diff --git a/docs/howtos/applications/cost.ipynb b/docs/howtos/applications/cost.ipynb
index b4085b0c2..433da0c18 100644
--- a/docs/howtos/applications/cost.ipynb
+++ b/docs/howtos/applications/cost.ipynb
@@ -236,7 +236,8 @@
     "from ragas.testset import TestsetGenerator\n",
     "from ragas.llms import llm_factory\n",
     "\n",
-    "tg = TestsetGenerator(llm=llm_factory(), knowledge_graph=kg)\n",
+    "# generator_llm = llm_factory()\n",
+    "tg = TestsetGenerator(llm=generator_llm, knowledge_graph=kg)\n",
     "# generating a testset\n",
     "testset = tg.generate(testset_size=10, token_usage_parser=get_token_usage_for_openai)"
    ]
diff --git a/tests/e2e/scratchpad_kg.json b/tests/e2e/scratchpad_kg.json
new file mode 100644
index 000000000..a132f5e58
--- /dev/null
+++ b/tests/e2e/scratchpad_kg.json
@@ -0,0 +1,1052578 @@
+{
+  "nodes": [
+    {
+      "id": "b570d71e-4f83-486e-b70f-b0c6217db455",
+      "properties": {
+        "page_content": "Mixture-Of-Experts With Expert Choice Routing\n\nYanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon Google, Mountain View, CA, USA {yanqiz, taole, hanxiaol, dunan, huangyp, vzhao, adai, zhifengc, qvl, jlaudon}@google.com\n\nAbstract\n\nSparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choice method. Instead of letting tokens select the top-k experts, we have experts selecting the top-k tokens. As a result, each token can be routed to a variable number of experts and each expert can have a fixed bucket size. We systematically study pre-training speedups using the same computational resources of the Switch Transformer top-1 and GShard top-2 gating of prior work and find that our method improves training convergence time by more than 2×. For the same computational cost, our method demonstrates higher performance in fine-tuning 11 selected tasks in the GLUE and SuperGLUE benchmarks. For a smaller activation cost, our method outperforms the T5 dense model in 7 out of the 11 tasks.\n\n1 Introduction\n\nScaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27]. The final model quality has been found to have a power-law relationship with the amount of data, model size, and compute time [16, 20]. However, training efficiency, which is defined as the total amount of computation used to achieve superior model quality than the state of the art system [21], should receive greater attention as we increase our efforts towards green AI [29].\n\nSparsely gated mixture-of-experts [31] (MoE) provides an effective way to scale model capacity given a fixed computational cost, and has recently played an important role in increasing the training efficiency of large-scale language models [10, 21]. MoE operate by adopting a number of experts, each as a sub-network, and by activating only one or a few experts for each input token. A gating network must be chosen and optimized in order to route each token to the most suited expert(s). For example, recent work has implemented sparse routing via k-means clustering [12], linear assignment to maximize token-expert affinities [22], or hashing [8, 28]. Many of the prior work use a routing strategy concerning the token choice, where each token selects the best one or two experts.\n\nWe argue that the independent token choice of prior work often leads to an imbalanced load of experts, which causes training inefficiency and sub-optimal training of the model. In order to mitigate this\n\nissue, previous sparsely gated networks introduce additional auxiliary losses as regularization to prevent too many tokens being routed to a single expert, but the effectiveness is still limited. Recent approaches [8, 22, 28] explore alternative strategies for routing, but they focus on pre-training only and do not demonstrate performance gain on downstream tasks. Moreover, none of the previous methods consider allocating a variable number of experts to each token based on importance, which can be beneficial.\n\nWe propose a very simple yet effective routing method we are calling expert choice. Unlike conventional MoE where tokens select one or two top-scoring experts, our method lets each expert pick the top-k tokens. Our method guarantees perfect load balancing, allows a variable number of experts for each token, and achieves substantial gains in training efficiency and downstream performance as demonstrated in our experiments. Our major contributions include:\n\nWe identify common pitfalls in conventional MoE such as load imbalance as described in Section 3.1. We then propose a heterogeneous, expert choice method to provide a fluid allocation of model parameters based on a learnt token-to-expert importance. This method intrinsically guarantees load balance without imposing an auxiliary loss.\n\nWe show our method provides over 2× faster training convergence in a 8B/64E (8 billion activated parameters, 64 experts) model, compared to the top-1 and top-2 gating counterparts in Switch Transformer [10] and GShard [21].\n\nWe show our method demonstrates strong scaling when increasing the number of experts from 16 to 128, evaluated in training perplexity.\n\nWe show our method demonstrates strong performance on downstream tasks selected from GLUE and SuperGLUE at all the evaluated scales. More specifically, our 8B/64E model outperforms a T5 11B dense model in 7 out of 11 tasks evaluated.\n\n2 Related Work\n\nScaling: Various approaches have been proposed to scale up neural network capacity to improve performance. Recent works have successfully scaled models to billions of parameters via various forms of model parallelism [2, 21, 26, 27, 33]. Model parallelism [30] splits weights and tensors across multiple cores while pipeline parallelism [18, 24] splits different layers across devices with micro-batches pipelined to the different layers. To enable continued scaling of neural networks, improving model training and serving efficiency has become a critical research area.\n\nConditional Computation: Computation decisions can be made dynamically based on the input [23, 25]. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation, by activating certain parameters and computation on demand, on a per-example or per-token basis [3]. Conditional convolution layers [1] with task-specific gating has been used to combat catastrophic forgetting when a sequence of learning problems are optimized. The gating decisions may be binary or sparse and continuous, stochastic or deterministic.\n\nMixture of Experts: Sparsely-gated MoE [31] is the first model to demonstrate massive improvements in model capacity, training time, or model quality with gating. Switch Transformer [10] simplifies the gating by selecting only the top expert per token using a softmax over the hidden state and demonstrates better scaling than previous work. All the prior work requires an auxiliary loss to explicitly encourage balancing. This loss term has to be carefully weighted to not overwhelm the primary loss. However, auxiliary loss does not guarantee balancing and a hard capacity factor has to be imposed. As a result, many tokens can still be unprocessed by the MoE layer. Hard MoE [12] with a single decoding layer can be efficiently trained to good effect on large scale hashtag prediction tasks.\n\nBase Layers [22] formulate a linear assignment that maximizes token-expert affinities while ensuring each expert receives an equal number of tokens. Hash layers [8, 28] devise hashing techniques on input tokens. However, the evaluations are limited to pre-training perplexity. THOR [? ] randomly activates experts during training and inference and is trained with a consistency regularization loss.\n\nTHOR has demonstrated strong performance on translation tasks. Different from these prior works, our method is a learnt method that enables heterogeneous MoE and effectively improves downstream fine-tuning performance.\n\n3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n5 Conclusion\n\nWe propose a new routing method for sparsely activated mixture-of-experts (MoE) models. This method addresses load imbalance and under-utilization of experts in conventional MoE methods, and enables selecting different numbers of experts for each token. Our model demonstrates more than 2x training efficiency improvements when compared to the state-of-the-art GShard and Switch Transformer models, and also achieves strong gains when finetuning on 11 datasets in the GLUE and SuperGLUE benchmark.\n\n6 Limitations\n\nThe expert choice method might not immediately apply to auto-regressive text generation as our current implementation takes in the past and future tokens to perform the top-k selection. One possible solution is to collect a large batch of input sequences, dispatch tokens of the same sequence into separate groups, and perform expert choice routing for each group. Another scenario where the expert choice method does not immediately apply is when the batch size becomes very small during serving or inference. A global top-k can be selected instead and we can cap the number of times each expert or token gets selected. We leave these possible improvements for future work.\n\nAnother long-standing issue with MoE has been the large memory footprint. Even though computational cost can be reduced using sparsely gated networks, the total number of parameters increases linearly or sub-linearly with the number of experts. Increasing the number of experts requires reservation of a large number of hardware devices. Therefore, dynamic (used) power is saved while static (reserved) power is not. Power saving techniques such as the ability to put hardware devices into low power states while not in use [17] can help with reducing the reserved power requirements.\n\nReferences\n\n[1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pages 3930–3939. Computer Vision Foundation / IEEE, 2020.\n\n[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. In Advances in Neural Information Processing Systems.\n\n[3] Kyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning, 2014.\n\n[4] Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems, 2021.\n\n[5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.\n\nTransformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.\n\n[6] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 933–941. JMLR.org, 2017.\n\n[7] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixtureof-experts, 2021.\n\n[8] Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan.\n\nTricks for training sparse translation models, 2021.\n\n[9] Richard L Dykstra. An iterative procedure for obtaining i-projections onto the intersection of convex sets. The annals of Probability, pages 975–984, 1985.\n\n[10] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.\n\n[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. NAS-FPN: learning scalable feature pyramid architecture for object detection. In CVPR, pages 7036–7045. Computer Vision Foundation / IEEE, 2019.\n\n[12] Sam Gross, Marc'Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision, 2017.\n\n[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.\n\n[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 630–645, Cham, 2016. Springer International Publishing.\n\n[15] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2016.\n\n[16] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.\n\n[17] Ping Huang, Zuocheng Xing, Tianran Wang, Qiang Wei, Hongyan Wang, and Guitao Fu. A brief survey on power gating design. In 2010 10th IEEE International Conference on Solid-State and Integrated Circuit Technology, pages 788–790, 2010.\n\n[18] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 103–112, 2019.\n\n[19] Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David A. Patterson. A domain-specific supercomputer for training deep neural networks. Commun. ACM, 63(7):67–78, 2020.\n\n[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.\n\n[21] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.\n\n[22] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6265–6274. PMLR, 18–24 Jul 2021.\n\n[24] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. New York, NY, USA, 2019. Association for Computing Machinery.\n\n[25] Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, Cédric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. In ICLR. OpenReview.net, 2021.\n\n[26] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.\n\n[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.\n\n[28] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models, 2021.\n\n[30] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 10435–10444, Red Hook, NY, USA, 2018. Curran Associates Inc.\n\n[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E.\n\nHinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In ICLR (Poster). OpenReview.net, 2017.\n\n[32] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 10–15 Jul 2018. [23] Min Lin, Jie Fu, and Yoshua Bengio. Conditional computation for continual learning, 2019.\n\n[29] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.\n\n[33] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.\n\n[34] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc.\n\n[35] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.\n\nGLUE: A multi-task benchmark and analysis platform for natural language understanding.\n\nIn Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, November 2018. Association for Computational Linguistics.\n\n[36] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake A. Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen. GSPMD: general and scalable parallelization for ML computation graphs. CoRR, abs/2105.04663, 2021.\n\n7 Checklist\n\n(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Yes (b) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes (c) Did you discuss any potential negative societal impacts of your work? N/A. Not any. (d) Did you describe the limitations of your work? Yes (a) Did you include the code, data, and instructions needed to reproduce the main experimental results? Yes. We include details in the experiment setup to help reproduce the main results. (b) Did you specify all the training details? Yes (c) Did you report error bars? Yes (d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? Yes (a) If your work uses existing assets, did you cite the creators? Yes (b) Did you mention the license of the assets? No. The used dataset is not released yet. (c) Did you include any new assets either in the supplemental material or as a URL? No. The dataset is not released yet.\n\n(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? No. Not using persons' data. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes. The dataset does not contain any personally identifiable information or offensive content.\n\nA Comparison On Fine-Tuning With A Dense Model\n\nOur 8B MoE model achieves stronger pre-training perplexity than its dense counterpart. However, a better perplexity does not always directly translate to downstream performance as demonstrated in Section 4.4. To this end, we compare fine-tuning performance of the 8B dense model and MoE model in Table 1. As shown in the table, our MoE model using expert choice routing consistently outperforms the dense model across the 11 tasks in GLUE and SuperGLUE.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 1: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nB Capacity Factor\n\nWe evaluate the downstream task fine-tuning performance by varying the capacity factors. Note that a capacity factor of n indicates on average how many experts each token can be received. EC-CF2 is our baseline expert choice, which matches GShard top-2 gating computational footprint. EC-CF1, however, matches Switch Transformer top-1 gating computational footprint. EC-CF0.5 further verifies that an aggressively lowered capacity factor can provide strong enough performance, that almost matches the top-2 gating baseline.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Top-2 78.1 87.0 88.3 85.0 82.6 90.1 90.7 81.6 94.7 68.2 67.2 83.0±0.3 EC-CAP2 78.2 88.0 88.5 85.7 83.0 90.8 91.1 80.0 95.4 70.4 64.1 83.2±0.4 EC-CAP3 78.5 91.7 89.3 86.3 83.5 90.9 91.1 81.8 94.9 70.0 65.6 84.0±0.4 EC-CF2 79.1 89.6 89.3 86.8 84.3 91.3 91.2 81.1 95.2 68.1 68.0 84.0±0.2 EC-CF1 77.4 90.6 88.0 85.5 83.6 90.3 91.2 79.8 95.3 66.5 64.9 83.0±0.2 EC-CF0.5 77.4 89.6 86.3 85.2 82.7 91.7 91.0 79.6 94.9 67.3 63.5 83.0 ±0.05 Hash Layers 76.1 85.2 86.7 83.4 82.5 90.0 90.3 75.7 94.0 67.4 63.3 81.3±1.0\n\nTable 2: Comparison between different routing methods in fine-tuning of 100M/64E models. We perform 3 independent fine-tuning runs for each method and report the average results. This gives more accurate difference between the variants of expert choice method, since they achieve close fine-tuning results. We do not report averaged results in other experiments.\n\nC Capped Expert Choice\n\nAs described in Section 4.5, the maximum number of experts each token is assigned can be capped by an entropy-regularized linear programming. Figure 1 compares the validation perplexity when training the 100M/64E models using the base expert choice method (EC-BASE), expert choice capped by two experts per token (EC-CAP2), expert choice capped by three experts per token (EC-CAP3), and GShard top-2 gating.\n\nAs shown in the figure, restricting the number of experts to 2 degrades the perplexity compared to the base expert choice method. This suggests that a more flexible allocation of experts (e.g. more than 2 experts for a token) can enhance model expressiveness. On the other hand, our EC-CAP2 and EC-CAP3 methods still outperform the top-2 gating method by a clear margin. We believe this confirms the effectiveness of a load balanced training, provided by our method. Finally, EC-CAP3 obtains comparable perplexity to EC-BASE. As indicated by Figure 3, only a little fraction of tokens use more than 3 experts therefore we see little or no difference between EC-BASE and EC-CAP3 variants. We present the fine-tuning results of these methods in Table 2.\n\nD Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [? ]. We use mod x to map a token ID to an expert ID. This in some way ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 2. Hashing based routing performs much worse than expert choice in terms of average scores and variance.\n\nE Fine-Tuning Details\n\nWe did a hyperparameter search for both baseline models and expert choice method. For fine-tuning of the 8B dense model, we use a constant learning rate of 0.0001 and a dropout rate of 0.1. We freeze the attention layer and feed-forward layer while leaving the embedding and layer normalization trainable. This setting has been found optimal for the 8B dense model. For MoE 8B/64E models including GShard top-2 gating and expert choice, we found continuing the learning rate from the pre-trained model while using a square root learning rate decay works better. In addition, we do not apply parameter freezing for fine-tuning MoE models. For models with 100M expert size, we use a constant learning rate of 0.0001 and no dropout is used.",
+        "document_metadata": {
+          "source": "../../datasets/Sample_Docs_Markdown/2202.09368v2.md"
+        },
+        "headlines": {
+          "1 Introduction": [],
+          "2 Related Work": [],
+          "3 Method": [
+            "3.1 Pitfalls Of Token-Choice Routing",
+            "3.2 Heterogeneous Moe Via Expert Choice",
+            "3.3 Expert Choice With Additional Constraint",
+            "3.4 Model Architecture"
+          ],
+          "4 Experiments": [
+            "4.1 Setup",
+            "4.2 Training Efficiency",
+            "4.3 Scaling The Number Of Experts",
+            "4.4 Fine-Tuning On Glue And Superglue",
+            "4.5 Heterogeneity Matters",
+            "4.6 Comparison With Hash Layer",
+            "4.7 Ablation"
+          ],
+          "5 Conclusion": [],
+          "6 Limitations": [],
+          "A Comparison On Fine-Tuning With A Dense Model": [],
+          "B Capacity Factor": [],
+          "C Capped Expert Choice": [],
+          "D Comparison With Hash Layer": [],
+          "E Fine-Tuning Details": []
+        },
+        "summary": "The paper introduces a new routing method for sparsely-activated Mixture-of-Experts (MoE) models called 'expert choice,' which addresses load imbalance and under-utilization issues in conventional MoE methods. Unlike traditional token-choice routing, expert choice allows experts to select the top-k tokens, ensuring perfect load balancing and enabling variable expert allocation per token. This method improves training efficiency, achieving over 2x faster convergence compared to existing models like Switch Transformer and GShard. It also demonstrates superior performance on downstream tasks in the GLUE and SuperGLUE benchmarks. The study highlights the benefits of flexible expert allocation and shows that expert choice outperforms hashing-based routing methods. The paper also discusses potential limitations and future improvements for the expert choice method.",
+        "summary_embedding": [
+          -0.009726986289024353,
+          -0.0337262786924839,
+          -0.023302342742681503,
+          -0.019878201186656952,
+          -0.025953780859708786,
+          0.04560471698641777,
+          -0.006874797400087118,
+          0.039665497839450836,
+          -0.006973279174417257,
+          0.034544438123703,
+          -0.025787118822336197,
+          -0.00363057479262352,
+          -0.01052999310195446,
+          0.0054657477885484695,
+          -0.024499278515577316,
+          0.036786794662475586,
+          0.010007280856370926,
+          0.003272630739957094,
+          0.006102093029767275,
+          -0.04048365354537964,
+          -0.018666116520762444,
+          -0.013522328808903694,
+          -0.03230207785964012,
+          0.008969432674348354,
+          0.02484775334596634,
+          0.0016713523073121905,
+          0.0026230288203805685,
+          0.003624893259257078,
+          -0.05311964824795723,
+          0.04339266195893288,
+          0.018681267276406288,
+          0.023711422458291054,
+          -0.016332851722836494,
+          -0.012832955457270145,
+          -0.039120059460401535,
+          -0.0197418425232172,
+          0.030786970630288124,
+          0.039968520402908325,
+          -0.019287310540676117,
+          0.03357476741075516,
+          0.018060073256492615,
+          0.02454473078250885,
+          -0.021272100508213043,
+          -0.029817303642630577,
+          -0.0061892117373645306,
+          -0.007840678095817566,
+          -0.008355814032256603,
+          0.01242387667298317,
+          0.016984347254037857,
+          -0.013007191941142082,
+          0.0212569497525692,
+          -0.01371171697974205,
+          0.03384748846292496,
+          0.008916404098272324,
+          -0.042332086712121964,
+          0.03984731063246727,
+          -0.008007340133190155,
+          0.012105703353881836,
+          0.03281721472740173,
+          0.010333028621971607,
+          -0.005287723150104284,
+          -0.021544819697737694,
+          -0.03378688171505928,
+          0.0010529992869123816,
+          0.00014014738553669304,
+          0.005950582213699818,
+          -0.003069985192269087,
+          -0.009742137044668198,
+          0.03227177634835243,
+          -0.003939277958124876,
+          -0.0740584209561348,
+          0.050089433789253235,
+          -0.026590125635266304,
+          -0.007628562860190868,
+          -0.028938541188836098,
+          -0.01736312545835972,
+          0.037362534552812576,
+          0.005662711802870035,
+          0.0098709212616086,
+          0.0106284748762846,
+          -0.0007892759749665856,
+          0.028923390433192253,
+          0.00910579226911068,
+          -0.024923507124185562,
+          0.04478655755519867,
+          -0.04790768027305603,
+          0.01801462098956108,
+          -0.04336235672235489,
+          0.022756904363632202,
+          1.6393929399782792e-05,
+          -0.020726662129163742,
+          -0.0013304532039910555,
+          -0.016666175797581673,
+          0.0218023881316185,
+          -0.01730252057313919,
+          0.026802239939570427,
+          0.013825350441038609,
+          -0.015802564099431038,
+          -0.029817303642630577,
+          0.025378039106726646,
+          0.003367325058206916,
+          0.010499690659344196,
+          -0.02075696364045143,
+          0.014332910999655724,
+          0.04393810033798218,
+          -0.02593863010406494,
+          -0.02271145209670067,
+          0.027681002393364906,
+          0.014408666640520096,
+          0.0633314698934555,
+          0.036544378846883774,
+          0.006624804809689522,
+          0.022362977266311646,
+          0.020575150847434998,
+          -0.028968842700123787,
+          -0.01963578537106514,
+          -0.046362269669771194,
+          -0.04757435619831085,
+          0.019878201186656952,
+          -0.04566532000899315,
+          0.0050869714468717575,
+          0.038544319570064545,
+          -0.013704141601920128,
+          -0.014446543529629707,
+          -0.04021093621850014,
+          0.017847958952188492,
+          0.0003908502112608403,
+          0.011976920068264008,
+          -0.0022082682698965073,
+          0.00033285003155469894,
+          -0.011825408786535263,
+          0.008878526277840137,
+          -0.03778676688671112,
+          -0.025196226313710213,
+          0.025181075558066368,
+          0.022862963378429413,
+          0.024120500311255455,
+          0.028802180662751198,
+          -0.03077181987464428,
+          -0.009984553791582584,
+          0.021378157660365105,
+          -0.04242299124598503,
+          -0.006556625012308359,
+          0.011332998983561993,
+          0.0026703758630901575,
+          0.003872992005199194,
+          0.032786913216114044,
+          0.052998438477516174,
+          -0.03172633796930313,
+          0.02065090648829937,
+          -0.010431510396301746,
+          0.04957429692149162,
+          0.00024691506405360997,
+          0.028968842700123787,
+          -0.019514575600624084,
+          -0.0134389977902174,
+          -0.029529431834816933,
+          -0.009795165620744228,
+          -0.01356020662933588,
+          -0.008484598249197006,
+          -0.002998017705976963,
+          -0.009575475007295609,
+          0.0040453351102769375,
+          0.0331808403134346,
+          0.00273855566047132,
+          0.030680913478136063,
+          0.014325335621833801,
+          0.0051475754007697105,
+          -0.08745196461677551,
+          0.026059838011860847,
+          -0.0033616432920098305,
+          0.005859675817191601,
+          0.0062460280023515224,
+          0.03013547509908676,
+          -0.0312718041241169,
+          0.020938776433467865,
+          0.026620427146553993,
+          0.0011145505122840405,
+          0.016969196498394012,
+          0.036847397685050964,
+          -0.04181694984436035,
+          0.009643655270338058,
+          0.012590537779033184,
+          -0.0033881578128784895,
+          0.04584713280200958,
+          0.011628445237874985,
+          0.019514575600624084,
+          0.018529755994677544,
+          0.003569970605894923,
+          0.00297907879576087,
+          0.006533898413181305,
+          0.03724132478237152,
+          0.003462019143626094,
+          -0.009833043441176414,
+          0.014976831153035164,
+          -0.010757259093225002,
+          -0.011219366453588009,
+          -0.037998881191015244,
+          0.014454119838774204,
+          0.008742166683077812,
+          -0.017454031854867935,
+          -0.02959003672003746,
+          0.0026949963066726923,
+          -0.031908150762319565,
+          0.026559824123978615,
+          -0.002445003716275096,
+          0.005738467443734407,
+          -0.010454237461090088,
+          -0.006757376715540886,
+          -0.035423196852207184,
+          0.028453707695007324,
+          -0.02095392718911171,
+          0.0053142374381423,
+          0.0007258309051394463,
+          -0.029438525438308716,
+          0.0060073984786868095,
+          0.017863109707832336,
+          -0.0025965142995119095,
+          0.0025302283465862274,
+          -0.02349930815398693,
+          0.024468975141644478,
+          0.025635607540607452,
+          0.004518806003034115,
+          -0.014037464745342731,
+          -0.013113250024616718,
+          -0.003605954349040985,
+          -0.009742137044668198,
+          -0.016196493059396744,
+          0.024468975141644478,
+          -0.014810169115662575,
+          0.00308892410248518,
+          0.021847840398550034,
+          0.014264730736613274,
+          -0.007045246660709381,
+          -0.004007457755506039,
+          -0.04066546633839607,
+          -0.01218903437256813,
+          -0.04760465770959854,
+          -0.011628445237874985,
+          -0.0034601252991706133,
+          -0.028696123510599136,
+          0.0061513339169323444,
+          -0.04481685906648636,
+          -0.01235569640994072,
+          0.02309022843837738,
+          0.018135828897356987,
+          0.0019488062243908644,
+          -0.009173971600830555,
+          -0.022605394944548607,
+          8.486965816700831e-05,
+          -0.01763584464788437,
+          -0.017044952139258385,
+          0.015393485315144062,
+          0.031059689819812775,
+          -0.03101423755288124,
+          0.04024123772978783,
+          0.003893824527040124,
+          -0.08066429197788239,
+          0.010393633507192135,
+          -0.026362858712673187,
+          0.07011914253234863,
+          -0.0025453795678913593,
+          0.014075342565774918,
+          0.06781618297100067,
+          -0.012431452050805092,
+          -0.02007516659796238,
+          0.03142331540584564,
+          -0.008249756880104542,
+          -1.1311515663692262e-05,
+          -0.02884763479232788,
+          -0.02739313244819641,
+          -0.053271155804395676,
+          -0.009704259224236012,
+          -0.010787560604512691,
+          0.03360506892204285,
+          -0.024317465722560883,
+          -0.0007656024536117911,
+          0.012620840221643448,
+          0.02983245439827442,
+          0.016635872423648834,
+          -0.0015681355725973845,
+          0.008272483013570309,
+          0.02881733328104019,
+          0.0274234339594841,
+          -0.0036154238041490316,
+          0.00814369972795248,
+          -0.034211114048957825,
+          -0.0068066176027059555,
+          -0.0493621826171875,
+          0.011302697472274303,
+          -0.02596893161535263,
+          -0.02596893161535263,
+          0.007643714081496,
+          -0.011476934887468815,
+          0.03748374432325363,
+          -0.024696242064237595,
+          -0.007503566797822714,
+          0.005037730094045401,
+          0.012855681590735912,
+          -0.017757052555680275,
+          -0.03612014651298523,
+          -0.0028502948116511106,
+          0.0026381798088550568,
+          0.007560383062809706,
+          -0.053240854293107986,
+          0.004499867558479309,
+          -0.018469152972102165,
+          0.0279688723385334,
+          -0.02139330841600895,
+          -0.013688990846276283,
+          0.042029064148664474,
+          0.011363301426172256,
+          0.027499189600348473,
+          -0.026468917727470398,
+          0.0017849853029474616,
+          0.012825379148125648,
+          -0.0020056227222085,
+          0.032514192163944244,
+          -0.030574856325984,
+          -0.03257479891180992,
+          0.001548249856568873,
+          -0.007840678095817566,
+          -0.0625436082482338,
+          -0.011386028490960598,
+          -0.00981789268553257,
+          0.017317671328783035,
+          0.0446956530213356,
+          0.04557441547513008,
+          -0.01068150345236063,
+          0.002174178371205926,
+          -0.02213571034371853,
+          -0.04542290419340134,
+          -0.02643861435353756,
+          0.0016486257081851363,
+          -0.019999410957098007,
+          -0.02789311669766903,
+          -0.0018313854234293103,
+          0.011211791075766087,
+          -0.013651113025844097,
+          0.01134057529270649,
+          -0.018560059368610382,
+          -0.01862066239118576,
+          -0.012332970276474953,
+          0.018575210124254227,
+          0.009901223704218864,
+          -0.0018796793883666396,
+          -0.015037435106933117,
+          -0.00980274099856615,
+          0.018969137221574783,
+          -0.011279970407485962,
+          0.0009388927719555795,
+          0.01188601367175579,
+          0.024347767233848572,
+          -0.007083124481141567,
+          0.010961798019707203,
+          -0.034756552428007126,
+          -0.01966608688235283,
+          0.007984613068401814,
+          0.002719616750255227,
+          0.020711511373519897,
+          0.008742166683077812,
+          -0.04275631532073021,
+          -0.044301725924015045,
+          0.016044981777668,
+          -0.0562104657292366,
+          0.028120383620262146,
+          -0.03327174484729767,
+          -0.012514783069491386,
+          -0.02671133354306221,
+          0.0013418165035545826,
+          -0.01828734017908573,
+          0.022438732907176018,
+          0.03778676688671112,
+          -0.0011486404109746218,
+          0.005503625608980656,
+          0.014658658765256405,
+          0.04384719207882881,
+          -0.05021063983440399,
+          -0.041180603206157684,
+          0.03754434734582901,
+          -0.01601468026638031,
+          0.0185146052390337,
+          0.00021744151308666915,
+          0.010348179377615452,
+          -0.014431392773985863,
+          0.022772056981921196,
+          0.013757170177996159,
+          -0.008764892816543579,
+          -0.030574856325984,
+          0.04945308715105057,
+          -0.023650817573070526,
+          -0.049513693898916245,
+          0.0018323323456570506,
+          0.018878230825066566,
+          0.018635815009474754,
+          -0.014870774000883102,
+          -0.02907489985227585,
+          -0.024317465722560883,
+          -0.006060427520424128,
+          -0.00947699323296547,
+          -0.03402930125594139,
+          0.011363301426172256,
+          0.0048445542342960835,
+          0.016666175797581673,
+          -0.00645056739449501,
+          -0.0073141781613230705,
+          -0.009370936080813408,
+          0.022559940814971924,
+          0.012635990977287292,
+          -0.03175663948059082,
+          -0.017484333366155624,
+          -0.02528713271021843,
+          0.004359719809144735,
+          0.010136065073311329,
+          0.022166013717651367,
+          0.024287162348628044,
+          -0.0323626808822155,
+          0.04218057543039322,
+          0.01550711877644062,
+          -0.008727015927433968,
+          -0.0019317612750455737,
+          0.02478714846074581,
+          0.04324115067720413,
+          0.0218023881316185,
+          0.026514369994401932,
+          0.002107892418280244,
+          8.705945947440341e-05,
+          -0.015620751306414604,
+          -0.0011391709558665752,
+          0.01862066239118576,
+          0.04911976307630539,
+          -0.012151156552135944,
+          0.01716616004705429,
+          -0.03578682243824005,
+          0.00396200455725193,
+          -0.02580226957798004,
+          -0.030786970630288124,
+          0.03745344281196594,
+          -0.015060162171721458,
+          0.0191964041441679,
+          0.025044716894626617,
+          -0.022469034418463707,
+          -0.011075431481003761,
+          -0.019423669204115868,
+          -0.026181045919656754,
+          -0.01702980138361454,
+          -0.03281721472740173,
+          -0.018135828897356987,
+          -0.002653330797329545,
+          -0.035635314881801605,
+          -0.04987731948494911,
+          -0.004329417832195759,
+          -0.046392571181058884,
+          -0.010507266037166119,
+          0.017378276214003563,
+          -0.011166337877511978,
+          -0.025711363181471825,
+          -0.005336964037269354,
+          0.003973367623984814,
+          -0.022196315228939056,
+          -0.0066588944755494595,
+          0.008439145050942898,
+          0.035938333719968796,
+          -0.02298417128622532,
+          -0.013060221448540688,
+          -0.0024317465722560883,
+          -0.02887793630361557,
+          -0.007878555916249752,
+          0.008999735116958618,
+          0.007723257411271334,
+          -0.0012357590021565557,
+          -0.05196816474199295,
+          -0.006916462909430265,
+          -0.012992041185498238,
+          0.029196109622716904,
+          0.026468917727470398,
+          0.0012168202083557844,
+          -0.02783251367509365,
+          0.008749742060899734,
+          -0.0072535742074251175,
+          0.02339324913918972,
+          -0.007992188446223736,
+          -0.022908415645360947,
+          -0.02034788578748703,
+          0.030786970630288124,
+          -0.0159086212515831,
+          0.009113367646932602,
+          0.009976978413760662,
+          0.0121360057964921,
+          -0.03899884968996048,
+          -0.001445980160497129,
+          0.0005648507503792644,
+          -0.010128489695489407,
+          0.00423093605786562,
+          -0.006738437805324793,
+          -0.007079336792230606,
+          -0.022090258076786995,
+          -0.0218023881316185,
+          0.03818069398403168,
+          -0.009113367646932602,
+          -0.0035661826841533184,
+          -0.01088604237884283,
+          0.03436262533068657,
+          -0.00595815759152174,
+          -0.01993880607187748,
+          -0.019347915425896645,
+          0.014431392773985863,
+          0.05105910077691078,
+          -0.028529461473226547,
+          -0.03984731063246727,
+          0.0011136035900563002,
+          0.030150625854730606,
+          0.004140029661357403,
+          -0.016908591613173485,
+          0.0035851215943694115,
+          0.056392278522253036,
+          -0.011742077767848969,
+          0.028362801298499107,
+          0.027771908789873123,
+          0.02295386977493763,
+          -0.02021152526140213,
+          -0.01678738370537758,
+          -0.020999381318688393,
+          0.011098157614469528,
+          -0.03027183562517166,
+          0.027059808373451233,
+          0.018969137221574783,
+          0.01538590993732214,
+          0.05054396390914917,
+          -0.03178694099187851,
+          -0.02515077404677868,
+          0.008499749936163425,
+          0.0021722845267504454,
+          0.011007251217961311,
+          -0.007727045100182295,
+          -0.01801462098956108,
+          -0.001950700068846345,
+          -0.014022313989698887,
+          -0.019817598164081573,
+          -0.020938776433467865,
+          0.00954517349600792,
+          0.015696506947278976,
+          0.002649543108418584,
+          0.010863316245377064,
+          0.0016590419691056013,
+          0.01686313934624195,
+          0.01321930717676878,
+          -0.0444229319691658,
+          -7.936555630294606e-05,
+          0.03378688171505928,
+          -0.0293476190418005,
+          -0.012264790013432503,
+          -0.016060132533311844,
+          -0.004439263138920069,
+          -0.01706010289490223,
+          -0.02762039750814438,
+          -0.006874797400087118,
+          -0.01889338344335556,
+          -0.00814369972795248,
+          -0.028196139261126518,
+          -0.009992130100727081,
+          0.01949942484498024,
+          0.022332675755023956,
+          0.001196934375911951,
+          -0.006783891003578901,
+          -0.005901341326534748,
+          0.02092362567782402,
+          0.032514192163944244,
+          0.01952972821891308,
+          0.0058331615291535854,
+          -0.015158643946051598,
+          -0.004295328166335821,
+          -0.012370847165584564,
+          0.016060132533311844,
+          0.012537509202957153,
+          6.344509893096983e-05,
+          0.03493836522102356,
+          0.004170331638306379,
+          -0.022726602852344513,
+          -0.001230077352374792,
+          0.024923507124185562,
+          -0.021590271964669228,
+          -0.0108330138027668,
+          0.040029123425483704,
+          0.0085679292678833,
+          -0.0024506852496415377,
+          -0.019287310540676117,
+          -0.019075196236371994,
+          0.02630225569009781,
+          -0.02133270539343357,
+          0.002083271974697709,
+          0.0482713058590889,
+          0.00035344602656550705,
+          -0.029802151024341583,
+          0.017954016104340553,
+          -0.013242034241557121,
+          -0.018560059368610382,
+          0.01848430372774601,
+          0.0012859469279646873,
+          -0.039968520402908325,
+          0.019014591351151466,
+          0.018408548086881638,
+          0.022635696455836296,
+          -0.02918095886707306,
+          0.0002127068000845611,
+          0.03233237937092781,
+          0.004037759732455015,
+          0.001595596899278462,
+          -0.008530051447451115,
+          -0.0006065162015147507,
+          0.010431510396301746,
+          0.001746160676702857,
+          0.02854461409151554,
+          -0.0030434709042310715,
+          0.027771908789873123,
+          -0.006727074272930622,
+          -0.008863375522196293,
+          0.019181253388524055,
+          -0.02109028771519661,
+          0.013166278600692749,
+          0.0685434341430664,
+          0.0021950111258774996,
+          0.025847723707556725,
+          0.007181606721132994,
+          0.024347767233848572,
+          0.02018122375011444,
+          -0.027271922677755356,
+          -0.02021152526140213,
+          -0.005397567991167307,
+          -0.0014033677289262414,
+          -0.03787767142057419,
+          0.011128460057079792,
+          -0.021302402019500732,
+          -0.013249609619379044,
+          -0.0022347825579345226,
+          -0.002316219499334693,
+          -0.006405114196240902,
+          0.034211114048957825,
+          -0.0014630251098424196,
+          -0.018560059368610382,
+          -0.035423196852207184,
+          0.037726160138845444,
+          0.04048365354537964,
+          0.0389382466673851,
+          -0.005984671879559755,
+          0.03548380360007286,
+          0.0030756669584661722,
+          0.0025302283465862274,
+          0.041453324258327484,
+          0.004791525658220053,
+          -0.011567841283977032,
+          0.02295386977493763,
+          0.012416300363838673,
+          0.03351416438817978,
+          -0.02045394293963909,
+          -0.02630225569009781,
+          -0.003299145260825753,
+          -0.026423463597893715,
+          0.012492056004703045,
+          -0.019726691767573357,
+          0.0018768386216834188,
+          0.007117214612662792,
+          0.015196521766483784,
+          -0.016044981777668,
+          0.014719262719154358,
+          -0.006102093029767275,
+          -0.021014532074332237,
+          0.011719351634383202,
+          -0.007780073676258326,
+          0.011961768381297588,
+          -0.02262054570019245,
+          0.001138224033638835,
+          -0.04063516482710838,
+          0.008280059322714806,
+          0.014022313989698887,
+          -0.01598437689244747,
+          0.03205966204404831,
+          0.020969079807400703,
+          -0.01801462098956108,
+          -0.0009682480012997985,
+          0.0034714885987341404,
+          -0.010393633507192135,
+          -0.011067855171859264,
+          -0.019923655316233635,
+          0.02786281518638134,
+          -0.015802564099431038,
+          -0.025908326730132103,
+          0.02454473078250885,
+          -0.01743887923657894,
+          0.015151068568229675,
+          -0.004299115855246782,
+          0.00888610165566206,
+          -0.019120648503303528,
+          0.02284781076014042,
+          -0.023741723969578743,
+          -0.0009128518868237734,
+          -0.020287280902266502,
+          0.0028389315120875835,
+          0.014135947450995445,
+          -0.018272189423441887,
+          -0.02139330841600895,
+          0.019150950014591217,
+          -0.005420294590294361,
+          -0.003931702114641666,
+          0.010014856234192848,
+          0.0030983935575932264,
+          -0.0029715034179389477,
+          -0.004549108445644379,
+          -0.03281721472740173,
+          -0.02915065549314022,
+          0.0033086147159337997,
+          0.011386028490960598,
+          0.015257125720381737,
+          0.027574945241212845,
+          0.03715042024850845,
+          0.006052851676940918,
+          0.0036949669010937214,
+          0.0009327376610599458,
+          -0.020726662129163742,
+          0.042635105550289154,
+          -0.020469093695282936,
+          0.040847279131412506,
+          0.032544493675231934,
+          -0.0197418425232172,
+          -0.023908386006951332,
+          0.0029052174650132656,
+          0.03115059621632099,
+          0.03151422366499901,
+          0.025211378931999207,
+          0.012310243211686611,
+          0.03690800443291664,
+          -0.004359719809144735,
+          -0.032211173325777054,
+          0.015802564099431038,
+          0.022256920114159584,
+          -0.008105821907520294,
+          0.009673957712948322,
+          0.027438584715127945,
+          0.013279912061989307,
+          -0.007757347077131271,
+          0.030150625854730606,
+          0.0012234487803652883,
+          0.02396899089217186,
+          0.02498411200940609,
+          -0.02190844528377056,
+          0.01743887923657894,
+          -0.0001154677156591788,
+          0.006727074272930622,
+          -0.009840618818998337,
+          0.004348356742411852,
+          0.011810258030891418,
+          0.018272189423441887,
+          0.018908534198999405,
+          0.003246116451919079,
+          -0.07060398161411285,
+          0.019726691767573357,
+          0.004507442936301231,
+          -0.02160542458295822,
+          0.00390140013769269,
+          -0.0024866692256182432,
+          0.012704171240329742,
+          0.02105998620390892,
+          0.012461753562092781,
+          -0.01790856383740902,
+          0.0013626492582261562,
+          0.011961768381297588,
+          -0.00595815759152174,
+          0.04730163514614105,
+          0.0019791084341704845,
+          -0.01437078882008791,
+          -0.016105586662888527,
+          0.013726867735385895,
+          0.02525683119893074,
+          -0.02355991117656231,
+          0.006325571332126856,
+          0.005621046293526888,
+          -0.004147605039179325,
+          -0.010742107406258583,
+          -0.030377892777323723,
+          -0.005280147306621075,
+          -0.03436262533068657,
+          0.023150833323597908,
+          0.015491968020796776,
+          0.014764715917408466,
+          -0.025044716894626617,
+          -0.004280176945030689,
+          0.0046400148421525955,
+          -0.021287251263856888,
+          0.003261267440393567,
+          0.01161329448223114,
+          0.0005104016163386405,
+          -0.020984230563044548,
+          0.00959820207208395,
+          0.013469300232827663,
+          0.06848283112049103,
+          0.0007442962378263474,
+          0.029771849513053894,
+          0.0422411784529686,
+          -0.0012243957025930285,
+          -0.006371024530380964,
+          0.0036286809481680393,
+          -0.019953956827521324,
+          0.003473382443189621,
+          0.025878025218844414,
+          -0.03224147483706474,
+          -0.055058982223272324,
+          -0.007545232307165861,
+          -0.04624105989933014,
+          0.001489539397880435,
+          0.014711687341332436,
+          0.023044776171445847,
+          -0.0037782976869493723,
+          -0.034817155450582504,
+          0.016908591613173485,
+          -0.003071879269555211,
+          0.007014944683760405,
+          -0.02593863010406494,
+          -0.0294839795678854,
+          -0.026014383882284164,
+          -0.02339324913918972,
+          -0.012908710166811943,
+          -0.022726602852344513,
+          0.012590537779033184,
+          -0.0015103721525520086,
+          -0.06587684899568558,
+          -0.009780014865100384,
+          0.016166189685463905,
+          0.001200722181238234,
+          0.011939042247831821,
+          0.011083006858825684,
+          -0.007083124481141567,
+          -0.0018275976181030273,
+          -0.022559940814971924,
+          0.03124150261282921,
+          0.005177877843379974,
+          0.03584742918610573,
+          -0.006503595970571041,
+          -0.009128518402576447,
+          0.01073453202843666,
+          0.04527139291167259,
+          0.0017916138749569654,
+          0.010226971469819546,
+          -0.020560000091791153,
+          0.0015065843472257257,
+          -0.02284781076014042,
+          -0.0031135445460677147,
+          -0.008863375522196293,
+          -0.0021211495622992516,
+          0.015893470495939255,
+          0.0019753205124288797,
+          -0.010204244405031204,
+          0.025438643991947174,
+          0.02251448854804039,
+          -0.009605777449905872,
+          -0.015969226136803627,
+          -0.02437806874513626,
+          -0.014545026235282421,
+          -0.0002627763315103948,
+          -0.010007280856370926,
+          -0.022453883662819862,
+          -0.020878173410892487,
+          -0.002566212322562933,
+          -0.02559015527367592,
+          0.022529639303684235,
+          0.01220418605953455,
+          -4.882668508798815e-05,
+          -0.006723286584019661,
+          -0.038544319570064545,
+          -0.003727162955328822,
+          -0.03118089959025383,
+          -0.005374841392040253,
+          0.03287781774997711,
+          -0.03745344281196594,
+          -0.0002231231628684327,
+          0.014491996727883816,
+          -0.01993880607187748,
+          0.0028938541654497385,
+          0.008961857296526432,
+          0.02769615314900875,
+          -0.027408283203840256,
+          0.0001795638381736353,
+          0.006954340264201164,
+          0.016908591613173485,
+          -0.009878496639430523,
+          0.027196168899536133,
+          -0.04275631532073021,
+          0.011037553660571575,
+          -0.037120118737220764,
+          -0.020423639565706253,
+          -0.01523439958691597,
+          0.008363389410078526,
+          -0.005018791649490595,
+          -0.02637801133096218,
+          -0.010454237461090088,
+          0.006026337388902903,
+          -0.00693918950855732,
+          -0.01902974210679531,
+          -0.010620899498462677,
+          0.027438584715127945,
+          0.00931033119559288,
+          0.007977037690579891,
+          -0.02955973520874977,
+          -0.01631770096719265,
+          0.006280118133872747,
+          0.052089374512434006,
+          -0.01584801822900772,
+          0.014938953332602978,
+          -0.006639956030994654,
+          0.02342355251312256,
+          -0.028484009206295013,
+          -0.01872672140598297,
+          0.0315445251762867,
+          0.013946558348834515,
+          0.007064185570925474,
+          0.028953691944479942,
+          0.027196168899536133,
+          0.027105262503027916,
+          -0.04654408246278763,
+          0.009886072017252445,
+          -0.002613559365272522,
+          -0.008757317438721657,
+          0.03402930125594139,
+          0.01916610263288021,
+          0.022150862962007523,
+          0.0009744031121954322,
+          -0.01730252057313919,
+          0.0139844361692667,
+          0.014885924756526947,
+          0.007681591901928186,
+          -0.014242004603147507,
+          -0.02660527639091015,
+          0.003367325058206916,
+          -0.007992188446223736,
+          0.00047276070108637214,
+          0.020726662129163742,
+          -0.009052763693034649,
+          0.030999086797237396,
+          -0.014605630189180374,
+          -0.02593863010406494,
+          0.008916404098272324,
+          -0.02268115058541298,
+          -0.0041741193272173405,
+          0.0004476667381823063,
+          0.010083036497235298,
+          -0.023847781121730804,
+          -0.04669559374451637,
+          0.019408518448472023,
+          0.028393102809786797,
+          0.005579380784183741,
+          0.01328748743981123,
+          -0.035332292318344116,
+          0.008992158807814121,
+          0.012310243211686611,
+          0.011348150670528412,
+          0.0019847899675369263,
+          0.035362593829631805,
+          -0.00028834378463216126,
+          0.012885984033346176,
+          -0.008620957843959332,
+          0.028362801298499107,
+          0.00667025800794363,
+          -0.0323626808822155,
+          -0.0378473699092865,
+          -0.01513591781258583,
+          0.015257125720381737,
+          0.006692984607070684,
+          -0.011840560473501682,
+          -0.02278720773756504,
+          -0.038817036896944046,
+          0.015787413343787193,
+          -0.012916285544633865,
+          -0.038271598517894745,
+          -0.010158791206777096,
+          -0.009742137044668198,
+          0.016635872423648834,
+          -0.006916462909430265,
+          -0.016802534461021423,
+          -0.016090434044599533,
+          0.005821797996759415,
+          7.161247049225494e-05,
+          0.0244083721190691,
+          -0.001251857029274106,
+          -0.006855858489871025,
+          -0.016393456608057022,
+          -0.01188601367175579,
+          -0.0035074723418802023,
+          0.03602924197912216,
+          -0.014219277538359165,
+          -0.02383263036608696,
+          0.014984406530857086,
+          0.02657497487962246,
+          0.011636020615696907,
+          0.03227177634835243,
+          0.007181606721132994,
+          -0.014022313989698887,
+          0.003016956616193056,
+          -0.033029329031705856,
+          -0.0046400148421525955,
+          0.005196816753596067,
+          -0.006427840795367956,
+          -0.015651054680347443,
+          -0.01815097965300083,
+          -0.015832867473363876,
+          0.030756669119000435,
+          -0.02487805485725403,
+          0.0218023881316185,
+          0.030347591266036034,
+          0.003174148965626955,
+          -0.02915065549314022,
+          0.013598084449768066,
+          -0.0072005451656877995,
+          0.01862066239118576,
+          0.0019014590652659535,
+          0.028014326468110085,
+          -0.007461901288479567,
+          -0.004818039946258068,
+          -0.02501441352069378,
+          -0.03696860745549202,
+          -0.023484155535697937,
+          -0.008817922323942184,
+          0.0027537066489458084,
+          0.020984230563044548,
+          -0.0375746488571167,
+          -0.017757052555680275,
+          0.009855770505964756,
+          0.015757111832499504,
+          -0.03027183562517166,
+          -0.0025908327661454678,
+          -0.010158791206777096,
+          -0.0004055278259329498,
+          0.005140000022947788,
+          -0.02227207086980343,
+          -0.008977008052170277,
+          -0.015726808458566666,
+          -0.0010435298318043351,
+          0.006166485138237476,
+          -0.0007409819518215954,
+          -0.05805889517068863,
+          -0.013969285413622856,
+          0.018635815009474754,
+          0.0026400736533105373,
+          0.008908828720450401,
+          0.015938924625515938,
+          -0.0111739132553339,
+          0.0031533162109553814,
+          -0.026756787672638893,
+          0.023893235251307487,
+          -0.009492144919931889,
+          0.009355784393846989,
+          -0.013742019422352314,
+          0.012514783069491386,
+          0.014014738611876965,
+          0.004795313347131014,
+          0.0010681503918021917,
+          -0.0054051438346505165,
+          0.01601468026638031,
+          0.028377952054142952,
+          -0.023756874725222588,
+          0.011454207822680473,
+          -0.0002514130319468677,
+          -0.0032158142421394587,
+          -0.029196109622716904,
+          0.00509454682469368,
+          0.003931702114641666,
+          -0.005109698045998812,
+          0.016651025041937828,
+          -0.036544378846883774,
+          0.01662072166800499,
+          0.02177208475768566,
+          0.032544493675231934,
+          0.014832896180450916,
+          -0.017287369817495346,
+          -0.011658747680485249,
+          -0.012385998852550983,
+          -0.002624922664836049,
+          0.022181164473295212,
+          0.012120855040848255,
+          -0.01815097965300083,
+          -0.024287162348628044,
+          0.011249667964875698,
+          0.0005283934879116714,
+          -0.01203752402216196,
+          0.010795135982334614,
+          -0.013166278600692749,
+          0.01349202636629343,
+          0.012711746618151665,
+          0.00866641104221344,
+          0.00888610165566206,
+          0.022862963378429413,
+          -0.011878437362611294,
+          -0.017711598426103592,
+          0.00656041270121932,
+          -0.008916404098272324,
+          -0.0034089903347194195,
+          -0.012598113156855106,
+          0.010795135982334614,
+          0.014878349378705025,
+          0.06490717828273773,
+          -0.013143551535904408,
+          0.0009526234352961183,
+          -0.020438792183995247,
+          -0.031938452273607254,
+          -0.0017944546416401863,
+          -0.020908474922180176,
+          0.0111739132553339,
+          0.008370965719223022,
+          0.01598437689244747,
+          0.012022373266518116,
+          -0.0021912232041358948,
+          0.00795431062579155,
+          0.019317612051963806,
+          -0.010423935018479824,
+          7.255941454786807e-05,
+          -0.006503595970571041,
+          0.0013683309080079198,
+          0.0035851215943694115,
+          -0.011719351634383202,
+          -0.0026949963066726923,
+          0.035695917904376984,
+          0.01068150345236063,
+          0.011651171371340752,
+          -0.03445352986454964,
+          0.0020435003098100424,
+          0.020150920376181602,
+          0.007602048572152853,
+          -0.008302785456180573,
+          -0.01563590206205845,
+          0.01706010289490223,
+          -0.015317730605602264,
+          -0.0022499337792396545,
+          0.002888172399252653,
+          -0.010151215828955173,
+          0.04581683129072189,
+          -0.019787294790148735,
+          -0.0011742077767848969,
+          0.031847547739744186,
+          0.0030207443051040173,
+          -0.0040983641520142555,
+          0.005408931523561478,
+          -0.003473382443189621,
+          -0.0069013116881251335,
+          -0.02139330841600895,
+          0.0031381649896502495,
+          0.00667025800794363,
+          -0.019287310540676117,
+          -0.0072005451656877995,
+          -0.007719469256699085,
+          0.006897523999214172,
+          0.023105379194021225,
+          -0.009492144919931889,
+          -0.00016145357221830636,
+          -0.008575504645705223,
+          0.009848194196820259,
+          -0.0011912527261301875,
+          -0.044301725924015045,
+          -0.006147546228021383,
+          0.021181194111704826,
+          0.000973456131760031,
+          0.0005757405888289213,
+          -0.007450537756085396,
+          0.005007428117096424,
+          -0.017423728480935097,
+          -0.008454295806586742,
+          0.02369627170264721,
+          0.01736312545835972,
+          -0.009113367646932602,
+          0.010863316245377064,
+          -0.0035680767614394426,
+          0.00433320552110672,
+          -0.020014561712741852,
+          -0.01095422264188528,
+          0.003462019143626094,
+          0.015272277407348156,
+          -0.0009819786064326763,
+          -0.029029447585344315,
+          -0.0012272365856915712,
+          -0.00024194362049456686,
+          0.02918095886707306,
+          0.01486319862306118,
+          -0.008961857296526432,
+          -0.021211495622992516,
+          -0.013242034241557121,
+          -0.001237652963027358,
+          -0.015726808458566666,
+          -0.009620928205549717,
+          0.011855711229145527,
+          0.0009137988090515137,
+          0.008992158807814121,
+          -0.018135828897356987,
+          -0.027347678318619728,
+          -0.016166189685463905,
+          0.0018465365283191204,
+          -0.012870832346379757,
+          0.003340810537338257,
+          0.015514694154262543,
+          -0.03430201858282089,
+          -0.015060162171721458,
+          -0.03805948421359062,
+          0.008280059322714806,
+          -0.0029658216517418623,
+          -0.02024182677268982,
+          -0.023605365306138992,
+          0.009484568610787392,
+          0.019181253388524055,
+          -0.0015548784285783768,
+          0.019847899675369263,
+          -0.000889651826582849,
+          0.020060013979673386,
+          -0.0073141781613230705,
+          0.0031703610438853502,
+          -0.013007191941142082,
+          0.0030245319940149784,
+          0.008333087898790836,
+          0.02890823967754841,
+          -0.023075077682733536,
+          -0.006503595970571041,
+          -0.004685468040406704,
+          0.004746072459965944,
+          0.012658718042075634,
+          0.014613205567002296,
+          0.020741812884807587,
+          -0.008272483013570309,
+          -0.0027612822595983744,
+          0.012484480626881123,
+          -0.003799130441620946,
+          0.007295239716768265,
+          0.002390081062912941,
+          0.0019677451346069574,
+          0.004162756260484457,
+          0.008477022871375084,
+          0.0053710537031292915,
+          0.027650700882077217,
+          -0.0028275682125240564,
+          0.0031703610438853502,
+          0.024620486423373222,
+          0.02322658896446228,
+          -0.018878230825066566,
+          -0.028074929490685463,
+          0.002024561632424593,
+          0.050695475190877914,
+          0.007245998363941908,
+          -0.006810405291616917,
+          -0.03927157074213028,
+          -0.0031760428100824356,
+          0.028635520488023758,
+          0.023650817573070526,
+          0.01584801822900772,
+          -0.010567869991064072,
+          -0.012098127976059914,
+          0.02772645466029644,
+          0.013295062817633152,
+          0.02616589516401291,
+          -0.025302285328507423,
+          0.010704229585826397,
+          -0.03869583085179329,
+          0.0449683703482151,
+          0.004439263138920069,
+          -0.014476845972239971,
+          0.0011884119594469666,
+          0.010386057198047638,
+          -0.004537744913250208,
+          0.006431628484278917,
+          -5.8059369621332735e-05,
+          -0.0008086882880888879,
+          0.011923890560865402,
+          0.012007221579551697,
+          0.004265025723725557,
+          -0.02399929240345955,
+          0.007359631359577179,
+          0.004272601567208767,
+          0.03005971945822239,
+          0.005431658122688532,
+          -0.0013863227795809507,
+          0.012385998852550983,
+          -0.005810434930026531,
+          0.008855799213051796,
+          -0.010719381272792816,
+          0.020469093695282936,
+          -0.02918095886707306,
+          1.0986003871948924e-05,
+          0.02051454596221447,
+          -0.0015785519499331713,
+          0.0038199631962925196,
+          -0.023741723969578743,
+          0.01186328660696745,
+          -0.052968136966228485,
+          0.021166043356060982,
+          -0.018741872161626816,
+          0.021741783246397972,
+          0.019590331241488457,
+          -0.009734561666846275,
+          -0.012052674777805805,
+          0.03545350208878517,
+          0.0026760573964565992,
+          -0.014264730736613274,
+          -0.010643625631928444,
+          -0.017999470233917236,
+          -0.02830219641327858,
+          0.041968461126089096,
+          -0.0006117243901826441,
+          0.0028275682125240564,
+          0.0024222771171480417,
+          -0.010666352696716785,
+          -0.008923979476094246,
+          0.001954487757757306,
+          0.020499395206570625,
+          -0.03369597718119621,
+          -0.011128460057079792,
+          -0.008689138107001781,
+          -0.02566591091454029,
+          -0.00020300064352340996,
+          -0.011961768381297588,
+          0.026211349293589592,
+          0.01240872498601675,
+          -0.03696860745549202,
+          0.003780191531404853,
+          0.023650817573070526,
+          0.001603172393515706,
+          0.018120678141713142,
+          0.002130619017407298,
+          0.0010795136913657188,
+          0.009742137044668198,
+          -0.010476963594555855,
+          0.026559824123978615,
+          -0.009204274043440819,
+          0.008658835664391518,
+          -0.001950700068846345,
+          -0.017287369817495346,
+          0.017969166859984398,
+          0.004083212930709124,
+          0.024120500311255455,
+          0.011954193003475666,
+          0.011605718173086643,
+          0.01112088467925787,
+          -0.019878201186656952,
+          0.020090317353606224,
+          -0.0018891488434746861,
+          0.009075489826500416,
+          0.024726543575525284,
+          -0.005863463506102562,
+          0.020832719281315804,
+          0.015257125720381737,
+          -0.00020229043730068952,
+          0.011295121163129807,
+          -0.020120618864893913,
+          -0.011779955588281155,
+          0.0085679292678833,
+          -0.009605777449905872,
+          -0.010325453244149685,
+          0.009901223704218864,
+          -0.010105762630701065,
+          -0.015052586793899536,
+          -0.0071247899904847145,
+          0.016635872423648834,
+          0.0028976418543606997,
+          -0.02610529027879238,
+          -1.7326075976598077e-05,
+          0.008901252411305904,
+          0.006166485138237476,
+          0.018741872161626816,
+          0.011317848227918148,
+          -0.006692984607070684,
+          0.010363331064581871,
+          0.011476934887468815,
+          0.0053104497492313385,
+          -0.005079395603388548,
+          -0.013431422412395477,
+          -0.0024241709616035223,
+          0.018090376630425453,
+          0.009810317307710648,
+          -0.01922670565545559,
+          0.004564259201288223,
+          0.0008877579239197075,
+          -0.01420412678271532,
+          -0.019347915425896645,
+          -0.0038483713287860155,
+          -0.004170331638306379,
+          0.0027915844693779945,
+          0.0509984977543354,
+          -0.0006216673064045608,
+          0.007893706671893597,
+          -0.025741664692759514,
+          -0.006299056578427553,
+          -0.00027556007262319326,
+          -0.0036097420379519463,
+          0.01272689737379551,
+          -0.01824188604950905,
+          -0.03724132478237152,
+          0.012711746618151665,
+          -0.0012584856012836099,
+          0.014923802576959133,
+          -0.00297907879576087,
+          0.014226853847503662,
+          -0.017347972840070724,
+          0.01916610263288021,
+          -0.0011117096291854978,
+          -0.016029831022024155,
+          0.0029374135192483664,
+          -0.011143610812723637,
+          -0.021317552775144577,
+          0.005916492082178593,
+          -0.006848283112049103,
+          -0.03896854817867279,
+          0.0029885482508689165,
+          -0.010423935018479824,
+          0.0011798894265666604,
+          -0.003912763670086861,
+          0.020166072994470596,
+          -0.0061513339169323444,
+          0.0008697660523466766,
+          0.02177208475768566,
+          0.0017026013229042292,
+          0.01601468026638031,
+          -0.014499573037028313,
+          0.010999675840139389,
+          0.012598113156855106,
+          0.02210540883243084,
+          -0.032180868089199066,
+          -0.01393140759319067,
+          0.014211702160537243,
+          0.0026116655208170414,
+          -0.005382417235523462,
+          0.020635755732655525,
+          -0.01294658798724413,
+          0.019650936126708984,
+          -0.0005577487172558904,
+          -0.011704200878739357,
+          -0.03524138405919075,
+          -0.016529815271496773,
+          -3.343888238305226e-05,
+          -0.008439145050942898,
+          0.025317436084151268,
+          0.01760554127395153,
+          -0.0108330138027668,
+          0.026468917727470398,
+          0.0037574649322777987,
+          0.014598054811358452,
+          -0.014257155358791351,
+          -0.005999823100864887,
+          -0.0003773563075810671,
+          -0.0119314668700099,
+          0.015264702029526234,
+          -0.03512017801403999,
+          0.0008342557121068239,
+          -0.00839369185268879,
+          0.005242269951850176,
+          -0.0254992488771677,
+          0.018257036805152893,
+          0.025817420333623886,
+          -0.0031419529113918543,
+          -0.005037730094045401,
+          -0.022802358493208885,
+          0.028968842700123787,
+          0.03049910068511963,
+          0.0024828813038766384,
+          -0.0013825349742546678,
+          -0.00352072948589921,
+          0.0033218718599528074,
+          0.0211205892264843,
+          -0.037998881191015244,
+          -0.009098216891288757,
+          -0.028696123510599136,
+          0.042332086712121964,
+          -0.008098246529698372,
+          -0.013431422412395477,
+          0.01862066239118576,
+          -0.0024715180043131113,
+          0.02355991117656231,
+          -0.004689255729317665,
+          -0.004855917301028967,
+          0.03148392215371132,
+          0.011514811776578426,
+          0.02772645466029644,
+          -0.04015033319592476,
+          0.014704111963510513,
+          -0.006867222022265196,
+          0.004867280833423138,
+          -0.013484450988471508,
+          0.005302873905748129,
+          0.02580226957798004,
+          -0.018060073256492615,
+          0.0068066176027059555,
+          0.030665762722492218,
+          0.00748084019869566,
+          0.002448791405186057,
+          0.0165904201567173,
+          0.003890036838129163,
+          0.010855740867555141,
+          0.000841831264551729,
+          0.0052574207074940205,
+          0.022423582151532173,
+          0.014454119838774204,
+          -0.016029831022024155,
+          -0.015120766125619411,
+          -0.00044790346873924136,
+          -0.004450626205652952,
+          0.005261208396404982,
+          0.013242034241557121,
+          0.0061172437854111195,
+          -0.015681356191635132,
+          0.01052999310195446,
+          0.014158673584461212,
+          0.013128400780260563,
+          -0.0039847311563789845,
+          -0.03412020578980446,
+          -0.02443867363035679,
+          -0.021438762545585632,
+          0.003104075090959668,
+          -0.0008281006012111902,
+          -0.0021211495622992516,
+          0.0017954016802832484,
+          0.017347972840070724,
+          -0.0008352026343345642,
+          0.00379155483096838,
+          0.01686313934624195,
+          -0.01415109820663929,
+          -0.022090258076786995,
+          -0.0035964848939329386,
+          -0.027529491111636162,
+          0.0053710537031292915,
+          0.001844642567448318,
+          -0.00965123064815998,
+          0.008530051447451115,
+          -0.0046059247106313705,
+          -0.007223271764814854,
+          -0.012196609750390053,
+          -0.02007516659796238,
+          0.005780132487416267,
+          0.016878290101885796,
+          0.005730891600251198,
+          -0.009272454306483269,
+          -0.003939277958124876,
+          0.011575416661798954,
+          0.017484333366155624,
+          0.017317671328783035,
+          -0.009295180439949036,
+          0.009628504514694214,
+          0.0004022135108243674,
+          0.004670316819101572,
+          -0.01834794320166111,
+          0.0045869858004152775,
+          -0.0027953721582889557,
+          -0.008658835664391518,
+          0.01105270441621542,
+          -0.00031201730598695576,
+          0.01628739945590496,
+          -0.013090522959828377,
+          0.005624833982437849,
+          0.010333028621971607,
+          -0.020590301603078842,
+          -0.0067308624275028706,
+          -0.01692374423146248,
+          0.026484068483114243,
+          -0.00504530593752861,
+          0.00045405857963487506,
+          -0.02978700026869774,
+          0.0033616432920098305,
+          0.000848933297675103,
+          -0.000845619011670351,
+          -0.012764775194227695,
+          0.010749682784080505,
+          0.03242328763008118,
+          0.0005179771687835455,
+          0.0216660276055336,
+          -0.0036154238041490316,
+          0.0007480839849449694,
+          -0.00785582885146141,
+          0.004977126140147448,
+          -0.004916521720588207,
+          -0.009901223704218864,
+          -0.010863316245377064,
+          0.021726632490754128,
+          -0.0033029329497367144,
+          0.0003356908564455807,
+          0.011332998983561993,
+          -0.012067826464772224,
+          0.0014620780711993575,
+          -0.0047687990590929985,
+          0.00971941091120243,
+          0.0061702728271484375,
+          0.009113367646932602,
+          -0.01176480483263731,
+          0.05451354384422302,
+          -0.0106284748762846,
+          0.005537715740501881,
+          -0.00439380994066596,
+          -0.022226616740226746,
+          -0.017317671328783035,
+          0.012711746618151665,
+          -0.017544938251376152,
+          0.03375658020377159,
+          0.024150803685188293,
+          0.0006467612111009657,
+          0.010393633507192135,
+          -0.006958128418773413,
+          0.023923536762595177,
+          -0.002602196065708995,
+          0.0010795136913657188,
+          -0.004855917301028967,
+          0.015802564099431038,
+          0.04511988162994385,
+          0.017393426969647408,
+          -0.001152428099885583,
+          0.00829521007835865,
+          0.020575150847434998,
+          0.00814369972795248,
+          -0.0020075165666639805,
+          0.014219277538359165,
+          -0.01024212222546339,
+          0.00900731049478054,
+          -0.004458202049136162,
+          0.027438584715127945,
+          0.00814369972795248,
+          -0.006310420110821724,
+          -0.0036836036015301943,
+          -0.0008209985098801553,
+          0.014173824340105057,
+          0.006950552575290203,
+          0.01963578537106514,
+          -0.00461350055411458,
+          -0.0012575386790558696,
+          0.013870803639292717,
+          0.014491996727883816,
+          -0.00552635220810771,
+          0.001724380999803543,
+          -0.01435563713312149,
+          -0.010204244405031204,
+          0.013575357384979725,
+          0.006086941808462143,
+          0.006310420110821724,
+          -0.014514723792672157,
+          -0.00980274099856615,
+          -0.004670316819101572,
+          0.0015435151290148497,
+          0.015052586793899536,
+          0.011060279794037342,
+          -0.004814252257347107,
+          0.006052851676940918,
+          0.013825350441038609,
+          -0.014014738611876965,
+          0.006147546228021383,
+          0.007208121009171009,
+          0.008855799213051796,
+          0.004787737503647804,
+          -0.010817863047122955,
+          -0.00428396463394165,
+          -0.0063937511295080185,
+          0.0012670081341639161,
+          0.016832837834954262,
+          0.013181429356336594,
+          0.0049581872299313545,
+          0.027529491111636162,
+          -0.00839369185268879,
+          -0.012302667833864689,
+          0.02339324913918972,
+          0.011257244274020195,
+          0.0031703610438853502,
+          0.014651083387434483,
+          -0.012635990977287292,
+          -0.012446602806448936,
+          0.006655106786638498,
+          0.004848341923207045,
+          0.006049063988029957,
+          0.009151245467364788,
+          0.00450365524739027,
+          0.007333117071539164,
+          -0.007098275702446699,
+          -0.007427811156958342,
+          0.0014810169814154506,
+          0.01642375811934471,
+          0.020393338054418564,
+          -0.024968961253762245,
+          -0.0018806264270097017,
+          0.0009052763343788683,
+          0.010878467001020908,
+          -0.003316190093755722,
+          0.012310243211686611,
+          5.65205882594455e-05,
+          -0.0074240234680473804,
+          0.010158791206777096,
+          -0.02119634486734867,
+          -0.012681444175541401,
+          0.0012717428617179394,
+          0.013037494383752346,
+          -0.019590331241488457,
+          0.018226735293865204,
+          -0.008893677033483982,
+          -0.013348091393709183,
+          0.013893529772758484,
+          0.020014561712741852,
+          0.004090788308531046,
+          -0.007738408166915178,
+          -0.021862991154193878,
+          -0.01127239502966404,
+          0.016060132533311844,
+          0.020272130146622658,
+          0.000947888707742095,
+          -0.0006472347304224968,
+          0.010173942893743515,
+          -0.013113250024616718,
+          -0.023605365306138992,
+          -0.005230906419456005,
+          0.010386057198047638,
+          0.01746918261051178,
+          0.018832778558135033,
+          -0.006685409229248762,
+          0.02443867363035679,
+          -0.0059127043932676315,
+          -0.015499543398618698,
+          2.86597842205083e-05,
+          -0.02802947722375393,
+          -0.017423728480935097,
+          0.00013778003631159663,
+          0.00246773031540215,
+          0.008598231710493565,
+          0.0008555619278922677,
+          0.01943882182240486,
+          0.000904329412151128,
+          0.017590390518307686,
+          -0.000659071491099894,
+          -0.002009410411119461,
+          -0.0013995799235999584,
+          -0.0015340456739068031,
+          0.02254479005932808,
+          -0.012166308239102364,
+          -0.01176480483263731,
+          0.022302372381091118,
+          0.006852070800960064,
+          0.014719262719154358,
+          -0.027635548263788223,
+          0.013757170177996159,
+          -0.015499543398618698,
+          0.017423728480935097,
+          -0.03408990427851677,
+          0.006420265417546034,
+          0.01489350013434887,
+          -0.010348179377615452,
+          -0.020499395206570625,
+          -0.029650641605257988,
+          -0.024150803685188293,
+          -0.0038710979279130697,
+          -0.010855740867555141,
+          -0.016847988590598106,
+          0.006121031939983368,
+          -0.005802859086543322,
+          0.018499454483389854,
+          -0.007079336792230606,
+          0.033877789974212646,
+          0.006969491485506296,
+          -0.004056698642671108,
+          0.003835114184767008,
+          0.022469034418463707,
+          -0.006102093029767275,
+          0.02109028771519661,
+          0.004408961161971092,
+          -0.001467759720981121,
+          0.013681414537131786,
+          -0.006552837323397398,
+          -0.002361672930419445,
+          0.005821797996759415,
+          -0.010484539903700352,
+          0.022014502435922623,
+          0.005924067925661802,
+          -0.007708106189966202,
+          -0.0006775368237867951,
+          -0.011454207822680473,
+          0.014726839028298855,
+          0.006530110724270344,
+          0.0022480397019535303,
+          -0.011295121163129807,
+          0.011249667964875698,
+          -0.004859705455601215,
+          0.0040642740204930305,
+          0.009484568610787392,
+          -0.01774190180003643,
+          -0.028862785547971725,
+          0.011416330002248287,
+          0.019120648503303528,
+          0.006382387597113848,
+          -0.0055755930952727795,
+          -0.0025681061670184135,
+          0.00839369185268879,
+          0.010651201009750366,
+          0.022150862962007523,
+          -0.012984465807676315,
+          -8.889415767043829e-05,
+          0.013204156421124935,
+          -0.007170243188738823,
+          -0.01056029461324215,
+          0.00487485621124506,
+          0.023438703268766403,
+          0.0068823727779090405,
+          -0.04327145218849182,
+          -0.018469152972102165,
+          0.01909034699201584,
+          0.007136153522878885,
+          -0.04154422879219055,
+          0.0026381798088550568,
+          -0.003689285134896636,
+          -0.023787178099155426,
+          0.016105586662888527,
+          -0.0011817833874374628,
+          -0.01828734017908573,
+          0.028756728395819664,
+          0.02830219641327858,
+          0.01834794320166111,
+          2.6987840101355687e-05,
+          -0.013332940638065338,
+          -0.0318172425031662,
+          -0.023590214550495148,
+          0.008280059322714806,
+          -0.02660527639091015,
+          -0.0072725131176412106,
+          0.011242092587053776,
+          -0.017454031854867935,
+          0.02972639724612236,
+          -0.008507325313985348,
+          0.026938600465655327,
+          0.011446632444858551,
+          0.001691238023340702,
+          -0.013446573168039322,
+          -0.018802475184202194,
+          -0.01350717805325985,
+          -0.008196728304028511,
+          0.014090493321418762,
+          0.017923714593052864,
+          0.011007251217961311,
+          0.011332998983561993,
+          0.02251448854804039,
+          -0.012158732861280441,
+          0.007969462312757969,
+          -2.888172457460314e-05,
+          -0.008961857296526432,
+          0.0026343921199440956,
+          -0.018393397331237793,
+          0.00868156272917986,
+          0.0062308767810463905,
+          0.005458172410726547,
+          -0.007208121009171009,
+          0.010742107406258583,
+          0.02712041325867176,
+          -0.018847929313778877,
+          0.011401179246604443,
+          -0.031362712383270264,
+          -0.014787442982196808,
+          -0.01056029461324215,
+          0.026862844824790955,
+          0.007757347077131271,
+          -0.015726808458566666,
+          -0.007514929864555597,
+          -0.01545409020036459,
+          0.012885984033346176,
+          0.0032442226074635983,
+          0.005764981731772423,
+          0.015817714855074883,
+          0.021893294528126717,
+          0.01746918261051178,
+          -0.00795431062579155,
+          0.006852070800960064,
+          0.004674104508012533,
+          0.028090080246329308,
+          -0.01777220331132412,
+          0.020802417770028114,
+          0.009954252280294895,
+          -0.003935490269213915,
+          0.005685438401997089,
+          -0.014257155358791351,
+          0.008530051447451115,
+          -0.007325541693717241,
+          0.030514251440763474,
+          0.0033067206386476755,
+          0.014075342565774918,
+          -0.02062060497701168,
+          0.006575563922524452,
+          -0.03172633796930313,
+          0.009022461250424385,
+          0.013704141601920128,
+          0.012976890429854393,
+          -0.007677804213017225,
+          -0.024181105196475983,
+          -0.0119314668700099,
+          0.0057990713976323605,
+          -0.01279507763683796,
+          -0.01046181283891201,
+          -0.0034828518982976675,
+          0.012446602806448936,
+          0.018847929313778877,
+          -0.01848430372774601,
+          -0.019923655316233635,
+          -0.009583051316440105,
+          0.02898399345576763,
+          -0.015787413343787193,
+          -0.001471547526307404,
+          0.02413565292954445,
+          0.021241798996925354,
+          0.015022284351289272,
+          0.002155239460989833,
+          0.020090317353606224,
+          -0.0007111532613635063,
+          -0.0237265732139349,
+          -0.02813553437590599,
+          0.007302815094590187,
+          -0.006912675220519304,
+          0.01943882182240486,
+          -0.0035548193845897913,
+          -0.005658924113959074,
+          -0.0034033088013529778,
+          -0.007503566797822714,
+          -0.0165904201567173,
+          -0.01787826046347618,
+          0.0021647089160978794,
+          -0.008454295806586742,
+          0.005863463506102562,
+          -0.0002795845502987504,
+          -0.013105674646794796,
+          0.013401119969785213,
+          -0.014075342565774918,
+          0.005772557109594345,
+          -0.002653330797329545,
+          0.001962063368409872,
+          0.0035926972050219774,
+          0.009113367646932602,
+          -0.007336904760450125,
+          -0.06381630152463913,
+          -0.013105674646794796,
+          -0.003704436356201768,
+          0.006071790587157011,
+          -0.00019578021601773798,
+          -0.016635872423648834,
+          -0.013196581043303013,
+          0.01706010289490223,
+          0.008863375522196293,
+          -0.013870803639292717,
+          0.016954045742750168,
+          0.009961827658116817,
+          0.003622999181970954,
+          -0.024044746533036232,
+          -0.0008148433989845216,
+          0.008757317438721657,
+          0.032756611704826355,
+          -0.008643684908747673,
+          -0.0003839849086944014,
+          0.0036684523802250624,
+          -0.0033332351595163345,
+          -0.010052734054625034,
+          -0.014582903124392033,
+          -0.011848135851323605,
+          0.011908739805221558,
+          0.01790856383740902,
+          -0.014264730736613274,
+          -0.02007516659796238,
+          -0.0007106798002496362,
+          0.012931437231600285,
+          -0.026590125635266304,
+          0.0007244104635901749,
+          -0.007787649054080248,
+          -0.01052999310195446,
+          0.018302490934729576,
+          -0.019014591351151466,
+          0.005249845329672098,
+          0.00048294031876139343,
+          0.008507325313985348,
+          -0.023544760420918465,
+          -0.024893205612897873,
+          0.0035131541080772877,
+          -0.023847781121730804,
+          0.03612014651298523,
+          -0.021590271964669228,
+          -0.012082977220416069,
+          0.034483831375837326,
+          -0.0037423139438033104,
+          0.01702980138361454,
+          0.0023995505180209875,
+          0.02739313244819641,
+          -0.011439057067036629,
+          -0.00617784820497036,
+          0.012499631382524967,
+          0.009575475007295609,
+          0.0032593735959380865,
+          0.03775646165013313,
+          -0.01257538702338934,
+          -0.013901105150580406,
+          -0.005556654185056686,
+          -0.012954163365066051,
+          0.0006775368237867951,
+          0.014431392773985863,
+          0.028393102809786797,
+          0.004257450345903635,
+          -0.00628769351169467,
+          -0.006143758539110422,
+          -0.0014649189542979002,
+          -0.004242299124598503,
+          -0.00947699323296547,
+          -0.00715130427852273,
+          0.017287369817495346,
+          -0.025241680443286896,
+          0.016711628064513206,
+          -0.006420265417546034,
+          0.021135739982128143,
+          -0.011757229454815388,
+          0.007007369305938482,
+          -0.008961857296526432,
+          0.003492321353405714,
+          0.0023635667748749256,
+          -0.005109698045998812,
+          0.009287605062127113,
+          -0.014772292226552963,
+          0.008454295806586742,
+          0.013742019422352314,
+          0.0011183382011950016,
+          -0.010590597055852413,
+          -0.019272159785032272,
+          0.0022972808219492435,
+          -0.000710206339135766,
+          0.01181783340871334,
+          -0.0047498601488769054,
+          -0.000981031684204936,
+          0.0002921315317507833,
+          -0.014476845972239971,
+          -0.009575475007295609,
+          0.0006050958181731403,
+          0.005302873905748129,
+          -0.008961857296526432,
+          0.010378481820225716,
+          -0.005693014245480299,
+          -0.00034160923678427935,
+          0.02627195231616497,
+          0.006526322569698095,
+          -0.007632351014763117,
+          -0.014098069630563259,
+          0.01563590206205845,
+          0.015370759181678295,
+          -0.007454325910657644,
+          0.01933276280760765,
+          -0.016196493059396744,
+          -0.0008366230758838356,
+          0.013916256837546825,
+          -0.009219424799084663,
+          0.0017698341980576515,
+          -0.008196728304028511,
+          0.03554440662264824,
+          -0.0008787619881331921,
+          -0.0035415622405707836,
+          -0.0006401326390914619,
+          0.012976890429854393,
+          -0.0026779514737427235,
+          0.001182730309665203,
+          0.01132542360574007,
+          0.005321812815964222,
+          0.0185146052390337,
+          0.00328020635060966,
+          -0.03378688171505928,
+          -0.007780073676258326,
+          -0.0065490491688251495,
+          -0.008090670220553875,
+          0.009022461250424385,
+          0.025196226313710213,
+          0.01990850456058979,
+          0.006408901885151863,
+          0.001881573349237442,
+          0.018423698842525482,
+          0.015757111832499504,
+          -0.01489350013434887,
+          0.016363153234124184,
+          0.02802947722375393,
+          -0.010030006989836693,
+          0.022741753607988358,
+          -0.005253633018583059,
+          -0.008893677033483982,
+          0.008946705609560013,
+          -0.0015378334792330861,
+          -0.017393426969647408,
+          -0.015681356191635132,
+          0.005022579338401556,
+          -0.008999735116958618,
+          -0.0022897052112966776,
+          -0.018605511635541916,
+          0.004670316819101572,
+          0.017105557024478912,
+          -0.004965762607753277,
+          -3.5243989259470254e-05,
+          0.019484274089336395,
+          -0.006965703796595335,
+          -0.016908591613173485,
+          0.006325571332126856,
+          -0.015007133595645428,
+          0.007946735247969627,
+          -0.013666263781487942,
+          0.010492115281522274,
+          -0.006098304875195026,
+          0.018817627802491188,
+          0.005446809343993664,
+          -0.008090670220553875,
+          -0.0008958069374784827,
+          -0.018984289839863777,
+          0.02065090648829937,
+          -0.033089932054281235,
+          0.00839369185268879,
+          -0.026226500049233437,
+          -0.0009781909175217152,
+          0.0008849170990288258,
+          0.014605630189180374,
+          -0.001218714052811265,
+          0.019696388393640518,
+          -0.002751812804490328,
+          -0.020090317353606224,
+          -0.007086912170052528,
+          -0.0010634156642481685,
+          -0.01618134044110775,
+          -0.030923331156373024,
+          -0.004272601567208767,
+          0.0036267871037125587,
+          -0.004711982328444719,
+          0.0021135739516466856,
+          0.017984317615628242,
+          -0.005613470915704966,
+          0.009173971600830555,
+          -0.02690829709172249,
+          -0.007416448090225458,
+          -0.00355103169567883,
+          0.0010776197304949164,
+          -0.011242092587053776,
+          0.0010596278589218855,
+          0.011226941831409931,
+          -0.019044892862439156,
+          0.013461724855005741,
+          -0.021378157660365105,
+          -0.004371083341538906,
+          -0.008507325313985348,
+          -0.00135412672534585,
+          -0.013446573168039322,
+          -0.028120383620262146,
+          0.025029566138982773,
+          -0.00021495578403118998,
+          -0.003528305096551776,
+          -0.011999646201729774,
+          0.023181134834885597,
+          0.002914686920121312,
+          0.027817361056804657,
+          -0.03157482668757439,
+          0.0013427634257823229,
+          0.011552689597010612,
+          0.0025984083767980337,
+          -0.017287369817495346,
+          0.004746072459965944,
+          -0.005234694108366966,
+          0.02210540883243084,
+          -0.0011694730492308736,
+          -0.0005340751959010959,
+          -0.008234606124460697,
+          0.030999086797237396,
+          0.009136094711720943,
+          0.0012944694608449936,
+          -0.013226882554590702,
+          -0.008060368709266186,
+          -0.002759388415142894,
+          -0.0011069749016314745,
+          0.012855681590735912,
+          0.01290113478899002,
+          -0.012105703353881836,
+          0.0013607552973553538,
+          -0.006878585088998079,
+          0.0011694730492308736,
+          -0.002935519441962242,
+          -0.0004966709529981017,
+          -0.01149208564311266,
+          0.003257479751482606,
+          0.016272246837615967,
+          -0.00780280027538538,
+          -0.006480869371443987,
+          0.0007750718505121768,
+          0.009181547909975052,
+          0.012461753562092781,
+          -0.012742049060761929,
+          -0.018226735293865204,
+          -0.011908739805221558,
+          0.016029831022024155,
+          0.010522416792809963,
+          -0.008204303681850433,
+          -0.013234458863735199,
+          -0.02583257108926773,
+          0.01169662456959486,
+          -0.03293842449784279,
+          -0.012242063879966736,
+          0.005821797996759415,
+          0.028559764847159386,
+          0.019999410957098007,
+          0.012226912193000317,
+          0.001332347048446536,
+          0.002950670663267374,
+          0.00023342114582192153,
+          -0.01916610263288021,
+          0.014408666640520096,
+          -0.011143610812723637,
+          -0.018166130408644676,
+          0.0075906855054199696,
+          0.011939042247831821,
+          0.03148392215371132,
+          0.006942977197468281,
+          -0.0010160685051232576,
+          0.01686313934624195,
+          -0.001218714052811265,
+          -0.013582932762801647,
+          0.01420412678271532,
+          -0.024256860837340355,
+          0.016438908874988556,
+          0.02860521711409092,
+          -0.004780162125825882,
+          0.0008531945641152561,
+          -0.020938776433467865,
+          0.00482561532407999,
+          -0.011529963463544846,
+          -0.002177966060116887,
+          -0.003287781961262226,
+          -0.03460504114627838,
+          0.007946735247969627,
+          -0.006825556512922049,
+          0.01046181283891201,
+          -0.00814369972795248,
+          0.006655106786638498,
+          -0.012431452050805092,
+          0.004253662656992674,
+          0.01513591781258583,
+          0.016105586662888527,
+          0.007060397882014513,
+          -0.025211378931999207,
+          -0.009196698665618896,
+          0.008105821907520294,
+          -0.01349202636629343,
+          -0.02295386977493763,
+          -0.006852070800960064,
+          0.009840618818998337,
+          -0.01061332318931818,
+          -0.00488243205472827,
+          -0.028180986642837524,
+          -0.013681414537131786,
+          -0.0014327229000627995,
+          -0.017817657440900803,
+          -0.005556654185056686,
+          -0.003100287402048707,
+          0.01571165770292282,
+          -0.0049733384512364864,
+          -0.006185423582792282,
+          0.004742284305393696,
+          -0.023878084495663643,
+          0.0037423139438033104,
+          -0.008280059322714806,
+          0.0008266801596619189,
+          -0.0032517979852855206,
+          -0.0014762822538614273,
+          -0.007969462312757969,
+          -0.014332910999655724,
+          0.006893736310303211,
+          -0.016469212248921394,
+          0.002954458352178335,
+          -0.015173795633018017,
+          -0.016499513760209084,
+          -0.00682934420183301,
+          0.00775355938822031,
+          0.021575121209025383,
+          0.006541473791003227,
+          0.021484214812517166,
+          0.005469535943120718,
+          0.028680972754955292,
+          0.0003953482082579285,
+          0.003041577059775591,
+          -8.958956641436089e-06,
+          0.003232859307900071,
+          0.019347915425896645,
+          0.006571775767952204,
+          0.018984289839863777,
+          -0.008742166683077812,
+          0.010514841414988041,
+          -0.010355755686759949,
+          0.010014856234192848,
+          0.006333146709948778,
+          -0.006473293993622065,
+          -0.003799130441620946,
+          -0.010348179377615452,
+          -0.0031419529113918543,
+          -0.016544966027140617,
+          0.0027366618160158396,
+          0.00693161366507411,
+          0.0071588801220059395,
+          -0.011582992039620876,
+          -0.005230906419456005,
+          -0.014704111963510513,
+          -0.0064354161731898785,
+          -0.01831764169037342,
+          0.014385939575731754,
+          0.029226411134004593,
+          -0.016332851722836494,
+          -0.01977214403450489,
+          -0.007772498298436403,
+          0.0007296186522580683,
+          -0.006636167876422405,
+          -0.01240872498601675,
+          -0.009151245467364788,
+          -0.035059571266174316,
+          -0.017666146159172058,
+          0.007586897816509008,
+          0.02484775334596634,
+          0.01387837901711464,
+          -0.013628385961055756,
+          -0.0006983695784583688,
+          -0.016984347254037857,
+          -0.010787560604512691,
+          -0.01535560842603445,
+          -0.02054484933614731,
+          -0.02498411200940609,
+          0.0077952248975634575,
+          -0.004033972043544054,
+          0.014931377954781055,
+          0.008113397285342216,
+          0.0032631612848490477,
+          -0.004371083341538906,
+          -0.0024109138175845146,
+          -0.013196581043303013,
+          0.0012925754999741912,
+          -0.004787737503647804,
+          -0.0017291157273575664,
+          0.03554440662264824,
+          -0.008098246529698372,
+          -0.005871038883924484,
+          0.011060279794037342,
+          0.011128460057079792,
+          -0.013764745555818081,
+          0.0006642796797677875,
+          -0.011249667964875698,
+          0.021847840398550034,
+          -0.021620575338602066,
+          0.019257009029388428,
+          0.0014507147716358304,
+          -0.010840589180588722,
+          0.028999146074056625,
+          0.01892368495464325,
+          0.00743159931153059,
+          0.006579351611435413,
+          0.01672677882015705,
+          0.0044316877610981464,
+          0.016196493059396744,
+          -0.0026552248746156693,
+          -0.004632438998669386,
+          0.009295180439949036,
+          0.015681356191635132,
+          -0.000702630786690861,
+          -0.006942977197468281,
+          0.016438908874988556,
+          0.004056698642671108,
+          0.0075414446182549,
+          -0.01356020662933588,
+          -0.008961857296526432,
+          0.001507531269453466,
+          0.006257391534745693,
+          -0.005939218681305647,
+          -0.0026003022212535143,
+          0.005530139897018671,
+          0.003943065647035837,
+          -0.013302638195455074,
+          0.01689344085752964,
+          -0.016166189685463905,
+          0.008742166683077812,
+          -0.0037972365971654654,
+          0.0121360057964921,
+          -0.006799042224884033,
+          0.0013153020991012454,
+          -0.01818128302693367,
+          -0.003869204083457589,
+          0.0038597346283495426,
+          0.012173883616924286,
+          -0.01262841559946537,
+          0.025484098121523857,
+          0.008166425861418247,
+          0.006602078210562468,
+          0.0059467945247888565,
+          -0.012855681590735912,
+          0.027211319655179977,
+          0.005969521123915911,
+          0.009886072017252445,
+          -0.008893677033483982,
+          0.011136035434901714,
+          0.035362593829631805,
+          0.03060515783727169,
+          0.014135947450995445,
+          0.004469565115869045,
+          -0.018999440595507622,
+          0.015120766125619411,
+          0.0018427487229928374,
+          0.0023806116078048944,
+          0.00863610953092575,
+          -0.011348150670528412,
+          0.01305264513939619,
+          -0.009484568610787392,
+          0.015241974964737892,
+          -0.021272100508213043,
+          -0.011749654076993465,
+          -0.014408666640520096,
+          -0.021075136959552765,
+          -0.01430260855704546,
+          0.034180812537670135,
+          -0.002346521709114313,
+          -0.010393633507192135,
+          0.0045112306252121925,
+          0.008477022871375084,
+          -0.006079366430640221,
+          0.0053142374381423,
+          -0.008204303681850433,
+          -0.013749594800174236,
+          -0.01889338344335556,
+          0.006628592498600483,
+          -0.01550711877644062,
+          0.007651289459317923,
+          -0.003304826794192195,
+          0.0037688282318413258,
+          0.007014944683760405,
+          0.018105527386069298,
+          0.00530666159465909,
+          -0.00135412672534585,
+          0.0014033677289262414,
+          0.0016865032957866788,
+          -0.00931033119559288,
+          0.0108330138027668,
+          0.015408637002110481,
+          -0.0011921997647732496,
+          -0.011279970407485962,
+          -0.009136094711720943,
+          -0.042877525091171265,
+          0.0006264019757509232,
+          -0.013128400780260563,
+          0.0064354161731898785,
+          -0.011196639388799667,
+          0.0022234192583709955,
+          0.015863168984651566,
+          -0.0054051438346505165,
+          -0.0010918239131569862,
+          0.02193874679505825,
+          0.013014768250286579,
+          0.021075136959552765,
+          0.02065090648829937,
+          0.014188976027071476,
+          -0.015302578918635845,
+          -0.03469594568014145,
+          0.002221525413915515,
+          -0.010030006989836693,
+          0.013772320933640003,
+          0.004238511435687542,
+          0.016060132533311844,
+          -0.023893235251307487,
+          -0.011878437362611294,
+          -0.006662682630121708,
+          0.014431392773985863,
+          0.01491622719913721,
+          -0.0032271775417029858,
+          0.02690829709172249,
+          0.009280029684305191,
+          0.0020510759204626083,
+          -0.02281750924885273,
+          0.003312402404844761,
+          0.005321812815964222,
+          -0.008370965719223022,
+          -0.016681326553225517,
+          -0.0004552422615233809,
+          -0.0012168202083557844,
+          -0.0007817004225216806,
+          -0.021650876849889755,
+          0.018832778558135033,
+          -0.004018820822238922,
+          0.0037574649322777987,
+          0.013840501196682453,
+          0.01516621932387352,
+          0.017984317615628242,
+          0.008370965719223022,
+          0.001497115008533001,
+          -0.006605865899473429,
+          -0.0007973249885253608,
+          -0.0005828426801599562,
+          -0.007389933802187443,
+          -0.005598319694399834,
+          0.006382387597113848,
+          -0.01134057529270649,
+          0.008810346014797688,
+          0.000896753859706223,
+          0.006113456096500158,
+          0.010605747811496258,
+          -0.004836978856474161,
+          -0.011628445237874985,
+          0.014628356322646141,
+          -0.0010548931313678622,
+          -0.0055755930952727795,
+          -0.011060279794037342,
+          0.003624893259257078,
+          0.013575357384979725,
+          -0.003096499713137746,
+          -0.013499601744115353,
+          0.00910579226911068,
+          0.002009410411119461,
+          -0.027438584715127945,
+          0.015499543398618698,
+          0.010295150801539421,
+          -0.013370818458497524,
+          0.004980913829058409,
+          -0.003371112747117877,
+          -0.03027183562517166,
+          0.004791525658220053,
+          0.002660906407982111,
+          -0.010643625631928444,
+          0.028938541188836098,
+          -0.003821857040748,
+          0.029256712645292282,
+          -0.004980913829058409,
+          -0.006905099377036095,
+          -0.011355726048350334,
+          0.009416389279067516,
+          -0.018681267276406288,
+          -0.0037953427527099848,
+          -0.008727015927433968,
+          -0.00915882084518671,
+          -0.0009838725673034787,
+          -0.005185453221201897,
+          0.006984642706811428,
+          0.005121061112731695,
+          -0.012704171240329742,
+          0.006757376715540886,
+          -0.010257272981107235,
+          -0.00020856392802670598,
+          0.0006718551740050316,
+          -0.008408842608332634,
+          0.013885954394936562,
+          0.005996035411953926,
+          -0.008340663276612759,
+          -0.0005032995832152665,
+          0.027817361056804657,
+          0.05051366239786148,
+          -0.018863080069422722,
+          0.0009270560112781823,
+          0.01452229917049408,
+          -0.0028824908658862114,
+          -0.0021590273827314377,
+          -0.011976920068264008,
+          0.005821797996759415,
+          -0.019423669204115868,
+          0.013317788951098919,
+          0.006473293993622065,
+          -0.008833073079586029,
+          0.003202557098120451,
+          -0.016711628064513206,
+          0.002621134975925088,
+          -0.0018058179412037134,
+          -0.018681267276406288,
+          -0.015529844909906387,
+          -0.012060250155627728,
+          -0.024181105196475983,
+          0.0004235197266098112,
+          -0.0008546149474568665,
+          -0.013870803639292717,
+          0.006579351611435413,
+          0.008431569673120975,
+          0.032483890652656555,
+          0.0011912527261301875,
+          0.010128489695489407,
+          -2.1513333194889128e-05,
+          -0.0010416359873488545,
+          -0.005988460034132004,
+          -0.01523439958691597,
+          0.00015387803432531655,
+          -0.002342734020203352,
+          0.022590242326259613,
+          0.005549078807234764,
+          0.017560089007019997,
+          0.0017887729918584228,
+          -0.016514664515852928,
+          -0.00596573343500495,
+          -0.007408872712403536,
+          -0.018060073256492615,
+          0.0062649669125676155,
+          -0.005473323632031679,
+          0.009605777449905872,
+          -0.015863168984651566,
+          -0.02731737680733204,
+          0.00792400911450386,
+          0.012234487570822239,
+          0.028514310717582703,
+          -0.0035623949952423573,
+          0.001716805505566299,
+          -0.005533927585929632,
+          0.016408607363700867,
+          0.032180868089199066,
+          -0.0008550884085707366,
+          0.006950552575290203,
+          -0.019999410957098007,
+          0.0017404790269210935,
+          0.013704141601920128,
+          -0.008401267230510712,
+          -0.01821158453822136,
+          0.003939277958124876,
+          -0.020635755732655525,
+          0.004583198111504316,
+          -0.020393338054418564,
+          0.030408194288611412,
+          -0.0023522034753113985,
+          0.008105821907520294,
+          0.004310478921979666,
+          0.01760554127395153,
+          -0.0031646795105189085,
+          0.010545143857598305,
+          0.013916256837546825,
+          0.014908651821315289,
+          0.0009980766335502267,
+          -0.011189064010977745,
+          -0.03790797293186188,
+          0.001859793672338128,
+          0.00817400123924017,
+          0.018635815009474754,
+          -0.01980244740843773,
+          0.011681473813951015,
+          -0.01296931505203247,
+          -0.020363036543130875,
+          -0.01003758329898119,
+          0.011575416661798954,
+          -0.006139970384538174,
+          0.03378688171505928,
+          -0.005874827038496733,
+          -0.008204303681850433,
+          0.005136212334036827,
+          0.010454237461090088,
+          0.0066778333857655525,
+          0.014704111963510513,
+          0.0326354019343853,
+          0.010196669027209282,
+          0.014173824340105057,
+          -0.004859705455601215,
+          0.02399929240345955,
+          -0.003931702114641666,
+          0.003954428713768721,
+          0.00509454682469368,
+          -0.022832660004496574,
+          -0.0040983641520142555,
+          -0.003931702114641666,
+          -0.016711628064513206,
+          -0.007458113599568605,
+          0.0032366469968110323,
+          0.008802770636975765,
+          -0.014045040123164654,
+          -0.00649980828166008,
+          -0.01350717805325985,
+          0.01218903437256813,
+          0.002730980049818754,
+          0.018302490934729576,
+          -0.029802151024341583,
+          0.009825468063354492,
+          -0.017484333366155624,
+          -0.017044952139258385,
+          0.006511171814054251,
+          -0.0038332203403115273,
+          0.006855858489871025,
+          -0.018272189423441887,
+          -0.01318900566548109,
+          0.012817803770303726,
+          0.0007897494360804558,
+          0.013060221448540688,
+          -0.009825468063354492,
+          0.0009706153068691492,
+          0.006064215209335089,
+          -0.009833043441176414,
+          -0.010961798019707203,
+          -0.007977037690579891,
+          -0.009681533090770245,
+          -0.008848223835229874,
+          0.005621046293526888,
+          0.01656011864542961,
+          -0.013742019422352314,
+          -0.003685497445985675,
+          0.016135888174176216,
+          0.006416477728635073,
+          -0.008166425861418247,
+          0.004075637552887201,
+          -0.003793448908254504,
+          -9.948806837201118e-05,
+          0.0013740125577896833,
+          0.012439027428627014,
+          -0.0017632056260481477,
+          0.024665940552949905,
+          -0.02654467150568962,
+          0.007598260883241892,
+          -0.0054051438346505165,
+          0.005446809343993664,
+          0.018226735293865204,
+          -0.02366596832871437,
+          -0.013082947582006454,
+          0.006276329979300499,
+          -0.01966608688235283,
+          -0.008060368709266186,
+          0.0054051438346505165,
+          -0.00016405766655225307,
+          -0.019923655316233635,
+          -0.012923861853778362,
+          -0.034786853939294815,
+          -0.02309022843837738,
+          0.024665940552949905,
+          0.008401267230510712,
+          -0.0073672072030603886,
+          0.018060073256492615,
+          -0.018438849598169327,
+          0.0172722190618515,
+          -0.002225313102826476,
+          -0.006613441277295351,
+          0.003935490269213915,
+          0.007674016058444977,
+          0.025362888351082802,
+          0.002653330797329545,
+          0.008264907635748386,
+          -0.0042082094587385654,
+          -0.014931377954781055,
+          -0.006321783177554607,
+          0.01980244740843773,
+          0.02471139281988144,
+          0.018605511635541916,
+          -0.018575210124254227,
+          0.010742107406258583,
+          -0.011749654076993465,
+          0.01818128302693367,
+          -0.006537686102092266,
+          0.00039037675014697015,
+          0.01859036087989807,
+          0.007969462312757969,
+          0.025241680443286896,
+          0.019923655316233635,
+          0.006852070800960064,
+          0.014128371141850948,
+          0.011908739805221558,
+          0.001204509986564517,
+          -0.012832955457270145,
+          -0.01154511421918869,
+          -0.02840825356543064,
+          0.042847223579883575,
+          0.0025434857234358788,
+          0.008317936211824417,
+          -0.027635548263788223,
+          -0.00021389048197306693,
+          -0.0034942151978611946,
+          -0.018832778558135033,
+          0.005939218681305647,
+          0.03260510042309761,
+          0.03284751623868942,
+          0.021438762545585632,
+          -2.888172457460314e-05,
+          0.0012234487803652883,
+          0.0329081192612648,
+          -0.013673839159309864,
+          -0.00937851145863533,
+          -0.01034060399979353,
+          -0.018393397331237793,
+          -0.006852070800960064,
+          -0.0032423287630081177,
+          -0.01031030248850584,
+          -0.0061172437854111195,
+          -0.009272454306483269,
+          0.0238629337400198,
+          0.0001374249259242788,
+          -0.0022594030015170574,
+          0.006605865899473429,
+          -0.020878173410892487,
+          0.011537538841366768,
+          -0.02813553437590599,
+          -0.013431422412395477,
+          -0.008651260286569595,
+          -0.006950552575290203,
+          -0.018802475184202194,
+          0.008643684908747673,
+          -0.006942977197468281,
+          0.021438762545585632,
+          0.002914686920121312,
+          -0.023590214550495148,
+          0.007530081085860729,
+          0.015317730605602264,
+          -0.0003825644962489605,
+          0.014037464745342731,
+          -0.003890036838129163,
+          0.009060339070856571,
+          0.0045112306252121925,
+          0.011681473813951015,
+          0.016954045742750168,
+          0.010651201009750366,
+          0.009030036628246307,
+          0.003437398700043559,
+          -0.001167579204775393,
+          0.01878732442855835,
+          -0.0007163614500313997,
+          -0.03405960276722908,
+          2.9236827685963362e-05,
+          -0.014628356322646141,
+          0.029499130323529243,
+          -0.009795165620744228,
+          0.009514871053397655,
+          0.008105821907520294,
+          0.007098275702446699,
+          -0.0067119235172867775,
+          0.007916433736681938,
+          -0.008742166683077812,
+          0.0052763596177101135,
+          0.015832867473363876,
+          -0.0042877523228526115,
+          -0.0014980619307607412,
+          0.007840678095817566,
+          0.00407184986397624,
+          -0.005454384721815586,
+          -0.019711541011929512,
+          -0.012287517078220844,
+          0.00726114958524704,
+          -0.02065090648829937,
+          -0.004367295652627945,
+          0.01161329448223114,
+          -0.005454384721815586,
+          -0.009620928205549717,
+          0.01393140759319067,
+          0.02136300690472126,
+          -0.002382505452260375,
+          0.022893264889717102,
+          -0.01349202636629343,
+          -0.009833043441176414,
+          -0.007984613068401814,
+          -0.004374871030449867,
+          -0.041120000183582306,
+          0.025984082370996475,
+          0.01068150345236063,
+          -0.006871009711176157,
+          0.01895398646593094,
+          0.0022499337792396545,
+          -0.002954458352178335,
+          -0.0011041341349482536,
+          0.00010499217023607343,
+          -0.03281721472740173,
+          0.03660498186945915,
+          -0.007007369305938482,
+          0.0005449649761430919,
+          0.010272424668073654,
+          0.042029064148664474,
+          -0.004590773954987526,
+          -0.005859675817191601,
+          0.01225721463561058,
+          0.005336964037269354,
+          0.031938452273607254,
+          -0.010416359640657902,
+          -0.014340486377477646,
+          -0.01815097965300083,
+          -0.004833190701901913,
+          -0.01939336769282818,
+          -0.0038313264958560467,
+          0.014037464745342731,
+          -0.019650936126708984,
+          -0.011242092587053776,
+          -0.001287840772420168,
+          -0.012537509202957153,
+          -0.0010747789638116956,
+          0.008598231710493565,
+          0.017832808196544647,
+          -0.011348150670528412,
+          -0.004450626205652952,
+          -2.2904154320713133e-05,
+          -0.005003640428185463,
+          0.0185146052390337,
+          0.0020056227222085,
+          -0.01702980138361454,
+          0.01321930717676878,
+          -0.00526499655097723,
+          0.006749800872057676,
+          0.0074429623782634735,
+          0.007939159870147705,
+          0.05145302787423134,
+          0.010317877866327763,
+          -0.001339922659099102,
+          -0.01607528328895569,
+          0.023196285590529442,
+          -0.01749948412179947,
+          -0.014181400649249554,
+          -0.02119634486734867,
+          -0.0029923359397798777,
+          -0.007060397882014513,
+          -0.028014326468110085,
+          -9.137987944995984e-05,
+          0.0016221113037317991,
+          -0.0024980325251817703,
+          -0.010045158676803112,
+          0.0005208179936744273,
+          -0.011529963463544846,
+          -0.006954340264201164,
+          -0.008545203134417534,
+          -0.016044981777668,
+          0.02240842953324318,
+          -0.021817538887262344,
+          -0.008855799213051796,
+          0.011514811776578426,
+          0.03118089959025383,
+          -0.005011215806007385,
+          -0.014173824340105057,
+          -0.019817598164081573,
+          0.0025643182452768087,
+          -0.01763584464788437,
+          0.012181458994746208,
+          -0.01225721463561058,
+          0.015045011416077614,
+          -0.03142331540584564,
+          -0.006079366430640221,
+          -0.008992158807814121,
+          0.012840530835092068,
+          0.019817598164081573,
+          -0.0025093958247452974,
+          0.00606800289824605,
+          -0.010295150801539421,
+          0.030211230739951134,
+          -0.00012286569108255208,
+          -0.018605511635541916,
+          0.00873459130525589,
+          0.007977037690579891,
+          0.012537509202957153,
+          -0.004435475450009108,
+          0.00257757562212646,
+          0.011567841283977032,
+          0.012742049060761929,
+          -0.01634800247848034,
+          -0.003174148965626955,
+          -0.017969166859984398,
+          0.005996035411953926,
+          0.006999793462455273,
+          0.021711481735110283,
+          -0.0013550736475735903,
+          0.018272189423441887,
+          -0.01563590206205845,
+          0.00019412307301536202,
+          -0.011749654076993465,
+          -0.0082270298153162,
+          -0.0028237802907824516,
+          -0.003958216868340969,
+          0.006685409229248762,
+          -0.028211290016770363,
+          -0.01274962443858385,
+          0.005287723150104284,
+          -0.004992276895791292,
+          -0.03405960276722908,
+          0.0053710537031292915,
+          -0.02725677192211151,
+          0.013128400780260563,
+          -0.002397656673565507,
+          -0.017847958952188492,
+          -0.00030846628942526877,
+          0.026332557201385498,
+          -0.003677921835333109,
+          -0.012537509202957153,
+          0.015544996596872807,
+          -0.006757376715540886,
+          -0.02119634486734867,
+          -0.005420294590294361,
+          0.014658658765256405,
+          -0.0010719380807131529,
+          -0.012393574230372906,
+          -0.029635490849614143,
+          0.012992041185498238,
+          0.0026874206960201263,
+          -0.007386146113276482,
+          0.010893618687987328,
+          0.025953780859708786,
+          0.008151275105774403,
+          -0.005924067925661802,
+          0.0015757111832499504,
+          0.0049581872299313545,
+          0.00300938100554049,
+          0.020287280902266502,
+          0.02728707529604435,
+          -0.008014915511012077,
+          -0.0011145505122840405,
+          -0.014954105019569397,
+          0.015802564099431038,
+          0.0013020449550822377,
+          0.010507266037166119,
+          0.021893294528126717,
+          0.007045246660709381,
+          -0.005738467443734407,
+          0.003905187826603651,
+          -0.023135682567954063,
+          0.01420412678271532,
+          -0.012272365391254425,
+          0.000622140767518431,
+          0.008749742060899734,
+          -0.007283876184374094,
+          0.01437078882008791,
+          -0.0015245762187987566,
+          0.0339989997446537,
+          0.025211378931999207,
+          0.0043218424543738365,
+          0.0007362472242675722,
+          -0.028559764847159386,
+          0.003547244006767869,
+          0.008651260286569595,
+          0.014499573037028313,
+          -0.0076664406806230545,
+          0.006261179223656654,
+          -0.004355932120233774,
+          0.00959820207208395,
+          -0.0065490491688251495,
+          -0.015893470495939255,
+          0.006113456096500158,
+          0.0012689019786193967,
+          0.006795254070311785,
+          0.0023408401757478714,
+          -0.008863375522196293,
+          -0.024362917989492416,
+          -0.025984082370996475,
+          0.021984200924634933,
+          -0.03918066248297691,
+          0.0073293293826282024,
+          0.011234517209231853,
+          0.03324144333600998,
+          -0.008158850483596325,
+          -0.004980913829058409,
+          0.006276329979300499,
+          -0.016711628064513206,
+          -0.004147605039179325,
+          0.008204303681850433,
+          0.012385998852550983,
+          -0.002420383272692561,
+          0.00974971242249012,
+          0.01318900566548109,
+          -0.012378423474729061,
+          0.03660498186945915,
+          0.016514664515852928,
+          0.01467380952090025,
+          -0.004007457755506039,
+          0.01312082540243864,
+          0.0066172294318675995,
+          0.019044892862439156,
+          0.025165924802422523,
+          -0.017454031854867935,
+          -0.002365460619330406,
+          -0.02292356640100479,
+          0.024165954440832138,
+          0.00726114958524704,
+          0.004556683823466301,
+          -0.008310360834002495,
+          0.014870774000883102,
+          0.0017954016802832484,
+          0.0028199926018714905,
+          -0.006696772295981646,
+          -0.010173942893743515,
+          0.019241856411099434,
+          0.0034317169338464737,
+          0.014870774000883102,
+          4.8501173296244815e-05,
+          -0.005503625608980656,
+          0.006052851676940918,
+          0.014832896180450916,
+          -0.023590214550495148,
+          -0.00203592493198812,
+          -0.019484274089336395,
+          -0.03202936053276062,
+          0.009022461250424385,
+          0.0016135887708514929,
+          -0.007052822504192591,
+          0.03230207785964012,
+          -0.010083036497235298,
+          -0.019787294790148735,
+          -0.008067944087088108,
+          -0.013113250024616718,
+          -0.010469388216733932,
+          0.0008271536789834499,
+          0.003908975515514612,
+          -0.00012428610352799296,
+          -0.009317907504737377,
+          -0.005749830510467291,
+          0.01990850456058979,
+          -0.0030358952935785055,
+          -0.00753386877477169,
+          0.00868156272917986
+        ],
+        "embedding": [
+          0.0040707582550429606,
+          -0.010230625151989194,
+          -0.021520242101903186,
+          -0.010115756270640508,
+          0.026738734586159035,
+          0.01796370233223267,
+          -0.012245199029994153,
+          0.06506924494147726,
+          -0.0014087587222033442,
+          0.009300068719271429,
+          0.003787396973047584,
+          0.002818772639754174,
+          0.022129077062176306,
+          -0.021526709808445683,
+          -0.0034388135367225824,
+          0.02763761747879405,
+          0.013210013805395276,
+          0.025255872672296602,
+          0.00029011480909986703,
+          -0.05438974762290585,
+          0.016671640187490865,
+          -0.016426252259893328,
+          -0.056591180651012554,
+          0.009335648425523075,
+          0.006141624459418827,
+          0.02568943956694641,
+          0.019377132054042116,
+          0.017946464337078746,
+          -0.057184776985927686,
+          0.03208527139096687,
+          0.030132392063755547,
+          -0.003846032371791839,
+          -0.008376524427531415,
+          -0.0010513963058713435,
+          -0.02734868388724371,
+          -0.014879400370545479,
+          0.04383920959748825,
+          0.02563613659516289,
+          -0.020117467074438997,
+          0.0398016699155716,
+          0.029896497884197775,
+          -0.0004195844530067313,
+          -0.023302720788253625,
+          -0.001168337916577517,
+          0.009149021369661628,
+          -0.02508145721900992,
+          -0.004307594138480351,
+          0.00550940415564303,
+          0.014016998918718476,
+          -0.028607379616884317,
+          -0.0018317656587526954,
+          -0.01973464940415756,
+          0.03762738392056255,
+          0.015358106058583114,
+          -0.03931907682104936,
+          0.011519637298832444,
+          0.004482080558226775,
+          0.0747867063383242,
+          0.05550679018263077,
+          0.023243129601931176,
+          0.02671631686665238,
+          -0.009686146566573536,
+          -0.04088582189725495,
+          -0.0065344623594924085,
+          0.019617295999325712,
+          -0.004722087402575777,
+          -0.002871844114437583,
+          0.003935087265300519,
+          0.03229890413393201,
+          -0.02813577047904649,
+          -0.05124172132597234,
+          0.03325235407841968,
+          -0.008687655557217192,
+          -0.011027174838246948,
+          -0.003653553063547375,
+          -0.02350737945182988,
+          0.016223087939183468,
+          0.03789252356462714,
+          0.0034350623378603083,
+          0.00195158657888739,
+          0.011789712862232108,
+          -0.0004435439660769551,
+          -0.0037330456918125894,
+          0.0031552713243866858,
+          0.04465432992298031,
+          -0.0032465421571915712,
+          -0.0075366743861922745,
+          -0.059958752731410256,
+          0.016784616861008125,
+          -0.028734506065772967,
+          -0.03035575736273943,
+          0.02680390892611741,
+          0.02338368719038086,
+          -0.0030658375437784277,
+          -0.03754896050794851,
+          0.07005659523248504,
+          0.0020711575553570695,
+          0.013034425083076923,
+          -0.02316832262132332,
+          0.021332142030732894,
+          -0.0318756233020585,
+          0.02270782509507388,
+          0.01570810309454529,
+          0.004788124787612736,
+          0.026727250200142456,
+          -0.022380142778058276,
+          -0.008593910379436904,
+          0.014439754382387034,
+          0.015007697617383174,
+          0.07662531963612977,
+          0.011195998137723795,
+          -0.010809283538470098,
+          -0.042227822120219305,
+          0.003824320933275175,
+          -0.03797629427607094,
+          -0.02347390684453348,
+          -0.0526876440280276,
+          -0.017799861680287913,
+          0.01701482796266917,
+          0.002730395144917155,
+          0.03789451759899825,
+          0.038338545350313946,
+          0.028463789278406586,
+          0.015494583705011921,
+          -0.05700237266350479,
+          0.047895170359075166,
+          -0.0064406894534929406,
+          -0.0007954721295065988,
+          0.012543029839729358,
+          -0.004703334868741216,
+          -0.0060569679284068,
+          0.012324901254711192,
+          -0.019883935809784068,
+          -0.04253565942154215,
+          0.02164409276211115,
+          -0.0075367555415808635,
+          0.03617159147722157,
+          0.07567745988686005,
+          -0.04225977020036783,
+          -0.022548837589622003,
+          0.029811602145850144,
+          -0.02175792196819533,
+          0.02069861458115363,
+          -0.00941331829664397,
+          -0.035479639885943264,
+          0.020352552580119105,
+          0.003678588588550461,
+          0.06776105274336729,
+          -0.02346047721661687,
+          0.02721075076349443,
+          -0.029386781707644253,
+          0.023530831921117993,
+          -0.02077660533100372,
+          0.020022475323372906,
+          0.024195118676513715,
+          -0.028378518720494666,
+          -0.012900969023913307,
+          0.019264089970719624,
+          -0.0374983624629265,
+          -0.019726171557384355,
+          0.00979636085642416,
+          -0.014479528435832063,
+          -0.014533384583777662,
+          0.005904623758767224,
+          0.010344475500263744,
+          0.05659646358026115,
+          0.04163866113482093,
+          0.007152494819169405,
+          -0.07584500130974763,
+          0.05471858597878099,
+          0.03580211013427615,
+          0.044161429444475306,
+          0.02258180566046248,
+          0.00980779221334715,
+          0.0011193505603397395,
+          0.018086566768892507,
+          0.015605066702191871,
+          -0.00954727399396313,
+          -0.027950079146919375,
+          0.03028514025485407,
+          -0.006351669761257431,
+          0.005095804801686436,
+          0.009620040085743162,
+          0.019748001227987134,
+          0.033709535059510916,
+          0.008203559956940196,
+          0.007389752602218563,
+          0.04197771878351742,
+          -0.02067363758471765,
+          0.007543739188471913,
+          0.011207941261799052,
+          0.013964827251172813,
+          0.03839562075248879,
+          0.0009775244543292637,
+          0.006469507611492216,
+          -0.022808235395746654,
+          -0.024795004398633292,
+          0.006249024864814684,
+          0.011292877109066925,
+          0.01596839619826819,
+          -0.0014105396434174883,
+          -0.04152075438277094,
+          0.008267788411079337,
+          -0.028834191212891387,
+          0.030940526405027906,
+          -0.0018271056434014425,
+          0.006185475047820603,
+          -0.007920574674727123,
+          0.012907560753043616,
+          -0.015165704767187252,
+          -0.021962244496450727,
+          0.0037071053072833147,
+          -0.013508376064336042,
+          -0.015178823540513864,
+          -0.02518557053061311,
+          0.011693256916760944,
+          -0.01716015674592331,
+          0.02048897513447166,
+          -0.003412078966361745,
+          -0.001807298511378529,
+          -0.0012575655363779904,
+          0.03116103961636643,
+          0.011668533361641842,
+          -0.011637010548030312,
+          -0.040260195687925236,
+          -0.011129376691522074,
+          -0.0034493180842871908,
+          -0.02098244789711689,
+          0.026658743116016938,
+          -0.01923736325953796,
+          -0.009723937181300192,
+          -0.013134679989565051,
+          -0.011974548914818263,
+          -0.004028712301052159,
+          0.019628329175867303,
+          -0.028192034966932684,
+          -0.012638520793012082,
+          -0.011568652932013111,
+          -0.0025919910519445737,
+          0.012497908463405111,
+          0.015113186712897145,
+          0.014880607940212641,
+          -0.03059227859452334,
+          -0.021899716217105084,
+          -0.0008513506358489523,
+          0.011985931118464679,
+          -0.01321584595585435,
+          -0.011735288366213981,
+          0.010710683541435907,
+          0.01961722172525993,
+          -0.023571946726192585,
+          -0.03364281880125354,
+          0.0014460076743756465,
+          0.016563167828467187,
+          -0.02891589059260944,
+          0.04595166331401113,
+          0.034271553243501465,
+          -0.027557971577291153,
+          -0.007138821520412525,
+          -0.039508407807651844,
+          0.06814329636830853,
+          0.00022136123564675344,
+          -0.039718016705753735,
+          0.0503457594963753,
+          -0.003580581891576583,
+          0.013820796398521455,
+          0.02054050419088757,
+          0.035682661890720495,
+          -0.0013220739026852943,
+          -0.0052759394697328675,
+          0.04733018719162478,
+          -0.0187845855131196,
+          -0.03982699468608443,
+          -0.023905460465070062,
+          0.002631233133039351,
+          -0.03090995554989513,
+          -0.012617612051488022,
+          -0.013605787969473513,
+          0.039672783793764745,
+          0.008669582242097126,
+          0.009974277556736362,
+          0.005535636914843583,
+          0.014180668493093584,
+          0.012042659947850441,
+          0.011870115226306776,
+          0.00451259454348581,
+          -0.014023048592193627,
+          -0.007769033996392623,
+          -0.05715987527668358,
+          0.003374564518384728,
+          -0.028801158523506985,
+          -0.00756331369170698,
+          -0.02163054894055572,
+          0.005367181573233089,
+          0.02308705505078254,
+          -0.005438909398653553,
+          -0.00873620909426313,
+          -0.0036530652237719077,
+          0.013920407174296543,
+          -0.01539613379172909,
+          -0.03908211213269472,
+          -0.016810726548844992,
+          0.007460173854386443,
+          -0.007069189426742819,
+          -0.041197222498539215,
+          -0.003843233633335066,
+          0.004373367480223065,
+          0.016194121660459908,
+          -0.029165150066068787,
+          -0.00735139702658342,
+          0.017257895082337448,
+          0.014153313280536095,
+          -0.002376178042741926,
+          -0.023223700836427403,
+          -0.0319877537205969,
+          0.02088060774052704,
+          0.0012829290529515294,
+          0.03251550040316258,
+          -0.04128453069131847,
+          -0.023376114412063578,
+          0.04083022735076889,
+          0.005690491480259003,
+          -0.05810403800689653,
+          0.016582434816685452,
+          0.010402038964417569,
+          0.016795644054313566,
+          -0.00908590043098552,
+          0.05003656634510042,
+          -0.037216197954986735,
+          -0.013027960261454162,
+          -0.007090943677204207,
+          -0.05175371787508879,
+          -0.0020065660448063322,
+          -0.014908489552053589,
+          0.010627472758306387,
+          -0.03275473161158777,
+          -0.017334608434133866,
+          0.025321047439048318,
+          -0.0033515195870701753,
+          0.011526068468647207,
+          -0.026783879816504975,
+          -0.0012830935419413407,
+          -0.017506908439998053,
+          -0.02114853058440477,
+          0.004878884893669383,
+          -0.020936900837115968,
+          -0.0025587027781733642,
+          0.01557850246191957,
+          0.023652335794299834,
+          0.011489428313303988,
+          0.0019089137782552768,
+          0.02054907941022243,
+          0.014361232983780595,
+          -0.0056413109308506085,
+          -0.0010300050480036684,
+          -0.03413262505887213,
+          -0.018821160485010483,
+          0.03361989907065425,
+          0.013125889701529858,
+          -0.003742167496379789,
+          0.008813100942466033,
+          -0.0383265293638935,
+          -0.035316957263713485,
+          0.03020499323039515,
+          -0.05001623971243151,
+          -0.00855479667673597,
+          -0.005939674940109814,
+          -0.026451831943555135,
+          -0.008032873049527808,
+          0.0009431571685654594,
+          -0.014387605128858954,
+          0.016382559541755014,
+          -0.024909510232038502,
+          0.003715736505444385,
+          -0.006016712902409688,
+          -0.009110197692342108,
+          0.05507263160354303,
+          -0.022103031032598353,
+          -0.0475415595313825,
+          0.021958434235402655,
+          -0.016670809659931243,
+          0.0323519363801999,
+          0.03867494951405658,
+          0.011124508964363016,
+          -0.008002445714431074,
+          0.034041838454818577,
+          0.018400123235267234,
+          -0.01161754789471199,
+          -0.025901844576206774,
+          0.021800298132870245,
+          0.00943488425697727,
+          -0.05906691123008531,
+          -0.00782549274387971,
+          -0.019357446956797447,
+          0.008682376344965976,
+          0.02863077694455766,
+          0.0024925977373894447,
+          -0.002682170968419412,
+          -0.013590235283325244,
+          -0.021511149507762284,
+          -0.03255453916518719,
+          0.009039307015853905,
+          0.008305286476500875,
+          0.009312625027668458,
+          -0.0014144165766983945,
+          -0.007853869334376857,
+          0.009067065760942163,
+          0.022485201611182105,
+          0.03966189080765722,
+          -0.041627289519629344,
+          0.002707086456239785,
+          -0.022678178050044222,
+          -0.03653753142622053,
+          0.0036574646778835588,
+          0.029996160786576367,
+          0.02901163765559891,
+          -0.01971777193183787,
+          -0.0036589934841109884,
+          0.009181234938554578,
+          -0.025979483002227535,
+          -0.00773645103995767,
+          0.004280770859737866,
+          0.04116668301017644,
+          0.006201085381597264,
+          0.01402728258112097,
+          0.001060407317110527,
+          -0.00920801784000087,
+          0.02109880889040528,
+          0.025211022083982727,
+          -0.029866678264942617,
+          0.014439702646820226,
+          0.009379206296908577,
+          0.04156435812161122,
+          -0.037309655469525846,
+          -0.011819731348299495,
+          -0.02766498020409985,
+          -0.010454143230794476,
+          0.027911532296566614,
+          -0.04127890742215716,
+          -0.006948536183086887,
+          0.010207784113909938,
+          -0.0016584043356136647,
+          0.016437495428919636,
+          -0.014607861750686952,
+          -0.03222311919805017,
+          -0.000610723203118213,
+          -0.02867445198803213,
+          -0.024772816792038133,
+          -0.016510309753766612,
+          -0.018581201775667577,
+          -0.07450654873867177,
+          0.00026832248829154255,
+          -0.06724696990816072,
+          -0.02447002541264633,
+          0.04885569249258513,
+          -0.010195604207134053,
+          0.03332905935704749,
+          -0.02815272112271314,
+          -0.005296340304911926,
+          -0.05908835042526019,
+          0.0038476172058970774,
+          0.025753576863447866,
+          0.02386003857017594,
+          0.005206447371391696,
+          0.024799057871126032,
+          0.012060738268265606,
+          -0.04146503002980051,
+          -0.024949907556862643,
+          -0.014503475223770958,
+          -0.0016340466065744286,
+          -0.0034009167013795906,
+          -0.031298432995411005,
+          0.036241272149880994,
+          0.008902644391348814,
+          0.03550223922667804,
+          0.04463301409716232,
+          -0.010385984413759852,
+          -0.03305004479734889,
+          -0.006952739338820577,
+          -0.0175981588092108,
+          0.042668533202572856,
+          -0.017112184620086308,
+          0.005149173014526686,
+          -0.03219589107997635,
+          0.024181808150698054,
+          0.0014339790422945406,
+          0.01254721190611376,
+          0.011099458649404119,
+          0.0027626836767364253,
+          -0.026883640446090158,
+          0.0024160770766463607,
+          0.023874514847956222,
+          -0.022558433617669373,
+          0.00917667715473093,
+          -0.006754091166465801,
+          -0.033686935718776144,
+          -0.01078389638118685,
+          -0.013595327486167114,
+          0.024712781935748277,
+          0.013168464807451798,
+          0.005875864604862434,
+          -0.02417814420558463,
+          -0.0023978800459714144,
+          0.028650463366555304,
+          0.020176980768301288,
+          -0.02118184201861616,
+          0.019082340279671355,
+          0.019734925740100277,
+          -0.04209645071102001,
+          -0.04217310001316092,
+          0.006553913719098956,
+          -0.003735719227468833,
+          0.0061793909220459,
+          -0.028285889794908054,
+          0.008181788076484304,
+          0.004432488437404309,
+          -0.01743905248466261,
+          0.041221269621866216,
+          -0.014892354419172107,
+          0.005087102208335201,
+          -0.022198383716186407,
+          -0.011936388003605221,
+          -0.00027487042024290816,
+          0.044599215801158845,
+          0.008068975171827777,
+          0.003496950921841073,
+          0.02238779276447619,
+          0.0054839808262270315,
+          0.04816723616939378,
+          -0.02685221260466647,
+          -0.0190356946337995,
+          0.016464136459560113,
+          -0.0176092780782677,
+          -0.012513274095432353,
+          -0.010695421445334002,
+          -0.004784348908457072,
+          -0.022203041334932165,
+          0.010788155380415583,
+          -0.018353897272137543,
+          -0.02277431189907027,
+          -0.003945465850796918,
+          0.016024639792926797,
+          -0.006248368166794934,
+          -0.0038769803892794875,
+          0.0016292556540434277,
+          0.02412864830338344,
+          0.02892955453819876,
+          -0.02408022470555968,
+          0.019190282372105023,
+          0.018941584034503536,
+          -0.006709516380602965,
+          -0.020956377933747532,
+          -0.01964885206488053,
+          0.01434623424964373,
+          -0.01352968586212152,
+          -0.013872556892707697,
+          0.004052798542613835,
+          0.009201147374299997,
+          -0.004086067985859617,
+          -0.04185829685685201,
+          -0.014668698123513235,
+          0.003179403563664022,
+          -0.0006415014901450319,
+          0.019360184344240947,
+          -0.0150817538567364,
+          -0.029771999278536083,
+          0.014221776097046751,
+          0.022944884879589678,
+          -0.04493425220030206,
+          0.02117577719533236,
+          -0.0030648751497261983,
+          -0.00016251533643103604,
+          -3.817296018601414e-05,
+          0.008727767382077138,
+          0.026914779443266063,
+          0.03456565094655899,
+          0.01974806772696219,
+          0.00450800007152441,
+          -0.008952599700759673,
+          0.025184190461155245,
+          0.009846044083064704,
+          -0.025168735183134738,
+          0.010983353565382029,
+          0.016740721743326522,
+          0.0070838930012654296,
+          0.009173879407277282,
+          -0.006078288608766152,
+          -0.018269347893097643,
+          0.013181494188066667,
+          -0.01435693747022231,
+          0.01678907949532146,
+          0.05154463552784974,
+          -0.00039632596424461964,
+          -0.02461439550772063,
+          0.008418147037164186,
+          -0.006544319842335593,
+          -0.010065129286193278,
+          0.02214628550460128,
+          0.0029515574751370417,
+          -0.030862152033847717,
+          0.007509556025223583,
+          0.00675485304374519,
+          0.01894936231559203,
+          -0.02355525295639599,
+          -0.01387885858544803,
+          -0.00431398020087544,
+          0.0021208056468763523,
+          0.01580766504149989,
+          -0.004739477121073512,
+          -0.013204823447960706,
+          0.02994098229517376,
+          0.025496728234662373,
+          -0.00026124946738213036,
+          0.015060064980768175,
+          0.022262345898533903,
+          0.009968408664790368,
+          0.007588431141088797,
+          -0.0083063413609779,
+          -0.031336272171937095,
+          0.0018957858383561667,
+          0.05800811417728381,
+          -0.014937085296865802,
+          0.04313875432235027,
+          0.02018476425923236,
+          0.015657644644987107,
+          0.017880263053253607,
+          -0.01662268082870562,
+          -0.008740053682875323,
+          -0.0017630394204179638,
+          -0.008294417990024647,
+          -0.027613464270562948,
+          -0.005953209556351086,
+          -0.02379961210881074,
+          -0.013878676554701543,
+          0.015418315245979704,
+          -0.014764728065382074,
+          0.0027420640870858457,
+          0.010395538988699663,
+          -0.006590633833835581,
+          -0.00841537209927409,
+          -0.003943772700797381,
+          0.03376338017637296,
+          0.011980354721458463,
+          0.027761929822019422,
+          -0.00011128539412582973,
+          0.009342430952107655,
+          -0.018419526564396063,
+          -0.006839752805318664,
+          0.03271190887864607,
+          0.006011520427944665,
+          0.004652835471939905,
+          0.01485920920301398,
+          0.014305626381398871,
+          0.038015677219487023,
+          -0.005102005401565164,
+          -0.018204460895989052,
+          0.02065495133597456,
+          -0.0049699013356374775,
+          0.019149036693562712,
+          0.008825981577388056,
+          -0.002660055848523568,
+          -0.004292379561953203,
+          0.03090270710860331,
+          -0.020243440437217503,
+          0.018735727616321387,
+          -0.02238236089688497,
+          -0.0276584413929402,
+          0.023028408906176946,
+          0.008320436733620496,
+          0.04047641118026783,
+          0.011463851066078041,
+          -0.02247143742166039,
+          -0.034588725413285185,
+          0.009827713930789184,
+          0.00337071450636392,
+          -0.024446280199879062,
+          0.017542777160283324,
+          0.00780836483490212,
+          0.008453886667283516,
+          0.016624116261219637,
+          -0.004175682824655864,
+          -0.0042784362933758835,
+          -0.001546656717896957,
+          -0.008819739495525477,
+          0.01406790007187259,
+          -0.015487658289541583,
+          -0.021457182907140035,
+          0.024687169583245905,
+          -0.03915911962859764,
+          0.01771290139419629,
+          -0.006510046907173026,
+          -0.00997450880725535,
+          -0.02059560682587115,
+          1.5939836949198004e-05,
+          -0.007159493752862143,
+          -0.0004782363814453761,
+          -0.017503181957042913,
+          0.016643971600145645,
+          0.024056442566869802,
+          -0.013209211443909658,
+          -0.00445521990927108,
+          0.0020313626821877526,
+          -0.029235692455820056,
+          0.006662209097255912,
+          0.016993027053110635,
+          0.00529940612845993,
+          0.0026104743851393716,
+          0.0001707934714983891,
+          -0.031289446611163674,
+          -0.005782208620284939,
+          0.03053516889069683,
+          0.05590704747438521,
+          0.021628254147289437,
+          0.02403314981042378,
+          0.02908606663381421,
+          0.02717206823892633,
+          -0.003198844932040613,
+          0.0016217508103442348,
+          -0.01620660606431983,
+          0.027098323143010743,
+          -0.011664168290676943,
+          0.04322175021589083,
+          0.008652323184522541,
+          -0.049493020736445056,
+          0.004537958157696794,
+          0.00943239454147692,
+          0.034918603888916544,
+          0.016678716235690318,
+          0.0005887414054803805,
+          -0.005894338664998101,
+          0.03279165663336323,
+          -0.02735726709032382,
+          -0.011962070182991011,
+          -0.00036014407560325284,
+          -0.019079003730607717,
+          -0.015938295169860884,
+          0.010357611943935459,
+          0.04278924678845588,
+          0.0009957483598272451,
+          -0.0071422790468850215,
+          0.01571201177743999,
+          -0.009445307674383,
+          0.02286338492075306,
+          0.007336440384050498,
+          0.0042809497342459904,
+          0.025581467690095977,
+          -0.0025007871391868105,
+          0.02300539823997411,
+          -0.000628898771317254,
+          0.004582863909134464,
+          -0.015960578501883316,
+          -0.00930649396114688,
+          0.027838455754803444,
+          -0.011112268986721,
+          -0.029253746504396393,
+          -0.008734785561831333,
+          0.00472579547270089,
+          -0.0003805825712986138,
+          0.0021854358149324227,
+          -0.01183885533661493,
+          -0.010840951201591629,
+          0.019965691788904057,
+          -0.004922943301424112,
+          0.003800048754664761,
+          0.012664049429916252,
+          -0.0030124219674414897,
+          -0.01910844996750822,
+          0.01889983984736845,
+          -0.003858283778519141,
+          -0.024168821338992813,
+          0.020083955711766945,
+          0.01850497889545421,
+          0.045156567155119454,
+          -0.01586824055785609,
+          0.004612757419198795,
+          0.017497774574602505,
+          -0.002172088240279357,
+          -0.019617327081566827,
+          -0.014944795772093997,
+          -0.018212462639041492,
+          -0.015975283672064702,
+          0.012837956594691596,
+          -0.01243789877725244,
+          0.019500571353639314,
+          -0.0380735226388175,
+          0.012590900534826689,
+          0.004832168601136412,
+          0.007505385730121269,
+          0.016011983211212992,
+          0.025994560026860358,
+          0.002157436199771672,
+          -0.033465476168065784,
+          -0.01014599891204166,
+          0.007845373376916211,
+          0.010093432683262838,
+          -0.002849185178237036,
+          -0.018147004183434554,
+          0.003633790611809425,
+          -0.0266147420637404,
+          -0.018644422659309035,
+          0.022926737299761104,
+          -0.04798772015599113,
+          0.017908415301694906,
+          0.024859125158586466,
+          -0.04343699525707025,
+          -0.04886699607556352,
+          -0.011459188432595229,
+          -0.036774381316312464,
+          0.0014128005215813578,
+          -0.0005256067584181517,
+          0.011194992852264102,
+          -0.009342140079829378,
+          -0.018598671981710183,
+          0.01584789414301952,
+          -0.0036412641242200418,
+          -0.004225840239935549,
+          -0.00616696736018045,
+          -0.0025598631463178384,
+          -0.010188803157718543,
+          -0.03244694662411977,
+          -0.03290891136780909,
+          -0.00033742479296272127,
+          -0.009743927051414455,
+          -0.02334288359945272,
+          -0.03447062782872289,
+          -0.008728968107967028,
+          -0.037753659749578956,
+          -0.00515561961442376,
+          0.04070890150714597,
+          0.027517147608335957,
+          0.009449414978133334,
+          0.008086900089929912,
+          -0.013989034604490105,
+          0.05691047488419298,
+          -0.020425883381389086,
+          0.039466620642512155,
+          0.005301493569940605,
+          -0.0037222353899918703,
+          0.013075425214878277,
+          0.027710826422351564,
+          -0.006253665106995653,
+          -0.0017412928073238833,
+          -0.01928096630475438,
+          -0.012920476859996416,
+          0.009444408011467955,
+          0.015799884767184885,
+          -0.0025083061619197524,
+          0.001274457251739847,
+          0.00930533161700852,
+          -0.0006652478540264343,
+          0.017258038491335247,
+          0.03996268817486439,
+          0.003833844612473671,
+          -0.00826404141706295,
+          -0.018822002570963614,
+          -0.003188844900317347,
+          -0.011807006674248829,
+          0.010482058734260632,
+          -0.015410635475145028,
+          -0.011540803927522013,
+          -0.01249428777269578,
+          -0.017081856786754605,
+          -0.002183034976477535,
+          0.003114834030705039,
+          0.0027475411722881747,
+          0.02261627635192778,
+          -0.015872027027080824,
+          -0.022487581027490516,
+          -0.0035823117885461506,
+          -0.02398778449664045,
+          8.13716819067491e-05,
+          -0.01050060782407259,
+          -0.015616181283381218,
+          0.013676480044963827,
+          -0.021513173575512936,
+          -0.01091399089076612,
+          -0.008699176177861256,
+          -0.0006986953850759474,
+          0.005853723710671249,
+          -0.00993355748621621,
+          -0.00010241843582159751,
+          0.01150872694850241,
+          0.005559552266827916,
+          -0.012336449703762907,
+          0.0039431718621504,
+          -0.012740550272925562,
+          -0.004432059174951158,
+          -0.03592309122711106,
+          -0.019665686718735042,
+          -0.01846795637714379,
+          0.005642094448288269,
+          -0.005108964028543505,
+          -0.020586678054151854,
+          0.00668364061048941,
+          0.0034775127050453774,
+          0.004713169526661792,
+          -0.010723667875209928,
+          0.013944735241748791,
+          0.010882420168586074,
+          -0.003518245916227596,
+          0.013827334494230662,
+          -0.001330399936559009,
+          -0.020286697504699764,
+          -0.011139922402071095,
+          0.03538258948179278,
+          -0.024961826464903997,
+          -0.0018019328351763667,
+          -0.0026423695690168066,
+          0.00597003516870726,
+          -0.03071835946501977,
+          -0.013634212232744325,
+          0.02164400349828422,
+          -0.005846552894831441,
+          -0.026367014426434226,
+          0.03399116480234033,
+          0.03473258231629753,
+          0.010041605592929843,
+          -0.013423384340589794,
+          0.04730054254696893,
+          -0.012968978954938308,
+          -0.018274800841400665,
+          0.026826644936918647,
+          0.0278389611094493,
+          0.02525164095926168,
+          -0.029782550447890466,
+          -0.003010713086487275,
+          -0.02393629084357885,
+          0.017993366287762866,
+          -0.0263476663180971,
+          0.01864011499179424,
+          -0.02332263152537849,
+          0.016014908881756034,
+          0.01868464825773604,
+          -0.003104041044182249,
+          0.026313491086684425,
+          -0.0052945248919367815,
+          0.015253731232325074,
+          0.0012371308759955573,
+          -0.02119765517013431,
+          0.007795466385020642,
+          -0.006469500272298062,
+          0.025762169721880456,
+          0.00833992768202466,
+          0.01801901639978976,
+          0.022019940825351116,
+          -0.0409522298213741,
+          0.02003698451081499,
+          -0.001367916864679543,
+          0.01782984895971748,
+          0.0038516134262842373,
+          -0.007885432294782304,
+          0.014221728513400065,
+          0.010991352188016265,
+          -0.012851099261924757,
+          0.003840539873231741,
+          0.0008923482291826014,
+          -0.028928104542716937,
+          0.007066004768771198,
+          -0.026914574065060926,
+          0.025866907400807087,
+          -0.005526774784152086,
+          -0.00998122210956923,
+          -0.020744532673097317,
+          -0.0071425207262770865,
+          0.003754468896273156,
+          -0.012747874997429746,
+          -0.011385805601999295,
+          -0.024700457669046855,
+          -0.02401291382882952,
+          0.029591582802488962,
+          -0.014054332096344003,
+          -0.04015978298687865,
+          0.001234440863385445,
+          -0.023301255402129767,
+          0.01193125007773239,
+          -0.005793124129267758,
+          -0.024440100649464773,
+          -0.042563399082257154,
+          -0.017548888269389457,
+          -0.0009394887961485922,
+          0.022739763644266633,
+          -0.00114156949394707,
+          0.041791395064755515,
+          0.0008816696205437911,
+          0.022207132857533028,
+          -0.015547580543606755,
+          0.005319731264840819,
+          -0.02071229859314312,
+          -0.041495579742912285,
+          0.03660432378985971,
+          -0.014637031447181937,
+          0.025854844497435178,
+          0.03324325296704186,
+          -0.013009347371118843,
+          0.001354504885302625,
+          0.02652570709474851,
+          -0.014618100923440427,
+          -0.010782232747863238,
+          -0.0034936699515378212,
+          0.003461670978569965,
+          -0.006468116285021298,
+          -0.027793221545159466,
+          -0.007752458229193391,
+          0.04193706715422074,
+          0.017894069367616637,
+          0.021928994684423043,
+          0.022606401934291136,
+          -0.017972404868085358,
+          -0.026402048654300563,
+          0.003656646576244765,
+          -0.013129722990153284,
+          -0.018090360146368677,
+          -0.01624598856366761,
+          0.012397150789705746,
+          0.011741002980073786,
+          -0.004052422390675261,
+          -0.01836750423615859,
+          -0.020929335029248472,
+          -0.016327412417122416,
+          -0.0042615345226698015,
+          -0.008458117095535,
+          -0.005002582649405199,
+          -0.012158173174049797,
+          -0.020409616620022143,
+          0.03706368540614453,
+          0.0285565642978937,
+          -0.0041732933884846566,
+          -0.00607154864704767,
+          -0.011142150176334957,
+          0.008834431806990914,
+          0.006021232046087652,
+          -0.005028870471297621,
+          -2.3396043292139426e-05,
+          0.005112033372713154,
+          0.022822312435625314,
+          0.007271235299348001,
+          0.0035791789459533856,
+          -0.06340305811365254,
+          -0.005392000517722054,
+          0.022560201096078862,
+          0.012581401789917604,
+          -0.0032248219865223496,
+          0.025180718735801837,
+          0.0006451724964970927,
+          -0.009714514747463593,
+          -0.018389422219956937,
+          0.024728862150150004,
+          0.008760329201960964,
+          0.020988700992999768,
+          -0.014974763313686516,
+          0.003845297486923296,
+          0.0006445983847138896,
+          0.00750394530065325,
+          -0.0019792992847026793,
+          -0.008578254004439607,
+          0.025776677078006516,
+          0.016593438298233645,
+          -0.005695578040595484,
+          0.01604018433477423,
+          -0.0017497260488899657,
+          -0.021460543443428924,
+          -0.010265213095609554,
+          0.00042017209251926397,
+          -0.010570864288876387,
+          -0.018015790888295944,
+          0.017569414447505878,
+          -0.002601036282535863,
+          0.0072399377790164435,
+          0.030281707151249355,
+          0.0423183067569767,
+          0.009566493398534713,
+          -0.005928603293907002,
+          -0.0037936911251953654,
+          0.015477873304924939,
+          0.01513127995244932,
+          0.010741256034652754,
+          0.03015858543221727,
+          -0.029247492875021843,
+          -0.006690306960497428,
+          0.0046653727410352576,
+          0.004610405633429778,
+          -0.011692179099382171,
+          0.030055897497741527,
+          0.04569730592994177,
+          0.006736813244286893,
+          -0.004799978513117371,
+          0.002121067028052434,
+          0.011299889107780231,
+          0.007768137218140627,
+          -0.018927080714961783,
+          -0.0012222962555201746,
+          -0.006009911026251445,
+          -0.014001766533604134,
+          0.026941585318261085,
+          -0.003960837039351648,
+          -0.011848232223811548,
+          0.02903357274500332,
+          0.1105267435019781,
+          -0.006011029338781496,
+          -0.0053646998281195206,
+          -0.03242321704621517,
+          -0.015194102811332289,
+          -0.008860590919147765,
+          0.0056988205877963076,
+          0.01635957585050017,
+          0.020327661913559888,
+          -0.0008943408274661007,
+          0.016820860249955754,
+          0.02822175614376368,
+          -0.013854827839129885,
+          0.007906244895327178,
+          -0.030891381690403466,
+          0.015702117924362928,
+          -0.005383974926093977,
+          -0.012032955400313954,
+          -0.0026169511828034054,
+          -0.008618307035330928,
+          -0.0037356680273096105,
+          0.0009666430093455632,
+          0.011296307527684261,
+          0.004427516814276546,
+          -0.014484198313369681,
+          0.002829529156309784,
+          -0.008595300985863424,
+          -0.005122996529848575,
+          -0.02393571073523279,
+          -0.024119245543788293,
+          0.005223147047205755,
+          0.0019389305548629142,
+          6.344308292376579e-05,
+          0.006848965054539454,
+          -0.003810712095343454,
+          0.030149081317333418,
+          -0.01700322008272963,
+          0.0012288912810468446,
+          0.01193226302726573,
+          0.012332749933413344,
+          0.01688602002694633,
+          0.0028600984823761366,
+          -1.9882692552530084e-06,
+          0.021172977047420227,
+          -0.010514036509402561,
+          0.01460334896316005,
+          0.013091417303145243,
+          -0.04103572381458182,
+          0.0089746401463863,
+          -0.0027841289484449873,
+          -0.024252447218794344,
+          0.004446472209246425,
+          -0.018987867343500102,
+          -0.00881974688349601,
+          0.0005365625696166774,
+          0.012054719659248512,
+          0.01960300421041832,
+          -0.008553943699846696,
+          0.02111201386480852,
+          -0.0003501657740808554,
+          0.004177088753111459,
+          -0.012879944670340026,
+          0.0065222126975051685,
+          -0.007563354540402343,
+          0.0008168128607271199,
+          -0.018213241937597355,
+          0.012526561612175514,
+          -0.0008285725179894544,
+          -0.015737003159436623,
+          -0.011337688275309921,
+          0.004737346137976519,
+          -0.0027993545140114705,
+          0.0030843163114700313,
+          0.010153740340626154,
+          -0.008815302484351509,
+          0.004919275910399053,
+          -0.0017541042538165297,
+          0.002562816687714991,
+          0.014285249186421434,
+          -0.029816964593980453,
+          0.005357227738099325,
+          0.009154842495811638,
+          -0.00048782303639333943,
+          -0.013074976152495576,
+          -0.004054505559072866,
+          0.0028322102449485015,
+          -0.016877328079971877,
+          -0.012128199639779175,
+          0.023421330164419888,
+          -0.012565145848820333,
+          0.008609070188894315,
+          -0.010936598722317274,
+          -0.018401801734378692,
+          -0.01066119413133089,
+          -0.005254229780560808,
+          0.01234969508270852,
+          -0.0038427767204741877,
+          -0.006947594635952202,
+          -0.01756894543880097,
+          -0.007890897373987378,
+          -0.047246698568222466,
+          0.009617876521097624,
+          0.027157555263862723,
+          -0.011355758309667893,
+          -0.04050311381480088,
+          -0.01017774338301804,
+          0.02437257208967103,
+          -0.004056329295095018,
+          -0.008345450168215274,
+          -0.001599171409135998,
+          0.018358386463243014,
+          0.0027207445181251406,
+          -0.007292967749349572,
+          -0.011267861428599221,
+          0.01657507035043879,
+          0.015105061280816214,
+          0.021419854579631276,
+          -0.02088952706825817,
+          0.006295984081511286,
+          0.020973020776936996,
+          -0.03073246543555915,
+          0.0023720435876790876,
+          0.03363966216709148,
+          0.010412588613998134,
+          0.013023311559810034,
+          -0.0010217662933572685,
+          -0.0075567381078977215,
+          -0.008248818794295818,
+          0.011848200323549865,
+          0.005066463524567732,
+          -0.023174489297821385,
+          0.021823825355858556,
+          0.020368868756448753,
+          0.0026042856858509524,
+          0.003457067317891926,
+          0.0005539745115410357,
+          0.01062670844108589,
+          0.009948889803890108,
+          0.05100741556350918,
+          -0.006493714124814164,
+          -0.014021751055930108,
+          0.015063988102064609,
+          0.02488059237697814,
+          -0.0023499941119596228,
+          0.01087282522970853,
+          -0.02612218935931224,
+          -0.021423880914461062,
+          0.023389294741183817,
+          0.008967763132667892,
+          0.009637806238448432,
+          -0.010815706635624337,
+          -0.011390999462441815,
+          0.031750675158698435,
+          -0.01307829462211914,
+          0.00759749911213318,
+          -0.02169895957209634,
+          0.01387879409130081,
+          -0.008606362268863441,
+          0.032229658009209815,
+          -0.0009066257436946125,
+          0.011062436424260079,
+          0.028010486866468023,
+          -0.007835805879241447,
+          -0.019657486763625547,
+          0.003967480939987674,
+          -0.015577952617624695,
+          0.02188429826317828,
+          -0.020502758155233298,
+          -0.0002362500702658591,
+          0.0040935406015014265,
+          -0.010056462958039452,
+          -0.0057038083589231615,
+          0.011475170331848666,
+          0.0015345076181482267,
+          0.006359262977587827,
+          -0.020293014401106,
+          0.01369706440475074,
+          -0.003989482215141903,
+          0.007887824197823247,
+          -0.015925992045349828,
+          0.014370223626942766,
+          -0.02309139937510304,
+          -0.021808736033141787,
+          0.008142621179624918,
+          0.0108996933426044,
+          -0.007589050511458233,
+          -0.025971539510030145,
+          0.02250992080022228,
+          -0.024980833805349545,
+          0.024979008145354243,
+          -0.006622339940663538,
+          0.002566378488088469,
+          0.004913874740363343,
+          -0.008102103508732738,
+          -0.0028098905427027783,
+          0.0057865587191235565,
+          -0.0073509382400023995,
+          -0.014123756810537343,
+          -0.019453380788786282,
+          -0.04083629135586275,
+          -0.03177927047231389,
+          0.022927581786088044,
+          0.011557232599801421,
+          0.0017073832658476193,
+          0.02339552660545296,
+          -0.02247471449131386,
+          -0.017890866287500053,
+          -0.009775068417716798,
+          0.025184899522229628,
+          -0.04408541669037361,
+          0.012104255392615549,
+          -0.014669458760022245,
+          -0.0245258242481677,
+          0.006274252098230249,
+          -0.004161960147481513,
+          0.009299171724026455,
+          -0.010779421730862708,
+          -0.04713627671791054,
+          -0.005192684784377356,
+          0.02330975631183273,
+          -0.0036802876364168935,
+          -0.006197548344287773,
+          0.03048284650641829,
+          -0.0019503797069087845,
+          -0.005038254331399784,
+          -0.005524875459245342,
+          0.027783596017224226,
+          -0.024950083604372984,
+          0.010988398531346805,
+          0.008304937912856357,
+          -0.008485065895193593,
+          0.006572665697403556,
+          0.0011518846966102158,
+          0.011927348472282199,
+          0.015632588644277183,
+          -0.015559413716826264,
+          0.02262990861928957,
+          0.011446850286896198,
+          0.006506075777765303,
+          0.008015225053785824,
+          0.00010054430757843335,
+          0.0022975742216194894,
+          -0.024948925449668525,
+          0.026047924967851216,
+          0.01888211557921096,
+          0.0001632181017876392,
+          0.012553094476616919,
+          -0.015042621225242777,
+          -0.02782009986290425,
+          0.02606646297992026,
+          -0.016847653820092335,
+          -0.00045272379465939274,
+          0.020053091486344633,
+          -0.016210036358455773,
+          -0.007930550661303182,
+          -0.01123514849773852,
+          -0.0162675150307978,
+          0.01803076559921804,
+          -0.012160217543572069,
+          -0.02216594536057653,
+          0.002594675636753233,
+          0.016443123828746582,
+          0.02134480656850119,
+          0.026694575953313482,
+          -0.011470930697351533,
+          0.017768927048191052,
+          0.0008653408391607816,
+          0.001865765599795092,
+          0.025745954171649543,
+          0.01702677011968492,
+          -0.008474889109168202,
+          0.025347176686722313,
+          -0.0016686223698600755,
+          -0.002421921060670558,
+          -0.012265504084322075,
+          -0.0165214436458303,
+          -0.007545471138114208,
+          -0.015333235556146196,
+          -0.0051830106590765685,
+          0.000762746782664525,
+          -0.0018118022696726346,
+          0.10876576838038368,
+          0.0034948044540582626,
+          -0.011608460689239234,
+          -0.005870263500417206,
+          0.007510774308250375,
+          0.0023525201199670765,
+          -0.006846154764389575,
+          0.016082721103476375,
+          -0.018616626204086575,
+          -0.023593744079394,
+          0.004743481974389998,
+          -0.0009262286319469741,
+          0.00870122878944883,
+          0.02029523251993269,
+          -0.008608837818169835,
+          -0.0200567711498164,
+          0.007089477722191928,
+          0.005535149644041221,
+          0.0023640746516071774,
+          0.005256431824732927,
+          -0.0006118671321824933,
+          0.008209920240361308,
+          0.02448422122961942,
+          -0.015622892878860465,
+          -0.029770836607612903,
+          0.010535371601298532,
+          -0.00015994156470228762,
+          -0.007193425789067776,
+          -0.011046316463797128,
+          0.014728841492009627,
+          -0.0011488451990980268,
+          -0.02751404133172324,
+          0.017663132712536528,
+          -0.019055335178900415,
+          0.014643246200933506,
+          0.0025348345163347464,
+          -0.00016387173410118788,
+          0.022565953069005714,
+          0.013101158251834834,
+          -0.04272085076882881,
+          -0.0034442020309914823,
+          0.01041205537746844,
+          0.011059422927946552,
+          0.0116446016862942,
+          0.0012992535381387887,
+          -0.018670801241345514,
+          -0.011788984490283379,
+          -0.009109849103208542,
+          -0.021268488414296117,
+          -0.012523033785938503,
+          -0.011393853408628214,
+          0.010743646386439633,
+          0.0012964030814325153,
+          0.023809566099779113,
+          -0.003030535542277875,
+          -0.006526351867141901,
+          -0.000430699447948508,
+          0.00880561288237257,
+          0.009694646009259077,
+          -0.02079011138439714,
+          0.008294338719512953,
+          -0.01823922789220214,
+          0.011819769685519291,
+          0.006416044253693268,
+          -0.022799568832306067,
+          -0.009911619276012527,
+          0.007152786695893951,
+          0.008393677195358034,
+          -0.013569953300860946,
+          -0.006503988731872938,
+          0.02299498754558268,
+          0.018870619152710703,
+          0.012233964644738905,
+          0.024994616203196145,
+          0.03838969722235787,
+          0.005744255912479976,
+          0.027682123067526515,
+          -0.016791471092091604,
+          0.008459745383680297,
+          -0.022493868174622695,
+          0.012902262638928338,
+          -0.00460399519266689,
+          -0.0050431343566427886,
+          -0.018188192498411616,
+          0.033765105245412236,
+          -0.009159751849801773,
+          -0.0027150644587327763,
+          0.034778224828120714,
+          -0.0066502021793471075,
+          0.004525230911930458,
+          0.005096748531775903,
+          -0.016067049494073893,
+          0.033274672789154185,
+          0.01686831040872425,
+          0.016353966592947262,
+          -0.021631941883929817,
+          0.011748317746828848,
+          -0.005675701689448899,
+          0.0001273442591116645,
+          -0.020738819037390263,
+          -0.0037528323797385733,
+          0.0027725541343763104,
+          0.0059860509282538546,
+          -0.016556134518370437,
+          0.02033623476757896,
+          0.00019647023567641008,
+          0.0052895744489259875,
+          0.03253462247073863,
+          0.01043311189804924,
+          0.019182938878855655,
+          0.010051051068017807,
+          0.019523739618278902,
+          0.0281939391326836,
+          0.025685787482623124,
+          0.0029932688462254147,
+          -0.018668945726139946,
+          -0.0009326901784812825,
+          -0.007880352863625262,
+          0.010492063055849727,
+          0.022849902214818293,
+          -0.022619228461640748,
+          -0.02772644874135741,
+          0.018566981184526675,
+          -0.0022608185345048414,
+          0.020031103834122595,
+          0.007544464625538984,
+          -0.020935181884939796,
+          0.003407355515103286,
+          -0.023178230166337178,
+          -0.0005418155871620593,
+          -0.031039187720144967,
+          -0.028591174077243604,
+          -0.0007659241904525392,
+          0.011945344534674635,
+          -0.01688153017778311,
+          0.03357788163306365,
+          0.017667778671101954,
+          -0.014289109343082812,
+          0.01604684528817524,
+          -0.011054587325539086,
+          -0.0036959091138078683,
+          -0.005819853870004682,
+          -0.00203755010992417,
+          -0.02157944474979592,
+          0.005138614763345542,
+          -0.0033509138150859593,
+          -0.0024871060745482883,
+          -0.016936575573598154,
+          -0.021901830366872595,
+          0.016375173843894617,
+          0.016742770709641697,
+          -0.006606252542838758,
+          0.039771905404651496,
+          -0.004238071340064403,
+          0.00018364315349545657,
+          0.011690521707403059,
+          0.0010018392078563795,
+          -0.013651921068215563,
+          -0.010829994512513501,
+          0.012362647379788833,
+          0.011321403447882543,
+          -0.018942043512252663,
+          0.005763773608775982,
+          -0.009131565038297889,
+          0.002089216671850269,
+          0.005438887787329238,
+          -0.008183835275629478,
+          0.006690280329947951,
+          -0.01673229763262922,
+          0.015833397876364723,
+          0.00295474085300386,
+          0.003186554859191664,
+          -0.011797859525926772,
+          -0.020991927615680524,
+          0.01576413532776261,
+          0.0032403514518527287,
+          0.018819824905201635,
+          -0.026755218746060662,
+          -0.00012946467986739473,
+          -0.009552681500884924,
+          -0.0028941357661144826,
+          -0.012533800273548772,
+          0.013975640628890588,
+          0.016864739319953755,
+          0.0063941926578529,
+          0.01781016436903586,
+          -0.0026268224264609724,
+          0.015368284773773678,
+          -0.018122416427266218,
+          0.008686009180697955,
+          -0.006801074032721299,
+          -0.018537521789181048,
+          0.0013237338264105415,
+          0.027863741654435434,
+          -0.004670012662973029,
+          0.011066853115455966,
+          0.011704884649534624,
+          0.018954056680143452,
+          -0.0012626215734593707,
+          -0.0030850559460101866,
+          0.015610404665171367,
+          -0.005838226405658572,
+          -0.006861883833171456,
+          0.020872966233916948,
+          0.032951653763679145,
+          -0.006716215648910268,
+          -0.0021292599951407863,
+          0.010886596327481677,
+          -0.004901684487088035,
+          -0.008644584974417106,
+          -0.009587037829603014,
+          -0.03158137900717491,
+          0.024369089446706894,
+          0.012092412439477474,
+          -0.009601042951586109,
+          0.022690862146548786,
+          0.020752792552278052,
+          0.011412743968296708,
+          -0.010471860708195726,
+          0.02291536038589634,
+          -0.01268035142112569,
+          0.024168869865226136,
+          0.032482831775271157,
+          0.004400973556358811,
+          0.0022312906286559167,
+          0.0021175489661843985,
+          0.01691348460546639,
+          0.006050357346231155,
+          -0.007068133214675996,
+          0.011656003476409986,
+          -0.018487923180009552,
+          0.01598530891381001,
+          -0.004432366596808033,
+          0.02808371007118948,
+          -0.01337145216499331,
+          -0.010473700033368562,
+          -0.00537325376270318,
+          0.00010726431372328729,
+          0.006445701087575359,
+          -0.003324002570256215,
+          0.005255443653813165,
+          0.0007072577303252132,
+          -0.017825649146944237,
+          0.003050128130393781,
+          0.004825967170196209,
+          0.009018799597337101,
+          0.004103653497834134,
+          0.007249957401824204,
+          -0.012360355929483,
+          0.01441885664789707,
+          0.004401820363035808,
+          -0.0026190938717927086,
+          -0.0007163225760398784,
+          0.011124538295057815,
+          -0.013232822471919533,
+          -0.004895624967029031,
+          0.005281228035321008,
+          0.0059275753353475414,
+          -0.007507924998741734,
+          -0.004484140989405051,
+          0.011908076549392969,
+          -0.0037917291321648483,
+          0.006351543504726163,
+          -0.004409464890606014,
+          -0.0037760571637986827,
+          0.005303661313434895,
+          -0.00974511544836468,
+          -0.004204167898342997,
+          -0.0007093125445559502,
+          -0.006869250680233214,
+          0.00786202375969793,
+          0.005971033227117194,
+          0.010606085272785863,
+          0.006969437776542303,
+          0.004293129058563288,
+          0.004114452739123398,
+          0.004071017374828283,
+          -0.002445681623750424,
+          -0.00592480769363354,
+          -0.004617501328052899,
+          -0.012067458701076778,
+          -0.008335239153829813,
+          0.009025028998949398,
+          0.008653779421856954,
+          -0.019363018792788574,
+          0.02048011578221327,
+          -0.001188320960771659,
+          0.00772165044232104,
+          -0.01602086976290928,
+          -0.000907553574811279,
+          0.005464880629734941,
+          0.008267702703609528,
+          0.016136762227127184,
+          -0.03654209526492064,
+          0.005179791231187316,
+          -0.0037321883493524766,
+          0.016026013709955444,
+          -0.008215703411373182,
+          -0.006979816286354862,
+          0.02564060526656963,
+          -0.01450763388985563,
+          -0.0009847758228920171,
+          -0.016142304341196283,
+          -0.01407778890808498,
+          0.010815725252959506,
+          0.02050940280233426,
+          0.020684808835848383,
+          -0.0004965802897137407,
+          -0.0005781445948759235,
+          0.00036179841583004193,
+          0.022775804864342608,
+          0.015545681427566235,
+          0.018697154644607316,
+          -0.004543500782944068,
+          0.007028436664826706,
+          -0.023057215501463973,
+          0.009912581556496658,
+          0.0011795417129232596,
+          -0.008215027850908477,
+          -0.008554137076267468,
+          -0.008703378564520125,
+          -0.010448538943182258,
+          -0.011773811327190638,
+          0.004106616839960258,
+          0.013895013678116032,
+          0.004190549543250594,
+          0.0031571190600981074,
+          -0.0225385788570314,
+          -0.006243193914542516,
+          -0.011229641377928767,
+          -0.01674214043453308,
+          -0.007367389908398082,
+          -0.02386334969150212,
+          -0.028021339684039734,
+          0.0002794207370355175,
+          -0.02411861258809571,
+          -0.015958336406924612,
+          0.005268815864773021,
+          -0.01546610225139129,
+          0.00898600137695074,
+          -0.008396481690803911,
+          -0.011873486990732417,
+          0.0074743033679134105,
+          0.015336444788692,
+          -0.017676977079590463,
+          -0.010165317702640217,
+          -0.01690524990506033,
+          -0.013460451917691695,
+          0.010918550283871637,
+          0.00595960977280955,
+          0.009250157926438213,
+          -0.012447979337850305,
+          0.0062932435120863365,
+          -0.012673905639312696,
+          0.015073013552721701,
+          -0.03769166067890086,
+          -0.029992249463405965,
+          -0.00014990754540999537,
+          0.007288036630677436,
+          -0.0020735742042505426,
+          -0.018695120921334806,
+          0.0003860116505754868,
+          -0.003608995261360232,
+          -0.014900191335144973,
+          -0.013866216738266176,
+          0.025251475305524053,
+          -0.006788483107616133,
+          0.01223227424986397,
+          0.0034937277040015626,
+          0.03438333560343413,
+          0.008803852645562129,
+          0.014666206816302367,
+          -0.005709421955160623,
+          0.003742433643393657,
+          -0.0033706258426092744,
+          0.006906464031352315,
+          0.0076578888497945185,
+          0.009276592191710524,
+          0.008025321004305927,
+          0.004576470294889512,
+          0.002887346543823031,
+          -0.001925468214331082,
+          0.02125241545487307,
+          0.025671028960818562,
+          -0.00037467906648160044,
+          0.008981063192139038,
+          0.0009387900084090577,
+          0.0002735789978680719,
+          -0.008374290340790827,
+          0.006547013284767913,
+          -0.008274637814401893,
+          0.004041066530254948,
+          0.012897549078838394,
+          -0.009150863940242148,
+          -0.010211568689323908,
+          0.016575088843292566,
+          -0.012380099999574783,
+          -0.03420455578618604,
+          0.026550654680719987,
+          0.005324512466403649,
+          -0.0035670106106754245,
+          -0.010657928468872608,
+          0.0037176027853158167,
+          -0.01103891334887315,
+          0.008096915671982816,
+          0.022587659771132487,
+          -0.0013527917944322358,
+          -0.0018085171207746743,
+          0.0038503794877980586,
+          -0.011870484733648887,
+          -0.007881489342351292,
+          0.006462772034381968,
+          0.0315696514747912,
+          0.01326417653779577,
+          -0.020400369539140532,
+          -0.015223808971473513,
+          0.020406862349006738,
+          -0.023142176712111575,
+          -0.006254959806032159,
+          0.009039506093607052,
+          0.0009087520492922078,
+          -0.022766046838415165,
+          0.0009662037093312435,
+          0.004701966191595815,
+          0.005946458676746661,
+          -0.006358986495019584,
+          0.03237532341275415,
+          0.001075992257265679,
+          0.015892020271402903,
+          -0.017189453434945788,
+          0.0017514632185789299,
+          -0.014189278671121677,
+          -0.014265447388465805,
+          -0.029809460132405256,
+          0.0005557491671071693,
+          -0.0030324198417091704,
+          0.011038984948790744,
+          0.024808067315720335,
+          0.01145103533194245,
+          0.014432246996007246,
+          0.010552852947578788,
+          0.01513689173494211,
+          -0.013653722180565311,
+          -0.029009757850432732,
+          -0.005171522291984226,
+          -0.018001048743500243,
+          -0.02297482628212251,
+          0.003962613844063587,
+          0.013645125285548414,
+          0.006278662690346082,
+          0.023786381386575035,
+          0.0021738516691483163,
+          0.0009580591750940011,
+          -2.690805459518147e-05,
+          0.008894091070322921,
+          -0.004952203441302471,
+          -0.006553986197076102,
+          0.003570189040872711,
+          -0.000652714137223501,
+          0.021228360881461853,
+          -0.005758946428212041,
+          7.897153237591187e-05,
+          0.014242175492137011,
+          -0.005622256348934008,
+          -0.0035009192529791146,
+          -0.017339926143652377,
+          0.006248842389703505,
+          -0.01622011586976866,
+          0.025783086900871728,
+          0.018450783573486916,
+          -0.009109295293546267,
+          -0.013469521573468915,
+          -0.011259555201313116,
+          0.02773097452736661,
+          0.004631282554715908,
+          0.010787055220617197,
+          0.018228322680582464,
+          0.01461254259249465,
+          0.01784830193964943,
+          0.004664725683522323,
+          -0.0034339781146649384,
+          -0.0007692550982545399,
+          0.021176646860433974,
+          -0.014700178590249292,
+          -0.002701208084390198,
+          -0.008842788249773623,
+          0.0021799005046816444,
+          -0.008431299000066363,
+          0.006086447848362669,
+          -0.010793016137040172,
+          -0.015251381840165954,
+          0.0189503457652375,
+          -0.02660767353735015,
+          0.016423753600066336,
+          -0.00593016170680532,
+          -0.00482891500342298,
+          -0.027459214027897378,
+          0.010728039528784636,
+          0.0027866316191608297,
+          0.0022519974195000687,
+          0.010770472550461752,
+          -0.021427634123401408,
+          -0.01867717541189363,
+          0.008532444554010673,
+          -0.011457117350255678,
+          -0.007152301670244725,
+          -0.00348619742388509,
+          0.0154715607296319,
+          -0.00017898955187849443,
+          -0.009925365923862246,
+          0.02085093731060963,
+          -0.007175309371250024,
+          0.01223979370120031,
+          0.005256127464980655,
+          -0.008756077893734508,
+          -0.0028438320920811362,
+          0.021022708898898203,
+          0.027557382862284808,
+          -0.008342206702778956,
+          0.014526066580760486,
+          -0.011883797998224506,
+          -0.01088335352319397,
+          -0.03479444009382274,
+          0.030104838505937447,
+          -0.013305732257226255,
+          0.026773747582411633,
+          -0.0006968265979630097,
+          -0.017037729111584094,
+          -0.016206207453228583,
+          0.01625225876515654,
+          -0.0024290584023212254,
+          -0.009366287391027166,
+          0.01465885301870365,
+          -0.02428628495473643,
+          0.0058336571592899735,
+          -0.0006627935593670669,
+          -0.011531762838191602,
+          0.013275508677482622,
+          0.0019223252930710324,
+          0.010738622785461513,
+          -0.02030405805119001,
+          0.0009661595529028194,
+          0.009263716373456188,
+          -0.016791855691574455,
+          0.006057830565306032,
+          -0.03730952786847912,
+          0.0029341721588188943,
+          -0.014833895684348656,
+          0.005725615593914644,
+          0.001498755253963699,
+          -0.01820708352937343,
+          -0.03652692232407375,
+          0.00043665701245599497,
+          0.01913883585802903,
+          -0.007975988799790905,
+          0.020538595326175857,
+          0.008243569241768276,
+          0.008421404577931517,
+          0.002314738067443065,
+          0.0016702403734608475,
+          0.008279538988821465,
+          0.03679456861213399,
+          -0.01775927634148114,
+          -0.004836606118958108,
+          -0.0012151329859941935,
+          0.010644195708319258,
+          -0.02028470571116305,
+          -0.009287515726906219,
+          -0.0036506912311261536,
+          -0.013134467570201307,
+          0.030615862647585222,
+          -0.020265555441523482,
+          0.025634449009756734,
+          -0.009300770899023007,
+          0.009274072905190077,
+          -0.02097685009125915,
+          -0.009797008690754956,
+          -0.010661540100131183,
+          0.0019532291898233895,
+          0.013465005282849381,
+          -0.009151430285942593,
+          0.0071467934925125145,
+          -0.006655608635025676,
+          0.01020327561163374,
+          -0.001472597695707201,
+          -0.02443508800166349,
+          -0.0024998811551575563,
+          -0.01107321258077183,
+          0.02456164421154801,
+          -0.0019619696941303388,
+          0.0005043179699158694,
+          0.010951580163871948,
+          -0.00853593956432799,
+          0.023927336608555987,
+          -0.008855702280766362,
+          0.0010093679746665581,
+          -0.004863269000020324,
+          -0.004005691415117686,
+          0.001698212044518236,
+          0.01822024022650613,
+          -0.005707404715849356,
+          0.03722587302228139,
+          -0.01708925706528118,
+          -0.0065500772743641925,
+          -0.00015538686795361824,
+          0.010852925330001283,
+          0.010035359503147549,
+          0.008754719437760327,
+          0.004067820167375348,
+          -0.014861417626433965,
+          0.009346669946626484,
+          -0.020136029447262147,
+          0.007501669351583119,
+          0.023178490169347605,
+          -0.006840359541885465,
+          0.0006676450965357345,
+          -0.01283320739092204,
+          -0.015610538827742276,
+          0.006308543164022682,
+          0.002329420005586303,
+          -0.002295739637504597,
+          -0.02181189272979859,
+          0.006298804075737111,
+          -0.014398470126723037,
+          0.004007419942728446,
+          -0.0007497656993578918,
+          0.013863867106289847,
+          0.016451166273826513,
+          -0.006740209707090658,
+          0.039292260347522444,
+          0.01871723939316731,
+          0.009109005310505462,
+          0.005674782595856756,
+          -0.040588444759736254,
+          -0.008993850998154852,
+          -0.0010197185016393919,
+          0.004866150984083949,
+          -0.01141354216871663,
+          -0.018944881712855702,
+          -0.002646110151132781,
+          0.000308585036997592,
+          -0.010681954707313127,
+          -0.009091585604191968,
+          0.01751599002710486,
+          -0.006034424764302979,
+          0.00872057856926618,
+          0.004146961781109887,
+          0.023758065635807138,
+          -0.009727802672285544,
+          0.01176257119851645,
+          -0.0071822083765114165,
+          -0.005456197751568526,
+          0.0006726765703478756,
+          0.011011628907288813,
+          -0.008031792262886802,
+          0.009763340867550022,
+          -0.004279891138185881,
+          0.008355720233269893,
+          -0.008552596384067825,
+          -0.023952129458400563,
+          -0.014972827087541732,
+          0.014347535404251018,
+          0.011651260496709961,
+          0.008137738462704715,
+          -0.013624435569539285,
+          0.0031728721674035352,
+          0.00675830277744976,
+          -0.009184784787581942,
+          0.009161771311910335,
+          0.017733082848622708,
+          -0.001411542568472906,
+          -0.0017830167499636113,
+          -0.0016130003349297027,
+          -0.012173528776899308,
+          0.0011196437753239162,
+          0.02284887358189995,
+          -0.003054479643937998,
+          0.014614283913976716,
+          0.010689486748036806,
+          0.009643292262102645,
+          -0.00810930837603355,
+          0.001915955511174678,
+          0.02341863048264366,
+          0.015022291783105098,
+          -0.004625715884342598,
+          -0.011422973894623295,
+          0.004259757717597968,
+          -0.010332744704689279,
+          0.007373328473086509,
+          -0.02117940113144938,
+          -0.004672698999328004,
+          0.008896311380763048,
+          -0.007028862917603513,
+          -0.01732432285099198,
+          -0.028026173555055017,
+          -0.016765805892242715,
+          0.019819329380181,
+          0.0027213760678977965,
+          -0.013291251876048186,
+          -0.0032919084721662363,
+          0.0031214883337300926,
+          0.002553005168048115,
+          -0.0071104806975682885,
+          0.005310583052470149,
+          0.007359183453835734,
+          -0.008674133980486269,
+          0.017665958608281126,
+          -0.000521269431423141,
+          -0.0008020251454616109,
+          -0.0005442374261144659,
+          0.0006127687598711624,
+          -0.016387752806126442,
+          0.01666161912459431,
+          -0.010949228376419892,
+          0.015198110119816624,
+          -0.008966726991404666,
+          0.008600499511690635,
+          -0.007617043351740469,
+          0.009458641139070436,
+          0.011597957400445055,
+          0.012755037944648426,
+          -0.004403680820063604,
+          0.008519241473418196,
+          0.004807816191718704,
+          -0.008147125240985166,
+          0.007017601015398136,
+          0.01296082929962557,
+          0.00582353932160707,
+          0.0013634237970372698,
+          -0.0008246894322793264,
+          0.0022035670958705576,
+          -0.002480976219285683,
+          0.0056293125704658135,
+          0.012714850523393719,
+          -0.002497639449055006,
+          -0.007135683201188999,
+          0.014988249771676699,
+          0.014955895630183352,
+          -0.022994948923833993,
+          0.007121634434082677,
+          0.005835971825639792,
+          -0.0030507120821557008,
+          0.002145266953387315,
+          -0.027334525318320587,
+          -0.020359474070022213,
+          0.01901995970350808,
+          -0.015260454474605018,
+          0.02850460517419317,
+          -0.011624138792780823,
+          -0.02284242513728605,
+          -0.01277489645734181,
+          -0.004209529633534193,
+          -0.022684500849916897,
+          -0.004894414098943742,
+          0.01919165657366256,
+          0.013401560972810082,
+          -0.004313549020265677,
+          -0.019089369171102138,
+          0.03863248408555474,
+          0.010497951840980463,
+          0.03508157919687289,
+          -0.008597970944278045,
+          0.0023354831464448313,
+          -0.01563401808449416,
+          0.015551176593436512,
+          0.001075882802133782,
+          -0.011575692834796734,
+          0.01505716108583101,
+          0.031836226915668404,
+          0.0034740889237207494,
+          -0.00617206855371193,
+          0.0018420207148235309,
+          0.00418433325800744,
+          -0.0008167766650900381,
+          0.011160989178055499,
+          -0.018497472318034093,
+          -0.011951262348399922,
+          0.0013093376768780122,
+          0.00216347903176653,
+          0.0059233460419514045,
+          -0.001778091863744448,
+          -0.013317639704511549,
+          0.011433308892643204,
+          0.011596810919469926,
+          -0.01518143105525228,
+          -0.007497373860338335,
+          0.0050957665991362405,
+          -0.018974405317819656,
+          0.01872472902494418,
+          0.011859522398789263,
+          0.000792413325332956,
+          -0.0006503947129906383,
+          0.00808635755624928,
+          0.003749813548995223,
+          0.0033657799335872368,
+          -0.007862215303617144,
+          -0.0028014669075116876,
+          0.0009892310856576799,
+          3.834379772674841e-06,
+          0.015402211364447726,
+          -0.015264893397161102,
+          0.02488855366696587,
+          -0.0025643639525691083,
+          0.00953993868921819,
+          -0.017460590918773188,
+          -0.0017911712931080155,
+          -0.00402374787529868,
+          0.01838051670627292,
+          0.006753072393532114,
+          -0.010130676216186492,
+          0.004921510943896829,
+          0.007975201081296845,
+          -0.012589786869771252,
+          0.00018319054705558588,
+          -0.0022606180909706688,
+          -0.014791528072476771,
+          -0.0031171076370692706,
+          0.017873825561816544,
+          0.007593688317551089,
+          0.0037375070591255375,
+          -0.015863143964928172,
+          0.007192324815779268,
+          0.007522990667369163,
+          0.024504875417877527,
+          -0.026309816223960274,
+          0.024779042752840836,
+          -0.022104087334088543,
+          0.02165781034613893,
+          0.0319051741666687,
+          0.007538146369110045,
+          -0.01690181186708103,
+          -0.006595947249169521,
+          0.006373991305923002,
+          -0.023410814842488123,
+          -0.005494017994983188,
+          0.016903736810723716,
+          -0.01792400152372258,
+          0.000330865061689199,
+          -0.014660413003063089,
+          0.0040825880154611895,
+          0.008248037865838212,
+          0.011540333451799686,
+          -0.0033578830043738085,
+          0.018327525495676525,
+          0.020171745352159914,
+          -0.01138759854371235,
+          -0.0004221149053107702,
+          -0.022321409200941777,
+          0.005046865873981027,
+          0.022509643273449936,
+          0.001111272240506322,
+          -0.009540467231190513,
+          0.018376261423745193,
+          0.008480532791922674,
+          -0.020801891594934313,
+          0.008436374227753589,
+          -0.036175982655998785,
+          -0.010454118523313122,
+          -0.009328507118928412,
+          -0.001218633166683384,
+          -0.004142773602181115,
+          -0.0002494371580772723,
+          -0.011365548842513265,
+          -7.699548642315061e-05,
+          -0.006889339420877534,
+          -0.0037173070609413474,
+          0.014953978004522174,
+          0.011678957032329096,
+          -0.003501047405385249,
+          -0.0073984932581048915,
+          0.008493128567102267,
+          0.006163937693564917,
+          -0.017034696006318337,
+          -0.013268418502042576,
+          0.0037122128689059085,
+          -0.020067656401141467,
+          0.005148329575092478,
+          0.00459729178556505,
+          0.014031669598917171,
+          -0.01849235806598133,
+          -0.009298127665365717,
+          0.019716512208448042,
+          0.0094466405158553,
+          -0.0008988916054855498,
+          0.004264260983102956,
+          -0.00032312548009733906,
+          0.002279987803645912,
+          -0.0008902345374346296,
+          -6.145528806973222e-06,
+          0.003602229465148649,
+          0.012503366799381277,
+          0.007759393855482671,
+          0.03694402922447051,
+          -0.016692148708641812,
+          0.005873765863350649,
+          -0.014187609702966002,
+          -0.0037592517335595654,
+          -0.00749939010396784,
+          -0.0004308157309481927,
+          -0.0022282234382577843,
+          0.0021999996319642854,
+          0.0052176773733353925,
+          0.009353171029508977,
+          0.002155405440765206,
+          -0.00823565584692709,
+          0.009691447250255694,
+          -0.015228173095533029,
+          0.003853858061067531,
+          -0.008207910291425473,
+          0.01017972550354623,
+          -0.009695280827515048,
+          0.0028705587799331246,
+          0.03849358164875786,
+          0.01130021300937487,
+          -0.004934175853966098,
+          -0.014300147534361459,
+          -0.0007796051618993811,
+          -0.01737877501558196,
+          -0.008579119859817895,
+          -0.005434792488602325,
+          -0.02617338653902575,
+          -0.00838916510137031,
+          0.015344260464413648,
+          0.011202567142310492,
+          0.014124853863170132,
+          -0.023894182185855824,
+          0.008897943489322673,
+          -0.03001670823127986,
+          -0.011951507895140876,
+          -0.001530967140156163,
+          -0.015467812535428609,
+          0.0010110999991167703,
+          0.023623196504289838,
+          -0.007762735863648054,
+          0.012654652822750325,
+          0.012867958170258448,
+          0.018777522196571934,
+          0.006390986057157016,
+          0.00038045736974183937,
+          -0.009444911783760569,
+          0.009534310165417939,
+          0.008806031651319704,
+          0.0080954543713255,
+          0.013161833638467553,
+          -0.0022840529693446553,
+          -0.018060162439023572,
+          0.005469288590000856,
+          0.014354475778235131,
+          -0.0033504276914600585,
+          -0.001615191819057537,
+          -0.011088493180661976,
+          0.016039040565630432,
+          -0.023559046333872156,
+          -0.009482398794557168,
+          0.009918576960319548,
+          -0.0027308309155029842,
+          0.009212063547988036,
+          0.0160181200081126,
+          -0.01881373001297218,
+          -0.0024168618343039703,
+          0.016244889977796682,
+          0.0024104546308670025,
+          -0.00022606425578303413,
+          0.016792244238218245,
+          -0.0007616857360728623,
+          -0.004279120886828465,
+          0.00997764266310123,
+          -0.0075631917138318655,
+          7.3878137053128405e-06,
+          -0.002416364915008295,
+          -0.0028253498168087964,
+          0.02054128727706496,
+          -0.021281801678871736,
+          -0.005379259148621041,
+          0.02257306103533458,
+          0.015687105622685975,
+          -0.007843745111084413,
+          6.258498042855072e-05,
+          -0.0219506172079994,
+          0.0019925886885244454,
+          -0.007768133999661569,
+          0.016524850592507592,
+          -0.007957949830060354,
+          0.0030085044673901665,
+          -0.01361024620290394,
+          0.011563781350927133,
+          -0.01277407169098418,
+          0.00873716215529134,
+          -0.002986943948333218,
+          -0.017567646684567492,
+          -0.0053109390854215125,
+          0.004561236136619266,
+          0.0014045175130801308,
+          0.044958041347040185,
+          -0.005539136232703934,
+          0.003729517522976541,
+          0.018086335927383804,
+          -0.014199453349727933,
+          0.02079719360289355,
+          0.016663293302592568,
+          0.00991445855219406,
+          -0.00033967370741635794,
+          -0.0018726748849858836,
+          0.023373354806639515,
+          0.013258226441642464,
+          0.00861986998150065,
+          -0.0014340218811281975,
+          -0.008516541227326329,
+          0.018239230461345617,
+          0.005458629943667832,
+          -0.012421896028975013,
+          0.012732721349758832,
+          0.013559877132508506,
+          -0.013720939472355347,
+          -0.011545210834395917,
+          0.01616358223872008,
+          -0.019904594319155373,
+          -0.018260016571763644,
+          -0.0036124537591368864,
+          -0.01614510591132948,
+          -0.03864699384205461,
+          0.01619401261888991,
+          0.018351607973242735,
+          0.006666313000123845,
+          0.013647410298896132,
+          0.002597680061427196,
+          0.004545530267244157,
+          -0.018531082430861854,
+          -0.0077618625909995275,
+          -0.008085144524135461,
+          0.003744735923004248,
+          0.011059457364156674,
+          -0.011335432904114519,
+          -0.009514379703581616,
+          -0.00034500702695418285,
+          -0.007433132060989644,
+          0.01971420270994951,
+          0.0020753610290475642,
+          0.0034506706065161024,
+          -0.00628437298309269,
+          0.017966485854173167,
+          0.026789678964152908,
+          -0.007532446825077476,
+          0.021886182869092073,
+          -0.002864230253056434,
+          -0.0015870360120997492,
+          -0.014393519359298496,
+          -0.0014439412842964876,
+          -0.011527182213472351,
+          0.0005453493437517964,
+          -0.0006791481533636673,
+          0.014093767997869645,
+          -0.03288824642147967,
+          -0.003957301296796813,
+          0.014444592778851293,
+          0.007538052312876599,
+          -0.005057755472988024,
+          0.011357919882949903,
+          -0.002964451028327082,
+          0.016977317604413924,
+          0.0075201055190542146,
+          0.023391646079994784,
+          -0.016588823248397513,
+          -0.004504858318770565,
+          0.006660877064954993,
+          0.0034631255931793638,
+          0.0014484203909961972,
+          0.01912983041187014,
+          0.023439941258581887,
+          -0.014715886385460541,
+          -0.0013596123366001458,
+          -0.0029020372473879614,
+          -0.021414221343490594,
+          0.014976806123253727,
+          0.0074729481304182865,
+          0.010967351226114886,
+          0.009806663105020302,
+          0.015465076881663691,
+          -0.02109587527354609,
+          0.0011671121739822202,
+          0.020567458365051382,
+          -0.008618333549105003,
+          -0.018815706906171112,
+          -0.009804829158883123,
+          -0.00966828714719191,
+          -0.006078893678150302,
+          -0.00803783313451146,
+          0.005523023167028105,
+          0.0022647506733877573,
+          0.01652997604669998,
+          -0.008204200434441975,
+          -0.008404609147866811,
+          0.008489247125986078,
+          0.007546785673285454,
+          -0.0033093451283570316,
+          0.010614050963656496,
+          0.0030633179859393587,
+          -0.006940540318011839,
+          0.00894884343338777,
+          -0.002787816848430143,
+          0.014347598920245487,
+          -0.02441769157218451,
+          0.004717742147185283,
+          -0.012096876825337124,
+          0.006405151160675699,
+          0.0041568790527989905,
+          0.0016702066682613428,
+          -0.0041159211459004,
+          0.0035217362856754353,
+          0.0008893479210743541,
+          -0.07433312151926533,
+          -0.012629681414004524,
+          -0.0035797012285864486,
+          0.033657838713146884,
+          -0.019695806809053914,
+          0.004477927056336216,
+          0.011966065644382396,
+          0.0096717896523898,
+          0.003070713232538593,
+          0.006136726101157844,
+          0.006112385569942848,
+          -0.0245258242481677,
+          -0.00015284418977092615,
+          0.004821040694679944,
+          0.0020347004045943454,
+          -0.007532408630122127,
+          0.00750565079228405,
+          0.015327710245879296,
+          0.020989926895388523,
+          -0.010970305087289488,
+          0.017227663358885748,
+          -0.003352718496981928,
+          -0.006068892126250978,
+          0.0014285750322333686,
+          -0.010153799722483554,
+          -0.017307415843726394,
+          -0.006495008086641125,
+          0.0068983699213297075,
+          -0.001982568363667046,
+          0.0044032687939897855,
+          0.003333596495330213,
+          -0.0032171586993948696,
+          0.0072365127663633925,
+          0.013920326463272646,
+          0.006415916101287133,
+          -0.013675562796300214,
+          0.003003782758807665,
+          -0.02500375035312058,
+          0.011945988879567646,
+          -0.012708975181025296,
+          -0.0009488161508767279,
+          -0.0047093877027523014,
+          -0.0035750724825159503,
+          -0.002213435677949806,
+          0.02043567538125188,
+          -0.014304009033474186,
+          -0.01802899775650995,
+          0.008187227768196673,
+          -0.011217013195839762,
+          0.008300740149622635,
+          -0.03382831122769974,
+          0.012386335491333264,
+          -0.0276029934382802,
+          -0.0002243752790856292,
+          -0.004521387639042107,
+          -0.0010631721066129095,
+          -0.009491184157278669,
+          -0.03274559774619223,
+          0.015845705808609598,
+          -0.007204751554079551,
+          -0.020338337341085228,
+          0.0007143424225187802,
+          -0.021267094083674304,
+          0.014150458196361147,
+          -0.01851165553224022,
+          0.0033450353123933058,
+          -0.018775439467431685,
+          0.002782289821140401,
+          -0.0018711350216089556,
+          0.010502195116579441,
+          -0.00557814442809019,
+          -0.0007130962242755068,
+          -0.013072988887382269,
+          -0.002121841212381393,
+          -0.006653444103331213,
+          0.009394907690516121,
+          0.0028678074167728367,
+          -0.009601577734902932,
+          0.011153571864462314,
+          0.013046116892519414,
+          0.019812860521973938,
+          -0.00539047787915588,
+          -0.00555139958411929,
+          -0.011553689090179406,
+          0.009852063270798534,
+          -0.016326212706634428,
+          -0.00827402913960903,
+          0.008982303148984887,
+          0.015216947130173639,
+          0.01026889105233137,
+          0.003352028462252994,
+          0.003150801750574548,
+          0.002531620731961765,
+          0.026744691368233237,
+          -0.0015120532859309638,
+          0.0008324701960729416,
+          -0.014508988496009931,
+          0.022127525293249926,
+          0.02772430884375744,
+          -0.0012206102581726523,
+          -0.00018859639594030037,
+          0.017437360169132967,
+          -0.004697052522250805,
+          -0.0038111184371524645,
+          0.024015246959315593,
+          -0.038396108183338655,
+          -0.008416825232533771,
+          -0.006788816882617399,
+          -0.006043428198944214,
+          -0.009587366247924467,
+          0.007294190273465812,
+          0.00041966080769004275,
+          0.0024709692753547823,
+          -0.0005250904505954556,
+          0.03452858797518386,
+          0.003403515654095309,
+          -0.0003392060101496309,
+          -0.0032980864446307116,
+          0.0019125121221204029,
+          0.0032987517853214346,
+          -0.009116761755779728,
+          -0.038030085122772624,
+          0.006006334000396066,
+          0.007932785488877407,
+          0.00975009749364395,
+          -0.008120121513849434,
+          -0.008864461019882246,
+          0.004909348738798099,
+          -0.02273129283865203,
+          -0.01372636307167243,
+          0.02625229144362647,
+          -0.008523028986127237,
+          0.029005877324974557,
+          -0.022435059061182174,
+          -0.013135836801818153,
+          -0.004126433322528265,
+          0.010283975013795537,
+          -0.019732856462444506,
+          0.004343860338939202,
+          -0.004022387065239562,
+          0.023733721026005954,
+          0.0077578144626961245,
+          -0.008371631446146718,
+          0.012386735614153616,
+          0.0064789380436815714,
+          -0.0026638642827013837,
+          0.00136333052963848,
+          -0.007209517067748103,
+          0.016916505654836485,
+          -0.011819784026452944,
+          -0.005833909403097991,
+          0.008977527508585844,
+          -0.0024556434204852946,
+          0.02509293948918158,
+          -0.0237762245071987,
+          0.005288171058938098,
+          -0.00019047961064753207,
+          0.01822542471604037,
+          0.008127347078975925,
+          0.008361588667386882,
+          -0.011835407590060994,
+          0.010004482616697591,
+          -0.03378107067882622,
+          -0.015433906677849963,
+          -0.007702910250795191,
+          -0.0013877702264578865,
+          -0.002059152251387601,
+          -0.015745218548968338,
+          -0.026393221735795527,
+          0.006294583740809379,
+          0.016267525575472445,
+          0.023291528020980265,
+          -0.004683197930892445,
+          0.012912830736650997,
+          -0.002831583991769044,
+          -0.008904271394710672,
+          -0.011441882849296418,
+          -0.012202629605208118,
+          -0.012470773094383122,
+          -0.014560694782340029,
+          -0.017045701603711442,
+          0.012139665577738337,
+          -0.020079061996873537,
+          0.013911518328924777,
+          0.0347210775522539,
+          -0.017176114991595492,
+          -0.0018941772168839354,
+          0.019702722555622765,
+          -0.004221469612316137,
+          0.01212088353057276,
+          0.0063707489618353725,
+          0.015298014532351866,
+          0.0038643174573634725,
+          -0.012181479717826009,
+          -0.007374005740809251,
+          0.019411457664902376,
+          -0.004902591844710158,
+          0.014591446725717338,
+          0.015069599884557399,
+          -0.008675672164385891,
+          0.002544198867387345,
+          0.01951865904496276,
+          -0.01762101777798757,
+          0.01665654707715257,
+          -0.00458402229480694,
+          -0.009936206685538294,
+          -0.013345740581684638,
+          -0.005336416724426228,
+          -0.019469457091169776,
+          0.0061608454109892425,
+          0.018931676738452464,
+          -0.00533013594306834,
+          0.009310871143727683,
+          -0.0037670793158045,
+          -0.005045151384034014,
+          0.0002041909886640482,
+          -0.04152751596227068,
+          -0.008340911336217609,
+          -0.0006651584370005978,
+          -0.02505748815737022,
+          0.025954102004571416,
+          0.007205602377144382,
+          0.022602344303400163,
+          -0.017168610245491406,
+          0.004459735690564842,
+          -0.005882024209102714,
+          0.013217517324383834,
+          0.02255689669623969,
+          0.0017984057469785134,
+          0.005699128497215882,
+          -0.02353849721739205,
+          -0.0034711576718386185,
+          0.015924913374553738,
+          -0.003604478141616307,
+          -0.0034993903155063817,
+          0.012592308877056896,
+          0.012864938348003884,
+          0.01567418642605425,
+          0.008801753550506258,
+          0.015606443428689292,
+          0.010589933994371628,
+          0.008928721556432368,
+          -0.005805129999077126,
+          -0.001025103064751441,
+          -0.015907755521365925,
+          0.0045874330999415985,
+          0.031701049450012876,
+          0.012594483199773746,
+          0.016783858873835704,
+          -0.01778536268185936,
+          0.00789523134933153,
+          -0.0012706411671644011,
+          0.008325108613530198,
+          -0.001677173669891956,
+          0.029540136288412976,
+          0.011739999605699826,
+          0.015670966746894652,
+          0.008732334090105053,
+          -0.016237731209426752,
+          0.045334690225148,
+          -0.016911928942220152,
+          -0.0021139311364009113,
+          -0.01058970056572469,
+          -0.021705338531028618,
+          0.007324792557651635,
+          -0.006556582104686972,
+          0.0018243188317438144,
+          0.010780292738415671,
+          -0.006471850255151019,
+          0.0006997395548596146,
+          0.003907433192252573,
+          -0.023129313269763224,
+          -0.002550875642832235,
+          -0.02237917712764273,
+          -0.003107775639788617,
+          -0.02621608419628401,
+          -0.00012482032027919036,
+          -0.016207377503696517,
+          -0.0038712739995960753,
+          -0.023315508569288476,
+          -0.014963211090562272,
+          -0.01633110043829169,
+          -0.004241081909202262,
+          -0.008295402942465353,
+          0.004976396647108597,
+          -0.007402107755970892,
+          -0.020321297548927468,
+          0.0020986169430507757,
+          0.016803273183070004,
+          0.01434729320047051,
+          0.018612970083178984,
+          0.00603868809307069,
+          0.014278001198851581,
+          -0.0026350854954985117,
+          0.017171016209107656,
+          -0.006968968261909459,
+          0.006881847166565678,
+          0.011454001418121107,
+          0.02140935294963631,
+          0.011121833270235448,
+          -0.012801756637470525,
+          0.0010870041670931706,
+          -0.009655388034828094,
+          0.002094601818569396,
+          -0.00828486034340067,
+          -0.009190975158350585,
+          0.0017762581946855365,
+          -0.007261996868906865,
+          0.004674405202069334,
+          0.0030526253833799426,
+          -0.0014082232541965316,
+          0.010419302200840906,
+          -0.0024200053377751805,
+          -0.012208685875688886,
+          -0.0033037330470612695,
+          -0.017975940046938498,
+          -0.006958042903103732,
+          -0.0014496522238488239,
+          -0.011711502944602638,
+          -0.01016130418405875,
+          0.003321436189059608,
+          -0.00538282506552848,
+          -0.009353581612407435,
+          0.013884147453136417,
+          0.0001309738696853529,
+          0.0005878330075918333,
+          0.023495280602974504,
+          0.006035028203642483,
+          0.008087594627921345,
+          0.013144288945385904,
+          -0.01023995000994454,
+          -0.022574958230635715,
+          -0.0243435727055662,
+          0.007867022215096418,
+          -0.01573478573423379,
+          0.02324699908478915,
+          -0.007929809922043611,
+          -0.00561630876431978,
+          0.0020982723335057625,
+          -0.013699317926847062,
+          0.008578431496569186,
+          -0.011271381061801356,
+          0.0014450645788554738,
+          -0.019581451799757368,
+          0.018059311800267413,
+          -0.010917911869373693,
+          -0.004302687883018328,
+          0.009466103226841532,
+          0.034049086327052606,
+          -0.0005689605174080737,
+          -0.007410970644432269,
+          -0.006594150877740108,
+          -0.00022810271313786974,
+          0.018766765773409123,
+          -0.0029041815756733333,
+          0.005758781908059541,
+          -0.01563519107805783,
+          0.006571628287171105,
+          -0.006132372252825422,
+          0.0012423261344297225,
+          0.01513074831647933,
+          -0.026306320707079913,
+          -0.015491710256868008,
+          -0.007857352164317122,
+          -0.024893430640551818,
+          0.002253574511878983,
+          0.017579477884206534,
+          0.020505606775521504,
+          -0.020324505029926956,
+          -0.02513332780748072,
+          0.005150966733930881,
+          -0.026580994694344065,
+          0.0037390577082486594,
+          0.1241031878517593,
+          -0.002138404898309707,
+          0.002720723037908521,
+          -0.011425036886011179,
+          0.0022399461009316216,
+          0.005389407121650262,
+          -0.019552543955456567,
+          0.05950746473186985,
+          -0.00638023587296763,
+          -0.004023798021748334,
+          0.002170545188776296,
+          0.001785855390676113,
+          -0.005831534307818096,
+          0.008643501093778434,
+          -0.017602787281866816,
+          -0.02184792441891879,
+          -0.0052215705748838905,
+          -0.021252169339074325,
+          -0.008509621422071373,
+          -0.013000137524907565,
+          0.002893880203681282,
+          -0.015821609145923173,
+          0.010397692426614709,
+          -0.01638916243796692,
+          -0.010283706937701179,
+          -0.01103621454761223,
+          -0.00373552043529251,
+          0.01563553822905027,
+          -0.03188533919653949,
+          -0.004873550301425776,
+          -0.005183590054152726,
+          0.018218971950313273,
+          0.006004328765748325,
+          0.004357734370247252,
+          -0.014230318971458851,
+          0.0064792070801456235,
+          0.0215589173714926,
+          0.0055272493764401305,
+          0.014643335833970546,
+          0.010618070641357003,
+          -0.0022177303813575846,
+          -0.0024402761885758007,
+          0.017541799738501007,
+          -0.005445774414228888,
+          -0.00023502936119626386,
+          -0.00537569046685066,
+          -0.005974703247446444,
+          -0.010712244203425036,
+          0.01843613166635215,
+          -0.009938564643456672,
+          -0.017388592763277214,
+          -0.0015797412875366156,
+          0.02028434083164951,
+          -0.012761534382003973,
+          0.0031906812114950982,
+          0.000788436210275669,
+          0.020589196433948156,
+          0.0007726855314648301,
+          -0.005481260996289263,
+          0.007165014826294827,
+          -0.02956334397512344,
+          0.004095715232997753,
+          0.0003568512047626187,
+          0.027290768799658912,
+          -0.0017631129386615335,
+          0.01552542586826711,
+          -0.020871286597113327,
+          0.004388125824324094,
+          -0.016926008631435403,
+          0.005099899758322398,
+          0.012560923968841436,
+          0.002781049890810361,
+          0.010889021753593997,
+          -0.008500358897706648,
+          -0.021353460542554444,
+          0.012663934009161902,
+          -0.018657094130860155,
+          -0.029156017587920958,
+          0.0018889388761887053,
+          -0.028158076315864772,
+          -0.003048854609750186,
+          0.006812072971079798,
+          -0.035217412591558014,
+          -0.02251675180883659,
+          0.007396524512996277,
+          -0.00995571600201846,
+          -0.011912753666073882,
+          0.010675240017836214,
+          -0.02645155743041798,
+          -0.018137648527851668,
+          0.0010666944556449141,
+          0.012588800077165337,
+          -0.004561825939541096,
+          0.007555558090958502,
+          -0.028558603070876605,
+          0.014816816621445352,
+          -0.01313764261744744,
+          0.016036065063831813,
+          -0.0037797584923144134,
+          0.022217516257304868,
+          0.011131597210722573,
+          0.008795376259031849,
+          -0.004950080558710245,
+          0.021409590863550987,
+          0.005574351961953293,
+          0.051358937509801736,
+          0.031972009383746405,
+          0.0019821003238211276,
+          -0.003251257665378234,
+          0.003313045083311649,
+          0.008294776423730887,
+          -0.00940948257229233,
+          0.005114631360856872,
+          0.028053227377164806,
+          0.016738013668710083,
+          -0.018755924318109216,
+          0.006606711923035118,
+          -0.02702692774986498,
+          0.017950884766781056,
+          -0.022054653845163,
+          0.0003755944703013671,
+          0.007766253634286795,
+          0.0016206993755471613,
+          0.010682347575070117,
+          0.0013842930079185415,
+          0.01705234255261425,
+          0.023975191051210525,
+          -0.002976402980885077,
+          0.017466027107477605,
+          0.012734909275073793,
+          0.018143161773162015,
+          -0.0038399373197265827,
+          0.006929440921265761,
+          -0.012003095293274786,
+          -0.0008883470597690258,
+          0.0020490527756307267,
+          -0.004231091383919565,
+          0.012128848448049833,
+          0.008340302519160299,
+          0.012989849936937843,
+          0.002974488532907779,
+          0.003198854381461613,
+          -0.005309753760578859,
+          -0.0017024982049936384,
+          -0.01763395035509857,
+          -0.01819680226734474,
+          0.028417724274372468,
+          -0.020217879775485576,
+          -0.050118184195774004,
+          0.0025509548991664425,
+          0.014185761006969567,
+          -0.0012021669304388173,
+          -0.0051260576846881515,
+          -0.0005774778872927327,
+          0.005718864494530703,
+          -0.016918749210038116,
+          -0.0015472638835434902,
+          0.008125845907627804,
+          0.020347547062899797,
+          -0.002036218493206914,
+          -0.01079739518866248,
+          0.0038112958713600906,
+          0.02169339303105242,
+          0.01667594322285835,
+          0.012338193092567552,
+          -0.009798409233733167,
+          0.0003198215558702285,
+          -1.6308740825163337e-05,
+          0.0076870827874737325,
+          0.004356644356656038,
+          0.003500023005601613,
+          -0.019968549869439228,
+          -0.008634740687755112,
+          0.012781565963545847,
+          -0.0025637758121512927,
+          0.010243567793632335,
+          -0.00042296044205132426,
+          -0.003069251553118065,
+          -0.009344199449486244,
+          -0.004694953215957636,
+          -0.009583993744231791,
+          0.013783878036521561,
+          -0.007705982042252044,
+          -0.01549567538665377,
+          0.00020666298495823114,
+          0.0005515775451348472,
+          0.001667367278489879,
+          -0.004518431526236256,
+          0.03302324454129141,
+          0.009531990824506103,
+          -0.0004937348108340977,
+          -0.004047613286672229,
+          -0.0327383112522095,
+          0.012456150771224677,
+          -0.009677533214778067,
+          0.0006455811519981003,
+          0.020012976827426605,
+          -0.00020277620481825933,
+          0.002168432791818704,
+          -0.015657551629104767,
+          0.0032924357559238502,
+          0.006161295735841883,
+          0.013814165416788555,
+          0.0005164426482972658,
+          0.01076512534105537,
+          -0.006498977875968572,
+          -0.010325144017898257,
+          0.004804158067334203,
+          0.0104064855510473,
+          -0.0012188100892986933,
+          0.005171197790296134
+        ],
+        "title": "Mixture-Of-Experts With Expert Choice Routing",
+        "keyphrases": [
+          "Mixture-Of-Experts",
+          "Expert Choice Routing",
+          "sparsely-activated models",
+          "training efficiency",
+          "downstream performance"
+        ]
+      },
+      "type": "document"
+    },
+    {
+      "id": "14339b1f-53f8-479b-9da7-9430b174b9ee",
+      "properties": {
+        "page_content": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer\n\nNoam Shazeer1, Azalia Mirhoseini∗†1, Krzysztof Maziarz∗2, Andy Davis1, Quoc Le1, Geoffrey Hinton1and Jeff Dean1 1Google Brain, {noam,azalia,andydavis,qvl,geoffhinton,jeff}@google.com 2Jagiellonian University, Cracow, krzysztof.maziarz@student.uj.edu.pl\n\nAbstract\n\nThe capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost.\n\n1 Introduction And Related Work 1.1 Conditional Computation\n\nExploiting scale in both training data and model size has been central to the success of deep learning. When datasets are sufficiently large, increasing the capacity (number of parameters) of neural networks can give much better prediction accuracy. This has been shown in domains such as text (Sutskever et al., 2014; Bahdanau et al., 2014; Jozefowicz et al., 2016; Wu et al., 2016), images (Krizhevsky et al., 2012; Le et al., 2012), and audio (Hinton et al., 2012; Amodei et al., 2015). For typical deep learning models, where the entire model is activated for every example, this leads to a roughly quadratic blow-up in training costs, as both the model size and the number of training examples increase. Unfortunately, the advances in computing power and distributed computation fall short of meeting such demand.\n\nVarious forms of conditional computation have been proposed as a way to increase model capacity without a proportional increase in computational costs (Davis & Arel, 2013; Bengio et al., 2013; Eigen et al., 2013; Ludovic Denoyer, 2014; Cho & Bengio, 2014; Bengio et al., 2015; Almahairi et al., 2015). In these schemes, large parts of a network are active or inactive on a per-example basis. The gating decisions may be binary or sparse and continuous, stochastic or deterministic. Various forms of reinforcement learning and back-propagation are proposed for trarining the gating decisions.\n\nWhile these ideas are promising in theory, no work to date has yet demonstrated massive improvements in model capacity, training time, or model quality. We blame this on a combination of the following challenges:\n\nModern computing devices, especially GPUs, are much faster at arithmetic than at branching. Most of the works above recognize this and propose turning on/off large chunks of the network with each gating decision.\n\nLarge batch sizes are critical for performance, as they amortize the costs of parameter transfers and updates. Conditional computation reduces the batch sizes for the conditionally active chunks of the network.\n\nNetwork bandwidth can be a bottleneck. A cluster of GPUs may have computational power thousands of times greater than the aggregate inter-device network bandwidth. To be computationally efficient, the relative computational versus network demands of an algorithm must exceed this ratio. Embedding layers, which can be seen as a form of conditional computation, are handicapped by this very problem. Since the embeddings generally need to be sent across the network, the number of (example, parameter) interactions is limited by network bandwidth instead of computational capacity.\n\nDepending on the scheme, loss terms may be necessary to achieve the desired level of sparsity per-chunk and/or per example. Bengio et al. (2015) use three such terms. These issues can affect both model quality and load-balancing.\n\nModel capacity is most critical for very large data sets. The existing literature on conditional computation deals with relatively small image recognition data sets consisting of up to 600,000 images. It is hard to imagine that the labels of these images provide a sufficient signal to adequately train a model with millions, let alone billions of parameters. In this work, we for the first time address all of the above challenges and finally realize the promise of conditional computation. We obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets.\n\n1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1). All parts of the network are trained jointly by back-propagation.\n\nWhile the introduced technique is generic, in this paper we focus on language modeling and machine translation tasks, which are known to benefit from very large models. In particular, we apply a MoE convolutionally between stacked LSTM layers (Hochreiter & Schmidhuber, 1997), as in Figure 1.\n\nThe MoE is called once for each position in the text, selecting a potentially different combination of experts at each position. The different experts tend to become highly specialized based on syntax and semantics (see Appendix E Table 9). On both language modeling and machine translation benchmarks, we improve on best published results at a fraction of the computational cost.\n\n1.3 Related Work On Mixtures Of Experts\n\nSince its introduction more than two decades ago (Jacobs et al., 1991; Jordan & Jacobs, 1994), the mixture-of-experts approach has been the subject of much research. Different types of expert architectures hae been proposed such as SVMs (Collobert et al., 2002), Gaussian Processes (Tresp, 2001; Theis & Bethge, 2015; Deisenroth & Ng, 2015), Dirichlet Processes (Shahbaba & Neal, 2009), and deep networks. Other work has focused on different expert configurations such as a hierarchical structure (Yao et al., 2009), infinite numbers of experts (Rasmussen & Ghahramani, 2002), and adding experts sequentially (Aljundi et al., 2016). Garmash & Monz (2016) suggest an ensemble model in the format of mixture of experts for machine translation. The gating network is trained on a pre-trained ensemble NMT model. The works above concern top-level mixtures of experts. The mixture of experts is the whole model. Eigen et al. (2013) introduce the idea of using multiple MoEs with their own gating networks as parts of a deep model. It is intuitive that the latter approach is more powerful, since complex problems may contain many sub-problems each requiring different experts. They also allude in their conclusion to the potential to introduce sparsity, turning MoEs into a vehicle for computational computation.\n\nOur work builds on this use of MoEs as a general purpose neural network component. While Eigen et al. (2013) uses two stacked MoEs allowing for two sets of gating decisions, our convolutional application of the MoE allows for different gating decisions at each position in the text. We also realize sparse gating and demonstrate its use as a practical way to massively increase model capacity.\n\n2 The Structure Of The Mixture-Of-Experts Layer\n\nThe Mixture-of-Experts (MoE) layer consists of a set of n \"expert networks\" E1, · · · , En, and a \"gating network\" G whose output is a sparse n-dimensional vector. Figure 1 shows an overview of the MoE module. The experts are themselves neural networks, each with their own parameters.\n\nAlthough in principle we only require that the experts accept the same sized inputs and produce the same-sized outputs, in our initial investigations in this paper, we restrict ourselves to the case where the models are feed-forward networks with identical architectures, but with separate parameters.\n\nLet us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows:\n\n$$y=\\sum_{i=1}^{n}G(x){i}E{i}(x)$$ $$(\\mathbf{l})$$ G(x)iEi(x) (1) We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example. If the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network. In the following we focus on ordinary MoEs. We provide more details on hierarchical MoEs in Appendix B. Our implementation is related to other models of conditional computation. A MoE whose experts are simple weight matrices is similar to the parameterized weight matrix proposed in (Cho & Bengio, 2014). A MoE whose experts have one hidden layer is similar to the block-wise dropout described in (Bengio et al., 2015), where the dropped-out layer is sandwiched between fully-activated layers.\n\n2.1 Gating Network\n\nSoftmax Gating: A simple choice of non-sparse gating function (Jordan & Jacobs, 1994) is to multiply the input by a trainable weight matrix Wg and then apply the Sof tmax function.\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ $\\eqref{eq:walpha}$. Gσ(x) = Sof tmax(x · Wg) (2) Noisy Top-K Gating: We add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to −∞ (which causes the corresponding gate values to equal 0). The sparsity serves to save computation, as described above. While this form of sparsity creates some theoretically scary discontinuities in the output of gating function, we have not yet observed this to be a problem in practice. The noise term helps with load balancing, as will be discussed in Appendix A. The amount of noise per component is controlled by a second trainable weight matrix Wnoise.\n\n$$G(x)=S o f t m a x(K e e p T o p K(H(x),k))$$ $$(4)$$ $$(S)$$ G(x) = Sof tmax(KeepT opK(H(x), k)) (3) $$H(x){i}=(x\\cdot W{g}){i}+S t a n d a r d N o r m a l()\\cdot S o f t p l u s((x\\cdot W{n o i s e}){i})$$ $KeepTopK(v,k){i}=\\begin{cases}v_{i}&\\text{if}v_{i}\\text{is in the top}k\\text{elements of}v.\\ -\\infty&\\text{otherwise.}\\end{cases}$ Training the Gating Network We train the gating network by simple back-propagation, along with the rest of the model. If we choose k > 1, the gate values for the top k experts have nonzero derivatives with respect to the weights of the gating network. This type of occasionally-sensitive behavior is described in (Bengio et al., 2013) with respect to noisy rectifiers. Gradients also backpropagate through the gating network to its inputs. Our method differs here from (Bengio et al., 2015) who use boolean gates and a REINFORCE-style approach to train the gating network.\n\n3 Addressing Performance Challenges 3.1 The Shrinking Batch Problem\n\nOn modern CPUs and GPUs, large batch sizes are necessary for computational efficiency, so as to amortize the overhead of parameter loads and updates. If the gating network chooses k out of n experts for each example, then for a batch of b examples, each expert receives a much smaller batch of approximately kb n b examples. This causes a naive MoE implementation to become very inefficient as the number of experts increases. The solution to this shrinking batch problem is to make the original batch size as large as possible. However, batch size tends to be limited by the memory necessary to store activations between the forwards and backwards passes. We propose the following techniques for increasing the batch size: Mixing Data Parallelism and Model Parallelism: In a conventional distributed training setting, multiple copies of the model on different devices asynchronously process distinct batches of data, and parameters are synchronized through a set of parameter servers. In our technique, these different batches run synchronously so that they can be combined for the MoE layer. We distribute the standard layers of the model and the gating network according to conventional data-parallel schemes, but keep only one shared copy of each expert. Each expert in the MoE layer receives a combined batch consisting of the relevant examples from all of the data-parallel input batches. The same set of devices function as data-parallel replicas (for the standard layers and the gating networks) and as model-parallel shards (each hosting a subset of the experts). If the model is distributed over d devices, and each device processes a batch of size b, each expert receives a batch of approximately kbd nexamples. Thus, we achieve a factor of d improvement in expert batch size.\n\nIn the case of a hierarchical MoE (Section B), the primary gating network employs data parallelism, and the secondary MoEs employ model parallelism. Each secondary MoE resides on one device.\n\nThis technique allows us to increase the number of experts (and hence the number of parameters) by proportionally increasing the number of devices in the training cluster. The total batch size increases, keeping the batch size per expert constant. The memory and bandwidth requirements per device also remain constant, as do the step times, as does the amount of time necessary to process a number of training examples equal to the number of parameters in the model. It is our goal to train a trillionparameter model on a trillion-word corpus. We have not scaled our systems this far as of the writing of this paper, but it should be possible by adding more hardware.\n\nTaking Advantage of Convolutionality: In our language models, we apply the same MoE to each time step of the previous layer. If we wait for the previous layer to finish, we can apply the MoE to all the time steps together as one big batch. Doing so increases the size of the input batch to the MoE layer by a factor of the number of unrolled time steps.\n\nIncreasing Batch Size for a Recurrent MoE: We suspect that even more powerful models may involve applying a MoE recurrently. For example, the weight matrices of a LSTM or other RNN could be replaced by a MoE. Sadly, such models break the convolutional trick from the last paragraph, since the input to the MoE at one timestep depends on the output of the MoE at the previous timestep. Gruslys et al. (2016) describe a technique for drastically reducing the number of stored activations in an unrolled RNN, at the cost of recomputing forward activations. This would allow for a large increase in batch size.\n\n3.2 Network Bandwidth\n\nAnother major performance concern in distributed computing is network bandwidth. Since the experts are stationary (see above) and the number of gating parameters is small, most of the communication involves sending the inputs and outputs of the experts across the network. To maintain computational efficiency, the ratio of an expert's computation to the size of its input and output must exceed the ratio of computational to network capacity of the computing device. For GPUs, this may be thousands to one. In our experiments, we use experts with one hidden layer containing thousands of RELU-activated units. Since the weight matrices in the expert have sizes input_size×hidden_size and hidden_size × output_size, the ratio of computation to input and output is equal to the size of the hidden layer. Conveniently, we can increase computational efficiency simply by using a larger hidden layer, or more hidden layers.\n\n4 Balancing Expert Utilization\n\nWe have observed that the gating network tends to converge to a state where it always produces large weights for the same few experts. This imbalance is self-reinforcing, as the favored experts are trained more rapidly and thus are selected even more by the gating network. Eigen et al. (2013) describe the same phenomenon, and use a hard constraint at the beginning of training to avoid this local minimum. Bengio et al. (2015) include a soft constraint on the batch-wise average of each gate.1 We take a soft constraint approach. We define the importance of an expert relative to a batch of training examples to be the batchwise sum of the gate values for that expert. We define an additional loss Limportance, which is added to the overall loss function for the model. This loss is equal to the square of the coefficient of variation of the set of importance values, multiplied by a hand-tuned scaling factor wimportance. This additional loss encourages all experts to have equal importance.\n\n$$I m p o r t a n c e(X)=\\sum_{x\\in X}G(x)$$ $$L_{importance}(X)=w_{importance}\\cdot CV(Importance(X))^{2}\\tag{7}$$ $$(6)$$\n\nWhile this loss function can ensure equal importance, experts may still receive very different numbers of examples. For example, one expert may receive a few examples with large weights, and another may receive many examples with small weights. This can cause memory and performance problems on distributed hardware. To solve this problem, we introduce a second loss function, Lload , which ensures balanced loads. Appendix A contains the definition of this function, along with experimental results.\n\n5 Experiments 5.1 1 Billion Word Language Modeling Benchmark\n\nDataset: This dataset, introduced by (Chelba et al., 2013) consists of shuffled unique sentences from news articles, totaling approximately 829 million words, with a vocabulary of 793,471 words.\n\nPrevious State-of-the-Art: The best previously published results (Jozefowicz et al., 2016) use models consisting of one or more stacked Long Short-Term Memory (LSTM) layers (Hochreiter & Schmidhuber, 1997; Gers et al., 2000). The number of parameters in the LSTM layers of these models vary from 2 million to 151 million. Quality increases greatly with parameter count, as do computational costs. Results for these models form the top line of Figure 2-right. MoE Models: Our models consist of two stacked LSTM layers with a MoE layer between them (see Figure 1). We vary the sizes of the layers and the number of experts. For full details on model architecture, training regimen, additional baselines and results, see Appendix C.\n\nLow Computation, Varied Capacity: To investigate the effects of adding capacity, we trained\n\na series of MoE models all with roughly equal computational costs: about 8 million multiply-andadds per training example per timestep in the forwards pass, excluding the softmax layer. We call this metric (ops/timestep). We trained models with flat MoEs containing 4, 32, and 256 experts, and models with hierarchical MoEs containing 256, 1024, and 4096 experts. Each expert had about 1 million parameters. For all the MoE layers, 4 experts were active per input.\n\nThe results of these models are shown in Figure 2-left. The model with 4 always-active experts performed (unsurprisingly) similarly to the computationally-matched baseline models, while the largest of the models (4096 experts) achieved an impressive 24% lower perplexity on the test set. Figure 2: Model comparison on 1-Billion-Word Language-Modeling Benchmark. On the left, we plot test perplexity as a function of model capacity for models with similar computational budgets of approximately 8-million-ops-per-timestep. On the right, we plot test perplexity as a function of computational budget. The top line represents the LSTM models from (Jozefowicz et al., 2016).\n\nThe bottom line represents 4-billion parameter MoE models with different computational budgets.\n\nVaried Computation, High Capacity: In addition to the largest model from the previous section, we trained two more MoE models with similarly high capacity (4 billion parameters), but higher computation budgets. These models had larger LSTMs, and fewer but larger and experts. Details\n\nTest Test #Parameters ops/timestep Training TFLOPS Perplexity Perplexity excluding embedding Time /GPU 10 epochs 100 epochs and softmax layers 10 epochs Best Published Results 34.7 30.6 151 million 151 million 59 hours, 32 k40s 1.09 Low-Budget MoE Model 34.1 4303 million 8.9 million 15 hours, 16 k40s 0.74 Medium-Budget MoE Model 31.3 4313 million 33.8 million 17 hours, 32 k40s 1.22 High-Budget MoE Model 28.0 4371 million 142.7 million 47 hours, 32 k40s 1.56\n\nTable 1: Summary of high-capacity MoE-augmented models with varying computational budgets, vs. best previously published results (Jozefowicz et al., 2016). Details in Appendix C. can be found in Appendix C.2. Results of these three models form the bottom line of Figure 2-right.\n\nTable 1 compares the results of these models to the best previously-published result on this dataset .\n\nEven the fastest of these models beats the best published result (when controlling for the number of training epochs), despite requiring only 6% of the computation.\n\nComputational Efficiency: We trained our models using TensorFlow (Abadi et al., 2016) on clusters containing 16-32 Tesla K40 GPUs. For each of our models, we determine computational efficiency in TFLOPS/GPU by dividing the number of floating point operations required to process one training batch by the observed step time and the number of GPUs in the cluster. The operation counts used here are higher than the ones we report in our ops/timestep numbers in that we include the backwards pass, we include the importance-sampling-based training of the softmax layer, and we count a multiply-and-add as two separate operations. For all of our MoE models, the floating point operations involved in the experts represent between 37% and 46% of the total.\n\nFor our baseline models wtih no MoE, observed computational efficiency ranged from 1.07-1.29 TFLOPS/GPU. For our low-computation MoE models, computation efficiency ranged from 0.740.90 TFLOPS/GPU, except for the 4-expert model which did not make full use of the available parallelism. Our highest-computation MoE model was more efficient at 1.56 TFLOPS/GPU, likely due to the larger matrices. These numbers represent a significant fraction of the theoretical maximum of 4.29 TFLOPS/GPU claimed by NVIDIA. Detailed results are in Appendix C, Table 7.\n\n5.2 100 BILLION WORD GOOGLE NEWS CORPUS\n\nOn the 1-billion-word corpus, adding additional capacity seems to produce diminishing returns as the number of parameters in the MoE layer exceeds 1 billion, as can be seen in Figure 2-left. We hypothesized that for a larger training set, even higher capacities would produce significant quality improvements. We constructed a similar training set consisting of shuffled unique sentences from Google's internal news corpus, totalling roughly 100 billion words. Similarly to the previous section, we tested a series of models with similar computational costs of about 8 million ops/timestep. In addition to a baseline LSTM model, we trained models augmented with MoE layers containing 32, 256, 1024, 4096, 16384, 65536, and 131072 experts. This corresponds to up to 137 billion parameters in the MoE layer. Details on architecture, training, and results are given in Appendix D.\n\nResults: Figure 3 shows test perplexity as a function of capacity after training on 10 billion words (top line) and 100 billion words (bottom line). When training over the full 100 billion words, test perplexity improves significantly up to 65536 experts (68 billion parameters), dropping 39% lower than the computationally matched baseline, but degrades at 131072 experts, possibly a result of too much sparsity. The widening gap between the two lines demonstrates (unsurprisingly) that increased model capacity helps more on larger training sets. Even at 65536 experts (99.994% layer sparsity), computational efficiency for the model stays at a respectable 0.72 TFLOPS/GPU.\n\n5.3 Machine Translation (Single Language Pair)\n\nModel Architecture: Our model was a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decreased the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We inserted MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). Each MoE layer contained up to 2048 experts each with about two million parameters, adding a total of about 8 billion parameters to the models. Further details on model architecture, testing procedure and results can be found in Appendix E.\n\nDatasets: We benchmarked our method on the WMT'14 En→Fr and En→De corpora, whose training sets have 36M sentence pairs and 5M sentence pairs, respectively. The experimental protocols were also similar to those in (Wu et al., 2016): newstest2014 was used as the test set to compare against previous work (Luong et al., 2015a; Zhou et al., 2016; Wu et al., 2016), while the combination of newstest2012 and newstest2013 was used as the development set. We also tested the same model on a Google's Production English to French data.\n\nTable 2: Results on WMT'14 En→ Fr newstest2014 (bold values represent best results). Model Test Test ops/timenstep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.69 40.35 85M 8.7B 3 days/64 k40s MoE with 2048 Experts (longer training) 2.63 40.56 85M 8.7B 6 days/64 k40s GNMT (Wu et al., 2016) 2.79 39.22 214M 278M 6 days/96 k80s GNMT+RL (Wu et al., 2016) 2.96 39.92 214M 278M 6 days/96 k80s PBMT (Durrani et al., 2014) 37.0 LSTM (6-layer) (Luong et al., 2015b) 31.5 LSTM (6-layer+PosUnk) (Luong et al., 2015b) 33.1 DeepAtt (Zhou et al., 2016) 37.7 DeepAtt+PosUnk (Zhou et al., 2016) 39.2\n\nTable 3: Results on WMT'14 En → De newstest2014 (bold values represent best results). Model Test Test ops/timestep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 4.64 26.03 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 5.25 24.91 214M 278M 1 day/96 k80s GNMT +RL (Wu et al., 2016) 8.08 24.66 214M 278M 1 day/96 k80s PBMT (Durrani et al., 2014) 20.7 DeepAtt (Zhou et al., 2016) 20.6\n\nTable 4: Results on the Google Production En→ Fr dataset (bold values represent best results). Model Eval Eval Test Test ops/timestep Total Training Perplexity BLEU Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.60 37.27 2.69 36.57 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 2.78 35.80 2.87 35.56 214M 278M 6 days/96 k80s\n\nResults: Tables 2, 3, and 4 show the results of our largest models, compared with published results. Our approach achieved BLEU scores of 40.56 and 26.03 on the WMT'14 En→Fr and En→De benchmarks. As our models did not use RL refinement, these results constitute significant gains of 1.34 and 1.12 BLEU score on top of the strong baselines in (Wu et al., 2016). The perplexity scores are also better.2 On the Google Production dataset, our model achieved 1.01 higher test BLEU score even after training for only one sixth of the time.\n\n5.4 Multilingual Machine Translation\n\nDataset: (Johnson et al., 2016) train a single GNMT (Wu et al., 2016) model on a very large combined dataset of twelve language pairs. Results are somewhat worse than those for 12 separately trained single-pair GNMT models. This is not surprising, given that the twelve models have 12 times the capacity and twelve times the aggregate training of the one model. We repeat this experiment with a single MoE-augmented model. See Appendix E for details on model architecture.\n\nWe train our model on the same dataset as (Johnson et al., 2016) and process the same number of training examples (about 3 billion sentence pairs). Our training time was shorter due to the lower computational budget of our model.\n\nResults: Results for the single-pair GNMT models, the multilingual GNMT model and the multilingual MoE model are given in Table 5. The MoE model achieves 19% lower perplexity on the dev set than the multilingual GNMT model. On BLEU score, the MoE model significantly beats the multilingual GNMT model on 11 of the 12 language pairs (by as much as 5.84 points), and even beats the monolingual GNMT models on 8 of 12 language pairs. The poor performance on English → Korean seems to be a result of severe overtraining, as for the rarer language pairs a small number of real examples were highly oversampled in the training corpus.\n\nTable 5: Multilingual Machine Translation (bold values represent best results). GNMT-Mono GNMT-Multi MoE-Multi MoE-Multi vs. GNMT-Multi Parameters 278M / model 278M 8.7B ops/timestep 212M 212M 102M training time, hardware various 21 days, 96 k20s 12 days, 64 k40s Perplexity (dev) 4.14 3.35 -19% French → English Test BLEU 36.47 34.40 37.46 +3.06 German → English Test BLEU 31.77 31.17 34.80 +3.63 Japanese → English Test BLEU 23.41 21.62 25.91 +4.29 Korean → English Test BLEU 25.42 22.87 28.71 +5.84 Portuguese → English Test BLEU 44.40 42.53 46.13 +3.60 Spanish → English Test BLEU 38.00 36.04 39.39 +3.35 English → French Test BLEU 35.37 34.00 36.59 +2.59 English → German Test BLEU 26.43 23.15 24.53 +1.38 English → Japanese Test BLEU 23.66 21.10 22.78 +1.68 English → Korean Test BLEU 19.75 18.41 16.62 -1.79 English → Portuguese Test BLEU 38.40 37.35 37.90 +0.55 English → Spanish Test BLEU 34.50 34.25 36.21 +1.96\n\n6 Conclusion\n\nThis work is the first to demonstrate major wins from conditional computation in deep networks.\n\nWe carefully identified the design considerations and challenges of conditional computing and addressed them with a combination of algorithmic and engineering solutions. While we focused on text, conditional computation may help in other domains as well, provided sufficiently large training sets. We look forward to seeing many novel implementations and applications of conditional computation in the years to come.\n\nAcknowledgments\n\nWe would like to thank all of the members of the Google Brain and Google Translate teams who helped us with this project, in particular Zhifeng Chen, Yonghui Wu, and Melvin Johnson. Thanks also to our anonymous ICLR reviewers for the helpful suggestions on making this paper better.\n\n2Reported perplexities relative to the tokenization used by both our models and GNMT.\n\nReferences\n\nMartín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Gordon Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda B. Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. CoRR, abs/1603.04467, 2016. URL http://arxiv.org/abs/1603.04467.\n\nRahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. CoRR, abs/1611.06194, 2016. URL http://arxiv.org/abs/1611.\n\n06194.\n\nA. Almahairi, N. Ballas, T. Cooijmans, Y. Zheng, H. Larochelle, and A. Courville. Dynamic Capacity Networks. ArXiv e-prints, November 2015.\n\nDario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan, Christopher Fougner, Tony Han, Awni Y. Hannun, Billy Jun, Patrick LeGresley, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595, 2015. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.\n\nEmmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297, 2015.\n\nYoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.\n\nCiprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling.\n\narXiv preprint arXiv:1312.3005, 2013.\n\nK. Cho and Y. Bengio. Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning. ArXiv e-prints, June 2014. Ronan Collobert, Samy Bengio, and Yoshua Bengio. A parallel mixture of SVMs for very large scale problems. Neural Computing, 2002.\n\nAndrew Davis and Itamar Arel. Low-rank approximations for conditional feedforward computation in deep neural networks. arXiv preprint arXiv:1312.4461, 2013. Marc Peter Deisenroth and Jun Wei Ng. Distributed Gaussian processes. In ICML, 2015.\n\nJohn Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization, 2010.\n\nNadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. Edinburgh's phrase-based machine translation systems for wmt-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation, 2014.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013. Ekaterina Garmash and Christof Monz. Ensemble learning for multi-source neural machine translation. In staff.science.uva.nl/c.monz, 2016.\n\nFelix A. Gers, Jürgen A. Schmidhuber, and Fred A. Cummins. Learning to forget: Continual prediction with lstm. Neural Computation, 2000.\n\nAudrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. Memory-efficient backpropagation through time. CoRR, abs/1606.03401, 2016. URL http://arxiv.org/ abs/1606.03401. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2015.\n\nGeoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 2012. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.\n\nSergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.\n\nRobert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computing, 1991.\n\nMelvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation.\n\nCoRR, abs/1611.04558, 2016. URL http://arxiv.org/abs/1611.04558.\n\nMichael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm.\n\nNeural Computing, 1994. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.\n\nDiederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.\n\nReinhard Kneser and Hermann. Ney. Improved backingoff for m-gram language modeling., 1995.\n\nAlex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.\n\nQuoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeffrey Dean, and Andrew Y. Ng. Building high-level features using large scale unsupervised learning. In ICML, 2012. Patrick Gallinari Ludovic Denoyer. Deep sequential neural network. arXiv preprint arXiv:1410.0510, 2014.\n\nMinh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attentionbased neural machine translation. EMNLP, 2015a.\n\nMinh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the rare word problem in neural machine translation. ACL, 2015b.\n\nCarl Edward Rasmussen and Zoubin Ghahramani. Infinite mixtures of Gaussian process experts.\n\nNIPS, 2002.\n\nHasim Sak, Andrew W Senior, and Françoise Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH, pp. 338–342, 2014.\n\nMike Schuster and Kaisuke Nakajima. Japanese and Korean voice search. ICASSP, 2012.\n\nBabak Shahbaba and Radford Neal. Nonlinear models using dirichlet process mixtures. JMLR, 2009.\n\nIlya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks.\n\nIn NIPS, 2014.\n\nLucas Theis and Matthias Bethge. Generative image modeling using spatial LSTMs. In NIPS, 2015. Volker Tresp. Mixtures of Gaussian Processes. In NIPS, 2001.\n\nYonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.\n\nBangpeng Yao, Dirk Walther, Diane Beck, and Li Fei-fei. Hierarchical mixture of classification experts uncovers interactions between brain regions. In NIPS. 2009.\n\nWojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization.\n\narXiv preprint arXiv:1409.2329, 2014.\n\nJie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with fast-forward connections for neural machine translation. arXiv preprint arXiv:1606.04199, 2016.\n\nAppendices A Load-Balancing Loss\n\nAs discussed in section 4, for load-balancing purposes, we want to define an additional loss function to encourage experts to receive roughly equal numbers of training examples. Unfortunately, the number of examples received by an expert is a discrete quantity, so it can not be used in backpropagation. Instead, we define a smooth estimator Load(X) of the number of examples assigned to each expert for a batch X of inputs. The smoothness allows us to back-propagate gradients through the estimator. This is the purpose of the noise term in the gating function. We define P(x, i) as the probability that G(x)iis nonzero, given a new random choice of noise on element i, but keeping the already-sampled choices of noise on the other elements. To compute P(x, i), we note that the G(x)iis nonzero if and only if H(x)iis greater than the k th-greatest element of H(x) excluding itself. The probability works out to be:\n\n$$P(x,i)=P r\\Big((x\\cdot W_{g}){i}+S t a n d x N o r a l()\\cdot S o f t p u s((x\\cdot W{n o i s e})_{i})$$ $$>k t h_e x c l u d i n(H(x),k,i)\\Big)$$ $$({\\boldsymbol{8}})$$\n\n$$(9)$$ $$(10)$$ (8) Where kth_excluding(v, k, i) means the kth highest component of v, excluding component i. Simplifying, we get:\n\n$$P(x,i)=\\Phi{\\Big(}{\\frac{(x\\cdot W_{g}){i}-k t h_c x c l u d i n g(H(x),k,i)}{S o f t p l u s((x\\cdot W{n o i s e})_{i})}}{\\Big)}$$\n\nWhere Φ is the CDF of the standard normal distribution.\n\n$$L o a d(X){i}=\\sum{x\\in X}P(x,i)$$\n\n$$(11)$$ P(x, i) (10) We can now define the load loss to be the square of the coefficient of variation of the load vector, multiplied by a hand-tuned scaling factor wload.\n\n$$L_{l o a d}(X)=w_{l o a d}\\cdot C V(L o a d(X))^{2}$$ Lload(X) = wload · CV (Load(X))2(11) Initial Load Imbalance: To avoid out-of-memory errors, we need to initialize the network in a state of approximately equal expert load (since the soft constraints need some time to work). To accomplish this, we initialize the matrices Wg and Wnoise to all zeros, which yields no signal and some noise.\n\nExperiments: We trained a set of models with identical architecture (the MoE-256 model described in Appendix C), using different values of wimportance and wload. We trained each model for 10 epochs, then measured perplexity on the test set. We also measured the coefficients of variation in Importance and Load, as well as ratio of the load on the most overloaded expert to the average load. This last value is significant for load balancing purposes on distributed hardware. All of these metrics were averaged over several training batches.\n\nwimportance wload Test Perplexity CV (Importance(X)) CV (Load(X)) max(Load(X)) mean(Load(X)) 0.0 0.0 39.8 3.04 3.01 17.80 0.2 0.0 35.6 0.06 0.17 1.47 0.0 0.2 35.7 0.22 0.04 1.15 0.1 0.1 35.6 0.06 0.05 1.14 0.01 0.01 35.7 0.48 0.11 1.37 1.0 1.0 35.7 0.03 0.02 1.07\n\nTable 6: Experiments with different combinations of losses.\n\n13 Results: Results are reported in Table 6. All the combinations containing at least one the two losses led to very similar model quality, where having no loss was much worse. Models with higher values of wload had lower loads on the most overloaded expert.\n\nB Hierachical Mixture Of Experts\n\nIf the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network.3If the hierarchical MoE consists of a groups of b experts each, we denote the primary gating network by Gprimary, the secondary gating networks by (G1, G2..Ga), and the expert networks by (E0,0, E0,1..Ea,b). The output of the MoE is given by:\n\n$$y_{H}=\\sum_{i=1}^{a}\\sum_{j=1}^{b}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}\\cdot E{i,j}(x)$$ $$(12)$$ $$(13)$$ $$(14)$$\n\nOur metrics of expert utilization change to the following:\n\n$$I m p o r t a n c e_{H}(X){i,j}=\\sum{x\\in X}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}$$ $$L o a d{H}(X){i,j}={\\frac{L o a d{p r i m a r y}(X){i}\\cdot L o a d{i}(X^{(i)})_{j}}{|X^{(i)}|}}$$\n\nLoadprimary and Loadi deonte the Load functions for the primary gating network and i th secondary gating network respectively. X(i) denotes the subset of X for which Gprimary(x)i > 0.\n\nIt would seem simpler to let LoadH(X)i,j = Loadi(Xi)j , but this would not have a gradient with respect to the primary gating network, so we use the formulation above.\n\nC 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS C.1 8-MILLION-OPERATIONS-PER-TIMESTEP MODELS Model Architecture: Our model consists of five layers: a word embedding layer, a recurrent Long Short-Term Memory (LSTM) layer (Hochreiter & Schmidhuber, 1997; Gers et al., 2000), a MoE layer, a second LSTM layer, and a softmax layer. The dimensionality of the embedding layer, the number of units in each LSTM layer, and the input and output dimensionality of the MoE layer are all equal to 512. For every layer other than the softmax, we apply drouput (Zaremba et al., 2014) to the layer output, dropping each activation with probability DropP rob, otherwise dividing by (1 − DropP rob). After dropout, the output of the previous layer is added to the layer output.\n\nThis residual connection encourages gradient flow (He et al., 2015).\n\nMoE Layer Architecture: Each expert in the MoE layer is a feed forward network with one ReLU-activated hidden layer of size 1024 and an output layer of size 512. Thus, each expert contains [512 ∗ 1024] + [1024 ∗ 512] = 1M parameters. The output of the MoE layer is passed through a sigmoid function before dropout. We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts.\n\nWe call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers. Thus, each example is processed by exactly 4 experts for a total of 4M ops/timestep. The two LSTM layers contribute 2M ops/timestep each for the desired total of 8M.\n\n3 We have not found the need for deeper hierarchies.\n\nComputationally-Matched Baselines: The MoE-4 model does not employ sparsity, since all 4 experts are always used. In addition, we trained four more computationally-matched baseline models with no sparsity:\n\nMoE-1-Wide: The MoE layer consists of a single \"expert\" containing one ReLU-activated hidden layer of size 4096.\n\nMoE-1-Deep: The MoE layer consists of a single \"expert\" containing four ReLU-activated hidden layers, each with size 1024.\n\n4xLSTM-512: We replace the MoE layer with two additional 512-unit LSTM layers.\n\nLSTM-2048-512: The model contains one 2048-unit LSTM layer (and no MoE). The output of the LSTM is projected down to 512 dimensions (Sak et al., 2014). The next timestep of the LSTM receives the projected output. This is identical to one of the models published in (Jozefowicz et al., 2016). We re-ran it to account for differences in training regimen, and obtained results very similar to the published ones. Training: The models were trained on a cluster of 16 K40 GPUs using the synchronous method described in Section 3. Each batch consisted of a set of sentences totaling roughly 300,000 words. In the interest of time, we limited training to 10 epochs, (27,000 steps). Training took 12-16 hours for all models, except for MoE-4, which took 18 hours (since all the expert computation was performed on only 4 of 16 GPUs). We used the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 1000 training steps, and decreased after that so as to be proportional to the inverse square root of the step number. The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016). For each model, we performed a hyper-parmeter search to find the best dropout probability, in increments of 0.1.\n\nTo ensure balanced expert utilization we set wimportance = 0.1 and wload = 0.1, as described in Section 4 and Appendix A.\n\nResults: We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol. Results are reported in Table 7. For each model, we report the test perplexity, the computational budget, the parameter counts, the value of DropP rob, and the computational efficiency.\n\nModel Test Test ops/timestep #Params excluding Total Drop- TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params P rob per GPU 10 epochs (final) (millions) (billions) (observed) Kneser-Ney 5-gram* 67.6 0.00001 1.8 LSTM-512-512* 54.1 2.4 2.4 0.8 0.1 LSTM-1024-512* 48.2 4.7 4.7 0.8 0.1 LSTM-2048-512* 45.0 43.7 9.4 9.4 0.8 0.1 0.61 LSTM-2048-512 44.7 9.4 9.4 0.8 0.1 1.21 4xLSTM-512 46.0 8.4 8.4 0.8 0.1 1.07 MoE-1-Wide 46.1 8.4 8.4 0.8 0.1 1.29 MoE-1-Deep 45.7 8.4 8.4 0.8 0.1 1.29 MoE-4 45.0 8.4 8.4 0.8 0.1 0.52 MoE-32 39.7 8.4 37.8 0.9 0.1 0.87 MoE-256 35.7 8.6 272.9 1.1 0.1 0.81 MoE-256-h 36.0 8.4 272.9 1.1 0.1 0.89 MoE-1024-h 34.6 8.5 1079.0 1.9 0.2 0.90 MoE-4096-h 34.1 8.9 4303.4 5.1 0.2 0.74 2xLSTM-8192-1024* 34.7 30.6 151.0 151.0 1.8 0.25 1.09 MoE-34M 31.3 33.8 4313.9 6.0 0.3 1.22 MoE-143M 28.0 142.7 4371.1 6.0 0.4 1.56\n\nC.2 More Expensive Models\n\nWe ran two additional models (MoE-34M and MoE-143M) to investigate the effects of adding more computation in the presence of a large MoE layer. These models have computation budgets of 34M and 143M ops/timestep. Similar to the models above, these models use a MoE layer between two LSTM layers. The dimensionality of the embedding layer, and the input and output dimensionality of the MoE layer are set to 1024 instead of 512. For MoE-34M, the LSTM layers have 1024 units.\n\nFor MoE-143M, the LSTM layers have 4096 units and an output projection of size 1024 (Sak et al., 2014). MoE-34M uses a hierarchical MoE layer with 1024 experts, each with a hidden layer of size 2048. MoE-143M uses a hierarchical MoE layer with 256 experts, each with a hidden layer of size 8192. Both models have 4B parameters in the MoE layers. We searched for the best DropP rob for each model, and trained each model for 10 epochs.\n\nThe two models achieved test perplexity of 31.3 and 28.0 respectively, showing that even in the presence of a large MoE, more computation is still useful. Results are reported at the bottom of Table 7. The larger of the two models has a similar computational budget to the best published model from the literature, and training times are similar. Comparing after 10 epochs, our model has a lower test perplexity by 18%.\n\nD 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS Model Architecture: The models are similar in structure to the 8-million-operations-per-timestep models described in the previous section. We vary the number of experts between models, using an ordinary MoE layer with 32 experts and hierarchical MoE layers with 256, 1024, 4096, 16384, 65536 and 131072 experts. For the hierarchical MoE layers, the first level branching factors are 32, 32, 64, 128, 256 and 256, respectively.\n\nTraining: Models are trained on a cluster of 32 Tesla K40 GPUs, except for the last two models, which are trained on clusters of 64 and 128 GPUs so as to have enough memory for all the parameters. For all models, training batch sizes are approximately 2.5 million words. Models are trained once-through over about 100 billion words. We implement several memory optimizations in order to fit up to 1 billion parameters per GPU. First, we do not store the activations of the hidden layers of the experts, but instead recompute them on the backwards pass. Secondly, we modify the optimizer on the expert parameters to require less auxiliary storage: The Adam optimizer (Kingma & Ba, 2015) keeps first and second moment estimates of the perparameter gradients. This triples the required memory. To avoid keeping a first-moment estimator, we set β1 = 0. To reduce the size of the second moment estimator, we replace it with a factored approximation. For a matrix of parameters, instead of maintaining a full matrix of second-moment estimators, we maintain vectors of row-wise and column-wise averages of that matrix. At each step, the matrix of estimators is taken to be the outer product of those two vectors divided by the mean of either one. This technique could similarly be applied to Adagrad (Duchi et al., 2010).\n\nModel Test Test ops/timestep #Params excluding Total TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params per GPU .1 epochs 1 epoch (millions) (billions) (observed) Kneser-Ney 5-gram 67.1 45.3 0.00001 76.0 4xLSTM-512 54.5 47.0 8.4 8.4 0.1 1.23 MoE-32 48.5 40.4 8.4 37.8 0.1 0.83 MoE-256-h 42.8 35.3 8.4 272.9 0.4 1.11 MoE-1024-h 40.3 32.7 8.5 1079.0 1.2 1.14 MoE-4096-h 38.9 30.9 8.6 4303.4 4.4 1.07 MoE-16384-h 38.2 29.7 8.8 17201.0 17.3 0.96 MoE-65536-h 38.2 28.9 9.2 68791.0 68.9 0.72 MoE-131072-h 39.8 29.2 9.7 137577.6 137.7 0.30\n\nResults: We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8. Perplexity after 100 billion training words is 39% lower for the 68-billion-parameter MoE model than for the baseline model. It is notable that the measured computational efficiency of the largest model (0.30 TFLOPS/GPU) is very low compared to the other models. This is likely a result of the fact that, for purposes of comparison to the other models, we did not increase the training batch size proportionally to the number of GPUs. For comparison, we include results for a computationally matched baseline model consisting of 4 LSTMs, and for an unpruned 5-gram model with Kneser-Ney smoothing (Kneser & Ney, 1995).4\n\nE Machine Translation - Experimental Details\n\nModel Architecture for Single Language Pair MoE Models: Our model is a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decrease the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We insert MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). We use an attention mechanism between the encoder and decoder, with the first decoder LSTM receiving output from and providing input for the attention 5. All of the layers in our model have input and output dimensionality of 512. Our LSTM layers have 2048 hidden units, with a 512-dimensional output projection. We add residual connections around all LSTM and MoE layers to encourage gradient flow (He et al., 2015). Similar to GNMT, to effectively deal with rare words, we used subword units (also known as \"wordpieces\") (Schuster & Nakajima, 2012) for inputs and outputs in our system.\n\nWe use a shared source and target vocabulary of 32K wordpieces. We also used the same beam search technique as proposed in (Wu et al., 2016).\n\nWe train models with different numbers of experts in the MoE layers. In addition to a baseline model with no MoE layers, we train models with flat MoE layers containing 32 experts, and models with hierarchical MoE layers containing 512 and 2048 experts. The flat MoE layers use k = 4 and the hierarchical MoE models use k = 2 at each level of the gating network. Thus, each input is processed by exactly 4 experts in each MoE layer. Each expert in the MoE layer is a feed forward network with one hidden layer of size 2048 and ReLU activation. Thus, each expert contains [512 ∗ 2048] + [2048 ∗ 512] = 2M parameters. The output of the MoE layer is passed through a sigmoid function. We use the strictly-balanced gating function described in Appendix F.\n\nModel Architecture for Multilingual MoE Model: We used the same model architecture as for the single-language-pair models, with the following exceptions: We used noisy-top-k gating as described in Section 2.1, not the scheme from Appendix F. The MoE layers in the encoder and decoder are non-hierarchical MoEs with n = 512 experts, and k = 2. Each expert has a larger hidden layer of size 8192. This doubles the amount of computation in the MoE layers, raising the computational budget of the entire model from 85M to 102M ops/timestep. Training: We trained our networks using the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 2000 training steps, held constant for an additional 8000 steps, and decreased after that so as to be proportional to the inverse square root of the step number. For the single-language-pair models, similarly to (Wu et al., 2016), we applied dropout (Zaremba et al., 2014) to the output of all embedding, LSTM and MoE layers, using DropP rob = 0.4. Training was done synchronously on a cluster of up to 64 GPUs as described in section 3. Each training batch consisted of a set of sentence pairs containing roughly 16000 words per GPU.\n\nTo ensure balanced expert utilization we set wimportance = 0.01 and wload = 0.01, as described in Section 4 and Appendix A. Metrics: We evaluated our models using the perplexity and the standard BLEU score metric. We reported tokenized BLEU score as computed by the multi-bleu.pl script, downloaded from the public implementation of Moses (on Github), which was also used in (Luong et al., 2015a). Results: Tables 2, 3 and 4 in Section 5.3 show comparisons of our results to other published methods. Figure 4 shows test perplexity as a function of number of words in the (training data's) source sentences processed for models with different numbers of experts. As can be seen from the Figure, as we increased the number of experts to approach 2048, the test perplexity of our model continued to improve.\n\nWe found that the experts indeed become highly specialized by syntax and/or semantics, as can be seen in Table 9. For example, one expert is used when the indefinite article \"a\" introduces the direct object in a verb phrase indicating importance or leadership.\n\nExpert 381 Expert 752 Expert 2004 ... with researchers , ... ... plays a core ... ... with rapidly growing ... ... to innovation . ... plays a critical ... ... under static conditions ... ... tics researchers . ... provides a legislative ... ... to swift ly ... ... the generation of ... ... play a leading ... ... to dras tically ... ... technology innovations is ... ... assume a leadership ... ... the rapid and ... ... technological innovations , ... ... plays a central ... ... the fast est ... ... support innovation throughout ... ... taken a leading ... ... the Quick Method ... ... role innovation will ... ... established a reconciliation ... ... rec urrent ) ... ... research scienti st ... ... played a vital ... ... provides quick access ... ... promoting innovation where ... ... have a central ... ... of volatile organic ... ... ... ...\n\nDue to some peculiarities in our infrastructure which have since been fixed, at the time we ran some of the machine translation experiments, our models ran faster if every expert received exactly the same batch size. To accommodate this, we used a different gating function which we describe below. Recall that we define the softmax gating function to be:\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ Gσ(x) = Sof tmax(x · Wg) (15) Sparse Gating (alternate formulation): To obtain a sparse gating vector, we multiply Gσ(x) component-wise with a sparse mask M(Gσ(x)) and normalize the output. The mask itself is a function of Gσ(x) and specifies which experts are assigned to each input example:\n\n$$(15)$$ $$G(x){i}=\\frac{G{\\sigma}(x){i}M(G{\\sigma}(x)){i}}{\\sum{j=1}^{n}G_{\\sigma}(x){j}M(G{\\sigma}(x))_{j}}\\tag{1}$$ $$(17)$$\n\n$$(16)$$\n\nTop-K Mask: To implement top-k gating in this formulation, we would let M(v) = T opK(v, k), where:\n\n$$T o p K(v,k){i}=\\begin{cases}1&{\\mathrm{if~}}v{i}{\\mathrm{~is~in~the~top~}}k{\\mathrm{~elements~of~}}v.\\ 0&{\\mathrm{otherwise.}}\\end{cases}$$ 0 otherwise. (17) Batchwise Mask: To force each expert to receive the exact same number of examples, we introduce an alternative mask function, Mbatchwise(X, m), which operates over batches of input vectors.\n\nInstead of keeping the top k values per example, we keep the top m values per expert across the training batch, where m = k|X| n, so that each example is sent to an average of k experts.\n\n$$M_{b a t c h w i s e}(X,m){j,i}=\\begin{cases}1&\\text{if}X{j,i}\\text{is in the top}m\\text{values for to expert}i\\ 0&\\text{otherwise}\\end{cases}$$\n\nAs our experiments suggest and also observed in (Ioffe & Szegedy, 2015), using a batchwise function during training (such as Mbatchwise) requires modifications to the inference when we may not have a large batch of examples. Our solution to this is to train a vector T of per-expert threshold values to approximate the effects of the batchwise mask. We use the following mask at inference time:\n\n$$(18)$$ $$M_{threshold}(x,T){i}=\\begin{cases}1&\\text{if}x{i}>T_{i}\\ 0&\\text{otherwise}\\end{cases}\\tag{1}$$ $$(19)$$\n\nTo learn the threshold values, we apply an additional loss at training time which is minimized when the batchwise mask and the threshold mask are identical.\n\n$$L_{batchwise}(X,T,m)=\\sum_{j=1}^{|X|}\\sum_{i=1}^{n}(M_{threshold}(x,T){i}-M{batchwise}(X,m){j,i})(X{j,i}-T_{i})\\tag{20}$$\n\nG ATTENTION FUNCTION The attention mechanism described in GNMT (Wu et al., 2016) involves a learned \"Attention Function\" A(xi, yj ) which takes a \"source vector\" xi and a \"target vector\" yj , and must be computed for every source time step i and target time step j. In GNMT, the attention function is implemented as a feed forward neural network with a hidden layer of size n. It can be expressed as:\n\n$$A_{G N M T}(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}t a n h((x_{i}U){d}+(y{j}W)_{d})$$\n\nWhere U and W are trainable weight matrices and V is a trainable weight vector.\n\nFor performance reasons, in our models, we used a slightly different attention function:\n\n$$A(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}tanh((x_{i}U){d})tanh((y{j}W)_{d})\\tag{22}$$\n\nWith our attention function, we can simultaneously compute the attention function on multiple source time steps and multiple target time steps using optimized matrix multiplications. We found little difference in quality between the two functions.\n\n$$(21)$$",
+        "document_metadata": {
+          "source": "../../datasets/Sample_Docs_Markdown/1701.06538v1.md"
+        },
+        "headlines": {
+          "1 Introduction And Related Work": [
+            "1.1 Conditional Computation",
+            "1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER",
+            "1.3 Related Work On Mixtures Of Experts"
+          ],
+          "2 The Structure Of The Mixture-Of-Experts Layer": [
+            "2.1 Gating Network"
+          ],
+          "3 Addressing Performance Challenges": [
+            "3.1 The Shrinking Batch Problem",
+            "3.2 Network Bandwidth"
+          ],
+          "4 Balancing Expert Utilization": [],
+          "5 Experiments": [
+            "5.1 1 Billion Word Language Modeling Benchmark",
+            "5.2 100 BILLION WORD GOOGLE NEWS CORPUS",
+            "5.3 Machine Translation (Single Language Pair)",
+            "5.4 Multilingual Machine Translation"
+          ],
+          "6 Conclusion": [],
+          "Appendices": [
+            "A Load-Balancing Loss",
+            "B Hierachical Mixture Of Experts",
+            "C 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS",
+            "D 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS",
+            "E Machine Translation - Experimental Details"
+          ]
+        },
+        "summary": "The paper introduces a Sparsely-Gated Mixture-of-Experts (MoE) layer to enhance neural network capacity without a proportional increase in computation. This approach uses a trainable gating network to select a sparse combination of experts for each input, significantly improving model capacity and performance in language modeling and machine translation tasks. The MoE layer, with up to 137 billion parameters, achieves better results than state-of-the-art models at lower computational costs. The paper addresses challenges like batch size and network bandwidth, proposing solutions like mixing data and model parallelism. Experiments show that MoE models outperform traditional models in various benchmarks, demonstrating the potential of conditional computation in deep learning.",
+        "summary_embedding": [
+          -0.01644345559179783,
+          -0.016995344310998917,
+          -0.016372699290513992,
+          -0.014349107630550861,
+          0.02282555401325226,
+          0.045934125781059265,
+          -0.014646278694272041,
+          0.03563220426440239,
+          -0.004829026758670807,
+          0.005617944523692131,
+          -0.00932550523430109,
+          -0.008143897168338299,
+          0.005575491581112146,
+          0.01538213063031435,
+          -0.005653322208672762,
+          -0.020306676626205444,
+          -0.013832597061991692,
+          -0.023235931992530823,
+          0.028344441205263138,
+          -0.020080260932445526,
+          0.011823155917227268,
+          -0.020943470299243927,
+          -0.025868017226457596,
+          -0.016500059515237808,
+          0.025061409920454025,
+          0.024806693196296692,
+          0.02583971433341503,
+          0.003827843815088272,
+          -0.03492465242743492,
+          0.033651065081357956,
+          0.014674580655992031,
+          0.012488252483308315,
+          -0.031500112265348434,
+          -0.009162768721580505,
+          -0.05351905897259712,
+          -0.05227377265691757,
+          0.04143410921096802,
+          0.022245362401008606,
+          -0.016231190413236618,
+          0.023717066273093224,
+          0.017264213413000107,
+          0.005816058721393347,
+          -0.020632149651646614,
+          -0.020844414830207825,
+          0.003700484987348318,
+          -0.013103820383548737,
+          -0.01983969286084175,
+          -0.013528349809348583,
+          -0.01828308403491974,
+          -0.01577835902571678,
+          0.009551920928061008,
+          0.0002856731880456209,
+          0.0029186424799263477,
+          -0.003424540627747774,
+          -0.047830358147621155,
+          0.0248208437114954,
+          0.019853845238685608,
+          0.03492465242743492,
+          0.04633035510778427,
+          0.013011839240789413,
+          0.01516986545175314,
+          -0.014271277002990246,
+          -0.04021712392568588,
+          0.005497660953551531,
+          -0.00041656987741589546,
+          -0.005380915477871895,
+          -0.005819596350193024,
+          0.008575502783060074,
+          0.008724087849259377,
+          -0.01143400277942419,
+          -0.04202845320105553,
+          0.013952880166471004,
+          -0.020561393350362778,
+          0.00010143388499272987,
+          0.01354957651346922,
+          0.008292482234537601,
+          0.02677368000149727,
+          0.03195294365286827,
+          0.023971782997250557,
+          -0.008724087849259377,
+          -0.01340806670486927,
+          0.019924599677324295,
+          -0.0022676968947052956,
+          0.007018893025815487,
+          0.04570771008729935,
+          -0.04033033177256584,
+          0.02163686975836754,
+          -0.025726506486535072,
+          -0.015339677222073078,
+          -0.005349075887352228,
+          -0.022471778094768524,
+          0.02806142158806324,
+          -0.007344366051256657,
+          0.017801949754357338,
+          0.017094401642680168,
+          0.01594817079603672,
+          0.013330236077308655,
+          -0.001894464367069304,
+          -0.022938761860132217,
+          0.0018360915128141642,
+          -0.01541043259203434,
+          0.02114158496260643,
+          0.0015230007702484727,
+          0.005469358991831541,
+          0.027594437822699547,
+          0.031160488724708557,
+          -0.023490650579333305,
+          0.009502392262220383,
+          0.02422650158405304,
+          0.06967949122190475,
+          0.015424583107233047,
+          -0.006435164716094732,
+          -0.0024640418123453856,
+          0.04143410921096802,
+          0.0015513027319684625,
+          0.0038349193055182695,
+          -0.06826439499855042,
+          -0.03022652305662632,
+          -0.0011365016689524055,
+          -0.02254253253340721,
+          0.02227366529405117,
+          0.01002597901970148,
+          -0.009750034660100937,
+          -0.022457627579569817,
+          -0.04516997188329697,
+          0.05190584436058998,
+          -0.00545874610543251,
+          -0.016259491443634033,
+          -0.005554265342652798,
+          -0.05340585112571716,
+          0.00039136342820711434,
+          0.015042506158351898,
+          -0.018481198698282242,
+          -0.03670767694711685,
+          0.036141637712717056,
+          0.0007398316520266235,
+          0.03741522878408432,
+          0.012459950521588326,
+          -0.01708024926483631,
+          -0.020094411447644234,
+          0.022259512916207314,
+          -0.011639192700386047,
+          -0.0038313816767185926,
+          -0.027339721098542213,
+          0.0065236082300543785,
+          -0.028103873133659363,
+          -0.0014566679019480944,
+          0.06730213016271591,
+          -0.04816998168826103,
+          0.02946236915886402,
+          -0.029660483822226524,
+          0.023660462349653244,
+          -0.008575502783060074,
+          0.0278916098177433,
+          -0.010238244198262691,
+          0.005840822588652372,
+          -0.013457595370709896,
+          -0.0010586712742224336,
+          -0.020745357498526573,
+          0.0034210027661174536,
+          -0.016259491443634033,
+          -0.027750099077820778,
+          -0.007054270710796118,
+          0.031273696571588516,
+          0.01698119193315506,
+          0.025655752047896385,
+          0.025429336354136467,
+          0.01246702577918768,
+          -0.04935866594314575,
+          0.02595292218029499,
+          0.0018926954362541437,
+          0.013514199294149876,
+          0.013181651011109352,
+          -0.0010418669553473592,
+          0.025599148124456406,
+          0.012049571610987186,
+          0.025061409920454025,
+          0.037443529814481735,
+          0.0027682881336659193,
+          0.04737752676010132,
+          -0.015693452209234238,
+          -0.005292471963912249,
+          0.028528403490781784,
+          0.000505455827806145,
+          0.02465103194117546,
+          -0.001686621573753655,
+          0.013330236077308655,
+          0.000776978034991771,
+          0.03834919258952141,
+          -0.006339645478874445,
+          0.029518973082304,
+          0.021608568727970123,
+          0.028316138312220573,
+          -0.014759486541152,
+          0.023122724145650864,
+          -0.019754787907004356,
+          -0.02292460948228836,
+          0.035547297447919846,
+          0.004082561936229467,
+          0.01886327564716339,
+          -0.028457649052143097,
+          -0.030679354444146156,
+          0.015452885068953037,
+          -0.01855195313692093,
+          0.016556663438677788,
+          0.03639635816216469,
+          0.0008273909334093332,
+          -0.0025719432160258293,
+          0.026292547583580017,
+          -0.04245298355817795,
+          0.012856177985668182,
+          -0.01532552670687437,
+          -0.02855670638382435,
+          -0.019528372213244438,
+          -0.00875946506857872,
+          -0.0018095583654940128,
+          -0.013691086322069168,
+          -0.0037570889107882977,
+          -0.007563706487417221,
+          -0.02459442801773548,
+          0.00935380719602108,
+          0.013627407141029835,
+          0.009849091991782188,
+          -0.018665160983800888,
+          0.0019952901639044285,
+          -0.008257105015218258,
+          -0.0160047747194767,
+          -0.03752843663096428,
+          0.015268922783434391,
+          -0.0024888061452656984,
+          0.009191070683300495,
+          0.04078316316008568,
+          0.028485950082540512,
+          -0.005055442452430725,
+          0.0329718142747879,
+          -0.033707667142152786,
+          0.017915157601237297,
+          -0.008320784196257591,
+          0.009191070683300495,
+          0.002989397384226322,
+          -0.025542544201016426,
+          0.027424626052379608,
+          -0.03792466223239899,
+          -0.038151077926158905,
+          0.022995365783572197,
+          0.011858533136546612,
+          0.0017096169758588076,
+          -0.019386861473321915,
+          -0.017717044800519943,
+          0.0008499440737068653,
+          0.002752368338406086,
+          -0.018834972754120827,
+          -0.01019579079002142,
+          0.02227366529405117,
+          -0.025500090792775154,
+          0.036085035651922226,
+          0.023391593247652054,
+          -0.03520767390727997,
+          -0.00608492735773325,
+          -0.01903308741748333,
+          0.059037946164608,
+          -0.0011506526498124003,
+          -0.010301923379302025,
+          0.022938761860132217,
+          -0.027268964797258377,
+          -0.004524780437350273,
+          0.022301966324448586,
+          0.01828308403491974,
+          -0.008278331719338894,
+          0.003983504604548216,
+          -0.022287815809249878,
+          -0.05281151086091995,
+          -0.010577867738902569,
+          -0.0003305583668407053,
+          0.054368119686841965,
+          -0.012367968447506428,
+          0.0020695829298347235,
+          -0.006413938011974096,
+          0.004100250545889139,
+          0.03704730048775673,
+          0.002140337834134698,
+          0.0025082635693252087,
+          -0.013698162510991096,
+          0.0371888130903244,
+          0.022316116839647293,
+          0.005883275531232357,
+          -0.012162779457867146,
+          0.005377377849072218,
+          -0.04899074137210846,
+          -0.0009118546731770039,
+          0.005741765722632408,
+          0.007386818993836641,
+          -0.025910470634698868,
+          0.017717044800519943,
+          0.009205221198499203,
+          -0.0024693484883755445,
+          -0.009792488068342209,
+          0.01173117384314537,
+          0.025429336354136467,
+          -0.019938750192523003,
+          -0.05700020492076874,
+          0.0034705311991274357,
+          0.012014193460345268,
+          -0.004054259974509478,
+          -0.019401011988520622,
+          0.006792477332055569,
+          0.0021934041287750006,
+          0.022698193788528442,
+          -0.011271266266703606,
+          -0.0029221803415566683,
+          0.040585052222013474,
+          0.009516543708741665,
+          -0.0016813150141388178,
+          -0.017717044800519943,
+          -0.011794853024184704,
+          0.018070818856358528,
+          0.004740582779049873,
+          0.0035289041697978973,
+          -0.01751893013715744,
+          -0.029264256358146667,
+          0.02800481766462326,
+          0.019967053085565567,
+          -0.048905834555625916,
+          -0.008200501091778278,
+          -0.019273653626441956,
+          0.0019599127117544413,
+          0.008695785887539387,
+          0.04613224044442177,
+          -0.00590803986415267,
+          0.008313708938658237,
+          -0.020179318264126778,
+          -0.05003791302442551,
+          -0.02708500251173973,
+          0.03707560524344444,
+          -0.028684064745903015,
+          -0.04276430234313011,
+          -0.010252394713461399,
+          0.016740625724196434,
+          -0.009403335861861706,
+          -0.018679311498999596,
+          -0.011738249100744724,
+          -0.04969828948378563,
+          -0.003497064346447587,
+          0.0337359681725502,
+          0.017264213413000107,
+          -0.008441068232059479,
+          -0.033877480775117874,
+          0.019641580060124397,
+          -0.007160402834415436,
+          -0.006474079564213753,
+          0.0007862646016292274,
+          0.008794843219220638,
+          0.02400008589029312,
+          0.011879759840667248,
+          -0.02763689123094082,
+          -0.03427370637655258,
+          -0.0024888061452656984,
+          0.014929298311471939,
+          -0.00878776703029871,
+          0.0245378240942955,
+          0.024679332971572876,
+          -0.022995365783572197,
+          -0.01654251106083393,
+          0.015750056132674217,
+          -0.05779265984892845,
+          -0.010287772864103317,
+          -0.029235953465104103,
+          -0.006183984223753214,
+          0.0009613831643946469,
+          0.0061096916906535625,
+          0.00012304731353651732,
+          0.03846240043640137,
+          0.008801918476819992,
+          0.006014172453433275,
+          0.009191070683300495,
+          0.01913214474916458,
+          0.028471799567341805,
+          0.009155693463981152,
+          -0.03517937287688255,
+          0.02850010246038437,
+          -0.017051948234438896,
+          0.007811348885297775,
+          -0.025754809379577637,
+          -0.013033065013587475,
+          0.00443633645772934,
+          0.02725481428205967,
+          -0.000503244751598686,
+          0.018849123269319534,
+          -0.021919889375567436,
+          0.037500135600566864,
+          -0.029915200546383858,
+          -0.05278320610523224,
+          -0.02005195803940296,
+          -0.008285406976938248,
+          0.015764206647872925,
+          0.017278363928198814,
+          -0.019924599677324295,
+          -0.0286557637155056,
+          -0.021750077605247498,
+          0.006470541935414076,
+          0.010542490519583225,
+          -0.021099131554365158,
+          -0.010627396404743195,
+          0.01502835564315319,
+          -0.00404364662244916,
+          -0.01886327564716339,
+          0.013684011064469814,
+          0.015693452209234238,
+          0.03169822692871094,
+          -0.03308502212166786,
+          0.004287751391530037,
+          -0.025967074558138847,
+          -0.04327373951673508,
+          0.004096712917089462,
+          0.03166992589831352,
+          0.027509532868862152,
+          -0.018636859953403473,
+          0.010754755698144436,
+          0.003990580327808857,
+          0.004234685096889734,
+          -0.0033803186379373074,
+          0.023603858426213264,
+          0.03291521221399307,
+          0.02763689123094082,
+          -0.011653343215584755,
+          0.004921008367091417,
+          -0.01559439580887556,
+          -0.0011108530452474952,
+          0.01865101046860218,
+          -0.017816102132201195,
+          0.033877480775117874,
+          -0.029405765235424042,
+          0.02104252763092518,
+          -0.035122767090797424,
+          0.006141531281173229,
+          -0.012099100276827812,
+          -0.00404364662244916,
+          0.04833979532122612,
+          -0.017646290361881256,
+          -0.0019546060357242823,
+          -0.008101443760097027,
+          -0.014575524255633354,
+          -0.02158026583492756,
+          0.008766540326178074,
+          -0.04033033177256584,
+          -0.012367968447506428,
+          -0.028429346159100533,
+          0.003424540627747774,
+          0.019910449162125587,
+          -0.06277380883693695,
+          -0.06237758323550224,
+          0.012275987304747105,
+          -0.04918885603547096,
+          -0.026419905945658684,
+          0.02963218092918396,
+          -0.002984090941026807,
+          0.0012780115939676762,
+          -0.0024003623984754086,
+          0.0068101659417152405,
+          -0.029207652434706688,
+          0.00425944896414876,
+          -0.000887974863871932,
+          0.03271709755063057,
+          -0.021056678146123886,
+          -0.008122670464217663,
+          0.004878554958850145,
+          -0.06860402226448059,
+          -0.028344441205263138,
+          0.011271266266703606,
+          -0.009459939785301685,
+          -0.0066615804098546505,
+          -0.017915157601237297,
+          -0.004383270163089037,
+          0.0008950503543019295,
+          0.0032830305863171816,
+          0.05801907554268837,
+          -0.01983969286084175,
+          -0.053773775696754456,
+          0.03633975237607956,
+          0.00887267291545868,
+          0.029603879898786545,
+          -0.025556694716215134,
+          -0.017533080652356148,
+          -0.01855195313692093,
+          -0.020504789426922798,
+          -0.04123599827289581,
+          0.00404364662244916,
+          -0.005918653216212988,
+          0.009212297387421131,
+          -0.0009693430620245636,
+          0.02763689123094082,
+          0.029235953465104103,
+          0.023108573630452156,
+          0.012750045396387577,
+          0.01079013291746378,
+          -0.022372720763087273,
+          -0.021594416350126266,
+          0.0012222920777276158,
+          0.04683978855609894,
+          0.024636881425976753,
+          -0.02675952948629856,
+          -0.011504758149385452,
+          0.03953787684440613,
+          0.0419718474149704,
+          -0.01974063739180565,
+          -0.02276895008981228,
+          0.00764861237257719,
+          0.03266049548983574,
+          -0.04477374628186226,
+          0.005932804197072983,
+          0.03410389646887779,
+          0.014405711553990841,
+          -0.005356151144951582,
+          -0.010294848121702671,
+          0.022570835426449776,
+          0.03152841329574585,
+          -0.008724087849259377,
+          0.01876421831548214,
+          0.016910437494516373,
+          0.029349161311984062,
+          -0.0067889392375946045,
+          0.0005766529939137399,
+          -0.024735936895012856,
+          0.055188875645399094,
+          -0.03059444949030876,
+          0.02909444272518158,
+          0.010316074825823307,
+          -0.00593634182587266,
+          0.039056744426488876,
+          -0.016641568392515182,
+          -0.0431322306394577,
+          0.01913214474916458,
+          -0.006979977712035179,
+          -0.0055613406002521515,
+          -0.03444352000951767,
+          -0.007592008449137211,
+          -0.007507102098315954,
+          -0.006714646704494953,
+          -0.022556684911251068,
+          -0.02606613002717495,
+          0.004415110219269991,
+          0.00973588414490223,
+          -0.006484692916274071,
+          0.00698705343529582,
+          0.02092931978404522,
+          -0.013507124036550522,
+          0.02525952458381653,
+          -0.046047333627939224,
+          -0.0008366775000467896,
+          -0.0038596836384385824,
+          -0.025712355971336365,
+          -0.0345001220703125,
+          -0.0005770951975136995,
+          -0.0024251264985650778,
+          -0.024622729048132896,
+          -0.00914154201745987,
+          -0.008752389810979366,
+          -0.0023119186516851187,
+          -0.004938696976751089,
+          -0.055188875645399094,
+          0.008695785887539387,
+          0.010846736840903759,
+          -0.002713453024625778,
+          0.0055507272481918335,
+          -0.010613245889544487,
+          -0.03039633482694626,
+          0.03039633482694626,
+          0.04129260033369064,
+          0.033707667142152786,
+          0.014334957115352154,
+          -0.0074646491557359695,
+          -0.017165156081318855,
+          0.006060163024812937,
+          0.021721776574850082,
+          0.00902833417057991,
+          0.007846726104617119,
+          0.03877372294664383,
+          0.00011331850691931322,
+          -0.008349086157977581,
+          -0.009764186106622219,
+          0.028740668669342995,
+          -0.001085204305127263,
+          0.023603858426213264,
+          0.050802066922187805,
+          0.030424637719988823,
+          0.020589696243405342,
+          -0.01681138016283512,
+          -0.02010856196284294,
+          -0.00851889792829752,
+          -0.02855670638382435,
+          0.0364246591925621,
+          0.012127402238547802,
+          -0.005232329946011305,
+          -0.011540135368704796,
+          0.019061388447880745,
+          -0.033339742571115494,
+          -0.0037358624394983053,
+          0.00048422932741232216,
+          0.013924578204751015,
+          -0.03379257395863533,
+          0.0002480846014805138,
+          0.031613320112228394,
+          0.014957600273191929,
+          -0.016995344310998917,
+          -0.0188774261623621,
+          0.010549565777182579,
+          0.020462337881326675,
+          0.018948180601000786,
+          0.00037743354914709926,
+          0.00897173024713993,
+          -0.00022121983056422323,
+          -0.0005319889169186354,
+          0.01590571738779545,
+          -0.007952858693897724,
+          0.036254845559597015,
+          -0.003525366308167577,
+          -0.01325240544974804,
+          -0.0037500134203583,
+          -0.021438755095005035,
+          -0.024467067793011665,
+          0.043556760996580124,
+          0.023971782997250557,
+          0.04661337286233902,
+          -0.005660397466272116,
+          -0.004245297983288765,
+          0.025825563818216324,
+          0.008221727795898914,
+          -0.03288691118359566,
+          0.000265110022155568,
+          0.005281858611851931,
+          -0.02892463095486164,
+          0.016641568392515182,
+          -0.02476423978805542,
+          -0.016896286979317665,
+          -0.006771250627934933,
+          -0.0015557248843833804,
+          -0.028188779950141907,
+          0.01703779585659504,
+          0.0031309074256569147,
+          -0.0027099153958261013,
+          -0.039764292538166046,
+          0.006880920846015215,
+          0.020419884473085403,
+          0.015311375260353088,
+          0.0037358624394983053,
+          0.011525984853506088,
+          -0.016613267362117767,
+          0.002126186853274703,
+          0.03379257395863533,
+          0.004878554958850145,
+          0.02005195803940296,
+          0.0027647505048662424,
+          0.023773670196533203,
+          0.03633975237607956,
+          -0.01681138016283512,
+          -0.02189158834517002,
+          0.01519816741347313,
+          -0.02233026921749115,
+          0.029943503439426422,
+          0.004889168310910463,
+          -0.012283062562346458,
+          0.022033097222447395,
+          -0.009827865287661552,
+          -0.0027346794959157705,
+          0.01659911684691906,
+          -0.022740647196769714,
+          -0.017717044800519943,
+          0.03183973580598831,
+          -0.039452970027923584,
+          0.012856177985668182,
+          -0.008547200821340084,
+          0.015863263979554176,
+          -0.023066120222210884,
+          0.011172209866344929,
+          0.007100261282175779,
+          -0.01573590561747551,
+          0.010797208175063133,
+          0.009693430736660957,
+          0.00545874610543251,
+          0.0018926954362541437,
+          0.02179253101348877,
+          -0.0006049550138413906,
+          -0.011257115751504898,
+          -0.018155725672841072,
+          -0.004694592207670212,
+          -0.002614396158605814,
+          -0.016556663438677788,
+          0.01865101046860218,
+          -0.041066184639930725,
+          -0.013280707411468029,
+          -0.008879749104380608,
+          -0.013988257385790348,
+          -0.010011828504502773,
+          0.024184048175811768,
+          0.025669902563095093,
+          -0.002361447084695101,
+          -0.031217092648148537,
+          0.01790100708603859,
+          0.024198200553655624,
+          0.003654494183138013,
+          -0.023646311834454536,
+          0.0302831269800663,
+          -0.008936353027820587,
+          0.014971751719713211,
+          0.026136886328458786,
+          0.014094389975070953,
+          0.004595535341650248,
+          -0.010500037111341953,
+          -0.02114158496260643,
+          0.01481609046459198,
+          0.010507113300263882,
+          0.033537857234478,
+          0.014745336025953293,
+          0.02071705460548401,
+          0.01594817079603672,
+          -0.023320838809013367,
+          -0.011214662343263626,
+          -0.003482913365587592,
+          -0.022514231503009796,
+          0.03093407303094864,
+          0.0006129149114713073,
+          0.02114158496260643,
+          0.026618018746376038,
+          -0.03319823369383812,
+          -0.01187268365174532,
+          0.032688796520233154,
+          0.02779255248606205,
+          0.05108508840203285,
+          0.025316128507256508,
+          0.015396281145513058,
+          0.027226513251662254,
+          -0.03585861995816231,
+          -0.036368053406476974,
+          0.001548649393953383,
+          0.04256619140505791,
+          -0.008426916785538197,
+          0.00967928022146225,
+          0.03900013864040375,
+          -0.006781863979995251,
+          -0.01611798256635666,
+          0.019457615911960602,
+          0.0026710000820457935,
+          0.002448122017085552,
+          -0.0033113325480371714,
+          -0.01335146278142929,
+          0.03175482898950577,
+          -0.0051474240608513355,
+          0.013570803217589855,
+          -0.019273653626441956,
+          0.027113303542137146,
+          0.0001799829478841275,
+          -0.003229964291676879,
+          0.028344441205263138,
+          -0.002660386962816119,
+          -0.051990751177072525,
+          0.024212351068854332,
+          0.019811391830444336,
+          -0.011561362072825432,
+          -0.0036332677118480206,
+          -0.003792466362938285,
+          0.04567940905690193,
+          0.034698236733675,
+          0.029009537771344185,
+          -0.0020501252729445696,
+          0.0168679840862751,
+          0.018792519345879555,
+          0.0053667644970119,
+          0.014278353191912174,
+          -0.0017025414854288101,
+          -0.007853801362216473,
+          0.021056678146123886,
+          -0.020787810906767845,
+          0.03226426616311073,
+          -0.016570813953876495,
+          -0.020122714340686798,
+          0.008738238364458084,
+          -0.011172209866344929,
+          -0.0009083169279620051,
+          -0.006311343517154455,
+          -0.0008813415770418942,
+          -0.0029664020985364914,
+          0.016938740387558937,
+          -0.004446949809789658,
+          0.04661337286233902,
+          -0.013931653462350368,
+          -0.008808993734419346,
+          0.006972902454435825,
+          -0.025981225073337555,
+          0.0013346156338229775,
+          0.0028726516757160425,
+          0.027000095695257187,
+          -0.01989629678428173,
+          -0.00970758218318224,
+          0.01606137864291668,
+          0.06039644405245781,
+          -0.008639181964099407,
+          -0.0026267783250659704,
+          0.027226513251662254,
+          -0.002787745790556073,
+          -0.03551899641752243,
+          0.012919857166707516,
+          -0.04494355618953705,
+          0.017589686438441277,
+          0.026349151507019997,
+          -0.04986810311675072,
+          -0.022089701145887375,
+          -0.02347649820148945,
+          -0.02043403498828411,
+          0.023434046655893326,
+          -0.009976450353860855,
+          0.013867974281311035,
+          -0.017801949754357338,
+          -0.009014182724058628,
+          0.020518941804766655,
+          0.009176919236779213,
+          0.03735862299799919,
+          -0.027551984414458275,
+          -0.016783079132437706,
+          -0.027820853516459465,
+          -0.04304732382297516,
+          0.009558996185660362,
+          -0.0278916098177433,
+          0.0027099153958261013,
+          -0.0016291331266984344,
+          -0.040302030742168427,
+          -0.037981268018484116,
+          0.005278320983052254,
+          -0.017985913902521133,
+          0.01516986545175314,
+          0.020915169268846512,
+          0.02731141820549965,
+          -0.0020253609400242567,
+          -0.030198222026228905,
+          0.029235953465104103,
+          -0.007344366051256657,
+          0.03223596513271332,
+          -0.00689507182687521,
+          0.005066055804491043,
+          -0.0070436573587358,
+          0.031160488724708557,
+          -0.023009516298770905,
+          0.018792519345879555,
+          -0.026801982894539833,
+          -0.004086099565029144,
+          -0.018990634009242058,
+          -0.01774534583091736,
+          0.004224071744829416,
+          0.004521242808550596,
+          7.888073741924018e-05,
+          0.0048750173300504684,
+          0.005777143407613039,
+          0.024806693196296692,
+          0.0021421066485345364,
+          0.02589631825685501,
+          -0.027877457439899445,
+          -0.025316128507256508,
+          0.01719345711171627,
+          -0.02077365852892399,
+          0.005678086541593075,
+          -0.006261814851313829,
+          -0.0302831269800663,
+          -0.01202834490686655,
+          0.023391593247652054,
+          0.02162271924316883,
+          -0.005490585695952177,
+          0.002644466934725642,
+          0.02206140011548996,
+          -0.026023678481578827,
+          0.00437973253428936,
+          -0.03781145438551903,
+          -0.018396291881799698,
+          0.029773691669106483,
+          -0.03195294365286827,
+          0.020745357498526573,
+          0.003185742534697056,
+          0.03483974561095238,
+          -0.017292514443397522,
+          -0.01947176828980446,
+          0.01654251106083393,
+          -0.01008258294314146,
+          0.004142703488469124,
+          0.01606137864291668,
+          0.01616043597459793,
+          -0.0015070808585733175,
+          0.05380208045244217,
+          0.0008167776977643371,
+          0.006909222807735205,
+          -0.049075648188591,
+          0.004496478475630283,
+          -0.044858649373054504,
+          0.0002768288250081241,
+          -0.013337312266230583,
+          -0.02514631673693657,
+          0.0014027173165231943,
+          0.001001182827167213,
+          0.001963450340554118,
+          -0.01913214474916458,
+          -0.01989629678428173,
+          0.011596739292144775,
+          -0.015863263979554176,
+          0.004928083624690771,
+          -0.06418890506029129,
+          -0.01855195313692093,
+          -0.004974074196070433,
+          0.0043337419629096985,
+          0.0007699025445617735,
+          0.03342464938759804,
+          -0.01478778850287199,
+          0.010117961093783379,
+          -0.01549533847719431,
+          -0.026363302022218704,
+          0.04010391607880592,
+          0.018750067800283432,
+          -0.023773670196533203,
+          0.024693485349416733,
+          0.012106175534427166,
+          0.03099067695438862,
+          -0.0318114347755909,
+          0.020349130034446716,
+          0.018523650243878365,
+          -0.0029823218937963247,
+          0.006551910191774368,
+          0.008575502783060074,
+          0.01832553744316101,
+          -0.015764206647872925,
+          -0.0003632825391832739,
+          0.011419852264225483,
+          0.010033054277300835,
+          0.031924642622470856,
+          0.01403071079403162,
+          -0.03183973580598831,
+          0.019047237932682037,
+          0.01616043597459793,
+          0.02298121340572834,
+          0.033820874989032745,
+          -0.027481229975819588,
+          0.038575608283281326,
+          -0.010386829264461994,
+          -0.024042539298534393,
+          0.020801961421966553,
+          -0.032688796520233154,
+          -0.006679269019514322,
+          0.030735958367586136,
+          0.04398128762841225,
+          -0.0015566092915832996,
+          -0.044632233679294586,
+          0.01535382866859436,
+          0.01316749956458807,
+          0.0027806703001260757,
+          0.028811423107981682,
+          0.011264191009104252,
+          0.021820832043886185,
+          0.025273675099015236,
+          0.011419852264225483,
+          -0.0037252490874379873,
+          0.024962352588772774,
+          -0.008101443760097027,
+          -0.004446949809789658,
+          -0.0007035697344690561,
+          0.03206615149974823,
+          0.014702882617712021,
+          -0.017603836953639984,
+          -0.036962397396564484,
+          -0.007620310410857201,
+          0.009452863596379757,
+          -0.007903330028057098,
+          -0.027127455919981003,
+          -0.03950957581400871,
+          -0.026787832379341125,
+          -0.005667473189532757,
+          0.01505665760487318,
+          -0.06724552065134048,
+          -0.007620310410857201,
+          -0.012233533896505833,
+          0.017717044800519943,
+          0.0022906921803951263,
+          0.0181132722645998,
+          0.001277127186767757,
+          -0.009403335861861706,
+          0.004209920763969421,
+          0.050802066922187805,
+          -0.03059444949030876,
+          0.014242975041270256,
+          0.011292492970824242,
+          0.0032105068676173687,
+          0.002787745790556073,
+          0.038688816130161285,
+          -0.010988246649503708,
+          -0.030424637719988823,
+          0.011575513519346714,
+          -0.017292514443397522,
+          0.003383856499567628,
+          0.05006621778011322,
+          -0.009481165558099747,
+          -0.027495380491018295,
+          0.020915169268846512,
+          -0.009148617275059223,
+          -0.015523640438914299,
+          -0.0027258351910859346,
+          -0.006035399157553911,
+          0.0006283925613388419,
+          -0.03832089155912399,
+          -0.015750056132674217,
+          0.03212275728583336,
+          0.008617955259978771,
+          0.014603826217353344,
+          0.025344429537653923,
+          0.010209942236542702,
+          -0.030481241643428802,
+          0.027481229975819588,
+          -0.0005678086308762431,
+          0.005221716593950987,
+          0.013323160819709301,
+          0.03424540534615517,
+          -0.014264201745390892,
+          -0.020264223217964172,
+          -0.040245428681373596,
+          -0.013853822834789753,
+          -0.0463869571685791,
+          0.001943992800079286,
+          0.006919836159795523,
+          -0.012919857166707516,
+          -0.04366996884346008,
+          -0.0329718142747879,
+          0.021127434447407722,
+          0.03178313374519348,
+          -0.017023645341396332,
+          0.0025294902734458447,
+          -0.008483520708978176,
+          -0.007882103323936462,
+          -0.0017520699184387922,
+          -0.013217028230428696,
+          -0.006378560326993465,
+          -0.006229975260794163,
+          0.015452885068953037,
+          -0.004652139265090227,
+          -0.0036934094969183207,
+          -0.0498398020863533,
+          -0.018778368830680847,
+          0.004082561936229467,
+          -0.008221727795898914,
+          0.007323139347136021,
+          -0.006955213379114866,
+          -0.002727604005485773,
+          -0.012474101036787033,
+          -0.03829259052872658,
+          0.024184048175811768,
+          0.017278363928198814,
+          0.0035501306410878897,
+          -0.008922201581299305,
+          -0.018367990851402283,
+          0.03294351324439049,
+          0.012523629702627659,
+          -0.017547233030200005,
+          -0.018580254167318344,
+          0.025981225073337555,
+          0.026957644149661064,
+          0.008122670464217663,
+          -0.006137993652373552,
+          0.029886899515986443,
+          0.005695775151252747,
+          -0.022938761860132217,
+          -0.003983504604548216,
+          -0.005055442452430725,
+          -0.0038844477385282516,
+          0.020561393350362778,
+          -0.010556641966104507,
+          0.0300001073628664,
+          0.038717120885849,
+          0.012870329432189465,
+          0.0039198254235088825,
+          -0.0033714743331074715,
+          -0.008922201581299305,
+          0.015976471826434135,
+          0.006481155287474394,
+          0.008525974117219448,
+          0.014759486541152,
+          -0.016358548775315285,
+          -0.020405733957886696,
+          0.006215824279934168,
+          -0.007107336539775133,
+          0.020122714340686798,
+          0.01708024926483631,
+          0.016429303213953972,
+          0.0007610581233166158,
+          -0.028287837281823158,
+          -0.0018024828750640154,
+          0.02411329373717308,
+          0.02010856196284294,
+          -0.004655676893889904,
+          -0.023264234885573387,
+          0.014398636296391487,
+          -0.015806660056114197,
+          -0.009969375096261501,
+          0.0033431723713874817,
+          0.006229975260794163,
+          -0.003083147807046771,
+          0.0571134127676487,
+          -0.020193468779325485,
+          -0.011419852264225483,
+          -0.010287772864103317,
+          -0.010867963545024395,
+          0.0012072566896677017,
+          -0.01828308403491974,
+          0.008964654989540577,
+          0.025613298639655113,
+          -0.009764186106622219,
+          0.03568880632519722,
+          0.0228114016354084,
+          0.006912760436534882,
+          0.005260631907731295,
+          -0.027070851996541023,
+          0.012304289266467094,
+          -0.016047226265072823,
+          0.005660397466272116,
+          0.018467046320438385,
+          0.017320817336440086,
+          0.0076132346875965595,
+          0.026306698098778725,
+          0.019386861473321915,
+          0.016684021800756454,
+          -0.022783100605010986,
+          0.025457637384533882,
+          -0.007832575589418411,
+          0.027877457439899445,
+          -0.01611798256635666,
+          -0.002253545681014657,
+          -0.012990612536668777,
+          0.008681634441018105,
+          0.002416282193735242,
+          -0.0019351483788341284,
+          0.00036129256477579474,
+          0.016783079132437706,
+          -0.014589674770832062,
+          0.01062032114714384,
+          0.013316085562109947,
+          -0.027919910848140717,
+          -0.010372678749263287,
+          0.016344398260116577,
+          -0.00668988237157464,
+          0.006877383217215538,
+          -0.011865608394145966,
+          -0.0017202302115038037,
+          0.011009473353624344,
+          -0.0436982698738575,
+          0.0104929618537426,
+          0.006279503460973501,
+          -0.015155714936554432,
+          0.014801939949393272,
+          -0.009799563325941563,
+          -0.008235878311097622,
+          -0.013592029921710491,
+          0.003792466362938285,
+          -0.00030225637601688504,
+          -0.0018290160223841667,
+          -0.00022995365725364536,
+          0.00389859895221889,
+          0.010047205723822117,
+          -0.0045990729704499245,
+          -0.010450509376823902,
+          -0.0028638073708862066,
+          -0.02265574224293232,
+          -0.0010047205723822117,
+          0.018735915422439575,
+          -0.004591997712850571,
+          0.0005222601466812193,
+          0.0145189194008708,
+          -0.005830209702253342,
+          0.010556641966104507,
+          -0.005221716593950987,
+          -0.008320784196257591,
+          -0.010768906213343143,
+          -0.033764272928237915,
+          0.0026710000820457935,
+          -0.015254771336913109,
+          0.008865597657859325,
+          -0.012544856406748295,
+          0.04027372971177101,
+          0.01022409275174141,
+          -0.00837738811969757,
+          -0.002796590095385909,
+          -0.0228114016354084,
+          -0.004475251771509647,
+          -0.016938740387558937,
+          -0.01184438169002533,
+          0.008342010900378227,
+          0.012629762291908264,
+          0.005614406894892454,
+          -0.018410442396998405,
+          0.003665107535198331,
+          0.0030955299735069275,
+          -0.005575491581112146,
+          -0.008370312862098217,
+          0.010853812098503113,
+          -0.0006248548161238432,
+          -0.01800006441771984,
+          -0.01957082375884056,
+          -0.027424626052379608,
+          0.0150000536814332,
+          0.014589674770832062,
+          -0.01011088490486145,
+          -0.022754797711968422,
+          -0.007471724878996611,
+          0.007107336539775133,
+          -0.02064630016684532,
+          0.0006779210525564849,
+          0.006891534198075533,
+          0.008717012591660023,
+          0.004117939155548811,
+          0.009636826813220978,
+          0.0007699025445617735,
+          0.0121698547154665,
+          0.018566103652119637,
+          0.03271709755063057,
+          -0.0396793857216835,
+          -0.009537769481539726,
+          0.0057028504088521,
+          -0.001465512323193252,
+          -0.013853822834789753,
+          0.009672204032540321,
+          0.0024375086650252342,
+          -0.0026426981203258038,
+          0.024424616247415543,
+          -0.008044839836657047,
+          -0.01505665760487318,
+          0.01184438169002533,
+          0.00887267291545868,
+          -0.019075538963079453,
+          0.02746707946062088,
+          0.006169833242893219,
+          -0.01011088490486145,
+          0.02259913645684719,
+          -0.012360893189907074,
+          0.012658064253628254,
+          -0.005522425286471844,
+          0.023660462349653244,
+          -0.019004784524440765,
+          0.0006306036957539618,
+          0.015665151178836823,
+          0.03883032873272896,
+          -0.00437973253428936,
+          -0.004567233379930258,
+          -0.03421710431575775,
+          0.005115584470331669,
+          0.028853876516222954,
+          0.02666047215461731,
+          -0.0012346742441877723,
+          -0.005596718285232782,
+          -0.0019864458590745926,
+          0.010768906213343143,
+          0.011589664034545422,
+          0.017561383545398712,
+          -0.029547275975346565,
+          0.00022011429246049374,
+          -0.016938740387558937,
+          0.025500090792775154,
+          0.014632128179073334,
+          0.006951675750315189,
+          0.04072656109929085,
+          0.0027789014857262373,
+          0.003841994795948267,
+          0.027452928945422173,
+          -0.00779719790443778,
+          0.003546592779457569,
+          -0.015566092915832996,
+          0.0003175129240844399,
+          0.022853855043649673,
+          0.0016370931407436728,
+          0.0020784272346645594,
+          0.013337312266230583,
+          0.029320860281586647,
+          -0.022471778094768524,
+          -0.023929331451654434,
+          0.003482913365587592,
+          0.007185167167335749,
+          -0.010570792481303215,
+          0.0038738346192985773,
+          0.006275965832173824,
+          -0.006007096730172634,
+          0.008900974877178669,
+          0.006332569755613804,
+          -0.0003166284877806902,
+          0.0039198254235088825,
+          -0.00308845448307693,
+          0.012962310574948788,
+          -0.02702839858829975,
+          0.036849189549684525,
+          0.004439874552190304,
+          0.010075507685542107,
+          0.015268922783434391,
+          0.001627364312298596,
+          0.004917470272630453,
+          0.005869124550372362,
+          0.010188715532422066,
+          0.018636859953403473,
+          -0.00956607237458229,
+          -0.027481229975819588,
+          -0.03166992589831352,
+          0.020363280549645424,
+          -0.01173117384314537,
+          0.0035872769076377153,
+          0.0074646491557359695,
+          -0.01828308403491974,
+          -0.007507102098315954,
+          -0.009976450353860855,
+          0.015764206647872925,
+          -0.03189634159207344,
+          0.012990612536668777,
+          -0.0043231286108493805,
+          -0.028669914230704308,
+          0.00851889792829752,
+          -0.010181640274822712,
+          -0.0045884596183896065,
+          -0.0035094465129077435,
+          -0.029264256358146667,
+          -0.007931631989777088,
+          0.03908504545688629,
+          -0.02141045406460762,
+          0.024184048175811768,
+          0.008257105015218258,
+          -0.00014449491573031992,
+          -0.0040365708991885185,
+          -0.013563727959990501,
+          0.005331386812031269,
+          -0.01778779923915863,
+          0.025457637384533882,
+          0.010294848121702671,
+          -0.013924578204751015,
+          0.0322076641023159,
+          -0.011179285123944283,
+          0.027438776567578316,
+          -0.018580254167318344,
+          0.036198243498802185,
+          0.007422196213155985,
+          0.00674648629501462,
+          0.01974063739180565,
+          -0.004025958012789488,
+          0.008228803053498268,
+          0.04559450224041939,
+          -0.018523650243878365,
+          0.02460857853293419,
+          0.020349130034446716,
+          -0.00022685811563860625,
+          0.017915157601237297,
+          0.0028390432707965374,
+          -0.023165177553892136,
+          0.009495317004621029,
+          -0.01135617308318615,
+          -0.009955224581062794,
+          0.01583496294915676,
+          -0.019316107034683228,
+          -0.018721764907240868,
+          -0.025613298639655113,
+          0.0029416377656161785,
+          -0.003350247861817479,
+          -0.0077618202194571495,
+          -0.0019616815261542797,
+          0.01454722136259079,
+          0.008724087849259377,
+          0.025669902563095093,
+          0.0008326975512318313,
+          0.011971740983426571,
+          0.01511326152831316,
+          -0.00035576484515331686,
+          0.02313687466084957,
+          -0.03229256719350815,
+          0.01246702577918768,
+          -0.004517704714089632,
+          -0.00509789539501071,
+          0.023858575150370598,
+          0.0003747802402358502,
+          -0.0004877670726273209,
+          -0.0024888061452656984,
+          0.0005496777012012899,
+          -0.014589674770832062,
+          -0.0057099261321127415,
+          0.000907432462554425,
+          -0.005175726022571325,
+          0.04299071803689003,
+          -0.00701535539701581,
+          -0.004772422835230827,
+          -0.039169952273368835,
+          -0.01918874867260456,
+          0.00623351288959384,
+          -0.01583496294915676,
+          -0.018778368830680847,
+          -0.006307805422693491,
+          0.005801907740533352,
+          -0.04893413558602333,
+          0.008342010900378227,
+          0.03336804360151291,
+          -0.012219383381307125,
+          0.00259493850171566,
+          -0.010981171391904354,
+          -0.006969364359974861,
+          -0.0029115669894963503,
+          -7.766463386360556e-05,
+          -0.002412744564935565,
+          -0.020830262452363968,
+          -0.005617944523692131,
+          0.003364398842677474,
+          -0.0008795727044343948,
+          -0.025047259405255318,
+          0.0006177793256938457,
+          0.018849123269319534,
+          -0.003979966975748539,
+          -0.0011152751976624131,
+          -0.001825478277169168,
+          0.006399787031114101,
+          -0.01508495956659317,
+          0.008009462617337704,
+          -0.0076344613917171955,
+          0.003841994795948267,
+          -0.0030495391692966223,
+          -0.002214630600064993,
+          0.02844349853694439,
+          0.021594416350126266,
+          -0.04808507859706879,
+          -0.01405901275575161,
+          0.023009516298770905,
+          0.007280686404556036,
+          -0.004740582779049873,
+          0.01178777776658535,
+          -0.031613320112228394,
+          -0.00028943203506059945,
+          -0.026787832379341125,
+          0.02200479619204998,
+          -0.009205221198499203,
+          -0.007386818993836641,
+          0.016967041417956352,
+          -0.026632171124219894,
+          0.016344398260116577,
+          0.00794578343629837,
+          -0.00030181417241692543,
+          -0.000493515923153609,
+          0.0063856360502541065,
+          0.0008972614887170494,
+          -0.01659911684691906,
+          0.006442239973694086,
+          -0.0048502529971301556,
+          0.007959933951497078,
+          -0.005087282508611679,
+          -0.05527378246188164,
+          -0.009933997876942158,
+          -0.005497660953551531,
+          -0.008936353027820587,
+          -0.012820800766348839,
+          0.008136821910738945,
+          0.02827368676662445,
+          0.01606137864291668,
+          -0.009863242506980896,
+          -4.8312376748071983e-05,
+          0.033000119030475616,
+          -0.0008300442132167518,
+          0.009955224581062794,
+          -0.011349096894264221,
+          0.00015709814033471048,
+          -0.0006911876262165606,
+          -0.014660430140793324,
+          -0.019061388447880745,
+          0.024042539298534393,
+          -0.028344441205263138,
+          0.02378782071173191,
+          -0.01843874529004097,
+          -0.0130825936794281,
+          0.01757553406059742,
+          -0.032575588673353195,
+          -0.005108508747071028,
+          0.007450498174875975,
+          0.0090566361322999,
+          0.021608568727970123,
+          0.0032794929575175047,
+          0.020589696243405342,
+          -0.03716050833463669,
+          -0.009134466759860516,
+          -0.010160413570702076,
+          -0.00794578343629837,
+          -0.010231168940663338,
+          -0.011596739292144775,
+          0.021863285452127457,
+          -0.015240620821714401,
+          -0.016952890902757645,
+          0.007054270710796118,
+          0.0030212372075766325,
+          -0.011313719674944878,
+          0.015750056132674217,
+          0.00017810352437663823,
+          0.004712280817329884,
+          0.013011839240789413,
+          -0.0008923970744945109,
+          0.04086806997656822,
+          0.0253019779920578,
+          -0.0030212372075766325,
+          -0.008568426594138145,
+          0.021438755095005035,
+          0.0004285097820684314,
+          0.0150000536814332,
+          0.009375033900141716,
+          0.023264234885573387,
+          -0.02092931978404522,
+          0.009997677057981491,
+          -0.005501199048012495,
+          0.014603826217353344,
+          -0.009495317004621029,
+          -0.015339677222073078,
+          0.00479718716815114,
+          -0.01577835902571678,
+          0.010153338313102722,
+          -0.010860888287425041,
+          0.008221727795898914,
+          0.01913214474916458,
+          0.002467579673975706,
+          -0.00941041111946106,
+          0.01146230474114418,
+          0.02075950801372528,
+          -0.025641601532697678,
+          0.008851447142660618,
+          -0.0031998935155570507,
+          -0.0021916350815445185,
+          0.021127434447407722,
+          0.010301923379302025,
+          -0.01594817079603672,
+          0.01573590561747551,
+          -0.008879749104380608,
+          -0.0015053119277581573,
+          -0.01882082223892212,
+          -0.03291521221399307,
+          -0.0015283073298633099,
+          0.012396270409226418,
+          0.004800724796950817,
+          0.0150000536814332,
+          0.011264191009104252,
+          -0.0008950503543019295,
+          -0.006601438857614994,
+          0.0008433107868768275,
+          -0.013485897332429886,
+          -0.004963461309671402,
+          0.010231168940663338,
+          0.017051948234438896,
+          -0.0037217114586383104,
+          0.015650998800992966,
+          0.0088160689920187,
+          0.006137993652373552,
+          -0.0010480580385774374,
+          0.0036138100549578667,
+          -0.002060738392174244,
+          -0.007783046457916498,
+          0.019938750192523003,
+          0.019075538963079453,
+          -0.018509499728679657,
+          -0.0024870370980352163,
+          -0.02141045406460762,
+          0.025882167741656303,
+          0.022188758477568626,
+          0.011165134608745575,
+          -0.04222656786441803,
+          0.00012050455552525818,
+          0.00752832880243659,
+          -0.0013744152383878827,
+          -0.031103884801268578,
+          0.008172199130058289,
+          0.005221716593950987,
+          0.004330204334110022,
+          0.020844414830207825,
+          -0.018424594774842262,
+          0.010853812098503113,
+          0.0070861103013157845,
+          -0.0005766529939137399,
+          0.01019579079002142,
+          -0.0073372903279960155,
+          0.008688710629940033,
+          0.038717120885849,
+          -0.018410442396998405,
+          0.005303084850311279,
+          0.0062370505183935165,
+          0.023320838809013367,
+          -0.011228813789784908,
+          -0.00031485961517319083,
+          0.00800238735973835,
+          0.021608568727970123,
+          -0.012934008613228798,
+          0.005883275531232357,
+          0.067075714468956,
+          -0.012912781909108162,
+          0.031613320112228394,
+          0.020533092319965363,
+          -0.004970536567270756,
+          -0.0104929618537426,
+          -0.012445799075067043,
+          -0.0181132722645998,
+          0.021283095702528954,
+          0.01175947580486536,
+          0.003419233951717615,
+          0.01941516436636448,
+          -0.01610383205115795,
+          0.006636816076934338,
+          0.017547233030200005,
+          0.012601460330188274,
+          -0.007379743270576,
+          -0.0036173479165881872,
+          0.019004784524440765,
+          0.018509499728679657,
+          -0.0030849166214466095,
+          0.00019424450874794275,
+          0.01508495956659317,
+          -0.022075550630688667,
+          -0.012530704960227013,
+          0.01644345559179783,
+          -0.0037358624394983053,
+          0.017773648723959923,
+          -0.02227366529405117,
+          0.024198200553655624,
+          -0.004531855694949627,
+          -0.017179306596517563,
+          0.004698129836469889,
+          -0.007514177821576595,
+          0.022033097222447395,
+          -0.001578720286488533,
+          0.004857328720390797,
+          0.000821199850179255,
+          -0.0018015984678640962,
+          0.01349297259002924,
+          0.002826661104336381,
+          -0.0005120891146361828,
+          0.010507113300263882,
+          -0.0009702275274321437,
+          -0.0013496510218828917,
+          0.024509521201252937,
+          1.9374700059415773e-05,
+          0.005989408120512962,
+          0.0032936439383774996,
+          -0.009261825121939182,
+          -0.004429261200129986,
+          -0.004443412180989981,
+          0.0011842612875625491,
+          0.003877372248098254,
+          -0.007029506377875805,
+          -0.02384442463517189,
+          0.022471778094768524,
+          -0.008540124632418156,
+          0.01175947580486536,
+          0.011349096894264221,
+          0.003665107535198331,
+          0.006824316922575235,
+          -0.009771261364221573,
+          -0.017320817336440086,
+          0.0006633278680965304,
+          -0.01617458648979664,
+          0.006912760436534882,
+          -0.001656550681218505,
+          0.0020235921256244183,
+          0.009445788338780403,
+          -0.002570174168795347,
+          0.0015123875346034765,
+          0.011306644417345524,
+          0.011497682891786098,
+          0.003254728624597192,
+          0.004577846731990576,
+          -0.018721764907240868,
+          -0.020915169268846512,
+          0.005288933869451284,
+          0.013924578204751015,
+          0.004273600410670042,
+          0.0015309606678783894,
+          0.017603836953639984,
+          -0.006166295614093542,
+          -0.022967062890529633,
+          -0.010117961093783379,
+          -0.01584911346435547,
+          0.039396367967128754,
+          0.01800006441771984,
+          -0.02525952458381653,
+          -0.000530220044311136,
+          -0.009728807955980301,
+          0.010450509376823902,
+          -0.00404364662244916,
+          0.014575524255633354,
+          0.005476434715092182,
+          -0.02426895499229431,
+          -0.016797229647636414,
+          -0.010662773624062538,
+          -0.005257094278931618,
+          0.020080260932445526,
+          0.027014248073101044,
+          -0.004956385586410761,
+          0.0061026159673929214,
+          0.019584976136684418,
+          -0.010938717983663082,
+          0.009849091991782188,
+          0.020957622677087784,
+          0.0031698227394372225,
+          -0.007570781745016575,
+          -0.004553082399070263,
+          -0.016514210030436516,
+          0.024467067793011665,
+          0.014250051230192184,
+          -0.005104971118271351,
+          -0.02465103194117546,
+          0.004110863897949457,
+          -0.02428310550749302,
+          -0.01616043597459793,
+          -0.007301913108676672,
+          -0.014143918640911579,
+          -0.004931621253490448,
+          0.0027594438288360834,
+          -0.004836102016270161,
+          0.011278342455625534,
+          -0.0031592093873769045,
+          -0.004531855694949627,
+          -0.0041851564310491085,
+          -0.025584997609257698,
+          -0.02524537406861782,
+          0.007691065315157175,
+          -0.003005317412316799,
+          0.005596718285232782,
+          0.000650945701636374,
+          -0.013400991447269917,
+          -0.011589664034545422,
+          0.00970758218318224,
+          -0.007988235913217068,
+          0.012813725508749485,
+          0.0031149876303970814,
+          -0.009183995425701141,
+          0.0017193458043038845,
+          -0.0012311364989727736,
+          -0.0059752571396529675,
+          0.019924599677324295,
+          0.00916984397917986,
+          0.011724098585546017,
+          -0.03118879161775112,
+          -0.0021633331198245287,
+          0.008349086157977581,
+          0.01828308403491974,
+          -0.02244347706437111,
+          -0.008313708938658237,
+          0.005179263651371002,
+          -0.011228813789784908,
+          -0.013507124036550522,
+          -0.028952933847904205,
+          -0.0030460015404969454,
+          8.175515540642664e-05,
+          -0.018636859953403473,
+          -0.029886899515986443,
+          0.0009038947173394263,
+          -0.007860877551138401,
+          0.016882136464118958,
+          -0.006643891800194979,
+          0.01790100708603859,
+          0.00870286114513874,
+          0.0036173479165881872,
+          0.011554286815226078,
+          0.0025082635693252087,
+          -0.013266556896269321,
+          0.008426916785538197,
+          -0.02671707607805729,
+          -0.01516986545175314,
+          0.009792488068342209,
+          0.014214673079550266,
+          -0.0026126273442059755,
+          5.074458022136241e-05,
+          0.0058372849598526955,
+          0.024254804477095604,
+          0.013584954664111137,
+          0.012474101036787033,
+          0.003042463678866625,
+          -0.005303084850311279,
+          0.005568416323512793,
+          0.0227123461663723,
+          0.008603804744780064,
+          0.004011806566268206,
+          0.007592008449137211,
+          0.00389859895221889,
+          0.0009932229295372963,
+          -0.005943417549133301,
+          -0.00843399204313755,
+          -0.021537812426686287,
+          -0.008908051066100597,
+          0.019938750192523003,
+          -0.010422207415103912,
+          -0.03005671128630638,
+          0.0006023016758263111,
+          -0.0032777239102870226,
+          0.007988235913217068,
+          0.02595292218029499,
+          -0.014172220602631569,
+          0.004970536567270756,
+          0.005536576267331839,
+          -0.007910405285656452,
+          -0.00614860700443387,
+          0.022967062890529633,
+          0.02639160305261612,
+          -0.0034404604230076075,
+          -0.04800017178058624,
+          -0.01671232469379902,
+          0.004305440001189709,
+          -0.007768895477056503,
+          -0.034047290682792664,
+          -0.015424583107233047,
+          0.002757675014436245,
+          -0.03206615149974823,
+          0.010068432427942753,
+          -0.012948159128427505,
+          -0.01214155275374651,
+          0.020264223217964172,
+          0.03223596513271332,
+          0.005154499784111977,
+          0.00014847487909719348,
+          -0.0214670579880476,
+          -0.0021686397958546877,
+          -0.021382151171565056,
+          0.010174565017223358,
+          -0.008801918476819992,
+          -0.010584943927824497,
+          0.0059611061587929726,
+          -0.012191081419587135,
+          0.013266556896269321,
+          0.010797208175063133,
+          0.024778390303254128,
+          0.01584911346435547,
+          -0.01876421831548214,
+          -0.0024746551644057035,
+          -0.01108730398118496,
+          -0.0016450530383735895,
+          -0.009820790030062199,
+          0.006360871717333794,
+          -0.0031592093873769045,
+          0.011264191009104252,
+          0.015184016898274422,
+          0.004432798828929663,
+          0.00046786724124103785,
+          0.038151077926158905,
+          -2.4487851987942122e-05,
+          -0.0003418349369894713,
+          0.006803090218454599,
+          -0.015750056132674217,
+          0.008837295696139336,
+          0.012191081419587135,
+          0.005051904823631048,
+          -0.007698140572756529,
+          0.006739411037415266,
+          0.03099067695438862,
+          -0.01516986545175314,
+          -0.009014182724058628,
+          -0.023504801094532013,
+          0.007549555506557226,
+          -0.0004656561359297484,
+          0.029971804469823837,
+          0.020533092319965363,
+          -0.027509532868862152,
+          -0.016018925234675407,
+          -0.02179253101348877,
+          0.01267929095774889,
+          -0.011497682891786098,
+          -0.003704022616147995,
+          0.012240610085427761,
+          0.02367461286485195,
+          0.0018838511314243078,
+          -0.014455240219831467,
+          -0.008950503543019295,
+          0.008943428285419941,
+          0.014617976732552052,
+          0.003497064346447587,
+          0.007570781745016575,
+          0.00743634719401598,
+          0.008886824361979961,
+          -0.014801939949393272,
+          -0.012254760600626469,
+          0.00767691433429718,
+          -0.010202866978943348,
+          0.01794346049427986,
+          -0.009403335861861706,
+          0.016768928617239,
+          -0.004645063541829586,
+          -0.011964665725827217,
+          -0.0320095494389534,
+          -0.002292460994794965,
+          0.005037753842771053,
+          0.004627374932169914,
+          0.0034705311991274357,
+          -0.0022217060904949903,
+          -0.013160424306988716,
+          0.004673365969210863,
+          0.005073131527751684,
+          1.875283123808913e-05,
+          -0.008313708938658237,
+          -0.00479718716815114,
+          0.004252373706549406,
+          -0.001031253719702363,
+          -0.0073372903279960155,
+          -0.0059858704917132854,
+          0.018481198698282242,
+          0.0066509670577943325,
+          0.00800238735973835,
+          0.03234917297959328,
+          0.02974538877606392,
+          -0.0016353242099285126,
+          -0.004606148693710566,
+          -0.00023304918431676924,
+          -0.01589156687259674,
+          -0.0057205394841730595,
+          -0.023320838809013367,
+          0.022259512916207314,
+          -0.009665128774940968,
+          0.021820832043886185,
+          0.019457615911960602,
+          0.0039304387755692005,
+          -0.008391539566218853,
+          -0.0011586126638576388,
+          -0.007308988366276026,
+          -0.003560743760317564,
+          0.016641568392515182,
+          0.006742948666214943,
+          0.01226891204714775,
+          0.00030933189555071294,
+          -0.028401045128703117,
+          0.026405755430459976,
+          0.000579748535528779,
+          0.0034015451092272997,
+          0.008908051066100597,
+          0.014129767194390297,
+          0.013209952972829342,
+          -0.01235381793230772,
+          0.01768874190747738,
+          -0.03280200436711311,
+          -0.0033484790474176407,
+          -0.016500059515237808,
+          0.002076658420264721,
+          -0.0018520113080739975,
+          -0.0008145665633492172,
+          -0.05354736000299454,
+          0.0003842879377771169,
+          0.005313698202371597,
+          -0.015339677222073078,
+          0.029349161311984062,
+          -0.020589696243405342,
+          -0.017165156081318855,
+          -0.032858606427907944,
+          -0.010238244198262691,
+          0.018099121749401093,
+          0.005617944523692131,
+          0.00035178486723452806,
+          -0.02660386823117733,
+          0.015750056132674217,
+          -0.0047476585023105145,
+          -0.00743634719401598,
+          -0.029858596622943878,
+          -0.010938717983663082,
+          0.009403335861861706,
+          0.004485865123569965,
+          -0.0130825936794281,
+          0.009898620657622814,
+          -0.002890340518206358,
+          0.010867963545024395,
+          -0.0138467475771904,
+          -0.00420284504070878,
+          0.0010020672343671322,
+          -0.005363226868212223,
+          0.012615610845386982,
+          -0.030537845566868782,
+          0.011391550302505493,
+          -0.01352127455174923,
+          0.0008548084879294038,
+          -0.017858553677797318,
+          -0.027707645669579506,
+          0.0020288988016545773,
+          -0.016132133081555367,
+          0.039990708231925964,
+          0.010188715532422066,
+          -0.01481609046459198,
+          0.016995344310998917,
+          -0.002080196049064398,
+          0.017377421259880066,
+          0.0014955832157284021,
+          0.017023645341396332,
+          -0.011971740983426571,
+          0.021056678146123886,
+          -0.0041745430789887905,
+          0.02156611531972885,
+          0.00600002147257328,
+          0.034047290682792664,
+          -0.02033497765660286,
+          0.0070825726725161076,
+          0.003792466362938285,
+          0.005798369646072388,
+          -0.0026816134341061115,
+          0.021424604579806328,
+          0.017985913902521133,
+          0.019698183983564377,
+          0.0037110981065779924,
+          -0.024084990844130516,
+          -0.006137993652373552,
+          0.004867942072451115,
+          0.023858575150370598,
+          0.011150983162224293,
+          -0.016415152698755264,
+          -0.021056678146123886,
+          0.015226469375193119,
+          -0.007860877551138401,
+          0.006771250627934933,
+          -0.00851889792829752,
+          0.00037544354563578963,
+          -0.013259481638669968,
+          0.012233533896505833,
+          -0.012523629702627659,
+          0.0032335021533071995,
+          0.01043635793030262,
+          -0.027806703001260757,
+          0.01511326152831316,
+          -0.00371817359700799,
+          -0.014313730411231518,
+          0.023320838809013367,
+          -0.015877414494752884,
+          -0.013988257385790348,
+          0.005501199048012495,
+          0.008610880002379417,
+          0.01746232621371746,
+          -0.002761212643235922,
+          0.002669231267645955,
+          -0.020504789426922798,
+          -0.012445799075067043,
+          0.004591997712850571,
+          0.02210385352373123,
+          -0.015820810571312904,
+          0.012431648559868336,
+          -0.018099121749401093,
+          0.01364863384515047,
+          0.016202887520194054,
+          0.007149789482355118,
+          0.00573115237057209,
+          -0.007726442534476519,
+          -0.0011471149045974016,
+          0.032151058316230774,
+          -0.0029876285698264837,
+          0.02704254910349846,
+          -0.014915147796273232,
+          0.009615600109100342,
+          0.00518987700343132,
+          -0.013089669868350029,
+          -0.014589674770832062,
+          0.011108530685305595,
+          0.02428310550749302,
+          -0.009530694223940372,
+          -0.023660462349653244,
+          0.0018484735628589988,
+          0.014264201745390892,
+          0.010337301529943943,
+          0.009403335861861706,
+          -0.001997058978304267,
+          -0.0037358624394983053,
+          0.01484439242631197,
+          -0.0032918748911470175,
+          -0.0286557637155056,
+          -0.01633024774491787,
+          -0.004400959238409996,
+          -0.017490629106760025,
+          0.020745357498526573,
+          0.02367461286485195,
+          0.010507113300263882,
+          0.0037570889107882977,
+          0.005540114361792803,
+          0.018297234550118446,
+          0.01843874529004097,
+          0.013330236077308655,
+          0.009304278530180454,
+          0.012700516730546951,
+          -0.004319590982049704,
+          0.02800481766462326,
+          -0.010860888287425041,
+          -0.01046465989202261,
+          -0.015707602724432945,
+          0.0016636261716485023,
+          -0.005193414632230997,
+          -0.012431648559868336,
+          -0.0069481381215155125,
+          0.004369119182229042,
+          0.013330236077308655,
+          -0.012240610085427761,
+          -0.007149789482355118,
+          0.01016748882830143,
+          0.011971740983426571,
+          0.01403071079403162,
+          0.01719345711171627,
+          -0.01633024774491787,
+          -0.011278342455625534,
+          0.0009834941010922194,
+          -0.02844349853694439,
+          -0.0160047747194767,
+          -0.01767459139227867,
+          0.009580222889780998,
+          -0.014469391666352749,
+          0.03724541515111923,
+          0.028471799567341805,
+          0.0073514413088560104,
+          0.031613320112228394,
+          -0.028401045128703117,
+          0.00970758218318224,
+          -0.014398636296391487,
+          0.007132100872695446,
+          -0.004188694059848785,
+          0.003902136581018567,
+          0.002448122017085552,
+          0.020419884473085403,
+          0.00791748147457838,
+          0.014264201745390892,
+          0.008150972425937653,
+          -0.021934039890766144,
+          -0.00662620272487402,
+          -0.012559006921947002,
+          -0.016401002183556557,
+          -0.02595292218029499,
+          0.013719388283789158,
+          0.005087282508611679,
+          -0.003074303502216935,
+          -0.0002704166399780661,
+          0.006566061172634363,
+          -0.0013195802457630634,
+          0.010443433187901974,
+          -0.027608588337898254,
+          -0.01311089564114809,
+          0.0069269114173948765,
+          0.01449061743915081,
+          -0.02114158496260643,
+          0.0014637433923780918,
+          0.008900974877178669,
+          -0.004602610599249601,
+          0.035660505294799805,
+          -0.014674580655992031,
+          -0.008610880002379417,
+          0.0027806703001260757,
+          0.005062518175691366,
+          -0.017660440877079964,
+          -0.02319347858428955,
+          0.0070648835971951485,
+          0.011575513519346714,
+          0.010867963545024395,
+          -0.028570856899023056,
+          0.02087271586060524,
+          -0.0054339817725121975,
+          0.03832089155912399,
+          0.00401534466072917,
+          -0.013337312266230583,
+          0.003962278366088867,
+          0.0011860302183777094,
+          -0.012360893189907074,
+          0.002060738392174244,
+          -0.000340508297085762,
+          0.003700484987348318,
+          0.011228813789784908,
+          -0.014801939949393272,
+          -0.016967041417956352,
+          0.016910437494516373,
+          -0.011476456187665462,
+          -0.0035395172890275717,
+          -0.00943871308118105,
+          -0.016669871285557747,
+          0.007308988366276026,
+          0.025047259405255318,
+          0.0041745430789887905,
+          0.001242634141817689,
+          -0.011632117442786694,
+          -0.0038596836384385824,
+          -0.008575502783060074,
+          0.0011718792375177145,
+          -0.0207029040902853,
+          0.027396325021982193,
+          -0.01801421493291855,
+          0.002410975517705083,
+          0.021099131554365158,
+          0.008709936402738094,
+          -0.003944589756429195,
+          0.016316095367074013,
+          0.026363302022218704,
+          0.01205664686858654,
+          0.0031539027113467455,
+          -0.00902833417057991,
+          1.648646139074117e-05,
+          0.014957600273191929,
+          0.0015123875346034765,
+          -0.0015663381200283766,
+          0.0032140444964170456,
+          -0.006583749782294035,
+          -0.001859086798503995,
+          -0.017561383545398712,
+          0.005564878229051828,
+          0.02125479280948639,
+          -0.004132090136408806,
+          0.020476488396525383,
+          0.015509488992393017,
+          0.011625041253864765,
+          0.010726453736424446,
+          -0.0012390963966026902,
+          -0.002150951186195016,
+          -0.0021668709814548492,
+          -0.007393894251435995,
+          -0.018297234550118446,
+          0.006470541935414076,
+          0.0034493047278374434,
+          0.01822648011147976,
+          0.020504789426922798,
+          -0.02308027073740959,
+          0.010146263055503368,
+          0.014985902234911919,
+          -0.013011839240789413,
+          0.0235755555331707,
+          -0.02006610855460167,
+          0.0024605041835457087,
+          0.02156611531972885,
+          0.013337312266230583,
+          -0.014271277002990246,
+          -0.009219372645020485,
+          0.004662752617150545,
+          -0.01543873455375433,
+          0.000872939417604357,
+          -0.01478778850287199,
+          -0.010783057659864426,
+          0.014533070847392082,
+          -0.01490099634975195,
+          0.01229721400886774,
+          -0.006969364359974861,
+          0.0070967236533761024,
+          -0.017165156081318855,
+          0.015014204196631908,
+          0.008632106706500053,
+          0.01757553406059742,
+          0.0021279556676745415,
+          -0.02518877014517784,
+          -0.006290116813033819,
+          0.021169887855648994,
+          0.009509467519819736,
+          -0.02497650496661663,
+          0.019613277167081833,
+          0.0020094411447644234,
+          -0.019528372213244438,
+          -0.022401023656129837,
+          -0.012686366215348244,
+          -0.005055442452430725,
+          -0.017264213413000107,
+          -0.0011276573641225696,
+          -0.024792540818452835,
+          0.003972891718149185,
+          0.009106164798140526,
+          -0.011978816241025925,
+          -0.019273653626441956,
+          -0.0002315014135092497,
+          -0.01040805596858263,
+          0.006219361908733845,
+          -0.01132787112146616,
+          -0.019853845238685608,
+          0.0070436573587358,
+          0.012127402238547802,
+          -0.0027647505048662424,
+          -0.005646246485412121,
+          -0.02562745101749897,
+          -0.006919836159795523,
+          0.020179318264126778,
+          -0.003994117956608534,
+          -0.004974074196070433,
+          -0.01703779585659504,
+          0.01108730398118496,
+          0.011165134608745575,
+          0.020901018753647804,
+          0.010931642726063728,
+          0.005674548447132111,
+          0.013634482398629189,
+          -0.0029416377656161785,
+          -0.007156865205615759,
+          -9.645892532716971e-06,
+          0.004284213297069073,
+          0.0069233737885952,
+          0.0016636261716485023,
+          0.020235922187566757,
+          -0.01692458800971508,
+          0.018934030085802078,
+          -0.012290137819945812,
+          0.008794843219220638,
+          -0.01855195313692093,
+          -0.0003155229496769607,
+          -0.01067692507058382,
+          0.00015002264990471303,
+          -0.0038455326575785875,
+          -0.009544845670461655,
+          1.9595809135353193e-05,
+          0.00143986358307302,
+          0.008511822670698166,
+          -0.016783079132437706,
+          0.009615600109100342,
+          -0.010839661583304405,
+          -0.004308977629989386,
+          -0.019542522728443146,
+          0.017448175698518753,
+          0.013344387523829937,
+          0.007308988366276026,
+          -0.008278331719338894,
+          -0.006300730165094137,
+          -0.02282555401325226,
+          0.008582578040659428,
+          -0.0014637433923780918,
+          0.002214630600064993,
+          -0.028301987797021866,
+          -0.004358506295830011,
+          0.0024587351363152266,
+          0.020306676626205444,
+          0.0033661676570773125,
+          -0.010712302289903164,
+          0.008978805504739285,
+          -0.023377442732453346,
+          -0.0030159305315464735,
+          -0.023349139839410782,
+          -0.004075486212968826,
+          -0.005112046375870705,
+          0.010103809647262096,
+          0.008101443760097027,
+          0.01549533847719431,
+          -0.009898620657622814,
+          0.01478778850287199,
+          -0.004793649073690176,
+          0.012049571610987186,
+          -0.0012152165872976184,
+          0.01337976474314928,
+          -0.0003111007681582123,
+          -0.01505665760487318,
+          0.02158026583492756,
+          -0.008738238364458084,
+          0.001741456682793796,
+          -0.0052853962406516075,
+          0.0027346794959157705,
+          -0.017702894285321236,
+          -0.004553082399070263,
+          -0.014455240219831467,
+          0.01957082375884056,
+          -0.022584985941648483,
+          0.004153316840529442,
+          -0.005055442452430725,
+          -0.016075529158115387,
+          0.029434068128466606,
+          0.022896308451890945,
+          -0.0035483618266880512,
+          0.015070809051394463,
+          0.0017989451298490167,
+          0.0015061964513733983,
+          0.016202887520194054,
+          -0.015806660056114197,
+          -0.013025989755988121,
+          0.035066161304712296,
+          0.0176321379840374,
+          0.0004313841927796602,
+          -0.0023260696325451136,
+          -0.0037323245778679848,
+          0.0032193511724472046,
+          0.008023613132536411,
+          -0.010450509376823902,
+          -0.01141277700662613,
+          -0.017051948234438896,
+          0.009997677057981491,
+          -0.006155682262033224,
+          0.004446949809789658,
+          -0.00025891896802932024,
+          -7.617214578203857e-05,
+          -0.01516986545175314,
+          0.022584985941648483,
+          -0.017717044800519943,
+          -0.00602832343429327,
+          0.004687516950070858,
+          0.0207029040902853,
+          -0.02541518583893776,
+          0.01616043597459793,
+          -0.007107336539775133,
+          -0.013429293408989906,
+          0.012573158368468285,
+          -0.015127412974834442,
+          -0.0107335289940238,
+          -0.003175129182636738,
+          -0.008009462617337704,
+          -0.022754797711968422,
+          0.017929309979081154,
+          -0.013627407141029835,
+          0.015650998800992966,
+          -0.00287972716614604,
+          0.000562502013053745,
+          0.002057200763374567,
+          0.0032582662533968687,
+          0.02250008098781109,
+          0.030735958367586136,
+          0.010415131226181984,
+          -0.005465821363031864,
+          -0.021000074222683907,
+          0.012983537279069424,
+          0.0020589695777744055,
+          0.013202877715229988,
+          -0.002253545681014657,
+          -0.002135031158104539,
+          -0.016613267362117767,
+          -0.023122724145650864,
+          -0.0013010069960728288,
+          -0.01761798746883869,
+          -0.02552839368581772,
+          -0.01774534583091736,
+          -0.007047194987535477,
+          0.011349096894264221,
+          0.020405733957886696,
+          0.0009755341452546418,
+          -0.005384453106671572,
+          0.01559439580887556,
+          -0.006941062398254871,
+          0.002080196049064398,
+          -0.006594363134354353,
+          -0.002032436430454254,
+          -0.008200501091778278,
+          -0.021863285452127457,
+          0.029660483822226524,
+          -0.010103809647262096,
+          0.01141277700662613,
+          -0.005472897086292505,
+          -0.00965097825974226,
+          0.005575491581112146,
+          0.00900003220885992,
+          0.024849144741892815,
+          0.006452853325754404,
+          0.03141520544886589,
+          0.020151015371084213,
+          -0.018891576677560806,
+          -0.0006071660900488496,
+          0.022684043273329735,
+          -0.0037570889107882977,
+          -0.004889168310910463,
+          -0.0031786670442670584,
+          -0.017985913902521133,
+          -0.012007118202745914,
+          0.007839650847017765,
+          -0.013471745885908604,
+          -0.006636816076934338,
+          -0.00837738811969757,
+          0.018070818856358528,
+          -0.017773648723959923,
+          0.025698205456137657,
+          -0.006488230545073748,
+          0.005154499784111977,
+          0.031924642622470856,
+          0.014363259077072144,
+          0.019698183983564377,
+          -0.011794853024184704,
+          -0.031075581908226013,
+          -0.00683846790343523,
+          -0.007882103323936462,
+          0.009558996185660362,
+          -0.01062032114714384,
+          0.005204027984291315,
+          -0.02006610855460167,
+          -0.018948180601000786,
+          0.012863253243267536,
+          -0.017476476728916168,
+          0.013924578204751015,
+          -0.009955224581062794,
+          0.027226513251662254,
+          -0.001228483160957694,
+          0.0121698547154665,
+          -0.002614396158605814,
+          -0.017801949754357338,
+          0.01205664686858654,
+          0.004089637193828821,
+          -0.023943481966853142,
+          0.010337301529943943,
+          -0.001864393474534154,
+          -0.014037786051630974,
+          -0.018410442396998405,
+          -0.001228483160957694,
+          -0.005200490355491638,
+          -0.009658053517341614,
+          0.025457637384533882,
+          0.014101465232670307,
+          0.016231190413236618,
+          0.0035359796602278948,
+          0.0016335553955286741,
+          -0.011073152534663677,
+          0.009573147632181644,
+          -0.006803090218454599,
+          0.000612030504271388,
+          -0.0034404604230076075,
+          0.011893910355865955,
+          -0.004252373706549406,
+          -0.01543873455375433,
+          0.002324300818145275,
+          0.008561351336538792,
+          0.0143278818577528,
+          5.947839599684812e-05,
+          0.009587298147380352,
+          0.0009790719486773014,
+          -0.009764186106622219,
+          -0.00046388726332224905,
+          -0.0060884649865329266,
+          0.0069375247694551945,
+          0.02400008589029312,
+          -0.012516554445028305,
+          -0.01681138016283512,
+          0.014504768885672092,
+          0.004033033270388842,
+          -0.017589686438441277,
+          0.018367990851402283,
+          0.004103788174688816,
+          -0.022358570247888565,
+          0.0035412863362580538,
+          0.010662773624062538,
+          -0.009523618966341019,
+          -0.014469391666352749,
+          -0.0010339069413021207,
+          -0.015509488992393017,
+          0.03130199760198593,
+          0.00025073791039176285,
+          0.016910437494516373,
+          -0.01062032114714384,
+          -0.0025967073161154985,
+          -0.013309009373188019,
+          0.013103820383548737,
+          -0.015750056132674217,
+          -0.007273610681295395,
+          0.00035620704875327647,
+          -0.02298121340572834,
+          0.01741987280547619,
+          0.0013425755314528942,
+          0.00794578343629837,
+          0.003436922561377287,
+          -0.010556641966104507,
+          0.009820790030062199,
+          0.0018237093463540077,
+          0.006668656133115292,
+          -0.014915147796273232,
+          0.023618008941411972,
+          0.006413938011974096,
+          0.011235889047384262,
+          -0.01352127455174923,
+          -0.01546703651547432,
+          0.002253545681014657,
+          0.04134920611977577,
+          -0.01306136790663004,
+          0.0002593612007331103,
+          0.01562269777059555,
+          7.081024523358792e-05,
+          0.004156854469329119,
+          -0.0023755980655550957,
+          -0.0034493047278374434,
+          -0.034585028886795044,
+          0.005370302125811577,
+          -0.0006328147719614208,
+          0.008073141798377037,
+          -0.008992956951260567,
+          -0.005430444143712521,
+          -0.0040684109553694725,
+          -0.008625030517578125,
+          -0.00314859626814723,
+          -0.01478778850287199,
+          0.008363237604498863,
+          -0.014533070847392082,
+          -0.028910480439662933,
+          0.03166992589831352,
+          -0.006887996103614569,
+          -0.00533846253529191,
+          -0.021396303549408913,
+          0.007570781745016575,
+          -0.01370523776859045,
+          0.002663924591615796,
+          0.010967019945383072,
+          0.008865597657859325,
+          0.008738238364458084,
+          0.002762981690466404,
+          0.0009516543359495699,
+          -0.004255911335349083,
+          0.010273621417582035,
+          -0.00835616234689951,
+          -0.0005943417199887335,
+          -0.0014186371117830276,
+          -0.02725481428205967,
+          -0.01405901275575161,
+          -0.010592019185423851,
+          -0.007811348885297775,
+          -0.00435143057256937,
+          0.012608535587787628,
+          0.007174553815275431,
+          -0.0017485321732237935,
+          -0.022938761860132217,
+          -0.014745336025953293,
+          -0.0034103896468877792,
+          0.008349086157977581,
+          -0.01838214136660099,
+          0.01064862310886383,
+          0.0016830838285386562,
+          0.009976450353860855,
+          0.014292503707110882,
+          0.00302831269800663,
+          0.0016742395237088203,
+          0.003458149265497923,
+          0.014886845834553242,
+          0.012601460330188274,
+          -0.005801907740533352,
+          -0.021382151171565056,
+          -0.0024605041835457087,
+          -0.007068421691656113,
+          -0.0034687623847275972,
+          -0.012332591228187084,
+          0.03039633482694626,
+          0.0029805530793964863,
+          0.0176321379840374,
+          0.007832575589418411,
+          0.009573147632181644,
+          0.010507113300263882,
+          0.029377464205026627,
+          0.006049550138413906,
+          -0.0029929352458566427,
+          -0.008603804744780064,
+          -0.02141045406460762,
+          -0.013146273791790009,
+          0.009898620657622814,
+          0.013294858857989311,
+          0.03597182780504227,
+          -0.034896351397037506,
+          0.012162779457867146,
+          -0.011101454496383667,
+          -0.008264180272817612,
+          -0.008533049374818802,
+          0.0044717141427099705,
+          0.010068432427942753,
+          0.014370334334671497,
+          -0.0023402206134051085,
+          -0.01175947580486536,
+          0.009070787578821182,
+          0.023547254502773285,
+          -0.012983537279069424,
+          0.03016991913318634,
+          0.027325568720698357,
+          0.023066120222210884,
+          0.01761798746883869,
+          -0.012537780217826366,
+          0.026462359353899956,
+          -6.484030018327758e-05,
+          0.0006261814851313829,
+          -0.018084971234202385,
+          -0.00587973790243268,
+          -0.004294826649129391,
+          -0.004362043924629688,
+          0.013316085562109947,
+          -0.0025471788831055164,
+          -0.0008468485320918262,
+          0.015099111013114452,
+          -0.0037323245778679848,
+          -3.927453508367762e-05,
+          -0.0059752571396529675,
+          0.015127412974834442,
+          0.0002699744363781065,
+          0.019457615911960602,
+          -0.04270770028233528,
+          0.019726485013961792,
+          -0.021396303549408913,
+          -0.003102605463936925,
+          -0.0020890405867248774,
+          0.009523618966341019,
+          0.00764861237257719,
+          -0.007142714224755764,
+          -0.014334957115352154,
+          0.004524780437350273,
+          0.014066088013350964,
+          -0.0006726144347339869,
+          0.005080206785351038,
+          -0.0027541371528059244,
+          0.0017228835495188832,
+          -0.010089659132063389,
+          -0.005862049292773008,
+          5.842812606715597e-05,
+          -0.010570792481303215,
+          -0.008646257221698761,
+          -0.03232087194919586,
+          0.01062032114714384,
+          0.010209942236542702,
+          0.007294837385416031,
+          0.004252373706549406,
+          -0.022018946707248688,
+          -0.012318440712988377,
+          0.012948159128427505,
+          -0.01132787112146616,
+          -0.002570174168795347,
+          -0.0020005968399345875,
+          0.005387990735471249,
+          0.0004705205501522869,
+          0.010160413570702076,
+          -0.003472300246357918,
+          0.010938717983663082,
+          -0.014603826217353344,
+          0.018084971234202385,
+          0.03229256719350815,
+          -0.00533846253529191,
+          -0.019967053085565567,
+          0.000907432462554425,
+          -0.009077862836420536,
+          0.003813692834228277,
+          0.007641536649316549,
+          -0.021962342783808708,
+          -0.02173592709004879,
+          0.005087282508611679,
+          -0.013153349049389362,
+          -0.007783046457916498,
+          0.014830241911113262,
+          0.005221716593950987,
+          0.010839661583304405,
+          0.01751893013715744,
+          0.010889190249145031,
+          0.00840569008141756,
+          -0.026533113792538643,
+          0.0006734988419339061,
+          -0.012686366215348244,
+          -0.00590803986415267,
+          0.002239394700154662,
+          -0.0017591454088687897,
+          0.004591997712850571,
+          -0.00010115749319083989,
+          -0.011200511828064919,
+          -0.0007057808106765151,
+          0.010903340764343739,
+          0.027919910848140717,
+          -0.004146241117268801,
+          -0.0022570835426449776,
+          -0.014257126487791538,
+          0.0010100272484123707,
+          0.036481261253356934,
+          -0.012275987304747105,
+          -0.017434025183320045,
+          0.01757553406059742,
+          0.027452928945422173,
+          0.005954030901193619,
+          -0.0032529598101973534,
+          -0.0027364485431462526,
+          0.00039600671152584255,
+          0.011978816241025925,
+          0.007719367276877165,
+          -0.002692226553335786,
+          -0.002131493529304862,
+          0.004761809483170509,
+          0.035773713141679764,
+          -0.0015203474322333932,
+          -0.010330225341022015,
+          -0.024240652099251747,
+          0.010974096134305,
+          0.00277182599529624,
+          0.004754733759909868,
+          0.002313687466084957,
+          0.023504801094532013,
+          0.019655730575323105,
+          0.014858543872833252,
+          0.00470520555973053,
+          -0.009771261364221573,
+          0.03005671128630638,
+          -0.008172199130058289,
+          -0.004542469047009945,
+          -0.02838689461350441,
+          -0.005087282508611679,
+          -0.015650998800992966,
+          0.008929276838898659,
+          -0.0002221042668679729,
+          -0.005016527138650417,
+          -0.013422218151390553,
+          0.019726485013961792,
+          0.01064862310886383,
+          -0.011073152534663677,
+          0.012608535587787628,
+          -0.027070851996541023,
+          0.02346234768629074,
+          -0.03826428949832916,
+          -0.0071993181481957436,
+          -0.006421013735234737,
+          -0.010514188557863235,
+          -0.02627839520573616,
+          -0.0030353881884366274,
+          0.007372668012976646,
+          0.014250051230192184,
+          0.0022570835426449776,
+          0.0022818478755652905,
+          0.02282555401325226,
+          0.0016415152931585908,
+          -0.01508495956659317,
+          0.017985913902521133,
+          0.03492465242743492,
+          0.012926933355629444,
+          0.00155837822239846,
+          0.015990622341632843,
+          0.0027789014857262373,
+          0.0021633331198245287,
+          0.00788917951285839,
+          -0.0034493047278374434,
+          -0.007938707247376442,
+          0.008582578040659428,
+          0.006990591064095497,
+          -0.02763689123094082,
+          0.003824306186288595,
+          -0.00010408719390397891,
+          -0.009898620657622814,
+          -0.022429324686527252,
+          0.011568437330424786,
+          0.01913214474916458,
+          -0.004224071744829416,
+          -0.006771250627934933,
+          -0.0043337419629096985,
+          0.0006836698739789426,
+          0.006785401608794928,
+          -0.002412744564935565,
+          -0.000515184598043561,
+          -0.0015176940942183137,
+          -0.02368876338005066,
+          -0.010316074825823307,
+          0.013570803217589855,
+          -0.005175726022571325,
+          0.010358527302742004,
+          0.005242943298071623,
+          -0.018792519345879555,
+          -0.016938740387558937,
+          0.006852618884295225,
+          -0.016188737004995346,
+          0.01364863384515047,
+          0.00027948213391937315,
+          -0.0023172253277152777,
+          0.024962352588772774,
+          0.03594352304935455,
+          -0.013188726268708706,
+          0.005119122099131346,
+          -0.019386861473321915,
+          -0.0245378240942955,
+          -0.019004784524440765,
+          0.022613288834691048,
+          0.008306633681058884,
+          -0.018792519345879555,
+          0.012304289266467094,
+          -0.002752368338406086,
+          -0.008865597657859325,
+          -0.0066721937619149685,
+          -0.006845543161034584,
+          -0.008547200821340084,
+          0.020533092319965363,
+          -0.004935159347951412,
+          0.0038950610905885696,
+          0.019047237932682037,
+          0.04075486212968826,
+          -0.008292482234537601,
+          -0.01349297259002924,
+          0.0054339817725121975,
+          0.004878554958850145,
+          0.0018626245437189937,
+          -0.004814875777810812,
+          -0.008016537874937057,
+          -0.0185944065451622,
+          -0.01740572229027748,
+          -0.009615600109100342,
+          -0.0020041344687342644,
+          0.020122714340686798,
+          -0.007768895477056503,
+          -0.01354957651346922,
+          -0.0245378240942955,
+          -0.028188779950141907,
+          -0.006580212153494358,
+          0.02319347858428955,
+          0.017179306596517563,
+          -0.00013200224202591926,
+          -4.3337418901501223e-05,
+          -0.012042495422065258,
+          -0.01968403346836567,
+          0.015509488992393017,
+          0.00037876018905080855,
+          -0.01773119531571865,
+          -0.0065448349341750145,
+          -0.015919867902994156,
+          0.005787756759673357,
+          -0.0051368107087910175,
+          0.004574308637529612,
+          0.05057565122842789,
+          0.0005271245026960969,
+          0.008441068232059479,
+          -0.026575567200779915,
+          0.009106164798140526,
+          -0.01583496294915676,
+          -0.017377421259880066,
+          -0.00012702727690339088,
+          -0.022301966324448586,
+          -0.0030955299735069275,
+          -0.018792519345879555,
+          -0.007386818993836641,
+          0.008568426594138145,
+          0.009799563325941563,
+          -0.013896276243031025,
+          -0.013669860549271107,
+          -0.024693485349416733,
+          -0.0035872769076377153,
+          -0.001539805089123547,
+          0.009311353787779808,
+          0.007825499400496483,
+          -0.03693409264087677,
+          0.002536565763875842,
+          -0.0009507699287496507,
+          0.0005779796629212797,
+          0.01208494883030653,
+          -0.008667483925819397,
+          -0.003063690150156617,
+          -0.0214670579880476,
+          0.006452853325754404,
+          0.007103798910975456,
+          -0.013500047847628593,
+          0.012375044636428356,
+          -0.012863253243267536,
+          -0.007669838611036539,
+          -0.0005426021525636315,
+          0.03379257395863533,
+          0.011794853024184704,
+          -0.0025153392925858498,
+          -0.00654129683971405,
+          -0.011150983162224293,
+          0.02654726430773735,
+          -0.011349096894264221,
+          -0.026405755430459976,
+          0.028910480439662933,
+          0.013245330192148685,
+          0.0005974372616037726,
+          0.020151015371084213,
+          -0.0017936384538188577,
+          -0.01397410687059164,
+          0.009601449593901634,
+          0.004266524687409401,
+          0.012424572370946407,
+          -0.00014460546663030982,
+          0.0214670579880476,
+          0.0018555490532889962,
+          0.0027399861719459295,
+          -0.0036226543597877026,
+          0.004687516950070858,
+          -0.03093407303094864,
+          -0.0006549257086589932,
+          -0.003102605463936925,
+          -0.008398614823818207,
+          0.011207587085664272,
+          -0.002421588869765401,
+          0.006558985915035009,
+          -0.022188758477568626,
+          -0.028485950082540512,
+          0.011342021636664867,
+          -0.016231190413236618,
+          -0.021396303549408913,
+          0.015339677222073078,
+          -0.0007641536649316549,
+          0.00614860700443387,
+          -0.007443422917276621,
+          -0.023278385400772095,
+          -0.019386861473321915,
+          0.02064630016684532,
+          0.012212308123707771,
+          -0.01190098561346531,
+          0.012750045396387577,
+          -0.021113283932209015,
+          -0.012948159128427505,
+          -0.001894464367069304,
+          0.0090566361322999,
+          -0.0019864458590745926,
+          0.01325240544974804,
+          -0.012198156677186489,
+          -0.00334494118578732,
+          0.0031592093873769045,
+          -0.011589664034545422,
+          0.008158047683537006,
+          0.01978308893740177,
+          0.008193425834178925,
+          -0.009339655749499798,
+          0.003422771580517292,
+          -0.011922212317585945,
+          0.013500047847628593,
+          0.015537790954113007,
+          0.018311386927962303,
+          -0.006498843897134066,
+          0.0020412809681147337,
+          -0.01370523776859045,
+          0.0063962494023144245,
+          -0.009827865287661552,
+          -0.004878554958850145,
+          -0.0005333155859261751,
+          0.004921008367091417,
+          -0.011179285123944283,
+          -0.0025772498920559883,
+          -0.026830283924937248,
+          -0.0013823752524331212,
+          -0.020533092319965363,
+          0.005303084850311279,
+          0.023207630962133408,
+          -0.007514177821576595,
+          0.0006045127520337701,
+          -0.0039092120714485645,
+          0.014476466923952103,
+          0.020688753575086594,
+          0.00870286114513874,
+          0.00482548912987113,
+          -0.006081389728933573,
+          0.02179253101348877,
+          0.011603815481066704,
+          -0.017816102132201195,
+          0.00164239970035851,
+          -0.001509734196588397,
+          -0.0014336726162582636,
+          0.011625041253864765,
+          0.004443412180989981,
+          -0.020844414830207825,
+          0.0028602697420865297,
+          -0.010089659132063389,
+          0.007818424142897129,
+          0.014702882617712021,
+          -0.00031906069489195943,
+          -0.02486329711973667,
+          -0.009827865287661552,
+          0.021919889375567436,
+          -0.015070809051394463,
+          -0.0008291598060168326,
+          0.00764861237257719,
+          0.023264234885573387,
+          -0.024792540818452835,
+          -0.006831392180174589,
+          -0.0023455272894352674,
+          -0.003933976404368877,
+          -0.005193414632230997,
+          0.0010489424457773566,
+          0.0214670579880476,
+          0.004415110219269991,
+          -0.004234685096889734,
+          0.005724077112972736,
+          0.006470541935414076,
+          0.02325008250772953,
+          0.0040577976033091545,
+          0.028047269210219383,
+          -0.00943871308118105,
+          -0.0023809047415852547,
+          -0.0019988277927041054,
+          -4.775960405822843e-05,
+          0.0160047747194767,
+          -0.005430444143712521,
+          0.020320827141404152,
+          0.009445788338780403,
+          0.0036934094969183207,
+          0.016514210030436516,
+          0.02319347858428955,
+          -0.006767712999135256,
+          -0.004793649073690176,
+          0.019160445779561996,
+          0.0053667644970119,
+          -0.008858522400259972,
+          -0.02194819226861,
+          0.01141277700662613,
+          0.0014124460285529494,
+          -0.0006956097786314785,
+          0.012106175534427166,
+          0.0024375086650252342,
+          0.006300730165094137,
+          0.008172199130058289,
+          -0.016684021800756454,
+          0.004241760354489088,
+          -0.004698129836469889,
+          -0.01860855706036091,
+          0.010910416021943092,
+          -0.012092024087905884,
+          -5.201595922699198e-05,
+          -0.0010002984199672937,
+          0.014165145345032215,
+          -0.01703779585659504,
+          -0.0019103841623291373,
+          -0.0070719593204557896,
+          -1.3024719009990804e-05,
+          -0.0009162768837995827,
+          0.008320784196257591,
+          -0.006254739128053188,
+          -0.010938717983663082,
+          -0.010938717983663082,
+          0.027509532868862152,
+          -0.004991763271391392,
+          0.003916287794709206,
+          0.019330257549881935
+        ],
+        "embedding": [
+          0.007204997251390758,
+          -0.03173324616100502,
+          -0.020505790884701024,
+          0.0037451608231038273,
+          0.009817085391002229,
+          0.026883177166072393,
+          -0.016477826654817607,
+          0.06949311853773875,
+          -0.01224876667472692,
+          0.031952566873802865,
+          0.013633671951171586,
+          -0.005041437355104989,
+          0.02663392750069342,
+          -0.022099132865267587,
+          0.006665702000169926,
+          0.0030467129715919033,
+          0.007931802508415542,
+          0.009984477355413895,
+          -0.010362836464183985,
+          -0.03180848181251013,
+          -0.013542377333474449,
+          -0.013572345022577053,
+          -0.03979985300694044,
+          -0.007698412130292187,
+          -0.005953452234356532,
+          0.018890228917932694,
+          0.02217446994660065,
+          0.005158357501784182,
+          -0.05468694715835175,
+          0.00789994048345594,
+          0.017130184174786147,
+          -0.0026591475669416114,
+          -0.02081141846859738,
+          -0.008403928318906045,
+          -0.030925346724945978,
+          -0.03855023691059801,
+          0.038853354785097736,
+          0.01612011653434478,
+          -0.035635702395196575,
+          0.026266623757525934,
+          0.043450373922427316,
+          -0.007254334835408145,
+          -0.015376804869420366,
+          -0.0017276202867766907,
+          0.004650414207094434,
+          -0.002365514361535186,
+          0.011906139905776183,
+          -0.0005664194302009289,
+          -0.0191723447249157,
+          -0.025546124518037998,
+          0.01025005257280377,
+          -0.014436532079436437,
+          0.029277529346694118,
+          -0.0045414983847733344,
+          -0.02260315456913697,
+          0.01717918656248626,
+          0.013666132349359846,
+          0.057804090189845364,
+          0.056600576093100735,
+          0.051208022172527734,
+          0.03553760896972431,
+          0.02094599510567085,
+          -0.02438674472832137,
+          -0.0027797634437333432,
+          0.03614248430646102,
+          -0.026729308357113124,
+          -0.002251394534452815,
+          0.018929251842788906,
+          0.00689697483318378,
+          -0.026603606434115974,
+          -0.0541443728344337,
+          0.027164630314707807,
+          -0.004335272389193875,
+          -0.013967321193992958,
+          0.003323121377857836,
+          -0.0245295956412367,
+          0.024489335129134787,
+          0.06212059558640156,
+          0.009214058207162371,
+          -0.002082293435905329,
+          0.013659934201796686,
+          -0.003344037927706573,
+          -0.00689041317919184,
+          0.020235785390418023,
+          0.0212739457082623,
+          0.0016782346585238381,
+          -0.013484221664059555,
+          -0.06465582506604173,
+          0.011882797526607326,
+          -0.032970655964890155,
+          -0.04027700439305653,
+          0.025396847912528105,
+          0.013861157517716707,
+          0.00604596157583048,
+          -0.00744611407420936,
+          0.04038084298991919,
+          0.021507532290328625,
+          0.005983822806623904,
+          -0.012711110369915742,
+          0.02410138142963901,
+          -0.018372791517880082,
+          0.021016674128298246,
+          0.013387285956415756,
+          0.02039383525127615,
+          0.010700292312227496,
+          0.0053934660497349735,
+          -0.014807678014449691,
+          0.009422805142740376,
+          0.0414357625490679,
+          0.12206398267002624,
+          -0.010877509834324155,
+          -0.011079302503786522,
+          -0.04107714853756596,
+          0.006631397986165343,
+          -0.031914267104222384,
+          -0.0179965295457466,
+          -0.057530036803978266,
+          -0.0320488164246567,
+          0.008208202363320509,
+          -0.0020203090915064162,
+          0.03502055780919601,
+          0.04464599164916666,
+          0.019332152079287082,
+          0.017133943087431935,
+          -0.039885565008919065,
+          0.04703059988373572,
+          -0.029370696964304416,
+          0.0008494086835971101,
+          0.0014082189719987074,
+          -0.030973762582071615,
+          -0.006561157852024518,
+          0.011634653995510726,
+          -0.019896826887411713,
+          -0.034275921680252924,
+          0.026274465530039866,
+          -0.007728234125236442,
+          0.04814514040690998,
+          0.06550407296280117,
+          -0.02749051703615343,
+          -0.010041099502187074,
+          0.02019601967763537,
+          -0.01871701432558228,
+          0.008532829759864928,
+          -0.020164138072967613,
+          -0.045196925901010855,
+          -0.004523517988516156,
+          -0.002113469149506399,
+          0.07929650663710812,
+          -0.01111269708042164,
+          0.03358254444720164,
+          -0.019973382759721527,
+          -0.0007145592863325417,
+          -0.0011740862928390678,
+          0.02166182480535788,
+          0.004543228023398081,
+          -0.013377688428761516,
+          -0.01758250073647576,
+          0.009548292014596834,
+          -0.04753617258004735,
+          -0.007125744019792479,
+          -0.009930358894299303,
+          -0.007931672044235996,
+          -0.024561936537429774,
+          0.012979195012881556,
+          0.009209145250061479,
+          0.04586756662401364,
+          0.04311807301838248,
+          -0.0036241105681042675,
+          -0.06579256720896123,
+          0.04967191755416882,
+          0.015475886231466817,
+          0.03157651656768371,
+          0.02515941109300714,
+          0.01773554572598468,
+          0.005824418191900522,
+          0.016317635547562007,
+          0.007650872352957098,
+          0.010015166465492963,
+          -0.033203307142867844,
+          0.040272188258443496,
+          -0.013408423788025047,
+          0.01036004294342104,
+          0.012391622448703885,
+          0.04214857862444935,
+          0.03418032388760535,
+          0.015263018079889338,
+          0.018786704580603317,
+          0.02704685343985817,
+          0.015348111464412869,
+          0.0013031600434754704,
+          0.03241427528588285,
+          0.011689924942195664,
+          0.04544406114742012,
+          0.016186615914968734,
+          0.011624500626103958,
+          -0.008272953078864385,
+          -0.02363080189189375,
+          0.02784298119317616,
+          -0.010585980875564109,
+          0.01455013604305903,
+          0.01749228557615982,
+          -0.054742056843690476,
+          -0.005876213780171779,
+          -0.010556605098730201,
+          0.003967456397644932,
+          0.014663888851767762,
+          0.0015548476559947807,
+          -0.003858367677774835,
+          0.012504286824484335,
+          -0.025806882498628832,
+          -0.014216299453397403,
+          0.0031620924516417956,
+          -0.006448390918604307,
+          -0.02785577502346934,
+          -0.00766402687921159,
+          0.005275000635711884,
+          -0.0090448751942,
+          0.019702662630533,
+          0.007009091837467867,
+          -0.01847307100043082,
+          -0.0006774105206839772,
+          0.032952656101827446,
+          0.015900033044760432,
+          -0.02170121261931376,
+          -0.017964879081469612,
+          -0.005062849745084668,
+          -0.03171847232054206,
+          -0.038177398521606544,
+          0.029426291950046455,
+          -0.025687201683643784,
+          0.030158867757101797,
+          0.015780411479509624,
+          -0.007011484049169482,
+          -0.012114023533118393,
+          0.020661407942701628,
+          -0.025067530380881376,
+          0.009777190971413269,
+          -0.004160771288889298,
+          -0.028825369485727344,
+          0.008307290529469729,
+          -0.014415903546305744,
+          -0.007561235629544945,
+          -0.03201704135271719,
+          -0.015182869388110948,
+          -0.00501561450194676,
+          -0.0003863746169672397,
+          -0.01073403445816663,
+          -0.03321103354401307,
+          -0.002304357889348034,
+          0.017001213960120047,
+          0.00028222146729658097,
+          -0.04002882852650754,
+          0.014509588052058002,
+          -0.004150541038999649,
+          -0.034333837380045466,
+          0.0564137596224324,
+          0.03584898691608747,
+          -0.006830698606679202,
+          0.006121456445207273,
+          -0.04505674213649008,
+          0.04358467937307492,
+          0.010847859073780811,
+          -0.04918983713702805,
+          0.049190442935189385,
+          -0.00528960443684769,
+          0.01419195371129309,
+          0.040316024601793406,
+          0.004600843996974165,
+          -0.006780970176765864,
+          0.005206651850992678,
+          0.03397517915308201,
+          -0.010037223444627569,
+          -0.027377913828275806,
+          -0.03328136706915221,
+          0.02562568440972934,
+          -0.03485770844248482,
+          -0.020069979230687374,
+          -0.0052041408068332524,
+          0.029818953377223505,
+          0.028789460147370508,
+          0.004071274130619537,
+          0.01928375754300958,
+          0.0014970947734202088,
+          0.010502599795027616,
+          0.016914145702515263,
+          0.011942576738188502,
+          -0.014909953528043828,
+          -0.003432489584767786,
+          -0.05733768990013957,
+          0.0030710780388681002,
+          -0.02627280695622545,
+          0.007930906480371135,
+          -0.031227917075485834,
+          0.013132797714872345,
+          0.002514351081823354,
+          -0.007152021475569949,
+          -0.0066245205132327055,
+          -0.005279224195782313,
+          0.028512607222275155,
+          -0.00921960825771865,
+          -0.0359704105589433,
+          -0.021347597388353182,
+          0.011200044645812517,
+          -0.0004171294784927543,
+          -0.016784468873218786,
+          -0.0018746544944553878,
+          0.007749103417240419,
+          0.02390272097796833,
+          -0.005128540108845805,
+          0.006397849195755672,
+          0.01214922062611769,
+          0.01163371266250103,
+          0.012307186722130844,
+          -0.004982589662882946,
+          -0.016343724553037307,
+          0.019806887143115643,
+          0.013139657463054087,
+          0.0205801609265216,
+          -0.03767676007629612,
+          -0.0249879011550972,
+          0.05547392586363707,
+          0.005903313549319051,
+          -0.05568806298774435,
+          0.018827693208840233,
+          -0.007931178941307645,
+          0.008183992600768387,
+          -0.010319022946041053,
+          0.05111352384595183,
+          -0.024897203290807753,
+          -0.02096678205188722,
+          0.006436523335212348,
+          -0.04917084523640266,
+          -0.010459284852442706,
+          -0.011138558192296276,
+          0.014789370043397917,
+          -0.03105725978752438,
+          -0.004934745004818259,
+          0.015595801375817122,
+          -4.687585727052446e-05,
+          0.0028886067775750254,
+          -0.00850352807499054,
+          -0.0034885579171087815,
+          0.016339681487281293,
+          -0.0034050317046826813,
+          -0.008309570448487812,
+          -0.003615165868022203,
+          -0.027638409919916757,
+          0.023352910161586186,
+          0.026364511529991978,
+          0.001249936082895606,
+          0.001987948967948376,
+          0.01756629246122769,
+          0.004656075485833687,
+          -0.006752579977941239,
+          -0.013681608638716132,
+          -0.03405936202918687,
+          -0.0401867651078737,
+          0.022991180183813725,
+          -0.02318908140203705,
+          0.018299512436825923,
+          0.00904081847012441,
+          0.019697091729447287,
+          -0.019236130984371835,
+          0.015159917756323988,
+          -0.0387297415849285,
+          -0.018793498297161176,
+          -0.005256288213225733,
+          -0.010414697711744128,
+          0.0007039894599589104,
+          -0.01739645696111503,
+          -0.013728845328097506,
+          0.022709522955937343,
+          -0.022789243441781803,
+          -0.0036763802589970026,
+          -0.01429671717544519,
+          0.0023674126354206932,
+          0.049448298344058644,
+          -0.009926033487593551,
+          -0.023371285193090733,
+          0.020846792916650956,
+          -0.026996519120163966,
+          0.011246908457705283,
+          0.04091365159640201,
+          0.005469162850197613,
+          0.005640919201612091,
+          0.029575753750229278,
+          -0.0007860452837141593,
+          -0.0052805237594885516,
+          -0.009782745443933828,
+          0.021178503581190618,
+          -0.0009176512672832309,
+          -0.05245650420332918,
+          -0.007698364493410058,
+          -0.039027418406703554,
+          0.005448840777995117,
+          0.017581625031764185,
+          0.025522517097715556,
+          -0.020775492472580457,
+          -0.015906320383873178,
+          -0.000643669839811819,
+          -0.0086896084429604,
+          0.0029399233482444638,
+          0.014381240985647907,
+          0.021086693776922936,
+          0.002436575081260609,
+          -0.004913514868101904,
+          -0.011582101006213432,
+          0.014400297663309155,
+          0.0467181039341492,
+          -0.01722692180137712,
+          0.0011622224560709315,
+          -0.013036540368441797,
+          -0.06208454638917413,
+          -0.019160212331432766,
+          0.0351496273293468,
+          0.02429814055116254,
+          -0.009896662169762772,
+          -0.01647101145477158,
+          -0.002915821099802562,
+          -0.03557863734177486,
+          -0.01786433999615293,
+          -0.010699364747311715,
+          0.02390901423299708,
+          0.007213809581960646,
+          0.0005601364043388203,
+          0.00916614601011734,
+          0.014737491634516238,
+          0.019303599465763203,
+          0.008864535607492504,
+          -0.0575839606069681,
+          0.018051837786406608,
+          0.006315830080225406,
+          0.02525619013082567,
+          -0.015671924543113065,
+          0.003988943108210209,
+          -0.03179878281917948,
+          -0.029440641244429775,
+          0.040586322748158446,
+          -0.03418722657593859,
+          -0.030650542761032924,
+          0.006566745611226466,
+          0.009571089937153852,
+          0.004491158558315032,
+          -0.011352813042040581,
+          -0.02535505987721779,
+          -0.013381948772812133,
+          -0.030918676056735445,
+          -0.02192708443081135,
+          -0.005232667170255381,
+          -0.03342408726436631,
+          -0.07498634115550039,
+          -0.006245638452568239,
+          -0.049946896465756174,
+          -0.03900385398225758,
+          0.04458524125677807,
+          -0.006106489377959481,
+          0.02398410968928338,
+          -0.04615183642824994,
+          -0.01165382599292015,
+          -0.04890013373371541,
+          0.013624324077017377,
+          0.01476398305411351,
+          0.020202554298470666,
+          0.00014553149242532853,
+          0.036224316512987846,
+          0.00021498496267774312,
+          -0.05055208658013767,
+          -0.057482169168904106,
+          -0.003960451575823462,
+          -0.010217078509789989,
+          -0.005593738972150311,
+          -0.034090261748104665,
+          0.05835341009022349,
+          0.0116359661127526,
+          0.04203678184748103,
+          0.03447818335186531,
+          -0.006747564131379555,
+          -0.025756962754841885,
+          -0.028123416424745064,
+          -0.008073625379614702,
+          0.047544241458556825,
+          -0.029605671083240564,
+          0.008171081332993893,
+          -0.026007452205474395,
+          0.0067084589781962805,
+          -0.0070645448617940995,
+          -0.0012298101311792947,
+          0.012372286141738867,
+          0.006377319686763927,
+          -0.015024537877377194,
+          0.007934745664790507,
+          0.026041755081605937,
+          -0.007303731324541215,
+          0.0407000170257006,
+          -0.012990703097937777,
+          -0.033189084013863494,
+          -0.022525679964848666,
+          0.01657717698191603,
+          0.03440284885162039,
+          0.010230633738702899,
+          -0.008707808754455952,
+          -0.0363777020540216,
+          0.011184076269109602,
+          0.031774012431843524,
+          0.01504703222643543,
+          -0.022509833303978505,
+          0.025230974050270107,
+          0.017502662694813216,
+          -0.018700579139537456,
+          -0.03511734747709348,
+          0.015162183119327901,
+          -0.017711638132862063,
+          0.018190335050681287,
+          -0.01590174153215615,
+          0.011103110869700727,
+          0.0025189370696524936,
+          -0.00920305384736575,
+          0.01929652151514404,
+          -0.006196715938358087,
+          0.011426611271994685,
+          0.007058186102245829,
+          -0.018191352599117545,
+          0.0027202185072220937,
+          0.057310566886380025,
+          -0.000531639834386418,
+          0.01494975681909879,
+          0.03845692752400366,
+          -0.001971953300066856,
+          0.045554909803324765,
+          -0.02532332999916258,
+          -0.042616337063845953,
+          0.02633963974172818,
+          -0.02970705577683783,
+          -0.01624154998005974,
+          -0.02620993449678864,
+          -0.021490817420853802,
+          -0.004038424045768319,
+          0.009940887982818171,
+          -0.02692933845395842,
+          -0.023801570856681796,
+          -0.001948460540662095,
+          0.02290911497874141,
+          -0.03300532378029689,
+          0.006525488380791647,
+          0.007039306647666766,
+          -0.003685585408710752,
+          0.03183735852538168,
+          -0.027197064244321705,
+          0.006861928286688311,
+          0.011209869849682412,
+          -0.001094897323854524,
+          -0.005328676992017907,
+          -0.004007474089818478,
+          0.010796946231051235,
+          -0.030543173497906445,
+          -0.012879972141706173,
+          0.00935894527375688,
+          0.0017573071057611829,
+          -0.010050882212831222,
+          -0.03102132152986303,
+          -0.002915141291985554,
+          0.007634137769426993,
+          -0.011094093612593696,
+          0.01199609810193786,
+          -0.010458016758814742,
+          -0.04929347607956551,
+          0.010973132159729983,
+          0.028148117094907874,
+          -0.025504019123994758,
+          0.012470127357818321,
+          0.009182729379865986,
+          -0.016020133756226275,
+          0.016320111225714855,
+          -0.012051271446461976,
+          0.024098195821827578,
+          0.04317620443481784,
+          0.024592886889234396,
+          0.007727787037954994,
+          0.00771746776156926,
+          -0.0003147538723468374,
+          0.022572868970938624,
+          -0.016683340264320543,
+          0.013672085891597506,
+          0.020827021901755415,
+          0.02092309219721837,
+          0.0057130785714458055,
+          -0.003901728981013356,
+          -0.0036968415923910314,
+          0.002227024825579113,
+          -0.02106678364798051,
+          0.014578546755457675,
+          0.03128101864424301,
+          -0.005234627629617124,
+          -0.022394744692442837,
+          0.002430617645746389,
+          -0.015780389432069438,
+          4.7277885648917e-05,
+          0.013357803020920043,
+          0.013777651543690907,
+          -0.027713499926132385,
+          -0.0018999112228756628,
+          0.028364630573666852,
+          0.009146793705085586,
+          -0.009891297866910313,
+          -0.01644269527910728,
+          -0.0003903768506971414,
+          0.006512217314869576,
+          0.02408785204739448,
+          0.0018406499545367782,
+          -0.007360473750031044,
+          0.01484497577088404,
+          0.016723277095327158,
+          0.003629911330658209,
+          -0.0004820567356447713,
+          0.02394406451074397,
+          0.023794019615473026,
+          0.001632266615911801,
+          -0.02064379152040184,
+          -0.02755693521684878,
+          0.027335938400284726,
+          0.02769248863677192,
+          -0.006832353240623308,
+          0.027324973360715134,
+          0.003937207909967385,
+          0.013580154030133622,
+          0.02076818768224654,
+          -0.01784151382870951,
+          -0.017617903734311997,
+          0.0017316476451235418,
+          -0.009605294810027804,
+          -0.009829196696575069,
+          0.004891780008779742,
+          -0.011655402383771472,
+          -0.01124379942906224,
+          0.043205454954628145,
+          0.004713701851783307,
+          0.003560551284131962,
+          0.007894840447501912,
+          -0.018881214526749082,
+          -0.00945538175144614,
+          0.0019040259741796817,
+          0.029505863720301916,
+          -0.001400002868254808,
+          0.03315866433970851,
+          -0.00830542045504805,
+          0.008602408964961391,
+          -0.03202191233287069,
+          -0.006978289755114341,
+          0.0326499698728755,
+          0.015314305362524091,
+          0.015104436300213667,
+          0.0021257042489266787,
+          -0.0010840650756005028,
+          0.02434348355847526,
+          -0.007174433778817674,
+          -0.04839373063710409,
+          0.022696994450018916,
+          -0.004779511451775743,
+          0.023588216956749154,
+          0.0382733980620746,
+          -0.004232286553944853,
+          0.003815507826545929,
+          0.023531223314846965,
+          -0.01887650634213847,
+          0.015055403379782985,
+          -0.01918944132635701,
+          -0.017597900721414636,
+          0.02176630010543207,
+          -0.008896608357798277,
+          0.01732387427588561,
+          0.01764363756727683,
+          -0.012180043444768312,
+          -0.019003360499321047,
+          0.01229139850966224,
+          0.0025710091163655626,
+          -0.017075658234265562,
+          0.01797412486853335,
+          0.018372010308966135,
+          0.006723179570342314,
+          0.015417677746651683,
+          -0.00047634684109332134,
+          0.017608296628666153,
+          -0.001049984114818522,
+          -0.008416990841821452,
+          0.01827827907328822,
+          -0.009568717277044265,
+          -0.01658089563620308,
+          0.009808478422454319,
+          -0.05193977747210182,
+          0.010644335931025957,
+          0.0016781339270411392,
+          -0.01562285100294917,
+          0.00405036330963539,
+          0.007102724536527262,
+          0.005195069570261247,
+          0.0028773630901681767,
+          0.004610474981005765,
+          0.042921486858787584,
+          0.013213672397112759,
+          -0.0064030584930089415,
+          -0.010711457508461704,
+          -0.017480309425763254,
+          -0.027151012983864797,
+          -0.005204425551873271,
+          0.009927884476991038,
+          0.022601230152836282,
+          0.008046297278661014,
+          -0.0031858480011267704,
+          -0.02589628103025927,
+          0.019723430185137954,
+          0.03604835658149313,
+          0.062423099201768716,
+          0.010930268386637612,
+          0.02925564269324381,
+          0.02439517070384333,
+          0.023930432096006114,
+          0.013036828091946605,
+          0.006910311087503408,
+          -0.032995841694293995,
+          0.012813650252889403,
+          0.01333714655191133,
+          0.03678966264802025,
+          0.012571138353467183,
+          -0.05610151838497368,
+          3.0777966587107344e-05,
+          0.009791774904454618,
+          0.041959488802722524,
+          0.01956296954085397,
+          0.000155217569370113,
+          -0.007285602454264675,
+          0.031547870838711856,
+          -0.026518385135879206,
+          -0.012299593280363843,
+          -0.0026551378470968594,
+          -0.006190117573235779,
+          -0.003807907001161504,
+          -0.004991957475076049,
+          0.036983035940964804,
+          -0.015648565314743963,
+          -0.019256463332683493,
+          0.0004300362961407231,
+          -0.0015686065121077472,
+          0.0011311524136286674,
+          0.0032236641835276464,
+          0.013795345278976164,
+          0.015385029265915279,
+          -0.004800448639353721,
+          0.026514926384796363,
+          -0.008699453553651681,
+          0.022186817580012775,
+          -0.013427553742493343,
+          -0.011927259975371873,
+          0.017776228476138103,
+          -0.021243797085501972,
+          -0.01667280589187595,
+          -0.003688045116580565,
+          0.006189970874536939,
+          0.0020911968266404467,
+          0.0006097761986763103,
+          -0.004685468039463817,
+          -0.01011357666327168,
+          0.022631082517271542,
+          0.012398000826637277,
+          -0.006721588925828951,
+          0.018142966317821054,
+          -0.0021330638043572654,
+          -0.011869184647489572,
+          0.025073455250141923,
+          -0.0015392945266926425,
+          -0.0337085890598252,
+          0.01267302113143573,
+          -0.013438601165482187,
+          0.02673290895281225,
+          -0.009586762706902736,
+          -0.011208527168019762,
+          0.019144928964011168,
+          -0.0017591316941248897,
+          -0.005459150067129732,
+          -0.029230079356228025,
+          0.004679497471366242,
+          -0.011723195942933286,
+          0.015899377439148873,
+          0.0022054234777254627,
+          0.029187759252864872,
+          -0.018290694895057157,
+          0.00792533824593637,
+          0.011719198071061566,
+          0.014612375997487872,
+          0.005419943379355563,
+          0.010515265723144554,
+          -0.006893314445458116,
+          -0.04424495426064596,
+          5.8241472764594054e-05,
+          -0.005719143565673274,
+          0.0038220021965914754,
+          0.013106940982933836,
+          -0.01203955335201393,
+          0.011260626096963215,
+          -0.0279896016489392,
+          -0.0295228261396555,
+          0.036474811902902415,
+          -0.04900747926861378,
+          0.019261089742811332,
+          0.00014104646541594166,
+          -0.03057197731717424,
+          -0.04331886560073797,
+          -0.03232957159604667,
+          -0.019401402365464362,
+          0.004102283356333861,
+          0.0002610070704924167,
+          0.0012280372705500212,
+          -0.014025012323929194,
+          -0.014135990809256075,
+          0.013269444858490211,
+          0.00992155580941576,
+          0.011225162609266308,
+          -0.006545044517282357,
+          0.002368862850663672,
+          -0.02063183105457681,
+          -0.026981383275694142,
+          -0.02774194980145225,
+          0.013337303400678109,
+          -0.0025902548982556788,
+          -0.02794339093668326,
+          -0.025606606546251152,
+          -0.0029852498102770587,
+          -0.01906873442473352,
+          0.006508845592362198,
+          0.03268861529625085,
+          0.0355975143313428,
+          0.010066906387952277,
+          0.01690250467284985,
+          -0.017201544758038183,
+          0.048345309114140714,
+          -0.019031171948182872,
+          0.03188329748476941,
+          -0.0005845169448621133,
+          -0.024140202991373587,
+          0.00011597263700677706,
+          0.03203623528903764,
+          -0.019285565857567896,
+          -0.01332755507465857,
+          -0.010914187315481855,
+          -0.003374912271519852,
+          0.02419326154642231,
+          0.02443825785057301,
+          -0.0037923303809811746,
+          0.00022800399604730175,
+          0.015893265651968885,
+          0.003154527829977343,
+          0.007688529734165207,
+          0.030623379985627763,
+          0.014052282574575454,
+          0.01127235345371133,
+          -0.04164707563955819,
+          -0.011811587719246128,
+          -0.003763086694358232,
+          0.005909410979841885,
+          0.002913220159138154,
+          -0.002671613084376487,
+          0.0045765194162907885,
+          -0.007241703279856166,
+          -0.005855949422953439,
+          0.0016973541436777935,
+          -0.001576326424468957,
+          0.008393873711385013,
+          -0.0037217094191171083,
+          -0.03498879678492834,
+          -0.026096985039773694,
+          -0.021280075788049788,
+          0.006437497348154639,
+          0.0070214556591898375,
+          -0.00972721128392027,
+          0.01949553762684561,
+          -0.0029404319702066146,
+          0.000271037702827666,
+          -0.010734867909488861,
+          -0.0029784341515646075,
+          -0.01206464664516104,
+          -0.019544778160755956,
+          0.00035139045159739705,
+          0.013457601448003152,
+          0.00025947376767512006,
+          -0.03165522602526639,
+          0.011979811705146604,
+          0.006642849985850598,
+          0.013181162886675883,
+          -0.024499404540711422,
+          -0.02310714308893576,
+          -0.028028912907092744,
+          0.010570122662701183,
+          -0.0066866321204157725,
+          -0.024617571284817428,
+          -0.004097807033448707,
+          -0.0074576460628928256,
+          -0.0020417256533259887,
+          -0.02470953739335365,
+          -0.011408461668188081,
+          0.023537349682260345,
+          -0.006373652338136973,
+          0.01936338676879376,
+          -0.017636435070085086,
+          -0.017036091876814843,
+          -0.002198685044404603,
+          0.015584758061011048,
+          -0.038943923931820196,
+          -0.002076529978164001,
+          -0.008452513321945455,
+          0.006551055965075869,
+          -0.028840587857625097,
+          -0.007128279504360441,
+          0.0199397499488538,
+          0.012424319366271584,
+          -0.03518875279748495,
+          0.013596027106451115,
+          0.022191403935861113,
+          0.01218507466930854,
+          -0.0038551718784437783,
+          0.03910075115429626,
+          -0.0025215641203030336,
+          -0.005422688956956598,
+          0.016759888275451905,
+          0.04462538468871483,
+          0.023708097134660096,
+          -0.03152743757744915,
+          -0.008668774531128668,
+          -0.015224825785549947,
+          0.004102648962663789,
+          -0.025867795193966987,
+          0.004676262922216584,
+          -0.011739753544166856,
+          0.01645951757031186,
+          0.011253400919230348,
+          0.013958707917471253,
+          0.0376596076542735,
+          -0.005077354977296842,
+          0.02668910096392324,
+          0.0010332284250006117,
+          -0.00856723931523911,
+          0.0014431148305642308,
+          -0.01412579094241589,
+          0.029892393553325205,
+          0.02269599036836208,
+          0.014421409701746036,
+          0.029248195108770955,
+          -0.03588867295960255,
+          0.02458655544632528,
+          -0.0010013539341895672,
+          0.008368648583431747,
+          0.018150734970019018,
+          0.02628022565029391,
+          0.015423867373844644,
+          0.012477080028047823,
+          -0.02513018750675568,
+          -0.022180001014682045,
+          -0.0027550350630288997,
+          -0.031944843677988374,
+          -0.0014668093006914267,
+          -0.009789309083948147,
+          0.026386027599666254,
+          0.0046988411391149,
+          -0.019803160322638583,
+          -0.003931806845108628,
+          0.004482097428470392,
+          -0.00244660100774939,
+          -0.010441732346892935,
+          -0.004649318716095894,
+          -0.01869464992812615,
+          -0.017838643679294407,
+          0.023677878122153204,
+          -0.023958455640817666,
+          -0.025238725541146534,
+          0.014582798102040157,
+          0.0013223590277629197,
+          -0.002295641913710489,
+          0.0008238518461567685,
+          -0.02115533275282518,
+          -0.04214275933987015,
+          -0.027817083427786927,
+          0.0037588803464689412,
+          0.03565056002940463,
+          -0.026788865866928923,
+          0.014162550819567465,
+          -0.011336583458519,
+          0.019175861037658344,
+          -0.005127229083828615,
+          0.015531116410874209,
+          -0.018251290727445846,
+          -0.044630374763679266,
+          0.021833307809708984,
+          -0.012087009688208446,
+          0.02720742725750304,
+          0.021497809277370574,
+          -0.026937653159100156,
+          -0.013572733244188407,
+          0.01358214612023832,
+          0.0020408253706382332,
+          -0.009652962552750464,
+          -0.0063411929391088626,
+          -0.010533309167065193,
+          0.003901271588567942,
+          -0.04496345296706673,
+          -0.008761188928422701,
+          0.06948704056954039,
+          0.005916180375965137,
+          0.03157905519638486,
+          0.014668818318138927,
+          -0.028448496686381258,
+          -0.04008949536631331,
+          -0.00462253589823598,
+          -0.0003701522833020314,
+          -0.00024410762276269235,
+          -0.01850585249309217,
+          0.017699615637510956,
+          -0.002712069912576265,
+          -0.009767338815933488,
+          -0.013119464222297351,
+          -0.013743610418034631,
+          -0.030375365173543157,
+          -0.0030316327877227783,
+          -0.008632689470358149,
+          -0.006565457264917239,
+          -0.0328089275944709,
+          -0.011713996290006943,
+          0.029773910045828737,
+          0.02740100904353622,
+          -0.0061926915497348915,
+          0.00638737826122608,
+          -0.010244154924811975,
+          0.0010396203272677069,
+          -0.001264454445070139,
+          -0.004059969507786729,
+          0.008715730726831364,
+          0.0014767015415289076,
+          0.031074767756160294,
+          0.004761858344125113,
+          0.00987870172044607,
+          -0.059926117793699066,
+          -0.009294021263133636,
+          0.021188432871397475,
+          0.016602826531924377,
+          -0.009125784136292983,
+          0.013946188378030194,
+          -0.0017670794602549428,
+          -0.006837334179368295,
+          -0.028290055353601393,
+          0.024556541126233806,
+          0.04543639597357669,
+          0.0031311343860416473,
+          -0.011199639773247286,
+          0.014965677855595444,
+          0.0018248533506196013,
+          0.015507721118783542,
+          -0.011409402378400194,
+          -0.01338737990904486,
+          0.03063410115541065,
+          0.009305362935335817,
+          0.0025578631174065385,
+          0.012727648343848194,
+          -0.0038680554900795763,
+          -0.01173215289303824,
+          -0.01568862495293563,
+          -0.004101703129244437,
+          -0.03649200408657064,
+          -0.0022107849490954137,
+          0.009659780679763728,
+          0.0034091693454195806,
+          0.008848380026939594,
+          0.01842435611499587,
+          0.03546926214224266,
+          0.013478278853234764,
+          0.005970293629431872,
+          -0.0069197533761813455,
+          0.023196445250565058,
+          0.010403914879205351,
+          0.010017272564548626,
+          0.015076807790462658,
+          -0.029649260409273275,
+          -0.012737152920662164,
+          0.00373276777994811,
+          0.013239909450830258,
+          0.0001682544254742693,
+          0.02030652051051447,
+          0.014143961923579586,
+          0.0010147733405793957,
+          -0.012509446388074923,
+          0.0020437769941487626,
+          0.007405578085797006,
+          0.013458276946120546,
+          0.0006779114540819997,
+          -0.004250745498730309,
+          0.0009149988500358458,
+          -0.008355906072311008,
+          0.040391939952386505,
+          -0.01831218885700067,
+          -0.016793844108223176,
+          0.035064882870542116,
+          0.11416006426189942,
+          0.0010630856112440689,
+          -0.015660014600661857,
+          -0.017142534874931403,
+          -0.0086899013413391,
+          0.016611825572132997,
+          0.01609650977226868,
+          0.009780106186629793,
+          0.009622367309076686,
+          -0.000672926017804395,
+          0.024310949465350565,
+          0.02944698490225121,
+          -0.0037169531700178868,
+          -0.010990838766452194,
+          -0.036819646475041955,
+          0.022923962979125287,
+          -0.004596366586768123,
+          -0.007108041562635549,
+          -0.002604794313552557,
+          0.012641223813290853,
+          0.013668775645501733,
+          0.005125100040230003,
+          0.0077993841846437815,
+          -0.011500851674334447,
+          -0.017334458072700107,
+          0.0056819751688881774,
+          -0.015982234644378358,
+          -0.0076533013358874935,
+          -0.013127172654362694,
+          -0.011837346554842119,
+          -0.006562312362371076,
+          0.0029877420008976425,
+          -0.008572723854050838,
+          0.00854083969164562,
+          -0.010939639783598493,
+          0.016185642759084087,
+          -0.008052282332943615,
+          -0.0011992212936609377,
+          0.0010378646897070758,
+          -0.0025399515501741718,
+          0.010712865639618352,
+          0.01451357949543628,
+          -0.02521401163091652,
+          0.020836357346214075,
+          -0.008252813367155418,
+          0.0006409807249997337,
+          0.015338809580802425,
+          -0.04672526649678653,
+          0.022728876784937357,
+          0.005129225753457052,
+          -0.018372222092794564,
+          0.0066182736895667805,
+          -0.02048375780196863,
+          -0.00896382274306482,
+          0.0021738974344880794,
+          0.004491167150658828,
+          0.0034280799511763956,
+          0.015611133773752265,
+          0.018126362497548308,
+          0.00014383671280220997,
+          0.007927708693242186,
+          -0.006973076292062532,
+          0.008118313055163714,
+          -0.006791836664375275,
+          -0.01164567445668625,
+          -0.010894577039363014,
+          0.00014731542306100506,
+          -0.008865132658571428,
+          -0.011415696755798914,
+          -0.017922318360382588,
+          0.00358538001727405,
+          0.00289327637128695,
+          0.010406054711003452,
+          -0.007757607207626101,
+          -0.010994775479337082,
+          -0.011716096880261664,
+          -0.0013333474316266438,
+          -0.01231144419953554,
+          0.009792603083441917,
+          -0.02884953407490196,
+          0.0029097409399144306,
+          0.025277274938032477,
+          0.009263530315933738,
+          -0.028037993274175277,
+          -0.02532252134134532,
+          -0.0018596440554143638,
+          -0.013847737486669237,
+          0.009910107581119272,
+          0.013952962280235523,
+          -0.011807495289410774,
+          0.003896325037406752,
+          -0.02144921729633576,
+          -0.01628545945817568,
+          -0.0040148671532777005,
+          -0.016032493124255323,
+          0.004580041957001551,
+          -0.011375180364988195,
+          -0.007789996450583554,
+          -0.022041665680945066,
+          -0.005890065942573431,
+          -0.050455058269193866,
+          0.014401045842891128,
+          0.030219548461217365,
+          -0.004580747668649223,
+          -0.028684680969792875,
+          -0.015525870112797619,
+          0.011121401958547706,
+          -0.008879749268609245,
+          -0.019454243014778435,
+          -0.007839434838464604,
+          0.018908534797107714,
+          -0.0019283067432864216,
+          0.0024769961730655835,
+          -0.006732653593696035,
+          0.027604805021030646,
+          0.01366593691107021,
+          0.03740746059041736,
+          -0.014662268336336578,
+          0.02539666271785517,
+          0.02371809783638648,
+          -0.036126848124425154,
+          -0.007627445211958312,
+          0.03657481892016626,
+          0.01092515982671624,
+          0.0011046573579274704,
+          0.012350047525582187,
+          -0.006340802546498992,
+          -0.031330519599469066,
+          -0.0013691360532301728,
+          -0.0014676630169621005,
+          -0.023111265345539388,
+          0.027359543575867225,
+          0.018091458372134273,
+          0.014237156751820934,
+          0.0030993618578507172,
+          -0.0014037082734049385,
+          0.013257269860435359,
+          0.01021315143964527,
+          0.030885417052806473,
+          -0.003225065956381245,
+          -0.012825341172770057,
+          0.013476039084668826,
+          0.01852164843904835,
+          -0.009965495459579544,
+          0.012541079912270015,
+          -0.00824122613985912,
+          -0.018124134280975132,
+          0.03192263766222747,
+          0.0001484308749572035,
+          0.010888733498667343,
+          -0.010649515858317311,
+          -0.013082335214246402,
+          0.032614733822945205,
+          -0.00999075761269065,
+          0.004257643407198955,
+          -0.030723533031261387,
+          -0.003820776505926347,
+          -0.010471792928758869,
+          0.01737608180148319,
+          0.004597677757231315,
+          0.02095148515295301,
+          0.029253627700524396,
+          -0.011604450820008121,
+          -0.01996325509217509,
+          -0.001158626158150122,
+          -0.020583793738536985,
+          0.004168757459099133,
+          -0.03632338333261702,
+          -0.019373660230711246,
+          0.022259216603413172,
+          -0.010364620831688015,
+          -0.004751193403783363,
+          0.00444187509644351,
+          0.0008572202661850934,
+          0.014943779567410408,
+          -0.025408263453313398,
+          0.017755758370543148,
+          0.004627392860417132,
+          0.007402073071525438,
+          -0.024913329140362466,
+          0.013760771254234878,
+          -0.006087346663657887,
+          -0.03780469033982951,
+          0.005509388120163784,
+          0.03756141956381493,
+          0.02381019168406692,
+          -0.024580249929652097,
+          0.01807584361080568,
+          -0.03111039245576191,
+          0.023245542501125073,
+          -0.004868090951368273,
+          5.1544460403943004e-05,
+          0.00640235757127173,
+          -0.004698373089469779,
+          -0.00621383035264185,
+          -0.0025132415458431587,
+          -0.008746634331769936,
+          -0.01611801559493664,
+          -0.01193535623883546,
+          -0.03381789115349148,
+          -0.024119754887378317,
+          0.013881102852385332,
+          0.0019232668966222687,
+          -0.0095263729740736,
+          0.018652966857927805,
+          -0.03775912497511837,
+          -0.012022352025516115,
+          -0.023289377270709736,
+          0.023579672990954228,
+          -0.028605627815260902,
+          0.019235397553197357,
+          -0.025838655372560505,
+          -0.021458526449310157,
+          0.01508157885177254,
+          -0.010064396230734452,
+          -0.0016708624560165368,
+          -0.015042071642751917,
+          -0.03670785416972153,
+          -0.006267614265977843,
+          0.02298763967146357,
+          -0.012506444728541897,
+          0.00029871598584066644,
+          0.02671390618094565,
+          -0.003059865833779805,
+          -0.011103452976820622,
+          -0.0017382684168257506,
+          0.010923055147701782,
+          -0.02938686260440486,
+          0.014300097108092281,
+          0.0038571851420598074,
+          -0.0037614499297117365,
+          0.0010991440540809893,
+          -0.010108413486243776,
+          0.02127952161226619,
+          0.009142545227414532,
+          -0.0030260119510809205,
+          0.040589877142650374,
+          0.008677453889680373,
+          0.0017339508605300814,
+          0.008732950895530873,
+          0.01063352447960556,
+          0.006618635029823979,
+          -0.014986307792626314,
+          0.016905861015681957,
+          0.013101237900847744,
+          0.0002162791516784468,
+          0.019309343995079022,
+          0.0021263376627204313,
+          -0.024516267132902416,
+          0.013339486637573757,
+          -0.02311659751478777,
+          0.0020575982953961246,
+          0.02391695586495566,
+          -0.020440172706421535,
+          -0.001656132876701879,
+          -0.006304133667169861,
+          0.000543457661904495,
+          0.022135707414160034,
+          -0.0230903871986617,
+          -0.023444763940153112,
+          0.009604600033024832,
+          0.01721387169484476,
+          0.01827905276638783,
+          0.02824773106648473,
+          -0.007100625424382998,
+          0.01348510178676837,
+          0.0002604386361122316,
+          -0.011802178189607182,
+          0.007664157334475143,
+          0.0074175377458313594,
+          -0.0044008460682833875,
+          0.025491035291032547,
+          -0.0017015687016376205,
+          -0.006134010425966364,
+          -0.005379657499404915,
+          -0.01584573519366957,
+          0.0034386456145185213,
+          -0.013860742817569676,
+          -0.013355238174583658,
+          0.0040342831827141845,
+          -0.0208160950322342,
+          0.10904592104153767,
+          -0.0037038089210379766,
+          -0.019472012964045247,
+          0.0063554319446707044,
+          0.003716719884684827,
+          -0.003692786470980806,
+          -0.003353843210051379,
+          0.007788605566526459,
+          -0.011254480693190834,
+          -0.0045751956417398095,
+          -0.013976039724381875,
+          -0.001989262807706565,
+          0.010067374586433014,
+          0.009201983771978006,
+          -0.005902980676079493,
+          -0.02029714318363332,
+          0.013031676901335606,
+          -0.016970144750162643,
+          0.008874670945038486,
+          0.0008311255074953139,
+          0.00884167159995975,
+          0.001716603195908536,
+          0.01854951250316076,
+          -0.009310881405486074,
+          -0.01325242738760595,
+          0.029667589863813217,
+          -0.005616511284128333,
+          -0.0038179509972925954,
+          -0.0004690716809300012,
+          -0.005068974756744239,
+          0.0021204231125051274,
+          -0.027958274497127258,
+          0.022127088236172387,
+          -0.017271838492631593,
+          0.010267709087809265,
+          0.01316800560257366,
+          0.005599367128954229,
+          0.01817806970002991,
+          -0.007115594036834107,
+          -0.02627643123342057,
+          0.006542742528003751,
+          -0.003433874053082844,
+          0.006631427284537971,
+          0.02316295746074422,
+          -0.014573351469966297,
+          -0.033520864104177005,
+          -0.013707022143307006,
+          -0.01877574599842729,
+          -0.005992004412448576,
+          -0.0043698881826095435,
+          -0.01481352026460129,
+          0.006424121402963946,
+          0.01382938882387991,
+          0.018136021761515388,
+          0.0012697483790911433,
+          -0.011531136974861559,
+          0.00609187274277241,
+          0.01076133059240578,
+          0.008400887242930113,
+          -0.01533504932731406,
+          0.011439227832642668,
+          -0.028718487721012004,
+          0.01522697877548034,
+          -0.012452665599305595,
+          -0.024938748080573947,
+          -0.0022082450217489366,
+          0.017589399637694264,
+          0.0037824832255756763,
+          -0.004355445812154662,
+          -0.018778668474337727,
+          0.019825087149714933,
+          0.024692878851503508,
+          0.003883497401229232,
+          0.019470881143244197,
+          0.041834635328088364,
+          -0.003544509491805351,
+          0.027139164223766474,
+          -0.02129723143323512,
+          0.01461354860170749,
+          -0.03441729954693155,
+          0.001893713808284104,
+          -0.004222657393633789,
+          0.0013677766871517654,
+          -0.003811928806805752,
+          0.020142937759360904,
+          -0.009158202577174012,
+          0.0030638058587375543,
+          0.024854053046139163,
+          -0.018665378030958756,
+          0.00805903734779081,
+          0.010625429315145887,
+          -0.012856237251902233,
+          0.028535299711292306,
+          -0.006335094045840286,
+          0.00305422126851784,
+          -0.0297883488914759,
+          -0.0029192872993071216,
+          0.0026067044379754047,
+          -0.006097749614529602,
+          -0.00968023644800927,
+          -0.007433916539502931,
+          -0.006202274026852405,
+          0.0038354607448269555,
+          -0.021471361566683354,
+          0.003195646827154078,
+          -0.004390751282171024,
+          -0.003323447423321121,
+          0.02518759723554948,
+          0.025857746688368054,
+          0.016893707217894103,
+          0.012739170140289531,
+          0.008553625305650129,
+          0.019805351588748434,
+          0.01654964927067909,
+          0.011549329129728739,
+          -0.020026013396046632,
+          0.01150644479272541,
+          -0.004159611921476011,
+          -0.0022937297602724912,
+          0.014815466072709131,
+          -0.01572171840394285,
+          -0.02276187015864222,
+          0.024693009795978448,
+          -0.019725315346836404,
+          0.00998768979020729,
+          -1.1702210281215188e-06,
+          -0.011129745730507545,
+          -0.002591293237053776,
+          -0.018034357254990988,
+          0.00012070821244376275,
+          -0.023051045150566527,
+          -0.028465270110388716,
+          -0.00048603008831669926,
+          -0.000574546300981984,
+          -0.012992623897880395,
+          0.02064747165046032,
+          0.014820638761258362,
+          -0.024820174379739057,
+          0.02666518374464728,
+          -0.017430076811109155,
+          -0.01495051200726725,
+          -0.005861181182847845,
+          0.00100881727574559,
+          -0.0055223282819321295,
+          0.0034719859452138017,
+          -0.010180752654175722,
+          -0.009961678082468036,
+          -0.012134423047419183,
+          -0.028073672819317354,
+          0.011574683159318163,
+          0.016120674453124317,
+          -0.015043147648505721,
+          0.031834693135959095,
+          -0.008802805725050891,
+          -0.0018698230152007877,
+          0.016179742266521146,
+          0.005096232332051617,
+          -0.017187077376166815,
+          -0.018999569809271876,
+          0.012251308792871571,
+          0.013279090498325945,
+          -0.009588297039279112,
+          0.006630431200804809,
+          -0.014130176783207386,
+          -0.005092662672039885,
+          0.007020236623084732,
+          -0.005049363863784624,
+          0.0055051712211635915,
+          -0.00995314895304919,
+          0.0052743456988306485,
+          0.005841370337169303,
+          -0.0029434616355351975,
+          0.004677049987338193,
+          -0.018568941264514256,
+          0.020031253265629777,
+          0.0018725390936655962,
+          0.01332532735283085,
+          -0.03518855189467458,
+          -0.0056298723325832025,
+          -0.005369475318146738,
+          -0.004650364775945294,
+          -0.020833510091412353,
+          0.020445280915606132,
+          0.0159991245532085,
+          0.013202409908533162,
+          0.023094507005426774,
+          -0.014905666713082246,
+          0.01572052456975675,
+          -0.020945766915121455,
+          0.013114000985348841,
+          0.003349101658930482,
+          -0.019829605346106578,
+          -0.001423750749031516,
+          0.04557459007627153,
+          -0.005847231973329081,
+          0.011986110554869603,
+          0.0012709842140249691,
+          0.026758922819943853,
+          -0.0001635536762342011,
+          0.0065419519929442995,
+          0.005997692702992206,
+          -0.009935128108338126,
+          -0.0008057798606777003,
+          0.00865712443881613,
+          0.046254320605750854,
+          -0.01181132866321511,
+          0.008170688314239364,
+          0.013099051984057827,
+          0.0034971494510865624,
+          -0.0008386400616584293,
+          -0.015346506644390764,
+          -0.03067678541405639,
+          0.019305962963428017,
+          0.007736916644077731,
+          -0.012756525606221703,
+          0.008734673666457216,
+          0.015137771675476126,
+          0.012891824153867332,
+          0.003895886821685762,
+          0.02155154331885366,
+          -0.012761690789748027,
+          0.018528313792219413,
+          0.024663453412919027,
+          0.009077230086690327,
+          9.184526542175535e-05,
+          -0.000751529849753638,
+          0.006791927267880182,
+          -0.0004433479107716552,
+          -0.009832230053513483,
+          0.015858232096630655,
+          -0.014861341148074698,
+          0.017790924034108806,
+          2.7216806238946785e-05,
+          0.03744841066552559,
+          -0.015216207585985075,
+          -0.010138319613387346,
+          -0.007774325714088762,
+          -0.0016997034352120362,
+          0.003971564314593291,
+          0.0018686537276849653,
+          0.006051669054361943,
+          -0.0008319624938748146,
+          -0.017326561560776794,
+          0.01667737942301944,
+          0.005719808546053257,
+          -0.0007405637872110778,
+          0.013750752065593804,
+          0.0038732323799683453,
+          -0.00367645612814479,
+          0.023580906347714314,
+          0.006238541579764762,
+          -0.005235795826199542,
+          0.0028182323816636243,
+          -0.006443210286549713,
+          -0.016255209299456314,
+          -0.003061441395209219,
+          0.004597970789372988,
+          -0.0016590920570710628,
+          -0.011138261849069419,
+          -0.0019423173609729848,
+          0.020355534503272887,
+          -0.004612410336124759,
+          0.013346750536413418,
+          -0.005400060374036387,
+          0.006252154576727367,
+          0.010601459925435238,
+          -0.012346569079682625,
+          0.0014154620309235822,
+          0.010406076751218612,
+          0.0013983417564564663,
+          -0.007140180075225315,
+          0.0010544410919327675,
+          0.007757599672903709,
+          0.006505350216372801,
+          0.00483063937962671,
+          0.00836630436955631,
+          0.013284451514874498,
+          0.0033749897293287438,
+          -0.006243293993633622,
+          0.0013949831763016042,
+          -0.013269921385800992,
+          -0.006308639321247727,
+          0.0010906169976769569,
+          -0.0010152835709869353,
+          -0.02017142197515583,
+          0.010787630901414118,
+          0.005355784839945777,
+          0.007838485470991034,
+          -0.01706932891696025,
+          0.009576030158827565,
+          -0.0008144311254998525,
+          0.02829832828245172,
+          0.007407953684956539,
+          -0.02769403281764041,
+          0.014266001222243134,
+          -0.010492199740754951,
+          0.022575782293320284,
+          -0.0016349316593207844,
+          -0.005656505606143177,
+          0.008025514398322551,
+          -0.009634281579228592,
+          -0.005991026158213869,
+          -0.022366083313133788,
+          -0.025825209755160946,
+          0.016871087849192924,
+          0.013583173973358155,
+          0.021125252303971605,
+          0.00012555627355712715,
+          0.015347535950809997,
+          0.0020128011217859444,
+          0.015164099887550086,
+          0.0024544903332736167,
+          0.012507136277293888,
+          -0.011798050869794805,
+          0.0104828761281283,
+          -0.0137805761114781,
+          0.014620357150314097,
+          -0.005246639228547017,
+          -0.006986408834884282,
+          -0.008849580196616098,
+          -0.005442165099441538,
+          -0.020891551850452786,
+          -0.009155930286949747,
+          -0.0013590151948670868,
+          0.010665872513236615,
+          -0.0005386423407222223,
+          -0.0027671778465775897,
+          -0.03237311596935737,
+          -0.004875062533405715,
+          -0.006670982068843501,
+          -0.01942244110096111,
+          -0.0045975373348006885,
+          -0.026137783023267008,
+          -0.01657173843643047,
+          -0.0007784046498232684,
+          -0.015037306676062236,
+          -0.006889680260149007,
+          -0.02009705080543122,
+          -0.027192867101205106,
+          3.0161964469866572e-05,
+          -0.009647644858034833,
+          -0.005380165957043753,
+          0.013815092649296373,
+          0.011902366689169847,
+          -0.01572592170758115,
+          -0.022321702398924333,
+          -0.01088937403483412,
+          -0.0165754067023749,
+          0.01646435743868707,
+          0.009914938007210463,
+          0.018834863207293908,
+          -0.003724245912768107,
+          0.005643505694722323,
+          0.0009581443359000331,
+          0.02327704258596552,
+          -0.02532422120223976,
+          -0.028694220894404807,
+          0.0019298778426469338,
+          0.0025970175688368323,
+          0.006086164866310174,
+          -0.009414665394188936,
+          0.001200390869257535,
+          -0.007782899559317819,
+          -0.0228264209954853,
+          -0.00898519580978142,
+          0.008156835270668449,
+          -0.010036630164522355,
+          0.02230664068547531,
+          0.012332230358869292,
+          0.013699197379189934,
+          0.010593587938184292,
+          0.01576174456497367,
+          -0.0001654241831584687,
+          0.014471905257729428,
+          -0.008675915624727887,
+          -0.0031070346099204425,
+          0.0038358404385932777,
+          0.005878417037187274,
+          -0.00625022998235275,
+          0.028077114529659795,
+          -0.0011836453716817525,
+          0.0002962695480296182,
+          0.014405782572641941,
+          0.02697667559474498,
+          -0.004710313427099278,
+          0.0210512043261309,
+          -0.002393165025068831,
+          -0.006135816286212752,
+          -0.008429776018235104,
+          0.008607617299702318,
+          -0.009529291136733401,
+          0.011133497451942684,
+          0.019864404836361926,
+          -0.014041315457579166,
+          -0.007351413712676263,
+          0.02950254974910368,
+          -0.014564388091367897,
+          -0.01825317106331622,
+          0.02325964952838705,
+          -0.007414295676409368,
+          -0.0039042587057605484,
+          0.00377144546378926,
+          0.007169856503233239,
+          -0.004036744966949995,
+          -0.012928341573802416,
+          0.016090527447479415,
+          -0.00011159893712484579,
+          0.014146137810890391,
+          -0.0004655839804281017,
+          -0.006372231942983921,
+          -0.010047382825720416,
+          0.002373224247019062,
+          0.026818813126567614,
+          0.013483627627077692,
+          -0.017895710143841333,
+          -0.013785856764479817,
+          0.010826610341182479,
+          -0.01538893818197323,
+          -0.004425206001945227,
+          0.02155595194812746,
+          0.0028768153668128553,
+          -0.020543027467698483,
+          0.008028975884363618,
+          0.0003835165718138906,
+          0.00037046947047892953,
+          -0.00029363296913641716,
+          0.030135739208689217,
+          0.001343516577762509,
+          0.00018948658255495327,
+          -0.00875053830853609,
+          0.008606573235959595,
+          -0.01280305089741873,
+          0.012786478836532904,
+          -0.02437993139337839,
+          0.007648181775166282,
+          -0.00629541989541625,
+          0.004579605315599212,
+          0.016658386927051592,
+          0.01218098627022323,
+          0.015031618813778413,
+          0.008237200519537417,
+          -0.008942965899756683,
+          0.0009093265012901892,
+          -0.025302175262520477,
+          -0.001333639155622843,
+          -0.017971876771137367,
+          -0.02036433247634014,
+          0.0002463842837918767,
+          0.022144712689630985,
+          0.018559482073124734,
+          0.011579840028464503,
+          -0.008657014440886656,
+          0.0166696911919226,
+          -2.9834620942882283e-05,
+          0.011845827764518911,
+          -0.00037901176785495324,
+          -0.006138374154553732,
+          0.004711058953619512,
+          -0.003622694588720417,
+          0.013542405259906128,
+          -0.014730567462694827,
+          0.012076044672004906,
+          0.01690223095263866,
+          -0.009108632851807137,
+          -0.008946251274440141,
+          -0.013992423874475428,
+          0.00048381889145447367,
+          -0.015909599902931202,
+          0.035489252674454214,
+          0.02252772349379229,
+          -0.01210464970923921,
+          -0.010469120693943236,
+          -0.014164999313775756,
+          0.019496048806752753,
+          0.010427539351244087,
+          0.0028018908451425276,
+          0.017091768406027965,
+          0.021395638416767112,
+          0.008817247593404623,
+          -0.006081063711583319,
+          -0.0021216503501144983,
+          -0.003781810594530787,
+          0.020143098218482827,
+          -0.003078172339607037,
+          -0.011350171884507555,
+          -0.012516379645909518,
+          0.005766102930704529,
+          -0.01626433668691071,
+          0.0019294643928566657,
+          0.008641558521764407,
+          -0.015057688533773374,
+          0.019407986696657772,
+          -0.028126197703647868,
+          0.012364374199657305,
+          -0.0044278434183731225,
+          -0.007251147925369614,
+          -0.03496220945653279,
+          0.006389489026974697,
+          0.008941259259632024,
+          0.003941752277454314,
+          0.023003838655784314,
+          -0.024528514847470928,
+          -0.018361931455105486,
+          0.0037791061321962543,
+          0.00047162060848090104,
+          -0.00711894577052754,
+          0.0010138871688747137,
+          0.004128392769213672,
+          -0.00674546985015164,
+          0.0017307097443508704,
+          0.0033354800298095546,
+          -0.002829898077850263,
+          0.013252947102333348,
+          0.0081724269619999,
+          -0.021449979263734438,
+          0.001368791856860285,
+          0.01590656734732547,
+          0.01452008402429858,
+          -0.02391847163018108,
+          0.006331397860689212,
+          -0.02064812934794864,
+          -0.000631340242718475,
+          -0.031272866000888556,
+          0.016859793097564958,
+          -0.0069008624795683276,
+          0.01899718801899956,
+          -0.0015377436942771224,
+          -0.0065810685260109004,
+          -0.012822620655323715,
+          0.02137604937338179,
+          -0.0019121137078344135,
+          -0.0025598772127157477,
+          0.015678293057066646,
+          -0.025700865215693987,
+          0.017825186616033165,
+          -0.0026230848396158204,
+          -0.017342967698019966,
+          0.017199139867162275,
+          0.004536019178633225,
+          0.007224666811309087,
+          -0.023505589665315567,
+          0.0007208041198255396,
+          0.003900164485347562,
+          -0.017250324025622685,
+          0.004470862192961212,
+          -0.030807400122398566,
+          0.0006579147990327597,
+          -0.013222907940559794,
+          -0.0020514994691204847,
+          -0.0046763235724575445,
+          -0.014050035576183385,
+          -0.039181147615786484,
+          -0.005429639279537522,
+          0.016065742005148733,
+          -0.0009029636981539927,
+          0.01800715198737081,
+          -0.002730931639143803,
+          0.002947680833843522,
+          0.004912692811020822,
+          0.00818378920043427,
+          0.006915895602313347,
+          0.02274116906914758,
+          -0.0057854922886970365,
+          0.00471999900678522,
+          0.007058014277137233,
+          0.014638081356210031,
+          -0.01719565951201012,
+          -0.015170377598372723,
+          -0.016015921063369496,
+          -0.006351213642269936,
+          0.01739665344303718,
+          -0.017488885259028475,
+          0.013303332445465686,
+          -0.00429428898520701,
+          0.014825338166443021,
+          -0.008686427208690176,
+          -0.01575945086724703,
+          0.0028564767084934386,
+          -0.007570417028925067,
+          0.001007670647768344,
+          -0.02323908008484089,
+          0.02019729564706863,
+          -0.00847581038203716,
+          0.010508144473197669,
+          0.01729824052022313,
+          -0.021364529179605813,
+          -0.009742851959432839,
+          -0.014342281441152413,
+          0.029896210049029176,
+          0.013925163184699139,
+          -0.009392252140772913,
+          0.003927230832659267,
+          0.008873561269230905,
+          0.011529536139797032,
+          -0.012209070214071445,
+          -0.0027679974125029362,
+          0.002411914349172495,
+          -0.0062056207925691356,
+          -0.002025687569076897,
+          0.03090134105002816,
+          0.0038285457265343245,
+          0.024877474647546086,
+          -0.013214997316123852,
+          -0.0033763842539861907,
+          0.00023379106197363714,
+          0.005134764372494098,
+          0.01183867975321486,
+          -0.0017966997954747142,
+          -0.008029791503028367,
+          -0.01179500476055557,
+          0.02599911330072515,
+          -0.021852083993691557,
+          0.007024505421035273,
+          0.026784024996764876,
+          0.001390202993990949,
+          0.0029956956045771046,
+          -0.013131691460546865,
+          -0.006298736864509989,
+          0.02523640664202355,
+          0.010030824059858138,
+          0.0041523782603032225,
+          -0.026465819482580182,
+          0.0032128899272377593,
+          -0.0061901409332948115,
+          -0.008681758126710213,
+          0.009866461769956222,
+          -0.0028095454857224164,
+          0.01188012041754054,
+          -0.012191996383095982,
+          0.028459939543805705,
+          0.010711502132045176,
+          0.01047154118931583,
+          0.025888652452698974,
+          -0.02079614305315134,
+          0.0042203106489994615,
+          -0.010829796140534066,
+          0.0006773493086316061,
+          0.005596640511307504,
+          -0.027510562413905437,
+          0.001969898603871003,
+          0.010390866563404346,
+          -0.005642706366801436,
+          -0.01882994179544761,
+          0.01187970346127799,
+          -0.019814385265412387,
+          0.024479579275617887,
+          0.003231341626501863,
+          0.020056914356101588,
+          -0.0053276448188067154,
+          0.009834685132275868,
+          0.0002742942778577165,
+          0.0031361331890882127,
+          -0.0008074598092128465,
+          0.009009012303347016,
+          0.005786777410144894,
+          0.013719989127068968,
+          -0.025528370602188686,
+          0.006821572597883499,
+          -0.0029197127345352626,
+          -0.030539872624857384,
+          -0.013383097680839732,
+          0.010291485622655026,
+          0.012218533188117704,
+          -0.009144425807890594,
+          -0.010378679518611277,
+          0.0030042579827305603,
+          0.005748243116238633,
+          -0.006692403885450407,
+          0.006913672151000981,
+          0.01629516912493545,
+          -0.002533517074771728,
+          -0.0019083178803585312,
+          -0.0035007340321378893,
+          -0.01154609935927537,
+          -0.010016998439072824,
+          0.023193275321990937,
+          -0.00482990342915347,
+          0.017943025772239603,
+          0.025873224580389702,
+          0.0039876567755322,
+          -0.0057577932736143506,
+          -0.005460363755206188,
+          0.01478435999841998,
+          0.014195457012279476,
+          0.004669295894868759,
+          -0.009702220679239768,
+          0.007974305230005956,
+          -0.007760086293456636,
+          0.008375077126777163,
+          -0.020260917110786147,
+          0.002199401871514877,
+          0.0026191373298767766,
+          -0.0022089262577237776,
+          -0.015213807046175918,
+          -0.01956788452420682,
+          -0.021898368819136302,
+          0.012634659463647929,
+          0.00034643102239488474,
+          -0.009241772775582378,
+          -0.013645250997151429,
+          0.0016665106990117016,
+          0.0025630500346914592,
+          0.0002095521560180936,
+          -0.007783022628694597,
+          0.009546147813008391,
+          -0.01265287317178631,
+          0.007445223311736028,
+          -0.004007029181436962,
+          0.0051846833906574515,
+          -0.0016916014415925766,
+          -0.016413938859402438,
+          -0.0210868129634149,
+          0.01920794362224444,
+          -0.012897042227841011,
+          0.02269303737601583,
+          -0.0014935878147854765,
+          3.062680855559304e-05,
+          0.0033132626755401546,
+          0.007274487651476732,
+          0.012110042834251237,
+          0.011426424896797977,
+          0.004307086698938411,
+          0.008206932254416481,
+          0.004673427176334148,
+          -0.013124571552764586,
+          0.004324963759928715,
+          0.0039216033705804084,
+          -0.010449887922072297,
+          0.005298323150690876,
+          -0.0010196552573491,
+          0.006128327172621373,
+          0.012167492341907742,
+          0.01923525204293908,
+          0.01908107550907106,
+          -0.0012031208519043073,
+          -0.0010601912677170903,
+          0.014474374858125607,
+          0.004540954805526089,
+          -0.004883537463424679,
+          -0.002772988629590886,
+          -0.013363713078069854,
+          -0.002572403646338979,
+          0.011541515690646252,
+          -0.02376161747414617,
+          -0.018215231053002656,
+          0.021831219299143226,
+          -0.004991882316041097,
+          0.033796174834149546,
+          -0.013140970283091536,
+          -0.016197177724633614,
+          -0.00464190076219332,
+          0.0026389653040644607,
+          -0.013788049032378498,
+          -0.010300859744205445,
+          0.010545541278382187,
+          0.013848738866656793,
+          0.009903590151120531,
+          -0.0005984085581408575,
+          0.02922816576443645,
+          0.0019900932217491907,
+          0.02245395592422119,
+          0.0018978407151245295,
+          -0.007029294098689854,
+          -0.01440575120349887,
+          0.010840472832356266,
+          0.005529335652955532,
+          0.009050748012161991,
+          0.017791269808484802,
+          0.014502057454389247,
+          0.006265567405788131,
+          0.013928705812292101,
+          0.0012903002372833693,
+          0.0010858377120770652,
+          -0.002355229463060164,
+          -0.01692711767136341,
+          -0.01456786846097011,
+          -0.01627640642218236,
+          0.003873864312554098,
+          0.012620350492241247,
+          0.00525890570023481,
+          0.0007393789113866752,
+          -0.007153210751899716,
+          0.009254566395058412,
+          0.0065087881512835375,
+          -0.012279152412209152,
+          -0.016530068283509593,
+          0.004060671955726967,
+          -0.02210570272789274,
+          0.007849467899009437,
+          0.01554233481902349,
+          0.00219885032769439,
+          -0.002655449524219968,
+          0.0006843524304842362,
+          0.006272944169764281,
+          0.005531927368102191,
+          -0.0036710328198476905,
+          -0.00544113514775027,
+          0.011264661763762068,
+          -0.006379654234440906,
+          0.009537150372938296,
+          -0.00290031548037575,
+          0.03718190949929003,
+          -0.011976080887998194,
+          0.012120408392610773,
+          -0.001751472085435711,
+          0.008481703962951994,
+          0.008067669857949423,
+          0.010772687289903202,
+          -0.003805089040741351,
+          -0.0022164543131735167,
+          -0.008753625032781523,
+          0.006751953442322185,
+          -0.01582177621255226,
+          0.013631301238130705,
+          -0.00030685726238985963,
+          -0.010728958775974582,
+          0.01055472059121978,
+          0.006788316621149835,
+          0.004413382316882087,
+          -0.003262024961860459,
+          -0.01972283203505511,
+          0.014937729522976615,
+          0.0018799533869990667,
+          0.019247008681353865,
+          -0.023276106189004046,
+          0.029043727846427652,
+          -0.02019024213343764,
+          0.023083643395685142,
+          0.01865875504942157,
+          0.001179436167200093,
+          -0.02070329911663634,
+          -0.007618131527214554,
+          0.014871709122115308,
+          -0.02659818889735726,
+          -0.006234719259825804,
+          0.005986829551040766,
+          -0.014639912579831785,
+          0.00037813423546252645,
+          -0.01924149909676064,
+          0.00010832498610532006,
+          0.011013130149723431,
+          -0.011782407678373089,
+          0.00438725832178584,
+          0.01608349750921335,
+          0.019347883652009713,
+          -0.009760017612545174,
+          0.003869246694835168,
+          -0.024240324890794748,
+          0.010958789926998691,
+          0.020511684636649225,
+          0.008883830392748497,
+          -0.01761944522796127,
+          0.021817973960311446,
+          0.010652131228027521,
+          -0.012575679152857703,
+          -0.0012753095453828826,
+          -0.028191274775392804,
+          -0.012421170815054556,
+          -0.02188248722244861,
+          -0.0011670175621882146,
+          -0.014436683528976771,
+          -0.0029783880907228383,
+          0.0009506986717516226,
+          -0.004719165363484719,
+          -0.01179163390107033,
+          -0.019543732196676634,
+          0.016606805875498898,
+          0.0227313413004497,
+          -0.006738735201626204,
+          -0.014547240981002825,
+          0.013866974327185123,
+          0.031694077725890536,
+          -0.016189209297529324,
+          -0.00542110554672736,
+          0.0028061798782555557,
+          -0.012964876141500577,
+          0.018946046750284537,
+          0.0039419760434645495,
+          0.000322115192590907,
+          -0.01991851369210666,
+          0.002489725351392778,
+          0.010835458511052207,
+          0.01684964045708685,
+          0.0005541314524621196,
+          0.010855852688653808,
+          -0.007549882748861744,
+          0.010096800951420057,
+          -0.011832260836468588,
+          -3.6179995603517093e-06,
+          -0.007800366796516945,
+          0.007001049318404058,
+          -0.004424776169203575,
+          0.034356203288485945,
+          -0.02817071276766299,
+          0.004989631403681216,
+          -0.014539644354369607,
+          -0.010888763939716889,
+          -0.0027322086509797843,
+          0.0004865534340981866,
+          -0.010775238981581461,
+          -0.005162220412866082,
+          -0.0011263271110752205,
+          0.00913081242511139,
+          0.005844327320514489,
+          -0.013886529519306775,
+          0.019171827605726503,
+          -0.030984919318656618,
+          0.004469458202753611,
+          -0.008553175891755315,
+          0.010511707988936336,
+          -0.0151185694875566,
+          0.013830711686954931,
+          0.01301959345025159,
+          0.008205984085944315,
+          0.007740367849216328,
+          -0.01532123052721834,
+          -0.01678314222392063,
+          0.001376468747213867,
+          0.0036953018347776244,
+          -0.00931855481896886,
+          -0.033391028922085425,
+          -0.007072913446166969,
+          0.005331350534425606,
+          0.011614453225043,
+          0.008156128603003278,
+          -0.02345206873048703,
+          0.005003870024809975,
+          -0.024178720646704597,
+          -0.012115112684614331,
+          -0.007750557665424973,
+          -0.011580152272614108,
+          -0.006117298954192534,
+          0.026449765548206453,
+          0.007989237291215073,
+          0.016691517078382376,
+          0.004954133565782356,
+          0.02742809127725913,
+          0.011670893621991404,
+          -0.011736840457474895,
+          -0.003349151747880166,
+          0.00035419034843196817,
+          0.005507651806218584,
+          -0.008706590052024251,
+          0.011793241980257321,
+          -0.0013295035978611142,
+          -0.0065061682758755625,
+          -0.004398354347597199,
+          0.009372816666628725,
+          0.014188676762501037,
+          -0.003007980559117429,
+          -0.008724255229333351,
+          0.015749776067989826,
+          -0.030899793617096834,
+          -0.015215033379100091,
+          0.0016152308157299517,
+          -0.007813175908301852,
+          0.017806770680528907,
+          0.01169056345888971,
+          -0.026835412610223673,
+          -0.004538235170332766,
+          0.010532316940606419,
+          0.004755930024691789,
+          0.0095408889025506,
+          0.00954782703036282,
+          0.012079064562163901,
+          -0.007829549014214643,
+          0.01046722658021748,
+          -0.011519369840999348,
+          -0.007092961453184232,
+          -0.007241799314569802,
+          -0.008077934089061052,
+          0.021467547268987214,
+          -0.025816030568238717,
+          -0.006558853543772633,
+          0.01134670818701549,
+          0.013027491377727365,
+          -0.002301991242932548,
+          -0.006299073927206214,
+          -0.017357989759336386,
+          0.008311464809363397,
+          -0.013922568511594478,
+          0.025633907424998367,
+          -0.004854655498817219,
+          -0.005336807899100369,
+          -0.005228056880188717,
+          0.01958682721272445,
+          -0.016774409737540904,
+          0.02072573220949266,
+          -0.014449250679704904,
+          -0.007857351393427215,
+          0.001673929749689194,
+          0.006890008607998747,
+          -0.002948835017234416,
+          0.03734128725793149,
+          -0.00556474133167756,
+          -0.0014544201137417158,
+          0.02209084868209573,
+          -0.01883610080898967,
+          0.017901421953179442,
+          0.01563604483998435,
+          0.012327159100148019,
+          0.0018161034827768648,
+          -0.010253034391420751,
+          0.019979728662282904,
+          0.020823925491027635,
+          0.0139490143219451,
+          0.009577521724445478,
+          -0.013478761809038993,
+          0.014934933784421556,
+          -0.0003725422088504494,
+          -0.0024362115214204105,
+          0.012785534912074068,
+          0.016441944143888537,
+          -0.02164522371903646,
+          -0.013152102018736346,
+          0.0063548333449661805,
+          -0.016547629104776748,
+          -0.024875443975589356,
+          -0.008991095813132973,
+          -0.021885127280977718,
+          -0.023336714382596175,
+          0.00688008599534903,
+          0.016482610587564443,
+          0.0035797836729533367,
+          0.013111012464332918,
+          0.006820471349350837,
+          0.018274678506489214,
+          -0.00980248993077149,
+          -0.01582838461385605,
+          -0.0009454586621769173,
+          -0.0004906493597104733,
+          0.009787688665732045,
+          -0.01835583040075359,
+          -0.013977939077760832,
+          0.004340902272034388,
+          -0.020118484730757877,
+          0.01421736173162507,
+          -0.006428386730058143,
+          0.016756376129345597,
+          -0.0004630315831872392,
+          0.011701948921076007,
+          0.02621773278421472,
+          -0.001391567406453007,
+          0.01262026995055899,
+          0.004696906079976175,
+          -0.0030577364576645635,
+          -0.010624066224426196,
+          0.0009420498021845824,
+          -0.0044079288776429516,
+          -0.0004845606666642234,
+          0.006481113327122954,
+          0.020269778435314,
+          -0.03321232649491074,
+          -0.009425498978427855,
+          0.01246754422779043,
+          -0.001658574527323383,
+          -0.005816985985181387,
+          0.013050062223188965,
+          0.00011195028631856881,
+          0.016573827662919836,
+          -0.0017453203044868104,
+          0.01417371165305189,
+          -0.008165910519539772,
+          -0.0007461950113024975,
+          0.0016492309267666983,
+          0.005076015595439873,
+          0.005829299318055953,
+          0.012146800191035666,
+          0.01148884639506138,
+          -0.021051327863689618,
+          -0.0026357096198729156,
+          0.0036146148489310443,
+          -0.010673282438301611,
+          0.014967444393862347,
+          0.005737933867764028,
+          0.01391411712868596,
+          0.0029556189893790237,
+          0.01167806258173173,
+          -0.024440974650111174,
+          -0.01029944007931845,
+          0.01385509328958837,
+          0.00869671728965227,
+          -0.00998315356917989,
+          -0.016580053751364735,
+          -0.007061005072623325,
+          -0.00477175200214453,
+          -0.0021182980051220428,
+          -0.009871524928312118,
+          -0.0005933941177237234,
+          0.006284920058476413,
+          5.466820734762947e-05,
+          -0.012842295882002122,
+          0.015810530885294202,
+          0.004572385791133295,
+          0.003685966629168175,
+          0.005093714133656017,
+          0.011877541830396259,
+          -0.013130170365831912,
+          0.01258190257102377,
+          0.011433121961378686,
+          0.01142665981860032,
+          -0.013710025169815697,
+          -0.001774030746825879,
+          -0.01109881340948764,
+          0.0065684584345163735,
+          0.009983753894796107,
+          0.005346443477816739,
+          -0.002062387245572686,
+          0.004950044847352647,
+          0.006105199826576752,
+          -0.06391185159544624,
+          -0.02131511418723256,
+          0.0027165737784844413,
+          0.033389688791557785,
+          -0.007167593552251769,
+          -0.0037646833460996797,
+          0.002883510089742145,
+          0.01356887065340832,
+          0.009730171586767032,
+          0.00418566245984083,
+          0.020320934773473808,
+          -0.008814826860651366,
+          -0.003014193747732543,
+          0.001559573791453607,
+          0.003499420226345028,
+          -0.008703837291821111,
+          0.0044844701302276645,
+          0.0053963630238108035,
+          0.027764369548110287,
+          0.001469968540767627,
+          0.015793086297019008,
+          -0.008505290826588508,
+          -0.004075888558568453,
+          -0.009166710762668385,
+          -0.002860047211143503,
+          -0.011127820563172684,
+          0.003937110717404271,
+          0.02054186721680431,
+          -0.006126180963243227,
+          0.009346433819831456,
+          0.01057504086030076,
+          0.0001650851447316893,
+          -0.012033423322524951,
+          0.0049587335923558,
+          -0.0032240390628760535,
+          -0.005568976687946761,
+          -0.0024667040429312737,
+          -0.03474027725033016,
+          0.026855947684394643,
+          -0.004369626674503285,
+          0.009252512430957932,
+          -0.005081834388513357,
+          -0.0037896610731020337,
+          -0.014201609270756739,
+          0.017871088029615106,
+          -0.008249444723433136,
+          -0.020092257756971717,
+          0.009532994544485572,
+          -0.004991989149115383,
+          0.006464417738670472,
+          -0.02643078108339741,
+          0.0029138922065812495,
+          -0.017232410749655366,
+          -0.003999293090834831,
+          -0.006924488971625995,
+          0.01318082244178942,
+          -0.007232695119856389,
+          -0.02660711037908311,
+          0.01979251106803667,
+          -0.019485860712345603,
+          -0.011977904020170568,
+          -0.0177530973032872,
+          -0.014524003566945937,
+          0.004874952064727108,
+          -0.015326044325348929,
+          0.002047771741041311,
+          -0.009697340332342246,
+          -0.003652304998155628,
+          -0.0008878698629674222,
+          0.0017782435347417623,
+          -0.013017776298713044,
+          -0.018272697230002275,
+          -0.01353020425912225,
+          -0.00757239449706798,
+          -0.0008250000619957538,
+          0.01299142281772714,
+          -0.001679606555140765,
+          -0.007329941421303076,
+          0.0044605462460153116,
+          0.010231192277666706,
+          0.0006558644644452857,
+          0.0024231733856020414,
+          -0.010728844680699885,
+          -0.014137434553169443,
+          0.005230053181525096,
+          -0.012086468475542818,
+          -0.009459717916037003,
+          -0.0028185227600210556,
+          0.014984527039312847,
+          0.008189464334213754,
+          -0.009139240774931914,
+          -0.005639157579163322,
+          0.014325949891859377,
+          0.009020303519690971,
+          -0.013310205617445609,
+          0.01567574314514385,
+          -0.025369802629268523,
+          0.021964786021408878,
+          0.030536905527039065,
+          0.004945288795196981,
+          -0.01168303908203322,
+          0.02281128628230144,
+          -0.004072286832145419,
+          0.0028278384348551146,
+          0.01740522162288968,
+          -0.042558465579514926,
+          -0.013146562547084278,
+          0.002903743316339479,
+          -0.015297177099602701,
+          -0.010963312697387696,
+          0.008804938803398296,
+          -0.0026727734429658817,
+          0.005273238046953793,
+          0.004257613402172282,
+          0.03512246556126453,
+          0.002258612643297096,
+          -0.0002885708728599208,
+          -0.0019720190285930487,
+          0.012312613129996533,
+          0.010451003939611476,
+          -0.015556620910136185,
+          -0.02387856865043814,
+          -0.004583090523619507,
+          -0.0037828776396525835,
+          -0.0014588238692793338,
+          -0.004361834167147271,
+          -0.008159679676329573,
+          0.012243319920992457,
+          -0.017279964797769668,
+          -0.013748032758693542,
+          0.018394195628121487,
+          -0.006411113522842359,
+          0.019002611495040334,
+          -0.014970362556522149,
+          -0.0077437581301661946,
+          0.0017311115715410025,
+          0.024439691379769795,
+          -0.025013956798520212,
+          0.007359134924497539,
+          -0.0031406470488477937,
+          0.017574824561755627,
+          0.00419409250132525,
+          -0.0046803792187546656,
+          0.010556972371044426,
+          0.015229789583480183,
+          -0.004608293492221483,
+          -0.014449487985966966,
+          -0.01172850291078532,
+          0.016823024209386785,
+          -0.012304887597699588,
+          0.0011773351576246157,
+          0.002262670226244277,
+          -0.0023810578015456716,
+          0.03477119000959688,
+          -0.03442178291363675,
+          0.004062693040714608,
+          0.0029249583507755944,
+          0.003291349443702718,
+          0.006461525089528308,
+          0.006047777261425826,
+          -0.020642802018978404,
+          0.008916088381876748,
+          -0.03316762661930299,
+          -0.006367254689957291,
+          -0.005404276207844267,
+          0.0009189006423526284,
+          0.009401330264376041,
+          -0.011106443887384533,
+          -0.02259405372588916,
+          -0.006010439054323966,
+          0.0070744551115721095,
+          0.010262607009446879,
+          -0.002856616377476421,
+          0.019719451109535182,
+          0.008426509638139982,
+          0.0015877648339862691,
+          -0.010655792508751512,
+          -0.015527851374834499,
+          -0.015460974859638219,
+          -0.017847462632188386,
+          -0.02236964311096428,
+          0.011841749854626384,
+          -0.00576688153869993,
+          0.009054902130717224,
+          0.014317707346504314,
+          -0.015366657284925976,
+          0.002910211008094564,
+          0.017761685988205086,
+          0.007974806171830457,
+          0.002692237107055498,
+          0.0032405477366987947,
+          0.02685230091638845,
+          -0.0009116884421301454,
+          -0.0032228977180251673,
+          0.003158213042682977,
+          0.015872199488167846,
+          0.0004966905858799325,
+          0.019981401695749582,
+          0.015201538351780685,
+          0.005359075814444414,
+          -0.004668738193285921,
+          0.019853988551405383,
+          -0.009844394286458195,
+          0.013476144827747423,
+          -0.00411789677405284,
+          -0.00830656434595337,
+          -0.008496132520586954,
+          0.006442581751999304,
+          -0.008138786054348571,
+          0.006977337946948795,
+          0.02419128791801387,
+          -0.0005734827808485311,
+          -0.0028842142002271044,
+          0.005503033191779872,
+          -0.01696639779775455,
+          -0.00389835761601045,
+          -0.03366077608692946,
+          -0.009942613075976742,
+          -0.01708630689497001,
+          -0.02910756562213045,
+          0.026536268897199546,
+          0.00219827495662445,
+          0.02586503862171508,
+          -0.01905122645609761,
+          0.008935637454859214,
+          -0.01385806889332683,
+          0.029747093931541325,
+          0.009163959012862627,
+          0.0008611776492791918,
+          0.003598831885031635,
+          -0.02103045281939351,
+          -0.01150941825152158,
+          0.027341774725604333,
+          -0.0006700608710615124,
+          -0.012089044078073347,
+          0.022396400579038037,
+          0.017491565611764764,
+          0.0051760367056280536,
+          -0.00559536785318233,
+          0.01979262595019399,
+          0.002306380546194741,
+          0.011973863166872444,
+          -0.0028136750606972223,
+          0.012911235736067303,
+          -0.01510421118359169,
+          -0.003338602157155174,
+          0.039758808572572274,
+          0.007290642519242007,
+          -0.001802921446521437,
+          -0.013696774990705774,
+          0.003941198181491921,
+          -0.0008308293948698925,
+          0.0200351513103389,
+          -0.00014327587644797734,
+          0.028291423013030337,
+          -0.0001085859276231927,
+          -0.010723064585989723,
+          0.020443848393732245,
+          -0.017860296930396913,
+          0.04540901629084378,
+          -0.018671578423261854,
+          0.000852031295629291,
+          -0.010639853151638835,
+          -0.019017194262730474,
+          0.003861296738239369,
+          0.0010977143012920602,
+          0.0008387482016061222,
+          0.0060155816314618,
+          -0.010780239332444652,
+          0.002152214497259421,
+          0.010556338812873054,
+          -0.027636192450621728,
+          0.0010938992111792307,
+          -0.014385202899929138,
+          -0.01353192238925814,
+          -0.026667778226211785,
+          -0.00017403398236549974,
+          -0.015357674131635848,
+          -0.013956722505869091,
+          -0.01985038526848704,
+          -0.008927903879720431,
+          -0.009580214158077466,
+          -0.0020314441852913386,
+          -0.004755584878514854,
+          -0.00018388548107372635,
+          -0.002918855567842974,
+          -0.0015941852262316128,
+          -0.014230272061190995,
+          0.02133992212037431,
+          0.015283345362445234,
+          0.004642765060980717,
+          0.012829570028541471,
+          0.01708952142208596,
+          -0.004770647543823141,
+          0.006604762788847952,
+          -0.0025369836845081078,
+          0.008368000846861623,
+          0.014785031726639733,
+          0.012074738887486398,
+          0.02124518119032885,
+          -0.015459818988572564,
+          -0.004992865135357168,
+          -0.012722391036055767,
+          -0.012210202603282509,
+          -0.017850065946244383,
+          -0.004300429148015694,
+          -0.0014387298144714717,
+          -0.023128300275416423,
+          0.004468952290723408,
+          -0.0026602931688877443,
+          -0.005768852559887139,
+          0.007815212168719318,
+          0.0005862972228454734,
+          -0.002265719947629544,
+          -0.00019323737674204774,
+          -0.02735820445266249,
+          0.00024009122317215594,
+          -0.00018701993387558173,
+          -0.014663710240203917,
+          -0.00450797561624596,
+          0.010584731989847334,
+          -0.011966527748437568,
+          -0.00602367292291343,
+          0.0111155226518017,
+          0.0015565179432852256,
+          0.006062862961835896,
+          0.016952559697728433,
+          0.0005012157272316869,
+          0.011160668242334889,
+          0.014993782560797772,
+          -0.01591303023837098,
+          -0.015356412848444082,
+          -0.0236025426434855,
+          -0.004514538154596937,
+          -0.009753334017065119,
+          0.007553951535816575,
+          -0.004616595532996391,
+          -0.013023153809236325,
+          0.006040285197946957,
+          -0.0006343451398642784,
+          0.008338564073158257,
+          -0.017608246105292328,
+          -8.346168119689692e-05,
+          -0.020944870308705715,
+          0.009230123321316848,
+          -0.00531237224627929,
+          -0.00445985089300067,
+          0.01410675485522911,
+          0.02895945580213926,
+          -0.007799830351886844,
+          -0.009455128322575888,
+          -0.005663072244883114,
+          0.012817229945992764,
+          0.012474431030896143,
+          -0.004423141749110039,
+          -0.0018294512146253298,
+          -0.012817101093394574,
+          0.0004755435377918658,
+          -0.01129485262081729,
+          0.0062747111125005155,
+          0.017199374798326412,
+          -0.021965117839794063,
+          -0.01296097993683748,
+          -0.016765996091978336,
+          -0.016882855472266523,
+          0.004372970343291638,
+          0.017290565252244258,
+          0.022670788364378462,
+          -0.008456547449078897,
+          -0.019564049850942417,
+          0.006898353991539946,
+          -0.01620125883985687,
+          0.0037108638122592746,
+          0.12887455743463638,
+          -0.0028864852421660366,
+          -0.011487804449328495,
+          -0.022683334123299446,
+          -0.005467192611880829,
+          0.01210557696712196,
+          -0.01555159294496261,
+          0.06709672179195036,
+          -0.015071089552631105,
+          0.004364902817496082,
+          -0.004899872946473657,
+          0.0006310685609937571,
+          -0.017610935588191347,
+          -0.005962991715597585,
+          -0.007768819050529025,
+          -0.02688607975562596,
+          0.000883763319535686,
+          -0.009935313912803587,
+          -0.012883104064962595,
+          0.002126368813790577,
+          -0.0013697265269800346,
+          -0.016131186838419288,
+          0.01575514418037863,
+          -0.020677990768697592,
+          -0.007251917191831732,
+          -0.0029325623780992118,
+          -0.00600710212646933,
+          0.014573093764769918,
+          -0.03253336117394778,
+          0.002973725493330224,
+          -0.013601417220439602,
+          0.011905788192118131,
+          0.001542117126832946,
+          -0.010789486980776022,
+          -0.013862139885514524,
+          0.007086571810314664,
+          0.015658913897173155,
+          0.017039624764452546,
+          0.01901374401756777,
+          0.004797954569087795,
+          0.003928999581093576,
+          -0.008046722893614655,
+          0.021974371139905104,
+          -0.00697335955527197,
+          0.0008975473383295958,
+          -0.00970589295353857,
+          -0.004346869073499034,
+          0.0031279458769925416,
+          0.02303983493541673,
+          -0.012560380384977343,
+          -0.01536378830925542,
+          0.0012035987157572355,
+          0.033313215063595594,
+          -0.010845084209743817,
+          0.013587126023659577,
+          0.0029867620644300073,
+          0.020679673435988446,
+          -0.0006640787253300063,
+          0.0025071780490108306,
+          0.022554193713122565,
+          -0.018082906254599396,
+          0.018230000220500718,
+          -0.004451773302171552,
+          0.01915083025790679,
+          -0.004183521043835777,
+          0.015019320101074747,
+          -0.027014055979204272,
+          -0.005362044589023088,
+          -0.012355559881051253,
+          0.003995309003198048,
+          0.020680455787871383,
+          0.0054391755324872375,
+          0.009865676762244514,
+          -0.01645386866657314,
+          -0.01760657627715638,
+          0.022969828924328126,
+          -0.021566687194482292,
+          -0.015826424849757402,
+          0.00012804774676298237,
+          -0.015021970627042762,
+          0.006000610438452896,
+          0.010126179023323158,
+          -0.02912121211344025,
+          -0.02046150879033127,
+          0.004881850039543051,
+          -0.0023469840669773042,
+          -0.00915786433296249,
+          0.006547377770403066,
+          -0.022329063637464074,
+          -0.015702777677743352,
+          -0.005692904224111613,
+          0.01207999389934878,
+          -0.008429160038192635,
+          0.010075061963131637,
+          -0.020106712664936504,
+          0.0036787045650556907,
+          -0.002764965946892891,
+          0.010307702251974629,
+          0.004697169247387421,
+          0.011192285826515207,
+          0.0030113901179382946,
+          0.0014774681536380047,
+          -0.02778270000997313,
+          0.010043261430740039,
+          -5.178406450222847e-05,
+          0.04216636255860709,
+          0.030977096295679525,
+          0.004479857028809617,
+          -5.798020745008712e-05,
+          -0.009996471945562616,
+          0.012476009017187798,
+          -0.019306907391548302,
+          -0.005092322343599383,
+          0.01856463597985508,
+          0.02195467141594795,
+          -0.02236232264139305,
+          0.0052217593371508145,
+          -0.016766955100332014,
+          0.016133299654723637,
+          -0.011208763137897244,
+          0.006248024409576809,
+          0.011601988603146743,
+          0.0022330878390492527,
+          0.010317781892717903,
+          -0.005895142237225119,
+          0.019670196605183973,
+          0.01186677200787666,
+          -0.002339367025714548,
+          0.02367408853110795,
+          0.006858134385159451,
+          0.004479683673962342,
+          0.004798719889930319,
+          -0.009604173975576537,
+          -0.009045602998213713,
+          0.009230969488015385,
+          0.004083616606774546,
+          -0.007944628589259331,
+          0.011836323287097144,
+          0.015100350195476968,
+          0.01885317701149666,
+          -0.0011707847499298092,
+          -0.00684548433712413,
+          -0.006550059418141426,
+          0.001899758393077906,
+          -0.024577381680573596,
+          -0.02855511854789239,
+          0.021238935710272536,
+          -0.010427135883958709,
+          -0.019187769881105637,
+          -0.0015864640188683347,
+          0.016360037306061392,
+          -0.008853972319021563,
+          -0.01627167079143546,
+          0.001270414822216867,
+          0.02255755055961612,
+          -0.005959411125391464,
+          0.006009654905468478,
+          0.005909233773418581,
+          0.005874361519568704,
+          -0.0027859463941992753,
+          -0.024599789577567636,
+          0.0036612374968223453,
+          0.03221346865585554,
+          0.016461563205582293,
+          0.016139782455492073,
+          -0.007158168059808459,
+          0.001838455290310879,
+          0.007530954381521755,
+          0.005438791523412295,
+          0.015674000912615536,
+          0.008604106101542998,
+          -0.006990462920227135,
+          0.00244736628387359,
+          0.012213048255418866,
+          -0.007329299946651028,
+          0.017424711328323798,
+          0.001973882532418776,
+          -0.0133178791530314,
+          -0.008373452013951822,
+          -0.010462150707776556,
+          -0.014575483571750984,
+          0.0027958704799249897,
+          -0.011401916623878957,
+          0.0008016041012669959,
+          0.003953958717263787,
+          0.010260753041492368,
+          0.013961649189650642,
+          -0.0019275791443041316,
+          0.017019168629298462,
+          0.011534594243860177,
+          -0.0017859096307142257,
+          -0.002970739203219076,
+          -0.03142778833370656,
+          0.010207789755749909,
+          -0.014730553506704018,
+          -0.0037611416875830746,
+          0.01825790347981639,
+          -0.007766890847082858,
+          -0.0032778433418881666,
+          -0.021003405429807113,
+          0.0024512381432602736,
+          0.008768616692363321,
+          0.01558713989712387,
+          -0.012819542714293536,
+          0.009201155584074711,
+          -0.0012254337002053017,
+          -0.0057451574939563045,
+          0.00977251392462179,
+          0.0042356831792011935,
+          -0.0033433823868435786,
+          0.011499764125503874
+        ],
+        "title": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer",
+        "keyphrases": [
+          "Sparsely-Gated Mixture-of-Experts Layer",
+          "conditional computation",
+          "neural network",
+          "language modeling",
+          "machine translation"
+        ]
+      },
+      "type": "document"
+    },
+    {
+      "id": "1e687808-8db8-4320-805d-0282a8d679bc",
+      "properties": {
+        "page_content": "Deepseekmoe: Towards Ultimate Expert Specialization In\n\nMixture-Of-Experts Language Models\n\nDamai Dai∗1,2, Chengqi Deng1, Chenggang Zhao∗1,3, R.X. Xu1, Huazuo Gao1, Deli Chen1, Jiashi Li1, Wangding Zeng1, Xingkai Yu∗1,4, Y. Wu1, Zhenda Xie1, Y.K. Li1, Panpan Huang1, Fuli Luo1, Chong Ruan1, Zhifang Sui2, Wenfeng Liang1 1DeepSeek-AI 2National Key Laboratory for Multimedia Information Processing, Peking University 3Institute for Interdisciplinary Information Sciences, Tsinghua University 4National Key Laboratory for Novel Software Technology, Nanjing University {daidamai, szf}@pku.edu.cn, {wenfeng.liang}@deepseek.com https://github.com/deepseek-ai/DeepSeek-MoE\n\nAbstract\n\nIn the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top- out of experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into ones and activating from them, allowing for a more flexible combination of activated experts; (2) isolating experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5× expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations.\n\nFurther, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\n1. Introduction\n\nRecent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023a). It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017) has emerged as a popular solution. It can\n\nenable parameter scaling, while concurrently keeping computational costs at a modest level.\n\nRecent applications of MoE architectures in Transformers (Vaswani et al., 2017) have yielded successful attempts at scaling language models to a substantial size (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022), accompanied with remarkable performance. These achievements underscore the considerable potential and promise of MoE language models.\n\nDespite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts.\n\nThis architecture manifests two potential issues: (1) Knowledge Hybridity: existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. (2) Knowledge Redundancy: tokens assigned to different experts may require common knowledge. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters.\n\nThese issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models.\n\nIn response to the aforementioned issues, we introduce DeepSeekMoE, an innovative MoE architecture specifically designed towards ultimate expert specialization. Our architecture involves two principal strategies: (1) Fine-Grained Expert Segmentation: while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. (2) Shared Expert Isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.\n\nThrough compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized.\n\nStarting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B (Lepikhin et al., 2021) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5× expert parameters and computation. Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE.\n\nThese studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization.\n\nLeveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B (DeepSeek-AI, 2024), a dense model trained on the same 2T corpus. We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B (Touvron et al., 2023b), which has approximately 2.5 times the activated parameters. Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard1. Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. The experimental results still validate its substantial advantages over the GShard architecture consistently. In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nOur contributions are summarized as follows: - Architectural Innovation. We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation.\n\nEmpirical Validation. We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. Experimental results validate the high 1https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models\n\nScalability. We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B.\n\nAlignment for MoE. We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B.\n\nPublic Release. In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization.\n\n2. Preliminaries: Mixture-Of-Experts For Transformers\n\nWe first introduce a generic MoE architecture commonly used in Transformer language models. A standard Transformer language model is constructed by stacking layers of standard Transformer blocks, where each block can be represented as follows:\n\n$$\\mathbf{u}{1:T}^{l}=\\text{Self-Att}\\left(\\mathbf{h}{1:T}^{l-1}\\right)+\\mathbf{h}{1:T}^{l-1},\\tag{1}$$ $$\\mathbf{h}{t}^{l}=\\text{FFN}\\left(\\mathbf{u}{t}^{l}\\right)+\\mathbf{u}{t}^{l},\\tag{2}$$\n\nwhere denotes the sequence length, Self-Att(·) denotes the self-attention module, FFN(·) denotes the Feed-Forward Network (FFN), u 1: ∈ R× are the hidden states of all tokens after the -th attention module, and h\n\n∈ Ris the output hidden state of the -th token after the -th Transformer block. For brevity, we omit the layer normalization in the above formulations.\n\nA typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022). An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. Then, each token will be assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts. If the -th FFN is substituted with an MoE layer, the computation for its output hidden state h\n\nis expressed as: where denotes the total number of experts, FFN(·) is the -th expert FFN, , denotes the gate value for the -th expert, , denotes the token-to-expert affinity, Topk(·, ) denotes the set comprising highest affinity scores among those calculated for the -th token and all experts, and e\n\nis the centroid of the -th expert in the -th layer. Note that ,is sparse, indicating that only out of gate values are nonzero. This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only experts. Also, in the above formulations, we omit the layer normalization operation for brevity.\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{N}\\left(g_{i,t}\\operatorname{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\operatorname{Topk}({s_{j,t}|1\\leqslant j\\leqslant N},K),\\ 0,&\\text{otherwise},\\end{cases}$$ $$s_{i,t}=\\operatorname{Softmax}{i}\\left(\\mathbf{u}{t}^{l}\\,\\mathbf{e}_{i}^{l}\\right),$$\n\n$$(3)$$\n\n(4) $\\binom{4}{5}$ .\n\n3. Deepseekmoe Architecture\n\nOn top of the generic MoE architecture outlined in Section 2, we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. As illustrated in Figure 2, our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. Both of these strategies are designed to elevate the level of expert specialization.\n\n3.1. Fine-Grained Expert Segmentation\n\nIn scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts.\n\nIn pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. The finer expert segmentation enables a more flexible and adaptable combination of activated experts. To be specific, on top of a typical MoE architecture shown in Figure 2(a), we segment each expert FFN into smaller experts by reducing the FFN intermediate hidden dimension to 1 times its original size. Since each expert becomes smaller, in response, we also increase the number of activated experts to times to keep the same computation cost, as illustrated in Figure 2(b). With the fine-grained expert segmentation, the output of an MoE layer can be expressed as:\n\n$$\\mathbf{h}{t}^{i}=\\sum{i=1}^{\\text{mN}}\\left(g_{i,t}\\text{FFN}{i}\\left(\\mathbf{u}{t}^{i}\\right)\\right)+\\mathbf{u}{t}^{i},\\tag{6}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\text{Tokr}(s_{i,t}|1\\leqslant j\\leqslant m\\text{N}),m\\text{K}),\\ 0,&\\text{otherwise},\\end{cases}$$ (7) $$s_{i,t}=\\text{Softmax}{i}\\left(\\mathbf{u}{t}^{T}\\mathbf{e}_{t}^{i}\\right),\\tag{8}$$ where the total number of expert parameters is equal to $N$ times the number of parameters in a\n\n$$(6)$$\n\n(7) (8) $\\frac{1}{2}$\n\nstandard FFN, and denotes the total number of fine-grained experts. With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to .\n\nFrom a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. As an illustrative example, we consider the case where = 16. A typical top-2 routing strategy can yield 16 2 = 120 possible combinations. By contrast, if each expert is split into 4 smaller experts, the fine-grained routing strategy can yield 64 8 = 4, 426, 165, 368 potential combinations. The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition.\n\n3.2. Shared Expert Isolation\n\nWith a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. This alleviation of redundancy will contribute to a more parameterefficient model with more specialized experts.\n\nTowards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate experts to serve as shared experts. Regardless of the router module, each token will be deterministically assigned to these shared experts. In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by , as depicted in Figure 2(c). With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows:\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{K_{s}}\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)+\\sum_{i=K_{s}+1}^{mN}\\left(g_{i,t}\\;\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},\\tag{9}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\mathrm{Topk}{s_{j,t}|K_{s}+1\\leqslant j\\leqslant mN},mK-K_{s}},\\ 0,&\\text{otherwise},\\end{cases}$$ (10) $$s_{i,t}=\\mathrm{Softmax}{i}\\left(\\mathbf{u}{t}^{l,T}e_{i}^{l}\\right).\\tag{11}$$ $\\mathbf{h}\\mathbf{M}\\mathbf{E}$ the number of closed-compact is $K_{s}$ the total number of routed currents.\n\nFinally, in DeepSeekMoE, the number of shared expert is , the total number of routed experts is − , and the number of nonzero gates is − .\n\nIt is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022). The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint.\n\n3.3. Load Balance Consideration\n\nAutomatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. Firstly, there is a risk of routing collapse (Shazeer et al., 2017), i.e., the model always selects only a few experts, preventing other experts from sufficient training.\n\nSecondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.\n\nExpert-Level Balance Loss. In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. The computation of the balance loss is as follows:\n\n$$\\begin{array}{l}{{{\\mathcal L}{\\mathrm{ExpBal}}=\\alpha{1}\\sum_{i=1}^{N^{\\prime}}f_{i}P_{i},}}\\ {{f_{i}=\\frac{N^{\\prime}}{K^{T}}\\sum_{t=1}^{T}\\mathds{1}\\,(\\mathrm{Token}\\;t\\;\\mathrm{s}\\mathrm{lects}\\;\\mathrm{Expert}\\;i),}}\\ {{P_{i}=\\frac{1}{T}\\sum_{t=1}^{T}s_{i,t},}}\\end{array}$$\n\n(12) $$\\begin{array}{l}\\small\\text{(13)}\\end{array}$$ = (14) $$\\begin{array}{l}\\small\\text{(14)}\\end{array}$$ .\n\n7 where 1 is a hyper-parameter called expert-level balance factor, ′is equal to ( − ) and ′ is equal to ( − ) for brevity. 1(·) denotes the indicator function.\n\nDevice-Level Balance Loss. In addition to the expert-level balance loss, we introduce a devicelevel balance loss. When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. Instead, our primary objective is to ensure balanced computation across the devices. If we partition all routed experts into groups {E1, E2, ..., E}, and deploy each group on a single device, the device-level balance loss is computed as follows:\n\n$$\\mathcal{L}{\\text{DevBal}}=\\alpha{2}\\sum_{i=1}^{D}f_{i}^{\\prime}P_{i}^{\\prime},$$ $$f_{i}^{\\prime}=\\frac{1}{|\\mathcal{E}{i}|}\\sum{j\\in\\mathcal{E}{i}}f{j},$$ $$P_{i}^{\\prime}=\\sum_{j\\in\\mathcal{E}{i}}P{j},$$\n\n(15) $$\\begin{array}{l}\\small\\text{(16)}\\end{array}$$ = (17) .\n\nwhere 2 is a hyper-parameter called device-level balance factor. In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices.\n\n4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n6. Alignment For Deepseekmoe 16B\n\nPrevious research indicates that MoE models typically do not emerge significant gains from fine-tuning (Artetxe et al., 2022; Fedus et al., 2021). However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B.\n\n6.1. Experimental Setup\n\nTraining Data. For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios.\n\nHyper-Parameters. During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).\n\nWe employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10−5 without incorporating any learning rate scheduling strategy.\n\nEvaluation Benchmarks. For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3, with the following adjustments: (1) We exclude Pile (Gao et al., 2020) since chat models are seldom employed for pure language modeling. (2) We exclude\n\nMetric # Shot LLaMA2 DeepSeek DeepSeekMoE SFT 7B Chat 7B Chat 16B # Total Params N/A 6.7B 6.9B 16.4B # Activated Params N/A 6.7B 6.9B 2.8B FLOPs per 4K Tokens N/A 187.9T 183.5T 74.4T HellaSwag (Acc.) 0-shot 67.9 71.0 72.2 PIQA (Acc.) 0-shot 76.9 78.4 79.7 ARC-easy (Acc.) 0-shot 69.7 70.2 69.9 ARC-challenge (Acc.) 0-shot 50.8 50.2 50.0 BBH (EM) 3-shot 39.3 43.1 42.2 RACE-middle (Acc.) 5-shot 63.9 66.1 64.8 RACE-high (Acc.) 5-shot 49.6 50.8 50.6 DROP (EM) 1-shot 40.0 41.7 33.8 GSM8K (EM) 0-shot 63.4 62.6 62.2 MATH (EM) 4-shot 13.5 14.7 15.2 HumanEval (Pass@1) 0-shot 35.4 45.1 45.7 MBPP (Pass@1) 3-shot 27.8 39.0 46.2 TriviaQA (EM) 5-shot 60.1 59.5 63.3 NaturalQuestions (EM) 0-shot 35.2 32.7 35.1 MMLU (Acc.) 0-shot 50.0 49.7 47.2 WinoGrande (Acc.) 0-shot 65.1 68.4 69.0 CLUEWSC (EM) 5-shot 48.4 66.2 68.2 CEval (Acc.) 0-shot 35.1 44.7 40.0 CMMLU (Acc.) 0-shot 36.9 51.2 49.3\n\nCHID (Zheng et al., 2019) due to the observed instability of results, hindering the derivation of solid conclusions. (3) We additionally include BBH (Suzgun et al., 2022) to provide a more comprehensive assessment of the reasoning ability of the chat models.\n\nTable 5 | Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations.\n\n6.2. Evaluations\n\nBaselines. In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. Correspondingly, we construct three chat models, including LLaMA2 SFT 7B3, DeepSeek Chat 7B, and DeepSeekMoE Chat 16B.\n\nSubsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks.\n\nResults. The evaluation results are presented in Table 5. Our key observations include: (1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intensive tasks (TriviaQA, NaturalQuestions). (2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. In addition, it also surpasses DeepSeek Chat 7B. (3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1). However, it is worth noting that, after supervised finetuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. (4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations.\n\n7. Deepseekmoe 145B Ongoing\n\nEncouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available.\n\n7.1. Experimental Setup\n\nTraining Data and Tokenization. For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study.\n\nModel Settings. For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps.\n\nSubsequently, the learning rate keeps constant during the remaining training process. The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. Also, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse.\n\nEvaluation Benchmarks. We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3).\n\n7.2. Evaluations\n\nBaselines. Apart from DeepSeekMoE 145B, we consider three additional models for comparison. DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters.\n\nResults. From the evaluation results presented in Table 6, we have the following observations: (1) Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. (2) On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. (3) At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5.\n\n8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\nAppendices A. Overview Of Hyper-Parameters\n\nWe present the overview of hyper-parameters for DeepSeekMoE across various sizes in Table 7.\n\n# Params # Layers Hidden # Attn # Shared # Routed Relative Sequence Batch Size Learning Size Heads Experts Experts Expert Size Length (Sequence) Rate 2.0B 9 1280 10 1 63 (7 activated) 0.25 2048 2048 1.08e-3 16.4B 28 2048 16 2 64 (6 activated) 0.25 4096 4608 4.2e-4 144.6B 62 4096 32 4 128 (12 activated) 0.125 4096 4608 3.0e-4\n\nTable 7 | Overview of hyper-parameters for DeepSeekMoE across various sizes. The relative expert size is in comparison to a standard FFN.\n\nB. Comparing Deepseekmoe With Larger Models\n\nComparisons among DeepSeekMoE, GShard×1.2, and GShard×1.5 are shown in Table 8. Comparisons among DeepSeekMoE, Dense×4, and Dense×16 are shown in Table 9.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 2.3B 2.8B 1.9B # Activated Expert Params N/A 0.28B 0.35B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.824 1.808 1.808 HellaSwag (Acc.) 0-shot 53.7 54.4 54.8 PIQA (Acc.) 0-shot 71.8 71.1 72.3 ARC-easy (Acc.) 0-shot 46.8 47.3 49.4 ARC-challenge (Acc.) 0-shot 31.7 34.1 34.3 RACE-middle (Acc.) 5-shot 43.7 46.4 44.0 RACE-high (Acc.) 5-shot 31.9 32.4 31.7 HumanEval (Pass@1) 0-shot 3.7 3.0 4.9 MBPP (Pass@1) 3-shot 2.4 2.6 2.2 TriviaQA (EM) 5-shot 15.2 15.7 16.6 NaturalQuestions (EM) 5-shot 4.5 4.7 5.7\n\nTable 8 | Comparison between DeepSeekMoE and larger GShard models.\n\nAt a larger scale of 13B total parameters, we also compare DeepSeekMoE with GShard×1.2 and GShard×1.5, and show results in Table 10. At a larger scale, DeepSeekMoE even outperforms GShard×1.5 distinctly.\n\nMetric # Shot Dense×4 Dense×16 DeepSeekMoE Relative Expert Size N/A 1 1 0.25 # Experts N/A 4 + 0 16 + 0 1 + 63 # Activated Experts N/A 4 + 0 16 + 0 1 + 7 # Total Expert Params N/A 0.47B 1.89B 1.89B # Activated Expert Params N/A 0.47B 1.89B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.908 1.806 1.808 HellaSwag (Acc.) 0-shot 47.6 55.1 54.8 PIQA (Acc.) 0-shot 70.0 71.9 72.3 ARC-easy (Acc.) 0-shot 43.9 51.9 49.4 ARC-challenge (Acc.) 0-shot 30.5 33.8 34.3 RACE-middle (Acc.) 5-shot 42.4 46.3 44.0 RACE-high (Acc.) 5-shot 30.7 33.0 31.7 HumanEval (Pass@1) 0-shot 1.8 4.3 4.9 MBPP (Pass@1) 3-shot 0.2 2.2 2.2 TriviaQA (EM) 5-shot 9.9 16.5 16.6 NaturalQuestions (EM) 5-shot 3.0 6.3 5.7\n\nTable 9 | Comparison between DeepSeekMoE and larger dense baselines.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 15.9B 19.8B 13.3B # Activated Expert Params N/A 2.37B 2.82B 2.05B # Training Tokens N/A 100B 100B 100B HellaSwag (Acc.) 0-shot 66.6 67.7 69.1 PIQA (Acc.) 0-shot 75.6 76.0 75.7 ARC-easy (Acc.) 0-shot 56.8 56.8 58.8 ARC-challenge (Acc.) 0-shot 39.9 37.6 38.5 RACE-middle (Acc.) 5-shot 51.6 50.6 52.4 RACE-high (Acc.) 5-shot 37.4 36.3 38.5 HumanEval (Pass@1) 0-shot 6.1 6.1 9.8 MBPP (Pass@1) 3-shot 7.0 11.6 10.6 TriviaQA (EM) 5-shot 36.5 36.7 38.2 NaturalQuestions (EM) 5-shot 12.6 12.1 13.7\n\nTable 10 | Comparison between DeepSeekMoE and larger GShard models at a larger scale.\n\nC. Training Benchmark Curves Of Deepseekmoe 16B\n\nWe present the benchmark curves during training of DeepSeekMoE 16B and DeepSeek 7B (Dense) in Figure 7 for reference.",
+        "document_metadata": {
+          "source": "../../datasets/Sample_Docs_Markdown/DeepSeekMoE_2.md"
+        },
+        "summary": "DeepSeekMoE is an innovative Mixture-of-Experts (MoE) architecture designed to enhance expert specialization in language models. It employs fine-grained expert segmentation and shared expert isolation to improve performance and reduce computational costs. Starting with a model of 2B parameters, DeepSeekMoE demonstrates comparable performance to larger models like GShard 2.9B and LLaMA2 7B, using significantly fewer computations. The architecture scales up to 16B and 145B parameters, maintaining advantages over traditional MoE architectures. DeepSeekMoE 16B achieves similar results to dense models with only 40% of the computations, and its chat model version shows adaptability in bilingual scenarios. The architecture's public release aims to contribute to advancements in large-scale language models.",
+        "headlines": {
+          "1. Introduction": [],
+          "2. Preliminaries: Mixture-Of-Experts For Transformers": [],
+          "3. Deepseekmoe Architecture": [
+            "3.1. Fine-Grained Expert Segmentation",
+            "3.2. Shared Expert Isolation",
+            "3.3. Load Balance Consideration"
+          ],
+          "4. Validation Experiments": [
+            "4.1. Experimental Setup",
+            "4.1.1. Training Data And Tokenization",
+            "4.1.2. Infrastructures",
+            "4.1.3. Hyper-Parameters",
+            "4.1.4. Evaluation Benchmarks",
+            "4.2. Evaluations",
+            "4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models",
+            "4.4. Ablation Studies",
+            "4.5. Analysis On Expert Specialization"
+          ],
+          "5. Scaling Up To Deepseekmoe 16B": [
+            "5.1. Experimental Setup",
+            "5.1.1. Training Data And Tokenization",
+            "5.1.2. Hyper-Parameters",
+            "5.1.3. Evaluation Benchmarks",
+            "5.2. Evaluations",
+            "5.2.1. Internal Comparison With Deepseek 7B",
+            "5.2.2. Comparison With Open Source Models"
+          ],
+          "6. Alignment For Deepseekmoe 16B": [
+            "6.1. Experimental Setup",
+            "6.2. Evaluations"
+          ],
+          "7. Deepseekmoe 145B Ongoing": [
+            "7.1. Experimental Setup",
+            "7.2. Evaluations"
+          ],
+          "8. Related Work": [],
+          "9. Conclusion": [],
+          "Appendices": [
+            "A. Overview Of Hyper-Parameters",
+            "B. Comparing Deepseekmoe With Larger Models",
+            "C. Training Benchmark Curves Of Deepseekmoe 16B"
+          ]
+        },
+        "summary_embedding": [
+          -0.012465757317841053,
+          -0.022808987647294998,
+          -0.018792515620589256,
+          -0.02308654971420765,
+          0.01797616109251976,
+          0.007200260180979967,
+          0.006490030791610479,
+          0.042613785713911057,
+          -0.014049487188458443,
+          0.03187054023146629,
+          -0.021486490964889526,
+          -0.023592690005898476,
+          0.009045224636793137,
+          -0.014441338367760181,
+          0.005865518003702164,
+          -0.007363531738519669,
+          -0.008743173442780972,
+          -0.005163452122360468,
+          0.018841497600078583,
+          -0.03100520372390747,
+          0.011110605672001839,
+          -0.046075135469436646,
+          -0.020017050206661224,
+          -0.021845687180757523,
+          0.016425084322690964,
+          0.037389107048511505,
+          0.022547753527760506,
+          -0.0028062239289283752,
+          -0.06844329088926315,
+          0.034678805619478226,
+          0.035168617963790894,
+          0.02052319049835205,
+          0.0005438972148112953,
+          -0.00031302153365686536,
+          -0.024833550676703453,
+          -0.028556134551763535,
+          0.03347059711813927,
+          -0.004379750229418278,
+          -0.01453930139541626,
+          0.015976088121533394,
+          0.0414055772125721,
+          0.012106560170650482,
+          -0.02754385396838188,
+          0.0014245413476601243,
+          0.015151568688452244,
+          0.0055961208418011665,
+          -0.01678428053855896,
+          -0.0022960014175623655,
+          -0.002424577483907342,
+          -0.021225256845355034,
+          -0.01400050614029169,
+          0.018825169652700424,
+          0.001296985661610961,
+          -0.008865626528859138,
+          -0.0542386956512928,
+          0.007330877240747213,
+          0.03840138763189316,
+          0.05623060464859009,
+          0.044148534536361694,
+          -0.020180322229862213,
+          0.03369917720556259,
+          -0.030025575309991837,
+          -0.02501314878463745,
+          0.011028970591723919,
+          0.003983817528933287,
+          -0.018531281501054764,
+          0.01784554310142994,
+          -0.005636938381940126,
+          -0.0006556359585374594,
+          0.01751900091767311,
+          -0.03941367194056511,
+          0.020751770585775375,
+          -0.01887415163218975,
+          0.031266435980796814,
+          0.02173139713704586,
+          0.0073798587545752525,
+          0.01872720755636692,
+          0.01678428053855896,
+          0.0036858476232737303,
+          -0.04107903689146042,
+          0.009118696674704552,
+          0.016523046419024467,
+          0.014196431264281273,
+          0.016269976273179054,
+          0.06553706526756287,
+          -0.05391215533018112,
+          -0.004485876299440861,
+          -0.05169166624546051,
+          0.0012561678886413574,
+          -0.004971608519554138,
+          -0.03670336678624153,
+          0.01048201136291027,
+          0.028425518423318863,
+          0.021372200921177864,
+          -0.01063711941242218,
+          0.024392718449234962,
+          -0.022743679583072662,
+          0.0032001156359910965,
+          -0.04062187671661377,
+          0.0019653772469609976,
+          0.004518530797213316,
+          0.016531210392713547,
+          0.0070778070949018,
+          0.004273623693734407,
+          -0.016702644526958466,
+          0.006045116577297449,
+          -0.01905374974012375,
+          0.03291547670960426,
+          -0.008465612307190895,
+          0.08653374016284943,
+          0.008751336485147476,
+          0.004738946910947561,
+          0.004281787667423487,
+          0.02398454025387764,
+          0.012131051160395145,
+          -0.00810233410447836,
+          -0.04718538001179695,
+          -0.015690363943576813,
+          0.011061624623835087,
+          -0.0383034273982048,
+          0.018335357308387756,
+          -0.008759500458836555,
+          0.01282495353370905,
+          -0.04107903689146042,
+          -0.06344719231128693,
+          0.01498013362288475,
+          0.001784758409485221,
+          -0.004065453074872494,
+          -0.0002903166168835014,
+          -0.018923133611679077,
+          0.003930754493921995,
+          -0.004200151655822992,
+          -0.010645282454788685,
+          -0.021502818912267685,
+          0.03159298002719879,
+          0.03657275065779686,
+          0.06527583301067352,
+          0.036801330745220184,
+          -0.0030164355412125587,
+          0.005775718949735165,
+          0.019690508022904396,
+          -0.020784424617886543,
+          0.003481758525595069,
+          -0.01214737817645073,
+          -0.021992631256580353,
+          -0.03820546343922615,
+          0.010971825569868088,
+          0.07379858940839767,
+          -0.029029620811343193,
+          0.013437220826745033,
+          -0.017616963014006615,
+          0.04205866530537605,
+          -0.004567512311041355,
+          0.026727497577667236,
+          -0.016669990494847298,
+          0.004059330560266972,
+          -0.025388672947883606,
+          0.01620466820895672,
+          -0.024964168667793274,
+          0.012571883387863636,
+          -0.000885746325366199,
+          -0.04222193360328674,
+          0.00011116473615402356,
+          0.026890767738223076,
+          -0.010971825569868088,
+          0.02261306345462799,
+          0.0004949158756062388,
+          -0.006302268709987402,
+          -0.05097327008843422,
+          0.038727931678295135,
+          -0.003912386484444141,
+          0.018645571544766426,
+          -0.0018959869630634785,
+          0.0163107942789793,
+          -0.0016408757073804736,
+          0.013216804713010788,
+          0.009543201886117458,
+          0.007114543113857508,
+          -0.033503253012895584,
+          0.02697240374982357,
+          -0.02504580281674862,
+          0.028049994260072708,
+          0.03435226157307625,
+          -0.0030184765346348286,
+          0.03255628049373627,
+          0.019249675795435905,
+          0.03840138763189316,
+          0.0327032245695591,
+          0.030335791409015656,
+          -0.00824927818030119,
+          0.013878053054213524,
+          0.008200296200811863,
+          0.03608293831348419,
+          0.013429056853055954,
+          0.019723162055015564,
+          -0.009730963967740536,
+          -0.014041324146091938,
+          0.048785436898469925,
+          0.007318632211536169,
+          -0.0179598331451416,
+          -0.005441013257950544,
+          -0.03510330989956856,
+          -0.002832755446434021,
+          -0.014425011351704597,
+          0.0323440283536911,
+          0.02158445306122303,
+          -0.01647406443953514,
+          0.035135965794324875,
+          0.024980494752526283,
+          -0.03751972317695618,
+          -0.005232842173427343,
+          -0.02026195637881756,
+          -0.023298801854252815,
+          -0.04215662553906441,
+          0.011624909937381744,
+          0.012833117507398129,
+          -0.013976015150547028,
+          -0.007469657808542252,
+          0.020457882434129715,
+          -0.023625344038009644,
+          -0.0018265966791659594,
+          -0.01356783788651228,
+          0.006547175347805023,
+          -0.020898714661598206,
+          -0.03667071461677551,
+          -0.01048201136291027,
+          0.025862159207463264,
+          0.012637191452085972,
+          0.04362606629729271,
+          -0.013518855907022953,
+          -0.005367541220039129,
+          0.033193036913871765,
+          0.012759645469486713,
+          0.010767736472189426,
+          0.03699725493788719,
+          -0.03027048148214817,
+          0.024621298536658287,
+          -0.020294612273573875,
+          -0.0032878739293664694,
+          0.01707816869020462,
+          -0.0030062312725931406,
+          0.01962519995868206,
+          -0.04535674303770065,
+          -0.024196794256567955,
+          0.04395261034369469,
+          -0.0007107399869710207,
+          0.016302630305290222,
+          -0.005959399044513702,
+          -0.012106560170650482,
+          0.0035899258218705654,
+          0.0027715289033949375,
+          -0.01438419334590435,
+          -0.008620720356702805,
+          0.007473739795386791,
+          -0.01962519995868206,
+          0.03660540655255318,
+          0.03794422745704651,
+          -0.004132802598178387,
+          -0.014131123200058937,
+          -0.03324201703071594,
+          0.010016689077019691,
+          -0.017878197133541107,
+          -0.023951886221766472,
+          0.04078514873981476,
+          -0.032115448266267776,
+          -0.00029312283731997013,
+          0.039478980004787445,
+          0.018808843567967415,
+          -0.014898497611284256,
+          0.00868602842092514,
+          -0.01709449477493763,
+          -0.026302991434931755,
+          -0.017159804701805115,
+          0.03356856107711792,
+          0.024800896644592285,
+          -0.010743245482444763,
+          0.010245268233120441,
+          0.005330805201083422,
+          0.02893165871500969,
+          0.024376392364501953,
+          0.00648594880476594,
+          -0.013812744058668613,
+          0.015029114671051502,
+          0.027641816064715385,
+          -0.0028021421749144793,
+          -7.66609373386018e-05,
+          -0.004010349046438932,
+          -0.007796200457960367,
+          -0.006453294772654772,
+          -0.013069860637187958,
+          -0.005089980084449053,
+          0.0009184005321003497,
+          -0.032295044511556625,
+          0.015967924147844315,
+          -0.005890008993446827,
+          0.010359558276832104,
+          -0.011535110883414745,
+          0.01288209855556488,
+          0.01722511276602745,
+          -0.013494365848600864,
+          -0.0716434046626091,
+          0.014612773433327675,
+          0.02458864450454712,
+          -0.006706364918500185,
+          -0.017943505197763443,
+          0.008481939323246479,
+          0.00016148542636074126,
+          0.038597315549850464,
+          -0.014180104248225689,
+          -0.031103165820240974,
+          0.02680913358926773,
+          -0.017437364906072617,
+          -0.015241367742419243,
+          -0.010661610402166843,
+          -0.00580837344750762,
+          0.04649963974952698,
+          0.01889047957956791,
+          0.011420820839703083,
+          -0.04042595252394676,
+          -0.007449248805642128,
+          0.015078096650540829,
+          0.014506647363305092,
+          -0.04705476388335228,
+          0.0052083516493439674,
+          0.015486273914575577,
+          0.019723162055015564,
+          0.024800896644592285,
+          0.04274440184235573,
+          -0.0021531390957534313,
+          0.01661284640431404,
+          -0.015053605660796165,
+          -0.05136512219905853,
+          -0.03340528905391693,
+          0.029943939298391342,
+          -0.017731253057718277,
+          -0.02906227484345436,
+          -0.01693122461438179,
+          -0.006151242647320032,
+          0.0017939424142241478,
+          0.007126788143068552,
+          -0.01916803978383541,
+          -0.03464614972472191,
+          -0.03970756009221077,
+          0.02277633361518383,
+          0.007983962073922157,
+          0.004269542172551155,
+          -0.042777057737112045,
+          0.0013459670590236783,
+          0.009184005670249462,
+          0.012073906138539314,
+          0.0009954441338777542,
+          -0.019984396174550056,
+          0.015494437888264656,
+          0.0032001156359910965,
+          -0.031070511788129807,
+          -0.035593122243881226,
+          -0.03595232218503952,
+          0.0023919232189655304,
+          0.003781769424676895,
+          0.06279410421848297,
+          0.012988224625587463,
+          -0.03840138763189316,
+          -0.04522612690925598,
+          0.029176564887166023,
+          -0.02052319049835205,
+          -0.016678154468536377,
+          0.0059430720284581184,
+          0.012898425571620464,
+          -0.009877908043563366,
+          0.013192313723266125,
+          0.023560035973787308,
+          0.041601505130529404,
+          -0.005987971555441618,
+          0.012914752587676048,
+          0.023037567734718323,
+          0.012955570593476295,
+          0.0362788625061512,
+          -0.00936360377818346,
+          0.0035409443080425262,
+          0.039936136454343796,
+          -0.012269831262528896,
+          0.03175625205039978,
+          -0.02189466916024685,
+          0.0020398697815835476,
+          0.02249877341091633,
+          0.027184655889868736,
+          -0.0006479826406575739,
+          0.02594379521906376,
+          -0.031543996185064316,
+          0.018368011340498924,
+          -0.014498483389616013,
+          -0.04888340085744858,
+          -0.01557607389986515,
+          0.013102514669299126,
+          0.030384771525859833,
+          0.02756018005311489,
+          -0.0041348435916006565,
+          -0.019723162055015564,
+          -0.01155143789947033,
+          -0.010596301406621933,
+          0.0018419033149257302,
+          0.03285016864538193,
+          0.010506502352654934,
+          0.010890189558267593,
+          -0.011935125105082989,
+          -0.004104230087250471,
+          0.03025415539741516,
+          -0.021192602813243866,
+          0.01650671847164631,
+          0.0041511706076562405,
+          -0.002029665280133486,
+          -0.04320156201720238,
+          -0.035266581922769547,
+          0.007845181971788406,
+          0.011747363954782486,
+          0.014874007552862167,
+          0.0368339866399765,
+          0.025094784796237946,
+          -0.01662100851535797,
+          0.030629679560661316,
+          -0.01905374974012375,
+          0.027984686195850372,
+          0.039054471999406815,
+          0.011624909937381744,
+          -0.018351683393120766,
+          0.012237177230417728,
+          0.002424577483907342,
+          -0.025127438828349113,
+          0.01680060662329197,
+          -0.032735876739025116,
+          0.01738838478922844,
+          -0.02354370802640915,
+          0.025143766775727272,
+          -0.02579685114324093,
+          -0.009290131740272045,
+          -0.004428731743246317,
+          -0.006130834110081196,
+          0.04888340085744858,
+          -0.026727497577667236,
+          -0.0142454132437706,
+          -0.0024694770108908415,
+          -0.0024531499948352575,
+          -0.008718682453036308,
+          0.010000362060964108,
+          -0.045781247317790985,
+          -0.01844964735209942,
+          -0.04016471654176712,
+          0.007355368230491877,
+          0.005412440747022629,
+          -0.030907239764928818,
+          -0.05694899708032608,
+          0.025421326979994774,
+          -0.06331657618284225,
+          -0.019886434078216553,
+          0.051626358181238174,
+          -0.018400665372610092,
+          -0.008212542161345482,
+          0.009641164913773537,
+          0.006375740747898817,
+          -0.029290854930877686,
+          0.0041348435916006565,
+          0.02504580281674862,
+          0.04418119043111801,
+          -0.0045797573402523994,
+          -0.02865409664809704,
+          0.0028674507047981024,
+          -0.027788760140538216,
+          -0.013461710885167122,
+          0.005261414684355259,
+          0.016449574381113052,
+          0.024539662525057793,
+          0.007628847379237413,
+          0.03585435822606087,
+          0.007294141221791506,
+          0.006318595726042986,
+          0.044279150664806366,
+          -0.025666235014796257,
+          -0.027380581945180893,
+          0.02519274689257145,
+          0.04045860469341278,
+          0.023151857778429985,
+          -0.018253721296787262,
+          -0.0142454132437706,
+          0.0053063142113387585,
+          0.0009296254720538855,
+          -0.022139575332403183,
+          0.002808264922350645,
+          -0.013902543112635612,
+          0.005143043119460344,
+          0.03892385587096214,
+          -0.001205145614221692,
+          0.014196431264281273,
+          0.007302304729819298,
+          0.006845145486295223,
+          0.004334850702434778,
+          -0.011731036007404327,
+          -0.023592690005898476,
+          0.00032220553839579225,
+          0.034972693771123886,
+          0.015404638834297657,
+          -0.030515389516949654,
+          -0.00995954405516386,
+          0.034711457788944244,
+          -0.00795130804181099,
+          -0.009118696674704552,
+          -0.009771781973540783,
+          0.003385836724191904,
+          0.02638462744653225,
+          -0.01606588624417782,
+          0.006649219896644354,
+          0.03716052696108818,
+          -0.019870106130838394,
+          0.005543057806789875,
+          -0.025405000895261765,
+          0.0018061877926811576,
+          0.02442537248134613,
+          -0.006216551177203655,
+          0.028098974376916885,
+          5.6315813708351925e-05,
+          0.03025415539741516,
+          -0.004677720367908478,
+          0.008432958275079727,
+          -0.020327266305685043,
+          0.03987082839012146,
+          -0.017633290961384773,
+          0.03402572125196457,
+          0.02067013457417488,
+          -0.01258004643023014,
+          0.04783846437931061,
+          -0.036801330745220184,
+          -0.020784424617886543,
+          0.00584919098764658,
+          0.012743317522108555,
+          0.0028919412288814783,
+          -0.0191517136991024,
+          -0.018400665372610092,
+          -0.010996315628290176,
+          0.02380494214594364,
+          -0.022547753527760506,
+          -0.016865916550159454,
+          -0.007441085297614336,
+          -0.01890680566430092,
+          -0.009673818945884705,
+          -0.007988044060766697,
+          -0.01709449477493763,
+          -0.016547536477446556,
+          0.044573038816452026,
+          -0.06334923207759857,
+          0.016996532678604126,
+          0.017159804701805115,
+          0.011477965861558914,
+          -0.034678805619478226,
+          -0.019510909914970398,
+          -0.017633290961384773,
+          -0.002347023691982031,
+          -0.030743969604372978,
+          0.001702102366834879,
+          0.01725776679813862,
+          0.01138000376522541,
+          -0.05022222548723221,
+          0.012335140258073807,
+          0.02772345207631588,
+          -0.012065742164850235,
+          0.0014949520118534565,
+          -0.023135529831051826,
+          -0.017176130786538124,
+          0.0031776658724993467,
+          0.022123249247670174,
+          0.005975726060569286,
+          -0.008188051171600819,
+          -0.016865916550159454,
+          -0.005241005681455135,
+          -0.006045116577297449,
+          -0.002359268954023719,
+          0.005094061605632305,
+          0.011780017986893654,
+          0.019690508022904396,
+          -0.0024633544962853193,
+          -0.004436895251274109,
+          -0.006396149750798941,
+          0.027364253997802734,
+          -0.001450052484869957,
+          -0.008416631259024143,
+          0.050907962024211884,
+          0.03556047007441521,
+          0.006094098091125488,
+          -0.01108611561357975,
+          -0.03990348428487778,
+          0.010661610402166843,
+          -0.022711025550961494,
+          0.0401320643723011,
+          0.01905374974012375,
+          -0.008298259228467941,
+          -0.028866350650787354,
+          0.010294250212609768,
+          -0.02290695160627365,
+          -0.009975871071219444,
+          0.009469729848206043,
+          0.00726556871086359,
+          -0.03263791650533676,
+          -0.013543346896767616,
+          0.03510330989956856,
+          0.011437147855758667,
+          -0.006841063499450684,
+          -0.036050282418727875,
+          0.008588065393269062,
+          0.01646590232849121,
+          0.011869817040860653,
+          -0.019233347848057747,
+          0.004449140280485153,
+          -0.012041252106428146,
+          0.017470018938183784,
+          0.008849299512803555,
+          -0.017796561121940613,
+          0.018237393349409103,
+          0.004710374400019646,
+          -0.008490103296935558,
+          -0.003912386484444141,
+          -0.0163107942789793,
+          -0.02006603218615055,
+          0.03840138763189316,
+          0.024490682408213615,
+          0.026335645467042923,
+          0.005457340274006128,
+          0.02008235827088356,
+          0.03227871656417847,
+          0.011282040737569332,
+          -0.03134807199239731,
+          0.027935704216361046,
+          0.016996532678604126,
+          -0.04604247957468033,
+          0.021551799029111862,
+          -0.02218855731189251,
+          -0.028670424595475197,
+          0.009649328887462616,
+          -0.0017367975087836385,
+          0.011192241683602333,
+          0.0034837995190173388,
+          -0.003642988856881857,
+          -0.03307874873280525,
+          -0.006236960180103779,
+          0.015771998092532158,
+          0.015502601861953735,
+          -0.0048695639707148075,
+          -0.003296037670224905,
+          -0.011886144056916237,
+          0.00487364549189806,
+          0.009796272963285446,
+          0.039054471999406815,
+          0.001381682581268251,
+          -0.004751192405819893,
+          -0.0018765984568744898,
+          0.031854212284088135,
+          0.0271193478256464,
+          -0.017274094745516777,
+          -0.026286665350198746,
+          0.00580837344750762,
+          -0.014253576286137104,
+          0.026156047359108925,
+          -0.00487364549189806,
+          0.011877980083227158,
+          0.008718682453036308,
+          -0.0007908449042588472,
+          -0.001373519073240459,
+          0.025584599003195763,
+          -0.02710302174091339,
+          -0.021764053031802177,
+          0.031233781948685646,
+          -0.025731543079018593,
+          0.028735732659697533,
+          0.013061696663498878,
+          -0.01557607389986515,
+          -0.015225040726363659,
+          -0.010302413254976273,
+          0.009943217039108276,
+          -0.012277995236217976,
+          0.023853924125432968,
+          0.003608293831348419,
+          0.006392067763954401,
+          0.0003584313380997628,
+          0.02770712412893772,
+          0.024833550676703453,
+          0.007820690982043743,
+          -0.018074123188853264,
+          -0.0016816934803500772,
+          -0.02320083975791931,
+          -0.01680060662329197,
+          0.023772288113832474,
+          -0.018090449273586273,
+          -0.03268689662218094,
+          -0.028556134551763535,
+          0.012114724144339561,
+          0.007392104249447584,
+          0.010816717520356178,
+          0.03461349755525589,
+          -0.030384771525859833,
+          -0.0344502255320549,
+          0.01766594499349594,
+          0.026270337402820587,
+          -0.005551221314817667,
+          -0.021535472944378853,
+          0.03085825778543949,
+          -0.005828781984746456,
+          0.017600636929273605,
+          0.0404912605881691,
+          -0.0016225076979026198,
+          0.010228941217064857,
+          -0.042026009410619736,
+          -0.014057651162147522,
+          -0.0066614653915166855,
+          0.006261450704187155,
+          0.034417569637298584,
+          0.01048201136291027,
+          0.01535565685480833,
+          0.03448288142681122,
+          -0.021192602813243866,
+          -0.0002903166168835014,
+          -0.0022449791431427,
+          -0.02818061038851738,
+          0.03347059711813927,
+          0.025715215131640434,
+          0.011918798089027405,
+          0.019706835970282555,
+          -0.03794422745704651,
+          -0.013037205673754215,
+          0.027070365846157074,
+          0.015053605660796165,
+          0.01273515447974205,
+          0.016694480553269386,
+          0.011249386705458164,
+          0.00868602842092514,
+          -0.021633435040712357,
+          -0.05169166624546051,
+          0.01214737817645073,
+          0.01993541419506073,
+          -0.016449574381113052,
+          0.01057181041687727,
+          0.024980494752526283,
+          0.007620683871209621,
+          -0.007045152597129345,
+          0.012808626517653465,
+          -0.02682545967400074,
+          0.002259265398606658,
+          0.0019010890973731875,
+          -0.014482156373560429,
+          0.038270771503448486,
+          -0.016367938369512558,
+          0.011363675817847252,
+          -0.018204739317297935,
+          0.023870250210165977,
+          0.00011531029304023832,
+          0.0007964573451317847,
+          0.014963806606829166,
+          -0.011347348801791668,
+          -0.04235255345702171,
+          0.0021613026037812233,
+          0.03342161700129509,
+          -0.015200549736618996,
+          0.026449935510754585,
+          -0.0020674217958003283,
+          0.032458316534757614,
+          0.042777057737112045,
+          0.017731253057718277,
+          0.0003497575526125729,
+          0.025682561099529266,
+          0.0036144163459539413,
+          0.0073880222626030445,
+          0.012400448322296143,
+          0.009322785772383213,
+          0.007245160173624754,
+          0.030809277668595314,
+          -0.010261595249176025,
+          0.02173139713704586,
+          -0.04760988429188728,
+          -0.014629100449383259,
+          0.012024925090372562,
+          -0.0013316808035597205,
+          -0.012449429370462894,
+          -0.033029764890670776,
+          0.012563719414174557,
+          -0.03503800183534622,
+          0.022074267268180847,
+          -0.0004959363141097128,
+          0.04333217814564705,
+          -0.02187834307551384,
+          -0.005518566817045212,
+          0.016694480553269386,
+          0.0016510800924152136,
+          0.023282473906874657,
+          -0.0012153501156717539,
+          -0.016416920349001884,
+          -0.0067798369564116,
+          -0.009257477708160877,
+          0.004138925112783909,
+          0.04375668615102768,
+          -0.003722583642229438,
+          -0.0001682458823779598,
+          0.01409030519425869,
+          -0.011959616094827652,
+          -0.03250729665160179,
+          0.045193471014499664,
+          -0.0030837850645184517,
+          0.02484987862408161,
+          0.04979771748185158,
+          -0.06410028040409088,
+          -0.019543563947081566,
+          0.013061696663498878,
+          -0.010441194288432598,
+          0.011363675817847252,
+          0.005547139327973127,
+          0.016114868223667145,
+          0.017894525080919266,
+          -0.005485912784934044,
+          0.01619650423526764,
+          0.005106307100504637,
+          0.028098974376916885,
+          -0.02336410991847515,
+          -0.03296445682644844,
+          -0.02636830136179924,
+          -0.017747581005096436,
+          0.0014480114914476871,
+          -0.015796490013599396,
+          -3.563585551091819e-06,
+          -0.0022204883862286806,
+          -0.04388730227947235,
+          -0.025274382904171944,
+          0.040687184780836105,
+          0.006706364918500185,
+          0.02458864450454712,
+          0.0077268099412322044,
+          0.03418898954987526,
+          0.005885927006602287,
+          -0.009037061594426632,
+          0.030058229342103004,
+          -0.020049704238772392,
+          0.03833607956767082,
+          0.022106921300292015,
+          -0.002832755446434021,
+          0.005865518003702164,
+          0.031086837872862816,
+          -0.033797141164541245,
+          0.008086007088422775,
+          -0.034384917467832565,
+          0.0046695563942193985,
+          -0.008620720356702805,
+          -0.015094423666596413,
+          -0.009526874870061874,
+          -0.02504580281674862,
+          0.011216731742024422,
+          -0.021535472944378853,
+          0.01590261608362198,
+          0.010416703298687935,
+          -0.002916431985795498,
+          0.03448288142681122,
+          -0.014204595237970352,
+          -0.0027796924114227295,
+          0.0024531499948352575,
+          -0.027070365846157074,
+          -0.006343086715787649,
+          -0.003377673216164112,
+          -0.03689929470419884,
+          -0.004502203781157732,
+          0.010988152585923672,
+          -0.00024184548237826675,
+          0.016898570582270622,
+          0.02787039615213871,
+          0.0005092020728625357,
+          -0.022302847355604172,
+          0.002340900944545865,
+          -0.04199335351586342,
+          0.0011877981014549732,
+          0.01483318954706192,
+          -0.017600636929273605,
+          -0.008612556383013725,
+          0.033046092838048935,
+          0.023298801854252815,
+          -0.009396257810294628,
+          -0.035005345940589905,
+          0.017796561121940613,
+          -0.011853490024805069,
+          0.027511198073625565,
+          -0.0002630197268445045,
+          0.019298657774925232,
+          -0.0076941559091210365,
+          0.04493223875761032,
+          0.001369437319226563,
+          0.008645210415124893,
+          -0.021959977224469185,
+          0.007020662073045969,
+          -0.020278284326195717,
+          -0.002406209474429488,
+          -0.006171651650220156,
+          -0.012751481495797634,
+          -0.010669773444533348,
+          0.0032735876739025116,
+          0.007730891928076744,
+          -0.0176822729408741,
+          0.0055961208418011665,
+          -0.015959760174155235,
+          -0.02741323597729206,
+          0.007543129846453667,
+          -0.05965929850935936,
+          -0.0179598331451416,
+          -0.03373183310031891,
+          -0.009477893821895123,
+          0.02008235827088356,
+          0.03510330989956856,
+          -0.012759645469486713,
+          0.00497977202758193,
+          -0.014898497611284256,
+          0.004477712791413069,
+          0.03490738570690155,
+          0.044442422688007355,
+          -0.014759717509150505,
+          0.003381754970178008,
+          -0.010955498553812504,
+          0.033029764890670776,
+          -0.03347059711813927,
+          0.0030062312725931406,
+          0.031086837872862816,
+          -0.005057325586676598,
+          0.007959471084177494,
+          0.011624909937381744,
+          0.020474210381507874,
+          -0.019396619871258736,
+          0.00877582747489214,
+          0.002100075827911496,
+          0.019233347848057747,
+          0.02171507105231285,
+          0.020343592390418053,
+          -0.03539719805121422,
+          0.028409190475940704,
+          0.018384337425231934,
+          0.02444170042872429,
+          0.044442422688007355,
+          -0.03974021226167679,
+          -0.004147088620811701,
+          -0.0032225653994828463,
+          -0.02997659333050251,
+          0.02008235827088356,
+          -0.007530884351581335,
+          -0.01841699331998825,
+          -0.007069643586874008,
+          0.03327467292547226,
+          0.014571955427527428,
+          -0.05051611363887787,
+          0.01980479806661606,
+          0.01740471087396145,
+          0.02055584453046322,
+          0.024474354460835457,
+          -0.01123305968940258,
+          -0.0019072118448093534,
+          0.008996243588626385,
+          0.006804327480494976,
+          0.019266001880168915,
+          0.015257694758474827,
+          0.002012317767366767,
+          -0.0022082431241869926,
+          -0.011837163008749485,
+          0.005224678665399551,
+          -0.009077879600226879,
+          0.010808554477989674,
+          -0.02458864450454712,
+          -0.009698309935629368,
+          0.021388528868556023,
+          0.0031164393294602633,
+          -0.013665799982845783,
+          -0.03729114308953285,
+          -0.045944519340991974,
+          0.0033837957307696342,
+          0.004543021321296692,
+          -0.05365091934800148,
+          0.005322641227394342,
+          -0.026547899469733238,
+          -0.002108239568769932,
+          -0.005502239800989628,
+          0.0019214981002733111,
+          0.005636938381940126,
+          0.0167679525911808,
+          -0.013494365848600864,
+          0.02248244546353817,
+          -0.015502601861953735,
+          0.03165828809142113,
+          0.0055961208418011665,
+          0.007375776767730713,
+          -0.0027796924114227295,
+          0.05590406060218811,
+          -0.003712379140779376,
+          -0.03699725493788719,
+          0.023282473906874657,
+          -0.01311067771166563,
+          -0.0001747512142173946,
+          0.030090883374214172,
+          0.00884113647043705,
+          -0.015257694758474827,
+          0.013437220826745033,
+          -0.030613351613283157,
+          -0.03249097242951393,
+          0.010996315628290176,
+          -0.013755599036812782,
+          -0.0015776080545037985,
+          -0.04568328335881233,
+          -0.009339112788438797,
+          0.004269542172551155,
+          -0.018335357308387756,
+          0.010775899514555931,
+          0.01631895825266838,
+          0.013992343097925186,
+          -0.024245774373412132,
+          0.009053388610482216,
+          0.006865554489195347,
+          0.013453547842800617,
+          0.0007331897504627705,
+          0.014049487188458443,
+          -0.008677864447236061,
+          -0.011306531727313995,
+          -0.018319029361009598,
+          0.0032450153958052397,
+          -0.026792805641889572,
+          0.005155288614332676,
+          -0.003155216109007597,
+          -0.03771565109491348,
+          -0.06521052122116089,
+          -0.0344502255320549,
+          0.016294466331601143,
+          0.03396041318774223,
+          -0.009045224636793137,
+          0.0035144127905368805,
+          -0.00023648813657928258,
+          -0.01093917153775692,
+          0.024523336440324783,
+          -0.005930826533585787,
+          -0.0004829256213270128,
+          0.0035389035474509,
+          0.02050686441361904,
+          -0.009722800925374031,
+          0.005845109466463327,
+          -0.051952898502349854,
+          -0.01586179807782173,
+          0.01812310516834259,
+          0.006306350696831942,
+          0.014629100449383259,
+          -0.009690145961940289,
+          0.01693122461438179,
+          -0.030074557289481163,
+          -0.028147956356406212,
+          0.04130761697888374,
+          0.008277850225567818,
+          0.011486129835247993,
+          -0.003055212553590536,
+          -0.01680060662329197,
+          0.02504580281674862,
+          -0.0020633398089557886,
+          -0.006461458280682564,
+          -0.00817988719791174,
+          0.015820980072021484,
+          0.044279150664806366,
+          0.003255219664424658,
+          0.0009821783751249313,
+          0.017045514658093452,
+          -0.002663361607119441,
+          -0.011575928889214993,
+          0.007677828427404165,
+          0.013592327944934368,
+          -0.016384266316890717,
+          0.010375885292887688,
+          -0.025862159207463264,
+          0.0332583449780941,
+          0.03131541982293129,
+          0.011575928889214993,
+          -0.015045441687107086,
+          0.006130834110081196,
+          0.005330805201083422,
+          0.008086007088422775,
+          -0.001369437319226563,
+          -0.0017133272485807538,
+          0.020719116553664207,
+          -0.03276853263378143,
+          -0.01693122461438179,
+          -0.005347132217139006,
+          0.003932795487344265,
+          0.005277741700410843,
+          0.03396041318774223,
+          -0.0033266509417444468,
+          0.017927179113030434,
+          -0.011657563969492912,
+          0.010579974390566349,
+          0.021764053031802177,
+          0.037095218896865845,
+          -0.024996822699904442,
+          -0.029862305149435997,
+          0.012914752587676048,
+          -0.001296985661610961,
+          0.014065815135836601,
+          0.0010173837654292583,
+          0.014139287173748016,
+          0.0029409227427095175,
+          0.060475654900074005,
+          -0.017763907089829445,
+          0.004628738854080439,
+          0.0004035860183648765,
+          -0.01858026348054409,
+          -0.0013531101867556572,
+          0.004261378664523363,
+          -0.010979988612234592,
+          0.03608293831348419,
+          0.002344982698559761,
+          0.039185091853141785,
+          0.013918871060013771,
+          0.009290131740272045,
+          0.0015143405180424452,
+          -0.013592327944934368,
+          0.008355404250323772,
+          0.0015592400450259447,
+          -0.001132694073021412,
+          0.009461566805839539,
+          0.019298657774925232,
+          0.009714636951684952,
+          0.025274382904171944,
+          0.009788108989596367,
+          0.012857607565820217,
+          -0.029470453038811684,
+          0.03510330989956856,
+          -0.0006224715034477413,
+          0.012996388599276543,
+          -0.02040890045464039,
+          -0.003136848099529743,
+          -0.0008240094175562263,
+          0.015649545937776566,
+          -0.015616890974342823,
+          -0.001201063860207796,
+          -0.01297189760953188,
+          0.040099408477544785,
+          -0.018988441675901413,
+          0.013976015150547028,
+          0.009673818945884705,
+          -0.02865409664809704,
+          -0.007596192881464958,
+          0.01995174214243889,
+          0.014294394291937351,
+          -0.0053144777193665504,
+          -0.020490536466240883,
+          0.005832863971590996,
+          0.015380147844552994,
+          -0.028572462499141693,
+          1.3584674888988957e-05,
+          0.014751553535461426,
+          -0.0164903923869133,
+          -0.00025409081717953086,
+          -0.004702210891991854,
+          0.003975654020905495,
+          -0.009265640750527382,
+          0.007028825581073761,
+          0.016057724133133888,
+          -0.024507008492946625,
+          -0.020049704238772392,
+          0.005722655914723873,
+          0.006571666337549686,
+          0.01444950234144926,
+          0.007739055436104536,
+          0.004355259705334902,
+          -0.020033378154039383,
+          -0.01799248717725277,
+          0.0032674651592969894,
+          -0.01649855636060238,
+          -0.0048614004626870155,
+          0.0020531355403363705,
+          -0.019363965839147568,
+          0.01875986158847809,
+          -0.010743245482444763,
+          0.00017538899555802345,
+          -0.003226647386327386,
+          -0.020800752565264702,
+          0.006204306147992611,
+          -0.004865481983870268,
+          0.019870106130838394,
+          0.003555230563506484,
+          0.02785406820476055,
+          0.02310287579894066,
+          -0.004428731743246317,
+          0.017600636929273605,
+          -0.016155686229467392,
+          0.007983962073922157,
+          -0.031135819852352142,
+          -0.001782717532478273,
+          -0.010408539324998856,
+          0.0036103345919400454,
+          -0.015943434089422226,
+          -0.009126860648393631,
+          -0.0048777274787425995,
+          -0.0006678812787868083,
+          -0.004820582456886768,
+          0.0004275664687156677,
+          0.0027817331720143557,
+          0.02231917530298233,
+          -0.013208640739321709,
+          -0.02803366631269455,
+          -0.017437364906072617,
+          0.006045116577297449,
+          0.019396619871258736,
+          -0.026025431230664253,
+          -0.009184005670249462,
+          -0.005012426059693098,
+          -0.004326687194406986,
+          -0.021404854953289032,
+          -0.015167895704507828,
+          -0.005632856860756874,
+          -0.008045189082622528,
+          -0.010522829368710518,
+          0.0046532293781638145,
+          -0.0035715578123927116,
+          -0.011739199981093407,
+          0.021666089072823524,
+          0.018057795241475105,
+          -0.019837452098727226,
+          -0.019837452098727226,
+          0.013641309924423695,
+          -0.006159406621009111,
+          0.0018878233386203647,
+          0.02008235827088356,
+          0.00029286774224601686,
+          0.005824700463563204,
+          0.020882388576865196,
+          -0.0035225762985646725,
+          -0.01797616109251976,
+          0.003577680326998234,
+          -0.0018888438353314996,
+          -0.012122887186706066,
+          0.003212361130863428,
+          0.02773977816104889,
+          0.0008076822850853205,
+          0.023429417982697487,
+          -0.03056436963379383,
+          -0.011412657797336578,
+          0.006061443593353033,
+          0.02369065210223198,
+          0.00211844383738935,
+          -0.0021694661118090153,
+          0.020131340250372887,
+          0.03941367194056511,
+          0.005775718949735165,
+          0.0015755671774968505,
+          -0.04003410041332245,
+          0.01027792226523161,
+          0.009208496659994125,
+          0.024621298536658287,
+          0.008339077234268188,
+          0.0014653591206297278,
+          -0.012906589545309544,
+          -0.009755454957485199,
+          -0.0017939424142241478,
+          0.014702572487294674,
+          -0.023870250210165977,
+          0.0024755997583270073,
+          -0.03157665207982063,
+          0.005094061605632305,
+          0.0021959978621453047,
+          -0.009559528902173042,
+          0.0335359051823616,
+          0.00980443600565195,
+          -0.02065380848944187,
+          0.015151568688452244,
+          0.005767555441707373,
+          0.007330877240747213,
+          0.022449791431427002,
+          -0.008228869177401066,
+          0.030450081452727318,
+          -0.005049162078648806,
+          0.01213921420276165,
+          0.02742956392467022,
+          0.032588932663202286,
+          -0.013102514669299126,
+          -0.003959327004849911,
+          0.0006097159348428249,
+          -0.0002561317232903093,
+          0.0007995187188498676,
+          -0.008204378187656403,
+          0.0037389106582850218,
+          -0.014800535514950752,
+          0.012710663489997387,
+          -0.019543563947081566,
+          -0.0034286954905837774,
+          0.0038532004691660404,
+          0.01991908811032772,
+          0.0016265894519165158,
+          -0.033633869141340256,
+          0.02055584453046322,
+          -0.015420965850353241,
+          0.0135596739128232,
+          0.001290863030590117,
+          -0.011110605672001839,
+          -0.00029210239881649613,
+          0.020588500425219536,
+          0.010947334580123425,
+          -0.007845181971788406,
+          -0.0135596739128232,
+          -0.028376536443829536,
+          -0.021535472944378853,
+          0.00823703221976757,
+          0.009445239789783955,
+          -0.0028837777208536863,
+          0.0021306893322616816,
+          -0.011869817040860653,
+          0.0006520643946714699,
+          -0.0008122742874547839,
+          0.01740471087396145,
+          -0.0335359051823616,
+          0.021943651139736176,
+          -0.019527235999703407,
+          -0.019543563947081566,
+          0.017045514658093452,
+          0.008049271069467068,
+          0.009445239789783955,
+          0.0033552234526723623,
+          -0.01995174214243889,
+          -0.012326976284384727,
+          0.0338951013982296,
+          -0.01138000376522541,
+          0.006298186723142862,
+          0.004408322740346193,
+          0.012914752587676048,
+          -0.008914608508348465,
+          -0.011926962062716484,
+          0.01784554310142994,
+          -0.009861581027507782,
+          0.013951525092124939,
+          0.0010010566329583526,
+          -0.0254703089594841,
+          0.02739690989255905,
+          -0.019233347848057747,
+          0.03487472981214523,
+          -0.021617108955979347,
+          0.00711862463504076,
+          0.0066614653915166855,
+          0.008792154490947723,
+          -0.006690037902444601,
+          -0.013224967755377293,
+          0.0012786176521331072,
+          0.03807484731078148,
+          -0.02623768337070942,
+          0.024180466309189796,
+          0.022400809451937675,
+          -0.00040766780148260295,
+          0.02065380848944187,
+          -0.008367649279534817,
+          -0.026711169630289078,
+          0.0179598331451416,
+          -0.012596373446285725,
+          -0.026123393326997757,
+          0.013257622718811035,
+          -0.022678371518850327,
+          -0.020278284326195717,
+          -0.026302991434931755,
+          0.004947117529809475,
+          -0.0025694805663079023,
+          -0.005151206627488136,
+          0.009004407562315464,
+          0.02561725303530693,
+          0.020915042608976364,
+          0.019576217979192734,
+          0.02111096680164337,
+          0.01161674689501524,
+          0.022449791431427002,
+          -0.007559456862509251,
+          0.04401791840791702,
+          -0.004947117529809475,
+          0.004632820375263691,
+          -0.0015980169409886003,
+          -0.015657709911465645,
+          0.009771781973540783,
+          0.0119106350466609,
+          0.0029572497587651014,
+          -0.029176564887166023,
+          -0.009061552584171295,
+          -0.03660540655255318,
+          0.005053244065493345,
+          0.0020664012990891933,
+          0.0176822729408741,
+          0.032115448266267776,
+          -0.005865518003702164,
+          0.004592002835124731,
+          -0.005102225113660097,
+          -0.013927034102380276,
+          -0.009420748800039291,
+          -0.02739690989255905,
+          -0.025437654927372932,
+          -0.007653337903320789,
+          -0.013739272020757198,
+          -0.030368445441126823,
+          -0.011722872965037823,
+          0.01551892887800932,
+          -0.007645174395292997,
+          0.011918798089027405,
+          -0.016702644526958466,
+          0.01385356206446886,
+          -0.004110352601855993,
+          -0.004763437435030937,
+          -0.004530776292085648,
+          -0.017600636929273605,
+          -0.006420640274882317,
+          0.014367866329848766,
+          -0.009477893821895123,
+          -0.01237595733255148,
+          -0.02215590327978134,
+          -0.0015459742862731218,
+          -0.01574750803411007,
+          0.003489922033622861,
+          -0.014596446417272091,
+          -0.006771673448383808,
+          -0.003908304497599602,
+          -0.018514955416321754,
+          0.01766594499349594,
+          0.014816862531006336,
+          -0.019380291923880577,
+          -0.0090697156265378,
+          0.02638462744653225,
+          0.042777057737112045,
+          -0.03218075633049011,
+          -0.008200296200811863,
+          0.0020378287881612778,
+          0.018090449273586273,
+          0.010171796195209026,
+          0.016702644526958466,
+          -0.024653952568769455,
+          0.0022286521270871162,
+          -0.021649762988090515,
+          -0.001220452249981463,
+          -0.023739634081721306,
+          0.010775899514555931,
+          0.025715215131640434,
+          -0.010971825569868088,
+          0.025976449251174927,
+          0.01634344831109047,
+          -0.01812310516834259,
+          0.012016761116683483,
+          0.007183933164924383,
+          0.014645427465438843,
+          -0.019755816087126732,
+          0.0167679525911808,
+          -0.015796490013599396,
+          0.031233781948685646,
+          0.00892277155071497,
+          -0.04829562455415726,
+          -0.03503800183534622,
+          -0.013388238847255707,
+          0.0013388239312916994,
+          -0.010702427476644516,
+          0.015037278644740582,
+          0.015967924147844315,
+          0.013959688134491444,
+          -0.019380291923880577,
+          -0.015820980072021484,
+          0.011984107084572315,
+          0.010065670125186443,
+          0.027462217956781387,
+          -0.006649219896644354,
+          -0.004212397150695324,
+          -0.00960034690797329,
+          -0.02249877341091633,
+          -0.016098542138934135,
+          0.0209803506731987,
+          -0.018041469156742096,
+          0.015935270115733147,
+          -0.026139721274375916,
+          -0.012449429370462894,
+          0.017453692853450775,
+          -0.05211617052555084,
+          -0.01453113742172718,
+          0.0007612520130351186,
+          0.03012353740632534,
+          0.004751192405819893,
+          0.0035184945445507765,
+          0.01841699331998825,
+          -0.01993541419506073,
+          -0.0039776950143277645,
+          -0.02099667675793171,
+          0.006832899991422892,
+          -0.011853490024805069,
+          -0.013984179124236107,
+          0.014482156373560429,
+          -0.013845399022102356,
+          -0.01601690612733364,
+          0.0034368589986115694,
+          0.007183933164924383,
+          -0.021094640716910362,
+          0.011951452121138573,
+          0.003945040516555309,
+          0.012171869166195393,
+          0.012171869166195393,
+          0.01199227012693882,
+          0.021551799029111862,
+          0.02295593172311783,
+          -0.007334959227591753,
+          -0.004767519421875477,
+          -0.01004117913544178,
+          -0.0015398516552522779,
+          0.008898280560970306,
+          0.0033797139767557383,
+          -0.002989904023706913,
+          0.007837017998099327,
+          0.010033016093075275,
+          -0.01498013362288475,
+          0.033193036913871765,
+          -0.0035878848284482956,
+          -0.04134026914834976,
+          -0.003238892648369074,
+          -0.03800953924655914,
+          0.020033378154039383,
+          -0.023298801854252815,
+          -0.016996532678604126,
+          0.0032776694279164076,
+          -0.00629410520195961,
+          -0.019902760162949562,
+          0.013608654960989952,
+          0.011404493823647499,
+          0.00258580781519413,
+          0.02021297626197338,
+          0.012269831262528896,
+          -0.005134879611432552,
+          0.026172375306487083,
+          -0.014718899503350258,
+          -0.008751336485147476,
+          0.015567909926176071,
+          0.006853308994323015,
+          0.0022347746416926384,
+          0.0011031010653823614,
+          -0.018825169652700424,
+          0.005098143592476845,
+          0.0164903923869133,
+          0.007216587662696838,
+          -0.000566347036510706,
+          0.014049487188458443,
+          0.012514738366007805,
+          -0.015135240741074085,
+          -0.0002213090192526579,
+          -0.026564225554466248,
+          0.0032817514147609472,
+          0.004567512311041355,
+          0.007506393827497959,
+          -0.006771673448383808,
+          0.006281859707087278,
+          0.00658799335360527,
+          0.014759717509150505,
+          -0.010155469179153442,
+          -0.016980206593871117,
+          -0.002510294783860445,
+          0.004285869188606739,
+          0.016694480553269386,
+          0.03157665207982063,
+          -0.010310577228665352,
+          -0.005734901409596205,
+          -0.0386626236140728,
+          0.02070278860628605,
+          0.03332365304231644,
+          0.011175914667546749,
+          -0.028213264420628548,
+          0.012302485294640064,
+          0.0006515541463159025,
+          0.0019276207312941551,
+          -0.026302991434931755,
+          0.02787039615213871,
+          0.00974729098379612,
+          0.0066614653915166855,
+          0.017861871048808098,
+          0.0022306928876787424,
+          0.0029266364872455597,
+          -0.016302630305290222,
+          0.01018812321126461,
+          0.009086042642593384,
+          -0.017306748777627945,
+          0.0009745250572450459,
+          0.02324981987476349,
+          -0.01423724927008152,
+          -0.003208279376849532,
+          0.0018133309204131365,
+          0.02651524543762207,
+          -0.013820908032357693,
+          -0.014294394291937351,
+          0.0005714492290280759,
+          0.01751900091767311,
+          -0.0007979880319908261,
+          0.012098397128283978,
+          0.04898136481642723,
+          -0.012424939312040806,
+          0.019657853990793228,
+          0.005461421795189381,
+          -0.015290348790585995,
+          -0.032605260610580444,
+          -0.004922627005726099,
+          -0.01830270327627659,
+          0.011200404725968838,
+          0.014343376271426678,
+          -0.015763835981488228,
+          0.0007525782566517591,
+          -0.03144603595137596,
+          0.007743136957287788,
+          0.014735226519405842,
+          0.018776189535856247,
+          -0.002440904499962926,
+          0.006134915631264448,
+          0.027674470096826553,
+          0.02458864450454712,
+          -0.007392104249447584,
+          0.013804581016302109,
+          0.02320083975791931,
+          -0.025062130764126778,
+          -0.003069498809054494,
+          0.006820654962211847,
+          0.004104230087250471,
+          0.0053063142113387585,
+          -0.016963878646492958,
+          0.03915243595838547,
+          -0.005922663025557995,
+          -0.01506176870316267,
+          0.0030266400426626205,
+          -0.009028897620737553,
+          0.0063390047289431095,
+          -0.012563719414174557,
+          0.012637191452085972,
+          -0.008114579133689404,
+          0.00022883480414748192,
+          -0.0059430720284581184,
+          -0.011110605672001839,
+          -0.012653518468141556,
+          -0.0007949267164804041,
+          -0.016531210392713547,
+          0.007432921789586544,
+          0.00981259997934103,
+          0.005045080557465553,
+          0.01025343220680952,
+          -0.0036797248758375645,
+          -0.003136848099529743,
+          -0.016751626506447792,
+          -0.009984034113585949,
+          0.0073716952465474606,
+          0.01872720755636692,
+          -0.028588788583874702,
+          -0.018547609448432922,
+          0.009290131740272045,
+          0.0022898786701261997,
+          0.018776189535856247,
+          0.0158944521099329,
+          -0.0002595757250674069,
+          0.012318813242018223,
+          -0.01964152604341507,
+          -0.009184005670249462,
+          0.009690145961940289,
+          -0.003569516818970442,
+          0.01048201136291027,
+          -0.014759717509150505,
+          0.010033016093075275,
+          -0.011159587651491165,
+          -0.011053460650146008,
+          0.01385356206446886,
+          0.01738838478922844,
+          0.008873790502548218,
+          -0.001294944784604013,
+          3.383413059054874e-05,
+          -0.010506502352654934,
+          -0.024033522233366966,
+          0.004771600943058729,
+          0.027021385729312897,
+          -0.013273949734866619,
+          0.003883813973516226,
+          0.008702355436980724,
+          -0.017502672970294952,
+          -0.019821124151349068,
+          -0.012653518468141556,
+          -0.022106921300292015,
+          0.020572172477841377,
+          0.023413091897964478,
+          -0.02026195637881756,
+          0.00877582747489214,
+          -0.004620575346052647,
+          0.014049487188458443,
+          -0.01400050614029169,
+          0.010612628422677517,
+          0.0317889042198658,
+          -0.020000724121928215,
+          -0.0028654097113758326,
+          0.0036633978597819805,
+          -0.015347493812441826,
+          0.018629245460033417,
+          0.03268689662218094,
+          -0.00176945177372545,
+          0.014041324146091938,
+          0.0008138049161061645,
+          -0.009118696674704552,
+          0.01905374974012375,
+          0.011094278655946255,
+          0.005347132217139006,
+          -0.0042491331696510315,
+          -0.012212686240673065,
+          -0.03010721132159233,
+          0.016702644526958466,
+          0.0129637336358428,
+          -0.0037674831692129374,
+          -0.0011551438365131617,
+          -0.006343086715787649,
+          -0.015404638834297657,
+          0.0035123717971146107,
+          -0.013078023679554462,
+          -0.00803294312208891,
+          -0.008759500458836555,
+          0.004314441699534655,
+          -0.0034960447810590267,
+          -0.009053388610482216,
+          0.007236996199935675,
+          0.0009510547970421612,
+          -0.0010602424154058099,
+          -0.01902109570801258,
+          -0.025078458711504936,
+          -0.00995954405516386,
+          -0.016849588602781296,
+          0.0038287099450826645,
+          0.0005234883283264935,
+          -0.027184655889868736,
+          0.0053961132653057575,
+          -0.008139070123434067,
+          -0.008134988136589527,
+          0.007126788143068552,
+          0.006881881505250931,
+          -0.007437003776431084,
+          -0.010816717520356178,
+          -0.00786150898784399,
+          -0.006865554489195347,
+          0.020033378154039383,
+          0.0034348180051892996,
+          0.012082069180905819,
+          -0.010498338378965855,
+          -0.018025141209363937,
+          -0.016963878646492958,
+          0.016718972474336624,
+          -0.008971752598881721,
+          -0.02083340659737587,
+          -0.01444950234144926,
+          -0.008424794301390648,
+          -0.007665583398193121,
+          -0.01527402177453041,
+          -0.021159948781132698,
+          -0.0032103201374411583,
+          -0.011347348801791668,
+          -0.0067431009374558926,
+          -0.0028960229828953743,
+          0.0013571919407695532,
+          0.00951871182769537,
+          0.00546958576887846,
+          0.00438791373744607,
+          0.021666089072823524,
+          0.010122815147042274,
+          -0.005461421795189381,
+          0.016441410407423973,
+          -0.006183897145092487,
+          -0.004440976772457361,
+          -0.01498013362288475,
+          -0.009918726049363613,
+          0.018547609448432922,
+          -0.0055226488038897514,
+          0.008457448333501816,
+          -0.0009536059224046767,
+          -0.00576347392052412,
+          0.015674035996198654,
+          0.0129637336358428,
+          0.016718972474336624,
+          0.004771600943058729,
+          -0.00012168807734269649,
+          0.012416775338351727,
+          -0.0015745466807857156,
+          0.012555556371808052,
+          -0.0038429962005466223,
+          0.014506647363305092,
+          -0.009722800925374031,
+          -0.018988441675901413,
+          0.004157293122261763,
+          0.006265532691031694,
+          -0.012531065382063389,
+          0.009404421783983707,
+          0.007743136957287788,
+          -0.013747435994446278,
+          -0.02261306345462799,
+          -0.000106317616882734,
+          0.00014362763613462448,
+          0.008122743107378483,
+          0.008971752598881721,
+          -0.008073761127889156,
+          0.012890261597931385,
+          0.004616493359208107,
+          -0.007179851643741131,
+          -0.019657853990793228,
+          0.026009103283286095,
+          0.015608727931976318,
+          0.012269831262528896,
+          -0.052508022636175156,
+          -0.0030980713199824095,
+          0.010743245482444763,
+          -0.015951598063111305,
+          -0.026792805641889572,
+          -0.006404313258826733,
+          -0.008759500458836555,
+          -0.04290767386555672,
+          -0.009624837897717953,
+          -0.0005724696675315499,
+          -0.011388166807591915,
+          0.010400376282632351,
+          0.01573934406042099,
+          0.014612773433327675,
+          0.005489994306117296,
+          -0.007232914678752422,
+          -0.017633290961384773,
+          -0.02788672223687172,
+          0.010057506151497364,
+          -0.013380075804889202,
+          -0.006412476766854525,
+          -0.002263347152620554,
+          -0.007690073922276497,
+          0.0016112826997414231,
+          0.01691489666700363,
+          0.042483169585466385,
+          0.023037567734718323,
+          -0.012335140258073807,
+          -0.020131340250372887,
+          -0.012800462543964386,
+          -0.009192168712615967,
+          0.00633492274209857,
+          0.011592255905270576,
+          -0.004738946910947561,
+          0.0096656559035182,
+          -0.004294032696634531,
+          0.012302485294640064,
+          0.012767808511853218,
+          0.01680060662329197,
+          -2.3023791072773747e-05,
+          0.005579793825745583,
+          -0.0024694770108908415,
+          -0.01738838478922844,
+          -0.004379750229418278,
+          0.008261523209512234,
+          -0.0008872769540175796,
+          0.0019317024853080511,
+          0.014204595237970352,
+          0.019102731719613075,
+          -0.009094206616282463,
+          -0.014612773433327675,
+          -0.016098542138934135,
+          -0.005277741700410843,
+          -0.0012622905196622014,
+          0.029927613213658333,
+          0.02037624642252922,
+          -0.019266001880168915,
+          -0.018237393349409103,
+          -0.018351683393120766,
+          0.00922482367604971,
+          -0.003563394071534276,
+          -0.0064369672909379005,
+          0.01905374974012375,
+          0.023021239787340164,
+          0.017910851165652275,
+          -0.003642988856881857,
+          -0.014865843579173088,
+          0.003804219188168645,
+          0.02847449854016304,
+          0.008367649279534817,
+          0.01078406348824501,
+          0.008726846426725388,
+          -0.0011908593587577343,
+          -0.032441988587379456,
+          -0.0113146947696805,
+          0.00951871182769537,
+          -0.008481939323246479,
+          0.026564225554466248,
+          -0.016531210392713547,
+          0.009388094767928123,
+          -0.005077734589576721,
+          0.0019286411115899682,
+          -0.02365799807012081,
+          0.0014929111348465085,
+          0.010122815147042274,
+          -0.024947840720415115,
+          0.00336950970813632,
+          -0.023592690005898476,
+          -0.00013763252354692668,
+          0.008367649279534817,
+          0.0022837561555206776,
+          0.01650671847164631,
+          -0.0113146947696805,
+          0.0051838611252605915,
+          0.02099667675793171,
+          -0.002832755446434021,
+          -0.01512707769870758,
+          -0.0007842120248824358,
+          0.0005092020728625357,
+          -0.001286781276576221,
+          0.02563357912003994,
+          0.015812816098332405,
+          -0.0036756431218236685,
+          0.020882388576865196,
+          0.008677864447236061,
+          0.0019133344758301973,
+          -0.021764053031802177,
+          0.0023796779569238424,
+          -0.032899148762226105,
+          0.017437364906072617,
+          -0.011126932688057423,
+          0.012490247376263142,
+          0.00868602842092514,
+          -0.01341272983700037,
+          -0.0007388022495433688,
+          -0.0107187544927001,
+          -0.0015725058037787676,
+          -0.007934981025755405,
+          0.009779945947229862,
+          -0.004363423213362694,
+          0.008339077234268188,
+          -0.00024579971795901656,
+          -0.01476788055151701,
+          0.018694553524255753,
+          9.678283277025912e-06,
+          -0.00010925139940809458,
+          0.005441013257950544,
+          0.009935053065419197,
+          0.00786559097468853,
+          0.002009256277233362,
+          0.023870250210165977,
+          -0.031119491904973984,
+          -0.016082214191555977,
+          -0.010204450227320194,
+          0.005543057806789875,
+          -0.007922735065221786,
+          -0.018531281501054764,
+          -0.03257260471582413,
+          0.01318415068089962,
+          0.001293924287892878,
+          -0.0009806477464735508,
+          0.01258004643023014,
+          -0.00942891277372837,
+          -0.0013878053287044168,
+          -0.01606588624417782,
+          -0.014808698557317257,
+          0.006416558753699064,
+          0.0033633869607001543,
+          -0.01222901325672865,
+          -0.023739634081721306,
+          0.014759717509150505,
+          0.0007418635650537908,
+          -0.016980206593871117,
+          -0.017502672970294952,
+          0.00018457300029695034,
+          -0.004804255440831184,
+          0.02204161323606968,
+          -0.016678154468536377,
+          0.003373591462150216,
+          0.0007219648687168956,
+          0.009232986718416214,
+          -0.02889900468289852,
+          0.00716352416202426,
+          0.0006107363733462989,
+          0.0035164537839591503,
+          -0.006518602836877108,
+          -0.021013004705309868,
+          0.011371839791536331,
+          -0.0020674217958003283,
+          -0.014751553535461426,
+          -0.020751770585775375,
+          -0.0020796670578420162,
+          -0.007420676294714212,
+          -0.02724996581673622,
+          0.017910851165652275,
+          0.0009985055075958371,
+          -0.01738838478922844,
+          0.008077843114733696,
+          0.006596156861633062,
+          0.009355440735816956,
+          0.02455599047243595,
+          0.010147306136786938,
+          -0.018776189535856247,
+          -0.0008087027235887945,
+          0.006979844067245722,
+          0.009306458756327629,
+          0.0036450298503041267,
+          0.04222193360328674,
+          -0.008130906149744987,
+          -0.006579829845577478,
+          0.0066614653915166855,
+          -0.008449285291135311,
+          -0.013747435994446278,
+          0.026727497577667236,
+          0.005796127952635288,
+          0.015984252095222473,
+          0.016123032197356224,
+          -0.00945340283215046,
+          -0.005008344538509846,
+          -0.0018786393338814378,
+          0.02065380848944187,
+          0.0073798587545752525,
+          -0.005502239800989628,
+          -0.009714636951684952,
+          0.005881845485419035,
+          -0.00898808054625988,
+          0.0021653843577951193,
+          -0.009526874870061874,
+          -0.005824700463563204,
+          -0.008069679141044617,
+          0.016400592401623726,
+          -0.0003523086488712579,
+          0.027315273880958557,
+          0.020751770585775375,
+          -0.008563575334846973,
+          -0.0005683879135176539,
+          0.017208784818649292,
+          -0.005820618476718664,
+          0.015608727931976318,
+          -0.011192241683602333,
+          -0.007706400938332081,
+          0.006408394780009985,
+          0.024017194285988808,
+          0.012237177230417728,
+          -0.0056042843498289585,
+          0.0002885308349505067,
+          -0.0030348035506904125,
+          -0.010612628422677517,
+          0.013388238847255707,
+          0.015323002822697163,
+          -0.01947825588285923,
+          -0.004722619894891977,
+          -0.0016592437168583274,
+          -0.003300119424238801,
+          0.00945340283215046,
+          0.02305389568209648,
+          0.000171817431692034,
+          0.009184005670249462,
+          0.0012765767751261592,
+          0.03862996771931648,
+          -0.0053144777193665504,
+          0.03102152980864048,
+          -0.0010423846542835236,
+          -0.0059430720284581184,
+          -0.009094206616282463,
+          -0.020441556349396706,
+          -0.011486129835247993,
+          -0.00041481092921458185,
+          -0.005040998570621014,
+          -0.0004956811899319291,
+          -0.020327266305685043,
+          0.0026960158720612526,
+          0.014637264423072338,
+          0.007661501411348581,
+          0.017927179113030434,
+          -0.002889900468289852,
+          0.017437364906072617,
+          0.019690508022904396,
+          -0.01400050614029169,
+          -0.029862305149435997,
+          -0.0006515541463159025,
+          0.01498013362288475,
+          -0.01661284640431404,
+          0.019445601850748062,
+          0.004085862077772617,
+          0.0029531680047512054,
+          0.004779764451086521,
+          -0.006049198564141989,
+          0.018482301384210587,
+          0.02039257436990738,
+          0.01453930139541626,
+          -0.013167822733521461,
+          0.014057651162147522,
+          0.009771781973540783,
+          0.006600238848477602,
+          -0.01646590232849121,
+          -0.013535182923078537,
+          0.002555194543674588,
+          0.0009245232213288546,
+          0.024392718449234962,
+          -0.020588500425219536,
+          0.0016602640971541405,
+          0.010588138364255428,
+          0.01769859902560711,
+          -0.004302196204662323,
+          0.008906444534659386,
+          0.0008740111952647567,
+          0.007134951651096344,
+          -0.0035919665824621916,
+          -0.006457376293838024,
+          -0.006673710886389017,
+          -0.010392212308943272,
+          -0.0020959940738976,
+          -0.02070278860628605,
+          0.002198038622736931,
+          -0.002830714685842395,
+          0.018237393349409103,
+          -0.01018812321126461,
+          0.038140155375003815,
+          0.03399306535720825,
+          -0.0007025764207355678,
+          0.011102442629635334,
+          -0.01709449477493763,
+          -0.003945040516555309,
+          -0.021959977224469185,
+          0.01844964735209942,
+          0.016392430290579796,
+          0.012269831262528896,
+          0.005302232690155506,
+          0.017796561121940613,
+          -0.0006301248213276267,
+          0.014490320347249508,
+          0.0029654132667928934,
+          0.007971717044711113,
+          -0.002265388146042824,
+          0.003938918001949787,
+          -0.031103165820240974,
+          -0.017029186710715294,
+          -0.01138000376522541,
+          0.004796091932803392,
+          -0.006375740747898817,
+          -0.0035205355379730463,
+          0.0084737753495574,
+          -0.005902254022657871,
+          0.0004270562494639307,
+          -0.025992777198553085,
+          -0.0157148540019989,
+          -0.010318740271031857,
+          -0.004040962550789118,
+          -0.01169838197529316,
+          -0.0027694879099726677,
+          -0.0022082431241869926,
+          0.0029552089981734753,
+          0.020947696641087532,
+          -0.012808626517653465,
+          -0.016294466331601143,
+          0.004379750229418278,
+          -0.002263347152620554,
+          -0.014016833156347275,
+          -0.016898570582270622,
+          0.006665547378361225,
+          0.012098397128283978,
+          -0.0017470018938183784,
+          -0.013608654960989952,
+          0.011739199981093407,
+          -0.006628811359405518,
+          0.04663025587797165,
+          -0.013584164902567863,
+          -0.0078002819791436195,
+          0.003720542648807168,
+          0.004698128905147314,
+          -0.009396257810294628,
+          0.007330877240747213,
+          0.013020878657698631,
+          0.01830270327627659,
+          0.0095758568495512,
+          -0.008465612307190895,
+          -0.0024388637393712997,
+          0.007796200457960367,
+          -0.016147522255778313,
+          0.006404313258826733,
+          0.0008525818702764809,
+          -0.014400520361959934,
+          0.006110425107181072,
+          0.028980640694499016,
+          0.013151495717465878,
+          0.01288209855556488,
+          -0.022351829335093498,
+          -0.004094025585800409,
+          -0.009355440735816956,
+          -0.004943036008626223,
+          -0.010090161114931107,
+          0.027070365846157074,
+          -0.018155759200453758,
+          0.02651524543762207,
+          0.00898808054625988,
+          0.0017684312770143151,
+          -0.01176369097083807,
+          0.006343086715787649,
+          0.02832755446434021,
+          0.0028592869639396667,
+          -0.020000724121928215,
+          -0.013592327944934368,
+          0.006314514204859734,
+          0.01707816869020462,
+          -0.002647034591063857,
+          -0.005428767763078213,
+          0.019657853990793228,
+          0.01784554310142994,
+          -0.004334850702434778,
+          -0.00936360377818346,
+          0.0008337036124430597,
+          0.00928196869790554,
+          0.016963878646492958,
+          0.015020951628684998,
+          0.009086042642593384,
+          -0.002647034591063857,
+          0.008037025108933449,
+          -0.008800318464636803,
+          -0.016449574381113052,
+          0.007220669183880091,
+          -0.008432958275079727,
+          -0.016963878646492958,
+          0.01872720755636692,
+          0.015257694758474827,
+          0.0181394312530756,
+          0.0024592725094407797,
+          -0.016735298559069633,
+          0.022302847355604172,
+          0.005755309946835041,
+          0.002998067531734705,
+          0.02160078100860119,
+          -0.022221211344003677,
+          0.013012715615332127,
+          0.02037624642252922,
+          0.027609162032604218,
+          -0.01737205684185028,
+          -0.010906516574323177,
+          -0.004857318475842476,
+          -0.006853308994323015,
+          -0.00176945177372545,
+          -0.008253359235823154,
+          -0.012620864436030388,
+          0.0028735732194036245,
+          -0.030466407537460327,
+          0.018106777220964432,
+          -0.005102225113660097,
+          0.010669773444533348,
+          -0.019266001880168915,
+          0.011363675817847252,
+          -0.0035838030744343996,
+          0.011135096661746502,
+          0.0019612954929471016,
+          -0.030041903257369995,
+          -0.008571738377213478,
+          0.015388311818242073,
+          -0.0030368445441126823,
+          -0.013641309924423695,
+          0.02006603218615055,
+          0.00018942011229228228,
+          -0.022270193323493004,
+          -0.004314441699534655,
+          -0.006934944540262222,
+          0.002193956868723035,
+          -0.0029613315127789974,
+          -0.0052981507033109665,
+          -0.030188847333192825,
+          0.020033378154039383,
+          0.009559528902173042,
+          -0.03072764165699482,
+          -0.003255219664424658,
+          -0.014996460638940334,
+          -0.0025184585247188807,
+          0.008881953544914722,
+          -0.004000144544988871,
+          -0.015551582910120487,
+          0.01784554310142994,
+          0.022711025550961494,
+          -0.0023164101876318455,
+          -0.016057724133133888,
+          -0.0090697156265378,
+          -0.027494871988892555,
+          0.030825603753328323,
+          -0.0066696288995444775,
+          0.0022878379095345736,
+          -0.005322641227394342,
+          0.009388094767928123,
+          0.004228724166750908,
+          0.0157148540019989,
+          0.006081852596253157,
+          -0.002647034591063857,
+          0.026596879586577415,
+          -0.01964152604341507,
+          0.009567692875862122,
+          -1.588864870427642e-05,
+          -0.003308282932266593,
+          0.01890680566430092,
+          0.0007939062779769301,
+          0.021649762988090515,
+          -0.008571738377213478,
+          -0.0001475818717153743,
+          -0.0034388997592031956,
+          0.010898353531956673,
+          -0.0028450009413063526,
+          -0.003465431509539485,
+          -0.001531688030809164,
+          0.011118769645690918,
+          0.01013097818940878,
+          0.003804219188168645,
+          -0.01737205684185028,
+          0.003142970846965909,
+          -0.008547247387468815,
+          -0.007028825581073761,
+          0.01302904263138771,
+          -0.011192241683602333,
+          -0.0018990482203662395,
+          -0.03160930797457695,
+          0.016286302357912064,
+          0.0533243790268898,
+          0.008547247387468815,
+          -0.014433175325393677,
+          -0.0018806802108883858,
+          -0.019755816087126732,
+          0.01535565685480833,
+          0.011004479601979256,
+          -0.016816934570670128,
+          -0.011984107084572315,
+          -0.012310649268329144,
+          0.007649256382137537,
+          0.021306892856955528,
+          -0.006522684823721647,
+          -0.008579902350902557,
+          0.025372346863150597,
+          -0.011788181029260159,
+          -0.010882026515901089,
+          -0.01661284640431404,
+          -0.01635977439582348,
+          0.00241641397587955,
+          0.019902760162949562,
+          -0.0169475506991148,
+          0.021176276728510857,
+          0.00936360377818346,
+          0.0010133018950000405,
+          0.003624620847404003,
+          0.008588065393269062,
+          0.0024878450203686953,
+          0.017878197133541107,
+          0.003883813973516226,
+          -0.01439235731959343,
+          0.0021408938337117434,
+          -0.0030286810360848904,
+          0.004624656867235899,
+          3.4320437407586724e-06,
+          -0.002749078907072544,
+          -0.0065635028295218945,
+          0.003710338147357106,
+          -0.021306892856955528,
+          0.021502818912267685,
+          -0.022678371518850327,
+          -0.004245051648467779,
+          0.005555302836000919,
+          -0.005118552595376968,
+          0.013469874858856201,
+          0.018792515620589256,
+          0.01423724927008152,
+          0.008702355436980724,
+          0.01462093647569418,
+          -0.011331021785736084,
+          0.02323349379003048,
+          -0.013282112777233124,
+          0.0005816536722704768,
+          0.012939243577420712,
+          0.01755165494978428,
+          0.018792515620589256,
+          -0.006245123688131571,
+          -0.010384049266576767,
+          -0.0002946535241790116,
+          0.0129637336358428,
+          -0.022286519408226013,
+          -0.012269831262528896,
+          -0.01386172603815794,
+          0.011486129835247993,
+          -0.004743028432130814,
+          0.025453981012105942,
+          -0.00633492274209857,
+          -0.014653591439127922,
+          -0.015918942168354988,
+          0.014196431264281273,
+          -0.048752784729003906,
+          0.0030348035506904125,
+          -0.0024919267743825912,
+          0.03043375350534916,
+          -0.027021385729312897,
+          0.01738838478922844,
+          -0.0028490826953202486,
+          -0.009355440735816956,
+          0.015429128892719746,
+          -0.015992414206266403,
+          -0.0046613928861916065,
+          -0.0016806729836389422,
+          -0.007539048325270414,
+          -0.00112248957157135,
+          0.017616963014006615,
+          -0.015845470130443573,
+          0.01282495353370905,
+          -0.00989423505961895,
+          0.0027021386194974184,
+          -0.004526694305241108,
+          -0.007751300930976868,
+          0.0196088720113039,
+          0.034548189491033554,
+          -0.007053316105157137,
+          0.0014990337658673525,
+          -0.022547753527760506,
+          0.013192313723266125,
+          -0.006730855442583561,
+          0.02351105399429798,
+          -0.005432849284261465,
+          -0.016898570582270622,
+          -0.013976015150547028,
+          -0.009739127941429615,
+          0.00839214026927948,
+          -0.022433463484048843,
+          -0.011110605672001839,
+          -0.00922482367604971,
+          -0.002573562553152442,
+          -0.018694553524255753,
+          0.004865481983870268,
+          -0.002426618244498968,
+          -0.012506574392318726,
+          -0.008816645480692387,
+          0.01048201136291027,
+          -0.009347276762127876,
+          0.003077662317082286,
+          -0.0011929002357646823,
+          -0.011453475803136826,
+          -0.02369065210223198,
+          0.0158944521099329,
+          0.0001974561164388433,
+          -0.010359558276832104,
+          0.008408467285335064,
+          -0.0018276170594617724,
+          0.001211268245242536,
+          0.016963878646492958,
+          -0.0020664012990891933,
+          0.003467472270131111,
+          0.023478399962186813,
+          0.004057289566844702,
+          -0.010971825569868088,
+          -0.00025906547671183944,
+          0.013535182923078537,
+          -0.005118552595376968,
+          -0.010008525103330612,
+          1.6646010408294387e-05,
+          -0.02984597720205784,
+          -0.008351322263479233,
+          0.006094098091125488,
+          0.018629245460033417,
+          -0.010620792396366596,
+          -0.015853634104132652,
+          0.026319319382309914,
+          -0.006359413731843233,
+          0.019070077687501907,
+          0.002889900468289852,
+          0.0025694805663079023,
+          0.01724144071340561,
+          0.013461710885167122,
+          0.00786150898784399,
+          -0.013927034102380276,
+          -0.02429475635290146,
+          0.0016398552106693387,
+          -0.018090449273586273,
+          0.010579974390566349,
+          -0.011028970591723919,
+          0.0032756286673247814,
+          -0.02398454025387764,
+          -0.012237177230417728,
+          0.004934872500598431,
+          -0.020343592390418053,
+          -0.00040588201954960823,
+          -0.0056940834037959576,
+          0.022237539291381836,
+          -0.005310396198183298,
+          0.008669701404869556,
+          0.0018255761824548244,
+          -0.012237177230417728,
+          0.0009877908742055297,
+          0.009469729848206043,
+          -0.029192892834544182,
+          -0.0007127808639779687,
+          -0.003138889092952013,
+          0.0011418780777603388,
+          0.005351213738322258,
+          0.009755454957485199,
+          0.007322713732719421,
+          0.0027062203735113144,
+          0.007253323681652546,
+          0.010449357330799103,
+          -0.00022041612828616053,
+          0.0197884701192379,
+          0.018172085285186768,
+          -0.007351286243647337,
+          0.008710519410669804,
+          0.024817224591970444,
+          -0.013159659691154957,
+          -0.008718682453036308,
+          0.023266147822141647,
+          -0.006461458280682564,
+          -0.0011122850701212883,
+          -0.01483318954706192,
+          0.00815947912633419,
+          0.0017367975087836385,
+          0.028131630271673203,
+          0.019657853990793228,
+          0.005820618476718664,
+          0.000639819074422121,
+          0.015788326039910316,
+          -0.013020878657698631,
+          0.0035286990460008383,
+          0.015461783856153488,
+          -0.003940958995372057,
+          -0.019821124151349068,
+          0.013878053054213524,
+          0.009502384811639786,
+          -0.003228688146919012,
+          0.01551892887800932,
+          0.011412657797336578,
+          -0.030841931700706482,
+          -0.0011898389784619212,
+          0.010498338378965855,
+          -0.0021449755877256393,
+          -0.008751336485147476,
+          -0.00014273475972004235,
+          -0.012384121306240559,
+          0.021388528868556023,
+          -0.004408322740346193,
+          0.012277995236217976,
+          -0.00815947912633419,
+          -0.017127150669693947,
+          -0.011559601873159409,
+          0.007914572022855282,
+          -0.010547320358455181,
+          -0.0006714528426527977,
+          -0.004022594541311264,
+          -0.005779800936579704,
+          0.017747581005096436,
+          0.004706292413175106,
+          -0.0008964610169641674,
+          -0.0018398624379187822,
+          0.007983962073922157,
+          -0.006979844067245722,
+          -0.008726846426725388,
+          0.004816500470042229,
+          -0.012833117507398129,
+          0.003981776535511017,
+          0.005543057806789875,
+          0.01506176870316267,
+          -0.014857680536806583,
+          -0.021159948781132698,
+          0.007020662073045969,
+          0.02695607766509056,
+          -0.02516009286046028,
+          -0.005232842173427343,
+          0.006245123688131571,
+          0.002093953313305974,
+          0.006914535537362099,
+          -0.006934944540262222,
+          -0.007208424154669046,
+          -0.03657275065779686,
+          -0.0020745648071169853,
+          0.0009005427709780633,
+          0.008963589556515217,
+          -0.008049271069467068,
+          -0.0136494729667902,
+          -0.011135096661746502,
+          0.002410291228443384,
+          0.007134951651096344,
+          -0.02204161323606968,
+          0.0049593630246818066,
+          0.009608510881662369,
+          -0.019559891894459724,
+          0.025127438828349113,
+          -0.010890189558267593,
+          -6.964537897147238e-05,
+          -0.00995954405516386,
+          0.021633435040712357,
+          -0.013486201874911785,
+          -0.018368011340498924,
+          -0.0006898208521306515,
+          0.008355404250323772,
+          0.011926962062716484,
+          -0.0017663904000073671,
+          -0.011371839791536331,
+          0.0003752686607185751,
+          0.01646590232849121,
+          0.019135385751724243,
+          0.002744997153058648,
+          -0.0024837632663547993,
+          -0.01019628718495369,
+          -0.009592183865606785,
+          -0.0073961857706308365,
+          -0.0007449248805642128,
+          0.002091912319883704,
+          0.02035992033779621,
+          0.013469874858856201,
+          0.0024837632663547993,
+          -0.015135240741074085,
+          -0.005257333163172007,
+          0.008939098566770554,
+          0.009412584826350212,
+          -0.011690218932926655,
+          -0.00898808054625988,
+          0.002659279853105545,
+          0.02249877341091633,
+          0.017633290961384773,
+          -0.0014000505907461047,
+          0.0032878739293664694,
+          -0.0016592437168583274,
+          0.012506574392318726,
+          -0.007555375341325998,
+          -0.010269759222865105,
+          -0.006943108048290014,
+          -0.014645427465438843,
+          -0.010800390504300594,
+          -0.0023062059190124273,
+          -0.010179960168898106,
+          0.022629389539361,
+          0.007988044060766697,
+          0.009779945947229862,
+          0.008265605196356773,
+          0.010269759222865105,
+          0.002087830565869808,
+          0.025715215131640434,
+          -0.004424649756401777,
+          -0.011984107084572315,
+          0.007236996199935675,
+          -0.022368155419826508,
+          -0.021535472944378853,
+          0.008257441222667694,
+          0.006367577239871025,
+          0.03595232218503952,
+          -0.026890767738223076,
+          0.0052981507033109665,
+          -0.009779945947229862,
+          -0.010400376282632351,
+          0.007996207103133202,
+          -0.0015439334092661738,
+          -0.004245051648467779,
+          0.022368155419826508,
+          0.00726556871086359,
+          0.0035797213204205036,
+          0.0032817514147609472,
+          0.019837452098727226,
+          -0.005685919895768166,
+          0.01573934406042099,
+          0.0259274672716856,
+          0.024621298536658287,
+          0.009388094767928123,
+          -0.009469729848206043,
+          0.02785406820476055,
+          0.0002895512734539807,
+          -0.0034327772445976734,
+          0.007220669183880091,
+          -0.00305113079957664,
+          -0.0009556467994116247,
+          -0.010033016093075275,
+          0.021764053031802177,
+          -0.001455154619179666,
+          0.004359341226518154,
+          0.02081707864999771,
+          -7.774515688652173e-05,
+          0.006220633164048195,
+          0.01057181041687727,
+          0.002181711606681347,
+          0.003396041225641966,
+          0.010775899514555931,
+          -0.039315707981586456,
+          -0.0006571666453965008,
+          -0.025600925087928772,
+          -0.0018286375561729074,
+          0.0007357408758252859,
+          -0.01266984548419714,
+          0.013600491918623447,
+          -0.002015379024669528,
+          -0.010751409456133842,
+          -0.00029337796149775386,
+          0.021649762988090515,
+          0.005416522268205881,
+          0.004071575589478016,
+          -0.004469549283385277,
+          0.018792515620589256,
+          -0.006955353543162346,
+          -0.017861871048808098,
+          -0.012759645469486713,
+          -0.012367794290184975,
+          -0.01918436773121357,
+          -0.01214737817645073,
+          -0.0024551907554268837,
+          -0.002667443361133337,
+          0.004738946910947561,
+          0.002481722505763173,
+          -0.003228688146919012,
+          -0.02070278860628605,
+          0.015037278644740582,
+          -0.01430255826562643,
+          0.012873934581875801,
+          -0.011053460650146008,
+          0.005963481031358242,
+          -0.004122598096728325,
+          0.012726990506052971,
+          -0.021404854953289032,
+          0.02070278860628605,
+          -0.01536382082849741,
+          0.0038532004691660404,
+          0.019592545926570892,
+          0.006551257334649563,
+          -0.008118661120533943,
+          -0.013404566794633865,
+          -0.022074267268180847,
+          0.021633435040712357,
+          -0.011633073911070824,
+          -0.018237393349409103,
+          -0.004608329851180315,
+          0.009649328887462616,
+          -0.017470018938183784,
+          -0.007367613259702921,
+          0.01707816869020462,
+          -0.0018949664663523436,
+          0.012980061583220959,
+          0.023004913702607155,
+          -0.0052981507033109665,
+          0.0036572751123458147,
+          -0.009167678654193878,
+          0.0010286086471751332,
+          -0.02470293454825878,
+          0.013527019880712032,
+          0.008971752598881721,
+          0.006269614677876234,
+          -0.013935198076069355,
+          -0.013306603766977787,
+          0.0036531933583319187,
+          0.004498121794313192,
+          0.011437147855758667,
+          0.009420748800039291,
+          0.004220560658723116,
+          -0.013918871060013771,
+          -0.027788760140538216,
+          -0.027070365846157074,
+          0.03595232218503952,
+          -0.0005002731923013926,
+          -0.020947696641087532,
+          0.03399306535720825,
+          0.016114868223667145,
+          0.0022980421781539917,
+          -0.012449429370462894,
+          -0.004314441699534655,
+          0.024621298536658287,
+          0.016400592401623726,
+          0.010465684346854687,
+          -0.0004742518358398229,
+          0.005571629852056503,
+          0.0016469983384013176,
+          0.03012353740632534,
+          0.015323002822697163,
+          0.0009459525463171303,
+          -0.038597315549850464,
+          0.011943289078772068,
+          -0.007547211833298206,
+          0.0013408648082986474,
+          0.006702283397316933,
+          0.017780235037207603,
+          0.011935125105082989,
+          0.015567909926176071,
+          0.014865843579173088,
+          -0.003065417055040598,
+          0.0029511270113289356,
+          0.0006163488142192364,
+          -0.009624837897717953,
+          -0.001767410896718502,
+          -0.008881953544914722,
+          -0.005714492406696081,
+          -0.002920513739809394,
+          0.005592038854956627,
+          0.007657419890165329,
+          -0.003983817528933287,
+          0.028049994260072708,
+          -0.0006433906382881105,
+          -0.017600636929273605,
+          -0.009061552584171295,
+          -0.03611559048295021,
+          0.00434301421046257,
+          -0.030939893797039986,
+          -0.011257549747824669,
+          0.0033674687147140503,
+          -0.020457882434129715,
+          -0.024768242612481117,
+          0.0010306495241820812,
+          0.014335212297737598,
+          0.007392104249447584,
+          0.0033327736891806126,
+          -0.01004934310913086,
+          0.003816464450210333,
+          -0.001855169073678553,
+          -0.006461458280682564,
+          0.028833696618676186,
+          0.019559891894459724,
+          0.020457882434129715,
+          -0.01190247107297182,
+          0.019298657774925232,
+          0.005845109466463327,
+          0.02035992033779621,
+          0.010465684346854687,
+          -0.0048695639707148075,
+          0.0074859848245978355,
+          0.009567692875862122,
+          -0.006155324634164572,
+          -0.021519144997000694,
+          0.013192313723266125,
+          -0.0010837126756086946,
+          -0.0052083516493439674,
+          -0.016269976273179054,
+          -0.003534821793437004,
+          -0.015812816098332405,
+          0.010547320358455181,
+          -0.017159804701805115,
+          0.012498411349952221,
+          -0.000802069844212383,
+          0.00168475485406816,
+          -0.015469946898519993,
+          -0.013951525092124939,
+          -0.0012439225101843476,
+          -0.01244126632809639,
+          -0.008971752598881721,
+          0.006163488142192364,
+          -0.007539048325270414,
+          -0.008318668231368065,
+          -0.00535937724635005,
+          -0.011437147855758667,
+          -0.010882026515901089,
+          0.004367504734545946,
+          -0.0010653446661308408,
+          0.020931368693709373,
+          0.00011671340325847268,
+          -0.005824700463563204,
+          0.01190247107297182,
+          0.02787039615213871,
+          0.0016704685986042023,
+          0.004583839327096939,
+          -0.014261740259826183,
+          -0.011118769645690918,
+          -0.02279266156256199,
+          0.016572028398513794,
+          0.0049675265327095985,
+          -0.0344502255320549,
+          0.01438419334590435,
+          0.00824927818030119,
+          0.0015102586476132274,
+          -0.006698201410472393,
+          -0.005334886722266674,
+          -0.005608365871012211,
+          0.006306350696831942,
+          -0.0022837561555206776,
+          0.0005107327597215772,
+          0.0035919665824621916,
+          0.034678805619478226,
+          -0.0040674940682947636,
+          -0.007228832691907883,
+          0.00981259997934103,
+          0.0005130287609063089,
+          0.020196648314595222,
+          -0.005518566817045212,
+          0.0037572786677628756,
+          -0.019282329827547073,
+          -0.016563864424824715,
+          0.0018429238116368651,
+          -0.01722511276602745,
+          0.023282473906874657,
+          -0.005392031744122505,
+          -0.01347803883254528,
+          -0.006906372029334307,
+          -0.020115012302994728,
+          0.012637191452085972,
+          0.02310287579894066,
+          0.007016580086201429,
+          0.01033506728708744,
+          0.012131051160395145,
+          -0.02444170042872429,
+          -0.020457882434129715,
+          0.012171869166195393,
+          0.0013571919407695532,
+          -0.008449285291135311,
+          -0.005404277238994837,
+          -0.001783737912774086,
+          0.007783954963088036,
+          -0.0032633834052830935,
+          0.007881917990744114,
+          0.06491663306951523,
+          -0.01093100756406784,
+          -0.002987863030284643,
+          -0.02966637909412384,
+          -0.004396077245473862,
+          -0.006628811359405518,
+          0.0056287748739123344,
+          0.009151351638138294,
+          -0.023135529831051826,
+          0.009232986718416214,
+          -0.030025575309991837,
+          -0.010628955438733101,
+          0.00824111420661211,
+          0.0028490826953202486,
+          -0.01220452319830656,
+          -0.0009592183632776141,
+          -0.013665799982845783,
+          0.0006107363733462989,
+          -0.009412584826350212,
+          -0.00576347392052412,
+          0.012718827463686466,
+          -0.02321716584265232,
+          0.006298186723142862,
+          0.007592111360281706,
+          -0.006514521315693855,
+          -0.013722945004701614,
+          -0.017861871048808098,
+          -0.011967780068516731,
+          -0.013314766809344292,
+          0.002996026771143079,
+          0.01190247107297182,
+          -0.02607441134750843,
+          0.012922916561365128,
+          -0.0037389106582850218,
+          0.00945340283215046,
+          0.004404240753501654,
+          0.013469874858856201,
+          0.018074123188853264,
+          0.009355440735816956,
+          0.0021429345943033695,
+          -0.008024780079722404,
+          0.012702500447630882,
+          -0.01993541419506073,
+          -0.01635161228477955,
+          0.013837235048413277,
+          0.012016761116683483,
+          0.003061335301026702,
+          0.009828926995396614,
+          0.0020959940738976,
+          0.011886144056916237,
+          0.010775899514555931,
+          -0.019739490002393723,
+          -0.0006648199632763863,
+          0.0007326795603148639,
+          0.03280118480324745,
+          -0.0005938990507274866,
+          0.008857463486492634,
+          0.004420568235218525,
+          0.013159659691154957,
+          -0.012784135527908802,
+          -0.01769859902560711,
+          0.0008087027235887945,
+          -0.01057181041687727,
+          0.021257910877466202,
+          -0.008192133158445358,
+          0.006216551177203655,
+          -0.018612917512655258,
+          -0.027086693793535233,
+          0.006392067763954401,
+          -0.016261812299489975,
+          -0.018221067264676094,
+          0.01317598670721054,
+          0.0077268099412322044,
+          0.0017572063952684402,
+          -0.005098143592476845,
+          -0.020457882434129715,
+          -0.009837090037763119,
+          0.028441844508051872,
+          0.011102442629635334,
+          -0.004120557103306055,
+          0.009584019891917706,
+          -0.009094206616282463,
+          -0.012490247376263142,
+          -0.0017541450215503573,
+          0.0036715613678097725,
+          0.008261523209512234,
+          0.005645101889967918,
+          -0.01386172603815794,
+          -0.00029873527819290757,
+          -0.012400448322296143,
+          -0.021225256845355034,
+          -0.008906444534659386,
+          0.016751626506447792,
+          0.007820690982043743,
+          -0.0038960592355579138,
+          0.015812816098332405,
+          -0.004404240753501654,
+          0.002189875114709139,
+          0.014294394291937351,
+          0.005334886722266674,
+          -0.004285869188606739,
+          -0.0095758568495512,
+          -0.004612411838024855,
+          0.004043003544211388,
+          -0.0025960123166441917,
+          0.006457376293838024,
+          0.017192458733916283,
+          0.0072900597006082535,
+          -0.013755599036812782,
+          0.013135168701410294,
+          0.0007791098323650658,
+          0.0049675265327095985,
+          -0.03699725493788719,
+          -0.003247056156396866,
+          0.015796490013599396,
+          -0.013020878657698631,
+          0.02501314878463745,
+          0.005400195252150297,
+          0.022743679583072662,
+          0.015233203768730164,
+          0.008669701404869556,
+          -0.001542912912555039,
+          0.00464098434895277,
+          0.02759283408522606,
+          -0.003151134354993701,
+          -0.008637047372758389,
+          -0.0020602785516530275,
+          0.009241150692105293,
+          0.0002063850115519017,
+          0.003000108525156975,
+          0.008281932212412357,
+          -0.03562577813863754,
+          0.011706545948982239,
+          -0.019576217979192734,
+          0.0024694770108908415,
+          0.003459308762103319,
+          -0.007183933164924383,
+          -0.014865843579173088,
+          -0.005857354495674372,
+          0.022841641679406166,
+          -0.003224606392905116,
+          0.021078312769532204,
+          0.002885818714275956,
+          0.016106704249978065,
+          -0.011575928889214993,
+          -0.009208496659994125,
+          -0.0004130251472815871,
+          -0.005351213738322258,
+          -0.007539048325270414,
+          0.010596301406621933,
+          0.0034409407526254654,
+          0.005534893833100796,
+          0.002265388146042824,
+          0.0036654386203736067,
+          -0.0036878883838653564,
+          0.02773977816104889,
+          0.003563394071534276,
+          0.007996207103133202,
+          -0.011567764915525913,
+          0.014082142151892185,
+          -0.017029186710715294,
+          0.0070778070949018,
+          0.005575711838901043,
+          -0.026156047359108925,
+          0.016849588602781296,
+          0.004775682929903269,
+          0.006628811359405518,
+          0.013812744058668613,
+          0.01618834026157856,
+          -0.010408539324998856,
+          -0.014816862531006336,
+          0.024049850180745125,
+          -0.0032041973900049925,
+          -0.015225040726363659,
+          -0.006775755435228348,
+          4.438935866346583e-05,
+          0.006559420842677355,
+          0.00373074715025723,
+          0.0013020879123359919,
+          -0.0006204306264407933,
+          -0.010743245482444763,
+          0.004645065870136023,
+          -0.01934763789176941,
+          -0.0007163524278439581,
+          -0.016449574381113052,
+          -0.015633217990398407,
+          0.018678225576877594,
+          -0.01830270327627659,
+          0.009633000940084457,
+          0.0012275953777134418,
+          0.028588788583874702,
+          -0.011706545948982239,
+          -0.012155541218817234,
+          0.0015102586476132274,
+          0.010849371552467346,
+          0.0067512644454836845,
+          -0.010033016093075275,
+          -0.007583947852253914,
+          -0.014180104248225689,
+          -0.010163633152842522,
+          0.015502601861953735,
+          -0.014498483389616013,
+          -0.014343376271426678,
+          0.008914608508348465
+        ],
+        "embedding": [
+          -0.006359657085039302,
+          -0.024594786241399234,
+          -0.02553538924848095,
+          0.005360188465088291,
+          0.012801870977589839,
+          0.021680505862506104,
+          -0.016668048966001273,
+          0.06860852474581301,
+          -0.007122272841157724,
+          0.03471775000705488,
+          -0.009004297848861675,
+          0.01059974144678391,
+          0.02565604599426148,
+          -0.022544127646041287,
+          0.010616674976540785,
+          0.013658194823359737,
+          0.016400329743363844,
+          0.02858546964917478,
+          0.0067612676434693686,
+          -0.04174737193606879,
+          0.03184155110488616,
+          -0.02223105127585514,
+          -0.05506239667974977,
+          -0.0015307551410555752,
+          -0.013169788751379287,
+          0.017681749722384007,
+          0.01682706317229353,
+          0.019652295823816928,
+          -0.05623980369271269,
+          0.030698582851533197,
+          0.02286327124296613,
+          0.016019032539689178,
+          0.005508113353872315,
+          8.58985091215149e-05,
+          -0.019206292853495906,
+          -0.018439290483495144,
+          0.03475023176950151,
+          0.01018612942141759,
+          -0.0252551158857426,
+          0.0334562130666827,
+          0.020462153537485785,
+          0.02260508249714211,
+          -0.03323524989392951,
+          -0.002094251009530707,
+          0.015177442347686773,
+          -0.00018698464350038317,
+          -0.0006788648344273403,
+          0.0007777616893581039,
+          0.013352398615059647,
+          -0.030751111528744813,
+          -0.0317733818445192,
+          -0.006729064081710058,
+          0.009413204899986514,
+          -0.005712894407943592,
+          -0.038961826880527066,
+          -0.002827692206530345,
+          0.017863300537973137,
+          0.07918455388563496,
+          0.048967158919816366,
+          0.018002566325350088,
+          0.040264962620950695,
+          -0.019715100359659787,
+          -0.02211350500879408,
+          0.004918148608475574,
+          0.01062585305866606,
+          -0.014478993521696027,
+          0.0034983131226364672,
+          0.000766228629514787,
+          0.009667268771502067,
+          -0.01558439890697323,
+          -0.04984718709870083,
+          0.029613955432821887,
+          -0.009686854114630624,
+          -0.015279319238124568,
+          0.007460629676325322,
+          -0.006235979350809034,
+          0.012514387473717085,
+          0.02800435263741497,
+          -0.008688925946672778,
+          -0.01236301543374469,
+          0.029755705020188972,
+          0.01158037219446542,
+          -0.005242392206024511,
+          0.009982679027728814,
+          0.052414361714483264,
+          -0.018788500381726227,
+          -0.020335831196027204,
+          -0.08583959943356621,
+          0.029309228216812346,
+          -0.0249285282010673,
+          -0.018929396555426307,
+          0.007710663193334574,
+          0.03096798035958051,
+          -0.017089977064591828,
+          -0.028015107605405817,
+          0.06740703271128606,
+          0.009398795412354399,
+          0.01118769022826835,
+          -0.03041417460855722,
+          0.014776002883150835,
+          0.004319194067673161,
+          0.022507636639041203,
+          0.0053562788646786845,
+          0.007548072258943124,
+          0.011123374308563128,
+          -0.006924362115585165,
+          -0.011118507147092024,
+          0.009563232643295304,
+          0.00657178473678069,
+          0.08313027447380966,
+          -0.004239370156003264,
+          -0.00930406163946907,
+          -0.039202672821898946,
+          0.009761029930556298,
+          -0.02827698465424993,
+          -0.037914533246900244,
+          -0.04496050911855815,
+          -0.017711307900236353,
+          0.01934435542287117,
+          0.004335173027873153,
+          0.03800509969714242,
+          0.031198127357716314,
+          0.031311207941680276,
+          -0.012184553894588813,
+          -0.058985863670856876,
+          0.041442017057707635,
+          -0.012456189292721711,
+          -0.0040966351720394135,
+          0.010849179519959421,
+          0.005243123049347636,
+          0.0011495799963862647,
+          0.003534183676934487,
+          -0.03139340512003631,
+          -0.023876261883905306,
+          0.008382957569752911,
+          -0.004778283217011976,
+          0.03703067589888386,
+          0.06888630688839406,
+          -0.03993532296478411,
+          -0.009472167678222372,
+          0.025246872942105057,
+          -0.01524567912700368,
+          0.01595917939878582,
+          -0.003736723164442467,
+          -0.030836797784967753,
+          0.0026246446163368533,
+          0.008894300082148884,
+          0.08342856279154155,
+          -0.0007476580521704578,
+          0.025149116238550588,
+          0.004972542633631468,
+          0.034495632546556576,
+          -0.009173329764686424,
+          0.014009381543772768,
+          0.004427261668151028,
+          -0.013426138607902904,
+          -0.022234568880321808,
+          0.01976282364689966,
+          -0.03592433789490043,
+          -0.010918278220831188,
+          0.013527180289641174,
+          -0.013023113052242677,
+          -0.008304770302495363,
+          0.003677063799167178,
+          0.0013981822663219421,
+          0.047001163242600175,
+          0.01828332714887986,
+          0.002483375662343689,
+          -0.06820498187523893,
+          0.05295312038995555,
+          0.025038753504342162,
+          0.025286870198436735,
+          -0.0029544808216316115,
+          0.029982752196635428,
+          0.0005600481523874767,
+          0.0128290258663539,
+          0.009023805227424919,
+          -0.0019219166367750596,
+          -0.04482843936478531,
+          0.04523479691409549,
+          -0.004678574027241186,
+          0.010486850804218904,
+          0.006736229865018014,
+          0.02134813656476443,
+          0.028083972673690036,
+          0.016857413440125445,
+          0.0189082447820146,
+          0.03843690613820022,
+          -0.011922027011229613,
+          0.016193433524453224,
+          0.03828207911467065,
+          -0.0013055128404331705,
+          0.043198163443700956,
+          0.024025562893260542,
+          -0.001257851946749882,
+          -0.011726713433348844,
+          -0.021907984099443396,
+          0.02752198420597673,
+          0.004431866336999851,
+          0.015798806714413912,
+          -0.012910782191382367,
+          -0.03180549180206364,
+          0.004550247372715741,
+          -0.022778272841118514,
+          0.031242218240017532,
+          -0.006106984949693483,
+          -0.006259993190497421,
+          0.02187449047033339,
+          0.009334146847221406,
+          -0.009381278960016145,
+          -0.020888853823931134,
+          -0.007502374908776156,
+          -0.010390018386728482,
+          -0.04316693435221676,
+          -0.006274513730720055,
+          0.019152443937064258,
+          -0.01668984824845791,
+          0.018573194036505578,
+          0.0035035262183249925,
+          -0.019779911381789077,
+          -0.0027495798165085993,
+          0.02425481236315913,
+          0.006857723545270935,
+          -0.014733535699087731,
+          -0.026326146471941727,
+          -0.019906660231056833,
+          -0.001033294559612807,
+          0.0159154063026895,
+          0.035649534442431415,
+          -0.026210592096240003,
+          0.014397127229517285,
+          0.005114311681443673,
+          -0.0033104236005207077,
+          0.00324988235167305,
+          0.03764004154553922,
+          -0.017822437760737605,
+          -0.008579701063708334,
+          -0.00032410984200965487,
+          0.018450351071676443,
+          0.017283379764846195,
+          0.006384411611332382,
+          -0.005115090309123598,
+          -0.030590047062583125,
+          -0.011776211359193503,
+          0.01183623914060493,
+          0.0016564847796208331,
+          -0.009530766634361575,
+          -0.004810598258154945,
+          0.03876242515526699,
+          0.027076785774401667,
+          0.005324763136165989,
+          -0.03416412480913939,
+          -0.007943016733586697,
+          0.002114307650456928,
+          -0.02961449347300628,
+          0.05142732770518828,
+          0.05566026433842563,
+          -0.0030729309664974704,
+          -0.00938186948393074,
+          -0.04547908981236149,
+          0.0651216359692231,
+          -0.01813722445944094,
+          -0.0482650853560993,
+          0.044045155892545554,
+          -0.014272653003441165,
+          0.013740652306746851,
+          0.04563732439184395,
+          0.02126513340521316,
+          -0.0054845632439384905,
+          -0.011635625101262898,
+          0.030992082069685376,
+          -0.010051935721068779,
+          -0.050967258971854425,
+          -0.009827942095033494,
+          0.013373878556654464,
+          -0.013017627321534534,
+          -0.009361001542607812,
+          -0.01925867665425919,
+          0.020329052915436022,
+          0.023020741670660188,
+          0.019387043947695172,
+          0.005628632318152202,
+          0.018451555161952665,
+          0.016226845400848455,
+          -0.01384033112836216,
+          0.004762589697426251,
+          -0.012718476561725138,
+          -0.024222765127494463,
+          -0.037487144314010595,
+          -0.019971524166659278,
+          -0.015034588610516958,
+          0.009483936496215338,
+          -0.014301924283553591,
+          -0.0024139665139412125,
+          0.013361045353529458,
+          -0.0005870798721107379,
+          -0.011192319340879789,
+          0.013927695434391159,
+          0.021337539669342293,
+          -0.027300952495815437,
+          -0.04765670695199488,
+          -0.00561706224251562,
+          0.0082776639906139,
+          -0.0030984973749633724,
+          -0.026269016057688548,
+          0.007040524859505573,
+          0.0017859008072463442,
+          0.020091542517503808,
+          -0.018716761130867725,
+          0.0035283051960672963,
+          0.008519383548064053,
+          -0.015247417050336866,
+          -0.016917626868166634,
+          -0.018580550690103854,
+          -0.024697180580294884,
+          0.034202028316737125,
+          0.0204395003463227,
+          0.030392895534267717,
+          -0.04527669147753553,
+          -0.018359428092075348,
+          0.05494107499558624,
+          0.015234754907198422,
+          -0.05066008106884831,
+          0.031139824660661085,
+          0.005016615710206006,
+          0.0019000453624929544,
+          -0.006094364589229031,
+          0.0637324238017742,
+          -0.02711381489292883,
+          -0.02187418906973867,
+          0.004601744864532875,
+          -0.04141121277496072,
+          -0.026689585608019588,
+          -0.005568100750837504,
+          0.01291019036002459,
+          -0.031853041426355644,
+          -0.02815800259372657,
+          -0.007964861641790374,
+          -0.010933515363060793,
+          0.013726639077745378,
+          -0.023956800669647334,
+          -0.015672058590455337,
+          -0.023642025374517057,
+          -0.01288435863544158,
+          -0.0042516881552192385,
+          -0.011162214160460695,
+          -0.02412733256710621,
+          0.010088528028211243,
+          0.03310175458095482,
+          0.023064360903748363,
+          0.001996159684818467,
+          0.009478229572933101,
+          0.014478772877286076,
+          0.00371783700962415,
+          0.007864103610320116,
+          -0.048511515124609,
+          -0.031398754206534776,
+          0.020227872561659397,
+          0.012159811587094092,
+          0.0172163448580163,
+          0.009578952796095119,
+          -0.04154545961112031,
+          -0.05095080393682759,
+          0.02968486144855133,
+          -0.03932395023720674,
+          -0.02063219739061335,
+          0.007512423249865194,
+          -0.011911709468952548,
+          -0.00810951734088105,
+          -0.0008079327452253181,
+          -0.01600701450676789,
+          0.023984909975645632,
+          -0.02418072541081482,
+          0.0033984265728439398,
+          -0.0104343782811971,
+          0.01588977364765743,
+          0.05109529246167992,
+          -0.027800269136366997,
+          -0.027889034481533438,
+          0.03706813210665186,
+          -0.02058120652429308,
+          0.030801396468451574,
+          0.03246127847621613,
+          0.008355517405358826,
+          -0.008605467586247932,
+          0.017693344563968573,
+          0.014499487024471321,
+          0.012431277080478217,
+          -0.040103870071440256,
+          0.017328848567062113,
+          0.02723900253727976,
+          -0.054909074068976875,
+          -0.008547596270650966,
+          -0.00777964838164648,
+          0.027134666572242665,
+          0.02765036594970255,
+          -0.002207173084941375,
+          -0.016838740117653725,
+          -0.028289233739722936,
+          -0.019433376533829132,
+          -0.011053380458946764,
+          0.03247641283550457,
+          0.0007514989009289282,
+          0.007919574219000797,
+          0.001316042436488591,
+          -0.0017934342041762042,
+          0.01924145734197619,
+          0.015260199553572459,
+          0.048259602293134723,
+          -0.03383050622747848,
+          0.0021712225589426266,
+          -0.03508567715696004,
+          -0.04578731401250471,
+          -0.008768228935292693,
+          0.010203318819548585,
+          0.024135236371743855,
+          -0.007467140460133838,
+          0.0028907962385387818,
+          -0.007582083854422456,
+          -0.005384477718912037,
+          -0.01867978482547249,
+          0.01641191108381631,
+          0.037168089880543244,
+          0.0031992495663980464,
+          -0.008533890994695162,
+          0.000778843942291118,
+          -0.004620258538699103,
+          0.010572690335163528,
+          0.033233014627100904,
+          -0.030497362482602697,
+          0.02974571152880676,
+          -0.004274442492343003,
+          0.025341347125910928,
+          -0.04307748954707389,
+          -0.018069609916024403,
+          -0.018491122727274337,
+          -0.02069307802429807,
+          0.05372967226264594,
+          -0.030768978301976772,
+          -0.03662045886993282,
+          -0.003205991669059688,
+          0.016815511161004534,
+          -0.0014152644009276566,
+          -0.0016015439520090104,
+          -0.03984716970293867,
+          -0.010828260334550217,
+          -0.05894985435251326,
+          -0.01638035185618134,
+          -0.021819442096404948,
+          -0.005315102328745982,
+          -0.06251579337157112,
+          0.0033713876515840233,
+          -0.07815974854883057,
+          0.001484860071649606,
+          0.04539427129131382,
+          -0.023530943197353476,
+          0.05142250021536681,
+          -0.024746600229063383,
+          -0.0005885740455257141,
+          -0.044698536284421976,
+          0.0026481001040031785,
+          0.02389178603305878,
+          0.051387657872255674,
+          -0.002949198236157656,
+          0.010674377196609455,
+          0.013322075519401089,
+          -0.026245090721950647,
+          -0.01036724558050635,
+          -0.005321884180696542,
+          0.005958798671856245,
+          0.013755178085972732,
+          -0.0074734329537091675,
+          0.038368016046563674,
+          0.010941421833174982,
+          0.03443115781919206,
+          0.034623617136788874,
+          -0.004361926531008214,
+          -0.03348681614127998,
+          -0.00632870804659212,
+          0.024433331840566692,
+          0.0321998807702602,
+          -0.02899983280320611,
+          0.005958471722097962,
+          -0.015602447981516634,
+          0.015633217539747047,
+          0.006321931116948646,
+          -0.011827541320878442,
+          0.02372204564999124,
+          0.015673426811623124,
+          0.006103300725621424,
+          -0.005165006670545588,
+          0.013589526056189892,
+          -0.025411864292353578,
+          0.03041791947375232,
+          -0.004456865298584878,
+          -0.04276058054583262,
+          -0.0024736078733106825,
+          -0.020059309867945227,
+          0.024883721794000956,
+          0.015578237413840115,
+          -0.00536523508503184,
+          -0.005290645798641171,
+          0.008242987948511412,
+          0.020469675635821673,
+          0.006668354647245447,
+          0.00017729517736814364,
+          0.02373604785121342,
+          0.029915687122900485,
+          -0.02432635861789771,
+          -0.029540533943782886,
+          0.008151297444985344,
+          -0.005730733861939129,
+          0.011318816579801282,
+          -0.02644855229221747,
+          0.00033932872948409137,
+          0.0035505684154028996,
+          -0.03570104343808711,
+          0.03408366016010711,
+          -0.009199700348630527,
+          0.005857598530788056,
+          -0.009538177736472163,
+          -0.0017768806148210118,
+          0.0027410153099351694,
+          0.03898108970435115,
+          -0.007367623392751247,
+          0.009827254226889936,
+          0.015465895035342481,
+          0.000981604396724216,
+          0.05143083558797524,
+          -0.027350877485564314,
+          -0.02183036962301852,
+          0.01885468378168636,
+          -0.007285239007489837,
+          -0.016846837665812908,
+          -0.020546383007536074,
+          -0.011863943814684079,
+          -0.016293192750336218,
+          0.01976516340568883,
+          -0.02387328798681798,
+          -0.011892336568452852,
+          0.007170651391639504,
+          0.0013725014050721622,
+          -0.009809669068563384,
+          0.0016617183704652932,
+          -0.006307590349798603,
+          0.02037693835615797,
+          0.031753656615094446,
+          -0.024892675105483313,
+          0.03577919637821846,
+          0.03178303188315581,
+          0.017260898395251155,
+          -0.02102289690701718,
+          -0.009661035772073294,
+          -0.00048542689442823136,
+          -0.014694631967051945,
+          -0.0326379186158631,
+          0.025443206465204822,
+          0.017194626456982255,
+          -0.011381421003142453,
+          -0.03680783761649673,
+          0.0017216115811536721,
+          0.022819859205829785,
+          -0.013600989058799166,
+          0.028141657788496834,
+          -0.01869047298692401,
+          -0.01366833755869951,
+          -0.00782959271979894,
+          0.031227914441058244,
+          -0.05252392878125891,
+          0.016957842026348195,
+          0.00020880696574616416,
+          0.001154370575234767,
+          0.010892067708194324,
+          -0.008368819756628046,
+          0.012224564683325483,
+          0.031700389750709,
+          0.018063405869147564,
+          0.0010794115307202266,
+          -0.002783668216641561,
+          0.01418234057248712,
+          0.017773269840982066,
+          -0.015653443355569053,
+          -0.000497740485657824,
+          0.019890882604100452,
+          0.022672054781919238,
+          -0.00521797370595221,
+          -0.016686942830774276,
+          -0.018443538351123584,
+          0.025270057639201107,
+          -0.010354431070738671,
+          0.03484047753830186,
+          0.04594785214094975,
+          0.0036441959528283004,
+          -0.05695476763071743,
+          -0.009412209619190693,
+          0.0010371110229966086,
+          -0.01099343927053564,
+          0.01606863921935641,
+          0.012470660511773616,
+          -0.03237187923799263,
+          0.006388672213167589,
+          0.012946570716567364,
+          0.022152554488829627,
+          -0.02540317621005867,
+          -0.023183252921885226,
+          -0.0030984019855787655,
+          -0.007890467768631811,
+          0.018648760402624687,
+          -0.008653405980661855,
+          0.0006228315253506687,
+          0.009672526897422602,
+          0.0005436494014569835,
+          0.011632326145022935,
+          0.005655942007108228,
+          0.021736827172951,
+          -0.001256608167230203,
+          0.013124413886287987,
+          -0.01986808920858981,
+          -0.0270450339809516,
+          0.006500423802582728,
+          0.04438242119211055,
+          -0.014813399694814117,
+          0.030621709833336064,
+          0.0092171224268983,
+          0.016433384739642534,
+          0.024875072913970443,
+          -0.00891470467241686,
+          -0.014124009014664287,
+          0.00845777541223868,
+          0.009019007848927453,
+          -0.04147358041382793,
+          0.012513030150663913,
+          -0.01710243399313324,
+          -0.033144939719206276,
+          0.01904674924133127,
+          -0.00534122554342279,
+          0.012367320751926314,
+          0.004270315800911569,
+          -0.002886460711688153,
+          -0.025696286516581048,
+          0.014137901708875033,
+          0.028135087538950946,
+          0.021797869775098042,
+          0.0296173052503884,
+          -0.012757078368361444,
+          0.0005460601780756187,
+          -0.008748930817524899,
+          0.005470975782187157,
+          0.04187083128854477,
+          0.0008687182830616019,
+          -0.0008032802520818105,
+          0.01507726552487313,
+          0.024945744517309208,
+          0.0376130540370386,
+          -0.015415560911023553,
+          -0.03407353556457147,
+          0.01564969972264396,
+          -0.010104443707865262,
+          0.018855792875436674,
+          0.005503259932317811,
+          0.0049634097823351596,
+          0.0038600156268000702,
+          0.019312419743287703,
+          -0.006635193789316492,
+          0.018798045806429395,
+          -0.02500983683871332,
+          -0.028785757874583755,
+          0.01829411751138161,
+          -0.005277972443258226,
+          0.03338623299668019,
+          0.016225885095995672,
+          -0.029522280454919586,
+          -0.01712097344863975,
+          0.01553706187924322,
+          -0.008465766956343768,
+          -0.024842700926275538,
+          0.01895228828512998,
+          0.007076862934523977,
+          0.008956105593078965,
+          0.0024373779389833218,
+          -0.01106503696703573,
+          0.01125466547367417,
+          -0.000818223040067914,
+          -0.015322790791698395,
+          0.014157131555941659,
+          -0.017853386731281715,
+          -0.014994640991761465,
+          0.01775991780643474,
+          -0.037498078532902435,
+          0.009570197712419421,
+          -0.007687297543935123,
+          0.009050494195378254,
+          -0.004222041469820445,
+          0.007460383234429904,
+          -0.0010804379186060264,
+          -0.014987857934522812,
+          -0.014903749199810126,
+          0.03228485430019625,
+          0.022096121825475306,
+          -0.000410816936759558,
+          0.006728733918931589,
+          -3.971636461500232e-05,
+          -0.026719848032785824,
+          -0.011253751790407687,
+          0.02245467425830765,
+          -0.005664635254398481,
+          0.005134496095445943,
+          -0.01109590855846966,
+          -0.0149616794609307,
+          0.0019726017255571673,
+          0.032668637270108894,
+          0.06066635618119076,
+          0.012001732730659747,
+          0.024422093585231484,
+          0.0189499396761026,
+          0.02119276033547528,
+          -0.008489818917377516,
+          0.006413901834269739,
+          -0.03097413287789443,
+          0.03509225573986143,
+          0.005392594635384506,
+          0.03108937318306421,
+          0.026664331248813712,
+          -0.04722525921137681,
+          -0.0058539171830594774,
+          0.017087531454837677,
+          0.022673712717885323,
+          0.0159020681868943,
+          0.0008382498574190522,
+          -0.008938174268517353,
+          0.035650940044285984,
+          -0.03315250299348084,
+          -0.023647111263155,
+          0.005059382024092534,
+          -0.01439537467317451,
+          -0.010488359956731476,
+          -0.006785295580877329,
+          0.02776449681317373,
+          -0.005584052419598926,
+          -0.004582019433008779,
+          0.009697064252638535,
+          -0.014400206558190682,
+          0.008165676294736836,
+          0.0007624778531368436,
+          0.008681263897172326,
+          0.039249866981334126,
+          0.0005918063050375113,
+          0.024275699182767528,
+          -0.004138313072843555,
+          0.011079529739650594,
+          -0.0158972266904151,
+          -0.016985058543605,
+          0.02276982363172991,
+          0.00351101120496393,
+          -0.03244126813432563,
+          -0.00768819222024939,
+          0.00928180979884261,
+          -0.0013793404539641824,
+          0.005266058566444064,
+          -0.00503433670609399,
+          0.003747448261046857,
+          0.03286230666764075,
+          0.0022383592721685153,
+          0.008606190020480513,
+          0.013593947677257625,
+          0.00030951915677929204,
+          -0.01490307680740127,
+          0.013254591138113545,
+          -2.1379308136707722e-05,
+          -0.012900073369514622,
+          0.03357519186606058,
+          0.006705902094864628,
+          0.04243347200485736,
+          -0.03302953674711988,
+          0.009031534935013593,
+          0.01822227379789576,
+          -0.008491770335368761,
+          -0.02725412856390648,
+          -0.03630788717786838,
+          0.008259145747740535,
+          -0.019003260334340084,
+          0.015781275447792516,
+          -0.010039277287105153,
+          0.029665124310477384,
+          -0.03563370794521629,
+          0.004714858110268169,
+          0.004053913355055313,
+          0.0009503673086552651,
+          0.02068360255395961,
+          0.018524093113914366,
+          -0.002785698934204024,
+          -0.031443835341186485,
+          -0.02066365433385339,
+          0.006869321907853112,
+          0.02431260052207595,
+          -0.010367048093621498,
+          -0.005612149485859478,
+          0.0006316058139530722,
+          -0.024369562712568606,
+          -0.016845124984022828,
+          0.030359419843284823,
+          -0.040974427176153246,
+          0.005252911449822132,
+          0.04272936053727608,
+          -0.04599092715016919,
+          -0.03802288716943514,
+          -0.002314988728575789,
+          -0.024124014894379466,
+          -0.006360771882090754,
+          -0.002068030981218464,
+          0.016669991113175926,
+          0.0036232707799735537,
+          -0.028763764013482795,
+          0.00943401978066451,
+          -0.005173988601101547,
+          0.003971420424839974,
+          -0.0031669592494977393,
+          -0.02216336363004636,
+          -0.01657772032717298,
+          -0.02531962163186398,
+          -0.035174102848865116,
+          -0.0015903434876311088,
+          -0.005158703150821607,
+          -0.028703075279214722,
+          -0.03941940849540492,
+          -0.021696000925262537,
+          -0.0160221488026189,
+          -0.00843213135542359,
+          0.040641882367348774,
+          0.021319060666253945,
+          0.019336844555345886,
+          0.0025252167889382917,
+          -0.020346900609125973,
+          0.049440028814233025,
+          -0.007642182460707816,
+          0.035628691886108926,
+          -0.0017882888000877158,
+          -0.013461669667950893,
+          0.00455558389906464,
+          0.03825428026120858,
+          -0.014248434654950971,
+          -0.0009902062564027718,
+          0.00027811655483232545,
+          -0.0011866253757890136,
+          0.0008535155950871286,
+          0.015433650396777696,
+          0.001307651096343185,
+          -0.007112199543314309,
+          0.030380673552683906,
+          -0.012803247194895328,
+          0.01637501752192592,
+          0.018350814869962304,
+          -0.0013359719137099667,
+          0.008064546833360553,
+          -0.021430659268086814,
+          -0.00731572118780415,
+          0.0044578576331236325,
+          0.008422210247967642,
+          -0.015096885054883586,
+          -0.009964588397149773,
+          -0.02359494467299634,
+          -0.01174145797396426,
+          0.0023855749166744366,
+          -0.004559202896782688,
+          0.004902138778592298,
+          0.034595817216889944,
+          -0.016068744500427584,
+          -0.035106033294403614,
+          0.009997207702122002,
+          -0.02663088875167486,
+          0.02222306406130687,
+          3.983384942889326e-05,
+          0.006838221769859366,
+          -0.0023434342406307616,
+          0.001765445386886839,
+          -0.00169484278667461,
+          0.002862331185439823,
+          -0.018217975204395134,
+          0.0031859734246840257,
+          -0.0032119988065673565,
+          0.0009052950782312657,
+          -0.0012459653052875638,
+          0.00687339892509406,
+          0.006332799551164718,
+          -0.0008019538898592668,
+          -0.002968451740267742,
+          -8.541086355097327e-06,
+          -0.020561816931299912,
+          -0.021642512405352372,
+          -0.028259049261410905,
+          -0.000989887278779396,
+          -0.012359733017834338,
+          -0.027804394906433664,
+          0.011443385424045703,
+          -0.006872684999639775,
+          0.008327844771213648,
+          -0.008969527138034904,
+          0.006219994886600349,
+          -0.008415106130309062,
+          -0.0004143996206881767,
+          0.013119286511794273,
+          -0.02925209282917943,
+          -0.02096907906398536,
+          -0.03096513357712367,
+          0.01762223841790055,
+          -0.00925741670612824,
+          -0.002656382260603827,
+          0.0025223527960181835,
+          0.005092659166535706,
+          -0.033606542487570154,
+          -0.006653438998497412,
+          0.007313682256676618,
+          0.009547290751154595,
+          -0.03287788897629887,
+          0.02446728061355753,
+          0.017023222076526946,
+          0.01874233066565012,
+          -0.02417566312664911,
+          0.04049827067897005,
+          0.014426879325647042,
+          -0.021740606525480655,
+          0.016712907564723732,
+          0.027385150577780248,
+          0.03177163221499527,
+          -0.03469098547555545,
+          0.004481376248622418,
+          -0.01518200063132508,
+          0.018918680020516044,
+          -0.025284921244460287,
+          0.020625170445627613,
+          -0.016707629602243063,
+          0.021278719363178687,
+          0.03520334001264928,
+          -0.005524369131116028,
+          0.03608133014180745,
+          -0.019234548780647943,
+          0.0059985801736444555,
+          0.004041778461022664,
+          -0.01645503683802124,
+          0.01024694092614158,
+          -0.0042700351372843386,
+          0.0006059159967286885,
+          0.009406090340935747,
+          0.019133229156115282,
+          0.019007219188041775,
+          -0.04442208547053466,
+          0.028588965123434923,
+          0.00804120101594811,
+          0.021472086070876265,
+          0.014177820346835521,
+          -0.02708807970982664,
+          0.004742981711172693,
+          0.002072673190279628,
+          -0.004659713164250794,
+          0.0061167904730676514,
+          -0.013796604518930328,
+          -0.034540857808309226,
+          0.006338792319814448,
+          -0.018918703778187926,
+          0.015036937364271984,
+          -0.009026670935442384,
+          0.0011667525636770179,
+          -0.01152652811400482,
+          -0.011938679695598347,
+          0.0006170592505446954,
+          -0.005495151258143897,
+          -0.0008075736908949407,
+          -0.03171721592827769,
+          -0.04427658539412448,
+          0.02036132271908089,
+          -0.0021221965958423804,
+          -0.04320396743189005,
+          0.004111179168317842,
+          -0.025352428814134187,
+          -0.01865918186335911,
+          -0.011237699157960728,
+          -0.025047923887906474,
+          -0.04219630696740606,
+          -0.011108496740166623,
+          0.007785006066683012,
+          0.016810642610918525,
+          -0.009081815651299525,
+          0.040965574143584864,
+          0.0073944364393447965,
+          0.015313576815177263,
+          -0.017731360025694955,
+          0.026137777790243826,
+          -0.019314392060123,
+          -0.03108486330987073,
+          0.02676829566273746,
+          -6.259161089416372e-05,
+          0.003053748984540415,
+          0.014358356168574803,
+          2.8186924071913628e-05,
+          -0.006181789640967465,
+          0.01947730374003864,
+          -0.028739300217145797,
+          -0.023460503178221397,
+          0.0012460072421725688,
+          -0.008850395570726385,
+          -0.016271242375881602,
+          -0.035896509136022194,
+          0.005227881053956234,
+          0.03324337105109496,
+          0.004970419987430122,
+          0.008273497228454482,
+          0.01814500594735864,
+          -0.02229593571615308,
+          -0.024230675455380678,
+          0.000639112529814716,
+          -0.010266640429960994,
+          -0.010257247303470066,
+          -0.021302483296731352,
+          0.019150757066356872,
+          0.01651375496287087,
+          -0.00390189952706486,
+          -0.006971692709239087,
+          -0.010621003777733253,
+          -0.01918206725910585,
+          -0.02020345113211036,
+          -0.013827322378214754,
+          -0.017980206764311026,
+          -0.034153741666571154,
+          -0.032195001867229935,
+          0.03225205339201753,
+          0.026354817588919596,
+          0.0014076409241492998,
+          -0.007959081015805435,
+          -0.023070419006792155,
+          -0.004483328183581872,
+          0.02879516595345542,
+          0.002549934782150634,
+          -0.0001541849716285838,
+          -0.00830090759167373,
+          0.015341095409025125,
+          0.00835695019244165,
+          0.005107844984733648,
+          -0.06676974053142512,
+          -0.006027540973808174,
+          0.01655283585235268,
+          0.01351155740684989,
+          -0.002359114145814062,
+          0.02591301169926259,
+          0.01108099264396926,
+          -0.015361038608114896,
+          -0.019765743462262668,
+          0.035195761515422364,
+          0.02408910733949848,
+          0.023468767772473746,
+          -0.0034492999385544727,
+          -0.004472311490618405,
+          0.007030833258457338,
+          -0.010684252427867153,
+          -0.013326964647936429,
+          -0.004558513842504658,
+          0.026925146859337397,
+          0.016099443021514896,
+          -0.0064331198280825575,
+          0.010848931851939124,
+          -0.0008320209125061849,
+          -0.019444491995041537,
+          -0.001841330829156697,
+          0.008730275423516445,
+          0.0012974082030866743,
+          -0.014611902493400719,
+          -0.005911807768916134,
+          -0.019130604101278263,
+          0.024719468317569562,
+          0.0209456329723249,
+          0.03381727436368338,
+          -0.013237902591972769,
+          -0.0003695142331921278,
+          0.005410360755058108,
+          0.0002841632253252816,
+          0.0023352645111704958,
+          0.004801123870325242,
+          0.02623649813035263,
+          -0.029010710171522985,
+          -0.020741774066460415,
+          0.0068426980682519505,
+          0.013445270043414486,
+          0.0022744032984779294,
+          0.033840485942696216,
+          0.035040080489222324,
+          0.004166066423508316,
+          -0.010046666684988869,
+          -0.006148765564658713,
+          0.007071740742464309,
+          0.01865897335451031,
+          -0.023267411753900076,
+          -0.0005936657905496225,
+          0.004833030519903764,
+          -0.004229582648167378,
+          0.03620610659071198,
+          -0.009961238880412731,
+          -0.007197670122096852,
+          0.022212791947347127,
+          0.09819205436926473,
+          0.00047695621444943017,
+          0.0025354521225913395,
+          -0.029358210234742176,
+          -0.023614016211567963,
+          -0.0012564198366457442,
+          0.00242680708457405,
+          0.00830167274308547,
+          0.028076260517106013,
+          0.0004852400049405906,
+          0.011579274037419117,
+          0.02773972145320728,
+          -0.017217305534161797,
+          0.006752070301141707,
+          -0.011376051474975524,
+          0.024577071515205502,
+          -0.012694034026935636,
+          -0.0019809719792138673,
+          -0.016512649783050146,
+          0.015907402258634816,
+          0.004556574687833768,
+          0.004907510649166062,
+          0.011347630984408937,
+          0.009824484201337644,
+          -0.02650731900265866,
+          0.01756773113723491,
+          -0.005671986127099093,
+          0.004449611508541289,
+          -0.027841346572384462,
+          -0.020124233970502267,
+          0.0025986391930124847,
+          -0.0014440758308989809,
+          -0.009579444248630085,
+          0.006005274872617889,
+          -0.0024984235450999127,
+          0.03765664613203687,
+          -0.03765496603611941,
+          0.01409350051861866,
+          0.0066453175608372555,
+          -0.0015390841089599882,
+          0.011806653942640485,
+          0.017306914776093522,
+          -0.008206256939290151,
+          0.02589800448884443,
+          -0.00798213351353373,
+          0.009846274212000523,
+          0.015230070803055296,
+          -0.03449531102975745,
+          0.009174739184806312,
+          0.010145762060174144,
+          -0.024165660737765746,
+          0.012146877603301336,
+          -0.013784371016298891,
+          -0.004818759788054113,
+          0.005930446700613062,
+          0.007349837554764034,
+          0.010733731974181452,
+          -0.02295571464086674,
+          0.016176424826629898,
+          -0.010947841926210105,
+          0.0037766665765835445,
+          0.013559846998199008,
+          0.024892849808979323,
+          0.0012426907454129122,
+          0.0005950515136086193,
+          -0.01379469431504576,
+          0.005534724429523084,
+          -0.015667467221535825,
+          -0.022401586867694584,
+          -0.014192041775234079,
+          -0.006366846766730035,
+          0.01029598687444803,
+          -0.003972386708834858,
+          0.016971476088808737,
+          -0.010309648640697812,
+          -0.010645860303528099,
+          0.0037818310327362103,
+          0.01040806357838676,
+          0.008316605962234849,
+          -0.02855914079113881,
+          -0.005676310556150579,
+          0.004961084138163161,
+          0.00029285971503001265,
+          0.009592497954559435,
+          -0.014556073011576487,
+          0.013689526487160009,
+          -0.013290113354287095,
+          -0.0010726537498214612,
+          0.013201860397060236,
+          -0.012443552043930505,
+          -0.004012305744087929,
+          -0.01592905426965635,
+          -0.028487066798154617,
+          0.008462143660968396,
+          -0.01365614159523485,
+          0.001076882584445294,
+          0.0019503240125505646,
+          0.004730085646042709,
+          -0.016919069570642004,
+          -0.007860046623166481,
+          -0.033515448866521046,
+          0.011065416478687504,
+          0.02314589687882036,
+          -0.006435733825418645,
+          -0.030351727858789472,
+          -0.009546333526852862,
+          0.011327489786114963,
+          -0.0071765122743172715,
+          -0.01676736383051139,
+          -0.0029390038145944096,
+          0.007617910650439855,
+          -0.008174620754097493,
+          -0.00508943241317381,
+          -0.0025173969010414098,
+          0.0217328037301181,
+          0.016094975605203832,
+          0.01769642142712967,
+          -0.021400862603034036,
+          -0.00976992221747187,
+          0.026043879119776864,
+          -0.03275550610951898,
+          0.012441465975914529,
+          0.024587446266013183,
+          0.003446527051428522,
+          0.005492228054838714,
+          0.0030377554109693714,
+          0.005459159163772825,
+          -0.006985179226831048,
+          0.019925583798328282,
+          0.010870421035240845,
+          -0.019709062313471467,
+          0.005385443078244229,
+          0.02086085539375208,
+          0.0037248737640193097,
+          0.012892299156840998,
+          -0.017546870090541213,
+          -0.0020341807891543716,
+          0.015594779287017958,
+          0.04816214672670215,
+          -0.0032851797178091685,
+          -0.010450846098750195,
+          0.029513742590677183,
+          0.023526004023637855,
+          0.005373455940309873,
+          0.0012120777808635214,
+          -0.02525630456877349,
+          -0.013075130701774785,
+          0.01962235086017781,
+          0.016944870494393874,
+          0.013484925533584295,
+          -0.0023255656480479004,
+          -0.019236081983457366,
+          0.021520156296467717,
+          -0.01701733319731161,
+          0.012422659010697477,
+          -0.023764941037007197,
+          -0.001490790833263327,
+          -0.0035419008954878724,
+          0.021998949056452454,
+          0.0021819635638621092,
+          0.008891505304514552,
+          0.024354552127458608,
+          -0.006978548952551799,
+          -0.019526615985982355,
+          -0.0051799020400150305,
+          -0.014578953088901798,
+          0.022506635912517146,
+          -0.024117708614271965,
+          -0.007008641019892289,
+          0.011379795968360019,
+          -0.012362093847309038,
+          -0.0102534051092702,
+          0.001164934138876502,
+          0.017686807394319725,
+          0.0009423993435129515,
+          -0.014784508280754078,
+          0.01944445593807992,
+          -0.0011359271135469456,
+          0.010410915597949927,
+          -0.01921419010977189,
+          0.009390638453648025,
+          -0.016682442516992777,
+          -0.007849184048209236,
+          0.010343008873053859,
+          0.0012604521199837046,
+          0.003948300703645506,
+          0.002592490121754231,
+          0.007773437307969315,
+          -0.03355658426684504,
+          0.022695505487314516,
+          -0.010751606023585792,
+          -0.003538523688786008,
+          -0.0014125330167969218,
+          -0.01678692701672328,
+          -0.0006918162525881059,
+          0.005816502852598842,
+          0.010878358093847386,
+          -0.016045147059003256,
+          -0.00873270513444376,
+          -0.04920244395446931,
+          -0.030314009649638693,
+          0.023156333024481057,
+          0.013733657599720879,
+          -0.010971506342695331,
+          0.018852159415605117,
+          -0.023760853159532587,
+          -0.010587249404421732,
+          0.0011379869499042905,
+          0.023319856329056813,
+          -0.03539290744576506,
+          0.01215369703816749,
+          -0.014493682360886549,
+          -0.02039288669534806,
+          0.009070984596830142,
+          0.011445354735283254,
+          0.0007259852045513916,
+          -0.0011514627555568814,
+          -0.029350945123925453,
+          -0.010246845821727087,
+          0.02391068177425878,
+          -0.00861718589055971,
+          -0.004723044986826333,
+          0.024716520302409263,
+          0.0033721801437248873,
+          -0.012524663490691517,
+          -0.017955758068233754,
+          0.021612671655463236,
+          -0.026077203994996058,
+          0.007645944309444457,
+          0.010032183011253845,
+          -0.0173991263874487,
+          0.021649599230588117,
+          -0.007529701797485131,
+          0.020327763440441377,
+          0.004227390938169872,
+          -0.011249233530251522,
+          0.020352283974471443,
+          0.013644443758970054,
+          0.004216958800094986,
+          -0.004081669404784771,
+          0.006869353098014179,
+          0.02212732880780841,
+          -0.011021739614266063,
+          0.02219580717345023,
+          0.028777213295471067,
+          0.00017328191188631918,
+          0.03260179554743322,
+          -0.021287655520443285,
+          -0.03159728983004563,
+          0.021740163175540124,
+          -0.01035146858119169,
+          -0.021564906188712388,
+          0.028914245422221707,
+          -0.013674731836508768,
+          -0.00786476052825481,
+          -0.0166718429993922,
+          0.00533027311342322,
+          0.016680336311088843,
+          -0.005246190451159961,
+          -0.014360934066628332,
+          0.0029628103834216345,
+          0.019809487418628386,
+          0.014587234544007958,
+          0.029780009648869975,
+          -0.010778199815920887,
+          0.035180242185168895,
+          -0.002320576872571577,
+          0.008990157317473909,
+          0.012415465232073404,
+          0.006489435152972723,
+          -0.011251531176923445,
+          0.013501894212117511,
+          -0.003446759011266526,
+          0.008230548711588447,
+          -0.001449023474280409,
+          -0.035111679577202855,
+          -0.013709382199848678,
+          -0.04709294504969336,
+          -0.0023752976043235697,
+          -0.0025098360125255215,
+          0.0030975417740913154,
+          0.09736815869389318,
+          -0.005723049647039619,
+          -0.011988817571623457,
+          0.001625059666070763,
+          0.009611697638805556,
+          -0.0009896494420056814,
+          -0.007606836952841873,
+          0.015973152482488318,
+          -0.01343292635775299,
+          -0.02140928016038583,
+          -0.010532701190806147,
+          -0.01288594454368977,
+          0.012079198190522715,
+          0.015929716343161652,
+          -0.0005947359502228821,
+          -0.015207331663761012,
+          0.006524010655124294,
+          -0.004566385899752684,
+          0.0064717704997400245,
+          0.008647099975176308,
+          -0.010414668624343924,
+          -0.0071698879102925555,
+          0.030899143952119196,
+          -0.025301269549419674,
+          -0.024361748354052457,
+          -0.0010579931222861095,
+          -0.009055796737486008,
+          -0.010335459737012047,
+          -0.002692653633719985,
+          -0.0017679501322462056,
+          -0.010519994985692081,
+          -0.010019865263564572,
+          0.006537076627706128,
+          -0.015472130358273033,
+          0.004812857776601667,
+          -0.0047457011269495634,
+          -0.013296069496526007,
+          0.017054968086538724,
+          0.022563785357424853,
+          -0.02841491200807702,
+          0.0006755428946168761,
+          -0.017422510783648978,
+          0.012941211381006863,
+          0.008866069010589792,
+          0.006049618782357251,
+          -0.022451528209097505,
+          -0.023622906582403243,
+          -0.015019272754405791,
+          -0.01927535750568903,
+          -0.024973490129133732,
+          -0.009073381957684092,
+          0.0067692803507600945,
+          -0.001267339788153158,
+          0.021918281111033645,
+          0.000816538714279555,
+          -0.01812545613159305,
+          0.0061084913570894905,
+          0.000568371220952269,
+          0.018292497018668842,
+          -0.027468677288882186,
+          0.0033806422195878524,
+          -0.02307582511848465,
+          0.033508234091571124,
+          0.0039195050746594975,
+          -0.026680111514794824,
+          -0.021558668725755275,
+          -0.003311830287529699,
+          0.018994577319066473,
+          -0.0025745912019675805,
+          -0.00724049447539589,
+          0.014775207617625293,
+          0.0155334760323838,
+          -0.004693118176196483,
+          0.002015625406296403,
+          0.027968079002437102,
+          0.01077998866180723,
+          0.0313892866933904,
+          -0.015236169297590675,
+          0.004945828495617267,
+          -0.027215016713379062,
+          0.010214419950212732,
+          -0.01503548137805401,
+          0.008859554812646242,
+          -0.01290890682557532,
+          0.03531384210182567,
+          -0.012005613861257432,
+          -0.004250698950192301,
+          0.03253288796441852,
+          -0.029150986393345252,
+          -0.0035020051536906334,
+          -0.009255437891304041,
+          -0.004142204892160768,
+          0.013240580426083011,
+          -0.001198093874819309,
+          0.008485141356876356,
+          -0.01854370462655459,
+          9.726096864073609e-05,
+          -0.003221852304410318,
+          0.005404595442127598,
+          -0.02005458688172967,
+          -0.003314118809778734,
+          0.0073427568419969814,
+          0.004259518237708634,
+          0.002408630356035373,
+          0.016420646142725346,
+          0.006662121128241274,
+          -0.0029357701493526907,
+          0.03424224243607692,
+          0.008171895850772245,
+          0.0152521424877615,
+          -0.0018638264193202728,
+          0.0030234211261443034,
+          0.019287963516760636,
+          0.019912956661192856,
+          -0.00048443399729935116,
+          -0.01058935153883094,
+          0.00195113175754497,
+          -0.001140732137433033,
+          0.011048741602327437,
+          0.024396046054912645,
+          -0.021158025221644738,
+          -0.02419562092713724,
+          0.016377661005279716,
+          -0.015076145951740961,
+          0.01903712233378994,
+          0.010541463112889985,
+          -0.010099351302095792,
+          -0.00031558616108876247,
+          -0.042316016201379245,
+          0.010964989031040692,
+          -0.02068939391415103,
+          -0.04004120028915868,
+          -0.00244383573783919,
+          0.0009542507389088937,
+          -0.029439456243505052,
+          0.02822814855547978,
+          0.0023755032182188156,
+          -0.017280221740813364,
+          0.018223709137876153,
+          -0.005548654680726505,
+          -0.0024040301239229064,
+          0.005397711787238679,
+          -0.01092026090471121,
+          -0.010648485086769927,
+          0.008235250252660972,
+          0.010957715075632535,
+          0.005425683656535705,
+          -0.005698919020608095,
+          -0.011284911899894062,
+          0.009975882651638084,
+          0.012318901214538077,
+          -0.0019922884373083155,
+          0.010050520101815593,
+          0.0031720752428247446,
+          0.007352859750323051,
+          -0.0073483049386141005,
+          0.0046483587496626965,
+          -0.005093220128411125,
+          -0.026080057085901446,
+          0.012863580653903103,
+          0.0052434400313520774,
+          -0.011863360599637747,
+          0.0103209502958491,
+          -0.01048162756158923,
+          0.005787157766911021,
+          0.001318945460232246,
+          -0.0059333230422061395,
+          0.004821320238893602,
+          -0.0038922239931512784,
+          0.011267634571983924,
+          0.013256611037442042,
+          0.004592529900805719,
+          -0.013076954459595086,
+          -0.02177318635989856,
+          0.009001291224623467,
+          0.01959767163356408,
+          0.024161754192119852,
+          -0.030416109293488064,
+          0.003784634588325551,
+          -0.0014915849060829743,
+          -0.0020550564975080885,
+          -0.015632785947151753,
+          0.01587790981601866,
+          0.01716742056401084,
+          0.012721563202528083,
+          0.01893393006967701,
+          -0.004009617703771589,
+          0.010539100665843704,
+          -0.02080068888791658,
+          0.013642451766699067,
+          0.0008609869904066794,
+          -0.013629067231663643,
+          0.0014620655123815298,
+          0.027046304278766087,
+          -0.011056783825898616,
+          0.006206969699250696,
+          0.008097527175511791,
+          0.022111870736907183,
+          -0.012509479100313492,
+          -0.003735868296238069,
+          0.01516254968167132,
+          0.0014878674796313539,
+          0.0035887079417890352,
+          0.016533149527222273,
+          0.04729052276110659,
+          0.00102751186013391,
+          0.0016713773449738153,
+          0.0192177920520243,
+          -0.005563519063161466,
+          -0.01114821419109643,
+          -0.003549999863325982,
+          -0.021807893906469106,
+          0.010998558936481785,
+          0.00431279785115251,
+          -0.02186324212032519,
+          0.01024093366826206,
+          0.0125495473550584,
+          0.005268320135120931,
+          -0.0031679212410026603,
+          0.02574575247827441,
+          -0.019807879421996258,
+          0.014935295894965128,
+          0.03329450628384787,
+          0.00493356778223407,
+          -0.011423949333937285,
+          0.007263264249871384,
+          0.0027354459678742013,
+          -0.004494406041906684,
+          -0.00606745413660583,
+          0.010497391290734088,
+          -0.008191702034391382,
+          0.009870458191510176,
+          -0.009580130479693,
+          0.039202091799313755,
+          -0.009619018331262844,
+          -0.010782138248435458,
+          -0.004558645539551881,
+          -0.006241773108460662,
+          0.003478414182938609,
+          -0.003395165694642073,
+          0.005812225135633631,
+          -0.0069045407353789945,
+          -0.014607146220271209,
+          0.0008522982757210454,
+          0.00020175973815948303,
+          0.014001922919116725,
+          0.0011804904415274088,
+          -0.004871874532755353,
+          -0.007877217154706205,
+          0.015243277056007988,
+          -0.006047906208304912,
+          0.0006531937033317192,
+          -0.0011653523616313422,
+          -0.0013662556399359335,
+          -0.007401050012179761,
+          -0.015166105795085528,
+          0.00974609244984173,
+          0.016253733908242134,
+          -0.013857834499211056,
+          -0.005142936208363415,
+          0.012274716212832219,
+          0.008540229125193836,
+          0.011742606168187876,
+          0.0007451338650642946,
+          -0.009255813682572937,
+          0.008292531177374192,
+          -0.015020523035432158,
+          -0.008781317965985481,
+          0.0026250469418136603,
+          0.002381147382977664,
+          0.005839723981046825,
+          0.0006240641828668715,
+          0.004502463508605272,
+          0.005798912705218846,
+          -0.00497936644919367,
+          0.01112666823857836,
+          0.008198424171006548,
+          -0.002452143482105151,
+          -0.01017569410837814,
+          -0.001127589984272638,
+          -0.015733298167661346,
+          -0.016758315411028353,
+          -0.0009286266498936771,
+          0.015175988113750882,
+          -0.02352862898718829,
+          0.018996352975391644,
+          -0.0014524363611708201,
+          -0.0033802546171857704,
+          -0.021392428456030312,
+          -0.0025923983911570183,
+          0.0034214188668518796,
+          0.01255768230394476,
+          0.01858853630745928,
+          -0.02729543519465384,
+          0.010575894388563116,
+          -0.007204725245442141,
+          0.030394929424014717,
+          -0.018328782140005044,
+          -0.004022941273610227,
+          0.026915618079621992,
+          -0.02200018306828758,
+          0.0008791185377513978,
+          -0.011454606955158956,
+          -0.008103032693853504,
+          0.019287674837688177,
+          0.023747720480736777,
+          0.02038287242167783,
+          0.01364489294869211,
+          0.0007758044919789085,
+          0.004768030222445397,
+          0.03572705648994403,
+          0.010568241270862248,
+          0.014828087092859878,
+          -0.007542489523110974,
+          9.457808640264811e-05,
+          -0.02554033312739386,
+          0.01635899434369978,
+          2.587383534566272e-05,
+          -0.012370727506722627,
+          0.007162422186621379,
+          -0.013370882403931942,
+          -0.011066483600541058,
+          -0.007151077450730025,
+          -0.01721910018182534,
+          0.02152239849400241,
+          -0.012322440280680164,
+          0.004587729883596212,
+          -0.009079747696669277,
+          -0.013272707286450417,
+          -0.012953682633400515,
+          -0.014202214807187545,
+          -0.01313141988034687,
+          -0.026041403779538307,
+          -0.027109731260915763,
+          -0.006573285295895077,
+          -0.023780366346334524,
+          -0.013799072648505526,
+          0.005588622594136253,
+          -0.022617902666316757,
+          0.012262325666428604,
+          -0.012689827812482627,
+          -0.01226264872059864,
+          0.012805734807835008,
+          0.020274844315616616,
+          -0.011765231803328348,
+          -0.028494489703400308,
+          -0.018583843821043854,
+          -0.019970998470704358,
+          0.012576201137505969,
+          0.002266288538944273,
+          0.013165889075444961,
+          -0.003018437854660793,
+          0.003227412667880239,
+          -0.014895831590298957,
+          0.018570624675927183,
+          -0.03442390703780963,
+          -0.03108003806963536,
+          -0.013611306765348361,
+          0.005182777066476028,
+          -0.0015782033203578553,
+          -0.011429046504256945,
+          -0.0018606648572920801,
+          -0.00433805271182878,
+          -0.025109572024446088,
+          -0.010134856632190067,
+          0.011010662054484161,
+          -0.015205629927469414,
+          0.0072933565255675036,
+          0.0009018832564038863,
+          0.013870902834260887,
+          0.009587432944616442,
+          0.013393301893492882,
+          -0.006095063698384775,
+          0.004356485794113208,
+          -0.0026476612747324386,
+          0.0004894024149759108,
+          0.009336366919474478,
+          0.00839591561027103,
+          0.010551161369079399,
+          0.002260707777260338,
+          0.007830059223001696,
+          -0.0060927446216633265,
+          0.018065002687169148,
+          0.015436944875538481,
+          -0.0002520689442496089,
+          0.013323098485251487,
+          0.0003769862591044255,
+          0.00022293304921127018,
+          -0.0029970303106340466,
+          0.007888437018607987,
+          -0.011785957933642163,
+          -0.001311704406355858,
+          0.015138190893483804,
+          -0.01707247730569978,
+          -0.013045187198628258,
+          0.03107034781283638,
+          0.0018583638488268697,
+          -0.02495564203304796,
+          0.031146554085787762,
+          0.004220047610942943,
+          -0.005835199594223787,
+          -0.013205086624848782,
+          0.0014059047822222418,
+          -0.011570624844100298,
+          0.003584232809362274,
+          0.014295032061446826,
+          -0.02231984223610487,
+          0.010521094801839404,
+          0.008446892441796151,
+          -0.009681535919980559,
+          -0.012337659795586223,
+          0.009847073709209602,
+          0.016809328445200358,
+          0.004472485504034416,
+          -0.03539093810210523,
+          -0.011957694102753411,
+          0.02145470732008716,
+          -0.029237428225009913,
+          -0.00810041956259673,
+          0.007452962400105787,
+          0.0065154832040426065,
+          -0.019353109017609053,
+          -0.008580955020412362,
+          0.005725794605897509,
+          -0.004940390374637084,
+          -0.003892210172017758,
+          0.017541623128534682,
+          0.00485047043626186,
+          0.013698341792101025,
+          -0.01689738578307474,
+          0.009771458057400557,
+          -0.022222817828602557,
+          0.004021882074676753,
+          -0.023373700500461992,
+          -0.009873403167702603,
+          0.002364143609079849,
+          0.012454271200139762,
+          0.014328556467632212,
+          0.027839363143659326,
+          0.030056404238348527,
+          0.013995696971422429,
+          0.007833677585250953,
+          -0.016265199386953717,
+          -0.018134327237616175,
+          -0.006409513840075933,
+          -0.004033173350204784,
+          -0.008714798059218596,
+          0.0066373547199279235,
+          0.01858354212009877,
+          0.004499712802472377,
+          0.02609078206670711,
+          -0.006227189596272876,
+          0.009039744622014585,
+          -3.127526911951914e-05,
+          0.0008833930131827993,
+          -0.004226083827636182,
+          -0.01971685546584527,
+          0.00481003211084742,
+          -0.0013721482162825508,
+          0.025874758880591167,
+          0.00940496860916206,
+          0.0024915013399541847,
+          0.01615978314811047,
+          -0.006378952757397484,
+          -0.01174653607178726,
+          -0.010540107733238588,
+          -0.0061926268398085176,
+          -0.021752582127981104,
+          0.029059100503124387,
+          0.021195595568253612,
+          -0.010800765264123782,
+          -0.013693582166598082,
+          -0.00903369526059487,
+          0.02795767990556847,
+          0.004080965688836953,
+          -0.0013050212744006933,
+          0.016313311850308505,
+          0.012084555958999215,
+          0.012329325712491252,
+          0.009794698207689061,
+          -0.00036157148861933123,
+          0.0015364709122308352,
+          0.03722522141440073,
+          -0.007427028510548186,
+          0.0026463292682538495,
+          -0.005239324411651515,
+          0.0025987932868461313,
+          -0.009705817520422487,
+          0.007242122655628283,
+          -0.004012676385296197,
+          -0.0158760051470016,
+          0.015003444174308607,
+          -0.022443331305169994,
+          -0.0016749787204471238,
+          -0.008150635190952799,
+          0.0016696640640947303,
+          -0.02781023337840043,
+          -0.0025731206607371845,
+          0.006695065701630532,
+          -0.015547340828489069,
+          0.011816010501093885,
+          -0.01740673229051557,
+          -0.011553604323149105,
+          0.007914961618894248,
+          -0.012034157288065447,
+          0.0016684130271324446,
+          -0.0008268957621923278,
+          0.004460864332388434,
+          0.013426055079012884,
+          -0.011008808570221245,
+          0.012166046631826273,
+          -0.005816284310138776,
+          -0.007872091214834688,
+          -0.0016055129487028885,
+          -0.0030049472973942043,
+          -0.0025600746761171543,
+          -0.0009966231170649917,
+          0.034355910234323155,
+          -0.004446570896048543,
+          0.003761242147499512,
+          -0.01362639743248326,
+          0.0019833052855181806,
+          -0.036176376394999044,
+          0.028344464817309477,
+          -0.008642835889481802,
+          0.011013502221008498,
+          -0.0017508365480865596,
+          -0.013243706268954915,
+          -0.01456517309747086,
+          0.015247299501245178,
+          0.007602596572394442,
+          -0.005854531057349665,
+          0.013558248087347848,
+          -0.019724373614448298,
+          0.008530500822910953,
+          0.004463694543554244,
+          -0.015561397976987216,
+          0.004668780190328835,
+          -0.007479912658259757,
+          0.012657201279639194,
+          -0.01774730033362932,
+          0.004314594593150765,
+          0.00958571617187387,
+          -0.01892876084898881,
+          0.009901041800918215,
+          -0.028675879564512306,
+          -0.004631920242208613,
+          -0.012323455524366223,
+          0.0053804608827285865,
+          -0.000660864620708182,
+          -0.02389088252276507,
+          -0.028944280660230762,
+          -0.0002099667627277945,
+          0.00458533068247518,
+          0.0018878351997997833,
+          0.011621407953503755,
+          0.005970807015151215,
+          0.008542958629199003,
+          0.01023896054756221,
+          -0.00205207247588841,
+          0.009402752124876478,
+          0.04169706217414899,
+          -0.022899912321317197,
+          -0.011574970753549647,
+          0.005321872977369877,
+          0.011499271505460519,
+          -0.02647653483798284,
+          -0.011039791758347888,
+          0.0033424345263339144,
+          -0.005219877979550397,
+          0.03609469691834737,
+          -0.02448746405881972,
+          0.019020618652020673,
+          -0.011392297116547224,
+          0.007625475937968714,
+          -0.021465240076078376,
+          -0.003921088264697311,
+          -0.014265745991522477,
+          -0.00011221615798127438,
+          -0.0031478236827989703,
+          -0.0076053733863387755,
+          0.007485039021296698,
+          -0.001907678661303243,
+          0.0020348246022161925,
+          0.006291436821831794,
+          -0.02173305455485444,
+          -0.0019125499969493854,
+          -0.016432460924913166,
+          0.030367263349759286,
+          -0.016242095671439496,
+          0.005072805486919414,
+          0.0053258139728101506,
+          -0.002810639419737258,
+          0.012368771814660942,
+          0.004165967191536186,
+          0.00513754440091394,
+          -0.004199767241827414,
+          -0.005142642360546972,
+          -0.0053886448467479215,
+          0.017868874549760376,
+          -0.013280048598849825,
+          0.035024701466710065,
+          -0.0100959186045929,
+          -0.007094001877261568,
+          0.0019733042325232155,
+          0.010380430710282744,
+          4.2084202709296443e-05,
+          0.01053565670076933,
+          0.0034270489770133685,
+          -0.0006311078385359693,
+          0.0019774779328212756,
+          -0.017162448473652843,
+          0.003089758682621882,
+          0.01682656561693936,
+          0.005463677441664502,
+          0.004469261499451007,
+          -0.013332130891003062,
+          -0.012050233723282738,
+          0.004538268704442613,
+          -0.007533940408255447,
+          0.0003675737140813486,
+          -0.018816953644773076,
+          0.013972696748575243,
+          -0.0011096683686631858,
+          0.0002715683052236706,
+          0.00524678369365015,
+          0.023056514997908218,
+          0.01813167860994747,
+          -0.002553352602551003,
+          0.03155164978556312,
+          0.0174738879054593,
+          0.0029268039949260673,
+          0.010637295711777793,
+          -0.033222781195984666,
+          -0.013301873512972214,
+          -0.0013721703891642354,
+          0.02238421933149517,
+          -0.01530189703335162,
+          -0.014059029663525449,
+          -0.0037393534403749947,
+          0.0010394601933309767,
+          -0.005177359933828505,
+          0.0042489306011644495,
+          0.011039889511148341,
+          -0.014697595240068863,
+          0.007597149702441915,
+          0.0026780496822907634,
+          0.02092734730183164,
+          0.0004688144432243786,
+          0.017181410417231698,
+          -0.0005083970646221274,
+          0.00749639055342427,
+          0.007319420826337891,
+          0.017541463125355265,
+          -0.01002811389296805,
+          0.023582113845081534,
+          -0.008823933397729605,
+          0.0019889258902730535,
+          -0.01257106158422629,
+          -0.028159594067544737,
+          -0.012624750220149823,
+          0.015247564545279736,
+          0.005765911899842925,
+          0.006478299226263732,
+          -0.016537090297144265,
+          0.006276997440048981,
+          0.015647122171767974,
+          -0.002251683151059454,
+          0.010295358869874306,
+          0.009730378299068289,
+          0.00012816405890877098,
+          0.010181120688239766,
+          -0.00801256937783534,
+          -0.00734257767723293,
+          -0.004497520124843868,
+          0.023982754195729983,
+          -0.005443550092842393,
+          0.022434177122972028,
+          0.012647958478747348,
+          0.00616791857573897,
+          -0.010853355357954727,
+          -0.003340111254357573,
+          0.023389634440057473,
+          0.011863246790550376,
+          0.00029642699874398923,
+          -0.009846034060495678,
+          0.0019458876579524617,
+          0.0054010373765486005,
+          0.012983913038716568,
+          -0.02049427270240938,
+          -0.004344571352918878,
+          0.008461947538684664,
+          -0.008472919299762712,
+          0.004318309005722874,
+          -0.030156444522730416,
+          -0.005812303237222271,
+          0.021354352236942434,
+          0.008392281015122372,
+          -0.010232039722494646,
+          -0.004395682831498155,
+          -0.0006527587802367882,
+          0.0040601850269714945,
+          -0.0024323167053212176,
+          -0.0023729138545912176,
+          -0.008541233856441836,
+          -0.006470933419694742,
+          0.014035696617797647,
+          -0.0052691760528871924,
+          0.008597459268477037,
+          0.00012084859681905111,
+          -0.0017255958688825231,
+          -0.007323529232751411,
+          0.03382792341957058,
+          0.008087732655845718,
+          0.005980666974986827,
+          0.0013613625176327891,
+          0.0042244330945601215,
+          0.000616791811672998,
+          -0.0028374856646453707,
+          0.018075360591135542,
+          0.0156738939725117,
+          0.005054696911814988,
+          0.009038525980341447,
+          0.014672270369917403,
+          -0.019700405219230957,
+          0.012625337070556061,
+          0.012184185186281482,
+          0.019382321278525114,
+          0.0007219498737851579,
+          0.002564970959687783,
+          -0.010601873823737926,
+          -0.0029963537053764736,
+          -0.002070567089566093,
+          0.029563028293936092,
+          0.0005903976560069006,
+          -0.010509758568838935,
+          0.010992421945232951,
+          0.020603868018231348,
+          -0.02101157589451638,
+          -0.001971974816402967,
+          0.005963911670920596,
+          -0.010704207628592966,
+          -0.014466113540150018,
+          -0.021269176952661614,
+          -0.013331247081394942,
+          0.0071597345125287174,
+          -0.008162102923896894,
+          0.02367549407145414,
+          -0.014256771777502395,
+          -0.03960796112042958,
+          -0.014803626428839152,
+          -0.00864320347698291,
+          -0.027531337207277483,
+          -0.007305441146142331,
+          0.00965056093889974,
+          0.016698589600806273,
+          -0.003330867073501157,
+          -0.025198703196464594,
+          0.030483790406960656,
+          0.008872821868559751,
+          0.023708422767352917,
+          -0.021464374508407488,
+          -0.0018617153194555344,
+          -0.00886376784514331,
+          0.014505918224188001,
+          -0.017588761818663975,
+          -0.011756112651591795,
+          0.020014632992762026,
+          0.014356419813769714,
+          0.007246557500044434,
+          -0.004355132555323843,
+          0.0067299398462013936,
+          0.007649499235136303,
+          -0.012515181210363267,
+          0.012239792251154642,
+          -0.013499277313916029,
+          -0.024858295126228433,
+          0.014430561986516884,
+          0.002229494935179828,
+          0.009563004428064055,
+          -0.0015984171085659323,
+          -0.016197659608768088,
+          0.010078159490026602,
+          0.00808729175515505,
+          -0.017508840440512225,
+          -0.004793934415922065,
+          0.007133527615305415,
+          -0.020818869260580315,
+          0.031234686259160864,
+          0.011872765152279094,
+          -0.010692949770503467,
+          -0.001989039629661815,
+          0.004431079028880923,
+          0.016511544608975148,
+          0.007893700385963683,
+          -0.01997314388229276,
+          -0.003708006454566117,
+          0.01530128306203401,
+          0.00822588305477249,
+          0.004861703074728471,
+          -0.010836328783594584,
+          0.03434874763691911,
+          0.006082873924472328,
+          4.192481378506604e-05,
+          -0.0052997095110967115,
+          -0.0076108924981838355,
+          0.004196112457289409,
+          0.018905391806961686,
+          0.013330078288599763,
+          -0.004200590783114919,
+          -0.006630107527854746,
+          0.007778448128358027,
+          -0.01706991160178345,
+          0.007329056462125937,
+          0.009390327499494826,
+          -0.012870674959022891,
+          -0.0017571189976160295,
+          0.016462608870101713,
+          0.00209846661139665,
+          0.004942714850185376,
+          -0.011305050739130339,
+          0.004252776146827917,
+          0.015841821944109317,
+          0.018727567181082646,
+          -0.008399642221255733,
+          0.021448055799259867,
+          -0.020972459569228434,
+          0.027243639465535856,
+          0.028499400850980086,
+          0.016573144403275928,
+          -0.01847096443459798,
+          -0.0022409373140116146,
+          -0.0041737366139895414,
+          -0.014761716560267328,
+          0.0034546020161337894,
+          0.005125870074458691,
+          -0.01174186608185822,
+          -0.009577921232380895,
+          -0.024331993794416026,
+          0.011075959943215117,
+          0.007918147902741738,
+          -0.0015045185909419564,
+          -0.011791155919533844,
+          0.016019077577552835,
+          0.010141868950981925,
+          -0.010938402325619172,
+          -0.005635351352447256,
+          -0.019967361901734836,
+          0.009750427567651833,
+          0.009128294017175602,
+          -0.00300556760891084,
+          -0.011496103417643132,
+          0.012977890524465716,
+          0.0014080059843484115,
+          -0.01292380658817521,
+          0.010841954636959441,
+          -0.03142011651894961,
+          -0.005510537354374977,
+          -0.011027026734432255,
+          -0.003620482919145167,
+          -0.014415606143566519,
+          0.003608572766363589,
+          0.0005034611321129689,
+          -0.011470488351158329,
+          -0.0035005774696278,
+          -0.016207453351921627,
+          0.004676327282262429,
+          0.015021513761697039,
+          0.001679614645967629,
+          -0.006917375094634678,
+          0.021994274901326064,
+          0.018518524483306403,
+          -0.021903538417919984,
+          -0.020116013379669034,
+          0.004708084983787958,
+          -0.03261690161800251,
+          0.015019348187107588,
+          0.00966124471577622,
+          0.011676937880751252,
+          -0.028768229226408037,
+          0.007114534979415504,
+          0.00900420165933673,
+          0.016398927481511824,
+          0.0050879301160439245,
+          0.009453255486450082,
+          0.01222758288108498,
+          -0.0104657614414275,
+          -0.003164502778525433,
+          -5.410382753651127e-06,
+          -0.005618225986727351,
+          0.015607620507415484,
+          0.0008527697935056332,
+          0.03603666488542239,
+          -0.005277207843139309,
+          -0.0009799879252482288,
+          -0.011735731235560851,
+          0.00812046801749312,
+          -0.011713349784345638,
+          0.006334172347602515,
+          -0.0024643499727789167,
+          0.0001240009282616789,
+          0.002965067818077541,
+          0.009442572075213767,
+          -0.01086095783353607,
+          -0.007595379345224076,
+          0.007444897256325813,
+          -0.01482814340478338,
+          -0.00019984717819268013,
+          -0.009339343730980451,
+          0.0019680804106275156,
+          -0.02241031895876346,
+          0.006629838013824249,
+          0.040935644507527846,
+          0.02140216465404127,
+          -0.005481040712269146,
+          -0.012431659209559418,
+          -0.004680230194268222,
+          -0.007229300034633229,
+          -0.006482401275991039,
+          -0.004794698160419071,
+          -0.022762405726589375,
+          -0.006928553505112732,
+          0.004320321244630314,
+          0.013300138724212552,
+          0.009741018117644266,
+          -0.008842135469729511,
+          0.004531760574809748,
+          -0.02662650543210291,
+          -0.016578081411548973,
+          -0.0060021425889846495,
+          -0.015840130031005147,
+          0.006125301421471048,
+          0.027521841635711787,
+          -0.0063432924325983525,
+          0.01789356321527744,
+          0.005142029893259461,
+          0.013034833065163448,
+          -0.005338869351330258,
+          0.00035250243782976347,
+          -0.006010938630154189,
+          0.0032958656042048292,
+          0.003074861351445954,
+          -0.0015947217852680714,
+          0.004151590621941346,
+          -0.007139285822575166,
+          -0.0026091937868967284,
+          0.0011629380311555433,
+          0.0117841884590861,
+          0.006877419488576261,
+          -0.00016252653621991807,
+          -0.01761285866765538,
+          0.022221104524808997,
+          -0.018933508559881868,
+          -0.008840161102111973,
+          0.011092684029552064,
+          -0.009660487856185668,
+          0.00702597023411946,
+          0.020387386977378873,
+          -0.006854213562767843,
+          -0.009583981919303215,
+          0.013028181087649146,
+          -0.0036507672754997286,
+          0.004464368543493108,
+          0.009867428978560914,
+          0.006036303973414791,
+          0.002482944923100156,
+          0.013244650094892797,
+          0.004530294330688855,
+          -0.0032625482538717137,
+          -0.020238533752799943,
+          -0.005186498720507101,
+          0.019015752262933585,
+          -0.02252458872825473,
+          -0.0027641215988942563,
+          0.0044506144202026456,
+          -0.0030317840657635457,
+          -0.01733962766050954,
+          0.004205861489436576,
+          -0.018770275720867473,
+          -0.0033883519277721433,
+          -0.008244668812709337,
+          0.009948090119054079,
+          -0.023700732627507225,
+          0.009109315916173436,
+          -0.011739192106106307,
+          0.02356509175177699,
+          -0.003925851077921686,
+          0.024734433836185024,
+          0.0015142645677087268,
+          -0.013482933030628261,
+          0.0055323884713736554,
+          -0.0008930661843169134,
+          -0.004477731568313757,
+          0.029338490404300612,
+          -0.005418258701133154,
+          0.002249996261871466,
+          0.013491124734151694,
+          -0.021332541202417246,
+          0.021773687217263173,
+          0.0046662696076723836,
+          0.01628697326539432,
+          -0.00986238951830334,
+          -0.013799209313714441,
+          0.025178874553620743,
+          0.03324555035648896,
+          0.008398796308070151,
+          0.011261509865513953,
+          -0.01226890254177634,
+          0.018966175575083295,
+          -0.0038328997984309597,
+          0.004344837325292706,
+          0.00878668880458693,
+          -0.007200842513742962,
+          -0.011500562922303248,
+          -0.004533014670236755,
+          0.027639894892383943,
+          -0.021838848944317424,
+          -0.008920629621611768,
+          -0.005739181836252042,
+          -0.01593272996094732,
+          -0.038167534267554606,
+          -0.004371701247332816,
+          0.019604405563120094,
+          -0.005974358141737469,
+          -0.0018341151594086288,
+          0.014334266690716505,
+          0.00402620072221496,
+          -0.002852569009185304,
+          -0.008195218707838913,
+          -0.0027944153624309856,
+          -0.0059390492900516225,
+          0.016651206007022145,
+          -0.004961864528562493,
+          -0.007929830585906699,
+          0.01145431100211762,
+          -0.005731957927601062,
+          0.012130694936714224,
+          0.0004848418056781251,
+          8.392048424584131e-05,
+          -0.0015174120840537315,
+          0.00822932723013239,
+          0.018620869354789082,
+          -0.0055803454666417465,
+          0.025682778991091174,
+          0.006956936033581638,
+          -0.003495709501259274,
+          -0.005489431577052735,
+          -0.0044154293119170445,
+          -0.024561016942073285,
+          -0.0020864636079347357,
+          0.006593060072668424,
+          0.024333813203596122,
+          -0.02515989187381735,
+          -0.004953266000463339,
+          0.020957806164031402,
+          -0.0012192724005613,
+          -0.011978722479596745,
+          0.014690559986694138,
+          0.0041346025614047875,
+          0.009881584250499358,
+          0.012176393203318002,
+          0.015764530703333238,
+          -0.004923548157246672,
+          -0.0015192775000207833,
+          0.012766482928380375,
+          0.0005745973040874939,
+          0.00997809510696241,
+          0.016495444272181515,
+          0.010467514633877887,
+          -0.021108281220544853,
+          0.0015078843789125156,
+          0.00522593579521476,
+          -0.017481430635971396,
+          0.0034362821987740755,
+          0.005397819166037461,
+          0.011626527721895694,
+          0.004048632382026344,
+          0.013207335578216437,
+          -0.015768003434275563,
+          -0.005916168437120099,
+          0.008153798523776654,
+          0.004319451296835019,
+          -0.02599474886669325,
+          -0.01485009085106089,
+          -0.0006781964116146299,
+          -0.004444144921332929,
+          0.013556265879295017,
+          0.01285768162554067,
+          0.02033626001285354,
+          0.024637349990579514,
+          -0.013833060730799217,
+          -0.00842819657792762,
+          0.009436072455719896,
+          0.015560357465711765,
+          0.005798181411384387,
+          0.007874051721372273,
+          0.009297440071118126,
+          0.0077320823375042,
+          -0.004073209576661068,
+          -0.002973928843567532,
+          0.020205003097225747,
+          -0.019725382393637726,
+          0.0006746659838642147,
+          -0.018417443693472958,
+          0.007530796896254817,
+          0.01140696723506382,
+          0.01129515042745398,
+          0.002816883643872417,
+          -0.005274757308627299,
+          0.005716917047812153,
+          -0.06254973656832136,
+          -0.009191453047502128,
+          0.003384751551032943,
+          0.027256466708746898,
+          -0.010885336140129142,
+          -0.003415707910562931,
+          0.007309577090125959,
+          0.005511970708769456,
+          0.0005572239548625122,
+          0.003186427278034852,
+          0.019698716902282903,
+          -0.02052581021967601,
+          -0.010793968505805883,
+          0.004219269922138548,
+          0.0033111837923746746,
+          -0.007269521534951044,
+          0.004805737092302134,
+          0.014143978208501908,
+          0.01986656063456352,
+          -0.0024048854610842748,
+          0.01629335842915328,
+          -0.0015498603704589015,
+          -0.014496994444493684,
+          -0.002671064325909516,
+          0.0037875833047555925,
+          -0.022466433279066646,
+          -0.005980897597195858,
+          0.0043262697442044435,
+          -0.011226121648007667,
+          0.006978495397818936,
+          0.008452177557316863,
+          -0.003018545465557218,
+          -0.0008303307644324117,
+          0.017796789945838845,
+          0.006591364879905198,
+          -0.013464486184025026,
+          0.0028668426952326957,
+          -0.023377605222213174,
+          0.004325185794796361,
+          0.001416245546006914,
+          0.007809449561523174,
+          -0.008431335139468173,
+          -0.014774499666285603,
+          -0.005933810712528226,
+          0.017746847020610673,
+          -0.01965188730040385,
+          -0.002406875192894708,
+          0.01683009662984443,
+          -0.0002518220110313279,
+          0.005456812494894182,
+          -0.02011241622618092,
+          -0.0009782974741561381,
+          -0.02933296756078644,
+          -0.01171823509606186,
+          0.001162345317872531,
+          0.00892221099991475,
+          -0.006551308022163148,
+          -0.03934356275100932,
+          0.00427623835776202,
+          -0.013111664387776261,
+          -0.008752857644809231,
+          -0.010214546496134218,
+          -0.00518087722852466,
+          0.015240297918667867,
+          -0.012498100033235827,
+          0.01086490299272505,
+          -0.00917777212602538,
+          0.005895055740265625,
+          0.0090554651297836,
+          0.014212841798768113,
+          -0.005544047915369086,
+          -0.0033589999440643385,
+          -0.007924830667307956,
+          0.0022291797354234005,
+          -0.0032976226056924502,
+          0.009615674261248298,
+          -0.0051932155872883,
+          -0.0065636389655770455,
+          0.02426106498596339,
+          0.011445591067574932,
+          0.013312596099532533,
+          0.0055118624272110965,
+          -0.007220620072349783,
+          -0.01916551187613823,
+          -0.003977616240384473,
+          -0.013204506708246246,
+          -0.006315966975649096,
+          0.01434099854765068,
+          0.012588249778961223,
+          0.006357972880218866,
+          0.0007747002406996095,
+          -0.004794439624135275,
+          0.006975593520375694,
+          0.0262335496080837,
+          -0.016481909971520166,
+          -0.0013084662210086592,
+          -0.01369313900978409,
+          0.01921748474212319,
+          0.03333761672336437,
+          -0.0008918419419301244,
+          -0.007699525186523558,
+          0.013757216065348164,
+          0.00819045303047094,
+          -0.007411380297164911,
+          0.02453611080891859,
+          -0.025045462880463647,
+          -0.015989319900023054,
+          -0.011650886878708265,
+          -0.011407253985176972,
+          -0.008386367959227559,
+          0.016331961716602287,
+          -0.00141225438918211,
+          -0.0018505754298268654,
+          0.009557291617446645,
+          0.03357924917013024,
+          0.006916860901290756,
+          0.007030308953738567,
+          -0.002739345982346728,
+          -0.009376522438071562,
+          0.011482637968411055,
+          -0.007259054391879614,
+          -0.04075661435928125,
+          0.004250195164583401,
+          -0.005160505473755424,
+          0.02405623169060588,
+          -0.02197586561375289,
+          -0.00619175761282745,
+          -0.006736129049831973,
+          -0.028200015908139763,
+          -0.005867722104196303,
+          0.02805071895018145,
+          -0.015402198567775478,
+          0.014773294766214671,
+          -0.009502185346558072,
+          -0.0013419858580438616,
+          0.004443749010136947,
+          0.013008783570355656,
+          -0.02289662952366337,
+          0.0013834888740540108,
+          -0.00048549197205067054,
+          0.027969842651517314,
+          0.010437729090975158,
+          -0.0047705656249623744,
+          0.014202218020412936,
+          0.008240861154198411,
+          -0.018637315941614406,
+          0.007314019317343154,
+          -0.005747966765101792,
+          0.014833896739762147,
+          -0.01314847940891154,
+          0.007762100494499689,
+          0.00705026393499352,
+          -0.01278023669088385,
+          0.03016342404000438,
+          -0.015204129985039077,
+          0.007776946006562982,
+          0.009278778571285689,
+          0.009656723862179986,
+          0.01257686993982689,
+          0.004362439504889069,
+          -0.02898418193036562,
+          0.0002144431781454467,
+          -0.03828100004144633,
+          -0.00833748467467171,
+          -0.002611937107022567,
+          -0.002145329666219461,
+          0.007050630951707937,
+          -0.009383645162261629,
+          -0.033976442057557676,
+          0.0006429978367640709,
+          0.01603027598131823,
+          0.017501067413108286,
+          0.0002196997337451293,
+          0.013075379412332973,
+          -0.006197537539554383,
+          -0.021148120787551232,
+          -0.016986448660412497,
+          -0.010207705210051413,
+          -0.02122194744719993,
+          -0.020550470201191853,
+          -0.00908564461857315,
+          0.012090079127547193,
+          -0.009927390005879807,
+          0.013888458967497627,
+          0.029513250508042935,
+          -0.024529175134769883,
+          0.0011144447737101862,
+          0.021977677894839922,
+          -0.00198308986575084,
+          0.014108189311291326,
+          0.001696775006906642,
+          0.014122064251288995,
+          -0.0004166140484018266,
+          -0.010556016685519523,
+          -0.009729362492661684,
+          0.017487929296614076,
+          -0.0026043257634797124,
+          0.016715798948897194,
+          0.012183487338360088,
+          0.003771721868999568,
+          -0.0030887189945795796,
+          0.010667776697461199,
+          -0.026471157050402344,
+          0.014249996582064706,
+          -0.006690961709923224,
+          -0.0181382691440242,
+          -0.008729104191096461,
+          0.003882731567105777,
+          -0.005609384377642956,
+          -0.007794812949125115,
+          0.012967089730551055,
+          -0.00944374111323783,
+          0.02088525981049842,
+          0.0023400121055514958,
+          -0.012196925240774426,
+          0.003648688547842322,
+          -0.03322041209304968,
+          0.0015191765655963997,
+          -0.018402382776013956,
+          -0.024659631472836927,
+          0.03840173429666435,
+          0.011991452465253198,
+          0.006639702644606365,
+          -0.01675838115087676,
+          0.007953102302963526,
+          0.0004257479565099406,
+          0.017484338819046517,
+          0.011068864013925452,
+          0.006466915839538296,
+          0.008364454017976605,
+          -0.023316137291380285,
+          -0.010898421318668592,
+          0.016204569093730564,
+          0.012099473359791148,
+          -0.01017229369980751,
+          0.019431669546223253,
+          0.017955119269367787,
+          0.007300372071724306,
+          -0.006898619924677706,
+          0.01647927363640854,
+          0.021012103522831126,
+          0.010291016618557653,
+          -0.00047955552103474145,
+          0.0008684297441778696,
+          -0.010534247928997246,
+          0.005037930235889528,
+          0.035417626740307205,
+          0.014712595692093886,
+          0.010650306363963653,
+          -0.03626584475133921,
+          0.021764456453493054,
+          -0.011961521673661574,
+          0.009092981192874177,
+          0.006089104887183293,
+          0.017982801101846038,
+          0.006916969397519662,
+          0.014196254958707752,
+          0.011745069030657836,
+          -0.006689305450954901,
+          0.025205280281804708,
+          -0.001601414806100461,
+          -0.011992525142592405,
+          0.003386930324498371,
+          -0.01859193251438437,
+          0.005303261319773131,
+          -0.0041710536965100615,
+          -0.004459393120257307,
+          0.0032461828144552565,
+          -0.0018650956576455287,
+          0.015678225251799887,
+          0.003894142628084471,
+          -0.02355212701026675,
+          -0.0014873755115206368,
+          -0.024284295126348117,
+          0.0027260592623277375,
+          -0.022185571998195344,
+          0.00022883605560765203,
+          -0.005557568353382136,
+          -0.009749382813201403,
+          -0.025385857431333838,
+          -0.009853073898348951,
+          -0.0035843255959309307,
+          0.007661137888845632,
+          -0.010521037433807653,
+          -0.004246698084422667,
+          -0.0019920499321964815,
+          -0.019465113363548715,
+          0.009014255846983305,
+          0.027122533756959888,
+          0.016848221859975445,
+          0.027170883580103264,
+          -0.00019554112591107156,
+          0.013614675849485926,
+          -0.002066822813103284,
+          -0.01603063005944364,
+          0.008824663429777616,
+          0.0075653522381218495,
+          -0.00531780740880296,
+          0.014643348459785112,
+          0.0072634484937344704,
+          -0.012800149015319017,
+          0.009859503660384916,
+          -0.007161712176469609,
+          -0.006437857349742842,
+          -0.009764856421740622,
+          -0.02058868349990357,
+          -0.014866137971603676,
+          0.0012450207572602557,
+          -0.002123265406747006,
+          0.006000473862248991,
+          -0.004054016790850226,
+          0.009584879012937315,
+          -0.00608816807208866,
+          -0.00925801057439376,
+          -0.013962683072792106,
+          -0.008924356118158748,
+          -0.004210064145949253,
+          -0.004381709838058668,
+          -0.011732699976353612,
+          -0.005479523097469261,
+          0.0010680258077037758,
+          -0.0019401640391035755,
+          -0.01397652201595301,
+          0.011650839243603377,
+          0.0058731838291845645,
+          -0.002462743134993977,
+          0.02219008688925881,
+          0.01097316136828334,
+          0.0018083001685932406,
+          0.01366095798812196,
+          0.0003942601242216818,
+          -0.0064063532958756175,
+          -0.03343312148366622,
+          -0.0005083355836126291,
+          -0.02815586367527057,
+          0.025770078162165123,
+          -0.0069114949102666145,
+          -0.010322087105114614,
+          -0.002519047562374146,
+          -0.006063669644021698,
+          0.005873609611264377,
+          -0.002722940652283699,
+          0.0021229525800901114,
+          -0.023673778230768945,
+          0.0028629715379660113,
+          -0.003953653437464966,
+          -0.0036366957271968823,
+          0.007469664864329332,
+          0.0346520690389124,
+          -0.004054282838167587,
+          -0.0034765253405897676,
+          0.005159466707114532,
+          0.011642913651743502,
+          0.027845571220425726,
+          -0.004445196354520585,
+          -0.0032038508043339413,
+          -0.02233838584152591,
+          0.0054112576531374175,
+          -0.00019544838850471866,
+          -0.004733192299951088,
+          0.016664217348411003,
+          -0.016719175418727594,
+          -0.01631658832047864,
+          -0.00545969677717507,
+          -0.014570848053102338,
+          -0.00022620258237138677,
+          0.014454101124666536,
+          0.011056312437637014,
+          -0.004035526363810078,
+          -0.012459378363360557,
+          -0.0022595493558020706,
+          -0.030370952502579162,
+          0.012766010284884204,
+          0.1141689619126009,
+          -0.00766197047210299,
+          -0.00896655658602649,
+          -0.009481112393590025,
+          0.001435269335646347,
+          0.001338755472877713,
+          -0.011739143028767118,
+          0.07166096084053437,
+          -0.0077855767070652395,
+          0.0002139580430342728,
+          -0.011312198831181402,
+          0.001844005266872301,
+          -0.001825412693163496,
+          0.01371932421024605,
+          -0.005025814503687022,
+          -0.018213360991353643,
+          -0.0033612481616769416,
+          -0.02941403830641238,
+          -0.0041137865872933515,
+          -0.005683630256743215,
+          -0.004288198570156578,
+          -0.014982999363445122,
+          0.0067149665791261335,
+          -0.019661973497372313,
+          -0.012688556129040908,
+          0.001124391650175233,
+          -0.0033623453063430534,
+          0.00996944383866679,
+          -0.020496336814603385,
+          -0.0044354321069609536,
+          0.0019875840782527733,
+          0.009706675240563883,
+          0.0011600364428219214,
+          -0.012755178230821602,
+          -0.0026514442584739412,
+          0.004056350166031111,
+          0.01909951327913368,
+          0.014360281326789544,
+          0.003732344102045803,
+          0.008623457695718206,
+          0.014461868085333023,
+          -0.001692149528228451,
+          0.01948035376572433,
+          0.01172550239477637,
+          0.0031966531144076173,
+          0.0004779413449255626,
+          0.0030912562273611517,
+          -0.004597633235071031,
+          0.01582547400214653,
+          -0.014811177637844808,
+          -0.02411202772171917,
+          -0.0071272238837422185,
+          0.018015541137524113,
+          -0.013081287852028074,
+          1.8837689271256874e-05,
+          0.004062803284646432,
+          0.027569726868200254,
+          0.0010564850664255057,
+          -0.01179755720247972,
+          0.0028876134800690334,
+          -0.01638215925131053,
+          0.022944183106912663,
+          0.0026043713064243836,
+          0.03279173899194346,
+          -0.005103903273968361,
+          0.00578213669463383,
+          -0.020744194169889544,
+          -0.007588793365216579,
+          -0.00819862611489168,
+          0.0017401442433921013,
+          0.010993500174794084,
+          0.005411238350013134,
+          0.01620925462604661,
+          -0.00641615069699833,
+          -0.020751730783437995,
+          0.014776878596707538,
+          -0.009552193304486048,
+          -0.03393319554458277,
+          0.008881987295131347,
+          -0.029102297123590998,
+          -0.0045267103602711,
+          0.00368971742585232,
+          -0.014269341759285646,
+          -0.016264809810970175,
+          0.012837302095591106,
+          -0.0011890334180728564,
+          -0.008937992768323412,
+          0.0014444613725248846,
+          -0.017110187133760626,
+          -0.02385668037818897,
+          0.006017405364919834,
+          0.00676670672597817,
+          -0.0077994500447595,
+          0.018312452858615755,
+          -0.027015660037024585,
+          0.006123912473734307,
+          -0.012973676426738582,
+          0.009041974762537394,
+          -0.01580183887345168,
+          0.01163516332546347,
+          0.010887662470499361,
+          -0.001503467664709027,
+          -0.0076234124970223326,
+          0.01231202541338883,
+          0.004852720134894811,
+          0.033760365210093034,
+          0.028658326381800395,
+          0.000995999783627898,
+          -0.0025605367058226737,
+          -0.0020356794893366384,
+          0.011667525096213252,
+          -0.014049700287539817,
+          0.011218246246280464,
+          0.02658113044439717,
+          0.017568631325838108,
+          -0.019133062654008885,
+          0.018050678772708717,
+          -0.010356118256625172,
+          0.012872052420999813,
+          -0.025620154415048252,
+          -0.005135768485044442,
+          0.004936490920616527,
+          0.005163008729256001,
+          0.007425278892536464,
+          0.001410212306439329,
+          0.021891387289347516,
+          0.014999616600766989,
+          0.00027613811156650413,
+          0.00558422822386443,
+          0.01477009526981092,
+          0.010497782599293994,
+          -0.0005567894749233204,
+          -0.00467776506604011,
+          -0.016998035358377343,
+          0.00957271797745408,
+          -0.006065116857898469,
+          0.005138070401846389,
+          0.01698914855850691,
+          -0.01230257809343985,
+          0.011109844802663414,
+          0.007518553660463051,
+          -0.015504773927595823,
+          -0.017169933945839305,
+          0.005552829003043926,
+          -0.02252633790984703,
+          -0.01306722099657118,
+          0.02715689072730398,
+          -0.010513941399990838,
+          -0.02790582732974509,
+          -0.0009847768069205477,
+          0.009797865241108237,
+          -0.002772952073769185,
+          -0.0027968741526000576,
+          0.0037541733651479613,
+          0.010801137722284016,
+          -0.014658156970099418,
+          0.002075415233345957,
+          0.004735264309137368,
+          0.01930804315461151,
+          -0.0014151522239496115,
+          -0.015046118414100617,
+          -0.002288495894227231,
+          0.021592829677937263,
+          0.017174609713578173,
+          0.00265560378681856,
+          -0.016499825201963983,
+          -3.965296928410772e-05,
+          -0.0074142747014567635,
+          0.00021360696629503726,
+          0.00027431174904249525,
+          -0.0056391089282546054,
+          -0.018931681700770225,
+          -0.003888248121485032,
+          0.011181675419027736,
+          -0.00247252920175839,
+          0.01977860312223846,
+          -0.0011527575128223392,
+          -0.015763587433200915,
+          0.001960734212846797,
+          -0.005280860341480037,
+          -0.010372829570478649,
+          0.01351900743232294,
+          -0.009359942374603997,
+          0.006788746813046239,
+          -0.003212015225284119,
+          0.004611143085183367,
+          -0.0003308099698088422,
+          -0.017017265064185255,
+          0.02924142709831441,
+          -0.0018238549856433466,
+          -0.004769274041546295,
+          -0.0014058363976586562,
+          -0.033393331166741945,
+          0.015017165991303838,
+          -0.014574579250749552,
+          0.006745699409061803,
+          0.002705853802154516,
+          0.01272741296341735,
+          0.003253095246601883,
+          -0.008898616461779179,
+          0.0027485606928249326,
+          0.017040360927723885,
+          0.013756402798365262,
+          -0.007292864293090161,
+          0.008694855886257706,
+          -0.0028934389922887676,
+          -0.0037483208383542015,
+          0.01363963339139356,
+          0.004170611039133969,
+          -0.006282657671540625,
+          -0.0009784372731140945
+        ],
+        "keyphrases": [
+          "DeepSeekMoE architecture",
+          "expert specialization",
+          "Mixture-of-Experts (MoE)",
+          "fine-grained expert segmentation",
+          "shared expert isolation"
+        ],
+        "title": "Deepseekmoe: Towards Ultimate Expert Specialization In Mixture-Of-Experts Language Models"
+      },
+      "type": "document"
+    },
+    {
+      "id": "65742957-6bda-446d-a61a-0fe20313c760",
+      "properties": {
+        "page_content": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity\n\nWilliam Fedus∗ liamfedus@google.com Barret Zoph∗ barretzoph@google.com Noam Shazeer noam@google.com Google, Mountain View, CA 94043, USA Editor: Alexander Clark\n\nAbstract\n\nIn deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) models defy this and instead select different parameters for each incoming example. The result is a sparsely-activated model—with an outrageous number of parameters—but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs, and training instability. We address these with the introduction of the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques mitigate the instabilities, and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large (Raffel et al., 2019) to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the \"Colossal Clean Crawled Corpus\", and achieve a 4x speedup over the T5-XXL model.12 Keywords: mixture-of-experts, natural language processing, sparsity, large-scale machine learning, distributed computing arXiv:2101.03961v3 [cs.LG] 16 Jun 2022 Contents\n\n1 Introduction 3 2 Switch Transformer 4 2.1 Simplifying Sparse Routing 5 2.2 Efficient Sparse Routing 6 2.3 Putting It All Together: The Switch Transformer 8 2.4 Improved Training and Fine-Tuning Techniques 8 3 Scaling Properties 11 3.1 Scaling Results on a Step-Basis 12 3.2 Scaling Results on a Time-Basis 13 3.3 Scaling Versus a Larger Dense Model 13 4 Downstream Results 14 4.1 Fine-Tuning 14 4.2 Distillation 16 4.3 Multilingual Learning 17 5 Designing Models with Data, Model, and Expert-Parallelism 18 5.1 Data Parallelism 20 5.2 Model Parallelism 20 5.3 Model and Data Parallelism 21 5.4 Expert and Data Parallelism 22 5.5 Expert, Model and Data Parallelism 22 5.6 Towards Trillion Parameter Models 22 6 Related Work 24 7 Discussion 25 8 Future Work 26 9 Conclusion 27 A Switch for Attention 27 B Preventing Token Dropping with No-Token-Left-Behind 29\n\nC Encouraging Exploration Across Experts 29 D Switch Transformers in Lower Compute Regimes 29 E Relation of Upstream to Downstream Model Performance 32 F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+        "document_metadata": {
+          "source": "../../datasets/Sample_Docs_Markdown/switch_transformers.md"
+        },
+        "summary": "The Switch Transformer is a sparsely-activated model that simplifies the Mixture of Experts (MoE) approach, allowing for efficient scaling to trillion-parameter models with reduced communication and computational costs. It achieves significant speedups in pre-training and fine-tuning by activating only a subset of neural network weights for each input, maintaining a constant computational cost. The model demonstrates superior scaling properties and improved performance across various natural language processing tasks, including multilingual settings. It also allows for effective distillation into smaller dense models, preserving a portion of the quality gains. The architecture is designed to maximize parameter count while keeping floating point operations per example constant, making it suitable for both large-scale and smaller-scale applications.",
+        "headlines": {
+          "1 Introduction": [],
+          "2 Switch Transformer": [
+            "2.1 Simplifying Sparse Routing",
+            "2.2 Efficient Sparse Routing",
+            "2.3 Putting It All Together: The Switch Transformer",
+            "2.4 Improved Training and Fine-Tuning Techniques"
+          ],
+          "3 Scaling Properties": [
+            "3.1 Scaling Results on a Step-Basis",
+            "3.2 Scaling Results on a Time-Basis",
+            "3.3 Scaling Versus a Larger Dense Model"
+          ],
+          "4 Downstream Results": [
+            "4.1 Fine-Tuning",
+            "4.2 Distillation",
+            "4.3 Multilingual Learning"
+          ],
+          "5 Designing Models with Data, Model, and Expert-Parallelism": [
+            "5.1 Data Parallelism",
+            "5.2 Model Parallelism",
+            "5.3 Model and Data Parallelism",
+            "5.4 Expert and Data Parallelism",
+            "5.5 Expert, Model and Data Parallelism",
+            "5.6 Towards Trillion Parameter Models"
+          ],
+          "6 Related Work": [],
+          "7 Discussion": [],
+          "8 Future Work": [],
+          "9 Conclusion": [],
+          "A Switch for Attention": [],
+          "B Preventing Token Dropping with No-Token-Left-Behind": [],
+          "C Encouraging Exploration Across Experts": [],
+          "D Switch Transformers in Lower Compute Regimes": [],
+          "E Relation of Upstream to Downstream Model Performance": [],
+          "F Pseudo Code for Switch Transformers": []
+        },
+        "summary_embedding": [
+          -0.0060447100549936295,
+          -0.00390331051312387,
+          -0.01784370467066765,
+          -0.0021820589900016785,
+          0.013227885589003563,
+          0.011338187381625175,
+          0.0165580902248621,
+          0.024240797385573387,
+          -0.005986624862998724,
+          0.03596621751785278,
+          -0.004852031823247671,
+          -0.0068811336532235146,
+          -0.003574162255972624,
+          0.014242436736822128,
+          0.011578272096812725,
+          -0.033302053809165955,
+          -0.024519605562090874,
+          0.01050176378339529,
+          0.016093410551548004,
+          -0.006795942317694426,
+          0.007473600562661886,
+          -0.030513975769281387,
+          -0.06124480068683624,
+          -0.006838538218289614,
+          -0.0026099516544491053,
+          0.01029265858232975,
+          0.0010329445358365774,
+          -0.010873507708311081,
+          -0.053469158709049225,
+          0.023311438038945198,
+          0.025294071063399315,
+          0.005092116538435221,
+          -0.029584616422653198,
+          0.0009361362317577004,
+          -0.05161044001579285,
+          -0.04448534920811653,
+          0.0379488542675972,
+          0.012561844661831856,
+          -0.04364892467856407,
+          0.039249956607818604,
+          -0.004875265993177891,
+          -0.0022440163884311914,
+          -0.021499186754226685,
+          -0.0034909071400761604,
+          -0.007039899472147226,
+          -0.009851212613284588,
+          -0.02874819189310074,
+          0.0014037203509360552,
+          0.021886419504880905,
+          -0.0056962003000080585,
+          0.023264970630407333,
+          -0.024271776899695396,
+          0.01936165988445282,
+          0.04240977764129639,
+          -0.06957805901765823,
+          0.024659009650349617,
+          0.017549408599734306,
+          0.05120771750807762,
+          0.06511713564395905,
+          0.018339363858103752,
+          0.021669568493962288,
+          -0.014141756109893322,
+          -0.017766257748007774,
+          -0.011872569099068642,
+          0.004673904739320278,
+          -0.014970434829592705,
+          -0.01891246810555458,
+          0.008449427783489227,
+          -0.020461400970816612,
+          0.008209343999624252,
+          -0.05576157942414284,
+          -0.014730350114405155,
+          -0.034200433641672134,
+          -0.01140014547854662,
+          0.017642343416810036,
+          -0.018215449526906013,
+          0.030126741155982018,
+          0.02997184917330742,
+          0.002638994250446558,
+          0.0007822110201232135,
+          0.021049996837973595,
+          0.006509389728307724,
+          -0.0053438181057572365,
+          0.012321759946644306,
+          0.028159597888588905,
+          -0.01450575515627861,
+          -0.010106786154210567,
+          -0.03454120084643364,
+          0.01861817203462124,
+          0.01589204929769039,
+          -0.04194509983062744,
+          -0.012569589540362358,
+          -0.013824224472045898,
+          0.032341714948415756,
+          -0.009549170732498169,
+          0.04098476096987724,
+          -0.010083552449941635,
+          0.001383390510454774,
+          -0.031149037182331085,
+          -0.009463978931307793,
+          0.017069239169359207,
+          0.006172496825456619,
+          -0.016589069738984108,
+          -0.0012846461031585932,
+          0.016170857474207878,
+          0.006617815233767033,
+          -0.030142230913043022,
+          0.04488807171583176,
+          0.023497309535741806,
+          0.055296897888183594,
+          0.05579255893826485,
+          -0.037391237914562225,
+          -0.002445377642288804,
+          0.0344172865152359,
+          -0.0228312686085701,
+          -0.015040136873722076,
+          -0.034014564007520676,
+          -0.01768881268799305,
+          0.035935238003730774,
+          -0.0318460576236248,
+          0.01556677371263504,
+          0.028066661208868027,
+          -0.025712283328175545,
+          0.003488970920443535,
+          -0.03943582624197006,
+          0.003622566582635045,
+          0.01220559049397707,
+          0.006095050368458033,
+          -0.0026293133851140738,
+          -0.013615119270980358,
+          0.008906363509595394,
+          0.003055269829928875,
+          0.00800798274576664,
+          -0.04615819454193115,
+          0.018478767946362495,
+          -0.00454224506393075,
+          0.04033420979976654,
+          0.042502716183662415,
+          -0.02730768360197544,
+          -0.016743963584303856,
+          0.008557853288948536,
+          0.012104909867048264,
+          0.04188314080238342,
+          -0.007287728600203991,
+          -0.00121784838847816,
+          -0.0007836631848476827,
+          0.007322579622268677,
+          0.047552235424518585,
+          0.004646798130124807,
+          0.0020697612781077623,
+          -0.027230238541960716,
+          0.014978179708123207,
+          -0.019733402878046036,
+          0.027648448944091797,
+          0.023373395204544067,
+          0.02942972257733345,
+          -0.04048910364508629,
+          0.017518429085612297,
+          -0.011818356812000275,
+          -0.005568413063883781,
+          -0.002726121572777629,
+          -0.02856232039630413,
+          -0.0074426219798624516,
+          0.033983584493398666,
+          0.015365412458777428,
+          0.012956822291016579,
+          0.03271345794200897,
+          0.0012052633101120591,
+          -0.042502716183662415,
+          0.03804178908467293,
+          0.024550583213567734,
+          0.04640602692961693,
+          0.023466331884264946,
+          0.010796061716973782,
+          0.006571347359567881,
+          0.013375034555792809,
+          -0.019377149641513824,
+          0.015907539054751396,
+          0.02560385875403881,
+          0.009750531986355782,
+          -0.03167567402124405,
+          -0.0009501734166406095,
+          0.01178737822920084,
+          -0.004546117503196001,
+          0.03701949119567871,
+          0.04104671627283096,
+          0.020569827407598495,
+          0.05030933395028114,
+          0.0020155487582087517,
+          0.017936641350388527,
+          0.005936284549534321,
+          0.02820606529712677,
+          0.030467506498098373,
+          -0.005285732913762331,
+          0.005076627247035503,
+          -0.01136916596442461,
+          -0.03385967016220093,
+          0.014110777527093887,
+          0.01561324205249548,
+          -0.008077683858573437,
+          -0.01697630248963833,
+          0.006033093202859163,
+          0.009169681929051876,
+          -0.01795213110744953,
+          -0.017456471920013428,
+          0.005971135571599007,
+          0.018866000697016716,
+          -0.013576395809650421,
+          0.024689987301826477,
+          -0.04442339017987251,
+          0.029119934886693954,
+          -0.033209118992090225,
+          -0.003384418087080121,
+          -0.010641167871654034,
+          -0.0030843124259263277,
+          0.0041666291654109955,
+          0.011314953677356243,
+          0.013406013138592243,
+          0.002042655134573579,
+          -0.013909416273236275,
+          0.024519605562090874,
+          0.018447788432240486,
+          0.012073930352926254,
+          -0.03148980438709259,
+          -0.014397329650819302,
+          -0.0015673263696953654,
+          -0.008743725717067719,
+          -0.034138478338718414,
+          0.026966920122504234,
+          0.014048820361495018,
+          -0.03844451159238815,
+          0.017797237262129784,
+          0.029538147151470184,
+          0.014420564286410809,
+          -0.004790074657648802,
+          -0.0463130883872509,
+          0.04045812413096428,
+          -0.030436528846621513,
+          -0.033333033323287964,
+          -0.010478530079126358,
+          -0.0003146269591525197,
+          0.02894955314695835,
+          -0.0064358157105743885,
+          -0.050835974514484406,
+          0.01184933539479971,
+          0.020631784573197365,
+          0.00010763872705865651,
+          0.005731051322072744,
+          -0.019593998789787292,
+          0.00028631053282879293,
+          -0.03395260497927666,
+          0.0090844901278615,
+          0.026750069111585617,
+          0.008867640048265457,
+          -0.016418687999248505,
+          0.03351890295743942,
+          -0.0004121613164898008,
+          -0.018695618957281113,
+          -0.003889757441356778,
+          0.006885006092488766,
+          0.034912943840026855,
+          0.004836542531847954,
+          -0.02946070022881031,
+          0.05665995925664902,
+          -0.02100352756679058,
+          0.021344292908906937,
+          0.05631919577717781,
+          -0.019439106807112694,
+          -0.008472662419080734,
+          -0.002065889071673155,
+          0.0026196325197815895,
+          -0.0260530486702919,
+          -0.008937342092394829,
+          0.032341714948415756,
+          0.041728246957063675,
+          -0.018742086365818977,
+          -0.001398879918269813,
+          0.006780453026294708,
+          0.009425255469977856,
+          0.004948840010911226,
+          0.02692045085132122,
+          -0.012128143571317196,
+          -0.014559967443346977,
+          0.05198218300938606,
+          0.028314489871263504,
+          -0.012081675231456757,
+          -0.0164031982421875,
+          0.019888296723365784,
+          -0.024844881147146225,
+          -0.020244551822543144,
+          -0.02431824430823326,
+          -0.020414933562278748,
+          -0.0042518205009400845,
+          0.009525936096906662,
+          0.037484172731637955,
+          -0.03968365862965584,
+          0.012747716158628464,
+          0.021700548008084297,
+          0.02981695532798767,
+          -0.019532041624188423,
+          -0.05910727381706238,
+          -0.0016128262504935265,
+          0.03296129032969475,
+          -0.015458349138498306,
+          -0.049379974603652954,
+          0.015164051204919815,
+          0.0260530486702919,
+          0.05957195535302162,
+          -0.001461805310100317,
+          -0.014954946003854275,
+          0.061926331371068954,
+          0.03026614524424076,
+          -0.004383479710668325,
+          -0.009494957514107227,
+          -0.0127864396199584,
+          0.003396035172045231,
+          0.00465454300865531,
+          -0.03364282101392746,
+          -0.041170634329319,
+          -0.011841590516269207,
+          -0.010284913703799248,
+          0.008108663372695446,
+          -0.040922801941633224,
+          0.03531566634774208,
+          -0.021917399019002914,
+          0.010950954630970955,
+          0.015473837964236736,
+          0.037391237914562225,
+          -0.021344292908906937,
+          -0.01284839678555727,
+          -0.01572941243648529,
+          -0.024968795478343964,
+          -0.014010096900165081,
+          0.021917399019002914,
+          0.007682706229388714,
+          3.4699722164077684e-05,
+          0.010075807571411133,
+          -0.024659009650349617,
+          -0.051052823662757874,
+          0.0291044469922781,
+          0.0015605497173964977,
+          -0.015938518568873405,
+          -0.004623564425855875,
+          -0.01977987214922905,
+          -0.01122201792895794,
+          -0.012608313001692295,
+          -0.039249956607818604,
+          0.021111954003572464,
+          -0.01135367713868618,
+          0.015876561403274536,
+          0.0009250032599084079,
+          0.003659353591501713,
+          0.028624277561903,
+          0.01116006076335907,
+          0.0051811798475682735,
+          -0.03255856782197952,
+          -0.018277406692504883,
+          0.010122275911271572,
+          -0.00561100896447897,
+          -0.018339363858103752,
+          -0.013351799920201302,
+          -0.05037129297852516,
+          -0.03305422514677048,
+          -0.008054450154304504,
+          -0.05415068939328194,
+          0.0020697612781077623,
+          -0.014296649023890495,
+          -0.02232012152671814,
+          -0.005990497302263975,
+          0.03107159025967121,
+          -0.003235333366319537,
+          0.025294071063399315,
+          0.0076594725251197815,
+          0.037360258400440216,
+          0.040117356926202774,
+          0.008426194079220295,
+          0.03065337985754013,
+          -0.0029991210903972387,
+          -0.016867877915501595,
+          0.05486319959163666,
+          -0.01881953328847885,
+          0.013529927469789982,
+          -0.008604321628808975,
+          0.008604321628808975,
+          0.03299226611852646,
+          -0.003794885240495205,
+          0.02007416822016239,
+          -0.0311955064535141,
+          -0.07998688519001007,
+          0.01637221872806549,
+          0.028159597888588905,
+          -0.06418777257204056,
+          0.014691627584397793,
+          -0.020275529474020004,
+          0.02052335813641548,
+          0.02411688305437565,
+          -0.0009114501299336553,
+          -0.014064309187233448,
+          0.005595519673079252,
+          0.0020833145827054977,
+          -0.01456771232187748,
+          -0.002691270550712943,
+          -0.009417511522769928,
+          0.008774704299867153,
+          -0.017007282003760338,
+          -0.013406013138592243,
+          -0.034231413155794144,
+          0.010873507708311081,
+          0.01233724970370531,
+          -0.003969140350818634,
+          0.0011500825639814138,
+          -0.03277541697025299,
+          -0.007876322604715824,
+          -0.02145271934568882,
+          0.03382869064807892,
+          0.012058441527187824,
+          -0.030699847266077995,
+          -0.007814365439116955,
+          0.015264731831848621,
+          -0.007876322604715824,
+          0.0008906363509595394,
+          0.024132372811436653,
+          0.02103450708091259,
+          0.0028364830650389194,
+          -0.0009700191440060735,
+          0.02438020147383213,
+          0.007427132688462734,
+          -0.01720864325761795,
+          0.028794659301638603,
+          0.014288905076682568,
+          0.060160547494888306,
+          -0.03271345794200897,
+          0.001297231181524694,
+          -0.04135650396347046,
+          -0.031118059530854225,
+          0.006408709101378918,
+          0.014350862242281437,
+          0.007597515359520912,
+          -0.028252532705664635,
+          0.005978880450129509,
+          0.02004319056868553,
+          0.01469937153160572,
+          0.009704063646495342,
+          0.0037580979987978935,
+          -0.036740683019161224,
+          -0.000691695255227387,
+          -0.011214273050427437,
+          -0.0038781403563916683,
+          0.035656433552503586,
+          -0.008186109364032745,
+          -0.05151750147342682,
+          -0.0006418389966711402,
+          -0.03255856782197952,
+          -0.0353776253759861,
+          0.021313315257430077,
+          -0.00839521549642086,
+          -0.0039207362569868565,
+          -0.013986863195896149,
+          -0.000537770101800561,
+          -0.010524998418986797,
+          0.027509044855833054,
+          0.028345469385385513,
+          0.022490503266453743,
+          -0.05384090170264244,
+          0.009479468688368797,
+          0.00023766436788719147,
+          -0.05817791447043419,
+          0.0009709871956147254,
+          -0.007117345929145813,
+          0.011191039346158504,
+          -0.020213572308421135,
+          -0.02414786070585251,
+          0.011500826105475426,
+          0.03392162546515465,
+          0.014304393902420998,
+          0.0321558453142643,
+          -0.013367289677262306,
+          -0.09250226616859436,
+          0.018292896449565887,
+          -0.008689512498676777,
+          0.04126356914639473,
+          0.00865078903734684,
+          -0.004278926644474268,
+          -0.015845581889152527,
+          -0.00865078903734684,
+          -0.006943091284483671,
+          0.01283290795981884,
+          0.004530628211796284,
+          -0.005626498255878687,
+          0.0016631665639579296,
+          0.013708055019378662,
+          -0.005967263597995043,
+          0.007481344975531101,
+          0.003651608945801854,
+          -0.013189162127673626,
+          -0.06059424951672554,
+          -0.04055105894804001,
+          -0.018540725111961365,
+          0.050835974514484406,
+          -0.0069159846752882,
+          -0.046684831380844116,
+          -0.010850274004042149,
+          0.05058814212679863,
+          0.012259802781045437,
+          -0.0334569476544857,
+          -0.0007841471815481782,
+          0.0050882440991699696,
+          0.015458349138498306,
+          0.004546117503196001,
+          -0.027292195707559586,
+          0.02820606529712677,
+          0.0008412640891037881,
+          0.02300165221095085,
+          -0.018339363858103752,
+          -0.005312839522957802,
+          0.035780347883701324,
+          -0.024844881147146225,
+          0.023791607469320297,
+          0.0055219451896846294,
+          0.031629208475351334,
+          0.002952652983367443,
+          0.010060317814350128,
+          0.004704883322119713,
+          0.0186491496860981,
+          0.004275054205209017,
+          0.0276639387011528,
+          0.011462102644145489,
+          0.006428070832043886,
+          0.021623101085424423,
+          -0.009378788061439991,
+          -0.014513500034809113,
+          0.01885051093995571,
+          -0.0014695499558001757,
+          0.025232113897800446,
+          -0.020538847893476486,
+          0.002726121572777629,
+          -0.0016883367206901312,
+          -0.017967619001865387,
+          -0.040922801941633224,
+          -0.0007405834621749818,
+          0.021747015416622162,
+          -0.011671207845211029,
+          -0.0228312686085701,
+          -0.013715799897909164,
+          -0.01585332676768303,
+          0.0038490979932248592,
+          0.034169457852840424,
+          -0.0398695282638073,
+          0.00963436160236597,
+          -0.000830615172162652,
+          -0.03125746175646782,
+          -0.0239619892090559,
+          -0.01093546487390995,
+          0.014614180661737919,
+          -0.01652711257338524,
+          -0.021173911169171333,
+          -0.028469383716583252,
+          -0.008410705253481865,
+          0.006048582494258881,
+          -0.05065010115504265,
+          -0.004054331686347723,
+          0.019965743646025658,
+          0.0276639387011528,
+          -0.003099801717326045,
+          -0.02177799493074417,
+          0.0012468908680602908,
+          0.022351099178195,
+          0.02936776541173458,
+          0.014226946979761124,
+          0.027586491778492928,
+          -0.002178186783567071,
+          -0.0011675080750137568,
+          -0.0073148347437381744,
+          0.01307299267500639,
+          0.0011287847300991416,
+          0.036616768687963486,
+          0.0122210793197155,
+          -0.020306508988142014,
+          0.0013959755888208747,
+          -0.006431943271309137,
+          0.029181893914937973,
+          -0.02270735427737236,
+          -9.832092473516241e-05,
+          0.03531566634774208,
+          0.027741385623812675,
+          0.004933350719511509,
+          -0.0143121387809515,
+          -0.05142456665635109,
+          0.021592123433947563,
+          -0.01765783317387104,
+          0.031025122851133347,
+          0.016666516661643982,
+          0.003397971158847213,
+          -0.0034212053287774324,
+          0.012368228286504745,
+          -0.022087780758738518,
+          -0.004406713880598545,
+          -0.021142931655049324,
+          0.04389675334095955,
+          -0.043184246867895126,
+          0.008472662419080734,
+          0.029445212334394455,
+          0.011926782317459583,
+          -0.019082851707935333,
+          -0.005425137002021074,
+          0.029212871566414833,
+          -0.00162637943867594,
+          0.01308848150074482,
+          0.0058665829710662365,
+          -0.026935940608382225,
+          0.01202746294438839,
+          -0.031660184264183044,
+          0.02177799493074417,
+          -0.013436991721391678,
+          0.018447788432240486,
+          0.03342596814036369,
+          -0.0030397805385291576,
+          -0.0038239278364926577,
+          -0.016263794153928757,
+          -0.00969631876796484,
+          0.0662323608994484,
+          0.01054823212325573,
+          0.03726732358336449,
+          0.0018974426202476025,
+          -0.009773765690624714,
+          0.007969259284436703,
+          0.0063041565008461475,
+          -0.012329504825174809,
+          0.010494019836187363,
+          0.024178840219974518,
+          0.0012730291346088052,
+          -0.002412462839856744,
+          -0.019408127292990685,
+          -0.03438630700111389,
+          0.003727119415998459,
+          -0.030932186171412468,
+          0.022691864520311356,
+          0.009208405390381813,
+          0.017456471920013428,
+          0.008240322582423687,
+          -0.015907539054751396,
+          0.03200095146894455,
+          0.05461536720395088,
+          0.0019768255297094584,
+          0.013375034555792809,
+          0.007938280701637268,
+          0.007036027032881975,
+          -0.011338187381625175,
+          0.04702559858560562,
+          0.000662652775645256,
+          0.005169562995433807,
+          0.022536970674991608,
+          0.02145271934568882,
+          0.022939693182706833,
+          0.002304037567228079,
+          -0.011694442480802536,
+          0.010478530079126358,
+          -0.0006302220281213522,
+          0.004445436876267195,
+          -0.010695381090044975,
+          -0.027865299955010414,
+          0.007492962293326855,
+          0.01302652433514595,
+          0.022598929703235626,
+          0.021592123433947563,
+          -0.03178409859538078,
+          -0.01855621486902237,
+          0.03364282101392746,
+          -0.02997184917330742,
+          0.03612111136317253,
+          -0.0014424435794353485,
+          -0.023807097226381302,
+          -0.0058356039226055145,
+          0.016929835081100464,
+          0.02656419761478901,
+          -0.029181893914937973,
+          0.01804506592452526,
+          0.029693040996789932,
+          -0.0074426219798624516,
+          0.0005939188995398581,
+          0.012112653814256191,
+          0.02447313815355301,
+          -0.009293596260249615,
+          -0.0024027819745242596,
+          -0.0013320822035893798,
+          -0.011926782317459583,
+          -0.018370341509580612,
+          -0.0044725434854626656,
+          -0.022629907354712486,
+          0.005564541090279818,
+          -0.007365175057202578,
+          -0.003481226274743676,
+          -0.008581086993217468,
+          0.01154729351401329,
+          0.001979729626327753,
+          -0.019857317209243774,
+          -0.022335609421133995,
+          0.01652711257338524,
+          0.03887821361422539,
+          -0.018215449526906013,
+          -0.02106548473238945,
+          0.024442158639431,
+          -0.01759587600827217,
+          -0.01601596549153328,
+          0.036678727716207504,
+          0.013739033602178097,
+          0.007736918982118368,
+          -0.03358086198568344,
+          -0.030049296095967293,
+          -0.03274443745613098,
+          0.006296411622315645,
+          0.03785591572523117,
+          -0.0028752065263688564,
+          0.02569679357111454,
+          0.03701949119567871,
+          0.0057891360484063625,
+          -0.028934063389897346,
+          -0.014823286794126034,
+          -0.045104920864105225,
+          0.060160547494888306,
+          0.0011481463443487883,
+          0.05161044001579285,
+          0.016294771805405617,
+          -0.01456771232187748,
+          0.00419760774821043,
+          -0.013251119293272495,
+          0.026192452758550644,
+          0.02280029095709324,
+          0.025929134339094162,
+          0.006582964211702347,
+          0.022505993023514748,
+          -0.0037522895727306604,
+          -0.012197845615446568,
+          0.007307090330868959,
+          0.014536733739078045,
+          0.0002497654058970511,
+          -0.011392400600016117,
+          0.015783624723553658,
+          -0.01307299267500639,
+          -0.011361422017216682,
+          0.023295948281884193,
+          0.01368482131510973,
+          -0.012344993650913239,
+          0.014064309187233448,
+          -0.008565598167479038,
+          0.01239920686930418,
+          -0.023915521800518036,
+          0.022691864520311356,
+          -0.03262052312493324,
+          0.009587894193828106,
+          0.0015053689712658525,
+          -0.0073148347437381744,
+          0.024550583213567734,
+          -0.0027454833034425974,
+          -0.05071205645799637,
+          0.03550153970718384,
+          -0.008015726692974567,
+          0.0037213109899312258,
+          0.006381602957844734,
+          0.011392400600016117,
+          -0.0012623801594600081,
+          0.03807276859879494,
+          0.01239920686930418,
+          0.015969496220350266,
+          0.02199484594166279,
+          0.011911292560398579,
+          -0.008433938957750797,
+          -0.004096927121281624,
+          -0.0016137943603098392,
+          -0.01784370467066765,
+          0.025944624096155167,
+          0.0018674320308491588,
+          0.03962169960141182,
+          -0.018168980255723,
+          -0.02300165221095085,
+          0.028066661208868027,
+          0.007597515359520912,
+          -0.02351279929280281,
+          -0.0017377089243382215,
+          0.01202746294438839,
+          -0.009185170754790306,
+          0.027385130524635315,
+          0.01196550577878952,
+          0.019113829359412193,
+          -0.018060555681586266,
+          -0.004340883810073137,
+          -0.005072754807770252,
+          -0.015938518568873405,
+          0.02238207869231701,
+          0.020771188661456108,
+          0.011152315884828568,
+          -0.011880313977599144,
+          -0.01307299267500639,
+          0.020600805059075356,
+          0.035036858171224594,
+          -0.0002744515077210963,
+          0.005672966130077839,
+          0.017487451434135437,
+          0.002373739378526807,
+          -0.04470219835639,
+          0.016232814639806747,
+          -0.014025585725903511,
+          0.011214273050427437,
+          0.019593998789787292,
+          -0.021917399019002914,
+          -0.03196997195482254,
+          -0.011539548635482788,
+          -0.0414804182946682,
+          0.00493722315877676,
+          -0.0069159846752882,
+          0.013576395809650421,
+          -0.010176488198339939,
+          -0.0014831031439825892,
+          0.016666516661643982,
+          0.0038103745318949223,
+          0.02177799493074417,
+          -0.011678952723741531,
+          -0.012073930352926254,
+          -0.020213572308421135,
+          -0.020786678418517113,
+          0.004278926644474268,
+          -0.030405549332499504,
+          -0.004488032776862383,
+          0.021375272423028946,
+          -0.04783104360103607,
+          -0.03689557686448097,
+          0.02898053266108036,
+          -0.008503641001880169,
+          0.020755698904395103,
+          0.022955182939767838,
+          -0.015094350092113018,
+          -0.0008717587334103882,
+          -0.018695618957281113,
+          0.02258343994617462,
+          -0.007837600074708462,
+          0.026300877332687378,
+          -0.0027551641687750816,
+          0.0006292539183050394,
+          -0.02659517526626587,
+          0.020306508988142014,
+          0.005227648187428713,
+          0.012275291606783867,
+          -0.02390003204345703,
+          0.006815304048359394,
+          -0.03133491054177284,
+          -0.01784370467066765,
+          0.0180140882730484,
+          -0.011152315884828568,
+          0.027989214286208153,
+          -0.006060199346393347,
+          0.028438406065106392,
+          0.022722844034433365,
+          0.024751944467425346,
+          0.02676555886864662,
+          -0.02363671362400055,
+          0.008495896123349667,
+          0.03726732358336449,
+          0.00865078903734684,
+          -0.015876561403274536,
+          0.008170620538294315,
+          -0.04863648861646652,
+          -0.008062195032835007,
+          0.05030933395028114,
+          0.002127846470102668,
+          0.023141054436564445,
+          0.016294771805405617,
+          -0.002873270306736231,
+          -0.005688455421477556,
+          0.010269423946738243,
+          -0.04296739399433136,
+          -0.023172033950686455,
+          0.022629907354712486,
+          -0.021561143919825554,
+          6.54212328754511e-07,
+          0.03122648410499096,
+          0.012120398692786694,
+          -0.0009109660750254989,
+          -0.05108380317687988,
+          0.02100352756679058,
+          0.01178737822920084,
+          0.014443797990679741,
+          -0.010300402529537678,
+          0.026068538427352905,
+          0.01733255758881569,
+          0.038413532078266144,
+          0.012755461037158966,
+          -0.002679653698578477,
+          -0.03782493621110916,
+          0.009177426807582378,
+          -0.021685058251023293,
+          0.0011471783509477973,
+          0.0074503663927316666,
+          -0.02273833192884922,
+          -0.024782923981547356,
+          0.0043912241235375404,
+          0.004588713403791189,
+          0.011880313977599144,
+          0.022087780758738518,
+          0.009417511522769928,
+          -0.021204888820648193,
+          0.015404135920107365,
+          -0.05446047708392143,
+          -0.015450604259967804,
+          -0.023884542286396027,
+          0.02566581591963768,
+          0.020012211054563522,
+          0.010184233076870441,
+          0.005545179359614849,
+          0.010904486291110516,
+          -0.020988039672374725,
+          -0.009239383973181248,
+          0.020631784573197365,
+          0.011074868962168694,
+          -0.012840652838349342,
+          0.04113965481519699,
+          -0.00011126903700642288,
+          0.031210994347929955,
+          -0.02280029095709324,
+          0.002941036131232977,
+          -0.001241082325577736,
+          -0.03649285435676575,
+          0.009130958467721939,
+          0.01981084980070591,
+          0.02151467651128769,
+          -0.01861817203462124,
+          0.0024143988266587257,
+          0.01048627495765686,
+          0.004766840487718582,
+          0.038351576775312424,
+          0.027121812105178833,
+          -0.044206541031599045,
+          0.011663463898003101,
+          0.012995545752346516,
+          -0.009363298304378986,
+          0.048419639468193054,
+          -0.017518429085612297,
+          0.017673322930932045,
+          0.008240322582423687,
+          -0.01798310875892639,
+          0.02318752370774746,
+          -0.01469937153160572,
+          -0.013282098807394505,
+          0.00650551775470376,
+          0.009192915633320808,
+          -0.0001770381786627695,
+          -0.037577107548713684,
+          -0.017859194427728653,
+          -0.005719434004276991,
+          0.0038297362625598907,
+          0.01562873087823391,
+          0.013498948886990547,
+          0.009967382065951824,
+          0.005161818582564592,
+          -0.00945623405277729,
+          -0.0016089539276435971,
+          0.008124152198433876,
+          -0.0005532593932002783,
+          0.0019216446671634912,
+          -0.01926872320473194,
+          0.02698240801692009,
+          0.02572777308523655,
+          -0.0018354853382334113,
+          -0.04652994126081467,
+          -0.0036709706764668226,
+          0.0037077576853334904,
+          -0.017735280096530914,
+          -0.0035315665882080793,
+          -0.04519785940647125,
+          -0.01810702309012413,
+          0.0143121387809515,
+          -0.014614180661737919,
+          -0.0460033044219017,
+          0.009076746180653572,
+          -0.013886182568967342,
+          0.0063970922492444515,
+          -0.00932457484304905,
+          0.005762029904872179,
+          0.003651608945801854,
+          0.005680711008608341,
+          -0.005909178406000137,
+          0.05108380317687988,
+          -0.03466511517763138,
+          0.028051171451807022,
+          0.008565598167479038,
+          -0.012437930330634117,
+          -0.02261441759765148,
+          0.00887538492679596,
+          -0.044981006532907486,
+          -0.033209118992090225,
+          0.016743963584303856,
+          -0.01781272701919079,
+          0.011074868962168694,
+          0.06161654368042946,
+          0.007330324035137892,
+          -0.023342415690422058,
+          -0.01233724970370531,
+          -0.021049996837973595,
+          -0.04191412031650543,
+          0.003442503046244383,
+          0.005463860463351011,
+          -0.004948840010911226,
+          -0.03807276859879494,
+          -0.021716037765145302,
+          0.04615819454193115,
+          -0.012004229240119457,
+          0.012422440573573112,
+          0.01080380566418171,
+          -0.0019410063978284597,
+          -0.04860550910234451,
+          -0.021019017323851585,
+          0.0395597442984581,
+          0.021886419504880905,
+          -0.008426194079220295,
+          0.0028364830650389194,
+          -0.018277406692504883,
+          -0.003289545886218548,
+          0.006296411622315645,
+          -0.009107724763453007,
+          -0.03562545403838158,
+          0.02067825198173523,
+          0.0054832217283546925,
+          -0.023373395204544067,
+          -0.025805220007896423,
+          -0.03296129032969475,
+          0.009944148361682892,
+          0.02453509531915188,
+          -0.01968693546950817,
+          0.005823987070471048,
+          0.00027203132049180567,
+          -0.000881923595443368,
+          0.00042740863864310086,
+          -0.009959637187421322,
+          -0.019764382392168045,
+          -0.020151615142822266,
+          0.03268248215317726,
+          -0.0060059865936636925,
+          0.0074387495405972,
+          -0.03630698472261429,
+          -0.010974188335239887,
+          0.011322698555886745,
+          -0.002395037328824401,
+          -0.013452480547130108,
+          -0.009092235006392002,
+          -0.01739451475441456,
+          -0.009882191196084023,
+          -0.007876322604715824,
+          0.022660886868834496,
+          0.006168624851852655,
+          0.005409647710621357,
+          0.0023795480374246836,
+          -0.008581086993217468,
+          0.02898053266108036,
+          0.017874684184789658,
+          -0.014188224449753761,
+          -0.00920066051185131,
+          0.03451022133231163,
+          0.02665713243186474,
+          -0.009115469641983509,
+          0.01798310875892639,
+          0.023264970630407333,
+          -0.0010484338272362947,
+          -0.01196550577878952,
+          0.001922612776979804,
+          -0.0009550138493068516,
+          -0.0032972905319184065,
+          0.00845717266201973,
+          -0.015365412458777428,
+          0.0315207801759243,
+          0.04287445917725563,
+          -0.002145271748304367,
+          0.009587894193828106,
+          -0.003727119415998459,
+          -0.008472662419080734,
+          0.017069239169359207,
+          0.009316830895841122,
+          0.04625113308429718,
+          0.01098193321377039,
+          0.005680711008608341,
+          -0.033983584493398666,
+          -0.001229465357027948,
+          -0.01184933539479971,
+          0.03308520466089249,
+          0.02061629481613636,
+          0.00889087375253439,
+          -0.008379725739359856,
+          0.01344473659992218,
+          0.008341003209352493,
+          0.00661007035523653,
+          0.020569827407598495,
+          -0.016542602330446243,
+          -0.017286090180277824,
+          0.030467506498098373,
+          -0.00672236829996109,
+          0.01289486512541771,
+          0.02029101923108101,
+          -0.002089123008772731,
+          0.010842529125511646,
+          0.07100307941436768,
+          -0.006184114143252373,
+          0.015713922679424286,
+          -0.019082851707935333,
+          0.0015789433382451534,
+          -0.007640110794454813,
+          -0.03370477631688118,
+          0.018246427178382874,
+          0.0039962464943528175,
+          0.00865078903734684,
+          0.01198099460452795,
+          0.009053511545062065,
+          -0.007946024648845196,
+          -0.00963436160236597,
+          0.015195030719041824,
+          0.0019080914789810777,
+          0.00033568276558071375,
+          0.001336922636255622,
+          -0.0065210070461034775,
+          -0.004011735785752535,
+          -0.021948376670479774,
+          0.02763296104967594,
+          0.007140580099076033,
+          0.030606910586357117,
+          -0.055296897888183594,
+          0.030095763504505157,
+          -0.00789568480104208,
+          0.02554190158843994,
+          -0.016201836988329887,
+          -0.011090358719229698,
+          0.001609921921044588,
+          0.018153492361307144,
+          0.004491904750466347,
+          0.006462921854108572,
+          0.011888058856129646,
+          0.023233991116285324,
+          -0.03444826602935791,
+          0.017053749412298203,
+          0.023110076785087585,
+          0.0048675211146473885,
+          0.0003535923024173826,
+          0.005165690556168556,
+          0.015458349138498306,
+          0.007198664825409651,
+          -0.023048119619488716,
+          -0.0025692922063171864,
+          0.017611365765333176,
+          -0.02306360937654972,
+          0.0036980770528316498,
+          -0.006923729553818703,
+          0.004623564425855875,
+          -0.0014705180656164885,
+          -0.03509881719946861,
+          -0.00100099784322083,
+          -0.025743262842297554,
+          0.01923774555325508,
+          0.009556914679706097,
+          -0.025867177173495293,
+          0.003932353109121323,
+          0.025650326162576675,
+          -0.0003695656778290868,
+          0.00821708794683218,
+          0.0004683101433329284,
+          0.00869725737720728,
+          -0.00974278710782528,
+          -0.01493945624679327,
+          0.004801691509783268,
+          0.01531120017170906,
+          -0.01493945624679327,
+          -0.007431004662066698,
+          -0.009510447271168232,
+          0.030560443177819252,
+          0.0006674932083114982,
+          0.004189862869679928,
+          -0.005045648664236069,
+          -0.01858719252049923,
+          -0.0032237162813544273,
+          -0.021142931655049324,
+          0.004367990419268608,
+          -0.0009046735358424485,
+          0.04197607934474945,
+          0.0038161829579621553,
+          -0.02029101923108101,
+          0.0032972905319184065,
+          -0.03136589005589485,
+          0.0005000148667022586,
+          -0.007276111748069525,
+          -0.02004319056868553,
+          0.0015005286550149322,
+          0.0016525175888091326,
+          0.012631546705961227,
+          -0.03255856782197952,
+          -0.009301341138780117,
+          0.018416810780763626,
+          -0.00020680672605521977,
+          -0.010687636211514473,
+          -0.017828216776251793,
+          0.00035189816844649613,
+          -0.011338187381625175,
+          -0.027973726391792297,
+          -0.02850036323070526,
+          -0.0026719090528786182,
+          0.013491204008460045,
+          -0.018277406692504883,
+          -0.014993669465184212,
+          0.0031249718740582466,
+          0.021592123433947563,
+          -0.009781510569155216,
+          0.008302279748022556,
+          0.02241305634379387,
+          -0.008255811408162117,
+          -0.005514200776815414,
+          -0.000941460661124438,
+          -0.009130958467721939,
+          0.004267309792339802,
+          0.006300284061580896,
+          0.043339136987924576,
+          -0.032403673976659775,
+          -0.0018238683696836233,
+          0.014815541915595531,
+          0.0034153966698795557,
+          0.0025905899237841368,
+          0.002503462601453066,
+          -0.0007841471815481782,
+          0.01344473659992218,
+          0.01178737822920084,
+          0.015466093085706234,
+          0.00100099784322083,
+          -0.0033011629711836576,
+          0.006408709101378918,
+          -0.021173911169171333,
+          0.015721667557954788,
+          0.027958236634731293,
+          -0.0033999073784798384,
+          0.005525817628949881,
+          -0.013731288723647594,
+          -0.008759214542806149,
+          0.013134949840605259,
+          0.0026641644071787596,
+          -0.005068882368505001,
+          -0.009394276887178421,
+          0.017967619001865387,
+          0.019082851707935333,
+          0.006668155547231436,
+          -0.037453193217515945,
+          -0.02898053266108036,
+          -0.009959637187421322,
+          0.020569827407598495,
+          0.014916222542524338,
+          0.0010339126456528902,
+          -0.010540487244725227,
+          -0.009595638141036034,
+          -0.011903548613190651,
+          0.010989678092300892,
+          0.02721474878489971,
+          -0.032372694462537766,
+          0.0008214184199459851,
+          -0.029383255168795586,
+          0.03068435750901699,
+          0.0020000594668090343,
+          0.011446612887084484,
+          -0.0008185141487047076,
+          -0.015140817500650883,
+          -0.029181893914937973,
+          0.013824224472045898,
+          -0.004395096562802792,
+          0.001024231780320406,
+          -0.0031753121875226498,
+          0.0201051477342844,
+          0.012887120246887207,
+          -0.0014472840121015906,
+          0.01968693546950817,
+          0.008178364485502243,
+          0.016480645164847374,
+          -0.010153254494071007,
+          -0.04135650396347046,
+          0.019965743646025658,
+          0.007562664337456226,
+          0.014219203032553196,
+          0.004739734344184399,
+          0.004588713403791189,
+          -0.020771188661456108,
+          0.002724185585975647,
+          0.004333139397203922,
+          0.014405074529349804,
+          0.011028401553630829,
+          0.0025615475606173277,
+          0.011678952723741531,
+          -0.02228914201259613,
+          0.01929970271885395,
+          -0.01631026156246662,
+          0.01006806269288063,
+          0.004445436876267195,
+          -0.026873983442783356,
+          0.007547175046056509,
+          0.007822110317647457,
+          -0.0019187404541298747,
+          -0.017549408599734306,
+          0.02160761132836342,
+          -0.0034831624943763018,
+          -0.03509881719946861,
+          0.0360281765460968,
+          -0.0043447562493383884,
+          0.019346170127391815,
+          0.03209388628602028,
+          -0.01771979033946991,
+          -0.01993476413190365,
+          0.00906900130212307,
+          0.010408828034996986,
+          -0.002873270306736231,
+          0.017859194427728653,
+          -0.002919738180935383,
+          -0.02232012152671814,
+          0.017766257748007774,
+          -0.028918573632836342,
+          -0.014234691858291626,
+          0.020337486639618874,
+          -0.03624502569437027,
+          -0.0021026763133704662,
+          0.03175312280654907,
+          -0.024070415645837784,
+          0.005734923295676708,
+          0.005196669604629278,
+          -0.0019351978553459048,
+          0.0012236569309607148,
+          -0.006451305001974106,
+          0.03801080957055092,
+          -0.010865762829780579,
+          0.0201051477342844,
+          0.01929970271885395,
+          -0.023388884961605072,
+          0.01977987214922905,
+          -0.001109423115849495,
+          0.01675945147871971,
+          -0.003940097521990538,
+          0.03466511517763138,
+          -0.006017603911459446,
+          -0.011330443434417248,
+          0.006431943271309137,
+          0.008488151244819164,
+          -0.009959637187421322,
+          0.008681767620146275,
+          -0.018122512847185135,
+          0.027152791619300842,
+          0.02091059274971485,
+          -0.0002691270783543587,
+          0.013800990767776966,
+          0.0004421719058882445,
+          -0.020833145827054977,
+          0.00789568480104208,
+          -0.027121812105178833,
+          -0.009967382065951824,
+          0.01646515540778637,
+          -0.005762029904872179,
+          0.007380664348602295,
+          -0.02064727433025837,
+          0.028004704043269157,
+          0.0077524082735180855,
+          -0.013591884635388851,
+          -0.0008824076503515244,
+          0.023760627955198288,
+          -0.005432881414890289,
+          0.009355553425848484,
+          0.02070923149585724,
+          0.015713922679424286,
+          0.013615119270980358,
+          0.016263794153928757,
+          0.03351890295743942,
+          0.005328328814357519,
+          0.005568413063883781,
+          0.003519949736073613,
+          0.016991792246699333,
+          -0.008557853288948536,
+          0.018029578030109406,
+          0.0008093173382803798,
+          -0.013576395809650421,
+          0.021499186754226685,
+          -0.0018684001406654716,
+          -0.013622863218188286,
+          -0.0040891822427511215,
+          -0.004337011836469173,
+          0.03271345794200897,
+          0.022041313350200653,
+          -0.004468671046197414,
+          -0.024442158639431,
+          -0.00394784240052104,
+          0.016682006418704987,
+          -0.003167567541822791,
+          0.011136826127767563,
+          0.0007952801533974707,
+          -0.01586107164621353,
+          -0.009595638141036034,
+          0.0063583687879145145,
+          0.010586955584585667,
+          -0.008921852335333824,
+          0.010346870869398117,
+          -0.0228312686085701,
+          -0.004011735785752535,
+          0.00046613195445388556,
+          -0.004844287410378456,
+          0.0053438181057572365,
+          -0.011314953677356243,
+          0.004948840010911226,
+          0.016062432900071144,
+          0.01026167906820774,
+          0.004832670092582703,
+          0.006989559158682823,
+          0.012066186405718327,
+          0.009161937050521374,
+          -0.01627928391098976,
+          0.00428667152300477,
+          -0.005072754807770252,
+          -0.02345084212720394,
+          0.005831731949001551,
+          0.00902253296226263,
+          -0.01122201792895794,
+          0.009386532939970493,
+          -0.010958699509501457,
+          0.03296129032969475,
+          0.01030814740806818,
+          -0.037391237914562225,
+          -0.0033379499800503254,
+          -0.0016244432190433145,
+          0.014885243959724903,
+          0.012298526242375374,
+          0.008550108410418034,
+          -0.017518429085612297,
+          -0.0047823297791182995,
+          -0.009835722856223583,
+          0.0012827098835259676,
+          -0.012306271120905876,
+          0.017456471920013428,
+          0.022567950189113617,
+          -0.036337960511446,
+          0.01968693546950817,
+          -0.010432061739265919,
+          -0.023652203381061554,
+          0.020476890727877617,
+          0.011500826105475426,
+          0.01942361705005169,
+          -0.009402021765708923,
+          0.018896980211138725,
+          -0.0001331114035565406,
+          0.025510922074317932,
+          -0.004228586331009865,
+          -0.022087780758738518,
+          -0.012499887496232986,
+          -0.03209388628602028,
+          -0.0051811798475682735,
+          0.006819176487624645,
+          -0.00045524101005867124,
+          0.013111716136336327,
+          0.014226946979761124,
+          -0.025712283328175545,
+          0.007295473478734493,
+          0.032589543610811234,
+          0.014637414366006851,
+          0.023311438038945198,
+          -0.011276230216026306,
+          -0.0010794125264510512,
+          -0.0026060794480144978,
+          -0.013770012184977531,
+          -0.03212486580014229,
+          -0.0017493258928880095,
+          -0.036585792899131775,
+          0.005680711008608341,
+          -0.016387708485126495,
+          -0.013096226379275322,
+          0.023264970630407333,
+          -0.034138478338718414,
+          0.0014608372002840042,
+          0.013708055019378662,
+          0.010540487244725227,
+          0.012391461990773678,
+          -0.00162637943867594,
+          0.015032391995191574,
+          -0.029243851080536842,
+          -0.010284913703799248,
+          -0.004491904750466347,
+          -0.01981084980070591,
+          -0.007338068913668394,
+          -0.018060555681586266,
+          -0.013127204962074757,
+          -0.003566417610272765,
+          -0.010060317814350128,
+          0.002247888594865799,
+          0.0032062907703220844,
+          -0.014714861288666725,
+          0.03206290677189827,
+          0.003157886676490307,
+          0.018958937376737595,
+          0.0033418224193155766,
+          0.004449309315532446,
+          0.018788553774356842,
+          -0.0026331855915486813,
+          -0.0058278595097362995,
+          0.030963165685534477,
+          -0.00032672801171429455,
+          0.008759214542806149,
+          0.019082851707935333,
+          0.004724245052784681,
+          0.0038529701996594667,
+          -0.0260530486702919,
+          0.012887120246887207,
+          -0.027447087690234184,
+          0.005080499220639467,
+          -0.0077175572514534,
+          -0.024302754551172256,
+          -0.028128618374466896,
+          -0.006571347359567881,
+          0.012058441527187824,
+          -0.02898053266108036,
+          -0.00613377382978797,
+          0.002315654419362545,
+          -0.00400399137288332,
+          -0.002673845272511244,
+          0.007241260726004839,
+          0.029150914400815964,
+          -0.033209118992090225,
+          0.008751469664275646,
+          0.005595519673079252,
+          -0.0005053392960689962,
+          0.013297587633132935,
+          0.008038961328566074,
+          0.003343758638948202,
+          0.020848635584115982,
+          0.006637176964432001,
+          0.002458930714055896,
+          -0.0025576751213520765,
+          -0.014242436736822128,
+          0.004000118933618069,
+          0.02154565416276455,
+          0.007562664337456226,
+          0.018385831266641617,
+          -0.008914108388125896,
+          0.002621568739414215,
+          -0.01455222349613905,
+          0.005552923772484064,
+          -0.002867461647838354,
+          0.004309905227273703,
+          -0.012236569076776505,
+          0.008093173615634441,
+          0.0028771425131708384,
+          -0.0043447562493383884,
+          0.018060555681586266,
+          -0.002319526858627796,
+          0.016480645164847374,
+          0.004015608225017786,
+          0.015620986931025982,
+          -0.0035412474535405636,
+          0.01178737822920084,
+          0.0005731051205657423,
+          -0.017905661836266518,
+          -0.017022771760821342,
+          -0.039249956607818604,
+          0.007702067960053682,
+          0.03531566634774208,
+          -0.0009564659558236599,
+          -0.03444826602935791,
+          0.010145509615540504,
+          -0.011942271143198013,
+          -0.009975126944482327,
+          -0.022862248122692108,
+          0.013862947933375835,
+          0.012430185452103615,
+          0.007415515370666981,
+          0.012252057902514935,
+          -0.006997303571552038,
+          0.01026167906820774,
+          -0.005216030869632959,
+          0.008356492035090923,
+          -0.0010823167394846678,
+          -0.037422213703393936,
+          0.012344993650913239,
+          0.02216522768139839,
+          -0.002635121811181307,
+          0.0016089539276435971,
+          -0.006145390681922436,
+          0.022103270515799522,
+          -0.013971373438835144,
+          -0.00650551775470376,
+          -0.00046443779137916863,
+          0.00797700323164463,
+          -0.01795213110744953,
+          0.004778457339853048,
+          0.0755879208445549,
+          -0.007775642443448305,
+          0.006122156511992216,
+          0.009270362555980682,
+          -0.015148562379181385,
+          -0.01637221872806549,
+          -0.004561606794595718,
+          -0.020260039716959,
+          0.008844406343996525,
+          -0.0018819533288478851,
+          -0.0110981035977602,
+          0.012468908913433552,
+          0.0053050946444272995,
+          0.002031038049608469,
+          -0.008379725739359856,
+          0.010811550542712212,
+          0.009525936096906662,
+          0.010432061739265919,
+          0.02360573597252369,
+          0.011477591469883919,
+          -0.00858883187174797,
+          -0.011795123107731342,
+          0.01129171997308731,
+          0.0028577810153365135,
+          0.016697494313120842,
+          0.007907302118837833,
+          0.0010648912284523249,
+          0.019501063972711563,
+          -0.002309845993295312,
+          0.02180897258222103,
+          -2.2689444449497387e-05,
+          -0.009301341138780117,
+          -0.004236331209540367,
+          -0.015349923633038998,
+          0.024271776899695396,
+          -0.0020155487582087517,
+          0.0015450604259967804,
+          -0.020213572308421135,
+          0.0013979118084535003,
+          0.013243375346064568,
+          0.011957760900259018,
+          -0.0020407189149409533,
+          -0.0051347119733691216,
+          0.0007154133054427803,
+          -0.016356728971004486,
+          0.032806396484375,
+          0.007578153628855944,
+          0.0007681738352403045,
+          0.0008838597568683326,
+          0.0009288755827583373,
+          -0.0032972905319184065,
+          0.0024608669336885214,
+          -0.004766840487718582,
+          0.009549170732498169,
+          -0.010943209752440453,
+          -0.010796061716973782,
+          0.010075807571411133,
+          -0.02402394637465477,
+          0.003566417610272765,
+          -0.00558777479454875,
+          -0.008062195032835007,
+          0.0076207490637898445,
+          0.00015537731815129519,
+          -0.007392281666398048,
+          -0.0030939930584281683,
+          0.0023408245760947466,
+          0.013429246842861176,
+          0.004406713880598545,
+          0.010850274004042149,
+          0.012956822291016579,
+          0.005359307397156954,
+          -0.006629432085901499,
+          0.015512561425566673,
+          -0.006946963258087635,
+          -0.004228586331009865,
+          -0.0012468908680602908,
+          -0.019996721297502518,
+          -0.0009419447160325944,
+          -0.004708755761384964,
+          0.0015092412941157818,
+          0.0001705036120256409,
+          -8.749050175538287e-05,
+          -0.0018490385264158249,
+          -0.011988739483058453,
+          -0.001831613015383482,
+          -0.011663463898003101,
+          -0.014498010277748108,
+          0.010253935120999813,
+          -0.0041201612912118435,
+          -0.022939693182706833,
+          -0.015349923633038998,
+          -0.020662762224674225,
+          0.02145271934568882,
+          -0.004387352149933577,
+          -0.005800753366202116,
+          0.006733985152095556,
+          -0.010215211659669876,
+          -0.015706177800893784,
+          -0.008069939911365509,
+          -0.0205853171646595,
+          0.010703125037252903,
+          0.003214035416021943,
+          0.0006791101768612862,
+          0.00882891658693552,
+          0.014397329650819302,
+          -0.01589204929769039,
+          0.0111755495890975,
+          0.012182355858385563,
+          0.011361422017216682,
+          -0.006854027509689331,
+          0.0008911203476600349,
+          -0.014103032648563385,
+          0.02596011385321617,
+          0.021623101085424423,
+          0.005169562995433807,
+          -0.04132552444934845,
+          -0.005862710531800985,
+          -0.013344055972993374,
+          -0.010524998418986797,
+          -0.012120398692786694,
+          0.011771888472139835,
+          -0.0038820127956569195,
+          0.0013117524795234203,
+          0.006288666743785143,
+          0.0015063370810821652,
+          0.004367990419268608,
+          -0.0022382077295333147,
+          -0.007709812838584185,
+          -0.03212486580014229,
+          -0.02817508764564991,
+          -0.00863530021160841,
+          0.020461400970816612,
+          0.002838419284671545,
+          -0.0009453330421820283,
+          -0.0036380558740347624,
+          0.0027048238553106785,
+          -0.005882072262465954,
+          -0.005018542055040598,
+          -0.014753584749996662,
+          -0.012739972211420536,
+          0.0101300198584795,
+          0.00802347157150507,
+          0.014195968396961689,
+          -0.011950016021728516,
+          0.0028190575540065765,
+          0.018788553774356842,
+          0.007690451107919216,
+          -0.023357905447483063,
+          0.01135367713868618,
+          -0.010912231169641018,
+          -0.009270362555980682,
+          -0.009433000348508358,
+          -0.009471723809838295,
+          0.0016980175860226154,
+          -0.008317768573760986,
+          0.003945906180888414,
+          -0.03615209087729454,
+          -0.011330443434417248,
+          -0.0013901671627536416,
+          -0.010346870869398117,
+          0.0009864765452221036,
+          0.02524760365486145,
+          0.006815304048359394,
+          0.01432762760668993,
+          -0.0006733016925863922,
+          0.017502939328551292,
+          -0.0018984107300639153,
+          0.018726596608757973,
+          0.012763205915689468,
+          -0.00797700323164463,
+          0.011717676185071468,
+          0.014738094992935658,
+          -0.0033863543067127466,
+          0.015620986931025982,
+          -0.010168743319809437,
+          0.005285732913762331,
+          -0.009556914679706097,
+          -0.003214035416021943,
+          0.002495717955753207,
+          0.030901208519935608,
+          -0.003618694143369794,
+          0.00858883187174797,
+          0.01098193321377039,
+          -0.015241498127579689,
+          0.0026951429899781942,
+          0.014962689951062202,
+          0.01141563430428505,
+          0.007760153152048588,
+          0.0024434414226561785,
+          -0.024225307628512383,
+          0.004193735308945179,
+          0.005037903785705566,
+          0.0031520782504230738,
+          -0.014366351068019867,
+          0.0021859314292669296,
+          0.015675200149416924,
+          0.008550108410418034,
+          -0.025325050577521324,
+          0.016418687999248505,
+          0.0009138703462667763,
+          -0.003909118939191103,
+          0.029119934886693954,
+          -0.011268485337495804,
+          -0.005723306443542242,
+          0.009463978931307793,
+          0.0009980935137718916,
+          0.0030029932968318462,
+          0.008937342092394829,
+          0.0228312686085701,
+          -0.003988501615822315,
+          -0.032341714948415756,
+          -0.022459525614976883,
+          0.015706177800893784,
+          -0.0047823297791182995,
+          -0.022939693182706833,
+          -0.026703599840402603,
+          -0.003113354789093137,
+          -0.02228914201259613,
+          -0.0022323993034660816,
+          -0.020058678463101387,
+          -0.014056564308702946,
+          0.001268188701942563,
+          0.029987338930368423,
+          0.016201836988329887,
+          0.008224832825362682,
+          -0.011051635257899761,
+          -0.029956359416246414,
+          -0.024271776899695396,
+          0.01810702309012413,
+          -0.007055388763546944,
+          -0.005541306920349598,
+          0.00591692328453064,
+          -0.009951893240213394,
+          -0.008480406366288662,
+          0.011802867986261845,
+          0.027137301862239838,
+          -0.002747419523075223,
+          -0.029987338930368423,
+          -0.02688947319984436,
+          -0.014335372485220432,
+          -0.01840132102370262,
+          -0.017704300582408905,
+          -0.008627555333077908,
+          0.0045035220682621,
+          0.00030300996149890125,
+          0.016186347231268883,
+          0.014962689951062202,
+          -0.000480653194244951,
+          0.0028597170021384954,
+          -2.7711375878425315e-05,
+          -0.001763847190886736,
+          0.006888878531754017,
+          -0.004933350719511509,
+          0.026997897773981094,
+          0.014614180661737919,
+          -0.006594581063836813,
+          0.018711106851696968,
+          0.028314489871263504,
+          0.020430423319339752,
+          0.003157886676490307,
+          -0.010695381090044975,
+          -0.014606435783207417,
+          0.009727297350764275,
+          -0.013421501964330673,
+          0.023264970630407333,
+          0.022552460432052612,
+          -0.01480779703706503,
+          -0.014885243959724903,
+          -0.017905661836266518,
+          0.015210519544780254,
+          -0.007562664337456226,
+          -0.008224832825362682,
+          0.012368228286504745,
+          0.022335609421133995,
+          -0.007024410180747509,
+          0.002371803391724825,
+          -0.0010232636705040932,
+          -0.021731525659561157,
+          -0.006122156511992216,
+          -0.004596457816660404,
+          -0.008666278794407845,
+          0.008681767620146275,
+          0.008131897076964378,
+          -0.001928421319462359,
+          -0.006319645792245865,
+          0.005471604876220226,
+          -0.013367289677262306,
+          0.01301103550940752,
+          -0.00789568480104208,
+          0.03379771113395691,
+          -0.009138703346252441,
+          -0.0006733016925863922,
+          -0.022598929703235626,
+          -0.011446612887084484,
+          0.00845717266201973,
+          0.0017967619933187962,
+          0.01682141050696373,
+          -0.0011626676423475146,
+          -0.005688455421477556,
+          0.009913169778883457,
+          -0.0010687635513022542,
+          -0.001450188341550529,
+          -0.006892750971019268,
+          -0.010401083156466484,
+          0.0002884887217078358,
+          0.013258864171802998,
+          -0.006025348324328661,
+          0.0018054747488349676,
+          0.006095050368458033,
+          -0.0175339188426733,
+          0.029027000069618225,
+          0.023884542286396027,
+          0.017255110666155815,
+          0.014660648070275784,
+          0.0006394187803380191,
+          0.011647974140942097,
+          -0.003572226269170642,
+          0.0038820127956569195,
+          -0.020337486639618874,
+          0.021080974489450455,
+          -0.010811550542712212,
+          0.01846327818930149,
+          0.011655719019472599,
+          -0.015233753249049187,
+          -0.005022414494305849,
+          0.021406250074505806,
+          0.009967382065951824,
+          0.00889087375253439,
+          0.0017541663255542517,
+          -0.02470547705888748,
+          0.017766257748007774,
+          0.010099041275680065,
+          -0.020399443805217743,
+          0.019547531381249428,
+          -0.009471723809838295,
+          0.01233724970370531,
+          -0.0005300253978930414,
+          0.004418330732733011,
+          0.0058743273839354515,
+          0.021499186754226685,
+          0.011570527218282223,
+          -0.03884723410010338,
+          -0.013615119270980358,
+          -0.0017357728211209178,
+          0.006857899948954582,
+          -0.0036303112283349037,
+          -0.015233753249049187,
+          -0.041635312139987946,
+          0.01604694314301014,
+          0.009998360648751259,
+          -0.01643417589366436,
+          0.004987563472241163,
+          -0.002427952131256461,
+          -0.0042053526267409325,
+          -0.008170620538294315,
+          0.003186929039657116,
+          -0.004170501604676247,
+          0.02324948087334633,
+          0.0018984107300639153,
+          -0.006946963258087635,
+          0.013258864171802998,
+          0.01344473659992218,
+          -0.021948376670479774,
+          0.0009874446550384164,
+          0.0021375271026045084,
+          0.000636998622212559,
+          0.023683181032538414,
+          -0.014707116410136223,
+          -0.0017657832941040397,
+          -0.01685238815844059,
+          -0.0013069120468571782,
+          -0.021592123433947563,
+          -0.001188805908896029,
+          -0.015187285840511322,
+          -0.008511385880410671,
+          0.025030752643942833,
+          -0.011005166918039322,
+          0.010997422970831394,
+          -0.006687517277896404,
+          0.006943091284483671,
+          -0.006904367823153734,
+          -0.01936165988445282,
+          -0.0008267428493127227,
+          0.009975126944482327,
+          -0.000512115890160203,
+          0.002941036131232977,
+          8.482827252009884e-05,
+          0.002464739140123129,
+          0.008867640048265457,
+          0.006234454456716776,
+          -0.0075278133153915405,
+          0.02489134855568409,
+          -0.012073930352926254,
+          0.0016399325104430318,
+          0.011911292560398579,
+          0.0004743606550619006,
+          -0.010424317792057991,
+          0.025588368996977806,
+          -0.02139076218008995,
+          -0.00800798274576664,
+          -0.00014799568452872336,
+          -0.0016544538084417582,
+          -0.006513262167572975,
+          0.006718495860695839,
+          0.02946070022881031,
+          0.0062654330395162106,
+          0.010447551496326923,
+          -0.0281441081315279,
+          -0.011493081226944923,
+          0.012538610957562923,
+          -0.022273652255535126,
+          -0.0007739823195151985,
+          -0.0004322490422055125,
+          -0.00619960343465209,
+          0.026378324255347252,
+          -0.02022906206548214,
+          -0.01093546487390995,
+          -0.018664639443159103,
+          0.005405775271356106,
+          -0.011121337302029133,
+          0.008294534869492054,
+          -0.0009777637897059321,
+          0.006540368776768446,
+          0.016805920749902725,
+          -0.019563021138310432,
+          0.016031453385949135,
+          0.005119222681969404,
+          -0.016217326745390892,
+          0.010222956538200378,
+          -0.017580386251211166,
+          -0.0008199662552215159,
+          0.020275529474020004,
+          -0.00400399137288332,
+          0.008751469664275646,
+          -0.003363120136782527,
+          0.005208286456763744,
+          -0.006513262167572975,
+          0.0001492057926952839,
+          -0.007249005138874054,
+          0.014428308233618736,
+          -0.016805920749902725,
+          -0.017828216776251793,
+          -0.008821171708405018,
+          0.0016709112096577883,
+          0.021297825500369072,
+          0.011136826127767563,
+          -0.010044828988611698,
+          -0.011059380136430264,
+          -0.011694442480802536,
+          0.041604332625865936,
+          0.005762029904872179,
+          0.027230238541960716,
+          -0.02261441759765148,
+          0.00545998802408576,
+          0.025170156732201576,
+          -0.014242436736822128,
+          0.0012788375606760383,
+          -0.0022788674104958773,
+          0.0053050946444272995,
+          0.011562783271074295,
+          -0.01819995976984501,
+          0.0049139889888465405,
+          -0.00019216447253711522,
+          -0.00782985519617796,
+          -0.000656844291370362,
+          0.003442503046244383,
+          0.00020196630794089288,
+          0.016325751319527626,
+          -0.004027225077152252,
+          -0.016511622816324234,
+          -0.004429947584867477,
+          -0.0003216455806978047,
+          -0.027896279469132423,
+          0.013514438644051552,
+          0.016589069738984108,
+          0.01566745527088642,
+          0.027989214286208153,
+          -0.014513500034809113,
+          0.016031453385949135,
+          0.009533680975437164,
+          0.013901671394705772,
+          -0.010610189288854599,
+          0.008952830918133259,
+          -0.01595400646328926,
+          0.007446494419127703,
+          -0.03454120084643364,
+          -0.007887939922511578,
+          -0.00802347157150507,
+          0.01984182931482792,
+          0.028763681650161743,
+          0.002923610620200634,
+          0.006493900436908007,
+          0.000196883876924403,
+          0.0024202074855566025,
+          -0.01714668609201908,
+          -0.020182594656944275,
+          -0.0003337466041557491,
+          0.0029332914855331182,
+          0.0006950835813768208,
+          -0.004495777189731598,
+          0.00472037261351943,
+          -0.012391461990773678,
+          0.0026835259050130844,
+          -0.013134949840605259,
+          0.005971135571599007,
+          -0.0059982421807944775,
+          0.009603383019566536,
+          0.0019429425010457635,
+          0.020864123478531837,
+          0.02300165221095085,
+          0.004100799560546875,
+          0.0046971384435892105,
+          -0.024333734065294266,
+          0.00920066051185131,
+          -0.013847459107637405,
+          0.012329504825174809,
+          0.019408127292990685,
+          0.008248066529631615,
+          0.006571347359567881,
+          0.01233724970370531,
+          0.012918098829686642,
+          -0.0031772484071552753,
+          0.024287264794111252,
+          0.005014669615775347,
+          -0.0009070937521755695,
+          0.002052335999906063,
+          -0.014954946003854275,
+          -0.0070979841984808445,
+          -0.0015286030247807503,
+          0.0018848575418815017,
+          -0.004731989465653896,
+          0.01906736195087433,
+          0.015256987884640694,
+          -0.01050176378339529,
+          0.017874684184789658,
+          -0.03937387093901634,
+          0.017379024997353554,
+          -0.0027377386577427387,
+          -0.0060911779291927814,
+          -0.00545998802408576,
+          0.00665653869509697,
+          0.004515138920396566,
+          0.003669034456834197,
+          0.03773200139403343,
+          -0.010408828034996986,
+          0.003963331691920757,
+          -0.005847221240401268,
+          0.007121218368411064,
+          -0.020864123478531837,
+          0.008302279748022556,
+          0.005336073227226734,
+          0.02160761132836342,
+          -0.0056497324258089066,
+          -0.0014482521219179034,
+          0.02328045852482319,
+          0.008859895169734955,
+          0.03469609469175339,
+          -0.01566745527088642,
+          -0.004418330732733011,
+          0.01048627495765686,
+          0.0207402091473341,
+          -0.00580462533980608,
+          -0.008976065553724766,
+          0.007903429679572582,
+          0.0035586729645729065,
+          0.021824462339282036,
+          -0.00515407370403409,
+          -0.0006650729919783771,
+          0.0054367538541555405,
+          -0.0060911779291927814,
+          0.009735042229294777,
+          0.011252996511757374,
+          -0.009076746180653572,
+          0.007539430167526007,
+          0.021530164405703545,
+          -0.009975126944482327,
+          -0.008658533915877342,
+          -0.009649851359426975,
+          0.0010658593382686377,
+          -0.03068435750901699,
+          0.001649613375775516,
+          0.005359307397156954,
+          0.015187285840511322,
+          -0.0004603234410751611,
+          -0.009014789015054703,
+          0.021019017323851585,
+          0.017255110666155815,
+          -0.005936284549534321,
+          -0.0028132491279393435,
+          0.0023795480374246836,
+          -0.0017154429806396365,
+          -0.006160879973322153,
+          -0.0006248975405469537,
+          -0.002656419761478901,
+          0.014730350114405155,
+          -0.0015469965292140841,
+          -0.006617815233767033,
+          -0.00756653631106019,
+          0.0033302053343504667,
+          0.010834784246981144,
+          -0.005661349277943373,
+          0.013057502917945385,
+          0.016805920749902725,
+          -0.00031704717548564076,
+          0.019671445712447166,
+          0.01037010457366705,
+          -0.010447551496326923,
+          0.015938518568873405,
+          -0.007984748110175133,
+          0.006563602481037378,
+          0.0164031982421875,
+          -0.01277095079421997,
+          -0.01246116403490305,
+          0.007210282143205404,
+          0.00665653869509697,
+          0.028918573632836342,
+          0.014792308211326599,
+          -0.02109646424651146,
+          0.014908477663993835,
+          0.015768134966492653,
+          -0.013018779456615448,
+          0.003537375247105956,
+          -0.01784370467066765,
+          0.020260039716959,
+          0.019051872193813324,
+          -0.015535795129835606,
+          -0.017224133014678955,
+          -0.014947201125323772,
+          -0.005092116538435221,
+          0.006331262644380331,
+          0.0033263331279158592,
+          -0.00617636926472187,
+          -0.027989214286208153,
+          -0.0049139889888465405,
+          -0.018633661791682243,
+          0.013870692811906338,
+          -0.008674023672938347,
+          0.0007715621031820774,
+          -0.004511266481131315,
+          0.007984748110175133,
+          -0.0033882902935147285,
+          0.032465629279613495,
+          -0.0051811798475682735,
+          -0.02832997962832451,
+          0.004398969002068043,
+          0.016356728971004486,
+          0.012623801827430725,
+          -0.029243851080536842,
+          0.010463040322065353,
+          0.004805563949048519,
+          -0.012530866079032421,
+          0.005990497302263975,
+          -0.02258343994617462,
+          -0.007632365915924311,
+          -0.0009395244996994734,
+          -0.016743963584303856,
+          -0.027121812105178833,
+          -0.008433938957750797,
+          -0.006993431597948074,
+          -0.005401902832090855,
+          -0.009525936096906662,
+          -0.008720491081476212,
+          -0.011477591469883919,
+          0.02303262986242771,
+          -0.007899557240307331,
+          -0.001866463921032846,
+          -0.0014637414133176208,
+          0.00930908601731062,
+          -0.023776117712259293,
+          -0.015605497173964977,
+          0.006389347370713949,
+          -0.002594462363049388,
+          0.022815778851509094,
+          -0.012747716158628464,
+          0.005940156988799572,
+          -0.009316830895841122,
+          0.009851212613284588,
+          0.015241498127579689,
+          0.004956584889441729,
+          0.017440982162952423,
+          -0.009889936074614525,
+          0.0021026763133704662,
+          -0.008627555333077908,
+          -0.00018986526993103325,
+          -1.027075541060185e-05,
+          0.014428308233618736,
+          0.007624621503055096,
+          0.014141756109893322,
+          0.017487451434135437,
+          -0.013553161174058914,
+          0.005003052763640881,
+          -0.0003274540649726987,
+          -0.017255110666155815,
+          0.0027725896798074245,
+          -0.007984748110175133,
+          -0.0008896682411432266,
+          0.001758038648404181,
+          6.395397940650582e-05,
+          -0.02112744189798832,
+          -0.0009124181815423071,
+          0.0018809852190315723,
+          0.0010222955606877804,
+          -0.025201136246323586,
+          0.00889087375253439,
+          -0.00889087375253439,
+          -0.0014918158994987607,
+          -0.019113829359412193,
+          0.019795360043644905,
+          0.026300877332687378,
+          -0.004619691986590624,
+          0.00036206303047947586,
+          -0.013537672348320484,
+          -0.02049238048493862,
+          0.0001489637652412057,
+          -0.011314953677356243,
+          -0.011345932260155678,
+          -0.0022169100120663643,
+          -0.003214035416021943,
+          0.008805682882666588,
+          0.0004189379105810076,
+          -0.009433000348508358,
+          -0.004956584889441729,
+          -0.011764144524931908,
+          -0.017549408599734306,
+          -0.010401083156466484,
+          0.01933068037033081,
+          -0.0018016024259850383,
+          0.0020794421434402466,
+          -0.0032508226577192545,
+          0.0005271211848594248,
+          0.011748654767870903,
+          0.013475715182721615,
+          0.013344055972993374,
+          -0.023915521800518036,
+          -0.017890173941850662,
+          -0.013320821337401867,
+          0.0029487807769328356,
+          -0.007187047973275185,
+          0.004844287410378456,
+          0.026362836360931396,
+          -0.0009695350890979171,
+          -0.01091997604817152,
+          0.002327271504327655,
+          0.019051872193813324,
+          -0.031272951513528824,
+          -0.013390523381531239,
+          -0.025077221915125847,
+          0.02225816436111927,
+          -0.013351799920201302,
+          0.011516314931213856,
+          0.010191977955400944,
+          -0.025944624096155167,
+          0.006768836174160242,
+          -0.007686578668653965,
+          0.026037558913230896,
+          0.01634124107658863,
+          0.0005934348446317017,
+          -0.00944074522703886,
+          0.013622863218188286,
+          -0.0008722427301108837,
+          -0.013576395809650421,
+          0.002201420720666647,
+          0.014645159244537354,
+          -0.014691627584397793,
+          -0.009510447271168232,
+          -0.004507394507527351,
+          0.004313777666538954,
+          0.012763205915689468,
+          -0.0030339721124619246,
+          -0.004031097516417503,
+          -0.011624740436673164,
+          0.013344055972993374,
+          -0.003930416889488697,
+          0.012383717112243176,
+          0.007380664348602295,
+          -0.014056564308702946,
+          -0.019532041624188423,
+          0.033333033323287964,
+          0.00400399137288332,
+          0.00338054564781487,
+          -0.010997422970831394,
+          0.02740062028169632,
+          -0.009897680021822453,
+          0.010571465827524662,
+          -0.006408709101378918,
+          0.0031985461246222258,
+          0.012321759946644306,
+          -0.012104909867048264,
+          -0.01981084980070591,
+          0.021468207240104675,
+          -0.00364192808046937,
+          -0.0010193913476541638,
+          0.008310024626553059,
+          -0.020182594656944275,
+          -0.00663330452516675,
+          0.002180122770369053,
+          0.009541425853967667,
+          -0.011818356812000275,
+          0.006369986105710268,
+          0.017905661836266518,
+          0.003912991378456354,
+          -0.0007672057254239917,
+          0.019996721297502518,
+          -0.01672847382724285,
+          0.011717676185071468,
+          -0.010850274004042149,
+          0.0016825281782075763,
+          0.0006065039779059589,
+          -0.0048597767017781734,
+          0.0058743273839354515,
+          -0.014536733739078045,
+          0.011988739483058453,
+          -6.244135147426277e-05,
+          -0.013979118317365646,
+          0.007601387333124876,
+          -0.013359544798731804,
+          -0.016836898401379585,
+          0.041542377322912216,
+          -0.005301222205162048,
+          -0.0038665232714265585,
+          -0.0056962003000080585,
+          0.01350669376552105,
+          0.02823704481124878,
+          -0.005336073227226734,
+          -0.01153180468827486,
+          -0.010780571959912777,
+          -0.016573579981923103,
+          0.015907539054751396,
+          -0.01810702309012413,
+          0.002495717955753207,
+          -0.010532742366194725,
+          -0.007903429679572582,
+          0.004302160814404488,
+          -0.003516077296808362,
+          0.014722606167197227,
+          0.00950270239263773,
+          0.023357905447483063,
+          0.00797700323164463,
+          -0.00027348342700861394,
+          0.010393339209258556,
+          0.011663463898003101,
+          0.003243078012019396,
+          -0.010509508661925793,
+          -0.00011949774489039555,
+          -0.02894955314695835,
+          -0.008449427783489227,
+          -0.004863648675382137,
+          -0.001485039247199893,
+          -0.00428667152300477,
+          -0.01239920686930418,
+          0.006617815233767033,
+          -0.0022052929271012545,
+          0.013909416273236275,
+          -0.009254872798919678,
+          -0.017766257748007774,
+          0.03389064967632294,
+          0.004495777189731598,
+          0.006466794293373823,
+          -0.008286789990961552,
+          -0.03705047070980072,
+          -0.0032566310837864876,
+          -0.008116407319903374,
+          -0.013529927469789982,
+          0.004538373090326786,
+          0.01283290795981884,
+          -0.014831030741333961,
+          -0.028097640722990036,
+          0.0019390701781958342,
+          -0.006683644838631153,
+          0.02978597767651081,
+          -0.007798876613378525,
+          0.00419760774821043,
+          0.0037251831963658333,
+          0.0011558911064639688,
+          0.01239920686930418,
+          -0.00845717266201973,
+          -0.0022536972537636757,
+          0.016387708485126495,
+          -0.010858018882572651,
+          0.011051635257899761,
+          -0.010966443456709385,
+          -0.02554190158843994,
+          -0.01456771232187748,
+          0.021344292908906937,
+          -0.0004823473282158375,
+          -0.00024903935263864696,
+          0.014823286794126034,
+          0.02354377694427967,
+          0.011260741390287876,
+          0.00969631876796484,
+          0.006408709101378918,
+          -0.001035848748870194,
+          0.005975008010864258,
+          0.006095050368458033,
+          -0.0024163350462913513,
+          -0.013398268260061741,
+          0.017564896494150162,
+          0.009773765690624714,
+          -0.01810702309012413,
+          0.0008514289511367679,
+          0.0036612898111343384,
+          0.0168833676725626,
+          0.006935346405953169,
+          0.009316830895841122,
+          -0.0143121387809515,
+          -0.023760627955198288,
+          -0.00014932679187040776,
+          0.002873270306736231,
+          0.015721667557954788,
+          0.010579210706055164,
+          -0.014319883659482002,
+          0.0005963391158729792,
+          0.008712747134268284,
+          0.020213572308421135,
+          -0.024209819734096527,
+          0.019702425226569176,
+          -0.004673904739320278,
+          -0.0021084847394376993,
+          0.007946024648845196,
+          0.011632485315203667,
+          -0.011872569099068642,
+          -0.00650551775470376,
+          0.006249943748116493,
+          0.003157886676490307,
+          0.03172214329242706,
+          -0.0017260919557884336,
+          0.009665340185165405,
+          -0.010997422970831394,
+          -0.0002405686245765537,
+          0.0006728176376782358,
+          -0.0006820144481025636,
+          -0.002884887158870697,
+          -0.011020656675100327,
+          0.0018974426202476025,
+          -0.020306508988142014,
+          0.0024473138619214296,
+          0.00463130883872509,
+          -0.005684582982212305,
+          -0.0032972905319184065,
+          -0.019222255796194077,
+          -0.0022672503255307674,
+          0.003293418325483799,
+          0.014335372485220432,
+          -0.031691163778305054,
+          0.004972074180841446,
+          0.010672146454453468,
+          -0.002309845993295312,
+          -0.010416572913527489,
+          0.011423379182815552,
+          -0.0026060794480144978,
+          0.011988739483058453,
+          -0.0038142469711601734,
+          -0.0004905760288238525,
+          0.025681305676698685,
+          -0.005308967083692551,
+          0.014707116410136223,
+          -0.01026167906820774,
+          -0.002993312431499362,
+          -0.028252532705664635,
+          0.005235392600297928,
+          -0.007403898518532515,
+          -0.0012081675231456757,
+          -0.004402841441333294,
+          -0.01130720879882574,
+          -0.0026990154292434454,
+          0.002122037811204791,
+          0.008000237867236137,
+          0.012910354882478714,
+          -0.0007725302129983902,
+          -0.010075807571411133,
+          0.004832670092582703,
+          0.0075355577282607555,
+          -0.015520306304097176,
+          -0.0070437719114124775,
+          -0.020306508988142014,
+          0.029662061482667923,
+          -0.013731288723647594,
+          -0.0011123273288831115,
+          -0.020864123478531837,
+          0.010160998441278934,
+          -0.0063119009137153625,
+          -0.002782270545139909,
+          0.004708755761384964,
+          -0.015109838917851448,
+          0.01344473659992218,
+          0.008751469664275646,
+          0.01450575515627861,
+          -0.02151467651128769,
+          -0.01762685552239418,
+          -0.0044725434854626656,
+          -0.002735802438110113,
+          -0.010648912750184536,
+          -0.0005334137240424752,
+          0.002013612538576126,
+          0.021251358091831207,
+          -0.00032503384863957763,
+          0.0042053526267409325,
+          -0.003214035416021943,
+          0.017224133014678955,
+          -0.010137764737010002,
+          -0.013057502917945385,
+          0.0024260159116238356,
+          0.005994369741529226,
+          0.016186347231268883,
+          0.03158273920416832,
+          -0.020693741738796234,
+          -0.00272999401204288,
+          0.0032740565948188305,
+          0.010656657628715038,
+          -0.004085310269147158,
+          6.304640555754304e-05,
+          -0.027803342789411545,
+          -0.04464024305343628,
+          -0.002422143705189228,
+          0.013057502917945385,
+          -0.015636475756764412,
+          0.012670270167291164,
+          0.008743725717067719,
+          0.010021595284342766,
+          -0.005196669604629278,
+          0.02620794251561165,
+          -0.004793947096914053,
+          0.014304393902420998,
+          0.011733165942132473,
+          0.011911292560398579,
+          0.0009874446550384164,
+          0.0007459079497493804,
+          -0.018354853615164757,
+          -0.008255811408162117,
+          -0.0015954007394611835,
+          0.01765783317387104,
+          -0.027478067204356194,
+          0.008526874706149101,
+          -0.011516314931213856,
+          0.013839714229106903,
+          0.0003666614356916398,
+          0.005661349277943373,
+          -0.01771979033946991,
+          0.012445674277842045,
+          -0.0009337160154245794,
+          -0.025325050577521324,
+          0.004805563949048519,
+          0.011276230216026306,
+          -0.010362359695136547,
+          0.006877261213958263,
+          0.019950253888964653,
+          0.0023079097736626863,
+          0.008495896123349667,
+          -0.014126266352832317,
+          0.010703125037252903,
+          0.009649851359426975,
+          -0.021080974489450455,
+          -0.01233724970370531,
+          -0.007059261202812195,
+          -0.006335135083645582,
+          -0.0035470561124384403,
+          -0.00013105422840453684,
+          0.010176488198339939,
+          -0.016000475734472275,
+          0.010099041275680065,
+          -0.006559730041772127,
+          0.006400964688509703,
+          0.00944074522703886,
+          0.001115231541916728,
+          -0.013576395809650421,
+          0.012716737575829029,
+          -0.0218709297478199,
+          0.008426194079220295,
+          -0.04287445917725563,
+          -0.005847221240401268,
+          -0.015078860335052013,
+          0.005204414017498493,
+          -0.003798757679760456,
+          -0.004453181754797697,
+          -0.006203475873917341,
+          -0.005061137955635786,
+          0.00920066051185131,
+          0.021747015416622162,
+          0.012135888449847698,
+          -0.016620049253106117,
+          -0.007551047019660473,
+          0.005425137002021074,
+          0.0004854935978073627,
+          -0.013336311094462872,
+          0.009316830895841122,
+          -0.01576039008796215,
+          0.0009259713697247207,
+          0.006339007057249546,
+          -0.01072635967284441,
+          0.013452480547130108,
+          0.004731989465653896,
+          -0.003401843598112464,
+          0.013390523381531239,
+          -0.0050417762249708176,
+          -0.0057891360484063625,
+          -0.007272239308804274,
+          -0.013739033602178097,
+          -0.003818119177594781,
+          -0.003072695340961218,
+          0.012747716158628464,
+          0.001529571134597063,
+          -0.0003243077953811735,
+          -0.00782985519617796,
+          0.009154192171990871,
+          0.002220782218500972,
+          -0.0013146566925570369,
+          0.005773646757006645,
+          -0.02300165221095085,
+          0.0001287550403503701,
+          0.004755223635584116,
+          -0.0009487213101238012,
+          -0.006246071308851242,
+          -0.012151377275586128,
+          -0.011903548613190651,
+          -0.038165703415870667,
+          0.009363298304378986,
+          0.0019197085639461875,
+          -0.0076207490637898445,
+          0.0064822835847735405,
+          0.017255110666155815,
+          0.01456771232187748,
+          0.016743963584303856,
+          -0.020693741738796234,
+          -0.02303262986242771,
+          0.005556796211749315,
+          -0.009402021765708923,
+          0.02261441759765148,
+          0.004046586807817221,
+          -0.005847221240401268,
+          0.0013243375578895211,
+          -0.013258864171802998,
+          0.008929597213864326,
+          0.005723306443542242,
+          -0.001592496526427567,
+          0.005425137002021074,
+          -0.013692565262317657,
+          -0.010904486291110516,
+          0.01048627495765686,
+          0.032496608793735504,
+          -0.0004261985304765403,
+          0.0024105266202241182,
+          0.023295948281884193,
+          0.015466093085706234,
+          0.016356728971004486,
+          -0.021111954003572464,
+          0.011167804710566998,
+          0.010060317814350128,
+          -0.0018645278178155422,
+          -0.014892988838255405,
+          0.00513083953410387,
+          -0.012716737575829029,
+          -0.010664401575922966,
+          0.014993669465184212,
+          0.007853088900446892,
+          0.017735280096530914,
+          -0.02369867078959942,
+          0.00789568480104208,
+          0.00042765066609717906,
+          0.00013710475468542427,
+          0.011524059809744358,
+          0.012956822291016579,
+          0.03370477631688118,
+          0.027152791619300842,
+          0.020662762224674225,
+          -0.009285852313041687,
+          0.014366351068019867,
+          -0.009192915633320808,
+          -0.011090358719229698,
+          -0.019470084458589554,
+          -0.015249243006110191,
+          -0.011136826127767563,
+          0.017998598515987396,
+          0.008790193125605583,
+          -0.009355553425848484,
+          -0.005947901867330074,
+          0.01586107164621353,
+          -0.003909118939191103,
+          -0.004635181277990341,
+          0.01646515540778637,
+          -0.026269899681210518,
+          0.016836898401379585,
+          -0.000745423894841224,
+          -0.014668392948806286,
+          -0.014691627584397793,
+          -0.004182118456810713,
+          -0.012670270167291164,
+          0.0003964299685321748,
+          -0.020817656069993973,
+          0.013181418180465698,
+          0.024597052484750748,
+          -0.0030920570716261864,
+          -0.004100799560546875,
+          -0.011299464851617813,
+          -0.0005111477803438902,
+          0.018385831266641617,
+          0.0034773540683090687,
+          0.009192915633320808,
+          -0.0023872926831245422,
+          0.013924905098974705,
+          0.00524313747882843,
+          0.0027958236169070005,
+          -0.0007565568666905165,
+          -0.016263794153928757,
+          0.012197845615446568,
+          0.0026835259050130844,
+          0.012135888449847698,
+          -0.022242674604058266,
+          0.007938280701637268,
+          -0.011764144524931908,
+          -0.0023698671720921993,
+          -0.02228914201259613,
+          0.0189744271337986,
+          -0.01610890030860901,
+          0.001046497724018991,
+          0.003841353114694357,
+          0.0006224773242138326,
+          0.009580149315297604,
+          0.01610890030860901,
+          -0.023388884961605072,
+          -0.0003460896550677717,
+          -0.014908477663993835,
+          -0.021917399019002914,
+          -0.01607792265713215,
+          0.004728117026388645,
+          -0.019501063972711563,
+          0.005444498732686043,
+          -0.0025557391345500946,
+          -0.015125328674912453,
+          0.018478767946362495,
+          0.014691627584397793,
+          -0.004379607271403074,
+          0.002412462839856744,
+          0.005425137002021074,
+          -0.0014405074762180448,
+          0.01971791312098503,
+          0.00013274837692733854,
+          -0.0010048701660707593,
+          -0.009425255469977856,
+          -0.0168833676725626,
+          -0.0065752193331718445,
+          -0.035005878657102585,
+          0.0260530486702919,
+          -0.004670032300055027,
+          -0.016511622816324234,
+          0.014319883659482002,
+          0.009115469641983509,
+          -0.014629669487476349,
+          -0.007059261202812195,
+          0.016294771805405617,
+          -0.024287264794111252,
+          0.02244403585791588,
+          -0.009611127898097038,
+          -0.008774704299867153,
+          0.00784921646118164,
+          0.05154848098754883,
+          0.009146448224782944,
+          0.003492843359708786,
+          0.011036145500838757,
+          0.005839476361870766,
+          -0.0013165927957743406,
+          -0.023683181032538414,
+          0.005568413063883781,
+          -0.007190920412540436,
+          -0.003601268632337451,
+          0.002536377403885126,
+          -0.004801691509783268,
+          0.031149037182331085,
+          -0.004379607271403074,
+          -0.005289605353027582,
+          0.004484160337597132,
+          -0.02196386642754078,
+          -0.0001332324172835797,
+          0.023203013464808464,
+          -2.3128106477088295e-05,
+          -0.023590246215462685,
+          -0.006571347359567881,
+          -0.015071115456521511,
+          -0.013770012184977531,
+          -0.018989915028214455,
+          0.005231520161032677,
+          -0.012755461037158966,
+          -0.0033940989524126053,
+          0.0016660707769915462,
+          -0.005940156988799572,
+          -0.002582845278084278,
+          0.009603383019566536,
+          0.05362405255436897,
+          0.014103032648563385,
+          -0.006335135083645582,
+          -0.005854965653270483,
+          0.009192915633320808,
+          -0.010524998418986797,
+          -0.019795360043644905,
+          -0.009820234030485153,
+          -0.029507169499993324,
+          -0.012747716158628464,
+          -0.0143121387809515,
+          -0.010191977955400944,
+          -0.010277168825268745,
+          0.00963436160236597,
+          -0.018122512847185135,
+          -0.007628493942320347,
+          -0.020337486639618874,
+          0.002941036131232977,
+          -0.011268485337495804,
+          -0.014714861288666725,
+          -0.008054450154304504,
+          -0.04807887226343155,
+          0.005796880926936865,
+          0.019593998789787292,
+          0.01263929158449173,
+          0.002515079453587532,
+          0.004383479710668325,
+          -0.01993476413190365,
+          -0.030792783945798874,
+          0.013839714229106903,
+          0.011283975094556808,
+          0.011733165942132473,
+          0.03283737599849701,
+          -0.011709931306540966,
+          -0.00813964195549488,
+          -0.004693266469985247,
+          -0.0017251238459721208,
+          0.012066186405718327,
+          0.00969631876796484,
+          -0.004917861428111792,
+          -0.012375972233712673,
+          0.016217326745390892,
+          -0.01981084980070591,
+          -0.023683181032538414,
+          -0.01672847382724285,
+          0.006157007534056902,
+          0.00813964195549488,
+          -0.008054450154304504,
+          0.002973950933665037,
+          0.0056962003000080585,
+          -0.007159941829741001,
+          -0.00022423222253564745,
+          0.004890755284577608,
+          -0.023048119619488716,
+          0.0008490087930113077,
+          -0.002257569460198283,
+          0.0061376458033919334,
+          0.021266845986247063,
+          -0.010168743319809437,
+          -0.0014075926737859845,
+          -0.0053902859799563885,
+          -0.0028984404634684324,
+          -0.006764963734894991,
+          0.02620794251561165,
+          0.009154192171990871,
+          -0.009882191196084023,
+          -0.01011453103274107,
+          -0.01129171997308731,
+          -0.0011278166202828288,
+          -0.017255110666155815,
+          -0.016186347231268883,
+          0.0015566773945465684,
+          -0.016712984070181847,
+          0.013971373438835144,
+          -0.00876695942133665,
+          0.00400399137288332,
+          -0.014110777527093887,
+          0.02019808255136013,
+          0.01929970271885395,
+          -0.009711808525025845,
+          0.023776117712259293,
+          -0.007709812838584185,
+          -0.038382552564144135,
+          -0.00800798274576664,
+          0.01843230053782463,
+          -0.006118284538388252,
+          0.010037084110081196,
+          -0.011314953677356243,
+          0.008209343999624252,
+          0.00015319912927225232,
+          0.0018122512847185135,
+          -0.0003112386621069163,
+          0.027555514127016068,
+          0.015535795129835606,
+          -0.005560668651014566,
+          -0.005494839046150446,
+          0.004255692940205336,
+          0.009541425853967667,
+          0.017580386251211166,
+          0.012151377275586128,
+          -0.021019017323851585,
+          -0.014257925562560558,
+          -0.016263794153928757,
+          0.011012911796569824,
+          0.015125328674912453,
+          -0.0040891822427511215,
+          0.01691434532403946,
+          0.014126266352832317,
+          -0.019098341464996338,
+          0.0019884423818439245,
+          -0.008193854242563248,
+          -0.00158184755127877,
+          -0.03029712475836277,
+          0.011043890379369259,
+          0.0074387495405972,
+          -0.0072218989953398705,
+          0.036616768687963486,
+          0.009858956560492516,
+          0.037577107548713684,
+          0.018680129200220108,
+          0.0008974128868430853,
+          0.009045767597854137,
+          -0.018370341509580612,
+          0.013065247796475887,
+          0.01500141341239214,
+          -0.012724482454359531,
+          -0.005734923295676708,
+          0.0013446672819554806,
+          0.0007551047019660473,
+          0.012631546705961227,
+          0.02004319056868553,
+          -0.01795213110744953,
+          0.009773765690624714,
+          -0.015032391995191574,
+          -0.005250881891697645,
+          0.006555858068168163,
+          -0.013932649977505207,
+          -0.008495896123349667,
+          -0.00963436160236597,
+          0.019408127292990685,
+          -0.012600568123161793,
+          0.016573579981923103,
+          -0.009595638141036034,
+          0.015520306304097176,
+          -0.029724018648266792,
+          -0.003669034456834197,
+          -0.006679772399365902,
+          -0.019593998789787292,
+          -0.012887120246887207,
+          0.009820234030485153,
+          0.010633422993123531,
+          -0.009301341138780117,
+          -0.011562783271074295,
+          0.020368466153740883,
+          -0.004185990896075964,
+          0.013529927469789982,
+          0.021189400926232338,
+          0.00737291993573308,
+          0.024132372811436653,
+          0.0005605200421996415,
+          0.014164989814162254,
+          0.005463860463351011,
+          0.010362359695136547,
+          -0.004677776712924242,
+          0.005161818582564592,
+          0.00932457484304905,
+          0.014203713275492191,
+          0.017766257748007774,
+          0.013196907006204128,
+          -0.00598275288939476,
+          0.007419387809932232,
+          0.02016710489988327,
+          0.0016254113288596272,
+          -0.030312614515423775,
+          -0.009719553403556347,
+          0.0001736498816171661,
+          0.00924712885171175,
+          0.009185170754790306,
+          3.817877222900279e-05,
+          -0.01093546487390995,
+          0.011144571006298065,
+          0.00969631876796484,
+          0.002286612056195736,
+          0.006280922330915928,
+          -0.01456771232187748,
+          -0.020957060158252716,
+          0.006106667220592499,
+          0.01798310875892639,
+          0.013917161151766777,
+          -0.014722606167197227,
+          0.022784801200032234,
+          0.006145390681922436,
+          -0.021406250074505806,
+          -0.01840132102370262,
+          -0.0009559819591231644,
+          0.0004542729293461889,
+          -0.002265314105898142,
+          0.03720536455512047,
+          -0.0011539548868313432,
+          -0.005955646280199289,
+          -0.0054754773154854774,
+          0.02775687538087368,
+          0.02049238048493862,
+          0.009177426807582378
+        ],
+        "embedding": [
+          0.0008215984936325879,
+          -0.019349418284688353,
+          -0.02322660633029795,
+          0.009747820845194852,
+          0.007353771220903641,
+          0.015252908783199839,
+          -0.012176175829980202,
+          0.051898051094966605,
+          -0.0068130353886146165,
+          0.008189212816890595,
+          0.008102287831931905,
+          0.01682801136011614,
+          0.02005557996049711,
+          -0.020779879580497183,
+          0.002332165268329781,
+          0.02337485065616377,
+          0.006074992895540845,
+          0.02576575146535689,
+          0.0007517286429767504,
+          -0.03763909942314134,
+          0.0011204729720314833,
+          0.00041212613589632806,
+          -0.06089176800314795,
+          0.012852035873395388,
+          -0.008901127092058543,
+          0.01702347055484216,
+          0.005803446395466281,
+          0.017889570275475296,
+          -0.05375047382413552,
+          0.021623778774252724,
+          0.014784317465883424,
+          -0.0018881880857885708,
+          0.0004735328554290247,
+          0.012393802850407656,
+          -0.019234442968055952,
+          -0.019566283611638122,
+          0.04407789373296535,
+          -0.0053565873209102995,
+          -0.026240866282480922,
+          0.048107902270426675,
+          -0.0054681314939387235,
+          -0.0009441397948041965,
+          -0.015205079626883548,
+          0.006885981841893229,
+          0.01403724256866746,
+          -0.026086548246792754,
+          0.0034778946051893067,
+          0.00044587533948775473,
+          0.032851328435891634,
+          -0.03882097333255334,
+          -0.011126344604254894,
+          -0.01648943336384904,
+          0.035810456555802166,
+          -0.019384589526165754,
+          -0.019430702768170285,
+          0.014297440272239852,
+          -0.007883453204663151,
+          0.050996900793086844,
+          0.06007149434153715,
+          0.03406832053483379,
+          0.025193439934671852,
+          0.00027785129663604897,
+          -0.008014391056168083,
+          -0.018806730908490114,
+          0.027193788903098558,
+          -0.006195006670328595,
+          -0.020466000344352826,
+          0.0068939884219589185,
+          0.017252276167086602,
+          -0.045073444648704136,
+          -0.049242269917908754,
+          0.023021140866008313,
+          -0.015676872472071106,
+          -0.024497517908898602,
+          0.005873633718973869,
+          -0.022828585938203683,
+          0.011748341004940436,
+          0.04624205727927451,
+          -0.015442165499674641,
+          0.018565551071491767,
+          0.015246505395507096,
+          0.004119269796224144,
+          0.004862735250819252,
+          -0.000145323321960046,
+          0.05155838952887077,
+          0.0027052423345323936,
+          -0.016218063670586664,
+          -0.07462470920944148,
+          0.043695974746295135,
+          -0.027608273720377708,
+          -0.03860912774695494,
+          0.002823110565150435,
+          0.026471469849508766,
+          -0.0026063577189074103,
+          -0.029230652014151735,
+          0.05684164470477843,
+          0.00022182868300572677,
+          0.015590845552956364,
+          -0.024939617549529242,
+          0.01035803286203917,
+          -0.010225298012384327,
+          0.019571847648759157,
+          0.01841613041441054,
+          0.0156981470538354,
+          0.021764297694793343,
+          -0.018831230007033416,
+          -0.014419529638789993,
+          -0.0029965475899652273,
+          0.028881185012095884,
+          0.09163526460163937,
+          0.022644302207811818,
+          -0.017933909936330133,
+          -0.04663928709016869,
+          0.004962951444105128,
+          -0.041333041174526026,
+          -0.027545628007617527,
+          -0.037142394579894954,
+          -0.013358143162788402,
+          0.02719686432396677,
+          -0.0034971800937504205,
+          0.04413635817371732,
+          0.04903857110328917,
+          0.03186067155630788,
+          0.007831051050269414,
+          -0.025257538623313574,
+          0.023517513773109414,
+          -0.0016527403209505964,
+          -0.0016101860707429347,
+          0.014448515185903624,
+          0.009905018312477684,
+          0.0024114582235163955,
+          0.00172043851384822,
+          -0.03818316221102173,
+          -0.02304537008110578,
+          0.01024969233570089,
+          -0.01901116356327339,
+          0.019791121460848986,
+          0.07985067340821365,
+          -0.051903191272236665,
+          -0.025924646976556942,
+          0.014766511532849252,
+          -0.002073086076894999,
+          0.030959269926800852,
+          0.006394417986617089,
+          -0.028177586693713442,
+          0.010725285336674526,
+          0.0009614052276924192,
+          0.07500253788978564,
+          -0.00036794182149960157,
+          0.004058458332826519,
+          -0.013218269166951234,
+          0.027613539268683245,
+          -0.012995170442480582,
+          0.023388707018488407,
+          0.026389423313105165,
+          -0.003068639956061266,
+          -0.02661967649632026,
+          0.026145274570635913,
+          -0.03300591322174331,
+          -0.004556072066733024,
+          0.020045722084112896,
+          0.0033437343914784865,
+          -0.010224220905483085,
+          -0.0025025498684079703,
+          0.03460103245395426,
+          0.04514362357278504,
+          0.03797031988058962,
+          -0.004586662475576956,
+          -0.06351723722596923,
+          0.06037983484369874,
+          0.019415753124363902,
+          0.028850618647994476,
+          -0.0034810237045849006,
+          0.02957938154808314,
+          0.02642652523231673,
+          0.013187653724133275,
+          -0.007856320804109569,
+          -0.007065882659482447,
+          -0.016672784625197085,
+          0.029020161154209707,
+          -0.013243058921970134,
+          -0.014900932834241732,
+          0.006001769901896033,
+          0.008648456186944271,
+          0.03484763413464497,
+          0.012279368984306564,
+          0.02379643508687133,
+          0.04933044959399941,
+          -0.021047723935647383,
+          -0.007264823701706426,
+          0.0007058774795716792,
+          -0.010512071832195085,
+          0.027489671668749157,
+          0.009731157570361245,
+          -0.03195919640758579,
+          -0.0033626031972691915,
+          -0.04282419484651207,
+          0.01873977910485268,
+          -0.014398285316695653,
+          0.012854631447494207,
+          -0.0017372959978092432,
+          -0.037331867825735776,
+          0.001088209292238142,
+          -0.012345684323522151,
+          0.037096063326414734,
+          0.005050219690024652,
+          0.0008906051515353004,
+          -0.003697709449630797,
+          0.006548479142737107,
+          -0.010727663712299397,
+          -0.007486226350610755,
+          -0.02704618481069485,
+          -0.019752552016312926,
+          -0.014632581296303486,
+          -0.0037501138119724717,
+          0.014588411795703245,
+          -0.007119137071719376,
+          0.016440258608998535,
+          -0.03686645352111013,
+          -0.023546402448713415,
+          0.005628703432152666,
+          0.024988145751166143,
+          0.011971199595808573,
+          -0.010868778329190568,
+          -0.03270046433624425,
+          -0.021950559285357267,
+          -0.011889914022657738,
+          -0.005123971523701009,
+          0.02619713302280345,
+          -0.021990810305904974,
+          0.01165257474485436,
+          -0.009117901804315863,
+          -0.007492082755249399,
+          0.010725902070291232,
+          0.01862811996181951,
+          -0.026694281868322685,
+          -0.01623776415010199,
+          -0.009403997538684658,
+          -0.02207329027097839,
+          0.025939540497196416,
+          0.014178803271471762,
+          -0.001141821318145744,
+          -0.006281187017652108,
+          -0.035993365198554964,
+          0.0026192776111449633,
+          -0.015254976771874396,
+          -0.0116872453009606,
+          -0.015780016491993135,
+          0.021421328283124968,
+          0.02889432250124136,
+          -0.01871009416296692,
+          -0.026540472640514536,
+          0.015703370682110186,
+          -0.0002849235245164438,
+          -0.02594930889995505,
+          0.04491828744809679,
+          0.044063522197410734,
+          -0.037441864491547405,
+          -0.007005339187980476,
+          -0.030381794393931814,
+          0.06901083845539985,
+          -0.017604082400603037,
+          -0.04060591381445593,
+          0.06044850935418443,
+          -0.01756490171206357,
+          0.025539717592705295,
+          0.03452018789549939,
+          0.0016217577522523927,
+          0.00018825054264059277,
+          -0.002563606092068712,
+          0.04770277039047993,
+          -0.0032010143255574255,
+          -0.03425328601475849,
+          -0.02105909767072799,
+          -0.00840881022563103,
+          -0.028699447677549002,
+          -0.015925160861589006,
+          -0.0055750742281166,
+          0.017513987082479483,
+          0.010399718005952185,
+          0.02267201252273823,
+          0.01258511878601025,
+          0.0015081978484714903,
+          0.01874981572091931,
+          0.011158889421057756,
+          -0.002607810686437211,
+          -0.003732023411827874,
+          -0.007787189972165309,
+          -0.05524012485122728,
+          -0.025110689689279583,
+          -0.03946383236703947,
+          0.004443572529325475,
+          -0.018745077598477586,
+          0.005926160888274727,
+          0.039430331582190524,
+          -0.0049500324516682,
+          0.004352795787899792,
+          0.015566139991780173,
+          0.013148156356408315,
+          -0.015279560337042242,
+          -0.03316846614871165,
+          0.0001592209771257332,
+          0.018362411161743347,
+          -0.012706582418607377,
+          -0.03424477956681259,
+          -0.001991022841226589,
+          0.0033467286047492175,
+          0.03357019284647071,
+          -0.024359110796680374,
+          0.00359117029951049,
+          0.005061818038334802,
+          0.011359582931760315,
+          -0.026793992746244076,
+          -0.018170136079817746,
+          -0.017421603453826397,
+          0.03996608926471779,
+          0.02587013058343799,
+          0.015763410145648683,
+          -0.04512536449983358,
+          -0.02293473890636517,
+          0.048575083783480195,
+          -0.009589860756342604,
+          -0.06486262416944208,
+          0.03447666009581266,
+          0.016105654870253654,
+          0.003526582643722421,
+          -0.01354754623151814,
+          0.05401575248587799,
+          -0.03563190295559527,
+          -0.017365166993914202,
+          -0.0046288709698227735,
+          -0.033657212157212645,
+          -0.007686252059880836,
+          -0.001385273642139782,
+          0.048229062789530236,
+          -0.029083855744728694,
+          -0.030465559177859865,
+          0.0024368661249980416,
+          -0.024144031154493057,
+          0.021693278579520123,
+          -0.016405088055634124,
+          -0.018439935250344958,
+          0.006948603359995389,
+          -0.033943037395199695,
+          -0.003007796423804453,
+          -0.016783195426536633,
+          -0.02270589491206155,
+          0.020874438753828814,
+          0.016724666903417947,
+          0.027033184180344852,
+          0.0019847647852695785,
+          0.017749524526371133,
+          0.0204661893602285,
+          -0.005604929968008821,
+          0.028462234633072944,
+          -0.020790459496727066,
+          -0.04183816355941877,
+          0.03946487892663695,
+          0.01192861758684541,
+          -0.0018024841275380242,
+          0.01077082591611702,
+          -0.01475239020295694,
+          -0.03952356245715099,
+          0.03683950018557526,
+          -0.056767672022551764,
+          -0.007507600090017693,
+          0.009530544352246083,
+          -0.00788856051420074,
+          -0.011204694027562521,
+          -0.010587701466318598,
+          -0.028604940377315504,
+          0.023749537635431462,
+          -0.013672419168235691,
+          0.01953449229707597,
+          -0.011449939491554354,
+          -0.012238180529977605,
+          0.05795536247491174,
+          -0.021518581774137322,
+          -0.040672955795925836,
+          0.026066659238628543,
+          -0.014061106927896723,
+          0.03328357230532424,
+          0.03957107406164427,
+          0.030670947051333917,
+          -0.004278577327978564,
+          0.025885946540364826,
+          0.0319106101668716,
+          -0.003462160036198642,
+          -0.04686868330851868,
+          0.00555303933159587,
+          0.02156143971867743,
+          -0.03739529489075782,
+          -0.01025626126223168,
+          -0.02141970191890338,
+          0.022134910980548407,
+          0.017807484142010496,
+          -0.007232643256393455,
+          0.011318041192005274,
+          -0.010489012862455913,
+          -0.013003217163997554,
+          -0.00847279880392288,
+          0.02357274819910594,
+          -0.0016485066139744577,
+          -0.005873113634992993,
+          0.006424249429163587,
+          0.004008703814301658,
+          0.0054639858216457724,
+          0.010198079500408441,
+          0.017694720520798957,
+          -0.017087021465208255,
+          -0.0025558869389290554,
+          -0.016503579212015177,
+          -0.0420082404922411,
+          -0.003563173424359332,
+          0.022924161700566468,
+          0.012345933179990547,
+          -0.00729908668365632,
+          -0.004234683369642856,
+          -0.0022962545569187934,
+          -0.020666205489143392,
+          -0.014796149043465308,
+          0.024562653223873056,
+          0.030217494971086438,
+          0.007810961433007407,
+          0.007781263640449456,
+          -0.0030348559388780655,
+          0.007310876971216526,
+          0.016099591640649783,
+          0.02311929597780174,
+          -0.04160162038353416,
+          0.027152100845897007,
+          -0.00025396315502635146,
+          0.023717123370023598,
+          -0.04186918759097667,
+          -0.017988531857836703,
+          -0.03106663560912813,
+          -0.023595079699181436,
+          0.02437134274097885,
+          -0.02317559303912261,
+          -0.012327263775257306,
+          0.000989566600374836,
+          0.00954921216572729,
+          -0.005155079395685017,
+          -0.017508404061379164,
+          -0.04582321461886958,
+          -0.0014367264231457263,
+          -0.030417229142190933,
+          -0.017829710557332023,
+          -0.001056472801554354,
+          0.005016912084754242,
+          -0.06276644565618253,
+          0.0109965681725148,
+          -0.07270669271977365,
+          -0.004542796125637244,
+          0.040139998232966566,
+          -0.017890441783630643,
+          0.055909102081208445,
+          -0.05029104308417623,
+          0.01673962668706085,
+          -0.04507293803055318,
+          0.015385506084493484,
+          0.025677372277539627,
+          0.030235171586316813,
+          -0.022189978375852563,
+          0.008461188938299745,
+          0.008033707483259569,
+          -0.045089505755429415,
+          -0.04144698743458133,
+          -0.013640256640978024,
+          0.01036782197518379,
+          0.0012696958402094232,
+          -0.016090728932978384,
+          0.023838437877040163,
+          0.011946482002767135,
+          0.03342604921859589,
+          0.04211801335157251,
+          0.002708015395637621,
+          -0.047156646610976634,
+          -0.013104464812562618,
+          -0.003933116891120681,
+          0.03155321708116329,
+          -0.018866109579446184,
+          0.015376812010630058,
+          -0.01955557376085726,
+          0.016255217383093355,
+          0.017868332102274922,
+          -0.020325179506828335,
+          0.013434364256590097,
+          0.004176904701252156,
+          -0.03877060944639854,
+          0.0006475542749369515,
+          0.00042468300294954125,
+          -0.02116992978038693,
+          0.0032654387271108293,
+          -0.00575003786159731,
+          -0.05149954071159492,
+          -0.031382926943128234,
+          -0.015730563640253065,
+          0.0382312502769352,
+          0.004297939848910064,
+          0.019724983545557365,
+          -0.017907869879607064,
+          -0.005165628493894463,
+          0.03698976707579373,
+          0.008501001373266799,
+          -0.01725527821298066,
+          0.006879620077191753,
+          0.02455792200929138,
+          -0.03224193972301503,
+          -0.039181328884413266,
+          -0.0008206986106407542,
+          -0.023306758893533952,
+          0.01672946960876128,
+          -0.02658587698869307,
+          0.005873620623736886,
+          0.0025331170979801232,
+          -0.03490809296591325,
+          0.02456764314036355,
+          0.003291113462353163,
+          -0.004303423672882112,
+          -0.020359544091059775,
+          -0.016892415165120203,
+          0.017174797487220006,
+          0.04391489511478065,
+          0.007327321684948843,
+          -0.003943713832544148,
+          0.00862684231313155,
+          -0.006361172785753631,
+          0.05826688726534012,
+          -0.02284956969636637,
+          -0.026680023999604643,
+          0.027527436347931256,
+          -0.022597669279069723,
+          -0.006277863878949772,
+          -0.017343446085471857,
+          -0.012065192152203348,
+          -0.0004183712490442143,
+          0.019339311388446943,
+          -0.019423562920897294,
+          -0.010212527447243049,
+          0.004783006556748757,
+          0.012824133870627983,
+          -0.004855945108275706,
+          0.006239949458562236,
+          0.006847581690471813,
+          0.03094005096378827,
+          0.026993131409761086,
+          -0.011331881651539968,
+          0.03297014291876675,
+          0.010265600643742144,
+          4.666113318251278e-05,
+          0.001970879281172325,
+          0.002523664335535017,
+          0.0034107811520368574,
+          -0.03143890864030425,
+          -0.01624584278940617,
+          0.013468729048885906,
+          0.005233309338223403,
+          -0.004550544967856448,
+          -0.023783653541634683,
+          -0.009402494239255264,
+          0.03334923712760375,
+          -0.003260424721959948,
+          0.02748395114281811,
+          -0.0058374897745283065,
+          -0.01985840930273332,
+          0.017595418271610163,
+          0.02881743246631328,
+          -0.05928552440605384,
+          0.017062540894682354,
+          -0.0011508547522925793,
+          0.008953761538519997,
+          0.01610561914042553,
+          -0.008894160513648953,
+          0.013301991014533684,
+          0.04080863883897929,
+          0.010738051773755976,
+          -0.023345002072886145,
+          0.004215726310838674,
+          0.01541749448570608,
+          0.01291813299324752,
+          -0.022747942296528025,
+          0.0008640791861357522,
+          0.011008522066277018,
+          0.005893793007012561,
+          0.0026106450193140156,
+          -0.006278394080583205,
+          -0.025659188287146268,
+          0.024180550333596625,
+          -0.0044491292740074925,
+          0.026967973346698617,
+          0.04390909273429254,
+          0.009596855239692042,
+          -0.03686020608700941,
+          -0.013109951664921729,
+          0.0002799680359670127,
+          -0.0038537577602593366,
+          -0.0037552094562213916,
+          0.011105295862811114,
+          -0.04185615640149602,
+          0.019950890919369135,
+          0.006453241355231516,
+          0.015314645427201533,
+          -0.020031193483734364,
+          -0.024530173488182337,
+          0.0031949828520724337,
+          -0.01492645564652531,
+          0.012875827757040941,
+          0.005057170413681063,
+          -0.004319834710426384,
+          0.005386862713494232,
+          0.00030848054013413157,
+          0.008384941028326983,
+          -0.010811526993301901,
+          0.03175068297908541,
+          0.004680987487229685,
+          0.009264547472281849,
+          -0.02856314241700808,
+          -0.021083721348429348,
+          0.020113396771629096,
+          0.03572366771051171,
+          -0.01708529918264109,
+          0.03346921724975398,
+          -0.009042046660366718,
+          0.0033219155387035227,
+          0.0151720168318272,
+          -0.015805318210244414,
+          -0.008057236707694303,
+          -0.0005483009439015931,
+          -0.002917527004423299,
+          -0.014538511124345527,
+          0.007302979537905532,
+          -0.0208529958115422,
+          -0.02965227242147972,
+          0.020064178921373875,
+          -0.00011892622932349096,
+          0.016455188521316146,
+          0.010335044657747514,
+          -0.007816155992852236,
+          -0.0052262330850905165,
+          0.006959344748883874,
+          0.04432147882994085,
+          0.026734484553994495,
+          0.05547661378166294,
+          -0.005787997952888261,
+          -0.004559444854138438,
+          -0.0082295581123363,
+          -0.006588172755707144,
+          0.04524009609473585,
+          0.015915003261638767,
+          -0.004910285095602864,
+          0.017495607570191165,
+          0.01557886571266685,
+          0.03858259260136037,
+          0.0027648525902558316,
+          -0.016912796336835408,
+          0.04014755669075221,
+          -0.004077405782937414,
+          0.008616827127381418,
+          0.011102692199778469,
+          0.0024185742218798562,
+          0.00807141794538637,
+          0.02996274720815659,
+          -0.012429186019907761,
+          0.01680942269506771,
+          -0.04571916592313758,
+          -0.02775530246832996,
+          0.014370280587419414,
+          -0.004931905540824582,
+          0.03419959665162407,
+          0.01668521021731861,
+          -0.023105305446763272,
+          -0.024761385385514877,
+          0.021529428247380984,
+          -0.002603042335561961,
+          -0.024135326707477325,
+          -0.015906833215844473,
+          0.003286605488908195,
+          -0.0023437997044735617,
+          0.0060251848371355166,
+          -0.013481652076194643,
+          0.0002908426939460218,
+          0.01953362396253743,
+          0.0038544889405965057,
+          0.020760460182873287,
+          -0.016224396121239458,
+          -0.00259233446682179,
+          -0.009282861237653619,
+          -0.02952130765234087,
+          0.017988370145301916,
+          0.006138236653154911,
+          -0.007607182955958238,
+          -0.009516767050028553,
+          0.0048347778733230365,
+          -0.005819209642901871,
+          -0.0025851701689263706,
+          -0.01073014664868326,
+          0.012433465684360061,
+          0.025784014942833097,
+          -0.016875624785552303,
+          0.007944349714123104,
+          8.651589872318479e-05,
+          -0.022306044741692753,
+          -0.022127016745066307,
+          0.020143472307319017,
+          0.002792646249762062,
+          0.008768813346304779,
+          -0.004074214276300631,
+          -0.00015024341785520542,
+          0.002990550758627699,
+          0.01951746553838439,
+          0.03896451381932467,
+          0.004232673242619471,
+          0.027876586543280867,
+          0.0145860303663199,
+          0.03994234200854897,
+          0.009657017763624678,
+          0.00696607158576712,
+          -0.026055387694656488,
+          0.02819890357011778,
+          0.00018297502510753014,
+          0.043190622821775236,
+          0.022488402221902314,
+          -0.0407730101161195,
+          -0.0021946518523759066,
+          0.01669520746276111,
+          0.013212623257794862,
+          0.017009844136059122,
+          -0.004537512994041156,
+          -0.014041213424740049,
+          0.028135404528975894,
+          -0.022859586430516003,
+          0.0008637006862481223,
+          -0.004486332934188017,
+          -0.025031795510001887,
+          0.007351370254175368,
+          -0.004567816916440959,
+          0.030025413528233364,
+          0.0014976740344892735,
+          -0.022285096307825804,
+          0.022837170656498543,
+          -0.009551269792616045,
+          0.011045530488684207,
+          0.008809047185288755,
+          0.020065001354321994,
+          0.031993362733068575,
+          0.009143210434415064,
+          0.02133640814803879,
+          0.012241777474889248,
+          0.014771538127546474,
+          -0.008804192497858033,
+          -0.02341427260922065,
+          0.023167424359504862,
+          -0.0011598649565273595,
+          -0.029627604902846688,
+          0.0021991474725787303,
+          0.003261878079344105,
+          -0.0005935713331682358,
+          -0.009144689470851402,
+          -0.005580557862988033,
+          -0.007240836451875213,
+          0.019631377011298905,
+          -0.007085213886352692,
+          0.007017573363709754,
+          -0.004074171791476129,
+          -0.00822871835665378,
+          -0.0421101573155962,
+          0.004210914075286222,
+          0.003408427700622687,
+          -0.01752179330665445,
+          0.006384440404174512,
+          0.011121481025573962,
+          0.043020575455262985,
+          -0.022809979913698254,
+          -0.0005737138409049003,
+          0.019206623212867476,
+          0.0009562551889820241,
+          -0.023427467810802533,
+          -0.038931688376706186,
+          0.007172345672967499,
+          -0.03467050750542982,
+          0.023185301620612612,
+          -0.0007075596339842658,
+          0.02387977204565927,
+          -0.03043468414449851,
+          0.013909304572370437,
+          0.004784668158421625,
+          0.012524879174804915,
+          0.021654512058639427,
+          0.03142146922870715,
+          -0.02053473475540882,
+          -0.050409862638621725,
+          -0.005406667316025549,
+          0.00035138762584085976,
+          0.022018487016014856,
+          -0.026014580970974616,
+          0.00025633004699302783,
+          -0.003349094027291921,
+          -0.02401366769212778,
+          -0.01735996745614562,
+          0.01962340791446025,
+          -0.050011480783403936,
+          0.015042913576492952,
+          0.0262737424555697,
+          -0.02507473901367252,
+          -0.05408889321034101,
+          -0.018988587082325706,
+          -0.028217893770659686,
+          -0.009285770094319318,
+          -0.00661159230987788,
+          0.028597405154740078,
+          0.003525576256054959,
+          -0.011462219358952466,
+          0.025401668390029348,
+          0.006155837251830549,
+          0.0026050693705924593,
+          0.015985722392086222,
+          0.007289910674212967,
+          -0.011452511453623235,
+          -0.014599577191293152,
+          -0.04558584556100621,
+          0.010197536827229795,
+          -0.0045431962637690155,
+          -0.019372598050508005,
+          -0.030147301333718016,
+          -0.0137828704002156,
+          -0.027060515748528666,
+          -0.0017291966950107505,
+          0.053783684195381654,
+          0.03292494290301388,
+          0.005560172296221419,
+          -0.0026085670565688026,
+          -0.010466890489356092,
+          0.036247483457995895,
+          -0.006789746401727701,
+          0.03662690472838588,
+          0.001784522073210422,
+          -0.0242673253380092,
+          0.01169382884548301,
+          0.02892627181542449,
+          0.009858451662166387,
+          -0.0005592283645355213,
+          -0.006160132931009508,
+          -0.008782830470079547,
+          0.0031207630972208394,
+          0.025613233596604418,
+          0.003268170392795548,
+          -0.010062991425127038,
+          0.018106967407336815,
+          0.004480291699976396,
+          0.02010374270557836,
+          0.03555357587371324,
+          0.023489837757983054,
+          -0.021782173652026497,
+          -0.026581762796336082,
+          -0.0013561486781077014,
+          0.0072266106510934505,
+          0.03551967295942857,
+          -0.01651315514034218,
+          0.0001274396319698452,
+          -0.008272735507120127,
+          -0.00894931703949637,
+          0.011149952898504355,
+          -0.0029671593121986116,
+          -0.0030752010285951576,
+          0.01662711184497578,
+          -0.0011672263615879709,
+          -0.033448222499147764,
+          0.0078288560210653,
+          -0.023496331346917084,
+          0.0051040606404415545,
+          -0.004113104164416575,
+          -0.007076525381595441,
+          -0.002157548492138289,
+          0.006302962169276006,
+          -0.008065841090139872,
+          -0.00035242603988955366,
+          -0.01586406637763306,
+          0.008271894439780969,
+          0.007438312999409211,
+          -0.0069617481460406095,
+          0.011402932583388205,
+          0.002666895909532698,
+          0.0005219881149353855,
+          -0.012159425670629751,
+          -0.012042145306711322,
+          -0.00957276588761916,
+          -0.03401863879157551,
+          -0.02604353712472079,
+          -0.024949533021906844,
+          -0.0014209149651479556,
+          -0.006566685925666972,
+          -0.03187796739029736,
+          -0.01209152137926458,
+          -0.011110763124407752,
+          0.007756961097537267,
+          -0.013260548545917543,
+          0.005433285079114047,
+          -6.399098520041351e-05,
+          -0.0044960488826674555,
+          0.016615636438440516,
+          -0.003643291599919672,
+          -0.019488010211707343,
+          -0.03927818878149417,
+          0.008209336177358258,
+          -0.02805593125673831,
+          -0.0038816771550190788,
+          0.007607119219721711,
+          -0.00528681677614909,
+          -0.029502897838743516,
+          -0.009149794737430706,
+          0.008767001199311985,
+          0.003507217008280066,
+          -0.01773349756494436,
+          0.03579670773605949,
+          0.026971617790626075,
+          0.011875179879209696,
+          -0.019891342676443626,
+          0.0508929813275565,
+          -0.0047010735337056275,
+          -0.012544935684845649,
+          0.018297503869683027,
+          0.02355022268131519,
+          0.03273167043015095,
+          -0.024830243320788693,
+          0.0009370814985214882,
+          -0.021937793653233454,
+          0.00535662615596573,
+          -0.035009915581788176,
+          0.0028595659314725012,
+          -0.014395027018699446,
+          0.015596459769560532,
+          0.03397372881171701,
+          -0.013254226072981457,
+          0.011195747181764677,
+          -0.016299384375589934,
+          0.00730244731485402,
+          0.00569944090461748,
+          -0.012916284280345044,
+          -0.005812311882399539,
+          -0.022902815152116493,
+          0.020140276669979015,
+          0.005809799950176385,
+          -0.0008130655714656685,
+          0.0104085826652653,
+          -0.044723432238900435,
+          0.03286429597337671,
+          0.008730225944501239,
+          0.017599189944921396,
+          0.0029762446259779998,
+          -0.0003068671951232415,
+          0.011611551606224214,
+          0.011228401175698422,
+          -0.016029727902740706,
+          0.004345913009102136,
+          -0.004613061137378433,
+          -0.03850533909474302,
+          0.009363351681220008,
+          -0.03631184057978666,
+          0.014427784677340159,
+          0.016026872002233575,
+          -0.004170582842643461,
+          -0.025166189571409366,
+          -0.005428471012333673,
+          0.010827910811085384,
+          -0.009783069664327864,
+          -0.009987805234276443,
+          -0.04163358183060102,
+          -0.025236440364963034,
+          0.034097508113604576,
+          -0.002119474613596975,
+          -0.025826403511707147,
+          -0.010830738112107807,
+          -0.011351132636491575,
+          0.020207687645133,
+          -0.006542026283946858,
+          -0.0006263975804936843,
+          -0.04150011545311013,
+          -0.0073396962917864515,
+          0.011192225907726928,
+          0.027746213794734322,
+          -0.035970017302005566,
+          0.02708126943501373,
+          0.002105633542220187,
+          0.014262674845755191,
+          -0.01626664983003942,
+          -0.005631805361903626,
+          -0.013990539604372344,
+          -0.035877093361375685,
+          0.016487723013906077,
+          -0.008607758248055054,
+          0.004714436224135091,
+          0.04458009880881818,
+          -0.0036872068506712727,
+          -0.009466125348499464,
+          0.007600705817195143,
+          -0.0034590883553048063,
+          -0.010351576085287273,
+          -0.0028564505793999295,
+          0.001959661298507,
+          -0.007080697863126256,
+          -0.04183476693971616,
+          -0.00499888976530217,
+          0.04684391782633148,
+          0.006361744381879857,
+          0.013276793765092465,
+          0.013516220521120684,
+          -0.018335746557932003,
+          -0.045577810515431215,
+          -0.0071616111758423025,
+          -0.0032763880176386808,
+          -0.022708028392758567,
+          -0.0240284945459861,
+          -0.008735040856657758,
+          0.005844145402902923,
+          0.006359368165686724,
+          -0.01120763599008222,
+          -0.022676131361171057,
+          -0.027085968700452136,
+          -0.010526103176253857,
+          -0.02123944716136792,
+          -0.010792145796733375,
+          -0.027142270190771887,
+          -0.028742234538796804,
+          0.02880426953386151,
+          0.007575182232859564,
+          -0.004084088301176117,
+          -0.010005765476121966,
+          -0.01255640546813973,
+          -0.0004774761617900028,
+          -0.0025307994917461232,
+          -0.005822065709999144,
+          0.0038614286313282164,
+          -0.011390482620811868,
+          0.023152086963976554,
+          0.0021220701260546895,
+          0.018329162172128138,
+          -0.05772324571231113,
+          -0.002709083851090071,
+          0.021497896395795905,
+          0.0033025410488410913,
+          -0.005412445141910314,
+          0.029352650459792132,
+          0.011223947499453582,
+          -0.006539460282439711,
+          -0.031355428613718234,
+          0.020942976176820262,
+          0.024122993515785947,
+          0.00817615634503138,
+          -0.012994160510030828,
+          0.01221095914384204,
+          0.01629778384790159,
+          -0.004494985018246821,
+          -0.011161098655072933,
+          0.005586857967039181,
+          0.020941039642372112,
+          0.013580622629911925,
+          0.0013398071867077221,
+          0.030905022925366163,
+          0.0006862084391508121,
+          -0.03521231612862609,
+          -0.01908191995763771,
+          -0.01275995164030872,
+          -0.003584331305986364,
+          -0.015264425022763126,
+          0.005294329654809353,
+          -0.002355465010876059,
+          0.006335052149475905,
+          0.022655379849785256,
+          0.03593231939948758,
+          0.0146937743203089,
+          0.00937330328448886,
+          0.003167462724350819,
+          -0.0013235176101076284,
+          0.012102872918129403,
+          0.008638011773075119,
+          0.0350701240268404,
+          -0.014653221697693864,
+          -0.011003994230902835,
+          -0.0037767288519369537,
+          0.010100131061186566,
+          -0.004113527518662366,
+          0.020414393256136372,
+          0.03274709615649469,
+          -0.0037222949335916876,
+          -0.005604552180246868,
+          -0.006160118231975786,
+          0.0041467454414452565,
+          0.01807879127455274,
+          -0.01252134965321004,
+          0.004364858987501461,
+          0.005549300665159658,
+          -0.0071897162833861,
+          0.018567059255813558,
+          0.0002622074720860951,
+          -0.012983550838417323,
+          0.03266376564402191,
+          0.11449509387240629,
+          0.0017251916835047628,
+          0.007125215166916139,
+          -0.040229997815548584,
+          -0.020537987527148888,
+          -0.00245728735683473,
+          0.005779972828836593,
+          0.007004732428236457,
+          0.02225914982152608,
+          0.006505827715591914,
+          0.0006708045351142873,
+          0.031708522931495536,
+          -0.017334083511709092,
+          -0.002522861743921599,
+          -0.01745656559688129,
+          0.013890451996979048,
+          -0.014312605700272691,
+          -0.013784963572164409,
+          -0.01572417853955075,
+          0.004507425871748977,
+          -0.000601158156253668,
+          0.0016158265715634729,
+          0.01791303234734911,
+          0.0011926917387973374,
+          -0.026393351252156246,
+          0.0006694008039871878,
+          0.001026031515214626,
+          -0.004514419527575117,
+          -0.011320929719992686,
+          -0.02223001723942815,
+          0.016235841442653918,
+          0.014188311619681008,
+          -0.004895218832118094,
+          0.009244697376301004,
+          0.010558796153959677,
+          0.02707485418319703,
+          -0.028380342401606758,
+          0.0040204430418629405,
+          0.007023041225985294,
+          0.01606556581447933,
+          0.017617862031190843,
+          0.0034870847111552862,
+          -0.011253315705053224,
+          0.03238289399766116,
+          -0.010563282687352038,
+          0.0065093536270233385,
+          0.014966802208644462,
+          -0.03541611540945151,
+          -5.821682910348165e-05,
+          0.0061985444938402745,
+          -0.011716201661363997,
+          0.009580035961315467,
+          -0.03120692534097433,
+          -0.0039291243987978845,
+          0.009317468331974374,
+          0.017611312558635673,
+          0.008804996982574765,
+          -0.0034129169537151925,
+          0.010822226698822738,
+          0.012510661458054741,
+          -0.0027147975086409966,
+          -0.008855213644027594,
+          0.01238502686635956,
+          -0.0007687053446954701,
+          0.00450108863711973,
+          -0.00799170686059026,
+          0.0029755595812552293,
+          0.002290650917244691,
+          -0.01554856300008412,
+          -0.011779430686394945,
+          0.00438393126679289,
+          0.0036931891075561957,
+          0.001535662168147833,
+          0.012014408636737167,
+          -0.006087255045485002,
+          -0.0077816220325470226,
+          0.0040573724156794284,
+          0.003136943055301706,
+          -5.505014316363865e-05,
+          -0.030625245024561508,
+          0.01585923084309149,
+          0.0067595435845396405,
+          0.010013308598626031,
+          -0.005315626260228517,
+          -0.00903142414091241,
+          -0.007240920015018893,
+          -0.01228521172101564,
+          -0.010657894223173453,
+          0.007658605592936815,
+          -0.00724924993432376,
+          0.011124174211132038,
+          -0.03128027787357322,
+          -0.013988715435985316,
+          0.01231792993245938,
+          -0.01528161242159377,
+          0.0007675821323979293,
+          -0.008389914312201402,
+          0.0009417242487395995,
+          -0.005119304379761705,
+          0.007887436211143328,
+          -0.03731503169547854,
+          0.007878001990271541,
+          0.02456170736765724,
+          -0.00330924625934981,
+          -0.020259771366167557,
+          -0.009613930457931786,
+          0.0241314385136084,
+          -0.008267831641886344,
+          -0.01645888287387047,
+          0.008421304862524506,
+          0.010140813149093993,
+          -0.0006309118552875393,
+          0.0008237784055549769,
+          -0.003726984191590917,
+          0.01669125388798323,
+          0.02054777616420997,
+          0.041930770964107515,
+          -0.014789610432004559,
+          0.01092349703703152,
+          0.02866938974792608,
+          -0.03378576980722847,
+          0.014728113846952417,
+          0.02518409645155824,
+          0.014590531327650454,
+          -0.0004446956315190408,
+          0.001609777186372437,
+          0.0068939321280085475,
+          -0.014273492115311677,
+          0.009247676157551494,
+          0.013380104440744367,
+          -0.018407746260686045,
+          0.020444430509441124,
+          0.03140071554008003,
+          0.0025839998101985736,
+          0.01355205268903058,
+          -0.006268939830271642,
+          -0.013860692186849113,
+          0.014717364068116991,
+          0.031429975488060896,
+          0.0006919565473572909,
+          -0.018561500446336806,
+          0.019128244642842268,
+          0.006988095806264863,
+          -0.003678529269242034,
+          -0.0021993078588142664,
+          -0.021334919445200388,
+          -0.021003586791369228,
+          0.017304350704792804,
+          0.013508779408333964,
+          0.027794485858413365,
+          -0.010594702782952029,
+          -0.007818089330662161,
+          0.014293688127901867,
+          -0.022368568887004094,
+          0.015426482139881204,
+          -0.02160820059598751,
+          -0.00571499813703566,
+          -0.0077402210272634235,
+          0.03274360648629609,
+          0.001441304806200403,
+          0.02734331677589297,
+          0.016457027764064002,
+          -0.033609420134045206,
+          -0.029790535553657092,
+          -0.00712106584238484,
+          -0.02307933297713104,
+          0.010356561171963577,
+          -0.03901183838032092,
+          -0.00071735150189382,
+          0.008251646656432826,
+          -0.008079917288471352,
+          -0.0007087526632966007,
+          -0.006419191399519291,
+          0.013989344185652506,
+          0.013661633160336152,
+          -0.0228103143826663,
+          0.023644639659577426,
+          0.008343481940133684,
+          0.025998483906975782,
+          -0.02076055376006461,
+          0.019148568238437927,
+          -0.017705363646416224,
+          -0.02777313933338929,
+          0.014358130220942483,
+          0.023585731219788467,
+          0.01635364038245971,
+          -0.007122853965024528,
+          0.007110003583832299,
+          -0.03556761320536796,
+          0.027585457849964405,
+          -0.013064062623866305,
+          -0.012504161289952458,
+          -0.0032379276562094555,
+          -0.011931827532360686,
+          -0.002574161005800279,
+          0.0008533907465694845,
+          0.0034345232246102806,
+          -0.03131462673429805,
+          -0.0002450959360520896,
+          -0.03727416416766574,
+          -0.028944814604815072,
+          0.01360126708805471,
+          -0.00036361284498363273,
+          -0.0049097813612814616,
+          0.02933242705521636,
+          -0.026339136251046315,
+          -0.02693295650849956,
+          -0.009590247544093352,
+          0.02853096456887074,
+          -0.0220673798697511,
+          -0.004519050393279848,
+          0.0015034014871316069,
+          -0.01828226786700525,
+          0.004436383319946364,
+          -0.0012748481042235446,
+          -0.010416065158876917,
+          -0.01105146139270597,
+          -0.03229479099927737,
+          -0.005822133182096844,
+          0.02106215163620738,
+          -0.006957125617688484,
+          0.002746241228040847,
+          0.013312248331456934,
+          -0.010968629118224582,
+          -0.0029946900704643812,
+          0.0017875284441794235,
+          0.023304175718480834,
+          -0.021033941064173063,
+          0.007714334541806199,
+          0.005955442031111047,
+          0.0016405586860131207,
+          0.018406562522853148,
+          -0.0054922197063461825,
+          0.0037690571959124617,
+          0.0162126534419016,
+          -0.009969304596173302,
+          0.017959307638726878,
+          0.007691549802475525,
+          0.006681837561745192,
+          0.002775700618222622,
+          0.007712770283579111,
+          0.009752652973700739,
+          -0.013506020126078112,
+          0.016911141425375933,
+          0.022468715726349706,
+          0.00023233555209372766,
+          0.021066580799163234,
+          -0.02302516292580905,
+          -0.028618544628453015,
+          0.018297878810036087,
+          -0.0225078941299981,
+          -0.012735208680311652,
+          0.016459925496937065,
+          -0.0043032724578332855,
+          -0.00797609922711709,
+          -0.014986057073351055,
+          0.02260618561608301,
+          0.02132930921464508,
+          -0.020249788476562527,
+          -0.012385936327068884,
+          0.0072304431443959505,
+          -0.004373475447051833,
+          0.007086656081865733,
+          0.03195810137269375,
+          -0.015982210897392764,
+          0.006818890093901413,
+          -0.0038306138017521626,
+          0.0030753642449792215,
+          0.017830546304667783,
+          0.009470018737705222,
+          -0.0019815241970236146,
+          0.030100238416495066,
+          -0.001330614013444443,
+          0.016392226282459177,
+          0.0004888028477871524,
+          -0.010948833692356176,
+          0.006343890495626752,
+          -0.026652806306173643,
+          -0.008627570055007363,
+          -0.001203530223345183,
+          -0.009581494451831351,
+          0.07598616930784373,
+          0.003930153552069478,
+          -0.01295957554290434,
+          -0.001673007115291578,
+          -0.001368287102181597,
+          -0.006897179139643672,
+          -0.00960679451603541,
+          0.0287166325268306,
+          -0.016613613331623346,
+          -0.01894527592131826,
+          0.0008370014132002703,
+          -0.006785618888421608,
+          0.00919924251249592,
+          0.01816363936999796,
+          -0.010225961928154075,
+          -0.01604759761423727,
+          0.001577138242530474,
+          -0.003520564936698765,
+          -0.004395434431595494,
+          0.0052399043934902014,
+          -0.010127122101423486,
+          0.006456662777583096,
+          0.026560491731327886,
+          -0.013684536319659296,
+          -0.01425196172663986,
+          0.004084982328707292,
+          -0.0010682084059972982,
+          0.009758641913073978,
+          -0.00864879062601467,
+          -0.00481222050200678,
+          -0.014375330954464725,
+          -0.011378890549403524,
+          0.02505641780042516,
+          -0.02861164504722546,
+          0.0034060847953066354,
+          0.014937227531980433,
+          -0.00016904224334342915,
+          0.01541381851648916,
+          0.0033028855218525644,
+          -0.03615935765718852,
+          -0.013677774380513626,
+          -0.02073839691250734,
+          0.019821074228011413,
+          0.01688422012576042,
+          -0.0033731840788147442,
+          -0.013840993586441808,
+          -0.026149880178874887,
+          -0.007430563876740068,
+          -0.02466234818567362,
+          -0.020708282040996014,
+          -0.019276991448222817,
+          0.006137002799644838,
+          0.005162892426268572,
+          0.022881759407670583,
+          0.0025854880108157857,
+          -0.015058889448404231,
+          0.0029948608544278386,
+          -0.006722836436935071,
+          0.011845298212977357,
+          -0.02747944486818378,
+          0.015311272959213262,
+          -0.01610493464983465,
+          0.025239768880017402,
+          0.007425208902253236,
+          0.0010569008485112994,
+          -0.008822681490988575,
+          0.00010704167529046774,
+          0.014883787023445807,
+          -0.004186968199973372,
+          -0.0007781259002896265,
+          0.02787555889813697,
+          0.02087373813833492,
+          -0.0017783425135865238,
+          0.008743718448136432,
+          0.028026471206552342,
+          0.019081083283793984,
+          0.0325261942213039,
+          -0.013815526479760766,
+          0.005113676604826083,
+          -0.028873600114081546,
+          0.010510475843707614,
+          -0.010871039729515013,
+          -0.010201824683340651,
+          -0.0067946599549165735,
+          0.027756887504389347,
+          -0.007962601205446472,
+          -0.0006687976124826922,
+          0.03469894503563926,
+          -0.014185294847328063,
+          -0.00010543064621374546,
+          0.0007386418834840544,
+          -0.009759481309152999,
+          0.02742722730386853,
+          -0.0073590367291510046,
+          0.018456032721630843,
+          -0.019560882612141943,
+          -0.013705917411254969,
+          -0.011894560208199602,
+          -0.001142768182039817,
+          -0.02576754877141608,
+          -0.008755240066826174,
+          -0.02249287216087102,
+          0.00859172262856532,
+          -0.0046012388553094375,
+          0.018845920253446127,
+          0.0011617844393195566,
+          -0.005275799428018491,
+          0.029069367036565145,
+          0.0088080893110627,
+          0.018100411389202663,
+          -0.0004971948011851863,
+          0.01216566778850985,
+          0.02369448391321832,
+          0.002637001729688794,
+          0.005012948614448268,
+          -0.024213704079796478,
+          0.003420184574622472,
+          -0.0004359900462023086,
+          8.265488264269535e-05,
+          0.016267948861008616,
+          -0.019572032453100303,
+          -0.02216264534756073,
+          0.021871880626992544,
+          -0.012742926070754964,
+          0.011676769226089007,
+          0.0035362870996336194,
+          -0.010732229460038356,
+          0.0006838777488770268,
+          -0.03083438636386314,
+          0.008820746379709355,
+          -0.03216685858285627,
+          -0.032230808828686675,
+          -0.013547681710236494,
+          -0.009994106571421729,
+          0.0008758088338659712,
+          0.028447187234357855,
+          -0.00795317029726568,
+          -0.015797892954136448,
+          0.005363565557248085,
+          -0.011199252697624111,
+          -0.00817162221789202,
+          -0.0007867892785855576,
+          0.005856939999700809,
+          0.009461959355310273,
+          0.005997804620690314,
+          0.011648335019189696,
+          0.007491946658046298,
+          -0.008028800679169293,
+          -0.008837725220736191,
+          0.015889923814605387,
+          -0.0014409716087425222,
+          -0.01176424391171685,
+          0.030430130082762363,
+          -0.013300620185884817,
+          -0.0005592926632214984,
+          -0.0021975669766032472,
+          -0.0015687208316663597,
+          -0.0021652076181411244,
+          -0.014225289143358926,
+          0.01870568404189425,
+          8.334501002782455e-05,
+          -0.014288806218067678,
+          0.0032841743905975634,
+          0.003841780428153241,
+          -0.0036498339483411074,
+          0.008388922960269383,
+          -0.008147275785908849,
+          0.017348147498892,
+          -0.015113395971380741,
+          0.008740452410210708,
+          0.0038335161236289234,
+          0.0043482078469644385,
+          -0.0065755348640655865,
+          -0.01592809184384243,
+          0.009642593996701513,
+          0.012051872087663658,
+          0.02173264012652684,
+          -0.02525660485859878,
+          0.00876161653901566,
+          -0.005374533920781607,
+          -0.014534179652013885,
+          -0.016127866637369738,
+          0.00982303248417706,
+          0.022322910435388785,
+          0.0063086346898091296,
+          0.012139263756190026,
+          0.00013383660542132047,
+          0.005848259030010402,
+          -0.014767198851314148,
+          0.010570158368602513,
+          -0.011355774405066537,
+          -0.018439261453084426,
+          -0.005243587266408389,
+          0.028894120979627323,
+          -0.005492024421080143,
+          0.017938747701972086,
+          0.0021350711250093858,
+          0.029007356490801176,
+          -0.008686781541806358,
+          0.0014883465513564527,
+          0.0039866022469891155,
+          -0.01295271525889559,
+          -0.004272008629260297,
+          0.01976273843646234,
+          0.04882868638601926,
+          0.010204475902027572,
+          -0.0009155109545661691,
+          0.010069071220988797,
+          0.005626418787776309,
+          0.002148101019776153,
+          -0.00805781810841286,
+          -0.030431071067211577,
+          0.008883707378543252,
+          0.005628218215212441,
+          -0.01997825125961566,
+          0.019838234890736068,
+          0.026483365072033395,
+          0.023040623182975964,
+          -0.011585643598171132,
+          0.016539930468533123,
+          -0.010092529089936188,
+          0.0177008735686489,
+          0.028665424114962996,
+          0.013810061902648515,
+          -0.010487150104822674,
+          -0.0020847870330548534,
+          0.006329342599564054,
+          0.004186981876004016,
+          0.00026239408223817384,
+          0.015190614519766204,
+          -0.019155180136775043,
+          0.01005728016778505,
+          0.014725826697964637,
+          0.03322436221319799,
+          8.049930133627564e-05,
+          -0.01645191835557271,
+          0.0016249281996128196,
+          -0.009976155662939693,
+          0.00856399492362911,
+          0.000632427127231252,
+          0.0032242260692672124,
+          -0.009688119824276678,
+          -0.012682779503915694,
+          -0.002851918183031152,
+          0.009403205508474241,
+          0.0021188716361101698,
+          0.002759004011231831,
+          0.007641925150177999,
+          0.0022830062031146204,
+          0.01713024474384216,
+          -0.0011181980841546123,
+          -0.00029141889484139217,
+          0.004601090256674874,
+          0.010509199590505558,
+          -0.01649852835819883,
+          -0.017548369964263514,
+          0.004337237475053179,
+          0.012217494210546133,
+          -0.013159179948224596,
+          -0.013241981632059755,
+          0.015704543608959117,
+          -0.01913575130892795,
+          0.024934102868065366,
+          -0.00942381892138331,
+          -0.004543611336526587,
+          0.000343200128934031,
+          -0.00727538880611703,
+          -0.0038751118497522673,
+          -0.0030064279504724797,
+          0.003984715814505561,
+          0.0019261818110367147,
+          0.007472350699291607,
+          0.008472369788801792,
+          0.014568547873264247,
+          0.006043895352236181,
+          0.0032580830408778395,
+          0.009109104498825864,
+          -0.014523846294121576,
+          -0.014800649887278973,
+          -0.002704791598011488,
+          -0.008895394038700809,
+          0.00902016867266151,
+          -0.006607182154874112,
+          0.00845408382182461,
+          -0.015501867835442603,
+          0.015573642225745032,
+          -0.012200552219016517,
+          -0.0008900215712478553,
+          -0.009666233545144905,
+          -2.7470277092642562e-05,
+          0.008191697904514857,
+          0.014605287653054787,
+          0.0070552942843382215,
+          -0.031921405225964206,
+          0.0017403427534633655,
+          -0.008846568352320488,
+          0.03446319360336241,
+          0.002612606662691094,
+          -0.01060863123202008,
+          0.020352181356568008,
+          -0.01592124132155158,
+          -0.00027704668715660693,
+          -0.01602075523652062,
+          -0.004797557413047212,
+          0.00707939446796679,
+          0.016399017978990946,
+          0.02759049512664933,
+          -0.005697020356865688,
+          -0.007995305872076221,
+          0.006384047872862793,
+          0.029590696670334232,
+          0.012246955508890425,
+          0.024748351124415785,
+          -0.007633712004672087,
+          0.014348228716100414,
+          0.007773260552660029,
+          0.0076045488653348975,
+          0.004321689250440021,
+          -0.002591801797145718,
+          -0.011698574606499644,
+          -0.014974971814360001,
+          -0.0037563183330623384,
+          -0.016987915346053498,
+          0.00012247833530362867,
+          0.021344400276501117,
+          -0.00475351423959752,
+          -0.0033847161875730026,
+          -0.01874186437694691,
+          0.0033461670218611346,
+          -0.010029445302549983,
+          -0.00367421246775698,
+          -0.013133469163793961,
+          -0.0254122035745965,
+          -0.018556369080030412,
+          -0.002155244024999317,
+          -0.00929309186646207,
+          -0.015194269781263309,
+          0.0070354690219415535,
+          -0.016305117309271634,
+          0.0080837166473439,
+          -0.00448473417024141,
+          -0.016508775226236883,
+          -0.00958925666995396,
+          0.01100335153658195,
+          -0.0041819993634961685,
+          -0.01929493648733792,
+          -0.01282300765645703,
+          -0.011300260564838231,
+          0.0051733704121938884,
+          0.005996780699975213,
+          -0.0038111648304178544,
+          -0.0028279639348479987,
+          0.016640121845366295,
+          -0.010197365811240557,
+          0.009878730312873492,
+          -0.03335474019383245,
+          -0.03551826040406959,
+          0.004497847402931096,
+          0.01507007013703286,
+          0.0013085881889541252,
+          -0.02078680101065954,
+          0.010806315988966003,
+          -0.016499186313125556,
+          -0.00858013792516617,
+          -0.008522534127498645,
+          0.009649521088411411,
+          -0.007283003380127847,
+          0.01299542385123007,
+          -5.3226845309105124e-05,
+          0.030477683597373687,
+          0.0074153455615142755,
+          0.004161881195438522,
+          -0.00679081048132963,
+          0.008354912600159931,
+          -0.001621776416108105,
+          0.004080191054849395,
+          0.02120524874226696,
+          0.012440848502230418,
+          -0.00218041171518331,
+          0.006458415098595872,
+          0.005678055538564355,
+          -0.017186537377121235,
+          0.02507960402044456,
+          0.024678178106806115,
+          -0.008205288425409454,
+          0.004108402466376321,
+          0.005783335302169668,
+          -0.008846663924040241,
+          -0.006897707231869481,
+          0.006527795262026734,
+          -0.018474181903007085,
+          -0.0008721381678008483,
+          0.010598084358464908,
+          -0.02255441909004843,
+          -0.004840511019512733,
+          0.021745651362109403,
+          0.004631567631335321,
+          -0.018558972627958834,
+          0.02764788406651861,
+          -0.004942361238163609,
+          -0.0027290999934532494,
+          -0.010876135202650048,
+          0.01671497143541799,
+          -0.012173527207375091,
+          0.008443317346655462,
+          0.011021541861818043,
+          -0.006295639322287843,
+          0.003083685676540999,
+          0.008277206401503182,
+          -0.004504302913715969,
+          -0.0068075424927125676,
+          -0.0010508416962215848,
+          0.010167160518175114,
+          0.0036980815941666336,
+          -0.022704963865525812,
+          -0.006020073261002279,
+          0.02934227992246887,
+          -0.015082922782242507,
+          -0.0050916398557337705,
+          0.005376294716056329,
+          0.007709342013963906,
+          -0.003476236561444464,
+          -0.0023399991233097337,
+          -0.00787486645740744,
+          0.010356414979428607,
+          -0.015950083845400274,
+          0.02076360898078935,
+          0.025608263494550438,
+          0.024504236990840842,
+          -0.014814235990238182,
+          0.010828584133720797,
+          -0.011703992394890898,
+          -0.011047470905668532,
+          -0.015542835222173864,
+          -0.001658296303013357,
+          0.001970208852236451,
+          0.013107754130616248,
+          0.009666887546609378,
+          0.0235853683657408,
+          0.006473544089109194,
+          0.008973300312205858,
+          0.014118601565410305,
+          -0.003902202854013708,
+          -0.018303438244719996,
+          -0.014584226585460755,
+          -0.011977613285028946,
+          -0.01756345491020528,
+          0.0004563602312336091,
+          0.020461659996271687,
+          0.012417744207683839,
+          0.024136626592375227,
+          0.002119213701333967,
+          0.013759902247467922,
+          -2.9525392347047554e-05,
+          -0.000892319792935443,
+          0.00045783825646242434,
+          -0.005238483120312843,
+          0.019255756091120846,
+          -0.004599752813817219,
+          0.01656589522110913,
+          0.0006443773300684925,
+          0.007133940816771615,
+          0.013554761340256566,
+          -0.01704388669741371,
+          -0.003443100892108368,
+          -0.010029620309829285,
+          0.000941391966877521,
+          -0.019633111750884034,
+          0.01934571074127927,
+          0.010273330977233852,
+          0.0017121345993533963,
+          -0.018327189396014198,
+          -0.0029706385099949693,
+          0.03448635131998772,
+          0.019458999453011265,
+          0.0050734354363614525,
+          0.02632443373898792,
+          0.013018762352749236,
+          0.014443611411502168,
+          0.010153370950346075,
+          0.02356766253713842,
+          -0.023479811300626968,
+          0.02095255383200973,
+          -0.011590347850340786,
+          -0.007414136601134069,
+          -0.0035774888741646705,
+          0.0025141508183687115,
+          -0.013588691733097813,
+          0.00982877622088977,
+          -0.0017916539399655826,
+          -0.025386211797192578,
+          0.009164316443206016,
+          -0.008811052979979271,
+          0.010464442616928296,
+          -0.007774569252594923,
+          -0.0002806136360286729,
+          -0.0127084026190163,
+          -0.015184317235990752,
+          0.001259201880504347,
+          -0.009158429132281081,
+          0.023908150915412617,
+          -0.021210848953875233,
+          -0.01321384014436847,
+          0.013997502713857512,
+          -0.006869443694669701,
+          -0.011364239609894453,
+          -0.00374449980963666,
+          0.023673050288879224,
+          -0.009243341446512421,
+          -0.01359367519009039,
+          0.014464166953530989,
+          0.003834735113400469,
+          0.0059713317014342395,
+          -0.007508862694272452,
+          -0.008428705474135382,
+          -0.003280788036976103,
+          0.005998840043285335,
+          0.026615077617284676,
+          -0.014059930972065399,
+          -0.0032833264512892022,
+          -0.009189999799498163,
+          0.004118204537632102,
+          -0.02359889635281631,
+          0.02771075351384059,
+          -0.008871395303200375,
+          0.021705177758560575,
+          -0.0015924449308442096,
+          -0.006239188158271628,
+          -0.024231036074349822,
+          0.027522945599928003,
+          0.003683870966132199,
+          -0.010221180899173243,
+          0.0012768725460662862,
+          -0.052005292736885915,
+          0.016636109796947025,
+          0.008429895913163985,
+          -0.019507994429834353,
+          0.003697519981610208,
+          -0.004773584856005853,
+          0.018983847580558853,
+          -0.022966993228850697,
+          -0.014078986007109447,
+          0.008283766047449966,
+          -0.01153285124362746,
+          0.0031439180465649996,
+          -0.02527749501821793,
+          -0.005754537200570427,
+          0.0011760345167229269,
+          -0.005276936074280004,
+          -0.001705933750221904,
+          -0.020022761608132548,
+          -0.025972227850795115,
+          0.017666017379157135,
+          0.015820875579600418,
+          -0.002278262448951789,
+          0.01320818011561616,
+          0.008709899932124187,
+          0.00836573261091119,
+          0.016034263394940008,
+          0.007124509234103267,
+          0.0081473711608784,
+          0.045498533239850446,
+          -0.019846323044145485,
+          0.007371629414298193,
+          -0.0014854149174929534,
+          0.013951741898629716,
+          -0.030467924817555014,
+          -0.0055144480998344705,
+          0.0062802744565489945,
+          -0.0049373689340857915,
+          0.03510454786147091,
+          -0.017326094216003073,
+          0.01694160526735264,
+          -0.0026075809870445238,
+          0.0007420394265014909,
+          -0.02042003877666893,
+          -0.026156937726134134,
+          -0.014294938857971834,
+          0.0011857406889443987,
+          -0.0010047941261608975,
+          -0.011946583993069242,
+          0.006797001710574735,
+          0.004917161312925866,
+          -0.0020713379451245515,
+          0.015652523103715316,
+          -0.02952061566240878,
+          0.007356800041369564,
+          -0.010733578620243498,
+          0.020008504830359524,
+          -0.00990331048284697,
+          -0.0013319400098308093,
+          0.007236960885181047,
+          -0.004122868383988748,
+          0.013216043472071927,
+          -0.0038337251451538677,
+          0.004227115653354611,
+          -0.005221321656766545,
+          -0.01794343569722877,
+          -0.0016435261495179767,
+          0.021429008432348098,
+          -0.008349684096540855,
+          0.031864497493526205,
+          -0.01264743878037828,
+          0.009174982784686633,
+          0.010692990337462762,
+          0.0006628831036334152,
+          -0.005286285851702483,
+          0.009052971814211658,
+          0.0021431970274545893,
+          -0.008690520448278274,
+          0.006934310986093342,
+          -0.018349449937310143,
+          0.008494451785507706,
+          0.03394632739506833,
+          0.0018511328780472266,
+          -0.0007519661670070526,
+          -0.007266663179613392,
+          -0.01335844201823745,
+          0.018540713029383066,
+          0.005488273906733518,
+          -0.002546531152114063,
+          -0.027536660869590543,
+          0.02007746621157705,
+          0.0012265798861291223,
+          -0.004755325069578531,
+          0.012466557901411086,
+          0.0020495867632654885,
+          0.019989531151046736,
+          -0.010171678639113316,
+          0.030757957899447096,
+          0.010050036220124393,
+          0.002959833051227505,
+          0.0038556928367700113,
+          -0.036655749074851536,
+          -0.01652152778069314,
+          -0.00019357502472740266,
+          0.014271581054865031,
+          -0.012430647252826173,
+          -0.022432580814844155,
+          0.0010346169306977334,
+          0.01148004051449691,
+          -0.0033898642762881046,
+          -0.009650665305271706,
+          0.01560757541276278,
+          -0.020342213902670575,
+          0.006701302394408393,
+          0.007693672045981921,
+          0.0287100214318104,
+          -0.0020735384554022153,
+          -0.0012324838849990056,
+          0.000626416803904792,
+          -0.0055669860801192935,
+          -0.0008037366126870824,
+          0.017251369734470945,
+          -0.013661316947002466,
+          0.0175926211217534,
+          -0.012735241490550407,
+          0.0064713656749838585,
+          -0.0021092378432037563,
+          -0.010629579860050812,
+          -0.020114807971202787,
+          0.0029609495764307357,
+          0.009378003163375899,
+          0.006920144698428806,
+          -0.010438984881449953,
+          0.010715409812532968,
+          0.012197673689706352,
+          -0.00753962861484439,
+          0.014519905132928041,
+          0.022173989895384192,
+          -0.007687503607783572,
+          0.013633321414009471,
+          -0.0041446334728745095,
+          -0.014056264816568229,
+          -0.009985463824440833,
+          0.023929750617160402,
+          -0.0005563806942667597,
+          0.008756997373618809,
+          0.00388125051641897,
+          0.0007856447663040696,
+          -0.011226530517316183,
+          -0.001653447552601681,
+          0.019294448360333633,
+          0.011831934675073843,
+          -0.0013888534822008192,
+          -0.0010525764356699885,
+          -0.0014566936525653407,
+          -0.009427557739483988,
+          0.02049188741459728,
+          -0.022030783927622997,
+          -0.0019604940727076573,
+          0.00461018148439991,
+          -0.022589816803875618,
+          0.0001863117949313752,
+          -0.024570217962989905,
+          -0.01569952241187498,
+          0.011763033781122343,
+          -0.0022530631823737554,
+          -0.016028810867854907,
+          -0.0003242701040942633,
+          7.424127301806556e-05,
+          -0.007815789940508527,
+          -0.004135500768181258,
+          0.006115909992703774,
+          -0.00034193217510010117,
+          -0.0032262189374874842,
+          0.013430154310506884,
+          -0.009996622196870008,
+          0.015431528911575157,
+          0.0067029980830614145,
+          -0.015702167721800483,
+          0.0021406610839557003,
+          0.018500219544264753,
+          -0.0058910087864916085,
+          0.00787021425357441,
+          -0.014681907249972176,
+          -0.0037495551894426303,
+          -0.005787661351944086,
+          0.006051351977821761,
+          0.024849195780885833,
+          0.012969434450643372,
+          0.01139340445077682,
+          0.017405361843075933,
+          0.004325000715054734,
+          -0.01572297405297993,
+          0.0007416798945675923,
+          0.01682727189828363,
+          0.010464036837245312,
+          0.009804326468113703,
+          0.0008975556667684365,
+          0.0043704860361487635,
+          -0.005142176468800175,
+          0.009317064072855551,
+          0.02915482449687158,
+          0.0013434689612997423,
+          0.008696141010760192,
+          0.017276730345147134,
+          0.027038741396131265,
+          -0.0035088997194497725,
+          -0.007469614319788499,
+          0.008739942310300076,
+          -0.009657991604173173,
+          -0.001081629174976152,
+          -0.017000351183542026,
+          -0.012310275736750868,
+          0.016510505259358398,
+          -0.006084627062230492,
+          0.02848913486719956,
+          -0.0010655806791907225,
+          -0.02026498848361482,
+          -0.01565505608194376,
+          0.001404549479440637,
+          -0.01362132895080652,
+          0.005287566256051299,
+          0.014187265685427415,
+          0.01917046513020831,
+          0.004687191682875024,
+          -0.022258337721085656,
+          0.037364962197399806,
+          0.01757108963349427,
+          0.019377870795760123,
+          -0.019422840506328382,
+          -0.006159311237412833,
+          -0.009077095160676276,
+          0.02116697012944498,
+          -0.00880692095488465,
+          -0.009875674856043954,
+          0.024014332004451508,
+          0.017812606845656484,
+          0.005679806286464216,
+          0.006736609904079586,
+          0.0020048663439319585,
+          0.0025414630253661444,
+          -0.015831593851012047,
+          0.017957902498542212,
+          -0.005858125698347523,
+          -0.0058068554681761945,
+          0.017529514573571298,
+          -0.005989581792993925,
+          -0.0030306021203732347,
+          0.00015839997806822045,
+          -0.019697052145531482,
+          0.011618112886207201,
+          0.008256007601576028,
+          -0.017229774429089857,
+          -0.0014757560402567304,
+          -0.0002404480326945295,
+          -0.014118694740134205,
+          0.02004885146995733,
+          0.01694893356262835,
+          -0.0013248884139405398,
+          -0.0051752815372013315,
+          0.008223868147393114,
+          0.007000679132597393,
+          0.005256194890343006,
+          -0.019496098876977206,
+          0.005230597149342019,
+          0.005103909294824105,
+          -0.0014716649591412774,
+          0.010274346323051634,
+          -0.009030594825985526,
+          0.033740380586893826,
+          -0.006505788197604522,
+          0.013065658941136721,
+          -0.0043634731684193745,
+          -0.0055529072007985125,
+          0.001816020676232658,
+          0.013054468478598695,
+          0.009753501030700457,
+          -0.00970446155762939,
+          -0.004172354962129865,
+          0.018697982092775737,
+          -0.011983156146516074,
+          0.0013300791598046338,
+          0.01564072865804035,
+          -0.004858814041438713,
+          0.004708235083775013,
+          0.008635204790345423,
+          -0.004489505237783172,
+          0.01801512858970922,
+          -0.016482572999490366,
+          0.0023953436660054026,
+          0.010694869684962088,
+          0.02390864159830096,
+          -0.019152187046410703,
+          0.021786596486708207,
+          -0.017951050280323393,
+          0.026538582634436314,
+          0.014484894489694262,
+          -0.0035222259457311444,
+          -0.01946200885147006,
+          -0.00850690913394181,
+          0.011654565129063417,
+          -0.021423803876241532,
+          -0.0007985948078290641,
+          -0.0007290222944686444,
+          -0.021367628543092133,
+          -0.011189364017231132,
+          -0.009961779129778331,
+          0.010701637745810138,
+          0.001918633898887013,
+          0.010380983868185359,
+          0.008002182827196827,
+          0.010784760574518087,
+          0.014405640442760822,
+          0.0008958107769384412,
+          -0.010743001445440941,
+          -0.023795589003655273,
+          0.016987708698679597,
+          0.00438463274382093,
+          0.0007862212695139549,
+          -0.01627468994515667,
+          0.011936651269737248,
+          0.013250643815768723,
+          -0.009385605548926286,
+          0.009121882469425302,
+          -0.03716901103204463,
+          -0.004552808410398574,
+          -0.0035704811925430366,
+          -0.0020309197365173475,
+          -0.00908678158053104,
+          0.01559445415882485,
+          -0.013682507836005851,
+          -0.0009590922904844632,
+          0.00011696985057536548,
+          -0.0074756526903548915,
+          -1.5467890432254378e-05,
+          0.020649928837031005,
+          0.008438785939908428,
+          -0.0037382105257999757,
+          0.010376213224654074,
+          0.008826031303751312,
+          -0.013900359449287676,
+          -0.01527474712786084,
+          0.019035366035884273,
+          -0.018776918681908793,
+          0.0063098418544296165,
+          0.01677916431403883,
+          0.020778813096283988,
+          -0.02891862501830075,
+          -0.0007231469961304435,
+          0.022093939332021185,
+          -0.0012807635799950273,
+          0.006370841109790851,
+          0.008781853138377092,
+          0.006754082980602544,
+          0.0029102159062355374,
+          -0.003711173531879644,
+          -4.843770941622283e-06,
+          -0.003940474588075707,
+          0.007155539022862352,
+          0.004366433868349658,
+          0.036453728071874844,
+          -0.013661129348165794,
+          0.002541308805735001,
+          -0.0058838191638835155,
+          -0.00347401304572066,
+          -0.00593469688967835,
+          0.004407665856950772,
+          -0.010055727994798024,
+          0.004330827357536078,
+          -0.001570861535288093,
+          -0.0003731129422346693,
+          0.004883969598300302,
+          -0.001701673379646408,
+          0.007036892564844683,
+          -0.011863650867472625,
+          -0.0003979727750011258,
+          -0.020174851412609312,
+          0.021117284557875267,
+          0.00011666551282792189,
+          0.003662462447326642,
+          0.03723741246290771,
+          0.00476064510662738,
+          -0.00238355131307445,
+          -0.012880057949435798,
+          -0.013551338766321525,
+          -0.012986444917747435,
+          -0.020703821629987552,
+          -0.0036665673548525388,
+          -0.021006311026247164,
+          -0.0052061406234528915,
+          0.010362862594785442,
+          -0.011540184420787572,
+          -0.004023659825622843,
+          -0.015881589631889473,
+          -0.006531981517815174,
+          -0.03114211633687585,
+          -0.01124607488650473,
+          0.007493228366452009,
+          -0.019149285949049886,
+          -0.0039932870926865805,
+          0.026850121590606017,
+          0.001095360635760988,
+          0.01387696424607248,
+          0.0038058248676879625,
+          0.008117686785954085,
+          -0.0045937472017383955,
+          -0.0005669025036141417,
+          -0.01928244877716143,
+          0.003391980330529026,
+          0.008525354854576925,
+          -0.00361507909530282,
+          0.0070008404454106905,
+          -0.008835727374865623,
+          -0.016950017602980382,
+          0.0021654956715219517,
+          0.009334379529385142,
+          -0.006639140003365139,
+          -0.017587472740268683,
+          -0.011049929670847685,
+          0.010401189523318744,
+          -0.02283429470064187,
+          0.006257145380266984,
+          0.016439581169238607,
+          -0.009363030375624585,
+          0.004651575425738153,
+          0.019809183749755425,
+          -0.011810674576781702,
+          -0.005811328501528397,
+          0.008471992070998505,
+          0.013891259999721588,
+          0.0095606134736748,
+          0.02954498207274139,
+          0.0026596809804008996,
+          0.000561870602466584,
+          0.004889263753490895,
+          -0.004277061142504552,
+          -0.017423200644235665,
+          -0.006243787897080996,
+          0.0038507494787521856,
+          0.012873996671952184,
+          -0.009879417960879032,
+          -0.00048575277472139055,
+          0.01682604801673206,
+          0.004530939990187663,
+          -0.0033901248079550284,
+          -0.011868399508738274,
+          -0.020080331108085103,
+          0.0015580259874809898,
+          -0.01362200460317948,
+          0.015354347040143093,
+          0.003390601069331352,
+          0.0015284610287850575,
+          -0.004239880336806429,
+          0.01828645373058079,
+          -0.005013418527935815,
+          0.012749833447770729,
+          -0.006148890994661865,
+          -0.004587831231448011,
+          0.007986533783001116,
+          0.009166578151006778,
+          0.0004939857995863664,
+          0.05421019830078428,
+          0.002838056604877719,
+          0.01333205648597435,
+          0.013419127695838135,
+          -0.014523733369484934,
+          0.014086525147734823,
+          0.012688569853552613,
+          0.018497344547821693,
+          0.013037819487160016,
+          -0.012353776077273462,
+          0.017354801119939964,
+          0.0024084723134891487,
+          0.003909681065813095,
+          0.0022202438064449653,
+          -0.004528576079628071,
+          0.019594683950225115,
+          0.004070222328547378,
+          -0.0074362443729372105,
+          0.004453618452623641,
+          0.005007543168224267,
+          -0.013105524222311019,
+          -0.0112958915514782,
+          0.027105208848863218,
+          -0.01219417557128091,
+          -0.018749971743148304,
+          0.006706594269933445,
+          -0.02008458683853808,
+          -0.03763686535076864,
+          -0.0021332074289244973,
+          0.009992361098878196,
+          0.002795183524495193,
+          -0.00253695604043486,
+          0.01614298064601032,
+          0.016730443047719683,
+          -0.016985296937992726,
+          -0.018245902728910347,
+          0.0025257780765481825,
+          0.0016333325293570182,
+          0.01861807236979066,
+          -0.023168663546030196,
+          -0.006378864850313035,
+          -1.3795465857380962e-05,
+          -0.006829008132963306,
+          0.004850306884033312,
+          0.004703280112191218,
+          -0.0071714759384722965,
+          -0.0027773519096287217,
+          0.002256846645341137,
+          0.01975383140755998,
+          0.013198754584066554,
+          0.01572971337890945,
+          0.002600741959577911,
+          -0.005033508062449482,
+          -0.016817359058724685,
+          -0.005450011991586895,
+          -0.013478330855299777,
+          -0.009091854345715177,
+          -0.003042052092498012,
+          0.008217376329142566,
+          -0.025053469059115735,
+          -0.011965083373220982,
+          0.0119294363235832,
+          0.00391377331459168,
+          0.0003099203772539202,
+          0.007607672728136279,
+          -0.003830132495364688,
+          0.010331053246451333,
+          0.013823890473448383,
+          0.0076112887952959766,
+          -0.004723377809559611,
+          0.009600411526173948,
+          0.011460781436250457,
+          0.006015477068316837,
+          -0.005072711638895421,
+          0.023859899048048694,
+          0.01821920335673539,
+          -0.01680365156728093,
+          -0.002149107383573353,
+          -0.0027829676356423997,
+          -0.01884308738652242,
+          0.009665830801645276,
+          0.00880322398336952,
+          0.008308268605449346,
+          0.003906369351868086,
+          0.00974531700889854,
+          -0.013875904819275925,
+          -0.0050570811397453235,
+          0.002870120865445989,
+          -0.005729556771404781,
+          -0.01901337075686543,
+          0.000297242372701423,
+          -0.011987192261104132,
+          -0.011327157550660895,
+          -0.01409710922165158,
+          -0.003221240464411861,
+          0.005283472511900512,
+          0.021472187640986972,
+          -0.004410881709548286,
+          -0.001855852749039012,
+          0.01351697599517034,
+          0.010060125186506398,
+          0.0021658738823899905,
+          0.0002206474541239167,
+          0.005418828324605768,
+          -0.0027919130944783397,
+          0.0008094042430809068,
+          -0.0015826797905407809,
+          0.021089276765434876,
+          -0.02016611086095107,
+          -0.009413977569213423,
+          -0.009288078098692112,
+          0.005842197094339927,
+          0.028793032462603636,
+          -0.007165634502373454,
+          -0.010104002083017078,
+          -0.01994315293556678,
+          0.0003786527922888834,
+          -0.07810609175343784,
+          -0.005210636756258548,
+          0.001870513618910238,
+          0.03126380113105983,
+          -0.010817888476436834,
+          0.0033673768058629823,
+          -0.002204487854177595,
+          0.014542299655712928,
+          -0.001760107965206874,
+          -0.0013283746982194259,
+          0.0071954806737858056,
+          -0.005640400818010697,
+          -0.006210989097047217,
+          0.005904163661619133,
+          -9.945941028848348e-05,
+          -0.010266938705916303,
+          0.00057263080955709,
+          0.02209498395848068,
+          0.0159398346608872,
+          -0.006575439759804175,
+          0.01816748552021211,
+          -0.00719294553484007,
+          -0.009838738592324234,
+          0.005639288780502127,
+          -0.0019007683459126327,
+          -0.015369619176147573,
+          -0.011475977270803784,
+          0.009862808084641291,
+          -0.030069890644150284,
+          -0.00010725753918890802,
+          0.00943698424646706,
+          -0.013540245857270492,
+          -0.008293042205008774,
+          0.007212974960781723,
+          0.006708343474532583,
+          -0.0012017968151025369,
+          -0.0078026000239507145,
+          -0.02527526731750743,
+          0.0027578400032925625,
+          -0.002446811971857853,
+          -0.004899702349784173,
+          0.0019960465173086183,
+          0.001479503476855694,
+          -0.012645905858466798,
+          0.01216748847452763,
+          -0.013378877368040743,
+          -0.012053251571418766,
+          0.012508842412556306,
+          -0.00027167241745418584,
+          0.01682680930386839,
+          -0.020607299598265426,
+          0.0010702151063620052,
+          -0.028075439824932304,
+          -0.007671848153932823,
+          -0.015486630894165054,
+          -0.0009776136330459065,
+          -0.0012939921569540803,
+          -0.03481880531471923,
+          0.012889447665794033,
+          -0.005200149829198305,
+          -0.006094702500154477,
+          -0.00459778303079732,
+          -0.024583504817899775,
+          0.006072281238752663,
+          0.00607350167867964,
+          -0.006506508970816635,
+          -0.016816439635445445,
+          0.008428048142588516,
+          -0.0011040593013238872,
+          0.004873670535200424,
+          -0.007475800310154908,
+          -0.0005638920132076605,
+          -0.013047135093134946,
+          -0.005463454269764494,
+          -0.0008522562324122834,
+          0.01827123304871433,
+          0.0044055851550317835,
+          -0.014526381222031981,
+          0.016862445236481526,
+          0.013250692156820124,
+          0.01602714293656341,
+          -0.0071089293427753115,
+          -0.0004265739315250755,
+          -0.02750630108787212,
+          -0.00327225092954903,
+          -0.023687644154808972,
+          0.00041213204365507744,
+          0.009447660230507965,
+          0.0203905264889617,
+          0.012694421477479161,
+          0.004620617778996077,
+          -0.001557511764988156,
+          0.0066751090724502,
+          0.02696959560423252,
+          -0.01099183126742224,
+          -0.006969111598423376,
+          -0.01884504192789845,
+          0.01113023717919988,
+          0.025135748541199255,
+          0.003381303139273845,
+          -0.011317957771935042,
+          0.008112598754655146,
+          -0.0009247845526290428,
+          -0.010302980598863016,
+          0.02889587382004902,
+          -0.0391134481322538,
+          -0.015599591108049343,
+          -0.004605824192962023,
+          0.007406520608110251,
+          -0.015596917115435033,
+          0.00770468362805995,
+          -0.012614339656105264,
+          0.001699049329817941,
+          -0.005851477164035241,
+          0.03535087340659861,
+          -0.0031950960727572925,
+          0.0006909856233673865,
+          -0.003395233050301427,
+          0.003531135044131077,
+          0.012355948349937309,
+          -0.004444221317023453,
+          -0.030726300006909638,
+          -0.0022983766869702346,
+          -0.011198252640110788,
+          0.009274666307515955,
+          -0.016207714291658314,
+          -0.017342435976846468,
+          -0.009424737744066259,
+          -0.021827243671837953,
+          -0.012403084982789827,
+          0.029747453318486338,
+          -0.005824001042844527,
+          0.01747569398142269,
+          -0.005219060548639168,
+          -0.008365003057800934,
+          0.00021272411948388247,
+          0.01344335658390732,
+          -0.028498240040284714,
+          0.00041963499785238845,
+          -0.002322931702641349,
+          0.0126393550661767,
+          -0.001595134597082556,
+          -0.00983073906977773,
+          0.006903313245694627,
+          0.0010522208685011098,
+          -0.015648940549713764,
+          -0.0007573245570947632,
+          -0.008533504878052644,
+          0.020475329445149498,
+          0.00027791387213409814,
+          0.001074213629646005,
+          0.013864331671732542,
+          -0.02102352993295736,
+          0.01576938071930521,
+          -0.021722775924283996,
+          0.01893267442161547,
+          -0.006577103101038035,
+          0.005772016664149475,
+          0.008283223343661555,
+          0.014068789332250016,
+          -0.014030042849491258,
+          -0.00663752723080222,
+          -0.034436220972634726,
+          -0.0122621879114314,
+          -0.001946072976826902,
+          -0.005308941654794638,
+          -0.014773123927909993,
+          -0.0011054822266082056,
+          -0.026205461509573584,
+          -0.006460373065662098,
+          0.012822188046808264,
+          0.019900312098359022,
+          -0.008409070094265829,
+          0.014163132434889265,
+          -0.006743711872055091,
+          -0.0028230771810882492,
+          -0.012652021749331811,
+          -0.018061180380009395,
+          -0.01083672324723357,
+          -0.01890966636827258,
+          -0.0018635408070944168,
+          0.004709222372611061,
+          -0.015760618199235518,
+          0.011435131737642627,
+          0.03198874888333405,
+          -0.01567271472650171,
+          0.008243406188694909,
+          0.016411496328333205,
+          0.0034236971031973503,
+          0.011728079139322837,
+          0.0038311415233052795,
+          0.004621789116267879,
+          -0.011109450058055078,
+          -0.011473158133621661,
+          0.006524490579223986,
+          0.00954511918454185,
+          -0.004913142601134428,
+          0.024347072255007807,
+          0.0008745419206472714,
+          0.005045794664611216,
+          0.005304506902446091,
+          0.011488187043525824,
+          -0.013135271908307747,
+          0.019746484188715552,
+          -0.002055210086078267,
+          -0.015571665869889405,
+          -0.01535499169234956,
+          -0.01105419264826655,
+          -0.018626399757822328,
+          0.0017108083455869296,
+          0.0181439079855921,
+          -0.0012493542718277685,
+          0.012137753522284387,
+          -0.00924460959150499,
+          -0.005019437549649743,
+          -0.0027156377206998474,
+          -0.038211954658821295,
+          -0.0021878970332906457,
+          -0.018809487178512628,
+          -0.02498945889484935,
+          0.03505216568400238,
+          -4.26477081213836e-05,
+          0.00997643255528116,
+          -0.010310197159232386,
+          -0.0002454902098986129,
+          -0.011065403358802515,
+          0.014954998647970473,
+          0.004624123403100598,
+          0.008872923678983478,
+          0.008560280683876702,
+          -0.016419500515801093,
+          -0.002997557798375716,
+          0.007472390947008284,
+          -0.00374822278027057,
+          0.012680407710694952,
+          0.015353684671120731,
+          0.020886474047003733,
+          0.011161262397502876,
+          0.002045862158732102,
+          0.00887271411363814,
+          0.009563361150297699,
+          -0.0003513237086717377,
+          -0.010459621030424415,
+          0.015419039131459582,
+          -0.02204848739950703,
+          0.012172257374410352,
+          0.04346634265794738,
+          0.019197433209794286,
+          0.029488039318675732,
+          -0.02272810439246353,
+          0.01448092574459596,
+          -0.003801043409441206,
+          0.009476229966122152,
+          0.0049309968883087685,
+          0.02346191840355209,
+          0.003973840862678076,
+          0.0073546443568171405,
+          0.010647581332452525,
+          -0.00972051412996416,
+          0.025644022498404723,
+          -0.00846669916801108,
+          5.1518540791212095e-05,
+          -0.004407265804145216,
+          -0.026400029105519137,
+          0.01615193008014211,
+          0.0081722536401398,
+          0.0010702314763197463,
+          0.0010263623479474385,
+          -0.0019566604229607643,
+          0.007982949510899019,
+          0.007079322685771456,
+          -0.015274148711633536,
+          -0.003886053737590683,
+          -0.018527455865296014,
+          0.0012864569064749438,
+          -0.024337133883771334,
+          0.010466651250597847,
+          -0.017568706814132484,
+          -0.014822354159709854,
+          -0.021666681078831483,
+          -0.013232016867264097,
+          -0.02011653224168991,
+          0.0035858355517491515,
+          -0.008632755352062642,
+          0.013025728891203198,
+          -0.016962939287139683,
+          -0.021494971929779273,
+          0.008430887589398292,
+          0.00870802359711456,
+          0.007481282677955534,
+          0.009728368603458082,
+          0.014996157358749115,
+          0.01736292711410622,
+          -0.022712386143092265,
+          -0.023163001611599246,
+          -0.006682209717364464,
+          -0.003531495447167166,
+          0.006762379833125297,
+          0.00933681196321128,
+          0.012884185500343249,
+          -0.007077895849501418,
+          -0.007554762054641417,
+          -0.013283067570021503,
+          0.006197966297581871,
+          -0.003993067576677973,
+          -0.009464789349849612,
+          -0.0015662739825834614,
+          -0.0061389795593180686,
+          0.002753664732282926,
+          -0.0001791088888074743,
+          0.0007595261237141144,
+          0.017122834676182343,
+          -0.008176319370751018,
+          -0.009787747261447871,
+          -0.02027228453943018,
+          -0.010683273300243025,
+          -0.015604745521156111,
+          -0.0016279406232527138,
+          -0.006506713191861036,
+          -0.014420323628589382,
+          0.010888206569219648,
+          -0.011930708571093279,
+          -0.00887210117616784,
+          0.010807147705360856,
+          0.0073900910661282935,
+          -0.003431679006289002,
+          0.037804020313161775,
+          0.0011945597874820288,
+          0.00778778394048267,
+          -0.003366543306746973,
+          0.005296727102519836,
+          -0.017761251541598876,
+          -0.026382400396890007,
+          0.0010357716464853195,
+          -0.01775777668381489,
+          0.010807713750049491,
+          -0.008515368407459745,
+          0.0016051389544388124,
+          -0.00034958600424912995,
+          -0.009585854918521076,
+          0.0036549720706351473,
+          -0.008110927907950148,
+          0.004212219431447508,
+          -0.03507890639477636,
+          0.011409377585617926,
+          -0.01819120811524444,
+          -0.008962242411035718,
+          0.01710177594046791,
+          0.037560936862121835,
+          0.0028513388998398104,
+          0.00030058353811717895,
+          0.0014792184333546447,
+          0.004427921639326168,
+          0.021438127785303025,
+          0.006339483221420334,
+          -0.001187747087595264,
+          -0.02237982599829809,
+          0.0062435650294483695,
+          0.002200578793697352,
+          0.010092828474934744,
+          0.023105023225115266,
+          -0.016797326021070008,
+          -0.015378520287395565,
+          -0.012732527281000097,
+          -0.010797463778182385,
+          0.009881499418260473,
+          0.02801767062785128,
+          0.01811139137849365,
+          -0.017358098300028903,
+          -0.020207173948076473,
+          0.0016242901745373126,
+          -0.026331540602868762,
+          0.007269751661479859,
+          0.11940631114242901,
+          -0.0045382135734049305,
+          -0.005994407567723939,
+          -0.008679712231646732,
+          -0.004482553208064535,
+          0.016709571294385293,
+          -0.013387043582971358,
+          0.069179848974036,
+          -0.007221545718220735,
+          0.0037392229364642915,
+          0.0028916224964057385,
+          0.00462729158956839,
+          -0.009727339449635306,
+          0.011570145487622916,
+          -0.011287365791681587,
+          -0.012991994493845187,
+          -0.013076194601468383,
+          -0.0180830261130804,
+          -0.007272768410218907,
+          -0.004155564814758654,
+          0.004204993170487392,
+          -0.005525575983182816,
+          0.008051108079520902,
+          -0.017551645696802648,
+          -0.010800070994485792,
+          0.013007176804761548,
+          -0.0023627485852619817,
+          0.005390986239687555,
+          -0.029668390278675554,
+          -0.0007650391259819867,
+          -0.0002446394046947245,
+          0.021280292985495608,
+          0.003137771886236958,
+          -0.0019077298815862946,
+          -0.010810159675564049,
+          0.007040107942856957,
+          0.023835993694726953,
+          0.021875366566457236,
+          0.0180511566130312,
+          0.012135351541044354,
+          0.020483975362807574,
+          -0.008667079096167644,
+          0.026324312888512866,
+          -0.001552207146726937,
+          -0.00397810465151493,
+          0.0077577136352257004,
+          -0.009849600842081875,
+          -0.00816392268697345,
+          0.00677353736462267,
+          -0.017288900282504148,
+          -0.020418071729964713,
+          -0.020383762873109595,
+          0.025741408097358233,
+          -0.011755094311186617,
+          0.002737129201159793,
+          -0.008266210469668866,
+          0.027287436144516414,
+          -0.006283892307996516,
+          -0.0029440642954701632,
+          0.009428876742171362,
+          -0.03788574758692254,
+          0.008834605607057885,
+          0.006463342392514337,
+          0.026711196829464153,
+          -3.9827332346130654e-05,
+          0.005565561841469247,
+          -0.0006738085601962865,
+          -0.006104320995853506,
+          -0.010792830169647851,
+          2.4565453208679464e-05,
+          0.017983908748172493,
+          0.0035181716362611077,
+          0.016970206528324653,
+          -0.005449808667944701,
+          -0.023171388778833197,
+          0.007092509502738752,
+          -0.01555320330471131,
+          -0.023306285898839978,
+          -0.002028783456704567,
+          -0.03196436161255752,
+          0.0033412618908711163,
+          0.007527633574188688,
+          -0.01890570959143866,
+          -0.017859735533232564,
+          0.010329340599904344,
+          0.004382396539605985,
+          -0.0016746397858953155,
+          0.006643785064338525,
+          -0.03307812909454471,
+          -0.01927475592253288,
+          0.004686970010943116,
+          0.012733525168279355,
+          -0.012993441126970916,
+          0.015291832952199082,
+          -0.020491167895268936,
+          0.016451976170508816,
+          -0.012259387189535237,
+          0.018393274368854132,
+          -0.006997668664226754,
+          0.011843450716200625,
+          0.011457415642860215,
+          0.009560877634897626,
+          -0.01790001115302499,
+          0.0214879415944361,
+          0.0066829070506954115,
+          0.03900547021123392,
+          0.02512484499576633,
+          -0.012651294056575184,
+          -0.0018425680226742782,
+          -0.00011361458320146007,
+          0.014982647266052349,
+          0.0061874424115559,
+          -0.003689464024837091,
+          0.02705902205734107,
+          0.018239469373522537,
+          -0.021070704644859155,
+          0.020037508334843183,
+          -0.00973804771680803,
+          0.018876283979591423,
+          -0.013303080480396818,
+          0.010592424494628194,
+          0.004717870260467274,
+          0.01070131238869639,
+          0.009087656015197565,
+          0.006708673669681232,
+          0.025928065153829043,
+          0.03663367449576964,
+          -0.00511073108390551,
+          0.002161381031392467,
+          0.004864092591944181,
+          0.012814518637357098,
+          0.001556833001747549,
+          0.007915766323215207,
+          -0.007012785514562176,
+          0.005834786334750152,
+          -0.002311533032619821,
+          -0.00122162928193159,
+          0.019525999280692766,
+          0.000824163362963678,
+          0.012875575199417254,
+          0.013259376059185821,
+          -0.002390133687709065,
+          -0.005682644446302278,
+          0.0019262218417782657,
+          -0.018193478045923098,
+          -0.015131696847666434,
+          0.029463723011600933,
+          -0.011447039651934148,
+          -0.008083114389964213,
+          0.0020642857121204753,
+          0.012160202510990566,
+          -0.00866944494225225,
+          -0.005378300635997713,
+          0.0038280364584418335,
+          0.014427318067525318,
+          -0.01799358399064332,
+          0.003381165206832161,
+          0.005252552821854107,
+          0.019433554824232185,
+          -0.00028791985674147885,
+          -0.013487678738391956,
+          0.0005317922815671783,
+          0.014982152801567637,
+          0.011902348165369078,
+          0.0016056390299304837,
+          -0.006935777197137747,
+          -0.00446584737370504,
+          0.004586814866182051,
+          0.006269466956110759,
+          0.0027516492952993445,
+          0.008157662045793032,
+          -0.0025001885993292637,
+          -0.016648747436203294,
+          0.017735757469839637,
+          -0.0030669873770852725,
+          0.018058248627079793,
+          -0.0038238312452176745,
+          -0.0147172470474118,
+          0.0043328417868095355,
+          -0.001888507803054499,
+          -0.018556943283494755,
+          0.009407930200720534,
+          0.002560949424273885,
+          0.010411159836451958,
+          0.0052571230238294825,
+          -0.007499470181994448,
+          -6.145889479758644e-05,
+          -0.015163819825329975,
+          0.022836682800304944,
+          0.016390944816242627,
+          0.001713572907204073,
+          -0.0035687505973373878,
+          -0.031210299824387475,
+          0.0029076511444359886,
+          -0.007822231980296341,
+          0.017821806420465356,
+          0.008250605126658402,
+          0.007908773724009905,
+          0.008351355849444162,
+          -0.00743970642051451,
+          -0.002411981861500092,
+          0.012543476434788293,
+          0.0202407777715137,
+          -0.0024635818448627447,
+          0.02535771413579989,
+          -0.012533449594824386,
+          -0.0022489292647974992,
+          0.0012362917358624392,
+          0.01578939075186126,
+          -0.0008001783766371555,
+          0.006311837812042848
+        ],
+        "keyphrases": [
+          "Switch Transformers",
+          "Mixture of Experts",
+          "sparsely-activated model",
+          "training instability",
+          "multilingual settings"
+        ],
+        "title": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity"
+      },
+      "type": "document"
+    },
+    {
+      "id": "8bd19de5-c2d4-458d-833b-5962c425f915",
+      "properties": {
+        "page_content": "1 Introduction\n\nScaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27]. The final model quality has been found to have a power-law relationship with the amount of data, model size, and compute time [16, 20]. However, training efficiency, which is defined as the total amount of computation used to achieve superior model quality than the state of the art system [21], should receive greater attention as we increase our efforts towards green AI [29].\n\nSparsely gated mixture-of-experts [31] (MoE) provides an effective way to scale model capacity given a fixed computational cost, and has recently played an important role in increasing the training efficiency of large-scale language models [10, 21]. MoE operate by adopting a number of experts, each as a sub-network, and by activating only one or a few experts for each input token. A gating network must be chosen and optimized in order to route each token to the most suited expert(s). For example, recent work has implemented sparse routing via k-means clustering [12], linear assignment to maximize token-expert affinities [22], or hashing [8, 28]. Many of the prior work use a routing strategy concerning the token choice, where each token selects the best one or two experts.\n\nWe argue that the independent token choice of prior work often leads to an imbalanced load of experts, which causes training inefficiency and sub-optimal training of the model. In order to mitigate this\n\nissue, previous sparsely gated networks introduce additional auxiliary losses as regularization to prevent too many tokens being routed to a single expert, but the effectiveness is still limited. Recent approaches [8, 22, 28] explore alternative strategies for routing, but they focus on pre-training only and do not demonstrate performance gain on downstream tasks. Moreover, none of the previous methods consider allocating a variable number of experts to each token based on importance, which can be beneficial.\n\nWe propose a very simple yet effective routing method we are calling expert choice. Unlike conventional MoE where tokens select one or two top-scoring experts, our method lets each expert pick the top-k tokens. Our method guarantees perfect load balancing, allows a variable number of experts for each token, and achieves substantial gains in training efficiency and downstream performance as demonstrated in our experiments. Our major contributions include:\n\nWe identify common pitfalls in conventional MoE such as load imbalance as described in Section 3.1. We then propose a heterogeneous, expert choice method to provide a fluid allocation of model parameters based on a learnt token-to-expert importance. This method intrinsically guarantees load balance without imposing an auxiliary loss.\n\nWe show our method provides over 2× faster training convergence in a 8B/64E (8 billion activated parameters, 64 experts) model, compared to the top-1 and top-2 gating counterparts in Switch Transformer [10] and GShard [21].\n\nWe show our method demonstrates strong scaling when increasing the number of experts from 16 to 128, evaluated in training perplexity.\n\nWe show our method demonstrates strong performance on downstream tasks selected from GLUE and SuperGLUE at all the evaluated scales. More specifically, our 8B/64E model outperforms a T5 11B dense model in 7 out of 11 tasks evaluated.\n\n",
+        "title": "Introduction",
+        "embedding": [
+          0.0024422307033091784,
+          -0.014919951558113098,
+          -0.01768755167722702,
+          -0.02871091105043888,
+          0.020494353026151657,
+          0.03838575258851051,
+          -0.01718577742576599,
+          0.0458810068666935,
+          -0.00809895433485508,
+          0.02212512120604515,
+          -0.01569613441824913,
+          -0.011674097739160061,
+          0.024147899821400642,
+          -0.016072465106844902,
+          -0.003586903912946582,
+          0.029040200635790825,
+          0.00016084715025499463,
+          0.0012985375942662358,
+          0.021921275183558464,
+          -0.021889913827180862,
+          -0.0052411919459700584,
+          -0.014426017180085182,
+          -0.01895766891539097,
+          -0.002695078030228615,
+          0.025621861219406128,
+          0.010239336639642715,
+          0.029526295140385628,
+          -0.00020776601741090417,
+          -0.05397212132811546,
+          0.04054965451359749,
+          0.015868619084358215,
+          0.025010323151946068,
+          -0.020776601508259773,
+          -0.014253532513976097,
+          -0.047417692840099335,
+          -0.029228366911411285,
+          0.05165141448378563,
+          0.018314769491553307,
+          -0.027127185836434364,
+          0.030765051022171974,
+          0.0260295532643795,
+          0.018973348662257195,
+          -0.037350840866565704,
+          -0.009784603491425514,
+          0.0013112779706716537,
+          0.00785982795059681,
+          -0.029761500656604767,
+          0.00613497756421566,
+          0.022031037136912346,
+          -0.01818932592868805,
+          0.02342659793794155,
+          -0.01548444852232933,
+          0.015492288395762444,
+          0.006064415443688631,
+          -0.06635968387126923,
+          0.04058101400732994,
+          0.0031008098740130663,
+          0.008036232553422451,
+          0.05513247475028038,
+          0.019961217418313026,
+          0.018205007538199425,
+          -0.0099727688357234,
+          -0.04525379091501236,
+          0.008130315691232681,
+          0.006036974955350161,
+          -0.019506484270095825,
+          -0.008647770620882511,
+          0.015061075799167156,
+          0.023410918191075325,
+          0.01010605227202177,
+          -0.055038392543792725,
+          0.029275408014655113,
+          -0.019929856061935425,
+          0.005562641192227602,
+          0.006781796459108591,
+          -0.01776595413684845,
+          0.01925559714436531,
+          0.01078031212091446,
+          0.005041265860199928,
+          0.0031556913163512945,
+          -0.01123504526913166,
+          0.038072142750024796,
+          0.004821739625185728,
+          0.014802348800003529,
+          0.04224314168095589,
+          -0.03951474279165268,
+          0.0034300992265343666,
+          -0.06153010204434395,
+          0.01611950621008873,
+          -0.012019068002700806,
+          -0.030733689665794373,
+          -0.003298775525763631,
+          -0.01593918167054653,
+          0.017828676849603653,
+          -0.003900512820109725,
+          0.041521843522787094,
+          0.011423210613429546,
+          -0.006993482355028391,
+          -0.02982422336935997,
+          -0.0009119163732975721,
+          -0.005151029210537672,
+          0.017985479906201363,
+          0.0025931550189852715,
+          -0.023175710812211037,
+          0.019396722316741943,
+          -0.03333664685487747,
+          -0.033023037016391754,
+          0.035281021147966385,
+          0.017907077446579933,
+          0.05739046260714531,
+          0.024100856855511665,
+          -0.011509452946484089,
+          0.011870103888213634,
+          0.025621861219406128,
+          -0.02924404665827751,
+          -0.017279859632253647,
+          -0.03810350224375725,
+          -0.06316086649894714,
+          0.019020389765501022,
+          -0.013657675124704838,
+          0.01680944673717022,
+          0.006762195844203234,
+          -0.012716848403215408,
+          -0.008624250069260597,
+          -0.06369400024414062,
+          0.045347873121500015,
+          -0.014026165939867496,
+          -0.011352648958563805,
+          -0.0018924347823485732,
+          -0.02450854890048504,
+          -0.003341896692290902,
+          0.028397301211953163,
+          -0.03418339043855667,
+          -0.05814312398433685,
+          0.040330126881599426,
+          0.013148060999810696,
+          0.03534374386072159,
+          0.04365438222885132,
+          -0.015782376751303673,
+          -0.022234883159399033,
+          0.026217719539999962,
+          -0.035124216228723526,
+          0.007946070283651352,
+          -0.0005517559475265443,
+          -0.02668813243508339,
+          0.01322646252810955,
+          0.010011970065534115,
+          0.06447802484035492,
+          -0.02052571438252926,
+          0.021889913827180862,
+          -0.02918132394552231,
+          0.057108212262392044,
+          -0.016260631382465363,
+          0.02201535739004612,
+          -0.023018905892968178,
+          -0.0031008098740130663,
+          -0.031219784170389175,
+          0.0018375532235950232,
+          -0.019584886729717255,
+          -0.011485932394862175,
+          -0.02443014644086361,
+          -0.01301477663218975,
+          -0.007569739129394293,
+          0.01749938726425171,
+          0.010686228983104229,
+          0.024931922554969788,
+          0.030827771872282028,
+          0.002036498859524727,
+          -0.07018571346998215,
+          0.026468606665730476,
+          0.018205007538199425,
+          0.01676240563392639,
+          0.01731122098863125,
+          0.019004710018634796,
+          0.006217299960553646,
+          0.026656771078705788,
+          0.011148802936077118,
+          0.011195844039320946,
+          -0.008059753105044365,
+          0.03985971212387085,
+          -0.03716267645359039,
+          -0.007585419807583094,
+          0.01726417988538742,
+          0.0046766954474151134,
+          0.026249079033732414,
+          0.018079563975334167,
+          0.01726417988538742,
+          0.012748208828270435,
+          0.007448215503245592,
+          0.01121936459094286,
+          0.00966699980199337,
+          0.03672362491488457,
+          0.01779731549322605,
+          -0.01638607494533062,
+          0.016433116048574448,
+          -0.02817777544260025,
+          -0.01050590444356203,
+          -0.004990304354578257,
+          0.02722126804292202,
+          0.00673867529258132,
+          -0.02339523658156395,
+          -0.0031027698423713446,
+          0.009196585975587368,
+          -0.022532811388373375,
+          0.008702651597559452,
+          0.006174178794026375,
+          -6.774446228519082e-05,
+          0.009525875560939312,
+          0.007271810434758663,
+          -0.030043749138712883,
+          0.023630443960428238,
+          -0.01404968649148941,
+          -0.008726172149181366,
+          -0.01792275905609131,
+          -0.007032683584839106,
+          -0.000910936389118433,
+          0.004002436064183712,
+          0.006323142908513546,
+          0.006664192769676447,
+          -0.022501451894640923,
+          -0.004837419837713242,
+          0.011454571969807148,
+          0.02488487958908081,
+          0.002355988137423992,
+          -0.0009266168344765902,
+          -0.020353229716420174,
+          -0.006664192769676447,
+          -0.01840885356068611,
+          0.04258811101317406,
+          -0.005386236123740673,
+          -0.03223901242017746,
+          0.009902206249535084,
+          0.014849389903247356,
+          0.010200135409832,
+          0.010286377742886543,
+          -0.042556751519441605,
+          0.00047947888378985226,
+          0.0019267357420176268,
+          0.005268632434308529,
+          -0.01114096213132143,
+          -0.019114473834633827,
+          0.019882814958691597,
+          -0.051431890577077866,
+          -0.014347615651786327,
+          0.029306767508387566,
+          0.02737807296216488,
+          -0.020070981234312057,
+          0.01941240206360817,
+          -0.007851987145841122,
+          -0.004453248810023069,
+          -0.0246967151761055,
+          -0.024336064234375954,
+          0.01234835758805275,
+          0.014378976076841354,
+          -0.0033595373388379812,
+          0.03434019535779953,
+          0.016213588416576385,
+          -0.034089308232069016,
+          -0.017091695219278336,
+          -0.0149669935926795,
+          0.050052009522914886,
+          -0.013006936758756638,
+          -0.015194360166788101,
+          0.06372536718845367,
+          -0.0255748201161623,
+          0.012622765265405178,
+          0.03515557944774628,
+          0.014684745110571384,
+          -0.002234464744105935,
+          -0.024947602301836014,
+          -0.0025970751885324717,
+          -0.05425437167286873,
+          -0.0051588695496320724,
+          -0.013720396906137466,
+          0.04224314168095589,
+          -0.04142775759100914,
+          -0.004076918121427298,
+          -0.005880170036107302,
+          0.01577453687787056,
+          0.03024759516119957,
+          0.006785716395825148,
+          0.00034153988235630095,
+          0.01049022376537323,
+          0.02308162860572338,
+          -0.0034457796718925238,
+          -0.014465218409895897,
+          -0.005151029210537672,
+          -0.018471574410796165,
+          -0.04838988184928894,
+          -0.011046879924833775,
+          -0.01742098480463028,
+          -0.004813899286091328,
+          -0.031454991549253464,
+          -0.00391815323382616,
+          0.03531238064169884,
+          -0.010647027753293514,
+          -0.015421726740896702,
+          0.008522327058017254,
+          0.00809895433485508,
+          -0.022877782583236694,
+          -0.04719816520810127,
+          0.004974624142050743,
+          0.00666027283295989,
+          -0.011321287602186203,
+          -0.0572650171816349,
+          0.006750435568392277,
+          -0.02461831271648407,
+          0.01837749220430851,
+          -0.021999677643179893,
+          -0.0099727688357234,
+          0.025872748345136642,
+          0.0016327272169291973,
+          0.0045120506547391415,
+          -0.02573162503540516,
+          0.0011064520804211497,
+          0.022595534101128578,
+          0.006699474062770605,
+          0.021105891093611717,
+          -0.027958249673247337,
+          -0.03418339043855667,
+          0.020839322358369827,
+          -0.0055077592842280865,
+          -0.06350583583116531,
+          -0.0026676373090595007,
+          -0.02251713164150715,
+          0.0027009581681340933,
+          0.019161514937877655,
+          0.04694727808237076,
+          -0.030184874311089516,
+          0.008663450367748737,
+          -0.023646123707294464,
+          -0.04650822654366493,
+          -0.030467122793197632,
+          -0.0027303590904921293,
+          -0.007957830093801022,
+          -0.026547009125351906,
+          -0.012991256080567837,
+          0.008467445150017738,
+          -0.03901296854019165,
+          0.009274988435208797,
+          -0.014245692640542984,
+          -0.026233399286866188,
+          -0.028648190200328827,
+          0.010427501983940601,
+          0.00468061538413167,
+          -0.002107060980051756,
+          -0.03603368252515793,
+          0.0060722557827830315,
+          0.004939342848956585,
+          -0.005162789486348629,
+          0.0010476503521203995,
+          0.002673517446964979,
+          0.004296444356441498,
+          0.005880170036107302,
+          -0.0027538796421140432,
+          -0.04183544963598251,
+          -0.026750855147838593,
+          0.016072465106844902,
+          0.02090204507112503,
+          -0.001751310657709837,
+          0.029197005555033684,
+          -0.050898753106594086,
+          -0.03230173513293266,
+          0.025951150804758072,
+          -0.06002477928996086,
+          0.006272181402891874,
+          -0.011995547451078892,
+          -0.022376008331775665,
+          -0.00047531374730169773,
+          -0.014073207043111324,
+          -0.02063547819852829,
+          0.026249079033732414,
+          0.02626476064324379,
+          0.017624830827116966,
+          0.017938438802957535,
+          0.013336226344108582,
+          0.02342659793794155,
+          -0.02220352366566658,
+          -0.06391353160142899,
+          0.035625990480184555,
+          -0.01566477306187153,
+          0.007781425025314093,
+          0.017860036343336105,
+          -0.012536522932350636,
+          -0.004010275937616825,
+          0.0303573589771986,
+          0.011713298968970776,
+          0.004022036213427782,
+          -0.018346130847930908,
+          0.05221591144800186,
+          -0.005986013449728489,
+          -0.04886029288172722,
+          -0.0021325417328625917,
+          -0.012120991013944149,
+          0.023489320650696754,
+          0.023348195478320122,
+          -0.00301064713858068,
+          -0.028742272406816483,
+          -0.022830741479992867,
+          -0.007197328377515078,
+          -0.015194360166788101,
+          0.02798961102962494,
+          -0.00789902824908495,
+          0.01466122455894947,
+          -0.0194908045232296,
+          -0.001328918500803411,
+          -0.004104358609765768,
+          0.01139185018837452,
+          0.04287036135792732,
+          -0.040643736720085144,
+          -0.009235787205398083,
+          -0.03361889347434044,
+          -0.03261534497141838,
+          0.00725613022223115,
+          0.017562108114361763,
+          0.03844847157597542,
+          -0.02182719111442566,
+          0.01078031212091446,
+          0.01853429712355137,
+          -0.010349099524319172,
+          -0.012418919242918491,
+          0.03393250331282616,
+          0.04873485118150711,
+          0.03452835977077484,
+          0.011956346221268177,
+          -0.00046306339208967984,
+          0.008992739953100681,
+          0.011415370739996433,
+          0.01152513362467289,
+          0.012967735528945923,
+          0.029338128864765167,
+          -0.008067593909800053,
+          0.006969961803406477,
+          -0.018126605078577995,
+          -0.020118022337555885,
+          -0.01611950621008873,
+          -0.011713298968970776,
+          0.030529843643307686,
+          -0.011768180876970291,
+          -0.005782167427241802,
+          0.0294792540371418,
+          -0.008498805575072765,
+          0.00907114241272211,
+          -0.009392592124640942,
+          -0.023755887523293495,
+          -0.03085913322865963,
+          -0.04202361777424812,
+          -0.005292152985930443,
+          -0.007753984536975622,
+          -0.035092856734991074,
+          -0.0529058501124382,
+          0.010380460880696774,
+          -0.07514073699712753,
+          -0.002481431933119893,
+          0.029965346679091454,
+          0.0010692109353840351,
+          0.0008305740193463862,
+          -0.0010888115502893925,
+          0.008208717219531536,
+          -0.03763309121131897,
+          0.0031478512100875378,
+          0.02243872918188572,
+          0.03089049458503723,
+          -0.0294792540371418,
+          -0.010991998016834259,
+          0.007103245705366135,
+          -0.03879344463348389,
+          -0.0038828724063932896,
+          0.0039887153543531895,
+          0.005190230440348387,
+          0.013312705792486668,
+          -0.03227037563920021,
+          -0.0069033196195960045,
+          0.004982464015483856,
+          0.01630767248570919,
+          0.03835438936948776,
+          -0.0023677486460655928,
+          -0.04512834548950195,
+          0.019004710018634796,
+          -0.0007732423837296665,
+          0.01531196292489767,
+          -0.00893785897642374,
+          -0.020839322358369827,
+          -0.0015778456581756473,
+          -0.00539799639955163,
+          0.0006272181635722518,
+          0.023332515731453896,
+          -0.008530166931450367,
+          0.01787571795284748,
+          -0.01707601360976696,
+          -0.015210039913654327,
+          -0.002385389059782028,
+          -0.006679873447865248,
+          0.018863586708903313,
+          0.0031125701498240232,
+          -0.021121570840477943,
+          -0.026092275977134705,
+          -0.017217138782143593,
+          0.027503516525030136,
+          0.0035927840508520603,
+          -0.040298767387866974,
+          -0.012795249931514263,
+          0.024289023131132126,
+          0.019726011902093887,
+          0.006613231264054775,
+          -0.019616248086094856,
+          0.008208717219531536,
+          0.03024759516119957,
+          -0.044657934457063675,
+          -0.02522985078394413,
+          0.0026931180618703365,
+          0.0033066156320273876,
+          -0.004888381343334913,
+          -0.011148802936077118,
+          0.0006757295341230929,
+          0.04365438222885132,
+          -0.010576466098427773,
+          0.021717429161071777,
+          0.019631927832961082,
+          0.01684080809354782,
+          -0.01898903027176857,
+          -0.013649835251271725,
+          -0.011454571969807148,
+          0.025010323151946068,
+          -0.014669064432382584,
+          0.027205588296055794,
+          0.022219203412532806,
+          0.002944005187600851,
+          0.06447802484035492,
+          -0.026672452688217163,
+          -0.025559140369296074,
+          0.011705459095537663,
+          -0.013579272665083408,
+          0.0009697380592115223,
+          -0.019475122913718224,
+          -0.007812785916030407,
+          -0.004092598333954811,
+          0.022109439596533775,
+          -0.028914757072925568,
+          -0.028507065027952194,
+          0.0017552308272570372,
+          0.010129572823643684,
+          -0.009925727732479572,
+          0.01257572416216135,
+          0.0027950408402830362,
+          0.004539491608738899,
+          0.0212470144033432,
+          -0.04149048030376434,
+          0.011681938543915749,
+          0.035469185560941696,
+          -0.02477511763572693,
+          -0.034402914345264435,
+          -0.031219784170389175,
+          0.011862263083457947,
+          0.0029890865553170443,
+          -0.04434432461857796,
+          -0.015037555247545242,
+          -0.029996708035469055,
+          -0.0015660853823646903,
+          -0.03572007268667221,
+          -0.00958859734237194,
+          0.010082531720399857,
+          -0.0037143074441701174,
+          0.01183090265840292,
+          -0.022987544536590576,
+          -0.005445037502795458,
+          0.028852034360170364,
+          0.04478337615728378,
+          0.012168032117187977,
+          0.020337548106908798,
+          -0.001616066787391901,
+          -0.024978963658213615,
+          -0.015492288395762444,
+          0.012599244713783264,
+          -0.0007776524871587753,
+          0.0031968525145202875,
+          0.0294478926807642,
+          -0.004907981958240271,
+          -0.03763309121131897,
+          0.00632706331089139,
+          0.04095734655857086,
+          -0.020572755485773087,
+          0.007146366871893406,
+          0.04192953556776047,
+          0.025935471057891846,
+          -0.019239917397499084,
+          -0.020008258521556854,
+          -0.031298186630010605,
+          0.004951103124767542,
+          -0.02817777544260025,
+          -0.005201990716159344,
+          0.03688042610883713,
+          -0.006185939069837332,
+          -0.026390204206109047,
+          0.012262115254998207,
+          -0.02297186478972435,
+          -0.009337710216641426,
+          0.01278741005808115,
+          0.01100767869502306,
+          -0.03625321015715599,
+          0.013328385539352894,
+          0.019537845626473427,
+          0.023332515731453896,
+          -0.008882977068424225,
+          -0.018973348662257195,
+          0.021074529737234116,
+          0.006166338454931974,
+          0.0032301736064255238,
+          -0.004202361684292555,
+          0.0075501385144889355,
+          0.019773053005337715,
+          -0.0036045443266630173,
+          0.011305606923997402,
+          -0.0034300992265343666,
+          0.03317984193563461,
+          -0.014473059214651585,
+          -0.002887163544073701,
+          0.01188578363507986,
+          -0.014622023329138756,
+          -0.009314189665019512,
+          0.059523001313209534,
+          0.007213008590042591,
+          0.040486931800842285,
+          -0.003974995110183954,
+          0.018847905099391937,
+          0.024633992463350296,
+          -0.01627631112933159,
+          -0.027205588296055794,
+          0.0006580890621989965,
+          -0.013869361020624638,
+          -0.03085913322865963,
+          -0.01699761115014553,
+          -0.03534374386072159,
+          -0.018471574410796165,
+          -0.00014565671153832227,
+          -0.01404968649148941,
+          0.0007394314161501825,
+          0.028428662568330765,
+          0.010239336639642715,
+          -0.012168032117187977,
+          -0.033901140093803406,
+          0.03885616362094879,
+          0.030874812975525856,
+          0.03261534497141838,
+          -0.007236529607325792,
+          0.019271278753876686,
+          -0.010074691846966743,
+          -0.0058135283179581165,
+          0.020807962864637375,
+          0.015437406487762928,
+          -0.00238342909142375,
+          0.014825869351625443,
+          0.01577453687787056,
+          0.0459437295794487,
+          -0.025621861219406128,
+          -0.03024759516119957,
+          -0.010176614858210087,
+          -0.0260295532643795,
+          0.027817124500870705,
+          0.012983416207134724,
+          -0.015445247292518616,
+          0.015727495774626732,
+          0.017044654116034508,
+          -0.029620377346873283,
+          0.02010234259068966,
+          -0.01374391745775938,
+          -0.03195676580071449,
+          0.030325997620821,
+          -0.016825126484036446,
+          0.013242143206298351,
+          -0.007013082969933748,
+          -0.004033796954900026,
+          -0.020086660981178284,
+          0.009298508986830711,
+          0.01131344772875309,
+          -0.028554106131196022,
+          0.038072142750024796,
+          0.04368574544787407,
+          0.003083169227465987,
+          0.009604278020560741,
+          0.014574982225894928,
+          -0.0007943129749037325,
+          -0.020353229716420174,
+          -0.012724688276648521,
+          0.006679873447865248,
+          -0.001151533331722021,
+          -0.028836354613304138,
+          0.0307807307690382,
+          -0.016103826463222504,
+          0.007495257072150707,
+          -0.01154081430286169,
+          -0.006087936460971832,
+          0.0063192229717969894,
+          -0.0007805926143191755,
+          0.006558349821716547,
+          0.0007997031207196414,
+          -0.03170587867498398,
+          0.014888591133058071,
+          0.031110020354390144,
+          -0.005758646875619888,
+          -0.01079599279910326,
+          0.01737394370138645,
+          -0.02090204507112503,
+          0.009180906228721142,
+          0.028005290776491165,
+          0.015186519362032413,
+          0.004206281621009111,
+          -0.0051980703137815,
+          -0.030310317873954773,
+          -0.020243465900421143,
+          -0.001436721533536911,
+          0.024304702877998352,
+          0.014841550029814243,
+          0.026437245309352875,
+          0.026735173538327217,
+          -0.00780886597931385,
+          0.0013436188455671072,
+          -0.0022677856031805277,
+          -0.03396386280655861,
+          0.04440704733133316,
+          -1.7808952179620974e-05,
+          0.01779731549322605,
+          0.021152932196855545,
+          -0.027095824480056763,
+          -0.027268309146165848,
+          -0.001135852886363864,
+          0.032333094626665115,
+          0.03484196960926056,
+          0.01699761115014553,
+          0.004590453114360571,
+          0.04202361777424812,
+          -0.02464967407286167,
+          -0.03741356357932091,
+          0.017891397699713707,
+          0.030043749138712883,
+          -0.009188746102154255,
+          0.0016738884150981903,
+          0.027017422020435333,
+          -0.011838742531836033,
+          -0.005574401468038559,
+          0.024367425590753555,
+          0.019083112478256226,
+          0.008914338424801826,
+          0.004104358609765768,
+          -0.009659159928560257,
+          0.02369316667318344,
+          0.002107060980051756,
+          0.024336064234375954,
+          -0.01960056647658348,
+          0.007318852003663778,
+          0.00945531390607357,
+          -0.026891978457570076,
+          0.033023037016391754,
+          0.009941407479345798,
+          -0.0502401739358902,
+          0.025668902322649956,
+          0.0032517341896891594,
+          -0.004304284695535898,
+          0.003394818166270852,
+          0.0059076109901070595,
+          0.007789265364408493,
+          0.023363877087831497,
+          0.00446108914911747,
+          -0.004096518736332655,
+          0.030106471851468086,
+          0.023912692442536354,
+          -0.0003197342448402196,
+          0.020666837692260742,
+          -0.0019110552966594696,
+          -0.02074524015188217,
+          -0.002781320596113801,
+          -0.0063780248165130615,
+          0.04105142876505852,
+          -0.0390443317592144,
+          0.014825869351625443,
+          0.02374020777642727,
+          0.003420298919081688,
+          -0.002857762621715665,
+          -0.02320707216858864,
+          -0.008443924598395824,
+          -0.011180163361132145,
+          0.024257661774754524,
+          -0.010145253501832485,
+          0.023301154375076294,
+          -0.02833458036184311,
+          -0.0038926727138459682,
+          0.01353223156183958,
+          -0.030498482286930084,
+          0.011031199246644974,
+          0.02722126804292202,
+          0.0010897915344685316,
+          -0.02010234259068966,
+          -0.005852729547768831,
+          0.036849066615104675,
+          0.055226556956768036,
+          0.0054959990084171295,
+          0.01630767248570919,
+          0.02867954969406128,
+          -0.02140381932258606,
+          -0.0106235072016716,
+          0.019537845626473427,
+          -0.018126605078577995,
+          0.014582822099328041,
+          0.04302716627717018,
+          -0.036347292363643646,
+          -0.03782125562429428,
+          -0.008130315691232681,
+          -0.032333094626665115,
+          0.008898657746613026,
+          -0.012003387324512005,
+          0.012889333069324493,
+          -0.009204426780343056,
+          -0.02841298282146454,
+          0.02798961102962494,
+          0.015170839615166187,
+          0.009384751319885254,
+          -0.014622023329138756,
+          -0.02182719111442566,
+          -0.02411653846502304,
+          -0.031078658998012543,
+          -0.008898657746613026,
+          -0.01765619032084942,
+          0.01975737139582634,
+          -0.0006556389853358269,
+          -0.04224314168095589,
+          -0.007848067209124565,
+          0.0194908045232296,
+          -0.012120991013944149,
+          0.040675096213817596,
+          0.02416357956826687,
+          0.016558559611439705,
+          -0.009267148561775684,
+          -0.03760172799229622,
+          0.05425437167286873,
+          -2.794612191792112e-05,
+          0.03027895651757717,
+          -0.013179421424865723,
+          -0.007655981462448835,
+          0.0026107956655323505,
+          0.027958249673247337,
+          0.005201990716159344,
+          0.009470994584262371,
+          -0.031909722834825516,
+          0.011078240349888802,
+          -0.024304702877998352,
+          -0.011932825669646263,
+          -0.006899399682879448,
+          -0.0034673402551561594,
+          0.02450854890048504,
+          0.00416316045448184,
+          0.006217299960553646,
+          0.02875795215368271,
+          0.012599244713783264,
+          -0.0017483705887570977,
+          -0.023834289982914925,
+          -0.013493030332028866,
+          -0.0015964661724865437,
+          0.003849551547318697,
+          -0.01676240563392639,
+          -0.013053977862000465,
+          -0.03835438936948776,
+          -0.010748950764536858,
+          0.0034810607321560383,
+          0.021152932196855545,
+          0.02151358313858509,
+          0.005688084755092859,
+          -0.010819513350725174,
+          -0.03085913322865963,
+          -0.007405094336718321,
+          -0.04735497012734413,
+          -0.016001902520656586,
+          0.018628379330039024,
+          -0.021043168380856514,
+          0.010356939397752285,
+          0.00260295532643795,
+          0.01590782031416893,
+          -0.0012289555743336678,
+          -0.01925559714436531,
+          0.003136090701445937,
+          -0.009086823090910912,
+          0.011548654176294804,
+          0.0010986118577420712,
+          0.03932657837867737,
+          -0.004084758460521698,
+          0.027660321444272995,
+          -0.029604697600007057,
+          0.008585048839449883,
+          -0.044124796986579895,
+          -0.0097610829398036,
+          -0.03866799920797348,
+          0.009855165146291256,
+          -0.008161676116287708,
+          -0.0338384211063385,
+          0.004523810930550098,
+          0.008357682265341282,
+          -0.002804841147735715,
+          -0.015892140567302704,
+          -0.006468187086284161,
+          0.02294050343334675,
+          -0.0047668581828475,
+          0.008953538723289967,
+          -0.03938930109143257,
+          -0.016166547313332558,
+          -0.007510937284678221,
+          0.03675498440861702,
+          -0.0036241449415683746,
+          0.017436664551496506,
+          -0.015727495774626732,
+          0.016213588416576385,
+          -0.04443840682506561,
+          -0.025825707241892815,
+          0.034057945013046265,
+          0.017389623448252678,
+          -0.021874234080314636,
+          0.026625411584973335,
+          0.019710330292582512,
+          0.04042420908808708,
+          -0.0580490417778492,
+          0.023802928626537323,
+          -0.0018473535310477018,
+          -0.009008420631289482,
+          0.019098792225122452,
+          0.010756791569292545,
+          0.03289759159088135,
+          -0.028961798176169395,
+          -0.014441697858273983,
+          0.006017374340444803,
+          0.020666837692260742,
+          8.042357512749732e-05,
+          0.006691633723676205,
+          -0.02358340285718441,
+          0.014496579766273499,
+          0.017562108114361763,
+          0.005190230440348387,
+          0.02703310176730156,
+          -0.026390204206109047,
+          0.028428662568330765,
+          0.014324095100164413,
+          -0.027770083397626877,
+          0.02132541686296463,
+          -0.027127185836434364,
+          -0.01390072237700224,
+          -0.001531784306280315,
+          0.02780144475400448,
+          0.011360488831996918,
+          -0.060244303196668625,
+          0.02063547819852829,
+          0.03173723816871643,
+          -0.006189859006553888,
+          0.03134522587060928,
+          -0.024053815752267838,
+          0.024665353819727898,
+          0.021372457966208458,
+          0.022799380123615265,
+          0.018440213054418564,
+          0.046037811785936356,
+          -0.00785982795059681,
+          0.01184658333659172,
+          -0.020118022337555885,
+          0.022579854354262352,
+          0.011595695279538631,
+          -0.009541556239128113,
+          -0.037570368498563766,
+          -0.021152932196855545,
+          0.007648141589015722,
+          0.013469509780406952,
+          -0.00831064023077488,
+          -0.03317984193563461,
+          -0.0458810068666935,
+          0.010090372525155544,
+          0.00569592509418726,
+          -0.057484544813632965,
+          -0.013555752113461494,
+          -0.009110343642532825,
+          0.005229431204497814,
+          -0.008459605276584625,
+          0.004061237443238497,
+          -0.028397301211953163,
+          0.015335483476519585,
+          -0.014457378536462784,
+          0.03716267645359039,
+          -0.011485932394862175,
+          0.022140800952911377,
+          -0.015837257727980614,
+          -0.004719816613942385,
+          -0.025292571634054184,
+          0.02488487958908081,
+          -0.021466542035341263,
+          -0.04199225455522537,
+          0.010835194028913975,
+          0.004480689764022827,
+          0.01622927002608776,
+          0.0537525974214077,
+          -0.005884090438485146,
+          -0.010317739099264145,
+          0.020321868360042572,
+          -0.019506484270095825,
+          -0.014002645388245583,
+          0.007679502479732037,
+          -0.005844889208674431,
+          -0.022031037136912346,
+          -0.01665264181792736,
+          -0.012262115254998207,
+          0.0247280765324831,
+          -0.00041602205601520836,
+          0.022611213847994804,
+          0.026390204206109047,
+          0.0195221658796072,
+          -0.02814641408622265,
+          0.009855165146291256,
+          -0.005127508658915758,
+          -0.011031199246644974,
+          0.004245482850819826,
+          0.021356778219342232,
+          -0.004755097441375256,
+          0.009000580757856369,
+          -0.01646447740495205,
+          -0.021121570840477943,
+          -0.03731948137283325,
+          -0.01249732170253992,
+          -0.003145891008898616,
+          0.0031478512100875378,
+          -0.035625990480184555,
+          -0.018126605078577995,
+          0.026060914620757103,
+          0.03070232830941677,
+          -0.028303219005465508,
+          0.0019051751587539911,
+          -0.002246225019916892,
+          -0.01132912840694189,
+          -0.003904432989656925,
+          -0.011791701428592205,
+          0.007040523923933506,
+          -0.015225720591843128,
+          8.691626862855628e-05,
+          -0.002042379230260849,
+          0.0023736287839710712,
+          -0.05792359635233879,
+          -0.017781633883714676,
+          0.02913428284227848,
+          0.002930284710600972,
+          0.00550383934751153,
+          -0.008279279805719852,
+          -0.01322646252810955,
+          -0.007009163033217192,
+          -0.02679789625108242,
+          0.024367425590753555,
+          0.015437406487762928,
+          0.019647609442472458,
+          -0.003559462958946824,
+          0.0017885518027469516,
+          0.018659740686416626,
+          0.0014171210350468755,
+          0.003351696999743581,
+          -0.007279650773853064,
+          0.006507388316094875,
+          0.02182719111442566,
+          -0.012795249931514263,
+          0.0158294178545475,
+          0.005778247490525246,
+          0.002685277722775936,
+          -0.02679789625108242,
+          -0.003682946553453803,
+          0.016793766990303993,
+          0.002657837001606822,
+          0.005296073388308287,
+          -0.0255434587597847,
+          0.038072142750024796,
+          0.04039284959435463,
+          0.025872748345136642,
+          0.020337548106908798,
+          -0.03220765292644501,
+          -0.004206281621009111,
+          -0.00758149940520525,
+          0.01121936459094286,
+          0.021811511367559433,
+          0.020055299624800682,
+          -0.010427501983940601,
+          -0.015743175521492958,
+          0.011062560603022575,
+          -0.02013370208442211,
+          -0.0019473163411021233,
+          0.018894946202635765,
+          0.011266405694186687,
+          -0.006201619748026133,
+          0.009988449513912201,
+          0.0004351325915195048,
+          0.005268632434308529,
+          0.02140381932258606,
+          -0.030529843643307686,
+          -0.025167128071188927,
+          0.0212470144033432,
+          -0.017123056575655937,
+          0.01154081430286169,
+          0.004876621067523956,
+          0.0032517341896891594,
+          0.020776601508259773,
+          0.07445079833269119,
+          -0.020070981234312057,
+          0.0043591661378741264,
+          -0.022768018767237663,
+          -0.02576298639178276,
+          -0.012614925391972065,
+          -0.022109439596533775,
+          0.005378395784646273,
+          0.013571432791650295,
+          -0.00024022944853641093,
+          0.020651157945394516,
+          0.01056862622499466,
+          -0.00644074659794569,
+          0.007216928992420435,
+          -0.01173681952059269,
+          -0.0007139506633393466,
+          -0.006585790775716305,
+          -0.0027087985072284937,
+          0.002058059675619006,
+          0.0015229640994220972,
+          0.002697037998586893,
+          0.02836594171822071,
+          0.012371878139674664,
+          0.013116699643433094,
+          -0.03327392414212227,
+          0.009815963916480541,
+          0.02151358313858509,
+          0.018095243722200394,
+          -0.03317984193563461,
+          -0.021388139575719833,
+          0.004543411545455456,
+          0.005017745308578014,
+          0.012928534299135208,
+          0.009196585975587368,
+          -0.006977802142500877,
+          0.02504168450832367,
+          -0.0355319082736969,
+          0.01814228482544422,
+          0.016527198255062103,
+          -0.011705459095537663,
+          -0.004931502975523472,
+          0.01986713521182537,
+          -0.0020090581383556128,
+          0.006860198453068733,
+          -0.005946812219917774,
+          0.0025794347748160362,
+          0.00653874920681119,
+          -0.021356778219342232,
+          -0.003559462958946824,
+          -0.012983416207134724,
+          -0.001520024030469358,
+          0.021419500932097435,
+          -0.017170097678899765,
+          0.008726172149181366,
+          -0.008655610494315624,
+          0.028836354613304138,
+          0.013414627872407436,
+          -0.029933987185359,
+          0.0007183607667684555,
+          0.023065946996212006,
+          0.01173681952059269,
+          -0.002052179304882884,
+          0.00922794733196497,
+          0.008851616643369198,
+          -0.016825126484036446,
+          6.958201265661046e-05,
+          0.012520842254161835,
+          0.018079563975334167,
+          0.007957830093801022,
+          0.006879799067974091,
+          0.006417225580662489,
+          0.019616248086094856,
+          -0.018785184249281883,
+          -0.02010234259068966,
+          0.00456693209707737,
+          0.010521584190428257,
+          -0.0065426696091890335,
+          -0.020996127277612686,
+          0.007060124538838863,
+          -0.016401754692196846,
+          0.037350840866565704,
+          0.009690520353615284,
+          -0.010803832672536373,
+          -0.0035790635738521814,
+          -0.031141381710767746,
+          -0.0027479995042085648,
+          -0.021983996033668518,
+          -0.010153094306588173,
+          0.019004710018634796,
+          -0.003353656968101859,
+          -0.00965131912380457,
+          -0.01906743273139,
+          -0.011015518568456173,
+          -0.008734012953937054,
+          0.0017326901433989406,
+          -0.008984900079667568,
+          0.01944376341998577,
+          0.01144673116505146,
+          -0.02844434417784214,
+          -0.031799960881471634,
+          -0.03361889347434044,
+          0.011054719798266888,
+          0.0025559139903634787,
+          -0.02913428284227848,
+          -0.011721138842403889,
+          -0.004555171821266413,
+          -0.004080838058143854,
+          -0.008000951260328293,
+          0.008145995438098907,
+          -0.003831910900771618,
+          0.006017374340444803,
+          -0.013830160722136497,
+          -0.005554800853133202,
+          -0.008130315691232681,
+          0.00788726843893528,
+          0.0149513129144907,
+          -0.005488158669322729,
+          -0.03093753568828106,
+          -0.017781633883714676,
+          0.00965131912380457,
+          -0.012826611287891865,
+          0.008145995438098907,
+          0.01684080809354782,
+          -0.004923662636429071,
+          0.00594289181753993,
+          0.01848725602030754,
+          0.018095243722200394,
+          -0.0030753291212022305,
+          0.014363295398652554,
+          -0.0011162522714585066,
+          -0.006385864689946175,
+          0.012607084587216377,
+          0.008953538723289967,
+          0.01487291045486927,
+          0.02507304586470127,
+          -0.010294217616319656,
+          -0.006087936460971832,
+          -0.0016954491147771478,
+          0.03274078667163849,
+          -0.023599082604050636,
+          -0.019804412499070168,
+          0.015688294544816017,
+          0.05425437167286873,
+          -0.0010662708664312959,
+          -0.0020913805346935987,
+          -0.022344646975398064,
+          -0.01102335937321186,
+          0.023489320650696754,
+          0.016793766990303993,
+          0.008036232553422451,
+          -0.010843033902347088,
+          -0.014386816881597042,
+          0.010545105673372746,
+          0.014418177306652069,
+          0.01922423578798771,
+          -0.024053815752267838,
+          0.021278375759720802,
+          -0.026750855147838593,
+          0.036127764731645584,
+          0.0075501385144889355,
+          -0.005974252708256245,
+          0.015539329499006271,
+          0.0036378654185682535,
+          -0.006526988931000233,
+          0.014324095100164413,
+          -0.01070190966129303,
+          0.011893624439835548,
+          0.006868038792163134,
+          -0.0015004234155640006,
+          0.021043168380856514,
+          -0.01108608115464449,
+          -0.0010045290691778064,
+          -0.0027048783376812935,
+          0.02090204507112503,
+          0.0031184502877295017,
+          -0.0038671919610351324,
+          0.01464554388076067,
+          0.0028773632366210222,
+          -0.004351325798779726,
+          -0.010074691846966743,
+          0.007350212894380093,
+          -0.03173723816871643,
+          -0.008138155564665794,
+          0.013924242928624153,
+          -0.005735125858336687,
+          0.002120781457051635,
+          -0.011015518568456173,
+          -0.002642156556248665,
+          -0.03609640523791313,
+          0.01787571795284748,
+          -0.000642898608930409,
+          0.017091695219278336,
+          0.0028656029608100653,
+          -0.00936123076826334,
+          -0.004809979349374771,
+          0.02297186478972435,
+          -0.009565076790750027,
+          -0.006930760573595762,
+          -0.018769502639770508,
+          -0.021497901529073715,
+          -0.025135766714811325,
+          0.021121570840477943,
+          -0.009580757468938828,
+          0.0018189327092841268,
+          -0.007706942968070507,
+          -0.0029126442968845367,
+          -0.00791470892727375,
+          0.012222914025187492,
+          0.018706781789660454,
+          -0.030576884746551514,
+          0.006617151666432619,
+          -0.006170258391648531,
+          -0.022109439596533775,
+          0.006456426810473204,
+          -0.009518035687506199,
+          0.01684080809354782,
+          -0.0009344570571556687,
+          -0.04123959317803383,
+          0.006303542293608189,
+          0.026484286412596703,
+          -0.017969800159335136,
+          -0.0015141437761485577,
+          0.010043330490589142,
+          0.008718332275748253,
+          -0.0026499966625124216,
+          -0.018894946202635765,
+          0.01811092346906662,
+          -0.017436664551496506,
+          0.018675420433282852,
+          0.026844937354326248,
+          -0.008757533505558968,
+          0.028993159532546997,
+          0.0001089056531782262,
+          0.01864405907690525,
+          0.0029832064174115658,
+          0.005641043186187744,
+          0.010082531720399857,
+          0.0038985528517514467,
+          0.020070981234312057,
+          0.00295772566460073,
+          0.007949990220367908,
+          0.038072142750024796,
+          -0.010788151994347572,
+          0.02182719111442566,
+          0.011305606923997402,
+          -0.00019184054690413177,
+          0.01615086756646633,
+          -0.007628540974110365,
+          -0.032489899545907974,
+          0.01566477306187153,
+          -0.027770083397626877,
+          -0.004668855108320713,
+          0.00922794733196497,
+          -0.0147396270185709,
+          -0.00934555009007454,
+          -0.014418177306652069,
+          0.009235787205398083,
+          0.012779570184648037,
+          -0.013908562250435352,
+          -0.0005326454411260784,
+          0.018973348662257195,
+          0.019381040707230568,
+          0.014527940191328526,
+          0.005982093047350645,
+          -0.005182390101253986,
+          0.013830160722136497,
+          0.0030204474460333586,
+          0.023646123707294464,
+          0.0016895688604563475,
+          0.005633202847093344,
+          -0.010592146776616573,
+          0.0145122604444623,
+          0.01508459635078907,
+          -0.0017875717021524906,
+          0.01092927623540163,
+          -0.004845260176807642,
+          -0.005817448254674673,
+          -0.01960056647658348,
+          -0.007636380847543478,
+          -0.009110343642532825,
+          0.006170258391648531,
+          0.04719816520810127,
+          0.0004782538453582674,
+          -0.002846002345904708,
+          -0.01765619032084942,
+          0.004559091757982969,
+          0.012685487046837807,
+          -0.0097454022616148,
+          -0.0034987013787031174,
+          -0.019365360960364342,
+          -0.022720977663993835,
+          0.0034144187811762094,
+          -0.0050843870267271996,
+          0.027001742273569107,
+          -0.003822110593318939,
+          0.02767600119113922,
+          -0.029102923348546028,
+          0.01956920698285103,
+          -0.016135187819600105,
+          -0.0097454022616148,
+          -0.0022560253273695707,
+          -0.012160192243754864,
+          -0.014331934973597527,
+          0.02300322614610195,
+          0.00936123076826334,
+          -0.032678067684173584,
+          -0.0016190068563446403,
+          0.01010605227202177,
+          -0.009306349791586399,
+          0.009008420631289482,
+          0.006860198453068733,
+          0.010803832672536373,
+          -0.0015798057429492474,
+          0.01190930511802435,
+          0.0013592992909252644,
+          -0.001451421994715929,
+          -9.536901052342728e-05,
+          -0.02294050343334675,
+          0.018346130847930908,
+          0.030482802540063858,
+          -0.05265496298670769,
+          -0.01914583332836628,
+          0.020118022337555885,
+          0.00569592509418726,
+          0.0005865469574928284,
+          0.013061817735433578,
+          -0.024367425590753555,
+          0.01027853786945343,
+          -0.018675420433282852,
+          -0.014316254295408726,
+          -0.022140800952911377,
+          -0.005460718180984259,
+          0.0151786794885993,
+          -0.027581918984651566,
+          0.025794345885515213,
+          0.017170097678899765,
+          -0.0015023835003376007,
+          0.006452506873756647,
+          0.0007472716388292611,
+          0.008851616643369198,
+          -0.03255262225866318,
+          -0.006617151666432619,
+          -0.0059664128348231316,
+          0.006009534001350403,
+          0.003655505832284689,
+          -0.02897747792303562,
+          0.014527940191328526,
+          -0.02074524015188217,
+          0.008263599127531052,
+          -0.029949666932225227,
+          0.011321287602186203,
+          0.03622184693813324,
+          -2.7471420253277756e-05,
+          0.009157384745776653,
+          -0.0031282505951821804,
+          0.04218042269349098,
+          0.026625411584973335,
+          0.0015709854196757078,
+          -0.008024472743272781,
+          -0.019694650545716286,
+          -0.002522593131288886,
+          0.00673867529258132,
+          -0.02918132394552231,
+          0.0012113150442019105,
+          -0.035092856734991074,
+          0.02209375984966755,
+          -0.006099696736782789,
+          -0.025512099266052246,
+          0.033556170761585236,
+          -0.020384589210152626,
+          0.002808761317282915,
+          -0.005347034893929958,
+          0.0027068383060395718,
+          0.030561204999685287,
+          0.0024990723468363285,
+          0.03005943074822426,
+          -0.03826030716300011,
+          -0.0013504790840670466,
+          0.0028244417626410723,
+          0.0038162304554134607,
+          -0.010435341857373714,
+          -0.015711814165115356,
+          0.016903528943657875,
+          -0.010200135409832,
+          0.004292524419724941,
+          0.02522985078394413,
+          -0.003655505832284689,
+          -0.013132380321621895,
+          0.021278375759720802,
+          -0.002808761317282915,
+          0.005417597014456987,
+          -0.005460718180984259,
+          0.000567926443181932,
+          0.023222751915454865,
+          0.02568458393216133,
+          -0.01136832870543003,
+          -0.011336968280375004,
+          -0.002781320596113801,
+          -0.00031654915073886514,
+          0.014104568399488926,
+          0.015264921821653843,
+          -0.008200877346098423,
+          -0.020541394129395485,
+          0.006817077286541462,
+          0.009808124043047428,
+          0.018549976870417595,
+          0.0012946174247190356,
+          -0.0255748201161623,
+          -0.008459605276584625,
+          -0.03027895651757717,
+          0.010153094306588173,
+          -0.016558559611439705,
+          -0.005601841956377029,
+          0.0013112779706716537,
+          -0.001341658877208829,
+          -0.009549396112561226,
+          0.014394656755030155,
+          0.015194360166788101,
+          -0.03901296854019165,
+          -0.015813738107681274,
+          -0.0007222809363156557,
+          -0.018502935767173767,
+          0.0024539909791201353,
+          -0.0011662337929010391,
+          0.001961036818102002,
+          0.0021658625919371843,
+          -0.003165491623803973,
+          0.004739417228847742,
+          -0.020070981234312057,
+          -0.0024324303958564997,
+          0.005629282910376787,
+          0.02113725244998932,
+          0.004535571206361055,
+          -0.0015014035161584616,
+          0.0022442650515586138,
+          0.0038907125126570463,
+          0.0017464105039834976,
+          0.0063897850923240185,
+          -0.00673867529258132,
+          0.015108116902410984,
+          -0.00020262086763978004,
+          0.015327643603086472,
+          -0.01906743273139,
+          0.007557978853583336,
+          -0.002369708614423871,
+          -0.005123588256537914,
+          0.015782376751303673,
+          -0.00037020572926849127,
+          -0.0020482593681663275,
+          -0.010294217616319656,
+          0.007605019956827164,
+          0.015586371533572674,
+          -0.022501451894640923,
+          -0.009063302539288998,
+          -0.014614183455705643,
+          0.02711150422692299,
+          0.0153746847063303,
+          0.008122474886476994,
+          -0.03223901242017746,
+          -0.0056488835252821445,
+          -0.012120991013944149,
+          -0.0059193712659180164,
+          -0.015296283178031445,
+          0.018863586708903313,
+          0.019349679350852966,
+          -0.0051784696988761425,
+          0.01619790866971016,
+          -0.0015592251438647509,
+          0.020855003967881203,
+          -0.00924362801015377,
+          0.006476027425378561,
+          -0.0007590319728478789,
+          -0.006315303035080433,
+          0.010168774053454399,
+          0.03521829843521118,
+          -0.017389623448252678,
+          0.008240078575909138,
+          -0.000719830859452486,
+          -0.002530433237552643,
+          -0.00306356861256063,
+          -0.004022036213427782,
+          0.006358424201607704,
+          0.014316254295408726,
+          0.009369071573019028,
+          0.0029342048801481724,
+          0.07056204229593277,
+          -0.003506541484966874,
+          0.020243465900421143,
+          0.006323142908513546,
+          -0.014574982225894928,
+          -0.005891930777579546,
+          0.0022109439596533775,
+          -0.028397301211953163,
+          0.031266823410987854,
+          0.020086660981178284,
+          0.0048413402400910854,
+          0.003782909596338868,
+          -0.014692584984004498,
+          0.023034587502479553,
+          0.004453248810023069,
+          0.0030616086442023516,
+          -0.0049119023606181145,
+          0.005864489823579788,
+          0.047762662172317505,
+          0.012403239496052265,
+          -0.007420775014907122,
+          -0.0022658256348222494,
+          0.014292733743786812,
+          0.0054254368878901005,
+          -0.011493772268295288,
+          -0.0003150791162624955,
+          -0.000643878651317209,
+          0.015037555247545242,
+          -0.001587645965628326,
+          0.021952634677290916,
+          -0.01114096213132143,
+          -0.01826772838830948,
+          0.0018375532235950232,
+          0.004394447430968285,
+          0.01571965590119362,
+          0.009486674331128597,
+          0.008318481035530567,
+          -0.005884090438485146,
+          0.004700215999037027,
+          0.006397624965757132,
+          0.005664564203470945,
+          0.00790294911712408,
+          -0.005852729547768831,
+          -0.008655610494315624,
+          -0.01826772838830948,
+          0.02637452259659767,
+          -0.002616675803437829,
+          0.003259574295952916,
+          -0.016856487840414047,
+          0.002438310533761978,
+          -0.0034712604247033596,
+          -0.007769664749503136,
+          0.010937117040157318,
+          0.017750274389982224,
+          -0.009392592124640942,
+          -0.008185196667909622,
+          0.0147239463403821,
+          -0.017028972506523132,
+          0.004457169212400913,
+          0.012622765265405178,
+          -0.004778618458658457,
+          0.006311382632702589,
+          0.003381097922101617,
+          -0.008898657746613026,
+          0.0012073949910700321,
+          0.0013671396300196648,
+          0.015539329499006271,
+          0.015249241143465042,
+          0.007075804751366377,
+          0.022031037136912346,
+          0.0016395874554291368,
+          0.0009217166807502508,
+          0.01654287800192833,
+          0.015609892085194588,
+          0.0064721074886620045,
+          0.022078078240156174,
+          -0.015445247292518616,
+          -0.016668323427438736,
+          0.003367377445101738,
+          0.006895479746162891,
+          0.0025441537145525217,
+          0.0030243676155805588,
+          0.0036751064471900463,
+          0.010545105673372746,
+          -0.014394656755030155,
+          -0.006946441251784563,
+          -0.009322029538452625,
+          0.018973348662257195,
+          0.022689616307616234,
+          -0.020588435232639313,
+          0.0003922563628293574,
+          0.004958943463861942,
+          0.017091695219278336,
+          -0.001889494713395834,
+          0.006558349821716547,
+          0.01760914921760559,
+          -0.006574030499905348,
+          -0.008389042690396309,
+          -0.02316003106534481,
+          -0.012426760047674179,
+          -0.0006556389853358269,
+          0.019898496568202972,
+          -0.011407529935240746,
+          0.01286581251770258,
+          -0.015633411705493927,
+          -0.0042768437415361404,
+          0.029918305575847626,
+          0.011877943761646748,
+          0.005739046260714531,
+          -0.01330486498773098,
+          -0.011674097739160061,
+          -0.013673355802893639,
+          0.014324095100164413,
+          0.010317739099264145,
+          0.003122370457276702,
+          -0.010121732950210571,
+          -0.0017013292526826262,
+          -0.005515599623322487,
+          -0.008961379528045654,
+          -0.00685235857963562,
+          0.011462411843240261,
+          -0.0027617199812084436,
+          0.003949514124542475,
+          -0.011987706646323204,
+          0.0009903187165036798,
+          -0.010748950764536858,
+          -0.006495628040283918,
+          -0.0015415846137329936,
+          -0.03481060639023781,
+          -0.0299026258289814,
+          -0.001531784306280315,
+          -0.005107908044010401,
+          0.005652803461998701,
+          0.004123959224671125,
+          0.011611375957727432,
+          0.0029185244347900152,
+          0.00937691144645214,
+          0.009251467883586884,
+          0.011062560603022575,
+          3.1728417525300756e-05,
+          -0.004202361684292555,
+          0.011995547451078892,
+          -0.013602794148027897,
+          -0.016244949772953987,
+          0.024916240945458412,
+          0.004751177504658699,
+          0.013461669906973839,
+          -0.028397301211953163,
+          0.011783861555159092,
+          -0.002575514605268836,
+          0.010654868558049202,
+          -0.031141381710767746,
+          -0.0039769550785422325,
+          0.0020443391986191273,
+          -0.0033242562785744667,
+          -0.0101844547316432,
+          -0.028052331879734993,
+          -0.0075383782386779785,
+          -0.017812995240092278,
+          -0.01630767248570919,
+          -0.024759436026215553,
+          0.0050138249062001705,
+          -0.012034748680889606,
+          0.021341098472476006,
+          -0.010286377742886543,
+          0.03725675866007805,
+          0.006040894892066717,
+          -0.0015719655202701688,
+          0.01361063402146101,
+          0.0030478881672024727,
+          -0.0020090581383556128,
+          0.016636962071061134,
+          -0.006942520849406719,
+          -0.0013749798526987433,
+          0.017969800159335136,
+          0.01102335937321186,
+          0.003243893850594759,
+          -0.003602584358304739,
+          0.0029949666932225227,
+          0.026562688872218132,
+          0.01814228482544422,
+          0.00426900340244174,
+          0.005840969271957874,
+          0.002998886862769723,
+          0.016417434439063072,
+          0.009808124043047428,
+          -0.014778828248381615,
+          -0.008145995438098907,
+          0.011611375957727432,
+          -0.01173681952059269,
+          -0.006832757964730263,
+          0.004845260176807642,
+          -0.0097454022616148,
+          -0.023097308352589607,
+          0.012748208828270435,
+          0.006256501190364361,
+          -0.0005919371615163982,
+          -0.019239917397499084,
+          -0.012920694425702095,
+          0.0017620910657569766,
+          0.01225427445024252,
+          0.016323352232575417,
+          -0.01914583332836628,
+          0.002738199196755886,
+          0.008153836242854595,
+          -0.006417225580662489,
+          -0.011580015532672405,
+          0.016950570046901703,
+          0.022313285619020462,
+          0.014888591133058071,
+          -0.04788810759782791,
+          -0.008945698849856853,
+          0.017546428367495537,
+          -0.012183712795376778,
+          -0.043967992067337036,
+          -0.0015455047832801938,
+          -0.0018395131919533014,
+          -0.025982512161135674,
+          0.0052490318194031715,
+          0.006182019133120775,
+          -0.014700425788760185,
+          0.016527198255062103,
+          0.03200380504131317,
+          0.00019159555085934699,
+          0.009102503769099712,
+          -0.011109601706266403,
+          -0.014426017180085182,
+          -0.009047621861100197,
+          0.008726172149181366,
+          -0.01175250019878149,
+          -0.009847325272858143,
+          0.00248927203938365,
+          -0.015555010177195072,
+          0.01848725602030754,
+          0.007569739129394293,
+          0.03150203078985214,
+          0.01132912840694189,
+          -0.00248927203938365,
+          -0.01760914921760559,
+          -0.020964767783880234,
+          -0.009533716365695,
+          0.003741748398169875,
+          0.007997031323611736,
+          0.007005242630839348,
+          0.010160934180021286,
+          0.013328385539352894,
+          0.022673936560750008,
+          -0.006252580787986517,
+          0.014825869351625443,
+          -3.0059307391638868e-05,
+          -0.0014680825406685472,
+          0.0025676742661744356,
+          -0.020149383693933487,
+          0.005143188871443272,
+          0.0018228527624160051,
+          -0.004904062021523714,
+          -0.007013082969933748,
+          -0.003083169227465987,
+          0.032066527754068375,
+          -0.016480157151818275,
+          0.015970543026924133,
+          -0.031078658998012543,
+          -0.009337710216641426,
+          0.001246596104465425,
+          0.02814641408622265,
+          0.012701167725026608,
+          -0.02841298282146454,
+          -0.011297767050564289,
+          -0.026515647768974304,
+          0.02278370037674904,
+          -0.014300573617219925,
+          0.004723736550658941,
+          0.015845099464058876,
+          0.01444953866302967,
+          0.01194850541651249,
+          -0.01165841706097126,
+          -0.00878889486193657,
+          0.011776020750403404,
+          0.021999677643179893,
+          0.007679502479732037,
+          0.0022266244050115347,
+          -0.0009261267841793597,
+          -0.001644487609155476,
+          -0.004304284695535898,
+          -0.017985479906201363,
+          0.006711234338581562,
+          -0.00819303747266531,
+          0.027127185836434364,
+          -0.007597180083394051,
+          0.016746724024415016,
+          -0.0062957024201750755,
+          -0.00582920853048563,
+          -0.02932244911789894,
+          -0.0203689094632864,
+          0.007605019956827164,
+          -0.006703393999487162,
+          0.012959894724190235,
+          -0.019365360960364342,
+          -0.0164958368986845,
+          0.010733271017670631,
+          -0.02010234259068966,
+          -0.002083540428429842,
+          -0.004719816613942385,
+          0.0022442650515586138,
+          0.00676611578091979,
+          0.00736197317019105,
+          -0.012826611287891865,
+          -0.006883719470351934,
+          0.016072465106844902,
+          -0.007663821801543236,
+          0.007640301249921322,
+          0.029871264472603798,
+          0.01768755167722702,
+          0.026844937354326248,
+          0.007048364263027906,
+          0.013751758262515068,
+          -0.010364780202507973,
+          -0.017546428367495537,
+          -0.023301154375076294,
+          0.01278741005808115,
+          -0.0025519938208162785,
+          0.03368161618709564,
+          0.00478645833209157,
+          -0.007820626720786095,
+          -0.0031027698423713446,
+          0.005417597014456987,
+          0.0008408643188886344,
+          -0.01414376962929964,
+          0.0029518455266952515,
+          -0.00551167968660593,
+          0.012826611287891865,
+          0.016033263877034187,
+          -0.01883222535252571,
+          0.02522985078394413,
+          -0.016856487840414047,
+          0.012779570184648037,
+          -0.003602584358304739,
+          0.01630767248570919,
+          0.009753242135047913,
+          -0.0003626105026341975,
+          -0.0015415846137329936,
+          -0.05394076183438301,
+          -0.010458862408995628,
+          0.0038456313777714968,
+          -0.004970703739672899,
+          0.000352075177943334,
+          -0.012638445943593979,
+          -0.038573917001485825,
+          0.0034810607321560383,
+          0.013391107320785522,
+          -0.011556494981050491,
+          0.013422468677163124,
+          -0.002016898477450013,
+          0.006617151666432619,
+          -0.007922549732029438,
+          -0.006260421127080917,
+          0.016480157151818275,
+          0.02573162503540516,
+          -0.021121570840477943,
+          -0.022344646975398064,
+          0.005778247490525246,
+          -0.005543040577322245,
+          -0.014684745110571384,
+          -0.005021665245294571,
+          -0.01598622277379036,
+          0.0025069124531000853,
+          0.023567723110318184,
+          -0.01426137238740921,
+          -0.01776595413684845,
+          0.010145253501832485,
+          0.007757904473692179,
+          -0.024853520095348358,
+          -0.002355988137423992,
+          -0.009815963916480541,
+          -0.010803832672536373,
+          0.018393171951174736,
+          -0.01362631469964981,
+          0.008906497620046139,
+          -0.013124539516866207,
+          0.010396140627563,
+          -0.02825617790222168,
+          -0.01226995512843132,
+          -0.0019502564100548625,
+          -0.008882977068424225,
+          0.028083693236112595,
+          -0.010772472247481346,
+          -0.010200135409832,
+          0.020070981234312057,
+          -0.008663450367748737,
+          0.012567883357405663,
+          0.007855907082557678,
+          0.028648190200328827,
+          -0.0025853149127215147,
+          0.009141704998910427,
+          0.0028930436819791794,
+          0.01390072237700224,
+          0.0011936745140701532,
+          0.035437826067209244,
+          -0.00957291666418314,
+          -0.00030307378619909286,
+          -0.008835935965180397,
+          -0.0017140696290880442,
+          0.0010427501983940601,
+          0.010991998016834259,
+          0.014151609502732754,
+          0.012693327851593494,
+          -0.006244740914553404,
+          -0.0077853454276919365,
+          -0.015500128269195557,
+          -0.004527730867266655,
+          -0.004888381343334913,
+          0.007428614888340235,
+          0.012442439794540405,
+          -0.020039619877934456,
+          0.005358795169740915,
+          -0.012144511565566063,
+          0.0011946546146646142,
+          -0.01587645895779133,
+          -0.0022560253273695707,
+          -0.013634154573082924,
+          -0.004080838058143854,
+          -0.004331725183874369,
+          0.003790749702602625,
+          0.008749693632125854,
+          -0.016401754692196846,
+          0.0199455376714468,
+          0.008153836242854595,
+          -0.0004074467869941145,
+          -0.0028107212856411934,
+          -0.024931922554969788,
+          -0.014010485261678696,
+          0.00014859679504297674,
+          0.016668323427438736,
+          0.0050138249062001705,
+          -0.0051784696988761425,
+          0.002026698552072048,
+          -0.012826611287891865,
+          0.005280392710119486,
+          -0.0009040761506184936,
+          0.0022285846062004566,
+          -0.0030067269690334797,
+          0.0057312059216201305,
+          0.006762195844203234,
+          0.003700587200000882,
+          0.012959894724190235,
+          0.01487291045486927,
+          -0.011760340072214603,
+          -0.00966699980199337,
+          0.012285635806620121,
+          0.015217880718410015,
+          -0.008914338424801826,
+          0.01941240206360817,
+          -0.02878931351006031,
+          0.002820521593093872,
+          0.005233351606875658,
+          -0.0042533231899142265,
+          -0.0021854632068425417,
+          7.748349162284285e-05,
+          0.028475703671574593,
+          -0.0010221695993095636,
+          -0.010388300754129887,
+          -0.0009854185627773404,
+          0.01707601360976696,
+          -0.0006600490887649357,
+          0.006123217288404703,
+          0.023724526166915894,
+          0.00023790188424754888,
+          0.009768922813236713,
+          0.009776762686669827,
+          -0.019506484270095825,
+          -0.003145891008898616,
+          0.004633574280887842,
+          -0.012442439794540405,
+          0.018393171951174736,
+          0.014841550029814243,
+          0.008389042690396309,
+          0.0038005500100553036,
+          0.007510937284678221,
+          0.020086660981178284,
+          0.006374104414135218,
+          -0.002336387522518635,
+          -0.0002511322672944516,
+          0.0066445921547710896,
+          -0.011721138842403889,
+          0.017405303195118904,
+          -0.004363086074590683,
+          -0.015437406487762928,
+          0.006679873447865248,
+          0.011674097739160061,
+          -0.006773956120014191,
+          -0.004033796954900026,
+          0.0022540653590112925,
+          0.005711605306714773,
+          0.005464638117700815,
+          -0.003888752544298768,
+          -0.006574030499905348,
+          0.0012505161575973034,
+          0.017812995240092278,
+          0.010639187879860401,
+          0.016433116048574448,
+          -0.014990514144301414,
+          -0.011697618290781975,
+          0.0023050266318023205,
+          -0.018016841262578964,
+          0.002996926661580801,
+          -0.011517293751239777,
+          0.006299622356891632,
+          -0.007420775014907122,
+          0.02568458393216133,
+          0.019616248086094856,
+          0.007573659066110849,
+          0.011932825669646263,
+          -0.015539329499006271,
+          0.016715364530682564,
+          -0.00913386419415474,
+          0.012120991013944149,
+          -0.018847905099391937,
+          -0.0017346502281725407,
+          0.005296073388308287,
+          0.027628960087895393,
+          0.010302058421075344,
+          0.006064415443688631,
+          -0.001778751495294273,
+          -0.005182390101253986,
+          -0.008318481035530567,
+          -0.009619958698749542,
+          -0.014755306765437126,
+          -0.026233399286866188,
+          -0.00022810160589870065,
+          0.016339033842086792,
+          -0.00195907661691308,
+          0.0022521051578223705,
+          0.014590661972761154,
+          0.013187261298298836,
+          -0.0024128297809511423,
+          -0.013908562250435352,
+          -0.007252209819853306,
+          0.010835194028913975,
+          0.005119668319821358,
+          -0.008357682265341282,
+          -0.009141704998910427,
+          0.008138155564665794,
+          -0.013273504562675953,
+          0.016433116048574448,
+          -0.016182228922843933,
+          -0.009949248284101486,
+          -0.013649835251271725,
+          -0.0005929171456955373,
+          -0.008263599127531052,
+          -0.031454991549253464,
+          0.012152351438999176,
+          0.005707685369998217,
+          -0.0006365284207277,
+          -0.025715945288538933,
+          0.02369316667318344,
+          -0.008569368161261082,
+          0.03656681999564171,
+          -0.01593918167054653,
+          0.015633411705493927,
+          0.0010623506968840957,
+          0.011493772268295288,
+          -0.003392858197912574,
+          0.0029479253571480513,
+          0.007953910157084465,
+          0.002550033852458,
+          0.003055728506296873,
+          -0.00539799639955163,
+          -0.0033066156320273876,
+          0.008898657746613026,
+          -0.006448586471378803,
+          0.008490965701639652,
+          -0.014849389903247356,
+          -0.02320707216858864,
+          -0.01144673116505146,
+          0.009502355009317398,
+          0.014104568399488926,
+          0.00861640926450491,
+          -0.014473059214651585,
+          0.00882809516042471,
+          -0.02435174584388733,
+          -0.0023461878299713135,
+          -0.0031282505951821804,
+          0.004782538395375013,
+          -0.017201457172632217,
+          0.016605600714683533,
+          0.02193695493042469,
+          -3.518914309097454e-05,
+          -0.0007879427867010236,
+          0.01382231991738081,
+          0.010160934180021286,
+          0.009423952549695969,
+          -0.010011970065534115,
+          0.0017826715484261513,
+          -8.403743413509801e-05,
+          0.02044731192290783,
+          -0.002397149335592985,
+          -0.009831644594669342,
+          -0.0035379023756831884,
+          -0.01399480551481247,
+          0.009698361158370972,
+          -0.029275408014655113,
+          -0.0052490318194031715,
+          -0.006523068994283676,
+          0.017170097678899765,
+          0.00684451824054122,
+          0.021639026701450348,
+          0.0065622697584331036,
+          0.005523439962416887,
+          -8.844755939207971e-05,
+          -0.008240078575909138,
+          0.012097470462322235,
+          -0.00809895433485508,
+          -0.007201248314231634,
+          0.006013453938066959,
+          0.004492450039833784,
+          0.016856487840414047,
+          0.008067593909800053,
+          -0.009180906228721142,
+          0.030106471851468086,
+          0.014904271811246872,
+          -0.013939923606812954,
+          0.01814228482544422,
+          -0.01826772838830948,
+          0.020776601508259773,
+          0.01853429712355137,
+          0.010239336639642715,
+          -0.015304123051464558,
+          -0.015680454671382904,
+          0.01784435659646988,
+          -0.010466703213751316,
+          -0.004657094832509756,
+          0.0018052122322842479,
+          -0.017938438802957535,
+          8.092890311672818e-06,
+          -0.011972026899456978,
+          0.015429566614329815,
+          -0.0021991836838424206,
+          -0.00012042098387610167,
+          -0.020776601508259773,
+          -0.00043831768562085927,
+          0.016903528943657875,
+          0.01615086756646633,
+          -0.0033340565860271454,
+          -0.04478337615728378,
+          -0.006178098730742931,
+          0.009855165146291256,
+          -0.00706404447555542,
+          -0.025982512161135674,
+          0.01803252287209034,
+          0.0034104986116290092,
+          -0.019428081810474396,
+          -0.01184658333659172,
+          -0.018502935767173767,
+          -0.019396722316741943,
+          -0.008820255286991596,
+          -0.004563012160360813,
+          -0.015272761695086956,
+          -0.003884832374751568,
+          0.008224397897720337,
+          -0.007213008590042591,
+          -0.006064415443688631,
+          0.004061237443238497,
+          -0.007354132831096649,
+          -0.0003518301818985492,
+          -0.010866554453969002,
+          -0.005629282910376787,
+          0.011133122257888317,
+          0.002452031010761857,
+          -0.01508459635078907,
+          -0.015366844832897186,
+          -0.011564334854483604,
+          -0.010537264868617058,
+          0.004139639902859926,
+          -0.009212266653776169,
+          -0.005080467090010643,
+          -0.020149383693933487,
+          0.0006615191232413054,
+          0.013469509780406952,
+          0.0104118213057518,
+          0.0006154578295536339,
+          0.004931502975523472,
+          0.022313285619020462,
+          -0.006189859006553888,
+          -0.0006242780946195126,
+          -9.363864592160098e-06,
+          -0.002955765463411808,
+          0.02687629871070385,
+          0.004312125034630299,
+          0.01286581251770258,
+          -0.005562641192227602,
+          0.009494515135884285,
+          -0.01975737139582634,
+          -0.0023755887523293495,
+          -0.010850873775780201,
+          -0.019773053005337715,
+          -0.011979866772890091,
+          -5.2125258662272245e-05,
+          0.004743337165564299,
+          -0.013069658540189266,
+          0.0021541023161262274,
+          -0.00467277504503727,
+          0.006734754890203476,
+          -0.01508459635078907,
+          -0.005782167427241802,
+          -0.007134606596082449,
+          -0.010662708431482315,
+          -0.02416357956826687,
+          0.018283409997820854,
+          0.02889907732605934,
+          0.001601366326212883,
+          -0.007554058451205492,
+          -0.01391640305519104,
+          -0.008177356794476509,
+          -0.004041636828333139,
+          -0.018048202618956566,
+          -0.008702651597559452,
+          -0.027926888316869736,
+          -0.013641994446516037,
+          -0.0003746158326976001,
+          0.010772472247481346,
+          0.0036065042950212955,
+          -0.018126605078577995,
+          0.005107908044010401,
+          -0.025527779012918472,
+          -0.025668902322649956,
+          -0.019428081810474396,
+          -0.017295541241765022,
+          0.0025676742661744356,
+          0.003888752544298768,
+          0.0021149013191461563,
+          0.017358262091875076,
+          0.020227786153554916,
+          0.004813899286091328,
+          -0.002424590289592743,
+          0.008592888712882996,
+          -0.014974833466112614,
+          0.0046766954474151134,
+          0.0023305073846131563,
+          -0.013422468677163124,
+          0.02358340285718441,
+          -0.011917144991457462,
+          -0.00353594240732491,
+          0.0008570348145440221,
+          0.0034575401805341244,
+          -0.028193457052111626,
+          -0.002152142347767949,
+          -0.0164958368986845,
+          0.02270529791712761,
+          -0.029651738703250885,
+          -0.0021443020086735487,
+          -0.00416316045448184,
+          -0.007895108312368393,
+          0.026578368619084358,
+          0.021356778219342232,
+          -0.0004645334556698799,
+          0.008984900079667568,
+          0.010709750466048717,
+          -0.0013063778169453144,
+          0.01869110018014908,
+          -0.017483705654740334,
+          -0.014151609502732754,
+          0.022234883159399033,
+          0.008757533505558968,
+          0.0036163046024739742,
+          -0.011242885142564774,
+          0.017562108114361763,
+          0.017468025907874107,
+          0.0059193712659180164,
+          -0.011485932394862175,
+          -0.016244949772953987,
+          0.0007933329907245934,
+          0.019773053005337715,
+          -0.012403239496052265,
+          0.0027440793346613646,
+          -0.010631348006427288,
+          0.008091114461421967,
+          -0.014708265662193298,
+          0.017750274389982224,
+          -0.027503516525030136,
+          -0.005060866475105286,
+          -0.0016092065488919616,
+          0.01795412041246891,
+          -0.008012712001800537,
+          0.015711814165115356,
+          -0.00696604186668992,
+          -0.009470994584262371,
+          0.014888591133058071,
+          -0.0005556760588660836,
+          -0.004739417228847742,
+          0.022909143939614296,
+          0.00653874920681119,
+          -0.0009614078444428742,
+          0.004884461406618357,
+          -0.018424533307552338,
+          0.025010323151946068,
+          -0.010074691846966743,
+          0.003000846831128001,
+          0.0012524762423709035,
+          0.0013710596831515431,
+          0.026844937354326248,
+          0.02634316310286522,
+          0.0057429661974310875,
+          0.008585048839449883,
+          -0.01351655088365078,
+          0.02519848942756653,
+          0.00467277504503727,
+          0.0012720768572762609,
+          -0.0009893386159092188,
+          -0.0024167499504983425,
+          -0.0017454305198043585,
+          -0.009180906228721142,
+          0.015743175521492958,
+          -0.01213667169213295,
+          -0.025935471057891846,
+          -0.009322029538452625,
+          -0.004492450039833784,
+          -0.017107374966144562,
+          0.020055299624800682,
+          0.0004699236014857888,
+          -0.003992635756731033,
+          0.011681938543915749,
+          0.007428614888340235,
+          0.0032360537443310022,
+          -0.008655610494315624,
+          0.0038671919610351324,
+          -0.007130686193704605,
+          -0.013046137988567352,
+          0.012450280599296093,
+          -0.019208556041121483,
+          -0.008984900079667568,
+          -0.008553687483072281,
+          0.009063302539288998,
+          0.010443182662129402,
+          0.00928282830864191,
+          0.015656933188438416,
+          0.004441488534212112,
+          0.00819303747266531,
+          0.012489481829106808,
+          -0.024931922554969788,
+          0.01707601360976696,
+          0.021764470264315605,
+          -0.0005096147651784122,
+          0.001671928446739912,
+          0.00045350813888944685,
+          -0.03446563705801964,
+          0.00027293790481053293,
+          0.0014034006744623184,
+          0.0077853454276919365,
+          -0.013406787998974323,
+          0.0022677856031805277,
+          0.022673936560750008,
+          -0.011109601706266403,
+          0.020996127277612686,
+          0.005970332771539688,
+          0.002479471731930971,
+          0.03343072906136513,
+          0.012230753898620605,
+          0.016339033842086792,
+          -0.0068915593437850475,
+          -0.026625411584973335,
+          0.0054371971637010574,
+          -0.018518615514039993,
+          0.009714040905237198,
+          0.0014386816183105111,
+          0.013148060999810696,
+          -0.01792275905609131,
+          -0.006025214213877916,
+          0.0027715202886611223,
+          -0.011039039120078087,
+          0.01654287800192833,
+          0.0014043806586414576,
+          0.019631927832961082,
+          0.001000609016045928,
+          -0.0028264017309993505,
+          -0.012967735528945923,
+          -0.01404968649148941,
+          0.0029773262795060873,
+          -0.009659159928560257,
+          -0.01699761115014553,
+          0.004966783802956343,
+          -0.008342001587152481,
+          0.007091485429555178,
+          -0.01587645895779133,
+          0.02044731192290783,
+          0.0005806668195873499,
+          0.00853800680488348,
+          0.015523649752140045,
+          0.0010319699067622423,
+          0.017750274389982224,
+          0.010341259650886059,
+          -0.0075501385144889355,
+          0.0062957024201750755,
+          0.0002594625111669302,
+          0.0029636058025062084,
+          0.0017895317869260907,
+          -0.01219155266880989,
+          0.0058135283179581165,
+          -0.004159240517765284,
+          0.00015165939112193882,
+          -0.006479947362095118,
+          -0.004092598333954811,
+          0.0034046184737235308,
+          -0.013406787998974323,
+          -0.00539015606045723,
+          0.002242304850369692,
+          0.001080971327610314,
+          0.02148222178220749,
+          -0.0017905118875205517,
+          0.00878889486193657,
+          0.02228192426264286,
+          -0.015570690855383873,
+          0.006703393999487162,
+          0.010059011168777943,
+          -0.004892301745712757,
+          -0.011156642809510231,
+          0.004661014769226313,
+          0.016166547313332558,
+          -0.01587645895779133,
+          -0.01131344772875309,
+          0.011501613073050976,
+          -0.021921275183558464,
+          0.004390527028590441,
+          -0.003963234834372997,
+          -0.007338452618569136,
+          0.02143518067896366,
+          -0.0012436560355126858,
+          0.032333094626665115,
+          -0.004770778119564056,
+          -0.015460927970707417,
+          -0.01930263824760914,
+          0.0012554163113236427,
+          -0.025512099266052246,
+          -0.005637123249471188,
+          -0.009188746102154255,
+          -0.006848438177257776,
+          0.00958859734237194,
+          0.0006865098839625716,
+          0.004919742234051228,
+          0.005950732156634331,
+          -0.004390527028590441,
+          0.004012236371636391,
+          -0.0031400108709931374,
+          0.007883348502218723,
+          -0.02143518067896366,
+          0.018628379330039024,
+          0.014120249077677727,
+          0.0005228451336733997,
+          -0.019741691648960114,
+          -0.008945698849856853,
+          0.020086660981178284,
+          0.04537923261523247,
+          -0.023222751915454865,
+          -0.0033085758332163095,
+          0.018330451101064682,
+          0.0038691519293934107,
+          0.0014475019415840507,
+          -0.00924362801015377,
+          -0.005570481065660715,
+          -0.04202361777424812,
+          0.0016415475402027369,
+          0.001603326410986483,
+          -0.009306349791586399,
+          0.007048364263027906,
+          -0.017812995240092278,
+          -0.007275730837136507,
+          -0.012638445943593979,
+          -0.020274827256798744,
+          -0.004610053263604641,
+          -0.010341259650886059,
+          -0.006503468379378319,
+          -0.011415370739996433,
+          0.013046137988567352,
+          -0.005774327088147402,
+          0.010082531720399857,
+          -0.004104358609765768,
+          0.01768755167722702,
+          -0.0013240183470770717,
+          0.012544362805783749,
+          -0.0005821368540637195,
+          0.009768922813236713,
+          0.00665243249386549,
+          -0.006789636332541704,
+          -0.008255759254097939,
+          -0.007667741738259792,
+          0.018659740686416626,
+          0.009768922813236713,
+          0.010498063638806343,
+          0.0014337814645841718,
+          -0.012552203610539436,
+          -0.011674097739160061,
+          5.987360782455653e-05,
+          -0.010333418846130371,
+          0.0006835698150098324,
+          0.006742595229297876,
+          0.026970380917191505,
+          -0.0033340565860271454,
+          -0.025527779012918472,
+          0.004649254493415356,
+          0.009847325272858143,
+          0.020917724817991257,
+          -0.013148060999810696,
+          0.007354132831096649,
+          -0.010498063638806343,
+          0.013108859769999981,
+          0.026311801746487617,
+          -0.009212266653776169,
+          0.018283409997820854,
+          -0.006205539684742689,
+          0.013947763480246067,
+          0.006836677901446819,
+          -0.00416316045448184,
+          -0.02963605709373951,
+          -0.008302800357341766,
+          0.0016131267184391618,
+          0.0036319850478321314,
+          -0.009369071573019028,
+          0.030874812975525856,
+          0.010482383891940117,
+          0.016088144853711128,
+          -0.009580757468938828,
+          0.010403981432318687,
+          0.00539015606045723,
+          0.019694650545716286,
+          0.012238594703376293,
+          0.00021413620561361313,
+          0.0019806374330073595,
+          -0.008498805575072765,
+          -0.045504678040742874,
+          0.00957291666418314,
+          0.006668113172054291,
+          0.02538665570318699,
+          -0.04196089506149292,
+          0.01569613441824913,
+          -0.010200135409832,
+          -0.0059899333864450455,
+          -0.003700587200000882,
+          0.007667741738259792,
+          -0.003696667030453682,
+          0.0290245208889246,
+          -0.005790007766336203,
+          0.0019355560652911663,
+          0.016684003174304962,
+          0.014073207043111324,
+          -0.007683422416448593,
+          0.01567261293530464,
+          0.023018905892968178,
+          0.01898903027176857,
+          0.013979124836623669,
+          -0.00748349679633975,
+          0.025621861219406128,
+          0.0006811197381466627,
+          -0.003218413097783923,
+          -0.008240078575909138,
+          -0.016166547313332558,
+          0.0024128297809511423,
+          -0.013116699643433094,
+          0.003659426001831889,
+          -0.00561752263456583,
+          -0.004633574280887842,
+          0.010905755683779716,
+          -0.010991998016834259,
+          0.0017454305198043585,
+          0.0018081523012369871,
+          0.007040523923933506,
+          0.0003800060076173395,
+          0.02703310176730156,
+          -0.032521262764930725,
+          0.014081047847867012,
+          -0.03258398175239563,
+          -0.01657423935830593,
+          0.006942520849406719,
+          -0.0044179679825901985,
+          0.00687195872887969,
+          -0.0195221658796072,
+          -0.017358262091875076,
+          0.00591545132920146,
+          0.01829908974468708,
+          0.01502971537411213,
+          0.00716988742351532,
+          0.006091856397688389,
+          0.004884461406618357,
+          -0.010059011168777943,
+          -0.006299622356891632,
+          -0.009557236917316914,
+          -0.00884377583861351,
+          -0.005049106199294329,
+          -0.02010234259068966,
+          0.015453087165951729,
+          -0.0070562041364610195,
+          -0.007922549732029438,
+          0.016746724024415016,
+          -0.01026285719126463,
+          -0.018424533307552338,
+          0.005539120174944401,
+          -0.0038926727138459682,
+          -0.005025585647672415,
+          -0.016213588416576385,
+          0.00922794733196497,
+          -0.0032752547413110733,
+          0.0039240336045622826,
+          -0.01309317909181118,
+          0.026672452688217163,
+          -0.006785716395825148,
+          0.001506303553469479,
+          0.011924984864890575,
+          -0.009408272802829742,
+          -0.024853520095348358,
+          -0.0021991836838424206,
+          -0.014896431006491184,
+          0.0027323190588504076,
+          -0.0028773632366210222,
+          -0.01673104427754879,
+          -0.008443924598395824,
+          0.0035967042203992605,
+          -0.02698606066405773,
+          -0.011384009383618832,
+          0.0099570881575346,
+          0.011783861555159092,
+          0.009431793354451656,
+          0.011932825669646263,
+          -0.0021011808421462774,
+          0.013148060999810696,
+          -0.029008839279413223,
+          -0.014496579766273499,
+          -0.01362631469964981,
+          -0.009549396112561226,
+          0.01412808895111084,
+          0.0017954119248315692,
+          0.007416854612529278,
+          -0.004480689764022827,
+          -0.005252952221781015,
+          -0.0011201724410057068,
+          0.011776020750403404,
+          0.028804993256926537,
+          0.0003814760420937091,
+          -0.003884832374751568,
+          -0.001314218039624393,
+          -0.011423210613429546,
+          0.028381621465086937,
+          -0.0041121989488601685,
+          -0.0072090886533260345,
+          0.03449700027704239,
+          0.01933399960398674,
+          0.027362391352653503,
+          -0.0001465142413508147,
+          -0.006025214213877916,
+          -0.0005091247148811817,
+          0.004931502975523472,
+          0.004974624142050743,
+          -0.01226995512843132,
+          -0.00842040404677391,
+          -0.004751177504658699,
+          0.03465380147099495,
+          0.0022952265571802855,
+          0.011133122257888317,
+          -0.0307493694126606,
+          0.015225720591843128,
+          -0.003577103605493903,
+          -0.007546218577772379,
+          0.003269374603405595,
+          0.036692261695861816,
+          0.02576298639178276,
+          0.0069738817401230335,
+          0.009447473101317883,
+          8.630374941276386e-05,
+          0.0398910753428936,
+          -0.02438310533761978,
+          -0.020196424797177315,
+          -0.02273665741086006,
+          -0.021717429161071777,
+          0.0016601680545136333,
+          0.007271810434758663,
+          0.003628065111115575,
+          -0.0009770882315933704,
+          0.0015259041683748364,
+          0.028585467487573624,
+          0.0015788257587701082,
+          -0.010341259650886059,
+          -0.0014308413956314325,
+          -0.034089308232069016,
+          0.013618473894894123,
+          -0.017389623448252678,
+          -0.010200135409832,
+          -0.016511518508195877,
+          -0.013006936758756638,
+          -0.025982512161135674,
+          -0.010607827454805374,
+          -0.0009599378099665046,
+          0.015970543026924133,
+          0.010004129260778427,
+          -0.0019120353972539306,
+          0.005613602697849274,
+          -0.023834289982914925,
+          -0.006703393999487162,
+          0.020917724817991257,
+          0.01567261293530464,
+          0.015790216624736786,
+          0.0046531748957931995,
+          0.01622927002608776,
+          -0.0003020937438122928,
+          -0.015923500061035156,
+          -0.0020913805346935987,
+          0.002767600119113922,
+          0.006413305643945932,
+          0.017091695219278336,
+          0.012222914025187492,
+          -0.018769502639770508,
+          0.0071267662569880486,
+          -0.012795249931514263,
+          0.03139226883649826,
+          -0.006197699345648289,
+          0.01205826923251152,
+          0.009682680480182171,
+          0.0032830950804054737,
+          -0.012489481829106808,
+          -0.006158498115837574,
+          -0.00479429867118597,
+          0.01569613441824913,
+          -0.00831064023077488,
+          -0.011062560603022575,
+          -0.005676324479281902,
+          0.009729721583425999,
+          -0.005535200238227844,
+          -0.0010349099757149816,
+          -0.015609892085194588,
+          -0.014347615651786327,
+          0.0030753291212022305,
+          -0.012826611287891865,
+          -0.0010711710201576352,
+          0.019036071375012398,
+          -0.005033425521105528,
+          -0.0004390527028590441,
+          0.014684745110571384,
+          0.016589920967817307,
+          0.004649254493415356,
+          0.018926307559013367,
+          -0.009549396112561226,
+          0.0022932663559913635,
+          -0.021623345091938972,
+          -0.0024716316256672144,
+          -0.032364457845687866,
+          0.019647609442472458,
+          0.0017836516490206122,
+          -0.020070981234312057,
+          0.010944956913590431,
+          -0.0016239070100709796,
+          -0.009165225550532341,
+          -0.0099727688357234,
+          0.002452031010761857,
+          -0.024179259315133095,
+          0.024147899821400642,
+          -0.016048945486545563,
+          0.008906497620046139,
+          0.01161921676248312,
+          0.041333675384521484,
+          0.0038573916535824537,
+          -0.00747565645724535,
+          0.021152932196855545,
+          -0.0002717128663789481,
+          0.02982422336935997,
+          -0.001121152425184846,
+          -0.012081789784133434,
+          -0.022689616307616234,
+          -0.005237271543592215,
+          -0.015202200040221214,
+          -0.014112408272922039,
+          0.0147239463403821,
+          -0.0195221658796072,
+          -0.015217880718410015,
+          -0.010004129260778427,
+          -0.019741691648960114,
+          -0.011509452946484089,
+          0.025982512161135674,
+          0.015758855268359184,
+          -0.011705459095537663,
+          -0.006425065919756889,
+          -0.0055077592842280865,
+          -0.02711150422692299,
+          0.021952634677290916,
+          0.008012712001800537,
+          -0.01983577385544777,
+          -0.0032772149424999952,
+          -0.009768922813236713,
+          0.004378766752779484,
+          0.01588430069386959,
+          -0.0001214010117109865,
+          0.05789223685860634,
+          -0.008804574608802795,
+          -0.0015945062041282654,
+          -0.020008258521556854,
+          0.009298508986830711,
+          -0.01575101539492607,
+          -0.006374104414135218,
+          -0.026672452688217163,
+          -0.008255759254097939,
+          -0.005119668319821358,
+          -0.021105891093611717,
+          -0.0022481849882751703,
+          -0.008287119679152966,
+          0.0057547264732420444,
+          -0.01280309073626995,
+          0.0008050933247432113,
+          -0.01665264181792736,
+          -0.00624866085126996,
+          -0.012089629657566547,
+          -0.012371878139674664,
+          0.01925559714436531,
+          -0.028303219005465508,
+          -0.0033869780600070953,
+          0.008153836242854595,
+          0.01353223156183958,
+          0.006268261466175318,
+          -0.021043168380856514,
+          -0.012614925391972065,
+          -0.011917144991457462,
+          -0.018894946202635765,
+          0.010733271017670631,
+          -0.010741110891103745,
+          0.013140220195055008,
+          -0.03186268359422684,
+          -0.012607084587216377,
+          -0.0019522164948284626,
+          0.019976899027824402,
+          0.01753074675798416,
+          -0.0004733536916319281,
+          -0.0059193712659180164,
+          -0.014002645388245583,
+          0.02278370037674904,
+          -0.013877201825380325,
+          -0.032678067684173584,
+          -0.0013494990998879075,
+          0.002218784298747778,
+          -0.00104079011362046,
+          -0.007816705852746964,
+          0.0027538796421140432,
+          0.004214121960103512,
+          0.004092598333954811,
+          -0.007546218577772379,
+          -0.0016464476939290762,
+          -0.01508459635078907,
+          0.0047668581828475,
+          -3.659793583210558e-05,
+          0.029730141162872314,
+          -0.0035339822061359882,
+          0.008992739953100681,
+          -0.02182719111442566,
+          0.0007546218694187701,
+          -0.01317158155143261,
+          -0.005782167427241802,
+          0.000176037588971667,
+          0.003330136416479945,
+          0.01956920698285103,
+          -0.025794345885515213,
+          -0.013038297183811665,
+          0.01205826923251152,
+          -0.019929856061935425,
+          -0.03572007268667221,
+          0.002736239228397608,
+          -0.029463572427630424,
+          -0.0013053978327661753,
+          -0.003777029225602746,
+          -0.015703974291682243,
+          -0.026217719539999962,
+          0.0060722557827830315,
+          -0.005366635508835316,
+          -0.005786087363958359,
+          0.003516341792419553,
+          -0.015233561396598816,
+          -0.015531489625573158,
+          0.005151029210537672,
+          0.007346292492002249,
+          -0.006503468379378319,
+          0.008561528287827969,
+          -0.0238186102360487,
+          0.011274246498942375,
+          -0.008145995438098907,
+          0.0037358682602643967,
+          0.005135348532348871,
+          0.026123635470867157,
+          0.008561528287827969,
+          -0.004000475630164146,
+          0.009306349791586399,
+          0.005472478456795216,
+          -0.010333418846130371,
+          0.011070400476455688,
+          0.02400677464902401,
+          0.0005184350302442908,
+          0.0032340935431420803,
+          -0.002973406109958887,
+          0.005746886134147644,
+          -0.001685648807324469,
+          -0.007612860295921564,
+          0.020933406427502632,
+          0.01593918167054653,
+          -0.019239917397499084,
+          0.0062369005754590034,
+          -0.020039619877934456,
+          0.010976317338645458,
+          -0.028742272406816483,
+          0.0005703765200451016,
+          0.005840969271957874,
+          -0.0050020646303892136,
+          0.011979866772890091,
+          0.006860198453068733,
+          0.03151771053671837,
+          0.026280440390110016,
+          -0.0018875346286222339,
+          0.0048178196884691715,
+          -0.02002394013106823,
+          0.01280309073626995,
+          0.009118183515965939,
+          0.010066851042211056,
+          -0.010694069787859917,
+          0.00791470892727375,
+          -0.00878889486193657,
+          0.014606342650949955,
+          -0.00260295532643795,
+          -0.014214331284165382,
+          0.006350583862513304,
+          -0.006287862081080675,
+          0.014441697858273983,
+          0.014081047847867012,
+          -0.013148060999810696,
+          -0.026923339813947678,
+          -0.006162418518215418,
+          0.018095243722200394,
+          -0.023912692442536354,
+          0.02443014644086361,
+          0.006464267149567604,
+          0.02265825681388378,
+          -0.009988449513912201,
+          -0.0030655288137495518,
+          -0.001984557369723916,
+          0.002657837001606822,
+          -0.00685235857963562,
+          0.00014345165982376784,
+          0.02243872918188572,
+          -0.009510194882750511,
+          0.005601841956377029,
+          0.012175872921943665,
+          -0.0016052864957600832,
+          0.03622184693813324,
+          0.018769502639770508,
+          0.024978963658213615,
+          -0.0017366103129461408,
+          0.010584305971860886,
+          0.0008266539080068469,
+          0.010537264868617058,
+          0.013845840469002724,
+          -0.010984158143401146,
+          0.0008913357742130756,
+          -0.014347615651786327,
+          0.017028972506523132,
+          0.024665353819727898,
+          0.009463153779506683,
+          0.0002709778491407633,
+          0.019506484270095825,
+          0.023191390559077263,
+          0.0031419708393514156,
+          -0.00707188481464982,
+          -0.007753984536975622,
+          0.017091695219278336,
+          0.01420649141073227,
+          0.007240449544042349,
+          0.0054254368878901005,
+          -0.007189488038420677,
+          -0.006036974955350161,
+          0.019192876294255257,
+          -0.019804412499070168,
+          -0.0018150125397369266,
+          0.0015288442373275757,
+          -0.04042420908808708,
+          0.010286377742886543,
+          -0.00031115900492295623,
+          -0.014669064432382584,
+          0.011180163361132145,
+          0.004072997719049454,
+          -0.010976317338645458,
+          -0.010176614858210087,
+          -0.016448795795440674,
+          -0.006879799067974091,
+          0.007593259681016207,
+          0.0024324303958564997,
+          0.008506646379828453,
+          -0.01050590444356203,
+          -0.005884090438485146,
+          0.019961217418313026,
+          0.003396778367459774,
+          0.0062369005754590034,
+          -0.0001754250843077898
+        ],
+        "keyphrases": [
+          "scaling up model capacity",
+          "training efficiency",
+          "sparsely gated mixture-of-experts",
+          "expert choice method",
+          "downstream performance"
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "3639a96c-e9a4-45ed-a3ac-1e18b2cfe8d4",
+      "properties": {
+        "page_content": "2 Related Work\n\nScaling: Various approaches have been proposed to scale up neural network capacity to improve performance. Recent works have successfully scaled models to billions of parameters via various forms of model parallelism [2, 21, 26, 27, 33]. Model parallelism [30] splits weights and tensors across multiple cores while pipeline parallelism [18, 24] splits different layers across devices with micro-batches pipelined to the different layers. To enable continued scaling of neural networks, improving model training and serving efficiency has become a critical research area.\n\nConditional Computation: Computation decisions can be made dynamically based on the input [23, 25]. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation, by activating certain parameters and computation on demand, on a per-example or per-token basis [3]. Conditional convolution layers [1] with task-specific gating has been used to combat catastrophic forgetting when a sequence of learning problems are optimized. The gating decisions may be binary or sparse and continuous, stochastic or deterministic.\n\nMixture of Experts: Sparsely-gated MoE [31] is the first model to demonstrate massive improvements in model capacity, training time, or model quality with gating. Switch Transformer [10] simplifies the gating by selecting only the top expert per token using a softmax over the hidden state and demonstrates better scaling than previous work. All the prior work requires an auxiliary loss to explicitly encourage balancing. This loss term has to be carefully weighted to not overwhelm the primary loss. However, auxiliary loss does not guarantee balancing and a hard capacity factor has to be imposed. As a result, many tokens can still be unprocessed by the MoE layer. Hard MoE [12] with a single decoding layer can be efficiently trained to good effect on large scale hashtag prediction tasks.\n\nBase Layers [22] formulate a linear assignment that maximizes token-expert affinities while ensuring each expert receives an equal number of tokens. Hash layers [8, 28] devise hashing techniques on input tokens. However, the evaluations are limited to pre-training perplexity. THOR [? ] randomly activates experts during training and inference and is trained with a consistency regularization loss.\n\nTHOR has demonstrated strong performance on translation tasks. Different from these prior works, our method is a learnt method that enables heterogeneous MoE and effectively improves downstream fine-tuning performance.\n\n",
+        "embedding": [
+          -0.013259558007121086,
+          -0.007197607308626175,
+          -0.015822459012269974,
+          0.013052377849817276,
+          0.029680538922548294,
+          0.02572108618915081,
+          -0.01866159960627556,
+          0.040392544120550156,
+          -0.039103418588638306,
+          0.017602676525712013,
+          0.020380431786179543,
+          -0.012024147436022758,
+          0.0011442891554906964,
+          0.006088807247579098,
+          0.009645408019423485,
+          -0.014533335343003273,
+          -0.004703766666352749,
+          0.0049493140541017056,
+          0.02260570414364338,
+          -0.03012559376657009,
+          -0.008847379125654697,
+          -0.007220627274364233,
+          -0.03612615540623665,
+          -0.01026694942265749,
+          0.03443801775574684,
+          0.0267186239361763,
+          0.03146075829863548,
+          -0.011226118542253971,
+          -0.04496586322784424,
+          0.025629006326198578,
+          0.015055122785270214,
+          0.009300106205046177,
+          -0.016912074759602547,
+          -0.01592988520860672,
+          -0.0355122871696949,
+          -0.04048462212085724,
+          0.05073622614145279,
+          0.013781346380710602,
+          -0.01992003060877323,
+          0.02774685248732567,
+          0.021807676181197166,
+          0.0150397764518857,
+          -0.03502119332551956,
+          -0.02219134382903576,
+          0.00023379754566121846,
+          0.0007141016540117562,
+          -0.032136011868715286,
+          -0.009069905616343021,
+          -0.008317917585372925,
+          -0.00694822333753109,
+          0.028268640860915184,
+          -0.04450546205043793,
+          0.01573037914931774,
+          0.012584302574396133,
+          -0.038274697959423065,
+          0.039932142943143845,
+          0.013228864409029484,
+          0.005678283050656319,
+          0.053621407598257065,
+          0.03726181387901306,
+          0.011333545669913292,
+          0.006449454929679632,
+          -0.04613221064209938,
+          0.002123601036146283,
+          0.010443436913192272,
+          -0.008847379125654697,
+          -0.00621158117428422,
+          0.0015471403021365404,
+          0.0041819787584245205,
+          0.010289969854056835,
+          -0.0367400236427784,
+          -0.0014166932087391615,
+          -0.011893700808286667,
+          -0.003069342114031315,
+          0.010834777727723122,
+          0.001008087070658803,
+          0.019367549568414688,
+          0.039011336863040924,
+          0.03511327505111694,
+          -0.008678564801812172,
+          -0.01622147299349308,
+          0.03317958861589432,
+          -0.002503432100638747,
+          0.016574447974562645,
+          0.02119380794465542,
+          -0.03713903948664665,
+          -0.0017399332718923688,
+          -0.06629779189825058,
+          -0.018983881920576096,
+          -0.01323653757572174,
+          -0.02991073951125145,
+          0.0056015495210886,
+          -0.01734178327023983,
+          -0.0033398279920220375,
+          -0.020273005589842796,
+          0.04646984115242958,
+          -0.018446745350956917,
+          -0.013482085429131985,
+          -0.032872654497623444,
+          0.004730623681098223,
+          -0.01812446489930153,
+          0.03631031885743141,
+          -0.0027451429050415754,
+          -0.0076541719026863575,
+          0.01571503095328808,
+          0.009092926047742367,
+          -0.05006096884608269,
+          0.034622177481651306,
+          -0.00013188579760026187,
+          0.074830561876297,
+          0.012898909859359264,
+          -0.033210281282663345,
+          0.00029134770738892257,
+          0.04054601117968559,
+          -0.03152214363217354,
+          -0.02452404424548149,
+          -0.04149750620126724,
+          -0.03597268834710121,
+          0.001932726358063519,
+          -0.028406761586666107,
+          0.024416616186499596,
+          0.00759278517216444,
+          -0.0010752289090305567,
+          -0.012430835515260696,
+          -0.05288476496934891,
+          0.050889693200588226,
+          -0.013812039978802204,
+          -0.01692742109298706,
+          -0.005755016580224037,
+          -0.02243689075112343,
+          -0.011356566101312637,
+          0.023541854694485664,
+          -0.032872654497623444,
+          -0.05638381466269493,
+          0.01705019548535347,
+          0.016160085797309875,
+          0.051380787044763565,
+          0.02021161839365959,
+          -0.018738333135843277,
+          -0.010044422000646591,
+          0.03278057277202606,
+          -0.005781873129308224,
+          0.02360324189066887,
+          -0.022022530436515808,
+          -0.028821121901273727,
+          0.013942486606538296,
+          -0.033455830067396164,
+          0.06227695196866989,
+          -0.01651306077837944,
+          0.023787401616573334,
+          -0.013474412262439728,
+          0.04542626440525055,
+          -0.03324097394943237,
+          0.03235086426138878,
+          -0.029388951137661934,
+          0.021500742062926292,
+          0.005755016580224037,
+          -0.00864787120372057,
+          0.0028065296355634928,
+          0.00858648493885994,
+          -0.008279550820589066,
+          -0.020902220159769058,
+          0.005098944529891014,
+          0.030800849199295044,
+          0.0026933476328849792,
+          0.024278495460748672,
+          0.04907878115773201,
+          0.02077944576740265,
+          -0.052915457636117935,
+          0.029726577922701836,
+          0.029695885255932808,
+          0.020426472648978233,
+          0.0444747656583786,
+          0.007554418407380581,
+          0.010604577139019966,
+          0.006587575655430555,
+          0.014993736520409584,
+          0.0070403036661446095,
+          -0.01655910164117813,
+          0.03879648447036743,
+          -0.0241403765976429,
+          -0.0031268922612071037,
+          0.024784937500953674,
+          0.0015365893486887217,
+          0.036617252975702286,
+          0.005482612177729607,
+          0.013512779027223587,
+          0.02136262133717537,
+          0.00933079980313778,
+          0.027808239683508873,
+          0.02021161839365959,
+          0.030171632766723633,
+          0.020150231197476387,
+          -0.024401269853115082,
+          0.03198254480957985,
+          -0.056813523173332214,
+          -0.015945231541991234,
+          -0.009323126636445522,
+          0.028283987194299698,
+          -0.006169377826154232,
+          -0.01812446489930153,
+          -0.017894264310598373,
+          0.008049349300563335,
+          0.002048785798251629,
+          -0.009676100686192513,
+          0.008931785821914673,
+          0.033916231244802475,
+          0.026288915425539017,
+          0.024002255871891975,
+          -0.037937067449092865,
+          -0.0052025350742042065,
+          -0.010903838090598583,
+          -0.013374658301472664,
+          -0.034990500658750534,
+          -0.00218306970782578,
+          0.009461247362196445,
+          0.0012622670037671924,
+          0.009630060754716396,
+          -0.002929303329437971,
+          0.0027355512138456106,
+          0.03015628643333912,
+          -0.0046423799358308315,
+          0.045610424131155014,
+          -0.014425908215343952,
+          -0.0034817850682884455,
+          -0.020503206178545952,
+          -0.006576065439730883,
+          0.003627578727900982,
+          0.01622147299349308,
+          0.011824640445411205,
+          0.0031134639866650105,
+          0.012661036103963852,
+          0.004577156621962786,
+          -0.004465892910957336,
+          0.04435199499130249,
+          -0.016697220504283905,
+          0.01414966769516468,
+          -0.011064978316426277,
+          0.008195143193006516,
+          0.003395459847524762,
+          -0.005912320222705603,
+          0.041466813534498215,
+          -0.047851044684648514,
+          -0.0132749043405056,
+          0.020718058571219444,
+          0.0033820313401520252,
+          -0.03176769241690636,
+          0.012001127935945988,
+          -0.0009505368652753532,
+          -0.01609869860112667,
+          -0.01435684785246849,
+          -0.02364928089082241,
+          -0.005543999373912811,
+          0.00993699487298727,
+          0.00022012938279658556,
+          0.05666005611419678,
+          0.018078425899147987,
+          -0.01528532337397337,
+          -0.01978190988302231,
+          -0.02140866219997406,
+          0.03962520509958267,
+          -0.004730623681098223,
+          -0.035819221287965775,
+          0.013934813439846039,
+          -0.012039494700729847,
+          -0.01883041486144066,
+          0.030938968062400818,
+          0.01470982190221548,
+          -0.003180605825036764,
+          -0.019029920920729637,
+          0.008862725459039211,
+          -0.05260852351784706,
+          -0.005716649815440178,
+          0.0007275299867615104,
+          0.06727997958660126,
+          -0.024984445422887802,
+          -0.010934531688690186,
+          0.0050490680150687695,
+          0.0011596358381211758,
+          0.02364928089082241,
+          -0.00858648493885994,
+          -0.003474111668765545,
+          -0.021961143240332603,
+          0.03003351204097271,
+          -0.008010982535779476,
+          0.005578529555350542,
+          -0.026979517191648483,
+          -0.017955651506781578,
+          -0.04849560558795929,
+          -0.02978796511888504,
+          -0.014510314911603928,
+          0.012607323005795479,
+          -0.05273129791021347,
+          0.017648717388510704,
+          0.026856742799282074,
+          -0.021086379885673523,
+          0.009668427519500256,
+          0.012208308093249798,
+          0.027286451309919357,
+          -0.034407325088977814,
+          -0.05260852351784706,
+          -0.02941964380443096,
+          0.014157340861856937,
+          -0.016942769289016724,
+          -0.05512538552284241,
+          0.0036122319288551807,
+          0.012868217192590237,
+          0.015546217560768127,
+          -0.004849560558795929,
+          -0.0013322862796485424,
+          0.02133192867040634,
+          -0.006986590102314949,
+          0.0062768044881522655,
+          -0.01721900887787342,
+          -0.00873227883130312,
+          0.011793946847319603,
+          0.010282295756042004,
+          -0.000877640035469085,
+          -0.03738458827137947,
+          -0.013466738164424896,
+          0.0236953217536211,
+          0.018600212410092354,
+          -0.07317311316728592,
+          0.011487012729048729,
+          -0.027501305565238,
+          -0.007170750759541988,
+          -0.0042126718908548355,
+          0.03302612155675888,
+          -0.019060615450143814,
+          0.016083352267742157,
+          -0.0020583774894475937,
+          -0.02808448113501072,
+          -0.027639426290988922,
+          -0.006537698674947023,
+          -0.03410039097070694,
+          -0.03207462653517723,
+          -0.028468148782849312,
+          -0.004178142175078392,
+          -0.03143006190657616,
+          -0.029220137745141983,
+          -0.024508696049451828,
+          0.0023173533845692873,
+          -0.00552865257486701,
+          0.017357129603624344,
+          0.01899922825396061,
+          -0.03373207151889801,
+          -0.045364875346422195,
+          0.013190497644245625,
+          -0.006587575655430555,
+          0.01259197574108839,
+          0.0009654040331952274,
+          -0.012062514200806618,
+          0.016282860189676285,
+          -0.003094280604273081,
+          -0.03015628643333912,
+          -0.046193599700927734,
+          -0.013297924771904945,
+          0.014610068872570992,
+          0.008095390163362026,
+          -0.0002774397435132414,
+          0.010006055235862732,
+          -0.04686885327100754,
+          -0.024171069264411926,
+          0.01066596433520317,
+          -0.04008561000227928,
+          -0.024002255871891975,
+          -0.0045694829896092415,
+          -0.025444846600294113,
+          -0.006391904782503843,
+          -0.006844632793217897,
+          -0.0222373828291893,
+          0.03729250654578209,
+          0.018262585625052452,
+          -0.00933079980313778,
+          0.02941964380443096,
+          0.012162268161773682,
+          0.020672019571065903,
+          -0.01617543213069439,
+          -0.04640845209360123,
+          0.030048860237002373,
+          -0.019444283097982407,
+          -0.03098500892519951,
+          8.272836566902697e-05,
+          -0.022252731025218964,
+          0.007792292162775993,
+          0.00012229410640429705,
+          0.015945231541991234,
+          -0.006422598380595446,
+          -0.04573319852352142,
+          0.02932756394147873,
+          0.004872580524533987,
+          -0.04956987500190735,
+          -0.0022137630730867386,
+          -0.04791243001818657,
+          -0.0065607186406850815,
+          0.030340446159243584,
+          -0.004914783872663975,
+          -0.027977053076028824,
+          -0.03459148481488228,
+          -0.0015605686930939555,
+          -0.011433299630880356,
+          0.014625415205955505,
+          -0.013643225654959679,
+          0.013727632351219654,
+          -0.008041676133871078,
+          -0.0188764538615942,
+          -0.010719677433371544,
+          0.015776418149471283,
+          0.03759944066405296,
+          -0.024155722931027412,
+          -0.022682437673211098,
+          -0.03572714328765869,
+          -0.03953312709927559,
+          0.013221191242337227,
+          0.024201761931180954,
+          0.01867694780230522,
+          -0.010972898453474045,
+          0.018078425899147987,
+          0.0068369596265256405,
+          -0.014878636226058006,
+          -0.025153258815407753,
+          0.0029810986015945673,
+          0.05441943556070328,
+          0.00807236973196268,
+          -0.00013368423969950527,
+          0.002990690292790532,
+          0.009798875078558922,
+          0.017188316211104393,
+          0.005555509123951197,
+          -0.014280114322900772,
+          0.04373812675476074,
+          -0.014433581382036209,
+          0.039686594158411026,
+          -0.01065829023718834,
+          -0.02070271223783493,
+          -0.03634101152420044,
+          0.023679975420236588,
+          0.005313798785209656,
+          -0.03443801775574684,
+          -0.012361775152385235,
+          0.029358256608247757,
+          0.005720486398786306,
+          -0.010773390531539917,
+          0.012599649839103222,
+          -0.035358820110559464,
+          -0.029557764530181885,
+          -0.022713132202625275,
+          -0.037108346819877625,
+          -0.00947659369558096,
+          -0.03453009948134422,
+          -0.06365815550088882,
+          -0.004247202072292566,
+          -0.061294760555028915,
+          -0.014878636226058006,
+          0.02820725366473198,
+          -0.0006987549131736159,
+          0.0162982065230608,
+          -0.00065223517594859,
+          -0.007700212299823761,
+          -0.03833608329296112,
+          0.005206371657550335,
+          0.04174305126070976,
+          0.01746455766260624,
+          -0.0380905345082283,
+          -0.017280396074056625,
+          -0.009031538851559162,
+          -0.04263316094875336,
+          -0.029342910274863243,
+          0.009031538851559162,
+          -0.013643225654959679,
+          6.396460958058015e-05,
+          -0.03941035270690918,
+          0.003128810552880168,
+          0.006840796209871769,
+          0.018109118565917015,
+          0.04444407299160957,
+          0.022129956632852554,
+          -0.06040465459227562,
+          0.007987963035702705,
+          -0.003629497019574046,
+          0.027240412309765816,
+          -0.014072933234274387,
+          0.018017038702964783,
+          -0.0006474393303506076,
+          -0.007009610068053007,
+          -0.02662654221057892,
+          0.009054559282958508,
+          -0.0012507569044828415,
+          -0.011878354474902153,
+          -0.0038596978411078453,
+          0.004316262435168028,
+          0.00914663914591074,
+          0.03182907775044441,
+          0.010282295756042004,
+          0.009537980891764164,
+          -0.04082224890589714,
+          -0.012860543094575405,
+          -0.01130285207182169,
+          0.04042323678731918,
+          0.009246393106877804,
+          -0.034990500658750534,
+          0.007316544186323881,
+          0.03253502771258354,
+          0.028406761586666107,
+          0.028115173801779747,
+          -0.011072651483118534,
+          0.012730096466839314,
+          0.020349739119410515,
+          -0.07004238665103912,
+          -0.005758853163570166,
+          -0.0069904266856610775,
+          -0.0037503524217754602,
+          0.0037407607305794954,
+          -0.02289729192852974,
+          -0.00019531087309587747,
+          0.042541082948446274,
+          -0.016896728426218033,
+          -0.00151644682046026,
+          -0.011003591120243073,
+          0.029235484078526497,
+          0.002560982247814536,
+          0.004086061846464872,
+          -0.012607323005795479,
+          0.058470968157052994,
+          -0.017863571643829346,
+          0.013343964703381062,
+          0.006077297497540712,
+          -0.01306772418320179,
+          0.024861671030521393,
+          -0.021715596318244934,
+          -0.007051813416182995,
+          0.005056741181761026,
+          0.0094382269307971,
+          -0.017157621681690216,
+          -0.003107708878815174,
+          -0.0026991027407348156,
+          -0.01741851679980755,
+          0.010811757296323776,
+          -0.0489560067653656,
+          -0.03471425920724869,
+          -0.0022463747300207615,
+          0.01005976926535368,
+          -0.020027456805109978,
+          0.010305316187441349,
+          0.014326154254376888,
+          0.007934249006211758,
+          0.055739253759384155,
+          -0.03143006190657616,
+          -0.0016229146858677268,
+          -0.003309134393930435,
+          -0.023925522342324257,
+          -0.028636962175369263,
+          -0.023004719987511635,
+          -0.0056130592711269855,
+          -0.015822459012269974,
+          -0.04386089742183685,
+          -0.008693911135196686,
+          -0.021439354866743088,
+          0.021270541474223137,
+          -0.04094502329826355,
+          0.016912074759602547,
+          0.008724604733288288,
+          0.0006124296924099326,
+          -0.009008519351482391,
+          -0.01995072327554226,
+          -0.0042625488713383675,
+          0.029757272452116013,
+          0.032166704535484314,
+          0.018861107528209686,
+          0.034867726266384125,
+          -0.005636079702526331,
+          -0.0231888797134161,
+          -0.013581839390099049,
+          0.03572714328765869,
+          -0.00854811817407608,
+          0.020058151334524155,
+          0.004980007652193308,
+          -0.0006061950698494911,
+          -0.0224061980843544,
+          -0.0077270688489079475,
+          0.03646378591656685,
+          -0.02555227279663086,
+          0.014425908215343952,
+          0.05122731998562813,
+          0.032749880105257034,
+          0.0006071542156860232,
+          -0.008640198037028313,
+          -0.01472516916692257,
+          0.005919993855059147,
+          -0.021792329847812653,
+          0.03198254480957985,
+          0.01987398974597454,
+          -0.001803238526917994,
+          -0.010535516776144505,
+          0.01325188484042883,
+          -0.030217673629522324,
+          0.012469202280044556,
+          0.006668145768344402,
+          0.003606477053835988,
+          -0.034867726266384125,
+          -0.02136262133717537,
+          -0.002419025171548128,
+          -0.0007759680738672614,
+          -0.01929081603884697,
+          -0.015553891658782959,
+          -0.02286659926176071,
+          0.002154294401407242,
+          -0.006821612827479839,
+          0.004281732253730297,
+          0.016697220504283905,
+          0.02837606891989708,
+          -0.0006138684111647308,
+          0.03701626509428024,
+          0.00303289364092052,
+          0.023879481479525566,
+          0.004945477470755577,
+          -0.003512478433549404,
+          -0.0070403036661446095,
+          -0.011755580082535744,
+          -0.014487294480204582,
+          0.057273924350738525,
+          0.017019502818584442,
+          0.05957593023777008,
+          -0.011064978316426277,
+          -0.010819431394338608,
+          0.030938968062400818,
+          0.004316262435168028,
+          -0.020733406767249107,
+          0.008670891635119915,
+          -0.016160085797309875,
+          -0.02871369570493698,
+          -0.0027470611967146397,
+          -0.03799845650792122,
+          -0.0188764538615942,
+          -0.022621052339673042,
+          -0.018308626487851143,
+          0.004684583283960819,
+          0.0196591354906559,
+          0.016436327248811722,
+          -0.00016737506666686386,
+          -0.011763254180550575,
+          0.031583528965711594,
+          0.019306162372231483,
+          0.028928549960255623,
+          -0.011809294112026691,
+          0.007331890985369682,
+          -0.01576107181608677,
+          0.0052831051871180534,
+          0.019137348979711533,
+          -0.01253058947622776,
+          0.009453574195504189,
+          -0.0008905887953005731,
+          0.011180078610777855,
+          0.03821330890059471,
+          -0.010036748833954334,
+          -0.03953312709927559,
+          -0.006280641537159681,
+          -0.014778882265090942,
+          0.007278177421540022,
+          0.0067065125331282616,
+          -0.020932912826538086,
+          0.012254348024725914,
+          0.007730905432254076,
+          -0.01479422952979803,
+          0.02128588780760765,
+          -0.007205280475318432,
+          -0.027624079957604408,
+          0.03468356654047966,
+          -0.014134320430457592,
+          0.01787891797721386,
+          -0.017986344173550606,
+          -0.0028755899984389544,
+          -0.025828514248132706,
+          0.022559665143489838,
+          0.01751059666275978,
+          -0.027317145839333534,
+          0.022989371791481972,
+          0.04564111679792404,
+          0.020150231197476387,
+          0.0007615805370733142,
+          0.012415489181876183,
+          0.0057665263302624226,
+          -0.036494478583335876,
+          -0.02057993970811367,
+          -0.006944386754184961,
+          -0.0027950196526944637,
+          -0.016574447974562645,
+          0.030217673629522324,
+          -0.03824400156736374,
+          -0.0031518307514488697,
+          -0.0052025350742042065,
+          -0.024078989401459694,
+          0.0055823661386966705,
+          0.009069905616343021,
+          0.003951778169721365,
+          0.017142275348305702,
+          -0.014172687195241451,
+          0.03020232729613781,
+          0.03566575422883034,
+          0.006322844885289669,
+          -0.017848225310444832,
+          0.011256812140345573,
+          -0.024508696049451828,
+          0.0048457239754498005,
+          0.043001484125852585,
+          0.014072933234274387,
+          0.008494404144585133,
+          -0.020426472648978233,
+          -0.02584386058151722,
+          0.007235974073410034,
+          0.008709258399903774,
+          0.04198860004544258,
+          0.02630426175892353,
+          0.03606477007269859,
+          0.009369166567921638,
+          -0.01199345476925373,
+          -0.013819713145494461,
+          -0.0049493140541017056,
+          -0.023265613242983818,
+          0.021761635318398476,
+          -0.02228342369198799,
+          0.01717296987771988,
+          0.012039494700729847,
+          -0.028759736567735672,
+          0.00017528820899315178,
+          0.01576107181608677,
+          0.03575783595442772,
+          0.04361535236239433,
+          0.023925522342324257,
+          0.0032765227369964123,
+          0.039686594158411026,
+          -0.04493517056107521,
+          -0.03716973215341568,
+          0.00753907160833478,
+          0.017571983858942986,
+          0.009154313243925571,
+          0.0058164033107459545,
+          0.03290334716439247,
+          -0.00947659369558096,
+          -0.003376276232302189,
+          0.012024147436022758,
+          0.02074875310063362,
+          0.002031520940363407,
+          -0.00429324246942997,
+          -0.008095390163362026,
+          0.02289729192852974,
+          0.021086379885673523,
+          0.03937966004014015,
+          -0.020058151334524155,
+          0.01780218444764614,
+          0.0009936995338648558,
+          -0.02552158012986183,
+          0.02559831365942955,
+          0.00021701208606828004,
+          -0.04530349001288414,
+          0.03339444100856781,
+          0.017249703407287598,
+          0.015108836814761162,
+          0.011786273680627346,
+          -0.006131011061370373,
+          0.012553608976304531,
+          0.030524607747793198,
+          -0.0031019540037959814,
+          0.0011423707474023104,
+          0.03115382231771946,
+          -0.003311052918434143,
+          -0.004807357210665941,
+          0.020886873826384544,
+          0.004887927323579788,
+          -0.013167478144168854,
+          0.01867694780230522,
+          -0.012169941328465939,
+          0.044075753539800644,
+          -0.024692857638001442,
+          0.006760226096957922,
+          0.010205562226474285,
+          -0.01883041486144066,
+          -0.007293524220585823,
+          -0.0013188580051064491,
+          -0.022268077358603477,
+          0.004500423092395067,
+          0.018155159428715706,
+          -0.012331082485616207,
+          0.0295731108635664,
+          -0.028606269508600235,
+          0.0012430836213752627,
+          0.015945231541991234,
+          -0.026933476328849792,
+          0.02767011895775795,
+          0.013006336987018585,
+          0.02713298425078392,
+          -0.009576347656548023,
+          -0.014456601813435555,
+          0.04797381907701492,
+          0.0563531219959259,
+          0.0060504404827952385,
+          -0.008578810840845108,
+          0.013742979615926743,
+          0.0003174850717186928,
+          -0.02360324189066887,
+          0.024048294872045517,
+          -0.03572714328765869,
+          0.0162982065230608,
+          0.03107708878815174,
+          -0.014617742039263248,
+          -0.014095953665673733,
+          -0.034622177481651306,
+          -0.0052831051871180534,
+          0.00429324246942997,
+          0.004381485749036074,
+          0.0036966388579458,
+          -0.025659700855612755,
+          -0.015599931590259075,
+          0.022252731025218964,
+          0.005379022099077702,
+          0.0010157604701817036,
+          -0.025076525285840034,
+          0.015530871227383614,
+          -0.01617543213069439,
+          -0.04162028059363365,
+          -0.006330518051981926,
+          -0.012492222711443901,
+          0.025214646011590958,
+          -6.204627425177023e-05,
+          -0.03477564454078674,
+          -0.006718022748827934,
+          -0.005191024858504534,
+          -0.009054559282958508,
+          0.044536154717206955,
+          0.023741360753774643,
+          0.03812123090028763,
+          -0.02170024812221527,
+          -0.03141471743583679,
+          0.02460077777504921,
+          -0.03489841893315315,
+          0.030770154669880867,
+          -0.0032439110800623894,
+          0.007186097092926502,
+          -0.0052025350742042065,
+          0.039348967373371124,
+          -0.012146921828389168,
+          0.008571137674152851,
+          -0.03588061034679413,
+          -0.004980007652193308,
+          -0.01945962943136692,
+          -0.008517424575984478,
+          0.0031153822783380747,
+          0.011870680376887321,
+          -0.009568674489855766,
+          0.005659099668264389,
+          0.04720648378133774,
+          0.031583528965711594,
+          0.0018675028113648295,
+          0.026565156877040863,
+          -0.02862161584198475,
+          -0.017986344173550606,
+          0.01464076153934002,
+          -0.0064072515815496445,
+          -0.00785751547664404,
+          -0.013973180204629898,
+          -0.031798385083675385,
+          -0.012799156829714775,
+          0.011187751777470112,
+          0.013581839390099049,
+          -0.011602113023400307,
+          0.003735005622729659,
+          0.012906583957374096,
+          -0.014172687195241451,
+          -0.008057023398578167,
+          -0.0393182709813118,
+          0.0011222282191738486,
+          0.024370577186346054,
+          -0.027977053076028824,
+          0.013121437281370163,
+          0.002286659786477685,
+          0.007389441132545471,
+          -0.023618588224053383,
+          -0.02087152563035488,
+          0.008218163624405861,
+          -0.012154594995081425,
+          0.005248575005680323,
+          -0.002712531015276909,
+          0.034744951874017715,
+          -0.00800330936908722,
+          0.03351721540093422,
+          -0.012975644320249557,
+          0.01953636296093464,
+          -0.025506233796477318,
+          -0.02389482781291008,
+          -0.04272524267435074,
+          -0.0012977562146261334,
+          0.0029580784030258656,
+          -0.02820725366473198,
+          -0.006495495326817036,
+          -0.001958623994141817,
+          0.017249703407287598,
+          -0.012246674858033657,
+          -0.003274604445323348,
+          0.006622105836868286,
+          -0.01354347262531519,
+          -0.0020104190334677696,
+          -0.03987075388431549,
+          -0.02825329452753067,
+          -0.03406969830393791,
+          0.016835341230034828,
+          -0.022667091339826584,
+          0.017234355211257935,
+          -0.01995072327554226,
+          0.0005678283050656319,
+          -0.047359950840473175,
+          -0.01573037914931774,
+          0.02991073951125145,
+          0.008141430094838142,
+          -0.04204998537898064,
+          0.021470047533512115,
+          0.01755663752555847,
+          0.017479903995990753,
+          -0.02311214618384838,
+          0.018492786213755608,
+          0.002338455058634281,
+          0.004933967255055904,
+          0.024416616186499596,
+          -0.02535276673734188,
+          0.02177698165178299,
+          -0.024907711893320084,
+          -0.021838368847966194,
+          -0.004020838066935539,
+          0.013827386312186718,
+          0.01726504974067211,
+          0.0020104190334677696,
+          -0.01709623634815216,
+          0.016743261367082596,
+          0.018323972821235657,
+          0.003915329463779926,
+          0.022636398673057556,
+          -0.02664189040660858,
+          0.016620486974716187,
+          0.011809294112026691,
+          -0.02323492057621479,
+          0.0036256604362279177,
+          -0.01483259629458189,
+          -0.007174587342888117,
+          0.017525942996144295,
+          0.03146075829863548,
+          0.014617742039263248,
+          -0.03683210536837578,
+          0.012860543094575405,
+          0.011747906915843487,
+          -0.0014147749170660973,
+          0.038274697959423065,
+          0.0027240412309765816,
+          0.0239715613424778,
+          0.03397761657834053,
+          0.0066105956211686134,
+          0.017664063721895218,
+          0.03548159450292587,
+          -0.022912638261914253,
+          -0.009277086704969406,
+          -0.023802747949957848,
+          0.014111299999058247,
+          0.014310807920992374,
+          -0.012753116898238659,
+          -0.012001127935945988,
+          -0.015239283442497253,
+          -0.014118974097073078,
+          0.01879972033202648,
+          -0.0008503036806359887,
+          -0.015745725482702255,
+          -0.029711231589317322,
+          0.024692857638001442,
+          0.0006781327538192272,
+          -0.044658929109573364,
+          0.012323408387601376,
+          -0.014218727126717567,
+          -0.0037868008948862553,
+          -0.0027835096698254347,
+          0.007769272197037935,
+          -0.005605386104434729,
+          0.0033359911758452654,
+          0.004266385454684496,
+          0.05135009437799454,
+          -0.0014166932087391615,
+          0.017311088740825653,
+          -0.008839705027639866,
+          0.01970517635345459,
+          -0.010988244786858559,
+          0.022912638261914253,
+          -0.014556354843080044,
+          -0.051503561437129974,
+          0.02618148922920227,
+          -0.011041957885026932,
+          0.040883637964725494,
+          0.05098177120089531,
+          -0.01841605268418789,
+          -0.010750371031463146,
+          0.03575783595442772,
+          0.004680746700614691,
+          -0.025337418541312218,
+          0.007888209074735641,
+          -0.0012296552304178476,
+          -0.007504541426897049,
+          -0.019613096490502357,
+          -0.012077861465513706,
+          0.019720522686839104,
+          -0.0038712078239768744,
+          0.017372475937008858,
+          0.040638089179992676,
+          0.03161422535777092,
+          -0.021301234140992165,
+          0.009522633627057076,
+          0.01331327110528946,
+          -0.009215699508786201,
+          -0.007796129211783409,
+          0.0254295002669096,
+          -0.017034849151968956,
+          -0.0008320794440805912,
+          -0.012177614495158195,
+          -0.00400165468454361,
+          -0.035941995680332184,
+          0.001303511206060648,
+          -0.016605140641331673,
+          -0.0018761353567242622,
+          -0.007692538667470217,
+          -0.030923621729016304,
+          0.03228947892785072,
+          0.028698349371552467,
+          -0.0070633236318826675,
+          -0.009077579714357853,
+          -0.009468920528888702,
+          0.005037557799369097,
+          0.0036333338357508183,
+          -0.0019969907589256763,
+          0.0034357449039816856,
+          -0.01792495884001255,
+          0.007443154696375132,
+          -0.006775572896003723,
+          -0.01325188484042883,
+          -0.051135238260030746,
+          -0.01867694780230522,
+          0.025045832619071007,
+          0.004830377176403999,
+          -0.01692742109298706,
+          -0.003261175937950611,
+          0.008103063330054283,
+          -0.005862443707883358,
+          -0.02496909908950329,
+          0.03135332837700844,
+          0.01668187417089939,
+          0.01875368133187294,
+          -0.0017456883797422051,
+          -0.0022578847128897905,
+          0.01746455766260624,
+          0.006084970664232969,
+          -0.01547715812921524,
+          -0.009499614126980305,
+          0.021761635318398476,
+          0.004120592027902603,
+          0.020932912826538086,
+          -0.010735023766756058,
+          0.021086379885673523,
+          0.007957269437611103,
+          -0.012123901396989822,
+          -0.0027202044147998095,
+          -0.0012718586949631572,
+          0.004105245228856802,
+          0.016912074759602547,
+          -0.013980853371322155,
+          0.024953750893473625,
+          0.020088844001293182,
+          0.007109363563358784,
+          0.0044889128766953945,
+          -0.01668187417089939,
+          -0.02452404424548149,
+          0.016574447974562645,
+          0.002522615483030677,
+          0.027977053076028824,
+          0.012139247730374336,
+          -0.004013164900243282,
+          -0.02759338542819023,
+          -0.013052377849817276,
+          -0.024217110127210617,
+          0.01005976926535368,
+          0.023664627224206924,
+          0.030632033944129944,
+          -0.015461810864508152,
+          -0.013489758595824242,
+          -0.025276033207774162,
+          0.00469993008300662,
+          0.008908765390515327,
+          -0.027347838506102562,
+          -0.01992003060877323,
+          0.018032385036349297,
+          -0.025291379541158676,
+          0.026488423347473145,
+          -0.0019269713666290045,
+          -0.012852869927883148,
+          0.006675818935036659,
+          0.07071764022111893,
+          -0.013090744614601135,
+          -0.0012200635392218828,
+          -0.028652308508753777,
+          -0.017618024721741676,
+          -0.005931503605097532,
+          -0.008617177605628967,
+          0.0004412179405335337,
+          0.018492786213755608,
+          -0.008624851703643799,
+          0.020902220159769058,
+          0.01668187417089939,
+          0.0013006337685510516,
+          0.0032074626069515944,
+          -0.03526674211025238,
+          0.03032509982585907,
+          -0.004224182106554508,
+          0.0014838350471109152,
+          0.00244204537011683,
+          0.006307498086243868,
+          0.018400706350803375,
+          0.016896728426218033,
+          0.005931503605097532,
+          0.009200353175401688,
+          -0.005912320222705603,
+          -0.003738842438906431,
+          0.016114046797156334,
+          0.019858643412590027,
+          -0.01111869141459465,
+          -0.012944950722157955,
+          0.005728159565478563,
+          0.014901655726134777,
+          -0.004853397142142057,
+          -0.00414744857698679,
+          0.0075736017897725105,
+          0.020472511649131775,
+          -0.01617543213069439,
+          -0.002286659786477685,
+          0.004742133431136608,
+          -0.017372475937008858,
+          0.0007548663415946066,
+          0.01875368133187294,
+          0.007565928157418966,
+          0.015853151679039,
+          -0.015484831295907497,
+          -0.009161986410617828,
+          0.0021255193278193474,
+          -0.022421544417738914,
+          0.003121137386187911,
+          -0.012192961759865284,
+          -0.01058155670762062,
+          -0.00239600520581007,
+          -0.007416298147290945,
+          -0.006487822160124779,
+          -0.006568392273038626,
+          0.004630869720131159,
+          0.009430553764104843,
+          -0.004151285160332918,
+          0.01738782227039337,
+          0.005252411589026451,
+          0.013988526538014412,
+          -0.0053560021333396435,
+          0.01489398255944252,
+          0.02136262133717537,
+          -0.010213236324489117,
+          0.00075822341023013,
+          0.01775614358484745,
+          -0.0006344905705191195,
+          0.006741042714565992,
+          0.004013164900243282,
+          1.707021874608472e-05,
+          0.010842450894415379,
+          -0.004343118984252214,
+          -0.01992003060877323,
+          -0.0055094691924750805,
+          -0.011440972797572613,
+          0.002133192727342248,
+          0.002252129837870598,
+          0.03716973215341568,
+          -0.02513791248202324,
+          0.03235086426138878,
+          0.012914257124066353,
+          -0.014011546969413757,
+          -0.00569362984970212,
+          -0.047482721507549286,
+          0.01338233146816492,
+          -0.014694475568830967,
+          3.977555752499029e-05,
+          0.017970997840166092,
+          -0.00914663914591074,
+          -0.0014799984637647867,
+          0.00502988463267684,
+          -0.005482612177729607,
+          -0.0009457410196773708,
+          -0.01924477517604828,
+          -0.0067563895136117935,
+          0.010382049717009068,
+          0.013842732645571232,
+          -0.03471425920724869,
+          -0.02243689075112343,
+          -0.030263712629675865,
+          0.00392875773832202,
+          0.01883041486144066,
+          -0.010942204855382442,
+          -0.025828514248132706,
+          -0.009537980891764164,
+          0.013405351899564266,
+          -0.01697346195578575,
+          0.006388068199157715,
+          -0.003997818101197481,
+          0.014326154254376888,
+          -0.008832031860947609,
+          0.001874217065051198,
+          -0.01899922825396061,
+          0.01487096305936575,
+          0.0028314681258052588,
+          -0.00420116214081645,
+          -0.030340446159243584,
+          0.007120873779058456,
+          -0.00020574184600263834,
+          -0.01450264174491167,
+          -0.012269695289433002,
+          0.017280396074056625,
+          -0.011134038679301739,
+          -0.004745970480144024,
+          0.016743261367082596,
+          0.006695002317428589,
+          0.0006618269253522158,
+          0.018277931958436966,
+          0.005352165549993515,
+          -0.0005002068355679512,
+          0.005336818750947714,
+          0.013106090947985649,
+          0.0012517160503193736,
+          0.024048294872045517,
+          -0.01617543213069439,
+          0.00997536163777113,
+          -0.014333827421069145,
+          0.033701375126838684,
+          -0.01597592607140541,
+          -0.0016027720412239432,
+          0.012177614495158195,
+          0.040883637964725494,
+          -0.013497431762516499,
+          0.01712692901492119,
+          -0.0291280560195446,
+          -0.015354383736848831,
+          0.03465287387371063,
+          0.01688138209283352,
+          0.010988244786858559,
+          -0.010359029285609722,
+          -0.01332094520330429,
+          0.008801338262856007,
+          0.01470982190221548,
+          0.008824358694255352,
+          -0.0241403765976429,
+          0.02908201701939106,
+          -0.014571702107787132,
+          0.04162028059363365,
+          -0.0001700127759249881,
+          -0.003117300570011139,
+          0.023204226046800613,
+          -0.0009577306336723268,
+          -0.02265174500644207,
+          0.014947696588933468,
+          0.00268183765001595,
+          0.01203182153403759,
+          -0.01066596433520317,
+          0.010527843609452248,
+          0.023787401616573334,
+          0.006537698674947023,
+          -0.005018374416977167,
+          -7.163796544773504e-05,
+          0.016912074759602547,
+          0.0005409715231508017,
+          -0.005033721216022968,
+          0.027731506153941154,
+          -0.007481521461158991,
+          -0.010842450894415379,
+          -0.01875368133187294,
+          -0.0012171859852969646,
+          -0.018247239291667938,
+          -0.00043474353151395917,
+          -0.007700212299823761,
+          0.0008320794440805912,
+          -0.010589230805635452,
+          -0.015154876746237278,
+          -0.003944104537367821,
+          -0.025076525285840034,
+          0.03480634093284607,
+          0.009676100686192513,
+          0.013597185723483562,
+          0.009200353175401688,
+          0.00365059869363904,
+          -0.000640245561953634,
+          0.0214547012001276,
+          -0.0030175470747053623,
+          0.0005649508093483746,
+          -0.005685956217348576,
+          -0.0119857806712389,
+          -0.0122390016913414,
+          0.013980853371322155,
+          -0.008317917585372925,
+          -0.008640198037028313,
+          -0.00844069104641676,
+          -0.01821654476225376,
+          -0.03477564454078674,
+          -0.0013610613532364368,
+          0.008893419057130814,
+          -0.028360720723867416,
+          0.014909329824149609,
+          -0.006441781762987375,
+          -0.01600661873817444,
+          0.0008009064476937056,
+          0.006161704193800688,
+          0.018554173409938812,
+          -0.007972615770995617,
+          -0.03425385802984238,
+          0.031384024769067764,
+          0.01646701991558075,
+          0.003899982897564769,
+          0.0033820313401520252,
+          0.008010982535779476,
+          0.008064696565270424,
+          -0.01395783293992281,
+          -0.0099139753729105,
+          0.028115173801779747,
+          -0.026212181895971298,
+          0.0239715613424778,
+          0.012561283074319363,
+          0.0003963767667300999,
+          0.027240412309765816,
+          -0.019474975764751434,
+          0.02630426175892353,
+          -0.0076042949222028255,
+          -5.1765175157925114e-05,
+          0.015200916677713394,
+          0.02099430002272129,
+          0.020319044589996338,
+          0.02272847853600979,
+          -0.02265174500644207,
+          0.038643017411231995,
+          -0.0023787401150912046,
+          0.03201323747634888,
+          0.006307498086243868,
+          -0.00015094928676262498,
+          0.008486730977892876,
+          0.008172123692929745,
+          -0.017679410055279732,
+          -0.0010176787618547678,
+          -0.03069342114031315,
+          0.010604577139019966,
+          -0.005229391623288393,
+          -0.01867694780230522,
+          -0.020794792100787163,
+          -0.033916231244802475,
+          0.010888490825891495,
+          0.001221981830894947,
+          -0.013037030585110188,
+          -0.004742133431136608,
+          0.0078076389618217945,
+          0.01306772418320179,
+          0.022713132202625275,
+          -0.0037100673653185368,
+          0.0032688493374735117,
+          0.019643789157271385,
+          -0.005298451986163855,
+          0.004331609234213829,
+          -0.01617543213069439,
+          0.02211461029946804,
+          0.005555509123951197,
+          0.030754808336496353,
+          0.02476959116756916,
+          0.008540444076061249,
+          0.004611686337739229,
+          -0.004204998724162579,
+          0.017817530781030655,
+          -0.01600661873817444,
+          -0.010481803677976131,
+          -0.001481916755437851,
+          -0.0038366776425391436,
+          0.06037395820021629,
+          -0.024692857638001442,
+          -0.02506117895245552,
+          -0.005559345707297325,
+          -0.011487012729048729,
+          0.006675818935036659,
+          -0.03247363865375519,
+          -0.028928549960255623,
+          -0.023511160165071487,
+          -0.0175873301923275,
+          -0.0019624605774879456,
+          -0.009998382069170475,
+          0.014049913734197617,
+          -0.00569362984970212,
+          0.011778600513935089,
+          -0.0239715613424778,
+          0.000355372263584286,
+          -0.016651181504130363,
+          -0.005685956217348576,
+          -0.014026893302798271,
+          -0.0070403036661446095,
+          -0.000628735579084605,
+          0.02949637733399868,
+          0.011824640445411205,
+          -0.019229428842663765,
+          -0.0032918695360422134,
+          0.012745442800223827,
+          -0.008279550820589066,
+          -0.007362584583461285,
+          0.012246674858033657,
+          0.01916804164648056,
+          -0.012208308093249798,
+          0.037937067449092865,
+          -0.0037983108777552843,
+          -0.00038582590059377253,
+          -0.005904647056013346,
+          -0.012584302574396133,
+          0.024877017363905907,
+          0.028314681723713875,
+          -0.03984006121754646,
+          -0.0013303679879754782,
+          0.0018694212194532156,
+          0.007600458338856697,
+          -0.005321471951901913,
+          0.0030866072047501802,
+          -0.027117637917399406,
+          -0.0012776136863976717,
+          -0.026872090995311737,
+          -0.01573037914931774,
+          -0.021470047533512115,
+          -0.005923830438405275,
+          0.0032458293717354536,
+          -0.023050758987665176,
+          0.030094899237155914,
+          0.02077944576740265,
+          -0.014487294480204582,
+          -0.0038596978411078453,
+          -0.003500968450680375,
+          0.008126083761453629,
+          -0.03189046308398247,
+          -0.004362302366644144,
+          -0.009745161049067974,
+          -9.597688767826185e-05,
+          -0.00962238758802414,
+          -0.04198860004544258,
+          0.0012239001225680113,
+          -0.003167177317664027,
+          -0.0017955651273950934,
+          -0.008494404144585133,
+          -0.016727915033698082,
+          0.02177698165178299,
+          0.020303698256611824,
+          -0.010880817659199238,
+          -0.008532770909368992,
+          0.045610424131155014,
+          0.008425343781709671,
+          0.0016411389224231243,
+          -0.010888490825891495,
+          0.002762407995760441,
+          0.0028314681258052588,
+          0.008486730977892876,
+          -0.013528125360608101,
+          0.011463993228971958,
+          -0.005298451986163855,
+          0.014118974097073078,
+          -0.0067065125331282616,
+          -0.016114046797156334,
+          0.023204226046800613,
+          -0.02829933539032936,
+          0.000396856339648366,
+          -0.010174869559705257,
+          -0.001934644766151905,
+          0.01395783293992281,
+          0.010259276255965233,
+          0.01726504974067211,
+          -0.03953312709927559,
+          -0.004815030377358198,
+          -0.008386977016925812,
+          -0.0004716715484391898,
+          -0.0059468504041433334,
+          -0.026994863525032997,
+          0.013443718664348125,
+          0.0023557201493531466,
+          -0.010880817659199238,
+          0.018323972821235657,
+          -0.004515769425779581,
+          -0.009960015304386616,
+          0.022375503554940224,
+          -0.0019020329928025603,
+          0.015362057834863663,
+          0.029235484078526497,
+          -0.016789302229881287,
+          0.0183700118213892,
+          0.032504335045814514,
+          -0.009369166567921638,
+          -0.0037139039486646652,
+          -0.0028142030350863934,
+          0.0009936995338648558,
+          0.016328901052474976,
+          0.017525942996144295,
+          0.0007447950774803758,
+          -0.03228947892785072,
+          0.00829489715397358,
+          -0.00218306970782578,
+          0.007918902672827244,
+          0.009514960460364819,
+          -0.014318481087684631,
+          -0.000949098146520555,
+          -0.025997327640652657,
+          0.006952059920877218,
+          -0.0209482591599226,
+          -0.009422880597412586,
+          0.024171069264411926,
+          -0.017786838114261627,
+          -0.019152695313096046,
+          0.003688965691253543,
+          0.02837606891989708,
+          -0.029680538922548294,
+          0.01130285207182169,
+          0.004139775410294533,
+          -0.005812566727399826,
+          0.006361211650073528,
+          0.000759182614274323,
+          -0.01895318739116192,
+          0.004500423092395067,
+          -0.009507287293672562,
+          -0.012169941328465939,
+          -0.020257657393813133,
+          -0.013835059478878975,
+          -0.0006699798395857215,
+          0.014617742039263248,
+          0.015822459012269974,
+          0.028851816430687904,
+          0.0016516897594556212,
+          -0.0069405497051775455,
+          -0.014656108804047108,
+          0.006879162974655628,
+          -0.0064801485277712345,
+          0.012990990653634071,
+          -0.008709258399903774,
+          0.02535276673734188,
+          -0.00811073649674654,
+          -0.0006037971470504999,
+          -0.010497150011360645,
+          -0.013198170810937881,
+          0.008762971498072147,
+          -0.003247747663408518,
+          0.0017994018271565437,
+          -0.011732560582458973,
+          0.004354629199951887,
+          0.01692742109298706,
+          -0.01863090693950653,
+          -0.02177698165178299,
+          -0.023449774831533432,
+          0.021976489573717117,
+          0.03766082972288132,
+          0.0009236801415681839,
+          -0.050214435905218124,
+          -0.0021063359454274178,
+          -0.007677191868424416,
+          -0.012077861465513706,
+          -0.02265174500644207,
+          0.011487012729048729,
+          0.018139811232686043,
+          -0.0008004268747754395,
+          -0.0019893173594027758,
+          0.0031978709157556295,
+          0.015108836814761162,
+          -0.0013207762967795134,
+          0.0053866952657699585,
+          0.00251494231633842,
+          -0.0075007048435509205,
+          0.004155121743679047,
+          0.029941432178020477,
+          -0.027731506153941154,
+          0.00941520743072033,
+          0.002618532394990325,
+          0.011755580082535744,
+          0.004669236950576305,
+          0.009177332744002342,
+          0.01470982190221548,
+          0.0032400742638856173,
+          -0.00709401722997427,
+          0.0196591354906559,
+          0.07206815481185913,
+          -0.0028314681258052588,
+          0.015385077334940434,
+          0.00972981471568346,
+          -0.012822176329791546,
+          -0.004040021449327469,
+          -0.010312989354133606,
+          -0.03848955035209656,
+          0.025398805737495422,
+          0.008034002967178822,
+          -0.012077861465513706,
+          0.006483985111117363,
+          -0.014126647263765335,
+          0.008878071792423725,
+          0.029020629823207855,
+          0.010366703383624554,
+          -0.020794792100787163,
+          0.004281732253730297,
+          0.03453009948134422,
+          0.0214547012001276,
+          0.004542626440525055,
+          -0.01780218444764614,
+          0.006219254340976477,
+          -0.0014195707626640797,
+          -0.0038712078239768744,
+          0.009614714421331882,
+          -0.004465892910957336,
+          0.022375503554940224,
+          -0.004078388214111328,
+          0.014111299999058247,
+          -0.013535798527300358,
+          -0.013911793008446693,
+          0.006115664262324572,
+          -0.001624832977540791,
+          0.0014790392015129328,
+          0.020150231197476387,
+          0.0028314681258052588,
+          -0.006760226096957922,
+          0.0024861672427505255,
+          -0.0005850933375768363,
+          0.013681592419743538,
+          0.004818866960704327,
+          0.007423971313983202,
+          0.006614432204514742,
+          -0.023050758987665176,
+          0.01879972033202648,
+          0.006434108596295118,
+          0.00976818148046732,
+          -0.013037030585110188,
+          -0.005682119634002447,
+          -0.005720486398786306,
+          -0.004757480230182409,
+          0.0018454419914633036,
+          -0.00196437886916101,
+          -0.01573037914931774,
+          -0.004435199312865734,
+          0.009960015304386616,
+          -0.024662164971232414,
+          0.0048764171078801155,
+          0.010842450894415379,
+          0.004991517867892981,
+          0.0009601286146789789,
+          -0.01124146580696106,
+          -0.03256572037935257,
+          5.1435461500659585e-05,
+          -0.005647589452564716,
+          -0.0021255193278193474,
+          0.004853397142142057,
+          0.010919184423983097,
+          0.0222373828291893,
+          0.004396832548081875,
+          -0.0013831222895532846,
+          0.021715596318244934,
+          0.0007064282544888556,
+          -0.0033014612272381783,
+          0.016866035759449005,
+          -0.00735491095110774,
+          -0.013129111379384995,
+          0.00012517160212155432,
+          0.003560436889529228,
+          0.014755862765014172,
+          0.0057358331978321075,
+          0.01617543213069439,
+          0.010351356118917465,
+          -0.02555227279663086,
+          -0.020840832963585854,
+          9.196036990033463e-05,
+          0.026319608092308044,
+          -0.012653362937271595,
+          -0.01912200078368187,
+          -0.00412826519459486,
+          -0.005187188275158405,
+          0.006925203371793032,
+          -0.012231328524649143,
+          0.008977825753390789,
+          0.016451673582196236,
+          -0.011364239268004894,
+          -0.026273569092154503,
+          -0.020472511649131775,
+          -0.005847096908837557,
+          0.009775854647159576,
+          0.009814221411943436,
+          -0.006741042714565992,
+          0.012983317486941814,
+          0.002706776140257716,
+          -0.021761635318398476,
+          0.022835904732346535,
+          0.013489758595824242,
+          0.005689792800694704,
+          -0.005229391623288393,
+          -0.004523443058133125,
+          -0.011479339562356472,
+          0.0068753263913095,
+          0.004070715047419071,
+          -0.007293524220585823,
+          -0.030923621729016304,
+          -0.0024880855344235897,
+          -0.006537698674947023,
+          -0.00125747115816921,
+          -0.009514960460364819,
+          0.00602742051705718,
+          -0.029972126707434654,
+          0.004020838066935539,
+          -0.0060926442965865135,
+          -0.0041819787584245205,
+          -0.006449454929679632,
+          -0.0009423839510418475,
+          -0.009238719940185547,
+          -0.028237948194146156,
+          -0.027271104976534843,
+          0.0011807375121861696,
+          0.01199345476925373,
+          -0.0016392205143347383,
+          -0.002380658406764269,
+          -0.024247802793979645,
+          -0.0004181978583801538,
+          0.008640198037028313,
+          -0.00744699127972126,
+          0.01846209354698658,
+          -0.003890391206368804,
+          -0.006952059920877218,
+          -0.0004949314170517027,
+          -0.0012392469216138124,
+          -0.026749316602945328,
+          0.017080888152122498,
+          0.0031000354792922735,
+          0.014326154254376888,
+          -0.02435522899031639,
+          0.011279832571744919,
+          -0.013366985134780407,
+          0.007780782412737608,
+          -0.01841605268418789,
+          -0.008762971498072147,
+          -0.00786518957465887,
+          -0.012392468750476837,
+          -0.007826822809875011,
+          -0.029143404215574265,
+          0.00844069104641676,
+          -0.008801338262856007,
+          -0.025813167914748192,
+          -0.02908201701939106,
+          0.009353820234537125,
+          -0.015776418149471283,
+          0.015561564825475216,
+          -0.009215699508786201,
+          0.04527279734611511,
+          0.020027456805109978,
+          0.015354383736848831,
+          0.008824358694255352,
+          -0.017863571643829346,
+          -0.0032937878277152777,
+          0.012653362937271595,
+          -0.026273569092154503,
+          0.0003601681091822684,
+          0.012553608976304531,
+          0.01487096305936575,
+          -0.011318199336528778,
+          0.001353388070128858,
+          0.01651306077837944,
+          0.015177897177636623,
+          0.01622147299349308,
+          0.0024036786053329706,
+          -0.009131292812526226,
+          0.010305316187441349,
+          0.023633934557437897,
+          0.02567504718899727,
+          -0.004803520627319813,
+          0.0019662973936647177,
+          0.0011625132756307721,
+          -0.018477439880371094,
+          -0.009077579714357853,
+          -0.013205844908952713,
+          -0.009783527813851833,
+          -0.02659584954380989,
+          0.011525379493832588,
+          -0.0038002291694283485,
+          -0.010013728402554989,
+          -0.01580711267888546,
+          -0.014978389255702496,
+          -0.005133474711328745,
+          0.006599085405468941,
+          0.021546781063079834,
+          -0.010712004266679287,
+          0.010919184423983097,
+          -0.010052095167338848,
+          0.0020545409061014652,
+          -0.00196437886916101,
+          0.0081491032615304,
+          0.027225064113736153,
+          0.010274622589349747,
+          -0.055524397641420364,
+          -0.006499331910163164,
+          0.002762407995760441,
+          -0.010650617070496082,
+          -0.051503561437129974,
+          0.002656899392604828,
+          0.007324217818677425,
+          -0.03185977041721344,
+          0.008655544370412827,
+          -0.0023154348600655794,
+          -0.0079265758395195,
+          0.006595248822122812,
+          0.019935376942157745,
+          -0.016206126660108566,
+          0.013259558007121086,
+          -6.013542588334531e-06,
+          0.004320099018514156,
+          -0.014924676157534122,
+          0.0062230913899838924,
+          -0.015147203579545021,
+          -0.014471948146820068,
+          0.01734178327023983,
+          -0.001127024064771831,
+          -0.002089071087539196,
+          0.012008801102638245,
+          0.02506117895245552,
+          0.02579781971871853,
+          -0.012162268161773682,
+          -0.01592988520860672,
+          -0.01879972033202648,
+          0.0007073874585330486,
+          -0.0055401623249053955,
+          -0.010712004266679287,
+          -0.0019624605774879456,
+          0.008417670615017414,
+          0.01209320779889822,
+          0.01987398974597454,
+          0.008417670615017414,
+          0.006315171252936125,
+          -3.35109798470512e-05,
+          0.009898628108203411,
+          0.0023422916419804096,
+          -0.020303698256611824,
+          -0.0065530454739928246,
+          0.012983317486941814,
+          0.019797256216406822,
+          0.0017936468357220292,
+          0.004297079052776098,
+          0.018185852095484734,
+          -0.00773857906460762,
+          -0.009208026342093945,
+          -0.0336092971265316,
+          -0.0006901223678141832,
+          -0.017203662544488907,
+          0.016236819326877594,
+          0.01945962943136692,
+          -0.02136262133717537,
+          -0.01022858265787363,
+          -0.01866159960627556,
+          0.02394086867570877,
+          -0.010735023766756058,
+          0.005912320222705603,
+          0.0022252730559557676,
+          0.008739951997995377,
+          0.0019068288384005427,
+          -0.008517424575984478,
+          -0.014272441156208515,
+          0.009277086704969406,
+          0.00048318158951587975,
+          0.007661845069378614,
+          -0.0061195008456707,
+          -0.012039494700729847,
+          0.001056045526638627,
+          -0.009860261343419552,
+          -0.028283987194299698,
+          0.011579093523323536,
+          0.0008670891402289271,
+          0.01821654476225376,
+          -0.019597750157117844,
+          0.01675860770046711,
+          0.0007380808820016682,
+          -0.013520452193915844,
+          -0.02788497321307659,
+          -0.015277650207281113,
+          0.003919166047126055,
+          0.003792555769905448,
+          0.012661036103963852,
+          0.0034817850682884455,
+          -0.01658979430794716,
+          0.008739951997995377,
+          -0.005647589452564716,
+          -0.005751179996877909,
+          -0.001697729923762381,
+          -0.01170186698436737,
+          -0.001447386690415442,
+          0.011072651483118534,
+          -0.01037437655031681,
+          -0.005574692506343126,
+          0.01468680240213871,
+          0.006411088164895773,
+          -0.002988772001117468,
+          0.02742457203567028,
+          0.01821654476225376,
+          0.02153143472969532,
+          0.0006488781073130667,
+          0.01066596433520317,
+          -0.0018473602831363678,
+          0.003558518597856164,
+          -0.03505188599228859,
+          0.025122566148638725,
+          -0.01576107181608677,
+          0.023373039439320564,
+          0.008532770909368992,
+          -0.00694822333753109,
+          -0.005881627090275288,
+          0.01331327110528946,
+          0.006088807247579098,
+          -0.004803520627319813,
+          0.00602742051705718,
+          -0.003128810552880168,
+          0.009844915010035038,
+          -0.0013102254597470164,
+          -0.02493840456008911,
+          0.020012110471725464,
+          -0.024416616186499596,
+          0.009990708902478218,
+          -0.002990690292790532,
+          0.009123619645833969,
+          -0.006894509773701429,
+          -5.931863415753469e-05,
+          -0.00457331957295537,
+          -0.040638089179992676,
+          -0.001970133977010846,
+          -0.010773390531539917,
+          0.004450546111911535,
+          0.016819994896650314,
+          -0.00021497384295798838,
+          -0.04220345616340637,
+          -0.005916156806051731,
+          0.01958240196108818,
+          -0.00021737176575697958,
+          0.009883281774818897,
+          -0.008271876722574234,
+          -0.004086061846464872,
+          -0.004320099018514156,
+          -0.007792292162775993,
+          0.016543753445148468,
+          0.013136784546077251,
+          -0.01563829742372036,
+          -0.023618588224053383,
+          0.012507569044828415,
+          -0.01879972033202648,
+          -0.019567055627703667,
+          0.00530996173620224,
+          -0.01267638336867094,
+          0.001553854439407587,
+          0.022268077358603477,
+          -0.004715276882052422,
+          0.011164732277393341,
+          -0.0056130592711269855,
+          0.013282578438520432,
+          -0.01543111726641655,
+          -0.0019471138948574662,
+          -0.0047651538625359535,
+          -0.02348046749830246,
+          0.025567619130015373,
+          -0.018937841057777405,
+          0.0019931539427489042,
+          -0.016528407111763954,
+          0.02228342369198799,
+          -0.020088844001293182,
+          -0.002275149803608656,
+          0.008010982535779476,
+          0.010174869559705257,
+          0.029803311452269554,
+          0.01345906499773264,
+          -0.007408624514937401,
+          0.009169659577310085,
+          0.008824358694255352,
+          0.019827950745821,
+          0.00156632368452847,
+          0.029803311452269554,
+          -0.005271594971418381,
+          0.01041274331510067,
+          -0.001517405966296792,
+          0.018968533724546432,
+          0.0067065125331282616,
+          0.029926085844635963,
+          0.0003174850717186928,
+          0.00015430638450197875,
+          -0.0001380004978273064,
+          0.018600212410092354,
+          -0.004369975998997688,
+          -0.005313798785209656,
+          0.007765435613691807,
+          -0.0059698703698813915,
+          -0.0015797520754858851,
+          -0.020441818982362747,
+          -0.005877790041267872,
+          0.001410938217304647,
+          0.013789019547402859,
+          -0.0005241860635578632,
+          0.005390532314777374,
+          0.0008200898300856352,
+          0.005586202722042799,
+          -0.014617742039263248,
+          0.011402606032788754,
+          -0.015668991953134537,
+          0.01883041486144066,
+          -0.01005976926535368,
+          0.01008278876543045,
+          -0.0009692406747490168,
+          0.00811073649674654,
+          0.011072651483118534,
+          -0.013574165292084217,
+          0.01614473946392536,
+          0.012139247730374336,
+          -0.010589230805635452,
+          0.016313552856445312,
+          -0.007228300906717777,
+          0.005083597730845213,
+          0.0038616161327809095,
+          0.0005870116874575615,
+          0.003153749043121934,
+          -0.018109118565917015,
+          -0.009461247362196445,
+          -0.0001405183138558641,
+          -0.012576629407703876,
+          -0.0030501587316393852,
+          0.012331082485616207,
+          -0.005505632609128952,
+          0.015507850795984268,
+          0.010197889059782028,
+          0.012285041622817516,
+          0.004933967255055904,
+          0.011172405444085598,
+          -0.0014186116168275476,
+          -0.0034299897961318493,
+          0.004684583283960819,
+          0.032964736223220825,
+          0.0008824358810670674,
+          0.025905247777700424,
+          -0.014978389255702496,
+          0.006084970664232969,
+          0.001282409531995654,
+          -0.0016181188402697444,
+          -0.008701585233211517,
+          0.016236819326877594,
+          0.020088844001293182,
+          0.0018934004474431276,
+          -0.011893700808286667,
+          -0.003499049926176667,
+          0.029404297471046448,
+          -0.005501795560121536,
+          0.007454664912074804,
+          0.010696657001972198,
+          -0.006196234375238419,
+          0.016605140641331673,
+          0.004381485749036074,
+          -0.024692857638001442,
+          0.0013457146706059575,
+          0.004047695081681013,
+          -0.019152695313096046,
+          0.020917566493153572,
+          0.02104034088551998,
+          0.0076042949222028255,
+          0.002739387797191739,
+          0.01345906499773264,
+          0.010497150011360645,
+          0.021623514592647552,
+          0.0032400742638856173,
+          0.004580993205308914,
+          0.002643470885232091,
+          -0.012952623888850212,
+          -0.00123445107601583,
+          -0.020963607355952263,
+          -0.011533052660524845,
+          -0.0020583774894475937,
+          0.018983881920576096,
+          0.0026530625764280558,
+          -0.013290251605212688,
+          -0.003959451336413622,
+          0.008855052292346954,
+          0.004381485749036074,
+          -0.006299824919551611,
+          -0.022053223103284836,
+          -0.008202817291021347,
+          0.024094335734844208,
+          0.008095390163362026,
+          0.010550864040851593,
+          -0.0031729324255138636,
+          -0.002160049509257078,
+          -0.0006109909154474735,
+          -0.006614432204514742,
+          -0.0006450414075516164,
+          -0.01597592607140541,
+          -0.0021063359454274178,
+          -0.016666527837514877,
+          0.015346710570156574,
+          0.015791764482855797,
+          0.01763337105512619,
+          0.018139811232686043,
+          -0.02102499268949032,
+          0.017602676525712013,
+          -0.0071438937447965145,
+          0.001838727737776935,
+          -0.0034414997790008783,
+          0.00010826624929904938,
+          0.019275467842817307,
+          0.029388951137661934,
+          0.0056015495210886,
+          0.011149385012686253,
+          -0.0013831222895532846,
+          0.005927667021751404,
+          -0.006457128562033176,
+          -0.013980853371322155,
+          -0.0033935413230210543,
+          -0.01892249472439289,
+          0.004907110705971718,
+          0.017848225310444832,
+          -0.015139530412852764,
+          -0.004922457505017519,
+          0.014349174685776234,
+          0.009867934510111809,
+          0.008340937085449696,
+          -0.008931785821914673,
+          -0.004112918395549059,
+          0.0008714054129086435,
+          0.01557691115885973,
+          -0.007585111539810896,
+          -0.016605140641331673,
+          0.008394651114940643,
+          -0.006146357394754887,
+          0.02123984694480896,
+          -0.004542626440525055,
+          -0.01605265960097313,
+          -0.010282295756042004,
+          0.003186360700055957,
+          -0.021439354866743088,
+          -0.014272441156208515,
+          0.01002140250056982,
+          0.0021312744356691837,
+          0.006936713121831417,
+          -0.017326436936855316,
+          0.013060051016509533,
+          0.004834213759750128,
+          0.03351721540093422,
+          -0.008133756928145885,
+          -0.0005424103001132607,
+          0.0003779127437155694,
+          0.005916156806051731,
+          0.0002481850970070809,
+          -0.01755663752555847,
+          0.026933476328849792,
+          0.006587575655430555,
+          0.013781346380710602,
+          -0.0003863054735120386,
+          -0.012645689770579338,
+          0.006994263269007206,
+          -0.01564597152173519,
+          0.001863666228018701,
+          -0.010205562226474285,
+          -0.020088844001293182,
+          -0.01058155670762062,
+          0.02165420912206173,
+          0.0070403036661446095,
+          -0.00732038076967001,
+          -0.0001907548139570281,
+          -0.0077769458293914795,
+          -0.016359593719244003,
+          -0.009852588176727295,
+          -0.015492504462599754,
+          0.008386977016925812,
+          -0.01663583517074585,
+          0.0029005282558500767,
+          0.008126083761453629,
+          0.009806548245251179,
+          0.005586202722042799,
+          0.021623514592647552,
+          0.002338455058634281,
+          0.016405634582042694,
+          0.0001647373428568244,
+          -0.0019049104303121567,
+          0.005651426035910845,
+          0.014771209098398685,
+          0.0072973608039319515,
+          -0.008555791340768337,
+          -0.0046423799358308315,
+          -0.012691729702055454,
+          0.0005198698490858078,
+          -0.009676100686192513,
+          0.01066596433520317,
+          0.0064302715472877026,
+          -0.0009404656011611223,
+          -0.00678324606269598,
+          0.021715596318244934,
+          -0.0038980646058917046,
+          0.007711722049862146,
+          -0.007957269437611103,
+          -0.004300915636122227,
+          0.01065829023718834,
+          0.0010848206002265215,
+          -0.013765999116003513,
+          0.008210490457713604,
+          0.00919268000870943,
+          -0.002263639820739627,
+          -0.003238155972212553,
+          -0.020672019571065903,
+          0.02874438837170601,
+          0.008885745890438557,
+          -0.010182542726397514,
+          0.014425908215343952,
+          -0.0052831051871180534,
+          0.019306162372231483,
+          0.016282860189676285,
+          0.0185388270765543,
+          -0.018323972821235657,
+          -0.019643789157271385,
+          0.007251320872455835,
+          -0.015561564825475216,
+          -0.0027835096698254347,
+          0.0029715069103986025,
+          -0.0022809049114584923,
+          -0.0002340373321203515,
+          -0.016068005934357643,
+          0.0009150475962087512,
+          -0.0046500531025230885,
+          0.0024938404094427824,
+          0.000545287795830518,
+          0.0077347420156002045,
+          0.0009395064553245902,
+          0.012760790064930916,
+          -0.006196234375238419,
+          -0.01783287711441517,
+          -0.006606759037822485,
+          0.013512779027223587,
+          0.006906019989401102,
+          -0.03600338473916054,
+          0.004980007652193308,
+          -0.012499895878136158,
+          -0.01751059666275978,
+          -0.002232946455478668,
+          -0.02157747559249401,
+          -0.02116311341524124,
+          -0.001316939597018063,
+          0.0009831486968323588,
+          -0.011510033160448074,
+          0.0021869062911719084,
+          0.006648962385952473,
+          -0.009300106205046177,
+          -0.00919268000870943,
+          0.0054595922119915485,
+          -0.0018751762108877301,
+          0.012645689770579338,
+          -0.012392468750476837,
+          -0.010443436913192272,
+          0.035174660384655,
+          0.013635552488267422,
+          -0.00766568211838603,
+          -0.013812039978802204,
+          -0.010873144492506981,
+          -0.014280114322900772,
+          0.010980571620166302,
+          -0.0030827706214040518,
+          -0.00466539990156889,
+          -0.037814296782016754,
+          0.0013543472159653902,
+          0.0011884109117090702,
+          0.01524695660918951,
+          -0.008862725459039211,
+          0.014533335343003273,
+          0.005793383345007896,
+          -0.00058077706489712,
+          -0.012354101985692978,
+          -6.549328645633068e-06,
+          0.01080408412963152,
+          0.010136502794921398,
+          -0.007197607308626175,
+          0.012100880965590477,
+          -0.013006336987018585,
+          0.007527561392635107,
+          -0.014633088372647762,
+          -0.0077769458293914795,
+          -0.01933685503900051,
+          -0.01787891797721386,
+          -0.007516051642596722,
+          0.01634424738585949,
+          0.009890954941511154,
+          -0.01787891797721386,
+          -0.004815030377358198,
+          -0.014525661244988441,
+          0.013451391831040382,
+          -0.011533052660524845,
+          -0.006015910767018795,
+          -0.01306772418320179,
+          0.001044535543769598,
+          -0.018784373998641968,
+          0.01385807991027832,
+          0.01775614358484745,
+          0.01639028638601303,
+          0.0004093255556654185,
+          -0.0053866952657699585,
+          -0.02248293161392212,
+          -0.01870764046907425,
+          -0.004462056327611208,
+          0.002572492463514209,
+          -0.013781346380710602,
+          -0.003744597313925624,
+          0.011732560582458973,
+          0.021915102377533913,
+          -0.0017984426813200116,
+          -0.013129111379384995,
+          0.00854811817407608,
+          -0.016958115622401237,
+          -0.017311088740825653,
+          -0.018277931958436966,
+          0.007358747534453869,
+          0.008172123692929745,
+          0.01605265960097313,
+          0.0024746570270508528,
+          0.006058114115148783,
+          -0.004623196553438902,
+          -0.007730905432254076,
+          -0.024278495460748672,
+          0.005110454745590687,
+          -0.0112030990421772,
+          0.010075115598738194,
+          -0.0021408661268651485,
+          -0.004070715047419071,
+          0.023465121164917946,
+          0.009760508313775063,
+          -0.004995354451239109,
+          -0.00018631866259966046,
+          0.020840832963585854,
+          -0.008686237968504429,
+          -0.00602742051705718,
+          -0.008057023398578167,
+          0.017771491780877113,
+          -0.021961143240332603,
+          0.00016090067219920456,
+          -0.011187751777470112,
+          -0.0222373828291893,
+          0.003272686153650284,
+          0.012200634926557541,
+          0.00606578728184104,
+          -0.003355174558237195,
+          0.003453009994700551,
+          -0.008433017879724503,
+          0.024094335734844208,
+          -0.01435684785246849,
+          0.002348046749830246,
+          0.013620206154882908,
+          -0.0021638860926032066,
+          -0.002465065335854888,
+          0.013727632351219654,
+          -0.005417388863861561,
+          0.009530307725071907,
+          0.0035738651640713215,
+          0.00011857732170028612,
+          -0.009660754352807999,
+          0.016988808289170265,
+          0.02535276673734188,
+          -0.00860950443893671,
+          0.0008152939844876528,
+          -0.003782964078709483,
+          0.00602742051705718,
+          -0.0070403036661446095,
+          0.03201323747634888,
+          -0.008287223987281322,
+          0.0005500836414285004,
+          0.0010100053623318672,
+          0.03161422535777092,
+          -0.0014771209098398685,
+          0.010044422000646591,
+          0.0014541008276864886,
+          -0.0034184798132628202,
+          0.022927986457943916,
+          0.0025667373556643724,
+          0.004531116224825382,
+          0.027102291584014893,
+          -0.013574165292084217,
+          -0.00493780430406332,
+          0.005543999373912811,
+          -0.015200916677713394,
+          0.012392468750476837,
+          -0.02048785798251629,
+          0.0009188842959702015,
+          -0.01522393710911274,
+          0.014280114322900772,
+          0.019981417804956436,
+          0.030386487022042274,
+          -0.014418235048651695,
+          -0.00619239779189229,
+          -0.00728201400488615,
+          0.03143006190657616,
+          0.00044217711547389627,
+          0.006288314703851938,
+          -0.00030045982566662133,
+          0.011471666395664215,
+          -0.015177897177636623,
+          -0.01962844282388687,
+          0.012499895878136158,
+          -0.019551709294319153,
+          -0.03413108363747597,
+          -0.004634706769138575,
+          0.005670609418302774,
+          -0.0030175470747053623,
+          0.0037100673653185368,
+          0.012146921828389168,
+          0.004358465783298016,
+          0.022145302966237068,
+          0.007243647240102291,
+          0.009752834215760231,
+          -0.016482368111610413,
+          3.229703224860714e-06,
+          -0.006848469842225313,
+          -0.025398805737495422,
+          0.016037313267588615,
+          -0.021516088396310806,
+          0.0014138157712295651,
+          -0.008433017879724503,
+          -0.002647307701408863,
+          0.009967688471078873,
+          0.004780500195920467,
+          0.014778882265090942,
+          0.005789546761661768,
+          0.016282860189676285,
+          0.019551709294319153,
+          -0.01573037914931774,
+          0.011586766690015793,
+          0.027025558054447174,
+          0.00113853404764086,
+          -0.017188316211104393,
+          -0.007784618996083736,
+          -0.01975121721625328,
+          -0.01945962943136692,
+          0.011256812140345573,
+          0.02041112445294857,
+          -0.00846371054649353,
+          -0.001863666228018701,
+          0.03904203325510025,
+          -0.013113764114677906,
+          0.023956215009093285,
+          -0.00775776244699955,
+          0.005240901838988066,
+          0.03468356654047966,
+          0.010827104561030865,
+          0.018692294135689735,
+          -0.00613868422806263,
+          -0.034407325088977814,
+          -0.002121682744473219,
+          -0.00680242944508791,
+          0.0065607186406850815,
+          -0.011049631983041763,
+          0.018032385036349297,
+          -0.013673919253051281,
+          -0.019812602549791336,
+          0.02036508545279503,
+          -0.02243689075112343,
+          0.021377967670559883,
+          0.007562091574072838,
+          0.008578810840845108,
+          -0.006269131321460009,
+          -0.011479339562356472,
+          -3.974558421759866e-05,
+          -0.0077270688489079475,
+          0.011318199336528778,
+          0.008034002967178822,
+          -0.011832313612103462,
+          -0.006806266028434038,
+          0.0031729324255138636,
+          -0.004634706769138575,
+          -0.02311214618384838,
+          0.012860543094575405,
+          0.008640198037028313,
+          0.013428371399641037,
+          0.007381767965853214,
+          0.005075924564152956,
+          0.010635270737111568,
+          0.0034817850682884455,
+          -0.013113764114677906,
+          -0.01203182153403759,
+          0.00914663914591074,
+          -0.006127174012362957,
+          -0.0007332850364036858,
+          -0.026687929406762123,
+          0.014702148735523224,
+          -0.022743824869394302,
+          -0.0032573393546044827,
+          -0.0018195443553850055,
+          0.0039786347188055515,
+          0.005486449226737022,
+          -0.01866159960627556,
+          -0.005636079702526331,
+          -0.00028990895953029394,
+          -0.009392186999320984,
+          0.01188602764159441,
+          -0.013620206154882908,
+          0.01646701991558075,
+          0.027056250721216202,
+          -0.01487096305936575,
+          -0.0036199053283780813,
+          0.00554783595725894,
+          0.0077040488831698895,
+          -0.012522916309535503,
+          0.016988808289170265,
+          0.015108836814761162,
+          -0.01205484103411436,
+          0.008847379125654697,
+          0.03083154186606407,
+          -0.006579902023077011,
+          0.0005088393809273839,
+          -0.00012493181566242129,
+          0.0024995955172926188,
+          0.029726577922701836,
+          0.0016066087409853935,
+          0.024293843656778336,
+          0.005873953457921743,
+          -0.0008862725226208568,
+          -0.014610068872570992,
+          -0.014364521019160748,
+          -0.009522633627057076,
+          -0.009990708902478218,
+          -0.005520978942513466,
+          -0.010497150011360645,
+          0.009706794284284115,
+          0.012699402868747711,
+          -0.004676910117268562,
+          0.0038884729146957397,
+          -0.0019164205295965075,
+          0.02131658047437668,
+          -0.005052904598414898,
+          0.013113764114677906,
+          -0.022206690162420273,
+          0.01962844282388687,
+          0.020595286041498184,
+          0.011839987710118294,
+          -0.016313552856445312,
+          -0.005643752869218588,
+          0.016819994896650314,
+          0.025214646011590958,
+          -0.00538285868242383,
+          0.001968215685337782,
+          0.019444283097982407,
+          0.0031384022440761328,
+          0.005252411589026451,
+          -0.0026626542676240206,
+          -0.01987398974597454,
+          -0.027194371446967125,
+          -0.003568110289052129,
+          -0.006737206131219864,
+          0.002689511049538851,
+          0.004097571596503258,
+          -0.011839987710118294,
+          -0.011609786190092564,
+          0.0046500531025230885,
+          -0.005026047583669424,
+          0.0011020856909453869,
+          -0.0099139753729105,
+          -0.007849842309951782,
+          -0.038059841841459274,
+          0.020150231197476387,
+          0.0004304272879380733,
+          0.004396832548081875,
+          0.004239528905600309,
+          0.015108836814761162,
+          -0.007723232265561819,
+          -0.0016229146858677268,
+          0.015085816383361816,
+          0.006069623865187168,
+          -0.0018233810551464558,
+          -0.0003510560200084001,
+          -0.0035719468723982573,
+          -0.007197607308626175,
+          0.0104741295799613,
+          -0.004980007652193308,
+          -0.010274622589349747,
+          0.0084023242816329,
+          -0.012852869927883148,
+          -0.015062796883285046,
+          0.008225836791098118,
+          -0.025153258815407753,
+          0.00501070125028491,
+          -0.000793712679296732,
+          0.010159522294998169,
+          -0.004661563318222761,
+          -0.008809012360870838,
+          0.0010032912250608206,
+          0.013873426243662834,
+          0.007032630033791065,
+          -0.004726787097752094,
+          0.016083352267742157,
+          -0.007400951348245144,
+          0.007124710362404585,
+          0.029220137745141983,
+          -0.007949596270918846,
+          0.008601831272244453,
+          0.018477439880371094,
+          0.0013687347527593374,
+          0.008471384644508362,
+          0.01130285207182169,
+          -0.03643308952450752,
+          -0.016743261367082596,
+          -0.0008809971041046083,
+          0.0051603312604129314,
+          -0.0011174323735758662,
+          0.0188764538615942,
+          0.00919268000870943,
+          0.02725575864315033,
+          -0.006177050992846489,
+          0.012684056535363197,
+          0.0019509505946189165,
+          0.023572547361254692,
+          0.012108555063605309,
+          0.01199345476925373,
+          -0.005747342947870493,
+          -0.0021312744356691837,
+          -0.03588061034679413,
+          0.0035661919973790646,
+          0.022175997495651245,
+          0.019490322098135948,
+          -0.02579781971871853,
+          0.017203662544488907,
+          -0.0057435063645243645,
+          0.0033129712101072073,
+          -0.013428371399641037,
+          0.017694758251309395,
+          0.004392995964735746,
+          0.017571983858942986,
+          -0.008156776428222656,
+          -0.002417106879875064,
+          0.016866035759449005,
+          0.010466456413269043,
+          -0.012914257124066353,
+          0.021470047533512115,
+          0.012806829996407032,
+          0.02119380794465542,
+          0.019198734313249588,
+          -0.018308626487851143,
+          0.00908525288105011,
+          0.00875529833137989,
+          -0.012200634926557541,
+          -0.0081491032615304,
+          0.00457331957295537,
+          0.007232137490063906,
+          -0.016605140641331673,
+          -0.005018374416977167,
+          -0.0005678283050656319,
+          0.00023475672060158104,
+          0.012768463231623173,
+          -0.012668709270656109,
+          0.000462079857243225,
+          0.00412826519459486,
+          0.011080324649810791,
+          -0.0065607186406850815,
+          0.01734178327023983,
+          -0.028729042038321495,
+          0.012223655357956886,
+          -0.022022530436515808,
+          -0.0239715613424778,
+          0.0014195707626640797,
+          -0.004481239710003138,
+          -0.0036333338357508183,
+          -0.0051027811132371426,
+          0.0015154876746237278,
+          0.005647589452564716,
+          0.009983035735785961,
+          0.028391415253281593,
+          0.007715558633208275,
+          0.0013601022073999047,
+          -0.0004165193240623921,
+          -0.02302006632089615,
+          -0.010827104561030865,
+          -0.006054277531802654,
+          -0.0021389478351920843,
+          -0.006445618346333504,
+          -0.0336092971265316,
+          -0.00022852211259305477,
+          -0.0057358331978321075,
+          0.0029216299299150705,
+          0.0008435894851572812,
+          -0.02007349766790867,
+          -0.021500742062926292,
+          0.00926941353827715,
+          -0.008870398625731468,
+          -0.014026893302798271,
+          -0.0168046485632658,
+          0.006472475361078978,
+          0.00037431585951708257,
+          -0.0036563538014888763,
+          0.010151849128305912,
+          0.01846209354698658,
+          -0.004676910117268562,
+          -0.0006843673763796687,
+          0.010090461932122707,
+          0.0026396343018859625,
+          -0.02165420912206173,
+          0.008617177605628967,
+          -0.014111299999058247,
+          0.003951778169721365,
+          -0.01130285207182169,
+          -0.01709623634815216,
+          -0.004964660853147507,
+          -0.003967124503105879,
+          -0.028391415253281593,
+          0.0011673092376440763,
+          0.011832313612103462,
+          -0.007067160215228796,
+          0.013635552488267422,
+          0.020441818982362747,
+          0.010405070148408413,
+          0.00084119156235829,
+          -0.014863288961350918,
+          0.011425626464188099,
+          -0.01457937527447939,
+          -0.012998663820326328,
+          0.01970517635345459,
+          0.012929603457450867,
+          0.0112030990421772,
+          -0.0008503036806359887,
+          -0.014019220136106014,
+          0.011072651483118534,
+          0.015745725482702255,
+          0.025245338678359985,
+          -0.006272967904806137,
+          -0.0007697334513068199,
+          -0.0162982065230608,
+          -0.018446745350956917,
+          0.034161776304244995,
+          -0.011932067573070526,
+          -0.013681592419743538,
+          0.01655910164117813,
+          0.019613096490502357,
+          0.012668709270656109,
+          -0.005616896320134401,
+          -0.004684583283960819,
+          -0.0014684883644804358,
+          0.012346428819000721,
+          0.005820239894092083,
+          -0.025199299678206444,
+          -0.010681310668587685,
+          0.004400669131428003,
+          0.0325964130461216,
+          0.011747906915843487,
+          0.006069623865187168,
+          -0.043001484125852585,
+          0.014172687195241451,
+          0.0008853133767843246,
+          0.0009505368652753532,
+          0.013980853371322155,
+          0.029481031000614166,
+          0.020257657393813133,
+          0.0009706794517114758,
+          0.01962844282388687,
+          -0.015914538875222206,
+          0.02269778586924076,
+          -0.020978953689336777,
+          -0.010919184423983097,
+          -0.0241403765976429,
+          -0.005006864201277494,
+          -0.008479057811200619,
+          0.0028257130179554224,
+          0.009031538851559162,
+          0.009637733921408653,
+          -0.012262022122740746,
+          0.02021161839365959,
+          0.013221191242337227,
+          -0.010044422000646591,
+          -0.001552895293571055,
+          -0.02070271223783493,
+          0.02547553926706314,
+          0.003771454095840454,
+          -0.0008680483442731202,
+          -0.010612250305712223,
+          -0.005018374416977167,
+          -0.021132420748472214,
+          0.0018368094461038709,
+          -0.0028506515081971884,
+          0.012906583957374096,
+          -0.0016085271490737796,
+          0.008348610252141953,
+          0.016942769289016724,
+          -0.015208589844405651,
+          -0.003878881223499775,
+          0.007846006192266941,
+          0.02104034088551998,
+          0.006437945179641247,
+          0.01190904714167118,
+          0.02854488231241703,
+          -0.0013936731265857816,
+          -0.027485959231853485,
+          0.012008801102638245,
+          -0.006403414998203516,
+          -0.003366684541106224,
+          0.01111869141459465,
+          0.02131658047437668,
+          -0.021746288985013962,
+          0.0011557991383597255,
+          -0.007335727568715811,
+          0.020150231197476387,
+          3.1173007300822064e-05,
+          0.013198170810937881,
+          0.009138965979218483,
+          0.003936431370675564,
+          -0.01111869141459465,
+          -0.01421105396002531,
+          -0.008026329800486565,
+          0.01170186698436737,
+          -0.026979517191648483,
+          -0.0024036786053329706,
+          0.001851196982897818,
+          -0.003021383658051491,
+          0.006311334669589996,
+          0.006230764556676149,
+          -0.022099263966083527,
+          -0.010635270737111568,
+          -0.0028564066160470247,
+          -0.010182542726397514,
+          -0.005206371657550335,
+          0.012476875446736813,
+          -0.005409715697169304,
+          0.0005798179190605879,
+          0.00875529833137989,
+          0.009630060754716396,
+          0.004300915636122227,
+          0.025506233796477318,
+          -0.012584302574396133,
+          0.01924477517604828,
+          -0.030647380277514458,
+          0.001827217754907906,
+          -0.008578810840845108,
+          0.024171069264411926,
+          -0.0024497187696397305,
+          -0.019075961783528328,
+          0.012047167867422104,
+          -0.008195143193006516,
+          0.001614282140508294,
+          -0.002810366451740265,
+          -0.008433017879724503,
+          -0.0065300255082547665,
+          0.012545935809612274,
+          -0.018155159428715706,
+          0.004308588802814484,
+          0.025444846600294113,
+          0.0429094024002552,
+          -0.013643225654959679,
+          -0.013121437281370163,
+          0.0052255550399422646,
+          0.009983035735785961,
+          0.024094335734844208,
+          -0.01144864596426487,
+          -0.009307780303061008,
+          -0.01159443985670805,
+          0.0075812749564647675,
+          -0.016896728426218033,
+          -0.023833442479372025,
+          0.011563746258616447,
+          -0.015960579738020897,
+          -0.016850687563419342,
+          -0.018155159428715706,
+          -0.017955651506781578,
+          -0.0027662445791065693,
+          0.013773673214018345,
+          0.005904647056013346,
+          -0.0028046113438904285,
+          -0.0015615278389304876,
+          -0.006111827678978443,
+          -0.021270541474223137,
+          0.016896728426218033,
+          0.005133474711328745,
+          -0.01614473946392536,
+          -0.004373812582343817,
+          -0.012039494700729847,
+          0.002562900772318244,
+          0.0037522707134485245,
+          -0.003075097221881151,
+          0.05506399646401405,
+          0.003518233308568597,
+          0.009507287293672562,
+          -0.02107103355228901,
+          0.004266385454684496,
+          -0.010796410962939262,
+          -0.01074269786477089,
+          -0.007707885466516018,
+          -0.012116228230297565,
+          -0.003617987036705017,
+          -0.011479339562356472,
+          0.0019969907589256763,
+          -0.01543111726641655,
+          0.008133756928145885,
+          -0.013673919253051281,
+          -0.013474412262439728,
+          -0.017571983858942986,
+          -0.0015864662127569318,
+          -0.003451091470196843,
+          0.016114046797156334,
+          0.01325188484042883,
+          -0.016374940052628517,
+          -0.0030904437880963087,
+          0.022268077358603477,
+          0.007895882241427898,
+          0.018968533724546432,
+          -0.0035048050340265036,
+          -0.013129111379384995,
+          -0.020012110471725464,
+          0.0006565514486283064,
+          0.013075397349894047,
+          -0.0075812749564647675,
+          0.002915875054895878,
+          -0.029097363352775574,
+          -0.006748715881258249,
+          0.0014090199256315827,
+          0.01600661873817444,
+          0.01668187417089939,
+          0.008862725459039211,
+          -0.0018732579192146659,
+          -0.020856179296970367,
+          0.0332716703414917,
+          -0.007753925397992134,
+          -0.022743824869394302,
+          0.0019758890848606825,
+          0.007328054402023554,
+          -0.005144984927028418,
+          0.0024036786053329706,
+          0.013351637870073318,
+          0.004903274122625589,
+          -0.008340937085449696,
+          -0.004887927323579788,
+          0.0011836150661110878,
+          -0.021715596318244934,
+          0.01838536001741886,
+          -0.0033263994846493006,
+          0.00854811817407608,
+          -0.004838050343096256,
+          0.004696093499660492,
+          -0.023158187046647072,
+          0.014126647263765335,
+          -0.008164450526237488,
+          -0.0069098565727472305,
+          0.008878071792423725,
+          -0.0023653118405491114,
+          0.02231411822140217,
+          -0.020273005589842796,
+          -0.008908765390515327,
+          0.008855052292346954,
+          -0.022129956632852554,
+          -0.02206856943666935,
+          -0.0004939722712151706,
+          -0.019275467842817307,
+          -0.0014780800556764007,
+          -0.00021233613369986415,
+          -0.032964736223220825,
+          -0.026856742799282074,
+          0.003044403623789549,
+          -0.0073472377844154835,
+          -0.008287223987281322,
+          0.012753116898238659,
+          -7.697334513068199e-05,
+          -0.013850406743586063,
+          -0.003665945492684841,
+          0.01041274331510067,
+          0.00023139962286222726,
+          0.01800169236958027,
+          -0.021930448710918427,
+          0.0020372758153826,
+          0.004431362729519606,
+          -0.004327772185206413,
+          0.006725695915520191,
+          0.01712692901492119,
+          0.0005990013014525175,
+          0.0003234798787161708,
+          0.006084970664232969,
+          0.0034932950511574745,
+          0.010850124061107635,
+          0.015745725482702255,
+          0.015032103285193443,
+          -0.004174305126070976,
+          0.016068005934357643,
+          -0.001623873831704259,
+          2.7096535632153973e-05,
+          -0.02460077777504921,
+          2.6736846848507412e-05,
+          0.012415489181876183,
+          0.004450546111911535,
+          -0.02410968206822872,
+          -0.0031997892074286938,
+          -0.03278057277202606,
+          0.005133474711328745,
+          -0.019382895901799202,
+          0.0013802447356283665,
+          0.017571983858942986,
+          0.00728201400488615,
+          -0.004446709528565407,
+          0.014402887783944607,
+          0.022789865732192993,
+          -0.0010502905352041125,
+          0.0006623064982704818,
+          0.004515769425779581,
+          -0.003224727464839816,
+          0.002537962282076478,
+          0.0022080079652369022,
+          -0.01597592607140541,
+          -0.01414966769516468,
+          0.008156776428222656,
+          0.004611686337739229,
+          0.011310526169836521,
+          0.007385604549199343,
+          -0.015446464531123638,
+          0.012223655357956886,
+          -0.013934813439846039,
+          0.008164450526237488,
+          -0.00473829684779048,
+          -0.013988526538014412,
+          -0.013466738164424896,
+          0.005359838716685772,
+          0.017618024721741676,
+          -0.006012073718011379,
+          0.033425137400627136,
+          -0.00925406627357006,
+          0.01533136423677206,
+          -0.019306162372231483,
+          -0.009491940960288048,
+          -0.010282295756042004,
+          0.01184766087681055,
+          -0.01567666418850422,
+          -0.0009020988363772631,
+          0.0035508451983332634,
+          -0.010573883540928364,
+          -0.010351356118917465,
+          0.005294615402817726,
+          -0.0030309753492474556,
+          0.03956381976604462,
+          -0.0024631470441818237,
+          0.025705739855766296,
+          0.006031257100403309,
+          -0.011517706327140331,
+          -0.010075115598738194,
+          0.002643470885232091,
+          0.00800330936908722,
+          0.0180937722325325,
+          -0.006568392273038626,
+          -0.004354629199951887,
+          0.011041957885026932,
+          0.017955651506781578,
+          0.007251320872455835,
+          0.011563746258616447,
+          -0.004093735013157129,
+          0.0037311690393835306,
+          0.002374903531745076,
+          -0.013666246086359024,
+          -0.0031096271704882383,
+          0.011916721239686012,
+          0.017019502818584442,
+          0.00786518957465887,
+          0.012852869927883148,
+          -0.0003946982033085078,
+          0.004611686337739229,
+          0.013996199704706669,
+          -0.013121437281370163,
+          -0.005240901838988066,
+          0.008241184055805206,
+          -0.01721900887787342,
+          0.009223372675478458,
+          0.0044083427637815475,
+          -0.004638543352484703,
+          0.00893945898860693,
+          -0.005348328500986099,
+          -0.0018483194289729,
+          -0.02099430002272129,
+          -0.014134320430457592,
+          -0.0033494196832180023,
+          0.020932912826538086,
+          -0.006299824919551611,
+          0.0015452218940481544,
+          -0.006790919695049524,
+          -0.02567504718899727,
+          0.01705019548535347,
+          0.004550299607217312,
+          0.0065530454739928246,
+          0.005966033786535263
+        ],
+        "title": "2 Related Work",
+        "keyphrases": [
+          "neural network capacity",
+          "model parallelism",
+          "conditional computation",
+          "mixture of experts",
+          "Switch Transformer"
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "86d0cb2e-1f6d-4e38-a077-7b7de2fe9e4a",
+      "properties": {
+        "page_content": "3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n",
+        "embedding": [
+          0.010104401037096977,
+          -0.020574966445565224,
+          -0.02167346142232418,
+          -0.031542472541332245,
+          0.03588413819670677,
+          0.03464615345001221,
+          -0.009755672886967659,
+          0.046869076788425446,
+          0.004803731571882963,
+          0.0364421010017395,
+          0.016547156497836113,
+          -0.007223033811897039,
+          0.009546436369419098,
+          -0.0017371026333421469,
+          0.005762734450399876,
+          0.04048734903335571,
+          0.011603932827711105,
+          0.01784616895020008,
+          0.011150586418807507,
+          -0.02915368229150772,
+          0.005383492447435856,
+          -0.01708768494427204,
+          -0.04317255690693855,
+          -0.006194285582751036,
+          0.009546436369419098,
+          0.005871712230145931,
+          0.027340294793248177,
+          0.0036812627222388983,
+          -0.055901139974594116,
+          0.023800702765583992,
+          0.021708333864808083,
+          0.010984940454363823,
+          -0.01007824670523405,
+          -0.010601338930428028,
+          -0.026573093608021736,
+          -0.021969880908727646,
+          0.04634598642587662,
+          0.007685098797082901,
+          -0.006399163510650396,
+          0.042475100606679916,
+          0.03703493997454643,
+          0.012728581205010414,
+          -0.012772172689437866,
+          -0.014175803400576115,
+          -0.003430614247918129,
+          -0.0012979229213669896,
+          -0.004712190479040146,
+          0.0065735275857150555,
+          0.009311044588685036,
+          -0.026172054931521416,
+          -0.001842810888774693,
+          -0.021900134161114693,
+          0.02798544242978096,
+          0.020609840750694275,
+          -0.049379922449588776,
+          0.028072625398635864,
+          -0.004890913609415293,
+          0.06461934745311737,
+          0.047461915761232376,
+          0.014602995477616787,
+          0.03794163465499878,
+          -0.034035876393318176,
+          -0.029275737702846527,
+          -0.003177786245942116,
+          0.03234454616904259,
+          0.007654585409909487,
+          -0.01033979281783104,
+          0.0010679803090170026,
+          0.011717269197106361,
+          -0.005706066265702248,
+          -0.044532597064971924,
+          0.010496720671653748,
+          -0.014925569295883179,
+          0.002881367225199938,
+          0.004034350160509348,
+          -0.020017001777887344,
+          0.03030448593199253,
+          0.0268520750105381,
+          0.014899414964020252,
+          0.010740830563008785,
+          0.010653648525476456,
+          0.03204812481999397,
+          -0.0026961052790284157,
+          0.009555154480040073,
+          0.0364421010017395,
+          -0.029415227472782135,
+          -0.01360040158033371,
+          -0.062178246676921844,
+          0.010540311224758625,
+          0.0028072623535990715,
+          -0.02815980650484562,
+          0.0018340926617383957,
+          -0.006569168530404568,
+          0.0036986989434808493,
+          -0.033094312995672226,
+          0.0727447122335434,
+          0.010653648525476456,
+          0.00023280335881281644,
+          -0.024044813588261604,
+          0.006394804455339909,
+          -0.015762517228722572,
+          0.021969880908727646,
+          0.0006560450419783592,
+          0.004008195362985134,
+          0.0229811929166317,
+          -0.02859571762382984,
+          -0.004690395202487707,
+          0.03815086930990219,
+          0.02148166112601757,
+          0.08676359057426453,
+          0.01007824670523405,
+          -0.017593340948224068,
+          -0.011211613193154335,
+          0.018482597544789314,
+          -0.04278895631432533,
+          -0.017689241096377373,
+          -0.03965040296316147,
+          -0.03215274587273598,
+          0.0114818774163723,
+          -0.025038689374923706,
+          0.01996469311416149,
+          0.02566639892756939,
+          -0.0011344566009938717,
+          -0.025422289967536926,
+          -0.07155904173851013,
+          0.04292844980955124,
+          -0.023696085438132286,
+          0.01780257746577263,
+          0.001439593848772347,
+          -0.0022144243121147156,
+          -0.00422179140150547,
+          0.0037183149252086878,
+          -0.020261110737919807,
+          -0.041882261633872986,
+          0.01856978051364422,
+          0.010479284450411797,
+          0.046834204345941544,
+          0.0633290484547615,
+          -0.021132932975888252,
+          -0.0072448295541107655,
+          0.017375385388731956,
+          -0.022894009947776794,
+          0.003243172774091363,
+          -0.0038142153061926365,
+          -0.029450101777911186,
+          0.013975284993648529,
+          0.028508534654974937,
+          0.08578714728355408,
+          -0.01956365443766117,
+          0.012022406794130802,
+          -0.033286113291978836,
+          0.037662651389837265,
+          -0.009799264371395111,
+          0.023417102172970772,
+          -0.025404853746294975,
+          -0.021708333864808083,
+          -0.025352545082569122,
+          0.010705957189202309,
+          -0.014995315112173557,
+          0.0059632533229887486,
+          -0.005889148451387882,
+          -0.0062945447862148285,
+          -0.018691834062337875,
+          0.010470565408468246,
+          0.007685098797082901,
+          0.047985006123781204,
+          0.029816266149282455,
+          0.005858634598553181,
+          -0.07769665867090225,
+          0.038325235247612,
+          0.017514877021312714,
+          0.036546722054481506,
+          0.01543994341045618,
+          0.033443041145801544,
+          -0.026538221165537834,
+          0.017680522054433823,
+          0.0032475318294018507,
+          0.01094134896993637,
+          -0.03197837993502617,
+          0.044916197657585144,
+          -0.00843050517141819,
+          0.002720080316066742,
+          0.01972058229148388,
+          0.013312701135873795,
+          0.04062684252858162,
+          0.027915697544813156,
+          0.034210242331027985,
+          0.03815086930990219,
+          -0.01332141924649477,
+          0.010453129187226295,
+          0.029380355030298233,
+          0.03515180945396423,
+          0.02796800620853901,
+          0.0027636713348329067,
+          0.0041890982538461685,
+          -0.026590529829263687,
+          -0.006721737328916788,
+          0.009964910335838795,
+          0.007039951626211405,
+          0.016390228644013405,
+          -0.012780890800058842,
+          -0.028124934062361717,
+          0.0220221895724535,
+          -0.028648026287555695,
+          0.030740395188331604,
+          0.008626664988696575,
+          -0.01083673071116209,
+          0.013208082877099514,
+          0.009895164519548416,
+          -0.029833702370524406,
+          0.004243587143719196,
+          -0.006250953767448664,
+          -0.004468080587685108,
+          -0.025021253153681755,
+          -0.01933698169887066,
+          0.00797280017286539,
+          0.0029532923363149166,
+          0.005021686665713787,
+          0.009380790404975414,
+          -0.022545281797647476,
+          0.004284998401999474,
+          0.02127242274582386,
+          0.022876573726534843,
+          -0.018325669690966606,
+          -0.018901070579886436,
+          -0.011438286863267422,
+          -0.01914518140256405,
+          -0.011865478940308094,
+          0.044393107295036316,
+          -0.012711144983768463,
+          -0.009694645181298256,
+          0.020574966445565224,
+          0.006774046458303928,
+          -0.005043482407927513,
+          0.017872322350740433,
+          -0.029484974220395088,
+          -0.010932630859315395,
+          -0.0035592077765613794,
+          0.011072122491896152,
+          0.010243892669677734,
+          -0.00997362844645977,
+          0.01550968922674656,
+          -0.050879452377557755,
+          -0.010932630859315395,
+          0.007881258614361286,
+          0.015082497149705887,
+          -0.0110285310074687,
+          0.02339966595172882,
+          0.012161897495388985,
+          0.016224581748247147,
+          -0.003081885864958167,
+          -0.014829669147729874,
+          0.008221268653869629,
+          0.017384104430675507,
+          0.007881258614361286,
+          0.05712168663740158,
+          0.02338222973048687,
+          -0.03947603702545166,
+          0.006538654677569866,
+          -0.029973194003105164,
+          0.05579652264714241,
+          0.007994594983756542,
+          -0.029833702370524406,
+          0.053215932101011276,
+          -0.023434538394212723,
+          -0.0024193022400140762,
+          0.060853078961372375,
+          -0.0007007258245721459,
+          -0.013548092916607857,
+          0.0013011923292651772,
+          0.02435866929590702,
+          -0.056075502187013626,
+          -0.04899631813168526,
+          -0.02148166112601757,
+          0.02857828140258789,
+          -0.02453303337097168,
+          0.0077156126499176025,
+          -0.015405070967972279,
+          0.04735729843378067,
+          0.036721084266901016,
+          0.018116433173418045,
+          -0.008731283247470856,
+          0.02008674666285515,
+          0.04749678820371628,
+          -0.007977158762514591,
+          -0.004742704331874847,
+          -0.02833417057991028,
+          0.005152460187673569,
+          -0.033216364681720734,
+          -0.009485408663749695,
+          -0.027462350204586983,
+          -0.011133149266242981,
+          -0.02125498652458191,
+          -0.0030317562632262707,
+          0.022336045280098915,
+          0.0041389684192836285,
+          5.1253518904559314e-05,
+          -0.0020389705896377563,
+          0.040522221475839615,
+          -0.024951506406068802,
+          -0.04470696300268173,
+          -0.00825178250670433,
+          0.01704409345984459,
+          0.0038556267973035574,
+          -0.05475033447146416,
+          -0.006250953767448664,
+          -0.0029707287903875113,
+          0.01304243691265583,
+          -0.023539157584309578,
+          -0.00863538309931755,
+          0.016573309898376465,
+          0.002870469354093075,
+          0.009807982482016087,
+          -0.02800287865102291,
+          -0.014629150740802288,
+          0.007406116463243961,
+          0.01841285265982151,
+          0.02817724272608757,
+          -0.0555872842669487,
+          -0.03225736320018768,
+          0.02067958563566208,
+          -0.009040780365467072,
+          -0.06409624963998795,
+          0.007833308540284634,
+          -0.001987751107662916,
+          0.01799437776207924,
+          0.016948193311691284,
+          0.04160328209400177,
+          -0.019581090658903122,
+          -0.022283734753727913,
+          -0.016599465161561966,
+          -0.0364072285592556,
+          -0.028700334951281548,
+          -0.009712082333862782,
+          -0.008299732580780983,
+          -0.028857262805104256,
+          -0.0017839629435911775,
+          0.01342603750526905,
+          -0.03759290650486946,
+          0.006176849361509085,
+          -0.021603714674711227,
+          0.0028486738447099924,
+          -0.03703493997454643,
+          0.007842026650905609,
+          -0.0030339357908815145,
+          -0.01613740064203739,
+          -0.02125498652458191,
+          -0.01561430748552084,
+          0.014725050888955593,
+          -0.0002036518562817946,
+          0.0017142173601314425,
+          0.0229637548327446,
+          0.007484580390155315,
+          0.009328480809926987,
+          -0.008535124361515045,
+          -0.029798829928040504,
+          -0.013382446952164173,
+          0.006586604751646519,
+          0.018534906208515167,
+          0.019999565556645393,
+          0.005501188337802887,
+          -0.04958915710449219,
+          -0.03584926575422287,
+          0.021621152758598328,
+          -0.05890020355582237,
+          0.006394804455339909,
+          -0.0017425514524802566,
+          -0.008622306399047375,
+          0.0033150978852063417,
+          -0.005832480266690254,
+          -0.026294110342860222,
+          0.015570716932415962,
+          0.0086135882884264,
+          0.012275234796106815,
+          0.007488939445465803,
+          0.015980472788214684,
+          0.02852597087621689,
+          -0.02451559714972973,
+          -0.05788889154791832,
+          0.03427998721599579,
+          -0.019075434654951096,
+          0.014210676774382591,
+          0.0440792515873909,
+          0.005475033540278673,
+          -0.006551732309162617,
+          0.025003815069794655,
+          0.013399883173406124,
+          -0.010034655220806599,
+          -0.03407074883580208,
+          0.04376539587974548,
+          -0.003430614247918129,
+          -0.07121030986309052,
+          0.005479392595589161,
+          0.0014810053398832679,
+          0.013103464618325233,
+          0.015283015556633472,
+          -0.003430614247918129,
+          -0.023643774911761284,
+          -0.01956365443766117,
+          -0.014088621363043785,
+          -0.018168741837143898,
+          0.029484974220395088,
+          0.003539591794833541,
+          0.005004250444471836,
+          -0.0006091846735216677,
+          -0.0014341449132189155,
+          0.004677317570894957,
+          0.014027593657374382,
+          0.04080120474100113,
+          -0.0364769771695137,
+          0.0025958458427339792,
+          -0.04275408387184143,
+          -0.03874370828270912,
+          -0.0046380856074392796,
+          0.018360542133450508,
+          0.026485910639166832,
+          -0.025910509750247,
+          0.02320786565542221,
+          0.0100608104839921,
+          -0.015675334259867668,
+          -0.005008609499782324,
+          0.028090061619877815,
+          0.025369981303811073,
+          0.01285063661634922,
+          -0.004533467348664999,
+          0.014690177515149117,
+          -0.012623962946236134,
+          0.0026045641861855984,
+          0.01574508100748062,
+          -0.00010305192699888721,
+          0.03490769863128662,
+          0.001937621389515698,
+          0.018918508663773537,
+          -0.04693882167339325,
+          -0.02699156664311886,
+          -0.027689022943377495,
+          -0.0172620490193367,
+          0.03874370828270912,
+          -0.006503781769424677,
+          -0.012728581205010414,
+          0.020261110737919807,
+          0.004282819107174873,
+          0.0017044093692675233,
+          -0.04066171497106552,
+          -0.028630590066313744,
+          -0.022283734753727913,
+          -0.044916197657585144,
+          -0.005618884228169918,
+          -0.02183038927614689,
+          -0.019057998433709145,
+          -0.05342517048120499,
+          0.0010527233825996518,
+          -0.07811512798070908,
+          -0.005370415281504393,
+          0.04048734903335571,
+          0.004686036147177219,
+          0.009424380958080292,
+          -0.028682898730039597,
+          0.01025261078029871,
+          -0.036337483674287796,
+          -0.02568383514881134,
+          0.017785141244530678,
+          0.04387001320719719,
+          -0.011813169345259666,
+          0.0019997386261820793,
+          0.019057998433709145,
+          -0.05056559666991234,
+          -0.005370415281504393,
+          -0.0031799657735973597,
+          0.007641507778316736,
+          0.012920381501317024,
+          -0.016974348574876785,
+          0.05457597225904465,
+          0.011054685339331627,
+          0.02223142609000206,
+          0.02606743760406971,
+          -0.00020024630066473037,
+          -0.038046251982450485,
+          0.017584621906280518,
+          0.008535124361515045,
+          0.02796800620853901,
+          0.0015681873774155974,
+          0.0073014977388083935,
+          -0.019807765260338783,
+          0.008979752659797668,
+          -0.009485408663749695,
+          0.01646869257092476,
+          0.018866198137402534,
+          0.02645103819668293,
+          -0.008644101209938526,
+          -0.002175192581489682,
+          0.0033434322103857994,
+          -0.0034785643219947815,
+          0.028700334951281548,
+          0.0076371487230062485,
+          -0.038848329335451126,
+          0.006813278421759605,
+          -0.002189359627664089,
+          0.023120682686567307,
+          0.030374230816960335,
+          -0.04108018800616264,
+          -0.02338222973048687,
+          -0.002796364715322852,
+          0.016974348574876785,
+          0.011019812896847725,
+          -0.020662149414420128,
+          -0.0031668883748352528,
+          0.044602345675230026,
+          -0.010479284450411797,
+          -0.032484035938978195,
+          0.01293781865388155,
+          -0.0018504393519833684,
+          0.0059806895442306995,
+          0.0024476363323628902,
+          0.012772172689437866,
+          0.013469628989696503,
+          -0.019598528742790222,
+          0.04087095335125923,
+          -0.004965018481016159,
+          0.013208082877099514,
+          -0.012031124904751778,
+          -0.010052092373371124,
+          -0.015195833519101143,
+          0.03926680237054825,
+          -0.0047906544059515,
+          0.02013905718922615,
+          0.024864325299859047,
+          0.006338136270642281,
+          0.05216974765062332,
+          -0.04125455394387245,
+          -0.03724417835474014,
+          0.005971971433609724,
+          0.0016215863870456815,
+          0.005871712230145931,
+          -0.011272640898823738,
+          -0.030932195484638214,
+          -0.012074715457856655,
+          0.004768858663737774,
+          -0.015535843558609486,
+          -0.015684053301811218,
+          0.00019043832435272634,
+          0.022144244983792305,
+          0.0011257383739575744,
+          0.005475033540278673,
+          0.00287700816988945,
+          0.014585559256374836,
+          0.03581439331173897,
+          -0.02589307352900505,
+          0.010758266784250736,
+          0.03147272393107414,
+          -0.009136680513620377,
+          -0.026764893904328346,
+          -0.027305422350764275,
+          0.015073779039084911,
+          -0.018918508663773537,
+          -0.028874699026346207,
+          3.807404209510423e-05,
+          -0.0034872826654464006,
+          0.008391273207962513,
+          -0.03584926575422287,
+          -0.0076763806864619255,
+          0.0021283321548253298,
+          -0.00196377607062459,
+          0.0007078094058670104,
+          -0.022318609058856964,
+          -0.013591683469712734,
+          0.01741897687315941,
+          0.024219177663326263,
+          0.005727861542254686,
+          0.03028704971075058,
+          -0.010243892669677734,
+          -0.00016768925706855953,
+          0.006525577511638403,
+          0.01589328981935978,
+          0.023050937801599503,
+          0.023312482982873917,
+          0.026573093608021736,
+          0.0009344827267341316,
+          -0.012327543459832668,
+          0.004252305254340172,
+          0.032484035938978195,
+          -0.016058936715126038,
+          -0.0030993223190307617,
+          0.014812232926487923,
+          0.03274558484554291,
+          0.010627493262290955,
+          -0.012188052758574486,
+          -0.02528279833495617,
+          0.015431225299835205,
+          -0.022283734753727913,
+          0.012510626576840878,
+          0.04777577146887779,
+          0.004393976181745529,
+          -0.029240863397717476,
+          0.02430635876953602,
+          -0.009546436369419098,
+          -0.024044813588261604,
+          0.027165930718183517,
+          0.003757546888664365,
+          -0.03271070867776871,
+          -6.98818766977638e-05,
+          -0.00018321855168323964,
+          0.014794795773923397,
+          -0.026015127077698708,
+          -0.015579435043036938,
+          0.02322530187666416,
+          0.009293608367443085,
+          0.0032627887558192015,
+          -0.021603714674711227,
+          0.001829733606427908,
+          0.015344043262302876,
+          0.0018177460879087448,
+          0.009424380958080292,
+          0.009145398624241352,
+          0.01505634281784296,
+          0.006939692422747612,
+          -0.013190646655857563,
+          -0.0032344546634703875,
+          -0.02720080316066742,
+          -0.006098385434597731,
+          0.0652819275856018,
+          -0.007292779628187418,
+          0.03686057776212692,
+          0.0015213270671665668,
+          0.012868072837591171,
+          0.025788454338908195,
+          -0.025160742923617363,
+          -0.02437610551714897,
+          -0.005047841463238001,
+          0.006961487699300051,
+          -0.04292844980955124,
+          -0.013269110582768917,
+          -0.023818138986825943,
+          -0.030217302963137627,
+          0.01933698169887066,
+          0.0005187332862988114,
+          0.0019267236348241568,
+          0.024829450994729996,
+          0.0051742554642260075,
+          -0.015038905665278435,
+          -0.017009221017360687,
+          0.0383601076900959,
+          0.016625620424747467,
+          0.028229551389813423,
+          -0.015989189967513084,
+          0.01625945419073105,
+          -0.008099214173853397,
+          0.003171247662976384,
+          0.03236198052763939,
+          -0.0026982848066836596,
+          -0.0030186788644641638,
+          0.009197707287967205,
+          0.01881388947367668,
+          0.040731459856033325,
+          -0.01745384931564331,
+          -0.030827578157186508,
+          -0.008914366364479065,
+          -0.014236831106245518,
+          0.021969880908727646,
+          -0.0009491947130300105,
+          -0.00041084547410719097,
+          -0.003934090491384268,
+          0.04352128505706787,
+          -0.029572155326604843,
+          0.014428631402552128,
+          -0.024445850402116776,
+          -0.039022691547870636,
+          0.017706677317619324,
+          -0.006102744489908218,
+          0.032693274319171906,
+          0.006883023772388697,
+          -0.007833308540284634,
+          -0.032867636531591415,
+          0.0016749853966757655,
+          0.0086179468780756,
+          -0.03194350749254227,
+          0.020418038591742516,
+          0.02472483366727829,
+          -0.00651250034570694,
+          0.0033063797745853662,
+          -0.016782546415925026,
+          0.008142804726958275,
+          -0.013844511471688747,
+          -0.022946318611502647,
+          0.022737082093954086,
+          -0.004298075567930937,
+          -0.028055187314748764,
+          0.006686864420771599,
+          -0.022458098828792572,
+          0.004707831423729658,
+          -0.0016771649243310094,
+          -0.00017014124023262411,
+          -0.00797280017286539,
+          0.010461847297847271,
+          -0.007571762427687645,
+          -0.01073211245238781,
+          -0.022353481501340866,
+          0.009991064667701721,
+          0.025631526485085487,
+          0.0017588981427252293,
+          -0.004672958515584469,
+          0.006848151329904795,
+          -0.03658159449696541,
+          0.00574093870818615,
+          0.02413199469447136,
+          -0.012205488979816437,
+          -0.008713847026228905,
+          -0.018255924805998802,
+          -0.029380355030298233,
+          -0.002366993110626936,
+          0.01130751334130764,
+          0.03584926575422287,
+          0.025788454338908195,
+          0.030409103259444237,
+          0.03391382470726967,
+          0.010810576379299164,
+          -0.00398204056546092,
+          0.009127962402999401,
+          -0.029031626880168915,
+          0.04006887599825859,
+          0.0040823002345860004,
+          0.02608487382531166,
+          0.016172273084521294,
+          -0.0459623858332634,
+          -0.00968592707067728,
+          0.014681459404528141,
+          0.027898261323571205,
+          0.044148996472358704,
+          0.025038689374923706,
+          -0.0010232995264232159,
+          0.027654150500893593,
+          -0.022196553647518158,
+          -0.02878751792013645,
+          0.01956365443766117,
+          -0.005235282704234123,
+          -0.00918898917734623,
+          -0.0013131798477843404,
+          0.021533969789743423,
+          -0.011385977268218994,
+          0.016058936715126038,
+          0.03415793180465698,
+          -0.008461019024252892,
+          0.012135743163526058,
+          0.0005590550135821104,
+          -0.008286654949188232,
+          0.026764893904328346,
+          -0.007214315701276064,
+          0.03485538810491562,
+          -0.01650356501340866,
+          -0.0018656961619853973,
+          0.000780279457103461,
+          -0.010766984894871712,
+          0.028020314872264862,
+          0.007270984351634979,
+          -0.06172490119934082,
+          0.00873564276844263,
+          0.011063403449952602,
+          0.013835793361067772,
+          -0.001278307056054473,
+          -0.0033957413397729397,
+          -0.025788454338908195,
+          0.019040562212467194,
+          0.008129727095365524,
+          0.02472483366727829,
+          0.010679802857339382,
+          0.019493909552693367,
+          -0.016643056645989418,
+          0.016599465161561966,
+          -0.0026590528432279825,
+          -0.02854340709745884,
+          0.007266624830663204,
+          0.013181927613914013,
+          0.0402083694934845,
+          -0.026538221165537834,
+          -0.002582768676802516,
+          0.010889039374887943,
+          -0.006970205809921026,
+          -0.011194176971912384,
+          -0.030443977564573288,
+          -0.0010756086558103561,
+          -0.028421353548765182,
+          0.02416686899960041,
+          -0.005139382556080818,
+          0.019406726583838463,
+          -0.027357731014490128,
+          0.0063337767496705055,
+          0.00043264098349027336,
+          -0.027654150500893593,
+          0.006390445400029421,
+          0.01782001368701458,
+          0.0031189383007586002,
+          -0.031176306307315826,
+          -0.006006844341754913,
+          0.023242738097906113,
+          0.03619799390435219,
+          0.006713018752634525,
+          -0.006848151329904795,
+          0.021533969789743423,
+          -0.015579435043036938,
+          -0.006150694563984871,
+          0.036546722054481506,
+          -0.025910509750247,
+          0.012231643311679363,
+          0.03759290650486946,
+          -0.03543078899383545,
+          -0.03196094557642937,
+          -0.0004021272761747241,
+          -0.04142891615629196,
+          0.001680434332229197,
+          -0.002975087845697999,
+          0.00835204217582941,
+          -0.0009715351043269038,
+          -0.027113622054457664,
+          0.024812014773488045,
+          -0.0003577189054340124,
+          0.013373728841543198,
+          -0.006848151329904795,
+          0.0021065366454422474,
+          -0.025143306702375412,
+          -0.01862208917737007,
+          -0.010793139226734638,
+          0.00015651904686819762,
+          0.0023059656377881765,
+          -0.008112290874123573,
+          -0.03944116458296776,
+          -0.020976005122065544,
+          0.007257906720042229,
+          0.00045770584256388247,
+          0.03584926575422287,
+          0.018482597544789314,
+          0.012188052758574486,
+          -0.02165602520108223,
+          -0.01898825354874134,
+          0.04756653308868408,
+          -0.026416165754199028,
+          0.0402432419359684,
+          -0.012135743163526058,
+          -0.017488721758127213,
+          0.002946753753349185,
+          0.02625923790037632,
+          -0.020923694595694542,
+          0.016817420721054077,
+          -0.01561430748552084,
+          0.005719143431633711,
+          -0.01409733947366476,
+          -0.004755781497806311,
+          0.009642336517572403,
+          -0.0017796038882806897,
+          0.018255924805998802,
+          0.0009072383400052786,
+          -0.0031123997177928686,
+          0.03183889016509056,
+          0.013652711175382137,
+          0.007763562723994255,
+          -0.016773829236626625,
+          -0.012222925201058388,
+          -0.010766984894871712,
+          0.007087901700288057,
+          -0.015073779039084911,
+          -0.007066106423735619,
+          -0.029659338295459747,
+          0.001694601378403604,
+          0.0007154378108680248,
+          -0.015256861224770546,
+          0.01235369872301817,
+          0.02706131339073181,
+          -0.013295264914631844,
+          -0.03157734498381615,
+          -0.006229158490896225,
+          -0.03944116458296776,
+          -0.021045750007033348,
+          0.019511345773935318,
+          -0.0005190056981518865,
+          0.009127962402999401,
+          0.004250125493854284,
+          0.00043509298120625317,
+          -0.022946318611502647,
+          -0.017976941540837288,
+          0.00714892940595746,
+          -0.009572590701282024,
+          0.00107724335975945,
+          0.00131862866692245,
+          0.01822105050086975,
+          -0.010156710632145405,
+          0.02800287865102291,
+          -0.017497440800070763,
+          0.015265579335391521,
+          -0.029851138591766357,
+          -0.02760184183716774,
+          -0.03199581801891327,
+          0.01304243691265583,
+          -0.014611713588237762,
+          -0.008195114322006702,
+          -0.010113119147717953,
+          0.0004999346565455198,
+          0.007567403372377157,
+          -0.009441817179322243,
+          0.00842614658176899,
+          0.011603932827711105,
+          0.013303983025252819,
+          0.007406116463243961,
+          -0.03244916349649429,
+          -0.02029598504304886,
+          -0.02397506684064865,
+          0.043974634259939194,
+          -0.030513722449541092,
+          0.01703537441790104,
+          -0.017392821609973907,
+          0.005823761690407991,
+          -0.0363723561167717,
+          -0.019075434654951096,
+          0.008953598327934742,
+          0.020749330520629883,
+          -0.025910509750247,
+          0.02741004154086113,
+          0.02934548258781433,
+          0.03013012185692787,
+          -0.036546722054481506,
+          0.025352545082569122,
+          -0.010505438782274723,
+          -0.02512587048113346,
+          0.027357731014490128,
+          0.03731392323970795,
+          0.03679082915186882,
+          -0.013460910879075527,
+          -0.012589089572429657,
+          -0.014690177515149117,
+          0.012318825349211693,
+          -0.0021555765997618437,
+          0.005801966413855553,
+          -0.037069812417030334,
+          0.01862208917737007,
+          0.009895164519548416,
+          0.005121946334838867,
+          0.03773239627480507,
+          -0.01898825354874134,
+          0.02054009400308132,
+          0.007471502758562565,
+          -0.01839541457593441,
+          0.006974565330892801,
+          0.000942111189942807,
+          -0.0004040343628730625,
+          0.003602798795327544,
+          0.017584621906280518,
+          0.005553497467190027,
+          -0.054541099816560745,
+          0.006582245696336031,
+          0.027898261323571205,
+          0.020418038591742516,
+          0.005235282704234123,
+          -0.026398729532957077,
+          0.014629150740802288,
+          0.015823544934391975,
+          0.002135960618034005,
+          0.007035592570900917,
+          0.016538437455892563,
+          -0.011080840602517128,
+          0.00042283302173018456,
+          0.0030339357908815145,
+          0.03720930591225624,
+          0.01475120522081852,
+          -0.013190646655857563,
+          -0.025056125596165657,
+          -0.016364073380827904,
+          0.0066519915126264095,
+          -0.005915303248912096,
+          -0.0004182014672551304,
+          -0.037488289177417755,
+          -0.03598875552415848,
+          0.013521937653422356,
+          -0.01818617805838585,
+          -0.03815086930990219,
+          -0.0042741005308926105,
+          -0.03222249075770378,
+          0.005200410261750221,
+          -0.014890696853399277,
+          -0.019319545477628708,
+          -0.03926680237054825,
+          -0.0038970382884144783,
+          0.00012893410166725516,
+          0.04045247659087181,
+          0.00037842465098947287,
+          0.03476820886135101,
+          -0.007253547664731741,
+          -0.0013676686212420464,
+          -0.004143327474594116,
+          0.022876573726534843,
+          -0.03147272393107414,
+          -0.04529980197548866,
+          0.026573093608021736,
+          0.005832480266690254,
+          0.016024062409996986,
+          0.035744644701480865,
+          -0.013870666734874249,
+          -0.010200301185250282,
+          0.024864325299859047,
+          -0.016747673973441124,
+          -0.010984940454363823,
+          -0.0022231426555663347,
+          0.0025980256032198668,
+          -0.012231643311679363,
+          -0.013521937653422356,
+          -0.011656241491436958,
+          0.05157690867781639,
+          0.005575293209403753,
+          0.027845950797200203,
+          0.04038273170590401,
+          0.0010271136416122317,
+          -0.04059197008609772,
+          0.0032889433205127716,
+          -0.0091018071398139,
+          -0.009642336517572403,
+          -0.004625008441507816,
+          0.033024564385414124,
+          0.001587803359143436,
+          0.017872322350740433,
+          -0.0042174323461949825,
+          -0.016416382044553757,
+          -0.020418038591742516,
+          -0.012318825349211693,
+          -0.00761535344645381,
+          -0.004145507235080004,
+          -0.03373945876955986,
+          -0.01256293524056673,
+          0.031716834753751755,
+          0.025335106998682022,
+          -0.022388353943824768,
+          -0.004925786517560482,
+          -0.01782001368701458,
+          -0.007672021631151438,
+          0.010479284450411797,
+          -0.002959831152111292,
+          -0.00416948227211833,
+          0.009250016883015633,
+          0.018133869394659996,
+          0.01007824670523405,
+          0.00766766257584095,
+          -0.05715656280517578,
+          -0.0019398009171709418,
+          0.011351104825735092,
+          0.010400820523500443,
+          -0.003968963399529457,
+          0.014541968703269958,
+          -0.012580371461808681,
+          -0.01265011727809906,
+          -0.01975545659661293,
+          0.0326235294342041,
+          0.006355572491884232,
+          0.030426539480686188,
+          -0.0005541510181501508,
+          0.009075652807950974,
+          0.02149909734725952,
+          0.004101916216313839,
+          0.012807045131921768,
+          0.00199102028273046,
+          0.019807765260338783,
+          0.00986029114574194,
+          -0.03142041712999344,
+          0.012414725497364998,
+          0.0010429153917357326,
+          -0.0024519956205040216,
+          -0.017872322350740433,
+          0.006922255735844374,
+          0.0023757112212479115,
+          -0.013949130661785603,
+          0.0041215321980416775,
+          -0.022370917722582817,
+          0.005954534746706486,
+          0.031089123338460922,
+          0.033233802765607834,
+          0.009206426329910755,
+          -0.015169679187238216,
+          -0.010810576379299164,
+          0.002057496691122651,
+          0.02008674666285515,
+          0.025404853746294975,
+          0.025003815069794655,
+          -0.0268172025680542,
+          -0.014393758960068226,
+          -0.003729212796315551,
+          -0.001253242138773203,
+          -0.010932630859315395,
+          0.03046141378581524,
+          0.02833417057991028,
+          -0.0008914365898817778,
+          0.0028835467528551817,
+          -0.0051916916854679585,
+          0.01523070689290762,
+          0.01898825354874134,
+          -0.017340512946248055,
+          0.0017665266059339046,
+          0.022283734753727913,
+          -0.020191365852952003,
+          0.022946318611502647,
+          -0.014672741293907166,
+          -0.011429568752646446,
+          0.02013905718922615,
+          0.09366840869188309,
+          -0.005248360335826874,
+          0.01120289508253336,
+          -0.028700334951281548,
+          -0.019214926287531853,
+          -0.008474096655845642,
+          -0.0035330532118678093,
+          0.016765110194683075,
+          0.026503346860408783,
+          0.009511562995612621,
+          0.021359605714678764,
+          0.015466097742319107,
+          -0.01666049286723137,
+          0.014411195181310177,
+          -0.009694645181298256,
+          0.016198428347706795,
+          -0.01362655684351921,
+          -0.004267562180757523,
+          0.010069528594613075,
+          -0.001332795829512179,
+          0.009816700592637062,
+          0.018674397841095924,
+          0.016799984499812126,
+          0.013190646655857563,
+          -0.024864325299859047,
+          0.00949412677437067,
+          0.002284170128405094,
+          -0.007763562723994255,
+          -0.02013905718922615,
+          -0.02322530187666416,
+          0.004999891389161348,
+          0.004428848624229431,
+          0.005296310409903526,
+          0.010331074707210064,
+          -0.0032889433205127716,
+          0.03811599686741829,
+          -0.03319893032312393,
+          0.01496044173836708,
+          0.006917896680533886,
+          -0.013103464618325233,
+          0.011935224756598473,
+          0.016198428347706795,
+          0.004049607086926699,
+          0.012249079532921314,
+          -0.011569059453904629,
+          0.014245549216866493,
+          0.02052265778183937,
+          -0.029781393706798553,
+          -0.00266123260371387,
+          -0.016372792422771454,
+          -0.008474096655845642,
+          0.0026677711866796017,
+          -0.016172273084521294,
+          -0.006006844341754913,
+          -0.00546195637434721,
+          0.01169983297586441,
+          0.0024825092405080795,
+          -0.0325886569917202,
+          0.005060918629169464,
+          0.00261110276915133,
+          0.005928380414843559,
+          0.018168741837143898,
+          0.01632048189640045,
+          0.01765436865389347,
+          -0.011525468900799751,
+          -0.01975545659661293,
+          0.006285826675593853,
+          0.0045029534958302975,
+          -0.01073211245238781,
+          0.0023473771288990974,
+          0.0024912275839596987,
+          0.01588457264006138,
+          -0.011185458861291409,
+          -0.005823761690407991,
+          -0.004141148179769516,
+          0.011664960533380508,
+          0.0031516316812485456,
+          -0.014541968703269958,
+          0.003940629307180643,
+          -0.021150369197130203,
+          0.020644713193178177,
+          0.011577778495848179,
+          -0.0019932000432163477,
+          -0.02088882215321064,
+          -0.02873520739376545,
+          0.01332141924649477,
+          -0.020836513489484787,
+          0.0031952227000147104,
+          0.015771236270666122,
+          -0.011673678644001484,
+          0.013260391540825367,
+          -0.018255924805998802,
+          -0.018866198137402534,
+          -0.0114818774163723,
+          -0.00473834527656436,
+          -0.005226564593613148,
+          -0.007576121482998133,
+          0.007502016611397266,
+          -0.01823848858475685,
+          -0.0306706503033638,
+          -0.033634841442108154,
+          0.019023125991225243,
+          0.01670408435165882,
+          -0.009467972442507744,
+          -0.015823544934391975,
+          -0.004642445128411055,
+          0.02319042943418026,
+          -0.0034415118861943483,
+          0.0007110786973498762,
+          -0.0026830281130969524,
+          0.013391165062785149,
+          -0.007610994391143322,
+          0.00016441992192994803,
+          -0.02284170128405094,
+          0.00044217653339728713,
+          0.0011725988006219268,
+          0.013391165062785149,
+          -0.027357731014490128,
+          -0.003380484413355589,
+          0.0003002332232426852,
+          -0.018098996952176094,
+          -0.004476799163967371,
+          0.04254484549164772,
+          -0.013417319394648075,
+          0.008979752659797668,
+          0.021551406010985374,
+          0.010104401037096977,
+          -0.00977310910820961,
+          0.010095682926476002,
+          0.007850744761526585,
+          -0.0028857262805104256,
+          0.026538221165537834,
+          0.022911446169018745,
+          0.01914518140256405,
+          0.01670408435165882,
+          -0.022161681205034256,
+          0.012528062798082829,
+          0.021969880908727646,
+          0.03752316161990166,
+          -0.017331793904304504,
+          -0.021952442824840546,
+          0.019860073924064636,
+          0.04676445946097374,
+          0.0021032672375440598,
+          -0.0006200824282132089,
+          -0.028648026287555695,
+          -0.017201021313667297,
+          0.024689961224794388,
+          0.020557530224323273,
+          0.016224581748247147,
+          -0.010121837258338928,
+          -0.031507596373558044,
+          0.014280421659350395,
+          -0.00309278373606503,
+          0.006865587551146746,
+          -0.01996469311416149,
+          0.01688716560602188,
+          -0.015004033222794533,
+          0.04257972165942192,
+          -0.012658835388720036,
+          0.004829886369407177,
+          0.017148712649941444,
+          0.011150586418807507,
+          -0.022318609058856964,
+          0.0009263094398193061,
+          -0.01666049286723137,
+          0.027933133766055107,
+          -0.018883634358644485,
+          0.003618055721744895,
+          0.01914518140256405,
+          -0.016608182340860367,
+          -0.014533249661326408,
+          0.004191277548670769,
+          0.019459037110209465,
+          0.006107103545218706,
+          -0.025247925892472267,
+          0.018098996952176094,
+          0.0019190951716154814,
+          0.009171552956104279,
+          -0.009930036962032318,
+          0.023870449513196945,
+          -0.027636714279651642,
+          -0.015387633815407753,
+          0.025003815069794655,
+          0.0041607641614973545,
+          0.013565529137849808,
+          -0.026311546564102173,
+          0.009380790404975414,
+          -0.029049063101410866,
+          0.02336479350924492,
+          0.0032584297005087137,
+          0.014280421659350395,
+          0.0015147883677855134,
+          0.002866110298782587,
+          -0.0015115190763026476,
+          0.014446067623794079,
+          -0.000550881668459624,
+          -0.022527845576405525,
+          -0.01569277234375477,
+          -0.03696519508957863,
+          -0.022737082093954086,
+          0.023818138986825943,
+          -0.002715721260756254,
+          0.003528693923726678,
+          0.02430635876953602,
+          -0.027462350204586983,
+          -0.01435888558626175,
+          0.009572590701282024,
+          0.02549203485250473,
+          -0.043625906109809875,
+          4.093470488442108e-05,
+          0.0013829254312440753,
+          -0.019232362508773804,
+          0.010627493262290955,
+          -0.005440160632133484,
+          0.015500971116125584,
+          0.0018776836805045605,
+          -0.037662651389837265,
+          0.011586496606469154,
+          0.03762777894735336,
+          -0.017924632877111435,
+          0.0028225192800164223,
+          0.017706677317619324,
+          0.012493189424276352,
+          -0.0012390750925987959,
+          -0.0017033196054399014,
+          0.01877901703119278,
+          -0.025945382192730904,
+          0.026102310046553612,
+          0.017026657238602638,
+          -0.014881978742778301,
+          0.007842026650905609,
+          -0.008753078989684582,
+          0.020731894299387932,
+          -0.0013208083109930158,
+          -0.013155773282051086,
+          0.017375385388731956,
+          0.002195898210629821,
+          0.01349578332155943,
+          -0.001912556472234428,
+          0.004304614383727312,
+          0.03107168711721897,
+          -0.01609380915760994,
+          0.02280682697892189,
+          0.01994725689291954,
+          3.6405323044164106e-05,
+          0.015405070967972279,
+          -0.019877510145306587,
+          -0.03312918543815613,
+          0.02221398986876011,
+          -0.013591683469712734,
+          -0.007310215849429369,
+          0.023347357288002968,
+          -0.010453129187226295,
+          -0.006250953767448664,
+          -0.004282819107174873,
+          0.004077941179275513,
+          0.012406007386744022,
+          -0.0230160653591156,
+          -0.015631744638085365,
+          0.022318609058856964,
+          0.017131276428699493,
+          0.022179117426276207,
+          0.023800702765583992,
+          -0.015378915704786777,
+          0.03182145208120346,
+          0.009223862551152706,
+          0.0065735275857150555,
+          0.0076371487230062485,
+          0.008395632728934288,
+          -0.005871712230145931,
+          0.01933698169887066,
+          0.014114775694906712,
+          0.007336370646953583,
+          -0.003565746359527111,
+          -0.02029598504304886,
+          -0.011447004973888397,
+          -0.027532095089554787,
+          -0.00045825072447769344,
+          -0.013914257287979126,
+          0.00704866973683238,
+          0.07797563821077347,
+          -0.006682505365461111,
+          0.0018144766800105572,
+          -0.007868180982768536,
+          0.000739412906114012,
+          0.0016771649243310094,
+          -0.005209128372371197,
+          0.016538437455892563,
+          -0.020662149414420128,
+          -0.020749330520629883,
+          -0.006163771729916334,
+          -0.010967503301799297,
+          0.013460910879075527,
+          0.01741897687315941,
+          0.0030034221708774567,
+          -0.02277195453643799,
+          0.011141867376863956,
+          -0.016355354338884354,
+          -0.011106994934380054,
+          0.010348510928452015,
+          -0.007231752388179302,
+          -0.011656241491436958,
+          0.03295481950044632,
+          -0.009206426329910755,
+          -0.03147272393107414,
+          0.015640461817383766,
+          -0.0024541751481592655,
+          -0.005976330488920212,
+          0.00470347236841917,
+          0.00853076484054327,
+          -0.004986814223229885,
+          0.0005544234300032258,
+          0.013504501432180405,
+          -0.002068394562229514,
+          0.005414006300270557,
+          0.005318105686455965,
+          -0.013565529137849808,
+          0.017296921461820602,
+          0.022527845576405525,
+          -0.033477913588285446,
+          -0.004755781497806311,
+          -0.013940411619842052,
+          0.010479284450411797,
+          0.018709270283579826,
+          0.015535843558609486,
+          -0.022580154240131378,
+          0.0025871277321130037,
+          -0.009154116734862328,
+          -0.012702426873147488,
+          -0.02741004154086113,
+          -0.018255924805998802,
+          0.00723611144348979,
+          -0.00851768720895052,
+          0.018656961619853973,
+          0.007907412946224213,
+          -0.005745298229157925,
+          0.005012968555092812,
+          -0.005178614519536495,
+          0.015518407337367535,
+          -0.0326235294342041,
+          0.010461847297847271,
+          -0.009337198920547962,
+          0.009581308811903,
+          0.01055774837732315,
+          -0.022388353943824768,
+          -0.014759923331439495,
+          -0.023417102172970772,
+          0.016555873677134514,
+          -0.010086964815855026,
+          -0.0020934592466801405,
+          0.02008674666285515,
+          0.0009088730439543724,
+          0.015413789078593254,
+          -0.001491903094574809,
+          0.023242738097906113,
+          0.014140930958092213,
+          0.016590746119618416,
+          -0.01215317938476801,
+          -0.001640112604945898,
+          -0.011124431155622005,
+          0.012719863094389439,
+          -0.02303350158035755,
+          -0.006813278421759605,
+          -0.026015127077698708,
+          0.03598875552415848,
+          -0.007602275814861059,
+          -0.01588457264006138,
+          0.02857828140258789,
+          -0.012414725497364998,
+          -0.0037052377592772245,
+          0.0058847893960773945,
+          -0.015431225299835205,
+          0.023451974615454674,
+          0.0004955755430273712,
+          0.011499314568936825,
+          -0.02549203485250473,
+          0.00863102450966835,
+          -0.007087901700288057,
+          0.006076589692384005,
+          -0.02128986082971096,
+          -0.002423661295324564,
+          0.012135743163526058,
+          -0.010426974855363369,
+          0.00411717314273119,
+          0.021185241639614105,
+          0.005331183318048716,
+          -0.01744513027369976,
+          0.03951091319322586,
+          -0.004025631584227085,
+          0.010653648525476456,
+          0.005475033540278673,
+          -0.0009028792264871299,
+          0.016799984499812126,
+          0.01742769405245781,
+          -0.004925786517560482,
+          -0.016547156497836113,
+          0.0032235567923635244,
+          -0.015021469444036484,
+          0.01561430748552084,
+          0.016233300790190697,
+          -0.02125498652458191,
+          -0.032658401876688004,
+          0.01779385842382908,
+          0.002061855746433139,
+          0.028072625398635864,
+          0.006983283441513777,
+          -0.025997690856456757,
+          -0.005296310409903526,
+          -0.03815086930990219,
+          0.012885509058833122,
+          -0.01801181398332119,
+          -0.02437610551714897,
+          0.002785466844215989,
+          -0.00986029114574194,
+          -0.01856978051364422,
+          0.02512587048113346,
+          0.013652711175382137,
+          -0.02552690915763378,
+          0.010889039374887943,
+          0.0013011923292651772,
+          -0.017549749463796616,
+          0.010052092373371124,
+          0.004350385162979364,
+          -0.012493189424276352,
+          0.007410475518554449,
+          0.005130664445459843,
+          -0.006978924386203289,
+          -0.02890957146883011,
+          -0.01670408435165882,
+          0.01972058229148388,
+          0.010993658564984798,
+          -0.0058455574326217175,
+          0.011290077120065689,
+          -0.00997362844645977,
+          0.0033456117380410433,
+          0.0035221553407609463,
+          0.00901462510228157,
+          -0.013051155023276806,
+          -0.007371243555098772,
+          -0.007162006571888924,
+          0.01973801851272583,
+          -0.016024062409996986,
+          0.007990236394107342,
+          -0.013443474657833576,
+          -0.007981518283486366,
+          0.011429568752646446,
+          -0.0013774766121059656,
+          0.014986597001552582,
+          -0.018744144588708878,
+          0.0073407297022640705,
+          0.01189163327217102,
+          -0.008883852511644363,
+          -0.010671084746718407,
+          -0.02437610551714897,
+          0.019110308960080147,
+          0.01438504084944725,
+          0.004694754257798195,
+          -0.03679082915186882,
+          0.003951527178287506,
+          -0.010453129187226295,
+          -0.0009775288635864854,
+          -0.008796670474112034,
+          0.019249800592660904,
+          0.018831325694918633,
+          0.007985876873135567,
+          0.013277828693389893,
+          -0.0023081451654434204,
+          0.013548092916607857,
+          -0.012414725497364998,
+          0.017593340948224068,
+          -0.004036529455333948,
+          -0.017575904726982117,
+          -0.005714784376323223,
+          0.019406726583838463,
+          -0.001491903094574809,
+          0.007903054356575012,
+          0.004372180439531803,
+          0.010226456448435783,
+          -0.012275234796106815,
+          -0.007493298500776291,
+          0.014045030809938908,
+          0.008378196507692337,
+          -0.0004288267809897661,
+          0.009250016883015633,
+          0.06758353859186172,
+          -0.00870512891560793,
+          0.023853013291954994,
+          0.006311981473118067,
+          -0.009345917031168938,
+          -0.008112290874123573,
+          0.0019932000432163477,
+          -0.02951984666287899,
+          0.023329921066761017,
+          0.011970097199082375,
+          -0.009180271066725254,
+          0.00036126066697761416,
+          -0.006726096384227276,
+          0.018325669690966606,
+          -0.009694645181298256,
+          0.016948193311691284,
+          -0.010052092373371124,
+          0.02435866929590702,
+          0.019162617623806,
+          0.005640679504722357,
+          0.005536061245948076,
+          0.0028203397523611784,
+          0.019459037110209465,
+          0.012868072837591171,
+          -0.002165384590625763,
+          -0.003123297356069088,
+          -0.009877728298306465,
+          0.019389290362596512,
+          -0.001910376944579184,
+          0.030792705714702606,
+          -0.0022144243121147156,
+          -0.005182973574846983,
+          -0.009842854924499989,
+          0.0041607641614973545,
+          0.011569059453904629,
+          0.009520281106233597,
+          0.012876790948212147,
+          -0.0013567708665505052,
+          -0.010679802857339382,
+          0.011909069493412971,
+          0.00863102450966835,
+          0.005326824262738228,
+          -0.0032518908847123384,
+          0.0045247492380440235,
+          -0.008404350839555264,
+          0.014001439325511456,
+          -0.0024781501851975918,
+          0.0015354941133409739,
+          -0.008774874731898308,
+          0.004426669329404831,
+          -0.01312961895018816,
+          0.0005348074482753873,
+          0.009607463143765926,
+          0.012868072837591171,
+          0.0015213270671665668,
+          -0.002273272257298231,
+          0.015405070967972279,
+          0.0025980256032198668,
+          0.0073014977388083935,
+          0.0009486498311161995,
+          -0.009537717327475548,
+          0.019057998433709145,
+          -0.01033979281783104,
+          -0.0019212746992707253,
+          0.010705957189202309,
+          0.006067871581763029,
+          0.008260500617325306,
+          0.01496916078031063,
+          0.009354635141789913,
+          0.003968963399529457,
+          0.0013545913388952613,
+          -0.00565375667065382,
+          -0.003535232739523053,
+          0.0036267738323658705,
+          0.0016771649243310094,
+          0.01646869257092476,
+          -0.019459037110209465,
+          -0.007039951626211405,
+          0.010679802857339382,
+          0.011438286863267422,
+          0.0017839629435911775,
+          0.015684053301811218,
+          0.014428631402552128,
+          0.01994725689291954,
+          -0.022562718018889427,
+          0.003831651760265231,
+          -0.002920599188655615,
+          0.019057998433709145,
+          0.008770515210926533,
+          -0.024114558473229408,
+          0.0003645300166681409,
+          -0.01063621137291193,
+          0.012676272541284561,
+          -0.0067086596973240376,
+          0.0023473771288990974,
+          0.01665177382528782,
+          -0.010121837258338928,
+          -0.009930036962032318,
+          -0.019424164667725563,
+          -0.016983065754175186,
+          0.010976221412420273,
+          0.014437349513173103,
+          0.0014188881032168865,
+          0.019232362508773804,
+          -0.0012412546202540398,
+          -0.008474096655845642,
+          0.030827578157186508,
+          0.016765110194683075,
+          -0.0004157494695391506,
+          -0.008962316438555717,
+          -0.0006544103380292654,
+          -0.015082497149705887,
+          0.01074954867362976,
+          0.004426669329404831,
+          0.005108869168907404,
+          0.003387023229151964,
+          9.174550359603018e-05,
+          -0.016712801530957222,
+          -0.014620431698858738,
+          -0.004668599460273981,
+          0.011045967228710651,
+          -0.003751008305698633,
+          0.009197707287967205,
+          -0.01609380915760994,
+          0.003121117828413844,
+          -0.008173318579792976,
+          -0.010714675299823284,
+          -0.008356400765478611,
+          -0.039406292140483856,
+          -0.02067958563566208,
+          0.0003173972072545439,
+          -0.00473834527656436,
+          -0.002358274767175317,
+          0.009223862551152706,
+          -0.003672544378787279,
+          0.013670147396624088,
+          -0.0015551100950688124,
+          0.0047906544059515,
+          0.008657178841531277,
+          0.01561430748552084,
+          -0.013504501432180405,
+          -0.0006631285650655627,
+          -0.014071185141801834,
+          -0.026782330125570297,
+          0.01707896590232849,
+          0.003199581755325198,
+          0.014446067623794079,
+          -0.015187115408480167,
+          0.009528999216854572,
+          -0.012589089572429657,
+          0.014463504776358604,
+          -0.0421612448990345,
+          -0.009520281106233597,
+          0.007031233515590429,
+          -0.0075979167595505714,
+          -0.005496829282492399,
+          -0.025369981303811073,
+          -0.0012205488746985793,
+          -0.011054685339331627,
+          -0.01666049286723137,
+          -0.019459037110209465,
+          0.00574093870818615,
+          -0.009311044588685036,
+          0.008591792546212673,
+          -0.006120180711150169,
+          0.02761927805840969,
+          0.00532246520742774,
+          -0.001346962875686586,
+          -0.006128899287432432,
+          0.009206426329910755,
+          -0.0007503106608055532,
+          0.012885509058833122,
+          0.008648460730910301,
+          0.005823761690407991,
+          0.023434538394212723,
+          0.0009339378448203206,
+          0.004690395202487707,
+          -0.006141976453363895,
+          0.014541968703269958,
+          0.024689961224794388,
+          0.011342386715114117,
+          0.011455723084509373,
+          0.005967612378299236,
+          0.0005950175691395998,
+          0.011429568752646446,
+          0.017392821609973907,
+          -0.003990759141743183,
+          -0.009171552956104279,
+          0.004869118332862854,
+          -0.01761949434876442,
+          -0.0033347138669341803,
+          0.012885509058833122,
+          -0.006128899287432432,
+          -0.02706131339073181,
+          0.01285063661634922,
+          0.004317691549658775,
+          -0.006673786789178848,
+          -0.016948193311691284,
+          -0.00661711860448122,
+          0.004646804183721542,
+          0.004729627165943384,
+          0.024236613884568214,
+          -0.022144244983792305,
+          -0.008644101209938526,
+          0.011917788535356522,
+          -0.014446067623794079,
+          -0.01063621137291193,
+          0.009790545329451561,
+          0.029484974220395088,
+          0.003496000776067376,
+          -0.05307644233107567,
+          -0.012467035092413425,
+          0.014236831106245518,
+          -0.01503018755465746,
+          -0.025980254635214806,
+          -0.0014428631402552128,
+          0.0038578063249588013,
+          -0.014010157436132431,
+          0.006625836715102196,
+          0.0011606112821027637,
+          -0.011708551086485386,
+          0.017139993607997894,
+          0.028874699026346207,
+          0.01312961895018816,
+          0.0114818774163723,
+          -0.025213053449988365,
+          -0.0008952508214861155,
+          -0.01914518140256405,
+          0.011752142570912838,
+          -0.030566031113266945,
+          -0.013984003104269505,
+          -0.0023342997301369905,
+          0.004141148179769516,
+          0.027444913983345032,
+          0.014376322738826275,
+          0.025962818413972855,
+          0.006778405513614416,
+          0.0037989583797752857,
+          -0.02125498652458191,
+          -0.018447725102305412,
+          0.0033456117380410433,
+          -0.0008304091752506793,
+          -0.002329940674826503,
+          -0.008216909132897854,
+          0.012275234796106815,
+          0.012039843015372753,
+          0.026172054931521416,
+          0.005087073426693678,
+          0.005553497467190027,
+          -2.538836997700855e-05,
+          -0.00494322320446372,
+          0.007593557704240084,
+          -0.019232362508773804,
+          -0.0029925242997705936,
+          -0.004559622146189213,
+          0.015405070967972279,
+          0.0045029534958302975,
+          0.0022275017108768225,
+          0.017357949167490005,
+          -0.0030099607538431883,
+          0.003729212796315551,
+          -0.019982129335403442,
+          -0.014672741293907166,
+          -0.0032518908847123384,
+          0.03525642678141594,
+          0.003971143160015345,
+          -0.02761927805840969,
+          -0.016224581748247147,
+          -0.0153353251516819,
+          0.030792705714702606,
+          -0.009581308811903,
+          0.004315512254834175,
+          0.02683463878929615,
+          0.01593688130378723,
+          0.012388571165502071,
+          0.006399163510650396,
+          -0.01822105050086975,
+          0.005021686665713787,
+          0.02931061014533043,
+          -0.0030317562632262707,
+          -0.010086964815855026,
+          -0.0066912234760820866,
+          0.007144570350646973,
+          -0.007798435632139444,
+          0.007632789667695761,
+          -0.0035068984143435955,
+          -0.013417319394648075,
+          0.029554719105362892,
+          -0.00968592707067728,
+          0.022370917722582817,
+          -0.010130555368959904,
+          0.00268738716840744,
+          -0.038848329335451126,
+          -0.002004097681492567,
+          0.0011453543556854129,
+          0.0019692247733473778,
+          0.004555262625217438,
+          -0.01881388947367668,
+          -0.008753078989684582,
+          0.011839324608445168,
+          -0.01276345457881689,
+          -0.007885618135333061,
+          -0.006137617398053408,
+          0.002983806189149618,
+          0.01977289281785488,
+          -0.00947669055312872,
+          0.0009857022669166327,
+          -0.0210980586707592,
+          0.01256293524056673,
+          -0.00282687833532691,
+          0.0017567186150699854,
+          0.010453129187226295,
+          0.007009437773376703,
+          0.03502975404262543,
+          0.01094134896993637,
+          0.005531701724976301,
+          -0.0032780456822365522,
+          -0.014053748920559883,
+          -0.03218761831521988,
+          0.013295264914631844,
+          -0.010453129187226295,
+          0.023800702765583992,
+          0.0036703648511320353,
+          -0.008103572763502598,
+          -0.0051960512064397335,
+          0.0037139558698982,
+          0.009590026922523975,
+          -0.0041106343269348145,
+          0.013478347100317478,
+          -0.010243892669677734,
+          0.021167805418372154,
+          0.0031298361718654633,
+          -0.005121946334838867,
+          0.012571653351187706,
+          -0.017689241096377373,
+          0.01055774837732315,
+          -0.01744513027369976,
+          0.0028225192800164223,
+          0.01360040158033371,
+          -0.010296201333403587,
+          0.0056276023387908936,
+          -0.052588220685720444,
+          -0.001197663601487875,
+          -0.016521001234650612,
+          -0.00675660977140069,
+          -0.008456660434603691,
+          -0.021394478157162666,
+          -0.03319893032312393,
+          -0.0036660057958215475,
+          0.015370197594165802,
+          0.006551732309162617,
+          0.023260174319148064,
+          0.0009895163821056485,
+          0.012257798574864864,
+          0.012807045131921768,
+          -0.0047906544059515,
+          0.012519344687461853,
+          0.033861514180898666,
+          -0.027741333469748497,
+          -0.019685709848999977,
+          0.005257078446447849,
+          -0.00028552126605063677,
+          -0.015788672491908073,
+          -0.013103464618325233,
+          -0.0017665266059339046,
+          0.0024955866392701864,
+          0.030443977564573288,
+          -0.020191365852952003,
+          -0.005579652264714241,
+          -0.007053028792142868,
+          0.004433208145201206,
+          -0.01918005384504795,
+          -0.0031450928654521704,
+          -0.011708551086485386,
+          -0.012780890800058842,
+          0.016041500493884087,
+          -0.011089558713138103,
+          0.011159304529428482,
+          -0.019511345773935318,
+          0.008626664988696575,
+          -0.015378915704786777,
+          -0.010627493262290955,
+          -0.0031690681353211403,
+          -0.010461847297847271,
+          0.030164994299411774,
+          -0.0041890982538461685,
+          0.0048386044800281525,
+          0.01670408435165882,
+          -0.00853076484054327,
+          0.011603932827711105,
+          -0.0006124540232121944,
+          0.007898694835603237,
+          -0.002704823622480035,
+          -0.0086135882884264,
+          0.0015278656501322985,
+          0.009537717327475548,
+          -0.010950067080557346,
+          0.03846472501754761,
+          -0.013731175102293491,
+          -0.008879492990672588,
+          -0.003859985852614045,
+          -0.004511671606451273,
+          0.002425840822979808,
+          0.0021762822289019823,
+          0.010984940454363823,
+          -0.0012870251666754484,
+          0.006835073698312044,
+          -0.024219177663326263,
+          -0.011717269197106361,
+          0.0013164491392672062,
+          0.006468909326940775,
+          0.003321636701002717,
+          0.008016390725970268,
+          -0.025579217821359634,
+          -0.0020782025530934334,
+          -0.012859354726970196,
+          0.005475033540278673,
+          -0.011499314568936825,
+          0.007332011591643095,
+          -0.01215317938476801,
+          -0.0013295264216139913,
+          0.009337198920547962,
+          0.012946536764502525,
+          0.013861947692930698,
+          -0.006608400493860245,
+          0.026573093608021736,
+          0.014899414964020252,
+          0.008709488436579704,
+          0.005405288189649582,
+          -0.033216364681720734,
+          -0.007802794687449932,
+          0.010365947149693966,
+          0.007519452832639217,
+          0.009232580661773682,
+          -0.017689241096377373,
+          -0.004176021087914705,
+          0.0029402151703834534,
+          -0.007039951626211405,
+          4.651980270864442e-05,
+          0.011080840602517128,
+          -0.010627493262290955,
+          0.005505547393113375,
+          -0.00031521765049546957,
+          0.0027549532242119312,
+          -0.006460190750658512,
+          0.012188052758574486,
+          -0.013286546804010868,
+          -0.008168959058821201,
+          0.0211154967546463,
+          0.02760184183716774,
+          -0.005531701724976301,
+          0.023696085438132286,
+          -0.021917570382356644,
+          0.008487174287438393,
+          -0.005043482407927513,
+          -0.014149649068713188,
+          -0.015300451777875423,
+          0.006883023772388697,
+          0.014777359552681446,
+          0.004760140553116798,
+          -0.013155773282051086,
+          -0.003027397207915783,
+          0.016983065754175186,
+          -0.0010952246375381947,
+          0.006216081324964762,
+          0.01593688130378723,
+          -0.00026971951592713594,
+          0.00017667989595793188,
+          -0.006002485286444426,
+          -0.01876158080995083,
+          -0.006290185730904341,
+          0.017924632877111435,
+          -0.0073799616657197475,
+          0.014262985438108444,
+          0.02320786565542221,
+          0.0005429807933978736,
+          0.011351104825735092,
+          -0.0038469084538519382,
+          0.01351321954280138,
+          0.01505634281784296,
+          -0.005235282704234123,
+          -0.0030600903555750847,
+          -0.0036965194158256054,
+          0.00011858122888952494,
+          0.012022406794130802,
+          -0.01741897687315941,
+          -0.011333668604493141,
+          0.0114818774163723,
+          -0.002121793571859598,
+          -0.008404350839555264,
+          -0.023486848920583725,
+          0.0016313943779096007,
+          0.0077156126499176025,
+          0.011447004973888397,
+          -0.016425101086497307,
+          -0.0005239096935838461,
+          0.000875089957844466,
+          0.010801857337355614,
+          -0.007497657556086779,
+          0.01159521471709013,
+          -0.007907412946224213,
+          -0.00863102450966835,
+          0.006381727289408445,
+          0.0015507509233430028,
+          0.007414834573864937,
+          -0.009999782778322697,
+          0.0015987011138349771,
+          -0.008025108836591244,
+          0.026590529829263687,
+          0.008317168802022934,
+          -0.0013011923292651772,
+          -0.005640679504722357,
+          0.004995532333850861,
+          0.008421787060797215,
+          -0.0100608104839921,
+          0.012815763242542744,
+          0.006107103545218706,
+          -0.002933676354587078,
+          0.010592620819807053,
+          0.015283015556633472,
+          0.0025631526950746775,
+          0.009328480809926987,
+          0.003443691413849592,
+          0.0059022256173193455,
+          -0.007022515404969454,
+          0.0017861425876617432,
+          -0.01550968922674656,
+          -0.020801641047000885,
+          -0.0035831828135997057,
+          0.016991784796118736,
+          -0.010583902709186077,
+          -0.001182406791485846,
+          0.017924632877111435,
+          0.01371373888105154,
+          -0.005034764297306538,
+          0.00551426550373435,
+          0.002217693720012903,
+          -0.0015529305674135685,
+          0.003323816228657961,
+          -0.014280421659350395,
+          -0.012493189424276352,
+          0.008726924657821655,
+          -0.02087138593196869,
+          0.01952878199517727,
+          -0.02165602520108223,
+          -0.035361044108867645,
+          -0.008831542916595936,
+          -0.009197707287967205,
+          -0.01707896590232849,
+          -0.013574247248470783,
+          0.023661212995648384,
+          0.016207145527005196,
+          -0.009354635141789913,
+          -0.034384604543447495,
+          0.03155990689992905,
+          0.00918898917734623,
+          0.027933133766055107,
+          -0.015544561669230461,
+          0.004343846347182989,
+          -0.0015213270671665668,
+          0.014707613736391068,
+          -0.006163771729916334,
+          -0.006686864420771599,
+          0.0091018071398139,
+          0.014620431698858738,
+          0.010897758416831493,
+          0.003245352301746607,
+          -0.003160349791869521,
+          0.007279702462255955,
+          0.002530459314584732,
+          0.0022285915911197662,
+          -0.008583074435591698,
+          -0.02282426506280899,
+          -0.009781827218830585,
+          0.017942069098353386,
+          0.022266298532485962,
+          0.012746017426252365,
+          -0.017131276428699493,
+          0.0020520477555692196,
+          -0.00484296353533864,
+          -0.014053748920559883,
+          -0.009991064667701721,
+          0.009642336517572403,
+          -0.033617403358221054,
+          0.02317299321293831,
+          0.004760140553116798,
+          -0.00627274950966239,
+          -0.0015006213216111064,
+          0.003836010815575719,
+          0.005679911468178034,
+          -0.004313332494348288,
+          0.0002961465797852725,
+          -0.002532638842239976,
+          0.010897758416831493,
+          0.016599465161561966,
+          0.005100150592625141,
+          -0.010400820523500443,
+          0.012388571165502071,
+          -0.0031407338101416826,
+          -0.002761491807177663,
+          -0.013914257287979126,
+          -0.006011203397065401,
+          -0.009502844884991646,
+          0.010261328890919685,
+          0.01483838725835085,
+          0.0054924702271819115,
+          0.006791482679545879,
+          0.004335128236562014,
+          -0.00013084120291750878,
+          -0.0002569146454334259,
+          0.008469737134873867,
+          -0.0005070181796327233,
+          -0.0045639812014997005,
+          0.0010821473551914096,
+          -0.0014679280575364828,
+          0.02320786565542221,
+          -0.00532246520742774,
+          -0.010950067080557346,
+          0.007196879480034113,
+          0.02033085748553276,
+          -0.022370917722582817,
+          0.022091934457421303,
+          -0.022179117426276207,
+          0.023853013291954994,
+          0.03482051566243172,
+          0.013975284993648529,
+          -0.011717269197106361,
+          -0.007096619810909033,
+          0.012815763242542744,
+          -0.012510626576840878,
+          -0.0009600924677215517,
+          0.0018079380970448256,
+          -0.022580154240131378,
+          0.015178397297859192,
+          -0.023347357288002968,
+          0.014280421659350395,
+          0.005121946334838867,
+          -0.00852640625089407,
+          -0.0073232934810221195,
+          -0.0009764390997588634,
+          0.018133869394659996,
+          -0.003990759141743183,
+          -0.0006255313055589795,
+          -0.03028704971075058,
+          -0.006930973846465349,
+          0.02163858897984028,
+          -0.0045247492380440235,
+          -0.013949130661785603,
+          0.007253547664731741,
+          -8.973622243502177e-06,
+          -0.016154836863279343,
+          -0.005876071285456419,
+          -0.03166452422738075,
+          -0.008142804726958275,
+          -0.02264990098774433,
+          -0.0075979167595505714,
+          -0.011577778495848179,
+          -0.012292671017348766,
+          -0.0038512677419930696,
+          -0.016276890411973,
+          0.001949608908034861,
+          -0.001063076313585043,
+          0.0033020207192748785,
+          0.0027091826777905226,
+          -0.009563872590661049,
+          -0.003968963399529457,
+          0.021324733272194862,
+          0.010705957189202309,
+          -0.016041500493884087,
+          -0.026206929236650467,
+          0.011638805270195007,
+          -0.012205488979816437,
+          -0.007593557704240084,
+          0.001547481631860137,
+          -0.005915303248912096,
+          -0.019267236813902855,
+          -0.01860465295612812,
+          0.005967612378299236,
+          0.0003718859807122499,
+          0.0051175872795283794,
+          0.013582965359091759,
+          0.014341449365019798,
+          -0.0019180054077878594,
+          -0.006734814494848251,
+          -1.330718441749923e-05,
+          -0.002195898210629821,
+          0.014027593657374382,
+          0.004348205402493477,
+          0.024445850402116776,
+          -0.008744360879063606,
+          0.0028944446239620447,
+          -0.017924632877111435,
+          0.00449859444051981,
+          -0.007393038831651211,
+          -0.005679911468178034,
+          -0.004064863547682762,
+          -0.011089558713138103,
+          0.005993766710162163,
+          -0.0011126610916107893,
+          -0.00247161160223186,
+          -0.007911772467195988,
+          -0.0020912797190248966,
+          -0.01707024872303009,
+          -0.0029794471338391304,
+          -0.020557530224323273,
+          -0.0013905538944527507,
+          -0.014916851185262203,
+          0.017000501975417137,
+          0.026172054931521416,
+          -0.00513502350077033,
+          -0.0006091846735216677,
+          -0.011229049414396286,
+          -4.08665910072159e-05,
+          -0.011909069493412971,
+          -0.00637736776843667,
+          -0.016416382044553757,
+          -0.0325189083814621,
+          -0.016904601827263832,
+          0.009049498476088047,
+          0.024288922548294067,
+          0.0019398009171709418,
+          -0.024986378848552704,
+          0.0029511128086596727,
+          -0.026607966050505638,
+          -0.014228112995624542,
+          -0.01496916078031063,
+          -0.006529936566948891,
+          -0.009136680513620377,
+          0.017392821609973907,
+          -0.0026350778061896563,
+          0.01571020856499672,
+          0.01747128553688526,
+          0.010906476527452469,
+          0.007497657556086779,
+          0.010880321264266968,
+          -0.009528999216854572,
+          -0.00016932391736190766,
+          0.009869009256362915,
+          -0.006311981473118067,
+          0.011839324608445168,
+          -0.00815152283757925,
+          -0.0011933045461773872,
+          0.01879645325243473,
+          0.008016390725970268,
+          -0.006255313288420439,
+          -0.0026699507143348455,
+          -0.02742747776210308,
+          0.017506157979369164,
+          -0.018552342429757118,
+          -0.00258494820445776,
+          -0.002944574225693941,
+          -0.016738956794142723,
+          0.02319042943418026,
+          0.017323076725006104,
+          -0.005021686665713787,
+          -0.0027549532242119312,
+          0.017889760434627533,
+          -0.006451472640037537,
+          0.00398204056546092,
+          0.0025762300938367844,
+          -0.0033281752839684486,
+          0.009790545329451561,
+          0.0027636713348329067,
+          -0.004577058367431164,
+          -0.0025587936397641897,
+          0.012196770869195461,
+          0.003901397343724966,
+          0.013530656695365906,
+          -0.027131058275699615,
+          -0.010566466487944126,
+          -0.0010303830495104194,
+          0.01935441792011261,
+          -0.005405288189649582,
+          -0.0007906323298811913,
+          -0.013312701135873795,
+          -0.0008854428306221962,
+          -0.005719143431633711,
+          0.023818138986825943,
+          -0.01992982067167759,
+          0.006120180711150169,
+          -0.008395632728934288,
+          0.013861947692930698,
+          -0.0032191977370530367,
+          0.010801857337355614,
+          -0.0007955363253131509,
+          -1.3375295566220302e-05,
+          0.009964910335838795,
+          0.012981409206986427,
+          -0.0022024367935955524,
+          0.024027377367019653,
+          -0.0004737800336442888,
+          -0.006403522565960884,
+          0.0027745692059397697,
+          -0.01895338110625744,
+          0.0229637548327446,
+          0.003402280155569315,
+          0.004315512254834175,
+          -0.0058629936538636684,
+          -0.008199472911655903,
+          0.039789892733097076,
+          0.021516533568501472,
+          0.010392102412879467,
+          -0.000592293159570545,
+          -0.01494300551712513,
+          0.028229551389813423,
+          0.007519452832639217,
+          0.00930232647806406,
+          -0.004012554418295622,
+          0.0027113622054457664,
+          -0.015178397297859192,
+          -0.010976221412420273,
+          0.019807765260338783,
+          -0.019999565556645393,
+          -0.008731283247470856,
+          -0.0026328982785344124,
+          -0.019703146070241928,
+          -0.029293173924088478,
+          0.015474816784262657,
+          0.009764390997588634,
+          0.0028443147893995047,
+          0.008495892398059368,
+          0.0037335718516260386,
+          -0.0015028008492663503,
+          -0.011647523380815983,
+          -0.003986400086432695,
+          -0.008574356324970722,
+          -0.01044441107660532,
+          0.006848151329904795,
+          -0.012004970572888851,
+          0.006717377807945013,
+          -0.008256141096353531,
+          -0.00024533577379770577,
+          0.01285063661634922,
+          0.005091432482004166,
+          0.0062771085649728775,
+          0.003443691413849592,
+          0.005078355316072702,
+          0.01591944508254528,
+          -0.013757329434156418,
+          0.016346637159585953,
+          0.017113838344812393,
+          0.0062945447862148285,
+          -0.0034131777938455343,
+          0.0023103246930986643,
+          -0.029188554733991623,
+          -0.00022381270537152886,
+          -0.0016869729151949286,
+          0.012632681056857109,
+          -0.02163858897984028,
+          0.004733986221253872,
+          0.021568842232227325,
+          -0.0016172273317351937,
+          0.009816700592637062,
+          0.014864541590213776,
+          -0.0001531135058030486,
+          0.028037751093506813,
+          0.011342386715114117,
+          0.02416686899960041,
+          -0.009459254331886768,
+          -0.02852597087621689,
+          0.006368649657815695,
+          -0.0038904997054487467,
+          0.014376322738826275,
+          0.0091018071398139,
+          0.0201216209679842,
+          -0.01369630265980959,
+          -0.006695582531392574,
+          0.007789717521518469,
+          0.0018918508430942893,
+          0.014306576922535896,
+          0.003217018209397793,
+          0.019302109256386757,
+          0.009119244292378426,
+          -0.0008625575574114919,
+          -0.007737408392131329,
+          8.588795753894374e-05,
+          0.006250953767448664,
+          -0.003500359831377864,
+          -0.03047885000705719,
+          0.0016390228411182761,
+          -0.004834245424717665,
+          0.004978095646947622,
+          -0.014245549216866493,
+          0.010409538634121418,
+          -0.0010015039006248116,
+          0.022196553647518158,
+          -0.0041498662903904915,
+          0.004064863547682762,
+          0.01708768494427204,
+          0.00938950851559639,
+          0.004991173278540373,
+          0.015300451777875423,
+          0.005658115725964308,
+          0.004511671606451273,
+          -0.008038186468183994,
+          -0.019493909552693367,
+          0.019162617623806,
+          -0.027462350204586983,
+          0.010653648525476456,
+          -0.00873564276844263,
+          0.004317691549658775,
+          0.003925372380763292,
+          -0.006887382827699184,
+          -0.002567511750385165,
+          0.003607157850638032,
+          0.001491903094574809,
+          -0.059144310653209686,
+          0.0037706242874264717,
+          0.0007388679659925401,
+          0.020452912896871567,
+          -0.00881846621632576,
+          0.0007590288296341896,
+          0.0026394370943307877,
+          0.00390357687138021,
+          -0.013347573578357697,
+          -0.0023604542948305607,
+          0.01571892574429512,
+          -0.01858721673488617,
+          -0.0059632533229887486,
+          0.004038709215819836,
+          -0.005557856522500515,
+          0.0035940806847065687,
+          -0.008779234252870083,
+          -0.002622000640258193,
+          0.02838647924363613,
+          -0.001655369414947927,
+          0.03140297904610634,
+          0.002506484277546406,
+          -0.0004756871494464576,
+          -0.01666049286723137,
+          0.007410475518554449,
+          -0.016738956794142723,
+          -0.00425884360447526,
+          0.008927443064749241,
+          -0.018726706504821777,
+          -0.00123580580111593,
+          0.001693511614575982,
+          -0.0013382446486502886,
+          -0.0077722808346152306,
+          -0.001763257198035717,
+          0.0010429153917357326,
+          -0.02048778533935547,
+          0.015997909009456635,
+          -0.023853013291954994,
+          0.01625945419073105,
+          -0.0034371528308838606,
+          -0.002946753753349185,
+          -0.0013033718569204211,
+          -0.0038098562508821487,
+          0.005666834302246571,
+          0.02531767077744007,
+          -0.015448661521077156,
+          -0.00522220553830266,
+          0.008365118876099586,
+          -0.003567925887182355,
+          0.002730978187173605,
+          -0.008761797100305557,
+          0.006656350567936897,
+          -0.0325886569917202,
+          -0.008112290874123573,
+          0.004413592163473368,
+          0.0031908636447042227,
+          -0.001236895564943552,
+          -0.02758440561592579,
+          0.003960245288908482,
+          -0.00854384247213602,
+          -0.024201741442084312,
+          -0.00027380615938454866,
+          -0.020592402666807175,
+          0.006717377807945013,
+          -0.00881846621632576,
+          0.01761949434876442,
+          -0.014568123035132885,
+          0.011098276823759079,
+          -0.014367603696882725,
+          0.016006626188755035,
+          -0.007406116463243961,
+          -0.009398226626217365,
+          -0.0069266147911548615,
+          -0.006338136270642281,
+          0.01044441107660532,
+          -0.014916851185262203,
+          -0.009511562995612621,
+          -0.007641507778316736,
+          0.005161178298294544,
+          0.011045967228710651,
+          0.013574247248470783,
+          0.0011333668371662498,
+          -0.00880538858473301,
+          -0.016547156497836113,
+          -0.005392210558056831,
+          -0.014881978742778301,
+          -0.0009301236714236438,
+          0.0042174323461949825,
+          0.020853949710726738,
+          0.008160240948200226,
+          -0.003753187833353877,
+          0.011586496606469154,
+          0.020801641047000885,
+          0.014132212847471237,
+          -0.014777359552681446,
+          0.0021697436459362507,
+          -0.010095682926476002,
+          0.013417319394648075,
+          0.022144244983792305,
+          0.002098908182233572,
+          0.002796364715322852,
+          0.006085308268666267,
+          0.007951004430651665,
+          -0.004396155476570129,
+          0.008291014470160007,
+          -0.025457162410020828,
+          -0.012667553499341011,
+          -0.006569168530404568,
+          0.005387851502746344,
+          -0.015378915704786777,
+          0.029973194003105164,
+          -0.003609337378293276,
+          0.004472440108656883,
+          -0.002182820811867714,
+          0.0326409637928009,
+          0.007196879480034113,
+          0.00473834527656436,
+          0.002874828642234206,
+          9.467427298659459e-05,
+          0.009136680513620377,
+          -0.01761077716946602,
+          -0.03888320177793503,
+          0.003254070645198226,
+          -0.000592293159570545,
+          0.01766308583319187,
+          -0.01514352485537529,
+          0.006878664717078209,
+          -0.0036899808328598738,
+          -0.01282448135316372,
+          -0.009633618406951427,
+          0.03138554468750954,
+          -0.005466315429657698,
+          0.03387894853949547,
+          -0.007558684796094894,
+          -0.010104401037096977,
+          0.021725770086050034,
+          0.021900134161114693,
+          -0.014210676774382591,
+          0.005531701724976301,
+          0.015762517228722572,
+          0.020278548821806908,
+          0.005209128372371197,
+          -0.00806434080004692,
+          0.02627667412161827,
+          0.012946536764502525,
+          -0.012807045131921768,
+          -0.0026350778061896563,
+          -0.009572590701282024,
+          0.006726096384227276,
+          -0.012301389127969742,
+          0.005387851502746344,
+          0.006434036418795586,
+          0.004986814223229885,
+          0.012963972985744476,
+          -0.01879645325243473,
+          -0.003831651760265231,
+          0.0073014977388083935,
+          0.013966566883027554,
+          0.00651250034570694,
+          0.017558468505740166,
+          -0.023643774911761284,
+          0.0028966241516172886,
+          -0.02493407018482685,
+          -0.008103572763502598,
+          0.0029968833550810814,
+          0.001748000388033688,
+          0.008875134401023388,
+          -0.01592816226184368,
+          -0.029484974220395088,
+          0.011516750790178776,
+          0.01708768494427204,
+          0.02568383514881134,
+          0.0011126610916107893,
+          0.008914366364479065,
+          0.00560144754126668,
+          -0.013739893212914467,
+          -0.00805998221039772,
+          -0.00997362844645977,
+          -0.01073211245238781,
+          -0.011159304529428482,
+          -0.002761491807177663,
+          0.009119244292378426,
+          -0.010592620819807053,
+          0.0006821996648795903,
+          0.016346637159585953,
+          -0.011717269197106361,
+          -0.0051960512064397335,
+          0.015535843558609486,
+          -0.005008609499782324,
+          0.0007977159111760557,
+          0.0016891524428501725,
+          0.013086027465760708,
+          0.0034480507019907236,
+          -0.0063730087131261826,
+          -0.01591944508254528,
+          0.01215317938476801,
+          -0.00417166156694293,
+          0.0016074193408712745,
+          0.00708354264497757,
+          -0.007933568209409714,
+          -0.0047819362953305244,
+          0.013373728841543198,
+          -0.00854384247213602,
+          0.0016259454423561692,
+          -0.007606634870171547,
+          -0.017549749463796616,
+          -0.0076938169077038765,
+          0.006394804455339909,
+          -0.026730021461844444,
+          -0.013478347100317478,
+          0.016233300790190697,
+          -0.003570105414837599,
+          0.001546391868032515,
+          -0.0005138292908668518,
+          -0.011124431155622005,
+          0.003160349791869521,
+          -0.019598528742790222,
+          -0.01064493041485548,
+          -0.004864759277552366,
+          -0.02662540227174759,
+          0.017549749463796616,
+          0.015091215260326862,
+          0.0033172776456922293,
+          -0.00699636060744524,
+          -0.005592729430645704,
+          0.002567511750385165,
+          0.010348510928452015,
+          0.01763693243265152,
+          0.009311044588685036,
+          -0.005592729430645704,
+          -0.019685709848999977,
+          -0.003443691413849592,
+          0.021324733272194862,
+          -6.375188240781426e-05,
+          -0.016590746119618416,
+          0.020609840750694275,
+          0.007109697442501783,
+          0.014306576922535896,
+          0.0055883703753352165,
+          0.0023451976012438536,
+          0.005383492447435856,
+          0.014114775694906712,
+          -0.0001486181718064472,
+          -0.01783744990825653,
+          -0.013984003104269505,
+          0.0004890369018539786,
+          0.03140297904610634,
+          0.00708354264497757,
+          0.017139993607997894,
+          -0.03623286634683609,
+          0.0006642183288931847,
+          -0.005335542373359203,
+          -0.0038490882143378258,
+          -0.0045988536439836025,
+          0.027671586722135544,
+          0.010766984894871712,
+          -0.0001184450084110722,
+          0.015605589374899864,
+          -0.010043373331427574,
+          0.0422309935092926,
+          -0.015378915704786777,
+          -0.00737560261040926,
+          -0.010052092373371124,
+          -0.01685229316353798,
+          -0.003940629307180643,
+          -0.0011769578559324145,
+          -0.01689588464796543,
+          -0.006647632457315922,
+          -0.006320699583739042,
+          0.021411914378404617,
+          0.0046380856074392796,
+          -0.01665177382528782,
+          0.0017065888969227672,
+          -0.023835577070713043,
+          -0.0010946798138320446,
+          -0.022475536912679672,
+          -0.006403522565960884,
+          -0.012327543459832668,
+          -0.01349578332155943,
+          -0.020993441343307495,
+          -0.02031342126429081,
+          -0.004908350296318531,
+          0.016547156497836113,
+          0.00532246520742774,
+          -0.0069091785699129105,
+          -0.0021043571177870035,
+          -0.014402477070689201,
+          -0.010278765112161636,
+          0.02144678682088852,
+          0.015605589374899864,
+          0.012432162649929523,
+          0.01332141924649477,
+          0.01685229316353798,
+          0.0037597264163196087,
+          -0.034593842923641205,
+          0.006128899287432432,
+          0.0041607641614973545,
+          -0.005435801576822996,
+          0.01782873272895813,
+          0.000511104823090136,
+          -0.010793139226734638,
+          0.013399883173406124,
+          -0.007593557704240084,
+          0.018098996952176094,
+          -0.003212659154087305,
+          -0.004184739198535681,
+          -0.005536061245948076,
+          -0.008740002289414406,
+          -0.007371243555098772,
+          0.00455090356990695,
+          0.0021043571177870035,
+          0.013643993064761162,
+          0.0005775811732746661,
+          -0.004625008441507816,
+          -0.011333668604493141,
+          0.0014602995943278074,
+          -0.0032257363200187683,
+          -0.001628125086426735,
+          -0.023835577070713043,
+          -0.012170616537332535,
+          0.0017120378324761987,
+          -0.0086179468780756,
+          -0.01235369872301817,
+          0.02856084331870079,
+          -0.0017011400777846575,
+          -0.0008009852026589215,
+          0.010688520967960358,
+          0.015675334259867668,
+          0.001266319421119988,
+          0.01837797835469246,
+          -0.012170616537332535,
+          -0.011185458861291409,
+          -0.01649484597146511,
+          -0.01282448135316372,
+          -0.01122033130377531,
+          0.02090625837445259,
+          0.0033979208674281836,
+          -0.00708354264497757,
+          0.010418256744742393,
+          -0.011106994934380054,
+          -0.005056559573858976,
+          -0.0091018071398139,
+          0.011708551086485386,
+          -0.02969421073794365,
+          0.02069702185690403,
+          -0.012711144983768463,
+          0.006024280562996864,
+          0.018639525398612022,
+          0.039231929928064346,
+          -0.006608400493860245,
+          -0.006869946606457233,
+          0.010034655220806599,
+          -0.003068808699026704,
+          0.029205990955233574,
+          -0.0059414575807750225,
+          0.004237048327922821,
+          -0.021551406010985374,
+          0.0018449904164299369,
+          -0.01074954867362976,
+          0.0012815763475373387,
+          0.01991238258779049,
+          -0.0191974900662899,
+          -0.013722456991672516,
+          -0.00675660977140069,
+          -0.016058936715126038,
+          -0.00823870487511158,
+          0.022283734753727913,
+          0.011377259157598019,
+          -0.006111462600529194,
+          -0.008273578248918056,
+          -0.005736579652875662,
+          -0.024201741442084312,
+          0.014716332778334618,
+          0.08174190670251846,
+          -0.009982346557080746,
+          -0.0001858067698776722,
+          0.0003367407189216465,
+          0.0017806936521083117,
+          0.007907412946224213,
+          -0.00031821453012526035,
+          0.05956278741359711,
+          -0.003509078174829483,
+          0.003899217816069722,
+          -0.012998845428228378,
+          0.01083673071116209,
+          -0.01438504084944725,
+          -0.005684270523488522,
+          -0.009712082333862782,
+          -0.01197881530970335,
+          -0.0008167869527824223,
+          -0.02203962579369545,
+          -0.001063076313585043,
+          -0.011996251530945301,
+          -0.005021686665713787,
+          -0.021324733272194862,
+          0.017514877021312714,
+          -0.02087138593196869,
+          -0.01648612879216671,
+          -0.004594494588673115,
+          -0.010705957189202309,
+          0.012118306942284107,
+          -0.016555873677134514,
+          0.002380070276558399,
+          0.007270984351634979,
+          0.014228112995624542,
+          0.007475861813873053,
+          -0.008199472911655903,
+          -0.008408710360527039,
+          -0.0076197125017642975,
+          0.004986814223229885,
+          0.0069876424968242645,
+          -0.0011088468600064516,
+          0.014637868851423264,
+          -0.02088882215321064,
+          -0.016058936715126038,
+          0.007589198648929596,
+          0.0017817834159359336,
+          0.009502844884991646,
+          -0.007820230908691883,
+          0.007449707482010126,
+          0.00020882829267065972,
+          0.022754518315196037,
+          -0.010156710632145405,
+          -0.0268520750105381,
+          -0.007933568209409714,
+          0.008644101209938526,
+          -0.013138337060809135,
+          -0.006102744489908218,
+          0.005331183318048716,
+          0.009807982482016087,
+          0.008081777021288872,
+          -0.009877728298306465,
+          0.0007110786973498762,
+          -0.018918508663773537,
+          0.013373728841543198,
+          0.003238813718780875,
+          0.022196553647518158,
+          -0.012667553499341011,
+          0.010880321264266968,
+          -0.02683463878929615,
+          -0.002574050333350897,
+          -0.014419913291931152,
+          0.0008592882077209651,
+          0.008434864692389965,
+          0.004742704331874847,
+          0.01684357412159443,
+          -0.017192302271723747,
+          -0.017096402123570442,
+          0.009311044588685036,
+          -0.01782873272895813,
+          -0.02376583032310009,
+          -0.0018471699440851808,
+          -0.02624180167913437,
+          -0.0038163948338478804,
+          0.007532530464231968,
+          -0.019389290362596512,
+          -0.01369630265980959,
+          0.0065953233279287815,
+          -0.0033412526827305555,
+          -0.014611713588237762,
+          0.017183585092425346,
+          -0.011970097199082375,
+          -0.03544822707772255,
+          0.003727033268660307,
+          0.008382555097341537,
+          -0.004729627165943384,
+          0.006747891660779715,
+          -0.024410977959632874,
+          0.016015345230698586,
+          -0.002397506730630994,
+          0.004293716512620449,
+          0.008029468357563019,
+          0.02606743760406971,
+          0.0010309278732165694,
+          0.0034545892849564552,
+          -0.005544779356569052,
+          0.014411195181310177,
+          0.005801966413855553,
+          0.023504285141825676,
+          0.026416165754199028,
+          0.0015529305674135685,
+          -0.010958785191178322,
+          -0.01081929448992014,
+          0.01592816226184368,
+          -0.011586496606469154,
+          -0.0048386044800281525,
+          0.022737082093954086,
+          0.013827075250446796,
+          -0.02203962579369545,
+          0.00017259323794860393,
+          -0.02149909734725952,
+          0.0069876424968242645,
+          -0.017148712649941444,
+          0.009467972442507744,
+          0.008731283247470856,
+          -0.005073996260762215,
+          0.02069702185690403,
+          0.010095682926476002,
+          0.030984506011009216,
+          0.01991238258779049,
+          0.002920599188655615,
+          0.011412132531404495,
+          -0.006451472640037537,
+          0.010470565408468246,
+          0.004537826403975487,
+          0.0022275017108768225,
+          -0.006608400493860245,
+          0.006930973846465349,
+          0.0023451976012438536,
+          -0.001170419156551361,
+          0.008583074435591698,
+          -0.007471502758562565,
+          0.009921318851411343,
+          0.0005939278053119779,
+          0.0055273426696658134,
+          -0.004287178162485361,
+          -0.010575184598565102,
+          -0.030339358374476433,
+          -0.003132015699520707,
+          0.020958568900823593,
+          -0.023539157584309578,
+          -0.008003313094377518,
+          0.005932739470154047,
+          0.016983065754175186,
+          -0.008282296359539032,
+          -0.012798327021300793,
+          -0.01708768494427204,
+          0.003105861134827137,
+          -0.0004893093137070537,
+          -0.010313638485968113,
+          0.01747128553688526,
+          0.0009546435903757811,
+          -0.004951941315084696,
+          0.008953598327934742,
+          -0.00680456031113863,
+          0.028212115168571472,
+          0.01505634281784296,
+          0.011150586418807507,
+          -0.012100870721042156,
+          0.014201958663761616,
+          -0.012711144983768463,
+          0.007680739741772413,
+          0.009799264371395111,
+          -0.006416599731892347,
+          -0.021220114082098007,
+          -0.017558468505740166,
+          0.023451974615454674,
+          0.006848151329904795,
+          0.012222925201058388,
+          -0.011987533420324326,
+          -0.0012815763475373387,
+          0.007475861813873053,
+          0.004834245424717665,
+          -0.01611996442079544,
+          -0.00048631246318109334,
+          0.0061986446380615234,
+          0.038604218512773514,
+          0.005087073426693678,
+          0.01072339341044426,
+          -0.0035984397400170565,
+          -0.0002066487359115854,
+          0.019162617623806,
+          -0.01952878199517727,
+          -0.006503781769424677,
+          -0.005832480266690254,
+          -0.02739260531961918,
+          0.016573309898376465,
+          -0.01685229316353798,
+          -0.016451256349682808,
+          0.010976221412420273,
+          -0.00041356991278007627,
+          -0.004422310274094343,
+          -0.01284191757440567,
+          -0.008766156621277332,
+          0.00794228632003069,
+          0.010810576379299164,
+          -0.009136680513620377,
+          0.00532246520742774,
+          0.00357446470297873,
+          -0.0035984397400170565,
+          0.02127242274582386,
+          0.0006467819330282509,
+          -0.002837776206433773,
+          0.007358166389167309
+        ],
+        "title": "Method",
+        "keyphrases": [
+          "mixture-of-experts (MoE) models",
+          "token-choice routing",
+          "load imbalance",
+          "expert choice method",
+          "Transformer architecture"
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "33412f87-7418-4ce2-9116-377a27fd2732",
+      "properties": {
+        "page_content": "4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n",
+        "title": "4 Experiments",
+        "embedding": [
+          0.009938324801623821,
+          -0.009144282899796963,
+          -0.026211904361844063,
+          -0.022352691739797592,
+          0.016043050214648247,
+          0.016247965395450592,
+          -0.02783413790166378,
+          0.0537557490170002,
+          -0.02827811799943447,
+          -0.01755429059267044,
+          0.010237157344818115,
+          0.001297788112424314,
+          0.014762339182198048,
+          -0.01001516729593277,
+          0.007487895432859659,
+          0.015283161774277687,
+          0.019347287714481354,
+          0.019910801202058792,
+          0.02535809576511383,
+          -0.04251963645219803,
+          0.0020213902462273836,
+          -0.032034873962402344,
+          -0.058332156389951706,
+          0.0034600566141307354,
+          0.001006959704682231,
+          0.01755429059267044,
+          0.011645940132439137,
+          -0.0015795447397977114,
+          -0.053584985435009,
+          0.04794985428452492,
+          0.016555335372686386,
+          0.0044910297729074955,
+          -0.007833687588572502,
+          -0.004798400681465864,
+          -0.04204150289297104,
+          -0.028961164876818657,
+          0.05720513314008713,
+          0.005165537819266319,
+          -0.005519868340343237,
+          0.05481446906924248,
+          0.020252324640750885,
+          0.010245694778859615,
+          -0.03961668908596039,
+          -0.009904172271490097,
+          0.010134699754416943,
+          0.001490962109528482,
+          -0.0010427129454910755,
+          -0.007479357533156872,
+          0.02295035682618618,
+          -0.04146091267466545,
+          -0.005549751687794924,
+          -0.03811398521065712,
+          0.01863008923828602,
+          0.013780459761619568,
+          -0.04426140338182449,
+          0.023223575204610825,
+          -0.0052978782914578915,
+          0.03527934476733208,
+          0.057854026556015015,
+          0.02783413790166378,
+          0.024162763729691505,
+          -0.02590453252196312,
+          -0.04340759664773941,
+          -0.0036180110182613134,
+          0.010143238119781017,
+          0.008119713515043259,
+          -0.01344747468829155,
+          -0.010527451522648335,
+          0.046173933893442154,
+          -0.023701708763837814,
+          -0.05327761545777321,
+          0.027680452913045883,
+          -0.013584083877503872,
+          -0.008081291802227497,
+          0.0036372218746691942,
+          -0.03517688810825348,
+          0.012807118706405163,
+          0.03548425808548927,
+          0.01004078146070242,
+          0.015018481761217117,
+          0.001224147155880928,
+          0.008418546058237553,
+          -0.0014920294051989913,
+          0.017912890762090683,
+          0.028517184779047966,
+          -0.01417321152985096,
+          -0.006309640593826771,
+          -0.0686461552977562,
+          0.025767924264073372,
+          -0.007368362508714199,
+          -0.020115714520215988,
+          0.00026281276950612664,
+          -0.007099412847310305,
+          0.0033597343135625124,
+          -0.020781684666872025,
+          0.07472527027130127,
+          -0.013105952180922031,
+          0.014386664144694805,
+          -0.02096952311694622,
+          0.007300057914108038,
+          -0.031932417303323746,
+          0.02236976847052574,
+          0.01146664097905159,
+          0.005916888825595379,
+          0.038318898528814316,
+          -0.02834642305970192,
+          -0.005869929678738117,
+          0.0122009152546525,
+          0.0292856115847826,
+          0.08387809246778488,
+          0.01419028826057911,
+          -0.014352511614561081,
+          -0.025563009083271027,
+          -0.016538260504603386,
+          -0.046105626970529556,
+          -0.0292856115847826,
+          -0.025170257315039635,
+          -0.027099864557385445,
+          0.036338064819574356,
+          -0.0010613900376483798,
+          0.012388752773404121,
+          0.05338007211685181,
+          0.015863751992583275,
+          -0.00874299369752407,
+          -0.039241012185811996,
+          0.0438857264816761,
+          -0.006198645569384098,
+          0.010809208266437054,
+          0.002330895746126771,
+          -0.0146428057923913,
+          -0.006185838486999273,
+          0.011833777651190758,
+          -0.013763383962213993,
+          -0.029456373304128647,
+          0.031949494034051895,
+          -0.01727253384888172,
+          0.03944592550396919,
+          0.06239628419280052,
+          -0.03562086820602417,
+          -0.02001325786113739,
+          0.03189826384186745,
+          -0.035825781524181366,
+          0.018288565799593925,
+          -0.0005250918911769986,
+          -0.030993228778243065,
+          0.026246055960655212,
+          0.01562468521296978,
+          0.06092773377895355,
+          -0.024196917191147804,
+          0.028705023229122162,
+          -0.019176525995135307,
+          0.01628211699426174,
+          -0.020627999678254128,
+          0.03596239164471626,
+          0.0023351646959781647,
+          -0.024265222251415253,
+          -0.015752756968140602,
+          0.014694035053253174,
+          -0.014275669120252132,
+          -0.022830823436379433,
+          0.019740039482712746,
+          0.02100367471575737,
+          -0.010450609028339386,
+          -0.0049051265232264996,
+          0.012013077735900879,
+          0.04989653453230858,
+          0.016205275431275368,
+          -0.014130521565675735,
+          -0.07479357719421387,
+          0.04801815748214722,
+          0.021703798323869705,
+          0.01958635449409485,
+          0.023104043677449226,
+          0.035825781524181366,
+          -0.001056053675711155,
+          0.030566323548555374,
+          0.0019007899099960923,
+          -0.013601160608232021,
+          -0.02044016122817993,
+          0.043646663427352905,
+          -0.0039019023533910513,
+          -0.0019274713704362512,
+          0.0076373121701180935,
+          0.018288565799593925,
+          0.03123229369521141,
+          0.005430218297988176,
+          0.010399380698800087,
+          0.039719145745038986,
+          -0.011586173437535763,
+          0.01414759736508131,
+          0.009460192173719406,
+          0.02091829478740692,
+          0.04299776628613472,
+          0.02245514839887619,
+          -0.008670419454574585,
+          -0.02194286324083805,
+          0.00032551426556892693,
+          0.0219599399715662,
+          0.01811780408024788,
+          0.02199409157037735,
+          -0.012013077735900879,
+          -0.0220111683011055,
+          0.009383349679410458,
+          -0.02199409157037735,
+          0.01682855375111103,
+          -0.00167666538618505,
+          0.01859593763947487,
+          0.013865840621292591,
+          -0.006258412264287472,
+          -0.01677732542157173,
+          0.0002502724528312683,
+          -0.018186109140515327,
+          -0.008380125276744366,
+          -0.010945817455649376,
+          -0.016495568677783012,
+          0.03364003449678421,
+          -0.020747533068060875,
+          0.006599935237318277,
+          -0.01810072734951973,
+          -0.011885006912052631,
+          -0.005835777148604393,
+          0.03596239164471626,
+          0.012209453620016575,
+          -0.01851055584847927,
+          -0.026058217510581017,
+          -0.011108040809631348,
+          -0.018817927688360214,
+          -0.006463326048105955,
+          0.0342206209897995,
+          -0.01750306226313114,
+          0.006659701932221651,
+          -0.004211407620459795,
+          -0.002409872831776738,
+          0.004258367232978344,
+          0.033435117453336716,
+          -0.03082246705889702,
+          -0.01676025055348873,
+          0.023582175374031067,
+          0.008059946820139885,
+          0.029046546667814255,
+          -0.016888320446014404,
+          -0.006275488529354334,
+          -0.024231068789958954,
+          -0.016973702237010002,
+          -0.01643580198287964,
+          -0.005118578672409058,
+          -0.02199409157037735,
+          -0.016025975346565247,
+          0.02790244296193123,
+          0.013686541467905045,
+          -0.00790199264883995,
+          -0.019313136115670204,
+          -0.002333030104637146,
+          0.02542640082538128,
+          -0.013592622242867947,
+          0.04064125940203667,
+          0.050545431673526764,
+          -0.05713682621717453,
+          -0.0122521435841918,
+          -0.03661128506064415,
+          0.07684271782636642,
+          -0.015377080999314785,
+          -0.03032725863158703,
+          0.033964481204748154,
+          -0.029388070106506348,
+          -0.008990597911179066,
+          0.026997406035661697,
+          0.025272715836763382,
+          -0.014719649218022823,
+          0.01123611256480217,
+          0.03756754845380783,
+          -0.030139420181512833,
+          -0.0439881831407547,
+          -0.03777246177196503,
+          0.019859572872519493,
+          -0.025289790704846382,
+          -0.01419028826057911,
+          -0.0110311983153224,
+          0.023530947044491768,
+          0.01801534742116928,
+          0.010664060711860657,
+          0.007547662127763033,
+          0.006843270733952522,
+          0.021157361567020416,
+          0.02144765481352806,
+          0.007073798682540655,
+          -0.03237639740109444,
+          0.0003433908859733492,
+          -0.03944592550396919,
+          -0.009093054570257664,
+          -0.025289790704846382,
+          -0.02095244638621807,
+          -0.023821242153644562,
+          0.00949434470385313,
+          0.03275207430124283,
+          -0.0008143192972056568,
+          0.0067365444265306,
+          0.015044095925986767,
+          0.015488076023757458,
+          -0.03807983547449112,
+          -0.03555256128311157,
+          0.006348061840981245,
+          0.01682855375111103,
+          -0.0055625587701797485,
+          -0.059220120310783386,
+          -0.00042770442087203264,
+          -0.003165493020787835,
+          -0.005016121547669172,
+          -0.028909936547279358,
+          0.010476223193109035,
+          0.013345018029212952,
+          0.010237157344818115,
+          0.0110311983153224,
+          -0.027526767924427986,
+          -0.014651344157755375,
+          -0.0036073385272175074,
+          0.006019345950335264,
+          0.02100367471575737,
+          -0.040709562599658966,
+          -0.022830823436379433,
+          0.03818229213356972,
+          0.0003810117777902633,
+          -0.06598227471113205,
+          -0.005387527868151665,
+          -0.010757979936897755,
+          0.015436847694218159,
+          -0.0005656477296724916,
+          0.034476764500141144,
+          -0.03121521882712841,
+          -0.019296059384942055,
+          -0.020815838128328323,
+          -0.051979828625917435,
+          -0.03531349450349808,
+          -0.015436847694218159,
+          0.007389707490801811,
+          -0.038353051990270615,
+          -0.024265222251415253,
+          0.004111085087060928,
+          -0.0268949493765831,
+          0.007398245856165886,
+          -0.013567008078098297,
+          0.015633223578333855,
+          -0.03852381557226181,
+          -0.010092009790241718,
+          0.004262635949999094,
+          -0.01861301250755787,
+          -0.005507061257958412,
+          0.014369587413966656,
+          0.013925607316195965,
+          0.014719649218022823,
+          0.0017268265364691615,
+          0.015266085974872112,
+          0.0062071834690868855,
+          -0.00769280968233943,
+          0.007833687588572502,
+          -0.04091447591781616,
+          0.0014461372047662735,
+          0.018852079287171364,
+          0.011432488448917866,
+          -0.011082426644861698,
+          0.004151640925556421,
+          -0.010373766534030437,
+          -0.03034433349967003,
+          0.029917430132627487,
+          -0.05419972911477089,
+          0.0024141420144587755,
+          0.005554020404815674,
+          -0.016410188749432564,
+          -0.008384393528103828,
+          -0.009690719656646252,
+          -0.03708941861987114,
+          -0.01248267199844122,
+          0.0039019023533910513,
+          0.00019050588889513165,
+          0.009349197149276733,
+          0.002870929194614291,
+          0.05471201241016388,
+          -0.003165493020787835,
+          -0.04484199360013008,
+          0.023308956995606422,
+          -0.03671374171972275,
+          0.018749622628092766,
+          0.04245132952928543,
+          0.020713381469249725,
+          -0.017212767153978348,
+          0.04791570082306862,
+          0.0195692777633667,
+          0.000224524803343229,
+          -0.016999315470457077,
+          0.025153182446956635,
+          -0.007099412847310305,
+          -0.061303410679101944,
+          -0.004448339343070984,
+          -0.018749622628092766,
+          0.018664240837097168,
+          0.016674868762493134,
+          -0.002407738473266363,
+          -0.010032243095338345,
+          0.009426039643585682,
+          -0.01510386262089014,
+          -0.017443295568227768,
+          0.01705908216536045,
+          -0.011458102613687515,
+          -0.004525181837379932,
+          0.007526317145675421,
+          0.013148642145097256,
+          0.01271320041269064,
+          0.008853988721966743,
+          0.023992003872990608,
+          -0.045627497136592865,
+          0.009127207100391388,
+          -0.01997910626232624,
+          -0.036918655037879944,
+          -0.002990462351590395,
+          0.01318279467523098,
+          0.04183658957481384,
+          -0.02779998630285263,
+          0.0019253368955105543,
+          0.005007583647966385,
+          -0.013515779748558998,
+          -0.021106131374835968,
+          0.00341309723444283,
+          0.04600317031145096,
+          -0.0015763428527861834,
+          0.021618416532874107,
+          -0.011304416693747044,
+          -0.03562086820602417,
+          0.008811297826468945,
+          0.04142676293849945,
+          -0.005028928630053997,
+          0.01663217879831791,
+          0.009434578008949757,
+          0.03548425808548927,
+          -0.04590071365237236,
+          -0.007833687588572502,
+          -0.02344556525349617,
+          -0.01813488081097603,
+          0.04105108603835106,
+          -0.03075416199862957,
+          -0.01365238893777132,
+          0.03524519130587578,
+          -0.005306416191160679,
+          0.02441890724003315,
+          -0.018271489068865776,
+          -0.02588745579123497,
+          0.0014600115828216076,
+          -0.03712356835603714,
+          -0.00708660576492548,
+          -0.01806657575070858,
+          -0.006160224322229624,
+          -0.06557244807481766,
+          0.00010459146869834512,
+          -0.07431544363498688,
+          -0.002902946900576353,
+          0.025136105716228485,
+          -0.006552976090461016,
+          0.04429555684328079,
+          -0.026228979229927063,
+          0.02098659984767437,
+          -0.05553166940808296,
+          -0.01367800310254097,
+          0.00978463888168335,
+          0.03375956416130066,
+          0.007611697539687157,
+          -0.001995776081457734,
+          0.0015507286880165339,
+          -0.026997406035661697,
+          -0.0018303507240489125,
+          -0.007282981649041176,
+          -0.001387437921948731,
+          4.89605481561739e-05,
+          -0.018852079287171364,
+          0.00950288213789463,
+          0.000838332693092525,
+          0.01610281690955162,
+          0.031454283744096756,
+          0.022711291909217834,
+          -0.03070293366909027,
+          -0.008862526156008244,
+          0.005195421166718006,
+          0.02587038092315197,
+          -0.02978082001209259,
+          0.003897633170709014,
+          -0.03464752435684204,
+          0.014864795841276646,
+          5.636466448777355e-05,
+          0.016657792031764984,
+          0.008008718490600586,
+          0.017016392201185226,
+          -0.040197279304265976,
+          0.0030160765163600445,
+          -0.0016254368238151073,
+          -0.027202321216464043,
+          0.020867066457867622,
+          0.002305281348526478,
+          -0.022147778421640396,
+          0.0054472945630550385,
+          -0.004207138437777758,
+          0.024213993921875954,
+          0.04473953694105148,
+          -0.0010592554463073611,
+          -0.02930268831551075,
+          -0.012704662047326565,
+          0.042212262749671936,
+          0.02633143588900566,
+          -0.028483033180236816,
+          0.017118848860263824,
+          0.02103782817721367,
+          -0.02341141365468502,
+          -0.021089056506752968,
+          0.013925607316195965,
+          -0.016179660335183144,
+          -0.0007694943924434483,
+          -0.005468640010803938,
+          0.010399380698800087,
+          0.01240582950413227,
+          -0.029883278533816338,
+          0.05676115304231644,
+          -0.016820017248392105,
+          -0.006834732368588448,
+          -0.03034433349967003,
+          -0.006168762221932411,
+          -0.014531810767948627,
+          0.05553166940808296,
+          0.0028175662737339735,
+          0.02728770114481449,
+          0.01658095046877861,
+          0.002264725510030985,
+          0.05628301948308945,
+          -0.03678204491734505,
+          -0.03451091796159744,
+          0.018681317567825317,
+          -0.0055625587701797485,
+          -0.00671093026176095,
+          -0.022728366777300835,
+          -0.014207364059984684,
+          -0.013353556394577026,
+          0.016965163871645927,
+          -0.0268778745085001,
+          -0.012260681949555874,
+          0.019740039482712746,
+          0.017690900713205338,
+          -2.0469710761972237e-06,
+          0.02293328195810318,
+          0.012644895352423191,
+          0.014284206554293633,
+          0.017878737300634384,
+          -0.020867066457867622,
+          0.019142374396324158,
+          0.024111535400152206,
+          0.009946862235665321,
+          -0.01763967238366604,
+          -0.017878737300634384,
+          0.007347017526626587,
+          -0.0038613462820649147,
+          -0.007125027012079954,
+          0.01437812577933073,
+          0.007043915335088968,
+          -0.004047049675136805,
+          -0.027151092886924744,
+          -0.018356870859861374,
+          0.02093537151813507,
+          -0.0023757205344736576,
+          0.01539415679872036,
+          -0.00928943045437336,
+          -0.010612832382321358,
+          0.03763585537672043,
+          0.01861301250755787,
+          -0.046139780431985855,
+          0.026280207559466362,
+          0.01051037572324276,
+          -0.006228528916835785,
+          -0.013729231432080269,
+          0.005626594182103872,
+          0.009101592935621738,
+          0.010220080614089966,
+          0.016572412103414536,
+          -0.005097233224660158,
+          -0.03328143432736397,
+          0.01564176194369793,
+          0.026211904361844063,
+          -0.020798761397600174,
+          0.012021616101264954,
+          0.00017663151083979756,
+          0.022762520238757133,
+          -0.005028928630053997,
+          -0.012311910279095173,
+          -0.01562468521296978,
+          0.009741948917508125,
+          -0.012235067784786224,
+          0.01123611256480217,
+          0.037396788597106934,
+          0.015914980322122574,
+          -0.03565501794219017,
+          0.015735680237412453,
+          -0.01686270721256733,
+          -0.015684451907873154,
+          0.03261546418070793,
+          0.008264861069619656,
+          -0.02636558935046196,
+          -0.007142103277146816,
+          -0.006527361460030079,
+          0.02001325786113739,
+          -0.02102075144648552,
+          -0.008508196100592613,
+          0.004418455995619297,
+          0.0035646480973809958,
+          0.009938324801623821,
+          0.006745082791894674,
+          0.005016121547669172,
+          0.019159449264407158,
+          0.013806073926389217,
+          -0.0005186883499845862,
+          0.012730276212096214,
+          0.011577636003494263,
+          0.002902946900576353,
+          0.0075391242280602455,
+          -0.010997045785188675,
+          -0.02295035682618618,
+          0.004734364803880453,
+          0.05945918336510658,
+          -0.005852853413671255,
+          0.03876287862658501,
+          0.010407919064164162,
+          0.028961164876818657,
+          0.021686721593141556,
+          -0.024299373850226402,
+          -0.01681147888302803,
+          -0.018151957541704178,
+          -0.021737949922680855,
+          -0.04453462362289429,
+          -0.01510386262089014,
+          -0.01629919372498989,
+          -0.013404784724116325,
+          0.005694898776710033,
+          -0.00013474156730808318,
+          -0.0013116624904796481,
+          0.011355645954608917,
+          -0.002003246918320656,
+          -0.012491210363805294,
+          -0.003355465130880475,
+          0.05324346199631691,
+          -0.0011056812945753336,
+          0.017912890762090683,
+          -0.004576410632580519,
+          0.015539304353296757,
+          -0.007300057914108038,
+          -0.016111355274915695,
+          0.029131926596164703,
+          0.0035561099648475647,
+          -0.0007540191290900111,
+          0.012106996960937977,
+          0.004469684325158596,
+          0.04149506613612175,
+          -0.005887005943804979,
+          -0.026058217510581017,
+          0.011910621076822281,
+          -0.017187153920531273,
+          0.020371858030557632,
+          0.00769707839936018,
+          -0.007607428822666407,
+          0.001932807732373476,
+          0.012909576296806335,
+          -0.027116939425468445,
+          0.019705887883901596,
+          -0.020645076408982277,
+          -0.045149363577365875,
+          0.009280892089009285,
+          -0.01175693515688181,
+          0.023035738617181778,
+          0.001117421081289649,
+          -0.010800670832395554,
+          -0.03507443144917488,
+          0.0021387888118624687,
+          0.0039040367119014263,
+          -0.03372541442513466,
+          0.006548706907778978,
+          0.01863008923828602,
+          0.015539304353296757,
+          0.009485806338489056,
+          -0.012465596199035645,
+          0.003758889390155673,
+          0.0031847036443650723,
+          -0.012371676973998547,
+          0.02776583470404148,
+          -0.004742903169244528,
+          -0.031983643770217896,
+          0.035859934985637665,
+          -0.02733892947435379,
+          0.009468729607760906,
+          -0.020730456337332726,
+          -0.012346062809228897,
+          -0.01608574204146862,
+          0.00441418681293726,
+          -0.02148180827498436,
+          -0.002561423694714904,
+          -0.02983205020427704,
+          0.027577996253967285,
+          0.009383349679410458,
+          -0.015249010175466537,
+          -0.004862436093389988,
+          0.009101592935621738,
+          -0.025255639106035233,
+          0.007846495136618614,
+          0.012585128657519817,
+          -0.010646984912455082,
+          -0.01367800310254097,
+          -0.00706099160015583,
+          -0.0014685495989397168,
+          0.005972386337816715,
+          0.0158210601657629,
+          0.04672037065029144,
+          0.004431263078004122,
+          0.03712356835603714,
+          0.02337726205587387,
+          0.013532855547964573,
+          0.0037012575194239616,
+          0.0047898623161017895,
+          -0.0067322757095098495,
+          0.04012897238135338,
+          0.006249873898923397,
+          0.0219599399715662,
+          0.008247784338891506,
+          -0.044671230018138885,
+          0.008503926917910576,
+          0.03264961391687393,
+          0.026621730998158455,
+          0.018715469166636467,
+          0.009588262997567654,
+          -0.0024013349320739508,
+          0.021191513165831566,
+          -0.020252324640750885,
+          -0.01705908216536045,
+          0.0035902622621506453,
+          -0.007530585862696171,
+          -0.0056650154292583466,
+          -0.009716334752738476,
+          0.03387909755110741,
+          -0.003229528432711959,
+          -0.008862526156008244,
+          0.018186109140515327,
+          -0.007432397920638323,
+          0.00475997943431139,
+          0.006245605181902647,
+          0.0011163539020344615,
+          0.02739015780389309,
+          0.008683227002620697,
+          0.027680452913045883,
+          0.005391797050833702,
+          0.007573276292532682,
+          0.0058571225963532925,
+          -0.009451653808355331,
+          0.018920384347438812,
+          -0.0097590247169137,
+          -0.033366814255714417,
+          -0.009443115442991257,
+          0.0016393112018704414,
+          0.014070754870772362,
+          0.0004023569927085191,
+          -0.014062216505408287,
+          -0.011893544346094131,
+          0.007859301753342152,
+          0.012533900327980518,
+          0.02107197977602482,
+          0.018169032409787178,
+          -0.005016121547669172,
+          -0.024726277217268944,
+          0.012593667022883892,
+          0.011056812480092049,
+          -0.01861301250755787,
+          0.008465506136417389,
+          0.012542438693344593,
+          0.057410046458244324,
+          -0.044705383479595184,
+          0.016538260504603386,
+          0.02151595987379551,
+          -0.004896588623523712,
+          -0.015249010175466537,
+          -0.010851899161934853,
+          0.0031633584294468164,
+          -0.017477449029684067,
+          0.014975790865719318,
+          -0.01798119582235813,
+          0.011193421669304371,
+          -0.014745263382792473,
+          0.00743666710332036,
+          0.012866885401308537,
+          -0.009682182222604752,
+          0.003942457959055901,
+          0.020371858030557632,
+          0.0028068935498595238,
+          -0.0077867284417152405,
+          0.005387527868151665,
+          0.01390853151679039,
+          0.020354781299829483,
+          0.006066305562853813,
+          -0.0005933965439908206,
+          0.019876649603247643,
+          -0.028175661340355873,
+          -0.025170257315039635,
+          0.01909114606678486,
+          -0.03551841154694557,
+          0.005472908727824688,
+          0.02790244296193123,
+          -0.04296361654996872,
+          -0.05020390823483467,
+          0.0027407235465943813,
+          -0.039719145745038986,
+          0.0004191663465462625,
+          -0.0023287611547857523,
+          0.01536854263395071,
+          -0.004057722166180611,
+          -0.017212767153978348,
+          0.0146428057923913,
+          -0.012397291138768196,
+          -0.0067322757095098495,
+          0.009835867211222649,
+          0.009016212075948715,
+          -0.024572592228651047,
+          -0.0016222350532189012,
+          -0.030890770256519318,
+          0.009169897064566612,
+          -0.017844585701823235,
+          -0.019142374396324158,
+          -0.025528857484459877,
+          -0.0005656477296724916,
+          -0.010638446547091007,
+          -0.001638244022615254,
+          0.051501695066690445,
+          0.021652569994330406,
+          0.020166942849755287,
+          -0.007726961746811867,
+          -0.023650480434298515,
+          0.05457540228962898,
+          0.0018132745753973722,
+          0.03814813867211342,
+          -0.012380215339362621,
+          -0.0002006448630709201,
+          -0.0008852921309880912,
+          0.022113624960184097,
+          -0.017101772129535675,
+          -0.0011547752656042576,
+          -0.011022660881280899,
+          -0.00586566049605608,
+          -0.0070780678652226925,
+          0.014045140706002712,
+          0.015974747017025948,
+          -0.002610517665743828,
+          0.014156135730445385,
+          0.01221799198538065,
+          0.022694215178489685,
+          0.03130060061812401,
+          0.016230888664722443,
+          -0.0034664601553231478,
+          -0.022199006751179695,
+          -0.012918113730847836,
+          -0.008200825192034245,
+          0.008999135345220566,
+          -0.008525271899998188,
+          -0.0031847036443650723,
+          -0.02631436102092266,
+          -0.020730456337332726,
+          -0.011312955059111118,
+          -0.01292665209621191,
+          -0.0031462821643799543,
+          0.02641681768000126,
+          -0.002469639526680112,
+          -0.03763585537672043,
+          -0.01511240005493164,
+          -0.03524519130587578,
+          -0.015607609413564205,
+          -0.0017876604106277227,
+          -0.01748598739504814,
+          0.016538260504603386,
+          -0.003780234605073929,
+          -0.007833687588572502,
+          -0.0008964983280748129,
+          -0.018169032409787178,
+          0.009690719656646252,
+          -0.011048275046050549,
+          -0.000543768925126642,
+          -0.0017503062263131142,
+          0.013345018029212952,
+          -0.008205094374716282,
+          0.008973521180450916,
+          -0.01412198320031166,
+          0.006868884898722172,
+          -0.048120614141225815,
+          -0.027731681242585182,
+          -0.027099864557385445,
+          0.022557605057954788,
+          -0.015214857645332813,
+          -0.023189423605799675,
+          0.008734455332159996,
+          -0.013686541467905045,
+          0.008939368650317192,
+          -0.012021616101264954,
+          0.008388662710785866,
+          0.019210679456591606,
+          0.009024749509990215,
+          -0.002614786848425865,
+          -0.013669464737176895,
+          -0.02390662208199501,
+          -0.02585330419242382,
+          0.023650480434298515,
+          -0.026485122740268707,
+          0.01071528997272253,
+          -0.012021616101264954,
+          0.008585038594901562,
+          -0.029917430132627487,
+          -0.020730456337332726,
+          0.0044910297729074955,
+          0.011347107589244843,
+          -0.021140284836292267,
+          0.044671230018138885,
+          0.04043634235858917,
+          0.029985735192894936,
+          -0.020252324640750885,
+          0.05139923840761185,
+          -0.022796671837568283,
+          0.007312864996492863,
+          0.036440521478652954,
+          0.030446790158748627,
+          0.026092370972037315,
+          -0.015581995248794556,
+          -0.012294834479689598,
+          -0.015496614389121532,
+          0.013404784724116325,
+          -0.024333525449037552,
+          0.002821835223585367,
+          -0.02042308636009693,
+          0.013643850572407246,
+          0.03451091796159744,
+          -0.004917933605611324,
+          0.029934506863355637,
+          -0.013720693998038769,
+          0.030361410230398178,
+          0.0017652479000389576,
+          -0.026587579399347305,
+          0.015667375177145004,
+          -0.014232978224754333,
+          0.004418455995619297,
+          0.017964119091629982,
+          -0.006087650544941425,
+          0.013575546443462372,
+          -0.061337560415267944,
+          0.028141509741544724,
+          0.01758844405412674,
+          0.01123611256480217,
+          -0.002180411946028471,
+          -0.024726277217268944,
+          -0.0021142419427633286,
+          0.009340658783912659,
+          -0.0016745307948440313,
+          0.011859391815960407,
+          0.01193623524159193,
+          -0.033418044447898865,
+          0.010578680783510208,
+          -0.013635313138365746,
+          0.024299373850226402,
+          0.006258412264287472,
+          0.0007476155878975987,
+          -0.03961668908596039,
+          -0.00560524920001626,
+          0.014830644242465496,
+          0.01095435582101345,
+          -0.005989462602883577,
+          -0.04624223709106445,
+          -0.030105268582701683,
+          0.025563009083271027,
+          0.005110040307044983,
+          -0.05167245492339134,
+          0.014557424932718277,
+          -0.015334390103816986,
+          0.018766699358820915,
+          -0.008708841167390347,
+          -0.025972837582230568,
+          -0.029985735192894936,
+          -0.017187153920531273,
+          0.016726097092032433,
+          0.048564594238996506,
+          0.004207138437777758,
+          0.013345018029212952,
+          -0.0048538981936872005,
+          0.006651164032518864,
+          -0.024760430678725243,
+          0.012030153535306454,
+          -0.027151092886924744,
+          -0.02971251681447029,
+          0.029012393206357956,
+          0.009981014765799046,
+          -0.011330030858516693,
+          0.019330210983753204,
+          -0.017084697261452675,
+          -0.0014141193823888898,
+          0.02240392006933689,
+          -0.0006163425859995186,
+          -0.007295788731426001,
+          -0.010382304899394512,
+          0.013618236407637596,
+          -0.0014354645973071456,
+          -0.01762259565293789,
+          -0.009571187198162079,
+          0.04241717979311943,
+          0.010092009790241718,
+          0.018835002556443214,
+          0.03261546418070793,
+          -0.0018933190731331706,
+          -0.02400907874107361,
+          -0.0027215126901865005,
+          -0.026638807728886604,
+          -0.017033468931913376,
+          -0.014548887498676777,
+          0.016674868762493134,
+          0.023530947044491768,
+          0.015249010175466537,
+          -0.027526767924427986,
+          -0.030019886791706085,
+          -0.01950097270309925,
+          -0.022199006751179695,
+          -0.0034493841230869293,
+          -0.0004213008505757898,
+          -0.026502197608351707,
+          0.007005494087934494,
+          0.0341181643307209,
+          0.03357172757387161,
+          -0.0010870042024180293,
+          -0.009016212075948715,
+          -0.010638446547091007,
+          0.0007721625734120607,
+          0.015231933444738388,
+          -0.015018481761217117,
+          0.0012326851719990373,
+          0.005028928630053997,
+          0.005383259151130915,
+          0.0029285612981766462,
+          -0.007095144130289555,
+          -0.05747834965586662,
+          0.003961668815463781,
+          0.02544347569346428,
+          0.005084426142275333,
+          -0.0008377990452572703,
+          0.02629728429019451,
+          0.0008276600856333971,
+          0.008123982697725296,
+          -0.03142013028264046,
+          0.011108040809631348,
+          0.01803242415189743,
+          0.015163629315793514,
+          -0.018151957541704178,
+          0.0122521435841918,
+          0.0064206356182694435,
+          0.004811207763850689,
+          -0.00030950538348406553,
+          0.010143238119781017,
+          0.028636718168854713,
+          0.019620506092905998,
+          -0.014736725017428398,
+          0.023821242153644562,
+          0.0006435577524825931,
+          -0.012072844430804253,
+          -0.024333525449037552,
+          -0.001040044822730124,
+          0.0029883277602493763,
+          -0.011176345869898796,
+          0.014702572487294674,
+          -0.0023949311580508947,
+          0.0007956422632560134,
+          0.04139260947704315,
+          0.0437491200864315,
+          0.013481627218425274,
+          -0.011295879259705544,
+          0.003304236801341176,
+          0.026502197608351707,
+          0.020132791250944138,
+          0.004862436093389988,
+          0.011458102613687515,
+          -0.03331558406352997,
+          -0.019620506092905998,
+          -0.002981924219056964,
+          0.0014290610561147332,
+          -0.0026788224931806326,
+          0.03876287862658501,
+          0.029524678364396095,
+          0.009682182222604752,
+          -0.005741858389228582,
+          -0.011364183388650417,
+          0.013643850572407246,
+          0.020201096311211586,
+          -0.023599252104759216,
+          -0.0038015798199921846,
+          0.0016745307948440313,
+          -0.017170077189803123,
+          0.02288205362856388,
+          -0.0070311082527041435,
+          -0.00928943045437336,
+          0.02645096927881241,
+          0.09186973422765732,
+          -0.00464471522718668,
+          0.012141148559749126,
+          -0.03948007896542549,
+          -0.021823331713676453,
+          -0.006975610740482807,
+          -0.0036671049892902374,
+          0.015402695164084435,
+          0.02739015780389309,
+          0.009135744534432888,
+          0.007133565377444029,
+          0.017878737300634384,
+          -0.004606293980032206,
+          0.012508286163210869,
+          -0.02826104313135147,
+          0.020166942849755287,
+          -0.005007583647966385,
+          -0.0038335975259542465,
+          -0.011389797553420067,
+          -0.005989462602883577,
+          0.011765473522245884,
+          0.005468640010803938,
+          0.017016392201185226,
+          0.0019893725402653217,
+          -0.0024760430678725243,
+          0.0014333301223814487,
+          0.003031018190085888,
+          -0.010877513326704502,
+          -0.009383349679410458,
+          -0.021396426483988762,
+          0.003827193984761834,
+          0.0009914844995364547,
+          -0.007884915918111801,
+          0.012755890376865864,
+          0.005502792075276375,
+          0.024043232202529907,
+          -0.028175661340355873,
+          -0.0030609015375375748,
+          0.00633525475859642,
+          -0.008956445381045341,
+          0.010390842333436012,
+          0.012380215339362621,
+          -0.008640536107122898,
+          0.022830823436379433,
+          -0.01466841995716095,
+          0.014762339182198048,
+          0.01170570682734251,
+          -0.021601341664791107,
+          -0.004155910108238459,
+          -0.009938324801623821,
+          -0.02192578837275505,
+          0.020252324640750885,
+          -0.018715469166636467,
+          0.005033197812736034,
+          0.00073320756200701,
+          0.006774966139346361,
+          -0.0010496501345187426,
+          -0.01740914396941662,
+          0.010348152369260788,
+          -0.008785683661699295,
+          0.006839001551270485,
+          -0.005058811977505684,
+          0.018254414200782776,
+          0.012166762724518776,
+          0.0028090281412005424,
+          -0.0011622461024671793,
+          0.014600115828216076,
+          -0.00372900627553463,
+          -0.018886230885982513,
+          -0.0036713741719722748,
+          0.01633334532380104,
+          0.01051037572324276,
+          0.0002606782363727689,
+          -0.010305461473762989,
+          -0.0146342683583498,
+          0.014139058999717236,
+          0.000977076473645866,
+          -0.0018122073961421847,
+          -0.0005285604856908321,
+          -0.03760170191526413,
+          0.002606248715892434,
+          0.013532855547964573,
+          0.012704662047326565,
+          -0.005818700883537531,
+          -0.017477449029684067,
+          0.007983104325830936,
+          -0.004190062638372183,
+          -0.0003204447857569903,
+          0.01661510206758976,
+          -0.019774191081523895,
+          0.01002370472997427,
+          -0.014779414981603622,
+          -0.03278622403740883,
+          -0.0029776552692055702,
+          -0.008380125276744366,
+          0.005822970066219568,
+          -0.005592442117631435,
+          -0.0071805245243012905,
+          -0.01414759736508131,
+          -0.005741858389228582,
+          -0.02979789674282074,
+          0.008947907015681267,
+          0.0317445807158947,
+          0.004473953507840633,
+          -0.028978241607546806,
+          -0.012158225290477276,
+          0.025580085813999176,
+          -0.0024931191001087427,
+          -0.006915844511240721,
+          0.003009672975167632,
+          0.0012572321575134993,
+          -0.005784548819065094,
+          -0.009656568057835102,
+          -0.012935190461575985,
+          0.013063261285424232,
+          0.01320840884000063,
+          0.011244650930166245,
+          -0.03357172757387161,
+          0.010664060711860657,
+          0.01813488081097603,
+          -0.02042308636009693,
+          0.011748396791517735,
+          0.028585489839315414,
+          0.0029435029719024897,
+          0.010399380698800087,
+          0.007334209978580475,
+          0.008286206051707268,
+          -0.017153002321720123,
+          0.016683407127857208,
+          0.009186973795294762,
+          -0.012559514492750168,
+          0.004563603550195694,
+          0.007381169591099024,
+          0.008542348630726337,
+          0.01960342936217785,
+          -0.011603250168263912,
+          0.013609698973596096,
+          -0.0068816919811069965,
+          0.03321312740445137,
+          -0.011919158510863781,
+          -0.014881872572004795,
+          0.03229101747274399,
+          0.027560919523239136,
+          -0.004563603550195694,
+          0.017246920615434647,
+          -0.01943266950547695,
+          -0.01267050951719284,
+          0.019296059384942055,
+          0.0005907283630222082,
+          0.019313136115670204,
+          -0.011304416693747044,
+          -0.02595576085150242,
+          0.029012393206357956,
+          -0.008106905966997147,
+          -0.001259366748854518,
+          -0.015829598531126976,
+          0.0043458822183310986,
+          -0.0005789885181002319,
+          0.037328481674194336,
+          0.012653433717787266,
+          0.0011985328746959567,
+          0.026570502668619156,
+          -0.00769280968233943,
+          -0.025067800655961037,
+          0.001990439835935831,
+          -0.015214857645332813,
+          0.013336479663848877,
+          -0.027970748022198677,
+          -0.00560524920001626,
+          0.014506196603178978,
+          0.003026749240234494,
+          -0.002644670195877552,
+          0.01666633039712906,
+          -0.007513510063290596,
+          0.011791087687015533,
+          -0.019244831055402756,
+          0.016717558726668358,
+          -0.011517869308590889,
+          0.004473953507840633,
+          -0.011910621076822281,
+          0.026092370972037315,
+          -0.025563009083271027,
+          -0.00999809056520462,
+          0.013660927303135395,
+          -0.008354511111974716,
+          0.0006376878009177744,
+          -0.02252345345914364,
+          -0.008089830167591572,
+          -0.020371858030557632,
+          0.01679440215229988,
+          -0.005682091694325209,
+          0.011902082711458206,
+          -0.00306303589604795,
+          0.0025528857950121164,
+          -0.008585038594901562,
+          0.010322538204491138,
+          -0.0009199780761264265,
+          -0.017417682334780693,
+          -0.005968117620795965,
+          -0.03025895357131958,
+          -0.030361410230398178,
+          0.03072001039981842,
+          -0.00880275946110487,
+          0.003923247568309307,
+          0.02728770114481449,
+          -0.02540932409465313,
+          -0.005485715810209513,
+          0.001918933354318142,
+          0.022728366777300835,
+          -0.030651705339550972,
+          0.007914799265563488,
+          -0.009391887113451958,
+          -0.016452878713607788,
+          0.007184793706983328,
+          -0.0008874266059137881,
+          0.015880826860666275,
+          -0.0025635582860559225,
+          -0.04924764111638069,
+          -0.0007081269286572933,
+          0.017135925590991974,
+          -0.013806073926389217,
+          0.008081291802227497,
+          0.02151595987379551,
+          -0.009630953893065453,
+          -0.008439891040325165,
+          -0.0011334300506860018,
+          0.025989914312958717,
+          -0.022625910118222237,
+          0.030019886791706085,
+          0.005767472553998232,
+          0.004713019821792841,
+          0.005340568721294403,
+          0.002962713595479727,
+          0.01730668731033802,
+          0.002080089645460248,
+          0.004258367232978344,
+          0.025238562375307083,
+          0.013148642145097256,
+          0.0037418133579194546,
+          -0.0032978332601487637,
+          -0.0030352871399372816,
+          0.016538260504603386,
+          -0.018442250788211823,
+          0.02100367471575737,
+          -0.0004714620590675622,
+          0.00013560871593654156,
+          0.016256503760814667,
+          -0.01484772004187107,
+          -0.027219397947192192,
+          0.03075416199862957,
+          -0.022335615009069443,
+          0.01757136732339859,
+          0.023770013824105263,
+          -0.01339624635875225,
+          -0.004922202788293362,
+          -0.007073798682540655,
+          0.001534719718620181,
+          0.018339794129133224,
+          -0.0134218605235219,
+          -0.023308956995606422,
+          -0.00030790449818596244,
+          0.0220111683011055,
+          0.02293328195810318,
+          0.028175661340355873,
+          -0.021140284836292267,
+          0.01412198320031166,
+          0.007466550450772047,
+          -0.008311820216476917,
+          0.016469955444335938,
+          0.008452698588371277,
+          -0.0025272713974118233,
+          0.021293969824910164,
+          0.0075903525575995445,
+          0.004576410632580519,
+          -0.008896678686141968,
+          -0.013242561370134354,
+          -0.01267050951719284,
+          -0.006749351508915424,
+          -0.006164493504911661,
+          -0.008264861069619656,
+          -0.01709323562681675,
+          0.09548988193273544,
+          -0.011867930181324482,
+          -0.0028410458471626043,
+          -0.009195511229336262,
+          0.0026041141245514154,
+          -0.006949996575713158,
+          0.0013479492627084255,
+          0.023086966946721077,
+          -0.022728366777300835,
+          -0.03671374171972275,
+          0.011167807504534721,
+          -0.004070529248565435,
+          0.006288295611739159,
+          0.03712356835603714,
+          0.00038314631092362106,
+          -0.016589488834142685,
+          -6.773765198886395e-05,
+          -0.003372541395947337,
+          0.01295226626098156,
+          0.0028197006322443485,
+          -0.0036158766597509384,
+          0.0109799699857831,
+          0.029200231656432152,
+          -0.017810434103012085,
+          -0.02144765481352806,
+          -0.012525362893939018,
+          0.0032615463715046644,
+          -0.002578499959781766,
+          0.003707661060616374,
+          0.0051996903494000435,
+          -0.014053679071366787,
+          -0.0134218605235219,
+          0.018817927688360214,
+          -0.016094280406832695,
+          0.0008025794522836804,
+          -0.012627819553017616,
+          -0.008222170174121857,
+          0.01753721572458744,
+          0.01757136732339859,
+          -0.03406693786382675,
+          -0.009921248070895672,
+          -0.03855796530842781,
+          0.018749622628092766,
+          0.01643580198287964,
+          0.014540349133312702,
+          -0.020884141325950623,
+          -0.012644895352423191,
+          -0.004520913120359182,
+          -0.02981497347354889,
+          -0.02243807166814804,
+          -0.026536351069808006,
+          0.009605339728295803,
+          0.00365856708958745,
+          0.027595072984695435,
+          0.005259457044303417,
+          -0.006262681446969509,
+          0.00801725685596466,
+          -0.01025423314422369,
+          0.00755620002746582,
+          -0.02146473154425621,
+          0.007107951212674379,
+          -0.0011409008875489235,
+          0.0012529630912467837,
+          0.005592442117631435,
+          -0.008853988721966743,
+          -0.01735791563987732,
+          -0.0079276068136096,
+          0.01466841995716095,
+          -0.015752756968140602,
+          -0.00821790099143982,
+          0.020798761397600174,
+          0.009383349679410458,
+          0.00931504461914301,
+          0.015616146847605705,
+          0.045661646872758865,
+          0.008273398503661156,
+          0.00783795677125454,
+          -0.011056812480092049,
+          0.009648029692471027,
+          -0.01813488081097603,
+          0.00745801255106926,
+          -0.019330210983753204,
+          -0.006292564328759909,
+          -0.019722962751984596,
+          0.04012897238135338,
+          -0.009366272948682308,
+          -0.01635042205452919,
+          0.033435117453336716,
+          -0.011808163486421108,
+          0.01910822093486786,
+          0.0035475720651447773,
+          -0.022830823436379433,
+          0.03352050110697746,
+          0.003876287955790758,
+          0.012918113730847836,
+          -0.025682542473077774,
+          0.0035859933122992516,
+          -0.00877714529633522,
+          0.006032153032720089,
+          -0.011586173437535763,
+          0.003494208911433816,
+          0.0023885276168584824,
+          -0.004841091111302376,
+          -0.008760069496929646,
+          0.018698394298553467,
+          0.006595666520297527,
+          0.0002183880569646135,
+          0.028056127950549126,
+          -0.000954130373429507,
+          0.018220260739326477,
+          6.110063259257004e-05,
+          0.006010808050632477,
+          0.027680452913045883,
+          0.002821835223585367,
+          0.006809118203818798,
+          -0.01714446395635605,
+          -0.0009199780761264265,
+          -0.009485806338489056,
+          0.009596801362931728,
+          0.02639974094927311,
+          -0.010612832382321358,
+          -0.031078608706593513,
+          0.022574681788682938,
+          -0.007048184517771006,
+          0.015966208651661873,
+          0.002999000484123826,
+          -0.014062216505408287,
+          -0.0023437028285115957,
+          -0.027680452913045883,
+          0.014454968273639679,
+          -0.017349377274513245,
+          -0.024265222251415253,
+          -0.007231753319501877,
+          -0.01045914739370346,
+          -0.004845359828323126,
+          0.038353051990270615,
+          0.012474133633077145,
+          -0.024572592228651047,
+          0.006817656569182873,
+          -0.003974475897848606,
+          -0.005989462602883577,
+          0.0057119750417768955,
+          0.005976655520498753,
+          0.004098278004676104,
+          0.010945817455649376,
+          -0.0065700518898665905,
+          -0.01320840884000063,
+          -0.03324728086590767,
+          -0.02240392006933689,
+          0.014941639266908169,
+          0.00028068936080671847,
+          -0.01315718051046133,
+          0.028483033180236816,
+          -0.01002370472997427,
+          -0.005575365852564573,
+          0.0071292961947619915,
+          0.00013887719251215458,
+          -0.0022796671837568283,
+          -0.017844585701823235,
+          0.002273263642564416,
+          0.006245605181902647,
+          -0.014950176700949669,
+          0.009844405576586723,
+          0.0032124524004757404,
+          -0.014540349133312702,
+          0.005635132547467947,
+          -0.0015027020126581192,
+          -0.003562513506039977,
+          -0.02481165900826454,
+          0.0059595792554318905,
+          0.00708233704790473,
+          -0.0018036692636087537,
+          0.002390662208199501,
+          -0.018169032409787178,
+          0.01801534742116928,
+          0.001955220242962241,
+          0.0071762558072805405,
+          -0.029166078194975853,
+          0.007282981649041176,
+          -0.006740813609212637,
+          0.0015080382581800222,
+          -0.0195692777633667,
+          0.009161358699202538,
+          0.020798761397600174,
+          0.012491210363805294,
+          -0.0005250918911769986,
+          0.010442070662975311,
+          0.008175211027264595,
+          -0.012704662047326565,
+          0.003950996324419975,
+          -0.02050846628844738,
+          -0.007308595813810825,
+          -0.010057857260107994,
+          0.011372721754014492,
+          -0.004219945520162582,
+          0.015983285382390022,
+          0.009511420503258705,
+          0.005600980017334223,
+          -0.009699258022010326,
+          0.00743666710332036,
+          0.01292665209621191,
+          0.005874198395758867,
+          -0.0011099503608420491,
+          0.017827508971095085,
+          0.07363239675760269,
+          0.005336299538612366,
+          0.013618236407637596,
+          6.870485958643258e-05,
+          -0.013071799650788307,
+          0.0015624684747308493,
+          -0.004939279053360224,
+          -0.01748598739504814,
+          0.026126522570848465,
+          0.026587579399347305,
+          -0.006382214371114969,
+          0.0004917400074191391,
+          0.009016212075948715,
+          0.005558289587497711,
+          0.005485715810209513,
+          0.022216081619262695,
+          -0.0077867284417152405,
+          0.01753721572458744,
+          0.024179840460419655,
+          0.005541213322430849,
+          -0.0019093280425295234,
+          -0.0022305732127279043,
+          0.009152821265161037,
+          0.004209273029118776,
+          -0.009050363674759865,
+          0.01150079257786274,
+          -0.01437812577933073,
+          0.01909114606678486,
+          0.011389797553420067,
+          0.02146473154425621,
+          0.007073798682540655,
+          -0.01704200729727745,
+          -0.018356870859861374,
+          0.0041281613521277905,
+          -0.0013479492627084255,
+          0.0059083509258925915,
+          0.0063224476762115955,
+          0.005477177910506725,
+          -0.011611787602305412,
+          -0.0041793896816670895,
+          0.010356690734624863,
+          -0.002471773885190487,
+          -0.0008500725380145013,
+          0.01757136732339859,
+          -0.01268758624792099,
+          0.01168009266257286,
+          -0.001839956152252853,
+          -0.0009984216885641217,
+          0.009682182222604752,
+          0.0014354645973071456,
+          -0.011825240217149258,
+          -0.004166582599282265,
+          0.0021003675647079945,
+          0.01217530108988285,
+          -0.013259637169539928,
+          -0.01048476155847311,
+          0.011825240217149258,
+          -0.006168762221932411,
+          0.009169897064566612,
+          -0.009579724632203579,
+          -0.0025315405800938606,
+          -0.00039648707024753094,
+          -0.008085560984909534,
+          -0.0014578771078959107,
+          0.007820880971848965,
+          0.0008324627415277064,
+          0.010518914088606834,
+          0.011560559272766113,
+          0.01150079257786274,
+          0.003560379147529602,
+          0.004002224653959274,
+          -0.002964847953990102,
+          -0.0064206356182694435,
+          -0.011432488448917866,
+          8.15119783510454e-05,
+          0.005776010919362307,
+          -0.02242099680006504,
+          -0.00024800453684292734,
+          0.008465506136417389,
+          0.005046004895120859,
+          -0.0007710952777415514,
+          0.020849989727139473,
+          0.001264702994376421,
+          0.007820880971848965,
+          -0.025221485644578934,
+          0.0011238247388973832,
+          0.006890229880809784,
+          0.007718423847109079,
+          0.012798581272363663,
+          -0.028636718168854713,
+          -0.0038528083823621273,
+          -0.0038634808734059334,
+          0.0171273872256279,
+          -0.00011606451880652457,
+          -0.0033106403425335884,
+          0.021686721593141556,
+          -0.0007833687704987824,
+          -0.003430173499509692,
+          -0.00755620002746582,
+          -0.005579635035246611,
+          0.008004449307918549,
+          0.012994957156479359,
+          0.024623820558190346,
+          -0.0019968433771282434,
+          -0.0055967108346521854,
+          -0.00809409935027361,
+          0.0244530588388443,
+          0.015249010175466537,
+          0.01863008923828602,
+          -0.0219087116420269,
+          0.010638446547091007,
+          -0.013865840621292591,
+          0.009067440405488014,
+          0.011714245192706585,
+          -0.010911665856838226,
+          0.0004290384822525084,
+          -0.010757979936897755,
+          -0.017135925590991974,
+          -0.022711291909217834,
+          0.00502039073035121,
+          0.02839765138924122,
+          -0.0002676154254004359,
+          0.007193332072347403,
+          -0.016358960419893265,
+          -0.0018442251021042466,
+          -0.011312955059111118,
+          -0.025750847533345222,
+          -0.016222350299358368,
+          -0.021772101521492004,
+          -0.02141350321471691,
+          -0.0038592119235545397,
+          -0.002131317974999547,
+          -0.01907406933605671,
+          0.00013153978215996176,
+          -0.0012817791430279613,
+          0.01902284100651741,
+          -0.0014888276346027851,
+          0.0015624684747308493,
+          0.012337524443864822,
+          0.012943727895617485,
+          -0.012824195437133312,
+          -0.004228483885526657,
+          -0.019261907786130905,
+          -0.024179840460419655,
+          0.0032487392891198397,
+          -0.007726961746811867,
+          0.013874378986656666,
+          -0.008234977722167969,
+          0.004260501824319363,
+          -0.017187153920531273,
+          0.014702572487294674,
+          -0.03579162806272507,
+          -0.024128612130880356,
+          0.0006531630642712116,
+          -0.0040299734100699425,
+          -0.009605339728295803,
+          -0.02585330419242382,
+          0.01198746357113123,
+          -0.019620506092905998,
+          0.002104636514559388,
+          -0.018442250788211823,
+          0.011398335918784142,
+          -0.015283161774277687,
+          0.018681317567825317,
+          -0.00025520852068439126,
+          0.03179580718278885,
+          0.004405648913234472,
+          0.007978835143148899,
+          -0.004525181837379932,
+          0.003327716374769807,
+          -0.005468640010803938,
+          0.006365138106048107,
+          0.006988417822867632,
+          0.008055677637457848,
+          0.007825150154531002,
+          -0.004768517334014177,
+          0.004061991348862648,
+          0.005110040307044983,
+          0.030515095219016075,
+          0.028021976351737976,
+          0.0027706066612154245,
+          0.0014674824196845293,
+          0.0034685947466641665,
+          0.004576410632580519,
+          -0.00038021133514121175,
+          0.008862526156008244,
+          -0.011287340894341469,
+          -0.005391797050833702,
+          0.01417321152985096,
+          -0.012448519468307495,
+          -0.0037695621140301228,
+          0.015377080999314785,
+          -0.015086785890161991,
+          -0.026092370972037315,
+          0.021823331713676453,
+          -0.006552976090461016,
+          -0.002450428670272231,
+          -0.006689585279673338,
+          0.008115444332361221,
+          -0.005946772173047066,
+          0.004194331355392933,
+          0.028107358142733574,
+          -0.005737589206546545,
+          -0.010945817455649376,
+          0.012636357918381691,
+          -0.002330895746126771,
+          -0.007948951795697212,
+          0.02291620522737503,
+          0.035894084721803665,
+          0.00999809056520462,
+          -0.026041142642498016,
+          -0.007381169591099024,
+          0.02144765481352806,
+          -0.00973341055214405,
+          -0.021755026653409004,
+          0.008273398503661156,
+          0.007522047962993383,
+          -0.005421680398285389,
+          0.018937459215521812,
+          0.004239156376570463,
+          -0.010792132467031479,
+          0.009041826240718365,
+          0.022711291909217834,
+          0.021686721593141556,
+          0.008947907015681267,
+          -0.023548023775219917,
+          0.01025423314422369,
+          -0.026228979229927063,
+          -0.013831689022481441,
+          -0.023821242153644562,
+          0.003581724362447858,
+          0.003857077332213521,
+          -0.003257277188822627,
+          0.026536351069808006,
+          0.011876468546688557,
+          0.011645940132439137,
+          0.0030950538348406553,
+          0.023052813485264778,
+          -0.015223395079374313,
+          -0.028107358142733574,
+          -0.0009733410552144051,
+          -0.015479537658393383,
+          -0.0018282162491232157,
+          0.002166537567973137,
+          0.007150641642510891,
+          0.005852853413671255,
+          0.01953512616455555,
+          0.000376209121895954,
+          0.0019584218971431255,
+          -2.4263486920972355e-05,
+          0.010100548155605793,
+          -0.015727141872048378,
+          -0.019705887883901596,
+          -0.008922292850911617,
+          6.470263178925961e-05,
+          0.011637401767075062,
+          -0.005208228249102831,
+          -3.448583447607234e-05,
+          0.007846495136618614,
+          -0.015897903591394424,
+          -0.0097590247169137,
+          -0.014412278309464455,
+          -0.0029306956566870213,
+          -0.005071619059890509,
+          0.03136890381574631,
+          0.014437892474234104,
+          -0.004747172351926565,
+          -0.011543483473360538,
+          -0.005741858389228582,
+          0.03370833769440651,
+          0.0017204229952767491,
+          0.012422905303537846,
+          0.0146342683583498,
+          0.015684451907873154,
+          0.027014482766389847,
+          0.012807118706405163,
+          0.003018211107701063,
+          0.002516598906368017,
+          0.02346264198422432,
+          -0.013951221480965614,
+          0.008730186149477959,
+          -0.011645940132439137,
+          -0.003521957667544484,
+          -0.010937280021607876,
+          0.015078248456120491,
+          -0.013003494590520859,
+          -0.011278802528977394,
+          0.023069890215992928,
+          -0.0004381102044135332,
+          0.013276712968945503,
+          -0.0004242358263581991,
+          -0.003953130915760994,
+          -0.02534101903438568,
+          0.0015464596217498183,
+          0.0006531630642712116,
+          -0.004401379730552435,
+          0.010578680783510208,
+          -0.027475539594888687,
+          0.0013660927070304751,
+          0.015240471810102463,
+          -0.007116489112377167,
+          0.00038981667603366077,
+          -0.010826284997165203,
+          0.005904081743210554,
+          0.00011606451880652457,
+          -0.02385539375245571,
+          0.015488076023757458,
+          -0.006698123179376125,
+          0.01243144366890192,
+          -0.0018548977095633745,
+          -0.013788998126983643,
+          0.008687496185302734,
+          0.019296059384942055,
+          0.027560919523239136,
+          -0.007825150154531002,
+          0.0005528406472876668,
+          -0.0001855698210420087,
+          -0.006027883850038052,
+          -0.02737308293581009,
+          0.02054261974990368,
+          -0.007863570936024189,
+          0.01953512616455555,
+          -0.009682182222604752,
+          -0.018186109140515327,
+          -0.014856258407235146,
+          0.011927696876227856,
+          0.007603159639984369,
+          0.0010304395109415054,
+          0.01489894837141037,
+          -0.024077383801341057,
+          0.020867066457867622,
+          0.0033383890986442566,
+          -0.0064718639478087425,
+          0.008303281851112843,
+          0.004230618476867676,
+          0.004695943556725979,
+          -0.008384393528103828,
+          0.007658657152205706,
+          0.007569007575511932,
+          -0.007253098301589489,
+          0.004326671827584505,
+          -0.04057295247912407,
+          0.010732365772128105,
+          -0.011090965010225773,
+          0.005878467578440905,
+          -0.004742903169244528,
+          -0.022745443508028984,
+          -0.035825781524181366,
+          0.004567872267216444,
+          0.019313136115670204,
+          -0.020235247910022736,
+          0.018408099189400673,
+          0.012124072760343552,
+          0.014574501663446426,
+          0.01859593763947487,
+          0.008853988721966743,
+          0.01461719162762165,
+          0.0438857264816761,
+          -0.024231068789958954,
+          6.483604374807328e-05,
+          0.008734455332159996,
+          0.011381260119378567,
+          -0.013404784724116325,
+          -0.010126162320375443,
+          0.005784548819065094,
+          -0.0015197781613096595,
+          0.04067540913820267,
+          -0.019210679456591606,
+          0.010886050760746002,
+          -0.010962894186377525,
+          0.008371586911380291,
+          -0.009562648832798004,
+          -0.0036158766597509384,
+          -0.012559514492750168,
+          -0.011893544346094131,
+          0.014745263382792473,
+          -0.019671734422445297,
+          -0.005460101645439863,
+          -0.007282981649041176,
+          0.01048476155847311,
+          -0.006096188444644213,
+          -0.020730456337332726,
+          0.018647165969014168,
+          -0.0018079383298754692,
+          0.029968658462166786,
+          -0.012755890376865864,
+          0.00574612757191062,
+          0.022591758519411087,
+          -0.010604294948279858,
+          0.018732545897364616,
+          -0.004362958483397961,
+          0.008853988721966743,
+          0.0055625587701797485,
+          -0.0065188235603272915,
+          -0.0014685495989397168,
+          0.01389145478606224,
+          -0.0027385889552533627,
+          0.029934506863355637,
+          -0.017400605604052544,
+          -0.009818791411817074,
+          0.009707796387374401,
+          0.0073769004084169865,
+          0.0042370217852294445,
+          0.003878422547131777,
+          0.009579724632203579,
+          -0.01071528997272253,
+          0.0009728074073791504,
+          -0.021686721593141556,
+          -0.015453923493623734,
+          -0.0011483717244118452,
+          -0.004683136474341154,
+          -0.0018730411538854241,
+          -0.002467504935339093,
+          -0.02044016122817993,
+          0.004627638962119818,
+          -0.0021579996682703495,
+          -8.451364556094632e-05,
+          -0.024487212300300598,
+          -0.00769280968233943,
+          -0.010057857260107994,
+          -0.005161269102245569,
+          -0.003139878623187542,
+          0.018271489068865776,
+          0.0034856710117310286,
+          -0.004862436093389988,
+          0.0293197650462389,
+          0.005404604133218527,
+          0.01536854263395071,
+          0.00280262460000813,
+          -0.05334591865539551,
+          -0.011014122515916824,
+          -0.004563603550195694,
+          -0.009912709705531597,
+          -0.007261636666953564,
+          -0.008456967771053314,
+          -0.010681137442588806,
+          -0.0019061262719333172,
+          0.0032914294861257076,
+          -0.004170851781964302,
+          0.0122521435841918,
+          -0.01700785383582115,
+          0.025494705885648727,
+          -0.004017166327685118,
+          0.015744218602776527,
+          -0.011722782626748085,
+          0.011654478497803211,
+          -0.006194376386702061,
+          -0.010851899161934853,
+          0.010544528253376484,
+          0.0146342683583498,
+          -0.0030544979963451624,
+          0.020371858030557632,
+          -0.014736725017428398,
+          0.003197510726749897,
+          -0.008303281851112843,
+          -0.02143057994544506,
+          -0.015991821885108948,
+          0.011295879259705544,
+          0.03130060061812401,
+          0.010066395625472069,
+          -0.0037524858489632607,
+          0.004072663839906454,
+          0.021669644862413406,
+          -0.004352285992354155,
+          0.0063181789591908455,
+          0.03025895357131958,
+          -0.0010982104577124119,
+          0.017742129042744637,
+          0.0001935742620844394,
+          -0.005028928630053997,
+          -0.002362913452088833,
+          0.01808365248143673,
+          -0.002766337711364031,
+          0.0057119750417768955,
+          0.02441890724003315,
+          0.010997045785188675,
+          -0.0064718639478087425,
+          0.008230708539485931,
+          0.014309820719063282,
+          0.013874378986656666,
+          -0.015351466834545135,
+          -0.0122009152546525,
+          -0.011534945107996464,
+          -0.015778370201587677,
+          0.007291519548743963,
+          -0.019261907786130905,
+          -0.004049184266477823,
+          0.007829418405890465,
+          -0.015138015151023865,
+          -0.012807118706405163,
+          -0.023548023775219917,
+          -0.015914980322122574,
+          0.01863008923828602,
+          0.005357644986361265,
+          -0.01465988252311945,
+          -0.004369362257421017,
+          0.0013714289525523782,
+          0.006181569304317236,
+          -0.0054985228925943375,
+          0.021840406581759453,
+          -0.004098278004676104,
+          -0.011125117540359497,
+          0.019330210983753204,
+          0.0017897948855534196,
+          0.005097233224660158,
+          -0.0064206356182694435,
+          0.004550796467810869,
+          -0.011364183388650417,
+          0.022096550092101097,
+          -0.003470729338005185,
+          0.012636357918381691,
+          -0.0037033918779343367,
+          0.004631908144801855,
+          -0.001978699816390872,
+          0.00821790099143982,
+          0.004017166327685118,
+          0.010629909113049507,
+          -0.005656477529555559,
+          0.004525181837379932,
+          0.012491210363805294,
+          -0.01641872711479664,
+          0.007265905383974314,
+          0.004559334367513657,
+          -4.365760105429217e-05,
+          0.005780279636383057,
+          -0.007645850069820881,
+          -0.005417411215603352,
+          -0.015488076023757458,
+          0.010100548155605793,
+          0.030873695388436317,
+          -0.0016649254830554128,
+          -0.007240291219204664,
+          0.01704200729727745,
+          0.015735680237412453,
+          -0.003816521493718028,
+          0.0061218030750751495,
+          0.009280892089009285,
+          -0.003148416755720973,
+          0.01370361726731062,
+          -0.024487212300300598,
+          -0.029524678364396095,
+          0.01763967238366604,
+          -0.020867066457867622,
+          0.022779595106840134,
+          -0.019210679456591606,
+          -0.027731681242585182,
+          -0.01847640424966812,
+          -0.0032530082389712334,
+          -0.019842496141791344,
+          -0.01611989364027977,
+          0.02532394416630268,
+          0.00685180863365531,
+          -0.006949996575713158,
+          -0.031112760305404663,
+          0.04050464928150177,
+          0.014489120803773403,
+          0.038318898528814316,
+          -0.011603250168263912,
+          -2.554753518779762e-05,
+          -0.007504971697926521,
+          0.012064306065440178,
+          -0.01146664097905159,
+          -0.014258592389523983,
+          0.007509240880608559,
+          0.028175661340355873,
+          0.015231933444738388,
+          -0.015223395079374313,
+          -0.007944682613015175,
+          0.015991821885108948,
+          -0.007043915335088968,
+          -0.0005480379913933575,
+          -0.008836911991238594,
+          -0.014830644242465496,
+          -0.010066395625472069,
+          -0.0005312286084517837,
+          0.012243606150150299,
+          0.016179660335183144,
+          -0.0007257367251440883,
+          0.01390853151679039,
+          -0.009955400601029396,
+          -0.014420815743505955,
+          -0.004602024797350168,
+          0.005327761638909578,
+          -0.02252345345914364,
+          0.02105490304529667,
+          0.002292474266141653,
+          -0.009537034668028355,
+          0.0030865157023072243,
+          -0.004572141449898481,
+          0.012602205388247967,
+          0.009391887113451958,
+          -0.0038421356584876776,
+          -0.0036137420684099197,
+          0.00572051340714097,
+          -0.001715086749754846,
+          0.015001405030488968,
+          -0.011714245192706585,
+          0.011842316016554832,
+          -0.007432397920638323,
+          -0.00019931078713852912,
+          -0.009648029692471027,
+          -0.005238111596554518,
+          -0.010493299923837185,
+          0.0028794673271477222,
+          0.012943727895617485,
+          -0.014548887498676777,
+          0.012807118706405163,
+          -0.0008132520597428083,
+          -0.00949434470385313,
+          -0.015138015151023865,
+          -0.0055155991576612,
+          -0.011364183388650417,
+          -0.016683407127857208,
+          0.007069529499858618,
+          0.0007855033036321402,
+          0.019347287714481354,
+          0.0030054040253162384,
+          -0.004572141449898481,
+          -0.00017262929759453982,
+          0.018903307616710663,
+          -0.02580207586288452,
+          0.03403278440237045,
+          -0.00010045584349427372,
+          0.022711291909217834,
+          0.04251963645219803,
+          0.00954557303339243,
+          -0.006663971114903688,
+          -0.013754845596849918,
+          0.005801625084131956,
+          -0.013609698973596096,
+          0.009981014765799046,
+          0.014335434883832932,
+          -0.03235932067036629,
+          -0.003944592550396919,
+          -0.013976835645735264,
+          0.011586173437535763,
+          -0.007957490161061287,
+          0.0032850259449332952,
+          -0.010587218217551708,
+          0.011765473522245884,
+          0.01679440215229988,
+          -0.009075978770852089,
+          -0.004311730153858662,
+          -0.027492616325616837,
+          0.00032524747075513005,
+          0.016674868762493134,
+          -0.010911665856838226,
+          -0.01684563048183918,
+          0.010945817455649376,
+          -0.007218946237117052,
+          -0.015453923493623734,
+          -0.006898768246173859,
+          -0.03278622403740883,
+          -0.00020504731219261885,
+          -0.01076651830226183,
+          -0.003927516285330057,
+          -0.0028794673271477222,
+          0.0011024795239791274,
+          -0.0033917520195245743,
+          -0.0067066615447402,
+          -0.0025891726836562157,
+          0.0067664277739822865,
+          0.005011852364987135,
+          0.002080089645460248,
+          0.0005987327895127237,
+          -0.00023506399884354323,
+          0.009255277924239635,
+          0.010168852284550667,
+          -0.007419590838253498,
+          -0.017212767153978348,
+          0.010834822431206703,
+          -0.020730456337332726,
+          0.005938234273344278,
+          0.00783795677125454,
+          0.011039736680686474,
+          -0.0220111683011055,
+          -0.010868974961340427,
+          0.009818791411817074,
+          0.003835732117295265,
+          0.01902284100651741,
+          0.010886050760746002,
+          0.0022455148864537477,
+          -0.0035646480973809958,
+          0.0028495839796960354,
+          -1.0947752343781758e-05,
+          -0.0012508286163210869,
+          0.015308775939047337,
+          0.0006504949415102601,
+          0.02974666841328144,
+          -0.012055767700076103,
+          0.011201960034668446,
+          -0.009340658783912659,
+          0.0013842361513525248,
+          -0.009741948917508125,
+          -0.001973363570868969,
+          -0.004194331355392933,
+          -0.006484671495854855,
+          0.013148642145097256,
+          0.007714154664427042,
+          -0.004239156376570463,
+          -0.01416467409580946,
+          0.01194477267563343,
+          -0.023035738617181778,
+          -0.012269220314919949,
+          -0.00881983619183302,
+          0.012935190461575985,
+          -0.0053960662335157394,
+          0.01170570682734251,
+          0.03457922115921974,
+          -0.0018559650052338839,
+          0.0007860369514673948,
+          -0.013746308162808418,
+          -0.008311820216476917,
+          -0.02728770114481449,
+          -0.01510386262089014,
+          -0.008922292850911617,
+          -0.03454506769776344,
+          -0.011919158510863781,
+          0.008239246904850006,
+          0.008111175149679184,
+          0.0014237246941775084,
+          -0.019176525995135307,
+          -0.008888140320777893,
+          -0.01492456253618002,
+          -0.017460372298955917,
+          -0.00924673955887556,
+          -0.01175693515688181,
+          -0.0037119300104677677,
+          0.020884141325950623,
+          -0.004162313882261515,
+          0.014267130754888058,
+          0.016939548775553703,
+          1.3132299500284716e-05,
+          0.015334390103816986,
+          0.009665105491876602,
+          -0.009127207100391388,
+          -0.004725826904177666,
+          0.006885961163789034,
+          0.0027706066612154245,
+          0.018749622628092766,
+          -0.006702392362058163,
+          -0.014796491712331772,
+          0.004298923071473837,
+          0.005571096669882536,
+          -0.012346062809228897,
+          0.010655523277819157,
+          -0.00476424815133214,
+          0.02054261974990368,
+          -0.018254414200782776,
+          -0.011885006912052631,
+          0.014472044073045254,
+          -0.008862526156008244,
+          0.02394077368080616,
+          0.013951221480965614,
+          -0.015291700139641762,
+          0.007167717441916466,
+          0.013029108755290508,
+          0.007615966722369194,
+          0.0018495614640414715,
+          0.022147778421640396,
+          -0.0011846584966406226,
+          0.0031590894795954227,
+          0.010237157344818115,
+          -0.018835002556443214,
+          -0.02042308636009693,
+          -0.003118533408269286,
+          0.02247222512960434,
+          0.007961759343743324,
+          -0.016222350299358368,
+          0.0037930416874587536,
+          0.012849809601902962,
+          0.012072844430804253,
+          0.0039787450805306435,
+          -0.001204936415888369,
+          -0.02248930186033249,
+          0.0035155541263520718,
+          0.0001910395221784711,
+          0.018664240837097168,
+          -0.011116579174995422,
+          0.001461078878492117,
+          -0.011389797553420067,
+          0.012747352942824364,
+          -0.005221035331487656,
+          0.01193623524159193,
+          0.0011761204805225134,
+          -0.005353375803679228,
+          0.0001780989987310022,
+          0.014941639266908169,
+          0.014070754870772362,
+          0.037738312035799026,
+          -0.005916888825595379,
+          -0.006480402313172817,
+          0.017170077189803123,
+          -0.012388752773404121,
+          0.018715469166636467,
+          0.006497478578239679,
+          0.0170249305665493,
+          -0.011090965010225773,
+          0.0011344973463565111,
+          0.04115354269742966,
+          0.015949131920933723,
+          -6.583659705938771e-05,
+          -0.008760069496929646,
+          -0.00805140845477581,
+          0.0244530588388443,
+          0.009067440405488014,
+          -2.7882166250492446e-05,
+          0.015744218602776527,
+          0.006096188444644213,
+          -0.016324806958436966,
+          -0.012918113730847836,
+          0.03555256128311157,
+          -0.007718423847109079,
+          -0.022045321762561798,
+          -0.0009311842732131481,
+          -0.009383349679410458,
+          -0.02884163148701191,
+          0.013959759846329689,
+          0.013788998126983643,
+          -0.008162403479218483,
+          0.006992687005549669,
+          0.008439891040325165,
+          -0.004117488861083984,
+          -0.004435532260686159,
+          0.0005346972029656172,
+          -0.0007049251580610871,
+          0.006177300587296486,
+          0.005092964507639408,
+          -0.014463506639003754,
+          -0.006787773221731186,
+          -0.005669284611940384,
+          -0.004064125940203667,
+          0.009016212075948715,
+          0.005058811977505684,
+          0.004870974458754063,
+          -0.008482581935822964,
+          0.018339794129133224,
+          0.02298451028764248,
+          -0.0122009152546525,
+          0.012115534394979477,
+          0.005233842879533768,
+          -0.0146428057923913,
+          -0.004614831879734993,
+          -0.01121903583407402,
+          -0.026570502668619156,
+          -0.006113264709711075,
+          0.0033490615896880627,
+          0.015428309328854084,
+          -0.022796671837568283,
+          0.0008500725380145013,
+          0.007790997624397278,
+          2.7365211280994117e-05,
+          -0.008879602886736393,
+          0.007573276292532682,
+          -0.00044157879892736673,
+          0.022301463410258293,
+          0.019808344542980194,
+          0.025631314143538475,
+          -0.008414276875555515,
+          -0.00499904528260231,
+          0.014301283285021782,
+          0.00433947890996933,
+          0.009374811314046383,
+          0.027543844655156136,
+          0.022164853289723396,
+          -0.013857303187251091,
+          0.006429173983633518,
+          0.000640355923678726,
+          -0.014250054024159908,
+          0.0066682398319244385,
+          0.013054722920060158,
+          0.012414366938173771,
+          0.0028410458471626043,
+          -0.0013415457215160131,
+          -0.015957670286297798,
+          0.004824014846235514,
+          0.018408099189400673,
+          -0.017118848860263824,
+          -0.03471583127975464,
+          -0.011842316016554832,
+          -0.012158225290477276,
+          -0.00624133599922061,
+          -0.012875423766672611,
+          0.01440373994410038,
+          0.004418455995619297,
+          0.019705887883901596,
+          0.0002309283590875566,
+          -0.01022861897945404,
+          0.014250054024159908,
+          0.005110040307044983,
+          -0.0016638581873849034,
+          0.0005352308508008718,
+          0.007231753319501877,
+          -0.0030331527814269066,
+          -0.0022476494777947664,
+          -0.00027055040118284523,
+          0.016239427030086517,
+          -0.02532394416630268,
+          0.008947907015681267,
+          -0.017067620530724525,
+          0.0017951311310753226,
+          0.015249010175466537,
+          -0.003775965655222535,
+          -0.016025975346565247,
+          0.0011216901475563645,
+          -0.008619191125035286,
+          -0.07288104295730591,
+          -0.005882736761122942,
+          0.004512374754995108,
+          0.023992003872990608,
+          -0.013336479663848877,
+          -0.0024824466090649366,
+          0.0018474268727004528,
+          0.008657612837851048,
+          -0.010134699754416943,
+          0.003995821345597506,
+          0.010390842333436012,
+          -0.003319178242236376,
+          -0.009221125394105911,
+          0.002508060773834586,
+          -0.008106905966997147,
+          -0.0038805571384727955,
+          -0.00783795677125454,
+          0.003402424743399024,
+          0.025221485644578934,
+          -0.010868974961340427,
+          0.024657974019646645,
+          0.009699258022010326,
+          -0.0075433929450809956,
+          -0.0027428579051047564,
+          -0.007761114276945591,
+          -0.027543844655156136,
+          -0.012397291138768196,
+          -0.003396020969375968,
+          -0.01292665209621191,
+          0.0029264267068356276,
+          0.001692674239166081,
+          -0.007167717441916466,
+          -0.004348016809672117,
+          0.0021547977812588215,
+          0.00030336863710545003,
+          -0.022147778421640396,
+          -0.0009407896432094276,
+          -0.014267130754888058,
+          0.02105490304529667,
+          -0.012371676973998547,
+          -0.004461146425455809,
+          0.0030544979963451624,
+          -0.010109085589647293,
+          0.004580679349601269,
+          0.019313136115670204,
+          -0.007308595813810825,
+          -0.009682182222604752,
+          0.014045140706002712,
+          -0.003690584795549512,
+          0.00185916677583009,
+          -0.01798119582235813,
+          0.010032243095338345,
+          -0.03232516720890999,
+          -0.003583858720958233,
+          -0.009981014765799046,
+          -0.0031847036443650723,
+          -0.0067151994444429874,
+          -0.03162504732608795,
+          0.015138015151023865,
+          -0.007039646618068218,
+          -0.02346264198422432,
+          -0.0011793222511187196,
+          -0.01859593763947487,
+          0.01419028826057911,
+          -0.01487333420664072,
+          0.01393414568156004,
+          -0.020645076408982277,
+          0.012533900327980518,
+          0.012798581272363663,
+          0.021123208105564117,
+          0.006493209395557642,
+          -0.005302147474139929,
+          0.002676687901839614,
+          0.0018730411538854241,
+          0.014352511614561081,
+          -0.008905217051506042,
+          0.00462336977943778,
+          -0.002390662208199501,
+          0.005737589206546545,
+          0.009426039643585682,
+          0.02107197977602482,
+          -0.0005896611255593598,
+          -0.0037823691964149475,
+          -0.017468910664319992,
+          0.00757754547521472,
+          -0.01559907104820013,
+          0.005485715810209513,
+          0.0025251370389014482,
+          0.023787088692188263,
+          0.010886050760746002,
+          0.0035859933122992516,
+          0.004900857340544462,
+          0.004520913120359182,
+          0.01861301250755787,
+          -0.006890229880809784,
+          -0.004747172351926565,
+          -0.018186109140515327,
+          0.019688811153173447,
+          0.007705616764724255,
+          -0.005306416191160679,
+          -0.011902082711458206,
+          0.006010808050632477,
+          0.002134519862011075,
+          0.0017577770631760359,
+          0.02288205362856388,
+          -0.02974666841328144,
+          -0.008964983746409416,
+          -0.007808073423802853,
+          -0.009682182222604752,
+          -0.017844585701823235,
+          0.017084697261452675,
+          -0.008012987673282623,
+          0.00461910106241703,
+          -0.012790042906999588,
+          0.033418044447898865,
+          0.002969117136672139,
+          0.00513992365449667,
+          0.013729231432080269,
+          -0.00024373548512812704,
+          0.017844585701823235,
+          -0.008367317728698254,
+          -0.04036803916096687,
+          -0.0031911071855574846,
+          0.0034515184815973043,
+          0.007645850069820881,
+          -0.010903127491474152,
+          0.004862436093389988,
+          -0.006339523941278458,
+          -0.01854470930993557,
+          -0.014232978224754333,
+          0.037840768694877625,
+          -0.005225304514169693,
+          0.017947042360901833,
+          -0.016145508736371994,
+          -0.0033682722132653,
+          0.007107951212674379,
+          0.022762520238757133,
+          -0.012602205388247967,
+          0.00791053008288145,
+          0.009571187198162079,
+          0.02199409157037735,
+          0.00978463888168335,
+          -0.01565883681178093,
+          0.023735860362648964,
+          0.005741858389228582,
+          -0.0023565099108964205,
+          0.005276532843708992,
+          -0.0030352871399372816,
+          0.016000360250473022,
+          -0.014813567511737347,
+          -0.011876468546688557,
+          0.002303146990016103,
+          -0.010168852284550667,
+          0.0037055264692753553,
+          -0.024589668959379196,
+          0.00037540867924690247,
+          -0.009844405576586723,
+          0.021089056506752968,
+          0.01027130987495184,
+          0.00877714529633522,
+          -0.019637582823634148,
+          -0.0048026698641479015,
+          -0.02645096927881241,
+          -0.014574501663446426,
+          -0.01022861897945404,
+          -0.0011451698374003172,
+          -0.004431263078004122,
+          -0.007765382993966341,
+          -0.03133475035429001,
+          -0.004202869720757008,
+          0.013541393913328648,
+          0.014975790865719318,
+          -0.0013714289525523782,
+          0.012089920230209827,
+          -0.00019544197130016983,
+          -0.008106905966997147,
+          -0.02236976847052574,
+          -0.010407919064164162,
+          -0.02091829478740692,
+          -0.03123229369521141,
+          -0.022199006751179695,
+          0.008670419454574585,
+          -0.012021616101264954,
+          -0.002941368380561471,
+          0.028961164876818657,
+          -0.022130701690912247,
+          0.007513510063290596,
+          0.005400335416197777,
+          0.0028666602447628975,
+          0.014856258407235146,
+          0.0054472945630550385,
+          0.0003634020104072988,
+          0.0037652929313480854,
+          -0.008341703563928604,
+          -0.008081291802227497,
+          0.01799827069044113,
+          -0.003331985557451844,
+          0.005097233224660158,
+          0.010809208266437054,
+          -0.006749351508915424,
+          -0.005438756663352251,
+          0.02392369881272316,
+          -0.015718603506684303,
+          0.002044870052486658,
+          -0.008610652759671211,
+          -0.014079293236136436,
+          -0.00949434470385313,
+          -0.005793086718767881,
+          -0.01813488081097603,
+          -0.015240471810102463,
+          0.01689685881137848,
+          -0.0004028906114399433,
+          0.016025975346565247,
+          -0.005289340391755104,
+          -0.005007583647966385,
+          0.014437892474234104,
+          -0.03558671474456787,
+          -0.003402424743399024,
+          0.006377945188432932,
+          -0.026211904361844063,
+          0.02484581060707569,
+          0.005276532843708992,
+          0.01605158858001232,
+          -0.013780459761619568,
+          0.0026318628806620836,
+          -0.006365138106048107,
+          0.004358689300715923,
+          0.018903307616710663,
+          0.007308595813810825,
+          0.004858167376369238,
+          -0.006689585279673338,
+          -0.008047140203416348,
+          0.016077203676104546,
+          -0.007415322121232748,
+          0.0018719738582149148,
+          0.016785863786935806,
+          0.018425175920128822,
+          0.012004539370536804,
+          0.0059553105384111404,
+          0.013071799650788307,
+          0.01750306226313114,
+          -0.007043915335088968,
+          -0.005835777148604393,
+          -0.013481627218425274,
+          -0.00802579429000616,
+          -0.0013105951948091388,
+          0.03859211876988411,
+          0.018920384347438812,
+          0.007385438773781061,
+          -0.02054261974990368,
+          0.013456013053655624,
+          0.0043928418308496475,
+          6.841302820248529e-06,
+          -0.0027385889552533627,
+          0.02293328195810318,
+          0.0024952536914497614,
+          0.007125027012079954,
+          0.006258412264287472,
+          -0.017212767153978348,
+          0.03309359401464462,
+          -0.020713381469249725,
+          0.0008532743086107075,
+          -0.013319403864443302,
+          -0.012166762724518776,
+          0.015530765987932682,
+          0.002104636514559388,
+          -0.004164448007941246,
+          -0.006296833511441946,
+          -0.0033575997222214937,
+          0.013515779748558998,
+          0.008499657735228539,
+          -0.012875423766672611,
+          0.004516643937677145,
+          -0.02542640082538128,
+          0.004315999336540699,
+          -0.015334390103816986,
+          0.021755026653409004,
+          -0.004559334367513657,
+          -0.004674598574638367,
+          -0.019261907786130905,
+          -0.017067620530724525,
+          -0.00999809056520462,
+          0.002403469290584326,
+          -0.004755710251629353,
+          0.004422725178301334,
+          -0.009468729607760906,
+          -0.008499657735228539,
+          0.0027599341701716185,
+          0.0036628360394388437,
+          0.012465596199035645,
+          0.013669464737176895,
+          0.019893724471330643,
+          0.017759205773472786,
+          -0.01536000519990921,
+          -0.0341864712536335,
+          0.0006056699785403907,
+          0.005084426142275333,
+          -0.0036436254158616066,
+          0.018664240837097168,
+          -0.001938143977895379,
+          -0.0009968207450583577,
+          -0.0007150641758926213,
+          0.0005419012159109116,
+          0.02004740945994854,
+          -0.011014122515916824,
+          -0.016657792031764984,
+          0.00021545309573411942,
+          0.008853988721966743,
+          0.00502039073035121,
+          -0.0025934416335076094,
+          -0.010296924039721489,
+          -0.0009455922991037369,
+          0.0020341973286122084,
+          0.006032153032720089,
+          -0.009101592935621738,
+          -0.0010496501345187426,
+          -0.0013671600027009845,
+          -0.0030544979963451624,
+          -0.006254143081605434,
+          -0.009844405576586723,
+          0.008222170174121857,
+          -0.012602205388247967,
+          -0.01123611256480217,
+          0.016222350299358368,
+          -0.015086785890161991,
+          0.001136631821282208,
+          0.023821242153644562,
+          0.022557605057954788,
+          -0.0008842248353175819,
+          0.006774966139346361,
+          -0.007936145178973675,
+          -0.011517869308590889,
+          -0.02785121463239193,
+          -0.019637582823634148,
+          -0.00806848518550396,
+          0.017947042360901833,
+          -0.002053408185020089,
+          -0.004049184266477823,
+          -0.0018869155319407582,
+          -0.008580769412219524,
+          0.008444160223007202,
+          -0.013814612291753292,
+          0.011620325967669487,
+          -0.034425534307956696,
+          0.013088875450193882,
+          -0.006702392362058163,
+          0.0010485828388482332,
+          0.015009943395853043,
+          0.04351005330681801,
+          -0.007351286243647337,
+          0.0029541754629462957,
+          0.001540056080557406,
+          0.01389145478606224,
+          0.028209814801812172,
+          0.004369362257421017,
+          0.0006136744632385671,
+          -0.01765674725174904,
+          0.004091874696314335,
+          0.002264725510030985,
+          -0.013456013053655624,
+          0.009579724632203579,
+          -0.023035738617181778,
+          -0.023155272006988525,
+          -0.009067440405488014,
+          -0.021123208105564117,
+          -0.002608383307233453,
+          0.002011785050854087,
+          0.012883962132036686,
+          -0.009878558106720448,
+          -0.012098458595573902,
+          -0.0005656477296724916,
+          -0.032991137355566025,
+          0.016709022223949432,
+          0.09665105491876602,
+          0.00875153113156557,
+          -0.003908305894583464,
+          -0.0021515958942472935,
+          0.002499522641301155,
+          0.009562648832798004,
+          -0.004768517334014177,
+          0.055360905826091766,
+          -0.018852079287171364,
+          0.008311820216476917,
+          -0.0008105839369818568,
+          0.008064216002821922,
+          -0.007223214954137802,
+          0.015530765987932682,
+          -0.012209453620016575,
+          -0.005951041355729103,
+          -0.009460192173719406,
+          -0.01123611256480217,
+          -0.0017246920615434647,
+          -0.016640717163681984,
+          -0.004986238200217485,
+          -0.020645076408982277,
+          0.013336479663848877,
+          -0.03234224393963814,
+          -0.011509330943226814,
+          -0.012474133633077145,
+          -0.0063224476762115955,
+          0.006066305562853813,
+          -0.015684451907873154,
+          -0.002595576224848628,
+          -0.002099300269037485,
+          0.015402695164084435,
+          0.0013863706262782216,
+          -0.00232235761359334,
+          -0.008328896015882492,
+          0.0005405671545304358,
+          0.005024659913033247,
+          0.003940323833376169,
+          0.006557244807481766,
+          0.006792041938751936,
+          -0.005848584230989218,
+          -0.008811297826468945,
+          0.010655523277819157,
+          -0.002510195365175605,
+          -0.0027919518761336803,
+          -0.008871064521372318,
+          0.010382304899394512,
+          -0.013481627218425274,
+          0.018356870859861374,
+          -0.013840226456522942,
+          -0.029507601633667946,
+          -0.015547842718660831,
+          0.02100367471575737,
+          -0.01996202953159809,
+          -0.0039381892420351505,
+          0.010553065687417984,
+          0.02829519473016262,
+          0.004113219678401947,
+          -0.001529383473098278,
+          -0.004909395705908537,
+          -0.017204230651259422,
+          0.009511420503258705,
+          0.008196556009352207,
+          0.018442250788211823,
+          -0.012371676973998547,
+          0.016487032175064087,
+          -0.018288565799593925,
+          0.006177300587296486,
+          -0.01663217879831791,
+          0.0006083381595090032,
+          0.00011866596469189972,
+          -0.0041793896816670895,
+          0.01608574204146862,
+          -0.01534292846918106,
+          -0.026263132691383362,
+          0.01900576427578926,
+          -0.013635313138365746,
+          -0.03307652100920677,
+          0.003374675754457712,
+          -0.034937821328639984,
+          -0.0025827689096331596,
+          0.009716334752738476,
+          -0.02149888314306736,
+          -0.018305642530322075,
+          0.007487895432859659,
+          -0.010066395625472069,
+          -0.008871064521372318,
+          0.00502039073035121,
+          -0.01689685881137848,
+          -0.012149686925113201,
+          -0.001152640674263239,
+          0.006753620691597462,
+          -0.00499477656558156,
+          0.0057632033713161945,
+          -0.026075294241309166,
+          0.014010988175868988,
+          -0.0170249305665493,
+          0.01144102681428194,
+          -0.010587218217551708,
+          0.022762520238757133,
+          0.011748396791517735,
+          -0.010356690734624863,
+          -0.0008303282083943486,
+          0.006036422215402126,
+          0.0005264259525574744,
+          0.026126522570848465,
+          0.03237639740109444,
+          -0.0011889275629073381,
+          0.008469774387776852,
+          0.007586083374917507,
+          0.003921112976968288,
+          -0.011014122515916824,
+          0.0015923518221825361,
+          0.014830644242465496,
+          0.01992787793278694,
+          -0.013729231432080269,
+          0.008879602886736393,
+          -0.026672959327697754,
+          0.02103782817721367,
+          -0.022096550092101097,
+          -0.000982412719167769,
+          0.010066395625472069,
+          -0.004777055233716965,
+          0.007090874947607517,
+          0.0066682398319244385,
+          0.021362274885177612,
+          0.03375956416130066,
+          0.00807702261954546,
+          0.008879602886736393,
+          0.007257367484271526,
+          -0.004798400681465864,
+          0.002044870052486658,
+          0.009135744534432888,
+          -0.005869929678738117,
+          -0.0040940092876553535,
+          -0.010083471424877644,
+          -0.006685316096991301,
+          0.016205275431275368,
+          0.0008079157560132444,
+          0.012602205388247967,
+          0.007018301170319319,
+          -0.007428129203617573,
+          -0.005669284611940384,
+          0.007014031987637281,
+          -0.023787088692188263,
+          -0.00950288213789463,
+          0.025221485644578934,
+          -0.013814612291753292,
+          -0.043202683329582214,
+          -0.0024333526380360126,
+          0.008853988721966743,
+          -0.017964119091629982,
+          -0.01245705783367157,
+          -0.004772786516696215,
+          0.010117623955011368,
+          0.0007241358398459852,
+          -0.007569007575511932,
+          0.028175661340355873,
+          0.00502039073035121,
+          -0.005169807001948357,
+          0.003571051638573408,
+          -2.8232359909452498e-05,
+          0.019244831055402756,
+          0.01536000519990921,
+          0.008294744417071342,
+          -0.03266669064760208,
+          -0.007304327096790075,
+          -0.005780279636383057,
+          0.013114489614963531,
+          0.002813297091051936,
+          0.003423769725486636,
+          -0.019740039482712746,
+          -0.020798761397600174,
+          0.019210679456591606,
+          0.004935009870678186,
+          0.019825419411063194,
+          -0.00867895781993866,
+          -0.006548706907778978,
+          -0.004841091111302376,
+          -0.004742903169244528,
+          0.00499477656558156,
+          0.0076415808871388435,
+          0.0038933642208576202,
+          0.03377664089202881,
+          -0.001280711847357452,
+          0.004042780492454767,
+          -0.0036350872833281755,
+          -0.0002441357064526528,
+          0.014787953346967697,
+          -0.00790199264883995,
+          0.008905217051506042,
+          0.018800850957632065,
+          -0.02887578494846821,
+          0.011398335918784142,
+          -0.01679440215229988,
+          -0.00828193686902523,
+          0.016000360250473022,
+          -0.0005320290802046657,
+          -0.004922202788293362,
+          -0.014583039097487926,
+          0.0008127184119075537,
+          0.007752575911581516,
+          0.01996202953159809,
+          -0.004091874696314335,
+          0.011287340894341469,
+          -0.009648029692471027,
+          -0.004305326379835606,
+          0.010083471424877644,
+          0.006087650544941425,
+          -0.0033213128335773945,
+          0.006232798099517822
+        ],
+        "keyphrases": [
+          "MoE models",
+          "expert choice",
+          "training efficiency",
+          "fine-tuning on GLUE and SuperGLUE",
+          "capacity factor"
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "d08eab82-ce8c-4126-8aee-d1681de9b6e5",
+      "properties": {
+        "page_content": "5 Conclusion\n\nWe propose a new routing method for sparsely activated mixture-of-experts (MoE) models. This method addresses load imbalance and under-utilization of experts in conventional MoE methods, and enables selecting different numbers of experts for each token. Our model demonstrates more than 2x training efficiency improvements when compared to the state-of-the-art GShard and Switch Transformer models, and also achieves strong gains when finetuning on 11 datasets in the GLUE and SuperGLUE benchmark.\n\n",
+        "title": "5 Conclusion",
+        "embedding": [
+          -0.014961956068873405,
+          -0.008189860731363297,
+          -0.023845616728067398,
+          0.002967504784464836,
+          -0.00690029701218009,
+          0.024916483089327812,
+          -0.0020832864101976156,
+          0.07137094438076019,
+          -0.014486853033304214,
+          0.04606231302022934,
+          -0.016666291281580925,
+          0.003244647989049554,
+          0.0073112985119223595,
+          -0.015520012937486172,
+          -0.04539867490530014,
+          0.030662959441542625,
+          0.012692022137343884,
+          0.0035896627232432365,
+          0.013717640191316605,
+          -0.03864166513085365,
+          -0.022352438420057297,
+          -0.008853496052324772,
+          -0.018159471452236176,
+          0.0027733161114156246,
+          0.019019179046154022,
+          -0.0032974369823932648,
+          0.000584922730922699,
+          0.008634798228740692,
+          -0.03864166513085365,
+          0.05327180400490761,
+          0.011213925667107105,
+          0.009728288277983665,
+          -0.0011019736994057894,
+          -0.001177386729978025,
+          -0.03257845342159271,
+          -0.007563932333141565,
+          0.07293953746557236,
+          0.025715861469507217,
+          -0.05375444516539574,
+          0.029727837070822716,
+          0.03912430629134178,
+          0.042442481964826584,
+          -0.03233712911605835,
+          -0.008084282279014587,
+          -0.014434064738452435,
+          -0.0042646098881959915,
+          -0.0013913713628426194,
+          -0.005045135505497456,
+          0.02479582279920578,
+          -0.01170411054044962,
+          0.0019154923502355814,
+          -0.010957521386444569,
+          0.014728175476193428,
+          0.0002771430881693959,
+          -0.04500652849674225,
+          0.03791769966483116,
+          0.018596867099404335,
+          0.017103686928749084,
+          0.03366440162062645,
+          0.007273592054843903,
+          -0.00380270485766232,
+          -0.01829521358013153,
+          -0.018657196313142776,
+          0.0008644224726594985,
+          0.003012752626091242,
+          0.00490185059607029,
+          -0.0039478749968111515,
+          -0.011915267445147038,
+          0.037223897874355316,
+          -0.010565373115241528,
+          -0.06045112758874893,
+          0.03665075823664665,
+          -0.04114537686109543,
+          0.0005467448499985039,
+          -0.0135818962007761,
+          -0.0067796362563967705,
+          0.0271637924015522,
+          0.014562266878783703,
+          -0.00047156744403764606,
+          0.001911721657961607,
+          0.012073635123670101,
+          0.00981124211102724,
+          0.0072698211297392845,
+          -0.007443271577358246,
+          0.05547386407852173,
+          -0.044704876840114594,
+          0.006198955699801445,
+          -0.05541353300213814,
+          0.03659042716026306,
+          -0.0024546959903091192,
+          -0.03007473796606064,
+          -0.021115664392709732,
+          -0.021779298782348633,
+          0.008619715459644794,
+          -0.0393354631960392,
+          0.033362749963998795,
+          0.006523231975734234,
+          -0.013197289779782295,
+          -0.008989240042865276,
+          0.021145828068256378,
+          -0.004758565686643124,
+          0.015218360349535942,
+          -0.015444600023329258,
+          0.008808248676359653,
+          0.05411642789840698,
+          -0.033151593059301376,
+          -0.019366079941391945,
+          0.044644545763731,
+          0.01785781793296337,
+          0.05526270717382431,
+          0.03119085170328617,
+          0.0022322272416204214,
+          0.01181722991168499,
+          0.01852145418524742,
+          -0.003199400147423148,
+          -0.028581559658050537,
+          -0.04585115611553192,
+          -0.05535320192575455,
+          0.023498717695474625,
+          -0.028551394119858742,
+          0.022548511624336243,
+          0.020633019506931305,
+          -0.009117442183196545,
+          -0.023845616728067398,
+          -0.03906397894024849,
+          0.01708860509097576,
+          0.0047095469199121,
+          -0.005278916098177433,
+          -0.012586443684995174,
+          -0.016877448186278343,
+          0.00349162588827312,
+          -0.01137983426451683,
+          -0.02776709757745266,
+          -0.03945612534880638,
+          0.016651209443807602,
+          -0.006040588021278381,
+          0.022186528891324997,
+          0.027917923405766487,
+          -0.020135292783379555,
+          -0.019350998103618622,
+          0.013732722960412502,
+          -0.04171851649880409,
+          0.030285894870758057,
+          0.01812930591404438,
+          -0.005037594120949507,
+          0.010459794662892818,
+          0.0033125197514891624,
+          0.053965602070093155,
+          -0.026213588193058968,
+          0.020844176411628723,
+          -0.04105488210916519,
+          0.04775156453251839,
+          -0.005836972501128912,
+          0.03984827175736427,
+          -0.024117104709148407,
+          -0.013838300481438637,
+          -0.01322745531797409,
+          -0.017556166276335716,
+          -0.006719305645674467,
+          -0.022020621225237846,
+          -0.0005962346913293004,
+          -0.0063196164555847645,
+          -0.0033030931372195482,
+          0.03646976873278618,
+          0.014072081074118614,
+          0.021628472954034805,
+          0.041507359594106674,
+          0.0015572801930829883,
+          -0.07173292338848114,
+          0.030828868970274925,
+          0.007684593088924885,
+          0.030919363722205162,
+          0.006839966867119074,
+          0.015897078439593315,
+          0.006066982634365559,
+          0.021387150511145592,
+          0.002356658922508359,
+          0.004947098437696695,
+          0.005754018202424049,
+          0.04135653376579285,
+          -0.02336297370493412,
+          0.001117998966947198,
+          0.015218360349535942,
+          0.012488407082855701,
+          0.03372473269701004,
+          0.023061320185661316,
+          0.016832200810313225,
+          0.015957407653331757,
+          0.004430518485605717,
+          0.0057163117453455925,
+          0.027118545025587082,
+          0.014901624992489815,
+          -0.0032898958306759596,
+          -0.020753679797053337,
+          0.0024188747629523277,
+          -0.024599747732281685,
+          -0.002049350645393133,
+          -0.01785781793296337,
+          0.034961506724357605,
+          0.0012782517587766051,
+          -0.026183422654867172,
+          -0.014984579756855965,
+          0.005836972501128912,
+          -0.03613794967532158,
+          -0.0010991457384079695,
+          0.0015563375782221556,
+          0.016952861100435257,
+          0.005870908498764038,
+          0.013981585390865803,
+          -0.03405654802918434,
+          0.012141506187617779,
+          -0.023679709061980247,
+          -0.006142395548522472,
+          0.004988575354218483,
+          -0.0007941940566524863,
+          0.013423528522253036,
+          -0.01253365445882082,
+          0.02022578939795494,
+          -0.005361870396882296,
+          -0.028837963938713074,
+          -0.007752465084195137,
+          0.02462991327047348,
+          0.004011976066976786,
+          -0.006097147706896067,
+          -0.010210931301116943,
+          -0.01613840088248253,
+          -0.030723290517926216,
+          -0.017948314547538757,
+          0.03155283257365227,
+          0.014449146576225758,
+          -0.015489847399294376,
+          0.020527441054582596,
+          0.007394252810627222,
+          0.0028939771000295877,
+          -0.014924249611794949,
+          -0.05559452623128891,
+          -0.009969609789550304,
+          -0.012797600589692593,
+          -0.000906370987650007,
+          0.0007253796211443841,
+          -0.04506685957312584,
+          0.02573094516992569,
+          -0.03674125298857689,
+          -0.0009040143340826035,
+          0.015791499987244606,
+          0.006938003934919834,
+          -0.006127313245087862,
+          0.011666404083371162,
+          -0.026590652763843536,
+          0.01303138118237257,
+          -0.030059656128287315,
+          -0.003970498684793711,
+          6.586861854884773e-05,
+          0.015173112042248249,
+          -0.023227229714393616,
+          0.034629687666893005,
+          0.021839629858732224,
+          -0.06274368613958359,
+          -0.008318062871694565,
+          -0.01835554465651512,
+          0.05456890910863876,
+          -0.01449439488351345,
+          -0.008144613355398178,
+          0.052306514233350754,
+          -0.009441718459129333,
+          0.002309525618329644,
+          0.031371843069791794,
+          0.027118545025587082,
+          0.01377042941749096,
+          -0.02897370606660843,
+          0.002942995633929968,
+          -0.07517176121473312,
+          -0.012488407082855701,
+          0.013415987603366375,
+          0.034086715430021286,
+          -0.03063279390335083,
+          0.0013838300947099924,
+          0.015474764630198479,
+          0.01895884983241558,
+          0.03170366212725639,
+          0.002475434448570013,
+          0.02963734231889248,
+          0.010452253744006157,
+          0.01011289469897747,
+          0.004140178207308054,
+          -0.004935786128044128,
+          -0.028340237215161324,
+          -0.0015544522320851684,
+          -0.03167349472641945,
+          0.0005425028502941132,
+          -0.008099365048110485,
+          -0.014094705693423748,
+          0.004355105571448803,
+          0.005403347313404083,
+          0.04651479050517082,
+          -0.006949315778911114,
+          -0.012563819997012615,
+          -0.01306154578924179,
+          -0.0002839773951563984,
+          -0.04144703224301338,
+          -0.0399991013109684,
+          0.014946873299777508,
+          0.005105465650558472,
+          -0.023709872737526894,
+          -0.0516730435192585,
+          -0.012880554422736168,
+          -0.022895412519574165,
+          0.035564810037612915,
+          -0.022729502990841866,
+          -0.012805141508579254,
+          0.0416581891477108,
+          -0.003783851396292448,
+          0.03731439262628555,
+          -0.057494934648275375,
+          -0.008604632690548897,
+          0.006545855663716793,
+          -0.005260062403976917,
+          0.014841294847428799,
+          -0.026424745097756386,
+          -0.024056773632764816,
+          -0.010361758060753345,
+          -0.015180653892457485,
+          -0.04760073870420456,
+          0.0017759781330823898,
+          -0.02028612047433853,
+          0.017118770629167557,
+          0.046152807772159576,
+          0.03052721731364727,
+          -0.02469024434685707,
+          0.014072081074118614,
+          -0.008385934866964817,
+          -0.054689567536115646,
+          -0.025595201179385185,
+          0.010097811929881573,
+          -0.020738597959280014,
+          -0.041235875338315964,
+          -0.02043694630265236,
+          0.0005128089687786996,
+          -0.022669173777103424,
+          0.016168564558029175,
+          -0.024539416655898094,
+          -0.03360407054424286,
+          0.000932294235099107,
+          0.0206631850451231,
+          -0.0028920918703079224,
+          -0.017224349081516266,
+          -0.02153797633945942,
+          -0.02562536671757698,
+          0.013793053105473518,
+          -0.004573803395032883,
+          0.0009091990068554878,
+          0.019592318683862686,
+          0.03254828602075577,
+          0.025534870103001595,
+          0.0035406441893428564,
+          -0.03333258256316185,
+          -0.021191077306866646,
+          0.026032596826553345,
+          0.0018824990838766098,
+          -0.0013084170641377568,
+          0.03245779126882553,
+          -0.03653009608387947,
+          -0.04325694590806961,
+          0.005554173607379198,
+          -0.04910900071263313,
+          0.0020418092608451843,
+          -0.021462563425302505,
+          -0.022503264248371124,
+          -0.013898631557822227,
+          0.002277475083246827,
+          0.003847952699288726,
+          0.02909436821937561,
+          0.011658863164484501,
+          -0.010482418350875378,
+          -0.023438386619091034,
+          0.008521678857505322,
+          0.0483548678457737,
+          -0.010060105472803116,
+          -0.02710346318781376,
+          0.03278961032629013,
+          -0.016455134376883507,
+          0.01168148685246706,
+          -0.011425082571804523,
+          -0.022895412519574165,
+          -0.01592724397778511,
+          0.017827652394771576,
+          0.01895884983241558,
+          0.006172561086714268,
+          -0.041235875338315964,
+          0.025474539026618004,
+          -0.0063497815281152725,
+          -0.03840034082531929,
+          0.014056999236345291,
+          0.004128866363316774,
+          0.01239791139960289,
+          0.0005486301961354911,
+          0.013174666091799736,
+          -0.054297421127557755,
+          -0.005844513885676861,
+          0.006036817561835051,
+          -0.03191481530666351,
+          -0.0058595966547727585,
+          -0.008378393948078156,
+          0.02280491590499878,
+          -0.014946873299777508,
+          0.015391810797154903,
+          -0.012405452318489552,
+          0.008144613355398178,
+          0.031975146383047104,
+          -0.03441853076219559,
+          -0.015067534521222115,
+          -0.036228444427251816,
+          -0.0041929674334824085,
+          -0.022458016872406006,
+          0.020361533388495445,
+          0.04334744065999985,
+          -0.017661744728684425,
+          0.00917777232825756,
+          0.0028185639530420303,
+          -0.004241986200213432,
+          -0.022005537524819374,
+          0.016817117109894753,
+          0.0569821260869503,
+          0.031100355088710785,
+          -0.008039034903049469,
+          -0.0006443105521611869,
+          0.02197537198662758,
+          -0.013121876865625381,
+          -0.008340687490999699,
+          0.006051899865269661,
+          0.017752239480614662,
+          0.010701116174459457,
+          0.02286524698138237,
+          -0.040089596062898636,
+          -0.010150601156055927,
+          0.009109900332987309,
+          -0.02469024434685707,
+          0.027812344953417778,
+          -0.03073837235569954,
+          0.018973931670188904,
+          0.017812570556998253,
+          -0.01956215314567089,
+          0.013627144508063793,
+          -0.010022399015724659,
+          -0.023830534890294075,
+          -0.02512763999402523,
+          -0.02496173046529293,
+          -0.008634798228740692,
+          0.017978480085730553,
+          -0.0139966681599617,
+          -0.05287965387105942,
+          0.04301562160253525,
+          -0.06298501044511795,
+          -0.029773086309432983,
+          0.03191481530666351,
+          -0.02027103677392006,
+          -0.014313403517007828,
+          0.008672504685819149,
+          0.01570100337266922,
+          -0.033362749963998795,
+          0.020452028140425682,
+          0.011643780395388603,
+          0.033151593059301376,
+          -0.028023501858115196,
+          -0.024931564927101135,
+          0.0034765431191772223,
+          -0.02739003300666809,
+          -0.006010422948747873,
+          -0.0002648884546943009,
+          0.014667844399809837,
+          0.00637617614120245,
+          -0.033754896372556686,
+          -0.009743370115756989,
+          0.003800819395110011,
+          0.0027506921906024218,
+          0.03348340839147568,
+          -0.004132636822760105,
+          -0.04205033555626869,
+          0.014170118607580662,
+          0.004321169573813677,
+          -0.03284993767738342,
+          0.006406341679394245,
+          0.003346455516293645,
+          -0.009185313247144222,
+          -0.016334474086761475,
+          0.005478760693222284,
+          0.03221647068858147,
+          0.030210481956601143,
+          0.017812570556998253,
+          0.007118995301425457,
+          0.009871572256088257,
+          -0.01338582206517458,
+          -0.01503736898303032,
+          0.010655868798494339,
+          -0.007782630156725645,
+          -0.03191481530666351,
+          -0.013204830698668957,
+          -0.014780964702367783,
+          0.025866687297821045,
+          -0.012563819997012615,
+          -0.01253365445882082,
+          0.005980257410556078,
+          0.045157354325056076,
+          0.014615056104958057,
+          -0.005267603788524866,
+          -0.01336319837719202,
+          0.007601638790220022,
+          0.012707104906439781,
+          -0.034358199685811996,
+          -0.022277025505900383,
+          0.022065868601202965,
+          0.0023283790796995163,
+          0.017375174909830093,
+          -0.024599747732281685,
+          -0.022216694429516792,
+          0.03239746019244194,
+          -0.03248795494437218,
+          0.030919363722205162,
+          0.01580658182501793,
+          0.006202726159244776,
+          -0.006444048136472702,
+          0.004739712458103895,
+          -0.014569807797670364,
+          0.028400568291544914,
+          -0.029350772500038147,
+          0.027193957939743996,
+          0.020421862602233887,
+          -0.006285680457949638,
+          0.060300301760435104,
+          -0.029682589694857597,
+          -0.016334474086761475,
+          0.009607627056539059,
+          -0.0043777297250926495,
+          0.015391810797154903,
+          -0.019094591960310936,
+          -0.007149160373955965,
+          -0.0060745240189135075,
+          0.005018740892410278,
+          -0.029667507857084274,
+          -0.015897078439593315,
+          0.02164355479180813,
+          -0.005987798795104027,
+          -0.0013272702926769853,
+          0.009660416282713413,
+          -0.02286524698138237,
+          0.017495835199952126,
+          -0.006542085204273462,
+          -0.04533834755420685,
+          -0.00045813448377884924,
+          0.02153797633945942,
+          -0.024388590827584267,
+          -0.015912160277366638,
+          -0.008582009002566338,
+          0.0038837739266455173,
+          -0.00595009233802557,
+          -0.02280491590499878,
+          -0.019245419651269913,
+          -0.005655981134623289,
+          -0.008385934866964817,
+          -0.027480527758598328,
+          -0.011779523454606533,
+          0.034086715430021286,
+          0.01294842641800642,
+          -0.004875455982983112,
+          -0.010301426984369755,
+          -0.029667507857084274,
+          0.016002656891942024,
+          0.04373958706855774,
+          0.0034218686632812023,
+          0.006941774394363165,
+          -0.0065948739647865295,
+          -0.0026451139710843563,
+          -0.014841294847428799,
+          0.009864031337201595,
+          0.0017646661726757884,
+          -0.0028223346453160048,
+          0.01790306530892849,
+          -0.011342127807438374,
+          -0.03381522744894028,
+          -0.026967719197273254,
+          0.04304578900337219,
+          -0.023785287514328957,
+          0.004483307711780071,
+          0.04651479050517082,
+          0.036620594561100006,
+          -0.005343017168343067,
+          -0.014305861666798592,
+          -0.03107019141316414,
+          0.0020663184113800526,
+          -0.02010512910783291,
+          0.009509589523077011,
+          0.03294043615460396,
+          -0.00039686134550720453,
+          -0.034297872334718704,
+          -0.013529106974601746,
+          -0.02099500223994255,
+          -0.010346675291657448,
+          0.015716087073087692,
+          0.017013192176818848,
+          -0.034358199685811996,
+          0.005795495584607124,
+          0.018310297280550003,
+          0.028943542391061783,
+          -0.011349668726325035,
+          -0.016983026638627052,
+          0.024765657261013985,
+          0.017224349081516266,
+          0.0009294662741012871,
+          -0.0029222569428384304,
+          -0.00034312953357584774,
+          0.0068098013289272785,
+          -0.004068535752594471,
+          0.033030930906534195,
+          0.0035274471156299114,
+          0.011206384748220444,
+          -0.02116091176867485,
+          0.00479250168427825,
+          0.020738597959280014,
+          -0.006870131939649582,
+          -0.001355550135485828,
+          0.07384449243545532,
+          0.00214173155836761,
+          0.01355173159390688,
+          0.017209265381097794,
+          0.03444869816303253,
+          0.01773715764284134,
+          -0.017993561923503876,
+          -0.03170366212725639,
+          -0.014313403517007828,
+          -0.007718529086560011,
+          -0.020527441054582596,
+          -0.0049697221256792545,
+          -0.012910719960927963,
+          -0.0433172769844532,
+          -0.0223222728818655,
+          -0.0033502262085676193,
+          -0.002466007834300399,
+          0.025263383984565735,
+          -0.0026319166645407677,
+          0.0009822554420679808,
+          -0.027948088943958282,
+          0.03565530478954315,
+          0.02622867003083229,
+          0.023996442556381226,
+          -0.007880667224526405,
+          0.011100806295871735,
+          -0.0043362523429095745,
+          -0.008491513319313526,
+          0.02754085883498192,
+          0.0074885194189846516,
+          -0.008137071505188942,
+          0.03478051349520683,
+          0.012111341580748558,
+          0.05272882804274559,
+          -0.013114335015416145,
+          -0.024343343451619148,
+          -0.017179099842905998,
+          -0.04018009081482887,
+          0.011372293345630169,
+          -0.007710987702012062,
+          0.0001368040539091453,
+          0.014139953069388866,
+          0.017390256747603416,
+          -0.008167237043380737,
+          0.0271637924015522,
+          0.0038498379290103912,
+          -0.015504930168390274,
+          0.025187969207763672,
+          -0.02892845869064331,
+          0.014253072440624237,
+          -0.0035576121881604195,
+          0.011839854530990124,
+          -0.012903179042041302,
+          0.011538201943039894,
+          0.02358921244740486,
+          -0.025761108845472336,
+          0.028551394119858742,
+          0.017254512757062912,
+          -0.0027412655763328075,
+          0.00480758398771286,
+          0.0026168341282755136,
+          -0.0012169786496087909,
+          -0.022080950438976288,
+          -0.034086715430021286,
+          0.02533879689872265,
+          -0.007473436649888754,
+          -0.023332808166742325,
+          0.02425284869968891,
+          -0.017118770629167557,
+          -0.008928908966481686,
+          -0.0017175329849123955,
+          -0.009358763694763184,
+          -0.007579015102237463,
+          0.04292512685060501,
+          -0.008664963766932487,
+          -0.004291004501283169,
+          -0.01640988700091839,
+          0.02689230628311634,
+          0.04226149246096611,
+          -0.008385934866964817,
+          -0.009072193875908852,
+          0.021899959072470665,
+          -0.012450700625777245,
+          0.0015129749663174152,
+          0.041628021746873856,
+          0.006153707858175039,
+          0.0015365416184067726,
+          0.005712541285902262,
+          -0.03393588960170746,
+          -0.029999325051903725,
+          0.005806807428598404,
+          0.016455134376883507,
+          0.016047904267907143,
+          0.04126603901386261,
+          0.03646976873278618,
+          -0.014969496987760067,
+          0.0025037142913788557,
+          0.0007546021952293813,
+          -0.035232994705438614,
+          0.028747467324137688,
+          -0.005486302077770233,
+          0.021447481587529182,
+          0.02485615201294422,
+          -0.026183422654867172,
+          -0.036620594561100006,
+          -0.00034996384056285024,
+          0.030210481956601143,
+          0.0190040972083807,
+          0.014056999236345291,
+          0.014072081074118614,
+          0.023227229714393616,
+          -0.020316284149885178,
+          -0.024946648627519608,
+          0.02012021094560623,
+          0.02506730891764164,
+          -0.009456800296902657,
+          0.01074636448174715,
+          0.017933230847120285,
+          0.009818783961236477,
+          -0.0012282906100153923,
+          0.03134167939424515,
+          0.009019404649734497,
+          0.016289226710796356,
+          0.008287898264825344,
+          -0.011515578255057335,
+          0.02500697784125805,
+          0.010271262377500534,
+          0.03393588960170746,
+          -0.012412993237376213,
+          0.003929021768271923,
+          0.003689585020765662,
+          -0.004087388981133699,
+          0.037163566797971725,
+          0.005946321878582239,
+          -0.04307595267891884,
+          0.0199995506554842,
+          0.011960514821112156,
+          -0.0001776920835254714,
+          -0.006489295978099108,
+          -0.0026469992008060217,
+          0.018446041271090508,
+          0.02043694630265236,
+          0.018038809299468994,
+          0.009532214142382145,
+          0.005659752059727907,
+          0.005618274677544832,
+          0.006138625089079142,
+          0.03167349472641945,
+          0.0020455799531191587,
+          -0.01978839375078678,
+          -0.00230009900406003,
+          0.005505155306309462,
+          0.021115664392709732,
+          -0.019637567922472954,
+          0.01653054729104042,
+          0.019215254113078117,
+          -0.0011934119975194335,
+          -0.01746566966176033,
+          -0.03671108931303024,
+          -0.007303757127374411,
+          -0.013853383250534534,
+          0.013898631557822227,
+          0.008951532654464245,
+          0.018687361851334572,
+          -0.0221563633531332,
+          0.016078069806098938,
+          0.019592318683862686,
+          -0.01950182393193245,
+          0.006496837362647057,
+          0.008061658591032028,
+          -0.014524560421705246,
+          -0.005580568220466375,
+          -0.016108235344290733,
+          0.002053121104836464,
+          0.06382963061332703,
+          -0.014464229345321655,
+          0.009954527020454407,
+          0.024931564927101135,
+          -0.009879114106297493,
+          -0.030119985342025757,
+          0.0009671727893874049,
+          -0.010429629124701023,
+          -0.002450925298035145,
+          0.024373508989810944,
+          -0.03577596694231033,
+          -0.04319661483168602,
+          -0.005290227942168713,
+          -0.03631894290447235,
+          -0.006560938432812691,
+          0.019034262746572495,
+          0.027948088943958282,
+          -0.014162576757371426,
+          -0.02760118804872036,
+          0.02292557805776596,
+          -0.0020418092608451843,
+          0.009765994735062122,
+          -0.019471658393740654,
+          -0.026545405387878418,
+          -0.03030097670853138,
+          -0.03212597221136093,
+          -0.00961516797542572,
+          -0.025097474455833435,
+          0.008408558554947376,
+          0.0012725958367809653,
+          -0.05061726272106171,
+          -0.030602630227804184,
+          0.010670951567590237,
+          -0.002435842528939247,
+          0.04005942866206169,
+          0.039757777005434036,
+          0.003970498684793711,
+          0.028008420020341873,
+          -0.03613794967532158,
+          0.050828419625759125,
+          -0.006530772894620895,
+          0.03592679277062416,
+          -0.007579015102237463,
+          0.0019890200346708298,
+          -0.014358650892972946,
+          0.04379991814494133,
+          0.000892702373676002,
+          0.002547076903283596,
+          -0.038762323558330536,
+          0.0021455022506415844,
+          -0.023996442556381226,
+          -0.00025451916735619307,
+          -0.0026752790436148643,
+          0.008996780961751938,
+          -0.0009907394414767623,
+          -0.014856377616524696,
+          0.002102139638736844,
+          0.03345324471592903,
+          0.02616834081709385,
+          -0.021779298782348633,
+          -0.027088379487395287,
+          -0.030617712065577507,
+          0.0004138292861171067,
+          -0.008725293911993504,
+          -0.002880779793485999,
+          -0.004860373213887215,
+          -0.024313177913427353,
+          0.014396357350051403,
+          -0.016771869733929634,
+          -0.013415987603366375,
+          -0.002205832628533244,
+          0.02545945718884468,
+          -0.01746566966176033,
+          -0.02259376086294651,
+          -0.005814348813146353,
+          -0.029335688799619675,
+          -0.01675678789615631,
+          0.00022930291015654802,
+          -0.02485615201294422,
+          0.017510918900370598,
+          0.01609315164387226,
+          -0.012103799730539322,
+          0.0002860983950085938,
+          0.0022001767065376043,
+          0.016123317182064056,
+          -0.021794380620121956,
+          -0.004166572820395231,
+          0.0025357650592923164,
+          -0.00017439275688957423,
+          -0.023151816800236702,
+          0.021387150511145592,
+          -0.019411327317357063,
+          0.01631939224898815,
+          -0.04844536632299423,
+          -0.011213925667107105,
+          -0.025037143379449844,
+          0.010587996803224087,
+          0.001273538451641798,
+          -0.02990882843732834,
+          -0.014072081074118614,
+          0.00785050168633461,
+          -0.0031447256915271282,
+          -0.02182454615831375,
+          0.008506596088409424,
+          0.03541398420929909,
+          -0.0014385045506060123,
+          0.009683039970695972,
+          -0.029773086309432983,
+          -0.011590991169214249,
+          -0.010158142074942589,
+          0.03079870343208313,
+          0.0037574570160359144,
+          -0.005131860263645649,
+          -0.00574270635843277,
+          0.0005769100971519947,
+          -0.03472018241882324,
+          -0.021899959072470665,
+          -0.003259730525314808,
+          -0.002062547951936722,
+          -0.005392035469412804,
+          0.034509025514125824,
+          0.03405654802918434,
+          0.03686191514134407,
+          -0.04379991814494133,
+          0.019215254113078117,
+          -0.008325604721903801,
+          -0.011108347214758396,
+          0.01325007900595665,
+          0.018777858465909958,
+          0.01840079203248024,
+          -0.016485299915075302,
+          0.001463956548832357,
+          0.003412442049011588,
+          0.013536648824810982,
+          -0.005603192374110222,
+          0.022985907271504402,
+          -0.00719063775613904,
+          0.013408446684479713,
+          0.014260614290833473,
+          -8.476607035845518e-06,
+          0.022955743595957756,
+          -0.006843737326562405,
+          0.009901737794280052,
+          -0.007043581921607256,
+          -0.014094705693423748,
+          0.0017872900934889913,
+          -0.03882265463471413,
+          -0.013016298413276672,
+          0.015384268946945667,
+          0.018777858465909958,
+          -0.012495948001742363,
+          -0.04874701797962189,
+          0.022020621225237846,
+          0.03671108931303024,
+          -0.019275585189461708,
+          0.0273297019302845,
+          -0.0279933363199234,
+          0.016334474086761475,
+          0.011613614857196808,
+          0.013272702693939209,
+          0.004773647990077734,
+          0.020738597959280014,
+          -0.02512763999402523,
+          0.033966053277254105,
+          -0.0019550842698663473,
+          0.025761108845472336,
+          0.016847282648086548,
+          -0.015391810797154903,
+          -0.04573049396276474,
+          -0.016108235344290733,
+          0.006753241643309593,
+          -0.012028386816382408,
+          -0.002897747792303562,
+          -0.043498266488313675,
+          -0.03420737385749817,
+          0.030210481956601143,
+          0.01418520137667656,
+          -0.0493503212928772,
+          0.00014446319255512208,
+          -0.014034374617040157,
+          -0.0030730832368135452,
+          -0.0052261268720030785,
+          -0.028943542391061783,
+          -0.02336297370493412,
+          0.00801641121506691,
+          0.00752245495095849,
+          0.03981810808181763,
+          -0.023046238347887993,
+          0.026741480454802513,
+          -0.01159853208810091,
+          -0.01719418354332447,
+          -0.02005987986922264,
+          -0.00045224284986034036,
+          -0.021326819434762,
+          -0.01481867115944624,
+          0.005094153806567192,
+          0.020738597959280014,
+          0.009660416282713413,
+          0.029396019876003265,
+          -0.012722187675535679,
+          -0.014856377616524696,
+          -0.0005095096421428025,
+          -0.03465985506772995,
+          -0.028506144881248474,
+          0.007032270077615976,
+          0.0006749471067450941,
+          -0.017269596457481384,
+          -0.02070843242108822,
+          -0.03704290837049484,
+          0.016213813796639442,
+          0.011990680359303951,
+          0.01692269556224346,
+          0.03245779126882553,
+          0.0198336411267519,
+          -0.01386092510074377,
+          0.01669645681977272,
+          -0.007888208143413067,
+          0.0009813127107918262,
+          0.01013551838696003,
+          0.018581783398985863,
+          -0.013114335015416145,
+          -0.0027733161114156246,
+          -0.014765881933271885,
+          -0.029003871604800224,
+          -0.01708860509097576,
+          -0.01806897483766079,
+          -0.0008795050671324134,
+          -0.006149936933070421,
+          -0.03891315311193466,
+          -0.013415987603366375,
+          0.024991896003484726,
+          0.02396627888083458,
+          -0.01534656248986721,
+          -0.007511143106967211,
+          -0.00023295573191717267,
+          -0.0208290945738554,
+          -0.004894309211522341,
+          -0.02342330478131771,
+          -0.003672617254778743,
+          -0.024147270247340202,
+          0.013234996236860752,
+          -0.0045813447795808315,
+          0.010052564553916454,
+          -0.053694114089012146,
+          -0.013174666091799736,
+          0.007326381281018257,
+          -0.010105352848768234,
+          -0.0016996223712339997,
+          0.01746566966176033,
+          -0.003291781060397625,
+          -0.009162689559161663,
+          -0.023845616728067398,
+          0.014155035838484764,
+          0.005772871430963278,
+          0.021025167778134346,
+          -0.002831761259585619,
+          -0.00406476529315114,
+          0.03348340839147568,
+          0.025384044274687767,
+          -5.541094287764281e-05,
+          -0.012269708327949047,
+          0.015897078439593315,
+          0.014788505621254444,
+          0.0014507592422887683,
+          0.0044493721798062325,
+          0.009102359414100647,
+          0.0048264372162520885,
+          -0.010889649391174316,
+          -0.00480758398771286,
+          0.024871235713362694,
+          0.006527002435177565,
+          0.005935009568929672,
+          -0.01463013794273138,
+          0.02897370606660843,
+          0.03752554953098297,
+          0.0196677315980196,
+          0.028068749234080315,
+          -0.015685921534895897,
+          -0.011032934300601482,
+          -0.007760006468743086,
+          -0.013762887567281723,
+          0.031432174146175385,
+          0.008861036971211433,
+          -0.009871572256088257,
+          -0.0104145472869277,
+          0.010037481784820557,
+          -0.009939444251358509,
+          0.002166240941733122,
+          0.02269933745265007,
+          0.009592544287443161,
+          0.005580568220466375,
+          -0.00028892638511024415,
+          0.007548849564045668,
+          0.015912160277366638,
+          0.023770203813910484,
+          -0.02975800260901451,
+          -0.027420196682214737,
+          -0.0009120269678533077,
+          0.0003007096820510924,
+          0.012993674725294113,
+          0.01767682656645775,
+          -0.0011594761162996292,
+          0.006658975500613451,
+          0.04334744065999985,
+          -0.015746252611279488,
+          -0.0034369511995464563,
+          -0.014698009938001633,
+          -0.013272702693939209,
+          -0.0022491952404379845,
+          -0.021945208311080933,
+          0.008687587454915047,
+          0.012382828630506992,
+          0.017118770629167557,
+          0.014245531521737576,
+          -0.008023952133953571,
+          0.012322497554123402,
+          0.018446041271090508,
+          0.011515578255057335,
+          -0.004219362046569586,
+          -0.014780964702367783,
+          -0.002564044902101159,
+          0.015248525887727737,
+          0.009034487418830395,
+          0.022835081443190575,
+          0.021326819434762,
+          0.006949315778911114,
+          0.01322745531797409,
+          -0.025052227079868317,
+          0.025746027007699013,
+          0.0144114401191473,
+          0.02419251762330532,
+          -0.03770654276013374,
+          -0.021145828068256378,
+          0.014501935802400112,
+          -0.007141618989408016,
+          -0.0004892423748970032,
+          0.015384268946945667,
+          -0.008091824129223824,
+          0.05125073343515396,
+          -0.0353536531329155,
+          0.00845380686223507,
+          0.02358921244740486,
+          -0.015881994739174843,
+          0.010271262377500534,
+          0.01818963512778282,
+          0.010595538653433323,
+          -0.010678492486476898,
+          -0.02093467302620411,
+          0.00037730109761469066,
+          0.02551978826522827,
+          -0.03818918392062187,
+          -0.018611948937177658,
+          -0.0065345438197255135,
+          -0.0004892423748970032,
+          0.022729502990841866,
+          -0.007552620489150286,
+          0.014645220711827278,
+          -0.008928908966481686,
+          0.011432623490691185,
+          0.001585560035891831,
+          -0.03134167939424515,
+          -0.007537537720054388,
+          0.014637679792940617,
+          0.01231495663523674,
+          -0.009667957201600075,
+          -0.006674057804048061,
+          -0.0010331592056900263,
+          -0.002075745025649667,
+          -0.014268155209720135,
+          0.02722412347793579,
+          0.0018617605092003942,
+          0.01592724397778511,
+          -0.00208705710247159,
+          -0.014207825064659119,
+          0.018551617860794067,
+          -0.01962248422205448,
+          -0.015256066806614399,
+          -0.004521014168858528,
+          0.001125540235079825,
+          -0.0011905840365216136,
+          -0.031100355088710785,
+          0.015361645258963108,
+          -0.013257619924843311,
+          0.034629687666893005,
+          0.006180102005600929,
+          -0.0012207493418827653,
+          -0.002481090370565653,
+          -0.011606073938310146,
+          -0.0056145042181015015,
+          -0.02821957692503929,
+          -0.017872901633381844,
+          0.01686236634850502,
+          -0.013536648824810982,
+          -0.009909279644489288,
+          -0.026515239849686623,
+          -0.021734051406383514,
+          -0.011040475219488144,
+          0.008793165907263756,
+          0.0014347339747473598,
+          0.022850165143609047,
+          0.00864988099783659,
+          -0.017375174909830093,
+          -0.0033671942073851824,
+          -0.041959840804338455,
+          -0.002809137338772416,
+          0.01033159252256155,
+          -0.017541082575917244,
+          -0.022246859967708588,
+          0.013204830698668957,
+          0.00048052275087684393,
+          -0.018596867099404335,
+          0.019411327317357063,
+          0.0019183203112334013,
+          0.004434289410710335,
+          0.0017147050239145756,
+          -0.0023151817731559277,
+          -0.01653054729104042,
+          0.00573139451444149,
+          0.009554837830364704,
+          0.01536918617784977,
+          -0.047238755971193314,
+          -0.007499831262975931,
+          0.003244647989049554,
+          0.006372405681759119,
+          0.006010422948747873,
+          0.01118376012891531,
+          0.004777418915182352,
+          -0.017767323181033134,
+          0.004471995867788792,
+          0.01906442828476429,
+          -0.022563595324754715,
+          0.01977331005036831,
+          -0.0019051230046898127,
+          0.0027883988805115223,
+          0.002799710724502802,
+          -0.007514914032071829,
+          -0.005490072537213564,
+          0.02583652175962925,
+          -0.010452253744006157,
+          0.009622709825634956,
+          0.021507810801267624,
+          0.02309148572385311,
+          -0.011470329947769642,
+          -0.01862703263759613,
+          0.013710098341107368,
+          0.0316433310508728,
+          0.0006923863547854125,
+          -0.014992120675742626,
+          -0.054086264222860336,
+          -0.011206384748220444,
+          0.03218630328774452,
+          0.015897078439593315,
+          0.017390256747603416,
+          -0.006941774394363165,
+          -0.012518571689724922,
+          0.019139841198921204,
+          0.023559046909213066,
+          0.017043357715010643,
+          -0.026545405387878418,
+          0.013928796164691448,
+          -0.020844176411628723,
+          0.03432803601026535,
+          0.0030429179314523935,
+          -0.008008869364857674,
+          0.004143949132412672,
+          -0.013431070372462273,
+          -0.02313673496246338,
+          0.014200284145772457,
+          -0.003846067236736417,
+          0.0028826650232076645,
+          0.006545855663716793,
+          -0.004864144138991833,
+          0.01939624547958374,
+          -0.0008766771061345935,
+          -0.005897303111851215,
+          0.004747253842651844,
+          0.02787267602980137,
+          0.0013659194810315967,
+          0.003993122838437557,
+          0.0011085723526775837,
+          0.010090271010994911,
+          -0.0019946759566664696,
+          -0.012586443684995174,
+          0.022563595324754715,
+          -0.012865472584962845,
+          -0.0056145042181015015,
+          0.01923033595085144,
+          -0.007903290912508965,
+          -0.00023342706845141947,
+          -0.010452253744006157,
+          -0.008838413283228874,
+          -0.03152266889810562,
+          0.026183422654867172,
+          -0.0036066307220607996,
+          0.014396357350051403,
+          0.021010085940361023,
+          -0.0012094373814761639,
+          -0.005410888697952032,
+          0.022337354719638824,
+          0.0002851557219401002,
+          -0.007458353880792856,
+          0.01386092510074377,
+          -0.02562536671757698,
+          -0.02612309157848358,
+          0.03858133405447006,
+          -0.012269708327949047,
+          0.0005542861763387918,
+          -0.0022491952404379845,
+          -0.00906465295702219,
+          -0.007511143106967211,
+          0.020542524755001068,
+          0.01708860509097576,
+          -0.02793300710618496,
+          -0.010595538653433323,
+          -0.010127977468073368,
+          -0.02909436821937561,
+          0.011590991169214249,
+          -0.0018532765097916126,
+          0.02153797633945942,
+          -0.005799266044050455,
+          -0.03155283257365227,
+          -0.004762336146086454,
+          0.03592679277062416,
+          -0.003981810994446278,
+          0.0010048793628811836,
+          0.015323938801884651,
+          0.0028675824869424105,
+          0.015610508620738983,
+          -0.029456350952386856,
+          0.025474539026618004,
+          -0.01640988700091839,
+          0.020255954936146736,
+          0.013483859598636627,
+          -0.014464229345321655,
+          0.027736932039260864,
+          0.013234996236860752,
+          0.005493842996656895,
+          0.006413882598280907,
+          0.0030052114743739367,
+          -0.010859483852982521,
+          -0.007077517919242382,
+          0.008340687490999699,
+          -0.002134190173819661,
+          0.0003591548011172563,
+          0.02594210021197796,
+          0.0003921480383723974,
+          0.013762887567281723,
+          0.001332926214672625,
+          -0.0001895932073239237,
+          0.0017618382116779685,
+          -0.002952422248199582,
+          -0.01944149285554886,
+          0.011304421350359917,
+          -0.015912160277366638,
+          -0.020964836701750755,
+          0.012443158775568008,
+          -0.017269596457481384,
+          -0.01785781793296337,
+          -0.028113998472690582,
+          0.01283530704677105,
+          -0.008861036971211433,
+          -0.007337693125009537,
+          0.009667957201600075,
+          0.011764440685510635,
+          0.015595425851643085,
+          0.02660573646426201,
+          0.019863806664943695,
+          -0.012744811363518238,
+          0.013762887567281723,
+          0.010489960201084614,
+          0.008830872364342213,
+          -0.00939647015184164,
+          0.0018146273214370012,
+          -0.010308968834578991,
+          0.020240871235728264,
+          0.010912273079156876,
+          0.004038370680063963,
+          0.019200170412659645,
+          -0.007669510785490274,
+          -0.01264677383005619,
+          -0.03505200147628784,
+          -0.004882997367531061,
+          -0.001891925698146224,
+          0.007262280210852623,
+          0.04998379200696945,
+          -0.014245531521737576,
+          -0.03300076723098755,
+          -0.016847282648086548,
+          -0.011802148073911667,
+          0.009645333513617516,
+          0.0012443158775568008,
+          0.010723740793764591,
+          -0.013491400517523289,
+          -0.035564810037612915,
+          0.0030825098510831594,
+          0.015776416286826134,
+          0.003531217575073242,
+          -0.0020041028037667274,
+          0.022337354719638824,
+          -0.028113998472690582,
+          0.0180086437612772,
+          -0.007514914032071829,
+          -0.011470329947769642,
+          -0.02088942378759384,
+          -0.0033671942073851824,
+          -0.0076581984758377075,
+          0.023845616728067398,
+          -0.005497613921761513,
+          -0.01978839375078678,
+          -0.003457689890637994,
+          0.011357210576534271,
+          -0.0009516188292764127,
+          -0.005659752059727907,
+          0.004928245209157467,
+          0.006406341679394245,
+          -0.0029769313987344503,
+          0.033905722200870514,
+          -0.0017128196777775884,
+          -0.0024792051408439875,
+          -0.0063196164555847645,
+          -0.0027733161114156246,
+          0.00947942491620779,
+          0.015022286213934422,
+          -0.047087930142879486,
+          -0.019878888502717018,
+          -0.007254738826304674,
+          -0.004079848062247038,
+          0.0051544844172894955,
+          0.012722187675535679,
+          -0.02104024961590767,
+          -0.00375934224575758,
+          -0.011191301979124546,
+          -0.008536760695278645,
+          -0.029516680166125298,
+          -0.010889649391174316,
+          0.03478051349520683,
+          -0.009426635690033436,
+          0.039094142615795135,
+          0.021130746230483055,
+          -0.0020436944905668497,
+          0.015052451752126217,
+          0.00261117797344923,
+          0.002690361812710762,
+          -0.00906465295702219,
+          -0.013936338014900684,
+          -0.0019626254215836525,
+          -0.005486302077770233,
+          0.0025489621330052614,
+          -0.03960695117712021,
+          0.0004977263743057847,
+          -0.0033276022877544165,
+          0.016998108476400375,
+          -0.035685472190380096,
+          -0.008891202509403229,
+          0.026982801035046577,
+          -0.012035928666591644,
+          0.005026282276958227,
+          -0.003996893297880888,
+          0.01653054729104042,
+          0.042231328785419464,
+          0.011455247178673744,
+          -0.003372850129380822,
+          -0.017782405018806458,
+          -0.0002785570686683059,
+          0.004988575354218483,
+          -0.03846067190170288,
+          0.004732171073555946,
+          -0.039697445929050446,
+          0.030180316418409348,
+          -0.00679094810038805,
+          -0.008333145640790462,
+          0.03541398420929909,
+          -0.018038809299468994,
+          0.01473571639508009,
+          0.009879114106297493,
+          -0.022186528891324997,
+          0.02693755365908146,
+          0.002552732825279236,
+          0.018551617860794067,
+          -0.04147719591856003,
+          0.00793345645070076,
+          -0.008295439183712006,
+          0.00950204860419035,
+          -0.01642496883869171,
+          -0.0017646661726757884,
+          0.016575796529650688,
+          -0.010127977468073368,
+          0.014449146576225758,
+          0.020135292783379555,
+          -0.001117998966947198,
+          0.0006494951667264104,
+          0.021899959072470665,
+          -0.006527002435177565,
+          0.0060330466367304325,
+          0.015489847399294376,
+          0.013868466019630432,
+          0.01346877682954073,
+          0.037827201187610626,
+          -0.002782742725685239,
+          0.0012320613022893667,
+          0.003993122838437557,
+          0.001916434965096414,
+          0.010241096839308739,
+          -0.0067796362563967705,
+          -0.001194354728795588,
+          -0.016605960205197334,
+          0.00782033707946539,
+          0.017013192176818848,
+          -0.006379947066307068,
+          -0.001372518134303391,
+          -0.030376389622688293,
+          -0.010037481784820557,
+          -0.04871685057878494,
+          0.006112230475991964,
+          -0.030919363722205162,
+          -0.010150601156055927,
+          -0.008416100405156612,
+          0.018385710194706917,
+          -0.003538758959621191,
+          0.012412993237376213,
+          0.001619496033526957,
+          -0.019984466955065727,
+          -0.005135631188750267,
+          0.008310521952807903,
+          -0.015059992671012878,
+          0.02325739525258541,
+          0.0071680136024951935,
+          -0.006440277211368084,
+          0.00266585242934525,
+          -0.013348115608096123,
+          -0.006142395548522472,
+          -0.015090158209204674,
+          -0.007017187308520079,
+          0.002581012900918722,
+          0.022654090076684952,
+          0.004143949132412672,
+          -0.0013828874798491597,
+          0.00010999705409631133,
+          -0.007752465084195137,
+          0.015791499987244606,
+          0.0027092150412499905,
+          -0.0003994536818936467,
+          0.004984804894775152,
+          0.015135405585169792,
+          0.008355769328773022,
+          -0.0059312391094863415,
+          0.008446265012025833,
+          0.006447818595916033,
+          -0.009592544287443161,
+          0.025821439921855927,
+          -0.008159696124494076,
+          0.009042028337717056,
+          -0.019712980836629868,
+          0.007639345247298479,
+          0.011093264445662498,
+          -0.014456688426434994,
+          -0.008023952133953571,
+          -0.02155306003987789,
+          0.02076876349747181,
+          0.007473436649888754,
+          0.0074885194189846516,
+          -0.024720408022403717,
+          0.014139953069388866,
+          -0.0055730268359184265,
+          -0.0013706327881664038,
+          -0.024780739098787308,
+          0.016213813796639442,
+          0.024886317551136017,
+          0.00906465295702219,
+          0.02027103677392006,
+          -0.003993122838437557,
+          0.008717752061784267,
+          -0.016017738729715347,
+          0.002801596187055111,
+          -0.012330039404332638,
+          -0.020783845335245132,
+          0.0033992447424679995,
+          0.033151593059301376,
+          -0.0016279799165204167,
+          -0.005870908498764038,
+          0.001966396113857627,
+          0.006131083704531193,
+          -0.004841519985347986,
+          -0.007707217242568731,
+          0.012247084639966488,
+          0.01344615314155817,
+          0.004649216774851084,
+          0.006006652023643255,
+          0.06690648943185806,
+          -0.004015746526420116,
+          0.004879226442426443,
+          -0.0011707880767062306,
+          -0.0031767762266099453,
+          -0.0034011302050203085,
+          -0.001456415164284408,
+          -0.02469024434685707,
+          0.019984466955065727,
+          0.0036556492559611797,
+          -0.009057111106812954,
+          0.009675499051809311,
+          -0.021085498854517937,
+          0.0006980423349887133,
+          0.007051123306155205,
+          0.011176219210028648,
+          0.001268825144506991,
+          0.016078069806098938,
+          0.03161316365003586,
+          0.015376728028059006,
+          -0.016666291281580925,
+          0.005180879030376673,
+          0.01116113644093275,
+          0.006406341679394245,
+          -0.00740933557972312,
+          0.02408693917095661,
+          0.0013781740562990308,
+          0.023815451189875603,
+          0.0019852493423968554,
+          0.020859258249402046,
+          0.010995227843523026,
+          -0.005554173607379198,
+          0.0007465895032510161,
+          -0.008582009002566338,
+          0.0064629013650119305,
+          0.012631691992282867,
+          0.0006551511469297111,
+          0.00011624219769146293,
+          0.0008257732843048871,
+          0.011372293345630169,
+          0.00801641121506691,
+          0.0018089713994413614,
+          -0.005953862797468901,
+          -0.007213261444121599,
+          -0.00876300036907196,
+          0.010957521386444569,
+          0.0025565035175532103,
+          -0.0005443881964311004,
+          -0.01063324511051178,
+          -0.002959963632747531,
+          -0.00010993813339155167,
+          0.013913714326918125,
+          0.016605960205197334,
+          0.009358763694763184,
+          -0.012654315680265427,
+          -0.004166572820395231,
+          0.0208290945738554,
+          -0.004852831829339266,
+          0.024916483089327812,
+          0.010670951567590237,
+          0.004422977566719055,
+          0.014418981969356537,
+          -0.016651209443807602,
+          -0.016349555924534798,
+          0.0030768539290875196,
+          -0.0021323049440979958,
+          0.011508036404848099,
+          0.010640786029398441,
+          -0.002104025101289153,
+          0.020406780764460564,
+          -0.007360316812992096,
+          -0.0025828981306403875,
+          0.015731168910861015,
+          0.016832200810313225,
+          -0.004215591587126255,
+          0.005275145173072815,
+          -0.015881994739174843,
+          -0.016334474086761475,
+          -0.005655981134623289,
+          0.00012619908375199884,
+          0.005810578353703022,
+          0.012873013503849506,
+          -5.217053694650531e-05,
+          0.014607514254748821,
+          -0.005870908498764038,
+          -0.025384044274687767,
+          -0.0007234942750073969,
+          0.003993122838437557,
+          0.01603282243013382,
+          -0.007088829763233662,
+          0.002594209974631667,
+          0.0016977370250970125,
+          0.021749133244156837,
+          -0.02099500223994255,
+          0.0038762325420975685,
+          0.0003674031177069992,
+          -0.01325007900595665,
+          0.0005245922366157174,
+          -0.007978703826665878,
+          -0.012458241544663906,
+          0.015331479720771313,
+          0.005561714991927147,
+          -0.013521566055715084,
+          0.018310297280550003,
+          -0.01052766665816307,
+          -0.0012150933034718037,
+          0.02199045568704605,
+          0.010648327879607677,
+          0.004362646955996752,
+          -0.015278690494596958,
+          -0.0056748343631625175,
+          -0.014871460385620594,
+          0.022382603958249092,
+          0.01708860509097576,
+          -0.004260839428752661,
+          0.003921480383723974,
+          0.013679933734238148,
+          0.0046944646164774895,
+          -0.018883436918258667,
+          -0.010648327879607677,
+          -0.0004682681174017489,
+          -0.010210931301116943,
+          0.004728400148451328,
+          0.0072396560572087765,
+          0.02010512910783291,
+          -0.016500383615493774,
+          -0.005995340179651976,
+          -0.006119771860539913,
+          -0.02930552512407303,
+          -0.029275359585881233,
+          -0.004411665257066488,
+          0.0024867465253919363,
+          -0.0024792051408439875,
+          -0.0026545405853539705,
+          -0.0019550842698663473,
+          0.002839302644133568,
+          0.0005000830278731883,
+          -0.008137071505188942,
+          0.013740263879299164,
+          -0.000598591344896704,
+          0.005554173607379198,
+          0.022126199677586555,
+          -0.002910945098847151,
+          -0.013544189743697643,
+          0.03194498270750046,
+          0.011455247178673744,
+          -0.009743370115756989,
+          -0.027344783768057823,
+          0.01211888249963522,
+          -0.011063099838793278,
+          0.017510918900370598,
+          -0.043558597564697266,
+          -0.004962180741131306,
+          -0.0038592645432800055,
+          -0.007356546353548765,
+          -0.011606073938310146,
+          -0.011002768762409687,
+          -0.016349555924534798,
+          -0.006862590555101633,
+          -0.010949979536235332,
+          -0.022231776267290115,
+          0.007963621988892555,
+          -0.005312851630151272,
+          0.02656048908829689,
+          -0.022216694429516792,
+          0.024448921903967857,
+          0.010158142074942589,
+          -0.003913938999176025,
+          0.002268048468977213,
+          -0.008589549921452999,
+          0.00490562105551362,
+          0.020074963569641113,
+          0.010014857165515423,
+          0.00011942369019379839,
+          0.021010085940361023,
+          0.008883661590516567,
+          -0.0038290992379188538,
+          -0.0038988564629107714,
+          -0.005433512851595879,
+          0.0246751606464386,
+          0.004422977566719055,
+          0.004660528618842363,
+          -0.008076741360127926,
+          0.011055557988584042,
+          0.027043132111430168,
+          0.007111453916877508,
+          -0.009336140006780624,
+          -0.005999110639095306,
+          0.0027959400322288275,
+          -0.011658863164484501,
+          -0.008416100405156612,
+          0.011093264445662498,
+          -0.01098768599331379,
+          -0.023543965071439743,
+          0.01002993993461132,
+          0.02291049435734749,
+          0.002745036268606782,
+          -0.015881994739174843,
+          -0.005640898831188679,
+          -0.0024716637562960386,
+          0.006685370113700628,
+          0.017827652394771576,
+          -0.010980145074427128,
+          -0.0144114401191473,
+          0.010542749427258968,
+          0.004057223908603191,
+          -0.00531662255525589,
+          0.02155306003987789,
+          0.030119985342025757,
+          0.011651321314275265,
+          -0.04367925971746445,
+          -0.005663522519171238,
+          0.005867138039320707,
+          -0.013823218643665314,
+          -0.047962721437215805,
+          0.004784960299730301,
+          -0.008416100405156612,
+          -0.013649768196046352,
+          0.014456688426434994,
+          -0.009562378749251366,
+          -0.014999662525951862,
+          0.02104024961590767,
+          0.026213588193058968,
+          0.0052977693267166615,
+          0.012910719960927963,
+          -0.01418520137667656,
+          -0.019909054040908813,
+          -0.020044798031449318,
+          0.00815215427428484,
+          -0.017706992104649544,
+          -0.009607627056539059,
+          0.0023151817731559277,
+          -0.013333032839000225,
+          0.02297082543373108,
+          0.012412993237376213,
+          0.03152266889810562,
+          0.009683039970695972,
+          -0.004702006001025438,
+          -0.02529354766011238,
+          -0.01719418354332447,
+          -0.002509370446205139,
+          -0.0010963176609948277,
+          0.0049697221256792545,
+          0.005456136539578438,
+          0.017510918900370598,
+          0.012963509187102318,
+          0.009517131373286247,
+          -0.013008756563067436,
+          0.015301315113902092,
+          -2.8662863769568503e-05,
+          -0.003806475317105651,
+          -0.014290779829025269,
+          -0.029561929404735565,
+          -0.0033615382853895426,
+          0.006383717525750399,
+          -0.0040911599062383175,
+          -0.004630363546311855,
+          0.0063196164555847645,
+          0.030014406889677048,
+          -0.009192855097353458,
+          0.002863811794668436,
+          -0.037435054779052734,
+          -0.003587777493521571,
+          -0.0034539191983640194,
+          0.026847057044506073,
+          0.01156836748123169,
+          -0.008755459450185299,
+          -0.006323386915028095,
+          -0.01344615314155817,
+          0.012888096272945404,
+          0.01366485096514225,
+          -0.009683039970695972,
+          0.03312142565846443,
+          0.0020116439554840326,
+          0.01396650355309248,
+          -0.0012084946502000093,
+          -0.01002993993461132,
+          0.016078069806098938,
+          0.028883211314678192,
+          -0.011281797662377357,
+          0.012767435051500797,
+          0.01190018467605114,
+          0.005512696225196123,
+          0.0017071636393666267,
+          -0.027254289016127586,
+          0.0022341127041727304,
+          -0.01021847315132618,
+          0.0214927289634943,
+          0.000284920068224892,
+          0.0015572801930829883,
+          -0.02093467302620411,
+          0.018114222213625908,
+          -0.02506730891764164,
+          -0.005060217808932066,
+          0.012443158775568008,
+          -0.0038102460093796253,
+          0.004717088304460049,
+          -0.02186979353427887,
+          -0.010535207577049732,
+          0.015007203444838524,
+          -0.018808024004101753,
+          -0.004396582953631878,
+          -0.012707104906439781,
+          -0.01553509570658207,
+          0.021356984972953796,
+          0.009750911965966225,
+          -0.018415875732898712,
+          -0.0005943493451923132,
+          0.029396019876003265,
+          -0.027239205315709114,
+          0.010753905400633812,
+          0.0180086437612772,
+          0.016711538657546043,
+          0.01344615314155817,
+          0.001777863479219377,
+          0.00884595513343811,
+          -0.0014799818163737655,
+          -0.012473324313759804,
+          -0.020014632493257523,
+          0.0196677315980196,
+          -0.01473571639508009,
+          0.019697897136211395,
+          0.010097811929881573,
+          -0.00854430254548788,
+          -0.005407118238508701,
+          -0.004815125372260809,
+          -0.003229565219953656,
+          -0.010120435617864132,
+          -0.003642451949417591,
+          0.015776416286826134,
+          0.022955743595957756,
+          -0.004068535752594471,
+          -0.018159471452236176,
+          0.017510918900370598,
+          -0.0015959293814375997,
+          0.0015139176975935698,
+          -0.0011679601157084107,
+          -0.006195184774696827,
+          0.015059992671012878,
+          0.010120435617864132,
+          -0.0002375512121943757,
+          -0.06654450297355652,
+          -0.009539755061268806,
+          -0.004287234041839838,
+          -0.004257068503648043,
+          0.002848729258403182,
+          -0.017812570556998253,
+          -0.03951645642518997,
+          0.011191301979124546,
+          0.016892530024051666,
+          -0.01471309270709753,
+          0.01592724397778511,
+          0.005678605288267136,
+          0.0017250742530450225,
+          -0.01740534044802189,
+          -0.002826105337589979,
+          0.021507810801267624,
+          0.03251812234520912,
+          -0.025052227079868317,
+          -0.01211888249963522,
+          0.012247084639966488,
+          -0.004155260976403952,
+          -0.007775088772177696,
+          -0.021326819434762,
+          -0.00029057604842819273,
+          0.01812930591404438,
+          0.02556503564119339,
+          -0.020844176411628723,
+          -0.024011526256799698,
+          4.7928548156050965e-05,
+          0.011085723526775837,
+          -0.025761108845472336,
+          -0.009924361482262611,
+          -0.015625590458512306,
+          -0.0034086713567376137,
+          0.011206384748220444,
+          -0.011870019137859344,
+          0.007624262943863869,
+          -0.012164130806922913,
+          0.005437283311039209,
+          -0.008121988736093044,
+          -0.02485615201294422,
+          0.011470329947769642,
+          -0.006055670790374279,
+          0.035444147884845734,
+          -0.006662745960056782,
+          -0.0030900510028004646,
+          0.02726937085390091,
+          -0.0008281299378722906,
+          0.005916156340390444,
+          0.014049457386136055,
+          0.03002949059009552,
+          -0.014871460385620594,
+          0.011176219210028648,
+          -0.000854524492751807,
+          -0.004882997367531061,
+          0.0005217642756178975,
+          0.03812885656952858,
+          0.004751024302095175,
+          0.0003579764743335545,
+          -0.0009945100173354149,
+          0.0015940440353006124,
+          -0.0011538201943039894,
+          0.015188194811344147,
+          0.025323713198304176,
+          -0.0012480865698307753,
+          -0.011093264445662498,
+          -0.01576133444905281,
+          -0.013868466019630432,
+          -8.383813838008791e-05,
+          0.0030881657730787992,
+          0.011085723526775837,
+          0.014056999236345291,
+          -0.006527002435177565,
+          0.01675678789615631,
+          -0.01146278902888298,
+          0.00815215427428484,
+          -0.024222683161497116,
+          0.013189747929573059,
+          -0.011666404083371162,
+          0.0006047186907380819,
+          0.0031447256915271282,
+          -0.0015261722728610039,
+          0.01325007900595665,
+          -0.00677586579695344,
+          0.004860373213887215,
+          0.012254626490175724,
+          -0.0007579015218652785,
+          0.0024414986837655306,
+          -0.021115664392709732,
+          -0.013906172476708889,
+          -0.003118331078439951,
+          0.021945208311080933,
+          -0.0030730832368135452,
+          -0.014637679792940617,
+          -4.306793925934471e-05,
+          0.004230673890560865,
+          -0.005192190874367952,
+          0.012171671725809574,
+          0.0008700784528627992,
+          -0.00031437829602509737,
+          0.020859258249402046,
+          0.0039063976146280766,
+          -0.010346675291657448,
+          0.01556526031345129,
+          -0.003041032701730728,
+          -0.0069530862383544445,
+          -0.010120435617864132,
+          0.006508149206638336,
+          0.006229120772331953,
+          -0.005177108105272055,
+          0.026364414021372795,
+          -0.03288010507822037,
+          0.0008507538586854935,
+          0.009705663658678532,
+          -0.0014102247077971697,
+          -0.0029486515559256077,
+          0.004102471750229597,
+          0.04105488210916519,
+          -0.009637792594730854,
+          -0.023830534890294075,
+          0.0002169304498238489,
+          0.01769191026687622,
+          -0.011492953635752201,
+          0.002967504784464836,
+          0.01823488436639309,
+          0.004890538286417723,
+          0.0034859697334468365,
+          0.010007316246628761,
+          -0.03704290837049484,
+          -0.004471995867788792,
+          -0.007940997369587421,
+          -0.011206384748220444,
+          0.007186866831034422,
+          0.02088942378759384,
+          0.01690761372447014,
+          0.014026833698153496,
+          0.005784183740615845,
+          0.030949529260396957,
+          0.0037970487028360367,
+          -0.003135299077257514,
+          0.0016317506087943912,
+          0.00720194960013032,
+          -0.013008756563067436,
+          0.018099140375852585,
+          -0.008137071505188942,
+          -0.015520012937486172,
+          0.004260839428752661,
+          -0.009569920599460602,
+          0.004815125372260809,
+          -0.011032934300601482,
+          0.0051243188790977,
+          -0.006904067937284708,
+          0.0005476875230669975,
+          -0.007439500652253628,
+          -0.00021657695469912142,
+          0.013076628558337688,
+          -3.543825732776895e-05,
+          0.011221466585993767,
+          0.012239543721079826,
+          -0.009057111106812954,
+          -0.024644995108246803,
+          0.012443158775568008,
+          -0.008031493052840233,
+          0.005772871430963278,
+          -0.010927355848252773,
+          0.0009073136607185006,
+          -0.0010576684726402164,
+          0.03791769966483116,
+          0.025414209812879562,
+          -0.0003365308803040534,
+          0.00593878049403429,
+          -0.028551394119858742,
+          0.007360316812992096,
+          -0.024991896003484726,
+          0.009826324880123138,
+          -0.013958961702883244,
+          -0.008287898264825344,
+          0.012171671725809574,
+          0.028460897505283356,
+          -0.0046529872342944145,
+          0.004226903431117535,
+          -0.007266050670295954,
+          -0.020150376483798027,
+          -0.007895749993622303,
+          -0.006530772894620895,
+          -0.011500495485961437,
+          -0.02055760659277439,
+          -0.010286344215273857,
+          0.01785781793296337,
+          -0.009026946499943733,
+          -0.0013678048271685839,
+          0.015286232344806194,
+          -0.0017995446687564254,
+          0.005490072537213564,
+          -0.017058439552783966,
+          0.0038837739266455173,
+          0.0062178089283406734,
+          0.01690761372447014,
+          -0.01553509570658207,
+          0.012209378182888031,
+          0.0033332582097500563,
+          -0.013091711327433586,
+          0.008106906898319721,
+          -0.021070415154099464,
+          -0.015082616358995438,
+          -0.0003834283852484077,
+          0.007695905398577452,
+          -0.010535207577049732,
+          -0.030919363722205162,
+          0.009554837830364704,
+          -0.0043777297250926495,
+          -0.007511143106967211,
+          -0.01613840088248253,
+          0.009901737794280052,
+          0.0022322272416204214,
+          0.03387555852532387,
+          -0.017284678295254707,
+          -0.006741929799318314,
+          0.019712980836629868,
+          0.009109900332987309,
+          -0.01642496883869171,
+          -0.009366304613649845,
+          0.004249527119100094,
+          -0.0012886211043223739,
+          0.006821113638579845,
+          -0.01463013794273138,
+          -0.009939444251358509,
+          0.0031145603861659765,
+          0.005637127906084061,
+          0.019471658393740654,
+          0.005324163939803839,
+          -0.0135818962007761,
+          -0.0032653864473104477,
+          0.009351222775876522,
+          0.006911609321832657,
+          0.009388929232954979,
+          -0.016349555924534798,
+          0.002688476350158453,
+          -0.012254626490175724,
+          0.002351003000512719,
+          0.011628697626292706,
+          -0.006361093837767839,
+          -0.006342240143567324,
+          0.022895412519574165,
+          0.017933230847120285,
+          -0.0018061433220282197,
+          -0.013234996236860752,
+          0.0004694464441854507,
+          -0.0002575828111730516,
+          0.01281268335878849,
+          -0.006557167507708073,
+          -0.005595650989562273,
+          0.0013838300947099924,
+          0.01576133444905281,
+          0.003944104071706533,
+          -0.005090383347123861,
+          -0.013943878933787346,
+          -0.015384268946945667,
+          0.0007640288094989955,
+          -0.027480527758598328,
+          0.0033502262085676193,
+          0.0014385045506060123,
+          0.020452028140425682,
+          0.017812570556998253,
+          0.019019179046154022,
+          -0.014358650892972946,
+          0.001570477499626577,
+          0.005007428582757711,
+          -0.00812953058630228,
+          0.002094598487019539,
+          -0.009879114106297493,
+          -0.0190040972083807,
+          -0.006504378281533718,
+          0.0058595966547727585,
+          0.018370626494288445,
+          0.004656758159399033,
+          -0.004309857729822397,
+          0.029411103576421738,
+          0.019366079941391945,
+          -0.02309148572385311,
+          0.011749358847737312,
+          -0.024554500356316566,
+          0.010301426984369755,
+          0.01962248422205448,
+          0.004592656623572111,
+          -0.016078069806098938,
+          -0.024554500356316566,
+          0.0050111995078623295,
+          -0.010301426984369755,
+          -0.008506596088409424,
+          0.008302980102598667,
+          -0.023785287514328957,
+          0.018084056675434113,
+          -0.011975597590208054,
+          0.0018268820131197572,
+          -0.010844401083886623,
+          -0.004241986200213432,
+          -0.010542749427258968,
+          -0.0031748907640576363,
+          0.007115224376320839,
+          0.01690761372447014,
+          0.005388265009969473,
+          -0.03577596694231033,
+          -0.011010310612618923,
+          0.015580343082547188,
+          -0.00520727364346385,
+          -0.0094190938398242,
+          -0.0020436944905668497,
+          0.019652649760246277,
+          -0.02164355479180813,
+          -0.006764553487300873,
+          -0.0376763753592968,
+          -0.02022578939795494,
+          -0.02755594067275524,
+          -0.028340237215161324,
+          -0.01314450055360794,
+          -0.004615280777215958,
+          0.015550177544355392,
+          0.0017986020538955927,
+          -0.008574468083679676,
+          -0.0008361425716429949,
+          -0.0108293192461133,
+          -0.0003174419398419559,
+          -0.010263720527291298,
+          -0.006417653523385525,
+          0.0033596528228372335,
+          0.007375399582087994,
+          0.002143617020919919,
+          -0.012262167409062386,
+          -0.008039034903049469,
+          -0.011809688992798328,
+          -0.010595538653433323,
+          0.00854430254548788,
+          -0.003346455516293645,
+          -0.019682815298438072,
+          0.016108235344290733,
+          0.008197402581572533,
+          0.012005763128399849,
+          0.013189747929573059,
+          0.0009275809279642999,
+          0.032095808535814285,
+          6.698803190374747e-05,
+          -0.0016854824498295784,
+          -6.112583832873497e-06,
+          -0.009026946499943733,
+          0.015376728028059006,
+          0.0077298409305512905,
+          0.008325604721903801,
+          -0.004079848062247038,
+          0.006293221842497587,
+          -0.007443271577358246,
+          0.0058181192725896835,
+          0.00018040223221760243,
+          -0.007971162907779217,
+          0.0003205056127626449,
+          0.007695905398577452,
+          0.002927913097664714,
+          -0.01418520137667656,
+          0.006817342713475227,
+          0.012699563056230545,
+          0.006934233009815216,
+          -0.01576133444905281,
+          -0.01380059402436018,
+          -0.012156588956713676,
+          -0.008499054238200188,
+          -0.028551394119858742,
+          0.022955743595957756,
+          0.043830085545778275,
+          0.000945962849073112,
+          -0.0190040972083807,
+          -0.0044380598701536655,
+          -0.015444600023329258,
+          -0.015007203444838524,
+          -0.0056748343631625175,
+          -0.005912385880947113,
+          -0.02375512197613716,
+          -0.008114447817206383,
+          0.015278690494596958,
+          0.014577348716557026,
+          0.000577381404582411,
+          -0.01570100337266922,
+          -0.005953862797468901,
+          -0.015987573191523552,
+          -0.013302868232131004,
+          -0.00908727664500475,
+          -0.021960290148854256,
+          -0.012518571689724922,
+          0.012043469585478306,
+          -0.01613840088248253,
+          0.015836747363209724,
+          0.020421862602233887,
+          0.00380836077965796,
+          -0.014109788462519646,
+          -0.015437058173120022,
+          -0.011289338581264019,
+          0.01198313944041729,
+          0.010912273079156876,
+          -0.010723740793764591,
+          0.03107019141316414,
+          -0.016440052539110184,
+          -0.009524672292172909,
+          -0.0033596528228372335,
+          0.007217032369226217,
+          -0.014260614290833473,
+          -0.004147719591856003,
+          -0.016621043905615807,
+          0.01220183726400137,
+          -0.014275697059929371,
+          0.006459130439907312,
+          0.007250967901200056,
+          -0.016334474086761475,
+          0.021326819434762,
+          0.025263383984565735,
+          0.0018579898169264197,
+          0.007635574787855148,
+          0.009962067939341068,
+          -0.013928796164691448,
+          0.010784070938825607,
+          -0.004984804894775152,
+          -0.0012575131841003895,
+          0.028129080310463905,
+          0.008536760695278645,
+          -0.0027506921906024218,
+          -0.015188194811344147,
+          0.013627144508063793,
+          0.020798929035663605,
+          -0.0023604296147823334,
+          -0.006870131939649582,
+          -0.013657309114933014,
+          0.013695016503334045,
+          0.0066363513469696045,
+          -0.00605944124981761,
+          -0.0075978683307766914,
+          -0.008989240042865276,
+          -0.0019230336183682084,
+          -0.021779298782348633,
+          0.012149048037827015,
+          -0.023830534890294075,
+          0.00014540585107170045,
+          -0.009494507685303688,
+          0.009667957201600075,
+          -0.0060028815641999245,
+          0.009939444251358509,
+          -0.010806694626808167,
+          0.013619602657854557,
+          0.007367858197540045,
+          0.0004548351571429521,
+          -0.011274255812168121,
+          0.017767323181033134,
+          0.004260839428752661,
+          0.00856692623347044,
+          0.00245281052775681,
+          -0.019170006737113,
+          0.030557380989193916,
+          -0.006625039502978325,
+          0.009434176608920097,
+          -0.010799153707921505,
+          0.012639232911169529,
+          0.021417316049337387,
+          0.028732385486364365,
+          0.011741816997528076,
+          0.0034765431191772223,
+          -0.015225901268422604,
+          0.024147270247340202,
+          0.0015846174210309982,
+          -0.010075188241899014,
+          0.02016545832157135,
+          0.004159031435847282,
+          0.005195961333811283,
+          -0.007032270077615976,
+          0.0090043218806386,
+          -0.02423776499927044,
+          -0.02551978826522827,
+          -0.007511143106967211,
+          -0.007582785561680794,
+          -0.009328598156571388,
+          0.01495441421866417,
+          -0.0017297875601798296,
+          -0.0019946759566664696,
+          0.0037178650964051485,
+          0.0032031708396971226,
+          0.007496060337871313,
+          0.002641343278810382,
+          -0.013574355281889439,
+          -0.012164130806922913,
+          -0.016711538657546043,
+          0.019215254113078117,
+          -0.008106906898319721,
+          0.001139680272899568,
+          -0.0070247286930680275,
+          0.021417316049337387,
+          0.010188307613134384,
+          0.007160472217947245,
+          0.012707104906439781,
+          -0.006877673324197531,
+          0.01729976199567318,
+          0.004649216774851084,
+          -0.012435617856681347,
+          0.015987573191523552,
+          0.026258835569024086,
+          -0.0012443158775568008,
+          -0.00364999333396554,
+          0.00190418038982898,
+          -0.04192967340350151,
+          0.00040935163269750774,
+          -0.014041916467249393,
+          0.018702445551753044,
+          -0.007888208143413067,
+          0.004988575354218483,
+          0.01459997333586216,
+          -0.005350558087229729,
+          -0.0010887763928622007,
+          0.014011750929057598,
+          0.007326381281018257,
+          0.015670839697122574,
+          0.00720194960013032,
+          0.015550177544355392,
+          -0.013657309114933014,
+          -0.03069312497973442,
+          -0.008174777962267399,
+          -0.003989351913332939,
+          0.015437058173120022,
+          0.0037103237118571997,
+          0.016228895634412766,
+          -0.030949529260396957,
+          -0.005116777960211039,
+          -0.006296992301940918,
+          0.002047465182840824,
+          0.014705551788210869,
+          -0.00354630034416914,
+          0.018038809299468994,
+          0.015489847399294376,
+          -0.008755459450185299,
+          -0.02159830741584301,
+          -0.020844176411628723,
+          0.013936338014900684,
+          0.010060105472803116,
+          -0.020361533388495445,
+          0.0020512358751147985,
+          -0.008634798228740692,
+          -0.006244203541427851,
+          -0.02963734231889248,
+          0.024283012375235558,
+          -0.003429410047829151,
+          0.01305400487035513,
+          0.011636238545179367,
+          0.010482418350875378,
+          0.013521566055715084,
+          0.02942618541419506,
+          -0.004690693691372871,
+          0.009683039970695972,
+          0.005984028335660696,
+          -0.009275808930397034,
+          -0.007164243143051863,
+          -0.009864031337201595,
+          0.015881994739174843,
+          -0.0019701668061316013,
+          0.0027412655763328075,
+          -0.003182432148605585,
+          0.017013192176818848,
+          -0.004626592621207237,
+          -0.024056773632764816,
+          -0.004204279277473688,
+          -0.001689253025688231,
+          -0.010429629124701023,
+          0.011802148073911667,
+          -0.008506596088409424,
+          -0.000715952948667109,
+          0.009735829196870327,
+          -0.004607739392668009,
+          -0.008989240042865276,
+          0.006892756093293428,
+          0.0002959963458124548,
+          -0.04117554426193237,
+          0.0005467448499985039,
+          0.004027058836072683,
+          -0.017752239480614662,
+          0.004668070003390312,
+          -0.0076280334033071995,
+          -0.010346675291657448,
+          0.0005962346913293004,
+          0.005927468184381723,
+          0.003774424782022834,
+          0.0221563633531332,
+          -0.0018711871234700084,
+          0.01438127551227808,
+          -0.006862590555101633,
+          -0.0024942876771092415,
+          -0.009750911965966225,
+          0.0135818962007761,
+          -0.020949754863977432,
+          -0.014720633625984192,
+          -0.01580658182501793,
+          -0.005007428582757711,
+          0.004577574320137501,
+          0.00458511570468545,
+          0.009539755061268806,
+          0.0013272702926769853,
+          -0.006444048136472702,
+          0.010701116174459457,
+          -0.006413882598280907,
+          -0.00032121260301209986,
+          -0.013016298413276672,
+          0.008446265012025833,
+          0.02357413060963154,
+          0.005075300578027964,
+          -0.0064930664375424385,
+          -0.013016298413276672,
+          0.01586691290140152,
+          0.03441853076219559,
+          -0.023016072809696198,
+          0.008664963766932487,
+          0.019712980836629868,
+          -0.0031371843069791794,
+          -0.002475434448570013,
+          -0.011583449319005013,
+          0.003039147239178419,
+          -0.019471658393740654,
+          0.009796159341931343,
+          0.006873902399092913,
+          -0.008227567188441753,
+          0.004882997367531061,
+          -0.015776416286826134,
+          0.0017165903700515628,
+          -0.002958078170195222,
+          -0.008710211142897606,
+          -0.005557944066822529,
+          -0.012692022137343884,
+          -0.006078294478356838,
+          0.0035180202685296535,
+          0.008393475785851479,
+          -0.004498390480875969,
+          0.01856670156121254,
+          -0.004038370680063963,
+          0.02253342978656292,
+          0.0022416538558900356,
+          -0.001372518134303391,
+          -0.009124983102083206,
+          0.0070247286930680275,
+          0.00026041080127470195,
+          -0.004769877530634403,
+          0.0017948313616216183,
+          -0.009072193875908852,
+          0.011334586888551712,
+          -0.0034312952775508165,
+          0.01752600073814392,
+          0.007009646389633417,
+          -0.008717752061784267,
+          -0.006308304611593485,
+          0.0027714308816939592,
+          -0.005712541285902262,
+          0.012322497554123402,
+          -0.004592656623572111,
+          0.01894376613199711,
+          -0.013423528522253036,
+          -0.018702445551753044,
+          0.004717088304460049,
+          0.008476430550217628,
+          0.013287785463035107,
+          -0.009532214142382145,
+          0.01002993993461132,
+          -0.0037819661665707827,
+          0.01823488436639309,
+          0.01806897483766079,
+          -0.007243426516652107,
+          0.015007203444838524,
+          -0.01962248422205448,
+          -0.0013781740562990308,
+          0.012737269513309002,
+          -0.002754462882876396,
+          -0.020361533388495445,
+          -0.011651321314275265,
+          -0.018928684294223785,
+          0.010489960201084614,
+          -0.015489847399294376,
+          0.023709872737526894,
+          0.011274255812168121,
+          0.010572914034128189,
+          -0.001968281576409936,
+          0.006198955699801445,
+          -0.0019202056573703885,
+          0.014539642259478569,
+          -0.0016835971036925912,
+          -0.004864144138991833,
+          -0.010595538653433323,
+          -0.002262392546981573,
+          -0.04057224094867706,
+          8.725529187358916e-05,
+          0.014698009938001633,
+          0.03354373946785927,
+          -0.02170388586819172,
+          0.014336027204990387,
+          -0.011651321314275265,
+          -0.0190040972083807,
+          -0.012043469585478306,
+          0.015158030204474926,
+          -0.015263608656823635,
+          0.019170006737113,
+          -0.009743370115756989,
+          -0.0047661070711910725,
+          0.001808028668165207,
+          0.01201330404728651,
+          0.003014638088643551,
+          0.0039478749968111515,
+          0.02402660809457302,
+          0.015203277580440044,
+          0.00648552505299449,
+          -0.003024064702913165,
+          0.011515578255057335,
+          0.00458511570468545,
+          -0.014863918535411358,
+          -0.005463677924126387,
+          -0.0011594761162996292,
+          -0.007639345247298479,
+          -0.010618162341415882,
+          -0.008529219776391983,
+          -0.0037800809368491173,
+          0.0008403845713473856,
+          0.018747692927718163,
+          -0.006010422948747873,
+          -0.0012942770263180137,
+          -0.012299873866140842,
+          0.012164130806922913,
+          -0.0011886986903846264,
+          0.018928684294223785,
+          -0.027359867468476295,
+          0.026741480454802513,
+          -0.03444869816303253,
+          -0.01388354878872633,
+          -0.006632580887526274,
+          -0.00015966364298947155,
+          0.011809688992798328,
+          0.0028845504857599735,
+          -0.015836747363209724,
+          -0.0008521678391844034,
+          -0.006930462550371885,
+          0.002935454249382019,
+          -0.003197514684870839,
+          0.009524672292172909,
+          0.005855826195329428,
+          -0.005659752059727907,
+          -0.021945208311080933,
+          -0.008891202509403229,
+          -0.012563819997012615,
+          -0.008355769328773022,
+          -0.013438611291348934,
+          0.013627144508063793,
+          -0.003331372980028391,
+          0.0006655204924754798,
+          0.023000990971922874,
+          -0.014283237978816032,
+          -0.01613840088248253,
+          0.015082616358995438,
+          -0.003555726958438754,
+          -0.0056446692906320095,
+          -0.01305400487035513,
+          0.007775088772177696,
+          0.0010850057005882263,
+          0.018249966204166412,
+          -0.008876119740307331,
+          0.022337354719638824,
+          -0.0045097023248672485,
+          0.012322497554123402,
+          0.018099140375852585,
+          -0.023453468456864357,
+          -0.030180316418409348,
+          0.004219362046569586,
+          -0.02793300710618496,
+          -0.0010472992435097694,
+          0.0019739374984055758,
+          -0.024946648627519608,
+          -0.014946873299777508,
+          -0.008853496052324772,
+          -0.0255047045648098,
+          -0.01630430854856968,
+          0.012918260879814625,
+          -0.0006914437399245799,
+          0.014253072440624237,
+          0.011621156707406044,
+          -0.016213813796639442,
+          0.01553509570658207,
+          -0.014418981969356537,
+          -0.02490140125155449,
+          0.0008055060170590878,
+          -0.008506596088409424,
+          0.012910719960927963,
+          0.00406476529315114,
+          0.001550681539811194,
+          0.009622709825634956,
+          -0.005320393014699221,
+          -0.0018759004306048155,
+          0.019049344584345818,
+          0.02186979353427887,
+          0.004275921732187271,
+          -0.00418542604893446,
+          0.00616879016160965,
+          -0.016651209443807602,
+          0.029275359585881233,
+          -0.01220183726400137,
+          -0.015097699128091335,
+          0.025097474455833435,
+          0.006285680457949638,
+          0.019215254113078117,
+          0.004453142639249563,
+          0.005128089804202318,
+          0.001582732074894011,
+          0.011357210576534271,
+          -0.0015261722728610039,
+          -0.01104801706969738,
+          -0.014388816431164742,
+          -0.018325379118323326,
+          0.030889200046658516,
+          0.008174777962267399,
+          0.017616497352719307,
+          -0.028672054409980774,
+          0.010799153707921505,
+          -0.008506596088409424,
+          -0.012888096272945404,
+          0.009818783961236477,
+          0.028445815667510033,
+          0.05140155926346779,
+          0.022578677162528038,
+          0.005735164973884821,
+          -0.000906370987650007,
+          0.034297872334718704,
+          -0.01303138118237257,
+          -0.009622709825634956,
+          -0.006455359980463982,
+          -0.01686236634850502,
+          0.004011976066976786,
+          0.012986132875084877,
+          -0.003627369413152337,
+          -0.008393475785851479,
+          -0.010867025703191757,
+          0.03541398420929909,
+          0.005780412815511227,
+          -0.010143060237169266,
+          0.013868466019630432,
+          -0.01956215314567089,
+          0.012216920033097267,
+          -0.017510918900370598,
+          -0.019049344584345818,
+          -0.013415987603366375,
+          0.0009813127107918262,
+          -0.02313673496246338,
+          0.006757012102752924,
+          -0.00854430254548788,
+          0.018913600593805313,
+          -0.010060105472803116,
+          -0.006017964333295822,
+          0.0034331807401031256,
+          -0.005946321878582239,
+          -0.014373733662068844,
+          0.00480758398771286,
+          -0.008023952133953571,
+          0.020783845335245132,
+          0.0005665408098138869,
+          0.008800706826150417,
+          0.0061348541639745235,
+          0.006425194907933474,
+          -0.0028298760298639536,
+          0.0012377172242850065,
+          0.012186754494905472,
+          0.015912160277366638,
+          0.009404012002050877,
+          -0.030346224084496498,
+          0.004253298044204712,
+          -0.013717640191316605,
+          0.014977038837969303,
+          0.00812953058630228,
+          0.01962248422205448,
+          0.016183648258447647,
+          0.013355657458305359,
+          -9.674084867583588e-05,
+          0.004547408781945705,
+          -0.0012961623724550009,
+          -0.0017043356783688068,
+          0.008499054238200188,
+          -0.00024226453388109803,
+          -0.0069530862383544445,
+          -0.0012047240743413568,
+          0.008031493052840233,
+          -0.008921368047595024,
+          -0.019094591960310936,
+          -0.00184950593393296,
+          -0.002364200074225664,
+          -0.01873260922729969,
+          0.0033671942073851824,
+          0.020180542021989822,
+          -0.015987573191523552,
+          -0.0016562598757445812,
+          0.01846112310886383,
+          0.02153797633945942,
+          -0.008204943500459194,
+          0.009230561554431915,
+          -0.004988575354218483,
+          -0.010693575255572796,
+          -0.02766151912510395,
+          -0.000485943048261106,
+          -0.02814416214823723,
+          0.035564810037612915,
+          -0.00886857882142067,
+          -0.018687361851334572,
+          0.012488407082855701,
+          -0.006560938432812691,
+          -0.01713385246694088,
+          -0.0068512787111103535,
+          0.005075300578027964,
+          -0.030165232717990875,
+          0.033966053277254105,
+          -0.016666291281580925,
+          -0.0009907394414767623,
+          0.020482193678617477,
+          0.05173337459564209,
+          -0.0031692348420619965,
+          -0.01746566966176033,
+          0.017254512757062912,
+          0.014426522888243198,
+          0.03188465163111687,
+          -0.01576133444905281,
+          -0.017390256747603416,
+          -0.010814236477017403,
+          -0.012873013503849506,
+          -0.011990680359303951,
+          -0.020814010873436928,
+          0.03604745492339134,
+          -0.012503488920629025,
+          -0.016726622357964516,
+          -0.0040496825240552425,
+          -0.008167237043380737,
+          -0.011289338581264019,
+          0.022654090076684952,
+          0.027450362220406532,
+          -0.0108293192461133,
+          -0.012963509187102318,
+          -0.0012235773028805852,
+          -0.023227229714393616,
+          0.02088942378759384,
+          -0.0005038536619395018,
+          -0.014698009938001633,
+          0.010867025703191757,
+          -0.0036914704833179712,
+          -0.0016949090640991926,
+          0.002579127438366413,
+          0.004751024302095175,
+          0.0569821260869503,
+          -0.015075075440108776,
+          0.002515026368200779,
+          -0.026515239849686623,
+          0.016213813796639442,
+          -0.014863918535411358,
+          -0.012149048037827015,
+          -0.013302868232131004,
+          -0.0031296429224312305,
+          -0.0158216655254364,
+          -0.02737494930624962,
+          -0.001026560552418232,
+          -0.017616497352719307,
+          0.005162025801837444,
+          -0.014464229345321655,
+          0.008069200441241264,
+          -0.024825986474752426,
+          -0.013506483286619186,
+          -0.011063099838793278,
+          -0.024373508989810944,
+          0.015082616358995438,
+          -0.030617712065577507,
+          -0.007790171541273594,
+          0.014449146576225758,
+          0.015218360349535942,
+          -0.007058664690703154,
+          -0.012179212644696236,
+          -0.0075563909485936165,
+          -0.00038366406806744635,
+          -0.05520237609744072,
+          0.002515026368200779,
+          -0.012699563056230545,
+          0.006836195942014456,
+          -0.028310071676969528,
+          0.005146943032741547,
+          0.014351109974086285,
+          0.023000990971922874,
+          0.024267930537462234,
+          0.008385934866964817,
+          0.0024490398354828358,
+          -0.01989397220313549,
+          0.02689230628311634,
+          0.005252521485090256,
+          -0.021839629858732224,
+          0.01313695963472128,
+          -0.003996893297880888,
+          -0.00015777831140439957,
+          -0.01303138118237257,
+          -0.005248750559985638,
+          0.012284791097044945,
+          0.007390482351183891,
+          -0.005791724659502506,
+          -0.007971162907779217,
+          -0.013159583322703838,
+          -0.002450925298035145,
+          0.00605944124981761,
+          0.010467336513102055,
+          0.005803036969155073,
+          0.01044471189379692,
+          -0.016847282648086548,
+          0.0029297983273863792,
+          0.005263833329081535,
+          -0.00219075009226799,
+          0.003046688623726368,
+          -0.007744923699647188,
+          0.01923033595085144,
+          -0.013461235910654068,
+          -0.00782033707946539,
+          0.00209271302446723,
+          -0.02259376086294651,
+          -0.016832200810313225,
+          0.009313516318798065,
+          -0.027902841567993164,
+          0.016500383615493774,
+          -0.002175667556002736,
+          -0.002022956032305956,
+          -0.007903290912508965,
+          0.012465782463550568,
+          0.0031635789200663567,
+          -0.01495441421866417,
+          -0.0007597868097946048,
+          0.0009223962551914155,
+          -0.028883211314678192,
+          0.011621156707406044,
+          0.015459681861102581,
+          -0.006029276177287102,
+          -0.00030047399923205376,
+          -0.029018955305218697,
+          0.009494507685303688,
+          0.0001621381234144792,
+          -0.001130253542214632,
+          0.008793165907263756,
+          0.020255954936146736,
+          0.008227567188441753,
+          -0.012759894132614136,
+          0.012827765196561813,
+          0.013393363915383816,
+          0.005806807428598404,
+          0.00752245495095849,
+          0.024644995108246803,
+          -0.0047962721437215805,
+          0.011621156707406044,
+          -0.011651321314275265,
+          0.00961516797542572,
+          -0.0104145472869277,
+          0.0004543638206087053,
+          0.016605960205197334,
+          0.012307415716350079,
+          -0.005459907464683056,
+          0.005033823195844889,
+          -0.015474764630198479,
+          0.014909166842699051,
+          -0.020633019506931305,
+          0.0008964730077423155,
+          0.012737269513309002,
+          -0.0036066307220607996,
+          0.005599421449005604,
+          0.0135818962007761,
+          0.03830984607338905,
+          0.015851831063628197,
+          0.012480865232646465,
+          0.006093377247452736,
+          -0.02616834081709385,
+          0.003171120071783662,
+          0.0023114110808819532,
+          0.006119771860539913,
+          -0.0024471546057611704,
+          0.004539867863059044,
+          -0.010459794662892818,
+          0.00020526499429252,
+          0.010210931301116943,
+          -0.017118770629167557,
+          0.012865472584962845,
+          -0.010919814929366112,
+          0.007345234509557486,
+          -0.004445601254701614,
+          -0.010919814929366112,
+          -0.021221240982413292,
+          -0.005999110639095306,
+          0.018265048041939735,
+          -0.031824320554733276,
+          0.010716198943555355,
+          0.007039811462163925,
+          0.0288228802382946,
+          -0.02892845869064331,
+          -0.011824771761894226,
+          0.0033841622062027454,
+          -0.006108460016548634,
+          -0.009215478785336018,
+          0.00837085209786892,
+          0.01767682656645775,
+          -0.009652874432504177,
+          0.01534656248986721,
+          0.010821777395904064,
+          -0.006236662156879902,
+          0.03924496844410896,
+          0.007250967901200056,
+          0.016108235344290733,
+          -0.007895749993622303,
+          0.005546632222831249,
+          0.005490072537213564,
+          0.01576133444905281,
+          0.008061658591032028,
+          -0.02021070569753647,
+          0.0027016736567020416,
+          -0.0011566481553018093,
+          0.024539416655898094,
+          0.0052562919445335865,
+          0.005595650989562273,
+          0.0020663184113800526,
+          0.00731506897136569,
+          0.015987573191523552,
+          0.015271149575710297,
+          -0.007213261444121599,
+          -0.007130307145416737,
+          0.007624262943863869,
+          0.003165464149788022,
+          -0.003484084503725171,
+          0.001214150688610971,
+          -0.012518571689724922,
+          -0.004053453449159861,
+          0.017013192176818848,
+          -0.0067381588742136955,
+          -0.001416823361068964,
+          -0.012737269513309002,
+          -0.02452433481812477,
+          0.013453694060444832,
+          0.0018674165476113558,
+          -0.011651321314275265,
+          0.025549951940774918,
+          -0.007695905398577452,
+          -0.011123429983854294,
+          -0.01625906117260456,
+          -0.016334474086761475,
+          0.0139966681599617,
+          0.018280131742358208,
+          0.0010058219777420163,
+          0.0046831523068249226,
+          -0.003197514684870839,
+          -0.0006815457600168884,
+          0.022895412519574165,
+          -0.010776530019938946,
+          0.009728288277983665,
+          -0.002498058369383216
+        ],
+        "keyphrases": [
+          "routing method",
+          "sparsely activated mixture-of-experts",
+          "load imbalance",
+          "training efficiency improvements",
+          "GLUE and SuperGLUE benchmark"
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "bd90d79e-2183-4ea8-ad44-6b8ece583bae",
+      "properties": {
+        "page_content": "6 Limitations\n\nThe expert choice method might not immediately apply to auto-regressive text generation as our current implementation takes in the past and future tokens to perform the top-k selection. One possible solution is to collect a large batch of input sequences, dispatch tokens of the same sequence into separate groups, and perform expert choice routing for each group. Another scenario where the expert choice method does not immediately apply is when the batch size becomes very small during serving or inference. A global top-k can be selected instead and we can cap the number of times each expert or token gets selected. We leave these possible improvements for future work.\n\nAnother long-standing issue with MoE has been the large memory footprint. Even though computational cost can be reduced using sparsely gated networks, the total number of parameters increases linearly or sub-linearly with the number of experts. Increasing the number of experts requires reservation of a large number of hardware devices. Therefore, dynamic (used) power is saved while static (reserved) power is not. Power saving techniques such as the ability to put hardware devices into low power states while not in use [17] can help with reducing the reserved power requirements.\n\nReferences\n\n[1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pages 3930–3939. Computer Vision Foundation / IEEE, 2020.\n\n[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. In Advances in Neural Information Processing Systems.\n\n[3] Kyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning, 2014.\n\n[4] Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems, 2021.\n\n[5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.\n\nTransformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.\n\n[6] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 933–941. JMLR.org, 2017.\n\n[7] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixtureof-experts, 2021.\n\n[8] Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan.\n\nTricks for training sparse translation models, 2021.\n\n[9] Richard L Dykstra. An iterative procedure for obtaining i-projections onto the intersection of convex sets. The annals of Probability, pages 975–984, 1985.\n\n[10] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.\n\n[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. NAS-FPN: learning scalable feature pyramid architecture for object detection. In CVPR, pages 7036–7045. Computer Vision Foundation / IEEE, 2019.\n\n[12] Sam Gross, Marc'Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision, 2017.\n\n[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.\n\n[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 630–645, Cham, 2016. Springer International Publishing.\n\n[15] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2016.\n\n[16] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.\n\n[17] Ping Huang, Zuocheng Xing, Tianran Wang, Qiang Wei, Hongyan Wang, and Guitao Fu. A brief survey on power gating design. In 2010 10th IEEE International Conference on Solid-State and Integrated Circuit Technology, pages 788–790, 2010.\n\n[18] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 103–112, 2019.\n\n[19] Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David A. Patterson. A domain-specific supercomputer for training deep neural networks. Commun. ACM, 63(7):67–78, 2020.\n\n[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.\n\n[21] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.\n\n[22] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6265–6274. PMLR, 18–24 Jul 2021.\n\n[24] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. New York, NY, USA, 2019. Association for Computing Machinery.\n\n[25] Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, Cédric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. In ICLR. OpenReview.net, 2021.\n\n[26] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.\n\n[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.\n\n[28] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models, 2021.\n\n[30] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 10435–10444, Red Hook, NY, USA, 2018. Curran Associates Inc.\n\n[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E.\n\nHinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In ICLR (Poster). OpenReview.net, 2017.\n\n[32] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 10–15 Jul 2018. [23] Min Lin, Jie Fu, and Yoshua Bengio. Conditional computation for continual learning, 2019.\n\n[29] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.\n\n[33] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.\n\n[34] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc.\n\n[35] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.\n\nGLUE: A multi-task benchmark and analysis platform for natural language understanding.\n\nIn Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, November 2018. Association for Computational Linguistics.\n\n[36] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake A. Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen. GSPMD: general and scalable parallelization for ML computation graphs. CoRR, abs/2105.04663, 2021.\n\n7 Checklist\n\n(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Yes (b) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes (c) Did you discuss any potential negative societal impacts of your work? N/A. Not any. (d) Did you describe the limitations of your work? Yes (a) Did you include the code, data, and instructions needed to reproduce the main experimental results? Yes. We include details in the experiment setup to help reproduce the main results. (b) Did you specify all the training details? Yes (c) Did you report error bars? Yes (d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? Yes (a) If your work uses existing assets, did you cite the creators? Yes (b) Did you mention the license of the assets? No. The used dataset is not released yet. (c) Did you include any new assets either in the supplemental material or as a URL? No. The dataset is not released yet.\n\n(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? No. Not using persons' data. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes. The dataset does not contain any personally identifiable information or offensive content.\n\n",
+        "title": "6 Limitations",
+        "embedding": [
+          -0.005417884327471256,
+          -0.002057158388197422,
+          -0.02763829380273819,
+          -0.01599692367017269,
+          0.03654416278004646,
+          -0.02751435525715351,
+          -0.03312699869275093,
+          0.03036494180560112,
+          0.005386899691075087,
+          0.039306219667196274,
+          0.025407399982213974,
+          -0.014332606457173824,
+          0.02131742797791958,
+          -0.005431163590401411,
+          -0.0021988023072481155,
+          0.018927184864878654,
+          -0.01384570449590683,
+          0.01158825308084488,
+          0.00927768461406231,
+          -0.02854127436876297,
+          -0.0014927954180166125,
+          -0.016377592459321022,
+          -0.013624385930597782,
+          0.010021315887570381,
+          0.019777048379182816,
+          0.013535858131945133,
+          0.02400866523385048,
+          0.020803967490792274,
+          -0.02866521291434765,
+          0.00207154406234622,
+          0.029214082285761833,
+          -0.0010053407168015838,
+          -0.03507460653781891,
+          -0.017953382804989815,
+          -0.0425640344619751,
+          -0.03293224051594734,
+          0.08179943263530731,
+          0.002959032543003559,
+          -0.00252082129009068,
+          0.022733869031071663,
+          0.005395752843469381,
+          0.00014842192467767745,
+          -0.006825472228229046,
+          -0.010499364696443081,
+          -0.03427785634994507,
+          0.003585364669561386,
+          -0.016802523285150528,
+          -0.0011071473127231002,
+          -0.025478221476078033,
+          -0.004550314974039793,
+          -0.017519596964120865,
+          -0.032755184918642044,
+          0.031161688268184662,
+          -0.0005953476065769792,
+          -0.03493295982480049,
+          0.01935211755335331,
+          -0.005988887045532465,
+          -0.025230344384908676,
+          0.07755011320114136,
+          0.0031958436593413353,
+          0.01846684142947197,
+          -0.0008542906143702567,
+          -0.007923213765025139,
+          0.007737305480986834,
+          0.025159522891044617,
+          -0.019104240462183952,
+          -0.02299945056438446,
+          0.021813182160258293,
+          -0.0003170392010360956,
+          -0.005537396762520075,
+          -0.03576511889696121,
+          0.019900986924767494,
+          -0.022840101271867752,
+          -0.016377592459321022,
+          0.003184777684509754,
+          -0.012606319040060043,
+          0.019989514723420143,
+          0.033870629966259,
+          0.016784818843007088,
+          0.008303881622850895,
+          0.008781930431723595,
+          0.04323684424161911,
+          0.00798075646162033,
+          0.0331801176071167,
+          0.03234795853495598,
+          0.006143810227513313,
+          -0.0033684722147881985,
+          -0.07485887408256531,
+          0.0303118247538805,
+          -0.0013168469304218888,
+          -0.031002338975667953,
+          -0.0057985526509583,
+          -0.0010662033455446362,
+          -0.036296285688877106,
+          -0.020910201594233513,
+          0.041926637291908264,
+          -0.0014042678521946073,
+          -0.0038487340789288282,
+          -0.024398185312747955,
+          -0.006055282894521952,
+          -0.02207876369357109,
+          6.712184404022992e-05,
+          0.009959346614778042,
+          -0.01853766292333603,
+          -0.016342181712388992,
+          -0.016891051083803177,
+          0.00800731498748064,
+          -0.007122039329260588,
+          0.039164576679468155,
+          0.08378244936466217,
+          -0.007617793511599302,
+          -0.013208306394517422,
+          -0.016165126115083694,
+          -0.03117939457297325,
+          -0.02871832810342312,
+          -0.03197614103555679,
+          -0.03842094540596008,
+          -0.03534018620848656,
+          0.011172173544764519,
+          -0.018148142844438553,
+          0.002722221426665783,
+          0.014332606457173824,
+          0.01792682334780693,
+          -0.014217520132660866,
+          -0.03310929238796234,
+          0.049292124807834625,
+          -0.018944889307022095,
+          -0.03254271671175957,
+          1.4307914852906833e-06,
+          0.008264044299721718,
+          -0.02122890017926693,
+          0.030081652104854584,
+          -0.04224533587694168,
+          -0.05980919674038887,
+          0.024309657514095306,
+          -0.011163320392370224,
+          0.013305687345564365,
+          0.05771994590759277,
+          -0.04791109636425972,
+          -0.008897015824913979,
+          -0.000875315919984132,
+          -0.010844621807336807,
+          0.012951577082276344,
+          0.01338536199182272,
+          -0.020927906036376953,
+          0.0021456859540194273,
+          -0.019918693229556084,
+          0.080949567258358,
+          -0.021512188017368317,
+          0.02303486131131649,
+          -0.03383522108197212,
+          0.02114037238061428,
+          -0.029957713559269905,
+          0.01624480076134205,
+          0.002892636926844716,
+          -0.011287258937954903,
+          -0.03027641400694847,
+          0.004176286049187183,
+          -0.02312338910996914,
+          0.009507856331765652,
+          0.005413458216935396,
+          -0.005324930418282747,
+          0.02753205969929695,
+          0.01435916405171156,
+          0.015386084094643593,
+          0.047379929572343826,
+          0.032755184918642044,
+          0.02939113788306713,
+          -0.05977378413081169,
+          0.05453295633196831,
+          0.016944168135523796,
+          0.03944786638021469,
+          -0.008379130624234676,
+          -0.000762996613048017,
+          -0.0042448947206139565,
+          0.013677502982318401,
+          0.030099358409643173,
+          0.03587135300040245,
+          -0.005559528712183237,
+          0.027018601074814796,
+          -0.023725377395749092,
+          0.018254375085234642,
+          0.02505328878760338,
+          0.005364768207073212,
+          0.01628021150827408,
+          0.012057448737323284,
+          0.03241878002882004,
+          0.017625829204916954,
+          -0.009906229563057423,
+          0.022769279778003693,
+          0.011260701343417168,
+          0.014058170840144157,
+          0.04904424771666527,
+          -0.01383685227483511,
+          0.01018066518008709,
+          -0.013624385930597782,
+          -0.002472131047397852,
+          0.01252664439380169,
+          -0.020573796704411507,
+          0.01570478267967701,
+          -0.018732424825429916,
+          -0.04143088310956955,
+          0.03289682790637016,
+          -0.015943806618452072,
+          0.008649138733744621,
+          0.03750025853514671,
+          0.0009483511094003916,
+          0.03781895712018013,
+          0.012075154110789299,
+          -0.03050658479332924,
+          -0.0330030620098114,
+          -0.0070423646830022335,
+          -0.022645341232419014,
+          -0.01435031183063984,
+          -0.0025562322698533535,
+          0.006546610500663519,
+          -0.004501624498516321,
+          0.013128631748259068,
+          -0.0165015310049057,
+          -0.01853766292333603,
+          -0.021848592907190323,
+          0.010021315887570381,
+          0.038350123912096024,
+          -0.00276205874979496,
+          -0.027992403134703636,
+          -0.011189878918230534,
+          -0.016758261248469353,
+          -0.009773438796401024,
+          0.0026248411741107702,
+          0.006887441501021385,
+          0.028983911499381065,
+          -0.01018066518008709,
+          -0.022379757836461067,
+          0.014961151406168938,
+          0.03390604257583618,
+          -0.005634776782244444,
+          -0.011703338474035263,
+          0.010410836897790432,
+          -0.024380480870604515,
+          -0.003375111846253276,
+          -0.005360341630876064,
+          0.01664317399263382,
+          -0.04606972262263298,
+          -0.009711469523608685,
+          -0.007940919138491154,
+          0.01660776324570179,
+          0.017068106681108475,
+          0.029302610084414482,
+          0.028045520186424255,
+          0.03785436972975731,
+          -0.0027664850931614637,
+          -0.0036738922353833914,
+          -0.0034083097707480192,
+          0.02021968737244606,
+          0.004603431094437838,
+          0.03339258208870888,
+          0.02314109541475773,
+          0.008029446005821228,
+          -0.02868291735649109,
+          -0.02500017359852791,
+          0.06420016288757324,
+          -0.012287620455026627,
+          -0.042811911553144455,
+          0.04486574977636337,
+          -0.02595626935362816,
+          0.03323323279619217,
+          0.03689827397465706,
+          0.011384639889001846,
+          0.0029568192549049854,
+          0.007724026683717966,
+          0.036437928676605225,
+          -0.011482019908726215,
+          -0.037146151065826416,
+          -0.0064270980656147,
+          0.017971087247133255,
+          -0.02781534753739834,
+          0.037181559950113297,
+          -0.0314449779689312,
+          0.03689827397465706,
+          0.020768556743860245,
+          0.009764585644006729,
+          -0.008724387735128403,
+          0.007768290117383003,
+          0.009463592432439327,
+          0.007197287864983082,
+          -0.041997458785772324,
+          -0.013704060576856136,
+          -0.06529790163040161,
+          -0.05779076740145683,
+          0.011030529625713825,
+          -0.020981023088097572,
+          0.002284010173752904,
+          -0.047415342181921005,
+          0.01294272392988205,
+          0.014385722577571869,
+          -0.008857178501784801,
+          0.0027930433861911297,
+          0.03305617719888687,
+          -0.01848454773426056,
+          -0.007538118865340948,
+          -0.05435590073466301,
+          -0.0238847266882658,
+          0.013190601021051407,
+          0.04320143163204193,
+          -0.04688417539000511,
+          -0.0128541961312294,
+          -0.01763468235731125,
+          0.011685633100569248,
+          -0.015439200215041637,
+          -0.03206466883420944,
+          0.01921047270298004,
+          0.0033817514777183533,
+          -0.013199454173445702,
+          -0.060198716819286346,
+          -0.010995118878781796,
+          0.0442991741001606,
+          0.009950493462383747,
+          0.004362193867564201,
+          -0.009835408069193363,
+          -0.033870629966259,
+          0.0515938401222229,
+          -0.015819868072867393,
+          -0.05612644925713539,
+          0.012022037990391254,
+          -0.0076045142486691475,
+          -0.023619143292307854,
+          -0.008795209228992462,
+          0.0637398213148117,
+          0.0008692296105436981,
+          -0.000270700576948002,
+          -0.01936982199549675,
+          -0.004446295090019703,
+          -0.013226011767983437,
+          -0.009127187542617321,
+          -0.0059800343587994576,
+          -0.052939459681510925,
+          -0.022769279778003693,
+          0.009082923643290997,
+          -0.010641008615493774,
+          -0.0031537930481135845,
+          -0.02754976600408554,
+          -0.018980301916599274,
+          -0.004264813382178545,
+          0.004647694993764162,
+          0.02662907913327217,
+          -0.042032867670059204,
+          -0.030081652104854584,
+          0.033870629966259,
+          0.01763468235731125,
+          -0.01628021150827408,
+          0.0017827230039983988,
+          0.018927184864878654,
+          -0.0012139336904510856,
+          0.03760649263858795,
+          0.015403789468109608,
+          -0.01288075465708971,
+          -0.03125021606683731,
+          -0.021813182160258293,
+          0.0017307130619883537,
+          0.01610315591096878,
+          0.024610651656985283,
+          -0.04578643664717674,
+          -0.02399095892906189,
+          0.009915082715451717,
+          -0.06324406713247299,
+          -0.028771445155143738,
+          -0.03849176689982414,
+          0.008064857684075832,
+          0.015952659770846367,
+          -0.013491595163941383,
+          -0.017776327207684517,
+          0.023371266201138496,
+          -0.009684910997748375,
+          0.002646972890943289,
+          -0.007082202006131411,
+          0.024132603779435158,
+          0.03321552649140358,
+          -0.014642452821135521,
+          -0.038350123912096024,
+          0.030028536915779114,
+          -0.00265582581050694,
+          0.011676780879497528,
+          0.046636298298835754,
+          0.014669010415673256,
+          -0.03689827397465706,
+          0.01922817900776863,
+          0.016492677852511406,
+          0.013562416657805443,
+          -0.01197777409106493,
+          0.007060070056468248,
+          -0.023548321798443794,
+          -0.010065579786896706,
+          -0.0017683373298496008,
+          -0.018289785832166672,
+          -0.014961151406168938,
+          0.03194073215126991,
+          0.006196926813572645,
+          0.002070437418296933,
+          0.0043444884940981865,
+          -0.015200176276266575,
+          -0.005333783570677042,
+          -0.008334866724908352,
+          0.016979578882455826,
+          0.01921047270298004,
+          0.01236729510128498,
+          0.013633239082992077,
+          0.01662546955049038,
+          0.027921581640839577,
+          0.03976656496524811,
+          -0.035446420311927795,
+          0.010340014472603798,
+          -0.010286898352205753,
+          -0.03333946689963341,
+          -0.0008493109489791095,
+          0.039979029446840286,
+          0.04132464900612831,
+          -0.02029050886631012,
+          0.008928000926971436,
+          0.008622581139206886,
+          -0.003306502941995859,
+          -0.004944262094795704,
+          0.02774452604353428,
+          0.044653281569480896,
+          0.036402519792318344,
+          -0.009463592432439327,
+          0.008786357007920742,
+          -0.014916887506842613,
+          0.008821767754852772,
+          0.02505328878760338,
+          -0.03436638414859772,
+          -0.0034503601491451263,
+          0.004926556721329689,
+          0.03806683421134949,
+          -0.02131742797791958,
+          -0.003974885679781437,
+          -0.04578643664717674,
+          -0.014846066012978554,
+          -0.015642814338207245,
+          0.011172173544764519,
+          0.007029085420072079,
+          -0.013110926374793053,
+          -0.007471723016351461,
+          0.00531607773154974,
+          -0.01565166562795639,
+          -0.03806683421134949,
+          -0.03208237513899803,
+          -0.04228074476122856,
+          -0.03500378131866455,
+          -0.021653832867741585,
+          -0.021441366523504257,
+          -0.05644514784216881,
+          0.03774813562631607,
+          -0.08640286326408386,
+          -0.021777771413326263,
+          0.02765599824488163,
+          -0.013898821547627449,
+          0.01338536199182272,
+          -0.03898752108216286,
+          0.0128541961312294,
+          -0.05998625233769417,
+          -0.013730619102716446,
+          0.05403720214962959,
+          0.014483102597296238,
+          0.03289682790637016,
+          0.009596383199095726,
+          0.02124660648405552,
+          -0.05499329790472984,
+          -0.027089422568678856,
+          -0.002164497971534729,
+          -0.010933149605989456,
+          0.008511921390891075,
+          -0.002127980347722769,
+          0.004437441937625408,
+          0.012995840981602669,
+          0.0012238930212333798,
+          0.010711830109357834,
+          0.009622941724956036,
+          -0.05092103034257889,
+          -0.00127700949087739,
+          -0.004291371908038855,
+          0.0258854478597641,
+          0.0031537930481135845,
+          -0.0010833556298166513,
+          -0.013721765950322151,
+          0.010021315887570381,
+          0.014784096740186214,
+          0.017351394519209862,
+          0.015518874861299992,
+          0.01065871398895979,
+          -0.04631759971380234,
+          -0.019794754683971405,
+          0.039022933691740036,
+          -0.005015084054321051,
+          0.007015806157141924,
+          -0.01853766292333603,
+          -0.005121317226439714,
+          -0.01759927161037922,
+          -0.003237894270569086,
+          0.01759927161037922,
+          -0.0014042678521946073,
+          -0.014908035285770893,
+          -0.021441366523504257,
+          -0.041041359305381775,
+          0.023707671090960503,
+          0.03392374888062477,
+          -0.0004260387213435024,
+          -0.023318149149417877,
+          0.007307947147637606,
+          -0.04231615737080574,
+          -0.010579039342701435,
+          0.004926556721329689,
+          0.020556090399622917,
+          -0.01921047270298004,
+          -0.007055643480271101,
+          0.013509300537407398,
+          0.009968198835849762,
+          -0.0295327827334404,
+          0.013217159546911716,
+          0.02475229650735855,
+          0.023743081837892532,
+          -0.026328084990382195,
+          0.002602709224447608,
+          -0.003523395396769047,
+          0.048265207558870316,
+          -0.012508939020335674,
+          -0.012535497546195984,
+          -0.0035676592960953712,
+          -0.0030453468207269907,
+          0.017811737954616547,
+          -0.03303847089409828,
+          -0.019175061956048012,
+          0.011605958454310894,
+          -0.012030890211462975,
+          -0.008560611866414547,
+          -0.005125743802636862,
+          -0.012889607809484005,
+          0.010162959806621075,
+          0.01756386086344719,
+          -0.02756747044622898,
+          -0.020857084542512894,
+          -0.0347382016479969,
+          0.04975247010588646,
+          -0.01288075465708971,
+          -0.0023415531031787395,
+          0.03426015377044678,
+          0.009490150958299637,
+          0.008109121583402157,
+          -0.018201259896159172,
+          0.03337487578392029,
+          0.02473459020256996,
+          0.014624747447669506,
+          -0.024327363818883896,
+          0.00028245814610272646,
+          0.025035584345459938,
+          0.0023282738402485847,
+          -0.017900265753269196,
+          -0.018643897026777267,
+          -0.003943901043385267,
+          0.007693042047321796,
+          -0.018998006358742714,
+          -0.010286898352205753,
+          0.004421949852257967,
+          -0.014093581587076187,
+          -0.00017318196478299797,
+          -0.014208667911589146,
+          -0.032719772309064865,
+          0.002005148446187377,
+          0.01525329239666462,
+          0.004298011306673288,
+          0.026345791295170784,
+          -0.009401623159646988,
+          -0.0028904236387461424,
+          0.009985905140638351,
+          -0.003054199507459998,
+          0.036367107182741165,
+          0.02951507642865181,
+          0.007427459117025137,
+          -0.009472444653511047,
+          -0.026328084990382195,
+          -0.014368017204105854,
+          0.005081479903310537,
+          0.008941279724240303,
+          0.001089995144866407,
+          0.02776223234832287,
+          0.026381202042102814,
+          0.014598188921809196,
+          -0.009738028049468994,
+          -0.003412736114114523,
+          0.000571002543438226,
+          -0.009286537766456604,
+          0.02671760693192482,
+          0.05623268336057663,
+          0.012269915081560612,
+          -0.032755184918642044,
+          0.009879671968519688,
+          -0.02678842842578888,
+          0.006055282894521952,
+          0.014270637184381485,
+          0.002030600095167756,
+          -0.056799259036779404,
+          0.02659366838634014,
+          0.014970004558563232,
+          0.014996563084423542,
+          -0.020857084542512894,
+          -0.05410802364349365,
+          0.000554680242203176,
+          0.006431524641811848,
+          0.0010390918469056487,
+          -0.007940919138491154,
+          -0.015456905588507652,
+          0.01718319207429886,
+          -0.0005671294638887048,
+          -0.007405327633023262,
+          0.007498281542211771,
+          0.02484082244336605,
+          -0.012588613666594028,
+          0.007825832813978195,
+          -0.01291616540402174,
+          -0.0009682697709649801,
+          0.005302798934280872,
+          0.030099358409643173,
+          0.0037115165032446384,
+          0.03493295982480049,
+          -0.003399456851184368,
+          0.016359886154532433,
+          0.028913090005517006,
+          -0.010587891563773155,
+          -0.006081840954720974,
+          0.0019055550219491124,
+          0.0035920043010264635,
+          -0.021441366523504257,
+          0.003401670139282942,
+          -0.025991681963205338,
+          -0.03406539186835289,
+          0.010189518332481384,
+          0.0030475601088255644,
+          0.009755733422935009,
+          0.015881838276982307,
+          -0.004410883877426386,
+          -0.0012902886373922229,
+          0.0005508071626536548,
+          0.015005415305495262,
+          0.0038730790838599205,
+          0.12500086426734924,
+          -0.01390767376869917,
+          0.024150308221578598,
+          0.005298372358083725,
+          0.018148142844438553,
+          0.03757108002901077,
+          0.01201318483799696,
+          0.017811737954616547,
+          0.007688615471124649,
+          0.003939474932849407,
+          0.04405129700899124,
+          -0.02103414013981819,
+          -0.01570478267967701,
+          0.016289064660668373,
+          -0.008786357007920742,
+          0.015332967042922974,
+          0.0014540646225214005,
+          -0.015766752883791924,
+          0.011667927727103233,
+          0.02029050886631012,
+          -0.010791505686938763,
+          0.02305256761610508,
+          -0.019086534157395363,
+          -0.031055456027388573,
+          0.01528870314359665,
+          -0.004957541357725859,
+          0.04858390614390373,
+          0.0051611545495688915,
+          -0.004966394044458866,
+          -0.02199023775756359,
+          -0.003468065755441785,
+          -0.02843504026532173,
+          -0.018891774117946625,
+          0.008560611866414547,
+          0.027089422568678856,
+          0.02758517675101757,
+          0.0017041548853740096,
+          -0.010056726634502411,
+          -0.007485002279281616,
+          0.025442810729146004,
+          0.0014927954180166125,
+          -0.00400365749374032,
+          0.006590874399989843,
+          -0.014474250376224518,
+          0.011659074574708939,
+          -0.014376870356500149,
+          0.003954967018216848,
+          0.0004039068298880011,
+          -0.01290731318295002,
+          0.0009544373606331646,
+          0.03569429740309715,
+          0.005404605530202389,
+          -0.01018066518008709,
+          -0.03151579946279526,
+          0.01284534391015768,
+          0.01419096253812313,
+          0.00324232061393559,
+          -0.0023039288353174925,
+          -0.0009594170260243118,
+          -0.011605958454310894,
+          0.029107850044965744,
+          -0.002613775199279189,
+          -0.004528183024376631,
+          0.0027200081385672092,
+          0.0006728091975674033,
+          -0.01940523274242878,
+          0.015828721225261688,
+          0.011260701343417168,
+          0.025389693677425385,
+          0.022362053394317627,
+          0.024097193032503128,
+          0.019830165430903435,
+          0.005900359712541103,
+          -0.005214271135628223,
+          -0.013252570293843746,
+          -0.02030821330845356,
+          0.043838828802108765,
+          0.00038813785067759454,
+          0.012863049283623695,
+          -0.000697707524523139,
+          -0.0684494823217392,
+          -0.033711280673742294,
+          0.019015712663531303,
+          0.02220270223915577,
+          0.024415891617536545,
+          0.0017849361756816506,
+          0.0028616522904485464,
+          0.022468285635113716,
+          -0.04118300601840019,
+          -0.007600088138133287,
+          0.005006231367588043,
+          -0.009074071422219276,
+          -0.004997378680855036,
+          -0.0025141816586256027,
+          0.023442087695002556,
+          -0.01660776324570179,
+          -0.012243356555700302,
+          0.034614261239767075,
+          -0.014943446032702923,
+          0.02662907913327217,
+          -0.00017829996068030596,
+          0.0008636966813355684,
+          0.012199092656373978,
+          0.013270275667309761,
+          0.03250730782747269,
+          -0.020963318645954132,
+          -0.0080781364813447,
+          0.003970459569245577,
+          -0.009622941724956036,
+          0.029037028551101685,
+          -0.009968198835849762,
+          -0.04592807963490486,
+          0.01666088029742241,
+          0.016837935894727707,
+          0.03693368285894394,
+          -0.005453295540064573,
+          0.010189518332481384,
+          -0.010782652534544468,
+          0.041749581694602966,
+          -0.005262961145490408,
+          -0.007569103501737118,
+          0.03410080075263977,
+          -0.009950493462383747,
+          0.0029014896135777235,
+          0.022344347089529037,
+          -0.011198732070624828,
+          -0.031692855060100555,
+          0.019564582034945488,
+          0.016041187569499016,
+          0.059419672936201096,
+          -0.018732424825429916,
+          0.016598910093307495,
+          0.019723933190107346,
+          -0.0020217474084347486,
+          -0.01651923544704914,
+          -0.03689827397465706,
+          0.008308308199048042,
+          -0.007077775429934263,
+          0.007489428389817476,
+          -0.015908395871520042,
+          0.03707532584667206,
+          -0.019759343937039375,
+          -0.013996201567351818,
+          0.008742093108594418,
+          -0.0036318416241556406,
+          -0.01338536199182272,
+          0.02664678543806076,
+          -0.0048822928220033646,
+          -0.03927081078290939,
+          -0.012818785384297371,
+          0.007223845925182104,
+          0.02939113788306713,
+          -0.02112266793847084,
+          0.01057018619030714,
+          0.022344347089529037,
+          -0.015518874861299992,
+          -0.02478770725429058,
+          0.01201318483799696,
+          -0.016784818843007088,
+          0.009481297805905342,
+          0.02480541169643402,
+          -0.03050658479332924,
+          -0.006852030288428068,
+          -0.028399629518389702,
+          -0.01374832447618246,
+          -0.012190240435302258,
+          0.019015712663531303,
+          0.0030187885276973248,
+          0.0060729882679879665,
+          -0.025177227333188057,
+          0.027408121153712273,
+          0.013996201567351818,
+          -0.013854557648301125,
+          -0.01834290288388729,
+          -0.010251487605273724,
+          -0.02496476098895073,
+          -0.0044662137515842915,
+          -0.02036133036017418,
+          -0.016528088599443436,
+          -7.096034096321091e-05,
+          -0.004227189347147942,
+          -0.018927184864878654,
+          0.005891506560146809,
+          -0.00015533814439550042,
+          0.004705238156020641,
+          0.06069447100162506,
+          0.007830259390175343,
+          0.02593856491148472,
+          -0.003412736114114523,
+          -0.020007221028208733,
+          0.03149809315800667,
+          -0.02018427476286888,
+          0.042882733047008514,
+          0.0008465444552712142,
+          -0.03032952919602394,
+          0.01010984368622303,
+          0.037004504352808,
+          -0.009056366048753262,
+          0.006121678277850151,
+          -0.01832519844174385,
+          0.0012006545439362526,
+          -0.013403067365288734,
+          -0.014031612314283848,
+          -0.0007010273402556777,
+          0.006639564409852028,
+          0.01563396118581295,
+          0.01521788164973259,
+          0.017811737954616547,
+          0.011358081363141537,
+          0.0008033872582018375,
+          0.02027280256152153,
+          -0.022609930485486984,
+          0.008888163603842258,
+          -0.009826554916799068,
+          0.021848592907190323,
+          0.006701533682644367,
+          0.0039372616447508335,
+          -0.011207584291696548,
+          0.004322356544435024,
+          -0.038314711302518845,
+          0.0028218147344887257,
+          0.01747533306479454,
+          0.022556813433766365,
+          -0.02595626935362816,
+          -0.04578643664717674,
+          -0.004820323549211025,
+          -0.05775535851716995,
+          -0.028293397277593613,
+          -0.010995118878781796,
+          -0.00369159784168005,
+          0.02209646999835968,
+          -0.002565084956586361,
+          0.024380480870604515,
+          -0.018148142844438553,
+          -0.014854918234050274,
+          0.01195121556520462,
+          -0.022379757836461067,
+          -0.008272897452116013,
+          0.016891051083803177,
+          0.017271719872951508,
+          -0.017971087247133255,
+          0.028045520186424255,
+          -0.024681473150849342,
+          -0.0035809383261948824,
+          -0.04398047551512718,
+          -0.01834290288388729,
+          -0.015934955328702927,
+          -0.0035411010030657053,
+          0.008604875765740871,
+          -0.011933510191738605,
+          -0.016722848638892174,
+          0.0011663500918075442,
+          0.009516708552837372,
+          -0.013633239082992077,
+          0.03346340358257294,
+          -0.01834290288388729,
+          -0.017360247671604156,
+          0.002666891785338521,
+          -0.012349589727818966,
+          -0.021777771413326263,
+          -0.02206105925142765,
+          0.01293387170881033,
+          -0.020626913756132126,
+          0.004922130145132542,
+          -0.017953382804989815,
+          0.013562416657805443,
+          -0.02671760693192482,
+          -0.01844913698732853,
+          -0.0029656721744686365,
+          -0.012925018556416035,
+          -0.05708254873752594,
+          0.019971810281276703,
+          0.05283322557806969,
+          -0.005479853600263596,
+          -0.014890329912304878,
+          0.033551931381225586,
+          -0.008600449189543724,
+          -0.024380480870604515,
+          0.02485852874815464,
+          -0.0019199406960979104,
+          0.023654554039239883,
+          -0.02505328878760338,
+          -0.0035676592960953712,
+          0.00015353992057498544,
+          0.019564582034945488,
+          -0.020927906036376953,
+          0.007405327633023262,
+          -0.013004693202674389,
+          0.022362053394317627,
+          -0.00020485822460614145,
+          0.006391687318682671,
+          0.046636298298835754,
+          -0.014031612314283848,
+          -0.00970261637121439,
+          0.04157252609729767,
+          -0.006059709005057812,
+          0.0032069096341729164,
+          -0.024433596059679985,
+          0.020502975210547447,
+          0.03619005158543587,
+          0.008914722129702568,
+          0.01560740265995264,
+          -0.03523395583033562,
+          0.028736034408211708,
+          0.004696385003626347,
+          0.028895383700728416,
+          0.029196377843618393,
+          -0.01944064348936081,
+          0.026115620508790016,
+          0.007799274753779173,
+          0.0128541961312294,
+          -0.0017174340318888426,
+          0.02018427476286888,
+          -0.018661601468920708,
+          -0.01673170179128647,
+          -0.01566937193274498,
+          0.0442991741001606,
+          -0.002087036380544305,
+          -0.010295751504600048,
+          -0.01721860282123089,
+          -0.02478770725429058,
+          0.0048778667114675045,
+          0.0029435402248054743,
+          -0.01470442209392786,
+          -0.01673170179128647,
+          -0.021972531452775,
+          0.015439200215041637,
+          0.010233782231807709,
+          -0.013527005910873413,
+          0.00711761275306344,
+          -0.002790830098092556,
+          -0.0038575867656618357,
+          -0.002963458886370063,
+          -0.01246467512100935,
+          -0.03760649263858795,
+          -0.022751573473215103,
+          -0.01515591237694025,
+          0.03333946689963341,
+          -0.010136401280760765,
+          0.020591503009200096,
+          -0.006081840954720974,
+          0.012553202919661999,
+          -0.0389166995882988,
+          0.00026378437178209424,
+          -0.015067384578287601,
+          -0.0304888803511858,
+          0.025194933637976646,
+          -0.010605597868561745,
+          0.008879310451447964,
+          0.032578129321336746,
+          -0.027124833315610886,
+          -0.024610651656985283,
+          0.01338536199182272,
+          -0.004353341180831194,
+          0.0017550581833347678,
+          -0.017625829204916954,
+          0.016944168135523796,
+          -0.01942293904721737,
+          -0.028045520186424255,
+          0.013057810254395008,
+          0.03192302584648132,
+          0.012606319040060043,
+          0.012199092656373978,
+          0.03693368285894394,
+          -0.006276601459830999,
+          -0.037181559950113297,
+          -0.006471361964941025,
+          -0.009499003179371357,
+          -0.03572971001267433,
+          -0.004537035711109638,
+          0.006435951218008995,
+          -0.003970459569245577,
+          -0.01669629104435444,
+          -0.02379619888961315,
+          -0.007940919138491154,
+          -0.012349589727818966,
+          -0.05240829288959503,
+          0.01516476459801197,
+          -0.019777048379182816,
+          -0.004286945331841707,
+          -0.03418932855129242,
+          0.05077938735485077,
+          0.02843504026532173,
+          0.007467296905815601,
+          0.006736944429576397,
+          -0.01293387170881033,
+          -0.0021423660218715668,
+          0.00507262721657753,
+          0.0032179756090044975,
+          0.01922817900776863,
+          -0.020910201594233513,
+          0.014987709932029247,
+          -0.003056412795558572,
+          0.005457721650600433,
+          -0.056693024933338165,
+          -0.020927906036376953,
+          0.010056726634502411,
+          0.005581660196185112,
+          -0.0003308716113679111,
+          0.0037911911495029926,
+          0.011783013120293617,
+          -0.01716548763215542,
+          -0.021689243614673615,
+          0.01610315591096878,
+          0.030063947662711143,
+          0.01522673387080431,
+          -0.006365128792822361,
+          0.019104240462183952,
+          0.019599994644522667,
+          -0.006865309551358223,
+          0.005993313621729612,
+          -0.01426178403198719,
+          0.009569825604557991,
+          0.010853474959731102,
+          0.0036583999171853065,
+          0.022822394967079163,
+          -0.00203502643853426,
+          -0.0013876688899472356,
+          -0.00814010575413704,
+          0.003932835068553686,
+          -0.005546249449253082,
+          -0.03025870770215988,
+          -0.014757538214325905,
+          -0.013633239082992077,
+          0.011242995969951153,
+          0.03426015377044678,
+          0.02126431092619896,
+          0.014722127467393875,
+          -0.012747963890433311,
+          -0.006807766854763031,
+          -0.001094974810257554,
+          0.006334144156426191,
+          0.003992591518908739,
+          0.01387226302176714,
+          -0.014474250376224518,
+          -0.008538479916751385,
+          0.002060478087514639,
+          0.003950540907680988,
+          -0.01197777409106493,
+          0.0385625883936882,
+          0.04635301232337952,
+          -0.02106955088675022,
+          -0.017953382804989815,
+          0.006577595137059689,
+          0.016811376437544823,
+          -0.005652482621371746,
+          -0.0037115165032446384,
+          -0.005873801186680794,
+          -0.00027249878621660173,
+          -0.01787370629608631,
+          0.015200176276266575,
+          0.01111020427197218,
+          -0.02767370454967022,
+          0.010835768654942513,
+          0.09653041511774063,
+          -0.02661137469112873,
+          0.009206862188875675,
+          -0.026345791295170784,
+          -0.011995479464530945,
+          -0.004554741084575653,
+          0.0048424554988741875,
+          0.02864750660955906,
+          0.02294633351266384,
+          0.0032179756090044975,
+          0.025389693677425385,
+          -0.0028328807093203068,
+          -0.010791505686938763,
+          0.013199454173445702,
+          -0.0006645097164437175,
+          0.013287981040775776,
+          -0.02395554818212986,
+          0.010286898352205753,
+          -0.011411197483539581,
+          -0.011358081363141537,
+          0.011437756009399891,
+          0.013757177628576756,
+          -0.0006883015157654881,
+          0.021671537309885025,
+          -0.004014723002910614,
+          0.02200794219970703,
+          0.0008731026900932193,
+          -4.9450923143012915e-06,
+          -0.04964623600244522,
+          -0.015085089951753616,
+          0.0024190146941691637,
+          -0.007932065986096859,
+          -0.02115807868540287,
+          0.001672063604928553,
+          -0.00224527926184237,
+          0.01566937193274498,
+          -0.01057018619030714,
+          0.01714778132736683,
+          0.0050682006403803825,
+          -0.010552480816841125,
+          -0.0037292218767106533,
+          0.009321948513388634,
+          -0.011783013120293617,
+          0.012473528273403645,
+          -0.009472444653511047,
+          0.004242681432515383,
+          0.023707671090960503,
+          -0.025336576625704765,
+          0.01191580481827259,
+          -0.009684910997748375,
+          0.000783468596637249,
+          0.016846787184476852,
+          -0.018714718520641327,
+          0.008317161351442337,
+          -0.0012515578418970108,
+          0.006254469510167837,
+          0.001487262430600822,
+          -0.016864493489265442,
+          0.012570908293128014,
+          -0.010853474959731102,
+          0.005417884327471256,
+          0.007245977874845266,
+          0.01778518036007881,
+          -0.014854918234050274,
+          0.010711830109357834,
+          -0.006064135581254959,
+          -0.002739926800131798,
+          -0.0035698723513633013,
+          -0.019086534157395363,
+          0.0028572259470820427,
+          0.00687416223809123,
+          -0.006099546328186989,
+          -0.0249293502420187,
+          0.002538526663556695,
+          -0.004267026670277119,
+          0.019635405391454697,
+          0.00019074915326200426,
+          -0.015279850922524929,
+          -0.007613366935402155,
+          -0.020981023088097572,
+          0.012792226858437061,
+          0.0021589649841189384,
+          0.014164404012262821,
+          0.011986627243459225,
+          -0.007781569380313158,
+          0.008188796229660511,
+          -0.024398185312747955,
+          0.006360702682286501,
+          0.02593856491148472,
+          -0.002722221426665783,
+          -0.0007989609148353338,
+          -0.004023576155304909,
+          -0.010340014472603798,
+          -0.00014330392878036946,
+          -0.007684188894927502,
+          -0.014421133324503899,
+          0.0031936303712427616,
+          0.00054444425040856,
+          -0.011251848191022873,
+          0.00483360281214118,
+          -0.012411559000611305,
+          0.015713635832071304,
+          0.01249123364686966,
+          -0.014075876213610172,
+          -0.020680028945207596,
+          0.00714859738945961,
+          -0.010003610514104366,
+          -0.0013223799178376794,
+          0.0026226278860121965,
+          0.0008360318024642766,
+          0.008742093108594418,
+          -0.026044797152280807,
+          0.003638481255620718,
+          -0.018838657066226006,
+          0.015501169487833977,
+          -0.011304965242743492,
+          0.014067023992538452,
+          -0.022698456421494484,
+          0.01689990423619747,
+          0.020520679652690887,
+          -0.019883282482624054,
+          0.0038664396852254868,
+          0.013580122031271458,
+          -0.021459072828292847,
+          0.0119423633441329,
+          0.015297556295990944,
+          0.00866684503853321,
+          -0.011384639889001846,
+          0.006307586096227169,
+          0.015315261669456959,
+          0.001169669907540083,
+          0.018608486279845238,
+          0.017121223732829094,
+          0.0027000894770026207,
+          0.01332339271903038,
+          -0.02661137469112873,
+          0.020963318645954132,
+          0.012597466818988323,
+          0.03948327526450157,
+          -0.00597560778260231,
+          -0.004873440135270357,
+          0.006157089490443468,
+          0.02204335294663906,
+          -0.0017041548853740096,
+          0.012243356555700302,
+          -0.009817702695727348,
+          -0.01810387894511223,
+          0.03142727166414261,
+          0.00804272573441267,
+          0.025124112144112587,
+          -0.010401983745396137,
+          -0.004286945331841707,
+          0.025088699534535408,
+          0.019139651209115982,
+          0.0030608391389250755,
+          -0.01010984368622303,
+          0.005670187994837761,
+          -0.0015215668827295303,
+          0.023388972505927086,
+          -0.013889968395233154,
+          0.011844982393085957,
+          0.029975419864058495,
+          0.024203425273299217,
+          -0.028116341680288315,
+          0.005807405803352594,
+          -0.012986987829208374,
+          -0.0049354094080626965,
+          -0.008649138733744621,
+          0.009375064633786678,
+          0.0030165754724293947,
+          -0.020733145996928215,
+          -0.004510477185249329,
+          -0.002430080436170101,
+          -0.0031095293816179037,
+          0.039235398173332214,
+          -0.017917970195412636,
+          0.03620775789022446,
+          0.01841372437775135,
+          0.016705144196748734,
+          -0.022468285635113716,
+          0.030719051137566566,
+          -0.016253653913736343,
+          -0.010800357908010483,
+          -0.0007917680195532739,
+          0.008339292369782925,
+          0.0066351378336548805,
+          -0.017289426177740097,
+          0.008423393592238426,
+          -0.018626190721988678,
+          0.02025509811937809,
+          -0.004899998661130667,
+          -0.00035604662843979895,
+          -0.02034362591803074,
+          0.0038332417607307434,
+          0.00898111704736948,
+          0.010410836897790432,
+          -0.007307947147637606,
+          -0.008162237703800201,
+          -0.009025380946695805,
+          -0.01733369007706642,
+          -0.017156634479761124,
+          0.011393492110073566,
+          -0.01201318483799696,
+          -0.01152628380805254,
+          0.02207876369357109,
+          -0.029762953519821167,
+          -0.022415168583393097,
+          0.018077321350574493,
+          -0.0025031156837940216,
+          -0.028948500752449036,
+          0.004541461821645498,
+          -0.019104240462183952,
+          0.004041281528770924,
+          -0.008950132876634598,
+          0.0008830620790831745,
+          0.00894570630043745,
+          -0.008733239956200123,
+          -0.03955409675836563,
+          -0.003014362184330821,
+          0.0037535671144723892,
+          -0.009136040695011616,
+          0.00711318664252758,
+          0.008011740632355213,
+          0.0008902549161575735,
+          0.013447331264615059,
+          -0.009968198835849762,
+          0.008202075026929379,
+          -0.03774813562631607,
+          0.02218499779701233,
+          0.009684910997748375,
+          -0.036437928676605225,
+          0.010295751504600048,
+          -0.00811354722827673,
+          0.010463953018188477,
+          0.02753205969929695,
+          0.00441752327606082,
+          0.020697735249996185,
+          -0.002020640764385462,
+          0.0011962282005697489,
+          0.011862688697874546,
+          -0.0033618328161537647,
+          0.014642452821135521,
+          -0.01619168370962143,
+          0.029338020831346512,
+          -0.0011873753974214196,
+          0.00014551711501553655,
+          0.032737478613853455,
+          -0.003027641447260976,
+          -0.02577921561896801,
+          0.009879671968519688,
+          -0.013766029849648476,
+          -0.030966928228735924,
+          0.030665934085845947,
+          -0.03119709901511669,
+          0.0013046744279563427,
+          0.00852077454328537,
+          -0.006546610500663519,
+          -0.0005032236222177744,
+          -0.02501787804067135,
+          -0.030117064714431763,
+          0.010039021261036396,
+          0.014571630395948887,
+          0.011304965242743492,
+          0.018183553591370583,
+          -0.027018601074814796,
+          0.013004693202674389,
+          -0.008073709905147552,
+          0.006303159520030022,
+          0.009587530978024006,
+          0.012119418010115623,
+          -0.019175061956048012,
+          0.025991681963205338,
+          -0.008432246744632721,
+          0.02397325448691845,
+          0.0037292218767106533,
+          -0.0030741184018552303,
+          -0.00026378437178209424,
+          -0.03240107372403145,
+          0.006241190247237682,
+          -0.006188074126839638,
+          0.003757993457838893,
+          0.0797455906867981,
+          0.0011641369201242924,
+          -0.0020881430245935917,
+          0.022733869031071663,
+          -0.009375064633786678,
+          -0.011030529625713825,
+          -0.015297556295990944,
+          -0.006347423419356346,
+          -0.013527005910873413,
+          -0.024628357961773872,
+          -0.006263322196900845,
+          -0.004260387271642685,
+          -0.005957902409136295,
+          0.01841372437775135,
+          0.0006362915737554431,
+          -0.028222573921084404,
+          0.0002011234755627811,
+          -0.010826916433870792,
+          -0.010844621807336807,
+          0.017953382804989815,
+          0.0037380747962743044,
+          0.007608940824866295,
+          -0.00400365749374032,
+          -0.01758156716823578,
+          -0.029338020831346512,
+          -0.026947777718305588,
+          -0.0002370877773500979,
+          -0.006980395410209894,
+          -0.0016432921402156353,
+          0.001767230685800314,
+          -0.023530615493655205,
+          -0.030028536915779114,
+          0.02115807868540287,
+          -0.015412641689181328,
+          0.005550675559788942,
+          0.011384639889001846,
+          -0.00044540411909110844,
+          1.79648632183671e-05,
+          0.020609207451343536,
+          -0.04808815196156502,
+          -0.015040826052427292,
+          -0.010809211060404778,
+          -0.008653565309941769,
+          -0.016775965690612793,
+          0.015377230942249298,
+          -0.010198370553553104,
+          -0.003839881392195821,
+          -0.014067023992538452,
+          -0.002136833034455776,
+          -0.0013688568724319339,
+          -0.030170179903507233,
+          -0.012668288312852383,
+          -0.015430347062647343,
+          0.031108571216464043,
+          0.008861605077981949,
+          -0.000989848398603499,
+          -0.0019409660017117858,
+          -0.0161562729626894,
+          0.017032695934176445,
+          -0.03788977861404419,
+          0.008923574350774288,
+          -0.008100268431007862,
+          0.028222573921084404,
+          0.02760288119316101,
+          -0.017935676500201225,
+          0.0038819320034235716,
+          -0.003523395396769047,
+          0.003890784690156579,
+          -0.0087730772793293,
+          -0.021848592907190323,
+          0.010782652534544468,
+          0.0102072237059474,
+          0.002379177138209343,
+          0.030612817034125328,
+          0.02590315416455269,
+          -0.001249344670213759,
+          0.014093581587076187,
+          -0.009313095360994339,
+          0.006958263460546732,
+          -0.02108725719153881,
+          0.020042631775140762,
+          -0.008688976988196373,
+          0.0028572259470820427,
+          -0.008476510643959045,
+          0.013978496193885803,
+          -0.014642452821135521,
+          -0.010933149605989456,
+          0.04061642661690712,
+          -0.018697014078497887,
+          -0.011729896999895573,
+          -0.013987348414957523,
+          0.00393504835665226,
+          0.00924227386713028,
+          0.008750945329666138,
+          0.030010830610990524,
+          -0.002717795083299279,
+          0.012314178980886936,
+          0.001447424991056323,
+          0.003954967018216848,
+          -0.017794031649827957,
+          -0.010100990533828735,
+          0.005147875752300024,
+          0.00666612247005105,
+          0.004992952570319176,
+          -0.000783468596637249,
+          -0.011304965242743492,
+          0.0025119686033576727,
+          0.00021924394241068512,
+          -0.018254375085234642,
+          0.003118382068350911,
+          -0.03284371271729469,
+          0.014952299185097218,
+          0.0020637980196624994,
+          0.01794452965259552,
+          0.009746880270540714,
+          -0.026434319093823433,
+          0.018201259896159172,
+          -0.001513820723630488,
+          0.00762664619833231,
+          0.027248771861195564,
+          -0.031126277521252632,
+          -0.012340736575424671,
+          0.014908035285770893,
+          0.022432874888181686,
+          0.013491595163941383,
+          0.008246338926255703,
+          -0.027018601074814796,
+          0.026345791295170784,
+          -0.015554285608232021,
+          0.00805157795548439,
+          -0.02204335294663906,
+          -0.007706320844590664,
+          -0.032631244510412216,
+          -0.00017110710905399173,
+          -0.0017550581833347678,
+          0.013465036638081074,
+          -0.0023836034815758467,
+          -0.02126431092619896,
+          0.0034105228260159492,
+          0.007878949865698814,
+          -0.014368017204105854,
+          -0.008883737027645111,
+          0.0017616976983845234,
+          -0.00552411749958992,
+          0.0007784889312461019,
+          0.008879310451447964,
+          -0.0035477406345307827,
+          -0.03034723550081253,
+          -0.010756094008684158,
+          0.0027023027651011944,
+          0.008897015824913979,
+          -0.0033640458714216948,
+          0.014270637184381485,
+          -0.011827277019619942,
+          0.0018646109383553267,
+          -0.003215762320905924,
+          0.0109862657263875,
+          0.0019863364286720753,
+          -0.008852752856910229,
+          0.0013998415088281035,
+          -0.00798075646162033,
+          -0.0276205874979496,
+          0.015023120678961277,
+          -0.00504606869071722,
+          -0.006493493914604187,
+          -0.0002827347780112177,
+          0.0015337393851950765,
+          -0.005152301862835884,
+          -0.013925379142165184,
+          0.011774160899221897,
+          0.0018491186201572418,
+          0.0014507448067888618,
+          -0.0036318416241556406,
+          -0.004138661548495293,
+          -0.002044985769316554,
+          0.01751074381172657,
+          0.02774452604353428,
+          -0.018626190721988678,
+          0.010490511544048786,
+          -0.01767009310424328,
+          0.00048026180593296885,
+          -0.001060117152519524,
+          0.01763468235731125,
+          0.011375786736607552,
+          0.02114037238061428,
+          0.01516476459801197,
+          -0.008719961158931255,
+          0.010384278371930122,
+          -0.008166664279997349,
+          0.015872985124588013,
+          -0.009348507039248943,
+          -0.0007602301193401217,
+          -0.0009821022395044565,
+          0.02209646999835968,
+          -0.018980301916599274,
+          -0.0017284998903051019,
+          0.009499003179371357,
+          0.022450579330325127,
+          -0.016908757388591766,
+          -0.011242995969951153,
+          0.031108571216464043,
+          0.003414949169382453,
+          0.006975968834012747,
+          0.017236309126019478,
+          0.03229483962059021,
+          0.006307586096227169,
+          0.012252209708094597,
+          0.007843539118766785,
+          0.015872985124588013,
+          0.011340375989675522,
+          -0.010800357908010483,
+          -0.029143260791897774,
+          0.018767835572361946,
+          -0.0001229010958923027,
+          0.00265803886577487,
+          -0.00462113693356514,
+          0.008958985097706318,
+          0.012677141465246677,
+          -0.011685633100569248,
+          0.009206862188875675,
+          -0.013040103949606419,
+          0.017678946256637573,
+          0.01843143068253994,
+          0.0008891483303159475,
+          -0.02121119573712349,
+          0.007431885693222284,
+          0.02103414013981819,
+          0.0035123294219374657,
+          -0.004178499337285757,
+          -0.003775698831304908,
+          -0.01669629104435444,
+          -0.0017550581833347678,
+          0.0016034548170864582,
+          0.03218860924243927,
+          -0.00483360281214118,
+          -0.008383556269109249,
+          0.004625563044101,
+          0.00016543580568395555,
+          0.007883376441895962,
+          0.015412641689181328,
+          0.004811470862478018,
+          0.005701172631233931,
+          -0.012376148253679276,
+          0.011667927727103233,
+          0.024451302364468575,
+          0.011154468171298504,
+          0.0030608391389250755,
+          -0.0005566168110817671,
+          -0.000995934708043933,
+          0.016528088599443436,
+          -0.015014268457889557,
+          -0.006582021247595549,
+          -0.007538118865340948,
+          0.019033417105674744,
+          -0.0024499993305653334,
+          -0.006679401732981205,
+          0.014270637184381485,
+          -0.0009505642810836434,
+          0.004501624498516321,
+          -0.010375426150858402,
+          0.01335880346596241,
+          0.008852752856910229,
+          0.014137845486402512,
+          -0.010242634452879429,
+          -0.018024204298853874,
+          0.005386899691075087,
+          -0.0026602521538734436,
+          -0.021724654361605644,
+          0.007737305480986834,
+          -0.010800357908010483,
+          0.008233060128986835,
+          -0.0007657631067559123,
+          0.00255401898175478,
+          0.0014551711501553655,
+          0.0009533307747915387,
+          -0.0058472431264817715,
+          0.007276962511241436,
+          -0.013465036638081074,
+          -0.0003488537622615695,
+          -0.007347784470766783,
+          -0.028753740713000298,
+          0.0026735311839729548,
+          -5.22865702805575e-05,
+          0.005807405803352594,
+          0.009383917786180973,
+          0.00576314190402627,
+          0.003853160422295332,
+          0.014120140112936497,
+          -0.027868464589118958,
+          -0.0041231694631278515,
+          -0.00735221104696393,
+          0.02032591961324215,
+          0.012544349767267704,
+          -0.01053477544337511,
+          -0.007847964763641357,
+          0.0048778667114675045,
+          0.011783013120293617,
+          -0.0097291748970747,
+          0.008365850895643234,
+          0.02777993679046631,
+          -0.009844260290265083,
+          -0.02131742797791958,
+          0.0007956410991027951,
+          0.0042448947206139565,
+          0.01851995848119259,
+          0.025141816586256027,
+          0.01249123364686966,
+          -0.0028793576639145613,
+          -0.009897377341985703,
+          -0.013305687345564365,
+          0.029160967096686363,
+          -0.006887441501021385,
+          -0.01245582289993763,
+          -0.011871540918946266,
+          0.010676419362425804,
+          -0.026930073276162148,
+          0.0032046963460743427,
+          -0.005621497984975576,
+          -0.0028793576639145613,
+          -0.004366619978100061,
+          -0.007631072774529457,
+          -0.006891867611557245,
+          -0.018785540014505386,
+          -0.005431163590401411,
+          0.012473528273403645,
+          0.006834324914962053,
+          0.006236764136701822,
+          -0.028311101719737053,
+          -0.009587530978024006,
+          -0.015270997770130634,
+          -0.017103517428040504,
+          -0.00213904632255435,
+          -0.031055456027388573,
+          -0.03293224051594734,
+          -0.017678946256637573,
+          -0.014784096740186214,
+          -6.719100929331034e-05,
+          -0.004089971538633108,
+          -0.01467786356806755,
+          0.0022884365171194077,
+          0.009684910997748375,
+          0.01335880346596241,
+          0.0109862657263875,
+          0.0015591910341754556,
+          -0.022326640784740448,
+          0.012553202919661999,
+          -0.017555007711052895,
+          -0.023672260344028473,
+          0.004603431094437838,
+          0.011313817463815212,
+          -0.0026204148307442665,
+          -0.0018491186201572418,
+          0.004492771811783314,
+          0.006050856318324804,
+          0.005223123822361231,
+          -0.044830337166786194,
+          -0.009525561705231667,
+          0.0070423646830022335,
+          0.0040302155539393425,
+          -0.0049708206206560135,
+          -0.013403067365288734,
+          0.012066301889717579,
+          -0.01659005880355835,
+          -0.027886170893907547,
+          -0.009906229563057423,
+          -0.016510384157299995,
+          0.002213188214227557,
+          -0.009321948513388634,
+          -0.0011386852711439133,
+          0.03220631182193756,
+          0.017448775470256805,
+          0.005417884327471256,
+          0.0003460872976575047,
+          0.003917342983186245,
+          0.008715534582734108,
+          -0.010481659322977066,
+          0.0014363591326400638,
+          0.008299455046653748,
+          0.01290731318295002,
+          0.005878227762877941,
+          0.0020394527819007635,
+          -0.013341098092496395,
+          0.025584453716874123,
+          0.01467786356806755,
+          0.01010984368622303,
+          0.0006910680094733834,
+          0.008109121583402157,
+          0.006732518319040537,
+          0.023760788142681122,
+          0.0040589869022369385,
+          0.00846765749156475,
+          -0.007936492562294006,
+          0.0068166195414960384,
+          -0.013562416657805443,
+          -0.021441366523504257,
+          0.015359525568783283,
+          -0.0060729882679879665,
+          -0.03436638414859772,
+          0.02032591961324215,
+          -0.007077775429934263,
+          0.002996656810864806,
+          -0.008866031654179096,
+          -0.011375786736607552,
+          -0.00276205874979496,
+          0.017032695934176445,
+          0.0061659421771764755,
+          -0.006086267530918121,
+          0.0008327120449393988,
+          0.0073212264105677605,
+          -0.0027155817952007055,
+          -0.018838657066226006,
+          -0.0022076552268117666,
+          -0.004240468610078096,
+          0.004306863993406296,
+          -0.053683090955019,
+          -0.016811376437544823,
+          0.012482380494475365,
+          -0.009003248997032642,
+          -0.01747533306479454,
+          0.009401623159646988,
+          -0.008848326280713081,
+          -0.018767835572361946,
+          0.004687532316893339,
+          0.008812914602458477,
+          0.002602709224447608,
+          0.0020660110749304295,
+          0.007502707652747631,
+          0.021405955776572227,
+          0.01153513602912426,
+          -0.016722848638892174,
+          0.0021600716281682253,
+          -0.012677141465246677,
+          -0.000435168098192662,
+          -0.019830165430903435,
+          -0.0169353149831295,
+          -0.00480261817574501,
+          0.010437395423650742,
+          0.04479492828249931,
+          0.013491595163941383,
+          0.010685272514820099,
+          -0.0007209460018202662,
+          0.0005815151962451637,
+          0.001700835069641471,
+          -0.04139547049999237,
+          -0.0019719505216926336,
+          -0.005165581125766039,
+          -0.005692319944500923,
+          -0.010835768654942513,
+          0.012951577082276344,
+          0.028169458732008934,
+          0.02868291735649109,
+          -0.002775337779894471,
+          0.0073831956833601,
+          -2.913454591180198e-05,
+          0.00390185066498816,
+          0.008056004531681538,
+          -0.013075515627861023,
+          0.015085089951753616,
+          -0.0016012416454032063,
+          0.009791144169867039,
+          0.009782291017472744,
+          -0.0008526307065039873,
+          0.014491955749690533,
+          -0.015846427530050278,
+          0.010764947161078453,
+          -0.00964064709842205,
+          0.004149727523326874,
+          0.008998822420835495,
+          0.03764190524816513,
+          0.002179990289732814,
+          -0.01659005880355835,
+          -0.01566937193274498,
+          -0.030063947662711143,
+          0.029692132025957108,
+          -0.013270275667309761,
+          0.016997285187244415,
+          0.017245162278413773,
+          -0.0015647240215912461,
+          0.004930983297526836,
+          0.010463953018188477,
+          0.0017318197060376406,
+          0.011375786736607552,
+          0.014509661123156548,
+          0.00969376415014267,
+          -0.023300444707274437,
+          -0.02013115957379341,
+          0.001953138504177332,
+          -0.015262145549058914,
+          -0.014341458678245544,
+          -0.007679762784391642,
+          -0.011189878918230534,
+          0.0072326986119151115,
+          -0.014854918234050274,
+          0.021724654361605644,
+          -0.016829082742333412,
+          0.0049752467311918736,
+          -0.02944425493478775,
+          -0.006621859036386013,
+          0.0034459338057786226,
+          -0.012066301889717579,
+          -0.00028384136385284364,
+          -0.008308308199048042,
+          -0.002790830098092556,
+          0.017740916460752487,
+          -0.004415310453623533,
+          -0.003992591518908739,
+          0.011136762797832489,
+          0.006701533682644367,
+          0.010100990533828735,
+          -0.015642814338207245,
+          0.005550675559788942,
+          -0.0037646330893039703,
+          0.014005054719746113,
+          -0.018679307773709297,
+          -0.004740648902952671,
+          0.0027664850931614637,
+          0.024327363818883896,
+          0.012889607809484005,
+          0.008941279724240303,
+          0.005895933136343956,
+          -0.007852391339838505,
+          -0.009136040695011616,
+          -0.03739402815699577,
+          0.030860694125294685,
+          -0.016147419810295105,
+          0.022822394967079163,
+          -0.003609709907323122,
+          0.0025119686033576727,
+          -0.0006523372139781713,
+          0.00687416223809123,
+          0.0042448947206139565,
+          -0.021494483575224876,
+          0.013102073222398758,
+          -0.0231942106038332,
+          0.008184369653463364,
+          0.003359619528055191,
+          -0.012588613666594028,
+          0.03693368285894394,
+          -0.002405735431239009,
+          0.014784096740186214,
+          -0.02855897881090641,
+          0.0024079487193375826,
+          0.009764585644006729,
+          -0.005590513348579407,
+          -0.005453295540064573,
+          -0.02942654862999916,
+          -0.0010999544756487012,
+          0.0009915082482621074,
+          0.00035881312214769423,
+          0.006944984197616577,
+          -0.01922817900776863,
+          -0.03128562867641449,
+          0.008184369653463364,
+          0.008688976988196373,
+          0.002562871901318431,
+          0.003731435164809227,
+          -0.0001748418580973521,
+          -0.0018535450799390674,
+          0.014987709932029247,
+          0.01242926437407732,
+          0.019706226885318756,
+          0.03284371271729469,
+          -0.02862980216741562,
+          -0.015713635832071304,
+          -0.002653612522408366,
+          -0.016368739306926727,
+          -0.02569068782031536,
+          -0.02222040854394436,
+          0.0044706398621201515,
+          0.0073212264105677605,
+          0.013562416657805443,
+          -0.030896106734871864,
+          0.013287981040775776,
+          -0.005621497984975576,
+          0.004984099417924881,
+          -0.03032952919602394,
+          -0.026345791295170784,
+          -0.006254469510167837,
+          -0.011765307746827602,
+          0.03208237513899803,
+          -0.010021315887570381,
+          0.004275879357010126,
+          -0.01522673387080431,
+          -0.013456183485686779,
+          -0.00711761275306344,
+          0.0019420725293457508,
+          -0.01335880346596241,
+          -0.012296473607420921,
+          0.038243889808654785,
+          -0.001566937193274498,
+          -0.014598188921809196,
+          0.019723933190107346,
+          0.005546249449253082,
+          0.005501985549926758,
+          0.0040943981148302555,
+          0.013252570293843746,
+          -0.008441099897027016,
+          -0.006404966581612825,
+          -0.0004611730691976845,
+          0.007299094460904598,
+          0.002870504977181554,
+          0.024699179455637932,
+          -0.0042515345849096775,
+          2.7509237042977475e-05,
+          -0.007728452794253826,
+          0.0005483173299580812,
+          0.01844913698732853,
+          0.005422310903668404,
+          -0.007051217369735241,
+          -0.0025717245880514383,
+          0.005298372358083725,
+          -0.03227713704109192,
+          -0.017749767750501633,
+          0.01713007688522339,
+          0.010065579786896706,
+          0.009950493462383747,
+          -0.00028065990773029625,
+          -0.03149809315800667,
+          -0.007901081815361977,
+          0.002615988254547119,
+          0.0026624652091413736,
+          -0.0015160338953137398,
+          0.018820952624082565,
+          -0.027301888912916183,
+          0.002101422054693103,
+          -0.0032511733006685972,
+          0.02108725719153881,
+          0.008445525541901588,
+          0.0005848349537700415,
+          0.024504419416189194,
+          0.0036561868619173765,
+          0.013704060576856136,
+          -0.0018258801428601146,
+          -0.010322309099137783,
+          -0.01747533306479454,
+          -0.010003610514104366,
+          -3.72092254110612e-05,
+          0.0010119802318513393,
+          0.001832519774325192,
+          -0.00021232773724477738,
+          0.0015082877362146974,
+          -0.0038885714020580053,
+          -0.0011630303924903274,
+          0.015970366075634956,
+          0.0014817294431850314,
+          -0.004446295090019703,
+          0.014111286960542202,
+          -0.002376964082941413,
+          -0.007631072774529457,
+          0.006838751491159201,
+          0.0003211889124941081,
+          0.003240107325837016,
+          0.009224568493664265,
+          0.011844982393085957,
+          -0.0016045613447204232,
+          0.020715441554784775,
+          -0.010472806170582771,
+          0.008480937220156193,
+          -0.008888163603842258,
+          -0.0004888379480689764,
+          -0.017519596964120865,
+          0.0034060964826494455,
+          -0.002910342300310731,
+          -0.01423522550612688,
+          -0.020485268905758858,
+          0.006931704934686422,
+          0.006577595137059689,
+          0.00115639076102525,
+          -0.003638481255620718,
+          0.014872624538838863,
+          -0.0038332417607307434,
+          0.0045591676607728004,
+          -0.004886719398200512,
+          -0.006958263460546732,
+          -0.016439560800790787,
+          0.01518247090280056,
+          0.0015492317033931613,
+          0.005307225044816732,
+          0.002216507913544774,
+          0.0034038834273815155,
+          -0.00872881431132555,
+          0.001222786377184093,
+          0.017254015430808067,
+          0.005462148226797581,
+          -0.009994757361710072,
+          0.004408670589327812,
+          -0.009224568493664265,
+          -0.004098824225366116,
+          0.0075115603394806385,
+          -0.009339653886854649,
+          -0.01057018619030714,
+          0.02678842842578888,
+          -0.005355915054678917,
+          -0.0005236956640146673,
+          -0.012659436091780663,
+          -0.004220549948513508,
+          0.0115971053019166,
+          -0.000843777961563319,
+          -0.009251126088202,
+          -0.004705238156020641,
+          0.008958985097706318,
+          -0.01152628380805254,
+          -0.003518969053402543,
+          0.014580483548343182,
+          -0.009853113442659378,
+          -0.009560972452163696,
+          0.02021968737244606,
+          0.0026890235021710396,
+          -0.011650222353637218,
+          -0.006834324914962053,
+          0.0004075032484252006,
+          -0.005581660196185112,
+          0.03217090293765068,
+          0.006772355642169714,
+          0.00970261637121439,
+          0.023601438850164413,
+          0.001833626301959157,
+          0.009543267078697681,
+          -0.0042161233723163605,
+          0.012615172192454338,
+          0.0036716791801154613,
+          0.00873766653239727,
+          0.002321634441614151,
+          0.021724654361605644,
+          -0.0025009026285260916,
+          0.0042161233723163605,
+          0.007772716693580151,
+          -0.011482019908726215,
+          -0.004132022149860859,
+          -0.004036854952573776,
+          -0.005453295540064573,
+          -0.00089689448941499,
+          0.0005737690371461213,
+          0.010880032554268837,
+          0.009950493462383747,
+          -0.0015060745645314455,
+          0.015864131972193718,
+          0.010428542271256447,
+          -0.01422637328505516,
+          -0.0011630303924903274,
+          0.0050682006403803825,
+          0.00449055852368474,
+          0.0009666099213063717,
+          -0.030595112591981888,
+          -0.01291616540402174,
+          0.003156006336212158,
+          -0.015563138760626316,
+          0.015864131972193718,
+          -0.019847871735692024,
+          -0.02379619888961315,
+          -0.019670816138386726,
+          0.0038841450586915016,
+          -0.015005415305495262,
+          -0.01374832447618246,
+          0.021795475855469704,
+          0.007839112542569637,
+          0.0015768965240567923,
+          -0.02122890017926693,
+          0.023442087695002556,
+          -0.009808849543333054,
+          0.023229623213410378,
+          -0.01200433261692524,
+          -0.002098102355375886,
+          -0.011765307746827602,
+          0.01561625488102436,
+          -0.010897737927734852,
+          0.0025163949467241764,
+          0.009516708552837372,
+          0.01659005880355835,
+          0.0030652654822915792,
+          0.0007928746636025608,
+          0.0017882559914141893,
+          0.00480261817574501,
+          0.014438839629292488,
+          0.006834324914962053,
+          0.009410476312041283,
+          -0.01249123364686966,
+          -0.010242634452879429,
+          0.004078905563801527,
+          0.005931344348937273,
+          -0.00015644473023712635,
+          -0.02207876369357109,
+          0.002936900593340397,
+          0.009897377341985703,
+          -0.0005375280743464828,
+          -0.01839601993560791,
+          -0.006210205610841513,
+          -0.009534413926303387,
+          0.027921581640839577,
+          0.0097291748970747,
+          -0.01855536922812462,
+          -0.008950132876634598,
+          0.021565305069088936,
+          0.012411559000611305,
+          0.015235587023198605,
+          -0.016165126115083694,
+          0.013668649829924107,
+          0.008264044299721718,
+          0.0013511513825505972,
+          0.0029546061996370554,
+          -0.01053477544337511,
+          0.028824562206864357,
+          0.0048822928220033646,
+          0.0009881885489448905,
+          -0.013491595163941383,
+          -0.007538118865340948,
+          -0.008264044299721718,
+          0.011455461382865906,
+          0.005957902409136295,
+          0.002804109361022711,
+          0.001394308521412313,
+          -0.007985183037817478,
+          -0.015368377789855003,
+          -0.0019985088147222996,
+          -0.003357406472787261,
+          0.0211934894323349,
+          -0.0032179756090044975,
+          0.003782338462769985,
+          -0.006152662914246321,
+          0.0018004284938797355,
+          -0.009968198835849762,
+          -0.0057188780046999454,
+          -0.0020803967490792274,
+          0.00804272573441267,
+          -0.018218964338302612,
+          0.015979217365384102,
+          -0.020857084542512894,
+          0.018006497994065285,
+          0.028275690972805023,
+          0.00324232061393559,
+          -0.006785634905099869,
+          -0.002070437418296933,
+          0.008472084067761898,
+          -0.034649673849344254,
+          0.003306502941995859,
+          0.017696652561426163,
+          -0.006909573450684547,
+          -0.013615533709526062,
+          -0.018856363371014595,
+          -0.0012172535061836243,
+          -0.007927639409899712,
+          0.005563954822719097,
+          0.005856095813214779,
+          -0.0010518176713958383,
+          0.018661601468920708,
+          -0.002057158388197422,
+          -0.01467786356806755,
+          -0.008131253533065319,
+          -0.0221495870500803,
+          0.01480180211365223,
+          -0.012252209708094597,
+          -0.013465036638081074,
+          0.012818785384297371,
+          0.009746880270540714,
+          -0.019936397671699524,
+          0.002547379583120346,
+          -0.02747894451022148,
+          -0.008480937220156193,
+          -0.005369194317609072,
+          -0.01014525443315506,
+          -0.011685633100569248,
+          -0.004138661548495293,
+          -0.011765307746827602,
+          0.003943901043385267,
+          0.0002846713177859783,
+          -0.011986627243459225,
+          -0.004607857670634985,
+          0.007153023965656757,
+          -0.012287620455026627,
+          -0.009295389987528324,
+          0.009746880270540714,
+          0.004368833266198635,
+          -0.019121944904327393,
+          -0.031657442450523376,
+          0.008657991886138916,
+          -0.03337487578392029,
+          0.004306863993406296,
+          0.015492316335439682,
+          0.004939835984259844,
+          -0.007276962511241436,
+          0.003184777684509754,
+          0.021972531452775,
+          0.014854918234050274,
+          -0.006719239056110382,
+          0.017103517428040504,
+          0.0036296285688877106,
+          0.006617432460188866,
+          -0.012606319040060043,
+          -1.0452126844029408e-05,
+          0.01843143068253994,
+          0.012225651182234287,
+          -0.0028749313205480576,
+          0.01335880346596241,
+          -0.006883014924824238,
+          0.008609301410615444,
+          -0.00020900795061606914,
+          -0.007670910097658634,
+          -0.018661601468920708,
+          -0.006342997308820486,
+          -0.015474610961973667,
+          -0.006095120217651129,
+          0.008427820168435574,
+          -0.005993313621729612,
+          -0.006312012672424316,
+          -0.010605597868561745,
+          0.010286898352205753,
+          -0.017484186217188835,
+          0.00045508681796491146,
+          -0.02572609856724739,
+          -0.00300329620949924,
+          -0.013659796677529812,
+          0.009392770007252693,
+          0.01156169455498457,
+          0.005687893368303776,
+          -0.015749046579003334,
+          -0.019599994644522667,
+          0.0007768290233798325,
+          -0.0029789512045681477,
+          -0.021529894322156906,
+          -0.007684188894927502,
+          -0.018502252176404,
+          -0.012376148253679276,
+          -0.009437033906579018,
+          0.014075876213610172,
+          0.003014362184330821,
+          -0.03256042301654816,
+          0.006205779500305653,
+          -0.027018601074814796,
+          -0.016846787184476852,
+          -0.01201318483799696,
+          -0.0021047419868409634,
+          0.0046565476804971695,
+          0.020113453269004822,
+          0.008516347967088223,
+          0.026575962081551552,
+          -3.7174642784520984e-05,
+          0.005205418448895216,
+          0.0012891821097582579,
+          0.011933510191738605,
+          0.009817702695727348,
+          -0.012323031201958656,
+          0.01620939001441002,
+          -0.008356998674571514,
+          0.0020128944888710976,
+          -0.022716162726283073,
+          -0.0012936084531247616,
+          -0.014438839629292488,
+          -0.006281028036028147,
+          -0.000598667364101857,
+          -0.01834290288388729,
+          -0.006294306833297014,
+          0.023743081837892532,
+          -0.019051123410463333,
+          0.008998822420835495,
+          0.008361424319446087,
+          -0.0003070798411499709,
+          0.01467786356806755,
+          0.029816070571541786,
+          -0.010915443301200867,
+          0.001500541577115655,
+          0.01944064348936081,
+          0.004315716680139303,
+          0.008503069169819355,
+          -0.0008183263125829399,
+          0.0036207756493240595,
+          0.012084007263183594,
+          -0.01155284233391285,
+          0.00648906733840704,
+          -0.005807405803352594,
+          0.008423393592238426,
+          -0.011269553564488888,
+          0.0041585806757211685,
+          -0.01949376054108143,
+          -0.01601462997496128,
+          0.019918693229556084,
+          0.012898460030555725,
+          -0.0003311482723802328,
+          -0.0051036118529737,
+          -0.03233025223016739,
+          -0.0008686763467267156,
+          -0.009817702695727348,
+          0.007431885693222284,
+          -0.013571269810199738,
+          0.011304965242743492,
+          -0.00642267195507884,
+          0.014146698638796806,
+          -0.013677502982318401,
+          0.018856363371014595,
+          -0.011269553564488888,
+          -0.01062330324202776,
+          0.017121223732829094,
+          0.009755733422935009,
+          -0.0037557801697403193,
+          0.029231788590550423,
+          -0.001606774516403675,
+          0.0029501798562705517,
+          -0.002301715547218919,
+          -0.030878400430083275,
+          0.0023747507948428392,
+          0.0028882105834782124,
+          0.00918030459433794,
+          0.004149727523326874,
+          -0.005820684600621462,
+          0.030701344832777977,
+          0.012154828757047653,
+          0.016997285187244415,
+          -0.011437756009399891,
+          -0.0008044939022511244,
+          0.020998729392886162,
+          0.013615533709526062,
+          0.0015481250593438745,
+          0.007400901056826115,
+          -0.0008199862204492092,
+          0.0007674229564145207,
+          -0.021777771413326263,
+          0.010286898352205753,
+          -0.018732424825429916,
+          -0.010428542271256447,
+          -0.002987803891301155,
+          -0.003027641447260976,
+          -0.003027641447260976,
+          0.0010352187091484666,
+          0.00528509309515357,
+          0.0015591910341754556,
+          0.023654554039239883,
+          -0.0009311988833360374,
+          -0.0037491407711058855,
+          -0.028134047985076904,
+          -0.017864855006337166,
+          0.003386177821084857,
+          0.011322670616209507,
+          0.029090143740177155,
+          0.0024920497089624405,
+          -0.009198009967803955,
+          -0.008330440148711205,
+          -0.012615172192454338,
+          0.01782059110701084,
+          0.010773799382150173,
+          -0.0025186080019921064,
+          0.0017229669028893113,
+          -0.011136762797832489,
+          0.014660158194601536,
+          -0.014607041142880917,
+          0.011127909645438194,
+          0.014120140112936497,
+          0.0019420725293457508,
+          -0.01249123364686966,
+          -0.0005477640661410987,
+          0.003862013341858983,
+          0.012145976535975933,
+          0.002469917992129922,
+          0.009171451441943645,
+          -0.02592085860669613,
+          -0.00666169635951519,
+          0.005679040681570768,
+          0.006648417096585035,
+          0.02296403981745243,
+          0.0152444401755929,
+          0.011225289665162563,
+          0.01108364574611187,
+          0.012349589727818966,
+          0.008600449189543724,
+          0.015005415305495262,
+          -0.013881116174161434,
+          0.0025938565377146006,
+          -0.0019044483778998256,
+          0.016492677852511406,
+          0.007812554016709328,
+          0.010729536414146423,
+          -0.02029050886631012,
+          0.0032069096341729164,
+          0.013429625891149044,
+          -0.008817341178655624,
+          0.008551758714020252,
+          -0.008410114794969559,
+          0.01378373522311449,
+          -0.005448868963867426,
+          0.0009323054691776633,
+          -0.0012393853394314647,
+          0.0030873974319547415,
+          0.032525014132261276,
+          -0.004218336660414934,
+          -0.004811470862478018,
+          0.01570478267967701,
+          -0.0045591676607728004,
+          0.0028240280225872993,
+          -0.01525329239666462,
+          0.00879078358411789,
+          0.02480541169643402,
+          0.00883504655212164,
+          -0.016253653913736343,
+          -0.014483102597296238,
+          0.004961967933923006,
+          -0.009321948513388634,
+          0.0011198731372132897,
+          0.007772716693580151,
+          -0.004424163140356541,
+          0.009321948513388634,
+          -0.0013799227308481932,
+          -0.0008244125638157129,
+          0.0049708206206560135,
+          -0.039164576679468155,
+          0.00252303434535861,
+          -0.015350672416388988,
+          0.006103972904384136,
+          0.004452934488654137,
+          -0.0024787706788629293,
+          -0.006462509278208017,
+          -0.006555463187396526,
+          -0.001566937193274498,
+          -0.05403720214962959,
+          0.0032201886642724276,
+          -0.0014573843218386173,
+          0.02393784187734127,
+          -0.01062330324202776,
+          0.00969376415014267,
+          0.015014268457889557,
+          0.013093221001327038,
+          -0.003078544745221734,
+          -0.0012659436324611306,
+          0.013411919586360455,
+          -0.0211934894323349,
+          -0.018626190721988678,
+          0.014837212860584259,
+          -0.007608940824866295,
+          0.00011750645353458822,
+          -0.004050134215503931,
+          0.004798191599547863,
+          -0.004634415730834007,
+          0.0033397008664906025,
+          0.013270275667309761,
+          -0.012730257585644722,
+          -0.01924588344991207,
+          -0.007515986915677786,
+          -0.006338570732623339,
+          -0.019511466845870018,
+          -0.016528088599443436,
+          0.00621463218703866,
+          -0.01714778132736683,
+          0.003678318578749895,
+          -0.0005380813381634653,
+          -0.00555510213598609,
+          0.013633239082992077,
+          0.001355577725917101,
+          0.0020538384560495615,
+          0.0007336718845181167,
+          0.009233420714735985,
+          -0.017431069165468216,
+          0.017652388662099838,
+          0.011216437444090843,
+          -0.014483102597296238,
+          0.00177940318826586,
+          -0.004669826943427324,
+          0.0026646784972399473,
+          0.021494483575224876,
+          -0.0065421839244663715,
+          0.0021224473603069782,
+          0.018767835572361946,
+          -0.0013876688899472356,
+          0.011198732070624828,
+          -0.003757993457838893,
+          0.005298372358083725,
+          -0.012101712636649609,
+          0.0012504513142630458,
+          0.00648906733840704,
+          -0.007812554016709328,
+          0.009065218269824982,
+          -0.04504280537366867,
+          0.012216798029839993,
+          -0.009144892916083336,
+          -0.024451302364468575,
+          -0.006059709005057812,
+          -0.0063164387829601765,
+          -0.0006456976407207549,
+          -0.01653694175183773,
+          0.013137484900653362,
+          -0.011216437444090843,
+          0.006745797581970692,
+          -0.012075154110789299,
+          0.0019885494839400053,
+          0.007839112542569637,
+          0.0003709856537170708,
+          0.002027280395850539,
+          -0.006068561691790819,
+          0.003687171498313546,
+          0.004926556721329689,
+          -0.002126873703673482,
+          -0.008989970199763775,
+          0.013544711284339428,
+          -0.0034171624574810266,
+          0.015952659770846367,
+          0.010756094008684158,
+          -0.008454378694295883,
+          -0.024681473150849342,
+          0.0157224889844656,
+          -0.004678679630160332,
+          -0.008941279724240303,
+          -0.0021556452848017216,
+          0.010065579786896706,
+          -3.074256528634578e-05,
+          -0.008392409421503544,
+          0.0027532060630619526,
+          0.009879671968519688,
+          0.007259257137775421,
+          -0.008967838250100613,
+          -0.004687532316893339,
+          -0.015456905588507652,
+          0.009525561705231667,
+          0.002750992774963379,
+          0.007462870329618454,
+          0.013898821547627449,
+          0.004132022149860859,
+          0.013376508839428425,
+          0.0004337848804425448,
+          -0.00759123545140028,
+          -0.026983190327882767,
+          -0.015766752883791924,
+          -0.010950854979455471,
+          0.0014153338270261884,
+          -0.0011785227106884122,
+          0.02018427476286888,
+          -0.0051611545495688915,
+          0.0046919588930904865,
+          0.0074540176428854465,
+          0.042032867670059204,
+          0.014182109385728836,
+          0.007887802086770535,
+          0.0007380982278846204,
+          -0.013819146901369095,
+          0.00038481809315271676,
+          -0.010463953018188477,
+          -0.03153350576758385,
+          0.009897377341985703,
+          -0.004129808861762285,
+          0.02315879985690117,
+          -0.014819507487118244,
+          0.014589335769414902,
+          -0.004510477185249329,
+          -0.01808617264032364,
+          -0.016944168135523796,
+          0.020626913756132126,
+          0.022840101271867752,
+          0.016377592459321022,
+          4.270761564839631e-05,
+          0.019511466845870018,
+          -0.007055643480271101,
+          0.0161562729626894,
+          -0.020113453269004822,
+          -0.004576873034238815,
+          0.006555463187396526,
+          0.02121119573712349,
+          0.0013467249227687716,
+          0.012765669263899326,
+          0.00711318664252758,
+          -0.0012825424782931805,
+          -0.001180735882371664,
+          -0.012703699991106987,
+          0.0019011285621672869,
+          -0.005174433812499046,
+          -0.0020471990574151278,
+          0.013252570293843746,
+          -0.01009213738143444,
+          -0.01837831363081932,
+          0.01149087306112051,
+          -0.010003610514104366,
+          0.0032622392755001783,
+          0.011234142817556858,
+          0.022344347089529037,
+          0.005515264812856913,
+          0.010880032554268837,
+          -0.006551036611199379,
+          8.638350118417293e-05,
+          -0.022397464141249657,
+          -0.01004787441343069,
+          -0.004138661548495293,
+          -0.01114561501890421,
+          0.003333061235025525,
+          -0.007586808875203133,
+          -0.025495927780866623,
+          0.00041497277561575174,
+          0.007458443753421307,
+          0.030099358409643173,
+          -0.013597827404737473,
+          0.02112266793847084,
+          0.0011148934718221426,
+          -0.0029501798562705517,
+          -0.010649860836565495,
+          -0.01425293181091547,
+          0.00576756801456213,
+          -0.01195121556520462,
+          -0.0014429986476898193,
+          6.57731870887801e-05,
+          -0.01830749213695526,
+          -3.8903697713976726e-05,
+          0.014005054719746113,
+          -0.037323202937841415,
+          -0.003957180306315422,
+          0.004992952570319176,
+          -0.0009284323896281421,
+          0.01387226302176714,
+          0.005643629468977451,
+          0.0036296285688877106,
+          0.0027554191183298826,
+          -0.003492410760372877,
+          0.00022380864538718015,
+          0.01929900050163269,
+          -0.015784457325935364,
+          0.02843504026532173,
+          0.00831273477524519,
+          -0.004891145508736372,
+          -0.0008930214098654687,
+          0.004503837786614895,
+          -0.028753740713000298,
+          0.010897737927734852,
+          -0.015952659770846367,
+          -0.018643897026777267,
+          0.011313817463815212,
+          0.015138207003474236,
+          -0.0025562322698533535,
+          0.002536313608288765,
+          0.015899542719125748,
+          0.011384639889001846,
+          0.005231976509094238,
+          -0.0016753834206610918,
+          -0.014146698638796806,
+          0.00203502643853426,
+          -0.01936982199549675,
+          0.009074071422219276,
+          -0.020024925470352173,
+          -0.0036318416241556406,
+          0.00849421601742506,
+          0.0038022571243345737,
+          0.015811016783118248,
+          0.009268831461668015,
+          0.018112732097506523,
+          -0.015359525568783283,
+          0.00891914777457714,
+          0.03973115235567093,
+          0.01147316675633192,
+          0.013677502982318401,
+          -0.005856095813214779,
+          -0.0005881547695025802,
+          0.025460515171289444,
+          -0.005262961145490408,
+          -0.008414541371166706,
+          0.020502975210547447,
+          0.018909478560090065,
+          0.014819507487118244,
+          -0.001221679849550128,
+          0.009065218269824982,
+          0.029214082285761833,
+          0.00552411749958992,
+          -0.005192139185965061,
+          -0.019511466845870018,
+          -0.006373981945216656,
+          -0.0020903560798615217,
+          0.013164042495191097,
+          0.007166303228586912,
+          0.016846787184476852,
+          -0.028966205194592476,
+          0.0034702790435403585,
+          0.003501263679936528,
+          -0.0001507734414190054,
+          0.013376508839428425,
+          0.02021968737244606,
+          0.003344127209857106,
+          -0.003842094447463751,
+          0.006493493914604187,
+          0.005174433812499046,
+          0.03125021606683731,
+          -0.025283461436629295,
+          -0.0037270088214427233,
+          0.0034038834273815155,
+          -0.015279850922524929,
+          0.013606680557131767,
+          -0.004523756448179483,
+          -0.002558445557951927,
+          -0.007852391339838505,
+          -0.010490511544048786,
+          0.016713997349143028,
+          0.01419096253812313,
+          -0.015456905588507652,
+          -0.025177227333188057,
+          -0.02117578312754631,
+          0.007122039329260588,
+          -0.02108725719153881,
+          -0.0072858151979744434,
+          -0.01009213738143444,
+          -0.006387260742485523,
+          -0.009578677825629711,
+          -0.006475788541138172,
+          -0.009746880270540714,
+          0.005563954822719097,
+          0.0007120932568795979,
+          0.003638481255620718,
+          0.0007353317341767251,
+          -0.01377488300204277,
+          -0.011136762797832489,
+          0.021671537309885025,
+          0.004860160872340202,
+          0.01296928245574236,
+          0.0026071355678141117,
+          0.005231976509094238,
+          0.006250043399631977,
+          -0.04298896715044975,
+          -0.006050856318324804,
+          -0.006679401732981205,
+          0.009357359260320663,
+          0.005563954822719097,
+          0.022716162726283073,
+          -0.012110565789043903,
+          0.015766752883791924,
+          -0.010437395423650742,
+          0.010039021261036396,
+          -0.010242634452879429,
+          0.0026646784972399473,
+          0.008215353824198246,
+          -0.007467296905815601,
+          0.0016565712867304683,
+          -0.004607857670634985,
+          0.007679762784391642,
+          -0.0018391592893749475,
+          0.007608940824866295,
+          -0.00960523635149002,
+          -0.015483464114367962,
+          -0.0023526190780103207,
+          -0.011605958454310894,
+          -0.017864855006337166,
+          5.754288940806873e-05,
+          -0.008587169460952282,
+          -0.0015315262135118246,
+          -0.019688520580530167,
+          -0.01924588344991207,
+          0.011960068717598915,
+          -0.0020615847315639257,
+          0.009091776795685291,
+          0.013916526921093464,
+          0.007763864006847143,
+          0.015306408517062664,
+          0.013376508839428425,
+          -0.004452934488654137,
+          -0.013571269810199738,
+          -0.014208667911589146,
+          0.0023526190780103207,
+          -0.014943446032702923,
+          0.01935211755335331,
+          0.008210928179323673,
+          -0.0017307130619883537,
+          0.0011884820414707065,
+          -0.020998729392886162,
+          -0.0001062330265995115,
+          -0.0022209342569112778,
+          0.0015569778624922037,
+          -0.018590779975056648,
+          0.02200794219970703,
+          -0.018289785832166672,
+          0.002113594673573971,
+          0.019688520580530167,
+          0.029780659824609756,
+          0.0028461599722504616,
+          -0.012925018556416035,
+          0.0007309053908102214,
+          0.008427820168435574,
+          0.03898752108216286,
+          -0.007985183037817478,
+          0.001860184594988823,
+          -0.011853835545480251,
+          -0.010313456878066063,
+          0.00435555400326848,
+          0.006365128792822361,
+          0.006812192965298891,
+          -0.026522846892476082,
+          -0.013597827404737473,
+          -0.010375426150858402,
+          -0.013695208355784416,
+          -0.0058118319138884544,
+          0.04072266072034836,
+          0.001315740286372602,
+          -0.0010070005664601922,
+          -0.00669268099591136,
+          0.00023556621454190463,
+          -0.017280573025345802,
+          0.03047117404639721,
+          0.09419328719377518,
+          -0.008817341178655624,
+          -0.009552120231091976,
+          -0.025035584345459938,
+          0.009198009967803955,
+          -0.001912194536998868,
+          0.011906951665878296,
+          0.06069447100162506,
+          -0.020662324503064156,
+          0.000730352068785578,
+          -0.0007264789892360568,
+          0.009897377341985703,
+          -0.00963179487735033,
+          -0.0026248411741107702,
+          -0.00735663715749979,
+          -0.008888163603842258,
+          0.0017605911707505584,
+          -0.03140956535935402,
+          0.0005306118400767446,
+          -0.006272174883633852,
+          0.005506412126123905,
+          -0.014244078658521175,
+          0.004581299610435963,
+          -0.010818063281476498,
+          0.004220549948513508,
+          -0.013677502982318401,
+          -0.005448868963867426,
+          0.01297813467681408,
+          -0.019688520580530167,
+          -0.0007790421950630844,
+          0.004647694993764162,
+          -0.012641730718314648,
+          0.008392409421503544,
+          0.0026248411741107702,
+          -0.011278406716883183,
+          -0.0015846426831558347,
+          -0.015085089951753616,
+          0.028789151459932327,
+          0.0018834230722859502,
+          0.007259257137775421,
+          -0.009950493462383747,
+          0.00101751321926713,
+          0.01293387170881033,
+          -0.0021711376029998064,
+          0.017121223732829094,
+          0.013571269810199738,
+          -0.001765017514117062,
+          -0.008808488957583904,
+          0.01714778132736683,
+          -0.00046089643728919327,
+          -0.013314539566636086,
+          0.009136040695011616,
+          0.011216437444090843,
+          0.0027266477700322866,
+          0.0037535671144723892,
+          -0.0015724701806902885,
+          0.023548321798443794,
+          -0.017254015430808067,
+          -0.011482019908726215,
+          -0.004072266165167093,
+          -0.009330800734460354,
+          0.021441366523504257,
+          0.006307586096227169,
+          0.03397686406970024,
+          -0.009773438796401024,
+          -0.0012305325362831354,
+          -0.018590779975056648,
+          -0.007259257137775421,
+          -0.010189518332481384,
+          -0.005993313621729612,
+          0.013553564436733723,
+          0.0019299000268802047,
+          0.012694846838712692,
+          -0.019971810281276703,
+          -0.017156634479761124,
+          0.00960523635149002,
+          -0.02294633351266384,
+          -0.0011375787435099483,
+          0.004922130145132542,
+          -0.027213361114263535,
+          -0.004435229115188122,
+          0.008166664279997349,
+          -0.02121119573712349,
+          -0.025230344384908676,
+          0.002651399467140436,
+          -0.0007049004198051989,
+          -0.02025509811937809,
+          0.009711469523608685,
+          0.006015445105731487,
+          -0.008565038442611694,
+          -0.002060478087514639,
+          -0.002961245598271489,
+          -0.020024925470352173,
+          0.0076399254612624645,
+          -0.015058532357215881,
+          -0.008038299158215523,
+          -0.006174794863909483,
+          0.004988525994122028,
+          -0.0012703699758276343,
+          0.02114037238061428,
+          0.006338570732623339,
+          -0.004032428842037916,
+          -0.011623663827776909,
+          0.0018280934309586883,
+          0.008472084067761898,
+          0.017138928174972534,
+          0.02285780757665634,
+          -0.0019442857010290027,
+          -0.006750223692506552,
+          0.001236065523698926,
+          0.015403789468109608,
+          0.005147875752300024,
+          -0.010791505686938763,
+          0.025354282930493355,
+          0.006785634905099869,
+          -0.008033872582018375,
+          0.015483464114367962,
+          -0.012668288312852383,
+          0.001964204479008913,
+          -0.009977051988244057,
+          -0.008219780400395393,
+          0.0042780926451087,
+          -0.0007159663364291191,
+          0.024238836020231247,
+          0.005581660196185112,
+          0.014571630395948887,
+          0.010419690050184727,
+          0.012774521484971046,
+          0.022432874888181686,
+          -0.01201318483799696,
+          0.0019874428398907185,
+          -0.0067280917428433895,
+          0.0018535450799390674,
+          -0.0005510838236659765,
+          -0.007684188894927502,
+          -0.006741371005773544,
+          0.0050283633172512054,
+          0.009445887058973312,
+          -0.0017561647109687328,
+          0.010348867624998093,
+          0.019192766398191452,
+          0.014828360639512539,
+          0.005391326267272234,
+          -0.0064093926921486855,
+          -0.019954103976488113,
+          0.003248960245400667,
+          0.021016433835029602,
+          -0.0034724920988082886,
+          0.010410836897790432,
+          0.0014175469987094402,
+          0.01855536922812462,
+          0.00669268099591136,
+          0.0050637745298445225,
+          -0.00666169635951519,
+          0.008742093108594418,
+          -0.01646612025797367,
+          0.004581299610435963,
+          0.0034437207505106926,
+          0.02206105925142765,
+          -0.007493854966014624,
+          -0.0067236656323075294,
+          0.0002651675895322114,
+          0.031728263944387436,
+          0.01384570449590683,
+          0.010782652534544468,
+          -0.019564582034945488,
+          0.009392770007252693,
+          -0.006705959793180227,
+          0.0015547646908089519,
+          0.0007773823454044759,
+          0.0115971053019166,
+          -0.007436312269419432,
+          -0.005621497984975576,
+          0.011756455525755882,
+          0.009233420714735985,
+          0.00918030459433794,
+          -0.0030453468207269907,
+          -0.017555007711052895,
+          0.006170368287712336,
+          -0.0047849128022789955,
+          -0.012137123383581638,
+          0.026522846892476082,
+          0.0157224889844656,
+          0.032879121601581573,
+          -0.003069691825658083,
+          0.0019619911909103394,
+          0.011402345262467861,
+          0.0018015351379290223,
+          0.014908035285770893,
+          -0.01011869590729475,
+          -0.0109862657263875,
+          0.03392374888062477,
+          -0.01246467512100935,
+          0.008135679177939892,
+          -0.024150308221578598,
+          -0.026469729840755463,
+          -0.0028129620477557182,
+          0.001937646185979247,
+          0.0049708206206560135,
+          0.0027952566742897034,
+          -0.004528183024376631,
+          -0.007338931784033775,
+          0.01296928245574236,
+          -0.001329019432887435,
+          0.0037668461445719004,
+          0.008976691402494907,
+          -0.0044418685138225555,
+          0.0031139557249844074,
+          -0.002733287401497364,
+          -0.000756910361815244,
+          0.010959707200527191
+        ],
+        "keyphrases": [
+          "expert choice method",
+          "auto-regressive text generation",
+          "large memory footprint",
+          "sparsely gated networks",
+          "power saving techniques"
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "dea84165-aef9-4273-9988-e3d233346670",
+      "properties": {
+        "page_content": "A Comparison On Fine-Tuning With A Dense Model\n\nOur 8B MoE model achieves stronger pre-training perplexity than its dense counterpart. However, a better perplexity does not always directly translate to downstream performance as demonstrated in Section 4.4. To this end, we compare fine-tuning performance of the 8B dense model and MoE model in Table 1. As shown in the table, our MoE model using expert choice routing consistently outperforms the dense model across the 11 tasks in GLUE and SuperGLUE.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 1: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\n",
+        "title": "A Comparison On Fine-Tuning With A Dense Model",
+        "embedding": [
+          0.016482073813676834,
+          -0.002563994377851486,
+          -0.0267450250685215,
+          -0.023007972165942192,
+          0.025922315195202827,
+          0.018462155014276505,
+          -0.01591036096215248,
+          0.03547411039471626,
+          -0.012068727053701878,
+          -0.011176296509802341,
+          0.003149651922285557,
+          -0.008659363724291325,
+          -0.021000003442168236,
+          0.016844624653458595,
+          -0.02533665858209133,
+          0.032490044832229614,
+          0.024039844051003456,
+          0.01954980380833149,
+          0.010472113266587257,
+          -0.021557772532105446,
+          -0.0034128492698073387,
+          -0.011873507872223854,
+          -0.011252989992499352,
+          -0.022938251495361328,
+          0.01864342950284481,
+          -0.007376495283097029,
+          0.017639445140957832,
+          -0.010311754420399666,
+          -0.029952196404337883,
+          0.032490044832229614,
+          0.012884464114904404,
+          0.014557771384716034,
+          0.02639641799032688,
+          -0.006728088948875666,
+          -0.05795220285654068,
+          -0.010625500231981277,
+          0.0629163458943367,
+          0.01878287084400654,
+          -0.027177294716238976,
+          0.06062949076294899,
+          0.024179287254810333,
+          0.018978090956807137,
+          -0.004779383074492216,
+          -0.004329681862145662,
+          0.01492032129317522,
+          -0.030314745381474495,
+          -0.018476098775863647,
+          -0.0148645443841815,
+          0.03494422882795334,
+          -0.03650598227977753,
+          -0.010472113266587257,
+          -0.01666335016489029,
+          0.018657373264431953,
+          0.011469125747680664,
+          -0.0498366616666317,
+          0.0530996099114418,
+          0.006975598633289337,
+          0.01773705519735813,
+          0.04239044710993767,
+          0.005661355331540108,
+          -0.007543826010078192,
+          -0.0029038849752396345,
+          -0.026758968830108643,
+          -0.02917131967842579,
+          0.011315738782286644,
+          0.015966137871146202,
+          -0.01385358814150095,
+          -0.02132072113454342,
+          0.03890438750386238,
+          0.01121812965720892,
+          -0.05678088590502739,
+          0.05452192202210426,
+          -0.025239048525691032,
+          0.018099604174494743,
+          -0.009293826296925545,
+          -0.03438645973801613,
+          0.014320719987154007,
+          0.01945219375193119,
+          0.022770920768380165,
+          0.0009926545899361372,
+          -0.012201197445392609,
+          -0.010939245112240314,
+          -0.0014963897410780191,
+          0.009684264659881592,
+          0.03611554577946663,
+          -0.026758968830108643,
+          -0.017848609015345573,
+          -0.05254184082150459,
+          -0.02526693604886532,
+          -0.005563745740801096,
+          -0.006602590903639793,
+          -0.010318727232515812,
+          -0.011308766901493073,
+          0.020693231374025345,
+          0.007257969584316015,
+          0.042920324951410294,
+          -0.004256474785506725,
+          0.01883864775300026,
+          -0.02076295204460621,
+          -0.008903387933969498,
+          0.00044708672794513404,
+          0.03296414762735367,
+          0.018252991139888763,
+          0.004782869480550289,
+          0.04950200021266937,
+          -0.015394425019621849,
+          0.020288847386837006,
+          0.007181276101619005,
+          0.0038242037408053875,
+          0.06832670420408249,
+          -0.005229084752500057,
+          -0.021892433986067772,
+          -0.03126295283436775,
+          0.008854582905769348,
+          -0.014794823713600636,
+          -0.00372310820966959,
+          -0.02597809210419655,
+          -0.011713149026036263,
+          0.04013147950172424,
+          -0.027637455612421036,
+          0.0076762959361076355,
+          0.027972117066383362,
+          0.019033867865800858,
+          -0.03313148021697998,
+          -0.03466534614562988,
+          0.045011959969997406,
+          0.02066534198820591,
+          0.0014536854578182101,
+          0.008087650872766972,
+          -0.014237054623663425,
+          -0.001226220279932022,
+          0.012473110109567642,
+          -0.014188249595463276,
+          -0.02958964742720127,
+          0.030454188585281372,
+          0.0055742040276527405,
+          0.0256852637976408,
+          0.03028685785830021,
+          -0.03555777668952942,
+          -0.03354980796575546,
+          0.019745023921132088,
+          -0.05968128517270088,
+          0.016440242528915405,
+          0.011992033571004868,
+          -0.004497012589126825,
+          0.010904384776949883,
+          0.0532948300242424,
+          0.08717929571866989,
+          -0.013016934506595135,
+          -0.004096116404980421,
+          -0.02710757404565811,
+          0.027065740898251534,
+          0.004817729815840721,
+          0.036394428461790085,
+          -0.007495020981878042,
+          -0.010953188873827457,
+          -0.02603386901319027,
+          0.0032385464292019606,
+          -0.013477093540132046,
+          -0.02132072113454342,
+          0.011280878446996212,
+          0.008241036906838417,
+          -0.004657371435314417,
+          -0.008310758508741856,
+          0.0006462276796810329,
+          0.03182072192430496,
+          0.0277071762830019,
+          -0.007313746493309736,
+          -0.07591236382722855,
+          0.031681280583143234,
+          0.011057770811021328,
+          0.030454188585281372,
+          0.0072161369025707245,
+          0.020037852227687836,
+          0.019061755388975143,
+          0.02533665858209133,
+          0.006578188389539719,
+          -0.014976098202168941,
+          -0.030482076108455658,
+          0.024374505504965782,
+          -0.01294721383601427,
+          -0.027986060827970505,
+          0.041916340589523315,
+          -0.01662151701748371,
+          0.013023906387388706,
+          -0.0030241538770496845,
+          0.010855579748749733,
+          0.025294825434684753,
+          -0.001602714415639639,
+          0.017848609015345573,
+          -0.00970518123358488,
+          0.045820724219083786,
+          0.03028685785830021,
+          -0.01878287084400654,
+          -0.007780877873301506,
+          -0.039992038160562515,
+          -0.024025900289416313,
+          0.01833665743470192,
+          0.01068824902176857,
+          0.017346616834402084,
+          -0.005978586617857218,
+          -0.03592032566666603,
+          -0.006846614647656679,
+          -0.021460162475705147,
+          0.02320319041609764,
+          -0.0015730829909443855,
+          0.0327131524682045,
+          -0.0027417833916842937,
+          -0.003228088142350316,
+          -0.016175301745533943,
+          0.024388451129198074,
+          -0.023998012766242027,
+          -0.020177293568849564,
+          -0.021111557260155678,
+          -0.01812749356031418,
+          0.022715143859386444,
+          0.004165837541222572,
+          -0.002523904899135232,
+          -0.022450203076004982,
+          -0.0008218378061428666,
+          0.008729085326194763,
+          0.028557773679494858,
+          -0.00485259061679244,
+          -0.022464146837592125,
+          -0.05692032724618912,
+          -0.03388446941971779,
+          0.021111557260155678,
+          -0.02998008392751217,
+          0.022603590041399002,
+          -0.012668329291045666,
+          -0.022464146837592125,
+          0.009698208421468735,
+          0.003107819240540266,
+          0.02350996434688568,
+          -0.006414343602955341,
+          -0.02526693604886532,
+          0.018211157992482185,
+          -0.020274903625249863,
+          0.013260958716273308,
+          0.029254985973238945,
+          -0.017262950539588928,
+          0.014111556112766266,
+          -0.03525100275874138,
+          -0.01833665743470192,
+          0.03937849402427673,
+          -0.010486057959496975,
+          -0.006825698539614677,
+          -0.0072509972378611565,
+          0.006532869767397642,
+          0.0019382473547011614,
+          0.005790339782834053,
+          -0.037760961800813675,
+          0.0075089652091264725,
+          0.009795818477869034,
+          0.023523908108472824,
+          0.01970319077372551,
+          0.05895618721842766,
+          -0.04685259610414505,
+          -0.01579880714416504,
+          -0.03965737670660019,
+          0.05374104529619217,
+          0.010297810658812523,
+          -0.03965737670660019,
+          0.015868527814745903,
+          -0.020651398226618767,
+          -0.01768127828836441,
+          0.003187998663634062,
+          0.009942232631146908,
+          -0.014850599691271782,
+          0.033159367740154266,
+          0.01281474344432354,
+          -0.021599605679512024,
+          -0.04947411268949509,
+          0.0050024907104671,
+          0.004908367525786161,
+          -0.019466139376163483,
+          0.010841635055840015,
+          0.004577192012220621,
+          0.0018894425593316555,
+          0.0021927293855696917,
+          0.023677295073866844,
+          0.03257371112704277,
+          0.023886458948254585,
+          0.02978486567735672,
+          -0.006905877497047186,
+          0.0050756982527673244,
+          -0.02279880829155445,
+          -0.01053486205637455,
+          -0.03053785301744938,
+          -0.0037265941500663757,
+          -0.009342631325125694,
+          -0.020944226533174515,
+          -0.024695223197340965,
+          -0.005731076467782259,
+          -0.006975598633289337,
+          0.007578686345368624,
+          -0.010827691294252872,
+          0.005399901419878006,
+          0.04952988773584366,
+          -0.03561355173587799,
+          -0.043478094041347504,
+          -0.007045319769531488,
+          0.02031673677265644,
+          0.002699950709939003,
+          -0.05951395258307457,
+          0.008854582905769348,
+          0.018350601196289062,
+          0.0024506975896656513,
+          -0.02405378967523575,
+          -0.02391434647142887,
+          0.026117533445358276,
+          -0.008729085326194763,
+          -0.013023906387388706,
+          -0.053490050137043,
+          0.0010527890408411622,
+          0.005856574513018131,
+          0.0057973116636276245,
+          0.03184860944747925,
+          -0.03873705863952637,
+          -0.004577192012220621,
+          0.01949402689933777,
+          0.013937253504991531,
+          -0.07686056941747665,
+          0.014627492986619473,
+          -0.004148406907916069,
+          0.027874507009983063,
+          0.045458175241947174,
+          0.039183273911476135,
+          -0.01941036246716976,
+          -6.231325824046507e-05,
+          0.0032559765968471766,
+          -0.0519561842083931,
+          -0.029645422473549843,
+          0.007076694630086422,
+          0.006240040995180607,
+          -0.034776899963617325,
+          -0.0442589707672596,
+          -0.0073067741468548775,
+          -0.02589442767202854,
+          0.013637452386319637,
+          -0.011685260571539402,
+          0.009907372295856476,
+          -0.034776899963617325,
+          0.010095619596540928,
+          0.018824703991413116,
+          -0.009670319966971874,
+          -0.026354586705565453,
+          -0.03572510555386543,
+          0.01180378720164299,
+          -0.017095619812607765,
+          0.0007935136090964079,
+          0.021348608657717705,
+          0.006473606918007135,
+          0.005654383450746536,
+          -0.007515937555581331,
+          -0.03564143925905228,
+          -0.006836156360805035,
+          -0.005201196298003197,
+          0.020400401204824448,
+          0.0342470183968544,
+          -0.011176296509802341,
+          -0.009021913632750511,
+          -0.011036854237318039,
+          0.04741036519408226,
+          -0.05148208141326904,
+          -0.009265937842428684,
+          0.01864342950284481,
+          -0.006484064739197493,
+          0.008143427781760693,
+          0.005086156073957682,
+          -0.026675302535295486,
+          0.02461155876517296,
+          0.01756972447037697,
+          0.0110647426918149,
+          0.028529886156320572,
+          0.00459810858592391,
+          0.030565742403268814,
+          -0.002150896703824401,
+          -0.03064940683543682,
+          0.014669325202703476,
+          -0.020400401204824448,
+          0.02735857106745243,
+          -0.015784863382577896,
+          -0.004960658028721809,
+          -0.008401395753026009,
+          0.032490044832229614,
+          0.01812749356031418,
+          0.005263945087790489,
+          -0.04149801656603813,
+          0.016705181449651718,
+          0.009161355905234814,
+          -0.025852594524621964,
+          0.009516933932900429,
+          0.009565738961100578,
+          0.0181832704693079,
+          0.01777888834476471,
+          -0.023217136040329933,
+          -0.0037056778091937304,
+          0.00459810858592391,
+          -0.013365539722144604,
+          -0.022701198235154152,
+          0.024597613140940666,
+          0.003393675899133086,
+          -0.01849004253745079,
+          0.003433765610679984,
+          0.016119524836540222,
+          0.017918329685926437,
+          0.0071220132522284985,
+          0.008234065026044846,
+          -0.04052191972732544,
+          -0.0022188748698681593,
+          -0.01960558071732521,
+          -0.005361554678529501,
+          0.017792832106351852,
+          0.0275119561702013,
+          0.05123108625411987,
+          0.00025666088913567364,
+          0.03321514651179314,
+          -0.0030642435885965824,
+          0.0009246764821000397,
+          -0.036087654531002045,
+          0.011287850327789783,
+          0.03140239417552948,
+          0.01526892650872469,
+          0.028027893975377083,
+          0.030733073130249977,
+          -0.029338650405406952,
+          0.010262950323522091,
+          0.02005179598927498,
+          0.008373507298529148,
+          0.021460162475705147,
+          -0.005542829632759094,
+          0.034721121191978455,
+          -0.009698208421468735,
+          0.007257969584316015,
+          -0.011985061690211296,
+          -0.017039842903614044,
+          0.022729087620973587,
+          -0.03985259681940079,
+          -0.0009752242476679385,
+          0.022659366950392723,
+          -0.021794823929667473,
+          0.001286354847252369,
+          -0.01939641684293747,
+          -0.0075298817828297615,
+          0.009586654603481293,
+          -0.03586454689502716,
+          0.01934063993394375,
+          -0.021446218714118004,
+          -0.014878488145768642,
+          -0.039183273911476135,
+          -0.0016332174418494105,
+          -0.06866136938333511,
+          8.072399214142933e-05,
+          0.031625501811504364,
+          -0.0016846366925165057,
+          0.023691238835453987,
+          0.007006973493844271,
+          0.014132472686469555,
+          -0.055386465042829514,
+          -0.0040473113767802715,
+          -0.0073625510558485985,
+          0.014432273805141449,
+          -0.009028886444866657,
+          -0.03299203887581825,
+          -0.011922312900424004,
+          -0.0023374008014798164,
+          0.004071713890880346,
+          0.0032472615130245686,
+          0.007599602919071913,
+          -0.029756976291537285,
+          -0.013330679386854172,
+          -0.03854183852672577,
+          -0.010262950323522091,
+          -0.004490040708333254,
+          0.038932278752326965,
+          0.0009220619103871286,
+          -0.03976893052458763,
+          0.008750001899898052,
+          0.021948210895061493,
+          0.00022212715703062713,
+          -0.019745023921132088,
+          0.010423308238387108,
+          -0.042278893291950226,
+          0.02487649768590927,
+          -0.012396416626870632,
+          0.019187254831194878,
+          -0.0077529894188046455,
+          0.014223109930753708,
+          -0.0075647421181201935,
+          -0.009691236540675163,
+          -0.009175300598144531,
+          0.012326695024967194,
+          -0.027930283918976784,
+          -0.009454185143113136,
+          -0.03115139901638031,
+          0.009231077507138252,
+          -0.005284861661493778,
+          0.008896416053175926,
+          0.02795817144215107,
+          -0.004960658028721809,
+          -0.005005977116525173,
+          0.003339641960337758,
+          0.038011956959962845,
+          0.004730578511953354,
+          0.006041335873305798,
+          0.027037853375077248,
+          0.04423108324408531,
+          -0.00539641547948122,
+          -0.00426693307235837,
+          0.02486255392432213,
+          -0.010214145295321941,
+          0.016035858541727066,
+          -0.0133237075060606,
+          -0.006267929449677467,
+          0.027637455612421036,
+          -0.026159366592764854,
+          0.04280877113342285,
+          -0.008519921451807022,
+          0.017541835084557533,
+          -0.025964148342609406,
+          -0.01733267307281494,
+          -0.003034612163901329,
+          0.03686853125691414,
+          -0.000990040018223226,
+          0.006226096767932177,
+          0.017500003799796104,
+          0.0028411359526216984,
+          0.03742630034685135,
+          -0.02861355058848858,
+          -0.04473307356238365,
+          0.012215141206979752,
+          0.02547609992325306,
+          -0.008896416053175926,
+          -0.04094024747610092,
+          -0.020860562101006508,
+          -0.004908367525786161,
+          0.007766933646053076,
+          -0.03276893123984337,
+          -0.0037126499228179455,
+          0.014557771384716034,
+          0.0040089646354317665,
+          0.010848607867956161,
+          0.00960059929639101,
+          0.014153389260172844,
+          0.02851594053208828,
+          0.006755977403372526,
+          -0.03865339234471321,
+          0.0037126499228179455,
+          0.018880480900406837,
+          -0.014237054623663425,
+          -0.011497014202177525,
+          -0.02653586119413376,
+          0.0009717381908558309,
+          -0.009077690541744232,
+          -0.029756976291537285,
+          -0.013149404898285866,
+          0.006156375631690025,
+          -0.013205181807279587,
+          -0.012005978263914585,
+          -0.01853187568485737,
+          0.018559763208031654,
+          -0.002811504527926445,
+          0.01681673526763916,
+          -0.010681277140974998,
+          -0.013491038233041763,
+          0.014027890749275684,
+          0.021794823929667473,
+          -0.04225100204348564,
+          0.0009682521340437233,
+          -0.015812750905752182,
+          0.02734462544322014,
+          -0.005061754025518894,
+          0.01172709371894598,
+          0.012772910296916962,
+          -0.009398408234119415,
+          0.01180378720164299,
+          -0.01950797066092491,
+          -0.024346617981791496,
+          0.01127390656620264,
+          0.030203191563487053,
+          -0.01676095835864544,
+          -0.010276894085109234,
+          0.011741037480533123,
+          0.025615543127059937,
+          -0.023774905130267143,
+          -0.03349402919411659,
+          -0.030816737562417984,
+          0.03151394799351692,
+          -0.007843626663088799,
+          0.014188249595463276,
+          0.027623509988188744,
+          -0.0002780130016617477,
+          -0.03622709959745407,
+          -0.011176296509802341,
+          -0.03182072192430496,
+          -0.01945219375193119,
+          0.011343627236783504,
+          0.009426296688616276,
+          -0.03207171708345413,
+          -0.003395418869331479,
+          0.01839243434369564,
+          0.029756976291537285,
+          0.013421316631138325,
+          0.008296813815832138,
+          0.004503984935581684,
+          0.038876499980688095,
+          0.027023909613490105,
+          -0.003834662027657032,
+          0.0009926545899361372,
+          0.006288845557719469,
+          0.007836654782295227,
+          -0.017764942720532417,
+          0.0006161604542285204,
+          -0.010276894085109234,
+          -0.0008305529481731355,
+          -0.019480083137750626,
+          -0.006264443509280682,
+          0.00691633578389883,
+          -0.007202192675322294,
+          0.05968128517270088,
+          -0.0037928293459117413,
+          0.03449801355600357,
+          -0.018768927082419395,
+          0.026019925251603127,
+          0.04264144226908684,
+          -0.01169920526444912,
+          -0.02903187833726406,
+          -0.010395419783890247,
+          -0.026075702160596848,
+          -0.04492829367518425,
+          0.009928288869559765,
+          -0.015994027256965637,
+          -0.022784864529967308,
+          -0.011427292600274086,
+          -0.014069723896682262,
+          0.002382719423621893,
+          0.012535858899354935,
+          -0.012250002473592758,
+          0.001576569047756493,
+          -0.00861055962741375,
+          0.03301992639899254,
+          -0.0013456178130581975,
+          0.011866535991430283,
+          0.006470120511949062,
+          0.003243775339797139,
+          -0.012605579569935799,
+          -0.0037196220364421606,
+          0.014188249595463276,
+          -0.00018835598893929273,
+          0.014195221476256847,
+          -0.001346489298157394,
+          0.019466139376163483,
+          0.03513944894075394,
+          -0.04046614095568657,
+          -0.027023909613490105,
+          -0.006250499282032251,
+          -0.04955777898430824,
+          0.01650996319949627,
+          0.00397759024053812,
+          0.011587651446461678,
+          0.00866633653640747,
+          0.008875499479472637,
+          -0.03140239417552948,
+          0.024374505504965782,
+          -0.011071715503931046,
+          -0.015213149599730968,
+          0.022296816110610962,
+          -0.0026250004302710295,
+          0.012487053871154785,
+          0.02102789282798767,
+          0.0008022287511266768,
+          -0.041107576340436935,
+          -0.0033117535058408976,
+          0.01874103955924511,
+          -0.028195224702358246,
+          -0.00014674117846880108,
+          0.009035858325660229,
+          0.00034076202427968383,
+          0.0032559765968471766,
+          -0.013163348659873009,
+          0.0034965146332979202,
+          -0.012222114019095898,
+          -0.043478094041347504,
+          0.026061756536364555,
+          -0.002492530271410942,
+          -0.03165339305996895,
+          0.0363665409386158,
+          -0.019173309206962585,
+          -0.009238049387931824,
+          0.008791834115982056,
+          -0.03988048434257507,
+          -0.032796818763017654,
+          0.01677490398287773,
+          -0.017346616834402084,
+          0.009886455722153187,
+          -0.04077291488647461,
+          0.006686256267130375,
+          0.013644424267113209,
+          -0.0008780504576861858,
+          -0.011399404145777225,
+          0.04163745790719986,
+          -0.022980082780122757,
+          -0.004368028603494167,
+          0.0128426318988204,
+          -0.027121517807245255,
+          0.004043825436383486,
+          -0.0054277898743748665,
+          -0.015394425019621849,
+          0.009865539148449898,
+          -0.021111557260155678,
+          0.018964147195219994,
+          0.02330080047249794,
+          0.0016175301279872656,
+          0.022213151678442955,
+          -0.011615539900958538,
+          -0.0015417084796354175,
+          0.019005978479981422,
+          -0.005473108496516943,
+          0.047382477670907974,
+          0.0022816238924860954,
+          0.030760960653424263,
+          0.013337651267647743,
+          -0.024806777015328407,
+          0.0023426299449056387,
+          0.023886458948254585,
+          0.004622510634362698,
+          0.0055219135247170925,
+          0.008122511208057404,
+          0.007209164556115866,
+          -0.02101394720375538,
+          0.0024071219377219677,
+          -0.03600399196147919,
+          0.006030877586454153,
+          -0.0008009214652702212,
+          -0.018225103616714478,
+          -0.008331675082445145,
+          0.02872510440647602,
+          0.025545820593833923,
+          -0.01849004253745079,
+          0.02268725447356701,
+          -0.012731078080832958,
+          -0.002875996520742774,
+          0.011427292600274086,
+          0.002492530271410942,
+          0.023482074961066246,
+          -0.005256973206996918,
+          0.003545319428667426,
+          0.010674304328858852,
+          0.006410857662558556,
+          0.014501994475722313,
+          0.0014685012865811586,
+          0.00866633653640747,
+          0.00848506111651659,
+          -0.038876499980688095,
+          0.011824702844023705,
+          0.03580877184867859,
+          -0.004193725995719433,
+          -0.013281874358654022,
+          -0.01945219375193119,
+          -0.008220121264457703,
+          0.010255977511405945,
+          0.017667334526777267,
+          0.02091633901000023,
+          0.007718128617852926,
+          0.0057798814959824085,
+          -0.029115542769432068,
+          0.021111557260155678,
+          0.010276894085109234,
+          -0.01157370675355196,
+          -0.002255478408187628,
+          0.0025134466122835875,
+          0.04660160094499588,
+          -0.05845419317483902,
+          0.004859562497586012,
+          0.016551796346902847,
+          -0.0180856604129076,
+          -0.002550050150603056,
+          -0.02422112040221691,
+          -0.001696837949566543,
+          -0.004978088662028313,
+          0.02614542283117771,
+          -0.013107571750879288,
+          0.0021840143017470837,
+          -0.03555777668952942,
+          0.009203189052641392,
+          -0.001525149680674076,
+          -0.012417333200573921,
+          0.00012604272342287004,
+          0.002182271331548691,
+          -0.021334664896130562,
+          -0.0246255025267601,
+          0.0032507474534213543,
+          0.014069723896682262,
+          0.03215538337826729,
+          -0.006567730102688074,
+          0.0011983318254351616,
+          0.03714741766452789,
+          -0.0256852637976408,
+          -0.014683269895613194,
+          0.026061756536364555,
+          -0.005229084752500057,
+          0.03505578264594078,
+          0.02542032301425934,
+          -0.03120717592537403,
+          -0.05243028700351715,
+          -0.007264941465109587,
+          -0.011566734872758389,
+          0.01010956335812807,
+          0.009454185143113136,
+          0.032545823603868484,
+          -0.0001861772034317255,
+          -0.0030328689608722925,
+          -0.009691236540675163,
+          -0.01823904737830162,
+          0.034776899963617325,
+          0.00019924991647712886,
+          0.01743028126657009,
+          0.004514443222433329,
+          -0.018601596355438232,
+          -0.025699207559227943,
+          0.013037851080298424,
+          0.003547062398865819,
+          -0.005121016874909401,
+          -0.05212351307272911,
+          0.006982570979744196,
+          0.005054781679064035,
+          0.006173805799335241,
+          0.0377051867544651,
+          0.015994027256965637,
+          -0.0001385707437293604,
+          -0.0017822462832555175,
+          -0.0192430317401886,
+          0.029254985973238945,
+          -0.029729088768363,
+          0.04052191972732544,
+          0.0009011455695144832,
+          0.012326695024967194,
+          0.008791834115982056,
+          0.03221116214990616,
+          -0.017262950539588928,
+          -0.003604582278057933,
+          -0.02339841052889824,
+          -0.018420321866869926,
+          0.009154384024441242,
+          -0.020749008283019066,
+          0.0057066744193434715,
+          -0.007739045191556215,
+          0.0037649408914148808,
+          0.0038381479680538177,
+          0.008366535417735577,
+          0.05220717936754227,
+          0.026270920410752296,
+          0.0006104956264607608,
+          -0.024025900289416313,
+          -0.010813746601343155,
+          -0.016928289085626602,
+          0.005159363616257906,
+          -0.020498011261224747,
+          -0.007739045191556215,
+          -0.032099608331918716,
+          -0.014613548293709755,
+          -0.014529882930219173,
+          -0.025754984468221664,
+          0.00850597769021988,
+          0.014641436748206615,
+          -0.006424801889806986,
+          -0.011176296509802341,
+          -0.019591636955738068,
+          -0.033661361783742905,
+          -0.02081872895359993,
+          -0.009035858325660229,
+          -0.01430677529424429,
+          0.029254985973238945,
+          -0.0028324208687990904,
+          -0.006334164645522833,
+          -0.006623507011681795,
+          -0.030063750222325325,
+          0.018615540117025375,
+          -0.018266934901475906,
+          -0.00789940357208252,
+          -0.012012950144708157,
+          0.02741434797644615,
+          0.0028637952636927366,
+          0.046573713421821594,
+          0.002319970401003957,
+          -0.014209166169166565,
+          -0.05190040543675423,
+          -0.03491634130477905,
+          -0.02947809360921383,
+          0.016886457800865173,
+          0.0032071718014776707,
+          0.0010702193249017,
+          0.002781872870400548,
+          0.006410857662558556,
+          0.002318227430805564,
+          -0.0056508975103497505,
+          -0.029896419495344162,
+          0.0032298313453793526,
+          0.010388447903096676,
+          0.005009463056921959,
+          -0.035083670169115067,
+          -0.021083669736981392,
+          -0.018364544957876205,
+          0.038876499980688095,
+          -0.03742630034685135,
+          0.03438645973801613,
+          -0.011510957963764668,
+          -0.01814143732190132,
+          0.004249502904713154,
+          -0.027037853375077248,
+          0.006121514830738306,
+          0.006292331963777542,
+          0.007578686345368624,
+          0.03600399196147919,
+          0.01837848871946335,
+          0.043840646743774414,
+          -0.0286693274974823,
+          0.03820717707276344,
+          -0.01326793059706688,
+          -0.0006126743974164128,
+          -0.0005874005146324635,
+          0.0038799806497991085,
+          0.02922709658741951,
+          -0.002659860998392105,
+          -0.0006985185318626463,
+          -0.0050931284204125404,
+          -0.004315737634897232,
+          -0.005309263709932566,
+          0.003034612163901329,
+          -0.025657374411821365,
+          0.012068727053701878,
+          0.024904387071728706,
+          -0.015338648110628128,
+          0.03628287464380264,
+          -0.004674801602959633,
+          0.03756574168801308,
+          -0.013511954806745052,
+          -0.027525901794433594,
+          0.027135463431477547,
+          -0.01570119708776474,
+          -0.013525898568332195,
+          0.005086156073957682,
+          0.0018092632526531816,
+          -0.0015643677907064557,
+          -0.02097211591899395,
+          0.024235064163804054,
+          0.02487649768590927,
+          0.003939243499189615,
+          0.0038102595135569572,
+          -0.02953387051820755,
+          -0.009984065778553486,
+          0.0181832704693079,
+          -0.004904881119728088,
+          0.020302793011069298,
+          0.02320319041609764,
+          -0.014118527993559837,
+          0.02193426713347435,
+          -0.01050697360187769,
+          0.016384465619921684,
+          0.022352593019604683,
+          -0.014069723896682262,
+          -0.016802791506052017,
+          -0.020274903625249863,
+          -0.004019422922283411,
+          0.004388945177197456,
+          -0.02284064143896103,
+          -0.03159761428833008,
+          0.0028463650960475206,
+          0.02958964742720127,
+          0.007108069024980068,
+          -0.03658964857459068,
+          -0.005738048814237118,
+          -0.017346616834402084,
+          0.030984068289399147,
+          -0.026521917432546616,
+          -0.0007778263534419239,
+          -0.010576695203781128,
+          -0.001526892650872469,
+          0.012005978263914585,
+          0.04088446870446205,
+          -0.0016088150441646576,
+          0.03095618076622486,
+          0.0171235091984272,
+          0.005145419389009476,
+          -0.0077320728451013565,
+          0.03525100275874138,
+          -0.016035858541727066,
+          -0.022561756893992424,
+          0.031123511493206024,
+          -0.014557771384716034,
+          -0.007843626663088799,
+          0.018043827265501022,
+          0.015547811053693295,
+          -0.015812750905752182,
+          0.026884466409683228,
+          -0.0008057148079387844,
+          0.011476097628474236,
+          0.011789842508733273,
+          -0.0009682521340437233,
+          -0.013177293352782726,
+          -0.0286693274974823,
+          -0.023984067142009735,
+          0.01802988350391388,
+          -0.008861555717885494,
+          0.03246215730905533,
+          0.018754983320832253,
+          0.015492034144699574,
+          -0.03923904895782471,
+          0.0365617610514164,
+          -0.013253985904157162,
+          -0.004510956816375256,
+          0.0032472615130245686,
+          -0.020749008283019066,
+          -0.00024380606191698462,
+          -0.009593627415597439,
+          -0.03377291560173035,
+          -0.03321514651179314,
+          -0.02339841052889824,
+          0.0003054656845051795,
+          0.01162251178175211,
+          -0.00419023958966136,
+          -0.031039845198392868,
+          -0.023189246654510498,
+          0.025057774037122726,
+          0.02826494537293911,
+          -0.008959164842963219,
+          -0.026312753558158875,
+          -0.006731574889272451,
+          -0.019689247012138367,
+          0.020776895806193352,
+          -0.0008392680902034044,
+          0.004573706071823835,
+          0.021794823929667473,
+          0.01076494250446558,
+          -0.006581674329936504,
+          -0.007885459810495377,
+          -0.036952197551727295,
+          0.008540838025510311,
+          0.007285858038812876,
+          -0.0034633970353752375,
+          -0.006787351798266172,
+          0.013686257414519787,
+          0.0005233442061580718,
+          -0.009203189052641392,
+          -0.018252991139888763,
+          0.0011312252609059215,
+          -0.0009516933932900429,
+          0.01317032054066658,
+          -0.00043771794298663735,
+          0.014174304902553558,
+          0.016551796346902847,
+          -0.006009961012750864,
+          -0.005710160359740257,
+          0.008980081416666508,
+          0.04478885233402252,
+          0.02330080047249794,
+          -0.02613147906959057,
+          0.011838647536933422,
+          0.015882473438978195,
+          0.0029282872565090656,
+          -0.015826696529984474,
+          0.006940738297998905,
+          0.023886458948254585,
+          -0.031235065311193466,
+          -0.0027138949371874332,
+          -0.007397411856800318,
+          0.01701195538043976,
+          0.032350603491067886,
+          0.027428291738033295,
+          0.00019129733846057206,
+          -0.005748507101088762,
+          -0.00554631557315588,
+          0.019033867865800858,
+          0.012563747353851795,
+          0.018601596355438232,
+          -0.0071603599935770035,
+          -0.012354583479464054,
+          -0.026103589683771133,
+          0.008450200781226158,
+          -0.0026947215665131807,
+          -0.011531874537467957,
+          0.032545823603868484,
+          0.012556775473058224,
+          0.015826696529984474,
+          -0.012828687205910683,
+          -0.016788847744464874,
+          0.026661358773708344,
+          0.022199207916855812,
+          -0.025308769196271896,
+          -0.021739047020673752,
+          0.010325699113309383,
+          0.0017683020560070872,
+          -0.0029840641655027866,
+          -0.013553787022829056,
+          -0.022617533802986145,
+          0.028585663065314293,
+          0.09086057543754578,
+          -0.015045818872749805,
+          -0.01337251253426075,
+          -0.01733267307281494,
+          -0.023984067142009735,
+          -0.019926298409700394,
+          0.0005407744902186096,
+          0.006145917344838381,
+          0.03313148021697998,
+          0.009886455722153187,
+          -0.003914841450750828,
+          -0.005783367436379194,
+          -0.006135459057986736,
+          0.012054783292114735,
+          -0.031988054513931274,
+          0.02051195502281189,
+          0.003050299361348152,
+          0.016440242528915405,
+          0.01002589799463749,
+          -0.010702192783355713,
+          -0.00901494175195694,
+          0.013435261324048042,
+          0.03438645973801613,
+          0.004789841361343861,
+          -0.021752992644906044,
+          0.001578312017954886,
+          0.004873506724834442,
+          0.010493029840290546,
+          -0.003998506814241409,
+          -0.01844821125268936,
+          0.005487052723765373,
+          0.011594623327255249,
+          -0.003092132043093443,
+          0.0028986558318138123,
+          -0.00843625608831644,
+          0.05429881438612938,
+          -0.009182272478938103,
+          0.01317032054066658,
+          0.0041205184534192085,
+          -0.019368529319763184,
+          -0.00744621641933918,
+          0.022017931565642357,
+          -0.0040089646354317665,
+          -0.00643526017665863,
+          -0.012605579569935799,
+          0.034274905920028687,
+          0.031235065311193466,
+          -0.03486056253314018,
+          0.017541835084557533,
+          -0.009579682722687721,
+          -0.014055779203772545,
+          0.007467132527381182,
+          -0.00782271008938551,
+          0.014216138049960136,
+          0.004636454861611128,
+          0.004761952906847,
+          0.010918328538537025,
+          -0.01878287084400654,
+          -0.011441237293183804,
+          0.003656873246654868,
+          0.0007006973028182983,
+          0.017960162833333015,
+          0.016384465619921684,
+          0.01050697360187769,
+          -0.009795818477869034,
+          0.018364544957876205,
+          0.023370521143078804,
+          0.0019609066657721996,
+          0.003187998663634062,
+          0.0096215158700943,
+          0.011887452565133572,
+          -0.006961654406040907,
+          -0.008847611024975777,
+          -0.015338648110628128,
+          0.0076762959361076355,
+          0.01885259337723255,
+          -0.015505978837609291,
+          0.003221116028726101,
+          0.013637452386319637,
+          -0.04261355474591255,
+          0.0191454216837883,
+          0.010018926113843918,
+          0.013407372869551182,
+          0.01278685498982668,
+          -0.022561756893992424,
+          0.004625997040420771,
+          -0.013470121659338474,
+          -0.018169326707720757,
+          0.002300797263160348,
+          -0.01430677529424429,
+          0.012480081990361214,
+          -0.0020515441428869963,
+          -0.006334164645522833,
+          -0.018057772889733315,
+          0.005640439223498106,
+          -0.005507969297468662,
+          -0.0011809015413746238,
+          -0.000824888062197715,
+          -0.011239045299589634,
+          -0.023412354290485382,
+          -0.03717530518770218,
+          0.016844624653458595,
+          0.01109263114631176,
+          -0.006511953193694353,
+          -0.008896416053175926,
+          -0.013463149778544903,
+          0.018824703991413116,
+          -0.004448208026587963,
+          0.00529183354228735,
+          -0.0115458182990551,
+          0.0037684268318116665,
+          -0.005141932982951403,
+          0.006609562784433365,
+          -0.006428287830203772,
+          0.01013047993183136,
+          0.020233070477843285,
+          0.008129483088850975,
+          -0.0475219190120697,
+          -0.005365040618926287,
+          0.009363546967506409,
+          -0.025768928229808807,
+          0.0037509966641664505,
+          0.027930283918976784,
+          0.008338646963238716,
+          0.002250249497592449,
+          0.01218028087168932,
+          0.0115458182990551,
+          -0.020902393385767937,
+          0.014906376600265503,
+          0.007501993328332901,
+          0.00018890068167820573,
+          0.0064213159494102,
+          0.0019347612978890538,
+          -0.0004309637297410518,
+          0.021460162475705147,
+          -0.007913348264992237,
+          0.0061807781457901,
+          -0.020023908466100693,
+          0.026828689500689507,
+          -0.019382473081350327,
+          -0.0014571715146303177,
+          0.025964148342609406,
+          0.023468131199479103,
+          0.00473406445235014,
+          0.015129484236240387,
+          -0.02618725597858429,
+          0.006086654495447874,
+          0.03246215730905533,
+          0.018169326707720757,
+          0.029645422473549843,
+          -0.014948209747672081,
+          -0.022770920768380165,
+          0.014090639539062977,
+          -0.015826696529984474,
+          -0.006337650585919619,
+          -0.014076695777475834,
+          0.008366535417735577,
+          0.0008431898895651102,
+          0.03820717707276344,
+          0.008450200781226158,
+          -0.026061756536364555,
+          0.029673311859369278,
+          -0.005385957192629576,
+          -0.029115542769432068,
+          0.020442234352231026,
+          -0.009440240450203419,
+          0.009238049387931824,
+          0.004082172177731991,
+          -0.016886457800865173,
+          0.025141438469290733,
+          -0.010276894085109234,
+          -0.004225100390613079,
+          0.01889442466199398,
+          0.027874507009983063,
+          0.0035244030877947807,
+          -0.028348609805107117,
+          0.010374504141509533,
+          -0.004953686147928238,
+          -0.0002974041854031384,
+          -0.0070836665108799934,
+          0.012152392417192459,
+          -0.016537850722670555,
+          0.0013874504948034883,
+          0.006825698539614677,
+          -0.024653390049934387,
+          -0.008603586815297604,
+          -0.001669820980168879,
+          -0.014153389260172844,
+          -0.03896016627550125,
+          0.002949203597381711,
+          -0.006623507011681795,
+          0.003548805369064212,
+          -0.005208168178796768,
+          0.009537850506603718,
+          -0.017290839925408363,
+          0.015101595781743526,
+          0.008324702270328999,
+          -0.0029770920518785715,
+          -0.0069372523576021194,
+          -0.00797609705477953,
+          -0.029143432155251503,
+          0.037454187870025635,
+          0.00015545632049907,
+          -0.0036254986189305782,
+          0.005588148254901171,
+          -0.014195221476256847,
+          0.00343725155107677,
+          0.02724701724946499,
+          0.02406773343682289,
+          -0.043784867972135544,
+          0.001980080036446452,
+          -0.008840639144182205,
+          -0.021766936406493187,
+          0.013491038233041763,
+          -0.003914841450750828,
+          0.0171235091984272,
+          0.0053510963916778564,
+          -0.029729088768363,
+          -0.00030350478482432663,
+          0.010269922204315662,
+          -0.012500998564064503,
+          0.0023984068538993597,
+          0.024751000106334686,
+          -0.00036146046477369964,
+          0.0031043333001434803,
+          -0.009440240450203419,
+          0.02254781313240528,
+          -0.01662151701748371,
+          0.015938250347971916,
+          0.006215638481080532,
+          0.006375997327268124,
+          0.014376496896147728,
+          -0.02005179598927498,
+          0.019982075318694115,
+          0.0038242037408053875,
+          0.004437749739736319,
+          0.0057066744193434715,
+          -0.001433640718460083,
+          0.021906377747654915,
+          -0.0003137450839858502,
+          0.006630479358136654,
+          0.016830680891871452,
+          -0.021083669736981392,
+          0.011413348838686943,
+          -0.006930280011147261,
+          -0.00016743964806664735,
+          -0.0015791835030540824,
+          -0.021083669736981392,
+          -0.027428291738033295,
+          0.035334669053554535,
+          0.001134711317718029,
+          0.00016090329154394567,
+          0.008087650872766972,
+          -0.013393428176641464,
+          -0.030203191563487053,
+          -0.02168327011168003,
+          -0.002867281436920166,
+          0.01281474344432354,
+          0.01762550137937069,
+          -0.017709165811538696,
+          0.007766933646053076,
+          0.011315738782286644,
+          0.02445817179977894,
+          0.03407968580722809,
+          -0.0011983318254351616,
+          0.020484067499637604,
+          0.026466140523552895,
+          0.009321714751422405,
+          0.014976098202168941,
+          -0.008415340445935726,
+          -0.00901494175195694,
+          0.012473110109567642,
+          0.015408368781208992,
+          0.011789842508733273,
+          0.0019678787793964148,
+          -0.024653390049934387,
+          -0.009795818477869034,
+          -0.010262950323522091,
+          0.006463148631155491,
+          -0.005745020695030689,
+          -0.001980080036446452,
+          0.02299402840435505,
+          -0.003981076180934906,
+          -0.018001995980739594,
+          -0.019270919263362885,
+          -0.016956178471446037,
+          -0.01233366783708334,
+          -0.009405380114912987,
+          0.015603587962687016,
+          -0.00584960263222456,
+          -0.015896417200565338,
+          -0.024095620959997177,
+          0.011957173235714436,
+          0.02158566191792488,
+          0.02694024331867695,
+          0.0003128735697828233,
+          -0.004622510634362698,
+          0.01677490398287773,
+          0.0057415347546339035,
+          0.005867032799869776,
+          -0.008931276388466358,
+          -0.0009673806489445269,
+          0.011657372117042542,
+          0.025252992287278175,
+          -0.010897411964833736,
+          -0.019996019080281258,
+          -0.032043829560279846,
+          0.0056125507690012455,
+          0.0004745394107885659,
+          -0.006905877497047186,
+          0.008575698360800743,
+          0.0024890443310141563,
+          0.004608566407114267,
+          0.013574703596532345,
+          -0.006926794070750475,
+          0.009328686632215977,
+          -0.029673311859369278,
+          0.007746017072349787,
+          0.011531874537467957,
+          0.02953387051820755,
+          -0.05231873318552971,
+          -0.019173309206962585,
+          0.0007743402966298163,
+          0.046880487352609634,
+          0.01660757325589657,
+          0.013644424267113209,
+          -0.017555780708789825,
+          -0.011350599117577076,
+          -0.013790839351713657,
+          -0.011057770811021328,
+          -0.0211673341691494,
+          -0.010269922204315662,
+          0.020804785192012787,
+          -0.006337650585919619,
+          0.0421394482254982,
+          0.029812753200531006,
+          -0.004646913148462772,
+          -0.0010850350372493267,
+          -0.01448805071413517,
+          0.014669325202703476,
+          -0.009412351995706558,
+          0.03870917111635208,
+          0.004476096481084824,
+          0.012326695024967194,
+          -0.009021913632750511,
+          -0.00043771794298663735,
+          -0.0496414415538311,
+          -0.015743030235171318,
+          0.017095619812607765,
+          -0.03207171708345413,
+          0.00023116912052500993,
+          0.030370522290468216,
+          0.020679285749793053,
+          -0.00015338648518081754,
+          -0.01228486280888319,
+          0.02608964592218399,
+          0.025503989309072495,
+          0.007962152361869812,
+          0.002156125847250223,
+          -0.01752789132297039,
+          -0.007481076754629612,
+          -0.0056683276779949665,
+          -0.012250002473592758,
+          0.00782271008938551,
+          -0.027470124885439873,
+          0.021976100280880928,
+          -0.012807771563529968,
+          -0.019870521500706673,
+          0.029505981132388115,
+          -0.009049802087247372,
+          0.02187849022448063,
+          0.009565738961100578,
+          -0.0139023931697011,
+          0.03717530518770218,
+          0.011106575839221478,
+          0.011176296509802341,
+          -0.008903387933969498,
+          0.014627492986619473,
+          -0.012033866718411446,
+          0.028585663065314293,
+          -0.017667334526777267,
+          0.015882473438978195,
+          0.006773407571017742,
+          -0.00792729202657938,
+          0.00477589713409543,
+          0.031179288402199745,
+          0.012821715325117111,
+          -0.004908367525786161,
+          0.028139447793364525,
+          0.007941236719489098,
+          0.022073708474636078,
+          0.004768925253301859,
+          0.008220121264457703,
+          0.027930283918976784,
+          0.02786056324839592,
+          -0.010172312147915363,
+          -0.03184860944747925,
+          -0.007257969584316015,
+          -0.005009463056921959,
+          0.018768927082419395,
+          -0.018225103616714478,
+          -0.004472610540688038,
+          -0.0027365542482584715,
+          0.0128426318988204,
+          -0.008854582905769348,
+          0.010960161685943604,
+          0.0010649901814758778,
+          -0.032294824719429016,
+          -0.02406773343682289,
+          -0.03678486868739128,
+          0.0014231825480237603,
+          -0.021920323371887207,
+          -0.015687253326177597,
+          -6.411075446521863e-05,
+          -0.027037853375077248,
+          0.0013708916958421469,
+          0.027428291738033295,
+          -0.008359563536942005,
+          -0.012368528172373772,
+          0.005915837828069925,
+          0.014571716077625751,
+          -0.0057241045869886875,
+          -0.004005478695034981,
+          0.0029021420050412416,
+          -0.0005425175186246634,
+          4.28131315857172e-05,
+          0.010353587567806244,
+          -0.010221117176115513,
+          -0.036087654531002045,
+          -0.02668924815952778,
+          0.013435261324048042,
+          0.0019086158135905862,
+          0.00792729202657938,
+          0.013658368960022926,
+          0.003020667936652899,
+          0.0001337774156127125,
+          0.011768925935029984,
+          0.00047628243919461966,
+          0.003034612163901329,
+          -0.0013830928364768624,
+          0.019033867865800858,
+          0.008617531508207321,
+          -0.015519922599196434,
+          0.018978090956807137,
+          0.002586653921753168,
+          -0.005487052723765373,
+          -0.010332670994102955,
+          -0.004078685771673918,
+          -0.010995022021234035,
+          -0.022045820951461792,
+          0.005577689968049526,
+          0.0076414356008172035,
+          -0.012319723144173622,
+          0.00911952368915081,
+          -0.020637454465031624,
+          0.016830680891871452,
+          0.026814745739102364,
+          0.009335658513009548,
+          -0.030203191563487053,
+          -0.00563695328310132,
+          -0.012912352569401264,
+          -0.01286354847252369,
+          -0.019996019080281258,
+          0.004333168268203735,
+          0.019577693194150925,
+          0.008833667263388634,
+          0.0057798814959824085,
+          0.01716534234583378,
+          0.004036853089928627,
+          -0.010967133566737175,
+          0.0015512950485572219,
+          0.0005643054028041661,
+          -0.0058705187402665615,
+          -0.0182808805257082,
+          0.008526894263923168,
+          -0.001230577938258648,
+          -0.009293826296925545,
+          0.012187252752482891,
+          0.015254982747137547,
+          0.00655727181583643,
+          0.0024332674220204353,
+          0.0171235091984272,
+          -0.00011231638200115412,
+          -0.004368028603494167,
+          0.016272911801934242,
+          0.07898009568452835,
+          0.00919621717184782,
+          0.013707173988223076,
+          0.0021038351114839315,
+          -0.02045617811381817,
+          -0.014934265054762363,
+          -0.0037998014595359564,
+          -0.04255777597427368,
+          0.013484066352248192,
+          0.02005179598927498,
+          -0.004497012589126825,
+          -0.013163348659873009,
+          -0.012152392417192459,
+          -0.00866633653640747,
+          -0.001097236294299364,
+          0.03399602323770523,
+          -0.008185259997844696,
+          0.014808767475187778,
+          0.038318730890750885,
+          0.01878287084400654,
+          -0.006288845557719469,
+          0.00964940432459116,
+          0.025197215378284454,
+          -0.010437252931296825,
+          0.00477589713409543,
+          -0.004971116315573454,
+          0.005654383450746536,
+          0.011008965782821178,
+          0.010771914385259151,
+          0.01814143732190132,
+          0.010304782539606094,
+          -0.0021770421881228685,
+          -0.01950797066092491,
+          0.005511455237865448,
+          -0.011455181054770947,
+          0.013302790932357311,
+          -0.0030572714749723673,
+          -0.0012018178822472692,
+          -0.02629880979657173,
+          0.012905380688607693,
+          0.009461157023906708,
+          -0.013539843261241913,
+          -0.00949601735919714,
+          0.002987550338730216,
+          -0.007278885692358017,
+          0.014320719987154007,
+          -0.010409364476799965,
+          0.0105836670845747,
+          0.009056774899363518,
+          -0.005103586707264185,
+          -0.015380480326712132,
+          -0.001689865835942328,
+          -0.004890936892479658,
+          0.008094622753560543,
+          -0.010988050140440464,
+          -0.011992033571004868,
+          0.015394425019621849,
+          0.012263946235179901,
+          0.012556775473058224,
+          0.017109565436840057,
+          -0.012626496143639088,
+          0.008847611024975777,
+          -0.0028637952636927366,
+          -0.016454186290502548,
+          -0.002661603968590498,
+          -0.018517931923270226,
+          0.011315738782286644,
+          0.004908367525786161,
+          0.005476594436913729,
+          -0.010569723322987556,
+          0.00014347300748340786,
+          -0.006637451238930225,
+          0.008331675082445145,
+          -0.0035331181716173887,
+          -0.0016166586428880692,
+          -0.012096615508198738,
+          -0.010576695203781128,
+          -0.0035052297171205282,
+          0.02314741350710392,
+          0.011789842508733273,
+          0.0009577939636074007,
+          0.009530877694487572,
+          0.0072719138115644455,
+          -0.007031375542283058,
+          -0.013414344750344753,
+          0.002541335066780448,
+          -0.001469372771680355,
+          0.005996016785502434,
+          0.021348608657717705,
+          -0.010520918294787407,
+          -0.0071394434198737144,
+          0.0020811755675822496,
+          0.016747014597058296,
+          0.0058530885726213455,
+          0.0009325200808234513,
+          0.023468131199479103,
+          0.0003483877517282963,
+          0.0030572714749723673,
+          -0.01864342950284481,
+          -0.015394425019621849,
+          0.010346615687012672,
+          0.026675302535295486,
+          0.00863844808191061,
+          0.0010275151580572128,
+          -0.017193229869008064,
+          0.0026162853464484215,
+          0.027372514829039574,
+          0.026019925251603127,
+          0.007543826010078192,
+          -0.0027417833916842937,
+          0.0009037601412273943,
+          -0.010151396505534649,
+          -0.0001501182996435091,
+          0.028390442952513695,
+          -0.015185261145234108,
+          -0.0006593004218302667,
+          -0.005256973206996918,
+          -0.013281874358654022,
+          -0.017151396721601486,
+          0.00957271084189415,
+          0.027233071625232697,
+          0.006222610827535391,
+          0.010409364476799965,
+          -0.01995418593287468,
+          0.01334462407976389,
+          -0.009007969871163368,
+          -0.008282870054244995,
+          -0.009265937842428684,
+          -0.021599605679512024,
+          -0.014376496896147728,
+          -0.011036854237318039,
+          -0.0032228592317551374,
+          -0.014766935259103775,
+          -0.0002703872451093048,
+          0.002027141861617565,
+          0.005727590527385473,
+          -0.003156624035909772,
+          -0.0036080684512853622,
+          -0.0061807781457901,
+          0.010548806749284267,
+          -0.00429482152685523,
+          0.0037684268318116665,
+          -0.023942235857248306,
+          -0.026507971808314323,
+          0.01839243434369564,
+          -0.008422312326729298,
+          -0.00397759024053812,
+          -0.015003986656665802,
+          -0.004782869480550289,
+          -0.023691238835453987,
+          0.01802988350391388,
+          -0.023523908108472824,
+          -0.010590638965368271,
+          0.004381972830742598,
+          0.013511954806745052,
+          -0.016747014597058296,
+          -0.01532470341771841,
+          -0.0022711658384650946,
+          -0.01228486280888319,
+          0.004256474785506725,
+          -0.03380080312490463,
+          0.007578686345368624,
+          -0.000818351749330759,
+          0.012975101359188557,
+          -0.020735062658786774,
+          0.02491833083331585,
+          0.0033117535058408976,
+          -0.013009962625801563,
+          -0.008373507298529148,
+          -0.007223108783364296,
+          -0.0026563750579953194,
+          0.005912351422011852,
+          0.001685508294031024,
+          -0.014766935259103775,
+          0.02081872895359993,
+          0.0016959664644673467,
+          0.009586654603481293,
+          -0.01112051960080862,
+          0.00803187396377325,
+          0.011162352748215199,
+          0.005082670133560896,
+          -0.003649901133030653,
+          0.007648407481610775,
+          -0.0031513948924839497,
+          0.019368529319763184,
+          0.020637454465031624,
+          -0.006341136526316404,
+          0.008038845844566822,
+          0.004015936981886625,
+          -0.017193229869008064,
+          -0.007034861948341131,
+          0.00405428372323513,
+          0.0029126000590622425,
+          -0.018727093935012817,
+          0.025099605321884155,
+          0.027009963989257812,
+          0.001078934408724308,
+          -0.02522510476410389,
+          0.010925300419330597,
+          0.011001993902027607,
+          0.014529882930219173,
+          0.02147410809993744,
+          -0.002708665793761611,
+          -0.023175302892923355,
+          0.009607571177184582,
+          -0.008415340445935726,
+          -0.01920119859278202,
+          0.031430285423994064,
+          0.022003987804055214,
+          0.011127491481602192,
+          -0.05491235852241516,
+          0.003093875013291836,
+          0.01579880714416504,
+          -0.013609563931822777,
+          -0.02953387051820755,
+          -0.011420320719480515,
+          -0.009935260750353336,
+          0.0029091141186654568,
+          0.013909365050494671,
+          -0.015213149599730968,
+          -0.01752789132297039,
+          0.014934265054762363,
+          0.030928291380405426,
+          0.0275119561702013,
+          0.025057774037122726,
+          -0.023426298052072525,
+          0.001805777195841074,
+          -0.025113550946116447,
+          0.00640737172216177,
+          -0.016077691689133644,
+          -0.005867032799869776,
+          0.0123615562915802,
+          -0.007613547146320343,
+          0.020735062658786774,
+          0.002405378967523575,
+          0.02183665707707405,
+          0.017695222049951553,
+          0.03692431002855301,
+          -0.016593627631664276,
+          -0.030231080949306488,
+          -0.011329683475196362,
+          -0.014048807322978973,
+          0.023370521143078804,
+          0.01789044216275215,
+          0.00967729277908802,
+          0.028446219861507416,
+          0.02928287349641323,
+          -0.0020550303161144257,
+          0.00960059929639101,
+          -2.5587109121261165e-05,
+          8.219467417802662e-05,
+          -0.018113549798727036,
+          -0.02314741350710392,
+          -0.0021630979608744383,
+          0.00967729277908802,
+          0.013923308812081814,
+          -0.005605578422546387,
+          -0.0027278391644358635,
+          0.021097613498568535,
+          -0.0032629487104713917,
+          -0.0027905881870537996,
+          -0.03555777668952942,
+          0.0018842134159058332,
+          -0.008924304507672787,
+          0.021808769553899765,
+          -0.00020491474424488842,
+          -0.008094622753560543,
+          -0.027525901794433594,
+          -0.004918825346976519,
+          0.034721121191978455,
+          0.005950698163360357,
+          0.00012168515240773559,
+          0.011469125747680664,
+          0.024193231016397476,
+          0.010513946413993835,
+          0.016328688710927963,
+          -0.001948705525137484,
+          0.001576569047756493,
+          0.022784864529967308,
+          -0.006644423585385084,
+          0.02187849022448063,
+          0.003827689914032817,
+          -0.013044822961091995,
+          -0.0017866039415821433,
+          -0.0019609066657721996,
+          -0.008659363724291325,
+          0.009900400415062904,
+          0.015533867292106152,
+          -0.011810759082436562,
+          0.018406378105282784,
+          -0.016021914780139923,
+          -0.0012401645071804523,
+          -0.023970123380422592,
+          -0.0005242157494649291,
+          0.013651397079229355,
+          -0.00317928334698081,
+          0.004985060542821884,
+          -0.019521916285157204,
+          0.0057624513283371925,
+          0.014111556112766266,
+          -0.012675301171839237,
+          0.004761952906847,
+          -0.02158566191792488,
+          0.006138945464044809,
+          0.011531874537467957,
+          -0.001174801029264927,
+          -0.009412351995706558,
+          -0.005385957192629576,
+          0.01165040023624897,
+          -0.015882473438978195,
+          0.004971116315573454,
+          0.01797410659492016,
+          0.012054783292114735,
+          0.010018926113843918,
+          0.006860558874905109,
+          0.00911952368915081,
+          0.008443228900432587,
+          0.012194225564599037,
+          -0.036952197551727295,
+          0.018071716651320457,
+          -0.01430677529424429,
+          0.009440240450203419,
+          -0.0017552294302731752,
+          -0.007829682901501656,
+          -0.008171316236257553,
+          -0.009175300598144531,
+          0.023133469745516777,
+          -0.004807271528989077,
+          0.0019243031274527311,
+          -0.00861055962741375,
+          0.030231080949306488,
+          0.016175301745533943,
+          -0.006484064739197493,
+          0.025768928229808807,
+          -0.004922311753034592,
+          -0.0020567732863128185,
+          -0.012431276962161064,
+          -0.004458666313439608,
+          0.01342828944325447,
+          -0.011643428355455399,
+          0.01600797101855278,
+          -0.04264144226908684,
+          0.0059088654816150665,
+          -0.011022910475730896,
+          -0.005121016874909401,
+          -0.016189245507121086,
+          -0.0171235091984272,
+          -0.030258968472480774,
+          0.02502988465130329,
+          0.00914741214364767,
+          -0.027079686522483826,
+          -0.0005503611755557358,
+          -0.006679283920675516,
+          0.0005150648066774011,
+          -0.01109263114631176,
+          -0.015812750905752182,
+          0.016551796346902847,
+          0.03787251561880112,
+          -0.012005978263914585,
+          -0.01367231272161007,
+          0.003060757415369153,
+          0.008450200781226158,
+          -0.017444226890802383,
+          -0.006375997327268124,
+          0.0019765938632190228,
+          -0.028097614645957947,
+          0.020274903625249863,
+          -0.005751993041485548,
+          0.0031374506652355194,
+          -0.017346616834402084,
+          0.01600797101855278,
+          -0.015394425019621849,
+          -0.0060587660409510136,
+          -0.02253386750817299,
+          -0.023523908108472824,
+          0.016342632472515106,
+          -0.033466141670942307,
+          -0.011315738782286644,
+          -0.01853187568485737,
+          0.004444722086191177,
+          -0.00848506111651659,
+          -0.001322086900472641,
+          0.009000997990369797,
+          0.00419023958966136,
+          0.032852593809366226,
+          -0.00848506111651659,
+          0.004137949086725712,
+          0.03307570144534111,
+          -0.008394423872232437,
+          0.024709166958928108,
+          0.0022641937248408794,
+          0.023328689858317375,
+          0.009454185143113136,
+          0.0013752492377534509,
+          -0.007369523402303457,
+          0.010925300419330597,
+          -0.0014310261467471719,
+          0.03731474652886391,
+          -0.011901396326720715,
+          -0.018922314047813416,
+          0.00861055962741375,
+          -0.004890936892479658,
+          0.006403885781764984,
+          0.02846016362309456,
+          0.019382473081350327,
+          -0.005738048814237118,
+          0.008275898173451424,
+          -0.01157370675355196,
+          -0.0025918828323483467,
+          0.008059762418270111,
+          0.011713149026036263,
+          0.017039842903614044,
+          0.0002762699732556939,
+          -0.019982075318694115,
+          0.02112550102174282,
+          0.0009569224785082042,
+          -0.006428287830203772,
+          -0.015213149599730968,
+          0.0078087663277983665,
+          -0.006972112692892551,
+          0.00922410562634468,
+          -0.00840836763381958,
+          0.016830680891871452,
+          0.0027034366503357887,
+          -0.02526693604886532,
+          0.0031130483839660883,
+          0.00911952368915081,
+          -0.002987550338730216,
+          -0.004950200207531452,
+          -0.016844624653458595,
+          -0.010904384776949883,
+          0.0058530885726213455,
+          0.015812750905752182,
+          -0.0054068733006715775,
+          -0.02628486417233944,
+          -0.005581176374107599,
+          0.00018236432515550405,
+          0.0008924304274842143,
+          0.014167333021759987,
+          0.0018388947937637568,
+          -0.02320319041609764,
+          0.017960162833333015,
+          -0.0017395421164110303,
+          0.004650399088859558,
+          -0.009614543057978153,
+          0.026410363614559174,
+          -0.004660857375711203,
+          -0.011587651446461678,
+          0.010799802839756012,
+          0.034721121191978455,
+          -0.0019452194683253765,
+          0.01868526265025139,
+          -0.02370518259704113,
+          -0.002764442702755332,
+          -0.00901494175195694,
+          0.004193725995719433,
+          -0.01532470341771841,
+          -0.0007220494444482028,
+          0.016133468598127365,
+          0.0023809764534235,
+          -0.010269922204315662,
+          0.002013197634369135,
+          0.024332674220204353,
+          -0.0009063746547326446,
+          0.003261205740272999,
+          0.015687253326177597,
+          0.005731076467782259,
+          0.018629485741257668,
+          0.011476097628474236,
+          -0.03792829439043999,
+          0.002319970401003957,
+          0.010723109357059002,
+          0.0031339647248387337,
+          0.002788844984024763,
+          0.014320719987154007,
+          -0.0053894431330263615,
+          -0.0061807781457901,
+          -0.0030468131881207228,
+          0.013023906387388706,
+          0.009893427602946758,
+          0.008101594634354115,
+          0.0035906380508095026,
+          0.0022746517788618803,
+          -0.018517931923270226,
+          0.013121516443789005,
+          0.0018458669073879719,
+          0.0069023915566504,
+          0.014739046804606915,
+          -0.015980081632733345,
+          -0.005469622556120157,
+          -0.009928288869559765,
+          0.004197211936116219,
+          0.0017953190254047513,
+          -0.004050797317177057,
+          0.004800299648195505,
+          -0.001281997188925743,
+          0.0014101098058745265,
+          -0.014599604532122612,
+          0.006389941554516554,
+          0.01864342950284481,
+          0.0008105080923996866,
+          -0.004981574602425098,
+          0.03661753609776497,
+          -0.01783466525375843,
+          0.010988050140440464,
+          -0.012061755172908306,
+          0.014153389260172844,
+          -0.02597809210419655,
+          0.041358571499586105,
+          0.016858568415045738,
+          0.00477589713409543,
+          0.006686256267130375,
+          -0.01079283095896244,
+          -0.007522909436374903,
+          -0.01218028087168932,
+          0.013769922778010368,
+          0.013846616260707378,
+          -0.015966137871146202,
+          -0.0008793577435426414,
+          -0.0018127493094652891,
+          -0.0011164095485582948,
+          0.02157171629369259,
+          0.006484064739197493,
+          -0.014697213657200336,
+          0.008840639144182205,
+          -0.0018894425593316555,
+          -0.008791834115982056,
+          -0.01869920641183853,
+          -0.008213148452341557,
+          0.014850599691271782,
+          -0.0072161369025707245,
+          -0.0022955681197345257,
+          0.008708168752491474,
+          0.004155379254370928,
+          0.000675423420034349,
+          -0.018420321866869926,
+          0.019577693194150925,
+          -0.011176296509802341,
+          0.006602590903639793,
+          -0.001901643699966371,
+          -0.02684263326227665,
+          0.015617532655596733,
+          -0.019828688353300095,
+          0.015227094292640686,
+          -0.00016624131239950657,
+          -0.027539845556020737,
+          0.0003377117100171745,
+          0.004664343316107988,
+          -0.023342633619904518,
+          -0.020442234352231026,
+          0.005110558588057756,
+          -0.0008105080923996866,
+          -0.004169323481619358,
+          -0.03352191671729088,
+          0.03681275621056557,
+          0.015826696529984474,
+          0.03879283368587494,
+          -0.019312752410769463,
+          -0.006187750026583672,
+          0.0034790842328220606,
+          -0.00667579798027873,
+          0.003147908952087164,
+          -0.026577694341540337,
+          0.016788847744464874,
+          0.022505979984998703,
+          0.013358567841351032,
+          -0.011866535991430283,
+          -0.009635459631681442,
+          -0.004490040708333254,
+          -0.008059762418270111,
+          0.014641436748206615,
+          -0.006672312039881945,
+          -0.013797811232507229,
+          -0.003052042331546545,
+          0.0004989418084733188,
+          0.003393675899133086,
+          0.016733070835471153,
+          -0.004967630375176668,
+          0.006229582708328962,
+          -0.0043436260893940926,
+          0.00048674060963094234,
+          -0.028223112225532532,
+          0.028892435133457184,
+          -0.021195223554968834,
+          0.03692431002855301,
+          -0.005713646300137043,
+          0.003475598292425275,
+          -0.005180279724299908,
+          0.007801793981343508,
+          0.018503988161683083,
+          -0.002971863141283393,
+          -0.010144423693418503,
+          -0.01772311143577099,
+          7.070157880662009e-05,
+          0.015589644201099873,
+          -0.0016062004724517465,
+          -0.0036882476415485144,
+          0.0189502015709877,
+          1.2528014849522151e-05,
+          -0.008708168752491474,
+          -0.009719124995172024,
+          -0.002668576082214713,
+          0.004842132329940796,
+          0.011587651446461678,
+          0.0275119561702013,
+          0.008017929270863533,
+          0.0029178292024880648,
+          0.006184264086186886,
+          -0.001894671586342156,
+          -0.02678685635328293,
+          -0.010067731142044067,
+          0.01266135647892952,
+          -0.029422316700220108,
+          0.0053720129653811455,
+          0.010988050140440464,
+          0.019061755388975143,
+          0.011134464293718338,
+          -0.021055780351161957,
+          -0.0006266186246648431,
+          0.01681673526763916,
+          -0.006145917344838381,
+          0.029756976291537285,
+          -0.01777888834476471,
+          0.011183268390595913,
+          0.026424307376146317,
+          0.018768927082419395,
+          -0.008847611024975777,
+          -0.01797410659492016,
+          0.008303786627948284,
+          0.0072509972378611565,
+          -0.001050174469128251,
+          0.008924304507672787,
+          -0.04043825343251228,
+          0.004451693966984749,
+          -0.01858765259385109,
+          0.0148645443841815,
+          0.004448208026587963,
+          0.011287850327789783,
+          0.006484064739197493,
+          -0.003172311233356595,
+          0.001901643699966371,
+          0.005528885405510664,
+          0.0006017804844304919,
+          -0.0276095662266016,
+          -0.00949601735919714,
+          -0.0085617545992136,
+          0.0012558518210425973,
+          -0.010262950323522091,
+          -0.0010606326395645738,
+          0.01570119708776474,
+          -0.007285858038812876,
+          0.002699950709939003,
+          -0.041051801294088364,
+          -0.013644424267113209,
+          -0.010632472112774849,
+          -0.012089643627405167,
+          -0.011357571929693222,
+          0.019215142354369164,
+          -0.00269820773974061,
+          -0.01010956335812807,
+          0.008331675082445145,
+          0.005469622556120157,
+          -0.030565742403268814,
+          -0.008603586815297604,
+          0.00922410562634468,
+          -0.006247012875974178,
+          0.0040647415444254875,
+          0.005103586707264185,
+          -0.012026894837617874,
+          -0.024890443310141563,
+          0.009956177324056625,
+          -0.012884464114904404,
+          0.016440242528915405,
+          0.019842632114887238,
+          0.0007159488159231842,
+          -0.005535857751965523,
+          0.0073276907205581665,
+          0.01023506186902523,
+          -0.005880977027118206,
+          0.01677490398287773,
+          -0.010820719413459301,
+          0.01635657623410225,
+          0.014223109930753708,
+          0.01844821125268936,
+          -6.938069418538362e-06,
+          -0.014557771384716034,
+          -0.00029609689954668283,
+          0.006749005056917667,
+          0.00505129573866725,
+          -0.00577290914952755,
+          -0.004197211936116219,
+          -0.012354583479464054,
+          0.010932273231446743,
+          -0.007264941465109587,
+          -0.006909363903105259,
+          -0.001948705525137484,
+          0.00015905132750049233,
+          -0.0050756982527673244,
+          -0.004577192012220621,
+          -0.002827191725373268,
+          -0.014766935259103775,
+          0.0006484065088443458,
+          -0.026661358773708344,
+          -0.00863844808191061,
+          -0.007244025357067585,
+          0.0016645919531583786,
+          -0.006341136526316404,
+          0.0030747016426175833,
+          0.04913945123553276,
+          -0.01446016225963831,
+          0.003154881065711379,
+          -0.003660359187051654,
+          -0.02380279265344143,
+          -0.018406378105282784,
+          0.00012734999472741038,
+          -0.009356575086712837,
+          -0.026480084285140038,
+          -0.016370519995689392,
+          0.011992033571004868,
+          -0.0036708174739032984,
+          0.0009708667057566345,
+          -0.010848607867956161,
+          -0.01717928610742092,
+          -0.02643825113773346,
+          -0.005657869391143322,
+          -0.0021142931655049324,
+          -0.010667332448065281,
+          -0.015589644201099873,
+          0.006473606918007135,
+          -0.010653388686478138,
+          0.009781873784959316,
+          0.02360757440328598,
+          0.004364542663097382,
+          0.019312752410769463,
+          0.00901494175195694,
+          -0.018266934901475906,
+          -0.0015434514498338103,
+          -0.0037998014595359564,
+          -0.018015939742326736,
+          0.006693228147923946,
+          0.01066036056727171,
+          -0.006755977403372526,
+          0.011008965782821178,
+          -0.007578686345368624,
+          -0.014725102111697197,
+          -0.004514443222433329,
+          -0.019968131557106972,
+          0.029003988951444626,
+          -0.02582470513880253,
+          -0.005005977116525173,
+          -0.003897411050274968,
+          -0.012849603779613972,
+          0.027079686522483826,
+          0.0128426318988204,
+          -0.019312752410769463,
+          0.024346617981791496,
+          0.0040473113767802715,
+          0.008366535417735577,
+          0.01374900620430708,
+          0.011789842508733273,
+          0.0007586530409753323,
+          0.011322710663080215,
+          0.027372514829039574,
+          -0.0071952203288674355,
+          -0.030258968472480774,
+          -0.002189243445172906,
+          0.01853187568485737,
+          0.014906376600265503,
+          -0.006909363903105259,
+          -0.002267679665237665,
+          -0.00014162104343995452,
+          0.017249006778001785,
+          -0.001689865835942328,
+          -0.000326599896652624,
+          -0.0008667207439430058,
+          -0.009007969871163368,
+          -0.007683268282562494,
+          0.006424801889806986,
+          -0.026173310354351997,
+          -0.0035278890281915665,
+          -0.02224103920161724,
+          0.011350599117577076,
+          -0.013023906387388706,
+          -0.002933516399934888,
+          -0.007725100964307785,
+          -0.005856574513018131,
+          0.027135463431477547,
+          0.021097613498568535,
+          -0.006700200494378805,
+          0.011113547720015049,
+          -0.008687252178788185,
+          -0.012187252752482891,
+          0.014711158350110054,
+          -0.0018476098775863647,
+          0.0014519424876198173,
+          -0.0021282373927533627,
+          0.005769423209130764,
+          -0.011232073418796062,
+          0.0013046566164121032,
+          0.022059764713048935,
+          0.024025900289416313,
+          -0.01367231272161007,
+          -0.020958170294761658,
+          -0.014934265054762363,
+          0.021892433986067772,
+          -0.0010449454421177506,
+          -0.011252989992499352,
+          0.0074741048738360405,
+          0.003100847126916051,
+          -0.010214145295321941,
+          -0.015715142711997032,
+          0.016300799325108528,
+          -0.02137649804353714,
+          -0.014446217566728592,
+          -0.014780879020690918,
+          -0.0036777895875275135,
+          -0.031179288402199745,
+          0.019326696172356606,
+          0.016886457800865173,
+          -0.004817729815840721,
+          -0.002276394749060273,
+          0.01631474308669567,
+          0.00587749108672142,
+          -0.00845717266201973,
+          -0.011720121838152409,
+          -0.013414344750344753,
+          -0.0017273409757763147,
+          0.0030293830204755068,
+          -0.015631476417183876,
+          -0.0017848608549684286,
+          0.0009211904252879322,
+          0.00805279053747654,
+          0.006030877586454153,
+          -0.0018493529642000794,
+          0.0016733070369809866,
+          -0.002238048240542412,
+          0.031792834401130676,
+          0.012975101359188557,
+          -0.00810856744647026,
+          0.007501993328332901,
+          -0.005535857751965523,
+          0.012696217745542526,
+          -0.003604582278057933,
+          -0.023328689858317375,
+          -0.02705179713666439,
+          -0.01382569968700409,
+          -0.004036853089928627,
+          -0.0043436260893940926,
+          -0.006009961012750864,
+          -0.014390440657734871,
+          0.006498008966445923,
+          -0.008157371543347836,
+          0.013281874358654022,
+          0.005197709891945124,
+          -0.002307769376784563,
+          0.014739046804606915,
+          0.01991235464811325,
+          0.012110560201108456,
+          -0.01446016225963831,
+          -0.014390440657734871,
+          0.012438248842954636,
+          -0.008729085326194763,
+          -0.002525647869333625,
+          0.020581677556037903,
+          0.013658368960022926,
+          -0.01436255220323801,
+          -0.0221294853836298,
+          0.005033865571022034,
+          -0.017695222049951553,
+          0.011950201354920864,
+          0.003034612163901329,
+          0.006651395466178656,
+          0.01849004253745079,
+          0.014501994475722313,
+          -0.010444224812090397,
+          0.003442480694502592,
+          0.011427292600274086,
+          -0.013386456295847893,
+          -0.030816737562417984,
+          -0.02518327161669731,
+          -0.004420319572091103,
+          -0.016175301745533943,
+          -0.03120717592537403,
+          0.0006919822189956903,
+          -0.007397411856800318,
+          0.017555780708789825,
+          0.009433268569409847,
+          0.017388449981808662,
+          0.021641438826918602,
+          0.0078087663277983665,
+          0.004950200207531452,
+          0.018657373264431953,
+          0.004699204117059708,
+          -0.0037440245505422354,
+          -0.006745519116520882,
+          -0.004751494619995356,
+          0.004723606165498495,
+          -0.01175498217344284,
+          0.0005673556588590145,
+          -0.011448209173977375,
+          0.020930282771587372,
+          0.018322711810469627,
+          0.005783367436379194,
+          0.0010597611544653773,
+          0.009942232631146908,
+          0.0016881227493286133,
+          0.00901494175195694,
+          0.009161355905234814,
+          -0.010709165595471859,
+          -0.0003250747686251998,
+          -0.013030878268182278,
+          -0.02816733531653881,
+          0.0090428302064538,
+          0.004472610540688038,
+          -0.0005625623743981123,
+          0.008645419962704182,
+          -0.00746016064658761,
+          -0.02380279265344143,
+          -0.0030136958230286837,
+          0.0062051801942288876,
+          -0.013016934506595135,
+          -0.006717630662024021,
+          -0.0014414843171834946,
+          0.007174304220825434,
+          0.023482074961066246,
+          -0.00970518123358488,
+          0.021139446645975113,
+          0.019745023921132088,
+          -0.0003472983662504703,
+          -0.008526894263923168,
+          -0.019745023921132088,
+          -0.033968131989240646,
+          -0.015575699508190155,
+          0.005358068738132715,
+          -0.0305936299264431,
+          -0.005434761755168438,
+          -0.007055778056383133,
+          -0.00016014071297831833,
+          -0.009656376205384731,
+          0.01023506186902523,
+          -0.0001039280541590415,
+          -0.00861055962741375,
+          0.013233070261776447,
+          -0.0034006480127573013,
+          0.005378984846174717,
+          -0.006923308130353689,
+          0.005392929073423147,
+          -0.01436255220323801,
+          -0.0011870021698996425,
+          0.001350846840068698,
+          0.014599604532122612,
+          -0.02056773193180561,
+          -0.011001993902027607,
+          0.0040298812091350555,
+          -0.0010562750976532698,
+          -0.006323706358671188,
+          -0.0201075728982687,
+          0.012668329291045666,
+          -0.03619920834898949,
+          0.008805778808891773,
+          -0.0035122018307447433,
+          0.002835906809195876,
+          -0.006278387736529112,
+          -0.021237054839730263,
+          -0.004430777858942747,
+          0.006337650585919619,
+          -0.019745023921132088,
+          0.0024158370215445757,
+          -0.01975896768271923,
+          0.010939245112240314,
+          -0.007725100964307785,
+          0.016454186290502548,
+          -0.0123615562915802,
+          -5.822803359478712e-05,
+          0.0030747016426175833,
+          0.01451593916863203,
+          -0.002084661740809679,
+          -0.0022397912107408047,
+          0.0024332674220204353,
+          -0.0017630730289965868,
+          0.003043327247723937,
+          -0.010925300419330597,
+          0.006982570979744196,
+          -0.006449204403907061,
+          0.01121812965720892,
+          0.005086156073957682,
+          0.019786855205893517,
+          0.006877989042550325,
+          0.023007972165942192,
+          -0.020846616476774216,
+          0.006072710268199444,
+          -0.015045818872749805,
+          0.018155381083488464,
+          0.013762950897216797,
+          0.02528088167309761,
+          0.011162352748215199,
+          -0.02582470513880253,
+          0.009830678813159466,
+          0.028501996770501137,
+          0.022017931565642357,
+          -0.008519921451807022,
+          -0.01929880864918232,
+          0.0026703192852437496,
+          0.03990837186574936,
+          0.01340040098875761,
+          0.00011972424545092508,
+          -0.011225101538002491,
+          0.016370519995689392,
+          0.00531972199678421,
+          -0.008631475269794464,
+          0.01492032129317522,
+          -0.02588048204779625,
+          0.0006937252474017441,
+          0.005738048814237118,
+          0.011190241202712059,
+          -0.022868528962135315,
+          0.0160637479275465,
+          0.014697213657200336,
+          0.026828689500689507,
+          -0.01980080083012581,
+          0.0015434514498338103,
+          -0.0033065243624150753,
+          0.008164344355463982,
+          -0.0070488061755895615,
+          0.002985807368531823,
+          0.01342828944325447,
+          -0.010883468203246593,
+          -0.04280877113342285,
+          -0.0006793452193960547,
+          0.009265937842428684,
+          0.016691237688064575,
+          -0.0032629487104713917,
+          0.005072211846709251,
+          -0.023970123380422592,
+          -0.020372513681650162,
+          -0.01053486205637455,
+          0.024806777015328407,
+          -0.0014911605976521969,
+          0.028041837736964226,
+          -0.0002359624340897426,
+          -0.009747013449668884,
+          -0.007101096678525209,
+          0.014083667658269405,
+          -0.005044323392212391,
+          0.007948208600282669,
+          0.03182072192430496,
+          0.014808767475187778,
+          0.005194223951548338,
+          -0.010939245112240314,
+          0.03611554577946663,
+          -0.00010321995068807155,
+          -0.005124502815306187,
+          -0.00845717266201973,
+          0.0007756475242786109,
+          0.0192430317401886,
+          -0.008478089235723019,
+          -0.006539841648191214,
+          0.011057770811021328,
+          0.0022310761269181967,
+          -0.0070104594342410564,
+          -0.023621518164873123,
+          0.015227094292640686,
+          -0.010297810658812523,
+          0.014557771384716034,
+          -0.0004311816010158509,
+          0.011615539900958538,
+          -0.031988054513931274,
+          -0.006592132616788149,
+          -0.0010510459542274475,
+          -0.006191235966980457,
+          0.015408368781208992,
+          -0.014780879020690918,
+          -0.009642431512475014,
+          -0.005884462967514992,
+          -0.011022910475730896,
+          0.021139446645975113,
+          0.00843625608831644,
+          -0.007543826010078192,
+          -0.002124751452356577,
+          -0.004336654208600521,
+          0.0011442980030551553,
+          -0.006550299935042858,
+          -0.010576695203781128,
+          -0.02132072113454342,
+          -0.014320719987154007,
+          -0.019842632114887238,
+          -0.01564542017877102,
+          0.018559763208031654,
+          0.009788846597075462,
+          -0.02218526229262352,
+          0.014501994475722313,
+          -0.002930030459538102,
+          0.0009935260750353336,
+          -0.0015112054534256458,
+          0.008903387933969498,
+          0.011134464293718338,
+          -0.024597613140940666,
+          0.0023426299449056387,
+          -0.014404385350644588,
+          -0.00024097364803310484,
+          -0.02847410924732685,
+          0.015868527814745903,
+          -0.017235063016414642,
+          -0.0007159488159231842,
+          0.020205182954669,
+          -0.013532870449125767,
+          -0.00944721233099699,
+          0.00328560802154243,
+          -0.027874507009983063,
+          0.006877989042550325,
+          0.008324702270328999,
+          -0.005661355331540108,
+          -0.027832673862576485,
+          -0.015213149599730968,
+          -0.037816740572452545,
+          -0.010995022021234035,
+          0.02476494386792183,
+          -0.0189502015709877,
+          -0.00014935572107788175,
+          0.009837650693953037,
+          -0.006417830009013414,
+          0.02502988465130329,
+          -0.01119721308350563,
+          0.005675299558788538,
+          -0.013072711415588856,
+          -0.01621713489294052,
+          0.0030102096498012543,
+          0.0037998014595359564,
+          -0.002534362953156233,
+          -0.012521914206445217,
+          7.544043910456821e-05,
+          0.002996265422552824,
+          0.0004237737157382071,
+          0.013693229295313358,
+          -0.0004531873273663223,
+          -0.019424306228756905,
+          0.014153389260172844,
+          -0.028808770701289177,
+          0.0016062004724517465,
+          -0.012926297262310982,
+          -0.009210160933434963,
+          -0.0072509972378611565,
+          0.007188248448073864,
+          0.005166335497051477,
+          0.0037579687777906656,
+          0.005821714177727699,
+          0.001241907593794167,
+          0.013735062442719936,
+          -0.004517929162830114,
+          -0.011406376026570797,
+          0.007662351708859205,
+          -0.016426296904683113,
+          0.02406773343682289,
+          0.01920119859278202,
+          0.0023600601125508547,
+          -0.02416534349322319,
+          0.011078687384724617,
+          0.003097361186519265,
+          -0.0071394434198737144,
+          0.0028237055521458387,
+          0.02324502356350422,
+          -0.004001992754638195,
+          0.010911356657743454,
+          0.001513820025138557,
+          -0.003372759558260441,
+          0.030677296221256256,
+          -0.009342631325125694,
+          -0.003859064308926463,
+          -0.01263346802443266,
+          -0.00851294957101345,
+          0.0051872520707547665,
+          -0.00021875003585591912,
+          -0.004361056722700596,
+          -0.0013212154153734446,
+          -0.006511953193694353,
+          0.02518327161669731,
+          0.00497460225597024,
+          0.0031984567176550627,
+          0.0076414356008172035,
+          -0.01585458405315876,
+          0.009684264659881592,
+          -0.02522510476410389,
+          -0.005497511010617018,
+          -0.022659366950392723,
+          -0.026159366592764854,
+          -0.030760960653424263,
+          -0.02026095986366272,
+          -0.014404385350644588,
+          0.006288845557719469,
+          0.0015190490521490574,
+          -0.007467132527381182,
+          0.0008680280297994614,
+          -0.007237053010612726,
+          -0.005166335497051477,
+          0.01492032129317522,
+          0.01061155553907156,
+          0.014390440657734871,
+          -0.0016184017295017838,
+          0.022324705496430397,
+          -0.004364542663097382,
+          -0.02253386750817299,
+          -0.008066734299063683,
+          -0.007704184390604496,
+          0.015715142711997032,
+          0.011029882356524467,
+          0.015924304723739624,
+          -0.0020236556883901358,
+          -0.004625997040420771,
+          -0.006689742207527161,
+          0.023217136040329933,
+          0.0013238298706710339,
+          -0.012598607689142227,
+          0.003942729905247688,
+          0.02897610142827034,
+          -0.0026668331120163202,
+          0.005832172464579344,
+          0.013686257414519787,
+          0.0101025914773345,
+          -0.010472113266587257,
+          -0.015533867292106152,
+          -0.008387451991438866,
+          -0.005420817527920008,
+          0.0022188748698681593,
+          -0.005692730192095041,
+          -0.006135459057986736,
+          -0.016928289085626602,
+          0.006100598722696304,
+          -0.01342828944325447,
+          -0.005009463056921959,
+          0.010430281050503254,
+          -0.010088646784424782,
+          0.0028254487551748753,
+          0.012933269143104553,
+          0.013302790932357311,
+          -0.000308733870042488,
+          0.02507171779870987,
+          0.0027836160734295845,
+          0.004549303557723761,
+          -0.012152392417192459,
+          -0.020707175135612488,
+          -0.007017431315034628,
+          0.018517931923270226,
+          0.013086655177175999,
+          0.002142181620001793,
+          -0.006588646676391363,
+          -0.02726096101105213,
+          0.0007133343024179339,
+          -0.009865539148449898,
+          0.008429284207522869,
+          -0.034972116351127625,
+          0.023119525983929634,
+          -0.028139447793364525,
+          0.008366535417735577,
+          0.01727689616382122,
+          0.05053387209773064,
+          0.00041789101669564843,
+          0.0023844626266509295,
+          6.50367364869453e-05,
+          0.015575699508190155,
+          0.02152988500893116,
+          -0.0009952690452337265,
+          -0.0021787851583212614,
+          -0.038011956959962845,
+          -0.005309263709932566,
+          -0.000978710362687707,
+          -0.016021914780139923,
+          0.014271914958953857,
+          -0.0015190490521490574,
+          -0.024751000106334686,
+          -0.0039531877264380455,
+          -0.012494025751948357,
+          -0.005054781679064035,
+          0.017193229869008064,
+          0.014752990566194057,
+          -0.015687253326177597,
+          -0.0010815489804372191,
+          -0.019173309206962585,
+          -0.00797609705477953,
+          0.02208765409886837,
+          0.0013586904387921095,
+          0.008010957390069962,
+          0.004406375344842672,
+          0.007746017072349787,
+          0.0013752492377534509,
+          0.01063944399356842,
+          -0.0017534863436594605,
+          0.05111953243613243,
+          -0.016858568415045738,
+          -0.005671813618391752,
+          -0.006860558874905109,
+          0.0004283491871319711,
+          -0.003201942890882492,
+          0.02147410809993744,
+          0.0023478588555008173,
+          0.0015085908817127347,
+          -0.004789841361343861,
+          -0.019368529319763184,
+          -0.0007015688461251557,
+          0.0009351345943287015,
+          0.0014475849457085133,
+          -0.007990040816366673,
+          0.006208666600286961,
+          -0.028390442952513695,
+          -0.015213149599730968,
+          0.008066734299063683,
+          -0.007773905526846647,
+          0.0004348855291027576,
+          -0.005490538664162159,
+          -0.0015164344804361463,
+          0.0007434015278704464,
+          0.022115541622042656,
+          -0.015059763565659523,
+          -0.0059646423906087875,
+          -0.012682273052632809,
+          0.0005586405168287456,
+          -0.029645422473549843,
+          0.003036355134099722,
+          -0.0017473857151344419,
+          0.022073708474636078,
+          -0.02624303288757801,
+          -0.024932274594902992,
+          0.006362053100019693,
+          0.00866633653640747,
+          -0.0028515940066426992,
+          -0.008687252178788185,
+          0.014599604532122612,
+          -0.02203187718987465,
+          0.019996019080281258,
+          0.0018022911390289664,
+          -0.03374502435326576,
+          -0.009258965961635113,
+          0.014501994475722313,
+          -0.01691434532403946,
+          0.00152950722258538,
+          -0.004800299648195505,
+          0.02051195502281189,
+          0.01777888834476471,
+          -0.010011954233050346,
+          -0.017290839925408363,
+          0.0005067854654043913,
+          0.019047811627388,
+          -0.005828686058521271,
+          0.02805578149855137,
+          -0.023886458948254585,
+          0.012298806570470333,
+          -0.024792833253741264,
+          0.01010956335812807,
+          -0.02060956507921219,
+          0.003900896990671754,
+          0.018810760229825974,
+          -0.009921316057443619,
+          0.022450203076004982,
+          -0.016984065994620323,
+          -0.022519923746585846,
+          0.0076414356008172035,
+          -0.010137451812624931,
+          -0.025294825434684753,
+          0.013560758903622627,
+          -0.023593628779053688,
+          -0.013058766722679138,
+          0.0042878491804003716,
+          -0.014613548293709755,
+          -0.011992033571004868,
+          0.015394425019621849,
+          0.004117032513022423,
+          0.004472610540688038,
+          0.0017343130894005299,
+          -0.021041836589574814,
+          -0.013205181807279587,
+          -0.00663396529853344,
+          0.003052042331546545,
+          -0.0020881476812064648,
+          -0.007599602919071913,
+          -0.0359761007130146,
+          0.020079685375094414,
+          -0.01565936580300331,
+          -0.01382569968700409,
+          -0.0037265941500663757,
+          0.019131477922201157,
+          0.009356575086712837,
+          0.005790339782834053,
+          0.011580679565668106,
+          -0.013358567841351032,
+          -0.007244025357067585,
+          0.006034363526850939,
+          0.018420321866869926,
+          -0.009698208421468735,
+          -0.004273904953151941,
+          -0.010346615687012672,
+          0.006138945464044809,
+          -0.002053287113085389,
+          -0.0039531877264380455,
+          0.00914741214364767,
+          0.011908368207514286,
+          -0.004486554767936468,
+          0.006752490997314453,
+          -0.008945221081376076,
+          0.012187252752482891,
+          -0.017541835084557533,
+          0.0074531883001327515,
+          0.006783865857869387,
+          0.00292480131611228,
+          0.007024403661489487,
+          -0.007202192675322294,
+          0.021097613498568535,
+          0.03586454689502716,
+          0.000885022571310401,
+          0.014962153509259224,
+          0.00420418381690979,
+          0.0039357575587928295,
+          -0.00036712532164528966,
+          0.021655382588505745,
+          -0.005880977027118206,
+          -0.009733069688081741,
+          -0.00405428372323513,
+          -0.014069723896682262,
+          0.01793227344751358,
+          -0.00866633653640747,
+          -0.0015608817338943481,
+          -0.03525100275874138,
+          0.008896416053175926,
+          -0.003384960815310478,
+          -0.005535857751965523,
+          -0.012194225564599037,
+          -0.009942232631146908,
+          0.02310558222234249,
+          -0.0035732078831642866,
+          0.026521917432546616,
+          -0.00013704558659810573,
+          0.027525901794433594,
+          -0.011455181054770947,
+          -0.00563695328310132,
+          -0.015729086473584175,
+          -0.004312251694500446,
+          0.015087652020156384,
+          0.00485259061679244,
+          0.02922709658741951,
+          0.012821715325117111,
+          -0.0006344622815959156,
+          0.010123508051037788,
+          0.004538845270872116,
+          0.01989840902388096,
+          0.015826696529984474,
+          0.013811754994094372,
+          -0.0033535861875861883,
+          -0.0060378494672477245,
+          0.006710658315569162,
+          0.02081872895359993,
+          0.0016515192110091448,
+          -0.00746016064658761,
+          0.002637201687321067,
+          -0.0071220132522284985,
+          0.016844624653458595,
+          0.011406376026570797,
+          0.026047812774777412,
+          -0.014006974175572395,
+          0.005281375255435705,
+          0.006198208313435316,
+          0.0019853091798722744,
+          0.003897411050274968,
+          -0.0078087663277983665,
+          0.02218526229262352,
+          0.00810856744647026,
+          0.003426793497055769,
+          0.0020637453999370337,
+          -0.004639940802007914,
+          -0.000789156008977443,
+          -0.0053894431330263615,
+          -0.012724106200039387,
+          0.005487052723765373,
+          0.004259960725903511,
+          -0.022352593019604683,
+          -0.0026162853464484215,
+          -0.0138047831133008,
+          0.007704184390604496,
+          0.018322711810469627,
+          0.017235063016414642,
+          0.003268177853897214,
+          -0.0026825203094631433,
+          -0.003932271618396044,
+          0.00030306901317089796,
+          0.008024902082979679,
+          0.0025465642102062702,
+          0.016537850722670555,
+          -0.024583669379353523,
+          -0.01878287084400654,
+          0.010060758329927921,
+          -0.006281873676925898,
+          -0.01311454363167286,
+          0.005860060919076204
+        ],
+        "keyphrases": [
+          "fine-tuning",
+          "dense model",
+          "MoE model",
+          "expert choice routing",
+          "downstream performance"
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "46fb6ac4-1a91-4f74-8c57-236109a5fc32",
+      "properties": {
+        "page_content": "B Capacity Factor\n\nWe evaluate the downstream task fine-tuning performance by varying the capacity factors. Note that a capacity factor of n indicates on average how many experts each token can be received. EC-CF2 is our baseline expert choice, which matches GShard top-2 gating computational footprint. EC-CF1, however, matches Switch Transformer top-1 gating computational footprint. EC-CF0.5 further verifies that an aggressively lowered capacity factor can provide strong enough performance, that almost matches the top-2 gating baseline.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Top-2 78.1 87.0 88.3 85.0 82.6 90.1 90.7 81.6 94.7 68.2 67.2 83.0±0.3 EC-CAP2 78.2 88.0 88.5 85.7 83.0 90.8 91.1 80.0 95.4 70.4 64.1 83.2±0.4 EC-CAP3 78.5 91.7 89.3 86.3 83.5 90.9 91.1 81.8 94.9 70.0 65.6 84.0±0.4 EC-CF2 79.1 89.6 89.3 86.8 84.3 91.3 91.2 81.1 95.2 68.1 68.0 84.0±0.2 EC-CF1 77.4 90.6 88.0 85.5 83.6 90.3 91.2 79.8 95.3 66.5 64.9 83.0±0.2 EC-CF0.5 77.4 89.6 86.3 85.2 82.7 91.7 91.0 79.6 94.9 67.3 63.5 83.0 ±0.05 Hash Layers 76.1 85.2 86.7 83.4 82.5 90.0 90.3 75.7 94.0 67.4 63.3 81.3±1.0\n\nTable 2: Comparison between different routing methods in fine-tuning of 100M/64E models. We perform 3 independent fine-tuning runs for each method and report the average results. This gives more accurate difference between the variants of expert choice method, since they achieve close fine-tuning results. We do not report averaged results in other experiments.\n\n",
+        "embedding": [
+          0.00878380797803402,
+          -0.02230931632220745,
+          -0.015165671706199646,
+          -0.002112389076501131,
+          -0.01730332337319851,
+          -0.0011611338704824448,
+          -0.006440163590013981,
+          0.05074397847056389,
+          -0.042286649346351624,
+          -0.006537329405546188,
+          -0.01575644128024578,
+          -0.005678381770849228,
+          -0.013284537009894848,
+          0.007656681817024946,
+          -0.008605021983385086,
+          0.038244541734457016,
+          0.0028061545453965664,
+          0.026398062705993652,
+          0.030253609642386436,
+          -0.007660568691790104,
+          -0.0003531984693836421,
+          -0.035757094621658325,
+          -0.05282721668481827,
+          -0.022464781999588013,
+          -0.01148890983313322,
+          0.020676927641034126,
+          0.011403403244912624,
+          -0.013937492854893208,
+          -0.02294672653079033,
+          0.030082598328590393,
+          0.0195420291274786,
+          0.025667374953627586,
+          -0.0036165190394967794,
+          -0.03728065639734268,
+          -0.0513969361782074,
+          -0.028014905750751495,
+          0.059232402592897415,
+          0.028496848419308662,
+          -0.028963245451450348,
+          0.035601627081632614,
+          0.0162772499024868,
+          0.02532535046339035,
+          -0.04981118440628052,
+          0.0055617825128138065,
+          0.01784745417535305,
+          -0.031683892011642456,
+          -0.03183935955166817,
+          -0.015344456769526005,
+          0.025092151015996933,
+          -0.016354983672499657,
+          -0.020910125225782394,
+          -0.04604891687631607,
+          0.023195471614599228,
+          0.038742031902074814,
+          -0.030642274767160416,
+          0.03675207123160362,
+          0.003350284183397889,
+          0.05873491242527962,
+          0.0325544998049736,
+          0.01063384860754013,
+          0.0017149801133200526,
+          -0.012211824767291546,
+          -0.048194341361522675,
+          -0.008340730331838131,
+          -0.0009648585109971464,
+          0.015235630795359612,
+          -0.019884053617715836,
+          0.0055034831166267395,
+          0.03799579665064812,
+          -0.008364050649106503,
+          -0.06597960740327835,
+          0.04483628645539284,
+          -0.03423352912068367,
+          1.5910934962448664e-05,
+          -0.0011455873027443886,
+          -0.03270996734499931,
+          0.04250429943203926,
+          0.027486322447657585,
+          0.033642761409282684,
+          0.02807709202170372,
+          -0.008022026158869267,
+          0.02874559350311756,
+          -0.025138791650533676,
+          0.006685021799057722,
+          0.014769233763217926,
+          -0.02674008719623089,
+          -0.01284923404455185,
+          -0.05550122633576393,
+          -0.0366898849606514,
+          0.00012388666800688952,
+          -0.02518543042242527,
+          -0.010587208904325962,
+          -0.009304617531597614,
+          0.007656681817024946,
+          -0.02033490315079689,
+          0.05926349386572838,
+          -0.037840332835912704,
+          0.014349476434290409,
+          -0.008830447681248188,
+          -0.00403044605627656,
+          -0.034948673099279404,
+          0.026600167155265808,
+          0.012499435804784298,
+          0.007835467346012592,
+          0.06616616994142532,
+          -0.018857980147004128,
+          -0.014870286919176579,
+          -0.0012155468575656414,
+          -0.01641716994345188,
+          0.07673782855272293,
+          0.0064595965668559074,
+          -0.007330203894525766,
+          -0.03892859071493149,
+          0.005040972959250212,
+          -0.0508994460105896,
+          -0.027735067531466484,
+          -0.03410915657877922,
+          -0.02756405435502529,
+          0.060289569199085236,
+          -0.028512395918369293,
+          0.0017295550787821412,
+          0.05969879776239395,
+          0.024050531908869743,
+          -0.004873847123235464,
+          -0.03622348979115486,
+          0.053200334310531616,
+          0.02229377068579197,
+          -0.0043763574212789536,
+          0.021438708528876305,
+          -0.021936198696494102,
+          -0.0052197580225765705,
+          0.012592715211212635,
+          -0.013851987197995186,
+          -0.039705920964479446,
+          0.03501085937023163,
+          -0.022371502593159676,
+          0.04325053468346596,
+          0.029336363077163696,
+          -0.009926480241119862,
+          -0.018593687564134598,
+          0.028108185157179832,
+          -0.033518388867378235,
+          0.039426080882549286,
+          0.021998384967446327,
+          -0.02530980296432972,
+          0.03911514952778816,
+          0.010493929497897625,
+          0.04974899813532829,
+          0.007866560481488705,
+          0.013976359739899635,
+          -0.011426723562180996,
+          0.016759194433689117,
+          -0.03389150649309158,
+          0.03146624192595482,
+          -0.013408909551799297,
+          -0.026724539697170258,
+          -0.014450529590249062,
+          0.017520975321531296,
+          0.0022406482603400946,
+          -0.03145069628953934,
+          0.0037214583717286587,
+          0.03283433988690376,
+          -0.01626170426607132,
+          -0.017365509644150734,
+          0.002685668645426631,
+          0.04011012986302376,
+          0.02782834693789482,
+          0.012157411314547062,
+          -0.06523337215185165,
+          0.0254497230052948,
+          0.04878511279821396,
+          -0.018718060106039047,
+          0.038493286818265915,
+          0.012312876991927624,
+          0.007306884042918682,
+          0.016603728756308556,
+          0.002194008557125926,
+          0.016743646934628487,
+          -0.010727128013968468,
+          0.022091664373874664,
+          -0.03799579665064812,
+          -0.009654414840042591,
+          0.011838707141578197,
+          -0.0014555469388142228,
+          0.049904465675354004,
+          -0.008154171518981457,
+          0.02073911391198635,
+          0.04567579925060272,
+          -0.000116234841698315,
+          0.005892146844416857,
+          0.025620734319090843,
+          0.038648754358291626,
+          0.013090205378830433,
+          0.005550122819840908,
+          0.007124212104827166,
+          -0.006261378061026335,
+          -0.004663968458771706,
+          0.018547048792243004,
+          0.0328032448887825,
+          0.01475368719547987,
+          -0.029709480702877045,
+          0.014100732281804085,
+          -0.01612178422510624,
+          -0.0195420291274786,
+          -0.0035193529911339283,
+          0.008037572726607323,
+          0.041882436722517014,
+          -0.01888907328248024,
+          -0.005736681632697582,
+          -0.014932473190128803,
+          -0.005464616697281599,
+          -0.018920166417956352,
+          -0.01707012578845024,
+          -0.02913425676524639,
+          -0.01416291855275631,
+          0.030471261590719223,
+          0.007427370175719261,
+          0.0003400810528546572,
+          -0.004457976669073105,
+          -0.0032783811911940575,
+          0.019231097772717476,
+          0.03058008849620819,
+          0.017256684601306915,
+          -0.01717895083129406,
+          -0.017147857695817947,
+          0.00652955612167716,
+          -0.007248584646731615,
+          0.0002594332618173212,
+          0.010571662336587906,
+          -0.013859760016202927,
+          -0.013486642390489578,
+          -0.0009648585109971464,
+          -0.033238548785448074,
+          0.01120907161384821,
+          0.008962593041360378,
+          -0.020676927641034126,
+          0.010385103523731232,
+          -0.011566642671823502,
+          0.0012272067833691835,
+          0.01416291855275631,
+          0.0003378948022145778,
+          0.008177491836249828,
+          -0.0328032448887825,
+          -0.01850040815770626,
+          0.009872066788375378,
+          -0.0005674495478160679,
+          -0.04645312950015068,
+          0.0049982196651399136,
+          0.008612795732915401,
+          -0.010711581446230412,
+          -0.028543489053845406,
+          -0.01823611743748188,
+          0.04424551501870155,
+          0.012965832836925983,
+          -0.008379597216844559,
+          0.03224356845021248,
+          0.007948179729282856,
+          -0.02806154452264309,
+          -0.030486809089779854,
+          -0.006941540166735649,
+          0.04029668867588043,
+          -0.014147371985018253,
+          -0.026398062705993652,
+          0.04023450240492821,
+          -0.018018465489149094,
+          -0.03126413747668266,
+          0.005355790723115206,
+          0.002260081470012665,
+          -0.019060084596276283,
+          0.00021704458049498498,
+          0.02125214971601963,
+          -0.003983806353062391,
+          -0.03448227420449257,
+          -0.0317305326461792,
+          0.0005280972691252828,
+          -0.02715984359383583,
+          -0.029180897399783134,
+          0.009234657511115074,
+          -0.0008647775393910706,
+          0.028403569012880325,
+          0.02832583710551262,
+          0.018935712054371834,
+          0.003678705310449004,
+          0.025931665673851967,
+          -0.023646321147680283,
+          0.014085185714066029,
+          -0.033145271241664886,
+          -0.01508016511797905,
+          -0.042286649346351624,
+          -0.03162170574069023,
+          -0.015896359458565712,
+          -0.024952232837677002,
+          -0.016075145453214645,
+          -0.007034819573163986,
+          0.012359516695141792,
+          -0.02230931632220745,
+          0.0047689080238342285,
+          -0.011621055193245411,
+          0.03410915657877922,
+          -0.0050759525038301945,
+          -0.05444405972957611,
+          -0.004838867578655481,
+          0.012219597585499287,
+          -0.037467215210199356,
+          -0.04384130612015724,
+          -0.0035912557505071163,
+          -0.023195471614599228,
+          -0.0015235630562528968,
+          -0.010167451575398445,
+          -0.0197907742112875,
+          0.028388023376464844,
+          -0.018407128751277924,
+          -0.009211338125169277,
+          -0.03382932022213936,
+          -0.0013010528637096286,
+          0.004438543692231178,
+          -0.014707047492265701,
+          0.005783321335911751,
+          -0.06358543783426285,
+          -0.01641716994345188,
+          0.02467239461839199,
+          -0.006708341650664806,
+          -0.08401361852884293,
+          0.01442720927298069,
+          -0.03373603895306587,
+          0.012250690720975399,
+          0.011838707141578197,
+          0.04051434248685837,
+          -0.02058364823460579,
+          0.007944293320178986,
+          -0.02860567346215248,
+          -0.017381057143211365,
+          -0.035477254539728165,
+          -0.024066077545285225,
+          -0.03805798292160034,
+          -0.013346723280847073,
+          -0.04738591983914375,
+          -0.017738627269864082,
+          -0.016370529308915138,
+          -0.0037136850878596306,
+          -0.02045927569270134,
+          0.035228509455919266,
+          -0.03927061706781387,
+          0.006078705657273531,
+          0.023817332461476326,
+          -0.012989153154194355,
+          -0.010144132189452648,
+          -0.0031384623143821955,
+          -0.009910933673381805,
+          -0.010493929497897625,
+          0.0010017815511673689,
+          0.0013107694685459137,
+          0.030238064005970955,
+          0.027750613167881966,
+          0.006280811037868261,
+          -0.02384842559695244,
+          0.007827693596482277,
+          0.004306397866457701,
+          0.019355470314621925,
+          -0.03905296325683594,
+          -0.012227371335029602,
+          -0.046515315771102905,
+          -0.014069639146327972,
+          0.05155239999294281,
+          -0.03609911724925041,
+          -0.004306397866457701,
+          0.014784780330955982,
+          -0.017365509644150734,
+          -0.002349474234506488,
+          0.005064292810857296,
+          -0.02795271947979927,
+          0.029693933203816414,
+          0.036658793687820435,
+          0.018578141927719116,
+          0.006615062244236469,
+          -0.002516599837690592,
+          0.023630773648619652,
+          -0.03134186938405037,
+          -0.042566485702991486,
+          0.03581928089261055,
+          -0.02715984359383583,
+          0.01494801975786686,
+          -0.005592875648289919,
+          -0.0026545755099505186,
+          -0.011916439980268478,
+          0.06234171614050865,
+          0.025387536734342575,
+          0.012382837012410164,
+          -0.05024648830294609,
+          0.01851595565676689,
+          -0.0023883406538516283,
+          -0.045333776623010635,
+          -0.024750126525759697,
+          -0.022107211872935295,
+          -0.0038477741181850433,
+          -0.006957086734473705,
+          -0.013774254359304905,
+          0.004971013404428959,
+          0.018034012988209724,
+          -0.012872553430497646,
+          -0.02493668533861637,
+          0.02649134211242199,
+          -0.023133285343647003,
+          -0.012631582096219063,
+          0.01120907161384821,
+          -0.004011013079434633,
+          0.00945230945944786,
+          -0.014567128382623196,
+          0.005977652966976166,
+          0.0023961139377206564,
+          -0.0065451026894152164,
+          -0.021407615393400192,
+          -0.010525022633373737,
+          -0.016759194433689117,
+          0.012188504450023174,
+          0.02845020964741707,
+          -0.003931337036192417,
+          0.03622348979115486,
+          0.002846964169293642,
+          0.0024816200602799654,
+          -0.029585108160972595,
+          0.010043079033493996,
+          0.0501532107591629,
+          0.0012388667091727257,
+          0.05012211576104164,
+          -0.003983806353062391,
+          -0.011038059368729591,
+          0.03501085937023163,
+          -0.009530042298138142,
+          0.012942513450980186,
+          0.011170204728841782,
+          0.007493442855775356,
+          0.00652955612167716,
+          -0.010999192483723164,
+          -0.0027089884970337152,
+          -0.0038808106910437346,
+          -0.0025010532699525356,
+          0.035943653434515,
+          -0.022122757509350777,
+          -0.013035791926085949,
+          0.03603693097829819,
+          -0.005118705797940493,
+          -0.016075145453214645,
+          -0.04909604415297508,
+          -0.021967291831970215,
+          -0.0097710145637393,
+          -0.0015206481330096722,
+          0.019573122262954712,
+          -0.02543417550623417,
+          0.020568102598190308,
+          -0.05755337327718735,
+          0.022340409457683563,
+          -0.06283920258283615,
+          0.013533282093703747,
+          0.028667859733104706,
+          0.002365020802244544,
+          0.02294672653079033,
+          -0.018733607605099678,
+          0.02911871112883091,
+          -0.045333776623010635,
+          -0.0071747382171452045,
+          0.016168424859642982,
+          0.03603693097829819,
+          -0.008022026158869267,
+          -0.0027361949905753136,
+          0.011170204728841782,
+          -0.008744941093027592,
+          -0.0325544998049736,
+          -0.006650041788816452,
+          0.02310219220817089,
+          -0.021065590903162956,
+          0.012048585340380669,
+          -0.02244923636317253,
+          -0.0104317432269454,
+          -0.005775548052042723,
+          0.027579601854085922,
+          0.03976810723543167,
+          -0.048007782548666,
+          -0.005472389981150627,
+          7.518220081692562e-05,
+          0.022620247676968575,
+          -0.0025690693873912096,
+          0.01475368719547987,
+          -0.012312876991927624,
+          0.037063002586364746,
+          -0.019215550273656845,
+          0.010610528290271759,
+          -0.009421216323971748,
+          -0.012818140909075737,
+          -0.018049558624625206,
+          -0.011154658161103725,
+          -0.021936198696494102,
+          0.01679028756916523,
+          -0.013828666880726814,
+          0.002605992369353771,
+          -0.032088104635477066,
+          0.03401587903499603,
+          -0.01941765658557415,
+          0.03264778107404709,
+          -0.014279517345130444,
+          0.005437409970909357,
+          -0.024874499067664146,
+          0.01572534814476967,
+          0.005666722077876329,
+          0.01612178422510624,
+          -0.021454256027936935,
+          0.031699441373348236,
+          0.030782192945480347,
+          -0.049904465675354004,
+          -0.02241814322769642,
+          0.0027284217067062855,
+          -0.012631582096219063,
+          0.010268504731357098,
+          -0.01121684443205595,
+          -0.02021053060889244,
+          0.0354461632668972,
+          -0.008348504081368446,
+          0.0052197580225765705,
+          -0.011100245639681816,
+          0.00485052727162838,
+          -0.03028470277786255,
+          -0.006988179869949818,
+          -0.006863807328045368,
+          0.036503326147794724,
+          -0.014380569569766521,
+          0.005235304590314627,
+          -0.0035407296381890774,
+          0.004601782187819481,
+          0.05105490982532501,
+          0.003196761943399906,
+          -0.00014137655671220273,
+          0.00534413056448102,
+          0.004959353245794773,
+          -0.0008458301308564842,
+          0.00011082998389611021,
+          -0.015826400369405746,
+          -0.007893767207860947,
+          0.00015473687381017953,
+          -0.032336849719285965,
+          -0.011263484135270119,
+          0.012219597585499287,
+          0.007610042113810778,
+          0.0029421867802739143,
+          -0.018857980147004128,
+          0.008698301389813423,
+          0.03504195064306259,
+          0.019479842856526375,
+          -0.019510935992002487,
+          0.0027770046144723892,
+          0.0032997578382492065,
+          -0.01207190565764904,
+          -0.028667859733104706,
+          -0.0040343329310417175,
+          0.0026526320725679398,
+          0.0031423489563167095,
+          -0.010734900832176208,
+          0.032212477177381516,
+          -0.027439681813120842,
+          3.349190956214443e-05,
+          -0.027719520032405853,
+          -0.004209231585264206,
+          0.01599741168320179,
+          -0.0036767618730664253,
+          0.03401587903499603,
+          -0.000832226884085685,
+          0.011605508625507355,
+          0.009755467996001244,
+          0.013043565675616264,
+          -0.04076308757066727,
+          0.00925020407885313,
+          0.011240164749324322,
+          -0.01680583320558071,
+          -0.028263650834560394,
+          0.006720001343637705,
+          0.007847127504646778,
+          0.01808065176010132,
+          -0.010159678757190704,
+          -0.018453769385814667,
+          -0.025543002411723137,
+          0.018034012988209724,
+          0.02179628051817417,
+          -0.0419757179915905,
+          0.007703321520239115,
+          0.015453282743692398,
+          0.03420243784785271,
+          -0.0009502836037427187,
+          -0.0036961950827389956,
+          -0.034326810389757156,
+          0.007015386130660772,
+          0.002021053107455373,
+          -0.013167938217520714,
+          0.050712887197732925,
+          -0.010641621425747871,
+          -0.019635308533906937,
+          0.016728101298213005,
+          -0.010727128013968468,
+          -0.018811339512467384,
+          0.02241814322769642,
+          0.014217331074178219,
+          -0.026429155841469765,
+          -0.0005212956457398832,
+          0.016588181257247925,
+          0.029211990535259247,
+          -0.017007939517498016,
+          0.008542835712432861,
+          0.006024292670190334,
+          -0.0019957898184657097,
+          0.015538789331912994,
+          -0.015328910201787949,
+          0.013680974952876568,
+          0.05121037736535072,
+          0.015103485435247421,
+          0.005631742067635059,
+          0.004601782187819481,
+          -0.006156438495963812,
+          0.012079678475856781,
+          -0.004271417856216431,
+          -0.004286964423954487,
+          -0.015119032002985477,
+          0.00011161946167703718,
+          0.05074397847056389,
+          -0.0008594333776272833,
+          0.028419116511940956,
+          -0.0018976522842422128,
+          0.014396116137504578,
+          0.021423162892460823,
+          -0.006914333440363407,
+          -0.02989603951573372,
+          -0.008379597216844559,
+          -0.008511742576956749,
+          -0.0237551461905241,
+          -0.024687940254807472,
+          -0.028698952868580818,
+          -0.026864459738135338,
+          -0.0002560324501246214,
+          -0.0013982189120724797,
+          0.009980892762541771,
+          0.004372470546513796,
+          0.0037972477730363607,
+          0.0008453443297185004,
+          0.004049879498779774,
+          0.023288751021027565,
+          0.013556602410972118,
+          0.020272716879844666,
+          -0.0042558712884783745,
+          0.007326317485421896,
+          -0.0076800016686320305,
+          0.00330947432667017,
+          -0.0035562762059271336,
+          -0.0010795143898576498,
+          -0.0020579760894179344,
+          0.006024292670190334,
+          0.03659660741686821,
+          0.02977166697382927,
+          0.002528259763494134,
+          0.001742186606861651,
+          0.012507209554314613,
+          -0.010120811872184277,
+          0.011815386824309826,
+          -0.0023786239326000214,
+          -0.011076925322413445,
+          0.011325670406222343,
+          0.02073911391198635,
+          -0.014652634970843792,
+          0.015290044248104095,
+          -0.014279517345130444,
+          -0.05519029498100281,
+          0.006082592532038689,
+          -0.017272230237722397,
+          0.022760167717933655,
+          -0.02555854804813862,
+          -0.003968259785324335,
+          -0.017225591465830803,
+          0.00731077091768384,
+          0.014675954356789589,
+          -0.024174904450774193,
+          -0.004325830843299627,
+          0.04082527384161949,
+          0.0013739274581894279,
+          -0.02518543042242527,
+          -0.005355790723115206,
+          -0.003744778223335743,
+          -0.027097657322883606,
+          -0.012538302689790726,
+          0.033114176243543625,
+          -0.009708828292787075,
+          -0.029600653797388077,
+          0.051490213721990585,
+          -0.01462931465357542,
+          0.0022134417667984962,
+          -0.014932473190128803,
+          -0.002841134322807193,
+          -0.03724956139922142,
+          -0.006308017764240503,
+          -0.02860567346215248,
+          0.035601627081632614,
+          -0.024719033390283585,
+          0.026942191645503044,
+          0.023304296657443047,
+          -0.003422186942771077,
+          -0.02139206975698471,
+          0.026833366602659225,
+          -0.030098143965005875,
+          0.0044618635438382626,
+          0.0270199254155159,
+          -0.0005859110387973487,
+          -0.0018228343687951565,
+          0.002858624095097184,
+          -0.021671907976269722,
+          -0.005351903848350048,
+          -0.0017965995939448476,
+          0.00853506289422512,
+          0.027843892574310303,
+          0.009273524396121502,
+          -0.007042592857033014,
+          -0.005923239979892969,
+          0.0012971663381904364,
+          0.012685994617640972,
+          -0.019635308533906937,
+          0.028030451387166977,
+          -0.011605508625507355,
+          0.011123565025627613,
+          -0.008014252409338951,
+          -0.03031579591333866,
+          -0.02345976233482361,
+          0.014963566325604916,
+          0.03696972504258156,
+          0.02308664470911026,
+          0.014279517345130444,
+          0.009195791557431221,
+          0.02174963988363743,
+          -0.00718639837577939,
+          -0.038617659360170364,
+          -0.0028994339518249035,
+          -0.008115305565297604,
+          -0.010361784137785435,
+          0.01812729239463806,
+          0.026289235800504684,
+          0.0024058304261416197,
+          -0.011970852501690388,
+          0.015383323654532433,
+          0.007726641371846199,
+          -0.01901344582438469,
+          0.01943320222198963,
+          -0.010136358439922333,
+          0.021485349163413048,
+          -0.0025651827454566956,
+          0.015709800645709038,
+          0.016852473840117455,
+          0.00045692318235524,
+          0.007194171659648418,
+          -0.023055551573634148,
+          0.029009884223341942,
+          -0.011395630426704884,
+          -0.05736681446433067,
+          0.0036942518781870604,
+          0.0003740891406778246,
+          -0.022091664373874664,
+          -0.013377816416323185,
+          0.015103485435247421,
+          -0.028932152315974236,
+          0.03724956139922142,
+          0.0032025917898863554,
+          -0.005064292810857296,
+          0.023630773648619652,
+          -0.006918220315128565,
+          -0.03488648682832718,
+          0.007959839887917042,
+          0.029072070494294167,
+          -0.01769198849797249,
+          -0.00037797578261233866,
+          0.043623652309179306,
+          0.005278057884424925,
+          -0.03370494768023491,
+          0.009032553061842918,
+          0.03152842819690704,
+          -0.009623321704566479,
+          -0.0007418625173158944,
+          0.012437249533832073,
+          -0.03302089869976044,
+          -0.01784745417535305,
+          0.019308829680085182,
+          0.005009879823774099,
+          0.018951259553432465,
+          -0.028512395918369293,
+          0.02241814322769642,
+          0.008620568551123142,
+          -0.01239838358014822,
+          -0.007524535991251469,
+          0.026569074019789696,
+          0.004496843088418245,
+          0.006377977319061756,
+          -0.015095711685717106,
+          0.013572148978710175,
+          0.049406975507736206,
+          -0.005655061919242144,
+          -0.018671421334147453,
+          0.05599871650338173,
+          -0.02047482319176197,
+          -0.013719840906560421,
+          0.024299276992678642,
+          -0.01822056993842125,
+          -0.007582835853099823,
+          0.03948826715350151,
+          -0.016292797401547432,
+          -0.03386041149497032,
+          0.009669961407780647,
+          -0.031295228749513626,
+          -0.00787822064012289,
+          -0.002689555287361145,
+          0.025014419108629227,
+          -0.013953039422631264,
+          -0.006148665212094784,
+          0.019122270867228508,
+          -0.027595147490501404,
+          -0.00879158079624176,
+          -0.017241137102246284,
+          0.013028019107878208,
+          -0.011325670406222343,
+          -0.02717539109289646,
+          -0.00024036443210206926,
+          -0.0009653443703427911,
+          0.007633361965417862,
+          -0.0025457495357841253,
+          -0.022635795176029205,
+          0.005946559831500053,
+          0.0035873691085726023,
+          -0.011955305933952332,
+          0.06072487309575081,
+          -0.00031287455931305885,
+          0.029196443036198616,
+          -0.02162526734173298,
+          -0.023428669199347496,
+          0.013284537009894848,
+          -0.011123565025627613,
+          0.022387050092220306,
+          0.0038846973329782486,
+          0.019713040441274643,
+          0.0015167614910751581,
+          0.03687644377350807,
+          0.010081945918500423,
+          -0.008962593041360378,
+          -0.032212477177381516,
+          0.01075822114944458,
+          -0.04051434248685837,
+          -0.011395630426704884,
+          0.008830447681248188,
+          -0.02454802207648754,
+          -0.013020245358347893,
+          -0.003565992694348097,
+          0.0206147413700819,
+          0.06498462706804276,
+          0.02373960055410862,
+          -0.009716601110994816,
+          -0.0272686704993248,
+          -0.004566802643239498,
+          -0.011706560850143433,
+          -0.006463483441621065,
+          -0.023133285343647003,
+          0.01192421279847622,
+          -0.045209404081106186,
+          -0.02557409554719925,
+          -0.021671907976269722,
+          -0.0018811339978128672,
+          -0.029320815578103065,
+          -0.016230611130595207,
+          -0.004403563681989908,
+          -0.012787047773599625,
+          0.007711094804108143,
+          -0.01812729239463806,
+          -0.04116729646921158,
+          -0.0016148991417139769,
+          -0.015678707510232925,
+          0.01651044934988022,
+          -0.02756405435502529,
+          -0.021407615393400192,
+          0.012631582096219063,
+          -0.027051018550992012,
+          0.02112777717411518,
+          -0.01108469907194376,
+          0.012763727456331253,
+          -0.010626074858009815,
+          0.031699441373348236,
+          0.0010979759972542524,
+          0.012499435804784298,
+          -0.006276924628764391,
+          0.005829961039125919,
+          -0.032461222261190414,
+          -0.02386397309601307,
+          -0.033767133951187134,
+          0.017878547310829163,
+          -0.014022999443113804,
+          -0.00747400987893343,
+          -0.022651340812444687,
+          0.0011533605866134167,
+          0.009988666512072086,
+          0.010921459645032883,
+          0.0028780573047697544,
+          0.0016090691788122058,
+          0.01081263367086649,
+          -0.008115305565297604,
+          -0.033642761409282684,
+          -0.021718546748161316,
+          -0.023786239326000214,
+          0.017412150278687477,
+          -0.041789159178733826,
+          -0.0016158708604052663,
+          -0.011473363265395164,
+          0.0005684212082996964,
+          -0.02703547105193138,
+          -0.00731465732678771,
+          -0.001748016569763422,
+          0.0021668020635843277,
+          -0.0013583808904513717,
+          0.015919679775834084,
+          0.019852960482239723,
+          0.01207190565764904,
+          -0.02479676716029644,
+          0.05217426270246506,
+          -0.030098143965005875,
+          -0.011496682651340961,
+          0.019324377179145813,
+          0.010369556955993176,
+          0.010703807696700096,
+          -0.021314335986971855,
+          -0.023397576063871384,
+          -0.016619274392724037,
+          0.015119032002985477,
+          0.004419110249727964,
+          -0.008200811222195625,
+          -0.013354497030377388,
+          0.006778301205486059,
+          0.04225555434823036,
+          -0.024843405932188034,
+          -0.00012303645780775696,
+          -0.019775226712226868,
+          0.01417069137096405,
+          -0.00422866502776742,
+          -0.04004794359207153,
+          0.0027964378241449594,
+          -0.02453247457742691,
+          0.002339757513254881,
+          0.01232065074145794,
+          -0.015297817066311836,
+          0.0104317432269454,
+          -0.023630773648619652,
+          0.046484220772981644,
+          0.02059919573366642,
+          -0.012670448049902916,
+          0.03058008849620819,
+          -0.014784780330955982,
+          -0.01075822114944458,
+          0.024454742670059204,
+          0.0051964386366307735,
+          0.031404055655002594,
+          0.0011339273769408464,
+          -0.02414381131529808,
+          0.01916891150176525,
+          -0.012584942393004894,
+          0.03227466344833374,
+          -0.006762754637748003,
+          -0.017925186082720757,
+          -0.006241944618523121,
+          -0.004442430101335049,
+          0.013206804171204567,
+          -0.014605995267629623,
+          -0.009397896938025951,
+          -0.03135741502046585,
+          -0.007120325695723295,
+          0.026833366602659225,
+          0.005351903848350048,
+          -0.0482565276324749,
+          0.0029091504402458668,
+          0.001017328118905425,
+          -0.009825427085161209,
+          0.02229377068579197,
+          -0.006335224024951458,
+          -0.009094739332795143,
+          -0.0033017010428011417,
+          0.035881467163562775,
+          0.05002883821725845,
+          0.012110771611332893,
+          0.004796114284545183,
+          0.012717087753117085,
+          0.036658793687820435,
+          -0.04306397587060928,
+          0.0127248615026474,
+          -0.01494801975786686,
+          -0.03187045082449913,
+          0.02008615806698799,
+          0.0035348995588719845,
+          -0.010284051299095154,
+          0.0337982252240181,
+          -0.015297817066311836,
+          -0.02153198793530464,
+          6.425102765206248e-05,
+          -0.013401136733591557,
+          -0.013556602410972118,
+          -0.007835467346012592,
+          0.0023261543828994036,
+          -0.007302997633814812,
+          -0.007893767207860947,
+          0.0006388665642589331,
+          0.027377495542168617,
+          -0.015017978847026825,
+          0.027921626344323158,
+          0.01528227049857378,
+          0.013432229869067669,
+          -0.021345429122447968,
+          0.012017492204904556,
+          -0.011776520870625973,
+          -2.3729762688162737e-05,
+          0.016230611130595207,
+          -0.0017237251158803701,
+          -0.005969879683107138,
+          0.025822840631008148,
+          -0.0008924698340706527,
+          -0.01952648162841797,
+          -0.006867693737149239,
+          0.008309637196362019,
+          0.0027167617809027433,
+          -0.0014555469388142228,
+          -0.009405669756233692,
+          0.00044502035598270595,
+          0.019852960482239723,
+          0.03765377402305603,
+          -0.034046970307826996,
+          0.004263644572347403,
+          0.003103482536971569,
+          0.0013544942485168576,
+          0.011675468645989895,
+          -0.00826299749314785,
+          -0.006311904173344374,
+          0.019775226712226868,
+          -0.006673361640423536,
+          -0.0016508505214005709,
+          0.006342997308820486,
+          -0.058579444885253906,
+          -0.0149557925760746,
+          0.043219443410634995,
+          -0.01955757476389408,
+          0.0007306884508579969,
+          0.03183935955166817,
+          -0.005037086084485054,
+          0.013066885061562061,
+          -0.029616201296448708,
+          -0.0018306076526641846,
+          -0.0016421056352555752,
+          0.02453247457742691,
+          -0.013066885061562061,
+          -0.002024939749389887,
+          0.017940733581781387,
+          -0.0015799193643033504,
+          -0.01384421344846487,
+          0.024858953431248665,
+          0.01280259434133768,
+          -0.0065528759732842445,
+          2.377530836383812e-05,
+          0.029569560661911964,
+          0.004279191140085459,
+          0.017132312059402466,
+          -0.023273203521966934,
+          0.0104317432269454,
+          -0.0028158710338175297,
+          -0.011271257884800434,
+          0.024330370128154755,
+          -0.005223644897341728,
+          0.011862026527523994,
+          0.025931665673851967,
+          0.031823813915252686,
+          -0.0011795953614637256,
+          -0.004792227875441313,
+          0.0010289880447089672,
+          0.00016068830154836178,
+          0.023646321147680283,
+          0.020381543785333633,
+          -0.011115792207419872,
+          -0.013883080333471298,
+          -0.019992878660559654,
+          0.005655061919242144,
+          -0.0072524710558354855,
+          0.0032744945492595434,
+          0.034948673099279404,
+          0.027346402406692505,
+          0.000937652017455548,
+          0.028045998886227608,
+          -0.016354983672499657,
+          0.017210043966770172,
+          0.011838707141578197,
+          -0.019339922815561295,
+          -0.014357250183820724,
+          0.00619919179007411,
+          -0.016852473840117455,
+          0.011154658161103725,
+          -0.00277117476798594,
+          -0.013276764191687107,
+          -0.006261378061026335,
+          0.08699855953454971,
+          -0.01742769591510296,
+          0.021687453612685204,
+          -0.056869324296712875,
+          -0.027641788125038147,
+          -0.011574415490031242,
+          -0.02359968051314354,
+          0.019837412983179092,
+          0.04312616214156151,
+          0.014030772261321545,
+          0.01943320222198963,
+          0.011310123838484287,
+          -0.006479030009359121,
+          0.03132632374763489,
+          0.0038438874762505293,
+          0.012958060018718243,
+          0.001303967903368175,
+          -0.010501702316105366,
+          -0.013921946287155151,
+          -0.012281783856451511,
+          -0.0015459113055840135,
+          0.012763727456331253,
+          0.005697815213352442,
+          0.010113039053976536,
+          -0.012141864746809006,
+          -0.013657654635608196,
+          -0.006743321195244789,
+          0.021438708528876305,
+          0.00853506289422512,
+          -0.014225104823708534,
+          0.027253123000264168,
+          0.012584942393004894,
+          -0.00026744944625534117,
+          0.010594981722533703,
+          -0.0025088265538215637,
+          0.053697824478149414,
+          -0.02740858867764473,
+          0.007885993458330631,
+          0.031932637095451355,
+          -0.009281297214329243,
+          -0.0027886645402759314,
+          -0.010952552780508995,
+          0.00701927300542593,
+          0.023304296657443047,
+          -0.010672714561223984,
+          0.006747208070009947,
+          0.028045998886227608,
+          -0.005079839378595352,
+          -0.009079192765057087,
+          -0.02139206975698471,
+          -0.008721621707081795,
+          0.01993069238960743,
+          -0.0048194341361522675,
+          0.007427370175719261,
+          -0.004842753987759352,
+          0.015655387192964554,
+          0.00845733005553484,
+          0.0008463159902021289,
+          0.0013272877549752593,
+          -0.0023747372906655073,
+          0.004321944434195757,
+          0.0003070445964112878,
+          0.01666591502726078,
+          0.03625458478927612,
+          0.011823160573840141,
+          0.00236113416031003,
+          0.011310123838484287,
+          -0.0028119843918830156,
+          -0.01548437587916851,
+          0.011457816697657108,
+          -0.00508761266246438,
+          0.002357247518375516,
+          0.009483402594923973,
+          -0.013735387474298477,
+          -0.0060670459643006325,
+          0.016603728756308556,
+          -0.008822673931717873,
+          -0.0034377335105091333,
+          0.008239678107202053,
+          -0.02321101725101471,
+          0.008014252409338951,
+          -0.0010435630101710558,
+          -0.01482364721596241,
+          0.00891595333814621,
+          -0.026973284780979156,
+          -0.0013496358878910542,
+          -0.0012806480517610908,
+          -0.015694255009293556,
+          0.014015225693583488,
+          0.0013127127895131707,
+          0.0011659921146929264,
+          0.0010503645753487945,
+          -0.02073911391198635,
+          0.00040761142736300826,
+          -0.01186980027705431,
+          -0.004100405611097813,
+          -0.0013826723443344235,
+          -0.000706882739905268,
+          -0.05671386048197746,
+          -0.010548342019319534,
+          -0.03389150649309158,
+          0.003078219247981906,
+          0.029693933203816414,
+          0.0003570851113181561,
+          -0.01113911159336567,
+          -0.011722107417881489,
+          0.025154337286949158,
+          -0.004388017114251852,
+          -0.004481296520680189,
+          -0.00277117476798594,
+          -0.008900406770408154,
+          -0.004908827133476734,
+          -0.0009944941848516464,
+          -0.01851595565676689,
+          0.03270996734499931,
+          0.011030285619199276,
+          0.0045978957787156105,
+          -0.03289652615785599,
+          -0.0015585428336635232,
+          -0.0015196764143183827,
+          -0.02464130148291588,
+          -0.016370529308915138,
+          0.023506401106715202,
+          0.011193525046110153,
+          0.0010474496521055698,
+          0.0028255877550691366,
+          0.02345976233482361,
+          -0.018313849344849586,
+          0.021703001111745834,
+          0.01055611576884985,
+          -0.0010940893553197384,
+          3.4888671507360414e-05,
+          0.00423255143687129,
+          -0.014808100648224354,
+          0.014862513169646263,
+          -0.00866720825433731,
+          0.029802760109305382,
+          -0.01756761595606804,
+          0.008970366790890694,
+          -0.009965346194803715,
+          0.019060084596276283,
+          0.03911514952778816,
+          0.030595634132623672,
+          0.026724539697170258,
+          0.012670448049902916,
+          -0.03647223487496376,
+          -0.01009749248623848,
+          -0.002440810203552246,
+          0.005538462661206722,
+          0.009537816047668457,
+          0.004842753987759352,
+          -0.015212311409413815,
+          0.01851595565676689,
+          -0.0005421863752417266,
+          0.006284697912633419,
+          -0.021594174206256866,
+          0.008239678107202053,
+          -0.002110445871949196,
+          0.025418629869818687,
+          0.025745106860995293,
+          0.011963079683482647,
+          -0.0018160328036174178,
+          -0.013035791926085949,
+          -0.007532309275120497,
+          0.02047482319176197,
+          0.013611014932394028,
+          0.022604702040553093,
+          -0.023319844156503677,
+          0.01344000268727541,
+          0.006047612521797419,
+          0.0022659115493297577,
+          0.01940210908651352,
+          0.02227822318673134,
+          0.021454256027936935,
+          0.0013272877549752593,
+          -0.008869313634932041,
+          0.01941765658557415,
+          -0.027346402406692505,
+          0.0049399202689528465,
+          0.0017655064584687352,
+          0.013696521520614624,
+          -0.0364411398768425,
+          0.0016217008233070374,
+          -0.0026273690164089203,
+          -0.0013807290233671665,
+          -0.0113412169739604,
+          -0.029569560661911964,
+          -0.018173931166529655,
+          -0.041509322822093964,
+          0.02204502560198307,
+          -0.004321944434195757,
+          0.031932637095451355,
+          0.007104779127985239,
+          -0.0095222694799304,
+          0.0037972477730363607,
+          0.012219597585499287,
+          0.013603242114186287,
+          0.004710608161985874,
+          0.007862674072384834,
+          -0.023413121700286865,
+          0.00718639837577939,
+          0.029289722442626953,
+          -0.00826299749314785,
+          0.01744324341416359,
+          0.005965993274003267,
+          -0.019852960482239723,
+          0.0041975718922913074,
+          0.011589962057769299,
+          0.007990933023393154,
+          -0.028559034690260887,
+          0.02439255639910698,
+          -0.01515012513846159,
+          -0.009988666512072086,
+          0.009825427085161209,
+          -0.012841460295021534,
+          0.02153198793530464,
+          -0.019852960482239723,
+          -0.01666591502726078,
+          0.00788210704922676,
+          0.003323077689856291,
+          -0.013976359739899635,
+          0.0032064784318208694,
+          0.023910611867904663,
+          -0.017769720405340195,
+          0.0004063968372065574,
+          -0.0045123896561563015,
+          0.02139206975698471,
+          -0.024890046566724777,
+          0.02151644229888916,
+          -0.010439516045153141,
+          -0.019635308533906937,
+          0.016106238588690758,
+          -0.00715141836553812,
+          0.013618788681924343,
+          0.011108018457889557,
+          0.0034727132879197598,
+          0.001421538763679564,
+          0.01245279610157013,
+          0.01572534814476967,
+          0.009615548886358738,
+          -0.011123565025627613,
+          -0.0012524699559435248,
+          -0.005752228200435638,
+          0.01416291855275631,
+          -0.02638251520693302,
+          -0.0002140081487596035,
+          0.021967291831970215,
+          -0.01836048997938633,
+          -0.015562108717858791,
+          0.006832714192569256,
+          0.00468728831037879,
+          0.009382350370287895,
+          0.003812794340774417,
+          -0.03501085937023163,
+          -0.00603595282882452,
+          -0.013167938217520714,
+          -0.003431903664022684,
+          0.03277215361595154,
+          -0.0025748994667083025,
+          -0.014862513169646263,
+          -0.024952232837677002,
+          -1.4673586520075332e-05,
+          0.010120811872184277,
+          0.01968194730579853,
+          -0.002547692973166704,
+          -0.016106238588690758,
+          0.008581702597439289,
+          0.013028019107878208,
+          0.008294090628623962,
+          0.009879840537905693,
+          -0.009483402594923973,
+          0.00682105403393507,
+          0.02689555287361145,
+          0.03647223487496376,
+          -0.009965346194803715,
+          -0.015468829311430454,
+          0.0007501216023229063,
+          -0.018795793876051903,
+          -0.003995466511696577,
+          -0.013028019107878208,
+          -0.005740568041801453,
+          0.0532936155796051,
+          -0.027377495542168617,
+          -0.004862187430262566,
+          -0.02033490315079689,
+          0.00032696363632567227,
+          0.0016090691788122058,
+          -0.001061052898876369,
+          0.007905426435172558,
+          -0.008698301389813423,
+          -0.03243012726306915,
+          0.007501216139644384,
+          0.009957573376595974,
+          0.000347368506481871,
+          0.03249231353402138,
+          0.0113412169739604,
+          -0.013867533765733242,
+          0.01679028756916523,
+          -0.0021881787106394768,
+          -0.0014001622330397367,
+          -0.01941765658557415,
+          -0.013377816416323185,
+          0.0063740904442965984,
+          0.021827373653650284,
+          -0.0017616198165342212,
+          -0.02059919573366642,
+          -0.00793651957064867,
+          0.0052197580225765705,
+          0.00891595333814621,
+          -0.008550609461963177,
+          0.02345976233482361,
+          0.004380243830382824,
+          -0.018578141927719116,
+          0.020910125225782394,
+          -0.018733607605099678,
+          0.0018704457907006145,
+          -0.010167451575398445,
+          -0.0062963576056063175,
+          0.03790251910686493,
+          0.013579921796917915,
+          -0.020770207047462463,
+          -0.013276764191687107,
+          -0.02715984359383583,
+          0.001835466013289988,
+          0.017241137102246284,
+          0.015686480328440666,
+          0.0036825919523835182,
+          -0.002462186850607395,
+          -0.009017006494104862,
+          -0.01311352476477623,
+          -0.025014419108629227,
+          -0.02516988478600979,
+          0.015818627551198006,
+          -0.0009609718690626323,
+          0.04959353432059288,
+          0.00310153909958899,
+          0.004415223840624094,
+          0.005348017439246178,
+          0.004158705472946167,
+          0.007831580936908722,
+          -0.019635308533906937,
+          -0.007306884042918682,
+          -0.0013107694685459137,
+          -0.021811826154589653,
+          0.004772794432938099,
+          0.012880327180027962,
+          -0.012025265954434872,
+          -0.0007020244374871254,
+          -0.001712065190076828,
+          0.01128680445253849,
+          -0.020521461963653564,
+          0.028014905750751495,
+          0.01874915324151516,
+          0.008294090628623962,
+          -0.0003578138421289623,
+          0.02729976363480091,
+          0.00299854320473969,
+          0.012429476715624332,
+          0.0007656681700609624,
+          0.020288264378905296,
+          0.00422866502776742,
+          0.008970366790890694,
+          -0.015935225412249565,
+          0.008542835712432861,
+          -0.015422189608216286,
+          0.035477254539728165,
+          0.0025496361777186394,
+          -0.0388353131711483,
+          0.02271352708339691,
+          -0.006922106724232435,
+          0.0014477736549451947,
+          -0.010618302039802074,
+          -0.026957739144563675,
+          0.01809619925916195,
+          0.01284923404455185,
+          0.010011985898017883,
+          -0.008441783487796783,
+          0.008193038403987885,
+          -0.02174963988363743,
+          0.005892146844416857,
+          -0.01598186604678631,
+          0.012965832836925983,
+          0.0016595955239608884,
+          0.01396858599036932,
+          -0.003507693065330386,
+          0.023646321147680283,
+          0.0071164388209581375,
+          -0.00330947432667017,
+          0.022184943780303,
+          0.0027109317015856504,
+          0.03491757810115814,
+          0.012934739701449871,
+          0.008022026158869267,
+          0.0211122315376997,
+          0.009537816047668457,
+          0.015290044248104095,
+          -0.013758707791566849,
+          -0.004096519201993942,
+          -0.00017647776985540986,
+          0.015017978847026825,
+          -0.006824940908700228,
+          -0.00743902986869216,
+          -0.02571401372551918,
+          0.009327936917543411,
+          -0.0008166803163476288,
+          0.020288264378905296,
+          -0.0077655077911913395,
+          -0.008706075139343739,
+          -0.015002432279288769,
+          -0.044369887560606,
+          -0.011247937567532063,
+          -0.01599741168320179,
+          0.0025418628938496113,
+          0.0024738467764109373,
+          -0.001715951832011342,
+          -0.020039519295096397,
+          0.02162526734173298,
+          -0.0005528746405616403,
+          -0.023972798138856888,
+          -0.0034377335105091333,
+          0.012903646565973759,
+          -0.01626170426607132,
+          0.012546075507998466,
+          -0.004792227875441313,
+          -0.016572635620832443,
+          0.013976359739899635,
+          -0.0011669638333842158,
+          -0.0036981385201215744,
+          -0.027361949905753136,
+          -0.011589962057769299,
+          -0.01455158181488514,
+          0.003192875301465392,
+          0.0019005672074854374,
+          0.010315144434571266,
+          0.011372310109436512,
+          -0.0179718267172575,
+          0.0017256684368476272,
+          0.015398870222270489,
+          0.008224131539463997,
+          0.004302510991692543,
+          0.002841134322807193,
+          0.012079678475856781,
+          -0.009273524396121502,
+          0.0004821863549295813,
+          -0.011403403244912624,
+          -0.01993069238960743,
+          0.005208098329603672,
+          0.010408422909677029,
+          -0.008643888868391514,
+          -0.03538397699594498,
+          -0.007699435111135244,
+          0.013626561500132084,
+          -0.009794333949685097,
+          -0.0009284212719649076,
+          -0.017163405194878578,
+          -0.0005212956457398832,
+          -0.005122592207044363,
+          0.007959839887917042,
+          -0.03019142337143421,
+          -0.0036223488859832287,
+          -0.007777167949825525,
+          -0.0017781381029635668,
+          -0.008177491836249828,
+          -0.0017470449674874544,
+          0.044121142476797104,
+          0.013696521520614624,
+          0.006941540166735649,
+          -0.006405183579772711,
+          -0.004928260110318661,
+          -0.006273037753999233,
+          -0.0003541701298672706,
+          -0.006720001343637705,
+          0.0004105264088138938,
+          -0.014225104823708534,
+          0.016090691089630127,
+          -0.0058455076068639755,
+          0.01430283673107624,
+          0.011862026527523994,
+          -0.007135872263461351,
+          0.026195956394076347,
+          0.0030374096240848303,
+          0.03370494768023491,
+          -0.012421702966094017,
+          0.0014322270872071385,
+          0.017381057143211365,
+          0.0844489261507988,
+          0.013471095822751522,
+          0.008325183764100075,
+          0.0027964378241449594,
+          -0.017940733581781387,
+          -0.009265750646591187,
+          0.003678705310449004,
+          -0.028279196470975876,
+          0.023024458438158035,
+          0.0020987859461456537,
+          0.00997311994433403,
+          -0.007528422866016626,
+          -0.0010843727504834533,
+          0.006024292670190334,
+          0.009763240814208984,
+          0.011815386824309826,
+          0.002611822448670864,
+          0.01285700686275959,
+          0.040483247488737106,
+          0.021547535434365273,
+          0.0014011339517310262,
+          -0.016479356214404106,
+          -0.0010639678221195936,
+          -0.015173444524407387,
+          0.006739434786140919,
+          0.007827693596482277,
+          -0.008744941093027592,
+          0.023055551573634148,
+          0.0021609722170978785,
+          0.010804860852658749,
+          0.013121298514306545,
+          -0.0049127135425806046,
+          0.0021026725880801678,
+          -0.003991579636931419,
+          0.0007374900160357356,
+          0.007497329730540514,
+          -0.006537329405546188,
+          0.008216357789933681,
+          -0.006082592532038689,
+          -0.013859760016202927,
+          0.006999839562922716,
+          -0.010120811872184277,
+          0.003097652457654476,
+          0.0058105275966227055,
+          -0.012514982372522354,
+          0.012048585340380669,
+          0.012491662986576557,
+          0.006261378061026335,
+          0.0006869637290947139,
+          -0.002522429684177041,
+          -0.005208098329603672,
+          0.006074819248169661,
+          0.011636601760983467,
+          -0.0005669636884704232,
+          -0.031792718917131424,
+          -0.0073651839047670364,
+          0.0075361961498856544,
+          -0.004582349210977554,
+          0.005433523561805487,
+          0.01876470074057579,
+          -0.0022892311681061983,
+          -0.013991906307637691,
+          0.00011848181020468473,
+          -0.015049071982502937,
+          -0.00731077091768384,
+          -0.0023144944570958614,
+          0.015834173187613487,
+          -0.010470609180629253,
+          0.016976846382021904,
+          0.012172957882285118,
+          0.00640129717066884,
+          -0.007159191649407148,
+          0.014388343319296837,
+          -0.023972798138856888,
+          0.0019977332558482885,
+          0.0008827532292343676,
+          -0.01161328237503767,
+          -0.0020502028055489063,
+          0.011768747121095657,
+          0.020023971796035767,
+          0.004609555471688509,
+          0.004329717718064785,
+          -0.0011154658859595656,
+          0.013657654635608196,
+          -0.011714334599673748,
+          -0.007816034369170666,
+          0.006673361640423536,
+          0.0006918220315128565,
+          -0.005713361781090498,
+          -0.024719033390283585,
+          -0.0027808912564069033,
+          0.009055872447788715,
+          0.0190445389598608,
+          -0.0047689080238342285,
+          -0.01943320222198963,
+          0.017474336549639702,
+          0.014201784506440163,
+          -0.00899368617683649,
+          -0.010408422909677029,
+          -0.015810852870345116,
+          0.005721135064959526,
+          0.024485835805535316,
+          -0.0014856683555990458,
+          -0.020505916327238083,
+          -0.011325670406222343,
+          -0.014442755840718746,
+          0.018422676250338554,
+          0.007299110759049654,
+          0.011045832186937332,
+          0.0003648583951871842,
+          -0.003826397703960538,
+          0.00024145754287019372,
+          0.014481622725725174,
+          0.020521461963653564,
+          -0.0021318222861737013,
+          -0.0002613765827845782,
+          0.0023008910939097404,
+          0.00033497982076369226,
+          -0.011372310109436512,
+          0.0018072878010571003,
+          0.0024660734925419092,
+          0.017412150278687477,
+          0.0034299602266401052,
+          0.0016693121287971735,
+          0.0049127135425806046,
+          -0.022184943780303,
+          -0.006168098654597998,
+          -0.01665036752820015,
+          -0.008962593041360378,
+          -0.015220084227621555,
+          -0.015919679775834084,
+          -0.005449070129543543,
+          -0.007800487335771322,
+          0.008091985248029232,
+          0.01823611743748188,
+          -0.01108469907194376,
+          -0.0011990285711362958,
+          0.019510935992002487,
+          -0.0031287455931305885,
+          0.004139272030442953,
+          -0.0028936038725078106,
+          -0.0014487452572211623,
+          -0.01876470074057579,
+          -0.003908017184585333,
+          0.015220084227621555,
+          0.010501702316105366,
+          0.012561622075736523,
+          -0.0066383820958435535,
+          0.01940210908651352,
+          -0.025931665673851967,
+          0.025263164192438126,
+          -0.020770207047462463,
+          -0.0014147371985018253,
+          0.021018952131271362,
+          0.0029655066318809986,
+          -0.004609555471688509,
+          -0.015103485435247421,
+          0.01232065074145794,
+          -0.02097231149673462,
+          -0.005550122819840908,
+          -0.034171342849731445,
+          0.03358057513833046,
+          -0.010696034878492355,
+          0.0009998382302001119,
+          -0.004753361456096172,
+          0.0417269729077816,
+          0.012895873747766018,
+          0.005635628942400217,
+          -0.00011544537119334564,
+          -0.014668181538581848,
+          -0.0025379762519150972,
+          0.01240615639835596,
+          -0.0051886653527617455,
+          0.0040071262046694756,
+          0.01260048896074295,
+          0.0066306088119745255,
+          0.0013049395056441426,
+          0.010626074858009815,
+          0.00833295751363039,
+          0.007955953478813171,
+          -0.01449716929346323,
+          -0.010649395175278187,
+          -0.009234657511115074,
+          0.018065106123685837,
+          0.005449070129543543,
+          0.00866720825433731,
+          -0.003737004939466715,
+          0.008402916602790356,
+          0.015686480328440666,
+          -0.007831580936908722,
+          -0.015771986916661263,
+          -0.010571662336587906,
+          -0.016588181257247925,
+          -0.0330519899725914,
+          -0.003834170987829566,
+          0.011955305933952332,
+          -0.006731661502271891,
+          -0.000800162146333605,
+          0.010696034878492355,
+          -0.003268664702773094,
+          0.014232877641916275,
+          0.028792232275009155,
+          -0.0058105275966227055,
+          -0.0016071258578449488,
+          0.0009478544816374779,
+          0.002372794086113572,
+          0.01599741168320179,
+          0.008193038403987885,
+          0.03504195064306259,
+          -0.015375549905002117,
+          -0.0427219532430172,
+          -0.003435790305957198,
+          0.012965832836925983,
+          -0.01494801975786686,
+          -0.021143324673175812,
+          0.005452956538647413,
+          -0.005934900138527155,
+          -0.0052158716134727,
+          0.028698952868580818,
+          0.0021434822119772434,
+          -0.01469927467405796,
+          0.008737168274819851,
+          0.0453648678958416,
+          -0.0031423489563167095,
+          0.013416683301329613,
+          -0.005585102364420891,
+          0.016992392018437386,
+          -0.032865431159734726,
+          -0.013292310759425163,
+          -0.02347530797123909,
+          0.00932016409933567,
+          0.01232842355966568,
+          -0.0008254252607002854,
+          0.032212477177381516,
+          0.016292797401547432,
+          0.012180731631815434,
+          -0.004201458301395178,
+          0.016292797401547432,
+          -0.032057009637355804,
+          -0.04063871502876282,
+          -0.007097005844116211,
+          -0.012864780612289906,
+          -0.009965346194803715,
+          0.001216518459841609,
+          0.007295224349945784,
+          0.0013253444340080023,
+          0.02768842689692974,
+          0.0032978144008666277,
+          -0.010874819941818714,
+          -2.778340785880573e-05,
+          -0.0068171676248312,
+          -0.002674008719623089,
+          -0.019339922815561295,
+          0.010571662336587906,
+          0.008729394525289536,
+          0.016712553799152374,
+          -0.011574415490031242,
+          -0.012079678475856781,
+          0.0012534415582194924,
+          -0.021189963445067406,
+          -0.019308829680085182,
+          -0.027455229312181473,
+          0.007831580936908722,
+          -0.015709800645709038,
+          0.012872553430497646,
+          0.011247937567532063,
+          0.013743161223828793,
+          0.0018344942945986986,
+          -0.013129071332514286,
+          0.027843892574310303,
+          -0.015841946005821228,
+          0.007217491511255503,
+          0.01344000268727541,
+          0.00871384795755148,
+          0.00820858497172594,
+          0.0021609722170978785,
+          -0.027641788125038147,
+          -0.011691015213727951,
+          0.015663161873817444,
+          -0.010423969477415085,
+          0.005794981028884649,
+          -0.009335710667073727,
+          -0.0017927129520103335,
+          -0.007816034369170666,
+          -0.008224131539463997,
+          -0.003958543296903372,
+          0.011574415490031242,
+          0.01462931465357542,
+          0.0012952230172231793,
+          0.026833366602659225,
+          -0.023008912801742554,
+          0.004003239795565605,
+          -0.005184778477996588,
+          0.0067744143307209015,
+          -0.001097004278562963,
+          0.006685021799057722,
+          0.018142838031053543,
+          -0.007163078524172306,
+          0.011504456400871277,
+          0.00587271386757493,
+          -0.015639841556549072,
+          -0.012172957882285118,
+          -0.02137652225792408,
+          0.002742024837061763,
+          -0.016603728756308556,
+          -0.0005878543597646058,
+          -0.001586721045896411,
+          -0.003968259785324335,
+          0.021936198696494102,
+          -0.0040926323272287846,
+          -0.023444214835762978,
+          0.024066077545285225,
+          0.011605508625507355,
+          0.023553041741251945,
+          -0.0030354661867022514,
+          0.030533447861671448,
+          -0.011706560850143433,
+          0.002701215213164687,
+          -0.027626240625977516,
+          0.03187045082449913,
+          -0.00419368501752615,
+          0.009763240814208984,
+          -0.011434496380388737,
+          -0.01148890983313322,
+          -0.019868506118655205,
+          -0.0043685841374099255,
+          0.008744941093027592,
+          0.012577168643474579,
+          0.0051031592302024364,
+          -0.015803080052137375,
+          0.023521948605775833,
+          0.004858300555497408,
+          -0.01239838358014822,
+          0.0009920649463310838,
+          0.009351257234811783,
+          -0.009032553061842918,
+          -0.006444049999117851,
+          -0.0058183008804917336,
+          0.01848486252129078,
+          -0.006852147169411182,
+          -0.01089036650955677,
+          -0.03267887234687805,
+          -0.01575644128024578,
+          -0.0033580574672669172,
+          0.0021706887055188417,
+          0.01927773654460907,
+          -0.007769394665956497,
+          -0.01795627921819687,
+          0.0028178144712001085,
+          0.010004213079810143,
+          -0.016044052317738533,
+          0.00011028342851204798,
+          -0.0018976522842422128,
+          0.01572534814476967,
+          -0.0047689080238342285,
+          0.018205024302005768,
+          0.0208634864538908,
+          0.036907538771629333,
+          0.0022659115493297577,
+          -0.0029557901434600353,
+          0.010734900832176208,
+          0.010610528290271759,
+          -0.008472876623272896,
+          -0.007384616881608963,
+          -0.007722754962742329,
+          -0.02215385064482689,
+          0.034451182931661606,
+          -0.009219110943377018,
+          -0.006063159089535475,
+          -0.00067287462297827,
+          0.022635795176029205,
+          -0.0034902032930403948,
+          -0.025263164192438126,
+          -0.024765674024820328,
+          -0.011146885342895985,
+          0.04645312950015068,
+          -0.021842919290065765,
+          -0.01153554953634739,
+          -0.026335876435041428,
+          0.01745878905057907,
+          -0.02017943747341633,
+          -0.0035193529911339283,
+          0.0011076926020905375,
+          0.005569555796682835,
+          -0.0033036444801837206,
+          -0.005324697587639093,
+          -0.004213118460029364,
+          0.0054685031063854694,
+          0.01318348478525877,
+          0.016572635620832443,
+          -0.014380569569766521,
+          0.021423162892460823,
+          -0.00932016409933567,
+          0.008325183764100075,
+          -0.016479356214404106,
+          0.007990933023393154,
+          -0.023257657885551453,
+          0.024485835805535316,
+          -0.007812147494405508,
+          -0.024205997586250305,
+          0.012297330424189568,
+          0.012499435804784298,
+          0.012693768367171288,
+          0.00015303646796382964,
+          0.012841460295021534,
+          -0.023677414283156395,
+          -0.0005470446776598692,
+          -0.006929880008101463,
+          -0.00426753144711256,
+          -0.01311352476477623,
+          -0.011317897588014603,
+          0.013657654635608196,
+          -0.0010814577108249068,
+          -0.012118545360863209,
+          0.019386563450098038,
+          -0.0066306088119745255,
+          -0.012996925972402096,
+          -0.02478121966123581,
+          0.006541215814650059,
+          -0.007827693596482277,
+          0.005324697587639093,
+          -0.0007005669758655131,
+          0.01599741168320179,
+          -0.005456843413412571,
+          -0.001588664366863668,
+          0.008814901113510132,
+          0.013704294338822365,
+          0.004566802643239498,
+          -0.012717087753117085,
+          -0.023661866784095764,
+          0.0007758706342428923,
+          0.011504456400871277,
+          0.014730367809534073,
+          -0.006580082233995199,
+          -0.017474336549639702,
+          -0.00017793526058085263,
+          -0.0010843727504834533,
+          -0.00970105454325676,
+          -0.0041975718922913074,
+          -0.00041465595131739974,
+          -0.011131338775157928,
+          -0.003937166649848223,
+          -0.01076599396765232,
+          0.017349964007735252,
+          0.0032025917898863554,
+          0.015546562150120735,
+          -0.0040693124756217,
+          -0.019728587940335274,
+          0.012056359089910984,
+          0.041011832654476166,
+          -0.019713040441274643,
+          0.021967291831970215,
+          -0.008099758997559547,
+          -0.0015993525739759207,
+          -0.014862513169646263,
+          0.015468829311430454,
+          -0.006665588356554508,
+          0.018733607605099678,
+          0.020925672724843025,
+          0.008643888868391514,
+          -0.0021551421377807856,
+          0.016836926341056824,
+          0.021687453612685204,
+          -0.0008210528176277876,
+          0.0062380582094192505,
+          0.019184457138180733,
+          0.008286317810416222,
+          0.005363564006984234,
+          -0.0020016198977828026,
+          -0.030067050829529762,
+          0.006805507466197014,
+          -0.0005766802933067083,
+          -0.004803887568414211,
+          0.021189963445067406,
+          0.02504551224410534,
+          0.010991419665515423,
+          0.0033036444801837206,
+          0.006148665212094784,
+          0.012095225043594837,
+          0.0026409721467643976,
+          0.0031540088821202517,
+          -0.0042558712884783745,
+          -0.0018685024697333574,
+          -0.009856520220637321,
+          0.004003239795565605,
+          0.008177491836249828,
+          -0.004691175185143948,
+          0.017489882186055183,
+          -0.01148890983313322,
+          -0.014504942111670971,
+          -0.010260730981826782,
+          -0.014186237938702106,
+          0.02176518738269806,
+          -0.017878547310829163,
+          -0.000856032595038414,
+          -0.0036301221698522568,
+          -6.595143349841237e-05,
+          0.00037141708889976144,
+          -0.00853506289422512,
+          0.02188955992460251,
+          0.008410690352320671,
+          -0.022651340812444687,
+          0.01154332235455513,
+          -0.004240324720740318,
+          0.005798867903649807,
+          -0.006995953153818846,
+          0.01030737068504095,
+          -0.01598186604678631,
+          0.010571662336587906,
+          -0.008845994248986244,
+          0.02296227216720581,
+          -0.013090205378830433,
+          -0.005425750277936459,
+          0.0041120657697319984,
+          -0.002913037082180381,
+          0.01693020574748516,
+          -0.0037991912104189396,
+          -0.010369556955993176,
+          0.003678705310449004,
+          0.011963079683482647,
+          -0.006350770592689514,
+          0.009506722912192345,
+          0.011185751296579838,
+          0.003565992694348097,
+          -0.006109798792749643,
+          -0.0015595145523548126,
+          0.00619919179007411,
+          -0.02927417680621147,
+          -0.00925020407885313,
+          0.0213609766215086,
+          -0.017862999811768532,
+          -0.004916600417345762,
+          0.017520975321531296,
+          0.015445509925484657,
+          -0.01916891150176525,
+          -0.00558121595531702,
+          0.008744941093027592,
+          -0.0020657493732869625,
+          0.023179924115538597,
+          -0.009351257234811783,
+          -0.01719449833035469,
+          0.01277927402406931,
+          -0.013867533765733242,
+          0.011022512800991535,
+          -0.013261217623949051,
+          0.000477813882753253,
+          -0.015639841556549072,
+          0.007952066138386726,
+          -0.014100732281804085,
+          -0.022216036915779114,
+          0.011675468645989895,
+          -0.008690528571605682,
+          -0.001947206910699606,
+          -0.023972798138856888,
+          0.0299115851521492,
+          0.032057009637355804,
+          0.026133770123124123,
+          -0.02005506493151188,
+          0.0026293122209608555,
+          -0.007695548236370087,
+          -0.018438221886754036,
+          -0.0034688266459852457,
+          -0.01166769489645958,
+          0.01654154248535633,
+          0.01940210908651352,
+          -0.002988826483488083,
+          -0.018142838031053543,
+          -0.016090691089630127,
+          0.004671741742640734,
+          -0.02202947810292244,
+          -0.01113911159336567,
+          -0.014613768085837364,
+          -0.0004333604301791638,
+          -0.02112777717411518,
+          0.007613928988575935,
+          0.002753684762865305,
+          0.008892633952200413,
+          0.0028294743970036507,
+          0.003171498654410243,
+          -0.019884053617715836,
+          -0.007170851808041334,
+          0.011076925322413445,
+          0.007190284784883261,
+          -0.012740408070385456,
+          0.02795271947979927,
+          -0.009265750646591187,
+          -0.0015750611200928688,
+          -0.0020754660945385695,
+          -0.018857980147004128,
+          0.013525509275496006,
+          0.010214091278612614,
+          -0.016090691089630127,
+          -0.0071669649332761765,
+          -0.010548342019319534,
+          0.0025612961035221815,
+          0.010797087103128433,
+          -0.001009554835036397,
+          -0.006129232235252857,
+          -0.010369556955993176,
+          0.008418463170528412,
+          -0.011154658161103725,
+          -0.00800647959113121,
+          -0.012942513450980186,
+          0.002843077527359128,
+          0.017520975321531296,
+          0.004154818598181009,
+          0.0020793527364730835,
+          -0.019479842856526375,
+          0.005351903848350048,
+          -0.01795627921819687,
+          0.0003585426020435989,
+          0.0009386236779391766,
+          -0.020365996286273003,
+          0.007959839887917042,
+          0.006428503431379795,
+          0.004543482791632414,
+          0.003585425904020667,
+          -0.005690041929483414,
+          0.008643888868391514,
+          -0.02109668403863907,
+          -0.010952552780508995,
+          0.025154337286949158,
+          -0.0025826727505773306,
+          0.015360003337264061,
+          0.012157411314547062,
+          -0.0024991098325699568,
+          -0.009592228569090366,
+          -0.026817819103598595,
+          -0.0035232396330684423,
+          -0.007835467346012592,
+          0.011481136083602905,
+          0.0019977332558482885,
+          -0.028932152315974236,
+          -0.0036262355279177427,
+          -0.026600167155265808,
+          -0.0027808912564069033,
+          -0.019060084596276283,
+          0.00800647959113121,
+          0.018205024302005768,
+          0.00252437312155962,
+          0.006615062244236469,
+          0.00041538471123203635,
+          0.0011018626391887665,
+          -0.0027439682744443417,
+          0.013541055843234062,
+          0.018034012988209724,
+          0.011706560850143433,
+          -0.019231097772717476,
+          0.011045832186937332,
+          -0.003169555449858308,
+          -0.020801300182938576,
+          0.006548989098519087,
+          -0.02558964118361473,
+          -0.007089232560247183,
+          -3.501012906781398e-05,
+          0.0018626725068315864,
+          -0.011760974302887917,
+          -1.905364842968993e-05,
+          -0.0007433199789375067,
+          0.001582834403961897,
+          0.005114818923175335,
+          0.013595468364655972,
+          -0.012219597585499287,
+          0.019308829680085182,
+          0.006583969108760357,
+          0.008185264654457569,
+          0.0020910126622766256,
+          0.0048544141463935375,
+          -0.013074658811092377,
+          -0.03389150649309158,
+          -0.017940733581781387,
+          -0.013028019107878208,
+          0.008340730331838131,
+          0.0029616199899464846,
+          0.005410203710198402,
+          -0.02454802207648754,
+          -0.016588181257247925,
+          0.012250690720975399,
+          0.01680583320558071,
+          0.00793651957064867,
+          0.007998705841600895,
+          -0.014139598235487938,
+          -0.00846510287374258,
+          0.0038147377781569958,
+          -8.122471626847982e-06,
+          0.01403854601085186,
+          0.00763724884018302,
+          -0.006937653291970491,
+          0.008278544060885906,
+          -0.0023961139377206564,
+          -0.02321101725101471,
+          -0.0019578952342271805,
+          0.010004213079810143,
+          -0.007784941233694553,
+          -0.006420730147510767,
+          -0.0009779758984223008,
+          0.008519516326487064,
+          0.011582189239561558,
+          -0.00443077040836215,
+          -0.0013904456282034516,
+          -0.0042830780148506165,
+          0.018982352688908577,
+          -0.015181218273937702,
+          -0.006273037753999233,
+          0.010937006212770939,
+          -0.004831094294786453,
+          0.013579921796917915,
+          0.006113685667514801,
+          0.04076308757066727,
+          -0.012413930147886276,
+          -0.013984132558107376,
+          -0.014738140627741814,
+          -0.008908180519938469,
+          -0.031046485528349876,
+          -0.01812729239463806,
+          -0.022262677550315857,
+          -0.005923239979892969,
+          0.0035446162801235914,
+          0.011691015213727951,
+          0.013774254359304905,
+          -0.007135872263461351,
+          -0.0002958705008495599,
+          0.002988826483488083,
+          0.0008429151494055986,
+          -0.01088259369134903,
+          -0.008954820223152637,
+          -0.0009687451529316604,
+          -0.014147371985018253,
+          0.002203725278377533,
+          -0.018920166417956352,
+          0.009716601110994816,
+          0.018982352688908577,
+          0.013719840906560421,
+          -0.007967612706124783,
+          0.01265490148216486,
+          -0.0136343352496624,
+          0.0013807290233671665,
+          -0.007952066138386726,
+          0.011675468645989895,
+          0.03488648682832718,
+          0.0038108511362224817,
+          -0.02257360890507698,
+          0.0009765184368006885,
+          -0.0012524699559435248,
+          -0.018780246376991272,
+          0.0056706084869802,
+          -0.021687453612685204,
+          0.01943320222198963,
+          -0.015461056493222713,
+          -0.006770527921617031,
+          6.097167351981625e-05,
+          0.0009036439005285501,
+          0.02479676716029644,
+          0.010742674581706524,
+          -0.015119032002985477,
+          0.019961785525083542,
+          0.011263484135270119,
+          -0.006168098654597998,
+          0.007990933023393154,
+          0.02229377068579197,
+          -0.01029959786683321,
+          0.010719354264438152,
+          0.01943320222198963,
+          -0.02031935751438141,
+          -0.009460083208978176,
+          -0.00301797641441226,
+          0.019324377179145813,
+          0.00845733005553484,
+          -0.009436762891709805,
+          0.0036301221698522568,
+          0.00021121461759321392,
+          0.004504616372287273,
+          -0.0014244538033381104,
+          -0.011240164749324322,
+          -0.006012632977217436,
+          -0.004038219340145588,
+          -0.0001887449761852622,
+          0.010517248883843422,
+          -0.015072392299771309,
+          -0.005942673422396183,
+          -0.0018490692600607872,
+          0.015367777086794376,
+          -0.022884540259838104,
+          -0.0017295550787821412,
+          0.005068179219961166,
+          -0.006397410295903683,
+          0.007139758672565222,
+          0.020023971796035767,
+          0.01783190667629242,
+          0.033518388867378235,
+          -0.014528262428939342,
+          0.0033483407460153103,
+          0.02232486382126808,
+          -0.0036106889601796865,
+          0.02031935751438141,
+          0.014403889887034893,
+          -0.004178138449788094,
+          -0.007470123004168272,
+          0.013572148978710175,
+          0.029989318922162056,
+          0.011955305933952332,
+          0.001268016523681581,
+          -0.02336648292839527,
+          -0.0017888263100758195,
+          0.021594174206256866,
+          0.009086965583264828,
+          0.0005387855344451964,
+          0.010346237570047379,
+          0.014815873466432095,
+          -0.013152391649782658,
+          -0.011201297864317894,
+          0.033238548785448074,
+          -0.017132312059402466,
+          -0.02451692894101143,
+          -0.00046615395694971085,
+          -0.005472389981150627,
+          -0.06116017699241638,
+          -0.002351417439058423,
+          0.014668181538581848,
+          -0.0032589479815214872,
+          0.018951259553432465,
+          0.007555629126727581,
+          0.00866720825433731,
+          0.0049982196651399136,
+          -0.01371206808835268,
+          0.003525183070451021,
+          -0.017163405194878578,
+          0.005775548052042723,
+          -0.006245831493288279,
+          -0.003826397703960538,
+          -0.0037739279214292765,
+          0.007497329730540514,
+          0.023413121700286865,
+          -0.0022348184138536453,
+          0.013533282093703747,
+          0.00669279508292675,
+          0.0023280975874513388,
+          0.01808065176010132,
+          0.006595628801733255,
+          0.00406542606651783,
+          0.0008555467356927693,
+          -0.003579595824703574,
+          -0.010315144434571266,
+          -0.018686966970562935,
+          -0.017754174768924713,
+          -0.014442755840718746,
+          0.006012632977217436,
+          0.002687611849978566,
+          -0.01860923506319523,
+          -0.006953199859708548,
+          0.013976359739899635,
+          -0.03348729386925697,
+          0.002287287963554263,
+          0.016852473840117455,
+          0.014248424209654331,
+          0.009755467996001244,
+          0.02165636047720909,
+          0.023273203521966934,
+          -0.0081386249512434,
+          -0.021874012425541878,
+          -0.006949313450604677,
+          0.001381700742058456,
+          -0.012468342669308186,
+          0.017894092947244644,
+          0.023661866784095764,
+          -0.013999679125845432,
+          -0.010804860852658749,
+          -0.004800001159310341,
+          -0.006078705657273531,
+          0.014248424209654331,
+          0.01640162244439125,
+          0.01968194730579853,
+          0.006144778802990913,
+          0.010113039053976536,
+          -0.017552068457007408,
+          0.012491662986576557,
+          0.0133855901658535,
+          -0.009149151854217052,
+          -0.019977333024144173,
+          -0.019184457138180733,
+          -0.021936198696494102,
+          -0.01665036752820015,
+          -0.024750126525759697,
+          0.028263650834560394,
+          0.002454413566738367,
+          0.013129071332514286,
+          0.00977878738194704,
+          -0.014528262428939342,
+          -0.0022348184138536453,
+          0.011263484135270119,
+          0.01680583320558071,
+          0.003188988659530878,
+          0.0005562754231505096,
+          -0.0002934413496404886,
+          0.014124051667749882,
+          -0.015134578570723534,
+          0.007656681817024946,
+          -0.004706721752882004,
+          -0.009910933673381805,
+          0.0034902032930403948,
+          0.005445183254778385,
+          -0.0077655077911913395,
+          -0.012763727456331253,
+          -0.007753848098218441,
+          0.0036689885891973972,
+          -0.007909313775599003,
+          -0.003435790305957198,
+          0.00037165998946875334,
+          -0.0005562754231505096,
+          0.018920166417956352,
+          -0.015064618550240993,
+          -0.0013243727153167129,
+          0.010338463820517063,
+          -0.0064090704545378685,
+          -0.00682105403393507,
+          0.004011013079434633,
+          0.00788210704922676,
+          -0.0063740904442965984,
+          0.007781054358929396,
+          0.016075145453214645,
+          -0.009685507975518703,
+          0.012227371335029602,
+          -0.002194008557125926,
+          0.00587271386757493,
+          0.023008912801742554,
+          -0.001738299964927137,
+          0.002992713125422597,
+          0.011838707141578197,
+          -0.021407615393400192,
+          0.0007097977213561535,
+          -0.015935225412249565,
+          -0.012748180888593197,
+          -0.015546562150120735,
+          -0.010493929497897625,
+          -0.02022607810795307,
+          -0.006704454775899649,
+          0.00937457662075758,
+          -0.004178138449788094,
+          0.010159678757190704,
+          -0.0011941703269258142,
+          -5.492916170624085e-05,
+          -0.020537009462714195,
+          0.0059815398417413235,
+          -0.00788210704922676,
+          0.0016731987707316875,
+          0.0018306076526641846,
+          0.027517415583133698,
+          0.005627855658531189,
+          -0.0029791099950671196,
+          0.005522916093468666,
+          0.013253443874418736,
+          -0.01812729239463806,
+          -0.010610528290271759,
+          0.012087452225387096,
+          0.009631095454096794,
+          -0.009382350370287895,
+          -0.009677735157310963,
+          -0.010820407420396805,
+          -0.031543973833322525,
+          0.006902673747390509,
+          -0.009561135433614254,
+          0.001341862604022026,
+          -0.0002895547077059746,
+          -0.04278413951396942,
+          0.006739434786140919,
+          -0.0019063971703872085,
+          -0.013346723280847073,
+          0.002019109670072794,
+          -0.019619761034846306,
+          0.00871384795755148,
+          -0.01599741168320179,
+          0.01331563014537096,
+          -0.012040812522172928,
+          -0.0023242109455168247,
+          0.009980892762541771,
+          0.023288751021027565,
+          0.0058455076068639755,
+          0.005740568041801453,
+          0.007120325695723295,
+          -8.714576688362285e-05,
+          0.005266397725790739,
+          -0.0022814578842371702,
+          0.0022387050557881594,
+          -0.0017791097052395344,
+          0.015103485435247421,
+          0.002036599675193429,
+          0.01654154248535633,
+          -0.007784941233694553,
+          -0.002930526854470372,
+          -0.008123078383505344,
+          0.01174542773514986,
+          -0.023288751021027565,
+          0.0014720651088282466,
+          0.0007588665466755629,
+          0.0006340082618407905,
+          -0.004123725462704897,
+          0.004535709507763386,
+          -0.007182511501014233,
+          0.031046485528349876,
+          0.016557088121771812,
+          -0.009102512151002884,
+          0.003991579636931419,
+          5.286438317853026e-05,
+          -0.0001824291975935921,
+          0.021734094247221947,
+          -0.01993069238960743,
+          -0.0056744953617453575,
+          0.0027187049854546785,
+          -0.01901344582438469,
+          0.0021881787106394768,
+          0.028512395918369293,
+          -0.014901380054652691,
+          0.001269959844648838,
+          0.00017319842299912125,
+          0.003350284183397889,
+          -0.018811339512467384,
+          0.001708178548142314,
+          0.014613768085837364,
+          0.007575062569230795,
+          -0.003012146335095167,
+          0.015429963357746601,
+          -0.012382837012410164,
+          0.009483402594923973,
+          0.0058688269928097725,
+          -0.0010124698746949434,
+          0.004318057559430599,
+          -0.000593198521528393,
+          -0.02977166697382927,
+          -0.005095385946333408,
+          0.004038219340145588,
+          -0.0019501218339428306,
+          -0.010999192483723164,
+          0.003651498816907406,
+          -0.009685507975518703,
+          -0.01212631817907095,
+          -0.007034819573163986,
+          0.03721847012639046,
+          0.015515469014644623,
+          0.0045123896561563015,
+          0.012437249533832073,
+          -0.0019404053455218673,
+          0.013035791926085949,
+          0.01848486252129078,
+          -0.016588181257247925,
+          -0.004115952178835869,
+          0.015367777086794376,
+          0.018671421334147453,
+          0.009825427085161209,
+          -0.004049879498779774,
+          0.025682920590043068,
+          0.0024952231906354427,
+          -0.020676927641034126,
+          0.006918220315128565,
+          0.0007384616765193641,
+          0.02596275880932808,
+          -0.00698429299518466,
+          -0.016463808715343475,
+          -0.01901344582438469,
+          0.00949894916266203,
+          -0.005600648932158947,
+          -0.005884373560547829,
+          -0.009677735157310963,
+          -0.004617328755557537,
+          0.026398062705993652,
+          0.008527289144694805,
+          0.0006412956863641739,
+          -0.015911906957626343,
+          0.008558382280170918,
+          -0.01862478069961071,
+          -0.013401136733591557,
+          -0.004003239795565605,
+          0.015771986916661263,
+          -0.02571401372551918,
+          -0.001339919283054769,
+          -0.008449556306004524,
+          -0.00013530367868952453,
+          0.012180731631815434,
+          0.021951746195554733,
+          0.0015624294755980372,
+          0.001924858777783811,
+          -0.006051499396562576,
+          -0.024081625044345856,
+          -0.015251177363097668,
+          -0.013727614656090736,
+          -0.007761621382087469,
+          -0.030673367902636528,
+          -0.010035306215286255,
+          -0.0006534414133056998,
+          0.00013821866014041007,
+          -0.0012388667091727257,
+          0.037591587752103806,
+          -0.006583969108760357,
+          -0.00244469684548676,
+          -0.006342997308820486,
+          0.009079192765057087,
+          -0.010664941743016243,
+          0.006261378061026335,
+          -0.006533442530781031,
+          -0.0002924696891568601,
+          0.004045992624014616,
+          0.002048259600996971,
+          0.02807709202170372,
+          0.002182348631322384,
+          -0.006591742392629385,
+          0.008954820223152637,
+          -0.01284923404455185,
+          -0.021967291831970215,
+          0.01693020574748516,
+          -0.024003891274333,
+          -0.013813120312988758,
+          -0.026600167155265808,
+          -0.016728101298213005,
+          -0.017256684601306915,
+          -0.01680583320558071,
+          -0.039954666048288345,
+          0.0023222677409648895,
+          0.02728421613574028,
+          -0.006471256725490093,
+          0.017132312059402466,
+          -0.0023747372906655073,
+          -0.009436762891709805,
+          -0.0010163565166294575,
+          -0.02007061243057251,
+          0.006300244480371475,
+          -0.0016595955239608884,
+          -0.005763887893408537,
+          0.013743161223828793,
+          -0.006560649257153273,
+          0.017396602779626846,
+          0.002920810366049409,
+          -0.019619761034846306,
+          -0.0033833205234259367,
+          0.01876470074057579,
+          0.023397576063871384,
+          -0.0033638873137533665,
+          -0.000545587157830596,
+          0.003752551507204771,
+          -0.010120811872184277,
+          0.01809619925916195,
+          -0.015585429035127163,
+          0.005002106539905071,
+          0.0012446966720744967,
+          0.014085185714066029,
+          0.005375223699957132,
+          0.0009070447413250804,
+          0.006992066279053688,
+          0.0006242916570045054,
+          0.006611175369471312,
+          -0.007411823607981205,
+          0.00134574924595654,
+          0.001861700788140297,
+          -0.0183449424803257,
+          0.03395369276404381,
+          0.013097978197038174,
+          0.012934739701449871,
+          -0.03684535250067711,
+          0.017660895362496376,
+          0.0016440489562228322,
+          -0.008449556306004524,
+          -0.007660568691790104,
+          0.030129237100481987,
+          0.011978626251220703,
+          0.00365538545884192,
+          0.009017006494104862,
+          -0.010594981722533703,
+          0.012157411314547062,
+          -0.011185751296579838,
+          -0.004586235620081425,
+          -0.005421863403171301,
+          -0.011193525046110153,
+          0.01823611743748188,
+          0.009825427085161209,
+          0.0022425916977226734,
+          -0.0046561951749026775,
+          -0.013393362984061241,
+          0.029989318922162056,
+          0.008022026158869267,
+          -0.004784454591572285,
+          0.008651661686599255,
+          -0.014310610480606556,
+          0.0005718219908885658,
+          0.000107489904621616,
+          0.007567289285361767,
+          -0.016728101298213005,
+          -5.696357402484864e-05,
+          -0.028139278292655945,
+          -0.017552068457007408,
+          -0.02335093729197979,
+          0.0002368421555729583,
+          0.0014438870130106807,
+          0.0177230816334486,
+          -0.0011941703269258142,
+          0.004527936223894358,
+          -7.360325253102928e-05,
+          0.007128098979592323,
+          0.004527936223894358,
+          0.014792554080486298,
+          0.012266237288713455,
+          0.02638251520693302,
+          -0.00570558849722147,
+          -0.028792232275009155,
+          0.0041975718922913074,
+          0.007909313775599003,
+          0.008962593041360378,
+          0.027455229312181473,
+          0.003634008811786771,
+          -0.0068949004635214806,
+          -0.00751287629827857,
+          0.008441783487796783,
+          0.025247616693377495,
+          -0.003908017184585333,
+          -0.018002919852733612,
+          0.018313849344849586,
+          0.020630288869142532,
+          0.0033600006718188524,
+          -0.0040693124756217,
+          0.01703903265297413,
+          0.011442270129919052,
+          -0.019122270867228508,
+          -0.011512229219079018,
+          0.002283401321619749,
+          0.0055889892391860485,
+          0.0028663973789662123,
+          -0.016494901850819588,
+          -0.007645022124052048,
+          -0.03488648682832718,
+          0.007131985388696194,
+          -0.0158652663230896,
+          0.0071436455473303795,
+          0.010369556955993176,
+          -0.03466883301734924,
+          -0.008099758997559547,
+          0.013214577920734882,
+          0.03211919590830803,
+          0.006758867762982845,
+          0.0075439694337546825,
+          -0.0043413774110376835,
+          -0.0014506885781884193,
+          -0.014559355564415455,
+          0.014707047492265701,
+          -0.0328032448887825,
+          0.02229377068579197,
+          0.01441943645477295,
+          0.004780567716807127,
+          -0.0018140894826501608,
+          -0.010276277549564838,
+          0.00558121595531702,
+          -0.0036301221698522568,
+          0.0033774906769394875,
+          -0.03448227420449257,
+          0.0192932840436697,
+          -0.008760487660765648,
+          0.022931179031729698,
+          0.02361522801220417,
+          0.02611822448670864,
+          -0.006335224024951458,
+          -0.0013700408162549138,
+          0.005464616697281599,
+          0.006486803293228149,
+          0.01694575324654579,
+          -0.009436762891709805,
+          -0.0016382189933210611,
+          -0.02073911391198635,
+          0.012701541185379028,
+          -0.023630773648619652,
+          -0.01063384860754013,
+          0.00269149849191308,
+          -0.013603242114186287,
+          -0.025371989235281944,
+          -0.010144132189452648,
+          -0.004100405611097813,
+          -0.0030587860383093357,
+          -0.0016440489562228322,
+          0.008099758997559547,
+          -0.01742769591510296,
+          -0.02465684711933136,
+          -0.014590448699891567,
+          -0.010742674581706524,
+          0.022651340812444687,
+          0.01075822114944458,
+          -0.016572635620832443,
+          -0.0039993529208004475,
+          -0.005682268645614386,
+          0.0071436455473303795,
+          0.007959839887917042,
+          -0.01416291855275631,
+          0.05506592243909836,
+          0.0006393523653969169,
+          0.0027187049854546785,
+          0.0018743324326351285,
+          0.008216357789933681,
+          0.0012913363752886653,
+          0.003585425904020667,
+          -0.002267854753881693,
+          0.004477410111576319,
+          -0.034575555473566055,
+          -0.018173931166529655,
+          0.0033153044059872627,
+          -0.011589962057769299,
+          0.00945230945944786,
+          -0.01915336400270462,
+          0.011589962057769299,
+          -0.01719449833035469,
+          -0.021718546748161316,
+          0.006109798792749643,
+          -0.0027886645402759314,
+          -0.002106559230014682,
+          -0.014652634970843792,
+          -0.017132312059402466,
+          0.007567289285361767,
+          0.02453247457742691,
+          -0.0024738467764109373,
+          0.01547660306096077,
+          -0.02022607810795307,
+          -0.005254738032817841,
+          -0.012810367159545422,
+          0.01089036650955677,
+          -0.010462836362421513,
+          -0.009017006494104862,
+          -0.009017006494104862,
+          -0.02164081484079361,
+          0.017816361039876938,
+          -0.002372794086113572,
+          0.00800647959113121,
+          -0.0206147413700819,
+          0.011224618181586266,
+          -0.010618302039802074,
+          0.016992392018437386,
+          0.000213036488275975,
+          -0.01915336400270462,
+          -0.014116278849542141,
+          0.01680583320558071,
+          -0.023972798138856888,
+          -0.0012835630914196372,
+          0.027346402406692505,
+          0.005934900138527155,
+          0.025807293131947517,
+          -0.006832714192569256,
+          -0.024983325973153114,
+          -0.01311352476477623,
+          0.016494901850819588,
+          -0.00957668200135231,
+          -0.00015947372594382614,
+          -0.00701927300542593,
+          0.007334090769290924,
+          -0.01482364721596241,
+          0.028683407232165337,
+          -0.01822056993842125,
+          0.007998705841600895,
+          0.0015604861546307802,
+          -0.008247450925409794,
+          0.014069639146327972,
+          -0.0048544141463935375,
+          -0.007493442855775356,
+          -0.0077072083950042725,
+          -0.004489069804549217,
+          -0.010664941743016243,
+          -0.0034571667201817036,
+          -0.029973771423101425,
+          0.0005674495478160679,
+          -0.0005266398075036705,
+          -0.012382837012410164,
+          -0.011753200553357601,
+          -0.004469636827707291,
+          -0.004671741742640734,
+          0.0019238870590925217,
+          0.011006966233253479,
+          0.00012704456457868218,
+          -0.01161328237503767,
+          -0.008379597216844559,
+          0.011776520870625973,
+          -0.00718639837577939,
+          0.005690041929483414,
+          -0.02557409554719925,
+          0.01194753311574459,
+          -0.013758707791566849,
+          0.0057600014843046665,
+          -0.009055872447788715,
+          0.019977333024144173,
+          0.03120194934308529,
+          0.0044618635438382626,
+          0.004166478756815195,
+          -0.007442916743457317,
+          0.0029791099950671196,
+          0.002434980357065797,
+          0.016215063631534576,
+          -0.008091985248029232,
+          0.017225591465830803,
+          -0.012748180888593197,
+          0.007769394665956497,
+          -0.012040812522172928,
+          0.01717895083129406,
+          0.006245831493288279,
+          0.0037953045684844255,
+          -0.009584455750882626,
+          -0.0007778139552101493,
+          -0.02897879108786583,
+          0.010672714561223984,
+          -0.01770753413438797,
+          -0.0009969233069568872,
+          0.016215063631534576,
+          0.009405669756233692,
+          -0.013657654635608196,
+          0.012794820591807365,
+          0.03134186938405037,
+          0.024827860295772552,
+          -0.004115952178835869,
+          -0.0012709314469248056,
+          0.01640162244439125,
+          -0.0039993529208004475,
+          0.004982673097401857,
+          -0.009211338125169277,
+          -0.03373603895306587,
+          -0.019184457138180733,
+          -0.018780246376991272,
+          0.0040071262046694756,
+          0.010058625601232052,
+          -0.00020538465469144285,
+          0.0110924718901515,
+          0.010781540535390377,
+          0.019650854170322418,
+          -0.019308829680085182,
+          -0.0034299602266401052,
+          -0.008519516326487064,
+          -0.005095385946333408,
+          0.028154823929071426,
+          0.008542835712432861,
+          0.021438708528876305,
+          -0.011038059368729591,
+          0.006179758347570896,
+          -0.020925672724843025,
+          -0.01371206808835268,
+          -0.01252275612205267,
+          0.021423162892460823,
+          0.0027944946195930243,
+          -0.0013661541743203998,
+          0.02572956122457981,
+          0.005359677132219076,
+          -0.015181218273937702,
+          0.006486803293228149,
+          -0.006191418506205082,
+          0.010408422909677029,
+          0.006261378061026335,
+          0.008053119294345379,
+          -0.025807293131947517,
+          -0.02507660537958145,
+          -0.01585749350488186,
+          0.010369556955993176,
+          0.013020245358347893,
+          0.01101473905146122,
+          -0.014139598235487938,
+          -0.018593687564134598,
+          0.0056395153515040874,
+          0.01331563014537096,
+          -0.008278544060885906,
+          -0.010968099348247051,
+          -0.0036126323975622654,
+          0.00970105454325676,
+          0.020490368828177452,
+          5.5475720728281885e-05,
+          0.007501216139644384,
+          0.01075822114944458,
+          0.005658948794007301,
+          -0.0043413774110376835,
+          0.008123078383505344,
+          -0.010983645915985107,
+          -0.01694575324654579,
+          0.01548437587916851,
+          -0.010385103523731232,
+          0.009156924672424793,
+          0.010268504731357098,
+          -0.010237411595880985,
+          -0.001987044932320714,
+          -0.007046479266136885,
+          0.01280259434133768,
+          0.025760654360055923,
+          -0.004353037569671869,
+          -0.003435790305957198,
+          -0.017210043966770172,
+          0.002506883116438985,
+          0.0023008910939097404,
+          0.018453769385814667,
+          0.0058183008804917336,
+          0.008822673931717873,
+          -0.010719354264438152,
+          -0.0022697981912642717,
+          0.025543002411723137,
+          0.010618302039802074,
+          -0.006704454775899649,
+          -0.00024813771597109735
+        ],
+        "title": "Capacity Factor",
+        "keyphrases": [
+          "capacity factors",
+          "downstream task fine-tuning",
+          "expert choice",
+          "gating computational footprint",
+          "fine-tuning performance"
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "d3110cfa-448f-46ee-b3ea-c6734cf305de",
+      "properties": {
+        "page_content": "C Capped Expert Choice\n\nAs described in Section 4.5, the maximum number of experts each token is assigned can be capped by an entropy-regularized linear programming. Figure 1 compares the validation perplexity when training the 100M/64E models using the base expert choice method (EC-BASE), expert choice capped by two experts per token (EC-CAP2), expert choice capped by three experts per token (EC-CAP3), and GShard top-2 gating.\n\nAs shown in the figure, restricting the number of experts to 2 degrades the perplexity compared to the base expert choice method. This suggests that a more flexible allocation of experts (e.g. more than 2 experts for a token) can enhance model expressiveness. On the other hand, our EC-CAP2 and EC-CAP3 methods still outperform the top-2 gating method by a clear margin. We believe this confirms the effectiveness of a load balanced training, provided by our method. Finally, EC-CAP3 obtains comparable perplexity to EC-BASE. As indicated by Figure 3, only a little fraction of tokens use more than 3 experts therefore we see little or no difference between EC-BASE and EC-CAP3 variants. We present the fine-tuning results of these methods in Table 2.\n\n",
+        "title": "C Capped Expert Choice",
+        "embedding": [
+          -0.0025292420759797096,
+          -0.00710443127900362,
+          -0.03731034696102142,
+          -0.015119067393243313,
+          3.892158201779239e-05,
+          0.01479687076061964,
+          -0.02490578405559063,
+          0.06395599246025085,
+          -0.031752459704875946,
+          0.013524195179343224,
+          -0.011389643885195255,
+          -0.03146248310804367,
+          -0.014837145805358887,
+          0.002412445843219757,
+          0.01028612069785595,
+          0.020378924906253815,
+          0.0013713486259803176,
+          0.009432300925254822,
+          0.005433037411421537,
+          -0.013242273591458797,
+          0.0022654435597360134,
+          -0.03563492372632027,
+          -0.02867548167705536,
+          0.004981962498277426,
+          0.034249480813741684,
+          0.023761987686157227,
+          0.00903761014342308,
+          -0.013854446820914745,
+          -0.04288434609770775,
+          0.04004901647567749,
+          0.017833571881055832,
+          0.01280730776488781,
+          -0.019879519939422607,
+          -0.036955930292606354,
+          -0.03078586980700493,
+          -0.02911044843494892,
+          0.03830915689468384,
+          0.022859837859869003,
+          -0.0014800898497924209,
+          0.012509276159107685,
+          0.033411771059036255,
+          0.012307903729379177,
+          -0.02360088936984539,
+          -0.013516140170395374,
+          0.01215486042201519,
+          0.0035884627141058445,
+          -0.03898577019572258,
+          -0.011365478858351707,
+          0.010390834882855415,
+          -0.04159555956721306,
+          0.01303284615278244,
+          -0.039533503353595734,
+          0.013500030152499676,
+          0.010060583241283894,
+          -0.040693409740924835,
+          0.009480630047619343,
+          -0.006468093488365412,
+          0.034346140921115875,
+          0.08435102552175522,
+          0.03172023966908455,
+          0.01726972870528698,
+          -0.031301382929086685,
+          -0.0481683649122715,
+          -0.01268648449331522,
+          -0.008747633546590805,
+          -0.016440073028206825,
+          -0.0063593522645533085,
+          0.0023298829328268766,
+          0.02923932671546936,
+          -0.018606843426823616,
+          -0.054096780717372894,
+          0.03360508754849434,
+          -0.028256626799702644,
+          0.00737427081912756,
+          -0.0055820532143116,
+          -0.01622259058058262,
+          0.037632543593645096,
+          0.03502275049686432,
+          0.01321810856461525,
+          0.03734256699681282,
+          -0.004065716173499823,
+          0.03325067088007927,
+          -0.0012092434335500002,
+          0.0340561643242836,
+          0.04088672623038292,
+          -0.02535685896873474,
+          -0.0193801149725914,
+          -0.057254306972026825,
+          -0.028691591694951057,
+          -0.021426063030958176,
+          0.0004978941869921982,
+          0.008199899457395077,
+          -0.0026259010192006826,
+          -0.018510185182094574,
+          -0.03308957442641258,
+          0.04613852873444557,
+          -0.012895911931991577,
+          0.007221227511763573,
+          -0.012646210379898548,
+          -0.01455522421747446,
+          -0.02828884683549404,
+          0.007462874986231327,
+          0.0039448924362659454,
+          -0.02221544459462166,
+          0.03605378046631813,
+          -0.01749526709318161,
+          -0.01784968189895153,
+          0.010181407444179058,
+          0.014579388312995434,
+          0.09305033087730408,
+          0.009891430847346783,
+          -0.0006464066100306809,
+          -0.007772989105433226,
+          -0.010946623980998993,
+          -0.05528890714049339,
+          -0.023649217560887337,
+          -0.04394759237766266,
+          -0.03685927018523216,
+          0.021184414625167847,
+          -0.01915457844734192,
+          0.0004923564265482128,
+          0.0317363478243351,
+          0.007176925428211689,
+          -0.0029420561622828245,
+          -0.04723399877548218,
+          0.026967842131853104,
+          0.024809125810861588,
+          0.051615867763757706,
+          0.00537665281444788,
+          -0.01221929956227541,
+          -0.006584889721125364,
+          0.03650485724210739,
+          -0.018751831725239754,
+          -0.02867548167705536,
+          0.017817461863160133,
+          0.0152721107006073,
+          0.05358126759529114,
+          0.03866357356309891,
+          0.0025131322909146547,
+          -0.007893812842667103,
+          0.0069594429805874825,
+          -0.0071004037745296955,
+          0.004671848379075527,
+          0.009134269319474697,
+          -0.046686261892318726,
+          0.019235126674175262,
+          9.407883771928027e-05,
+          0.05799535661935806,
+          -0.026693973690271378,
+          0.018284646794199944,
+          0.005928414408117533,
+          0.036827053874731064,
+          -0.028385505080223083,
+          0.046815142035484314,
+          -0.018010780215263367,
+          -0.021748259663581848,
+          -0.013781952671706676,
+          -0.0015817830571904778,
+          -0.009005390107631683,
+          -0.03573158383369446,
+          0.01416053343564272,
+          0.03134971112012863,
+          -0.0021587160881608725,
+          0.016101766377687454,
+          0.0020167483016848564,
+          0.04194997623562813,
+          0.013201998546719551,
+          0.01603732816874981,
+          -0.07481401413679123,
+          0.029062118381261826,
+          0.029223216697573662,
+          -0.029690401628613472,
+          0.04455976560711861,
+          0.014982134103775024,
+          0.0014508907916024327,
+          0.02337535098195076,
+          0.003131346544250846,
+          0.0004191070911474526,
+          -0.011011063121259212,
+          0.0411444827914238,
+          -0.01603732816874981,
+          -0.015376824885606766,
+          0.024712465703487396,
+          -0.011736005544662476,
+          0.033895064145326614,
+          -0.0009348730673082173,
+          0.029915938153862953,
+          0.02841772511601448,
+          -0.013137559406459332,
+          0.007458847481757402,
+          0.013959160074591637,
+          0.041885536164045334,
+          -0.0005522647988982499,
+          0.005771343596279621,
+          0.032010216265916824,
+          -0.01308923028409481,
+          0.0281760785728693,
+          -0.00663724634796381,
+          0.018026890233159065,
+          0.019927849993109703,
+          -0.009891430847346783,
+          -0.010334450751543045,
+          -0.007189007941633463,
+          -0.02864326350390911,
+          0.015594307333230972,
+          0.04852278158068657,
+          0.02666175551712513,
+          0.026275118812918663,
+          -0.005364570766687393,
+          -0.021184414625167847,
+          0.023536449298262596,
+          -0.03252572938799858,
+          0.002066084649413824,
+          -0.02712894044816494,
+          -0.04733065515756607,
+          0.024003634229302406,
+          -0.0006433859816752374,
+          -0.009464520029723644,
+          0.0012585797812789679,
+          0.007760906592011452,
+          0.03172023966908455,
+          0.0423043929040432,
+          0.02500244230031967,
+          -0.020733339712023735,
+          -0.008119350299239159,
+          -0.010994953103363514,
+          -0.01520767156034708,
+          0.006814454682171345,
+          0.02335924096405506,
+          -0.01900959014892578,
+          0.006057293154299259,
+          0.009980034083127975,
+          0.0008528137113898993,
+          -0.0016804557526484132,
+          0.016850873827934265,
+          -0.010866074822843075,
+          0.025163540616631508,
+          -0.02242487110197544,
+          0.016528677195310593,
+          0.012791198678314686,
+          -0.020701121538877487,
+          -0.0028131776489317417,
+          -0.03605378046631813,
+          0.008115322329103947,
+          -0.004542969632893801,
+          -0.008079075254499912,
+          -0.02326258271932602,
+          0.0007093356107361615,
+          -0.0043335421942174435,
+          -0.012042091228067875,
+          -0.027547795325517654,
+          -0.02453525923192501,
+          0.0004686951288022101,
+          0.009392025880515575,
+          -0.016045382246375084,
+          0.026758413761854172,
+          0.004265075083822012,
+          -0.05799535661935806,
+          0.012920076958835125,
+          -0.024245280772447586,
+          0.06907891482114792,
+          0.008812072686851025,
+          -0.0105760982260108,
+          0.015489593148231506,
+          -0.023939194157719612,
+          -0.011510467156767845,
+          0.01462771836668253,
+          -0.00922287255525589,
+          -0.003711300203576684,
+          -0.0029380286578089,
+          0.02770889364182949,
+          -0.01515128742903471,
+          -0.010382779873907566,
+          -0.045848552137613297,
+          0.05499893054366112,
+          -0.029529303312301636,
+          0.0008810058934614062,
+          0.007825345732271671,
+          -0.007084294222295284,
+          0.018703503534197807,
+          0.01973453164100647,
+          0.02186102792620659,
+          0.01009280327707529,
+          0.01975064165890217,
+          0.015707075595855713,
+          0.0022775260731577873,
+          -0.0622805692255497,
+          -0.002076153177767992,
+          -0.03217131271958351,
+          -0.03159135952591896,
+          -0.02806330844759941,
+          -0.03067309968173504,
+          -0.010076693259179592,
+          0.018155768513679504,
+          0.01971842162311077,
+          -0.02537296898663044,
+          -0.015803735703229904,
+          0.008852346800267696,
+          0.004800726659595966,
+          -0.0041885534301400185,
+          -0.023230362683534622,
+          0.0024990360252559185,
+          -0.03511941060423851,
+          -0.028691591694951057,
+          -0.06714573502540588,
+          0.004402008838951588,
+          -0.008115322329103947,
+          0.010962733998894691,
+          -0.044914182275533676,
+          0.03302513435482979,
+          0.03373396769165993,
+          -0.010463329032063484,
+          -0.0003755602228920907,
+          -0.005481366999447346,
+          -0.006588917225599289,
+          0.008054911158978939,
+          0.020282264798879623,
+          0.030463673174381256,
+          -0.028127748519182205,
+          0.005465256981551647,
+          0.04649294540286064,
+          0.019815079867839813,
+          -0.04839390516281128,
+          -0.026355668902397156,
+          0.014426345005631447,
+          0.021442173048853874,
+          0.000837207306176424,
+          0.015699021518230438,
+          -0.021893247961997986,
+          -0.010986898094415665,
+          -0.02608180046081543,
+          -0.04185331612825394,
+          -0.056803230196237564,
+          -0.010849964804947376,
+          -0.03550604730844498,
+          -0.02126496471464634,
+          -0.009762551635503769,
+          0.015425154007971287,
+          -0.019782861694693565,
+          -0.017430827021598816,
+          -0.021989906206727028,
+          0.03956572338938713,
+          -0.045171938836574554,
+          0.02758001536130905,
+          0.02456747740507126,
+          -0.0045671346597373486,
+          -0.01797856017947197,
+          -0.010495549067854881,
+          -0.007740769535303116,
+          -0.002712491201236844,
+          0.0010280079441145062,
+          0.002889699302613735,
+          0.034120600670576096,
+          -5.767442053183913e-05,
+          0.014442455023527145,
+          -0.035667143762111664,
+          -0.010833854787051678,
+          0.01868739351630211,
+          0.026242898777127266,
+          0.005098758731037378,
+          -0.004414091352373362,
+          -0.045880772173404694,
+          -0.045848552137613297,
+          0.03133360296487808,
+          -0.038921330124139786,
+          0.01639174297451973,
+          0.005868002772331238,
+          0.009287312626838684,
+          -0.04030677303671837,
+          -0.002058029640465975,
+          -0.02184491790831089,
+          -0.01926734670996666,
+          0.04101560637354851,
+          0.013338932767510414,
+          0.03102751635015011,
+          -0.0027165187057107687,
+          0.020266154780983925,
+          -0.039307963103055954,
+          -0.049843788146972656,
+          0.019927849993109703,
+          -0.05061705783009529,
+          0.02277928777039051,
+          0.030334794893860817,
+          0.01315366942435503,
+          -0.023987524211406708,
+          0.029690401628613472,
+          -0.00795422401279211,
+          0.016182316467165947,
+          0.0012464975006878376,
+          0.028401615098118782,
+          -0.0399523563683033,
+          -0.03308957442641258,
+          -0.009053720161318779,
+          0.020169496536254883,
+          0.0004284205788280815,
+          0.0036609568633139133,
+          -0.021329404786229134,
+          -0.002535283099859953,
+          0.008352942764759064,
+          -0.018848491832613945,
+          -0.041885536164045334,
+          0.04146667942404747,
+          -0.014434400014579296,
+          0.007233310025185347,
+          0.0005955599481239915,
+          -0.0030608661472797394,
+          0.0036347785498946905,
+          -0.007676329929381609,
+          0.009947814978659153,
+          -0.005002099554985762,
+          -0.013121449388563633,
+          -0.009762551635503769,
+          -0.003427364630624652,
+          -0.008131432346999645,
+          0.0062143635004758835,
+          0.024003634229302406,
+          -0.011083557270467281,
+          0.019428445026278496,
+          -0.010294175706803799,
+          -0.010197517462074757,
+          -0.01303284615278244,
+          0.013073120266199112,
+          0.04085450619459152,
+          -0.02103942632675171,
+          0.0340561643242836,
+          -0.0007098389905877411,
+          -0.02042725309729576,
+          0.029674291610717773,
+          0.012106531299650669,
+          0.004760452080518007,
+          0.037986960262060165,
+          -0.002492995001375675,
+          0.00475642504170537,
+          -0.03534494712948799,
+          -0.010656647384166718,
+          -0.01865517348051071,
+          -0.05429009720683098,
+          0.02993204817175865,
+          -0.010721086524426937,
+          0.00354013335891068,
+          0.01216291543096304,
+          -0.0006494271801784635,
+          0.012396507896482944,
+          -0.02479301579296589,
+          -0.03724590688943863,
+          -0.018864601850509644,
+          -0.006721823010593653,
+          -0.016738105565309525,
+          -0.020620571449398994,
+          0.005002099554985762,
+          -0.06514811515808105,
+          0.0015445291064679623,
+          -0.0563521571457386,
+          0.004945715423673391,
+          0.03946906328201294,
+          -0.010841909795999527,
+          -0.001440822146832943,
+          -0.01950899325311184,
+          0.009480630047619343,
+          -0.048103928565979004,
+          -0.045171938836574554,
+          0.016142040491104126,
+          0.033282890915870667,
+          -0.010890239849686623,
+          -0.00915843341499567,
+          0.014192752540111542,
+          -0.037632543593645096,
+          -0.0017086479347199202,
+          0.01374973263591528,
+          0.012106531299650669,
+          -0.01480492576956749,
+          0.01056804321706295,
+          -0.022038236260414124,
+          -0.021409953013062477,
+          -0.008054911158978939,
+          0.029883719980716705,
+          0.01416053343564272,
+          -0.011945432983338833,
+          0.004009332042187452,
+          0.010350560769438744,
+          0.03325067088007927,
+          -0.0076521653681993484,
+          -0.005300131160765886,
+          0.010849964804947376,
+          0.016142040491104126,
+          -0.0022654435597360134,
+          0.012340123765170574,
+          -0.007285666652023792,
+          -0.015223781578242779,
+          -0.013717513531446457,
+          -0.003880453296005726,
+          -0.014974079094827175,
+          0.0018657187465578318,
+          -0.003938851412385702,
+          0.015014354139566422,
+          -0.014982134103775024,
+          -0.020523913204669952,
+          -0.02596903219819069,
+          0.02442248910665512,
+          0.02994815818965435,
+          -0.01227568369358778,
+          -0.034571677446365356,
+          -0.02100720815360546,
+          -3.951940743718296e-05,
+          0.022038236260414124,
+          -0.02455136738717556,
+          0.029851499944925308,
+          0.033282890915870667,
+          -0.05554666370153427,
+          -0.002996426774188876,
+          0.004804754164069891,
+          0.006383516825735569,
+          -0.00431340467184782,
+          -0.03615044057369232,
+          -0.0028333149384707212,
+          0.03934018313884735,
+          -0.0007793126278556883,
+          0.04887719824910164,
+          0.0021526748314499855,
+          -0.017108630388975143,
+          -0.032686829566955566,
+          -0.01668977551162243,
+          -0.03863135352730751,
+          0.04900607839226723,
+          -0.020298374816775322,
+          0.034120600670576096,
+          0.014998244121670723,
+          -0.026581205427646637,
+          0.060798466205596924,
+          -0.0376003235578537,
+          -0.006427818909287453,
+          -0.00015921035083010793,
+          -0.007039992138743401,
+          -0.01772080361843109,
+          -0.01228373870253563,
+          0.0002982834121212363,
+          -0.00780923618003726,
+          0.004357706755399704,
+          -0.004990017041563988,
+          -0.0010285114403814077,
+          0.004426173400133848,
+          0.014941859990358353,
+          -0.004418118391185999,
+          0.028852690011262894,
+          0.013822226785123348,
+          0.008127405308187008,
+          0.020523913204669952,
+          -0.02572738565504551,
+          0.016625335440039635,
+          0.027998870238661766,
+          -0.00560621777549386,
+          -0.010390834882855415,
+          -0.008119350299239159,
+          -0.0011427904246374965,
+          0.006991662550717592,
+          -0.028820471838116646,
+          0.015304330736398697,
+          0.014579388312995434,
+          -0.0030205915682017803,
+          -0.020991098135709763,
+          -0.02360088936984539,
+          0.018864601850509644,
+          -0.023165924474596977,
+          0.027773331850767136,
+          -0.021071646362543106,
+          0.00044855783926323056,
+          0.026710083708167076,
+          0.03573158383369446,
+          -0.011083557270467281,
+          0.015006299130618572,
+          -0.00043949607061222196,
+          -0.005058484151959419,
+          -0.020491693168878555,
+          0.0045751892030239105,
+          -0.0008291523554362357,
+          0.009593399241566658,
+          0.011913212947547436,
+          0.005022237077355385,
+          -0.018606843426823616,
+          0.01515128742903471,
+          0.01609371230006218,
+          -0.030689209699630737,
+          0.00428924011066556,
+          0.010664702393114567,
+          0.014176643453538418,
+          0.005070566199719906,
+          0.003234046744182706,
+          -0.023101484403014183,
+          -0.004462420474737883,
+          -0.016738105565309525,
+          -0.01950899325311184,
+          0.044463109225034714,
+          0.0016653528437018394,
+          -0.034088384360075,
+          0.021538831293582916,
+          -0.007748824078589678,
+          0.005171252880245447,
+          0.0281760785728693,
+          -0.009488685056567192,
+          -0.0505204014480114,
+          -0.013185889460146427,
+          -0.006077430211007595,
+          0.016005108132958412,
+          -0.01926734670996666,
+          -0.009247037582099438,
+          0.030479783192276955,
+          -0.004977934993803501,
+          0.0051269507966935635,
+          -0.008079075254499912,
+          0.024841345846652985,
+          0.026581205427646637,
+          0.00314342905767262,
+          0.007970334030687809,
+          0.018945150077342987,
+          -0.006592944730073214,
+          0.009649783372879028,
+          -0.009174543432891369,
+          -0.0035300645977258682,
+          -0.0002856976061593741,
+          -0.005348460748791695,
+          0.06611470878124237,
+          0.0017881902167573571,
+          0.030705319717526436,
+          -0.0016039340989664197,
+          0.02758001536130905,
+          0.015497648157179356,
+          -0.010986898094415665,
+          -0.02266651950776577,
+          0.007644110359251499,
+          -0.012251519598066807,
+          -0.05416122078895569,
+          -0.026855072006583214,
+          -0.022875946015119553,
+          -0.016085656359791756,
+          -0.029883719980716705,
+          -0.006685575935989618,
+          2.6902143872575834e-05,
+          0.009859210811555386,
+          0.004510750062763691,
+          -0.01786579191684723,
+          -0.032010216265916824,
+          0.00969811249524355,
+          -0.0011850787559524179,
+          0.025340748950839043,
+          -0.005348460748791695,
+          0.010487494058907032,
+          0.007221227511763573,
+          -0.005783426109701395,
+          0.019557323306798935,
+          -0.027451135218143463,
+          0.014007490128278732,
+          -1.3608394056063844e-06,
+          0.011333259753882885,
+          0.035860463976860046,
+          -0.014394125901162624,
+          -0.01726972870528698,
+          -0.005384707823395729,
+          0.008107267320156097,
+          0.023165924474596977,
+          -0.009263147599995136,
+          -0.014877419918775558,
+          0.0021103867329657078,
+          0.005175280384719372,
+          -0.032477401196956635,
+          0.017591925337910652,
+          -0.014200807549059391,
+          -0.05764094367623329,
+          -0.004764479584991932,
+          -0.021877137944102287,
+          0.019219016656279564,
+          -0.025195760652422905,
+          -0.006890976335853338,
+          -0.018123548477888107,
+          0.0015183506766334176,
+          0.014168588444590569,
+          -0.023633107542991638,
+          0.01103522814810276,
+          0.05899416655302048,
+          0.0044221458956599236,
+          0.01762414537370205,
+          0.009464520029723644,
+          -0.005880084820091724,
+          -0.008071020245552063,
+          0.003036701353266835,
+          0.021812697872519493,
+          -0.0002548624179325998,
+          -0.031301382929086685,
+          0.039050206542015076,
+          -0.0004669331246986985,
+          0.004591299220919609,
+          -0.010721086524426937,
+          -0.0007571616442874074,
+          -0.010463329032063484,
+          0.009843100793659687,
+          -0.02385864593088627,
+          0.008892621845006943,
+          -0.019927849993109703,
+          0.03731034696102142,
+          0.0008155597024597228,
+          -0.003642833326011896,
+          -0.025904593989253044,
+          0.0013814171543344855,
+          -0.03199410438537598,
+          0.016182316467165947,
+          0.02911044843494892,
+          -0.012324013747274876,
+          -0.0014539114199578762,
+          0.012267629615962505,
+          -0.02737058699131012,
+          -0.001367321121506393,
+          -0.009843100793659687,
+          0.01402360014617443,
+          0.015111012384295464,
+          0.0030165640637278557,
+          0.034926094114780426,
+          0.01267842948436737,
+          0.009931704960763454,
+          0.013016736134886742,
+          0.0001655032392591238,
+          0.015747351571917534,
+          -0.0038140001706779003,
+          0.005578025709837675,
+          -0.010874129831790924,
+          -0.013604744337499142,
+          -0.021168306469917297,
+          0.01592455804347992,
+          0.021893247961997986,
+          0.011663510464131832,
+          0.03750366345047951,
+          0.014780761674046516,
+          0.026468437165021896,
+          0.0022714848164469004,
+          -0.03582824394106865,
+          0.001979494234547019,
+          0.007426627911627293,
+          -0.025082992389798164,
+          -0.004889330826699734,
+          0.0193801149725914,
+          0.011204380542039871,
+          -0.02501855231821537,
+          0.005851892754435539,
+          0.005396790336817503,
+          -0.002138578798621893,
+          -0.0091020492836833,
+          -0.03579602390527725,
+          0.0021989906672388315,
+          0.019992288202047348,
+          0.025791823863983154,
+          0.003517982317134738,
+          0.015787625685334206,
+          0.007865620777010918,
+          -0.009657838381826878,
+          0.03534494712948799,
+          0.006633218843489885,
+          -0.039275746792554855,
+          -0.015223781578242779,
+          -0.008650974370539188,
+          0.01797856017947197,
+          -0.00029677312704734504,
+          0.0031857171561568975,
+          -0.00969811249524355,
+          0.0047805896028876305,
+          -0.007023882120847702,
+          -0.009303421713411808,
+          0.03395950421690941,
+          0.007011800073087215,
+          0.011317149735987186,
+          -0.004273130092769861,
+          0.018413526937365532,
+          -0.014466620050370693,
+          -0.006540587637573481,
+          0.010366669856011868,
+          0.02314981445670128,
+          -0.025276310741901398,
+          0.015167397446930408,
+          -0.005718986503779888,
+          0.003246129024773836,
+          0.016174260526895523,
+          -0.00431340467184782,
+          -0.01562652736902237,
+          -0.02972262166440487,
+          0.0038683710154145956,
+          -0.014257192611694336,
+          -0.008401271887123585,
+          -0.020475583150982857,
+          -0.014853255823254585,
+          0.030350904911756516,
+          -0.0193801149725914,
+          0.0023802260402590036,
+          -0.0014559251721948385,
+          0.016238700598478317,
+          -0.02031448483467102,
+          -0.013483921065926552,
+          0.025066882371902466,
+          0.05261467769742012,
+          0.0019684189464896917,
+          0.02970651164650917,
+          0.07938919961452484,
+          -0.030689209699630737,
+          -0.013645019382238388,
+          0.019315676763653755,
+          -0.03028646484017372,
+          0.0034535429440438747,
+          0.03895355015993118,
+          -0.04501084238290787,
+          -0.06166839599609375,
+          0.013177834451198578,
+          -0.034603897482156754,
+          -0.000111384317278862,
+          0.010439164005219936,
+          0.008747633546590805,
+          -0.019347894936800003,
+          -0.00610562227666378,
+          0.028804361820220947,
+          -0.007414545398205519,
+          -0.003664984367787838,
+          -0.029319874942302704,
+          -0.011470193043351173,
+          -0.018123548477888107,
+          -0.005388735327869654,
+          -0.0063593522645533085,
+          0.0211360864341259,
+          -0.0006786262383684516,
+          0.003614641260355711,
+          -0.019573433324694633,
+          0.00046743653365410864,
+          0.013725567609071732,
+          0.018590733408927917,
+          0.049972668290138245,
+          -9.590377885615453e-05,
+          0.013983325101435184,
+          -0.01715696044266224,
+          -0.026355668902397156,
+          0.041273362934589386,
+          -0.006955415476113558,
+          0.040822289884090424,
+          -0.02725781872868538,
+          -0.0021748258732259274,
+          -0.006568779703229666,
+          0.025807933881878853,
+          -0.01913846842944622,
+          0.013934995979070663,
+          -0.010729141533374786,
+          -0.0117521146312356,
+          -0.020604461431503296,
+          -0.00766827492043376,
+          0.0020328580867499113,
+          -0.026387887075543404,
+          -0.005968688987195492,
+          -0.004108004737645388,
+          0.027209488674998283,
+          0.047846172004938126,
+          0.005416927393525839,
+          -0.004124114289879799,
+          -0.009077884256839752,
+          -0.015392934903502464,
+          -0.008667084388434887,
+          -0.01726972870528698,
+          -0.015819845721125603,
+          -0.01579567976295948,
+          -0.028014980256557465,
+          -0.019766751676797867,
+          -0.019831189885735512,
+          -0.007325941231101751,
+          0.0029380286578089,
+          -0.0007752851815894246,
+          0.0076521653681993484,
+          -0.04639628529548645,
+          -0.0002455489302519709,
+          -0.03065699152648449,
+          -0.01426524668931961,
+          0.017060300335288048,
+          -0.03859913349151611,
+          -0.00634324224665761,
+          0.0004344617482274771,
+          -0.0020217825658619404,
+          0.0015999067109078169,
+          -0.015119067393243313,
+          0.007797153666615486,
+          -0.02795054018497467,
+          -0.007865620777010918,
+          0.002593681216239929,
+          0.017817461863160133,
+          0.0013310740469023585,
+          0.021538831293582916,
+          -0.03101140633225441,
+          0.011760169640183449,
+          -0.039662379771471024,
+          -0.02619457058608532,
+          -0.042691025882959366,
+          0.027048390358686447,
+          -0.029770949855446815,
+          -0.01168767549097538,
+          0.008804017677903175,
+          -0.0007818297599442303,
+          0.00716887041926384,
+          -0.0152721107006073,
+          -0.0013622867409139872,
+          -0.0027265872340649366,
+          0.007039992138743401,
+          -0.005400817841291428,
+          -0.021748259663581848,
+          -0.017527485266327858,
+          -0.017285838723182678,
+          0.020749449729919434,
+          -0.035054970532655716,
+          0.02044336311519146,
+          -0.012549551203846931,
+          0.02253764122724533,
+          -0.020250046625733376,
+          -0.017898011952638626,
+          -0.006641273852437735,
+          0.040210116654634476,
+          -0.02033059485256672,
+          0.011309094727039337,
+          0.039275746792554855,
+          0.04240104928612709,
+          -0.028611043468117714,
+          0.02571127563714981,
+          -0.022376542910933495,
+          0.015489593148231506,
+          0.055256687104701996,
+          0.01772080361843109,
+          0.02983538992702961,
+          -0.004909468349069357,
+          -0.026919512078166008,
+          -0.008804017677903175,
+          -0.011011063121259212,
+          0.0034736802335828543,
+          0.003671025624498725,
+          -0.02901378832757473,
+          -0.002052995376288891,
+          0.012025982141494751,
+          -0.02397141419351101,
+          0.043850935995578766,
+          0.0013300671707838774,
+          0.01280730776488781,
+          0.005952578969299793,
+          -0.016979752108454704,
+          0.02207045629620552,
+          -0.010704976506531239,
+          0.0005127454060129821,
+          0.009665893390774727,
+          -0.0060170185752213,
+          -0.004683930426836014,
+          -0.03511941060423851,
+          0.01121243555098772,
+          0.024116402491927147,
+          -0.004635601304471493,
+          0.003783794352784753,
+          -0.01221124455332756,
+          -0.002809150144457817,
+          0.005316241178661585,
+          0.020588351413607597,
+          0.0016633390914648771,
+          0.03180078789591789,
+          -0.011647401377558708,
+          0.00777701660990715,
+          -0.011840718798339367,
+          0.023810315877199173,
+          0.008062966167926788,
+          -0.004031483083963394,
+          -0.023536449298262596,
+          -0.010793580673635006,
+          0.017946342006325722,
+          0.01760803535580635,
+          -0.030818087980151176,
+          -0.040596749633550644,
+          -0.021103866398334503,
+          0.016343414783477783,
+          0.0010471383575350046,
+          -0.04143446311354637,
+          -0.012420672923326492,
+          0.015135177411139011,
+          0.01168767549097538,
+          0.02948097325861454,
+          0.006536560133099556,
+          0.007442737463861704,
+          -0.00027009123004972935,
+          0.010551933199167252,
+          0.036472637206315994,
+          9.779164975043386e-05,
+          -0.009021500125527382,
+          -0.014998244121670723,
+          0.022956496104598045,
+          -0.029787059873342514,
+          0.0323324128985405,
+          0.011292984709143639,
+          -0.030721429735422134,
+          0.017704693600535393,
+          0.004957797471433878,
+          -0.018010780215263367,
+          0.03515163064002991,
+          -0.03653707727789879,
+          0.00525582954287529,
+          0.028691591694951057,
+          -0.008010609075427055,
+          -0.01913846842944622,
+          -0.002654093084856868,
+          0.006234501022845507,
+          -0.019702311605215073,
+          -0.0030326738487929106,
+          -0.001486130990087986,
+          0.015779569745063782,
+          -0.015280165709555149,
+          0.014297466725111008,
+          0.026403997093439102,
+          0.014659937471151352,
+          -0.023182032629847527,
+          -0.01714085042476654,
+          -0.020056728273630142,
+          -0.003956974949687719,
+          -0.007442737463861704,
+          0.008441546931862831,
+          -0.019460665062069893,
+          0.021200524643063545,
+          -0.02524409070611,
+          -0.05213138461112976,
+          -0.0007012806599959731,
+          0.005590108223259449,
+          -0.020620571449398994,
+          0.004824891686439514,
+          -0.046202968806028366,
+          0.014007490128278732,
+          0.012074311263859272,
+          0.03885689005255699,
+          -0.007772989105433226,
+          -0.0030105228070169687,
+          0.006488230545073748,
+          0.010777470655739307,
+          0.02853049337863922,
+          -0.032139092683792114,
+          -0.0010682825231924653,
+          0.019798971712589264,
+          -0.0019231099868193269,
+          -0.0015676870243623853,
+          -0.0054733119904994965,
+          -0.04417313262820244,
+          -0.033186230808496475,
+          0.03708481043577194,
+          0.010986898094415665,
+          -0.003620682517066598,
+          0.03065699152648449,
+          -0.023101484403014183,
+          0.009488685056567192,
+          -0.016142040491104126,
+          0.016593115404248238,
+          0.010954678989946842,
+          0.02912655845284462,
+          -0.02948097325861454,
+          -0.0013653073692694306,
+          0.03315401449799538,
+          0.010841909795999527,
+          -0.014120258390903473,
+          0.00024693337036296725,
+          0.018300756812095642,
+          0.020991098135709763,
+          -0.017237508669495583,
+          0.016270920634269714,
+          0.005686766933649778,
+          -0.004236883018165827,
+          -0.019798971712589264,
+          0.0013149641454219818,
+          0.0223282128572464,
+          -0.015135177411139011,
+          0.03006092645227909,
+          -0.009657838381826878,
+          -0.010004199109971523,
+          0.04613852873444557,
+          0.02606569230556488,
+          -0.013411426916718483,
+          4.716527837445028e-05,
+          0.006238528527319431,
+          0.017994670197367668,
+          0.007551478687673807,
+          -0.012348177842795849,
+          0.0006000908324494958,
+          -0.02856271341443062,
+          -0.0029923992697149515,
+          0.007458847481757402,
+          -0.008884566836059093,
+          0.009367860853672028,
+          0.021345512941479683,
+          0.005686766933649778,
+          -0.006101594772189856,
+          0.003542147111147642,
+          -0.010318340733647346,
+          0.019347894936800003,
+          0.03250962123274803,
+          -0.01865517348051071,
+          -0.018413526937365532,
+          0.012460947036743164,
+          -0.026017362251877785,
+          0.03785808011889458,
+          0.009915594942867756,
+          -0.0058196731843054295,
+          0.029738731682300568,
+          0.09479019045829773,
+          -0.012791198678314686,
+          0.0223282128572464,
+          -0.027435027062892914,
+          -0.007861592806875706,
+          -0.009762551635503769,
+          -0.010471384041011333,
+          0.01416053343564272,
+          0.042336612939834595,
+          -0.0006982600898481905,
+          0.028127748519182205,
+          0.010237791575491428,
+          0.0009162460919469595,
+          0.02372976765036583,
+          -0.013604744337499142,
+          0.009738387539982796,
+          -0.004172443877905607,
+          -0.0050745937041938305,
+          -0.009198708459734917,
+          0.004587271716445684,
+          -0.0017318058526143432,
+          0.01678643375635147,
+          -0.00408383971080184,
+          0.0025614616461098194,
+          -0.019428445026278496,
+          -0.019815079867839813,
+          0.006427818909287453,
+          -0.012203189544379711,
+          -0.018284646794199944,
+          -0.017656365409493446,
+          0.00240640458650887,
+          -0.0062143635004758835,
+          0.0012867719633504748,
+          -0.007704521995037794,
+          -0.024486929178237915,
+          0.01844574511051178,
+          -0.0038764257915318012,
+          0.0015425154706463218,
+          0.021635491400957108,
+          -0.017801353707909584,
+          -0.014821035787463188,
+          0.016866983845829964,
+          -0.010898293927311897,
+          0.022747067734599113,
+          -0.025985142216086388,
+          -0.0026661755982786417,
+          0.0012575729051604867,
+          -0.0069594429805874825,
+          -0.015223781578242779,
+          -0.015578197315335274,
+          -0.0008940951083786786,
+          0.026210680603981018,
+          0.003989194519817829,
+          -0.011405753903090954,
+          -0.01157490722835064,
+          0.01880016177892685,
+          0.021796587854623795,
+          -0.007680357433855534,
+          0.0020117140375077724,
+          0.005139033310115337,
+          0.0028051226399838924,
+          -0.009287312626838684,
+          0.02303704433143139,
+          0.0323324128985405,
+          -0.005553861148655415,
+          -0.000351898925146088,
+          0.01526405569165945,
+          -0.006125759799033403,
+          -0.005896194837987423,
+          0.01063248235732317,
+          0.001617023372091353,
+          0.007877702824771404,
+          0.008852346800267696,
+          -0.025695165619254112,
+          -0.013459756039083004,
+          0.02210267446935177,
+          -0.006351297255605459,
+          5.386721750255674e-05,
+          -0.005948551464825869,
+          -0.025775713846087456,
+          0.021087756380438805,
+          0.001424712361767888,
+          0.013798062689602375,
+          -0.008050883188843727,
+          -0.027692783623933792,
+          -0.0038784395437687635,
+          -0.018864601850509644,
+          -0.00020942768605891615,
+          0.01374973263591528,
+          -0.013290602713823318,
+          -0.010406944900751114,
+          0.0018183961510658264,
+          0.0030628798995167017,
+          -0.00862680934369564,
+          -0.013886665925383568,
+          0.005892167333513498,
+          0.0020640708971768618,
+          -0.014104149304330349,
+          -0.040338993072509766,
+          -0.008236146531999111,
+          -0.02207045629620552,
+          0.010737196542322636,
+          0.028578823432326317,
+          -0.012992571108043194,
+          -0.013540305197238922,
+          -0.005054456647485495,
+          0.02712894044816494,
+          0.0071527608670294285,
+          -0.006621136795729399,
+          0.008328777737915516,
+          0.006496285554021597,
+          0.024841345846652985,
+          -0.007434682920575142,
+          -0.0007128596189431846,
+          0.0016160164959728718,
+          0.005642464850097895,
+          -0.008973171003162861,
+          -0.016343414783477783,
+          0.0005160177242942154,
+          -0.012082366272807121,
+          -0.02630733884871006,
+          0.008812072686851025,
+          0.021909357979893684,
+          -0.006766125094145536,
+          0.001452904543839395,
+          0.010028364136815071,
+          0.016866983845829964,
+          -0.017994670197367668,
+          0.004301322158426046,
+          0.00243862415663898,
+          -0.007495094556361437,
+          -0.01996006816625595,
+          0.010423054918646812,
+          0.001549563487060368,
+          0.01913846842944622,
+          0.004414091352373362,
+          0.0016411880496889353,
+          -0.008352942764759064,
+          0.009440355934202671,
+          -0.008264338597655296,
+          -0.0015143232885748148,
+          0.017318058758974075,
+          0.0399201363325119,
+          0.017463047057390213,
+          0.0328962542116642,
+          -0.03708481043577194,
+          -0.026983952149748802,
+          0.011051337234675884,
+          0.014724376611411572,
+          0.007197062950581312,
+          -0.003234046744182706,
+          -0.015417098999023438,
+          0.030930858105421066,
+          0.008618754334747791,
+          0.005271939095109701,
+          -0.009287312626838684,
+          0.007861592806875706,
+          -0.007096376270055771,
+          0.05271133780479431,
+          0.00677820760756731,
+          -0.014643827453255653,
+          0.016182316467165947,
+          -0.016142040491104126,
+          -0.013701403513550758,
+          -0.005654547363519669,
+          -0.010439164005219936,
+          0.013314767740666866,
+          -0.018977370113134384,
+          0.008377106860280037,
+          0.014031654223799706,
+          0.010294175706803799,
+          0.0036508883349597454,
+          0.012299848720431328,
+          0.014015545137226582,
+          0.012541496194899082,
+          0.01657700724899769,
+          0.0003000454162247479,
+          -0.029674291610717773,
+          -0.004651710856705904,
+          -0.002055009128525853,
+          0.0091020492836833,
+          -0.0211199764162302,
+          -0.008707358501851559,
+          0.0003111209371127188,
+          -0.0018264510435983539,
+          0.002591667464002967,
+          -0.028917130082845688,
+          -0.011437973007559776,
+          -0.038341376930475235,
+          0.027306146919727325,
+          0.007950196973979473,
+          0.021893247961997986,
+          0.002742697251960635,
+          -0.028449945151805878,
+          0.010302230715751648,
+          0.02242487110197544,
+          0.0193962249904871,
+          -0.012775088660418987,
+          -0.013411426916718483,
+          -0.030350904911756516,
+          0.0030266328249126673,
+          0.010535823181271553,
+          0.0009686030098237097,
+          0.009561179205775261,
+          0.005984798539429903,
+          -0.0205722413957119,
+          0.009786716662347317,
+          -0.027869991958141327,
+          0.01725361868739128,
+          -0.04945715144276619,
+          0.0055739982053637505,
+          -0.01068081147968769,
+          -0.02092665806412697,
+          -0.005392762832343578,
+          -0.007648137863725424,
+          0.028208298608660698,
+          -0.005690794438123703,
+          -0.022457091137766838,
+          0.016609225422143936,
+          0.016029272228479385,
+          -0.016295084729790688,
+          0.009488685056567192,
+          0.01062442734837532,
+          0.003735464997589588,
+          -0.001605947851203382,
+          0.011123832315206528,
+          0.03215520456433296,
+          0.0035602706484496593,
+          0.01833297684788704,
+          0.010664702393114567,
+          0.009247037582099438,
+          0.006504340562969446,
+          0.0014307535020634532,
+          0.030834197998046875,
+          -0.005235692020505667,
+          0.0060532656498253345,
+          0.007164843380451202,
+          0.001605947851203382,
+          0.015223781578242779,
+          0.001339128939434886,
+          -0.0020117140375077724,
+          0.009730332531034946,
+          -0.010793580673635006,
+          0.010414999909698963,
+          -0.020024508237838745,
+          -0.00027915299870073795,
+          0.01589233987033367,
+          -0.011083557270467281,
+          -0.024116402491927147,
+          -0.00019784874166361988,
+          -0.0164642371237278,
+          0.009182598441839218,
+          0.0017267714720219374,
+          -0.011325204744935036,
+          -0.0051672253757715225,
+          -0.0016885106451809406,
+          -0.005880084820091724,
+          0.022875946015119553,
+          -0.01215486042201519,
+          -0.019702311605215073,
+          -0.018735721707344055,
+          -0.008852346800267696,
+          0.011977652087807655,
+          0.03162357956171036,
+          -0.015473484061658382,
+          0.00608145771548152,
+          -0.006762097589671612,
+          0.00014549182378686965,
+          0.027773331850767136,
+          -0.0032642525620758533,
+          -0.0074467649683356285,
+          0.013540305197238922,
+          0.01762414537370205,
+          0.005259856581687927,
+          -0.007132623344659805,
+          -0.011800444684922695,
+          -0.012823417782783508,
+          -0.003695190418511629,
+          0.012815362773835659,
+          -0.006037155631929636,
+          -0.017994670197367668,
+          0.04298100247979164,
+          -0.023053154349327087,
+          0.0038663572631776333,
+          -0.00921481754630804,
+          -0.004893358331173658,
+          -0.0025393106043338776,
+          0.010431109927594662,
+          -0.006480175536125898,
+          -0.04336763918399811,
+          -0.043399859219789505,
+          0.007664247881621122,
+          -0.006645301356911659,
+          -0.0004505715623963624,
+          0.047395095229148865,
+          0.010278066620230675,
+          -0.02946486324071884,
+          0.01726972870528698,
+          -0.025195760652422905,
+          -0.006274775601923466,
+          -0.0050745937041938305,
+          -0.022714847698807716,
+          0.009343696758151054,
+          0.0008794955792836845,
+          -0.011292984709143639,
+          -0.012960351072251797,
+          -0.022231554612517357,
+          -0.009053720161318779,
+          -0.001134735532104969,
+          0.012581770308315754,
+          0.004325487185269594,
+          -0.0036267235409468412,
+          -0.026758413761854172,
+          0.017285838723182678,
+          -0.01168767549097538,
+          0.0012213258305564523,
+          -0.000624255568254739,
+          0.00898122601211071,
+          0.019637873396277428,
+          0.016238700598478317,
+          -0.02360088936984539,
+          -0.017124740406870842,
+          -0.026516767218708992,
+          0.017060300335288048,
+          0.00505042914301157,
+          0.026371778920292854,
+          -0.012436782009899616,
+          0.003979125991463661,
+          -0.020846109837293625,
+          -0.013693348504602909,
+          -0.008642919361591339,
+          -0.014031654223799706,
+          0.005827728193253279,
+          -0.003018577815964818,
+          0.03421726077795029,
+          -0.0012364288559183478,
+          -0.007434682920575142,
+          0.004824891686439514,
+          0.008618754334747791,
+          0.025904593989253044,
+          -0.021796587854623795,
+          -0.017189180478453636,
+          -0.011292984709143639,
+          0.010978843085467815,
+          0.005650519859045744,
+          -0.016883093863725662,
+          -0.02548573724925518,
+          -0.016423963010311127,
+          -0.002593681216239929,
+          -0.024148622527718544,
+          0.006230473518371582,
+          0.02970651164650917,
+          0.014442455023527145,
+          0.002138578798621893,
+          -0.01596483401954174,
+          0.03007703647017479,
+          0.010938568972051144,
+          0.011800444684922695,
+          -0.008022691123187542,
+          0.017930231988430023,
+          -0.020266154780983925,
+          0.005296103656291962,
+          -0.024019744247198105,
+          -0.017301948741078377,
+          -0.013250328600406647,
+          0.03511941060423851,
+          -0.010551933199167252,
+          -0.018735721707344055,
+          0.004748370032757521,
+          -0.005187362432479858,
+          0.01678643375635147,
+          -0.01514323242008686,
+          -0.021103866398334503,
+          0.025195760652422905,
+          0.007535369135439396,
+          0.012074311263859272,
+          -0.03136582300066948,
+          0.0022211417090147734,
+          -0.019122358411550522,
+          0.0024205006193369627,
+          -0.012017927132546902,
+          0.0015173438005149364,
+          0.004639628808945417,
+          0.00766827492043376,
+          0.004305349662899971,
+          0.024841345846652985,
+          0.004365761764347553,
+          0.021168306469917297,
+          -0.016512567177414894,
+          -0.0013491975842043757,
+          0.03724590688943863,
+          0.008026719093322754,
+          -0.001129701267927885,
+          0.007712577003985643,
+          0.018864601850509644,
+          0.006721823010593653,
+          -0.009553124196827412,
+          -0.00437784381210804,
+          -0.0011729963589459658,
+          0.023278692737221718,
+          0.01315366942435503,
+          0.0064600384794175625,
+          -0.0015102957841008902,
+          0.008747633546590805,
+          0.006681548431515694,
+          0.001100502209737897,
+          -0.003042742609977722,
+          -0.013894720934331417,
+          -0.020395034924149513,
+          -0.025082992389798164,
+          0.025421299040317535,
+          0.0031353740487247705,
+          -0.006335187237709761,
+          0.009657838381826878,
+          0.006315050181001425,
+          -0.012066256254911423,
+          0.02878825180232525,
+          -0.005718986503779888,
+          -0.024873564019799232,
+          -0.005598163232207298,
+          -0.003679080633446574,
+          -0.011913212947547436,
+          0.030624771490693092,
+          -0.00027487383340485394,
+          0.0059606339782476425,
+          0.01267842948436737,
+          -0.005489421542733908,
+          -0.021232744678854942,
+          -0.02102331817150116,
+          -0.025936812162399292,
+          -0.007619945798069239,
+          0.015014354139566422,
+          -7.803194603184238e-05,
+          0.028852690011262894,
+          0.0065003130584955215,
+          -0.008497931063175201,
+          -0.0058760573156178,
+          -0.023182032629847527,
+          -0.004478530492633581,
+          0.008489876054227352,
+          -0.017898011952638626,
+          0.002058029640465975,
+          -0.015344604849815369,
+          0.011349369771778584,
+          0.0006912120152264833,
+          -0.01327449269592762,
+          -0.002196976915001869,
+          -0.0005774364108219743,
+          -0.01473243162035942,
+          -0.011003008112311363,
+          -0.005553861148655415,
+          0.01256566122174263,
+          -0.01975064165890217,
+          -0.006447955965995789,
+          -0.011719895526766777,
+          0.019879519939422607,
+          -0.010414999909698963,
+          0.009448410011827946,
+          -0.03357286751270294,
+          0.00329043110832572,
+          -0.0003179172636009753,
+          -0.01622259058058262,
+          0.004114045761525631,
+          -0.0016613253392279148,
+          0.02360088936984539,
+          0.01420886255800724,
+          0.014812980778515339,
+          0.004994044546037912,
+          0.017173070460557938,
+          -0.0015143232885748148,
+          -0.002843383466824889,
+          -0.004998072050511837,
+          -0.00487724831327796,
+          -0.010672756470739841,
+          0.006588917225599289,
+          -0.003246129024773836,
+          0.019637873396277428,
+          0.004945715423673391,
+          -0.012928131967782974,
+          0.0066050267778337,
+          0.01656089723110199,
+          0.013258383609354496,
+          -0.00032295158598572016,
+          -0.00684264674782753,
+          0.01562652736902237,
+          0.06869228184223175,
+          -0.0011226532515138388,
+          0.01809133030474186,
+          0.016246754676103592,
+          -0.03191355615854263,
+          -0.022730957716703415,
+          0.01156685221940279,
+          -0.01575540564954281,
+          0.03418504074215889,
+          0.01109161227941513,
+          0.011204380542039871,
+          -0.014966024085879326,
+          0.00651642307639122,
+          0.01913846842944622,
+          0.0170119721442461,
+          -0.005863975267857313,
+          -0.006041183136403561,
+          -0.019170688465237617,
+          0.023101484403014183,
+          0.010769415646791458,
+          -0.006443928461521864,
+          0.004212718456983566,
+          0.008203926496207714,
+          0.003908645361661911,
+          -0.00013315775140654296,
+          0.01633535884320736,
+          -0.018477965146303177,
+          0.001423705485649407,
+          -0.003340774215757847,
+          0.014925749972462654,
+          -0.006298940163105726,
+          -0.01962176337838173,
+          0.0076400828547775745,
+          -0.0019301580032333732,
+          0.011204380542039871,
+          0.013645019382238388,
+          -0.0025232008192688227,
+          0.00663724634796381,
+          -0.003495831275358796,
+          0.0004875738231930882,
+          0.00816767942160368,
+          -0.01668977551162243,
+          0.003558256896212697,
+          0.0003438440035097301,
+          -0.022022126242518425,
+          0.010914403945207596,
+          0.014837145805358887,
+          0.012082366272807121,
+          0.0018032931257039309,
+          -0.008127405308187008,
+          -0.022505421191453934,
+          -0.008949005976319313,
+          0.003348829224705696,
+          0.0002846907591447234,
+          -0.025340748950839043,
+          -0.001157893449999392,
+          -0.012960351072251797,
+          -0.01726972870528698,
+          0.0056666298769414425,
+          0.012726759538054466,
+          0.004663793370127678,
+          -0.004430200904607773,
+          -0.0016019203467294574,
+          -0.0048329466953873634,
+          -0.01509490329772234,
+          0.0019784874748438597,
+          0.012718704529106617,
+          -0.003324664430692792,
+          0.014224972575902939,
+          0.022151004523038864,
+          0.004055647645145655,
+          0.0039026043377816677,
+          0.0030447563622146845,
+          4.613481451087864e-06,
+          0.01581178978085518,
+          0.01925123669207096,
+          -0.0019513021688908339,
+          0.0013199985260143876,
+          -0.00045661276089958847,
+          0.006464065983891487,
+          0.006472120992839336,
+          0.001662332215346396,
+          -0.011462138034403324,
+          0.003439446911215782,
+          -0.01902570016682148,
+          -0.013201998546719551,
+          0.005622327793389559,
+          0.0047805896028876305,
+          0.02147439308464527,
+          -0.01891293004155159,
+          0.0022553750313818455,
+          0.006544615142047405,
+          0.0072051179595291615,
+          -0.0009097015135921538,
+          -0.008409326896071434,
+          0.018268538638949394,
+          0.006149924360215664,
+          0.0034172958694398403,
+          -0.004168416373431683,
+          -0.011333259753882885,
+          0.007950196973979473,
+          0.00033704767702147365,
+          0.0019331786315888166,
+          -0.006383516825735569,
+          -0.013943050988018513,
+          -0.022618189454078674,
+          0.023649217560887337,
+          0.01726972870528698,
+          0.019798971712589264,
+          -0.01797856017947197,
+          0.006258665584027767,
+          -0.011019118130207062,
+          0.010414999909698963,
+          0.01167962048202753,
+          -0.0017408676212653518,
+          -0.004293267615139484,
+          -0.001821416663005948,
+          -0.021055536344647408,
+          -0.01749526709318161,
+          0.005626355297863483,
+          0.02055613324046135,
+          0.0001618533715372905,
+          -0.009826990775763988,
+          -0.029674291610717773,
+          -0.004389926325529814,
+          -0.005678711924701929,
+          -0.0199439600110054,
+          -0.01150241307914257,
+          -0.015392934903502464,
+          -0.015642637386918068,
+          -0.003228005487471819,
+          0.0022271827328950167,
+          -0.005578025709837675,
+          0.004023428075015545,
+          0.018574625253677368,
+          0.0013532249722629786,
+          0.020411144942045212,
+          0.010068638250231743,
+          0.017189180478453636,
+          0.013596689328551292,
+          -0.013411426916718483,
+          0.006282830610871315,
+          -0.009593399241566658,
+          -0.012050146237015724,
+          0.018026890233159065,
+          -0.014386070892214775,
+          -2.73583791567944e-05,
+          -0.023182032629847527,
+          0.001140776788815856,
+          -0.02395530417561531,
+          0.01892904005944729,
+          -0.034475017338991165,
+          -0.011961543001234531,
+          0.005743151530623436,
+          -0.01714085042476654,
+          -0.01949288509786129,
+          -0.028369395062327385,
+          -0.0008281455375254154,
+          -0.018059110268950462,
+          -0.00408383971080184,
+          -0.016770323738455772,
+          0.0012011885410174727,
+          -0.0028836580459028482,
+          0.010785525664687157,
+          -0.008497931063175201,
+          0.051390331238508224,
+          0.0029923992697149515,
+          0.005537751130759716,
+          -0.01263010036200285,
+          0.001191119896247983,
+          -0.012066256254911423,
+          0.013427536003291607,
+          -0.0016885106451809406,
+          -0.0038200414273887873,
+          0.008409326896071434,
+          0.010704976506531239,
+          -0.0032561977859586477,
+          0.004293267615139484,
+          0.02395530417561531,
+          0.02772500365972519,
+          -0.001674414612352848,
+          -0.0120340371504426,
+          0.004990017041563988,
+          0.009061774238944054,
+          0.014861310832202435,
+          0.008264338597655296,
+          -0.016625335440039635,
+          -0.01267842948436737,
+          0.013572524301707745,
+          0.005328323692083359,
+          -0.011832663789391518,
+          0.01868739351630211,
+          -0.013065065257251263,
+          -0.022714847698807716,
+          0.005392762832343578,
+          -0.00798241700977087,
+          0.012815362773835659,
+          -0.00915843341499567,
+          0.006625164300203323,
+          0.009770606644451618,
+          0.007088321726769209,
+          0.025276310741901398,
+          -0.004192580934613943,
+          -0.004216745961457491,
+          -0.015038518235087395,
+          -0.0017388538690283895,
+          -0.0009162460919469595,
+          0.027563905343413353,
+          0.02875603176653385,
+          0.008779852651059628,
+          -0.059960756450891495,
+          -0.003125305287539959,
+          0.016512567177414894,
+          -0.010527768172323704,
+          -0.0182202085852623,
+          0.015320440754294395,
+          -0.0021828808821737766,
+          0.002150661079213023,
+          0.01689920201897621,
+          0.009907539933919907,
+          -0.005340405739843845,
+          0.014845200814306736,
+          0.03531273081898689,
+          0.009432300925254822,
+          0.008203926496207714,
+          -0.007656192872673273,
+          -0.01865517348051071,
+          -0.029303764924407005,
+          -0.004961824975907803,
+          -0.004389926325529814,
+          0.001964391442015767,
+          0.004196608439087868,
+          -0.00904566515237093,
+          0.037890300154685974,
+          0.018735721707344055,
+          0.01904180832207203,
+          0.006238528527319431,
+          0.008393216878175735,
+          -0.02279539778828621,
+          -0.03805140033364296,
+          0.003969057463109493,
+          -0.024019744247198105,
+          0.0018687392584979534,
+          0.010729141533374786,
+          0.01962176337838173,
+          0.005022237077355385,
+          0.011889047920703888,
+          -0.008360997773706913,
+          0.004212718456983566,
+          -2.6917876311927103e-05,
+          0.005783426109701395,
+          -0.01599705219268799,
+          -0.01950899325311184,
+          -0.009013445116579533,
+          0.0043738167732954025,
+          0.010278066620230675,
+          -0.005408872850239277,
+          -0.005300131160765886,
+          0.005936469417065382,
+          -0.02443859912455082,
+          0.0029541384428739548,
+          -0.011993762105703354,
+          0.0037092864513397217,
+          -0.009464520029723644,
+          0.016947532072663307,
+          0.020233936607837677,
+          -0.00956923421472311,
+          -0.0017650322988629341,
+          0.0009832025971263647,
+          0.02300482615828514,
+          -0.009899485856294632,
+          0.01656089723110199,
+          0.005771343596279621,
+          0.029046008363366127,
+          0.01962176337838173,
+          0.015747351571917534,
+          -0.010382779873907566,
+          0.012936186976730824,
+          0.022618189454078674,
+          -0.007495094556361437,
+          0.013572524301707745,
+          -0.00857042521238327,
+          0.012710649520158768,
+          -0.006133814807981253,
+          -0.010704976506531239,
+          0.0034354194067418575,
+          0.01327449269592762,
+          0.011397698894143105,
+          0.003228005487471819,
+          0.02429361082613468,
+          -0.0053202686831355095,
+          -0.010374724864959717,
+          -0.019090138375759125,
+          0.0036468608304858208,
+          -0.0017569774063304067,
+          0.0043738167732954025,
+          0.01420886255800724,
+          -0.02337535098195076,
+          -0.000712356180883944,
+          0.01533654984086752,
+          -0.011599071323871613,
+          -0.001532446825876832,
+          -0.022618189454078674,
+          -0.01599705219268799,
+          0.009851155802607536,
+          -0.0015102957841008902,
+          -0.005968688987195492,
+          -0.024615807458758354,
+          0.01846185512840748,
+          -0.0029138640966266394,
+          0.002593681216239929,
+          0.036214880645275116,
+          0.008892621845006943,
+          0.039050206542015076,
+          -0.0068063996732234955,
+          0.012001817114651203,
+          -0.011292984709143639,
+          -0.011019118130207062,
+          -0.026001252233982086,
+          0.013604744337499142,
+          -0.01015724241733551,
+          0.019315676763653755,
+          0.009311476722359657,
+          0.006472120992839336,
+          -0.0026118047535419464,
+          0.0050745937041938305,
+          0.0008633857360109687,
+          -0.015779569745063782,
+          0.026001252233982086,
+          -0.02195768617093563,
+          0.028240516781806946,
+          0.005638437811285257,
+          -0.0120340371504426,
+          0.012823417782783508,
+          0.011139941401779652,
+          0.004776562098413706,
+          0.01051165908575058,
+          0.0030447563622146845,
+          -0.0054612294770777225,
+          0.005437064915895462,
+          -0.00634324224665761,
+          -0.04262658953666687,
+          -0.025807933881878853,
+          -0.006109649781137705,
+          -0.003950933925807476,
+          0.0075998082756996155,
+          -0.013419480994343758,
+          -0.021699929609894753,
+          0.0030064955353736877,
+          0.023745877668261528,
+          -0.021699929609894753,
+          0.011429918929934502,
+          0.0018697461346164346,
+          0.011655456386506557,
+          -0.01784968189895153,
+          0.013363096863031387,
+          -0.002575557678937912,
+          0.02996426820755005,
+          0.000515262596309185,
+          -0.0031635661143809557,
+          0.012517331168055534,
+          0.000442516669863835,
+          -0.021571051329374313,
+          -0.00028745963936671615,
+          -0.012452892027795315,
+          -0.02020171657204628,
+          0.02182880789041519,
+          -0.008115322329103947,
+          -0.006556697189807892,
+          -0.004216745961457491,
+          0.024116402491927147,
+          -0.0006927223294042051,
+          0.0011951474007219076,
+          -0.005863975267857313,
+          -0.011736005544662476,
+          0.0182040985673666,
+          -0.02688729204237461,
+          -0.0030306600965559483,
+          -0.007285666652023792,
+          0.0007954224711284041,
+          -0.005739124026149511,
+          -0.005275966599583626,
+          0.017189180478453636,
+          -0.017930231988430023,
+          0.005586080718785524,
+          -0.014426345005631447,
+          -0.003695190418511629,
+          0.0008608686039224267,
+          0.011019118130207062,
+          0.03157525137066841,
+          0.004091894719749689,
+          0.030850308015942574,
+          0.014329686760902405,
+          -0.008594590239226818,
+          0.0012535455171018839,
+          0.01878405176103115,
+          -0.010535823181271553,
+          0.02160327136516571,
+          -0.01308923028409481,
+          -0.008650974370539188,
+          0.007241365034133196,
+          -0.007632027845829725,
+          0.00458324421197176,
+          0.011260765604674816,
+          0.015409043990075588,
+          0.0014871378662064672,
+          0.0009711202001199126,
+          -0.016029272228479385,
+          0.010737196542322636,
+          -0.0046799033880233765,
+          -0.008916785940527916,
+          0.00578745361417532,
+          0.007269557099789381,
+          -0.014007490128278732,
+          0.0009691064478829503,
+          -0.007877702824771404,
+          0.0050141820684075356,
+          -0.01889682002365589,
+          -0.008940950967371464,
+          -0.024470819160342216,
+          -0.00391670037060976,
+          -0.013612799346446991,
+          0.005650519859045744,
+          0.001747915637679398,
+          -0.005952578969299793,
+          0.016174260526895523,
+          0.015844009816646576,
+          0.005924386903643608,
+          0.006045210640877485,
+          -0.02666175551712513,
+          -0.0038764257915318012,
+          0.0007294728420674801,
+          -0.0013038887409493327,
+          0.0016653528437018394,
+          -0.011244655586779118,
+          -0.011421863920986652,
+          -0.018703503534197807,
+          -0.010189462453126907,
+          -0.0027084636967629194,
+          0.014514949172735214,
+          -0.0037233824841678143,
+          -0.0014196779811754823,
+          -0.0058760573156178,
+          0.006971525494009256,
+          0.012509276159107685,
+          0.022618189454078674,
+          -0.014498839154839516,
+          -0.0132100535556674,
+          0.015417098999023438,
+          0.03197799623012543,
+          -0.007998526096343994,
+          0.008066993206739426,
+          -0.010141132399439812,
+          -0.00607340270653367,
+          -0.005578025709837675,
+          -0.01809133030474186,
+          0.010616372339427471,
+          0.00018186478700954467,
+          0.02548573724925518,
+          0.00578745361417532,
+          -0.004013359546661377,
+          0.007813263684511185,
+          0.010350560769438744,
+          0.012082366272807121,
+          0.00754745164886117,
+          0.030512001365423203,
+          0.0005930428160354495,
+          0.02360088936984539,
+          -0.0012686484260484576,
+          -0.010938568972051144,
+          -0.0021869083866477013,
+          0.0017297921003773808,
+          0.00484502874314785,
+          0.010076693259179592,
+          0.0281760785728693,
+          0.0035220098216086626,
+          -0.004772534593939781,
+          0.01633535884320736,
+          0.006391571834683418,
+          -0.0005880084936507046,
+          -0.012718704529106617,
+          -0.01227568369358778,
+          -0.006335187237709761,
+          -0.0007934087188914418,
+          0.002002652268856764,
+          -0.016440073028206825,
+          -0.03550604730844498,
+          0.010020309127867222,
+          0.006089512724429369,
+          -0.016995862126350403,
+          -0.01773691363632679,
+          -0.006178116425871849,
+          0.019122358411550522,
+          0.00040048008668236434,
+          -0.009061774238944054,
+          -0.0024305693805217743,
+          0.007575643714517355,
+          0.022263772785663605,
+          -0.0014569320483133197,
+          0.031075846403837204,
+          -0.015860119834542274,
+          -0.01479687076061964,
+          0.012960351072251797,
+          0.0024245281238108873,
+          0.008288503624498844,
+          -0.025195760652422905,
+          0.010125023312866688,
+          -0.006931250914931297,
+          0.022747067734599113,
+          0.007559533696621656,
+          0.010390834882855415,
+          -0.012251519598066807,
+          -0.006242556031793356,
+          0.016367578878998756,
+          -0.010906348936259747,
+          -0.004708095453679562,
+          -0.003236060496419668,
+          -0.026242898777127266,
+          0.0012092434335500002,
+          0.011526577174663544,
+          -0.0015515772392973304,
+          0.015803735703229904,
+          -0.006886948831379414,
+          -0.002825259929522872,
+          -0.0003078486188314855,
+          -0.00019105241517536342,
+          -0.008272393606603146,
+          -0.041982196271419525,
+          -0.002799081616103649,
+          0.02078166976571083,
+          -0.005102786235511303,
+          -0.02210267446935177,
+          0.01725361868739128,
+          0.003308554645627737,
+          0.0036851216573268175,
+          -0.01328254770487547,
+          -0.021571051329374313,
+          -0.015827899798750877,
+          0.01308923028409481,
+          -0.018252428621053696,
+          -0.023536449298262596,
+          -0.0016804557526484132,
+          -0.017946342006325722,
+          0.002428555628284812,
+          -0.02197379618883133,
+          -0.02055613324046135,
+          -0.021780479699373245,
+          -0.005497476551681757,
+          0.004192580934613943,
+          -0.025662945583462715,
+          0.026371778920292854,
+          -0.008441546931862831,
+          -0.006149924360215664,
+          -0.014861310832202435,
+          0.008892621845006943,
+          0.0017126754391938448,
+          0.021764369681477547,
+          -0.02679063379764557,
+          0.009585344232618809,
+          -0.0023681437596678734,
+          -0.015417098999023438,
+          -0.005944523960351944,
+          -0.0015868174377828836,
+          0.009673947468400002,
+          0.03499053418636322,
+          0.006524477619677782,
+          -0.021877137944102287,
+          -0.023343130946159363,
+          0.012847582809627056,
+          0.003991208504885435,
+          -0.010801635682582855,
+          -0.010737196542322636,
+          -0.01715696044266224,
+          -0.015425154007971287,
+          0.014579388312995434,
+          0.01367723848670721,
+          0.018107440322637558,
+          -0.0032380742486566305,
+          0.008199899457395077,
+          -0.01880016177892685,
+          -0.005747179035097361,
+          0.0033790350425988436,
+          0.006262693088501692,
+          -0.018397416919469833,
+          0.035892684012651443,
+          0.005332351196557283,
+          -0.004454365465790033,
+          -0.00581161817535758,
+          0.0052276370115578175,
+          0.0028333149384707212,
+          -0.008844291791319847,
+          -0.01316172443330288,
+          0.00737427081912756,
+          -0.009085939265787601,
+          0.007003745064139366,
+          0.009134269319474697,
+          -0.005541778635233641,
+          -0.019895629957318306,
+          -0.003364939009770751,
+          0.011727950535714626,
+          -0.017237508669495583,
+          -0.01110772229731083,
+          -0.020523913204669952,
+          -0.003024619072675705,
+          0.003979125991463661,
+          -0.00014410738367587328,
+          0.0018687392584979534,
+          -0.01639174297451973,
+          -0.004176471382379532,
+          -0.02911044843494892,
+          0.0008155597024597228,
+          -0.02182880789041519,
+          -0.018848491832613945,
+          0.011921267956495285,
+          0.00975449662655592,
+          0.00604923814535141,
+          0.003705258946865797,
+          0.00047549145529046655,
+          0.014120258390903473,
+          0.002132537541911006,
+          -0.016528677195310593,
+          0.022408761084079742,
+          -0.0063593522645533085,
+          0.015594307333230972,
+          0.029400425031781197,
+          0.01380611676722765,
+          -0.00687486631795764,
+          -0.017801353707909584,
+          -0.01003641914576292,
+          -0.02210267446935177,
+          0.0047685070894658566,
+          -0.01772080361843109,
+          -0.024277500808238983,
+          0.014635773375630379,
+          -0.023069264367222786,
+          0.0011488316813483834,
+          -0.01693142205476761,
+          0.006737933028489351,
+          -0.00637546181678772,
+          0.012597880326211452,
+          0.017833571881055832,
+          0.0016703871078789234,
+          0.0027789443265646696,
+          -0.011880993843078613,
+          -0.0036106137558817863,
+          0.006689603440463543,
+          -0.005175280384719372,
+          -0.004905440844595432,
+          0.001707641058601439,
+          -0.010922458954155445,
+          -0.019766751676797867,
+          -0.012775088660418987,
+          -0.025646835565567017,
+          -0.0010632482590153813,
+          0.0049215503968298435,
+          0.006246583536267281,
+          -0.004273130092769861,
+          -0.00355221563950181,
+          0.0033448017202317715,
+          -0.008594590239226818,
+          0.0031917584128677845,
+          -0.0010144152911379933,
+          -0.005900222342461348,
+          0.006170061882585287,
+          0.0034515291918069124,
+          0.0030387151055037975,
+          0.01427330169826746,
+          0.00243862415663898,
+          -0.005171252880245447,
+          -0.038244716823101044,
+          0.0062546380795538425,
+          -0.018848491832613945,
+          0.001827457919716835,
+          -0.011446028016507626,
+          7.104683027137071e-05,
+          -0.006717795506119728,
+          -0.006967497989535332,
+          0.016770323738455772,
+          0.003882467048242688,
+          0.010116968303918839,
+          0.015940668061375618,
+          -0.010600262321531773,
+          -0.008949005976319313,
+          0.012501221150159836,
+          -9.49441164266318e-06,
+          -0.00481280917301774,
+          0.0016542773228138685,
+          -0.01566680148243904,
+          0.008916785940527916,
+          -0.00951285008341074,
+          0.009593399241566658,
+          -0.012944241985678673,
+          0.007386353332549334,
+          -0.008731523528695107,
+          -0.011405753903090954,
+          0.00024882121942937374,
+          0.008763742633163929,
+          0.01581178978085518,
+          0.003097113221883774,
+          0.006025073118507862,
+          0.0035602706484496593,
+          0.02147439308464527,
+          -0.01255760621279478,
+          0.0009464520262554288,
+          -0.005465256981551647,
+          -0.023455901071429253,
+          -0.015844009816646576,
+          0.025324638932943344,
+          0.03937240317463875,
+          -0.011711840517818928,
+          -0.005429009906947613,
+          0.0011880993843078613,
+          0.009085939265787601,
+          0.0023983498103916645,
+          -0.004490613006055355,
+          -0.01309728529304266,
+          -0.02608180046081543,
+          -0.02018560655415058,
+          0.004555052146315575,
+          0.01760803535580635,
+          -0.010721086524426937,
+          -0.004013359546661377,
+          -0.005940496921539307,
+          -0.00431340467184782,
+          -0.0028534522280097008,
+          -0.008642919361591339,
+          -0.00551761407405138,
+          -0.005723014008253813,
+          -0.004514777567237616,
+          -0.022151004523038864,
+          0.009424245916306973,
+          0.011663510464131832,
+          0.008369052782654762,
+          0.0028172051534056664,
+          0.020829999819397926,
+          -0.02453525923192501,
+          -0.0067057134583592415,
+          -0.000449816434411332,
+          -0.009786716662347317,
+          0.02500244230031967,
+          0.00798241700977087,
+          -0.016947532072663307,
+          0.0012626072857528925,
+          0.006363379769027233,
+          0.00024693337036296725,
+          0.018945150077342987,
+          -0.010439164005219936,
+          0.021554941311478615,
+          -0.011486303061246872,
+          -0.004454365465790033,
+          -0.0022734985686838627,
+          0.006882921326905489,
+          0.03228408098220825,
+          0.012324013747274876,
+          -0.00950479507446289,
+          0.01068886648863554,
+          0.017688583582639694,
+          0.011437973007559776,
+          -0.009947814978659153,
+          0.021989906206727028,
+          -0.004196608439087868,
+          0.015296275727450848,
+          0.016528677195310593,
+          0.008344887755811214,
+          -0.011454083025455475,
+          -0.008465711027383804,
+          0.022167114540934563,
+          0.003783794352784753,
+          -0.02664564549922943,
+          -0.010012254118919373,
+          0.0029319876339286566,
+          0.023552559316158295,
+          0.007712577003985643,
+          0.0028756032697856426,
+          0.001929151127114892,
+          0.024631917476654053,
+          -0.005662602372467518,
+          0.027515575289726257,
+          -0.016754213720560074,
+          0.004542969632893801,
+          -0.001554597751237452,
+          0.014200807549059391,
+          -0.01831686682999134,
+          0.001941233524121344,
+          -0.0036448470782488585,
+          0.004631573799997568,
+          0.008892621845006943,
+          0.007120541296899319,
+          0.019283456727862358,
+          -0.004933632910251617,
+          -0.013258383609354496,
+          0.008457656018435955,
+          0.008868456818163395,
+          -0.01736638695001602,
+          0.027547795325517654,
+          0.020378924906253815,
+          0.00798241700977087,
+          -0.016866983845829964,
+          0.005130978301167488,
+          0.04043565317988396,
+          0.018606843426823616,
+          0.013016736134886742,
+          -0.010358615778386593,
+          -0.022634299471974373,
+          0.007587726227939129,
+          0.009359806776046753,
+          0.0024144595954567194,
+          0.004369789268821478,
+          0.004164388868957758,
+          -0.014321631751954556,
+          0.0001999883388634771,
+          0.03592490404844284,
+          -0.010785525664687157,
+          -0.01599705219268799,
+          -0.004301322158426046,
+          -0.011727950535714626,
+          -0.018477965146303177,
+          0.010414999909698963,
+          0.004865166265517473,
+          -0.020362814888358116,
+          0.00950479507446289,
+          0.001946267788298428,
+          -0.0164481271058321,
+          0.0041281417943537235,
+          0.0032199507113546133,
+          -0.003926768898963928,
+          -0.004841001238673925,
+          0.001975466962903738,
+          -0.009118159301578999,
+          0.005702876951545477,
+          -0.0011528590694069862,
+          -0.003743519773706794,
+          0.010052529163658619,
+          0.01654478721320629,
+          0.0024547341745346785,
+          0.010431109927594662,
+          0.0019734532106667757,
+          0.013298657722771168,
+          -0.018300756812095642,
+          0.00657280720770359,
+          -0.005972716491669416,
+          0.001719723455607891,
+          -0.0024104320909827948,
+          -0.005429009906947613,
+          -0.03685927018523216,
+          0.0029984405264258385,
+          -0.0011961542768403888,
+          0.014450510032474995,
+          -0.01736638695001602,
+          0.00032269986695609987,
+          0.01309728529304266,
+          -0.014361905865371227,
+          -0.00022830638044979423,
+          0.01846185512840748,
+          0.001276703318580985,
+          0.02279539778828621,
+          0.012315958738327026,
+          0.013371151871979237,
+          0.002072125906124711,
+          -0.02326258271932602,
+          0.02442248910665512,
+          -0.009915594942867756,
+          0.004458392970263958,
+          0.0211360864341259,
+          0.03312179446220398,
+          -0.021168306469917297,
+          -0.008546260185539722,
+          -0.015344604849815369,
+          -0.002984344493597746,
+          0.021667709574103355,
+          0.0040757847018539906,
+          0.028192188590765,
+          0.013226163573563099,
+          0.01648840308189392,
+          -0.025936812162399292,
+          0.0021244827657938004,
+          0.014378015883266926,
+          -0.007696467451751232,
+          -0.017930231988430023,
+          -0.01736638695001602,
+          -0.013234218582510948,
+          0.010648592375218868,
+          -0.02149050123989582,
+          0.026484547182917595,
+          -0.0009686030098237097,
+          0.01678643375635147,
+          0.009013445116579533,
+          -0.010471384041011333,
+          0.004929605405777693,
+          0.00627880310639739,
+          0.0037636570632457733,
+          -0.01726972870528698,
+          0.00510681327432394,
+          -0.007712577003985643,
+          0.0061539518646895885,
+          -0.004043565131723881,
+          0.029335984960198402,
+          -0.008010609075427055,
+          0.008489876054227352,
+          -0.011969597078859806,
+          -0.011309094727039337,
+          0.011317149735987186,
+          0.000714873312972486,
+          -0.012404562905430794,
+          0.0065123955719172955,
+          -0.002897754078730941,
+          0.01455522421747446,
+          -0.015932613983750343,
+          0.00405162014067173,
+          0.030334794893860817,
+          -0.01566680148243904,
+          -0.009327586740255356,
+          0.0030326738487929106,
+          0.0057673160918056965,
+          0.0034656254574656487,
+          0.0019684189464896917,
+          0.012847582809627056,
+          -0.00378983560949564,
+          -0.010946623980998993,
+          0.012348177842795849,
+          -0.011486303061246872,
+          0.008530150167644024,
+          0.0016834763810038567,
+          -0.008804017677903175,
+          0.029046008363366127,
+          -0.010133077390491962,
+          0.01257371623069048,
+          0.021087756380438805,
+          -0.023069264367222786,
+          -0.0077528515830636024,
+          -0.0105599882081151,
+          -0.019074028357863426,
+          -0.011518522165715694,
+          -0.005022237077355385,
+          0.00107231002766639,
+          0.0010400903411209583,
+          0.0060129910707473755,
+          3.240842852392234e-05,
+          0.004784617107361555,
+          0.0011589003261178732,
+          -0.0033689665142446756,
+          -0.010914403945207596,
+          0.03006092645227909,
+          -0.018977370113134384,
+          0.024728575721383095,
+          -0.00021584643400274217,
+          0.016947532072663307,
+          0.0017348263645544648,
+          -0.008034773170948029,
+          0.014136368408799171,
+          0.018300756812095642,
+          -0.019589543342590332,
+          -0.008123377338051796,
+          0.004619491286575794,
+          -0.0026601343415677547,
+          -0.009931704960763454,
+          -0.011760169640183449,
+          0.0038280964363366365,
+          -0.028949350118637085,
+          -0.0044221458956599236,
+          -0.011325204744935036,
+          -0.0035441608633846045,
+          -0.00777701660990715,
+          -0.01833297684788704,
+          -0.009118159301578999,
+          -0.01216291543096304,
+          -0.012009872123599052,
+          -0.008280448615550995,
+          -0.018945150077342987,
+          0.006186171434819698,
+          -0.02395530417561531,
+          0.014990189112722874,
+          -0.014200807549059391,
+          -0.005147087853401899,
+          0.015650691464543343,
+          0.03254184126853943,
+          0.006391571834683418,
+          0.0010622413828969002,
+          0.0016713939839974046,
+          0.004269102588295937,
+          0.0031696073710918427,
+          -0.0152721107006073,
+          -0.016641445457935333,
+          -0.0032380742486566305,
+          -0.001123660011216998,
+          0.02020171657204628,
+          0.01846185512840748,
+          -0.0036972041707485914,
+          -0.005384707823395729,
+          0.00869930349290371,
+          -0.002859493251889944,
+          -0.008042828179895878,
+          0.005324296187609434,
+          0.003431391902267933,
+          0.029642071574926376,
+          -0.011808499693870544,
+          -0.006077430211007595,
+          0.016085656359791756,
+          0.008457656018435955,
+          0.006798344664275646,
+          -0.010841909795999527,
+          0.00798241700977087,
+          0.006484203040599823,
+          0.017527485266327858,
+          0.0111641064286232,
+          0.0035723529290407896,
+          -0.010173352435231209,
+          -0.010817745700478554,
+          0.0006177109898999333,
+          0.006979580037295818,
+          -0.004446310922503471,
+          -0.022022126242518425,
+          0.0016673665959388018,
+          -0.0008487862069159746,
+          -0.021748259663581848,
+          -0.017060300335288048,
+          0.032815705984830856,
+          0.012879802845418453,
+          0.009867265820503235,
+          -0.0027708893176168203,
+          0.022505421191453934,
+          0.008449601009488106,
+          0.0018314853077754378,
+          0.008812072686851025,
+          0.0012213258305564523,
+          0.006395599339157343,
+          0.003228005487471819,
+          -0.03595712035894394,
+          0.004800726659595966,
+          -0.0024023770820349455,
+          0.006762097589671612,
+          -0.0031514838337898254,
+          0.004269102588295937,
+          -0.0044745029881596565,
+          -0.010390834882855415,
+          -0.02092665806412697,
+          0.022054346278309822,
+          0.015634581446647644,
+          0.021587161347270012,
+          -0.005437064915895462,
+          -0.0020902494434267282,
+          0.007913949899375439,
+          0.025195760652422905,
+          0.0005698849563486874,
+          0.01633535884320736,
+          0.02372976765036583,
+          0.014893529936671257,
+          0.020169496536254883,
+          -0.0031837034039199352,
+          0.04001679643988609,
+          -0.00114581105299294,
+          -0.0026359695475548506,
+          -0.0025513931177556515,
+          -0.010495549067854881,
+          0.01315366942435503,
+          -7.123561954358593e-05,
+          -0.005038346629589796,
+          0.006029100622981787,
+          0.0003780773840844631,
+          0.011639346368610859,
+          -0.01068081147968769,
+          -0.010978843085467815,
+          0.0021808671299368143,
+          0.019412335008382797,
+          0.01227568369358778,
+          0.024116402491927147,
+          -0.021345512941479683,
+          0.005904249846935272,
+          -0.027209488674998283,
+          -0.012968406081199646,
+          0.0032481427770107985,
+          0.004204663448035717,
+          -0.01456327922642231,
+          -0.021877137944102287,
+          -0.009802826680243015,
+          0.011429918929934502,
+          0.007031937129795551,
+          0.010382779873907566,
+          0.0071004037745296955,
+          -0.0003302513505332172,
+          0.006717795506119728,
+          -0.01997617818415165,
+          -0.020475583150982857,
+          0.0019543226808309555,
+          0.0013421495677903295,
+          -0.022022126242518425,
+          -0.006991662550717592,
+          0.010302230715751648,
+          -6.689351721433923e-05,
+          -0.016472293063998222,
+          0.016254810616374016,
+          -0.01598094217479229,
+          -0.01109161227941513,
+          -0.0014670006930828094,
+          0.007688412442803383,
+          0.0024587614461779594,
+          0.015127122402191162,
+          -0.009553124196827412,
+          0.009609508328139782,
+          0.011760169640183449,
+          -0.013234218582510948,
+          0.024454709142446518,
+          -0.0023399516940116882,
+          -0.015827899798750877,
+          0.016303138807415962,
+          -0.010060583241283894,
+          -0.02550184726715088,
+          0.025775713846087456,
+          -0.013983325101435184,
+          -0.010294175706803799,
+          -0.024631917476654053,
+          -0.019992288202047348,
+          -0.007845482788980007,
+          -0.01162323635071516,
+          -0.034926094114780426,
+          -0.018236318603157997,
+          0.029545413330197334,
+          0.018590733408927917,
+          0.0211360864341259,
+          0.00325821153819561,
+          0.003961002454161644,
+          0.013814171776175499,
+          -0.01566680148243904,
+          0.0010541864903643727,
+          0.0017771146958693862,
+          9.546327783027664e-05,
+          0.016963642090559006,
+          -0.001924116862937808,
+          -0.0028051226399838924,
+          -0.017914121970534325,
+          -0.020636681467294693,
+          0.0041885534301400185,
+          0.002146633807569742,
+          0.019235126674175262,
+          0.014659937471151352,
+          -0.02490578405559063,
+          0.005888139829039574,
+          -0.013403371907770634,
+          0.03428170084953308,
+          -0.01416053343564272,
+          0.022376542910933495,
+          0.00863486435264349,
+          0.02195768617093563,
+          0.0026681891176849604,
+          -0.00037128105759620667,
+          0.008779852651059628,
+          0.010334450751543045,
+          0.0034636117052286863,
+          0.002946083666756749,
+          -0.001038076588883996,
+          8.589807112002745e-05,
+          -0.004897385835647583,
+          0.03075364977121353,
+          0.012710649520158768,
+          -0.0016975724138319492,
+          -0.018171878531575203,
+          -0.009585344232618809,
+          -0.0062143635004758835,
+          -0.01725361868739128,
+          -0.010141132399439812,
+          0.029915938153862953,
+          0.003501872532069683,
+          0.0007470929995179176,
+          3.530316462274641e-05,
+          0.0032159232068806887,
+          0.028385505080223083,
+          -0.02337535098195076,
+          -0.009738387539982796,
+          -0.0036972041707485914,
+          -0.004168416373431683,
+          0.014974079094827175,
+          0.0021124002523720264,
+          -0.006709740497171879,
+          -0.0015425154706463218,
+          0.012549551203846931,
+          0.013846391811966896,
+          0.002241278998553753,
+          -0.006504340562969446,
+          -0.009295366704463959,
+          -0.022698739543557167,
+          0.0036911629140377045,
+          0.0006811433704569936,
+          0.0085140410810709,
+          0.010841909795999527,
+          -0.005698849447071552,
+          -0.02843383513391018,
+          -0.009432300925254822,
+          0.007390380837023258,
+          0.004071757197380066,
+          -0.0037314374931156635,
+          0.005984798539429903,
+          -0.006681548431515694,
+          0.006826536729931831,
+          0.019911739975214005,
+          0.024277500808238983,
+          0.020958878099918365,
+          0.007869647815823555,
+          0.020974988117814064,
+          0.028949350118637085,
+          -0.007015827111899853,
+          0.0006962463376112282,
+          0.00956923421472311,
+          0.014224972575902939,
+          0.0014670006930828094,
+          0.017334168776869774,
+          0.002881644293665886,
+          -0.0058075906708836555,
+          -0.01110772229731083,
+          -0.0008281455375254154,
+          0.0317041277885437,
+          -0.016327304765582085,
+          -0.00031187606509774923,
+          0.006141869351267815,
+          0.019412335008382797,
+          0.009714222513139248,
+          0.004675875883549452,
+          0.005316241178661585,
+          0.013483921065926552,
+          -0.010463329032063484,
+          -0.012710649520158768,
+          -0.0021345512941479683,
+          0.00498599000275135,
+          0.011671565473079681,
+          -0.01626286469399929,
+          0.0032300192397087812,
+          -0.021313294768333435,
+          0.009891430847346783,
+          -0.019090138375759125,
+          -0.011591016314923763,
+          0.0024668164551258087,
+          0.006544615142047405,
+          0.006298940163105726,
+          0.009271202608942986,
+          0.029545413330197334,
+          -0.0005583059974014759,
+          0.034475017338991165,
+          -0.005078621208667755,
+          -0.017785243690013885,
+          -0.016053438186645508,
+          0.00038210483035072684,
+          -0.020266154780983925,
+          -0.003217936959117651,
+          0.016399798914790154,
+          -0.007184980437159538,
+          -0.009746442548930645,
+          0.00451880507171154,
+          0.010253901593387127,
+          -0.0063593522645533085,
+          -0.005525668617337942,
+          -0.037278126925230026,
+          0.021458283066749573,
+          -0.0013814171543344855,
+          0.010326395742595196,
+          0.019122358411550522,
+          0.025807933881878853,
+          -0.01428135670721531,
+          -0.0026641618460416794,
+          0.0181879885494709,
+          0.003195785917341709,
+          0.023633107542991638,
+          -0.0047805896028876305,
+          0.008433491922914982,
+          -0.018300756812095642,
+          0.007732714526355267,
+          -0.022167114540934563,
+          -0.010616372339427471,
+          0.004651710856705904,
+          -0.027676673606038094,
+          0.0012646210379898548,
+          -0.0002251599362352863,
+          -0.013491976074874401,
+          -0.004764479584991932,
+          -0.0021667710971087217,
+          0.0012515317648649216,
+          -0.007555506192147732,
+          -0.014708267524838448,
+          -0.01639174297451973,
+          -0.027290038764476776,
+          0.00852209609001875,
+          0.0020701121538877487,
+          -0.003348829224705696,
+          0.0015102957841008902,
+          0.005247774533927441,
+          0.007229282520711422,
+          0.012815362773835659,
+          -0.007201090455055237,
+          0.060991786420345306,
+          -0.014047764241695404,
+          0.0020388993434607983,
+          -0.00508264871314168,
+          0.018606843426823616,
+          0.00868319347500801,
+          0.0027366559952497482,
+          -0.011832663789391518,
+          0.0030105228070169687,
+          -0.015473484061658382,
+          -0.013943050988018513,
+          -0.005932441912591457,
+          -5.896446600672789e-05,
+          0.002843383466824889,
+          -0.019702311605215073,
+          -0.0011528590694069862,
+          -0.01416053343564272,
+          2.218246845586691e-05,
+          0.002350020222365856,
+          -0.007986444048583508,
+          0.017785243690013885,
+          -0.018622953444719315,
+          -0.0085140410810709,
+          0.015167397446930408,
+          0.005179307889193296,
+          0.007881729863584042,
+          -0.013306712731719017,
+          -0.012533441185951233,
+          0.000388649437809363,
+          -0.001509288907982409,
+          -0.0013310740469023585,
+          -0.0091020492836833,
+          -0.008151569403707981,
+          -0.012654264457523823,
+          -0.005884112324565649,
+          -0.0018234304152429104,
+          0.0009761545225046575,
+          0.011236600577831268,
+          -0.008066993206739426,
+          0.005702876951545477,
+          -0.020298374816775322,
+          0.021635491400957108,
+          -0.003149470081552863,
+          -0.0211199764162302,
+          -0.008763742633163929,
+          0.01586817391216755,
+          0.0007712577353231609,
+          0.0006418757257051766,
+          0.03157525137066841,
+          0.004059675149619579,
+          -0.0011045295977964997,
+          -0.005453174468129873,
+          -0.0013743691379204392,
+          -0.007720632012933493,
+          0.020974988117814064,
+          -0.00757967121899128,
+          0.010398889891803265,
+          0.003677066881209612,
+          0.0211521964520216,
+          -0.025759605690836906,
+          0.006766125094145536,
+          -0.011003008112311363,
+          -0.0026077774818986654,
+          -0.00857042521238327,
+          -0.012863692827522755,
+          0.008868456818163395,
+          -0.02656509540975094,
+          -0.031043626368045807,
+          0.006262693088501692,
+          -0.012106531299650669,
+          -0.03785808011889458,
+          0.008216009475290775,
+          -0.018365196883678436,
+          -0.004393953830003738,
+          0.004365761764347553,
+          -0.014966024085879326,
+          -0.01726972870528698,
+          -0.012058201245963573,
+          -0.01609371230006218,
+          -0.0010884198127314448,
+          0.015634581446647644,
+          -0.02429361082613468,
+          -0.013757787644863129,
+          -0.006649328861385584,
+          0.007595780771225691,
+          -0.006371434312313795,
+          -0.0039247553795576096,
+          -0.02244098111987114,
+          0.007160815875977278,
+          -0.0059082768857479095,
+          0.01022168155759573,
+          -0.009673947468400002,
+          0.008264338597655296,
+          0.0252602007240057,
+          0.0036911629140377045,
+          -0.002927960129454732,
+          0.005002099554985762,
+          0.011792389675974846,
+          -0.0006484203040599823,
+          0.0293682049959898,
+          -0.007031937129795551,
+          0.031526919454336166,
+          0.0020439336076378822,
+          0.003568325424566865,
+          -0.015602362342178822,
+          -0.004655738361179829,
+          0.012622045353055,
+          0.008034773170948029,
+          -0.013717513531446457,
+          -0.003211895702406764,
+          -0.027789441868662834,
+          0.012871747836470604,
+          -0.023536449298262596,
+          -0.0039750984869897366,
+          0.005541778635233641,
+          -0.018252428621053696,
+          0.009400080889463425,
+          0.0028917130548506975,
+          0.03238074108958244,
+          0.017801353707909584,
+          -0.0016633390914648771,
+          0.002116427756845951,
+          0.007877702824771404,
+          -0.0020127207972109318,
+          0.019541213288903236,
+          0.004748370032757521,
+          -0.017285838723182678,
+          -0.015650691464543343,
+          -0.019573433324694633,
+          0.0007576650241389871,
+          0.007901867851614952,
+          -0.01051165908575058,
+          0.012412617914378643,
+          -0.0013975270558148623,
+          0.022134894505143166,
+          -0.00551761407405138,
+          -0.008876511827111244,
+          -0.01315366942435503,
+          -0.01268648449331522,
+          0.01602121815085411,
+          -0.001299861236475408,
+          0.016423963010311127,
+          0.007785071153193712,
+          0.010302230715751648,
+          -0.011115777306258678,
+          -0.00014310052210930735,
+          -0.01602121815085411,
+          -0.005195417441427708,
+          -0.007535369135439396,
+          -0.0008845299016684294,
+          0.03937240317463875,
+          -0.006230473518371582,
+          -0.005863975267857313,
+          0.0012736828066408634,
+          -0.0117521146312356,
+          0.024358050897717476,
+          0.013798062689602375,
+          0.022457091137766838,
+          -0.010414999909698963,
+          0.00035970209864899516,
+          -0.008175734430551529,
+          0.01842963509261608,
+          0.006665438879281282,
+          -0.0037314374931156635,
+          -0.011373533867299557,
+          -0.003958988469094038,
+          0.006492258049547672,
+          0.017543595284223557,
+          0.00019470229744911194,
+          -0.015972888097167015,
+          -0.004498667549341917,
+          0.004156333860009909,
+          0.004341596737504005,
+          0.009408135898411274,
+          -0.004095922224223614,
+          0.019444555044174194,
+          0.008155597373843193,
+          -0.016383688896894455,
+          -0.00754745164886117,
+          -0.015441264025866985,
+          0.0019059933256357908,
+          0.013338932767510414,
+          -0.015392934903502464,
+          0.009134269319474697,
+          0.005352488253265619,
+          -0.028611043468117714,
+          0.013886665925383568,
+          -0.012613990344107151,
+          -0.004760452080518007,
+          0.028095528483390808,
+          0.006383516825735569,
+          -0.02300482615828514,
+          -0.005424982402473688,
+          0.002648051828145981,
+          -0.014498839154839516,
+          0.02219933457672596,
+          -0.006439901422709227,
+          0.003471666481345892,
+          -0.007760906592011452,
+          -0.004869193769991398,
+          0.027531685307621956,
+          -0.00200970028527081,
+          -0.0009978021262213588,
+          0.007325941231101751
+        ],
+        "keyphrases": [
+          "Capped Expert Choice",
+          "entropy-regularized linear programming",
+          "validation perplexity",
+          "load balanced training",
+          "fine-tuning results"
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "e96a5505-edee-45dd-be65-af56dee281e4",
+      "properties": {
+        "page_content": "D Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [? ]. We use mod x to map a token ID to an expert ID. This in some way ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 2. Hashing based routing performs much worse than expert choice in terms of average scores and variance.\n\n",
+        "title": "D Comparison With Hash Layer",
+        "embedding": [
+          -0.015004903078079224,
+          -0.027099307626485825,
+          -0.030763821676373482,
+          -0.012313068844377995,
+          0.005327117629349232,
+          -0.009055723436176777,
+          -0.007592933718115091,
+          0.055405039340257645,
+          -0.039389755576848984,
+          -0.0020019104704260826,
+          -0.016120845451951027,
+          -0.011890820227563381,
+          -0.0219267625361681,
+          0.00947797205299139,
+          -0.006043432280421257,
+          0.06357856839895248,
+          0.02605876699090004,
+          0.014650516211986542,
+          -0.0008935080841183662,
+          -0.05407797545194626,
+          0.01372307725250721,
+          0.008814438246190548,
+          -0.03872622177004814,
+          -0.0012384700821712613,
+          0.021429114043712616,
+          0.0030575317796319723,
+          -0.0006979353493079543,
+          -0.023012544959783554,
+          -0.03145751729607582,
+          0.046960070729255676,
+          0.024007845669984818,
+          0.01857893541455269,
+          -0.006804987788200378,
+          -0.05121271684765816,
+          -0.024294370785355568,
+          -0.013783399015665054,
+          0.046990230679512024,
+          0.038092851638793945,
+          -0.036946747452020645,
+          0.04632669687271118,
+          0.04659814387559891,
+          0.062311820685863495,
+          -0.03190992400050163,
+          -0.010179205797612667,
+          0.000586718088015914,
+          -0.0037851566448807716,
+          -0.014424311928451061,
+          -0.0021338630467653275,
+          0.03184960409998894,
+          -0.0449996292591095,
+          -0.011257447302341461,
+          -0.07594441622495651,
+          0.025395233184099197,
+          0.012343229725956917,
+          -0.005538241937756538,
+          0.050549183040857315,
+          0.0019906002562493086,
+          0.027808083221316338,
+          0.023600677028298378,
+          -0.00661648390814662,
+          -0.004599492996931076,
+          -0.014469552785158157,
+          -3.43724932463374e-05,
+          0.015306509099900723,
+          0.010194285772740841,
+          0.0011376205366104841,
+          -0.013157566078007221,
+          -0.008015181869268417,
+          0.04210421070456505,
+          -0.015427151694893837,
+          -0.039118312299251556,
+          0.031216232106089592,
+          -0.0038567879237234592,
+          0.0019679798278957605,
+          -0.028697820380330086,
+          -0.00024034235684666783,
+          0.021821200847625732,
+          0.03218137100338936,
+          -0.00124601018615067,
+          0.020252849906682968,
+          0.012456332333385944,
+          0.0006861538859084249,
+          -0.002107472624629736,
+          0.0172820296138525,
+          0.018880542367696762,
+          -0.03504662960767746,
+          -0.02144419401884079,
+          -0.06122603639960289,
+          -0.015442232601344585,
+          -0.018232088536024094,
+          -0.01572875864803791,
+          -0.015909722074866295,
+          -0.007057582959532738,
+          -0.0011168851051479578,
+          -0.05905447527766228,
+          0.0761253759264946,
+          0.005873778834939003,
+          -0.0025391464587301016,
+          -0.023660998791456223,
+          -0.00031456572469323874,
+          -0.013579814694821835,
+          0.0022752410732209682,
+          -0.013421471230685711,
+          0.0437932051718235,
+          0.06086410954594612,
+          -0.029512157663702965,
+          0.010978462174534798,
+          0.022107727825641632,
+          -0.0007973711472004652,
+          0.1003141850233078,
+          0.006884159054607153,
+          0.005941640119999647,
+          -0.037549957633018494,
+          0.002186644123867154,
+          -0.01580416038632393,
+          -0.04240581765770912,
+          -0.03700706735253334,
+          -0.05407797545194626,
+          0.019619476050138474,
+          -0.046748943626880646,
+          0.00444114999845624,
+          0.04273758456110954,
+          -0.015140626579523087,
+          -0.0038096620701253414,
+          -0.01768919825553894,
+          0.013383770361542702,
+          -0.022861741483211517,
+          0.011189586482942104,
+          -0.0014024684205651283,
+          0.000858163577504456,
+          -0.004241335671395063,
+          -0.01000578235834837,
+          -0.029059747233986855,
+          -0.007672104984521866,
+          0.003811547067016363,
+          0.04047553986310959,
+          0.047653764486312866,
+          -0.0007059467607177794,
+          0.008761657401919365,
+          0.0011036897776648402,
+          0.012320608831942081,
+          -0.02206248603761196,
+          -0.018443213775753975,
+          0.010133964940905571,
+          0.0034119191113859415,
+          0.018322570249438286,
+          0.005353508051484823,
+          0.04521075263619423,
+          -0.03761028125882149,
+          0.0037681912072002888,
+          0.02813984826207161,
+          0.017870161682367325,
+          -0.006635334342718124,
+          0.0455123595893383,
+          -0.002667329041287303,
+          -0.015118005685508251,
+          -0.004588183015584946,
+          -0.01981552131474018,
+          0.0016041673952713609,
+          -0.0403548963367939,
+          0.014771158806979656,
+          0.00011899303353857249,
+          -6.232407031347975e-05,
+          0.02773268148303032,
+          -0.003076382214203477,
+          0.06930907815694809,
+          0.02446025423705578,
+          -0.0023204819299280643,
+          -0.07292835414409637,
+          0.027431074529886246,
+          0.0247920211404562,
+          0.01482393965125084,
+          0.049644362181425095,
+          0.06182925030589104,
+          -0.016799459233880043,
+          0.02823033183813095,
+          0.012810719199478626,
+          -0.007140524219721556,
+          -0.029994726181030273,
+          0.02352527529001236,
+          -0.04255662113428116,
+          -0.027114387601614,
+          0.046628303825855255,
+          -0.010450650937855244,
+          0.041229553520679474,
+          -0.007977481000125408,
+          0.04726167395710945,
+          0.03507678955793381,
+          -0.007902080193161964,
+          0.022198209539055824,
+          0.005432679783552885,
+          0.03423229232430458,
+          0.00433935783803463,
+          -0.007438360247761011,
+          -0.007193305529654026,
+          -0.03655466064810753,
+          -0.005575942806899548,
+          -0.024203889071941376,
+          0.033357635140419006,
+          0.006936940364539623,
+          -0.0062696365639567375,
+          -0.049855489283800125,
+          0.009251766838133335,
+          -0.014537413604557514,
+          0.01063915528357029,
+          -0.0010669315233826637,
+          0.051242876797914505,
+          -0.008957700803875923,
+          0.00243546930141747,
+          -0.021263230592012405,
+          0.002537261229008436,
+          -0.012177346274256706,
+          -0.006503381300717592,
+          0.014303669333457947,
+          -0.03100510686635971,
+          0.023600677028298378,
+          -0.006578783039003611,
+          -0.0004493459709919989,
+          -0.010986002162098885,
+          0.0029481996316462755,
+          0.040415216237306595,
+          0.015374370850622654,
+          0.017387591302394867,
+          -0.012426171451807022,
+          -0.004840777721256018,
+          0.01446201279759407,
+          -0.00019792901002801955,
+          0.009349789470434189,
+          0.007743736729025841,
+          -0.018533695489168167,
+          0.01826224848628044,
+          -0.020328251644968987,
+          0.014673137106001377,
+          0.0023921134416013956,
+          -0.006541082169860601,
+          -0.042948707938194275,
+          0.0064166695810854435,
+          -0.01943851262331009,
+          0.008188605308532715,
+          -0.002505215583369136,
+          -0.035559359937906265,
+          0.014469552785158157,
+          -0.03426245227456093,
+          0.009176366031169891,
+          -0.0007106593693606555,
+          0.011423330754041672,
+          -0.017719358205795288,
+          0.005127303767949343,
+          -0.0004592424083966762,
+          -0.01776459999382496,
+          0.01362505555152893,
+          -0.038002368062734604,
+          0.0039095692336559296,
+          0.016724057495594025,
+          -0.015743838623166084,
+          0.04246613755822182,
+          0.017824919894337654,
+          -0.03468470275402069,
+          0.00354764168150723,
+          -0.04116923362016678,
+          0.05814965441823006,
+          0.004049061797559261,
+          -0.005813457537442446,
+          0.020735418424010277,
+          -0.01219242624938488,
+          -0.024354692548513412,
+          0.020976703613996506,
+          0.0028464074712246656,
+          0.00209427741356194,
+          -0.027958884835243225,
+          -0.016347050666809082,
+          -0.043401118367910385,
+          -0.04892050847411156,
+          -0.004369518253952265,
+          0.04611557349562645,
+          -0.03272426128387451,
+          -0.022288691252470016,
+          -0.01286350004374981,
+          0.02528967149555683,
+          0.013361150398850441,
+          -0.007585393264889717,
+          0.030326493084430695,
+          0.02560635842382908,
+          -0.00020405538089107722,
+          -0.008761657401919365,
+          0.019906003028154373,
+          -0.04602508991956711,
+          0.002431699074804783,
+          -0.0633372813463211,
+          -0.010307388380169868,
+          -0.004475080408155918,
+          -0.03827381506562233,
+          -0.002079196972772479,
+          -0.0005141441361047328,
+          0.0239324439316988,
+          -0.02617940865457058,
+          -0.019091665744781494,
+          -0.0003159795014653355,
+          0.021006865426898003,
+          -0.019906003028154373,
+          -0.03010028786957264,
+          0.004007590934634209,
+          -0.03317667171359062,
+          -0.006620253901928663,
+          -0.052962031215429306,
+          -0.005176314618438482,
+          -0.009734337218105793,
+          0.01912182755768299,
+          -0.01895594410598278,
+          -0.000523569353390485,
+          0.00784175843000412,
+          0.02225852943956852,
+          0.035891126841306686,
+          -0.03896750882267952,
+          -0.007574083283543587,
+          -0.006273407023400068,
+          -0.017176467925310135,
+          0.02835097350180149,
+          -0.01207178458571434,
+          -0.008980321697890759,
+          -0.003890718799084425,
+          0.011363009922206402,
+          -0.046960070729255676,
+          0.0005061327246949077,
+          -0.013949282467365265,
+          -0.00020111000048927963,
+          0.01545731257647276,
+          0.04370272532105446,
+          -0.038424618542194366,
+          -0.0023751480039209127,
+          -0.04578380659222603,
+          -0.02859225869178772,
+          -0.038907185196876526,
+          -0.030763821676373482,
+          -0.023178428411483765,
+          -0.014575114473700523,
+          0.017538394778966904,
+          0.004173474386334419,
+          -0.021187828853726387,
+          0.009153745137155056,
+          -0.020901301875710487,
+          0.06828361749649048,
+          -0.007562772836536169,
+          0.015336669981479645,
+          0.001535363495349884,
+          -0.024580897763371468,
+          -0.009915300644934177,
+          -0.00974187720566988,
+          0.02385704219341278,
+          -0.007442130707204342,
+          0.0009670245344750583,
+          -0.0019246239680796862,
+          0.02560635842382908,
+          -0.0013713652733713388,
+          -0.005428909789770842,
+          -0.03721819445490837,
+          -0.0062168557196855545,
+          -0.010118884965777397,
+          0.030733661726117134,
+          -0.036132410168647766,
+          -7.999630179256201e-05,
+          -0.011468571610748768,
+          -0.0461457334458828,
+          0.00017471947649028152,
+          -0.04219469428062439,
+          0.003749340772628784,
+          -0.0007229121401906013,
+          -0.021534675732254982,
+          -0.022077566012740135,
+          0.01723678782582283,
+          -0.026858022436499596,
+          -0.022771259769797325,
+          0.006857768632471561,
+          0.0017804185627028346,
+          0.010020862333476543,
+          4.367986912257038e-05,
+          0.030055047944188118,
+          -0.043521761894226074,
+          -0.023133188486099243,
+          0.02761203795671463,
+          -0.022786341607570648,
+          -0.002165908692404628,
+          -0.04219469428062439,
+          -0.007347878534346819,
+          -0.040053289383649826,
+          0.020614776760339737,
+          0.02438485249876976,
+          -0.006861538626253605,
+          -0.0181416068226099,
+          0.026933424174785614,
+          -0.0037248353473842144,
+          -0.05555584281682968,
+          0.01296906266361475,
+          0.0052894167602062225,
+          0.022891903296113014,
+          -0.008279087021946907,
+          -0.024475334212183952,
+          -0.040626343339681625,
+          0.004689974710345268,
+          -0.01695026271045208,
+          -0.04919195547699928,
+          0.0204639732837677,
+          -0.032211530953645706,
+          -0.016573255881667137,
+          -0.007834218442440033,
+          0.0031461287289857864,
+          -0.002904843771830201,
+          0.019921083003282547,
+          0.001642810762859881,
+          -0.009847438894212246,
+          -0.03501646965742111,
+          -0.021474353969097137,
+          -0.03365924209356308,
+          -0.021745799109339714,
+          0.006529772188514471,
+          0.005892629269510508,
+          -0.014311209321022034,
+          0.04267726466059685,
+          -0.015487473458051682,
+          0.034775182604789734,
+          -0.0016041673952713609,
+          -0.016467692330479622,
+          0.05468118563294411,
+          0.012923821806907654,
+          -0.00233367714099586,
+          0.009937920607626438,
+          -0.00265036360360682,
+          0.0019472443964332342,
+          0.012637295760214329,
+          -0.004316737409681082,
+          0.031638480722904205,
+          -0.0005683390190824866,
+          0.009953001514077187,
+          -0.02708422765135765,
+          -0.013104785233736038,
+          -0.011574134230613708,
+          -0.040173932909965515,
+          0.02875814214348793,
+          -0.008037802763283253,
+          -0.002033956116065383,
+          0.018126526847481728,
+          0.012667456641793251,
+          -0.010948301292955875,
+          -0.03374972194433212,
+          -0.015291429124772549,
+          -0.00853545218706131,
+          -0.007879459299147129,
+          -0.012275367975234985,
+          -0.04457738250494003,
+          0.008550533093512058,
+          -0.0535954050719738,
+          -0.01996632292866707,
+          -0.05443990230560303,
+          -0.008867219090461731,
+          0.03038681484758854,
+          -4.72732208436355e-05,
+          0.012916280888020992,
+          0.01054113358259201,
+          0.0054062893614172935,
+          -0.03646417707204819,
+          -0.013677836395800114,
+          0.024897582828998566,
+          0.02136879228055477,
+          -0.01472591795027256,
+          -0.038092851638793945,
+          0.017432833090424538,
+          -0.01666373759508133,
+          -0.047925207763910294,
+          -0.011815418489277363,
+          0.011430870741605759,
+          -0.012554354034364223,
+          -0.060803789645433426,
+          -0.01764395646750927,
+          -0.018563855439424515,
+          -0.008610853925347328,
+          0.028893863782286644,
+          0.020403653383255005,
+          -0.004795536864548922,
+          -0.03679594397544861,
+          0.0018567625666037202,
+          0.015653356909751892,
+          0.006435520015656948,
+          0.032000407576560974,
+          -0.008550533093512058,
+          0.03803252801299095,
+          0.011355469934642315,
+          0.02589288353919983,
+          0.014213187620043755,
+          0.0016013398999348283,
+          0.0071970755234360695,
+          -0.01568351686000824,
+          -0.01362505555152893,
+          -0.011174505576491356,
+          -0.028320813551545143,
+          -0.009349789470434189,
+          -0.018292410299181938,
+          0.02124815061688423,
+          -0.0316988006234169,
+          0.04141051694750786,
+          0.01854877546429634,
+          -0.0007818195736035705,
+          0.010412951000034809,
+          0.001439226558431983,
+          0.00450524128973484,
+          0.007555232848972082,
+          -0.0016211328329518437,
+          0.025018226355314255,
+          0.023721320554614067,
+          -0.034051328897476196,
+          -0.013806018978357315,
+          0.014235807582736015,
+          -0.010103804059326649,
+          0.016889940947294235,
+          -0.005165004637092352,
+          -0.005662654526531696,
+          0.03160832077264786,
+          -0.024852342903614044,
+          0.02103702537715435,
+          0.003636238630861044,
+          -0.014243348501622677,
+          -0.02495790459215641,
+          0.0059944214299321175,
+          -0.023540355265140533,
+          0.04406465217471123,
+          -0.03531807288527489,
+          0.02756679803133011,
+          0.004497700836509466,
+          -0.015351750887930393,
+          0.03059793822467327,
+          -0.011166965588927269,
+          -0.017342351377010345,
+          -0.003223415231332183,
+          0.030160609632730484,
+          -0.010827658697962761,
+          -0.023600677028298378,
+          0.018699578940868378,
+          -0.007121674250811338,
+          0.027219951152801514,
+          -0.01413024589419365,
+          -0.0025466864462941885,
+          0.03332747519016266,
+          0.006303567439317703,
+          -0.0016456382581964135,
+          0.009266847744584084,
+          0.012086864560842514,
+          0.031186070293188095,
+          0.02969312109053135,
+          -0.025274591520428658,
+          -0.013361150398850441,
+          0.04032473638653755,
+          0.010978462174534798,
+          -0.002395883435383439,
+          0.012184886261820793,
+          -0.0032498056534677744,
+          -0.013594894669950008,
+          0.002346872352063656,
+          0.0007521302322857082,
+          -0.012109484523534775,
+          0.006420440040528774,
+          -0.028154930099844933,
+          -0.0017398901982232928,
+          0.01883530057966709,
+          0.013519493862986565,
+          -0.005915249697864056,
+          -0.013557193800807,
+          0.01525372825562954,
+          -0.01984568126499653,
+          0.024158649146556854,
+          -0.013821099884808064,
+          0.006582553032785654,
+          0.01425842847675085,
+          -0.010239526629447937,
+          -0.03281474485993385,
+          0.00847513135522604,
+          -3.4195771149825305e-05,
+          -0.009493052028119564,
+          0.003455274971202016,
+          -0.009794658049941063,
+          -0.008950160816311836,
+          -0.0032611158676445484,
+          0.011513812467455864,
+          -0.02940659411251545,
+          -0.0002133627567673102,
+          0.027762841433286667,
+          0.024988064542412758,
+          -0.013542113825678825,
+          -0.005741826258599758,
+          -0.010759797878563404,
+          0.009681555442512035,
+          -0.004282806534320116,
+          -0.009583533741533756,
+          0.027913644909858704,
+          0.02185136266052723,
+          -0.03212105110287666,
+          0.006420440040528774,
+          -0.005485461093485355,
+          0.0019547846168279648,
+          0.004999121185392141,
+          0.015238648280501366,
+          -0.018729738891124725,
+          0.00609621312469244,
+          -0.006454370450228453,
+          0.013994523324072361,
+          -0.014371531084179878,
+          0.023706238716840744,
+          0.017915401607751846,
+          0.009349789470434189,
+          -0.01482393965125084,
+          -0.006503381300717592,
+          0.010390330106019974,
+          -0.001478812424466014,
+          -0.010556213557720184,
+          0.035800643265247345,
+          0.00704627251252532,
+          -0.0011885165004059672,
+          0.01562319602817297,
+          0.028969265520572662,
+          0.02316334843635559,
+          -0.003754995996132493,
+          0.011853119358420372,
+          0.06774073094129562,
+          -0.018247168511152267,
+          0.005658884532749653,
+          0.0005353508167900145,
+          0.020041724666953087,
+          0.006130144000053406,
+          -0.032000407576560974,
+          -0.034051328897476196,
+          0.0001823774364311248,
+          -0.026073846966028214,
+          -0.04735215753316879,
+          -0.014047304168343544,
+          -0.01809636689722538,
+          -0.03172896057367325,
+          -0.0042639560997486115,
+          -0.00963631458580494,
+          0.005787067115306854,
+          0.023600677028298378,
+          0.004113153088837862,
+          -0.027385834604501724,
+          -0.016678817570209503,
+          0.02699374593794346,
+          0.025274591520428658,
+          0.03736899420619011,
+          -0.005681504961103201,
+          0.011174505576491356,
+          0.022529976442456245,
+          0.002050921553745866,
+          0.00927438773214817,
+          -0.004026441369205713,
+          -0.008052882738411427,
+          0.027174709364771843,
+          0.02703898772597313,
+          0.029270872473716736,
+          -0.0011461031390354037,
+          -0.034775182604789734,
+          -0.015298969112336636,
+          -0.018081285059452057,
+          -0.0006102810730226338,
+          0.008693795651197433,
+          -0.009033102542161942,
+          0.018352732062339783,
+          -0.0004740870790556073,
+          0.018518613651394844,
+          0.025591278448700905,
+          0.0022865512873977423,
+          -0.022394252941012383,
+          0.010495892725884914,
+          -0.0009608981781639159,
+          0.0071669151075184345,
+          -0.029843922704458237,
+          -0.04125971347093582,
+          -0.056430500000715256,
+          0.007068892940878868,
+          0.03254329785704613,
+          -0.042375657707452774,
+          0.01609068550169468,
+          0.04741247743368149,
+          -0.009018022567033768,
+          -0.023676078766584396,
+          -0.015864480286836624,
+          0.005410059355199337,
+          -0.022741099819540977,
+          -0.018307490274310112,
+          0.037067390978336334,
+          -0.015502553433179855,
+          -0.017870161682367325,
+          0.02079574018716812,
+          -0.02295222505927086,
+          0.02041873335838318,
+          -0.02818509005010128,
+          -0.002237540204077959,
+          -0.03311634808778763,
+          0.04014377295970917,
+          -0.012260288000106812,
+          0.00731017766520381,
+          -0.013097245246171951,
+          0.044456738978624344,
+          -0.008083043619990349,
+          0.017101066187024117,
+          0.0012997337616980076,
+          -0.001241297577507794,
+          -0.00830924790352583,
+          0.004893559031188488,
+          0.02838113345205784,
+          -0.017598716542124748,
+          0.0069030094891786575,
+          0.011264988221228123,
+          -0.03136703372001648,
+          -0.0187448188662529,
+          0.0037870416417717934,
+          0.021263230592012405,
+          0.015314050018787384,
+          0.04394400864839554,
+          0.028562096878886223,
+          -0.009930380620062351,
+          -0.008346948772668839,
+          0.009756957180798054,
+          -0.031276553869247437,
+          0.03709755092859268,
+          -0.013579814694821835,
+          0.031759120523929596,
+          0.028622418642044067,
+          -0.011264988221228123,
+          -0.001501432852819562,
+          -0.0027540407609194517,
+          0.02058461681008339,
+          0.03803252801299095,
+          0.007528842426836491,
+          0.005142384208738804,
+          0.02560635842382908,
+          0.010910600423812866,
+          -0.02626989036798477,
+          0.01826224848628044,
+          -0.004509011283516884,
+          -0.004780456889420748,
+          0.025259511545300484,
+          0.014265968464314938,
+          0.01409254502505064,
+          0.0017427176935598254,
+          0.028848623856902122,
+          -0.019875841215252876,
+          0.010118884965777397,
+          0.002904843771830201,
+          -0.012697616592049599,
+          0.015314050018787384,
+          0.0068276082165539265,
+          0.021263230592012405,
+          -0.004618343431502581,
+          0.02985900454223156,
+          0.01768919825553894,
+          0.01649785414338112,
+          0.0386357419192791,
+          0.006024581845849752,
+          -0.049041152000427246,
+          0.012049163691699505,
+          0.017176467925310135,
+          -0.0031423585023730993,
+          -0.021655317395925522,
+          -0.004041521809995174,
+          -0.002746500540524721,
+          0.007924700155854225,
+          0.0039057990070432425,
+          -0.01568351686000824,
+          -0.017824919894337654,
+          -0.004324277397245169,
+          0.010707016102969646,
+          0.021308470517396927,
+          0.010164125822484493,
+          0.005930330138653517,
+          0.006375198718160391,
+          0.040505699813365936,
+          0.02708422765135765,
+          -0.02495790459215641,
+          0.033387795090675354,
+          0.011543973349034786,
+          -0.015110465697944164,
+          -0.015223568305373192,
+          -0.022514894604682922,
+          0.0028690281324088573,
+          -0.02838113345205784,
+          -0.004659814294427633,
+          0.01883530057966709,
+          0.02209264598786831,
+          -0.013542113825678825,
+          -0.002821902045980096,
+          0.024701539427042007,
+          0.0009255537297576666,
+          -0.011363009922206402,
+          -0.021459273993968964,
+          0.011016163043677807,
+          0.0067785969004035,
+          -4.6360157284652814e-05,
+          0.008821978233754635,
+          0.04922211542725563,
+          -0.0055834827944636345,
+          0.027958884835243225,
+          0.049734845757484436,
+          0.0027917413972318172,
+          -0.02871290035545826,
+          0.02225852943956852,
+          0.007359188981354237,
+          -0.004169704392552376,
+          0.0221831277012825,
+          -0.005225325468927622,
+          -0.04819665476679802,
+          0.022861741483211517,
+          -0.028109688311815262,
+          -0.006005731411278248,
+          0.009289467707276344,
+          0.011385629884898663,
+          -0.002250735415145755,
+          -0.026692138984799385,
+          0.036253053694963455,
+          -0.006620253901928663,
+          0.015223568305373192,
+          -0.023374471813440323,
+          -0.027008825913071632,
+          -0.014974743127822876,
+          -0.007687185425311327,
+          -0.009749417193233967,
+          0.0031348182819783688,
+          -0.00901048257946968,
+          0.016241488978266716,
+          -0.03619273379445076,
+          0.003183829365298152,
+          0.012418631464242935,
+          0.00450524128973484,
+          0.04762360453605652,
+          0.00959861371666193,
+          0.00757785327732563,
+          -0.005915249697864056,
+          -0.012162266299128532,
+          0.023540355265140533,
+          0.010933221317827702,
+          0.03516726940870285,
+          -0.004399679135531187,
+          0.007223465945571661,
+          0.030447136610746384,
+          0.024927744641900063,
+          0.02201724611222744,
+          -0.016844701021909714,
+          -0.004181014373898506,
+          -0.00476914644241333,
+          -0.014590195380151272,
+          0.011822959408164024,
+          -0.005805917549878359,
+          0.008987861685454845,
+          0.012380930595099926,
+          0.007434590253978968,
+          0.011581674218177795,
+          0.06249278411269188,
+          0.015879560261964798,
+          -0.00888229999691248,
+          -0.026481015607714653,
+          -0.0032498056534677744,
+          0.007657025009393692,
+          0.00959861371666193,
+          0.0019736348185688257,
+          -0.009794658049941063,
+          0.014265968464314938,
+          0.00824892707169056,
+          -0.03402116894721985,
+          0.002678639255464077,
+          -0.004015131387859583,
+          0.006130144000053406,
+          0.01492950227111578,
+          -0.0403548963367939,
+          -0.00259381253272295,
+          -0.020720338448882103,
+          -0.004799306858330965,
+          0.022620458155870438,
+          -0.010412951000034809,
+          0.005979340989142656,
+          0.0016503508668392897,
+          -0.028803382068872452,
+          0.003958580084145069,
+          -0.0054364497773349285,
+          0.02777792140841484,
+          -0.028079528361558914,
+          -0.008821978233754635,
+          -0.013677836395800114,
+          -0.007600473705679178,
+          -0.017417753115296364,
+          0.029180390760302544,
+          -0.019589316099882126,
+          -0.002714454894885421,
+          -0.031155910342931747,
+          -0.014884261414408684,
+          -0.01476361881941557,
+          0.026375453919172287,
+          -0.01308970432728529,
+          -0.04481866583228111,
+          0.008482671342790127,
+          -0.012441251426935196,
+          0.00222057499922812,
+          -0.026194490492343903,
+          0.0030914624221622944,
+          0.028652578592300415,
+          0.02058461681008339,
+          0.006507151760160923,
+          -0.028411295264959335,
+          -0.010412951000034809,
+          0.01817176677286625,
+          0.04620605334639549,
+          -0.020765580236911774,
+          0.0128257991746068,
+          -0.012252748012542725,
+          0.01069947611540556,
+          -0.044004328548908234,
+          -0.0042337956838309765,
+          0.001988715259358287,
+          0.021821200847625732,
+          -0.015412071719765663,
+          0.002610777737572789,
+          0.05253978073596954,
+          0.017794759944081306,
+          -0.05163496360182762,
+          0.01866941712796688,
+          -0.012019002810120583,
+          0.034533899277448654,
+          0.05534471943974495,
+          -0.004535401705652475,
+          0.006936940364539623,
+          0.020765580236911774,
+          -0.022032326087355614,
+          0.010450650937855244,
+          0.014876720495522022,
+          -0.00576821668073535,
+          -0.02259029634296894,
+          -0.013579814694821835,
+          0.021640237420797348,
+          0.012592054903507233,
+          -0.012305528856813908,
+          0.03872622177004814,
+          -0.0007318660500459373,
+          0.029587557539343834,
+          -0.01678437925875187,
+          -0.03166864067316055,
+          -0.002508985809981823,
+          -0.018081285059452057,
+          0.025274591520428658,
+          0.010390330106019974,
+          0.0015985122881829739,
+          0.006699425633996725,
+          -0.035921286791563034,
+          0.01857893541455269,
+          0.02503330633044243,
+          0.0003843121521640569,
+          0.001393043203279376,
+          -0.02541031315922737,
+          0.02168547920882702,
+          -0.004761606454849243,
+          0.018186848610639572,
+          0.022605376318097115,
+          -0.0027955116238445044,
+          -0.019167067483067513,
+          0.005836077965795994,
+          0.0012083094334229827,
+          0.04346143826842308,
+          -0.001625845325179398,
+          0.0026239731814712286,
+          -0.01826224848628044,
+          -0.0032724260818213224,
+          -0.019001184031367302,
+          0.02572700008749962,
+          -0.005545781925320625,
+          -0.017432833090424538,
+          -0.018352732062339783,
+          0.03136703372001648,
+          -0.0007634404464624822,
+          -0.032332174479961395,
+          -0.003114083083346486,
+          -0.012433711439371109,
+          -0.0018379121320322156,
+          0.018639257177710533,
+          -0.03157815709710121,
+          -0.017251869663596153,
+          -0.0017493154155090451,
+          0.022077566012740135,
+          0.026948504149913788,
+          0.01883530057966709,
+          -0.0024712849408388138,
+          -0.0001750729134073481,
+          0.011076483875513077,
+          -0.0004952937597408891,
+          0.011506272479891777,
+          0.01299168262630701,
+          -0.007370498962700367,
+          0.03287506476044655,
+          -0.006367658730596304,
+          -0.0036173881962895393,
+          0.023887204006314278,
+          -0.01063915528357029,
+          -0.008731496520340443,
+          0.018654337152838707,
+          -0.018352732062339783,
+          -0.015502553433179855,
+          -0.007804058026522398,
+          0.005662654526531696,
+          -0.004886019043624401,
+          -0.0015080304583534598,
+          3.1014766136649996e-05,
+          0.03724835440516472,
+          -0.027265191078186035,
+          0.03178928419947624,
+          0.043220154941082,
+          -0.009847438894212246,
+          -0.02863749861717224,
+          -0.0016013398999348283,
+          -0.003496745601296425,
+          0.02455073595046997,
+          0.006631563883274794,
+          -0.0006592920981347561,
+          -0.04195340722799301,
+          0.015954962000250816,
+          -0.016226409003138542,
+          -0.011679695919156075,
+          -0.006024581845849752,
+          -0.004791766870766878,
+          0.009410110302269459,
+          0.028154930099844933,
+          -0.02801920659840107,
+          0.01190590113401413,
+          0.012418631464242935,
+          0.024083247408270836,
+          -0.005342198070138693,
+          -0.01409254502505064,
+          -0.01103124301880598,
+          -0.011287608183920383,
+          0.012531733140349388,
+          -0.0164224524050951,
+          -0.005319577641785145,
+          0.00519893504679203,
+          -0.005327117629349232,
+          0.01100862305611372,
+          -0.016920102760195732,
+          -0.06375952810049057,
+          -0.011114184744656086,
+          0.03667530044913292,
+          -0.014801319688558578,
+          -0.018880542367696762,
+          0.03712771087884903,
+          0.009545832872390747,
+          0.021640237420797348,
+          -0.023947523906826973,
+          0.0155628751963377,
+          -0.006767286919057369,
+          0.0029953254852443933,
+          -0.019785359501838684,
+          -0.010955841280519962,
+          0.00870133563876152,
+          -0.004429839551448822,
+          -0.004143313970416784,
+          -0.024249130859971046,
+          0.02617940865457058,
+          0.008264007046818733,
+          -0.02271093986928463,
+          0.007592933718115091,
+          0.008656094782054424,
+          0.005376128945499659,
+          -0.01604544371366501,
+          0.013572274707257748,
+          0.011822959408164024,
+          -0.01345917209982872,
+          0.02275617979466915,
+          -0.025063466280698776,
+          -0.004486390855163336,
+          0.013775859028100967,
+          0.030447136610746384,
+          0.014454471878707409,
+          -0.021670397371053696,
+          -0.02148943394422531,
+          0.011242367327213287,
+          0.0015909721842035651,
+          0.0034119191113859415,
+          0.000781348324380815,
+          -0.00709528336301446,
+          -0.04451705887913704,
+          0.01984568126499653,
+          -0.0020660017617046833,
+          -0.015035063959658146,
+          0.00412446353584528,
+          0.015909722074866295,
+          0.013609975576400757,
+          0.007487371563911438,
+          -0.006371428724378347,
+          0.01096338126808405,
+          0.033629078418016434,
+          -0.019001184031367302,
+          -0.022680778056383133,
+          -0.0063638887368142605,
+          -0.0010094379540532827,
+          0.01838289201259613,
+          -0.010360169224441051,
+          0.004395908676087856,
+          0.01809636689722538,
+          0.07220450043678284,
+          -0.014710837043821812,
+          0.00022785397595725954,
+          -0.03206072747707367,
+          -0.02340463362634182,
+          0.00108766695484519,
+          -0.0036569740623235703,
+          0.022198209539055824,
+          0.021293390542268753,
+          0.018066205084323883,
+          0.010691936127841473,
+          -0.007234776392579079,
+          0.002895418554544449,
+          0.025395233184099197,
+          -0.013353610411286354,
+          0.030809063464403152,
+          0.007909620180726051,
+          0.016316890716552734,
+          0.001337434514425695,
+          -0.015080304816365242,
+          0.012509113177657127,
+          0.033267151564359665,
+          -0.010857819579541683,
+          0.002940659411251545,
+          -0.004331817384809256,
+          0.0099454615265131,
+          -0.00028275573276914656,
+          -0.008158445358276367,
+          -0.009206525981426239,
+          -0.023057786747813225,
+          0.014115165919065475,
+          -0.014205647632479668,
+          -0.004735215567052364,
+          0.005263026338070631,
+          -0.016105765476822853,
+          0.062191177159547806,
+          -0.008769197389483452,
+          0.008324328809976578,
+          0.0077361962758004665,
+          -0.008497752249240875,
+          -0.005425139795988798,
+          0.008173525333404541,
+          0.005968030542135239,
+          0.011619375087320805,
+          -0.005149924196302891,
+          0.0037813864182680845,
+          -0.0024335843045264482,
+          -0.011189586482942104,
+          -0.00015857882681302726,
+          -0.023133188486099243,
+          0.016392290592193604,
+          0.006484531331807375,
+          -0.023615757003426552,
+          -0.0007064180099405348,
+          -0.0049425698816776276,
+          -0.00741196982562542,
+          0.002096162410452962,
+          -0.017251869663596153,
+          -0.0034119191113859415,
+          0.014612815342843533,
+          -0.004799306858330965,
+          -0.003898258786648512,
+          0.008075503632426262,
+          0.013127405196428299,
+          -0.008135824464261532,
+          -0.009183906018733978,
+          0.02156483568251133,
+          0.003148013725876808,
+          0.0034816653933376074,
+          0.006156534422188997,
+          -0.009726796299219131,
+          0.024988064542412758,
+          -0.01915198750793934,
+          -0.01435645017772913,
+          0.0050669824704527855,
+          0.010020862333476543,
+          0.019257549196481705,
+          -0.020328251644968987,
+          -0.003347827820107341,
+          -0.010940761305391788,
+          -0.008791817352175713,
+          0.022891903296113014,
+          -0.002870913129299879,
+          -0.023555437102913857,
+          -0.016482774168252945,
+          0.0025184110272675753,
+          -0.01833765022456646,
+          -0.005379898939281702,
+          -0.008339408785104752,
+          -0.005866238847374916,
+          -0.02103702537715435,
+          -0.018563855439424515,
+          0.0033384026028215885,
+          -0.029964566230773926,
+          -0.0036456636153161526,
+          -0.011046322993934155,
+          -0.0005975571111775935,
+          0.0076834154315292835,
+          -0.04578380659222603,
+          0.0005622126045636833,
+          -0.035227593034505844,
+          0.002356297569349408,
+          0.008656094782054424,
+          0.005304497200995684,
+          -0.013783399015665054,
+          0.020071886479854584,
+          0.017372511327266693,
+          -0.01981552131474018,
+          0.015713678672909737,
+          0.005334658082574606,
+          0.00018390902550891042,
+          0.0037983518559485674,
+          0.007672104984521866,
+          -0.007385579403489828,
+          0.0072272359393537045,
+          0.010186745785176754,
+          0.0028407524805516005,
+          -0.017025664448738098,
+          -0.012652375735342503,
+          -0.019167067483067513,
+          -0.008573153056204319,
+          0.006853998638689518,
+          0.021142587065696716,
+          -0.00959861371666193,
+          0.004901099018752575,
+          0.00671827606856823,
+          0.008625934831798077,
+          -0.016271648928523064,
+          0.01972503960132599,
+          -0.009900220669806004,
+          -0.0018906932091340423,
+          -0.018865462392568588,
+          -0.000623005093075335,
+          -0.00036051354254595935,
+          0.01372307725250721,
+          -0.009221606887876987,
+          -0.00444114999845624,
+          -0.0037531109992414713,
+          0.006337498314678669,
+          -0.010940761305391788,
+          -0.0013581699458882213,
+          0.00020865016267634928,
+          0.0267977025359869,
+          0.014363990165293217,
+          0.0008280029869638383,
+          -0.03966120257973671,
+          -0.027989046648144722,
+          0.025274591520428658,
+          0.00513484375551343,
+          0.012953981757164001,
+          -0.02409832738339901,
+          -0.018081285059452057,
+          0.0328449048101902,
+          0.0012808834435418248,
+          -0.000869473849888891,
+          -0.034986305981874466,
+          0.016603415831923485,
+          -0.015577955171465874,
+          0.02875814214348793,
+          0.025259511545300484,
+          -0.014801319688558578,
+          0.027702519670128822,
+          0.005391208920627832,
+          -0.022967305034399033,
+          0.008324328809976578,
+          0.006963330786675215,
+          0.019875841215252876,
+          -0.01515570655465126,
+          0.01601528376340866,
+          0.003886948572471738,
+          -0.017342351377010345,
+          -0.007502451539039612,
+          0.014205647632479668,
+          0.02756679803133011,
+          0.005802147556096315,
+          0.005485461093485355,
+          0.001331779407337308,
+          -0.036494337022304535,
+          0.015638276934623718,
+          -0.0019415892893448472,
+          0.01345917209982872,
+          -0.01637721061706543,
+          0.004181014373898506,
+          0.007924700155854225,
+          -0.015170786529779434,
+          -0.013270668685436249,
+          -0.029346274212002754,
+          0.005519391503185034,
+          -0.009116044268012047,
+          0.0020245308987796307,
+          -0.0033723332453519106,
+          0.0071669151075184345,
+          0.020569536834955215,
+          -0.014333830215036869,
+          -0.0035646071191877127,
+          0.005112223327159882,
+          0.008407269604504108,
+          -0.001880325493402779,
+          0.002735190326347947,
+          0.002011335687711835,
+          -0.00020111000048927963,
+          0.02835097350180149,
+          -0.016000203788280487,
+          -0.013134945183992386,
+          0.004101843107491732,
+          -0.00937240943312645,
+          0.002397768432274461,
+          -0.00646568089723587,
+          0.009628774598240852,
+          -0.0328449048101902,
+          -0.0018595900619402528,
+          -0.014409231022000313,
+          -0.021881522610783577,
+          -0.0013072739820927382,
+          0.008927540853619576,
+          0.009485512040555477,
+          -0.02099178545176983,
+          0.008422350510954857,
+          0.020433813333511353,
+          0.007702265866100788,
+          0.00867871567606926,
+          0.02299746498465538,
+          0.027793001383543015,
+          0.010028403252363205,
+          -0.008814438246190548,
+          -0.016000203788280487,
+          0.019830601289868355,
+          -0.015909722074866295,
+          0.013308369554579258,
+          0.00810566358268261,
+          0.004361978266388178,
+          0.03002488799393177,
+          0.018518613651394844,
+          0.00933470856398344,
+          0.011792798526585102,
+          0.004591953009366989,
+          0.01425842847675085,
+          -0.0042111752554774284,
+          0.01482393965125084,
+          0.00023786825477145612,
+          -0.00845251139253378,
+          0.00014538357208948582,
+          -0.0051876250654459,
+          0.026601657271385193,
+          -0.011755097657442093,
+          -0.00017742921772878617,
+          -0.011355469934642315,
+          -0.027340592816472054,
+          -0.010571293532848358,
+          0.005477920640259981,
+          -0.024249130859971046,
+          0.004637193866074085,
+          0.004486390855163336,
+          -0.01296906266361475,
+          -0.016482774168252945,
+          -0.015306509099900723,
+          0.010925681330263615,
+          -0.0006616484024561942,
+          0.008919999934732914,
+          0.0057003553956747055,
+          -0.011023703031241894,
+          0.003400608664378524,
+          0.020162368193268776,
+          0.021640237420797348,
+          0.0011074598878622055,
+          0.02646593563258648,
+          0.0013864454813301563,
+          -0.010616534389555454,
+          -0.0071669151075184345,
+          0.0017643957398831844,
+          -0.007962401024997234,
+          -0.005168774630874395,
+          -0.009093424305319786,
+          -0.0035551819019019604,
+          0.020252849906682968,
+          -0.004799306858330965,
+          -0.015758918598294258,
+          0.0006687172572128475,
+          -0.01649785414338112,
+          -0.0038831785786896944,
+          -0.008972781710326672,
+          0.04044537618756294,
+          -0.021459273993968964,
+          -0.007547692861407995,
+          -0.02981376275420189,
+          0.0024769401643425226,
+          -0.0015928571810945868,
+          0.00629979744553566,
+          0.004927489440888166,
+          -0.03402116894721985,
+          -0.05658130347728729,
+          0.027355672791600227,
+          0.002610777737572789,
+          0.01386634074151516,
+          0.022982385009527206,
+          -0.001093322061933577,
+          -0.033960845321416855,
+          0.010887980461120605,
+          -0.008497752249240875,
+          -0.007969941012561321,
+          0.005247946362942457,
+          -0.002974590053781867,
+          0.0007238546386361122,
+          0.02136879228055477,
+          -0.011958681978285313,
+          -0.030824143439531326,
+          -0.013187726959586143,
+          -0.0007653255015611649,
+          0.01425842847675085,
+          -0.004267726093530655,
+          0.03758012130856514,
+          0.01604544371366501,
+          -0.02013220638036728,
+          0.034986305981874466,
+          0.007102823816239834,
+          0.016075605526566505,
+          0.007751276716589928,
+          0.012305528856813908,
+          0.02642069384455681,
+          0.02237917296588421,
+          -0.04065650328993797,
+          0.008301707915961742,
+          -0.01403222419321537,
+          0.0008421407546848059,
+          0.010028403252363205,
+          0.029587557539343834,
+          0.000784647127147764,
+          0.0027615807484835386,
+          0.0011649535736069083,
+          -0.008716416545212269,
+          -0.038213491439819336,
+          0.0007181995315477252,
+          0.021911682561039925,
+          0.02111242711544037,
+          0.03480534255504608,
+          0.012652375735342503,
+          0.008188605308532715,
+          0.029512157663702965,
+          -0.005545781925320625,
+          -0.003053761785849929,
+          -0.014778698794543743,
+          -0.021655317395925522,
+          -0.004049061797559261,
+          -0.008098123595118523,
+          0.012614674866199493,
+          -0.0045731025747954845,
+          0.002378917997702956,
+          0.02177596092224121,
+          0.018774978816509247,
+          -0.0017568555194884539,
+          -0.011822959408164024,
+          0.02304270677268505,
+          -0.0025297212414443493,
+          -0.016151007264852524,
+          -0.009364869445562363,
+          0.015434691682457924,
+          0.03782140463590622,
+          0.0018954058177769184,
+          0.002769120968878269,
+          0.015080304816365242,
+          -0.008218766190111637,
+          0.024113407358527184,
+          -0.02103702537715435,
+          -0.03365924209356308,
+          -0.02340463362634182,
+          0.03854525834321976,
+          0.012901200912892818,
+          -0.008369569666683674,
+          0.011513812467455864,
+          0.004878478590399027,
+          0.03549903631210327,
+          0.0010339433792978525,
+          -0.04113907366991043,
+          0.006318647880107164,
+          0.007321488112211227,
+          -0.004026441369205713,
+          -0.026827862486243248,
+          0.0010782418539747596,
+          -0.02964787930250168,
+          0.017055824398994446,
+          -0.0014712722040712833,
+          0.015374370850622654,
+          0.02115766704082489,
+          0.009379949420690536,
+          0.00980219803750515,
+          0.006665494758635759,
+          -0.0008567498298361897,
+          0.009922840632498264,
+          0.008565613068640232,
+          -0.006737126037478447,
+          0.007087743375450373,
+          0.015713678672909737,
+          0.00824892707169056,
+          0.02364591881632805,
+          0.028215250000357628,
+          0.018729738891124725,
+          -0.014801319688558578,
+          -0.018729738891124725,
+          0.005511851515620947,
+          0.03236233443021774,
+          0.012757938355207443,
+          -0.001835084636695683,
+          -0.03209088742733002,
+          0.011574134230613708,
+          0.030567778274416924,
+          0.021519595757126808,
+          -0.004705055151134729,
+          -0.011566594243049622,
+          -0.024535655975341797,
+          -0.02720486931502819,
+          0.007381809409707785,
+          0.0069030094891786575,
+          -0.008218766190111637,
+          -0.019287709146738052,
+          -0.01149873249232769,
+          -0.019574236124753952,
+          0.03190992400050163,
+          0.021293390542268753,
+          -0.00450524128973484,
+          -0.009591073729097843,
+          0.0011281953193247318,
+          -0.004331817384809256,
+          0.013934201560914516,
+          -0.0003169220290146768,
+          -0.01229798886924982,
+          0.01160429511219263,
+          -0.0139869824051857,
+          -0.009379949420690536,
+          -0.01661849580705166,
+          -0.028577178716659546,
+          -0.004968960303813219,
+          0.016362130641937256,
+          0.008784277364611626,
+          0.011672155931591988,
+          -0.017749518156051636,
+          -0.002957624848932028,
+          0.014446931891143322,
+          0.0067333560436964035,
+          -0.010782417841255665,
+          0.0011187701020389795,
+          -0.0009953001281246543,
+          0.009470431134104729,
+          -0.028170010074973106,
+          0.012124565429985523,
+          -0.02782316319644451,
+          -0.0009792773053050041,
+          0.013572274707257748,
+          -0.0014703297056257725,
+          0.0007945435936562717,
+          -0.015608116053044796,
+          -0.010556213557720184,
+          -0.003159323940053582,
+          -0.008671175688505173,
+          -0.005210245493799448,
+          -0.02390228398144245,
+          0.01609068550169468,
+          -0.005990650970488787,
+          0.011106644757091999,
+          -0.038092851638793945,
+          0.001858647563494742,
+          0.010797498747706413,
+          -0.012237667106091976,
+          -0.024927744641900063,
+          0.003643778618425131,
+          0.040988270193338394,
+          0.0017691082321107388,
+          0.006876619067043066,
+          0.0011640110751613975,
+          0.0008100951090455055,
+          -0.021172748878598213,
+          -0.00022054946748539805,
+          -0.009455351158976555,
+          -0.008731496520340443,
+          -0.0077135758474469185,
+          0.011182046495378017,
+          0.006567472591996193,
+          -0.007393119391053915,
+          0.02299746498465538,
+          -0.0036494338419288397,
+          -0.002514640800654888,
+          0.002282781060785055,
+          0.023178428411483765,
+          -0.00608490314334631,
+          0.02503330633044243,
+          0.006869079079478979,
+          0.06514691561460495,
+          -0.0035947677679359913,
+          -0.005949180107563734,
+          0.0032592308707535267,
+          -0.015819240361452103,
+          0.0011008622823283076,
+          0.011875740252435207,
+          -0.022198209539055824,
+          0.015027523972094059,
+          0.0014175486285239458,
+          -0.010179205797612667,
+          -0.012199967168271542,
+          -0.002771005965769291,
+          0.012486492283642292,
+          0.022047406062483788,
+          0.021338632330298424,
+          0.001983060035854578,
+          0.01609068550169468,
+          0.03172896057367325,
+          0.013225427828729153,
+          -0.011378089897334576,
+          0.0013402621261775494,
+          -0.00019215607608202845,
+          0.017598716542124748,
+          -0.007819138467311859,
+          0.019694877788424492,
+          -0.00296705006621778,
+          0.011883280239999294,
+          0.010118884965777397,
+          0.025561116635799408,
+          0.01895594410598278,
+          0.0010179205564782023,
+          0.00916882511228323,
+          0.008301707915961742,
+          0.017734438180923462,
+          -0.001773820840753615,
+          0.011197126470506191,
+          -0.007280017249286175,
+          0.0012592055136337876,
+          0.019544076174497604,
+          0.02041873335838318,
+          -0.024520576000213623,
+          0.003958580084145069,
+          -0.008218766190111637,
+          -0.0024053086526691914,
+          -0.0005197992431931198,
+          0.0064166695810854435,
+          0.010103804059326649,
+          -0.0061942352913320065,
+          -0.02847161516547203,
+          -0.006122604012489319,
+          -0.004912409465759993,
+          0.005368588492274284,
+          -0.02263553813099861,
+          -0.013519493862986565,
+          0.00188126799184829,
+          0.01106894388794899,
+          -0.007114133797585964,
+          -0.0011592984665185213,
+          0.002567421877756715,
+          0.021534675732254982,
+          0.011528893373906612,
+          -0.0068426881916821,
+          -0.008799358271062374,
+          -0.001767223235219717,
+          0.006963330786675215,
+          0.022605376318097115,
+          0.002514640800654888,
+          0.012712697498500347,
+          0.014597735367715359,
+          0.006518461741507053,
+          -0.009658935479819775,
+          0.01446201279759407,
+          0.0005636264104396105,
+          0.001902946038171649,
+          0.0005537299439311028,
+          -0.0008228191290982068,
+          0.004784226883202791,
+          0.00927438773214817,
+          0.01972503960132599,
+          0.0071970755234360695,
+          0.00015186337986961007,
+          0.0028765681199729443,
+          0.014100085012614727,
+          -0.016105765476822853,
+          -0.0037926966324448586,
+          -0.0012771133333444595,
+          0.012056703679263592,
+          0.007042502518743277,
+          -0.0021489434875547886,
+          0.004561792127788067,
+          0.0123960105702281,
+          0.006503381300717592,
+          -0.008791817352175713,
+          -0.016407372429966927,
+          0.005643804091960192,
+          -0.007449670694768429,
+          -0.009206525981426239,
+          -0.007717345841228962,
+          0.009553372859954834,
+          0.005681504961103201,
+          0.010307388380169868,
+          -0.019694877788424492,
+          0.007528842426836491,
+          -0.0021998395677655935,
+          -0.009930380620062351,
+          -0.003935959655791521,
+          0.010737176984548569,
+          0.019860761240124702,
+          -0.0017945562722161412,
+          0.011709856800734997,
+          -0.01606052555143833,
+          0.0024222740903496742,
+          0.009093424305319786,
+          -0.0027766611892729998,
+          0.008859679102897644,
+          0.015668436884880066,
+          -0.008625934831798077,
+          -0.03483550623059273,
+          -0.007721116300672293,
+          0.015298969112336636,
+          -0.003477895399555564,
+          0.011219747364521027,
+          -0.01658833585679531,
+          0.013270668685436249,
+          -0.017251869663596153,
+          -0.018639257177710533,
+          -0.012999222613871098,
+          -0.01776459999382496,
+          -0.010631615296006203,
+          -0.028079528361558914,
+          -0.01190590113401413,
+          0.019106745719909668,
+          0.0027615807484835386,
+          0.004486390855163336,
+          -0.00036286984686739743,
+          7.49302635085769e-05,
+          0.0012884235475212336,
+          0.02067509852349758,
+          -0.010428030975162983,
+          0.0032139900140464306,
+          0.015879560261964798,
+          -0.025229349732398987,
+          -0.0054138293489813805,
+          0.011815418489277363,
+          -0.017251869663596153,
+          0.008618393912911415,
+          -0.024234050884842873,
+          0.002989670494571328,
+          -0.01562319602817297,
+          0.018201928585767746,
+          -0.020147288218140602,
+          -0.013730617240071297,
+          0.010020862333476543,
+          -0.015065224841237068,
+          -0.013142486102879047,
+          -0.024927744641900063,
+          -5.4754465963924304e-05,
+          -0.006032121833413839,
+          -0.0070274220779538155,
+          -0.020388571545481682,
+          -0.0058775488287210464,
+          -0.007721116300672293,
+          0.0042564161121845245,
+          -0.012418631464242935,
+          0.03447357565164566,
+          0.012705156579613686,
+          0.01695026271045208,
+          -0.00990776065737009,
+          -0.027144549414515495,
+          0.004328047391027212,
+          0.03190992400050163,
+          0.013632595539093018,
+          0.007404429838061333,
+          0.007811598014086485,
+          -0.0012884235475212336,
+          -0.004659814294427633,
+          0.02197200432419777,
+          0.00021937131532467902,
+          0.013949282467365265,
+          0.004331817384809256,
+          -0.015200947411358356,
+          -0.0028859933372586966,
+          0.021006865426898003,
+          0.021610077470541,
+          -0.0015174556756392121,
+          0.0006545794894918799,
+          -0.0049425698816776276,
+          0.01366275642067194,
+          -0.008935080841183662,
+          -0.009493052028119564,
+          0.007242316380143166,
+          -0.01752331480383873,
+          -0.01817176677286625,
+          -0.0024618597235530615,
+          0.017719358205795288,
+          0.004245105665177107,
+          -0.005394978914409876,
+          0.003647548845037818,
+          0.014756078831851482,
+          0.019227389246225357,
+          0.006326187867671251,
+          -0.02194184437394142,
+          -0.0033384026028215885,
+          0.018473373726010323,
+          -0.013006762601435184,
+          -0.010013322345912457,
+          0.01143841166049242,
+          0.026737380772829056,
+          -0.0024618597235530615,
+          -0.055525682866573334,
+          -0.0018360271351411939,
+          0.012554354034364223,
+          -0.009327168576419353,
+          -0.040505699813365936,
+          -0.004607032984495163,
+          2.5757279217941687e-05,
+          -0.023057786747813225,
+          0.029104989022016525,
+          -0.00974187720566988,
+          -0.008648554794490337,
+          0.015321590006351471,
+          0.01981552131474018,
+          -0.010345089249312878,
+          -0.0054590702056884766,
+          0.009877599775791168,
+          -0.015344209969043732,
+          -0.02263553813099861,
+          0.005168774630874395,
+          -0.023103026673197746,
+          -0.02438485249876976,
+          -0.009817278943955898,
+          0.005285646766424179,
+          0.01924246922135353,
+          0.013806018978357315,
+          0.01809636689722538,
+          0.019348030909895897,
+          0.0058775488287210464,
+          -0.011830499395728111,
+          -0.015713678672909737,
+          0.008754117414355278,
+          -0.014688217081129551,
+          -0.0007342223543673754,
+          -0.001240355079062283,
+          0.008527912199497223,
+          0.006213085725903511,
+          0.013165106065571308,
+          -0.015834320336580276,
+          -0.020840981975197792,
+          -2.8039941753377207e-05,
+          -0.004177244380116463,
+          0.007272477261722088,
+          -0.015879560261964798,
+          -0.013549653813242912,
+          0.006522231735289097,
+          0.0022054945584386587,
+          -0.010081184096634388,
+          -0.0009868174092844129,
+          0.02964787930250168,
+          -0.015608116053044796,
+          -0.005820997525006533,
+          -0.025500794872641563,
+          0.0007535439799539745,
+          -0.011981301940977573,
+          0.0239324439316988,
+          0.009183906018733978,
+          -0.023510195314884186,
+          0.009515671990811825,
+          -0.027355672791600227,
+          0.017779679968953133,
+          -0.006887929514050484,
+          0.0034062638878822327,
+          0.01891070231795311,
+          0.008218766190111637,
+          0.02177596092224121,
+          0.011551513336598873,
+          -0.0007243258878588676,
+          -0.0010066103423014283,
+          0.03390052542090416,
+          -0.009930380620062351,
+          0.016347050666809082,
+          -0.0027201101183891296,
+          -0.008754117414355278,
+          0.010216906666755676,
+          -0.004079222679138184,
+          -0.007404429838061333,
+          0.011325309053063393,
+          0.01785508170723915,
+          0.002763465978205204,
+          0.007457210682332516,
+          -0.021987084299325943,
+          -0.005538241937756538,
+          -0.010058563202619553,
+          0.009078343398869038,
+          -0.000991530017927289,
+          0.008060422725975513,
+          0.0034703551791608334,
+          -0.020388571545481682,
+          -0.01833765022456646,
+          0.0028181320521980524,
+          -0.015608116053044796,
+          -0.006009501405060291,
+          -0.0075891632586717606,
+          -0.0065561626106500626,
+          0.015298969112336636,
+          -0.017327269539237022,
+          -0.004248876124620438,
+          0.0068502286449074745,
+          0.016724057495594025,
+          -0.006126374006271362,
+          -0.016151007264852524,
+          0.019830601289868355,
+          0.013798478990793228,
+          0.0322718508541584,
+          0.002786086406558752,
+          0.01589464209973812,
+          -0.01711614616215229,
+          -0.014914421364665031,
+          -0.024897582828998566,
+          0.018699578940868378,
+          -0.006458140444010496,
+          0.014507253654301167,
+          -0.01654309406876564,
+          -0.0030462215654551983,
+          -0.018654337152838707,
+          0.0002771005965769291,
+          -0.0007808770169503987,
+          -0.020162368193268776,
+          0.011958681978285313,
+          -0.0016795690171420574,
+          0.023419713601469994,
+          -0.01621132716536522,
+          -0.014514793641865253,
+          0.002884108340367675,
+          -0.021459273993968964,
+          0.008339408785104752,
+          -0.0031875993590801954,
+          -0.006597633473575115,
+          0.0005037764203734696,
+          0.00910850428044796,
+          -0.006642874330282211,
+          -0.03803252801299095,
+          -0.007076432928442955,
+          -0.004000050947070122,
+          0.006963330786675215,
+          -0.006243246141821146,
+          -0.01756855472922325,
+          -0.018081285059452057,
+          0.013489332981407642,
+          0.02646593563258648,
+          -0.005741826258599758,
+          0.007246086373925209,
+          0.01026968751102686,
+          0.01964963786303997,
+          -0.01654309406876564,
+          0.019544076174497604,
+          0.0015721217496320605,
+          0.00870133563876152,
+          0.003347827820107341,
+          0.008482671342790127,
+          0.013730617240071297,
+          0.0023826882243156433,
+          -0.00746475113555789,
+          -0.00976449716836214,
+          -0.006759746931493282,
+          0.006133913993835449,
+          0.016648657619953156,
+          -0.002486365381628275,
+          -0.01478623878210783,
+          0.00413577351719141,
+          -0.003489205613732338,
+          -0.02168547920882702,
+          -0.002601352520287037,
+          -0.012855960056185722,
+          -0.014529873616993427,
+          0.015909722074866295,
+          -0.013406391255557537,
+          -0.017749518156051636,
+          -0.00034331256756559014,
+          0.007117903791368008,
+          -0.006303567439317703,
+          -0.03103526681661606,
+          0.005839847959578037,
+          -0.007758816704154015,
+          0.030416974797844887,
+          -0.0065109217539429665,
+          0.008188605308532715,
+          0.005655114538967609,
+          0.011121724732220173,
+          0.0267977025359869,
+          -0.0002858189109247178,
+          0.015819240361452103,
+          0.008098123595118523,
+          -0.006936940364539623,
+          -0.003790811635553837,
+          0.026435773819684982,
+          -0.01505014393478632,
+          0.02124815061688423,
+          0.003585342550650239,
+          -0.012117025442421436,
+          0.017794759944081306,
+          -0.0037418007850646973,
+          0.014688217081129551,
+          0.0007851183763705194,
+          0.0030160609167069197,
+          0.00608490314334631,
+          0.0017625106265768409,
+          -0.022560136392712593,
+          -0.010850279591977596,
+          0.0024392392951995134,
+          -0.011770177632570267,
+          0.023660998791456223,
+          0.015434691682457924,
+          -0.007506221998482943,
+          0.016331970691680908,
+          -0.027838243171572685,
+          0.009206525981426239,
+          -0.013127405196428299,
+          0.011076483875513077,
+          0.0031875993590801954,
+          -0.004007590934634209,
+          -0.007596703711897135,
+          -0.013376230373978615,
+          -0.006835148204118013,
+          -0.00609621312469244,
+          0.013406391255557537,
+          0.03317667171359062,
+          0.008301707915961742,
+          -0.0060698227025568485,
+          -0.010691936127841473,
+          -0.003943499643355608,
+          0.004497700836509466,
+          0.011657075956463814,
+          0.0052064755000174046,
+          -0.00963631458580494,
+          -0.005745596252381802,
+          -0.007012341637164354,
+          -0.013760778121650219,
+          -0.007170685101300478,
+          0.009153745137155056,
+          0.004181014373898506,
+          0.006397819612175226,
+          0.0053836689330637455,
+          -0.0025485714431852102,
+          -0.0010848394595086575,
+          0.010714557021856308,
+          -0.004644733853638172,
+          -0.004614573437720537,
+          0.006820067763328552,
+          0.026164328679442406,
+          -0.0013110439758747816,
+          0.01456003449857235,
+          -0.0032875065226107836,
+          -0.002005680464208126,
+          -0.001410951022990048,
+          -0.00970417633652687,
+          -0.017553474754095078,
+          0.002520296024158597,
+          0.048015691339969635,
+          -0.01472591795027256,
+          -0.014379071071743965,
+          -0.004859628155827522,
+          0.03026617132127285,
+          -0.0014118935214355588,
+          -0.006024581845849752,
+          0.01764395646750927,
+          -0.007939780130982399,
+          0.014167946763336658,
+          0.008995401673018932,
+          -0.026043687015771866,
+          0.0014345140662044287,
+          -0.00726116681471467,
+          -0.0006875676335766912,
+          0.011649535968899727,
+          0.025108708068728447,
+          0.012184886261820793,
+          0.00275215576402843,
+          0.002882223343476653,
+          0.020479053258895874,
+          0.015200947411358356,
+          -0.00540251936763525,
+          0.002458089729771018,
+          0.0017087870510295033,
+          0.005274336785078049,
+          -0.009447811171412468,
+          -0.011219747364521027,
+          -0.017387591302394867,
+          0.00014950709010008723,
+          -0.002073541982099414,
+          0.0005749366246163845,
+          -0.022439494729042053,
+          -0.00841481052339077,
+          0.013466712087392807,
+          -0.01262221485376358,
+          -0.00963631458580494,
+          0.002147058490663767,
+          -0.0011687236838042736,
+          -0.005240405909717083,
+          0.0055307019501924515,
+          0.02405308559536934,
+          -0.007442130707204342,
+          -0.014967203140258789,
+          0.003630583407357335,
+          0.01100862305611372,
+          0.013994523324072361,
+          -0.0005881318938918412,
+          0.008399729616940022,
+          -0.005251716356724501,
+          0.011589214205741882,
+          0.010737176984548569,
+          -0.005304497200995684,
+          -0.010593914426863194,
+          -0.007178225088864565,
+          0.00746475113555789,
+          -0.015427151694893837,
+          -0.004316737409681082,
+          -2.169266554119531e-05,
+          -0.005319577641785145,
+          0.014997363090515137,
+          0.021293390542268753,
+          -0.00672958604991436,
+          0.020192528143525124,
+          -0.006284717004746199,
+          0.00540251936763525,
+          -0.0034929756075143814,
+          -0.0011517582461237907,
+          -0.008980321697890759,
+          -0.03540855646133423,
+          -0.011951141990721226,
+          0.027355672791600227,
+          -0.01039787009358406,
+          -0.008256467059254646,
+          0.023314151912927628,
+          0.003951040096580982,
+          0.009417650289833546,
+          -0.008633474819362164,
+          0.009515671990811825,
+          -0.009342248551547527,
+          0.007581623271107674,
+          0.004316737409681082,
+          0.0009962426265701652,
+          -0.004361978266388178,
+          -0.024731699377298355,
+          -0.005809687543660402,
+          -0.006220625713467598,
+          -0.02450549602508545,
+          -0.006872849073261023,
+          0.004120693542063236,
+          -0.008927540853619576,
+          -0.022695859894156456,
+          0.0007804057677276433,
+          -0.005949180107563734,
+          0.0007690955535508692,
+          -0.020433813333511353,
+          0.01039787009358406,
+          0.020569536834955215,
+          0.0346243791282177,
+          -0.03230201452970505,
+          0.00651469174772501,
+          -0.0012752283364534378,
+          0.010586374439299107,
+          -0.026888184249401093,
+          -0.0009274387848563492,
+          0.008060422725975513,
+          0.007442130707204342,
+          0.001478812424466014,
+          -0.0026579038240015507,
+          -0.003954810090363026,
+          0.025877803564071655,
+          -0.002356297569349408,
+          0.0025693068746477365,
+          -0.008837059140205383,
+          -0.013594894669950008,
+          -0.02405308559536934,
+          0.012486492283642292,
+          0.012350769713521004,
+          0.01299168262630701,
+          -0.010940761305391788,
+          -0.0008407270070165396,
+          -0.006032121833413839,
+          -0.003853017929941416,
+          0.007879459299147129,
+          0.006612713914364576,
+          -0.01202654279768467,
+          0.023057786747813225,
+          -0.011815418489277363,
+          -0.0030462215654551983,
+          0.0029972107149660587,
+          -0.0039057990070432425,
+          0.001874670386314392,
+          0.011747557669878006,
+          -0.011393170803785324,
+          -0.018684497103095055,
+          -0.009357329457998276,
+          0.013248047791421413,
+          0.010827658697962761,
+          -0.00870133563876152,
+          -0.034292612224817276,
+          -0.02156483568251133,
+          0.03172896057367325,
+          -0.01345917209982872,
+          0.010654235258698463,
+          -0.00619800528511405,
+          0.011453491635620594,
+          0.012343229725956917,
+          0.0015655241440981627,
+          -0.0034194590989500284,
+          -0.01063915528357029,
+          -0.011385629884898663,
+          -0.00843743048608303,
+          -0.0003246978158131242,
+          -0.0029764752835035324,
+          -0.014612815342843533,
+          -0.005926559679210186,
+          0.017085986211895943,
+          -0.0015589265385642648,
+          0.007668334990739822,
+          0.0027201101183891296,
+          0.013134945183992386,
+          0.006940710358321667,
+          7.58727837819606e-05,
+          0.01802096515893936,
+          -0.014363990165293217,
+          0.012750397436320782,
+          0.041651804000139236,
+          -0.0020999324042350054,
+          -0.006669264752417803,
+          -0.019227389246225357,
+          -0.007083973381668329,
+          -0.029104989022016525,
+          0.005357278510928154,
+          -0.016362130641937256,
+          -0.026375453919172287,
+          0.011641995050013065,
+          -0.016271648928523064,
+          -0.012117025442421436,
+          -0.023555437102913857,
+          0.008165985345840454,
+          -0.0038944887928664684,
+          0.011151885613799095,
+          0.0022752410732209682,
+          0.01216980628669262,
+          -0.009146205149590969,
+          -0.020494135096669197,
+          0.005353508051484823,
+          0.013798478990793228,
+          -0.002884108340367675,
+          -0.02515394799411297,
+          -0.012908740900456905,
+          0.005296957213431597,
+          -0.017719358205795288,
+          -0.010111344046890736,
+          -0.03966120257973671,
+          -0.017659036442637444,
+          0.0207505002617836,
+          0.010593914426863194,
+          -0.007909620180726051,
+          0.005545781925320625,
+          -0.0012167921522632241,
+          0.004591953009366989,
+          0.0030236011371016502,
+          0.0038096620701253414,
+          0.013180186972022057,
+          -0.002090507186949253,
+          0.004354438278824091,
+          0.00976449716836214,
+          0.013617515563964844,
+          0.006590093486011028,
+          -0.01349687296897173,
+          -0.019785359501838684,
+          0.011657075956463814,
+          -0.005063212476670742,
+          0.005342198070138693,
+          -0.0019161412492394447,
+          -0.00466735428199172,
+          -0.012674996629357338,
+          0.010111344046890736,
+          0.01308970432728529,
+          0.016030363738536835,
+          0.017779679968953133,
+          0.014115165919065475,
+          0.005919019691646099,
+          -0.015427151694893837,
+          0.001035828492604196,
+          -7.470199761883123e-06,
+          0.004018901381641626,
+          -0.007083973381668329,
+          0.006224395707249641,
+          0.003477895399555564,
+          0.010201826691627502,
+          0.011355469934642315,
+          -0.0016701437998563051,
+          0.012750397436320782,
+          -0.007332798093557358,
+          -0.003246035659685731,
+          0.0022563906386494637,
+          0.003076382214203477,
+          -0.009319628588855267,
+          0.009568453766405582,
+          0.0017172697698697448,
+          -0.004780456889420748,
+          0.010578833520412445,
+          -0.02450549602508545,
+          -0.008181065320968628,
+          -0.01276547834277153,
+          0.0011866315035149455,
+          -0.023480035364627838,
+          0.01262221485376358,
+          0.03377988189458847,
+          0.0009773923084139824,
+          -0.00225262064486742,
+          -0.00038784657954238355,
+          0.010744716972112656,
+          -0.01666373759508133,
+          -0.009493052028119564,
+          -0.0007412912673316896,
+          -0.01862417720258236,
+          -0.024580897763371468,
+          0.01833765022456646,
+          0.017839999869465828,
+          -0.013308369554579258,
+          -0.016241488978266716,
+          -0.002853947691619396,
+          -0.01833765022456646,
+          -0.010020862333476543,
+          -0.0019180262461304665,
+          -0.012984142638742924,
+          -0.019574236124753952,
+          -0.002957624848932028,
+          -0.0291200689971447,
+          0.0021150128450244665,
+          0.010096264071762562,
+          0.011385629884898663,
+          -0.010744716972112656,
+          0.0062394761480391026,
+          -0.01190590113401413,
+          -0.0033685630187392235,
+          0.006593863479793072,
+          0.01256189402192831,
+          0.03390052542090416,
+          -0.006789907347410917,
+          -0.010737176984548569,
+          0.01525372825562954,
+          0.007634404581040144,
+          0.002652248600497842,
+          0.018729738891124725,
+          0.0013921007048338652,
+          0.010616534389555454,
+          0.0018614751752465963,
+          0.015412071719765663,
+          0.01349687296897173,
+          -0.008935080841183662,
+          0.017734438180923462,
+          0.0116646159440279,
+          -0.0014986053574830294,
+          -0.019393272697925568,
+          0.014477092772722244,
+          0.010330009274184704,
+          0.013014303520321846,
+          0.0009905875194817781,
+          0.014914421364665031,
+          -0.012463872320950031,
+          -0.01768919825553894,
+          -0.01431874930858612,
+          0.00016482302453368902,
+          -0.006929399911314249,
+          0.010111344046890736,
+          -0.0006734298658557236,
+          -0.02405308559536934,
+          -0.0012714582262560725,
+          0.0035928827710449696,
+          -0.002141403267160058,
+          0.0018944633193314075,
+          -0.005342198070138693,
+          0.011973761953413486,
+          0.013263127766549587,
+          0.007871919311583042,
+          0.012750397436320782,
+          -0.008927540853619576,
+          0.009447811171412468,
+          -0.004312967415899038,
+          0.012840880081057549,
+          -0.0025108708068728447,
+          -0.0020999324042350054,
+          -3.84960112569388e-05,
+          0.0003195139579474926,
+          0.007110363803803921,
+          0.00963631458580494,
+          0.008874759078025818,
+          0.026088926941156387,
+          -0.012780558317899704,
+          0.012584514915943146,
+          0.009530752897262573,
+          -0.013700457289814949,
+          0.03133687376976013,
+          0.017025664448738098,
+          0.011920981109142303,
+          -0.017101066187024117,
+          0.010518512688577175,
+          0.023781640455126762,
+          0.024188809096813202,
+          -0.00870133563876152,
+          -0.005964260548353195,
+          -0.0003664042742457241,
+          0.022967305034399033,
+          0.015374370850622654,
+          -0.017508234828710556,
+          0.01519340742379427,
+          0.011174505576491356,
+          0.009183906018733978,
+          -0.015057684853672981,
+          0.040777143090963364,
+          -0.008686255663633347,
+          -0.017387591302394867,
+          -0.011687235906720161,
+          -0.014160406775772572,
+          -0.016392290592193604,
+          -0.005176314618438482,
+          -0.006537312176078558,
+          -0.009953001514077187,
+          0.0018482798477634788,
+          0.0016239603282883763,
+          -0.031216232106089592,
+          -0.003966120071709156,
+          -0.006314877886325121,
+          -0.013655216433107853,
+          -0.015065224841237068,
+          0.008482671342790127,
+          -0.010812578722834587,
+          0.0030575317796319723,
+          0.004482620861381292,
+          -0.020690178498625755,
+          0.01744791306555271,
+          0.023148268461227417,
+          0.0033968386705964804,
+          0.011249907314777374,
+          -0.00466735428199172,
+          0.0058775488287210464,
+          -0.0025448014494031668,
+          0.025395233184099197,
+          -0.007479831110686064,
+          0.010345089249312878,
+          -0.002599467523396015,
+          -0.01519340742379427,
+          -0.04545203968882561,
+          -0.0024147338699549437,
+          0.006865308620035648,
+          0.0181416068226099,
+          -0.01356473471969366,
+          0.007649484556168318,
+          0.013519493862986565,
+          -0.013338529504835606,
+          0.0050368220545351505,
+          0.023314151912927628,
+          -0.006152764428406954,
+          0.013044463470578194,
+          0.00953829288482666,
+          0.02304270677268505,
+          -0.011626915074884892,
+          -0.02414356917142868,
+          0.006013271398842335,
+          0.013707997277379036,
+          -0.005255486350506544,
+          0.017146306112408638,
+          0.012207507155835629,
+          -0.0139869824051857,
+          -0.004478850401937962,
+          -0.014514793641865253,
+          -0.010345089249312878,
+          0.012848420068621635,
+          -0.0047502960078418255,
+          0.018036045134067535,
+          -0.002073541982099414,
+          -0.004825697746127844,
+          -0.016724057495594025,
+          0.011792798526585102,
+          0.007849298417568207,
+          -0.007212155964225531,
+          -0.004037751816213131,
+          -0.02124815061688423,
+          -0.021519595757126808,
+          0.003251690650358796,
+          -0.024897582828998566,
+          0.02622465044260025,
+          -0.0005909594474360347,
+          0.03218137100338936,
+          0.000863818742800504,
+          -0.002763465978205204,
+          0.019378192722797394,
+          0.01707090437412262,
+          -0.013383770361542702,
+          0.0038794083520770073,
+          0.009862519800662994,
+          0.0032931615132838488,
+          -0.0008280029869638383,
+          -0.00904818344861269,
+          -0.0006696598138660192,
+          -0.0007083030650392175,
+          0.009236686863005161,
+          0.0010527938138693571,
+          0.0009330938919447362,
+          -0.008181065320968628,
+          -0.024882502853870392,
+          -0.006597633473575115,
+          0.010767337866127491,
+          0.005734285805374384,
+          -0.006337498314678669,
+          -0.012878580018877983,
+          0.007415739819407463,
+          -0.006529772188514471,
+          -0.01084273960441351,
+          -0.00873903650790453,
+          0.009779578074812889,
+          0.0020660017617046833,
+          -0.01026968751102686,
+          0.021006865426898003,
+          0.0021432882640510798,
+          -0.006910549942404032,
+          0.014379071071743965,
+          0.001902946038171649,
+          -0.017387591302394867,
+          -0.00996808148920536,
+          0.009847438894212246,
+          -0.01000578235834837,
+          0.002179104136303067,
+          0.010058563202619553,
+          -0.0009604269289411604,
+          0.0136853763833642,
+          0.003257345873862505,
+          -0.011958681978285313,
+          -0.012690076604485512,
+          -0.0282604917883873,
+          -0.00847513135522604,
+          -0.028245411813259125,
+          0.012509113177657127,
+          0.0029764752835035324,
+          0.0017945562722161412,
+          0.013504412956535816,
+          0.0018209468107670546,
+          -0.0207505002617836,
+          0.017779679968953133,
+          -0.01768919825553894,
+          0.011257447302341461,
+          0.004716365598142147,
+          0.007962401024997234,
+          -0.006797447334975004,
+          0.011302688159048557,
+          0.009787118062376976,
+          -0.02185136266052723,
+          0.018156686797738075,
+          0.02173071913421154,
+          -0.02070525847375393,
+          0.010201826691627502,
+          0.009900220669806004,
+          0.007864379324018955,
+          -0.0006555219879373908,
+          -0.013971902430057526,
+          -0.008505292236804962,
+          -0.017960643395781517,
+          0.003438309533521533,
+          0.003342172596603632,
+          -0.006360118743032217,
+          -0.005455300211906433,
+          -0.015027523972094059,
+          0.011038783006370068,
+          0.00530072720721364,
+          -0.021640237420797348,
+          -0.017266949638724327,
+          -0.0038605581503361464,
+          -0.01924246922135353,
+          0.004803077317774296,
+          0.027551718056201935,
+          -0.008346948772668839,
+          -0.015268809162080288,
+          0.024611057713627815,
+          0.02630005218088627,
+          0.009440271183848381,
+          -0.010986002162098885,
+          0.0015410187188535929,
+          -0.004855858162045479,
+          -0.011340389028191566,
+          0.005312037654221058,
+          -0.004723905585706234,
+          -0.010035943239927292,
+          0.008542993105947971,
+          -0.003732375567778945,
+          0.00858069397509098,
+          0.022349011152982712,
+          -0.006642874330282211,
+          0.004459999967366457,
+          0.009123584255576134,
+          -0.0012422400759533048,
+          0.01339885126799345,
+          -0.02577224187552929,
+          0.0108050387352705,
+          -0.025455554947257042,
+          -0.002605122746899724,
+          0.01086535956710577,
+          0.01919722743332386,
+          0.025470634922385216,
+          -0.00963631458580494,
+          0.0008741864003241062,
+          -0.0009552430710755289,
+          0.027958884835243225,
+          0.021142587065696716,
+          -0.00422625569626689,
+          -0.01678437925875187,
+          -0.014017143286764622,
+          -0.004248876124620438,
+          -0.002265815855935216,
+          0.016648657619953156,
+          -0.024203889071941376,
+          0.0044336095452308655,
+          -0.018473373726010323,
+          -0.002045266330242157,
+          -0.0013402621261775494,
+          0.014115165919065475,
+          0.015261268243193626,
+          0.002590042306110263,
+          -0.0019566696137189865,
+          0.019875841215252876,
+          0.005647574085742235,
+          -0.008196145296096802,
+          0.0042564161121845245,
+          0.005805917549878359,
+          -0.001035828492604196,
+          0.007698495872318745,
+          -0.05235881730914116,
+          0.008203686214983463,
+          0.01780983991920948,
+          0.010292308405041695,
+          0.005168774630874395,
+          0.001149873249232769,
+          -0.005640034098178148,
+          -0.015366830863058567,
+          -0.010887980461120605,
+          0.017749518156051636,
+          -0.015472392551600933,
+          0.0064128995873034,
+          0.00392087921500206,
+          -0.007826678454875946,
+          0.021293390542268753,
+          0.008075503632426262,
+          0.015035063959658146,
+          0.009651395492255688,
+          0.03047729656100273,
+          0.011423330754041672,
+          0.010827658697962761,
+          -0.005624953657388687,
+          0.00304810656234622,
+          0.0010999197838827968,
+          0.005621183663606644,
+          0.0196043960750103,
+          -0.019634557887911797,
+          -0.004821927286684513,
+          0.006066052708774805,
+          -0.0279739648103714,
+          -0.005851158406585455,
+          0.012320608831942081,
+          -0.010081184096634388,
+          -0.006722046062350273,
+          0.007121674250811338,
+          -0.0014543068828061223,
+          0.0004229554324410856,
+          0.01572875864803791,
+          0.0022073795553296804,
+          -0.013285748660564423,
+          0.01568351686000824,
+          -0.002940659411251545,
+          -0.03136703372001648,
+          -0.011740017682313919,
+          -0.011845579370856285,
+          -0.007155604660511017,
+          -0.005191395059227943,
+          -0.03091462515294552,
+          0.0008732439018785954,
+          -0.011280068196356297,
+          0.016799459233880043,
+          -0.0010509087005630136,
+          0.006853998638689518,
+          0.006752206478267908,
+          -0.010797498747706413,
+          -0.011657075956463814,
+          -0.01797572337090969,
+          -0.0011395055335015059,
+          -0.01372307725250721,
+          -0.0075891632586717606,
+          0.01752331480383873,
+          -0.021217988803982735,
+          -0.007321488112211227,
+          0.025546036660671234,
+          -0.011310229077935219,
+          -0.0027653509750962257,
+          0.006691885180771351,
+          0.0026484786067157984,
+          0.0027069146744906902,
+          0.002339332364499569,
+          0.009033102542161942,
+          -0.0008756002062000334,
+          0.002906728768721223,
+          -0.02005680650472641,
+          0.03287506476044655,
+          0.011355469934642315,
+          -0.0011395055335015059,
+          0.010691936127841473,
+          -0.02687310427427292,
+          -0.025425394997000694,
+          0.019257549196481705,
+          -0.0311257503926754,
+          -0.006111293565481901,
+          -0.01250157319009304,
+          -0.0009213123703375459,
+          -0.03218137100338936,
+          -0.025108708068728447,
+          -0.029195470735430717,
+          -0.040746983140707016,
+          0.008482671342790127,
+          -0.00225262064486742,
+          0.005519391503185034,
+          0.007773897144943476,
+          -0.025636518374085426,
+          0.013021843507885933,
+          -0.008625934831798077,
+          -0.0013242393033578992,
+          0.018986104056239128,
+          -0.007434590253978968,
+          0.004486390855163336,
+          0.017025664448738098,
+          -0.0005692815175279975,
+          -0.006620253901928663,
+          -0.014145325869321823,
+          -0.006808757781982422,
+          0.026767540723085403,
+          0.025455554947257042,
+          0.022484734654426575,
+          -0.003114083083346486,
+          0.00249202037230134,
+          -0.003898258786648512,
+          0.019619476050138474,
+          -0.018066205084323883,
+          -0.008331868797540665,
+          0.003496745601296425,
+          0.007204615511000156,
+          0.008897379972040653,
+          0.004241335671395063,
+          0.03655466064810753,
+          0.020644936710596085,
+          0.015140626579523087,
+          -0.011740017682313919,
+          -0.013308369554579258,
+          -0.01103124301880598,
+          -0.013738158158957958,
+          0.02421896904706955,
+          0.024490416049957275,
+          -0.006925629917532206,
+          -0.02304270677268505,
+          -0.011461031623184681,
+          -0.00904818344861269,
+          -0.02053937502205372,
+          0.004901099018752575,
+          0.02254505641758442,
+          0.009990702383220196,
+          0.014326289296150208,
+          -0.010608994401991367,
+          0.011468571610748768,
+          0.03299570828676224,
+          -0.012124565429985523,
+          0.006118833553045988,
+          -0.0026465936098247766,
+          -0.016362130641937256,
+          0.012742857448756695,
+          -0.015208487398922443,
+          -0.019016264006495476,
+          -0.00466735428199172,
+          -0.021881522610783577,
+          0.018126526847481728,
+          0.00035768598900176585,
+          0.003087692428380251,
+          -0.012818259187042713,
+          -0.02443009428679943,
+          0.0060170418582856655,
+          0.00046089180978015065,
+          0.003898258786648512,
+          -0.00619800528511405,
+          0.00571920583024621,
+          -0.01711614616215229,
+          0.003393068676814437,
+          -0.017719358205795288,
+          0.0219267625361681,
+          -0.010043483227491379,
+          -0.011936061084270477,
+          0.004886019043624401,
+          0.008098123595118523,
+          0.009666475467383862,
+          0.0017879586666822433,
+          -0.0028690281324088573,
+          0.007562772836536169,
+          -0.017327269539237022,
+          0.017085986211895943,
+          0.01006610319018364,
+          0.011400710791349411,
+          0.003739915555343032,
+          0.002853947691619396,
+          -0.012546814046800137,
+          0.02801920659840107,
+          -0.0008986919419839978,
+          -0.008135824464261532,
+          0.00841481052339077,
+          -0.002000025473535061,
+          0.011958681978285313,
+          0.013263127766549587,
+          -0.00798502191901207,
+          0.0006074535194784403,
+          0.003754995996132493,
+          0.008369569666683674,
+          0.015321590006351471,
+          -0.0040339813567698,
+          0.020267929881811142,
+          0.0063337283208966255,
+          -0.00022243449348025024,
+          0.012275367975234985,
+          0.0037022149190306664,
+          0.015019983984529972,
+          -0.023962603881955147,
+          -0.009206525981426239,
+          -0.023721320554614067,
+          -0.0016739139100536704,
+          -0.00741196982562542,
+          0.008754117414355278,
+          -0.0029727050568908453,
+          -0.011521353386342525,
+          0.005455300211906433,
+          0.025907963514328003,
+          0.023299071937799454,
+          -0.004267726093530655,
+          0.016648657619953156,
+          -0.006744666490703821,
+          -0.023389553651213646,
+          -0.010707016102969646,
+          -0.006624023895710707,
+          -0.03145751729607582,
+          0.022514894604682922,
+          -0.007004801649600267,
+          0.005195165053009987,
+          0.02242441289126873,
+          -0.007238546386361122,
+          -0.0033384026028215885,
+          0.01202654279768467,
+          -0.004316737409681082,
+          -0.026330212131142616,
+          0.00830924790352583,
+          -0.013285748660564423,
+          0.014575114473700523,
+          0.011996382847428322,
+          0.031638480722904205,
+          -0.013225427828729153,
+          -0.002260160632431507,
+          0.009410110302269459,
+          0.012388470582664013,
+          0.020267929881811142,
+          -0.022695859894156456,
+          -0.003210220020264387,
+          -0.001869957777671516,
+          -0.0014656170969828963,
+          -0.0029255792032927275,
+          -0.004490160848945379,
+          0.008784277364611626,
+          -0.01972503960132599,
+          -0.002989670494571328,
+          -0.005082062911242247,
+          0.003183829365298152,
+          0.003858672920614481,
+          0.0026805242523550987,
+          0.002616432961076498,
+          -0.012516653165221214,
+          -0.008037802763283253,
+          -0.006526002194732428,
+          -0.015819240361452103,
+          0.012388470582664013,
+          0.000850152166094631,
+          -0.01597004383802414,
+          0.00847513135522604,
+          0.005673964973539114,
+          0.0030914624221622944,
+          0.003460929961875081,
+          0.0034703551791608334,
+          0.05558600276708603,
+          -0.008814438246190548,
+          -0.007140524219721556,
+          -0.019348030909895897,
+          0.011166965588927269,
+          -0.013904041610658169,
+          -0.0017417751951143146,
+          0.0007804057677276433,
+          0.02194184437394142,
+          -0.028969265520572662,
+          -0.018307490274310112,
+          0.010782417841255665,
+          -0.011423330754041672,
+          -0.01127252820879221,
+          -0.0005843618419021368,
+          0.0030613020062446594,
+          -0.03172896057367325,
+          0.003935959655791521,
+          0.005572172813117504,
+          -0.004154623951762915,
+          0.01462035533040762,
+          -0.004637193866074085,
+          -0.005481690634042025,
+          0.02230377122759819,
+          0.015057684853672981,
+          -0.006235706154257059,
+          0.002520296024158597,
+          -0.00810566358268261,
+          0.014273508451879025,
+          -0.024611057713627815,
+          -0.013557193800807,
+          0.002192299347370863,
+          0.0026258581783622503,
+          -0.017221707850694656,
+          -0.007404429838061333,
+          0.0009679670911282301,
+          -0.003574032336473465,
+          0.017598716542124748,
+          -0.004659814294427633,
+          0.015223568305373192,
+          -0.0119662219658494,
+          0.03305602818727493,
+          -0.004128233529627323,
+          -0.023253830149769783,
+          0.006390279158949852,
+          0.0029990957118570805,
+          0.011363009922206402,
+          0.0004262542352080345,
+          0.01833765022456646,
+          0.015268809162080288,
+          0.0019161412492394447,
+          0.006872849073261023,
+          0.005293187219649553,
+          -0.01744791306555271,
+          0.011920981109142303,
+          -0.008158445358276367,
+          0.008791817352175713,
+          -0.0108050387352705,
+          0.0030839224345982075,
+          -0.008316787891089916,
+          0.009093424305319786,
+          -0.004595723003149033,
+          0.0025221810210496187,
+          -0.0001197588280774653,
+          -0.001998140476644039,
+          -0.0016692013014107943,
+          -0.017659036442637444,
+          -0.010186745785176754,
+          0.02058461681008339,
+          -0.02115766704082489,
+          -0.007132984232157469,
+          0.00661648390814662,
+          -0.025048386305570602,
+          -0.010887980461120605,
+          0.017327269539237022,
+          -0.015140626579523087,
+          -0.007276247255504131,
+          0.011709856800734997,
+          0.0024844801519066095,
+          -0.01323296781629324,
+          0.014205647632479668,
+          0.002084852196276188,
+          -0.036946747452020645,
+          0.011385629884898663,
+          -0.003988740500062704,
+          0.00010674028453649953,
+          -0.01054113358259201,
+          -0.047653764486312866,
+          0.006861538626253605,
+          0.0006300740060396492,
+          -0.013994523324072361,
+          0.004784226883202791,
+          0.009116044268012047,
+          0.0155628751963377,
+          -0.0032875065226107836,
+          0.009425190277397633,
+          0.014605275355279446,
+          0.0007756931590847671,
+          -3.722950350493193e-05,
+          0.019875841215252876,
+          0.0010103804524987936,
+          0.022273611277341843,
+          -0.011672155931591988,
+          -0.005168774630874395,
+          -0.023434793576598167,
+          0.010051023215055466,
+          -0.0020584615413099527,
+          -0.0005358220660127699,
+          -0.019167067483067513,
+          -0.0032139900140464306,
+          -0.015577955171465874,
+          0.02859225869178772,
+          -0.003751225769519806,
+          -0.0033685630187392235,
+          0.013082164339721203,
+          -0.003721065353602171,
+          -0.007238546386361122,
+          0.014431851916015148,
+          0.030703501775860786,
+          0.00422625569626689,
+          -0.003709755139425397,
+          0.007325258105993271,
+          -0.006469450891017914,
+          -0.012886120937764645,
+          0.0010895520681515336,
+          -0.0008284742361865938,
+          -0.01086535956710577,
+          -0.0039057990070432425,
+          0.0011451606405898929,
+          -0.011755097657442093,
+          -0.01515570655465126,
+          -0.014348910190165043,
+          0.016241488978266716,
+          -0.003223415231332183,
+          -0.00413577351719141,
+          -0.021187828853726387,
+          0.0029104987625032663,
+          -0.015065224841237068,
+          -0.004874708596616983,
+          0.014627896249294281,
+          -0.011898360215127468,
+          0.03377988189458847,
+          -0.006631563883274794,
+          0.0239324439316988,
+          -0.004297886975109577,
+          -0.0011366780381649733,
+          0.0004455758898984641,
+          -0.004482620861381292,
+          -0.014884261414408684,
+          -0.005968030542135239,
+          0.008565613068640232,
+          0.0028916485607624054,
+          0.007102823816239834,
+          0.01826224848628044,
+          -0.0015118005685508251,
+          0.045844126492738724,
+          -0.006737126037478447,
+          0.02572700008749962,
+          -0.008158445358276367,
+          -0.002590042306110263,
+          0.00386432814411819,
+          0.012531733140349388,
+          0.005157464183866978,
+          0.002361952792853117,
+          -0.022906983271241188,
+          -0.02177596092224121,
+          0.020569536834955215,
+          -0.0062470161356031895,
+          -0.0011479882523417473,
+          -0.027069147676229477,
+          0.00037135250749997795,
+          0.009078343398869038,
+          0.0009217836195603013,
+          -0.001966094831004739,
+          0.00671827606856823,
+          0.015306509099900723,
+          0.003590997541323304,
+          -0.014680677093565464,
+          -0.004935029894113541,
+          -0.026692138984799385,
+          -0.004297886975109577,
+          0.010744716972112656,
+          0.006314877886325121,
+          0.012222587130963802,
+          -0.013481792993843555,
+          -0.031547997146844864,
+          0.007796517573297024,
+          -0.009251766838133335,
+          -0.007023652084171772,
+          0.03139719367027283,
+          -0.02103702537715435,
+          -0.010247067548334599,
+          -0.02099178545176983,
+          -0.018654337152838707,
+          -0.008075503632426262,
+          0.020690178498625755,
+          -0.010624075308442116,
+          0.0046975151635706425,
+          -0.02500314638018608,
+          -0.00152405328117311,
+          0.017055824398994446,
+          0.02053937502205372,
+          0.006205545272678137,
+          0.014326289296150208
+        ],
+        "keyphrases": [
+          "Hash Layers",
+          "token ID",
+          "expert ID",
+          "load balance",
+          "Hashing based routing"
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "2d39419a-1397-4001-8e1e-a802772c4f79",
+      "properties": {
+        "page_content": "E Fine-Tuning Details\n\nWe did a hyperparameter search for both baseline models and expert choice method. For fine-tuning of the 8B dense model, we use a constant learning rate of 0.0001 and a dropout rate of 0.1. We freeze the attention layer and feed-forward layer while leaving the embedding and layer normalization trainable. This setting has been found optimal for the 8B dense model. For MoE 8B/64E models including GShard top-2 gating and expert choice, we found continuing the learning rate from the pre-trained model while using a square root learning rate decay works better. In addition, we do not apply parameter freezing for fine-tuning MoE models. For models with 100M expert size, we use a constant learning rate of 0.0001 and no dropout is used.",
+        "title": "E Fine-Tuning Details",
+        "embedding": [
+          0.022390764206647873,
+          -0.0034248128067702055,
+          -0.016889112070202827,
+          -0.013610072433948517,
+          0.024709414690732956,
+          0.026149997487664223,
+          -0.02159501425921917,
+          0.05106521025300026,
+          -0.016985150054097176,
+          0.006918224971741438,
+          0.008664073422551155,
+          -0.005028318148106337,
+          0.013939348049461842,
+          0.006582088768482208,
+          0.030732421204447746,
+          0.0230218768119812,
+          0.008314217440783978,
+          0.024915212765336037,
+          0.007045133039355278,
+          -0.03493069112300873,
+          0.022267285734415054,
+          -0.0422845184803009,
+          0.0016892541898414493,
+          -0.017520224675536156,
+          0.0009578156750649214,
+          -0.003855272429063916,
+          0.014789978042244911,
+          0.014254904352128506,
+          -0.01691655069589615,
+          -0.0006456895498558879,
+          0.039293594658374786,
+          0.015421089716255665,
+          0.0020151000935584307,
+          0.005889237392693758,
+          -0.05144936591386795,
+          -0.019770275801420212,
+          0.042311958968639374,
+          0.03193976730108261,
+          -0.0038278328720480204,
+          0.021361777558922768,
+          0.03226904198527336,
+          0.015064374543726444,
+          -0.023460911586880684,
+          -0.014995775185525417,
+          -0.019139165058732033,
+          -0.019098004326224327,
+          -0.017163507640361786,
+          0.007635085843503475,
+          0.004493244457989931,
+          -0.06272706389427185,
+          -0.01129142101854086,
+          -0.024627096951007843,
+          0.009597022086381912,
+          0.02502497099339962,
+          -0.044781528413295746,
+          -0.0015443384181708097,
+          -0.00871209241449833,
+          0.06366001814603806,
+          0.05512627959251404,
+          0.0056354207918047905,
+          0.022527962923049927,
+          0.0003787959576584399,
+          -0.040638137608766556,
+          -0.015023214742541313,
+          0.025903040543198586,
+          0.011387459933757782,
+          0.027220144867897034,
+          -0.023309992626309395,
+          -0.024174341931939125,
+          0.003937591332942247,
+          -0.04969322681427002,
+          0.03051290474832058,
+          -0.018123896792531013,
+          0.014186304993927479,
+          -0.005940686911344528,
+          -0.04017166420817375,
+          0.020140711218118668,
+          0.023749027401208878,
+          0.039952147752046585,
+          -0.011373739689588547,
+          0.015187853015959263,
+          -0.026561593636870384,
+          -0.00991943757981062,
+          0.040308862924575806,
+          0.052601832896471024,
+          -0.006551219150424004,
+          -0.022774919867515564,
+          -0.036549631506204605,
+          -0.026410674676299095,
+          0.015352491289377213,
+          -0.02174593135714531,
+          -0.005419333465397358,
+          -0.019619358703494072,
+          -0.006074455101042986,
+          -0.029278120025992393,
+          0.047168780118227005,
+          -0.032735515385866165,
+          0.011908813379704952,
+          -0.02487405389547348,
+          -0.0023117915261536837,
+          -0.016710754483938217,
+          0.005851508118212223,
+          -0.0023615260142832994,
+          -0.002745681209489703,
+          0.02949763648211956,
+          -0.01580524444580078,
+          -0.00518266623839736,
+          -0.005628560669720173,
+          0.0011584681924432516,
+          0.05894039571285248,
+          -0.024627096951007843,
+          -0.04722365736961365,
+          -0.016271719709038734,
+          0.03597339615225792,
+          -0.043354667723178864,
+          0.01352775376290083,
+          -0.008101560175418854,
+          -0.013013259507715702,
+          0.012965240515768528,
+          -0.04494616761803627,
+          0.03976007178425789,
+          0.01974283717572689,
+          0.034162379801273346,
+          0.012690844014286995,
+          -0.04423273354768753,
+          0.03284527361392975,
+          0.021938009187579155,
+          0.014817417599260807,
+          0.004798511043190956,
+          -0.021732212975621223,
+          0.016930270940065384,
+          0.008163299411535263,
+          -0.022706320509314537,
+          -0.0387173630297184,
+          0.0006821328424848616,
+          -0.0073538292199373245,
+          0.04705902189016342,
+          0.05186096206307411,
+          -0.026945747435092926,
+          -0.03528740629553795,
+          0.014693939127027988,
+          -0.01673819310963154,
+          0.026712510734796524,
+          0.004249717574566603,
+          -0.03471117094159126,
+          0.021512694656848907,
+          0.027000626549124718,
+          0.06689789891242981,
+          0.008897310122847557,
+          0.03155561164021492,
+          -0.03427213802933693,
+          0.005693729966878891,
+          -0.03163792937994003,
+          0.00961760152131319,
+          -0.005920107010751963,
+          -0.007587066385895014,
+          -0.02540912665426731,
+          0.0006538357120007277,
+          -0.015174132771790028,
+          0.0062905424274504185,
+          -7.942495722090825e-05,
+          0.007401848677545786,
+          0.0049631488509476185,
+          0.03336662799119949,
+          0.011627556756138802,
+          0.06398928910493851,
+          0.007834023796021938,
+          0.01986631564795971,
+          -0.07842255383729935,
+          0.028317730873823166,
+          0.040994856506586075,
+          0.017739741131663322,
+          0.00991943757981062,
+          0.007984941825270653,
+          0.0012545069912448525,
+          0.025971639901399612,
+          0.016326598823070526,
+          0.023762747645378113,
+          -0.031418412923812866,
+          0.02243192307651043,
+          -0.01140803936868906,
+          0.009898858144879341,
+          0.021732212975621223,
+          0.00917170662432909,
+          0.0123341279104352,
+          -0.01002233661711216,
+          -0.007415568456053734,
+          0.030677542090415955,
+          -0.0015400510746985674,
+          0.00259133311919868,
+          0.0060950350016355515,
+          0.020209310576319695,
+          -0.005601121112704277,
+          -0.0056731500662863255,
+          0.007628226187080145,
+          -0.0274396613240242,
+          0.004613293334841728,
+          -0.014515581540763378,
+          0.008259338326752186,
+          -0.013678671792149544,
+          -0.010289872996509075,
+          -0.03328431025147438,
+          -0.007820303551852703,
+          -0.04431505501270294,
+          0.006654118187725544,
+          0.0023580959532409906,
+          0.01473509892821312,
+          -0.01882360875606537,
+          0.04211987927556038,
+          -0.006077885162085295,
+          -0.001488601672463119,
+          -0.018988246098160744,
+          -0.02525820955634117,
+          -0.01018697489053011,
+          -0.02695946767926216,
+          0.02335115149617195,
+          0.01634031906723976,
+          0.023159075528383255,
+          0.015325050801038742,
+          -0.012313548475503922,
+          0.02746710181236267,
+          0.029717154800891876,
+          0.0010469945846125484,
+          -0.006835905835032463,
+          -0.038168568164110184,
+          -0.03122633509337902,
+          0.008444556035101414,
+          -0.017726020887494087,
+          0.013431714847683907,
+          -0.00541247334331274,
+          -0.01422746479511261,
+          -0.004575563594698906,
+          -0.023762747645378113,
+          0.019043125212192535,
+          0.02012699283659458,
+          0.006005856208503246,
+          0.002215752610936761,
+          -0.032433681190013885,
+          0.038909438997507095,
+          -0.01458417996764183,
+          -0.00680160615593195,
+          0.011696156114339828,
+          -0.018590370193123817,
+          -0.04447969049215317,
+          0.014968335628509521,
+          -0.008135859854519367,
+          -0.01580524444580078,
+          6.827759352745488e-05,
+          0.007491027936339378,
+          -0.006427740678191185,
+          -0.030046429485082626,
+          -0.03558924049139023,
+          -0.005481072701513767,
+          0.00847199559211731,
+          -0.009775379672646523,
+          0.04601631313562393,
+          0.03797649219632149,
+          -0.012704563327133656,
+          0.002215752610936761,
+          -0.02090902253985405,
+          0.040967416018247604,
+          9.405587479704991e-05,
+          -0.00932262558490038,
+          -0.0003839409037027508,
+          -0.008094700053334236,
+          -0.005031748209148645,
+          0.024407578632235527,
+          -0.00319843553006649,
+          0.016024762764573097,
+          0.007676245644688606,
+          0.014597900211811066,
+          -0.006715857423841953,
+          -0.013863889500498772,
+          0.0021608732640743256,
+          0.02388622611761093,
+          -0.018384573981165886,
+          0.006842765491455793,
+          -0.027906136587262154,
+          0.0010975864715874195,
+          0.011744175106287003,
+          0.006232233252376318,
+          0.03454653546214104,
+          0.0224593635648489,
+          0.029634835198521614,
+          -0.03369590640068054,
+          -0.011627556756138802,
+          -0.03471117094159126,
+          -0.03907407820224762,
+          -0.015105534344911575,
+          0.004887689836323261,
+          -0.0009843978332355618,
+          -0.025189610198140144,
+          -0.009164847433567047,
+          -0.0026496422942727804,
+          -0.007662525400519371,
+          0.026835989207029343,
+          -0.015283891931176186,
+          0.007532187271863222,
+          0.00599556602537632,
+          -0.028207972645759583,
+          -0.03177512809634209,
+          -0.0002012956392718479,
+          0.003044087439775467,
+          -0.01789066009223461,
+          -0.0331745520234108,
+          -0.004390345886349678,
+          0.01697143167257309,
+          0.011174802668392658,
+          -0.041653405874967575,
+          0.00736754946410656,
+          0.021402936428785324,
+          -0.005913247354328632,
+          0.011325720697641373,
+          -0.05452260747551918,
+          0.007024553604424,
+          -0.005014598369598389,
+          -0.0038758523296564817,
+          0.016697034239768982,
+          -0.03117145597934723,
+          0.006071025040000677,
+          0.03292759507894516,
+          0.025422846898436546,
+          -0.03965030983090401,
+          0.00899334903806448,
+          -0.007168611511588097,
+          0.0097959591075778,
+          -0.007655665744096041,
+          0.055702514946460724,
+          -0.03641243278980255,
+          -0.014803697355091572,
+          -0.030595222488045692,
+          -0.0227886401116848,
+          -0.03490325063467026,
+          0.0001791081012925133,
+          -0.05101033300161362,
+          0.003769523464143276,
+          -0.0037043543998152018,
+          0.017465345561504364,
+          0.017588824033737183,
+          -0.008952190168201923,
+          -0.018947087228298187,
+          0.004757351242005825,
+          -0.04008934646844864,
+          -0.028976282104849815,
+          0.0049494290724396706,
+          0.026534153148531914,
+          -0.031363531947135925,
+          -0.013459154404699802,
+          -0.0084102563560009,
+          -0.04329978674650192,
+          0.0007151461904868484,
+          -0.0018041577422991395,
+          0.021814530715346336,
+          0.0021831681951880455,
+          -0.0057554692029953,
+          -0.03226904198527336,
+          -0.01306127943098545,
+          0.00024459886481054127,
+          -0.00391358183696866,
+          0.03366846591234207,
+          0.006746727041900158,
+          -0.02487405389547348,
+          -0.028235411271452904,
+          -0.006650688126683235,
+          -0.0272064246237278,
+          -0.031418412923812866,
+          0.027343623340129852,
+          0.015846405178308487,
+          -0.016326598823070526,
+          0.0035671561490744352,
+          -0.025121010839939117,
+          0.017698582261800766,
+          0.01223122887313366,
+          0.014776257798075676,
+          0.030677542090415955,
+          -0.009988036938011646,
+          -0.0024575646966695786,
+          -0.020991342142224312,
+          -0.05183352157473564,
+          0.024105742573738098,
+          -0.012368427589535713,
+          0.007470448035746813,
+          0.034189820289611816,
+          -0.008499435149133205,
+          0.03196720778942108,
+          0.028317730873823166,
+          0.0037043543998152018,
+          0.016203120350837708,
+          -0.0460437536239624,
+          0.03520508483052254,
+          0.018329694867134094,
+          -0.06069653108716011,
+          0.016326598823070526,
+          0.019070565700531006,
+          0.013219057582318783,
+          0.049857866019010544,
+          -0.028262851759791374,
+          -0.006492909975349903,
+          -0.004112519323825836,
+          0.004455515183508396,
+          -0.027576860040426254,
+          0.018453173339366913,
+          0.007415568456053734,
+          -0.03978750854730606,
+          0.007772284094244242,
+          -0.013925628736615181,
+          -0.014001087285578251,
+          -0.004304597154259682,
+          0.0400344654917717,
+          -0.03635755181312561,
+          -0.02261028066277504,
+          -0.013088718988001347,
+          -0.0036494750529527664,
+          -0.002870874712243676,
+          0.02812565304338932,
+          0.03709842264652252,
+          -0.018755009397864342,
+          0.010468230582773685,
+          -0.007443008478730917,
+          -0.014254904352128506,
+          0.00044632324716076255,
+          0.01163441687822342,
+          -0.016930270940065384,
+          0.003308194223791361,
+          0.007093152496963739,
+          -0.0071343122981488705,
+          -0.02121085859835148,
+          0.022267285734415054,
+          0.024256661534309387,
+          -0.013500313274562359,
+          0.00056208431487903,
+          -0.0007906052633188665,
+          0.030787300318479538,
+          -0.03811369091272354,
+          -0.03940335288643837,
+          -0.01440582238137722,
+          -0.00802610069513321,
+          0.034491654485464096,
+          -0.03794905170798302,
+          0.0024078304413706064,
+          -0.006842765491455793,
+          -0.005597691051661968,
+          -0.005782908760011196,
+          0.0013642656849697232,
+          -0.011956832371652126,
+          -0.01691655069589615,
+          -0.02782381698489189,
+          -0.016504956409335136,
+          -0.02723386511206627,
+          -0.007161751855164766,
+          -0.053287822753190994,
+          0.01197741273790598,
+          -0.010111515410244465,
+          -0.020826702937483788,
+          0.04072045907378197,
+          -0.00449667451903224,
+          0.006071025040000677,
+          -0.01583268493413925,
+          0.007189191412180662,
+          -0.050598736852407455,
+          -0.01819249615073204,
+          -0.0011936252703890204,
+          0.04316258803009987,
+          0.0065649389289319515,
+          -0.022171247750520706,
+          -0.0007284372695721686,
+          -0.022692600265145302,
+          -0.008863010443747044,
+          0.006355711724609137,
+          0.013520893640816212,
+          -0.027700338512659073,
+          -0.022692600265145302,
+          0.009562722407281399,
+          0.015407370403409004,
+          0.024613376706838608,
+          0.04428761452436447,
+          -0.011538377963006496,
+          -0.061190444976091385,
+          0.01717722788453102,
+          -0.015036934986710548,
+          0.028029615059494972,
+          -0.005323294550180435,
+          -0.0008073262870311737,
+          -0.01771230250597,
+          -0.014762538485229015,
+          -0.004976868629455566,
+          0.019948633387684822,
+          -0.006108754780143499,
+          0.0035568661987781525,
+          0.031390972435474396,
+          -0.016422636806964874,
+          0.0037146443501114845,
+          -0.0012545069912448525,
+          0.0039787511341273785,
+          -0.01640891842544079,
+          -0.05007738247513771,
+          0.025793282315135002,
+          -0.040939975529909134,
+          0.010536829940974712,
+          -0.0021385785657912493,
+          -0.02108738012611866,
+          -0.01575036533176899,
+          0.01370611134916544,
+          -0.015668047592043877,
+          0.013500313274562359,
+          0.030046429485082626,
+          0.02266516163945198,
+          0.045055925846099854,
+          -0.036933783441782,
+          -0.007285230327397585,
+          0.03844296559691429,
+          -0.014309783466160297,
+          -0.003944451455026865,
+          -0.013129877857863903,
+          0.0026324924547225237,
+          0.027576860040426254,
+          -0.01006349641829729,
+          0.04911699518561363,
+          -0.006359141785651445,
+          0.009631321765482426,
+          0.010660308413207531,
+          -0.00233408622443676,
+          -0.012642824091017246,
+          0.04192780330777168,
+          -0.029003722593188286,
+          0.012937800958752632,
+          0.012073451653122902,
+          -0.011099343188107014,
+          0.020442547276616096,
+          -0.0004986301064491272,
+          0.002894884441047907,
+          0.015517128631472588,
+          0.008616053499281406,
+          -0.04483640938997269,
+          -0.006372861564159393,
+          -0.01175103522837162,
+          -0.020497428253293037,
+          0.022500522434711456,
+          -0.013692391104996204,
+          -0.019619358703494072,
+          -0.03147329390048981,
+          -0.019770275801420212,
+          -0.0037180741783231497,
+          -0.003611745545640588,
+          0.00996059738099575,
+          -0.0018041577422991395,
+          0.026616472750902176,
+          -0.03605571761727333,
+          0.011435478925704956,
+          0.030458025634288788,
+          -0.005467352457344532,
+          -0.024297820404171944,
+          -0.03391542285680771,
+          0.025930481031537056,
+          -0.02073066495358944,
+          -0.019646797329187393,
+          0.004397206008434296,
+          0.027906136587262154,
+          0.012807462364435196,
+          0.002601622836664319,
+          -0.005957836750894785,
+          0.00736754946410656,
+          -0.01684795320034027,
+          -0.007840882986783981,
+          -0.024407578632235527,
+          -0.04244915768504143,
+          0.015489689074456692,
+          -0.004716191906481981,
+          -0.011428618803620338,
+          0.016065921634435654,
+          -0.02248680219054222,
+          -0.01577780582010746,
+          0.008458275347948074,
+          0.01261538453400135,
+          0.00167038943618536,
+          0.002545028692111373,
+          -0.006139624398201704,
+          0.0024181201588362455,
+          -0.0041605387814342976,
+          0.008375956676900387,
+          0.04516568407416344,
+          -0.016299158334732056,
+          -0.006609528791159391,
+          0.035863637924194336,
+          0.027576860040426254,
+          -0.008417116478085518,
+          -0.010660308413207531,
+          -0.013726690784096718,
+          0.0022929266560822725,
+          -0.033778224140405655,
+          0.023749027401208878,
+          0.051559124141931534,
+          0.007326389662921429,
+          -0.024531057104468346,
+          -0.030650101602077484,
+          -0.015709206461906433,
+          -0.005398753564804792,
+          0.049171872437000275,
+          0.02446245774626732,
+          -0.043683942407369614,
+          -0.02601279877126217,
+          0.03161048889160156,
+          0.04469921067357063,
+          0.0077517046593129635,
+          -0.011997992172837257,
+          0.0050557577051222324,
+          0.030293386429548264,
+          0.01628543995320797,
+          -0.03528740629553795,
+          0.011353160254657269,
+          0.028592128306627274,
+          -0.0013273936929181218,
+          -0.012265528552234173,
+          -0.004163968842476606,
+          -0.006300832144916058,
+          0.015366210602223873,
+          -0.008773831650614738,
+          -0.009302045218646526,
+          -0.016697034239768982,
+          -0.026712510734796524,
+          0.07606274634599686,
+          -0.00541247334331274,
+          0.022774919867515564,
+          0.008890450932085514,
+          0.00915798731148243,
+          0.04044606164097786,
+          0.003930731676518917,
+          -0.009494123049080372,
+          0.011867653578519821,
+          -0.006976534146815538,
+          -0.02621859684586525,
+          -0.011819634586572647,
+          -0.01605220139026642,
+          -0.0019053415162488818,
+          0.016450077295303345,
+          -0.006221943534910679,
+          0.03018362820148468,
+          0.03737282007932663,
+          0.013150458224117756,
+          -0.01854921132326126,
+          -0.01580524444580078,
+          0.00403363024815917,
+          0.01849433220922947,
+          0.03986983001232147,
+          -0.011373739689588547,
+          -0.0018436022801324725,
+          0.0026925168931484222,
+          -0.006414020899683237,
+          0.003155561164021492,
+          -0.011435478925704956,
+          -0.013712971471250057,
+          -0.008142719976603985,
+          0.022006608545780182,
+          0.0311988964676857,
+          -0.0031864307820796967,
+          -0.027508260682225227,
+          -0.004013050347566605,
+          0.022377043962478638,
+          0.01575036533176899,
+          0.0004411783011164516,
+          0.0066198185086250305,
+          0.014103986322879791,
+          0.006335131824016571,
+          -0.0015443384181708097,
+          0.02487405389547348,
+          -0.02645183354616165,
+          0.0016129376599565148,
+          0.021718492731451988,
+          -0.02431154064834118,
+          0.02436641976237297,
+          -0.011654996313154697,
+          -0.030759861692786217,
+          -0.027371061965823174,
+          -0.007141171954572201,
+          0.02764545939862728,
+          -0.02057974599301815,
+          -9.443102317163721e-05,
+          0.0246682558208704,
+          0.01406282652169466,
+          0.0007717405096627772,
+          0.0036494750529527664,
+          0.008506295271217823,
+          0.008835570886731148,
+          -0.02988179214298725,
+          -0.009535282850265503,
+          -0.00014116420061327517,
+          -0.013171037659049034,
+          0.04244915768504143,
+          -0.05485188588500023,
+          -0.013905048370361328,
+          -0.011847074143588543,
+          -0.02347462996840477,
+          -0.01819249615073204,
+          0.004479524679481983,
+          0.0017295562429353595,
+          -0.006379721220582724,
+          -0.024023424834012985,
+          0.0349581278860569,
+          0.00917170662432909,
+          -0.020113272592425346,
+          -0.019962353631854057,
+          0.03048546425998211,
+          -0.01867268979549408,
+          0.03254343941807747,
+          0.0033322039525955915,
+          -0.012622244656085968,
+          -0.005096917040646076,
+          -0.01267712377011776,
+          -0.02159501425921917,
+          -0.03572643920779228,
+          -0.020634625107049942,
+          0.04552239924669266,
+          0.015393650159239769,
+          0.003906721714884043,
+          -0.01572292670607567,
+          0.023282554000616074,
+          -0.008211318403482437,
+          0.009940017946064472,
+          0.011997992172837257,
+          0.05221767723560333,
+          0.0049631488509476185,
+          0.03630267456173897,
+          0.006163633894175291,
+          -0.06206851452589035,
+          0.0012793742353096604,
+          0.020236751064658165,
+          0.020140711218118668,
+          0.03397030010819435,
+          0.016450077295303345,
+          0.004887689836323261,
+          -0.021526414901018143,
+          -0.021197138354182243,
+          -0.04703158140182495,
+          -0.00927460566163063,
+          0.005168946459889412,
+          -0.011661856435239315,
+          -0.02812565304338932,
+          0.04722365736961365,
+          0.024736855179071426,
+          -0.003387083299458027,
+          0.018535491079092026,
+          -0.01009779516607523,
+          0.0016318024136126041,
+          0.01610708236694336,
+          -0.018425732851028442,
+          0.036220353096723557,
+          -0.005337014328688383,
+          0.03081474080681801,
+          -0.025546325370669365,
+          0.02034650929272175,
+          0.02182825095951557,
+          -0.012313548475503922,
+          0.034519094973802567,
+          0.005607980769127607,
+          -0.0623977929353714,
+          -0.0028760195709764957,
+          0.027906136587262154,
+          0.013452294282615185,
+          0.0015837829560041428,
+          0.0022826369386166334,
+          -0.01634031906723976,
+          0.019166603684425354,
+          -0.002447274746373296,
+          0.0115109384059906,
+          0.04524800181388855,
+          -0.0025621785316616297,
+          0.003800393082201481,
+          0.026794830337166786,
+          0.0058549377135932446,
+          -0.014296064153313637,
+          0.0036529048811644316,
+          -0.012039151974022388,
+          0.04524800181388855,
+          -0.03778441622853279,
+          -0.00043946332880295813,
+          0.0522451177239418,
+          -0.005186096299439669,
+          0.004208558239042759,
+          -0.015915004536509514,
+          -0.01867268979549408,
+          -0.005957836750894785,
+          0.02001723274588585,
+          0.004736771807074547,
+          0.04058326035737991,
+          -0.040336303412914276,
+          0.010145815089344978,
+          -0.020895302295684814,
+          0.0041742585599422455,
+          0.010776927694678307,
+          0.015764085575938225,
+          0.013390555046498775,
+          -0.038936879485845566,
+          -0.006983393803238869,
+          0.015283891931176186,
+          0.06398928910493851,
+          -0.006307692267000675,
+          -0.012491906061768532,
+          0.04601631313562393,
+          -0.005313004367053509,
+          -0.015078093856573105,
+          0.016422636806964874,
+          0.004225708078593016,
+          0.02848236821591854,
+          0.02001723274588585,
+          -0.02812565304338932,
+          -0.04137900844216347,
+          0.009137407876551151,
+          0.01473509892821312,
+          0.005309574771672487,
+          -0.001153323333710432,
+          -0.006846195552498102,
+          -0.013966787606477737,
+          -0.013054419308900833,
+          -0.0038209729827940464,
+          -0.007587066385895014,
+          0.023309992626309395,
+          -0.02352951094508171,
+          0.007635085843503475,
+          0.02949763648211956,
+          -0.013493454083800316,
+          -0.020360229536890984,
+          -0.005282134748995304,
+          -0.005127786658704281,
+          -0.002359810983762145,
+          -0.02481917478144169,
+          -0.010488810949027538,
+          -0.018988246098160744,
+          -0.019701676443219185,
+          0.04036374390125275,
+          0.016806792467832565,
+          0.005889237392693758,
+          0.009343205019831657,
+          -0.007292089983820915,
+          0.04500104486942291,
+          -0.028015894815325737,
+          0.05216279625892639,
+          -0.014872296713292599,
+          -0.003529426408931613,
+          -0.005518801975995302,
+          0.02654787339270115,
+          -0.014940896071493626,
+          0.019976073876023293,
+          -0.03215928375720978,
+          0.00703141326084733,
+          0.009144267067313194,
+          -0.005402183625847101,
+          -0.00858861394226551,
+          -0.01124340109527111,
+          0.001927636214531958,
+          -0.0026925168931484222,
+          0.02576584182679653,
+          0.033723343163728714,
+          0.010255573317408562,
+          0.004397206008434296,
+          -0.032790396362543106,
+          -0.005638850387185812,
+          -0.0007215773803181946,
+          -0.0030749570578336716,
+          -0.033723343163728714,
+          -0.018864767625927925,
+          -0.037510018795728683,
+          -0.013925628736615181,
+          0.009809679351747036,
+          -0.0035637260880321264,
+          0.021965449675917625,
+          0.019125444814562798,
+          -0.0017475634813308716,
+          -0.01605220139026642,
+          -0.013870748691260815,
+          0.006942234467715025,
+          0.013308236375451088,
+          -0.01565432734787464,
+          0.0015494833933189511,
+          0.0007811728864908218,
+          -0.010584849864244461,
+          0.021005060523748398,
+          -0.01989375427365303,
+          -0.025601204484701157,
+          -0.008273057639598846,
+          -0.03303735330700874,
+          0.00472648162394762,
+          0.0199211947619915,
+          -0.0038072532042860985,
+          0.01971539668738842,
+          0.009013929404318333,
+          0.002479859394952655,
+          -0.006678127683699131,
+          -0.03386054188013077,
+          -0.031418412923812866,
+          -0.034436777234077454,
+          -0.013857029378414154,
+          0.013994227163493633,
+          0.0008519157418049872,
+          0.02009955234825611,
+          0.008156439289450645,
+          0.00359802576713264,
+          0.018439453095197678,
+          -0.0007511607254855335,
+          -0.013356255367398262,
+          -0.013500313274562359,
+          -0.001734701101668179,
+          -0.01732814684510231,
+          0.0061979335732758045,
+          -0.03084217943251133,
+          0.0460437536239624,
+          -0.02230844460427761,
+          0.009343205019831657,
+          0.014940896071493626,
+          -0.009933157823979855,
+          -0.0026856570038944483,
+          -0.027247583493590355,
+          0.02248680219054222,
+          0.04596143215894699,
+          -0.013342536054551601,
+          0.033394068479537964,
+          0.03056778386235237,
+          0.0224593635648489,
+          -0.025450287386775017,
+          0.02759058028459549,
+          0.013664951547980309,
+          -0.008986488915979862,
+          0.030320826917886734,
+          -0.020442547276616096,
+          0.025875601917505264,
+          -0.007086292840540409,
+          -0.022884678095579147,
+          0.004294306971132755,
+          0.025367967784404755,
+          0.010866106487810612,
+          0.014186304993927479,
+          -0.01918032392859459,
+          0.004061069805175066,
+          0.0029669133946299553,
+          -0.019619358703494072,
+          0.04725109785795212,
+          0.002308361465111375,
+          0.021128538995981216,
+          0.0113943200558424,
+          -0.012773162685334682,
+          0.022034049034118652,
+          -0.016203120350837708,
+          -0.01933124102652073,
+          -0.0028365750331431627,
+          0.010927845723927021,
+          0.015379930846393108,
+          -0.048540763556957245,
+          0.03355870768427849,
+          0.028098214417696,
+          0.009645041078329086,
+          0.0058309282176196575,
+          0.008725812658667564,
+          -0.002208892721682787,
+          0.006352281663566828,
+          -0.00680160615593195,
+          -0.0009157987078651786,
+          0.044754087924957275,
+          -0.01175103522837162,
+          -0.0019019115716218948,
+          -0.0049871583469212055,
+          0.030101308599114418,
+          0.0017158363480120897,
+          -0.008252478204667568,
+          -0.008808131329715252,
+          -0.01458417996764183,
+          0.00013408990344032645,
+          0.00420512817800045,
+          -0.004887689836323261,
+          -0.016244279220700264,
+          -0.027096666395664215,
+          0.01319847721606493,
+          0.004808800760656595,
+          -0.03421725705265999,
+          -0.01060542929917574,
+          -0.02075810357928276,
+          0.010502530261874199,
+          -0.006544359494000673,
+          -0.0030458024702966213,
+          0.00018693269521463662,
+          -0.007923202589154243,
+          -0.014076546765863895,
+          0.03314711153507233,
+          0.003947881516069174,
+          0.04434249550104141,
+          -0.00342652783729136,
+          0.030979378148913383,
+          -0.02621859684586525,
+          0.03473861142992973,
+          -0.004194838460534811,
+          -0.015791526064276695,
+          0.02505241148173809,
+          -0.03498556837439537,
+          0.031089136376976967,
+          0.02225356549024582,
+          0.004983728751540184,
+          -0.0004171686014160514,
+          0.02949763648211956,
+          -0.04563215747475624,
+          -5.0806247600121424e-05,
+          -0.00553252175450325,
+          -0.0035843057557940483,
+          0.012148910202085972,
+          -0.01048195082694292,
+          -0.012965240515768528,
+          0.0027748357970267534,
+          0.015338771045207977,
+          0.02261028066277504,
+          0.03056778386235237,
+          0.00863663386553526,
+          -0.013116158545017242,
+          0.013520893640816212,
+          -0.0012802317505702376,
+          0.0026925168931484222,
+          -0.015969883650541306,
+          0.020181871950626373,
+          0.00204253988340497,
+          -0.03290015459060669,
+          -0.005786338821053505,
+          -0.06234291195869446,
+          -0.009075668640434742,
+          0.007154891733080149,
+          0.009151127189397812,
+          -0.004925419110804796,
+          -0.029717154800891876,
+          -0.038964319974184036,
+          0.017616262659430504,
+          0.02675366960465908,
+          -0.006623248569667339,
+          -0.008773831650614738,
+          -0.0224593635648489,
+          -0.016024762764573097,
+          0.03320199251174927,
+          -0.0007057138136588037,
+          0.012786882929503918,
+          0.004338896367698908,
+          0.03578132018446922,
+          -0.0038689924404025078,
+          -0.010715187527239323,
+          -0.04810172691941261,
+          -0.02949763648211956,
+          0.016244279220700264,
+          0.015475969761610031,
+          -0.009624461643397808,
+          0.019290082156658173,
+          -0.005782908760011196,
+          -0.02550516650080681,
+          -0.017245827242732048,
+          0.03594595566391945,
+          -0.012080310843884945,
+          0.028893964365124702,
+          0.01544852927327156,
+          -0.021512694656848907,
+          0.002509013982489705,
+          -0.0018195925513282418,
+          -0.009398084133863449,
+          0.005700589623302221,
+          0.02484661340713501,
+          0.0013719830894842744,
+          0.005611410830169916,
+          0.011922532692551613,
+          0.008252478204667568,
+          0.002426695078611374,
+          -0.022294726222753525,
+          -0.003380223410204053,
+          0.015215292572975159,
+          -0.03701610490679741,
+          0.008760112337768078,
+          -0.005282134748995304,
+          0.004908269736915827,
+          0.008382816798985004,
+          0.04346442595124245,
+          -0.005048897583037615,
+          0.005216965917497873,
+          -0.011483498848974705,
+          0.01699887029826641,
+          0.011606977321207523,
+          0.0033596435096114874,
+          -0.00541247334331274,
+          -0.010955285280942917,
+          -0.026794830337166786,
+          -0.008691512979567051,
+          0.01388446893543005,
+          0.012505626305937767,
+          0.01457046065479517,
+          0.007209771312773228,
+          0.004942568950355053,
+          -0.016834232956171036,
+          -0.006149914115667343,
+          0.010550550185143948,
+          0.01915288344025612,
+          -0.022061487659811974,
+          -0.02293955720961094,
+          0.01966051757335663,
+          -0.003011502791196108,
+          0.022322164848446846,
+          -0.021883130073547363,
+          -0.020675785839557648,
+          0.026287196204066277,
+          0.0712333619594574,
+          -0.0003322771517559886,
+          -0.010756347328424454,
+          -0.02228100597858429,
+          -0.011888233944773674,
+          -0.01664215512573719,
+          -0.017163507640361786,
+          0.027851257473230362,
+          0.03591851890087128,
+          -0.008238758891820908,
+          0.018466891720891,
+          -0.0020288201048970222,
+          0.0038038231432437897,
+          0.004218847956508398,
+          -0.030430585145950317,
+          0.01326021645218134,
+          -0.007552767172455788,
+          0.006366001442074776,
+          0.012999540194869041,
+          0.003989040851593018,
+          -0.008897310122847557,
+          -0.0018710419535636902,
+          0.009885137900710106,
+          0.012930940836668015,
+          -0.005299284588545561,
+          0.02126573771238327,
+          -0.0062253731302917,
+          0.009981176815927029,
+          -0.011318860575556755,
+          -0.018151337280869484,
+          0.00951470248401165,
+          0.01926264353096485,
+          -0.014474421739578247,
+          0.011936252936720848,
+          -0.007600786164402962,
+          0.03440933674573898,
+          -0.011421759612858295,
+          0.0029754883144050837,
+          -0.0011610407382249832,
+          -0.019454719498753548,
+          -0.002123143756762147,
+          0.0010718618286773562,
+          -0.002905174158513546,
+          0.02052486687898636,
+          0.004064499866217375,
+          0.03770209476351738,
+          0.021402936428785324,
+          -0.04862308129668236,
+          0.03633011132478714,
+          -0.009686200879514217,
+          -0.01388446893543005,
+          0.00857489462941885,
+          -0.009754800237715244,
+          -0.0019207763252779841,
+          -0.016271719709038734,
+          -0.014158865436911583,
+          0.013870748691260815,
+          -0.007086292840540409,
+          0.003745513968169689,
+          -0.004808800760656595,
+          0.015009495429694653,
+          0.0020545446313917637,
+          0.00772426463663578,
+          0.017259547486901283,
+          -0.006348851602524519,
+          -0.01661471463739872,
+          0.03520508483052254,
+          0.021869409829378128,
+          0.0004092368471901864,
+          0.0024215502198785543,
+          -0.004359476268291473,
+          0.02815309353172779,
+          0.009432383812963963,
+          -0.010042916052043438,
+          -0.015942443162202835,
+          -0.0019224913557991385,
+          -0.003222445258870721,
+          0.03657706826925278,
+          0.03180256858468056,
+          -0.02001723274588585,
+          0.0018933366518467665,
+          -0.0011258836602792144,
+          -0.009864558465778828,
+          0.009164847433567047,
+          -0.025903040543198586,
+          0.01816505566239357,
+          -0.006636968348175287,
+          0.007621366064995527,
+          0.02878420613706112,
+          -0.0044486550614237785,
+          0.012656544335186481,
+          0.013973647728562355,
+          0.007587066385895014,
+          0.003308194223791361,
+          0.006242522969841957,
+          0.018096456304192543,
+          -0.006667837966233492,
+          0.005230685696005821,
+          -0.029031163081526756,
+          -0.02075810357928276,
+          -0.034134939312934875,
+          0.01122968178242445,
+          0.011600117199122906,
+          -0.01343857403844595,
+          -0.014028526842594147,
+          0.009315765462815762,
+          0.026849709451198578,
+          0.005649140570312738,
+          -0.0046990420669317245,
+          -0.020620906725525856,
+          0.005553101655095816,
+          -0.005587401334196329,
+          -0.02711038663983345,
+          -0.01364437211304903,
+          0.02469569630920887,
+          0.0045275441370904446,
+          0.019070565700531006,
+          -0.008279917761683464,
+          0.0073538292199373245,
+          0.0004371052491478622,
+          -0.014611619524657726,
+          0.008279917761683464,
+          0.02845492959022522,
+          0.022870957851409912,
+          -0.008622913621366024,
+          0.0019224913557991385,
+          -0.0018333124462515116,
+          -0.007271510548889637,
+          0.021691052243113518,
+          -0.0008163299644365907,
+          -0.009974316693842411,
+          0.025079850107431412,
+          0.011668716557323933,
+          -0.016532396897673607,
+          0.014501861296594143,
+          -0.02594420127570629,
+          0.006297402549535036,
+          0.016175679862499237,
+          0.009603881277143955,
+          -0.005957836750894785,
+          -0.0030921068973839283,
+          0.016902832314372063,
+          0.03128121420741081,
+          0.006379721220582724,
+          0.0027782658580690622,
+          -0.021005060523748398,
+          -0.014639060012996197,
+          -1.834223439800553e-05,
+          0.00466131279245019,
+          0.042641233652830124,
+          -0.009775379672646523,
+          -0.018069017678499222,
+          0.013747270219027996,
+          -0.010433931834995747,
+          -0.0021728782448917627,
+          -0.014323503710329533,
+          0.01864525116980076,
+          0.005433053243905306,
+          0.015860125422477722,
+          0.007758564315736294,
+          -0.005210105795413256,
+          0.02314535528421402,
+          0.0026890868321061134,
+          -0.028674446046352386,
+          0.01834341324865818,
+          0.01613452099263668,
+          0.017012590542435646,
+          -0.006204793695360422,
+          -0.00229978677816689,
+          0.013685530982911587,
+          -0.007978081703186035,
+          0.016326598823070526,
+          0.030732421204447746,
+          0.03133609518408775,
+          -0.0006752729532308877,
+          -0.024119462817907333,
+          0.014611619524657726,
+          -0.003472832264378667,
+          -0.012046011164784431,
+          -0.0255600456148386,
+          0.018604090437293053,
+          -0.0039650313556194305,
+          -0.01631287857890129,
+          -0.009953737258911133,
+          -0.025614924728870392,
+          0.00036314676981419325,
+          -0.010598569177091122,
+          0.003944451455026865,
+          -0.011661856435239315,
+          0.03536972403526306,
+          0.012073451653122902,
+          0.019015686586499214,
+          0.0018607521196827292,
+          -0.02728874422609806,
+          -0.018178775906562805,
+          0.03226904198527336,
+          -0.012882920913398266,
+          -0.00656150933355093,
+          -0.0058309282176196575,
+          -0.0037146443501114845,
+          -0.03569899871945381,
+          0.0154348099604249,
+          0.027343623340129852,
+          0.013795290142297745,
+          0.0010615718783810735,
+          -0.024448739364743233,
+          -0.01557200774550438,
+          0.01249876618385315,
+          0.01422746479511261,
+          -0.026026519015431404,
+          0.027700338512659073,
+          -0.006441460456699133,
+          -0.016203120350837708,
+          0.011620696634054184,
+          -0.007813443429768085,
+          -0.010536829940974712,
+          0.00691136484965682,
+          -0.02767289988696575,
+          0.002752541098743677,
+          0.024435019120573997,
+          -0.019043125212192535,
+          0.00564571050927043,
+          0.012011712417006493,
+          0.035891078412532806,
+          0.013911908492445946,
+          0.01867268979549408,
+          -0.00025253064814023674,
+          -0.02914092130959034,
+          0.01291722059249878,
+          -0.00158464047126472,
+          -0.022377043962478638,
+          0.026794830337166786,
+          -0.015736646950244904,
+          0.017136069014668465,
+          -0.005803488660603762,
+          -0.011552097275853157,
+          0.020236751064658165,
+          0.013225916773080826,
+          0.01237528771162033,
+          -0.0002709666732698679,
+          0.004743631463497877,
+          0.007539046928286552,
+          -0.011346300132572651,
+          0.020181871950626373,
+          -0.013911908492445946,
+          -0.00023923955450300127,
+          -0.004709331784397364,
+          -0.008547455072402954,
+          -0.031418412923812866,
+          0.03427213802933693,
+          0.0014825991820544004,
+          -0.01492717582732439,
+          0.005913247354328632,
+          -0.025422846898436546,
+          -0.025971639901399612,
+          -0.006880495231598616,
+          -0.008725812658667564,
+          -0.0039787511341273785,
+          -0.0054227630607783794,
+          -0.007820303551852703,
+          -0.016889112070202827,
+          0.007792863994836807,
+          0.019399840384721756,
+          0.02764545939862728,
+          -0.005378173664212227,
+          0.0374002605676651,
+          0.01346601452678442,
+          0.02228100597858429,
+          -0.004397206008434296,
+          0.01006349641829729,
+          0.01565432734787464,
+          0.011819634586572647,
+          -0.015530848875641823,
+          0.03572643920779228,
+          -0.0014242900069803,
+          -0.03130865469574928,
+          0.002548458520323038,
+          -0.016957711428403854,
+          -0.012828041799366474,
+          -0.009603881277143955,
+          -0.004218847956508398,
+          0.025464005768299103,
+          0.0020785543601959944,
+          0.023995984345674515,
+          -0.011703016236424446,
+          -0.002551888581365347,
+          0.0007665955345146358,
+          -0.01557200774550438,
+          -0.01082494668662548,
+          -0.026739951223134995,
+          -0.009946877136826515,
+          -0.027096666395664215,
+          0.005443342961370945,
+          0.013514033518731594,
+          0.009020788595080376,
+          -0.00950098317116499,
+          -0.003594595706090331,
+          0.0013291086070239544,
+          -0.025464005768299103,
+          -0.02129317820072174,
+          0.01473509892821312,
+          -0.0007095725159160793,
+          0.004534404259175062,
+          0.010269293561577797,
+          0.015064374543726444,
+          -0.013054419308900833,
+          -0.013253356330096722,
+          -0.01337683480232954,
+          -0.007244070526212454,
+          -0.01974283717572689,
+          -0.0016472372226417065,
+          0.015297611244022846,
+          0.008266198448836803,
+          0.00876697152853012,
+          -0.014200025238096714,
+          0.01562688685953617,
+          0.0028468649834394455,
+          0.0038895721081644297,
+          0.014021667651832104,
+          0.034107498824596405,
+          -0.036522191017866135,
+          -0.0035071317106485367,
+          -0.029964111745357513,
+          0.02182825095951557,
+          0.012162630446255207,
+          0.020332789048552513,
+          -0.012299828231334686,
+          -0.008739531971514225,
+          -0.0362752340734005,
+          -0.0041605387814342976,
+          -0.018027858808636665,
+          0.00719605153426528,
+          0.013088718988001347,
+          -0.009192286990582943,
+          0.04672974348068237,
+          0.0196879580616951,
+          -0.025395406410098076,
+          -0.014446982182562351,
+          0.01989375427365303,
+          0.022857237607240677,
+          -0.006684987805783749,
+          0.02421550080180168,
+          0.0057280296459794044,
+          0.015036934986710548,
+          -0.012485045939683914,
+          -0.0071823312900960445,
+          -0.03081474080681801,
+          -0.013610072433948517,
+          0.007456728257238865,
+          0.005559961311519146,
+          0.011600117199122906,
+          0.003968460951000452,
+          0.009933157823979855,
+          -0.007676245644688606,
+          -0.02711038663983345,
+          0.027247583493590355,
+          0.018178775906562805,
+          0.022226126864552498,
+          -0.020716944709420204,
+          0.0006019576103426516,
+          -0.006770736537873745,
+          0.013932487927377224,
+          0.0048465305007994175,
+          0.0007541619706898928,
+          -0.02182825095951557,
+          0.035452041774988174,
+          -0.02979947254061699,
+          -0.016861671581864357,
+          -0.0007215773803181946,
+          -0.01789066009223461,
+          0.01664215512573719,
+          -0.013616932556033134,
+          -0.029415316879749298,
+          0.016573555767536163,
+          0.015544568188488483,
+          0.015311331488192081,
+          0.0035500063095241785,
+          0.015846405178308487,
+          -0.012416446581482887,
+          0.001987660536542535,
+          -0.022774919867515564,
+          -0.004122809041291475,
+          0.009507843293249607,
+          -0.025820722803473473,
+          -0.024078303948044777,
+          0.004119379445910454,
+          0.010502530261874199,
+          -0.017972977831959724,
+          0.039595432579517365,
+          0.011531517840921879,
+          -0.008828711695969105,
+          0.005649140570312738,
+          0.0005252122646197677,
+          -0.006383151281625032,
+          0.04700414091348648,
+          0.027014346793293953,
+          -0.022212406620383263,
+          0.013445434160530567,
+          -0.004071359988301992,
+          0.012828041799366474,
+          -0.004894549492746592,
+          0.017108628526329994,
+          -0.02012699283659458,
+          0.009850838221609592,
+          -0.011353160254657269,
+          0.006084744818508625,
+          0.007936921902000904,
+          -0.023872505873441696,
+          -0.02660275250673294,
+          -0.036467310041189194,
+          0.008965909481048584,
+          -0.023749027401208878,
+          -0.01312301866710186,
+          0.028619566932320595,
+          -0.011147363111376762,
+          -0.03161048889160156,
+          0.027700338512659073,
+          0.00910996738821268,
+          -0.014014807529747486,
+          0.024352699518203735,
+          0.019962353631854057,
+          0.0069628143683075905,
+          0.0067124273627996445,
+          -0.01048195082694292,
+          -0.013534612953662872,
+          0.0013625507708638906,
+          0.004215417895466089,
+          0.00524097541347146,
+          -0.029387878254055977,
+          -0.009905718266963959,
+          0.0016369473887607455,
+          0.01897452585399151,
+          0.00673300726339221,
+          0.01124340109527111,
+          -0.006029865704476833,
+          0.002486719284206629,
+          -6.742010737070814e-05,
+          0.010296733118593693,
+          -0.004287447314709425,
+          -0.0019259213004261255,
+          0.0127800228074193,
+          0.028647007420659065,
+          -0.024627096951007843,
+          0.008629773743450642,
+          -0.016655875369906425,
+          -0.015393650159239769,
+          0.003227590350434184,
+          0.007038273382931948,
+          0.0018093027174472809,
+          -0.007456728257238865,
+          -0.00771740498021245,
+          0.006897645071148872,
+          -0.008451416157186031,
+          0.0019173463806509972,
+          -0.02012699283659458,
+          0.00431831693276763,
+          0.027782658115029335,
+          0.007292089983820915,
+          -0.034491654485464096,
+          -0.0019430710235610604,
+          -0.015009495429694653,
+          -0.011895093135535717,
+          -0.009041368961334229,
+          0.018617810681462288,
+          0.01631287857890129,
+          -0.013562052510678768,
+          0.0067913164384663105,
+          0.002584473229944706,
+          0.027933575212955475,
+          -0.011483498848974705,
+          -0.007278370205312967,
+          0.01923520304262638,
+          -0.00697310408577323,
+          0.002812565304338932,
+          0.027810096740722656,
+          -0.003769523464143276,
+          -0.017040029168128967,
+          0.0008077550446614623,
+          0.00961760152131319,
+          0.0084102563560009,
+          -0.01798669807612896,
+          0.010344752110540867,
+          0.0020116702653467655,
+          -0.006071025040000677,
+          0.03421725705265999,
+          0.059544067829847336,
+          -0.0027542561292648315,
+          0.0035002718213945627,
+          0.04563215747475624,
+          -0.02034650929272175,
+          -0.027028067037463188,
+          -0.01006349641829729,
+          -0.014460701495409012,
+          0.012011712417006493,
+          0.011723595671355724,
+          -0.005168946459889412,
+          0.01336311548948288,
+          -4.842135604121722e-05,
+          0.030704982578754425,
+          -0.01819249615073204,
+          0.03888200223445892,
+          0.010852386243641376,
+          0.011312000453472137,
+          0.015160413458943367,
+          0.020030952990055084,
+          0.008540594950318336,
+          -0.0007533044554293156,
+          0.006180783733725548,
+          -0.014337223023176193,
+          0.006321412045508623,
+          -0.012800602242350578,
+          -0.012526205740869045,
+          0.014076546765863895,
+          0.0013908478431403637,
+          0.02797473594546318,
+          -0.007463587913662195,
+          -0.005103777162730694,
+          -0.014090266078710556,
+          -0.00472648162394762,
+          0.0034351027570664883,
+          0.012251809239387512,
+          -0.00010434574505779892,
+          -0.007648805622011423,
+          -0.025971639901399612,
+          0.00926088634878397,
+          -3.971247861045413e-05,
+          -0.014117705635726452,
+          0.0012853766093030572,
+          -0.0011653281981125474,
+          -0.014625339768826962,
+          0.026355795562267303,
+          -0.004942568950355053,
+          0.0037386538460850716,
+          0.011517798528075218,
+          0.0009732504840940237,
+          -0.0034848370123654604,
+          0.0023358012549579144,
+          0.011346300132572651,
+          -0.00023559521650895476,
+          -0.015640607103705406,
+          -0.004654452670365572,
+          -0.009226586669683456,
+          0.005529091693460941,
+          -0.006506629753857851,
+          0.009302045218646526,
+          0.003512276802212,
+          0.006307692267000675,
+          -0.01509181410074234,
+          -0.016573555767536163,
+          -0.012128330767154694,
+          -0.006784456316381693,
+          0.011620696634054184,
+          0.0010504245292395353,
+          -0.002776550827547908,
+          -0.004356046207249165,
+          -0.0005886664730496705,
+          -0.024256661534309387,
+          -0.007573346607387066,
+          0.003195005701854825,
+          -0.004441795405000448,
+          0.00019036265439353883,
+          0.003772953525185585,
+          -0.020264189690351486,
+          0.033064793795347214,
+          0.014090266078710556,
+          -0.008965909481048584,
+          0.006787886377424002,
+          -0.003872422268614173,
+          -0.00663010822609067,
+          -0.028619566932320595,
+          -0.0017681432655081153,
+          -0.01870013028383255,
+          0.019976073876023293,
+          0.025793282315135002,
+          -0.0227886401116848,
+          0.0012124900240451097,
+          -0.009013929404318333,
+          0.011456059291958809,
+          -0.0024747145362198353,
+          0.0008373384480364621,
+          0.028921402990818024,
+          -0.01679307222366333,
+          -0.019303802400827408,
+          -0.02350207045674324,
+          0.003423097776249051,
+          0.004486384801566601,
+          0.024887774139642715,
+          -0.0014148575719445944,
+          0.00871209241449833,
+          -0.011360020376741886,
+          -0.0012416447279974818,
+          0.016806792467832565,
+          0.01867268979549408,
+          0.021608734503388405,
+          -0.009254026226699352,
+          -0.0022003178019076586,
+          -0.011037603951990604,
+          -0.0013711255742236972,
+          0.026739951223134995,
+          0.003776383586227894,
+          -0.02540912665426731,
+          -0.008808131329715252,
+          -0.007051993161439896,
+          -0.005374743603169918,
+          0.004510394297540188,
+          -0.0026324924547225237,
+          0.002447274746373296,
+          0.001353118335828185,
+          -0.012210649438202381,
+          0.020442547276616096,
+          0.0015186137752607465,
+          0.0048396703787148,
+          -0.010728907771408558,
+          -0.013774710707366467,
+          -0.026657631620764732,
+          -0.01006349641829729,
+          -0.028647007420659065,
+          -0.003505416912958026,
+          0.0070657129399478436,
+          -0.005841217935085297,
+          0.0009269460570067167,
+          0.013959928415715694,
+          -0.0020408248528838158,
+          -0.005848078057169914,
+          0.0199211947619915,
+          -0.013095579110085964,
+          -0.010310453362762928,
+          -0.008197599090635777,
+          -0.017726020887494087,
+          0.005971556529402733,
+          -0.010296733118593693,
+          0.0003800822014454752,
+          -0.005144936498254538,
+          -0.0018264524405822158,
+          -0.04769013077020645,
+          -0.006715857423841953,
+          -0.014995775185525417,
+          -0.023364871740341187,
+          -0.006691847462207079,
+          0.018357133492827415,
+          0.007038273382931948,
+          -0.018206216394901276,
+          -0.008732672780752182,
+          -0.004747061524540186,
+          8.54139852890512e-06,
+          -0.03391542285680771,
+          0.023543229326605797,
+          0.0018213075818493962,
+          0.011922532692551613,
+          -0.012368427589535713,
+          0.012738863006234169,
+          0.007100012619048357,
+          0.020716944709420204,
+          -0.007326389662921429,
+          -0.01730070635676384,
+          -0.01920776255428791,
+          -0.0009861128637567163,
+          -0.010255573317408562,
+          -0.024325260892510414,
+          -0.01054369006305933,
+          0.0012836616951972246,
+          0.004342326428741217,
+          0.006846195552498102,
+          0.010818086564540863,
+          0.02558748424053192,
+          -0.012491906061768532,
+          0.002020245185121894,
+          0.019797716289758682,
+          0.002472999505698681,
+          0.02210264839231968,
+          0.01890592649579048,
+          0.007278370205312967,
+          -0.001317103742621839,
+          0.002605052897706628,
+          -0.0033390638418495655,
+          -0.01583268493413925,
+          0.007292089983820915,
+          -0.00865035317838192,
+          -0.03320199251174927,
+          0.025107290595769882,
+          0.027069225907325745,
+          0.010756347328424454,
+          -0.006897645071148872,
+          0.004397206008434296,
+          0.0056354207918047905,
+          0.008190738968551159,
+          0.02952507697045803,
+          -0.017383025959134102,
+          0.00253473874181509,
+          0.00788890291005373,
+          -0.011017024517059326,
+          -0.008321077562868595,
+          0.018933366984128952,
+          0.033064793795347214,
+          -0.005837787874042988,
+          -0.039924707263708115,
+          -0.00871209241449833,
+          0.016271719709038734,
+          -0.01406282652169466,
+          -0.015105534344911575,
+          -0.007552767172455788,
+          0.004613293334841728,
+          -0.020113272592425346,
+          0.0019619357772171497,
+          -0.003622035263106227,
+          -0.0028434349223971367,
+          0.008808131329715252,
+          0.028537249192595482,
+          0.003776383586227894,
+          0.02652043290436268,
+          -0.011805914342403412,
+          0.008019241504371166,
+          -0.026561593636870384,
+          0.011517798528075218,
+          -0.02073066495358944,
+          -0.021375495940446854,
+          0.006091604940593243,
+          0.0019173463806509972,
+          0.02807077392935753,
+          0.010516250506043434,
+          0.01923520304262638,
+          0.0023975404910743237,
+          0.00319843553006649,
+          -0.03358614444732666,
+          -0.022061487659811974,
+          -0.0015906429616734385,
+          -0.0195919182151556,
+          0.006890784949064255,
+          -0.006757016759365797,
+          0.014446982182562351,
+          0.016889112070202827,
+          0.015215292572975159,
+          -0.008760112337768078,
+          0.019276361912488937,
+          -2.2576088667847216e-05,
+          0.01712234877049923,
+          -0.02416062168776989,
+          0.00414681900292635,
+          0.02147153578698635,
+          -0.0009320910321548581,
+          0.017547663301229477,
+          0.0003359214751981199,
+          -0.0007018551114015281,
+          0.0069868238642811775,
+          -0.0046201529912650585,
+          -0.012903501279652119,
+          -0.006444890517741442,
+          0.023008156567811966,
+          -0.03163792937994003,
+          0.0110650435090065,
+          0.009020788595080376,
+          -0.006695277523249388,
+          -0.014954615384340286,
+          -0.007271510548889637,
+          0.03737282007932663,
+          0.004040490370243788,
+          -0.00011104487930424511,
+          0.014433261938393116,
+          0.019221482798457146,
+          0.007984941825270653,
+          0.012382147833704948,
+          -0.017081189900636673,
+          0.020854143425822258,
+          0.04041862115263939,
+          -0.007319530006498098,
+          -0.0038689924404025078,
+          0.007669385522603989,
+          0.008821851573884487,
+          -0.01037905178964138,
+          0.008444556035101414,
+          -0.004006190691143274,
+          0.002958338474854827,
+          0.009226586669683456,
+          -0.022212406620383263,
+          0.026438115164637566,
+          -0.022870957851409912,
+          -0.0019756555557250977,
+          -0.004510394297540188,
+          1.4858684153296053e-05,
+          0.004795080982148647,
+          -3.577338793547824e-05,
+          0.02022303082048893,
+          -0.003222445258870721,
+          -0.02197916992008686,
+          -0.006592378951609135,
+          -0.00663010822609067,
+          0.00737440912052989,
+          -0.02019559033215046,
+          0.0004523256793618202,
+          0.024352699518203735,
+          0.005223825573921204,
+          0.0037420839071273804,
+          -0.014001087285578251,
+          -0.00024116890563163906,
+          -0.0035534361377358437,
+          0.021348057314753532,
+          0.01676563359797001,
+          0.024942653253674507,
+          0.02639695443212986,
+          0.003330488922074437,
+          0.012416446581482887,
+          -0.007223491091281176,
+          0.003690634621307254,
+          -0.0243389792740345,
+          0.025985360145568848,
+          -0.011538377963006496,
+          -0.010468230582773685,
+          -0.003851842600852251,
+          -0.010056636296212673,
+          -0.010310453362762928,
+          0.004647592548280954,
+          0.01394620817154646,
+          0.010276153683662415,
+          0.026177437976002693,
+          -0.023762747645378113,
+          0.013342536054551601,
+          0.012135190889239311,
+          -0.018658969551324844,
+          0.025491446256637573,
+          -0.012423306703567505,
+          -0.006366001442074776,
+          -0.0016335173277184367,
+          -0.005961266346275806,
+          0.010811226442456245,
+          -0.021787092089653015,
+          0.009061948396265507,
+          -0.026136277243494987,
+          -0.006064165383577347,
+          -0.003941021393984556,
+          0.0014946040464565158,
+          -0.0014380097854882479,
+          -0.0005633705877698958,
+          -0.028592128306627274,
+          0.0006589806289412081,
+          0.008197599090635777,
+          5.399503788794391e-05,
+          0.014995775185525417,
+          -0.011085623875260353,
+          -0.008423976600170135,
+          -0.0052135358564555645,
+          -0.006201363634318113,
+          -0.008327937684953213,
+          0.03229648247361183,
+          0.003476262092590332,
+          -0.01122968178242445,
+          0.003707784228026867,
+          0.0034248128067702055,
+          -0.030979378148913383,
+          0.006472330074757338,
+          0.008259338326752186,
+          -0.017904380336403847,
+          0.020387668162584305,
+          0.002598193008452654,
+          0.015997322276234627,
+          -0.01813761703670025,
+          0.007957502268254757,
+          -0.017561383545398712,
+          -0.012052871286869049,
+          8.306928975798655e-07,
+          -0.019290082156658173,
+          0.012464466504752636,
+          -0.02352951094508171,
+          0.022130087018013,
+          -0.02070322446525097,
+          -0.003759233746677637,
+          -0.02344719134271145,
+          0.02055230736732483,
+          -0.0018779018428176641,
+          0.007504747714847326,
+          0.013500313274562359,
+          0.0017972978530451655,
+          0.004441795405000448,
+          0.016600996255874634,
+          0.0035431464202702045,
+          0.014515581540763378,
+          0.01798669807612896,
+          0.02233588509261608,
+          0.006811895873397589,
+          0.0033184841740876436,
+          -0.020319068804383278,
+          0.004229137673974037,
+          -0.007346969563513994,
+          0.05205303803086281,
+          0.005710879806429148,
+          -0.007984941825270653,
+          0.00466817244887352,
+          0.0035362865310162306,
+          0.013445434160530567,
+          0.0038141130935400724,
+          0.02129317820072174,
+          -0.016985150054097176,
+          0.014337223023176193,
+          -0.020826702937483788,
+          0.00656150933355093,
+          0.017643703147768974,
+          0.03193976730108261,
+          0.009391224011778831,
+          -0.012505626305937767,
+          -0.020140711218118668,
+          0.011353160254657269,
+          -0.008273057639598846,
+          0.00014716661826241761,
+          -0.018562931567430496,
+          0.007244070526212454,
+          -0.007744844537228346,
+          0.011593257077038288,
+          0.002666792133823037,
+          0.02317279390990734,
+          0.012279248796403408,
+          -0.013212197460234165,
+          0.00680160615593195,
+          -0.009507843293249607,
+          -0.010008616372942924,
+          0.006297402549535036,
+          -0.03462885320186615,
+          -0.02225356549024582,
+          0.009809679351747036,
+          0.030430585145950317,
+          -0.0020459697116166353,
+          -0.03144585341215134,
+          -0.0038758523296564817,
+          -0.015242732129991055,
+          -0.025971639901399612,
+          0.017780901864171028,
+          0.02952507697045803,
+          -0.011154222302138805,
+          0.0038278328720480204,
+          -0.00014041388931218535,
+          0.009603881277143955,
+          0.008135859854519367,
+          0.021348057314753532,
+          -0.008177019655704498,
+          -0.009624461643397808,
+          0.007244070526212454,
+          0.024242941290140152,
+          -0.00978223979473114,
+          0.022006608545780182,
+          -0.014103986322879791,
+          0.009597022086381912,
+          -0.018302254378795624,
+          -0.00991943757981062,
+          -0.01559944823384285,
+          -6.082815525587648e-05,
+          -0.0032567449379712343,
+          0.003855272429063916,
+          -0.00950098317116499,
+          -0.0010512820445001125,
+          0.01026243343949318,
+          0.003989040851593018,
+          -0.004352616146206856,
+          0.00851315539330244,
+          0.005234115291386843,
+          0.01666959375143051,
+          0.003155561164021492,
+          -0.0362752340734005,
+          0.005402183625847101,
+          0.004050780087709427,
+          -0.0024284101091325283,
+          0.01572292670607567,
+          0.02705550752580166,
+          0.00799180194735527,
+          0.014282343909144402,
+          0.015311331488192081,
+          0.018000418320298195,
+          0.014296064153313637,
+          -0.010420211590826511,
+          0.0020939891692250967,
+          0.016203120350837708,
+          -0.0017338435864076018,
+          -0.010660308413207531,
+          -0.004808800760656595,
+          -0.024229221045970917,
+          0.030320826917886734,
+          0.013678671792149544,
+          -0.008506295271217823,
+          -0.027727779000997543,
+          -0.00933634489774704,
+          0.020620906725525856,
+          0.002779980655759573,
+          -0.011462918482720852,
+          -0.006952524185180664,
+          -0.007216630969196558,
+          0.018947087228298187,
+          0.0019070565467700362,
+          0.0015417659888044,
+          -0.005203245673328638,
+          -0.007957502268254757,
+          0.0195919182151556,
+          -0.018425732851028442,
+          -0.006050445605069399,
+          -0.0074224285781383514,
+          0.00836909655481577,
+          0.00043989208643324673,
+          0.03904663771390915,
+          0.009425523690879345,
+          0.030650101602077484,
+          0.0009218011400662363,
+          -0.018466891720891,
+          0.007477307692170143,
+          0.008115280419588089,
+          0.00893160980194807,
+          0.0305403433740139,
+          -0.017904380336403847,
+          0.003666624892503023,
+          0.013534612953662872,
+          0.0007691680220887065,
+          0.0005582256126217544,
+          0.0005740891792811453,
+          0.004640732891857624,
+          -0.004657882731407881,
+          -0.015105534344911575,
+          -0.0008634918485768139,
+          -0.01780834048986435,
+          0.016011042520403862,
+          0.012491906061768532,
+          0.006760446820408106,
+          -0.01129142101854086,
+          0.003366503631696105,
+          0.0092334458604455,
+          -0.015544568188488483,
+          -0.00865035317838192,
+          0.007854603230953217,
+          -0.01216948963701725,
+          0.017437905073165894,
+          -0.00021179989562369883,
+          -0.018000418320298195,
+          -0.002262057038024068,
+          0.007635085843503475,
+          0.005974986590445042,
+          0.008087839931249619,
+          -0.02281607873737812,
+          -0.016600996255874634,
+          -0.0032207302283495665,
+          -0.02385878562927246,
+          -0.02764545939862728,
+          0.0025827581994235516,
+          0.009535282850265503,
+          -0.007518467493355274,
+          -0.00851315539330244,
+          0.01920776255428791,
+          0.011888233944773674,
+          0.023913664743304253,
+          -0.024640817195177078,
+          0.019468439742922783,
+          -0.00032477412605658174,
+          -0.016189400106668472,
+          0.004585853312164545,
+          -0.025285648182034492,
+          0.02436641976237297,
+          0.0011018739314749837,
+          0.00720291119068861,
+          -0.0031212614849209785,
+          -0.006482620257884264,
+          0.002944618696346879,
+          -0.007120592053979635,
+          0.009974316693842411,
+          0.0017921528778970242,
+          -0.020634625107049942,
+          -0.0010975864715874195,
+          -0.007285230327397585,
+          -0.009315765462815762,
+          0.011819634586572647,
+          -0.027631739154458046,
+          -0.0007005688967183232,
+          -0.00535416416823864,
+          -0.0031932906713336706,
+          -0.003143556183204055,
+          0.017136069014668465,
+          -0.022555401548743248,
+          0.035863637924194336,
+          0.008225038647651672,
+          0.012629104778170586,
+          -0.00316928094252944,
+          0.01607964187860489,
+          0.008938469924032688,
+          -0.0029497635550796986,
+          -0.0013676956295967102,
+          -0.007470448035746813,
+          -0.0061670639552176,
+          0.011545238085091114,
+          0.0025467437226325274,
+          -0.012457606382668018,
+          0.009912577457726002,
+          -0.0038038231432437897,
+          -0.02078554406762123,
+          -0.008773831650614738,
+          0.011799054220318794,
+          0.010975864715874195,
+          -0.002409545239061117,
+          0.005546241533011198,
+          0.0067364368587732315,
+          0.010111515410244465,
+          -0.004966578911989927,
+          -0.001727841212414205,
+          0.004486384801566601,
+          -0.017396746203303337,
+          -0.006407161243259907,
+          -0.014968335628509521,
+          -0.009631321765482426,
+          0.020799264311790466,
+          0.01849433220922947,
+          0.007497887592762709,
+          5.8094909036299214e-05,
+          -0.0019001965411007404,
+          -0.009116827510297298,
+          -0.0002758972404990345,
+          0.011826494708657265,
+          -0.02848236821591854,
+          0.019825154915452003,
+          0.01789066009223461,
+          0.022870957851409912,
+          -0.028180532157421112,
+          -0.007429288700222969,
+          0.0007117162458598614,
+          -0.007003973703831434,
+          -0.017314426600933075,
+          -0.0033973732497543097,
+          -0.008849291130900383,
+          0.014501861296594143,
+          -0.02004467323422432,
+          0.01509181410074234,
+          0.0022894968278706074,
+          0.0016755344113335013,
+          -0.0011944827856495976,
+          0.010838666930794716,
+          0.007175471633672714,
+          -0.0037180741783231497,
+          -0.002824570285156369,
+          -0.018274815753102303,
+          -0.007971221581101418,
+          0.02182825095951557,
+          0.002627347595989704,
+          -0.023433471098542213,
+          0.002495294203981757,
+          0.004558413755148649,
+          -0.025875601917505264,
+          0.006925084628164768,
+          -0.037564896047115326,
+          -0.009432383812963963,
+          -0.015407370403409004,
+          0.0019790856167674065,
+          -0.02009955234825611,
+          0.006221943534910679,
+          0.001680679270066321,
+          0.010132094845175743,
+          0.003201865591108799,
+          -0.007971221581101418,
+          -0.00945296324789524,
+          0.002495294203981757,
+          -0.004530974198132753,
+          0.0019756555557250977,
+          0.01044079102575779,
+          0.005916676949709654,
+          -0.02842748910188675,
+          -0.020538587123155594,
+          0.005388463847339153,
+          -0.014639060012996197,
+          -0.0007507320260629058,
+          -0.005446773022413254,
+          0.00019089858687948436,
+          -0.00044417951721698046,
+          -0.004589283373206854,
+          0.007587066385895014,
+          0.014625339768826962,
+          -0.00019818724831566215,
+          0.009754800237715244,
+          0.005326724145561457,
+          -0.0005595118855126202,
+          0.020497428253293037,
+          -9.680251423560549e-06,
+          0.0218968503177166,
+          0.024270381778478622,
+          -0.013342536054551601,
+          0.02314535528421402,
+          -0.009933157823979855,
+          0.013699251227080822,
+          -0.008698373101651669,
+          0.017040029168128967,
+          -0.0031006818171590567,
+          0.012121470645070076,
+          -0.006451750639826059,
+          -0.011757895350456238,
+          -0.006077885162085295,
+          0.009686200879514217,
+          -0.0006636968464590609,
+          -0.010193834081292152,
+          0.019962353631854057,
+          -0.014789978042244911,
+          0.011558957397937775,
+          -0.0038827122189104557,
+          0.001445727190002799,
+          0.0020442549139261246,
+          -0.0004028056573588401,
+          0.03125377371907234,
+          0.00919914711266756,
+          -0.00639001140370965,
+          -0.017822060734033585,
+          0.006846195552498102,
+          -0.029387878254055977,
+          -0.011737314984202385,
+          -0.004115949384868145,
+          -0.01474881824105978,
+          -0.0022963567171245813,
+          -0.0019979504868388176,
+          0.031363531947135925,
+          -0.005192955955862999,
+          -0.020675785839557648,
+          -0.005196386016905308,
+          -0.01664215512573719,
+          -0.00736754946410656,
+          -0.00799180194735527,
+          -0.0062528131529688835,
+          0.003152131102979183,
+          -8.617768617114052e-05,
+          -0.023378591984510422,
+          0.009116827510297298,
+          -0.0047127618454396725,
+          0.017437905073165894,
+          -0.001723553752526641,
+          0.0224593635648489,
+          -0.020154431462287903,
+          0.011888233944773674,
+          -0.009610741399228573,
+          0.002551888581365347,
+          0.018439453095197678,
+          0.0027748357970267534,
+          -0.0068153259344398975,
+          0.0040953694842755795,
+          0.015695486217737198,
+          -0.015517128631472588,
+          -0.0025467437226325274,
+          -0.011716735549271107,
+          0.02121085859835148,
+          3.5398235922912136e-05,
+          -0.007779144216328859,
+          -0.015050654299557209,
+          -0.02359810844063759,
+          0.016065921634435654,
+          -0.006321412045508623,
+          0.014200025238096714,
+          0.01181277446448803,
+          0.02362554892897606,
+          0.0022243275307118893,
+          -8.730314584681764e-05,
+          0.009103108197450638,
+          -0.007758564315736294,
+          0.008863010443747044,
+          0.021183418110013008,
+          0.0035362865310162306,
+          -0.005745179019868374,
+          0.006293972488492727,
+          -0.0014320074114948511,
+          0.02261028066277504,
+          -0.018562931567430496,
+          0.001973940758034587,
+          0.014762538485229015,
+          0.0020511148031800985,
+          -0.010639728978276253,
+          0.015942443162202835,
+          -0.0013462584465742111,
+          -0.0037180741783231497,
+          -0.01750650443136692,
+          0.0031349812634289265,
+          -0.00891789048910141,
+          0.009144267067313194,
+          -0.01352775376290083,
+          0.010015476495027542,
+          -0.02258284203708172,
+          0.007401848677545786,
+          0.00016388767107855529,
+          -0.002584473229944706,
+          0.02621859684586525,
+          -0.006465470418334007,
+          0.0009226586553268135,
+          0.030403144657611847,
+          -0.0035843057557940483,
+          -0.01966051757335663,
+          0.00133854104205966,
+          -0.0032790396362543106,
+          -0.004242857918143272,
+          0.00691136484965682,
+          0.0015572007978335023,
+          -0.016573555767536163,
+          -0.005491362418979406,
+          0.014213744550943375,
+          0.02565608359873295,
+          0.0015229012351483107,
+          -0.022377043962478638,
+          0.002049399772658944,
+          0.01572292670607567,
+          -0.0029617685358971357,
+          -0.0002458851085975766,
+          0.005820638500154018,
+          0.009789098985493183,
+          0.0006461183074861765,
+          -0.006595808546990156,
+          0.010866106487810612,
+          -0.02797473594546318,
+          -0.008554314263164997,
+          -0.01640891842544079,
+          -0.008760112337768078,
+          -0.04157108813524246,
+          0.024435019120573997,
+          0.00459957355633378,
+          0.003469402203336358,
+          0.008965909481048584,
+          0.005529091693460941,
+          -0.004009620752185583,
+          -0.018933366984128952,
+          -0.002056259661912918,
+          -0.006376291625201702,
+          0.01130514033138752,
+          -0.008828711695969105,
+          -0.007957502268254757,
+          0.005559961311519146,
+          0.0029754883144050837,
+          -0.005275275092571974,
+          0.024092022329568863,
+          -0.001313673797994852,
+          0.0031075417064130306,
+          0.0038106830324977636,
+          -0.004157108720391989,
+          -0.0013128162827342749,
+          0.015640607103705406,
+          0.014090266078710556,
+          -4.6465207560686395e-05,
+          0.009672480635344982,
+          0.008259338326752186,
+          0.001888191793113947,
+          -0.006403731182217598,
+          -0.009391224011778831,
+          0.007024553604424,
+          0.018206216394901276,
+          -0.0035465762484818697,
+          -0.004294306971132755,
+          0.019605638459324837,
+          -0.014529300853610039,
+          -0.0008008951554074883,
+          0.012642824091017246,
+          -0.015201572328805923,
+          0.017904380336403847,
+          0.008972769603133202,
+          0.011538377963006496,
+          -0.010509390383958817,
+          -0.04598887264728546,
+          0.004139958880841732,
+          -0.00737440912052989,
+          0.009727359749376774,
+          0.0072372108697891235,
+          0.02248680219054222,
+          -0.013486593961715698,
+          -0.03534228354692459,
+          0.004805370699614286,
+          0.00216258829459548,
+          0.020442547276616096,
+          0.008122139610350132,
+          0.018809888511896133,
+          0.008629773743450642,
+          0.01559944823384285,
+          -0.006674697622656822,
+          0.009638180956244469,
+          0.022857237607240677,
+          -0.011997992172837257,
+          -0.014213744550943375,
+          -0.004242857918143272,
+          0.0022792068775743246,
+          -0.005628560669720173,
+          -0.009768519550561905,
+          0.001305956393480301,
+          -0.013486593961715698,
+          0.0048465305007994175,
+          0.01422746479511261,
+          0.015078093856573105,
+          0.010502530261874199,
+          0.006492909975349903,
+          0.0033390638418495655,
+          -0.003954741172492504,
+          0.0006765591679140925,
+          0.006506629753857851,
+          0.018535491079092026,
+          -0.013877608813345432,
+          0.026163717731833458,
+          -0.0008446270949207246,
+          0.013074998743832111,
+          0.005621700547635555,
+          0.003694064449518919,
+          0.005162086337804794,
+          0.008698373101651669,
+          0.0032258753199130297,
+          0.013555193319916725,
+          -0.004122809041291475,
+          0.0014294348657131195,
+          0.0016095077153295279,
+          -0.008780691772699356,
+          0.007882042787969112,
+          -0.020113272592425346,
+          0.011757895350456238,
+          0.009528422728180885,
+          0.00034042331390082836,
+          0.010248714126646519,
+          0.018755009397864342,
+          0.015352491289377213,
+          -0.031390972435474396,
+          -0.0072372108697891235,
+          -0.007504747714847326,
+          -0.002130003646016121,
+          0.01676563359797001,
+          0.002109423978254199,
+          0.014076546765863895,
+          0.0424765981733799,
+          0.006129334680736065,
+          0.02335115149617195,
+          0.0017852929886430502,
+          -0.019317522644996643,
+          -0.00250558415427804,
+          0.0039821807295084,
+          -0.02147153578698635,
+          0.0016832518158480525,
+          -0.002937758807092905,
+          -0.01598360389471054,
+          -0.0028897393494844437,
+          -0.0028039903845638037,
+          0.0029960679821670055,
+          -0.00598870636895299,
+          -0.002406115410849452,
+          0.0039238715544342995,
+          0.0035431464202702045,
+          0.022088928148150444,
+          -0.019098004326224327,
+          0.0013333960669115186,
+          -0.006125904619693756,
+          0.006136194337159395,
+          -0.015517128631472588,
+          -0.010015476495027542,
+          -0.013857029378414154,
+          0.02944275736808777,
+          -0.021718492731451988,
+          -0.009549002163112164,
+          0.013006399385631084,
+          -0.012423306703567505,
+          -0.006582088768482208,
+          0.0013899903278797865,
+          -0.002208892721682787,
+          -0.026479274034500122,
+          -0.0006332559278234839,
+          -0.005144936498254538,
+          -0.015325050801038742,
+          -0.01343857403844595,
+          -0.04494616761803627,
+          0.021691052243113518,
+          0.004105659201741219,
+          -0.007854603230953217,
+          -0.0023495210334658623,
+          -0.020058393478393555,
+          0.014501861296594143,
+          0.0007142886752262712,
+          0.01920776255428791,
+          -0.011490358039736748,
+          -0.014350943267345428,
+          -0.004047350026667118,
+          0.009946877136826515,
+          -0.020291630178689957,
+          0.02034650929272175,
+          0.002613627817481756,
+          -0.024860333651304245,
+          -0.012608524411916733,
+          0.0040953694842755795,
+          -0.0020065251737833023,
+          -0.008389676921069622,
+          0.0038758523296564817,
+          -0.0040953694842755795,
+          0.00276283104903996,
+          -0.0058446479961276054,
+          0.014268624596297741,
+          -0.02293955720961094,
+          0.021608734503388405,
+          0.0013436859007924795,
+          0.0006739866803400218,
+          0.014529300853610039,
+          0.014872296713292599,
+          0.016203120350837708,
+          -0.014680218882858753,
+          -0.007593926507979631,
+          0.02009955234825611,
+          0.035808760672807693,
+          -0.009686200879514217,
+          0.007223491091281176,
+          0.011785334907472134,
+          0.02591676078736782,
+          0.011504078283905983,
+          -0.002399255521595478,
+          -0.015791526064276695,
+          0.014803697355091572,
+          0.004664742387831211,
+          0.004551554098725319,
+          -0.008540594950318336,
+          -0.031006818637251854,
+          0.0025930481497198343,
+          -0.008238758891820908,
+          -0.023419750854372978,
+          -0.0031281213741749525,
+          0.011023884639143944,
+          0.021787092089653015,
+          0.008780691772699356,
+          -0.004867109935730696,
+          0.01440582238137722,
+          -0.01849433220922947,
+          0.00805354118347168,
+          0.010392772033810616,
+          -0.011010164394974709,
+          0.014042247086763382,
+          -0.02675366960465908,
+          -0.042696114629507065,
+          0.0023718157317489386,
+          0.006684987805783749,
+          0.01257422473281622,
+          0.005717739462852478,
+          0.0070794327184557915,
+          -0.006441460456699133,
+          -0.008375956676900387,
+          -0.006136194337159395,
+          0.0023718157317489386,
+          -0.009082527831196785,
+          0.018384573981165886,
+          0.005436482839286327,
+          -0.008074120618402958,
+          0.0014217174611985683,
+          0.017945539206266403,
+          -0.007648805622011423,
+          0.009247166104614735,
+          0.008856151252985,
+          0.015393650159239769,
+          0.017726020887494087,
+          -0.008595474064350128,
+          0.038580164313316345,
+          0.010934704914689064,
+          -0.014543021097779274,
+          0.00045275440788827837,
+          -0.0044486550614237785,
+          0.030622662976384163,
+          -0.0003732222830876708,
+          -0.01583268493413925,
+          0.023927384987473488,
+          0.002188313053920865,
+          -0.00013923484948463738,
+          -0.014721378684043884,
+          0.0032498850487172604,
+          -0.006475760135799646,
+          0.01643635705113411,
+          -0.009192286990582943,
+          0.001149893389083445,
+          -0.020291630178689957,
+          -0.005611410830169916,
+          0.0004626155423466116,
+          -0.008897310122847557,
+          0.008677792735397816,
+          -0.0017750031547620893,
+          0.0038998620584607124,
+          -0.013459154404699802,
+          -0.018892208114266396,
+          0.0027079517021775246,
+          0.008938469924032688,
+          0.02012699283659458,
+          0.015105534344911575,
+          0.0004733341629616916,
+          -0.011881373822689056,
+          -0.00945296324789524,
+          -0.0035431464202702045,
+          0.005179236177355051,
+          0.0023049316368997097,
+          -0.01941356062889099,
+          -0.012004852294921875,
+          0.010132094845175743,
+          -0.0005805203691124916,
+          -0.003659765003249049,
+          0.00720291119068861,
+          -0.007209771312773228,
+          -0.00598870636895299,
+          0.009926297701895237,
+          -0.0018161626067012548,
+          0.0035105617716908455,
+          -0.006969674024730921,
+          -0.014268624596297741,
+          -0.00933634489774704,
+          0.003944451455026865,
+          -0.01664215512573719,
+          0.02797473594546318,
+          -0.00991943757981062,
+          0.0017518509412184358,
+          0.008876730687916279,
+          0.013171037659049034,
+          0.0017295562429353595,
+          0.02263772115111351,
+          -0.02446245774626732,
+          -0.006966243963688612,
+          0.0029926381539553404,
+          -0.022047769278287888,
+          -0.01780834048986435,
+          -0.017918098717927933,
+          -0.04151620715856552,
+          0.012656544335186481,
+          0.026561593636870384,
+          -0.01646379753947258,
+          0.02979947254061699,
+          -0.0002896170481108129,
+          0.005837787874042988,
+          0.011222821660339832,
+          -0.008314217440783978,
+          -0.003522566519677639,
+          -0.017753461375832558,
+          -0.014014807529747486,
+          0.02004467323422432,
+          -0.005186096299439669,
+          0.005148366559296846,
+          -0.01124340109527111,
+          -0.009747940115630627,
+          -0.0050660474225878716,
+          -0.0002876876969821751,
+          0.01628543995320797,
+          0.015421089716255665,
+          -0.005734889302402735,
+          -0.012313548475503922,
+          -0.020154431462287903,
+          0.025189610198140144,
+          -0.0020854142494499683,
+          -0.009837118908762932,
+          0.009432383812963963,
+          0.02317279390990734,
+          -0.013925628736615181,
+          -0.00645861029624939,
+          0.003143556183204055,
+          0.005676580127328634,
+          -0.0029531936161220074,
+          0.018933366984128952,
+          -0.011483498848974705,
+          0.0018813317874446511,
+          -0.010420211590826511,
+          0.016710754483938217,
+          0.02042882889509201,
+          0.021046221256256104,
+          -0.019756557419896126,
+          0.014501861296594143,
+          -0.01157267764210701,
+          -0.0038244028110057116,
+          0.014076546765863895,
+          0.014789978042244911,
+          0.002035679994150996,
+          0.007346969563513994,
+          0.025779562070965767,
+          -0.007024553604424,
+          0.02090902253985405,
+          -0.011113063432276249,
+          -0.015338771045207977,
+          -0.014433261938393116,
+          -0.02573840320110321,
+          0.003189860610291362,
+          -0.006451750639826059,
+          -0.018151337280869484,
+          -0.007298950105905533,
+          -0.006969674024730921,
+          0.004476095084100962,
+          -0.004801940638571978,
+          -0.02352951094508171,
+          0.01223122887313366,
+          -0.0202504713088274,
+          0.02276119962334633,
+          -0.023282554000616074,
+          -0.0097959591075778,
+          -0.018069017678499222,
+          -0.011133642867207527,
+          -0.008327937684953213,
+          -0.008952190168201923,
+          0.0079643614590168,
+          -0.006678127683699131,
+          0.005656000226736069,
+          -0.005247835535556078,
+          0.0027045216411352158,
+          -0.005467352457344532,
+          0.009363784454762936,
+          0.019701676443219185,
+          0.02162245288491249,
+          0.00472648162394762,
+          0.016065921634435654,
+          0.015489689074456692,
+          0.006623248569667339,
+          -0.016546115279197693,
+          0.006753586698323488,
+          -0.0075802067294716835,
+          0.008115280419588089,
+          0.020085832104086876,
+          -0.010735767893493176,
+          -0.00633856188505888,
+          -0.009686200879514217,
+          -0.0017870080191642046,
+          0.02398226410150528,
+          -0.004692182410508394,
+          -0.002123143756762147,
+          -0.001323963631875813,
+          0.016450077295303345,
+          0.0029549086466431618,
+          0.020058393478393555,
+          0.0019893755670636892,
+          0.01509181410074234,
+          -0.007868323475122452,
+          -0.019221482798457146,
+          -0.014872296713292599,
+          -0.02070322446525097,
+          0.010084075853228569,
+          -0.00909624807536602,
+          -0.01804157719016075,
+          -0.014995775185525417,
+          0.001756138401106,
+          -0.005930396728217602,
+          7.590121185785392e-06,
+          0.005515371914952993,
+          0.007477307692170143,
+          0.00945296324789524,
+          0.014172585681080818,
+          0.001274229260161519,
+          -0.0008952189818955958,
+          0.023556949570775032,
+          -0.015338771045207977,
+          0.00530271464958787,
+          -0.010310453362762928,
+          -0.01129142101854086,
+          -0.008266198448836803,
+          0.0006778453825972974,
+          0.006849625613540411,
+          -0.028537249192595482,
+          0.0061739240773022175,
+          -0.02807077392935753,
+          0.004194838460534811,
+          0.0071343122981488705,
+          -0.001848747255280614,
+          -0.007792863994836807,
+          0.0005758041515946388,
+          -0.009267745539546013,
+          0.011517798528075218,
+          0.021252017468214035,
+          0.021814530715346336,
+          -0.002745681209489703,
+          -0.005234115291386843,
+          0.013788430020213127,
+          0.020140711218118668,
+          0.016559835523366928,
+          0.003587735816836357,
+          0.005642280448228121,
+          -0.025491446256637573,
+          -0.0003918726579286158,
+          -0.012361567467451096,
+          -0.018123896792531013,
+          0.01918032392859459,
+          8.628487557871267e-05,
+          0.00598870636895299,
+          -0.008218178525567055,
+          -0.006726147141307592,
+          -0.004853390157222748,
+          0.01424118410795927,
+          0.010872965678572655,
+          -0.02842748910188675,
+          -0.004805370699614286,
+          -0.022171247750520706,
+          -0.004393775947391987,
+          0.023008156567811966,
+          0.0032207302283495665,
+          -0.003690634621307254,
+          -0.0011661855969578028,
+          -0.009864558465778828,
+          0.004386915825307369,
+          0.006071025040000677,
+          0.0006191073916852474,
+          0.04810172691941261,
+          0.008465135470032692,
+          -0.006894215010106564,
+          -0.00863663386553526,
+          0.0007318672141991556,
+          -0.008533734828233719,
+          0.004736771807074547,
+          0.01002233661711216,
+          0.0018967665964737535,
+          -0.013191617093980312,
+          -0.016381477937102318,
+          -0.009706780314445496,
+          -0.009555862285196781,
+          -0.004215417895466089,
+          -0.023515790700912476,
+          -0.00026753669953905046,
+          -0.009830258786678314,
+          0.002063119551166892,
+          -0.005117496941238642,
+          -0.011888233944773674,
+          0.01944100111722946,
+          -0.02619115822017193,
+          0.007271510548889637,
+          0.01768486201763153,
+          0.006726147141307592,
+          0.012560505419969559,
+          -0.005978416185826063,
+          -0.013520893640816212,
+          0.003927301615476608,
+          -0.014021667651832104,
+          -0.010557410307228565,
+          0.002400970319285989,
+          0.0027816956862807274,
+          -0.01966051757335663,
+          -0.01780834048986435,
+          0.002287781797349453,
+          0.011984271928668022,
+          0.00244898977689445,
+          0.005395323503762484,
+          0.012992680072784424,
+          -0.0004488957056310028,
+          0.02037394978106022,
+          0.0005959551781415939,
+          -0.02947019785642624,
+          0.006016145925968885,
+          0.010845526121556759,
+          0.005539381876587868,
+          -0.0004690467321779579,
+          0.010914125479757786,
+          0.008259338326752186,
+          0.0028674446512013674,
+          -0.02398226410150528,
+          0.0020974192302674055,
+          -0.015338771045207977,
+          0.010090935975313187,
+          -0.026355795562267303,
+          0.014995775185525417,
+          -0.02604023925960064,
+          0.01130514033138752,
+          -0.02850980870425701,
+          0.01225866936147213,
+          0.002647927263751626,
+          0.001377985579892993,
+          0.0071823312900960445,
+          -0.011545238085091114,
+          -0.00035114193451590836,
+          -0.015530848875641823,
+          0.0009037938434630632,
+          -0.0009681055671535432,
+          -0.0261225588619709,
+          -0.029223239049315453,
+          -0.0013376835267990828,
+          -0.007264650426805019,
+          0.004393775947391987,
+          0.009864558465778828,
+          -0.012018571607768536,
+          -0.02174593135714531,
+          0.007038273382931948,
+          0.002641067374497652,
+          -0.0014877441572025418,
+          0.026328355073928833,
+          -0.025875601917505264,
+          -0.02096390165388584,
+          0.003472832264378667,
+          0.01938612200319767,
+          0.005100347101688385,
+          0.006105324719101191,
+          -0.0193586815148592,
+          0.01831597462296486,
+          -0.0070657129399478436,
+          -0.02192429080605507,
+          -0.008595474064350128,
+          0.02096390165388584,
+          0.014378382824361324,
+          0.012382147833704948,
+          0.010475090704858303,
+          -0.0009543857304379344,
+          0.002027105074375868,
+          -0.0014980339910835028,
+          0.013109298422932625,
+          0.008924749679863453,
+          -0.004887689836323261,
+          -0.015174132771790028,
+          -0.0014011377934366465,
+          -0.008794412016868591,
+          0.0028640148229897022,
+          0.013630651868879795,
+          0.011346300132572651,
+          -0.017163507640361786,
+          -0.01583268493413925,
+          -0.03158305212855339,
+          0.002167733386158943,
+          -0.01983887515962124,
+          -0.008986488915979862,
+          -0.00034042331390082836,
+          0.006746727041900158,
+          0.008664073422551155,
+          0.008005521260201931,
+          0.015297611244022846,
+          0.01651867665350437,
+          0.019729116931557655,
+          0.027659179642796516,
+          -0.00056208431487903,
+          -0.004603003151714802,
+          0.004050780087709427,
+          0.0025107290130108595,
+          -0.028537249192595482,
+          -0.00927460566163063,
+          -0.012368427589535713,
+          -0.0006302547408267856,
+          0.016093362122774124,
+          0.001071004313416779,
+          0.007895763032138348,
+          -0.015119253657758236,
+          0.013829589821398258,
+          -0.010914125479757786,
+          -0.010584849864244461,
+          -0.02848236821591854,
+          0.0002185526245739311,
+          0.021608734503388405,
+          -0.025450287386775017,
+          0.013280796818435192,
+          -0.018027858808636665,
+          0.02004467323422432,
+          -0.008060400374233723,
+          -0.004736771807074547,
+          0.0014577320544049144,
+          0.005656000226736069,
+          -0.00611561443656683,
+          0.01020069420337677,
+          0.01136687956750393,
+          0.012704563327133656,
+          -0.013095579110085964,
+          0.009507843293249607,
+          -0.0021643033251166344,
+          0.02192429080605507,
+          0.024558497592806816,
+          0.015283891931176186,
+          -0.012340988032519817,
+          -0.017588824033737183,
+          0.00875325221568346,
+          0.008732672780752182,
+          -0.005803488660603762,
+          0.012649684213101864,
+          -0.01631287857890129,
+          0.01003605592995882,
+          0.01457046065479517,
+          0.005076337605714798,
+          0.020840423181653023,
+          -0.005381603725254536,
+          -0.011071903631091118,
+          0.01562688685953617,
+          0.012059731408953667,
+          -0.004263437353074551,
+          -0.011901953257620335,
+          0.009226586669683456,
+          0.011462918482720852,
+          0.008677792735397816,
+          0.016175679862499237,
+          0.009075668640434742,
+          0.0077517046593129635,
+          0.012251809239387512,
+          -0.028345171362161636,
+          0.004887689836323261,
+          -0.0031641360837966204,
+          -0.017396746203303337,
+          0.008396536111831665,
+          -0.01003605592995882,
+          -0.006496340036392212,
+          0.024407578632235527,
+          0.017314426600933075,
+          0.0020391098223626614,
+          -0.010132094845175743,
+          -0.0007327247294597328,
+          -0.00010531042062211782,
+          0.0023838207125663757,
+          0.0028897393494844437,
+          0.011860793456435204,
+          -0.01651867665350437,
+          -0.019495880231261253,
+          0.029031163081526756,
+          -0.007511607371270657,
+          0.006132764276117086,
+          0.001967080868780613
+        ],
+        "keyphrases": [
+          "hyperparameter search",
+          "8B dense model",
+          "constant learning rate",
+          "MoE 8B/64E models",
+          "square root learning rate decay"
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "23ceada7-a437-4b2d-b989-f16e928f439a",
+      "properties": {
+        "page_content": "1 Introduction 3 ",
+        "embedding": [
+          0.009373236447572708,
+          -0.00019147605053149164,
+          -0.011275587603449821,
+          0.02236648090183735,
+          -0.016502436250448227,
+          0.024490466341376305,
+          -0.009955023415386677,
+          0.09330756217241287,
+          0.00554082915186882,
+          0.0107676787301898,
+          0.006459683645516634,
+          -0.0031074818689376116,
+          0.005069858860224485,
+          -0.029274048283696175,
+          -0.0028881572652608156,
+          0.012771611101925373,
+          0.004903634078800678,
+          0.0010071611031889915,
+          -0.00228790077380836,
+          -0.00550850760191679,
+          0.008833005093038082,
+          0.009594869799911976,
+          -0.036237023770809174,
+          0.07295425236225128,
+          0.023493116721510887,
+          -0.021886276081204414,
+          0.03270936384797096,
+          0.024564342573285103,
+          -0.023142196238040924,
+          -0.0063627189956605434,
+          0.006759811658412218,
+          0.008625224232673645,
+          0.0010533347958698869,
+          -0.021516887471079826,
+          -0.06157708168029785,
+          0.0061087640933692455,
+          0.01108165830373764,
+          0.020574945956468582,
+          -0.03252466768026352,
+          -0.003042838769033551,
+          -0.011358699761331081,
+          -0.008699102327227592,
+          -0.01935596391558647,
+          0.006593586876988411,
+          0.04015254229307175,
+          -0.0013990594306960702,
+          -0.043292347341775894,
+          -0.018090806901454926,
+          -0.023991791531443596,
+          -0.014581616036593914,
+          0.029440272599458694,
+          -0.0307700727134943,
+          0.045434799045324326,
+          0.010758443735539913,
+          0.027667207643389702,
+          -0.0015398887917399406,
+          0.01821085810661316,
+          0.09737084060907364,
+          0.003970927558839321,
+          -0.02190474607050419,
+          0.005037537310272455,
+          -0.05751380696892738,
+          -0.011543394066393375,
+          -0.022107908502221107,
+          -0.010001196525990963,
+          -0.0067828986793756485,
+          -0.016742538660764694,
+          0.04137152433395386,
+          -0.02829516865313053,
+          -0.02114749886095524,
+          0.02690996043384075,
+          -0.002694228198379278,
+          0.0043634031899273396,
+          0.03933988884091377,
+          0.002493373118340969,
+          -0.005203762091696262,
+          0.069408118724823,
+          0.05537135526537895,
+          -0.001163574168458581,
+          0.031915176659822464,
+          -0.00717537384480238,
+          -0.0074200937524437904,
+          0.0161607526242733,
+          0.005069858860224485,
+          0.022772807627916336,
+          -0.01561590377241373,
+          -0.07964018732309341,
+          -0.10867413133382797,
+          0.0017811458092182875,
+          -0.0003122488269582391,
+          0.006099529564380646,
+          -0.008445147424936295,
+          -0.008514408022165298,
+          -0.04657990485429764,
+          -0.01678871363401413,
+          -0.003474561730399728,
+          0.0430707149207592,
+          0.00898999534547329,
+          -0.008334331214427948,
+          -0.01108165830373764,
+          0.03139803186058998,
+          -0.027556391432881355,
+          -0.01748131588101387,
+          -0.007743308786302805,
+          -0.020870456472039223,
+          -0.015976058319211006,
+          0.043255407363176346,
+          0.020741170272231102,
+          -0.0013759726425632834,
+          0.022348010912537575,
+          -0.03315262869000435,
+          0.00933167990297079,
+          -0.013057887554168701,
+          -0.01091543398797512,
+          0.01863565668463707,
+          -0.006025651935487986,
+          -0.03577528893947601,
+          -0.014886361546814442,
+          0.015458913519978523,
+          0.02661444991827011,
+          -0.013741256669163704,
+          -0.010287472978234291,
+          -0.009424027055501938,
+          -0.014535441994667053,
+          -0.027593329548835754,
+          0.0079002995043993,
+          -0.004626592621207237,
+          0.003342967014759779,
+          -0.007669431157410145,
+          0.04181479290127754,
+          0.02417648583650589,
+          0.04436357319355011,
+          -0.02628200128674507,
+          0.003675416810438037,
+          0.017398204654455185,
+          0.019503720104694366,
+          0.021627703681588173,
+          -0.006436596624553204,
+          0.00369388610124588,
+          0.02476750686764717,
+          -0.07927079498767853,
+          -0.010028901509940624,
+          -0.017527490854263306,
+          0.025709448382258415,
+          -0.009433262050151825,
+          0.018728002905845642,
+          -0.04750337824225426,
+          0.019485250115394592,
+          0.007235399447381496,
+          0.010730739682912827,
+          -0.01968841440975666,
+          0.006709021050482988,
+          -0.02216331660747528,
+          -0.012660794891417027,
+          0.004765113350003958,
+          0.011423342861235142,
+          0.03219221904873848,
+          0.017065754160284996,
+          0.03143497183918953,
+          0.03298640251159668,
+          -0.004991363734006882,
+          0.034020692110061646,
+          0.026429755613207817,
+          -0.02631893940269947,
+          0.026134245097637177,
+          -0.019208207726478577,
+          0.012134416028857231,
+          -0.04089131951332092,
+          -0.03645865619182587,
+          0.025044549256563187,
+          0.01748131588101387,
+          -0.006067208014428616,
+          0.040780503302812576,
+          0.0038855066522955894,
+          -0.013131765648722649,
+          -0.00817734096199274,
+          0.017499785870313644,
+          0.01250380463898182,
+          -0.014997177757322788,
+          0.005356134846806526,
+          0.006769046653062105,
+          -0.02308678813278675,
+          0.0019115861505270004,
+          0.0045896535739302635,
+          -0.027002308517694473,
+          -0.010416759178042412,
+          0.020925864577293396,
+          -0.030751602724194527,
+          0.006810602732002735,
+          -0.0014486960135400295,
+          0.05489115044474602,
+          0.0032944846898317337,
+          0.028442922979593277,
+          0.034149978309869766,
+          -0.01456314604729414,
+          -0.04414194077253342,
+          -0.003213681047782302,
+          0.004421120043843985,
+          0.0161607526242733,
+          -0.010970842093229294,
+          -0.0005338820046745241,
+          -0.01942984201014042,
+          0.01731509156525135,
+          0.03333732485771179,
+          0.0022463444620370865,
+          -0.019023513421416283,
+          0.010860025882720947,
+          0.038083966821432114,
+          0.02090739645063877,
+          -0.002989739179611206,
+          0.027500983327627182,
+          0.008200427517294884,
+          -0.03113946132361889,
+          -0.009890380315482616,
+          0.01893116720020771,
+          0.009308593347668648,
+          -0.0269469004124403,
+          0.05725523456931114,
+          0.0020755024161189795,
+          -0.021129028871655464,
+          0.026743736118078232,
+          0.012180590070784092,
+          0.049461137503385544,
+          0.021812397986650467,
+          -0.022311072796583176,
+          0.026577511802315712,
+          0.008893030695617199,
+          0.032469261437654495,
+          0.005651645828038454,
+          -0.00040199869545176625,
+          0.002104360843077302,
+          0.03051150031387806,
+          -0.0017707566730678082,
+          0.007401624694466591,
+          -0.03723437339067459,
+          0.01070303563028574,
+          0.0019427533261477947,
+          0.00814501941204071,
+          0.021165968850255013,
+          0.022384950891137123,
+          -0.006778281182050705,
+          0.008223514072597027,
+          -0.009770329110324383,
+          -0.021221376955509186,
+          -0.018986575305461884,
+          0.005268405191600323,
+          -0.0123375803232193,
+          0.08414673060178757,
+          0.006907567381858826,
+          -0.05141889676451683,
+          0.06157708168029785,
+          0.010823086835443974,
+          0.014156819321215153,
+          -0.007627875078469515,
+          0.011164771392941475,
+          0.0864369347691536,
+          -0.0008519025286659598,
+          -0.00424566026777029,
+          -0.03786233440041542,
+          0.04454826936125755,
+          -0.020593415945768356,
+          0.037751518189907074,
+          -0.0031813594978302717,
+          -0.012032833881676197,
+          0.019780760630965233,
+          -0.060801368206739426,
+          0.043920308351516724,
+          -0.008976143784821033,
+          -0.0009165455121546984,
+          0.03198905289173126,
+          0.06349790096282959,
+          -0.009040786884725094,
+          0.019337493926286697,
+          0.018247798085212708,
+          0.015569730661809444,
+          -0.021258315071463585,
+          0.020796580240130424,
+          -0.0016149209113791585,
+          -0.0142491664737463,
+          -0.033004872500896454,
+          0.004381872713565826,
+          0.026706798002123833,
+          0.025173833593726158,
+          -0.027427105233073235,
+          0.00783565640449524,
+          -0.024194953963160515,
+          0.013482684269547462,
+          0.020593415945768356,
+          -0.03904437646269798,
+          -0.026411285623908043,
+          0.004485763143748045,
+          -0.003559982869774103,
+          -0.04905480891466141,
+          -0.011958956718444824,
+          0.01857101358473301,
+          0.0034791789948940277,
+          0.013381103053689003,
+          -0.0331895686686039,
+          -0.036828044801950455,
+          -0.006542795803397894,
+          0.004314920864999294,
+          -0.0001486211986048147,
+          -0.006436596624553204,
+          -0.035701408982276917,
+          0.0030012824572622776,
+          0.07155057787895203,
+          -0.01592988334596157,
+          0.0028350576758384705,
+          -0.03740059956908226,
+          0.0222926028072834,
+          0.03392834588885307,
+          -0.03908131644129753,
+          0.017324326559901237,
+          -0.06593587249517441,
+          -0.021572295576334,
+          0.023659341037273407,
+          -0.007928003557026386,
+          -0.011718854308128357,
+          -0.03937682881951332,
+          -0.008929969742894173,
+          -0.004912868607789278,
+          -0.003446857677772641,
+          -0.024619750678539276,
+          -0.016040701419115067,
+          0.009622573852539062,
+          -0.007928003557026386,
+          -0.00038872379809617996,
+          0.015246515162289143,
+          -0.01649320125579834,
+          -0.01435074768960476,
+          -0.0038624198641628027,
+          0.027057716622948647,
+          0.03344814106822014,
+          0.01913432963192463,
+          -0.016881059855222702,
+          -0.055629927664995193,
+          0.0015387344174087048,
+          -0.0015075673582032323,
+          -0.004356476943939924,
+          -0.02711312472820282,
+          0.01589294523000717,
+          0.010813851840794086,
+          -0.014443094842135906,
+          0.026392817497253418,
+          0.0029112440533936024,
+          -0.016253098845481873,
+          0.016465498134493828,
+          -0.039561521261930466,
+          -0.006492005195468664,
+          0.013011714443564415,
+          0.0008392047602683306,
+          -0.009548695757985115,
+          0.014212227426469326,
+          -0.02328995242714882,
+          0.01676100865006447,
+          -0.03703121095895767,
+          0.021756989881396294,
+          -0.02354852482676506,
+          0.025783326476812363,
+          0.004104831255972385,
+          0.004903634078800678,
+          0.0001385207287967205,
+          0.02236648090183735,
+          0.006662847474217415,
+          -0.0007001068443059921,
+          -0.06257443130016327,
+          0.01767524518072605,
+          -0.026669858023524284,
+          0.02090739645063877,
+          -0.006542795803397894,
+          -0.04247969016432762,
+          0.018792646005749702,
+          0.0180723387748003,
+          0.008717571385204792,
+          -0.04414194077253342,
+          0.0005454253987409174,
+          0.0029320220928639174,
+          -0.0011347156250849366,
+          -0.04229499772191048,
+          0.01797999069094658,
+          0.004199486691504717,
+          -0.023160666227340698,
+          -0.004656605422496796,
+          0.008435912430286407,
+          0.001627618563361466,
+          -0.030991705134510994,
+          0.005356134846806526,
+          0.004040188156068325,
+          -0.0046242838725447655,
+          -0.008985377848148346,
+          -0.026134245097637177,
+          -0.002957417629659176,
+          -0.020131679251790047,
+          -0.037382129579782486,
+          0.042073365300893784,
+          0.0010117784840986133,
+          -0.0060164169408380985,
+          -0.01735202968120575,
+          0.0010550662409514189,
+          0.01141410879790783,
+          -0.009733390063047409,
+          0.0033845233265310526,
+          -0.014904830604791641,
+          0.027057716622948647,
+          0.02354852482676506,
+          -0.00550850760191679,
+          -0.03512885794043541,
+          -0.02774108573794365,
+          0.0052360836416482925,
+          0.0026064985431730747,
+          -0.03546130657196045,
+          -0.05289645120501518,
+          0.023326890543103218,
+          -0.021923214197158813,
+          0.020131679251790047,
+          0.010481402277946472,
+          -0.030456092208623886,
+          0.028239760547876358,
+          -0.022772807627916336,
+          -0.02061188593506813,
+          -0.001733817858621478,
+          0.005840957630425692,
+          0.0088560925796628,
+          0.002608807059004903,
+          -0.04137152433395386,
+          0.024749036878347397,
+          -0.03134262561798096,
+          -0.007904916070401669,
+          0.024952201172709465,
+          -0.006348866969347,
+          -0.014258400537073612,
+          -0.007119965739548206,
+          0.003691577585414052,
+          0.030659254640340805,
+          0.0076140230521559715,
+          -0.03139803186058998,
+          -0.010213595815002918,
+          -0.002126293256878853,
+          0.023105258122086525,
+          0.005559298675507307,
+          0.021055150777101517,
+          -0.041482340544462204,
+          0.016437793150544167,
+          0.0362185537815094,
+          -0.027685677632689476,
+          0.0021874732337892056,
+          -0.03919213265180588,
+          -0.026429755613207817,
+          0.026429755613207817,
+          -0.010536810383200645,
+          0.0200208630412817,
+          0.0041694738902151585,
+          -0.024121077731251717,
+          0.005577768199145794,
+          0.026411285623908043,
+          -0.00010345767077524215,
+          -0.000978302676230669,
+          0.05452176183462143,
+          -0.008689867332577705,
+          0.021258315071463585,
+          -0.00016622488328721374,
+          0.014433860778808594,
+          0.011349465698003769,
+          -0.017204275354743004,
+          -0.01873723790049553,
+          0.004361094441264868,
+          -0.02282821759581566,
+          -0.013279520906507969,
+          -0.00630731089040637,
+          0.011931252665817738,
+          -0.016345446929335594,
+          -0.002054724143818021,
+          -0.014849422499537468,
+          0.028018126264214516,
+          -0.026429755613207817,
+          -0.029015475884079933,
+          -0.010878494940698147,
+          -0.010112013667821884,
+          -0.006672082003206015,
+          -0.02203403040766716,
+          -0.015218811109662056,
+          0.04092825949192047,
+          0.012097476981580257,
+          -0.0010181274265050888,
+          0.0024218042381107807,
+          0.014221461489796638,
+          0.044806838035583496,
+          -0.026374347507953644,
+          0.010758443735539913,
+          0.027168532833456993,
+          -0.012411457486450672,
+          0.004467293620109558,
+          0.004324155393987894,
+          0.019282085821032524,
+          -0.01784146949648857,
+          -0.07816263288259506,
+          -0.020002393051981926,
+          -0.0386011116206646,
+          -0.01360273640602827,
+          0.04558255523443222,
+          -0.002902009291574359,
+          0.016668662428855896,
+          -0.006007182411849499,
+          -0.011155536398291588,
+          0.010795382782816887,
+          0.03394681587815285,
+          0.0025418554432690144,
+          -0.0037054296117275953,
+          -0.004377255216240883,
+          0.01626233384013176,
+          -0.004307994619011879,
+          -0.02740863524377346,
+          0.008657545782625675,
+          0.004963659681379795,
+          0.008851475082337856,
+          -0.019577596336603165,
+          -0.005088328383862972,
+          0.005887131206691265,
+          0.017887644469738007,
+          -0.02005780301988125,
+          0.0027080802246928215,
+          0.03241385146975517,
+          -0.033004872500896454,
+          0.015477383509278297,
+          -0.022551175206899643,
+          0.026245061308145523,
+          0.022551175206899643,
+          -0.0063581014983356,
+          0.013528858311474323,
+          -0.018552543595433235,
+          -0.004201795440167189,
+          0.0053976913914084435,
+          0.018063103780150414,
+          0.009816502220928669,
+          0.013418042100965977,
+          -0.019078921526670456,
+          -0.024065667763352394,
+          0.03549824655056,
+          0.011469516903162003,
+          0.00992731936275959,
+          -0.050938691943883896,
+          0.008223514072597027,
+          0.010398290120065212,
+          -0.026559041813015938,
+          -0.025210773572325706,
+          0.03559059277176857,
+          0.010610688477754593,
+          -0.02295750193297863,
+          0.01391671597957611,
+          0.05444788187742233,
+          -0.008625224232673645,
+          0.02325301431119442,
+          0.01254997868090868,
+          0.010906198993325233,
+          -4.3900970922550187e-05,
+          0.022218724712729454,
+          -0.0075124409049749374,
+          0.0037746899761259556,
+          0.0031328771729022264,
+          -0.0238809734582901,
+          0.026928430423140526,
+          -0.008560581132769585,
+          -0.01689029484987259,
+          0.001642625080421567,
+          0.028018126264214516,
+          -0.0199839249253273,
+          0.00649662222713232,
+          0.015135698951780796,
+          -0.014997177757322788,
+          0.009622573852539062,
+          0.02437964826822281,
+          0.010970842093229294,
+          -0.04454826936125755,
+          -0.01976229064166546,
+          0.012429927475750446,
+          0.012799316085875034,
+          0.02661444991827011,
+          0.009110046550631523,
+          0.006196494214236736,
+          0.0003523621126078069,
+          -0.010047370567917824,
+          -0.01873723790049553,
+          0.014951004646718502,
+          -0.015375801362097263,
+          -0.029828131198883057,
+          -0.028646087273955345,
+          -0.005850192159414291,
+          -0.004857460502535105,
+          -0.013141000643372536,
+          -0.009733390063047409,
+          0.0158005990087986,
+          -0.015135698951780796,
+          0.01959606632590294,
+          0.006602821405977011,
+          -0.00042508551268838346,
+          0.033300384879112244,
+          0.03064078651368618,
+          0.02144300937652588,
+          -0.00038497219793498516,
+          -0.04281214252114296,
+          0.004885164555162191,
+          0.0005820756778120995,
+          0.007678666152060032,
+          0.0036130824591964483,
+          0.00820966251194477,
+          0.04809439927339554,
+          0.004931338131427765,
+          0.013390337117016315,
+          0.006921419408172369,
+          0.03732671961188316,
+          -0.029865069314837456,
+          0.0032044462859630585,
+          0.032007522881031036,
+          -0.017961522564291954,
+          -0.03165660426020622,
+          -0.014590850099921227,
+          0.006944505963474512,
+          0.007794099859893322,
+          -0.005222231615334749,
+          -0.00103255663998425,
+          0.02111055888235569,
+          -0.0013344164472073317,
+          0.008265070617198944,
+          0.020297905430197716,
+          0.0007843736675567925,
+          0.01606840454041958,
+          -0.016345446929335594,
+          0.0039432235062122345,
+          -0.009816502220928669,
+          -0.02988353930413723,
+          -0.00913775060325861,
+          0.002717314986512065,
+          0.004215647466480732,
+          0.004573492798954248,
+          -0.0077479262836277485,
+          0.013270285911858082,
+          0.03592304512858391,
+          -0.0024633603170514107,
+          0.010739973746240139,
+          -0.02018708735704422,
+          0.017398204654455185,
+          0.0026388198602944613,
+          0.006602821405977011,
+          0.01985463872551918,
+          -0.02476750686764717,
+          0.007443180773407221,
+          -0.011672680266201496,
+          0.017296621575951576,
+          -0.005600854754447937,
+          0.03952458128333092,
+          0.0018192389979958534,
+          0.014877126552164555,
+          0.016151517629623413,
+          0.029440272599458694,
+          0.01764754205942154,
+          -0.039302948862314224,
+          -0.014803249388933182,
+          0.008232749067246914,
+          -0.0063211629167199135,
+          -0.012513039633631706,
+          0.0017615220276638865,
+          -0.011783497408032417,
+          0.032044462859630585,
+          -0.010204360820353031,
+          0.012743907049298286,
+          -0.014840187504887581,
+          -0.01439692173153162,
+          -0.04824215546250343,
+          -0.0015548951923847198,
+          0.00037256305222399533,
+          0.0019970072899013758,
+          -0.036292433738708496,
+          0.002897392027080059,
+          0.024989139288663864,
+          -0.02870149537920952,
+          0.002140145283192396,
+          -0.04318153113126755,
+          0.01896810531616211,
+          0.005854809656739235,
+          -0.021165968850255013,
+          -0.01606840454041958,
+          -0.012679263949394226,
+          0.0038947411812841892,
+          0.04488071799278259,
+          0.00553159462288022,
+          0.008902265690267086,
+          -0.03291252627968788,
+          -0.00022769345378037542,
+          -0.0012697733473032713,
+          0.003028986742720008,
+          0.010315177030861378,
+          0.014424625784158707,
+          0.03206293284893036,
+          0.00712458323687315,
+          -0.024435056373476982,
+          -0.010130482725799084,
+          0.01385207287967205,
+          0.021646173670887947,
+          0.016945702955126762,
+          0.008006498217582703,
+          -0.007567849475890398,
+          -0.005633176304399967,
+          -0.030234457924962044,
+          0.01441539078950882,
+          0.016871824860572815,
+          0.007138435263186693,
+          0.006750577129423618,
+          -0.004326464142650366,
+          0.01546814851462841,
+          -0.03150884807109833,
+          -0.026411285623908043,
+          0.023659341037273407,
+          -0.01883881911635399,
+          0.010370585136115551,
+          -0.006986062508076429,
+          0.020371781662106514,
+          -0.02328995242714882,
+          -0.02279127761721611,
+          -0.004040188156068325,
+          0.012697733938694,
+          -0.0001104558541555889,
+          -0.007364685647189617,
+          -0.004379563964903355,
+          -0.006233432795852423,
+          -0.0046750749461352825,
+          -3.780461702262983e-05,
+          -0.0146277891471982,
+          0.03812090680003166,
+          0.004192560911178589,
+          -0.00451115844771266,
+          -0.018063103780150414,
+          -0.018820350989699364,
+          0.00834356527775526,
+          0.013168704695999622,
+          0.03126874566078186,
+          0.001201667357236147,
+          -0.010629157535731792,
+          0.015643607825040817,
+          0.0024148779921233654,
+          -0.005171440541744232,
+          0.03185976669192314,
+          -0.0030751603189855814,
+          -0.01968841440975666,
+          0.006764429155737162,
+          -0.025210773572325706,
+          0.007660196628421545,
+          0.02437964826822281,
+          -0.0008998076082207263,
+          -0.020168619230389595,
+          0.02262505330145359,
+          -0.02064882405102253,
+          -0.03014211170375347,
+          0.010416759178042412,
+          -0.034648653119802475,
+          -0.01867259480059147,
+          0.01001043152064085,
+          0.02321607433259487,
+          0.007623257581144571,
+          0.007350833620876074,
+          0.002272894373163581,
+          -0.011201709508895874,
+          0.01826626807451248,
+          -0.03856417164206505,
+          0.009982727468013763,
+          0.025321589782834053,
+          0.0061087640933692455,
+          -0.013113295659422874,
+          -0.014258400537073612,
+          -0.012226763181388378,
+          0.003841641591861844,
+          0.022440358996391296,
+          0.031675074249506,
+          0.027390167117118835,
+          -0.009834972210228443,
+          0.0014302266063168645,
+          -0.002456434303894639,
+          0.009493287652730942,
+          -0.040189482271671295,
+          -0.013870542868971825,
+          0.003945532254874706,
+          -0.04096519947052002,
+          -0.0082789221778512,
+          -0.0030543820466846228,
+          0.007558614481240511,
+          0.022348010912537575,
+          0.00971492100507021,
+          -0.018386319279670715,
+          -0.037788454443216324,
+          -0.0238440353423357,
+          0.0051437364891171455,
+          0.04303377494215965,
+          0.03021598979830742,
+          -0.016908764839172363,
+          0.011238648556172848,
+          0.045434799045324326,
+          -0.005799401551485062,
+          -0.015911415219306946,
+          0.021627703681588173,
+          -0.012346814386546612,
+          0.016086874529719353,
+          0.016530141234397888,
+          0.0024218042381107807,
+          -0.04875929653644562,
+          0.029329456388950348,
+          0.017952287569642067,
+          0.006935271434485912,
+          0.02628200128674507,
+          -0.023068320006132126,
+          -0.03512885794043541,
+          -0.002382556675001979,
+          -0.0009690679726190865,
+          0.014138349331915379,
+          -0.009770329110324383,
+          -0.0013436510926112533,
+          0.003975545056164265,
+          0.052231550216674805,
+          0.010305942967534065,
+          -0.002495681867003441,
+          -0.01265155989676714,
+          0.02338230051100254,
+          0.013805899769067764,
+          -0.010204360820353031,
+          0.015578964725136757,
+          -0.019337493926286697,
+          0.02413954585790634,
+          0.046653781086206436,
+          0.007493971846997738,
+          0.009465583600103855,
+          0.018257033079862595,
+          0.010749208740890026,
+          -0.014932534657418728,
+          0.00727233849465847,
+          -0.013547327369451523,
+          0.006949123460799456,
+          0.010416759178042412,
+          0.007184608839452267,
+          -0.015791364014148712,
+          0.016216160729527473,
+          -0.017952287569642067,
+          0.012162120081484318,
+          0.00358768692240119,
+          -0.014932534657418728,
+          0.009290123358368874,
+          -0.0123375803232193,
+          -0.025026079267263412,
+          -0.0044626761227846146,
+          -0.00965027790516615,
+          0.0034999572671949863,
+          0.024749036878347397,
+          -0.021775459870696068,
+          0.016206925734877586,
+          0.014295339584350586,
+          0.0030520735308527946,
+          0.017823001369833946,
+          -0.04137152433395386,
+          0.009871911257505417,
+          0.010970842093229294,
+          0.05167746916413307,
+          0.03322650492191315,
+          -0.015542026609182358,
+          0.014101410284638405,
+          0.04159316048026085,
+          -0.019078921526670456,
+          -0.016142282634973526,
+          0.012217528186738491,
+          -0.005614707246422768,
+          -0.040189482271671295,
+          -0.04838990792632103,
+          -0.03021598979830742,
+          -0.003042838769033551,
+          -0.015246515162289143,
+          -0.0037262076511979103,
+          -0.01376896072179079,
+          0.0005832299939356744,
+          0.008717571385204792,
+          0.00018700299551710486,
+          0.015117228962481022,
+          -0.009484052658081055,
+          -0.0018411714117974043,
+          -0.023807097226381302,
+          -0.019005045294761658,
+          0.0009771483018994331,
+          -0.0177121851593256,
+          0.009364001452922821,
+          -0.020297905430197716,
+          -0.011174005456268787,
+          0.0038855066522955894,
+          0.024435056373476982,
+          0.03437161073088646,
+          -0.01870029978454113,
+          0.011451046913862228,
+          0.010481402277946472,
+          -0.006958357989788055,
+          -0.04613664001226425,
+          0.01185737457126379,
+          0.008583667688071728,
+          -0.0037192816380411386,
+          0.008943822234869003,
+          -0.009622573852539062,
+          0.001551432185806334,
+          -0.006427362095564604,
+          0.029643436893820763,
+          0.018487900495529175,
+          -0.02391791343688965,
+          -0.004525010474026203,
+          0.015578964725136757,
+          0.01857101358473301,
+          0.005134501960128546,
+          -0.011663445271551609,
+          -0.010555279441177845,
+          -0.01708422414958477,
+          -0.027353227138519287,
+          -0.04074356332421303,
+          -0.049128685146570206,
+          0.006593586876988411,
+          -0.0061087640933692455,
+          0.0055546811781823635,
+          0.029015475884079933,
+          -0.0002024422719841823,
+          0.041777852922677994,
+          0.0063858055509626865,
+          -0.0017072680639103055,
+          -0.013150234706699848,
+          0.009705686010420322,
+          0.021572295576334,
+          -0.05267481878399849,
+          -0.02354852482676506,
+          0.0022070968989282846,
+          0.017536723986268044,
+          0.004187943413853645,
+          -0.0009448268101550639,
+          0.020556475967168808,
+          -0.00783565640449524,
+          -0.00719846086576581,
+          0.04779888689517975,
+          -0.010509106330573559,
+          -0.007526292931288481,
+          0.011321761645376682,
+          -0.010656861588358879,
+          0.020371781662106514,
+          0.023954851552844048,
+          0.03331885486841202,
+          0.016373150050640106,
+          -0.008833005093038082,
+          0.022107908502221107,
+          -0.0215353574603796,
+          0.004848225507885218,
+          0.025450875982642174,
+          0.001217828132212162,
+          0.0021066695917397738,
+          -0.025395467877388,
+          0.025284651666879654,
+          -0.004425737541168928,
+          0.014092176221311092,
+          -0.0017188114579766989,
+          0.013371868059039116,
+          -0.014738606289029121,
+          0.025450875982642174,
+          0.0010175502393394709,
+          0.03136109560728073,
+          0.017065754160284996,
+          -0.029902009293437004,
+          -0.015218811109662056,
+          -0.012032833881676197,
+          -0.04151928052306175,
+          0.007544762454926968,
+          0.006713638082146645,
+          0.021516887471079826,
+          -0.02245882898569107,
+          -0.004580419044941664,
+          -0.01896810531616211,
+          -0.015643607825040817,
+          0.0051437364891171455,
+          -0.028683027252554893,
+          0.000924048712477088,
+          -0.010112013667821884,
+          -0.0034076101146638393,
+          0.0009985036449506879,
+          -0.02962496690452099,
+          0.020390251651406288,
+          0.0013782812748104334,
+          -0.016336211934685707,
+          0.02681761421263218,
+          0.019614536315202713,
+          -0.00643197912722826,
+          -0.016317741945385933,
+          0.003005899954587221,
+          -0.00890688318759203,
+          -0.001478708814829588,
+          0.04802051931619644,
+          0.010084309615194798,
+          0.012300641275942326,
+          0.004732791800051928,
+          -0.014064472168684006,
+          -0.00013296547695063055,
+          0.0014475416392087936,
+          0.010102778673171997,
+          -0.010333647020161152,
+          -0.008057288825511932,
+          0.005947156809270382,
+          -0.0045896535739302635,
+          -0.012697733938694,
+          0.01454467698931694,
+          0.016604019328951836,
+          -0.0036061564460396767,
+          -0.0193005558103323,
+          0.023345360532402992,
+          0.010305942967534065,
+          0.0048666950315237045,
+          0.0042133391834795475,
+          -0.0018053868552669883,
+          -0.006926036439836025,
+          0.01750902086496353,
+          -0.008555963635444641,
+          0.013547327369451523,
+          0.019023513421416283,
+          -0.03664335235953331,
+          0.018127746880054474,
+          0.030696194618940353,
+          0.04310765117406845,
+          -0.0011774261947721243,
+          -0.01449850294739008,
+          -0.018090806901454926,
+          -0.01555126067250967,
+          0.004229499492794275,
+          0.015643607825040817,
+          0.02015014924108982,
+          0.005379221867769957,
+          0.0068244547583162785,
+          -0.011460281908512115,
+          -0.018386319279670715,
+          0.009502522647380829,
+          -0.0032991021871566772,
+          -0.0200208630412817,
+          0.032303035259246826,
+          0.0003789119073189795,
+          -0.008966908790171146,
+          -0.03051150031387806,
+          0.006141085643321276,
+          0.003566908882930875,
+          0.017370499670505524,
+          0.007558614481240511,
+          -0.001290551503188908,
+          -0.008158870972692966,
+          0.0035415133461356163,
+          0.008943822234869003,
+          -0.008999230340123177,
+          0.04181479290127754,
+          -0.0009627190884202719,
+          0.013464215211570263,
+          0.006699786055833101,
+          -0.021036682650446892,
+          0.0002056167140835896,
+          -0.005900983233004808,
+          -0.01689029484987259,
+          0.001274390728212893,
+          -0.02077811025083065,
+          -0.0027311670128256083,
+          0.025118425488471985,
+          -0.00929935835301876,
+          -0.029661906883120537,
+          0.030234457924962044,
+          -0.012513039633631706,
+          -0.02123984508216381,
+          0.0044395895674824715,
+          0.006879862863570452,
+          0.013944420032203197,
+          0.00624266779050231,
+          -0.004894399084150791,
+          0.01590218022465706,
+          0.008315861225128174,
+          0.001127789611928165,
+          0.02328995242714882,
+          -0.019633004441857338,
+          -0.00524531863629818,
+          -0.0022244120482355356,
+          -0.020888926461338997,
+          -0.01635468192398548,
+          0.006335014943033457,
+          -0.023696279153227806,
+          -0.018792646005749702,
+          -0.001254766946658492,
+          -0.0092347152531147,
+          0.016049936413764954,
+          0.03051150031387806,
+          0.0035368960816413164,
+          -0.01254997868090868,
+          0.007974176667630672,
+          -0.00046606454998254776,
+          -0.0038831979036331177,
+          -0.05042154714465141,
+          -0.021258315071463585,
+          0.011829670518636703,
+          0.007909533567726612,
+          -0.01681641675531864,
+          -0.001661094487644732,
+          0.0036061564460396767,
+          0.06763505935668945,
+          -0.02232954278588295,
+          -0.016576314345002174,
+          -0.030603846535086632,
+          0.002384865190833807,
+          0.003539204830303788,
+          0.003673108061775565,
+          0.003989397082477808,
+          -0.014830953441560268,
+          0.030252927914261818,
+          -0.0012963231420144439,
+          0.04613664001226425,
+          -0.009604103863239288,
+          0.008860709145665169,
+          0.028110474348068237,
+          0.0199839249253273,
+          0.015163403004407883,
+          -0.0238440353423357,
+          -0.000534170598257333,
+          -0.00905463844537735,
+          0.0022417271975427866,
+          -0.01439692173153162,
+          0.008200427517294884,
+          0.02774108573794365,
+          -0.016447028145194054,
+          0.00649662222713232,
+          0.017102692276239395,
+          -0.020427191630005836,
+          -0.022569645196199417,
+          -0.0011924325954169035,
+          -0.0008091919589787722,
+          0.02668832801282406,
+          0.01299324445426464,
+          -0.005600854754447937,
+          0.010619922541081905,
+          0.022089440375566483,
+          -0.007678666152060032,
+          0.01728738658130169,
+          0.014119880273938179,
+          -0.014489268884062767,
+          0.011811201460659504,
+          0.007295425049960613,
+          0.0029158613178879023,
+          -0.03147191181778908,
+          -0.022237194702029228,
+          -0.0018965797498822212,
+          0.007946472615003586,
+          -0.001829628017731011,
+          -0.02236648090183735,
+          -0.008389739319682121,
+          0.02777802385389805,
+          -0.00840820837765932,
+          0.011663445271551609,
+          0.0002845446579158306,
+          -0.009936554357409477,
+          0.029902009293437004,
+          0.03457477316260338,
+          -0.002913552802056074,
+          0.007041470613330603,
+          -0.029809661209583282,
+          0.019743822515010834,
+          -0.018100041896104813,
+          -0.0038139375392347574,
+          -0.03571987897157669,
+          0.03856417164206505,
+          0.016280803829431534,
+          -0.0028535271994769573,
+          -0.004751261323690414,
+          0.028516801074147224,
+          0.02299444191157818,
+          -0.0386011116206646,
+          0.03274630010128021,
+          0.008135784417390823,
+          0.0031651987228542566,
+          -0.0038624198641628027,
+          -0.020722702145576477,
+          0.009114664047956467,
+          -0.033725179731845856,
+          0.0020974348299205303,
+          -0.01533886231482029,
+          0.005619324278086424,
+          -0.0246751606464386,
+          -0.0146277891471982,
+          0.0036823428235948086,
+          -0.006699786055833101,
+          0.005259170662611723,
+          -0.0036615647841244936,
+          0.003359127789735794,
+          -0.002754253800958395,
+          -0.02203403040766716,
+          -0.033060282468795776,
+          0.026115775108337402,
+          -0.01250380463898182,
+          0.004513467196375132,
+          -0.009484052658081055,
+          -0.03106558322906494,
+          0.02005780301988125,
+          -0.013205642811954021,
+          -0.03248772770166397,
+          -0.009303975850343704,
+          0.026041897013783455,
+          0.000664899533148855,
+          -0.008980761282145977,
+          0.00920239370316267,
+          0.04377255216240883,
+          0.03634783998131752,
+          -0.017952287569642067,
+          0.014239931479096413,
+          0.019928516820073128,
+          0.0010221676202490926,
+          0.013242581859230995,
+          -0.019614536315202713,
+          -0.013362633064389229,
+          0.004407268017530441,
+          -0.019448310136795044,
+          -0.014757075347006321,
+          -0.040817443281412125,
+          -0.00986267626285553,
+          0.02602342888712883,
+          -0.01544967945665121,
+          -0.015357332304120064,
+          0.03919213265180588,
+          0.007036853116005659,
+          -0.04613664001226425,
+          -0.00831124372780323,
+          0.027851901948451996,
+          0.00270115421153605,
+          -0.012429927475750446,
+          0.004880547057837248,
+          0.013353399001061916,
+          -0.027131594717502594,
+          -0.001456776401028037,
+          0.023622402921319008,
+          0.03054843842983246,
+          -0.03440855070948601,
+          -0.01519110705703497,
+          0.014812483452260494,
+          -0.0010071611031889915,
+          -0.00793262105435133,
+          -0.014747840352356434,
+          -0.0068244547583162785,
+          0.022274134680628777,
+          0.006515091750770807,
+          -0.01286395825445652,
+          0.006196494214236736,
+          -0.011755792424082756,
+          -0.03472252935171127,
+          0.02266199141740799,
+          -0.01286395825445652,
+          -0.00313749467022717,
+          0.01735202968120575,
+          0.003996323328465223,
+          0.007023001089692116,
+          0.012245233170688152,
+          -0.012706968933343887,
+          -0.017102692276239395,
+          0.028849251568317413,
+          0.02460128255188465,
+          -0.022384950891137123,
+          0.015772894024848938,
+          -0.0045550232753157616,
+          0.01918973959982395,
+          -0.012808550149202347,
+          -0.007309277541935444,
+          -0.0019646857399493456,
+          0.0003399529668968171,
+          0.003322188975289464,
+          0.06257443130016327,
+          -0.03974621742963791,
+          -0.019466780126094818,
+          0.008597520180046558,
+          0.016973407939076424,
+          -0.02321607433259487,
+          -0.01456314604729414,
+          0.028719965368509293,
+          -0.025487814098596573,
+          0.02484138496220112,
+          0.014507737942039967,
+          -0.014535441994667053,
+          -0.0188295841217041,
+          0.015301923267543316,
+          0.02668832801282406,
+          -0.010444463230669498,
+          0.007309277541935444,
+          -0.01326105184853077,
+          0.004903634078800678,
+          0.006468918174505234,
+          0.010324412025511265,
+          -0.008597520180046558,
+          0.005199145060032606,
+          0.021516887471079826,
+          0.0002038852107943967,
+          -0.00269191968254745,
+          -0.00965027790516615,
+          0.014618555083870888,
+          0.007803334388881922,
+          0.000943095306865871,
+          -0.0005249358364380896,
+          -3.19607715937309e-05,
+          -0.006616673897951841,
+          0.021332193166017532,
+          -0.026559041813015938,
+          -0.002375630661845207,
+          -0.007669431157410145,
+          0.024102607741951942,
+          -0.01685335673391819,
+          0.02022402733564377,
+          -0.0070091490633785725,
+          0.026669858023524284,
+          0.00461735762655735,
+          0.00012438007979653776,
+          0.012300641275942326,
+          0.018395552411675453,
+          -0.02292056381702423,
+          0.002876613987609744,
+          -0.024989139288663864,
+          -0.0009125053184106946,
+          -0.0027265497483313084,
+          0.0269469004124403,
+          -0.00020720393513329327,
+          -0.02783343195915222,
+          0.03165660426020622,
+          -0.006048738490790129,
+          0.007323129568248987,
+          -0.012457631528377533,
+          -0.007757160812616348,
+          0.017407439649105072,
+          -0.03361436352133751,
+          0.015006412751972675,
+          0.020002393051981926,
+          0.012817785143852234,
+          -0.017527490854263306,
+          0.030197519809007645,
+          -0.001605686149559915,
+          -0.01935596391558647,
+          0.011534160003066063,
+          0.00199585291557014,
+          0.01070303563028574,
+          -0.0044557503424584866,
+          0.00849132053554058,
+          -0.02127678506076336,
+          -0.012577682733535767,
+          0.019743822515010834,
+          0.018681829795241356,
+          -0.004564258269965649,
+          -0.0005191641394048929,
+          -0.0017672936664894223,
+          -0.024435056373476982,
+          -0.018035398796200752,
+          0.001420991844497621,
+          -0.0008836468332447112,
+          0.0029851216822862625,
+          -0.0051437364891171455,
+          -0.03492569550871849,
+          0.004130226559937,
+          0.021646173670887947,
+          0.014600085094571114,
+          0.0067828986793756485,
+          -0.021313723176717758,
+          0.008320478722453117,
+          0.013990594074130058,
+          -0.023825565353035927,
+          0.03139803186058998,
+          -0.00910542905330658,
+          -0.03719743341207504,
+          0.004407268017530441,
+          0.0051206499338150024,
+          -0.016410090029239655,
+          -0.0004279713612049818,
+          0.01183890551328659,
+          -0.0061780246905982494,
+          0.016945702955126762,
+          0.021036682650446892,
+          -0.016908764839172363,
+          0.0088560925796628,
+          0.013381103053689003,
+          -0.012365284375846386,
+          -0.0014613937819376588,
+          -0.0006545104552060366,
+          -0.00712458323687315,
+          -0.0014867892023175955,
+          0.059323813766241074,
+          -0.012171355076134205,
+          -0.0038901239167898893,
+          -0.025968020781874657,
+          0.024952201172709465,
+          -0.01989157684147358,
+          -0.009668746963143349,
+          -0.001956605352461338,
+          -0.010887729935348034,
+          -0.009982727468013763,
+          0.005956391338258982,
+          -0.014323043636977673,
+          0.018293971195816994,
+          0.03054843842983246,
+          0.027334757149219513,
+          0.003559982869774103,
+          0.014193757437169552,
+          -0.00628422386944294,
+          -0.004693544004112482,
+          0.02434271015226841,
+          -0.019448310136795044,
+          0.0064781527034938335,
+          0.0007786019705235958,
+          -0.0007474347949028015,
+          -0.004432663321495056,
+          -0.010832320898771286,
+          -0.04100213572382927,
+          -0.0292555782943964,
+          0.02249576710164547,
+          0.006173407193273306,
+          0.020888926461338997,
+          0.002327148336917162,
+          0.009677981957793236,
+          -0.0028004273772239685,
+          0.012032833881676197,
+          0.011303291656076908,
+          -0.02391791343688965,
+          -0.014516972936689854,
+          -0.0033337322529405355,
+          -0.004559640772640705,
+          -0.012559212744235992,
+          -0.010555279441177845,
+          0.018949635326862335,
+          0.01800769567489624,
+          0.009548695757985115,
+          -0.015837537124753,
+          0.00036159681621938944,
+          -0.0007029927219264209,
+          -0.0013702008873224258,
+          0.0008253526757471263,
+          -0.007083026692271233,
+          -0.033466607332229614,
+          0.009354766458272934,
+          -0.016770243644714355,
+          0.006196494214236736,
+          0.015412740409374237,
+          -0.010943138040602207,
+          -0.0032575458753854036,
+          0.0046312096528708935,
+          -0.02183086797595024,
+          -0.007817186415195465,
+          -0.018358614295721054,
+          0.006704403553158045,
+          -0.027851901948451996,
+          0.0012097477447241545,
+          0.00803420227020979,
+          0.0010573748731985688,
+          0.026263531297445297,
+          0.00920239370316267,
+          -0.030880888924002647,
+          0.02321607433259487,
+          -0.003158272709697485,
+          0.008805301040410995,
+          -0.0037700727116316557,
+          -0.012402222491800785,
+          0.01636391505599022,
+          0.0074200937524437904,
+          0.00840820837765932,
+          -0.008126549422740936,
+          0.019651474431157112,
+          -0.0001968148717423901,
+          0.007138435263186693,
+          0.017195040360093117,
+          -0.022975971922278404,
+          -0.006085677538067102,
+          -0.0026896109338849783,
+          0.017933817580342293,
+          0.026540571823716164,
+          -0.01368584856390953,
+          -0.017407439649105072,
+          -0.009585634805262089,
+          0.023991791531443596,
+          0.014600085094571114,
+          0.015874475240707397,
+          0.002256733598187566,
+          -0.023142196238040924,
+          -0.0199839249253273,
+          0.02186780609190464,
+          -0.004266438540071249,
+          0.0017430525040253997,
+          0.026374347507953644,
+          -0.028405984863638878,
+          -0.014406155794858932,
+          -0.022089440375566483,
+          0.006034886464476585,
+          0.0012293715262785554,
+          0.009050020948052406,
+          -0.003119025146588683,
+          -0.015883710235357285,
+          -0.003446857677772641,
+          0.0013517314800992608,
+          -0.005240701138973236,
+          0.003998631611466408,
+          0.010130482725799084,
+          -0.003250619862228632,
+          0.005610089749097824,
+          0.011617272160947323,
+          -0.0238440353423357,
+          -0.004372637718915939,
+          0.010305942967534065,
+          -0.02262505330145359,
+          -0.02598648890852928,
+          -0.0027219324838370085,
+          0.014313808642327785,
+          0.016668662428855896,
+          0.006399658042937517,
+          0.006538178771734238,
+          -0.006399658042937517,
+          -0.027981188148260117,
+          0.01860795170068741,
+          -0.004351859912276268,
+          0.00405634893104434,
+          -0.013778195716440678,
+          0.020888926461338997,
+          0.017592133954167366,
+          0.007646344602108002,
+          0.009677981957793236,
+          -0.013519623316824436,
+          -0.03680957481265068,
+          -0.00034803335438482463,
+          0.00910542905330658,
+          -0.006335014943033457,
+          0.003012825967743993,
+          -0.008006498217582703,
+          -0.017832236364483833,
+          0.01883881911635399,
+          -0.007863360457122326,
+          0.0021932448726147413,
+          0.004642753396183252,
+          -0.032044462859630585,
+          0.008763745427131653,
+          0.019540658220648766,
+          0.01857101358473301,
+          -0.01446156483143568,
+          -0.015680547803640366,
+          -0.01276237703859806,
+          0.0107676787301898,
+          0.002585720270872116,
+          0.000738777220249176,
+          -0.0188295841217041,
+          -0.021720051765441895,
+          -0.028405984863638878,
+          0.006256519816815853,
+          0.0005335934110917151,
+          -0.022902093827724457,
+          -0.011238648556172848,
+          -0.005503890570253134,
+          0.01896810531616211,
+          0.01276237703859806,
+          0.007489354349672794,
+          0.02421342395246029,
+          -0.008306626230478287,
+          0.017130397260189056,
+          0.004940572660416365,
+          0.01368584856390953,
+          -0.0142491664737463,
+          -0.0002904606517404318,
+          -0.00988114532083273,
+          0.01926361583173275,
+          0.029477212578058243,
+          -0.008833005093038082,
+          -0.004679691977798939,
+          0.011330995708703995,
+          -0.004598888102918863,
+          0.022107908502221107,
+          0.02341923862695694,
+          -0.01085079088807106,
+          0.03198905289173126,
+          0.0008190038497559726,
+          0.015181872062385082,
+          -0.010527575388550758,
+          -0.013085591606795788,
+          -0.009585634805262089,
+          0.00042508551268838346,
+          -0.028812311589717865,
+          -0.027464043349027634,
+          0.009290123358368874,
+          0.0075216758996248245,
+          0.003876271890476346,
+          -0.015458913519978523,
+          0.009188542142510414,
+          0.0030913210939615965,
+          0.01613304764032364,
+          0.009377853944897652,
+          0.09301205724477768,
+          0.012106711976230145,
+          0.01566207781434059,
+          0.013667378574609756,
+          0.0011185549665242434,
+          -0.013150234706699848,
+          0.005042154807597399,
+          -0.0016368533251807094,
+          9.480012522544712e-05,
+          -0.005910217761993408,
+          -0.014166053384542465,
+          -0.01774912327528,
+          -0.008588285185396671,
+          0.025968020781874657,
+          0.015292689204216003,
+          -0.014877126552164555,
+          -0.004912868607789278,
+          -0.020833518356084824,
+          -0.003654638770967722,
+          -0.013177938759326935,
+          0.009073108434677124,
+          -0.0216092336922884,
+          -0.006501239724457264,
+          0.013565797358751297,
+          0.021184436976909637,
+          0.004109448287636042,
+          -0.015689780935645103,
+          -0.006191876716911793,
+          0.027223940938711166,
+          0.015588199719786644,
+          0.009322444908320904,
+          -0.006542795803397894,
+          -0.004328772891312838,
+          0.025376997888088226,
+          -0.006939888931810856,
+          0.018811115995049477,
+          0.01009354367852211,
+          -0.008523642085492611,
+          -0.0034607097040861845,
+          0.0010025438386946917,
+          0.006787515711039305,
+          0.003964001778513193,
+          0.01446156483143568,
+          0.018644889816641808,
+          0.012365284375846386,
+          -0.009613338857889175,
+          -0.008676014840602875,
+          0.004268747288733721,
+          -0.01913432963192463,
+          0.020205557346343994,
+          -0.019540658220648766,
+          -0.011635741218924522,
+          -0.02345617674291134,
+          0.0006851004436612129,
+          -0.0014579306589439511,
+          0.005637793801724911,
+          -0.006972210016101599,
+          -0.01859871670603752,
+          -0.029366394504904747,
+          0.01935596391558647,
+          0.0013690466294065118,
+          -0.01162650715559721,
+          -0.011700384318828583,
+          0.016807181760668755,
+          -0.005157588515430689,
+          0.01368584856390953,
+          0.0034768704790621996,
+          0.011469516903162003,
+          -0.008648310787975788,
+          0.024859854951500893,
+          0.0030913210939615965,
+          0.0005491769989021122,
+          0.007276955991983414,
+          -0.008648310787975788,
+          0.014064472168684006,
+          0.019466780126094818,
+          0.015819067135453224,
+          -0.0010123556712642312,
+          0.015717485919594765,
+          -0.00630731089040637,
+          -0.005817870609462261,
+          0.012457631528377533,
+          -0.003691577585414052,
+          0.0077987173572182655,
+          0.004176400136202574,
+          0.0026849934365600348,
+          0.020851988345384598,
+          0.005152971483767033,
+          0.020242497324943542,
+          0.02018708735704422,
+          -0.036754168570041656,
+          -0.003506883280351758,
+          -0.012235998176038265,
+          -0.016382385045289993,
+          0.034390080720186234,
+          0.006353484466671944,
+          -0.01009354367852211,
+          0.003511500544846058,
+          -0.01385207287967205,
+          -0.00631654541939497,
+          0.010989311151206493,
+          -0.005623941775411367,
+          0.006588969379663467,
+          0.006579734850674868,
+          -0.01242069248110056,
+          -0.004236425738781691,
+          0.018460195511579514,
+          -8.520467963535339e-05,
+          -0.007914151065051556,
+          -0.01242069248110056,
+          -0.0034837964922189713,
+          0.005222231615334749,
+          -0.04281214252114296,
+          -0.00917007215321064,
+          0.0011710773687809706,
+          0.0009078879957087338,
+          0.006866010837256908,
+          0.009146985597908497,
+          -0.02027943544089794,
+          0.0027773405890911818,
+          0.01893116720020771,
+          -0.03195211663842201,
+          -0.0032390763517469168,
+          -0.01206977292895317,
+          0.010509106330573559,
+          0.010795382782816887,
+          0.0011618426069617271,
+          -0.004658914171159267,
+          -0.0024310387670993805,
+          0.001940444577485323,
+          0.011201709508895874,
+          -0.0031282599084079266,
+          0.015505087561905384,
+          -0.0007918768678791821,
+          0.010758443735539913,
+          -0.00044788370723836124,
+          0.004721248522400856,
+          0.017730653285980225,
+          0.016280803829431534,
+          -0.002516459906473756,
+          -0.014378451742231846,
+          0.009548695757985115,
+          0.002673450158908963,
+          0.013020948506891727,
+          -0.0068244547583162785,
+          0.01939290203154087,
+          -0.02476750686764717,
+          0.0026665241457521915,
+          -0.001568747335113585,
+          0.003139803186058998,
+          -0.0066397604532539845,
+          -0.013981359079480171,
+          -0.001972766127437353,
+          -0.031878236681222916,
+          -0.03298640251159668,
+          -0.01649320125579834,
+          0.00920239370316267,
+          0.005102180410176516,
+          0.011109362356364727,
+          -0.005840957630425692,
+          -0.008740657940506935,
+          0.005166823510080576,
+          0.031157929450273514,
+          0.0008772979490458965,
+          -0.005060624331235886,
+          0.0028142796363681555,
+          0.013122530654072762,
+          0.02552475407719612,
+          -0.007410859223455191,
+          -0.028092004358768463,
+          -0.0006781744305044413,
+          -0.0001631947379792109,
+          -0.02375168912112713,
+          -0.02807353436946869,
+          -0.009013081900775433,
+          -0.0054207779467105865,
+          -0.0020778109319508076,
+          0.0020697305444628,
+          0.01099854614585638,
+          0.007397007197141647,
+          -0.004229499492794275,
+          0.005157588515430689,
+          -0.00984420720487833,
+          0.006898332387208939,
+          0.0036223172210156918,
+          0.01169115025550127,
+          -0.026466695591807365,
+          -0.007974176667630672,
+          -0.025469345971941948,
+          -0.007438563276082277,
+          0.006889097858220339,
+          -0.01446156483143568,
+          0.015301923267543316,
+          0.022052500396966934,
+          0.018589481711387634,
+          -0.010666096583008766,
+          0.029532620683312416,
+          0.016770243644714355,
+          -0.03461171314120293,
+          0.021221376955509186,
+          0.03014211170375347,
+          -0.01681641675531864,
+          -0.0005194527329877019,
+          -0.00033158401492983103,
+          0.019633004441857338,
+          0.012919367291033268,
+          0.010361351072788239,
+          -0.013399572111666203,
+          0.00460812309756875,
+          -0.01517263799905777,
+          -0.05197297781705856,
+          0.0039085932075977325,
+          0.024398118257522583,
+          0.004668148700147867,
+          -0.004483454395085573,
+          0.016253098845481873,
+          0.0005252244300208986,
+          -0.006787515711039305,
+          0.03169354423880577,
+          0.02744557522237301,
+          0.01922667771577835,
+          0.0016841812757775187,
+          -0.006884480360895395,
+          -0.029403334483504295,
+          -0.01676100865006447,
+          0.006404275074601173,
+          0.014110645279288292,
+          0.006990679539740086,
+          -0.01790611259639263,
+          -0.02430577203631401,
+          0.005356134846806526,
+          0.023530054837465286,
+          0.004211030434817076,
+          0.017859939485788345,
+          0.021018212661147118,
+          5.732016870751977e-05,
+          0.01582830213010311,
+          -0.008648310787975788,
+          -0.0006066053756512702,
+          -0.015754424035549164,
+          -0.00013974722241982818,
+          -0.014055237174034119,
+          0.0012859341222792864,
+          0.002643437357619405,
+          0.006173407193273306,
+          0.006621290929615498,
+          0.006672082003206015,
+          -0.009257802739739418,
+          -0.004968277178704739,
+          -3.21411389450077e-05,
+          0.016465498134493828,
+          -0.0068244547583162785,
+          0.004707396030426025,
+          -0.012143651023507118,
+          -0.004317229613661766,
+          0.008546729572117329,
+          0.0007001068443059921,
+          0.007840273901820183,
+          0.012152886018157005,
+          0.004116374533623457,
+          0.010112013667821884,
+          -0.0010764215840026736,
+          0.0024125694762915373,
+          -0.004737408831715584,
+          0.004280290566384792,
+          0.01198666077107191,
+          -0.004174091387540102,
+          -0.006187259219586849,
+          -0.02450893446803093,
+          0.037751518189907074,
+          -0.023179136216640472,
+          -0.002844292437657714,
+          -0.01057374943047762,
+          0.022514237090945244,
+          -0.0042548952624201775,
+          0.01877417601644993,
+          -0.011174005456268787,
+          -0.009368618950247765,
+          0.014784779399633408,
+          -0.006810602732002735,
+          -0.025709448382258415,
+          -0.01015818677842617,
+          -0.005707054398953915,
+          0.02421342395246029,
+          -0.007216930389404297,
+          -0.012688498944044113,
+          0.009142368100583553,
+          -0.0035761436447501183,
+          0.004940572660416365,
+          -0.020334843546152115,
+          -0.006588969379663467,
+          0.00010259191913064569,
+          -0.007553997449576855,
+          -0.009770329110324383,
+          -0.016216160729527473,
+          0.009327062405645847,
+          -0.003246002597734332,
+          0.0037423684261739254,
+          0.014156819321215153,
+          -0.008676014840602875,
+          -0.0014325352385640144,
+          0.021129028871655464,
+          0.0033129542134702206,
+          -0.004628901369869709,
+          0.009908849373459816,
+          -0.01169115025550127,
+          0.01952218823134899,
+          -0.015412740409374237,
+          -0.002085891319438815,
+          -0.007710987236350775,
+          -0.0077063702046871185,
+          -0.006584352347999811,
+          0.007004531566053629,
+          0.0031074818689376116,
+          0.021036682650446892,
+          -0.003495339769870043,
+          -0.009724155068397522,
+          -0.017832236364483833,
+          0.006805985234677792,
+          0.018848054111003876,
+          -0.000645852938760072,
+          -0.001607994781807065,
+          -0.0123375803232193,
+          -0.013141000643372536,
+          0.005268405191600323,
+          -0.0039940145798027515,
+          0.00269191968254745,
+          -0.026245061308145523,
+          -0.010028901509940624,
+          0.0045388625003397465,
+          0.0315273180603981,
+          -0.0075124409049749374,
+          -0.010527575388550758,
+          0.005106797907501459,
+          0.008075758814811707,
+          -0.01959606632590294,
+          -0.016945702955126762,
+          0.003282941412180662,
+          0.018183154985308647,
+          -0.0012697733473032713,
+          -0.0032021375373005867,
+          0.0013794356491416693,
+          -0.0016483967192471027,
+          -0.015200342051684856,
+          0.002309833187609911,
+          0.011090893298387527,
+          -0.030529970303177834,
+          0.003931680228561163,
+          -0.049498073756694794,
+          0.027464043349027634,
+          0.013713552616536617,
+          0.002952800365164876,
+          -0.008837622590363026,
+          -0.022551175206899643,
+          -0.023659341037273407,
+          -0.002936639590188861,
+          0.042073365300893784,
+          0.009096194989979267,
+          -0.010952373035252094,
+          0.022846685722470284,
+          -0.0017118854448199272,
+          -0.004234116990119219,
+          -0.01347345020622015,
+          0.0077479262836277485,
+          -0.0062011112459003925,
+          0.008117315359413624,
+          0.009373236447572708,
+          0.008232749067246914,
+          0.01800769567489624,
+          0.0010452544083818793,
+          0.010582984425127506,
+          -0.02870149537920952,
+          -0.0035899956710636616,
+          0.010906198993325233,
+          0.008528259582817554,
+          0.001420991844497621,
+          0.02031637355685234,
+          0.013584266416728497,
+          -0.00829277466982603,
+          -0.002938948106020689,
+          0.019152799621224403,
+          -0.014618555083870888,
+          0.014960238710045815,
+          0.004155621863901615,
+          0.009724155068397522,
+          0.0007053014123812318,
+          -0.001627618563361466,
+          0.025340059772133827,
+          0.003255237126722932,
+          -0.026392817497253418,
+          -0.021627703681588173,
+          0.010666096583008766,
+          -0.017869174480438232,
+          -0.00643197912722826,
+          -0.005836340133100748,
+          0.0038023940287530422,
+          0.010786147788167,
+          0.010462932288646698,
+          0.01926361583173275,
+          0.011044720187783241,
+          0.0016599401133134961,
+          -0.0030382215045392513,
+          0.013214877806603909,
+          -0.01804463379085064,
+          0.0013055579038336873,
+          -0.018654124811291695,
+          0.00895767379552126,
+          0.012125181034207344,
+          -0.00037631465238519013,
+          0.007295425049960613,
+          0.002249807585030794,
+          -0.0028373664245009422,
+          0.014877126552164555,
+          0.0079002995043993,
+          0.011257118545472622,
+          -0.027500983327627182,
+          0.0024541255552321672,
+          -0.02572791650891304,
+          0.0200208630412817,
+          -0.016530141234397888,
+          0.008463616482913494,
+          0.013094826601445675,
+          -0.0031467291992157698,
+          0.012956305406987667,
+          0.010980077087879181,
+          -0.002865070477128029,
+          0.004982129205018282,
+          0.023991791531443596,
+          -0.013861307874321938,
+          -0.05721829831600189,
+          0.0028927745297551155,
+          0.004825138952583075,
+          0.007854125462472439,
+          0.012439161539077759,
+          0.017231978476047516,
+          -0.0056978194043040276,
+          -0.014867891557514668,
+          -0.006898332387208939,
+          -0.002091662958264351,
+          -0.004252586513757706,
+          -0.0015918341232463717,
+          -0.0003933411499019712,
+          0.016253098845481873,
+          -0.0017038050573319197,
+          -0.017268918454647064,
+          0.001054489053785801,
+          -0.0022094056475907564,
+          0.004414193797856569,
+          0.01827550120651722,
+          -0.015782129019498825,
+          0.0057670800015330315,
+          0.01632697694003582,
+          0.0161976907402277,
+          0.006792133208364248,
+          0.008615989238023758,
+          -0.006191876716911793,
+          0.004991363734006882,
+          0.009456348605453968,
+          0.013944420032203197,
+          -0.006122616585344076,
+          -0.012079007923603058,
+          0.007881829515099525,
+          -0.01072150468826294,
+          0.02127678506076336,
+          -0.0030728515703231096,
+          -0.013676613569259644,
+          -0.01538503635674715,
+          -0.009761094115674496,
+          -0.009770329110324383,
+          0.005448481999337673,
+          0.001754595898091793,
+          0.003354510525241494,
+          -0.0028696877416223288,
+          0.01242069248110056,
+          0.0014094484504312277,
+          -0.016567079350352287,
+          0.0004233539802953601,
+          0.010416759178042412,
+          0.02905241586267948,
+          -0.0009754167986102402,
+          -0.018164684996008873,
+          -0.010204360820353031,
+          -0.01985463872551918,
+          0.02988353930413723,
+          0.0020085505675524473,
+          -0.0003895895497407764,
+          0.005157588515430689,
+          -0.006556647829711437,
+          -0.005517742596566677,
+          -0.007881829515099525,
+          0.01141410879790783,
+          0.0021389909088611603,
+          0.02220025658607483,
+          0.0008761436329223216,
+          0.0074708848260343075,
+          -0.006704403553158045,
+          0.004700470250099897,
+          -0.002366395900025964,
+          -0.004208721686154604,
+          0.003599230432882905,
+          0.0066397604532539845,
+          -0.0029597263783216476,
+          0.0034768704790621996,
+          0.005665497854351997,
+          -0.013547327369451523,
+          0.004811286926269531,
+          -0.02199709229171276,
+          0.030566908419132233,
+          0.006395040545612574,
+          0.015874475240707397,
+          0.004114065784960985,
+          -0.017952287569642067,
+          0.006727490108460188,
+          0.01183890551328659,
+          -0.0041209920309484005,
+          0.031213337555527687,
+          -0.002458743052557111,
+          -0.043550919741392136,
+          -0.00550850760191679,
+          0.02005780301988125,
+          0.03307875245809555,
+          0.0011791577562689781,
+          -0.01705651916563511,
+          0.0024471995420753956,
+          -0.0011197092244401574,
+          0.009142368100583553,
+          0.019577596336603165,
+          0.00720769539475441,
+          0.0040771267376840115,
+          -0.003915519453585148,
+          -0.014073706232011318,
+          -0.0031282599084079266,
+          0.020796580240130424,
+          -0.005845575127750635,
+          -0.008394356817007065,
+          0.009031551890075207,
+          0.01668713055551052,
+          -0.01863565668463707,
+          0.01070303563028574,
+          0.007443180773407221,
+          -0.02790731005370617,
+          -0.017102692276239395,
+          0.012365284375846386,
+          0.023437708616256714,
+          0.002456434303894639,
+          -0.014701667241752148,
+          -0.012079007923603058,
+          0.02245882898569107,
+          0.01642855815589428,
+          0.0061364686116576195,
+          -0.010610688477754593,
+          -0.0017430525040253997,
+          -0.011007781140506268,
+          -0.02114749886095524,
+          -0.0006423898739740252,
+          0.018349379301071167,
+          -0.014323043636977673,
+          0.008814536035060883,
+          -0.0005018490483053029,
+          0.0024310387670993805,
+          0.006339631974697113,
+          0.005799401551485062,
+          -0.011894313618540764,
+          0.005180675536394119,
+          0.0058317226357758045,
+          -0.005928687285631895,
+          0.004176400136202574,
+          0.0065058572217822075,
+          0.0012339889071881771,
+          -0.025081487372517586,
+          -0.003864728379994631,
+          -8.809052815195173e-05,
+          0.002162077696993947,
+          0.0010816160356625915,
+          -0.026965368539094925,
+          -0.012402222491800785,
+          -0.014840187504887581,
+          -0.025340059772133827,
+          -0.003996323328465223,
+          -0.01456314604729414,
+          -0.02031637355685234,
+          -0.001719965715892613,
+          -0.0035692176315933466,
+          0.0008086147718131542,
+          -0.0006602821522392333,
+          0.011072424240410328,
+          0.014036767184734344,
+          -0.0067321076057851315,
+          0.003070542821660638,
+          -0.0027911928482353687,
+          0.0323953814804554,
+          0.008583667688071728,
+          0.003964001778513193,
+          0.010832320898771286,
+          0.011183240450918674,
+          0.004116374533623457,
+          -0.006755194626748562,
+          -0.012910132296383381,
+          0.011487985961139202,
+          -0.00905463844537735,
+          -0.015985293313860893,
+          0.013464215211570263,
+          -0.010259768925607204,
+          0.0074708848260343075,
+          -0.0010060068452730775,
+          -0.015726720914244652,
+          -0.008315861225128174,
+          -0.0015952971298247576,
+          -0.01622539572417736,
+          -0.03507344797253609,
+          0.009631807915866375,
+          0.013972125016152859,
+          0.0013932876754552126,
+          -0.007397007197141647,
+          -0.028405984863638878,
+          -0.007424711249768734,
+          -0.006939888931810856,
+          0.004471911117434502,
+          0.01830320619046688,
+          0.00926703680306673,
+          -0.011469516903162003,
+          -0.008435912430286407,
+          -0.010361351072788239,
+          -0.003213681047782302,
+          0.005743992980569601,
+          0.0025372381787747145,
+          -0.000359576748451218,
+          -0.010361351072788239,
+          0.02173851989209652,
+          -0.01108165830373764,
+          0.0008034202619455755,
+          0.01448003388941288,
+          -0.026078836992383003,
+          0.0024841385893523693,
+          -0.02908935397863388,
+          -0.01681641675531864,
+          0.02648516371846199,
+          0.0025372381787747145,
+          0.013935185968875885,
+          -0.017333561554551125,
+          -0.004176400136202574,
+          0.018450962379574776,
+          -0.006921419408172369,
+          -0.025672508403658867,
+          -0.005748610477894545,
+          -0.010555279441177845,
+          -0.018432492390275,
+          0.012272937223315239,
+          0.0026064985431730747,
+          -0.014489268884062767,
+          -0.01542197447270155,
+          0.011801966466009617,
+          -0.005448481999337673,
+          -0.007807951886206865,
+          -0.02262505330145359,
+          0.026632919907569885,
+          -0.005268405191600323,
+          -0.019208207726478577,
+          -0.00136673788074404,
+          -0.014258400537073612,
+          0.0058825137093663216,
+          -0.00915160309523344,
+          0.005550064146518707,
+          0.010250533930957317,
+          -0.030326806008815765,
+          -0.009761094115674496,
+          -0.012836254201829433,
+          -0.01418452337384224,
+          -0.03911825641989708,
+          -0.005171440541744232,
+          0.011737323366105556,
+          -0.001036019646562636,
+          -0.01827550120651722,
+          -0.007609405554831028,
+          -0.013353399001061916,
+          -0.005009833257645369,
+          -0.0030543820466846228,
+          0.0073046600446105,
+          -0.007784865330904722,
+          -0.010619922541081905,
+          0.018524838611483574,
+          0.014433860778808594,
+          0.012125181034207344,
+          -0.019873108714818954,
+          -2.845266408257885e-06,
+          -0.00039622699841856956,
+          0.015135698951780796,
+          0.001699187676422298,
+          0.015034116804599762,
+          0.01057374943047762,
+          -0.020408721640706062,
+          -0.004926720634102821,
+          0.007632492575794458,
+          -0.013177938759326935,
+          -0.009419409558176994,
+          -0.0034791789948940277,
+          0.031915176659822464,
+          -0.00627498934045434,
+          0.027556391432881355,
+          -0.004074818454682827,
+          0.008994612842798233,
+          0.01527421921491623,
+          -0.0025187686551362276,
+          -0.026245061308145523,
+          -0.0038831979036331177,
+          -0.005799401551485062,
+          -0.010232064872980118,
+          -0.0006856776308268309,
+          -0.007821803912520409,
+          0.020574945956468582,
+          -0.005693201906979084,
+          0.011404873803257942,
+          0.0019889269024133682,
+          0.010813851840794086,
+          -0.016779478639364243,
+          -0.024749036878347397,
+          0.011912782676517963,
+          0.010102778673171997,
+          -0.0022255664225667715,
+          0.020925864577293396,
+          0.005448481999337673,
+          -0.0019150491571053863,
+          0.024121077731251717,
+          -0.004418811295181513,
+          -0.006667464505881071,
+          -0.01731509156525135,
+          0.0034214621409773827,
+          0.01599452644586563,
+          0.019208207726478577,
+          -0.019633004441857338,
+          0.011811201460659504,
+          0.038970500230789185,
+          0.00713381776586175,
+          -0.016945702955126762,
+          -0.026411285623908043,
+          -0.005730140954256058,
+          -0.008837622590363026,
+          0.005314578767865896,
+          -0.03274630010128021,
+          0.0002919035905506462,
+          -0.005157588515430689,
+          -0.005725523456931114,
+          0.016105344519019127,
+          0.011718854308128357,
+          -0.0038624198641628027,
+          -0.0014302266063168645,
+          0.012965540401637554,
+          -0.029957417398691177,
+          0.0003561137127690017,
+          0.008708336390554905,
+          -0.016142282634973526,
+          -0.007332364097237587,
+          0.007877212017774582,
+          -0.013353399001061916,
+          -0.004550406243652105,
+          -0.013399572111666203,
+          0.01265155989676714,
+          -0.00977956410497427,
+          -0.010398290120065212,
+          0.019651474431157112,
+          -0.004116374533623457,
+          -0.013307224959135056,
+          -0.0021828559692949057,
+          0.004026336129754782,
+          -0.03446395695209503,
+          0.020999742671847343,
+          0.005840957630425692,
+          0.00155258656013757,
+          -0.01164497621357441,
+          0.011995895765721798,
+          0.007766395807266235,
+          0.005323813296854496,
+          -0.009936554357409477,
+          -0.008126549422740936,
+          -0.008980761282145977,
+          0.003707738360390067,
+          0.011423342861235142,
+          -0.004164856858551502,
+          -0.009696451015770435,
+          0.0034122273791581392,
+          0.010582984425127506,
+          0.016964172944426537,
+          -0.0064781527034938335,
+          -0.013889011926949024,
+          0.0011878153309226036,
+          -0.002876613987609744,
+          -0.014637024141848087,
+          -0.0035345873329788446,
+          0.009123899042606354,
+          -0.016345446929335594,
+          -0.000193640444194898,
+          0.011580333113670349,
+          -0.010185890831053257,
+          -0.01918973959982395,
+          -0.025210773572325706,
+          -0.000573129509575665,
+          -0.006561265327036381,
+          0.000608914066106081,
+          -0.021553825587034225,
+          0.014720136299729347,
+          -0.0006037194980308414,
+          -0.022052500396966934,
+          0.017398204654455185,
+          -0.005069858860224485,
+          -0.00199585291557014,
+          0.030289866030216217,
+          0.00716613931581378,
+          -0.0015006412286311388,
+          -0.01972535252571106,
+          0.007821803912520409,
+          -0.005660880822688341,
+          -0.0067321076057851315,
+          -0.0007705215830355883,
+          0.02044565975666046,
+          -0.005106797907501459,
+          -0.005993330385535955,
+          0.025026079267263412,
+          0.02081504836678505,
+          0.013307224959135056,
+          -0.014904830604791641,
+          0.004610431846231222,
+          -0.026337409391999245,
+          0.022274134680628777,
+          0.009705686010420322,
+          -0.011358699761331081,
+          -0.007794099859893322,
+          0.0008917272207327187,
+          -0.014683197252452374,
+          0.025949550792574883,
+          0.011174005456268787,
+          -0.011958956718444824,
+          -0.008883796632289886,
+          -0.0023029071744531393,
+          0.0030797775834798813,
+          -0.008288157172501087,
+          -0.008551346138119698,
+          -0.004458059091120958,
+          0.018718767911195755,
+          0.005517742596566677,
+          -0.001165305613540113,
+          -0.012559212744235992,
+          -0.014646259136497974,
+          0.011958956718444824,
+          0.014341513626277447,
+          -0.004615048877894878,
+          -0.028092004358768463,
+          0.010564514435827732,
+          0.016530141234397888,
+          -0.00984420720487833,
+          0.004400341771543026,
+          -0.005730140954256058,
+          0.031675074249506,
+          -0.0019877725280821323,
+          -0.01754595898091793,
+          -0.008061906322836876,
+          0.026078836992383003,
+          0.0023410003632307053,
+          0.022237194702029228,
+          0.005891748704016209,
+          -0.0054900385439395905,
+          -0.014673963189125061,
+          0.0011179777793586254,
+          0.005864044185727835,
+          0.01198666077107191,
+          0.021479947492480278,
+          0.00395014975219965,
+          -0.006279606372117996,
+          0.00994578842073679,
+          -0.013399572111666203,
+          -0.012679263949394226,
+          0.00181231286842376,
+          -0.017435142770409584,
+          0.007600171025842428,
+          -0.023899443447589874,
+          -0.0014613937819376588,
+          0.013011714443564415,
+          0.009410175494849682,
+          0.0011231722310185432,
+          0.006099529564380646,
+          0.008385121822357178,
+          0.0008888413431122899,
+          -0.014655493199825287,
+          -0.005374604370445013,
+          -0.009225481189787388,
+          0.011377169750630856,
+          0.010435228236019611,
+          -0.010407524183392525,
+          -0.022255664691329002,
+          0.018284736201167107,
+          -0.016243863850831985,
+          0.007447798270732164,
+          -0.005032919812947512,
+          -0.0016022231429815292,
+          0.006127233617007732,
+          0.013205642811954021,
+          -0.02661444991827011,
+          0.012799316085875034,
+          -0.0021666951943188906,
+          -0.03017904981970787,
+          0.0025557074695825577,
+          -0.004317229613661766,
+          -0.0004957888158969581,
+          -0.014997177757322788,
+          0.01527421921491623,
+          -0.010619922541081905,
+          0.009350149892270565,
+          0.00536075234413147,
+          -0.0005206071073189378,
+          -0.003250619862228632,
+          0.004712013527750969,
+          -0.0014579306589439511,
+          -0.012790081091225147,
+          0.016613252460956573,
+          0.011765027418732643,
+          -0.005263787694275379,
+          0.007258486468344927,
+          -0.004656605422496796,
+          -0.014313808642327785,
+          -0.030234457924962044,
+          0.005979478359222412,
+          0.010213595815002918,
+          0.002398717449977994,
+          -0.00888841412961483,
+          -0.0008876870269887149,
+          0.0038531851023435593,
+          0.013759725727140903,
+          0.004485763143748045,
+          -0.010555279441177845,
+          -0.02476750686764717,
+          0.011866609565913677,
+          0.013002479448914528,
+          -0.012079007923603058,
+          -0.005887131206691265,
+          0.013787430711090565,
+          -0.006427362095564604,
+          0.001054489053785801,
+          0.0037746899761259556,
+          8.174165850505233e-05,
+          0.01162650715559721,
+          0.014710902236402035,
+          -0.012134416028857231,
+          0.022643523290753365,
+          -0.0015572039410471916,
+          -0.004610431846231222,
+          -0.006085677538067102,
+          0.0023525438737124205,
+          -0.011044720187783241,
+          0.0007659042021259665,
+          -0.0025764857418835163,
+          0.01698264107108116,
+          -0.0033291149884462357,
+          0.011875844560563564,
+          -0.008154253475368023,
+          0.00728157302364707,
+          -0.0005664920900017023,
+          -0.01106318924576044,
+          0.003303719451650977,
+          -0.01248533558100462,
+          0.011958956718444824,
+          -0.005600854754447937,
+          0.004275673069059849,
+          -0.022939033806324005,
+          0.017527490854263306,
+          0.009918084368109703,
+          -0.012374518439173698,
+          -0.017111927270889282,
+          0.013732021674513817,
+          -0.006048738490790129,
+          -0.047614194452762604,
+          -0.011894313618540764,
+          -0.02833210676908493,
+          0.00019970072025898844,
+          0.014720136299729347,
+          0.02816588245332241,
+          -0.012965540401637554,
+          -0.00265959813259542,
+          0.01748131588101387,
+          0.01072150468826294,
+          0.0009956177091225982,
+          0.01596682332456112,
+          -0.006939888931810856,
+          0.00965027790516615,
+          0.0034607097040861845,
+          0.0022982899099588394,
+          -0.001016973052173853,
+          -0.00042421973193995655,
+          -0.007406241726130247,
+          0.029606498777866364,
+          -0.008168105967342854,
+          0.004778965376317501,
+          -0.012162120081484318,
+          0.00811269786208868,
+          -0.0222926028072834,
+          0.0017257374711334705,
+          0.014710902236402035,
+          -0.021720051765441895,
+          0.006468918174505234,
+          0.003730825148522854,
+          -0.015680547803640366,
+          -0.0036500212736427784,
+          0.013962890021502972,
+          -0.01880188100039959,
+          -0.0025372381787747145,
+          0.004912868607789278,
+          0.012984010390937328,
+          0.008929969742894173,
+          0.004760495852679014,
+          -0.005762462504208088,
+          0.020999742671847343,
+          0.0011185549665242434,
+          0.008357417769730091,
+          0.004058657679706812,
+          0.00638118851929903,
+          0.0006348866736516356,
+          0.01599452644586563,
+          -0.009257802739739418,
+          0.003559982869774103,
+          0.019910046830773354,
+          -0.02522924356162548,
+          -0.01699187606573105,
+          0.03274630010128021,
+          0.0016726378817111254,
+          0.021978622302412987,
+          -0.0013690466294065118,
+          0.0031351859215646982,
+          0.0066397604532539845,
+          0.006214963737875223,
+          -0.0033129542134702206,
+          0.00624266779050231,
+          0.007276955991983414,
+          0.011238648556172848,
+          0.01741667278110981,
+          8.679189340909943e-05,
+          0.006057973485440016,
+          0.01347345020622015,
+          -0.02358546294271946,
+          0.006076443009078503,
+          0.005032919812947512,
+          0.007752543780952692,
+          -0.004238734487444162,
+          0.01926361583173275,
+          -0.003153655445203185,
+          -0.005254553165286779,
+          0.017527490854263306,
+          0.028646087273955345,
+          -0.0004600042593665421,
+          0.01566207781434059,
+          0.026503633707761765,
+          -0.004522701725363731,
+          -0.009936554357409477,
+          -0.006418127100914717,
+          -0.018284736201167107,
+          -0.0013078665360808372,
+          -0.0008201581658795476,
+          -0.0017569046467542648,
+          -0.01553279161453247,
+          0.0009119281312450767,
+          0.01622539572417736,
+          0.01353809330612421,
+          -0.025949550792574883,
+          0.01952218823134899,
+          0.0006902950117364526,
+          0.00324831111356616,
+          -0.00979803316295147,
+          0.014747840352356434,
+          -0.024250362068414688,
+          -0.002902009291574359,
+          0.017305856570601463,
+          -0.0161607526242733,
+          -0.012392988428473473,
+          0.0073138945735991,
+          -0.009211628697812557,
+          -0.011635741218924522,
+          0.006750577129423618,
+          0.0038624198641628027,
+          0.0108046168461442,
+          -0.00038843523361720145,
+          -0.00716613931581378,
+          -0.006588969379663467,
+          0.025173833593726158,
+          0.007383155170828104,
+          -0.005670115351676941,
+          -0.020297905430197716,
+          -0.009825737215578556,
+          -0.011977425776422024,
+          0.016650192439556122,
+          0.0017222744645550847,
+          -0.00720769539475441,
+          -0.0022648139856755733,
+          -0.013741256669163704,
+          -0.007831038907170296,
+          -0.02249576710164547,
+          0.009853441268205643,
+          0.014701667241752148,
+          0.009096194989979267,
+          0.010121248662471771,
+          0.0024218042381107807,
+          0.012300641275942326,
+          -0.000943095306865871,
+          -0.0007555151823908091,
+          0.009114664047956467,
+          0.014369217678904533,
+          -0.003474561730399728,
+          0.019946984946727753,
+          -0.012402222491800785,
+          0.009687216952443123,
+          -0.022089440375566483,
+          0.005517742596566677,
+          -0.00977956410497427,
+          0.005730140954256058,
+          0.005176058039069176,
+          -0.007503206375986338,
+          0.019873108714818954,
+          0.00545771699398756,
+          0.0006037194980308414,
+          -0.0003411073121242225,
+          0.014175288379192352,
+          0.0057901665568351746,
+          0.00893920473754406,
+          -0.01918973959982395,
+          0.0017661392921581864,
+          0.0014279178576543927,
+          0.010518341325223446,
+          -0.0028489097021520138,
+          0.010462932288646698,
+          -0.007567849475890398,
+          0.012365284375846386,
+          0.004995981231331825,
+          -0.00709226168692112,
+          0.026263531297445297,
+          0.0064781527034938335,
+          -0.0011947413440793753,
+          0.01022282987833023,
+          0.015440444462001324,
+          -0.015708250924944878,
+          -0.0012524583144113421,
+          -0.020427191630005836,
+          0.00012603943469002843,
+          0.0022255664225667715,
+          -0.02484138496220112,
+          0.0238809734582901,
+          0.023530054837465286,
+          0.0177121851593256,
+          -0.008948439732193947,
+          -0.004940572660416365,
+          -0.007928003557026386,
+          -0.0054623340256512165,
+          0.0018515604315325618,
+          -0.0005136810359545052,
+          0.023142196238040924,
+          0.009303975850343704,
+          0.035867635160684586,
+          0.01326105184853077,
+          -0.0020893544424325228,
+          0.0005399422952905297,
+          -0.004557332023978233,
+          -0.021258315071463585,
+          0.00941479206085205,
+          -0.01555126067250967,
+          -0.019041983410716057,
+          0.013205642811954021,
+          -0.00550850760191679,
+          -0.0007543608080595732,
+          -0.004416502546519041,
+          -0.008814536035060883,
+          0.015569730661809444,
+          -0.024102607741951942,
+          -0.014646259136497974,
+          0.016234630718827248,
+          -0.004765113350003958,
+          -0.017730653285980225,
+          -0.009276271797716618,
+          -0.02988353930413723,
+          -0.0038831979036331177,
+          0.02236648090183735,
+          0.002456434303894639,
+          0.0013182556722313166,
+          -0.001848097424954176,
+          -0.007678666152060032,
+          -0.006898332387208939,
+          0.002126293256878853,
+          0.021627703681588173,
+          -0.017989225685596466,
+          0.01101701520383358,
+          -0.01741667278110981,
+          -0.010001196525990963,
+          -0.0029089353047311306,
+          -0.019946984946727753,
+          0.009276271797716618,
+          0.004688926972448826,
+          -0.011127832345664501,
+          -0.0018746472196653485,
+          -0.017896879464387894,
+          -0.021165968850255013,
+          0.013667378574609756,
+          -0.003936297260224819,
+          0.012522274628281593,
+          0.008712953887879848,
+          0.006953740958124399,
+          0.002585720270872116,
+          0.058363400399684906,
+          -0.012291406281292439,
+          0.021018212661147118,
+          -0.013048653490841389,
+          0.03017904981970787,
+          -0.005365369841456413,
+          0.01781376637518406,
+          0.00918392464518547,
+          0.00460812309756875,
+          0.0018931166268885136,
+          0.0292555782943964,
+          0.022440358996391296,
+          -0.016465498134493828,
+          0.013233347795903683,
+          0.00042941427091136575,
+          0.009031551890075207,
+          -0.004982129205018282,
+          -0.011072424240410328,
+          -0.011487985961139202,
+          -0.020427191630005836,
+          -0.008509790524840355,
+          -0.00817734096199274,
+          -0.0032021375373005867,
+          -0.002143608406186104,
+          -0.003262163372710347,
+          -0.009918084368109703,
+          0.00938247051090002,
+          -0.02170158177614212,
+          0.028110474348068237,
+          0.01790611259639263,
+          0.003031295258551836,
+          0.0009298204095102847,
+          -0.003749294439330697,
+          0.007517058402299881,
+          0.0030405300203710794,
+          -0.0053699868731200695,
+          0.0003194634336978197,
+          0.01942984201014042,
+          -0.006935271434485912,
+          0.002050106879323721,
+          -0.009340914897620678,
+          0.024896793067455292,
+          -0.020168619230389595,
+          -0.0014579306589439511,
+          -0.002174775581806898,
+          -0.0004100790829397738,
+          -0.004116374533623457,
+          0.011783497408032417,
+          -0.0007058785413391888,
+          0.004146387334913015,
+          0.012263702228665352,
+          0.008482086472213268,
+          0.019005045294761658,
+          0.0216092336922884,
+          -0.005134501960128546,
+          0.0044049592688679695,
+          -0.007461650297045708,
+          -0.007226164918392897,
+          -0.012272937223315239,
+          -0.0008438221411779523,
+          -0.002327148336917162,
+          -0.022588113322854042,
+          -0.02820282056927681,
+          -0.0005673578125424683,
+          0.00369388610124588,
+          0.003262163372710347,
+          0.0034791789948940277,
+          -0.014304574579000473,
+          -0.01972535252571106,
+          0.023234544321894646,
+          -0.009834972210228443,
+          -0.02354852482676506,
+          0.013445746153593063,
+          -0.0011023941915482283,
+          -0.012900897301733494,
+          0.0027034629601985216,
+          -0.007590936031192541,
+          -0.032598547637462616,
+          0.004146387334913015,
+          -0.006150320637971163,
+          0.007447798270732164,
+          -0.033097218722105026,
+          -0.017222745344042778,
+          -0.0038347155787050724,
+          0.007475502323359251,
+          0.0015745189739391208,
+          0.062352798879146576,
+          -0.005651645828038454,
+          0.017231978476047516,
+          -0.015264985151588917,
+          -0.0012997862650081515,
+          -0.004875930026173592,
+          0.004940572660416365,
+          0.013104061596095562,
+          -0.01658554933965206,
+          -0.012134416028857231,
+          0.010961607098579407,
+          0.009017699398100376,
+          0.01276237703859806,
+          0.026208123192191124,
+          0.001071804203093052,
+          -0.014018298126757145,
+          -0.0033498930279165506,
+          -0.004520393442362547,
+          0.006621290929615498,
+          -0.006709021050482988,
+          0.0246751606464386,
+          -0.0005027148290537298,
+          -0.022846685722470284,
+          0.0003916096466127783,
+          -0.012725437991321087,
+          0.016899529844522476,
+          -0.013676613569259644,
+          -0.005217614118009806,
+          -0.017924582585692406,
+          -0.0254324059933424,
+          -0.018894227221608162,
+          0.007637109607458115,
+          0.0021701580844819546,
+          0.012291406281292439,
+          -0.0016680205008015037,
+          0.004857460502535105,
+          -0.007720222231000662,
+          0.0238809734582901,
+          -0.012291406281292439,
+          -0.010564514435827732,
+          0.002860453212633729,
+          -0.026559041813015938,
+          0.0019970072899013758,
+          0.01708422414958477,
+          -0.002918170066550374,
+          -0.008366651833057404,
+          -0.020870456472039223,
+          -0.03097323514521122,
+          0.016040701419115067,
+          -0.004843608476221561,
+          -0.003841641591861844,
+          -0.009567164815962315,
+          0.017102692276239395,
+          0.02035331353545189,
+          -0.002636511344462633,
+          0.03937682881951332,
+          0.02220025658607483,
+          -0.014747840352356434,
+          0.01658554933965206,
+          -0.016317741945385933,
+          -0.002957417629659176,
+          -0.011608037166297436,
+          0.014978708699345589,
+          -0.004079435486346483,
+          -0.030271397903561592,
+          0.025284651666879654,
+          0.0026526721194386482,
+          -0.0008380504441447556,
+          -0.006935271434485912,
+          0.0006423898739740252,
+          0.022218724712729454,
+          0.0013009405229240656,
+          -0.010379820130765438,
+          -0.010121248662471771,
+          0.009243950247764587,
+          0.009511756710708141,
+          0.0009482898749411106,
+          -0.03134262561798096,
+          0.013104061596095562,
+          0.005563916172832251,
+          -0.003269089385867119,
+          0.02123984508216381,
+          -0.012734672985970974,
+          0.021498417481780052,
+          0.020094741135835648,
+          -0.022237194702029228,
+          0.0007347370265051723,
+          -0.03387293592095375,
+          -0.014757075347006321,
+          0.00905463844537735,
+          -0.010213595815002918,
+          0.012476100586354733,
+          -0.0054023084230721,
+          -0.005291492212563753,
+          0.008925352245569229,
+          -0.01093390304595232,
+          -0.011266352608799934,
+          0.006958357989788055,
+          -0.0034122273791581392,
+          -0.007115348242223263,
+          0.003271397901698947,
+          0.012282171286642551,
+          0.020925864577293396,
+          -0.010730739682912827,
+          -0.0062934583984315395,
+          -0.009368618950247765,
+          0.028904659673571587,
+          0.00917007215321064,
+          -0.007623257581144571,
+          0.0039847795851528645,
+          0.005009833257645369,
+          0.006667464505881071,
+          0.00632577994838357,
+          0.004695852752774954,
+          0.013205642811954021,
+          0.007817186415195465,
+          0.023179136216640472,
+          -0.01446156483143568,
+          0.02345617674291134,
+          0.00627498934045434,
+          -0.010730739682912827,
+          -0.0012028217315673828,
+          0.0009032706147991121,
+          0.009567164815962315,
+          -0.022181786596775055,
+          -0.011358699761331081,
+          -0.005448481999337673,
+          -0.024859854951500893,
+          0.01926361583173275,
+          0.003287558676674962,
+          -0.0007843736675567925,
+          0.006528943777084351,
+          0.0025187686551362276,
+          0.0016934159211814404,
+          0.0018769559683278203,
+          -0.00024255557218566537,
+          -0.004224882461130619,
+          0.004252586513757706,
+          0.022902093827724457,
+          0.004321846645325422,
+          0.00014443672262132168,
+          0.0554082915186882,
+          -0.0029481828678399324,
+          -0.01141410879790783,
+          0.02245882898569107,
+          0.0018365540308877826,
+          0.011958956718444824,
+          -0.006298075895756483,
+          0.012429927475750446,
+          -0.026559041813015938,
+          -0.01099854614585638,
+          -0.03265395388007164,
+          0.017259683459997177,
+          0.002401025965809822,
+          -0.030419152230024338,
+          -0.007650961633771658,
+          0.0051852925680577755,
+          0.017158102244138718,
+          -0.011672680266201496,
+          0.026337409391999245,
+          0.007466267328709364,
+          -0.008994612842798233,
+          -0.00020330803818069398,
+          -0.01070303563028574,
+          0.006104147061705589,
+          -0.01689029484987259,
+          -0.01286395825445652,
+          -0.005356134846806526,
+          0.0019854637794196606,
+          0.00979803316295147,
+          0.02652210369706154,
+          0.022680461406707764,
+          -0.012208294123411179,
+          0.015652842819690704,
+          0.022865155711770058,
+          0.0018908079946413636,
+          0.012429927475750446,
+          -0.006376571021974087,
+          -0.024582812562584877,
+          -0.0378069244325161,
+          0.00639042304828763,
+          0.04159316048026085,
+          -0.005065241362899542,
+          0.004848225507885218,
+          0.027630269527435303,
+          0.00046029285294935107,
+          0.01714886724948883,
+          -0.005494655575603247,
+          0.005845575127750635,
+          0.01676100865006447,
+          0.0050513893365859985,
+          -0.04369867220520973,
+          0.011571098119020462,
+          0.0034976485185325146,
+          -0.005342282820492983,
+          0.014895596541464329,
+          -0.022311072796583176,
+          0.00012365861039143056,
+          0.0075632319785654545,
+          0.01334416400641203,
+          0.008200427517294884,
+          -0.004834373481571674,
+          0.004728174302726984,
+          -0.005868661683052778
+        ],
+        "title": "1 Introduction 3",
+        "keyphrases": [
+          "Introduction"
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "47c02fce-72d4-4324-913d-f36ecb5160fd",
+      "properties": {
+        "page_content": "2 Switch Transformer 4 2.1 Simplifying Sparse Routing 5 2.2 Efficient Sparse Routing 6 2.3 Putting It All Together: The Switch Transformer 8 2.4 Improved Training and Fine-Tuning Techniques 8 ",
+        "embedding": [
+          0.03316810354590416,
+          0.0006708452710881829,
+          -0.017672521993517876,
+          0.002751410473138094,
+          -0.007615511305630207,
+          0.019335461780428886,
+          0.034437984228134155,
+          0.057870324701070786,
+          -0.03555668890476227,
+          0.048315975815057755,
+          -0.011119023896753788,
+          -0.004195145331323147,
+          0.03028063289821148,
+          0.0179899912327528,
+          -0.011209730058908463,
+          0.001036503235809505,
+          0.0006793489446863532,
+          0.0007010805420577526,
+          0.017612051218748093,
+          -0.04595762491226196,
+          0.018035344779491425,
+          0.0009675289620645344,
+          -0.059351854026317596,
+          0.008488554507493973,
+          -0.024520812556147575,
+          -0.01762716844677925,
+          0.012955817393958569,
+          0.012260406278073788,
+          -0.03809645399451256,
+          0.034891512244939804,
+          0.024278929457068443,
+          0.003327770624309778,
+          -0.005680453032255173,
+          -0.009335142560303211,
+          -0.03256339579820633,
+          -0.025382516905665398,
+          0.021603107452392578,
+          0.03576833754777908,
+          -0.049616094678640366,
+          0.04940444603562355,
+          0.019698284566402435,
+          -0.0011952384375035763,
+          -0.008284466341137886,
+          0.006708452943712473,
+          0.023900989443063736,
+          -0.014611199498176575,
+          -0.017581814900040627,
+          -0.004780953750014305,
+          0.02580581232905388,
+          -0.012744170613586903,
+          0.0013350766384974122,
+          -0.07056914269924164,
+          0.026440752670168877,
+          0.009962525218725204,
+          -0.013636111281812191,
+          0.03818715736269951,
+          0.009312466718256474,
+          0.05723538622260094,
+          0.035859040915966034,
+          0.03422633558511734,
+          0.017158521339297295,
+          0.002755190012976527,
+          -0.014467581175267696,
+          -0.03377280756831169,
+          0.01649334467947483,
+          -0.0113684656098485,
+          -0.03395421802997589,
+          0.013137228786945343,
+          -0.006243585143238306,
+          0.004047748167067766,
+          -0.05055338889360428,
+          -0.003365564625710249,
+          -0.05070456489920616,
+          -0.0017942748963832855,
+          -0.017218992114067078,
+          -0.003101005917415023,
+          0.03486127778887749,
+          0.03897327557206154,
+          -0.004614659585058689,
+          0.01395358145236969,
+          0.02783157490193844,
+          -0.01479261089116335,
+          0.005737144500017166,
+          0.00222796224988997,
+          0.030462045222520828,
+          -0.013583199121057987,
+          -0.01650846377015114,
+          -0.05324432998895645,
+          0.05019056424498558,
+          -0.004943468142300844,
+          -0.03510316088795662,
+          -0.027695516124367714,
+          -0.009131054393947124,
+          0.003350446932017803,
+          -0.011973170563578606,
+          0.05324432998895645,
+          0.029010750353336334,
+          -0.014240817166864872,
+          -0.0005191964446566999,
+          0.009592142887413502,
+          0.0290409866720438,
+          0.0069012027233839035,
+          -0.036191631108522415,
+          0.004875438753515482,
+          0.06062173470854759,
+          0.007555040530860424,
+          0.00041620753472670913,
+          0.0489509180188179,
+          0.025775576010346413,
+          0.0024244915693998337,
+          0.04616927355527878,
+          -0.026652399450540543,
+          -0.031898219138383865,
+          0.03431704267859459,
+          -0.004985041916370392,
+          -0.05448397621512413,
+          -0.03126328065991402,
+          -0.04299456998705864,
+          0.014127434231340885,
+          -0.022615989670157433,
+          0.039638452231884,
+          0.045413389801979065,
+          -0.037794098258018494,
+          -0.021119343116879463,
+          -0.07449973374605179,
+          0.006077291443943977,
+          0.033137865364551544,
+          0.005703129805624485,
+          -0.011277759447693825,
+          0.023326518014073372,
+          0.003941924776881933,
+          -0.03126328065991402,
+          0.00961481872946024,
+          -0.03731033578515053,
+          -0.0009084756602533162,
+          -0.022268284112215042,
+          -0.00158546247985214,
+          0.0030953367240726948,
+          -0.015162993222475052,
+          0.0004497497866395861,
+          0.0014427897986024618,
+          -0.017234109342098236,
+          0.02633492834866047,
+          0.012547641061246395,
+          -0.02922239899635315,
+          0.03516362980008125,
+          -0.013749493286013603,
+          0.0394570417702198,
+          0.021724049001932144,
+          0.028058340772986412,
+          -0.05230703577399254,
+          0.004981262609362602,
+          -0.021104225888848305,
+          0.037340570241212845,
+          0.032865747809410095,
+          0.016478227451443672,
+          -0.02264622412621975,
+          -0.0021051312796771526,
+          0.0319586917757988,
+          0.004334983415901661,
+          0.054090917110443115,
+          -0.02650122344493866,
+          -0.002630469389259815,
+          0.029116574674844742,
+          0.024263812229037285,
+          -0.0018538006115704775,
+          0.029796868562698364,
+          0.02043904922902584,
+          -0.042964331805706024,
+          0.01938081532716751,
+          0.025563929229974747,
+          0.05176280066370964,
+          0.04142233356833458,
+          0.02940380945801735,
+          -0.014550728723406792,
+          -0.004678909666836262,
+          -0.02931310422718525,
+          0.013280847109854221,
+          0.007120408583432436,
+          -0.016266580671072006,
+          -0.021391460672020912,
+          -0.023024166002869606,
+          -0.01171617116779089,
+          0.007422761060297489,
+          0.019773872569203377,
+          0.01912381500005722,
+          0.014641433954238892,
+          0.015427551232278347,
+          -0.011610347777605057,
+          0.0028893589042127132,
+          0.0160700511187315,
+          0.01514031644910574,
+          0.017234109342098236,
+          -0.0007648580940440297,
+          -0.011527200229465961,
+          -0.05650973692536354,
+          -0.01764228567481041,
+          0.004017512779682875,
+          0.005332747474312782,
+          -0.013855316676199436,
+          -0.013159905560314655,
+          0.024777812883257866,
+          0.02288810722529888,
+          -0.0030065206810832024,
+          -0.025065047666430473,
+          -0.03123304434120655,
+          -0.0004428996180649847,
+          -0.005578408949077129,
+          -0.009713083505630493,
+          -0.05061385780572891,
+          0.019531991332769394,
+          -0.020559990778565407,
+          0.015767699107527733,
+          0.0018566352082416415,
+          0.014951346442103386,
+          -0.010355583392083645,
+          -0.014656552113592625,
+          -0.004531512502580881,
+          0.009584583342075348,
+          -0.02343234233558178,
+          0.02879910357296467,
+          0.027861811220645905,
+          0.03906398266553879,
+          -0.02482316456735134,
+          -0.02157287299633026,
+          -0.015200787223875523,
+          -0.035919513553380966,
+          -0.02142169512808323,
+          -0.011232406832277775,
+          -0.01685616932809353,
+          -0.0354357473552227,
+          0.008140848949551582,
+          0.02254040166735649,
+          -0.007687320001423359,
+          -0.002987623680382967,
+          -0.05254891514778137,
+          0.0034184763208031654,
+          0.0072526875883340836,
+          -0.005121100693941116,
+          -0.02359863556921482,
+          -0.004996379837393761,
+          0.004758276976644993,
+          -0.004860321059823036,
+          -0.02025763690471649,
+          -0.004229160025715828,
+          0.014316405169665813,
+          -0.036100924015045166,
+          0.04299456998705864,
+          -0.016961991786956787,
+          -0.0025322046130895615,
+          -0.005873203277587891,
+          0.025548812001943588,
+          0.018035344779491425,
+          -0.0014890875900164247,
+          -0.004100659862160683,
+          0.03165633976459503,
+          -0.017869049683213234,
+          -0.04710656777024269,
+          -0.02063557878136635,
+          0.018730755895376205,
+          0.040908332914114,
+          -0.002668263390660286,
+          -0.0011149259516969323,
+          0.02306951954960823,
+          0.0037624025717377663,
+          -0.007793143391609192,
+          0.032775044441223145,
+          0.020454166457057,
+          -0.01676546223461628,
+          1.5036442164273467e-05,
+          0.04003151133656502,
+          -0.022872989997267723,
+          0.005820291582494974,
+          0.030114339664578438,
+          0.026984987780451775,
+          -0.037431277334690094,
+          0.002757079666480422,
+          0.026652399450540543,
+          -0.005812732502818108,
+          -0.014029170386493206,
+          0.02588140033185482,
+          0.006784040946513414,
+          -0.030507396906614304,
+          -0.00842052511870861,
+          0.016720110550522804,
+          -0.03002363257110119,
+          -0.033833280205726624,
+          0.057265620678663254,
+          -0.02560928277671337,
+          -0.03815692290663719,
+          -0.008435643278062344,
+          -0.014097199775278568,
+          -0.02069604955613613,
+          -0.010075907222926617,
+          0.05366762354969978,
+          -0.025639517232775688,
+          0.02359863556921482,
+          -0.04245033487677574,
+          0.021829871460795403,
+          -0.01729458011686802,
+          -0.008722878061234951,
+          0.010620142333209515,
+          0.005688012111932039,
+          -0.03712892532348633,
+          -0.05929138511419296,
+          0.033047161996364594,
+          0.029328221455216408,
+          0.045141272246837616,
+          -0.02385563589632511,
+          -0.001808447646908462,
+          0.0442039780318737,
+          0.02317534200847149,
+          0.010068348608911037,
+          -0.0412711575627327,
+          -0.004750718362629414,
+          -0.011360906064510345,
+          0.022570636123418808,
+          -0.013719257898628712,
+          -0.03220057487487793,
+          -0.008556583896279335,
+          -0.04934397712349892,
+          -0.011958053335547447,
+          -0.029267750680446625,
+          0.06845267117023468,
+          0.008866495452821255,
+          0.01175396516919136,
+          0.024006811901926994,
+          0.02783157490193844,
+          -0.03186798468232155,
+          0.0007024978403933346,
+          -0.008972319774329662,
+          -0.009440965950489044,
+          0.008186202496290207,
+          -0.00898743700236082,
+          -0.0029214839451014996,
+          -0.02589651755988598,
+          -0.016130521893501282,
+          0.003979718778282404,
+          -0.036463748663663864,
+          -0.018231874331831932,
+          -0.009365377947688103,
+          0.005846747197210789,
+          0.016387522220611572,
+          -0.016100287437438965,
+          -0.03537527844309807,
+          -0.03186798468232155,
+          -0.018821461126208305,
+          -0.012010964564979076,
+          -0.009289789944887161,
+          0.003407138166949153,
+          0.0008442256948910654,
+          0.023039283230900764,
+          0.023825401440262794,
+          0.002604013541713357,
+          0.028118809685111046,
+          -0.03867092356085777,
+          -0.01650846377015114,
+          0.01570722833275795,
+          0.015034493058919907,
+          -0.04284339025616646,
+          0.0247475765645504,
+          -0.03344022110104561,
+          -0.03939656913280487,
+          -0.021996166557073593,
+          -0.053123388439416885,
+          -0.04021292179822922,
+          0.0015656206523999572,
+          -0.04737868532538414,
+          -0.026607045903801918,
+          0.02940380945801735,
+          -0.0289502814412117,
+          0.005827850196510553,
+          0.010468965396285057,
+          -0.014150111004710197,
+          -0.010650377720594406,
+          -0.019017990678548813,
+          0.05212562158703804,
+          -0.011285318061709404,
+          -0.001033668639138341,
+          0.02763504534959793,
+          -0.017702756449580193,
+          0.005162674002349377,
+          -0.005812732502818108,
+          -0.004282071720808744,
+          0.022071754559874535,
+          0.024278929457068443,
+          0.04344809800386429,
+          -0.007097731810063124,
+          -0.08961737155914307,
+          0.016735227778553963,
+          0.040575746446847916,
+          -0.014807728119194508,
+          0.00541967386379838,
+          -0.023659106343984604,
+          0.05626785755157471,
+          0.011345788836479187,
+          -0.013537846505641937,
+          -0.0009372936910949647,
+          0.012668582610785961,
+          -0.005098423920571804,
+          -0.028315339237451553,
+          -0.016614286229014397,
+          -0.042329393327236176,
+          -0.006855849642306566,
+          -0.009055466391146183,
+          0.018095815554261208,
+          -0.02186010777950287,
+          0.022253166884183884,
+          0.007914084941148758,
+          0.008934524841606617,
+          0.013515169732272625,
+          -0.026108164340257645,
+          0.03247269243001938,
+          -0.04178515821695328,
+          0.00832981988787651,
+          0.012222612276673317,
+          -0.03924539312720299,
+          0.016432873904705048,
+          -0.02195081301033497,
+          -0.030144574120640755,
+          -0.011648141779005527,
+          0.030205044895410538,
+          0.024324283003807068,
+          -0.014452463947236538,
+          0.008148407563567162,
+          0.008284466341137886,
+          0.0229183416813612,
+          -0.03464962914586067,
+          0.003811534959822893,
+          0.030235279351472855,
+          0.03903374820947647,
+          0.0062398058362305164,
+          0.00339957932010293,
+          -0.004554189275950193,
+          -0.04450633376836777,
+          0.0027608589734882116,
+          -0.029978280887007713,
+          -0.008269349113106728,
+          -0.035224102437496185,
+          0.024157987907528877,
+          0.04580644890666008,
+          0.023281166329979897,
+          0.014974022284150124,
+          0.00031628936994820833,
+          -0.028738632798194885,
+          0.0008456429932266474,
+          -0.029857339337468147,
+          0.006198232527822256,
+          0.031112102791666985,
+          -0.0010667385067790747,
+          -0.028738632798194885,
+          -0.00025251181796193123,
+          -0.047167036682367325,
+          -0.03933610022068024,
+          0.017324814572930336,
+          -0.02780134044587612,
+          0.021527519449591637,
+          0.00541589455679059,
+          0.003002741141244769,
+          -0.022268284112215042,
+          0.005910997278988361,
+          0.01904822699725628,
+          0.02089257910847664,
+          -0.054000210016965866,
+          0.024067282676696777,
+          -0.024233577772974968,
+          -0.05309315025806427,
+          0.0020503299310803413,
+          -0.027332693338394165,
+          0.003017858834937215,
+          -0.0003706183924805373,
+          -0.026002340018749237,
+          0.00039353108149953187,
+          0.019169166684150696,
+          -0.014611199498176575,
+          0.033651866018772125,
+          -0.01649334467947483,
+          -0.09403172135353088,
+          0.0020408814307302237,
+          -0.025034811347723007,
+          -0.011338230222463608,
+          0.03885233402252197,
+          -0.0027627486269921064,
+          0.0054612476378679276,
+          -0.007226231973618269,
+          0.012781964614987373,
+          0.017612051218748093,
+          0.029615456238389015,
+          0.00965261273086071,
+          0.00511354161426425,
+          0.030068986117839813,
+          0.005665335804224014,
+          9.962288459064439e-05,
+          -0.0040061743929982185,
+          -0.020393695682287216,
+          -0.05152091756463051,
+          -0.022872989997267723,
+          -0.05396997556090355,
+          0.020030872896313667,
+          0.0071733202785253525,
+          -0.030129456892609596,
+          -0.0049888212233781815,
+          0.04870903491973877,
+          0.020484402775764465,
+          -0.02176940254867077,
+          -0.007056158501654863,
+          0.019108697772026062,
+          0.024157987907528877,
+          0.005593526642769575,
+          -0.001357753062620759,
+          0.010620142333209515,
+          0.0015826279995962977,
+          0.011897582560777664,
+          -0.02195081301033497,
+          -0.03090045601129532,
+          0.006889864336699247,
+          -0.028390927240252495,
+          0.022782282903790474,
+          0.006470350082963705,
+          -0.0012490949593484402,
+          0.00991717167198658,
+          0.006421217694878578,
+          -0.0027117265854030848,
+          0.023976577445864677,
+          -0.01087714172899723,
+          0.015004257671535015,
+          0.037854570895433426,
+          0.014467581175267696,
+          0.05294197425246239,
+          -0.0076684230007231236,
+          -0.010854465886950493,
+          0.008397849276661873,
+          -0.0028251090552657843,
+          -0.017778344452381134,
+          -0.00031487210071645677,
+          -0.007675981614738703,
+          0.0089949956163764,
+          0.013515169732272625,
+          -0.0398198626935482,
+          -0.011958053335547447,
+          -0.002184499055147171,
+          -0.01649334467947483,
+          -0.0004769143124576658,
+          -0.0031085647642612457,
+          8.887518924893811e-05,
+          0.03906398266553879,
+          -0.0022090652491897345,
+          -0.016977109014987946,
+          -0.00726402597501874,
+          0.02000063844025135,
+          -0.05245821177959442,
+          -0.00741520244628191,
+          0.003911689389497042,
+          0.03359139710664749,
+          -0.011217288672924042,
+          0.005767379887402058,
+          -0.028904927894473076,
+          0.01694687455892563,
+          -0.01394602283835411,
+          -0.026304693892598152,
+          -0.0039494833908975124,
+          0.0024887414183467627,
+          0.01416522916406393,
+          0.01518566906452179,
+          -0.01079399511218071,
+          0.009849142283201218,
+          0.017869049683213234,
+          0.03331927955150604,
+          -0.021300755441188812,
+          0.011255082674324512,
+          -0.004013733472675085,
+          0.03099116124212742,
+          -0.007093952503055334,
+          -0.00710907019674778,
+          0.017445756122469902,
+          0.015253698453307152,
+          -0.00209379312582314,
+          0.010998083278536797,
+          -0.013689023442566395,
+          -0.0068785264156758785,
+          0.01684105023741722,
+          -0.03168657422065735,
+          0.015321727842092514,
+          0.01139870099723339,
+          0.005332747474312782,
+          -0.009191525168716908,
+          -0.02297881245613098,
+          -0.020998401567339897,
+          0.003896571695804596,
+          -0.016357285901904106,
+          0.03368210047483444,
+          0.013809964060783386,
+          0.007354731671512127,
+          -0.020136697217822075,
+          -0.0016043595969676971,
+          0.011580112390220165,
+          0.009433407336473465,
+          0.005030394531786442,
+          0.03537527844309807,
+          -0.0637359693646431,
+          0.02684892900288105,
+          0.011232406832277775,
+          -0.004818747751414776,
+          -0.01417278777807951,
+          0.0011470509925857186,
+          -0.004542850889265537,
+          0.0102270832285285,
+          0.01056723017245531,
+          0.02368934266269207,
+          0.0020087563898414373,
+          0.014633875340223312,
+          -0.03168657422065735,
+          0.035586923360824585,
+          -0.009131054393947124,
+          0.0034241455141454935,
+          -0.0006117919692769647,
+          -0.012003405950963497,
+          0.00869264267385006,
+          -0.009645054116845131,
+          0.004576865583658218,
+          0.08411455154418945,
+          0.013568081893026829,
+          0.0101363779976964,
+          0.005291174165904522,
+          -0.027967633679509163,
+          0.03546598181128502,
+          -0.014860640279948711,
+          -0.021376343443989754,
+          0.0014824735699221492,
+          0.0035658732522279024,
+          0.009622378274798393,
+          -0.02595698833465576,
+          -0.007921643555164337,
+          -0.03897327557206154,
+          -0.010778876952826977,
+          -0.018307462334632874,
+          0.010559671558439732,
+          -0.0035450865980237722,
+          0.036826569586992264,
+          -0.01948663778603077,
+          -0.01913893222808838,
+          0.03437751531600952,
+          0.0582633838057518,
+          0.03386351466178894,
+          -0.00868508405983448,
+          0.008707760833203793,
+          0.004512615501880646,
+          -0.013106994330883026,
+          0.02124028466641903,
+          0.015813050791621208,
+          0.008155967108905315,
+          0.026879163458943367,
+          -0.010091024450957775,
+          0.03625210002064705,
+          0.023900989443063736,
+          -0.017082933336496353,
+          -0.02692451700568199,
+          -0.0052382624708116055,
+          -0.02604769356548786,
+          -0.0040666451677680016,
+          -0.00307266041636467,
+          0.006965452805161476,
+          0.005820291582494974,
+          1.740595507726539e-05,
+          0.01718875765800476,
+          -0.023326518014073372,
+          -0.018035344779491425,
+          0.007400084752589464,
+          -0.022752048447728157,
+          0.01746087335050106,
+          0.0035791012924164534,
+          -0.03138422220945358,
+          -0.009153731167316437,
+          0.028300222009420395,
+          0.000411955697927624,
+          -0.030386457219719887,
+          -0.006039496976882219,
+          0.005521717946976423,
+          -0.028103692457079887,
+          0.0013946022372692823,
+          -0.025911634787917137,
+          -0.0071959965862333775,
+          -0.004474821500480175,
+          0.008881613612174988,
+          0.009395613335072994,
+          -0.00038762573967687786,
+          -0.02394634112715721,
+          -0.015200787223875523,
+          -0.029721280559897423,
+          0.027075693011283875,
+          -0.02631981112062931,
+          0.015079845674335957,
+          -0.02553369477391243,
+          0.02246481366455555,
+          -0.01737016811966896,
+          -0.007449217140674591,
+          -0.004365218337625265,
+          0.030855102464556694,
+          0.061468325555324554,
+          -0.030598104000091553,
+          0.0010865803342312574,
+          0.010098583996295929,
+          0.003289976390078664,
+          -0.017869049683213234,
+          0.03649398311972618,
+          0.02589651755988598,
+          -3.9772385207470506e-05,
+          -0.0032918660435825586,
+          -0.013016288168728352,
+          -0.041906099766492844,
+          -0.0024263812229037285,
+          0.011648141779005527,
+          -0.0197285208851099,
+          0.02210198901593685,
+          0.031081868335604668,
+          0.01684105023741722,
+          -0.03044692613184452,
+          -0.023659106343984604,
+          -0.03625210002064705,
+          0.032593630254268646,
+          -0.013122111558914185,
+          0.043478332459926605,
+          0.014807728119194508,
+          -0.025140635669231415,
+          -0.010899818502366543,
+          -0.008722878061234951,
+          0.014399551786482334,
+          -0.02263110689818859,
+          0.027151281014084816,
+          -0.012177258729934692,
+          0.02192057855427265,
+          -0.007033482193946838,
+          -0.003972159698605537,
+          0.011617906391620636,
+          -0.007097731810063124,
+          -0.02043904922902584,
+          -0.0012557089794427156,
+          0.005975247360765934,
+          0.017581814900040627,
+          -0.0066064088605344296,
+          0.034619394689798355,
+          0.03062833845615387,
+          -0.0040326304733753204,
+          0.023356754332780838,
+          0.02774086967110634,
+          0.00572202680632472,
+          0.01966805011034012,
+          0.011557435616850853,
+          -0.001473969896323979,
+          -0.021542636677622795,
+          0.001912381500005722,
+          -0.032865747809410095,
+          0.010544554330408573,
+          0.005465026944875717,
+          -0.02279740199446678,
+          0.021542636677622795,
+          0.01702246256172657,
+          -0.019184285774827003,
+          -0.00036636655568145216,
+          -0.024898752570152283,
+          0.0007379298331215978,
+          0.02668263390660286,
+          -0.025669753551483154,
+          0.027362927794456482,
+          0.020650696009397507,
+          0.010363142006099224,
+          -0.024535929784178734,
+          -0.005619982723146677,
+          0.0035299689043313265,
+          -0.007698658388108015,
+          0.024157987907528877,
+          0.027952516451478004,
+          0.03174704313278198,
+          -0.016629403457045555,
+          0.0036811453755944967,
+          0.013061640784144402,
+          0.0036282336805015802,
+          -0.019320344552397728,
+          -0.0264861062169075,
+          0.013832640834152699,
+          0.005907217971980572,
+          0.0009306797292083502,
+          0.020862342789769173,
+          0.003520520403981209,
+          -0.004195145331323147,
+          -0.0012292531318962574,
+          -0.007127967197448015,
+          -0.029948044568300247,
+          -0.0075248051434755325,
+          0.02545810490846634,
+          0.012139464728534222,
+          0.03017481043934822,
+          -0.026380281895399094,
+          0.011239965446293354,
+          0.042510803788900375,
+          -0.018987756222486496,
+          -0.009380496107041836,
+          0.017869049683213234,
+          0.003953262697905302,
+          -0.015321727842092514,
+          -0.014308846555650234,
+          -0.009818906895816326,
+          -0.004414350725710392,
+          0.03310763090848923,
+          -0.00987937767058611,
+          -0.046108800917863846,
+          -0.004021292086690664,
+          -0.033289045095443726,
+          -0.006746246945112944,
+          7.907234248705208e-05,
+          0.023659106343984604,
+          -0.02052975445985794,
+          0.007929202169179916,
+          0.025473223999142647,
+          -0.005427232943475246,
+          0.0356171615421772,
+          0.012388906441628933,
+          -0.02580581232905388,
+          0.011821994557976723,
+          -0.025155752897262573,
+          0.01417278777807951,
+          -0.006825614254921675,
+          0.005313850473612547,
+          -0.013099434785544872,
+          -0.04411327466368675,
+          -0.05170232802629471,
+          0.0030896677635610104,
+          -0.02518598921597004,
+          0.04142233356833458,
+          0.01938081532716751,
+          -0.02105887234210968,
+          0.0025643296539783478,
+          -0.017355050891637802,
+          0.017324814572930336,
+          0.04680421203374863,
+          0.011678377166390419,
+          -0.009667730890214443,
+          0.01966805011034012,
+          -0.014059404842555523,
+          0.02376493066549301,
+          0.020121578127145767,
+          0.003779409918934107,
+          -0.03477057069540024,
+          -0.007793143391609192,
+          -0.06434067338705063,
+          0.0033806823194026947,
+          0.013076758943498135,
+          0.008246672339737415,
+          0.03809645399451256,
+          -0.0009150896221399307,
+          0.03099116124212742,
+          0.02692451700568199,
+          0.024324283003807068,
+          0.001319014118053019,
+          -0.03368210047483444,
+          -0.0024055945686995983,
+          0.04641115665435791,
+          0.038640689104795456,
+          -0.012842435389757156,
+          -0.014648993499577045,
+          -0.016735227778553963,
+          -0.005170233082026243,
+          0.026470987126231194,
+          -0.018972638994455338,
+          0.011648141779005527,
+          -0.008019908331334591,
+          -0.004353880416601896,
+          -0.010657936334609985,
+          0.007154423277825117,
+          -0.020862342789769173,
+          -0.03885233402252197,
+          -0.0290409866720438,
+          -0.03229127824306488,
+          0.02615351788699627,
+          0.02677334100008011,
+          -0.042601510882377625,
+          -0.008239113725721836,
+          -0.04226892068982124,
+          0.010771318338811398,
+          0.021451931446790695,
+          0.0014409000286832452,
+          0.01665963977575302,
+          -0.013167464174330235,
+          0.03791503980755806,
+          0.02352304756641388,
+          0.0016922308132052422,
+          -0.005986585281789303,
+          -0.007543702609837055,
+          -0.00833737850189209,
+          -0.007936760783195496,
+          0.00445592449977994,
+          0.03386351466178894,
+          -0.01888193190097809,
+          -0.008193761110305786,
+          0.004743159748613834,
+          0.0003840825520455837,
+          -0.0012216942850500345,
+          0.0161607563495636,
+          0.011965611949563026,
+          -0.004784733057022095,
+          -0.0023848076816648245,
+          -0.01364366989582777,
+          -0.004055306781083345,
+          -0.005533056333661079,
+          0.007071276195347309,
+          -0.009743318893015385,
+          -0.006666879169642925,
+          0.031172573566436768,
+          0.0013246831949800253,
+          -0.026289576664566994,
+          -0.014815287664532661,
+          0.0032351750414818525,
+          -0.0013624773127958179,
+          -0.009456084109842777,
+          0.03380304202437401,
+          0.052609387785196304,
+          -0.0049548065289855,
+          -0.023659106343984604,
+          0.010121259838342667,
+          -0.001774432952515781,
+          0.002562440000474453,
+          0.0005012442707084119,
+          0.01876099221408367,
+          0.02668263390660286,
+          -0.0038927923887968063,
+          -0.004395453725010157,
+          -0.01913893222808838,
+          0.008624613285064697,
+          0.022600872442126274,
+          0.025140635669231415,
+          -0.022041520103812218,
+          0.014921111054718494,
+          0.0019596240017563105,
+          -0.0197285208851099,
+          0.015767699107527733,
+          0.0160700511187315,
+          -0.007233790587633848,
+          0.012509847059845924,
+          -0.005174012389034033,
+          0.00825423188507557,
+          -0.010431171394884586,
+          0.010990524664521217,
+          0.01975875534117222,
+          0.008299584500491619,
+          -0.010242201387882233,
+          -0.017687639221549034,
+          0.007517246529459953,
+          0.005623762030154467,
+          -0.0018358484376221895,
+          0.017173638567328453,
+          -0.005706909112632275,
+          0.004890556447207928,
+          -0.005427232943475246,
+          -0.024551047012209892,
+          -0.004176247864961624,
+          -0.004229160025715828,
+          -0.010809112340211868,
+          0.009758437052369118,
+          -0.026365164667367935,
+          0.016901521012187004,
+          0.023235812783241272,
+          -0.0057787178084254265,
+          0.0007317882846109569,
+          -0.0089042903855443,
+          0.001393657410517335,
+          -0.017475992441177368,
+          0.017339933663606644,
+          -0.029539868235588074,
+          -0.02615351788699627,
+          0.04311550781130791,
+          -0.025684870779514313,
+          -0.026395399123430252,
+          0.0006264371913857758,
+          -0.01824699155986309,
+          -0.0022525284439325333,
+          0.009675289504230022,
+          0.006859628949314356,
+          -0.015419992618262768,
+          -0.017143404111266136,
+          0.015412434004247189,
+          0.013689023442566395,
+          -0.03238198533654213,
+          0.022963695228099823,
+          -0.014958905056118965,
+          -0.019169166684150696,
+          -0.00019806470663752407,
+          -0.0031936015002429485,
+          -0.03570786491036415,
+          -0.02562440000474453,
+          -0.006088629364967346,
+          -0.005041732918471098,
+          -0.0012708266731351614,
+          0.032865747809410095,
+          0.0005503766005858779,
+          -0.0008706816006451845,
+          -0.01605493389070034,
+          -0.009592142887413502,
+          -0.04411327466368675,
+          0.006436335388571024,
+          0.01650846377015114,
+          -0.021739166229963303,
+          -0.01334131695330143,
+          -0.0006311614997684956,
+          0.059442561119794846,
+          -0.02001575566828251,
+          0.005139997694641352,
+          0.01139870099723339,
+          -0.018715638667345047,
+          -0.030764397233724594,
+          0.014679228886961937,
+          0.03289598599076271,
+          0.0009817017707973719,
+          -0.02201128378510475,
+          -0.014323963783681393,
+          -0.011376024223864079,
+          -0.0024396092630922794,
+          0.017778344452381134,
+          0.0068180556409060955,
+          -0.022510165348649025,
+          0.025140635669231415,
+          -0.002588895848020911,
+          0.0007237570243887603,
+          -0.03192845731973648,
+          -4.650445771403611e-05,
+          0.010151495225727558,
+          0.02299392968416214,
+          -0.013099434785544872,
+          0.0008319426560774446,
+          -0.005676673725247383,
+          -0.002031432930380106,
+          -0.00829202588647604,
+          -0.0070901731960475445,
+          -0.026516340672969818,
+          -0.024263812229037285,
+          0.026108164340257645,
+          -0.005593526642769575,
+          0.012555199675261974,
+          -0.03271457180380821,
+          -0.01483040489256382,
+          0.032261043787002563,
+          0.0014616867993026972,
+          -0.025231340900063515,
+          0.019365696236491203,
+          -0.025503458455204964,
+          -0.00900255423039198,
+          0.006005482282489538,
+          0.0051551153883337975,
+          0.006859628949314356,
+          0.019259873777627945,
+          0.015949109569191933,
+          0.023190459236502647,
+          4.854179860558361e-05,
+          0.015933992341160774,
+          -0.022328754886984825,
+          -0.018201638013124466,
+          0.02798275090754032,
+          -0.0021278078202158213,
+          0.008564143441617489,
+          0.02808857522904873,
+          0.0037246085703372955,
+          -0.015737462788820267,
+          8.344228990608826e-05,
+          -0.004561747889965773,
+          0.010302671231329441,
+          0.0010998082580044866,
+          0.014958905056118965,
+          -0.005994144361466169,
+          -0.0002851092431228608,
+          0.015858404338359833,
+          0.01452805195003748,
+          0.007921643555164337,
+          -0.011572553776204586,
+          -0.01894240267574787,
+          0.0021693813614547253,
+          -0.0005267552915029228,
+          0.04281315580010414,
+          0.007222452666610479,
+          0.004985041916370392,
+          -0.05433279648423195,
+          0.004678909666836262,
+          -0.01842840388417244,
+          0.004788512364029884,
+          0.024460341781377792,
+          0.002974395640194416,
+          -0.00863973144441843,
+          0.011224847286939621,
+          -0.0015457787085324526,
+          -0.0029233735986053944,
+          0.006254923529922962,
+          -0.043024804443120956,
+          -0.02246481366455555,
+          0.01623634621500969,
+          0.008896730840206146,
+          0.022767165675759315,
+          0.016478227451443672,
+          -0.004969924222677946,
+          0.00691254111006856,
+          0.05956350266933441,
+          0.0008007625001482666,
+          -0.009425848722457886,
+          -0.027060575783252716,
+          0.00016924670489970595,
+          -0.006988129112869501,
+          -0.03422633558511734,
+          0.03906398266553879,
+          0.008435643278062344,
+          0.02001575566828251,
+          0.009690407663583755,
+          0.02439987100660801,
+          -0.011852229945361614,
+          0.011746406555175781,
+          0.014580964110791683,
+          -0.009856700897216797,
+          -0.006825614254921675,
+          -0.0015542823821306229,
+          -0.015004257671535015,
+          0.010990524664521217,
+          -0.0031180132646113634,
+          0.014399551786482334,
+          -0.0074454378336668015,
+          0.03703821823000908,
+          -0.0315958671271801,
+          0.024944106116890907,
+          0.02482316456735134,
+          0.007449217140674591,
+          -0.02421845868229866,
+          -0.007407643832266331,
+          0.012827317230403423,
+          0.017657402902841568,
+          0.021346107125282288,
+          0.025034811347723007,
+          0.015004257671535015,
+          0.03721962869167328,
+          -0.03344022110104561,
+          0.015676992014050484,
+          0.009191525168716908,
+          0.028920045122504234,
+          0.013106994330883026,
+          0.014301287941634655,
+          0.02052975445985794,
+          -0.006621526554226875,
+          -0.021618224680423737,
+          -0.0015608962858095765,
+          0.006595070473849773,
+          -0.00929734855890274,
+          -0.01545022800564766,
+          -0.01762716844677925,
+          -0.0020068667363375425,
+          0.00934270117431879,
+          -0.029721280559897423,
+          -0.020590225234627724,
+          -0.022842753678560257,
+          0.0077629080042243,
+          0.01675034500658512,
+          -0.026607045903801918,
+          0.010159053839743137,
+          0.011557435616850853,
+          0.016206109896302223,
+          0.009796231053769588,
+          0.009040349163115025,
+          0.004285851027816534,
+          0.01588863879442215,
+          -0.013673905283212662,
+          0.019637813791632652,
+          0.010476524941623211,
+          0.011625465005636215,
+          -0.012169700115919113,
+          -0.035405512899160385,
+          0.007464334834367037,
+          0.01703757978975773,
+          0.0003247930435463786,
+          -0.018549343571066856,
+          -0.0040061743929982185,
+          -0.0014862529933452606,
+          -0.0291316919028759,
+          -0.0013294074451550841,
+          -0.0020068667363375425,
+          0.01329596433788538,
+          -0.008760672062635422,
+          -0.0009996539447456598,
+          0.004656233359128237,
+          -0.020212285220623016,
+          -0.0032276161946356297,
+          -0.010559671558439732,
+          0.010385818779468536,
+          0.001661050715483725,
+          -0.005782497115433216,
+          0.010506759397685528,
+          -0.018927285447716713,
+          -0.01694687455892563,
+          0.023039283230900764,
+          0.006375864613801241,
+          -0.019320344552397728,
+          0.01167081855237484,
+          -0.016568932682275772,
+          -0.005563291721045971,
+          -0.005971467588096857,
+          -0.03168657422065735,
+          0.004690248053520918,
+          0.018398167565464973,
+          -0.005638879723846912,
+          0.011633023619651794,
+          0.009577024728059769,
+          0.018473755568265915,
+          -0.013507611118257046,
+          0.023039283230900764,
+          0.004236718639731407,
+          0.004391674418002367,
+          -0.005087085999548435,
+          -0.008662407286465168,
+          0.005162674002349377,
+          -0.0009509940282441676,
+          -1.5781988622620702e-05,
+          0.0223741065710783,
+          -0.0438411571085453,
+          0.005929894279688597,
+          0.006270041223615408,
+          0.009992759674787521,
+          0.022661343216896057,
+          0.010249760001897812,
+          -0.022600872442126274,
+          0.0049283504486083984,
+          0.005899658892303705,
+          0.006942776031792164,
+          -4.81284259876702e-05,
+          0.0014049956807866693,
+          -0.00842052511870861,
+          -0.011746406555175781,
+          0.016614286229014397,
+          0.023099754005670547,
+          -0.01148940622806549,
+          0.014853081665933132,
+          -0.024883635342121124,
+          -0.0031917118467390537,
+          0.013129670172929764,
+          0.03676610067486763,
+          -0.035859040915966034,
+          -0.010605024173855782,
+          0.020302990451455116,
+          0.005064409226179123,
+          0.011942935176193714,
+          -0.03694751113653183,
+          -0.02385563589632511,
+          -0.02193569578230381,
+          0.01788416877388954,
+          0.0003573904687073082,
+          -0.0022487489040941,
+          -0.01170861255377531,
+          -0.02010646089911461,
+          -0.010091024450957775,
+          0.012283082120120525,
+          0.00025794474640861154,
+          -0.011897582560777664,
+          -0.001873642555437982,
+          0.006069732364267111,
+          0.03555668890476227,
+          -0.011995847336947918,
+          0.02113446034491062,
+          -0.008322261273860931,
+          -0.019456403329968452,
+          -0.020514637231826782,
+          -0.010506759397685528,
+          0.0010327238123863935,
+          0.005037953611463308,
+          0.012388906441628933,
+          0.010234642773866653,
+          0.005480144638568163,
+          -0.007241349667310715,
+          0.004357659723609686,
+          -0.0060735116712749004,
+          0.0019426167709752917,
+          -0.008246672339737415,
+          -0.025911634787917137,
+          0.02492898888885975,
+          0.00829202588647604,
+          0.008586819283664227,
+          -0.006065953057259321,
+          0.0032295058481395245,
+          -0.022147342562675476,
+          0.004406792111694813,
+          0.018564462661743164,
+          -0.003560204291716218,
+          0.002891248557716608,
+          0.013424464501440525,
+          -0.02343234233558178,
+          -0.008042584173381329,
+          0.006504364777356386,
+          0.00034912300179712474,
+          0.010703288950026035,
+          0.014271052554249763,
+          -0.017959756776690483,
+          0.006077291443943977,
+          0.010552112944424152,
+          -0.009577024728059769,
+          -0.005714467726647854,
+          0.009176407940685749,
+          0.006844511721283197,
+          -0.03658469021320343,
+          0.023538164794445038,
+          -0.0037321674171835184,
+          -0.0027325134724378586,
+          0.02774086967110634,
+          -0.019879696890711784,
+          -0.019017990678548813,
+          -0.005166453309357166,
+          0.015918875113129616,
+          -0.0013719258131459355,
+          0.01048408355563879,
+          -0.0034865057095885277,
+          0.003429814474657178,
+          -0.0005040788091719151,
+          0.004818747751414776,
+          -0.01057478878647089,
+          0.00075399229535833,
+          -0.008140848949551582,
+          0.001432396355085075,
+          0.009289789944887161,
+          -0.007604172918945551,
+          -0.008186202496290207,
+          0.003972159698605537,
+          -0.0011952384375035763,
+          0.003414697013795376,
+          -0.007014585193246603,
+          0.018972638994455338,
+          -0.03126328065991402,
+          -0.0003994363942183554,
+          0.00568423280492425,
+          -0.009289789944887161,
+          0.023296283558011055,
+          0.011013200506567955,
+          0.010385818779468536,
+          0.005117321386933327,
+          0.012131906114518642,
+          -0.013258170336484909,
+          0.0008073764620348811,
+          -0.006304055918008089,
+          -0.009335142560303211,
+          0.004686468280851841,
+          0.026213988661766052,
+          0.016100287437438965,
+          0.026093047112226486,
+          0.018201638013124466,
+          -0.0002548739721532911,
+          0.004002395085990429,
+          0.004731821361929178,
+          -0.0054234531708061695,
+          0.01720387488603592,
+          -0.019078461453318596,
+          0.0028496752493083477,
+          0.007868731394410133,
+          -0.023825401440262794,
+          -0.003443042514845729,
+          -0.003862557001411915,
+          0.031535398215055466,
+          0.01851910911500454,
+          0.015102522447705269,
+          0.010763759724795818,
+          0.006738687865436077,
+          -0.0160700511187315,
+          0.03044692613184452,
+          -0.007861172780394554,
+          0.016463110223412514,
+          0.010884701274335384,
+          -0.007177099585533142,
+          0.0030839985702186823,
+          0.010703288950026035,
+          0.002562440000474453,
+          -0.022479930892586708,
+          0.020318107679486275,
+          -0.018624933436512947,
+          0.0076684230007231236,
+          0.009501436725258827,
+          -0.008866495452821255,
+          0.023296283558011055,
+          -0.01079399511218071,
+          -0.00863973144441843,
+          -0.005389438476413488,
+          -0.005272277165204287,
+          0.05578409135341644,
+          0.002758969319984317,
+          -0.019320344552397728,
+          -0.024611517786979675,
+          0.024551047012209892,
+          0.0039834980852901936,
+          -0.0061906734481453896,
+          0.021965930238366127,
+          0.012010964564979076,
+          -0.03480080887675285,
+          0.015586286783218384,
+          0.004225380253046751,
+          -0.008012348785996437,
+          0.020151814445853233,
+          0.0031028955709189177,
+          -0.006141541060060263,
+          0.030401574447751045,
+          -0.003520520403981209,
+          -0.001451293472200632,
+          0.008737996220588684,
+          -0.001432396355085075,
+          -0.017869049683213234,
+          0.05327456444501877,
+          0.001982300542294979,
+          0.025216223672032356,
+          -0.0056124236434698105,
+          -0.009100819006562233,
+          -0.017566697672009468,
+          -0.011028318665921688,
+          0.024233577772974968,
+          -0.012184818275272846,
+          -0.015110081061720848,
+          0.0011602789163589478,
+          0.013696582056581974,
+          0.0032654101960361004,
+          -0.0013568082358688116,
+          -0.006243585143238306,
+          0.03431704267859459,
+          -0.005136218387633562,
+          -0.018095815554261208,
+          -0.016085168346762657,
+          -0.0024887414183467627,
+          0.020998401567339897,
+          0.019017990678548813,
+          0.015737462788820267,
+          -0.02069604955613613,
+          -0.004890556447207928,
+          -0.013311082497239113,
+          -0.03362163156270981,
+          0.010620142333209515,
+          0.0004606155853252858,
+          0.023825401440262794,
+          -0.02579069323837757,
+          0.006825614254921675,
+          0.003658468835055828,
+          -0.04589715600013733,
+          0.019350579008460045,
+          0.014467581175267696,
+          0.00031416345154866576,
+          -0.014762375503778458,
+          0.0025529915001243353,
+          -0.0028950280975550413,
+          0.006266261916607618,
+          -0.012600553222000599,
+          -0.017869049683213234,
+          0.0024660651106387377,
+          0.01056723017245531,
+          0.023129988461732864,
+          0.001661050715483725,
+          -0.021557753905653954,
+          -0.0009731980971992016,
+          0.00042234905413351953,
+          -0.013159905560314655,
+          0.0009486319031566381,
+          0.02124028466641903,
+          0.02255551889538765,
+          0.02211710810661316,
+          -0.004274512641131878,
+          0.005703129805624485,
+          0.013144788332283497,
+          -0.007573937531560659,
+          -0.026531457901000977,
+          -0.008239113725721836,
+          -0.024112636223435402,
+          0.021633343771100044,
+          0.004414350725710392,
+          -0.00011361851647961885,
+          0.026440752670168877,
+          -0.023900989443063736,
+          0.012086553499102592,
+          0.009629936888813972,
+          -0.001953955041244626,
+          0.010053230449557304,
+          -0.00024920483701862395,
+          0.019244756549596786,
+          -0.025488341227173805,
+          0.012849994003772736,
+          -0.002855344209820032,
+          0.0067349085584282875,
+          0.015933992341160774,
+          -0.011942935176193714,
+          0.009486319497227669,
+          0.00047148141311481595,
+          -0.00418758625164628,
+          -0.011678377166390419,
+          -0.000992095097899437,
+          0.003769961418583989,
+          0.03079463355243206,
+          -0.011307994835078716,
+          0.004130895249545574,
+          0.015737462788820267,
+          0.009886936284601688,
+          0.014150111004710197,
+          0.01782369799911976,
+          0.0016667197924107313,
+          0.019002873450517654,
+          -0.014633875340223312,
+          0.006591291166841984,
+          0.009516553953289986,
+          -0.01522346306592226,
+          -0.006829394027590752,
+          -0.026289576664566994,
+          -0.0005437626386992633,
+          -0.003998615778982639,
+          -0.011610347777605057,
+          -0.009047907777130604,
+          -0.028496751561760902,
+          -0.007800702471286058,
+          -0.026213988661766052,
+          -0.007313158363103867,
+          -0.008019908331334591,
+          0.000944380066357553,
+          -0.000724229437764734,
+          0.01782369799911976,
+          -0.004678909666836262,
+          -0.004875438753515482,
+          0.021542636677622795,
+          -0.011421376839280128,
+          -0.00038644467713311315,
+          -0.012041199952363968,
+          0.0013596427161246538,
+          -0.0072526875883340836,
+          0.011549877002835274,
+          0.005752262193709612,
+          -0.00023007158597465605,
+          -0.002220403403043747,
+          0.0080803781747818,
+          -0.005465026944875717,
+          0.019002873450517654,
+          -0.03386351466178894,
+          0.01975875534117222,
+          0.010053230449557304,
+          -0.004678909666836262,
+          -0.020166931673884392,
+          -0.02052975445985794,
+          -0.021739166229963303,
+          0.005408335942775011,
+          0.026712870225310326,
+          -0.013386670500040054,
+          -0.006304055918008089,
+          -0.00480363005772233,
+          -0.01079399511218071,
+          0.0024944106116890907,
+          0.015231022611260414,
+          -0.027257105335593224,
+          0.02387075312435627,
+          -0.007914084941148758,
+          0.009902054443955421,
+          -0.008851378224790096,
+          0.003518630750477314,
+          0.00126704724971205,
+          -0.009803789667785168,
+          -0.0031917118467390537,
+          -0.018050462007522583,
+          0.010302671231329441,
+          0.025291811674833298,
+          -0.008934524841606617,
+          -0.025760458782315254,
+          0.007997231557965279,
+          -0.011005641892552376,
+          -0.012668582610785961,
+          -0.015556051395833492,
+          0.0027816458605229855,
+          0.015858404338359833,
+          0.002666373737156391,
+          0.022525284439325333,
+          0.0009061135351657867,
+          -0.0053176297806203365,
+          -0.006198232527822256,
+          0.0136058758944273,
+          0.010544554330408573,
+          -0.033289045095443726,
+          -0.008783348836004734,
+          0.0290409866720438,
+          -0.020061109215021133,
+          -0.004395453725010157,
+          0.003435483667999506,
+          0.026455869898200035,
+          -0.01171617116779089,
+          0.009629936888813972,
+          0.0003321156546007842,
+          0.001567510305903852,
+          -0.02009134367108345,
+          0.0010034333681687713,
+          0.07752326130867004,
+          0.0179899912327528,
+          -0.017128286883234978,
+          -0.00349784386344254,
+          -0.013658788055181503,
+          -0.012842435389757156,
+          0.00894964300096035,
+          -0.025382516905665398,
+          0.00860193744301796,
+          -0.00961481872946024,
+          -0.021890342235565186,
+          0.015752580016851425,
+          -0.00013605876301880926,
+          0.020408814772963524,
+          -0.03359139710664749,
+          0.012215052731335163,
+          0.015737462788820267,
+          0.01631193421781063,
+          0.01676546223461628,
+          0.0009358763927593827,
+          -0.013930905610322952,
+          -7.009624823695049e-05,
+          0.00649680569767952,
+          0.017566697672009468,
+          0.025397635996341705,
+          0.011534759774804115,
+          -0.012555199675261974,
+          0.03126328065991402,
+          0.005204247776418924,
+          0.0229183416813612,
+          0.013001170009374619,
+          -0.0021807195153087378,
+          -0.007808261085301638,
+          -0.007604172918945551,
+          0.014883317053318024,
+          0.01833769679069519,
+          -0.0036471306812018156,
+          -0.014490257948637009,
+          0.005960129667073488,
+          0.004985041916370392,
+          -0.006481688003987074,
+          -0.005438570864498615,
+          -0.0011205950286239386,
+          0.013840199448168278,
+          -0.009395613335072994,
+          0.026727987453341484,
+          0.00771755538880825,
+          0.0014087751042097807,
+          -0.015246139839291573,
+          -0.0015911315567791462,
+          -0.006617746781557798,
+          0.007728893309831619,
+          0.004796071443706751,
+          0.009350260719656944,
+          -0.0005513214273378253,
+          0.00126704724971205,
+          0.012925582006573677,
+          -0.03480080887675285,
+          -0.0007577717187814415,
+          0.023114871233701706,
+          -0.0026210208889096975,
+          -0.006889864336699247,
+          -0.009592142887413502,
+          -0.018352815881371498,
+          0.001622311770915985,
+          0.002823219168931246,
+          0.010582348331809044,
+          -0.0019917490426450968,
+          0.010703288950026035,
+          0.02885957434773445,
+          0.006727349944412708,
+          -0.016357285901904106,
+          0.0228276364505291,
+          0.006258702836930752,
+          0.008722878061234951,
+          0.014225699007511139,
+          -0.006236026529222727,
+          0.006118864752352238,
+          -0.010242201387882233,
+          -0.0066290851682424545,
+          0.02359863556921482,
+          0.01044628955423832,
+          -0.011549877002835274,
+          -0.004765836056321859,
+          0.014618758112192154,
+          -0.0265768114477396,
+          0.012744170613586903,
+          0.0019218300003558397,
+          0.013719257898628712,
+          1.0371232747274917e-05,
+          -0.016281697899103165,
+          -0.020076226443052292,
+          0.00599036505445838,
+          0.0032502925023436546,
+          -0.013054082170128822,
+          -0.018972638994455338,
+          0.0008801301009953022,
+          -0.014709463343024254,
+          -0.030159691348671913,
+          -0.021028637886047363,
+          0.014732140116393566,
+          0.0007105291006155312,
+          0.0076684230007231236,
+          0.010045671835541725,
+          -0.0025246459990739822,
+          0.007876290008425713,
+          0.02754434011876583,
+          0.014051846228539944,
+          0.018912168219685555,
+          -0.00829202588647604,
+          -0.002518976805731654,
+          0.011043435893952847,
+          0.011542318388819695,
+          0.01737016811966896,
+          -0.0018755322089418769,
+          -0.01983434334397316,
+          -0.013681463897228241,
+          0.01762716844677925,
+          -0.023371871560811996,
+          -0.02306951954960823,
+          0.013144788332283497,
+          -0.02341722510755062,
+          0.0002966836909763515,
+          -0.002492520958185196,
+          0.007573937531560659,
+          -0.003401468973606825,
+          -0.006001702975481749,
+          0.0018027785699814558,
+          -0.028451398015022278,
+          -0.03607068955898285,
+          -0.0024471678771078587,
+          0.009017672389745712,
+          -0.001181065570563078,
+          0.004221600946038961,
+          0.005933673586696386,
+          0.009070583619177341,
+          -0.01570722833275795,
+          -0.011610347777605057,
+          -0.007384967058897018,
+          0.010166613385081291,
+          0.008012348785996437,
+          0.027876928448677063,
+          -0.005449909251183271,
+          0.0035734320990741253,
+          0.01800510846078396,
+          0.009773554280400276,
+          -0.002562440000474453,
+          -0.004350101109594107,
+          0.012933140620589256,
+          -0.011345788836479187,
+          0.006292717531323433,
+          -0.019516874104738235,
+          0.0011782310903072357,
+          0.0018897049594670534,
+          0.009992759674787521,
+          0.019531991332769394,
+          -0.019199403002858162,
+          -0.0007086393889039755,
+          -0.008276907727122307,
+          -0.016795698553323746,
+          0.005669115111231804,
+          0.025216223672032356,
+          -0.004675130359828472,
+          0.018640050664544106,
+          -0.005994144361466169,
+          0.026365164667367935,
+          -0.01044628955423832,
+          0.013794846832752228,
+          0.0047167036682367325,
+          -0.004353880416601896,
+          0.008057702332735062,
+          0.01426349300891161,
+          -0.005204247776418924,
+          0.01570722833275795,
+          -0.012880229391157627,
+          0.016281697899103165,
+          0.0032011603470891714,
+          -0.0014805839164182544,
+          0.0018452969379723072,
+          0.0064627910032868385,
+          -0.02254040166735649,
+          0.011958053335547447,
+          0.01202608272433281,
+          -0.003779409918934107,
+          0.02195081301033497,
+          -0.011973170563578606,
+          -0.006973011419177055,
+          -0.0014730250695720315,
+          0.021905461326241493,
+          -0.02929798699915409,
+          0.01860981434583664,
+          0.0223741065710783,
+          -0.0005872258334420621,
+          -0.011935376562178135,
+          0.02124028466641903,
+          0.011345788836479187,
+          0.002981954487040639,
+          -0.009388054721057415,
+          0.020030872896313667,
+          -0.0006160438060760498,
+          -0.0027627486269921064,
+          0.009750877507030964,
+          -0.01087714172899723,
+          0.007577716838568449,
+          0.014769934117794037,
+          -0.00952411349862814,
+          0.0017215212574228644,
+          0.006220908835530281,
+          0.016251463443040848,
+          -0.015661874786019325,
+          -0.025820929557085037,
+          -0.018972638994455338,
+          0.024808047339320183,
+          -0.0022808739449828863,
+          -0.0064627910032868385,
+          -0.024883635342121124,
+          0.008186202496290207,
+          -0.026002340018749237,
+          0.013084317557513714,
+          -0.011519641615450382,
+          -0.002025763737037778,
+          -0.0007865896914154291,
+          0.005344085860997438,
+          0.00807281956076622,
+          0.018141167238354683,
+          0.0015618412289768457,
+          -0.014694346114993095,
+          -0.010914935730397701,
+          0.010302671231329441,
+          -0.001181065570563078,
+          -0.007048599887639284,
+          0.005657776724547148,
+          -0.005929894279688597,
+          0.0006406100001186132,
+          0.02378004789352417,
+          0.007403864059597254,
+          -0.0020975726656615734,
+          -0.005087085999548435,
+          -0.011436494998633862,
+          -0.006213350221514702,
+          -0.00021152885165065527,
+          -0.029615456238389015,
+          -0.0005579353892244399,
+          0.007815820164978504,
+          0.0039268070831894875,
+          0.0010818560840561986,
+          -0.007282922975718975,
+          -0.016916640102863312,
+          0.008473437279462814,
+          -2.8035545255988836e-05,
+          -0.02071116678416729,
+          -0.018624933436512947,
+          0.0003132185956928879,
+          0.010340466164052486,
+          0.0046335565857589245,
+          0.010506759397685528,
+          0.014543170109391212,
+          0.018035344779491425,
+          0.01746087335050106,
+          -0.002492520958185196,
+          -0.01895751990377903,
+          -0.006084850057959557,
+          0.013628552667796612,
+          -0.02833045832812786,
+          0.001375705236569047,
+          0.013636111281812191,
+          -0.0034997337497770786,
+          -0.005453688558191061,
+          -0.023674223572015762,
+          0.019093578681349754,
+          0.005933673586696386,
+          -0.004138453863561153,
+          0.00837517250329256,
+          0.010211966000497341,
+          -0.016523580998182297,
+          -0.00286101340316236,
+          0.003199270460754633,
+          -0.007294261362403631,
+          0.003316432237625122,
+          -0.01334131695330143,
+          -0.0058845411986112595,
+          -0.005079526919871569,
+          -0.002764638513326645,
+          0.003686814568936825,
+          0.012713935226202011,
+          -0.0014427897986024618,
+          -0.012245288118720055,
+          0.012910464778542519,
+          0.0003134548314847052,
+          0.019184285774827003,
+          -0.008186202496290207,
+          0.008768231607973576,
+          -0.020151814445853233,
+          -0.012751729227602482,
+          0.000203733827220276,
+          -0.012736611999571323,
+          0.015072287060320377,
+          -0.0049056741409003735,
+          0.006368305999785662,
+          0.0025511018466204405,
+          -0.01456584595143795,
+          -0.007426540832966566,
+          0.003635792527347803,
+          0.000584391294978559,
+          0.01622122712433338,
+          0.014898434281349182,
+          0.015367081388831139,
+          0.0012717714998871088,
+          0.004350101109594107,
+          -0.02938869222998619,
+          -0.0074454378336668015,
+          0.013711699284613132,
+          0.022404342889785767,
+          0.0019284439040347934,
+          0.008511231280863285,
+          0.013469817116856575,
+          -0.0009835914243012667,
+          -0.0113684656098485,
+          -0.019290108233690262,
+          0.016916640102863312,
+          -0.025911634787917137,
+          0.01417278777807951,
+          0.006640423554927111,
+          -0.003197380807250738,
+          0.0028458957094699144,
+          0.010204407386481762,
+          0.019940167665481567,
+          -0.01207143533974886,
+          -0.011806877329945564,
+          -0.012116788886487484,
+          0.019547108560800552,
+          -0.008458320051431656,
+          -0.009599701501429081,
+          0.000633523624856025,
+          -0.02080187201499939,
+          -0.003979718778282404,
+          0.01421814039349556,
+          -0.02264622412621975,
+          -0.006285158917307854,
+          0.006613967474550009,
+          -0.00873043667525053,
+          -0.02449057623744011,
+          0.00832981988787651,
+          0.006870967335999012,
+          -0.0026474767364561558,
+          0.007630628999322653,
+          -0.017702756449580193,
+          -0.04359927400946617,
+          0.010559671558439732,
+          0.0019426167709752917,
+          -0.013477375730872154,
+          0.0014342861250042915,
+          -0.003274858696386218,
+          -0.006474129389971495,
+          -0.004168689250946045,
+          -0.011648141779005527,
+          0.01965293288230896,
+          0.023009048774838448,
+          -0.007959437556564808,
+          0.0020427710842341185,
+          -0.027045458555221558,
+          0.01983434334397316,
+          -0.007709996309131384,
+          -0.0011999626876786351,
+          -0.010597465559840202,
+          0.0031784838065505028,
+          0.04311550781130791,
+          -0.015382198616862297,
+          -0.011300436221063137,
+          -0.014512934722006321,
+          0.0008928856113925576,
+          -0.02124028466641903,
+          -0.017687639221549034,
+          0.0012472053058445454,
+          -0.021210048347711563,
+          0.012600553222000599,
+          0.0022298519033938646,
+          0.013507611118257046,
+          0.00108563550747931,
+          -0.008397849276661873,
+          -0.005427232943475246,
+          -0.0223741065710783,
+          0.005831629503518343,
+          0.011814435943961143,
+          0.009221760556101799,
+          0.013855316676199436,
+          0.006198232527822256,
+          0.005616203416138887,
+          -0.0018538006115704775,
+          0.0033806823194026947,
+          -0.016266580671072006,
+          0.014588522724807262,
+          -0.018352815881371498,
+          -0.004516394808888435,
+          -0.004380336031317711,
+          -0.012003405950963497,
+          -0.0015542823821306229,
+          0.029086340218782425,
+          -0.004746939055621624,
+          -0.00023727607913315296,
+          -0.004350101109594107,
+          0.0006084850174374878,
+          0.017612051218748093,
+          0.00868508405983448,
+          0.02677334100008011,
+          -0.008889172226190567,
+          0.00495858583599329,
+          -0.022842753678560257,
+          0.004391674418002367,
+          0.006455232389271259,
+          -0.011360906064510345,
+          0.008065260946750641,
+          0.017052698880434036,
+          0.005790056195110083,
+          0.04459703713655472,
+          -0.01895751990377903,
+          0.0017479771049693227,
+          -0.03555668890476227,
+          0.023371871560811996,
+          -0.0007969830767251551,
+          0.004486159887164831,
+          -0.0028062118217349052,
+          0.011685935780405998,
+          -0.000542817753739655,
+          -0.008926966227591038,
+          0.010922495275735855,
+          0.013235493563115597,
+          -0.02140657790005207,
+          0.0011952384375035763,
+          -0.035314805805683136,
+          0.0029233735986053944,
+          0.0004955751355737448,
+          0.006326732225716114,
+          -0.0066442028619349,
+          -0.01800510846078396,
+          0.01021952461451292,
+          -0.00873043667525053,
+          0.00952411349862814,
+          0.0008295804727822542,
+          0.023825401440262794,
+          -0.020680932328104973,
+          0.03156563267111778,
+          -0.0004497497866395861,
+          -0.005117321386933327,
+          0.013106994330883026,
+          -0.011111465282738209,
+          -0.004599541891366243,
+          -0.01563164032995701,
+          -0.008216436952352524,
+          0.013122111558914185,
+          0.002330006333068013,
+          0.015321727842092514,
+          -0.011935376562178135,
+          0.009660172276198864,
+          0.017566697672009468,
+          -0.007838496007025242,
+          0.0007502128719352186,
+          0.004229160025715828,
+          0.011050994507968426,
+          0.005805173888802528,
+          -0.00930490717291832,
+          -0.006334291305392981,
+          -0.006750026252120733,
+          -0.022071754559874535,
+          -0.01667475700378418,
+          -0.012222612276673317,
+          -0.00526471808552742,
+          0.004391674418002367,
+          0.0061566587537527084,
+          -0.0018178962636739016,
+          0.008488554507493973,
+          -0.0006769868195988238,
+          -0.016553815454244614,
+          0.0006580897606909275,
+          0.013182582333683968,
+          0.015155433677136898,
+          0.007740231696516275,
+          0.0011309884721413255,
+          0.029086340218782425,
+          0.0013794846599921584,
+          -0.007861172780394554,
+          0.005060629919171333,
+          0.0003578628820832819,
+          -0.025034811347723007,
+          0.007543702609837055,
+          -0.019531991332769394,
+          0.005083306692540646,
+          -0.016372404992580414,
+          0.007929202169179916,
+          0.028632810339331627,
+          0.009115937165915966,
+          -0.0006354133365675807,
+          -0.012479611672461033,
+          0.007997231557965279,
+          -0.01596422865986824,
+          -0.01727946288883686,
+          0.011081229895353317,
+          5.4535707022296265e-05,
+          0.011194612830877304,
+          0.014694346114993095,
+          -0.008737996220588684,
+          -0.02043904922902584,
+          -0.00034061932819895446,
+          -0.015193228609859943,
+          0.00864729005843401,
+          0.01074864249676466,
+          0.0011461060494184494,
+          -0.014905992895364761,
+          0.014989140443503857,
+          0.0042782919481396675,
+          -0.00987181905657053,
+          -0.016735227778553963,
+          -0.03090045601129532,
+          0.01210922934114933,
+          0.011171936057507992,
+          0.025563929229974747,
+          -0.009645054116845131,
+          -0.005941232666373253,
+          -0.011648141779005527,
+          0.004323645029217005,
+          0.005948791280388832,
+          -0.019637813791632652,
+          0.014905992895364761,
+          -0.01382508222013712,
+          -0.0024963002651929855,
+          0.0027533001266419888,
+          -0.0024944106116890907,
+          -0.00895720161497593,
+          -0.005465026944875717,
+          0.010400936007499695,
+          -0.0032238366547971964,
+          0.0014853081665933132,
+          0.013212817721068859,
+          -0.0006325787398964167,
+          0.0199099313467741,
+          -0.02376493066549301,
+          0.03271457180380821,
+          0.006202011834830046,
+          3.097344597335905e-05,
+          0.00033376915962435305,
+          -0.008019908331334591,
+          -0.017491109669208527,
+          0.00215048436075449,
+          0.02299392968416214,
+          -0.004471042193472385,
+          -0.0113684656098485,
+          0.00046510365791618824,
+          0.0060735116712749004,
+          -0.013673905283212662,
+          -0.012970935553312302,
+          -0.01267614122480154,
+          0.01359831728041172,
+          -0.00378318945877254,
+          -0.005158894695341587,
+          0.029902692884206772,
+          0.0063002766110002995,
+          0.030945809558033943,
+          -0.013537846505641937,
+          -0.005412115249782801,
+          4.6858778659952804e-05,
+          0.013893111608922482,
+          0.004308527335524559,
+          -0.02122516743838787,
+          0.009403171949088573,
+          0.00011208312935195863,
+          0.017218992114067078,
+          -0.0033353292383253574,
+          -0.009697966277599335,
+          -0.005865644197911024,
+          0.015276375226676464,
+          -0.003953262697905302,
+          0.011436494998633862,
+          0.006149100139737129,
+          0.016568932682275772,
+          0.012623229064047337,
+          -0.01232843566685915,
+          0.005858085583895445,
+          -0.009939848445355892,
+          0.011897582560777664,
+          -0.02107398957014084,
+          -0.007539922837167978,
+          0.01175396516919136,
+          0.01390822883695364,
+          0.00245661661028862,
+          -0.0011621685698628426,
+          0.0009722532122395933,
+          0.0043387627229094505,
+          -0.005653997417539358,
+          0.012865111231803894,
+          -0.01176152378320694,
+          0.013938464224338531,
+          0.004731821361929178,
+          -0.007460555527359247,
+          -0.000914617208763957,
+          0.0007818654412403703,
+          -0.004463483113795519,
+          -0.006538379471749067,
+          -0.01176152378320694,
+          -0.016689874231815338,
+          -0.014490257948637009,
+          -0.01202608272433281,
+          0.014701904729008675,
+          0.011897582560777664,
+          0.00860193744301796,
+          0.008662407286465168,
+          0.011376024223864079,
+          -0.01885169744491577,
+          0.0071733202785253525,
+          -0.002624800195917487,
+          -0.003779409918934107,
+          0.02087746001780033,
+          -0.00232622679322958,
+          -0.014293728396296501,
+          0.0023791384883224964,
+          0.01718875765800476,
+          0.01513275783509016,
+          -0.00019511203572619706,
+          0.006663099862635136,
+          0.004387895110994577,
+          0.02281251922249794,
+          -0.019426167011260986,
+          -0.0032219470012933016,
+          -0.007815820164978504,
+          0.012192376889288425,
+          0.02263110689818859,
+          -0.004523953888565302,
+          -0.015276375226676464,
+          -0.03458916023373604,
+          0.013832640834152699,
+          -0.014671669341623783,
+          0.0014390103751793504,
+          0.013568081893026829,
+          -0.028315339237451553,
+          0.013749493286013603,
+          -0.022495048120617867,
+          0.007675981614738703,
+          0.017793461680412292,
+          -0.016977109014987946,
+          -0.016130521893501282,
+          -0.0036849246826022863,
+          0.004244277253746986,
+          0.025594163686037064,
+          -0.020590225234627724,
+          -0.029706163331866264,
+          -0.0022525284439325333,
+          -0.007808261085301638,
+          -0.011897582560777664,
+          -0.008004790171980858,
+          -0.009554348886013031,
+          0.02492898888885975,
+          -0.019093578681349754,
+          0.003085888223722577,
+          -0.02993292734026909,
+          -0.0066064088605344296,
+          -0.011315553449094296,
+          -0.007573937531560659,
+          -0.019199403002858162,
+          -0.005665335804224014,
+          -0.010725965723395348,
+          -0.0004521119117271155,
+          -0.008050143718719482,
+          0.0004632139462046325,
+          -0.019199403002858162,
+          0.012131906114518642,
+          0.008231555111706257,
+          0.011685935780405998,
+          0.013507611118257046,
+          -0.007951878942549229,
+          -0.01966805011034012,
+          -0.00599036505445838,
+          0.01114925928413868,
+          0.0017319145845249295,
+          0.009085701778531075,
+          -0.0057295854203403,
+          0.016901521012187004,
+          -0.025775576010346413,
+          0.004119556862860918,
+          0.018806343898177147,
+          0.007400084752589464,
+          0.01539731677621603,
+          -0.02054487355053425,
+          0.011648141779005527,
+          -0.00618311483412981,
+          -0.0016988448332995176,
+          -6.315010068647098e-06,
+          -0.003911689389497042,
+          -0.005676673725247383,
+          0.019093578681349754,
+          0.016190992668271065,
+          -0.010393377393484116,
+          -0.002549211960285902,
+          0.0016260911943390965,
+          -0.015556051395833492,
+          -0.004433247726410627,
+          -0.00013015343574807048,
+          -0.0034090278204530478,
+          0.005548174027353525,
+          0.0041497922502458096,
+          -0.012842435389757156,
+          -0.005053071305155754,
+          0.0019445064244791865,
+          -0.00023975632211659104,
+          -0.01711316779255867,
+          -0.007959437556564808,
+          -0.00476205674931407,
+          -0.009894495829939842,
+          -0.004637335892766714,
+          0.004183806944638491,
+          0.02210198901593685,
+          -0.0010005987714976072,
+          -0.005767379887402058,
+          -0.001888760132715106,
+          -0.0203634612262249,
+          0.0026153516955673695,
+          0.00038644467713311315,
+          -0.0020919034723192453,
+          -0.016100287437438965,
+          0.008715319447219372,
+          0.002876131096854806,
+          0.017581814900040627,
+          -0.026546575129032135,
+          -0.002184499055147171,
+          -0.014550728723406792,
+          -0.01426349300891161,
+          -0.02615351788699627,
+          0.01417278777807951,
+          -0.011685935780405998,
+          -0.0012207494582980871,
+          -0.01879122667014599,
+          -0.011481847614049911,
+          0.0073622907511889935,
+          0.003960821777582169,
+          0.022268284112215042,
+          -0.03582880645990372,
+          -0.028239751234650612,
+          0.00726402597501874,
+          -0.006761364638805389,
+          0.006304055918008089,
+          -0.019244756549596786,
+          0.010597465559840202,
+          -0.009697966277599335,
+          -0.021965930238366127,
+          -0.0006708452710881829,
+          0.02308463677763939,
+          -0.029086340218782425,
+          -0.04021292179822922,
+          -0.02569998800754547,
+          0.016266580671072006,
+          0.006836952641606331,
+          0.022389225661754608,
+          0.014082081615924835,
+          -0.0040666451677680016,
+          -0.007298040669411421,
+          -0.010544554330408573,
+          0.0036509099882096052,
+          0.019804108887910843,
+          -0.011890023946762085,
+          -0.0061075263656675816,
+          0.00015991629334166646,
+          0.021346107125282288,
+          -0.0009604425868019462,
+          0.013938464224338531,
+          0.006213350221514702,
+          -0.017400404438376427,
+          -0.004637335892766714,
+          0.0001971198507817462,
+          -0.005170233082026243,
+          0.0018972638063132763,
+          0.0011744516668841243,
+          -0.0025038591120392084,
+          0.010862024500966072,
+          0.007974554784595966,
+          -0.004622218664735556,
+          -0.013575640507042408,
+          -0.003023528028279543,
+          0.0017999439733102918,
+          -0.018730755895376205,
+          0.021799637004733086,
+          0.008889172226190567,
+          -0.0054839239455759525,
+          -0.004773394670337439,
+          0.0052382624708116055,
+          0.022147342562675476,
+          0.004263174720108509,
+          0.005298732779920101,
+          0.007638187613338232,
+          0.007071276195347309,
+          0.00863973144441843,
+          -0.0014796389732509851,
+          0.01052943617105484,
+          -0.003289976390078664,
+          0.017400404438376427,
+          -0.002144815167412162,
+          -0.015737462788820267,
+          -0.0009599701152183115,
+          -0.007475673221051693,
+          0.009622378274798393,
+          -0.00995496567338705,
+          0.011474289000034332,
+          0.015427551232278347,
+          0.008881613612174988,
+          0.0011961832642555237,
+          0.010189289227128029,
+          -0.014323963783681393,
+          0.027514105662703514,
+          -0.00039683806244283915,
+          -0.0010270546190440655,
+          0.009645054116845131,
+          0.005544394254684448,
+          -0.00041526267887093127,
+          -0.0011054774513468146,
+          0.018473755568265915,
+          -0.006663099862635136,
+          -0.02606281079351902,
+          0.022434577345848083,
+          -0.01974363811314106,
+          -0.018640050664544106,
+          0.010007877834141254,
+          -0.006867188028991222,
+          0.009577024728059769,
+          0.004459703806787729,
+          0.009206642396748066,
+          0.03168657422065735,
+          0.005438570864498615,
+          -0.01966805011034012,
+          -0.002471734071150422,
+          -0.010824230499565601,
+          0.008571702055633068,
+          -0.02176940254867077,
+          -0.0059223356656730175,
+          -0.009947407059371471,
+          0.004051527474075556,
+          -0.0021183593198657036,
+          -0.0018434072844684124,
+          0.017959756776690483,
+          0.0021674917079508305,
+          0.012131906114518642,
+          -0.001987969735637307,
+          -0.004769615363329649,
+          0.03186798468232155,
+          -0.0008933580247685313,
+          -0.014233258552849293,
+          -0.004951027221977711,
+          -0.0053176297806203365,
+          -0.039789628237485886,
+          -0.01083178911358118,
+          -0.0023696899879723787,
+          0.01703757978975773,
+          -0.0011120913550257683,
+          0.00833737850189209,
+          -0.007698658388108015,
+          -0.00695789372548461,
+          0.0004176248039584607,
+          0.003666027681902051,
+          -0.0009273727191612124,
+          0.02149728499352932,
+          -0.007793143391609192,
+          0.002653145929798484,
+          -0.002394256182014942,
+          -0.02817928045988083,
+          -0.02527669444680214,
+          -0.002486851764842868,
+          -0.022510165348649025,
+          0.0024981899186968803,
+          0.013205258175730705,
+          -0.012131906114518642,
+          -0.026017459109425545,
+          0.0031104544177651405,
+          -0.02379516512155533,
+          0.03613115847110748,
+          0.011655700393021107,
+          -0.0028591237496584654,
+          -0.014603639952838421,
+          -0.007581496611237526,
+          -0.0037718513049185276,
+          -0.020937930792570114,
+          -0.006791599560528994,
+          0.009289789944887161,
+          -0.0014210582012310624,
+          0.0016147529240697622,
+          0.0021618225146085024,
+          -0.04220845177769661,
+          -0.016251463443040848,
+          0.013961140997707844,
+          -0.0319586917757988,
+          0.018670285120606422,
+          0.03088533878326416,
+          -0.0011215399717912078,
+          0.0031028955709189177,
+          -0.0033258807379752398,
+          0.005257159471511841,
+          0.010295112617313862,
+          0.005646438337862492,
+          -0.00903278961777687,
+          -0.010967847891151905,
+          -0.015949109569191933,
+          0.025140635669231415,
+          0.009108378551900387,
+          -0.01263078860938549,
+          0.008768231607973576,
+          0.011353347450494766,
+          0.004002395085990429,
+          -0.004603321198374033,
+          0.0035715424455702305,
+          -0.02501969411969185,
+          -0.005895879585295916,
+          -0.0032219470012933016,
+          0.013099434785544872,
+          0.01267614122480154,
+          0.0037246085703372955,
+          0.00495858583599329,
+          0.007736452389508486,
+          -0.002653145929798484,
+          -0.0060735116712749004,
+          -0.02387075312435627,
+          0.020061109215021133,
+          -0.00042943545849993825,
+          0.0036981527227908373,
+          0.0080803781747818,
+          0.0025246459990739822,
+          0.002176940208300948,
+          -0.008012348785996437,
+          0.011519641615450382,
+          0.01903310790657997,
+          0.016115404665470123,
+          -0.006220908835530281,
+          0.007589055225253105,
+          -0.01259299460798502,
+          0.003051873529329896,
+          0.010635259561240673,
+          -0.002365910680964589,
+          -0.008307143114507198,
+          -0.025473223999142647,
+          0.00033754855394363403,
+          -0.01083934772759676,
+          -0.0007251743227243423,
+          -0.0002626689965836704,
+          0.0010629590833559632,
+          -0.00726402597501874,
+          -0.018141167238354683,
+          0.0011952384375035763,
+          0.0026588148903101683,
+          0.020302990451455116,
+          -0.027317576110363007,
+          -0.002471734071150422,
+          0.017763227224349976,
+          0.009886936284601688,
+          0.007778025697916746,
+          0.019290108233690262,
+          0.005982805974781513,
+          0.0013719258131459355,
+          -0.007033482193946838,
+          -0.002777866320684552,
+          0.0197285208851099,
+          -0.002549211960285902,
+          0.008276907727122307,
+          -0.0145204933360219,
+          -0.014550728723406792,
+          -0.016871286556124687,
+          0.010514318943023682,
+          -0.013507611118257046,
+          -0.008261790499091148,
+          0.016992228105664253,
+          -0.002679601777344942,
+          -0.011958053335547447,
+          0.01718875765800476,
+          0.0015892419032752514,
+          0.0070674968883395195,
+          -0.003622564487159252,
+          0.0023791384883224964,
+          0.006326732225716114,
+          -0.0019019880564883351,
+          0.0035016234032809734,
+          0.005381879862397909,
+          -0.026032576337456703,
+          0.019154049456119537,
+          0.004372777417302132,
+          0.020665813237428665,
+          -0.009478759951889515,
+          0.017687639221549034,
+          -0.002324337139725685,
+          0.00802746694535017,
+          0.020378578454256058,
+          -0.016977109014987946,
+          0.008397849276661873,
+          0.003934365697205067,
+          0.015556051395833492,
+          -0.010378260165452957,
+          -0.0022449695970863104,
+          -0.012691258452832699,
+          -0.00199552858248353,
+          -0.006780261639505625,
+          -0.0024755136109888554,
+          -0.0010185509454458952,
+          -0.0005735254962928593,
+          -0.0030613220296800137,
+          -0.006602629087865353,
+          0.005249600391834974,
+          0.021875225007534027,
+          -0.009320025332272053,
+          -0.016478227451443672,
+          0.004017512779682875,
+          0.0018452969379723072,
+          0.021708931773900986,
+          0.0401826873421669,
+          -0.011890023946762085,
+          0.02376493066549301,
+          -0.019063344225287437,
+          -0.011481847614049911,
+          0.008798466064035892,
+          0.00307266041636467,
+          -0.013054082170128822,
+          -0.035586923360824585,
+          0.00587698258459568,
+          0.012645905837416649,
+          -0.009660172276198864,
+          0.02113446034491062,
+          0.014384434558451176,
+          0.005449909251183271,
+          -0.017869049683213234,
+          0.020151814445853233,
+          -0.006674438249319792,
+          0.0055670710280537605,
+          -0.00898743700236082,
+          0.016175875440239906,
+          -0.00648924708366394,
+          0.013666346669197083,
+          -0.016342168673872948,
+          0.004618438892066479,
+          0.008964760228991508,
+          0.02843628078699112,
+          -0.02133098989725113,
+          0.014497816562652588,
+          -0.016432873904705048,
+          -0.004875438753515482,
+          -0.005442350637167692,
+          0.009697966277599335,
+          -0.0001299172145081684,
+          0.006228467915207148,
+          -0.00011285082291578874,
+          -0.016357285901904106,
+          -0.0007147809374146163,
+          -0.005257159471511841,
+          -0.010703288950026035,
+          -0.0003616423055063933,
+          0.0022865431383252144,
+          0.005918555893003941,
+          0.01298605278134346,
+          0.011421376839280128,
+          -0.01201852411031723,
+          0.012736611999571323,
+          -0.041633982211351395,
+          0.001619477174244821,
+          -0.004289630334824324,
+          -0.0005791945732198656,
+          0.0007459610351361334,
+          -0.015661874786019325,
+          -0.012124347500503063,
+          -0.0070674968883395195,
+          0.010196847841143608,
+          -0.00552927702665329,
+          -0.009765995666384697,
+          -0.011769082397222519,
+          0.007592834532260895,
+          0.001629870617762208,
+          0.006625305861234665,
+          0.010552112944424152,
+          0.012260406278073788,
+          -0.033047161996364594,
+          -0.01263834722340107,
+          -0.019607579335570335,
+          0.00012212219007778913,
+          -0.005623762030154467,
+          -0.0010818560840561986,
+          -0.014422228559851646,
+          -0.02553369477391243,
+          -0.0010998082580044866,
+          0.0154351107776165,
+          0.013175023719668388,
+          -0.004206483252346516,
+          0.009796231053769588,
+          -0.0034676087088882923,
+          0.004180027637630701,
+          -0.018141167238354683,
+          0.002394256182014942,
+          0.004100659862160683,
+          0.0036905938759446144,
+          0.020726284012198448,
+          -0.019788991659879684,
+          0.01921452023088932,
+          0.014278611168265343,
+          0.0019936386961489916,
+          0.011028318665921688,
+          -0.0011801207438111305,
+          0.0066442028619349,
+          -0.008465878665447235,
+          -0.012608111836016178,
+          0.005559511948376894,
+          -0.0010506759863346815,
+          0.008405407890677452,
+          0.006504364777356386,
+          0.005627541337162256,
+          0.008087937720119953,
+          0.020302990451455116,
+          -0.008163525722920895,
+          0.0028043221682310104,
+          -0.01596422865986824,
+          -0.0006836007814854383,
+          -0.0009590252884663641,
+          -0.004414350725710392,
+          0.013205258175730705,
+          0.0024736239574849606,
+          -0.013885552063584328,
+          -0.011391141451895237,
+          -0.02405216544866562,
+          0.011973170563578606,
+          0.014437345787882805,
+          0.003580990945920348,
+          0.010620142333209515,
+          -0.004951027221977711,
+          0.001301061944104731,
+          0.01545022800564766,
+          -0.007033482193946838,
+          -0.028904927894473076,
+          0.0055670710280537605,
+          0.010234642773866653,
+          0.023553283885121346,
+          -0.0020767857786267996,
+          -0.004845203831791878,
+          0.00418758625164628,
+          0.0017914402997121215,
+          0.014414669945836067,
+          0.005638879723846912,
+          0.012230170890688896,
+          0.019063344225287437,
+          -0.006013041362166405,
+          0.007010805420577526,
+          0.005752262193709612,
+          0.01694687455892563,
+          -0.0160700511187315,
+          -7.039151387289166e-05,
+          0.02122516743838787,
+          0.009486319497227669,
+          0.01206387672573328,
+          -0.010068348608911037,
+          0.014588522724807262,
+          0.0036093364469707012,
+          0.0019142711535096169,
+          -0.004860321059823036,
+          0.0018065579934045672,
+          0.00434632133692503,
+          0.0047167036682367325,
+          0.02842116355895996,
+          -0.0028723515570163727,
+          0.008050143718719482,
+          -0.040243156254291534,
+          0.016810815781354904,
+          0.00032148606260307133,
+          0.009131054393947124,
+          -0.0036527998745441437,
+          0.018367933109402657,
+          0.03088533878326416,
+          0.014104758389294147,
+          0.00511354161426425,
+          0.005771159194409847,
+          0.0034524910151958466,
+          -0.006413658615201712,
+          -0.011028318665921688,
+          0.004323645029217005,
+          0.004667571280151606,
+          -0.0074454378336668015,
+          0.014505375176668167,
+          0.0032994248904287815,
+          -0.014286169782280922,
+          -0.004992600530385971,
+          0.013273287564516068,
+          0.0006491136737167835,
+          -0.010680613107979298,
+          0.03483104333281517,
+          -0.012426700443029404,
+          -0.0016100286738947034,
+          -0.0001238937838934362,
+          -0.01563164032995701,
+          -0.0069843498058617115,
+          0.020333226770162582,
+          -0.014490257948637009,
+          -0.010884701274335384,
+          -0.037340570241212845,
+          0.012237729504704475,
+          0.0029082559049129486,
+          0.003282417543232441,
+          -0.014724581502377987,
+          -0.03942680358886719,
+          0.005022835917770863,
+          0.011194612830877304,
+          -0.007846055552363396,
+          0.009229319170117378,
+          0.00679915864020586,
+          -0.01198073010891676,
+          0.012789523229002953,
+          0.029691044241189957,
+          0.01817140355706215,
+          -0.01210922934114933,
+          0.004773394670337439,
+          0.024702223017811775,
+          0.008677525445818901,
+          -0.013522729277610779,
+          0.020756520330905914,
+          -0.02798275090754032,
+          0.007543702609837055,
+          0.0159793458878994,
+          -0.0014191684313118458,
+          0.0035847704857587814,
+          -8.899329259293154e-05,
+          0.015933992341160774,
+          0.004478600807487965,
+          0.008889172226190567,
+          0.03235175088047981,
+          -0.010673053562641144,
+          -0.00025322046712972224,
+          -0.013734376057982445,
+          -0.009463642723858356,
+          -0.0015391646884381771,
+          0.00645145308226347,
+          -0.02254040166735649,
+          -0.0007809206144884229,
+          0.002675822237506509,
+          -0.026274457573890686,
+          0.010642818175256252,
+          0.020680932328104973,
+          -0.016825933009386063,
+          -2.1598738385364413e-05,
+          0.014550728723406792,
+          0.017067816108465195,
+          0.010673053562641144,
+          -0.012509847059845924,
+          -0.0007492680451832712,
+          -0.009493878111243248,
+          -0.008722878061234951,
+          0.004229160025715828,
+          -0.02394634112715721,
+          0.03833833336830139,
+          -0.01229064166545868,
+          0.006024379748851061,
+          0.015601404942572117,
+          -0.004387895110994577,
+          -0.004973703529685736,
+          0.004667571280151606,
+          0.011557435616850853,
+          -0.023659106343984604,
+          0.03340998291969299,
+          -0.016024697571992874,
+          0.00491323322057724,
+          0.01703757978975773,
+          0.027075693011283875,
+          -0.007638187613338232,
+          0.0013416905421763659,
+          -0.00026030687149614096,
+          0.028889810666441917,
+          -4.538244684226811e-05,
+          -0.02325093001127243,
+          -0.01895751990377903,
+          -0.005136218387633562,
+          0.01938081532716751,
+          0.00757015822455287,
+          -0.015752580016851425,
+          0.028935162350535393,
+          -0.004939688835293055,
+          -0.01956222578883171,
+          -0.002256307750940323,
+          -0.0008097385871224105,
+          -0.0036206748336553574,
+          0.0317772775888443,
+          0.01017417199909687,
+          -0.01817140355706215,
+          -0.01965293288230896,
+          -0.008760672062635422,
+          -0.02686404623091221,
+          -0.0022789842914789915,
+          0.0004492773732636124,
+          -0.008874054998159409,
+          0.00301596918143332,
+          -0.012494729831814766,
+          -0.019622696563601494,
+          0.0012320876121520996,
+          0.014089640229940414,
+          0.04746938869357109,
+          0.001130043645389378,
+          0.007275364361703396,
+          -0.02624422311782837,
+          0.01272905245423317,
+          -0.011685935780405998,
+          -0.0033409984316676855,
+          -0.0027344031259417534,
+          -0.00925199594348669,
+          -0.00039022407145239413,
+          -0.006285158917307854,
+          -0.015646757557988167,
+          -0.029328221455216408,
+          0.01879122667014599,
+          0.002141035860404372,
+          -0.006311614532023668,
+          -0.013537846505641937,
+          -0.02387075312435627,
+          0.012759287841618061,
+          -0.01675034500658512,
+          -0.008624613285064697,
+          -0.0356171615421772,
+          -0.024989459663629532,
+          0.011285318061709404,
+          0.006882305722683668,
+          -0.0017583705484867096,
+          -0.0028855795972049236,
+          -0.013726817443966866,
+          -0.02124028466641903,
+          -0.03640327602624893,
+          0.009115937165915966,
+          -0.005355423782020807,
+          0.01622122712433338,
+          -0.011247524060308933,
+          -5.229168164078146e-05,
+          -0.006319173611700535,
+          0.008458320051431656,
+          0.020665813237428665,
+          0.01140625961124897,
+          0.003021638374775648,
+          0.002367800334468484,
+          0.01700734533369541,
+          -0.004463483113795519,
+          -0.02376493066549301,
+          -0.02518598921597004,
+          0.00286479271017015,
+          0.00599036505445838,
+          -0.01782369799911976,
+          0.009463642723858356,
+          0.0007790309027768672,
+          0.0019237196538597345,
+          0.009267113171517849,
+          0.0034317043609917164,
+          -0.0069843498058617115,
+          -0.010400936007499695,
+          -0.001878366805613041,
+          0.026969870552420616,
+          0.007815820164978504,
+          -0.010249760001897812,
+          -0.010718407109379768,
+          0.0023848076816648245,
+          -0.007203555665910244,
+          0.0011961832642555237,
+          0.012222612276673317,
+          -0.013794846832752228,
+          -0.001549558131955564,
+          -0.011307994835078716,
+          -0.010000319220125675,
+          -0.011988288722932339,
+          -0.017475992441177368,
+          0.0011961832642555237,
+          -0.000834304781164974,
+          -0.03854998201131821,
+          0.013855316676199436,
+          -0.0006779316463507712,
+          -0.007124187890440226,
+          -0.0015949109802022576,
+          0.007936760783195496,
+          0.0005286450032144785,
+          -0.003512961557134986,
+          -0.0002489686303306371,
+          0.018201638013124466,
+          -0.027181517332792282,
+          0.012970935553312302,
+          0.021618224680423737,
+          -0.017143404111266136,
+          -0.001795219723135233,
+          -0.009206642396748066,
+          0.010113701224327087,
+          0.009161289781332016,
+          0.012494729831814766,
+          -0.001033668639138341,
+          0.003919248003512621,
+          4.290220749680884e-05,
+          -0.0010431171394884586,
+          0.006999467499554157,
+          0.0028118810150772333,
+          0.010982965119183064,
+          0.00807281956076622,
+          0.006364526227116585,
+          -0.01026487722992897,
+          0.010453848168253899,
+          0.0025945650413632393,
+          -0.0006132092676125467,
+          0.0011253192787989974,
+          0.01390067022293806,
+          0.023659106343984604,
+          0.012351111508905888,
+          -0.015540934167802334,
+          0.019819226115942,
+          -0.013205258175730705,
+          0.006663099862635136,
+          0.00278731482103467,
+          0.0002310164418304339,
+          0.03477057069540024,
+          0.012555199675261974,
+          0.024021929129958153,
+          0.025639517232775688,
+          0.026652399450540543,
+          0.019970402121543884,
+          -0.0020030871964991093,
+          -0.015057169832289219,
+          -0.01623634621500969,
+          -0.002135366667062044,
+          0.019879696890711784,
+          0.0018481315346434712,
+          0.00027849528123624623,
+          -0.010461406782269478,
+          -0.002199616516008973,
+          -0.004304748028516769,
+          0.010642818175256252,
+          0.0053176297806203365,
+          -0.0034543806686997414,
+          -0.0037510644178837538,
+          -0.0007029702537693083,
+          0.0008220216841436923,
+          -0.008133290335536003,
+          -0.007400084752589464,
+          -0.004924571141600609,
+          0.0021334770135581493,
+          -0.03053763322532177,
+          0.05000915378332138,
+          -0.00803502555936575,
+          0.014354199171066284,
+          -0.009728201664984226,
+          0.003121792571619153,
+          0.004901894833892584,
+          -0.013507611118257046,
+          -0.002435829723253846,
+          -0.003199270460754633,
+          0.0026909399311989546,
+          -0.0147472582757473,
+          0.002702278085052967,
+          0.008571702055633068,
+          -0.014596081338822842,
+          0.0032011603470891714,
+          0.01149696484208107,
+          0.006753805559128523,
+          0.005933673586696386,
+          0.009025231003761292,
+          0.015193228609859943,
+          0.02464175410568714,
+          -0.00073462282307446,
+          -0.0005399832152761519,
+          0.0025322046130895615,
+          0.010695730336010456,
+          0.015828169882297516,
+          0.001142326625995338,
+          0.024883635342121124,
+          -0.0021126901265233755,
+          0.019063344225287437,
+          0.028194399550557137,
+          -0.010673053562641144,
+          -0.007490790449082851,
+          0.001940727001056075,
+          -0.01365122850984335,
+          0.008737996220588684,
+          -0.0019426167709752917,
+          -0.008019908331334591,
+          -0.013477375730872154,
+          0.001699789660051465,
+          0.02201128378510475,
+          0.00832981988787651,
+          -0.006511923391371965,
+          0.01263078860938549,
+          -0.002673932584002614,
+          0.0030197484884411097,
+          0.002269535791128874,
+          -0.0013917677570134401,
+          -0.0030367558356374502,
+          -0.010695730336010456,
+          0.01587352156639099,
+          -0.005230703391134739,
+          -0.021376343443989754,
+          0.009765995666384697,
+          -0.00434632133692503,
+          -0.008322261273860931,
+          0.03235175088047981,
+          -0.01622122712433338,
+          -0.006731129251420498,
+          -0.003365564625710249,
+          0.016296815127134323,
+          0.008019908331334591,
+          -0.0007880069897510111
+        ],
+        "title": "Switch Transformer",
+        "keyphrases": [
+          "Switch Transformer",
+          "Simplifying Sparse Routing",
+          "Efficient Sparse Routing",
+          "Improved Training",
+          "Fine-Tuning Techniques"
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "d17e0745-ef0b-4687-8d99-5321eb6cab7e",
+      "properties": {
+        "page_content": "3 Scaling Properties 11 3.1 Scaling Results on a Step-Basis 12 3.2 Scaling Results on a Time-Basis 13 3.3 Scaling Versus a Larger Dense Model 13 ",
+        "keyphrases": [
+          "Scaling Properties",
+          "Scaling Results on a Step-Basis",
+          "Scaling Results on a Time-Basis",
+          "Scaling Versus a Larger Dense Model"
+        ],
+        "title": "Scaling Properties",
+        "embedding": [
+          -0.020178575068712234,
+          0.010542738251388073,
+          -0.025771139189600945,
+          0.015961479395627975,
+          0.009152155369520187,
+          -0.016702115535736084,
+          -0.001157245016656816,
+          0.07732853293418884,
+          -0.004825475625693798,
+          0.01630912534892559,
+          0.00782203208655119,
+          0.006609050091356039,
+          0.04265463352203369,
+          -0.01061831321567297,
+          0.03403906524181366,
+          0.045133501291275024,
+          0.008094102144241333,
+          0.006782873068004847,
+          -0.0005828736466355622,
+          -0.042836014181375504,
+          -0.026269935071468353,
+          -0.020103000104427338,
+          -0.027645403519272804,
+          0.029096446931362152,
+          0.025499068200588226,
+          0.0028454058337956667,
+          0.03932932764291763,
+          -0.0032780736219137907,
+          -0.010232879780232906,
+          0.0061329263262450695,
+          -0.011638578958809376,
+          0.002119883894920349,
+          -0.04081059992313385,
+          -0.002091543283313513,
+          -0.04446844011545181,
+          -0.02823488973081112,
+          0.06620386242866516,
+          -0.015024347230792046,
+          -0.0017486208816990256,
+          0.0305474903434515,
+          0.011358950287103653,
+          0.01002126932144165,
+          -0.01330123282968998,
+          -0.00594398844987154,
+          0.01584055833518505,
+          -0.04274532571434975,
+          -0.048398349434137344,
+          0.013391923159360886,
+          0.007130519021302462,
+          0.006314306985586882,
+          0.014230807311832905,
+          -0.0615786612033844,
+          0.040961749851703644,
+          -0.01794154942035675,
+          -0.02935340255498886,
+          0.01768459379673004,
+          -0.00886496901512146,
+          0.0859743282198906,
+          0.023745723068714142,
+          0.01330123282968998,
+          0.025272341445088387,
+          -0.028113968670368195,
+          -0.03201364725828171,
+          -0.006930244620889425,
+          0.0015370103064924479,
+          -0.006253846455365419,
+          -0.035550568252801895,
+          0.003850555745884776,
+          -0.03279963135719299,
+          -0.007655766326934099,
+          -0.033464692533016205,
+          -0.0024070697836577892,
+          0.0023749503307044506,
+          -0.01073167659342289,
+          0.006182050332427025,
+          -0.01117756962776184,
+          0.030834676697850227,
+          0.010504950769245625,
+          -0.014759833924472332,
+          0.029020871967077255,
+          0.052811939269304276,
+          0.039057254791259766,
+          -0.04205003380775452,
+          0.02893018163740635,
+          0.01701953262090683,
+          -0.03422044590115547,
+          -0.033011242747306824,
+          -0.055865176022052765,
+          0.009061465039849281,
+          -0.021266857162117958,
+          -0.0681990459561348,
+          -0.033736761659383774,
+          0.012983816675841808,
+          -0.03210433945059776,
+          -0.026919880881905556,
+          0.050967905670404434,
+          -0.01117001287639141,
+          0.010421818122267723,
+          -0.02122151292860508,
+          0.0011147339828312397,
+          -0.002512875013053417,
+          0.03597378730773926,
+          0.01862172596156597,
+          -0.01503946166485548,
+          0.02306554652750492,
+          -0.011600791476666927,
+          -0.0017051651375368237,
+          0.030199844390153885,
+          -0.006102696061134338,
+          0.05272125080227852,
+          0.0037315248046070337,
+          -0.01797178015112877,
+          -0.03198342025279999,
+          -0.03736437112092972,
+          -0.060036927461624146,
+          0.009363765828311443,
+          -0.00215578218922019,
+          -0.04594971239566803,
+          0.010527623817324638,
+          -0.02753959782421589,
+          0.02655711956322193,
+          0.02265744097530842,
+          -0.007481943350285292,
+          0.005112661514431238,
+          -0.032527558505535126,
+          0.04428705945611,
+          0.02823488973081112,
+          -0.02261209487915039,
+          -0.001841200515627861,
+          -0.02264232560992241,
+          0.004058387596160173,
+          0.019241442903876305,
+          -0.02985219843685627,
+          -0.007198536302894354,
+          -0.01142696850001812,
+          -0.016959071159362793,
+          0.050030771642923355,
+          0.025000272318720818,
+          -0.005399846937507391,
+          0.005161785054951906,
+          -0.011154897511005402,
+          -0.04071991145610809,
+          0.040689680725336075,
+          0.013452382758259773,
+          -0.03385768458247185,
+          0.0025582199450582266,
+          -0.028386039659380913,
+          0.062334414571523666,
+          -0.0035784849897027016,
+          -0.006261404138058424,
+          -0.02492469549179077,
+          -0.01747298240661621,
+          0.007746456656605005,
+          0.0467054657638073,
+          -0.0063445367850363255,
+          0.005543440114706755,
+          -0.018939141184091568,
+          0.0396013967692852,
+          -0.013928506523370743,
+          0.034432053565979004,
+          0.011661251075565815,
+          -0.007716226391494274,
+          0.014034312218427658,
+          0.01913563720881939,
+          0.028159314766526222,
+          0.00200652121566236,
+          -0.0012942251050844789,
+          0.022551635280251503,
+          -0.029217366129159927,
+          0.0015946364728733897,
+          0.02794770337641239,
+          0.011729269288480282,
+          0.012998932041227818,
+          0.01797178015112877,
+          -0.029459208250045776,
+          0.01653585024178028,
+          -0.02264232560992241,
+          0.026164129376411438,
+          -0.014880754053592682,
+          0.04005485028028488,
+          -0.003034343710169196,
+          -0.02941386215388775,
+          -0.011812401935458183,
+          0.021599387750029564,
+          0.030214959755539894,
+          -0.03056260570883751,
+          0.030456800013780594,
+          0.03518780693411827,
+          0.006627943832427263,
+          0.007213651668280363,
+          0.02353411354124546,
+          0.03476458415389061,
+          0.05646977946162224,
+          -0.00728922663256526,
+          -0.03286009281873703,
+          -0.027932588011026382,
+          0.010346243157982826,
+          -0.0043493518605828285,
+          0.01748809777200222,
+          0.026073439046740532,
+          -0.008320827968418598,
+          -0.02283882163465023,
+          0.0023995121009647846,
+          0.008373730815947056,
+          0.02237025462090969,
+          -0.011200242675840855,
+          -0.02078317664563656,
+          0.00010415205179015175,
+          0.017216026782989502,
+          -0.07104067504405975,
+          -0.010784579440951347,
+          -0.01410232950001955,
+          -0.024864235892891884,
+          -0.024380555376410484,
+          -0.008948101662099361,
+          0.013150081969797611,
+          0.01469937339425087,
+          0.02447124570608139,
+          0.0008743104408495128,
+          -0.02051110565662384,
+          -0.0035520337987691164,
+          -0.01797178015112877,
+          0.02985219843685627,
+          -0.04546603187918663,
+          -0.022294679656624794,
+          0.026405969634652138,
+          0.01073167659342289,
+          -0.0017287824302911758,
+          0.009650951251387596,
+          -0.039268866181373596,
+          0.037001609802246094,
+          -0.018304310739040375,
+          -0.001512448419816792,
+          0.009356208145618439,
+          0.009227730333805084,
+          -0.017548557370901108,
+          0.02332250215113163,
+          0.007973182015120983,
+          -0.021750537678599358,
+          0.03146950900554657,
+          0.022143529728055,
+          0.014631356112658978,
+          0.011661251075565815,
+          -0.03295078128576279,
+          0.015764983370900154,
+          0.013429710641503334,
+          -0.00931842066347599,
+          0.05486758425831795,
+          0.018833335489034653,
+          -0.004228431731462479,
+          0.022899281233549118,
+          -0.023216696456074715,
+          0.011381623335182667,
+          0.01455578114837408,
+          -0.0037031841930001974,
+          0.029020871967077255,
+          0.02543860860168934,
+          0.022521406412124634,
+          -0.04894249141216278,
+          0.01937747746706009,
+          0.03219502791762352,
+          -0.00419064424932003,
+          -0.0698314756155014,
+          0.025952519848942757,
+          0.0018770986935123801,
+          0.01748809777200222,
+          0.017352063208818436,
+          0.02144823782145977,
+          0.01674746163189411,
+          0.002259697997942567,
+          0.043924298137426376,
+          -0.039057254791259766,
+          0.019044946879148483,
+          0.0010344353504478931,
+          -0.026859421283006668,
+          -0.014638913795351982,
+          -0.007379916962236166,
+          -0.01726137287914753,
+          -0.010580525733530521,
+          0.026723386719822884,
+          0.02049599029123783,
+          0.033676303923130035,
+          -0.01191064901649952,
+          0.028809260576963425,
+          -0.0014926099684089422,
+          0.00100326060783118,
+          -0.0124850207939744,
+          -0.00945445615798235,
+          -0.017442753538489342,
+          -0.016732346266508102,
+          0.003291299333795905,
+          0.0034613434690982103,
+          -0.03875495493412018,
+          0.01202401239424944,
+          -0.007024713326245546,
+          -0.01751832850277424,
+          0.023458538576960564,
+          -0.033887915313243866,
+          0.0400850772857666,
+          -0.015780098736286163,
+          -0.05078652501106262,
+          0.006786651909351349,
+          0.01837988570332527,
+          -0.030381225049495697,
+          0.00044494890607893467,
+          0.004428706131875515,
+          0.030668411403894424,
+          -0.0060875811614096165,
+          -0.01933213323354721,
+          0.001051439787261188,
+          0.002350388327613473,
+          0.024622395634651184,
+          -0.0022294679656624794,
+          -0.016898611560463905,
+          -0.006306749302893877,
+          0.0045609623193740845,
+          0.020888982340693474,
+          0.0007798414444550872,
+          -0.025015385821461678,
+          0.02170519344508648,
+          0.025483952835202217,
+          -0.014412187971174717,
+          -0.05244917795062065,
+          0.03824104368686676,
+          0.033706534653902054,
+          -0.007478164508938789,
+          -0.04226164519786835,
+          0.016006823629140854,
+          -0.029081331565976143,
+          0.01798689365386963,
+          0.00021491693041753024,
+          -0.00454206857830286,
+          -0.016248665750026703,
+          0.019014716148376465,
+          0.021599387750029564,
+          -0.013271003030240536,
+          -0.005369617138057947,
+          -0.042291875928640366,
+          -0.036487698554992676,
+          0.017140451818704605,
+          -0.01304427720606327,
+          0.00782203208655119,
+          -0.012326313182711601,
+          -0.02421429008245468,
+          0.009605606086552143,
+          -0.05263055860996246,
+          -0.04640316590666771,
+          0.0015011121286079288,
+          0.024894466623663902,
+          -0.0001595344947418198,
+          0.0009215449099428952,
+          -0.010172420181334019,
+          0.01961931772530079,
+          0.020375071093440056,
+          0.024894466623663902,
+          -0.0358830988407135,
+          -0.002492091851308942,
+          0.028113968670368195,
+          0.0014963886933401227,
+          -0.018153158947825432,
+          0.014517993666231632,
+          -0.025499068200588226,
+          -0.006265182979404926,
+          0.0008100715349428356,
+          -0.0326484814286232,
+          0.004504281096160412,
+          -0.005407404620200396,
+          -0.0013424042845144868,
+          0.010247995145618916,
+          0.0316811166703701,
+          -0.021145937964320183,
+          0.004625201225280762,
+          0.040447838604450226,
+          -0.0043040066957473755,
+          -0.013134967535734177,
+          0.008260367438197136,
+          0.017911318689584732,
+          -0.017125336453318596,
+          -0.022083070129156113,
+          0.02329227328300476,
+          -0.02379106916487217,
+          0.014805179089307785,
+          0.045828793197870255,
+          -0.012696631252765656,
+          0.006484351120889187,
+          0.042291875928640366,
+          0.02540837787091732,
+          0.013679108582437038,
+          -0.01423836499452591,
+          -0.008676031604409218,
+          0.012054242193698883,
+          -0.059099797159433365,
+          9.500037413090467e-05,
+          -0.005857076961547136,
+          0.022899281233549118,
+          -0.002246472518891096,
+          0.014087215065956116,
+          -0.021372662857174873,
+          0.007104067597538233,
+          0.03009403869509697,
+          -0.02333761751651764,
+          0.014381958171725273,
+          -0.057678982615470886,
+          0.016369584947824478,
+          0.012394330464303493,
+          0.012160047888755798,
+          -0.0045118387788534164,
+          -0.0124850207939744,
+          0.027207067236304283,
+          -0.007920279167592525,
+          0.009280633181333542,
+          0.01745786890387535,
+          -0.022324910387396812,
+          -0.053023550659418106,
+          0.017805512994527817,
+          -0.010323570109903812,
+          -0.030653296038508415,
+          0.007485722191631794,
+          -0.02167496271431446,
+          -0.021312203258275986,
+          -0.044438209384679794,
+          0.012046684511005878,
+          0.037666670978069305,
+          -0.016732346266508102,
+          -0.04274532571434975,
+          -0.0004893493023701012,
+          -0.022249335423111916,
+          -0.013822701759636402,
+          0.010746791027486324,
+          -0.03225548937916756,
+          0.020178575068712234,
+          0.018138045445084572,
+          0.048882029950618744,
+          -0.010316013358533382,
+          -0.004345573019236326,
+          0.02823488973081112,
+          -0.029217366129159927,
+          0.005097546149045229,
+          -0.02471308596432209,
+          -0.03446228429675102,
+          0.02888483554124832,
+          0.0050597586669027805,
+          0.029020871967077255,
+          0.015235957689583302,
+          -0.007988297380506992,
+          0.016974186524748802,
+          -0.021281972527503967,
+          -0.027690747752785683,
+          -0.018062468618154526,
+          -0.026980342343449593,
+          -0.04504281282424927,
+          0.028355810791254044,
+          -0.05734645202755928,
+          -0.019966963678598404,
+          0.028612766414880753,
+          0.009552703239023685,
+          -0.0021274415776133537,
+          0.005853298120200634,
+          -0.019453052431344986,
+          -0.036517929285764694,
+          -0.011374065652489662,
+          0.028794145211577415,
+          0.009598048403859138,
+          -0.010187535546720028,
+          0.005762607790529728,
+          0.009598048403859138,
+          -0.04407544806599617,
+          -0.016898611560463905,
+          -0.00010893454600591213,
+          0.013694223947823048,
+          -0.02170519344508648,
+          -0.04425682872533798,
+          -0.04211049526929855,
+          0.009386437945067883,
+          -0.02729775756597519,
+          0.007912722416222095,
+          -0.030411455780267715,
+          -0.04845881089568138,
+          -0.004073502495884895,
+          -0.0071909790858626366,
+          0.02382129803299904,
+          0.02584671415388584,
+          -2.5447581720072776e-05,
+          0.03201364725828171,
+          -0.032527558505535126,
+          0.011389181017875671,
+          0.006571262609213591,
+          -0.021538928151130676,
+          -0.009008562192320824,
+          -0.0022294679656624794,
+          -0.029897544533014297,
+          0.029292942956089973,
+          0.01913563720881939,
+          0.00495017459616065,
+          -0.0025941182393580675,
+          -0.027146607637405396,
+          -0.02149358205497265,
+          -0.05789059400558472,
+          0.014752276241779327,
+          0.05380953103303909,
+          -0.030502144247293472,
+          -0.01051250845193863,
+          -0.017835743725299835,
+          0.019740238785743713,
+          0.04827743023633957,
+          0.007950509898364544,
+          0.03149973601102829,
+          -0.023050431162118912,
+          0.009514915756881237,
+          0.006975589785724878,
+          0.011018862016499043,
+          -0.044861432164907455,
+          -0.0014614351093769073,
+          -0.013376807793974876,
+          -0.0042586615309119225,
+          -0.003671064740046859,
+          -0.015598718076944351,
+          -0.025000272318720818,
+          -0.0045005022548139095,
+          0.006911350879818201,
+          -0.0050710951909422874,
+          0.002527989912778139,
+          -0.018017124384641647,
+          0.03479481488466263,
+          -0.04017576947808266,
+          0.03827127441763878,
+          -0.006427669432014227,
+          0.004568520002067089,
+          0.04981916397809982,
+          -0.018077583983540535,
+          -0.007648208644241095,
+          -0.008033642545342445,
+          -0.0038524451665580273,
+          -0.01141185313463211,
+          0.010096845217049122,
+          -0.014608683064579964,
+          0.0017864084802567959,
+          0.020465759560465813,
+          -0.02584671415388584,
+          -0.020344840362668037,
+          -0.005841962061822414,
+          0.0031590426806360483,
+          -0.0405687615275383,
+          0.006170713808387518,
+          0.0012951696990057826,
+          0.030169613659381866,
+          0.014155232347548008,
+          -0.03128812834620476,
+          0.000974919879809022,
+          0.01565917767584324,
+          -0.012492578476667404,
+          -0.02330738678574562,
+          0.03225548937916756,
+          0.003327197628095746,
+          -0.004972847178578377,
+          0.003485905472189188,
+          0.01769970916211605,
+          -0.013732011429965496,
+          0.01863684132695198,
+          -0.006522138603031635,
+          0.012265852652490139,
+          0.017397407442331314,
+          -0.03618539869785309,
+          0.016173088923096657,
+          0.007950509898364544,
+          -0.019241442903876305,
+          0.008139447309076786,
+          0.019044946879148483,
+          -0.025499068200588226,
+          0.021145937964320183,
+          -0.02097967080771923,
+          0.016626540571451187,
+          -0.04537534341216087,
+          -0.008192350156605244,
+          0.02896041050553322,
+          0.009802102111279964,
+          0.0013943621888756752,
+          -0.022234220057725906,
+          -0.0013839706080034375,
+          0.04809604957699776,
+          0.005229802802205086,
+          -0.011525215581059456,
+          -0.011850189417600632,
+          0.00513911247253418,
+          0.00043975311564281583,
+          0.0009342982666566968,
+          -0.02753959782421589,
+          -0.04588925465941429,
+          -0.006393660791218281,
+          -0.036941152065992355,
+          0.0326484814286232,
+          0.027343101799488068,
+          -0.0037126310635358095,
+          -0.016868380829691887,
+          0.0189089123159647,
+          -0.0006636445759795606,
+          0.03282986208796501,
+          -0.007478164508938789,
+          0.04150589182972908,
+          -0.052842170000076294,
+          0.0018478132551535964,
+          -0.011517658829689026,
+          -0.02193191833794117,
+          -0.007406368386000395,
+          -0.014638913795351982,
+          -0.018002009019255638,
+          0.01892402581870556,
+          0.01083748135715723,
+          0.0031533746514469385,
+          0.016974186524748802,
+          -0.0028246226720511913,
+          0.0035274717956781387,
+          0.012409445829689503,
+          -0.012084471993148327,
+          -0.003589821280911565,
+          0.04458935931324959,
+          0.009530031122267246,
+          -0.00979454442858696,
+          -0.017321832478046417,
+          -0.0024864235892891884,
+          0.052116647362709045,
+          -0.008494650945067406,
+          0.021569158881902695,
+          0.03606447950005531,
+          0.007958066649734974,
+          0.03074398636817932,
+          -0.004685661755502224,
+          0.026678040623664856,
+          0.003858113195747137,
+          -0.019180981442332268,
+          -0.006310528144240379,
+          -0.020405299961566925,
+          -0.01340703759342432,
+          -0.005849519278854132,
+          -0.023186467587947845,
+          0.026058323681354523,
+          -0.006525917444378138,
+          -0.00215578218922019,
+          0.0006490019150078297,
+          -0.030623065307736397,
+          0.021418007090687752,
+          0.018334539607167244,
+          0.014722046442329884,
+          0.04099198058247566,
+          0.007164527662098408,
+          0.02403290942311287,
+          -0.002614901401102543,
+          -0.004761236719787121,
+          0.014004082418978214,
+          0.024818891659379005,
+          0.024380555376410484,
+          0.03606447950005531,
+          -0.005350723396986723,
+          0.028567420318722725,
+          0.032527558505535126,
+          -0.027464022859930992,
+          0.014419745653867722,
+          0.016883496195077896,
+          -0.006008227355778217,
+          -0.021614503115415573,
+          -0.0022559193894267082,
+          0.02073783054947853,
+          0.002458082977682352,
+          -0.00838128849864006,
+          0.0363667793571949,
+          -0.02498515695333481,
+          -0.015946364030241966,
+          0.0031514852307736874,
+          -0.03267871215939522,
+          0.02495492622256279,
+          0.0028246226720511913,
+          -0.00792783685028553,
+          -0.02608855441212654,
+          -0.0076595451682806015,
+          0.0029757730662822723,
+          -0.011381623335182667,
+          -0.0038543345872312784,
+          0.04189888387918472,
+          -0.006654395256191492,
+          0.01318031270056963,
+          -0.0022389148361980915,
+          0.004957732278853655,
+          -0.0058306255377829075,
+          0.028854606673121452,
+          0.0006863171583972871,
+          0.0049199447967112064,
+          -0.022098183631896973,
+          -0.007580191362649202,
+          0.0049652899615466595,
+          0.015538258478045464,
+          0.027826784178614616,
+          0.019649548456072807,
+          0.010127075016498566,
+          0.027010571211576462,
+          -0.02172030881047249,
+          0.00020688706717919558,
+          -0.017397407442331314,
+          0.050726063549518585,
+          0.036034248769283295,
+          -0.023005086928606033,
+          -0.012787321582436562,
+          0.010323570109903812,
+          -0.03594356030225754,
+          0.012137374840676785,
+          0.02445613034069538,
+          -0.0005446136929094791,
+          -0.0065863775089383125,
+          0.0048632631078362465,
+          -0.008101659826934338,
+          -0.026028094813227654,
+          -0.00919750053435564,
+          0.02654200606048107,
+          0.0002616790879983455,
+          0.0062425103969872,
+          0.011827516369521618,
+          0.016173088923096657,
+          -0.017442753538489342,
+          -0.029338287189602852,
+          0.000420150812715292,
+          0.037424832582473755,
+          0.01109443698078394,
+          0.004372024443000555,
+          0.019286787137389183,
+          -0.009280633181333542,
+          -0.014140116982161999,
+          0.02474331669509411,
+          0.019906504079699516,
+          -0.0013216211227700114,
+          0.004938838537782431,
+          -0.007973182015120983,
+          0.00968118105083704,
+          -0.059553246945142746,
+          0.03258801996707916,
+          0.008570225909352303,
+          0.017079992219805717,
+          -0.0009522473555989563,
+          -0.007988297380506992,
+          0.028159314766526222,
+          -0.0003171795979142189,
+          0.008056314662098885,
+          0.016943957656621933,
+          -0.030895136296749115,
+          0.008366173133254051,
+          -0.018531035631895065,
+          -0.0010495503665879369,
+          0.019286787137389183,
+          0.01013463269919157,
+          -0.013021604157984257,
+          -0.012636170722544193,
+          -0.009371323511004448,
+          0.023443423211574554,
+          -0.007153191603720188,
+          0.015077250078320503,
+          0.0004227487079333514,
+          -0.009734083898365498,
+          -0.02241560071706772,
+          0.006733749061822891,
+          0.026647809892892838,
+          -0.025090962648391724,
+          0.01700441725552082,
+          -0.01529641728848219,
+          0.01435928512364626,
+          -0.029247596859931946,
+          0.005486758425831795,
+          0.04507303982973099,
+          0.011812401935458183,
+          -0.040659449994564056,
+          -0.010890384204685688,
+          -0.0003204860258847475,
+          -0.01154788862913847,
+          -0.007617978844791651,
+          0.012318755500018597,
+          0.05692322924733162,
+          -0.03051725961267948,
+          0.008532438427209854,
+          0.028975525870919228,
+          -0.014034312218427658,
+          0.016671886667609215,
+          -0.019966963678598404,
+          0.033222850412130356,
+          0.006850890815258026,
+          0.008539996109902859,
+          0.001844034530222416,
+          -0.0016796585405245423,
+          -0.0096585089340806,
+          0.0041037327609956264,
+          -0.018954256549477577,
+          -0.017064876854419708,
+          0.010172420181334019,
+          0.013429710641503334,
+          0.006714855320751667,
+          -0.00838128849864006,
+          -0.027222182601690292,
+          -0.001923388452269137,
+          0.019710008054971695,
+          -0.016671886667609215,
+          -0.032497331500053406,
+          0.014404630288481712,
+          0.012855338864028454,
+          -0.015538258478045464,
+          0.0391177162528038,
+          0.0041490779258310795,
+          0.01037647295743227,
+          0.015167939476668835,
+          -0.006321864202618599,
+          -0.008736491203308105,
+          0.024441014975309372,
+          0.0170497614890337,
+          -0.004334236960858107,
+          -0.016278894618153572,
+          0.015455125831067562,
+          0.030623065307736397,
+          0.013331462629139423,
+          0.005947767291218042,
+          -0.012401888146996498,
+          0.0378178246319294,
+          -0.004390918184071779,
+          -0.011555446311831474,
+          -0.03171134740114212,
+          -0.014034312218427658,
+          -0.02049599029123783,
+          -0.0033555382397025824,
+          -0.0039412458427250385,
+          0.012273410335183144,
+          -0.024531705304980278,
+          0.004594971425831318,
+          0.008509766310453415,
+          0.00636343052610755,
+          0.08131889998912811,
+          0.01798689365386963,
+          0.007633093744516373,
+          0.023413192480802536,
+          -0.01154788862913847,
+          0.029474321752786636,
+          -0.02708614617586136,
+          0.017427638173103333,
+          -0.013278559781610966,
+          0.011109552346169949,
+          -0.005350723396986723,
+          0.02590717375278473,
+          0.02332250215113163,
+          -0.0007576412172056735,
+          0.002580892527475953,
+          0.013225657865405083,
+          -0.0034084408544003963,
+          -0.019740238785743713,
+          0.012076914310455322,
+          0.030880020931363106,
+          0.03654816001653671,
+          0.012961144559085369,
+          0.02169007807970047,
+          0.008441748097538948,
+          -0.01049739308655262,
+          -0.02026926539838314,
+          -0.006866005714982748,
+          0.005524545907974243,
+          0.021614503115415573,
+          0.008358615450561047,
+          0.009582933969795704,
+          0.00642011221498251,
+          -0.008751606568694115,
+          -0.006091360002756119,
+          0.00636343052610755,
+          -0.014147674664855003,
+          0.03225548937916756,
+          0.005977997090667486,
+          -0.018092699348926544,
+          -0.028809260576963425,
+          -0.009137040004134178,
+          -0.01605216972529888,
+          -0.033041469752788544,
+          -0.01015730481594801,
+          0.008547553792595863,
+          -0.03482504561543465,
+          -0.008471978828310966,
+          0.01051250845193863,
+          -0.018077583983540535,
+          -0.03775736317038536,
+          0.014744718559086323,
+          -0.028129084035754204,
+          0.023911988362669945,
+          -0.024622395634651184,
+          0.010089287534356117,
+          0.002883193315938115,
+          -0.006457899697124958,
+          -0.011563003063201904,
+          -0.0022823705803602934,
+          -0.002686697756871581,
+          -0.0020103000570088625,
+          -0.03140904754400253,
+          -0.014321497641503811,
+          -0.017367178574204445,
+          -0.002590339398011565,
+          -0.033706534653902054,
+          0.025090962648391724,
+          0.030411455780267715,
+          0.012658843770623207,
+          0.007323235739022493,
+          -0.008751606568694115,
+          -0.01841011457145214,
+          -0.007874934002757072,
+          -0.033011242747306824,
+          -0.0064805722795426846,
+          -0.05928117781877518,
+          -0.010905499570071697,
+          -0.008932987228035927,
+          0.01772993803024292,
+          -0.013202984817326069,
+          -0.010814809240400791,
+          -0.02100990153849125,
+          -0.04099198058247566,
+          -0.019014716148376465,
+          0.007202315144240856,
+          -0.03379722312092781,
+          0.05925094708800316,
+          0.013210542500019073,
+          0.006968032103031874,
+          -0.024577051401138306,
+          0.010051500052213669,
+          -0.000378112104954198,
+          0.0096585089340806,
+          -0.005902422126382589,
+          -0.010913057252764702,
+          0.013339020311832428,
+          0.01698930189013481,
+          -0.008517323061823845,
+          0.0012687183916568756,
+          0.016173088923096657,
+          0.01682303659617901,
+          0.02614901401102543,
+          0.022249335423111916,
+          0.019014716148376465,
+          0.032769400626420975,
+          -0.029232481494545937,
+          0.06293901801109314,
+          -0.011479870416224003,
+          0.005369617138057947,
+          0.016928842291235924,
+          -0.019211212173104286,
+          0.029791738837957382,
+          -0.015251072123646736,
+          -0.02143312245607376,
+          0.00335175939835608,
+          0.005607679020613432,
+          0.013943621888756752,
+          -0.05217710882425308,
+          0.033434461802244186,
+          0.03963162750005722,
+          0.017352063208818436,
+          -0.00019826677453238517,
+          0.009174827486276627,
+          -0.0190751776099205,
+          0.038120124489068985,
+          -0.007342129480093718,
+          0.019997194409370422,
+          0.03727368265390396,
+          -0.0008039310341700912,
+          0.029292942956089973,
+          -0.012326313182711601,
+          0.02894529700279236,
+          -0.006605271250009537,
+          0.012666401453316212,
+          0.01198622491210699,
+          -0.009325978346168995,
+          -0.023262042552232742,
+          -0.015326648019254208,
+          0.0017731827683746815,
+          0.0012384883593767881,
+          -0.02843138575553894,
+          0.030245188623666763,
+          -0.019241442903876305,
+          -0.0007533901371061802,
+          0.006095138844102621,
+          -0.017412522807717323,
+          0.000406925129937008,
+          -0.0006593934958800673,
+          0.009507358074188232,
+          -0.005441413260996342,
+          0.010686331428587437,
+          0.01584055833518505,
+          -0.011842631734907627,
+          -0.02240048535168171,
+          0.01701953262090683,
+          -0.010066614486277103,
+          0.006968032103031874,
+          -0.03310193121433258,
+          -0.003370653372257948,
+          -0.03775736317038536,
+          -0.02847672998905182,
+          0.03948047757148743,
+          -0.0163242407143116,
+          -0.02265744097530842,
+          0.021599387750029564,
+          -0.03524826839566231,
+          -0.010111959651112556,
+          0.021508697420358658,
+          -0.010988632217049599,
+          -0.011736826039850712,
+          0.027554713189601898,
+          0.002395733492448926,
+          -0.009401553310453892,
+          -0.04990985244512558,
+          -0.026012979447841644,
+          0.004870820790529251,
+          0.011215358041226864,
+          -0.013308790512382984,
+          0.04546603187918663,
+          -0.01479762140661478,
+          -0.042352333664894104,
+          -0.015228400006890297,
+          0.01529641728848219,
+          -0.0099003491923213,
+          -0.019468167796730995,
+          -0.014230807311832905,
+          0.009726526215672493,
+          0.0024070697836577892,
+          -0.03149973601102829,
+          -0.01108688022941351,
+          0.02169007807970047,
+          -0.009847446344792843,
+          0.011744383722543716,
+          -0.03585286810994148,
+          -0.021856343373656273,
+          -0.01844034530222416,
+          0.020677370950579643,
+          0.029202252626419067,
+          -0.00115252158138901,
+          0.010368915274739265,
+          0.011895534582436085,
+          0.024652626365423203,
+          0.010452047921717167,
+          -0.0195588581264019,
+          0.015183054842054844,
+          -0.016037054359912872,
+          0.04032691940665245,
+          -0.0027830563485622406,
+          0.004417369607836008,
+          -0.03355538100004196,
+          0.0029323173221200705,
+          0.009303305298089981,
+          0.020223919302225113,
+          -0.009091694839298725,
+          0.03821081295609474,
+          -0.024335209280252457,
+          0.002350388327613473,
+          -0.009514915756881237,
+          0.01352040097117424,
+          0.03727368265390396,
+          0.002677250886335969,
+          -0.005146670155227184,
+          -0.007164527662098408,
+          0.019483283162117004,
+          0.024818891659379005,
+          -0.03760621324181557,
+          -0.013694223947823048,
+          0.039027027785778046,
+          0.00967362429946661,
+          -0.00875916425138712,
+          0.00459875026717782,
+          0.014737160876393318,
+          -0.023050431162118912,
+          0.017412522807717323,
+          0.015364435501396656,
+          0.009023677557706833,
+          -0.01889379695057869,
+          0.017352063208818436,
+          -0.016868380829691887,
+          -0.011419410817325115,
+          0.02753959782421589,
+          0.026285050436854362,
+          -0.025483952835202217,
+          0.016913726925849915,
+          -0.02401779405772686,
+          0.028068624436855316,
+          0.002153892768546939,
+          0.002055644989013672,
+          0.00874404888600111,
+          -0.004096175078302622,
+          -0.02867322601377964,
+          0.0012375436490401626,
+          0.006571262609213591,
+          0.006616607774049044,
+          0.0036238301545381546,
+          0.010028827004134655,
+          0.014926099218428135,
+          -0.008449305780231953,
+          0.008434190414845943,
+          0.00838884525001049,
+          0.02917202189564705,
+          -0.009356208145618439,
+          -0.027373332530260086,
+          -0.006102696061134338,
+          -0.0029512110631912947,
+          0.028506960719823837,
+          0.01292335707694292,
+          -0.00814700499176979,
+          0.02167496271431446,
+          0.09613163769245148,
+          0.006185829173773527,
+          -0.020163459703326225,
+          -0.02353411354124546,
+          -0.013437268324196339,
+          0.005891085602343082,
+          -0.006684625055640936,
+          0.009371323511004448,
+          0.004704555496573448,
+          -0.013815144076943398,
+          -0.009862561710178852,
+          0.03748529404401779,
+          -0.002057534409686923,
+          -0.008298155851662159,
+          -0.012885569594800472,
+          0.01841011457145214,
+          0.00780691672116518,
+          -0.008993446826934814,
+          0.004145299084484577,
+          -0.02519676648080349,
+          -0.03171134740114212,
+          -0.010656101629137993,
+          -0.005123997572809458,
+          0.02329227328300476,
+          -0.006423891056329012,
+          0.019014716148376465,
+          0.0054225195199251175,
+          0.012099587358534336,
+          -0.009991039521992207,
+          -0.0012252627639099956,
+          -0.018939141184091568,
+          0.019060062244534492,
+          -0.009371323511004448,
+          -0.0009862561710178852,
+          -0.01176705677062273,
+          0.003833551425486803,
+          -0.019634433090686798,
+          -0.0015766873257234693,
+          -0.029066216200590134,
+          0.016475390642881393,
+          -0.0054225195199251175,
+          0.0069491383619606495,
+          0.011842631734907627,
+          -0.012401888146996498,
+          0.0007992075989022851,
+          0.006166934967041016,
+          -0.0027981712482869625,
+          -0.03703184053301811,
+          0.011797286570072174,
+          0.007274111732840538,
+          0.02147846855223179,
+          -0.020405299961566925,
+          -0.034371595829725266,
+          -0.012076914310455322,
+          -0.0017467314610257745,
+          0.013369250111281872,
+          0.014109887182712555,
+          0.01059564109891653,
+          -0.025952519848942757,
+          -0.008426633663475513,
+          0.018153158947825432,
+          0.012144932523369789,
+          0.023141121491789818,
+          -0.006964253261685371,
+          0.041626811027526855,
+          -0.001176138874143362,
+          0.024335209280252457,
+          -0.006695961579680443,
+          -0.0018657624023035169,
+          -0.01934724859893322,
+          0.00885741226375103,
+          0.031348586082458496,
+          0.010860154405236244,
+          -0.015303974971175194,
+          -0.022944627329707146,
+          0.00953758880496025,
+          0.011245587840676308,
+          -0.006231173872947693,
+          0.009038791991770267,
+          -0.024365440011024475,
+          -0.00780691672116518,
+          0.0011780281784012914,
+          0.0012687183916568756,
+          0.021085476502776146,
+          -0.033918142318725586,
+          -0.01984604448080063,
+          -0.008373730815947056,
+          -0.014842966571450233,
+          -0.005003077443689108,
+          0.002580892527475953,
+          0.004020600114017725,
+          0.01200889702886343,
+          -0.012568153440952301,
+          -0.002246472518891096,
+          -0.024773545563220978,
+          0.00021161051699891686,
+          0.019271671772003174,
+          -0.007769129239022732,
+          -0.023201582953333855,
+          0.012658843770623207,
+          -0.016838151961565018,
+          0.010300897993147373,
+          0.026209475472569466,
+          -0.010399146005511284,
+          0.014412187971174717,
+          -0.015961479395627975,
+          -0.01936236210167408,
+          0.01328611746430397,
+          -0.005709705408662558,
+          -0.002926649060100317,
+          0.009975924156606197,
+          0.005796616896986961,
+          -0.00010568717698333785,
+          0.02982196770608425,
+          0.034432053565979004,
+          -0.023262042552232742,
+          0.013233215548098087,
+          -0.008124332875013351,
+          0.006427669432014227,
+          -0.0009295747731812298,
+          -0.02212841436266899,
+          0.014344170689582825,
+          0.014132560230791569,
+          -0.0022237999364733696,
+          -0.002837848151102662,
+          0.023624803870916367,
+          -0.0008270759717561305,
+          -0.01027822494506836,
+          0.031590428203344345,
+          0.002308822004124522,
+          0.0005960993003100157,
+          0.009016119875013828,
+          0.021508697420358658,
+          0.04171750321984291,
+          0.012658843770623207,
+          -0.00953758880496025,
+          0.0013452382991090417,
+          -0.005294041708111763,
+          0.009325978346168995,
+          -0.012069357559084892,
+          0.02072271518409252,
+          0.00896321702748537,
+          0.0244863610714674,
+          -0.003982812166213989,
+          -0.004791466984897852,
+          -0.02001230977475643,
+          -0.01537954993546009,
+          0.00814700499176979,
+          0.033192623406648636,
+          -0.000691512948833406,
+          -0.015674293041229248,
+          -0.0030362331308424473,
+          0.033041469752788544,
+          -0.0033668745309114456,
+          -0.030955595895648003,
+          -0.03122766688466072,
+          -0.00484059052541852,
+          -0.0019167757127434015,
+          0.012915799394249916,
+          -0.0017618464771658182,
+          0.007172085344791412,
+          0.015349320136010647,
+          -0.04576833173632622,
+          -0.046977534890174866,
+          0.015901019796729088,
+          -0.010777021758258343,
+          0.007247660309076309,
+          0.019407708197832108,
+          -0.033948373049497604,
+          -0.008789394050836563,
+          0.0060271210968494415,
+          0.012787321582436562,
+          0.008177234791219234,
+          0.006310528144240379,
+          0.015183054842054844,
+          -0.027872128412127495,
+          -0.019286787137389183,
+          0.020103000104427338,
+          0.0011317384196445346,
+          -0.004341794643551111,
+          0.0024788661394268274,
+          0.01132872048765421,
+          -0.009152155369520187,
+          0.01352040097117424,
+          0.0006357762613333762,
+          0.019301902502775192,
+          -0.011215358041226864,
+          -0.006333200726658106,
+          -0.010920614004135132,
+          0.0016258112154901028,
+          0.014147674664855003,
+          0.01724625751376152,
+          0.0027887243777513504,
+          0.006831996608525515,
+          0.001246045925654471,
+          0.00991546455770731,
+          -0.015016789548099041,
+          -0.0006310528260655701,
+          0.01071656122803688,
+          0.014019196853041649,
+          0.0035161355044692755,
+          0.009718968532979488,
+          0.006605271250009537,
+          0.0025393262039870024,
+          0.0024335209745913744,
+          -0.025967633351683617,
+          -0.009605606086552143,
+          -0.013905834406614304,
+          0.011804844252765179,
+          -0.004402254708111286,
+          0.0057852803729474545,
+          -0.010293340310454369,
+          -0.012968702241778374,
+          0.020118115469813347,
+          -0.0048292544670403,
+          -0.010520066134631634,
+          -0.026466431096196175,
+          -0.039571166038513184,
+          0.027630288153886795,
+          -0.009492243640124798,
+          0.0037579762283712626,
+          -0.01039158832281828,
+          0.022793475538492203,
+          0.0052713691256940365,
+          0.007814474403858185,
+          0.006166934967041016,
+          0.0067375279031693935,
+          -0.0031873835250735283,
+          -0.02097967080771923,
+          -0.0024316315539181232,
+          -0.01703464612364769,
+          0.017533443868160248,
+          -0.010210207663476467,
+          0.021735424175858498,
+          0.015719639137387276,
+          0.00023640862491447479,
+          0.006268961820751429,
+          0.02613389864563942,
+          0.012757091782987118,
+          0.0160824004560709,
+          -0.025816483423113823,
+          -0.0037749805487692356,
+          -0.01035380084067583,
+          0.024652626365423203,
+          -0.005505652166903019,
+          -0.00023735332069918513,
+          0.014865638688206673,
+          -0.022264450788497925,
+          -0.009870119392871857,
+          0.012878011912107468,
+          -0.006722413003444672,
+          -0.02234002575278282,
+          -0.001692884135991335,
+          -0.019664663821458817,
+          -0.007198536302894354,
+          -0.022143529728055,
+          0.005312935449182987,
+          0.018984487280249596,
+          -0.008721376769244671,
+          0.016399815678596497,
+          0.02915690653026104,
+          -0.006695961579680443,
+          0.024818891659379005,
+          0.0160824004560709,
+          0.0010372694814577699,
+          -0.00046148098772391677,
+          -0.005894864443689585,
+          0.004477829672396183,
+          -0.009333535097539425,
+          -0.012817551381886005,
+          -0.01748809777200222,
+          0.009061465039849281,
+          -0.018304310739040375,
+          0.015470240265130997,
+          0.03473435714840889,
+          -0.004020600114017725,
+          0.0042586615309119225,
+          -0.022959740832448006,
+          -0.016611425206065178,
+          -0.004447599872946739,
+          -0.014850524254143238,
+          0.06517603993415833,
+          -0.004795245360583067,
+          -0.014502878300845623,
+          -0.012666401453316212,
+          0.0036672858987003565,
+          -0.0028869721572846174,
+          -0.0036691753193736076,
+          -0.012182720005512238,
+          -0.015137709677219391,
+          -0.006964253261685371,
+          0.022445829585194588,
+          -0.032769400626420975,
+          0.005475422367453575,
+          0.030728871002793312,
+          -0.005641687661409378,
+          -0.0037579762283712626,
+          0.0023787289392203093,
+          -0.017397407442331314,
+          -0.012447233311831951,
+          -0.013565746136009693,
+          0.0008299099863506854,
+          0.0005710650002583861,
+          -0.0033215293660759926,
+          -0.020420415326952934,
+          0.01913563720881939,
+          -0.00850220862776041,
+          0.005309156607836485,
+          -0.029716163873672485,
+          -0.0055283247493207455,
+          0.012651286087930202,
+          0.0045005022548139095,
+          -0.019891388714313507,
+          0.013225657865405083,
+          -0.017790399491786957,
+          -0.0017023311229422688,
+          0.009356208145618439,
+          -0.01768459379673004,
+          -0.007391253020614386,
+          -0.024879351258277893,
+          -0.045798562467098236,
+          0.0036937373224645853,
+          -0.005743714049458504,
+          0.04011530801653862,
+          0.010452047921717167,
+          0.025816483423113823,
+          -0.011751941405236721,
+          -0.01798689365386963,
+          -0.008275482803583145,
+          -0.010897941887378693,
+          -0.015387107618153095,
+          -0.014193019829690456,
+          0.005683253984898329,
+          0.0029474322218447924,
+          -0.004885935690253973,
+          0.024380555376410484,
+          -0.006930244620889425,
+          -0.02188657410442829,
+          -0.003960139583796263,
+          -0.022037724032998085,
+          -0.01682303659617901,
+          0.006000669673085213,
+          -0.001827030093409121,
+          0.008630686439573765,
+          -0.005592563655227423,
+          -0.004398475866764784,
+          -0.016414931043982506,
+          0.00762931490316987,
+          0.020647140219807625,
+          0.010897941887378693,
+          -0.03691092133522034,
+          0.026511775329709053,
+          0.005849519278854132,
+          0.0007633093628101051,
+          0.017624134197831154,
+          0.004904829431325197,
+          0.007255217991769314,
+          0.01792643405497074,
+          -0.024773545563220978,
+          -0.004016821272671223,
+          -0.011880419217050076,
+          -0.025045616552233696,
+          -0.00943934079259634,
+          0.007799359038472176,
+          -0.020178575068712234,
+          -0.015689408406615257,
+          0.008366173133254051,
+          -0.02167496271431446,
+          0.01443486101925373,
+          -0.03503665700554848,
+          0.02026926539838314,
+          0.037878282368183136,
+          0.0072967843152582645,
+          0.0190751776099205,
+          0.002031083218753338,
+          0.00969629641622305,
+          -0.0014718268066644669,
+          -0.008328385651111603,
+          -0.02191680483520031,
+          0.01340703759342432,
+          -0.009257960133254528,
+          -0.003017339389771223,
+          -0.0030456800013780594,
+          0.00804120022803545,
+          -0.0009701964445412159,
+          -0.010127075016498566,
+          -0.0026111227925866842,
+          -0.015092364512383938,
+          0.008320827968418598,
+          -0.005660581402480602,
+          0.032315950840711594,
+          0.0008440803503617644,
+          0.018062468618154526,
+          0.0015332315815612674,
+          0.031620658934116364,
+          0.01863684132695198,
+          0.021841228008270264,
+          -0.0016891054110601544,
+          -0.008298155851662159,
+          0.00850220862776041,
+          -0.0019630654715001583,
+          -0.019906504079699516,
+          -0.01700441725552082,
+          -0.0028170649893581867,
+          0.0033196399454027414,
+          0.0006197165348567069,
+          0.007572633679956198,
+          0.006125368643552065,
+          -0.024093369022011757,
+          -0.01129093300551176,
+          0.003548254957422614,
+          0.0007727562915533781,
+          -0.005600121337920427,
+          0.009983481839299202,
+          -0.020647140219807625,
+          -0.0044060335494577885,
+          0.025317687541246414,
+          0.019755354151129723,
+          -0.025831598788499832,
+          -0.01549291331321001,
+          -0.005660581402480602,
+          0.0007500837091356516,
+          -0.018787991255521774,
+          -0.014389515854418278,
+          0.0037390822544693947,
+          -0.007504615932703018,
+          0.006779094226658344,
+          -0.01772993803024292,
+          -0.009514915756881237,
+          -0.0051202187314629555,
+          -0.030154500156641006,
+          0.012326313182711601,
+          0.013694223947823048,
+          0.030245188623666763,
+          -0.008676031604409218,
+          -0.01635446958243847,
+          -0.029323171824216843,
+          -0.006336979568004608,
+          -0.004765015561133623,
+          0.0033007462043315172,
+          0.004659210331737995,
+          -0.015115037560462952,
+          -0.025468837469816208,
+          -0.013376807793974876,
+          0.0063898819498717785,
+          -0.020571565255522728,
+          0.015251072123646736,
+          -0.008660916239023209,
+          0.03146950900554657,
+          -0.017805512994527817,
+          0.001601249212399125,
+          0.01222806517034769,
+          0.0005838182987645268,
+          0.007565075997263193,
+          0.0014954439830034971,
+          0.008071430027484894,
+          0.021130822598934174,
+          0.00826792512089014,
+          -0.01841011457145214,
+          0.007122961338609457,
+          0.006423891056329012,
+          -0.019422823563218117,
+          -0.0005328050465323031,
+          0.02052622102200985,
+          0.0030966931954026222,
+          0.00015670042193960398,
+          0.0021595610305666924,
+          0.004016821272671223,
+          0.01674746163189411,
+          -0.010436933487653732,
+          0.02495492622256279,
+          -0.00806387234479189,
+          -0.02732798643410206,
+          -0.01469181664288044,
+          -0.004077281337231398,
+          -0.03104628622531891,
+          -0.01108688022941351,
+          0.0009371322812512517,
+          -0.001392472768202424,
+          -0.006934023462235928,
+          0.0020443086978048086,
+          0.023866644129157066,
+          -0.003994148690253496,
+          -0.0021671184804290533,
+          0.01912052184343338,
+          0.09897326678037643,
+          0.002624348271638155,
+          -0.007746456656605005,
+          0.00437958212569356,
+          -0.011260703206062317,
+          -0.00647301459684968,
+          -0.018833335489034653,
+          -0.04924479126930237,
+          0.0016125855036079884,
+          1.3513492376659997e-05,
+          -0.004247325472533703,
+          -0.017125336453318596,
+          -0.013822701759636402,
+          0.013467498123645782,
+          -0.0060875811614096165,
+          0.01306694932281971,
+          -0.0052222455851733685,
+          0.0032686267513781786,
+          0.02329227328300476,
+          0.017881089821457863,
+          -0.006639279890805483,
+          0.009628279134631157,
+          -0.0029058658983558416,
+          0.008955659344792366,
+          0.018032239750027657,
+          0.014193019829690456,
+          -0.00769733265042305,
+          0.011661251075565815,
+          0.004946395754814148,
+          0.03981300815939903,
+          -0.007308120373636484,
+          -0.016928842291235924,
+          -0.00953758880496025,
+          -0.0009928690269589424,
+          -0.0053620594553649426,
+          0.011238030157983303,
+          -0.002766051795333624,
+          -0.016838151961565018,
+          -0.0017174461390823126,
+          -0.00780691672116518,
+          0.014306383207440376,
+          0.01117756962776184,
+          0.020163459703326225,
+          0.011457198299467564,
+          -0.002864299574866891,
+          0.028053509071469307,
+          -0.006129147484898567,
+          5.936785237281583e-05,
+          -0.013868046924471855,
+          0.004825475625693798,
+          -0.018017124384641647,
+          -0.002760383766144514,
+          0.005891085602343082,
+          -0.005078652407974005,
+          -0.0011563003063201904,
+          -0.0096585089340806,
+          0.005709705408662558,
+          -0.0027868349570780993,
+          0.010149747133255005,
+          -0.0036200513131916523,
+          -0.0018497026758268476,
+          0.009666066616773605,
+          0.00722120888531208,
+          -0.008056314662098885,
+          0.00922017265111208,
+          -0.021342432126402855,
+          -0.0007930670981295407,
+          -0.0074025895446538925,
+          0.003123144619166851,
+          -0.0019989637657999992,
+          -0.0030192285776138306,
+          -0.005569891072809696,
+          0.02031460963189602,
+          0.0017769616097211838,
+          -0.007980739697813988,
+          -0.022264450788497925,
+          0.022113298997282982,
+          -0.0031590426806360483,
+          0.014601126313209534,
+          -0.008827181532979012,
+          0.008925429545342922,
+          0.0035161355044692755,
+          -0.003438670886680484,
+          -0.002607343951240182,
+          -0.0018232513684779406,
+          -0.00361060444265604,
+          0.004307785537093878,
+          -0.001872375258244574,
+          0.0004959621583111584,
+          0.02311089262366295,
+          0.0009701964445412159,
+          -0.0027509366627782583,
+          0.012658843770623207,
+          0.006318085361272097,
+          0.0017004417022690177,
+          0.023881759494543076,
+          -0.008592898957431316,
+          -0.007727562915533781,
+          0.00012221925135236233,
+          -0.007258996833115816,
+          0.031590428203344345,
+          -0.009749199263751507,
+          0.00227103428915143,
+          0.020798292011022568,
+          -0.010285782627761364,
+          -0.003967697266489267,
+          0.003967697266489267,
+          -0.015553372912108898,
+          0.020344840362668037,
+          -0.023655032739043236,
+          0.022566750645637512,
+          0.012001339346170425,
+          -0.021372662857174873,
+          0.013913392089307308,
+          -0.00045439580571837723,
+          -0.013021604157984257,
+          0.006370988208800554,
+          0.007980739697813988,
+          -0.0420198030769825,
+          -0.008222579956054688,
+          -0.007882491685450077,
+          -0.03340423107147217,
+          0.009635835886001587,
+          -0.016430046409368515,
+          -0.0064956871792674065,
+          -0.0034084408544003963,
+          -0.006439005956053734,
+          -0.002450525527819991,
+          -0.02261209487915039,
+          -0.02327715791761875,
+          0.00642011221498251,
+          -0.02732798643410206,
+          -0.008184792473912239,
+          -0.0003235562762711197,
+          -0.002083985833451152,
+          -0.005668139085173607,
+          0.0052260239608585835,
+          0.016475390642881393,
+          0.0034821266308426857,
+          0.014049426652491093,
+          -0.005448970943689346,
+          0.011449640616774559,
+          -0.005626572761684656,
+          -0.01260594092309475,
+          -0.002288038842380047,
+          0.0027414897922426462,
+          -0.014011639170348644,
+          0.004251104313880205,
+          -0.008139447309076786,
+          -0.0060724662616848946,
+          0.012681515887379646,
+          -0.018939141184091568,
+          0.0018100256565958261,
+          0.008766721934080124,
+          7.917681796243414e-05,
+          -0.01258326880633831,
+          -0.028401155024766922,
+          0.00256955623626709,
+          -0.0054149623028934,
+          -0.002854852704331279,
+          0.00033772661117836833,
+          0.012417003512382507,
+          0.011540330946445465,
+          0.010701446793973446,
+          -0.002269144868478179,
+          0.007081395015120506,
+          0.028975525870919228,
+          0.013376807793974876,
+          -0.0074630496092140675,
+          0.0002914368233177811,
+          0.0022502511274069548,
+          0.0068886782974004745,
+          -0.020178575068712234,
+          -0.010323570109903812,
+          -0.0272372979670763,
+          -0.008766721934080124,
+          0.0014708820963278413,
+          0.0013594086049124599,
+          0.0244863610714674,
+          0.007493279874324799,
+          0.0018279748037457466,
+          -0.0028095075394958258,
+          0.010520066134631634,
+          0.007149412762373686,
+          0.0048028030432760715,
+          0.007009598426520824,
+          -0.018017124384641647,
+          -0.011593233793973923,
+          0.02146335318684578,
+          -0.0025884502101689577,
+          0.005543440114706755,
+          -0.0015200059860944748,
+          -0.0038203257136046886,
+          -0.004285112954676151,
+          0.01961931772530079,
+          -0.005671917926520109,
+          0.00012599800538737327,
+          0.029444092884659767,
+          -0.0035350292455404997,
+          0.0036238301545381546,
+          -0.006397439632564783,
+          -0.007224987726658583,
+          -0.04002461954951286,
+          0.01633935607969761,
+          -0.006983147002756596,
+          0.009431783109903336,
+          -0.010119517333805561,
+          0.009817216545343399,
+          0.010988632217049599,
+          0.0045609623193740845,
+          -0.010066614486277103,
+          0.0023995121009647846,
+          0.015047019347548485,
+          -0.011147339828312397,
+          -0.008313270285725594,
+          0.02100990153849125,
+          -0.006370988208800554,
+          -0.001745786750689149,
+          -0.0019177203066647053,
+          0.011948436498641968,
+          0.0027452686335891485,
+          0.018319424241781235,
+          0.005123997572809458,
+          -0.012076914310455322,
+          -0.0012970591196790338,
+          0.010308455675840378,
+          0.001875209272839129,
+          -0.006775315385311842,
+          0.005894864443689585,
+          -0.021554043516516685,
+          -0.0011270149843767285,
+          0.0018024682067334652,
+          0.023428307846188545,
+          0.005078652407974005,
+          0.026889652013778687,
+          -0.00035520337405614555,
+          0.0012167604872956872,
+          0.007727562915533781,
+          -0.007092731073498726,
+          -0.027146607637405396,
+          -0.01282510906457901,
+          -0.00828304048627615,
+          0.00018822944548446685,
+          0.0023749503307044506,
+          0.021735424175858498,
+          0.01841011457145214,
+          -0.02167496271431446,
+          0.007935394532978535,
+          -0.0047574578784406185,
+          -2.664320527401287e-05,
+          0.00888008438050747,
+          -0.00601200619712472,
+          0.003933688625693321,
+          0.00038921221857890487,
+          -0.0014944992726668715,
+          -0.007784244138747454,
+          -0.00886496901512146,
+          -0.0027112597599625587,
+          0.012084471993148327,
+          -0.003540697507560253,
+          0.008713819086551666,
+          -0.017654363065958023,
+          0.012386772781610489,
+          -0.008479535579681396,
+          0.03428090363740921,
+          0.010459605604410172,
+          -0.009061465039849281,
+          -0.013724453747272491,
+          -0.022083070129156113,
+          0.048851799219846725,
+          0.010346243157982826,
+          -0.0010835592402145267,
+          -0.00513911247253418,
+          -0.0077540138736367226,
+          -0.00036866520531475544,
+          -0.009582933969795704,
+          0.004315343219786882,
+          -0.004443821031600237,
+          0.006707297638058662,
+          0.0014160900609567761,
+          -0.0349157378077507,
+          0.0048179179430007935,
+          -0.014979002065956593,
+          -0.0012743865372613072,
+          -0.0008365228422917426,
+          -0.006892457138746977,
+          -0.00413396256044507,
+          -0.004625201225280762,
+          -0.008630686439573765,
+          0.01435928512364626,
+          0.0028850827366113663,
+          0.00026970895123668015,
+          -0.005585006438195705,
+          -0.013475055806338787,
+          0.008056314662098885,
+          -0.012386772781610489,
+          0.012462347745895386,
+          -0.006930244620889425,
+          -0.008668473921716213,
+          0.018304310739040375,
+          0.008252810686826706,
+          -0.0032421755604445934,
+          -0.02146335318684578,
+          -0.012333869934082031,
+          0.030895136296749115,
+          -0.010814809240400791,
+          0.005970439873635769,
+          -0.004451378248631954,
+          0.001610696199350059,
+          -0.006627943832427263,
+          0.008154562674462795,
+          0.014049426652491093,
+          0.01772993803024292,
+          0.017639247700572014,
+          -0.01771482452750206,
+          -0.0014689926756545901,
+          0.004799024201929569,
+          -0.02028438076376915,
+          -0.022959740832448006,
+          0.03682022914290428,
+          -0.002288038842380047,
+          0.014994116500020027,
+          0.0015162271447479725,
+          0.016369584947824478,
+          -0.011540330946445465,
+          0.017881089821457863,
+          0.004659210331737995,
+          -0.012280968017876148,
+          0.0019479504553601146,
+          -0.022007493302226067,
+          0.0396013967692852,
+          -0.010520066134631634,
+          -0.013210542500019073,
+          0.00734590832144022,
+          -0.02096455730497837,
+          -0.005093767773360014,
+          -0.024395670741796494,
+          -0.006080023944377899,
+          -0.00045415962813422084,
+          0.0004756513226311654,
+          0.007542403414845467,
+          -0.015523143112659454,
+          -0.028325580060482025,
+          -0.001077891094610095,
+          -0.011774614453315735,
+          -0.006148041225969791,
+          -0.01306694932281971,
+          -0.029761508107185364,
+          -0.011736826039850712,
+          0.039299096912145615,
+          -0.006979368627071381,
+          -0.0028964190278202295,
+          -0.0008029863820411265,
+          -0.0008398292702622712,
+          0.0038184362929314375,
+          0.019483283162117004,
+          -0.005887307226657867,
+          0.023609688505530357,
+          -0.005777723155915737,
+          0.010096845217049122,
+          0.004821696784347296,
+          -0.013822701759636402,
+          -0.013384365476667881,
+          0.014004082418978214,
+          0.003289409913122654,
+          -0.020390184596180916,
+          0.023670148104429245,
+          -0.008358615450561047,
+          0.01866707019507885,
+          -0.017216026782989502,
+          0.012099587358534336,
+          -0.02661758102476597,
+          -0.012757091782987118,
+          -0.014593568630516529,
+          -0.017805512994527817,
+          0.008819623850286007,
+          -0.011835074052214622,
+          0.01653585024178028,
+          -0.005743714049458504,
+          -0.0018837115494534373,
+          -0.0011723600327968597,
+          -0.037001609802246094,
+          0.0032591798808425665,
+          0.026693155989050865,
+          0.03675977140665054,
+          0.012311197817325592,
+          0.0030097817070782185,
+          -0.007950509898364544,
+          0.03654816001653671,
+          -0.010913057252764702,
+          -0.0122431805357337,
+          0.013187870383262634,
+          -0.006355873309075832,
+          -5.260741454549134e-05,
+          -0.01002126932144165,
+          0.02191680483520031,
+          0.013369250111281872,
+          0.04171750321984291,
+          -0.007319456897675991,
+          -0.000410467735491693,
+          -0.00897833239287138,
+          0.0007189089665189385,
+          0.007122961338609457,
+          0.0036616178695112467,
+          0.0017968000611290336,
+          -0.015704523772001266,
+          0.0067526428028941154,
+          -0.010867712087929249,
+          0.009046349674463272,
+          -0.0007529177819378674,
+          0.001029711915180087,
+          0.010043942369520664,
+          0.009348650462925434,
+          0.009076579473912716,
+          0.01958908885717392,
+          -0.012870454229414463,
+          -0.0059213158674538136,
+          -0.008895199745893478,
+          0.015024347230792046,
+          -0.0124850207939744,
+          0.0160824004560709,
+          0.006790430285036564,
+          -0.006091360002756119,
+          0.008411518298089504,
+          0.003138259518891573,
+          -0.0036559496074914932,
+          -0.0035180249251425266,
+          0.02422940544784069,
+          -0.007164527662098408,
+          -0.016399815678596497,
+          -0.00943934079259634,
+          0.018818221986293793,
+          -0.006114032585173845,
+          0.005214687902480364,
+          -0.005709705408662558,
+          -0.003502909792587161,
+          -0.013830259442329407,
+          -0.003361206501722336,
+          -0.010603198781609535,
+          0.0021274415776133537,
+          0.02028438076376915,
+          -0.0005294986767694354,
+          0.0034273345954716206,
+          0.0001297767594223842,
+          0.00804875697940588,
+          0.010913057252764702,
+          0.011472313664853573,
+          0.004387139342725277,
+          0.02333761751651764,
+          0.04126405343413353,
+          -0.00022826067288406193,
+          0.0005219411104917526,
+          -0.012537923641502857,
+          0.013928506523370743,
+          -0.0018374216742813587,
+          0.028340695425868034,
+          -0.01960420422255993,
+          0.0026848085690289736,
+          0.006125368643552065,
+          -0.01794154942035675,
+          -0.0003625247336458415,
+          0.001641870941966772,
+          0.00031694344943389297,
+          0.014147674664855003,
+          0.0051051038317382336,
+          -0.006627943832427263,
+          0.005199573002755642,
+          0.00804875697940588,
+          -0.0022237999364733696,
+          -0.015991710126399994,
+          0.012409445829689503,
+          0.0020461981184780598,
+          -0.0003955888678319752,
+          0.010127075016498566,
+          0.027887243777513504,
+          0.006185829173773527,
+          0.005566112697124481,
+          0.028053509071469307,
+          0.0027641623746603727,
+          0.010006154887378216,
+          0.00484059052541852,
+          0.007738898973912001,
+          0.007580191362649202,
+          -0.006321864202618599,
+          -0.014283710159361362,
+          -0.016974186524748802,
+          -0.01723114214837551,
+          -0.0030097817070782185,
+          -0.00016000683535821736,
+          0.03222525864839554,
+          0.001197866746224463,
+          0.01073167659342289,
+          0.01866707019507885,
+          -0.0017901872051879764,
+          -0.004885935690253973,
+          -0.025499068200588226,
+          -0.00489727221429348,
+          -0.0003348925383761525,
+          0.007073837332427502,
+          0.0005772055010311306,
+          0.004753679037094116,
+          -0.007799359038472176,
+          -0.00035071608726866543,
+          0.01655096560716629,
+          0.0014293156564235687,
+          0.013097180053591728,
+          -0.02351899817585945,
+          -0.023231811821460724,
+          0.010006154887378216,
+          0.020465759560465813,
+          0.005048422608524561,
+          -0.011661251075565815,
+          0.0046667675487697124,
+          0.01039158832281828,
+          0.0021425564773380756,
+          0.026753615587949753,
+          0.007323235739022493,
+          -0.018606610596179962,
+          0.008608013391494751,
+          0.010663658380508423,
+          0.01234898529946804,
+          -0.01176705677062273,
+          -0.0035010206047445536,
+          -0.007217430509626865,
+          -0.005690811667591333,
+          0.004107511602342129,
+          0.004353130701929331,
+          0.020934326574206352,
+          0.008509766310453415,
+          0.02008788473904133,
+          0.006151820067316294,
+          0.002475087298080325,
+          0.017382292076945305,
+          0.008441748097538948,
+          0.012069357559084892,
+          0.012757091782987118,
+          0.018712416291236877,
+          -0.02400267869234085,
+          0.017110222950577736,
+          -0.014963886700570583,
+          -0.007051164750009775,
+          0.0011336278403177857,
+          0.0048179179430007935,
+          0.004020600114017725,
+          -0.008237695321440697,
+          0.001068444224074483,
+          -0.010769464075565338,
+          0.00554721849039197,
+          -0.020919211208820343,
+          -0.00011064679711125791,
+          0.012160047888755798,
+          -0.003557701827958226,
+          -0.014767391607165337,
+          -0.010179977864027023,
+          0.02636062540113926,
+          0.011804844252765179,
+          0.012061799876391888,
+          -0.006544811185449362,
+          0.002210574224591255,
+          -0.015440010465681553,
+          0.013920948840677738,
+          0.018727531656622887,
+          -0.008872526697814465,
+          0.023246927186846733,
+          -0.014449975453317165,
+          0.006499466020613909,
+          -0.010149747133255005,
+          -0.006268961820751429,
+          -0.009212614968419075,
+          0.004292670637369156,
+          0.0041037327609956264,
+          0.016717230901122093,
+          0.008366173133254051,
+          -0.008940544910728931,
+          0.0086458008736372,
+          -0.0041982014663517475,
+          -0.008698703721165657,
+          -0.0172916017472744,
+          -0.014125002548098564,
+          -0.010754348710179329,
+          -0.0018138044979423285,
+          -0.0004062166262883693,
+          -0.022748131304979324,
+          0.00751973083242774,
+          0.0005649244994856417,
+          0.009114367887377739,
+          -0.001027822494506836,
+          -0.006873563397675753,
+          0.016974186524748802,
+          0.012076914310455322,
+          0.006896235514432192,
+          0.009552703239023685,
+          0.034643664956092834,
+          0.00945445615798235,
+          -0.007633093744516373,
+          0.013097180053591728,
+          -0.009446898475289345,
+          -0.027569828554987907,
+          0.0030910251662135124,
+          0.00241462723352015,
+          -0.000520996458362788,
+          0.0008346334798261523,
+          -0.005664360243827105,
+          0.0037220779340714216,
+          0.002367392648011446,
+          0.004062166437506676,
+          -0.0027981712482869625,
+          0.010671216063201427,
+          -0.026889652013778687,
+          0.014404630288481712,
+          0.014827851206064224,
+          0.011396737769246101,
+          -0.013482613489031792,
+          0.008260367438197136,
+          0.000955553783569485,
+          -0.017200913280248642,
+          -0.03243687003850937,
+          -0.01002126932144165,
+          0.01305939257144928,
+          0.011162455193698406,
+          0.004364467225968838,
+          0.0009560260805301368,
+          -0.0006178271723911166,
+          0.013792471028864384,
+          0.015213284641504288,
+          -0.004330458119511604,
+          -0.028310464695096016,
+          -0.023050431162118912,
+          0.032739169895648956,
+          0.007799359038472176,
+          0.006280297879129648,
+          0.019982079043984413,
+          0.006503244861960411,
+          0.0008549442864023149,
+          -0.02590717375278473,
+          0.014049426652491093,
+          -0.013505285605788231,
+          -0.004579856060445309,
+          0.006986925844103098,
+          -0.006499466020613909,
+          -0.016641655936837196,
+          0.013898276723921299,
+          0.001303671975620091,
+          -0.007243881467729807,
+          0.006677067838609219,
+          -0.008222579956054688,
+          -0.01001371257007122,
+          -0.01635446958243847,
+          -0.0008351058349944651,
+          -3.5425870009930804e-05,
+          -0.032074108719825745,
+          0.008721376769244671,
+          -0.012598383240401745,
+          0.007769129239022732,
+          0.014570895582437515,
+          0.004224652890115976,
+          -0.021750537678599358,
+          0.005562333855777979,
+          -0.012500136159360409,
+          -0.013497727923095226,
+          0.0069491383619606495,
+          -0.007334571797400713,
+          -0.00636343052610755,
+          -0.015024347230792046,
+          0.0051655638962984085,
+          0.006412554532289505,
+          0.019437937065958977,
+          -0.009552703239023685,
+          -0.0008813955937512219,
+          -0.014563338831067085,
+          0.014253480359911919,
+          -0.015455125831067562,
+          0.016278894618153572,
+          0.001953618600964546,
+          -0.004583634901791811,
+          -0.03237640857696533,
+          -0.003948803525418043,
+          0.0026281271129846573,
+          -0.006257625296711922,
+          -0.007723784074187279,
+          0.019241442903876305,
+          0.02005765400826931,
+          0.005101324990391731,
+          -0.01481273677200079,
+          -8.428404726146255e-06,
+          0.008192350156605244,
+          0.008018527179956436,
+          0.002970104804262519,
+          0.001875209272839129,
+          -0.016384700313210487,
+          -0.003960139583796263,
+          -0.022672556340694427,
+          -0.032557789236307144,
+          -0.006257625296711922,
+          0.0010920614004135132,
+          0.004084839019924402,
+          0.023216696456074715,
+          0.013905834406614304,
+          0.013527958653867245,
+          -0.006918908096849918,
+          0.021614503115415573,
+          0.009567818604409695,
+          -0.018092699348926544,
+          -0.009862561710178852,
+          -0.010043942369520664,
+          0.01629400998353958,
+          0.007213651668280363,
+          0.011147339828312397,
+          0.02823488973081112,
+          0.017352063208818436,
+          0.005558555014431477,
+          0.002766051795333624,
+          -0.007980739697813988,
+          -0.01364132110029459,
+          -0.01199378166347742,
+          -0.009257960133254528,
+          -0.0228690505027771,
+          0.009182385168969631,
+          -0.010520066134631634,
+          0.00665061641484499,
+          -0.008698703721165657,
+          -0.0022219105158001184,
+          0.0060271210968494415,
+          -0.01582544483244419,
+          0.0004508532292675227,
+          -0.01202401239424944,
+          -0.0008365228422917426,
+          0.00802608486264944,
+          0.010059057734906673,
+          -0.01889379695057869,
+          0.00792783685028553,
+          0.005150448996573687,
+          0.010920614004135132,
+          -0.02964058890938759,
+          -0.029323171824216843,
+          0.008320827968418598,
+          -0.01165369339287281,
+          -0.009824774228036404,
+          -0.026239704340696335,
+          -0.009250402450561523,
+          -0.013603533618152142,
+          -0.001517171855084598,
+          0.0019290565978735685,
+          0.021735424175858498,
+          -0.009159712120890617,
+          -0.03518780693411827,
+          -0.0252118818461895,
+          -0.015931248664855957,
+          -0.011215358041226864,
+          0.0019800697918981314,
+          -0.0023466094862669706,
+          -0.001889379695057869,
+          0.006896235514432192,
+          -0.02308066189289093,
+          -0.02049599029123783,
+          0.0006617552135139704,
+          0.010452047921717167,
+          -0.013339020311832428,
+          -0.010119517333805561,
+          0.026421085000038147,
+          -0.015553372912108898,
+          0.01001371257007122,
+          -0.013225657865405083,
+          -0.005925094708800316,
+          0.0010117627680301666,
+          -0.001555904163978994,
+          -0.004145299084484577,
+          0.00558878481388092,
+          -0.03403906524181366,
+          0.00513911247253418,
+          0.031832266598939896,
+          0.02382129803299904,
+          -0.004791466984897852,
+          -0.002518543042242527,
+          -0.03521803766489029,
+          -0.002465640427544713,
+          -0.012689073570072651,
+          0.02753959782421589,
+          -0.003945024684071541,
+          0.007474386133253574,
+          0.001541733741760254,
+          0.0036162727046757936,
+          -0.005600121337920427,
+          0.003094803774729371,
+          0.0005252475384622812,
+          -0.005857076961547136,
+          0.005671917926520109,
+          0.014963886700570583,
+          0.0009494132827967405,
+          0.012938471511006355,
+          -0.01635446958243847,
+          -0.008993446826934814,
+          -0.008872526697814465,
+          -0.024592164903879166,
+          -0.00558878481388092,
+          0.009839889593422413,
+          0.019226327538490295,
+          -0.01467670127749443,
+          -0.015946364030241966,
+          0.01527374517172575,
+          0.006624164991080761,
+          -0.02658735029399395,
+          -0.0007855095900595188,
+          -0.02403290942311287,
+          0.026436200365424156,
+          0.0163242407143116,
+          -0.0023371626157313585,
+          0.00244863610714674,
+          -0.006900014355778694,
+          -0.011691480875015259,
+          -0.013936064206063747,
+          0.00029309000819921494,
+          0.007867377251386642,
+          -0.032315950840711594,
+          0.0076142000034451485,
+          0.009159712120890617,
+          -0.0013546851696446538,
+          -0.02218887396156788,
+          0.017200913280248642,
+          0.00838884525001049,
+          0.0016569859581068158,
+          -0.019876273348927498,
+          0.008101659826934338,
+          0.0014264816418290138,
+          -0.004693218972533941,
+          -0.0049539534375071526,
+          -0.010406702756881714,
+          4.165491554886103e-05,
+          -0.023851528763771057,
+          -0.005974218249320984,
+          -0.007531067356467247,
+          -0.0029946668073534966,
+          0.017352063208818436,
+          -0.014510435983538628,
+          0.00020676899293903261,
+          0.009590490721166134,
+          0.018848450854420662,
+          0.010164862498641014,
+          -0.014593568630516529,
+          0.023216696456074715,
+          0.0019800697918981314,
+          -0.007364802062511444,
+          0.007047385908663273,
+          -0.005112661514431238,
+          -0.009303305298089981,
+          0.009431783109903336,
+          -0.0029927773866802454,
+          0.006563704926520586,
+          -0.008335943333804607,
+          -0.007281669415533543,
+          -0.013527958653867245,
+          -0.0013764130417257547,
+          0.014895869418978691,
+          0.003892122069373727,
+          -0.023896872997283936,
+          0.01528886053711176,
+          -0.002108547603711486,
+          -0.0024335209745913744,
+          -0.009998597204685211,
+          -0.019437937065958977,
+          -0.012976259924471378,
+          -0.009152155369520187,
+          -0.004776351619511843,
+          0.013210542500019073,
+          0.012968702241778374,
+          -0.011623463593423367,
+          -0.009567818604409695,
+          -0.004995519760996103,
+          -0.030487030744552612,
+          0.035792406648397446,
+          -0.019483283162117004,
+          0.02008788473904133,
+          -0.01910540647804737,
+          0.005192015320062637,
+          -0.001130793709307909,
+          0.0021614502184093,
+          -0.007799359038472176,
+          0.004995519760996103,
+          -0.019649548456072807,
+          -0.015961479395627975,
+          -0.006605271250009537,
+          -0.008033642545342445,
+          -0.021130822598934174,
+          -0.014608683064579964,
+          -0.007070058491080999,
+          0.012794879265129566,
+          -0.0011988113401457667,
+          -0.0014633245300501585,
+          0.020919211208820343,
+          0.007814474403858185,
+          0.011162455193698406,
+          0.026164129376411438,
+          0.022566750645637512,
+          0.03149973601102829,
+          -0.002148224739357829,
+          -0.01676257699728012,
+          0.006918908096849918,
+          0.0005550052737817168,
+          0.0007992075989022851,
+          0.002907755319029093,
+          4.006075323559344e-05,
+          0.011132225394248962,
+          -0.01469937339425087,
+          -0.003922352101653814,
+          -0.007640651427209377,
+          -0.010043942369520664,
+          0.006465457379817963,
+          0.007217430509626865,
+          0.0008539995760656893,
+          0.022476060315966606,
+          -0.017790399491786957,
+          0.007890049368143082,
+          -0.0018478132551535964,
+          0.01612774468958378,
+          0.01676257699728012,
+          0.011442082934081554,
+          -0.004477829672396183,
+          -0.005687032826244831,
+          -0.015916133299469948,
+          0.04226164519786835,
+          -0.005033307243138552,
+          0.00028010053210891783,
+          0.0086458008736372,
+          0.008351057767868042,
+          0.0008747827960178256,
+          -0.0095753762871027,
+          -0.010572968982160091,
+          -0.009038791991770267,
+          0.012326313182711601,
+          0.004734785296022892,
+          -0.011600791476666927,
+          -0.016414931043982506,
+          -0.006219837814569473,
+          0.021145937964320183,
+          0.009938136674463749,
+          -0.0077200052328407764,
+          0.001680603250861168,
+          -0.009076579473912716,
+          0.006268961820751429,
+          0.015266187489032745,
+          0.00838128849864006,
+          0.02821977436542511,
+          0.018818221986293793,
+          -0.030003348365426064,
+          0.02637574076652527,
+          0.009605606086552143,
+          0.005063537508249283,
+          0.0035236929543316364,
+          0.016868380829691887,
+          0.010777021758258343,
+          -0.009416667744517326,
+          -0.009666066616773605,
+          0.0013915281742811203,
+          0.0077200052328407764,
+          0.004564741160720587,
+          -0.016399815678596497,
+          0.0006296357605606318,
+          -0.0038108788430690765,
+          -0.014427303336560726,
+          -0.009152155369520187,
+          -0.010081729851663113,
+          -0.005350723396986723,
+          0.02237025462090969,
+          0.00523736048489809,
+          -0.009741641581058502,
+          -0.03213457018136978,
+          0.005101324990391731,
+          -0.016218435019254684,
+          -0.020858751609921455,
+          -0.014427303336560726,
+          -0.008766721934080124,
+          0.00010432918497826904,
+          -0.0113136051222682,
+          -0.022113298997282982,
+          -0.016218435019254684,
+          0.0007222153944894671,
+          -0.004621422849595547,
+          0.00804120022803545,
+          -0.005690811667591333,
+          0.031167207285761833,
+          -0.005932651925832033,
+          0.00530537823215127,
+          -0.006563704926520586,
+          0.004984183702617884,
+          0.004848148208111525,
+          -0.023186467587947845,
+          -0.005728599149733782,
+          0.006359652150422335,
+          -0.0013244551373645663,
+          0.00010574621410341933,
+          0.00646167853847146,
+          -0.012099587358534336,
+          0.005539661273360252,
+          0.003895900910720229,
+          -0.0006830107304267585,
+          -0.0014510435285046697,
+          0.005762607790529728,
+          0.005959103349596262,
+          0.015946364030241966,
+          -0.007890049368143082,
+          -0.017669478431344032,
+          -0.009144597686827183,
+          0.00047187256859615445,
+          -0.002110437024384737,
+          0.027675632387399673,
+          0.010565411299467087,
+          0.005516988690942526,
+          0.018138045445084572,
+          0.012477463111281395,
+          -0.0022200210951268673,
+          0.022989971563220024,
+          -0.0015974704874679446,
+          -0.015341762453317642,
+          0.009288189932703972,
+          -0.006609050091356039,
+          0.0014680479653179646,
+          -0.016717230901122093,
+          0.001082614529877901,
+          -0.007081395015120506,
+          0.0034159983042627573,
+          0.010240437462925911,
+          -0.001951729180291295,
+          0.013202984817326069,
+          -0.00954514555633068,
+          0.004889714531600475,
+          -0.0029852199368178844,
+          0.005789059214293957,
+          -0.023836413398385048,
+          0.006159377750009298,
+          -0.004821696784347296,
+          -0.0043002283200621605,
+          -0.002405180362984538,
+          0.006280297879129648,
+          -0.0032931887544691563,
+          0.013255887664854527,
+          -0.003586042672395706,
+          0.025529297068715096,
+          0.003841108875349164,
+          -0.0026999234687536955,
+          0.01410232950001955,
+          -0.0021274415776133537,
+          0.01036135759204626,
+          0.017200913280248642,
+          -0.02164473384618759,
+          -0.001862928387708962,
+          -0.01771482452750206,
+          0.00402437848970294,
+          0.00397147610783577,
+          0.006805545650422573,
+          -0.0016862713964655995,
+          0.0042435466311872005,
+          -0.02655711956322193,
+          -0.005411183461546898,
+          -0.006454120855778456,
+          0.014510435983538628,
+          -0.017064876854419708,
+          -0.018833335489034653,
+          -0.002526100492104888,
+          0.0037806485779583454,
+          0.0001020265044644475,
+          0.0008993447409011424,
+          0.0055736699141561985,
+          0.013490171171724796,
+          -0.001459545805118978,
+          -0.006302970461547375,
+          -0.011185127310454845,
+          -0.0017571230418980122,
+          0.01582544483244419,
+          -0.004194422625005245,
+          -0.015462683513760567,
+          -0.006794209126383066,
+          -0.013966294005513191,
+          -0.02702568657696247,
+          -0.00242029526270926,
+          -0.007281669415533543,
+          0.015228400006890297,
+          0.008970774710178375,
+          -0.0006919853040017188,
+          0.014261038042604923,
+          0.00027088981005363166,
+          0.010232879780232906,
+          -0.020148344337940216,
+          -0.024622395634651184,
+          -0.02311089262366295,
+          -0.010905499570071697,
+          0.003536918666213751,
+          -0.021176166832447052,
+          -0.01605216972529888,
+          0.02052622102200985,
+          -0.0048292544670403,
+          -0.0034802372101694345,
+          0.01073167659342289,
+          -0.004931280855089426,
+          -0.009114367887377739,
+          -0.007304341532289982,
+          -0.016248665750026703,
+          -0.0165660809725523,
+          0.01469181664288044,
+          0.003967697266489267,
+          -0.01549291331321001,
+          0.02658735029399395,
+          0.030487030744552612,
+          0.00922017265111208,
+          -0.008411518298089504,
+          -0.002813286380842328,
+          -0.009847446344792843,
+          -0.025136306881904602,
+          -0.0014633245300501585,
+          -0.007429040968418121,
+          -0.0040017059072852135,
+          -0.015764983370900154,
+          -0.005864634644240141,
+          -0.002218131674453616,
+          -0.005539661273360252,
+          -0.007995855063199997,
+          -0.002146335318684578,
+          0.03210433945059776,
+          0.009643393568694592,
+          0.010784579440951347,
+          0.0018393110949546099,
+          -0.010875268839299679,
+          0.014230807311832905,
+          -0.030623065307736397,
+          -0.016142860054969788,
+          0.013134967535734177,
+          0.001925277872942388,
+          0.010225323028862476,
+          -0.0023296051658689976,
+          0.0030437905807048082,
+          0.013051834888756275,
+          -0.004972847178578377,
+          0.015024347230792046,
+          0.00886496901512146,
+          -0.0017873531905934215,
+          -0.002206795383244753,
+          0.002907755319029093,
+          -0.002322047483175993,
+          0.008812067098915577,
+          0.01635446958243847,
+          -0.014971444383263588,
+          -0.00279439240694046,
+          0.007500837091356516,
+          0.0247282013297081,
+          -0.0026470208540558815,
+          -0.013021604157984257,
+          -0.0069907046854496,
+          0.021826114505529404,
+          0.007077616173774004,
+          -0.014638913795351982,
+          -0.0032610693015158176,
+          -0.006336979568004608,
+          0.031590428203344345,
+          0.02589205838739872,
+          0.02102501690387726,
+          -0.0013896387536078691,
+          -0.01865195669233799,
+          -0.004681882914155722,
+          -0.016974186524748802,
+          0.016248665750026703,
+          0.0027112597599625587,
+          -0.008910314179956913,
+          -0.0039412458427250385,
+          -0.015107479877769947,
+          -0.010043942369520664,
+          -0.0032535118516534567,
+          0.007421483285725117,
+          -0.017548557370901108,
+          -0.009242845699191093,
+          -0.003255401039496064,
+          -0.015417338348925114,
+          -0.002473197877407074,
+          0.01015730481594801,
+          -0.0065863775089383125,
+          -0.013679108582437038,
+          0.004976626019924879,
+          0.0007236324017867446,
+          0.012998932041227818,
+          -0.011925764381885529,
+          -0.026511775329709053,
+          -0.0021746759302914143,
+          0.011759499087929726,
+          0.0189089123159647,
+          -0.0008625017944723368,
+          -0.01862172596156597,
+          -0.03225548937916756,
+          -0.017427638173103333,
+          -0.012235622853040695,
+          -0.025529297068715096,
+          0.001894103130325675,
+          0.0020726495422422886,
+          0.0086458008736372,
+          -0.01505457703024149,
+          -0.043742917478084564,
+          0.005475422367453575,
+          -0.004326679278165102,
+          0.006336979568004608,
+          0.003963918425142765,
+          0.008018527179956436,
+          0.017397407442331314,
+          0.00670351879671216,
+          0.012273410335183144,
+          -0.00420198030769825,
+          -0.011819958686828613,
+          0.00852488074451685,
+          0.03470412641763687,
+          0.0057701654732227325,
+          -0.011880419217050076,
+          0.008955659344792366,
+          -0.014669143594801426,
+          -0.0009654730092734098,
+          0.003629498416557908,
+          -0.0012120370520278811,
+          -0.004043272230774164,
+          0.014654028229415417,
+          0.0009919243166223168,
+          -0.008494650945067406,
+          0.0008034586790017784,
+          0.02354922890663147,
+          -0.02007276937365532,
+          -0.001997074345126748,
+          -0.0067979879677295685,
+          -0.02631527930498123,
+          0.010709003545343876,
+          0.0014718268066644669,
+          -0.019543742761015892,
+          -0.01839500106871128,
+          0.0009612218709662557,
+          -0.014835408888757229,
+          0.03219502791762352,
+          0.01957397349178791,
+          -0.0038259937427937984,
+          -0.004296449478715658,
+          0.018969371914863586,
+          0.033192623406648636,
+          0.005093767773360014,
+          0.00524869654327631,
+          -0.011117110028862953,
+          0.009975924156606197,
+          0.01260594092309475,
+          0.0041528563015162945,
+          -0.000520996458362788,
+          0.017185797914862633,
+          -0.010293340310454369,
+          -0.015311532653868198,
+          -0.010875268839299679,
+          0.00218979106284678,
+          -0.014759833924472332,
+          -0.0045005022548139095,
+          -0.0030211179982870817,
+          -0.0009853114606812596,
+          0.02983708307147026,
+          -0.004999298602342606,
+          0.01798689365386963,
+          0.024320095777511597,
+          0.006571262609213591,
+          0.018757760524749756,
+          -0.016868380829691887,
+          -0.00021290946460794657,
+          0.0006858448032289743,
+          0.01909029111266136,
+          -0.0029058658983558416,
+          -0.00601200619712472,
+          -0.016460275277495384,
+          -0.005751271732151508,
+          -0.01176705677062273,
+          -0.03612493723630905,
+          0.02705591730773449,
+          -0.009061465039849281,
+          -0.024637511000037193,
+          -0.012658843770623207,
+          -0.028053509071469307,
+          -0.01366399321705103,
+          0.02353411354124546,
+          0.01844034530222416,
+          -0.0124850207939744,
+          -0.00780691672116518,
+          -0.02099478617310524,
+          -0.011404295451939106,
+          0.04873088002204895,
+          -0.0035255823750048876,
+          -0.02029949426651001,
+          0.009975924156606197,
+          -0.01889379695057869,
+          -0.015266187489032745,
+          0.012855338864028454,
+          0.006412554532289505,
+          0.04383360967040062,
+          0.011366507969796658,
+          0.00022991388686932623,
+          -0.01027822494506836,
+          0.000404563412303105,
+          -0.028809260576963425,
+          0.011638578958809376,
+          -0.014132560230791569,
+          -0.017382292076945305,
+          -0.0018648176919668913,
+          -0.011336278170347214,
+          -0.015311532653868198,
+          -0.0037107416428625584,
+          0.001894103130325675,
+          0.01236410066485405,
+          0.007217430509626865,
+          -0.008902756497263908,
+          0.01527374517172575,
+          0.018470576032996178,
+          -0.010127075016498566,
+          0.01933213323354721,
+          0.0025525519158691168,
+          0.006627943832427263,
+          -0.003502909792587161,
+          -0.01061831321567297,
+          -0.015508027747273445,
+          0.013852931559085846,
+          0.004523174837231636,
+          -0.016490506008267403,
+          0.0066014924086630344,
+          0.016384700313210487,
+          0.014858081936836243,
+          -0.008078987710177898,
+          -0.01198622491210699,
+          0.0007396921282634139,
+          0.005063537508249283,
+          -0.003856223775073886,
+          0.005033307243138552,
+          0.006378545891493559,
+          -0.006068687420338392,
+          -0.008517323061823845,
+          -0.0027377111837267876,
+          0.010965959168970585,
+          -0.0034802372101694345,
+          -0.018848450854420662,
+          0.003138259518891573,
+          -0.007651987485587597,
+          -0.008305712603032589,
+          0.02636062540113926,
+          0.0363667793571949,
+          -0.003128812648355961,
+          0.003111808327957988,
+          0.0034745691809803247,
+          0.0060422359965741634,
+          -0.010164862498641014,
+          0.012039126828312874,
+          0.00909925252199173,
+          -0.01609751395881176,
+          0.00792783685028553,
+          -0.015235957689583302,
+          -0.002306932583451271,
+          -0.012190277688205242,
+          0.008245253004133701,
+          -0.0004520340880844742,
+          0.004345573019236326,
+          0.014404630288481712,
+          -0.01505457703024149,
+          -0.01165369339287281,
+          -0.008585341274738312,
+          -0.009008562192320824,
+          0.027630288153886795,
+          0.01769970916211605,
+          -0.027615172788500786,
+          -0.000787398952525109,
+          -0.002512875013053417,
+          -0.023685263469815254,
+          0.0027641623746603727,
+          -0.0007160748937167227,
+          0.0017306717345491052,
+          0.0018251407891511917,
+          0.003895900910720229,
+          0.004644094966351986,
+          -0.00991546455770731,
+          0.013611091300845146,
+          0.013331462629139423,
+          -0.005611457396298647,
+          0.01445753313601017,
+          -0.01958908885717392,
+          0.007209872826933861,
+          -0.01585567370057106,
+          -0.008139447309076786,
+          0.006782873068004847,
+          0.024199174717068672,
+          0.0045458474196493626,
+          0.002652689116075635,
+          0.01517549715936184,
+          -1.436961792933289e-05,
+          0.0216598492115736,
+          0.010527623817324638,
+          0.0022237999364733696,
+          0.009650951251387596,
+          0.0025790033396333456,
+          0.005350723396986723,
+          -0.00977942906320095,
+          0.0009650006541050971,
+          -0.010542738251388073,
+          0.016490506008267403,
+          0.01457845326513052,
+          -0.0027112597599625587,
+          0.012507692910730839,
+          -0.006144262850284576,
+          -0.006488129962235689,
+          -0.01410232950001955,
+          0.005112661514431238,
+          0.009885234758257866,
+          -0.0038108788430690765,
+          0.002359835198149085,
+          0.005501873325556517,
+          -0.005135334096848965,
+          0.01339948084205389,
+          -0.005524545907974243,
+          0.003389547113329172,
+          0.02980685420334339,
+          -0.004345573019236326,
+          -0.0013556298799812794,
+          0.015568488277494907,
+          0.0029474322218447924,
+          -0.0036653964780271053,
+          0.003438670886680484,
+          -0.02608855441212654,
+          0.022022608667612076,
+          0.019725123420357704,
+          0.019468167796730995,
+          0.02421429008245468,
+          0.02335273288190365,
+          -0.007572633679956198,
+          -0.0016796585405245423,
+          0.009061465039849281,
+          0.0037220779340714216,
+          0.009688738733530045,
+          0.011381623335182667,
+          0.03594356030225754,
+          -0.002499649301171303,
+          0.005490537267178297,
+          -0.01961931772530079,
+          0.012961144559085369,
+          0.012681515887379646,
+          -0.004356909543275833,
+          0.0007760627195239067,
+          -0.022249335423111916,
+          0.013414595276117325,
+          -0.024395670741796494,
+          -0.0077540138736367226,
+          0.007791801821440458,
+          -0.01797178015112877,
+          0.031348586082458496,
+          -0.00478768814355135,
+          -0.012530365958809853,
+          0.009069022722542286,
+          0.00576638663187623,
+          0.005093767773360014,
+          -0.018561266362667084,
+          0.008728934451937675,
+          0.0011685813078656793,
+          -0.0035614806693047285,
+          -0.01039158832281828,
+          0.010467163287103176,
+          0.015976594761013985,
+          0.008827181532979012,
+          -0.00075150077464059,
+          -0.00804875697940588,
+          0.03621562942862511,
+          0.0008757274481467903,
+          0.0003894483670592308,
+          -0.0043946970254182816,
+          -0.0027282643131911755,
+          0.007882491685450077,
+          -0.00717964256182313,
+          -0.016188204288482666,
+          -0.014888311736285686,
+          -0.005267590284347534,
+          0.028083739802241325,
+          0.0036238301545381546,
+          0.013293675146996975,
+          -0.0075083947740495205,
+          -0.013293675146996975,
+          0.008351057767868042,
+          0.00518067879602313,
+          -0.0020972113125026226,
+          8.472686749882996e-05,
+          0.0027188172098249197,
+          -0.009378880262374878,
+          -0.034159984439611435,
+          -0.004235989414155483,
+          0.01676257699728012,
+          0.006733749061822891,
+          -0.006854669190943241,
+          0.01837988570332527,
+          -0.016702115535736084,
+          0.010671216063201427,
+          -0.0005975163076072931,
+          0.02283882163465023,
+          0.0163242407143116,
+          0.003030564868822694
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "8f4b7200-614f-45df-b481-ec48e0567e94",
+      "properties": {
+        "page_content": "4 Downstream Results 14 4.1 Fine-Tuning 14 4.2 Distillation 16 4.3 Multilingual Learning 17 ",
+        "keyphrases": [
+          "Downstream Results",
+          "Fine-Tuning",
+          "Distillation",
+          "Multilingual Learning"
+        ],
+        "embedding": [
+          0.007475246209651232,
+          0.018637971952557564,
+          -0.022418024018406868,
+          -0.003915054257959127,
+          -0.025920359417796135,
+          -0.01815968006849289,
+          -0.013469329103827477,
+          0.07850165665149689,
+          -0.036689650267362595,
+          0.014703632332384586,
+          0.017773959785699844,
+          -0.022819172590970993,
+          -0.010807863436639309,
+          -0.039034824818372726,
+          -0.02303517609834671,
+          0.02710837498307228,
+          0.017064236104488373,
+          -0.007691249251365662,
+          0.015552215278148651,
+          -0.02539578080177307,
+          -0.010514716617763042,
+          -0.0059323678724467754,
+          -0.04039255902171135,
+          0.03233873471617699,
+          -0.02058199979364872,
+          0.002258388325572014,
+          0.04107142612338066,
+          -0.006360516417771578,
+          -0.03289416804909706,
+          0.017465384677052498,
+          -0.020350567996501923,
+          -0.00360647845081985,
+          0.01323789730668068,
+          -0.029422692954540253,
+          -0.022634027525782585,
+          0.012574459426105022,
+          0.06967639178037643,
+          0.011378728784620762,
+          0.010777005925774574,
+          0.03610335662961006,
+          0.009380700998008251,
+          0.007938110269606113,
+          0.008539832197129726,
+          -0.007521532475948334,
+          -0.002765609649941325,
+          -0.047397226095199585,
+          -0.0303792767226696,
+          -0.04141085967421532,
+          0.011671875603497028,
+          -0.05992539972066879,
+          0.01718766614794731,
+          -0.060480836778879166,
+          0.016693945974111557,
+          -0.014703632332384586,
+          -0.002127243671566248,
+          0.057271648198366165,
+          0.014317912049591541,
+          0.05264301598072052,
+          0.05779622867703438,
+          0.020921431481838226,
+          0.030240418389439583,
+          -0.005635363515466452,
+          0.014194482006132603,
+          -0.025920359417796135,
+          0.002937254961580038,
+          -0.04468176141381264,
+          -0.009534989483654499,
+          -0.028512394055724144,
+          -0.029129546135663986,
+          0.025735212489962578,
+          -0.003972912207245827,
+          0.04144171625375748,
+          -0.007787679322063923,
+          -0.0030394706409424543,
+          -0.01823682337999344,
+          0.014634202234447002,
+          0.00830840039998293,
+          0.00699695385992527,
+          -0.010993009433150291,
+          0.03243130445480347,
+          0.0004541848611552268,
+          -0.0026498937513679266,
+          0.012991037219762802,
+          -0.016786517575383186,
+          0.03557877987623215,
+          -0.016925377771258354,
+          -0.04036170244216919,
+          -0.049773260951042175,
+          0.02891354262828827,
+          -0.026151791214942932,
+          -0.02970041148364544,
+          -0.02309689112007618,
+          0.0036836224608123302,
+          -0.013939906843006611,
+          -0.033449605107307434,
+          0.02380661480128765,
+          -0.007452103309333324,
+          0.001406912342645228,
+          -0.021631157025694847,
+          0.011209012009203434,
+          -0.003843696089461446,
+          0.005519647616893053,
+          0.004609349649399519,
+          0.0007053847657516599,
+          0.02632150799036026,
+          -0.017928248271346092,
+          0.00661894865334034,
+          -0.016678515821695328,
+          0.004844638518989086,
+          0.07368787378072739,
+          0.01564478874206543,
+          0.007540818769484758,
+          -0.02072085812687874,
+          0.025087203830480576,
+          -0.02229459397494793,
+          -0.018745973706245422,
+          -0.03391246870160103,
+          -0.04693436250090599,
+          0.0011436586501076818,
+          -0.027432380244135857,
+          0.04838467016816139,
+          0.025056347250938416,
+          0.006591948214918375,
+          -0.046378929167985916,
+          -0.053167592734098434,
+          0.034221045672893524,
+          0.025904929265379906,
+          0.031582724303007126,
+          -0.01393219269812107,
+          0.02979298308491707,
+          0.019393982365727425,
+          0.0023509610909968615,
+          -0.06424546241760254,
+          -0.0021214578300714493,
+          0.00862469058483839,
+          -0.0044126324355602264,
+          0.01812882162630558,
+          0.011733590625226498,
+          0.017912819981575012,
+          0.001125336973927915,
+          0.03468390926718712,
+          -0.03631936013698578,
+          0.02301974780857563,
+          -0.006086655426770449,
+          -0.022448882460594177,
+          0.0007304565515369177,
+          -0.009596704505383968,
+          0.09985509514808655,
+          0.018607115373015404,
+          -0.006129084620624781,
+          -0.015968792140483856,
+          0.003992198035120964,
+          -0.012682461179792881,
+          0.026861514896154404,
+          0.018406540155410767,
+          0.010159854777157307,
+          0.01717223785817623,
+          0.004825352691113949,
+          0.0030587567016482353,
+          0.0013577330391854048,
+          0.020767144858837128,
+          -0.007043240126222372,
+          0.019378554075956345,
+          0.006148370914161205,
+          0.07208327949047089,
+          0.03388161212205887,
+          -0.0052612158469855785,
+          0.026614654809236526,
+          -0.04964983090758324,
+          0.021044863387942314,
+          0.02553463913500309,
+          0.030857570469379425,
+          0.04526805505156517,
+          0.0030356133356690407,
+          -0.01653965748846531,
+          -0.016987092792987823,
+          -0.020828859880566597,
+          0.03162900730967522,
+          -0.06344316154718399,
+          0.01741909794509411,
+          -0.013153038918972015,
+          -0.014549343846738338,
+          0.020906003192067146,
+          -0.0063450876623392105,
+          0.01806710660457611,
+          -0.023621469736099243,
+          0.04292288050055504,
+          -0.00019466786761768162,
+          0.027324378490447998,
+          0.006726950407028198,
+          0.0067732366733253,
+          0.010908151045441628,
+          0.027386093512177467,
+          0.016940806061029434,
+          -0.0021947447676211596,
+          -0.007737535517662764,
+          -0.013716190122067928,
+          0.03545534610748291,
+          -0.0014773061266168952,
+          0.010576431639492512,
+          -0.019440269097685814,
+          -0.05048298463225365,
+          -0.0035293344408273697,
+          0.05329102277755737,
+          0.005943939555436373,
+          0.020412283018231392,
+          0.023528896272182465,
+          0.05344530940055847,
+          0.0020018848590552807,
+          -0.03142843395471573,
+          0.01661680079996586,
+          -0.01977970264852047,
+          -0.052735585719347,
+          0.026136361062526703,
+          -0.0027559667360037565,
+          -0.0024358194787055254,
+          -0.008269828744232655,
+          0.012867606244981289,
+          -0.04230572655797005,
+          -0.017650529742240906,
+          0.02061285637319088,
+          -0.0011561946012079716,
+          0.013461614958941936,
+          -0.01480391900986433,
+          -0.032554734498262405,
+          0.0017675601411610842,
+          -0.020427711308002472,
+          3.8240483263507485e-05,
+          0.009689277037978172,
+          -0.02221745066344738,
+          0.01800539158284664,
+          0.03056442365050316,
+          -0.022803744301199913,
+          -0.019270552322268486,
+          0.01731109619140625,
+          -0.018761401996016502,
+          0.014194482006132603,
+          0.00035775493597611785,
+          0.003957483451813459,
+          0.02880554087460041,
+          0.0005978654371574521,
+          -0.033418748527765274,
+          -0.02209402061998844,
+          -0.032585594803094864,
+          0.02317403443157673,
+          -0.03468390926718712,
+          -0.024207763373851776,
+          0.01882311701774597,
+          -0.008362401276826859,
+          0.012481886893510818,
+          0.006757807917892933,
+          0.005187929142266512,
+          0.016231082379817963,
+          0.01397847943007946,
+          -0.021584870293736458,
+          0.05446361005306244,
+          0.03206101432442665,
+          -0.024269478395581245,
+          -0.0248403437435627,
+          -0.04853895679116249,
+          0.018360253423452377,
+          -0.018483683466911316,
+          -0.051501285284757614,
+          0.0009676741319708526,
+          -0.003297902876511216,
+          -0.007864823564887047,
+          0.021260865032672882,
+          0.03378903865814209,
+          0.005824366118758917,
+          0.025704355910420418,
+          0.0031937584280967712,
+          0.004107913933694363,
+          -0.014927349053323269,
+          -0.021152863278985023,
+          -0.008015253581106663,
+          -0.03388161212205887,
+          0.019270552322268486,
+          -0.014024765230715275,
+          -0.01820596680045128,
+          0.022541454061865807,
+          0.008586118929088116,
+          0.04875496029853821,
+          0.01447991468012333,
+          0.0017743102507665753,
+          -0.022757457569241524,
+          -0.01633908413350582,
+          -0.014078766107559204,
+          0.02485577203333378,
+          -0.05665449798107147,
+          -0.0132224690169096,
+          0.011579303070902824,
+          -0.015544501133263111,
+          -0.05514247715473175,
+          0.005191786214709282,
+          -0.029808413237333298,
+          0.012782747857272625,
+          0.00558136310428381,
+          -0.005284358747303486,
+          0.010939008556306362,
+          0.01324561145156622,
+          -0.022464310750365257,
+          0.009635276161134243,
+          0.02153858356177807,
+          -0.008524403907358646,
+          -0.03366560861468315,
+          0.027648381888866425,
+          0.019424840807914734,
+          0.028651254251599312,
+          -0.058845385909080505,
+          0.00558907724916935,
+          0.028543252497911453,
+          -0.02470148541033268,
+          -0.013523329980671406,
+          -0.011533016338944435,
+          -0.03366560861468315,
+          0.007120384369045496,
+          0.02971583977341652,
+          0.007575533352792263,
+          -0.04468176141381264,
+          0.007679677568376064,
+          -0.011602446436882019,
+          0.009295842610299587,
+          -0.04212058335542679,
+          0.05168642848730087,
+          0.007108812686055899,
+          0.004316202364861965,
+          0.013052752241492271,
+          0.005288215819746256,
+          -0.011293870396912098,
+          0.03854110464453697,
+          -0.024207763373851776,
+          -0.010206141509115696,
+          -0.017465384677052498,
+          0.022526025772094727,
+          0.021662013605237007,
+          -0.04366346076130867,
+          -0.049032676964998245,
+          0.006044226698577404,
+          0.010661290027201176,
+          -0.015444213524460793,
+          -0.03527020290493965,
+          0.02459348365664482,
+          -0.007394245360046625,
+          -0.02064371481537819,
+          0.006414517294615507,
+          -0.029299262911081314,
+          -0.008964124135673046,
+          0.0019999563228338957,
+          0.0058937957510352135,
+          0.015536786057054996,
+          0.0009117447771131992,
+          0.010946722701191902,
+          -0.005990225821733475,
+          0.03780052438378334,
+          -0.012875321321189404,
+          -0.02951526641845703,
+          -0.020967718213796616,
+          0.019501984119415283,
+          0.024917487055063248,
+          0.03162900730967522,
+          -0.008022967725992203,
+          -0.04039255902171135,
+          -0.035825639963150024,
+          -0.003938197158277035,
+          -0.025843214243650436,
+          -0.035856496542692184,
+          -0.005191786214709282,
+          -0.003957483451813459,
+          -0.004836923908442259,
+          0.01324561145156622,
+          -0.02542663738131523,
+          -0.002941112034022808,
+          0.0429537370800972,
+          0.02146144025027752,
+          -0.0017280238680541515,
+          -0.033418748527765274,
+          -0.014271626248955727,
+          -0.03443704918026924,
+          -0.036597076803445816,
+          0.020319709554314613,
+          -0.0013972693122923374,
+          0.020273422822356224,
+          -0.021260865032672882,
+          -0.009249556809663773,
+          0.016061365604400635,
+          0.018283110111951828,
+          0.0148656340315938,
+          0.007347958628088236,
+          -0.039806265383958817,
+          -0.030641566962003708,
+          -0.01643165573477745,
+          -0.009851278737187386,
+          -0.021692872047424316,
+          -0.029283834621310234,
+          0.012682461179792881,
+          0.040917135775089264,
+          -0.02468605525791645,
+          0.005211072042584419,
+          0.015467356890439987,
+          -0.021708300337195396,
+          -0.057395078241825104,
+          0.03289416804909706,
+          -0.015220495872199535,
+          -0.02062828466296196,
+          0.0032169015612453222,
+          0.01279046293348074,
+          0.005014355294406414,
+          -0.022047733888030052,
+          0.015251354314386845,
+          0.01982598938047886,
+          0.009974709711968899,
+          -0.017743103206157684,
+          0.019285980612039566,
+          -0.03193758428096771,
+          0.02479405701160431,
+          0.0015737360808998346,
+          -0.022711170837283134,
+          -0.001737666898407042,
+          -0.06436888873577118,
+          -0.003911197185516357,
+          0.0006523483316414058,
+          0.020227137953042984,
+          0.03357303515076637,
+          -0.003241973463445902,
+          0.01581450365483761,
+          0.03369646519422531,
+          0.02561178244650364,
+          -0.0017135593807324767,
+          0.0020906003192067146,
+          -0.01655508577823639,
+          0.02309689112007618,
+          0.018622543662786484,
+          0.05415503680706024,
+          -0.01823682337999344,
+          -0.008370115421712399,
+          -0.017496241256594658,
+          -0.014634202234447002,
+          0.03755366429686546,
+          -0.0181133933365345,
+          -0.005079927388578653,
+          0.0010925508104264736,
+          -0.04853895679116249,
+          0.016771089285612106,
+          0.011239869520068169,
+          0.014209911227226257,
+          0.017588814720511436,
+          -0.00450520496815443,
+          0.013484758324921131,
+          -0.022587740793824196,
+          -0.017588814720511436,
+          -0.013253326527774334,
+          0.054031603038311005,
+          -0.07307072728872299,
+          -0.04116399958729744,
+          0.05032869800925255,
+          -0.02803410217165947,
+          0.02806496061384678,
+          0.003409761469811201,
+          0.024207763373851776,
+          -0.05208757892251015,
+          -0.013531044125556946,
+          0.02883639931678772,
+          0.014989065006375313,
+          -0.03465305268764496,
+          -0.0110624385997653,
+          -0.03378903865814209,
+          -0.051470424979925156,
+          -0.044002894312143326,
+          -0.022865459322929382,
+          0.009843564592301846,
+          0.006009511649608612,
+          -0.025102633982896805,
+          -0.012613032013177872,
+          -0.01561393029987812,
+          -0.021677441895008087,
+          0.00471735093742609,
+          -0.027401521801948547,
+          -0.03551706299185753,
+          -0.012512744404375553,
+          0.022896315902471542,
+          -0.0012661246582865715,
+          0.01993398927152157,
+          -0.017635101452469826,
+          -0.006931381765753031,
+          -0.024161476641893387,
+          -0.0010607289150357246,
+          0.03675136715173721,
+          0.009820421226322651,
+          0.04668750241398811,
+          -0.0010134782642126083,
+          0.009311271831393242,
+          -0.03354217857122421,
+          -0.000637883844319731,
+          0.008439545519649982,
+          0.005176357459276915,
+          -0.04934125393629074,
+          -0.003911197185516357,
+          -0.04764408618211746,
+          0.01885397545993328,
+          0.002808038843795657,
+          -0.01198045164346695,
+          -0.002281531458720565,
+          -0.012952464632689953,
+          -0.0004305595357436687,
+          0.026969516649842262,
+          -0.0102987140417099,
+          -0.0075948191806674,
+          -0.009357557632029057,
+          -0.034221045672893524,
+          0.03486905246973038,
+          -0.014348769560456276,
+          -0.017743103206157684,
+          -0.008732692338526249,
+          -0.019270552322268486,
+          -0.018375683575868607,
+          0.011425015516579151,
+          0.0012391243362799287,
+          -0.00016139955550897866,
+          -0.01235074270516634,
+          -0.005083784461021423,
+          0.02152315527200699,
+          0.01800539158284664,
+          -0.0023509610909968615,
+          0.03273988142609596,
+          -0.04175029322504997,
+          0.019100835546851158,
+          0.0014223410980775952,
+          -0.015637073665857315,
+          0.03709080070257187,
+          0.01435648463666439,
+          -0.02562721259891987,
+          0.001886168960481882,
+          0.03397418558597565,
+          -0.0020057419314980507,
+          0.010753863491117954,
+          -0.01074614841490984,
+          -0.011695018969476223,
+          0.025133490562438965,
+          -0.016678515821695328,
+          -0.0015814504586160183,
+          -0.021245436742901802,
+          0.014171338640153408,
+          -0.014873349107801914,
+          0.029052402824163437,
+          0.009396130219101906,
+          0.022803744301199913,
+          0.020998576655983925,
+          -0.013862763531506062,
+          0.006129084620624781,
+          0.019147122278809547,
+          -0.03456047922372818,
+          0.013099038042128086,
+          -0.009141555055975914,
+          0.004084771033376455,
+          -0.01147901639342308,
+          -0.007228385657072067,
+          0.012096167542040348,
+          -0.015073922462761402,
+          0.014564773067831993,
+          -0.013986193574965,
+          0.024207763373851776,
+          0.039219971746206284,
+          -0.026599224656820297,
+          0.007664248812943697,
+          0.001095443731173873,
+          0.0008090469054877758,
+          0.03480733931064606,
+          0.004443489946424961,
+          -0.03277073800563812,
+          -0.006946810521185398,
+          -0.0072823865339159966,
+          0.027756383642554283,
+          -0.026676369830965996,
+          -0.01720309443771839,
+          0.03807824105024338,
+          0.030795855447649956,
+          -0.002358675468713045,
+          -0.03227701783180237,
+          -0.019208837300539017,
+          0.003047185018658638,
+          0.01893111877143383,
+          -0.025735212489962578,
+          -0.029422692954540253,
+          -0.009619846940040588,
+          0.011841592378914356,
+          0.017511671409010887,
+          -0.00911841168999672,
+          -0.01280589122325182,
+          0.0030992571264505386,
+          -0.006811808329075575,
+          0.010784721001982689,
+          -0.004269916098564863,
+          0.0074598174542188644,
+          -0.05878366902470589,
+          -0.006645949091762304,
+          0.011232155375182629,
+          0.04067027568817139,
+          0.00450134789571166,
+          0.03452961891889572,
+          -0.06273344159126282,
+          -0.0036122642923146486,
+          -0.0038880538195371628,
+          -0.01735738292336464,
+          -0.015698788687586784,
+          -0.016832804307341576,
+          -0.004455061629414558,
+          -0.013847334310412407,
+          0.02058199979364872,
+          0.024500910192728043,
+          0.012520458549261093,
+          0.014742203988134861,
+          -0.0058976528234779835,
+          0.042490873485803604,
+          -0.009372986853122711,
+          -0.014186767861247063,
+          0.00041947010322473943,
+          -0.002705823164433241,
+          0.004146486055105925,
+          0.007818536832928658,
+          -0.006572662387043238,
+          0.035825639963150024,
+          0.020890574902296066,
+          0.008832979016005993,
+          0.00911841168999672,
+          0.005693221464753151,
+          0.0458543486893177,
+          0.027432380244135857,
+          -0.045761775225400925,
+          -0.013153038918972015,
+          -0.0098589938133955,
+          -0.017758531495928764,
+          -0.01808253675699234,
+          0.006534090265631676,
+          -0.029823841527104378,
+          -0.00904898252338171,
+          0.008262114599347115,
+          0.019301410764455795,
+          0.015876220539212227,
+          0.016801945865154266,
+          -0.01825225166976452,
+          -0.04684179276227951,
+          -0.01720309443771839,
+          -0.004208201076835394,
+          0.0246089119464159,
+          -0.008030682802200317,
+          -0.0034599050413817167,
+          -0.003587192390114069,
+          -4.5985012548044324e-05,
+          0.01820596680045128,
+          0.01360818836838007,
+          -0.0073248157277703285,
+          -0.010831006802618504,
+          0.012960179708898067,
+          0.037461090832948685,
+          0.028543252497911453,
+          -0.02294260263442993,
+          0.012620746158063412,
+          0.01393219269812107,
+          -0.022850031033158302,
+          0.02237173728644848,
+          0.04051598906517029,
+          0.020196279510855675,
+          0.0015100922901183367,
+          -0.02641407959163189,
+          0.014950492419302464,
+          -0.01068443339318037,
+          0.014773061498999596,
+          0.0029681124724447727,
+          0.009226413443684578,
+          0.025997502729296684,
+          0.02632150799036026,
+          -0.0033962612506002188,
+          -0.020288852974772453,
+          0.023220321163535118,
+          0.0009445309406146407,
+          -0.0037183372769504786,
+          0.012173310853540897,
+          0.030703281983733177,
+          -0.009897565469145775,
+          0.004424204118549824,
+          -0.03403589874505997,
+          0.006742379162460566,
+          0.011371014639735222,
+          0.02465519867837429,
+          -0.004300773609429598,
+          0.004293059464544058,
+          0.004624778404831886,
+          0.009743277914822102,
+          -0.02798781543970108,
+          -0.026815228164196014,
+          -0.007201385218650103,
+          0.016508799046278,
+          -0.016092222183942795,
+          0.02301974780857563,
+          -0.014695917256176472,
+          0.017851104959845543,
+          -0.03471476584672928,
+          0.04144171625375748,
+          0.024392908439040184,
+          -0.03372732549905777,
+          -0.020134564489126205,
+          -0.013376756571233273,
+          -0.05677792802453041,
+          0.011671875603497028,
+          0.03604163974523544,
+          -0.010074996389448643,
+          -0.0062910872511565685,
+          -0.017449956387281418,
+          0.007313244044780731,
+          -0.013299612328410149,
+          -0.014773061498999596,
+          0.007475246209651232,
+          0.011810734868049622,
+          0.018452826887369156,
+          -0.008755835704505444,
+          0.029006116092205048,
+          0.012775033712387085,
+          -0.03511591628193855,
+          0.016508799046278,
+          0.030101560056209564,
+          0.010098139755427837,
+          0.024917487055063248,
+          -0.001642201328650117,
+          -0.010599575005471706,
+          -0.007598676718771458,
+          0.03597992658615112,
+          0.022479739040136337,
+          0.01562164444476366,
+          0.010838720947504044,
+          -0.010591860860586166,
+          0.026043789461255074,
+          -0.012235026806592941,
+          -0.0222020223736763,
+          0.00449749082326889,
+          -0.018375683575868607,
+          -0.00942698772996664,
+          -0.00900269579142332,
+          0.018375683575868607,
+          -0.001383769093081355,
+          -0.010429858230054379,
+          0.010260142385959625,
+          0.0023914617486298084,
+          -0.003322974545881152,
+          -0.006456946488469839,
+          -0.004354774486273527,
+          0.004744351375848055,
+          0.03397418558597565,
+          0.009218699298799038,
+          -0.010476144962012768,
+          0.03687479719519615,
+          0.007386530749499798,
+          -0.027463236823678017,
+          0.01973341591656208,
+          -0.01902369223535061,
+          -0.019949419423937798,
+          -5.614028486888856e-05,
+          0.020844288170337677,
+          -0.012289026752114296,
+          -0.02465519867837429,
+          0.0018485613400116563,
+          -0.009797278791666031,
+          0.007309386972337961,
+          0.022726600989699364,
+          0.00014801989891566336,
+          0.028496965765953064,
+          -0.017048807814717293,
+          -0.016184795647859573,
+          0.024300336837768555,
+          0.011155011132359505,
+          -0.012150168418884277,
+          0.034221045672893524,
+          -0.006279515568166971,
+          0.05955510959029198,
+          -0.0363810732960701,
+          0.003913125488907099,
+          -0.024423766881227493,
+          -0.01806710660457611,
+          -0.014155910350382328,
+          -0.040886279195547104,
+          0.029345549643039703,
+          -0.05350702628493309,
+          0.019579127430915833,
+          0.0058089373633265495,
+          -0.016678515821695328,
+          -0.0376770943403244,
+          0.0015978434821590781,
+          0.017789389938116074,
+          -0.008053826168179512,
+          -0.02808038890361786,
+          0.004204344004392624,
+          -0.0008818514761514962,
+          0.006468518171459436,
+          -0.04273773357272148,
+          0.00906441081315279,
+          0.05292073264718056,
+          -0.03363475203514099,
+          0.010993009433150291,
+          0.009635276161134243,
+          0.0032496878411620855,
+          -0.016987092792987823,
+          0.03613421320915222,
+          0.003791623981669545,
+          -0.003820552956312895,
+          0.02295803278684616,
+          -0.03443704918026924,
+          -0.04474347457289696,
+          0.019316839054226875,
+          -0.0011455873027443886,
+          0.03320274502038956,
+          -0.003004755824804306,
+          0.026938658207654953,
+          0.0062949443235993385,
+          0.023420896381139755,
+          0.027740955352783203,
+          -0.0019594556652009487,
+          0.010954436846077442,
+          -0.00828525796532631,
+          -0.014402770437300205,
+          -0.010445287451148033,
+          -0.02954612299799919,
+          0.0102987140417099,
+          0.027000373229384422,
+          -0.025133490562438965,
+          0.02048942632973194,
+          -0.03613421320915222,
+          0.007749107200652361,
+          -0.017527099698781967,
+          0.011016152799129486,
+          0.051532141864299774,
+          0.047335512936115265,
+          0.0026981087867170572,
+          -0.025287779048085213,
+          0.009642990306019783,
+          0.01743452623486519,
+          -0.024932917207479477,
+          0.031814154237508774,
+          -0.0024628196842968464,
+          -0.007976681925356388,
+          -0.013099038042128086,
+          0.031845010817050934,
+          -0.0038282673340290785,
+          -0.0073711019940674305,
+          -0.000637883844319731,
+          -0.01242017187178135,
+          0.01324561145156622,
+          -0.006090512964874506,
+          -0.01657051593065262,
+          -0.017002521082758904,
+          0.012620746158063412,
+          -0.012582174502313137,
+          0.012065310031175613,
+          0.013014179654419422,
+          0.03388161212205887,
+          0.013407614082098007,
+          -0.019116263836622238,
+          0.023405466228723526,
+          0.02638322301208973,
+          0.0657266229391098,
+          -0.012952464632689953,
+          0.012435601092875004,
+          0.007614105474203825,
+          -0.021168293431401253,
+          -0.002835039282217622,
+          -0.02895982936024666,
+          0.0015014136442914605,
+          -0.028543252497911453,
+          -0.008917837403714657,
+          -0.034190189093351364,
+          -0.023606041446328163,
+          -0.03770795091986656,
+          0.01110101118683815,
+          -0.001489842077717185,
+          -0.026198076084256172,
+          0.01902369223535061,
+          -0.006495518609881401,
+          -0.010753863491117954,
+          -0.013477043248713017,
+          -0.026136361062526703,
+          0.010090425610542297,
+          -0.029345549643039703,
+          0.003562120720744133,
+          0.006237086374312639,
+          0.012119310908019543,
+          0.00829297211021185,
+          0.019548270851373672,
+          0.00910298340022564,
+          0.018468255177140236,
+          0.0013934121234342456,
+          -0.02726266346871853,
+          -0.026876943185925484,
+          0.010931294411420822,
+          0.011201297864317894,
+          0.014896491542458534,
+          -0.013947621919214725,
+          -0.006807951256632805,
+          0.02894440107047558,
+          0.0014638060238212347,
+          -0.010792435146868229,
+          -0.022016875445842743,
+          -0.014981349930167198,
+          -0.02153858356177807,
+          -0.025688927620649338,
+          -0.001375090447254479,
+          -0.02298888936638832,
+          -0.00899498164653778,
+          -0.007116526830941439,
+          -0.0008288150420412421,
+          -0.0005269894609227777,
+          0.0005901510012336075,
+          0.01817510835826397,
+          -0.03968283534049988,
+          -0.0009941923199221492,
+          0.0003018256102222949,
+          -0.04705779254436493,
+          0.024130620062351227,
+          -0.004995069000869989,
+          0.013014179654419422,
+          -0.027355235069990158,
+          0.020335137844085693,
+          0.007922681048512459,
+          0.011749019846320152,
+          -0.0015428784536197782,
+          -0.01718766614794731,
+          0.014742203988134861,
+          0.00575107941403985,
+          -0.018730545416474342,
+          0.0016566658159717917,
+          0.028203818947076797,
+          0.016693945974111557,
+          0.00786096602678299,
+          0.014240768738090992,
+          0.01198045164346695,
+          0.005241929553449154,
+          -0.00236060400493443,
+          0.047335512936115265,
+          -0.012967893853783607,
+          0.012204169295728207,
+          0.017527099698781967,
+          -0.005203357897698879,
+          -0.0003975322761107236,
+          0.006510947365313768,
+          -0.004740494303405285,
+          0.0062949443235993385,
+          0.003673979314044118,
+          0.029438121244311333,
+          -0.00906441081315279,
+          0.018452826887369156,
+          0.034961625933647156,
+          -0.014734489843249321,
+          0.0008148326887749135,
+          -0.005461789667606354,
+          0.007020097225904465,
+          0.009704705327749252,
+          -0.014256197027862072,
+          -0.014911920763552189,
+          0.028157532215118408,
+          -0.01199587993323803,
+          0.02953069470822811,
+          -0.004705779254436493,
+          0.014626488089561462,
+          0.03527020290493965,
+          0.00764882005751133,
+          0.004250630270689726,
+          -0.00861697643995285,
+          -0.0078146792948246,
+          0.013315041549503803,
+          -0.021260865032672882,
+          0.011679589748382568,
+          -0.021245436742901802,
+          0.027679240331053734,
+          -0.031690724194049835,
+          -0.005049069877713919,
+          -0.012636174447834492,
+          -0.01112415362149477,
+          0.02474777027964592,
+          0.0017627386841922998,
+          0.0034714764915406704,
+          -0.0057587940245866776,
+          -0.013369042426347733,
+          0.01110872533172369,
+          0.01965627260506153,
+          -0.007166670635342598,
+          0.012960179708898067,
+          0.004632492549717426,
+          -0.030286705121397972,
+          -0.014965921640396118,
+          0.030147844925522804,
+          0.0029777553863823414,
+          -0.021862588822841644,
+          0.014379628002643585,
+          -0.0062910872511565685,
+          -0.004401060752570629,
+          0.036535363644361496,
+          -0.009851278737187386,
+          -0.0074328170157969,
+          -0.0009223520755767822,
+          -0.001847596955485642,
+          -0.001201516599394381,
+          0.02394547499716282,
+          0.021785443648695946,
+          -0.02641407959163189,
+          -0.007999825291335583,
+          -0.014009336940944195,
+          0.007498389575630426,
+          -0.013816476799547672,
+          0.02371404320001602,
+          0.031783297657966614,
+          -0.00948870275169611,
+          0.00596708245575428,
+          0.004536062944680452,
+          0.013199325650930405,
+          -0.01517421007156372,
+          -0.023374609649181366,
+          -0.0017212738748639822,
+          0.005943939555436373,
+          -0.01569107361137867,
+          -0.006746236234903336,
+          -0.04384860768914223,
+          -0.011749019846320152,
+          -0.0026730368845164776,
+          -0.029268404468894005,
+          -0.04011484235525131,
+          -0.031845010817050934,
+          -0.005546648055315018,
+          0.02052028477191925,
+          0.016771089285612106,
+          0.0017193452222272754,
+          0.02386832982301712,
+          -0.005724078975617886,
+          0.005623792298138142,
+          0.021924303844571114,
+          0.00011264218483120203,
+          0.015073922462761402,
+          -0.0006962239276617765,
+          0.038170814514160156,
+          0.014665059745311737,
+          -0.0013162682298570871,
+          -0.05671621486544609,
+          -0.0188385471701622,
+          0.012937036342918873,
+          0.021168293431401253,
+          -0.0041271997615695,
+          0.02139972522854805,
+          -0.005245787091553211,
+          -0.020859716460108757,
+          -0.012543601915240288,
+          0.007035525981336832,
+          0.0037067655939608812,
+          0.007733678445219994,
+          0.0048176380805671215,
+          0.0036103357560932636,
+          0.029345549643039703,
+          0.026753513142466545,
+          0.00048239060561172664,
+          0.008940980769693851,
+          0.04224401339888573,
+          -0.006121370475739241,
+          0.0008119398262351751,
+          -0.00742510287091136,
+          0.022402595728635788,
+          -0.018653400242328644,
+          0.02377575822174549,
+          0.019995706155896187,
+          0.008771263994276524,
+          0.023467181250452995,
+          0.016169367358088493,
+          -0.00515707116574049,
+          -0.009226413443684578,
+          0.032647307962179184,
+          -0.010437573306262493,
+          -0.029098687693476677,
+          -0.007228385657072067,
+          -0.012975607998669147,
+          0.011826164089143276,
+          0.005608363542705774,
+          0.01191873662173748,
+          0.0033789039589464664,
+          0.0006263122195377946,
+          -0.02129172347486019,
+          -0.011949594132602215,
+          -0.00022540490317624062,
+          0.007760678883641958,
+          0.011193583719432354,
+          0.017465384677052498,
+          -0.0034406189806759357,
+          -0.018607115373015404,
+          0.033387891948223114,
+          -0.020782573148608208,
+          0.041565146297216415,
+          0.00011203949543414637,
+          -0.023189464583992958,
+          0.004038484301418066,
+          -0.00030399527167901397,
+          0.023405466228723526,
+          -0.009982423856854439,
+          -0.019069978967308998,
+          0.017943676561117172,
+          0.06066598370671272,
+          0.012273598462343216,
+          -0.016231082379817963,
+          -0.00787639431655407,
+          -0.02970041148364544,
+          0.01806710660457611,
+          0.006171513814479113,
+          0.026630083099007607,
+          0.02800324372947216,
+          0.021662013605237007,
+          -0.0042776307091116905,
+          0.02314317785203457,
+          -0.014626488089561462,
+          0.010530145838856697,
+          -0.004532205406576395,
+          0.02295803278684616,
+          0.0036450503394007683,
+          -0.013500186614692211,
+          0.034282758831977844,
+          0.0021793157793581486,
+          -0.002719323383644223,
+          -0.015590786933898926,
+          0.00575107941403985,
+          0.004875496029853821,
+          -0.00640294561162591,
+          0.02633693628013134,
+          0.03059528023004532,
+          0.003114685881882906,
+          -0.021631157025694847,
+          -0.010993009433150291,
+          0.016801945865154266,
+          0.020319709554314613,
+          0.0015920577570796013,
+          0.016925377771258354,
+          0.013816476799547672,
+          0.033295318484306335,
+          -0.0029989699833095074,
+          -0.0008153148810379207,
+          0.0041271997615695,
+          -0.0058937957510352135,
+          0.005461789667606354,
+          0.011417300440371037,
+          0.034190189093351364,
+          -0.00020033313194289804,
+          -0.018499113619327545,
+          0.009581275284290314,
+          0.011826164089143276,
+          -0.039034824818372726,
+          0.004393346607685089,
+          -0.022510597482323647,
+          0.0006914024124853313,
+          0.00992070883512497,
+          -0.022865459322929382,
+          -0.021615726873278618,
+          0.004046198911964893,
+          0.013029608875513077,
+          0.018360253423452377,
+          0.0156293585896492,
+          -0.019378554075956345,
+          -0.014194482006132603,
+          0.02718551829457283,
+          0.002271888544782996,
+          0.009781849570572376,
+          0.024346621707081795,
+          -0.006599662825465202,
+          0.0015824147267267108,
+          0.019116263836622238,
+          0.004138771444559097,
+          0.03150557726621628,
+          -0.002854325110092759,
+          -0.028311820700764656,
+          0.012466458603739738,
+          0.006210085935890675,
+          -0.009650705382227898,
+          -0.006896666716784239,
+          -0.027694668620824814,
+          -0.015035350807011127,
+          -0.030240418389439583,
+          0.003689408302307129,
+          -0.02305060438811779,
+          -0.0016373797552660108,
+          0.012235026806592941,
+          -0.011471301317214966,
+          0.006437660660594702,
+          -0.013777905143797398,
+          -0.02389918826520443,
+          -0.003689408302307129,
+          -0.0001803239283617586,
+          -0.012975607998669147,
+          -0.03033299185335636,
+          -0.02132258005440235,
+          -0.017835674807429314,
+          -0.014317912049591541,
+          -0.012073024176061153,
+          -0.019532842561602592,
+          -0.0008567796903662384,
+          0.02056656964123249,
+          0.007907251827418804,
+          -0.035085055977106094,
+          0.004208201076835394,
+          -0.004289202392101288,
+          -0.010128997266292572,
+          0.008871551603078842,
+          -0.02135343849658966,
+          -0.00038403208600357175,
+          0.02223287895321846,
+          0.011440443806350231,
+          -0.0165859442204237,
+          -0.004188915248960257,
+          -0.010954436846077442,
+          0.03059528023004532,
+          0.0094192735850811,
+          -0.008223542012274265,
+          -0.0058860816061496735,
+          0.024192335084080696,
+          -0.0023471040185540915,
+          -0.00023818186309654266,
+          -0.0024821057450026274,
+          -0.0009724955889396369,
+          0.0008399044745601714,
+          -0.024269478395581245,
+          0.030857570469379425,
+          0.00450520496815443,
+          0.002742466516792774,
+          -0.024979202076792717,
+          0.024315765127539635,
+          0.02314317785203457,
+          -0.030764997005462646,
+          0.0073749590665102005,
+          0.0012931249802932143,
+          0.010653575882315636,
+          0.00905669666826725,
+          0.0026267506182193756,
+          -0.008809836581349373,
+          0.03221530467271805,
+          -0.025256920605897903,
+          0.008123255334794521,
+          0.012929322198033333,
+          0.040053125470876694,
+          0.02226373739540577,
+          -0.007301672361791134,
+          0.009473273530602455,
+          0.020119136199355125,
+          0.013515615835785866,
+          0.017835674807429314,
+          -0.017665958032011986,
+          -0.01272874791175127,
+          0.0001393412094330415,
+          -0.0038109098095446825,
+          0.010468430817127228,
+          0.0013490543933585286,
+          -0.03155186399817467,
+          0.005631506443023682,
+          0.01067671924829483,
+          -0.007560104597359896,
+          -0.019285980612039566,
+          0.023667756468057632,
+          -0.004019198473542929,
+          0.007020097225904465,
+          -0.003754980629310012,
+          -0.006499375682324171,
+          0.011571588926017284,
+          -0.05936996266245842,
+          -0.020057421177625656,
+          0.008871551603078842,
+          0.0053036450408399105,
+          0.01447220053523779,
+          0.010020995512604713,
+          -0.02963869646191597,
+          -0.007197528146207333,
+          -0.02874382585287094,
+          0.018344825133681297,
+          -0.00987442210316658,
+          0.020057421177625656,
+          0.0221248771995306,
+          -0.008493546396493912,
+          0.0058436524122953415,
+          -0.003068399615585804,
+          0.018637971952557564,
+          -0.012451029382646084,
+          0.022603169083595276,
+          -0.016801945865154266,
+          -0.015004493296146393,
+          -0.00944241601973772,
+          0.011386442929506302,
+          -0.007938110269606113,
+          0.01449534296989441,
+          -0.024238621816039085,
+          -0.02141515351831913,
+          0.007922681048512459,
+          0.013531044125556946,
+          0.014603344723582268,
+          -0.01806710660457611,
+          0.007999825291335583,
+          -0.005616077687591314,
+          0.01105472445487976,
+          0.006152227986603975,
+          0.009712420403957367,
+          -0.0003524513158481568,
+          -0.022649455815553665,
+          -0.008354687131941319,
+          0.007563961669802666,
+          0.0008644941262900829,
+          0.004250630270689726,
+          -0.008794407360255718,
+          -0.029021544381976128,
+          0.007787679322063923,
+          -0.014896491542458534,
+          -0.019347695633769035,
+          -0.01653965748846531,
+          -0.009689277037978172,
+          -0.010838720947504044,
+          0.0018543470650911331,
+          0.02389918826520443,
+          -0.020072849467396736,
+          -0.010607290081679821,
+          0.01323789730668068,
+          -0.013083609752357006,
+          0.016153937205672264,
+          0.0035949070006608963,
+          -0.007629534229636192,
+          0.00552350515499711,
+          0.0003452190721873194,
+          0.03684393689036369,
+          0.0019903131760656834,
+          -0.02152315527200699,
+          0.01312218140810728,
+          -0.015498214401304722,
+          0.02382204309105873,
+          0.007976681925356388,
+          -0.0021349580492824316,
+          0.011571588926017284,
+          0.024886630475521088,
+          0.0027366806752979755,
+          0.020443139597773552,
+          -0.011008437722921371,
+          -0.029484407976269722,
+          0.007961252704262733,
+          -0.008817550726234913,
+          -0.01991856098175049,
+          -0.0198722742497921,
+          0.003754980629310012,
+          -0.010931294411420822,
+          0.023575183004140854,
+          -0.022649455815553665,
+          -0.0003042363387066871,
+          0.00038089812733232975,
+          0.0015814504586160183,
+          -0.014888777397572994,
+          0.013585045002400875,
+          0.007968967780470848,
+          -0.017542527988553047,
+          0.00779925100505352,
+          -0.01652422919869423,
+          -0.02130715176463127,
+          -2.7196247174288146e-05,
+          -0.008516688831150532,
+          -0.0022622456308454275,
+          -0.017064236104488373,
+          -0.0030201845802366734,
+          0.00033726359833963215,
+          -0.0006306516006588936,
+          0.016956234350800514,
+          0.019162550568580627,
+          -0.007745250128209591,
+          0.01894654706120491,
+          0.003976769279688597,
+          0.02209402061998844,
+          0.0031648294534534216,
+          -0.006985382176935673,
+          -0.022402595728635788,
+          0.02642950788140297,
+          -0.02567349746823311,
+          -0.0024338907096534967,
+          0.01235074270516634,
+          -0.014904206618666649,
+          -0.0012005523312836885,
+          -0.0297312680631876,
+          -0.004678778816014528,
+          -0.0014011265011504292,
+          -0.0018177037127315998,
+          0.04551491513848305,
+          -0.003259330755099654,
+          -0.012967893853783607,
+          -0.0131067531183362,
+          0.0006841701688244939,
+          0.0066112345084548,
+          0.005006640683859587,
+          0.01640079915523529,
+          -0.01154073141515255,
+          -0.02391461655497551,
+          0.016030507162213326,
+          -0.005855224095284939,
+          0.01647794246673584,
+          0.020196279510855675,
+          0.0026286791544407606,
+          -0.03307931497693062,
+          0.012505030259490013,
+          -0.01905454881489277,
+          -0.00944241601973772,
+          0.019270552322268486,
+          0.02801867388188839,
+          -0.00867869146168232,
+          -0.01072300598025322,
+          0.011633303947746754,
+          -0.000418505776906386,
+          -0.01066900510340929,
+          -0.02479405701160431,
+          -0.009403844363987446,
+          -0.013762475922703743,
+          0.013731618411839008,
+          0.006113655865192413,
+          -0.02065914310514927,
+          0.006483946926891804,
+          -0.006414517294615507,
+          0.01230445597320795,
+          -0.01154073141515255,
+          0.003432904602959752,
+          0.018514541909098625,
+          0.02150772698223591,
+          -0.0230814628303051,
+          0.025827785953879356,
+          -0.027648381888866425,
+          0.021075719967484474,
+          -0.0008799228817224503,
+          0.026707226410508156,
+          -0.029484407976269722,
+          -0.018313968554139137,
+          -0.013006465509533882,
+          -0.011378728784620762,
+          -0.025195205584168434,
+          -0.006078941281884909,
+          0.01441048551350832,
+          0.013970764353871346,
+          0.009311271831393242,
+          0.026799799874424934,
+          -0.012096167542040348,
+          0.018653400242328644,
+          0.02064371481537819,
+          -0.01193416491150856,
+          0.005681650247424841,
+          0.011733590625226498,
+          -0.009558131918311119,
+          0.012813606299459934,
+          0.001909312093630433,
+          0.010445287451148033,
+          -0.033264461904764175,
+          0.022078590467572212,
+          0.014819348230957985,
+          -0.0037299087271094322,
+          -0.03872625157237053,
+          -0.0016798089491203427,
+          -0.020057421177625656,
+          -0.02315860614180565,
+          -0.0014310197439044714,
+          0.011957308277487755,
+          0.028682110831141472,
+          0.018622543662786484,
+          -0.036689650267362595,
+          -0.000610401330050081,
+          -0.01579907536506653,
+          -0.011232155375182629,
+          -0.014024765230715275,
+          0.012921607121825218,
+          -0.009018125012516975,
+          0.014958206564188004,
+          -0.0004170593456365168,
+          0.002379890065640211,
+          -0.003363475203514099,
+          -0.01231217011809349,
+          0.016138508915901184,
+          0.010275570675730705,
+          -0.009457845240831375,
+          0.010429858230054379,
+          0.013816476799547672,
+          0.0026074645575135946,
+          -0.004644064232707024,
+          -0.012597602792084217,
+          -0.02485577203333378,
+          0.011617874726653099,
+          -0.0018205966334789991,
+          0.007498389575630426,
+          -0.009797278791666031,
+          0.0034676194190979004,
+          -0.047366369515657425,
+          0.003726051654666662,
+          0.0032265447080135345,
+          0.030641566962003708,
+          0.011749019846320152,
+          0.018622543662786484,
+          0.014919634908437729,
+          0.0005795437609776855,
+          0.010591860860586166,
+          0.02127629518508911,
+          0.03786223754286766,
+          0.00537307420745492,
+          0.014896491542458534,
+          -5.394048002926866e-06,
+          -0.007563961669802666,
+          0.0074559603817760944,
+          -0.006125227548182011,
+          -0.02062828466296196,
+          -0.006360516417771578,
+          -0.00020177957776468247,
+          0.0008881194517016411,
+          -0.00860926229506731,
+          0.004308488219976425,
+          -0.017465384677052498,
+          -0.010838720947504044,
+          -0.02794152870774269,
+          0.00216581579297781,
+          -0.025118062272667885,
+          -0.015891648828983307,
+          -0.016308225691318512,
+          -0.010869579389691353,
+          -0.003953625913709402,
+          0.022772885859012604,
+          -0.0049063535407185555,
+          -0.033418748527765274,
+          -0.0037029082886874676,
+          0.02727809175848961,
+          -0.008269828744232655,
+          -0.010530145838856697,
+          0.011725876480340958,
+          0.010059568099677563,
+          -0.008022967725992203,
+          0.01362361665815115,
+          0.010229283943772316,
+          -0.009357557632029057,
+          -0.0013104823883622885,
+          -0.01560621615499258,
+          0.02963869646191597,
+          0.017110522836446762,
+          0.002422319259494543,
+          -0.004528348334133625,
+          -0.017449956387281418,
+          -0.0016084507806226611,
+          -0.0013914835872128606,
+          0.02480948530137539,
+          -0.021569442003965378,
+          0.027756383642554283,
+          -0.0031474721617996693,
+          -0.02227916568517685,
+          0.013747047632932663,
+          0.02055114135146141,
+          -0.02721637673676014,
+          0.0010732648661360145,
+          -0.006588091142475605,
+          0.015505928546190262,
+          -0.008223542012274265,
+          -0.0057665081694722176,
+          -0.010622718371450901,
+          0.007517675403505564,
+          0.013214753940701485,
+          -0.01967170089483261,
+          0.017542527988553047,
+          0.0110624385997653,
+          0.01280589122325182,
+          -0.02547292411327362,
+          0.013307327404618263,
+          -0.006213943008333445,
+          -0.01275189034640789,
+          -0.0020751715637743473,
+          -0.004825352691113949,
+          0.017573386430740356,
+          0.006059655454009771,
+          0.002231388119980693,
+          -0.00758324796333909,
+          0.003984483890235424,
+          -0.006244800519198179,
+          0.010514716617763042,
+          0.011432729661464691,
+          -0.020273422822356224,
+          -0.004377917852252722,
+          -0.013006465509533882,
+          -0.009349843487143517,
+          0.003344189142808318,
+          -0.007178242318332195,
+          0.03650450333952904,
+          0.00986670795828104,
+          0.004296916536986828,
+          0.02231002226471901,
+          -0.011139582842588425,
+          -0.012913892976939678,
+          0.01660137251019478,
+          0.09436244517564774,
+          3.441342414589599e-05,
+          -0.003969055134803057,
+          0.002773324027657509,
+          -0.010476144962012768,
+          -0.0008866729913279414,
+          -0.01235074270516634,
+          -0.018468255177140236,
+          0.008123255334794521,
+          -0.00514549994841218,
+          -0.021739158779382706,
+          -0.009149269200861454,
+          -0.020011134445667267,
+          0.017141379415988922,
+          0.003322974545881152,
+          0.012242740951478481,
+          0.013939906843006611,
+          0.01905454881489277,
+          0.0031899013556540012,
+          0.014819348230957985,
+          -0.009519560262560844,
+          0.007243814412504435,
+          -0.011347871273756027,
+          0.007263100706040859,
+          0.008123255334794521,
+          0.009210984222590923,
+          0.006244800519198179,
+          0.015012207441031933,
+          0.012003595009446144,
+          0.029006116092205048,
+          0.010283284820616245,
+          -0.0012179097393527627,
+          -0.005049069877713919,
+          -0.016693945974111557,
+          -0.003693265374749899,
+          0.00904898252338171,
+          -0.01984141767024994,
+          -0.022634027525782585,
+          -0.013693046756088734,
+          -0.002790681552141905,
+          0.0019382410682737827,
+          -0.008940980769693851,
+          0.023405466228723526,
+          0.026954086497426033,
+          -0.006800237111747265,
+          0.004269916098564863,
+          -0.007479103282094002,
+          -0.0011108724866062403,
+          -0.00866326317191124,
+          -0.010514716617763042,
+          -0.0013066251995041966,
+          -0.0013596616918221116,
+          0.005295930430293083,
+          -0.021939732134342194,
+          -0.013677617534995079,
+          -0.004107913933694363,
+          0.009766421280801296,
+          -0.00040548774995841086,
+          0.0024956059642136097,
+          0.011363300494849682,
+          -0.01393219269812107,
+          -0.004944925662130117,
+          0.00558136310428381,
+          -0.009550417773425579,
+          -0.008385544642806053,
+          0.006275658495724201,
+          0.012134739197790623,
+          -0.0010173354530707002,
+          -0.011725876480340958,
+          -0.012613032013177872,
+          -0.010453001596033573,
+          0.006109798792749643,
+          0.0221248771995306,
+          -0.007548532914370298,
+          0.008501260541379452,
+          -0.008424116298556328,
+          0.003648907644674182,
+          -0.017727673053741455,
+          0.003157115075737238,
+          -0.0038880538195371628,
+          -0.0021542441099882126,
+          0.00992842298001051,
+          -0.006607376970350742,
+          -0.00817725621163845,
+          0.00011372702283551916,
+          0.0006518661393783987,
+          -0.007228385657072067,
+          0.02132258005440235,
+          0.028203818947076797,
+          0.029931843280792236,
+          0.002574678510427475,
+          -0.02629064954817295,
+          0.02312774769961834,
+          -0.009681562893092632,
+          -0.009573561139404774,
+          0.009156984277069569,
+          0.0005901510012336075,
+          -0.009411558508872986,
+          -0.01896197721362114,
+          -0.002981612691655755,
+          0.005230357870459557,
+          0.019162550568580627,
+          0.00224681687541306,
+          -0.0018312039319425821,
+          -0.022880887612700462,
+          0.003407832933589816,
+          0.02727809175848961,
+          0.0030780425295233727,
+          -0.009789563715457916,
+          -0.008339257910847664,
+          0.020813431590795517,
+          -0.003297902876511216,
+          0.004624778404831886,
+          0.003791623981669545,
+          -0.008339257910847664,
+          -0.003342260606586933,
+          -0.004941068589687347,
+          0.0026518225204199553,
+          -0.01805167831480503,
+          0.009758706204593182,
+          0.01116272620856762,
+          -0.010499288327991962,
+          0.004215915687382221,
+          -0.012705604545772076,
+          0.006210085935890675,
+          -0.0004380328464321792,
+          0.0077761076390743256,
+          -0.0015698788920417428,
+          -0.012289026752114296,
+          -0.02971583977341652,
+          -0.01185702160000801,
+          -0.01817510835826397,
+          -0.014634202234447002,
+          0.0007473317673429847,
+          0.02784895710647106,
+          -0.004266059026122093,
+          0.007563961669802666,
+          0.001351947314105928,
+          -0.009974709711968899,
+          0.012188740074634552,
+          -0.020026562735438347,
+          0.01480391900986433,
+          -0.013639045879244804,
+          0.013091323897242546,
+          0.015451927669346333,
+          0.01797453500330448,
+          -0.009396130219101906,
+          0.0004013894940726459,
+          -0.0020983146969228983,
+          -0.008100111968815327,
+          -0.00537307420745492,
+          -0.012813606299459934,
+          -0.008053826168179512,
+          -0.017635101452469826,
+          0.015328497625887394,
+          0.024099761620163918,
+          0.0009990137768909335,
+          0.01728023961186409,
+          0.0008259221212938428,
+          -1.7749129256117158e-05,
+          -0.01111643947660923,
+          -0.004092485178261995,
+          0.014348769560456276,
+          0.005068355705589056,
+          -0.025210633873939514,
+          -0.0036334788892418146,
+          0.01652422919869423,
+          0.006102084647864103,
+          -0.01393219269812107,
+          0.013469329103827477,
+          -0.003799338359385729,
+          -0.0021812445484101772,
+          -0.01355418749153614,
+          -0.0009030660730786622,
+          0.007675820495933294,
+          0.0052727870643138885,
+          -0.021122006699442863,
+          -0.008786693215370178,
+          0.0032496878411620855,
+          0.012435601092875004,
+          -0.004258344415575266,
+          0.00309347128495574,
+          0.01885397545993328,
+          -0.00536150299012661,
+          0.03465305268764496,
+          -0.012520458549261093,
+          -0.014503058046102524,
+          0.0020713144913315773,
+          0.020134564489126205,
+          -0.0062563722021877766,
+          -0.0007400995236821473,
+          0.011340157128870487,
+          0.0005506147863343358,
+          -0.008161826990544796,
+          0.02974669821560383,
+          0.006040369160473347,
+          0.01738824136555195,
+          -0.0010877293534576893,
+          0.014595630578696728,
+          -0.001114729675464332,
+          0.0002372175658820197,
+          -0.0009802100248634815,
+          -0.01035271491855383,
+          -0.005068355705589056,
+          -0.020026562735438347,
+          -0.006009511649608612,
+          -0.020273422822356224,
+          0.029160402715206146,
+          0.012389314360916615,
+          0.007494532503187656,
+          -0.02638322301208973,
+          0.002289246069267392,
+          0.006441517733037472,
+          -0.007170527707785368,
+          -0.006900524254888296,
+          -0.012204169295728207,
+          -0.022032305598258972,
+          0.009318985976278782,
+          0.007259243167936802,
+          0.010568717494606972,
+          0.004250630270689726,
+          -0.010182998143136501,
+          0.0032034015748649836,
+          -0.018529970198869705,
+          0.012705604545772076,
+          0.0024801772087812424,
+          0.003133971942588687,
+          0.0016942734364420176,
+          -0.0034483333583921194,
+          0.014665059745311737,
+          -3.730631942744367e-05,
+          0.011039295233786106,
+          0.014094195328652859,
+          0.01111643947660923,
+          0.007976681925356388,
+          0.004401060752570629,
+          0.010576431639492512,
+          -0.00816954206675291,
+          -0.030734140425920486,
+          -0.0264449380338192,
+          0.004080913495272398,
+          -0.01110872533172369,
+          -0.009627562016248703,
+          0.007984396070241928,
+          0.011139582842588425,
+          0.01069214753806591,
+          -0.016323653981089592,
+          -0.005666221491992474,
+          -0.0043200599029660225,
+          -2.2917169189895503e-05,
+          -0.011787591502070427,
+          -0.0007203313871286809,
+          0.012157882563769817,
+          0.002040456747636199,
+          0.006491661071777344,
+          0.01894654706120491,
+          -0.00906441081315279,
+          -0.007826250977814198,
+          0.01567564532160759,
+          -0.0074328170157969,
+          -0.019625414162874222,
+          -0.008910123258829117,
+          0.016215652227401733,
+          -0.008215827867388725,
+          0.011286156252026558,
+          0.009565846994519234,
+          0.010491574183106422,
+          -0.016632230952382088,
+          -0.007999825291335583,
+          0.02804953046143055,
+          0.03230787441134453,
+          -0.010807863436639309,
+          -0.002273817081004381,
+          0.002725109225139022,
+          -0.012127025052905083,
+          0.021122006699442863,
+          0.00860926229506731,
+          -0.010113568976521492,
+          0.016987092792987823,
+          0.0026903944090008736,
+          0.008138683624565601,
+          -0.0036238357424736023,
+          0.0012304455740377307,
+          -0.017604243010282516,
+          -0.018375683575868607,
+          -0.00948098860681057,
+          0.009164698421955109,
+          0.006742379162460566,
+          0.0073749590665102005,
+          8.919765969039872e-05,
+          0.011093296110630035,
+          -0.011332442983984947,
+          0.005604506004601717,
+          0.0008119398262351751,
+          0.027740955352783203,
+          -0.029885556548833847,
+          0.019903132691979408,
+          -0.003267045132815838,
+          -0.0018283110111951828,
+          -0.013585045002400875,
+          -0.012844463810324669,
+          -0.020828859880566597,
+          -0.011209012009203434,
+          0.014973635785281658,
+          0.016076793894171715,
+          -0.0005472397315315902,
+          -0.006908238399773836,
+          -0.002619036240503192,
+          0.0057973661459982395,
+          -0.006861952133476734,
+          0.002761752577498555,
+          -0.0017646672204136848,
+          0.0014956278027966619,
+          -0.0020751715637743473,
+          -0.015559929423034191,
+          0.007444388698786497,
+          -0.003652764717116952,
+          -0.0015669859712943435,
+          -0.018761401996016502,
+          0.03604163974523544,
+          -0.013060466386377811,
+          0.00904126837849617,
+          0.014155910350382328,
+          0.0018128822557628155,
+          -0.011471301317214966,
+          -0.013777905143797398,
+          0.011216727085411549,
+          -0.02149229682981968,
+          -0.021955160424113274,
+          -0.004096342250704765,
+          0.0223563089966774,
+          -0.015760503709316254,
+          -0.004046198911964893,
+          0.019532842561602592,
+          -0.006271800957620144,
+          -0.009658419527113438,
+          -0.022680314257740974,
+          -0.016231082379817963,
+          -1.1903066479135305e-05,
+          -0.016801945865154266,
+          -0.017912819981575012,
+          -0.017079664394259453,
+          -0.007864823564887047,
+          -0.01970255747437477,
+          -0.0006914024124853313,
+          0.0033924041781574488,
+          -0.01805167831480503,
+          -0.022695742547512054,
+          0.01275189034640789,
+          0.00818497035652399,
+          -0.0026306079234927893,
+          0.008524403907358646,
+          0.00596322538331151,
+          0.0027231804560869932,
+          -2.3640393919777125e-05,
+          0.007035525981336832,
+          0.02892897091805935,
+          0.017017949372529984,
+          0.006599662825465202,
+          0.017002521082758904,
+          -0.0062988013960421085,
+          -0.01029099989682436,
+          -0.010321857407689095,
+          -0.004169628955423832,
+          0.010445287451148033,
+          -0.019964847713708878,
+          0.033357031643390656,
+          -0.006202371325343847,
+          -0.018730545416474342,
+          -0.006410660222172737,
+          0.00987442210316658,
+          -0.010476144962012768,
+          -0.019532842561602592,
+          -0.007922681048512459,
+          -0.006885095499455929,
+          0.016092222183942795,
+          -0.0032901884987950325,
+          0.00898726750165224,
+          0.010113568976521492,
+          -0.015421070158481598,
+          0.010823292657732964,
+          -0.006240943446755409,
+          -0.014271626248955727,
+          0.025195205584168434,
+          0.022448882460594177,
+          -0.005064498633146286,
+          -0.007826250977814198,
+          -0.002401104662567377,
+          0.021245436742901802,
+          0.008940980769693851,
+          -0.0003459422732703388,
+          0.009172412566840649,
+          0.005608363542705774,
+          0.003905411111190915,
+          0.0024608911480754614,
+          0.001322053954936564,
+          0.011224441230297089,
+          0.03311017155647278,
+          0.030147844925522804,
+          -0.008046111091971397,
+          -0.002551535377278924,
+          0.01528992597013712,
+          0.00899498164653778,
+          0.028280962258577347,
+          -0.0031455436255782843,
+          -0.035023342818021774,
+          0.001998027553781867,
+          -0.008447259664535522,
+          0.014703632332384586,
+          0.0006648842245340347,
+          0.015876220539212227,
+          0.0017540600383654237,
+          -0.0014879134250804782,
+          0.018514541909098625,
+          0.019316839054226875,
+          0.00617922842502594,
+          -0.017635101452469826,
+          -0.01312989555299282,
+          0.008740406483411789,
+          0.006514804437756538,
+          -0.005685507319867611,
+          -0.012327599339187145,
+          -0.00015826558228582144,
+          0.029808413237333298,
+          -0.010198426432907581,
+          -0.017449956387281418,
+          0.01242017187178135,
+          0.011525302194058895,
+          0.007787679322063923,
+          -0.025951215997338295,
+          0.004169628955423832,
+          0.008354687131941319,
+          0.014819348230957985,
+          -0.00899498164653778,
+          -0.015004493296146393,
+          -0.000610401330050081,
+          0.011193583719432354,
+          -0.016261938959360123,
+          0.001589164836332202,
+          0.02218659222126007,
+          -0.02053571306169033,
+          0.03122786059975624,
+          -0.010059568099677563,
+          0.017804818227887154,
+          0.003477262333035469,
+          0.020088277757167816,
+          0.006495518609881401,
+          0.0012304455740377307,
+          0.004161914810538292,
+          0.013885905966162682,
+          -0.003245830535888672,
+          0.011795305646955967,
+          -0.01485020574182272,
+          -0.004536062944680452,
+          0.0003758355451282114,
+          -0.0021581011824309826,
+          -0.0024512482341378927,
+          0.008215827867388725,
+          0.02226373739540577,
+          -0.01581450365483761,
+          0.0015843433793634176,
+          -0.005284358747303486,
+          0.006800237111747265,
+          0.023312894627451897,
+          -0.011903307400643826,
+          -0.009519560262560844,
+          0.012890749610960484,
+          0.019085407257080078,
+          -0.0053537883795797825,
+          -0.027478665113449097,
+          -0.0015043065650388598,
+          0.014680488966405392,
+          0.017789389938116074,
+          -0.0031821869779378176,
+          0.036658793687820435,
+          -0.013060466386377811,
+          -0.024038046598434448,
+          -0.008856122381985188,
+          0.03789309412240982,
+          0.004605492111295462,
+          -0.005095356144011021,
+          0.0015264854300767183,
+          -0.010152140632271767,
+          0.008393258787691593,
+          0.015066208317875862,
+          -0.00662280572578311,
+          -0.01976427435874939,
+          0.01726480945944786,
+          -0.0153053542599082,
+          -0.009835850447416306,
+          -0.002852396573871374,
+          -0.005477218423038721,
+          -0.009450131095945835,
+          0.012481886893510818,
+          -0.01449534296989441,
+          0.0003293081244919449,
+          0.000391987559851259,
+          0.0029063974507153034,
+          0.004636350087821484,
+          0.013855048455297947,
+          -0.010831006802618504,
+          0.007679677568376064,
+          0.016061365604400635,
+          -0.0013828048249706626,
+          0.016724802553653717,
+          0.007131956052035093,
+          -0.007093383930623531,
+          -0.013446185737848282,
+          0.03212273120880127,
+          -0.0024531767703592777,
+          0.016076793894171715,
+          -0.011401872150599957,
+          -0.008215827867388725,
+          -0.02559635415673256,
+          -0.010576431639492512,
+          0.028358107432723045,
+          0.004952639807015657,
+          -0.024099761620163918,
+          0.003521620063111186,
+          0.0347764827311039,
+          0.00823125708848238,
+          0.006507089827209711,
+          0.008431831374764442,
+          0.009542703628540039,
+          0.005137785337865353,
+          -0.0022873173002153635,
+          -0.0016113437013700604,
+          -0.00553121929988265,
+          0.009581275284290314,
+          0.0021214578300714493,
+          -0.005130071192979813,
+          -0.013099038042128086,
+          -0.01360818836838007,
+          -0.028280962258577347,
+          0.005928510800004005,
+          -0.03517762944102287,
+          0.015876220539212227,
+          -0.02136886678636074,
+          0.02385290153324604,
+          -0.004871638957411051,
+          -0.005616077687591314,
+          -0.020211707800626755,
+          0.008539832197129726,
+          -0.002129172207787633,
+          0.006352802272886038,
+          0.002316246274858713,
+          -0.004316202364861965,
+          0.012204169295728207,
+          -0.012188740074634552,
+          -0.024254050105810165,
+          0.028527824208140373,
+          -0.006067369598895311,
+          -0.02480948530137539,
+          0.0025843214243650436,
+          0.003236187621951103,
+          0.006499375682324171,
+          0.005635363515466452,
+          -0.03227701783180237,
+          0.0025669641327112913,
+          -0.003905411111190915,
+          0.009234127588570118,
+          0.006171513814479113,
+          -0.028666682541370392,
+          0.023544326424598694,
+          0.008917837403714657,
+          -0.0013567687710747123,
+          -0.007891823537647724,
+          -0.018715115264058113,
+          0.00014283679774962366,
+          0.0042390585877001286,
+          0.011687304824590683,
+          0.004354774486273527,
+          0.0015592715935781598,
+          -0.011093296110630035,
+          0.0029642554000020027,
+          -0.011216727085411549,
+          0.016647659242153168,
+          -0.020134564489126205,
+          0.011633303947746754,
+          -0.005018212366849184,
+          -0.005110784899443388,
+          -0.0036373359616845846,
+          0.007899537682533264,
+          -0.01404019445180893,
+          0.02872839756309986,
+          0.005133928265422583,
+          -0.014533915556967258,
+          -0.012065310031175613,
+          0.012026737444102764,
+          0.0069930967874825,
+          0.011031581088900566,
+          0.0014358413172885776,
+          -0.005677792709320784,
+          0.0035794780123978853,
+          0.0027771813329309225,
+          0.01231217011809349,
+          -0.0030298277270048857,
+          0.0188385471701622,
+          0.011401872150599957,
+          -0.0030626137740910053,
+          -0.0058976528234779835,
+          -0.00829297211021185,
+          2.452935768815223e-05,
+          -0.01715680956840515,
+          0.011371014639735222,
+          0.0068349516950547695,
+          -0.010460715740919113,
+          0.002080957405269146,
+          0.005006640683859587,
+          -0.001408840878866613,
+          0.013361327350139618,
+          -0.001426198286935687,
+          0.0005809901631437242,
+          1.5458919733646326e-05,
+          0.007633391302078962,
+          -0.010576431639492512,
+          -0.024485481902956963,
+          -0.031042715534567833,
+          0.004034627228975296,
+          -0.0005313287838362157,
+          0.015081637538969517,
+          0.021260865032672882,
+          -0.015374784357845783,
+          -0.00021925750479567796,
+          0.005943939555436373,
+          -0.011332442983984947,
+          -0.015012207441031933,
+          -0.017727673053741455,
+          0.013214753940701485,
+          -0.016740230843424797,
+          0.002659536898136139,
+          0.019548270851373672,
+          -0.025226064026355743,
+          0.012975607998669147,
+          -0.034251902252435684,
+          -0.007228385657072067,
+          -0.011039295233786106,
+          -0.014209911227226257,
+          0.005045212805271149,
+          0.016894519329071045,
+          -0.014842490665614605,
+          -0.0005539897829294205,
+          -0.004528348334133625,
+          0.0012420171406120062,
+          -0.022896315902471542,
+          -0.02136886678636074,
+          0.016123080626130104,
+          -0.01888483203947544,
+          -0.02135343849658966,
+          -0.009558131918311119,
+          -0.0014609131030738354,
+          -0.001360625959932804,
+          -0.014711346477270126,
+          0.019594557583332062,
+          0.0005718293250538409,
+          0.012466458603739738,
+          6.394508091034368e-05,
+          0.015845362097024918,
+          -0.00558136310428381,
+          -0.006865809205919504,
+          -0.006703807041049004,
+          0.0036643364001065493,
+          -0.011201297864317894,
+          0.0029873985331505537,
+          -0.0005356681067496538,
+          0.0015168423997238278,
+          0.0024724628310650587,
+          -0.004358631558716297,
+          0.008061540313065052,
+          0.0003941572504118085,
+          0.016879091039299965,
+          -0.0017627386841922998,
+          0.008470403030514717,
+          0.014541629701852798,
+          0.0030433277133852243,
+          -0.013585045002400875,
+          -0.0038282673340290785,
+          0.011795305646955967,
+          0.002084814477711916,
+          0.006896666716784239,
+          0.022742029279470444,
+          -0.0031436148565262556,
+          0.014163624495267868,
+          0.0012092309771105647,
+          -8.505418918502983e-06,
+          0.005118499509990215,
+          -0.008979552425444126,
+          0.004246773198246956,
+          -0.0013500186614692211,
+          -0.0006889916840009391,
+          -0.002725109225139022,
+          0.014749918133020401,
+          0.001008656807243824,
+          -0.013392184861004353,
+          -0.010221569798886776,
+          -0.0018890618812292814,
+          0.03203015774488449,
+          0.00025168206775560975,
+          0.010807863436639309,
+          -0.02882097102701664,
+          -0.00742895994335413,
+          0.009380700998008251,
+          -0.025148918852210045,
+          -0.020921431481838226,
+          0.005650792270898819,
+          0.012682461179792881,
+          -0.003945911768823862,
+          -0.0004481579817365855,
+          0.04890924692153931,
+          0.020165421068668365,
+          -0.0035987640731036663,
+          -0.004802209325134754,
+          -0.0063026584684848785,
+          -0.0026884658727794886,
+          0.011617874726653099,
+          -0.012204169295728207,
+          -0.006148370914161205,
+          -0.0069275242276489735,
+          0.004474347457289696,
+          0.007830108515918255,
+          -0.013662189245223999,
+          -0.006044226698577404,
+          0.002401104662567377,
+          -0.011147296987473965,
+          -0.013723904266953468,
+          0.006588091142475605,
+          -0.0063412305898964405,
+          3.5859869967680424e-05,
+          0.006453089416027069,
+          -0.012250455096364021,
+          0.01155615970492363,
+          0.004983497317880392,
+          0.019100835546851158,
+          -0.01653965748846531,
+          -0.001184159191325307,
+          0.00904126837849617,
+          -0.015953363850712776,
+          0.0031841155141592026,
+          -0.013145324774086475,
+          0.006167656742036343,
+          -0.0012063381727784872,
+          0.0058513665571808815,
+          -0.002252602484077215,
+          -0.00450520496815443,
+          0.007189814001321793,
+          -0.03320274502038956,
+          -0.009812707081437111,
+          0.013685332611203194,
+          -0.025874072685837746,
+          0.010483859106898308,
+          0.004748208448290825,
+          0.009511846117675304,
+          -0.007961252704262733,
+          0.011949594132602215,
+          0.00112340843770653,
+          -0.015374784357845783,
+          0.020890574902296066,
+          0.007675820495933294,
+          0.013762475922703743,
+          0.005708650220185518,
+          0.005928510800004005,
+          0.009990138001739979,
+          0.013515615835785866,
+          -0.012119310908019543,
+          -0.01636994071304798,
+          0.011455873027443886,
+          0.003450262127444148,
+          -5.23433591297362e-05,
+          -0.01230445597320795,
+          -0.0148656340315938,
+          0.01066900510340929,
+          0.01279046293348074,
+          0.012921607121825218,
+          0.01275189034640789,
+          0.006981525104492903,
+          0.015343926846981049,
+          -0.02062828466296196,
+          -0.012767319567501545,
+          -0.025904929265379906,
+          0.014965921640396118,
+          -0.0026576081290841103,
+          -0.001909312093630433,
+          -0.017496241256594658,
+          -0.005072213243693113,
+          -0.02237173728644848,
+          0.007344101555645466,
+          0.016123080626130104,
+          0.0009990137768909335,
+          -0.0024551053065806627,
+          0.02389918826520443,
+          -0.013739332556724548,
+          0.04357088729739189,
+          -0.007521532475948334,
+          -0.019193409010767937,
+          -0.010831006802618504,
+          -0.004593920893967152,
+          0.013893621042370796,
+          0.005338359624147415,
+          -0.00617922842502594,
+          0.003953625913709402,
+          0.020149992778897285,
+          -0.001371233258396387,
+          -0.002293103141710162,
+          -0.011965022422373295,
+          0.01441048551350832,
+          -0.006534090265631676,
+          0.008539832197129726,
+          -0.011085581965744495,
+          0.019471125677227974,
+          0.00515707116574049,
+          0.010699862614274025,
+          0.009588989429175854,
+          -0.014348769560456276,
+          -0.014603344723582268,
+          -0.00595936831086874,
+          0.00558136310428381,
+          -0.019316839054226875,
+          -0.009095268324017525,
+          0.03465305268764496,
+          -7.075544272083789e-05,
+          -0.006676806602627039,
+          -0.005114641971886158,
+          0.011471301317214966,
+          -0.009288128465414047,
+          -0.001129194162786007,
+          -0.005716364830732346,
+          -0.010877293534576893,
+          -0.001112801139242947,
+          -0.005785794463008642,
+          -0.006526376120746136,
+          -0.013261040672659874,
+          -0.0058976528234779835,
+          0.0014483771519735456,
+          -0.007668105885386467,
+          -0.0006889916840009391,
+          0.004532205406576395,
+          0.012512744404375553,
+          0.0110624385997653,
+          -0.00276753818616271,
+          0.006865809205919504,
+          -0.0012603388167917728,
+          0.018637971952557564,
+          0.0135464733466506,
+          -0.013739332556724548,
+          -0.019378554075956345,
+          -0.0028793970122933388,
+          0.00617537135258317,
+          0.014611058868467808,
+          0.012890749610960484,
+          -0.006032655015587807,
+          -0.0003430493816267699,
+          -0.024485481902956963,
+          -0.005176357459276915,
+          0.008547547273337841,
+          -0.01273646205663681,
+          -0.007579390425235033,
+          -0.0029951129108667374,
+          -0.0010154069168493152,
+          -0.004485919140279293,
+          -0.016925377771258354,
+          -0.000276512757409364,
+          0.010877293534576893,
+          -0.006483946926891804,
+          0.017110522836446762,
+          0.018622543662786484,
+          -0.014973635785281658,
+          -0.007251529023051262,
+          0.0009271735325455666,
+          -0.03236959129571915,
+          0.0024146048817783594,
+          -0.000498060486279428,
+          0.02067457139492035,
+          0.0053152162581682205,
+          0.0005901510012336075,
+          0.002809967380017042,
+          -0.006460803560912609,
+          0.002123386599123478,
+          0.007309386972337961,
+          -0.008840694092214108,
+          -0.01073072012513876,
+          -0.010468430817127228,
+          -0.011039295233786106,
+          -0.009735562838613987,
+          -0.0025631068274378777,
+          -0.022572312504053116,
+          0.029931843280792236,
+          0.011795305646955967,
+          0.003629621583968401,
+          -0.01069214753806591,
+          -0.0010134782642126083,
+          0.018637971952557564,
+          -0.0035235488321632147,
+          0.023729471489787102,
+          0.013006465509533882,
+          -0.011826164089143276,
+          -0.017125951126217842,
+          0.023467181250452995,
+          -0.020983148366212845,
+          0.0062525151297450066,
+          -0.003961340524256229,
+          0.0019044906366616488,
+          0.003965197596698999,
+          -0.011417300440371037,
+          0.008022967725992203,
+          -0.016940806061029434,
+          -0.00298354122787714,
+          -0.00433934573084116,
+          0.006217800546437502,
+          -0.002638322301208973,
+          0.026768941432237625,
+          -0.0019739202689379454,
+          -0.004042341839522123,
+          0.013099038042128086,
+          0.039127398282289505,
+          0.029283834621310234,
+          0.018761401996016502,
+          -0.006665234919637442,
+          -0.006194657180458307,
+          -0.012705604545772076,
+          0.004678778816014528,
+          -0.017095094546675682,
+          -0.003265116596594453,
+          0.004663350060582161,
+          0.0010684434091672301,
+          -0.009704705327749252,
+          -0.0003309956518933177,
+          -0.019162550568580627,
+          0.0239609032869339,
+          -0.01236617099493742,
+          0.00011173815437359735,
+          -0.010962151922285557,
+          -0.01404790859669447,
+          -0.018468255177140236,
+          0.0023876044433563948,
+          -0.0072823865339159966,
+          -0.01891569048166275,
+          0.0024782486725598574,
+          -0.019579127430915833,
+          -0.03690565377473831,
+          0.021986018866300583,
+          0.0029970414470881224,
+          0.011664161458611488,
+          0.025025488808751106,
+          -0.011371014639735222,
+          0.01578364707529545,
+          0.02713923342525959,
+          0.017002521082758904,
+          0.004566920455545187,
+          0.011756733991205692,
+          0.01023699901998043,
+          0.012597602792084217,
+          -0.03227701783180237,
+          -0.00698923971503973,
+          0.015027636662125587,
+          0.009558131918311119,
+          -0.008323829621076584,
+          -0.010861864313483238,
+          0.0030548993963748217,
+          -0.02798781543970108,
+          0.006476232316344976,
+          -0.022495169192552567,
+          -0.007444388698786497,
+          0.012659317813813686,
+          -0.009781849570572376,
+          0.0008239935268647969,
+          0.004875496029853821,
+          0.005415503401309252,
+          -0.011702733114361763,
+          -0.01229674182832241,
+          -0.007714392617344856,
+          0.00742510287091136,
+          0.01316846814006567,
+          -0.004875496029853821,
+          -0.0001758640428306535,
+          -0.014950492419302464,
+          0.003351903520524502,
+          0.011625589802861214,
+          0.015482786111533642,
+          0.015729647129774094,
+          -0.00287361117079854,
+          -0.0024705342948436737,
+          0.0181133933365345,
+          0.00044237219844944775,
+          -0.007201385218650103,
+          0.008532118052244186,
+          0.010846436023712158,
+          0.000777707202360034,
+          0.017449956387281418,
+          0.009581275284290314,
+          -0.01657051593065262,
+          0.004809923470020294,
+          0.004995069000869989,
+          0.03292502835392952,
+          0.010105853900313377,
+          0.00742895994335413,
+          -0.0031243290286511183,
+          -0.0016412369441241026,
+          -0.009596704505383968,
+          0.012659317813813686,
+          0.017959104850888252,
+          -0.0068388087674975395,
+          0.004686493426561356,
+          0.02556549571454525,
+          0.004177343565970659,
+          0.044866908341646194,
+          0.004852352663874626,
+          0.0035274059046059847,
+          -0.006048083771020174,
+          -0.009018125012516975,
+          -0.0015804861905053258,
+          -0.0018958119908347726,
+          -0.004077056422829628,
+          -0.006206228863447905,
+          -0.0016711303032934666,
+          0.019316839054226875,
+          -0.004088628105819225,
+          0.00595936831086874,
+          0.009897565469145775,
+          -0.00216967286542058,
+          -0.006931381765753031,
+          0.014503058046102524,
+          -0.01155615970492363,
+          0.010522431693971157,
+          -0.021569442003965378,
+          -0.002233316656202078,
+          0.00411562854424119,
+          0.014148196205496788,
+          -0.010977580212056637,
+          0.00021202526113484055,
+          -0.003330688923597336,
+          0.015112495049834251,
+          0.004532205406576395,
+          -0.0032940455712378025,
+          0.0011947664897888899,
+          -0.01903912052512169,
+          -0.008084683679044247,
+          0.03230787441134453,
+          0.004007626790553331,
+          0.015390212647616863,
+          -0.005508076399564743,
+          0.008200399577617645,
+          0.005866795312613249,
+          0.013824190944433212,
+          -0.02794152870774269,
+          0.011664161458611488,
+          0.020196279510855675,
+          0.0013345899060368538,
+          0.00780696514993906,
+          -0.001996099017560482,
+          -0.01566021703183651,
+          0.01638536900281906,
+          -0.026799799874424934,
+          0.0049024964682757854,
+          -0.014287054538726807,
+          0.012011309154331684,
+          0.004212058149278164,
+          -0.004146486055105925,
+          0.007992110215127468,
+          0.006966096349060535,
+          0.017743103206157684,
+          0.025750642642378807,
+          0.026568368077278137,
+          0.0002040697872871533,
+          0.0148656340315938,
+          -0.007031668908894062,
+          -3.0887702450854704e-05,
+          -0.026059217751026154,
+          -0.01715680956840515,
+          0.003988340962678194,
+          -0.0363810732960701,
+          -7.202108827186748e-05,
+          -0.008030682802200317,
+          -0.018576256930828094,
+          0.041688576340675354,
+          -0.006117513403296471,
+          -0.013569616712629795,
+          0.007197528146207333,
+          -0.005724078975617886,
+          0.004836923908442259,
+          -0.009550417773425579,
+          0.030194131657481194,
+          -0.002079028869047761,
+          -0.017604243010282516,
+          -0.03314103186130524,
+          -0.00574336526915431,
+          -0.025766070932149887,
+          -0.00450520496815443,
+          -0.01633908413350582,
+          0.014109623618423939,
+          -0.020828859880566597,
+          0.002302746055647731,
+          0.0037974095903337,
+          0.0020539569668471813,
+          0.01965627260506153,
+          0.0014638060238212347,
+          -0.007386530749499798,
+          0.006267943885177374,
+          -0.005106927827000618,
+          -0.012651603668928146,
+          -0.025087203830480576,
+          0.0011369085405021906,
+          0.00449363375082612,
+          0.00041971117025241256,
+          0.013808762654662132,
+          -0.0037279801908880472,
+          0.022495169192552567,
+          -0.016323653981089592,
+          0.012775033712387085,
+          -0.01517421007156372,
+          0.0007073133601807058,
+          0.004875496029853821,
+          0.0006644020322710276,
+          -7.636043301317841e-05,
+          -0.00536150299012661,
+          -0.02558092586696148,
+          0.007725963834673166,
+          0.021708300337195396,
+          -0.0025708212051540613,
+          0.0063450876623392105,
+          -0.006213943008333445,
+          0.000337986828526482,
+          0.0035061913076788187,
+          0.008138683624565601,
+          -0.015382498502731323,
+          0.0012535888236016035,
+          0.03369646519422531,
+          0.013855048455297947,
+          -0.007984396070241928,
+          -0.01230445597320795,
+          0.0021388153545558453,
+          0.013515615835785866,
+          -0.016076793894171715,
+          0.004586206283420324,
+          0.01279817707836628,
+          0.024315765127539635,
+          0.0020944576244801283,
+          -0.00553121929988265,
+          -0.028219247236847878,
+          0.021230008453130722,
+          -0.011810734868049622,
+          0.015205067582428455,
+          0.0018620614428073168,
+          0.028358107432723045,
+          0.002680751495063305,
+          0.0014271625550463796,
+          0.0010935150785371661,
+          0.03351132199168205,
+          0.00942698772996664,
+          -0.023590611293911934,
+          -0.0024281051009893417,
+          -0.0009517631842754781,
+          0.003047185018658638,
+          0.029268404468894005,
+          0.03153643757104874,
+          -0.004755923058837652,
+          -0.039281684905290604,
+          0.008802121505141258,
+          -0.024115189909934998,
+          0.002595893107354641,
+          0.011000723578035831,
+          0.00617922842502594,
+          -0.01360047422349453,
+          -0.0012748033041134477,
+          0.0012034452520310879,
+          0.0038514104671776295,
+          0.014950492419302464,
+          -0.011448157951235771,
+          -0.002935326425358653,
+          0.009735562838613987,
+          0.0032130444888025522,
+          -0.012543601915240288,
+          0.017557958140969276,
+          0.0020867432467639446,
+          -0.003022113349288702,
+          -0.004208201076835394,
+          0.03357303515076637,
+          0.008532118052244186,
+          -0.016308225691318512,
+          -0.0054039317183196545,
+          -0.026028359308838844,
+          -0.009234127588570118,
+          -0.013531044125556946,
+          -0.006904381327331066,
+          -0.022772885859012604,
+          -0.007679677568376064,
+          -0.019347695633769035,
+          -0.007513818331062794,
+          -0.02895982936024666,
+          0.006117513403296471,
+          -0.0047752088867127895,
+          0.03375818207859993,
+          -0.0035505490377545357,
+          -0.031582724303007126,
+          0.01741909794509411,
+          -0.0035717636346817017,
+          0.019594557583332062,
+          -0.010777005925774574,
+          0.009156984277069569,
+          0.01069214753806591,
+          0.01312989555299282,
+          0.02152315527200699,
+          0.00026566439191810787,
+          0.006503232754766941,
+          0.0030857569072395563,
+          0.025010060518980026,
+          -0.0033576893620193005,
+          -0.007818536832928658,
+          0.002827324904501438,
+          -0.011571588926017284,
+          0.019255124032497406,
+          0.0030317562632262707,
+          0.014503058046102524,
+          -0.022526025772094727,
+          0.022418024018406868,
+          -0.004570777527987957,
+          0.013801048509776592,
+          0.0012468387139961123,
+          0.005330645013600588,
+          -0.012582174502313137,
+          -0.007351816166192293,
+          -0.004269916098564863,
+          -0.0035949070006608963,
+          0.015220495872199535,
+          -0.01361590251326561,
+          0.005130071192979813,
+          -0.03523934632539749,
+          0.027015801519155502,
+          0.008601547218859196,
+          0.003278616815805435,
+          -0.0017087379237636924,
+          0.010121283121407032,
+          0.005924653261899948,
+          0.01235074270516634,
+          0.005727936513721943,
+          -0.013099038042128086,
+          0.017465384677052498,
+          0.001782024628482759,
+          0.008370115421712399,
+          0.0036874795332551003,
+          -0.016771089285612106,
+          -0.010645861737430096,
+          0.01324561145156622,
+          0.01819053664803505,
+          0.003342260606586933,
+          0.0013596616918221116,
+          -0.03145929425954819,
+          0.030086129903793335,
+          0.0017289882525801659,
+          0.008562975563108921,
+          -0.007891823537647724,
+          0.02894440107047558,
+          -0.007444388698786497,
+          0.010622718371450901,
+          0.022603169083595276,
+          0.01567564532160759,
+          0.007128098513931036,
+          -0.008416402153670788,
+          0.007668105885386467,
+          0.027632953599095345,
+          0.007976681925356388,
+          0.005573648493736982,
+          0.018298538401722908,
+          0.009156984277069569,
+          0.012049880810081959,
+          -0.01576821878552437,
+          -0.04017655551433563,
+          0.001536128344014287,
+          -0.015444213524460793,
+          -0.018668830394744873,
+          0.011949594132602215,
+          0.011949594132602215,
+          0.004917925223708153,
+          0.02477862872183323,
+          0.015652501955628395,
+          -0.006858095061033964,
+          0.0018466326873749495,
+          -0.013006465509533882,
+          -0.019903132691979408,
+          0.005770365707576275,
+          -0.003265116596594453,
+          -0.023220321163535118,
+          -0.001962348585948348,
+          -0.011031581088900566,
+          1.3439917893265374e-05,
+          -0.006024940405040979,
+          -0.0069699534215033054,
+          0.04431147128343582,
+          0.008200399577617645,
+          -0.010321857407689095,
+          -0.014387342147529125,
+          0.013577330857515335,
+          -0.011841592378914356,
+          0.029175832867622375,
+          -0.0034155473113059998,
+          -0.001836989657022059,
+          -0.025982074439525604,
+          -0.0010607289150357246,
+          -0.009465559385716915,
+          -0.005496504716575146,
+          0.005230357870459557,
+          -0.01198045164346695,
+          -0.009789563715457916,
+          0.0028813255485147238,
+          -0.008709548972547054,
+          0.013307327404618263,
+          -0.009087554179131985,
+          0.006599662825465202,
+          -0.01272874791175127,
+          -0.0007323851459659636,
+          0.018684258684515953,
+          -4.550286030280404e-05,
+          0.016308225691318512,
+          -0.015104779973626137,
+          -0.0005366324330680072,
+          -0.00022648973390460014,
+          -0.03323360159993172,
+          0.016693945974111557,
+          0.012273598462343216,
+          0.0001559753727633506,
+          -0.02312774769961834,
+          0.002248745411634445,
+          0.024207763373851776,
+          0.0014117337996140122,
+          0.004354774486273527,
+          0.025040917098522186,
+          -0.005091499071568251,
+          0.0074598174542188644,
+          0.013446185737848282,
+          0.007922681048512459,
+          -0.020026562735438347,
+          -0.0020925288554280996,
+          0.009226413443684578,
+          0.007668105885386467,
+          0.007710535079240799,
+          -0.0008996910182759166,
+          0.0165859442204237,
+          0.0067848083563148975,
+          -0.019285980612039566,
+          -0.017696816474199295,
+          0.015120209194719791,
+          0.010098139755427837,
+          -0.0009392272913828492,
+          0.004393346607685089,
+          0.005731793586164713,
+          0.011394158005714417,
+          -0.011695018969476223,
+          -0.0013037322787567973,
+          -0.02298888936638832,
+          0.021662013605237007,
+          0.016663087531924248,
+          -0.016045935451984406,
+          0.034961625933647156,
+          -0.004250630270689726,
+          -0.007363387383520603,
+          -0.019239693880081177,
+          -0.02541120909154415,
+          -0.0021966733038425446,
+          0.015359355136752129,
+          -0.004458918701857328,
+          0.002956541022285819,
+          0.007961252704262733,
+          -0.032523877918720245,
+          -0.020365996286273003,
+          -0.011671875603497028,
+          0.0005154178361408412,
+          0.0052612158469855785,
+          0.0027347521390765905,
+          0.009704705327749252,
+          0.005018212366849184,
+          0.01116272620856762,
+          0.005862938240170479,
+          0.007706678006798029,
+          -0.017959104850888252,
+          -0.02135343849658966,
+          0.008161826990544796,
+          0.00027048587799072266,
+          -0.016107652336359024,
+          -0.021631157025694847,
+          0.007023954298347235,
+          0.004077056422829628,
+          -0.001974884420633316,
+          0.013492472469806671,
+          -0.020458567887544632,
+          0.017665958032011986,
+          0.009596704505383968,
+          -0.003650836180895567,
+          0.022556884214282036,
+          0.013137610629200935,
+          0.01318389642983675,
+          0.00037053192500025034,
+          -0.016925377771258354,
+          -0.0017810603603720665,
+          0.02649122290313244,
+          0.0009932280518114567,
+          0.0014628416392952204,
+          0.012528173625469208,
+          0.004543777089565992,
+          0.020288852974772453,
+          0.0034849767107516527,
+          -0.018715115264058113,
+          0.029484407976269722,
+          0.00019563216483220458,
+          0.012119310908019543,
+          0.018730545416474342,
+          0.021862588822841644,
+          0.0062525151297450066,
+          0.004632492549717426,
+          0.007097241003066301,
+          0.01903912052512169,
+          0.006545661948621273,
+          -0.013075895607471466,
+          -0.004185057710856199,
+          0.0069699534215033054,
+          0.0020057419314980507,
+          -0.004755923058837652,
+          -0.012196454219520092,
+          0.018298538401722908,
+          -0.007043240126222372,
+          0.0018292752793058753,
+          -0.022664885967969894,
+          0.0069930967874825,
+          -0.008848408237099648,
+          -0.004038484301418066,
+          -0.00823897123336792,
+          0.004786780569702387,
+          0.015876220539212227,
+          -0.004678778816014528,
+          0.03400504216551781,
+          -0.010098139755427837,
+          0.014595630578696728,
+          -0.009342129342257977,
+          -0.02227916568517685,
+          0.018483683466911316,
+          -0.01116272620856762,
+          -0.0014011265011504292,
+          0.00012547941878437996,
+          0.0022391024976968765,
+          -0.009596704505383968,
+          -0.014842490665614605,
+          0.004667207598686218,
+          -0.010869579389691353,
+          0.0303792767226696,
+          -0.0004961318918503821,
+          0.02315860614180565,
+          0.003916982561349869,
+          0.0021735301706939936,
+          0.01731109619140625,
+          -0.005558219738304615,
+          -0.003257402218878269,
+          0.011664161458611488,
+          -0.0072129569016397,
+          0.020011134445667267,
+          -0.00020913235493935645,
+          -0.001489842077717185,
+          0.016940806061029434,
+          -0.011772163212299347,
+          -0.012481886893510818,
+          0.018545400351285934,
+          0.012659317813813686,
+          0.0014657345600426197,
+          0.006144513376057148,
+          0.002813824685290456,
+          -0.0002004536654567346,
+          -0.003560192184522748,
+          -0.020921431481838226,
+          -0.005704793147742748,
+          0.002511034719645977,
+          0.002727037761360407,
+          -0.00455920584499836,
+          0.004401060752570629,
+          0.0012593745486810803,
+          -0.0004539437941275537,
+          -0.002684608567506075,
+          0.0025418922305107117,
+          -0.0031841155141592026,
+          0.004690350499004126,
+          0.005878366995602846,
+          -0.014395056292414665,
+          -0.02391461655497551,
+          0.0065533765591681,
+          0.013631331734359264,
+          -0.010020995512604713,
+          0.0019806702621281147,
+          0.03150557726621628,
+          -0.03786223754286766,
+          0.01738824136555195,
+          0.005434789694845676,
+          -0.007336387410759926,
+          0.005704793147742748,
+          -0.014287054538726807
+        ],
+        "title": "4 Downstream Results"
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "9cc3633d-79f3-4045-b6a2-d3df9e569d10",
+      "properties": {
+        "page_content": "5 Designing Models with Data, Model, and Expert-Parallelism 18 5.1 Data Parallelism 20 5.2 Model Parallelism 20 5.3 Model and Data Parallelism 21 5.4 Expert and Data Parallelism 22 5.5 Expert, Model and Data Parallelism 22 5.6 Towards Trillion Parameter Models 22 ",
+        "keyphrases": [
+          "Designing Models",
+          "Data Parallelism",
+          "Model Parallelism",
+          "Expert Parallelism",
+          "Trillion Parameter Models"
+        ],
+        "embedding": [
+          -0.004642441403120756,
+          0.005233492702245712,
+          -0.022710707038640976,
+          0.004320049658417702,
+          0.027123892679810524,
+          -0.0023946543224155903,
+          -0.029717355966567993,
+          0.04969131574034691,
+          -0.019472461193799973,
+          0.01448613591492176,
+          0.021908311173319817,
+          -0.0279119610786438,
+          0.043014224618673325,
+          -0.02931615710258484,
+          -0.037196844816207886,
+          0.025519099086523056,
+          0.024114903062582016,
+          0.01901394873857498,
+          -0.010932662524282932,
+          -0.05880425497889519,
+          0.020575758069753647,
+          -0.04052106291055679,
+          -0.04877429082989693,
+          0.019085591658949852,
+          0.03307022899389267,
+          0.024630729109048843,
+          0.0260635819286108,
+          0.01029504369944334,
+          -0.05450569838285446,
+          0.01792498119175434,
+          0.012745220214128494,
+          -0.014686735346913338,
+          0.02420087344944477,
+          -0.021263526752591133,
+          -0.03960403427481651,
+          -0.017409155145287514,
+          0.05966396629810333,
+          -0.004216167610138655,
+          -0.01887066476047039,
+          0.022108910605311394,
+          0.006763062439858913,
+          0.0028657044749706984,
+          -0.030462438240647316,
+          0.002539730630815029,
+          0.006548134610056877,
+          -0.02255309373140335,
+          -0.047570694237947464,
+          0.003571384120732546,
+          0.01928619109094143,
+          -0.024874314665794373,
+          0.008267557248473167,
+          -0.046309784054756165,
+          0.025963282212615013,
+          0.00751530984416604,
+          -0.022108910605311394,
+          4.404901119414717e-05,
+          0.028943615034222603,
+          0.037196844816207886,
+          0.007013811729848385,
+          0.017695724964141846,
+          0.00950697436928749,
+          -0.033901285380125046,
+          -0.05513615533709526,
+          0.018369166180491447,
+          -0.012265214696526527,
+          -0.035477422177791595,
+          -0.017967967316508293,
+          0.017595425248146057,
+          -0.026450451463460922,
+          -0.011283711530268192,
+          -0.0028764507733285427,
+          0.014801363460719585,
+          0.003943925723433495,
+          -0.006902765482664108,
+          0.009463989175856113,
+          -0.014521957375109196,
+          0.029975268989801407,
+          0.040062546730041504,
+          0.027654048055410385,
+          -0.001376433647237718,
+          -0.02430117316544056,
+          0.004194675013422966,
+          -0.010445492342114449,
+          0.02546178363263607,
+          0.02044680155813694,
+          -0.00542692793533206,
+          -0.005899769254028797,
+          -0.06487955152988434,
+          -0.017265869304537773,
+          0.010130265727639198,
+          -0.04218316823244095,
+          0.005924844183027744,
+          0.025934625416994095,
+          -0.009299211204051971,
+          -0.0171512421220541,
+          0.027553748339414597,
+          0.028069574385881424,
+          0.007192918099462986,
+          -0.01659242808818817,
+          -0.017409155145287514,
+          0.0003219439822714776,
+          0.047398753464221954,
+          0.0451061874628067,
+          -0.01139833964407444,
+          0.04106554388999939,
+          -0.009227568283677101,
+          0.005846037063747644,
+          0.03155140578746796,
+          -0.013518961146473885,
+          0.03499025106430054,
+          -0.025791339576244354,
+          -0.009772052057087421,
+          -0.016950642690062523,
+          0.008518306538462639,
+          -0.038772981613874435,
+          -0.007185753900557756,
+          -0.017867667600512505,
+          -0.04335810989141464,
+          0.020719043910503387,
+          -0.04120882973074913,
+          0.013017462566494942,
+          0.017982294782996178,
+          -0.02114889957010746,
+          0.03243977576494217,
+          -0.04295691102743149,
+          0.029574070125818253,
+          -0.013239555060863495,
+          -0.021335169672966003,
+          -0.015431818552315235,
+          -0.02100561372935772,
+          0.02230951003730297,
+          0.018039610236883163,
+          -0.03513353690505028,
+          -0.027768677100539207,
+          0.008819205686450005,
+          -0.032353803515434265,
+          0.025691039860248566,
+          0.0003951537946704775,
+          -0.006573209539055824,
+          0.002344504464417696,
+          -0.0020274859853088856,
+          -0.018612749874591827,
+          0.0076227737590670586,
+          -0.053502701222896576,
+          -0.020246202126145363,
+          -0.026536423712968826,
+          0.01940082013607025,
+          0.042068541049957275,
+          -0.0010889676632359624,
+          0.026822993531823158,
+          -0.013318361714482307,
+          0.0033672028221189976,
+          0.010115936398506165,
+          0.018999621272087097,
+          -0.00021448006737045944,
+          -0.006286639254540205,
+          -0.007938001304864883,
+          0.022982949391007423,
+          -0.004549305886030197,
+          0.001577928545884788,
+          0.009292046539485455,
+          0.009901008568704128,
+          0.00019175592751707882,
+          0.05324478819966316,
+          0.0067773908376693726,
+          0.000919712008908391,
+          0.061039503663778305,
+          0.0037110873963683844,
+          -0.04668232426047325,
+          0.024043260142207146,
+          0.0024286846164613962,
+          -0.00724306795746088,
+          0.03946075215935707,
+          0.012451485730707645,
+          -0.0010235938243567944,
+          0.017208555713295937,
+          0.013339853845536709,
+          0.01571838930249214,
+          -0.004660351667553186,
+          0.03467502444982529,
+          -0.0015474804677069187,
+          -0.017796024680137634,
+          -0.0029695862904191017,
+          0.0010003099450841546,
+          0.04794323444366455,
+          0.0009313539485447109,
+          0.029545413330197334,
+          0.009693245403468609,
+          0.01960141770541668,
+          -0.017380498349666595,
+          0.02044680155813694,
+          0.02574835531413555,
+          0.015016291290521622,
+          0.012465814128518105,
+          -0.029330484569072723,
+          -0.03258306160569191,
+          0.01591898873448372,
+          -0.010946990922093391,
+          -0.01806826703250408,
+          0.010861019603908062,
+          0.006372610107064247,
+          -0.0444757342338562,
+          -0.0026239105500280857,
+          -0.01911424845457077,
+          -0.000753590720705688,
+          0.0021922639571130276,
+          -0.006401267368346453,
+          0.011090275831520557,
+          0.03828581050038338,
+          -0.0343024842441082,
+          -0.006039472296833992,
+          0.016004959121346474,
+          -0.024917300790548325,
+          -0.006129025481641293,
+          -0.003829297609627247,
+          0.0020471876487135887,
+          -0.03355740010738373,
+          0.0253901407122612,
+          0.013332690112292767,
+          -0.04043509066104889,
+          -0.008969655260443687,
+          -0.008145765401422977,
+          0.006798883900046349,
+          -0.02054710127413273,
+          0.003039437811821699,
+          -0.02093397080898285,
+          0.006709330715239048,
+          -0.015517789870500565,
+          -0.02230951003730297,
+          -0.02633582428097725,
+          0.017280198633670807,
+          0.003111080499365926,
+          -0.00244122208096087,
+          -0.010932662524282932,
+          0.030978266149759293,
+          -0.013848517090082169,
+          -0.0075368029065430164,
+          -0.020805014297366142,
+          -0.03857238218188286,
+          0.03261171653866768,
+          0.01520256232470274,
+          -0.022395480424165726,
+          -0.04120882973074913,
+          -0.01995963230729103,
+          0.008718905970454216,
+          0.009026968851685524,
+          -0.015804359689354897,
+          0.052557021379470825,
+          -0.014758378267288208,
+          -0.001351358718238771,
+          0.0011588193010538816,
+          -0.0005270209512673318,
+          0.015575103461742401,
+          0.024731028825044632,
+          -0.010846691206097603,
+          0.006917094346135855,
+          0.013934488408267498,
+          0.0006443357560783625,
+          -0.005061550531536341,
+          -0.0090484619140625,
+          0.05284358933568001,
+          -0.030491095036268234,
+          -0.047742635011672974,
+          0.04545007273554802,
+          -0.032840974628925323,
+          0.009772052057087421,
+          0.016778700053691864,
+          0.001771363546140492,
+          0.03524816408753395,
+          0.007343367673456669,
+          -0.00307705020532012,
+          -0.015317190438508987,
+          0.0018662899965420365,
+          -0.012673578225076199,
+          0.02412923239171505,
+          -0.03587862104177475,
+          0.0015277786878868937,
+          0.007644266821444035,
+          0.02437281608581543,
+          0.0067021665163338184,
+          -0.021822338923811913,
+          0.02655075117945671,
+          -0.008102779276669025,
+          0.002068680478259921,
+          -0.005036475602537394,
+          -0.0005041848635300994,
+          -0.01732318289577961,
+          -0.005151103716343641,
+          -0.015876002609729767,
+          -0.02357041835784912,
+          -0.024487445130944252,
+          -0.01950111798942089,
+          -0.01676437072455883,
+          0.01785333827137947,
+          0.00706754345446825,
+          0.005398270674049854,
+          0.007572623901069164,
+          0.012995969504117966,
+          0.007196500431746244,
+          0.005982158239930868,
+          -0.03272634372115135,
+          0.013712395913898945,
+          0.007823373191058636,
+          -0.004277063999325037,
+          -0.023341162130236626,
+          -0.0018430062336847186,
+          0.008704577572643757,
+          0.012687906622886658,
+          0.0016218096716329455,
+          -0.009549960494041443,
+          0.012129094451665878,
+          0.0023803256917744875,
+          -0.00688127288594842,
+          0.0008588157943449914,
+          -0.06619777530431747,
+          0.05814514309167862,
+          -0.0012635965831577778,
+          0.009528467431664467,
+          -0.057887230068445206,
+          -0.018913649022579193,
+          0.05909082666039467,
+          0.008109943941235542,
+          -0.048487719148397446,
+          -0.011899838224053383,
+          0.016563771292567253,
+          -0.02391430363059044,
+          -0.00041127335862256587,
+          0.029129885137081146,
+          -0.03143677860498428,
+          -0.04163868725299835,
+          -0.050837598741054535,
+          -0.040033891797065735,
+          -0.035477422177791595,
+          0.012172079645097256,
+          -0.011584610678255558,
+          -0.005043639801442623,
+          -0.026751350611448288,
+          0.006845451425760984,
+          -0.00787352304905653,
+          -0.03037646785378456,
+          0.0027152549009770155,
+          -0.02784031815826893,
+          0.0009698618669062853,
+          0.010158922523260117,
+          -0.005982158239930868,
+          -0.004366617184132338,
+          -0.03209589049220085,
+          -0.012917162850499153,
+          0.006064547225832939,
+          0.0251179002225399,
+          0.0007222470594570041,
+          0.02100561372935772,
+          0.021349499002099037,
+          0.010352357290685177,
+          -0.007099783048033714,
+          -0.0015698687639087439,
+          0.0029731683898717165,
+          0.004764233715832233,
+          0.003082423470914364,
+          -0.009951158426702023,
+          0.035162195563316345,
+          -0.05676960572600365,
+          -0.029860639944672585,
+          0.006770226638764143,
+          -0.01111893355846405,
+          -0.024071916937828064,
+          0.020174559205770493,
+          -0.0023713705595582724,
+          -0.04275631159543991,
+          0.01176371704787016,
+          -0.025590740144252777,
+          0.033299487084150314,
+          0.02742479182779789,
+          0.02524685673415661,
+          -0.010130265727639198,
+          -0.024745358154177666,
+          -0.004506320226937532,
+          -0.02216622419655323,
+          -0.015976302325725555,
+          0.020088588818907738,
+          -0.010173250921070576,
+          -0.005824544467031956,
+          0.011656252667307854,
+          -0.002566596493124962,
+          -0.0036824301350861788,
+          0.014385836198925972,
+          0.02441580221056938,
+          0.009986979886889458,
+          -0.04665366932749748,
+          0.00562036270275712,
+          -0.01613391563296318,
+          -0.029516756534576416,
+          -0.0067845555022358894,
+          0.011727895587682724,
+          0.0015833016950637102,
+          -0.010338028892874718,
+          -0.011835359036922455,
+          -0.025819996371865273,
+          0.014729720540344715,
+          -0.0009125477517955005,
+          -0.017265869304537773,
+          0.024143559858202934,
+          0.00978638045489788,
+          0.02381400391459465,
+          0.05378927290439606,
+          -0.004954086616635323,
+          0.009170254692435265,
+          0.021335169672966003,
+          0.021765025332570076,
+          0.0260635819286108,
+          -0.009893844835460186,
+          -0.04014851897954941,
+          -0.01577570289373398,
+          0.003718251595273614,
+          0.03711087256669998,
+          -0.018598422408103943,
+          -0.0006416491232812405,
+          0.027310162782669067,
+          -0.04023449122905731,
+          -0.001844797283411026,
+          0.007672923617064953,
+          0.011849688366055489,
+          0.05548003688454628,
+          -0.020790686830878258,
+          -0.013526124879717827,
+          0.016105258837342262,
+          -0.01596197299659252,
+          -0.018956635147333145,
+          -0.007844866253435612,
+          -0.020174559205770493,
+          0.012172079645097256,
+          -0.007350531872361898,
+          0.034245166927576065,
+          -0.00932786799967289,
+          0.0020794267766177654,
+          -0.04765666648745537,
+          0.022223537787795067,
+          0.05049371346831322,
+          -0.026149554178118706,
+          -0.020532771944999695,
+          -0.029330484569072723,
+          -0.013361346907913685,
+          0.011362518183887005,
+          -0.004449006170034409,
+          -0.058030515909194946,
+          -0.015245547518134117,
+          -0.02136382646858692,
+          -0.030032582581043243,
+          -0.04481961950659752,
+          -0.03413053974509239,
+          -0.0667136013507843,
+          0.021062927320599556,
+          -0.07714476436376572,
+          -0.032955601811409,
+          0.031236179172992706,
+          -0.016033615916967392,
+          -0.029574070125818253,
+          0.013318361714482307,
+          -0.008776219561696053,
+          -0.056339748203754425,
+          -0.03885895386338234,
+          0.05249970406293869,
+          0.016921984031796455,
+          -0.0033313813619315624,
+          -0.029803326353430748,
+          0.0007710536010563374,
+          -0.03166603296995163,
+          -0.028212860226631165,
+          0.004731994587928057,
+          0.012609099969267845,
+          0.023341162130236626,
+          -0.013261047191917896,
+          0.009972651489078999,
+          -0.002568387659266591,
+          0.013970308937132359,
+          0.02332683466374874,
+          0.02507491409778595,
+          -0.014357179403305054,
+          -0.004914683289825916,
+          -0.020260531455278397,
+          0.04398856312036514,
+          0.0203035157173872,
+          0.014149416238069534,
+          -0.006816794630140066,
+          -0.01901394873857498,
+          0.025275513529777527,
+          0.010775049217045307,
+          -0.005233492702245712,
+          1.3845774446963333e-05,
+          -0.006748734042048454,
+          0.03771267086267471,
+          -0.015603761188685894,
+          0.0014167326735332608,
+          0.002708090702071786,
+          -0.005896186921745539,
+          -0.02536148391664028,
+          -0.053502701222896576,
+          -0.016606757417321205,
+          0.0203035157173872,
+          -0.0028997347690165043,
+          -0.010108772665262222,
+          0.0023373402655124664,
+          -0.0171512421220541,
+          -0.02497461438179016,
+          0.025017600506544113,
+          -0.003610787680372596,
+          -0.01527420524507761,
+          -0.04633844271302223,
+          -0.0219226386398077,
+          -0.015446146950125694,
+          -0.015259875915944576,
+          0.025203870609402657,
+          0.028356146067380905,
+          0.009542795829474926,
+          0.01283835619688034,
+          0.017065269872546196,
+          -0.03272634372115135,
+          0.0004092584131285548,
+          -0.02563372626900673,
+          -0.0067845555022358894,
+          -0.005785140674561262,
+          -0.009822201915085316,
+          -0.01732318289577961,
+          0.02276802249252796,
+          -0.015302862040698528,
+          0.035964589565992355,
+          0.0021367408335208893,
+          -0.009263389743864536,
+          0.04851637780666351,
+          -0.016406157985329628,
+          0.002040023449808359,
+          0.008539799600839615,
+          -0.003204215783625841,
+          -2.553667400206905e-05,
+          -0.0020991284400224686,
+          -0.035018909722566605,
+          0.018383493646979332,
+          0.04843040555715561,
+          -0.0009501601452939212,
+          -0.030290495604276657,
+          -0.022524436935782433,
+          0.02563372626900673,
+          0.004309303127229214,
+          -0.006000068504363298,
+          0.01952977664768696,
+          -0.013963145203888416,
+          0.028069574385881424,
+          -0.048802945762872696,
+          0.0008507560123689473,
+          0.02461640164256096,
+          0.013347018510103226,
+          -0.04169600084424019,
+          0.048344433307647705,
+          -0.00925622507929802,
+          -0.038916267454624176,
+          -0.02240980975329876,
+          0.03762669861316681,
+          0.014371507801115513,
+          0.02100561372935772,
+          -0.018111253157258034,
+          0.012186408042907715,
+          0.009206075221300125,
+          -0.03708221763372421,
+          0.017523782327771187,
+          -0.029660040512681007,
+          0.004001239780336618,
+          0.004058553837239742,
+          0.02868570201098919,
+          -0.01623421534895897,
+          0.01816856674849987,
+          0.02556208334863186,
+          -0.006229325197637081,
+          -0.005118864588439465,
+          0.006662762723863125,
+          0.021091584116220474,
+          0.03742609918117523,
+          0.034846965223550797,
+          -0.02391430363059044,
+          0.012881341390311718,
+          0.013024627231061459,
+          0.03132215142250061,
+          -0.0009779216488823295,
+          0.0019397237338125706,
+          0.02020321600139141,
+          0.025590740144252777,
+          -0.003840043907985091,
+          -0.007307546213269234,
+          -0.008812041021883488,
+          0.0013379257870838046,
+          -0.00915592536330223,
+          0.0036860124673694372,
+          0.03289828822016716,
+          -0.010001308284699917,
+          -0.03928880766034126,
+          -0.0025271931663155556,
+          0.001104191760532558,
+          0.010273550637066364,
+          0.0023319669999182224,
+          0.05705617740750313,
+          -0.016606757417321205,
+          -0.006437088828533888,
+          0.013755381107330322,
+          -0.015732716768980026,
+          -0.024788344278931618,
+          -0.013769710436463356,
+          0.005043639801442623,
+          -0.023126235231757164,
+          -0.032525744289159775,
+          0.00379705848172307,
+          0.04387393593788147,
+          -0.016921984031796455,
+          0.013253883458673954,
+          0.03341411426663399,
+          -0.014385836198925972,
+          0.006494402885437012,
+          -0.0009349361062049866,
+          0.003689594566822052,
+          -0.00751530984416604,
+          -0.04871697723865509,
+          0.0206187441945076,
+          0.078864187002182,
+          0.005863947793841362,
+          0.02963138371706009,
+          -0.011964316479861736,
+          0.012702235020697117,
+          -0.013841352425515652,
+          0.024774014949798584,
+          9.951606625691056e-05,
+          0.014858677983283997,
+          -0.008045465685427189,
+          -0.019902316853404045,
+          -0.003714669495820999,
+          -0.004825130105018616,
+          -0.01394165214151144,
+          0.009356524795293808,
+          -0.014873006381094456,
+          -0.023226534947752953,
+          0.006652016658335924,
+          0.0009465779876336455,
+          -0.03441711142659187,
+          -0.0020203215535730124,
+          -0.02153576910495758,
+          0.023197878152132034,
+          0.026221195235848427,
+          -0.0019253952195867896,
+          0.0018125580390915275,
+          0.005254985764622688,
+          -0.010409671813249588,
+          0.01952977664768696,
+          -0.003539144992828369,
+          0.03759804368019104,
+          0.01328970491886139,
+          0.006046636495739222,
+          0.041466742753982544,
+          0.01816856674849987,
+          -0.03473233804106712,
+          -0.014930319972336292,
+          0.012422828935086727,
+          -0.005072297062724829,
+          0.0164778009057045,
+          -0.001104191760532558,
+          0.007866358384490013,
+          -0.0036519821733236313,
+          0.02384266071021557,
+          0.012931491248309612,
+          -0.028943615034222603,
+          -0.004463334567844868,
+          0.006261564325541258,
+          -0.018741706386208534,
+          0.01409926638007164,
+          -0.000238435561186634,
+          -0.04983460158109665,
+          0.0016110632568597794,
+          0.034560397267341614,
+          0.004470498766750097,
+          -0.014844349585473537,
+          0.006394103169441223,
+          0.04467633366584778,
+          0.012387007474899292,
+          0.010008472949266434,
+          0.0066126128658652306,
+          0.014686735346913338,
+          -0.010982812382280827,
+          -0.015317190438508987,
+          -0.028399132192134857,
+          -0.007422174792736769,
+          -0.003703922964632511,
+          -0.02430117316544056,
+          -0.010015636682510376,
+          -0.03292694315314293,
+          -0.0036681017372757196,
+          0.014973306097090244,
+          -0.006917094346135855,
+          -0.005183342844247818,
+          -0.015560775063931942,
+          -0.010230564512312412,
+          -0.004183928482234478,
+          0.039403434842824936,
+          0.025060584768652916,
+          0.01366224605590105,
+          -0.004932594019919634,
+          -0.00029597352840937674,
+          -0.035792648792266846,
+          -0.0056955874897539616,
+          0.021550098434090614,
+          0.026149554178118706,
+          0.006852615624666214,
+          -0.00939951092004776,
+          -0.05292956158518791,
+          -0.015618089586496353,
+          0.0007746357587166131,
+          -0.00939951092004776,
+          0.02296862192451954,
+          0.015861675143241882,
+          0.012730891816318035,
+          0.03384396806359291,
+          -0.0060323080979287624,
+          -0.01746646873652935,
+          -0.010818034410476685,
+          0.03665236011147499,
+          -0.004388109780848026,
+          0.004237660672515631,
+          0.016262872144579887,
+          -0.005849619396030903,
+          -0.035792648792266846,
+          0.023541761562228203,
+          0.0064478348940610886,
+          0.0024143559858202934,
+          0.016090931370854378,
+          -0.0026418212801218033,
+          -0.001410463941283524,
+          -0.019200220704078674,
+          0.012702235020697117,
+          0.023197878152132034,
+          0.035334136337041855,
+          -0.0018859917763620615,
+          0.013898666948080063,
+          0.043186165392398834,
+          0.004656769800931215,
+          -0.008790548890829086,
+          0.02676567994058132,
+          -0.019099920988082886,
+          -0.003883029567077756,
+          0.017480798065662384,
+          -0.01564674638211727,
+          0.04671098291873932,
+          0.011670581065118313,
+          0.005208417773246765,
+          -0.040492404252290726,
+          0.003263321006670594,
+          -0.003961836453527212,
+          -0.014142251573503017,
+          0.03160871937870979,
+          0.015689732506871223,
+          -0.04381662234663963,
+          0.014034787192940712,
+          0.016893327236175537,
+          0.006902765482664108,
+          0.012086108326911926,
+          0.022882649675011635,
+          -0.0038113868795335293,
+          0.03665236011147499,
+          0.017380498349666595,
+          -0.006086039822548628,
+          0.034560397267341614,
+          -0.01265924982726574,
+          0.003514070063829422,
+          0.01422105822712183,
+          -0.007758894935250282,
+          -0.001464195898734033,
+          -0.006175593007355928,
+          0.015517789870500565,
+          0.03619384765625,
+          -0.007680087815970182,
+          -0.03811386972665787,
+          0.03983329236507416,
+          -0.012193572707474232,
+          0.005885440390557051,
+          -0.04986325651407242,
+          0.010051458142697811,
+          -0.01858409307897091,
+          -0.007415010128170252,
+          9.391898493049666e-05,
+          0.008260393515229225,
+          -0.010359521955251694,
+          -0.0012224020902067423,
+          0.015661075711250305,
+          0.00903413351625204,
+          0.014958977699279785,
+          0.024286845698952675,
+          0.02994661219418049,
+          0.008704577572643757,
+          -0.015460475347936153,
+          -0.006064547225832939,
+          0.03903089463710785,
+          -0.01222222950309515,
+          -0.013225225731730461,
+          0.04530678689479828,
+          -0.025447456166148186,
+          -0.027697034180164337,
+          0.03665236011147499,
+          -0.03539144992828369,
+          0.021306512877345085,
+          0.04608052968978882,
+          -0.004029897041618824,
+          -0.02974601276218891,
+          0.01527420524507761,
+          -0.030920950695872307,
+          0.010911169461905956,
+          -0.0026633141096681356,
+          0.0012098646257072687,
+          0.005294389091432095,
+          0.016320187598466873,
+          0.005358867347240448,
+          0.007332621142268181,
+          0.011197740212082863,
+          -0.01792498119175434,
+          0.02107725664973259,
+          -0.03146543353796005,
+          -0.013769710436463356,
+          0.01057444978505373,
+          -0.027510762214660645,
+          -0.0005879171658307314,
+          0.013053284026682377,
+          -0.016563771292567253,
+          -0.030433781445026398,
+          0.018111253157258034,
+          0.009184583090245724,
+          0.06522343307733536,
+          0.01995963230729103,
+          0.016750043258070946,
+          0.009277718141674995,
+          -0.01577570289373398,
+          0.05049371346831322,
+          -0.000637171498965472,
+          0.03215320408344269,
+          0.020117245614528656,
+          -0.016248544678092003,
+          -0.016406157985329628,
+          0.014744049869477749,
+          0.015345847234129906,
+          0.008281885646283627,
+          -0.008016807958483696,
+          0.003514070063829422,
+          0.008955326862633228,
+          0.0017525573493912816,
+          0.02689463645219803,
+          0.006408431567251682,
+          0.02321220561861992,
+          0.03229648992419243,
+          0.0035570557229220867,
+          0.025232527405023575,
+          0.001274342997930944,
+          0.015790032222867012,
+          -0.01914290525019169,
+          0.014958977699279785,
+          0.01901394873857498,
+          0.00896249059587717,
+          0.0193578340113163,
+          -0.006103950552642345,
+          -0.013719560578465462,
+          0.008253228850662708,
+          0.005799469538033009,
+          0.01058877818286419,
+          0.010166086256504059,
+          -0.013726724311709404,
+          0.014887334778904915,
+          -0.032038576900959015,
+          0.014006130397319794,
+          -0.050522368401288986,
+          -0.005785140674561262,
+          0.030204525217413902,
+          0.017108255997300148,
+          -0.025304170325398445,
+          0.018053937703371048,
+          -0.015345847234129906,
+          -0.007852029986679554,
+          -0.038601040840148926,
+          -0.014421657659113407,
+          -0.003650191007182002,
+          0.022710707038640976,
+          0.011226397007703781,
+          0.015603761188685894,
+          0.017308855429291725,
+          -0.011247890070080757,
+          -0.007715909276157618,
+          -0.00841800682246685,
+          -0.03911686688661575,
+          -0.04120882973074913,
+          -0.012193572707474232,
+          0.0006353803910315037,
+          0.00048045325092971325,
+          -0.010875348001718521,
+          -0.02226652391254902,
+          0.009499810636043549,
+          0.022839665412902832,
+          0.0010853855637833476,
+          -0.020074259489774704,
+          0.007393517531454563,
+          -0.022710707038640976,
+          -0.006143353879451752,
+          -0.0424983985722065,
+          -0.020045602694153786,
+          -0.02819853276014328,
+          0.013504632748663425,
+          -0.0126950703561306,
+          0.026665380224585533,
+          -0.003232872812077403,
+          0.0014373299200087786,
+          -0.006444253027439117,
+          0.009707573801279068,
+          -0.008826369419693947,
+          0.006100368220359087,
+          -0.03656638786196709,
+          0.05568063631653786,
+          -0.006125443149358034,
+          0.029831983149051666,
+          -0.016004959121346474,
+          0.05158267915248871,
+          0.00843233522027731,
+          -0.008919505402445793,
+          0.03762669861316681,
+          -0.003850790439173579,
+          0.010882512666285038,
+          0.008797712624073029,
+          -0.03246843069791794,
+          -0.005441256333142519,
+          -0.01834050938487053,
+          0.0008409051806665957,
+          0.010345193557441235,
+          -0.017552439123392105,
+          0.031150206923484802,
+          0.030175868421792984,
+          -0.004663933999836445,
+          0.058976199477910995,
+          0.003345709992572665,
+          -0.018927978351712227,
+          0.008511141873896122,
+          -0.059606652706861496,
+          -0.009772052057087421,
+          -0.028571072965860367,
+          0.011391174979507923,
+          0.015546446666121483,
+          0.026364481076598167,
+          0.0051905070431530476,
+          -0.014185236766934395,
+          0.027611061930656433,
+          0.04281362518668175,
+          0.0002937347162514925,
+          0.020031273365020752,
+          0.012351186014711857,
+          0.00960727408528328,
+          0.009091447107493877,
+          0.0021080838050693274,
+          0.017122583463788033,
+          0.01869872212409973,
+          -0.004921847488731146,
+          -0.0012752384645864367,
+          -0.01581868901848793,
+          -0.006078875623643398,
+          0.0036663105711340904,
+          -0.01732318289577961,
+          -0.008704577572643757,
+          -0.003494368400424719,
+          0.014958977699279785,
+          -0.012723728083074093,
+          0.017738711088895798,
+          -0.02384266071021557,
+          -0.0539325587451458,
+          0.00913443323224783,
+          -0.010789377614855766,
+          -0.02490297146141529,
+          0.009005476720631123,
+          -0.014973306097090244,
+          0.01401329506188631,
+          0.004506320226937532,
+          -0.006172011140733957,
+          -0.014127923175692558,
+          0.010373850353062153,
+          0.003519443329423666,
+          0.040807630866765976,
+          0.006315296050161123,
+          0.006766644772142172,
+          -0.01124072540551424,
+          0.013332690112292767,
+          -0.025304170325398445,
+          0.02342713437974453,
+          -0.010732063092291355,
+          -0.06098219007253647,
+          0.03086363710463047,
+          -0.012623428367078304,
+          -0.010674749501049519,
+          0.03413053974509239,
+          -0.012100436724722385,
+          -0.0006783659919165075,
+          -0.002858540276065469,
+          0.006641270127147436,
+          -0.02286832220852375,
+          0.008396513760089874,
+          -0.04226914048194885,
+          0.010954155586659908,
+          -0.04522081837058067,
+          -0.019945302978157997,
+          0.045020218938589096,
+          0.02149278298020363,
+          0.00312182679772377,
+          0.05103819817304611,
+          0.017652738839387894,
+          0.013776874169707298,
+          0.0035212342627346516,
+          0.014285536482930183,
+          -0.016076602041721344,
+          -0.014242551289498806,
+          -0.006856197956949472,
+          0.019372161477804184,
+          -0.005785140674561262,
+          -0.020776357501745224,
+          0.015016291290521622,
+          0.0035176521632820368,
+          0.0029534667264670134,
+          0.010087279602885246,
+          0.0010773257818073034,
+          -0.013139255344867706,
+          -0.013017462566494942,
+          0.013798367232084274,
+          0.032067231833934784,
+          -0.03395859897136688,
+          -0.0020077840890735388,
+          0.010889677330851555,
+          0.00779471592977643,
+          -0.003854372538626194,
+          -0.03117886371910572,
+          0.01304611936211586,
+          -0.02209458127617836,
+          0.006884855218231678,
+          -0.009026968851685524,
+          -0.013698067516088486,
+          -0.056483034044504166,
+          0.001492852927185595,
+          0.019902316853404045,
+          0.0005453793564811349,
+          -0.009865188039839268,
+          0.0406930036842823,
+          0.016463471576571465,
+          0.021378155797719955,
+          0.0018483793828636408,
+          0.013805530965328217,
+          0.014887334778904915,
+          -0.0006734405760653317,
+          3.0084299851296237e-06,
+          0.005118864588439465,
+          -0.0004316467384342104,
+          -0.0038615367375314236,
+          -0.0015510625671595335,
+          -0.0039725829847157,
+          0.03054841049015522,
+          0.02739613503217697,
+          0.006859780289232731,
+          0.003499741666018963,
+          0.020260531455278397,
+          -0.008998312056064606,
+          -0.005029311403632164,
+          0.010037129744887352,
+          0.013719560578465462,
+          0.006931422743946314,
+          0.007823373191058636,
+          0.01627720147371292,
+          0.002285399241372943,
+          0.003814968978986144,
+          0.013490303419530392,
+          0.012630592100322247,
+          0.0073720249347388744,
+          0.011541624553501606,
+          -0.008797712624073029,
+          0.019759032875299454,
+          -0.022639065980911255,
+          -0.0016880790935829282,
+          0.0008315020822919905,
+          -0.018842006102204323,
+          -0.0004016463935840875,
+          0.00967891700565815,
+          0.012816863134503365,
+          0.011462817899882793,
+          -0.002675851574167609,
+          0.02497461438179016,
+          -0.03630847483873367,
+          -0.011899838224053383,
+          0.004452588502317667,
+          0.028628388419747353,
+          -0.01360493153333664,
+          -0.01957276090979576,
+          0.013368511572480202,
+          -0.008231735788285732,
+          0.02700926549732685,
+          -0.01050280686467886,
+          0.0017534529324620962,
+          -0.013640752993524075,
+          0.05722811818122864,
+          -0.006132607348263264,
+          0.01367657445371151,
+          -0.0012071779929101467,
+          -0.00041284054168500006,
+          0.011011469177901745,
+          0.0015295697376132011,
+          0.04493424668908119,
+          0.026522094383835793,
+          0.009091447107493877,
+          0.018670065328478813,
+          0.02276802249252796,
+          0.016921984031796455,
+          0.005520063452422619,
+          -0.005788723006844521,
+          0.006401267368346453,
+          0.0006600075867027044,
+          -0.011305204592645168,
+          0.02132084034383297,
+          -0.007296800147742033,
+          -0.016721386462450027,
+          0.004011986311525106,
+          -0.010524299927055836,
+          -0.006343953311443329,
+          -0.013647917658090591,
+          0.007758894935250282,
+          0.03679564595222473,
+          -0.001645989017561078,
+          -0.009284882806241512,
+          -0.03928880766034126,
+          0.008281885646283627,
+          0.01693631336092949,
+          0.013733888976275921,
+          -0.020647400990128517,
+          0.005036475602537394,
+          0.020217545330524445,
+          -0.035792648792266846,
+          0.005409017205238342,
+          -0.021578755229711533,
+          0.011169083416461945,
+          -0.010424000211060047,
+          0.027367478236556053,
+          0.0006586642703041434,
+          0.003519443329423666,
+          0.0067881373688578606,
+          -0.009772052057087421,
+          0.02381400391459465,
+          -0.02167905494570732,
+          -0.0003752281772904098,
+          0.005434092134237289,
+          0.008718905970454216,
+          -0.029373470693826675,
+          0.006354699842631817,
+          0.018354836851358414,
+          -0.01591898873448372,
+          0.024817001074552536,
+          0.0285854022949934,
+          -0.008009644225239754,
+          -0.009105775505304337,
+          -0.010911169461905956,
+          0.005828126333653927,
+          0.002023903885856271,
+          0.029187200590968132,
+          -0.003188096219673753,
+          -0.0006156786694191396,
+          -0.007895016111433506,
+          0.010667584836483002,
+          0.0025522680953145027,
+          0.015001962892711163,
+          -0.008561291731894016,
+          -0.03143677860498428,
+          0.039403434842824936,
+          -0.0009385182056576014,
+          -0.008561291731894016,
+          -0.01690765656530857,
+          -0.007415010128170252,
+          0.0015958391595631838,
+          -0.008059794083237648,
+          0.00271346396766603,
+          -0.03476099669933319,
+          0.02146412618458271,
+          0.0020937554072588682,
+          -0.0005014982889406383,
+          0.0013191195903345942,
+          -0.01437867246568203,
+          -0.006849033758044243,
+          -0.0024089827202260494,
+          -0.013225225731730461,
+          0.012480142526328564,
+          0.004574380815029144,
+          -0.018526779487729073,
+          0.013991801999509335,
+          0.009428167715668678,
+          0.016821684315800667,
+          -0.0032973513007164,
+          0.005570212844759226,
+          0.020948300138115883,
+          -0.011348189786076546,
+          -0.03590727597475052,
+          -0.0036197430454194546,
+          -0.00271346396766603,
+          0.030118554830551147,
+          0.006268728524446487,
+          -0.023484447970986366,
+          0.01981634646654129,
+          0.004388109780848026,
+          0.007128439843654633,
+          0.0009922501631081104,
+          -0.008661591447889805,
+          -0.004739158786833286,
+          0.002597044687718153,
+          -0.002835256280377507,
+          -0.00011233337863814086,
+          -0.01022340077906847,
+          0.009478317573666573,
+          -0.00980070885270834,
+          0.005871111992746592,
+          -0.00884069874882698,
+          0.013597767800092697,
+          -0.006791719701141119,
+          -0.026665380224585533,
+          0.004051389638334513,
+          -0.002688389038667083,
+          -0.05235642194747925,
+          -0.0036448179744184017,
+          0.02013157308101654,
+          -0.0074579957872629166,
+          0.005681259091943502,
+          -0.0013862844789400697,
+          0.0028889882378280163,
+          0.017409155145287514,
+          0.008804877288639545,
+          0.026178210973739624,
+          0.016033615916967392,
+          0.01921454817056656,
+          -0.01623421534895897,
+          0.011054455302655697,
+          -0.009564288891851902,
+          0.039260152727365494,
+          0.002247786847874522,
+          -0.0068669444881379604,
+          -0.01736616902053356,
+          0.04418916255235672,
+          -0.00967891700565815,
+          -0.0015304653206840158,
+          -0.01455777883529663,
+          -0.014142251573503017,
+          0.03269768878817558,
+          -0.002256742212921381,
+          0.011512967757880688,
+          -0.004760651383548975,
+          0.008031136356294155,
+          0.0206617284566164,
+          -0.02556208334863186,
+          -0.02766837738454342,
+          -0.017767367884516716,
+          0.007214411161839962,
+          -0.004112286027520895,
+          0.04106554388999939,
+          -0.017910653725266457,
+          -0.0016836014110594988,
+          0.02718120627105236,
+          -0.0269662793725729,
+          -0.03418785333633423,
+          -0.010158922523260117,
+          0.014615092426538467,
+          -0.0010486687533557415,
+          -0.005057968199253082,
+          -0.01942947693169117,
+          -0.020647400990128517,
+          -0.000571797601878643,
+          0.0021868906915187836,
+          0.015603761188685894,
+          0.008862190879881382,
+          0.015388833358883858,
+          0.0019827093929052353,
+          0.0031576480250805616,
+          -0.012236557900905609,
+          0.013683739118278027,
+          0.010810869745910168,
+          0.002262115478515625,
+          -0.014521957375109196,
+          -0.011011469177901745,
+          -0.022366823628544807,
+          -0.008847862482070923,
+          -0.00808128621429205,
+          -0.00023933110060170293,
+          -0.0034531739074736834,
+          -0.015317190438508987,
+          0.02805524691939354,
+          0.00480363704264164,
+          0.00878338422626257,
+          0.015804359689354897,
+          -0.023584747686982155,
+          0.019099920988082886,
+          0.013920159079134464,
+          -0.010739227756857872,
+          0.003189887385815382,
+          -0.006770226638764143,
+          -0.012229394167661667,
+          -0.014858677983283997,
+          0.0038078047800809145,
+          0.015087934210896492,
+          0.011921330355107784,
+          -0.0003962731861975044,
+          -0.015345847234129906,
+          -0.01268074195832014,
+          -0.011713567189872265,
+          0.02563372626900673,
+          -0.030634380877017975,
+          -0.002115248003974557,
+          0.023942960426211357,
+          -0.01995963230729103,
+          -0.011899838224053383,
+          -0.03143677860498428,
+          0.0027994350530207157,
+          -0.02960272692143917,
+          -0.029402127489447594,
+          0.01897096447646618,
+          0.049146831035614014,
+          -0.013261047191917896,
+          -0.010345193557441235,
+          0.01627720147371292,
+          -0.016463471576571465,
+          -0.0021671890281140804,
+          -0.033614713698625565,
+          0.02616388164460659,
+          -0.021664725616574287,
+          0.0038472083397209644,
+          -0.008740399032831192,
+          -0.03499025106430054,
+          0.012687906622886658,
+          0.023541761562228203,
+          0.03338545560836792,
+          -0.001543898251838982,
+          0.00379705848172307,
+          -0.007637102622538805,
+          0.017839010804891586,
+          0.018111253157258034,
+          0.002573760924860835,
+          0.03301291540265083,
+          0.0011032961774617434,
+          -0.04642441123723984,
+          -0.00022880859614815563,
+          0.005151103716343641,
+          -0.0003194812743458897,
+          0.002509282436221838,
+          -0.01366224605590105,
+          -0.026808666065335274,
+          0.015059277415275574,
+          -0.026536423712968826,
+          -0.00040813899249769747,
+          -0.016434814780950546,
+          -0.010818034410476685,
+          -0.007780387531965971,
+          0.005509316921234131,
+          0.004495573695749044,
+          0.024000274017453194,
+          0.009557124227285385,
+          -0.0006985154468566179,
+          0.015016291290521622,
+          0.014529122039675713,
+          0.03069169446825981,
+          -0.0006680673686787486,
+          0.02219488099217415,
+          0.02686597965657711,
+          -0.013110597617924213,
+          0.007343367673456669,
+          0.0028083904180675745,
+          0.026350151747465134,
+          -0.03163737803697586,
+          0.02517521381378174,
+          0.01250163558870554,
+          0.003983329050242901,
+          0.02657940983772278,
+          -0.011985808610916138,
+          -0.00912010483443737,
+          -0.00014709123934153467,
+          0.004001239780336618,
+          -0.003247201442718506,
+          -0.0046925912611186504,
+          0.040635690093040466,
+          0.011491474695503712,
+          -0.024186545982956886,
+          -0.012401335872709751,
+          -0.0158903319388628,
+          0.028040917590260506,
+          -0.019773360341787338,
+          0.00017183032468892634,
+          -0.016492128372192383,
+          0.004918265156447887,
+          0.014314194209873676,
+          0.0014597182162106037,
+          0.01879902184009552,
+          0.016019288450479507,
+          -0.0006586642703041434,
+          0.018354836851358414,
+          0.007751730736345053,
+          -0.026393137872219086,
+          0.004839458502829075,
+          0.007436503190547228,
+          -0.014020458795130253,
+          0.007529638707637787,
+          0.015503461472690105,
+          -0.00407646456733346,
+          -0.013683739118278027,
+          -0.01230820082128048,
+          -0.00862576998770237,
+          0.004595873411744833,
+          -0.010638928040862083,
+          0.019300520420074463,
+          0.0018806185107678175,
+          -0.0038113868795335293,
+          0.019859332591295242,
+          -0.006734405644237995,
+          -0.023025935515761375,
+          0.01897096447646618,
+          -0.01437867246568203,
+          0.02441580221056938,
+          0.012530292384326458,
+          -0.022123238071799278,
+          0.00579588720574975,
+          0.009929666295647621,
+          0.04226914048194885,
+          -0.005878276191651821,
+          0.00615410041064024,
+          -0.02703792229294777,
+          -0.005584541708230972,
+          -0.02609223872423172,
+          0.0067845555022358894,
+          0.003377949120476842,
+          -0.011276546865701675,
+          -0.00021190541156101972,
+          -0.0012447903864085674,
+          0.014142251573503017,
+          0.012809699401259422,
+          -0.014873006381094456,
+          -0.0011149381753057241,
+          0.0036860124673694372,
+          -0.01528853364288807,
+          -0.0024698791094124317,
+          0.0011955361114814878,
+          -0.008317707106471062,
+          0.02293996512889862,
+          0.0393461212515831,
+          -0.021693382412195206,
+          -0.01760975457727909,
+          0.016878999769687653,
+          -0.003571384120732546,
+          -0.022438466548919678,
+          -0.007508145645260811,
+          0.01113326195627451,
+          0.0054376740008592606,
+          -0.0025558501947671175,
+          -0.03372934088110924,
+          0.021507112309336662,
+          0.006472909823060036,
+          0.014958977699279785,
+          0.00788785144686699,
+          0.0026955532375723124,
+          0.030347811058163643,
+          -0.020002616569399834,
+          -0.018770365044474602,
+          -0.00569916982203722,
+          -0.020188888534903526,
+          0.019099920988082886,
+          0.008790548890829086,
+          -0.011770880781114101,
+          0.02829883247613907,
+          -0.026493437588214874,
+          0.015560775063931942,
+          0.009299211204051971,
+          -0.0051976717077195644,
+          0.010603106580674648,
+          0.0028227188158780336,
+          -0.004914683289825916,
+          -0.04854503273963928,
+          -0.005100953858345747,
+          0.008074122481048107,
+          0.02104859985411167,
+          -0.0060251434333622456,
+          -0.031408119946718216,
+          0.02974601276218891,
+          0.0008785175159573555,
+          -0.002616746351122856,
+          -0.010058622807264328,
+          0.00706754345446825,
+          -0.020561428740620613,
+          -0.006863362155854702,
+          0.007053215056657791,
+          0.020461129024624825,
+          0.0059427544474601746,
+          0.028485102578997612,
+          -0.006028725765645504,
+          0.05232776328921318,
+          -0.0006617986364290118,
+          0.009435332380235195,
+          0.0026418212801218033,
+          -0.002097337506711483,
+          -0.0017946474254131317,
+          -0.013504632748663425,
+          -0.026292838156223297,
+          -0.021120242774486542,
+          -0.006010815035551786,
+          -0.00208300887607038,
+          0.010316535830497742,
+          0.008704577572643757,
+          -0.02034650184214115,
+          0.0009967278456315398,
+          -0.039546720683574677,
+          0.009170254692435265,
+          -0.027023592963814735,
+          -0.009843694977462292,
+          -0.0035122791305184364,
+          -0.004481245297938585,
+          -0.022567423060536385,
+          0.022366823628544807,
+          0.019930975511670113,
+          -0.032181862741708755,
+          0.002450177213177085,
+          0.008847862482070923,
+          0.0079451659694314,
+          0.01339716836810112,
+          2.5690593247418292e-05,
+          -0.02412923239171505,
+          -0.014744049869477749,
+          -0.0006953810807317495,
+          0.01029504369944334,
+          -0.008740399032831192,
+          -0.026421794667840004,
+          0.0019253952195867896,
+          0.03788461163640022,
+          -0.007017393596470356,
+          0.013461646623909473,
+          -0.00047776667634025216,
+          -0.00869741290807724,
+          -0.0222378671169281,
+          0.004080046899616718,
+          0.019701717421412468,
+          0.025576412677764893,
+          -0.009378017857670784,
+          0.004954086616635323,
+          -0.0029928700532764196,
+          0.006343953311443329,
+          -0.00632246071472764,
+          0.0006474701222032309,
+          0.01826886646449566,
+          -0.029545413330197334,
+          0.004739158786833286,
+          -0.008031136356294155,
+          -0.011978644877672195,
+          0.010832362808287144,
+          -0.0004674680531024933,
+          0.014256879687309265,
+          -0.016506457701325417,
+          -0.0018573347479104996,
+          0.030003925785422325,
+          0.00407646456733346,
+          -0.025017600506544113,
+          0.012752384878695011,
+          -0.02556208334863186,
+          -0.007192918099462986,
+          -0.0103093720972538,
+          0.023685047402977943,
+          0.01331119704991579,
+          0.00925622507929802,
+          0.027152549475431442,
+          -0.0006125443032942712,
+          0.01792498119175434,
+          -0.004878861829638481,
+          -0.0033277992624789476,
+          0.0016853924607858062,
+          -0.03209589049220085,
+          0.0047785621136426926,
+          0.029889296740293503,
+          -0.03395859897136688,
+          -0.004402438644319773,
+          0.010524299927055836,
+          0.02359907701611519,
+          -0.016993626952171326,
+          0.008281885646283627,
+          0.014092101715505123,
+          -0.006326042581349611,
+          0.00897681899368763,
+          0.025089243426918983,
+          0.049462057650089264,
+          -0.00248599867336452,
+          0.006175593007355928,
+          0.009578617289662361,
+          0.013626424595713615,
+          -0.0007370233652181923,
+          0.008812041021883488,
+          -0.05089491233229637,
+          0.01258760690689087,
+          -0.006394103169441223,
+          -0.022682050243020058,
+          -0.0053624496795237064,
+          -0.016248544678092003,
+          0.017093926668167114,
+          0.026135224848985672,
+          0.02352743409574032,
+          0.01518823392689228,
+          0.013339853845536709,
+          0.02841345965862274,
+          0.012007301673293114,
+          0.0052728960290551186,
+          0.004689008928835392,
+          -0.003426307812333107,
+          0.0022065923549234867,
+          -0.004126614425331354,
+          0.009557124227285385,
+          -0.005287224892526865,
+          0.02819853276014328,
+          0.007723073475062847,
+          0.043472737073898315,
+          -0.03401591256260872,
+          -0.006397685036063194,
+          0.015790032222867012,
+          -0.016735713928937912,
+          -0.0017821099609136581,
+          0.007687252014875412,
+          0.02027485892176628,
+          -0.014543450437486172,
+          0.011985808610916138,
+          -0.0033994419500231743,
+          0.010323700495064259,
+          -0.0042698998004198074,
+          0.01339716836810112,
+          -0.002792270854115486,
+          0.016004959121346474,
+          0.01824020966887474,
+          0.006576791871339083,
+          0.005738573148846626,
+          0.006017979234457016,
+          0.00019522612274158746,
+          -0.022366823628544807,
+          -0.011892673559486866,
+          0.00208300887607038,
+          0.004230496007949114,
+          -0.00625081779435277,
+          0.004456170368939638,
+          0.014622257091104984,
+          -0.02034650184214115,
+          0.016578100621700287,
+          0.019515447318553925,
+          -0.004753487184643745,
+          -0.00034903385676443577,
+          0.0015304653206840158,
+          -0.011641924269497395,
+          -0.015245547518134117,
+          -0.022180551663041115,
+          -0.011161918751895428,
+          -0.012859849259257317,
+          -0.005681259091943502,
+          0.01654944382607937,
+          0.00994399469345808,
+          -0.014729720540344715,
+          0.019271861761808395,
+          -0.027768677100539207,
+          0.019071262329816818,
+          0.004742741119116545,
+          -0.0055308095179498196,
+          -0.014593600295484066,
+          -0.00994399469345808,
+          0.04416050761938095,
+          0.00787352304905653,
+          0.002009575255215168,
+          -0.010882512666285038,
+          -0.004201839212328196,
+          -0.018741706386208534,
+          -0.002480625407770276,
+          -0.0090484619140625,
+          0.024774014949798584,
+          0.01630585826933384,
+          0.008074122481048107,
+          -0.020160231739282608,
+          0.008281885646283627,
+          0.005462748929858208,
+          0.016176901757717133,
+          0.012394172139465809,
+          0.01819722354412079,
+          -0.010402507148683071,
+          -0.021765025332570076,
+          0.0010424000211060047,
+          -0.007880687713623047,
+          0.017767367884516716,
+          0.008983983658254147,
+          0.0038221334107220173,
+          0.006182757206261158,
+          -0.014299864880740643,
+          0.012773877941071987,
+          0.027969276532530785,
+          -0.0004370199458207935,
+          0.021750696003437042,
+          -0.00995832309126854,
+          -0.00834636390209198,
+          0.006852615624666214,
+          0.017409155145287514,
+          0.018354836851358414,
+          -0.03137946501374245,
+          -0.009471152909100056,
+          -0.008862190879881382,
+          -0.011075947433710098,
+          -0.011706402525305748,
+          0.0003340336843393743,
+          -0.019973959773778915,
+          -0.0126950703561306,
+          -0.0104884784668684,
+          -0.01749512553215027,
+          0.005401853006333113,
+          -0.009041297249495983,
+          0.004320049658417702,
+          -0.013089105486869812,
+          -0.034388452768325806,
+          -0.01504494808614254,
+          0.020074259489774704,
+          -0.002005993155762553,
+          -0.009793545119464397,
+          0.008145765401422977,
+          -0.0016997209750115871,
+          0.010373850353062153,
+          0.007629937957972288,
+          -0.013812695629894733,
+          -0.014350014738738537,
+          0.00013623290578834713,
+          -0.004474081099033356,
+          0.010538628324866295,
+          -0.010595941916108131,
+          -0.004581545013934374,
+          -0.012974477373063564,
+          -0.004277063999325037,
+          -0.017452139407396317,
+          -0.015790032222867012,
+          0.006730823311954737,
+          -0.018956635147333145,
+          0.01360493153333664,
+          -0.023728033527731895,
+          -0.012387007474899292,
+          0.020074259489774704,
+          0.002625701716169715,
+          -0.016850342974066734,
+          -0.01869872212409973,
+          -0.008045465685427189,
+          -0.019615747034549713,
+          -0.009456824511289597,
+          0.003207797883078456,
+          0.02686597965657711,
+          0.005838872864842415,
+          0.014314194209873676,
+          -0.0017068851739168167,
+          0.032038576900959015,
+          0.006286639254540205,
+          0.029889296740293503,
+          -0.005863947793841362,
+          0.009363689459860325,
+          -0.02458774484694004,
+          -0.003499741666018963,
+          -0.009112940169870853,
+          0.0036340714432299137,
+          0.00044642301509156823,
+          -0.006852615624666214,
+          -0.0012734474148601294,
+          -0.0023283849004656076,
+          -0.007400681730359793,
+          -0.009585781022906303,
+          0.015575103461742401,
+          0.006591120269149542,
+          0.005129611119627953,
+          0.004183928482234478,
+          0.03332814201712608,
+          -0.006684255786240101,
+          -0.013655081391334534,
+          0.01366224605590105,
+          0.023470120504498482,
+          -0.016893327236175537,
+          -0.000970757391769439,
+          0.009084283374249935,
+          0.011290875263512135,
+          -0.023871317505836487,
+          0.01940082013607025,
+          -0.004696173127740622,
+          -0.004871697630733252,
+          0.008074122481048107,
+          0.002426893450319767,
+          0.01705094240605831,
+          -0.0029212275985628366,
+          -0.0019988289568573236,
+          -0.009936830028891563,
+          0.003943925723433495,
+          -0.015102262608706951,
+          -0.0007929941639304161,
+          0.01255895011126995,
+          -0.011405503377318382,
+          0.008088450878858566,
+          -0.027825990691781044,
+          -0.029831983149051666,
+          -0.01705094240605831,
+          0.01050280686467886,
+          0.002369579393416643,
+          -0.022395480424165726,
+          0.018956635147333145,
+          -0.0038113868795335293,
+          -0.017280198633670807,
+          0.018598422408103943,
+          -0.006501567084342241,
+          -0.003922433126717806,
+          -0.0005767230177298188,
+          0.020633071660995483,
+          0.005842455197125673,
+          -0.005623945035040379,
+          -0.002880033105611801,
+          -0.0069529153406620026,
+          -0.00317376758903265,
+          -0.002838838379830122,
+          0.001708676340058446,
+          -0.0206187441945076,
+          0.00434154225513339,
+          0.01268074195832014,
+          0.01513091940432787,
+          -0.0007625460275448859,
+          0.02829883247613907,
+          -0.005086625460535288,
+          0.00397974718362093,
+          0.006852615624666214,
+          -0.0393461212515831,
+          5.3284191380953416e-05,
+          -0.018569765612483025,
+          -0.008582784794270992,
+          0.010044294409453869,
+          0.009478317573666573,
+          0.001631660503335297,
+          -0.008016807958483696,
+          0.0028065992519259453,
+          -0.009213239885866642,
+          -2.209446938650217e-05,
+          0.007286053616553545,
+          0.022080253809690475,
+          0.013196568936109543,
+          -0.004434677772223949,
+          0.02991795539855957,
+          -0.011434161104261875,
+          0.007443667389452457,
+          0.012322529219090939,
+          0.008898012340068817,
+          -0.0026382391806691885,
+          0.02076202817261219,
+          -0.009198911488056183,
+          -0.009772052057087421,
+          -0.01241566427052021,
+          0.014163744635879993,
+          0.01571838930249214,
+          -0.025261184200644493,
+          -0.0024770433083176613,
+          -0.003428098978474736,
+          0.005262149963527918,
+          -0.002656149910762906,
+          0.012974477373063564,
+          -0.01882767863571644,
+          0.025160884484648705,
+          0.031150206923484802,
+          -0.008704577572643757,
+          -0.011584610678255558,
+          0.0019110667053610086,
+          0.00267406040802598,
+          0.01268074195832014,
+          0.0033385457936674356,
+          0.006519477814435959,
+          -0.003358247457072139,
+          -0.02626418136060238,
+          -0.021836668252944946,
+          0.008403678424656391,
+          -0.01436434406787157,
+          -0.004072882700711489,
+          0.00931353960186243,
+          0.016076602041721344,
+          0.0010352357057854533,
+          0.0060358899645507336,
+          0.022452794015407562,
+          -0.02514655701816082,
+          0.0036824301350861788,
+          -0.01083952747285366,
+          -0.0023892810568213463,
+          -0.006017979234457016,
+          -0.034531738609075546,
+          0.011713567189872265,
+          0.006168428808450699,
+          -0.017667068168520927,
+          0.0023767435923218727,
+          -0.014063444919884205,
+          0.016463471576571465,
+          -0.0007370233652181923,
+          0.03160871937870979,
+          0.008195914328098297,
+          -0.006813212297856808,
+          0.00489319022744894,
+          0.02332683466374874,
+          0.04083628952503204,
+          0.010037129744887352,
+          0.004835876170545816,
+          -0.019300520420074463,
+          -0.00960727408528328,
+          -0.010746391490101814,
+          -0.007708745077252388,
+          -0.01942947693169117,
+          0.009528467431664467,
+          -0.012816863134503365,
+          -0.002960630925372243,
+          0.01213625818490982,
+          -0.0010952363954856992,
+          -0.020991284400224686,
+          0.014729720540344715,
+          0.014271208085119724,
+          -0.0316946916282177,
+          -0.013948816806077957,
+          -0.031236179172992706,
+          0.03054841049015522,
+          -0.020919643342494965,
+          -0.0257053691893816,
+          0.024501772597432137,
+          -0.007572623901069164,
+          0.018498122692108154,
+          0.010595941916108131,
+          0.005842455197125673,
+          -0.0002697792078834027,
+          0.004731994587928057,
+          0.007175007369369268,
+          -0.025576412677764893,
+          0.028786001726984978,
+          0.006759480573236942,
+          -0.016420487314462662,
+          0.008554127998650074,
+          -0.025347156450152397,
+          -0.024114903062582016,
+          0.02170771174132824,
+          0.019773360341787338,
+          0.00048358761705458164,
+          0.01078221295028925,
+          -0.0007495608297176659,
+          -0.0009940412128344178,
+          -0.013970308937132359,
+          -0.008940997533500195,
+          -0.000573140918277204,
+          0.02132084034383297,
+          -0.017036613076925278,
+          0.008446663618087769,
+          0.01255895011126995,
+          8.630695811007172e-05,
+          -0.02805524691939354,
+          -0.00924189668148756,
+          -0.004491991829127073,
+          -0.028800329193472862,
+          0.0076227737590670586,
+          -0.00469975546002388,
+          0.003743326524272561,
+          -0.02381400391459465,
+          0.0019468879327178001,
+          -0.04201122745871544,
+          -0.03089229390025139,
+          0.00140509067568928,
+          -0.029101228341460228,
+          -0.012250886298716068,
+          -0.02371370419859886,
+          0.01131236832588911,
+          -0.01503061968833208,
+          -0.0006752316257916391,
+          0.007665759418159723,
+          -0.01725153997540474,
+          0.006583956070244312,
+          -0.019859332591295242,
+          0.004363034851849079,
+          -0.011441324837505817,
+          0.012673578225076199,
+          0.017595425248146057,
+          0.020174559205770493,
+          0.005333792418241501,
+          -0.002863913308829069,
+          0.03103557974100113,
+          -0.008403678424656391,
+          -0.016363171860575676,
+          -0.006476492155343294,
+          0.017910653725266457,
+          -0.00014597181871067733,
+          0.03576399013400078,
+          0.0016558398492634296,
+          -0.0008068748866207898,
+          0.0036000411491841078,
+          0.007175007369369268,
+          0.0047785621136426926,
+          -0.006831123027950525,
+          0.0269662793725729,
+          0.010352357290685177,
+          0.012752384878695011,
+          -0.006150518078356981,
+          -0.009657423943281174,
+          -0.006759480573236942,
+          0.014077773317694664,
+          0.016463471576571465,
+          0.012107601389288902,
+          0.010409671813249588,
+          0.00489319022744894,
+          -0.017065269872546196,
+          0.01894230581820011,
+          -0.02422953210771084,
+          0.011369682848453522,
+          -0.020890984684228897,
+          0.007343367673456669,
+          -0.0007303068996407092,
+          0.00834636390209198,
+          -0.0020167394541203976,
+          -0.005745737347751856,
+          0.00823890045285225,
+          0.012236557900905609,
+          -0.01230820082128048,
+          -0.0014015085762366652,
+          -0.0253901407122612,
+          -0.01201446633785963,
+          0.0067057483829557896,
+          0.018297523260116577,
+          0.024243859574198723,
+          -0.03089229390025139,
+          0.0038078047800809145,
+          -0.000754934037104249,
+          -0.020719043910503387,
+          0.0035212342627346516,
+          0.013282540254294872,
+          -0.015546446666121483,
+          -0.0009528467198833823,
+          0.023054592311382294,
+          -0.002648985479027033,
+          0.009198911488056183,
+          0.013182240538299084,
+          -0.009299211204051971,
+          0.015317190438508987,
+          0.013741052709519863,
+          0.01729452610015869,
+          -0.009277718141674995,
+          0.016176901757717133,
+          -0.018541108816862106,
+          0.008647263050079346,
+          0.02059008739888668,
+          0.020461129024624825,
+          -0.02069038711488247,
+          0.026636723428964615,
+          0.018254537135362625,
+          -0.020174559205770493,
+          -0.0016925566596910357,
+          -0.007773223333060741,
+          0.011842523701488972,
+          0.014235386624932289,
+          -0.004312884993851185,
+          -0.025089243426918983,
+          -0.01346881128847599,
+          0.010280714370310307,
+          -0.010509971529245377,
+          -0.004739158786833286,
+          -0.015259875915944576,
+          0.010366685688495636,
+          0.0007634415524080396,
+          0.020676057785749435,
+          0.03258306160569191,
+          0.014335686340928078,
+          -0.006662762723863125,
+          0.02599193900823593,
+          0.03215320408344269,
+          0.00541976373642683,
+          -0.005946336779743433,
+          0.010022801347076893,
+          0.0010567285353317857,
+          -0.009213239885866642,
+          -0.003437054343521595,
+          -0.001859125797636807,
+          -0.0142927011474967,
+          -0.0064263422973454,
+          0.014930319972336292,
+          0.03100692294538021,
+          -0.005222746171057224,
+          -0.020575758069753647,
+          0.00858994945883751,
+          0.00912010483443737,
+          0.014564942568540573,
+          0.004997072275727987,
+          0.018999621272087097,
+          0.015489133074879646,
+          0.016219887882471085,
+          0.005835290532559156,
+          -0.023685047402977943,
+          -0.008031136356294155,
+          0.009750558994710445,
+          -0.02114889957010746,
+          -0.006451417226344347,
+          -0.004843040369451046,
+          -0.039747320115566254,
+          -0.021550098434090614,
+          0.014550614170730114,
+          0.026436123996973038,
+          0.01167774572968483,
+          -0.0009832947980612516,
+          -0.015073605813086033,
+          0.0029140631668269634,
+          0.0038113868795335293,
+          0.01346881128847599,
+          0.012601935304701328,
+          0.0013746425975114107,
+          0.00011899389937752858,
+          0.02301160618662834,
+          0.0028674956411123276,
+          -0.012580442242324352,
+          0.005785140674561262,
+          -0.008181585930287838,
+          -0.007923672907054424,
+          0.003062721574679017,
+          -0.005161850247532129,
+          -0.004452588502317667,
+          0.010775049217045307,
+          0.006096786353737116,
+          -0.003505114698782563,
+          -0.010194743983447552,
+          0.0011641924502328038,
+          0.010638928040862083,
+          0.002199428156018257,
+          0.0020453964825719595,
+          0.008847862482070923,
+          -0.011534460820257664,
+          -0.01043116394430399,
+          -0.015503461472690105,
+          0.007694416679441929,
+          -0.015016291290521622,
+          0.006902765482664108,
+          0.022108910605311394,
+          -0.004488409496843815,
+          -0.026708366349339485,
+          -0.008253228850662708,
+          0.008203078992664814,
+          -0.014507628977298737,
+          0.005745737347751856,
+          0.018096923828125,
+          -0.0009582199272699654,
+          0.009850858710706234,
+          -0.01346881128847599,
+          0.015489133074879646,
+          0.013253883458673954,
+          0.02139248326420784,
+          -0.0004813488048966974,
+          -0.007758894935250282,
+          0.011706402525305748,
+          0.0036681017372757196,
+          -0.017122583463788033,
+          0.009714738465845585,
+          0.03427382558584213,
+          -0.001524196588434279,
+          0.005964247509837151,
+          0.0016943477094173431,
+          0.0043558706529438496,
+          -0.001223297556862235,
+          -0.0010379223385825753,
+          0.004724830389022827,
+          0.012816863134503365,
+          0.005677676759660244,
+          -0.0038758653681725264,
+          0.009721902199089527,
+          0.008804877288639545,
+          -0.004735576454550028,
+          -0.02703792229294777,
+          -0.004499156028032303,
+          -0.0008422484388574958,
+          -0.0067773908376693726,
+          0.007386353332549334,
+          -0.016721386462450027,
+          0.007099783048033714,
+          -0.006043054163455963,
+          0.014350014738738537,
+          -0.0033134708646684885,
+          0.009736230596899986,
+          0.019128577783703804,
+          0.01422105822712183,
+          0.01237984374165535,
+          -0.007286053616553545,
+          0.010080115869641304,
+          0.007952329702675343,
+          0.009793545119464397,
+          0.0027403298299759626,
+          -0.006071711424738169,
+          0.0021421140991151333,
+          -0.004835876170545816,
+          0.0064263422973454,
+          0.0034191436134278774,
+          0.015603761188685894,
+          -0.01558943185955286,
+          -0.017308855429291725,
+          -0.000402318051783368,
+          0.032840974628925323,
+          0.001868081046268344,
+          0.0019182309042662382,
+          -0.009234732948243618,
+          0.010266385972499847,
+          0.000753590720705688,
+          -0.012859849259257317,
+          -0.007257396355271339,
+          0.026350151747465134,
+          -0.0021295766346156597,
+          0.0017749457620084286,
+          -0.0034728755708783865,
+          -0.0016692728968337178,
+          0.012745220214128494,
+          0.012365514412522316,
+          -0.011684910394251347,
+          0.018053937703371048,
+          -0.004585127346217632,
+          0.02293996512889862,
+          0.013225225731730461,
+          0.0018376329680904746,
+          -0.0005776185425929725,
+          -0.008038301020860672,
+          0.0074579957872629166,
+          -0.02339847758412361,
+          0.002009575255215168,
+          -0.0090484619140625,
+          -0.010445492342114449,
+          -0.0055308095179498196,
+          -0.025791339576244354,
+          0.00451706675812602,
+          0.002878241939470172,
+          -0.011684910394251347,
+          0.0016979299252852798,
+          0.005645437631756067,
+          0.007637102622538805,
+          0.013425825163722038,
+          -0.01420672982931137,
+          -0.005749319680035114,
+          0.010724899359047413,
+          0.0009832947980612516,
+          0.005520063452422619,
+          -0.020776357501745224,
+          0.01676437072455883,
+          -0.0065947026014328,
+          -0.02689463645219803,
+          0.007063961587846279,
+          -0.00406930036842823,
+          0.01662108674645424,
+          0.00806695781648159,
+          0.002824509982019663,
+          -0.020532771944999695,
+          0.018713049590587616,
+          -0.004284228198230267,
+          -0.012494471855461597,
+          -0.019343504682183266,
+          -0.0030179449822753668,
+          -0.007973822765052319,
+          0.018713049590587616,
+          -0.007651431020349264,
+          -0.0336720272898674,
+          0.004660351667553186,
+          -0.022223537787795067,
+          -0.017480798065662384,
+          -0.022223537787795067,
+          0.010323700495064259,
+          -0.005627527367323637,
+          0.011004305444657803,
+          -0.009714738465845585,
+          0.007551131304353476,
+          -0.029402127489447594,
+          0.019372161477804184,
+          0.007020975928753614,
+          0.017896324396133423,
+          0.019687389954924583,
+          0.0273244921118021,
+          -0.011649088934063911,
+          0.0048609510995447636,
+          -0.022710707038640976,
+          -1.0858333553187549e-05,
+          -0.019372161477804184,
+          -0.007472324185073376,
+          0.003976164851337671,
+          0.017738711088895798,
+          0.00680246576666832,
+          0.003290186868980527,
+          -0.016076602041721344,
+          -0.022037267684936523,
+          -0.03215320408344269,
+          -0.027123892679810524,
+          -0.004230496007949114,
+          0.006139772012829781,
+          0.013741052709519863,
+          -0.002645403379574418,
+          -0.015904659405350685,
+          3.201529034413397e-05,
+          0.012759549543261528,
+          -0.01960141770541668,
+          -0.002000619890168309,
+          -0.03278365731239319,
+          -0.010072951205074787,
+          -0.029545413330197334,
+          0.018211551010608673,
+          0.021936967968940735,
+          0.004760651383548975,
+          -0.011534460820257664,
+          -0.012967312708497047,
+          -0.018383493646979332,
+          0.0016504667000845075,
+          -0.006361864041537046,
+          -0.013003134168684483,
+          -0.03490427881479263,
+          -0.012143422849476337,
+          0.011935658752918243,
+          0.010087279602885246,
+          -0.006394103169441223,
+          -0.0018331552855670452,
+          0.011462817899882793,
+          -0.005681259091943502,
+          -0.005237075034528971,
+          -0.0036967587657272816,
+          -0.02146412618458271,
+          -0.015302862040698528,
+          0.006644852459430695,
+          -0.006390520837157965,
+          0.01257327850908041,
+          -0.013898666948080063,
+          0.0016003168420866132,
+          -0.008675919845700264,
+          0.006755898240953684,
+          0.00275824056006968,
+          -0.010072951205074787,
+          -0.010617434978485107,
+          0.016707057133316994,
+          0.010825199075043201,
+          -0.016878999769687653,
+          -0.003650191007182002,
+          0.01023772917687893,
+          0.00939951092004776,
+          -0.009700410068035126,
+          -0.028313159942626953,
+          -0.013432989828288555,
+          0.014242551289498806,
+          -0.01103296224027872,
+          0.01887066476047039,
+          -0.012093272991478443,
+          -0.00831054337322712,
+          0.020432472229003906,
+          0.008246064186096191,
+          -0.003408397315070033,
+          0.0007831432740204036,
+          -0.004123032093048096,
+          -0.003528398694470525,
+          -0.006956497672945261,
+          0.015016291290521622,
+          -0.015102262608706951,
+          0.008940997533500195,
+          -0.007096200715750456,
+          0.014314194209873676,
+          -0.00787352304905653,
+          0.0034782488364726305,
+          0.023341162130236626,
+          -0.006723659113049507,
+          -0.013411496765911579,
+          0.008389350026845932,
+          0.01448613591492176,
+          0.02122054249048233,
+          0.00913443323224783,
+          -0.003068094840273261,
+          -0.01296014804393053,
+          -0.014271208085119724,
+          -0.00896249059587717,
+          0.025203870609402657,
+          -0.01518823392689228,
+          -0.006652016658335924,
+          0.009456824511289597,
+          -0.0005064237047918141,
+          -0.011971480213105679,
+          -0.002588089322671294,
+          -0.025719698518514633,
+          -0.005351703148335218,
+          0.02286832220852375,
+          -0.03258306160569191,
+          -0.005910515319555998,
+          0.017065269872546196,
+          -0.01402762345969677,
+          -0.017423482611775398,
+          0.010982812382280827,
+          -0.015159576199948788,
+          0.009220404550433159,
+          -0.008038301020860672,
+          0.01430702954530716,
+          0.000650156696792692,
+          -0.02367071807384491,
+          0.005143939517438412,
+          0.021206213161349297,
+          -0.02437281608581543,
+          0.004577962681651115,
+          -0.0443897619843483,
+          0.002100919606164098,
+          0.003943925723433495,
+          0.014607928693294525,
+          -0.00925622507929802,
+          0.0002228756929980591,
+          -0.004044225439429283,
+          -0.012279544025659561,
+          0.017552439123392105,
+          -0.008948162198066711,
+          -0.02230951003730297,
+          0.002145696198567748,
+          0.005176178645342588,
+          0.007038886658847332,
+          -0.009743395261466503,
+          0.013239555060863495,
+          -0.0019146488048136234,
+          -0.010037129744887352,
+          0.016262872144579887,
+          -0.0007576206116937101,
+          -0.020604414865374565,
+          -0.02013157308101654,
+          -0.00244122208096087,
+          -0.024358488619327545,
+          0.012071779929101467,
+          -0.004760651383548975,
+          -0.0015743464464321733,
+          -0.005480659659951925,
+          0.016535114496946335,
+          0.00027179415337741375,
+          0.017867667600512505,
+          -0.0027295835316181183,
+          0.012200736440718174,
+          0.00022567423002328724,
+          0.00352481659501791,
+          -0.02325519174337387,
+          0.0005046326550655067,
+          -0.003368993755429983,
+          -0.012472978793084621,
+          -0.016334515064954758,
+          -0.01957276090979576,
+          -0.018010953441262245,
+          -0.010058622807264328,
+          0.016606757417321205,
+          0.025619398802518845,
+          0.006863362155854702,
+          -0.020432472229003906,
+          0.013755381107330322,
+          -0.004667516332119703,
+          0.00959294568747282,
+          0.0013459855690598488,
+          -0.009478317573666573,
+          0.029688699170947075,
+          0.013110597617924213,
+          0.018741706386208534,
+          -0.003703922964632511,
+          -0.028169874101877213,
+          -0.01819722354412079,
+          -0.0177816953510046,
+          0.010810869745910168,
+          -0.010825199075043201,
+          0.010338028892874718,
+          -0.011727895587682724,
+          -0.00787352304905653,
+          -0.001703303074464202,
+          -0.021234869956970215,
+          0.032181862741708755,
+          -0.024988943710923195,
+          0.0046102022752165794,
+          0.017409155145287514,
+          -0.0015886749606579542,
+          -0.010323700495064259,
+          0.023556090891361237,
+          0.002005993155762553,
+          -0.022925635799765587,
+          -0.00966458860784769,
+          0.010037129744887352,
+          -0.012988805770874023,
+          0.005176178645342588,
+          -1.1285110304015689e-05,
+          0.007171425502747297,
+          -0.028327489271759987,
+          0.012387007474899292,
+          0.018053937703371048,
+          0.013490303419530392,
+          0.02276802249252796,
+          0.0023122653365135193,
+          0.013003134168684483,
+          -0.012286707758903503,
+          0.006386938970535994,
+          0.01988798938691616,
+          0.0079451659694314,
+          -0.028040917590260506,
+          0.020676057785749435,
+          -0.021693382412195206,
+          0.018598422408103943,
+          -0.0042698998004198074,
+          -0.0031003339681774378,
+          0.03387262672185898,
+          -0.010667584836483002,
+          -0.004456170368939638,
+          -0.01069624163210392,
+          -0.004989908076822758,
+          -0.003675265936180949,
+          -0.011634760536253452,
+          0.02633582428097725,
+          0.024716701358556747,
+          -0.021019943058490753,
+          0.01211476605385542,
+          0.002511073602363467,
+          0.012243722565472126,
+          -0.00697082607075572,
+          0.023441461846232414,
+          0.014507628977298737,
+          -0.007314710877835751,
+          0.009019805118441582,
+          0.004051389638334513,
+          -0.003911686595529318,
+          0.0006891123484820127,
+          -0.013447318226099014,
+          -0.007386353332549334,
+          -0.006053800694644451,
+          0.011555952951312065,
+          0.008554127998650074,
+          -0.008016807958483696,
+          0.003718251595273614,
+          0.0003978403692599386,
+          0.0012931491946801543,
+          -0.01002996601164341,
+          0.0074579957872629166,
+          0.0035122791305184364,
+          0.00869741290807724,
+          0.011212068609893322,
+          0.0033672028221189976,
+          -0.009213239885866642,
+          -0.00451706675812602,
+          0.01824020966887474,
+          0.016191231086850166,
+          0.00976488832384348,
+          0.014958977699279785,
+          -0.01652078703045845,
+          0.0253901407122612,
+          0.024530429393053055,
+          0.0007558295619674027,
+          -0.01176371704787016,
+          0.0053552850149571896,
+          0.0028137636836618185,
+          -0.008632934652268887,
+          0.011054455302655697,
+          -0.006017979234457016,
+          0.015216890722513199,
+          0.003354665357619524,
+          -0.00806695781648159,
+          -0.006161264609545469,
+          -0.027095235884189606,
+          -0.037540730088949203,
+          0.002767195925116539,
+          -0.004599455744028091,
+          -0.006723659113049507,
+          -0.0026579408440738916,
+          -0.009986979886889458,
+          0.0014865841949358582,
+          0.006261564325541258,
+          -0.006010815035551786,
+          0.01202879473567009,
+          -0.023226534947752953,
+          -0.005423345603048801,
+          0.000444184202933684,
+          -0.010459821671247482,
+          -2.4809052774799056e-05,
+          -0.007046050857752562,
+          -0.010051458142697811,
+          0.005906933452934027,
+          -0.022782349959015846,
+          -0.000383287959266454,
+          -0.01763841137290001,
+          0.01889932155609131,
+          -0.008632934652268887,
+          0.0045206486247479916,
+          0.0016504667000845075,
+          -0.013074777089059353,
+          0.01504494808614254,
+          0.011491474695503712,
+          -0.010509971529245377,
+          0.002151069464161992,
+          0.008876519277691841,
+          0.0025558501947671175,
+          0.0011767299147322774,
+          -0.021793682128190994,
+          -0.01139833964407444,
+          0.02289697900414467,
+          0.002111665904521942,
+          -0.017423482611775398,
+          -9.363912977278233e-05,
+          0.009686080738902092,
+          0.003426307812333107,
+          0.016019288450479507,
+          0.00607529329136014,
+          0.013741052709519863,
+          0.014256879687309265,
+          -0.0008301587658934295,
+          0.020160231739282608,
+          -0.0009206075337715447,
+          -0.002340922364965081,
+          0.0024251025170087814,
+          0.006770226638764143,
+          0.0007200082764029503,
+          0.004807219374924898,
+          -0.010015636682510376,
+          -0.010739227756857872,
+          -0.0019021113403141499,
+          0.0014695690479129553,
+          0.00425198907032609,
+          0.016076602041721344,
+          0.0008444872801192105,
+          -0.01383418869227171,
+          0.014077773317694664,
+          0.04267033934593201,
+          0.0034191436134278774,
+          0.021062927320599556,
+          0.0010970274452120066,
+          -0.0005614989786408842,
+          0.009077118709683418,
+          0.00858994945883751,
+          -0.026636723428964615,
+          0.021621739491820335,
+          -0.006809630431234837,
+          0.00551648112013936,
+          -0.030175868421792984,
+          0.006286639254540205,
+          -0.017208555713295937,
+          0.002500327071174979,
+          -0.01166341733187437,
+          0.006394103169441223,
+          0.012315364554524422,
+          -0.006616195198148489,
+          0.017523782327771187,
+          -0.0008525470620952547,
+          0.016047945246100426,
+          0.0012421037536114454,
+          -0.0021188303362578154,
+          -0.00931353960186243,
+          0.012480142526328564,
+          0.018913649022579193,
+          0.018440809100866318,
+          0.0005261254264041781,
+          0.01222222950309515,
+          -0.005262149963527918,
+          -0.010015636682510376,
+          0.009284882806241512,
+          -0.004047807771712542,
+          -0.0005579168209806085,
+          0.0037934763822704554,
+          -0.011197740212082863,
+          -0.0033564562909305096,
+          0.01676437072455883,
+          0.019515447318553925,
+          0.0028657044749706984,
+          -0.011577446013689041,
+          -0.02037515863776207,
+          0.0006282161339186132,
+          0.014350014738738537,
+          -0.003111080499365926,
+          0.013683739118278027,
+          -0.0036716838367283344,
+          -0.0029695862904191017,
+          -0.008647263050079346,
+          0.0004466469108592719,
+          0.020074259489774704,
+          0.0051546860486269,
+          -0.014034787192940712,
+          -0.004534977488219738,
+          0.021936967968940735,
+          0.010610271245241165,
+          0.008145765401422977,
+          0.006576791871339083,
+          -0.007529638707637787,
+          0.01659242808818817,
+          -0.011169083416461945,
+          -0.00037948196404613554,
+          -0.012666413560509682,
+          -0.003236454911530018,
+          -0.005856783594936132,
+          -0.008496813476085663,
+          -0.007350531872361898,
+          -0.008568456396460533,
+          0.005487823858857155,
+          0.011161918751895428,
+          -0.01596197299659252,
+          -0.012236557900905609,
+          0.012250886298716068,
+          -0.002503909170627594,
+          -0.0004728412313852459,
+          -0.01620555855333805,
+          0.014156579971313477,
+          0.008059794083237648,
+          0.0033851133193820715,
+          -0.009779216721653938,
+          0.01558943185955286,
+          -0.005817380268126726,
+          0.0002970929490402341,
+          -0.0060323080979287624,
+          -0.0004099300713278353,
+          0.0014982260763645172,
+          -0.004090792965143919,
+          -0.005548720248043537,
+          -0.0018967381911352277,
+          0.007701580878347158,
+          -0.01952977664768696,
+          0.009686080738902092,
+          -0.010832362808287144,
+          -0.026565080508589745,
+          -0.013827024027705193,
+          0.022223537787795067,
+          0.025805668905377388,
+          0.027410462498664856,
+          0.007701580878347158,
+          0.002815554616972804,
+          0.00562036270275712,
+          0.0034155615139752626,
+          -0.0025845072232186794,
+          0.005376778077334166,
+          0.009026968851685524,
+          -0.0038078047800809145,
+          0.03057706728577614,
+          0.005641855765134096,
+          -0.004545723553746939,
+          0.014328522607684135,
+          0.018369166180491447,
+          0.01831185072660446,
+          -0.007687252014875412,
+          0.020174559205770493,
+          -0.016148244962096214,
+          -0.0008395618642680347,
+          -0.002708090702071786,
+          0.03507622331380844,
+          -0.029373470693826675,
+          -0.0012644920498132706,
+          -0.0027510763611644506,
+          0.02041814476251602,
+          0.006028725765645504,
+          -0.02304026484489441,
+          0.010531463660299778,
+          0.028112560510635376,
+          0.0025791339576244354,
+          0.012895669788122177,
+          -0.02051844447851181,
+          -0.009700410068035126,
+          0.00017686770297586918,
+          0.005308717489242554,
+          0.0029964521527290344,
+          0.005505734588950872,
+          -0.022323837503790855,
+          0.01623421534895897,
+          -0.01457210723310709,
+          0.019873660057783127,
+          0.0037218336947262287,
+          0.026292838156223297,
+          0.005366031546145678,
+          0.006064547225832939,
+          0.006960079539567232,
+          -0.013877173885703087,
+          0.014987634494900703,
+          0.0007853821152821183,
+          -0.013862845487892628,
+          0.002903316868469119,
+          -0.016721386462450027,
+          0.01167774572968483,
+          -0.0168360136449337,
+          0.015947645530104637,
+          -0.004359452985227108,
+          0.007565459702163935,
+          -0.0036089965142309666,
+          -0.0009138910681940615,
+          -0.013755381107330322,
+          0.012666413560509682,
+          -0.030032582581043243,
+          0.010287879034876823,
+          -0.010954155586659908,
+          -0.031092893332242966,
+          0.0037504907231777906,
+          0.002140322932973504,
+          0.01131236832588911,
+          -0.013633589260280132,
+          -0.017982294782996178,
+          0.01630585826933384,
+          -0.0019880824256688356,
+          0.004481245297938585,
+          -0.020819343626499176,
+          -0.007085454184561968,
+          0.003800640581175685,
+          0.024945957586169243,
+          -0.0017928563756868243,
+          -0.014536285772919655,
+          0.007325456943362951,
+          0.014371507801115513,
+          -0.009614438749849796,
+          0.020461129024624825,
+          0.005634691566228867,
+          -0.007701580878347158,
+          0.007253814488649368,
+          -0.0025468948297202587,
+          0.022954292595386505,
+          -0.00470691965892911,
+          -0.008496813476085663,
+          0.013905830681324005,
+          0.008375021629035473,
+          -0.02328384853899479,
+          0.005186925176531076,
+          -0.008726069703698158,
+          -0.0013871800620108843,
+          -0.0007809044909663498,
+          -0.001941514783538878,
+          0.010037129744887352,
+          0.028485102578997612,
+          -0.00834636390209198,
+          0.008948162198066711,
+          0.003689594566822052,
+          -0.00317376758903265,
+          5.6950279940792825e-06,
+          0.0006322460249066353,
+          -0.019830675795674324,
+          -0.00903413351625204,
+          -0.004749905318021774,
+          -0.010395343415439129,
+          0.0007531429291702807,
+          0.016463471576571465,
+          0.027582405135035515,
+          -0.0073720249347388744,
+          0.007167843170464039,
+          -0.0007070230203680694,
+          0.020461129024624825,
+          0.031752005219459534,
+          -0.014328522607684135,
+          0.004588709212839603,
+          -0.009370853193104267,
+          -0.012867012992501259,
+          -0.020561428740620613,
+          0.004259153269231319,
+          -0.0014668825315311551,
+          -0.02304026484489441,
+          -0.0009725484414957464,
+          0.00046657249913550913,
+          -0.005176178645342588,
+          -0.009549960494041443,
+          0.00488244416192174,
+          -0.011885508894920349,
+          0.0045278132893145084,
+          -0.002625701716169715,
+          0.004889608360826969,
+          0.010911169461905956,
+          0.034875623881816864,
+          -0.005946336779743433,
+          -0.002511073602363467,
+          -0.011498639360070229,
+          0.0033224262297153473,
+          0.020160231739282608,
+          0.000652843271382153,
+          0.008195914328098297,
+          -0.0066197775304317474,
+          0.00834636390209198,
+          -0.006841869559139013,
+          -0.012401335872709751,
+          0.01520256232470274,
+          0.008948162198066711,
+          0.003125408897176385,
+          -0.01076788455247879,
+          -0.0501498281955719,
+          -0.0015143457567319274,
+          0.032525744289159775,
+          0.010531463660299778,
+          -0.005720662418752909,
+          -0.012652085162699223,
+          -0.018541108816862106,
+          -0.008754727430641651,
+          -0.0020704714115709066,
+          0.0013638961827382445,
+          -0.023455791175365448,
+          0.0024645058438181877,
+          -0.005305135622620583,
+          -0.003789894049987197,
+          -0.0037289978936314583,
+          0.011441324837505817,
+          0.04390259087085724,
+          -0.015431818552315235,
+          0.00813860073685646,
+          -0.009657423943281174,
+          -0.01527420524507761,
+          -0.0006761271506547928,
+          0.018598422408103943,
+          -0.010216236114501953,
+          -0.012494471855461597,
+          -0.016964970156550407,
+          -0.010674749501049519,
+          -0.006472909823060036,
+          0.009979816153645515,
+          0.009628767147660255,
+          -0.011649088934063911,
+          -0.007751730736345053,
+          -0.004491991829127073,
+          -0.007053215056657791,
+          0.013511796481907368,
+          0.006193503737449646,
+          0.021091584116220474,
+          -0.0161625724285841,
+          -0.01185685209929943,
+          0.012021630071103573,
+          0.008754727430641651,
+          0.0030698860064148903,
+          -0.015360175631940365,
+          -0.005541556049138308,
+          -0.007465159986168146,
+          -0.0023122653365135193,
+          0.025662383064627647,
+          -0.004864533431828022,
+          0.016463471576571465,
+          -0.0003806013846769929,
+          -0.005584541708230972,
+          0.004574380815029144,
+          0.005545137915760279,
+          0.025232527405023575,
+          0.011806702241301537,
+          -0.0026042088866233826,
+          -0.01338283997029066,
+          0.0023283849004656076,
+          -0.010624599643051624,
+          -0.0022191298194229603,
+          -0.009356524795293808,
+          0.006086039822548628,
+          0.01574704609811306,
+          0.0034334722440689802,
+          0.021979954093694687,
+          -0.00030358557705767453,
+          -0.020848000422120094,
+          -0.004330795723944902,
+          0.00831054337322712,
+          0.0021779353264719248,
+          0.017437811940908432,
+          0.009707573801279068,
+          0.013432989828288555,
+          0.0058675301261246204,
+          0.012472978793084621,
+          -0.011298039928078651,
+          -0.004316467326134443,
+          -0.006086039822548628,
+          -0.002656149910762906,
+          0.014163744635879993,
+          6.481417221948504e-05,
+          0.03819984197616577,
+          -0.015073605813086033,
+          -0.019687389954924583,
+          0.002557641128078103,
+          -0.011520132422447205,
+          -0.012422828935086727,
+          0.00462811253964901,
+          -0.012809699401259422,
+          -0.0017874831100925803,
+          0.013167912140488625,
+          -0.020676057785749435,
+          -0.022452794015407562,
+          -0.013175075873732567,
+          0.011262218467891216,
+          -0.005183342844247818,
+          -0.005577377509325743,
+          -0.00020630833751056343,
+          0.010946990922093391,
+          0.0024698791094124317,
+          -0.007303964346647263,
+          0.018397822976112366,
+          -0.007680087815970182,
+          -0.007809044793248177,
+          0.013705231249332428,
+          -0.003943925723433495,
+          0.007114111445844173,
+          0.002369579393416643,
+          0.015933316200971603,
+          0.022925635799765587,
+          -0.004782144445925951,
+          0.0034746667370200157,
+          0.007880687713623047,
+          0.014701063744723797,
+          0.0020597251132130623,
+          0.008453828282654285,
+          -0.023785347118973732,
+          0.01360493153333664,
+          0.007164261303842068,
+          -0.007930837571620941,
+          0.013755381107330322,
+          0.01518823392689228,
+          0.0019182309042662382,
+          0.005459167063236237,
+          -0.005050804000347853,
+          0.01964440383017063,
+          -0.0009895635303109884,
+          0.002396445255726576,
+          -0.01967306062579155,
+          0.012738056480884552,
+          -0.004391692113131285,
+          0.011190575547516346,
+          0.009886680170893669,
+          0.01627720147371292,
+          0.029086900874972343,
+          -0.0010692659998312593,
+          0.00294630229473114,
+          -0.008955326862633228,
+          0.011498639360070229,
+          0.0038615367375314236,
+          -0.009901008568704128,
+          -0.0024286846164613962,
+          0.00641917809844017,
+          0.01792498119175434,
+          0.022753693163394928,
+          0.019314847886562347,
+          0.004255570936948061,
+          -0.0029588397592306137,
+          0.026923293247818947,
+          0.0013603140832856297,
+          -0.00029351082048378885,
+          -0.0027546584606170654,
+          -0.009177418425679207,
+          -0.002872868673875928,
+          -0.014744049869477749,
+          0.011985808610916138,
+          -0.0020848000422120094,
+          0.019730376079678535,
+          -0.000688216823618859,
+          0.02342713437974453,
+          0.010244893841445446,
+          -0.015431818552315235,
+          -0.0068669444881379604,
+          0.0017149450723081827,
+          0.002801226219162345,
+          -0.0171512421220541,
+          -0.0021725622937083244,
+          -0.003159439191222191,
+          0.0017015120247378945,
+          0.010825199075043201,
+          8.04300289019011e-05,
+          9.75011134869419e-05,
+          0.001118520274758339,
+          -0.0015600178157910705,
+          -0.002203010255470872,
+          0.0011821030639111996,
+          -0.0007885164814069867,
+          -0.005598870106041431,
+          0.008790548890829086,
+          0.013139255344867706,
+          0.01436434406787157,
+          0.015059277415275574,
+          0.02391430363059044,
+          0.01644914411008358,
+          0.013905830681324005,
+          -0.0027600317262113094,
+          -0.0025074915029108524,
+          0.024774014949798584,
+          0.003986911382526159,
+          -0.004710501525551081,
+          0.004581545013934374,
+          0.0026668962091207504,
+          0.0019343504682183266,
+          -0.006086039822548628,
+          0.0055308095179498196,
+          -0.016090931370854378,
+          5.056960799265653e-05,
+          -0.0036143697798252106,
+          -0.002208383521065116,
+          -0.010617434978485107,
+          0.0009152343845926225,
+          -0.017208555713295937,
+          0.009127268567681313,
+          0.01571838930249214,
+          0.003442427609115839,
+          0.01058877818286419,
+          0.007708745077252388,
+          -0.030949607491493225,
+          -0.009299211204051971,
+          0.011362518183887005,
+          0.003689594566822052,
+          0.011226397007703781,
+          0.012358350679278374,
+          0.008181585930287838,
+          -0.017910653725266457,
+          0.008246064186096191,
+          -0.002319429535418749,
+          0.011806702241301537,
+          0.010424000211060047,
+          -0.008948162198066711
+        ],
+        "title": "Designing Models with Data, Model, and Expert-Parallelism"
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "9204ecdb-832f-4b40-b0fe-5cc34ec75d3b",
+      "properties": {
+        "page_content": "6 Related Work 24 ",
+        "keyphrases": [
+          "Related Work"
+        ],
+        "embedding": [
+          0.002109470311552286,
+          0.024481380358338356,
+          -0.01675625890493393,
+          0.01014935877174139,
+          0.03144679218530655,
+          0.0017690955428406596,
+          0.0074690403416752815,
+          0.09587687253952026,
+          0.01939389668405056,
+          0.014093012548983097,
+          0.001160902320407331,
+          -0.013068686239421368,
+          -0.015330738388001919,
+          -0.026103228330612183,
+          0.03235161304473877,
+          0.036431845277547836,
+          -0.005620986223220825,
+          -0.00097150873625651,
+          -0.033068642020225525,
+          0.002234310144558549,
+          0.0004380058671813458,
+          0.024430163204669952,
+          -0.015834365040063858,
+          0.04742627218365669,
+          0.02104988880455494,
+          0.008480561897158623,
+          0.051182132214307785,
+          0.01811349019408226,
+          -0.03865121677517891,
+          0.0019728937186300755,
+          -0.008903096430003643,
+          0.010678594000637531,
+          8.36265753605403e-05,
+          -0.0034208204597234726,
+          -0.028698187321424484,
+          -0.018420787528157234,
+          0.014630783349275589,
+          0.01811349019408226,
+          -0.0005980567075312138,
+          0.012718708254396915,
+          -0.036909863352775574,
+          -0.027776293456554413,
+          -0.030541973188519478,
+          0.0029363997746258974,
+          0.03361494839191437,
+          -0.029739584773778915,
+          -0.01758425496518612,
+          -0.044626448303461075,
+          0.005258204415440559,
+          0.03020053170621395,
+          0.029500575736165047,
+          -0.023098541423678398,
+          0.01559535600244999,
+          -0.0012782729463651776,
+          -0.01341866422444582,
+          0.003841220634058118,
+          -0.0015738335205242038,
+          0.07300026714801788,
+          0.023798497393727303,
+          0.019069526344537735,
+          0.001734951394610107,
+          -0.016662361100316048,
+          -0.0010467326501384377,
+          -0.03286377713084221,
+          0.009978638030588627,
+          -0.01825006678700447,
+          0.018045201897621155,
+          -0.005023463163524866,
+          0.020264573395252228,
+          -0.008489097468554974,
+          -0.0041101062670350075,
+          -0.007831822149455547,
+          -0.026905616745352745,
+          0.05551844462752342,
+          -0.0061459532007575035,
+          0.00418479647487402,
+          0.042850952595472336,
+          0.027008049190044403,
+          -0.015689251944422722,
+          0.04995294287800789,
+          -0.005655130371451378,
+          -0.024703318253159523,
+          -0.012863821350038052,
+          -0.01522830594331026,
+          0.019701194018125534,
+          -0.005778903141617775,
+          -0.056850068271160126,
+          -0.13206970691680908,
+          0.004724701400846243,
+          -0.0011918453965336084,
+          0.011318797245621681,
+          0.014229589141905308,
+          -0.002349546644836664,
+          -0.03841220587491989,
+          -0.051182132214307785,
+          0.04595807194709778,
+          -0.008531778119504452,
+          -0.021391332149505615,
+          0.009551835246384144,
+          -0.01522830594331026,
+          0.008096439763903618,
+          0.016414817422628403,
+          0.0179598405957222,
+          -0.03494657203555107,
+          0.02656417526304722,
+          -0.03824148699641228,
+          -0.005292348563671112,
+          -0.00846775807440281,
+          0.026291022077202797,
+          0.03970968723297119,
+          -0.02081087976694107,
+          -0.025249624624848366,
+          -0.033444229513406754,
+          0.004285095259547234,
+          -0.0017040082020685077,
+          -0.029995666816830635,
+          0.011079788208007812,
+          -0.046162936836481094,
+          -0.021203538402915,
+          0.012360194697976112,
+          0.020486511290073395,
+          -0.0011032839538529515,
+          0.004673485178500414,
+          -0.02465210109949112,
+          -0.02274002693593502,
+          0.021664485335350037,
+          0.003265037667006254,
+          -0.01618434302508831,
+          -0.01535634696483612,
+          0.0282543133944273,
+          0.03605625778436661,
+          -0.00591121194884181,
+          0.0010622042464092374,
+          -0.03387103229761124,
+          0.01354670524597168,
+          -0.018574437126517296,
+          0.011933392845094204,
+          0.03660256415605545,
+          -0.022022999823093414,
+          -0.004733237437903881,
+          0.005288080777972937,
+          -0.019359752535820007,
+          -0.0030473684892058372,
+          0.022415656596422195,
+          -0.007490380667150021,
+          0.014912472106516361,
+          -0.03055904433131218,
+          0.012991862371563911,
+          -0.017789119854569435,
+          0.04548005387187004,
+          -0.03424661606550217,
+          0.017319638282060623,
+          -0.018284210935235023,
+          0.032027244567871094,
+          -0.019735338166356087,
+          -0.0157319325953722,
+          0.009287218563258648,
+          0.0017146782483905554,
+          0.016355063766241074,
+          -0.02403750643134117,
+          0.04097302258014679,
+          0.023320477455854416,
+          0.024447236210107803,
+          -0.01644042506814003,
+          0.021630341187119484,
+          -0.03378567099571228,
+          0.015211233869194984,
+          -0.014681999571621418,
+          -0.017908625304698944,
+          0.026512959972023964,
+          0.02734949253499508,
+          0.0033183880150318146,
+          0.018420787528157234,
+          0.0003475771227385849,
+          -0.026683680713176727,
+          -0.007976935245096684,
+          -0.007575741037726402,
+          0.03168580308556557,
+          -0.013956435024738312,
+          0.003331192070618272,
+          -0.021288899704813957,
+          0.03762689232826233,
+          -0.0013903084909543395,
+          -0.00194301747251302,
+          0.011071251705288887,
+          -0.013393056578934193,
+          0.029381070286035538,
+          -0.030132243409752846,
+          0.041314464062452316,
+          0.024378947913646698,
+          -0.0024221031926572323,
+          -0.019052455201745033,
+          0.01622702367603779,
+          0.012752852402627468,
+          0.005198452156037092,
+          -0.057737816125154495,
+          0.011199292726814747,
+          -0.006252653896808624,
+          -0.03568067401647568,
+          -0.021220611408352852,
+          -0.00606912886723876,
+          -0.022057143971323967,
+          0.007443432230502367,
+          0.004575320519506931,
+          -0.025847148150205612,
+          -0.009201857261359692,
+          -0.0024583812337368727,
+          0.002289794385433197,
+          -0.008659819141030312,
+          -0.03516850993037224,
+          0.00754159688949585,
+          0.0003875898546539247,
+          0.0006898192223161459,
+          -0.016406280919909477,
+          -0.0011064850259572268,
+          -0.023559486493468285,
+          -0.0224839448928833,
+          0.06524953991174698,
+          -0.0035787373781204224,
+          -0.0295347198843956,
+          0.03892436996102333,
+          -0.0030708424746990204,
+          0.007925719022750854,
+          0.008809199556708336,
+          -0.02647881582379341,
+          -0.023439982905983925,
+          -0.0022663201671093702,
+          0.05275276303291321,
+          0.0034955109003931284,
+          -0.01458810269832611,
+          0.0006594095611944795,
+          0.008292769081890583,
+          -0.004041817970573902,
+          -0.052616186439991,
+          -0.0314980112016201,
+          -0.01964997872710228,
+          -0.00800254289060831,
+          -0.030354179441928864,
+          -0.012300441972911358,
+          0.02801530249416828,
+          0.030763909220695496,
+          -0.0016613280167803168,
+          -0.042953383177518845,
+          0.017123308032751083,
+          -0.02466917410492897,
+          -0.020349934697151184,
+          -0.03163458779454231,
+          0.05312835052609444,
+          0.019871914759278297,
+          -0.02620566077530384,
+          0.0058045112527906895,
+          0.012616275809705257,
+          -0.009048208594322205,
+          0.02079380862414837,
+          0.019342679530382156,
+          0.09280389547348022,
+          0.026666607707738876,
+          -0.0018149768002331257,
+          -0.0026653804816305637,
+          -0.015211233869194984,
+          -0.015194161795079708,
+          -0.01675625890493393,
+          -0.04561663046479225,
+          -0.021783988922834396,
+          0.005377708934247494,
+          -0.0013038810575380921,
+          0.039368245750665665,
+          0.0044302078895270824,
+          0.022415656596422195,
+          -0.01001278217881918,
+          0.024686245247721672,
+          -0.013145511038601398,
+          -0.002503195544704795,
+          -0.02622273378074169,
+          -0.009603051468729973,
+          -0.030115170404314995,
+          -0.003077244618907571,
+          -0.015142945572733879,
+          0.016372136771678925,
+          -0.004588124807924032,
+          -0.009893277660012245,
+          0.04466059431433678,
+          0.008937240578234196,
+          -0.018984166905283928,
+          -0.021527908742427826,
+          -0.03557823970913887,
+          0.009449402801692486,
+          -0.0026995246298611164,
+          -0.0031199248041957617,
+          -0.019564617425203323,
+          0.01213825773447752,
+          -0.027912870049476624,
+          -0.040529146790504456,
+          0.044899601489305496,
+          0.006828837096691132,
+          -0.025198407471179962,
+          -0.01381132286041975,
+          0.006905661430209875,
+          -0.0014351228019222617,
+          -0.026410527527332306,
+          -0.001253731781616807,
+          0.012838213704526424,
+          -0.00121211854275316,
+          0.007302587386220694,
+          -0.019718267023563385,
+          0.008331180550158024,
+          0.02106696180999279,
+          0.03177116438746452,
+          -0.029978593811392784,
+          -0.026871472597122192,
+          0.01149805448949337,
+          -0.04868960753083229,
+          -0.0007074248278513551,
+          -0.04063158109784126,
+          0.012069969438016415,
+          0.024430163204669952,
+          0.005902675911784172,
+          0.021732773631811142,
+          -0.02874940261244774,
+          -0.04612879455089569,
+          0.0013113500317558646,
+          0.01078102644532919,
+          -0.04353383556008339,
+          0.024481380358338356,
+          -0.014724679291248322,
+          0.022296153008937836,
+          -0.003619283437728882,
+          0.04059743508696556,
+          0.01580875739455223,
+          -0.010763954371213913,
+          0.03006395511329174,
+          -0.00937257893383503,
+          -6.443714482884388e-06,
+          0.018967093899846077,
+          -0.011941928416490555,
+          -0.03156629949808121,
+          -0.03557823970913887,
+          0.015475851483643055,
+          0.007567205000668764,
+          -0.017823264002799988,
+          0.009662804193794727,
+          0.01554413978010416,
+          0.007511720526963472,
+          0.02451552450656891,
+          0.019086599349975586,
+          -0.05367465689778328,
+          0.035373374819755554,
+          -0.04029013589024544,
+          -0.012923574075102806,
+          0.007780605927109718,
+          0.0009704417316243052,
+          -0.03554409742355347,
+          -0.0024370411410927773,
+          0.012104113586246967,
+          -0.014238124713301659,
+          -0.047221407294273376,
+          -0.017703760415315628,
+          -0.029244493693113327,
+          0.020076779648661613,
+          -0.021801061928272247,
+          0.00022700548288412392,
+          -0.04022184759378433,
+          -0.009125033393502235,
+          -0.016022158786654472,
+          -0.05278690904378891,
+          -0.036670852452516556,
+          0.024464307352900505,
+          -0.005795975215733051,
+          -0.02145962044596672,
+          -0.0009016198455356061,
+          -0.059410881251096725,
+          0.007989739067852497,
+          0.0016175807686522603,
+          -0.006777620408684015,
+          -0.025095975026488304,
+          -0.011148076504468918,
+          0.014571030624210835,
+          0.017635472118854523,
+          -0.0397779755294323,
+          0.005680738482624292,
+          -0.01373449806123972,
+          0.01014935877174139,
+          0.005706346593797207,
+          -0.017635472118854523,
+          -0.024088721722364426,
+          0.0036897058598697186,
+          -0.0037665304262191057,
+          0.01168584730476141,
+          0.0030644405633211136,
+          -0.010678594000637531,
+          0.02480575069785118,
+          -0.04520690068602562,
+          0.003469902789220214,
+          -0.03632941097021103,
+          0.039880406111478806,
+          -0.006495931185781956,
+          0.02992737665772438,
+          0.015100265853106976,
+          -0.0030793785117566586,
+          -0.005181380081921816,
+          -0.0501236617565155,
+          0.005595378112047911,
+          0.005138699896633625,
+          0.006700796075165272,
+          0.04435329511761665,
+          -0.003719581989571452,
+          -0.017925696447491646,
+          0.01232605054974556,
+          0.016628216952085495,
+          -0.011643166653811932,
+          0.03168580308556557,
+          0.015023441053926945,
+          -0.000559644540771842,
+          -0.011216364800930023,
+          0.03574896231293678,
+          -0.002466917270794511,
+          0.009150641039013863,
+          0.061220522969961166,
+          -0.023542415350675583,
+          -0.010311543941497803,
+          -0.004592392593622208,
+          -0.008732374757528305,
+          0.07368315011262894,
+          -0.009611587971448898,
+          -0.008472025394439697,
+          0.0179598405957222,
+          -0.015663644298911095,
+          -0.011916320770978928,
+          0.01600508578121662,
+          -0.02569349855184555,
+          -0.02169862948358059,
+          0.012505307793617249,
+          0.026273949071764946,
+          -0.021101105958223343,
+          0.028032375499606133,
+          -0.015689251944422722,
+          -0.011139540001749992,
+          0.01201021671295166,
+          -0.016329456120729446,
+          -0.012300441972911358,
+          0.035407520830631256,
+          -0.004592392593622208,
+          0.03366616740822792,
+          0.03660256415605545,
+          0.006192901637405157,
+          -0.03131021559238434,
+          -0.04548005387187004,
+          -0.010670057497918606,
+          -0.03480999544262886,
+          0.017413534224033356,
+          -0.0003048969083465636,
+          0.013691818341612816,
+          -0.04469473659992218,
+          -0.00844641774892807,
+          0.05510871484875679,
+          -0.02209128811955452,
+          0.029090845957398415,
+          0.04343140125274658,
+          -0.03185652568936348,
+          -0.02412286587059498,
+          -0.021510835736989975,
+          0.08549703657627106,
+          0.040665723383426666,
+          -0.010055461898446083,
+          -0.0359879694879055,
+          0.01592826284468174,
+          -0.026666607707738876,
+          -0.025078903883695602,
+          2.39409419009462e-05,
+          0.04339725896716118,
+          0.00754159688949585,
+          -0.015527067705988884,
+          -0.017908625304698944,
+          0.01937682554125786,
+          -0.033853959292173386,
+          0.001105417963117361,
+          0.019991420209407806,
+          0.008087903261184692,
+          -0.04510446637868881,
+          0.012069969438016415,
+          0.017891552299261093,
+          0.02031579054892063,
+          -0.02504475973546505,
+          0.015322202816605568,
+          0.026512959972023964,
+          0.027451924979686737,
+          0.027947014197707176,
+          -0.005676470696926117,
+          0.015339274890720844,
+          -0.006965413689613342,
+          0.03865121677517891,
+          -0.0018619250040501356,
+          -0.007840358652174473,
+          0.01695258729159832,
+          -0.029483502730727196,
+          -0.04005112871527672,
+          -0.017080627381801605,
+          -0.07129305601119995,
+          0.015834365040063858,
+          0.027178769931197166,
+          -0.014272268861532211,
+          -0.00818180013448,
+          -0.008058027364313602,
+          0.0450020357966423,
+          0.010251791216433048,
+          -0.021869350224733353,
+          0.01799398474395275,
+          -0.018950022757053375,
+          -0.01213825773447752,
+          0.013008934445679188,
+          0.028220169246196747,
+          0.016090447083115578,
+          0.019479257985949516,
+          -0.0237472802400589,
+          0.003971395548433065,
+          0.006991021800786257,
+          0.012658956460654736,
+          -0.0007340999436564744,
+          -0.01065298542380333,
+          0.030115170404314995,
+          0.0077379257418215275,
+          0.0027976890560239553,
+          0.001323087140917778,
+          0.04855303093791008,
+          -0.03187359496951103,
+          0.008971384726464748,
+          -0.010584697127342224,
+          -0.02236444130539894,
+          0.05346979200839996,
+          -0.003363202093169093,
+          -0.012744316831231117,
+          0.031259000301361084,
+          0.027588501572608948,
+          -0.013316231779754162,
+          -0.0118821756914258,
+          -0.0011043510166928172,
+          -0.027008049190044403,
+          0.020725520327687263,
+          -0.02067430317401886,
+          -0.0068587129935622215,
+          0.01007253397256136,
+          0.016824547201395035,
+          0.011660238727927208,
+          -0.017754975706338882,
+          -0.011848031543195248,
+          0.05125042051076889,
+          0.041280318051576614,
+          -0.010883458890020847,
+          0.007140402682125568,
+          -0.0028446372598409653,
+          0.014161300845444202,
+          0.028442105278372765,
+          0.006534343119710684,
+          -0.003913776949048042,
+          -0.002244980074465275,
+          -0.010951747186481953,
+          -0.03236868605017662,
+          0.019957276061177254,
+          0.025864219292998314,
+          0.012223618105053902,
+          0.028100663796067238,
+          -0.03779761120676994,
+          -0.00943233072757721,
+          -0.0053819771856069565,
+          0.006555683445185423,
+          -0.04670924320816994,
+          0.014101548120379448,
+          0.021544979885220528,
+          -0.03520265221595764,
+          -0.021408403292298317,
+          -0.0025330716744065285,
+          0.011259044520556927,
+          -0.03096877597272396,
+          -0.035509951412677765,
+          -0.032163821160793304,
+          0.04742627218365669,
+          -0.0021820268593728542,
+          -0.0063508180901408195,
+          -0.04059743508696556,
+          -0.01738792657852173,
+          0.015646573156118393,
+          -0.02429358661174774,
+          -0.006760548334568739,
+          0.025744715705513954,
+          -0.006133148912340403,
+          0.006167293526232243,
+          -0.024447236210107803,
+          -0.015262450091540813,
+          -0.03622698038816452,
+          0.02878354676067829,
+          0.012846749275922775,
+          0.0025800198782235384,
+          -0.01966704986989498,
+          -0.024327730759978294,
+          -0.025539850816130638,
+          0.004728969652205706,
+          0.010891994461417198,
+          0.0009005528409034014,
+          0.0289030522108078,
+          -0.05428925156593323,
+          0.030729765072464943,
+          0.006235581822693348,
+          -0.026239804923534393,
+          0.006389230489730835,
+          0.010644449852406979,
+          -0.023030253127217293,
+          0.003160471096634865,
+          0.01677333004772663,
+          -0.009346970357000828,
+          0.011446838267147541,
+          0.02555692195892334,
+          -0.014093012548983097,
+          0.048484742641448975,
+          -0.004524104297161102,
+          0.009918885305523872,
+          -0.025915436446666718,
+          0.041690047830343246,
+          0.008587262593209743,
+          -0.0009405656019225717,
+          0.0015044781612232327,
+          0.03892436996102333,
+          -0.01886466145515442,
+          -0.0224839448928833,
+          -0.0112334368750453,
+          0.013828394934535027,
+          -0.033973462879657745,
+          -0.008886024355888367,
+          -0.007848894223570824,
+          -0.017439141869544983,
+          -0.011933392845094204,
+          0.005783171392977238,
+          0.010746882297098637,
+          0.005492945667356253,
+          0.010635913349688053,
+          0.018011057749390602,
+          0.008143387734889984,
+          0.038104910403490067,
+          -0.009346970357000828,
+          0.03687571734189987,
+          -0.012360194697976112,
+          -0.01600508578121662,
+          -0.016858691349625587,
+          0.019223175942897797,
+          0.041177887469530106,
+          -0.040255993604660034,
+          0.029568864032626152,
+          -0.014442989602684975,
+          0.0147673599421978,
+          0.005253936164081097,
+          0.01838664337992668,
+          0.011165148578584194,
+          -0.003881766926497221,
+          -0.0050405352376401424,
+          0.022893676534295082,
+          0.011020035482943058,
+          -0.00015231507131829858,
+          -0.0012836079113185406,
+          -0.02425944246351719,
+          -0.0474945604801178,
+          0.020127996802330017,
+          -0.012940646149218082,
+          -0.022808315232396126,
+          -0.005467337556183338,
+          0.010490800254046917,
+          0.005224060267210007,
+          0.029722511768341064,
+          0.015211233869194984,
+          0.005224060267210007,
+          -0.014442989602684975,
+          0.006577023304998875,
+          -0.004024745896458626,
+          -0.035885538905858994,
+          0.0050917514599859715,
+          0.020503582432866096,
+          -0.03131021559238434,
+          -0.018403714522719383,
+          -0.004159188363701105,
+          0.03337594121694565,
+          -0.0031113887671381235,
+          0.004270156845450401,
+          -0.016568465158343315,
+          -0.004618000704795122,
+          -0.021118177101016045,
+          0.017217205837368965,
+          0.002994018141180277,
+          0.01503197755664587,
+          -0.010542016476392746,
+          0.0003016958653461188,
+          0.006623971741646528,
+          -0.00018725951667875051,
+          -0.0041762604378163815,
+          0.005744758993387222,
+          -0.006700796075165272,
+          -0.0016666629817336798,
+          0.0013561643427237868,
+          0.010695666074752808,
+          -0.014272268861532211,
+          0.00950915552675724,
+          0.007405019830912352,
+          0.013512561097741127,
+          -0.02799823135137558,
+          -0.025761786848306656,
+          -0.055689163506031036,
+          0.01675625890493393,
+          -0.009355506859719753,
+          -0.011284653097391129,
+          -0.006585559342056513,
+          -0.006081932689994574,
+          -0.025130119174718857,
+          -0.014391773380339146,
+          -0.014622246846556664,
+          -0.011122467927634716,
+          0.010328616015613079,
+          -0.0007314324611797929,
+          -0.0015524934278801084,
+          0.01744767837226391,
+          0.002784885000437498,
+          -0.022569306194782257,
+          -0.01746474951505661,
+          0.013589384965598583,
+          -0.01876222901046276,
+          -0.0006722136167809367,
+          -0.034588057547807693,
+          -0.04923591390252113,
+          -0.019308535382151604,
+          0.006141684949398041,
+          0.026530031114816666,
+          -0.0002678184537217021,
+          -0.004404599778354168,
+          -0.003990601748228073,
+          0.005714882630854845,
+          -0.01118222065269947,
+          -0.000805055839009583,
+          0.005859995726495981,
+          0.0036342216189950705,
+          0.020264573395252228,
+          0.039368245750665665,
+          0.018523219972848892,
+          0.0006119278259575367,
+          -0.016337992623448372,
+          0.012547987513244152,
+          0.016269704326987267,
+          -0.006116076838225126,
+          0.015919726341962814,
+          0.00982498936355114,
+          -0.004507032223045826,
+          0.046060506254434586,
+          0.029227422550320625,
+          -0.04349968954920769,
+          0.009517691098153591,
+          0.014622246846556664,
+          -0.0250276867300272,
+          0.03404175117611885,
+          -0.017618399113416672,
+          -0.02028164640069008,
+          0.0032500994857400656,
+          0.021101105958223343,
+          -0.0002234043349744752,
+          -0.015783149749040604,
+          0.022022999823093414,
+          -0.01002131775021553,
+          -0.0014329887926578522,
+          0.007601349148899317,
+          0.026530031114816666,
+          -0.0011822424130514264,
+          -0.03356373310089111,
+          -0.002336742589250207,
+          -0.009475011378526688,
+          0.004985050763934851,
+          0.011224900372326374,
+          0.01811349019408226,
+          0.002705926541239023,
+          0.009398186579346657,
+          -0.0231156125664711,
+          0.012573596090078354,
+          -0.02337169460952282,
+          -0.008019614964723587,
+          -0.02400336228311062,
+          -0.011626094579696655,
+          0.01450274232774973,
+          0.006141684949398041,
+          0.01657700166106224,
+          -0.02053772658109665,
+          -0.009483546949923038,
+          -0.014280805364251137,
+          0.024447236210107803,
+          0.012419947423040867,
+          -0.02760557271540165,
+          -0.016585538163781166,
+          -0.012437019497156143,
+          -0.015211233869194984,
+          0.012437019497156143,
+          -0.01966704986989498,
+          0.02053772658109665,
+          0.04817744344472885,
+          -0.03568067401647568,
+          -0.011702919378876686,
+          0.03387103229761124,
+          -0.004997855052351952,
+          -0.013136974535882473,
+          0.02991030551493168,
+          0.01027739979326725,
+          -0.01014935877174139,
+          0.005172844044864178,
+          0.012633347883820534,
+          0.01400765124708414,
+          -0.020759664475917816,
+          -0.020998673513531685,
+          -0.019735338166356087,
+          -0.0031498009338974953,
+          -0.007951326668262482,
+          -0.02965422347187996,
+          0.017498893663287163,
+          0.0166794341057539,
+          -0.004447279963642359,
+          0.01650017686188221,
+          -0.028066519647836685,
+          -0.01457956712692976,
+          -0.01142976526170969,
+          -0.01387107465416193,
+          0.0314980112016201,
+          -0.0068587129935622215,
+          0.05749880522489548,
+          -0.006939805578440428,
+          0.00023194037203211337,
+          0.061083946377038956,
+          0.011404157616198063,
+          -0.0015354212373495102,
+          0.037456169724464417,
+          0.027434851974248886,
+          0.0093213627114892,
+          0.018215922638773918,
+          -0.01773790456354618,
+          0.013538168743252754,
+          -0.02081087976694107,
+          0.00026995246298611164,
+          -0.0068074967712163925,
+          0.016133127734065056,
+          -0.0385146401822567,
+          0.0359538272023201,
+          0.017200132831931114,
+          -0.002206567907705903,
+          0.01142976526170969,
+          -0.023303406313061714,
+          -0.010303007438778877,
+          0.016022158786654472,
+          -0.02197178266942501,
+          0.015424635261297226,
+          -0.004131446126848459,
+          -0.004946638830006123,
+          0.024720389395952225,
+          0.033819813281297684,
+          -0.016978194937109947,
+          -0.009893277660012245,
+          -0.042338788509368896,
+          0.0023068664595484734,
+          -0.018335426226258278,
+          0.02258637733757496,
+          0.010354223661124706,
+          -0.013854002580046654,
+          0.0009037538548000157,
+          -0.009227465838193893,
+          -0.016431888565421104,
+          0.004865546245127916,
+          0.008408005349338055,
+          -0.02492525428533554,
+          -0.039641398936510086,
+          -0.011404157616198063,
+          -0.012932109646499157,
+          -0.013136974535882473,
+          -0.016978194937109947,
+          0.0186427254229784,
+          0.0073836799710989,
+          0.011796815320849419,
+          0.009893277660012245,
+          -0.010610305704176426,
+          -0.019479257985949516,
+          0.011515126563608646,
+          0.005919747985899448,
+          0.0023900929372757673,
+          -0.026291022077202797,
+          0.003915911074727774,
+          -0.029585935175418854,
+          0.003687571967020631,
+          -0.010874922387301922,
+          0.03375152498483658,
+          -0.0027507408522069454,
+          -0.029363999143242836,
+          -0.026154445484280586,
+          -0.02492525428533554,
+          0.016884298995137215,
+          -0.010243255645036697,
+          0.00846775807440281,
+          -0.029961522668600082,
+          0.019103670492768288,
+          -0.008937240578234196,
+          -0.027025122195482254,
+          0.00028835830744355917,
+          0.004193332511931658,
+          -0.010874922387301922,
+          -0.021493764594197273,
+          -0.011327332817018032,
+          -0.011941928416490555,
+          -0.052343033254146576,
+          0.0030366983264684677,
+          -0.009389651007950306,
+          0.008766518905758858,
+          -0.0109176030382514,
+          0.00855311844497919,
+          -0.008681158535182476,
+          -0.004142116289585829,
+          -0.021510835736989975,
+          -0.014485670253634453,
+          -0.022296153008937836,
+          0.033068642020225525,
+          0.017353782430291176,
+          -0.023593630641698837,
+          -0.006713600363582373,
+          -0.008689695037901402,
+          -0.004310703370720148,
+          0.0166965052485466,
+          0.012923574075102806,
+          -0.02593250758945942,
+          0.0029022556263953447,
+          0.023474127054214478,
+          -0.003399480367079377,
+          -0.008548850193619728,
+          0.02903962880373001,
+          0.005518553778529167,
+          0.009142105467617512,
+          0.004101570229977369,
+          0.025232551619410515,
+          -0.005347833037376404,
+          0.004575320519506931,
+          0.054903849959373474,
+          0.0026355041190981865,
+          -0.005667934659868479,
+          0.03786589950323105,
+          0.0004075962060596794,
+          -0.01548438798636198,
+          0.024361874908208847,
+          0.012249225750565529,
+          0.025352057069540024,
+          0.013444272801280022,
+          -0.001758425496518612,
+          0.002910791663452983,
+          -0.00014244526391848922,
+          0.03416125476360321,
+          0.0029684100300073624,
+          0.01600508578121662,
+          -0.03947067633271217,
+          0.02195470966398716,
+          0.015689251944422722,
+          -0.0014649989316239953,
+          0.016944050788879395,
+          -0.004024745896458626,
+          -0.029620079323649406,
+          0.0031391310039907694,
+          0.0011331600835546851,
+          0.02390092983841896,
+          0.024617956951260567,
+          0.008343985304236412,
+          -0.019445113837718964,
+          -0.007823286578059196,
+          -0.002910791663452983,
+          0.05971817672252655,
+          0.011856568045914173,
+          0.03136143460869789,
+          -0.018693940714001656,
+          -0.00846775807440281,
+          -0.02106696180999279,
+          0.0021809597965329885,
+          0.025659354403614998,
+          -0.0029769460670650005,
+          -0.01540756318718195,
+          -0.010798098519444466,
+          -0.001411648583598435,
+          -0.004767381586134434,
+          -0.001800038735382259,
+          0.009142105467617512,
+          0.02079380862414837,
+          -0.013563777320086956,
+          0.01059323363006115,
+          0.028851835057139397,
+          -0.007784874178469181,
+          0.01915488764643669,
+          -0.027264131233096123,
+          -0.02746899612247944,
+          -0.010046926327049732,
+          0.012505307793617249,
+          0.009082352742552757,
+          0.013119902461767197,
+          0.020059708505868912,
+          0.008642747066915035,
+          0.008809199556708336,
+          0.014946616254746914,
+          0.015697788447141647,
+          -0.014084476046264172,
+          0.04698239639401436,
+          -0.00393298314884305,
+          -0.002027310896664858,
+          -0.010584697127342224,
+          -0.01065298542380333,
+          0.012266297824680805,
+          -0.0007298319251276553,
+          0.013478416949510574,
+          0.013085758313536644,
+          -0.02326926216483116,
+          0.01937682554125786,
+          0.00982498936355114,
+          0.0006257988861761987,
+          -0.0007986538112163544,
+          0.0166794341057539,
+          -0.013290624134242535,
+          0.026120301336050034,
+          -0.005949624348431826,
+          -0.01912074349820614,
+          0.010303007438778877,
+          0.01710623688995838,
+          0.0028659775853157043,
+          -0.02579593099653721,
+          -0.029995666816830635,
+          -0.0031049868557602167,
+          -0.03240283206105232,
+          0.02144254744052887,
+          0.007456236053258181,
+          0.014332021586596966,
+          0.014878327958285809,
+          -0.01336744800209999,
+          -0.000287024537101388,
+          0.003220223356038332,
+          0.016056302934885025,
+          0.0052454001270234585,
+          -0.05865970999002457,
+          0.004421671852469444,
+          -0.007823286578059196,
+          -0.005031999200582504,
+          -0.06091322377324104,
+          0.009500619024038315,
+          -0.006598363630473614,
+          0.022825386375188828,
+          -0.006692260038107634,
+          0.03803662210702896,
+          0.00777206989005208,
+          -0.0019302134169265628,
+          0.007819018326699734,
+          0.008220212534070015,
+          0.022944891825318336,
+          -0.007183082867413759,
+          0.00777206989005208,
+          0.0030751104932278395,
+          0.01720866933465004,
+          -0.0033525321632623672,
+          -0.012547987513244152,
+          0.0011875773780047894,
+          0.020981600508093834,
+          -0.006222777534276247,
+          0.009065280668437481,
+          0.03574896231293678,
+          0.002298330422490835,
+          -0.029773728922009468,
+          0.013666209764778614,
+          0.0020614550448954105,
+          -0.017891552299261093,
+          0.01964997872710228,
+          -0.005680738482624292,
+          0.011250508949160576,
+          -0.002041182015091181,
+          0.002332474570721388,
+          0.023525342345237732,
+          0.001920610317029059,
+          0.018540292978286743,
+          -0.0014820710057392716,
+          -0.010755417868494987,
+          0.027315348386764526,
+          -0.008766518905758858,
+          -0.022244935855269432,
+          -0.015381954610347748,
+          -0.03817319869995117,
+          -0.010268863290548325,
+          -0.0056508625857532024,
+          -0.03042246773838997,
+          0.005078947637230158,
+          0.027793366461992264,
+          0.019103670492768288,
+          -0.023730207234621048,
+          -0.023542415350675583,
+          -0.018198849633336067,
+          -0.004167724400758743,
+          0.007144670467823744,
+          -0.03322229161858559,
+          -0.012300441972911358,
+          0.012923574075102806,
+          -0.006056324578821659,
+          0.004511300474405289,
+          -0.009927421808242798,
+          -0.03096877597272396,
+          0.06944926828145981,
+          -0.015620964579284191,
+          -0.013896683230996132,
+          -0.02567642740905285,
+          -0.003478438826277852,
+          -0.011668775230646133,
+          0.023474127054214478,
+          -0.0034250884782522917,
+          0.005403317045420408,
+          0.015774613246321678,
+          0.005155771970748901,
+          0.011207828298211098,
+          -0.00033610680839046836,
+          0.009551835246384144,
+          0.014656390994787216,
+          0.005360636860132217,
+          -0.003478438826277852,
+          0.00591121194884181,
+          0.015262450091540813,
+          -0.027947014197707176,
+          0.008685426786541939,
+          -0.019428040832281113,
+          0.008681158535182476,
+          0.000938965065870434,
+          0.011617559008300304,
+          0.0031988832633942366,
+          -0.01290650200098753,
+          -0.005437461193650961,
+          -0.04073401167988777,
+          -0.009594515897333622,
+          0.023542415350675583,
+          0.017942769452929497,
+          0.03544166311621666,
+          0.0022321760188788176,
+          0.009253074415028095,
+          0.022671738639473915,
+          0.03841220587491989,
+          -0.018198849633336067,
+          0.0015813024947419763,
+          0.010550552979111671,
+          -0.0015770344762131572,
+          0.04916762560606003,
+          0.02362777478992939,
+          0.0012644018279388547,
+          -0.01149805448949337,
+          0.0006620771018788218,
+          0.00023580827109981328,
+          0.006154489237815142,
+          0.006611167918890715,
+          -0.0018437859835103154,
+          -0.002035846933722496,
+          -0.017669616267085075,
+          0.005539894104003906,
+          -0.006542879156768322,
+          -0.019735338166356087,
+          -0.0068587129935622215,
+          0.004835669882595539,
+          0.0289030522108078,
+          0.0002462115662638098,
+          0.004950906615704298,
+          -0.0005164307658560574,
+          0.0007132933242246509,
+          0.026786113157868385,
+          -0.009210393764078617,
+          0.01588558219373226,
+          0.002345278626307845,
+          -0.013597921468317509,
+          0.010174967348575592,
+          -0.008459221571683884,
+          0.0021574855782091618,
+          -0.012974790297448635,
+          0.014878327958285809,
+          0.015569748356938362,
+          -0.010192039422690868,
+          -0.025095975026488304,
+          -0.011762671172618866,
+          0.026769040152430534,
+          -0.015552676282823086,
+          0.011327332817018032,
+          0.003011090215295553,
+          0.0007778471917845309,
+          -0.00181390973739326,
+          -0.0009202924557030201,
+          0.03827562928199768,
+          -0.00943233072757721,
+          -0.016158735379576683,
+          -0.005749027244746685,
+          0.012479699216783047,
+          0.008343985304236412,
+          -0.014972224831581116,
+          -0.01925732009112835,
+          0.018796373158693314,
+          -0.017908625304698944,
+          0.015697788447141647,
+          -0.04182662442326546,
+          -0.023303406313061714,
+          0.026615392416715622,
+          -9.683077223598957e-05,
+          -0.03503193333745003,
+          0.01168584730476141,
+          0.016329456120729446,
+          0.0028617095667868853,
+          0.008271428756415844,
+          -0.008702498860657215,
+          0.0028190291486680508,
+          -0.007754997815936804,
+          -0.010849314741790295,
+          0.0013636333169415593,
+          0.007114794570952654,
+          -0.007955594919621944,
+          0.02082795277237892,
+          -0.014195444993674755,
+          -0.015211233869194984,
+          -0.005680738482624292,
+          -0.0128552857786417,
+          0.01553560420870781,
+          -0.00581731554120779,
+          -0.013256479986011982,
+          0.017473286017775536,
+          0.014485670253634453,
+          0.015433171764016151,
+          0.02478867769241333,
+          0.018301282078027725,
+          -0.02250101789832115,
+          -0.002558679785579443,
+          3.8512240280397236e-05,
+          -0.0024733194150030613,
+          0.01227483432739973,
+          0.035236798226833344,
+          0.0016421218169853091,
+          -0.009816452860832214,
+          -0.02364484779536724,
+          0.01277846097946167,
+          0.01309429481625557,
+          -0.014041795395314693,
+          -0.018711013719439507,
+          -0.004114374052733183,
+          -0.013964971527457237,
+          0.00652153929695487,
+          -0.0023111344780772924,
+          0.000925094005651772,
+          -0.0009939158335328102,
+          -0.003484840737655759,
+          -0.02091331221163273,
+          0.005356369074434042,
+          -0.01953047327697277,
+          -0.031907740980386734,
+          -0.012530915439128876,
+          0.008036687038838863,
+          0.024754533544182777,
+          0.002035846933722496,
+          -0.0224839448928833,
+          -0.0014415248297154903,
+          -0.00016365200281143188,
+          -0.0093213627114892,
+          -0.01495515275746584,
+          0.0016197147779166698,
+          0.0005118960398249328,
+          0.010038389824330807,
+          -0.007682441268116236,
+          -0.01722574047744274,
+          0.005471605341881514,
+          0.01734524592757225,
+          -0.022381512448191643,
+          -0.0025202676188200712,
+          -0.017174525186419487,
+          -0.01559535600244999,
+          0.0873408243060112,
+          -0.010891994461417198,
+          -0.012539451941847801,
+          -0.01418690849095583,
+          0.0038092106115072966,
+          -0.006333746016025543,
+          0.010678594000637531,
+          0.03264183923602104,
+          -0.006026448681950569,
+          -0.006338014267385006,
+          -0.0026717823930084705,
+          -0.0008578726556152105,
+          -0.019359752535820007,
+          0.001124624046497047,
+          0.007921450771391392,
+          -0.004566784482449293,
+          -0.003316253889352083,
+          0.01450274232774973,
+          -0.04435329511761665,
+          0.0021916297264397144,
+          0.005096019711345434,
+          -0.02260345034301281,
+          0.02002556435763836,
+          0.015919726341962814,
+          -0.022057143971323967,
+          0.003002554178237915,
+          0.00606912886723876,
+          0.03311985731124878,
+          0.022296153008937836,
+          0.009218929335474968,
+          0.015723396092653275,
+          -0.01836957037448883,
+          0.012863821350038052,
+          -0.002146815648302436,
+          -0.0002396761701675132,
+          0.008954312652349472,
+          0.004472888074815273,
+          0.013145511038601398,
+          0.0016581269446760416,
+          0.0009016198455356061,
+          -0.022688809782266617,
+          -0.006850176956504583,
+          -0.026820257306098938,
+          -0.027895798906683922,
+          -0.010379832237958908,
+          0.03182237967848778,
+          -0.008988456800580025,
+          -0.014332021586596966,
+          0.006628239993005991,
+          -0.008292769081890583,
+          -0.002063589170575142,
+          -0.024617956951260567,
+          0.016337992623448372,
+          -0.007272711489349604,
+          0.0024413091596215963,
+          0.0005537759861908853,
+          0.0112334368750453,
+          -0.0064660548232495785,
+          -0.022808315232396126,
+          0.010584697127342224,
+          -0.0024583812337368727,
+          0.03426368907094002,
+          -0.00924453791230917,
+          -0.01836957037448883,
+          0.004780185874551535,
+          0.00041613224311731756,
+          -0.01941096968948841,
+          -0.005962428171187639,
+          -0.007793410215526819,
+          -0.021425476297736168,
+          -0.014613711275160313,
+          -0.013597921468317509,
+          -0.012027288787066936,
+          -0.01655992865562439,
+          0.006278262007981539,
+          0.00014591304352506995,
+          0.008574457839131355,
+          -0.01027739979326725,
+          -0.000391057605156675,
+          0.012539451941847801,
+          -0.015433171764016151,
+          -0.019308535382151604,
+          -0.007784874178469181,
+          -0.0041976007632911205,
+          -0.0013294891687110066,
+          0.005915479734539986,
+          -0.0218522772192955,
+          -0.037319593131542206,
+          0.011335869319736958,
+          -0.011002963408827782,
+          -0.004342713393270969,
+          0.022791242226958275,
+          0.000883480766788125,
+          -0.022705882787704468,
+          0.009415258653461933,
+          0.012599203735589981,
+          0.0045027644373476505,
+          -0.015612428076565266,
+          0.005510017741471529,
+          0.0034677686635404825,
+          -0.01110539585351944,
+          -0.01860858127474785,
+          0.0034528307151049376,
+          -0.008685426786541939,
+          0.005731955170631409,
+          -0.02786165475845337,
+          0.013657674193382263,
+          0.014784432016313076,
+          0.01084077823907137,
+          0.00021033351367805153,
+          -0.00475457776337862,
+          -0.01719159632921219,
+          -0.002552277874201536,
+          -0.00642764288932085,
+          0.07013215124607086,
+          -0.012752852402627468,
+          -0.017635472118854523,
+          0.016056302934885025,
+          0.0011384951649233699,
+          -0.011617559008300304,
+          0.009662804193794727,
+          -0.01848907582461834,
+          0.020623087882995605,
+          -0.03868535906076431,
+          0.012291906401515007,
+          -0.006060592830181122,
+          0.0075842770747840405,
+          -0.008655550889670849,
+          0.0006172628491185606,
+          -0.021408403292298317,
+          0.023918000981211662,
+          0.027656789869070053,
+          0.010431048460304737,
+          -0.01847200281918049,
+          -0.027895798906683922,
+          0.016824547201395035,
+          0.017003804445266724,
+          0.002428505104035139,
+          -0.009637195616960526,
+          0.011754135601222515,
+          -0.023952145129442215,
+          0.008945776149630547,
+          -0.006346550304442644,
+          -0.0032372954301536083,
+          0.009218929335474968,
+          0.007814750075340271,
+          0.007951326668262482,
+          0.009364042431116104,
+          -0.03235161304473877,
+          0.016798939555883408,
+          0.028800619766116142,
+          0.004592392593622208,
+          0.028800619766116142,
+          -0.018796373158693314,
+          0.01823299378156662,
+          -0.006752012297511101,
+          0.009022600948810577,
+          0.021254755556583405,
+          0.011463910341262817,
+          -0.030081026256084442,
+          0.0027678129263222218,
+          -0.03380274400115013,
+          0.0017712295521050692,
+          0.02673489600419998,
+          -0.0044302078895270824,
+          0.001891801250167191,
+          -0.02284245938062668,
+          0.008860415779054165,
+          0.004443012177944183,
+          -0.020059708505868912,
+          -0.014357629232108593,
+          0.00600510835647583,
+          0.0025800198782235384,
+          -0.026547104120254517,
+          0.022518089041113853,
+          -0.023815568536520004,
+          -0.0010488666594028473,
+          0.017618399113416672,
+          0.0006231313454918563,
+          -0.01579168438911438,
+          0.008420809172093868,
+          0.02159619703888893,
+          -0.02579593099653721,
+          -0.032027244567871094,
+          0.006431910675019026,
+          0.0075842770747840405,
+          0.021630341187119484,
+          0.01978655532002449,
+          -0.008723839186131954,
+          0.02287660352885723,
+          0.007840358652174473,
+          0.0038177466485649347,
+          -0.0372513048350811,
+          -0.017635472118854523,
+          0.019957276061177254,
+          -0.014536886475980282,
+          0.018454931676387787,
+          -0.011412693187594414,
+          -0.001605843659490347,
+          0.009355506859719753,
+          -0.008139119483530521,
+          0.027793366461992264,
+          -0.01405886746942997,
+          -0.0036576958373188972,
+          0.004425940103828907,
+          -0.03341008350253105,
+          -0.0024178349412977695,
+          0.025727642700076103,
+          0.001300679985433817,
+          -0.02454966865479946,
+          -0.0030132243409752846,
+          -0.0023580826818943024,
+          0.003779334481805563,
+          -0.005475873593240976,
+          -0.007733657956123352,
+          -0.010644449852406979,
+          0.003572335233911872,
+          -0.006453251000493765,
+          -0.011352941393852234,
+          0.00214574858546257,
+          0.04985050857067108,
+          0.015817293897271156,
+          -0.007123330608010292,
+          0.0005857861833646894,
+          0.01728549413383007,
+          0.042953383177518845,
+          -0.022825386375188828,
+          0.03122485615313053,
+          0.0009693747269921005,
+          0.0038540249224752188,
+          -0.02478867769241333,
+          -0.026052013039588928,
+          0.02632516622543335,
+          0.010721273720264435,
+          -0.02542034536600113,
+          -0.002863843459635973,
+          0.00415492057800293,
+          0.012223618105053902,
+          0.014929544180631638,
+          0.014690535143017769,
+          -0.031241929158568382,
+          -0.015475851483643055,
+          -0.003343996126204729,
+          -0.008954312652349472,
+          -0.00606912886723876,
+          -0.005322224926203489,
+          0.013478416949510574,
+          0.008143387734889984,
+          -0.025778859853744507,
+          -0.0013454942964017391,
+          -0.005390513222664595,
+          -0.005343564786016941,
+          -0.01033715158700943,
+          0.009278682060539722,
+          -0.009227465838193893,
+          0.0040610237047076225,
+          -0.010448120534420013,
+          0.010832242667675018,
+          -0.015074657276272774,
+          0.010516408830881119,
+          0.004186930600553751,
+          -0.02094745635986328,
+          -0.012496771290898323,
+          -0.01838664337992668,
+          0.007183082867413759,
+          0.01937682554125786,
+          0.01835249923169613,
+          0.0062996018677949905,
+          -0.014084476046264172,
+          -0.016039229929447174,
+          -0.015151482075452805,
+          0.017908625304698944,
+          0.01014935877174139,
+          -0.005501481704413891,
+          -0.018028128892183304,
+          -0.025659354403614998,
+          0.022518089041113853,
+          0.005988036282360554,
+          -0.03612454608082771,
+          0.02250101789832115,
+          -0.008326913230121136,
+          -6.268659035413293e-06,
+          0.012761388905346394,
+          -0.010900530964136124,
+          -0.004400331526994705,
+          -0.0023474127519875765,
+          0.019206102937459946,
+          -0.0014916741056367755,
+          -0.0008450685418210924,
+          0.005706346593797207,
+          -0.00011263579654041678,
+          -0.008817736059427261,
+          -0.013213799335062504,
+          -0.0037537263706326485,
+          0.001452194876037538,
+          -0.0033226560335606337,
+          -0.0024498451966792345,
+          0.004545444622635841,
+          0.014681999571621418,
+          0.013802786357700825,
+          0.005578306037932634,
+          0.004746041726320982,
+          0.0007917182520031929,
+          -0.007242835126817226,
+          0.01059323363006115,
+          -0.021408403292298317,
+          0.006816032808274031,
+          -0.005569770000874996,
+          -0.020725520327687263,
+          -0.005902675911784172,
+          0.00018845988961402327,
+          0.02157912403345108,
+          -0.03431490436196327,
+          0.0038198805414140224,
+          0.017294028773903847,
+          0.009346970357000828,
+          -0.01836957037448883,
+          0.010883458890020847,
+          0.06518124788999557,
+          -0.005271008238196373,
+          0.008284232579171658,
+          -0.0010888794204220176,
+          0.011779743246734142,
+          -0.006184365600347519,
+          0.005322224926203489,
+          -0.0017360183410346508,
+          0.0346904918551445,
+          -0.007823286578059196,
+          -0.012940646149218082,
+          -0.023183900862932205,
+          -0.012496771290898323,
+          -0.016397744417190552,
+          0.02786165475845337,
+          -0.014477133750915527,
+          0.002011305885389447,
+          -0.012078505009412766,
+          0.0065386113710701466,
+          0.002077460289001465,
+          0.0014596638502553105,
+          -0.018215922638773918,
+          -0.009927421808242798,
+          0.016747722402215004,
+          0.0034442946780472994,
+          0.0023666187189519405,
+          0.00530088460072875,
+          -0.014528350904583931,
+          0.030524900183081627,
+          -0.0074477000162005424,
+          0.020059708505868912,
+          0.004391795489937067,
+          -0.005232596304267645,
+          0.015185626223683357,
+          -0.009466474875807762,
+          0.015859974548220634,
+          -0.0118821756914258,
+          -0.003529655048623681,
+          -0.030149314552545547,
+          -0.008565922267735004,
+          0.020623087882995605,
+          -0.014519814401865005,
+          0.0005495079676620662,
+          0.016525784507393837,
+          0.005014927126467228,
+          -0.007959863170981407,
+          0.011224900372326374,
+          0.003382408292964101,
+          -0.01758425496518612,
+          -0.010832242667675018,
+          -0.004647877067327499,
+          -0.01072981022298336,
+          0.011301725171506405,
+          0.013589384965598583,
+          -0.022398585453629494,
+          0.0015290192095562816,
+          -0.006914197467267513,
+          -0.008207408711314201,
+          0.00716174254193902,
+          0.013162583112716675,
+          -0.027571428567171097,
+          0.017370853573083878,
+          0.0021254755556583405,
+          -0.0002107336331391707,
+          0.004250951111316681,
+          -0.0004980249213986099,
+          0.014605174772441387,
+          -0.007575741037726402,
+          -0.002740070689469576,
+          -0.006845909170806408,
+          0.010934675112366676,
+          -0.0004094634496141225,
+          0.0157319325953722,
+          -0.00683737313374877,
+          -0.002915059681981802,
+          0.007234299089759588,
+          0.016286775469779968,
+          -0.02557399496436119,
+          0.015424635261297226,
+          -0.00810070801526308,
+          0.0002974278468172997,
+          0.00031343294540420175,
+          0.009253074415028095,
+          0.005591110326349735,
+          -0.015902653336524963,
+          -0.0072300308384001255,
+          0.015569748356938362,
+          0.005117359571158886,
+          0.018950022757053375,
+          0.021476691588759422,
+          0.007033701986074448,
+          0.023866785690188408,
+          -0.005744758993387222,
+          -0.013085758313536644,
+          -0.0020561201963573694,
+          -0.009338434785604477,
+          -0.002368752844631672,
+          0.006286798045039177,
+          -0.004054621793329716,
+          -0.025915436446666718,
+          -0.012821141630411148,
+          -0.007276979275047779,
+          -0.00038012079312466085,
+          0.009475011378526688,
+          -0.010311543941497803,
+          0.01707209274172783,
+          0.02236444130539894,
+          7.760800144751556e-06,
+          -0.006175829563289881,
+          -0.02094745635986328,
+          0.007251371163874865,
+          -0.007763533852994442,
+          -0.011737063527107239,
+          -0.010098142549395561,
+          -0.0028702456038445234,
+          0.0018395179649814963,
+          0.005407585296779871,
+          0.004775917623192072,
+          -0.017772048711776733,
+          -0.006862981244921684,
+          0.009423795156180859,
+          -0.01799398474395275,
+          -0.0023239385336637497,
+          0.00031343294540420175,
+          0.006666651926934719,
+          0.012872357852756977,
+          0.01758425496518612,
+          -0.013614993542432785,
+          -0.022449800744652748,
+          0.003002554178237915,
+          -0.009739628992974758,
+          0.03274427354335785,
+          -0.008168996311724186,
+          -0.005514285992830992,
+          -0.018284210935235023,
+          -0.006175829563289881,
+          -0.009346970357000828,
+          -0.00956890732049942,
+          -0.0004897556500509381,
+          0.014861255884170532,
+          -0.009304290637373924,
+          0.01482711173593998,
+          -0.016491640359163284,
+          -0.003205285407602787,
+          0.00409090006724,
+          0.00969694834202528,
+          -0.030012737959623337,
+          -0.006278262007981539,
+          -0.00799400731921196,
+          -0.02145962044596672,
+          -0.013708890415728092,
+          -0.012471163645386696,
+          0.007588544860482216,
+          0.0014073805650696158,
+          0.01642335206270218,
+          -0.007187350653111935,
+          -0.014195444993674755,
+          0.009978638030588627,
+          -0.02031579054892063,
+          -0.012983325868844986,
+          -0.008292769081890583,
+          -0.04073401167988777,
+          0.020469438284635544,
+          0.01354670524597168,
+          0.023337550461292267,
+          -0.0024434432853013277,
+          0.020964529365301132,
+          0.008723839186131954,
+          -0.01742207072675228,
+          0.006372158415615559,
+          -0.012505307793617249,
+          0.005744758993387222,
+          -0.02506183087825775,
+          -0.0066026318818330765,
+          0.005505749955773354,
+          0.01246262714266777,
+          -0.0037665304262191057,
+          2.7758822398027405e-05,
+          0.014280805364251137,
+          -0.01041397638618946,
+          -0.01787448115646839,
+          0.004067426081746817,
+          0.0067562805488705635,
+          -0.01713184453547001,
+          0.03670499846339226,
+          -0.00796413142234087,
+          -0.00017445544654037803,
+          0.013179655186831951,
+          0.02104988880455494,
+          -0.020264573395252228,
+          -0.01450274232774973,
+          0.01937682554125786,
+          -0.0010200574761256576,
+          -0.0004940236685797572,
+          0.017183061689138412,
+          -0.006828837096691132,
+          0.008032419718801975,
+          -0.005766099318861961,
+          -0.00484847417101264,
+          -0.00503626698628068,
+          0.0011683712946251035,
+          -0.016901372000575066,
+          0.0042637549340724945,
+          -0.000693553767632693,
+          -0.0006658115889877081,
+          0.019581690430641174,
+          -0.006453251000493765,
+          0.0020102388225495815,
+          0.011976072564721107,
+          0.00038332180702127516,
+          -0.05049924924969673,
+          -0.0047204336151480675,
+          0.0034869748633354902,
+          -0.003279975615441799,
+          -0.003469902789220214,
+          0.009807917289435863,
+          0.01195046491920948,
+          0.01110539585351944,
+          -0.005130163859575987,
+          0.013452808372676373,
+          0.00938111450523138,
+          0.006632507778704166,
+          0.0007287649204954505,
+          -0.004329909570515156,
+          0.011813887394964695,
+          -0.004588124807924032,
+          0.020127996802330017,
+          -0.004421671852469444,
+          -0.01889880560338497,
+          -0.016414817422628403,
+          0.0010739413555711508,
+          0.027025122195482254,
+          -0.0037665304262191057,
+          0.026103228330612183,
+          -0.0013839064631611109,
+          0.0033205219078809023,
+          -0.0018032396910712123,
+          -0.0032223574817180634,
+          -0.004827133845537901,
+          -0.035134363919496536,
+          0.009039673022925854,
+          -0.007084918208420277,
+          0.020008491352200508,
+          -0.02943228743970394,
+          0.003026028396561742,
+          -0.006175829563289881,
+          -0.005787439178675413,
+          -0.0025608139112591743,
+          0.005031999200582504,
+          -2.869245145120658e-05,
+          0.03803662210702896,
+          -0.014844183810055256,
+          -0.021271826699376106,
+          -0.010396904312074184,
+          0.006995289586484432,
+          0.02798115834593773,
+          -0.0004838871245738119,
+          0.01913781464099884,
+          0.020896241068840027,
+          -0.013068686239421368,
+          0.004844206385314465,
+          -0.0031156567856669426,
+          -0.002942801918834448,
+          0.00011677043949021026,
+          -0.009440867230296135,
+          0.005198452156037092,
+          -0.03530508652329445,
+          0.007008093874901533,
+          -0.0005225660861469805,
+          0.02427651546895504,
+          5.1282964705023915e-05,
+          0.004814330022782087,
+          -0.003311985870823264,
+          0.02274002693593502,
+          0.003945787437260151,
+          0.01719159632921219,
+          -0.0013017470482736826,
+          0.004767381586134434,
+          0.010396904312074184,
+          -0.019086599349975586,
+          -0.019820699468255043,
+          -0.015142945572733879,
+          0.001055268687196076,
+          0.0011448971927165985,
+          -0.015270986594259739,
+          -0.018830517306923866,
+          -0.0059923045337200165,
+          0.017515966668725014,
+          -0.011702919378876686,
+          0.01264188438653946,
+          0.013538168743252754,
+          0.009159177541732788,
+          -0.001976094674319029,
+          0.01227483432739973,
+          0.01592826284468174,
+          -0.018215922638773918,
+          0.009022600948810577,
+          -0.009577443823218346,
+          -0.008937240578234196,
+          -0.010960282757878304,
+          -0.01811349019408226,
+          0.03438319265842438,
+          0.014750287868082523,
+          -0.011677310802042484,
+          -0.008587262593209743,
+          -0.0010019184555858374,
+          0.004758845549076796,
+          0.00044013987644575536,
+          0.005663666408509016,
+          -0.004647877067327499,
+          -0.026137372478842735,
+          -0.014673463068902493,
+          -0.016466032713651657,
+          -0.005719150882214308,
+          0.002590690040960908,
+          0.014801504090428352,
+          -0.021732773631811142,
+          -0.016397744417190552,
+          -0.0263080932199955,
+          0.01925732009112835,
+          -0.017533037811517715,
+          0.01651724986732006,
+          0.012249225750565529,
+          0.002076393226161599,
+          -0.01265041995793581,
+          -0.02801530249416828,
+          0.014741751365363598,
+          -0.024361874908208847,
+          0.00670933211222291,
+          -0.0016261168057098985,
+          0.0500553734600544,
+          -0.00277634896337986,
+          -0.006192901637405157,
+          0.009500619024038315,
+          0.003561665304005146,
+          0.013845467008650303,
+          -0.02197178266942501,
+          -0.005403317045420408,
+          -0.003525387030094862,
+          0.011566342785954475,
+          -0.004571052733808756,
+          -0.0056892745196819305,
+          -0.002590690040960908,
+          -0.007375143934041262,
+          0.001402045600116253,
+          0.019291464239358902,
+          -0.006880053319036961,
+          -0.007008093874901533,
+          -0.0030921828001737595,
+          0.029842017218470573,
+          0.01963290572166443,
+          -0.008548850193619728,
+          -0.006444714963436127,
+          0.008365325629711151,
+          -0.01787448115646839,
+          -0.0014916741056367755,
+          0.017370853573083878,
+          0.021220611408352852,
+          -0.011224900372326374,
+          0.006999557837843895,
+          0.002942801918834448,
+          -0.0033269240520894527,
+          -0.021510835736989975,
+          -0.01137001346796751,
+          -0.0075202565640211105,
+          -0.0018896672409027815,
+          0.01632091961801052,
+          -0.011310260742902756,
+          0.02325218915939331,
+          0.01567218080163002,
+          0.009517691098153591,
+          0.0010883458890020847,
+          -0.041280318051576614,
+          -0.01548438798636198,
+          0.01687576249241829,
+          -0.0034613667521625757,
+          0.017362317070364952,
+          0.013649137690663338,
+          0.0031007188372313976,
+          -0.019479257985949516,
+          0.023952145129442215,
+          0.0008594731334596872,
+          0.0014233856927603483,
+          -0.001049933722242713,
+          0.015330738388001919,
+          0.0019632906187325716,
+          0.00962012354284525,
+          0.02106696180999279,
+          0.020640159025788307,
+          -0.01797691360116005,
+          -0.009987173601984978,
+          0.011301725171506405,
+          -0.017037948593497276,
+          0.019684122875332832,
+          0.001101149944588542,
+          0.003862560959532857,
+          0.015501460060477257,
+          0.027025122195482254,
+          0.03277841582894325,
+          0.011062716133892536,
+          -0.004933834541589022,
+          -0.007959863170981407,
+          -0.001679467037320137,
+          0.004545444622635841,
+          0.014041795395314693,
+          -0.02337169460952282,
+          -0.031054135411977768,
+          0.003213821444660425,
+          0.013529633171856403,
+          -0.02506183087825775,
+          0.007725121919065714,
+          0.01913781464099884,
+          0.02181813307106495,
+          0.0050917514599859715,
+          0.015552676282823086,
+          0.014519814401865005,
+          0.0034528307151049376,
+          -0.002049718052148819,
+          0.010046926327049732,
+          -0.013623529113829136,
+          0.005155771970748901,
+          0.0022833922412246466,
+          -0.015509995631873608,
+          0.02053772658109665,
+          -0.00037345202872529626,
+          0.00026034939219243824,
+          0.003395212348550558,
+          0.001364700379781425,
+          -0.00950915552675724,
+          -0.03426368907094002,
+          0.016551394015550613,
+          -0.004159188363701105,
+          0.010439584031701088,
+          -0.017020875588059425,
+          -0.0075714727863669395,
+          -0.007511720526963472,
+          0.03421247377991676,
+          0.009304290637373924,
+          0.007631225045770407,
+          -0.004293631296604872,
+          0.0008301304769702256,
+          0.003205285407602787,
+          0.007699513342231512,
+          -0.01799398474395275,
+          -0.018045201897621155,
+          0.009662804193794727,
+          0.004903958644717932,
+          8.744445949560031e-06,
+          0.01642335206270218,
+          0.009073817171156406,
+          0.012078505009412766,
+          0.02299610897898674,
+          0.008040955290198326,
+          -0.01728549413383007,
+          0.0033013159409165382,
+          0.019957276061177254,
+          -0.0008957513491623104,
+          0.01707209274172783,
+          0.007486112415790558,
+          -0.007008093874901533,
+          -0.026103228330612183,
+          0.00020659899746533483,
+          0.018164705485105515,
+          0.023952145129442215,
+          -0.023969218134880066,
+          -0.019052455201745033,
+          0.01850614883005619,
+          -0.0121980095282197,
+          0.010627377778291702,
+          -0.002761411014944315,
+          0.000420933763962239,
+          0.0031754090450704098,
+          -0.017217205837368965,
+          0.004562516696751118,
+          0.017037948593497276,
+          0.01014935877174139,
+          0.00015778347733430564,
+          0.005386244971305132,
+          0.01809641718864441,
+          0.022449800744652748,
+          -0.030661476776003838,
+          -0.010968819260597229,
+          -0.0029769460670650005,
+          0.004579588770866394,
+          -0.014613711275160313,
+          -0.0028531732968986034,
+          0.005108823534101248,
+          0.00937257893383503,
+          -0.022057143971323967,
+          -0.0109176030382514,
+          -0.0023025984410196543,
+          -0.0024626492522656918,
+          0.008903096430003643,
+          -0.004733237437903881,
+          0.0030559045262634754,
+          -0.016798939555883408,
+          0.005492945667356253,
+          -0.019615834578871727,
+          0.017183061689138412,
+          -0.0049252985045313835,
+          -0.0008722771890461445,
+          -0.010217647068202496,
+          0.0021062693558633327,
+          -0.0028702456038445234,
+          -0.02390092983841896,
+          0.022893676534295082,
+          -0.0157148614525795,
+          -0.0036897058598697186,
+          0.043738700449466705,
+          0.002827565185725689,
+          0.006828837096691132,
+          0.02364484779536724,
+          0.002174557652324438,
+          -0.004647877067327499,
+          0.026103228330612183,
+          0.03175409138202667,
+          -0.010550552979111671,
+          -0.009748164564371109,
+          0.017618399113416672,
+          0.009278682060539722,
+          0.006935537327080965,
+          0.011489517986774445,
+          -0.0027784830890595913,
+          -0.017686687409877777,
+          -0.00911649689078331,
+          0.0011790413409471512,
+          0.015492923557758331,
+          -0.0032308935187757015,
+          -0.0020059708040207624,
+          0.011924856342375278,
+          7.60908515076153e-05,
+          -0.007524524815380573,
+          0.02837381698191166,
+          0.002735802670940757,
+          -0.01784033700823784,
+          -0.029210349544882774,
+          0.0118821756914258,
+          -0.0125138433650136,
+          -0.018796373158693314,
+          -0.017294028773903847,
+          -0.0021030684001743793,
+          -0.011301725171506405,
+          -0.002526669530197978,
+          0.012829677201807499,
+          0.01163463108241558,
+          -0.005211255978792906,
+          0.01620141603052616,
+          0.006329478230327368,
+          -0.015902653336524963,
+          -0.01612459123134613,
+          -0.004276559222489595,
+          -0.003909509163349867,
+          -0.0035040469374507666,
+          -0.003911643289029598,
+          0.018198849633336067,
+          -0.009090889245271683,
+          -0.0044302078895270824,
+          -0.018557364121079445,
+          0.01760132610797882,
+          0.013700353913009167,
+          -0.009398186579346657,
+          -0.001582369557581842,
+          -0.001988898729905486,
+          0.028203096240758896,
+          0.0062996018677949905,
+          0.0017947036540135741,
+          0.004297899082303047,
+          -0.00968841277062893,
+          -0.0010846114018931985,
+          -0.02463502809405327,
+          0.020093852654099464,
+          -0.018437858670949936,
+          -0.026939760893583298,
+          -0.007170278578996658,
+          0.024959398433566093,
+          -0.0018395179649814963,
+          0.0015076791169121861,
+          -0.00831837672740221,
+          0.018932949751615524,
+          0.0029812140855938196,
+          -0.02823724038898945,
+          -0.0125138433650136,
+          -0.010183502919971943,
+          -0.015552676282823086,
+          -0.0013518963241949677,
+          0.020332861691713333,
+          -0.01360645703971386,
+          0.00035237864358350635,
+          0.01671357825398445,
+          -0.01149805448949337,
+          0.013469880446791649,
+          -0.003260769648477435,
+          -0.009125033393502235,
+          0.013845467008650303,
+          -0.008536046370863914,
+          -0.00408022990450263,
+          -0.003094316693022847,
+          0.005275276489555836,
+          -0.0012387937167659402,
+          -0.019086599349975586,
+          -0.009440867230296135,
+          0.01084077823907137,
+          0.002966275904327631,
+          0.008728107437491417,
+          -0.014050331898033619,
+          -0.005629522260278463,
+          0.001096881926059723,
+          -7.188950985437259e-05,
+          -0.0018960692686960101,
+          -0.012582131661474705,
+          -0.011327332817018032,
+          -0.020879168063402176,
+          -0.007665369194000959,
+          -0.01206143293529749,
+          0.011173684149980545,
+          0.015245378017425537,
+          -0.010738345794379711,
+          -0.005642326548695564,
+          0.008305572904646397,
+          0.04100716486573219,
+          -0.009782308712601662,
+          0.004383259452879429,
+          0.021613268181681633,
+          0.0032372954301536083,
+          0.007592813111841679,
+          0.02874940261244774,
+          -0.0017701626056805253,
+          -0.010704201646149158,
+          0.005117359571158886,
+          -0.01825006678700447,
+          -0.002957739867269993,
+          0.027554357424378395,
+          -0.021271826699376106,
+          0.005787439178675413,
+          -0.023730207234621048,
+          -0.008079367689788342,
+          -0.0033888102043420076,
+          -0.0121980095282197,
+          -0.014238124713301659,
+          -0.002049718052148819,
+          0.0006028582574799657,
+          0.023713136091828346,
+          -0.007439163979142904,
+          0.0050405352376401424,
+          0.009159177541732788,
+          -0.025130119174718857,
+          0.007426360156387091,
+          -0.010994426906108856,
+          -0.010166430845856667,
+          -0.004916762467473745,
+          -0.026803184300661087,
+          0.015501460060477257,
+          -0.011540734209120274,
+          -0.006717868149280548,
+          -0.018557364121079445,
+          0.033205218613147736,
+          0.017635472118854523,
+          0.011276116594672203,
+          -0.02313268557190895,
+          -0.015014905482530594,
+          -0.020110923796892166,
+          -0.037046439945697784,
+          -0.00030409664032049477,
+          0.0008386665140278637,
+          -0.002919327700510621,
+          0.007716585882008076,
+          0.025864219292998314,
+          0.0017701626056805253,
+          -0.006444714963436127,
+          -0.004575320519506931,
+          0.007375143934041262,
+          0.009005528874695301,
+          -0.014417381957173347,
+          0.00911649689078331,
+          0.018301282078027725,
+          0.0002280724875163287,
+          0.007725121919065714,
+          0.00813058391213417,
+          0.0020305118523538113,
+          0.00548867741599679,
+          0.028407961130142212,
+          0.0034122844226658344,
+          0.003109254874289036,
+          -0.0001357764849672094,
+          -7.043905043246923e-06,
+          -0.011148076504468918,
+          0.0010947479167953134,
+          -0.018591508269309998,
+          0.00600510835647583,
+          -0.01860858127474785,
+          0.0199231319129467,
+          -0.004127178341150284,
+          0.008092171512544155,
+          -0.02207421511411667,
+          -0.0243960190564394,
+          0.004268023185431957,
+          0.027178769931197166,
+          0.01259066816419363,
+          -0.007878770120441914,
+          -0.015859974548220634,
+          0.01888173446059227,
+          0.01104564405977726,
+          -0.011267581023275852,
+          -0.012505307793617249,
+          -0.017789119854569435,
+          -9.489682270213962e-05,
+          -0.02144254744052887,
+          0.02528376877307892,
+          0.015902653336524963,
+          0.0038881690707057714,
+          -0.013171118684113026,
+          0.02425944246351719,
+          -0.011549270711839199,
+          0.0070465062744915485,
+          -0.01503197755664587,
+          -0.015561211854219437,
+          -0.0186427254229784,
+          -0.005945356097072363,
+          -0.001342293224297464,
+          0.011651703156530857,
+          -0.027571428567171097,
+          0.013768642209470272,
+          0.014391773380339146,
+          -0.002705926541239023,
+          0.0022215060889720917,
+          -0.0055356258526444435,
+          0.014571030624210835,
+          -0.006875785067677498,
+          0.007336731534451246,
+          -0.006991021800786257,
+          -0.0021873617079108953,
+          -0.0065172710455954075,
+          -0.01418690849095583,
+          -0.014656390994787216,
+          -0.0053307609632611275,
+          -0.014110084623098373,
+          0.0035125829745084047,
+          0.019342679530382156,
+          -0.026052013039588928,
+          -0.00018205786182079464,
+          -0.001230257679708302,
+          -6.235315231606364e-05,
+          -0.0032735737040638924,
+          0.012795533053576946,
+          0.016918443143367767,
+          -0.0067861564457416534,
+          -0.005117359571158886,
+          -0.0007324994658119977,
+          -0.017498893663287163,
+          -0.0004924231325276196,
+          0.021203538402915,
+          -0.0031433990225195885,
+          -0.005962428171187639,
+          -0.010934675112366676,
+          -0.038856081664562225,
+          -0.01078956201672554,
+          -0.01862565241754055,
+          -0.021544979885220528,
+          -0.006939805578440428,
+          0.0007244969019666314,
+          0.008028151467442513,
+          0.01381132286041975,
+          -0.010712738148868084,
+          -0.011096860282123089,
+          0.0033290579449385405,
+          -0.01622702367603779,
+          0.0016463898355141282,
+          -0.019342679530382156,
+          -0.0013614993076771498,
+          0.008783590979874134,
+          0.014212517067790031,
+          0.0044515482150018215,
+          0.00632094219326973,
+          0.0019995688926428556,
+          -0.011301725171506405,
+          -0.009978638030588627,
+          0.019189031794667244,
+          -0.022552233189344406,
+          0.003687571967020631,
+          -0.004865546245127916,
+          0.0065386113710701466,
+          -0.0034400266595184803,
+          -0.02159619703888893,
+          0.006367890164256096,
+          -0.02069137617945671,
+          0.010362760163843632,
+          -0.0010099209612235427,
+          -0.004950906615704298,
+          0.006658115889877081,
+          -0.0021521507296711206,
+          -0.011719991452991962,
+          0.01772083155810833,
+          -0.006777620408684015,
+          -0.004511300474405289,
+          -0.007191618904471397,
+          0.006244117859750986,
+          0.01588558219373226,
+          0.011037107557058334,
+          -0.0053819771856069565,
+          0.0062483856454491615,
+          0.011455373838543892,
+          -0.0026995246298611164,
+          -0.00767817348241806,
+          -0.01760132610797882,
+          0.00048788837739266455,
+          0.013956435024738312,
+          -0.020657232031226158,
+          0.0059282840229570866,
+          0.011967536993324757,
+          0.024310659617185593,
+          -0.007494648452848196,
+          0.006991021800786257,
+          -0.01913781464099884,
+          0.015561211854219437,
+          -0.00414425041526556,
+          -0.02250101789832115,
+          -0.0015524934278801084,
+          -0.013802786357700825,
+          0.0009058878640644252,
+          -0.005305152852088213,
+          0.00010189905151491985,
+          -0.0060435207560658455,
+          -0.0010563357500359416,
+          -0.004724701400846243,
+          -0.015262450091540813,
+          0.0013454942964017391,
+          -0.01610751822590828,
+          0.006790424697101116,
+          0.015288058668375015,
+          0.0144515261054039,
+          0.0028489052783697844,
+          0.00439606374129653,
+          -0.0030196262523531914,
+          0.002827565185725689,
+          -0.011916320770978928,
+          -0.004750309512019157,
+          0.007481844164431095,
+          -0.018284210935235023,
+          0.0005361703806556761,
+          -0.009338434785604477,
+          0.009099424816668034,
+          -0.022927820682525635,
+          0.003696108004078269,
+          -0.00207212520763278,
+          -0.014545422978699207,
+          -0.007072114385664463,
+          -0.007908646948635578,
+          0.023013180121779442,
+          0.010285935364663601,
+          0.01458810269832611,
+          0.005049071274697781,
+          -0.018130561336874962,
+          -0.01567218080163002,
+          0.0008263959898613393,
+          -0.001281473902054131,
+          0.02209128811955452,
+          -0.018932949751615524,
+          0.004464352037757635,
+          0.00844641774892807,
+          -0.02878354676067829,
+          -0.012283369898796082,
+          0.001568498439155519,
+          -0.01335891243070364,
+          0.009483546949923038,
+          0.009850597009062767,
+          -0.00800254289060831,
+          -0.0037729323375970125,
+          -0.004417404066771269,
+          -0.01630384847521782,
+          -0.006854445207864046,
+          -0.002769947052001953,
+          0.01169438287615776,
+          -0.009731092490255833,
+          0.0021916297264397144,
+          0.005445997230708599,
+          0.0121980095282197,
+          -0.007853162474930286,
+          -0.011207828298211098,
+          -0.010687129572033882,
+          0.002074259100481868,
+          0.015416099689900875,
+          -0.01823299378156662,
+          -0.010260327719151974,
+          0.015347810462117195,
+          -0.017515966668725014,
+          0.009893277660012245,
+          -0.00962012354284525,
+          -0.021732773631811142,
+          -0.006730672437697649,
+          0.016978194937109947,
+          0.000151514817844145,
+          0.02453259564936161,
+          0.01124197244644165,
+          -0.00622704578563571,
+          -0.006397766526788473,
+          -0.02349119819700718,
+          0.01201021671295166,
+          -0.006901393178850412,
+          0.0069739497266709805,
+          -0.000578317150939256,
+          -6.068595394026488e-05,
+          0.011711454950273037,
+          -0.001123557100072503,
+          -0.004107972141355276,
+          -0.011907784268260002,
+          -0.008087903261184692,
+          0.02951764687895775,
+          -0.007554400712251663,
+          0.017686687409877777,
+          0.01259066816419363,
+          -0.006884321104735136,
+          0.008361057378351688,
+          0.009048208594322205,
+          0.00388603494502604,
+          -0.010823706164956093,
+          0.01458810269832611,
+          -0.010806634090840816,
+          -0.016337992623448372,
+          0.008373861201107502,
+          0.007106258533895016,
+          -0.012027288787066936,
+          -0.008971384726464748,
+          0.0018619250040501356,
+          -0.0011854433687403798,
+          0.0003201017389073968,
+          -0.011011499911546707,
+          -0.009150641039013863,
+          -0.002554411767050624,
+          -0.005416121333837509,
+          0.0021830936893820763,
+          0.0027976890560239553,
+          0.008480561897158623,
+          0.002078527119010687,
+          -0.01645749621093273,
+          -0.01976948231458664,
+          -0.022279080003499985,
+          -0.015629500150680542,
+          0.00491249468177557,
+          0.0001392442500218749,
+          0.010268863290548325,
+          -0.007716585882008076,
+          0.014758823439478874,
+          -0.02079380862414837,
+          0.011412693187594414,
+          0.013324768282473087,
+          0.006508735008537769,
+          -0.013760106638073921,
+          -0.023098541423678398,
+          -0.012573596090078354,
+          0.0030004202853888273,
+          0.006952609401196241,
+          0.015125873498618603,
+          -0.00032917127828113735,
+          -0.008757983334362507,
+          0.00399700365960598,
+          -0.012573596090078354,
+          0.006769084371626377,
+          -0.00616302527487278,
+          0.0069227335043251514,
+          0.0011267580557614565,
+          -0.006901393178850412,
+          0.007985470816493034,
+          -0.005740491207689047,
+          0.0033717381302267313,
+          -0.008655550889670849,
+          -0.01663675345480442,
+          0.016431888565421104,
+          -0.0013518963241949677,
+          -0.007434896193444729,
+          -0.00860433466732502,
+          -0.004257353022694588,
+          -0.0015055451076477766,
+          -0.0036790359299629927,
+          0.009526227600872517,
+          0.011335869319736958,
+          0.027042193338274956,
+          0.00886895228177309,
+          0.008164728060364723,
+          0.004287228919565678,
+          -0.0005313688889145851,
+          0.013162583112716675,
+          0.018830517306923866,
+          0.00020873300672974437,
+          0.002744338707998395,
+          -0.0045539806596934795,
+          -0.02168155647814274,
+          -0.004562516696751118,
+          0.017686687409877777,
+          0.00937257893383503,
+          0.002082795137539506,
+          -0.007302587386220694,
+          0.0013999115908518434,
+          0.0019451514817774296,
+          -0.02299610897898674,
+          0.015919726341962814,
+          -0.005919747985899448,
+          -0.0041613224893808365,
+          0.015014905482530594,
+          -0.016594072803854942,
+          0.0052667404524981976,
+          -0.009176249615848064,
+          0.0031988832633942366,
+          0.0023346086964011192,
+          0.008267160505056381,
+          0.03762689232826233,
+          -0.01496368832886219,
+          0.006534343119710684,
+          0.021135250106453896,
+          -0.008629942312836647,
+          -0.017772048711776733,
+          -0.003045234363526106,
+          -0.006000840570777655,
+          0.004191198851913214,
+          0.012223618105053902,
+          0.0030921828001737595,
+          -0.0007869167602621019,
+          0.017515966668725014,
+          0.001877930131740868,
+          0.01206143293529749,
+          -0.014852720312774181,
+          -1.1286920198472217e-05,
+          -0.008937240578234196,
+          0.013410128653049469,
+          0.0019728937186300755,
+          -0.015902653336524963,
+          -0.002094532363116741,
+          0.02043529413640499,
+          -0.018215922638773918,
+          0.013717425987124443,
+          -0.006090468727052212,
+          0.02673489600419998,
+          0.004711897578090429,
+          0.0073153916746377945,
+          -0.004818597808480263,
+          0.009705484844744205,
+          0.013614993542432785,
+          0.024361874908208847,
+          -0.0015610294649377465,
+          0.002044382970780134,
+          0.0058557274751365185,
+          0.013691818341612816,
+          0.01610751822590828,
+          0.006478859111666679,
+          -0.0016549259889870882,
+          0.029978593811392784,
+          0.01720866933465004,
+          0.007306855637580156,
+          0.004438743926584721,
+          0.0001512480666860938,
+          -0.017200132831931114,
+          -0.005889871623367071,
+          -0.03168580308556557,
+          0.0036342216189950705,
+          0.013947899453341961,
+          0.010371295735239983,
+          -0.015108801424503326,
+          0.0083482526242733,
+          0.007050774060189724,
+          -0.018335426226258278,
+          0.013196727260947227,
+          0.0012217216426506639,
+          0.008826271630823612,
+          0.004387527704238892,
+          0.008006811141967773,
+          0.008510437794029713,
+          0.022159576416015625,
+          0.008783590979874134,
+          0.0063508180901408195,
+          -0.008006811141967773,
+          -0.01912074349820614,
+          0.0026995246298611164,
+          -0.016926979646086693,
+          0.001235592644661665,
+          0.010388367809355259,
+          -0.006141684949398041,
+          0.002078527119010687,
+          0.014357629232108593,
+          0.011113932356238365,
+          0.00018219123012386262,
+          -0.009807917289435863,
+          0.021374259144067764,
+          -0.0026162981521338224,
+          0.002697390504181385,
+          -0.02134011499583721,
+          0.0044515482150018215,
+          0.013649137690663338,
+          -0.00956037174910307,
+          0.012018753215670586,
+          -0.01951340213418007,
+          0.013469880446791649,
+          0.001887533231638372,
+          -0.01213825773447752,
+          -0.001184376422315836,
+          -0.010772489942610264,
+          0.023218045011162758,
+          -0.01574900560081005,
+          0.014084476046264172,
+          0.02490818314254284,
+          -0.002483989344909787,
+          -0.005569770000874996,
+          0.0022684542927891016,
+          0.012727244757115841,
+          -0.014605174772441387,
+          -0.006329478230327368,
+          0.0015738335205242038,
+          0.030354179441928864,
+          -0.009765236638486385,
+          0.002919327700510621,
+          -0.00886895228177309,
+          0.0199231319129467,
+          -0.01315404660999775,
+          -0.006760548334568739,
+          0.005612450186163187,
+          0.02673489600419998,
+          -0.00268245255574584,
+          0.027554357424378395,
+          0.016722114756703377,
+          0.0010526011465117335,
+          0.006171561311930418,
+          -0.01521977037191391,
+          0.006905661430209875,
+          0.017686687409877777,
+          0.01246262714266777,
+          0.017396461218595505,
+          -0.009048208594322205,
+          0.009543299674987793,
+          0.027537284418940544,
+          -0.00713186664506793,
+          0.0038177466485649347,
+          0.012308978475630283,
+          0.011156612075865269,
+          0.003747324226424098,
+          -0.0017541574779897928,
+          0.004259487148374319,
+          -0.015236842446029186,
+          0.013136974535882473,
+          -0.00943233072757721,
+          0.006393498741090298,
+          0.021510835736989975,
+          0.007925719022750854,
+          9.296288044424728e-05,
+          0.0006012577214278281,
+          0.012445555068552494,
+          0.03315400332212448,
+          -0.003975663334131241,
+          0.013768642209470272,
+          -0.0067861564457416534,
+          0.004250951111316681,
+          -0.01007253397256136,
+          -0.012872357852756977,
+          -0.0001271737419301644,
+          -0.014630783349275589,
+          -0.0010915468446910381,
+          7.54239663365297e-05,
+          0.008783590979874134,
+          -0.017618399113416672,
+          0.0015279522631317377,
+          -0.006833104882389307,
+          -0.007187350653111935,
+          -0.011865103617310524,
+          0.01137001346796751,
+          -0.010968819260597229,
+          -0.007332463748753071,
+          0.007281247526407242,
+          0.0076739052310585976,
+          -6.271993333939463e-05,
+          -0.02156205289065838,
+          -0.01059323363006115,
+          0.002249248092994094,
+          0.0036064793821424246,
+          0.014750287868082523,
+          -0.02067430317401886,
+          -0.004289363045245409,
+          0.01002131775021553,
+          -0.009133568964898586,
+          -0.008224480785429478,
+          0.0033717381302267313,
+          -0.01052494440227747,
+          -0.017703760415315628,
+          -0.012078505009412766,
+          0.00831837672740221,
+          -0.019803626462817192,
+          -0.029876161366701126,
+          -0.017635472118854523,
+          0.009329898282885551,
+          -0.016448961570858955,
+          -0.004519836511462927,
+          -0.025352057069540024,
+          0.02181813307106495,
+          0.008489097468554974,
+          0.011173684149980545,
+          0.006811764556914568,
+          -0.006547147408127785,
+          0.012949181720614433,
+          0.0010008513927459717,
+          -0.004289363045245409,
+          0.014050331898033619,
+          0.015424635261297226,
+          -0.010900530964136124,
+          -0.010866386815905571,
+          0.048757895827293396,
+          -0.004455816000699997,
+          0.008809199556708336,
+          -0.0011715723667293787,
+          0.02555692195892334,
+          0.022381512448191643,
+          0.01118222065269947,
+          0.009944493882358074,
+          0.026171516627073288,
+          -0.005813047289848328,
+          0.0008845477714203298,
+          0.018711013719439507,
+          0.002041182015091181,
+          0.028425034135580063,
+          0.016209950670599937,
+          -0.003617149544879794,
+          -0.008412273600697517,
+          -0.0032436975743621588,
+          0.0024818554520606995,
+          0.017353782430291176,
+          -0.003011090215295553,
+          -0.00116517033893615,
+          0.011976072564721107,
+          -0.018796373158693314,
+          -0.02606908418238163,
+          -0.028544537723064423,
+          -0.008096439763903618,
+          -0.003841220634058118,
+          0.0006578090251423419,
+          0.021135250106453896,
+          0.0029641420114785433,
+          0.0074690403416752815,
+          0.01937682554125786,
+          0.005283812526613474,
+          0.0044302078895270824,
+          0.004472888074815273,
+          -0.0020977333188056946,
+          -0.01002131775021553,
+          0.004835669882595539,
+          0.004472888074815273,
+          -0.0012729378649964929,
+          0.01953047327697277,
+          0.008126315660774708,
+          0.008736643008887768,
+          -0.003049502382054925,
+          0.002296196296811104,
+          0.006598363630473614,
+          0.018454931676387787,
+          0.0014372568111866713,
+          -0.026171516627073288,
+          0.008011079393327236,
+          0.009867669083178043,
+          -0.004639341030269861,
+          -0.004054621793329716,
+          0.003964993637055159,
+          -0.003877498907968402,
+          -0.004150652326643467,
+          -0.003719581989571452,
+          0.021203538402915,
+          0.022313224151730537,
+          0.02017921209335327,
+          0.003395212348550558,
+          0.012377266772091389,
+          0.0010584697593003511,
+          -0.02750314027070999,
+          0.007648297119885683,
+          -0.007055042311549187,
+          0.01418690849095583,
+          -0.0359196811914444,
+          -0.00670933211222291,
+          -0.004733237437903881,
+          0.001975027611479163,
+          0.03120778501033783,
+          0.0021478827111423016,
+          0.006564219482243061,
+          0.00552282202988863,
+          0.011959000490605831,
+          -0.004310703370720148,
+          0.024378947913646698,
+          -0.002678184537217021,
+          0.004737505689263344,
+          -0.002415701048448682,
+          -0.01084077823907137,
+          -0.020110923796892166,
+          0.01110539585351944,
+          0.008689695037901402,
+          0.0616643987596035,
+          -0.006513003259897232,
+          0.023047324270009995,
+          -0.01373449806123972,
+          0.007234299089759588,
+          0.0218522772192955,
+          0.024447236210107803,
+          0.02748606912791729,
+          -0.0010691398056223989,
+          0.004707629326730967,
+          -0.014647855423390865,
+          -0.018523219972848892,
+          0.009585979394614697,
+          -0.007242835126817226,
+          0.00597523245960474,
+          -0.022415656596422195,
+          -0.02016214095056057,
+          -0.008988456800580025,
+          -0.0057532950304448605,
+          -0.001239860663190484,
+          0.004618000704795122,
+          0.005309420637786388,
+          -0.0034549646079540253,
+          0.025864219292998314,
+          -0.0005332361324690282,
+          -0.0033141199965029955,
+          -0.028595754876732826,
+          -0.009184785187244415,
+          0.011890712194144726,
+          -0.05049924924969673,
+          0.018779302015900612,
+          -0.00272940075956285,
+          -0.02748606912791729,
+          -0.01744767837226391,
+          -0.0035595311783254147,
+          0.015629500150680542,
+          -0.012334587052464485,
+          0.010610305704176426,
+          0.0026205661706626415,
+          -0.0006882187444716692,
+          0.0037772003561258316,
+          0.014545422978699207,
+          0.0224839448928833,
+          0.004370455630123615,
+          0.009329898282885551,
+          0.005556966178119183,
+          0.009978638030588627,
+          -0.007580008823424578,
+          0.019974347203969955,
+          0.001809641718864441,
+          0.0045326403342187405,
+          0.0020337130408734083,
+          -0.016961123794317245,
+          -0.004536908585578203,
+          -0.006017912644892931,
+          0.028083590790629387,
+          0.015339274890720844,
+          -0.01041397638618946,
+          -0.005347833037376404,
+          -0.005569770000874996,
+          -0.011924856342375278,
+          0.0013028139946982265,
+          0.013512561097741127,
+          0.010448120534420013,
+          -0.038856081664562225,
+          0.03317107632756233,
+          0.01728549413383007,
+          -0.01848907582461834,
+          -0.024208227172493935,
+          -0.002460515359416604,
+          0.020708447322249413,
+          0.023832641541957855,
+          -0.03271012753248215,
+          -0.013427200727164745,
+          0.014468598179519176,
+          -0.014383237808942795,
+          6.825502350693569e-05,
+          -0.0005495079676620662,
+          -0.0205547995865345,
+          0.011250508949160576,
+          -0.014810039661824703,
+          0.019291464239358902,
+          -0.0025928239338099957,
+          0.022518089041113853,
+          0.025727642700076103,
+          -0.004528372548520565,
+          0.0018352499464526772,
+          -0.01355524081736803,
+          -0.008207408711314201,
+          0.03277841582894325,
+          0.0018053737003356218,
+          -0.005428925156593323,
+          -0.007917182520031929,
+          -0.0023986289743334055,
+          -0.012676028534770012,
+          0.01490393653512001,
+          0.007349535822868347,
+          0.0012035825056955218,
+          0.005241132341325283,
+          -0.0010878123575821519,
+          0.025215480476617813,
+          0.004763113800436258,
+          -0.004763113800436258,
+          0.008834808133542538,
+          -0.015108801424503326,
+          0.01937682554125786,
+          -0.0029022556263953447,
+          -0.008361057378351688,
+          0.0053179566748440266,
+          0.014656390994787216,
+          0.004792989697307348,
+          -0.0038518907967954874,
+          -0.010482264682650566,
+          -0.004857010208070278,
+          0.007669637445360422,
+          0.027059266343712807,
+          0.014519814401865005,
+          -0.007597080897539854,
+          -0.011412693187594414,
+          0.01451127789914608,
+          0.007883038371801376,
+          0.0006924867630004883,
+          -0.0071788146160542965,
+          0.012095577083528042,
+          -0.01554413978010416,
+          0.014681999571621418,
+          -0.009346970357000828,
+          -0.020862096920609474,
+          0.008988456800580025,
+          -0.02506183087825775,
+          0.0017125442391261458,
+          -0.01835249923169613,
+          0.002155351685360074,
+          -0.007968398742377758,
+          -0.011395621113479137,
+          0.009500619024038315,
+          -0.000326237000990659,
+          0.03878779336810112,
+          0.011190756224095821,
+          0.009679876267910004,
+          -0.0014489938039332628,
+          0.07416117191314697,
+          -0.023439982905983925,
+          -0.01033715158700943,
+          -0.028954267501831055,
+          -0.012949181720614433,
+          -0.0063636223785579205,
+          0.01002131775021553,
+          0.008104975335299969,
+          -0.0004057289334014058,
+          -0.011301725171506405,
+          -0.007072114385664463,
+          0.015783149749040604,
+          -0.026615392416715622,
+          -0.0007479710620827973,
+          0.02159619703888893,
+          0.0030729766003787518,
+          0.006564219482243061,
+          0.015006368979811668,
+          -0.007268443237990141,
+          0.01966704986989498,
+          -0.016346529126167297,
+          -0.012744316831231117,
+          0.005894139874726534,
+          0.016329456120729446,
+          0.005339297000318766,
+          0.0008904163260012865,
+          0.011848031543195248,
+          -0.0006097938166931272,
+          0.005108823534101248,
+          0.02323511801660061,
+          -0.02014506794512272,
+          0.014417381957173347,
+          -0.008412273600697517,
+          0.01825006678700447,
+          0.007644029334187508,
+          0.018028128892183304,
+          -0.005134431645274162,
+          -0.008877487853169441,
+          -0.008442149497568607,
+          0.010943210683763027,
+          0.015962406992912292,
+          -0.012215081602334976,
+          -0.0019686257001012564,
+          0.004048219881951809,
+          -0.018028128892183304,
+          -0.006897125393152237,
+          -0.012180937454104424,
+          0.008049491792917252,
+          0.021510835736989975,
+          -0.012471163645386696,
+          -0.01675625890493393,
+          -0.022176647558808327,
+          0.012317514047026634,
+          -0.008190336637198925,
+          0.02772507816553116,
+          -0.013623529113829136,
+          -0.00523686408996582,
+          0.0211864672601223,
+          0.0004182662523817271,
+          0.0012174536241218448,
+          -0.027776293456554413,
+          -0.010166430845856667,
+          -0.014997833408415318
+        ],
+        "title": "6 Related Work"
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "be3079d3-622e-4848-bff6-15e3ac00f52b",
+      "properties": {
+        "page_content": "7 Discussion 25 ",
+        "keyphrases": [
+          "Discussion"
+        ],
+        "title": "7 Discussion",
+        "embedding": [
+          -0.015314480289816856,
+          0.01669096015393734,
+          -0.018997248262166977,
+          0.02089332602918148,
+          -0.021677281707525253,
+          -0.005907013546675444,
+          -0.010774830356240273,
+          0.0937100350856781,
+          -0.00431403424590826,
+          0.05294434353709221,
+          0.0064402855932712555,
+          -0.027529601007699966,
+          -0.014475829899311066,
+          -0.01842295378446579,
+          -0.024448473006486893,
+          0.03755693882703781,
+          -0.012051037512719631,
+          0.033710088580846786,
+          -0.006408380344510078,
+          -0.030100245028734207,
+          0.01741110533475876,
+          -0.012816760689020157,
+          0.011850490234792233,
+          0.07365535199642181,
+          -0.013801263645291328,
+          0.025287123396992683,
+          0.04072922468185425,
+          0.02200544998049736,
+          -0.03755693882703781,
+          0.04565173387527466,
+          -0.010009106248617172,
+          0.004895164165645838,
+          -0.0015485400799661875,
+          -0.04878755658864975,
+          -0.024029148742556572,
+          0.00932086631655693,
+          0.03347307816147804,
+          -0.010191421955823898,
+          0.033710088580846786,
+          0.02512303926050663,
+          -0.030592497438192368,
+          -0.039453018456697464,
+          -0.0070100207813084126,
+          0.0217137448489666,
+          0.010382852517068386,
+          -0.006080213002860546,
+          -0.025505902245640755,
+          -0.018559690564870834,
+          0.0007360977469943464,
+          0.012707372196018696,
+          -0.016827696934342384,
+          -0.04269822686910629,
+          0.01159524917602539,
+          0.0005062096752226353,
+          0.014184125699102879,
+          -0.00020211352966725826,
+          0.011440280824899673,
+          0.034111179411411285,
+          0.00420920317992568,
+          -0.02865995466709137,
+          0.003798993770033121,
+          -0.044375527650117874,
+          -0.004564717877656221,
+          -0.004081582184880972,
+          -0.016408370807766914,
+          -0.02123972587287426,
+          -0.013500443659722805,
+          0.01520509086549282,
+          -0.02880580723285675,
+          -0.021130336448550224,
+          -0.014421135187149048,
+          -0.008003639057278633,
+          -0.0213126502931118,
+          0.04120324179530144,
+          -0.017447568476200104,
+          -0.012944381684064865,
+          0.03330899402499199,
+          0.01408385206013918,
+          -0.019799435511231422,
+          0.044448453933000565,
+          0.02428438887000084,
+          -0.019070174545049667,
+          -0.025378281250596046,
+          0.030829505994915962,
+          0.029790310189127922,
+          -0.01874200627207756,
+          -0.06716493517160416,
+          -0.11777564138174057,
+          0.026891497895121574,
+          0.022114837542176247,
+          0.006271644029766321,
+          -0.012962613254785538,
+          -0.012224236503243446,
+          -0.0341658741235733,
+          -0.009398350492119789,
+          -0.00598905561491847,
+          -0.01561530027538538,
+          -0.0026207815390080214,
+          0.023099340498447418,
+          -0.012780298478901386,
+          0.03454873710870743,
+          0.0032497690990567207,
+          0.0071695465594530106,
+          0.024685483425855637,
+          0.022461237385869026,
+          -0.00037574031739495695,
+          0.02880580723285675,
+          0.016700075939297676,
+          -0.0015508190263062716,
+          0.007880575954914093,
+          -0.024612557142972946,
+          0.022953487932682037,
+          -0.030009087175130844,
+          -0.010428431443870068,
+          0.00737465126439929,
+          -0.015505910851061344,
+          0.006800358183681965,
+          -0.04754781350493431,
+          -0.017155863344669342,
+          0.016453949734568596,
+          -0.0017046474386006594,
+          -0.005405646748840809,
+          -0.03053780272603035,
+          -0.02102094702422619,
+          -0.027183201164007187,
+          0.006253412459045649,
+          4.162627010373399e-05,
+          -0.011121229268610477,
+          -0.005282584112137556,
+          0.016435718163847923,
+          0.006668179761618376,
+          0.04557880759239197,
+          -0.02499541826546192,
+          0.029188670217990875,
+          0.002563808113336563,
+          -0.005733814090490341,
+          0.002456697868183255,
+          0.02665448747575283,
+          -0.008040102198719978,
+          -0.01068367250263691,
+          -0.02235184796154499,
+          -0.003769367467612028,
+          0.006385591346770525,
+          0.02900635451078415,
+          -0.01597081497311592,
+          0.023901527747511864,
+          -0.0013240644475445151,
+          0.021640818566083908,
+          0.006157697178423405,
+          0.021549660712480545,
+          0.0255423653870821,
+          -0.012470361776649952,
+          -0.0430263951420784,
+          0.03489513695240021,
+          0.006426611915230751,
+          -0.010510473512113094,
+          -0.009380118921399117,
+          0.014931618236005306,
+          -0.012133078649640083,
+          0.010865988209843636,
+          0.015414752997457981,
+          0.06450313329696655,
+          -0.022096605971455574,
+          -0.0035095682833343744,
+          0.04204189404845238,
+          -0.01353690680116415,
+          0.02561529166996479,
+          -0.008409290574491024,
+          -0.04204189404845238,
+          0.019544193521142006,
+          0.04448491707444191,
+          0.008309016935527325,
+          0.004799448419362307,
+          -0.012333625927567482,
+          0.008514121174812317,
+          0.02242477424442768,
+          -0.003452594857662916,
+          -0.0010278021218255162,
+          -0.014074736274778843,
+          0.023281656205654144,
+          -0.04393797367811203,
+          0.003309021471068263,
+          0.012762066908180714,
+          -0.0032725585624575615,
+          -0.034311726689338684,
+          -0.01408385206013918,
+          0.04587051272392273,
+          -0.02650863490998745,
+          0.04572466015815735,
+          0.010182306170463562,
+          -0.0075751980766654015,
+          -0.002725612837821245,
+          -0.007570640183985233,
+          -0.01061986293643713,
+          -0.014995427802205086,
+          -0.06351862847805023,
+          0.01214219443500042,
+          0.011686407029628754,
+          0.0002243331982754171,
+          0.01436644047498703,
+          -0.021677281707525253,
+          -0.013646295294165611,
+          -0.00037232189788483083,
+          0.007383767049759626,
+          -0.011987227015197277,
+          -0.0016602081013843417,
+          0.010364620946347713,
+          0.022734710946679115,
+          0.03617134317755699,
+          -0.024867799133062363,
+          0.010701904073357582,
+          0.02435731515288353,
+          -0.022151300683617592,
+          0.04054690897464752,
+          0.01696443371474743,
+          0.008126701228320599,
+          -0.019197793677449226,
+          0.06319046020507812,
+          0.01896078512072563,
+          -0.040218740701675415,
+          0.025432975962758064,
+          -0.000753759581129998,
+          0.01708293706178665,
+          0.026745645329356194,
+          -0.012634445913136005,
+          0.026071079075336456,
+          -0.02359159104526043,
+          0.021549660712480545,
+          -0.01819506101310253,
+          -0.052907880395650864,
+          0.02603461593389511,
+          0.014566987752914429,
+          -0.01453964039683342,
+          -0.010100264102220535,
+          -0.041677262634038925,
+          -0.026198700070381165,
+          -0.025761142373085022,
+          0.0063126650638878345,
+          0.024047380313277245,
+          0.03068365529179573,
+          0.029735615476965904,
+          -0.010109379887580872,
+          -0.040984466671943665,
+          0.010364620946347713,
+          -0.054803960025310516,
+          0.020783936604857445,
+          -0.025141270831227303,
+          0.029553299769759178,
+          0.0002243331982754171,
+          -0.010920682922005653,
+          0.020072907209396362,
+          -0.007365535479038954,
+          0.012406552210450172,
+          0.029808541759848595,
+          -0.008204185403883457,
+          0.045979902148246765,
+          0.04481308534741402,
+          -0.0034844998735934496,
+          -0.0023837718181312084,
+          0.004040561616420746,
+          -0.018204176798462868,
+          -0.010437547229230404,
+          -0.032379187643527985,
+          -0.014074736274778843,
+          0.02138557657599449,
+          -0.01175933238118887,
+          0.021458502858877182,
+          0.004427981097251177,
+          0.026472171768546104,
+          -0.009024604223668575,
+          0.02568821609020233,
+          -0.01901547983288765,
+          0.010391968302428722,
+          0.005587961990386248,
+          -0.027092043310403824,
+          -0.014402903616428375,
+          0.01896078512072563,
+          -0.005309931468218565,
+          0.004931627307087183,
+          0.032999057322740555,
+          0.00916134100407362,
+          0.03061072900891304,
+          -0.0073883249424397945,
+          -0.025943458080291748,
+          -0.010556052438914776,
+          -0.035223305225372314,
+          -0.016052857041358948,
+          0.024813104420900345,
+          -0.012716487981379032,
+          -0.0217137448489666,
+          0.009999990463256836,
+          -0.005027342587709427,
+          -0.0037306256126612425,
+          0.017228789627552032,
+          0.011431165039539337,
+          -0.009234266355633736,
+          0.021330881863832474,
+          0.005095710977911949,
+          -0.0022117116022855043,
+          0.005628983024507761,
+          -0.009735633619129658,
+          0.021257957443594933,
+          -0.024667251855134964,
+          0.001667044940404594,
+          -0.009516855701804161,
+          0.01627163402736187,
+          0.005911571439355612,
+          0.009343655779957771,
+          -0.0852506011724472,
+          -0.020236991345882416,
+          0.06257058680057526,
+          -0.03839558735489845,
+          -0.009361887350678444,
+          0.009972643107175827,
+          -0.017000894993543625,
+          0.019252488389611244,
+          0.02497718669474125,
+          0.013710105791687965,
+          -0.01644483394920826,
+          0.0024361873511224985,
+          0.02082039974629879,
+          0.005670004058629274,
+          -0.029735615476965904,
+          -0.004749312065541744,
+          0.0068185897544026375,
+          0.002308566588908434,
+          -0.024047380313277245,
+          0.027092043310403824,
+          0.006882400251924992,
+          0.0013126697158440948,
+          0.023792138323187828,
+          -0.03234272450208664,
+          0.005856877192854881,
+          -0.004876932594925165,
+          -0.00793527066707611,
+          0.004968089982867241,
+          -0.041677262634038925,
+          0.03624426946043968,
+          0.002105741063132882,
+          -0.030993590131402016,
+          0.01408385206013918,
+          0.009817675687372684,
+          0.011777563951909542,
+          0.02220599539577961,
+          -0.012579751200973988,
+          -0.006855052895843983,
+          0.01461256667971611,
+          -0.023281656205654144,
+          4.2480874981265515e-05,
+          0.010364620946347713,
+          0.000994757516309619,
+          0.006595253478735685,
+          -0.013582484796643257,
+          0.014949849806725979,
+          -0.0014881481183692813,
+          -0.035843174904584885,
+          -0.05560614541172981,
+          -0.014448482543230057,
+          0.0003068023652303964,
+          9.008936467580497e-05,
+          0.04477662220597267,
+          -0.013381938450038433,
+          -0.00829534325748682,
+          0.0055834040977060795,
+          -0.029024586081504822,
+          -0.07099355012178421,
+          0.035369157791137695,
+          0.022661784663796425,
+          -0.024940723553299904,
+          0.014758418314158916,
+          -0.03228802978992462,
+          -0.02789423055946827,
+          0.01589788869023323,
+          -0.020492233335971832,
+          -0.017119400203227997,
+          0.0018675917526707053,
+          -0.007629892788827419,
+          -0.005888781975954771,
+          -0.03201455622911453,
+          0.026417477056384087,
+          -0.02269824780523777,
+          0.01121238712221384,
+          -0.040911540389060974,
+          -0.013281664811074734,
+          0.005510477814823389,
+          -2.746301333900192e-06,
+          0.01567910984158516,
+          0.026289857923984528,
+          -0.02116679958999157,
+          0.00796261802315712,
+          -0.04754781350493431,
+          -0.012379204854369164,
+          -0.006859610788524151,
+          -0.017265252768993378,
+          0.03108474798500538,
+          0.006203276105225086,
+          0.01651776023209095,
+          -0.00870555266737938,
+          0.036262501031160355,
+          -0.030446644872426987,
+          -0.00527346832677722,
+          0.018577922135591507,
+          -0.006581580266356468,
+          -0.017401989549398422,
+          0.04196896776556969,
+          0.00760254543274641,
+          -0.019598888233304024,
+          -0.0020943463314324617,
+          -0.005838645622134209,
+          0.0072287991642951965,
+          0.009726517833769321,
+          -0.009917949326336384,
+          0.036116648465394974,
+          -0.011185039766132832,
+          0.06468544900417328,
+          0.0025205081328749657,
+          0.04565173387527466,
+          0.05367360636591911,
+          -0.02971738390624523,
+          -0.01061986293643713,
+          0.007739281747490168,
+          0.0238103698939085,
+          0.026818571612238884,
+          -0.017219673842191696,
+          -0.025232428684830666,
+          0.03615311160683632,
+          -0.015788499265909195,
+          -0.0039106616750359535,
+          0.010319042019546032,
+          -0.030556034296751022,
+          -0.028714649379253387,
+          0.04200543090701103,
+          -0.020528696477413177,
+          -0.007775744888931513,
+          0.018632616847753525,
+          -0.01936187781393528,
+          0.005574288312345743,
+          0.008422964252531528,
+          -0.03174108266830444,
+          0.024029148742556572,
+          0.010154958814382553,
+          -0.005246120970696211,
+          -0.015569721348583698,
+          0.046052828431129456,
+          -0.017593421041965485,
+          0.0016066529788076878,
+          -0.031412914395332336,
+          -0.021203262731432915,
+          -0.04477662220597267,
+          0.035843174904584885,
+          -0.03547854349017143,
+          0.029899699613451958,
+          -0.038213275372982025,
+          0.010574284009635448,
+          0.036061953753232956,
+          -0.00958066526800394,
+          0.0019553308375179768,
+          0.018924321979284286,
+          -0.020510464906692505,
+          -0.06891515851020813,
+          0.0106563251465559,
+          0.022734710946679115,
+          0.021002715453505516,
+          -0.015405637212097645,
+          -0.0364995114505291,
+          0.03330899402499199,
+          -0.0038628040347248316,
+          -0.03163169324398041,
+          0.005783950909972191,
+          0.018796700984239578,
+          -0.015834078192710876,
+          -0.010382852517068386,
+          -0.05425701290369034,
+          -0.005382857285439968,
+          -0.0376298651099205,
+          -0.024047380313277245,
+          0.0347675159573555,
+          -0.013117581605911255,
+          -0.04510479047894478,
+          0.03901546075940132,
+          -0.033837709575891495,
+          0.011996342800557613,
+          -0.0033477635588496923,
+          0.022224226966500282,
+          -0.02242477424442768,
+          0.015587952919304371,
+          0.042370062321424484,
+          -0.041458483785390854,
+          0.004712848924100399,
+          0.039380092173814774,
+          0.03296259418129921,
+          -0.01585230976343155,
+          0.06618043035268784,
+          -0.005820414051413536,
+          -0.013728337362408638,
+          -0.06771188229322433,
+          -0.00018388200260233134,
+          -0.07817677408456802,
+          -0.001375340623781085,
+          0.03365539386868477,
+          -0.008901541121304035,
+          -0.019325414672493935,
+          -0.033071983605623245,
+          0.03312667831778526,
+          0.009206918999552727,
+          -0.025177733972668648,
+          0.011713754385709763,
+          -0.04579758644104004,
+          -0.010802177712321281,
+          0.0192160252481699,
+          0.027985388413071632,
+          0.005551498848944902,
+          0.00091556430561468,
+          -0.012105731293559074,
+          -0.019197793677449226,
+          -0.007543292827904224,
+          -0.016645381227135658,
+          0.03132175654172897,
+          -0.004416586831212044,
+          0.012652677483856678,
+          0.0019040547776967287,
+          -0.007648124359548092,
+          -0.03387417271733284,
+          0.04324517399072647,
+          -0.01773015595972538,
+          0.012570635415613651,
+          0.013181391172111034,
+          -0.008810384199023247,
+          -0.0016602081013843417,
+          0.011257966049015522,
+          0.0009468997595831752,
+          0.004343660548329353,
+          0.006030076649039984,
+          0.022187763825058937,
+          0.031212368980050087,
+          -0.0015850031049922109,
+          0.0024908818304538727,
+          0.01916133053600788,
+          0.007939828559756279,
+          0.026490403339266777,
+          -0.0023814928717911243,
+          0.024193231016397476,
+          -0.030628960579633713,
+          -0.028295325115323067,
+          0.007643566466867924,
+          0.01916133053600788,
+          -0.00929807685315609,
+          -0.00896535161882639,
+          -0.015861425548791885,
+          -0.02915220707654953,
+          0.02386506460607052,
+          -0.0023792139254510403,
+          0.008290785364806652,
+          -0.01992705464363098,
+          0.0021137173753231764,
+          -0.0013696432579308748,
+          0.0315040722489357,
+          0.019416572526097298,
+          0.013728337362408638,
+          -0.004619412124156952,
+          0.011084766127169132,
+          0.0028988122940063477,
+          -0.027164969593286514,
+          0.0028646280989050865,
+          -0.011048302985727787,
+          -0.007023694459348917,
+          -0.014348208904266357,
+          0.043682731688022614,
+          0.004453049506992102,
+          -0.025031881406903267,
+          0.009571549482643604,
+          0.00199065450578928,
+          -0.01679123379290104,
+          -0.009097530506551266,
+          -0.0017673183465376496,
+          0.033965326845645905,
+          0.027875998988747597,
+          -0.028204167261719704,
+          -0.02499541826546192,
+          0.006207833997905254,
+          0.013783032074570656,
+          0.002743844408541918,
+          0.009872370399534702,
+          -0.004425702150911093,
+          -5.647499347105622e-05,
+          -0.01207838486880064,
+          -0.009389234706759453,
+          -0.02025522291660309,
+          -0.009617128409445286,
+          0.020802168175578117,
+          0.025998152792453766,
+          0.020364612340927124,
+          0.031066516414284706,
+          -0.04634453356266022,
+          -0.010957146063446999,
+          0.016453949734568596,
+          0.04856877774000168,
+          0.025159502401947975,
+          -0.006408380344510078,
+          -0.03290789946913719,
+          -0.008787594735622406,
+          0.010893335565924644,
+          -0.010200537741184235,
+          0.012461245991289616,
+          -0.01731083169579506,
+          0.004026887938380241,
+          -0.03194162994623184,
+          0.013427517376840115,
+          -0.0015154953580349684,
+          0.02700088731944561,
+          -0.02020052820444107,
+          0.012169541791081429,
+          0.01894255355000496,
+          -0.02957153134047985,
+          -0.011987227015197277,
+          -0.0315040722489357,
+          -0.003691883524879813,
+          0.001616908237338066,
+          -0.01589788869023323,
+          0.0061440235003829,
+          0.014402903616428375,
+          -0.010492241941392422,
+          0.0011018677614629269,
+          0.024466704577207565,
+          0.019106637686491013,
+          0.0077939764596521854,
+          -0.00453053368255496,
+          0.0063400124199688435,
+          -0.009361887350678444,
+          -0.028623491525650024,
+          -0.03219687193632126,
+          0.018331797793507576,
+          0.02180490270256996,
+          0.003411573823541403,
+          -0.006175928749144077,
+          0.005095710977911949,
+          0.057283446192741394,
+          -0.026891497895121574,
+          0.018833164125680923,
+          0.0016043740324676037,
+          -0.03901546075940132,
+          -0.031832240521907806,
+          -0.003343205666169524,
+          0.06089328974485397,
+          -0.028222398832440376,
+          0.01121238712221384,
+          0.015606184490025043,
+          0.01817682944238186,
+          0.002261848421767354,
+          0.0003395621315576136,
+          -0.034184105694293976,
+          0.014329977333545685,
+          -0.01207838486880064,
+          0.01894255355000496,
+          0.031394682824611664,
+          -0.04072922468185425,
+          -0.004804006312042475,
+          -0.021950755268335342,
+          -0.0012830435298383236,
+          -0.024029148742556572,
+          -0.024175001308321953,
+          0.00619416031986475,
+          0.00550136249512434,
+          0.0071695465594530106,
+          -0.0047265226021409035,
+          -0.006444843485951424,
+          -0.0015838636318221688,
+          -0.00231198500841856,
+          0.00924794003367424,
+          -0.007014578673988581,
+          0.0008050356991589069,
+          -0.01159524917602539,
+          0.01718321070075035,
+          0.0007492016884498298,
+          -0.03832266107201576,
+          -0.013491327874362469,
+          -0.020036444067955017,
+          0.011950763873755932,
+          -0.0008283948409371078,
+          -0.01908840611577034,
+          -0.020036444067955017,
+          -0.011340008117258549,
+          0.010948030278086662,
+          0.01696443371474743,
+          -0.02123972587287426,
+          0.0044781179167330265,
+          0.008669089525938034,
+          -0.0035095682833343744,
+          -0.007680029142647982,
+          -0.0018493601819500327,
+          0.0215131975710392,
+          0.04940742999315262,
+          0.004020051099359989,
+          0.006016402970999479,
+          -0.007771186996251345,
+          -0.017338179051876068,
+          -0.04102092981338501,
+          0.03923423960804939,
+          0.01339105423539877,
+          0.029735615476965904,
+          -0.00248176627792418,
+          -0.04058337211608887,
+          -0.0016624870477244258,
+          -0.01783042959868908,
+          0.004172740038484335,
+          -0.0034298053942620754,
+          -0.008637184277176857,
+          0.004475838970392942,
+          -0.017319947481155396,
+          -0.0326344259083271,
+          -0.016873275861144066,
+          -0.029334520921111107,
+          0.0009634220623411238,
+          0.022661784663796425,
+          -0.021604355424642563,
+          0.009361887350678444,
+          -0.0025979920756071806,
+          -0.039963498711586,
+          -0.007433903869241476,
+          -0.003502731444314122,
+          -0.01398357842117548,
+          0.013719221577048302,
+          -0.014220588840544224,
+          0.0023245192132890224,
+          -0.005054689943790436,
+          0.01467637624591589,
+          0.025232428684830666,
+          -0.02138557657599449,
+          0.005373741500079632,
+          -0.0001051873477990739,
+          -0.00041220334242098033,
+          -0.013609832152724266,
+          0.0005908153252676129,
+          -0.0012819040566682816,
+          -0.020109370350837708,
+          0.018131250515580177,
+          0.004927069414407015,
+          0.02776661142706871,
+          0.016636265441775322,
+          -0.03312667831778526,
+          0.010072916746139526,
+          -0.002771191531792283,
+          -0.01696443371474743,
+          0.015496795065701008,
+          -0.012898802757263184,
+          0.0008266856311820447,
+          0.02422969415783882,
+          0.017620768398046494,
+          -0.018377376720309258,
+          0.003115311497822404,
+          0.00978121254593134,
+          -0.04247945174574852,
+          0.062351811677217484,
+          -0.00193937832955271,
+          -0.03347307816147804,
+          0.016499528661370277,
+          0.018304450437426567,
+          0.01673653908073902,
+          -0.0045669968239963055,
+          0.00888786744326353,
+          -0.002016862388700247,
+          0.03555146977305412,
+          0.012944381684064865,
+          0.04776659235358238,
+          0.029407447203993797,
+          -0.014566987752914429,
+          -0.006458517163991928,
+          -0.01819506101310253,
+          -0.00801275484263897,
+          0.01683681271970272,
+          0.008960793726146221,
+          0.01742933690547943,
+          0.021458502858877182,
+          -0.013546021655201912,
+          0.015861425548791885,
+          0.005382857285439968,
+          0.00898814108222723,
+          -0.05330897495150566,
+          -0.0430263951420784,
+          0.007543292827904224,
+          -0.01565176248550415,
+          -0.033071983605623245,
+          -0.005929803010076284,
+          -0.004872374702244997,
+          -0.022607089951634407,
+          0.0025205081328749657,
+          0.041057392954826355,
+          -0.024175001308321953,
+          0.027711916714906693,
+          0.007306283339858055,
+          0.006303549278527498,
+          0.0255423653870821,
+          -0.02200544998049736,
+          -0.00387875665910542,
+          0.043974436819553375,
+          -0.04933450371026993,
+          0.009516855701804161,
+          -0.0050455741584300995,
+          -0.009197804145514965,
+          -0.015050122514367104,
+          0.0315040722489357,
+          0.009653591550886631,
+          -0.0064721908420324326,
+          0.015761151909828186,
+          0.01950773037970066,
+          0.0006551953847520053,
+          0.016326328739523888,
+          -0.020528696477413177,
+          -0.034238800406455994,
+          0.030027318745851517,
+          0.0009104367345571518,
+          -0.008144932799041271,
+          0.0414949469268322,
+          0.009945295751094818,
+          0.010975377634167671,
+          0.0034252475015819073,
+          -0.007392882835119963,
+          -0.026125773787498474,
+          0.00699178921058774,
+          0.023226961493492126,
+          0.025925226509571075,
+          0.011768448166549206,
+          0.020419307053089142,
+          -0.006025518756359816,
+          0.008286227472126484,
+          0.04732903465628624,
+          0.008760247379541397,
+          0.006900631822645664,
+          0.042588841170072556,
+          0.035369157791137695,
+          -0.02484956756234169,
+          -0.020984483882784843,
+          -0.00975386518985033,
+          0.022807635366916656,
+          0.0006956465658731759,
+          0.02054692804813385,
+          0.011221502907574177,
+          -0.0035665417090058327,
+          -0.005460341461002827,
+          0.030501339584589005,
+          -0.014393787831068039,
+          0.002228803699836135,
+          -0.0006791242631152272,
+          -0.017392873764038086,
+          -0.003037827555090189,
+          0.004402913153171539,
+          -0.020437538623809814,
+          0.009279845282435417,
+          0.02422969415783882,
+          0.00045806702109985054,
+          0.018979016691446304,
+          0.019124869257211685,
+          0.006358243990689516,
+          -0.0029717383440583944,
+          -0.024302620440721512,
+          -0.05061070993542671,
+          0.021257957443594933,
+          0.019343646243214607,
+          0.0026891496963799,
+          0.0029238807037472725,
+          0.012944381684064865,
+          0.024685483425855637,
+          -0.00985413882881403,
+          -0.02032814919948578,
+          0.03773925453424454,
+          -0.02319049835205078,
+          -0.008824057877063751,
+          -0.03159523010253906,
+          0.006791242398321629,
+          -0.023500433191657066,
+          -0.01207838486880064,
+          -0.020164065062999725,
+          -0.0066317166201770306,
+          0.0058340877294540405,
+          0.002695986535400152,
+          -0.0036235153675079346,
+          -0.012552403844892979,
+          0.001857336494140327,
+          0.004516860004514456,
+          -0.031759314239025116,
+          -0.02123972587287426,
+          -0.027456674724817276,
+          -0.0253418181091547,
+          -0.006207833997905254,
+          -0.02242477424442768,
+          0.003976751118898392,
+          -0.01965358294546604,
+          -0.046818554401397705,
+          -0.008546026423573494,
+          -0.010583399794995785,
+          0.0036736519541591406,
+          -0.006367359776049852,
+          0.014958965592086315,
+          -0.03157699853181839,
+          0.0030606170184910297,
+          0.003632631152868271,
+          -0.032105714082717896,
+          0.011896069161593914,
+          -0.011750216595828533,
+          -0.005565172526985407,
+          -0.00603919243440032,
+          0.0063400124199688435,
+          0.019471267238259315,
+          -0.032105714082717896,
+          -0.010391968302428722,
+          -0.028550565242767334,
+          0.024594325572252274,
+          0.004010935313999653,
+          0.014557871967554092,
+          0.006125791929662228,
+          -0.027365516871213913,
+          -0.02275294065475464,
+          -0.03713761270046234,
+          -0.02816770412027836,
+          0.010975377634167671,
+          0.02282586693763733,
+          -0.0024133978877216578,
+          0.016390139237046242,
+          -0.0016271634958684444,
+          0.024138538166880608,
+          0.010273464024066925,
+          -0.0008152909576892853,
+          -0.029024586081504822,
+          0.009644475765526295,
+          0.00855514220893383,
+          -0.04350041598081589,
+          -0.030009087175130844,
+          0.023026414215564728,
+          -0.013400170020759106,
+          0.009945295751094818,
+          0.021494966000318527,
+          0.017839545384049416,
+          0.026526866480708122,
+          0.006941652856767178,
+          0.027748379856348038,
+          -0.0115223228931427,
+          -0.016973547637462616,
+          0.031194137409329414,
+          0.015551489777863026,
+          -0.012725603766739368,
+          0.02712850645184517,
+          0.01391065213829279,
+          0.0023792139254510403,
+          0.008318132720887661,
+          0.021494966000318527,
+          0.005305373575538397,
+          0.0013343195896595716,
+          0.04977206140756607,
+          0.01817682944238186,
+          -0.008842289447784424,
+          -0.019525961950421333,
+          0.007429345976561308,
+          0.00982679147273302,
+          -0.007338188122957945,
+          0.021987218409776688,
+          0.00015268901188392192,
+          -0.04503186419606209,
+          0.01824975572526455,
+          0.019124869257211685,
+          0.024065611883997917,
+          0.0017057869117707014,
+          0.008277111686766148,
+          -0.027930693700909615,
+          0.00732451444491744,
+          -0.01023700088262558,
+          0.01985413022339344,
+          0.008381943218410015,
+          0.03226979821920395,
+          -0.03886960819363594,
+          0.020929789170622826,
+          -0.001477892859838903,
+          0.002021420281380415,
+          0.02601638436317444,
+          -0.020109370350837708,
+          -0.029735615476965904,
+          0.0013912931317463517,
+          -0.028678186237812042,
+          -0.011896069161593914,
+          0.004143113736063242,
+          0.015688225626945496,
+          0.004525975789874792,
+          -0.013719221577048302,
+          0.01617136225104332,
+          -0.008117586374282837,
+          -0.01589788869023323,
+          -0.004799448419362307,
+          -0.0009816535748541355,
+          0.043208710849285126,
+          0.016882391646504402,
+          0.013327243737876415,
+          -0.024466704577207565,
+          -0.03828619793057442,
+          -0.0034753840882331133,
+          -0.00029070733580738306,
+          0.03573378548026085,
+          0.007005462888628244,
+          0.007055599708110094,
+          -0.012233352288603783,
+          -0.011804911307990551,
+          0.0008768223342485726,
+          -0.016262518242001534,
+          0.0013081118231639266,
+          -0.011431165039539337,
+          -0.011494975537061691,
+          -0.004047398455440998,
+          0.004104371648281813,
+          0.010255232453346252,
+          0.013974462635815144,
+          0.012935265898704529,
+          0.006540559232234955,
+          -0.0032953477930277586,
+          -0.012689140625298023,
+          -0.0116225965321064,
+          -0.007488598581403494,
+          0.006271644029766321,
+          0.031066516414284706,
+          0.0032657217234373093,
+          -0.007023694459348917,
+          0.0047265226021409035,
+          0.032999057322740555,
+          -0.011504091322422028,
+          0.001646534539759159,
+          0.017474915832281113,
+          -0.02497718669474125,
+          0.016973547637462616,
+          -0.0008699854952283204,
+          0.015150396153330803,
+          0.006139465607702732,
+          0.015013659372925758,
+          -0.013518675230443478,
+          0.0009047393687069416,
+          0.011886953376233578,
+          0.008076565340161324,
+          -0.04886048287153244,
+          -0.00996352732181549,
+          0.009170456789433956,
+          -0.02173197641968727,
+          -0.01505923829972744,
+          -0.002342750784009695,
+          -0.0019188678124919534,
+          0.013345475308597088,
+          -0.017292600125074387,
+          0.014111199416220188,
+          0.03704645484685898,
+          -0.015086585655808449,
+          -0.02178667113184929,
+          0.008983583189547062,
+          0.02658156119287014,
+          0.013655411079525948,
+          -0.004981763660907745,
+          -0.008842289447784424,
+          0.014721955172717571,
+          0.009325424209237099,
+          -0.02116679958999157,
+          -0.013454864732921124,
+          0.040838614106178284,
+          -0.013044655323028564,
+          -0.016253402456641197,
+          0.023336350917816162,
+          -0.013309012167155743,
+          -0.02317226678133011,
+          0.017520494759082794,
+          0.014284398406744003,
+          -0.0061531392857432365,
+          0.020437538623809814,
+          0.0220783744007349,
+          0.003309021471068263,
+          -0.004156787414103746,
+          0.019945286214351654,
+          -0.006431169807910919,
+          -0.015350942499935627,
+          -0.007319956552237272,
+          -0.011303544975817204,
+          -0.012479477562010288,
+          0.014512293040752411,
+          0.008008196949958801,
+          -0.01928895153105259,
+          -0.005893339868634939,
+          -0.020638084039092064,
+          -0.004149950575083494,
+          0.04627160727977753,
+          -0.028222398832440376,
+          -0.0023336349986493587,
+          0.04051044583320618,
+          -0.003101638052612543,
+          -0.04470369592308998,
+          -0.01864173263311386,
+          0.007762071210891008,
+          -0.01752961054444313,
+          -0.009024604223668575,
+          -0.0422242097556591,
+          -0.005820414051413536,
+          0.020492233335971832,
+          0.015250669792294502,
+          -0.015697341412305832,
+          0.010082032531499863,
+          -0.020455770194530487,
+          0.06388325989246368,
+          0.0004905419191345572,
+          -0.009047393687069416,
+          -0.01092979870736599,
+          -0.02650863490998745,
+          0.011968995444476604,
+          0.02650863490998745,
+          0.014503177255392075,
+          0.012543288059532642,
+          0.022041911259293556,
+          0.02047400176525116,
+          0.012069269083440304,
+          0.025852300226688385,
+          0.02123972587287426,
+          0.030902432277798653,
+          0.024266157299280167,
+          -0.0003660548245534301,
+          0.01807655580341816,
+          -0.008509563282132149,
+          -0.04656331241130829,
+          -0.008573373779654503,
+          -0.003154053585603833,
+          0.00038371660048142076,
+          -0.011704638600349426,
+          -0.008855962194502354,
+          -0.00014143674343358725,
+          -0.020528696477413177,
+          -0.019416572526097298,
+          -0.041130319237709045,
+          -0.006490422412753105,
+          0.0028919754549860954,
+          0.03447581082582474,
+          0.011677291244268417,
+          -0.0007338188006542623,
+          -0.01835002936422825,
+          0.018577922135591507,
+          0.017228789627552032,
+          0.010145843029022217,
+          0.010692788287997246,
+          -0.00012954352132510394,
+          -0.014904270879924297,
+          0.050428394228219986,
+          0.021476734429597855,
+          0.0075159454718232155,
+          -0.027219664305448532,
+          -0.01492250245064497,
+          0.004343660548329353,
+          -0.005255236756056547,
+          -0.008477658964693546,
+          -0.0130537711083889,
+          -0.019270719960331917,
+          0.0033272530417889357,
+          0.014949849806725979,
+          -0.018304450437426567,
+          -0.015961699187755585,
+          0.003577936440706253,
+          0.003413852769881487,
+          0.024794872850179672,
+          -0.006148581393063068,
+          0.015907004475593567,
+          -0.012762066908180714,
+          -0.02025522291660309,
+          0.028422946110367775,
+          0.02061985246837139,
+          0.007857786491513252,
+          0.007392882835119963,
+          -0.006554232910275459,
+          0.006385591346770525,
+          -0.0011622596066445112,
+          0.012707372196018696,
+          -0.03810388594865799,
+          0.029370984062552452,
+          -0.004181855823844671,
+          5.131177022121847e-05,
+          -0.012278931215405464,
+          0.0220783744007349,
+          0.006517769768834114,
+          -0.0004099244251847267,
+          0.007739281747490168,
+          0.0025227870792150497,
+          -0.010419315658509731,
+          -0.01582496240735054,
+          0.002310845535248518,
+          0.036535974591970444,
+          -0.0031654483173042536,
+          0.016973547637462616,
+          -0.0038559671957045794,
+          0.02255239523947239,
+          0.007926154881715775,
+          -0.022807635366916656,
+          -0.023281656205654144,
+          -0.006025518756359816,
+          0.0015929794171825051,
+          0.03197809308767319,
+          -0.016134899109601974,
+          -0.03325429931282997,
+          0.022789403796195984,
+          0.01711028441786766,
+          -0.020145833492279053,
+          0.03578848019242287,
+          0.013454864732921124,
+          -0.0035095682833343744,
+          0.032999057322740555,
+          -0.0033454846125096083,
+          0.0018664522795006633,
+          0.009863254614174366,
+          0.014630797319114208,
+          0.0032360954210162163,
+          -0.012561519630253315,
+          -0.00414767162874341,
+          -0.009047393687069416,
+          -0.01595258340239525,
+          -0.006390149239450693,
+          0.004345939494669437,
+          -0.025943458080291748,
+          -0.0025729236658662558,
+          -0.0008483355632051826,
+          -0.001078508561477065,
+          0.015396521426737309,
+          0.01283499225974083,
+          -0.017119400203227997,
+          0.020729241892695427,
+          -0.007784860674291849,
+          -0.025906994938850403,
+          0.02393799088895321,
+          0.015405637212097645,
+          -0.00026165085728280246,
+          0.005268910434097052,
+          0.036955296993255615,
+          0.02395622245967388,
+          -0.009845023043453693,
+          -2.443629455228802e-06,
+          -0.001749086775816977,
+          0.009407466277480125,
+          -0.005979939829558134,
+          0.0009018906857818365,
+          0.00822241697460413,
+          0.0027005444280803204,
+          0.024375546723604202,
+          -0.008017312735319138,
+          -0.01575203612446785,
+          0.005838645622134209,
+          -0.017292600125074387,
+          0.01137647032737732,
+          -0.0018527786014601588,
+          -0.006567906588315964,
+          -0.01357336901128292,
+          0.008336364291608334,
+          0.014484945684671402,
+          0.047292571514844894,
+          0.014092967845499516,
+          0.0056153093464672565,
+          -0.017247021198272705,
+          0.01283499225974083,
+          0.02783953584730625,
+          0.001371922204270959,
+          0.012069269083440304,
+          -0.02811300940811634,
+          0.021968986839056015,
+          -0.002764354692772031,
+          0.003744299290701747,
+          0.0065998113714158535,
+          -0.0012978565646335483,
+          -0.0001713478413876146,
+          0.03440288454294205,
+          0.020145833492279053,
+          0.021950755268335342,
+          0.08576108515262604,
+          0.009644475765526295,
+          -0.016927970573306084,
+          -0.010774830356240273,
+          -0.0032429322600364685,
+          -0.028678186237812042,
+          -0.013454864732921124,
+          0.044521380215883255,
+          -0.004936185199767351,
+          0.004553323145955801,
+          -0.00706471549347043,
+          -0.0006426612380892038,
+          -0.013655411079525948,
+          0.009462160989642143,
+          0.010391968302428722,
+          0.0036508627235889435,
+          -0.0020054676569998264,
+          -0.0018641733331605792,
+          -0.023208729922771454,
+          -0.003976751118898392,
+          -0.025177733972668648,
+          -0.0072287991642951965,
+          -0.0035938890650868416,
+          0.011431165039539337,
+          0.0018687312258407474,
+          0.023482201620936394,
+          -1.8819062461261638e-05,
+          -0.007885133847594261,
+          0.00307201175019145,
+          0.017985397949814796,
+          -0.020382843911647797,
+          0.010018222033977509,
+          0.019179562106728554,
+          -0.0031358220148831606,
+          -0.0007053320878185332,
+          0.0021444829180836678,
+          -0.008500447496771812,
+          -0.009626244194805622,
+          -0.012306278571486473,
+          0.02415676973760128,
+          -0.006517769768834114,
+          -0.01810390315949917,
+          -0.010884219780564308,
+          0.00918413046747446,
+          0.000970258901361376,
+          0.01864173263311386,
+          0.00678668450564146,
+          -0.005460341461002827,
+          0.014886039309203625,
+          -0.01183225866407156,
+          -0.006927979178726673,
+          -0.00954420305788517,
+          0.010774830356240273,
+          -0.009726517833769321,
+          0.02242477424442768,
+          -0.013473096303641796,
+          0.009115762077271938,
+          -0.01881493255496025,
+          0.0065587908029556274,
+          0.00035124170244671404,
+          -0.024448473006486893,
+          0.011613480746746063,
+          -0.0036690940614789724,
+          -0.004936185199767351,
+          -0.008322690613567829,
+          0.005806740373373032,
+          0.015578837133944035,
+          -0.016982663422822952,
+          0.00034810815122909844,
+          -0.014138546772301197,
+          -0.02776661142706871,
+          -0.006134907715022564,
+          0.006955326069146395,
+          -0.008017312735319138,
+          -0.02464902028441429,
+          0.02054692804813385,
+          0.014594335108995438,
+          -0.02499541826546192,
+          -0.0005406786222010851,
+          -0.0014676376013085246,
+          -0.01723790541291237,
+          -0.025560596957802773,
+          0.0240109171718359,
+          -0.003445758018642664,
+          -0.007525061257183552,
+          0.020729241892695427,
+          -0.0003711824247147888,
+          -0.03551500663161278,
+          0.02929805777966976,
+          -0.004589786287397146,
+          0.00822241697460413,
+          0.022114837542176247,
+          0.003903825068846345,
+          -0.0007594568887725472,
+          0.006800358183681965,
+          0.02206014283001423,
+          0.008436637930572033,
+          -0.02756606414914131,
+          0.005164078902453184,
+          -0.014557871967554092,
+          -0.0027552389074116945,
+          -0.0035391945857554674,
+          -0.015907004475593567,
+          -0.03402002155780792,
+          0.0066590639762580395,
+          0.011531438678503036,
+          0.011358239687979221,
+          0.004120324272662401,
+          0.0054557835683226585,
+          0.018687311559915543,
+          0.0003073721018154174,
+          0.02082039974629879,
+          -0.0021365066058933735,
+          0.016581570729613304,
+          0.047365497797727585,
+          0.00021920558356214315,
+          -0.03394709527492523,
+          0.0007190057076513767,
+          0.00652688555419445,
+          -0.005200542043894529,
+          0.0028691859915852547,
+          0.03680944815278053,
+          0.01708293706178665,
+          -0.021968986839056015,
+          0.014010925777256489,
+          0.0013833168195560575,
+          -0.013208738528192043,
+          -0.019525961950421333,
+          0.0011759332846850157,
+          -0.02670918218791485,
+          0.007584313862025738,
+          -0.0025888762902468443,
+          -0.009229708462953568,
+          -0.020948020741343498,
+          -0.004721964709460735,
+          0.043062858283519745,
+          0.019307183101773262,
+          0.009001814760267735,
+          -0.001074520405381918,
+          -0.011048302985727787,
+          -0.03817681223154068,
+          0.010355505160987377,
+          0.029535068199038506,
+          -0.0015553769189864397,
+          0.006080213002860546,
+          -0.017000894993543625,
+          0.009589781053364277,
+          -0.0077347238548099995,
+          0.006335454527288675,
+          0.011467628180980682,
+          0.013409285806119442,
+          -0.0021102989558130503,
+          0.011823142878711224,
+          0.009762980975210667,
+          0.01006380096077919,
+          -0.0406927615404129,
+          0.007338188122957945,
+          0.0057429298758506775,
+          0.008568815886974335,
+          -0.00844575371593237,
+          0.016061972826719284,
+          -0.01970827765762806,
+          0.016326328739523888,
+          -0.03290789946913719,
+          -0.003418410662561655,
+          0.015733804553747177,
+          -0.01829533465206623,
+          -0.02443024143576622,
+          0.007702818606048822,
+          -0.0030287117697298527,
+          0.005984497722238302,
+          -0.006476748734712601,
+          0.00960801262408495,
+          -0.04557880759239197,
+          0.011075650341808796,
+          -0.02317226678133011,
+          -0.005578846205025911,
+          0.03617134317755699,
+          -0.004854143131524324,
+          -0.020291686058044434,
+          -0.00022817892022430897,
+          0.019489498808979988,
+          -0.020729241892695427,
+          -0.04350041598081589,
+          0.012725603766739368,
+          -0.014065620489418507,
+          0.013272549025714397,
+          -0.015843193978071213,
+          -0.007885133847594261,
+          0.015606184490025043,
+          0.013418401591479778,
+          0.009434813633561134,
+          -0.0028144915122538805,
+          -0.000781106820795685,
+          0.016034625470638275,
+          -0.03374655172228813,
+          0.006166812963783741,
+          -0.018723774701356888,
+          -0.03941655531525612,
+          -0.0012499988079071045,
+          0.023755675181746483,
+          -0.0042205979116261005,
+          0.02408384345471859,
+          0.01467637624591589,
+          -0.030063781887292862,
+          -0.02109387330710888,
+          -0.014484945684671402,
+          0.039671797305345535,
+          0.007971733808517456,
+          -0.023281656205654144,
+          -0.0027871441561728716,
+          -0.0007218543905764818,
+          0.014092967845499516,
+          -0.017985397949814796,
+          -0.0030879643745720387,
+          0.004181855823844671,
+          -0.004703733138740063,
+          -0.005975381936877966,
+          -0.03927070274949074,
+          0.012570635415613651,
+          0.019179562106728554,
+          -0.012935265898704529,
+          0.006271644029766321,
+          0.012844108045101166,
+          -0.01859615370631218,
+          0.012816760689020157,
+          -0.004300360567867756,
+          0.02796715684235096,
+          0.01680946536362171,
+          0.014594335108995438,
+          -0.009498624131083488,
+          -0.010993609204888344,
+          0.0011394702596589923,
+          -0.0007719910936430097,
+          -0.011285313405096531,
+          0.004767543636262417,
+          -0.003935730084776878,
+          0.020018212497234344,
+          0.017839545384049416,
+          0.029954394325613976,
+          -0.019179562106728554,
+          0.012798530049622059,
+          -0.03664536401629448,
+          0.00568823516368866,
+          0.0010044430382549763,
+          -0.00441430788487196,
+          -0.007739281747490168,
+          0.020236991345882416,
+          -0.01965358294546604,
+          0.0046672699972987175,
+          -0.011577017605304718,
+          -0.026599792763590813,
+          0.0009919088333845139,
+          0.007438461761921644,
+          -0.015961699187755585,
+          0.0018926600459963083,
+          -0.013546021655201912,
+          -0.01985413022339344,
+          -0.007497713901102543,
+          0.026563329622149467,
+          -0.008413848467171192,
+          -0.0005198832950554788,
+          -0.0005409634904935956,
+          -0.01391065213829279,
+          0.007333630230277777,
+          -0.003530078800395131,
+          -0.006567906588315964,
+          0.0017080658581107855,
+          -0.02428438887000084,
+          -0.007443019654601812,
+          0.005041016265749931,
+          0.02339104562997818,
+          0.015551489777863026,
+          -0.018441185355186462,
+          -0.024029148742556572,
+          -0.002853233367204666,
+          0.026526866480708122,
+          0.014247936196625233,
+          -0.026891497895121574,
+          0.01846853271126747,
+          -0.003641746938228607,
+          0.020638084039092064,
+          0.006216949783265591,
+          0.006618042942136526,
+          -0.0032657217234373093,
+          -0.008199627511203289,
+          0.007119410205632448,
+          -0.009525971487164497,
+          -0.02138557657599449,
+          -0.009735633619129658,
+          0.015496795065701008,
+          -0.010647210292518139,
+          0.003821783233433962,
+          0.000417046103393659,
+          -0.015916120260953903,
+          0.006289875600486994,
+          -0.009790328331291676,
+          0.015432984568178654,
+          0.011996342800557613,
+          0.00036747913691215217,
+          0.00036235153675079346,
+          0.030483108013868332,
+          -0.012297162786126137,
+          -0.007917039096355438,
+          0.00522333150729537,
+          -0.009047393687069416,
+          0.004010935313999653,
+          -0.007834997028112411,
+          -0.03241565078496933,
+          -0.0031859588343650103,
+          0.014311745762825012,
+          0.018605269491672516,
+          -0.01044666301459074,
+          -0.007958060130476952,
+          0.028058314695954323,
+          0.006285317707806826,
+          -0.007306283339858055,
+          -0.005018226802349091,
+          0.04718318209052086,
+          -0.0003959659079555422,
+          0.020984483882784843,
+          0.009535087272524834,
+          0.003142658853903413,
+          -0.004181855823844671,
+          -0.0003748857125174254,
+          0.008846846409142017,
+          -0.003760251682251692,
+          0.0021718302741646767,
+          -0.012415667995810509,
+          0.013254317454993725,
+          -0.0037078361492604017,
+          -0.028404714539647102,
+          0.008327248506247997,
+          -0.014603450894355774,
+          0.0001921431685332209,
+          -0.018377376720309258,
+          0.0027005444280803204,
+          0.012798530049622059,
+          -0.012561519630253315,
+          -0.0017525051953271031,
+          -0.007101178634911776,
+          -0.00947127677500248,
+          -0.0021923407912254333,
+          -0.0010414757998660207,
+          -0.02027345448732376,
+          -0.010893335565924644,
+          0.0008016172796487808,
+          -0.012762066908180714,
+          0.011886953376233578,
+          0.0002697695745155215,
+          -0.0006933676195330918,
+          0.00975386518985033,
+          -0.0009805141016840935,
+          0.019544193521142006,
+          0.00699178921058774,
+          0.0047265226021409035,
+          -0.006089328788220882,
+          0.004712848924100399,
+          0.020091138780117035,
+          -0.035770248621702194,
+          0.004095255862921476,
+          0.0005959429545328021,
+          0.009120319969952106,
+          -0.006604369264096022,
+          -0.016216941177845,
+          -0.002313124481588602,
+          -0.019598888233304024,
+          0.009899717755615711,
+          -0.01595258340239525,
+          -0.012680024839937687,
+          -0.000525010924320668,
+          0.0003065174969378859,
+          -0.004571554716676474,
+          0.006157697178423405,
+          0.011850490234792233,
+          0.004327707923948765,
+          0.0032133059576153755,
+          0.013937999494373798,
+          -0.020018212497234344,
+          0.004459886346012354,
+          -0.007944386452436447,
+          -0.0019747018814086914,
+          0.015195975080132484,
+          -0.0020134439691901207,
+          0.0006033494719304144,
+          -0.014895155094563961,
+          -0.002074975287541747,
+          0.005765719339251518,
+          -0.0012830435298383236,
+          -0.0036987203639000654,
+          0.013381938450038433,
+          -0.004958974663168192,
+          0.00013353166286833584,
+          -0.008587047457695007,
+          -0.00039767511771060526,
+          -0.004949858877807856,
+          0.006467632949352264,
+          -0.016982663422822952,
+          0.009735633619129658,
+          0.01721055805683136,
+          0.0061440235003829,
+          -0.002125111874192953,
+          -0.003600725904107094,
+          -0.00301959621720016,
+          0.03309021517634392,
+          0.006704642903059721,
+          0.02060162089765072,
+          0.029334520921111107,
+          0.008896983228623867,
+          -0.0037853200919926167,
+          0.0013092512963339686,
+          -0.008331806398928165,
+          -0.007456693332642317,
+          -0.0230446457862854,
+          -0.013299896381795406,
+          0.01464902888983488,
+          -0.006084770895540714,
+          0.012470361776649952,
+          -5.740081178373657e-05,
+          0.007383767049759626,
+          -0.00607109721750021,
+          -0.01896078512072563,
+          -0.030902432277798653,
+          -0.002962622558698058,
+          0.022114837542176247,
+          -0.005405646748840809,
+          -0.026964424178004265,
+          0.0004817110311705619,
+          0.010474010370671749,
+          -0.018094787374138832,
+          -0.015669994056224823,
+          0.010273464024066925,
+          -0.007429345976561308,
+          0.01886962726712227,
+          -0.008641742169857025,
+          0.017739271745085716,
+          -0.0009041696321219206,
+          -0.008641742169857025,
+          0.022114837542176247,
+          -0.024339083582162857,
+          0.013299896381795406,
+          -0.0004179007082711905,
+          -0.0031677272636443377,
+          -0.0003310161118861288,
+          0.010528705082833767,
+          -0.010765714570879936,
+          -0.0184503011405468,
+          0.005478573031723499,
+          -0.01346398051828146,
+          0.009352771565318108,
+          -0.008896983228623867,
+          0.011148576624691486,
+          0.02470371499657631,
+          -0.014475829899311066,
+          0.004977205768227577,
+          0.007206009700894356,
+          0.0012534172274172306,
+          0.0014949849573895335,
+          -0.005793066695332527,
+          0.013199622742831707,
+          0.0069781155325472355,
+          -0.010647210292518139,
+          0.02692796103656292,
+          0.010993609204888344,
+          -0.012752951122820377,
+          -0.0035847732797265053,
+          -0.007301725447177887,
+          -0.018796700984239578,
+          -0.000998745672404766,
+          -0.018997248262166977,
+          -0.011048302985727787,
+          -0.008587047457695007,
+          -0.011112113483250141,
+          0.0019507730612531304,
+          -0.005528709385544062,
+          -0.01259798277169466,
+          -0.0038377356249839067,
+          0.011804911307990551,
+          -0.006704642903059721,
+          -0.03318137302994728,
+          0.01812213473021984,
+          0.019762972369790077,
+          0.008140374906361103,
+          -0.03792157024145126,
+          0.019872359931468964,
+          0.00757975596934557,
+          -0.002256151055917144,
+          -0.012853223830461502,
+          0.0012841830030083656,
+          -0.01896078512072563,
+          0.013263433240354061,
+          -0.019872359931468964,
+          -0.001451685093343258,
+          0.0016818580916151404,
+          0.007958060130476952,
+          0.005100268870592117,
+          0.022880561649799347,
+          -0.005195984151214361,
+          -0.004268455319106579,
+          0.005478573031723499,
+          0.0038103885017335415,
+          -0.0025524133816361427,
+          0.030829505994915962,
+          -0.01051958929747343,
+          -0.023154035210609436,
+          -0.002796259941533208,
+          0.014521408826112747,
+          -0.009735633619129658,
+          -0.015387405641376972,
+          0.020437538623809814,
+          -0.01797628216445446,
+          0.0022401984315365553,
+          0.014731070958077908,
+          0.004284407943487167,
+          0.030720116570591927,
+          0.01290791854262352,
+          -0.005346394143998623,
+          -0.000578565988689661,
+          0.00010262354044243693,
+          0.0010055825114250183,
+          0.011613480746746063,
+          -0.004895164165645838,
+          -0.004762985743582249,
+          -0.005815856158733368,
+          0.012725603766739368,
+          -9.649888670537621e-05,
+          -0.0031996325124055147,
+          -0.00015781662659719586,
+          -0.015414752997457981,
+          0.014876923523843288,
+          0.0026731970719993114,
+          0.0036280732601881027,
+          0.007420230191200972,
+          -6.366789602907375e-05,
+          -0.0059617082588374615,
+          -0.01384684257209301,
+          0.006567906588315964,
+          -0.011340008117258549,
+          -0.0115223228931427,
+          -0.028696417808532715,
+          0.014949849806725979,
+          -0.02040107548236847,
+          0.00958066526800394,
+          -0.002073835814371705,
+          0.009863254614174366,
+          -0.02054692804813385,
+          -0.015788499265909195,
+          -0.009325424209237099,
+          0.012607098557054996,
+          -0.009662707336246967,
+          0.02144027128815651,
+          0.007174104452133179,
+          0.015770267695188522,
+          0.021257957443594933,
+          0.024247925728559494,
+          0.004867816809564829,
+          0.01629898138344288,
+          -0.02818593569099903,
+          -0.0007919318159110844,
+          -0.007748397532850504,
+          0.012014574371278286,
+          -0.0036166785284876823,
+          0.006394707132130861,
+          0.013108465820550919,
+          0.01723790541291237,
+          -0.010938914492726326,
+          0.002903370186686516,
+          -2.533540828153491e-05,
+          -0.00958066526800394,
+          -0.00711485231295228,
+          -0.017784850671887398,
+          -0.01859615370631218,
+          0.01631721295416355,
+          0.007511387579143047,
+          0.012780298478901386,
+          0.016052857041358948,
+          0.016289865598082542,
+          -0.0023199613206088543,
+          0.0021091594826430082,
+          -0.01943480409681797,
+          -0.0025934341829270124,
+          -0.012917034327983856,
+          -0.0013593879994004965,
+          0.025560596957802773,
+          -0.0017069263849407434,
+          0.017939819023013115,
+          0.007433903869241476,
+          0.034731052815914154,
+          0.0010015943553298712,
+          0.006927979178726673,
+          0.005278026219457388,
+          0.015633530914783478,
+          0.0027119391597807407,
+          0.02386506460607052,
+          -0.02040107548236847,
+          0.0013001355109736323,
+          0.03691883385181427,
+          0.022862330079078674,
+          -0.008195069618523121,
+          -0.01790335588157177,
+          -0.01943480409681797,
+          0.016681844368577003,
+          -0.03019140288233757,
+          -0.00757975596934557,
+          0.02386506460607052,
+          0.00624885456636548,
+          0.01319050695747137,
+          -0.0006648809066973627,
+          -0.018650848418474197,
+          0.012889686971902847,
+          0.008195069618523121,
+          -0.009498624131083488,
+          0.004423423204571009,
+          -0.006381033454090357,
+          -0.009489508345723152,
+          -0.0027825862634927034,
+          -0.0027871441561728716,
+          -0.017474915832281113,
+          -0.00865541584789753,
+          0.0004555032355710864,
+          -0.004541928414255381,
+          0.0021102989558130503,
+          -0.003470826428383589,
+          -0.0217137448489666,
+          0.0058340877294540405,
+          -0.012087500654160976,
+          -0.0025683660060167313,
+          0.0018128971569240093,
+          -0.009498624131083488,
+          -0.024667251855134964,
+          -0.00650409609079361,
+          -0.007142199203372002,
+          -0.005268910434097052,
+          0.01277118269354105,
+          -0.015460331924259663,
+          0.009257055819034576,
+          -0.014876923523843288,
+          0.023354582488536835,
+          -0.0005976521642878652,
+          0.008609836921095848,
+          0.013965346850454807,
+          -0.013454864732921124,
+          -0.011494975537061691,
+          -0.03442111611366272,
+          -0.0044781179167330265,
+          -0.020455770194530487,
+          -0.00565633038058877,
+          -0.00660892715677619,
+          0.01689150743186474,
+          0.0018197339959442616,
+          -0.017301715910434723,
+          0.01696443371474743,
+          0.004854143131524324,
+          -0.0013138091890141368,
+          -0.01561530027538538,
+          -0.0044484916143119335,
+          -9.236830373993143e-05,
+          -0.007452135439962149,
+          0.01956242509186268,
+          0.00161348981782794,
+          -0.002983133075758815,
+          0.0025136712938547134,
+          0.007803092245012522,
+          -0.007092062849551439,
+          -0.03389240428805351,
+          -0.012579751200973988,
+          -0.002392887370660901,
+          6.17094337940216e-05,
+          -0.007493156008422375,
+          0.003685046685859561,
+          0.014339093118906021,
+          -0.011185039766132832,
+          -0.007921596989035606,
+          -0.00013324679457582533,
+          0.009808559902012348,
+          0.015451216138899326,
+          -0.009334539994597435,
+          0.018094787374138832,
+          -0.01567910984158516,
+          0.01874200627207756,
+          0.0007093202439136803,
+          -0.002285777358338237,
+          -0.0022162694949656725,
+          0.010282579809427261,
+          -0.008473101072013378,
+          -0.006918863393366337,
+          -0.009211476892232895,
+          -0.008814942091703415,
+          -0.0073290723375976086,
+          0.0013046934036538005,
+          -0.020291686058044434,
+          -0.020528696477413177,
+          -0.006230622995644808,
+          0.0030013646464794874,
+          -0.006125791929662228,
+          0.011567901819944382,
+          -0.01211484707891941,
+          -0.014348208904266357,
+          0.014147662557661533,
+          0.007392882835119963,
+          0.0029216017574071884,
+          -0.017092052847146988,
+          -0.007274378091096878,
+          -0.007661798037588596,
+          0.008299901150166988,
+          -0.0031700062099844217,
+          0.034803979098796844,
+          -0.003343205666169524,
+          -0.010455778799951077,
+          0.003015038324519992,
+          -0.001371922204270959,
+          0.0058112982660532,
+          -0.016499528661370277,
+          0.005701908841729164,
+          0.0011172505328431726,
+          0.006814031861722469,
+          0.020528696477413177,
+          -0.0016294424422085285,
+          -0.012871455401182175,
+          0.008796710520982742,
+          0.010191421955823898,
+          0.002445303136482835,
+          -0.0016009557293727994,
+          -0.0016898344038054347,
+          -0.00801275484263897,
+          0.019945286214351654,
+          0.041604336351156235,
+          0.0035164051223546267,
+          0.01037373673170805,
+          0.009389234706759453,
+          -0.01641748659312725,
+          -0.009489508345723152,
+          0.014685492031276226,
+          -0.008436637930572033,
+          0.005683677736669779,
+          -0.006253412459045649,
+          0.019981749355793,
+          -0.011458512395620346,
+          -0.005232447292655706,
+          0.009981758892536163,
+          -0.003575657494366169,
+          0.014904270879924297,
+          0.006107560358941555,
+          -0.019817667081952095,
+          0.008600721135735512,
+          0.0014482666738331318,
+          -0.0035072893369942904,
+          -0.026818571612238884,
+          0.0024703715462237597,
+          0.002686870750039816,
+          0.009945295751094818,
+          -0.0232451930642128,
+          0.01252505648881197,
+          0.010145843029022217,
+          0.004908837843686342,
+          0.004651317372918129,
+          0.0037511358968913555,
+          -0.009329982101917267,
+          0.005369183607399464,
+          0.01923425681889057,
+          -0.008154048584401608,
+          -0.025852300226688385,
+          -0.0004278710694052279,
+          0.001534866401925683,
+          -0.010811293497681618,
+          -0.0058112982660532,
+          0.010528705082833767,
+          -0.009480392560362816,
+          -0.005337278358638287,
+          0.015569721348583698,
+          0.014138546772301197,
+          -0.012570635415613651,
+          0.02816770412027836,
+          -0.006203276105225086,
+          0.0061804866418242455,
+          0.025724679231643677,
+          -0.007365535479038954,
+          -0.012178657576441765,
+          -0.025141270831227303,
+          0.013245201669633389,
+          0.0010956006590276957,
+          0.019398340955376625,
+          -0.02747490629553795,
+          -0.021640818566083908,
+          -0.018359145149588585,
+          0.008359153755009174,
+          0.008208743296563625,
+          0.0027301707305014133,
+          0.02277117222547531,
+          -0.00683682132512331,
+          -0.0010357784340158105,
+          -0.0013331801164895296,
+          0.019270719960331917,
+          0.00732451444491744,
+          -0.0015758873196318746,
+          0.011686407029628754,
+          0.012789414264261723,
+          0.01567910984158516,
+          -0.011558786034584045,
+          -0.014594335108995438,
+          -0.014968081377446651,
+          0.00016992349992506206,
+          -0.01044666301459074,
+          -0.004863258916884661,
+          0.006513211876153946,
+          0.026271626353263855,
+          -0.021203262731432915,
+          0.010018222033977509,
+          0.006376475561410189,
+          0.002236780012026429,
+          -0.0016647659940645099,
+          -0.004175018984824419,
+          0.018113018944859505,
+          -0.017721040174365044,
+          0.00532816257327795,
+          -0.0010152680333703756,
+          0.0027073812671005726,
+          0.012470361776649952,
+          -0.023135803639888763,
+          -0.005565172526985407,
+          0.015341827645897865,
+          -0.004122603219002485,
+          -0.00947127677500248,
+          0.026052847504615784,
+          -0.03700999170541763,
+          0.021276189014315605,
+          0.015423868782818317,
+          -0.0024065610487014055,
+          -0.011896069161593914,
+          -0.01881493255496025,
+          0.0034024580381810665,
+          -0.027019118890166283,
+          -0.0018687312258407474,
+          0.005405646748840809,
+          -0.01267090905457735,
+          -0.010173190385103226,
+          0.009589781053364277,
+          0.011786679737269878,
+          -0.005515035707503557,
+          -0.009936179965734482,
+          -0.008003639057278633,
+          -0.009507739916443825,
+          -0.011996342800557613,
+          0.0057429298758506775,
+          -0.00898814108222723,
+          -0.003976751118898392,
+          0.0007685726741328835,
+          0.02678210847079754,
+          -4.931769581162371e-06,
+          -0.012661793269217014,
+          0.013892421498894691,
+          0.02541474439203739,
+          -0.009726517833769321,
+          -0.02235184796154499,
+          -0.008983583189547062,
+          -0.0009565852815285325,
+          -0.0037055572029203176,
+          -0.008263438008725643,
+          0.0009070183150470257,
+          -0.011804911307990551,
+          0.0040701874531805515,
+          0.003933451138436794,
+          -0.0009845022577792406,
+          -0.022151300683617592,
+          0.008331806398928165,
+          0.007999081164598465,
+          0.011440280824899673,
+          -0.0414949469268322,
+          -0.002910207025706768,
+          -0.012880571186542511,
+          -0.0042205979116261005,
+          -0.01429351419210434,
+          0.025232428684830666,
+          0.005847761407494545,
+          0.0065770223736763,
+          -0.02470371499657631,
+          0.008103912696242332,
+          0.022242458537220955,
+          -0.0014699165476486087,
+          0.0018698706990107894,
+          -0.01159524917602539,
+          -0.00257520261220634,
+          0.028751112520694733,
+          -0.007803092245012522,
+          0.00506836362183094,
+          -0.003101638052612543,
+          0.00027831559418700635,
+          0.004441654775291681,
+          0.013372822664678097,
+          0.0007588871521875262,
+          -0.00982679147273302,
+          0.014494061470031738,
+          0.00021607204689644277,
+          0.005756603553891182,
+          -0.0031107538379728794,
+          -0.01329078059643507,
+          0.0025432975962758064,
+          -0.009899717755615711,
+          -0.03144937753677368,
+          -0.01579761505126953,
+          -0.020510464906692505,
+          -0.013427517376840115,
+          -0.002880580723285675,
+          0.0334366150200367,
+          0.014776649884879589,
+          -0.014010925777256489,
+          -0.0012955776182934642,
+          -0.0034047369845211506,
+          0.027784842997789383,
+          0.011485859751701355,
+          -0.008030986413359642,
+          0.02020052820444107,
+          -0.0017593420343473554,
+          -0.00285551231354475,
+          0.018277103081345558,
+          -0.004343660548329353,
+          0.010802177712321281,
+          0.0024726504925638437,
+          -0.02421146258711815,
+          0.012743835337460041,
+          -0.002525066025555134,
+          0.001482450752519071,
+          0.010957146063446999,
+          -0.00960801262408495,
+          -0.01290791854262352,
+          0.0015417032409459352,
+          0.008523236960172653,
+          -0.012196889147162437,
+          -0.024375546723604202,
+          -0.0033477635588496923,
+          -0.008550584316253662,
+          -0.007948944345116615,
+          0.0016191871836781502,
+          0.014703723601996899,
+          -0.0004247375181876123,
+          0.006522327661514282,
+          -0.004936185199767351,
+          -0.014165894128382206,
+          -0.004694617353379726,
+          0.02144027128815651,
+          0.01866907998919487,
+          0.00954420305788517,
+          -4.436812378116883e-05,
+          0.016882391646504402,
+          -0.010255232453346252,
+          -0.005875108297914267,
+          -0.016927970573306084,
+          -0.02484956756234169,
+          0.0009457602864131331,
+          0.0106563251465559,
+          -0.024120306596159935,
+          0.0034844998735934496,
+          -0.048678167164325714,
+          -0.008700994774699211,
+          0.017174094915390015,
+          -0.01908840611577034,
+          0.005059247836470604,
+          0.00242023472674191,
+          -0.03041018173098564,
+          0.015268901363015175,
+          -0.019762972369790077,
+          0.01061986293643713,
+          -0.009311750531196594,
+          -0.02588876336812973,
+          0.016818581148982048,
+          -0.006677295546978712,
+          0.016335444524884224,
+          -0.009079298935830593,
+          -0.008304459042847157,
+          -0.0056426567025482655,
+          -0.006221507675945759,
+          0.00036690940032713115,
+          0.0006888097268529236,
+          0.001456242986023426,
+          0.0036736519541591406,
+          0.004899722058326006,
+          -0.01750226318836212,
+          -0.009945295751094818,
+          -0.01551502663642168,
+          -0.014895155094563961,
+          -0.01810390315949917,
+          0.009083856828510761,
+          -0.01520509086549282,
+          0.010027337819337845,
+          -0.0034252475015819073,
+          -0.017438452690839767,
+          0.007092062849551439,
+          -0.006809473969042301,
+          0.005232447292655706,
+          -0.005970824044197798,
+          -0.006595253478735685,
+          0.009899717755615711,
+          0.006905189715325832,
+          0.007871460169553757,
+          0.00706471549347043,
+          -0.012844108045101166,
+          -0.004193250555545092,
+          -0.003876477712765336,
+          0.03318137302994728,
+          0.0005891061155125499,
+          -0.01023700088262558,
+          -0.005788508802652359,
+          -1.0931792530755047e-05,
+          7.641572301508859e-05,
+          0.005086595192551613,
+          0.0007127386052161455,
+          0.016499528661370277,
+          -0.019197793677449226,
+          -0.0051367320120334625,
+          -0.0060209608636796474,
+          -0.005697350949048996,
+          -0.012534172274172306,
+          -0.016918854787945747,
+          0.002527344971895218,
+          0.01916133053600788,
+          0.009261613711714745,
+          0.01259798277169466,
+          -0.03110297955572605,
+          0.0029671804513782263,
+          0.021494966000318527,
+          -0.025706447660923004,
+          -0.010474010370671749,
+          -0.00793527066707611,
+          -0.028277093544602394,
+          -0.015551489777863026,
+          0.013855958357453346,
+          0.014311745762825012,
+          0.023208729922771454,
+          -0.017319947481155396,
+          -0.0007452135323546827,
+          -0.03274381533265114,
+          -0.0012978565646335483,
+          -0.0030811275355517864,
+          -0.008381943218410015,
+          -0.024375546723604202,
+          0.025834068655967712,
+          0.015979930758476257,
+          0.008559700101613998,
+          0.004355055280029774,
+          -0.000295692530926317,
+          0.024831335991621017,
+          -0.004131719004362822,
+          0.007629892788827419,
+          -0.001804920844733715,
+          0.017347294837236404,
+          -0.009553318843245506,
+          -0.012889686971902847,
+          0.005109384655952454,
+          -0.01068367250263691,
+          -7.24987912690267e-05,
+          -0.00688695814460516,
+          -0.027456674724817276,
+          -0.029443910345435143,
+          0.0053965309634804726,
+          0.0013400169555097818,
+          0.009872370399534702,
+          -0.022935256361961365,
+          0.018240639939904213,
+          -0.0066590639762580395,
+          -0.004744754172861576,
+          -0.0006774150533601642,
+          -0.0023906086571514606,
+          0.02720143273472786,
+          -0.02220599539577961,
+          -0.0026230604853481054,
+          -0.0015439821872860193,
+          -0.010027337819337845,
+          0.006595253478735685,
+          0.0008027567528188229,
+          0.013372822664678097,
+          0.016818581148982048,
+          -0.015004543587565422,
+          -0.04123970493674278,
+          -0.002021420281380415,
+          -0.035095684230327606,
+          -0.0077347238548099995,
+          -0.002716497052460909,
+          0.021768439561128616,
+          -0.006435727700591087,
+          0.037082917988300323,
+          -0.0014414298348128796,
+          -0.02464902028441429,
+          -0.0270738136023283,
+          0.01599816232919693,
+          0.023135803639888763,
+          -0.014457598328590393,
+          -0.0026549657341092825,
+          0.019525961950421333,
+          -0.004120324272662401,
+          0.01121238712221384,
+          0.012160426005721092,
+          0.015323596075177193,
+          -0.004138555843383074,
+          0.0018687312258407474,
+          0.008267995901405811,
+          -0.019690046086907387,
+          -0.028751112520694733,
+          -0.0036189574748277664,
+          0.00030025039450265467,
+          -0.015323596075177193,
+          0.0074475775472819805,
+          -0.01106653455644846,
+          -0.007060157600790262,
+          0.006016402970999479,
+          0.007876018062233925,
+          -0.007944386452436447,
+          0.025925226509571075,
+          -0.014193241484463215,
+          -0.002069277921691537,
+          0.023901527747511864,
+          0.0006523467018269002,
+          -0.016873275861144066,
+          -0.0047265226021409035,
+          0.0032452112063765526,
+          0.027948925271630287,
+          0.01679123379290104,
+          -0.007201451808214188,
+          -0.017848661169409752,
+          0.02443024143576622,
+          -0.011230618692934513,
+          -0.007629892788827419,
+          0.0001251992944162339,
+          0.01200545858591795,
+          0.01842295378446579,
+          -0.004234271589666605,
+          0.004813122097402811,
+          -0.0009480392327532172,
+          0.033491309732198715,
+          0.002347308676689863,
+          0.009453045204281807,
+          -0.01644483394920826,
+          0.009307192638516426,
+          -0.0023974452633410692,
+          -0.00021179902250878513,
+          -0.0021410644985735416,
+          -0.034731052815914154,
+          -0.01530536450445652,
+          -0.0042205979116261005,
+          0.003985866904258728,
+          -0.021914292126893997,
+          0.0041294400580227375,
+          0.001722879009321332,
+          -0.02665448747575283,
+          0.016216941177845,
+          -0.016918854787945747,
+          -0.00944392941892147,
+          0.017821313813328743,
+          -0.0070692733861505985,
+          -0.0073883249424397945,
+          -0.012206004932522774,
+          0.022716479375958443,
+          0.00975386518985033,
+          -0.008846846409142017,
+          0.009890601970255375,
+          -0.01482222881168127,
+          0.008158606477081776,
+          -0.0015451216604560614,
+          0.007834997028112411,
+          0.021185031160712242,
+          -0.017638999968767166,
+          0.017438452690839767,
+          0.00011950194311793894,
+          -0.009744749404489994,
+          -0.008523236960172653,
+          0.00436417106539011,
+          0.027037350460886955,
+          0.00035893311724066734,
+          0.017693692818284035,
+          0.007429345976561308,
+          0.03301728889346123,
+          -0.0034867788199335337,
+          0.004471281077712774,
+          0.012133078649640083,
+          0.016089320182800293,
+          -0.017174094915390015,
+          -0.013026423752307892,
+          0.0020795331802219152,
+          -0.009790328331291676,
+          0.0013650853652507067,
+          -0.009762980975210667,
+          -0.02165905013680458,
+          0.02623516321182251,
+          0.021531429141759872,
+          -0.017019126564264297,
+          0.0008016172796487808,
+          -0.008810384199023247,
+          -0.0351686105132103,
+          -0.018404724076390266,
+          0.0108568724244833,
+          0.015241554006934166,
+          0.0010813572444021702,
+          0.010309926234185696,
+          -0.01016407459974289,
+          -0.00021920558356214315,
+          -0.0036189574748277664,
+          -0.01723790541291237,
+          -0.01793070323765278,
+          -0.015405637212097645,
+          -0.009671823121607304,
+          -0.009029162116348743,
+          0.0039699142798781395,
+          -0.011303544975817204,
+          -0.006536001339554787,
+          0.004514581058174372,
+          0.004731080494821072,
+          -0.009735633619129658,
+          -0.011850490234792233,
+          0.013126697391271591,
+          -0.001399269443936646,
+          0.00046974659198895097,
+          0.016581570729613304,
+          -0.0035961680114269257,
+          -0.0213126502931118,
+          0.00387875665910542,
+          0.006025518756359816,
+          0.012515940703451633,
+          0.00429124478250742,
+          -0.010483126156032085,
+          0.01599816232919693,
+          0.00568823516368866,
+          -0.012333625927567482,
+          0.017146747559309006,
+          -0.022023679688572884,
+          -0.007538734935224056,
+          0.03296259418129921,
+          -0.013637179508805275,
+          0.009316308423876762,
+          0.0003042385505978018,
+          -0.006431169807910919,
+          0.02206014283001423,
+          0.026545098051428795,
+          0.01627163402736187,
+          0.00515040522441268,
+          0.012798530049622059,
+          0.00031363917514681816,
+          -0.0011656780261546373,
+          0.015843193978071213,
+          -0.0030765696428716183,
+          -0.026198700070381165,
+          -0.0025501344352960587,
+          0.0009018906857818365,
+          0.002495439723134041,
+          0.0029808541294187307,
+          -0.010628978721797466,
+          -0.005715582519769669,
+          -0.0010101404041051865,
+          -0.006226065568625927,
+          -0.0034092948772013187,
+          0.001910891616716981,
+          -0.011850490234792233,
+          0.0011998621048405766,
+          -0.02650863490998745,
+          0.00031335430685430765,
+          -0.012570635415613651,
+          -0.027912462130188942,
+          0.021075641736388206,
+          -0.005984497722238302,
+          0.011148576624691486,
+          0.022917024791240692,
+          0.019252488389611244,
+          -0.008792152628302574,
+          -0.004981763660907745,
+          0.0012978565646335483,
+          0.008673647418618202,
+          -0.014202357269823551,
+          -0.014594335108995438,
+          -0.009808559902012348,
+          -0.0019439362222328782,
+          -0.020218759775161743,
+          -0.021257957443594933,
+          -0.018231524154543877,
+          0.0010454639559611678,
+          0.01513216458261013,
+          0.0006540559115819633,
+          0.00459206523373723,
+          -0.016399255022406578,
+          0.00898814108222723,
+          -0.011030071415007114,
+          -0.011987227015197277,
+          0.003685046685859561,
+          0.004494070541113615,
+          -0.010793061926960945,
+          0.004712848924100399,
+          -0.005715582519769669,
+          -0.00301959621720016,
+          0.002584318397566676,
+          -0.014211473055183887,
+          -0.002613944699987769,
+          0.0006301270332187414,
+          0.025451207533478737,
+          0.0005329871783033013,
+          0.009772096760571003,
+          0.004045119509100914,
+          0.009188688360154629,
+          0.008267995901405811,
+          0.0003355739754624665,
+          0.0071695465594530106,
+          0.0014049668097868562,
+          -0.005401088856160641,
+          0.017620768398046494,
+          0.0011964436853304505,
+          -0.006586138159036636,
+          -0.007926154881715775,
+          -0.021403808146715164,
+          -0.003926614299416542,
+          0.0011098439572378993,
+          0.02180490270256996,
+          0.002579760504886508,
+          0.004731080494821072,
+          -0.013318127952516079,
+          -0.004443933721631765,
+          -0.004279850050806999,
+          -0.0008688460220582783,
+          0.0037830411456525326,
+          0.025651752948760986,
+          -0.004168182145804167,
+          -0.008819499984383583,
+          -0.01894255355000496,
+          -0.017711924389004707,
+          0.020145833492279053,
+          -0.006007287185639143,
+          0.006394707132130861,
+          -0.017420221120119095,
+          0.007789418566972017,
+          0.0064995381981134415,
+          0.018340913578867912,
+          -0.009717402048408985,
+          0.011704638600349426,
+          0.005246120970696211,
+          -0.0020077466033399105,
+          0.011959879659116268,
+          -0.0064995381981134415,
+          0.034038253128528595,
+          -0.014876923523843288,
+          0.009708286263048649,
+          0.0022048749960958958,
+          0.004213761072605848,
+          0.007844112813472748,
+          0.019070174545049667,
+          -0.004863258916884661,
+          -0.012762066908180714,
+          -0.005195984151214361,
+          -0.004863258916884661,
+          0.016800349578261375,
+          0.021841365844011307,
+          0.0046353647485375404,
+          -0.0008283948409371078,
+          0.0027415654622018337,
+          0.006171370856463909,
+          0.009726517833769321,
+          0.019070174545049667,
+          -0.01669096015393734,
+          0.005350952036678791,
+          -0.016353676095604897,
+          -0.009662707336246967,
+          0.00836826954036951,
+          -0.005843203514814377,
+          0.014147662557661533,
+          0.012616214342415333,
+          -0.000646079599391669,
+          -0.0004167612351011485,
+          0.015834078192710876,
+          -0.0021091594826430082,
+          0.006157697178423405,
+          0.012178657576441765,
+          -0.01100272499024868,
+          -0.001858475967310369,
+          -0.02680034004151821,
+          0.008053775876760483,
+          -0.023500433191657066,
+          -0.019471267238259315,
+          0.003286232240498066,
+          -0.025177733972668648,
+          0.005191426258534193,
+          0.018632616847753525,
+          0.01758430525660515,
+          -0.009143109433352947,
+          -0.010191421955823898,
+          -0.0009360748226754367,
+          0.02027345448732376,
+          -0.013555137440562248,
+          0.02032814919948578,
+          -0.002256151055917144,
+          0.004931627307087183,
+          0.007648124359548092,
+          -0.004412028938531876,
+          0.016216941177845,
+          -0.012734719552099705,
+          0.0024430241901427507,
+          -0.012588866986334324,
+          -0.0032702796161174774,
+          0.022151300683617592,
+          -0.02102094702422619,
+          0.027237895876169205,
+          0.009257055819034576,
+          -0.010109379887580872,
+          -0.0008483355632051826,
+          0.004177297931164503,
+          0.0215131975710392,
+          -0.03287143632769585,
+          -0.0010705322492867708,
+          -0.03239741921424866,
+          -0.001720600062981248,
+          -0.00011544257722562179,
+          0.03288966789841652,
+          -0.012251583859324455,
+          -0.019981749355793,
+          -0.002964901505038142,
+          0.006782126612961292,
+          -0.03305375203490257,
+          -0.001533726928755641,
+          0.007133083418011665,
+          -0.0037260677199810743,
+          0.01741110533475876,
+          0.0037807621993124485,
+          0.0022983115632086992,
+          -0.015186859294772148,
+          -0.012452131137251854,
+          0.0192160252481699,
+          0.016007278114557266,
+          -0.007351861800998449,
+          0.013509559445083141,
+          -0.022041911259293556,
+          -0.00624885456636548,
+          0.004491791594773531,
+          0.0030082014854997396,
+          -0.0008990420028567314,
+          0.034603431820869446,
+          0.018532343208789825,
+          -0.005829529836773872,
+          0.007456693332642317,
+          -0.0009639917989261448,
+          0.022096605971455574,
+          0.011841374449431896,
+          0.032452113926410675,
+          -0.00053782993927598,
+          0.03004555031657219,
+          -0.02858702838420868,
+          -0.00673199025914073,
+          -0.005259794648736715,
+          0.01412943098694086,
+          -0.025068344548344612,
+          0.015287132933735847,
+          -3.9169288356788456e-05,
+          0.028915196657180786,
+          0.009562434628605843,
+          0.0014608008787035942,
+          0.004250223748385906,
+          0.006344570312649012,
+          -0.0017376920441165566,
+          0.005241563078016043,
+          -0.0016909738769754767,
+          -0.026399245485663414,
+          -0.023445740342140198,
+          -0.0028122125659137964,
+          -0.018650848418474197,
+          0.014521408826112747,
+          -0.005770277231931686,
+          -0.006773010827600956,
+          -0.011795795522630215,
+          0.015688225626945496,
+          0.010638094507157803,
+          0.00916134100407362,
+          0.001238604192622006,
+          0.021185031160712242,
+          0.01832268200814724,
+          0.01577938348054886,
+          -0.003901546122506261,
+          0.0099908746778965,
+          -0.000999315408989787,
+          0.011485859751701355,
+          0.009699170477688313,
+          -0.011367355473339558,
+          0.020510464906692505,
+          -0.0032748375087976456,
+          -0.01415677834302187,
+          0.0075159454718232155,
+          -0.001373061677441001,
+          0.010638094507157803,
+          -0.02818593569099903,
+          0.0026754760183393955,
+          0.008591605350375175,
+          0.011412933468818665,
+          0.009917949326336384,
+          -0.01565176248550415,
+          -0.014895155094563961,
+          0.02450316771864891,
+          -0.005282584112137556,
+          0.01819506101310253,
+          0.0029762962367385626,
+          0.017912471666932106,
+          0.011914300732314587,
+          0.002582039451226592,
+          -0.021458502858877182,
+          -0.0015006823232397437,
+          0.0022686852607876062,
+          -0.0005760021740570664,
+          -0.008021870627999306,
+          -0.01054693665355444,
+          0.005597077775746584,
+          -0.036535974591970444,
+          0.004371007904410362,
+          -0.015031890943646431,
+          -0.02588876336812973,
+          0.004560159984976053,
+          -0.012935265898704529,
+          0.002186643425375223,
+          0.0032429322600364685,
+          0.02895165979862213,
+          -0.009999990463256836,
+          -0.02824063040316105,
+          1.6219646568060853e-05,
+          0.0030856854282319546,
+          -0.004113487433642149,
+          0.013309012167155743,
+          0.015788499265909195,
+          -0.0020054676569998264,
+          0.0026230604853481054,
+          0.042370062321424484,
+          -0.001320646028034389,
+          -0.006804916076362133,
+          0.00527346832677722,
+          -0.003721509827300906,
+          -0.012297162786126137,
+          -0.0003475384146440774,
+          0.008518679067492485,
+          0.011549670249223709,
+          -0.018587037920951843,
+          -0.00565633038058877,
+          0.010082032531499863,
+          0.030227866023778915,
+          0.017301715910434723,
+          0.013236085884273052,
+          -0.0024863239377737045,
+          0.005054689943790436,
+          -0.0018402443965896964,
+          -0.00023558546672575176,
+          -0.015578837133944035,
+          -0.029334520921111107,
+          -0.01970827765762806,
+          -0.0046991752460598946,
+          -0.01214219443500042,
+          -0.009908833540976048,
+          -0.011631712317466736,
+          0.0077347238548099995,
+          -0.0014608008787035942,
+          0.007014578673988581,
+          0.004357334226369858,
+          -0.010738367214798927,
+          0.010765714570879936,
+          0.022716479375958443,
+          0.017292600125074387,
+          -0.0018470812356099486,
+          0.01269825641065836,
+          -0.003842293517664075,
+          0.004008656367659569,
+          0.002835002029314637,
+          -0.010665440931916237,
+          -0.00978121254593134,
+          -0.0006865308387205005,
+          0.012415667995810509,
+          0.011349123902618885,
+          -0.01513216458261013,
+          0.002525066025555134,
+          0.011896069161593914,
+          0.010720135644078255,
+          0.0032953477930277586,
+          -0.020309917628765106,
+          0.015123048797249794,
+          0.0021205542143434286,
+          0.004703733138740063,
+          -0.0017638999270275235,
+          0.0008722644415684044,
+          -0.004024608992040157,
+          0.014211473055183887,
+          -0.01211484707891941,
+          0.014329977333545685,
+          0.024302620440721512,
+          0.01725613698363304,
+          0.005469457246363163,
+          0.007142199203372002,
+          0.008354595862329006,
+          0.0027278917841613293,
+          -0.009220592677593231,
+          -0.003871919820085168,
+          -0.0018413838697597384,
+          -0.00596626615151763,
+          0.023099340498447418,
+          -0.0092251505702734,
+          0.005414762534201145,
+          0.018906090408563614,
+          -0.008997256867587566,
+          -0.006467632949352264,
+          0.009279845282435417,
+          -0.002962622558698058,
+          -0.012187773361802101,
+          0.013263433240354061,
+          -0.002878301776945591,
+          0.01030081044882536,
+          0.014530524611473083,
+          -0.0013844562927260995,
+          -0.0022504536900669336,
+          -0.00414767162874341,
+          -0.0074475775472819805,
+          0.06745664030313492,
+          -0.015123048797249794,
+          0.004466723185032606,
+          -0.010984493419528008,
+          -0.017155863344669342,
+          0.014530524611473083,
+          0.02027345448732376,
+          0.02942567877471447,
+          -0.010701904073357582,
+          0.003972193226218224,
+          0.000644940126221627,
+          0.0017958050593733788,
+          0.018085671588778496,
+          0.006800358183681965,
+          -0.0009081577882170677,
+          -0.017265252768993378,
+          -0.01842295378446579,
+          -0.015241554006934166,
+          0.00473563838750124,
+          -0.01881493255496025,
+          0.0038149463944137096,
+          0.0046991752460598946,
+          -0.001347993267700076,
+          -0.013783032074570656,
+          0.005879666190594435,
+          0.003363716183230281,
+          -0.020510464906692505,
+          -0.0018744284752756357,
+          -0.008049217984080315,
+          -0.04740196093916893,
+          0.017721040174365044,
+          -0.005820414051413536,
+          0.011722870171070099,
+          -0.0032429322600364685,
+          0.016800349578261375,
+          0.007292609661817551,
+          0.004040561616420746,
+          0.005241563078016043,
+          0.009635359980165958,
+          0.009562434628605843,
+          0.013090234249830246,
+          -0.00982679147273302,
+          0.013072002679109573,
+          0.013637179508805275,
+          -0.007939828559756279,
+          -0.004498628433793783,
+          0.013454864732921124,
+          -0.006125791929662228,
+          -0.0011434584157541394,
+          0.0047584278509020805,
+          0.009744749404489994,
+          0.0062807598151266575,
+          -0.0013685037847608328,
+          0.004858701024204493,
+          -0.0004552183672785759,
+          0.02824063040316105,
+          0.0019040547776967287,
+          -0.00737465126439929,
+          -0.015496795065701008,
+          -0.01800362952053547,
+          -0.022096605971455574,
+          -0.010802177712321281,
+          -0.014074736274778843,
+          0.002639013109728694,
+          -0.05677296593785286,
+          0.022716479375958443,
+          0.0028144915122538805,
+          0.0037465780042111874,
+          -0.0024339084047824144,
+          -0.013810379430651665,
+          0.030665423721075058,
+          -0.004749312065541744,
+          -0.022096605971455574,
+          -0.01602550968527794,
+          0.011941648088395596,
+          0.002258430002257228,
+          0.0018539180746302009,
+          -0.009863254614174366,
+          -0.0026731970719993114,
+          -0.0013673643115907907,
+          0.006736548151820898,
+          0.014038273133337498,
+          0.005478573031723499,
+          0.007164988666772842,
+          0.02393799088895321,
+          0.007201451808214188,
+          -0.01711028441786766,
+          0.007538734935224056,
+          -0.008281669579446316,
+          0.0039995405822992325,
+          -0.005765719339251518,
+          -0.002180946059525013,
+          0.003687325632199645,
+          -0.00339334225282073,
+          -0.0075478507205843925,
+          -0.011349123902618885,
+          0.002987690968438983,
+          0.004881490487605333,
+          -0.003343205666169524,
+          -0.002072696341201663,
+          0.01467637624591589,
+          0.0217137448489666,
+          0.015423868782818317,
+          -0.0008209883235394955,
+          -0.005628983024507761,
+          0.014320861548185349,
+          0.0107565987855196,
+          0.005319047253578901,
+          -0.007898807525634766,
+          -0.006513211876153946,
+          0.007092062849551439,
+          -0.01738375797867775,
+          0.0050774794071912766,
+          -0.01950773037970066,
+          0.0003783041029237211,
+          0.005852319300174713,
+          0.016681844368577003,
+          -0.0005318477051332593,
+          -0.006162255071103573,
+          0.0037374624516814947,
+          0.0077666291035711765,
+          0.003691883524879813,
+          0.01515951193869114,
+          0.014357324689626694,
+          -0.010273464024066925,
+          0.022114837542176247,
+          0.000282588618574664,
+          -0.007493156008422375,
+          -0.0023814928717911243,
+          -0.0027051023207604885,
+          0.01985413022339344,
+          -0.0009167037787847221,
+          0.015123048797249794,
+          -0.006435727700591087,
+          0.020802168175578117,
+          0.007206009700894356,
+          0.004767543636262417,
+          0.025797605514526367,
+          0.01651776023209095,
+          0.002832723082974553,
+          0.012360973283648491,
+          0.08350037783384323,
+          0.018240639939904213,
+          -0.010975377634167671,
+          -0.017547842115163803,
+          -0.0047265226021409035,
+          -0.0071695465594530106,
+          0.003413852769881487,
+          0.017493147403001785,
+          -0.017292600125074387,
+          -0.023354582488536835,
+          0.0006494980189017951,
+          0.000889356539119035,
+          0.0037875990383327007,
+          -0.01554237399250269,
+          0.01780308224260807,
+          0.015296248719096184,
+          -0.0013821773463860154,
+          0.0046376436948776245,
+          0.005765719339251518,
+          -0.007593429647386074,
+          -0.007620777003467083,
+          0.010401084087789059,
+          -0.015387405641376972,
+          -0.014102083630859852,
+          0.01190518494695425,
+          -0.004876932594925165,
+          -0.00021450527128763497,
+          0.004020051099359989,
+          0.010082032531499863,
+          -0.00183112861122936,
+          0.015287132933735847,
+          0.02353689633309841,
+          0.004532812628895044,
+          0.003794435877352953,
+          0.015268901363015175,
+          0.01530536450445652,
+          -0.00046746767475269735,
+          -0.0036166785284876823,
+          -0.01671830751001835,
+          0.009626244194805622,
+          0.006845937110483646,
+          -0.015323596075177193,
+          0.016216941177845,
+          0.02193252369761467,
+          0.006257970351725817,
+          0.006485864520072937,
+          0.00468550156801939,
+          -0.0004227434692438692,
+          0.01916133053600788,
+          0.000195134271052666,
+          -0.0034571527503430843,
+          0.006230622995644808,
+          0.007520503364503384,
+          -0.012898802757263184,
+          0.017839545384049416,
+          0.013664526864886284,
+          0.012379204854369164,
+          0.007319956552237272,
+          0.0037853200919926167,
+          0.010720135644078255,
+          -0.022443005815148354,
+          0.016508644446730614,
+          -0.018514111638069153
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "8f4dd4ba-f551-4c79-af63-003b9ccf3976",
+      "properties": {
+        "page_content": "8 Future Work 26 ",
+        "keyphrases": [],
+        "embedding": [
+          -0.024294501170516014,
+          0.01452074758708477,
+          -0.01423195656388998,
+          0.00711597828194499,
+          -0.015378094278275967,
+          0.004546194337308407,
+          -0.010901841334998608,
+          0.11226730793714523,
+          0.006362415850162506,
+          0.04544840380549431,
+          0.02425840124487877,
+          5.9330373915145174e-05,
+          -0.00413782661780715,
+          -0.018969925120472908,
+          -0.018825531005859375,
+          0.03312066197395325,
+          -0.018969925120472908,
+          0.014204883016645908,
+          -0.029601026326417923,
+          -0.0012081821914762259,
+          -0.01162381749600172,
+          -0.0036031126510351896,
+          -0.03490755334496498,
+          0.01664155349135399,
+          -0.001300685340538621,
+          0.011109409853816032,
+          0.04822801426053047,
+          0.030413249507546425,
+          -0.05245157331228256,
+          0.018013307824730873,
+          -0.007968813180923462,
+          0.023933513090014458,
+          0.019258717074990273,
+          -0.005049321334809065,
+          0.012463115155696869,
+          0.006443637888878584,
+          0.027326801791787148,
+          -0.008451634086668491,
+          0.0013762671733275056,
+          0.012914350256323814,
+          -0.0050673708319664,
+          -0.00494553754106164,
+          -0.009110437706112862,
+          0.026081392541527748,
+          0.035539280623197556,
+          -0.01277897972613573,
+          -0.012950449250638485,
+          -0.04808361828327179,
+          0.0016458802856504917,
+          0.007197200786322355,
+          0.0050809080712497234,
+          -0.022200770676136017,
+          0.001969641540199518,
+          0.0006689561414532363,
+          0.02433059923350811,
+          -0.00736415758728981,
+          0.012986548244953156,
+          0.07804563641548157,
+          0.020558273419737816,
+          0.007833442650735378,
+          -0.022200770676136017,
+          -0.02465548925101757,
+          -0.0032579179387539625,
+          -0.03779545798897743,
+          0.016109095886349678,
+          -0.02696581371128559,
+          0.029564928263425827,
+          -0.011903583072125912,
+          0.009882049635052681,
+          0.0025224045384675264,
+          -0.0248179342597723,
+          -0.005893131252378225,
+          -0.007359645329415798,
+          0.04508741572499275,
+          0.00730549730360508,
+          -0.017210109159350395,
+          0.026370182633399963,
+          0.018753333017230034,
+          -0.02730875089764595,
+          0.046711862087249756,
+          0.01647910848259926,
+          -0.03963649645447731,
+          -0.005058346316218376,
+          0.021496843546628952,
+          0.024529142305254936,
+          -0.004372468683868647,
+          -0.04992465674877167,
+          -0.11732114106416702,
+          0.016452033072710037,
+          0.0011562900617718697,
+          -0.005071883089840412,
+          0.015332970768213272,
+          0.0010463015642017126,
+          -0.028680507093667984,
+          -0.04238000512123108,
+          0.04259659722447395,
+          -0.00870432611554861,
+          0.010405482724308968,
+          -0.0004030094132758677,
+          -0.020215334370732307,
+          0.01059500128030777,
+          0.005374210886657238,
+          0.005992402788251638,
+          -0.02342812903225422,
+          0.0422356091439724,
+          -0.02474573627114296,
+          0.019872397184371948,
+          0.015026130713522434,
+          0.0034542051143944263,
+          -0.002572040306404233,
+          0.005545679945498705,
+          -0.033644095063209534,
+          -0.022327115759253502,
+          0.018148677423596382,
+          -0.006168384570628405,
+          -0.01757109723985195,
+          0.013952190987765789,
+          -0.03044934757053852,
+          -0.029474681243300438,
+          0.04494302347302437,
+          0.04559279978275299,
+          0.019511409103870392,
+          0.027687789872288704,
+          -0.03633345663547516,
+          -0.032073795795440674,
+          0.02905954420566559,
+          0.012327744625508785,
+          0.00182637432590127,
+          -0.003952819854021072,
+          0.03254307806491852,
+          0.02994396537542343,
+          0.004620647989213467,
+          -0.02211052179336548,
+          -0.03440216928720474,
+          0.021478792652487755,
+          -0.02243541181087494,
+          -0.009548136033117771,
+          0.024186205118894577,
+          -0.0227603018283844,
+          0.01911432109773159,
+          0.004505583085119724,
+          0.004850777797400951,
+          -0.011632842011749744,
+          0.0054193343967199326,
+          -0.006348878610879183,
+          0.03633345663547516,
+          -0.014078537002205849,
+          0.023067140951752663,
+          0.003984406590461731,
+          0.050466138869524,
+          -0.01692131906747818,
+          0.00340231298469007,
+          -0.01360925193876028,
+          0.015712007880210876,
+          0.0010214835638180375,
+          0.0005175103433430195,
+          0.000724796496797353,
+          0.024872081354260445,
+          0.014502697624266148,
+          0.012291645631194115,
+          0.02424035221338272,
+          0.03005226142704487,
+          -0.013672425411641598,
+          0.0012894044630229473,
+          0.005112494342029095,
+          -0.020504126325249672,
+          0.0355934277176857,
+          -0.01125380489975214,
+          -0.04353516921401024,
+          0.03377043828368187,
+          0.05858837440609932,
+          -0.01819380186498165,
+          0.01096501387655735,
+          -0.01952945813536644,
+          0.000879344530403614,
+          -0.0008167356136254966,
+          -0.014917834661900997,
+          0.03696518391370773,
+          -0.028102925047278404,
+          -0.019998742267489433,
+          -0.010802569799125195,
+          0.016677651554346085,
+          -0.007828930392861366,
+          0.003255661576986313,
+          0.01038743369281292,
+          -0.006457175128161907,
+          0.027417048811912537,
+          -0.037614963948726654,
+          0.052595969289541245,
+          0.013573153875768185,
+          0.01576615683734417,
+          -0.030503496527671814,
+          0.026009194552898407,
+          0.013419733382761478,
+          0.00960228405892849,
+          -0.06082649901509285,
+          -0.0021546478383243084,
+          0.026514578610658646,
+          0.004909438546746969,
+          -0.015937626361846924,
+          -0.014782464131712914,
+          -0.01398828998208046,
+          0.012273596599698067,
+          -0.007657460402697325,
+          -0.000787969387602061,
+          0.02068462036550045,
+          0.018500640988349915,
+          0.023969611153006554,
+          0.00773868290707469,
+          -0.024619389325380325,
+          0.023049091920256615,
+          -0.0033887759782373905,
+          -0.010910865850746632,
+          0.005360673647373915,
+          -0.008514807559549809,
+          -0.005405797157436609,
+          -0.02044997736811638,
+          0.04905828461050987,
+          -0.00019558223721105605,
+          -0.04313807934522629,
+          -0.007720633409917355,
+          0.0005521990242414176,
+          0.009854976087808609,
+          -0.004986148327589035,
+          -0.004424360580742359,
+          -0.03593636676669121,
+          -0.009854976087808609,
+          0.06310072541236877,
+          -0.02582870051264763,
+          -0.02730875089764595,
+          -0.01687619462609291,
+          0.004620647989213467,
+          0.0031721831765025854,
+          -0.028157074004411697,
+          -0.018807481974363327,
+          -0.0032285875640809536,
+          -0.012057003565132618,
+          -0.011524545960128307,
+          -0.025341367349028587,
+          0.022724201902747154,
+          0.05815518647432327,
+          0.013879992999136448,
+          -0.038842324167490005,
+          0.015278822742402554,
+          -0.013022646307945251,
+          -0.02059437334537506,
+          -0.04526790976524353,
+          0.09941612929105759,
+          0.023807166144251823,
+          0.003623418277129531,
+          0.0062270453199744225,
+          -0.00327822333201766,
+          0.0001518688368378207,
+          0.018717234954237938,
+          0.04006968066096306,
+          0.07746805250644684,
+          0.04956366866827011,
+          -0.007409281097352505,
+          -0.0322723388671875,
+          0.003966357093304396,
+          -0.021226102486252785,
+          -0.019006025046110153,
+          -0.045556701719760895,
+          -0.0063308291137218475,
+          -0.00036803868715651333,
+          -0.016605453565716743,
+          0.01588347740471363,
+          0.033553846180438995,
+          0.01112745888531208,
+          -0.005292988382279873,
+          0.04429324343800545,
+          0.005144080612808466,
+          0.014250006526708603,
+          -0.0033210907131433487,
+          -0.011831386014819145,
+          -0.02012508735060692,
+          -0.014376352541148663,
+          -0.0018714978359639645,
+          -0.008961530402302742,
+          0.015296871773898602,
+          -0.013419733382761478,
+          0.05848007649183273,
+          0.006195458583533764,
+          -0.006764015182852745,
+          -0.05191009119153023,
+          -0.03360799327492714,
+          -0.010540853254497051,
+          0.021749533712863922,
+          -0.0235725250095129,
+          -0.012381892651319504,
+          0.02202027477324009,
+          -0.0120028555393219,
+          -0.020486075431108475,
+          0.03530463948845863,
+          0.007066342514008284,
+          -0.028915148228406906,
+          0.005202741362154484,
+          0.020215334370732307,
+          -0.011867485009133816,
+          -0.0019312864169478416,
+          0.02341008000075817,
+          0.003925745841115713,
+          -0.025233069434762,
+          -0.004307039547711611,
+          0.005947279278188944,
+          0.021984176710247993,
+          0.015405167825520039,
+          0.025178922340273857,
+          -0.05801079049706459,
+          -0.014430500566959381,
+          0.043823957443237305,
+          -0.04173022881150246,
+          -0.020702669396996498,
+          -0.045881591737270355,
+          -0.016127144917845726,
+          0.03407727926969528,
+          0.028915148228406906,
+          0.02738094888627529,
+          -0.04660356789827347,
+          -0.03783155605196953,
+          -0.031063027679920197,
+          -0.00024550012312829494,
+          -0.05306525528430939,
+          0.030178606510162354,
+          0.0006040910957381129,
+          -0.0036865912843495607,
+          -0.023049091920256615,
+          0.049527570605278015,
+          0.01749889925122261,
+          -0.005482506938278675,
+          0.004674796015024185,
+          -0.007652948144823313,
+          0.018058430403470993,
+          -0.006592545658349991,
+          0.010640124790370464,
+          -0.034384120255708694,
+          -0.02878880314528942,
+          0.03631540387868881,
+          0.00046590028796344995,
+          -0.018112579360604286,
+          0.040502868592739105,
+          0.005446408409625292,
+          0.015179550275206566,
+          0.02218271978199482,
+          -0.018347222357988358,
+          -0.03573782369494438,
+          0.041405338793992996,
+          -0.05086322873830795,
+          0.004361187573522329,
+          0.0067188916727900505,
+          0.000985948834568262,
+          -0.04006968066096306,
+          -0.00813576951622963,
+          0.019655803218483925,
+          0.010179865173995495,
+          -0.044401541352272034,
+          -0.015305896289646626,
+          -0.004252891521900892,
+          0.021785633638501167,
+          -0.014340253546833992,
+          0.01498100720345974,
+          -0.04772263020277023,
+          -0.030070310458540916,
+          0.0056900754570961,
+          -0.05819128453731537,
+          -0.021406596526503563,
+          0.029691273346543312,
+          0.008726888336241245,
+          -0.013961215503513813,
+          -0.005094444844871759,
+          -0.03469095751643181,
+          0.013816820457577705,
+          0.02664092369377613,
+          -0.02804877795279026,
+          -0.00885323341935873,
+          -0.010739396326243877,
+          0.01009864266961813,
+          0.017950134351849556,
+          -0.04003358259797096,
+          0.022146621719002724,
+          -0.021659286692738533,
+          0.01274288073182106,
+          -0.014502697624266148,
+          -0.02994396537542343,
+          -0.01675887405872345,
+          -0.01637081243097782,
+          -0.01224652212113142,
+          0.02010703831911087,
+          -0.006646693684160709,
+          0.0017428958090022206,
+          0.002045223256573081,
+          -0.0347631573677063,
+          -0.0073867193423211575,
+          -0.027435097843408585,
+          0.03667639195919037,
+          0.027001911774277687,
+          0.02954687736928463,
+          -0.0042754532769322395,
+          0.018157701939344406,
+          -0.02979956939816475,
+          -0.028500013053417206,
+          -0.008420048281550407,
+          -0.0028269882313907146,
+          0.020052891224622726,
+          0.0463508740067482,
+          0.021875880658626556,
+          -0.0029646148905158043,
+          0.011632842011749744,
+          0.006800113711506128,
+          0.004968099296092987,
+          0.025251120328903198,
+          -0.03467290848493576,
+          0.011867485009133816,
+          -0.0421995110809803,
+          0.04981636255979538,
+          0.037542764097452164,
+          0.04310198128223419,
+          0.04104435071349144,
+          -0.03171280771493912,
+          0.0012860201532021165,
+          -0.0017541766865178943,
+          -0.01046865526586771,
+          0.03180305287241936,
+          -0.008650178089737892,
+          0.008907382376492023,
+          0.034546565264463425,
+          0.00457101222127676,
+          -0.013663400895893574,
+          -0.011578693985939026,
+          -0.023969611153006554,
+          -0.021550990641117096,
+          -0.000988204963505268,
+          0.025846749544143677,
+          -0.018347222357988358,
+          0.012327744625508785,
+          -0.02416815422475338,
+          0.004546194337308407,
+          -0.0022155647166073322,
+          -0.005121518857777119,
+          -0.025467712432146072,
+          0.014376352541148663,
+          -0.02631603367626667,
+          -0.002851806115359068,
+          0.03534073755145073,
+          0.006633156910538673,
+          -0.0163437370210886,
+          -0.04526790976524353,
+          -0.005414821673184633,
+          0.0007496143807657063,
+          0.00883067212998867,
+          -0.008478708565235138,
+          0.03772325813770294,
+          -0.040827758610248566,
+          -0.023319832980632782,
+          0.039744794368743896,
+          -0.015468341298401356,
+          -0.004737969022244215,
+          0.007720633409917355,
+          -0.02211052179336548,
+          -0.038842324167490005,
+          -0.0004580036911647767,
+          0.05786639824509621,
+          0.034709006547927856,
+          -0.002942053135484457,
+          -0.022363213822245598,
+          0.0013954447349533439,
+          -0.03815644606947899,
+          -0.014944908209145069,
+          -0.0017113093053922057,
+          0.05107982084155083,
+          0.003921233583241701,
+          -0.009575209580361843,
+          0.0023464227560907602,
+          -0.003269198816269636,
+          -0.021966127678751945,
+          0.016397885978221893,
+          0.01187650952488184,
+          -0.005450920667499304,
+          -0.018347222357988358,
+          0.005008710082620382,
+          -0.008911894634366035,
+          0.009412765502929688,
+          -0.027579491958022118,
+          -0.016072995960712433,
+          0.017354503273963928,
+          0.025052575394511223,
+          0.03180305287241936,
+          0.02017923630774021,
+          0.018590888008475304,
+          0.011118434369564056,
+          0.045809391885995865,
+          -0.013419733382761478,
+          0.001081836293451488,
+          0.006033014040440321,
+          -0.01724620722234249,
+          -0.042740993201732635,
+          0.006412051618099213,
+          -0.07840662449598312,
+          -0.004950049798935652,
+          0.03180305287241936,
+          -0.022236868739128113,
+          0.017607195302844048,
+          -0.0008167356136254966,
+          0.02880685217678547,
+          0.016632527112960815,
+          -0.027778036892414093,
+          -0.004611623473465443,
+          -0.015134426765143871,
+          0.011010137386620045,
+          0.012174324132502079,
+          0.0008443737751804292,
+          0.021984176710247993,
+          0.00013847279478795826,
+          0.00972863007336855,
+          -0.009295444004237652,
+          0.012625560164451599,
+          0.005144080612808466,
+          0.011037211865186691,
+          -0.021226102486252785,
+          0.006109723821282387,
+          -0.00761233689263463,
+          -0.000742281845305115,
+          0.008451634086668491,
+          0.03624320775270462,
+          -0.03862572833895683,
+          0.0006170640699565411,
+          -0.03873402625322342,
+          -0.008189918473362923,
+          0.06941801309585571,
+          -0.03167670965194702,
+          -0.01505320519208908,
+          0.006610595155507326,
+          -0.00080432667164132,
+          0.003090960904955864,
+          0.002966871252283454,
+          0.027850233018398285,
+          -0.043499067425727844,
+          0.019060172140598297,
+          -0.015332970768213272,
+          -0.026279935613274574,
+          -0.02433059923350811,
+          0.021370496600866318,
+          0.03275967389345169,
+          0.00581190874800086,
+          0.00792820192873478,
+          0.0388784222304821,
+          0.01985434629023075,
+          -0.031135225668549538,
+          0.01411463599652052,
+          -0.014222932048141956,
+          -0.0028269882313907146,
+          0.014268055558204651,
+          0.015802254900336266,
+          -0.014674167148768902,
+          0.012047979049384594,
+          0.0005457125371322036,
+          0.0025878336746245623,
+          0.015035155229270458,
+          0.021695386618375778,
+          0.0051892041228711605,
+          0.012842152267694473,
+          -0.020251434296369553,
+          -0.006633156910538673,
+          0.019150419160723686,
+          -0.010360359214246273,
+          -0.03923941031098366,
+          -0.0016537768533453345,
+          0.048877790570259094,
+          -0.03481730446219444,
+          -0.019511409103870392,
+          0.004349906928837299,
+          0.032055746763944626,
+          -0.009647407568991184,
+          -0.03481730446219444,
+          -0.012201398611068726,
+          0.022200770676136017,
+          0.014439525082707405,
+          -0.016722774133086205,
+          -0.03552123159170151,
+          0.0044401539489626884,
+          -0.004094959236681461,
+          -0.007440867833793163,
+          -0.004056604113429785,
+          0.016776923090219498,
+          0.005617877934128046,
+          -0.0017801227513700724,
+          -0.011768212541937828,
+          -0.02564820647239685,
+          -0.04945537447929382,
+          0.02308518998324871,
+          -0.0063082673586905,
+          0.019475309178233147,
+          0.01059500128030777,
+          -0.003131571924313903,
+          -0.03840913623571396,
+          0.010053519159555435,
+          -0.002939797006547451,
+          0.004234842024743557,
+          0.042488303035497665,
+          -0.037037380039691925,
+          0.00377683830447495,
+          0.023049091920256615,
+          -0.016154218465089798,
+          -0.0248179342597723,
+          0.00680913869291544,
+          -0.01299557276070118,
+          0.002973639639094472,
+          0.03451046347618103,
+          -0.01604592241346836,
+          0.0260633435100317,
+          0.02035973034799099,
+          0.0007941738585941494,
+          0.05006905272603035,
+          0.00674596568569541,
+          0.006118748802691698,
+          -0.006290218327194452,
+          0.014719290658831596,
+          -0.0037226900458335876,
+          -0.014349278062582016,
+          0.0008415535558015108,
+          0.0310269296169281,
+          -0.013067769818007946,
+          -0.01514345221221447,
+          -0.006204483564943075,
+          0.03461876139044762,
+          -0.005960816517472267,
+          0.007720633409917355,
+          -0.03797595202922821,
+          -0.006064600311219692,
+          0.00780185591429472,
+          0.020756816491484642,
+          -0.00798686221241951,
+          0.012463115155696869,
+          0.0011811080621555448,
+          -0.010161816142499447,
+          0.010441581718623638,
+          0.033553846180438995,
+          -0.00027962480089627206,
+          0.012806054204702377,
+          -0.011641867458820343,
+          -0.017137911170721054,
+          -0.011046236380934715,
+          0.018518690019845963,
+          0.0612235851585865,
+          -0.03321090713143349,
+          0.020305581390857697,
+          0.010640124790370464,
+          0.0044446662068367004,
+          0.015179550275206566,
+          0.016975466161966324,
+          0.020666569471359253,
+          0.0019245179137215018,
+          -0.007337083574384451,
+          0.03400508314371109,
+          0.018735283985733986,
+          -0.014583920128643513,
+          -0.0007501784712076187,
+          0.0025946020614355803,
+          -0.05219888314604759,
+          0.0003074039414059371,
+          -0.02216467075049877,
+          0.0019211337203159928,
+          -0.000984820770099759,
+          -0.011632842011749744,
+          0.002883392618969083,
+          0.017688417807221413,
+          0.004706382751464844,
+          0.01162381749600172,
+          -0.029853718355298042,
+          -0.009322518482804298,
+          0.0017553047509863973,
+          -0.021460743620991707,
+          0.04313807934522629,
+          0.037290073931217194,
+          -0.04653136804699898,
+          -0.01795915886759758,
+          0.004638697486370802,
+          0.049274880439043045,
+          0.0011342923389747739,
+          0.002632957184687257,
+          -0.029492730274796486,
+          -0.010567927733063698,
+          0.015278822742402554,
+          0.006705354433506727,
+          0.0028630869928747416,
+          0.017950134351849556,
+          -0.013455832377076149,
+          0.018040381371974945,
+          -0.005893131252378225,
+          0.008808109909296036,
+          -0.015666885301470757,
+          0.029727371409535408,
+          -0.011118434369564056,
+          0.000694338115863502,
+          0.01662350259721279,
+          0.018572838976979256,
+          -0.031171325594186783,
+          0.013356560841202736,
+          0.03176695480942726,
+          0.013870968483388424,
+          -0.018184777349233627,
+          -0.015594687312841415,
+          -0.044473737478256226,
+          0.026334084570407867,
+          0.007648435886949301,
+          0.0016470083501189947,
+          -0.00259685842320323,
+          -0.020161187276244164,
+          -0.018275024369359016,
+          -0.027038009837269783,
+          -0.01567590981721878,
+          -0.010694272816181183,
+          0.005139568354934454,
+          0.00446045957505703,
+          0.003928001970052719,
+          0.017814764752984047,
+          0.007034756243228912,
+          -0.02059437334537506,
+          -0.016100069507956505,
+          -0.014710266143083572,
+          0.0045304009690880775,
+          -0.024384748190641403,
+          -0.048047520220279694,
+          -0.03452851250767708,
+          -0.015450291335582733,
+          0.020504126325249672,
+          0.030918633565306664,
+          -0.006353390868753195,
+          0.011262829415500164,
+          -0.010730371810495853,
+          0.01472831517457962,
+          -0.0003917285066563636,
+          -0.0055772666819393635,
+          0.003946051467210054,
+          -0.012896301224827766,
+          0.03270552307367325,
+          0.01022498868405819,
+          0.021893929690122604,
+          0.010071569122374058,
+          -0.004316064063459635,
+          0.029402483254671097,
+          0.013636326417326927,
+          -0.013248263858258724,
+          0.0024005710147321224,
+          0.013645350933074951,
+          -0.006533884909003973,
+          0.048191916197538376,
+          -0.0014304154319688678,
+          -0.019078223034739494,
+          0.009313493967056274,
+          0.007562701124697924,
+          -0.0363876037299633,
+          0.05389552563428879,
+          -0.01178626250475645,
+          -0.0281209759414196,
+          0.010802569799125195,
+          0.01763427071273327,
+          -0.02581065148115158,
+          -0.006023989524692297,
+          0.015170525759458542,
+          -0.021388545632362366,
+          -0.0033662142232060432,
+          0.019258717074990273,
+          0.03617101162672043,
+          0.014944908209145069,
+          0.0013785234186798334,
+          -0.026677021756768227,
+          -0.017020590603351593,
+          0.023861315101385117,
+          -0.022200770676136017,
+          0.02516087330877781,
+          0.0013401684118434787,
+          -0.002479537157341838,
+          -0.02664092369377613,
+          0.009845951572060585,
+          -0.014565871097147465,
+          0.015666885301470757,
+          -0.025142822414636612,
+          0.015044179745018482,
+          0.008988603949546814,
+          0.00914202444255352,
+          0.015360044315457344,
+          -0.05277646332979202,
+          -0.03174890577793121,
+          -0.030900582671165466,
+          0.014746365137398243,
+          0.028355617076158524,
+          -0.03093668259680271,
+          -0.016569355502724648,
+          0.012887275777757168,
+          0.006452662870287895,
+          0.015278822742402554,
+          -0.012273596599698067,
+          4.801424074685201e-05,
+          0.02342812903225422,
+          -0.03270552307367325,
+          -0.013762672431766987,
+          0.0297815203666687,
+          0.006890360731631517,
+          -0.012562386691570282,
+          0.040936052799224854,
+          0.007571726106107235,
+          -0.03467290848493576,
+          -0.0036008565220981836,
+          -0.01303167175501585,
+          -0.0023396543692797422,
+          -0.005942767020314932,
+          0.005256889387965202,
+          0.008528344333171844,
+          0.012174324132502079,
+          0.0060736252926290035,
+          -0.023482277989387512,
+          0.02781413495540619,
+          0.005974353291094303,
+          0.014629043638706207,
+          0.01812160387635231,
+          -0.014638068154454231,
+          0.007513065356761217,
+          0.005653976462781429,
+          0.0037881191819906235,
+          0.013248263858258724,
+          0.0013954447349533439,
+          0.05891326069831848,
+          -0.014015363529324532,
+          0.005536655429750681,
+          0.0513325110077858,
+          0.011795287020504475,
+          -0.002567528048530221,
+          0.020468026399612427,
+          0.023789117112755775,
+          -0.013167042285203934,
+          -0.01766134425997734,
+          -0.011686990968883038,
+          -0.0031744393054395914,
+          -0.011389175429940224,
+          0.03146011382341385,
+          0.002407339634373784,
+          0.02158709056675434,
+          -0.03153231367468834,
+          0.03568367660045624,
+          0.006425588857382536,
+          -0.011307952925562859,
+          0.0132392393425107,
+          -0.0244208462536335,
+          -0.019493358209729195,
+          -0.006646693684160709,
+          -0.02472768723964691,
+          -0.006159360054880381,
+          -0.007106953766196966,
+          0.0310269296169281,
+          0.025918947532773018,
+          0.008537368848919868,
+          0.005667513702064753,
+          0.003379751229658723,
+          -0.0239876601845026,
+          -0.013230214826762676,
+          0.005568241700530052,
+          0.011064286343753338,
+          0.03451046347618103,
+          0.021334398537874222,
+          -0.0032669424545019865,
+          -0.006136798299849033,
+          -0.007806368172168732,
+          -0.011524545960128307,
+          0.017002541571855545,
+          -0.02887905016541481,
+          -0.03261527791619301,
+          -0.014078537002205849,
+          -0.003988918848335743,
+          -0.0025788089260458946,
+          0.00024634620058350265,
+          0.021226102486252785,
+          0.004410823807120323,
+          0.021532941609621048,
+          0.0016210622852668166,
+          0.012093102559447289,
+          -0.02664092369377613,
+          -0.007355133071541786,
+          0.031893301755189896,
+          0.0021501355804502964,
+          -0.014313179068267345,
+          0.013879992999136448,
+          -0.012860202230513096,
+          -0.012842152267694473,
+          -0.016749849542975426,
+          0.029925916343927383,
+          0.0033910321071743965,
+          -0.03423972427845001,
+          -0.010712322778999805,
+          -0.023698870092630386,
+          0.0020170211791992188,
+          -0.004401798825711012,
+          0.02615359053015709,
+          -0.03855353221297264,
+          0.0010874767322093248,
+          0.014881735667586327,
+          -0.005667513702064753,
+          -0.010306211188435555,
+          0.00447174021974206,
+          0.004006968345493078,
+          -0.017923060804605484,
+          -0.010495729744434357,
+          -0.012950449250638485,
+          -0.020468026399612427,
+          -0.009132999926805496,
+          -0.010243037715554237,
+          0.013690474443137646,
+          0.010784519836306572,
+          -0.008266627788543701,
+          0.008893844671547413,
+          -0.002168184844776988,
+          -0.02622578665614128,
+          -0.020088989287614822,
+          -0.022886646911501884,
+          0.03552123159170151,
+          -0.0030413249041885138,
+          -0.03837303817272186,
+          -0.0033323715906590223,
+          -0.0018940595909953117,
+          0.0011529058683663607,
+          0.0067279161885380745,
+          -0.01352803036570549,
+          -0.0182118508964777,
+          0.013753647916018963,
+          0.007237812038511038,
+          -0.013546079397201538,
+          0.011650891974568367,
+          0.026514578610658646,
+          0.017787689343094826,
+          0.02341008000075817,
+          0.011226730421185493,
+          0.037614963948726654,
+          0.017300356179475784,
+          -0.002073425566777587,
+          0.043065883219242096,
+          -0.001112294616177678,
+          -0.01844649389386177,
+          0.046314775943756104,
+          0.00023196307301986963,
+          -0.0017214621184393764,
+          0.027579491958022118,
+          -0.00013339638826437294,
+          0.0037046405486762524,
+          0.011668941006064415,
+          -0.027272652834653854,
+          0.005117006599903107,
+          0.004706382751464844,
+          0.03655004873871803,
+          0.005717149470001459,
+          -0.017760615795850754,
+          -0.022381264716386795,
+          0.012643609195947647,
+          0.018663085997104645,
+          -0.0011258317390456796,
+          0.014006339013576508,
+          0.006786576937884092,
+          -0.03920331224799156,
+          0.015784205868840218,
+          -0.0069760954938828945,
+          0.010622075758874416,
+          0.005762272980064154,
+          0.005153105594217777,
+          -0.016172267496585846,
+          0.0007783806649968028,
+          0.009575209580361843,
+          0.03815644606947899,
+          -0.013094844296574593,
+          0.03331920504570007,
+          -0.032308436930179596,
+          -0.009818877093493938,
+          -0.025846749544143677,
+          0.00782441720366478,
+          0.011542594991624355,
+          0.007355133071541786,
+          -0.042416103184223175,
+          0.005103469826281071,
+          -0.011479422450065613,
+          0.0010835283901542425,
+          -0.002407339634373784,
+          0.008203455246984959,
+          0.04097215086221695,
+          -0.02299494296312332,
+          0.025955045595765114,
+          -0.007350620813667774,
+          0.01837429590523243,
+          -0.0002003766130656004,
+          -0.009191660210490227,
+          -0.019728001207113266,
+          -0.004864315036684275,
+          0.003530915128067136,
+          -0.000880472594872117,
+          -0.007567213382571936,
+          -3.902479147654958e-05,
+          -0.005035784561187029,
+          0.0045394254848361015,
+          -0.006520348135381937,
+          0.01597372442483902,
+          -0.01729133166372776,
+          0.04227171093225479,
+          0.001362730166874826,
+          -0.011190632358193398,
+          -0.0151073532178998,
+          -0.03402313217520714,
+          0.0027705838438123465,
+          -0.006655718665570021,
+          0.02696581371128559,
+          0.014674167148768902,
+          0.020919261500239372,
+          -0.0008658074657432735,
+          0.008785548619925976,
+          0.020648520439863205,
+          -0.017850862815976143,
+          0.024944279342889786,
+          -0.01807648129761219,
+          0.002287762239575386,
+          -0.0040430668741464615,
+          0.016533255577087402,
+          0.01869918406009674,
+          0.01398828998208046,
+          0.0020847064442932606,
+          -0.008591517806053162,
+          -0.032001595944166183,
+          -0.008095158264040947,
+          -0.005893131252378225,
+          0.014583920128643513,
+          0.010279136709868908,
+          0.019637754186987877,
+          0.0015691702719777822,
+          0.010360359214246273,
+          0.00032376122544519603,
+          0.004873339552432299,
+          0.02456524223089218,
+          0.006777551956474781,
+          -0.03978089243173599,
+          0.0005428923177532852,
+          0.001857960713095963,
+          0.009182635694742203,
+          -0.021623188629746437,
+          0.014908809214830399,
+          -0.00537872314453125,
+          0.028752705082297325,
+          -0.007007681764662266,
+          0.02084706351161003,
+          -0.006136798299849033,
+          -0.004661259241402149,
+          -0.003783606691285968,
+          0.006524860393255949,
+          0.030846435576677322,
+          -0.00328047969378531,
+          0.024854032322764397,
+          0.023103240877389908,
+          0.0074724541045725346,
+          0.007851491682231426,
+          -0.017923060804605484,
+          0.0030774236656725407,
+          0.022904695942997932,
+          -0.0017146936152130365,
+          0.007887590676546097,
+          -0.004877852275967598,
+          0.006303755100816488,
+          -0.016677651554346085,
+          0.01627153903245926,
+          0.01452074758708477,
+          -0.0058209337294101715,
+          0.02781413495540619,
+          0.015567612834274769,
+          0.009737654589116573,
+          -0.002993945265188813,
+          0.004408567678183317,
+          0.014538796618580818,
+          0.00432283291593194,
+          0.004140082746744156,
+          -0.015657860785722733,
+          0.013573153875768185,
+          0.010802569799125195,
+          0.013130943290889263,
+          -0.03687493875622749,
+          -0.016036897897720337,
+          -0.02326568402349949,
+          -0.0019121089717373252,
+          0.0194572601467371,
+          0.00011612254456849769,
+          0.001237512449733913,
+          0.025612108409404755,
+          -0.0003043017059098929,
+          -0.008162843994796276,
+          -0.007075367495417595,
+          -0.020973410457372665,
+          0.0017135655507445335,
+          -0.0001287430350203067,
+          -0.04270489513874054,
+          0.007296472322195768,
+          0.007025731261819601,
+          0.001146137248724699,
+          -0.007910151965916157,
+          -0.02572040446102619,
+          -0.01134405191987753,
+          0.08793670684099197,
+          -0.012345793657004833,
+          -0.02283249981701374,
+          -0.008591517806053162,
+          -0.004950049798935652,
+          -0.008659202605485916,
+          -0.003582807257771492,
+          0.021370496600866318,
+          0.01860893703997135,
+          0.0015624017687514424,
+          0.015179550275206566,
+          -0.008108695968985558,
+          0.02936638332903385,
+          0.01208407711237669,
+          0.019998742267489433,
+          -0.0028044264763593674,
+          0.007856003940105438,
+          -0.0044401539489626884,
+          0.013546079397201538,
+          -0.022128572687506676,
+          -0.0011641867458820343,
+          -0.0031383405439555645,
+          0.00407465361058712,
+          -0.02202027477324009,
+          0.007883078418672085,
+          -0.011515521444380283,
+          -0.013302412815392017,
+          -0.0032015135511755943,
+          -0.030467398464679718,
+          -0.013320461846888065,
+          -0.002380265388637781,
+          0.039095014333724976,
+          0.009620334021747112,
+          0.01804940588772297,
+          0.0012442809529602528,
+          0.02267005480825901,
+          0.0072197625413537025,
+          0.0017428958090022206,
+          0.014096586033701897,
+          0.009137512184679508,
+          -0.00923227146267891,
+          0.04624257981777191,
+          0.015405167825520039,
+          -0.0012262315722182393,
+          -0.03277772292494774,
+          0.003679822664707899,
+          0.016424959525465965,
+          0.009295444004237652,
+          -0.0028021703474223614,
+          -0.0017451519379392266,
+          0.00537872314453125,
+          -0.0046793087385594845,
+          0.017399627715349197,
+          0.015008081682026386,
+          -0.02227296680212021,
+          0.017562072724103928,
+          -0.0018331428291276097,
+          0.02259785681962967,
+          0.0006396258249878883,
+          -0.00021673388255294412,
+          0.02837366610765457,
+          -0.0018444237066432834,
+          0.024150105193257332,
+          0.00885323341935873,
+          -0.002257303800433874,
+          0.0033684703521430492,
+          0.017300356179475784,
+          0.004404054954648018,
+          0.006655718665570021,
+          -0.004176181275397539,
+          -0.017968183383345604,
+          0.023067140951752663,
+          0.01650618202984333,
+          0.002729972591623664,
+          -0.01763427071273327,
+          -0.006786576937884092,
+          0.009182635694742203,
+          0.0013807795476168394,
+          0.006651206407696009,
+          -0.005459945183247328,
+          0.01026108767837286,
+          0.014908809214830399,
+          -0.006939996499568224,
+          0.03241673484444618,
+          -0.010992088355123997,
+          -0.02299494296312332,
+          -0.011885534040629864,
+          0.023121289908885956,
+          0.01943921111524105,
+          -0.004207768011838198,
+          -0.015134426765143871,
+          0.031315721571445465,
+          -0.011533570475876331,
+          -0.0036143935285508633,
+          -0.030900582671165466,
+          -0.036189060658216476,
+          0.04501521959900856,
+          -0.0004785912751685828,
+          -0.0446903295814991,
+          0.027453146874904633,
+          0.01692131906747818,
+          0.010694272816181183,
+          0.010243037715554237,
+          -0.022128572687506676,
+          0.015657860785722733,
+          -0.0049274880439043045,
+          0.0045349132269620895,
+          -0.008244066499173641,
+          0.006664743181318045,
+          -0.011010137386620045,
+          0.027669738978147507,
+          -0.01221042312681675,
+          -0.002286634175106883,
+          -0.0010615306673571467,
+          0.00028300905250944197,
+          0.005870569497346878,
+          -0.005062858574092388,
+          -0.014656118117272854,
+          0.011930657550692558,
+          0.01389804296195507,
+          -0.0018703697714954615,
+          0.0179862342774868,
+          0.020648520439863205,
+          -0.03581002354621887,
+          -0.0014416963094845414,
+          0.0012826359597966075,
+          -0.0005392259918153286,
+          0.008826159872114658,
+          0.012075052596628666,
+          0.023211536929011345,
+          -0.0138619439676404,
+          -0.0368388369679451,
+          0.008762986399233341,
+          0.023644722998142242,
+          -0.0022990431170910597,
+          -0.023446178063750267,
+          -0.0020858345087617636,
+          0.00829821452498436,
+          0.020540224388241768,
+          -0.0012905325274914503,
+          0.008059060201048851,
+          0.0077702696435153484,
+          -0.017210109159350395,
+          -0.009999371133744717,
+          -0.005766785237938166,
+          -0.015332970768213272,
+          -0.027778036892414093,
+          0.010775495320558548,
+          0.005978866014629602,
+          0.01911432109773159,
+          -0.022868597880005836,
+          -0.021370496600866318,
+          -0.021911978721618652,
+          0.0009171354467980564,
+          0.006926459725946188,
+          -0.020973410457372665,
+          0.0014687704388052225,
+          -0.010766470804810524,
+          0.025792602449655533,
+          -0.013636326417326927,
+          0.0014394401805475354,
+          0.0020486076828092337,
+          0.028319519013166428,
+          0.002041839063167572,
+          0.008401998318731785,
+          -0.020504126325249672,
+          -0.0035106095019727945,
+          0.07266691327095032,
+          -0.013943166472017765,
+          -0.0045258887112140656,
+          -0.01175016351044178,
+          0.017020590603351593,
+          -0.01009864266961813,
+          0.0036707979161292315,
+          0.04772263020277023,
+          -0.014403426088392735,
+          0.004473996814340353,
+          0.021911978721618652,
+          0.008415536023676395,
+          0.000976360053755343,
+          -0.00967448204755783,
+          -0.012932399287819862,
+          -0.0023464227560907602,
+          -0.017697442322969437,
+          0.02301299385726452,
+          -0.021081706508994102,
+          0.027074109762907028,
+          -0.015784205868840218,
+          -0.034131426364183426,
+          0.01712888665497303,
+          0.029240038245916367,
+          -0.015594687312841415,
+          -0.009909124113619328,
+          -0.000990461208857596,
+          0.010983063839375973,
+          0.035629529505968094,
+          0.014448549598455429,
+          0.01887967810034752,
+          0.005320062395185232,
+          -0.003212794428691268,
+          0.0018410393968224525,
+          0.02012508735060692,
+          0.015892501920461655,
+          -0.0013345279730856419,
+          0.0064707123674452305,
+          -0.003801656188443303,
+          0.02548576146364212,
+          -0.014845636673271656,
+          -0.02846391312777996,
+          -0.00235995976254344,
+          -0.026370182633399963,
+          -0.007576238363981247,
+          0.029438581317663193,
+          -0.009448864497244358,
+          0.012445065192878246,
+          0.025431614369153976,
+          -0.019475309178233147,
+          -0.027326801791787148,
+          -0.02117195352911949,
+          0.01729133166372776,
+          -0.005915693007409573,
+          0.005288476124405861,
+          0.0017293586861342192,
+          0.015892501920461655,
+          -0.016118120402097702,
+          -0.01514345221221447,
+          0.0020294301211833954,
+          -0.020829014480113983,
+          0.024150105193257332,
+          0.018365271389484406,
+          -0.004354419186711311,
+          0.008492245338857174,
+          0.019728001207113266,
+          -0.010549877770245075,
+          0.0012713550822809339,
+          -0.017905011773109436,
+          -0.004877852275967598,
+          -0.0004585677233990282,
+          0.009936198592185974,
+          -0.02458329126238823,
+          -0.007707096636295319,
+          -0.004516863729804754,
+          -0.00182637432590127,
+          0.01522467378526926,
+          -0.015621760860085487,
+          -0.0004402362974360585,
+          0.008550906553864479,
+          0.0022144364193081856,
+          -0.017065713182091713,
+          -0.0015274309553205967,
+          0.00039313864544965327,
+          -0.011154533363878727,
+          0.0017169497441500425,
+          -0.010279136709868908,
+          -0.023879364132881165,
+          -0.004006968345493078,
+          -0.01141624990850687,
+          -0.0045326570980250835,
+          0.005789346992969513,
+          -0.00901567842811346,
+          0.0007039268384687603,
+          0.020666569471359253,
+          0.016316663473844528,
+          0.009304468519985676,
+          -0.029149791225790977,
+          0.003840011078864336,
+          -0.0037001282908022404,
+          -0.0027412534691393375,
+          -0.01692131906747818,
+          0.0030210192780941725,
+          -0.023536425083875656,
+          0.005089932586997747,
+          -0.018275024369359016,
+          0.014746365137398243,
+          0.005802884232252836,
+          -0.0032579179387539625,
+          -0.0006774167995899916,
+          -0.0002590371877886355,
+          -0.0010288162156939507,
+          0.0007428458775393665,
+          0.01675887405872345,
+          0.050430040806531906,
+          0.0032082819379866123,
+          -0.034456316381692886,
+          -0.004128801636397839,
+          0.006813650950789452,
+          0.0019380550365895033,
+          0.0089931171387434,
+          0.012634584680199623,
+          0.04021407663822174,
+          -0.03732617199420929,
+          0.01564883440732956,
+          -0.021298298612236977,
+          0.005396772641688585,
+          -0.014177808538079262,
+          0.0049229757860302925,
+          -0.025088675320148468,
+          0.01009864266961813,
+          0.012237497605383396,
+          -0.0019910752307623625,
+          -0.005247864872217178,
+          -0.015576637350022793,
+          0.016975466161966324,
+          0.010937940329313278,
+          -0.019998742267489433,
+          -0.009218734689056873,
+          0.001989946933463216,
+          -0.03416752442717552,
+          -0.01618129201233387,
+          -0.005198229104280472,
+          -0.009710581041872501,
+          0.0042867339216172695,
+          -0.00875847414135933,
+          0.012327744625508785,
+          0.013852919451892376,
+          -0.008031985722482204,
+          -0.002881136490032077,
+          0.014683191664516926,
+          -0.003145109163597226,
+          0.0037204339168965816,
+          0.00328047969378531,
+          0.021803682669997215,
+          -0.03490755334496498,
+          0.019132370129227638,
+          -0.003659517038613558,
+          -0.0015939881559461355,
+          -0.008677251636981964,
+          -0.0006633157026953995,
+          -0.022796399891376495,
+          -0.007711608894169331,
+          0.004473996814340353,
+          -0.005234327632933855,
+          0.0025539910420775414,
+          -0.0190240740776062,
+          0.021893929690122604,
+          0.010567927733063698,
+          0.008388461545109749,
+          -0.021081706508994102,
+          -0.005302012898027897,
+          -0.014096586033701897,
+          -0.01786891184747219,
+          0.007549164351075888,
+          -0.013744622468948364,
+          0.0009086747886613011,
+          0.004415336064994335,
+          0.016406910493969917,
+          0.0015285591362044215,
+          0.003025531768798828,
+          0.009069826453924179,
+          -0.015757132321596146,
+          -0.03317480906844139,
+          0.008478708565235138,
+          -0.0101347416639328,
+          0.026695072650909424,
+          0.0256662555038929,
+          0.015576637350022793,
+          0.009493988007307053,
+          0.006430101115256548,
+          -0.00327822333201766,
+          -0.03535878658294678,
+          -0.023283734917640686,
+          0.011921633034944534,
+          -0.020468026399612427,
+          0.017598170787096024,
+          0.004169412888586521,
+          -0.003429387230426073,
+          0.012318720109760761,
+          0.01671374961733818,
+          0.028193172067403793,
+          0.025359416380524635,
+          -0.01657838001847267,
+          0.01687619462609291,
+          -0.019565556198358536,
+          -0.006366928108036518,
+          0.02862635813653469,
+          -0.0040362984873354435,
+          -0.011172582395374775,
+          0.007643923629075289,
+          -0.0095120370388031,
+          0.01918651908636093,
+          0.000988204963505268,
+          -0.010495729744434357,
+          -0.01038743369281292,
+          0.0036346991546452045,
+          -0.016397885978221893,
+          -0.01911432109773159,
+          0.01554956380277872,
+          0.027525344863533974,
+          0.01369949895888567,
+          -0.020305581390857697,
+          0.008663714863359928,
+          0.002093731192871928,
+          0.024962328374385834,
+          0.005667513702064753,
+          0.03718177601695061,
+          -0.0056765382178127766,
+          0.00643461337313056,
+          -0.016452033072710037,
+          -0.005392259918153286,
+          0.01766134425997734,
+          -0.003770069684833288,
+          -0.007467941846698523,
+          -0.007851491682231426,
+          0.0030165070202201605,
+          0.016289589926600456,
+          0.01224652212113142,
+          0.0024321575183421373,
+          -0.01729133166372776,
+          -0.0031044979114085436,
+          -0.010035470128059387,
+          -0.0036031126510351896,
+          -0.0010271240025758743,
+          -0.018681135028600693,
+          -0.0007941738585941494,
+          0.012165299616754055,
+          -0.04537620767951012,
+          -0.001970769604668021,
+          -0.015910550951957703,
+          -0.022056374698877335,
+          -0.024132056161761284,
+          0.023771068081259727,
+          0.009620334021747112,
+          0.013645350933074951,
+          0.007319034077227116,
+          0.01604592241346836,
+          -0.041008252650499344,
+          0.01960165612399578,
+          0.01017084065824747,
+          -0.005644951947033405,
+          0.0022730971686542034,
+          -0.02705606073141098,
+          0.006254119332879782,
+          0.008216992020606995,
+          -0.003864829195663333,
+          0.01551346480846405,
+          -0.026911664754152298,
+          -0.002569784177467227,
+          -0.011885534040629864,
+          0.011831386014819145,
+          0.027344850823283195,
+          -0.006145822815597057,
+          3.934206688427366e-05,
+          -0.015161501243710518,
+          0.005207253620028496,
+          0.002380265388637781,
+          -0.01837429590523243,
+          0.006637669168412685,
+          -0.014204883016645908,
+          -0.0036031126510351896,
+          0.00458229286596179,
+          -0.0012916605919599533,
+          -0.01683107204735279,
+          0.0019301583524793386,
+          -0.0011935170041397214,
+          -0.004171669017523527,
+          -0.005103469826281071,
+          -0.0022674566134810448,
+          0.012643609195947647,
+          -0.008925431407988071,
+          -0.008623103611171246,
+          -0.0035444521345198154,
+          -0.022363213822245598,
+          -0.0011991574428975582,
+          -0.00114275305531919,
+          -0.007070854771882296,
+          0.00494553754106164,
+          0.002865343354642391,
+          -0.004661259241402149,
+          0.01597372442483902,
+          -0.008203455246984959,
+          -0.00552763044834137,
+          0.008230529725551605,
+          -0.02259785681962967,
+          0.005685563199222088,
+          0.012390917167067528,
+          -0.04089995473623276,
+          0.0030480935238301754,
+          -0.004550706595182419,
+          0.028427815064787865,
+          -0.011425274424254894,
+          0.0042799655348062515,
+          0.021460743620991707,
+          -0.009011166170239449,
+          -0.026171639561653137,
+          0.00989107508212328,
+          0.053570639342069626,
+          0.004886876791715622,
+          -0.0008945736917667091,
+          -0.003650492522865534,
+          0.00023027094721328467,
+          -0.011975781060755253,
+          -0.0064797368831932545,
+          -0.014159759506583214,
+          0.016984490677714348,
+          0.005112494342029095,
+          -0.015784205868840218,
+          -0.020052891224622726,
+          -0.015558588318526745,
+          -0.013148992322385311,
+          0.023608623072504997,
+          -0.010585976764559746,
+          -0.01579323038458824,
+          0.004232585895806551,
+          0.02259785681962967,
+          0.010306211188435555,
+          -0.00575324846431613,
+          -0.02109975554049015,
+          -0.019096272066235542,
+          0.018067454919219017,
+          0.006303755100816488,
+          0.014331229031085968,
+          0.0024908180348575115,
+          -0.013708524405956268,
+          0.014177808538079262,
+          0.003025531768798828,
+          0.021081706508994102,
+          -0.0005378159112296999,
+          -0.007052805740386248,
+          0.005013222806155682,
+          -0.02456524223089218,
+          0.020666569471359253,
+          -0.008695301599800587,
+          0.013148992322385311,
+          -0.020648520439863205,
+          -0.008623103611171246,
+          0.004480765201151371,
+          -0.0030864484142512083,
+          -0.012634584680199623,
+          0.021496843546628952,
+          0.0024028271436691284,
+          -0.008271140046417713,
+          -0.0032895042095333338,
+          0.0008240682072937489,
+          -0.027778036892414093,
+          -0.001954976236447692,
+          0.006394002120941877,
+          -0.02465548925101757,
+          0.015405167825520039,
+          0.0015533770201727748,
+          -0.014240982010960579,
+          0.004015992861241102,
+          -0.00792820192873478,
+          -0.005274938885122538,
+          0.0003812937065958977,
+          0.013654375448822975,
+          -0.029907865449786186,
+          0.015982748940587044,
+          -0.0031766954343765974,
+          -0.0071295155212283134,
+          0.006935484241694212,
+          0.0018365271389484406,
+          0.00864566583186388,
+          -0.008027473464608192,
+          1.942426388268359e-05,
+          0.017778664827346802,
+          0.0063263168558478355,
+          0.008568955585360527,
+          0.00513054383918643,
+          -0.0031766954343765974,
+          -0.003219562815502286,
+          0.0006644437671639025,
+          0.002129829954355955,
+          -0.020810965448617935,
+          -0.005911180749535561,
+          -0.003986662719398737,
+          0.0014089817414060235,
+          0.01026108767837286,
+          -0.003939283080399036,
+          0.0013593458570539951,
+          -0.02573845349252224,
+          -0.004268684424459934,
+          0.005911180749535561,
+          0.0030187631491571665,
+          0.007580750621855259,
+          0.03028690442442894,
+          0.015396143309772015,
+          0.019258717074990273,
+          0.0020215334370732307,
+          -0.0015240467619150877,
+          -0.011948706582188606,
+          -0.0011399327777326107,
+          0.00792820192873478,
+          -0.00844712182879448,
+          0.009827901609241962,
+          -0.022814448922872543,
+          0.01072134729474783,
+          0.005261402111500502,
+          0.003183464054018259,
+          -0.000386370113119483,
+          -0.014096586033701897,
+          0.012129200622439384,
+          0.021821731701493263,
+          -0.021496843546628952,
+          0.001207054010592401,
+          -0.013780721463263035,
+          -0.00023619340208824724,
+          -0.011569669470191002,
+          -0.03656809777021408,
+          -0.008144794963300228,
+          -0.030665941536426544,
+          -0.00015750927559565753,
+          -0.014033413492143154,
+          -0.0030119947623461485,
+          -0.01079354528337717,
+          -0.010694272816181183,
+          0.001760945189744234,
+          -0.009791802614927292,
+          0.009692531079053879,
+          0.010414507240056992,
+          0.004737969022244215,
+          -0.0007857132004573941,
+          0.011235755868256092,
+          -0.009367641992866993,
+          -0.014872711151838303,
+          -0.010414507240056992,
+          -0.002763815224170685,
+          0.024547193199396133,
+          -0.010766470804810524,
+          -0.014845636673271656,
+          -0.00612326106056571,
+          -0.006430101115256548,
+          -0.011605768464505672,
+          -0.01837429590523243,
+          0.014195858500897884,
+          0.02754339389503002,
+          -0.013690474443137646,
+          0.0023667283821851015,
+          -0.010847693309187889,
+          0.013266313821077347,
+          0.023969611153006554,
+          -0.001973025733605027,
+          -0.0289692971855402,
+          -0.018717234954237938,
+          0.0012521775206550956,
+          -0.011425274424254894,
+          -0.0072287870571017265,
+          -0.01432220358401537,
+          -0.004706382751464844,
+          -0.015206624753773212,
+          0.017769640311598778,
+          -0.00926837045699358,
+          -0.02028753235936165,
+          -0.008456147275865078,
+          -0.0030165070202201605,
+          -0.00600142776966095,
+          -0.008690789341926575,
+          -0.022525658831000328,
+          0.011434298940002918,
+          0.008253091014921665,
+          0.009593259543180466,
+          -0.01567590981721878,
+          0.019673852249979973,
+          0.013383634388446808,
+          -0.024799883365631104,
+          -0.003431643359363079,
+          -0.01464709360152483,
+          -0.0020666569471359253,
+          -0.013843894936144352,
+          -0.011271853931248188,
+          0.0006982863997109234,
+          0.0014394401805475354,
+          0.010802569799125195,
+          0.0024231327697634697,
+          0.022633954882621765,
+          -4.344900844444055e-06,
+          -0.013627301901578903,
+          0.00643461337313056,
+          0.01522467378526926,
+          -0.014538796618580818,
+          0.02806682698428631,
+          0.012914350256323814,
+          -0.017670368775725365,
+          0.015937626361846924,
+          0.013049720786511898,
+          -0.015964699909090996,
+          0.005762272980064154,
+          0.004977123811841011,
+          -0.00854188110679388,
+          -0.01361827738583088,
+          0.018320146948099136,
+          0.01175016351044178,
+          0.021532941609621048,
+          0.001636855537071824,
+          0.012508238665759563,
+          -0.00686779897660017,
+          0.007409281097352505,
+          -0.022200770676136017,
+          0.01985434629023075,
+          -0.00963838305324316,
+          -0.02010703831911087,
+          0.014926859177649021,
+          -0.0026103954296559095,
+          0.012896301224827766,
+          0.011262829415500164,
+          -0.004832728765904903,
+          -0.05097152292728424,
+          -0.01038743369281292,
+          0.011470397934317589,
+          0.003955075982958078,
+          -0.014990031719207764,
+          0.019565556198358536,
+          -0.0036911035422235727,
+          0.030268853530287743,
+          -0.005049321334809065,
+          0.02631603367626667,
+          -0.005256889387965202,
+          0.018735283985733986,
+          -0.015125402249395847,
+          -0.016993515193462372,
+          0.0071295155212283134,
+          0.00026777986204251647,
+          -0.004674796015024185,
+          0.008307239040732384,
+          -0.030918633565306664,
+          -0.010531828738749027,
+          -0.007147565018385649,
+          0.01886162906885147,
+          -0.006177409086376429,
+          0.026604825630784035,
+          0.0019233898492529988,
+          0.009466913528740406,
+          0.017580121755599976,
+          -0.01352803036570549,
+          0.019565556198358536,
+          -0.05479799583554268,
+          -0.0020316862501204014,
+          -0.0044333855621516705,
+          0.027669738978147507,
+          -0.004205511882901192,
+          -0.0012668427079916,
+          0.004088190384209156,
+          3.040549358956923e-07,
+          -0.005017735064029694,
+          0.007661973126232624,
+          -3.456531703704968e-05,
+          0.01361827738583088,
+          -0.027272652834653854,
+          -0.02739899791777134,
+          0.00661961967125535,
+          0.004828216042369604,
+          0.016975466161966324,
+          -0.010306211188435555,
+          -0.001425903057679534,
+          0.017850862815976143,
+          -0.01439440157264471,
+          0.00471991952508688,
+          -0.010495729744434357,
+          -0.016316663473844528,
+          -0.018455518409609795,
+          0.0018771382747218013,
+          0.019493358209729195,
+          -0.006412051618099213,
+          0.005536655429750681,
+          0.001989946933463216,
+          0.013627301901578903,
+          0.0012352562043815851,
+          0.011425274424254894,
+          -0.011777237989008427,
+          0.027705838903784752,
+          -0.009710581041872501,
+          0.015982748940587044,
+          0.004088190384209156,
+          -0.0017146936152130365,
+          0.006168384570628405,
+          -0.007251348812133074,
+          -0.029655175283551216,
+          -0.004467227961868048,
+          -0.010243037715554237,
+          -0.0018342708935961127,
+          -0.02433059923350811,
+          -0.024799883365631104,
+          0.0040430668741464615,
+          0.0030390687752515078,
+          0.00963838305324316,
+          0.016100069507956505,
+          0.022236868739128113,
+          0.007589775137603283,
+          0.008731400594115257,
+          -0.0021726973354816437,
+          0.004106239881366491,
+          -0.020901212468743324,
+          0.008302726782858372,
+          -0.016614478081464767,
+          -0.010576952248811722,
+          -0.0019211337203159928,
+          -0.02597309648990631,
+          0.002477281028404832,
+          -0.008131257258355618,
+          0.007319034077227116,
+          -0.00026369054103270173,
+          0.0021140368189662695,
+          0.016749849542975426,
+          -0.00894348043948412,
+          0.010613051243126392,
+          -0.005089932586997747,
+          0.00327822333201766,
+          -0.004904926288872957,
+          -0.014583920128643513,
+          -0.0011207553325220942,
+          -0.014421476051211357,
+          0.012102127075195312,
+          -0.023139338940382004,
+          -0.0045213764533400536,
+          -0.05512288585305214,
+          -0.006096187047660351,
+          -0.021911978721618652,
+          0.005978866014629602,
+          0.013519004918634892,
+          -0.012273596599698067,
+          -0.01590152643620968,
+          -0.02043192833662033,
+          0.022561758756637573,
+          -0.012038953602313995,
+          -0.0012228472623974085,
+          -0.01059500128030777,
+          0.035467084497213364,
+          -0.030431298539042473,
+          -0.001330015598796308,
+          -0.0032263314351439476,
+          0.008916406892240047,
+          -0.006615107413381338,
+          -0.024294501170516014,
+          -0.005324574653059244,
+          -0.012806054204702377,
+          0.004453691188246012,
+          -0.007661973126232624,
+          -0.017814764752984047,
+          0.005324574653059244,
+          -0.022778350859880447,
+          0.0031947449315339327,
+          0.012562386691570282,
+          -0.006800113711506128,
+          -0.00527945114299655,
+          0.004900414030998945,
+          0.007711608894169331,
+          0.009773753583431244,
+          0.0076303863897919655,
+          -0.0014890760648995638,
+          0.006344366353005171,
+          -0.009385691024363041,
+          -0.009277394972741604,
+          0.012824103236198425,
+          0.02102755755186081,
+          -0.002097115386277437,
+          0.006046551279723644,
+          -0.019385062158107758,
+          -0.011957732029259205,
+          -0.02747119590640068,
+          -0.013546079397201538,
+          -0.02100950852036476,
+          0.0009092388208955526,
+          0.01125380489975214,
+          0.00643461337313056,
+          0.01432220358401537,
+          -0.00019713335495907813,
+          -0.0030526057817041874,
+          0.0016278307884931564,
+          -0.046964555978775024,
+          -0.010215964168310165,
+          0.006380465347319841,
+          -0.005306525621563196,
+          0.009038240648806095,
+          0.020829014480113983,
+          -0.007684534881263971,
+          -0.021984176710247993,
+          0.0028540624771267176,
+          -0.0006937740836292505,
+          -0.0010694273514673114,
+          0.014078537002205849,
+          0.014069512486457825,
+          -0.006678280420601368,
+          0.013654375448822975,
+          0.004363444168120623,
+          0.022399313747882843,
+          -0.029330285266041756,
+          0.008826159872114658,
+          0.011235755868256092,
+          -0.024781834334135056,
+          0.018347222357988358,
+          0.004832728765904903,
+          -0.018094530329108238,
+          0.004155875649303198,
+          0.015712007880210876,
+          0.019709952175617218,
+          -0.002291146432980895,
+          0.012354818172752857,
+          -0.005103469826281071,
+          0.003623418277129531,
+          0.019312864169478416,
+          0.004977123811841011,
+          -0.021081706508994102,
+          -0.017589146271348,
+          -0.011641867458820343,
+          0.018500640988349915,
+          -0.005969841033220291,
+          0.012390917167067528,
+          0.018365271389484406,
+          -0.0007405896903946996,
+          -0.009909124113619328,
+          0.022706152871251106,
+          -0.0006666999543085694,
+          0.006831700447946787,
+          0.0014180064899846911,
+          0.0004551834717858583,
+          -0.019006025046110153,
+          0.000448414939455688,
+          0.008153819479048252,
+          0.012815078720450401,
+          0.015919575467705727,
+          0.01398828998208046,
+          -0.01597372442483902,
+          -0.003149621421471238,
+          -0.001759817125275731,
+          -0.012463115155696869,
+          -0.023373981937766075,
+          0.014953932724893093,
+          -0.006588033400475979,
+          0.018085505813360214,
+          -0.005906668026000261,
+          -0.001572554581798613,
+          -0.01303167175501585,
+          0.018265999853610992,
+          0.001610909472219646,
+          0.01109135989099741,
+          -0.00012359612446743995,
+          -0.002317092614248395,
+          0.00717915128916502,
+          0.0005073575302958488,
+          -0.017228158190846443,
+          -0.013248263858258724,
+          0.013455832377076149,
+          0.005780322477221489,
+          -0.0029127229936420918,
+          0.02216467075049877,
+          -0.0010936812032014132,
+          0.024366699159145355,
+          0.019836297258734703,
+          0.01277897972613573,
+          -0.006520348135381937,
+          0.008456147275865078,
+          0.026081392541527748,
+          -0.021659286692738533,
+          0.013943166472017765,
+          0.004891389049589634,
+          -0.02061242237687111,
+          -0.03183915093541145,
+          0.0007620233809575438,
+          -0.0006949021480977535,
+          0.010495729744434357,
+          -0.02167733758687973,
+          -0.00934959203004837,
+          -0.015964699909090996,
+          -0.009909124113619328,
+          0.008388461545109749,
+          0.00376330129802227,
+          0.008587004616856575,
+          0.008546394295990467,
+          -0.01327533833682537,
+          0.0032105380669236183,
+          0.0165513064712286,
+          0.0033210907131433487,
+          -0.012255546636879444,
+          0.014006339013576508,
+          0.03220013901591301,
+          0.005450920667499304,
+          -0.031063027679920197,
+          0.016948392614722252,
+          0.0012792516499757767,
+          0.017697442322969437,
+          -0.004329601302742958,
+          -0.007968813180923462,
+          -0.008442609570920467,
+          0.01472831517457962,
+          -0.03218208998441696,
+          -0.0022753532975912094,
+          0.018933827057480812,
+          0.003181207925081253,
+          0.008559931069612503,
+          0.009430814534425735,
+          0.001052506035193801,
+          -0.017905011773109436,
+          0.004864315036684275,
+          -0.014403426088392735,
+          -0.0007287447806447744,
+          0.011795287020504475,
+          -0.01360925193876028,
+          -0.009191660210490227,
+          0.014331229031085968,
+          -0.00767550989985466,
+          -0.02821122296154499,
+          0.03292211890220642,
+          -0.013347536325454712,
+          -0.008013936690986156,
+          0.0372539758682251,
+          0.004769555758684874,
+          -0.005789346992969513,
+          0.011073310859501362,
+          0.015712007880210876,
+          -0.007314521819353104,
+          0.004113008268177509,
+          0.018320146948099136,
+          -0.019980693235993385,
+          -0.011280879378318787,
+          0.018482591956853867,
+          0.03019665740430355,
+          -0.0013221190311014652,
+          0.0007479222840629518,
+          -0.017589146271348,
+          -0.015423217788338661,
+          -0.005455432925373316,
+          0.008361387066543102,
+          0.01804940588772297,
+          -0.012544337660074234,
+          -0.01733645424246788,
+          0.02373497001826763,
+          0.0045236325822770596,
+          0.005062858574092388,
+          0.0071340277791023254,
+          0.022525658831000328,
+          -0.013049720786511898,
+          -0.025594057515263557,
+          0.014656118117272854,
+          -0.010360359214246273,
+          -0.01782378926873207,
+          -0.017760615795850754,
+          0.014087561517953873,
+          -0.012634584680199623,
+          -0.015531513839960098,
+          0.021641237661242485,
+          0.0013367841020226479,
+          -0.0032060258090496063,
+          0.02225491777062416,
+          0.010748421773314476,
+          -0.0167408250272274,
+          -0.03783155605196953,
+          -0.021370496600866318,
+          -0.00038298583240248263,
+          0.009584235027432442,
+          0.006538397632539272,
+          0.009029215201735497,
+          -0.025449663400650024,
+          -0.0029826643876731396,
+          -0.025052575394511223,
+          -0.0005820933729410172,
+          -0.006849749479442835,
+          -0.010694272816181183,
+          -0.004122033249586821,
+          -0.00376330129802227,
+          0.021695386618375778,
+          0.01647910848259926,
+          -0.0009487218922004104,
+          0.006344366353005171,
+          -0.002160288393497467,
+          -0.008623103611171246,
+          -0.011578693985939026,
+          0.01646105945110321,
+          0.016903268173336983,
+          -0.02017923630774021,
+          -0.0028224759735167027,
+          0.011190632358193398,
+          0.0026690559461712837,
+          -0.013509980402886868,
+          -0.019096272066235542,
+          -0.009493988007307053,
+          -0.009173611178994179,
+          -0.024547193199396133,
+          -0.002287762239575386,
+          -0.006023989524692297,
+          -0.025449663400650024,
+          0.012842152267694473,
+          0.04230780899524689,
+          -0.01606397144496441,
+          -0.0032285875640809536,
+          0.006664743181318045,
+          -0.0028292443603277206,
+          0.0072107380256056786,
+          0.005311037879437208,
+          -0.008975067175924778,
+          0.02052217535674572,
+          -0.004376980941742659,
+          -0.004986148327589035,
+          -0.002004612237215042,
+          0.005148593336343765,
+          -0.011389175429940224,
+          -0.02458329126238823,
+          -0.025016477331519127,
+          0.006935484241694212,
+          -0.0006949021480977535,
+          0.0011822361266240478,
+          -0.009087876416742802,
+          0.0012871483340859413,
+          -0.01749889925122261,
+          0.009137512184679508,
+          0.012941424734890461,
+          -0.0024592315312474966,
+          -0.0045258887112140656,
+          -0.013519004918634892,
+          -0.011136483401060104,
+          0.005771297495812178,
+          0.017047664150595665,
+          0.006127773318439722,
+          -0.01800428330898285,
+          0.0007157717482186854,
+          0.012661658227443695,
+          0.02680336870253086,
+          -0.023482277989387512,
+          0.013717548921704292,
+          0.012733856216073036,
+          0.001949335914105177,
+          0.0020666569471359253,
+          0.019728001207113266,
+          -0.007007681764662266,
+          -0.004864315036684275,
+          0.00606008805334568,
+          -0.015242723748087883,
+          0.0029578465037047863,
+          0.026099441573023796,
+          -0.008117720484733582,
+          0.014286105521023273,
+          -0.05353453755378723,
+          -0.010405482724308968,
+          0.0009977937443181872,
+          -0.016352761536836624,
+          -0.0055817789398133755,
+          0.024095958098769188,
+          -0.00013755621330346912,
+          -0.003215050557628274,
+          -0.0055772666819393635,
+          0.012237497605383396,
+          -0.02176758460700512,
+          -0.04353516921401024,
+          0.0018748820293694735,
+          -0.01590152643620968,
+          -0.016361786052584648,
+          0.00014249159721657634,
+          -0.018933827057480812,
+          0.007454404607415199,
+          -0.00963838305324316,
+          -0.008496757596731186,
+          -0.014665142633020878,
+          0.027290701866149902,
+          -0.000774996355175972,
+          -0.0012668427079916,
+          -0.01812160387635231,
+          -0.01184041053056717,
+          -0.015784205868840218,
+          -0.011849435046315193,
+          -0.008749449625611305,
+          -0.0028247321024537086,
+          -0.008226016536355019,
+          -0.0034451803658157587,
+          0.0045258887112140656,
+          0.017444750294089317,
+          -0.020413879305124283,
+          -0.02035973034799099,
+          -0.0028946734964847565,
+          -0.01009864266961813,
+          0.004832728765904903,
+          0.0138619439676404,
+          0.015125402249395847,
+          -0.0054328711703419685,
+          0.007359645329415798,
+          -0.007846979424357414,
+          0.0016571611631661654,
+          -0.010243037715554237,
+          0.03862572833895683,
+          0.0027322289533913136,
+          0.0008325288654305041,
+          -0.0019042124040424824,
+          -8.156604053510819e-06,
+          0.005446408409625292,
+          -0.004977123811841011,
+          -0.017507923766970634,
+          0.010549877770245075,
+          -0.02326568402349949,
+          0.021406596526503563,
+          -0.010279136709868908,
+          0.01017084065824747,
+          -0.005762272980064154,
+          -0.026586774736642838,
+          -0.002233613980934024,
+          0.024781834334135056,
+          0.0010919891064986587,
+          0.009227759204804897,
+          -0.01782378926873207,
+          -0.008636641316115856,
+          -0.0030751675367355347,
+          -0.022291017696261406,
+          -0.009584235027432442,
+          -0.00901567842811346,
+          0.002319348743185401,
+          -0.010576952248811722,
+          0.014935883693397045,
+          0.017580121755599976,
+          0.005053833592683077,
+          -0.006831700447946787,
+          0.009710581041872501,
+          -0.02979956939816475,
+          0.008781036362051964,
+          -0.003436155617237091,
+          -0.005473482422530651,
+          -0.019583605229854584,
+          0.019421162083745003,
+          0.0001576502836542204,
+          -0.0027389973402023315,
+          -0.002817963482812047,
+          0.02100950852036476,
+          0.008361387066543102,
+          0.009683506563305855,
+          -0.005468970164656639,
+          -0.004638697486370802,
+          -0.01221042312681675,
+          0.003219562815502286,
+          -0.00019558223721105605,
+          -0.0002899185928981751,
+          -0.0016176780918613076,
+          0.005320062395185232,
+          -0.010486705228686333,
+          -0.025142822414636612,
+          0.017192060127854347,
+          -0.00894348043948412,
+          -0.006664743181318045,
+          0.017155960202217102,
+          -0.01514345221221447,
+          0.0011213193647563457,
+          0.004088190384209156,
+          0.0004159824165981263,
+          0.004162644501775503,
+          0.004760530777275562,
+          0.008884820155799389,
+          -0.0063082673586905,
+          -0.020558273419737816,
+          0.011235755868256092,
+          -0.013482906855642796,
+          -0.005139568354934454,
+          0.021911978721618652,
+          -0.006529372651129961,
+          -0.002199771348387003,
+          -0.010486705228686333,
+          -0.02183978073298931,
+          0.01481856219470501,
+          -0.02689361572265625,
+          0.0016481364145874977,
+          -0.0029804082587361336,
+          0.01588347740471363,
+          -0.011939682066440582,
+          0.02368082106113434,
+          -0.006655718665570021,
+          -0.006263143848627806,
+          0.0021208052057772875,
+          -0.0260633435100317,
+          0.008198942989110947,
+          -0.035449035465717316,
+          0.004338625818490982,
+          -0.01125380489975214,
+          0.029258087277412415,
+          0.018645036965608597,
+          -0.006425588857382536,
+          0.011398199945688248,
+          -0.02904149517416954,
+          0.004214536398649216,
+          0.01411463599652052,
+          0.0038783662021160126,
+          -0.0040295301005244255,
+          0.007449892349541187,
+          -0.010865742340683937,
+          -0.023536425083875656,
+          -0.02954687736928463,
+          0.001267970772460103,
+          -0.03481730446219444,
+          0.0013785234186798334,
+          0.0012916605919599533,
+          -0.016912294551730156,
+          0.010324260219931602,
+          -0.006759502459317446,
+          -0.020666569471359253,
+          0.017228158190846443,
+          0.0031157787889242172,
+          -0.012878251262009144,
+          -0.0042777094058692455,
+          0.006750477943569422,
+          0.02160513959825039,
+          0.014638068154454231,
+          -0.009737654589116573,
+          0.0008404254913330078,
+          0.010585976764559746,
+          -0.006389489863067865,
+          -0.010197914205491543,
+          -0.022796399891376495,
+          -0.00883067212998867,
+          0.004004712216556072,
+          0.006443637888878584,
+          -0.006664743181318045,
+          0.02001679129898548,
+          0.011226730421185493,
+          0.0012205911334604025,
+          0.01352803036570549,
+          -0.01389804296195507,
+          0.018338195979595184,
+          -0.0031090101692825556,
+          -0.014629043638706207,
+          0.0019290302880108356,
+          -0.03593636676669121,
+          0.005965328775346279,
+          -0.006213508080691099,
+          0.025756502524018288,
+          -0.01439440157264471,
+          -0.0017530486220493913,
+          -0.018455518409609795,
+          -0.0025336854159832,
+          0.015684934332966805,
+          -0.012598485685884953,
+          -0.011280879378318787,
+          0.013537054881453514,
+          -0.013176066800951958,
+          -0.0020305581856518984,
+          0.016930343583226204,
+          -0.010089618153870106,
+          0.01233676914125681,
+          -0.007188176270574331,
+          -0.00027215119916945696,
+          -0.00182637432590127,
+          -0.0012566898949444294,
+          -0.004566499963402748,
+          -0.0013412964763119817,
+          0.0028337568510323763,
+          -0.020143138244748116,
+          0.029582977294921875,
+          -0.013509980402886868,
+          -0.010766470804810524,
+          0.004250635392963886,
+          -0.003032300155609846,
+          0.021875880658626556,
+          -0.011650891974568367,
+          0.00391220860183239,
+          0.000977488118223846,
+          -0.010342310182750225,
+          -0.008167356252670288,
+          -0.004559731110930443,
+          0.013013621792197227,
+          0.026279935613274574,
+          -0.019980693235993385,
+          0.009394715540111065,
+          0.00997229665517807,
+          -0.02350032702088356,
+          -0.01869918406009674,
+          -0.004543938208371401,
+          -0.013446807861328125,
+          0.01788696087896824,
+          -0.00039454875513911247,
+          -0.005658488720655441,
+          0.0036843351554125547,
+          0.005035784561187029,
+          -0.014340253546833992,
+          -0.000682493147905916,
+          -0.003898671828210354,
+          0.004345394670963287,
+          -0.006281193345785141,
+          0.011768212541937828,
+          0.0006520348251797259,
+          0.02059437334537506,
+          -0.00926837045699358,
+          -0.014132685028016567,
+          0.0064842491410672665,
+          -0.00997229665517807,
+          0.009882049635052681,
+          -0.021208051592111588,
+          0.0010152790928259492,
+          0.0074724541045725346,
+          -0.005098957102745771,
+          0.004841753281652927,
+          0.009484962560236454,
+          -0.02739899791777134,
+          0.005969841033220291,
+          0.005482506938278675,
+          -0.0015691702719777822,
+          0.016497157514095306,
+          0.009503012523055077,
+          -0.0007095672772265971,
+          0.0006988504319451749,
+          -0.014719290658831596,
+          0.019385062158107758,
+          0.006921947468072176,
+          0.019096272066235542,
+          0.006773039698600769,
+          0.015522489324212074,
+          -0.00013586408749688417,
+          -0.005622390192002058,
+          0.006782064214348793,
+          -0.006064600311219692,
+          -0.004595830105245113,
+          0.03557537868618965,
+          -0.0006255247280932963,
+          0.012300670146942139,
+          0.006696329917758703,
+          -0.00030599383171647787,
+          0.0056765382178127766,
+          0.0120028555393219,
+          0.0012454090174287558,
+          -0.017697442322969437,
+          0.011235755868256092,
+          -0.015991773456335068,
+          -0.019637754186987877,
+          0.0019380550365895033,
+          0.009394715540111065,
+          -0.00780185591429472,
+          -0.0026081393007189035,
+          0.026099441573023796,
+          0.008550906553864479,
+          0.009358617477118969,
+          0.003530915128067136,
+          -0.007901127450168133,
+          -0.002162544522434473,
+          0.005956304259598255,
+          0.000220682195504196,
+          0.001759817125275731,
+          -0.004805654287338257,
+          -0.02581065148115158,
+          0.003303041448816657,
+          -0.005446408409625292,
+          -0.00340456934645772,
+          -0.0015026130713522434,
+          0.018383320420980453,
+          -0.01352803036570549,
+          0.01544126681983471,
+          0.006033014040440321,
+          -0.004113008268177509,
+          -0.014304154552519321,
+          0.0062315575778484344,
+          0.004081421997398138,
+          0.020883163437247276,
+          -0.001704540802165866,
+          -0.019746050238609314,
+          -0.0227603018283844,
+          0.003797143930569291,
+          -5.8026020269608125e-05,
+          -0.0034880477469414473,
+          0.0022945308592170477,
+          -0.002249407349154353,
+          0.008216992020606995,
+          -0.0034857916180044413,
+          -0.006673768162727356,
+          0.002287762239575386,
+          -0.006154847331345081,
+          0.02017923630774021,
+          -0.002326117129996419,
+          -0.0006571111734956503,
+          0.0023667283821851015,
+          -0.004013736732304096,
+          0.001205925946123898,
+          -0.002102755941450596,
+          0.016794972121715546,
+          -0.017192060127854347,
+          -0.00393702695146203,
+          -0.009078850969672203,
+          -0.017480850219726562,
+          0.0014721547486260533,
+          -0.009259345009922981,
+          0.006664743181318045,
+          0.013943166472017765,
+          0.014764414168894291,
+          -0.004990661051124334,
+          -0.0017846350092440844,
+          0.01918651908636093,
+          0.011677965521812439,
+          0.0010474296286702156,
+          0.011353076435625553,
+          0.0009002141305245459,
+          -0.010992088355123997,
+          -0.0006266528507694602,
+          -0.010910865850746632,
+          -0.007607824634760618,
+          0.023554475978016853,
+          0.02274225279688835,
+          0.012318720109760761,
+          0.0025742966681718826,
+          -0.0023486788850277662,
+          -0.0029285161290317774,
+          0.006294730585068464,
+          0.000976360053755343,
+          -0.011885534040629864,
+          0.0021456230897456408,
+          0.0006243966636247933,
+          -0.022615905851125717,
+          0.013925116509199142,
+          -0.02929418720304966,
+          0.0240057110786438,
+          -0.0018365271389484406,
+          0.009187147952616215,
+          0.019547507166862488,
+          -0.0011280878679826856,
+          0.013464856892824173,
+          0.01096501387655735,
+          0.0019414392299950123,
+          -0.02465548925101757,
+          -0.00811320822685957,
+          -0.010378408245742321,
+          -0.005256889387965202,
+          0.006989632733166218,
+          0.009114949963986874,
+          0.000154830064275302,
+          0.0030841922853142023,
+          -0.0021241893991827965,
+          0.001891803345642984,
+          -0.01969190314412117,
+          0.011903583072125912,
+          0.0022178208455443382,
+          -0.010215964168310165,
+          -0.023933513090014458,
+          -0.005703612230718136,
+          0.009864000603556633,
+          0.0022099241614341736,
+          -0.003519634250551462,
+          0.008365900255739689,
+          -0.0063308291137218475,
+          -0.0008658074657432735,
+          0.00024973045219667256,
+          -0.003395544597879052,
+          0.0016526487888768315,
+          -0.005866057239472866,
+          0.005117006599903107,
+          0.005766785237938166,
+          -0.0008415535558015108,
+          0.008230529725551605,
+          -0.01046865526586771,
+          0.013952190987765789,
+          -0.011407224461436272,
+          0.005613365210592747,
+          -0.008537368848919868,
+          0.006831700447946787,
+          0.008893844671547413,
+          0.005861544515937567,
+          0.013148992322385311,
+          -0.0003739611420314759,
+          -0.0013164785923436284,
+          -0.016217391937971115,
+          -0.03485340252518654,
+          0.006366928108036518,
+          -0.0009064186015166342,
+          0.006588033400475979,
+          -0.02160513959825039,
+          0.0050809080712497234,
+          0.012580436654388905,
+          -0.017255231738090515,
+          0.0010169711895287037,
+          0.005784834735095501,
+          0.015910550951957703,
+          -0.011154533363878727,
+          0.000819555833004415,
+          0.007043780758976936,
+          0.018825531005859375,
+          0.0031202910467982292,
+          0.004189718514680862,
+          -0.02225491777062416,
+          -0.012670683674514294,
+          0.015341995283961296,
+          -0.030738139525055885,
+          0.008641153573989868,
+          0.020702669396996498,
+          -0.009286419488489628,
+          0.03109912760555744,
+          0.01788696087896824,
+          -0.013573153875768185,
+          -0.0073731825686991215,
+          -0.008573467843234539,
+          0.022453460842370987,
+          -0.0067324284464120865,
+          0.009683506563305855,
+          -0.014620019122958183,
+          0.011037211865186691,
+          0.0046838209964334965,
+          0.005202741362154484,
+          0.010983063839375973,
+          -0.027128256857395172,
+          0.007702583912760019,
+          4.857828389503993e-05,
+          -0.026586774736642838,
+          0.00804552249610424,
+          -0.005771297495812178,
+          0.020919261500239372,
+          -0.001577066839672625,
+          0.012607510201632977,
+          0.024962328374385834,
+          0.00027088209753856063,
+          -0.012381892651319504,
+          -0.0073686703108251095,
+          0.009900099597871304,
+          -0.012860202230513096,
+          0.028355617076158524,
+          -0.00042472509085200727,
+          0.0009227758855558932,
+          -0.005049321334809065,
+          0.008352362550795078,
+          -0.017598170787096024,
+          0.004701870493590832,
+          -0.013970240019261837,
+          -0.016731800511479378,
+          0.0011574181262403727,
+          0.012147250585258007,
+          -0.0006035270052962005,
+          0.013257289305329323,
+          0.011073310859501362,
+          0.0040385546162724495,
+          0.014195858500897884,
+          0.003397800726816058,
+          0.0023283734917640686,
+          0.022309066727757454,
+          0.010450606234371662,
+          0.001882778713479638,
+          -0.0018940595909953117,
+          -0.0022798655554652214,
+          0.03169475868344307,
+          -6.616940663661808e-05,
+          -0.004652234259992838,
+          0.010874766856431961,
+          0.01432220358401537,
+          0.006384977605193853,
+          0.01088379230350256,
+          0.015712007880210876,
+          0.0012713550822809339,
+          0.003219562815502286,
+          0.01046865526586771,
+          0.002475024899467826,
+          0.01240896712988615,
+          0.003131571924313903,
+          -0.006493274122476578,
+          -0.0038084248080849648,
+          0.006218020338565111,
+          0.013284362852573395,
+          -0.01000839564949274,
+          0.017489874735474586,
+          0.009493988007307053,
+          0.00943983905017376,
+          -0.005044809076935053,
+          0.007070854771882296,
+          0.012950449250638485,
+          0.0034384119790047407,
+          -0.007291960064321756,
+          -0.011389175429940224,
+          0.002443438395857811,
+          -0.027759985998272896,
+          0.012363843619823456,
+          0.0014349278062582016,
+          -0.016822047531604767,
+          -0.007314521819353104,
+          -0.003946051467210054,
+          -0.015919575467705727,
+          0.0032872480805963278,
+          -0.002628444693982601,
+          0.026117490604519844,
+          -0.0007608952582813799,
+          -0.02532331645488739,
+          -0.008411023765802383,
+          0.016993515193462372,
+          -0.007341595832258463,
+          0.005153105594217777,
+          0.012255546636879444,
+          0.028084876015782356,
+          0.017020590603351593,
+          -0.0006706482381559908,
+          -0.003896415699273348,
+          -0.004904926288872957,
+          -0.003957332111895084,
+          0.010775495320558548,
+          -0.0159286018460989,
+          -0.008591517806053162,
+          -0.01647910848259926,
+          -0.016587404534220695,
+          0.0023712406400591135,
+          0.009575209580361843,
+          -0.004505583085119724,
+          -0.007201713044196367,
+          -0.0020192773081362247,
+          0.001828630454838276,
+          -0.003657260909676552,
+          0.02043192833662033,
+          -0.01860893703997135,
+          -0.02781413495540619,
+          0.0031586461700499058,
+          -0.0038851348217576742,
+          0.0073776948265731335,
+          0.006660230923444033,
+          0.014656118117272854,
+          -0.002908210502937436,
+          -0.012616534717381,
+          0.04371566325426102,
+          -0.009277394972741604,
+          0.017074737697839737,
+          -0.006750477943569422,
+          0.016325687989592552,
+          0.015062229707837105,
+          0.0060871620662510395,
+          0.0054238466545939445,
+          0.019258717074990273,
+          -0.01647910848259926,
+          -0.002655518939718604,
+          0.007639411371201277,
+          0.009864000603556633,
+          0.02498037740588188,
+          0.008474196307361126,
+          0.0010474296286702156,
+          -0.02044997736811638,
+          -0.003066142788156867,
+          -0.019258717074990273,
+          -0.0009453376405872405,
+          -0.015260772779583931,
+          -0.003970869351178408,
+          0.01233676914125681,
+          -0.02218271978199482,
+          -0.013248263858258724,
+          -0.02160513959825039,
+          -0.006375952623784542,
+          0.009818877093493938,
+          0.0006221404764801264,
+          0.018437469378113747,
+          0.00897957943379879,
+          0.00789661519229412,
+          0.007869540713727474,
+          0.015161501243710518,
+          -0.006014964543282986,
+          0.002693873830139637,
+          0.00550055643543601,
+          0.0009013422532007098,
+          0.011578693985939026,
+          -0.00045151717495173216,
+          -0.013762672431766987,
+          0.012580436654388905,
+          0.0045281448401510715,
+          0.009236783720552921,
+          -0.02582870051264763,
+          -0.023789117112755775,
+          0.01112745888531208,
+          0.0067369407042860985,
+          0.004092703107744455,
+          -0.013356560841202736,
+          0.010658174753189087,
+          -0.003269198816269636,
+          -0.006466199643909931,
+          -0.005762272980064154,
+          0.0036346991546452045,
+          -0.016993515193462372,
+          -0.007783806417137384,
+          0.00637144036591053,
+          0.01664155349135399,
+          0.000557275430765003,
+          0.014881735667586327,
+          0.010288161225616932,
+          0.02283249981701374,
+          0.012733856216073036,
+          -0.026099441573023796,
+          -0.002752534346655011,
+          0.007143052760511637,
+          0.002515635918825865,
+          -0.023608623072504997,
+          -0.009340567514300346,
+          -0.002231357851997018,
+          -0.0010158431250602007,
+          0.026947762817144394,
+          -0.0025449662934988737,
+          0.011362100951373577,
+          -0.009295444004237652,
+          -0.004399542696774006,
+          -0.018843580037355423,
+          0.031008880585432053,
+          -0.003934770356863737,
+          0.005793859250843525,
+          -0.007607824634760618,
+          -0.013762672431766987,
+          -0.012165299616754055,
+          -0.004742481280118227,
+          0.00123638438526541,
+          0.06407539546489716,
+          -0.017562072724103928,
+          0.010477680712938309,
+          -0.008456147275865078,
+          0.006046551279723644,
+          0.022236868739128113,
+          0.010279136709868908,
+          0.0060826498083770275,
+          -0.011163557879626751,
+          -0.01472831517457962,
+          -0.004823703784495592,
+          -0.002286634175106883,
+          0.008451634086668491,
+          0.005157617852091789,
+          -0.0058344705030322075,
+          -0.02144269458949566,
+          -0.025016477331519127,
+          -0.023554475978016853,
+          -0.0012172068236395717,
+          -0.01022498868405819,
+          -0.003677566535770893,
+          0.00041711051017045975,
+          0.0013548335991799831,
+          0.008794573135674,
+          0.01960165612399578,
+          -0.007643923629075289,
+          -0.02391546405851841,
+          -0.027850233018398285,
+          0.014574895612895489,
+          -0.04227171093225479,
+          0.01860893703997135,
+          0.0004881800268776715,
+          -0.027453146874904633,
+          -0.012643609195947647,
+          0.013049720786511898,
+          0.017092788591980934,
+          -0.010901841334998608,
+          0.021731484681367874,
+          0.010685248300433159,
+          -0.011470397934317589,
+          0.0029307722579687834,
+          0.022633954882621765,
+          0.026586774736642838,
+          0.012634584680199623,
+          0.0032060258090496063,
+          0.01853674091398716,
+          0.013221190311014652,
+          -0.018013307824730873,
+          0.007698071654886007,
+          0.01567590981721878,
+          -0.012463115155696869,
+          0.008122232742607594,
+          -0.010432557202875614,
+          0.012986548244953156,
+          -0.005351649131625891,
+          -0.001114550861530006,
+          0.030665941536426544,
+          -0.028608309105038643,
+          0.021208051592111588,
+          -0.031171325594186783,
+          0.008000398986041546,
+          -0.004990661051124334,
+          -0.0004560295201372355,
+          0.010712322778999805,
+          -0.051513005048036575,
+          0.03245283290743828,
+          0.018717234954237938,
+          -5.53468125872314e-05,
+          -0.026297984644770622,
+          -0.01312191877514124,
+          0.021460743620991707,
+          0.007792831398546696,
+          -0.029907865449786186,
+          -0.019475309178233147,
+          0.009548136033117771,
+          -0.0010711194481700659,
+          0.0011500855907797813,
+          -0.007779294159263372,
+          -0.009173611178994179,
+          0.003406825475394726,
+          -0.02615359053015709,
+          0.02781413495540619,
+          -0.018455518409609795,
+          0.02285054884850979,
+          0.007670997641980648,
+          0.006384977605193853,
+          -0.0005039732786826789,
+          -0.008510295301675797,
+          -0.009620334021747112,
+          0.01680399663746357,
+          -0.0053290873765945435,
+          -0.002637469442561269,
+          -0.0173635296523571,
+          -0.0045349132269620895,
+          0.0067188916727900505,
+          0.0027886333409696817,
+          0.0022166927810758352,
+          0.011118434369564056,
+          -0.001003434183076024,
+          0.014051462523639202,
+          0.017372554168105125,
+          0.02622578665614128,
+          0.002996201394125819,
+          0.001920005539432168,
+          0.004189718514680862,
+          0.015991773456335068,
+          -0.0003677566419355571,
+          -0.00531555013731122,
+          0.0037858630530536175,
+          0.01088379230350256,
+          0.013879992999136448,
+          0.01175016351044178,
+          0.008284677751362324,
+          -0.00035986004513688385,
+          0.0044424100778996944,
+          0.016948392614722252,
+          0.020233385264873505,
+          -0.01026108767837286,
+          0.00556372944265604,
+          0.009069826453924179,
+          -0.00795076321810484,
+          0.0038512919563800097,
+          -0.007287447806447744,
+          0.007061830256134272,
+          -0.03245283290743828,
+          0.02936638332903385,
+          -0.0020666569471359253,
+          -0.008550906553864479,
+          -0.006168384570628405,
+          -0.013672425411641598,
+          0.013194115832448006,
+          -0.019637754186987877,
+          0.004367956425994635,
+          -0.02507062442600727,
+          0.008365900255739689,
+          0.002070041373372078,
+          -0.00022604061814490706,
+          0.030575694516301155,
+          0.012688732706010342,
+          0.01795915886759758,
+          -0.00787405390292406,
+          0.06663841009140015,
+          -0.004361187573522329,
+          -0.0046928455121815205,
+          -0.029276136308908463,
+          -0.018518690019845963,
+          -0.009412765502929688,
+          0.0005434563499875367,
+          0.0006892617093399167,
+          0.007413793820887804,
+          -0.0017293586861342192,
+          -0.00839297380298376,
+          0.012417991645634174,
+          -0.01141624990850687,
+          -0.013004597276449203,
+          -0.0007862772908993065,
+          0.015802254900336266,
+          0.005022247321903706,
+          -0.008383949287235737,
+          0.016054946929216385,
+          0.0030187631491571665,
+          0.0011709552491083741,
+          -0.006949021480977535,
+          0.012201398611068726,
+          -0.005107982084155083,
+          0.009078850969672203,
+          0.02573845349252224,
+          -0.003400056855753064,
+          -0.0020170211791992188,
+          0.004103983752429485,
+          -0.01187650952488184,
+          0.0018004282610490918,
+          0.006660230923444033,
+          -0.011037211865186691,
+          0.002903698245063424,
+          0.002323861001059413,
+          0.005536655429750681,
+          -0.010243037715554237,
+          -0.011407224461436272,
+          -0.0008404254913330078,
+          0.005649464204907417,
+          0.01554956380277872,
+          0.0001692131772870198,
+          0.0042777094058692455,
+          0.022652005776762962,
+          -0.01680399663746357,
+          -0.012075052596628666,
+          -0.02283249981701374,
+          0.007147565018385649,
+          0.01423195656388998,
+          0.002506611403077841,
+          -0.018987976014614105,
+          -0.012896301224827766,
+          0.02234516479074955,
+          0.009530086070299149,
+          0.034456316381692886,
+          0.006375952623784542,
+          0.0017146936152130365,
+          0.011154533363878727,
+          0.02756144292652607,
+          0.010423531755805016,
+          -0.019764099270105362,
+          0.0020621446892619133,
+          -0.012156275101006031
+        ],
+        "title": "8 Future Work"
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "1deb6775-c7ea-4a7a-8654-e306d12e688d",
+      "properties": {
+        "page_content": "9 Conclusion 27 ",
+        "keyphrases": [],
+        "embedding": [
+          -0.0026620388962328434,
+          0.009736468084156513,
+          -0.01410017628222704,
+          0.027085084468126297,
+          -0.02409333363175392,
+          -0.023739280179142952,
+          0.002801447408273816,
+          0.09375333786010742,
+          -0.007904241792857647,
+          0.016392672434449196,
+          -0.00255582295358181,
+          -0.03965397924184799,
+          -0.002425265731289983,
+          -0.03738803789019585,
+          -0.047655586153268814,
+          0.007165155373513699,
+          -0.002056828932836652,
+          -0.002341178013011813,
+          -0.018729424104094505,
+          -0.010240994393825531,
+          -0.027563055977225304,
+          -0.00033054203959181905,
+          0.017215846106410027,
+          0.05247071012854576,
+          -0.00470448425039649,
+          -0.0022095143795013428,
+          0.020588204264640808,
+          0.01972077414393425,
+          -0.032856155186891556,
+          0.04528342932462692,
+          -0.008333531208336353,
+          -0.01013477798551321,
+          0.036502905189991,
+          -0.024925358593463898,
+          -0.01793280430138111,
+          0.013728420250117779,
+          0.02297806553542614,
+          -0.014091324992477894,
+          0.013666460290551186,
+          -0.018764829263091087,
+          -0.01830456033349037,
+          0.01579963229596615,
+          -0.004806274548172951,
+          -0.004509754944592714,
+          -0.004859382752329111,
+          -0.028182649984955788,
+          -0.021101580932736397,
+          -0.06061394140124321,
+          -0.015905847772955894,
+          -0.003113457001745701,
+          -0.0059967790730297565,
+          -0.0376712791621685,
+          0.01110842451453209,
+          -0.003370145568624139,
+          0.009320455603301525,
+          0.03606033697724342,
+          0.028288865461945534,
+          0.08199876546859741,
+          0.0022404941264539957,
+          0.011338559910655022,
+          -0.021455634385347366,
+          -0.029917510226368904,
+          -0.009603697806596756,
+          -0.00012419528502505273,
+          -0.009072617627680302,
+          -0.022376174107193947,
+          0.011188087053596973,
+          -0.01899496465921402,
+          -0.006395089440047741,
+          -0.028430486097931862,
+          0.01365760900080204,
+          0.021136987954378128,
+          0.005231138784438372,
+          0.016764428466558456,
+          -0.007868836633861065,
+          0.0030692003201693296,
+          0.05133774131536484,
+          0.036078039556741714,
+          -0.009506333619356155,
+          0.04036208614706993,
+          -0.007550188340246677,
+          -0.010471128858625889,
+          0.008178633637726307,
+          0.028182649984955788,
+          0.04556667059659958,
+          0.010612750425934792,
+          -0.0727933794260025,
+          -0.09934738278388977,
+          -0.010108224116265774,
+          0.004580565728247166,
+          0.022712524980306625,
+          -0.03039548359811306,
+          -0.02081833966076374,
+          -0.026518598198890686,
+          -0.046416398137807846,
+          0.02150874398648739,
+          0.002179641043767333,
+          0.005049686413258314,
+          -0.03901668265461922,
+          0.008023735135793686,
+          0.018145235255360603,
+          0.017313210293650627,
+          0.009824981912970543,
+          0.010559642687439919,
+          0.02203982323408127,
+          -0.011250046081840992,
+          0.024358872324228287,
+          0.0323781818151474,
+          0.009718765504658222,
+          0.05640070512890816,
+          -0.020729824900627136,
+          -0.0049478961154818535,
+          -0.03560006618499756,
+          -0.029563458636403084,
+          0.011418221518397331,
+          -0.004403539001941681,
+          -0.021225500851869583,
+          -0.04762018099427223,
+          0.014162135310471058,
+          0.017667263746261597,
+          -0.009585995227098465,
+          0.004894787911325693,
+          0.02825346030294895,
+          -0.03774208948016167,
+          -0.02223455347120762,
+          0.019826989620923996,
+          0.03239588439464569,
+          0.008992956019937992,
+          -0.007616573479026556,
+          0.010957952588796616,
+          0.006669480819255114,
+          0.015967806801199913,
+          -0.024695223197340965,
+          -0.005629448685795069,
+          0.018410775810480118,
+          -0.024128738790750504,
+          0.010046265088021755,
+          0.0012579959584400058,
+          -0.001970528392121196,
+          0.030802644789218903,
+          -0.044398292899131775,
+          -0.008298126049339771,
+          -0.005554212257266045,
+          0.02388090081512928,
+          -0.006204785779118538,
+          0.012869840487837791,
+          -0.01720699481666088,
+          0.029811294749379158,
+          0.0013188488082960248,
+          0.01239186804741621,
+          0.0006068696384318173,
+          0.011639504693448544,
+          -0.02643008530139923,
+          0.0038591818884015083,
+          -0.002400924451649189,
+          -0.03961857408285141,
+          0.031369131058454514,
+          0.012728218920528889,
+          0.0037750941701233387,
+          0.03660912066698074,
+          0.0031532880384474993,
+          0.054524220526218414,
+          -0.020499691367149353,
+          -0.0051381997764110565,
+          0.009727616794407368,
+          0.00795734953135252,
+          0.027580758556723595,
+          -0.015976659953594208,
+          -0.04273424297571182,
+          0.022871848195791245,
+          0.03616655245423317,
+          0.0002338688645977527,
+          0.025244006887078285,
+          -0.03558236360549927,
+          0.007231540512293577,
+          -0.007213837932795286,
+          -0.008218464441597462,
+          0.055834218859672546,
+          -0.0100551163777709,
+          0.024164143949747086,
+          -0.009453224949538708,
+          0.019579151645302773,
+          -0.01991550251841545,
+          0.004472136963158846,
+          -0.0227656327188015,
+          -0.0074705262668430805,
+          0.06341096013784409,
+          -0.03887506201863289,
+          0.03274993970990181,
+          0.01571997068822384,
+          -0.002052403287962079,
+          -0.022482389584183693,
+          0.03370588272809982,
+          0.010046265088021755,
+          0.004113657865673304,
+          -0.06093258783221245,
+          -0.0016231135232374072,
+          0.035210609436035156,
+          -0.014498486183583736,
+          0.011781126260757446,
+          -0.014560445211827755,
+          -0.022287661209702492,
+          0.018481586128473282,
+          0.010754371993243694,
+          -0.010710114613175392,
+          -0.0028280015103518963,
+          0.005558638367801905,
+          0.010471128858625889,
+          0.014294905588030815,
+          -0.0005191860836930573,
+          0.02747454307973385,
+          0.0150118637830019,
+          -0.008116673678159714,
+          0.01593240164220333,
+          -0.013232745230197906,
+          0.010692412033677101,
+          -0.006930594798177481,
+          0.02209293097257614,
+          -0.013002610765397549,
+          -0.030625617131590843,
+          0.014409972354769707,
+          0.013551393523812294,
+          0.04145964980125427,
+          -0.0002031657932093367,
+          -0.008138801902532578,
+          0.04850531369447708,
+          -0.03760046884417534,
+          0.04698288440704346,
+          0.02729751542210579,
+          -0.04698288440704346,
+          0.009798427112400532,
+          0.018906449899077415,
+          -0.01830456033349037,
+          0.003505128435790539,
+          -0.0057710702531039715,
+          0.008882313966751099,
+          0.0009399011032655835,
+          0.014206391759216785,
+          0.003244014224037528,
+          0.02928021550178528,
+          0.04528342932462692,
+          -0.02372157759964466,
+          -0.028660621494054794,
+          0.0412118136882782,
+          -0.025615762919187546,
+          0.007258094381541014,
+          -0.0412118136882782,
+          0.06741176545619965,
+          0.017260102555155754,
+          0.014082473702728748,
+          0.015224295668303967,
+          -0.0005056324880570173,
+          0.0017757989699020982,
+          0.03958316892385483,
+          0.019490638747811317,
+          0.04740775004029274,
+          0.009957751259207726,
+          -0.0043836236000061035,
+          -0.053249627351760864,
+          0.032094940543174744,
+          -0.0034498076420277357,
+          0.0035715135745704174,
+          -0.030324673280119896,
+          0.01626875251531601,
+          0.02497846633195877,
+          0.002735062502324581,
+          0.02207522839307785,
+          -0.010356061160564423,
+          0.021809687837958336,
+          0.02007482759654522,
+          0.05640070512890816,
+          -0.014170986600220203,
+          -0.00489036226645112,
+          0.01175457239151001,
+          -0.018393073230981827,
+          -0.031953319907188416,
+          0.032271966338157654,
+          -0.018516991287469864,
+          -0.013347812928259373,
+          0.019065774977207184,
+          -0.0007838963065296412,
+          0.03476804122328758,
+          -0.005558638367801905,
+          -0.020712122321128845,
+          -0.017410574480891228,
+          -0.022110633552074432,
+          -0.0010062861256301403,
+          0.01040916983038187,
+          -0.030430888757109642,
+          0.00484168017283082,
+          0.013480582274496555,
+          -0.01138281635940075,
+          -0.009152280166745186,
+          0.01111727673560381,
+          -0.004633673466742039,
+          0.002014785073697567,
+          0.018853342160582542,
+          0.007492654956877232,
+          -0.008523834869265556,
+          -0.0014095749938860536,
+          0.003102392889559269,
+          0.0005645491764880717,
+          -0.03441398963332176,
+          -0.013595649972558022,
+          -0.008453024551272392,
+          0.01607402414083481,
+          0.016498887911438942,
+          0.03584790602326393,
+          -0.0714125707745552,
+          0.011789977550506592,
+          0.035901013761758804,
+          0.016339562833309174,
+          0.000724703015293926,
+          -0.04418586194515228,
+          -0.029775889590382576,
+          0.020375771448016167,
+          0.01302031334489584,
+          0.010046265088021755,
+          -0.013454028405249119,
+          -0.004485413897782564,
+          0.012763625010848045,
+          -0.011728018522262573,
+          -0.018605506047606468,
+          0.030625617131590843,
+          0.020499691367149353,
+          -0.03260831534862518,
+          -0.021243203431367874,
+          0.008430896326899529,
+          0.01988009735941887,
+          -0.012444976717233658,
+          0.025633465498685837,
+          -0.022482389584183693,
+          0.019490638747811317,
+          -0.001468215137720108,
+          -0.014799431897699833,
+          -0.02713819220662117,
+          -0.023438334465026855,
+          0.02025185339152813,
+          -0.03657371550798416,
+          -0.006908466573804617,
+          0.019472936168313026,
+          0.0032019703648984432,
+          -0.008599071763455868,
+          0.0008060246473178267,
+          -0.01640152372419834,
+          -0.038237765431404114,
+          0.029563458636403084,
+          -0.0561174601316452,
+          -0.003299335017800331,
+          0.01527740340679884,
+          0.0009227516129612923,
+          -0.009152280166745186,
+          0.008160930126905441,
+          0.01793280430138111,
+          -0.0023124110884964466,
+          -0.008006032556295395,
+          -0.028111839666962624,
+          -0.027580758556723595,
+          -0.009470928460359573,
+          -0.008935422636568546,
+          0.013073421083390713,
+          0.006381812505424023,
+          0.014896796084940434,
+          -0.006311001721769571,
+          -0.04907179996371269,
+          -0.025066979229450226,
+          0.016339562833309174,
+          0.01293179951608181,
+          0.020570501685142517,
+          0.0008663243497721851,
+          -0.02223455347120762,
+          -0.002160832053050399,
+          0.021225500851869583,
+          -0.017083076760172844,
+          -0.01575537584722042,
+          -0.006549987476319075,
+          -0.013648757711052895,
+          0.011612950824201107,
+          -0.026890354230999947,
+          0.011294303461909294,
+          -0.02892616204917431,
+          0.012276801280677319,
+          0.006704885978251696,
+          0.004992152564227581,
+          0.0151092279702425,
+          -0.0027306366246193647,
+          0.012498084455728531,
+          0.017180440947413445,
+          -0.0073997159488499165,
+          -0.04393802583217621,
+          -0.04854071885347366,
+          -0.012329909019172192,
+          -0.004717761185020208,
+          -0.02095996029675007,
+          0.00849728100001812,
+          0.011993558146059513,
+          0.0390520878136158,
+          0.0034984899684786797,
+          0.0045849913731217384,
+          -0.01707422360777855,
+          -0.016773279756307602,
+          0.009293901734054089,
+          -0.006572116166353226,
+          -0.009621400386095047,
+          0.021030770614743233,
+          0.031227508559823036,
+          0.0006782335112802684,
+          -0.023650765419006348,
+          -0.008315828628838062,
+          -0.008979679085314274,
+          0.0402204655110836,
+          -0.03604263439774513,
+          0.0002623590989969671,
+          -0.046239372342824936,
+          0.029970619827508926,
+          0.00285012973472476,
+          0.020481988787651062,
+          0.0024186272639781237,
+          -0.027279812842607498,
+          -0.0227656327188015,
+          -0.021632662042975426,
+          -0.0019583578687161207,
+          0.027616163715720177,
+          0.001016243826597929,
+          -0.021650364622473717,
+          0.04138883948326111,
+          -0.01610942929983139,
+          0.019101180136203766,
+          0.00011299282050458714,
+          0.011312006041407585,
+          -0.04167208448052406,
+          0.007231540512293577,
+          -0.02044658362865448,
+          0.01989779993891716,
+          0.04662882909178734,
+          -0.0034741489216685295,
+          -0.03331642225384712,
+          0.011533289216458797,
+          -0.011409370228648186,
+          0.021491041406989098,
+          0.01742827706038952,
+          -0.019118882715702057,
+          0.00047078036004677415,
+          0.04312370344996452,
+          0.019242800772190094,
+          -0.00578434718772769,
+          -0.030360078439116478,
+          -0.007620999123901129,
+          0.0010770967928692698,
+          0.010878290049731731,
+          -0.03202413022518158,
+          0.038981277495622635,
+          -0.037706684321165085,
+          -0.006470325402915478,
+          0.04407964646816254,
+          0.005939245689660311,
+          -0.010736668482422829,
+          0.026129139587283134,
+          -0.019207395613193512,
+          -0.04563748091459274,
+          0.0004276301187928766,
+          0.054347194731235504,
+          0.022907253354787827,
+          -0.025598060339689255,
+          -0.018481586128473282,
+          0.016746725887060165,
+          -0.007138601504266262,
+          0.013427474536001682,
+          -0.004531883168965578,
+          0.00865217950195074,
+          -0.014250649139285088,
+          -0.0037308374885469675,
+          -0.06309231370687485,
+          0.029563458636403084,
+          -0.02439427748322487,
+          -0.0195260439068079,
+          0.01194045040756464,
+          -0.009462077170610428,
+          -0.012533489614725113,
+          0.02533251978456974,
+          -0.02425265684723854,
+          0.03597182407975197,
+          0.015135781839489937,
+          0.0024186272639781237,
+          -0.023579955101013184,
+          0.05077125504612923,
+          0.04754937067627907,
+          -0.015879293903708458,
+          -0.03604263439774513,
+          0.0064039407297968864,
+          -0.0072979251854121685,
+          -0.031900208443403244,
+          0.00047575923963449895,
+          0.001995975850149989,
+          0.008559240959584713,
+          -0.07343067228794098,
+          -0.004664653446525335,
+          -0.07930795848369598,
+          0.007979477755725384,
+          0.031015075743198395,
+          0.003403338138014078,
+          0.0006732546607963741,
+          -0.007824580185115337,
+          0.015250849537551403,
+          0.02331441454589367,
+          -0.02333211898803711,
+          -0.014330310747027397,
+          -0.016906049102544785,
+          -0.010444574989378452,
+          0.030165348201990128,
+          0.0368923619389534,
+          -0.009276199154555798,
+          0.007027959916740656,
+          -0.00438583642244339,
+          -0.014170986600220203,
+          -0.01679983362555504,
+          0.01257774606347084,
+          0.01864091120660305,
+          -0.014224094338715076,
+          -0.02080063708126545,
+          -0.00154677068348974,
+          -0.0028523425571620464,
+          -0.00985153578221798,
+          0.03876884654164314,
+          -0.023650765419006348,
+          0.02133171632885933,
+          -0.006076441146433353,
+          -0.01828685775399208,
+          0.05501989647746086,
+          -0.022747930139303207,
+          0.02604062668979168,
+          0.00314443651586771,
+          -0.0062490422278642654,
+          0.0011235662968829274,
+          0.027085084468126297,
+          0.004456647206097841,
+          -0.013772676698863506,
+          0.03639668971300125,
+          0.0011141618015244603,
+          -0.002290282864123583,
+          0.014047068543732166,
+          0.010179034434258938,
+          -0.003905651392415166,
+          -0.020906852558255196,
+          -0.011453626677393913,
+          0.0139320008456707,
+          -0.005571915302425623,
+          -0.007160729728639126,
+          -0.0034055509604513645,
+          -0.005881711840629578,
+          0.013338961638510227,
+          -0.006983703002333641,
+          0.03436088189482689,
+          -0.0323781818151474,
+          -0.006727014202624559,
+          0.007067790720611811,
+          0.004978875629603863,
+          0.02368617057800293,
+          0.020694419741630554,
+          0.017437128350138664,
+          0.02005712501704693,
+          0.010400318540632725,
+          -0.010940250009298325,
+          -0.0029342174530029297,
+          -0.0001850482076406479,
+          -0.016357267275452614,
+          0.02752765081822872,
+          0.036644525825977325,
+          -0.020322663709521294,
+          -0.04291126877069473,
+          0.0014018301153555512,
+          0.01148903276771307,
+          -0.028979269787669182,
+          -0.03381209820508957,
+          0.01830456033349037,
+          0.034290071576833725,
+          0.019791584461927414,
+          -0.03400682657957077,
+          -0.02409333363175392,
+          0.008289274759590626,
+          0.01347173098474741,
+          0.008090119808912277,
+          0.02025185339152813,
+          -0.003168777795508504,
+          0.009435522370040417,
+          -0.0005703578935936093,
+          -0.004082677885890007,
+          -0.013418623246252537,
+          -0.03529912233352661,
+          0.014480783604085445,
+          -0.0067668454721570015,
+          -0.0022194720804691315,
+          0.009789575822651386,
+          -0.017755776643753052,
+          -0.027014274150133133,
+          0.005819752346724272,
+          0.026890354230999947,
+          0.010382615961134434,
+          -0.012781327590346336,
+          -0.02281874045729637,
+          -0.006811101920902729,
+          0.0010627133306115866,
+          0.011896193958818913,
+          9.674231841927394e-05,
+          0.00715187843888998,
+          0.02080063708126545,
+          0.0003053710388485342,
+          -0.016374969854950905,
+          -0.037175606936216354,
+          0.03324561193585396,
+          -0.01565801165997982,
+          -0.010639304295182228,
+          0.031245211139321327,
+          -0.010789777152240276,
+          -0.004762018099427223,
+          -0.03351115435361862,
+          -0.008253869600594044,
+          -0.02386319823563099,
+          0.01110842451453209,
+          0.013383218087255955,
+          0.025580357760190964,
+          -0.02625305764377117,
+          -0.03437858447432518,
+          0.03184710070490837,
+          0.015684565529227257,
+          0.0005410378216765821,
+          -0.012648557312786579,
+          -0.004916916601359844,
+          -0.03873344138264656,
+          -0.03653831034898758,
+          0.019083477556705475,
+          0.0051204971969127655,
+          0.002542545786127448,
+          -0.022730227559804916,
+          -0.0076608299277722836,
+          0.02223455347120762,
+          0.04252181202173233,
+          0.004611545242369175,
+          0.022871848195791245,
+          -0.012329909019172192,
+          0.02352684736251831,
+          -0.031085887923836708,
+          -0.019826989620923996,
+          0.024482792243361473,
+          -0.019437531009316444,
+          -0.00976302195340395,
+          0.01013477798551321,
+          0.025757383555173874,
+          0.020552799105644226,
+          0.017552196979522705,
+          0.032856155186891556,
+          0.029634268954396248,
+          0.01862320862710476,
+          0.025721978396177292,
+          0.009532887488603592,
+          -0.019083477556705475,
+          -0.016020916402339935,
+          -0.00664292648434639,
+          -0.027633866295218468,
+          -0.004558437503874302,
+          -0.024836845695972443,
+          -0.016065172851085663,
+          0.021066175773739815,
+          0.0036224087234586477,
+          0.0028147243428975344,
+          0.013436325825750828,
+          -0.03653831034898758,
+          -0.00505853770300746,
+          -0.0048107001930475235,
+          0.008001606911420822,
+          -0.0013974044704809785,
+          -0.0008524941513314843,
+          0.017100779339671135,
+          0.01384348701685667,
+          -0.018056722357869148,
+          -0.0036334728356450796,
+          -0.01778233051300049,
+          0.030501699075102806,
+          -0.018552398309111595,
+          -0.006532284896820784,
+          -0.03958316892385483,
+          0.019986312836408615,
+          0.023438334465026855,
+          0.027899406850337982,
+          0.004341579508036375,
+          0.001363105489872396,
+          0.006833230145275593,
+          0.005660428665578365,
+          -0.0009476460400037467,
+          -0.0063065760768949986,
+          -0.003874671645462513,
+          0.014631256461143494,
+          -0.012117477133870125,
+          0.009568292647600174,
+          -0.0123122064396739,
+          -0.02223455347120762,
+          -0.024907656013965607,
+          0.0015157910529524088,
+          0.02294265851378441,
+          0.03922911360859871,
+          0.031758587807416916,
+          -0.016507739201188087,
+          -0.03506898880004883,
+          0.02278333529829979,
+          0.02223455347120762,
+          -0.009771873243153095,
+          -0.001461576670408249,
+          -0.005439145024865866,
+          -0.014525040052831173,
+          -0.017295507714152336,
+          0.0014593638479709625,
+          -0.01665821112692356,
+          -0.013896595686674118,
+          0.0386626310646534,
+          0.013542542234063148,
+          0.012197138741612434,
+          -0.0007601083489134908,
+          -0.017569899559020996,
+          -0.01665821112692356,
+          0.0002082829741993919,
+          0.019207395613193512,
+          -0.01320619136095047,
+          -0.015348214656114578,
+          0.011196938343346119,
+          -0.034095339477062225,
+          0.011250046081840992,
+          0.007793600205332041,
+          0.0019760604482144117,
+          0.022128336131572723,
+          0.008253869600594044,
+          -0.014445378445088863,
+          -0.02025185339152813,
+          -0.005226713139563799,
+          0.0026310591492801905,
+          -0.0037551787681877613,
+          0.016693616285920143,
+          0.008218464441597462,
+          0.027917109429836273,
+          0.027226705104112625,
+          -0.009258495643734932,
+          0.0123122064396739,
+          0.022323066368699074,
+          -0.010754371993243694,
+          -0.004567288793623447,
+          0.00045031163608655334,
+          -0.013958554714918137,
+          0.03784830868244171,
+          0.043725594878196716,
+          -0.016330711543560028,
+          0.013179637491703033,
+          -0.01194045040756464,
+          -0.03167007490992546,
+          0.04737234115600586,
+          -0.018888747319579124,
+          -0.01880023442208767,
+          0.00986923836171627,
+          0.007687384262681007,
+          -0.02115469053387642,
+          -0.017198143526911736,
+          0.007758195046335459,
+          -0.013091123662889004,
+          0.010966803878545761,
+          0.003606918966397643,
+          0.040432896465063095,
+          0.01972077414393425,
+          -0.006864210125058889,
+          -0.024111036211252213,
+          -0.018570100888609886,
+          0.015144633129239082,
+          0.00215087435208261,
+          0.03976019471883774,
+          -0.0032395885791629553,
+          0.0244296845048666,
+          -0.029616566374897957,
+          0.008050289005041122,
+          -0.013108826242387295,
+          0.004257491789758205,
+          -0.0070722163654863834,
+          -0.016923751682043076,
+          0.012427274137735367,
+          -0.018322262912988663,
+          0.002898812061175704,
+          -0.03289156034588814,
+          -0.028642918914556503,
+          -0.018499288707971573,
+          0.03455561026930809,
+          0.05169179290533066,
+          -0.01748138666152954,
+          -0.009152280166745186,
+          0.02207522839307785,
+          -0.013073421083390713,
+          -0.0024650967679917812,
+          -0.008758395910263062,
+          -0.0002622207975946367,
+          0.03432547673583031,
+          -0.03368818014860153,
+          -0.028483595699071884,
+          -0.0011606313055381179,
+          -0.02494306117296219,
+          -0.03653831034898758,
+          0.000632870418485254,
+          -0.0047929976135492325,
+          -0.03618425503373146,
+          0.015622605569660664,
+          0.00015973893459886312,
+          0.0184461809694767,
+          0.004832828883081675,
+          0.011338559910655022,
+          -0.010798628441989422,
+          0.006417217664420605,
+          0.03239588439464569,
+          -0.04340694472193718,
+          0.017906250432133675,
+          0.007727215066552162,
+          -0.024695223197340965,
+          0.034449394792318344,
+          0.014710918068885803,
+          -0.015215444378554821,
+          -0.019809287041425705,
+          0.009050489403307438,
+          -0.002253771061077714,
+          -0.01918969303369522,
+          0.01691490039229393,
+          -0.01897726207971573,
+          0.010887141339480877,
+          0.03370588272809982,
+          0.013586798682808876,
+          0.005833029747009277,
+          0.04089316725730896,
+          0.05463043600320816,
+          -0.031209805980324745,
+          0.002666464541107416,
+          -0.01410017628222704,
+          0.03632587566971779,
+          -0.00863447692245245,
+          0.009807278402149677,
+          -0.00397867476567626,
+          -0.010426872409880161,
+          -0.014348013326525688,
+          0.02062360942363739,
+          -0.009621400386095047,
+          0.018322262912988663,
+          0.0006389557383954525,
+          -0.0035892161540687084,
+          -0.00022640054521616548,
+          0.0034918515011668205,
+          -0.021668067201972008,
+          -0.012622003443539143,
+          0.009444373659789562,
+          0.021809687837958336,
+          0.018393073230981827,
+          0.02674873359501362,
+          -0.021119285374879837,
+          0.007253668736666441,
+          -0.0009448799537494779,
+          -0.005656003020703793,
+          0.013781527988612652,
+          0.027439137920737267,
+          0.01675557717680931,
+          0.022181443870067596,
+          0.015144633129239082,
+          0.05463043600320816,
+          -0.016516590490937233,
+          -0.03926451876759529,
+          0.034838851541280746,
+          -0.0025159919168800116,
+          -0.041318029165267944,
+          -0.030165348201990128,
+          -0.0317939929664135,
+          -0.03958316892385483,
+          -0.02264171466231346,
+          -0.0018941856687888503,
+          0.018711721524596214,
+          0.004978875629603863,
+          -0.010993357747793198,
+          -0.02117239311337471,
+          0.0037264118436723948,
+          -0.011612950824201107,
+          0.03563547506928444,
+          -0.01830456033349037,
+          -0.010957952588796616,
+          -0.013985108584165573,
+          -0.03513979911804199,
+          -0.00742184417322278,
+          -0.014586999081075191,
+          0.011462477967143059,
+          -0.014728620648384094,
+          -0.030147645622491837,
+          -0.0024119887966662645,
+          -0.022393876686692238,
+          -0.008860185742378235,
+          -0.008599071763455868,
+          0.00868315901607275,
+          -0.030661022290587425,
+          0.022146038711071014,
+          0.030625617131590843,
+          0.00010808309161802754,
+          0.0036047061439603567,
+          0.016330711543560028,
+          0.0009996476583182812,
+          -0.011550991795957088,
+          0.008815929293632507,
+          0.026465490460395813,
+          -0.03894587233662605,
+          -0.01374612282961607,
+          -0.027580758556723595,
+          0.000777257839217782,
+          0.0016806471394374967,
+          -0.02281874045729637,
+          0.004899213556200266,
+          0.0005875073256902397,
+          -0.01707422360777855,
+          -0.04177829995751381,
+          -0.03887506201863289,
+          0.01320619136095047,
+          0.035741690546274185,
+          -0.002186279743909836,
+          0.005284246988594532,
+          -0.008970827795565128,
+          0.013515988364815712,
+          0.023031173273921013,
+          -0.02081833966076374,
+          -0.014702066779136658,
+          0.021791985258460045,
+          0.015321659855544567,
+          -0.018339965492486954,
+          -0.0155517952516675,
+          0.02156185172498226,
+          -0.016242198646068573,
+          0.03313939645886421,
+          -0.006815527565777302,
+          0.021278608590364456,
+          0.052895575761795044,
+          -0.01710963062942028,
+          0.04436288774013519,
+          -0.006651777774095535,
+          -0.01693260297179222,
+          0.015711119398474693,
+          0.018924154341220856,
+          -0.01473747193813324,
+          0.020747527480125427,
+          0.037706684321165085,
+          0.020765231922268867,
+          0.020517393946647644,
+          -0.028873054310679436,
+          0.013108826242387295,
+          0.019260503351688385,
+          0.0506296344101429,
+          0.009134577587246895,
+          -0.013790379278361797,
+          -0.005164753645658493,
+          0.010639304295182228,
+          0.00314443651586771,
+          -0.02945724129676819,
+          0.0012380804400891066,
+          0.00031864806078374386,
+          -0.04068073257803917,
+          0.006793399341404438,
+          0.010028562508523464,
+          0.03253750503063202,
+          0.0134894335642457,
+          0.0027129340451210737,
+          -0.03653831034898758,
+          0.01465781033039093,
+          -0.01356024481356144,
+          0.02660711109638214,
+          0.03057250939309597,
+          0.022323066368699074,
+          -0.02692575938999653,
+          -0.0022947085089981556,
+          -0.007687384262681007,
+          0.017861993983387947,
+          0.055975839495658875,
+          0.020765231922268867,
+          -0.020198745653033257,
+          -0.021278608590364456,
+          -0.01789739914238453,
+          -0.013648757711052895,
+          -0.002648761961609125,
+          0.022553199902176857,
+          0.02368617057800293,
+          -0.01240072026848793,
+          0.022146038711071014,
+          -0.027085084468126297,
+          -0.007713938131928444,
+          -0.0033458045218139887,
+          -0.033581964671611786,
+          0.0027439137920737267,
+          0.024659818038344383,
+          0.00932930689305067,
+          0.00922309048473835,
+          -0.007820154540240765,
+          0.0041933199390769005,
+          -0.025775086134672165,
+          0.0007910880376584828,
+          0.0015677926130592823,
+          0.02474833093583584,
+          -0.008811503648757935,
+          0.008793801069259644,
+          -0.012338760308921337,
+          -0.020216448232531548,
+          -0.01229450386017561,
+          -0.023845495656132698,
+          0.013321259059011936,
+          0.021437931805849075,
+          0.006408366374671459,
+          0.020198745653033257,
+          0.005226713139563799,
+          0.017419425770640373,
+          0.025013871490955353,
+          -0.007364310324192047,
+          -0.00212432025000453,
+          0.021738877519965172,
+          0.01140051893889904,
+          -0.00790866743773222,
+          0.019118882715702057,
+          -0.0010660325642675161,
+          0.011471329256892204,
+          0.01076322328299284,
+          0.014277203008532524,
+          -0.012365314178168774,
+          -0.03549385070800781,
+          -0.0027107212226837873,
+          0.006045461632311344,
+          0.016569698229432106,
+          0.01640152372419834,
+          0.017773479223251343,
+          0.015861591324210167,
+          0.014887944795191288,
+          -0.005062963347882032,
+          0.0036157702561467886,
+          -0.007342182099819183,
+          0.004450008273124695,
+          -0.04450451210141182,
+          0.012223693542182446,
+          -0.006855358835309744,
+          0.005594043526798487,
+          -0.013126528821885586,
+          0.002008146606385708,
+          -0.019083477556705475,
+          -0.012241396121680737,
+          -0.014091324992477894,
+          0.02713819220662117,
+          0.02444738708436489,
+          -0.0022626223508268595,
+          -0.0031400108709931374,
+          -0.011011060327291489,
+          0.022553199902176857,
+          0.0026067181024700403,
+          0.0017392872832715511,
+          -0.009789575822651386,
+          0.016419226303696632,
+          0.033174801617860794,
+          -0.0045849913731217384,
+          -0.011418221518397331,
+          0.013480582274496555,
+          -0.01057734526693821,
+          -0.006271170452237129,
+          0.022836443036794662,
+          -0.017808884382247925,
+          -0.015410173684358597,
+          0.01502956636250019,
+          -0.004149063024669886,
+          0.001848822575993836,
+          0.009984305128455162,
+          0.015604902990162373,
+          0.009833833202719688,
+          -0.007461674977093935,
+          0.006213637068867683,
+          0.01204666681587696,
+          -0.011143830604851246,
+          -0.01783544011414051,
+          0.007682958617806435,
+          -0.010090521536767483,
+          0.013498285785317421,
+          0.020694419741630554,
+          -0.031953319907188416,
+          -0.014418824575841427,
+          -0.012790178880095482,
+          -0.00841761939227581,
+          0.013498285785317421,
+          -0.008484004065394402,
+          0.022376174107193947,
+          0.021278608590364456,
+          -0.008253869600594044,
+          -0.016295306384563446,
+          -0.02547414042055607,
+          0.017224697396159172,
+          0.012701665051281452,
+          0.0007081067888066173,
+          -0.0437610000371933,
+          0.0021221074275672436,
+          0.011427072808146477,
+          0.012152882292866707,
+          0.001480385661125183,
+          -0.0010759903816506267,
+          -0.009692211635410786,
+          0.06925284117460251,
+          -0.01455159392207861,
+          -0.0004137998912483454,
+          -0.007187283597886562,
+          -0.02658940851688385,
+          0.005678131245076656,
+          -0.028465893119573593,
+          0.0076431273482739925,
+          0.002303559798747301,
+          0.014852539636194706,
+          -0.004815125837922096,
+          0.013250447809696198,
+          0.005589617881923914,
+          0.024872250854969025,
+          0.023296711966395378,
+          0.01991550251841545,
+          -0.02170347236096859,
+          -0.0010394785786047578,
+          0.0041601271368563175,
+          -0.025651168078184128,
+          -0.003427679417654872,
+          -0.003272780915722251,
+          -0.014507337473332882,
+          -0.00269080582074821,
+          -0.025615762919187546,
+          0.014622405171394348,
+          -0.00782900582998991,
+          0.005377185996621847,
+          -0.02589900605380535,
+          -0.0006267851567827165,
+          0.018534693866968155,
+          0.03349345177412033,
+          0.008103396743535995,
+          0.005151476711034775,
+          -0.01013477798551321,
+          0.01778233051300049,
+          -0.0017912888433784246,
+          0.04124721884727478,
+          0.01644578017294407,
+          -0.005549786612391472,
+          -0.006430494599044323,
+          0.040999382734298706,
+          0.02007482759654522,
+          -0.027226705104112625,
+          -0.0025690998882055283,
+          0.005748942028731108,
+          0.025279412046074867,
+          -0.028642918914556503,
+          -0.008550389669835567,
+          -0.019968610256910324,
+          -0.0044367313385009766,
+          0.008780524134635925,
+          0.016047470271587372,
+          -0.00874069333076477,
+          -0.012551192194223404,
+          0.0018886534962803125,
+          0.011710315942764282,
+          0.017490237951278687,
+          -0.009267346933484077,
+          0.006204785779118538,
+          0.016313008964061737,
+          -0.02804102748632431,
+          0.0006511263200081885,
+          -0.009585995227098465,
+          0.024323467165231705,
+          0.027191299945116043,
+          0.0027660420164465904,
+          -0.0030736259650439024,
+          0.00939126592129469,
+          0.007479377556592226,
+          -0.007143027149140835,
+          0.037919119000434875,
+          -0.010418021120131016,
+          -0.00046386526082642376,
+          -0.004290684591978788,
+          0.003370145568624139,
+          0.014808283187448978,
+          -0.014967607334256172,
+          0.0015711118467152119,
+          -0.008700861595571041,
+          -0.0021530871745198965,
+          0.006386238150298595,
+          -0.006341981235891581,
+          0.015188890509307384,
+          0.0034807873889803886,
+          -0.029705079272389412,
+          0.010019711218774319,
+          0.0017337552271783352,
+          0.005434719379991293,
+          -0.025704275816679,
+          -0.03037778101861477,
+          0.015126930549740791,
+          -0.01738402061164379,
+          -0.01883563958108425,
+          -0.017746925354003906,
+          -0.022924955934286118,
+          0.019844692200422287,
+          -0.0025027149822562933,
+          -0.02731521800160408,
+          0.03073183447122574,
+          0.031192103400826454,
+          -0.005098368972539902,
+          0.03197102248668671,
+          -0.010586196556687355,
+          0.012613152153789997,
+          0.0349273681640625,
+          -0.012630854733288288,
+          -0.0015025140019133687,
+          0.013852338306605816,
+          -0.008886739611625671,
+          0.005704685114324093,
+          -0.008284849114716053,
+          -0.022340768948197365,
+          -0.020552799105644226,
+          -0.018924154341220856,
+          -0.012223693542182446,
+          0.00976302195340395,
+          -0.0029076633509248495,
+          0.012807881459593773,
+          -0.010143629275262356,
+          -0.015348214656114578,
+          0.02333211898803711,
+          -0.0035670879296958447,
+          -0.01989779993891716,
+          -0.005930394399911165,
+          0.0222699586302042,
+          0.007483803667128086,
+          -0.01991550251841545,
+          0.032626017928123474,
+          0.027279812842607498,
+          -0.025739680975675583,
+          -0.014498486183583736,
+          -0.024358872324228287,
+          0.009966602548956871,
+          -0.0032418014016002417,
+          -0.025456437841057777,
+          0.006833230145275593,
+          -0.025084681808948517,
+          0.001977166859433055,
+          -0.003029369283467531,
+          -0.008961976505815983,
+          -0.0017160525312647223,
+          -0.016790982335805893,
+          0.006063164211809635,
+          -0.014179837889969349,
+          -0.017198143526911736,
+          -0.03113899566233158,
+          0.017693817615509033,
+          0.018587803468108177,
+          0.04705369472503662,
+          -0.005147051066160202,
+          -0.0034918515011668205,
+          0.0008707500528544188,
+          0.026182247325778008,
+          0.013622203841805458,
+          -0.012701665051281452,
+          0.0034874258562922478,
+          -0.007474951911717653,
+          0.014153284020721912,
+          -0.01638382114470005,
+          0.0076077221892774105,
+          0.022393876686692238,
+          -0.014427675865590572,
+          -0.0023677321150898933,
+          0.008537111803889275,
+          0.0026620388962328434,
+          -0.005629448685795069,
+          0.07944957911968231,
+          -0.00874069333076477,
+          0.008692010305821896,
+          -0.007174006663262844,
+          0.0024783737026154995,
+          -0.011019911617040634,
+          0.000400799501221627,
+          0.03558236360549927,
+          -0.007873262278735638,
+          0.01102876290678978,
+          0.024659818038344383,
+          0.008523834869265556,
+          0.004376985132694244,
+          0.007921944372355938,
+          0.02839508093893528,
+          -0.009647955186665058,
+          0.015613754279911518,
+          0.012064369395375252,
+          -0.011745721101760864,
+          -0.004244214855134487,
+          -0.012082071974873543,
+          0.0007075535831972957,
+          -0.0009885834297165275,
+          0.004545160569250584,
+          0.012595449574291706,
+          0.02554495260119438,
+          0.014109027571976185,
+          0.014817134477198124,
+          0.006434920243918896,
+          0.019632259383797646,
+          0.0021110433153808117,
+          -0.0111349793151021,
+          -0.009258495643734932,
+          0.00836893729865551,
+          0.005501104518771172,
+          0.0034697232767939568,
+          -0.006850933190435171,
+          0.009506333619356155,
+          -0.017861993983387947,
+          0.02062360942363739,
+          -0.02080063708126545,
+          -0.009524036198854446,
+          -0.003230737056583166,
+          -0.020729824900627136,
+          0.007342182099819183,
+          0.045141804963350296,
+          0.004700058605521917,
+          -0.005868434906005859,
+          0.04776180163025856,
+          -0.015162336640059948,
+          -0.03912289813160896,
+          0.034732636064291,
+          0.014932201243937016,
+          -0.029439538717269897,
+          -0.0017459257505834103,
+          0.00014438739162869751,
+          0.028837649151682854,
+          0.002750552259385586,
+          -0.00041960857925005257,
+          0.015153485350310802,
+          -0.02188050001859665,
+          0.016242198646068573,
+          -0.004478775430470705,
+          0.0007838963065296412,
+          -0.01915428787469864,
+          0.02602292411029339,
+          -0.006182657089084387,
+          -0.005855157971382141,
+          0.0013011462287977338,
+          -0.014728620648384094,
+          -0.004215448163449764,
+          -0.00939126592129469,
+          -0.00669603468850255,
+          -0.00978072453290224,
+          -0.019402125850319862,
+          0.015702268108725548,
+          0.0030758387874811888,
+          -0.02749224565923214,
+          -0.0004948449204675853,
+          0.003244014224037528,
+          -0.01620679348707199,
+          -0.009568292647600174,
+          -0.0009376882808282971,
+          0.014224094338715076,
+          -0.02658940851688385,
+          0.013126528821885586,
+          -0.002604505280032754,
+          -0.010887141339480877,
+          0.022199146449565887,
+          -0.007634276058524847,
+          0.016313008964061737,
+          0.009453224949538708,
+          0.013091123662889004,
+          -0.0008679840248078108,
+          0.0008591326768510044,
+          0.00792637001723051,
+          0.004801848903298378,
+          -0.03788371384143829,
+          0.0070899189449846745,
+          0.014507337473332882,
+          -0.01748138666152954,
+          -0.00872299075126648,
+          -0.006850933190435171,
+          -0.034467097371816635,
+          0.0058064754121005535,
+          -0.011913896538317204,
+          0.022022120654582977,
+          0.01438341848552227,
+          0.002887747948989272,
+          0.011373965069651604,
+          -0.014294905588030815,
+          0.01194045040756464,
+          0.0025270560290664434,
+          -0.00489036226645112,
+          0.05983502045273781,
+          -0.023579955101013184,
+          -0.02205752581357956,
+          -0.008315828628838062,
+          0.02929791808128357,
+          -0.0014814921887591481,
+          0.01646348275244236,
+          0.04195532575249672,
+          0.030448591336607933,
+          -0.004149063024669886,
+          0.0025093534495681524,
+          0.0067491428926587105,
+          -0.00010379572631791234,
+          -0.006961574777960777,
+          0.01328585296869278,
+          -0.027970217168331146,
+          0.01710963062942028,
+          -0.002752765081822872,
+          -0.02731521800160408,
+          -0.012321057729423046,
+          0.009081468917429447,
+          0.023916305974125862,
+          -0.005894988775253296,
+          0.014586999081075191,
+          -0.008559240959584713,
+          -0.018764829263091087,
+          -0.00912572629749775,
+          0.009506333619356155,
+          0.020694419741630554,
+          0.0006511263200081885,
+          0.02823575772345066,
+          -0.030625617131590843,
+          0.028483595699071884,
+          0.006815527565777302,
+          -0.003624621545895934,
+          0.015374768525362015,
+          -0.005231138784438372,
+          0.0009896898409351707,
+          0.009285050444304943,
+          0.015082674100995064,
+          -0.021951310336589813,
+          -0.046558018773794174,
+          0.013940852135419846,
+          0.02515549212694168,
+          0.024146441370248795,
+          0.0037573915906250477,
+          -0.012064369395375252,
+          -0.0193313155323267,
+          0.022163741290569305,
+          -0.016684764996170998,
+          -0.010152480565011501,
+          0.010541939176619053,
+          0.007337756454944611,
+          0.0044367313385009766,
+          0.0267841387540102,
+          0.019968610256910324,
+          0.003317037597298622,
+          -0.004443369805812836,
+          -0.0021044048480689526,
+          -0.0027239981573075056,
+          0.010825182311236858,
+          -0.030873455107212067,
+          0.0033944868482649326,
+          0.006125123705714941,
+          0.0008646647329442203,
+          -0.007116473279893398,
+          0.006833230145275593,
+          0.0144719323143363,
+          -0.02409333363175392,
+          -0.05679016187787056,
+          0.021136987954378128,
+          -0.01799476332962513,
+          0.015888145193457603,
+          -0.0005501657724380493,
+          0.0004071613948326558,
+          0.014985309913754463,
+          0.013914298266172409,
+          -0.003914502914994955,
+          -0.013569096103310585,
+          -0.0155517952516675,
+          0.014684364199638367,
+          -0.010710114613175392,
+          0.013338961638510227,
+          -0.011462477967143059,
+          -0.011657207272946835,
+          -0.006864210125058889,
+          0.01652544178068638,
+          0.024465089663863182,
+          0.012604300864040852,
+          0.01707422360777855,
+          0.0020557225216180086,
+          0.007820154540240765,
+          0.00505853770300746,
+          -0.0038414793089032173,
+          0.014507337473332882,
+          -0.034095339477062225,
+          0.013347812928259373,
+          -0.012453828006982803,
+          -0.0022637287620455027,
+          -0.004425667226314545,
+          -0.004049485549330711,
+          -0.02189820259809494,
+          -0.019862394779920578,
+          -0.0076608299277722836,
+          -0.018322262912988663,
+          0.00484168017283082,
+          0.02625305764377117,
+          -0.011630653403699398,
+          -0.021278608590364456,
+          -0.0042796204797923565,
+          -0.027102787047624588,
+          0.00949748232960701,
+          -0.02802332490682602,
+          0.041353434324264526,
+          0.008838057518005371,
+          0.00135757343377918,
+          0.006700460333377123,
+          -0.02189820259809494,
+          0.012170584872364998,
+          0.011073019355535507,
+          -0.019278205931186676,
+          -0.015135781839489937,
+          0.006656203418970108,
+          0.025084681808948517,
+          -0.0015257487539201975,
+          0.010913695208728313,
+          -0.0018875470850616693,
+          0.015534092672169209,
+          -0.03620195761322975,
+          -0.008191910572350025,
+          -0.00039858664968051016,
+          -0.02368617057800293,
+          -0.00914342887699604,
+          0.025952113792300224,
+          -0.018853342160582542,
+          0.015542943961918354,
+          -0.005470125004649162,
+          -0.006408366374671459,
+          -0.01365760900080204,
+          0.01701226457953453,
+          -2.6744170099846087e-05,
+          0.023066578432917595,
+          0.012453828006982803,
+          0.015224295668303967,
+          -0.015711119398474693,
+          -0.006961574777960777,
+          -0.0033922740258276463,
+          -0.0043327282182872295,
+          -0.015056120231747627,
+          -0.01130315475165844,
+          0.0134894335642457,
+          0.008461875841021538,
+          -0.008616774342954159,
+          0.015286254696547985,
+          0.0001645794982323423,
+          -0.0031046057119965553,
+          -0.023031173273921013,
+          0.0005709110992029309,
+          -0.004908064845949411,
+          -0.007567890919744968,
+          -0.013648757711052895,
+          -0.017808884382247925,
+          0.025598060339689255,
+          0.01862320862710476,
+          -0.04361937567591667,
+          0.01084288489073515,
+          -0.016065172851085663,
+          0.025226304307579994,
+          -0.010232143104076385,
+          0.010019711218774319,
+          -0.0024296913761645555,
+          -0.0216857697814703,
+          0.0028656194917857647,
+          -0.006514582317322493,
+          -0.019437531009316444,
+          -0.021827390417456627,
+          0.014790580607950687,
+          -0.007041236851364374,
+          -0.0037950098048895597,
+          -0.0030692003201693296,
+          -0.030537104234099388,
+          -0.009639102965593338,
+          0.01954374648630619,
+          0.004801848903298378,
+          0.007669681683182716,
+          0.016985710710287094,
+          -0.011506735347211361,
+          0.028129542246460915,
+          -0.011232343502342701,
+          -0.011250046081840992,
+          0.0025204175617545843,
+          -0.0056338743306696415,
+          0.016569698229432106,
+          -0.0116749107837677,
+          -0.022588606923818588,
+          -0.002199556678533554,
+          0.009754170663654804,
+          0.021738877519965172,
+          -0.005859583616256714,
+          0.01357794739305973,
+          0.024571305140852928,
+          0.013462879694998264,
+          -0.0015489835059270263,
+          0.007266945671290159,
+          0.06128663942217827,
+          0.019950907677412033,
+          0.013754974119365215,
+          0.0019052497809752822,
+          0.025704275816679,
+          -0.014277203008532524,
+          0.000535782368388027,
+          -0.013542542234063148,
+          -0.0025580357760190964,
+          -0.018083276227116585,
+          -0.011931599117815495,
+          -0.005682556889951229,
+          0.010364912450313568,
+          -0.006757994182407856,
+          0.011276599951088428,
+          -0.010471128858625889,
+          -0.005386037286370993,
+          -0.01610942929983139,
+          0.00011700358300004154,
+          0.008253869600594044,
+          0.015321659855544567,
+          -0.019649963825941086,
+          -0.026111437007784843,
+          0.002363306237384677,
+          -0.0060941437259316444,
+          0.011550991795957088,
+          0.0044632856734097,
+          0.0021652576979249716,
+          0.0145692965015769,
+          -0.022323066368699074,
+          0.032997775822877884,
+          -0.007156304083764553,
+          -0.018888747319579124,
+          0.017136184498667717,
+          -0.005403739865869284,
+          0.020552799105644226,
+          0.0182514525949955,
+          0.007846708409488201,
+          0.00985153578221798,
+          0.003244014224037528,
+          -0.0030669874977320433,
+          -0.02007482759654522,
+          0.012896394357085228,
+          -0.002684167120605707,
+          0.007948498241603374,
+          -0.008568092249333858,
+          -0.014082473702728748,
+          -0.004554011858999729,
+          -0.004970024339854717,
+          0.009134577587246895,
+          -0.0023987116292119026,
+          0.014374567195773125,
+          0.0036644525825977325,
+          -0.0019738476257771254,
+          -0.01076322328299284,
+          -0.0002287516836076975,
+          0.028111839666962624,
+          -0.0011849724687635899,
+          0.005496678873896599,
+          0.008895590901374817,
+          -0.011798828840255737,
+          -0.010435723699629307,
+          -0.004312812816351652,
+          -0.00028628535801544785,
+          0.003912290092557669,
+          0.015268552117049694,
+          0.0030138795264065266,
+          0.0018443968147039413,
+          -0.015339363366365433,
+          0.004952321760356426,
+          0.0015146846417337656,
+          0.009400117211043835,
+          -0.005005429964512587,
+          -0.0034498076420277357,
+          -0.018729424104094505,
+          -0.011258897371590137,
+          -0.01148903276771307,
+          -0.0014848114224150777,
+          -0.006244616582989693,
+          -0.022659417241811752,
+          0.011090721935033798,
+          0.018198344856500626,
+          -0.01546328142285347,
+          -0.0013863402418792248,
+          -0.0368923619389534,
+          -0.0007130856392905116,
+          0.008886739611625671,
+          0.013338961638510227,
+          0.00309354136697948,
+          0.03234277665615082,
+          0.003520618425682187,
+          -0.0068199532106518745,
+          0.014764025807380676,
+          -0.020481988787651062,
+          -0.013011462055146694,
+          -0.0201102327555418,
+          0.007682958617806435,
+          0.004135786090046167,
+          -0.00655883876606822,
+          -0.018216047435998917,
+          0.011373965069651604,
+          -0.009169982746243477,
+          -0.010196737945079803,
+          -0.0026310591492801905,
+          -0.013675311580300331,
+          0.00013850955292582512,
+          0.013693015091121197,
+          -0.012170584872364998,
+          -0.0195260439068079,
+          -0.016675913706421852,
+          -0.0061339749954640865,
+          -0.02189820259809494,
+          -0.02281874045729637,
+          -0.003064774675294757,
+          -0.0025381201412528753,
+          0.015374768525362015,
+          -0.012781327590346336,
+          0.004872659686952829,
+          -0.01665821112692356,
+          0.003991951700299978,
+          -0.004268555901944637,
+          -0.038627225905656815,
+          0.008758395910263062,
+          0.006063164211809635,
+          -0.00715187843888998,
+          -0.0037507531233131886,
+          0.019419828429818153,
+          -0.00709877023473382,
+          -0.008501706644892693,
+          -0.00043675804045051336,
+          0.004102593753486872,
+          0.014277203008532524,
+          0.0008995168609544635,
+          -0.013046867214143276,
+          -0.00026069945306517184,
+          -0.003642324125394225,
+          -0.008846908807754517,
+          -0.006271170452237129,
+          -0.0040738265961408615,
+          0.0008314722217619419,
+          -0.021225500851869583,
+          0.010825182311236858,
+          -0.0035737263970077038,
+          0.0032949093729257584,
+          0.03556466102600098,
+          -0.005332929082214832,
+          -0.026642518118023872,
+          -0.009240793064236641,
+          0.01664050854742527,
+          0.005133774131536484,
+          -0.012878691777586937,
+          -0.012471530586481094,
+          -0.0046425252221524715,
+          -0.014719769358634949,
+          -0.007629850413650274,
+          -0.01229450386017561,
+          0.0018001401331275702,
+          0.020889149978756905,
+          -0.025297114625573158,
+          -0.015613754279911518,
+          -0.007439546752721071,
+          -0.015923550352454185,
+          0.013781527988612652,
+          0.002002614550292492,
+          0.0018510353984311223,
+          -0.0074351211078464985,
+          0.02639468014240265,
+          0.004441156983375549,
+          -0.016233347356319427,
+          -0.01548098400235176,
+          0.0036047061439603567,
+          0.01311767753213644,
+          -0.0021641512867063284,
+          -0.026359274983406067,
+          0.030076835304498672,
+          0.0006124016945250332,
+          0.010648155584931374,
+          0.004379197955131531,
+          0.023208199068903923,
+          0.014047068543732166,
+          0.0007872155983932316,
+          0.0034675104543566704,
+          0.012613152153789997,
+          -1.3069548913335893e-05,
+          0.018216047435998917,
+          -0.0009874770184978843,
+          -0.03333412483334541,
+          0.012843286618590355,
+          0.00543029373511672,
+          -0.006058738566935062,
+          0.0047398898750543594,
+          0.016313008964061737,
+          -0.006757994182407856,
+          -0.011205789633095264,
+          0.016870643943548203,
+          0.0041601271368563175,
+          0.016215644776821136,
+          0.007085493300110102,
+          0.013710717670619488,
+          -0.01581733487546444,
+          -0.0057887728326022625,
+          -0.008922145701944828,
+          0.0016197942895814776,
+          -0.00228585721924901,
+          0.0025890152901411057,
+          0.024111036211252213,
+          0.003662239760160446,
+          0.01988009735941887,
+          0.002788170473650098,
+          0.0021210010163486004,
+          -0.02154414914548397,
+          0.006580967456102371,
+          0.013383218087255955,
+          0.00013837125152349472,
+          -0.028147244825959206,
+          0.0018001401331275702,
+          -0.017764627933502197,
+          0.014038216322660446,
+          0.017419425770640373,
+          0.00392999267205596,
+          0.002414201619103551,
+          0.007461674977093935,
+          0.004450008273124695,
+          -0.01579963229596615,
+          0.024465089663863182,
+          -0.028819944709539413,
+          0.010072818957269192,
+          -0.006563264410942793,
+          -0.023031173273921013,
+          -0.023916305974125862,
+          0.013834635727107525,
+          0.014648959040641785,
+          -0.00046690789167769253,
+          0.04252181202173233,
+          0.005217861849814653,
+          -0.00339006120339036,
+          0.014887944795191288,
+          0.013161934912204742,
+          0.013701866380870342,
+          -0.03781289979815483,
+          -0.015135781839489937,
+          0.007474951911717653,
+          0.010382615961134434,
+          -0.014994161203503609,
+          -0.0038857359904795885,
+          -0.00031975447200238705,
+          0.003423253772780299,
+          -0.00664292648434639,
+          -0.0017083075363188982,
+          -2.6139097826671787e-05,
+          0.004219873808324337,
+          -0.018924154341220856,
+          -0.009886940941214561,
+          -0.008501706644892693,
+          0.01202896423637867,
+          -0.0003352442872710526,
+          0.026642518118023872,
+          -0.005266543943434954,
+          0.003834840841591358,
+          -0.00877167284488678,
+          0.008178633637726307,
+          -0.014852539636194706,
+          0.0028523425571620464,
+          0.0024562454782426357,
+          0.009479779750108719,
+          0.02949264645576477,
+          -0.0012790178880095482,
+          0.010356061160564423,
+          -0.006164954509586096,
+          0.0178531426936388,
+          0.01709192804992199,
+          -0.007842282764613628,
+          -0.007952923886477947,
+          0.011922747828066349,
+          0.02715589478611946,
+          0.011719167232513428,
+          -0.01519774179905653,
+          0.009506333619356155,
+          0.036467500030994415,
+          -0.01266625989228487,
+          -0.028696026653051376,
+          0.008121099323034286,
+          0.008103396743535995,
+          0.012763625010848045,
+          -0.01579963229596615,
+          -0.0030913285445421934,
+          -0.0029209405183792114,
+          -0.012418422847986221,
+          0.03365277498960495,
+          -0.004669079091399908,
+          -0.012719367630779743,
+          0.002144235884770751,
+          0.007567890919744968,
+          -0.010957952588796616,
+          -0.00732005387544632,
+          -0.03365277498960495,
+          0.00931160431355238,
+          -0.024677520617842674,
+          -0.007651978638023138,
+          -0.02517319656908512,
+          -0.024535899981856346,
+          0.00801488384604454,
+          0.007842282764613628,
+          -0.02476603351533413,
+          0.002686380175873637,
+          -0.015764227136969566,
+          0.011179235763847828,
+          -0.015126930549740791,
+          0.0016795407282188535,
+          -0.0201102327555418,
+          0.014179837889969349,
+          0.007412992883473635,
+          0.0019318037666380405,
+          -0.006142826285213232,
+          -0.013321259059011936,
+          0.0005742303328588605,
+          -0.02481914311647415,
+          -0.016410375013947487,
+          -0.022482389584183693,
+          0.0047752950340509415,
+          -0.02225225605070591,
+          0.005394888576120138,
+          -0.0010643729474395514,
+          0.0004124168772250414,
+          -0.004903639201074839,
+          -0.03460871800780296,
+          -0.015286254696547985,
+          -0.02860751375555992,
+          -0.0018255878239870071,
+          -0.0033435916993767023,
+          0.030430888757109642,
+          -0.009754170663654804,
+          -0.007103195879608393,
+          0.00823174137622118,
+          0.018499288707971573,
+          -0.0009360286640003324,
+          -0.022588606923818588,
+          -0.0036024930886924267,
+          -0.00976302195340395,
+          0.014542742632329464,
+          0.0014471932081505656,
+          -0.02095996029675007,
+          0.0018598866881802678,
+          -0.008953125216066837,
+          -0.008298126049339771,
+          0.020021719858050346,
+          -0.014224094338715076,
+          0.0001842183992266655,
+          -0.0034121894277632236,
+          0.01683523878455162,
+          -0.0017957144882529974,
+          0.008395491167902946,
+          0.009798427112400532,
+          -0.00484168017283082,
+          -0.008067991584539413,
+          -0.011223492212593555,
+          0.029970619827508926,
+          0.016959156841039658,
+          -0.011365113779902458,
+          0.019030369818210602,
+          -0.005695833824574947,
+          0.010630453005433083,
+          -0.016135983169078827,
+          -0.012630854733288288,
+          -0.0019141010707244277,
+          0.0037618172354996204,
+          0.02264171466231346,
+          -0.012498084455728531,
+          0.00900623295456171,
+          -0.011019911617040634,
+          -0.010187885724008083,
+          0.006687183398753405,
+          -0.021119285374879837,
+          -0.0022947085089981556,
+          0.007266945671290159,
+          0.017322061583399773,
+          0.006793399341404438,
+          -0.0009188791736960411,
+          0.01436571590602398,
+          -0.03643209487199783,
+          0.023031173273921013,
+          0.009293901734054089,
+          0.01675557717680931,
+          -0.009647955186665058,
+          0.025084681808948517,
+          -0.00651015667244792,
+          -0.00836893729865551,
+          -0.0030758387874811888,
+          0.0278640016913414,
+          -0.028341973200440407,
+          -0.004257491789758205,
+          0.003027156461030245,
+          -0.01644578017294407,
+          0.015003012493252754,
+          -0.015056120231747627,
+          -0.019225098192691803,
+          -0.0015888145426288247,
+          0.027987919747829437,
+          0.020145637914538383,
+          -0.017047669738531113,
+          0.013905446976423264,
+          -0.0007213837816379964,
+          0.01865861378610134,
+          -0.0014914498897269368,
+          0.002449606778100133,
+          -0.0235445499420166,
+          -0.011126128025352955,
+          0.003907864447683096,
+          0.01022329181432724,
+          0.00497444998472929,
+          0.02368617057800293,
+          0.00957714393734932,
+          -0.010984506458044052,
+          0.003137798048555851,
+          0.01175457239151001,
+          0.0008934315992519259,
+          -0.012737070210278034,
+          -0.009462077170610428,
+          0.014808283187448978,
+          -0.01464010775089264,
+          -0.007041236851364374,
+          0.0065942443907260895,
+          0.00832910556346178,
+          0.03522831201553345,
+          0.002270367229357362,
+          -0.008138801902532578,
+          0.0001835268922150135,
+          0.037919119000434875,
+          0.0017968208994716406,
+          -0.023013470694422722,
+          -0.0005200158921070397,
+          -0.0014294905122369528,
+          0.00578434718772769,
+          -0.00012903586321044713,
+          0.001651880331337452,
+          0.015985511243343353,
+          -0.002372157759964466,
+          -0.0036091317888349295,
+          0.0005570809007622302,
+          0.003617983078584075,
+          0.0023854346945881844,
+          0.01575537584722042,
+          -0.008568092249333858,
+          0.006603095680475235,
+          -0.008191910572350025,
+          -0.009515184909105301,
+          -0.0005355057655833662,
+          0.0034011253155767918,
+          0.018534693866968155,
+          -0.01697685942053795,
+          0.007532485760748386,
+          0.0133920693770051,
+          0.012506935745477676,
+          -0.014489634893834591,
+          0.017083076760172844,
+          0.002892173593863845,
+          -8.899740350898355e-05,
+          0.020765231922268867,
+          0.006080866791307926,
+          -0.006585393100976944,
+          -0.033387232571840286,
+          0.01202011201530695,
+          0.012958354316651821,
+          0.02621765248477459,
+          -0.004536308813840151,
+          -0.001487024244852364,
+          0.0035604494623839855,
+          0.008501706644892693,
+          0.015392471104860306,
+          -0.006226914003491402,
+          -0.00489036226645112,
+          -0.008616774342954159,
+          -0.0019937630277127028,
+          0.001215952099300921,
+          0.009116875007748604,
+          0.013666460290551186,
+          -0.02081833966076374,
+          0.015666862949728966,
+          0.015985511243343353,
+          0.014710918068885803,
+          -0.020499691367149353,
+          0.011533289216458797,
+          -0.012887543067336082,
+          0.009382414631545544,
+          -0.020375771448016167,
+          -0.005213436204940081,
+          0.009453224949538708,
+          -0.004823977593332529,
+          -0.03703398257493973,
+          -0.0020446584094315767,
+          -0.0028656194917857647,
+          0.007258094381541014,
+          -0.0029209405183792114,
+          0.012347611598670483,
+          0.00865217950195074,
+          -0.008599071763455868,
+          -0.003348017344251275,
+          -0.0036688782274723053,
+          -0.008413193747401237,
+          0.01601206511259079,
+          -0.011267748661339283,
+          0.004961173050105572,
+          0.02101306803524494,
+          0.004100380931049585,
+          -0.018747126683592796,
+          0.00718285795301199,
+          -0.02336752414703369,
+          0.0014117879327386618,
+          0.024482792243361473,
+          -0.006368535105139017,
+          -0.006886338349431753,
+          -0.002708508400246501,
+          -0.020694419741630554,
+          -0.014356864616274834,
+          -0.004118083510547876,
+          0.02389860339462757,
+          -0.004474349785596132,
+          -0.02207522839307785,
+          0.015498686581850052,
+          0.023934008553624153,
+          -0.0089265713468194,
+          -0.00028213628684170544,
+          -0.0072979251854121685,
+          -0.014232946559786797,
+          -0.0003125627408735454,
+          0.0011772274738177657,
+          0.019048072397708893,
+          -0.0001921016228152439,
+          -0.0001586325088283047,
+          0.007116473279893398,
+          -0.005571915302425623,
+          -0.002701869932934642,
+          0.029899807646870613,
+          0.00985153578221798,
+          -0.004540734458714724,
+          -0.01648118533194065,
+          0.0193313155323267,
+          -0.007550188340246677,
+          0.014861390925943851,
+          -0.001994869438931346,
+          0.012161733582615852,
+          -0.009347009472548962,
+          -0.007603296544402838,
+          -0.0014095749938860536,
+          -0.0017647348577156663,
+          -0.024358872324228287,
+          0.015436727553606033,
+          0.00502755818888545,
+          -0.007647552993148565,
+          -0.04468153789639473,
+          -0.004531883168965578,
+          -0.01050653401762247,
+          0.0026797414757311344,
+          -0.008594646118581295,
+          -0.005846306681632996,
+          -0.006164954509586096,
+          0.0017127332976087928,
+          -0.042273975908756256,
+          0.010807479731738567,
+          0.021791985258460045,
+          0.0003241801168769598,
+          -0.002584589645266533,
+          -0.0076254247687757015,
+          0.006178231444209814,
+          0.016490036621689796,
+          -0.0029120889957994223,
+          0.004370346665382385,
+          0.008293700404465199,
+          -0.011312006041407585,
+          -0.010541939176619053,
+          0.02315509133040905,
+          0.018924154341220856,
+          0.009745319373905659,
+          0.014250649139285088,
+          0.00701468251645565,
+          -0.009417819790542126,
+          0.01634841598570347,
+          -0.024004818871617317,
+          -0.0122148422524333,
+          -0.005948096979409456,
+          -0.017888547852635384,
+          -0.005248841363936663,
+          -0.018694018945097923,
+          -0.015843888744711876,
+          0.01086058747023344,
+          0.03078494220972061,
+          0.00012578576570376754,
+          -0.007576742675155401,
+          -0.008377788588404655,
+          0.009488631039857864,
+          0.014312608167529106,
+          0.013259299099445343,
+          0.009515184909105301,
+          0.010834033600986004,
+          -0.020535096526145935,
+          -0.009302753023803234,
+          0.0018443968147039413,
+          -0.02876683697104454,
+          0.0026310591492801905,
+          -0.014746323227882385,
+          -0.0201102327555418,
+          -0.02278333529829979,
+          0.00823174137622118,
+          -0.007771471980959177,
+          0.0027549779042601585,
+          -0.005492253229022026,
+          -0.012896394357085228,
+          0.009285050444304943,
+          0.014330310747027397,
+          -0.004443369805812836,
+          -0.015118079259991646,
+          0.01104646548628807,
+          0.006589818745851517,
+          0.004363708198070526,
+          0.0003681601956486702,
+          -0.005200159270316362,
+          -0.0008176420233212411,
+          -0.0031599265057593584,
+          0.013569096103310585,
+          0.017322061583399773,
+          0.010125926695764065,
+          0.01864091120660305,
+          0.0027217853348702192,
+          0.006842081900686026,
+          -0.002210620790719986,
+          -0.003874671645462513,
+          -0.0010411381954327226,
+          -0.02517319656908512,
+          -0.0019849117379635572,
+          -0.024181846529245377,
+          0.012630854733288288,
+          0.001221484155394137,
+          -0.01878253184258938,
+          0.003179841907694936,
+          -0.051620982587337494,
+          -0.011471329256892204,
+          -0.011090721935033798,
+          0.005673705600202084,
+          -0.0055984691716730595,
+          -0.02005712501704693,
+          -0.018587803468108177,
+          0.012462679296731949,
+          -0.006001204717904329,
+          -0.00492576789110899,
+          -0.00031062652124091983,
+          0.004120296332985163,
+          0.009108023717999458,
+          -0.014445378445088863,
+          0.0068199532106518745,
+          -0.007355459034442902,
+          -0.03092656284570694,
+          0.0038813103456050158,
+          0.006337555591017008,
+          -0.013728420250117779,
+          -0.03940614312887192,
+          0.005014281254261732,
+          0.013754974119365215,
+          0.010267548263072968,
+          -0.013383218087255955,
+          -0.009771873243153095,
+          0.011896193958818913,
+          -0.0032086088322103024,
+          0.03041318617761135,
+          -0.008758395910263062,
+          -0.016242198646068573,
+          -0.01464010775089264,
+          0.006505731027573347,
+          0.02152644656598568,
+          0.013551393523812294,
+          -0.03476804122328758,
+          0.0009957751026377082,
+          -0.010612750425934792,
+          -0.0005637193680740893,
+          0.01220599003136158,
+          -0.0024783737026154995,
+          0.002996176714077592,
+          -0.001240293262526393,
+          0.021136987954378128,
+          0.010966803878545761,
+          0.01691490039229393,
+          0.02692575938999653,
+          0.00610742112621665,
+          0.0058241779915988445,
+          -0.01901266723871231,
+          -8.051775694184471e-06,
+          -0.019968610256910324,
+          0.009904643520712852,
+          -0.022836443036794662,
+          0.014834837056696415,
+          -0.004219873808324337,
+          -0.007461674977093935,
+          -2.411642981314799e-06,
+          0.006505731027573347,
+          -0.008612348698079586,
+          -0.016162537038326263,
+          -0.006085292436182499,
+          0.03404223173856735,
+          -0.015826186165213585,
+          0.01683523878455162,
+          -0.010984506458044052,
+          0.01465781033039093,
+          0.015330511145293713,
+          -0.0035958546213805676,
+          -0.018888747319579124,
+          -0.014055919833481312,
+          -0.014215243048965931,
+          -0.010143629275262356,
+          0.0222699586302042,
+          0.019101180136203766,
+          0.009931197389960289,
+          0.0010704583255574107,
+          0.005509955808520317,
+          -0.006036610342562199,
+          0.014666661620140076,
+          0.007138601504266262,
+          0.002217259258031845,
+          -0.01084288489073515,
+          -0.005545360967516899,
+          -0.009860387071967125,
+          0.011763423681259155,
+          0.005266543943434954,
+          -0.004100380931049585,
+          0.008377788588404655,
+          0.005642725620418787,
+          0.0016806471394374967,
+          -0.0015611541457474232,
+          -0.005262118298560381,
+          0.0008071310585364699,
+          0.0003344144788570702,
+          0.0026399106718599796,
+          -0.006961574777960777,
+          0.016197942197322845,
+          -0.012055518105626106,
+          -0.013454028405249119,
+          -0.024341169744729996,
+          -0.006430494599044323,
+          -0.005377185996621847,
+          0.001866525155492127,
+          -0.03643209487199783,
+          0.012807881459593773,
+          0.0003496277204249054,
+          0.0014671087265014648,
+          0.0013785953633487225,
+          -0.0025093534495681524,
+          0.02476603351533413,
+          -0.016729023307561874,
+          -0.010834033600986004,
+          -0.010497682727873325,
+          -0.0018952920800074935,
+          0.002427478553727269,
+          0.005872860550880432,
+          -0.002425265731289983,
+          0.014339162036776543,
+          -0.004372559487819672,
+          -0.009550590068101883,
+          0.006948297843337059,
+          -0.0010084989480674267,
+          -0.020375771448016167,
+          0.008621199987828732,
+          0.014578147791326046,
+          0.018003614619374275,
+          0.026536300778388977,
+          0.00432608975097537,
+          -0.009015084244310856,
+          -0.029262512922286987,
+          -0.004295110236853361,
+          0.03855641186237335,
+          -0.02552725002169609,
+          0.006740291137248278,
+          -0.006952723488211632,
+          0.026500895619392395,
+          0.015950104221701622,
+          0.005235564429312944,
+          0.012790178880095482,
+          -0.008275997824966908,
+          0.0050010038539767265,
+          -0.0009587102103978395,
+          0.008581369183957577,
+          -0.014073622412979603,
+          0.014480783604085445,
+          0.0013099975185468793,
+          -0.005076240282505751,
+          -0.004104806575924158,
+          -0.021986715495586395,
+          -0.024642115458846092,
+          0.019632259383797646,
+          -0.004611545242369175,
+          -0.012489233165979385,
+          0.013524839654564857,
+          -0.011444775387644768,
+          -0.01757875084877014,
+          0.03351115435361862,
+          -0.01258659828454256,
+          -0.015534092672169209,
+          0.0012945076450705528,
+          0.013808081857860088,
+          0.015056120231747627,
+          0.03547614812850952,
+          -0.008050289005041122,
+          0.02299576811492443,
+          0.0035781520418822765,
+          -0.010834033600986004,
+          -0.005563064012676477,
+          -0.02101306803524494,
+          -0.013454028405249119,
+          0.008793801069259644,
+          -0.017375169321894646,
+          0.00932930689305067,
+          0.017631858587265015,
+          0.002112149726599455,
+          0.0012934012338519096,
+          0.02115469053387642,
+          -0.02427035942673683,
+          0.015534092672169209,
+          -0.001430596923455596,
+          0.006129549350589514,
+          0.0012579959584400058,
+          -0.01807442493736744,
+          -0.02134941890835762,
+          -0.01428605429828167,
+          0.003914502914994955,
+          -0.030129943042993546,
+          0.005518807098269463,
+          -0.005386037286370993,
+          -0.020765231922268867,
+          -0.01589699648320675,
+          -0.008864611387252808,
+          -0.0072271148674190044,
+          0.009913494810461998,
+          -0.0016740086721256375,
+          -0.01772037148475647,
+          -0.01915428787469864,
+          0.0033878483809530735,
+          -0.00610299501568079,
+          -0.007253668736666441,
+          0.014914498664438725,
+          -0.010603899136185646,
+          0.011081870645284653,
+          0.01039146725088358,
+          -0.00519573362544179,
+          0.019278205931186676,
+          -0.00878937542438507,
+          0.03795452415943146,
+          -0.002117681782692671,
+          -0.007992755621671677,
+          -0.004801848903298378,
+          -0.0026686773635447025,
+          0.010603899136185646,
+          0.003272780915722251,
+          0.0038813103456050158,
+          0.007377587258815765,
+          0.013781527988612652,
+          -0.011099573224782944,
+          -0.0012248033890500665,
+          0.003967610653489828,
+          0.033369529992341995,
+          -0.014153284020721912,
+          -0.0128521379083395,
+          0.0032949093729257584,
+          -0.010161331854760647,
+          0.0002121554280165583,
+          -0.00024368830781895667,
+          -0.004615970887243748,
+          0.031900208443403244,
+          0.003887948812916875,
+          0.004503116477280855,
+          0.012515787035226822,
+          0.007085493300110102,
+          -0.026270762085914612,
+          0.0007169580785557628,
+          0.0035361081827431917,
+          0.002909876173362136,
+          0.018039019778370857,
+          -0.019844692200422287,
+          -0.019667666405439377,
+          0.008382214233279228,
+          -0.0033325275871902704,
+          -0.004204384051263332,
+          0.0020933407358825207,
+          -0.013418623246252537,
+          -0.00885133445262909,
+          -0.01897726207971573,
+          -0.0013011462287977338,
+          0.004664653446525335,
+          -0.005023132544010878,
+          0.019950907677412033,
+          -0.01793280430138111,
+          -0.04057451710104942,
+          -0.0018654187442734838,
+          0.01121464092284441,
+          0.016233347356319427,
+          0.015622605569660664,
+          0.020181043073534966,
+          -0.002648761961609125,
+          -0.013586798682808876,
+          0.015516390092670918,
+          0.0015854953089728951,
+          0.005341780371963978,
+          -0.0018997177248820662,
+          0.015826186165213585,
+          0.009940048679709435,
+          -0.0034785745665431023,
+          -0.004443369805812836,
+          0.03459101542830467,
+          -0.0046026939526200294,
+          -0.014374567195773125,
+          0.023066578432917595,
+          -0.012958354316651821,
+          0.004157914314419031,
+          0.005355057306587696,
+          -0.0008641115273348987,
+          0.012223693542182446,
+          0.021986715495586395,
+          0.018127532675862312,
+          -0.021809687837958336,
+          -0.00524441571906209,
+          -0.006430494599044323,
+          -0.023084281012415886,
+          -0.010789777152240276,
+          0.010028562508523464,
+          0.004916916601359844,
+          0.024854548275470734,
+          0.0007888752152211964,
+          0.005819752346724272,
+          -0.001608730060979724,
+          -0.008970827795565128,
+          0.0059259687550365925,
+          -0.0035737263970077038,
+          0.006806676276028156,
+          -0.009904643520712852,
+          0.019313612952828407,
+          -0.0012413996737450361,
+          -0.024164143949747086,
+          0.0014648959040641785,
+          0.004388049244880676,
+          -0.008992956019937992,
+          -0.0043504307977855206,
+          0.006926169153302908,
+          -0.0024960762821137905,
+          0.01303801592439413,
+          0.015312808565795422,
+          0.003624621545895934,
+          0.012604300864040852,
+          0.011630653403699398,
+          0.005492253229022026,
+          -0.006846507545560598,
+          0.0014018301153555512,
+          -0.0023478164803236723,
+          -0.004118083510547876,
+          0.01772037148475647,
+          -0.010267548263072968,
+          0.018552398309111595,
+          -0.011444775387644768,
+          0.007253668736666441,
+          0.01954374648630619,
+          -0.0020236363634467125,
+          -0.0012369740288704634,
+          -0.0024540324229747057,
+          0.017224697396159172,
+          0.01492334995418787,
+          -0.002376583404839039,
+          -0.0004226512392051518,
+          0.007868836633861065,
+          0.007426269818097353,
+          0.018959559500217438,
+          -0.0044876267202198505,
+          0.0015866017201915383,
+          -0.002130958717316389,
+          -0.0025934409350156784,
+          -0.021862797439098358,
+          0.0014814921887591481,
+          0.005660428665578365,
+          0.005191307980567217,
+          0.011170384474098682,
+          0.007076642010360956,
+          0.018747126683592796,
+          0.0018764829728752375,
+          -0.021437931805849075,
+          0.011648355983197689,
+          -0.01917199045419693,
+          -0.018083276227116585,
+          0.011993558146059513,
+          -0.023456037044525146,
+          -0.0032086088322103024,
+          -0.02586359903216362,
+          -0.0054834019392728806,
+          -0.00010566280980128795,
+          0.007138601504266262,
+          0.0133920693770051,
+          0.0029143018182367086,
+          0.007837857119739056,
+          -0.015534092672169209,
+          -0.010550790466368198,
+          0.009878089651465416,
+          0.008568092249333858,
+          -0.011223492212593555,
+          0.013967406004667282,
+          -0.007904241792857647,
+          -0.011869640089571476,
+          0.002974048489704728,
+          -0.010373763740062714,
+          -0.0032152472995221615,
+          -0.00157332478556782,
+          0.0063242786563932896,
+          0.0023367523681372404,
+          0.007802451495081186,
+          0.017808884382247925,
+          0.008780524134635925,
+          -0.001572218257933855,
+          -0.008324679918587208,
+          -0.025279412046074867,
+          -0.0014648959040641785,
+          -0.00465580215677619,
+          0.007010256871581078,
+          0.031085887923836708,
+          -0.001259102369658649,
+          0.012250247411429882,
+          0.002670890185981989,
+          0.007497080601751804,
+          -0.0001987401192309335,
+          -0.0004624822468031198,
+          -0.016082875430583954,
+          -0.006465899758040905,
+          -0.0037241990212351084,
+          -0.02097766287624836,
+          0.019756179302930832,
+          0.010559642687439919,
+          0.008050289005041122,
+          0.00912572629749775,
+          -0.015374768525362015,
+          -0.0030869028996676207,
+          0.01880023442208767,
+          -0.0028412784449756145,
+          3.170575655531138e-05,
+          0.017056521028280258,
+          -0.013454028405249119,
+          0.005864009261131287,
+          -0.0031333724036812782,
+          -0.015401322394609451,
+          0.006364109460264444,
+          -0.0036378984805196524,
+          -0.00865217950195074,
+          -0.002591228112578392,
+          0.024730628356337547,
+          0.02244698442518711,
+          0.012082071974873543,
+          -0.016605103388428688,
+          -0.011258897371590137,
+          -0.002659826073795557,
+          -0.001719371764920652,
+          0.011497884057462215,
+          -0.015516390092670918,
+          -0.024288062006235123,
+          0.008470727130770683,
+          -0.008581369183957577,
+          -0.019225098192691803,
+          -0.00020731484983116388,
+          0.02207522839307785,
+          -0.016162537038326263,
+          -0.004180043004453182,
+          0.013223893940448761,
+          -1.0778920113807544e-05,
+          -0.010975655168294907,
+          0.014055919833481312,
+          0.00489036226645112,
+          0.013480582274496555,
+          -0.011453626677393913,
+          -0.0065765418112277985,
+          0.007204986177384853,
+          -0.020588204264640808,
+          -0.00844859890639782,
+          -0.003230737056583166,
+          -0.018587803468108177,
+          0.015684565529227257,
+          -0.010462277568876743,
+          0.00914342887699604,
+          0.004204384051263332,
+          0.0035338953603059053,
+          0.010754371993243694,
+          0.0016286455793306231,
+          0.007603296544402838,
+          -0.004700058605521917,
+          -0.0269788671284914,
+          -0.030094537883996964,
+          0.0011893981136381626,
+          0.0024186272639781237,
+          0.013728420250117779,
+          -0.0026310591492801905,
+          -0.016056321561336517,
+          -0.00966565776616335,
+          0.017605304718017578,
+          -0.030342375859618187,
+          0.016790982335805893,
+          0.01842847838997841,
+          0.0024739480577409267,
+          0.001541238627396524,
+          0.006302150432020426,
+          0.0009874770184978843,
+          -0.016135983169078827,
+          -0.00489036226645112,
+          -0.0011407157871872187,
+          0.02170347236096859,
+          -0.005062963347882032,
+          0.015427876263856888,
+          -0.021774282678961754,
+          0.005518807098269463,
+          -0.012011260725557804,
+          0.0017912888433784246,
+          0.0004840573528781533,
+          0.005992353428155184,
+          0.003629047190770507,
+          -0.008992956019937992,
+          0.010090521536767483,
+          -0.005031983833760023,
+          0.00042514066444709897,
+          0.004275194834917784,
+          0.05392232909798622,
+          0.00392999267205596,
+          0.03795452415943146,
+          -0.015312808565795422,
+          -0.00885133445262909,
+          0.025421032682061195,
+          0.011480181477963924,
+          -0.007443972397595644,
+          0.01917199045419693,
+          0.010179034434258938,
+          0.018481586128473282,
+          0.003328101709485054,
+          -0.0008696436416357756,
+          0.02223455347120762,
+          0.018198344856500626,
+          0.006368535105139017,
+          0.008059140294790268,
+          0.008391065523028374,
+          -0.004016293212771416,
+          0.019118882715702057,
+          -0.0005769963609054685,
+          -0.0195260439068079,
+          0.007231540512293577,
+          -0.01277247630059719,
+          0.015516390092670918,
+          0.0006262319511733949,
+          0.0021331715397536755,
+          0.010533087886869907,
+          0.0034365307074040174,
+          0.0023190497886389494,
+          0.0027726804837584496,
+          0.0007540230872109532,
+          -0.0025890152901411057,
+          0.012506935745477676,
+          0.010913695208728313,
+          0.0031909062527120113,
+          -0.007005831226706505,
+          0.010913695208728313,
+          0.013445177115499973,
+          0.0033258888870477676,
+          -0.012524638324975967,
+          -0.011612950824201107,
+          -0.008700861595571041,
+          -0.015790781006217003,
+          0.010187885724008083,
+          0.003611344611272216,
+          -0.009993156418204308,
+          -0.025969816371798515,
+          0.00849728100001812,
+          0.00986923836171627,
+          -0.017746925354003906,
+          0.0030625618528574705,
+          0.00826272089034319,
+          0.000512824219185859,
+          0.006032184697687626,
+          0.020358068868517876,
+          0.0006129549001343548,
+          0.012179436162114143,
+          0.01130315475165844,
+          -0.0029452815651893616,
+          -0.004872659686952829,
+          0.015604902990162373,
+          0.01411787886172533,
+          -0.008992956019937992,
+          -0.016020916402339935,
+          -0.012347611598670483,
+          -0.035175204277038574,
+          -0.006381812505424023,
+          -0.00357593921944499,
+          -0.01237416546791792,
+          -0.002407562918961048,
+          -0.00023013469763100147,
+          0.012701665051281452,
+          -0.024730628356337547,
+          0.010311804711818695,
+          -0.021083878353238106,
+          -0.017454832792282104,
+          -0.00013359983859118074,
+          0.00674471678212285,
+          -0.01697685942053795,
+          0.00044173692003823817,
+          0.006850933190435171,
+          0.0009907962521538138,
+          0.0027992345858365297,
+          0.048399098217487335,
+          -0.02641238272190094,
+          -0.0008840270456857979,
+          -0.009293901734054089,
+          0.017587602138519287,
+          0.019295910373330116,
+          0.012356462888419628,
+          -0.004372559487819672,
+          0.008864611387252808,
+          -0.006164954509586096,
+          0.003783945459872484,
+          0.024465089663863182,
+          0.004157914314419031,
+          0.025279412046074867,
+          0.005213436204940081,
+          0.0017149461200460792,
+          -0.01475517451763153,
+          -0.005209010560065508,
+          -0.014445378445088863,
+          -0.0013365515042096376,
+          -0.021614959463477135,
+          -0.03675074130296707,
+          0.02062360942363739,
+          -0.017534494400024414,
+          0.015003012493252754,
+          -0.004934619180858135,
+          0.0008032586192712188,
+          0.005076240282505751,
+          0.022323066368699074,
+          0.015604902990162373,
+          -0.022199146449565887,
+          0.030678726732730865,
+          0.004290684591978788,
+          0.03328101709485054,
+          0.0026376976165920496,
+          -0.0014604702591896057,
+          0.0017868631985038519,
+          0.020535096526145935,
+          0.0003781179548241198,
+          -0.011639504693448544,
+          -0.0020601481664925814,
+          0.012285652570426464,
+          -0.017649561166763306,
+          0.003852543421089649,
+          -0.03018305078148842,
+          -0.012630854733288288,
+          0.0027439137920737267,
+          -0.0032749937381595373,
+          0.00048710001283325255,
+          -0.01436571590602398,
+          0.03271453455090523,
+          -0.0046203965321183205,
+          0.011807680130004883,
+          0.023385226726531982,
+          -0.0036378984805196524,
+          -0.01148903276771307,
+          -0.016870643943548203,
+          0.01883563958108425,
+          -0.013799230568110943,
+          0.01842847838997841,
+          0.009771873243153095,
+          0.0011595248943194747,
+          -0.0014859178336337209,
+          -0.0021951310336589813,
+          0.00015379194519482553,
+          0.0016441354528069496,
+          0.007289073895663023,
+          -0.004509754944592714,
+          -0.020535096526145935,
+          0.021314013749361038,
+          -0.011391667649149895,
+          -0.00618708273395896,
+          0.021986715495586395,
+          -0.013268150389194489,
+          0.006341981235891581,
+          0.015020715072751045,
+          -0.02788170427083969,
+          -0.03540533781051636,
+          0.019791584461927414,
+          -0.003113457001745701,
+          0.008156504482030869,
+          0.005594043526798487,
+          0.00841761939227581,
+          0.008935422636568546,
+          0.00994889996945858,
+          -0.008913294412195683,
+          0.05919772759079933,
+          -0.021190095692873,
+          0.0032108216546475887,
+          0.0007175113423727453,
+          -0.017295507714152336,
+          0.014082473702728748,
+          0.00939126592129469,
+          0.010612750425934792,
+          -0.013091123662889004,
+          -0.008174207992851734,
+          -0.017083076760172844,
+          3.744529385585338e-05,
+          0.0015666862018406391,
+          -0.0008696436416357756,
+          -0.006541136186569929,
+          -0.003989738877862692,
+          -0.0394415482878685,
+          -0.02765156887471676,
+          0.03813154995441437,
+          -0.005279821343719959,
+          0.016684764996170998,
+          -0.006443771533668041,
+          -0.008718564175069332,
+          0.0014471932081505656,
+          0.010364912450313568,
+          0.004135786090046167,
+          -0.012338760308921337,
+          -0.005828603636473417,
+          0.007412992883473635,
+          -0.0561174601316452,
+          -0.0022438133601099253,
+          0.0057710702531039715,
+          -0.007209412287920713,
+          0.0033192504197359085,
+          0.018322262912988663,
+          0.02460671029984951,
+          -0.011196938343346119,
+          0.011453626677393913,
+          -0.004427880048751831,
+          0.011825382709503174,
+          0.02315509133040905,
+          0.0027815320063382387,
+          0.005116071552038193,
+          0.013604501262307167,
+          0.004691207315772772,
+          -0.008762821555137634,
+          -0.009754170663654804,
+          -0.018127532675862312,
+          -0.011312006041407585,
+          0.006032184697687626,
+          0.0003062008472625166,
+          0.004396900534629822,
+          0.014560445211827755,
+          0.017959358170628548,
+          -0.015808483585715294,
+          0.02133171632885933,
+          0.013631055131554604,
+          -0.005859583616256714,
+          -1.9491951661620988e-06,
+          -0.014967607334256172,
+          -0.011922747828066349,
+          -0.004135786090046167,
+          -0.010772074572741985,
+          0.01212632842361927,
+          -0.04163667932152748,
+          0.019083477556705475,
+          -0.0032285242341458797,
+          -0.010081670247018337,
+          -0.011851937510073185,
+          0.006846507545560598,
+          0.032112643122673035,
+          0.0030470718629658222,
+          -0.014445378445088863,
+          0.0012358676176518202,
+          0.004531883168965578,
+          -0.023827793076634407,
+          0.01585274003446102,
+          -0.008333531208336353,
+          0.001153992721810937,
+          -0.009050489403307438,
+          -0.015773078426718712,
+          0.04443369805812836,
+          -0.014808283187448978,
+          0.02203982323408127,
+          0.034626420587301254,
+          0.029404133558273315,
+          -0.0010887141106650233,
+          -0.012551192194223404,
+          -0.0139320008456707,
+          0.017083076760172844,
+          0.0032019703648984432,
+          0.012179436162114143,
+          0.017516791820526123,
+          0.017410574480891228,
+          -0.013799230568110943,
+          0.014073622412979603,
+          -0.006346406880766153,
+          -0.006727014202624559,
+          0.00046995055163279176,
+          -0.0020490840543061495,
+          0.0073997159488499165,
+          0.013400920666754246,
+          0.02062360942363739,
+          -0.01086058747023344,
+          0.0017249038210138679,
+          0.02264171466231346,
+          0.017047669738531113,
+          0.011922747828066349,
+          -0.008072417229413986,
+          3.713411570060998e-05,
+          0.005164753645658493,
+          0.010356061160564423,
+          0.008382214233279228,
+          0.004580565728247166,
+          -0.01011707540601492,
+          0.024535899981856346,
+          0.004270769190043211,
+          0.0008060246473178267,
+          0.022588606923818588,
+          -0.01076322328299284,
+          -0.00701468251645565,
+          -0.0016242199344560504,
+          0.007882113568484783,
+          -0.006797824986279011,
+          -0.03558236360549927,
+          0.02025185339152813,
+          -0.015339363366365433,
+          -0.01707422360777855,
+          0.01022329181432724,
+          -0.010984506458044052,
+          0.007992755621671677,
+          -0.0025757383555173874,
+          0.013666460290551186,
+          -0.009435522370040417,
+          0.0015622605569660664,
+          0.007005831226706505,
+          -0.0029497072100639343,
+          0.017596453428268433,
+          -0.014941052533686161,
+          0.009320455603301525,
+          -0.008815929293632507,
+          0.04609775170683861,
+          0.0140293650329113,
+          0.0026376976165920496,
+          -0.0030382205732166767,
+          -0.008112248033285141,
+          0.0006129549001343548,
+          0.003963185008615255,
+          2.6778745450428687e-05,
+          -0.008205187506973743,
+          -0.02621765248477459,
+          -0.006757994182407856,
+          -0.002617782214656472,
+          0.01672017201781273,
+          -0.008395491167902946,
+          0.019402125850319862,
+          0.013701866380870342,
+          -0.019136585295200348,
+          -0.02281874045729637,
+          0.022287661209702492,
+          -0.00809454545378685,
+          -0.007054513785988092,
+          -0.0067491428926587105,
+          -0.01810983009636402,
+          -0.00024811396724544466,
+          0.025704275816679,
+          0.003976461943238974,
+          0.0015102589968591928,
+          0.0023920731619000435,
+          0.004770869389176369,
+          0.025704275816679,
+          0.001831119880080223,
+          0.023119686171412468,
+          0.004016293212771416,
+          0.009453224949538708,
+          -0.012064369395375252,
+          0.0011827596463263035,
+          0.0014770664274692535,
+          -0.019649963825941086,
+          -0.011896193958818913,
+          0.0036578141152858734,
+          0.0065765418112277985,
+          0.006850933190435171,
+          0.017145035788416862,
+          0.016923751682043076,
+          -0.014047068543732166,
+          0.009134577587246895,
+          -0.008581369183957577,
+          -0.004507542122155428,
+          0.00966565776616335,
+          0.012453828006982803,
+          -0.00846630148589611,
+          0.0016253263456746936,
+          0.009709914214909077,
+          -0.007174006663262844,
+          0.018198344856500626,
+          0.01583503745496273,
+          0.0001586325088283047,
+          -0.002617782214656472,
+          0.03094426542520523,
+          0.007492654956877232,
+          -0.015241998247802258,
+          0.009187685325741768,
+          -0.025456437841057777
+        ],
+        "title": "9 Conclusion"
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "b8ffe9ad-9e6f-4478-8767-d2c5561f1266",
+      "properties": {
+        "page_content": "A Switch for Attention 27 ",
+        "keyphrases": [
+          "Switch",
+          "Attention"
+        ],
+        "embedding": [
+          -0.004847599659115076,
+          -0.013338678516447544,
+          -0.017561476677656174,
+          0.018164733424782753,
+          -0.036252863705158234,
+          0.0018600418698042631,
+          0.006209715269505978,
+          0.06599438190460205,
+          0.014593069441616535,
+          -0.005357495043426752,
+          -0.02489631250500679,
+          0.024474991485476494,
+          0.01628793403506279,
+          -0.022100266069173813,
+          -0.018451998010277748,
+          -0.02642839401960373,
+          -0.014775004237890244,
+          0.0039068059995770454,
+          -0.036712486296892166,
+          -0.008148754946887493,
+          0.020644789561629295,
+          -0.02066393941640854,
+          -0.034203704446554184,
+          0.016929492354393005,
+          -0.03311209753155708,
+          -0.00733004929497838,
+          0.03295888751745224,
+          0.011835324577987194,
+          -0.04140448570251465,
+          0.04393241927027702,
+          -0.03259501978754997,
+          0.010667112655937672,
+          -2.788491838146001e-05,
+          0.008067362941801548,
+          -0.04025542363524437,
+          -0.014545192010700703,
+          -0.0072869593277573586,
+          -0.023517441004514694,
+          0.001378872781060636,
+          0.02303866483271122,
+          -0.045158080756664276,
+          -0.006899151485413313,
+          -0.05542302504181862,
+          -0.02713698148727417,
+          0.010226639918982983,
+          -0.015732554718852043,
+          -0.02164064161479473,
+          -0.03544852137565613,
+          -0.0038780795875936747,
+          -0.018672235310077667,
+          0.024723954498767853,
+          -0.03631031513214111,
+          -0.0005529854097403586,
+          0.00697575556114316,
+          0.007416228763759136,
+          0.0014470982132479548,
+          -0.003313124878332019,
+          0.010542631149291992,
+          -0.011088434606790543,
+          0.037287015467882156,
+          0.003653055289760232,
+          -0.01817430928349495,
+          -0.0034160614013671875,
+          -0.024800557643175125,
+          -0.015694253146648407,
+          -0.011126737110316753,
+          -0.022119417786598206,
+          -0.0168050117790699,
+          -0.022866306826472282,
+          -0.0010132081806659698,
+          -0.042974866926670074,
+          0.012716270983219147,
+          -0.04653695598244667,
+          0.01056178193539381,
+          0.006458678282797337,
+          0.007713069207966328,
+          0.03776579350233078,
+          0.03698059916496277,
+          -0.05538472160696983,
+          -0.002126959152519703,
+          -0.010284092277288437,
+          -0.02424517832696438,
+          0.010657536797225475,
+          0.026409242302179337,
+          0.011777871288359165,
+          -0.02905208244919777,
+          -0.02650499902665615,
+          -0.09989167004823685,
+          0.05113319680094719,
+          0.0056447600945830345,
+          -0.02842009998857975,
+          -0.025164427235722542,
+          0.007488044910132885,
+          -0.006329408846795559,
+          0.01437283307313919,
+          -0.008871705271303654,
+          0.028688212856650352,
+          -0.014803730882704258,
+          -0.007205567788332701,
+          -0.0008031455217860639,
+          0.029109535738825798,
+          -0.0010479193879291415,
+          -0.025624051690101624,
+          -0.025164427235722542,
+          0.03203963860869408,
+          0.03052671067416668,
+          -0.02981812320649624,
+          0.024091970175504684,
+          0.0008827418787404895,
+          0.004270675126463175,
+          0.01378872711211443,
+          -0.01235240139067173,
+          -0.013941935263574123,
+          -0.0073731387965381145,
+          0.004337703809142113,
+          -0.002435769187286496,
+          0.002875045407563448,
+          -0.018384970724582672,
+          -0.014075992628932,
+          -0.009834043681621552,
+          -0.0008546138415113091,
+          0.030737372115254402,
+          0.004285038448870182,
+          -0.012371552176773548,
+          -0.020568184554576874,
+          -0.005850633606314659,
+          0.012591789476573467,
+          0.01174914464354515,
+          -0.014028115198016167,
+          0.010102157481014729,
+          0.01855732873082161,
+          0.019601058214902878,
+          -0.0066980659030377865,
+          0.0014459012309089303,
+          0.023517441004514694,
+          0.002047961112111807,
+          0.025815561413764954,
+          0.03431861102581024,
+          -0.03956598788499832,
+          -0.0008270842372439802,
+          -0.028305193409323692,
+          0.0037583857774734497,
+          0.015598498284816742,
+          0.02265564538538456,
+          -0.013032262213528156,
+          0.024015367031097412,
+          0.014822881668806076,
+          0.009977675974369049,
+          -0.009541991166770458,
+          0.029473405331373215,
+          0.018643507733941078,
+          -0.0008181072189472616,
+          -0.03347596526145935,
+          -0.001977341715246439,
+          0.054235659539699554,
+          0.021104414016008377,
+          -0.0035405431408435106,
+          -0.00020632220548577607,
+          -0.01043730042874813,
+          0.010274517349898815,
+          -0.01779128797352314,
+          -0.006635825149714947,
+          -0.021295923739671707,
+          -0.004670452792197466,
+          0.00989149697124958,
+          -0.005845845676958561,
+          0.033514268696308136,
+          0.00400495482608676,
+          -0.031580016016960144,
+          0.05132470652461052,
+          0.029224442318081856,
+          0.025183578953146935,
+          0.02834349498152733,
+          0.015943216159939766,
+          0.0025758109986782074,
+          0.00035818375181406736,
+          0.05224395543336868,
+          0.013530189171433449,
+          0.003772749099880457,
+          0.014803730882704258,
+          -0.035295311361551285,
+          0.006875212769955397,
+          -0.0050558666698634624,
+          -0.025834713131189346,
+          0.008967460133135319,
+          0.0135972173884511,
+          0.02169809490442276,
+          0.008057788014411926,
+          -0.0009449826902709901,
+          0.004304189700633287,
+          -0.025585750117897987,
+          -0.03711465746164322,
+          0.019725540652871132,
+          -0.02087460085749626,
+          -0.01820303499698639,
+          -0.01625920832157135,
+          -0.020625637844204903,
+          -0.0026284761261194944,
+          0.04366430267691612,
+          0.00017774531443137676,
+          -0.009977675974369049,
+          0.026332639157772064,
+          0.018672235310077667,
+          0.012074711732566357,
+          -0.05971284955739975,
+          0.003966652788221836,
+          -0.009355268441140652,
+          -0.020204314962029457,
+          -0.0038541408721357584,
+          0.00015784621064085513,
+          0.03276737779378891,
+          -0.004438246600329876,
+          0.01959148421883583,
+          0.01880629174411297,
+          0.005544217303395271,
+          0.006554433144629002,
+          0.008124816231429577,
+          0.015761282294988632,
+          0.006875212769955397,
+          -0.0067076412960886955,
+          0.039834100753068924,
+          0.02196620963513851,
+          -0.006295894738286734,
+          0.025241032242774963,
+          -0.06572626531124115,
+          0.040025610476732254,
+          -0.03397389128804207,
+          0.03052671067416668,
+          0.0010760474251583219,
+          -0.027749814093112946,
+          -0.07373139262199402,
+          0.0029277107678353786,
+          0.005917662288993597,
+          0.03077567368745804,
+          -0.008775950409471989,
+          -0.004912233911454678,
+          -0.009790954180061817,
+          0.01677628420293331,
+          0.01882544346153736,
+          -0.013999388553202152,
+          0.01795407198369503,
+          0.012045985087752342,
+          -0.015349534340202808,
+          0.0011514545185491443,
+          0.004505275283008814,
+          -0.009259513579308987,
+          -0.020127711817622185,
+          0.027481699362397194,
+          -0.02309611812233925,
+          -0.012620516121387482,
+          -0.0157134048640728,
+          0.0023316354490816593,
+          -0.019026529043912888,
+          0.012180042453110218,
+          -0.0004691997601184994,
+          0.05978945270180702,
+          -0.011270369403064251,
+          -0.02016601338982582,
+          0.0089483093470335,
+          0.014468587934970856,
+          0.009939374402165413,
+          0.02522188052535057,
+          -0.005553793162107468,
+          0.039834100753068924,
+          0.010312818922102451,
+          -0.027673209086060524,
+          0.03742107376456261,
+          -0.05982775613665581,
+          0.006616673897951841,
+          -0.019409548491239548,
+          0.002752957632765174,
+          -0.006286319345235825,
+          0.013549339957535267,
+          0.01759977824985981,
+          0.014391983859241009,
+          0.020912902429699898,
+          0.0032676411792635918,
+          6.702853715978563e-05,
+          -0.014190898276865482,
+          -0.018136005848646164,
+          -0.010466027073562145,
+          0.06400267779827118,
+          0.033629175275564194,
+          -0.005525066517293453,
+          -0.0273284912109375,
+          0.00811524037271738,
+          0.03194388374686241,
+          0.03274822607636452,
+          -0.028822271153330803,
+          -0.0005173765239305794,
+          0.004268281627446413,
+          -0.018193459138274193,
+          0.0030162842012941837,
+          -0.044315438717603683,
+          0.03487398847937584,
+          -0.0241877268999815,
+          0.037018902599811554,
+          0.04795413091778755,
+          0.005022352561354637,
+          -0.019917050376534462,
+          -0.03000963293015957,
+          -0.014746277593076229,
+          -0.005209074821323156,
+          -0.03272907808423042,
+          -0.0379764549434185,
+          0.0254133902490139,
+          0.01751359924674034,
+          0.021544886752963066,
+          -0.06415588408708572,
+          0.014707976020872593,
+          0.05676359310746193,
+          0.014075992628932,
+          -0.014794155023992062,
+          -0.017283786088228226,
+          -0.010169186629354954,
+          -0.0022598193027079105,
+          0.018566904589533806,
+          0.010485177859663963,
+          -0.014535617083311081,
+          -0.03090973012149334,
+          -0.017140153795480728,
+          -0.004510062746703625,
+          -0.025758108124136925,
+          0.03680824115872383,
+          -0.015071844682097435,
+          -0.010418149642646313,
+          0.008507836610078812,
+          0.037918999791145325,
+          -0.033629175275564194,
+          0.012409854680299759,
+          -0.055959250777959824,
+          -0.02093205414712429,
+          -0.008479109965264797,
+          -0.01735081523656845,
+          0.0037129020784050226,
+          -0.0025087823159992695,
+          0.014085568487644196,
+          0.02476225607097149,
+          -0.01907440647482872,
+          0.006564009003341198,
+          -0.011500181630253792,
+          0.015646375715732574,
+          -0.007665191777050495,
+          -0.020778845995664597,
+          -0.05235886201262474,
+          0.0008205011254176497,
+          0.03027774766087532,
+          -0.01257263869047165,
+          0.015234628692269325,
+          -0.0014913849299773574,
+          0.0011592345545068383,
+          -0.02137252688407898,
+          0.007320473901927471,
+          -0.006564009003341198,
+          0.036003898829221725,
+          -0.018605206161737442,
+          -0.023019514977931976,
+          0.011902352795004845,
+          -0.011605512350797653,
+          -0.01891162246465683,
+          0.025011219084262848,
+          -0.03263332322239876,
+          0.01043730042874813,
+          0.005817119497805834,
+          -0.06281531602144241,
+          -0.05163112282752991,
+          0.0039762286469340324,
+          -0.009786166250705719,
+          -0.009872346185147762,
+          -0.004897871054708958,
+          -0.08097047358751297,
+          -0.009671260602772236,
+          -0.04385581240057945,
+          0.02552829682826996,
+          0.0168050117790699,
+          0.017341239377856255,
+          -0.0038876549806445837,
+          0.005750090815126896,
+          -0.03761258348822594,
+          0.02330677956342697,
+          -0.014842033386230469,
+          0.017427420243620872,
+          -0.019189313054084778,
+          -0.029971331357955933,
+          0.04063844308257103,
+          0.02700292505323887,
+          0.02533678710460663,
+          0.006592735182493925,
+          -0.043626002967357635,
+          0.028017928823828697,
+          -0.021391678601503372,
+          -0.005075017921626568,
+          -0.0064491028897464275,
+          0.0048859016969799995,
+          0.021927906200289726,
+          -0.008129604160785675,
+          0.008268448524177074,
+          -0.0007175644277594984,
+          -0.001700849155895412,
+          -0.05339301750063896,
+          -0.013520613312721252,
+          0.02661990374326706,
+          -0.005697425454854965,
+          -0.030392654240131378,
+          0.010801170021295547,
+          0.010063855908811092,
+          -0.06534324586391449,
+          -0.010944802314043045,
+          0.0004428671090863645,
+          0.013156743720173836,
+          0.024398386478424072,
+          -0.06534324586391449,
+          0.013338678516447544,
+          -0.030584163963794708,
+          0.010628811083734035,
+          0.045541103929281235,
+          0.006157049909234047,
+          0.0551932118833065,
+          0.004232373088598251,
+          0.0008839388028718531,
+          0.014545192010700703,
+          0.0316949225962162,
+          0.013980237767100334,
+          0.02951170690357685,
+          -0.014238775707781315,
+          0.0012543911579996347,
+          0.03596559911966324,
+          0.005199499428272247,
+          0.04301317036151886,
+          0.0017367572290822864,
+          0.00875679962337017,
+          -0.04370260611176491,
+          -0.03975749760866165,
+          -0.008866917341947556,
+          -0.0007863883511163294,
+          -0.001040737726725638,
+          -0.03994900733232498,
+          -0.0033107309136539698,
+          -0.03968089446425438,
+          -0.04423883557319641,
+          0.02323017455637455,
+          0.03994900733232498,
+          -0.02426433004438877,
+          0.01786789298057556,
+          -0.007732220459729433,
+          0.019763842225074768,
+          0.002664384199306369,
+          -0.013204621151089668,
+          0.006477829068899155,
+          0.0224066823720932,
+          -0.028190286830067635,
+          0.03617625683546066,
+          -0.0029588311444967985,
+          0.002010856056585908,
+          -0.007052359636873007,
+          0.004778177011758089,
+          0.01732208952307701,
+          0.026102827861905098,
+          -0.031139543280005455,
+          -0.025432541966438293,
+          0.0445452481508255,
+          0.03265247121453285,
+          0.03768918663263321,
+          -0.04940960556268692,
+          0.005448462441563606,
+          -0.005620821379125118,
+          -0.06055549532175064,
+          0.04075334966182709,
+          -9.201461944030598e-05,
+          0.014688825234770775,
+          -0.03060331381857395,
+          -0.022559890523552895,
+          -0.04703488200902939,
+          0.007018845062702894,
+          0.0016769103240221739,
+          -0.015483591705560684,
+          -0.015502742491662502,
+          -0.04504317790269852,
+          -0.027098679915070534,
+          -0.0336100235581398,
+          -0.0188541691750288,
+          -0.01513887383043766,
+          -0.0037224777042865753,
+          0.003921169321984053,
+          0.01097352895885706,
+          0.016709256917238235,
+          0.01247688289731741,
+          -0.008210995234549046,
+          -0.05186093598604202,
+          -0.01637411303818226,
+          0.022349229082465172,
+          -0.008622742258012295,
+          0.035161253064870834,
+          -0.006678914651274681,
+          -0.013022687286138535,
+          -0.053278110921382904,
+          -0.01699652150273323,
+          -0.04243863746523857,
+          0.02393876202404499,
+          0.05389094352722168,
+          0.00978137832134962,
+          -0.012936507351696491,
+          0.007511983625590801,
+          -0.0024537232238799334,
+          -0.019007377326488495,
+          -0.017475297674536705,
+          0.02566235326230526,
+          -0.017044398933649063,
+          -0.0073635634034872055,
+          -0.027290189638733864,
+          0.019763842225074768,
+          0.027347642928361893,
+          0.01561764907091856,
+          0.0010646764421835542,
+          0.02003195695579052,
+          0.004567516036331654,
+          -0.003028253559023142,
+          0.027826417237520218,
+          0.01593364030122757,
+          -0.0025303272996097803,
+          0.01721675880253315,
+          -0.012314099818468094,
+          0.005477189086377621,
+          0.042093921452760696,
+          -0.01270669512450695,
+          0.018815867602825165,
+          0.03678908944129944,
+          -0.00822535902261734,
+          0.027998777106404305,
+          -0.04132787883281708,
+          0.021295923739671707,
+          0.046115633100271225,
+          -0.02368979901075363,
+          0.009206848219037056,
+          -0.001639805268496275,
+          0.002424996579065919,
+          -0.014583494514226913,
+          0.002250243676826358,
+          -0.02188960462808609,
+          -0.0060325684025883675,
+          0.00822535902261734,
+          0.016307085752487183,
+          -0.02778811566531658,
+          -0.01008300669491291,
+          0.010456451214849949,
+          0.02932019717991352,
+          0.025777259841561317,
+          -0.025451691821217537,
+          0.01462179608643055,
+          -0.009527627378702164,
+          0.03301634266972542,
+          -0.016910342499613762,
+          0.03188643231987953,
+          0.004318553023040295,
+          -0.007095449138432741,
+          -0.02956916019320488,
+          -0.012783299200236797,
+          0.04653695598244667,
+          0.005893723573535681,
+          0.010580932721495628,
+          0.003533361479640007,
+          -0.01718803122639656,
+          0.021046960726380348,
+          0.02573895826935768,
+          -0.010963953100144863,
+          -0.004222797695547342,
+          -0.0013704941375181079,
+          0.033188700675964355,
+          0.03148426115512848,
+          -0.014401559717953205,
+          -0.017724260687828064,
+          0.0016350174555554986,
+          0.01283117663115263,
+          -0.028764817863702774,
+          -0.004581879358738661,
+          0.058333978056907654,
+          0.004543577320873737,
+          -0.02713698148727417,
+          -0.015004816465079784,
+          -0.007162477821111679,
+          -0.011299096047878265,
+          -0.023900460451841354,
+          0.029722368344664574,
+          -0.009800529107451439,
+          -0.02137252688407898,
+          -0.02183215133845806,
+          0.00822535902261734,
+          -0.026658207178115845,
+          -0.03129275143146515,
+          0.03014368936419487,
+          0.017810439690947533,
+          0.04014051705598831,
+          0.006338984239846468,
+          -0.015837885439395905,
+          -0.024666501209139824,
+          4.4511136366054416e-05,
+          0.01978299394249916,
+          -0.01490906160324812,
+          -0.0013297982513904572,
+          -0.034012194722890854,
+          0.004964899271726608,
+          0.010983104817569256,
+          0.018068978562951088,
+          -0.01577085629105568,
+          0.0026715658605098724,
+          0.029281895607709885,
+          -0.013444009236991405,
+          0.012055560946464539,
+          -0.017695533111691475,
+          0.016623076051473618,
+          -0.01658477447926998,
+          -0.005161197390407324,
+          0.033629175275564194,
+          -0.03678908944129944,
+          0.008129604160785675,
+          0.003705720417201519,
+          0.009379207156598568,
+          0.025049520656466484,
+          -0.002621294464915991,
+          -0.005333556327968836,
+          0.024302631616592407,
+          -0.021334225311875343,
+          0.024934615939855576,
+          0.008234934881329536,
+          0.010188337415456772,
+          0.006640612613409758,
+          -0.017504023388028145,
+          -0.011710843071341515,
+          -0.02183215133845806,
+          -0.0018253306625410914,
+          0.01132782269269228,
+          -0.0047135422937572,
+          -0.008455171249806881,
+          -0.019285067915916443,
+          0.0004862561181653291,
+          0.009680835530161858,
+          0.04557940363883972,
+          -0.014870759099721909,
+          0.013846180401742458,
+          0.01295565813779831,
+          -0.010130884125828743,
+          0.007282171864062548,
+          0.03140765801072121,
+          0.03372493013739586,
+          0.0008701740298420191,
+          0.004385581240057945,
+          0.022310927510261536,
+          0.026773111894726753,
+          0.0032125820871442556,
+          0.0057213641703128815,
+          -0.007492832839488983,
+          0.00989149697124958,
+          0.0008456368232145905,
+          0.01951487921178341,
+          0.011959806084632874,
+          -0.03014368936419487,
+          -0.02497291751205921,
+          -0.011385275982320309,
+          -0.023823857307434082,
+          0.01948615349829197,
+          -0.031637467443943024,
+          -0.02060648798942566,
+          0.011844899505376816,
+          0.009455811232328415,
+          0.0021125958301126957,
+          -0.00465369550511241,
+          -0.02093205414712429,
+          0.03378238156437874,
+          -0.015129297971725464,
+          0.060708701610565186,
+          -0.009757439605891705,
+          -0.03713380917906761,
+          -0.023670649155974388,
+          -0.008163117803633213,
+          -0.017255060374736786,
+          -0.03768918663263321,
+          -0.020070258527994156,
+          0.022808853536844254,
+          -0.01091607566922903,
+          -0.017532749101519585,
+          -0.0008426444255746901,
+          -0.000540118315257132,
+          -0.011624663136899471,
+          0.0006493389373645186,
+          0.021908756345510483,
+          0.005046291276812553,
+          0.0008887265576049685,
+          0.008124816231429577,
+          -0.021851303055882454,
+          0.005754878744482994,
+          -0.004373611882328987,
+          0.02278970181941986,
+          -0.026390092447400093,
+          -0.0026308700907975435,
+          -0.029684066772460938,
+          -0.05986605957150459,
+          -0.030316049233078957,
+          -0.017618929967284203,
+          0.012926931492984295,
+          0.007468894124031067,
+          0.011509757488965988,
+          -0.025566598400473595,
+          -0.004541183356195688,
+          -0.011978956870734692,
+          0.011710843071341515,
+          0.022483285516500473,
+          -0.005496339872479439,
+          -0.020836299285292625,
+          -0.003054586239159107,
+          -0.022061964496970177,
+          0.006812972016632557,
+          0.006927878130227327,
+          -0.0273284912109375,
+          0.05484849214553833,
+          -0.0005957759567536414,
+          0.0325184166431427,
+          -0.013568490743637085,
+          -0.019581908360123634,
+          -0.040102217346429825,
+          -0.011069283820688725,
+          -0.007248657289892435,
+          0.022425832226872444,
+          -0.025126125663518906,
+          -0.009609019383788109,
+          -0.019342521205544472,
+          -0.008230146951973438,
+          0.023670649155974388,
+          0.04224712774157524,
+          0.0333419106900692,
+          -0.003293973859399557,
+          0.00793809350579977,
+          0.002160473261028528,
+          -0.016623076051473618,
+          -0.014363258145749569,
+          -0.014315379783511162,
+          -0.003641085932031274,
+          0.007224718574434519,
+          0.03537191450595856,
+          0.030124539509415627,
+          -0.04305147007107735,
+          0.018126431852579117,
+          0.03784239664673805,
+          0.010054280050098896,
+          0.014344106428325176,
+          0.019744692370295525,
+          0.011729993857443333,
+          0.02905208244919777,
+          0.015081420540809631,
+          -0.01306098885834217,
+          -0.005917662288993597,
+          -0.010025554336607456,
+          -0.03339936211705208,
+          0.0243217833340168,
+          -0.010446876287460327,
+          -0.010284092277288437,
+          0.016766710206866264,
+          0.019581908360123634,
+          -0.0005595685797743499,
+          0.019821295514702797,
+          0.0024776619393378496,
+          -0.032863132655620575,
+          -0.009230786934494972,
+          -0.0018791928887367249,
+          0.014775004237890244,
+          0.012926931492984295,
+          -0.015196327120065689,
+          0.025260182097554207,
+          -0.023440835997462273,
+          0.01603897102177143,
+          0.011902352795004845,
+          0.014822881668806076,
+          0.015426138415932655,
+          0.022061964496970177,
+          -0.008862130343914032,
+          -0.004617787431925535,
+          0.015090996399521828,
+          0.004258705768734217,
+          -0.006161837372928858,
+          -0.015081420540809631,
+          0.016565624624490738,
+          -0.007842338643968105,
+          -0.004950535949319601,
+          -0.02765405923128128,
+          -0.005285678897053003,
+          -0.010839471593499184,
+          -0.007291747257113457,
+          0.015512318350374699,
+          -0.005146834068000317,
+          0.0074401674792170525,
+          -0.0003764370339922607,
+          0.021736396476626396,
+          -0.00026108213933184743,
+          0.0004120459489058703,
+          0.016489019617438316,
+          0.050520364195108414,
+          -0.02079799771308899,
+          -0.01209386344999075,
+          -0.0022514406591653824,
+          -0.0008977036341093481,
+          -0.010169186629354954,
+          -0.004342491738498211,
+          -0.005199499428272247,
+          -0.006501767784357071,
+          0.01270669512450695,
+          0.02060648798942566,
+          -0.025394240394234657,
+          -0.010992679744958878,
+          -0.023191872984170914,
+          -0.0377083383500576,
+          0.010178761556744576,
+          -0.01490906160324812,
+          -0.04730299487709999,
+          0.03937447816133499,
+          -0.014152596704661846,
+          0.010734140872955322,
+          0.029205290600657463,
+          -0.0032197635155171156,
+          -0.04722639173269272,
+          -0.0060421437956392765,
+          0.005414948333054781,
+          -0.00098208780400455,
+          -0.0031216146890074015,
+          0.0401788204908371,
+          -0.017829591408371925,
+          0.003597995964810252,
+          -0.0185860563069582,
+          0.01997450366616249,
+          -0.01689119078218937,
+          0.036137957125902176,
+          0.03598474711179733,
+          -0.04680506885051727,
+          -0.0033059432171285152,
+          -0.03292058780789375,
+          0.020912902429699898,
+          0.007784885820001364,
+          0.011059708893299103,
+          -0.014765429310500622,
+          0.020434128120541573,
+          -0.023536590859293938,
+          0.028726516291499138,
+          0.011835324577987194,
+          0.033054642379283905,
+          -0.023440835997462273,
+          -0.01746572181582451,
+          -0.02713698148727417,
+          -0.003203006461262703,
+          0.01937124691903591,
+          -0.02361319586634636,
+          0.010130884125828743,
+          0.019083982333540916,
+          0.01414302084594965,
+          0.026964621618390083,
+          0.020702242851257324,
+          -0.0015284899855032563,
+          -0.015090996399521828,
+          -0.00834984052926302,
+          0.04511978104710579,
+          0.007382714655250311,
+          0.0004383785999380052,
+          -0.005467613227665424,
+          0.011318246833980083,
+          0.026524148881435394,
+          0.030833126977086067,
+          -0.015646375715732574,
+          0.01625920832157135,
+          -0.008167905732989311,
+          -0.011011830531060696,
+          -0.028783967718482018,
+          -0.0021628672257065773,
+          -0.03186728060245514,
+          -0.03893400356173515,
+          0.007957245223224163,
+          -0.03521870821714401,
+          -0.002626082394272089,
+          0.015359110198915005,
+          0.021870452910661697,
+          0.02464734949171543,
+          -0.028592457994818687,
+          0.028879722580313683,
+          0.019332945346832275,
+          -0.008354628458619118,
+          -0.001125121838413179,
+          -0.027175283059477806,
+          -0.003918775357306004,
+          0.0018947530770674348,
+          -0.014918637461960316,
+          -0.021276772022247314,
+          -0.007176841143518686,
+          -0.012084287591278553,
+          -0.0061187478713691235,
+          0.009446235373616219,
+          -0.007813611999154091,
+          0.002906165784224868,
+          -0.025375088676810265,
+          0.03349511697888374,
+          0.003466332796961069,
+          0.007861489430069923,
+          -0.01577085629105568,
+          0.0006451496738009155,
+          0.0021137928124517202,
+          -0.003650661325082183,
+          -0.009269089438021183,
+          0.03259501978754997,
+          -0.010398998856544495,
+          0.0054532503709197044,
+          -0.013233347795903683,
+          0.0019222826231271029,
+          0.01863393373787403,
+          0.016824161633849144,
+          -0.017590202391147614,
+          0.0043329158797860146,
+          -0.005060654599219561,
+          0.004687209613621235,
+          -0.005194711498916149,
+          0.027998777106404305,
+          -0.011931079439818859,
+          -9.934586705639958e-05,
+          -0.0038948366418480873,
+          -0.009316966868937016,
+          0.024111121892929077,
+          0.000726541446056217,
+          0.014899485744535923,
+          0.023459987714886665,
+          0.023000363260507584,
+          -0.013242923654615879,
+          -0.007023632992058992,
+          0.022674795240163803,
+          0.024226028472185135,
+          0.01209386344999075,
+          -0.009541991166770458,
+          0.002733806613832712,
+          0.004045650828629732,
+          0.029664915055036545,
+          0.0029444678220897913,
+          0.010178761556744576,
+          -0.02669650875031948,
+          -0.02688801847398281,
+          0.013846180401742458,
+          -0.012400278821587563,
+          0.0049888379871845245,
+          0.012132165022194386,
+          0.013568490743637085,
+          0.005003201309591532,
+          -0.001164620858617127,
+          0.009302603080868721,
+          0.012065136805176735,
+          0.006147474516183138,
+          0.02355574257671833,
+          0.012821601703763008,
+          -0.015119723044335842,
+          -0.006875212769955397,
+          0.002081475453451276,
+          0.00019390397937968373,
+          -0.008857342414557934,
+          -0.03096718341112137,
+          -0.06235568970441818,
+          -0.02183215133845806,
+          -0.014526041224598885,
+          0.004560334142297506,
+          0.02503037080168724,
+          0.025317635387182236,
+          -0.0049888379871845245,
+          -0.04653695598244667,
+          -0.0034783021546900272,
+          0.006980543024837971,
+          0.014727126806974411,
+          -0.004538789391517639,
+          0.02451329305768013,
+          -0.023574894294142723,
+          0.005223438143730164,
+          0.0072007798589766026,
+          -0.023862158879637718,
+          0.03904891014099121,
+          0.02368979901075363,
+          -0.01174914464354515,
+          -0.01735081523656845,
+          -0.018432848155498505,
+          0.012180042453110218,
+          -0.0034495757427066565,
+          0.022636493667960167,
+          0.03297803923487663,
+          -0.03979580104351044,
+          0.043357886373996735,
+          -0.006937453523278236,
+          -0.002997133182361722,
+          -0.01056178193539381,
+          -0.0028606820851564407,
+          0.0014662492321804166,
+          0.0029708005022257566,
+          -0.004320946522057056,
+          0.01779128797352314,
+          -0.018796715885400772,
+          0.023402534425258636,
+          0.008287599310278893,
+          0.022368380799889565,
+          -0.004325734451413155,
+          0.020146863535046577,
+          -0.07281213998794556,
+          0.007004482205957174,
+          -0.004548364784568548,
+          -0.010896924883127213,
+          -0.04967772215604782,
+          -0.02721358649432659,
+          0.06036398559808731,
+          -0.03437606245279312,
+          0.02175554819405079,
+          0.02688801847398281,
+          0.0251452773809433,
+          -0.032346054911613464,
+          0.00023968686582520604,
+          -0.003961865324527025,
+          0.024628199636936188,
+          -0.00918290950357914,
+          0.034471817314624786,
+          0.021161865442991257,
+          0.015129297971725464,
+          0.015004816465079784,
+          -0.002413027221336961,
+          -0.01574213057756424,
+          0.08441765606403351,
+          -0.011461880058050156,
+          -0.041902411729097366,
+          0.03000963293015957,
+          -0.0404469333589077,
+          0.00390441226772964,
+          0.01295565813779831,
+          -0.0061187478713691235,
+          0.014995240606367588,
+          -0.0024142242036759853,
+          -0.009585080668330193,
+          -0.021085262298583984,
+          0.0020240223966538906,
+          -0.03301634266972542,
+          -0.02252158895134926,
+          -0.004790146369487047,
+          0.003511816496029496,
+          0.0008617954445071518,
+          -0.0029875575564801693,
+          -0.004189283587038517,
+          0.006338984239846468,
+          0.04347279295325279,
+          -0.023364232853055,
+          0.011030982248485088,
+          -0.005654335953295231,
+          -0.026600753888487816,
+          0.033839836716651917,
+          0.013558914884924889,
+          0.005046291276812553,
+          -0.01497608982026577,
+          0.008359416387975216,
+          0.002378316130489111,
+          -0.031388506293296814,
+          0.00649698032066226,
+          -0.029741518199443817,
+          -0.004256311804056168,
+          0.005496339872479439,
+          -0.00047727907076478004,
+          -0.0066118864342570305,
+          0.030545860528945923,
+          -0.01645071804523468,
+          -0.021046960726380348,
+          -0.009800529107451439,
+          -0.004708754830062389,
+          -0.009503688663244247,
+          -0.007339624688029289,
+          -0.00037075160071253777,
+          0.0007103827665559947,
+          0.03320785239338875,
+          0.02661990374326706,
+          0.011174614541232586,
+          -0.010772443376481533,
+          -0.0252793338149786,
+          0.003966652788221836,
+          -0.0022693946957588196,
+          0.02164064161479473,
+          0.025202728807926178,
+          -0.0262368842959404,
+          -0.008546138182282448,
+          -0.036003898829221725,
+          -0.010226639918982983,
+          -0.009144607000052929,
+          0.011423577554523945,
+          -0.011346973478794098,
+          0.038819096982479095,
+          -0.010446876287460327,
+          0.013367405161261559,
+          -0.026390092447400093,
+          0.008531775325536728,
+          0.013453585095703602,
+          -0.003674600040540099,
+          -0.04557940363883972,
+          0.01490906160324812,
+          0.01587618701159954,
+          -0.0016505776438862085,
+          0.008689770475029945,
+          -0.02407282032072544,
+          -0.01533038355410099,
+          0.06955646723508835,
+          0.01037027221173048,
+          -0.01620175503194332,
+          -0.013252499513328075,
+          -0.024666501209139824,
+          -0.011902352795004845,
+          -0.013769576326012611,
+          0.01735081523656845,
+          -0.010858622379601002,
+          0.004232373088598251,
+          -0.007669979706406593,
+          0.024091970175504684,
+          0.007459318730980158,
+          0.0012986778747290373,
+          0.02650499902665615,
+          -0.003511816496029496,
+          -0.02495376579463482,
+          0.03916381672024727,
+          -0.017235908657312393,
+          -0.011366124264895916,
+          -0.019304217770695686,
+          -0.0021149897947907448,
+          -0.02125762216746807,
+          0.03571663424372673,
+          -0.021525735035538673,
+          -0.0011867642169818282,
+          -0.011797022074460983,
+          -0.019668087363243103,
+          -0.007574224378913641,
+          0.009939374402165413,
+          0.01904567889869213,
+          0.010351121425628662,
+          -0.0029253168031573296,
+          0.00827802438288927,
+          -0.008498260751366615,
+          0.031714074313640594,
+          0.0035668755881488323,
+          0.015148448757827282,
+          0.020395826548337936,
+          0.013175895437598228,
+          0.0045723035000264645,
+          0.018461573868989944,
+          -0.010762867517769337,
+          0.006812972016632557,
+          -0.026466695591807365,
+          -0.014382408931851387,
+          0.023919612169265747,
+          -0.013290801085531712,
+          -0.014334531500935555,
+          0.0010844259522855282,
+          -0.0039068059995770454,
+          0.0022454559803009033,
+          -0.012333250604569912,
+          -0.0054628257639706135,
+          -0.017006097361445427,
+          0.009834043681621552,
+          0.009776590391993523,
+          0.01863393373787403,
+          0.004569910001009703,
+          0.029645763337612152,
+          -0.007751371245831251,
+          -0.007780097890645266,
+          -0.0073635634034872055,
+          -0.02298121154308319,
+          0.005237801466137171,
+          0.021870452910661697,
+          0.0006702853715978563,
+          -0.005596882663667202,
+          -0.023459987714886665,
+          0.0061378986574709415,
+          -0.011959806084632874,
+          0.021027809008955956,
+          -0.0007313292007893324,
+          0.018672235310077667,
+          0.0028032290283590555,
+          -0.03288228437304497,
+          -0.0033298819325864315,
+          -0.008421656675636768,
+          -0.00590329896658659,
+          -0.014889910817146301,
+          -0.004088740795850754,
+          -0.0015464440220966935,
+          -0.0031886431388556957,
+          0.037401922047138214,
+          -0.007128963712602854,
+          -0.009000974707305431,
+          -0.0014949757605791092,
+          0.015722978860139847,
+          0.020012805238366127,
+          0.0009467780473642051,
+          -0.008474322035908699,
+          0.010427725501358509,
+          -8.408490248257294e-05,
+          0.010762867517769337,
+          -0.0011221294989809394,
+          -0.008503048680722713,
+          0.001982129644602537,
+          -0.0003803270810749382,
+          -0.027979625388979912,
+          -0.005884147714823484,
+          0.0011592345545068383,
+          -0.013376981019973755,
+          0.038302019238471985,
+          0.005328768398612738,
+          -0.007004482205957174,
+          0.0015428533079102635,
+          0.018346667289733887,
+          0.01132782269269228,
+          0.010226639918982983,
+          -0.003126402385532856,
+          0.004490911960601807,
+          -0.04511978104710579,
+          -0.027749814093112946,
+          -0.025949619710445404,
+          -0.03156086429953575,
+          0.015445290133357048,
+          0.018317941576242447,
+          -0.006875212769955397,
+          0.013673821464180946,
+          0.012812025845050812,
+          -0.001474627759307623,
+          0.009695199318230152,
+          -0.012802449986338615,
+          -0.0017247878713533282,
+          -0.00949890073388815,
+          -0.011710843071341515,
+          0.018001949414610863,
+          0.011797022074460983,
+          0.024206876754760742,
+          0.03929787129163742,
+          -0.011950230225920677,
+          0.0026859291829168797,
+          0.0011843702523037791,
+          0.006631037220358849,
+          -0.005711788777261972,
+          -0.013989812694489956,
+          0.010580932721495628,
+          -0.000983883161097765,
+          0.014707976020872593,
+          -0.01209386344999075,
+          0.0019665693398565054,
+          -0.00619535194709897,
+          -0.013961086049675941,
+          0.011366124264895916,
+          -0.01805940270423889,
+          0.012371552176773548,
+          -0.03941277787089348,
+          -0.007315685972571373,
+          -0.016489019617438316,
+          0.051554519683122635,
+          0.007933305576443672,
+          -0.01782001554965973,
+          -0.0017511205514892936,
+          0.025624051690101624,
+          0.004100710153579712,
+          -0.009771803393959999,
+          0.008641893044114113,
+          -0.00548676447942853,
+          0.011050133034586906,
+          0.0009144607465714216,
+          0.0029995269142091274,
+          -0.026026222854852676,
+          -0.005855421535670757,
+          -0.0054628257639706135,
+          0.01989790052175522,
+          0.000962338293902576,
+          0.016900766640901566,
+          0.08870747685432434,
+          -0.0035070287995040417,
+          -0.009451023302972317,
+          -0.0068608494475483894,
+          -0.019763842225074768,
+          -0.004562728106975555,
+          -0.008900431916117668,
+          -0.005639972630888224,
+          0.0030426166485995054,
+          -0.007186416536569595,
+          0.0054724011570215225,
+          0.014229200780391693,
+          -0.010925651527941227,
+          -0.008699346333742142,
+          -0.007195991929620504,
+          -0.01430580485612154,
+          0.016852889209985733,
+          0.031656619161367416,
+          0.0116821164265275,
+          0.04902658611536026,
+          -0.02533678710460663,
+          -0.014861184172332287,
+          0.004873932339251041,
+          0.004251524340361357,
+          0.00989149697124958,
+          0.007464106194674969,
+          -0.012409854680299759,
+          0.02330677956342697,
+          0.007756159175187349,
+          -0.0037990817800164223,
+          -0.012687544338405132,
+          0.015588922426104546,
+          0.015464440919458866,
+          -0.006061294581741095,
+          0.031522564589977264,
+          0.0025542660150676966,
+          0.010667112655937672,
+          0.010398998856544495,
+          0.0049026585184037685,
+          0.01557934656739235,
+          0.014095143415033817,
+          -0.0006271955789998174,
+          0.002482449635863304,
+          -0.002757745562121272,
+          0.013319527730345726,
+          0.018289214000105858,
+          0.012984384782612324,
+          -0.005831482820212841,
+          0.02560489997267723,
+          0.026581602171063423,
+          -0.00262368842959404,
+          -0.031120391562581062,
+          0.03864673897624016,
+          -0.020261768251657486,
+          0.0006313848425634205,
+          -0.008890856057405472,
+          0.018384970724582672,
+          -0.009403145872056484,
+          -0.004950535949319601,
+          -0.009163758717477322,
+          -0.028822271153330803,
+          -0.003729659365490079,
+          -0.003921169321984053,
+          0.012668393552303314,
+          0.01869138516485691,
+          -0.010628811083734035,
+          0.02112356387078762,
+          -0.012649241834878922,
+          0.020510731264948845,
+          -0.008570076897740364,
+          -0.022004511207342148,
+          0.008713710121810436,
+          -0.001651774626225233,
+          0.0019426306243985891,
+          -0.015177175402641296,
+          0.021219318732619286,
+          0.04684337228536606,
+          0.004869144409894943,
+          -0.00040815590182319283,
+          -0.007990758866071701,
+          -0.0033969103824347258,
+          -0.010734140872955322,
+          0.021295923739671707,
+          0.0058793602511286736,
+          -0.032863132655620575,
+          0.009752651676535606,
+          0.015885762870311737,
+          0.0014782185899093747,
+          0.017810439690947533,
+          -0.0013908421387895942,
+          0.03156086429953575,
+          0.006587947718799114,
+          0.004258705768734217,
+          0.0036554490216076374,
+          -0.014928212389349937,
+          0.0025279333349317312,
+          -0.0027745026163756847,
+          -0.02727103792130947,
+          0.00483084237203002,
+          -0.0012160891201347113,
+          -0.020912902429699898,
+          0.02233007736504078,
+          -0.03841692581772804,
+          -0.019083982333540916,
+          -0.003083312651142478,
+          -0.018844593316316605,
+          -0.013252499513328075,
+          0.01937124691903591,
+          -0.013999388553202152,
+          0.03636777028441429,
+          -0.007899791933596134,
+          0.007962032221257687,
+          -0.0036865693982690573,
+          -0.016948644071817398,
+          0.038876552134752274,
+          0.0045579406432807446,
+          -0.011078859679400921,
+          -0.008694558404386044,
+          0.022425832226872444,
+          -0.02213856764137745,
+          0.017168881371617317,
+          0.005673486739397049,
+          0.029856424778699875,
+          -0.02233007736504078,
+          0.01068626344203949,
+          0.025777259841561317,
+          0.009613807313144207,
+          -0.006003841757774353,
+          -0.010226639918982983,
+          -0.031139543280005455,
+          0.031771525740623474,
+          0.04151938855648041,
+          0.004936173092573881,
+          -0.005544217303395271,
+          0.0022514406591653824,
+          0.013511037454009056,
+          0.014583494514226913,
+          0.010552207008004189,
+          0.007119387853890657,
+          0.0020240223966538906,
+          -0.032097093760967255,
+          0.018739264458417892,
+          0.002458510920405388,
+          0.015177175402641296,
+          0.009063215926289558,
+          -0.026792263612151146,
+          -0.0009743076516315341,
+          -0.0021437162067741156,
+          0.007617314346134663,
+          0.02003195695579052,
+          0.013444009236991405,
+          0.028017928823828697,
+          0.00994895026087761,
+          0.0016924705123528838,
+          -0.012563062831759453,
+          -0.03315039724111557,
+          0.010178761556744576,
+          0.016737982630729675,
+          0.004593848716467619,
+          -0.01686246506869793,
+          0.010466027073562145,
+          0.007282171864062548,
+          -0.01145230419933796,
+          -0.014995240606367588,
+          -0.0064634657464921474,
+          -0.0038852612487971783,
+          -0.0302011426538229,
+          0.006094809155911207,
+          0.0020324010401964188,
+          -0.007081086281687021,
+          0.011337397620081902,
+          0.015598498284816742,
+          0.013137592934072018,
+          -0.022885456681251526,
+          0.013913208618760109,
+          -0.010762867517769337,
+          -0.015004816465079784,
+          -0.008986611850559711,
+          0.006970967631787062,
+          -0.0020946417935192585,
+          -0.006980543024837971,
+          0.01155763491988182,
+          0.0011813779128715396,
+          -0.030354350805282593,
+          0.032997190952301025,
+          -0.0037033266853541136,
+          -0.03060331381857395,
+          -0.024685652926564217,
+          0.008584440685808659,
+          0.02771151252090931,
+          0.0022203202825039625,
+          -0.004696785472333431,
+          0.03182898089289665,
+          -0.010580932721495628,
+          -0.0029899515211582184,
+          -0.03380153328180313,
+          -0.0034280307590961456,
+          -0.012342826463282108,
+          -0.020893752574920654,
+          -0.012294949032366276,
+          0.00763167766854167,
+          -0.007464106194674969,
+          -0.011634238995611668,
+          0.01805940270423889,
+          -0.01686246506869793,
+          0.002427390543743968,
+          0.003870897926390171,
+          -0.0024441475979983807,
+          0.021161865442991257,
+          -0.017427420243620872,
+          0.0018133613048121333,
+          -0.0035692695528268814,
+          -0.02188960462808609,
+          0.006755518727004528,
+          -0.001415977836586535,
+          -0.03192473575472832,
+          -0.01861478202044964,
+          -0.020146863535046577,
+          -0.0251452773809433,
+          0.006592735182493925,
+          0.009202060289680958,
+          0.015339959412813187,
+          0.005544217303395271,
+          0.01786789298057556,
+          -0.005458037834614515,
+          -0.005525066517293453,
+          -0.027883870527148247,
+          0.001639805268496275,
+          0.009570717811584473,
+          0.014219624921679497,
+          -0.024436689913272858,
+          -0.010542631149291992,
+          0.010868198238313198,
+          0.02162148989737034,
+          -0.004100710153579712,
+          0.014899485744535923,
+          0.014420710504055023,
+          -0.0241877268999815,
+          -4.5165711526351515e-06,
+          -0.013645094819366932,
+          -0.0254133902490139,
+          -0.002503994619473815,
+          -0.0002746972895693034,
+          -0.004031287506222725,
+          -0.015311232767999172,
+          0.015004816465079784,
+          0.010523480363190174,
+          0.01631666161119938,
+          -0.02464734949171543,
+          -0.004459791351109743,
+          -0.004115073475986719,
+          -0.013999388553202152,
+          -0.021678943186998367,
+          0.012543912045657635,
+          -0.026351790875196457,
+          0.014114294201135635,
+          -0.001409993157722056,
+          0.020319221541285515,
+          0.01978299394249916,
+          0.017561476677656174,
+          -0.0019031316041946411,
+          -0.0026955048087984324,
+          0.009024913422763348,
+          0.0071146003901958466,
+          -0.016192179173231125,
+          0.015406987629830837,
+          -0.008503048680722713,
+          -0.0061283232644200325,
+          0.012266222387552261,
+          -0.008823827840387821,
+          0.002645233180373907,
+          0.014047265984117985,
+          -0.0059655397199094296,
+          0.0009988448582589626,
+          -0.008388142101466656,
+          -0.004206040408462286,
+          -0.009479749947786331,
+          0.007468894124031067,
+          -0.03330360725522041,
+          0.013128018006682396,
+          -0.0036722063086926937,
+          0.0033418512903153896,
+          -0.000942588783800602,
+          -0.007559861522167921,
+          -0.02093205414712429,
+          -0.008273236453533173,
+          -0.014324955642223358,
+          -0.002482449635863304,
+          -0.012065136805176735,
+          0.01939997263252735,
+          0.009139820002019405,
+          -0.005496339872479439,
+          0.009364844299852848,
+          0.006918302271515131,
+          -0.019036104902625084,
+          -0.0004512456653174013,
+          0.02317272312939167,
+          0.018183885142207146,
+          -0.00739229004830122,
+          0.03326530382037163,
+          -0.012026834301650524,
+          0.015081420540809631,
+          0.014286654070019722,
+          0.01830836571753025,
+          -0.006353347562253475,
+          -0.030507558956742287,
+          0.017896618694067,
+          0.020070258527994156,
+          -0.020759694278240204,
+          -0.013242923654615879,
+          0.006578371860086918,
+          0.042974866926670074,
+          -0.007085873745381832,
+          0.016086848452687263,
+          0.0012603758368641138,
+          -0.006827335339039564,
+          -0.011145887896418571,
+          -0.011193765327334404,
+          0.06863722205162048,
+          0.023364232853055,
+          0.013424858450889587,
+          -0.007124175783246756,
+          0.0022011692635715008,
+          -0.02112356387078762,
+          -0.001406402327120304,
+          0.0026428394485265017,
+          0.002515963977202773,
+          -0.008752011694014072,
+          -0.025011219084262848,
+          0.01916058547794819,
+          0.030450105667114258,
+          0.009163758717477322,
+          -0.011270369403064251,
+          -0.013874907046556473,
+          0.02303866483271122,
+          0.002209547907114029,
+          0.017341239377856255,
+          0.011797022074460983,
+          0.001982129644602537,
+          -0.03765088692307472,
+          0.003921169321984053,
+          0.015033543109893799,
+          0.0038852612487971783,
+          0.0036434796638786793,
+          0.0078567024320364,
+          0.012400278821587563,
+          0.011212916113436222,
+          0.0034280307590961456,
+          0.027979625388979912,
+          -0.026677357032895088,
+          -0.0019486153032630682,
+          0.0007307307096198201,
+          -0.0005475992220453918,
+          0.009015338495373726,
+          0.013721698895096779,
+          -0.015196327120065689,
+          -0.011174614541232586,
+          0.0011562422150745988,
+          0.0065592210739851,
+          -0.02284715510904789,
+          -0.01091607566922903,
+          0.01414302084594965,
+          -0.01984044723212719,
+          0.011816173791885376,
+          -0.0015763675328344107,
+          0.0006122338818386197,
+          -0.02323017455637455,
+          -0.0008929158793762326,
+          0.006683702580630779,
+          0.010935226455330849,
+          -0.007397077511996031,
+          0.022368380799889565,
+          -0.024111121892929077,
+          -0.01942870020866394,
+          0.017638079822063446,
+          -0.009747864678502083,
+          0.013156743720173836,
+          0.014669673517346382,
+          -0.013989812694489956,
+          -0.003739234758540988,
+          0.0007684343145228922,
+          -0.004940960556268692,
+          -0.010466027073562145,
+          0.00400495482608676,
+          0.0021664579398930073,
+          -0.005735727492719889,
+          0.002329241717234254,
+          0.017293361946940422,
+          -0.011548059061169624,
+          -0.01628793403506279,
+          -0.00590329896658659,
+          0.012400278821587563,
+          -0.000718761351890862,
+          0.018049826845526695,
+          0.01587618701159954,
+          -0.005917662288993597,
+          0.009834043681621552,
+          -0.022808853536844254,
+          0.01078201923519373,
+          0.008167905732989311,
+          -0.008498260751366615,
+          0.006769882049411535,
+          -0.0022670007310807705,
+          -0.004732693545520306,
+          0.03311209753155708,
+          0.020070258527994156,
+          0.007928518578410149,
+          -0.01283117663115263,
+          -0.005888935644179583,
+          -0.026792263612151146,
+          0.0003776339872274548,
+          -0.0028056229930371046,
+          -0.011672540567815304,
+          -0.026524148881435394,
+          0.011528908275067806,
+          0.002091050846502185,
+          -0.004423883277922869,
+          -0.020702242851257324,
+          0.013740849681198597,
+          0.015627223998308182,
+          0.0005272512789815664,
+          0.002843925030902028,
+          0.015167600475251675,
+          0.010657536797225475,
+          0.018298789858818054,
+          -0.010504329577088356,
+          -0.0056447600945830345,
+          0.001198135083541274,
+          -0.026524148881435394,
+          0.0036722063086926937,
+          -0.006085233297199011,
+          0.003061767667531967,
+          0.01786789298057556,
+          -0.011481030844151974,
+          -0.01989790052175522,
+          0.003468726761639118,
+          0.0032987615559250116,
+          -0.01935209520161152,
+          0.014880334958434105,
+          -0.023249326273798943,
+          -0.0004925400717183948,
+          -0.030430955812335014,
+          -0.0060325684025883675,
+          -0.002427390543743968,
+          0.0017834377940744162,
+          -0.0049792625941336155,
+          0.0019569939468055964,
+          -0.015435714274644852,
+          0.010829896666109562,
+          0.03188643231987953,
+          0.010284092277288437,
+          -0.0012071121018379927,
+          0.001107766292989254,
+          -0.0043161590583622456,
+          -0.006841698195785284,
+          -0.01139485090970993,
+          -0.009786166250705719,
+          0.01299396064132452,
+          -0.0058793602511286736,
+          0.02495376579463482,
+          -0.009283452294766903,
+          0.0024513292592018843,
+          0.006846486125141382,
+          0.008708922192454338,
+          -0.011059708893299103,
+          -0.018298789858818054,
+          0.00947017502039671,
+          -0.011864051222801208,
+          0.00815833080559969,
+          -0.008929158560931683,
+          -0.011021406389772892,
+          0.0016050940612331033,
+          0.0032245514448732138,
+          -0.012649241834878922,
+          -0.014707976020872593,
+          -0.005333556327968836,
+          -0.0006906333146616817,
+          -0.005802756175398827,
+          0.00834984052926302,
+          -0.008163117803633213,
+          -0.008373779244720936,
+          -0.005391009617596865,
+          -0.0011041754623875022,
+          -0.0023172723595052958,
+          0.0032413084991276264,
+          0.007765734568238258,
+          0.020051108673214912,
+          0.00900576263666153,
+          0.012505609542131424,
+          0.023268477991223335,
+          0.01513887383043766,
+          0.010523480363190174,
+          -0.008301963098347187,
+          0.01948615349829197,
+          -0.006243229378014803,
+          0.004411913920193911,
+          0.022828003391623497,
+          0.010829896666109562,
+          -0.00691351480782032,
+          0.02757745422422886,
+          -0.010150035843253136,
+          -0.024915464222431183,
+          -0.01917973719537258,
+          0.010858622379601002,
+          -0.02413027361035347,
+          0.003813444869592786,
+          -0.004356855060905218,
+          -0.022253474220633507,
+          0.01174914464354515,
+          0.01596236787736416,
+          0.021085262298583984,
+          0.001044927048496902,
+          -0.004809297621250153,
+          0.0007546694832853973,
+          0.016115576028823853,
+          0.01276414841413498,
+          -0.005836270283907652,
+          0.0012879054993391037,
+          -0.003830202156677842,
+          0.011433153413236141,
+          0.010734140872955322,
+          -0.0018672235310077667,
+          0.007306110579520464,
+          -0.001930661266669631,
+          0.0016565624391660094,
+          0.0016936674946919084,
+          0.011825748719274998,
+          -0.005946388468146324,
+          -0.001229255460202694,
+          0.0012376339873299003,
+          0.011978956870734692,
+          0.0015727767022326589,
+          -0.00798118393868208,
+          0.015895338729023933,
+          0.011337397620081902,
+          -0.014717550948262215,
+          0.01916058547794819,
+          -0.010063855908811092,
+          -0.003739234758540988,
+          0.0006499374285340309,
+          0.015636799857020378,
+          0.004601030144840479,
+          0.01759977824985981,
+          -0.0032077941577881575,
+          0.008139179088175297,
+          -0.015158024616539478,
+          -0.020434128120541573,
+          -0.04672846570611,
+          0.008617954328656197,
+          -0.0020431734155863523,
+          -0.006884788163006306,
+          0.0019498122856020927,
+          0.01901695318520069,
+          0.013300376944243908,
+          0.009417509660124779,
+          -0.0034591511357575655,
+          0.014746277593076229,
+          -0.018021101132035255,
+          -0.019218038767576218,
+          -0.015943216159939766,
+          0.00912545621395111,
+          0.0018001949647441506,
+          -0.002961224876344204,
+          -0.005630397237837315,
+          0.020568184554576874,
+          -0.006420376244932413,
+          0.011184190399944782,
+          -3.63569961336907e-05,
+          -0.018442422151565552,
+          -0.007382714655250311,
+          -0.004452609922736883,
+          0.00857965275645256,
+          0.0029468617867678404,
+          0.022387530654668808,
+          0.0116821164265275,
+          0.00011879610974574462,
+          0.010475602932274342,
+          -0.010150035843253136,
+          -0.0066884905099868774,
+          0.016594350337982178,
+          0.013578066602349281,
+          0.0002670668181963265,
+          0.012361977249383926,
+          0.014353682287037373,
+          0.004675240255892277,
+          0.0019426306243985891,
+          0.01590491458773613,
+          0.05787435173988342,
+          0.018902046605944633,
+          0.003605177626013756,
+          -0.008440807461738586,
+          0.008120028302073479,
+          0.004897871054708958,
+          0.013740849681198597,
+          -0.00372726540081203,
+          -0.0234982892870903,
+          0.006832122802734375,
+          0.010236214846372604,
+          -0.01948615349829197,
+          -0.009958525188267231,
+          0.006022992543876171,
+          0.00836420338600874,
+          -0.015857037156820297,
+          -0.0004030689306091517,
+          -0.011844899505376816,
+          -0.016086848452687263,
+          0.0004171329492237419,
+          0.013884482905268669,
+          -0.023440835997462273,
+          -0.0011951427441090345,
+          -0.0009521642932668328,
+          -0.008215783163905144,
+          -0.009556354023516178,
+          0.0005706402589567006,
+          -0.0021832152269780636,
+          -0.018480725586414337,
+          -0.0022059569600969553,
+          -0.00936005637049675,
+          -0.007444955408573151,
+          0.022636493667960167,
+          -0.0102457907050848,
+          -0.006109172478318214,
+          -0.02213856764137745,
+          -0.023957913741469383,
+          0.01784874126315117,
+          -0.016249632462859154,
+          0.003097675973549485,
+          -0.018155157566070557,
+          0.002120974473655224,
+          -0.019036104902625084,
+          0.01677628420293331,
+          0.010389422997832298,
+          -0.01942870020866394,
+          0.02131507359445095,
+          0.006511343643069267,
+          0.004809297621250153,
+          -0.012649241834878922,
+          -0.002219123300164938,
+          -0.012314099818468094,
+          0.0088142529129982,
+          0.009795742109417915,
+          0.010647961869835854,
+          0.008680195547640324,
+          0.008282812312245369,
+          0.006865637376904488,
+          0.005788392852991819,
+          -0.012697120197117329,
+          -0.0401788204908371,
+          0.01882544346153736,
+          0.016077272593975067,
+          -0.01628793403506279,
+          0.016278358176350594,
+          -0.013865331187844276,
+          -0.020012805238366127,
+          -0.01378872711211443,
+          -0.008689770475029945,
+          -0.014583494514226913,
+          0.011825748719274998,
+          0.01601981930434704,
+          -0.011366124264895916,
+          0.00036416843067854643,
+          0.006372498814016581,
+          0.016173027455806732,
+          -0.009546778164803982,
+          0.005979903042316437,
+          -0.0015272931195795536,
+          -0.01049475371837616,
+          0.01593364030122757,
+          -0.016986945644021034,
+          -0.006248017307370901,
+          0.004277857020497322,
+          -0.022579040378332138,
+          -0.018672235310077667,
+          -0.011461880058050156,
+          0.014947363175451756,
+          -0.0052282256074249744,
+          -0.02194705791771412,
+          -0.0046608769334852695,
+          -0.03539106622338295,
+          0.008459959179162979,
+          0.00566391134634614,
+          -0.0017786500975489616,
+          -0.015541044995188713,
+          0.036712486296892166,
+          0.004751844331622124,
+          -0.02047242969274521,
+          0.005027140025049448,
+          -0.02041497640311718,
+          0.013807878829538822,
+          0.009518052451312542,
+          -0.017149729654192924,
+          0.002458510920405388,
+          0.0007905776146799326,
+          -0.0023807098623365164,
+          -0.00625759270042181,
+          0.004285038448870182,
+          0.004735087510198355,
+          -0.01896907575428486,
+          -0.004124648869037628,
+          0.015464440919458866,
+          -0.0033442452549934387,
+          -0.010513904504477978,
+          -0.011835324577987194,
+          -0.0037751428317278624,
+          0.0025662353727966547,
+          -0.0037631734739989042,
+          -0.0003393319493625313,
+          -0.01866265945136547,
+          -0.01650817133486271,
+          0.034529272466897964,
+          0.0020754907745867968,
+          0.017590202391147614,
+          -0.00494574848562479,
+          -0.0015979124000295997,
+          -0.008694558404386044,
+          0.015339959412813187,
+          0.013874907046556473,
+          0.011471454985439777,
+          0.0029708005022257566,
+          0.009436660446226597,
+          0.01773383468389511,
+          0.00483802380040288,
+          0.01869138516485691,
+          -0.009168545715510845,
+          -0.00888128113001585,
+          -0.011385275982320309,
+          0.029090384021401405,
+          0.009522839449346066,
+          -0.007023632992058992,
+          -0.025394240394234657,
+          -0.0019905080553144217,
+          0.009460599161684513,
+          0.024168575182557106,
+          -0.018767990171909332,
+          -0.028898874297738075,
+          -0.013204621151089668,
+          0.01418132334947586,
+          0.01549316756427288,
+          -0.014085568487644196,
+          -0.0022394713014364243,
+          0.011241642758250237,
+          0.023823857307434082,
+          -0.027826417237520218,
+          0.00936005637049675,
+          -0.0033298819325864315,
+          0.0037320530973374844,
+          -0.01939997263252735,
+          -0.02476225607097149,
+          0.00017475297499913722,
+          0.0034040920436382294,
+          0.009111093357205391,
+          -0.015301656909286976,
+          0.0017211970407515764,
+          0.020759694278240204,
+          -0.009628170169889927,
+          0.015406987629830837,
+          -0.0007564648985862732,
+          -0.010418149642646313,
+          0.0013369799125939608,
+          -0.005323980934917927,
+          -0.009814892895519733,
+          0.015560195781290531,
+          0.005414948333054781,
+          0.00428982637822628,
+          0.002503994619473815,
+          -0.00953720323741436,
+          -0.003291579894721508,
+          -0.008981823921203613,
+          -0.010284092277288437,
+          0.015110147185623646,
+          0.013769576326012611,
+          -0.020644789561629295,
+          0.024666501209139824,
+          -0.0029037718195468187,
+          -0.0049002645537257195,
+          -0.00507022999227047,
+          -0.008459959179162979,
+          0.0170827005058527,
+          0.0006918302387930453,
+          -0.00016442938067484647,
+          0.015282506123185158,
+          0.0038996245712041855,
+          0.001734363380819559,
+          -0.017178455367684364,
+          -0.007047571707516909,
+          -0.00023445025726687163,
+          0.005917662288993597,
+          0.0065257069654762745,
+          0.016661379486322403,
+          -0.006310258060693741,
+          -0.01014045998454094,
+          0.007023632992058992,
+          0.00955156609416008,
+          0.015924064442515373,
+          0.013022687286138535,
+          0.014449437148869038,
+          0.018767990171909332,
+          0.019189313054084778,
+          -0.008412080816924572,
+          -0.01727421209216118,
+          0.017235908657312393,
+          -0.008139179088175297,
+          -0.003289185930043459,
+          -0.04117467254400253,
+          0.01705397479236126,
+          0.020817147567868233,
+          -0.02137252688407898,
+          0.001020988216623664,
+          -0.004663270898163319,
+          -0.007784885820001364,
+          -0.021602340042591095,
+          0.018155157566070557,
+          -0.004852387122809887,
+          0.0019210857572034001,
+          0.004888295195996761,
+          -0.01820303499698639,
+          0.005855421535670757,
+          0.023574894294142723,
+          -0.009000974707305431,
+          0.004907446447759867,
+          -0.0021772305481135845,
+          -0.003554906230419874,
+          -0.012917356565594673,
+          0.006597523111850023,
+          -0.02457074634730816,
+          0.023670649155974388,
+          0.018193459138274193,
+          0.011002255603671074,
+          -0.007952457293868065,
+          0.007473681587725878,
+          -0.015349534340202808,
+          -0.0027266251854598522,
+          -0.00199888669885695,
+          0.014104719273746014,
+          0.002975588198751211,
+          0.00817269366234541,
+          0.0058697848580777645,
+          0.002906165784224868,
+          0.0032748228404670954,
+          -0.018241336569190025,
+          0.016144301742315292,
+          -0.006147474516183138,
+          0.007947669364511967,
+          0.013635518960654736,
+          0.0026931108441203833,
+          -0.009599443525075912,
+          -0.010312818922102451,
+          -0.004821266978979111,
+          0.006592735182493925,
+          0.004931385163217783,
+          0.011911928653717041,
+          0.009977675974369049,
+          0.009824467822909355,
+          -0.02503037080168724,
+          0.012879054062068462,
+          0.0032556718215346336,
+          -0.03282483294606209,
+          -0.002697898540645838,
+          0.0016972583252936602,
+          -0.0030761309899389744,
+          0.010255365632474422,
+          0.02317272312939167,
+          -0.017504023388028145,
+          -0.009206848219037056,
+          0.01801152527332306,
+          0.005199499428272247,
+          -0.016077272593975067,
+          -0.016852889209985733,
+          0.007377926725894213,
+          0.009901071898639202,
+          0.011098010465502739,
+          -0.014985665678977966,
+          0.009077578783035278,
+          0.007052359636873007,
+          0.0254133902490139,
+          -0.03085227683186531,
+          0.018902046605944633,
+          0.012266222387552261,
+          0.0169390682131052,
+          -0.00888128113001585,
+          -0.010896924883127213,
+          0.0008031455217860639,
+          0.012888629920780659,
+          -0.0017571052303537726,
+          0.015598498284816742,
+          -0.002170048886910081,
+          0.005256952252238989,
+          -0.02368979901075363,
+          -0.005041503347456455,
+          -0.0026404454838484526,
+          0.023057816550135612,
+          0.01485160831362009,
+          0.02661990374326706,
+          0.006233653984963894,
+          -0.012007683515548706,
+          0.010743716731667519,
+          -0.006429951637983322,
+          0.0003551913832779974,
+          -0.021219318732619286,
+          -0.00039409188320860267,
+          -0.019610634073615074,
+          -0.003442394081503153,
+          -0.016900766640901566,
+          0.03609965369105339,
+          0.02047242969274521,
+          0.014123870059847832,
+          0.025509145110845566,
+          -9.672756277723238e-05,
+          0.008670619688928127,
+          -0.004225191660225391,
+          -0.0018923592288047075,
+          -0.015253779478371143,
+          0.00774658378213644,
+          -0.006789033301174641,
+          0.001808573491871357,
+          0.008622742258012295,
+          -0.004129436798393726,
+          0.016632651910185814,
+          -0.022483285516500473,
+          0.014411135576665401,
+          0.01378872711211443,
+          0.014315379783511162,
+          0.016603926196694374,
+          -0.010284092277288437,
+          -0.0031862494070082903,
+          -0.0028223800472915173,
+          0.001198135083541274,
+          -0.009039277210831642,
+          -0.0023711344692856073,
+          -0.0028918024618178606,
+          -0.0088142529129982,
+          -0.005951176397502422,
+          0.022387530654668808,
+          0.0024968129582703114,
+          0.018298789858818054,
+          0.0024477385450154543,
+          0.015885762870311737,
+          0.005673486739397049,
+          0.005783604923635721,
+          -0.0157134048640728,
+          -0.0005550800706259906,
+          0.013482310809195042,
+          0.0021999722812324762,
+          -0.0031790677458047867,
+          -0.021813001483678818,
+          -0.021659793332219124,
+          0.0007851914269849658,
+          0.00673636794090271,
+          0.02162148989737034,
+          0.0017666807398200035,
+          -0.015272930264472961,
+          0.0008462352561764419,
+          -0.021295923739671707,
+          -0.0243217833340168,
+          -0.014564343728125095,
+          0.0011191371595486999,
+          0.029856424778699875,
+          0.0016350174555554986,
+          -0.009010550566017628,
+          0.009407933801412582,
+          -0.021985359489917755,
+          -0.0023615588434040546,
+          0.002669172128662467,
+          -0.0204915814101696,
+          0.009202060289680958,
+          -0.031522564589977264,
+          0.015474015846848488,
+          -0.013463160023093224,
+          0.00035459292121231556,
+          0.022310927510261536,
+          -0.005165984854102135,
+          -0.0008492275956086814,
+          -0.011385275982320309,
+          0.00044556023203767836,
+          0.0159910935908556,
+          -0.004129436798393726,
+          -0.004514850676059723,
+          -0.0014111900236457586,
+          0.003509422531351447,
+          -0.0016745164757594466,
+          -0.005941601004451513,
+          0.0019881143234670162,
+          0.0011125539895147085,
+          -0.019581908360123634,
+          0.0006631037103943527,
+          0.018404120579361916,
+          -0.012974809855222702,
+          -0.03246096149086952,
+          0.01653689704835415,
+          -0.02405366860330105,
+          -0.010801170021295547,
+          -0.0035261798184365034,
+          0.017006097361445427,
+          0.0022658039815723896,
+          -0.007492832839488983,
+          0.020721392706036568,
+          0.007933305576443672,
+          0.01008300669491291,
+          0.004426277242600918,
+          0.003159916726872325,
+          -0.0002832254976965487,
+          -0.0026428394485265017,
+          -5.97533971813391e-06,
+          0.0064634657464921474,
+          0.015971941873431206,
+          -0.018375394865870476,
+          0.027883870527148247,
+          -0.0070284209214150906,
+          -0.0034280307590961456,
+          0.0043233404867351055,
+          -0.006808184087276459,
+          -0.008368991315364838,
+          0.010877774097025394,
+          -0.02721358649432659,
+          0.017944496124982834,
+          -0.015090996399521828,
+          -0.011921503581106663,
+          -0.011883202008903027,
+          0.014229200780391693,
+          0.006664551794528961,
+          -0.01951487921178341,
+          0.005740515422075987,
+          -0.01490906160324812,
+          -0.01596236787736416,
+          0.002160473261028528,
+          -0.004100710153579712,
+          0.014928212389349937,
+          -0.0116821164265275,
+          -0.01164381392300129,
+          0.00536228297278285,
+          -0.019859597086906433,
+          0.008321113884449005,
+          -0.0077705224975943565,
+          0.009403145872056484,
+          -0.025892166420817375,
+          0.011576785705983639,
+          0.015110147185623646,
+          -0.004735087510198355,
+          -0.017197607085108757,
+          0.015857037156820297,
+          0.016469869762659073,
+          -0.004447821993380785,
+          -0.011481030844151974,
+          0.00822535902261734,
+          0.013147168792784214,
+          0.0025518720503896475,
+          -0.004984050523489714,
+          -0.017130577936768532,
+          -0.01126079447567463,
+          0.00923557486385107,
+          -0.0016074879094958305,
+          -0.036655034869909286,
+          -0.01672840677201748,
+          -0.02131507359445095,
+          -0.011433153413236141,
+          -0.009474962018430233,
+          0.006003841757774353,
+          0.006607098504900932,
+          -0.002515963977202773,
+          -0.02738594450056553,
+          -0.00035638833651319146,
+          0.004857175052165985,
+          -0.014813306741416454,
+          -0.001930661266669631,
+          -0.003621934913098812,
+          -0.0036985387559980154,
+          -0.0048164790496230125,
+          -0.005060654599219561,
+          0.013769576326012611,
+          -0.003739234758540988,
+          0.01239070389419794,
+          0.011011830531060696,
+          -0.00787106528878212,
+          -0.0026619904674589634,
+          -0.03211624547839165,
+          -0.014449437148869038,
+          0.008182269521057606,
+          0.0068800002336502075,
+          -0.002951649483293295,
+          -0.0005161795997992158,
+          0.008321113884449005,
+          0.00029130480834282935,
+          -0.005084593314677477,
+          0.005237801466137171,
+          0.013195046223700047,
+          -0.013501462526619434,
+          0.024858010932803154,
+          0.010456451214849949,
+          -0.00863710604608059,
+          -0.012534336186945438,
+          -0.01078201923519373,
+          0.018509451299905777,
+          0.009068002924323082,
+          0.017331663519144058,
+          -0.0023519834503531456,
+          0.01247688289731741,
+          0.0014231594977900386,
+          0.0262368842959404,
+          -0.006482616998255253,
+          -0.006808184087276459,
+          -0.010839471593499184,
+          0.0021568825468420982,
+          -0.02778811566531658,
+          0.00447415467351675,
+          0.01478458009660244,
+          -0.009987251833081245,
+          -0.002028810093179345,
+          0.006406012922525406,
+          -0.006468253675848246,
+          0.011548059061169624,
+          -0.009977675974369049,
+          0.0024800559040158987,
+          -0.017551900818943977,
+          -0.0020096590742468834,
+          0.016546472907066345,
+          0.0010138065554201603,
+          0.006640612613409758,
+          0.04749450460076332,
+          -0.009723925031721592,
+          -0.009393570944666862,
+          0.026102827861905098,
+          -1.738365608616732e-05,
+          0.019993655383586884,
+          -0.0006035560509189963,
+          0.0010419347090646625,
+          0.004675240255892277,
+          0.013645094819366932,
+          0.018538178876042366,
+          0.005304829683154821,
+          0.00196776632219553,
+          -0.01084904745221138,
+          -0.003356214612722397,
+          -0.011605512350797653,
+          -0.033514268696308136,
+          -0.005419735796749592,
+          0.003049798309803009,
+          -0.0069422414526343346,
+          0.007579012308269739,
+          -0.01847114972770214,
+          0.007138539105653763,
+          0.02483886107802391,
+          -0.017006097361445427,
+          -0.015158024616539478,
+          -0.008646680973470211,
+          -0.006487404927611351,
+          0.0004410716937854886,
+          -0.01696779578924179,
+          0.001875602058134973,
+          -0.007076298352330923,
+          0.0036817817017436028,
+          0.005103744100779295,
+          -0.0135972173884511,
+          0.022004511207342148,
+          0.0030210718978196383,
+          0.020721392706036568,
+          -0.006645400542765856,
+          -0.0001726583286654204,
+          0.005932025611400604,
+          -0.010935226455330849,
+          -0.021468281745910645,
+          0.004325734451413155,
+          0.01049475371837616,
+          -0.02162148989737034,
+          -0.022253474220633507,
+          -0.0029109534807503223,
+          0.0015655950410291553,
+          -0.01030324399471283,
+          0.006990118883550167,
+          -0.0016421991167590022,
+          0.0126396669074893,
+          -0.00013046625826973468,
+          -0.0069422414526343346,
+          0.016527321189641953,
+          0.02175554819405079,
+          -0.004735087510198355,
+          -0.0029444678220897913,
+          -0.0004012735153082758,
+          -0.015550619922578335,
+          0.011662965640425682,
+          0.008866917341947556,
+          -0.018681811168789864,
+          0.015272930264472961,
+          0.017915770411491394,
+          0.016785860061645508,
+          0.0015955185517668724,
+          -0.014516465365886688,
+          -0.0005520877311937511,
+          -0.0038948366418480873,
+          -0.00017789493722375482,
+          -0.01549316756427288,
+          -0.005098956637084484,
+          -0.004912233911454678,
+          -0.014803730882704258,
+          0.0020802784711122513,
+          -0.026581602171063423,
+          -0.00043299238313920796,
+          -0.011940655298531055,
+          -0.0003560890909284353,
+          0.020261768251657486,
+          -0.019218038767576218,
+          -0.0006852470687590539,
+          -0.011308671906590462,
+          -0.018356243148446083,
+          0.013070564717054367,
+          -0.005089380778372288,
+          -0.021927906200289726,
+          0.0066118864342570305,
+          0.007344412617385387,
+          -0.008910007774829865,
+          -0.014640947803854942,
+          0.007985970936715603,
+          -0.012553486973047256,
+          0.0009617398027330637,
+          0.020376674830913544,
+          0.016718832775950432,
+          0.009518052451312542,
+          -0.00857965275645256,
+          -0.004433458670973778,
+          0.011739569716155529,
+          0.01689119078218937,
+          0.004895477090030909,
+          0.02694547176361084,
+          -0.024743104353547096,
+          -0.029090384021401405,
+          0.04308977350592613,
+          0.007009269669651985,
+          -0.009546778164803982,
+          0.013415282592177391,
+          -0.024628199636936188,
+          0.009599443525075912,
+          0.033322758972644806,
+          0.008924370631575584,
+          -0.017312513664364815,
+          0.014832457527518272,
+          0.002272985642775893,
+          -0.021525735035538673,
+          0.0035237858537584543,
+          0.005410160403698683,
+          -0.015416563488543034,
+          0.005395797081291676,
+          0.016948644071817398,
+          0.006640612613409758,
+          0.01286947913467884,
+          -0.00235318043269217,
+          0.008067362941801548,
+          0.010255365632474422,
+          0.004277857020497322,
+          -0.007947669364511967,
+          -0.006951816845685244,
+          0.009316966868937016,
+          -0.039834100753068924,
+          -0.01485160831362009,
+          -0.013128018006682396,
+          -0.0052138627506792545,
+          -0.007847126573324203,
+          0.021659793332219124,
+          -0.013405706733465195,
+          -0.01874883845448494,
+          0.02164064161479473,
+          0.024398386478424072,
+          -0.006980543024837971,
+          -0.004665664862841368,
+          0.0037200837396085262,
+          -0.016517747193574905,
+          -0.0020060683600604534,
+          0.005491552408784628,
+          0.035046350210905075,
+          -0.0072773839347064495,
+          -0.0060325684025883675,
+          0.008699346333742142,
+          -0.014545192010700703,
+          0.02956916019320488,
+          0.005285678897053003,
+          -0.006329408846795559,
+          -0.00924993772059679,
+          -0.022119417786598206,
+          -0.0030019208788871765,
+          -0.017130577936768532,
+          -0.02183215133845806,
+          0.0038804735522717237,
+          -0.0010461239144206047,
+          -0.0008893250487744808,
+          0.004881113767623901,
+          0.0022394713014364243,
+          0.018241336569190025,
+          -0.009800529107451439,
+          0.011117161251604557,
+          -0.018432848155498505,
+          -0.0024501322768628597,
+          0.006812972016632557,
+          -0.0017630899092182517,
+          -0.010772443376481533,
+          -0.021027809008955956,
+          0.023364232853055,
+          -0.006894363556057215,
+          0.012496034614741802,
+          -0.01920846290886402,
+          0.013386555947363377,
+          -0.0166709553450346,
+          -0.002225107979029417,
+          0.0032293391413986683,
+          -0.005271315574645996,
+          -0.004964899271726608,
+          -0.003289185930043459,
+          0.01939997263252735,
+          -0.011567209847271442,
+          -0.012084287591278553,
+          0.0251452773809433,
+          0.0241877268999815,
+          0.004268281627446413,
+          -0.01257263869047165,
+          0.001366903306916356,
+          0.007263020612299442,
+          0.0028941964264959097,
+          0.0016015032306313515,
+          0.002992345253005624,
+          0.002399860881268978,
+          0.002276576356962323,
+          0.006712429225444794,
+          0.01978299394249916,
+          -0.016766710206866264,
+          0.0060517191886901855,
+          0.023440835997462273,
+          0.01874883845448494,
+          -0.006908726878464222,
+          -0.002071899827569723,
+          -0.012266222387552261,
+          0.005970327649265528,
+          0.012984384782612324,
+          0.0052138627506792545,
+          0.0015452471561729908,
+          0.0019450244726613164,
+          0.007162477821111679,
+          -0.024302631616592407,
+          0.030181992799043655,
+          0.0028271679766476154,
+          0.014516465365886688,
+          6.553236016770825e-05,
+          0.015177175402641296,
+          -0.013530189171433449,
+          -0.002506388584151864,
+          0.013798302970826626,
+          -0.006248017307370901,
+          0.008790313266217709,
+          0.007574224378913641,
+          -0.005898511037230492,
+          -0.012582213617861271,
+          -0.00044137093937024474,
+          -0.0036674183793365955,
+          -0.00715290242806077,
+          -0.0026165067683905363,
+          -0.002841531066223979,
+          0.0013214197242632508,
+          -0.008316325955092907,
+          0.006410800851881504,
+          -0.007344412617385387,
+          0.007306110579520464,
+          -0.002270591678097844,
+          -0.003770355135202408,
+          0.0008683786145411432,
+          0.025968769565224648,
+          0.00806257501244545,
+          0.03135020285844803,
+          0.003291579894721508,
+          0.0016589562874287367,
+          -0.0008611970115453005,
+          -0.014535617083311081,
+          -0.0262368842959404,
+          0.011346973478794098,
+          -0.02771151252090931,
+          0.00031449549715034664,
+          -0.0032676411792635918,
+          0.003902018303051591,
+          0.012534336186945438,
+          0.011098010465502739,
+          0.009096729569137096,
+          -0.018930774182081223,
+          -0.013568490743637085,
+          -0.002460904885083437,
+          -0.011921503581106663,
+          0.002963618841022253,
+          0.012218344956636429,
+          -0.005118107423186302,
+          0.012792875058948994,
+          -0.0004982255049981177,
+          0.002997133182361722,
+          -0.03699975088238716,
+          -0.012256646528840065,
+          0.00241183047182858,
+          -0.00994895026087761,
+          0.007708281744271517,
+          0.009039277210831642,
+          -0.001272345194593072,
+          0.012208769097924232,
+          0.01601981930434704,
+          0.0058697848580777645,
+          -0.0035668755881488323,
+          -0.0023519834503531456,
+          -0.0011442728573456407,
+          -0.017963647842407227,
+          0.012610940262675285,
+          -0.017446570098400116,
+          0.00590329896658659,
+          0.004404732491821051,
+          0.016575198620557785,
+          0.017398692667484283,
+          -0.0053000422194600105,
+          0.020510731264948845,
+          0.0064634657464921474,
+          -0.0066214618273079395,
+          0.027175283059477806,
+          -0.00019285666348878294,
+          -0.006540069822221994,
+          -0.021334225311875343,
+          -0.003413667669519782,
+          0.019725540652871132,
+          -0.010236214846372604,
+          -0.0010856229346245527,
+          0.010734140872955322,
+          0.008479109965264797,
+          -2.2947548131924123e-05,
+          0.01765723153948784,
+          -0.009441448375582695,
+          0.0009192485013045371,
+          0.007315685972571373,
+          0.004950535949319601,
+          0.011232067830860615,
+          0.01174914464354515,
+          0.01740826852619648,
+          0.003947502002120018,
+          0.0023304384667426348,
+          -0.0037583857774734497,
+          0.004866750445216894,
+          0.011672540567815304,
+          0.03675078973174095,
+          0.007861489430069923,
+          0.007124175783246756,
+          -0.007176841143518686,
+          -0.004009742755442858,
+          0.01020748820155859,
+          -0.013415282592177391,
+          -0.013242923654615879,
+          0.015119723044335842,
+          0.005170772783458233,
+          0.032288603484630585,
+          -0.0121608916670084,
+          0.009996827691793442,
+          0.025585750117897987,
+          -0.008368991315364838,
+          -0.004658483434468508,
+          -0.004842811729758978,
+          0.0017319695325568318,
+          0.0024896312970668077,
+          0.007827975787222385,
+          0.0018037857953459024,
+          0.020587336272001266,
+          -0.009666472673416138,
+          -0.031771525740623474,
+          0.028075380250811577,
+          0.013491886667907238,
+          0.017398692667484283,
+          0.02164064161479473,
+          0.0079955467954278,
+          0.012677968479692936,
+          0.00971435010433197,
+          0.005893723573535681,
+          0.00416773883625865,
+          0.01609642431139946,
+          0.0073539880104362965,
+          -0.01767638325691223,
+          0.02054903469979763,
+          0.015483591705560684,
+          0.010753292590379715,
+          -0.02951170690357685,
+          -0.01116503868252039,
+          0.012936507351696491,
+          -0.01203641016036272,
+          -0.01061923522502184,
+          0.009929798543453217,
+          -0.011892776936292648,
+          0.005170772783458233,
+          0.015158024616539478,
+          0.010025554336607456,
+          0.0014123870059847832,
+          -0.02309611812233925,
+          -0.016489019617438316,
+          0.005496339872479439,
+          0.0021341408137232065,
+          0.012563062831759453,
+          0.0126396669074893,
+          0.017226334661245346,
+          -0.004950535949319601,
+          -0.00040037580765783787,
+          -0.017752986401319504,
+          -0.009580292738974094,
+          -0.007784885820001364,
+          0.02752000093460083,
+          -0.02164064161479473,
+          -0.0019330551149323583,
+          0.004105497617274523,
+          -0.02169809490442276,
+          0.0006218093913048506,
+          0.00840250588953495,
+          0.0025949617847800255,
+          -0.005357495043426752,
+          -0.008919582702219486,
+          0.003964259289205074,
+          -0.010753292590379715,
+          0.01414302084594965,
+          0.0011592345545068383,
+          -0.018729688599705696,
+          0.008368991315364838,
+          0.0013776757987216115,
+          -0.01212258916348219,
+          0.019629785791039467,
+          0.01852860301733017,
+          -0.007823187857866287,
+          0.005118107423186302,
+          0.041634295135736465,
+          0.0024992069229483604,
+          -0.022770551964640617,
+          -0.007545498199760914,
+          0.005323980934917927,
+          0.026102827861905098,
+          0.006066082511097193,
+          0.00822535902261734,
+          -0.0002223312621936202,
+          0.0030402229167521,
+          0.0017738622846081853,
+          -0.007406653370708227,
+          0.016211330890655518,
+          -0.00697575556114316,
+          0.004349673166871071,
+          0.014545192010700703,
+          -0.00877116248011589,
+          0.002016840735450387,
+          -0.01866265945136547,
+          0.0021233682055026293,
+          -0.025183578953146935,
+          -0.009546778164803982,
+          -0.003923563286662102,
+          -0.010264941491186619,
+          -0.008340264670550823,
+          0.0025303272996097803,
+          -0.011375700123608112,
+          -0.017398692667484283,
+          -0.010571357794106007,
+          0.01945742592215538,
+          -0.02066393941640854,
+          -0.0008234934648498893,
+          0.0065592210739851,
+          0.019189313054084778,
+          0.012151315808296204,
+          0.0029037718195468187,
+          0.0073539880104362965,
+          0.01561764907091856,
+          0.0017020460218191147,
+          -0.02188960462808609,
+          -0.027692360803484917,
+          0.02424517832696438,
+          0.001624245080165565,
+          0.0013681002892553806,
+          0.001211899914778769,
+          -0.011825748719274998,
+          -0.00985319446772337,
+          0.009048852138221264,
+          0.0006984134088270366,
+          -0.018183885142207146,
+          0.015474015846848488,
+          -0.002918135141953826,
+          -0.0017726654186844826,
+          -0.0031431596726179123,
+          0.014066416770219803,
+          0.014631371945142746,
+          0.0044095199555158615,
+          0.0055490052327513695,
+          0.005683062132447958,
+          0.012065136805176735,
+          0.010580932721495628,
+          -0.019878748804330826,
+          -0.025183578953146935,
+          0.019198887050151825,
+          -0.008459959179162979,
+          0.016852889209985733,
+          0.01382702961564064,
+          0.013635518960654736,
+          -0.028324345126748085,
+          -0.0037320530973374844,
+          -0.024436689913272858,
+          0.004550758749246597,
+          0.0008234934648498893,
+          -0.0019940990023314953,
+          -0.002559053711593151,
+          1.956245796463918e-05,
+          -0.021774698048830032,
+          0.006602310575544834,
+          0.025164427235722542,
+          -0.003890048945322633,
+          -0.011308671906590462,
+          0.02874566614627838,
+          -0.028918026015162468,
+          0.002788865938782692,
+          0.023076968267560005,
+          -0.004907446447759867,
+          0.07740838080644608,
+          0.008728072978556156,
+          0.0031670983880758286,
+          -0.015110147185623646,
+          0.0053000422194600105,
+          0.014487739652395248,
+          0.00834984052926302,
+          0.013817453756928444,
+          -0.00955156609416008,
+          -0.015158024616539478,
+          -0.008282812312245369,
+          0.01187362615019083,
+          0.006554433144629002,
+          -0.0031216146890074015,
+          0.002810410689562559,
+          0.0005098956753499806,
+          -0.010963953100144863,
+          -0.011739569716155529,
+          0.024111121892929077,
+          -0.025777259841561317,
+          -0.00871849711984396,
+          -0.028132833540439606,
+          -0.01366424560546875,
+          -0.00023789145052433014,
+          0.012754572555422783,
+          0.011011830531060696,
+          -0.022119417786598206,
+          -0.013606793247163296,
+          -0.020510731264948845,
+          -0.0020635214168578386,
+          0.018576480448246002,
+          0.0015536256833001971,
+          -0.000921043916605413,
+          -0.002472874242812395,
+          -0.001449492061510682,
+          0.009130244143307209,
+          -0.01295565813779831,
+          0.014344106428325176,
+          -0.006200139410793781,
+          0.0027649272233247757,
+          0.0038014755118638277,
+          0.00733004929497838,
+          0.01343443337827921,
+          -0.0034783021546900272,
+          0.006324620917439461,
+          -0.010753292590379715,
+          0.02361319586634636,
+          -0.01545486506074667,
+          -0.0031240086536854506,
+          -0.0015739736845716834,
+          0.008440807461738586,
+          -0.007325261365622282,
+          0.006760306656360626,
+          -0.00854135025292635,
+          -0.009269089438021183,
+          0.014468587934970856,
+          0.025566598400473595,
+          -0.02399621531367302,
+          0.004878719802945852,
+          -0.022770551964640617,
+          -0.0006918302387930453,
+          -0.007325261365622282,
+          -0.006425164174288511,
+          -0.002341211074963212,
+          -0.0401788204908371,
+          -0.005319193005561829,
+          0.012314099818468094,
+          0.0026284761261194944,
+          -0.01634538732469082,
+          0.009111093357205391,
+          -0.002007265342399478,
+          0.003736840793862939,
+          -0.03994900733232498,
+          0.017130577936768532,
+          0.025068672373890877,
+          -0.015129297971725464,
+          0.007253445219248533,
+          0.005845845676958561,
+          -0.007698706351220608,
+          0.0005371259758248925,
+          0.004426277242600918,
+          0.002757745562121272,
+          -0.02880311943590641,
+          0.020434128120541573,
+          0.018375394865870476,
+          0.007579012308269739,
+          0.004888295195996761,
+          -0.008929158560931683,
+          0.002621294464915991,
+          0.00989149697124958,
+          0.005544217303395271,
+          0.02405366860330105,
+          0.0031431596726179123,
+          -0.0006415588431991637,
+          0.0016374114202335477,
+          -0.0037224777042865753,
+          0.009767015464603901,
+          -0.005970327649265528,
+          0.005740515422075987,
+          -0.017877468839287758,
+          -0.0043329158797860146,
+          -0.0018349061720073223,
+          -0.004172526299953461,
+          -0.007794461213052273,
+          -0.001229255460202694,
+          -0.011844899505376816,
+          0.021410830318927765,
+          0.007377926725894213,
+          0.014210049994289875,
+          -0.01603897102177143,
+          0.0032556718215346336,
+          -0.021334225311875343,
+          -0.0036099653225392103,
+          -0.008311538025736809,
+          0.005946388468146324,
+          0.04404732212424278,
+          0.010006402619183064,
+          0.016661379486322403,
+          0.00033514268579892814,
+          0.020070258527994156,
+          -0.0066884905099868774,
+          -0.0055585806258022785,
+          -0.00047518443898297846,
+          -0.003830202156677842,
+          -0.0032867921981960535,
+          -0.0016050940612331033,
+          -0.008570076897740364,
+          -0.008321113884449005,
+          0.006678914651274681,
+          0.007890216074883938,
+          0.00667412718757987,
+          -0.0214491318911314,
+          0.01830836571753025,
+          -0.0007079889182932675,
+          -0.006037355866283178,
+          0.019763842225074768,
+          -0.0015620043268427253,
+          -0.002276576356962323,
+          -0.010418149642646313,
+          0.008258873596787453,
+          -0.0024082395248115063,
+          0.0469965785741806,
+          -0.009034489281475544,
+          0.02188960462808609,
+          -0.006439527031034231,
+          0.0023771191481500864,
+          0.003736840793862939,
+          0.0009264301043003798,
+          -0.006745943333953619,
+          -0.0021125958301126957,
+          -0.015541044995188713,
+          0.0056016705930233,
+          -0.015215477906167507,
+          0.012342826463282108,
+          -0.016795435920357704,
+          0.00372726540081203,
+          0.02361319586634636,
+          -0.0050558666698634624,
+          -0.00902012549340725,
+          0.0036913573276251554,
+          0.006587947718799114,
+          -0.0028271679766476154,
+          -0.02106611058115959,
+          0.013635518960654736,
+          -0.00193664594553411,
+          0.017427420243620872,
+          -0.007516771554946899,
+          -0.010963953100144863,
+          0.021104414016008377,
+          0.010092582553625107,
+          0.025049520656466484,
+          0.017312513664364815,
+          0.018863745033740997,
+          -0.001241224817931652,
+          0.014966514892876148,
+          -0.020721392706036568,
+          0.002918135141953826,
+          -0.012668393552303314,
+          -0.01989790052175522,
+          -0.0028175923507660627,
+          -0.004445428028702736,
+          0.01116503868252039,
+          -0.008546138182282448,
+          -0.010954378172755241,
+          0.007899791933596134,
+          -0.023345081135630608,
+          0.00324849016033113,
+          -0.008301963098347187,
+          0.017839165404438972,
+          0.0035477245692163706,
+          -0.012658817693591118,
+          -0.008795101195573807,
+          -0.006841698195785284,
+          -0.01174914464354515,
+          -0.020568184554576874,
+          0.03008623793721199,
+          0.01636453904211521,
+          0.015435714274644852,
+          -0.005678274668753147,
+          -0.002378316130489111,
+          -0.008689770475029945,
+          -0.0036267226096242666,
+          0.0008923173882067204,
+          -0.02822858840227127
+        ],
+        "title": "A Switch for Attention"
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "44ddb1e4-8508-4bcf-9d46-226ebb75db2f",
+      "properties": {
+        "page_content": "B Preventing Token Dropping with No-Token-Left-Behind 29\n\n",
+        "keyphrases": [
+          "Preventing Token Dropping",
+          "No-Token-Left-Behind"
+        ],
+        "embedding": [
+          0.025537114590406418,
+          -0.019454315304756165,
+          -0.016811933368444443,
+          0.022433646023273468,
+          -0.0004264498711563647,
+          -0.0028263733256608248,
+          -0.020252350717782974,
+          0.05167718976736069,
+          -0.02958049066364765,
+          0.00320987356826663,
+          -0.031212029978632927,
+          -0.00974489189684391,
+          0.01094194408506155,
+          -0.049123477190732956,
+          -0.010197111405432224,
+          0.038660354912281036,
+          -0.002062698360532522,
+          0.016102569177746773,
+          0.016013899818062782,
+          -0.003059133654460311,
+          0.010631597600877285,
+          0.013460187241435051,
+          -0.01815972663462162,
+          0.039192378520965576,
+          -0.005244862288236618,
+          0.027842549607157707,
+          -0.010347851552069187,
+          0.005479839164763689,
+          -0.04174608737230301,
+          0.018115390092134476,
+          0.008871487341821194,
+          0.009558684192597866,
+          -0.0016282127471640706,
+          -0.052067339420318604,
+          -0.011119285598397255,
+          0.012387273833155632,
+          0.04429979994893074,
+          0.015579413622617722,
+          -0.07313545793294907,
+          -0.028835657984018326,
+          -0.0168474018573761,
+          0.003110119141638279,
+          -0.05912551283836365,
+          0.0050763883627951145,
+          0.019170571118593216,
+          0.025129230692982674,
+          -0.004646336194127798,
+          -0.01648385263979435,
+          -0.0052049607038497925,
+          0.0036554429680109024,
+          0.017237553372979164,
+          -0.042207177728414536,
+          0.011943921446800232,
+          -0.02472134679555893,
+          0.015464141964912415,
+          0.015251332893967628,
+          0.0375608392059803,
+          0.018283864483237267,
+          0.007479359861463308,
+          -0.01447989884763956,
+          0.01440009567886591,
+          0.012804025784134865,
+          0.034670181572437286,
+          0.004735006485134363,
+          -0.0008046851144172251,
+          -0.008299562148749828,
+          -0.011722245253622532,
+          -0.023072073236107826,
+          0.017343956977128983,
+          -0.021405067294836044,
+          -0.050719548016786575,
+          0.02541297674179077,
+          -0.003491402370855212,
+          -0.006645856890827417,
+          0.023781437426805496,
+          0.010489724576473236,
+          0.07838475704193115,
+          0.05848708748817444,
+          0.0006278982618823647,
+          -0.000581900414545089,
+          -0.005510874092578888,
+          0.021937090903520584,
+          -0.001937451190315187,
+          0.014329158701002598,
+          0.041426874697208405,
+          -0.00011402476957300678,
+          -0.046995386481285095,
+          -0.04749194160103798,
+          0.04476088657975197,
+          -0.021334132179617882,
+          -0.0384475439786911,
+          -0.02028781920671463,
+          -0.02064250037074089,
+          -0.04121406748890877,
+          -0.00320987356826663,
+          0.0588417686522007,
+          0.017228685319423676,
+          -0.019383380189538002,
+          -0.022256305441260338,
+          0.005936492700129747,
+          -0.025022825226187706,
+          -0.0026623329613357782,
+          -0.05469198897480965,
+          0.01869175024330616,
+          0.03926331549882889,
+          0.012874961830675602,
+          -0.0023054340854287148,
+          0.009425678290426731,
+          -0.022185368463397026,
+          0.0798034816980362,
+          -0.018815888091921806,
+          0.009212869219481945,
+          -0.020411957055330276,
+          0.04468994960188866,
+          -0.031247496604919434,
+          -0.016785332933068275,
+          -0.00320987356826663,
+          -0.034067220985889435,
+          -0.036638665944337845,
+          0.011669042520225048,
+          0.0056571802124381065,
+          -0.015269066207110882,
+          -0.012617817148566246,
+          0.013229643926024437,
+          -0.027186386287212372,
+          0.008844885975122452,
+          0.027150917798280716,
+          0.019738061353564262,
+          -0.021405067294836044,
+          -0.01984446682035923,
+          -0.014072014018893242,
+          -0.013149840757250786,
+          -0.058628957718610764,
+          0.012724222615361214,
+          -0.04224264249205589,
+          0.019879935309290886,
+          0.019436581060290337,
+          0.011988257057964802,
+          -0.026211010292172432,
+          0.04163968563079834,
+          -0.022912466898560524,
+          0.007377388887107372,
+          0.028392305597662926,
+          0.04745647311210632,
+          -0.012431609444320202,
+          0.009913366287946701,
+          -0.008441435173153877,
+          0.015747888013720512,
+          -0.018620813265442848,
+          0.021511472761631012,
+          0.02498735673725605,
+          0.004351506475359201,
+          0.014630638994276524,
+          0.03674507141113281,
+          0.005346833262592554,
+          0.01222766749560833,
+          -0.007204481400549412,
+          -0.02152920700609684,
+          0.007794140372425318,
+          -0.007452758494764566,
+          0.016155771911144257,
+          0.007075909059494734,
+          0.03532634302973747,
+          -0.00984242931008339,
+          0.024029714986681938,
+          0.006654724013060331,
+          0.03472338244318962,
+          0.013265112414956093,
+          -0.08313749730587006,
+          0.02347995899617672,
+          -0.0027244023513048887,
+          0.031034687533974648,
+          -0.010223712772130966,
+          0.03021891973912716,
+          -0.007612365763634443,
+          -0.020341021940112114,
+          -0.0064995503053069115,
+          -0.00956755131483078,
+          0.016599124297499657,
+          0.011411897838115692,
+          -0.011713378131389618,
+          0.03709975257515907,
+          -0.009053261950612068,
+          -0.024650409817695618,
+          0.022522317245602608,
+          -0.003440416883677244,
+          0.019206037744879723,
+          -0.0175301656126976,
+          -0.008538972586393356,
+          0.009434545412659645,
+          -0.02516469918191433,
+          0.018461205065250397,
+          0.01779617741703987,
+          -0.00274656992405653,
+          -0.01745922863483429,
+          0.0034071654081344604,
+          -0.014905516989529133,
+          -0.025022825226187706,
+          0.01737942546606064,
+          -0.00849907100200653,
+          -0.0026157808024436235,
+          0.0168474018573761,
+          -0.029917439445853233,
+          -0.00881828460842371,
+          0.0021292013116180897,
+          0.021121321246027946,
+          0.0020128211472183466,
+          0.04458354786038399,
+          0.007740938104689121,
+          0.01790258102118969,
+          -0.0017235336126759648,
+          0.0003020340227521956,
+          -0.007173446472734213,
+          0.012502545490860939,
+          -0.019418848678469658,
+          -0.01103061530739069,
+          -0.03181498870253563,
+          0.018053321167826653,
+          -0.022185368463397026,
+          0.039192378520965576,
+          0.03630171716213226,
+          -0.014976453967392445,
+          -0.022717392072081566,
+          -0.0026645497418940067,
+          -0.003846084466204047,
+          -0.017388291656970978,
+          0.018408004194498062,
+          0.025217900052666664,
+          -0.0026157808024436235,
+          -0.00030425077420659363,
+          0.010826672427356243,
+          0.02401198074221611,
+          -0.05298951268196106,
+          0.0066192555241286755,
+          -0.013309448026120663,
+          -0.009700557217001915,
+          -0.034084953367710114,
+          0.010933076962828636,
+          0.019791264086961746,
+          0.01067593228071928,
+          0.013814869336783886,
+          0.044654481112957,
+          -0.07682415097951889,
+          -0.0029549456667155027,
+          -0.03552141785621643,
+          0.06685758382081985,
+          -0.026636630296707153,
+          0.03678053990006447,
+          -0.03527313843369484,
+          -0.0006434156093746424,
+          -0.018141992390155792,
+          0.017672037705779076,
+          -0.010968545451760292,
+          0.007763105444610119,
+          0.02481001615524292,
+          0.06412653625011444,
+          -0.006934036035090685,
+          -0.025093762204051018,
+          0.001193727133795619,
+          -0.02489868737757206,
+          0.020695703104138374,
+          0.03848301246762276,
+          0.025483911857008934,
+          -0.01815972663462162,
+          0.008995626121759415,
+          -0.009922233410179615,
+          0.016155771911144257,
+          -0.014178418554365635,
+          0.025661254301667213,
+          -0.021688813343644142,
+          0.027062248438596725,
+          0.016368580982089043,
+          -0.03748990222811699,
+          -0.03177952021360397,
+          0.0035667724441736937,
+          0.003726379247382283,
+          -0.00783404242247343,
+          0.0015772271435707808,
+          0.005289197433739901,
+          0.034067220985889435,
+          -0.014870049431920052,
+          0.00894242338836193,
+          0.003912587650120258,
+          -0.033535197377204895,
+          -0.007878377102315426,
+          -0.035237669944763184,
+          -0.0650487095117569,
+          0.03780911862850189,
+          -0.08483996987342834,
+          -0.012910430319607258,
+          -0.017831645905971527,
+          0.01191732008010149,
+          -0.011057215742766857,
+          0.009039961732923985,
+          0.06430387496948242,
+          0.01702474243938923,
+          0.010471990332007408,
+          0.0023896710481494665,
+          -0.04312935099005699,
+          -0.0010507458355277777,
+          -0.007581330835819244,
+          -0.01966712437570095,
+          0.001189293572679162,
+          -0.029988376423716545,
+          -0.03589383512735367,
+          -0.017831645905971527,
+          0.0019474267028272152,
+          -0.027682941406965256,
+          -0.05171265825629234,
+          -0.002861841581761837,
+          0.020802108570933342,
+          -0.02383464016020298,
+          -0.02755880355834961,
+          -0.0328080989420414,
+          -0.040185488760471344,
+          -0.006588221061974764,
+          0.005807920359075069,
+          0.04075298085808754,
+          -0.023072073236107826,
+          -0.02995290793478489,
+          0.02037649042904377,
+          -0.017964651808142662,
+          -0.04057563841342926,
+          0.011243424378335476,
+          0.006925168912857771,
+          -0.0393342487514019,
+          0.027186386287212372,
+          0.04330668970942497,
+          -0.025749923661351204,
+          -0.00035828439285978675,
+          -0.014533101581037045,
+          -0.01913510262966156,
+          0.000581900414545089,
+          -0.029740098863840103,
+          -0.02709771692752838,
+          -0.03126523271203041,
+          0.0014054280472919345,
+          -0.018674015998840332,
+          -0.0025248934980481863,
+          0.007563597057014704,
+          -0.020500628277659416,
+          -0.03731256350874901,
+          0.009815828874707222,
+          0.0009986519580706954,
+          0.0007620124379172921,
+          -0.016253309324383736,
+          0.01440009567886591,
+          -0.021068120375275612,
+          -0.005524174310266972,
+          -0.002821939764544368,
+          0.0011904019629582763,
+          0.01320304349064827,
+          0.05518854409456253,
+          -0.026636630296707153,
+          0.0665738433599472,
+          -0.009141932241618633,
+          -0.01913510262966156,
+          0.008920256048440933,
+          0.006428613793104887,
+          -0.03731256350874901,
+          0.006348810624331236,
+          -0.02605140395462513,
+          0.009532082825899124,
+          0.004101012367755175,
+          -0.0023497692309319973,
+          0.00025395795819349587,
+          -0.012662152759730816,
+          0.01067593228071928,
+          -0.0247036125510931,
+          -0.014905516989529133,
+          -0.018727216869592667,
+          -0.011819782666862011,
+          0.034404169768095016,
+          -0.012724222615361214,
+          0.02417158894240856,
+          -0.027789346873760223,
+          0.03741896525025368,
+          0.015623748302459717,
+          -0.018141992390155792,
+          0.01977352984249592,
+          -0.022557783871889114,
+          -0.021688813343644142,
+          0.024419866502285004,
+          -0.01594296284019947,
+          0.02170654758810997,
+          0.022256305441260338,
+          0.01904643140733242,
+          0.0366741344332695,
+          -0.012600083835422993,
+          0.032116468995809555,
+          -0.03559235483407974,
+          0.02234497480094433,
+          -0.07061721384525299,
+          0.006096099503338337,
+          0.017317356541752815,
+          0.036886945366859436,
+          -0.02835683710873127,
+          0.00717788003385067,
+          0.021227726712822914,
+          -0.013398118317127228,
+          -0.007408423349261284,
+          0.010746869258582592,
+          0.00956755131483078,
+          -0.034510571509599686,
+          -0.012289736419916153,
+          0.028764722868800163,
+          -0.050896890461444855,
+          -0.013637528754770756,
+          -0.0027753878384828568,
+          0.007483793422579765,
+          0.0027776046190410852,
+          -0.047704748809337616,
+          -0.004488945938646793,
+          -0.02878245711326599,
+          -0.007918278686702251,
+          0.010152776725590229,
+          0.02427799254655838,
+          0.021671079099178314,
+          -0.01966712437570095,
+          -0.03284356743097305,
+          0.006645856890827417,
+          0.029545024037361145,
+          0.009993169456720352,
+          0.026441553607583046,
+          0.07192953675985336,
+          -0.036195311695337296,
+          -0.01502078864723444,
+          0.009833562187850475,
+          -0.00788724422454834,
+          -0.016359714791178703,
+          0.03475885093212128,
+          0.03279036656022072,
+          -0.03692241013050079,
+          -0.011057215742766857,
+          -0.020075010135769844,
+          -0.029527289792895317,
+          -0.05774225294589996,
+          0.0027709542773663998,
+          -0.0008872595499269664,
+          -0.02081984281539917,
+          0.00427170330658555,
+          0.014187285676598549,
+          -0.055685099214315414,
+          0.005280330311506987,
+          -0.016528187319636345,
+          -0.021458270028233528,
+          -0.028587380424141884,
+          -0.03569876030087471,
+          -0.013176442123949528,
+          0.008973458781838417,
+          -0.037702713161706924,
+          0.007523695006966591,
+          -0.022575518116354942,
+          0.030502665787935257,
+          0.008574441075325012,
+          -0.008303996175527573,
+          0.025217900052666664,
+          -0.021156789734959602,
+          0.013362649828195572,
+          -0.006867533084005117,
+          -0.04146234318614006,
+          -0.003145587397739291,
+          0.0023608531337231398,
+          -0.011021748185157776,
+          0.0055463421158492565,
+          0.023515427485108376,
+          -0.02791348472237587,
+          0.029970642179250717,
+          0.0017833862220868468,
+          -0.0061404346488416195,
+          0.0061315675266087055,
+          -0.01710454747080803,
+          -0.011748846620321274,
+          -0.021227726712822914,
+          0.0023165177553892136,
+          -0.0034382001031190157,
+          -0.04011455178260803,
+          -0.04394511878490448,
+          -0.048520516604185104,
+          -0.007541429251432419,
+          -0.014001077972352505,
+          -0.00903552770614624,
+          -0.0083217304199934,
+          -0.005462104920297861,
+          -0.004034509416669607,
+          -0.0031123359221965075,
+          -0.007027139887213707,
+          -0.010294649749994278,
+          -0.009372475557029247,
+          -0.004280570428818464,
+          0.02294793538749218,
+          -0.019808998331427574,
+          -0.019188303500413895,
+          -0.016599124297499657,
+          0.00631334213539958,
+          -0.024508535861968994,
+          -0.03545048087835312,
+          0.003611107589676976,
+          0.038412075489759445,
+          0.03373027220368385,
+          0.011465100571513176,
+          -0.018026720732450485,
+          0.0026290814857929945,
+          0.019596189260482788,
+          0.022699657827615738,
+          -0.05909004807472229,
+          -0.01332718227058649,
+          -0.0008351656142622232,
+          -0.02410065196454525,
+          -0.0027798213995993137,
+          -0.015384337864816189,
+          0.0010551793966442347,
+          -0.006526151672005653,
+          -0.0005539137637242675,
+          -0.007155712228268385,
+          -0.020802108570933342,
+          -6.234647298697382e-05,
+          0.0027111016679555178,
+          -0.0008235276327468455,
+          -0.002099274890497327,
+          0.016616858541965485,
+          0.019649391993880272,
+          -0.0164040494710207,
+          0.00757246371358633,
+          -0.011216823011636734,
+          0.018567610532045364,
+          0.03137163445353508,
+          -0.0016437300946563482,
+          0.020713437348604202,
+          -0.025306571274995804,
+          0.003990174271166325,
+          0.022735126316547394,
+          -0.034670181572437286,
+          -0.027168652042746544,
+          -0.0014575219247490168,
+          0.014453297480940819,
+          -0.0006240188959054649,
+          0.01984446682035923,
+          -0.06370091438293457,
+          -0.00586998974904418,
+          -0.025058293715119362,
+          0.00200284575112164,
+          -0.016439517959952354,
+          0.035503681749105453,
+          -0.003141153836622834,
+          0.012245400808751583,
+          0.03259528800845146,
+          0.006415313575416803,
+          0.016714395955204964,
+          -0.01779617741703987,
+          0.014408962801098824,
+          0.0354682132601738,
+          -0.047101788222789764,
+          -0.011881851591169834,
+          -0.006645856890827417,
+          -0.044512610882520676,
+          0.013371516950428486,
+          0.011296626180410385,
+          0.019702592864632607,
+          0.027594272047281265,
+          -0.024384398013353348,
+          0.03165538236498833,
+          -0.007324186619371176,
+          -0.009363608434796333,
+          0.026246478781104088,
+          -0.004805942997336388,
+          0.019472049549221992,
+          0.006362111307680607,
+          -0.0072133480571210384,
+          0.010569527745246887,
+          0.012245400808751583,
+          -0.011216823011636734,
+          0.006876400206238031,
+          -0.029296746477484703,
+          -8.721578342374414e-05,
+          0.012316337786614895,
+          -0.010516325943171978,
+          0.002083757659420371,
+          -0.03390761464834213,
+          0.025448445230722427,
+          -0.013140973635017872,
+          -0.01267101988196373,
+          0.03461697697639465,
+          0.004225151147693396,
+          0.016599124297499657,
+          0.03447510674595833,
+          0.0026889340952038765,
+          -0.009000059217214584,
+          0.006597088184207678,
+          -0.001766760484315455,
+          0.007341920398175716,
+          0.018993228673934937,
+          0.00934587512165308,
+          -0.023550894111394882,
+          -0.02374597080051899,
+          -0.013699597679078579,
+          0.008836018852889538,
+          0.01001090370118618,
+          0.0020848659332841635,
+          0.003216523677110672,
+          0.008707446977496147,
+          0.009159666486084461,
+          0.015792222693562508,
+          -0.013353782705962658,
+          -0.0031256363727152348,
+          0.01763656921684742,
+          -0.03773818165063858,
+          0.0036465758457779884,
+          -0.010285782627761364,
+          0.0010391079122200608,
+          0.00934587512165308,
+          -0.00047992929467000067,
+          0.0473145991563797,
+          0.005643879529088736,
+          0.008405966684222221,
+          0.010436521843075752,
+          0.019365645945072174,
+          -0.006774429231882095,
+          -0.017964651808142662,
+          0.005967527162283659,
+          0.026477022096514702,
+          -0.029456352815032005,
+          0.03766724467277527,
+          -0.00445569446310401,
+          -0.001095081097446382,
+          0.002797555411234498,
+          -0.006539452355355024,
+          -0.018833622336387634,
+          -0.00405446020886302,
+          -0.03509579971432686,
+          -0.009833562187850475,
+          0.01098627969622612,
+          0.027665207162499428,
+          -0.023941045626997948,
+          -0.02622874453663826,
+          0.03138937056064606,
+          0.023107541725039482,
+          -0.0005497573292814195,
+          0.030963752418756485,
+          -0.028232699260115623,
+          0.020624767988920212,
+          0.026920374482870102,
+          0.018815888091921806,
+          0.03748990222811699,
+          -0.030981486663222313,
+          -0.011713378131389618,
+          0.022167634218931198,
+          0.038944099098443985,
+          0.038837697356939316,
+          0.0012380623957142234,
+          -0.03748990222811699,
+          0.02347995899617672,
+          0.03422682732343674,
+          0.03426229581236839,
+          0.00675226142629981,
+          -0.02339128777384758,
+          -0.009638487361371517,
+          0.013469054363667965,
+          0.0005458780215121806,
+          0.010090706869959831,
+          -0.010684799402952194,
+          0.008295129053294659,
+          -0.013593193143606186,
+          -0.016625726595520973,
+          -0.0005375651526264846,
+          -0.018815888091921806,
+          -0.033801209181547165,
+          -0.030857346951961517,
+          0.016944939270615578,
+          -0.014417828992009163,
+          -0.03470565006136894,
+          -0.019879935309290886,
+          -0.013185309246182442,
+          -0.005847821943461895,
+          -0.04834317788481712,
+          -0.02800215594470501,
+          -0.020429691299796104,
+          0.027310525998473167,
+          -0.01879815384745598,
+          -0.007865076884627342,
+          -0.02727505750954151,
+          -0.003212090115994215,
+          0.0007664459408260882,
+          0.0384475439786911,
+          -0.013894673436880112,
+          -0.0001251778594451025,
+          -0.02488095313310623,
+          0.013238511048257351,
+          -0.02984650246798992,
+          0.0065039838664233685,
+          -0.015827691182494164,
+          -0.02638835273683071,
+          0.011305493302643299,
+          -0.011491701938211918,
+          -0.002347552450373769,
+          -0.011172487400472164,
+          0.01377053465694189,
+          -0.005151758436113596,
+          -0.010658198967576027,
+          0.007723203860223293,
+          0.0014763644430786371,
+          -0.008552273735404015,
+          -0.02259325236082077,
+          0.04962003231048584,
+          0.0036598765291273594,
+          -0.0062645734287798405,
+          -0.0013056736206635833,
+          -0.012387273833155632,
+          -0.006291174795478582,
+          -0.008303996175527573,
+          -0.019099634140729904,
+          0.002118117408826947,
+          0.017778443172574043,
+          0.026175541803240776,
+          0.026884907856583595,
+          -0.001146066701039672,
+          0.015419806353747845,
+          -0.00015406505553983152,
+          -0.04600227624177933,
+          -0.022717392072081566,
+          -0.03169085085391998,
+          -0.03169085085391998,
+          0.03197459504008293,
+          -0.019649391993880272,
+          -0.0030480497516691685,
+          -0.011872985400259495,
+          0.02463267557322979,
+          0.02913713827729225,
+          0.0003538508608471602,
+          -0.0030347490683197975,
+          -0.0001627935707801953,
+          0.0027842549607157707,
+          -0.01045425608754158,
+          0.0033827810548245907,
+          -0.016058234497904778,
+          0.024845484644174576,
+          -5.153975143912248e-05,
+          0.01727301999926567,
+          0.032471150159835815,
+          -0.02913713827729225,
+          0.033801209181547165,
+          0.010188245214521885,
+          0.008729614317417145,
+          0.015925228595733643,
+          -0.008135521784424782,
+          -0.0011011771857738495,
+          0.035822898149490356,
+          0.02073117159307003,
+          -0.031673114746809006,
+          0.01457743626087904,
+          -0.02372823655605316,
+          -0.026884907856583595,
+          0.020890777930617332,
+          -0.009806961752474308,
+          -0.028924329206347466,
+          0.029704630374908447,
+          0.020181413739919662,
+          0.026104606688022614,
+          0.032985441386699677,
+          -0.015818823128938675,
+          -0.006654724013060331,
+          -0.002751003485172987,
+          0.0004120409139432013,
+          -0.006344377063214779,
+          0.012600083835422993,
+          0.004087711684405804,
+          -0.0014054280472919345,
+          -0.001630429527722299,
+          0.018478939309716225,
+          0.0006295607890933752,
+          0.011598106473684311,
+          0.006277874112129211,
+          -0.00751039432361722,
+          -0.029030734673142433,
+          -0.008051284588873386,
+          0.011633574962615967,
+          0.03731256350874901,
+          -0.034865256398916245,
+          0.00859217531979084,
+          -0.022699657827615738,
+          -0.019188303500413895,
+          0.01737942546606064,
+          -0.03271942958235741,
+          0.004770474974066019,
+          0.016776464879512787,
+          0.0006539452006109059,
+          0.013903540559113026,
+          0.009425678290426731,
+          0.011598106473684311,
+          0.012963633053004742,
+          0.0057724518701434135,
+          0.002511593047529459,
+          0.015357737429440022,
+          0.012910430319607258,
+          0.07249703258275986,
+          -0.025040559470653534,
+          0.00045665327343158424,
+          0.03802192583680153,
+          -0.027168652042746544,
+          -0.003965789917856455,
+          0.03396081551909447,
+          -0.006468515843153,
+          -0.02401198074221611,
+          0.029562756419181824,
+          -0.01860307902097702,
+          -0.02816176228225231,
+          -0.016794199123978615,
+          -0.030272122472524643,
+          -0.013983343727886677,
+          0.012165597639977932,
+          0.01762770302593708,
+          -0.011145886965095997,
+          0.025005090981721878,
+          0.0008401533705182374,
+          -0.019915401935577393,
+          -0.008787250146269798,
+          0.02259325236082077,
+          -0.012316337786614895,
+          -0.006694625597447157,
+          0.0025780959986150265,
+          0.012520279735326767,
+          0.00962075311690569,
+          0.015340003184974194,
+          -0.041604217141866684,
+          -0.010303516872227192,
+          0.011119285598397255,
+          -0.0031034687999635935,
+          0.0030037143733352423,
+          0.0276474729180336,
+          0.01717548258602619,
+          0.011828649789094925,
+          -0.03163764625787735,
+          0.027168652042746544,
+          0.01853214204311371,
+          -0.023409022018313408,
+          0.03827020525932312,
+          -0.03936971724033356,
+          -0.025005090981721878,
+          -0.006676891352981329,
+          0.006109400186687708,
+          0.0049566831439733505,
+          -0.00631334213539958,
+          -0.010064106434583664,
+          -0.024916421622037888,
+          -0.03279036656022072,
+          -0.019010962918400764,
+          -0.0005746959359385073,
+          0.015295667573809624,
+          0.011802048422396183,
+          0.005235995166003704,
+          0.008286261931061745,
+          0.04057563841342926,
+          -0.0033606134820729494,
+          -0.002994847483932972,
+          -0.031939126551151276,
+          -0.012289736419916153,
+          0.03165538236498833,
+          0.027044514194130898,
+          -0.005945359356701374,
+          0.007204481400549412,
+          -0.005351266823709011,
+          0.00907986331731081,
+          0.0013245161389932036,
+          -0.013486788608133793,
+          0.026264213025569916,
+          0.04632148891687393,
+          -0.0022533400915563107,
+          -0.03279036656022072,
+          -0.006167036015540361,
+          -0.026884907856583595,
+          -0.01566808484494686,
+          0.012733088806271553,
+          -0.026689831167459488,
+          0.0027088848873972893,
+          -0.011988257057964802,
+          -0.04366137459874153,
+          -0.008756215684115887,
+          -0.009443412534892559,
+          0.01753903180360794,
+          -0.00571481604129076,
+          -0.0062468391843140125,
+          -0.0061227004043757915,
+          -0.0053335330449044704,
+          0.00350913661532104,
+          -0.0031832722015678883,
+          -0.03341105952858925,
+          0.020677968859672546,
+          -0.01869175024330616,
+          -0.023089807480573654,
+          -0.02002180740237236,
+          -0.003923671320080757,
+          0.002082649152725935,
+          -0.005572943482547998,
+          -0.010179378092288971,
+          0.014870049431920052,
+          -0.019383380189538002,
+          -0.024526270106434822,
+          -0.03837660700082779,
+          0.012014857493340969,
+          0.010046372190117836,
+          -0.011553770862519741,
+          -0.01186411827802658,
+          -0.0016160205705091357,
+          -0.004899047315120697,
+          0.01568581722676754,
+          -0.02346222475171089,
+          0.0049522495828568935,
+          0.029367681592702866,
+          0.018008986487984657,
+          -0.026477022096514702,
+          0.012050325982272625,
+          -0.01897549442946911,
+          0.02913713827729225,
+          -0.027079982683062553,
+          -0.011819782666862011,
+          0.02443760074675083,
+          0.0022965669631958008,
+          -0.0027044513262808323,
+          -0.025803126394748688,
+          0.015393204987049103,
+          -0.002391887828707695,
+          0.019915401935577393,
+          -0.021511472761631012,
+          0.02773614414036274,
+          0.004276136867702007,
+          -0.030360791832208633,
+          0.015047390013933182,
+          0.026104606688022614,
+          -0.011367563158273697,
+          0.02365729957818985,
+          0.01957845501601696,
+          -0.004899047315120697,
+          -0.010729135014116764,
+          0.0175301656126976,
+          -0.008188724517822266,
+          0.023799171671271324,
+          -0.0407884456217289,
+          0.0064951167441904545,
+          -0.044867292046546936,
+          -0.036106642335653305,
+          0.003602240700274706,
+          -0.006237972527742386,
+          -0.020979449152946472,
+          -0.040220957249403,
+          0.003229824360460043,
+          0.009922233410179615,
+          0.01842573843896389,
+          0.027842549607157707,
+          -0.0218838881701231,
+          0.02878245711326599,
+          0.0034226826392114162,
+          -0.027487866580486298,
+          0.0070936428382992744,
+          -0.0022810494992882013,
+          -0.032205138355493546,
+          -0.009239469654858112,
+          -0.022025762125849724,
+          -0.005586243700236082,
+          -0.049832843244075775,
+          0.013052303344011307,
+          -0.011074949987232685,
+          0.015934094786643982,
+          -0.017521297559142113,
+          -0.02833910472691059,
+          -0.0022489065304398537,
+          0.037773650139570236,
+          0.003945839125663042,
+          -0.036816008388996124,
+          -0.02924354374408722,
+          0.019010962918400764,
+          -0.02984650246798992,
+          -0.007071475498378277,
+          0.005936492700129747,
+          0.020252350717782974,
+          -0.010773470625281334,
+          -0.020411957055330276,
+          -0.010037505067884922,
+          -0.006065064575523138,
+          -0.00823305919766426,
+          0.013070037588477135,
+          0.04163968563079834,
+          -0.004659636877477169,
+          0.010356718674302101,
+          0.0027155352290719748,
+          0.04603774473071098,
+          -0.001809987355954945,
+          -0.018656281754374504,
+          0.028587380424141884,
+          -0.0014364627422764897,
+          -0.0059143248945474625,
+          0.016164638102054596,
+          -0.013238511048257351,
+          0.002917260630056262,
+          -0.0300238449126482,
+          0.028906594961881638,
+          0.011970522813498974,
+          -0.026867173612117767,
+          -0.00939907692372799,
+          -0.02782481536269188,
+          -0.02791348472237587,
+          -0.05022299289703369,
+          -0.011092684231698513,
+          -0.01700701005756855,
+          0.04518650472164154,
+          -0.05593337491154671,
+          -0.01728188805282116,
+          0.006198070477694273,
+          -0.006920735351741314,
+          -0.024224791675806046,
+          0.006645856890827417,
+          -0.01320304349064827,
+          -0.022185368463397026,
+          0.011145886965095997,
+          -0.033269185572862625,
+          -0.003879335941746831,
+          -0.015907494351267815,
+          -0.007315319497138262,
+          0.0017301838379353285,
+          -0.0014353543519973755,
+          0.06015409156680107,
+          0.009957700967788696,
+          0.02862284891307354,
+          -0.009443412534892559,
+          0.009780360385775566,
+          0.026104606688022614,
+          0.020766640082001686,
+          0.023444490507245064,
+          0.009904499165713787,
+          -0.0047305733896791935,
+          -0.019702592864632607,
+          0.0024184889625757933,
+          0.010241447016596794,
+          0.010072972625494003,
+          0.005621712189167738,
+          -0.028410039842128754,
+          -0.02126319520175457,
+          -0.006916301790624857,
+          -0.008667545393109322,
+          -0.004380324389785528,
+          0.031158827245235443,
+          -0.016794199123978615,
+          -0.029367681592702866,
+          -0.023284882307052612,
+          -0.005036486312747002,
+          -0.04635695740580559,
+          0.0003114552528131753,
+          0.015827691182494164,
+          0.002108142012730241,
+          5.7462664699414745e-05,
+          -0.0026623329613357782,
+          0.004570966120809317,
+          -0.011119285598397255,
+          0.014072014018893242,
+          0.0034515007864683867,
+          0.0058034867979586124,
+          -0.00996656809002161,
+          -0.0005685998476110399,
+          0.0024916420225054026,
+          0.013921274803578854,
+          0.006597088184207678,
+          -0.02259325236082077,
+          -0.04802396148443222,
+          -0.027416929602622986,
+          -0.0029615960083901882,
+          -0.013273979537189007,
+          -0.000881163461599499,
+          -0.010853273794054985,
+          -0.0049655502662062645,
+          0.0034382001031190157,
+          -0.0008839344372972846,
+          -0.011855251155793667,
+          -0.008073452860116959,
+          -0.004245101939886808,
+          -0.008937990292906761,
+          -0.023072073236107826,
+          0.023355819284915924,
+          -0.009443412534892559,
+          -0.01621784083545208,
+          -0.020890777930617332,
+          -0.007501527667045593,
+          -0.023426756262779236,
+          -0.010587261989712715,
+          0.009673955850303173,
+          0.006557186134159565,
+          -0.016412915661931038,
+          -0.006716793403029442,
+          -0.004449044354259968,
+          0.008286261931061745,
+          -3.664586984086782e-05,
+          -0.006162602454423904,
+          -0.029083935543894768,
+          0.007523695006966591,
+          0.008711880072951317,
+          0.0083217304199934,
+          -0.0015317834913730621,
+          -0.002019471488893032,
+          0.01470157504081726,
+          0.08065471798181534,
+          -0.012697621248662472,
+          0.009363608434796333,
+          0.0008722963975742459,
+          -0.032400213181972504,
+          0.01244934368878603,
+          -0.03848301246762276,
+          -0.001967377495020628,
+          0.02230950817465782,
+          0.008574441075325012,
+          0.024260258302092552,
+          -0.011606973595917225,
+          0.010090706869959831,
+          0.04827224090695381,
+          -0.006348810624331236,
+          0.0029793300200253725,
+          0.0026800669729709625,
+          2.926820525317453e-05,
+          0.0056616137735545635,
+          0.007031573448330164,
+          -0.0015417590038850904,
+          -0.009682822972536087,
+          -0.019347911700606346,
+          -0.016191240400075912,
+          0.01780504360795021,
+          -0.013850337825715542,
+          -0.0012258702190592885,
+          -0.03177952021360397,
+          -0.004198549780994654,
+          -0.010002036578953266,
+          0.015632616356015205,
+          -0.03057360090315342,
+          0.00029787758830934763,
+          0.012298603542149067,
+          -0.035663291811943054,
+          0.004311604890972376,
+          0.001890899264253676,
+          0.0007936013280414045,
+          0.01835480146110058,
+          -0.003152237506583333,
+          -0.007723203860223293,
+          0.022132165729999542,
+          0.0022522315848618746,
+          -0.013628661632537842,
+          -0.022841529920697212,
+          0.002338685328140855,
+          -0.015987297520041466,
+          -0.03231154382228851,
+          -0.019720327109098434,
+          -0.00039347552228718996,
+          -0.009097597561776638,
+          0.023054338991642,
+          -0.007049307692795992,
+          -0.028729254379868507,
+          -0.01922377198934555,
+          0.007816308178007603,
+          0.004668503999710083,
+          -0.017140014097094536,
+          -0.005834521260112524,
+          0.021812953054904938,
+          -0.03738350048661232,
+          -0.021103588864207268,
+          -0.009443412534892559,
+          0.002622431144118309,
+          0.004641902633011341,
+          -0.008969024755060673,
+          -0.017255285754799843,
+          0.0024362229742109776,
+          0.004056677222251892,
+          -0.015304534696042538,
+          -0.0016359713627025485,
+          0.028285901993513107,
+          0.019418848678469658,
+          0.004522197414189577,
+          -0.00288179237395525,
+          0.004067760892212391,
+          -0.0051916600205004215,
+          0.006211371161043644,
+          0.0025936132296919823,
+          -0.026618896052241325,
+          -0.008290695026516914,
+          0.04848505184054375,
+          0.003999041393399239,
+          0.0057680183090269566,
+          0.004797075875103474,
+          -0.007084776181727648,
+          -0.014541967771947384,
+          0.0005589014617726207,
+          0.020057275891304016,
+          0.013415852561593056,
+          0.022752860561013222,
+          0.015428673475980759,
+          -0.007918278686702251,
+          -0.004167515318840742,
+          0.0002563132729846984,
+          -0.02285926416516304,
+          0.036816008388996124,
+          0.0016891737468540668,
+          -0.0025337606202811003,
+          0.005626145750284195,
+          -0.0063887122087180614,
+          0.00031976812169887125,
+          0.025501646101474762,
+          -0.004087711684405804,
+          -0.014666106551885605,
+          0.02321394719183445,
+          0.013477921485900879,
+          -0.007257683668285608,
+          -0.034510571509599686,
+          0.008840452879667282,
+          0.007044874131679535,
+          0.01585429161787033,
+          -0.01737942546606064,
+          -0.003799532540142536,
+          -0.009115330874919891,
+          -0.013318315148353577,
+          0.03660319745540619,
+          -0.00723994942381978,
+          -0.006597088184207678,
+          -0.0010075189638882875,
+          -0.030449463054537773,
+          0.00680103013291955,
+          0.02383464016020298,
+          0.00819759164005518,
+          -0.027345994487404823,
+          -0.0068542324006557465,
+          0.004083278123289347,
+          -0.020270084962248802,
+          -0.010853273794054985,
+          0.005107422824949026,
+          -0.023781437426805496,
+          0.011278891935944557,
+          -0.0028175064362585545,
+          -0.0030702173244208097,
+          0.0018432388314977288,
+          0.0020050625316798687,
+          0.04784662276506424,
+          -0.02940315008163452,
+          0.01904643140733242,
+          -0.003897070186212659,
+          0.009239469654858112,
+          0.00018468410416971892,
+          0.011190221644937992,
+          -0.007270983885973692,
+          -0.007869509980082512,
+          -0.00019646066357381642,
+          -0.03745443373918533,
+          -0.0007570247398689389,
+          0.010401054285466671,
+          0.038305673748254776,
+          -0.007754238322377205,
+          -0.007115810643881559,
+          0.023621831089258194,
+          0.011314360424876213,
+          0.009106464684009552,
+          -0.008410400710999966,
+          -0.005989694967865944,
+          -0.007563597057014704,
+          -0.0011288868263363838,
+          -0.012564615346491337,
+          0.029296746477484703,
+          -0.00907986331731081,
+          0.003713078796863556,
+          -0.030077045783400536,
+          0.02933221310377121,
+          0.0056571802124381065,
+          0.02401198074221611,
+          -0.004273919854313135,
+          0.0019607271533459425,
+          0.005821220576763153,
+          0.025998201221227646,
+          -0.009274938143789768,
+          -0.023089807480573654,
+          0.0012979150051251054,
+          0.01763656921684742,
+          0.04387418180704117,
+          0.030609069392085075,
+          -0.017476962879300117,
+          0.023444490507245064,
+          -0.03596476837992668,
+          0.0003726933500729501,
+          0.006198070477694273,
+          -0.006570486817508936,
+          -0.0053202323615550995,
+          -0.014027679339051247,
+          0.0004497258923947811,
+          0.004637469071894884,
+          -0.01798238418996334,
+          -0.008175423368811607,
+          -0.010223712772130966,
+          0.01518926303833723,
+          0.01799125224351883,
+          0.004903480876237154,
+          -0.0010851057013496757,
+          7.121352246031165e-05,
+          -0.021724281832575798,
+          0.010081839747726917,
+          -0.026547959074378014,
+          0.03438643366098404,
+          -0.009088730439543724,
+          -0.00010841358744073659,
+          0.007355221081525087,
+          -0.01279515866190195,
+          0.01762770302593708,
+          0.01541093923151493,
+          -0.0019751363433897495,
+          0.0059187584556639194,
+          -0.039653465151786804,
+          0.020890777930617332,
+          -0.009709424339234829,
+          -0.04571852833032608,
+          -0.0011194655671715736,
+          0.011332094669342041,
+          -0.01621784083545208,
+          0.020358756184577942,
+          0.03365933522582054,
+          -0.01603163406252861,
+          0.0136020602658391,
+          -0.015517343766987324,
+          -0.009195134975016117,
+          0.027878016233444214,
+          0.006100533064454794,
+          -0.015818823128938675,
+          0.007975914515554905,
+          0.0057680183090269566,
+          0.02046515978872776,
+          -0.008525672368705273,
+          -2.9960943720652722e-05,
+          0.004695104900747538,
+          -0.010170510970056057,
+          0.040220957249403,
+          5.850177331012674e-05,
+          0.019436581060290337,
+          -0.01895776018500328,
+          -0.010906476527452469,
+          -0.006472949404269457,
+          0.003294110530987382,
+          -0.0067300936207175255,
+          -0.010321250185370445,
+          -0.018727216869592667,
+          -0.009239469654858112,
+          -0.00939907692372799,
+          0.031495776027441025,
+          -0.0014785812236368656,
+          -0.011704511009156704,
+          -0.0002506328164599836,
+          -0.013619794510304928,
+          -0.004240668378770351,
+          -0.029314478859305382,
+          -0.0054177697747945786,
+          -0.007324186619371176,
+          -0.01966712437570095,
+          -0.006295608356595039,
+          -0.01328284665942192,
+          0.005861122626811266,
+          0.014790245331823826,
+          0.008565573953092098,
+          0.008379366248846054,
+          -0.0040079085156321526,
+          -0.0021713196765631437,
+          -0.006827631499618292,
+          -0.037241626530885696,
+          0.017388291656970978,
+          0.0016326463082805276,
+          -0.01586315967142582,
+          0.00886705331504345,
+          -0.03656772896647453,
+          0.03571649268269539,
+          0.02330261655151844,
+          -0.02888886071741581,
+          0.00575915165245533,
+          0.025909531861543655,
+          0.024863218888640404,
+          -0.0036554429680109024,
+          0.009984302334487438,
+          -0.015543945133686066,
+          -0.007412856910377741,
+          -0.0053290994837880135,
+          0.02629968151450157,
+          -0.009727157652378082,
+          0.002292133402079344,
+          0.04206530377268791,
+          -0.0011837516212835908,
+          0.004788209218531847,
+          -0.011376430280506611,
+          0.005360133945941925,
+          -0.008441435173153877,
+          0.0027266191318631172,
+          -0.036354921758174896,
+          -0.006929602473974228,
+          -0.03413815796375275,
+          0.0062601398676633835,
+          0.01191732008010149,
+          -0.005954226478934288,
+          0.02842777408659458,
+          0.002972679678350687,
+          0.002906176960095763,
+          0.022433646023273468,
+          -0.025944998487830162,
+          -0.026796236634254456,
+          0.012378406710922718,
+          0.006951770279556513,
+          0.031123358756303787,
+          0.013637528754770756,
+          0.007222215179353952,
+          0.003555688541382551,
+          -0.0056660473346710205,
+          -0.04518650472164154,
+          0.00819759164005518,
+          -0.013406985439360142,
+          0.024402132257819176,
+          0.0046020010486245155,
+          0.0080601517111063,
+          0.03605344146490097,
+          0.01332718227058649,
+          0.004615301266312599,
+          0.016359714791178703,
+          0.0007514827884733677,
+          0.00819759164005518,
+          0.03004157729446888,
+          0.028658317402005196,
+          -0.007155712228268385,
+          -0.04649883136153221,
+          -0.032027799636125565,
+          0.010383320041000843,
+          -0.0023586363531649113,
+          -0.0006478491122834384,
+          0.010019770823419094,
+          -0.023373553529381752,
+          -0.01033011730760336,
+          0.023515427485108376,
+          -4.7383316996274516e-05,
+          -0.00653058523312211,
+          -0.03156671300530434,
+          -0.012050325982272625,
+          0.01771637424826622,
+          -0.012928164564073086,
+          0.011021748185157776,
+          0.014772512018680573,
+          -0.010871008038520813,
+          -0.015419806353747845,
+          -0.014932118356227875,
+          -0.002702234545722604,
+          0.019986338913440704,
+          0.009860163554549217,
+          -0.020004073157906532,
+          0.022486848756670952,
+          0.016634592786431313,
+          0.0062734405510127544,
+          0.012768557295203209,
+          -0.020571565255522728,
+          0.02638835273683071,
+          -0.013983343727886677,
+          0.014692707918584347,
+          -0.012050325982272625,
+          -0.00686309952288866,
+          0.011890718713402748,
+          0.004101012367755175,
+          -0.0014353543519973755,
+          -0.003888203063979745,
+          0.001440896186977625,
+          -0.012334072031080723,
+          -0.0013622011756524444,
+          -0.020589299499988556,
+          0.009283805266022682,
+          -0.011225690133869648,
+          -0.00047217062092386186,
+          0.023196212947368622,
+          0.01413408387452364,
+          0.003201006446033716,
+          0.0028441075701266527,
+          -0.014045413583517075,
+          -0.01984446682035923,
+          0.0050675212405622005,
+          -0.006308908574283123,
+          -0.0028441075701266527,
+          -0.02073117159307003,
+          -0.0254307109862566,
+          0.017583368346095085,
+          0.03321598470211029,
+          0.016058234497904778,
+          -0.027629738673567772,
+          -0.013876939192414284,
+          -0.004205200355499983,
+          0.014009945094585419,
+          0.011881851591169834,
+          -0.006632556207478046,
+          0.006397579330950975,
+          -0.008193157613277435,
+          -0.0036000239197164774,
+          -0.013433586806058884,
+          0.015996165573596954,
+          0.01711341366171837,
+          0.004468995146453381,
+          0.018301598727703094,
+          0.015632616356015205,
+          0.015446407720446587,
+          -0.030431728810071945,
+          -0.0014043196570128202,
+          0.018283864483237267,
+          -0.010915343649685383,
+          -0.03585836663842201,
+          -0.014692707918584347,
+          0.00228326627984643,
+          0.018461205065250397,
+          -0.0015905278269201517,
+          -0.02135186456143856,
+          -0.014719309285283089,
+          0.0002265255170641467,
+          -0.03373027220368385,
+          0.017335090786218643,
+          0.00018468410416971892,
+          -0.00043060630559921265,
+          0.015526210889220238,
+          0.0074483249336481094,
+          -0.0007398448069579899,
+          0.005107422824949026,
+          -0.019188303500413895,
+          -0.006521718110889196,
+          -0.007545862812548876,
+          0.02959822490811348,
+          -0.00028429992380551994,
+          0.0023209513165056705,
+          -0.005705948919057846,
+          -0.009390209801495075,
+          0.007528128568083048,
+          -0.010383320041000843,
+          0.00022555567556992173,
+          0.012422742322087288,
+          0.025271102786064148,
+          -0.008405966684222221,
+          0.011314360424876213,
+          0.039830803871154785,
+          0.00038350006798282266,
+          -0.03436870127916336,
+          0.00023594676167704165,
+          -0.01435576006770134,
+          -0.005129590630531311,
+          0.009008926339447498,
+          0.007270983885973692,
+          -0.0007564705447293818,
+          -0.001497423741966486,
+          -0.006885267328470945,
+          -0.021919356659054756,
+          -0.012360673397779465,
+          -0.030147982761263847,
+          -0.00988676492124796,
+          -0.0047261398285627365,
+          0.007222215179353952,
+          -0.020482894033193588,
+          0.006477382965385914,
+          -0.01611143723130226,
+          -0.009718290530145168,
+          -0.015242465771734715,
+          -0.002522676717489958,
+          -0.013539991341531277,
+          -0.014391228556632996,
+          0.0070936428382992744,
+          0.006167036015540361,
+          0.00934587512165308,
+          -0.0033716971520334482,
+          -0.009057695046067238,
+          -0.0019197171786800027,
+          -0.012422742322087288,
+          0.014719309285283089,
+          0.00864537712186575,
+          0.01637744903564453,
+          -0.003936972003430128,
+          -0.009115330874919891,
+          0.00031588878482580185,
+          0.01683853566646576,
+          0.012786291539669037,
+          0.010268048383295536,
+          -0.012165597639977932,
+          0.014231621287763119,
+          -0.003688694443553686,
+          0.01700701005756855,
+          -0.004191899672150612,
+          0.010241447016596794,
+          -0.004792642779648304,
+          0.004938948899507523,
+          0.019613923504948616,
+          0.018993228673934937,
+          0.004912347998470068,
+          0.004903480876237154,
+          0.06632556021213531,
+          0.013406985439360142,
+          0.033517464995384216,
+          0.011287759058177471,
+          0.0065039838664233685,
+          -0.01510945986956358,
+          -0.008437002077698708,
+          -0.012422742322087288,
+          0.026743033900856972,
+          0.017246419563889503,
+          0.007270983885973692,
+          -0.018656281754374504,
+          -0.0030236653983592987,
+          0.028658317402005196,
+          0.014657239429652691,
+          0.008219758979976177,
+          0.006286741234362125,
+          0.019454315304756165,
+          0.006082798819988966,
+          0.008366065099835396,
+          0.003460367675870657,
+          0.0060029951855540276,
+          0.006078365258872509,
+          0.015313401818275452,
+          0.012617817148566246,
+          0.01208579447120428,
+          0.005018752533942461,
+          0.00560397794470191,
+          0.02949182130396366,
+          0.014426696114242077,
+          0.029615959152579308,
+          -0.0007819632883183658,
+          0.005710382480174303,
+          0.031939126551151276,
+          0.012147863395512104,
+          0.01421388704329729,
+          0.02614007517695427,
+          -0.0017612185329198837,
+          -0.022486848756670952,
+          0.0037507638335227966,
+          0.02214989997446537,
+          -0.014630638994276524,
+          0.023284882307052612,
+          -0.0033606134820729494,
+          -0.0008224192424677312,
+          -0.026725299656391144,
+          -0.019117368385195732,
+          0.0027776046190410852,
+          -0.027328260242938995,
+          0.009939967654645443,
+          -0.0029926307033747435,
+          -0.007523695006966591,
+          0.005847821943461895,
+          -0.007900545373558998,
+          -0.010516325943171978,
+          0.004562098998576403,
+          0.016874004155397415,
+          -0.005630579311400652,
+          -0.0067389607429504395,
+          0.013300580903887749,
+          -0.000335839664330706,
+          -0.0005372880259528756,
+          0.011819782666862011,
+          -0.00394362211227417,
+          -0.015632616356015205,
+          0.00935474131256342,
+          -0.0013743933523073792,
+          -0.012236534617841244,
+          -0.015606014989316463,
+          0.0013888023095205426,
+          -0.005976394284516573,
+          -0.01569468528032303,
+          0.0029704628977924585,
+          0.007590197958052158,
+          -0.002387454267591238,
+          -0.0031322867143899202,
+          0.007049307692795992,
+          0.012883828952908516,
+          0.011447366327047348,
+          -0.002442873315885663,
+          0.005488706286996603,
+          0.003828350454568863,
+          -0.014178418554365635,
+          -0.010862140916287899,
+          -0.038412075489759445,
+          -0.005816787481307983,
+          0.01922377198934555,
+          0.01860307902097702,
+          0.0006855340907350183,
+          0.003285243408754468,
+          0.009195134975016117,
+          0.006699059158563614,
+          -0.0007453867001459002,
+          0.004801509436219931,
+          -0.025483911857008934,
+          0.002972679678350687,
+          0.012165597639977932,
+          -0.008352764882147312,
+          -0.0007985890260897577,
+          -0.01826613023877144,
+          -0.021316397935152054,
+          0.0041120960377156734,
+          0.003169971751049161,
+          -0.025111496448516846,
+          -0.011793181300163269,
+          0.0029638127889484167,
+          0.008716314099729061,
+          -0.002309867413714528,
+          -0.027612006291747093,
+          0.009070996195077896,
+          -0.025324305519461632,
+          0.013167575001716614,
+          -3.078356940022786e-06,
+          -0.021032651886343956,
+          0.03624851629137993,
+          0.002489425241947174,
+          0.01603163406252861,
+          0.004083278123289347,
+          -0.007705469615757465,
+          -0.009461146779358387,
+          -0.02649475634098053,
+          0.01350452285259962,
+          -0.0066901920363307,
+          0.0014752560527995229,
+          0.019596189260482788,
+          -0.004171948879957199,
+          -0.004588700365275145,
+          -0.013539991341531277,
+          0.003693128004670143,
+          -0.010064106434583664,
+          0.000359392783138901,
+          0.013096638023853302,
+          -0.0006201395881362259,
+          0.00035828439285978675,
+          -0.0025514946319162846,
+          -0.001351117272861302,
+          -0.005475405603647232,
+          0.0001952137245098129,
+          -0.0015495176194235682,
+          -0.009824695996940136,
+          -0.002011712873354554,
+          0.005120723508298397,
+          -0.005032053217291832,
+          0.019170571118593216,
+          0.015703551471233368,
+          -0.016856269910931587,
+          -0.004788209218531847,
+          -0.013797136023640633,
+          0.008969024755060673,
+          -0.022806061431765556,
+          0.008756215684115887,
+          0.0060162958689033985,
+          0.0159606970846653,
+          0.008552273735404015,
+          -0.0020737822633236647,
+          0.004034509416669607,
+          0.017166616395115852,
+          0.017521297559142113,
+          -0.0012602299684658647,
+          -0.006663591135293245,
+          -0.00728871813043952,
+          -0.007160145789384842,
+          -0.011083817109465599,
+          -0.002338685328140855,
+          -0.009673955850303173,
+          0.023338085040450096,
+          0.015242465771734715,
+          0.022664189338684082,
+          0.008490203879773617,
+          -0.0014785812236368656,
+          -0.007248816546052694,
+          0.016182372346520424,
+          0.0029748964589089155,
+          0.005240428727120161,
+          0.006317775696516037,
+          -0.0034581508953124285,
+          0.004801509436219931,
+          -0.004562098998576403,
+          0.008295129053294659,
+          0.008734048344194889,
+          -0.009638487361371517,
+          0.01835480146110058,
+          0.009266071021556854,
+          -0.01808878965675831,
+          0.008632076904177666,
+          -0.003972440026700497,
+          -0.0028928762767463923,
+          0.006823197938501835,
+          0.0044822958298027515,
+          -0.0036066740285605192,
+          0.008561140857636929,
+          0.007430591154843569,
+          -0.007226648740470409,
+          0.007599065080285072,
+          0.007519261445850134,
+          -0.003460367675870657,
+          0.014648373238742352,
+          0.012130129151046276,
+          -0.0003546821535564959,
+          0.014559702016413212,
+          0.0028729254845529795,
+          -0.008663111366331577,
+          -0.011500569060444832,
+          0.016155771911144257,
+          -0.0035401713103055954,
+          0.007417290471494198,
+          0.0036975613329559565,
+          0.03470565006136894,
+          0.006339943502098322,
+          -0.015934094786643982,
+          -0.03392534703016281,
+          0.003786231856793165,
+          -0.0037064284551888704,
+          0.014533101581037045,
+          -0.013238511048257351,
+          0.006699059158563614,
+          0.01462177187204361,
+          -0.00494338246062398,
+          0.026512490585446358,
+          0.004426876548677683,
+          0.007656700909137726,
+          0.006277874112129211,
+          -0.010658198967576027,
+          0.0006594871520064771,
+          -0.0017290754476562142,
+          0.020145945250988007,
+          0.0016060450579971075,
+          -0.013637528754770756,
+          -0.014949852600693703,
+          -0.014178418554365635,
+          0.006344377063214779,
+          0.013797136023640633,
+          -0.010968545451760292,
+          0.020890777930617332,
+          -0.011544903740286827,
+          0.012342939153313637,
+          0.03342879191040993,
+          0.02454400435090065,
+          0.011748846620321274,
+          0.0008745131781324744,
+          0.0004896276514045894,
+          -0.018230661749839783,
+          -0.009718290530145168,
+          -0.00956755131483078,
+          -0.0016891737468540668,
+          0.0017656520940363407,
+          0.011606973595917225,
+          0.005728116724640131,
+          0.012307470664381981,
+          -3.5606761230155826e-05,
+          -0.019418848678469658,
+          0.016350846737623215,
+          -0.02817949652671814,
+          -0.010170510970056057,
+          -0.007426157593727112,
+          0.029704630374908447,
+          0.024207057431340218,
+          -0.01577448844909668,
+          0.011092684231698513,
+          0.010853273794054985,
+          -0.0056483130902051926,
+          0.013832603581249714,
+          -0.010064106434583664,
+          -0.0009210652206093073,
+          0.009372475557029247,
+          0.011305493302643299,
+          0.006118267308920622,
+          0.020571565255522728,
+          0.024136120453476906,
+          0.027434663847088814,
+          -0.006167036015540361,
+          -0.005994128528982401,
+          0.012112395837903023,
+          0.00996656809002161,
+          0.012679887004196644,
+          0.020323287695646286,
+          0.0006567161763086915,
+          -0.016545921564102173,
+          0.03509579971432686,
+          0.009665088728070259,
+          0.003959139343351126,
+          0.0003106239892076701,
+          0.00403007585555315,
+          0.023621831089258194,
+          -0.02161787636578083,
+          0.004287220537662506,
+          0.0048502786085009575,
+          -0.013841470703482628,
+          0.03759630769491196,
+          0.008481336757540703,
+          0.012520279735326767,
+          0.00751039432361722,
+          0.0023564195726066828,
+          0.00026518033700995147,
+          -0.006193636916577816,
+          -0.00881385151296854,
+          0.016785332933068275,
+          -0.009425678290426731,
+          -0.018443472683429718,
+          -0.00962075311690569,
+          -0.01425822265446186,
+          0.0019285841844975948,
+          -0.007975914515554905,
+          -0.020748905837535858,
+          0.0003084072086494416,
+          -0.012068060226738453,
+          0.019897669553756714,
+          -0.004143130965530872,
+          -0.0021247677505016327,
+          -0.010853273794054985,
+          -0.00016237792442552745,
+          0.005630579311400652,
+          0.013460187241435051,
+          -0.018133124336600304,
+          -0.0067433943040668964,
+          0.01904643140733242,
+          -0.014231621287763119,
+          -0.0127153554931283,
+          -0.026335150003433228,
+          -0.020075010135769844,
+          0.001347792218439281,
+          -0.008441435173153877,
+          0.005147324874997139,
+          0.008583308197557926,
+          0.0050808219239115715,
+          -0.0013644179562106729,
+          0.005018752533942461,
+          -0.021458270028233528,
+          0.00033473127405159175,
+          -0.008995626121759415,
+          0.018904559314250946,
+          0.004899047315120697,
+          0.0037928821984678507,
+          -0.014249355532228947,
+          -0.0029881971422582865,
+          -0.010126175358891487,
+          -0.016412915661931038,
+          0.002062698360532522,
+          -0.03197459504008293,
+          0.013220776803791523,
+          -0.005351266823709011,
+          -0.014001077972352505,
+          -0.002966029569506645,
+          -0.012564615346491337,
+          -0.009044394828379154,
+          0.010338984429836273,
+          -0.002567012095823884,
+          -0.025022825226187706,
+          -0.0003854397509712726,
+          0.0034891855902969837,
+          0.018904559314250946,
+          -0.0035734225530177355,
+          0.0030236653983592987,
+          -0.0004976633936166763,
+          -0.007408423349261284,
+          -0.00913306511938572,
+          0.020571565255522728,
+          0.007333053275942802,
+          -0.01431142445653677,
+          0.006162602454423904,
+          0.0053202323615550995,
+          -0.03137163445353508,
+          -0.0205538310110569,
+          -0.0018365884898230433,
+          0.01528680045157671,
+          -0.0006112725241109729,
+          -0.0010607213480398059,
+          0.003118986263871193,
+          -0.015348870307207108,
+          -0.011553770862519741,
+          -0.0138237364590168,
+          0.006942903157323599,
+          -0.013743933290243149,
+          0.007860642857849598,
+          -0.010002036578953266,
+          0.0067256600596010685,
+          0.0014475465286523104,
+          0.006517284549772739,
+          0.021050386130809784,
+          -0.0007021598285064101,
+          0.003285243408754468,
+          -0.005608411505818367,
+          0.00420963391661644,
+          -0.015073991380631924,
+          0.009700557217001915,
+          0.00876951590180397,
+          -0.0018210711423307657,
+          0.017609968781471252,
+          0.002079324098303914,
+          -0.011323227547109127,
+          0.000915523327421397,
+          0.0010956353507936,
+          0.004646336194127798,
+          -0.018017852678894997,
+          -0.019418848678469658,
+          0.015934094786643982,
+          -0.004134263843297958,
+          0.010569527745246887,
+          -0.00086398352868855,
+          0.008751781657338142,
+          -0.004360373597592115,
+          0.00642418023198843,
+          0.025732189416885376,
+          -0.011598106473684311,
+          0.0012956982245668769,
+          -0.0067389607429504395,
+          0.006636989768594503,
+          -0.007412856910377741,
+          0.02958049066364765,
+          0.011270025745034218,
+          0.008804984390735626,
+          0.004393625073134899,
+          0.024313461035490036,
+          0.0026667665224522352,
+          -0.01279515866190195,
+          0.0017346173990517855,
+          0.033623866736888885,
+          0.0026601161807775497,
+          -0.003872685832902789,
+          0.00988676492124796,
+          0.012245400808751583,
+          0.0009875681716948748,
+          0.006747827865183353,
+          0.00675226142629981,
+          0.019525252282619476,
+          -0.0026046971324831247,
+          0.019330177456140518,
+          -0.004154214635491371,
+          -0.006215804722160101,
+          -0.026991311460733414,
+          0.0056660473346710205,
+          -0.02126319520175457,
+          0.01604936644434929,
+          -0.0027753878384828568,
+          0.016359714791178703,
+          0.012502545490860939,
+          -0.001784494612365961,
+          -0.022912466898560524,
+          -0.004404708743095398,
+          0.016811933368444443,
+          -0.0041608652099967,
+          0.011802048422396183,
+          -0.007536995690315962,
+          0.0015794439241290092,
+          0.007568030618131161,
+          -0.017308488488197327,
+          -0.0037418967112898827,
+          0.0006234647007659078,
+          -0.010507458820939064,
+          0.006588221061974764,
+          0.005963093601167202,
+          -0.008875920437276363,
+          0.017317356541752815,
+          -0.013921274803578854,
+          -0.003786231856793165,
+          -0.002456173999235034,
+          0.019276974722743034,
+          0.0006001886795274913,
+          0.008556706830859184,
+          -0.011571505106985569,
+          -0.015366604551672935,
+          6.151518755359575e-05,
+          0.012688754126429558,
+          -0.005630579311400652,
+          -0.0009875681716948748,
+          0.013593193143606186,
+          -0.008210891857743263,
+          0.032985441386699677,
+          -0.017929183319211006,
+          -0.028285901993513107,
+          -0.005874423310160637,
+          -0.0032187404576689005,
+          -0.010569527745246887,
+          0.003150020958855748,
+          0.013469054363667965,
+          0.0161203034222126,
+          -0.005519740749150515,
+          0.01222766749560833,
+          0.01701587624847889,
+          -0.0019906535744667053,
+          -0.023621831089258194,
+          -0.0023542027920484543,
+          -0.001258013304322958,
+          -0.013371516950428486,
+          -0.0045975674875080585,
+          -0.016705529764294624,
+          0.002338685328140855,
+          0.006707926280796528,
+          -0.003854951588436961,
+          -0.004067760892212391,
+          -0.005235995166003704,
+          0.007000538986176252,
+          0.0021624527871608734,
+          -0.019117368385195732,
+          0.00560397794470191,
+          -0.017335090786218643,
+          0.016874004155397415,
+          -0.005156191997230053,
+          -0.009000059217214584,
+          0.021582409739494324,
+          -0.008494637906551361,
+          -0.0056660473346710205,
+          0.001044095610268414,
+          0.0255903173238039,
+          -0.011225690133869648,
+          0.0056882151402533054,
+          -0.008069018833339214,
+          0.017335090786218643,
+          5.936769593972713e-05,
+          0.005284763872623444,
+          -0.0027244023513048887,
+          0.024242525920271873,
+          -0.006601521745324135,
+          -0.02596273273229599,
+          -0.0030657837633043528,
+          0.02895979769527912,
+          -0.02021688222885132,
+          -0.022132165729999542,
+          0.005280330311506987,
+          -0.000599080347456038,
+          0.0055374749936163425,
+          -0.003584506455808878,
+          0.0056837815791368484,
+          0.009815828874707222,
+          0.000171106425113976,
+          -0.006965070962905884,
+          0.024863218888640404,
+          -0.02771840989589691,
+          0.010143909603357315,
+          0.0047305733896791935,
+          -0.002263315487653017,
+          -0.010933076962828636,
+          0.02135186456143856,
+          0.014444430358707905,
+          0.006450781598687172,
+          -0.0017390509601682425,
+          0.01692720502614975,
+          -0.024082917720079422,
+          0.0021569107193499804,
+          0.01452423445880413,
+          0.01718435063958168,
+          -0.0033051942009478807,
+          -0.008241926319897175,
+          0.002852974459528923,
+          -0.0056571802124381065,
+          -0.01510945986956358,
+          0.011802048422396183,
+          -0.0018022286240011454,
+          0.015419806353747845,
+          -0.030857346951961517,
+          0.013185309246182442,
+          0.0006018512649461627,
+          0.014240488409996033,
+          0.002691150875762105,
+          0.007661134470254183,
+          -0.01333604846149683,
+          0.012121262960135937,
+          -0.017955783754587173,
+          0.00302144861780107,
+          0.017166616395115852,
+          0.020961714908480644,
+          -0.0036177579313516617,
+          -0.0031212028115987778,
+          0.01165130827575922,
+          0.014160685241222382,
+          0.016067100688815117,
+          0.003670960199087858,
+          0.0036864776629954576,
+          -0.017689771950244904,
+          -0.0300238449126482,
+          -0.00582565413787961,
+          -0.0014142950531095266,
+          -0.012369539588689804,
+          -0.0009338115924037993,
+          0.019436581060290337,
+          -0.001129995216615498,
+          0.018762685358524323,
+          -0.03536181151866913,
+          0.008069018833339214,
+          -0.007545862812548876,
+          -0.012484811246395111,
+          -0.0032519919332116842,
+          -0.014914384111762047,
+          -0.015127194114029408,
+          0.0067256600596010685,
+          -0.011500569060444832,
+          0.009124197997152805,
+          -0.01666119322180748,
+          -0.025324305519461632,
+          0.008889221586287022,
+          -0.01125229150056839,
+          -0.013007967732846737,
+          0.007856209762394428,
+          -0.01772524043917656,
+          0.007355221081525087,
+          0.007971481420099735,
+          0.0016016116132959723,
+          -0.00631334213539958,
+          0.002755437046289444,
+          0.0076167993247509,
+          0.003537954529747367,
+          0.015809956938028336,
+          -0.0006428614142350852,
+          -0.00048824213445186615,
+          -0.022717392072081566,
+          -0.007803007494658232,
+          -0.018106523901224136,
+          -0.002391887828707695,
+          -0.007036007009446621,
+          -0.03055586665868759,
+          -0.0031877057626843452,
+          0.010746869258582592,
+          0.003923671320080757,
+          0.014063146896660328,
+          0.004553232342004776,
+          -0.009540949948132038,
+          0.011926187202334404,
+          0.005502006970345974,
+          -0.0005051449406892061,
+          -0.0033561799209564924,
+          -0.005927625577896833,
+          0.008264093659818172,
+          0.00026725855423137546,
+          0.008960157632827759,
+          0.0006661374354735017,
+          0.025874063372612,
+          -0.006894134450703859,
+          -0.00030729881837032735,
+          0.010489724576473236,
+          -0.009736024774610996,
+          -0.0011155862594023347,
+          0.0007852884591557086,
+          0.0057768854312598705,
+          0.012662152759730816,
+          -0.021937090903520584,
+          0.012555748224258423,
+          -0.023621831089258194,
+          -0.011216823011636734,
+          0.007705469615757465,
+          0.0020471811294555664,
+          -0.010995146818459034,
+          0.0026002635713666677,
+          -0.01342471968382597,
+          -0.017343956977128983,
+          -0.00894242338836193,
+          0.017148882150650024,
+          -0.018585344776511192,
+          -0.03048493154346943,
+          -0.004703972022980452,
+          -0.007621232885867357,
+          -0.02303660660982132,
+          0.008343897759914398,
+          -0.024136120453476906,
+          -0.0030635669827461243,
+          0.000919956830330193,
+          0.015561679378151894,
+          -0.0011737762251868844,
+          0.014604037627577782,
+          -0.03064453788101673,
+          -0.0005134578095749021,
+          0.022433646023273468,
+          -0.021334132179617882,
+          -0.015162661671638489,
+          0.0061537353321909904,
+          -0.011048348620533943,
+          -0.019117368385195732,
+          0.011997124180197716,
+          -0.0007004972430877388,
+          0.002314300974830985,
+          -0.019986338913440704,
+          0.034067220985889435,
+          -0.037880051881074905,
+          0.005253729410469532,
+          0.009904499165713787,
+          0.023143010213971138,
+          0.008796117268502712,
+          0.016430649906396866,
+          0.02995290793478489,
+          0.0036554429680109024,
+          0.014391228556632996,
+          0.007763105444610119,
+          -0.005138457752764225,
+          -0.004885746631771326,
+          -0.009221736341714859,
+          -6.1956807257956825e-06,
+          -0.002870708703994751,
+          0.0058877235278487206,
+          0.005395602434873581,
+          0.022575518116354942,
+          -0.022912466898560524,
+          0.027416929602622986,
+          0.02488095313310623,
+          0.006769995670765638,
+          -0.00984242931008339,
+          0.0009886764455586672,
+          -0.013451320119202137,
+          0.023710502311587334,
+          -0.00353352096863091,
+          -0.021334132179617882,
+          -0.0012868312187492847,
+          0.019791264086961746,
+          0.015091725625097752,
+          0.0028884427156299353,
+          -0.009869030676782131,
+          -0.009381342679262161,
+          -0.0002915044024121016,
+          -0.03057360090315342,
+          0.007674435153603554,
+          0.012112395837903023,
+          -0.012076927348971367,
+          -0.01699814200401306,
+          -0.011713378131389618,
+          -0.013460187241435051,
+          0.001321190968155861,
+          -0.007333053275942802,
+          0.015224731527268887,
+          -0.013584326021373272,
+          0.0038660354912281036,
+          0.018283864483237267,
+          -0.0007010514382272959,
+          -0.022380443289875984,
+          -0.021901622414588928,
+          -0.004355940036475658,
+          -0.00917740073055029,
+          0.011145886965095997,
+          -0.008649811148643494,
+          -0.017166616395115852,
+          -0.019365645945072174,
+          0.0011859684018418193,
+          -0.02259325236082077,
+          -0.0016160205705091357,
+          -0.015322268940508366,
+          0.0006932927644811571,
+          -0.013619794510304928,
+          0.0014054280472919345,
+          -0.00885818712413311,
+          0.014586303383111954,
+          -0.0033517463598400354,
+          -0.0021114670671522617,
+          0.022628720849752426,
+          -0.011979389935731888,
+          0.017432628199458122,
+          0.003635491942986846,
+          0.0010668174363672733,
+          -0.006903001572936773,
+          -0.017867112532258034,
+          0.009558684192597866,
+          0.0024761247914284468,
+          -0.007767539005726576,
+          0.018230661749839783,
+          -0.012254267930984497,
+          0.0041231801733374596,
+          -0.0023497692309319973,
+          -0.01103061530739069,
+          0.0055507756769657135,
+          0.007355221081525087,
+          0.0013821519678458571,
+          -0.0013655262300744653,
+          0.003917021211236715,
+          -0.010117308236658573,
+          0.012218800373375416,
+          0.01492325123399496,
+          0.00881828460842371,
+          0.004480078816413879,
+          -0.026991311460733414,
+          0.006393145769834518,
+          0.023001138120889664,
+          -0.004515547305345535,
+          -0.0018498890567570925,
+          -0.0002811133163049817,
+          0.014683840796351433,
+          0.022841529920697212,
+          -0.007723203860223293,
+          0.009939967654645443,
+          -0.031141093000769615,
+          0.03349972888827324,
+          0.01629764400422573,
+          -0.013965609483420849,
+          -0.023675033822655678,
+          0.02376370318233967,
+          -0.01125229150056839,
+          0.0029748964589089155,
+          -0.011606973595917225,
+          -0.015517343766987324,
+          -0.024614941328763962,
+          0.01737942546606064,
+          0.015162661671638489,
+          0.007426157593727112,
+          -0.014471031725406647,
+          0.03064453788101673,
+          0.0006894134567119181,
+          0.004650769755244255,
+          -0.007324186619371176,
+          0.005351266823709011,
+          0.021546941250562668,
+          -0.003529087407514453,
+          -0.0031810556538403034,
+          -0.005258162971585989,
+          -0.01541093923151493,
+          0.0070936428382992744,
+          -0.014586303383111954,
+          0.0031345034949481487,
+          -0.010738002136349678,
+          -0.00030480496934615076,
+          0.009062129072844982,
+          0.027434663847088814,
+          0.008951290510594845,
+          0.011926187202334404,
+          0.023710502311587334,
+          -0.014497633092105389,
+          -0.010268048383295536,
+          -0.009558684192597866,
+          0.014125216752290726,
+          0.016368580982089043,
+          0.01984446682035923,
+          -0.0009166316594928503,
+          0.01622670888900757,
+          -0.013442453928291798,
+          -0.009718290530145168,
+          -0.009931100532412529,
+          -0.01744149439036846,
+          0.0007132436148822308,
+          0.00978922750800848,
+          -0.0026445987168699503,
+          -0.005169492214918137,
+          -0.011722245253622532,
+          0.012830627150833607,
+          -0.0015118326991796494,
+          -0.007372955325990915,
+          -0.010897609405219555,
+          0.01510945986956358,
+          -0.005293630994856358,
+          -0.004686237778514624,
+          0.016075968742370605,
+          0.023533159866929054,
+          -0.01993313618004322,
+          0.0026645497418940067,
+          -0.013406985439360142,
+          0.004267269745469093,
+          -0.007971481420099735,
+          0.031212029978632927,
+          -0.017920315265655518,
+          0.0205538310110569,
+          0.0014575219247490168,
+          -0.001433137571439147,
+          -0.009762626141309738,
+          0.0561816543340683,
+          -0.0012702054809778929,
+          -0.028853392228484154,
+          0.024207057431340218,
+          0.007927145808935165,
+          -0.012724222615361214,
+          -0.002011712873354554,
+          0.01262668427079916,
+          0.015650350600481033,
+          0.004085495136678219,
+          -0.011935054324567318,
+          -0.007275417447090149,
+          -0.007004972547292709,
+          -0.017601102590560913,
+          0.030520398169755936,
+          -0.01895776018500328,
+          0.004868012387305498,
+          0.011358696036040783,
+          0.021777484565973282,
+          -0.004686237778514624,
+          -0.01510945986956358,
+          0.00225888192653656,
+          0.007319753058254719,
+          -0.024685878306627274,
+          -0.004198549780994654,
+          0.00016542596858926117,
+          -0.0025315438397228718,
+          -0.004218501038849354,
+          -0.020429691299796104,
+          0.02383464016020298,
+          0.00984242931008339,
+          -0.00750596122816205,
+          -0.017929183319211006,
+          -0.01700701005756855,
+          -0.010844406671822071,
+          -0.017831645905971527,
+          -0.002586962888017297,
+          0.017973517999053,
+          0.016803067177534103,
+          0.001548409229144454,
+          0.013930141925811768,
+          -0.0055463421158492565,
+          0.0049566831439733505,
+          -0.010427655652165413,
+          0.012307470664381981,
+          -0.01753903180360794,
+          0.0038704690523445606,
+          0.01975579559803009,
+          -0.007297585252672434,
+          0.01948978379368782,
+          0.020677968859672546,
+          0.0067433943040668964,
+          -0.008623209781944752,
+          0.00010411861148895696,
+          0.0024207057431340218,
+          0.024047449231147766,
+          -0.0016115870093926787,
+          -0.002083757659420371,
+          0.0024362229742109776,
+          0.007807441055774689,
+          0.028906594961881638,
+          0.009860163554549217,
+          0.008184290491044521,
+          0.025182433426380157,
+          -0.0031677549704909325,
+          -0.009159666486084461,
+          -0.0055419085547327995,
+          -0.014958719722926617,
+          0.0020216882694512606,
+          0.005958660040050745,
+          0.003815049771219492,
+          -0.0029881971422582865,
+          -0.0038616019301116467,
+          -0.001410969882272184,
+          -0.01886909082531929,
+          0.010693666525185108,
+          0.0016670060576871037,
+          0.0020649151410907507,
+          0.02835683710873127,
+          -0.012272002175450325,
+          0.008312863297760487,
+          0.01788484677672386,
+          -0.003032532287761569,
+          0.00930153951048851,
+          0.013043436221778393,
+          -0.008095620200037956,
+          0.008964591659605503,
+          -0.010640464723110199,
+          -0.01853214204311371,
+          -0.004074411001056433,
+          -0.0021790785249322653,
+          0.022185368463397026,
+          -0.010915343649685383,
+          -0.02356862835586071,
+          0.025271102786064148,
+          0.009860163554549217,
+          -0.022894732654094696,
+          -0.009603018872439861,
+          -0.00011146163888042793,
+          0.00801138300448656,
+          0.00962075311690569,
+          0.022274039685726166,
+          -0.01566808484494686,
+          -0.007670001592487097,
+          0.0003970777615904808,
+          0.008804984390735626,
+          0.019170571118593216,
+          0.0070892092771828175,
+          -0.005001018289476633,
+          -0.0011915103532373905,
+          0.01173997949808836,
+          -0.012662152759730816,
+          -0.023320350795984268,
+          0.007355221081525087,
+          0.00723994942381978,
+          -0.006787729449570179,
+          0.011881851591169834,
+          -0.013575458899140358,
+          -0.02399424836039543,
+          -0.012990233488380909,
+          0.004398058634251356,
+          -0.023621831089258194,
+          -0.01518926303833723,
+          -0.0027199687901884317,
+          -0.025874063372612,
+          -0.005151758436113596,
+          0.008335030637681484,
+          -0.007749804761260748,
+          0.009141932241618633,
+          0.004484512377530336,
+          0.006574920378625393,
+          -0.002575879218056798,
+          -0.0026290814857929945,
+          -9.774541103979573e-05,
+          0.0006384278531186283,
+          -0.002926127752289176,
+          0.0009011143120005727,
+          -0.021937090903520584,
+          0.009939967654645443,
+          -0.008707446977496147,
+          0.010480857454240322,
+          -0.009221736341714859,
+          -0.005377868190407753,
+          -0.015898628160357475,
+          0.016013899818062782,
+          -0.018762685358524323,
+          0.01995087042450905,
+          -1.6495836462127045e-05,
+          0.006406446453183889,
+          -0.0002840228262357414,
+          0.018408004194498062,
+          -0.014417828992009163,
+          -0.007607932202517986,
+          0.011562637984752655,
+          0.006428613793104887,
+          0.024473069235682487,
+          -0.01728188805282116,
+          0.0019540770445019007,
+          0.013655262999236584,
+          0.011837516911327839,
+          0.01218333188444376,
+          -0.00036299502244219184,
+          -0.019330177456140518,
+          0.011456233449280262,
+          0.0007354112458415329,
+          0.005240428727120161,
+          -0.004205200355499983,
+          0.014515367336571217,
+          0.020181413739919662,
+          -0.021387333050370216,
+          0.015357737429440022,
+          0.0009304864797741175,
+          0.00236972002312541,
+          0.012121262960135937,
+          0.005612845066934824,
+          -0.01824839599430561,
+          -0.014151818118989468,
+          0.012928164564073086,
+          0.0066901920363307,
+          -0.028197230771183968,
+          0.024331195279955864,
+          -0.014426696114242077,
+          -0.0004627493617590517,
+          0.010613863356411457,
+          -0.001321190968155861,
+          -0.024685878306627274,
+          -0.00837493222206831,
+          0.000817431544419378,
+          -0.004912347998470068,
+          -0.008286261931061745,
+          0.0030524833127856255,
+          0.018301598727703094,
+          0.009647354483604431,
+          0.0017789526609703898,
+          0.020571565255522728,
+          0.005732550285756588,
+          0.012582349590957165,
+          0.03873129189014435,
+          -0.001021927921101451,
+          0.004471211694180965,
+          -0.02019914798438549,
+          0.011580372229218483,
+          0.011385297402739525,
+          -0.006056197918951511,
+          -0.01528680045157671,
+          -0.025377508252859116,
+          -0.002099274890497327,
+          -0.009780360385775566,
+          0.010862140916287899,
+          0.0018709483556449413,
+          -0.011527169495821,
+          0.004300521221011877,
+          0.00701827323064208,
+          -0.010046372190117836,
+          -0.012759690172970295,
+          0.01904643140733242,
+          0.009266071021556854,
+          0.008778383024036884,
+          0.0008296237210743129,
+          -0.001856539398431778,
+          -0.01528680045157671,
+          0.014302557334303856,
+          -0.020979449152946472,
+          -0.002135851653292775,
+          0.00805571861565113,
+          0.004114313051104546,
+          -0.010959678329527378,
+          0.0057724518701434135,
+          -0.009629620239138603,
+          0.0011582588776946068,
+          -0.00793157983571291,
+          0.035397280007600784,
+          -0.004114313051104546,
+          0.0050896890461444855,
+          0.010401054285466671,
+          -0.00304583297111094,
+          0.006583787500858307,
+          -0.012467077933251858,
+          -0.008836018852889538,
+          0.013247378170490265,
+          0.02612234093248844,
+          -0.010392187163233757,
+          -0.019738061353564262,
+          0.0009947726503014565,
+          0.003639925504103303,
+          -0.00837049912661314,
+          -0.0006694625481031835,
+          0.010995146818459034,
+          0.014417828992009163,
+          -0.008587741293013096,
+          -0.006459648720920086,
+          0.024739079177379608,
+          0.013477921485900879,
+          -0.004351506475359201,
+          -0.016891736537218094,
+          0.009022227488458157,
+          0.0002779267088044435,
+          0.006942903157323599,
+          0.0063798450864851475,
+          -0.007151278667151928,
+          -0.004522197414189577,
+          0.023231681436300278,
+          -0.011004013940691948,
+          -0.002135851653292775,
+          -0.01094194408506155,
+          -0.016492720693349838,
+          -0.022717392072081566,
+          0.002875142265111208,
+          -0.024136120453476906,
+          -4.776432615472004e-05,
+          0.025501646101474762,
+          -0.0016969323623925447,
+          -0.009833562187850475,
+          0.0018576477887108922,
+          -0.016891736537218094,
+          0.012857227586209774,
+          6.400904385372996e-05,
+          -0.001757893362082541,
+          -0.0013788269134238362,
+          0.027133185416460037,
+          0.03816379979252815,
+          0.02481001615524292,
+          -0.002146935323253274,
+          0.01658139005303383,
+          -0.007953747175633907,
+          0.012068060226738453,
+          -0.0054488047026097775,
+          0.01391240768134594,
+          0.015428673475980759,
+          0.01116362027823925,
+          0.004335989244282246,
+          -0.004992151167243719,
+          -0.006433047354221344,
+          0.006526151672005653,
+          -0.008663111366331577,
+          0.007758671883493662,
+          0.0223272405564785,
+          0.0027776046190410852,
+          0.01537547167390585,
+          0.012059193104505539,
+          -0.011332094669342041,
+          0.016457252204418182,
+          -0.01728188805282116,
+          0.0027111016679555178,
+          -0.016058234497904778,
+          -0.0028640583623200655,
+          0.035822898149490356,
+          0.015703551471233368,
+          -0.0027088848873972893,
+          -0.0050852554850280285,
+          -0.007137978449463844,
+          0.008184290491044521,
+          -0.019525252282619476,
+          -0.0007836258737370372,
+          0.00467737065628171,
+          0.0001997858053073287,
+          -0.019330177456140518,
+          0.025022825226187706,
+          0.0056660473346710205,
+          -0.02092624641954899,
+          0.009328140877187252,
+          -0.015029655769467354,
+          0.012564615346491337,
+          0.020270084962248802,
+          0.0011560420971363783,
+          0.01045425608754158,
+          0.00598082784563303,
+          0.009425678290426731,
+          -0.0026445987168699503,
+          -0.00044169012107886374,
+          -0.01711341366171837,
+          -0.00751039432361722,
+          -0.0014708224916830659,
+          0.012812892906367779,
+          -0.003662093309685588,
+          0.00984242931008339,
+          -0.010064106434583664,
+          0.009266071021556854,
+          -0.0030125814955681562,
+          -0.005107422824949026,
+          -0.013433586806058884,
+          0.003036965848878026,
+          0.001497423741966486,
+          0.012919297441840172,
+          -0.007173446472734213,
+          -0.0029638127889484167,
+          -0.022256305441260338,
+          0.007120244204998016,
+          0.011677909642457962,
+          -0.01788484677672386,
+          0.017787309363484383,
+          0.004797075875103474,
+          -0.014107482507824898,
+          -0.01023257989436388,
+          0.011562637984752655,
+          -0.0019540770445019007,
+          -0.009381342679262161,
+          0.012546881102025509,
+          -0.018177460879087448,
+          -0.009736024774610996,
+          7.412302693410311e-06,
+          0.0205538310110569,
+          0.0022012460976839066,
+          0.010560660623013973,
+          -0.007696602493524551,
+          0.00886262021958828,
+          0.002527110278606415,
+          0.0010834431741386652,
+          0.010773470625281334,
+          -4.464700032258406e-05,
+          0.010365585796535015,
+          0.007625666446983814,
+          -0.014009945094585419,
+          -0.006707926280796528,
+          -0.021103588864207268,
+          -0.001193727133795619,
+          -0.001488556619733572,
+          -0.014027679339051247,
+          0.0068497988395392895,
+          -0.0012070277007296681,
+          0.006938469596207142,
+          0.016882870346307755,
+          -0.004743873607367277,
+          -0.011500569060444832,
+          0.0004724477184936404,
+          0.011935054324567318,
+          0.006836498621851206,
+          -0.004664070438593626,
+          -0.014497633092105389,
+          0.015747888013720512,
+          0.010613863356411457,
+          -0.0005594556569121778,
+          0.0050852554850280285,
+          -0.01879815384745598,
+          -0.01904643140733242,
+          -0.014568569138646126,
+          -0.0050675212405622005,
+          0.010046372190117836,
+          0.004205200355499983,
+          0.0033273620065301657,
+          -0.0014386795228347182,
+          0.0023076506331562996,
+          -0.00571481604129076,
+          -0.00014935444050934166,
+          0.03057360090315342,
+          -0.0015805523144081235,
+          -0.0003696453059092164,
+          0.012103528715670109,
+          0.014905516989529133,
+          0.012812892906367779,
+          -0.005151758436113596,
+          0.012653285637497902,
+          0.00907986331731081,
+          0.010480857454240322,
+          0.01709567941725254,
+          -0.02693810872733593,
+          0.02135186456143856,
+          -0.011261158622801304,
+          -0.010631597600877285,
+          0.018106523901224136,
+          0.003500269493088126,
+          -0.0022400394082069397,
+          -0.002502725925296545,
+          0.0059098913334310055,
+          0.01674986444413662,
+          -0.003901503747329116,
+          -0.017849380150437355,
+          0.0004815918509848416,
+          -0.013123239390552044,
+          -0.00751039432361722,
+          -0.017503563314676285,
+          -0.017476962879300117,
+          0.0054266368970274925,
+          -0.0028396740090101957,
+          0.008844885975122452,
+          0.003846084466204047,
+          -0.021050386130809784,
+          -0.010463123209774494,
+          -0.005741417407989502,
+          -0.001937451190315187,
+          0.06781522929668427,
+          -0.005169492214918137,
+          0.018008986487984657,
+          -0.0013566592242568731,
+          0.006220238283276558,
+          0.0070936428382992744,
+          -0.01851440779864788,
+          -0.006681324914097786,
+          -0.005626145750284195,
+          -0.03383667767047882,
+          0.006162602454423904,
+          0.013735066168010235,
+          -0.007749804761260748,
+          0.005267030093818903,
+          0.0003879335999954492,
+          -0.020890777930617332,
+          -0.020766640082001686,
+          -0.006996105425059795,
+          0.001219219877384603,
+          -0.008663111366331577,
+          0.021546941250562668,
+          -0.007944880053400993,
+          -0.0011349829146638513,
+          0.006167036015540361,
+          0.004531064536422491,
+          -0.0015739020891487598,
+          -0.017202084884047508,
+          0.005630579311400652,
+          -2.2150315999169834e-05,
+          -0.03630171716213226,
+          0.0036975613329559565,
+          -0.0022810494992882013,
+          -0.012334072031080723,
+          -0.006632556207478046,
+          0.011855251155793667,
+          0.016457252204418182,
+          -0.009611885994672775,
+          0.01975579559803009,
+          -0.020075010135769844,
+          0.0014741476625204086,
+          0.007594631519168615,
+          0.011083817109465599,
+          -0.004322688560932875,
+          -0.002511593047529459,
+          -0.020075010135769844,
+          -0.0007210022886283696,
+          0.029988376423716545,
+          -0.03591156750917435,
+          0.0038106164429336786,
+          -0.010445388965308666,
+          -0.004178598988801241,
+          -0.007816308178007603,
+          0.01842573843896389,
+          -0.002203462878242135,
+          -0.025625785812735558,
+          0.017521297559142113,
+          0.005484272725880146,
+          0.002493858803063631,
+          0.0016492720460519195,
+          -0.022486848756670952,
+          0.004699538461863995,
+          -0.022983403876423836,
+          -0.0008013600017875433,
+          -0.008237493224442005,
+          -0.018674015998840332,
+          0.011509436182677746,
+          -0.00984242931008339,
+          0.010525193065404892,
+          -0.024739079177379608,
+          0.006189203355461359,
+          0.0028241565451025963,
+          -0.012786291539669037,
+          -0.011713378131389618,
+          -0.006446348037570715,
+          0.0054399375803768635,
+          -0.024685878306627274,
+          0.0008146605687215924,
+          0.01657252386212349,
+          0.015570546500384808,
+          0.005958660040050745,
+          0.0057901861146092415,
+          -0.006916301790624857,
+          -0.01710454747080803,
+          0.003584506455808878,
+          -0.0038992869667708874,
+          -0.022894732654094696,
+          0.0013622011756524444,
+          -0.0022267389576882124,
+          0.006521718110889196,
+          0.023533159866929054,
+          0.012812892906367779,
+          0.021919356659054756,
+          0.008481336757540703,
+          0.014763644896447659,
+          0.005847821943461895,
+          0.024934155866503716,
+          0.0042428853921592236,
+          -0.023338085040450096,
+          -0.004136480391025543,
+          -0.000975375936832279,
+          0.0014176202239468694,
+          0.014072014018893242,
+          -0.0063887122087180614,
+          -0.024065183475613594,
+          -0.03890863060951233,
+          -0.0058832899667322636,
+          0.013664130121469498,
+          0.00757246371358633,
+          -0.008303996175527573,
+          0.006809897255152464,
+          -0.0019197171786800027,
+          -0.0035468214191496372,
+          0.0027842549607157707,
+          0.0026845005340874195,
+          -0.020979449152946472,
+          0.028924329206347466,
+          0.014728176407516003,
+          0.046818044036626816,
+          0.03204553201794624,
+          0.009904499165713787,
+          -0.003105685580521822,
+          0.01950751803815365,
+          0.001809987355954945,
+          -0.01186411827802658,
+          0.016528187319636345,
+          0.022965669631958008,
+          -0.02498735673725605,
+          0.02126319520175457,
+          -0.008703012950718403,
+          -0.00784290861338377,
+          -0.00407884456217289,
+          -0.011935054324567318,
+          0.0170868132263422,
+          0.031673114746809006,
+          -0.006583787500858307,
+          0.000389319087844342,
+          -0.02417158894240856,
+          0.01710454747080803,
+          -0.010604996234178543,
+          0.016856269910931587,
+          -0.007244382984936237,
+          0.045044634491205215,
+          0.009665088728070259,
+          0.006557186134159565,
+          0.0035623388830572367,
+          -0.011066082864999771,
+          0.0006689083529636264,
+          0.0055507756769657135,
+          -0.002966029569506645,
+          0.004420226439833641,
+          0.019826732575893402,
+          -0.0053246659226715565,
+          0.023001138120889664,
+          0.005107422824949026,
+          -0.009549817070364952,
+          0.004167515318840742,
+          0.014861182309687138,
+          0.002815289655700326,
+          0.020128212869167328,
+          -0.006220238283276558,
+          -0.013353782705962658,
+          -0.0026290814857929945,
+          0.002706668106839061,
+          0.003090168349444866,
+          -0.0019518602639436722,
+          -0.006685758475214243,
+          -0.001582769094966352,
+          -0.01647498644888401,
+          -0.005608411505818367,
+          -0.018017852678894997,
+          -0.01311437226831913,
+          0.002930561313405633,
+          -0.013814869336783886,
+          0.01824839599430561,
+          0.029349947348237038,
+          -0.004087711684405804,
+          -0.0026645497418940067,
+          0.018283864483237267,
+          -0.014825713820755482,
+          -0.013442453928291798,
+          0.004236234817653894,
+          0.010551794432103634,
+          0.005630579311400652,
+          0.003850518027320504,
+          0.03587609902024269,
+          0.003948055673390627,
+          0.010374452918767929,
+          -0.013273979537189007,
+          -0.02372823655605316,
+          -0.000808564480394125,
+          -0.0043670241720974445,
+          -0.008428134955465794,
+          0.00013272871728986502,
+          -0.009470012970268726,
+          -0.009363608434796333,
+          0.01646611839532852,
+          0.01568581722676754,
+          -0.016758732497692108,
+          -0.021582409739494324,
+          0.0001651488710194826,
+          0.0019540770445019007,
+          -0.0037795817479491234,
+          -0.0311942957341671,
+          -0.011722245253622532
+        ],
+        "title": "Preventing Token Dropping with No-Token-Left-Behind"
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "460c555c-6aea-4625-af79-5ad4ea6dd448",
+      "properties": {
+        "page_content": "C Encouraging Exploration Across Experts 29 ",
+        "keyphrases": [
+          "Encouraging Exploration",
+          "Experts"
+        ],
+        "embedding": [
+          0.029896028339862823,
+          0.007837229408323765,
+          -0.022426677867770195,
+          -0.02060125023126602,
+          -0.0056159826926887035,
+          0.03129303827881813,
+          -0.005033894907683134,
+          0.0729425847530365,
+          -0.001533219125121832,
+          -0.0002334171877009794,
+          -0.04105348512530327,
+          -0.01453822385519743,
+          0.016559232026338577,
+          -0.02227766253054142,
+          -0.02035910077393055,
+          0.05126097798347473,
+          0.010049163363873959,
+          0.008787197060883045,
+          -0.022743333131074905,
+          -0.01933462731540203,
+          0.0470513179898262,
+          -0.040829963982105255,
+          -0.06571538001298904,
+          0.061505720019340515,
+          0.04552391916513443,
+          0.010365818627178669,
+          0.024457000195980072,
+          0.007217888254672289,
+          -0.06351741403341293,
+          0.04153778403997421,
+          0.019949311390519142,
+          0.008465884253382683,
+          -0.008838420733809471,
+          -0.03708597645163536,
+          -0.02613341249525547,
+          -0.024401118978857994,
+          0.018803762272000313,
+          -0.012619663029909134,
+          -0.013113273307681084,
+          0.015842100605368614,
+          -0.007562484126538038,
+          -0.030492085963487625,
+          -0.016037682071328163,
+          0.019148359075188637,
+          0.00031636469066143036,
+          -0.002261993009597063,
+          -0.041463274508714676,
+          -0.036601677536964417,
+          -0.014938700012862682,
+          -0.023302137851715088,
+          -0.02192375436425209,
+          -0.0339939258992672,
+          0.00943913497030735,
+          -0.010915310122072697,
+          0.007962960749864578,
+          0.03365864232182503,
+          0.030436204746365547,
+          -0.008545048534870148,
+          0.03002641536295414,
+          0.04511412978172302,
+          0.022780586034059525,
+          -0.0023202018346637487,
+          -0.02482953481376171,
+          -0.000566662463825196,
+          0.009844268672168255,
+          -0.041463274508714676,
+          -0.01307601947337389,
+          0.011865276843309402,
+          -0.006072339601814747,
+          -0.04101623222231865,
+          -0.014473030343651772,
+          0.010552086867392063,
+          -0.02872253768146038,
+          0.029784265905618668,
+          0.021495336666703224,
+          0.015385744161903858,
+          0.06273508816957474,
+          0.03393804654479027,
+          -0.030249936506152153,
+          -0.0006822068826295435,
+          0.0351860411465168,
+          -0.03609875589609146,
+          0.03326747938990593,
+          0.001204921631142497,
+          0.03285769000649452,
+          -0.01790967583656311,
+          -0.03591248765587807,
+          -0.08165992796421051,
+          0.052415840327739716,
+          0.0031362888403236866,
+          -0.02225903607904911,
+          -0.0011286681983619928,
+          0.003944226540625095,
+          -0.010617280378937721,
+          -0.05465105548501015,
+          0.026040278375148773,
+          0.008135258220136166,
+          0.0008545048185624182,
+          0.0029989161994308233,
+          -0.013578942976891994,
+          0.02455013431608677,
+          0.009993283078074455,
+          -0.015963174402713776,
+          -0.022668825462460518,
+          0.030845994129776955,
+          0.0056159826926887035,
+          0.01548819150775671,
+          0.020508116111159325,
+          -0.011343725956976414,
+          0.08039330691099167,
+          -0.027902958914637566,
+          0.005285356659442186,
+          -0.017024902626872063,
+          0.021420828998088837,
+          -0.013923538848757744,
+          -0.02676672302186489,
+          0.000443550874479115,
+          -0.04485335573554039,
+          0.021718857809901237,
+          0.00806540809571743,
+          -0.01486419327557087,
+          0.01780722849071026,
+          0.01503183413296938,
+          -0.029467610642313957,
+          -0.031423427164554596,
+          0.023767806589603424,
+          -0.010598653927445412,
+          0.014119120314717293,
+          0.020526742562651634,
+          0.04522589221596718,
+          0.030752860009670258,
+          0.0222404096275568,
+          -0.004028047434985638,
+          0.024419745430350304,
+          0.041798558086156845,
+          0.027213767170906067,
+          0.046678781509399414,
+          0.02423347719013691,
+          -0.02546284720301628,
+          -0.002768409438431263,
+          -0.03900453820824623,
+          0.018803762272000313,
+          -0.0029407073743641376,
+          0.01745332032442093,
+          -0.010216804221272469,
+          -0.00982564128935337,
+          0.024997176602482796,
+          0.04455532506108284,
+          -0.04906301200389862,
+          0.01417500153183937,
+          0.008279616013169289,
+          0.04973357915878296,
+          0.014910760335624218,
+          -0.009583492763340473,
+          0.03220575302839279,
+          -0.001704353024251759,
+          -0.014454402960836887,
+          0.0264873206615448,
+          -0.020545369014143944,
+          -0.021886499598622322,
+          0.013467182405292988,
+          -0.008177168667316437,
+          0.005229476373642683,
+          0.012498588301241398,
+          0.011893217451870441,
+          0.02715788595378399,
+          0.0011496233055368066,
+          -0.016913142055273056,
+          -0.07972273975610733,
+          0.04734934866428375,
+          0.0036461977288126945,
+          -0.018338093534111977,
+          0.04943554848432541,
+          0.002908110385760665,
+          0.0007311022491194308,
+          -0.020210087299346924,
+          -0.0017765318043529987,
+          -0.0043516880832612514,
+          -0.00693382928147912,
+          0.015860727056860924,
+          -0.04433180391788483,
+          -0.0029942593537271023,
+          0.022426677867770195,
+          -0.013141212984919548,
+          0.010701101273298264,
+          0.005406430922448635,
+          0.08411866426467896,
+          -0.016335710883140564,
+          0.028163734823465347,
+          0.017742035910487175,
+          0.015246042981743813,
+          -0.031516559422016144,
+          0.003408706048503518,
+          0.018328780308365822,
+          0.013755897991359234,
+          -0.011129518039524555,
+          0.009816328063607216,
+          0.03481350466609001,
+          0.0048429700545966625,
+          -0.025369713082909584,
+          -0.01563720591366291,
+          -0.03447822108864784,
+          0.01683863438665867,
+          0.000613520503975451,
+          0.01678275503218174,
+          0.024307984858751297,
+          -0.009956029243767262,
+          0.02646869421005249,
+          -0.0024750372394919395,
+          -0.01634502410888672,
+          0.016717560589313507,
+          -0.01307601947337389,
+          0.009047972038388252,
+          -0.013914225623011589,
+          -0.005001298151910305,
+          -0.003718376625329256,
+          -0.018487107008695602,
+          0.02516481839120388,
+          -0.0036718095652759075,
+          -0.012265753000974655,
+          0.015702400356531143,
+          0.023972703143954277,
+          0.026226546615362167,
+          -0.014612731523811817,
+          -0.01563720591366291,
+          0.006700994446873665,
+          0.04414553567767143,
+          0.009392567910254002,
+          0.021178681403398514,
+          -0.01552544441074133,
+          -0.005643922835588455,
+          0.03689970821142197,
+          -0.017667528241872787,
+          0.016400905326008797,
+          -0.0005166028859093785,
+          -0.016242576763033867,
+          0.018291525542736053,
+          -0.003939569927752018,
+          0.006426248699426651,
+          0.03639678284525871,
+          0.0035018399357795715,
+          -0.024792281910777092,
+          -0.030417578294873238,
+          0.028312748298048973,
+          0.026617709547281265,
+          0.004137479700148106,
+          -0.027660811319947243,
+          0.04354947805404663,
+          0.026375560089945793,
+          0.014333329163491726,
+          -0.02065712958574295,
+          -0.02587263658642769,
+          -0.010803548619151115,
+          0.011874590069055557,
+          -0.010654534213244915,
+          0.040867216885089874,
+          0.03283906355500221,
+          -0.00010513960296520963,
+          -0.01047758013010025,
+          0.004083927720785141,
+          0.021551217883825302,
+          0.01868268847465515,
+          0.010281998664140701,
+          0.0032131243497133255,
+          -0.01935325376689434,
+          -0.04202207922935486,
+          0.024922668933868408,
+          -0.023767806589603424,
+          0.00702230678871274,
+          -0.03948883339762688,
+          0.006477472838014364,
+          -0.010216804221272469,
+          0.009434478357434273,
+          -0.034385088831186295,
+          0.045337650924921036,
+          0.016754813492298126,
+          -0.006887262221425772,
+          -0.0130946459248662,
+          -0.011800083331763744,
+          0.030175428837537766,
+          -0.002717185765504837,
+          0.02034047432243824,
+          0.027847077697515488,
+          -0.013411302119493484,
+          -0.01632639765739441,
+          0.0054855952039361,
+          -0.052490346133708954,
+          0.02069438435137272,
+          -0.033211600035429,
+          0.009224927052855492,
+          -0.00036293172161094844,
+          -0.04641800746321678,
+          -0.022147275507450104,
+          0.028461763635277748,
+          0.037272244691848755,
+          -0.012424081563949585,
+          -0.016894515603780746,
+          0.004402911756187677,
+          -0.008880331180989742,
+          0.003743988461792469,
+          -0.005853474605828524,
+          -0.02447562664747238,
+          -0.03188909590244293,
+          -0.0067801582626998425,
+          -0.026207920163869858,
+          0.02844313718378544,
+          -0.011902530677616596,
+          0.0119584109634161,
+          -0.03704872354865074,
+          -0.02712063305079937,
+          0.022445304319262505,
+          0.005662549752742052,
+          -0.020470863208174706,
+          -0.04869047552347183,
+          -0.057966627180576324,
+          0.04004763811826706,
+          0.06381544470787048,
+          0.025630488991737366,
+          -0.00016007413796614856,
+          -0.027660811319947243,
+          0.02641281485557556,
+          -0.014938700012862682,
+          -0.037980061024427414,
+          0.01676412671804428,
+          0.011828023009002209,
+          0.007939676754176617,
+          0.01905522495508194,
+          0.028536271303892136,
+          -0.001995396800339222,
+          -0.004321419633924961,
+          -0.031237157061696053,
+          -0.04362398386001587,
+          -0.029467610642313957,
+          -0.02093653194606304,
+          -0.01437058299779892,
+          -0.03574484586715698,
+          -0.01940913498401642,
+          0.024047208949923515,
+          0.005806907545775175,
+          0.005862787831574678,
+          -0.02546284720301628,
+          0.012032918632030487,
+          -0.010980503633618355,
+          0.0053784907795488834,
+          0.010896682739257812,
+          0.0053272671066224575,
+          0.017797915264964104,
+          -0.0036834513302892447,
+          -0.00501061137765646,
+          -0.023786434903740883,
+          0.0012631304562091827,
+          0.011082950979471207,
+          0.0033947357442229986,
+          -0.0089455246925354,
+          0.024270731955766678,
+          0.01614944264292717,
+          -0.03513016179203987,
+          -0.013644137419760227,
+          0.03809182345867157,
+          0.0029290656093508005,
+          0.030827367678284645,
+          -0.07957372814416885,
+          -0.016969023272395134,
+          0.004898850806057453,
+          -0.03652717173099518,
+          -0.0075019472278654575,
+          0.005606669466942549,
+          0.019800297915935516,
+          -0.010570713318884373,
+          -0.0037649436853826046,
+          -0.027902958914637566,
+          -0.009215613827109337,
+          0.02356291189789772,
+          0.011064324527978897,
+          -0.02166297845542431,
+          0.013811778277158737,
+          0.00014312082203105092,
+          -0.01903659850358963,
+          -0.02933722361922264,
+          0.014584790915250778,
+          -0.03540956228971481,
+          -0.002244530478492379,
+          0.009881521575152874,
+          0.000486043281853199,
+          -0.016205323860049248,
+          0.05748233199119568,
+          0.023767806589603424,
+          0.018496420234441757,
+          -2.3683694962528534e-05,
+          0.0024890073109418154,
+          -0.02097378671169281,
+          -0.011734888888895512,
+          -0.05547063425183296,
+          0.002099008532240987,
+          0.02326488308608532,
+          -0.010328564792871475,
+          -0.023395271971821785,
+          -0.018971404060721397,
+          0.01658717356622219,
+          -0.015124968253076077,
+          -0.024699147790670395,
+          0.03514878824353218,
+          -0.010123670101165771,
+          -0.01340198889374733,
+          0.05174527317285538,
+          0.01275005005300045,
+          -0.04347497224807739,
+          -0.00793502014130354,
+          -0.0371604822576046,
+          0.04068094864487648,
+          -0.024028582498431206,
+          -0.03680657222867012,
+          0.0004083345702383667,
+          0.0027055440004915,
+          0.054017744958400726,
+          -0.005108402110636234,
+          0.04623173922300339,
+          0.023991329595446587,
+          0.0029919310472905636,
+          0.004945417400449514,
+          0.05032963678240776,
+          0.011092264205217361,
+          0.051149215549230576,
+          0.01966990903019905,
+          -0.025630488991737366,
+          0.002516947453841567,
+          -0.03645266219973564,
+          0.0035693622194230556,
+          -0.021551217883825302,
+          -0.043288704007864,
+          -0.004991984460502863,
+          0.019204240292310715,
+          -0.01682000793516636,
+          -0.005117715802043676,
+          0.0472375862300396,
+          -0.046678781509399414,
+          -0.03624776750802994,
+          0.01909247785806656,
+          -0.013001511804759502,
+          -0.03159106522798538,
+          0.0020047102589160204,
+          -0.0069245160557329655,
+          -0.031795963644981384,
+          -0.032168496400117874,
+          -5.2824463637080044e-05,
+          0.014230881817638874,
+          -0.029858773574233055,
+          -0.0166058000177145,
+          -0.04515138268470764,
+          0.01451959740370512,
+          -0.036992840468883514,
+          0.03580072522163391,
+          -0.03526054695248604,
+          0.0003108348755631596,
+          0.027847077697515488,
+          -0.009145762771368027,
+          -0.03125578537583351,
+          0.027958840131759644,
+          0.0016065621748566628,
+          -0.01836603321135044,
+          0.010095730423927307,
+          -0.0018673375016078353,
+          0.03453410416841507,
+          -0.030138175934553146,
+          0.0003000662545673549,
+          0.034422341734170914,
+          -0.03295082226395607,
+          -0.02393544837832451,
+          0.01079423539340496,
+          0.05971754714846611,
+          0.008088691160082817,
+          -0.027008872479200363,
+          -0.012358887121081352,
+          0.014808312989771366,
+          -0.009499671868979931,
+          0.009304090403020382,
+          0.020843397825956345,
+          -0.03749576583504677,
+          -0.022426677867770195,
+          -0.007134067825973034,
+          0.00888498779386282,
+          -0.007981587201356888,
+          -0.0022526795510202646,
+          0.036266397684812546,
+          0.017173917964100838,
+          6.915930134709924e-05,
+          0.03369589522480965,
+          -0.020899279043078423,
+          -0.019297374412417412,
+          -0.004482075572013855,
+          0.00969525333493948,
+          -0.017304304987192154,
+          0.00484762666746974,
+          -0.006649770773947239,
+          -0.003913958091288805,
+          -0.04030841216444969,
+          -0.03494389355182648,
+          -0.03477625176310539,
+          0.007203917950391769,
+          0.07160145044326782,
+          -0.009779074229300022,
+          -0.0028918120078742504,
+          0.018058691173791885,
+          0.01743469201028347,
+          -0.001707845483906567,
+          -0.021532591432332993,
+          -0.03198223188519478,
+          -0.028331374749541283,
+          0.008670778945088387,
+          0.00515962578356266,
+          -0.007688215002417564,
+          0.04485335573554039,
+          0.021867873147130013,
+          0.004153778310865164,
+          -0.015395057387650013,
+          0.024736400693655014,
+          -0.024959923699498177,
+          0.03520466759800911,
+          -0.00905262865126133,
+          0.0023958731908351183,
+          -0.0019977251067757607,
+          0.000827728770673275,
+          -0.0044378372840583324,
+          0.03593111410737038,
+          -0.00347389979287982,
+          0.0261892918497324,
+          0.014603417366743088,
+          -0.005308640189468861,
+          0.03168420121073723,
+          -0.0626605823636055,
+          -0.0012316977372393012,
+          0.02004244551062584,
+          0.0047498359344899654,
+          0.001548353466205299,
+          -0.004866253584623337,
+          -0.004782433155924082,
+          0.01258240919560194,
+          0.02354428544640541,
+          -0.003101363545283675,
+          -0.034999772906303406,
+          -0.02067575789988041,
+          0.048876743763685226,
+          0.003289960091933608,
+          0.0009470568038523197,
+          0.02069438435137272,
+          0.009229583665728569,
+          0.03464586287736893,
+          -0.01519016269594431,
+          0.026040278375148773,
+          0.01708078384399414,
+          0.009383254684507847,
+          0.008572988212108612,
+          0.015721026808023453,
+          0.011399607174098492,
+          -0.000368752604117617,
+          -0.026096157729625702,
+          0.018990030512213707,
+          0.01870131492614746,
+          0.0015646519605070353,
+          -0.04675328731536865,
+          0.014975953847169876,
+          0.014510284177958965,
+          -0.0008236541762016714,
+          0.012386827729642391,
+          -0.01875719614326954,
+          0.003408706048503518,
+          -0.011725575663149357,
+          0.016121502965688705,
+          -0.00448673265054822,
+          -0.015003894455730915,
+          -0.00463574705645442,
+          -0.0009453105158172548,
+          -0.017034215852618217,
+          -0.010421698912978172,
+          -0.003874376183375716,
+          0.01773272082209587,
+          0.039190806448459625,
+          -0.045039623975753784,
+          -0.010291311889886856,
+          -0.015553385019302368,
+          -0.011986351571977139,
+          -0.0326155424118042,
+          -0.001544860890135169,
+          -0.00967662688344717,
+          0.006808098405599594,
+          -0.005546132102608681,
+          -0.013523062691092491,
+          -0.04172405228018761,
+          -0.011408920399844646,
+          -0.027344154193997383,
+          -0.013737271539866924,
+          0.02941173128783703,
+          0.031181277707219124,
+          0.0015390400076285005,
+          0.009373941458761692,
+          0.010309938341379166,
+          0.009639373049139977,
+          -0.004787089768797159,
+          -0.015711713582277298,
+          -0.028461763635277748,
+          0.019446387887001038,
+          0.015097028575837612,
+          0.032168496400117874,
+          -0.01337404828518629,
+          0.0002654320269357413,
+          0.019949311390519142,
+          -0.002558857901021838,
+          -0.010729041881859303,
+          -0.0014517268864437938,
+          0.01098981685936451,
+          -0.02028459496796131,
+          -0.004447150509804487,
+          0.03229888528585434,
+          0.0009418179979547858,
+          0.01273142360150814,
+          0.03349100053310394,
+          0.004018733743578196,
+          0.02415897138416767,
+          0.005392461083829403,
+          0.018533675000071526,
+          0.029635252431035042,
+          -0.017816541716456413,
+          0.009220270439982414,
+          0.007990900427103043,
+          0.030175428837537766,
+          0.009169046767055988,
+          -0.014575477689504623,
+          -0.0339939258992672,
+          -0.02004244551062584,
+          0.0001397010637447238,
+          0.03136754408478737,
+          0.0005989682977087796,
+          -0.0033248853869736195,
+          -0.011297158896923065,
+          -0.01771409437060356,
+          0.014612731523811817,
+          0.001167668029665947,
+          0.033360615372657776,
+          0.031162651255726814,
+          -0.01486419327557087,
+          -0.02060125023126602,
+          -0.028275495395064354,
+          0.008647495880723,
+          0.0612821988761425,
+          -0.014240195043385029,
+          -0.011949097737669945,
+          0.029430357739329338,
+          0.015069087967276573,
+          0.020806144922971725,
+          0.003799868980422616,
+          -0.0026450068689882755,
+          0.009909462183713913,
+          0.01965128257870674,
+          0.026617709547281265,
+          -0.0229296013712883,
+          -0.029150955379009247,
+          -0.013653450645506382,
+          -0.012619663029909134,
+          -0.006966426502913237,
+          0.009099195711314678,
+          0.004433180205523968,
+          -0.03811044991016388,
+          0.009117823094129562,
+          0.00342966103926301,
+          -0.012098112143576145,
+          -0.014761745929718018,
+          -0.013588257133960724,
+          0.015804847702383995,
+          -0.01342061534523964,
+          0.02937447652220726,
+          -0.013904912397265434,
+          -0.03028719127178192,
+          -0.013634824194014072,
+          -0.01907385140657425,
+          -0.019763043150305748,
+          -0.034105684608221054,
+          -0.019763043150305748,
+          0.015450937673449516,
+          0.020806144922971725,
+          -0.006165473721921444,
+          -0.017425378784537315,
+          0.0009272657916881144,
+          -0.003071095095947385,
+          0.0020908592268824577,
+          0.028890179470181465,
+          0.009741820394992828,
+          -0.025015803053975105,
+          0.03509290888905525,
+          -0.018198391422629356,
+          0.003418019274249673,
+          -0.011492740362882614,
+          0.022482557222247124,
+          -0.014510284177958965,
+          0.030268562957644463,
+          -0.008018841035664082,
+          -0.006808098405599594,
+          -0.016000429168343544,
+          -0.018179764971137047,
+          0.005909354891628027,
+          0.04008489102125168,
+          -0.009639373049139977,
+          0.008675435557961464,
+          -0.019893432036042213,
+          0.013290227390825748,
+          0.0213463231921196,
+          -0.0006047891802154481,
+          -0.003813839051872492,
+          -0.01159518864005804,
+          -0.024084463715553284,
+          -0.005946608725935221,
+          0.0006787143065594137,
+          0.008852390572428703,
+          -0.0050478652119636536,
+          0.026394188404083252,
+          0.01940913498401642,
+          -0.0019651283510029316,
+          0.01808663085103035,
+          -0.008000214584171772,
+          -0.004433180205523968,
+          0.028610777109861374,
+          -0.002068739850074053,
+          0.0066683972254395485,
+          0.01785379648208618,
+          -0.0118093965575099,
+          -0.022780586034059525,
+          0.0059559219516813755,
+          0.01998656615614891,
+          0.02060125023126602,
+          0.029486237093806267,
+          -0.010570713318884373,
+          0.011129518039524555,
+          -0.014622044749557972,
+          -0.009946716018021107,
+          0.0041235098615288734,
+          0.0004292897356208414,
+          0.01940913498401642,
+          0.010850115679204464,
+          0.022389423102140427,
+          0.0027590959798544645,
+          -0.03609875589609146,
+          -0.002391216577962041,
+          -0.029206834733486176,
+          -0.0017311290139332414,
+          0.0025518727488815784,
+          0.003222437808290124,
+          -0.004312105942517519,
+          0.03416156768798828,
+          -0.011343725956976414,
+          -0.02516481839120388,
+          0.022724706679582596,
+          -0.0038976597134023905,
+          -0.0015809503383934498,
+          0.04656702280044556,
+          -0.030901875346899033,
+          -0.013020139187574387,
+          0.012647602707147598,
+          -0.0008125944877974689,
+          0.012181933037936687,
+          -0.0027102006133645773,
+          0.006118906661868095,
+          0.0175930205732584,
+          0.008503138087689877,
+          0.005401774309575558,
+          0.026375560089945793,
+          0.028331374749541283,
+          0.0010035192826762795,
+          0.009751134552061558,
+          -0.00637968210503459,
+          0.015171535313129425,
+          0.01613081619143486,
+          0.0007421618793159723,
+          -0.0006734755588695407,
+          0.04492786154150963,
+          -0.04392201453447342,
+          0.025891263037919998,
+          -0.024084463715553284,
+          0.018515046685934067,
+          0.005369177553802729,
+          -0.015246042981743813,
+          0.010878056287765503,
+          -0.031814590096473694,
+          -0.011446173302829266,
+          -0.024140343070030212,
+          -0.0009394896333105862,
+          -0.018747882917523384,
+          -0.025313831865787506,
+          0.009993283078074455,
+          0.0009202807559631765,
+          -0.006468159146606922,
+          0.0012631304562091827,
+          -0.008763913065195084,
+          -0.01996793784201145,
+          -0.0021467397455126047,
+          0.010281998664140701,
+          0.029635252431035042,
+          0.01451959740370512,
+          0.007124754134565592,
+          0.008833764120936394,
+          -0.04481610283255577,
+          -0.0322243794798851,
+          0.026729470118880272,
+          0.0029220804572105408,
+          -0.039228059351444244,
+          0.010729041881859303,
+          -0.006836038548499346,
+          -0.01725773885846138,
+          0.03937707468867302,
+          -0.026245173066854477,
+          -0.005704460199922323,
+          0.01733224466443062,
+          0.02393544837832451,
+          -0.008596272207796574,
+          0.0024307984858751297,
+          0.01483625266700983,
+          0.009881521575152874,
+          -0.003434317884966731,
+          0.013271600939333439,
+          -0.013523062691092491,
+          -0.012861811555922031,
+          0.023730553686618805,
+          0.029523491859436035,
+          0.019483640789985657,
+          -0.0024820221588015556,
+          0.0067801582626998425,
+          0.0035903172101825476,
+          0.026394188404083252,
+          0.03619188815355301,
+          -0.02192375436425209,
+          0.03127441182732582,
+          -0.006063025910407305,
+          -0.0355772040784359,
+          0.029430357739329338,
+          -0.0018033079104498029,
+          0.02415897138416767,
+          0.021830620244145393,
+          0.023302137851715088,
+          -0.009061941877007484,
+          -0.015450937673449516,
+          -0.01047758013010025,
+          0.03812907636165619,
+          0.014705865643918514,
+          -0.009583492763340473,
+          -0.015646519139409065,
+          -0.020470863208174706,
+          -0.008512451313436031,
+          0.01909247785806656,
+          -0.015981800854206085,
+          0.026859857141971588,
+          0.02909507416188717,
+          0.016736187040805817,
+          -0.0013271600473672152,
+          0.015469564124941826,
+          -0.02190512605011463,
+          -0.013085332699120045,
+          -0.03423607349395752,
+          -0.006426248699426651,
+          0.0044704340398311615,
+          0.010552086867392063,
+          0.0005579311400651932,
+          -0.006700994446873665,
+          0.02970976009964943,
+          0.01320640742778778,
+          -0.008037467487156391,
+          -0.010831489227712154,
+          0.03168420121073723,
+          -0.01613081619143486,
+          0.00866612233221531,
+          -0.03544681519269943,
+          -0.0072644553147256374,
+          -0.01790967583656311,
+          -0.018012123182415962,
+          0.04753561317920685,
+          -0.0235256589949131,
+          -0.0005762669024989009,
+          0.007483320310711861,
+          -0.008060751482844353,
+          -0.01030062511563301,
+          -0.0185802411288023,
+          0.014901447109878063,
+          0.0013876971788704395,
+          0.0027730660513043404,
+          -0.02522069774568081,
+          -0.0026822604704648256,
+          0.0009488030336797237,
+          -0.049584563821554184,
+          0.004912820644676685,
+          -0.03561445698142052,
+          -0.027251020073890686,
+          -0.00028114838642068207,
+          -0.056327469646930695,
+          0.02132769487798214,
+          -0.019893432036042213,
+          -0.0009668477578088641,
+          -0.025239326059818268,
+          0.012880438007414341,
+          0.01836603321135044,
+          -0.04712582379579544,
+          -0.012275067158043385,
+          -0.011902530677616596,
+          0.011557934805750847,
+          -0.010775608941912651,
+          -0.02553735487163067,
+          0.02484816312789917,
+          -0.038855522871017456,
+          0.007175977807492018,
+          -0.00043103599455207586,
+          0.017211170867085457,
+          0.007613707799464464,
+          0.02259431779384613,
+          0.00892689824104309,
+          0.012275067158043385,
+          -0.011651068925857544,
+          -0.0036881081759929657,
+          -0.03513016179203987,
+          0.004586851689964533,
+          0.008982778526842594,
+          0.02032184787094593,
+          -0.021048294380307198,
+          0.02199826017022133,
+          0.03231751173734665,
+          0.022203154861927032,
+          0.02747454307973385,
+          0.0038347942754626274,
+          0.022519811987876892,
+          0.008107318542897701,
+          -0.03233613818883896,
+          0.017695467919111252,
+          -0.006333115044981241,
+          -0.005192223004996777,
+          -0.020135579630732536,
+          -0.018608180806040764,
+          0.04556117206811905,
+          0.025276578962802887,
+          0.020116953179240227,
+          0.03296945244073868,
+          -0.03226163238286972,
+          -0.014780372381210327,
+          -0.016056308522820473,
+          -0.007231858558952808,
+          -0.013560316525399685,
+          -0.02747454307973385,
+          0.0016461441991850734,
+          -0.011893217451870441,
+          0.007883796468377113,
+          0.005290013737976551,
+          0.02000519260764122,
+          0.012526528909802437,
+          0.042729899287223816,
+          0.013979420065879822,
+          0.0056765200570225716,
+          0.013178466819226742,
+          0.0038813611026853323,
+          -0.009429821744561195,
+          0.03349100053310394,
+          -0.004423866979777813,
+          0.013495123013854027,
+          -0.01972579024732113,
+          0.03585660457611084,
+          0.0030361698009073734,
+          0.01081286184489727,
+          0.01468723826110363,
+          0.00033499152050353587,
+          -0.0336027629673481,
+          -0.005057178437709808,
+          0.027008872479200363,
+          0.036601677536964417,
+          0.012293693609535694,
+          -0.00782791618257761,
+          -0.02455013431608677,
+          0.018077317625284195,
+          -0.006226010620594025,
+          -0.0016985321417450905,
+          0.034105684608221054,
+          0.025090310722589493,
+          0.03103226236999035,
+          -0.014184314757585526,
+          0.0037719286046922207,
+          -0.02000519260764122,
+          0.008018841035664082,
+          0.019781669601798058,
+          0.036955587565898895,
+          -0.01081286184489727,
+          -0.012489275075495243,
+          -0.0006239980575628579,
+          0.03391941636800766,
+          -0.039898622781038284,
+          0.01355100329965353,
+          0.026524575427174568,
+          -0.02553735487163067,
+          0.019856177270412445,
+          -0.007311022374778986,
+          -0.034068431705236435,
+          0.010393759235739708,
+          -0.013774524442851543,
+          0.005350550636649132,
+          0.03255965933203697,
+          0.0006839531124569476,
+          -0.032131243497133255,
+          0.00471025425940752,
+          -0.024661894887685776,
+          -0.019241493195295334,
+          -0.007227201480418444,
+          -0.0002667417284101248,
+          0.03233613818883896,
+          -0.02844313718378544,
+          0.028461763635277748,
+          0.029840147122740746,
+          0.027549048885703087,
+          -0.008805823512375355,
+          0.010356505401432514,
+          -0.0261892918497324,
+          0.037570271641016006,
+          0.011660382151603699,
+          -0.011651068925857544,
+          0.013942166231572628,
+          0.009993283078074455,
+          -0.013513749465346336,
+          0.00011066943261539564,
+          -0.01421225443482399,
+          -0.01320640742778778,
+          -0.02423347719013691,
+          0.002894140314310789,
+          -0.03065972588956356,
+          -0.026897111907601357,
+          0.014500970020890236,
+          0.020824771374464035,
+          -0.004544941242784262,
+          0.00868940632790327,
+          0.00935997162014246,
+          -0.006076996214687824,
+          0.021066920831799507,
+          -0.00177769607398659,
+          0.01084080245345831,
+          -0.012368200346827507,
+          -0.001345786964520812,
+          -0.003157244063913822,
+          0.01854298822581768,
+          -0.07778555154800415,
+          -0.02484816312789917,
+          0.05010611563920975,
+          0.032373394817113876,
+          -0.014789685606956482,
+          -0.002556529361754656,
+          0.010822176001966,
+          -0.0005678266170434654,
+          0.018533675000071526,
+          0.004435508977621794,
+          0.015218102373182774,
+          -0.010961877182126045,
+          -0.011353040114045143,
+          -0.01113883126527071,
+          0.028238240629434586,
+          -0.002343485364690423,
+          -0.01275936421006918,
+          -0.02417759783565998,
+          0.02006107196211815,
+          0.0112692192196846,
+          -0.00757179781794548,
+          -0.0006111921393312514,
+          0.0031619006767868996,
+          -0.01307601947337389,
+          0.00035681980079971254,
+          -0.009145762771368027,
+          -0.01965128257870674,
+          0.019064538180828094,
+          0.025276578962802887,
+          0.009844268672168255,
+          0.01725773885846138,
+          0.029281342402100563,
+          0.008372750133275986,
+          0.01486419327557087,
+          0.005746370181441307,
+          0.007315678987652063,
+          -0.009844268672168255,
+          -0.01047758013010025,
+          0.008274959400296211,
+          0.0032131243497133255,
+          -0.014780372381210327,
+          -0.004163091536611319,
+          0.019614029675722122,
+          0.011399607174098492,
+          0.00034576014149934053,
+          0.009215613827109337,
+          0.022110020741820335,
+          0.013234347105026245,
+          -0.01804937794804573,
+          0.011865276843309402,
+          0.010542773641645908,
+          0.03581935167312622,
+          -0.00903400219976902,
+          -0.02877841889858246,
+          -0.006938486360013485,
+          0.021551217883825302,
+          0.004018733743578196,
+          -0.007418126333504915,
+          0.015683772042393684,
+          -0.01616806909441948,
+          0.07208574563264847,
+          0.0012747722212225199,
+          0.015786219388246536,
+          -0.027027498930692673,
+          -0.014789685606956482,
+          0.007497290149331093,
+          -0.014780372381210327,
+          0.021458083763718605,
+          0.002114142756909132,
+          0.026356933638453484,
+          0.014053926803171635,
+          0.014016672968864441,
+          0.0042399270460009575,
+          0.010542773641645908,
+          -0.0030734234023839235,
+          0.014631357975304127,
+          0.0019709491170942783,
+          0.007078187074512243,
+          -0.00109956378582865,
+          0.012200559489428997,
+          0.004558911547064781,
+          -0.0010244745062664151,
+          -0.0025239326059818268,
+          0.010011909529566765,
+          0.00854039192199707,
+          -0.007143381051719189,
+          0.004463449120521545,
+          -0.003844107501208782,
+          -0.030734233558177948,
+          -0.007418126333504915,
+          0.016037682071328163,
+          0.008503138087689877,
+          0.032392021268606186,
+          -0.0036485260352492332,
+          -0.018570927903056145,
+          0.02840588241815567,
+          0.0003201482759322971,
+          0.010216804221272469,
+          0.0182822123169899,
+          0.005313297268003225,
+          -0.027306901291012764,
+          0.03164694830775261,
+          -0.01905522495508194,
+          0.016652366146445274,
+          -0.023413898423314095,
+          -0.014798998832702637,
+          0.004258553963154554,
+          0.0030478115659207106,
+          -0.029448984190821648,
+          -0.005266730207949877,
+          0.01634502410888672,
+          0.024307984858751297,
+          0.00815388560295105,
+          0.009490358643233776,
+          0.010226117447018623,
+          0.004009420517832041,
+          0.013765211217105389,
+          0.01577690616250038,
+          -0.012796617113053799,
+          -0.007921050302684307,
+          0.0012421752326190472,
+          -0.015646519139409065,
+          0.013904912397265434,
+          -0.00281730480492115,
+          -0.0008748779073357582,
+          0.013886285945773125,
+          -0.02265019901096821,
+          -0.010375131852924824,
+          0.010132983326911926,
+          -0.009834954515099525,
+          -0.004193360451608896,
+          0.038222212344408035,
+          -0.006510069593787193,
+          0.008801166899502277,
+          -0.01565583236515522,
+          0.00038184956065379083,
+          0.012954945676028728,
+          -0.004202673677355051,
+          -0.026580454781651497,
+          -0.04381025210022926,
+          0.004917477257549763,
+          0.02905782125890255,
+          0.013001511804759502,
+          0.04056918993592262,
+          0.0019290389027446508,
+          -0.01996793784201145,
+          -0.025704994797706604,
+          0.02838725596666336,
+          -0.00868940632790327,
+          0.0017846811097115278,
+          -0.025555981323122978,
+          0.013560316525399685,
+          0.00861955527216196,
+          0.012833870947360992,
+          -0.007506603840738535,
+          -0.01501320768147707,
+          0.014277448877692223,
+          0.014072554185986519,
+          -0.03220575302839279,
+          0.007539200596511364,
+          0.004200345370918512,
+          0.0038557492662221193,
+          0.011520680971443653,
+          -0.01844054087996483,
+          -0.009974655695259571,
+          0.0004563567927107215,
+          0.0036881081759929657,
+          -0.012787303887307644,
+          0.0006915202829986811,
+          0.015273982658982277,
+          -0.02130906842648983,
+          -0.016540605574846268,
+          -0.01785379648208618,
+          -0.007674245163798332,
+          -0.003962853457778692,
+          0.00456123985350132,
+          0.01455685030668974,
+          -0.013802465051412582,
+          -0.008475197479128838,
+          -0.005136342253535986,
+          -0.004558911547064781,
+          0.02747454307973385,
+          0.006808098405599594,
+          -0.02613341249525547,
+          0.0068127550184726715,
+          0.012200559489428997,
+          -0.0011408920399844646,
+          -0.011082950979471207,
+          0.024065837264060974,
+          0.0008603257010690868,
+          0.003939569927752018,
+          -0.000613520503975451,
+          0.007902423851191998,
+          0.02229628898203373,
+          -0.00866612233221531,
+          0.007851200178265572,
+          0.014435776509344578,
+          0.013737271539866924,
+          0.00621669739484787,
+          -0.014649984426796436,
+          -0.012293693609535694,
+          -0.011362353339791298,
+          0.017071470618247986,
+          0.013485808856785297,
+          -0.012768677435815334,
+          -0.022110020741820335,
+          -0.012628976255655289,
+          -0.0011630113003775477,
+          0.0070688738487660885,
+          0.00862886942923069,
+          -0.008754599839448929,
+          0.023153122514486313,
+          0.020731637254357338,
+          0.004405240062624216,
+          -0.0046147918328642845,
+          -0.013737271539866924,
+          0.030119549483060837,
+          0.0014261150499805808,
+          0.0351860411465168,
+          0.013485808856785297,
+          0.005709116812795401,
+          0.012722110375761986,
+          -0.005541475489735603,
+          -0.018971404060721397,
+          0.029523491859436035,
+          0.0035181385464966297,
+          -0.010067789815366268,
+          0.06199001893401146,
+          0.0020198444835841656,
+          -0.0015972488326951861,
+          -0.008386720903217793,
+          -0.008866360411047935,
+          0.014426463283598423,
+          -0.004414553754031658,
+          0.01834740675985813,
+          -0.0015169207472354174,
+          0.01563720591366291,
+          -0.00870803277939558,
+          -0.006044399458914995,
+          0.0037020782474428415,
+          0.004223628900945187,
+          -0.007054903544485569,
+          0.014808312989771366,
+          0.02201688662171364,
+          0.006901232525706291,
+          -0.036974214017391205,
+          -0.02741866186261177,
+          -0.014249508269131184,
+          -0.037868302315473557,
+          0.014733805321156979,
+          0.008423973806202412,
+          9.793626668397337e-05,
+          0.014277448877692223,
+          -0.035968367010354996,
+          0.012247126549482346,
+          0.0122564397752285,
+          -0.0027614242862910032,
+          -0.0062120407819747925,
+          -0.0004831328405998647,
+          -0.02028459496796131,
+          -0.0011024741688743234,
+          0.01875719614326954,
+          0.013616196811199188,
+          -0.0007695200038142502,
+          0.02132769487798214,
+          -0.00475449301302433,
+          0.014612731523811817,
+          -0.006654427386820316,
+          0.025630488991737366,
+          -0.013830405659973621,
+          -0.017043529078364372,
+          -0.009788387455046177,
+          0.01468723826110363,
+          0.021532591432332993,
+          0.009467075578868389,
+          0.03034307062625885,
+          -0.007981587201356888,
+          0.009411195293068886,
+          -0.013849032111465931,
+          -0.014994580298662186,
+          0.00266130524687469,
+          0.024103090167045593,
+          0.004801060073077679,
+          0.015422997064888477,
+          0.013830405659973621,
+          -0.011772142723202705,
+          0.007939676754176617,
+          -0.007371559273451567,
+          0.0031130053102970123,
+          -0.008731316775083542,
+          -0.005969892255961895,
+          0.0038487643469125032,
+          -0.0006664904649369419,
+          0.012470648624002934,
+          0.002131605288013816,
+          0.018393972888588905,
+          -0.007595081347972155,
+          -0.005578729324042797,
+          -0.021811991930007935,
+          -0.001249160384759307,
+          0.0257981289178133,
+          -0.028554897755384445,
+          -0.02425210364162922,
+          0.016037682071328163,
+          0.0023073959164321423,
+          -0.0003850510693155229,
+          0.01243339478969574,
+          -0.04101623222231865,
+          -0.033360615372657776,
+          -0.0235256589949131,
+          -0.014854880049824715,
+          -0.0001042664734995924,
+          0.015078401193022728,
+          -0.01611218973994255,
+          -0.010486893355846405,
+          0.0219423808157444,
+          -0.015199475921690464,
+          0.024047208949923515,
+          -0.014286762103438377,
+          -0.0028359314892441034,
+          -0.004363330081105232,
+          0.006030429154634476,
+          0.00011692687985487282,
+          0.019893432036042213,
+          -0.019129732623696327,
+          -0.0030757517088204622,
+          -0.02071301080286503,
+          -0.001526234089396894,
+          0.00614219019189477,
+          0.001925546326674521,
+          -0.024587387219071388,
+          0.0016915469896048307,
+          0.020824771374464035,
+          -0.016615113243460655,
+          0.02715788595378399,
+          -0.01909247785806656,
+          0.020862026140093803,
+          -0.009611433371901512,
+          0.011688321828842163,
+          -0.008228392340242863,
+          -0.024307984858751297,
+          0.06638594716787338,
+          -0.0069245160557329655,
+          -0.00215488881804049,
+          -0.016456784680485725,
+          0.0016473083524033427,
+          -0.021066920831799507,
+          0.02581675723195076,
+          0.006896575912833214,
+          0.019427761435508728,
+          -0.014091180637478828,
+          0.026356933638453484,
+          -0.002856886712834239,
+          0.0064402190037071705,
+          0.02741866186261177,
+          -0.0018719942308962345,
+          -0.016792068257927895,
+          0.005313297268003225,
+          0.02291097491979599,
+          -0.020172834396362305,
+          -0.01747194677591324,
+          -0.021197307854890823,
+          -0.003471571486443281,
+          -0.004633418750017881,
+          -0.011259905993938446,
+          -0.002584469737485051,
+          0.002989602740854025,
+          -0.03349100053310394,
+          0.024680521339178085,
+          0.0036275710444897413,
+          -0.005108402110636234,
+          0.004400583449751139,
+          -0.026710843667387962,
+          0.000846355629619211,
+          0.00281730480492115,
+          -0.012694169767200947,
+          0.011716262437403202,
+          0.01646609790623188,
+          0.008447257801890373,
+          0.002136262133717537,
+          0.011846650391817093,
+          0.017015589401125908,
+          -0.017760662361979485,
+          0.008177168667316437,
+          -0.011362353339791298,
+          0.018654748797416687,
+          -0.010701101273298264,
+          -0.01371864415705204,
+          -0.01569308526813984,
+          0.001958143198862672,
+          -0.02060125023126602,
+          -0.004074614495038986,
+          0.00590469827875495,
+          -0.025071684271097183,
+          -0.0032573631033301353,
+          0.019837550818920135,
+          0.005057178437709808,
+          0.020862026140093803,
+          0.00023560001864098012,
+          -0.00982564128935337,
+          -0.031497932970523834,
+          -0.0025448875967413187,
+          -0.021495336666703224,
+          -0.012265753000974655,
+          0.016298457980155945,
+          0.007604394573718309,
+          -0.015497504733502865,
+          -0.005723086651414633,
+          0.001080936985090375,
+          -0.007734782062470913,
+          -0.035968367010354996,
+          0.03721636161208153,
+          0.006267921067774296,
+          0.014296075329184532,
+          -0.03390078991651535,
+          0.005313297268003225,
+          -0.003185184206813574,
+          -0.0005832519382238388,
+          -0.011511367745697498,
+          0.0032177811954170465,
+          -0.015627892687916756,
+          -0.00793502014130354,
+          -0.01939050666987896,
+          -0.009443791583180428,
+          -0.035670336335897446,
+          -0.021402202546596527,
+          -0.00012332983897067606,
+          0.012666230089962482,
+          0.008228392340242863,
+          0.001559995231218636,
+          0.02030322141945362,
+          -0.025555981323122978,
+          0.019465014338493347,
+          0.0015646519605070353,
+          0.02546284720301628,
+          0.006961769424378872,
+          -0.0007718483684584498,
+          0.008582302369177341,
+          -0.029933281242847443,
+          -0.013727957382798195,
+          -0.01755576767027378,
+          -0.0169038288295269,
+          -0.021551217883825302,
+          0.0064728157594799995,
+          -0.004742851015180349,
+          -0.017695467919111252,
+          -0.012638289481401443,
+          -0.004682314116507769,
+          0.004735866095870733,
+          -0.02287372015416622,
+          0.02255706489086151,
+          0.011278532445430756,
+          0.037942808121442795,
+          -0.01587004028260708,
+          0.02032184787094593,
+          0.00797693058848381,
+          -0.006519382819533348,
+          -0.01325297448784113,
+          -0.027213767170906067,
+          0.002933722222223878,
+          -0.0015553385019302368,
+          -0.014333329163491726,
+          0.004628761671483517,
+          0.012116738595068455,
+          0.026207920163869858,
+          0.027604930102825165,
+          -0.0209551602602005,
+          -0.008717346005141735,
+          -0.011371666565537453,
+          -0.006375025026500225,
+          0.0041048829443752766,
+          -0.019856177270412445,
+          -0.0012002650182694197,
+          -0.036340903490781784,
+          -0.00281730480492115,
+          -0.006351741496473551,
+          0.0003335362998768687,
+          0.006719620898365974,
+          -0.004540284629911184,
+          -0.0007153858314268291,
+          -0.00262405164539814,
+          -0.024680521339178085,
+          0.024028582498431206,
+          -0.021066920831799507,
+          -0.0072644553147256374,
+          -0.018496420234441757,
+          0.028629405423998833,
+          0.0025006490759551525,
+          0.005997832398861647,
+          -0.029914654791355133,
+          -0.011399607174098492,
+          -0.012964258901774883,
+          0.0038301374297589064,
+          0.00043481955071911216,
+          -0.00815388560295105,
+          0.03198223188519478,
+          0.008917584083974361,
+          0.004824343603104353,
+          0.012656915932893753,
+          0.020265966653823853,
+          -0.02004244551062584,
+          -0.019539522007107735,
+          -0.034422341734170914,
+          0.02227766253054142,
+          -0.017779288813471794,
+          -0.006370368413627148,
+          -0.002556529361754656,
+          0.0033318703062832355,
+          -0.007348275743424892,
+          -0.013625510036945343,
+          0.010254058055579662,
+          0.007990900427103043,
+          -0.011259905993938446,
+          -0.002887155395001173,
+          -0.009900148957967758,
+          0.002933722222223878,
+          -0.00788845308125019,
+          0.006458845920860767,
+          -0.009476388804614544,
+          0.007092157378792763,
+          0.0063563985750079155,
+          0.003799868980422616,
+          -0.011474113911390305,
+          0.028908805921673775,
+          0.010859428904950619,
+          0.01871994324028492,
+          0.010011909529566765,
+          -0.017462633550167084,
+          0.013262287713587284,
+          -0.006719620898365974,
+          0.026282425969839096,
+          -0.028461763635277748,
+          -0.021048294380307198,
+          -0.012628976255655289,
+          0.008866360411047935,
+          -0.006314488127827644,
+          -0.0033435120712965727,
+          -0.0006222518277354538,
+          0.0019756059627979994,
+          -0.01737881265580654,
+          -0.0012736080680042505,
+          0.02166297845542431,
+          0.006766187958419323,
+          0.007045590318739414,
+          -0.00759042426943779,
+          0.052788376808166504,
+          0.005886071361601353,
+          0.0219423808157444,
+          0.0003335362998768687,
+          0.01998656615614891,
+          -0.015739653259515762,
+          0.009900148957967758,
+          -0.03231751173734665,
+          -0.00177420349791646,
+          0.007520573679357767,
+          -0.007786005735397339,
+          0.0009982804767787457,
+          -0.003443631110712886,
+          0.009769761003553867,
+          0.002465723780915141,
+          -0.020489489659667015,
+          -0.027027498930692673,
+          -0.012954945676028728,
+          0.01650335267186165,
+          0.009038658812642097,
+          -0.011576561257243156,
+          -0.00351115339435637,
+          -0.00550422165542841,
+          0.0026636335533112288,
+          0.003341183764860034,
+          0.028834300115704536,
+          -0.038892775774002075,
+          0.0011263397755101323,
+          -0.00023865597904659808,
+          0.030920501798391342,
+          0.0037416601553559303,
+          -0.016046995297074318,
+          -0.00021944708714727312,
+          0.012470648624002934,
+          0.008526421152055264,
+          0.007236515171825886,
+          -0.0005649161757901311,
+          0.004423866979777813,
+          0.007031620014458895,
+          -0.009499671868979931,
+          0.014808312989771366,
+          -0.015283295884728432,
+          -0.009318061172962189,
+          -0.004035032354295254,
+          -0.007967617362737656,
+          0.003930256702005863,
+          0.005448341369628906,
+          0.0031293039210140705,
+          0.0007840722100809216,
+          -0.0006030429503880441,
+          -0.0012573095737025142,
+          -0.006468159146606922,
+          0.018328780308365822,
+          0.01998656615614891,
+          -0.0185802411288023,
+          0.0016158756334334612,
+          0.025276578962802887,
+          -0.025053057819604874,
+          0.014147060923278332,
+          0.023041361942887306,
+          0.003681123023852706,
+          -0.002554201055318117,
+          0.005648579448461533,
+          0.0007252813666127622,
+          0.010552086867392063,
+          0.007837229408323765,
+          0.01785379648208618,
+          0.010505519807338715,
+          -0.014482343569397926,
+          0.01771409437060356,
+          -0.006198070477694273,
+          -0.0011414741165935993,
+          -0.009243553504347801,
+          0.010784922167658806,
+          -0.002402858342975378,
+          0.004978014621883631,
+          -0.00016021965711843222,
+          -0.00423527043312788,
+          -0.007217888254672289,
+          0.013197094202041626,
+          0.00355539214797318,
+          0.017173917964100838,
+          -0.008023497648537159,
+          -0.0038720478769391775,
+          -0.02199826017022133,
+          -0.020265966653823853,
+          0.0024447685573250055,
+          0.019800297915935516,
+          0.02326488308608532,
+          -0.01666167937219143,
+          0.0074926335364580154,
+          0.008647495880723,
+          0.02265019901096821,
+          -0.008801166899502277,
+          -0.014147060923278332,
+          0.0034133626613765955,
+          -0.011073637753725052,
+          0.004714910872280598,
+          -0.012694169767200947,
+          -0.015301923267543316,
+          0.016689620912075043,
+          0.0212531890720129,
+          0.002812647959217429,
+          0.013709330931305885,
+          0.00645418930798769,
+          -0.004260882269591093,
+          0.019912058487534523,
+          -0.01678275503218174,
+          -0.017993496730923653,
+          -0.005369177553802729,
+          -0.0030920500867068768,
+          -0.00935997162014246,
+          -0.002068739850074053,
+          0.00021711873705498874,
+          0.0261892918497324,
+          0.007450723554939032,
+          0.003045483259484172,
+          -0.008503138087689877,
+          -0.002073396695777774,
+          -0.009224927052855492,
+          -0.027195140719413757,
+          0.008647495880723,
+          -0.007869826629757881,
+          -0.006742904428392649,
+          0.005103745497763157,
+          -0.01706215739250183,
+          -0.019129732623696327,
+          -0.007725468836724758,
+          -0.0022352170199155807,
+          -0.005681176669895649,
+          0.0166058000177145,
+          0.01771409437060356,
+          -0.022184528410434723,
+          -0.006598547101020813,
+          -0.00918301660567522,
+          -0.0012142350897192955,
+          -0.014957327395677567,
+          0.006724277976900339,
+          0.006826725322753191,
+          0.02199826017022133,
+          -0.023656046018004417,
+          -0.008251676335930824,
+          0.0015576668083667755,
+          -0.0015145923243835568,
+          0.02482953481376171,
+          -0.003285303246229887,
+          -0.017229797318577766,
+          -0.02836862951517105,
+          0.011828023009002209,
+          -0.008111975155770779,
+          -0.00614219019189477,
+          -0.019912058487534523,
+          -0.007450723554939032,
+          0.0017648900393396616,
+          -0.013020139187574387,
+          0.0027777228970080614,
+          -0.02712063305079937,
+          0.012377514503896236,
+          -0.010421698912978172,
+          -0.006673054303973913,
+          -0.0030221997294574976,
+          -0.011446173302829266,
+          0.026561828330159187,
+          0.019818924367427826,
+          0.016866575926542282,
+          0.015003894455730915,
+          0.008638182654976845,
+          0.0006239980575628579,
+          -0.00817251205444336,
+          0.003315571928396821,
+          -0.023693300783634186,
+          0.02548147365450859,
+          -0.002238709479570389,
+          -0.02190512605011463,
+          0.00493610417470336,
+          -0.004659030586481094,
+          -0.01079423539340496,
+          -0.000244331342400983,
+          0.012805930338799953,
+          -0.0003213124582543969,
+          -0.0021642022766172886,
+          0.010747668333351612,
+          0.013336794450879097,
+          0.011362353339791298,
+          0.030510712414979935,
+          0.016084248200058937,
+          -0.013513749465346336,
+          -0.02065712958574295,
+          0.011697635985910892,
+          -0.009723193943500519,
+          0.0044448222033679485,
+          0.025015803053975105,
+          0.017704781144857407,
+          -0.0017800243804231286,
+          -0.009047972038388252,
+          -0.0012771005276590586,
+          0.008079377934336662,
+          0.00935065746307373,
+          0.010552086867392063,
+          -0.012004978023469448,
+          -0.009425165131688118,
+          -0.01049620658159256,
+          -0.0027963495813310146,
+          0.001626353245228529,
+          -0.010859428904950619,
+          -0.0028475732542574406,
+          -0.001629845704883337,
+          0.011110891588032246,
+          0.010254058055579662,
+          -0.006319144740700722,
+          -0.054129503667354584,
+          0.007040933705866337,
+          0.010710414499044418,
+          0.009751134552061558,
+          -0.010505519807338715,
+          0.01870131492614746,
+          0.0012503245379775763,
+          -0.0014796671457588673,
+          0.010775608941912651,
+          0.010766295716166496,
+          0.0024936639238148928,
+          0.013495123013854027,
+          0.01583278737962246,
+          0.011092264205217361,
+          0.006840695161372423,
+          0.029951907694339752,
+          -0.021234560757875443,
+          -0.003932585008442402,
+          -0.022668825462460518,
+          -0.03349100053310394,
+          -0.007064217235893011,
+          0.01422156859189272,
+          0.004244584124535322,
+          0.008982778526842594,
+          0.020433608442544937,
+          0.03494389355182648,
+          0.013672077096998692,
+          -0.002324858447536826,
+          -0.002270142314955592,
+          -0.02162572368979454,
+          0.0037207049317657948,
+          -0.0014249508967623115,
+          0.00477311946451664,
+          0.006971083115786314,
+          -0.004305121023207903,
+          0.002726498991250992,
+          0.0090945390984416,
+          -0.0015588310780003667,
+          -0.00982564128935337,
+          -3.0923412850825116e-05,
+          0.010226117447018623,
+          0.01547887735068798,
+          -0.00524810329079628,
+          -0.017481260001659393,
+          0.007129410747438669,
+          0.022668825462460518,
+          -0.004165419843047857,
+          -0.004298136103898287,
+          -0.006840695161372423,
+          -0.013904912397265434,
+          -0.0024191567208617926,
+          -0.025630488991737366,
+          0.021811991930007935,
+          -0.012982885353267193,
+          0.017648901790380478,
+          0.01664305292069912,
+          -0.009313404560089111,
+          0.00016051069542299956,
+          -0.0027078723069280386,
+          0.03613600879907608,
+          0.0050478652119636536,
+          0.027008872479200363,
+          0.01692245528101921,
+          0.025239326059818268,
+          0.008722002618014812,
+          0.015786219388246536,
+          0.011883903294801712,
+          -0.004025719128549099,
+          0.00954623892903328,
+          0.011101577430963516,
+          0.008479854092001915,
+          -0.0032759900204837322,
+          0.013085332699120045,
+          0.0017660543089732528,
+          -0.018887583166360855,
+          -0.013904912397265434,
+          0.002901125466451049,
+          0.010654534213244915,
+          0.006738247815519571,
+          0.01840328611433506,
+          -0.008260989561676979,
+          -0.01723911054432392,
+          -0.0035088250879198313,
+          0.011697635985910892,
+          0.024661894887685776,
+          -0.01422156859189272,
+          -0.009620746597647667,
+          -0.02747454307973385,
+          0.0028662001714110374,
+          -0.0023341719061136246,
+          0.008503138087689877,
+          -0.0019860833417624235,
+          0.012163305655121803,
+          -0.00026732380501925945,
+          -0.015152908861637115,
+          -0.008614898659288883,
+          0.020396355539560318,
+          -0.026971617713570595,
+          0.017518512904644012,
+          0.0008731316193006933,
+          -0.01483625266700983,
+          0.017406752333045006,
+          0.0021502322051674128,
+          0.014249508269131184,
+          0.007902423851191998,
+          0.0070688738487660885,
+          -0.01245202124118805,
+          0.010207490995526314,
+          -0.024345237761735916,
+          0.010281998664140701,
+          0.011073637753725052,
+          0.0017159946728497744,
+          0.018868956714868546,
+          0.01356962975114584,
+          0.004668343812227249,
+          -0.019427761435508728,
+          0.009066599421203136,
+          -0.02544422075152397,
+          0.013271600939333439,
+          -0.001437756814993918,
+          0.024792281910777092,
+          -0.008600928820669651,
+          -0.0006106100627221167,
+          0.006058369297534227,
+          0.014910760335624218,
+          -0.003485541557893157,
+          -0.0020338145550340414,
+          0.006999023258686066,
+          -0.0006164309452287853,
+          -0.0017707109218463302,
+          0.0005800504586659372,
+          -0.019483640789985657,
+          -0.02294822782278061,
+          -0.0005075805238448083,
+          -0.009089882485568523,
+          0.01613081619143486,
+          -0.02745591476559639,
+          0.001415637438185513,
+          -0.00733430590480566,
+          0.003969838377088308,
+          -0.00020416728511918336,
+          -0.017145976424217224,
+          0.012209872715175152,
+          -0.005266730207949877,
+          0.0016321741277351975,
+          0.023134496062994003,
+          0.03362138941884041,
+          0.00048517013783566654,
+          -0.016223950311541557,
+          0.0175930205732584,
+          0.0066963378340005875,
+          -0.013392675668001175,
+          -0.020768892019987106,
+          -0.01049620658159256,
+          -0.017294991761446,
+          -0.009010718204081059,
+          0.0025518727488815784,
+          0.003837122581899166,
+          -0.0069571128115057945,
+          -0.010803548619151115,
+          0.010682474821805954,
+          -0.013755897991359234,
+          -0.02294822782278061,
+          0.00965800043195486,
+          -0.0012247127015143633,
+          0.016140129417181015,
+          0.0010663848370313644,
+          -0.004260882269591093,
+          0.019632656127214432,
+          -0.01664305292069912,
+          0.009741820394992828,
+          0.017499886453151703,
+          0.02296685427427292,
+          -0.026654962450265884,
+          -0.00935997162014246,
+          -0.012610348872840405,
+          -0.013625510036945343,
+          -0.0008585794712416828,
+          0.02740003541111946,
+          0.012172618880867958,
+          0.004596164915710688,
+          0.015450937673449516,
+          -0.0038417791947722435,
+          0.009089882485568523,
+          -0.025425594300031662,
+          0.016754813492298126,
+          -0.0118093965575099,
+          0.02615203894674778,
+          0.016223950311541557,
+          0.010077103041112423,
+          0.020806144922971725,
+          -0.0028149764984846115,
+          0.0018452182412147522,
+          -0.007380872964859009,
+          0.01773272082209587,
+          0.014715178869664669,
+          0.028312748298048973,
+          -0.011446173302829266,
+          0.016773441806435585,
+          0.005648579448461533,
+          0.005331923719495535,
+          -0.002933722222223878,
+          0.018794449046254158,
+          -0.0038464360404759645,
+          0.002593782963231206,
+          0.011297158896923065,
+          0.00281730480492115,
+          -0.01998656615614891,
+          -0.0023155452217906713,
+          0.012852497398853302,
+          0.0004173569323029369,
+          0.0030315129552036524,
+          0.0005468714516609907,
+          0.010952563025057316,
+          -0.0073575894348323345,
+          -0.018766509369015694,
+          0.017229797318577766,
+          0.016438158228993416,
+          -0.014128434471786022,
+          -0.026524575427174568,
+          -0.007324992213398218,
+          -0.000489535799715668,
+          0.021439457312226295,
+          0.0012293693143874407,
+          0.0047335377894341946,
+          0.014910760335624218,
+          0.010244744829833508,
+          0.003844107501208782,
+          -0.003769600298255682,
+          0.0009255195618607104,
+          -0.018459167331457138,
+          -0.003918614704161882,
+          0.01163244154304266,
+          -0.008330839686095715,
+          -1.7971959096030332e-05,
+          -0.0029593342915177345,
+          -0.008978121913969517,
+          -0.009965342469513416,
+          0.016717560589313507,
+          0.002754439366981387,
+          0.007525230757892132,
+          0.01193047035485506,
+          0.040196653455495834,
+          -0.007157351355999708,
+          -0.004172405228018761,
+          0.024624640122056007,
+          0.004095569718629122,
+          -0.001629845704883337,
+          0.0219423808157444,
+          -0.005178252700716257,
+          -0.01711803674697876,
+          0.0021607098169624805,
+          0.01842191442847252,
+          0.007324992213398218,
+          -0.007082843687385321,
+          0.012181933037936687,
+          -0.014780372381210327,
+          0.020824771374464035,
+          0.004991984460502863,
+          -0.009974655695259571,
+          -0.00937859807163477,
+          0.005313297268003225,
+          0.007939676754176617,
+          0.0043866136111319065,
+          0.024065837264060974,
+          0.019064538180828094,
+          -0.02032184787094593,
+          0.007166664581745863,
+          0.009807014837861061,
+          0.017034215852618217,
+          -0.02097378671169281,
+          -0.001526234089396894,
+          0.009816328063607216,
+          0.01552544441074133,
+          0.0025704996660351753,
+          0.003152587218210101,
+          -0.02004244551062584,
+          -0.005313297268003225,
+          -0.01940913498401642,
+          -0.016810694709420204,
+          -0.016140129417181015,
+          -0.006337771657854319,
+          0.00517359608784318,
+          0.01585141383111477,
+          -0.01646609790623188,
+          0.0033877508249133825,
+          0.0028103196527808905,
+          -0.024047208949923515,
+          -0.01404461357742548,
+          0.0029360507614910603,
+          0.002756767673417926,
+          -0.00086672866018489,
+          -0.0002302157081430778,
+          -0.007529887370765209,
+          -0.018161138519644737,
+          0.016279829666018486,
+          -0.012992198579013348,
+          -0.01404461357742548,
+          0.004889537114650011,
+          0.01402598712593317,
+          0.005993175785988569,
+          -0.0066683972254395485,
+          0.0029057820793241262,
+          -0.015022520907223225,
+          -0.009983968921005726,
+          0.012563781812787056,
+          0.0004697448166552931,
+          -0.02553735487163067,
+          0.00868940632790327,
+          0.02231491729617119,
+          -0.00725979870185256,
+          0.002270142314955592,
+          -0.003681123023852706,
+          -0.01179077010601759,
+          -0.02287372015416622,
+          0.0077813491225242615,
+          -0.005653236526995897,
+          -0.013681390322744846,
+          -0.0002440402895445004,
+          0.03188909590244293,
+          -0.012163305655121803,
+          -0.002957005752250552,
+          0.009723193943500519,
+          -0.0010244745062664151,
+          -0.009862895123660564,
+          -0.0017241439782083035,
+          -0.01826358586549759,
+          0.004225957207381725,
+          0.0022573363967239857,
+          -0.004682314116507769,
+          0.008028154261410236,
+          -0.0109711904078722,
+          -0.018626809120178223,
+          0.01340198889374733,
+          -0.007674245163798332,
+          0.002428470179438591,
+          -0.007511260453611612,
+          -0.006575263570994139,
+          -0.012172618880867958,
+          -0.03466448932886124,
+          0.013299541547894478,
+          -0.012470648624002934,
+          -0.02127181552350521,
+          -0.014584790915250778,
+          -0.005695146508514881,
+          0.014463717117905617,
+          0.004328404553234577,
+          -0.03485075756907463,
+          -0.018077317625284195,
+          0.00970456749200821,
+          0.015441624447703362,
+          -0.009751134552061558,
+          -0.0132436603307724,
+          0.011343725956976414,
+          0.038930028676986694,
+          0.0026776036247611046,
+          -0.007436753250658512,
+          0.000422304670792073,
+          0.01051483303308487,
+          -0.011697635985910892,
+          -0.004032704047858715,
+          0.021849246695637703,
+          0.010086417198181152,
+          -0.0056672063656151295,
+          -0.006193413864821196,
+          0.027828451246023178,
+          0.00942050851881504,
+          0.018338093534111977,
+          -0.006579920183867216,
+          -0.007641647942364216,
+          -0.014761745929718018,
+          -0.004277180880308151,
+          -0.015087714418768883,
+          -0.02097378671169281,
+          0.009834954515099525,
+          0.009862895123660564,
+          -0.011064324527978897,
+          0.006682367529720068,
+          0.0029407073743641376,
+          0.0229296013712883,
+          -0.0072644553147256374,
+          0.003131632227450609,
+          -0.015981800854206085,
+          0.004833656828850508,
+          -0.0036322276573628187,
+          0.0046730004251003265,
+          0.006076996214687824,
+          -0.010309938341379166,
+          0.006426248699426651,
+          -0.017015589401125908,
+          -0.00982564128935337,
+          0.0054390281438827515,
+          -0.005578729324042797,
+          0.00025306266616098583,
+          -0.001689218683168292,
+          0.017015589401125908,
+          -0.002021008636802435,
+          -0.005606669466942549,
+          -0.0004228867474012077,
+          0.012973572127521038,
+          -0.01163244154304266,
+          -0.02158847078680992,
+          0.009266837500035763,
+          0.003974495455622673,
+          0.009881521575152874,
+          -0.005117715802043676,
+          -0.0005270804977044463,
+          -0.009448448196053505,
+          0.0010093401651829481,
+          -0.02713925950229168,
+          -0.01662442646920681,
+          0.003101363545283675,
+          0.0011211010860279202,
+          0.02484816312789917,
+          0.024680521339178085,
+          -0.0012736080680042505,
+          -0.007422782946377993,
+          -0.013765211217105389,
+          -0.0036531828809529543,
+          -0.008116631768643856,
+          0.011706949211657047,
+          -0.01113883126527071,
+          -0.013951479457318783,
+          -7.86727980539581e-07,
+          -0.042096588760614395,
+          0.0005590952932834625,
+          -0.014826939441263676,
+          0.014873506501317024,
+          0.004787089768797159,
+          0.020880652591586113,
+          -0.0036345559637993574,
+          -0.003210796043276787,
+          -0.0045030307956039906,
+          0.008912927471101284,
+          0.009457762353122234,
+          0.013681390322744846,
+          0.004633418750017881,
+          -0.01968853734433651,
+          0.025630488991737366,
+          -0.016261203214526176,
+          -0.03511153534054756,
+          0.016531292349100113,
+          -0.02067575789988041,
+          0.011408920399844646,
+          -0.007874483242630959,
+          0.0025332458317279816,
+          -0.0043586730025708675,
+          0.0209551602602005,
+          -0.01688520237803459,
+          -0.015916608273983,
+          0.00019732775399461389,
+          -0.012610348872840405,
+          0.0008271466940641403,
+          -0.01838465966284275,
+          -0.011539307422935963,
+          -0.006673054303973913,
+          0.02320900373160839,
+          0.006258607842028141,
+          -0.0031805275939404964,
+          -0.025965770706534386,
+          0.015069087967276573,
+          -0.02425210364162922,
+          0.013429928570985794,
+          -0.0004976850468665361,
+          -2.5630051823100075e-05,
+          -0.016931768506765366,
+          0.031069517135620117,
+          0.004633418750017881,
+          0.0031665575224906206,
+          0.010142297483980656,
+          0.023358017206192017,
+          0.005364520940929651,
+          0.005154969170689583,
+          0.0015972488326951861,
+          -1.1186999472556636e-05,
+          -0.0166058000177145,
+          0.01771409437060356,
+          -0.007972273975610733,
+          0.020582623779773712,
+          -0.0111202048137784,
+          0.010067789815366268,
+          0.0037463167682290077,
+          -0.0033202285412698984,
+          0.005825534462928772,
+          -0.005806907545775175,
+          -0.013318167999386787,
+          0.017462633550167084,
+          -0.01974441669881344,
+          0.010933936573565006,
+          -0.006165473721921444,
+          0.005550788715481758,
+          -0.004139808006584644,
+          0.01289906445890665,
+          -0.006901232525706291,
+          -0.02099241316318512,
+          0.008754599839448929,
+          -0.01641021855175495,
+          0.013765211217105389,
+          0.014240195043385029,
+          -0.005718430038541555,
+          -0.013802465051412582,
+          -0.007804632652550936,
+          -0.007324992213398218,
+          0.025127563625574112,
+          -0.018990030512213707,
+          -0.013327481225132942,
+          -0.01062659453600645,
+          0.0006071175448596478,
+          0.006263264454901218,
+          0.014482343569397926,
+          -0.0012922348687425256,
+          0.007059560157358646,
+          0.017248423770070076,
+          -0.012032918632030487,
+          0.0024890073109418154,
+          -0.018114570528268814,
+          0.005704460199922323,
+          -0.0029919310472905636,
+          0.001448234310373664,
+          -0.0013504435773938894,
+          0.026710843667387962,
+          0.014612731523811817,
+          -0.019008658826351166,
+          -0.018487107008695602,
+          -0.006449532229453325,
+          -0.030436204746365547,
+          0.001108295051380992,
+          -0.020135579630732536,
+          -0.02287372015416622,
+          0.03770066052675247,
+          -0.015702400356531143,
+          -0.018971404060721397,
+          0.004533299710601568,
+          -0.005187565926462412,
+          -0.0011059667449444532,
+          -0.004305121023207903,
+          -0.01569308526813984,
+          0.0006711471942253411,
+          -0.018095944076776505,
+          -0.0049547310918569565,
+          0.024643268436193466,
+          0.0021246203687042,
+          0.01417500153183937,
+          -0.002210769336670637,
+          -0.015860727056860924,
+          0.0008812808664515615,
+          -0.005783624015748501,
+          -0.005345894023776054,
+          -0.0016577859641984105,
+          -0.009145762771368027,
+          -0.0023306794464588165,
+          0.013327481225132942,
+          0.011967724189162254,
+          0.0033761090599000454,
+          -0.019166985526680946,
+          -0.005704460199922323,
+          0.021495336666703224,
+          -0.0032317512668669224,
+          -0.013951479457318783,
+          -0.011334412731230259,
+          -0.007050246931612492,
+          0.005611326079815626,
+          0.005201536230742931,
+          0.0028079913463443518,
+          0.001922053750604391,
+          0.011064324527978897,
+          0.013048078864812851,
+          0.00109956378582865,
+          -0.02743728831410408,
+          0.011110891588032246,
+          0.001075698179192841,
+          -0.00645418930798769,
+          -0.006230667233467102,
+          0.00574171356856823,
+          -0.016438158228993416,
+          -0.017565080896019936,
+          -0.01223781332373619,
+          -0.00574171356856823,
+          -0.005518191959708929,
+          0.015078401193022728,
+          -0.030734233558177948,
+          0.010403072461485863,
+          -0.01854298822581768,
+          -0.014053926803171635,
+          0.001118190586566925,
+          0.006896575912833214,
+          -0.0028033347334712744,
+          0.014426463283598423,
+          -0.010132983326911926,
+          0.016475411131978035,
+          -0.01051483303308487,
+          -0.003876704489812255,
+          0.005536818876862526,
+          -0.01688520237803459,
+          -0.0002786745026241988,
+          0.01148342713713646,
+          0.021160054951906204,
+          -0.026282425969839096,
+          0.01514359563589096,
+          -0.0035064967814832926,
+          -0.0053272671066224575,
+          0.025611860677599907,
+          0.00524810329079628,
+          -0.0027777228970080614,
+          -0.002700887154787779,
+          -0.003732346696779132,
+          -0.003078080015257001,
+          -0.012517215684056282,
+          -0.0054855952039361,
+          -0.029635252431035042,
+          -0.011874590069055557,
+          -0.027232393622398376,
+          -0.016056308522820473,
+          0.0017381140496581793,
+          -0.008847733959555626,
+          0.0004837149172089994,
+          -0.01778860203921795,
+          0.008717346005141735,
+          0.024978550150990486,
+          -0.012712797150015831,
+          0.0035530636087059975,
+          0.0028894837014377117,
+          0.008805823512375355,
+          0.005867444444447756,
+          0.007059560157358646,
+          0.012293693609535694,
+          0.0012817572569474578,
+          0.0036718095652759075,
+          0.0029407073743641376,
+          -0.01966990903019905,
+          -0.013606883585453033,
+          -0.010766295716166496,
+          0.018468480557203293,
+          -0.0008836092310957611,
+          -0.027064751833677292,
+          0.0029290656093508005,
+          0.024382492527365685,
+          0.017024902626872063,
+          -0.01713666319847107,
+          0.004731209482997656,
+          0.01648472622036934,
+          0.023488406091928482,
+          0.0036974214017391205,
+          0.041127994656562805,
+          -0.0001826300285756588,
+          0.028461763635277748,
+          0.005415744613856077,
+          0.009397224523127079,
+          0.0002062045969069004,
+          -0.03254103288054466,
+          -0.0020489490125328302,
+          -0.013318167999386787,
+          -0.012051545083522797,
+          0.033397868275642395,
+          0.016615113243460655,
+          -0.017816541716456413,
+          0.0054855952039361,
+          -0.00030937965493649244,
+          0.005588042549788952,
+          -0.02617066539824009,
+          -0.0012631304562091827,
+          0.0069571128115057945,
+          0.006179443560540676,
+          0.00517359608784318,
+          -0.028890179470181465,
+          0.003296945011243224,
+          -0.005210849456489086,
+          -0.0061841001734137535,
+          0.0024075149558484554,
+          -0.009252866730093956,
+          -0.022091394290328026,
+          0.020210087299346924,
+          0.009355314075946808,
+          0.016810694709420204,
+          0.012722110375761986,
+          -0.008940868079662323,
+          -0.008139914833009243,
+          -0.02328351140022278,
+          -0.01066384743899107,
+          0.001526234089396894,
+          0.0019057553727179766,
+          -0.002250351244583726,
+          0.017704781144857407,
+          0.0025379026774317026,
+          -0.00637968210503459,
+          -0.0008271466940641403,
+          -0.0012479961151257157,
+          -0.01579553261399269,
+          -0.003152587218210101,
+          -0.008917584083974361,
+          0.00839137751609087,
+          0.01404461357742548,
+          -0.012619663029909134,
+          -0.007138724438846111,
+          0.014435776509344578,
+          0.005681176669895649,
+          0.024289358407258987,
+          -0.010822176001966,
+          0.0014179657446220517,
+          0.02583538368344307,
+          -0.010878056287765503,
+          -0.010878056287765503,
+          0.012945631518959999,
+          0.015981800854206085,
+          0.014333329163491726,
+          0.0013876971788704395,
+          0.005373834166675806,
+          -0.012377514503896236,
+          -0.007855856791138649,
+          0.009751134552061558,
+          -0.028797045350074768,
+          8.30202698125504e-05,
+          0.00817251205444336,
+          -0.025332460179924965,
+          -0.0009947880171239376,
+          -0.013476495631039143,
+          0.0006007145857438445,
+          -0.010067789815366268,
+          -0.004526314325630665,
+          0.00039320028736256063,
+          0.012042231857776642,
+          -0.007720812223851681,
+          -0.019893432036042213,
+          0.0025774845853447914,
+          -0.020396355539560318,
+          0.021066920831799507,
+          -0.016065621748566628,
+          0.013290227390825748,
+          0.004768462851643562,
+          0.015981800854206085,
+          0.024680521339178085,
+          -0.006831381935626268,
+          -0.005713773425668478,
+          0.01519016269594431,
+          -0.013960792683064938,
+          -0.007557827513664961,
+          -0.0030478115659207106,
+          0.007930363528430462,
+          0.011734888888895512,
+          0.012200559489428997,
+          0.00201169541105628,
+          -0.01179077010601759,
+          0.01081286184489727,
+          -0.00044704339234158397,
+          0.002244530478492379,
+          0.014733805321156979,
+          -0.0072644553147256374,
+          -0.009648687206208706,
+          -0.0029034537728875875,
+          0.0034366461914032698,
+          -0.015199475921690464,
+          -0.00423527043312788,
+          0.0053272671066224575,
+          -0.02939310297369957,
+          -0.013178466819226742,
+          0.0018638450419530272,
+          -0.006896575912833214,
+          -0.03978686407208443,
+          -0.013541690073907375,
+          -0.009006061591207981,
+          -0.008950181305408478,
+          0.0041561066173017025,
+          -0.008475197479128838,
+          -0.012004978023469448,
+          0.021215934306383133,
+          0.027251020073890686,
+          0.024345237761735916,
+          -0.007851200178265572,
+          -0.01131578627973795,
+          0.021495336666703224,
+          -0.02192375436425209,
+          -0.005452997982501984,
+          -0.010980503633618355,
+          -0.0019139045616611838,
+          0.0034506162628531456,
+          0.007795319426804781,
+          0.015758279711008072,
+          -0.0018382332054898143,
+          0.013979420065879822,
+          -0.025053057819604874,
+          0.014202941209077835,
+          -0.004379628226161003,
+          0.0056672063656151295,
+          0.001721815555356443,
+          0.003895331406965852,
+          -0.006887262221425772,
+          0.019297374412417412,
+          -0.002672947011888027,
+          -0.013783838599920273,
+          0.023022735491394997,
+          -0.005839504301548004,
+          -0.002350470284000039,
+          0.0007084007957018912,
+          -0.00501061137765646,
+          0.025239326059818268,
+          0.0111202048137784,
+          -0.011017757467925549,
+          0.016512665897607803,
+          0.006910545751452446,
+          0.014938700012862682,
+          -0.024382492527365685,
+          0.005737056955695152,
+          0.010719728656113148,
+          -0.011706949211657047,
+          0.015786219388246536,
+          -0.02548147365450859,
+          0.018198391422629356,
+          0.0047335377894341946,
+          -0.011949097737669945,
+          -0.019204240292310715,
+          0.030845994129776955,
+          0.02069438435137272,
+          -0.0004354016564320773,
+          0.010803548619151115,
+          -0.017490573227405548,
+          -0.004342374857515097,
+          0.010114356875419617,
+          -0.0020605907775461674,
+          0.00767890177667141,
+          -0.013970105908811092,
+          0.0013131899759173393,
+          -0.009555553086102009,
+          0.007143381051719189,
+          -0.015711713582277298,
+          -0.0010431013070046902,
+          -0.03138617053627968,
+          0.0010221460834145546,
+          -0.01434264238923788,
+          0.0031432739924639463,
+          0.012973572127521038,
+          0.0029593342915177345,
+          0.010281998664140701,
+          0.008116631768643856,
+          0.021644352003932,
+          0.02060125023126602,
+          0.016521979123353958,
+          0.002498320769518614,
+          0.009495015256106853,
+          -0.006193413864821196,
+          0.01178145594894886,
+          0.0013294884702190757,
+          -0.007371559273451567,
+          0.010226117447018623,
+          -0.01852436177432537,
+          -0.013234347105026245,
+          0.023171748965978622,
+          -0.003755630226805806,
+          0.00841000396758318,
+          -0.01709941029548645,
+          -0.006738247815519571,
+          0.01113883126527071,
+          -0.009639373049139977,
+          0.014286762103438377,
+          -0.015897981822490692,
+          0.021122800186276436,
+          0.030529338866472244,
+          -0.016615113243460655,
+          -0.01891552470624447,
+          -0.02646869421005249,
+          0.0064728157594799995,
+          -0.01532986294478178,
+          -0.010095730423927307,
+          -0.002369097201153636,
+          0.0025402309838682413,
+          -3.296072100056335e-05,
+          0.024438371881842613,
+          0.005317953880876303,
+          -0.0013329809298738837,
+          -0.009788387455046177,
+          -0.011744203045964241,
+          0.019129732623696327,
+          0.002738140756264329,
+          -0.00970456749200821,
+          -0.00501061137765646,
+          -0.024121716618537903,
+          -0.0059233251959085464,
+          0.008512451313436031,
+          -0.0027590959798544645,
+          -0.022706080228090286,
+          0.014147060923278332,
+          0.006808098405599594,
+          -0.025555981323122978,
+          -0.004563568159937859,
+          -0.002268978161737323,
+          0.0012584737269207835,
+          -0.007948990911245346,
+          0.007362246047705412,
+          -0.003385422518476844,
+          0.007352932821959257,
+          0.006645113695412874,
+          -0.007511260453611612,
+          0.013998046517372131,
+          0.0005355207249522209,
+          0.01706215739250183,
+          -0.012312320061028004,
+          -0.01569308526813984,
+          -0.01143686007708311,
+          0.014202941209077835,
+          0.0011909515596926212,
+          0.009797701612114906,
+          -0.004184046760201454,
+          -0.0043982551433146,
+          0.0014074882492423058,
+          0.0008958330727182329,
+          0.003196825971826911,
+          -0.01907385140657425,
+          0.03414294123649597,
+          0.009983968921005726,
+          -0.00013373466208577156,
+          7.618073868798092e-05,
+          -0.020414981991052628,
+          0.007460036780685186,
+          0.0005011775647290051,
+          -0.0047521647065877914,
+          -0.004945417400449514,
+          0.0033691239077597857,
+          0.014230881817638874,
+          0.013783838599920273,
+          0.008414660580456257,
+          0.008195796050131321,
+          0.013113273307681084,
+          0.01434264238923788,
+          -0.0012805931037291884,
+          0.011222652159631252,
+          -0.014072554185986519,
+          -0.00235512712970376,
+          -0.016242576763033867,
+          -0.016726873815059662,
+          0.029281342402100563,
+          -0.019465014338493347,
+          0.022799214348196983,
+          0.008177168667316437,
+          0.0024191567208617926,
+          0.024661894887685776,
+          0.006221354007720947,
+          0.025313831865787506,
+          0.017509199678897858,
+          0.015721026808023453,
+          0.0023493061307817698,
+          0.009518299251794815,
+          0.014184314757585526,
+          0.005513535346835852,
+          0.008475197479128838,
+          0.00039640176692046225,
+          -0.005862787831574678,
+          -0.010747668333351612,
+          0.00355539214797318,
+          0.009741820394992828,
+          -0.011883903294801712,
+          -0.008260989561676979,
+          0.017285678535699844,
+          -0.0053272671066224575,
+          -0.011380979791283607,
+          0.00090747483773157,
+          0.005154969170689583,
+          0.012684856541454792,
+          -0.0027404692955315113,
+          -0.007776692509651184,
+          0.002151396358385682,
+          -0.023991329595446587,
+          0.010375131852924824,
+          0.011101577430963516,
+          0.0039046446327120066,
+          0.004917477257549763,
+          0.003913958091288805,
+          0.007720812223851681,
+          0.0004953566822223365,
+          -0.013085332699120045,
+          0.010719728656113148,
+          -0.025891263037919998,
+          -0.008107318542897701,
+          -0.021439457312226295,
+          -0.011585874482989311,
+          -0.007115440908819437,
+          -0.01820770464837551,
+          -0.014817626215517521,
+          0.00671030767261982,
+          -0.021458083763718605,
+          0.0028103196527808905,
+          -0.001522741629742086,
+          -0.01737881265580654,
+          -0.002635693410411477,
+          0.0118093965575099,
+          0.017267052084207535,
+          0.029504865407943726,
+          0.011548621580004692,
+          0.005983862094581127,
+          0.009024688974022865,
+          0.0062446375377476215,
+          -0.007548514287918806,
+          0.03930256515741348,
+          0.0011152802035212517,
+          -0.015758279711008072,
+          0.014016672968864441,
+          -0.014752432703971863,
+          -0.007040933705866337,
+          -0.007036277092993259,
+          0.0053784907795488834,
+          0.0084984814748168,
+          0.020731637254357338,
+          -0.01840328611433506,
+          0.0005104909650981426,
+          0.007562484126538038,
+          0.023153122514486313,
+          0.004745179321616888,
+          0.011716262437403202,
+          -0.0019022627966478467,
+          0.024270731955766678,
+          -0.003932585008442402,
+          -0.0010052655125036836,
+          -0.015376430004835129,
+          -0.008982778526842594,
+          -0.006840695161372423,
+          -0.004731209482997656,
+          -0.017015589401125908,
+          -0.01909247785806656,
+          -0.006095623131841421,
+          0.005303983576595783,
+          -0.007944333367049694,
+          0.010431013070046902,
+          0.007478663697838783,
+          0.014268135651946068,
+          0.018077317625284195,
+          0.025704994797706604,
+          0.010403072461485863,
+          0.005974548868834972,
+          0.010077103041112423,
+          -0.009862895123660564,
+          -0.002845244947820902,
+          -0.006836038548499346,
+          -0.04403377324342728,
+          0.005117715802043676,
+          0.0030245280358940363,
+          -0.01628914475440979,
+          0.0005954757798463106,
+          -0.009965342469513416,
+          0.015376430004835129,
+          0.00629120459780097,
+          -0.011427546851336956,
+          -0.015534758567810059,
+          0.015767592936754227,
+          -0.010440326295793056,
+          0.014873506501317024,
+          0.007422782946377993,
+          0.011604501865804195,
+          -0.006235324312001467,
+          0.019539522007107735,
+          -0.004875567276030779,
+          -0.01535780355334282,
+          0.017481260001659393,
+          0.008633526042103767,
+          -0.0006752217886969447,
+          0.010421698912978172,
+          0.0014191300142556429,
+          -0.02037772908806801,
+          -0.010188864544034004,
+          0.031423427164554596,
+          -0.001983755035325885,
+          -0.015804847702383995,
+          -0.002610081573948264,
+          -0.01062659453600645,
+          0.005769653711467981,
+          0.006645113695412874,
+          -0.004691627342253923,
+          -0.01163244154304266,
+          -0.020526742562651634,
+          -0.018785135820508003,
+          -0.028648031875491142,
+          0.02577950246632099,
+          -0.0017509199678897858,
+          -0.006901232525706291,
+          0.011911843903362751,
+          -5.009592860005796e-05,
+          0.0008358780178241432,
+          -0.0023702613543719053,
+          -0.010542773641645908,
+          0.07178772240877151,
+          -0.019837550818920135,
+          0.010570713318884373,
+          0.00016967857663985342,
+          -0.002754439366981387,
+          0.0017683826154097915,
+          -0.003969838377088308,
+          0.00954623892903328,
+          0.003795212134718895,
+          -0.011744203045964241,
+          -0.01370001770555973,
+          0.003641541115939617,
+          -0.014826939441263676,
+          -0.010524147190153599,
+          -0.0012503245379775763,
+          -0.006575263570994139,
+          -0.019008658826351166,
+          0.007534543983638287,
+          0.0007706842152401805,
+          -0.004507687408477068,
+          -0.008586958982050419,
+          -0.008079377934336662,
+          -0.01776997558772564,
+          0.010235431604087353,
+          -0.0021071576047688723,
+          -0.003415690967813134,
+          -0.03974960744380951,
+          -0.023749180138111115,
+          -0.008871017955243587,
+          -0.028834300115704536,
+          0.00815388560295105,
+          -0.015106341801583767,
+          -0.023786434903740883,
+          0.0068732923828065395,
+          0.007539200596511364,
+          0.006253950763493776,
+          -0.01761164702475071,
+          0.011241278611123562,
+          -0.014053926803171635,
+          -0.002642678329721093,
+          -0.0075531709007918835,
+          0.012889751233160496,
+          0.009145762771368027,
+          -8.149228960974142e-05,
+          -0.013774524442851543,
+          -0.009574179537594318,
+          0.01531123649328947,
+          -0.010086417198181152,
+          0.007893109694123268,
+          -0.004593836609274149,
+          -0.014771059155464172,
+          -0.020526742562651634,
+          0.01755576767027378,
+          0.0033132436219602823,
+          -0.006556636653840542,
+          0.01483625266700983,
+          0.0017450990853831172,
+          -0.00021871947683393955,
+          0.024643268436193466,
+          -0.011185398325324059,
+          -0.011772142723202705,
+          -0.008763913065195084,
+          -0.015544071793556213,
+          0.009630059823393822,
+          -0.019446387887001038,
+          0.011185398325324059,
+          -0.01289906445890665,
+          -0.022352170199155807,
+          -0.005015267990529537,
+          0.00177769607398659,
+          0.02453150600194931,
+          0.0018242630176246166,
+          -0.020563997328281403,
+          -0.012852497398853302,
+          0.006919859442859888,
+          -0.025276578962802887,
+          0.00023836494074203074,
+          0.0002408388099865988,
+          0.015450937673449516,
+          -0.016186697408556938,
+          0.016196010634303093,
+          -0.008600928820669651,
+          -0.0049873278476297855,
+          -0.0028545584063977003,
+          0.016037682071328163,
+          -0.0023621120490133762,
+          -0.024997176602482796,
+          -0.024103090167045593,
+          -0.01273142360150814,
+          -0.013578942976891994,
+          -0.0024075149558484554,
+          0.0226315725594759,
+          0.0024773655459284782,
+          0.026003023609519005,
+          0.007474007084965706,
+          -0.015152908861637115,
+          -0.015711713582277298,
+          -8.636726852273569e-05,
+          0.008265646174550056,
+          0.005411088000983,
+          0.001437756814993918,
+          -0.0017439349321648479,
+          -0.007422782946377993,
+          -0.007762722205370665,
+          -0.0033248853869736195,
+          0.03105088882148266,
+          0.0074414098635315895,
+          -0.0022596647031605244,
+          -0.005029238294810057,
+          -0.002712528919801116,
+          0.005611326079815626,
+          0.0027102006133645773,
+          -0.007162007968872786,
+          0.007767379283905029,
+          -0.004624105058610439,
+          0.0053365807980299,
+          0.025239326059818268,
+          -0.002100172685459256,
+          0.015572011470794678,
+          0.016894515603780746,
+          -0.01533917710185051,
+          -0.026394188404083252,
+          0.025611860677599907,
+          0.0019942326471209526,
+          -0.011520680971443653,
+          -0.011557934805750847,
+          -0.0226315725594759,
+          -0.006337771657854319,
+          -0.004884880501776934,
+          0.006752218119800091,
+          -0.009355314075946808,
+          -0.01695970818400383,
+          -0.006272577680647373,
+          -0.007376216351985931,
+          0.0015658161137253046,
+          -0.003650854341685772,
+          0.017704781144857407,
+          0.01840328611433506,
+          -0.01131578627973795,
+          0.024792281910777092,
+          -0.01532986294478178,
+          0.0567372590303421,
+          0.0015530101954936981,
+          -0.006770845036953688,
+          -0.010142297483980656,
+          -0.012116738595068455,
+          -0.012330947443842888,
+          0.0010780264856293797,
+          0.012507901526987553,
+          -0.0001615293585928157,
+          -0.016261203214526176,
+          -0.008028154261410236,
+          0.00652869651094079,
+          0.007474007084965706,
+          -0.0028010064270347357,
+          0.005308640189468861,
+          0.026952991262078285,
+          0.0002060590632027015,
+          -0.014491656795144081,
+          0.02974701300263405,
+          -0.003955868538469076,
+          0.003459929721429944,
+          0.0024890073109418154,
+          0.008824450895190239,
+          -0.005704460199922323,
+          0.00614219019189477,
+          0.013234347105026245,
+          0.003997778985649347,
+          0.012638289481401443,
+          0.0015611593844369054,
+          -0.0008975793025456369,
+          0.009252866730093956,
+          -0.009555553086102009,
+          0.00508511858060956,
+          -0.002896468620747328,
+          7.610797183588147e-05,
+          -0.003762615378946066,
+          -0.004076942801475525,
+          -0.014985267072916031,
+          0.014491656795144081,
+          -0.006035085767507553,
+          -0.004340046551078558,
+          -0.003755630226805806,
+          -0.011706949211657047,
+          -0.019614029675722122,
+          -0.014333329163491726,
+          0.017937617376446724,
+          -0.016019055619835854,
+          -0.012973572127521038,
+          0.026971617713570595,
+          0.015907295048236847,
+          -0.03125578537583351,
+          -0.013783838599920273,
+          -0.008330839686095715,
+          -0.014566164463758469,
+          0.018626809120178223,
+          -0.0074088131077587605,
+          0.007650961633771658,
+          -0.013280914165079594,
+          0.01453822385519743,
+          0.005876758135855198,
+          -0.021141426637768745,
+          0.010617280378937721,
+          -0.009075912646949291
+        ],
+        "title": "C Encouraging Exploration Across Experts"
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "12f04ba2-cf28-4e5a-aaa8-f47613191ec5",
+      "properties": {
+        "page_content": "D Switch Transformers in Lower Compute Regimes 29 ",
+        "embedding": [
+          0.023223955184221268,
+          0.008607305586338043,
+          -0.01453364733606577,
+          0.043360237032175064,
+          -0.007881038822233677,
+          0.03894453123211861,
+          0.03295178711414337,
+          0.07682663947343826,
+          0.0186422448605299,
+          0.02312435209751129,
+          -0.010591053403913975,
+          -0.009802534244954586,
+          0.008665407076478004,
+          0.01330521795898676,
+          -0.00808024313300848,
+          0.023273756727576256,
+          -0.016957305371761322,
+          0.01791182905435562,
+          0.009520327672362328,
+          -0.021547315642237663,
+          0.02274254336953163,
+          0.002058449201285839,
+          -0.04790874570608139,
+          0.03579045459628105,
+          -0.01648419350385666,
+          0.01867544651031494,
+          -0.016533995047211647,
+          -0.012840406969189644,
+          -0.023522762581706047,
+          0.02939930371940136,
+          -0.0005597446579486132,
+          0.009113618172705173,
+          -0.018758447840809822,
+          -0.025431808084249496,
+          -0.048473160713911057,
+          -0.027888666838407516,
+          -0.008001391775906086,
+          -0.017513418570160866,
+          -0.04266301915049553,
+          0.006719010882079601,
+          -0.04774274304509163,
+          -0.01855924353003502,
+          -0.026710039004683495,
+          0.03499363735318184,
+          0.02893449179828167,
+          0.0012471048394218087,
+          -0.011661779135465622,
+          0.01660039648413658,
+          0.01438424363732338,
+          -0.01910705678164959,
+          -0.009603329934179783,
+          -0.057968586683273315,
+          0.004353453870862722,
+          0.01889125257730484,
+          -0.026079224422574043,
+          0.02126510813832283,
+          0.025049999356269836,
+          0.11447633802890778,
+          -0.006486604921519756,
+          -0.0010707256151363254,
+          0.02028568461537361,
+          -0.02486739493906498,
+          -0.01766282320022583,
+          -0.03291858732700348,
+          -0.028585882857441902,
+          -0.01542176865041256,
+          -0.03071073442697525,
+          0.007947440259158611,
+          0.0005597446579486132,
+          -0.03542524576187134,
+          -0.04130178689956665,
+          -0.015604373067617416,
+          0.010881559923291206,
+          0.015371967107057571,
+          0.010873259976506233,
+          0.010449949651956558,
+          0.0406709723174572,
+          0.034628428518772125,
+          -0.02661043591797352,
+          0.01175308134406805,
+          0.01771262288093567,
+          -0.032221369445323944,
+          0.018625644966959953,
+          0.012201291508972645,
+          0.01673319935798645,
+          -0.001978559885174036,
+          -0.052291251718997955,
+          -0.04757673665881157,
+          0.03821411356329918,
+          0.00491371750831604,
+          -0.021846123039722443,
+          -0.02601282112300396,
+          0.003295178757980466,
+          -0.020401887595653534,
+          -0.008972514420747757,
+          0.04877196624875069,
+          -0.006113096140325069,
+          -0.01459174882620573,
+          0.025398606434464455,
+          0.008989115245640278,
+          0.0039674947038292885,
+          0.016807902604341507,
+          0.0043908050283789635,
+          -0.008997415192425251,
+          0.030146321281790733,
+          -0.011819482780992985,
+          -0.014367643743753433,
+          0.04452226310968399,
+          0.0186422448605299,
+          0.048738766461610794,
+          0.013047912158071995,
+          -0.024551987648010254,
+          -0.022344134747982025,
+          0.02325715683400631,
+          -0.003189351176843047,
+          -0.026710039004683495,
+          0.003627186641097069,
+          -0.012159790843725204,
+          -0.000483227166114375,
+          -0.00844960194081068,
+          0.026378029957413673,
+          0.013056212104856968,
+          -0.008881212212145329,
+          -0.01224279310554266,
+          -0.02101610228419304,
+          0.012201291508972645,
+          0.00771088432520628,
+          -0.02647763304412365,
+          0.006582057569175959,
+          0.01943906396627426,
+          0.0009275471675209701,
+          0.015172762796282768,
+          -0.00734982592985034,
+          -0.034163616597652435,
+          0.01629328913986683,
+          0.00948712695389986,
+          0.035724055022001266,
+          0.04681311920285225,
+          -0.001914233318530023,
+          -0.00859900563955307,
+          -0.0402393639087677,
+          -0.0012087164213880897,
+          0.0311423446983099,
+          0.00649075536057353,
+          -0.05415049567818642,
+          0.02388797141611576,
+          -0.005996893160045147,
+          0.03655407577753067,
+          0.004627360496670008,
+          0.005270625930279493,
+          -0.00020218764257151634,
+          0.03871212527155876,
+          -0.018775049597024918,
+          -0.0005281001213006675,
+          0.07596341520547867,
+          0.040139760822057724,
+          -0.02481759339570999,
+          -0.005204224493354559,
+          -0.0001957031199708581,
+          -0.01050805114209652,
+          0.015521370805799961,
+          0.022427136078476906,
+          -0.030810337513685226,
+          -0.024004174396395683,
+          0.005996893160045147,
+          0.004648111294955015,
+          0.04249701648950577,
+          0.007715034298598766,
+          -0.05577733367681503,
+          0.023622365668416023,
+          0.03731769323348999,
+          0.0220287274569273,
+          0.031374748796224594,
+          0.022559938952326775,
+          -0.004332703538239002,
+          0.01850944198668003,
+          -0.006075745448470116,
+          0.010151143185794353,
+          0.023921171203255653,
+          -0.021032702177762985,
+          -0.032885387539863586,
+          -0.016002783551812172,
+          -0.0109064606949687,
+          -0.017563220113515854,
+          0.003915618639439344,
+          0.011238468810915947,
+          0.02104930393397808,
+          -0.001071763108484447,
+          0.02762306109070778,
+          0.003759989980608225,
+          0.005399279296398163,
+          0.008134194649755955,
+          0.01791182905435562,
+          0.010383548215031624,
+          -0.019638270139694214,
+          -0.04359264299273491,
+          -0.00263116299174726,
+          0.035392045974731445,
+          -0.0014027334982529283,
+          -0.0031416250858455896,
+          -0.00401937123388052,
+          0.008715208619832993,
+          0.020385287702083588,
+          0.01194398570805788,
+          -0.016608696430921555,
+          -0.004839015658944845,
+          0.037550099194049835,
+          -0.016251789405941963,
+          0.009097017347812653,
+          -0.0258468184620142,
+          0.018227236345410347,
+          -0.005399279296398163,
+          0.008931013755500317,
+          0.013272017240524292,
+          -0.0014255590504035354,
+          0.00751997996121645,
+          0.025697413831949234,
+          0.009578429162502289,
+          -0.0377161018550396,
+          -0.01973787136375904,
+          0.021995525807142258,
+          0.00754903070628643,
+          0.023240555077791214,
+          -0.03778250515460968,
+          -0.02330695651471615,
+          0.011454273946583271,
+          0.015637574717402458,
+          -0.002007610397413373,
+          -0.002900919411331415,
+          -0.0402393639087677,
+          -0.038778528571128845,
+          0.006465854588896036,
+          0.014475545845925808,
+          -0.013819830492138863,
+          0.00028843191103078425,
+          -0.02727445214986801,
+          -0.00632060132920742,
+          0.05219164863228798,
+          -0.028287075459957123,
+          0.027888666838407516,
+          0.012334095314145088,
+          0.015313866548240185,
+          -0.01762962155044079,
+          -0.015513070859014988,
+          -0.0150399599224329,
+          0.008806510828435421,
+          -0.015919780358672142,
+          0.0648079514503479,
+          0.03321739286184311,
+          -0.013462921604514122,
+          -0.04116898402571678,
+          -0.01347122248262167,
+          0.002583436667919159,
+          -0.004127273801714182,
+          0.0014608348719775677,
+          0.02964830957353115,
+          -0.003932218998670578,
+          -0.015181062743067741,
+          -0.026129024103283882,
+          0.02016948163509369,
+          0.016102384775877,
+          -0.003091824008151889,
+          -0.030561331659555435,
+          0.029930515214800835,
+          -0.017048608511686325,
+          0.030826937407255173,
+          0.03612246364355087,
+          0.00788933876901865,
+          -0.0036126612685620785,
+          0.004760163836181164,
+          0.00200968561694026,
+          -0.0313415490090847,
+          -0.05806818976998329,
+          0.010325446724891663,
+          0.02085009776055813,
+          -0.032852184027433395,
+          -0.026278428733348846,
+          0.01188588421791792,
+          0.003847141982987523,
+          -0.010026639327406883,
+          0.001663152244873345,
+          0.0011174142127856612,
+          -0.00038388418033719063,
+          -0.007300024386495352,
+          -0.00700121745467186,
+          -0.04126858711242676,
+          -0.0063538020476698875,
+          0.04863916337490082,
+          -0.03834691643714905,
+          -0.0377161018550396,
+          0.005249875597655773,
+          -0.01296490989625454,
+          -0.006594507489353418,
+          0.01120526809245348,
+          0.034163616597652435,
+          -0.03489403426647186,
+          0.05401769280433655,
+          -0.027971668168902397,
+          0.02080029807984829,
+          -0.00859900563955307,
+          -0.015205963514745235,
+          -0.0016185387503355742,
+          0.02825387567281723,
+          -0.02443578466773033,
+          -0.028951091691851616,
+          -0.002340656006708741,
+          0.01588658057153225,
+          0.016044283285737038,
+          0.003556634997949004,
+          0.0351596400141716,
+          0.04485427215695381,
+          0.0056192344054579735,
+          -0.004855616018176079,
+          -0.06620237976312637,
+          -0.025979621335864067,
+          -0.010234144516289234,
+          0.020899899303913116,
+          -0.012400496751070023,
+          -0.011620277538895607,
+          -0.00472281314432621,
+          0.008706907741725445,
+          -0.036155663430690765,
+          -0.05584373697638512,
+          0.05268966034054756,
+          0.02085009776055813,
+          -0.03041192702949047,
+          -0.0025626863352954388,
+          0.026876043528318405,
+          -0.011429373174905777,
+          -0.028320277109742165,
+          -0.017264412716031075,
+          -0.019986877217888832,
+          0.012591401115059853,
+          0.007748235482722521,
+          0.025232603773474693,
+          0.026544034481048584,
+          0.0011475024512037635,
+          -0.020451689139008522,
+          -0.044289860874414444,
+          0.017978230491280556,
+          0.036321669816970825,
+          0.02926649898290634,
+          -0.04004015773534775,
+          -0.03456202521920204,
+          -0.03363240510225296,
+          -0.02186272293329239,
+          0.008798209950327873,
+          -0.007138170767575502,
+          -0.02923329919576645,
+          0.008237946778535843,
+          0.0010442687198519707,
+          -0.007573931012302637,
+          0.014774353243410587,
+          -0.03758329898118973,
+          0.062085483223199844,
+          -0.0478755459189415,
+          -0.02508319914340973,
+          -0.004056721925735474,
+          -0.025182802230119705,
+          -0.04824075475335121,
+          0.010815158486366272,
+          -0.021663518622517586,
+          0.0021912523079663515,
+          -0.024518785998225212,
+          -0.03894453123211861,
+          -0.03834691643714905,
+          0.03426321968436241,
+          -0.008773310109972954,
+          -0.021115705370903015,
+          0.021082503721117973,
+          -0.026129024103283882,
+          -0.0016403266927227378,
+          0.009072116576135159,
+          0.027772463858127594,
+          0.010806858539581299,
+          -0.018360039219260216,
+          0.03758329898118973,
+          -0.012491798959672451,
+          -0.02885149046778679,
+          0.008200596086680889,
+          -0.043526239693164825,
+          0.023605763912200928,
+          -0.04203220456838608,
+          -0.04837355762720108,
+          0.07735785096883774,
+          0.00859900563955307,
+          0.026793040335178375,
+          -0.014351042918860912,
+          -0.04143458977341652,
+          0.0336822047829628,
+          -9.992660488933325e-05,
+          -0.01703200675547123,
+          -0.02063429355621338,
+          -0.012566500343382359,
+          0.0347280316054821,
+          0.021281708031892776,
+          -0.0220287274569273,
+          0.014558548107743263,
+          0.007831237278878689,
+          -0.01910705678164959,
+          -0.005888990592211485,
+          0.013379920274019241,
+          -0.03084353730082512,
+          -0.018609045073390007,
+          0.008823110722005367,
+          0.01296490989625454,
+          -0.06839363276958466,
+          -0.013396520167589188,
+          -0.006573757156729698,
+          -0.00281169218942523,
+          -0.00595124252140522,
+          -0.05162723362445831,
+          0.011852683499455452,
+          -0.013562524691224098,
+          0.02765626087784767,
+          0.0010017302120104432,
+          0.0025128850247710943,
+          0.027805665507912636,
+          -0.022725943475961685,
+          -0.023057950660586357,
+          0.009395824745297432,
+          0.047510337084531784,
+          0.01880824938416481,
+          0.015596073120832443,
+          0.033781807869672775,
+          0.015562872402369976,
+          0.03396441042423248,
+          0.022510137408971786,
+          0.020816897973418236,
+          0.0020958001259714365,
+          0.02377176843583584,
+          -0.0013612326001748443,
+          -0.04163379594683647,
+          -0.009312822483479977,
+          -0.04608270153403282,
+          0.02511640079319477,
+          -0.014691350981593132,
+          0.036155663430690765,
+          -0.012973209843039513,
+          0.0017254037084057927,
+          0.028320277109742165,
+          -0.010383548215031624,
+          -0.02486739493906498,
+          0.020999502390623093,
+          -0.04963518679141998,
+          0.028187474235892296,
+          -0.008200596086680889,
+          0.016168786212801933,
+          -0.0037309392355382442,
+          0.013886231929063797,
+          -0.03469482809305191,
+          0.0258468184620142,
+          -0.013844731263816357,
+          -0.022759143263101578,
+          0.02126510813832283,
+          -0.03851291909813881,
+          0.027772463858127594,
+          0.01624348759651184,
+          -0.015322166495025158,
+          -0.03960854560136795,
+          0.03336679935455322,
+          0.033150993287563324,
+          0.032553378492593765,
+          -0.0347280316054821,
+          0.013338418677449226,
+          0.0423642136156559,
+          -0.05255685746669769,
+          -0.005967842880636454,
+          -0.02685944177210331,
+          0.02426978014409542,
+          -0.01311431359499693,
+          -0.00978593435138464,
+          0.008329248987138271,
+          0.05800178647041321,
+          -0.024087175726890564,
+          -0.0041438741609454155,
+          0.0018260437063872814,
+          -0.07616262137889862,
+          -0.014118637889623642,
+          -0.01370362751185894,
+          0.012425397522747517,
+          0.004959368612617254,
+          0.0026871892623603344,
+          -0.018758447840809822,
+          0.03122534602880478,
+          0.019721271470189095,
+          0.018907852470874786,
+          0.0330347903072834,
+          -0.05036560446023941,
+          0.018758447840809822,
+          -0.005880690645426512,
+          -0.026029422879219055,
+          -0.00212277565151453,
+          -0.0235061626881361,
+          -0.011595377698540688,
+          -0.06328071653842926,
+          -0.044123854488134384,
+          -0.04960198700428009,
+          0.025016797706484795,
+          0.03549164906144142,
+          -0.05109602212905884,
+          -0.021331509575247765,
+          -0.005839189514517784,
+          0.008457901887595654,
+          -0.02227773331105709,
+          -0.03409721702337265,
+          0.006283250171691179,
+          0.0009960237657651305,
+          -0.0025045848451554775,
+          0.0005753075238317251,
+          0.05939621850848198,
+          0.0022472788114100695,
+          0.028917891904711723,
+          -0.022941747680306435,
+          -0.027208050712943077,
+          0.006864264141768217,
+          -0.006191947963088751,
+          0.005486431065946817,
+          0.017463617026805878,
+          -0.007441128138452768,
+          0.006158747244626284,
+          -0.0047975145280361176,
+          0.007466028444468975,
+          0.04621550440788269,
+          -0.005984443239867687,
+          0.023057950660586357,
+          0.018077831715345383,
+          -0.009744432754814625,
+          0.04047176614403725,
+          0.009951937943696976,
+          0.007142320740967989,
+          0.024751191958785057,
+          -0.008947613649070263,
+          -0.021630316972732544,
+          -0.008486952632665634,
+          0.03535884618759155,
+          -0.012035287916660309,
+          -0.007835387252271175,
+          -0.026278428733348846,
+          0.020518090575933456,
+          0.01518936362117529,
+          0.04641471058130264,
+          -0.02219473011791706,
+          -0.009080417454242706,
+          -0.007573931012302637,
+          0.005258175544440746,
+          0.014458945952355862,
+          -0.011869283393025398,
+          0.013645526021718979,
+          0.003253677859902382,
+          -0.01791182905435562,
+          -0.0436258427798748,
+          -0.006768811959773302,
+          -0.009453926235437393,
+          -0.006042544264346361,
+          -0.0072585237212479115,
+          0.0011620278237387538,
+          0.023008150979876518,
+          -0.00014642068708781153,
+          -0.01654229499399662,
+          -0.007428677752614021,
+          0.020833497866988182,
+          0.01635139063000679,
+          0.011346370913088322,
+          0.013886231929063797,
+          -0.00018052931409329176,
+          0.01621858775615692,
+          0.04100298136472702,
+          -0.0070634689182043076,
+          -0.011196968145668507,
+          -0.022095128893852234,
+          0.0177956260740757,
+          -0.00961992982774973,
+          0.004436456132680178,
+          0.00422480097040534,
+          0.05521291866898537,
+          -0.026842841878533363,
+          -0.013944333419203758,
+          -0.015737175941467285,
+          0.022061927244067192,
+          0.00933772325515747,
+          -0.03144115209579468,
+          0.011130565777420998,
+          0.006582057569175959,
+          -0.017994830384850502,
+          -0.024452384561300278,
+          -0.006901614833623171,
+          -0.025747215375304222,
+          0.007187971845269203,
+          -0.0018654696177691221,
+          0.006345501635223627,
+          0.04262981936335564,
+          -0.017015406861901283,
+          -0.01546326931566,
+          0.006598657928407192,
+          0.010889860801398754,
+          0.02159711718559265,
+          -0.010806858539581299,
+          0.022808944806456566,
+          -0.04103618115186691,
+          0.015529671683907509,
+          0.02579701691865921,
+          0.013670426793396473,
+          -0.013230516575276852,
+          0.004448906518518925,
+          0.032984986901283264,
+          -0.02592981979250908,
+          -0.0005654510459862649,
+          -0.002218228066340089,
+          0.009819135069847107,
+          0.03592325747013092,
+          -0.00855750497430563,
+          0.020401887595653534,
+          -0.014168438501656055,
+          -0.017546620219945908,
+          0.024933796375989914,
+          -0.046149104833602905,
+          0.024966996163129807,
+          0.0035524850245565176,
+          -0.01711500994861126,
+          0.035392045974731445,
+          -0.017380615696310997,
+          0.011371271684765816,
+          0.024136977270245552,
+          -0.016177086159586906,
+          0.019040655344724655,
+          0.0055113318376243114,
+          -0.02707524783909321,
+          -0.013985834084451199,
+          -0.0018488692585378885,
+          0.021115705370903015,
+          -0.004357604309916496,
+          -0.013653826899826527,
+          -0.04146779328584671,
+          0.021680118516087532,
+          -0.022725943475961685,
+          0.009927037172019482,
+          -0.020186083391308784,
+          0.020601091906428337,
+          0.004635660909116268,
+          -0.007881038822233677,
+          0.017131609842181206,
+          0.032603178173303604,
+          -0.03213836997747421,
+          -0.019472265616059303,
+          0.013396520167589188,
+          0.012848706915974617,
+          0.02028568461537361,
+          0.007478478830307722,
+          0.020899899303913116,
+          0.03456202521920204,
+          0.009860635735094547,
+          0.0175798200070858,
+          0.017845427617430687,
+          0.013446321710944176,
+          -0.008130044676363468,
+          0.0037952656857669353,
+          -0.007619582116603851,
+          -0.012209592387080193,
+          0.0022970798891037703,
+          -0.0389777310192585,
+          -0.009968538768589497,
+          0.028735287487506866,
+          0.021846123039722443,
+          -0.012168090790510178,
+          -0.033067990094423294,
+          0.000248616881435737,
+          0.02151411399245262,
+          -0.01075705699622631,
+          0.024966996163129807,
+          -0.03502683714032173,
+          -0.0343296192586422,
+          0.006984617095440626,
+          -0.026145625859498978,
+          0.005565282888710499,
+          -0.026544034481048584,
+          -0.00019012641860172153,
+          0.026444431394338608,
+          -0.012217892333865166,
+          0.010350347496569157,
+          -0.01919005811214447,
+          0.020451689139008522,
+          0.008574104867875576,
+          -8.741146302781999e-05,
+          0.012309194542467594,
+          0.007374726235866547,
+          -0.041202183812856674,
+          -0.011429373174905777,
+          -0.02672663889825344,
+          0.03366560488939285,
+          -0.011811182834208012,
+          -0.006113096140325069,
+          -0.009669731371104717,
+          0.013346719555556774,
+          -0.019538667052984238,
+          -0.04189940169453621,
+          0.010823458433151245,
+          0.004980118945240974,
+          0.017231212928891182,
+          -0.021763119846582413,
+          0.008266997523605824,
+          -0.01512296125292778,
+          0.003008821979165077,
+          0.010292246006429195,
+          0.007312474772334099,
+          0.003932218998670578,
+          0.0065447064116597176,
+          0.00808024313300848,
+          -0.01419333927333355,
+          -0.012275993824005127,
+          0.011562176048755646,
+          -0.009570129215717316,
+          -0.00560263404622674,
+          0.06656759232282639,
+          0.03258657827973366,
+          0.0025398607831448317,
+          -0.02320735529065132,
+          -0.01512296125292778,
+          -0.03911053389310837,
+          -0.002927894936874509,
+          0.02661043591797352,
+          0.03057793155312538,
+          -0.01957186870276928,
+          -0.0046066101640462875,
+          -0.012441997416317463,
+          -0.015006759203970432,
+          0.03688608109951019,
+          0.004598309751600027,
+          0.024070575833320618,
+          -0.0029341201297938824,
+          0.0147245517000556,
+          0.009669731371104717,
+          -0.02699224464595318,
+          -0.0002613265533000231,
+          -0.0032702782191336155,
+          -0.006017643958330154,
+          0.012068488635122776,
+          0.03529244288802147,
+          0.0013384070480242372,
+          -0.024468984454870224,
+          0.01668339967727661,
+          0.0003351205086801201,
+          -0.016343090683221817,
+          0.02541520819067955,
+          0.001071763108484447,
+          -0.017181411385536194,
+          0.016500795260071754,
+          0.002089574933052063,
+          -0.017845427617430687,
+          0.039376143366098404,
+          -0.015928080305457115,
+          -0.05544532462954521,
+          0.00670241005718708,
+          -0.0008647768991068006,
+          -0.02694244496524334,
+          0.021464312449097633,
+          0.006914065219461918,
+          -0.00192772108130157,
+          0.01400243490934372,
+          0.014483846724033356,
+          -0.002081274753436446,
+          0.022393936291337013,
+          0.00719212181866169,
+          0.029465705156326294,
+          0.02554801106452942,
+          0.011802881956100464,
+          -0.013263717293739319,
+          -0.044920675456523895,
+          0.013902832753956318,
+          0.05833379551768303,
+          0.0381145104765892,
+          0.031059343367815018,
+          0.01434274297207594,
+          -0.004880516789853573,
+          0.0020750495605170727,
+          0.016284989193081856,
+          0.017148209735751152,
+          -0.03794850781559944,
+          0.0013653826899826527,
+          0.021962326020002365,
+          0.00855750497430563,
+          -0.007627882529050112,
+          -0.02792186848819256,
+          0.001390283228829503,
+          -0.017994830384850502,
+          0.008059492334723473,
+          0.02320735529065132,
+          -0.002274254336953163,
+          0.014624949544668198,
+          0.019173458218574524,
+          -0.00793083943426609,
+          0.016666797921061516,
+          -0.013446321710944176,
+          0.010474850423634052,
+          0.016550594940781593,
+          -0.004197825212031603,
+          -0.021613717079162598,
+          0.01158707682043314,
+          -0.045352283865213394,
+          -0.044289860874414444,
+          0.022642942145466805,
+          0.002205777680501342,
+          -0.020916501060128212,
+          0.004739413503557444,
+          0.00719212181866169,
+          -0.007129870355129242,
+          0.018144233152270317,
+          -0.04037216678261757,
+          0.013371619395911694,
+          -0.001614388544112444,
+          0.031241947785019875,
+          -0.0495687834918499,
+          0.019090456888079643,
+          0.036454472690820694,
+          -0.002919594757258892,
+          -0.01050805114209652,
+          -0.0003117762098554522,
+          -0.006308150943368673,
+          0.00015926006017252803,
+          -0.003587760729715228,
+          0.023024750873446465,
+          0.021215306594967842,
+          0.01321391575038433,
+          -0.023987572640180588,
+          -0.017928428947925568,
+          -0.018658846616744995,
+          0.035093240439891815,
+          -0.008798209950327873,
+          0.026212027296423912,
+          0.046912722289562225,
+          -0.01826043613255024,
+          0.00944562628865242,
+          -0.01071555633097887,
+          -0.01557117234915495,
+          -6.3183638303598855e-06,
+          0.02999691665172577,
+          -0.012026987969875336,
+          0.016152186319231987,
+          -0.03914373740553856,
+          0.010723856277763844,
+          0.018758447840809822,
+          0.013587424531579018,
+          -0.015330466441810131,
+          -0.0317731611430645,
+          -0.016774700954556465,
+          -0.015031659044325352,
+          -0.0003994470462203026,
+          0.006685809697955847,
+          0.03582365810871124,
+          0.05262325704097748,
+          0.04774274304509163,
+          0.024253180250525475,
+          0.008026291616261005,
+          -0.004270452074706554,
+          -0.05879860743880272,
+          0.008399800397455692,
+          0.030212722718715668,
+          0.00123361696023494,
+          -0.019306261092424393,
+          -9.441475413041189e-05,
+          -0.007810486946254969,
+          0.02528240531682968,
+          0.05023280158638954,
+          -0.027855465188622475,
+          0.025232603773474693,
+          -0.0026353129651397467,
+          0.0036126612685620785,
+          0.01194398570805788,
+          0.011852683499455452,
+          -0.04329383373260498,
+          -0.05398448929190636,
+          0.017397215589880943,
+          -0.01243369746953249,
+          -0.0034196816850453615,
+          0.013089412823319435,
+          -0.01782882586121559,
+          -0.024236580356955528,
+          -0.025730615481734276,
+          0.037550099194049835,
+          0.036188866943120956,
+          0.005191774107515812,
+          -0.004502857569605112,
+          -0.013462921604514122,
+          0.00810929387807846,
+          0.03665367513895035,
+          0.022891947999596596,
+          -0.0027535909321159124,
+          -0.0003333048371132463,
+          0.02385476976633072,
+          -0.017895227298140526,
+          -0.006536406464874744,
+          -0.013421420939266682,
+          0.026676837354898453,
+          -0.015537971630692482,
+          0.001002248958684504,
+          0.0018716948106884956,
+          0.007673533633351326,
+          0.01012624241411686,
+          0.009561828337609768,
+          -0.013761729001998901,
+          -0.0043078032322227955,
+          -0.039840951561927795,
+          0.020435089245438576,
+          -0.013180715031921864,
+          0.01867544651031494,
+          -0.010806858539581299,
+          0.0005216156132519245,
+          0.02863568440079689,
+          0.023456361144781113,
+          -0.015347067266702652,
+          -0.010109641589224339,
+          -0.015687374398112297,
+          0.02036868780851364,
+          -0.0073539759032428265,
+          0.028370078653097153,
+          0.003654162399470806,
+          0.0042829024605453014,
+          -0.03046172857284546,
+          0.026195425540208817,
+          0.0037662149406969547,
+          -0.021065903827548027,
+          0.0036375620402395725,
+          0.02481759339570999,
+          0.02765626087784767,
+          -0.019937077537178993,
+          -0.018409840762615204,
+          -0.01621858775615692,
+          0.02008648030459881,
+          -0.0020345861557871103,
+          0.03681968152523041,
+          -0.010425048880279064,
+          0.024170177057385445,
+          0.02194572426378727,
+          -0.025448407977819443,
+          0.0258468184620142,
+          -0.01226769294589758,
+          -0.004320253152400255,
+          0.018360039219260216,
+          -0.013886231929063797,
+          -0.0011091140331700444,
+          -0.027390655130147934,
+          0.016210287809371948,
+          0.00700121745467186,
+          -0.02533220499753952,
+          -0.007055168505758047,
+          0.024767791852355003,
+          -0.011346370913088322,
+          0.029515504837036133,
+          -0.007378876209259033,
+          -0.0032267021015286446,
+          0.0036292618606239557,
+          -0.004519457928836346,
+          -0.01012624241411686,
+          0.009478827007114887,
+          -0.006295700557529926,
+          -0.05833379551768303,
+          -0.021032702177762985,
+          0.024983597919344902,
+          -0.012109989300370216,
+          0.023223955184221268,
+          0.037981707602739334,
+          0.0021373010240495205,
+          -0.02617882564663887,
+          0.0007885188679210842,
+          0.022128328680992126,
+          -0.029034094884991646,
+          -0.01673319935798645,
+          -0.020833497866988182,
+          -0.03781570494174957,
+          -0.0019318711711093783,
+          -0.025299005210399628,
+          -0.006324751302599907,
+          0.04511987790465355,
+          0.007685983553528786,
+          -0.012035287916660309,
+          -0.007084219250828028,
+          0.014873955398797989,
+          -0.0004430231056176126,
+          0.006150446832180023,
+          0.008482802659273148,
+          0.029830913990736008,
+          -0.00192253349814564,
+          0.014774353243410587,
+          0.05484770983457565,
+          0.016467593610286713,
+          -0.012184691615402699,
+          -0.006387002766132355,
+          -0.01867544651031494,
+          -0.003214251948520541,
+          -0.004502857569605112,
+          0.0011827782727777958,
+          -0.02338995970785618,
+          0.04040536656975746,
+          -0.02888469025492668,
+          0.014525347389280796,
+          -0.02224453166127205,
+          0.0020491115283221006,
+          -0.05006679892539978,
+          0.010591053403913975,
+          -0.013803229667246342,
+          -0.02036868780851364,
+          -0.009918737225234509,
+          0.0002780566574074328,
+          0.05185963958501816,
+          0.0004969743895344436,
+          0.01867544651031494,
+          -0.004502857569605112,
+          -0.019206659868359566,
+          -0.015454969368875027,
+          -0.02046828903257847,
+          0.005270625930279493,
+          -6.296478386502713e-05,
+          0.018160834908485413,
+          0.0006251087179407477,
+          -0.0006152522400952876,
+          0.02418677881360054,
+          0.0288016889244318,
+          -0.019306261092424393,
+          -0.027506858110427856,
+          0.05899780988693237,
+          -0.013645526021718979,
+          -0.02579701691865921,
+          -0.02020268328487873,
+          0.005864090286195278,
+          -0.003253677859902382,
+          0.012251093052327633,
+          0.004847316071391106,
+          -0.002434033202007413,
+          -0.00297147105447948,
+          -0.02592981979250908,
+          -0.0004222725983709097,
+          -0.013064512051641941,
+          -0.015338766388595104,
+          -0.022676141932606697,
+          0.0008803398231975734,
+          0.0018696197075769305,
+          0.011670079082250595,
+          -0.008540904149413109,
+          -0.019173458218574524,
+          0.00047259253915399313,
+          -0.0006634971359744668,
+          -0.011138866655528545,
+          0.02792186848819256,
+          -0.012002087198197842,
+          -0.03562445193529129,
+          0.019854074344038963,
+          0.0024547837674617767,
+          0.017729224637150764,
+          0.0129234092310071,
+          0.014409144409000874,
+          0.009047216735780239,
+          0.0032184019219130278,
+          0.01812763325870037,
+          -0.02426978014409542,
+          0.002842817921191454,
+          -0.001082138391211629,
+          0.0030025967862457037,
+          -0.02868548594415188,
+          0.023904571309685707,
+          0.003488158341497183,
+          -0.012350695207715034,
+          -0.0005498881218954921,
+          0.002855268307030201,
+          -0.01867544651031494,
+          0.005129522643983364,
+          0.02524920366704464,
+          -0.0015614748699590564,
+          0.012358995154500008,
+          0.016193687915802002,
+          -0.01258310116827488,
+          0.0029133695643395185,
+          -0.0009591917041689157,
+          -0.005187624134123325,
+          -0.0020449613220989704,
+          0.016475893557071686,
+          0.020302286371588707,
+          0.010192643851041794,
+          -0.012026987969875336,
+          -0.03376520797610283,
+          0.00971953198313713,
+          -0.012566500343382359,
+          0.01673319935798645,
+          -0.007656933274120092,
+          0.018575843423604965,
+          -0.029432503506541252,
+          0.028270475566387177,
+          -0.013238816522061825,
+          -0.00683106342330575,
+          -0.015546271577477455,
+          -0.025016797706484795,
+          -0.023373357951641083,
+          0.011238468810915947,
+          0.025365406647324562,
+          -0.006938965991139412,
+          0.015977881848812103,
+          -0.012085088528692722,
+          -0.005245725624263287,
+          0.06291550397872925,
+          0.01586167886853218,
+          0.012649502605199814,
+          -0.023323558270931244,
+          -0.00913021806627512,
+          -0.01385303121060133,
+          -0.0298641137778759,
+          -0.0007330112857744098,
+          0.025066599249839783,
+          0.02754005789756775,
+          0.014857355505228043,
+          0.008615606464445591,
+          0.011761381290853024,
+          0.009578429162502289,
+          0.0034985337406396866,
+          -0.00613384647294879,
+          -0.009644830599427223,
+          2.2695854568155482e-05,
+          0.00281169218942523,
+          -0.008296048268675804,
+          -0.012184691615402699,
+          -0.0036583123728632927,
+          -0.009287921711802483,
+          0.015197663567960262,
+          -0.016633598133921623,
+          0.006611107848584652,
+          -0.011122265830636024,
+          0.0015365742146968842,
+          -0.019538667052984238,
+          -0.01336331944912672,
+          0.011827782727777958,
+          0.01396093424409628,
+          -0.0007003292557783425,
+          -0.00424555130302906,
+          -0.003886567894369364,
+          0.02219473011791706,
+          -0.023921171203255653,
+          0.013537623919546604,
+          0.01910705678164959,
+          0.017845427617430687,
+          -0.012226192280650139,
+          0.023622365668416023,
+          0.008781610056757927,
+          0.008781610056757927,
+          -0.03808131068944931,
+          -0.018277037888765335,
+          0.007225322537124157,
+          -0.0038222414441406727,
+          0.00012969059753231704,
+          0.02016948163509369,
+          0.003577385563403368,
+          0.011570476926863194,
+          -0.03374860808253288,
+          -0.0065322560258209705,
+          -0.011221867986023426,
+          0.006743911188095808,
+          0.006557156797498465,
+          -0.009503726847469807,
+          0.00756148062646389,
+          0.037848904728889465,
+          -0.002983921440318227,
+          0.0122842937707901,
+          0.0069555663503706455,
+          -0.002749440725892782,
+          -0.010084740817546844,
+          0.0147245517000556,
+          -0.016849402338266373,
+          0.013612325303256512,
+          -0.02546500973403454,
+          -0.002585511887446046,
+          -0.016301589086651802,
+          0.01973787136375904,
+          0.004345153924077749,
+          0.018609045073390007,
+          -0.0025917370803654194,
+          -0.020534690469503403,
+          -0.009188319556415081,
+          -0.017513418570160866,
+          0.002475534100085497,
+          -0.006407753098756075,
+          0.0018457566620782018,
+          -0.006357952021062374,
+          -0.00014655037375632674,
+          0.009196619503200054,
+          -0.019771073013544083,
+          -0.018791649490594864,
+          -0.019771073013544083,
+          -0.015488170087337494,
+          0.006250049453228712,
+          -0.014425745233893394,
+          -0.006785412319004536,
+          -0.002328205620869994,
+          0.005843339487910271,
+          0.041833002120256424,
+          -0.020833497866988182,
+          -0.028652284294366837,
+          0.02503339946269989,
+          -0.00022877422452438623,
+          -0.009171719662845135,
+          0.01370362751185894,
+          -0.0012211666908115149,
+          -0.005154423415660858,
+          0.018990853801369667,
+          -0.01889125257730484,
+          0.007229472976177931,
+          0.008673707023262978,
+          0.02109910361468792,
+          -0.008682007901370525,
+          -0.0017918053781613708,
+          0.00472281314432621,
+          0.0207504965364933,
+          -0.00898081436753273,
+          0.006478304974734783,
+          -0.006590357515960932,
+          0.006075745448470116,
+          0.002392532303929329,
+          0.043061431497335434,
+          -0.010109641589224339,
+          0.004353453870862722,
+          -0.006428503897041082,
+          -0.011221867986023426,
+          0.003284803591668606,
+          0.03212176635861397,
+          -0.012350695207715034,
+          -0.009453926235437393,
+          -0.0004974931362085044,
+          0.004258001688867807,
+          -0.0012782305711880326,
+          0.009669731371104717,
+          -0.002780566457659006,
+          -0.019422464072704315,
+          0.009370923973619938,
+          -0.0021808771416544914,
+          -0.010109641589224339,
+          -0.008839711546897888,
+          0.012417096644639969,
+          -0.014873955398797989,
+          0.003652087412774563,
+          -0.009022315964102745,
+          0.0038969432935118675,
+          0.017729224637150764,
+          0.02312435209751129,
+          -0.013545923866331577,
+          0.00015848191105760634,
+          -0.014019034802913666,
+          -0.02991391532123089,
+          -0.022344134747982025,
+          0.004980118945240974,
+          0.001614388544112444,
+          -0.007856138050556183,
+          0.007237772922962904,
+          0.0004977525095455348,
+          -0.011686678975820541,
+          0.007922539487481117,
+          0.001984784845262766,
+          -0.020003478974103928,
+          0.017646221444010735,
+          -0.005183474160730839,
+          0.014425745233893394,
+          0.004206125624477863,
+          0.012691003270447254,
+          0.004094073083251715,
+          -0.0008103068685159087,
+          -0.004677162040024996,
+          0.005909741390496492,
+          -0.01829363778233528,
+          0.02249353751540184,
+          0.04160059615969658,
+          -0.0004712956433650106,
+          0.012035287916660309,
+          0.013628926128149033,
+          0.01829363778233528,
+          -0.005058970768004656,
+          -0.010259045287966728,
+          0.002500434871762991,
+          -0.01744701713323593,
+          -0.005166873335838318,
+          -0.004938618279993534,
+          0.028204074129462242,
+          0.00562338437885046,
+          0.005818439181894064,
+          0.00597199285402894,
+          0.0006686847191303968,
+          0.013745129108428955,
+          0.007490929216146469,
+          0.026842841878533363,
+          0.020617693662643433,
+          -0.020783696323633194,
+          -0.011346370913088322,
+          -0.0039674947038292885,
+          0.018061231821775436,
+          0.018360039219260216,
+          -0.00022812576207797974,
+          -0.048307154327631,
+          0.026361430063843727,
+          -0.007295874413102865,
+          0.02486739493906498,
+          -0.00315615045838058,
+          0.021912524476647377,
+          0.01962166838347912,
+          -0.01523086428642273,
+          0.01834343932569027,
+          0.01923985965549946,
+          0.001934983767569065,
+          0.03384821116924286,
+          -0.028054671362042427,
+          -0.0063413516618311405,
+          -0.010416748933494091,
+          -0.005830889567732811,
+          0.01736401580274105,
+          -0.0006453404203057289,
+          -0.028585882857441902,
+          0.0021539016161113977,
+          0.023954372853040695,
+          -0.0046896119602024555,
+          0.022858746349811554,
+          -0.009495426900684834,
+          -0.028287075459957123,
+          -0.026461033150553703,
+          0.001020405674353242,
+          -0.023788368329405785,
+          -0.018924452364444733,
+          0.004780914168804884,
+          0.02850288152694702,
+          0.0039031682536005974,
+          -0.016334790736436844,
+          0.00545323034748435,
+          -0.02118210680782795,
+          0.013786629773676395,
+          0.005399279296398163,
+          -0.001994122751057148,
+          0.01665019802749157,
+          0.006893314886838198,
+          0.0053038266487419605,
+          0.0021497514098882675,
+          0.03335019573569298,
+          -0.01262460183352232,
+          -0.006266649812459946,
+          -0.008549204096198082,
+          0.003048247890546918,
+          0.00037999346386641264,
+          0.011072465218603611,
+          0.02516620233654976,
+          0.0008331324206665158,
+          0.005581883247941732,
+          -0.0003052916727028787,
+          0.017463617026805878,
+          -0.015388567931950092,
+          -0.03881172835826874,
+          0.00280131702311337,
+          -0.006171197630465031,
+          -0.021680118516087532,
+          0.015396867878735065,
+          -0.02885149046778679,
+          -0.007673533633351326,
+          -0.0008206821512430906,
+          0.0013581200037151575,
+          0.031756557524204254,
+          -0.005353628192096949,
+          -0.017347414046525955,
+          0.0014421595260500908,
+          -0.028702085837721825,
+          -0.0030129719525575638,
+          0.009927037172019482,
+          0.024385983124375343,
+          0.019306261092424393,
+          -0.01804463192820549,
+          0.018028030171990395,
+          0.027772463858127594,
+          -0.003533809445798397,
+          -0.023987572640180588,
+          -0.010707256384193897,
+          -0.009935337118804455,
+          0.011570476926863194,
+          0.0023116052616387606,
+          -0.013313517905771732,
+          0.010615954175591469,
+          -0.01075705699622631,
+          -0.0019412089604884386,
+          -0.0010655380319803953,
+          -0.013794929720461369,
+          0.052888862788677216,
+          0.006005193572491407,
+          0.0049220179207623005,
+          -0.03957534581422806,
+          0.005108772311359644,
+          -0.011180367320775986,
+          0.013911132700741291,
+          0.009736132808029652,
+          0.014873955398797989,
+          0.00668165972456336,
+          -0.001169290510006249,
+          0.01557117234915495,
+          -0.011147166602313519,
+          0.013728528283536434,
+          0.005652435123920441,
+          -0.0300799198448658,
+          0.0023302806075662374,
+          0.0006536406581290066,
+          0.0051253726705908775,
+          -0.004789214581251144,
+          -1.1988275218755007e-05,
+          0.007304174825549126,
+          0.025149600580334663,
+          0.014508746564388275,
+          0.008254547603428364,
+          -0.011578776873648167,
+          -0.023057950660586357,
+          0.01508146058768034,
+          0.009495426900684834,
+          -0.000578420062083751,
+          0.002919594757258892,
+          -0.030594531446695328,
+          0.005615084432065487,
+          0.004050496965646744,
+          0.010740457102656364,
+          0.01347122248262167,
+          -0.010815158486366272,
+          0.024385983124375343,
+          -0.015454969368875027,
+          -0.003757914761081338,
+          0.011163766495883465,
+          -0.05494731292128563,
+          0.00810929387807846,
+          0.03867892548441887,
+          0.004399104975163937,
+          -0.027307653799653053,
+          -0.01543836947530508,
+          -0.01910705678164959,
+          -0.000976829556748271,
+          0.0006375589873641729,
+          0.005527932196855545,
+          0.006988767068833113,
+          -0.016832802444696426,
+          0.011213568039238453,
+          -0.01880824938416481,
+          -0.034163616597652435,
+          -0.011869283393025398,
+          0.019605068489909172,
+          0.012940009124577045,
+          -0.023439761251211166,
+          0.018443040549755096,
+          -0.010425048880279064,
+          0.011279969476163387,
+          -0.01247519813477993,
+          -0.032885387539863586,
+          0.003793190699070692,
+          -0.015737175941467285,
+          0.007283424027264118,
+          0.0165173951536417,
+          -0.03159055486321449,
+          0.007839537225663662,
+          0.006611107848584652,
+          -0.013039612211287022,
+          -0.03029572404921055,
+          0.018160834908485413,
+          -0.0019806348718702793,
+          0.019040655344724655,
+          -0.01982087455689907,
+          0.019040655344724655,
+          -0.007590531371533871,
+          -0.00685181375592947,
+          -0.03313439339399338,
+          -0.007042718585580587,
+          -0.010682355612516403,
+          -0.0283866785466671,
+          -0.011977186426520348,
+          -0.009935337118804455,
+          0.02342315949499607,
+          -0.040438566356897354,
+          0.003598136128857732,
+          -0.012915108352899551,
+          0.017679423093795776,
+          0.014135237783193588,
+          -0.023871371522545815,
+          0.020684095099568367,
+          -0.018625644966959953,
+          -0.013628926128149033,
+          -0.006868414115160704,
+          -0.011304870247840881,
+          0.005648285150527954,
+          -0.000889158749487251,
+          -0.004494557622820139,
+          -0.005295526701956987,
+          -0.02270934358239174,
+          -0.03180636093020439,
+          -0.002793016843497753,
+          0.0006531218532472849,
+          -0.0025979620404541492,
+          0.012857007794082165,
+          0.02846967987716198,
+          -0.008453751914203167,
+          -0.0032163269352167845,
+          0.005855789873749018,
+          0.017679423093795776,
+          0.0008793022716417909,
+          0.023107752203941345,
+          -0.015347067266702652,
+          -0.02363896556198597,
+          0.0012605926021933556,
+          0.01574547588825226,
+          -0.010532951913774014,
+          -0.0017357789911329746,
+          0.01489055622369051,
+          -0.006270799785852432,
+          0.003816016251221299,
+          -0.02134811133146286,
+          -0.0288016889244318,
+          -0.0022701043635606766,
+          -0.002444408368319273,
+          -0.015703976154327393,
+          -0.0013435946311801672,
+          -0.012616301886737347,
+          -0.00893931370228529,
+          0.021447712555527687,
+          -0.01493205688893795,
+          -0.01139617245644331,
+          0.01037524826824665,
+          -0.01826043613255024,
+          0.008872912265360355,
+          0.0013321818551048636,
+          0.017696022987365723,
+          0.02066749334335327,
+          -0.002402907470241189,
+          0.011338070966303349,
+          0.013944333419203758,
+          0.0035794605500996113,
+          -0.0056316847912967205,
+          0.014052236452698708,
+          0.010225844569504261,
+          -0.011528975330293179,
+          0.001997235231101513,
+          0.011578776873648167,
+          0.01423483993858099,
+          -0.03095974028110504,
+          -0.022128328680992126,
+          -0.014575148932635784,
+          0.015181062743067741,
+          0.0192730613052845,
+          0.0013487822143360972,
+          -0.002954870695248246,
+          0.0049925693310797215,
+          -0.009113618172705173,
+          0.005171023774892092,
+          0.003899018280208111,
+          -0.0186422448605299,
+          0.004660561215132475,
+          0.0014815854374319315,
+          0.007685983553528786,
+          -0.006100645754486322,
+          0.010723856277763844,
+          -0.003911468666046858,
+          -0.011553876101970673,
+          -0.007163071073591709,
+          -0.01894105225801468,
+          -0.00889781303703785,
+          0.020020078867673874,
+          0.00022086309036239982,
+          -0.02211172878742218,
+          0.005656585097312927,
+          0.013413120992481709,
+          -0.015023359097540379,
+          -0.02334015816450119,
+          0.010026639327406883,
+          -0.0005804951069876552,
+          0.0008767084800638258,
+          0.010267345234751701,
+          -0.011761381290853024,
+          0.016907503828406334,
+          -0.014276341535151005,
+          0.010964562185108662,
+          0.0068227630108594894,
+          -0.04189940169453621,
+          0.010441649705171585,
+          0.019671469926834106,
+          -0.01224279310554266,
+          -0.01829363778233528,
+          0.0008450639434158802,
+          0.0277392640709877,
+          -0.0012159791076555848,
+          0.011935685761272907,
+          0.019986877217888832,
+          -0.0007024043006822467,
+          -0.012317494489252567,
+          -0.004851466044783592,
+          0.0918997973203659,
+          0.016309890896081924,
+          0.0136621268466115,
+          0.010200943797826767,
+          0.011836082674562931,
+          -0.030129719525575638,
+          -0.007225322537124157,
+          -0.008906112983822823,
+          -0.010823458433151245,
+          -0.017098408192396164,
+          -0.014434045180678368,
+          0.006777111906558275,
+          0.012790605425834656,
+          0.015919780358672142,
+          0.015952982008457184,
+          -0.00895591452717781,
+          0.007781436201184988,
+          0.008204746060073376,
+          0.0037662149406969547,
+          0.013031311333179474,
+          -0.0056316847912967205,
+          -0.0003042541502509266,
+          -0.011230168864130974,
+          0.010939661413431168,
+          0.014417444355785847,
+          0.011230168864130974,
+          -0.00262701278552413,
+          0.004590009804815054,
+          0.02500019781291485,
+          0.00793083943426609,
+          0.016060883179306984,
+          -0.016002783551812172,
+          0.0045941597782075405,
+          -0.013189014978706837,
+          0.040903378278017044,
+          0.015255765058100224,
+          0.0037765903398394585,
+          -0.005241575185209513,
+          0.0122842937707901,
+          0.004681312013417482,
+          0.018243836238980293,
+          -0.014940356835722923,
+          0.013545923866331577,
+          -0.009453926235437393,
+          -0.0038616673555225134,
+          0.03067753277719021,
+          -0.006677509751170874,
+          0.0029112945776432753,
+          -0.014093737117946148,
+          0.004560959059745073,
+          0.007980640977621078,
+          0.003305553924292326,
+          0.004830715712159872,
+          0.016633598133921623,
+          -0.01894105225801468,
+          -0.011138866655528545,
+          0.020551292225718498,
+          -0.013637226074934006,
+          -0.0017949179746210575,
+          0.017613021656870842,
+          0.006100645754486322,
+          -0.0013819830492138863,
+          0.010856659151613712,
+          -0.012334095314145088,
+          -0.007358125876635313,
+          0.008433002047240734,
+          0.006183648016303778,
+          4.756402631755918e-05,
+          0.005577733274549246,
+          0.024452384561300278,
+          0.005320427007973194,
+          -0.017231212928891182,
+          0.012699303217232227,
+          0.0008367637638002634,
+          0.010483150370419025,
+          0.017928428947925568,
+          0.011952285654842854,
+          0.01948886550962925,
+          -0.015977881848812103,
+          0.00719212181866169,
+          0.006100645754486322,
+          0.01572057604789734,
+          -0.02689264342188835,
+          -0.006992917042225599,
+          -0.0011672154068946838,
+          -0.0017814300954341888,
+          0.012715904042124748,
+          0.014840754680335522,
+          -0.00022125216491986066,
+          0.008972514420747757,
+          -0.0024278080090880394,
+          -0.010408448986709118,
+          0.02172992005944252,
+          -0.022725943475961685,
+          -0.02588001824915409,
+          -0.024634988978505135,
+          0.0073664262890815735,
+          -0.012558200396597385,
+          0.008304348215460777,
+          -0.005270625930279493,
+          0.009835734963417053,
+          0.004432306159287691,
+          0.015181062743067741,
+          0.002600037259981036,
+          -0.007586381398141384,
+          -0.014176739379763603,
+          0.021414512768387794,
+          -0.0033906311728060246,
+          0.009254720993340015,
+          -0.009960237890481949,
+          0.014301242306828499,
+          0.0038222414441406727,
+          0.00560263404622674,
+          -0.010242444463074207,
+          0.016641898080706596,
+          -0.014774353243410587,
+          0.002942420309409499,
+          0.016832802444696426,
+          -0.01736401580274105,
+          -0.012566500343382359,
+          -0.006719010882079601,
+          -0.011703279800713062,
+          0.006905765272676945,
+          -0.028071271255612373,
+          -0.011869283393025398,
+          0.007723334711045027,
+          -0.018077831715345383,
+          0.0071464707143604755,
+          -0.00963653065264225,
+          -0.016558896750211716,
+          0.0235061626881361,
+          0.027855465188622475,
+          -0.011379571631550789,
+          -0.004123123362660408,
+          -0.0025564611423760653,
+          0.003811866044998169,
+          -0.026660237461328506,
+          0.0006546781514771283,
+          -0.0056192344054579735,
+          -0.008773310109972954,
+          -0.003048247890546918,
+          0.016849402338266373,
+          0.002066749380901456,
+          0.0013435946311801672,
+          0.013462921604514122,
+          -0.006810312625020742,
+          -0.020534690469503403,
+          0.01109736505895853,
+          0.014458945952355862,
+          -0.01762962155044079,
+          0.003143700072541833,
+          0.0009285847190767527,
+          -0.019389264285564423,
+          0.01547987014055252,
+          0.00855750497430563,
+          0.016110684722661972,
+          -0.0004414668073877692,
+          0.013819830492138863,
+          -0.011819482780992985,
+          -0.005349477753043175,
+          -0.008590705692768097,
+          0.025780417025089264,
+          0.00685181375592947,
+          0.015073160640895367,
+          0.010873259976506233,
+          0.006270799785852432,
+          -0.00476431380957365,
+          0.027390655130147934,
+          0.023738566786050797,
+          -0.0007309362408705056,
+          -0.005361928138881922,
+          0.010076440870761871,
+          -0.005851639900356531,
+          0.0027100148145109415,
+          -0.017231212928891182,
+          0.0080511923879385,
+          0.0024713841266930103,
+          0.010400149039924145,
+          0.029465705156326294,
+          0.012151490896940231,
+          -0.014301242306828499,
+          0.02571401558816433,
+          -0.003058623056858778,
+          -0.023987572640180588,
+          -0.00014538316463585943,
+          -0.01425144076347351,
+          -0.009080417454242706,
+          0.010840059258043766,
+          -0.0014691351680085063,
+          -0.015106361359357834,
+          0.015297265723347664,
+          0.01453364733606577,
+          0.009927037172019482,
+          -0.008105143904685974,
+          0.004266302101314068,
+          0.00011510041076689959,
+          0.01315581426024437,
+          0.004901267122477293,
+          0.014326142147183418,
+          -0.004185375291854143,
+          0.0029237449634820223,
+          -5.317314571584575e-05,
+          -0.0014784728409722447,
+          -0.012201291508972645,
+          0.007881038822233677,
+          0.012358995154500008,
+          0.0033034789375960827,
+          0.013562524691224098,
+          0.00297147105447948,
+          -0.015820179134607315,
+          -0.03290198743343353,
+          -0.0009156156447716057,
+          0.010840059258043766,
+          -0.009794234298169613,
+          -0.01141277328133583,
+          0.010723856277763844,
+          0.007208722177892923,
+          0.009005715139210224,
+          0.013288618065416813,
+          0.003759989980608225,
+          -0.002585511887446046,
+          -0.010275646112859249,
+          0.021148905158042908,
+          0.007453578058630228,
+          -0.002770191291347146,
+          -0.0036686877720057964,
+          -0.010134542360901833,
+          -0.009586729109287262,
+          -0.0034196816850453615,
+          -0.014392544515430927,
+          0.013313517905771732,
+          0.007565631065517664,
+          0.007059318944811821,
+          0.011952285654842854,
+          0.016467593610286713,
+          0.0035919109359383583,
+          0.009022315964102745,
+          -0.010690655559301376,
+          -0.006175347603857517,
+          -0.020136281847953796,
+          -0.01850944198668003,
+          -0.014135237783193588,
+          0.0015510995872318745,
+          -0.006864264141768217,
+          -0.0004572890466079116,
+          -0.004614910576492548,
+          0.02131490968167782,
+          -0.011288270354270935,
+          -0.0012854932574555278,
+          -3.089878737227991e-05,
+          -0.008856311440467834,
+          -0.01298151072114706,
+          0.020916501060128212,
+          0.0016662648413330317,
+          0.0157786775380373,
+          0.007395477034151554,
+          0.026046022772789,
+          0.007026118226349354,
+          0.002187102334573865,
+          -0.015106361359357834,
+          -0.02431958168745041,
+          -0.011487474665045738,
+          0.021082503721117973,
+          -0.004193675238639116,
+          -0.0001918124035000801,
+          0.02334015816450119,
+          -0.011827782727777958,
+          0.007167221512645483,
+          -0.015313866548240185,
+          0.030312323942780495,
+          -0.008856311440467834,
+          0.012275993824005127,
+          0.012765705585479736,
+          0.01826043613255024,
+          -0.0018997079459950328,
+          0.01741381548345089,
+          0.0021829521283507347,
+          -0.008997415192425251,
+          -0.008383200503885746,
+          -0.0017679422162473202,
+          -0.021148905158042908,
+          -0.012782305479049683,
+          0.00771918473765254,
+          0.0050299204885959625,
+          -0.0054158796556293964,
+          0.013504423201084137,
+          0.0015355367213487625,
+          -0.0007049980922602117,
+          -0.00982743501663208,
+          0.00670241005718708,
+          -0.0013653826899826527,
+          -0.01523086428642273,
+          -0.0017181411385536194,
+          0.011454273946583271,
+          -0.027174849063158035,
+          -0.008856311440467834,
+          0.010010039433836937,
+          -0.00751997996121645,
+          -0.010433349758386612,
+          -0.00786028802394867,
+          0.0022763293236494064,
+          0.008154945448040962,
+          0.004158399533480406,
+          -0.009022315964102745,
+          0.0026456883642822504,
+          0.0015687375562265515,
+          -0.00035249904613010585,
+          -0.011968886479735374,
+          -0.005710536614060402,
+          -0.00898081436753273,
+          0.019937077537178993,
+          -0.007565631065517664,
+          -0.00580183882266283,
+          0.003598136128857732,
+          -0.0068227630108594894,
+          0.015056559816002846,
+          -0.013902832753956318,
+          -0.007885188795626163,
+          -0.01812763325870037,
+          0.021364711225032806,
+          -0.014683051034808159,
+          -0.00026443914975970984,
+          -0.008034592494368553,
+          -0.013280317187309265,
+          -0.0020646743942052126,
+          -0.0029693960677832365,
+          0.017214611172676086,
+          -0.0024278080090880394,
+          0.0003117762098554522,
+          -0.010516351088881493,
+          0.002209927886724472,
+          0.00788933876901865,
+          -0.013064512051641941,
+          0.0020677868742495775,
+          -0.0011402397649362683,
+          0.006241749040782452,
+          -0.007627882529050112,
+          -0.012051887810230255,
+          -0.007997240871191025,
+          0.019156858325004578,
+          0.012010387144982815,
+          -0.00773163465783,
+          -0.017081808298826218,
+          -0.007154771126806736,
+          0.013628926128149033,
+          0.014102037064731121,
+          0.005565282888710499,
+          -0.033649004995822906,
+          -0.003977870102971792,
+          -0.004859765991568565,
+          0.009121918119490147,
+          -0.022443735972046852,
+          -0.0074867792427539825,
+          -0.005270625930279493,
+          -0.02363896556198597,
+          -0.0010634629288688302,
+          0.010898160748183727,
+          0.006474155001342297,
+          -0.007470178417861462,
+          -0.009121918119490147,
+          -0.008291898295283318,
+          0.009951937943696976,
+          -0.01198548637330532,
+          0.0051959240809082985,
+          -0.0031416250858455896,
+          0.005805988796055317,
+          0.02126510813832283,
+          -0.018609045073390007,
+          0.0034093065187335014,
+          -0.005025770049542189,
+          -0.00947052612900734,
+          0.006710710469633341,
+          -0.007104970049113035,
+          -0.014649850316345692,
+          -0.006743911188095808,
+          -0.009495426900684834,
+          0.0004225319717079401,
+          0.007042718585580587,
+          0.0013684951700270176,
+          -0.028021469712257385,
+          0.004258001688867807,
+          -0.003836766816675663,
+          0.014832454733550549,
+          -0.00011587855260586366,
+          -0.008881212212145329,
+          0.011960585601627827,
+          -0.002284629736095667,
+          0.0050299204885959625,
+          0.012516699731349945,
+          -0.01071555633097887,
+          -0.023190755397081375,
+          -0.006868414115160704,
+          -0.010076440870761871,
+          0.005067271180450916,
+          -0.004361754283308983,
+          0.022941747680306435,
+          -0.03535884618759155,
+          0.014085437171161175,
+          -0.0074494280852377415,
+          -0.006528106052428484,
+          0.006972166709601879,
+          -0.005345327779650688,
+          0.027722662314772606,
+          0.00489296717569232,
+          0.008433002047240734,
+          -0.003654162399470806,
+          -0.008549204096198082,
+          -0.020385287702083588,
+          0.02443578466773033,
+          0.010574452579021454,
+          -0.0014286716468632221,
+          0.0023468811996281147,
+          0.006760511547327042,
+          0.011562176048755646,
+          0.024468984454870224,
+          -0.024717990309000015,
+          -0.007926689460873604,
+          -0.018111033365130424,
+          -0.013255417346954346,
+          -0.008416401222348213,
+          -0.011113965883851051,
+          -0.0006640158826485276,
+          -0.007237772922962904,
+          -0.0021310760639607906,
+          -0.0026228628121316433,
+          -0.0015479869907721877,
+          -0.004270452074706554,
+          -0.0014743227511644363,
+          -0.003079373622313142,
+          -0.015737175941467285,
+          -0.003106349380686879,
+          0.005224974825978279,
+          0.00948712695389986,
+          -0.004847316071391106,
+          0.0009778671665117145,
+          0.01056615263223648,
+          0.014467245899140835,
+          -0.007237772922962904,
+          -0.021464312449097633,
+          0.02385476976633072,
+          -0.010682355612516403,
+          0.000497233762871474,
+          -0.013238816522061825,
+          -0.009121918119490147,
+          -0.0034508074168115854,
+          0.0024402583949267864,
+          -0.0032267021015286446,
+          -0.015239164233207703,
+          -0.0031810509972274303,
+          -0.011421073228120804,
+          -0.016177086159586906,
+          0.01648419350385666,
+          0.03278578445315361,
+          -0.0015230864519253373,
+          0.016509095206856728,
+          -1.2742101716867182e-05,
+          0.010433349758386612,
+          -0.004805814940482378,
+          0.0005177248967811465,
+          0.003600211115553975,
+          -0.009578429162502289,
+          -0.004946918226778507,
+          0.007511679548770189,
+          -0.004021446220576763,
+          -0.006727310828864574,
+          -0.00961992982774973,
+          -0.017231212928891182,
+          -0.00893931370228529,
+          -0.005440779961645603,
+          -0.009263021871447563,
+          0.012350695207715034,
+          0.004345153924077749,
+          -0.008781610056757927,
+          0.014824154786765575,
+          0.00424555130302906,
+          -0.007034418173134327,
+          -0.011080765165388584,
+          -0.00013040390331298113,
+          0.0230911523103714,
+          0.023140953853726387,
+          -0.00809269305318594,
+          -0.026444431394338608,
+          0.01593638025224209,
+          -0.025597812607884407,
+          -0.004365904256701469,
+          -0.011371271684765816,
+          -0.01012624241411686,
+          -0.03838011622428894,
+          0.0087899100035429,
+          0.011180367320775986,
+          0.01635139063000679,
+          0.002259728964418173,
+          -0.0031167245469987392,
+          0.014608349651098251,
+          -0.008831411600112915,
+          -0.016235187649726868,
+          -0.004556809086352587,
+          0.003867892548441887,
+          -0.008706907741725445,
+          0.015089760534465313,
+          0.005436629988253117,
+          0.007793886587023735,
+          -0.0019754471722990274,
+          0.002940345322713256,
+          0.0036769879516214132,
+          0.021298309788107872,
+          0.004332703538239002,
+          0.004685461986809969,
+          0.0035836107563227415,
+          -0.0004712956433650106,
+          -0.010931361466646194,
+          0.007142320740967989,
+          -0.0013446321245282888,
+          -0.007055168505758047,
+          -0.00889781303703785,
+          0.030743934214115143,
+          0.02948230504989624,
+          0.010649154894053936,
+          -0.008266997523605824,
+          0.009495426900684834,
+          0.00908871740102768,
+          -0.015015059150755405,
+          0.033233996480703354,
+          -0.004955218639224768,
+          0.0014608348719775677,
+          0.012649502605199814,
+          0.016849402338266373,
+          -0.008287748321890831,
+          -0.006374552380293608,
+          0.007295874413102865,
+          -0.007395477034151554,
+          0.006988767068833113,
+          0.025946419686079025,
+          0.007706734351813793,
+          -0.004287052433937788,
+          -0.030528130009770393,
+          0.0038637423422187567,
+          -0.0031623756512999535,
+          -0.009478827007114887,
+          -0.005328727420419455,
+          0.01209338940680027,
+          -0.011670079082250595,
+          0.007818787358701229,
+          0.008466202765703201,
+          -0.0101677430793643,
+          0.0050880215130746365,
+          -0.00849110260605812,
+          0.0030752234160900116,
+          -0.01527236495167017,
+          -0.01654229499399662,
+          -0.005428329575806856,
+          0.01258310116827488,
+          -0.003857517149299383,
+          -0.008831411600112915,
+          0.012541599571704865,
+          0.01330521795898676,
+          0.01796162873506546,
+          -0.006864264141768217,
+          -0.0028677184600383043,
+          0.00908871740102768,
+          0.025863418355584145,
+          0.0016901278868317604,
+          -0.011852683499455452,
+          0.0013799079461023211,
+          0.008848011493682861,
+          0.017397215589880943,
+          0.007291724439710379,
+          -0.004818265326321125,
+          -0.01330521795898676,
+          -0.01741381548345089,
+          -0.001454609795473516,
+          0.003328379476442933,
+          -0.008565804921090603,
+          0.01103926356881857,
+          0.015056559816002846,
+          -0.009155118837952614,
+          0.007773135788738728,
+          0.0006463779718615115,
+          0.015961281955242157,
+          -0.022061927244067192,
+          -0.026212027296423912,
+          0.013197315856814384,
+          -0.003984095063060522,
+          -0.02028568461537361,
+          -0.00384506699629128,
+          0.001266817795112729,
+          0.022393936291337013,
+          -0.00034731143387034535,
+          -0.00038621859857812524,
+          0.00439495500177145,
+          -0.007138170767575502,
+          0.011006062850356102,
+          -0.02888469025492668,
+          -0.004490407183766365,
+          -3.7837231502635404e-05,
+          -0.0040878476575016975,
+          0.00018014023953583091,
+          -0.014898856170475483,
+          0.006727310828864574,
+          0.01733081415295601,
+          -0.008889512158930302,
+          0.01826043613255024,
+          -0.004697912372648716,
+          0.024070575833320618,
+          0.010607653297483921,
+          -0.0012657803017646074,
+          -0.0052332752384245396,
+          0.001739929080940783,
+          -0.028370078653097153,
+          0.004436456132680178,
+          -0.007358125876635313,
+          -0.010342047549784184,
+          -0.00489296717569232,
+          -0.024087175726890564,
+          0.013587424531579018,
+          0.006893314886838198,
+          0.018775049597024918,
+          0.004552658647298813,
+          0.008773310109972954,
+          0.020352086052298546,
+          -0.010856659151613712,
+          -0.0006178460316732526,
+          0.0036292618606239557,
+          0.012674403376877308,
+          0.0069182151928544044,
+          -0.004519457928836346,
+          -0.024800993502140045,
+          -0.030561331659555435,
+          0.0040359715931117535,
+          0.0016081634676083922,
+          -0.00978593435138464,
+          -0.0022763293236494064,
+          -0.011113965883851051,
+          0.0034508074168115854,
+          -0.02749025821685791,
+          -0.010250745341181755,
+          0.007312474772334099,
+          0.030146321281790733,
+          -0.006793712265789509,
+          -0.0005296564195305109,
+          -0.020518090575933456,
+          0.002039773855358362,
+          -0.02699224464595318,
+          -0.014268041588366032,
+          0.008320949040353298,
+          -0.00786028802394867,
+          0.021962326020002365,
+          -0.02999691665172577,
+          -0.001976484665647149,
+          -0.0046066101640462875,
+          -0.011852683499455452,
+          0.004560959059745073,
+          -0.013296918012201786,
+          -0.0063413516618311405,
+          0.0033470550552010536,
+          0.004174999892711639,
+          -0.00864050630480051,
+          0.016998806968331337,
+          -0.018841451033949852,
+          -0.012549900449812412,
+          -0.00456510903313756,
+          -0.01711500994861126,
+          -0.003201801562681794,
+          0.015712276101112366,
+          -0.007528279908001423,
+          0.005590183660387993,
+          0.017812225967645645,
+          0.0006666096742264926,
+          -0.007084219250828028,
+          -0.02672663889825344,
+          0.01593638025224209,
+          0.001115339226089418,
+          -0.004038046579807997,
+          -0.0005029401509091258,
+          0.02833687700331211,
+          0.00314577529206872,
+          0.007590531371533871,
+          0.014027335681021214,
+          -0.006528106052428484,
+          -0.017148209735751152,
+          0.008814810775220394,
+          -0.002479684306308627,
+          -0.01880824938416481,
+          0.010350347496569157,
+          -9.426885299035348e-06,
+          -0.004631510935723782,
+          -0.003811866044998169,
+          -0.016940705478191376,
+          -0.0064492542296648026,
+          -0.012466898187994957,
+          -0.0015282740350812674,
+          0.015330466441810131,
+          -0.005735436920076609,
+          -0.00933772325515747,
+          -0.014467245899140835,
+          -0.021829521283507347,
+          0.018459642305970192,
+          -0.011960585601627827,
+          8.036764484131709e-06,
+          -0.0022161530796438456,
+          -0.006457554176449776,
+          -0.0023323558270931244,
+          -0.012491798959672451,
+          0.000848176539875567,
+          -0.0028635684866458178,
+          -0.02164691686630249,
+          -0.0032059517689049244,
+          0.020318886265158653,
+          0.02850288152694702,
+          0.005743737332522869,
+          -2.742956166912336e-05,
+          -0.010765357874333858,
+          -0.02669343911111355,
+          0.01406053639948368,
+          0.009180019609630108,
+          -0.020435089245438576,
+          -0.01509806141257286,
+          0.01706520840525627,
+          0.0012720053782686591,
+          -0.005573583301156759,
+          -0.02558121085166931,
+          0.0032785783987492323,
+          0.003392706159502268,
+          -0.03125854581594467,
+          -0.0028635684866458178,
+          0.008823110722005367,
+          7.243220170494169e-05,
+          -0.01434274297207594,
+          0.016367990523576736,
+          -0.0017606796463951468,
+          0.011387872509658337,
+          0.007266823668032885,
+          0.0157786775380373,
+          -0.03678647801280022,
+          -0.033233996480703354,
+          0.013844731263816357,
+          0.010516351088881493,
+          -0.018526043742895126,
+          -0.011504075489938259,
+          0.013653826899826527,
+          -0.0002509512996766716,
+          -0.008125894702970982,
+          -0.0022618041839450598,
+          0.01266610249876976,
+          -0.03452882543206215,
+          -0.021447712555527687,
+          -0.009188319556415081,
+          0.011113965883851051,
+          0.002952795708552003,
+          0.011105665937066078,
+          0.014649850316345692,
+          -0.014110337011516094,
+          -0.010043240152299404,
+          -0.03894453123211861,
+          -0.004772614222019911,
+          0.009163418784737587,
+          -0.01733081415295601,
+          -0.0095286276191473,
+          0.003585685743018985,
+          -0.004751863423734903,
+          0.0084205511957407,
+          0.008881212212145329,
+          0.014525347389280796,
+          -0.002135226037353277,
+          0.015181062743067741,
+          0.0007112232269719243,
+          0.01943906396627426,
+          0.01283210702240467,
+          0.002712089801207185,
+          0.019854074344038963,
+          0.0007480554049834609,
+          -0.003438357263803482,
+          -0.004258001688867807,
+          0.002369706751778722,
+          0.009918737225234509,
+          -0.003120874520391226,
+          0.00016457737365271896,
+          0.020152881741523743,
+          0.007968190126121044,
+          0.009769333526492119,
+          -0.0021891773212701082,
+          0.03545844927430153,
+          -0.011769681237637997,
+          0.005814289208501577,
+          -0.005897291004657745,
+          -0.015513070859014988,
+          0.020833497866988182,
+          0.012765705585479736,
+          0.0016413643024861813,
+          0.018326837569475174,
+          0.00528307631611824,
+          0.0007226360030472279,
+          -0.00543248001486063,
+          -0.022227931767702103,
+          0.0013591574970632792,
+          0.017596419900655746,
+          -0.007698433939367533,
+          0.006096495781093836,
+          -0.004536058288067579,
+          0.023406559601426125,
+          0.0005312127177603543,
+          0.006897464860230684,
+          0.016417792066931725,
+          -0.006067445036023855,
+          0.003957119770348072,
+          -0.0006334088975563645,
+          0.0022784045431762934,
+          -0.011329771019518375,
+          -0.003199726575985551,
+          0.016318190842866898,
+          0.0010800632881000638,
+          0.0020034604240208864,
+          0.004614910576492548,
+          0.0002532857470214367,
+          0.0009825360029935837,
+          0.002994296606630087,
+          -0.0062085483223199844,
+          0.010101341642439365,
+          -0.0068601141683757305,
+          -0.006934816017746925,
+          -0.017762424424290657,
+          0.0063164508901536465,
+          0.029764512553811073,
+          -0.01050805114209652,
+          -0.02016948163509369,
+          0.0039052434731274843,
+          -0.005556982941925526,
+          0.003988245502114296,
+          -0.0008751521818339825,
+          -0.005025770049542189,
+          -0.01787862740457058,
+          -0.009121918119490147,
+          0.007914239540696144,
+          -0.0018135933205485344,
+          0.018575843423604965,
+          0.029681509360671043,
+          0.005673185456544161,
+          -7.55447763367556e-05,
+          0.009595029056072235,
+          0.004598309751600027,
+          0.001292755943723023,
+          0.0017752049025148153,
+          0.004760163836181164,
+          0.006922365631908178,
+          -0.020451689139008522,
+          -0.01001833938062191,
+          -0.019754473119974136,
+          -0.003938443958759308,
+          -0.009113618172705173,
+          -0.00839150045067072,
+          0.00650320528075099,
+          -0.002766041085124016,
+          -0.0017285163048654795,
+          0.0005358815542422235,
+          -0.0019080081256106496,
+          0.01054955180734396,
+          -0.019754473119974136,
+          -0.0054158796556293964,
+          -0.021779721602797508,
+          -0.004482107236981392,
+          0.003309704130515456,
+          -0.008354149758815765,
+          -0.016525695100426674,
+          -0.0038077160716056824,
+          0.03320079296827316,
+          -0.017613021656870842,
+          -0.00844960194081068,
+          -0.0062583498656749725,
+          0.0072585237212479115,
+          0.00927962176501751,
+          -0.02071729488670826,
+          0.0012356920633465052,
+          -0.00017806519463192672,
+          -0.01277400553226471,
+          -0.007851988077163696,
+          -0.017945028841495514,
+          -0.02634483017027378,
+          0.004818265326321125,
+          -0.0011962661519646645,
+          -0.01749681867659092,
+          -0.008947613649070263,
+          -0.015687374398112297,
+          -0.006494905333966017,
+          0.00843715202063322,
+          -0.024087175726890564,
+          0.010151143185794353,
+          0.012043587863445282,
+          -0.004193675238639116,
+          0.003143700072541833,
+          0.0063538020476698875,
+          0.016509095206856728,
+          0.005648285150527954,
+          0.005864090286195278,
+          -0.01425144076347351,
+          -0.017314214259386063,
+          0.0022244532592594624,
+          0.005710536614060402,
+          0.014367643743753433,
+          -0.004901267122477293,
+          0.0059180413372814655,
+          0.007980640977621078,
+          0.0020491115283221006,
+          0.01404393557459116,
+          0.025315605103969574,
+          -0.020302286371588707,
+          -0.01065745484083891,
+          0.0044987075962126255,
+          0.015396867878735065,
+          0.003959194757044315,
+          0.007138170767575502,
+          -0.014525347389280796,
+          -0.009935337118804455,
+          -0.001842644065618515,
+          0.007017817813903093,
+          -0.017596419900655746,
+          0.021829521283507347,
+          -0.00685181375592947,
+          0.02546500973403454,
+          0.014757752418518066,
+          0.02282554656267166,
+          0.017546620219945908,
+          -0.010914760641753674,
+          8.55309481266886e-05,
+          -0.0015376117080450058,
+          0.01542176865041256,
+          -0.008362449705600739,
+          -0.006768811959773302,
+          0.0028822438325732946,
+          0.009312822483479977,
+          -0.004901267122477293,
+          0.009379223920404911,
+          0.016807902604341507,
+          -0.005366078112274408,
+          0.0011962661519646645,
+          -0.014351042918860912,
+          -0.004197825212031603,
+          -0.010350347496569157,
+          0.0029361951164901257,
+          0.0033034789375960827,
+          -0.022045327350497246,
+          0.011910784989595413,
+          0.013545923866331577,
+          0.0024859094992280006,
+          -0.026909243315458298,
+          0.0063164508901536465,
+          0.0032578278332948685,
+          0.002357256365939975,
+          0.0122842937707901,
+          0.027224650606513023,
+          0.00406087189912796,
+          -0.006756361573934555,
+          -0.003811866044998169,
+          0.006868414115160704,
+          0.012126590125262737,
+          0.00263738795183599,
+          -0.006793712265789509,
+          -0.021298309788107872,
+          -0.009254720993340015,
+          -0.028917891904711723,
+          0.002919594757258892,
+          -0.026411231607198715,
+          0.0005919078830629587,
+          0.006432653870433569,
+          -0.009702932089567184,
+          -0.024419184774160385,
+          0.01842644065618515,
+          0.016152186319231987,
+          0.0017088033491745591,
+          0.009354324080049992,
+          -0.015703976154327393,
+          -0.00526232598349452,
+          0.015737175941467285,
+          -0.009362624026834965,
+          -0.011869283393025398,
+          -0.029714711010456085,
+          0.009503726847469807,
+          0.0017129534389823675,
+          0.0038741177413612604,
+          -0.010350347496569157,
+          0.0015791127225384116,
+          -0.014757752418518066,
+          0.006196097936481237,
+          -0.0008502515847794712,
+          -0.017247812822461128,
+          0.010200943797826767,
+          0.00368528813123703,
+          0.003071073442697525,
+          -0.003888642881065607,
+          -0.0014660225715488195,
+          0.010275646112859249,
+          0.016658497974276543,
+          -0.0032931037712842226,
+          -0.009960237890481949,
+          -0.0021165506914258003,
+          0.014409144409000874,
+          -0.005781088024377823,
+          0.016450993716716766,
+          -0.0011371271684765816,
+          0.0053162770345807076,
+          -0.003025422338396311,
+          0.002425733022391796,
+          0.0144008444622159,
+          -0.0005436629871837795,
+          0.007864437997341156,
+          0.019588468596339226,
+          -0.006474155001342297,
+          -0.01347122248262167,
+          0.00961992982774973,
+          -0.010649154894053936,
+          0.015977881848812103,
+          0.0031935013830661774,
+          -0.009387524798512459,
+          -0.01018434390425682,
+          0.014683051034808159,
+          0.023987572640180588,
+          -0.022061927244067192,
+          0.009221520274877548,
+          0.009412424638867378,
+          0.004104448016732931,
+          -0.01787862740457058,
+          0.007279274053871632,
+          -0.007075919304043055,
+          -0.0035939859226346016,
+          0.01018434390425682,
+          0.00889781303703785,
+          0.002546085976064205,
+          0.021547315642237663,
+          -0.011371271684765816,
+          -0.0012004162417724729,
+          -0.005573583301156759,
+          0.010840059258043766,
+          -0.017978230491280556,
+          0.0012720053782686591,
+          -0.00913021806627512,
+          -0.004436456132680178,
+          0.005212524440139532,
+          0.018227236345410347,
+          -0.011960585601627827,
+          -0.005980292800813913,
+          0.003371955594047904,
+          -0.014226539991796017,
+          -0.010068140923976898,
+          -0.00997683871537447,
+          -0.0044032554142177105,
+          0.008665407076478004,
+          0.0074369776993989944,
+          -0.018443040549755096,
+          -0.010624254122376442,
+          0.0008336511673405766,
+          -0.02159711718559265,
+          0.0036168114747852087,
+          -0.031109143048524857,
+          0.015322166495025158,
+          -0.0021705019753426313,
+          -0.006059145089238882,
+          -0.0023738567251712084,
+          0.018227236345410347,
+          -0.005540382582694292,
+          0.0023427309934049845,
+          0.0068725645542144775,
+          0.004569259472191334,
+          0.01659209653735161,
+          0.01523086428642273,
+          -0.010516351088881493,
+          -0.01207678858190775,
+          0.025348806753754616,
+          -0.007594681810587645,
+          0.01243369746953249,
+          -0.03698568418622017,
+          0.004237251356244087,
+          -0.014541947282850742,
+          -0.0031395500991493464,
+          -0.018144233152270317,
+          0.013936033472418785,
+          0.003759989980608225,
+          0.005171023774892092,
+          -0.0026436131447553635,
+          0.013338418677449226,
+          0.008715208619832993,
+          -0.008906112983822823,
+          0.008117593824863434,
+          -0.00849110260605812,
+          0.012840406969189644,
+          -0.024883994832634926,
+          0.005025770049542189,
+          0.012383895926177502,
+          -0.0011682529002428055,
+          0.007229472976177931,
+          -0.01662529818713665,
+          0.004868066404014826,
+          0.00908871740102768,
+          -0.021331509575247765,
+          0.02197892591357231,
+          -0.008354149758815765,
+          -0.00166522734798491,
+          -0.020783696323633194,
+          -0.009844035841524601,
+          0.014367643743753433,
+          -0.0006354839424602687,
+          -0.012508398853242397,
+          0.015579472295939922,
+          0.019771073013544083,
+          0.0005976142711006105,
+          0.023057950660586357,
+          -0.017297614365816116,
+          0.010408448986709118,
+          -0.0017492668703198433,
+          -0.00029595394153147936,
+          -0.0074867792427539825,
+          0.0122842937707901,
+          0.0015116736758500338,
+          -0.0006769849569536746,
+          -0.01419333927333355,
+          -0.005133672617375851,
+          -0.013786629773676395,
+          0.02885149046778679,
+          0.02312435209751129,
+          -0.0006583094946108758,
+          0.0031706758309155703,
+          -0.0038077160716056824,
+          -0.0034279818646609783,
+          0.004370054695755243,
+          -0.01914025843143463,
+          -0.01965487003326416,
+          -0.0036707627587020397,
+          -0.013819830492138863,
+          0.018526043742895126,
+          0.018775049597024918,
+          -0.006901614833623171,
+          0.019721271470189095,
+          -0.013745129108428955,
+          0.013778329826891422,
+          -0.0066443090327084064,
+          -0.010051540099084377,
+          -0.003716413863003254,
+          0.002954870695248246,
+          -0.0063164508901536465,
+          0.025149600580334663,
+          0.008233796805143356,
+          -0.0230911523103714,
+          0.013902832753956318,
+          -0.010732156224548817,
+          -0.015703976154327393,
+          0.0015178987523540854,
+          -0.0177956260740757,
+          0.025216002017259598,
+          0.012840406969189644,
+          0.0003945188072975725,
+          -0.01400243490934372,
+          0.016011083498597145,
+          0.008183995261788368,
+          0.002693414455279708,
+          0.011620277538895607,
+          0.02113230526447296,
+          -8.961620187619701e-05,
+          -0.0177956260740757,
+          -0.0012678552884608507,
+          0.019339462742209435,
+          -0.02224453166127205,
+          -0.004177074879407883,
+          0.010640854015946388,
+          0.017729224637150764,
+          0.009893836453557014,
+          -0.0005239500314928591,
+          -0.0070510185323655605,
+          0.0010364872869104147,
+          -0.010781957767903805,
+          -0.01995367743074894,
+          0.01855924353003502,
+          -0.011329771019518375,
+          0.020916501060128212,
+          0.016251789405941963,
+          0.02148091420531273,
+          -0.005843339487910271,
+          -0.021331509575247765,
+          0.005440779961645603,
+          -0.010541251860558987,
+          -0.010914760641753674,
+          0.026361430063843727,
+          -0.009072116576135159,
+          -0.004407405387610197,
+          0.006050844676792622,
+          -0.012275993824005127,
+          0.013537623919546604,
+          -0.014367643743753433,
+          0.0038844929076731205,
+          -0.013413120992481709,
+          -0.022676141932606697,
+          0.009213220328092575,
+          0.004768464248627424,
+          0.013205615803599358,
+          -0.01902405545115471,
+          -0.01662529818713665,
+          0.006976316682994366,
+          0.01812763325870037,
+          -0.00788933876901865,
+          0.008997415192425251,
+          0.013272017240524292,
+          -0.003542109625414014,
+          -0.010350347496569157,
+          0.037417296320199966,
+          0.004872216377407312,
+          -0.002952795708552003,
+          0.006988767068833113,
+          0.006594507489353418,
+          0.024070575833320618,
+          -0.005710536614060402,
+          0.02410377562046051,
+          -0.019970277324318886,
+          7.839797035558149e-05,
+          -0.00401937123388052,
+          -0.0002860974636860192,
+          0.017214611172676086,
+          -0.01736401580274105,
+          0.017812225967645645,
+          -0.0022618041839450598,
+          0.004469656851142645,
+          0.04777594283223152,
+          -0.012715904042124748,
+          -0.028585882857441902,
+          -0.018409840762615204,
+          -0.025481609627604485,
+          -0.0009156156447716057,
+          -0.001976484665647149,
+          -0.01990387588739395,
+          0.0017326663946732879,
+          -0.0029590206686407328,
+          -0.00722117256373167,
+          0.004938618279993534,
+          0.012334095314145088,
+          -0.009653130546212196,
+          0.021331509575247765,
+          0.020551292225718498,
+          0.013446321710944176,
+          0.006951416376978159,
+          0.005648285150527954,
+          -0.005561132915318012,
+          0.0008476577932015061,
+          0.00026612510555423796,
+          -0.0013684951700270176,
+          -0.03213836997747421,
+          0.02172992005944252,
+          -0.015869978815317154,
+          0.00472281314432621,
+          0.011802881956100464,
+          -0.03358260169625282,
+          -0.004668861627578735,
+          -0.006586207542568445,
+          0.009138518944382668,
+          -0.008167395368218422,
+          0.01817743480205536,
+          -0.016741501167416573,
+          -0.00506312120705843,
+          0.008806510828435421,
+          0.013014711439609528,
+          0.00927962176501751,
+          0.015264065004885197,
+          0.016035983338952065,
+          0.010707256384193897,
+          -0.013728528283536434,
+          0.02101610228419304,
+          -0.019007453694939613,
+          -0.0044032554142177105,
+          0.01957186870276928,
+          -0.012358995154500008,
+          0.016002783551812172,
+          0.022974949330091476,
+          -0.00579353841021657,
+          -0.024551987648010254,
+          0.01593638025224209,
+          0.008914412930607796,
+          0.01435934379696846,
+          0.044887471944093704,
+          -0.0012554050190374255,
+          -0.01689920388162136,
+          -0.027324253693223,
+          -0.02224453166127205,
+          -0.01105586439371109,
+          0.013911132700741291,
+          -0.003187276190146804,
+          0.0033657304011285305,
+          0.02194572426378727,
+          -0.0030316475313156843,
+          -0.014840754680335522,
+          0.011869283393025398,
+          0.001804255647584796,
+          0.0643763393163681,
+          0.004258001688867807,
+          -0.005681485868990421,
+          0.018775049597024918,
+          -0.0062583498656749725,
+          -0.012217892333865166,
+          0.006561306770890951,
+          -0.0063164508901536465,
+          -0.012906808406114578,
+          -0.01850944198668003,
+          0.010134542360901833,
+          -0.023140953853726387,
+          0.004278752487152815,
+          0.004756013862788677,
+          0.0025481609627604485,
+          -0.017513418570160866,
+          -0.010109641589224339,
+          -0.022178130224347115,
+          0.0015490244841203094,
+          -0.017812225967645645,
+          -0.018609045073390007,
+          -0.04027256369590759,
+          -0.016691699624061584,
+          0.016401192173361778,
+          -0.00908871740102768,
+          0.015031659044325352,
+          -0.008453751914203167,
+          -0.010350347496569157,
+          -0.009304522536695004,
+          -0.006308150943368673,
+          -0.005374378524720669,
+          0.014981858432292938,
+          0.025863418355584145,
+          -0.004681312013417482,
+          -0.004469656851142645,
+          0.004419855773448944,
+          -0.013753429055213928,
+          0.011487474665045738,
+          -0.010748757049441338,
+          0.011636878363788128,
+          -0.00386996753513813,
+          0.01889125257730484,
+          0.0003467926580924541,
+          -0.008723508566617966,
+          -0.02778906375169754,
+          -0.012740804813802242,
+          -0.0010925136739388108,
+          -0.00773163465783,
+          0.010300545953214169,
+          -0.005229125265032053,
+          0.003774515353143215,
+          0.0015510995872318745,
+          0.0034217569045722485,
+          -0.020534690469503403,
+          -0.007088369689881802,
+          -0.00034860832965932786,
+          0.0030025967862457037,
+          0.008183995261788368,
+          0.006880864500999451,
+          -0.005627534352242947,
+          0.0035732353571802378,
+          -0.017646221444010735,
+          -0.010649154894053936,
+          0.01580357737839222,
+          -0.013628926128149033,
+          -0.002390457084402442,
+          0.008673707023262978,
+          -0.013844731263816357,
+          -0.026577236130833626,
+          0.009578429162502289,
+          0.011421073228120804,
+          0.00367491296492517,
+          -0.021464312449097633,
+          -0.0003724714042618871,
+          0.012989810667932034,
+          0.0008263884810730815,
+          0.020103080198168755,
+          0.022393936291337013,
+          0.00212277565151453,
+          0.00032215146347880363,
+          0.003147850278764963,
+          0.005839189514517784,
+          -0.028270475566387177,
+          0.024850795045495033,
+          0.022775745019316673,
+          -0.011238468810915947,
+          0.015330466441810131,
+          0.00405464693903923,
+          0.0009856485994532704,
+          -0.01948886550962925,
+          0.0042829024605453014,
+          0.021530715748667717,
+          0.00335120502859354,
+          0.007291724439710379,
+          0.0015095985727384686,
+          0.0011630653170868754,
+          0.017397215589880943,
+          0.0006385965389199555,
+          0.004199900198727846,
+          0.0041832998394966125,
+          -0.020816897973418236,
+          0.00827114749699831,
+          -0.011080765165388584,
+          -0.0028594182804226875,
+          -0.007727484684437513,
+          0.0003626149264164269,
+          0.02699224464595318,
+          0.011213568039238453,
+          -0.014309542253613472,
+          0.014450645074248314,
+          0.011578776873648167,
+          -0.00803874246776104,
+          -0.012043587863445282,
+          -0.0007942252559587359,
+          -0.00632060132920742,
+          0.0041438741609454155,
+          0.017264412716031075,
+          -0.0013487822143360972,
+          0.009702932089567184,
+          0.014492146670818329,
+          -0.00175756704993546,
+          -0.012524999678134918,
+          0.01351272314786911,
+          0.011329771019518375,
+          0.013164115138351917,
+          -0.0073622758500278,
+          -0.0023655565455555916,
+          -0.011761381290853024,
+          -0.007424527313560247,
+          0.005627534352242947,
+          0.0022472788114100695,
+          -0.012466898187994957,
+          -0.007735785096883774,
+          -0.0016880529001355171,
+          0.0009555603610351682,
+          0.011728180572390556,
+          -0.007831237278878689,
+          0.0076652332209050655,
+          -0.003666612785309553,
+          0.012491798959672451,
+          0.000912503048311919,
+          0.04883836954832077,
+          0.005610933993011713,
+          0.003147850278764963,
+          0.0052083744667470455,
+          0.0030772986356168985,
+          0.01728101260960102,
+          -0.008026291616261005,
+          0.024120377376675606,
+          0.0076776836067438126,
+          -0.010599353350698948,
+          -0.026710039004683495,
+          -0.015762077644467354,
+          0.021364711225032806,
+          -0.004739413503557444,
+          -0.001546949497424066,
+          0.019389264285564423,
+          0.01574547588825226,
+          0.011761381290853024,
+          0.023705366998910904,
+          0.00614214688539505,
+          0.013039612211287022,
+          0.0011423148680478334,
+          0.02104930393397808,
+          0.004697912372648716,
+          -0.0027556659188121557,
+          0.016011083498597145,
+          0.009586729109287262,
+          0.019156858325004578,
+          -0.004531908314675093,
+          -0.00043420412112027407,
+          0.019920475780963898,
+          -0.0144008444622159,
+          -0.008665407076478004,
+          0.003533809445798397,
+          -0.009163418784737587,
+          -0.0018540568416938186,
+          0.0026124874129891396,
+          -0.022227931767702103,
+          -0.012018687091767788,
+          -0.007715034298598766,
+          0.00489296717569232,
+          0.008922713808715343,
+          0.0010593128390610218,
+          0.02672663889825344,
+          -0.011479174718260765,
+          0.011977186426520348,
+          0.00929622258991003,
+          0.010192643851041794,
+          -0.005370228551328182,
+          0.002778491470962763,
+          0.0021414512302726507,
+          -0.01762962155044079,
+          -0.00683106342330575,
+          -0.018227236345410347,
+          0.007407926954329014,
+          0.01169497985392809,
+          0.026876043528318405,
+          -0.01489055622369051,
+          -0.001508561079390347,
+          -0.011089065112173557,
+          0.010059840977191925,
+          0.014840754680335522,
+          0.005527932196855545
+        ],
+        "keyphrases": [
+          "Switch Transformers",
+          "Lower Compute Regimes"
+        ],
+        "title": "D Switch Transformers in Lower Compute Regimes"
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "6df469f9-914b-4652-b189-540abdf1161a",
+      "properties": {
+        "page_content": "E Relation of Upstream to Downstream Model Performance 32 ",
+        "embedding": [
+          -0.018082372844219208,
+          0.023730995133519173,
+          -0.010728987865149975,
+          0.047936949878931046,
+          -0.03446846455335617,
+          -0.022373968735337257,
+          6.639356433879584e-05,
+          0.06883515417575836,
+          -0.040982190519571304,
+          -0.011789164505898952,
+          -0.011102170683443546,
+          -0.019473325461149216,
+          0.0032950290478765965,
+          -0.020541982725262642,
+          -0.018150225281715393,
+          0.06174468994140625,
+          -0.013655075803399086,
+          -0.0006122520426288247,
+          0.0013835305580869317,
+          0.010686581023037434,
+          -0.011840052902698517,
+          -0.03555408492684364,
+          -0.012162347324192524,
+          0.02306944504380226,
+          -0.017946671694517136,
+          0.0030490681529045105,
+          0.011161540634930134,
+          -0.005020997021347284,
+          -0.02865021489560604,
+          -0.0051015703938901424,
+          0.03867524489760399,
+          0.03236507251858711,
+          -0.004855609033256769,
+          -0.017844894900918007,
+          -0.03636830300092697,
+          0.01401977613568306,
+          0.01845555566251278,
+          -0.04328913614153862,
+          -0.03602904453873634,
+          0.020287541672587395,
+          -0.04037152975797653,
+          0.025851348415017128,
+          -0.025495128706097603,
+          0.022017749026417732,
+          0.03172048553824425,
+          -0.010008067823946476,
+          0.0009758926462382078,
+          0.005945470649749041,
+          0.01615709252655506,
+          -0.010118326172232628,
+          -0.01730208285152912,
+          -0.0348077192902565,
+          0.03915020450949669,
+          0.004249188117682934,
+          -0.02006702497601509,
+          0.01911710575222969,
+          0.04488363862037659,
+          0.1082228347659111,
+          -0.006721520330756903,
+          0.028073478490114212,
+          0.05166877061128616,
+          -0.012043607421219349,
+          0.002571988618001342,
+          -0.03602904453873634,
+          -0.02803955227136612,
+          -0.012798452749848366,
+          -0.06045551598072052,
+          0.016767755150794983,
+          0.011365094222128391,
+          0.017760079354047775,
+          -0.02951531857252121,
+          0.020253615453839302,
+          0.02417202852666378,
+          0.04423905164003372,
+          -0.014910325407981873,
+          0.030634865164756775,
+          0.02975279837846756,
+          0.021084794774651527,
+          -0.006059969775378704,
+          0.02388366125524044,
+          0.0036957759875804186,
+          0.010949504561722279,
+          0.013510892167687416,
+          0.015546430833637714,
+          0.022730188444256783,
+          -0.011585610918700695,
+          -0.027547631412744522,
+          -0.06934403628110886,
+          0.037114664912223816,
+          0.01806541159749031,
+          -0.023188184946775436,
+          -0.007624790538102388,
+          -0.004177096299827099,
+          0.005703750532120466,
+          -0.023001592606306076,
+          0.04657992348074913,
+          0.020525021478533745,
+          0.0024362860713154078,
+          0.011721313931047916,
+          0.04043937847018242,
+          -0.0042152623645961285,
+          0.03552015870809555,
+          0.005071885418146849,
+          0.030007241293787956,
+          0.01556339394301176,
+          -0.0288537684828043,
+          -0.007315219379961491,
+          0.0028221902903169394,
+          0.003935375716537237,
+          0.10937631130218506,
+          -0.014002813957631588,
+          0.006509484723210335,
+          -0.0004625020665116608,
+          0.015554912388324738,
+          -0.024850541725754738,
+          -0.01725119538605213,
+          0.00812519434839487,
+          -0.037352144718170166,
+          0.04885294288396835,
+          -0.022153452038764954,
+          0.013748371042311192,
+          0.01494425069540739,
+          0.005110051482915878,
+          0.016479386016726494,
+          -0.005339049734175205,
+          0.048513684421777725,
+          0.015936575829982758,
+          -0.001556339324451983,
+          0.0006970661925151944,
+          0.005203347187489271,
+          0.01067809946835041,
+          -0.008668004535138607,
+          -0.06221964955329895,
+          -0.0252746120095253,
+          0.0008497316157445312,
+          0.007446681149303913,
+          0.018438592553138733,
+          0.0020864277612417936,
+          0.011458389461040497,
+          -0.008205767720937729,
+          0.00984692107886076,
+          -0.05506133660674095,
+          0.020474132150411606,
+          -0.02403632551431656,
+          0.01446929108351469,
+          0.04261062294244766,
+          0.0181162990629673,
+          0.06048944219946861,
+          -0.0016040473710745573,
+          0.00034826804767362773,
+          0.0008396599441766739,
+          0.00017042340186890215,
+          0.026071865111589432,
+          0.004707184620201588,
+          0.03619867190718651,
+          -0.004170734900981188,
+          -0.013841667212545872,
+          0.004982830490916967,
+          -0.0059836371801793575,
+          0.018862662836909294,
+          0.0063313753344118595,
+          0.0233578123152256,
+          0.034587204456329346,
+          0.006967481225728989,
+          -0.010627211071550846,
+          -0.0009297749493271112,
+          0.022051675245165825,
+          -0.023815808817744255,
+          -0.023289961740374565,
+          0.02822614461183548,
+          0.028718065470457077,
+          -0.03390869125723839,
+          0.01740385964512825,
+          0.01787881925702095,
+          -0.010499989613890648,
+          -0.01446929108351469,
+          -0.013595705851912498,
+          0.018252002075314522,
+          -0.012722120620310307,
+          0.0033459176775068045,
+          -0.046885255724191666,
+          -0.06876730173826218,
+          -0.02227219194173813,
+          -0.048751164227724075,
+          0.007815622724592686,
+          -0.004187698010355234,
+          0.06045551598072052,
+          0.005635899491608143,
+          -0.013010487891733646,
+          0.013476965948939323,
+          0.02454521134495735,
+          0.018557332456111908,
+          0.026258455589413643,
+          0.008621356450021267,
+          -0.022187378257513046,
+          -0.0016793198883533478,
+          0.0016931021818891168,
+          0.03494342416524887,
+          0.016029872000217438,
+          -0.0022645373828709126,
+          -0.0023875178303569555,
+          -0.020626798272132874,
+          0.01029643602669239,
+          -0.002057802863419056,
+          0.024613061919808388,
+          0.007561180274933577,
+          0.04990463703870773,
+          0.038302063941955566,
+          -0.024731801822781563,
+          -0.04715665802359581,
+          -0.014774622395634651,
+          -0.041083965450525284,
+          -0.04542645066976547,
+          0.03548623248934746,
+          -0.012009681202471256,
+          0.03867524489760399,
+          -0.006895389407873154,
+          0.03418009728193283,
+          -0.03562193736433983,
+          -0.013264930807054043,
+          0.0021479178685694933,
+          -0.01122939120978117,
+          0.024765728041529655,
+          0.012425270862877369,
+          -0.036877185106277466,
+          0.011424464173614979,
+          0.011483834125101566,
+          -0.015376802533864975,
+          0.015453135594725609,
+          -0.009134482592344284,
+          -0.00803613942116499,
+          0.036266524344682693,
+          -0.0024256843607872725,
+          -0.017174862325191498,
+          -0.05119381099939346,
+          -0.00827361922711134,
+          -0.021814195439219475,
+          0.006585817784070969,
+          -0.022577522322535515,
+          0.020100949332118034,
+          -0.01032188069075346,
+          -0.01535135880112648,
+          -0.03378995135426521,
+          0.001905137556605041,
+          0.04837798327207565,
+          -0.003979903180152178,
+          -0.010118326172232628,
+          0.06822448968887329,
+          0.02451128512620926,
+          0.00183834636118263,
+          0.01592809520661831,
+          -0.022781075909733772,
+          0.010457582771778107,
+          -0.012611862272024155,
+          0.03602904453873634,
+          0.03643615171313286,
+          0.003526147687807679,
+          -0.008646801114082336,
+          -0.025732608512043953,
+          -0.00999110471457243,
+          0.002476572757586837,
+          -0.004346724599599838,
+          -0.04895471781492233,
+          0.018268965184688568,
+          -0.005572288762778044,
+          -0.006488281302154064,
+          0.03667363151907921,
+          0.022679299116134644,
+          -0.004376409575343132,
+          0.002985457656905055,
+          0.02875199168920517,
+          0.04389979690313339,
+          -0.0567236952483654,
+          -0.055807702243328094,
+          0.016377609223127365,
+          0.011186984367668629,
+          -0.022611448541283607,
+          -0.02164456807076931,
+          -0.001569061540067196,
+          0.01508843433111906,
+          0.004217382986098528,
+          0.022085601463913918,
+          -0.011314205825328827,
+          0.003244140651077032,
+          -0.010169214569032192,
+          -0.008358432911336422,
+          0.022611448541283607,
+          0.01256097387522459,
+          -0.04423905164003372,
+          -0.0008041440160013735,
+          0.01435055211186409,
+          0.009261703118681908,
+          -0.028429698199033737,
+          0.01769222877919674,
+          0.0023853974416851997,
+          0.011585610918700695,
+          0.022340042516589165,
+          -0.011882460676133633,
+          0.025851348415017128,
+          0.010499989613890648,
+          -0.0031190398149192333,
+          0.011670424602925777,
+          0.008214249275624752,
+          -0.022899815812706947,
+          -0.02259448543190956,
+          -0.010898616164922714,
+          -0.00786651112139225,
+          0.004537556320428848,
+          1.3956232578493655e-05,
+          -0.01950725167989731,
+          0.029549244791269302,
+          -0.01735297217965126,
+          -0.02132227271795273,
+          -0.037216443568468094,
+          -0.011899422854185104,
+          -0.010262510739266872,
+          0.02904035896062851,
+          0.03738607093691826,
+          -0.013799259439110756,
+          0.004073198884725571,
+          0.009482220746576786,
+          0.011068244464695454,
+          -0.0639159306883812,
+          0.053093649446964264,
+          0.022730188444256783,
+          0.012798452749848366,
+          -0.019422436133027077,
+          0.065815769135952,
+          0.0029282080940902233,
+          0.01191638596355915,
+          -0.007900436408817768,
+          -0.02832792140543461,
+          -0.007022610399872065,
+          -0.013188597746193409,
+          0.025003207847476006,
+          -0.05492563545703888,
+          -0.027954738587141037,
+          -0.030787531286478043,
+          -0.020558945834636688,
+          0.013604187406599522,
+          -0.016055315732955933,
+          -0.002962133614346385,
+          -0.03562193736433983,
+          -0.017344491556286812,
+          0.010584804229438305,
+          -0.04563000425696373,
+          -0.017285121604800224,
+          -0.0030617902521044016,
+          0.0011131855426356196,
+          0.04135537147521973,
+          0.0010888014221563935,
+          -0.017522599548101425,
+          0.00937196146696806,
+          0.026903044432401657,
+          0.05119381099939346,
+          -0.003013022243976593,
+          -0.006912352051585913,
+          0.001623130519874394,
+          0.006526447832584381,
+          -0.005326327867805958,
+          0.005203347187489271,
+          1.655697815294843e-05,
+          -0.038302063941955566,
+          0.027852961793541908,
+          -0.025444241240620613,
+          -0.02541031502187252,
+          -0.003863283898681402,
+          0.008252415806055069,
+          0.02179723232984543,
+          -0.00820152647793293,
+          -0.03684325888752937,
+          0.019625991582870483,
+          0.03070271760225296,
+          -0.004711425397545099,
+          -0.0031169194262474775,
+          -0.00047098350478336215,
+          -0.01473221555352211,
+          -0.031025011092424393,
+          -0.03928590565919876,
+          0.013646594248712063,
+          -0.0467156246304512,
+          0.011042800731956959,
+          -0.03514697775244713,
+          0.0077901785261929035,
+          -0.0003336906083859503,
+          -0.004987071268260479,
+          0.012688194401562214,
+          0.05207587778568268,
+          0.005958192981779575,
+          0.02770029567182064,
+          -0.008154879324138165,
+          -0.03314536437392235,
+          0.006424670573323965,
+          -0.03426491096615791,
+          0.07972528785467148,
+          -0.005631658714264631,
+          -2.9949991585453972e-05,
+          0.005415382329374552,
+          0.007684160489588976,
+          -0.03314536437392235,
+          -0.029108211398124695,
+          0.032924845814704895,
+          0.0081039909273386,
+          0.0018171428237110376,
+          0.026156678795814514,
+          0.03207670524716377,
+          -0.007463643793016672,
+          -0.03565586358308792,
+          0.002756459405645728,
+          -0.0015775428619235754,
+          0.0106102479621768,
+          -0.019727768376469612,
+          0.018235038965940475,
+          -0.04077863693237305,
+          0.033772990107536316,
+          0.011186984367668629,
+          -0.021763307973742485,
+          0.0152580626308918,
+          -0.025817422196269035,
+          -0.0146643640473485,
+          -0.02870110422372818,
+          0.03088930808007717,
+          0.02946443110704422,
+          0.028480585664510727,
+          0.0007718086126260459,
+          0.04030367732048035,
+          -0.018608221784234047,
+          0.024731801822781563,
+          0.013756852596998215,
+          -0.0024362860713154078,
+          0.028158292174339294,
+          0.04033760353922844,
+          -0.031109824776649475,
+          0.0006530688260681927,
+          -0.010737469419836998,
+          0.0035855176392942667,
+          -0.03450239077210426,
+          0.0014619836583733559,
+          -0.022255228832364082,
+          -0.022085601463913918,
+          0.00032229372300207615,
+          0.013587224297225475,
+          0.005992118734866381,
+          0.0025889514945447445,
+          0.004779276438057423,
+          0.0024914152454584837,
+          -0.01139901950955391,
+          -0.031245527788996696,
+          -0.05601125583052635,
+          -0.004643573891371489,
+          -0.015766948461532593,
+          0.03714859113097191,
+          -0.040507230907678604,
+          0.014961213804781437,
+          0.018557332456111908,
+          -0.016564199700951576,
+          0.010949504561722279,
+          0.005415382329374552,
+          0.02407025173306465,
+          -0.04963323101401329,
+          -0.028599325567483902,
+          0.015173248946666718,
+          0.041457150131464005,
+          -0.030770568177103996,
+          0.009931734763085842,
+          0.00635257875546813,
+          0.0024723319802433252,
+          0.01610620506107807,
+          -0.0006970661925151944,
+          0.02817525528371334,
+          0.014214849099516869,
+          -0.008795225992798805,
+          -0.0056740655563771725,
+          0.018472518771886826,
+          0.020999979227781296,
+          0.028904657810926437,
+          -0.015105397440493107,
+          -0.05760576203465462,
+          -0.016275832429528236,
+          0.02074553817510605,
+          -0.00039306050166487694,
+          0.02222130261361599,
+          0.004588444717228413,
+          0.03231418505311012,
+          0.037589624524116516,
+          0.0037572660949081182,
+          0.020202726125717163,
+          0.0066960761323571205,
+          -0.056045182049274445,
+          -0.01574150286614895,
+          0.011008874513208866,
+          -0.023951511830091476,
+          -0.015079952776432037,
+          -0.005763120483607054,
+          0.004503630567342043,
+          -0.0410161167383194,
+          0.009736662730574608,
+          -0.00865104142576456,
+          0.026258455589413643,
+          0.02751370519399643,
+          0.00678089028224349,
+          0.011195465922355652,
+          -0.015410728752613068,
+          -0.006013322155922651,
+          -0.02016880176961422,
+          -0.026309344917535782,
+          0.026529861614108086,
+          -0.02317122183740139,
+          -0.025681721046566963,
+          -0.0036258043255656958,
+          0.01223019789904356,
+          0.0015839040279388428,
+          0.02498624473810196,
+          -0.018574295565485954,
+          -0.007200720254331827,
+          0.01993132196366787,
+          -0.0116789061576128,
+          -0.009405887685716152,
+          -0.0017874578479677439,
+          0.006874185521155596,
+          -0.0054662711918354034,
+          -0.015207174234092236,
+          0.004766554571688175,
+          0.03504519909620285,
+          -0.0038590431213378906,
+          0.009109037928283215,
+          -0.01178068295121193,
+          0.0034116485621780157,
+          0.06299994140863419,
+          -0.027089634910225868,
+          -0.04437475651502609,
+          0.04600318521261215,
+          0.051227737218141556,
+          -0.0011746757663786411,
+          -0.01542769093066454,
+          -0.019337622448801994,
+          -0.03356943652033806,
+          0.03823421150445938,
+          -0.043933723121881485,
+          0.014206367544829845,
+          -0.012891748920083046,
+          0.024002401158213615,
+          -0.001378229702822864,
+          -0.001866971142590046,
+          -0.020525021478533745,
+          0.04257669672369957,
+          0.03589334338903427,
+          -0.011331168003380299,
+          0.005279679782688618,
+          -0.0004161193501204252,
+          0.015224137343466282,
+          -0.02245878241956234,
+          0.02803955227136612,
+          0.0029388098046183586,
+          -0.010304917581379414,
+          0.005962433759123087,
+          -0.01664053276181221,
+          -0.012476159259676933,
+          -0.0033607601653784513,
+          0.013748371042311192,
+          -0.007289774715900421,
+          0.035350531339645386,
+          -0.04118574410676956,
+          -0.0035473511088639498,
+          0.0041262079030275345,
+          0.005890341475605965,
+          0.009838439524173737,
+          -0.00034800299908965826,
+          -0.036266524344682693,
+          -0.024494322016835213,
+          -0.029260877519845963,
+          0.015605800785124302,
+          -0.04708880931138992,
+          -0.01988043263554573,
+          0.03504519909620285,
+          0.008663764223456383,
+          -0.01782793179154396,
+          -0.02069464884698391,
+          -0.007603587117046118,
+          -0.004728388041257858,
+          0.013799259439110756,
+          -0.009838439524173737,
+          0.0021447374019771814,
+          -0.013672038912773132,
+          0.023103371262550354,
+          -0.015105397440493107,
+          0.02454521134495735,
+          -0.04183033108711243,
+          0.011144577525556087,
+          -0.014316625893115997,
+          0.010796839371323586,
+          0.03999834507703781,
+          0.017675265669822693,
+          -0.018557332456111908,
+          -0.008455969393253326,
+          -0.028785917907953262,
+          -0.005207587964832783,
+          -0.005614695604890585,
+          0.016097722575068474,
+          -0.02164456807076931,
+          0.010355805978178978,
+          0.01518173050135374,
+          0.011170021258294582,
+          -0.014995139092206955,
+          -0.011025837622582912,
+          0.0367075577378273,
+          -0.002381156897172332,
+          0.021865084767341614,
+          0.014503217302262783,
+          -0.01115305908024311,
+          0.015419209375977516,
+          0.009880846366286278,
+          0.03952338546514511,
+          0.00752301374450326,
+          -0.02937961556017399,
+          -0.02846362441778183,
+          -0.005334808956831694,
+          0.030770568177103996,
+          -0.02917606197297573,
+          -0.0024680914357304573,
+          0.048038724809885025,
+          -0.019761692732572556,
+          0.015673652291297913,
+          -0.02464698813855648,
+          -0.020016135647892952,
+          0.0055468445643782616,
+          -0.022764114663004875,
+          -0.05774146318435669,
+          -0.018489481881260872,
+          -0.019965248182415962,
+          -0.026088828220963478,
+          0.03952338546514511,
+          -0.027157485485076904,
+          -0.048649389296770096,
+          0.014291182160377502,
+          0.012179309502243996,
+          0.009363480843603611,
+          -0.025342464447021484,
+          -0.00943981297314167,
+          -0.02230611816048622,
+          -0.01774311624467373,
+          -0.00170476408675313,
+          0.0032823069486767054,
+          0.004656296223402023,
+          -0.021576715633273125,
+          -0.03976086527109146,
+          0.004270391538739204,
+          0.028141330927610397,
+          0.00511429226025939,
+          0.02435861900448799,
+          0.0042152623645961285,
+          0.013595705851912498,
+          0.008769781328737736,
+          0.01161105465143919,
+          0.009117519482970238,
+          -0.04325520992279053,
+          -0.004329761490225792,
+          -0.006263523828238249,
+          -0.040066197514534,
+          -0.010313399136066437,
+          -0.012484640814363956,
+          0.004085920751094818,
+          0.01854036934673786,
+          0.007993732579052448,
+          0.0026313585694879293,
+          -0.014265737496316433,
+          0.005521400365978479,
+          -0.00511429226025939,
+          0.010389731265604496,
+          0.02188204601407051,
+          -0.0005422803806141019,
+          -0.0357576385140419,
+          0.019388511776924133,
+          0.031584784388542175,
+          0.01644546166062355,
+          -0.014749177731573582,
+          -0.019778655841946602,
+          0.02798866480588913,
+          0.022526634857058525,
+          -0.020558945834636688,
+          -0.04094826430082321,
+          0.02927783876657486,
+          -0.026953931897878647,
+          0.018268965184688568,
+          0.029023397713899612,
+          0.004105004016309977,
+          -0.02354440465569496,
+          0.008163360878825188,
+          7.255584205267951e-05,
+          -0.0025211002212017775,
+          -0.011288761161267757,
+          -0.014146997593343258,
+          0.0001689656637609005,
+          -0.008379636332392693,
+          0.003568554762750864,
+          0.02880288101732731,
+          -0.016716865822672844,
+          -0.014588030986487865,
+          0.02530853822827339,
+          0.01982954517006874,
+          0.0038060343358665705,
+          0.001432298682630062,
+          -0.034298837184906006,
+          -0.005754638928920031,
+          0.010864690877497196,
+          -0.024714838713407516,
+          0.01063569262623787,
+          0.009176889434456825,
+          -0.00511429226025939,
+          -0.03302662447094917,
+          -0.0002263477217638865,
+          -0.01921888254582882,
+          -0.016954345628619194,
+          0.030193831771612167,
+          0.023001592606306076,
+          0.0009414369123987854,
+          0.012806934304535389,
+          -0.027361039072275162,
+          -0.016954345628619194,
+          -0.016606608405709267,
+          0.006607021205127239,
+          -0.0021701816003769636,
+          -0.023714032024145126,
+          -0.00695051858201623,
+          -0.014978175982832909,
+          0.015699096024036407,
+          0.031008047983050346,
+          -0.005360253155231476,
+          -0.00413680961355567,
+          0.00024556342395953834,
+          0.0027331355959177017,
+          0.00522030983120203,
+          -0.026631638407707214,
+          0.0023853974416851997,
+          0.017607415094971657,
+          0.009354999288916588,
+          0.012315012514591217,
+          0.008684967644512653,
+          -0.0008836572524160147,
+          -0.04505326971411705,
+          0.020253615453839302,
+          -0.030380424112081528,
+          -0.008795225992798805,
+          0.03972693905234337,
+          3.036412272194866e-05,
+          -0.010542397387325764,
+          0.019625991582870483,
+          -0.0036872944328933954,
+          -0.04172855615615845,
+          0.016894975677132607,
+          -0.002285741036757827,
+          -0.030787531286478043,
+          0.018676072359085083,
+          -0.020253615453839302,
+          0.008099749684333801,
+          0.014910325407981873,
+          0.026156678795814514,
+          -0.03009205497801304,
+          -0.007018369622528553,
+          0.020609835162758827,
+          -0.028548438102006912,
+          0.0010400332976132631,
+          0.0032632239162921906,
+          0.018659109249711037,
+          0.0036046006716787815,
+          0.007366107776761055,
+          -0.051566995680332184,
+          -0.028768954798579216,
+          0.014893362298607826,
+          -0.0010236005764454603,
+          0.0018341057002544403,
+          0.05794501677155495,
+          0.02412113919854164,
+          -0.07450073957443237,
+          0.01711549237370491,
+          0.026394158601760864,
+          0.025020169094204903,
+          -0.027157485485076904,
+          -0.009142964147031307,
+          0.036300450563430786,
+          0.021050868555903435,
+          0.009126001037657261,
+          -0.005398419685661793,
+          0.0022242506965994835,
+          -0.023103371262550354,
+          0.021525828167796135,
+          0.013290374539792538,
+          -0.014392958953976631,
+          -0.036775410175323486,
+          -0.024138102307915688,
+          -0.025088021531701088,
+          -0.0032971494365483522,
+          -0.015139322727918625,
+          -0.018438592553138733,
+          0.03518090397119522,
+          -0.005597732961177826,
+          0.0011502917623147368,
+          0.029142137616872787,
+          -0.002459609881043434,
+          -0.003973542246967554,
+          0.01532591413706541,
+          0.019863469526171684,
+          -0.014783103950321674,
+          -0.002510498510673642,
+          -0.015453135594725609,
+          0.0020228170324116945,
+          0.026716452091932297,
+          -0.024002401158213615,
+          0.015656689181923866,
+          0.0029282080940902233,
+          0.028904657810926437,
+          -0.00970273744314909,
+          0.011042800731956959,
+          -0.01720878854393959,
+          0.00021826387092005461,
+          -0.014986657537519932,
+          0.034451503306627274,
+          -0.006848741322755814,
+          -0.016284313052892685,
+          -0.036504004150629044,
+          0.03847169131040573,
+          0.020779462531208992,
+          -0.014384477399289608,
+          -0.046037111431360245,
+          -0.01535135880112648,
+          -0.028904657810926437,
+          0.0007373528787866235,
+          0.008328747935593128,
+          0.010720506310462952,
+          0.014867917634546757,
+          0.017030678689479828,
+          -0.007794418837875128,
+          0.0002812118618749082,
+          -0.009066631086170673,
+          -0.03367121145129204,
+          0.010211621411144733,
+          -0.038878798484802246,
+          0.01736145280301571,
+          -0.008167601190507412,
+          0.007785937748849392,
+          0.01697978936135769,
+          -0.038030657917261124,
+          -0.010576322674751282,
+          -0.03156781941652298,
+          0.011195465922355652,
+          -0.011008874513208866,
+          0.015800872817635536,
+          -0.030109018087387085,
+          0.011568647809326649,
+          0.020558945834636688,
+          -0.0038081547245383263,
+          0.033874765038490295,
+          -0.006047247909009457,
+          -0.01859125867486,
+          -0.05183839797973633,
+          -0.008027657866477966,
+          0.028531474992632866,
+          0.03986264392733574,
+          -0.0026186364702880383,
+          0.027394965291023254,
+          -0.004440019838511944,
+          0.012238679453730583,
+          0.00418133707717061,
+          -0.021474938839673996,
+          0.027666371315717697,
+          -0.02354440465569496,
+          0.005682547111064196,
+          -0.049565382301807404,
+          -0.03891272470355034,
+          -0.057809315621852875,
+          -0.03155085816979408,
+          -0.0028836806304752827,
+          -0.010330361314117908,
+          0.01270515751093626,
+          0.015758465975522995,
+          0.02103390544652939,
+          -0.0023557126987725496,
+          0.007035332731902599,
+          -0.004804720636457205,
+          -0.028768954798579216,
+          0.03060094080865383,
+          -0.004745350684970617,
+          0.003000300144776702,
+          -0.013078339397907257,
+          0.04047330468893051,
+          0.011220909655094147,
+          0.0116789061576128,
+          0.00045879147364757955,
+          -0.025003207847476006,
+          -0.05160091817378998,
+          0.007472125347703695,
+          -0.008481413125991821,
+          0.006424670573323965,
+          -0.004749591462314129,
+          -0.027479778975248337,
+          0.005249994806945324,
+          0.01587720587849617,
+          -0.0055001964792609215,
+          -0.0029790964908897877,
+          -0.024239879101514816,
+          -0.008905484341084957,
+          -0.03915020450949669,
+          -0.006318653002381325,
+          -0.005004033911973238,
+          0.008778262883424759,
+          -0.027428891509771347,
+          0.011729794554412365,
+          0.011857016012072563,
+          0.008065824396908283,
+          -0.009948697872459888,
+          -0.025206761434674263,
+          -0.007484847214072943,
+          -0.005097329616546631,
+          -0.01518173050135374,
+          0.01363811269402504,
+          -0.026343271136283875,
+          -0.027479778975248337,
+          -0.02627541869878769,
+          0.014791585505008698,
+          -0.016428498551249504,
+          -0.01921888254582882,
+          -0.005020997021347284,
+          0.024901429191231728,
+          0.020728575065732002,
+          0.02764940820634365,
+          0.0014895482454448938,
+          0.024019362404942513,
+          0.027598518878221512,
+          -0.00041585430153645575,
+          -0.0012404067674651742,
+          0.007200720254331827,
+          -0.0023281481117010117,
+          0.0031529655680060387,
+          -0.01328189391642809,
+          0.010949504561722279,
+          0.003430731827393174,
+          0.009032705798745155,
+          -0.026088828220963478,
+          -0.005691028665751219,
+          -0.00013152816973160952,
+          -0.02016880176961422,
+          0.0123998261988163,
+          -0.04257669672369957,
+          -0.0047368695959448814,
+          0.024392545223236084,
+          -0.014189404435455799,
+          0.025902237743139267,
+          0.031686559319496155,
+          -0.012442233972251415,
+          0.009583997540175915,
+          0.0020567427854985,
+          -0.02751370519399643,
+          -0.010805320926010609,
+          -0.0046817404218018055,
+          0.021288348361849785,
+          9.351089829578996e-06,
+          -0.019049255177378654,
+          -0.008684967644512653,
+          -0.029430504888296127,
+          0.01494425069540739,
+          0.02312033250927925,
+          0.014409921132028103,
+          -0.04420512542128563,
+          0.0029918185900896788,
+          0.0025529053527861834,
+          -0.008337229490280151,
+          -0.01663205213844776,
+          -0.010601767338812351,
+          -0.020813388749957085,
+          0.01731056533753872,
+          -0.016182536259293556,
+          -0.00791315920650959,
+          0.015758465975522995,
+          0.01532591413706541,
+          0.01332430075854063,
+          0.01630127616226673,
+          -0.0074254777282476425,
+          -0.013417595997452736,
+          0.0008804767276160419,
+          0.005771602038294077,
+          0.039082352072000504,
+          0.023849735036492348,
+          0.041457150131464005,
+          -0.006560373120009899,
+          0.0028285514563322067,
+          0.0011555926175788045,
+          -0.019710805267095566,
+          0.0015934455441311002,
+          -0.010525434277951717,
+          -0.0007967227720655501,
+          -0.015648208558559418,
+          -0.00786651112139225,
+          0.005979396402835846,
+          0.024765728041529655,
+          0.006475559435784817,
+          0.01877784915268421,
+          0.007985251024365425,
+          -0.0190662182867527,
+          0.002330268267542124,
+          -0.0035240272991359234,
+          -0.009897809475660324,
+          -0.0021468577906489372,
+          -0.035350531339645386,
+          0.0176413394510746,
+          -0.01753956265747547,
+          0.009176889434456825,
+          0.00035197866964153945,
+          -0.01806541159749031,
+          -0.008210008032619953,
+          0.01077987626194954,
+          -0.030295608565211296,
+          0.013587224297225475,
+          -0.01044061966240406,
+          -0.025783497840166092,
+          -0.04081256315112114,
+          0.038505617529153824,
+          0.01735297217965126,
+          -0.007756252773106098,
+          0.02988850139081478,
+          0.01611468568444252,
+          -0.013315819203853607,
+          -0.014367514289915562,
+          -0.009668811224400997,
+          -0.010508471168577671,
+          0.01032188069075346,
+          0.012586417607963085,
+          0.01854036934673786,
+          0.005491715390235186,
+          0.0019242207054048777,
+          0.003341676900163293,
+          0.003956579603254795,
+          -0.01508843433111906,
+          0.0004513702297117561,
+          -0.007811381947249174,
+          0.0052542355842888355,
+          -0.021763307973742485,
+          0.005330568179488182,
+          -0.04115181788802147,
+          -0.0017514119390398264,
+          0.01387559249997139,
+          0.01921888254582882,
+          0.0008030838216654956,
+          0.00867648608982563,
+          0.014783103950321674,
+          -0.022170415148139,
+          0.007124387193471193,
+          -0.016462422907352448,
+          0.0058097681030631065,
+          -0.015376802533864975,
+          0.012645787559449673,
+          0.01418092381209135,
+          0.017056122422218323,
+          0.013553299009799957,
+          -0.0017132455250248313,
+          0.016886495053768158,
+          0.044510457664728165,
+          -0.01325644925236702,
+          -0.0012393465731292963,
+          0.024155065417289734,
+          0.019371548667550087,
+          -0.005678306333720684,
+          -0.009804514236748219,
+          0.01887962594628334,
+          0.011288761161267757,
+          -0.02417202852666378,
+          0.017844894900918007,
+          0.014749177731573582,
+          0.022204341366887093,
+          0.052890095859766006,
+          -0.005555325653403997,
+          -0.005652862135320902,
+          0.012510084547102451,
+          0.009685774333775043,
+          -0.009634885936975479,
+          -0.012501603923738003,
+          0.025613868609070778,
+          -0.02127138525247574,
+          -0.028633251786231995,
+          -0.010576322674751282,
+          -0.008718892931938171,
+          -0.01456258725374937,
+          -0.014418402686715126,
+          0.009779069572687149,
+          0.009100556373596191,
+          -0.005529881455004215,
+          0.004864090587943792,
+          0.016420016065239906,
+          -0.0031466044019907713,
+          0.031025011092424393,
+          -0.013307337649166584,
+          -0.03265344351530075,
+          0.004554518964141607,
+          0.009736662730574608,
+          -0.00592426722869277,
+          -0.0024532489478588104,
+          -0.01882873848080635,
+          0.0024256843607872725,
+          0.050651002675294876,
+          -0.0063865045085549355,
+          0.0011004634434357285,
+          -0.02398543804883957,
+          -0.004673258867114782,
+          0.007489087991416454,
+          0.006933555472642183,
+          0.005741917062550783,
+          0.022662336006760597,
+          0.021915972232818604,
+          0.00367669272236526,
+          -0.004149531479924917,
+          -0.0003323653945699334,
+          -0.028768954798579216,
+          -0.010550878010690212,
+          0.0036809334997087717,
+          0.016411535441875458,
+          0.005245754029601812,
+          0.013383670710027218,
+          0.007378829643130302,
+          0.007785937748849392,
+          -0.01187397912144661,
+          -0.027004821226000786,
+          -0.0057758428156375885,
+          -0.005724953953176737,
+          0.012806934304535389,
+          -0.01187397912144661,
+          0.0048174429684877396,
+          -0.012162347324192524,
+          -0.021050868555903435,
+          0.03185619041323662,
+          0.009244740940630436,
+          -0.005279679782688618,
+          0.002268778160214424,
+          -0.007065017241984606,
+          0.015105397440493107,
+          -0.010067437775433064,
+          0.0019496649038046598,
+          0.0029133656062185764,
+          0.0357576385140419,
+          -0.007344903890043497,
+          0.012213235720992088,
+          0.02103390544652939,
+          0.01835377886891365,
+          -0.04261062294244766,
+          -0.015410728752613068,
+          0.02985457517206669,
+          -0.024019362404942513,
+          0.01845555566251278,
+          0.0058394530788064,
+          0.007489087991416454,
+          -0.023137295618653297,
+          -0.00018818136595655233,
+          0.00791315920650959,
+          0.0047368695959448814,
+          0.026343271136283875,
+          0.02330692484974861,
+          0.04037152975797653,
+          0.009507664479315281,
+          0.00305754947476089,
+          -0.006420430261641741,
+          0.0064373929053545,
+          -0.0031699282117187977,
+          -0.0003058609727304429,
+          -0.014409921132028103,
+          0.008468691259622574,
+          0.000756436085794121,
+          -0.013103784061968327,
+          -0.007086221128702164,
+          0.00927018467336893,
+          -0.0110852075740695,
+          0.027683334425091743,
+          0.004630851559340954,
+          0.00511429226025939,
+          -0.005860656965523958,
+          -0.007476366125047207,
+          0.020016135647892952,
+          -0.02985457517206669,
+          -0.010313399136066437,
+          -0.017191825434565544,
+          0.0015054509276524186,
+          -0.004073198884725571,
+          -0.007031091954559088,
+          0.012747564353048801,
+          -0.006284727249294519,
+          -0.006072692107409239,
+          -0.004342483822256327,
+          -0.014011295512318611,
+          -0.006276246160268784,
+          0.006725761108100414,
+          -0.02627541869878769,
+          -0.0035473511088639498,
+          -0.01032188069075346,
+          -0.027717258781194687,
+          -0.03448542580008507,
+          -0.01721726916730404,
+          0.03151693195104599,
+          -0.01184853445738554,
+          -0.005033718887716532,
+          0.006310171447694302,
+          -0.0026143956929445267,
+          0.017149418592453003,
+          0.011170021258294582,
+          -0.016224944964051247,
+          0.011738276109099388,
+          -0.0012340457178652287,
+          0.006861463654786348,
+          0.0008582130540162325,
+          -0.00943981297314167,
+          -0.011373575776815414,
+          -0.019388511776924133,
+          -0.00015664110833313316,
+          -0.004550278186798096,
+          -0.012756045907735825,
+          0.004775035660713911,
+          0.007120146416127682,
+          0.012527047656476498,
+          0.008646801114082336,
+          -0.0012404067674651742,
+          0.00803189817816019,
+          -0.012730601243674755,
+          -0.013917999342083931,
+          0.004982830490916967,
+          0.00899029802531004,
+          -0.01821807585656643,
+          -0.00784530770033598,
+          0.04142322391271591,
+          -0.022051675245165825,
+          -0.0028200699016451836,
+          -0.007192238699644804,
+          0.009405887685716152,
+          -0.009075112640857697,
+          -0.0048725721426308155,
+          0.012408307753503323,
+          -0.005644380580633879,
+          -0.009931734763085842,
+          -0.036775410175323486,
+          -0.0013983730459585786,
+          0.043221283704042435,
+          -0.019015328958630562,
+          0.01759045198559761,
+          0.0007219803519546986,
+          0.02651289850473404,
+          -0.001526654465124011,
+          0.013756852596998215,
+          -0.03555408492684364,
+          -0.009762107394635677,
+          0.003136002691462636,
+          0.015487060882151127,
+          -0.006238079629838467,
+          -0.016699902713298798,
+          0.03438365086913109,
+          -0.017794005572795868,
+          -0.022187378257513046,
+          0.00362368393689394,
+          -0.03470594435930252,
+          0.007981009781360626,
+          0.0283618476241827,
+          0.011619536206126213,
+          -0.015156285837292671,
+          -0.013578742742538452,
+          0.008222730830311775,
+          0.00781986303627491,
+          -0.002936689415946603,
+          0.0009637005859985948,
+          0.01740385964512825,
+          0.023476552218198776,
+          0.036741483956575394,
+          -0.03312839940190315,
+          -0.021441012620925903,
+          0.0008216369315050542,
+          0.015190212056040764,
+          -0.004999793134629726,
+          0.029956351965665817,
+          0.030431311577558517,
+          0.012043607421219349,
+          0.010584804229438305,
+          -0.018031485378742218,
+          0.009405887685716152,
+          0.00692507391795516,
+          0.011543204076588154,
+          0.02079642564058304,
+          -0.0011524120345711708,
+          0.024613061919808388,
+          0.01582631841301918,
+          0.005890341475605965,
+          -0.0006186130922287703,
+          -0.02040628157556057,
+          -0.007667197845876217,
+          0.016988271847367287,
+          0.004847127944231033,
+          -9.746999421622604e-05,
+          0.005936989560723305,
+          0.0035240272991359234,
+          0.019337622448801994,
+          0.006004840601235628,
+          -0.006632465403527021,
+          0.008231211453676224,
+          0.0004375879361759871,
+          -0.009329554624855518,
+          -0.0025677478406578302,
+          0.014248774386942387,
+          0.023663144558668137,
+          -0.00216382066719234,
+          0.00714135030284524,
+          -0.021491901949048042,
+          0.01639457233250141,
+          -0.0011566528119146824,
+          -0.0040647173300385475,
+          -0.0034116485621780157,
+          -0.021966861560940742,
+          0.02664860151708126,
+          0.0004627671150956303,
+          0.012348937802016735,
+          -0.0005454609054140747,
+          0.0036194431595504284,
+          0.021661529317498207,
+          -0.02359529212117195,
+          -7.586889114463702e-05,
+          -0.009982624091207981,
+          -0.010788357816636562,
+          -0.028497548773884773,
+          0.0033586397767066956,
+          0.03402743116021156,
+          0.013392152264714241,
+          -0.008735856041312218,
+          -0.0009186430834233761,
+          -0.011746757663786411,
+          0.016767755150794983,
+          0.010279472917318344,
+          -0.002824310678988695,
+          0.0009409068152308464,
+          -0.017912745475769043,
+          -0.003307751379907131,
+          0.0028073478024452925,
+          -0.005623177159577608,
+          -0.007938602939248085,
+          -0.010898616164922714,
+          -0.0037297015078365803,
+          0.018608221784234047,
+          0.006602780427783728,
+          -0.007128627970814705,
+          0.005987877957522869,
+          -0.013002007268369198,
+          0.001110004959627986,
+          -0.000263983994955197,
+          0.013332782313227654,
+          -0.016191018745303154,
+          -0.02398543804883957,
+          0.016428498551249504,
+          -0.0005292932037264109,
+          -0.009176889434456825,
+          -0.019473325461149216,
+          -0.010754432529211044,
+          -0.022390931844711304,
+          0.010041994042694569,
+          -0.006963240448385477,
+          0.006013322155922651,
+          0.001960266614332795,
+          -0.023018555715680122,
+          0.009397406131029129,
+          0.007955566048622131,
+          -0.01122939120978117,
+          0.031160714104771614,
+          -0.024918392300605774,
+          0.018268965184688568,
+          -0.0009165227529592812,
+          0.014698289334774017,
+          0.02069464884698391,
+          -0.0013474845327436924,
+          -0.03255166485905647,
+          0.006670631933957338,
+          -0.009516146034002304,
+          0.01578390970826149,
+          -0.0068402597680687904,
+          -0.002148978179320693,
+          -0.006428911350667477,
+          -0.04077863693237305,
+          -0.017912745475769043,
+          -0.0069717220030725,
+          -0.0007262210128828883,
+          0.055468443781137466,
+          -0.009456776082515717,
+          -0.001471525290980935,
+          -0.022153452038764954,
+          -0.0021839640103280544,
+          -0.033874765038490295,
+          0.0011142457369714975,
+          0.017675265669822693,
+          0.005860656965523958,
+          -0.0045630005188286304,
+          0.010881653986871243,
+          0.010101363062858582,
+          0.0056740655563771725,
+          0.013841667212545872,
+          -0.00018341056420467794,
+          -0.04213566333055496,
+          0.005975155625492334,
+          -0.012951118871569633,
+          -0.006382263731211424,
+          0.021424051374197006,
+          0.011042800731956959,
+          0.0014047340955585241,
+          0.018082372844219208,
+          0.0029833372682332993,
+          0.002404480706900358,
+          -0.010864690877497196,
+          -0.017132455483078957,
+          0.017336009070277214,
+          0.0024150824174284935,
+          0.007662957068532705,
+          -0.004830164834856987,
+          -0.004520593211054802,
+          0.013146190904080868,
+          0.0011661943281069398,
+          0.012603380717337132,
+          0.015173248946666718,
+          -0.008044620975852013,
+          -0.002330268267542124,
+          0.024630025029182434,
+          -0.005330568179488182,
+          0.0022730189375579357,
+          -0.07456858456134796,
+          0.038166359066963196,
+          0.028141330927610397,
+          -0.009329554624855518,
+          -0.00543658621609211,
+          -0.015376802533864975,
+          -0.015512505546212196,
+          -0.0046817404218018055,
+          -0.024494322016835213,
+          -0.014130034483969212,
+          0.02583438530564308,
+          -0.014791585505008698,
+          0.00025165945407934487,
+          0.004363687243312597,
+          0.0048725721426308155,
+          0.005385697353631258,
+          0.006233838852494955,
+          -0.00752301374450326,
+          0.017675265669822693,
+          -0.021542789414525032,
+          0.002406601095572114,
+          -0.010712025687098503,
+          -0.01610620506107807,
+          -0.0016803800826892257,
+          0.007573902141302824,
+          -0.008701929822564125,
+          0.0065773362293839455,
+          0.030380424112081528,
+          -0.015062990598380566,
+          0.031873151659965515,
+          0.003867524676024914,
+          -0.016911938786506653,
+          -0.04420512542128563,
+          0.009999586269259453,
+          0.021474938839673996,
+          0.015266544185578823,
+          -0.029498355463147163,
+          0.0037106184754520655,
+          -0.03165263682603836,
+          -0.013239486142992973,
+          -0.01329885609447956,
+          -0.012815415859222412,
+          -0.024256842210888863,
+          0.00754421716555953,
+          0.010025030933320522,
+          -0.017047641798853874,
+          -0.010008067823946476,
+          -0.02717444859445095,
+          -0.01025402918457985,
+          -0.00856198649853468,
+          -0.023256035521626472,
+          -0.03270433098077774,
+          -0.008239693008363247,
+          0.0036957759875804186,
+          -0.01349392905831337,
+          -0.0123998261988163,
+          -0.019659915938973427,
+          -0.003948098048567772,
+          0.002957893069833517,
+          0.007824104279279709,
+          -0.006089654751121998,
+          0.02030450478196144,
+          -0.01587720587849617,
+          -0.008358432911336422,
+          0.007226164452731609,
+          0.012781490571796894,
+          0.003311991924419999,
+          0.022492708638310432,
+          0.014138516038656235,
+          0.02965102158486843,
+          0.02427380532026291,
+          0.00798949133604765,
+          0.0037805901374667883,
+          0.024307731539011,
+          0.007768974639475346,
+          -0.026241492480039597,
+          -0.019371548667550087,
+          -0.0016570561565458775,
+          0.022390931844711304,
+          -0.005249994806945324,
+          -0.010126807726919651,
+          0.017573488876223564,
+          -0.0007357626454904675,
+          0.015071472153067589,
+          -0.030109018087387085,
+          -0.013951925560832024,
+          0.0027055710088461637,
+          -0.030770568177103996,
+          -0.008595912717282772,
+          -0.0008905483991838992,
+          0.00686570443212986,
+          -0.0012181430356577039,
+          -0.006623983848839998,
+          -0.011144577525556087,
+          0.03155085816979408,
+          0.0074424403719604015,
+          0.0021267144475132227,
+          0.010423657484352589,
+          -0.004864090587943792,
+          0.00592426722869277,
+          0.004486667923629284,
+          -0.032907884567976,
+          0.00645859632641077,
+          -0.004978589713573456,
+          0.014537142589688301,
+          -0.0169289018958807,
+          -0.005589251406490803,
+          0.0024299249053001404,
+          -0.0051015703938901424,
+          -4.8271169362124056e-05,
+          0.02250967174768448,
+          -0.01602138951420784,
+          0.0030639106407761574,
+          0.0016517553012818098,
+          -0.0036618502344936132,
+          0.006967481225728989,
+          0.03619867190718651,
+          0.0044527421705424786,
+          -0.006085413973778486,
+          -0.021254422143101692,
+          -0.005712232086807489,
+          -0.0049446639604866505,
+          -0.007645994424819946,
+          -0.042508844286203384,
+          -0.005996359512209892,
+          -0.0010326120536774397,
+          -0.02588527463376522,
+          -0.024528248235583305,
+          -0.007896196097135544,
+          0.01187397912144661,
+          -0.01253552921116352,
+          -0.008455969393253326,
+          -0.014503217302262783,
+          0.007187997922301292,
+          0.0006684414111077785,
+          0.010991912335157394,
+          -0.015156285837292671,
+          0.008197286166250706,
+          9.607851097825915e-05,
+          -0.0023832772858440876,
+          -0.010008067823946476,
+          0.0110852075740695,
+          0.010728987865149975,
+          0.008061583153903484,
+          0.008040379732847214,
+          -0.008214249275624752,
+          0.015911132097244263,
+          -0.005512918811291456,
+          0.005275439005345106,
+          0.010822284035384655,
+          -0.01223019789904356,
+          -0.013375189155340195,
+          -0.0007044874364510179,
+          -0.009499182924628258,
+          -0.020050061866641045,
+          0.010423657484352589,
+          0.009880846366286278,
+          -0.009736662730574608,
+          0.011042800731956959,
+          0.0027607001829892397,
+          0.0019083180231973529,
+          -0.007484847214072943,
+          0.012968081049621105,
+          0.08813884854316711,
+          -0.0005459910025820136,
+          0.018387705087661743,
+          0.01601290889084339,
+          -0.0056571029126644135,
+          -0.030923234298825264,
+          -0.0014535022201016545,
+          -0.03418009728193283,
+          -0.008566227741539478,
+          -0.014384477399289608,
+          -0.026190605014562607,
+          -0.01898140273988247,
+          -0.014291182160377502,
+          0.01549554243683815,
+          0.006496762856841087,
+          0.0023048240691423416,
+          0.00865528266876936,
+          0.006280486937612295,
+          0.031008047983050346,
+          -0.010839246213436127,
+          -0.01044061966240406,
+          -0.02680126577615738,
+          0.011161540634930134,
+          0.000879946630448103,
+          0.019659915938973427,
+          -0.00891396589577198,
+          0.019439399242401123,
+          0.024528248235583305,
+          0.016199499368667603,
+          0.03136426582932472,
+          0.025393351912498474,
+          0.009388924576342106,
+          0.011806127615272999,
+          -0.005644380580633879,
+          -0.0026822469662874937,
+          0.014757659286260605,
+          -0.014520179480314255,
+          0.001473645563237369,
+          -0.002315425779670477,
+          -0.01387559249997139,
+          0.0033925652969628572,
+          -0.016555719077587128,
+          0.0036682114005088806,
+          0.012315012514591217,
+          -0.003678813111037016,
+          -0.0025762293953448534,
+          0.001580723444931209,
+          -6.443886377383024e-05,
+          -0.005360253155231476,
+          -0.000742653792258352,
+          -0.008540783077478409,
+          -0.008829151280224323,
+          0.00013795548875350505,
+          0.016080759465694427,
+          0.006547651253640652,
+          0.00362368393689394,
+          0.013858629390597343,
+          -0.008451728150248528,
+          -0.013214042410254478,
+          0.0054111420176923275,
+          -0.00035091847530566156,
+          0.003598239738494158,
+          0.014588030986487865,
+          -0.004520593211054802,
+          -0.007226164452731609,
+          0.002864597365260124,
+          0.00047389898099936545,
+          0.0006806334131397307,
+          -0.007768974639475346,
+          0.022950705140829086,
+          0.01935458555817604,
+          0.0010066378163173795,
+          0.012382864020764828,
+          0.01893051527440548,
+          0.0022624169941991568,
+          -0.03173745051026344,
+          -0.003996866289526224,
+          -0.007361866999417543,
+          0.013587224297225475,
+          -0.0021171728149056435,
+          -0.01682712510228157,
+          0.00807854626327753,
+          -0.009583997540175915,
+          -0.005576529540121555,
+          -0.009346517734229565,
+          -0.002582590328529477,
+          0.017912745475769043,
+          0.004461223725229502,
+          -0.024138102307915688,
+          0.01577542908489704,
+          0.011263317428529263,
+          -0.003971422091126442,
+          0.0013326420448720455,
+          -0.015724539756774902,
+          -0.004083800595253706,
+          -0.006602780427783728,
+          0.010813802480697632,
+          -0.0031529655680060387,
+          -0.013383670710027218,
+          -0.01774311624467373,
+          0.029532281681895256,
+          -0.005869138054549694,
+          0.005402660463005304,
+          -0.004821683745831251,
+          0.0011301483027637005,
+          0.02206863835453987,
+          0.014604994095861912,
+          -0.005606214515864849,
+          0.013502410613000393,
+          -0.013443040661513805,
+          -0.0063653006218373775,
+          0.028293995186686516,
+          -0.006051488686352968,
+          -0.007645994424819946,
+          0.007692642044275999,
+          -0.007302497047930956,
+          -0.024239879101514816,
+          0.0036872944328933954,
+          -0.02412113919854164,
+          0.004707184620201588,
+          -0.015122360549867153,
+          0.022238265722990036,
+          0.015164767391979694,
+          -0.008621356450021267,
+          -0.003013022243976593,
+          0.0026546823792159557,
+          -0.002449008170515299,
+          -0.00984692107886076,
+          0.0027925053145736456,
+          0.015945058315992355,
+          -0.004804720636457205,
+          0.015071472153067589,
+          -0.010092882439494133,
+          -0.024324694648385048,
+          -0.0005115352687425911,
+          -0.011068244464695454,
+          -0.01411307230591774,
+          0.0018902950687333941,
+          -0.027581557631492615,
+          0.007081980351358652,
+          -0.019948285073041916,
+          0.000553942343685776,
+          -0.015724539756774902,
+          -0.006992925424128771,
+          -0.007171035278588533,
+          0.020525021478533745,
+          -0.003299269825220108,
+          0.0041198465041816235,
+          0.0022730189375579357,
+          -0.007811381947249174,
+          0.01184853445738554,
+          -0.027021784335374832,
+          0.003909931518137455,
+          0.01677623577415943,
+          0.0033183530904352665,
+          -0.01835377886891365,
+          -0.0013994332402944565,
+          0.023900622501969337,
+          -0.008871558122336864,
+          -0.0029833372682332993,
+          -0.007400033064186573,
+          0.008812189102172852,
+          0.007192238699644804,
+          0.003044827375560999,
+          -0.007764733862131834,
+          0.023442627862095833,
+          0.01701371558010578,
+          0.008468691259622574,
+          0.019422436133027077,
+          -0.012925674207508564,
+          -0.00535177206620574,
+          0.022204341366887093,
+          -0.001378229702822864,
+          -0.014163960702717304,
+          0.0011460509849712253,
+          -0.005271198693662882,
+          -0.009354999288916588,
+          0.006242320407181978,
+          0.01518173050135374,
+          -0.010389731265604496,
+          -0.002245454117655754,
+          -0.014477772638201714,
+          0.005771602038294077,
+          -0.011729794554412365,
+          -0.0025741090066730976,
+          -0.01132268738001585,
+          -0.007730808574706316,
+          0.007349144667387009,
+          -0.005635899491608143,
+          -0.01669142208993435,
+          -0.009100556373596191,
+          -0.006204153876751661,
+          -0.003032105276361108,
+          0.010033512488007545,
+          0.025342464447021484,
+          -0.01387559249997139,
+          0.015716059133410454,
+          -0.0048895347863435745,
+          0.008803707547485828,
+          0.016046835109591484,
+          0.013358226045966148,
+          -0.013061377219855785,
+          -0.01630127616226673,
+          0.0011131855426356196,
+          -0.005805527325719595,
+          0.010762914083898067,
+          0.0016157092759385705,
+          0.010270991362631321,
+          0.0018309251172468066,
+          0.009219296276569366,
+          -0.01625886932015419,
+          0.002569868229329586,
+          0.0031423636246472597,
+          -0.008392359130084515,
+          -0.018760886043310165,
+          -0.010627211071550846,
+          -0.009066631086170673,
+          0.021763307973742485,
+          0.018285928294062614,
+          0.007879232987761497,
+          -0.016852568835020065,
+          0.02106783166527748,
+          -0.007450921926647425,
+          0.007289774715900421,
+          -0.014443847350776196,
+          0.00716255372390151,
+          0.02580045908689499,
+          -0.004596926271915436,
+          0.0035346290096640587,
+          -0.015758465975522995,
+          -0.005805527325719595,
+          -0.0007999032968655229,
+          0.00023284131020773202,
+          0.014647400937974453,
+          0.011186984367668629,
+          0.013773815706372261,
+          0.000118408483103849,
+          0.013197079300880432,
+          -0.020677685737609863,
+          -0.019422436133027077,
+          -0.021474938839673996,
+          -0.013129227794706821,
+          -0.004965867381542921,
+          0.0026292381808161736,
+          0.0047156657092273235,
+          0.002610154915601015,
+          0.009600959718227386,
+          -0.00987236574292183,
+          -0.005843693856149912,
+          -3.207034387742169e-05,
+          0.008379636332392693,
+          -0.0006069511291570961,
+          0.008328747935593128,
+          -0.013790778815746307,
+          0.010245547629892826,
+          0.007484847214072943,
+          -0.0016284313751384616,
+          0.010186177678406239,
+          0.003610961837694049,
+          -0.006878426298499107,
+          -0.008099749684333801,
+          0.00025099681806750596,
+          -0.010423657484352589,
+          0.000959989964030683,
+          -0.009812995791435242,
+          0.006123580504208803,
+          0.009397406131029129,
+          0.01921888254582882,
+          0.02150886505842209,
+          0.014070665463805199,
+          3.816635944531299e-05,
+          0.027717258781194687,
+          0.006450114771723747,
+          0.022051675245165825,
+          -0.008061583153903484,
+          0.030244721099734306,
+          0.0008868377772159874,
+          -0.022713225334882736,
+          0.011967274360358715,
+          0.0088206697255373,
+          0.0026568027678877115,
+          -0.012959599494934082,
+          -0.016267351806163788,
+          -0.006132062058895826,
+          -0.03412920981645584,
+          -0.02098301611840725,
+          0.005177902989089489,
+          0.006657909601926804,
+          0.021339235827326775,
+          0.008964854292571545,
+          -0.00597091531381011,
+          -0.01830288954079151,
+          0.005936989560723305,
+          -0.00392265385016799,
+          -0.010720506310462952,
+          -0.013180116191506386,
+          0.012289567850530148,
+          -0.0028306718450039625,
+          0.015393765643239021,
+          0.01005895622074604,
+          -0.00754421716555953,
+          -0.0031593265011906624,
+          -0.007739289663732052,
+          -0.010313399136066437,
+          0.02135619893670082,
+          -0.024256842210888863,
+          -0.00801069475710392,
+          -0.0072516086511313915,
+          -0.008222730830311775,
+          -0.009092075750231743,
+          0.003127521136775613,
+          0.013332782313227654,
+          -0.0032653443049639463,
+          0.0030787531286478043,
+          0.004486667923629284,
+          -0.0012563093332573771,
+          0.008210008032619953,
+          -0.01225564256310463,
+          -0.037793178111314774,
+          0.024918392300605774,
+          -0.007539976388216019,
+          0.020541982725262642,
+          0.014910325407981873,
+          0.0020323586650192738,
+          -0.0012139022583141923,
+          -0.018523406237363815,
+          0.011721313931047916,
+          -0.03183922544121742,
+          0.009075112640857697,
+          -0.016470905393362045,
+          0.034875571727752686,
+          -0.013366707600653172,
+          -0.020677685737609863,
+          0.01769222877919674,
+          0.0011725553777068853,
+          -0.011687387712299824,
+          -0.027157485485076904,
+          -0.005432345438748598,
+          0.005780083127319813,
+          0.008430524729192257,
+          0.015537949278950691,
+          -0.02746281772851944,
+          -0.010957986116409302,
+          -0.025851348415017128,
+          -0.004618129692971706,
+          0.015597319230437279,
+          -0.007552698720246553,
+          -5.648753813147778e-06,
+          0.0014651642413809896,
+          0.014435365796089172,
+          0.0029515319038182497,
+          0.012035125866532326,
+          0.0008990298374556005,
+          0.0031805301550775766,
+          -0.001866971142590046,
+          0.010220102965831757,
+          0.005729194730520248,
+          0.019965248182415962,
+          -0.01208601426333189,
+          -0.00860863458365202,
+          0.011644980870187283,
+          0.008880039677023888,
+          -0.03041434846818447,
+          -0.00784530770033598,
+          0.002970615169033408,
+          -0.015275025740265846,
+          0.02888769470155239,
+          0.005644380580633879,
+          -0.010194659233093262,
+          0.01263730600476265,
+          -0.023442627862095833,
+          -0.017624378204345703,
+          -0.009473739191889763,
+          -0.0030363460537046194,
+          -0.012968081049621105,
+          -0.0031020769383758307,
+          -0.0016676579834893346,
+          0.025189798325300217,
+          0.020423242822289467,
+          -0.011687387712299824,
+          0.016581162810325623,
+          -0.010330361314117908,
+          0.004957386292517185,
+          -0.005979396402835846,
+          0.019422436133027077,
+          0.024528248235583305,
+          -0.0010442740749567747,
+          0.006908111274242401,
+          -0.0042555490508675575,
+          0.0058097681030631065,
+          -0.010092882439494133,
+          0.011687387712299824,
+          -0.012102977372705936,
+          0.002580470172688365,
+          -0.00922777783125639,
+          -0.004172855522483587,
+          -0.012204754166305065,
+          0.046274591237306595,
+          0.0030617902521044016,
+          0.011695869266986847,
+          0.026920005679130554,
+          0.0027925053145736456,
+          0.009448294527828693,
+          0.030109018087387085,
+          0.020338429138064384,
+          -0.00865528266876936,
+          -0.026869118213653564,
+          0.002639839891344309,
+          -0.014952732250094414,
+          -0.01046606432646513,
+          0.0013220403343439102,
+          -0.009626404382288456,
+          0.013366707600653172,
+          0.0030066610779613256,
+          0.047461990267038345,
+          0.003341676900163293,
+          -0.009923254139721394,
+          -0.015800872817635536,
+          0.00781986303627491,
+          0.0014651642413809896,
+          0.006513725500553846,
+          -0.004999793134629726,
+          -0.01490184385329485,
+          0.0038738856092095375,
+          -0.013332782313227654,
+          -0.017200306057929993,
+          0.01782793179154396,
+          -0.004639333114027977,
+          -0.013807740993797779,
+          -0.006869944743812084,
+          0.01725967600941658,
+          -0.009719699621200562,
+          0.03009205497801304,
+          -0.003854802343994379,
+          -0.013629631139338017,
+          0.022577522322535515,
+          -0.0021309552248567343,
+          0.015800872817635536,
+          -0.0026016735937446356,
+          -0.005122773814946413,
+          -0.017607415094971657,
+          -0.013790778815746307,
+          0.01602138951420784,
+          0.004406094551086426,
+          -0.008332989178597927,
+          -0.0009811935015022755,
+          0.01229804940521717,
+          0.01153472252190113,
+          0.0058648972772061825,
+          0.015393765643239021,
+          -0.020999979227781296,
+          0.011246354319155216,
+          0.02093212865293026,
+          0.003155085723847151,
+          -0.007421236950904131,
+          0.0005931688356213272,
+          0.0012881146976724267,
+          0.0009610501583665609,
+          -0.0022772594820708036,
+          -0.015156285837292671,
+          -0.005402660463005304,
+          0.0017885180423036218,
+          0.013307337649166584,
+          0.007552698720246553,
+          0.005033718887716532,
+          -0.02188204601407051,
+          0.0017196065746247768,
+          -0.013672038912773132,
+          0.006327134557068348,
+          -0.001542557030916214,
+          0.0017259676242247224,
+          0.017624378204345703,
+          0.005593492183834314,
+          0.00413468899205327,
+          0.005551085341721773,
+          -0.02751370519399643,
+          -0.020541982725262642,
+          -0.004864090587943792,
+          0.00905814953148365,
+          -0.006797852925956249,
+          0.011458389461040497,
+          -0.01115305908024311,
+          0.012433752417564392,
+          0.009075112640857697,
+          0.007026851177215576,
+          -0.010126807726919651,
+          -0.006255042273551226,
+          -0.011008874513208866,
+          -0.010805320926010609,
+          4.200950206723064e-05,
+          -0.010974949225783348,
+          -0.011661943979561329,
+          0.015546430833637714,
+          0.00383783970028162,
+          0.004359446465969086,
+          -0.015173248946666718,
+          -0.0025529053527861834,
+          -0.011076726019382477,
+          -0.007018369622528553,
+          0.0062295980751514435,
+          0.018947478383779526,
+          0.0024574894923716784,
+          -0.0027734222821891308,
+          -0.008256656117737293,
+          -0.006445874460041523,
+          -0.0036300448700785637,
+          -0.022424858063459396,
+          -0.03643615171313286,
+          0.01325644925236702,
+          0.007641753647476435,
+          0.008973335847258568,
+          0.001959206536412239,
+          -0.012577936053276062,
+          -0.028735028579831123,
+          -0.005127014592289925,
+          0.0134684843942523,
+          0.02369706891477108,
+          -0.022475745528936386,
+          0.011907904408872128,
+          0.023849735036492348,
+          0.012951118871569633,
+          -0.0031444840133190155,
+          0.022662336006760597,
+          0.01256097387522459,
+          0.01911710575222969,
+          -0.0024532489478588104,
+          -0.006314412225037813,
+          0.0070607769303023815,
+          -0.012391345575451851,
+          -0.0013050774578005075,
+          -0.009354999288916588,
+          0.008829151280224323,
+          0.0027607001829892397,
+          -0.0023832772858440876,
+          0.007917399518191814,
+          -0.030482200905680656,
+          0.010661136358976364,
+          -0.022153452038764954,
+          0.013366707600653172,
+          0.014359032735228539,
+          -0.008557746186852455,
+          0.00946525763720274,
+          0.0007617369410581887,
+          0.0053093647584319115,
+          0.02317122183740139,
+          -0.002353592310100794,
+          0.009185370989143848,
+          -0.010576322674751282,
+          -0.0018553092377260327,
+          -0.018252002075314522,
+          0.003301390213891864,
+          0.003998986445367336,
+          -0.004596926271915436,
+          -0.012425270862877369,
+          0.02059287205338478,
+          -0.00473262881860137,
+          -0.0001673754013609141,
+          -0.024239879101514816,
+          -0.00029923487454652786,
+          -0.005843693856149912,
+          0.021525828167796135,
+          -0.017726154997944832,
+          -0.0005565927713178098,
+          0.009388924576342106,
+          0.010211621411144733,
+          0.0017916986253112555,
+          -0.0319579653441906,
+          0.023001592606306076,
+          -0.02993938885629177,
+          -0.025427278131246567,
+          -0.005025237333029509,
+          -0.004864090587943792,
+          -0.006564613897353411,
+          -0.022323081269860268,
+          0.012459196150302887,
+          0.012331975623965263,
+          -0.004196179565042257,
+          -0.014723733998835087,
+          -0.0018319853115826845,
+          -0.003093595616519451,
+          -0.00624656118452549,
+          -0.005093088839203119,
+          -0.0032377797178924084,
+          -0.005941230338066816,
+          0.0021161127369850874,
+          -0.010432139039039612,
+          -0.007739289663732052,
+          -0.00784106645733118,
+          0.007213442120701075,
+          0.0018574295099824667,
+          0.01206056959927082,
+          -0.002894282341003418,
+          -0.020677685737609863,
+          0.011950311250984669,
+          0.001987831201404333,
+          0.015724539756774902,
+          0.004719906486570835,
+          0.006738482974469662,
+          0.010941023007035255,
+          0.013790778815746307,
+          -0.011526240967214108,
+          -0.00982147641479969,
+          -0.01648786850273609,
+          -0.005907304584980011,
+          0.013739889487624168,
+          -0.012026644311845303,
+          -0.0040074680000543594,
+          -0.00012821512063965201,
+          -0.020083988085389137,
+          0.005440826993435621,
+          0.011102170683443546,
+          0.005152458790689707,
+          -0.011144577525556087,
+          -0.026292381808161736,
+          0.013790778815746307,
+          0.0058097681030631065,
+          -0.013392152264714241,
+          -0.003216576064005494,
+          0.01682712510228157,
+          0.026190605014562607,
+          0.02641112171113491,
+          0.011195465922355652,
+          0.015291988849639893,
+          -0.009007261134684086,
+          0.031143750995397568,
+          0.0076163094490766525,
+          0.007221923675388098,
+          -0.024765728041529655,
+          0.01574150286614895,
+          -0.0015467978082597256,
+          0.0022221303079277277,
+          -0.00020103600400034338,
+          -0.028582364320755005,
+          -0.009728181175887585,
+          -0.036741483956575394,
+          -0.015953538939356804,
+          -0.013205560855567455,
+          -0.004758073017001152,
+          0.0037212201859802008,
+          0.006416189484298229,
+          0.00021481829753611237,
+          0.011840052902698517,
+          -0.0017810967983677983,
+          0.009142964147031307,
+          0.006869944743812084,
+          -0.015444654040038586,
+          0.0025062577333301306,
+          -0.009354999288916588,
+          -0.0076884012669324875,
+          -0.0030087814666330814,
+          -0.03156781941652298,
+          0.0018690915312618017,
+          -0.02483357861638069,
+          -0.010627211071550846,
+          -0.008973335847258568,
+          -0.004953145515173674,
+          -0.0028985231183469296,
+          0.0023896382190287113,
+          -0.017794005572795868,
+          -0.018506444990634918,
+          -0.0110852075740695,
+          -0.020491095259785652,
+          -0.013460002839565277,
+          -0.0070607769303023815,
+          -0.003810275113210082,
+          -0.002542303642258048,
+          0.03251773864030838,
+          0.006717279553413391,
+          -0.003307751379907131,
+          -0.032348111271858215,
+          0.014223330654203892,
+          -0.005381457041949034,
+          -0.014842473901808262,
+          -0.00867648608982563,
+          0.014995139092206955,
+          -0.022662336006760597,
+          0.018319852650165558,
+          0.00943981297314167,
+          -0.021254422143101692,
+          0.013621150515973568,
+          0.027903851121664047,
+          0.025732608512043953,
+          0.010220102965831757,
+          -0.00850685779005289,
+          -6.033896170265507e-06,
+          0.002220009919255972,
+          0.001824564067646861,
+          0.015809355303645134,
+          0.011509277857840061,
+          0.015308951027691364,
+          -0.002446887781843543,
+          0.015436172485351562,
+          -0.02780207432806492,
+          -0.0015902650775387883,
+          -0.0076163094490766525,
+          -0.005894582252949476,
+          0.008405080996453762,
+          -0.008108231239020824,
+          -0.018913552165031433,
+          -0.025953125208616257,
+          -0.00010051800200017169,
+          0.024002401158213615,
+          -0.016886495053768158,
+          -0.014359032735228539,
+          -0.010881653986871243,
+          -0.0004110835143364966,
+          -0.007366107776761055,
+          0.010576322674751282,
+          0.022916778922080994,
+          -0.0005102100549265742,
+          0.0060218037106096745,
+          0.012289567850530148,
+          -0.02717444859445095,
+          0.01340063288807869,
+          -0.017895782366394997,
+          -0.0074254777282476425,
+          -0.00798949133604765,
+          -0.010881653986871243,
+          0.0033374361228197813,
+          -0.011526240967214108,
+          0.014800066128373146,
+          0.01606379635632038,
+          0.0034158893395215273,
+          -0.013137709349393845,
+          -0.0027522186283022165,
+          -0.020423242822289467,
+          0.00011118603288196027,
+          -0.007752011995762587,
+          0.003678813111037016,
+          0.009957179427146912,
+          0.004340363200753927,
+          0.021525828167796135,
+          0.008786744438111782,
+          -0.03680933639407158,
+          -0.0021691215224564075,
+          0.012925674207508564,
+          -0.00213307561352849,
+          -0.028378808870911598,
+          -0.012713639065623283,
+          0.009134482592344284,
+          0.00038484414108097553,
+          0.003937496338039637,
+          0.003950218204408884,
+          -0.004558759741485119,
+          0.017471712082624435,
+          -0.04043937847018242,
+          -0.01840466819703579,
+          -0.00040207200800068676,
+          -0.016470905393362045,
+          -0.007103183772414923,
+          0.008481413125991821,
+          0.0026037939824163914,
+          0.010499989613890648,
+          0.009668811224400997,
+          -0.018438592553138733,
+          0.01178068295121193,
+          -0.0009260643273591995,
+          -0.01373140886425972,
+          0.0013167394790798426,
+          0.0007988431607373059,
+          0.009448294527828693,
+          0.0021055107936263084,
+          -0.00862983800470829,
+          -0.00786651112139225,
+          -0.0100165493786335,
+          0.004842887166887522,
+          0.029684947803616524,
+          -0.00023959993268363178,
+          -0.005856416188180447,
+          0.005016756244003773,
+          -0.010381249710917473,
+          -0.013273412361741066,
+          -0.01432510744780302,
+          0.0006949458038434386,
+          0.013460002839565277,
+          0.0073067378252744675,
+          0.014749177731573582,
+          -0.011170021258294582,
+          -0.005071885418146849,
+          0.011390537954866886,
+          -0.01170435082167387,
+          0.013409114442765713,
+          -0.01625886932015419,
+          -0.015995945781469345,
+          -0.007993732579052448,
+          0.0025401832535862923,
+          0.015139322727918625,
+          0.00922777783125639,
+          -0.000635575910564512,
+          0.0333828441798687,
+          -0.008803707547485828,
+          0.020474132150411606,
+          -0.01425725594162941,
+          -0.01697130873799324,
+          0.005283920560032129,
+          -0.016564199700951576,
+          -0.007531495299190283,
+          -0.004859849810600281,
+          -0.020558945834636688,
+          0.007166794501245022,
+          0.018726961687207222,
+          -0.004189818166196346,
+          0.016148611903190613,
+          0.008167601190507412,
+          -0.0039523388259112835,
+          0.0030893548391759396,
+          -0.00927018467336893,
+          -0.012069051153957844,
+          0.004098643083125353,
+          0.0146643640473485,
+          -0.01759045198559761,
+          0.017522599548101425,
+          0.004516352899372578,
+          0.007285534404218197,
+          0.015512505546212196,
+          0.012985044158995152,
+          -0.02680126577615738,
+          -0.011865497566759586,
+          0.022441819310188293,
+          -0.01206056959927082,
+          0.005597732961177826,
+          0.011840052902698517,
+          0.0038993298076093197,
+          -0.009397406131029129,
+          -0.013510892167687416,
+          0.0011259076418355107,
+          -0.00991477258503437,
+          0.013544817455112934,
+          -0.00015120769967325032,
+          -0.041083965450525284,
+          0.007997972890734673,
+          -0.006441633682698011,
+          0.003487981390208006,
+          0.0027331355959177017,
+          -0.0058394530788064,
+          0.023256035521626472,
+          0.02259448543190956,
+          -0.015504023991525173,
+          -0.00588610116392374,
+          -0.013960406184196472,
+          0.012798452749848366,
+          0.008693449199199677,
+          0.007976769469678402,
+          -0.02493535540997982,
+          -0.003610961837694049,
+          0.007213442120701075,
+          -0.0064373929053545,
+          -0.00473262881860137,
+          0.014545624144375324,
+          -0.009346517734229565,
+          -0.0013951925793662667,
+          -0.003825117601081729,
+          0.0015043907333165407,
+          0.013044414110481739,
+          0.0036703317891806364,
+          0.015529467724263668,
+          -0.01950725167989731,
+          0.013833185657858849,
+          -0.0058648972772061825,
+          0.0010368528310209513,
+          0.009897809475660324,
+          -0.014197885990142822,
+          -0.026139715686440468,
+          0.01630127616226673,
+          0.00692507391795516,
+          0.00037477246951311827,
+          -0.023086408153176308,
+          -0.00645859632641077,
+          -0.01993132196366787,
+          0.0029409301932901144,
+          -0.00028465743525885046,
+          0.0038738856092095375,
+          0.025088021531701088,
+          -0.000675862655043602,
+          -0.015860242769122124,
+          -0.005797046236693859,
+          0.001527714659459889,
+          0.0015923853497952223,
+          0.007145591080188751,
+          -0.0193036962300539,
+          -0.025088021531701088,
+          -0.0050464412197470665,
+          -0.013858629390597343,
+          0.018319852650165558,
+          -0.017760079354047775,
+          0.015716059133410454,
+          -0.003555832663550973,
+          0.008922447450459003,
+          0.00562741793692112,
+          -0.0029812168795615435,
+          0.03294181078672409,
+          -0.015597319230437279,
+          0.010304917581379414,
+          0.007548457942903042,
+          -0.011382057331502438,
+          -0.011449907906353474,
+          0.01774311624467373,
+          -0.010508471168577671,
+          0.0032505018170922995,
+          0.0038463210221379995,
+          -0.00929562933743,
+          0.023476552218198776,
+          0.0027500982396304607,
+          0.014757659286260605,
+          -0.01070354413241148,
+          -0.019914358854293823,
+          0.0032907885033637285,
+          0.010686581023037434,
+          0.0013496049214154482,
+          0.00812519434839487,
+          -0.02888769470155239,
+          0.00483864638954401,
+          -0.009643367491662502,
+          0.02106783166527748,
+          0.009728181175887585,
+          0.0018097215797752142,
+          -0.02999027818441391,
+          0.0008857776410877705,
+          -0.014070665463805199,
+          0.02651289850473404,
+          0.005165180657058954,
+          0.003301390213891864,
+          -0.01139901950955391,
+          -0.003975662402808666,
+          0.012823897413909435,
+          -0.005754638928920031,
+          -0.009643367491662502,
+          -0.0038039139471948147,
+          0.011178502812981606,
+          0.004821683745831251,
+          -0.013426077552139759,
+          -0.013934962451457977,
+          -0.011882460676133633,
+          0.010160733014345169,
+          0.0012520686723291874,
+          0.0037424236070364714,
+          -0.004323400557041168,
+          0.007484847214072943,
+          -0.011763720773160458,
+          0.022950705140829086,
+          0.009668811224400997,
+          0.01592809520661831,
+          0.01893051527440548,
+          -0.008176082745194435,
+          0.009643367491662502,
+          0.008837632834911346,
+          0.01046606432646513,
+          0.016174055635929108,
+          0.022373968735337257,
+          0.021899009123444557,
+          0.019710805267095566,
+          -0.021949898451566696,
+          -0.0010585864074528217,
+          -0.006602780427783728,
+          -0.0073237004689872265,
+          -0.011958792805671692,
+          -0.018998365849256516,
+          0.001702643814496696,
+          -0.03372209891676903,
+          -0.019015328958630562,
+          -0.0017206667689606547,
+          0.007090461906045675,
+          0.013027451001107693,
+          -0.014333589002490044,
+          0.012985044158995152,
+          0.008583190850913525,
+          0.00036417070077732205,
+          -0.014486254192888737,
+          -0.010949504561722279,
+          -0.012544010765850544,
+          0.004367928020656109,
+          0.021610641852021217,
+          0.004153772257268429,
+          0.0049446639604866505,
+          -0.01301896944642067,
+          0.00937196146696806,
+          0.02822614461183548,
+          0.0015351359033957124,
+          -0.00953310914337635,
+          0.0058479346334934235,
+          -0.004073198884725571,
+          0.008752819150686264,
+          -0.012162347324192524,
+          -0.010669617913663387,
+          0.000553412246517837,
+          0.005754638928920031,
+          -0.013748371042311192,
+          0.02140708826482296,
+          -0.004196179565042257,
+          0.018285928294062614,
+          -2.4102455427055247e-06,
+          -0.0007198599632829428,
+          0.0024511285591870546,
+          0.003971422091126442,
+          0.02198382467031479,
+          0.005275439005345106,
+          -0.0008550324710085988,
+          -0.023730995133519173,
+          -0.0005356542533263564,
+          -0.0029939389787614346,
+          -0.020134875550866127,
+          0.004906497895717621,
+          0.0069589996710419655,
+          -0.004359446465969086,
+          0.02717444859445095,
+          0.013960406184196472,
+          0.0013156792847439647,
+          0.005669824779033661,
+          0.0014089748729020357,
+          0.007366107776761055,
+          -0.008498376235365868,
+          -0.00946525763720274,
+          0.0041262079030275345,
+          -0.0027755426708608866,
+          0.0051185330376029015,
+          -0.027089634910225868,
+          0.013570261187851429,
+          0.0050634038634598255,
+          0.0038166360463947058,
+          0.006208394654095173,
+          0.008693449199199677,
+          -0.005996359512209892,
+          0.0012128421803936362,
+          0.006975962780416012,
+          0.008260896429419518,
+          0.001960266614332795,
+          0.031075898557901382,
+          -0.027428891509771347,
+          -0.00036019503022544086,
+          -0.005945470649749041,
+          -0.005233032163232565,
+          -0.0032547423616051674,
+          -0.01816718839108944,
+          -0.013782297261059284,
+          -0.00392053322866559,
+          -0.001568001345731318,
+          0.023900622501969337,
+          0.013578742742538452,
+          0.01611468568444252,
+          0.01149231567978859,
+          -0.006344097200781107,
+          -0.018862662836909294,
+          0.009176889434456825,
+          -0.0035367493983358145,
+          -0.00619991309940815,
+          0.031686559319496155,
+          0.016199499368667603,
+          -0.0036830538883805275,
+          0.01887962594628334,
+          -0.0051354956813156605,
+          0.011178502812981606,
+          -0.016174055635929108,
+          0.001324160723015666,
+          -0.006869944743812084,
+          -0.0031741689890623093,
+          0.013426077552139759,
+          -0.002298463135957718,
+          0.007930121384561062,
+          -0.0024574894923716784,
+          -0.00526695791631937,
+          0.0076714386232197285,
+          -0.0031317619141191244,
+          -0.004162253811955452,
+          0.013977369293570518,
+          -0.003746664384379983,
+          -0.011161540634930134,
+          -0.015368320979177952,
+          0.011127614416182041,
+          -0.01328189391642809,
+          0.006216876208782196,
+          -0.0036745723336935043,
+          -0.007925881072878838,
+          -0.002419323194772005,
+          0.03060094080865383,
+          -0.0020779462065547705,
+          0.017794005572795868,
+          0.004851368721574545,
+          -0.02203471213579178,
+          -0.02083035185933113,
+          -0.008167601190507412,
+          0.03151693195104599,
+          -0.00507612619549036,
+          -0.009159926325082779,
+          -0.019795618951320648,
+          -0.01091557927429676,
+          0.004268271382898092,
+          -0.012823897413909435,
+          -0.0003002950397785753,
+          -5.681884431396611e-05,
+          0.003638526424765587,
+          -0.0007946024416014552,
+          -0.01408762764185667,
+          -0.004656296223402023,
+          0.00031142690568231046,
+          0.01253552921116352,
+          0.005059163086116314,
+          0.002565627684816718,
+          0.014308144338428974,
+          0.005470511503517628,
+          -0.02751370519399643,
+          -0.006551892030984163,
+          0.013315819203853607,
+          0.012527047656476498,
+          0.012348937802016735,
+          -0.002095969393849373,
+          -0.007900436408817768,
+          -0.02659771218895912,
+          -0.010381249710917473,
+          -0.0362325981259346,
+          -0.0013803500914946198,
+          -0.023052481934428215,
+          -0.018896589055657387,
+          -0.02103390544652939,
+          -0.021254422143101692,
+          -0.010330361314117908,
+          -0.01067809946835041,
+          0.025681721046566963,
+          0.010983430780470371,
+          0.0026716452557593584,
+          0.00791315920650959,
+          -0.023374775424599648,
+          0.0027119319420307875,
+          -0.009541589766740799,
+          -0.006963240448385477,
+          0.02359529212117195,
+          0.00011436655768193305,
+          0.016793198883533478,
+          -0.0014216969721019268,
+          -0.011907904408872128,
+          0.007395792752504349,
+          -0.007701123598963022,
+          -0.007662957068532705,
+          0.010932542383670807,
+          -0.012628824450075626,
+          -0.013485447503626347,
+          0.016428498551249504,
+          0.026156678795814514,
+          -0.002173362299799919,
+          0.0005157759878784418,
+          -0.024155065417289734,
+          -0.0011969394981861115,
+          -0.0020355391316115856,
+          0.0009202333749271929,
+          -0.0015828438336029649,
+          0.007421236950904131,
+          0.002241213573142886,
+          0.0034116485621780157,
+          0.02006702497601509,
+          -0.012603380717337132,
+          0.01816718839108944,
+          0.007506051100790501,
+          -0.001392011996358633,
+          0.021491901949048042,
+          0.010949504561722279,
+          0.0016284313751384616,
+          -0.023052481934428215,
+          -0.004363687243312597,
+          -0.010033512488007545,
+          -0.001446080976165831,
+          0.014435365796089172,
+          0.013154672458767891,
+          -0.020016135647892952,
+          -0.025257648900151253,
+          4.452741995919496e-05,
+          -0.001272212015464902,
+          0.009829957969486713,
+          -0.015444654040038586,
+          -0.0020620436407625675,
+          0.0006551892147399485,
+          -0.018557332456111908,
+          0.015571875497698784,
+          0.013417595997452736,
+          0.0033925652969628572,
+          0.01332430075854063,
+          -0.005283920560032129,
+          0.015275025740265846,
+          -0.007743530441075563,
+          0.010203140787780285,
+          -0.003099956549704075,
+          -0.013146190904080868,
+          0.006288968026638031,
+          -0.00784106645733118,
+          -0.01163649931550026,
+          -0.010118326172232628,
+          -0.017369935289025307,
+          -0.021542789414525032,
+          -0.020915165543556213,
+          -0.019235845655202866,
+          0.015291988849639893,
+          -0.011365094222128391,
+          0.009261703118681908,
+          0.004961627069860697,
+          -0.006081173662096262,
+          0.021101756021380424,
+          0.005890341475605965,
+          -0.0017641340382397175,
+          0.007209201343357563,
+          -0.003203853964805603,
+          -0.0008216369315050542,
+          -0.004393372219055891,
+          0.032348111271858215,
+          -0.0013315819669514894,
+          0.024494322016835213,
+          -0.0006445874460041523,
+          0.008205767720937729,
+          0.0016125286929309368,
+          0.0015892048832029104,
+          -0.023137295618653297,
+          0.012908711098134518,
+          -0.011314205825328827,
+          -0.014571068808436394,
+          -0.02412113919854164,
+          -0.00022078178881201893,
+          0.006178709678351879,
+          0.017047641798853874,
+          0.0041431705467402935,
+          -0.0009424970485270023,
+          0.008646801114082336,
+          -0.01950725167989731,
+          0.002160640200600028,
+          0.009007261134684086,
+          -0.00156694115139544,
+          -0.008290581405162811,
+          -0.02030450478196144,
+          -0.0019570861477404833,
+          -0.027394965291023254,
+          0.005568047985434532,
+          0.0007946024416014552,
+          -0.01504602748900652,
+          0.010949504561722279,
+          -0.007904677651822567,
+          0.021339235827326775,
+          0.01830288954079151,
+          0.0011916386429220438,
+          -0.006581577006727457,
+          -0.015300470404326916,
+          -0.015962019562721252,
+          0.013527854345738888,
+          0.015308951027691364,
+          0.011110651306807995,
+          0.006178709678351879,
+          0.005398419685661793,
+          -0.004329761490225792,
+          0.01446929108351469,
+          0.00937196146696806,
+          -0.028599325567483902,
+          1.502469149272656e-05,
+          -0.010983430780470371,
+          0.023680105805397034,
+          -0.02330692484974861,
+          0.014002813957631588,
+          -0.020999979227781296,
+          0.01935458555817604,
+          -0.0003612551954574883,
+          0.012722120620310307,
+          0.009278666228055954,
+          8.89223229023628e-05,
+          -0.013375189155340195,
+          0.005852175410836935,
+          0.001298716408200562,
+          -0.0011555926175788045,
+          0.0048174429684877396,
+          0.022492708638310432,
+          -0.02951531857252121,
+          -0.004893775563687086,
+          -0.005190624855458736,
+          -2.678602686501108e-05,
+          -0.010313399136066437,
+          -0.024188991636037827,
+          0.0033586397767066956,
+          -0.015012102201581001,
+          0.002998179756104946,
+          0.018862662836909294,
+          0.00516094034537673,
+          0.02111871913075447,
+          -0.02330692484974861,
+          -0.009829957969486713,
+          -0.013553299009799957,
+          0.017437785863876343,
+          -0.0029536522924900055,
+          0.0016825003549456596,
+          0.006517966277897358,
+          -0.013909517787396908,
+          0.0058097681030631065,
+          -0.0073237004689872265,
+          -0.00441033486276865,
+          0.06323742121458054,
+          -0.0005393648752942681,
+          -0.007014128845185041,
+          -0.01022858452051878,
+          -0.010092882439494133,
+          0.015317432582378387,
+          0.006929314695298672,
+          -0.003528268076479435,
+          -0.010695062577724457,
+          -0.004397612996399403,
+          -0.023256035521626472,
+          -0.01648786850273609,
+          -0.004217382986098528,
+          -0.0074254777282476425,
+          0.01178068295121193,
+          -0.000947267864830792,
+          -0.009999586269259453,
+          0.003445574315264821,
+          -0.0054111420176923275,
+          -0.00846020970493555,
+          0.006378022953867912,
+          -0.01615709252655506,
+          0.001985711045563221,
+          0.008939409628510475,
+          -0.015554912388324738,
+          0.018252002075314522,
+          -0.014528661035001278,
+          -0.0048174429684877396,
+          0.022170415148139,
+          -0.021288348361849785,
+          0.005394178908318281,
+          0.0011725553777068853,
+          0.01935458555817604,
+          -0.019625991582870483,
+          -0.009456776082515717,
+          0.03173745051026344,
+          -0.012645787559449673,
+          0.008400839753448963,
+          0.00853230245411396,
+          0.006759686395525932,
+          0.0018902950687333941,
+          0.0011736155720427632,
+          0.007412755396217108,
+          0.014859436079859734,
+          -0.01636064611375332,
+          -0.01950725167989731,
+          0.007832585833966732,
+          -0.004448501393198967,
+          -0.0036003601271659136,
+          0.010372769087553024,
+          -0.009278666228055954,
+          0.0025529053527861834,
+          -0.0053560128435492516,
+          0.01470677088946104,
+          -0.005682547111064196,
+          -0.006187191233038902,
+          0.018099335953593254,
+          -0.01039821282029152,
+          0.0053093647584319115,
+          0.02927783876657486,
+          -0.012544010765850544,
+          -0.016911938786506653,
+          -0.0009944457560777664,
+          -0.00522030983120203,
+          -0.013332782313227654,
+          0.01740385964512825,
+          -0.01702219620347023,
+          -0.005525640677660704,
+          -0.02393454872071743,
+          -0.004520593211054802,
+          0.005826731212437153,
+          0.0190662182867527,
+          0.013273412361741066,
+          -0.019337622448801994,
+          0.003812395269051194,
+          -0.011271798983216286,
+          0.023493515327572823,
+          0.0031953726429492235,
+          0.012196272611618042,
+          -0.0010442740749567747,
+          -0.03894665092229843,
+          -0.012764527462422848,
+          -0.03562193736433983,
+          0.014825510792434216,
+          -0.004813202191144228,
+          -0.024952318519353867,
+          -0.02627541869878769,
+          -0.0021161127369850874,
+          -0.014723733998835087,
+          0.007187997922301292,
+          0.012611862272024155,
+          0.01425725594162941,
+          0.012645787559449673,
+          0.03775925189256668,
+          0.0021277745254337788,
+          -0.0004309618379920721,
+          -0.0008476112852804363,
+          0.022289155051112175,
+          0.01149231567978859,
+          -0.01720878854393959,
+          0.009142964147031307,
+          0.0043721687979996204,
+          -0.012187791056931019,
+          -0.004596926271915436,
+          -0.017709191888570786,
+          -0.016479386016726494,
+          0.014842473901808262,
+          -0.01877784915268421,
+          0.007561180274933577,
+          0.01578390970826149,
+          -0.010169214569032192,
+          0.009185370989143848,
+          -0.003369241487234831,
+          -0.005173662211745977,
+          -0.01490184385329485,
+          0.0010071678552776575,
+          -0.008693449199199677,
+          0.01582631841301918,
+          -0.012739082798361778,
+          0.01864214614033699,
+          -0.014172442257404327,
+          -0.00826937798410654,
+          0.021339235827326775,
+          0.022475745528936386,
+          -0.009567034430801868,
+          0.008710411377251148,
+          -0.021525828167796135,
+          0.0008878979715518653,
+          0.014630438759922981,
+          -0.019490288570523262,
+          0.02856540121138096,
+          -0.012925674207508564,
+          0.014613475650548935,
+          0.0011322686914354563,
+          -0.009287147782742977,
+          0.002555025741457939,
+          -0.010550878010690212,
+          0.013672038912773132,
+          -0.0019411834655329585,
+          0.012094495818018913,
+          -0.01615709252655506,
+          0.015809355303645134,
+          -0.011653462424874306,
+          0.012510084547102451,
+          0.009244740940630436,
+          -0.005860656965523958,
+          -0.0005868077860213816,
+          -0.013069857843220234,
+          0.0052542355842888355,
+          -0.003812395269051194,
+          0.023849735036492348,
+          0.006399226374924183,
+          -0.015809355303645134,
+          0.0014821270015090704,
+          0.0028094681911170483,
+          0.015537949278950691,
+          0.011314205825328827,
+          0.007086221128702164,
+          -0.013095302507281303,
+          0.01935458555817604,
+          -0.00987236574292183,
+          0.008998779579997063,
+          -0.0011036439100280404,
+          0.011407501064240932,
+          -0.022577522322535515,
+          -0.00588610116392374,
+          0.007048054598271847,
+          -0.015291988849639893,
+          0.012857822701334953,
+          -0.0076163094490766525,
+          0.0006000599823892117,
+          0.007137109525501728,
+          0.006590058095753193,
+          0.019439399242401123,
+          0.004276752937585115,
+          0.002393878996372223,
+          0.003301390213891864,
+          -0.011619536206126213,
+          -0.0159789826720953,
+          -0.008668004535138607,
+          -0.020185764878988266,
+          0.005088848061859608,
+          0.010237066075205803,
+          0.02064376138150692,
+          0.0008592731901444495,
+          -0.008718892931938171,
+          -0.00801069475710392,
+          -0.02011791244149208,
+          0.00045190032687969506,
+          0.021135682240128517,
+          0.012416789308190346,
+          -0.011543204076588154,
+          -0.011415982618927956,
+          0.020491095259785652,
+          -0.00654341047629714,
+          -0.02103390544652939,
+          -0.012934155762195587,
+          -0.006836019456386566,
+          -0.0027267744299024343,
+          -0.007981009781360626,
+          0.016581162810325623,
+          0.004775035660713911,
+          0.0076332720927894115,
+          0.03613082319498062
+        ],
+        "title": "E Relation of Upstream to Downstream Model Performance",
+        "keyphrases": [
+          "Upstream to Downstream Model Performance"
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "addd7c61-bdd3-470e-8c4f-439af824f89e",
+      "properties": {
+        "page_content": "F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+        "embedding": [
+          0.0002445570513886185,
+          -0.025035678422212698,
+          -0.026056660824720954,
+          0.0021234386221176942,
+          -0.007526403638487191,
+          0.018963965001103752,
+          -0.007728070932730228,
+          0.062213823247851435,
+          -0.00501600735659763,
+          0.031461940272958254,
+          0.014585783857839997,
+          0.010118033107401152,
+          0.009983318452435566,
+          -0.03282616973330728,
+          0.015192449242879527,
+          0.01480596701302375,
+          0.019405922018728892,
+          0.022192910412596127,
+          0.008048967251594976,
+          -0.0367043822293015,
+          0.017737679976556845,
+          -0.022757028540067756,
+          -0.06290354368907101,
+          0.016842087557557506,
+          -0.0050319451760917995,
+          0.019311667659179976,
+          0.008548884527537198,
+          0.02344214718766695,
+          -0.042417710048062586,
+          0.01950972309054391,
+          0.021637767293446648,
+          0.010581067128107188,
+          -0.014214247204390872,
+          0.01016365063627519,
+          -0.025782209417014867,
+          -0.02611397356963557,
+          0.034251067799009986,
+          0.0071402676991737055,
+          -0.020147891665364803,
+          0.056472902950962155,
+          -0.006027534100459245,
+          0.005473493744726429,
+          -0.03139313966858068,
+          0.00043634965151753214,
+          0.013171990459657583,
+          -0.016474119463665594,
+          0.0075792906007691105,
+          -0.0032442837467945356,
+          0.02492845005795393,
+          -0.04366675539501486,
+          -0.006033838756045725,
+          -0.018660330752604155,
+          0.02088611082654532,
+          -0.019387265112916365,
+          -0.01222746328548615,
+          0.005977009213829137,
+          0.0012690574665308455,
+          0.05430319716489232,
+          0.058075720286678886,
+          0.02554562736016709,
+          0.02997758641056458,
+          0.00014178556329408164,
+          0.004139467932725944,
+          -0.004454862143349584,
+          0.031094774362479788,
+          0.0017837745263941452,
+          -0.013542834385584962,
+          0.01983846902748998,
+          0.010136662804503968,
+          -0.0387179389182337,
+          -0.06156798393873734,
+          0.015844092498186847,
+          -0.020820695961906926,
+          -0.02678200120240037,
+          -4.570339833017677e-05,
+          -0.015544616932001905,
+          0.005434882475939711,
+          0.050835936350804696,
+          -0.004985250023269435,
+          0.0016509410141492726,
+          0.022801366331672433,
+          -0.004080282248725238,
+          0.0078367492683837,
+          -0.00034876063727630653,
+          0.039772801289415724,
+          0.0288510926430687,
+          -0.022609202992981695,
+          -0.07091181813750308,
+          0.045194222785472735,
+          -0.03225547853169315,
+          -0.032585463285183205,
+          0.01670145480899693,
+          0.03286138231600811,
+          -0.009733112712376757,
+          -0.030197857075781466,
+          0.0731500616714208,
+          -0.005545147701735842,
+          0.017072915518860705,
+          -0.021002529816498382,
+          0.010525361666209598,
+          -0.012745643075817463,
+          0.025352505498532282,
+          -0.003102906957075941,
+          0.016391641977220053,
+          0.02534354270624064,
+          -0.008108690979345334,
+          -0.006451703723989414,
+          0.0065097623653779324,
+          0.04453924610212246,
+          0.0889545538715994,
+          0.019774398083206815,
+          -0.003981259900744729,
+          -0.046615274629371053,
+          0.00732791197790897,
+          -0.02916202133853672,
+          -0.04303144138453202,
+          -0.04367640508681853,
+          -0.010910501188273782,
+          0.023755744360759886,
+          -0.003399898430475389,
+          0.03910227884474275,
+          0.04137187598162877,
+          0.03363334616934253,
+          0.00916121492790106,
+          -0.031686589382074935,
+          0.026907320660055866,
+          -0.01693827759642556,
+          0.002542422168436765,
+          0.00734783450287368,
+          0.008189080458799796,
+          -0.006352648760821389,
+          -0.002333605867785783,
+          -0.03831808898525347,
+          -0.036762698962852246,
+          0.012773386000855496,
+          -0.008349610778914014,
+          0.027936454763857345,
+          0.07015560257394027,
+          -0.03713244149832974,
+          -0.0066245405611366675,
+          0.02802618921440782,
+          0.0017169880395367697,
+          0.023346396749133396,
+          0.007605522879978981,
+          -0.023484167590494896,
+          0.008594274917753499,
+          -0.0071765874950466295,
+          0.0677537727828362,
+          -0.008126000179550069,
+          0.011046754551416857,
+          -0.019516727128214784,
+          0.031654663874166704,
+          0.0007057013357055938,
+          0.021535694192642306,
+          0.03313372731162823,
+          -0.005794425715079591,
+          -0.03865395674172617,
+          0.032271538528276474,
+          -0.029790294001121286,
+          0.0013647166052139265,
+          0.014194415049797637,
+          -0.004899479294176156,
+          -0.015920769574443834,
+          0.0013708196727143433,
+          0.029757298661028133,
+          0.045717552753231847,
+          0.042911032378569906,
+          0.0016019404554248433,
+          -0.06844613496660937,
+          0.06365820914938357,
+          0.012895947255192629,
+          0.03208498337942363,
+          0.009724409704032419,
+          0.042361427690196844,
+          0.008120380964731038,
+          -0.0004384537671641681,
+          -0.005540416382489673,
+          -0.005125920468958917,
+          -0.021186731704155592,
+          0.03258917022091572,
+          -0.01443327736487225,
+          -0.007107908179884635,
+          -0.00206346150761894,
+          0.014042865018508809,
+          0.028517557440008168,
+          0.017283745982594707,
+          0.0029564774390786045,
+          0.04378212980825259,
+          -0.019454732028746872,
+          -0.009437148551161395,
+          0.018820938021162286,
+          0.0008903198503990673,
+          0.025297040557527045,
+          -0.0011429799732923295,
+          -0.01670583390791835,
+          -0.009421860087826363,
+          -0.0328764813310764,
+          0.009563088287577815,
+          -0.0015347427243049344,
+          0.019228666906882602,
+          -0.007174098839673089,
+          -0.04068735475327182,
+          0.008903031576514094,
+          -0.011472954071988202,
+          0.015735403376626513,
+          -0.007084760348074423,
+          0.008729329609340102,
+          -0.00638454270428521,
+          0.016210007209816298,
+          -0.02460475066044075,
+          -0.009047036293783832,
+          -0.022598561172568143,
+          -0.020144302113939427,
+          -0.010886272544512816,
+          -0.010886770387412563,
+          0.008591903125597681,
+          -0.012216526799178626,
+          0.023956058440750026,
+          -0.017402060516065108,
+          -0.018328740623422317,
+          -0.00046940812181371226,
+          0.022921229116282273,
+          0.009026508358954174,
+          -0.00446208425714566,
+          -0.04421346673289319,
+          -0.01214251475197077,
+          -0.016860336137191526,
+          0.005985059770480569,
+          0.02804423359971541,
+          -0.03510963462663318,
+          0.012588036039507566,
+          -0.0018909139908325232,
+          0.0019847602212350957,
+          0.011374423039138836,
+          0.015486762163219968,
+          -0.027040748338117617,
+          -0.016329845569406535,
+          -0.012600824227928017,
+          -0.01765236203915568,
+          0.01673827406092187,
+          0.0016641690585652725,
+          0.003997011410859195,
+          -0.00499922800986537,
+          -0.02782499674212141,
+          -0.008774389172636544,
+          -0.011506812701874534,
+          -0.009783555199790996,
+          -0.015740884114025612,
+          0.013493654174963239,
+          0.014254624157312108,
+          -0.033270488877124596,
+          -0.018446763439664365,
+          0.02247061755496804,
+          -0.0035396859846893343,
+          -0.03759408156279411,
+          0.04544560120093284,
+          0.052893430652542234,
+          -0.02220682841881151,
+          0.002828355769475809,
+          -0.03469141962159207,
+          0.05950248906327849,
+          -0.03444371252359212,
+          -0.03378776392860449,
+          0.06521072243320349,
+          -0.009713026070923203,
+          0.015249223153809483,
+          0.03131206469454657,
+          0.0034865285730464384,
+          -0.0025997752599464445,
+          0.00912836988165194,
+          0.033031075123542926,
+          -0.014878258361522118,
+          -0.013101723481793925,
+          -0.01797681408884685,
+          -0.002397237505725697,
+          -0.03438738791694146,
+          -0.021713511748922477,
+          0.00836495266415799,
+          0.013064910474172053,
+          0.017110113503472666,
+          0.017514586931181946,
+          0.016195079102773242,
+          -0.0024109742901844617,
+          0.01545592172528996,
+          0.001046760697009891,
+          -8.863738247355372e-05,
+          -0.010979717992873124,
+          -0.017762086340346584,
+          -0.052667758929128256,
+          -0.018996338856872796,
+          -0.04421628171000429,
+          -0.005982494173649518,
+          0.0012420734363470585,
+          0.009886466834005627,
+          0.04949340056515024,
+          -0.01295176306623405,
+          0.003508142432105508,
+          0.010930632194984236,
+          0.01019261004578912,
+          -0.006556730312565122,
+          -0.032765993356945154,
+          -0.006485728779853869,
+          0.025709969056450885,
+          -0.022932446214570953,
+          -0.021792413124305354,
+          -0.007047777916806282,
+          0.0038659532636633486,
+          0.02321123092409853,
+          -0.026003302747277822,
+          0.017643847938685852,
+          0.010129447679208158,
+          0.009551458127273026,
+          -0.017747795509094742,
+          -0.02891499441428287,
+          -0.030321565228144824,
+          0.039350178167852384,
+          0.02899873253152933,
+          0.011338359735211436,
+          -0.033634061116901,
+          -0.013629902520109783,
+          0.0376352218933715,
+          -0.0019251015746424632,
+          -0.07267257827375968,
+          0.036685408572352814,
+          0.0050526642351482985,
+          -0.00019289056191322955,
+          -0.013581714265389013,
+          0.05300479823846402,
+          -0.04139717386779843,
+          -0.03335524761191713,
+          -0.00920987783097114,
+          -0.022616612319255443,
+          -0.005274093640024995,
+          -0.0012667939280239298,
+          0.032502321377997974,
+          -0.020089629343721754,
+          -0.024354491736044894,
+          -0.010206744478678104,
+          -0.02465036256948816,
+          0.017966455981144307,
+          -0.012995995315432267,
+          -0.02135780149071486,
+          0.010967719068923269,
+          -0.03402154331207373,
+          -2.0626689329697705e-05,
+          -0.01092309016851862,
+          -0.018683564965672157,
+          0.01571041043301007,
+          0.016259448102797114,
+          0.015403085986247105,
+          0.0020047641027155683,
+          0.015172380396260792,
+          0.008099191720720747,
+          0.004074377809743208,
+          0.01893667379047588,
+          -0.03384452762579117,
+          -0.038822883991989264,
+          0.050315952008617104,
+          0.0008458969404873289,
+          0.0012049589464131246,
+          0.006406225290437734,
+          -0.0042817559898658255,
+          -0.0411169553122232,
+          0.012933323148113837,
+          -0.05784927886369837,
+          0.006262727192870561,
+          0.012928199517490538,
+          -0.01786898028943474,
+          -0.02291183998443847,
+          0.00046953379111417065,
+          -0.024864892714864046,
+          0.02151263275001755,
+          -0.017247234755598105,
+          0.020405349906524557,
+          -0.023068655052252782,
+          -0.006872857342813033,
+          0.05842934123173711,
+          -0.02308642746390288,
+          -0.04925938088613823,
+          0.032829595473385986,
+          -0.007184802367095134,
+          0.03878513331161335,
+          0.055971017386540985,
+          0.016337316575743225,
+          0.006439501224944926,
+          0.026084366083114555,
+          0.013204436213767278,
+          -0.012028383848550473,
+          -0.0383295105300241,
+          0.015805232298071025,
+          0.030751727443425712,
+          -0.04685643586863969,
+          -0.010258856888449854,
+          -0.02541648783174773,
+          0.01599307149086869,
+          0.020046184897703115,
+          -0.0044598427609889935,
+          -0.0012270315730225135,
+          -0.009664326045898607,
+          -0.01520444928791987,
+          -0.02631191309701645,
+          0.02193635473056903,
+          -0.013740214173212537,
+          0.0007504924364311022,
+          -0.001111971939163715,
+          0.0012797636039507028,
+          0.008478066024469051,
+          0.01501219347281274,
+          0.027440349509054086,
+          -0.022728634568050663,
+          0.006916913421268421,
+          -0.008499438828073956,
+          -0.03788213665656746,
+          -0.006718262893819488,
+          0.029198375723488434,
+          0.006169843558767785,
+          -0.016427547826608675,
+          -0.0049009468913267205,
+          0.0022509753971959087,
+          -0.016307639896946134,
+          0.00372456949880952,
+          0.010347360137213142,
+          0.035374109073612686,
+          0.01856811791368892,
+          0.019247903053478954,
+          0.0035591273657627456,
+          0.010292179556783772,
+          0.004040044337368795,
+          0.025869815038962875,
+          -0.04526297785801751,
+          0.030173313035569235,
+          0.01521363974209499,
+          0.009059160353158267,
+          -0.03914219183565228,
+          -0.02367109515190293,
+          -0.025737979702884085,
+          -0.021654077593318197,
+          0.02346511753268162,
+          -0.02730833269330862,
+          -0.006289840626554075,
+          0.006531934540756005,
+          -0.0006804032161296996,
+          0.003928945341546481,
+          -0.0033962179480786624,
+          -0.037975452991752895,
+          -0.009840510213526213,
+          -0.04023861493405766,
+          -0.0241245373830213,
+          0.003822063151525241,
+          0.003249755356115233,
+          -0.06594891648887229,
+          -0.001768385222533634,
+          -0.06222620899240395,
+          -0.013594921028316139,
+          0.03441942883093182,
+          -0.024438842229617237,
+          0.04820486752403123,
+          -0.05486229161412509,
+          0.018219866569279347,
+          -0.046562594596398506,
+          0.011354841468658838,
+          0.021738373879790916,
+          0.04347115691988908,
+          -0.007572536447300832,
+          0.014626550834376365,
+          0.009906079088662038,
+          -0.04530647508884267,
+          -0.04520033458696902,
+          -0.011400230794947595,
+          0.011239630851440557,
+          0.003287355301854498,
+          -0.012304101523783504,
+          0.02623659451177029,
+          0.015903981896266153,
+          0.04300392046214319,
+          0.03414364576071245,
+          -0.0005691335103219632,
+          -0.049506948352089984,
+          -0.027336474868358992,
+          -0.014910043127825028,
+          0.02552137904202476,
+          -0.03055258097953423,
+          0.003401887357920115,
+          -0.03228200232009397,
+          0.013338682663233296,
+          0.018207379090305786,
+          -0.02224210534533673,
+          0.032569850137027664,
+          0.014446434357859929,
+          -0.040301668913492676,
+          0.0012399866793165108,
+          0.0002450513991178417,
+          -0.038300625347349834,
+          0.008119751344998158,
+          -0.013712452991626932,
+          -0.05381257202269606,
+          -0.022199278450525977,
+          -0.014717114803771204,
+          0.02206967996899982,
+          -0.0023470596291977116,
+          0.017222681546831476,
+          -0.029965631581610256,
+          0.0053103260868707786,
+          0.039279027237044066,
+          -0.001286692479487674,
+          -0.01148541970963179,
+          -0.0008706049086180766,
+          0.022795196067681996,
+          -0.01968201411368477,
+          -0.045534614872904305,
+          0.009877856016973188,
+          -0.020566606587457816,
+          0.016002665913994348,
+          -0.015445999456355206,
+          -0.0052940252527317235,
+          0.010360748741468617,
+          -0.03170100753615497,
+          0.02004511444257144,
+          0.009722541379665509,
+          0.006411072442536516,
+          -0.017460683536795665,
+          -0.0033423349847155888,
+          0.015743901543629382,
+          0.047005401368949463,
+          0.0007329338628268773,
+          0.006153613674047577,
+          0.02501606022399451,
+          -0.013054974002495664,
+          0.05036387123988784,
+          -0.02807526639433245,
+          -0.03099731580589117,
+          0.04344605263104773,
+          -0.019489740746149623,
+          -0.017636713524640496,
+          -0.017960525325193457,
+          -0.009792405274000215,
+          -0.0062801692104839674,
+          0.02801421653619339,
+          -0.00712206060099242,
+          -0.005749470909187103,
+          0.020504942834135256,
+          0.013170883165927672,
+          -0.011218763497707175,
+          0.008523945766715354,
+          0.01511514654097483,
+          0.027931571407773977,
+          0.03214582418497515,
+          -0.02345593941074465,
+          0.0333746849861944,
+          0.009324947617983663,
+          -0.004234113006464689,
+          0.005131158074842622,
+          0.0020437263732432953,
+          0.00176803337042029,
+          -0.029372739581870692,
+          -0.01539406767336474,
+          0.015958225909710817,
+          0.007396588215584237,
+          -0.003291618431912445,
+          -0.0316834931316845,
+          -0.0028368107695447525,
+          0.02614797456533501,
+          0.003832488818435695,
+          0.026426519154286857,
+          -0.015605455255333021,
+          -0.02018445776455915,
+          0.008876565653043164,
+          0.023669947107607083,
+          -0.055401884612295824,
+          0.01882053182190406,
+          0.011046506385076857,
+          -2.6003594081234644e-05,
+          0.01591257026228522,
+          -0.0046432737555961385,
+          0.005094314300671912,
+          0.0482159962414822,
+          0.015126643838581288,
+          -0.013318945197936635,
+          0.0020418779929576307,
+          0.015136735547940301,
+          0.001727083579122356,
+          -0.020116332736417462,
+          0.002896210161586164,
+          0.012858057228188658,
+          0.0092589465483404,
+          -0.012984552574495295,
+          -0.005651788746934606,
+          -0.016856929103895944,
+          0.03249171052801259,
+          -0.005843029913406538,
+          0.02336833175997056,
+          0.04800276014595125,
+          0.02261889070736121,
+          -0.03578655846417886,
+          -0.011451513080556048,
+          0.0077334037670644135,
+          0.009415939321950421,
+          -0.012399032514032437,
+          0.006084821107593102,
+          -0.034655634244304866,
+          0.013104268092544735,
+          -0.0016781301605750378,
+          0.012249966178417542,
+          -0.025567673456449492,
+          -0.023488408901029257,
+          0.007732021927617865,
+          -0.006986715221001125,
+          0.01461433573263053,
+          0.00011845838735106802,
+          -0.005506714145045683,
+          -0.006827219009979501,
+          0.0005242424465709722,
+          0.002885486381705293,
+          -0.0042669493792680474,
+          0.021618593454103052,
+          0.00010034640948597114,
+          0.011602625527297699,
+          -0.023477647311478233,
+          -0.0162254255990209,
+          0.031352657169434826,
+          0.03529846802317018,
+          -0.0007059281329032357,
+          0.02568807107901636,
+          -0.00277252926066966,
+          0.010828396988798257,
+          0.00810821945842862,
+          -0.010307381937016405,
+          0.0003407672690264756,
+          -0.003768030765932326,
+          -0.0027367413382233317,
+          -0.012887542072043335,
+          0.0031930180018598015,
+          -0.014086667286395425,
+          -0.01504545079945656,
+          0.02010182276528891,
+          -0.002519095978589373,
+          0.011477055159845283,
+          0.006165264376334778,
+          -0.01139265727997407,
+          -0.016539552549751473,
+          0.01574258576985793,
+          0.05233886430069725,
+          0.022428487725773775,
+          0.044334901638531425,
+          -0.00706941862193002,
+          0.011585028247041065,
+          -0.016760013805097753,
+          0.001211362507398395,
+          0.048283299647824234,
+          0.015403290676022106,
+          -0.013446978976128664,
+          0.024015090204073844,
+          0.013753743307879434,
+          0.0398341544586196,
+          0.0034302789804899806,
+          -0.020152697770903245,
+          0.041614050886963766,
+          0.00392650113407049,
+          0.013465009394885544,
+          0.020487015678247938,
+          0.002430045518615804,
+          0.0035516244267459565,
+          0.02938659684630997,
+          -0.010142701862398358,
+          0.010335745659719125,
+          -0.033187389861872627,
+          -0.021792906328508686,
+          0.02532137764121119,
+          0.0005114146307176893,
+          0.033795722059780535,
+          0.007069260093915293,
+          -0.017628598898476375,
+          -0.017395163521872672,
+          0.021985468266342154,
+          -0.007987351575783343,
+          -0.025037973555114328,
+          0.002627649768366858,
+          0.008700784263932844,
+          -0.004480915144455101,
+          0.006798551384053197,
+          -0.014580806303786968,
+          -0.009741248878656092,
+          0.025066050175629597,
+          0.007330922027951371,
+          0.010472464570407973,
+          -0.024286928122882124,
+          -0.011159235410800293,
+          0.006107141408585794,
+          -0.04181412282877586,
+          0.008647396691893692,
+          0.00528510107024571,
+          -0.006647986735406744,
+          -0.002548771993161688,
+          -0.0011322362644692566,
+          0.00513401828632921,
+          -0.0035963203271502263,
+          -0.014021183815351008,
+          0.022993860073007393,
+          0.02875036422479212,
+          -0.012996574913859303,
+          0.014373647963439562,
+          -0.0093541571449835,
+          -0.019474535598737786,
+          -0.0233122546137944,
+          0.014269723193443946,
+          0.008784376430480408,
+          0.004317715767550436,
+          -0.008273153355003107,
+          0.0010203136295909875,
+          0.001909823045834306,
+          0.02406426043795662,
+          0.049377235716785324,
+          0.005265955549193235,
+          0.029166436381966566,
+          0.017136545810044906,
+          0.03882940870384413,
+          0.0007533463090350569,
+          0.016233986042907378,
+          -0.03311355052072762,
+          0.03082356484543373,
+          -0.0009431717656252606,
+          0.04077734731808,
+          0.0369941736537998,
+          -0.05477277258937169,
+          -0.014658721668336077,
+          0.028327060073149418,
+          0.010896122046930724,
+          0.01436699901738847,
+          -0.003307011398652852,
+          -0.01163199508062231,
+          0.029659278016495517,
+          -0.023936970669211955,
+          -0.010748756889752312,
+          -0.00767811402724157,
+          -0.010785978668072403,
+          0.009408061520238972,
+          -0.0038320809092061126,
+          0.038383104318077226,
+          0.0070413219383093,
+          -0.009684310528333753,
+          0.019154521368080882,
+          -0.006292471963504016,
+          0.01596901434992771,
+          0.0037167338451907566,
+          0.0019879266696342887,
+          0.03104390318244226,
+          -0.003561483519923973,
+          0.028949607591091022,
+          0.015156617705082505,
+          0.017473301120296506,
+          -0.0029321911929339165,
+          -0.017573294769065502,
+          0.027284173131869906,
+          0.009113069584708108,
+          -0.01971507323375489,
+          0.003667023499937692,
+          -0.0025339283534597176,
+          -0.0077697455666997945,
+          -0.010491575421861018,
+          -9.407001004405342e-05,
+          -0.00211520268689493,
+          0.02087980536494742,
+          -0.005582301812064227,
+          -0.0016376727058019977,
+          -0.0064225805186745605,
+          0.0052835419390265485,
+          -0.019662445004865352,
+          0.015117937189193789,
+          0.007575238218238001,
+          -0.018537762525347407,
+          0.014673223139419801,
+          0.019165766004277593,
+          0.0343151018851732,
+          -0.023949650834192267,
+          -0.0009279394158895472,
+          0.01730410035639405,
+          0.005928527001253011,
+          -0.018975612465080345,
+          -0.033873819454780706,
+          0.00825263613547798,
+          -0.01745801647450136,
+          0.027401724356781083,
+          0.008830481774058282,
+          0.0347823637819511,
+          -0.02309575756624861,
+          0.020559439488958053,
+          -1.2525886083543457e-05,
+          0.01593239963830551,
+          0.01455178268663934,
+          0.027993448238508672,
+          -0.017573784980205873,
+          -0.0379374065982424,
+          -0.0030450459335320053,
+          -0.0029968228865394214,
+          0.016851016821808874,
+          -0.013002468483615099,
+          -0.003337198735398279,
+          -0.0019044899095842511,
+          -0.029686441224288696,
+          -0.021487593939979652,
+          0.027796807698461906,
+          -0.04637191752173605,
+          0.0030176823995715597,
+          0.03281049040516897,
+          -0.030047693785808505,
+          -0.04246000386774685,
+          -0.0014603539776430917,
+          -0.03560639883119047,
+          0.0010602583020630973,
+          -0.01222013383414755,
+          0.029128085483749913,
+          0.007388529661657186,
+          -0.020518571527741426,
+          0.018184513437823,
+          0.0076594003031722,
+          -0.004161472492195519,
+          0.006100954284405263,
+          -0.005239699097592758,
+          -0.028136388241670696,
+          -0.015254134018585082,
+          -0.03037392857155718,
+          -1.6317930920712043e-06,
+          -0.002478230556191563,
+          -0.012845180214134591,
+          -0.024383682688014996,
+          -0.000484419407302015,
+          -0.014810859291916117,
+          3.958274758340876e-06,
+          0.05136363130801806,
+          0.02120810926469503,
+          0.005060247818396135,
+          0.012869784532979883,
+          -0.019665446839172217,
+          0.04743686110578455,
+          0.003514534022065111,
+          0.03583012040901262,
+          0.002062991366066978,
+          -0.02173965243546218,
+          -0.0005002960506349255,
+          0.0319890423992129,
+          -0.0016894669649440992,
+          -0.017249094049398075,
+          0.006136324758346181,
+          -0.009220681391996718,
+          0.00837047468537671,
+          0.012871598576797918,
+          0.01579543926540845,
+          -0.0061999233512262265,
+          0.014725777187858035,
+          -0.003927118678273384,
+          0.009336164668870259,
+          0.03122006739418818,
+          0.019771411252821138,
+          -0.024716950288097537,
+          -0.037736399088695305,
+          -0.012523788001500705,
+          0.001234987976751515,
+          0.034477269953960235,
+          -0.013836458205716058,
+          0.003837278407556882,
+          -0.011027628278972294,
+          -0.0054689562005572675,
+          0.005739209890610896,
+          -0.006985434270987544,
+          0.0015806421600885957,
+          0.012196603223058041,
+          -0.006859901421089723,
+          -0.025670287745499374,
+          -0.004011122451643543,
+          -0.023258896941014143,
+          0.012997299613183496,
+          -0.010144866692641839,
+          -0.006816231069049567,
+          -0.005213359047878286,
+          -0.0014167278420750282,
+          -0.0062535968890290545,
+          0.0006662897611595949,
+          -0.014236807237438571,
+          0.01012846684096945,
+          -0.00899852930003816,
+          0.0016620397335915663,
+          0.007838787681725669,
+          0.007725632848979133,
+          0.0033110678307931205,
+          -0.008206916937522372,
+          -0.008777095103563134,
+          -0.01631820698659747,
+          -0.032412484976539334,
+          -0.03000207979123948,
+          -0.02276516767125653,
+          -0.003870969029785218,
+          0.0034978865240734053,
+          -0.032594545526009384,
+          -0.020635972136797354,
+          -0.002367523739258502,
+          0.0030103817989158062,
+          -0.014587341442812854,
+          0.0014779462725289245,
+          0.007969995140846214,
+          -0.0029319878689507335,
+          0.019355924479361485,
+          -0.005448177691636249,
+          -0.021203447333791942,
+          -0.028956667864566195,
+          0.017851274292380093,
+          -0.015891501231440428,
+          -0.010070179095955473,
+          0.0064388768290910715,
+          -0.005633920123057214,
+          -0.03957042012546069,
+          -0.008945181709739484,
+          0.010992989857329521,
+          0.0035378401849368002,
+          -0.025936727040303652,
+          0.031903561853958325,
+          0.030132441728296204,
+          0.01845519141895959,
+          -0.017902241158092712,
+          0.04710441323879975,
+          -0.006739916600200837,
+          -0.012103058303430145,
+          0.016276540022040643,
+          0.023963592487909288,
+          0.028774476787161263,
+          -0.02643829153200483,
+          -0.005349178006882811,
+          -0.010545703050010008,
+          0.005354643441490569,
+          -0.02271588009302058,
+          0.008468528451368553,
+          -0.013601178278798316,
+          0.011576067943282815,
+          0.019227032134009135,
+          -0.013197207810056759,
+          0.02629307658052319,
+          0.0045263515787155985,
+          -6.067227645639678e-05,
+          -0.0037770406201685248,
+          -0.016287414859552752,
+          -0.012319957965328086,
+          -0.034674880061192896,
+          0.020297690806102293,
+          0.01889209267823917,
+          0.003953684662429981,
+          0.021845046751532583,
+          -0.049434011185276405,
+          0.02806744160678342,
+          0.0017849117483741536,
+          0.021763434864703356,
+          0.013889518910180847,
+          -0.001843068102127134,
+          0.013886691089114936,
+          -0.0032117229736863266,
+          -0.019971320584891997,
+          0.006079730961157539,
+          -0.006352535944493006,
+          -0.03316544192496317,
+          0.014736816574462923,
+          -0.03268321622529288,
+          0.013639134163846678,
+          0.015454170145821088,
+          -0.010170396823729307,
+          -0.021708948733883182,
+          -0.008303375382721087,
+          0.0010570623547459114,
+          -0.0014149688171089135,
+          -0.006849845444850982,
+          -0.04346483266888068,
+          -0.03020319096411844,
+          0.03124837430439466,
+          -0.0016720020681567616,
+          -0.03404132565569305,
+          -0.007209824644083462,
+          -0.02711263625315296,
+          0.004780610743996515,
+          -0.004212647822173262,
+          0.0025676935176963222,
+          -0.048750272936498695,
+          -0.01018434342955251,
+          0.011179560412168381,
+          0.031648611061279724,
+          -0.029298960703078986,
+          0.021370266671953315,
+          3.770954396468699e-05,
+          0.006629905234129844,
+          -0.00943632381701963,
+          -0.00484754585233744,
+          -0.012115337995959813,
+          -0.02607251387584821,
+          0.021868809148577698,
+          -0.0020587964818437345,
+          -0.0013502026980372474,
+          0.0341522429571705,
+          -0.00669289309893371,
+          -0.010113192232031375,
+          0.0005934017402521538,
+          -0.011327797122243258,
+          -0.017269539623150844,
+          -0.010467173785198161,
+          0.0005779464354975677,
+          0.0030406502310922306,
+          -0.024755136158113783,
+          0.004943446717510779,
+          0.04566273647510376,
+          0.013607936503018211,
+          0.02534200532143803,
+          0.017104750405207106,
+          -0.03792550084708064,
+          -0.04394052120951292,
+          0.0012600504352102988,
+          -0.005807815271334456,
+          -0.01944685298932453,
+          -0.01732108197375621,
+          0.0011677414026425442,
+          0.00790730598557849,
+          0.0037240231183698074,
+          -0.011427532982681041,
+          -0.02086964367350831,
+          -0.032539146090370744,
+          -0.0136797717535217,
+          -0.009464326388205681,
+          -0.007302204556505735,
+          -0.032703238152349344,
+          -0.015929357859286307,
+          0.026848214129378756,
+          0.01012098852614113,
+          -0.005646819862636667,
+          -0.0107564788588665,
+          -0.024118155140225247,
+          0.007122339033521904,
+          -0.0027183981945039814,
+          -0.009219030314883272,
+          -0.004672057262936543,
+          -0.016450690196817726,
+          0.02348093022809639,
+          0.009139210149358477,
+          0.024997129282118148,
+          -0.058533911093319214,
+          -0.014411932474734742,
+          0.014256038602455438,
+          0.0020188651815054508,
+          0.001417136751974749,
+          0.029836680491427268,
+          0.0028786535004858044,
+          -0.006526872714072965,
+          -0.02967220833325222,
+          0.028817295504169067,
+          0.027449694157944637,
+          0.01800200378294509,
+          -0.010560094796033294,
+          0.0033996190760546383,
+          0.006004691351249025,
+          -0.005679195734863024,
+          -0.011927786207717698,
+          0.0021260244673648285,
+          0.018998574211603905,
+          0.017174632765532803,
+          0.014659925822127794,
+          0.02455156987384756,
+          -0.005882590537857664,
+          -0.03911277764764406,
+          -0.009781110378157883,
+          -0.0119630929326396,
+          -0.004815125964940769,
+          -0.0026631770014339242,
+          0.008903827530107938,
+          -0.011824767329332145,
+          0.013088406716464123,
+          0.030264448923574067,
+          0.042592268021664714,
+          0.021292401951863188,
+          0.006150997237661769,
+          -0.005463801502425317,
+          0.0007399518998122848,
+          0.0038649546282459707,
+          0.011355675550395135,
+          0.035762872771750435,
+          -0.018147218572765546,
+          -0.019704502923991365,
+          0.0015558311499702044,
+          0.023543487977413267,
+          -0.007150197116420024,
+          0.02608209797565159,
+          0.023985156585794257,
+          -0.0003434024317322991,
+          -0.01271338268931938,
+          0.000985282363050309,
+          0.006710364567868783,
+          0.014540943712513688,
+          -0.02154205198718209,
+          0.004039619025717627,
+          0.001865104196024055,
+          -0.009756600776161738,
+          0.029738871754072076,
+          0.00039154552800346406,
+          -0.012981811693538832,
+          0.03825344802886,
+          0.11025963334575106,
+          0.001476339401950677,
+          0.0031187927813794386,
+          -0.031522639386920355,
+          -0.015113380587160163,
+          0.0021786482865071505,
+          -0.005948756393737655,
+          0.020082591515778573,
+          0.012751062800862873,
+          0.0055847042526916265,
+          0.010100961720464159,
+          0.038084609873482715,
+          -0.009203643420546616,
+          0.009804311800033327,
+          -0.01878272400403416,
+          0.017185477997947226,
+          -0.00972716681599207,
+          -0.00857276597428136,
+          -0.007641511960016827,
+          0.004167416313632853,
+          0.00984005584819001,
+          -0.004596999098606572,
+          0.018810908413859465,
+          0.002496310438422421,
+          -0.01613133688770545,
+          0.011505030137914962,
+          0.007437924384075958,
+          -0.005386885126779478,
+          -0.031755982499042136,
+          -0.013811586180238287,
+          0.01190667400285204,
+          -0.0013786018401773159,
+          0.0007177632889203917,
+          0.012342640512536284,
+          0.01205074763420124,
+          0.031217897495485192,
+          -0.0244324065837541,
+          0.0025071378510626696,
+          0.0070486374045588465,
+          0.0159322020159265,
+          0.01098799814774876,
+          0.00029844505852147526,
+          -0.015456787478044924,
+          0.027139764983358832,
+          -0.011978684728746428,
+          0.010378658252224036,
+          0.01146607671795495,
+          -0.04131658683485702,
+          -0.0009967932006190922,
+          -0.005916133731248287,
+          -0.015698754902727707,
+          0.013905118384587169,
+          -0.040163447416240027,
+          -0.005682084326086952,
+          0.004297114532998809,
+          0.010724201884417054,
+          0.0058749102353125955,
+          -0.017715270733111323,
+          0.009962853888582856,
+          0.00955216765250444,
+          0.0025392532243413593,
+          -0.006356691980285544,
+          -0.006218169970979339,
+          -0.008606942887730427,
+          -0.004280171880086814,
+          -0.015669721147884084,
+          0.0057481312044666884,
+          0.0077227094511494055,
+          -0.018751123856436715,
+          -0.01585864749751703,
+          0.009183099781375234,
+          0.006837438250956817,
+          -0.001445622524955126,
+          0.004828557942941401,
+          -0.010034012200008746,
+          -0.004543834537450213,
+          0.01097540631691429,
+          -0.009334158551770033,
+          -0.004804430123454511,
+          -0.02609985520241612,
+          0.013342299390862539,
+          0.011103136832190223,
+          -0.008261332966534102,
+          -0.006944131083745896,
+          -0.016931502665633764,
+          -0.0031318730206304824,
+          -0.0026508016484082988,
+          0.0016110884325639078,
+          0.017071266355203386,
+          -0.009906320666640836,
+          0.010877416311741906,
+          -0.03496707406301918,
+          -0.015357787514985604,
+          0.01622830705930841,
+          -0.01290003882907745,
+          -0.005625420341281716,
+          -0.009838333350984609,
+          0.0006911383780218238,
+          -0.007545081680638308,
+          0.006074752194220675,
+          -0.041320607445282646,
+          0.0035102345636816475,
+          0.018825308640738208,
+          -0.009723969339534365,
+          -0.020829421583878845,
+          -0.009055117268112308,
+          0.010487235049013567,
+          -0.015831829161845813,
+          -0.01780922120240258,
+          0.0016219998217122911,
+          0.010559783293374071,
+          -0.0055135690971065965,
+          -0.005160232484458576,
+          0.006163424486940105,
+          0.020247627989655837,
+          0.015220964429515309,
+          0.04337432385613251,
+          -0.011915355138717245,
+          0.009617347907919507,
+          0.027812934121040484,
+          -0.030652481946156104,
+          0.01037470561447702,
+          0.02599512377987521,
+          0.0037340561802882417,
+          -0.0013375196064018082,
+          -0.010197691376747194,
+          0.004161692215514214,
+          -0.012254402844820636,
+          0.008715552098450461,
+          0.004697361473725416,
+          -0.027978648968732336,
+          0.017196771853396124,
+          0.019448218659009996,
+          0.0038720194629308794,
+          0.0006995062900842749,
+          -0.01527901684789497,
+          -0.011825577013200781,
+          0.019194236643713274,
+          0.027690065919038755,
+          0.005460028121722145,
+          -0.015512497062157167,
+          0.02370643287056251,
+          0.012916257803027072,
+          -0.004973882005506947,
+          0.0022332716531575122,
+          -0.016984605526823043,
+          -0.01443316577897661,
+          0.030942241604983316,
+          0.011563751997976407,
+          0.01908030186367601,
+          -0.021614212853976106,
+          -0.003982310717524619,
+          0.019931579069642925,
+          -0.021029366993395197,
+          0.013006879711144009,
+          -0.019497570222553013,
+          -0.014067925104548129,
+          -0.010687576618198498,
+          0.02884494104893037,
+          0.009229770955288097,
+          0.026701705044769222,
+          0.012715371448003833,
+          -0.021444875656527786,
+          -0.021050591467198086,
+          -0.0027703267406452336,
+          -0.02721606030674425,
+          0.01688256721881862,
+          -0.055990128879151335,
+          0.0016563858333845547,
+          0.010911810464298247,
+          -0.008604898990983188,
+          0.004039521277648332,
+          -0.007858378479197747,
+          0.010021929060776957,
+          0.020926274809879138,
+          -0.02956229623532218,
+          0.01256060127924585,
+          0.01156331569664224,
+          0.0218688589850221,
+          -0.018861065515394883,
+          0.015339087155577661,
+          -0.00841001512858443,
+          -0.027396412391138852,
+          0.008323898095545889,
+          0.01763697170382491,
+          0.014160184414353919,
+          -0.011992345081843988,
+          0.004485633614765869,
+          -0.04000932898876895,
+          0.027999519977069676,
+          -0.016823500188831896,
+          0.002010511519234648,
+          -0.004537991252901591,
+          -0.0067219583064939085,
+          -0.008523354091034399,
+          2.682477256914193e-05,
+          -0.0005371719103087346,
+          -0.030920653652724776,
+          0.0012939076000836422,
+          -0.04144943703666512,
+          -0.02260198516292792,
+          0.013327519374598822,
+          -0.0012897465780583367,
+          -0.005059170593504863,
+          0.02999548027478466,
+          -0.02863650640718448,
+          -0.027272295036911975,
+          -0.014376307736824656,
+          0.03534844385341626,
+          -0.015358480569131909,
+          -0.000540068394391221,
+          -0.009287193719601908,
+          -0.02236594406404711,
+          0.01091078016405303,
+          -0.006469935602381157,
+          -0.0019272207412132534,
+          -0.01387843655423569,
+          -0.03972775413987396,
+          -0.005948332199741248,
+          0.019500395452013777,
+          0.0006122786247500248,
+          0.003656719755271774,
+          0.014473976840529128,
+          -0.0072278373928449675,
+          -0.0036623750417377524,
+          -0.0038809301345520044,
+          0.026929584793529297,
+          -0.022225170775945557,
+          0.009744440210592443,
+          0.004704883568603746,
+          -0.0005991305039095727,
+          0.017317686588588826,
+          0.000351879037187266,
+          0.0024030356296394478,
+          0.0029060795825935717,
+          -0.0035301907818594274,
+          0.02922907976800865,
+          0.0064070048516525,
+          0.0016291092834589405,
+          0.0001988403049139903,
+          0.004746570638478093,
+          0.01395197992587789,
+          -0.01069245020264236,
+          0.0157940052093005,
+          0.02865126209030693,
+          0.00021287181382353683,
+          0.018345429962063935,
+          -0.015921361819192573,
+          -0.036059435316760584,
+          0.019807131219096722,
+          -0.029642240399677812,
+          -0.003504127677138975,
+          0.022473005924273325,
+          -0.012903852741467797,
+          -0.0031750649075827082,
+          -0.01107957248024579,
+          0.027243132530764567,
+          0.023512079029549708,
+          -0.014856101416774535,
+          -0.015142539116300868,
+          0.004337433267586975,
+          -0.010210607143984678,
+          0.010939153927790107,
+          0.029382995339774957,
+          -0.011230337698862454,
+          0.011717621080350227,
+          0.001161341258722717,
+          -0.003778338045364591,
+          0.019300005069271462,
+          0.0027681684372537606,
+          -0.007401677666787755,
+          0.025628069381915978,
+          -0.008115857591022265,
+          0.013043669211084909,
+          -0.004672358463640539,
+          -0.012902494254220482,
+          -0.003106794914564588,
+          -0.017558145644151636,
+          -0.01225024815282763,
+          0.0011522376202485873,
+          -0.005528597019035285,
+          0.06711658458916447,
+          0.010930891527455397,
+          -0.017888423424597618,
+          -0.0059190199704595775,
+          0.00955567741445378,
+          -0.000966061435587513,
+          -0.009466328431452424,
+          0.027063767833815658,
+          -0.016888493246695876,
+          -0.009681104485277698,
+          -3.814655691584766e-05,
+          -0.0233149747136092,
+          0.0030596902889784496,
+          0.010827987186229144,
+          0.0012052973350181351,
+          -0.013994504803884577,
+          0.005318095523120499,
+          -0.004843763992616558,
+          0.0034188401938723923,
+          0.003678626789714963,
+          -0.0025749484272345573,
+          -0.005333294277998365,
+          0.02307826358450567,
+          -0.012791914224263883,
+          -0.008519982879775168,
+          0.01173424692936401,
+          0.001777123104204978,
+          0.003219553788359312,
+          -0.004564745566711077,
+          -0.002015107048152335,
+          -0.017123545909339455,
+          -0.011278616226036531,
+          0.023713473402287875,
+          -0.02733523566514418,
+          -0.00150943777243135,
+          0.017328382962199716,
+          -0.005705544203971564,
+          0.016650187392502975,
+          9.750950361601131e-05,
+          -0.02100066506382883,
+          0.00010090567939930427,
+          -0.024778741347883073,
+          0.010484659495766551,
+          0.013963183459158656,
+          -0.009531952698151101,
+          -0.020878167882780043,
+          -0.027881362421494915,
+          -0.010190031350698889,
+          -0.027246821338460653,
+          -0.0280692647056191,
+          -0.02020582179742232,
+          0.009778340348478517,
+          0.005715364307462615,
+          0.021929057840860695,
+          -0.006380523858228711,
+          -0.027552858236741976,
+          0.015095163660340326,
+          -0.013269344817111162,
+          0.008282918921343354,
+          -0.019991891695114384,
+          0.01754833974582726,
+          -0.014720830704852575,
+          0.02112161088830394,
+          -0.00041409266970033855,
+          -0.004490472334793758,
+          -0.012076561942010787,
+          0.003976913901514849,
+          0.008101357042612863,
+          0.00176590819486174,
+          -0.008977295353115363,
+          0.026242190932756135,
+          0.01707634775369835,
+          -0.007385093173634561,
+          0.01604386075569055,
+          0.031855080733360075,
+          0.014762206220119051,
+          0.038860660002328094,
+          -0.007238177370595944,
+          -0.009237098188374397,
+          -0.02764495845127417,
+          -0.004407870047633689,
+          -0.006820430763682645,
+          -0.004516837004332235,
+          -0.015251921783911087,
+          0.03164122113689215,
+          -0.010371400055822811,
+          -0.01686972159902524,
+          0.03979035122593764,
+          -0.021846728842184655,
+          -0.007084905708666271,
+          0.008642771511425465,
+          -0.013996636040226265,
+          0.03176661229383477,
+          0.007275612529363987,
+          0.018452818352301472,
+          -0.015039890816023917,
+          -0.015988701055097997,
+          -0.0019911190487748244,
+          -0.002806138139429132,
+          -0.020828646560807865,
+          -0.0063794261154720665,
+          -0.01614570050728432,
+          -0.0026319562705484634,
+          -0.00570753504957681,
+          0.015917008930969468,
+          0.0013916917451795444,
+          -0.015420032290910069,
+          0.04488171544231248,
+          0.005033499525006087,
+          0.022768662319611596,
+          0.010777933460735837,
+          0.012372342667053587,
+          0.015819823672091563,
+          0.006759824420801962,
+          -0.0001668726099910387,
+          -0.02211654164646909,
+          0.004261169256891085,
+          -0.005641717035723983,
+          0.0041005490092982915,
+          0.02012094437568505,
+          -0.024436624754739972,
+          -0.029708548399946657,
+          0.024493078535528394,
+          -0.01833101179919427,
+          0.0030908990956393764,
+          -0.0031025537219982234,
+          -0.010059653447038065,
+          -0.011612233298907742,
+          -0.03293899346571458,
+          -0.0005929867206193162,
+          -0.027438115397399914,
+          -0.03635734152790657,
+          -0.006287687204976032,
+          -0.007850474352061472,
+          -0.003574235211373211,
+          0.02449669755605501,
+          0.004269460596739739,
+          -0.029959856511873897,
+          0.01782698044631415,
+          -0.013252733005984911,
+          -0.013342147906120882,
+          0.0029581361690999564,
+          0.004786570391076097,
+          -0.0027618259744494854,
+          0.010720003765859173,
+          0.007898965393358715,
+          0.007358694001293221,
+          -0.010710502803266644,
+          -0.004049671482838962,
+          0.0013263710826446039,
+          0.0030943253037901135,
+          -0.013325799375903166,
+          0.030553973934607993,
+          -0.018812559793353154,
+          -0.004819166852190956,
+          -0.0006499911473215254,
+          0.0034556893023524674,
+          -0.0110929170801761,
+          -0.014862886414702483,
+          0.021004665462647988,
+          0.0031088550726355926,
+          -0.012570481398485388,
+          0.0096952264731202,
+          -0.0018868240406799254,
+          -0.0024767709620225923,
+          0.01176108203579327,
+          -0.002080730326597438,
+          0.014924914157541781,
+          -0.011040818699356567,
+          0.01490073610670544,
+          0.0034649445207609085,
+          0.008645740731109337,
+          0.003603641937848282,
+          -0.01905231764119889,
+          0.009780124784430865,
+          0.016356631955067272,
+          0.013415470760506568,
+          -0.029496530287207723,
+          0.008881639987471632,
+          -0.005975399013716009,
+          -0.009870580036511721,
+          -0.024372445588075428,
+          0.002590423942920311,
+          0.028960961301291905,
+          0.012044179251559706,
+          0.007934516374398097,
+          -0.0077949340827692054,
+          -0.005480592942515702,
+          -0.019923500717513334,
+          0.009041031662948522,
+          -0.01095651774810384,
+          -0.025157908389819465,
+          -0.007578712371715301,
+          0.03430545904801765,
+          -0.010740874257824011,
+          0.01343875386664532,
+          -0.005526757815283282,
+          0.02436552949831661,
+          -0.0027538610388853284,
+          0.0013690344627799137,
+          0.008416180197734094,
+          -0.006091433734127276,
+          -0.001708607133233843,
+          0.013349212286052506,
+          0.041066250931277824,
+          0.0038375614886832564,
+          -0.0050950709988837216,
+          0.01149207527692021,
+          -0.00326389523728132,
+          -0.005712157453687712,
+          -0.00309579814793649,
+          -0.025491974096530076,
+          0.015133356727314608,
+          0.0010745378801251735,
+          -0.017162228172591873,
+          0.021785292616181687,
+          0.027891598422935242,
+          0.013262622112280296,
+          -0.0178471114368855,
+          0.014456217536546996,
+          -0.009278392107874806,
+          0.01629461595238019,
+          0.02423194076020779,
+          0.010067296472207578,
+          -0.011865648674106681,
+          -0.0070073298858753805,
+          0.008985354045706703,
+          0.008096016282907443,
+          0.0004236756235618935,
+          0.020370823466310733,
+          -0.028334250141960886,
+          0.01483270667238479,
+          0.007137078270349882,
+          0.03314918453195744,
+          -0.0006231629401653488,
+          -0.014397909717816884,
+          -0.008918359605259833,
+          -0.002915473990900902,
+          0.011759189078587877,
+          0.003523992368256188,
+          -0.0006853609664806012,
+          -0.004749148984168426,
+          -0.014181883848149463,
+          -0.006778789302277201,
+          0.012376777107762334,
+          -0.0031646329407784245,
+          0.0030514749941358095,
+          -0.0007326603315688041,
+          -0.007811941797535734,
+          0.015661953163847418,
+          0.001669500869963878,
+          -0.0007143628544145883,
+          0.0016729300148826723,
+          6.604560293656475e-06,
+          -0.00467413979302959,
+          -0.019114577369302277,
+          0.008922356418978664,
+          0.00867683974577237,
+          -0.006578536023513047,
+          -0.006467824108991175,
+          0.009324373505311057,
+          -0.01432602608910974,
+          0.01852478469282692,
+          -0.0061254590933158826,
+          -0.006603842196861297,
+          0.00014277641689784565,
+          -0.012328738339684425,
+          -0.003642202698645936,
+          0.0034895681343535914,
+          0.0051453312116433265,
+          -0.000908593523739809,
+          0.0036062764245630334,
+          0.003940522150829955,
+          0.01083964694261132,
+          0.002824079422783228,
+          0.0015266576108775834,
+          0.007280263953807289,
+          -0.01039293741283943,
+          -0.007403606843286745,
+          -0.004993622657038095,
+          -0.006341662331986892,
+          0.007288958567606623,
+          -0.004364975055335228,
+          0.010457360052582658,
+          -0.0062292104145735515,
+          0.020471836369529147,
+          -0.006133009423930811,
+          -0.0024055488697334888,
+          -0.017695183455939345,
+          -0.007234478249839231,
+          0.007156119488831356,
+          0.007878044165936736,
+          0.009359297231370758,
+          -0.04487830669749157,
+          0.005038301058510894,
+          -0.0054192943462008595,
+          0.02941616633783806,
+          -0.009828941844770915,
+          -0.004335130343692854,
+          0.016067824555917796,
+          -0.019100842323053376,
+          0.0037066021889614513,
+          -0.013965982913833292,
+          0.003695036285467973,
+          0.0034558765116311525,
+          0.014205693020177169,
+          0.031307058208919394,
+          -0.007482180869231769,
+          -0.009383819319265733,
+          -0.006384083847148052,
+          0.026967380724567013,
+          0.018643952476557566,
+          0.023548246269020395,
+          -0.011885146764471029,
+          0.00960518654541979,
+          0.001357650512536872,
+          0.018271473493526338,
+          -0.009663920325675169,
+          -0.011163336807225407,
+          -0.000516828412255004,
+          -0.010288703356943627,
+          -0.0032111641654394615,
+          -0.021092460896394025,
+          -0.0033456309643941965,
+          0.025876107752976232,
+          0.0038584058642743987,
+          -0.0036689325941524813,
+          -0.015624044740149702,
+          2.2644755460272697e-05,
+          -0.011931276845804481,
+          -0.0055870010742218364,
+          -0.015697330226950804,
+          -0.028927689083316704,
+          -0.016192017705678192,
+          -0.002914436546908901,
+          -0.017224654082473787,
+          -0.009460990414684924,
+          0.002071634636546831,
+          -0.015533259213802986,
+          0.004918377471991415,
+          -0.015669642231243106,
+          -0.010915412245035698,
+          -0.00046472075835247,
+          0.004672390433470177,
+          -0.004682173537415046,
+          -0.024109027987987815,
+          -0.013628794248618615,
+          -0.0022054473551170926,
+          0.006500170428351522,
+          0.010016298501187265,
+          0.008324628078950177,
+          -0.0069374343736907625,
+          0.011201167788973098,
+          -0.009746811993090597,
+          0.010155637867574787,
+          -0.02773759960959338,
+          -0.04317847702979315,
+          -0.000791583637601122,
+          0.009241551203900427,
+          0.001979688190287368,
+          -0.01884488538245213,
+          0.0038489955050695704,
+          -0.011280060242688173,
+          -0.00847814532468811,
+          -0.007522173019881079,
+          0.01573091287228822,
+          0.004138348969142638,
+          0.013463997539337106,
+          0.0042065270170965705,
+          0.0285467221116185,
+          0.012635383326761538,
+          0.01030078781909351,
+          -0.003092067773698843,
+          0.004763648809248413,
+          -0.0022835042548997662,
+          0.009077632103272405,
+          0.02331475506951857,
+          0.0076225652102231605,
+          -0.002326207128504976,
+          0.00409858249509217,
+          -0.0005698076375460636,
+          -0.005442088641415082,
+          0.018565262151290427,
+          0.02915691852605287,
+          -0.0069811288298562155,
+          0.008120076286611811,
+          -0.0009110258445641983,
+          -0.009900329477406562,
+          0.0038288023984988134,
+          0.005765264445222657,
+          -0.014230725839572923,
+          -0.0024175374911696653,
+          0.009367987848760041,
+          -0.024664375361024386,
+          -0.004113757123212052,
+          0.020371264792326226,
+          0.0024813232005061487,
+          -0.016599537494340785,
+          0.03488073638324099,
+          -0.003709931703179228,
+          0.003010315958426392,
+          -0.00944047118505417,
+          0.016362178907316286,
+          -0.017989464717541103,
+          0.0005390907615384354,
+          0.013131172782591922,
+          -0.005251614069791733,
+          0.005783823857695169,
+          0.003744510952053123,
+          -0.010417674399178015,
+          -0.010824994224858631,
+          -0.0033951133363820767,
+          0.01085129781083312,
+          0.002642227227069058,
+          -0.024446190958178605,
+          -0.017731767444262785,
+          0.019381370848622046,
+          -0.01001206050737818,
+          -0.008290790009659911,
+          0.0009331678418736739,
+          0.0029270629970877303,
+          -0.004637651319894003,
+          -0.004290917518172777,
+          -0.003574121539057569,
+          0.009121344354139133,
+          -0.009485343005168732,
+          0.01637203851901565,
+          0.004043708359084711,
+          0.01843096334879628,
+          -0.011588935298420727,
+          0.005808551475129831,
+          -0.001368952044937682,
+          -0.007242291047654279,
+          -0.019004856431577073,
+          0.0059537808323522675,
+          0.0073777483129045685,
+          0.016051495408653346,
+          0.011060627996179347,
+          0.018569088420830965,
+          0.011345955812555888,
+          0.0039209921807600205,
+          0.001180658155391396,
+          -0.011006781869601802,
+          -0.016371246428912994,
+          -0.014232536816028922,
+          -0.007416908142074357,
+          -0.019561375671062513,
+          -0.0013253236527051844,
+          0.017798265625542643,
+          0.014528038148955117,
+          0.017704367192118375,
+          -0.0048389172759564884,
+          0.016018290057587425,
+          -2.7789810195444745e-05,
+          0.0009215076153434622,
+          -0.005897137279173464,
+          0.001794778099552753,
+          0.016314258230521846,
+          -0.003313666704140227,
+          0.0063152813149371136,
+          0.0002935873451081568,
+          0.01645203621687452,
+          0.022650990579075773,
+          -0.018899997585540014,
+          -0.0014442758993479705,
+          -0.006467599303099793,
+          -0.0021225224281092863,
+          -0.019894752041664215,
+          0.020266472457121338,
+          0.014019662562577204,
+          -0.005002061958298289,
+          -0.00943116174529331,
+          -0.004354703314807638,
+          0.02678118874395904,
+          0.020206530983694557,
+          0.006044260997975214,
+          0.026222728100119382,
+          0.01976601108150248,
+          0.01984865298844998,
+          0.011918890024498122,
+          0.009052182076567042,
+          -0.013092243192366595,
+          0.018358346459726883,
+          -0.01681770553983246,
+          0.0008504895530001561,
+          -0.003223133403363917,
+          0.002743153478851782,
+          -0.007937179998086906,
+          0.013021430914653432,
+          -0.00030955362267617273,
+          -0.025935204965869492,
+          0.007680669831594167,
+          -0.013183173230049736,
+          0.001375652741144253,
+          6.07690288088255e-06,
+          -0.001857629339163609,
+          -0.017652365850892467,
+          -0.0082709435899296,
+          0.0008448530375262078,
+          -0.01435595518048752,
+          0.023866770609514233,
+          -0.026826645910954506,
+          -0.02140002089501948,
+          0.00988141127158139,
+          -0.004436844338114915,
+          -0.012632978351515892,
+          -0.006972217855086974,
+          0.021864041851582767,
+          -0.005139891275508354,
+          -0.016596866242234234,
+          0.014168148679282804,
+          0.0005869000756971533,
+          0.00412958763467087,
+          0.0012364712526909765,
+          -0.008001798306987228,
+          -0.0030416500521329983,
+          0.006860667487819997,
+          0.028693213258586023,
+          -0.011895509688811421,
+          0.0016632663352826491,
+          -0.013799461077520309,
+          -0.0037922015693749754,
+          -0.02974150200357857,
+          0.027752493864472517,
+          -0.011830981692069491,
+          0.01989293240244598,
+          -0.004540523332199857,
+          -0.007667738626842118,
+          -0.021173122202316107,
+          0.035703796333655495,
+          0.004594443089494466,
+          -0.0006664409717898689,
+          0.007256660335524136,
+          -0.047588498838074554,
+          0.01105469709139337,
+          0.006498789757276137,
+          -0.023261890747840144,
+          -0.000964111432535798,
+          -0.003995007813335293,
+          0.02143067593410374,
+          -0.019461700447829744,
+          -0.012777737727876326,
+          0.0021024453304811634,
+          -0.021925886491102078,
+          -0.007056889535073896,
+          -0.0340604475320119,
+          -0.006137802768771592,
+          0.006503135515471076,
+          0.0038385769709340417,
+          0.0020799029890387023,
+          -0.021308499975514292,
+          -0.0309544649831362,
+          0.013308972238482328,
+          0.01651668347077017,
+          0.010007831710070633,
+          0.01200701454477496,
+          0.0067050757442456005,
+          0.011425518484475204,
+          0.00865023114755134,
+          0.00974582806570851,
+          0.008884712478547614,
+          0.039109252942848706,
+          -0.016122498485808056,
+          0.008671479031657285,
+          0.0005699279397862279,
+          0.02156543966774511,
+          -0.02391743870774471,
+          -0.009877938786506814,
+          0.015475531666380014,
+          0.003045729682805746,
+          0.030465388674890067,
+          -0.01511217342930631,
+          0.005735960079359902,
+          0.0030955813611916066,
+          0.003343849304714193,
+          -0.017685941897160877,
+          -0.019795694199965495,
+          -0.015889472669342385,
+          0.0005064000414834533,
+          -0.0007719666713392906,
+          -0.012817517834744753,
+          0.01370266779873307,
+          0.0029457043321269045,
+          0.004380574233134056,
+          0.003997906783171028,
+          -0.02610528890228495,
+          0.0033844277218836307,
+          -0.008397899012048187,
+          0.013703609594527308,
+          -0.0012702392316682096,
+          2.036849934035612e-05,
+          0.007261982817787643,
+          -0.00406175072535136,
+          0.018353316401139956,
+          -0.007206833180919801,
+          -0.004174154024217347,
+          -0.004097578846173991,
+          -0.00573702992914836,
+          -0.0017184308802689716,
+          0.022375713882789285,
+          -0.008746034802967245,
+          0.032729322137817145,
+          -0.014835515993879445,
+          0.01049714579786462,
+          0.013344412726432523,
+          0.006468296817982878,
+          -0.0036641997047404426,
+          0.0003384623491490165,
+          -0.0035345323089769998,
+          -0.006579417470328358,
+          0.015682816949756224,
+          -0.018934655843627496,
+          0.006860259740057359,
+          0.03727162736659208,
+          -0.002536806673033066,
+          -0.00615871353663963,
+          -0.008853057427430264,
+          -0.012968501800661,
+          0.028268388127570685,
+          0.005935705363468247,
+          -0.0013092005585187737,
+          -0.03024974000885413,
+          0.015180604705329429,
+          -0.0037501687534873948,
+          0.0016387074669205704,
+          0.00860407823411825,
+          0.00044285387135976006,
+          0.020127437993717013,
+          -0.0011272951769363546,
+          0.021290480413673304,
+          0.010032359217391118,
+          -0.0010666180914844765,
+          0.011529468490024128,
+          -0.036558434912810016,
+          -0.008579049583792913,
+          0.0038048395503202733,
+          0.019444233855445248,
+          -0.0006623807457136332,
+          -0.015853103419331053,
+          -0.004242407392496502,
+          0.009029483146616182,
+          -0.01059695082306233,
+          -0.013492380798775181,
+          0.016259633969459952,
+          -0.019096544374819852,
+          0.011560512412214506,
+          0.008617334373339341,
+          0.02512750386847096,
+          0.0041202753709624245,
+          0.002613588241299414,
+          0.0035069606381226207,
+          0.006602772841623022,
+          -0.002625433099855357,
+          0.011789793979724633,
+          -0.011592113209370399,
+          0.02102479518464762,
+          -0.008294039267346376,
+          0.007068661518691129,
+          0.0032541671568032306,
+          -0.022166914870027622,
+          -0.018234159989683987,
+          -0.0023642465680105544,
+          0.021312775708470108,
+          0.006016177710817625,
+          -0.012618763344460982,
+          0.0055411096806088885,
+          0.004690341026678943,
+          -0.00547301082703577,
+          0.009118596747082881,
+          0.024398283179066196,
+          -0.0028098889343668772,
+          0.01068095200128235,
+          -0.007137857371825465,
+          -0.027832295014375272,
+          -0.007421428223393234,
+          0.021707007996690945,
+          0.00020946296332069377,
+          0.016793613517175475,
+          0.014396690762872469,
+          0.012889706504719996,
+          -0.00488698200749025,
+          -0.005354238088985329,
+          0.018317372626432005,
+          0.014340608961907827,
+          -0.0048105230366691145,
+          0.001495716612807014,
+          -0.0018529641162211795,
+          -0.017240719704808234,
+          0.026933447120216628,
+          -0.024232238634425658,
+          -0.003438477892192365,
+          0.00892792234782449,
+          -0.012586276728773793,
+          -0.004923824123059311,
+          -0.023468301644204742,
+          -0.02117100775850015,
+          0.014296432515879623,
+          -0.003314359969711106,
+          -0.01750062790930347,
+          -0.006856892040084997,
+          6.621834660421394e-05,
+          -0.003983545180909168,
+          -0.010826144842427663,
+          -0.0028995230412062357,
+          0.0018029406642546938,
+          -0.01286603500476852,
+          0.00937188453692113,
+          -0.00635685147895941,
+          0.010242542894487791,
+          0.0062482230941642404,
+          -0.022974002980633818,
+          0.005235079251883088,
+          0.020160136045340955,
+          -0.008953506137454955,
+          0.0007027144749750591,
+          -0.015318432934167311,
+          -0.006155069535076527,
+          -8.304773776972749e-05,
+          0.010665400866776814,
+          0.013561082526851403,
+          0.019453190615040503,
+          0.008724224096424808,
+          0.017716608184445207,
+          0.0035170519664773133,
+          -0.013169473712817174,
+          0.009862672596286218,
+          0.011645070824075182,
+          0.011745045118639314,
+          0.011761235635339118,
+          -0.0051506969158396755,
+          0.00628898514628358,
+          -0.0049857719357619505,
+          0.008525048536699957,
+          0.027507594807546505,
+          0.0013209667681023782,
+          0.017047984110417937,
+          0.016368796085341072,
+          0.020105811276523177,
+          -0.003759327296302425,
+          3.4626946806238876e-05,
+          0.00035688794161846735,
+          -0.001063984476108484,
+          0.003291497239446139,
+          -0.018279629509070614,
+          -0.01135428882896519,
+          0.017033095140582537,
+          -5.492833742539044e-05,
+          0.024685530931127835,
+          -0.0010090516352729089,
+          -0.0184210790691242,
+          -0.006072072044463014,
+          0.0019766255786695347,
+          -0.0143465314042477,
+          -0.00209268265697904,
+          0.008867797876246725,
+          0.02090777280180668,
+          0.007785784409274212,
+          -0.014694559759226521,
+          0.03266241399157856,
+          0.017596572028431588,
+          0.02301544904486291,
+          -0.02019106568899837,
+          -0.008116927224141509,
+          -0.007678137378694171,
+          0.020318874678520133,
+          -0.0068566512882342135,
+          -0.002510061555870801,
+          0.021473513752723834,
+          0.006645783892214728,
+          0.011211922100954015,
+          0.012286872605920514,
+          0.0003084092164612384,
+          0.0028492628853667647,
+          -0.015161921733689642,
+          0.011855756525919763,
+          -0.005289433905822296,
+          -0.006904878573181685,
+          0.021517401941638887,
+          -0.010444313612915583,
+          -0.009104739247715266,
+          0.005434542886627656,
+          -0.011460046711120444,
+          0.013324525803587403,
+          -0.0016717915349416276,
+          -0.013225789210352699,
+          -0.014868291792978007,
+          -0.004059908595020105,
+          -0.017086039690741954,
+          0.018935544362274584,
+          0.006810606440007176,
+          -0.0016747502037076753,
+          -0.005949836930379681,
+          0.004577809518378731,
+          -0.002387527459083855,
+          -0.0006415824062693734,
+          -0.015923707971787795,
+          0.0035047065077239213,
+          0.009700583796235219,
+          -0.0031324165789907116,
+          0.010643108490671052,
+          -0.013353561795075988,
+          0.033779753889958034,
+          -0.003236875179603599,
+          0.0033068852921624326,
+          0.005106828066670305,
+          -0.003034255184307467,
+          0.001481657575335246,
+          0.013519816427440722,
+          0.009066434673988428,
+          -0.008339461460990311,
+          -0.0009211583209818409,
+          0.013497039042975171,
+          -0.018668722901146233,
+          0.0013796912320793106,
+          0.013796986913155082,
+          -0.015849997808276967,
+          0.0049427770053857125,
+          0.011400118227502903,
+          -0.00639345486098034,
+          0.015340608222895147,
+          -0.01371015934577743,
+          0.007008681477695071,
+          0.009775206231977338,
+          0.028490233702896408,
+          -0.01966685875736091,
+          0.01928758485650401,
+          -0.025635008258084145,
+          0.031283028009357144,
+          0.012122242939206308,
+          -0.005796910058436286,
+          -0.025521126197222552,
+          -0.011593941553769327,
+          0.01283377684428954,
+          -0.021686338476615465,
+          0.0014883363235708173,
+          0.004930128822274494,
+          -0.013589466747020829,
+          -0.014766714233107183,
+          -0.01202283835955618,
+          0.011738578705966321,
+          0.0006399892787922201,
+          0.01589633486439762,
+          -0.001565909739326631,
+          0.006195453613357957,
+          0.00799709051068248,
+          -0.007262779704915372,
+          -0.006717513726969218,
+          -0.01982600498555257,
+          0.01452043414139846,
+          0.004256754663990435,
+          -0.005262312998149605,
+          -0.0224549938750886,
+          0.009397743096285409,
+          0.017707219262338467,
+          -0.005753255460862127,
+          0.013047488144162397,
+          -0.03383526087792666,
+          0.0009452367103428395,
+          -0.005822570832057675,
+          -0.0027621026437538738,
+          -0.011741384193082396,
+          0.005215022845011562,
+          -0.015301048137828595,
+          0.002594097201808009,
+          -0.006446736651739076,
+          -0.016963075831388622,
+          0.002297125878220545,
+          0.02099207084363147,
+          -0.0032941933998622765,
+          -0.0028818889706579017,
+          0.01784874229400347,
+          0.008499720521713671,
+          -0.01436426626087195,
+          -0.004000179664908013,
+          0.006397941984552902,
+          -0.018873859156915206,
+          0.0011599963694687036,
+          0.02198209878751397,
+          0.012652203334020664,
+          -0.026378312815268385,
+          -0.0016397712646032201,
+          0.013833745513458322,
+          0.008230921549345205,
+          0.008667892596847433,
+          0.0040942632709300665,
+          0.014593014365768302,
+          -0.006350891941908016,
+          -0.001374093276890415,
+          -6.3005496707736715e-06,
+          -0.006943874751919358,
+          0.011912879044321251,
+          0.003688355066059739,
+          0.040741622477833674,
+          -0.010639328654064204,
+          0.011006744363538067,
+          -0.01080331270092793,
+          -0.004477468739500037,
+          -0.004503151926350927,
+          0.0027311865257027917,
+          -0.013139063275489816,
+          -0.0001322676781706513,
+          0.0021051078165230533,
+          -0.0019788024041678294,
+          0.00649268452600941,
+          0.003305903531478554,
+          0.013176851471483939,
+          -0.017229299828998786,
+          -0.0024884474570636707,
+          -0.009910839696360717,
+          0.019347666346683084,
+          -0.010724923526200773,
+          0.01051871403061451,
+          0.03256893707453101,
+          0.013242686936629418,
+          0.00022445440868234016,
+          -0.014947304198470308,
+          -0.008836459838385262,
+          -0.008992977994567497,
+          -0.01739161854615299,
+          -0.012908166671408042,
+          -0.022654409684101404,
+          -0.00294050967213434,
+          0.006390611639719366,
+          -0.0010555036392130776,
+          -0.002785796833161689,
+          -0.027340941664129582,
+          -0.00792516100590229,
+          -0.03210910565574123,
+          -0.013679170743667392,
+          0.0056845041078574104,
+          -0.021558178294402857,
+          0.005195998493021035,
+          0.029445968279023734,
+          -0.008256723194649932,
+          0.018403154314213917,
+          0.006923789005793937,
+          0.011433558571124386,
+          -0.0022324062216246424,
+          -0.011870704203994723,
+          -0.011035736680889242,
+          0.00032924465177187114,
+          0.003455564411617082,
+          0.004938023824535148,
+          0.00916834019939669,
+          -0.01732811783799426,
+          -0.010720420451770168,
+          0.00025951769018448803,
+          0.01732173296763328,
+          -0.0036441011455210317,
+          -0.014654182169779525,
+          -0.009890543675136605,
+          0.00934755871928037,
+          -0.017560953324850772,
+          0.0028803276180087,
+          0.008144645056336303,
+          -0.006844751836577803,
+          0.009686357271083834,
+          0.00910269542748879,
+          -0.003634975911996508,
+          -0.01125116892179341,
+          0.015123292946443665,
+          0.011417498955214564,
+          -0.0005683452620824292,
+          0.009547451569266946,
+          0.0031907584166949686,
+          -0.004914343407341038,
+          0.0004946517266559977,
+          -0.013981094295278457,
+          -0.008438177812073363,
+          -0.009698303451884585,
+          -0.0004873949117926506,
+          0.008310275049300703,
+          -0.011244097579407173,
+          0.002589333020166005,
+          0.021677389796728952,
+          0.0016179416889548232,
+          -0.011231602157838225,
+          -0.0045296258349024725,
+          -0.03198651985714177,
+          -0.005752566699226788,
+          -0.017524199510403467,
+          0.014980719037744585,
+          -0.002529640898744345,
+          0.01028755110586937,
+          -0.0021848874276835985,
+          0.015817733088597956,
+          -0.005450667684071798,
+          0.012212046004369696,
+          -0.0008714643856749736,
+          -0.004560548606687623,
+          0.006746171236132563,
+          0.0034423786329218276,
+          -0.0009995673054631,
+          0.05643134636651925,
+          0.010296478797092773,
+          0.014377556149215558,
+          0.013825497434998718,
+          -0.017485595187900386,
+          0.013330532940316836,
+          0.013333976202885402,
+          0.012417102160127097,
+          0.004536479576179834,
+          -0.017256228281240734,
+          0.024337236913894427,
+          0.012755805964855683,
+          0.009622694631579495,
+          -0.0018516129725815202,
+          -0.0015877067248695194,
+          0.019827793457522933,
+          0.018266833138588977,
+          -0.009808363592736615,
+          0.011083663940891296,
+          0.004458465015627857,
+          -0.01068421391840819,
+          -0.0050807436529387556,
+          0.024563333063926475,
+          -0.01325209293057542,
+          -0.016539133573613984,
+          0.0059168302415277515,
+          -0.02488439856049784,
+          -0.029755917745609557,
+          0.001920092403971772,
+          0.006364691645397147,
+          0.002128416061225683,
+          -0.0021827593501562285,
+          0.00636889353893684,
+          0.01505350329221805,
+          -0.02223481012881443,
+          -0.01913380824474348,
+          0.004876437698740333,
+          -0.0031323018872010133,
+          0.015711985535422395,
+          -0.02142499388661488,
+          -0.011646214273628826,
+          0.003244037052013272,
+          -0.006889665577990765,
+          0.004421548112756594,
+          0.004704830748153097,
+          -0.004955793799612646,
+          -0.005239972361031197,
+          0.0007828188358756139,
+          0.019667585310240637,
+          0.017115475760556154,
+          0.02008255444898991,
+          0.012206187649127842,
+          -0.004915394518125827,
+          -0.00997481157928988,
+          0.002664275191354035,
+          -0.017453428141032505,
+          -0.005114569347643741,
+          0.006659824028894164,
+          0.013233844584497872,
+          -0.023794827807567007,
+          -0.007354308951883939,
+          0.012842257789962674,
+          0.011294252132532885,
+          -0.00824836528452984,
+          0.004256336058369505,
+          -0.011193345143979632,
+          0.009248664100690892,
+          0.010363170772580968,
+          0.010981447978991254,
+          -0.01250390246078834,
+          0.008163162620777558,
+          0.002528845973804195,
+          0.007003579605495149,
+          -0.005812703866552746,
+          0.008764871082692903,
+          0.013124123077340675,
+          -0.012212447674686537,
+          -0.0013940507123620205,
+          -0.001544873017903471,
+          -0.020493969663982807,
+          0.005782189204516662,
+          0.007033733269347455,
+          0.012878926833341835,
+          -0.008338295354334415,
+          0.015844568159504493,
+          -0.013423240193733334,
+          -0.00468920516522183,
+          0.0063319500880227525,
+          -0.004924778574104498,
+          -0.012053197667708071,
+          0.0017742020845283848,
+          -0.0035640186446405536,
+          -0.008666749197879684,
+          -0.0013894177719374007,
+          0.0016400693271668141,
+          0.004501038858023909,
+          0.02043221461176279,
+          0.001901506572332553,
+          -0.0056076433740056926,
+          0.019660750419391273,
+          0.003538773720733224,
+          -0.0018659146421562808,
+          0.005354994676719758,
+          0.0006789623595561136,
+          -0.0021516503031452773,
+          -0.0027566542393029298,
+          -0.010552067426241374,
+          0.01699044655141112,
+          -0.01883386136434379,
+          -0.006428123254736374,
+          -0.012514829515253224,
+          0.006923832110523631,
+          0.023065674483579466,
+          -0.0031719793455512423,
+          -0.006131653355136594,
+          -0.015640226612880566,
+          0.003589774262421098,
+          -0.07234225978543331,
+          -0.010636515019878534,
+          0.005436135654579842,
+          0.030362090167699224,
+          -0.008735845879763853,
+          0.0012226978854869956,
+          -0.00802187367908004,
+          0.010460901462092434,
+          -0.0024797678489701584,
+          0.005049311815982583,
+          0.0037643360764659196,
+          -0.003379308997651945,
+          -0.0029803517660911536,
+          0.0047612984744651015,
+          -0.010726918354479238,
+          -0.018870259480662628,
+          0.0028549213021684043,
+          0.025034678854420084,
+          0.009004274035878184,
+          0.004138885468102832,
+          0.009765532829693857,
+          -0.008537379113922283,
+          -0.0014902115840393893,
+          0.008462743183234337,
+          0.007328272099011593,
+          -0.009076904245691212,
+          -0.0008002047467975742,
+          0.008347212876126893,
+          -0.021799374855548816,
+          0.0020253943807330738,
+          0.012246389250647798,
+          -0.008912465890948629,
+          -0.006244789880293639,
+          0.002413038795382646,
+          0.004202974630810528,
+          -0.004339960146144246,
+          -0.009919301413657872,
+          -0.023862183602380987,
+          0.006400371798756263,
+          -0.007410957922309058,
+          -0.0001606112434129549,
+          0.0031662502228666104,
+          0.00012871401736865864,
+          -0.01687187870925491,
+          0.01374600947628664,
+          -0.008834671549515985,
+          -0.0033344289547729495,
+          0.012937173759658114,
+          -0.003267867578821276,
+          0.01864135643890992,
+          -0.014050228437162755,
+          0.007864998326137568,
+          -0.018988002335416778,
+          -0.003944450943320823,
+          -0.0032084001089808037,
+          -0.0023149926514803745,
+          -0.007750311931438483,
+          -0.045019596490520515,
+          0.013573650825957857,
+          -0.004374086208559833,
+          -0.005869005771503507,
+          -0.01250037134701792,
+          -0.016625210662158708,
+          0.011639132814162461,
+          0.004045997475412364,
+          -0.0014885281342871186,
+          -0.020321856013396592,
+          0.006222318025479057,
+          0.011121009394355636,
+          0.00904225037228108,
+          -0.011585148988670874,
+          -0.0018474735072673144,
+          -0.011325387916132704,
+          -0.0025890917877355725,
+          -0.009148717095934154,
+          0.018495348432506,
+          0.002404017871470538,
+          -0.010232069372010613,
+          0.022941087706292104,
+          -0.0024570420644715457,
+          0.020988341201251853,
+          -0.00442363926883869,
+          -0.004485898239018673,
+          -0.020447941631096866,
+          -0.004158168168969816,
+          -0.019358617104349624,
+          0.00036464612100595666,
+          0.0017749921060331022,
+          0.02078748568527368,
+          0.007256675717496499,
+          0.012612722550842472,
+          0.006032799184116686,
+          0.00631983142330159,
+          0.030151438614535002,
+          -0.010852408231267204,
+          0.0022451749744724233,
+          -0.017863828302314567,
+          0.010653806941450981,
+          0.028289026813870025,
+          0.009882042879445822,
+          -0.006920783006708277,
+          0.014053717954960316,
+          0.004932716665921658,
+          -0.00616100713292227,
+          0.025975267677767474,
+          -0.038836632659249616,
+          -0.021059874446788137,
+          -0.0061122622002863615,
+          0.001439949038430725,
+          -3.496872289060673e-05,
+          0.005030571386336924,
+          -0.011007922189367617,
+          0.009222155733548683,
+          0.0026655561130744807,
+          0.03101907459085608,
+          -0.002317540524509921,
+          -0.007348120500263025,
+          0.003987101796623003,
+          0.001511347369284844,
+          0.021229069065914295,
+          -0.01197457000398312,
+          -0.038429200853184124,
+          -0.0031917847963042626,
+          -0.0039488592790253305,
+          0.008371872140716302,
+          -0.012355524487988753,
+          -0.010839396359961,
+          -0.004574476927098761,
+          -0.01882180934875166,
+          -0.010292867074240718,
+          0.025595330740853118,
+          -0.007162452868646569,
+          0.01788200594893745,
+          -0.0076384034977797525,
+          -0.008112046063674713,
+          0.00683079906571602,
+          0.011572398977364099,
+          -0.01932794114157753,
+          0.0034055065356524052,
+          -0.0027824032743918262,
+          0.01627342501880651,
+          0.011718826886349412,
+          -0.013342112644751174,
+          0.006234666245077415,
+          0.01365078200475753,
+          -0.015476695744659024,
+          -0.0015378873781631448,
+          -0.0069121070205125585,
+          0.014866073481143516,
+          -0.003093423905641203,
+          -0.0033762696975023986,
+          0.011958295888528918,
+          -0.021982979614167224,
+          0.01697002117600754,
+          -0.02183323542526832,
+          0.022295922951583082,
+          -0.0006195453455557405,
+          -0.0006320640234089136,
+          0.008423668060539851,
+          0.008314680408345411,
+          -0.013378756052181496,
+          0.0016194432891316378,
+          -0.03416272433582183,
+          -0.009962172336151867,
+          -0.015051382650973002,
+          0.0038009542554177348,
+          -0.010106196779819915,
+          -0.01203081704994182,
+          -0.021869748479536547,
+          -0.013231917312580397,
+          0.009488867268928662,
+          0.02421155136770049,
+          -0.004349181861466944,
+          0.015665735941744613,
+          -0.008077879618601087,
+          -0.00346140349979159,
+          -0.019022501109520248,
+          -0.020158795455478176,
+          -0.011854460388465065,
+          -0.01570428154296937,
+          -0.007073290993912972,
+          0.004794071576762192,
+          -0.014492538032767992,
+          0.013391092874532436,
+          0.03615243857749714,
+          -0.017029836110560772,
+          0.008609740828901341,
+          0.019179133878629843,
+          0.0009923228421593012,
+          0.015393048602064972,
+          0.003790868078763006,
+          0.011710499244312703,
+          -0.014885526287290077,
+          -0.011829126494406653,
+          -0.0005627435441977781,
+          0.0020582596099707785,
+          0.003728052108395647,
+          0.027914733124690644,
+          0.0038604865752177283,
+          0.005302574635039923,
+          0.012293011832771443,
+          0.013320598642264739,
+          -0.010706185765391238,
+          0.015853648683536427,
+          0.004090821159159474,
+          -0.016980287717038522,
+          -0.0031197192883179663,
+          -0.006894138060845641,
+          -0.012739354473335233,
+          -0.002311031608856898,
+          0.0159372988694303,
+          -0.008091532959711314,
+          0.015169701976466646,
+          -0.004316674756425857,
+          -0.015090694840235596,
+          0.0007287957651103694,
+          -0.03810971752015664,
+          -0.0018729979445690102,
+          -0.01301867399306695,
+          -0.023136174288569147,
+          0.04241471786668668,
+          0.00415464531754789,
+          0.020836251111427635,
+          -0.006703690210920993,
+          0.006147438331654669,
+          -0.020673361951422737,
+          0.011783676068789637,
+          -0.0037516561524537765,
+          0.009702463136945385,
+          0.009366943837185086,
+          -0.018879598518030567,
+          -0.01132474738893142,
+          0.018484010977000243,
+          0.0008964474295952695,
+          0.001899853004269779,
+          0.014452586238316783,
+          0.019746940887134506,
+          0.01136527620507647,
+          -0.0020960934961233306,
+          0.02093114800466563,
+          0.018023169361765227,
+          -0.0002475129435970239,
+          -0.007118840223520255,
+          0.013527604253725194,
+          -0.02531530780276405,
+          0.014601163833111726,
+          0.03268673940622878,
+          0.013556815596615061,
+          0.020396267313433348,
+          -0.016161264086739215,
+          0.016259608464481445,
+          -0.006718110409512467,
+          0.017378727943854342,
+          0.001211924791251576,
+          0.01911265175070918,
+          0.014022657097776944,
+          0.01308266544853342,
+          0.006784756253744209,
+          -0.013367188766230505,
+          0.024435211011756285,
+          -0.008342072323415817,
+          -0.0004209982405190692,
+          -0.002823543536521797,
+          -0.023279946325307846,
+          0.01506543259962226,
+          0.0017095020186682082,
+          0.003411744637061027,
+          -0.0035317155649076414,
+          -0.001957702240702538,
+          0.008037874596341196,
+          0.008176809711854463,
+          -0.015780325305129808,
+          0.0031090160698441006,
+          -0.01924979849975192,
+          -0.008566500568783712,
+          -0.024111269290772643,
+          0.003704817310374187,
+          -0.015182531891766549,
+          -0.009971369164255839,
+          -0.019503345629597463,
+          -0.011275291123690537,
+          -0.010129027262987365,
+          0.00026549411615768147,
+          -0.0077553590287942975,
+          0.014297350661562049,
+          -0.017119422159171904,
+          -0.021341131823214652,
+          0.007945720599865192,
+          0.01439604029848557,
+          0.0037256533134793816,
+          0.005472351446140726,
+          0.008325407758477418,
+          0.009082034215555286,
+          -0.010830069811539129,
+          -0.010352768708989164,
+          0.001902868536460922,
+          0.0010307757364948558,
+          0.011572880679973516,
+          0.01480960939553474,
+          0.011328449881374384,
+          -0.013429607187664061,
+          0.0019326261581148884,
+          -0.014862631307250641,
+          0.004686987038977201,
+          -0.0011432874921856958,
+          -0.007026108142521568,
+          -0.0023122147402109927,
+          -0.01067305755615663,
+          0.008329587642130216,
+          0.0051774766682198365,
+          0.0014058609424645312,
+          0.0175095191841222,
+          -0.0016739751490535205,
+          -0.01295313808457633,
+          -0.012767766732175707,
+          -0.011653259281510757,
+          -0.015653450921388538,
+          0.006739771869002279,
+          -0.006908534840842188,
+          -0.006309735406143942,
+          0.00969748644524724,
+          -0.004300978285892996,
+          -0.0071082865141610775,
+          0.009870599453242536,
+          0.0003007194077865931,
+          -0.0011846789159672513,
+          0.03344372215649544,
+          -0.004482311849310301,
+          0.004153681423879411,
+          0.00022516282287549753,
+          -0.00096173441700326,
+          -0.01608715959652527,
+          -0.026963713856310283,
+          -0.0014323662880336336,
+          -0.02273696636049197,
+          0.017938234567205,
+          -0.0058065575851078514,
+          -0.001006759040376689,
+          0.002493842516193206,
+          -0.0042740143630253385,
+          -6.58295494211919e-05,
+          -0.011768915967276026,
+          0.0022431022211178383,
+          -0.029090994997079067,
+          0.009898341521414653,
+          -0.009544941198003441,
+          -0.02154260971685405,
+          0.01133866603878075,
+          0.040150135462722274,
+          -0.0034082812055093122,
+          0.0006477463440760081,
+          0.0010662318198694314,
+          0.010346430535548179,
+          0.014953186668393513,
+          -0.0019239187336010772,
+          -0.01603518108536549,
+          -0.010550241019652155,
+          0.007810013487665037,
+          0.0003366114893320438,
+          0.0030211673366186986,
+          0.025036691996012227,
+          -0.019097101590128545,
+          -0.016027314610013832,
+          -0.011102216330632247,
+          -0.011450582072890358,
+          0.009254748792585386,
+          0.019696746434919966,
+          0.013185751991632784,
+          -0.020709092762886617,
+          -0.017120964154502993,
+          0.00025737692261907426,
+          -0.02662183259737709,
+          -0.0015175233130328546,
+          0.12569047385271798,
+          -0.004250752280902458,
+          -0.009469510190778525,
+          -0.016304049953896647,
+          -0.00863767582150345,
+          0.02183050591695681,
+          -0.008697910752438377,
+          0.06936580824413162,
+          0.0006861269173917291,
+          -0.0029257096931296273,
+          -0.0020896063525880225,
+          -0.002993306834763656,
+          -0.013871927123029641,
+          0.001864949832879178,
+          -0.016847962023455045,
+          -0.01637518023758693,
+          -0.006332388047537449,
+          -0.013325712872379694,
+          -0.0017155737214029734,
+          -0.0061798136617018905,
+          0.004759205228314821,
+          -0.007830001529147273,
+          0.011495001353183024,
+          -0.026107152853762815,
+          -0.010130127689151806,
+          0.005962957535201191,
+          -0.011637257947784772,
+          0.0011307756612654312,
+          -0.030816554116329904,
+          -0.00441741097001083,
+          7.409036337997905e-05,
+          0.016501246370776126,
+          0.0015748302561177006,
+          -0.00940138978060357,
+          -0.011216292373500447,
+          0.006508185064247167,
+          0.021984022174755303,
+          0.02029850289834145,
+          0.020629050747183544,
+          0.009741513502990842,
+          0.017418812192334685,
+          -0.007031919478404256,
+          0.02625468226435525,
+          0.006196721544177288,
+          0.002206946290342702,
+          0.005596780938975919,
+          -0.01211153541496144,
+          -0.008534755574472074,
+          0.0074101096913398124,
+          -0.0219026467376419,
+          -0.0238583746438936,
+          -0.012260621283748488,
+          0.025739429552984337,
+          -0.009054977254128021,
+          -0.0019759724570709175,
+          -0.0030737356218219617,
+          0.024727831360321823,
+          -0.005149317224364814,
+          0.002332136352952536,
+          0.015587381251136284,
+          -0.028035348333895063,
+          0.008643405636450286,
+          0.007622365990511162,
+          0.02564089666732,
+          -0.0032690983288409517,
+          0.0030897564314706603,
+          -0.009711753474608342,
+          -0.009285462125807194,
+          -0.009613024258506932,
+          0.004240295765592783,
+          0.018961596475867332,
+          0.005797211858953558,
+          0.009922313409983964,
+          -0.006335322307636977,
+          -0.015248479856831972,
+          0.018517674622994848,
+          -0.018526204881631703,
+          -0.017805480495530986,
+          -0.0031014632675223346,
+          -0.023499333544416315,
+          0.007508363381003107,
+          0.006594465447638183,
+          -0.013304818586245387,
+          -0.0232989529893049,
+          0.020407788403667446,
+          0.003426820233878876,
+          -0.006520960224510217,
+          0.0028020004149410366,
+          -0.03282622678411901,
+          -0.018099884128291804,
+          0.004341732255376245,
+          0.012442902960644058,
+          -0.006914419410521592,
+          0.01684289728480369,
+          -0.0213314411022478,
+          0.021170682961259797,
+          -0.018444729425471068,
+          0.015400297173773751,
+          -0.005856807329212788,
+          0.01661347113847185,
+          0.011889835104373263,
+          0.0013585650928227913,
+          -0.019926344687386053,
+          0.02233564528528766,
+          0.0006518460244140545,
+          0.029977976063273147,
+          0.026669882018303386,
+          -0.005623183408671759,
+          -0.004497323576380724,
+          -0.017964594171161242,
+          0.011149712487916564,
+          0.0034380718823694635,
+          0.00295265329839474,
+          0.023835186931253683,
+          0.02803963164846445,
+          -0.028097235508129332,
+          0.017703040589712025,
+          -0.012893023698870347,
+          0.0089348477172039,
+          -0.011630331709306501,
+          0.0096768368423985,
+          0.00669273666692988,
+          0.003825890215203911,
+          0.008289398654583944,
+          0.011807275797649763,
+          0.024893949220958504,
+          0.02893733381281628,
+          0.004069052493622539,
+          0.012465630768347759,
+          -0.003600044227856378,
+          0.004954707230784128,
+          0.005985890100113585,
+          0.0014569711209781388,
+          -0.00648453455551435,
+          0.0031945554565833674,
+          -0.006090168119075504,
+          -0.003744222112779555,
+          0.015266602618825367,
+          -0.00028966361595811053,
+          0.015634624351841753,
+          0.012395295870300452,
+          -0.006577117978669073,
+          -0.003069372773552921,
+          -0.0029369830414480856,
+          -0.007598245840968154,
+          -0.017753359726409172,
+          0.025057809370078198,
+          -0.01606095813358513,
+          -0.019631204097884002,
+          0.0008782653868466668,
+          0.0031106969086903075,
+          -0.013675045629189149,
+          -0.008463196400259825,
+          0.008777824363912681,
+          0.014471929439779247,
+          -0.018478175346087953,
+          0.0017862044141924377,
+          -0.0004910793014860654,
+          0.013862986726603731,
+          -0.003641460118879092,
+          -0.00902817956483427,
+          -0.002076665354355939,
+          0.010393957588184604,
+          0.016413601414994997,
+          0.0038017609859922327,
+          -0.008644753777502864,
+          -0.0005484639867925445,
+          0.006452371249117759,
+          0.007391332924183688,
+          0.0036307379469783137,
+          -4.895894824382588e-05,
+          -0.008129811261078888,
+          -0.013655410823646597,
+          0.018829640339982075,
+          -0.007579460691893014,
+          0.016191139169828096,
+          -0.004984562798832502,
+          -0.011846989328030879,
+          0.004110855640416496,
+          0.0004775146965349116,
+          -0.014308885325290177,
+          0.002778951512640393,
+          -0.003583348080466357,
+          0.0017351729844533707,
+          0.006444402813802743,
+          -0.004945263366121632,
+          0.003803172980003619,
+          -0.008178062077842394,
+          0.02700102856098108,
+          0.015477045085093356,
+          0.005818362056942523,
+          -0.00876269298951217,
+          -0.024697543974427114,
+          0.008183329610246195,
+          0.00518473714350693,
+          0.018661120751406378,
+          0.014464207183253898,
+          0.0019868192486975638,
+          0.009827229842256728,
+          -0.013395371817208287,
+          -0.0022376892192351944,
+          0.009711801873474911,
+          0.017963581533215926,
+          -0.0059700736266576305,
+          0.019169874693002874,
+          -0.010316234316657931,
+          -0.0008125759237295765,
+          0.004003428870809722,
+          0.017109043212624506,
+          -0.004590883969173384,
+          0.013958072401225709
+        ],
+        "title": "F Pseudo Code for Switch Transformers 33",
+        "keyphrases": [
+          "Switch Transformers",
+          "Mixture-of-Experts",
+          "sparsely-activated expert model",
+          "natural language tasks",
+          "pre-training and fine-tuning"
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "294d1382-75f0-4a45-bc91-9727e57552f3",
+      "properties": {
+        "page_content": "1 Introduction And Related Work 1.1 Conditional Computation\n\nExploiting scale in both training data and model size has been central to the success of deep learning. When datasets are sufficiently large, increasing the capacity (number of parameters) of neural networks can give much better prediction accuracy. This has been shown in domains such as text (Sutskever et al., 2014; Bahdanau et al., 2014; Jozefowicz et al., 2016; Wu et al., 2016), images (Krizhevsky et al., 2012; Le et al., 2012), and audio (Hinton et al., 2012; Amodei et al., 2015). For typical deep learning models, where the entire model is activated for every example, this leads to a roughly quadratic blow-up in training costs, as both the model size and the number of training examples increase. Unfortunately, the advances in computing power and distributed computation fall short of meeting such demand.\n\nVarious forms of conditional computation have been proposed as a way to increase model capacity without a proportional increase in computational costs (Davis & Arel, 2013; Bengio et al., 2013; Eigen et al., 2013; Ludovic Denoyer, 2014; Cho & Bengio, 2014; Bengio et al., 2015; Almahairi et al., 2015). In these schemes, large parts of a network are active or inactive on a per-example basis. The gating decisions may be binary or sparse and continuous, stochastic or deterministic. Various forms of reinforcement learning and back-propagation are proposed for trarining the gating decisions.\n\nWhile these ideas are promising in theory, no work to date has yet demonstrated massive improvements in model capacity, training time, or model quality. We blame this on a combination of the following challenges:\n\nModern computing devices, especially GPUs, are much faster at arithmetic than at branching. Most of the works above recognize this and propose turning on/off large chunks of the network with each gating decision.\n\nLarge batch sizes are critical for performance, as they amortize the costs of parameter transfers and updates. Conditional computation reduces the batch sizes for the conditionally active chunks of the network.\n\nNetwork bandwidth can be a bottleneck. A cluster of GPUs may have computational power thousands of times greater than the aggregate inter-device network bandwidth. To be computationally efficient, the relative computational versus network demands of an algorithm must exceed this ratio. Embedding layers, which can be seen as a form of conditional computation, are handicapped by this very problem. Since the embeddings generally need to be sent across the network, the number of (example, parameter) interactions is limited by network bandwidth instead of computational capacity.\n\nDepending on the scheme, loss terms may be necessary to achieve the desired level of sparsity per-chunk and/or per example. Bengio et al. (2015) use three such terms. These issues can affect both model quality and load-balancing.\n\nModel capacity is most critical for very large data sets. The existing literature on conditional computation deals with relatively small image recognition data sets consisting of up to 600,000 images. It is hard to imagine that the labels of these images provide a sufficient signal to adequately train a model with millions, let alone billions of parameters. In this work, we for the first time address all of the above challenges and finally realize the promise of conditional computation. We obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets.\n\n1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1). All parts of the network are trained jointly by back-propagation.\n\nWhile the introduced technique is generic, in this paper we focus on language modeling and machine translation tasks, which are known to benefit from very large models. In particular, we apply a MoE convolutionally between stacked LSTM layers (Hochreiter & Schmidhuber, 1997), as in Figure 1.\n\nThe MoE is called once for each position in the text, selecting a potentially different combination of experts at each position. The different experts tend to become highly specialized based on syntax and semantics (see Appendix E Table 9). On both language modeling and machine translation benchmarks, we improve on best published results at a fraction of the computational cost.\n\n1.3 Related Work On Mixtures Of Experts\n\nSince its introduction more than two decades ago (Jacobs et al., 1991; Jordan & Jacobs, 1994), the mixture-of-experts approach has been the subject of much research. Different types of expert architectures hae been proposed such as SVMs (Collobert et al., 2002), Gaussian Processes (Tresp, 2001; Theis & Bethge, 2015; Deisenroth & Ng, 2015), Dirichlet Processes (Shahbaba & Neal, 2009), and deep networks. Other work has focused on different expert configurations such as a hierarchical structure (Yao et al., 2009), infinite numbers of experts (Rasmussen & Ghahramani, 2002), and adding experts sequentially (Aljundi et al., 2016). Garmash & Monz (2016) suggest an ensemble model in the format of mixture of experts for machine translation. The gating network is trained on a pre-trained ensemble NMT model. The works above concern top-level mixtures of experts. The mixture of experts is the whole model. Eigen et al. (2013) introduce the idea of using multiple MoEs with their own gating networks as parts of a deep model. It is intuitive that the latter approach is more powerful, since complex problems may contain many sub-problems each requiring different experts. They also allude in their conclusion to the potential to introduce sparsity, turning MoEs into a vehicle for computational computation.\n\nOur work builds on this use of MoEs as a general purpose neural network component. While Eigen et al. (2013) uses two stacked MoEs allowing for two sets of gating decisions, our convolutional application of the MoE allows for different gating decisions at each position in the text. We also realize sparse gating and demonstrate its use as a practical way to massively increase model capacity.\n\n",
+        "embedding": [
+          -0.011026207357645035,
+          -0.0030965988989919424,
+          -0.017856089398264885,
+          0.0010101930238306522,
+          0.02138679102063179,
+          0.0306765865534544,
+          -0.014028750360012054,
+          0.025452885776758194,
+          -0.030126724392175674,
+          0.021791953593492508,
+          0.010809156112372875,
+          -0.003031483618542552,
+          0.0071120485663414,
+          -0.010223117657005787,
+          0.014021515846252441,
+          0.00018878943228628486,
+          0.01843489333987236,
+          0.003255769843235612,
+          0.03244193643331528,
+          -0.030329305678606033,
+          0.006352368742227554,
+          -0.010158002376556396,
+          -0.03186313435435295,
+          -0.031226450577378273,
+          0.018391482532024384,
+          0.023152140900492668,
+          0.040400486439466476,
+          -0.0036120957229286432,
+          -0.027333997189998627,
+          0.01920180767774582,
+          0.016163088381290436,
+          0.000524540722835809,
+          -0.03342590481042862,
+          -0.03348378464579582,
+          -0.058574918657541275,
+          -0.044220589101314545,
+          0.04589911922812462,
+          0.018217841163277626,
+          -0.003402279457077384,
+          0.018926875665783882,
+          0.037419646978378296,
+          -0.001556438859552145,
+          -0.03244193643331528,
+          -0.025858048349618912,
+          0.0014379650820046663,
+          0.016626130789518356,
+          -0.016466960310935974,
+          -0.03426516801118851,
+          -0.015902627259492874,
+          -0.01490419078618288,
+          0.02338366210460663,
+          -0.025742286816239357,
+          0.01846383325755596,
+          -0.000727121951058507,
+          -0.040603067725896835,
+          0.033049680292606354,
+          0.026639433577656746,
+          -0.004460404627025127,
+          0.04691202566027641,
+          0.04639110341668129,
+          0.0038635137025266886,
+          0.006019556894898415,
+          -0.04427846893668175,
+          -0.0017617333214730024,
+          0.02446891926229,
+          -0.015251473523676395,
+          -0.024526799097657204,
+          0.021010568365454674,
+          0.018594063818454742,
+          -0.00322140334174037,
+          -0.037680111825466156,
+          0.00494153518229723,
+          -0.03119751065969467,
+          0.01049804873764515,
+          0.005907413549721241,
+          0.01499101147055626,
+          0.02413610741496086,
+          0.04008214548230171,
+          0.021242089569568634,
+          0.011192613281309605,
+          -0.003957569133490324,
+          0.022862739861011505,
+          -0.006706885993480682,
+          0.010975562036037445,
+          0.027218235656619072,
+          -0.02911381796002388,
+          0.005502250976860523,
+          -0.06227926164865494,
+          0.004485727287828922,
+          -0.013276305980980396,
+          -0.0250043123960495,
+          0.013565707951784134,
+          -0.0021397643722593784,
+          0.0019570793956518173,
+          0.010577634908258915,
+          0.03658038377761841,
+          -0.0005281582125462592,
+          -0.013869579881429672,
+          0.005491398740559816,
+          -0.0063632214441895485,
+          -0.01565663516521454,
+          0.01856512390077114,
+          0.007361657451838255,
+          -0.0035723031032830477,
+          0.014448382891714573,
+          -0.0009423645096831024,
+          -0.023730944842100143,
+          0.026205329224467278,
+          0.008710993453860283,
+          0.08670476824045181,
+          -0.009463437832891941,
+          -0.022775918245315552,
+          -0.001728271134197712,
+          0.03600158169865608,
+          -0.0008876494830474257,
+          -0.011489249765872955,
+          -0.04349708557128906,
+          -0.04118186980485916,
+          0.007216956466436386,
+          -0.022110294550657272,
+          0.03328120335936546,
+          0.03843255341053009,
+          -0.015164652839303017,
+          -0.010158002376556396,
+          -0.05591242015361786,
+          0.07883304357528687,
+          -0.013695938512682915,
+          -0.035162314772605896,
+          -0.008703758008778095,
+          -0.027044596150517464,
+          -0.002848798641934991,
+          0.00745209539309144,
+          -0.01866641454398632,
+          -0.02755104750394821,
+          0.04659368470311165,
+          0.012415335513651371,
+          0.03773799166083336,
+          0.022139234468340874,
+          -0.01235022023320198,
+          -0.013630823232233524,
+          0.03244193643331528,
+          0.0003031935775652528,
+          0.004037154838442802,
+          -0.017783738672733307,
+          -0.007980253547430038,
+          0.005802505649626255,
+          -0.022732509300112724,
+          0.057648833841085434,
+          -0.03794057294726372,
+          0.02171960286796093,
+          -0.030300365760922432,
+          0.031023869290947914,
+          -0.016163088381290436,
+          0.03941652178764343,
+          -0.01902816630899906,
+          0.01389128528535366,
+          -0.0250043123960495,
+          -0.017957380041480064,
+          0.006417484022676945,
+          -0.011178143322467804,
+          -0.016987884417176247,
+          -0.019910842180252075,
+          -0.015613225288689137,
+          0.02323896251618862,
+          0.0025485444348305464,
+          0.03073446825146675,
+          0.0352201946079731,
+          0.019476739689707756,
+          -0.0614689365029335,
+          0.019447797909379005,
+          0.004789599217474461,
+          0.021632781252264977,
+          0.019968722015619278,
+          -0.0019860195461660624,
+          0.03134221211075783,
+          0.009297031909227371,
+          0.02205241471529007,
+          0.03252875804901123,
+          -0.005419048015028238,
+          0.0330207422375679,
+          -0.03001096285879612,
+          -0.01251662615686655,
+          0.02878100611269474,
+          0.008117719553411007,
+          0.025091134011745453,
+          0.008877399377524853,
+          0.015801336616277695,
+          -0.007813847623765469,
+          0.0341494083404541,
+          0.0012118698796257377,
+          0.01969379000365734,
+          0.023962466046214104,
+          0.036725085228681564,
+          -0.030589766800403595,
+          0.0233113132417202,
+          -0.04074776917695999,
+          -0.019375449046492577,
+          0.024092696607112885,
+          0.02338366210460663,
+          0.017002353444695473,
+          -0.03368636593222618,
+          -0.03212359547615051,
+          0.02480173110961914,
+          0.01720493473112583,
+          0.013695938512682915,
+          0.009246386587619781,
+          0.013703173957765102,
+          0.0147884301841259,
+          0.017595628276467323,
+          -0.037882693111896515,
+          0.009969891048967838,
+          0.00405524205416441,
+          -0.009940951131284237,
+          -0.04303404316306114,
+          0.006829881574958563,
+          -0.0030152045655995607,
+          -0.022240525111556053,
+          -0.0033299291972070932,
+          -0.0182033721357584,
+          -0.003758605569601059,
+          -0.002535883104428649,
+          0.01681424304842949,
+          0.021459141746163368,
+          -0.021632781252264977,
+          -0.0020529436878859997,
+          -0.017291756346821785,
+          -0.021835362538695335,
+          -0.03235511854290962,
+          0.003633800894021988,
+          -0.005400960333645344,
+          0.02464256063103676,
+          0.03140009194612503,
+          0.017885029315948486,
+          -0.004778746515512466,
+          0.03938758000731468,
+          -0.03368636593222618,
+          0.016626130789518356,
+          -0.0012697502970695496,
+          0.007488270755857229,
+          0.008862929418683052,
+          -0.024353159591555595,
+          0.04178961366415024,
+          -0.0409214086830616,
+          -0.005690362304449081,
+          0.011330079287290573,
+          0.010035006329417229,
+          -0.010794686153531075,
+          0.0044133770279586315,
+          -0.023354722186923027,
+          -0.02649473212659359,
+          0.0013222043635323644,
+          -0.012762618251144886,
+          -0.016553781926631927,
+          0.02205241471529007,
+          -0.02539500594139099,
+          0.03976380452513695,
+          0.030618706718087196,
+          -0.02490302175283432,
+          -0.01976614072918892,
+          -0.006916702259331942,
+          0.04123975336551666,
+          -0.009108920581638813,
+          -0.01268303208053112,
+          0.02569887787103653,
+          -0.025612056255340576,
+          0.0019950633868575096,
+          0.036956604570150375,
+          0.022587807849049568,
+          -0.004930682480335236,
+          -0.01800079084932804,
+          0.0038237208500504494,
+          -0.05845915898680687,
+          0.0035361277405172586,
+          -0.041471272706985474,
+          0.05614394322037697,
+          -0.03736176714301109,
+          0.0041999430395662785,
+          0.0013303437735885382,
+          -0.012111463584005833,
+          0.04028472676873207,
+          0.022935090586543083,
+          -0.009405557066202164,
+          -0.022804860025644302,
+          0.02480173110961914,
+          0.015468524768948555,
+          -0.004123975057154894,
+          -0.015208062715828419,
+          -0.0024273572489619255,
+          -0.05116623267531395,
+          0.002250098856166005,
+          -0.005104323849081993,
+          -0.002651643706485629,
+          -0.048359036445617676,
+          0.015671106055378914,
+          0.028607364743947983,
+          -0.001820518053136766,
+          -0.011402430012822151,
+          -0.01437603309750557,
+          0.02708800509572029,
+          -0.017566686496138573,
+          -0.042397357523441315,
+          -0.006793706677854061,
+          0.013970870524644852,
+          0.008776108734309673,
+          -0.03209465742111206,
+          0.006746678613126278,
+          -0.003434837330132723,
+          0.02543841488659382,
+          -0.0076691466383636,
+          0.004728101193904877,
+          0.018188901245594025,
+          0.0015220724744722247,
+          0.01614861935377121,
+          -0.01747986674308777,
+          -0.006120847538113594,
+          0.02141573093831539,
+          0.021676192060112953,
+          0.0006932076648809016,
+          -0.016872122883796692,
+          -0.03316544368863106,
+          0.032181475311517715,
+          0.002027621027082205,
+          -0.0637841522693634,
+          -0.01426027249544859,
+          -0.03542277589440346,
+          0.01033164281398058,
+          0.004677455872297287,
+          0.03247087821364403,
+          -0.02228393591940403,
+          0.018246781080961227,
+          -0.011713536456227303,
+          -0.03542277589440346,
+          -0.024512330070137978,
+          0.013666998594999313,
+          -0.030879167839884758,
+          -0.03987956419587135,
+          -0.017118114978075027,
+          0.023991405963897705,
+          -0.013117135502398014,
+          -0.029026998206973076,
+          -0.01969379000365734,
+          -0.031255390495061874,
+          0.0030748937278985977,
+          0.027739159762859344,
+          0.01774032786488533,
+          -0.024946432560682297,
+          -0.034033648669719696,
+          0.0057265376672148705,
+          -0.01178588718175888,
+          -0.0018594063585624099,
+          0.0010210456093773246,
+          0.018261251971125603,
+          0.006489834748208523,
+          -0.011257728561758995,
+          -0.017552217468619347,
+          -0.04413377121090889,
+          -0.008407121524214745,
+          0.0005385585827752948,
+          -0.009904775768518448,
+          0.034699272364377975,
+          0.03180525451898575,
+          -0.012125934474170208,
+          -0.011612245813012123,
+          0.014745020307600498,
+          -0.047201428562402725,
+          -0.015367234125733376,
+          -0.025640996173024178,
+          -0.005306905135512352,
+          0.01684318296611309,
+          0.008928044699132442,
+          -0.003950334154069424,
+          0.042889341711997986,
+          0.002532265381887555,
+          -0.013507827185094357,
+          0.007922373712062836,
+          0.006956494878977537,
+          0.028144322335720062,
+          0.0057265376672148705,
+          -0.03637780249118805,
+          0.04381542652845383,
+          -0.02464256063103676,
+          0.010027770884335041,
+          0.013348656706511974,
+          -0.011728006415069103,
+          -0.013761053793132305,
+          0.018550653010606766,
+          -0.005636099260300398,
+          0.025858048349618912,
+          -0.020779047161340714,
+          0.061526816338300705,
+          -0.006761149037629366,
+          -0.05186079815030098,
+          -0.0147884301841259,
+          -0.04089247062802315,
+          0.021271029487252235,
+          0.013377596624195576,
+          -0.002273612655699253,
+          -0.020344944670796394,
+          -0.013030314818024635,
+          0.019288627430796623,
+          0.004000979475677013,
+          -0.009629843756556511,
+          -0.01235022023320198,
+          0.006366838701069355,
+          -0.01578686572611332,
+          -0.005990616511553526,
+          0.007376127410680056,
+          0.007625736761838198,
+          0.04187643527984619,
+          -0.03142903372645378,
+          -0.0031201126985251904,
+          -0.01594603806734085,
+          -0.048127513378858566,
+          0.0049560051411390305,
+          0.03154479339718819,
+          0.0364067442715168,
+          -0.023947997018694878,
+          0.0003253509057685733,
+          0.013175015337765217,
+          0.0018973903497681022,
+          -0.017653508111834526,
+          0.011113028042018414,
+          0.03336802497506142,
+          0.02457020990550518,
+          -0.0187966451048851,
+          -0.005245407111942768,
+          0.0062438431195914745,
+          -0.0020746488589793444,
+          0.00621852045878768,
+          -0.04031366482377052,
+          0.03637780249118805,
+          -0.01235022023320198,
+          0.02998202294111252,
+          -0.028708655387163162,
+          -0.013102664612233639,
+          -0.03481503203511238,
+          -0.010223117657005787,
+          0.03472821041941643,
+          -0.032876040786504745,
+          -0.007188016548752785,
+          0.017262814566493034,
+          -0.002754742978140712,
+          -0.006724973674863577,
+          0.018550653010606766,
+          -0.04636216163635254,
+          -0.019520148634910583,
+          -0.027536578476428986,
+          0.0008266037912108004,
+          -0.0018811115296557546,
+          -0.05871962010860443,
+          -0.061295293271541595,
+          -0.005350315012037754,
+          -0.054233890026807785,
+          -0.019491208717226982,
+          0.03935864195227623,
+          -0.009477907791733742,
+          -0.002881356282159686,
+          -0.019838491454720497,
+          0.009543023072183132,
+          -0.028694184496998787,
+          -0.0034185582771897316,
+          0.026205329224467278,
+          0.017190465703606606,
+          -0.018319131806492805,
+          -0.006236608140170574,
+          0.001457861391827464,
+          -0.062221381813287735,
+          -0.027507638558745384,
+          0.0053466977551579475,
+          -0.02477279119193554,
+          -0.008609702810645103,
+          -0.019780611619353294,
+          0.0038526610005646944,
+          0.01460755430161953,
+          0.00634151604026556,
+          0.037622228264808655,
+          0.0022898914758116007,
+          -0.06980370730161667,
+          0.013182250782847404,
+          0.024454450234770775,
+          0.023137671872973442,
+          -0.023600714281201363,
+          -0.00623299041762948,
+          -0.02228393591940403,
+          -0.022529928013682365,
+          -0.02636450156569481,
+          0.0061859628185629845,
+          0.004109505098313093,
+          0.005440753418952227,
+          -0.013884049840271473,
+          0.009311501868069172,
+          0.0421079583466053,
+          0.005227319430559874,
+          0.0153382932767272,
+          0.0037296651862561703,
+          -0.011366254650056362,
+          -0.020402824506163597,
+          -0.003644653595983982,
+          0.02924404852092266,
+          0.011460309848189354,
+          -0.03681190311908722,
+          -0.00158537901006639,
+          0.04427846893668175,
+          0.03530701622366905,
+          0.02198006398975849,
+          -0.011764181777834892,
+          0.009043805301189423,
+          0.020706696435809135,
+          -0.06013768911361694,
+          -0.00040742341661825776,
+          0.010042241774499416,
+          0.0057844179682433605,
+          -0.02148808166384697,
+          -0.020373884588479996,
+          -0.008754403330385685,
+          0.01953461952507496,
+          -0.013789994642138481,
+          0.008276890963315964,
+          6.890248914714903e-05,
+          0.033049680292606354,
+          0.011236024089157581,
+          -0.012531096115708351,
+          -0.017103644087910652,
+          0.05825657770037651,
+          -0.02539500594139099,
+          0.02878100611269474,
+          0.026842014864087105,
+          -0.023658594116568565,
+          0.04734613001346588,
+          -0.005386490374803543,
+          -0.04057412967085838,
+          0.015801336616277695,
+          0.009022099897265434,
+          -0.02171960286796093,
+          -0.025293715298175812,
+          0.0006579368491657078,
+          -0.007430390454828739,
+          0.002532265381887555,
+          -0.04642004519701004,
+          -0.010266527533531189,
+          -0.0025811020750552416,
+          0.017624568194150925,
+          -0.015642166137695312,
+          0.006019556894898415,
+          0.02600274793803692,
+          -0.00706140324473381,
+          0.041037172079086304,
+          -0.041934315115213394,
+          -0.008045368827879429,
+          0.00811048410832882,
+          -0.03750646859407425,
+          -0.03180525451898575,
+          -0.0013927459949627519,
+          -0.0012588977115228772,
+          -0.018058670684695244,
+          -0.018709823489189148,
+          0.00033439468825235963,
+          -0.008305830880999565,
+          -0.006869674660265446,
+          -0.04254205897450447,
+          0.011662891134619713,
+          -0.013522298075258732,
+          -0.006417484022676945,
+          0.003910541534423828,
+          -0.010592104867100716,
+          -0.019114986062049866,
+          0.024266337975859642,
+          0.03484397381544113,
+          0.024266337975859642,
+          0.023861175402998924,
+          -0.0021126328501850367,
+          -0.021271029487252235,
+          -0.01455690898001194,
+          0.02685648389160633,
+          0.001517550554126501,
+          0.007206104230135679,
+          0.024845141917467117,
+          -0.004927065223455429,
+          0.0015410644700750709,
+          -0.011293903924524784,
+          0.0363488607108593,
+          -0.008703758008778095,
+          0.00639216136187315,
+          0.04485727474093437,
+          0.02795621007680893,
+          0.003259387332946062,
+          -0.006649005692452192,
+          -0.0039539518766105175,
+          0.005277964752167463,
+          -0.03394682705402374,
+          0.01840595155954361,
+          0.019389918074011803,
+          0.0038237208500504494,
+          -0.0056071593426167965,
+          0.006431954447180033,
+          -0.024179518222808838,
+          0.011496485210955143,
+          0.01183653250336647,
+          0.00784278754144907,
+          -0.03660932183265686,
+          0.012545567005872726,
+          0.019259687513113022,
+          0.0014054073253646493,
+          -0.015974977985024452,
+          -0.0116701265797019,
+          -0.014238567091524601,
+          0.024222927168011665,
+          0.001188355963677168,
+          -0.003324502846226096,
+          0.0009622608777135611,
+          0.030994929373264313,
+          0.012777088209986687,
+          0.016727421432733536,
+          -0.022732509300112724,
+          0.02675519324839115,
+          -0.0074159204959869385,
+          0.0068009416572749615,
+          -0.0010110974544659257,
+          -0.014462853781878948,
+          -0.013363126665353775,
+          0.03565429896116257,
+          -0.0019968722481280565,
+          0.04138445109128952,
+          -0.005755477584898472,
+          0.013384832069277763,
+          0.03770904988050461,
+          -0.0027728306595236063,
+          -0.02600274793803692,
+          -0.007712556980550289,
+          0.0007709843921475112,
+          -0.015309353359043598,
+          -0.016293318942189217,
+          -0.027507638558745384,
+          -0.013269071467220783,
+          -0.009427262470126152,
+          -0.017885029315948486,
+          -0.012342985719442368,
+          0.025062192231416702,
+          -0.0037658405490219593,
+          -0.0182033721357584,
+          -0.03727494925260544,
+          0.028940176591277122,
+          0.0004478944465517998,
+          0.03692766651511192,
+          -0.008240715600550175,
+          0.012994139455258846,
+          -0.008616937324404716,
+          0.013920225203037262,
+          0.016307789832353592,
+          0.013023079372942448,
+          0.022370757535099983,
+          0.0005584549508057535,
+          0.00988307036459446,
+          0.03565429896116257,
+          -0.031486913561820984,
+          -0.03348378464579582,
+          0.014976541511714458,
+          -0.01522253267467022,
+          0.022095825523138046,
+          0.0029211491346359253,
+          -0.0050283558666706085,
+          0.021401260048151016,
+          0.006457277107983828,
+          -0.034207288175821304,
+          0.01966485008597374,
+          -0.012444276362657547,
+          -0.03319438174366951,
+          0.03435198962688446,
+          -0.014802900142967701,
+          0.004131210036575794,
+          -0.012480450794100761,
+          -0.0025485444348305464,
+          -0.024353159591555595,
+          0.026711782440543175,
+          0.019809551537036896,
+          -0.025915928184986115,
+          0.02523583360016346,
+          0.030358245596289635,
+          0.013348656706511974,
+          -0.004142062738537788,
+          0.008450531400740147,
+          -0.0048510972410440445,
+          -0.022443106397986412,
+          -0.02549629658460617,
+          0.002917531644925475,
+          0.011641185730695724,
+          -0.005719302222132683,
+          0.021459141746163368,
+          -0.049661342054605484,
+          -0.007481035776436329,
+          0.0005797079065814614,
+          -0.016438020393252373,
+          -0.004764276556670666,
+          0.00960090383887291,
+          0.013761053793132305,
+          0.0030622323974967003,
+          -0.03154479339718819,
+          0.0318920761346817,
+          0.029258519411087036,
+          0.020070012658834457,
+          -0.008732697926461697,
+          0.017566686496138573,
+          -0.006374074146151543,
+          0.012096993625164032,
+          0.031949955970048904,
+          0.01827572099864483,
+          0.008038134314119816,
+          0.0005860385717824101,
+          -0.027898330241441727,
+          0.02015683241188526,
+          0.011330079287290573,
+          0.035827938467264175,
+          0.02128550037741661,
+          0.021864304319024086,
+          0.02225499600172043,
+          -0.022023474797606468,
+          0.006493452005088329,
+          -0.006558567751199007,
+          -0.016235439106822014,
+          0.01658272184431553,
+          -0.010628280229866505,
+          0.03504655510187149,
+          0.012147638946771622,
+          -0.05186079815030098,
+          -0.012314044870436192,
+          0.019809551537036896,
+          0.022356286644935608,
+          0.04372860863804817,
+          0.017335165292024612,
+          0.0025666318833827972,
+          0.04838797450065613,
+          -0.02642238140106201,
+          -0.0306765865534544,
+          0.005871238186955452,
+          0.02025812305510044,
+          -0.024859612807631493,
+          0.009760074317455292,
+          0.02464256063103676,
+          -0.018217841163277626,
+          -0.016365669667720795,
+          0.015613225288689137,
+          0.0020999715197831392,
+          0.0030459535773843527,
+          -0.014137276448309422,
+          -0.008096014149487019,
+          0.02487408183515072,
+          0.006301723420619965,
+          0.036985546350479126,
+          -0.01771138794720173,
+          0.019143927842378616,
+          0.00183589244261384,
+          -0.01473778486251831,
+          0.0284481942653656,
+          0.0011639377335086465,
+          -0.03362848609685898,
+          0.025742286816239357,
+          0.012965199537575245,
+          0.010512519627809525,
+          0.009912010282278061,
+          -0.0009658784256316721,
+          0.021907713264226913,
+          0.04233947768807411,
+          0.010592104867100716,
+          0.0004639472172129899,
+          0.03319438174366951,
+          0.024946432560682297,
+          0.003083937568590045,
+          0.01415898185223341,
+          -0.006649005692452192,
+          -0.013052019290626049,
+          0.011532660573720932,
+          -0.02061987668275833,
+          0.0278838612139225,
+          -0.023846706375479698,
+          -0.012393631041049957,
+          -0.004572547972202301,
+          -0.0021144417114555836,
+          -0.0011386150727048516,
+          -0.012096993625164032,
+          -0.008074309676885605,
+          -0.0187677051872015,
+          0.020402824506163597,
+          -0.007965783588588238,
+          0.037419646978378296,
+          -0.007188016548752785,
+          -0.0012118698796257377,
+          0.015048892237246037,
+          -0.026610491797327995,
+          0.002241055015474558,
+          -0.0010174281196668744,
+          0.021734071895480156,
+          -0.01571451686322689,
+          -0.005599924363195896,
+          0.012929024174809456,
+          0.03912711888551712,
+          -0.009260856546461582,
+          0.0003251247981097549,
+          0.029504509642720222,
+          0.0017798208864405751,
+          -0.03817209228873253,
+          0.028838885948061943,
+          -0.046506863087415695,
+          0.008667582646012306,
+          0.009116156026721,
+          -0.022168176248669624,
+          -0.019737200811505318,
+          -0.025640996173024178,
+          -0.004406142048537731,
+          0.0033009888138622046,
+          -0.01203911378979683,
+          0.020807987079024315,
+          -0.00955025851726532,
+          -0.025423945859074593,
+          0.02998202294111252,
+          0.014665434136986732,
+          0.009470673277974129,
+          -0.02675519324839115,
+          0.014267507009208202,
+          -0.036754023283720016,
+          -0.024193987250328064,
+          -0.0038273383397608995,
+          -0.012531096115708351,
+          0.0076980870217084885,
+          -0.006909467279911041,
+          -0.027666809037327766,
+          -0.014021515846252441,
+          0.00466660363599658,
+          -0.01584474742412567,
+          0.039705920964479446,
+          0.035625357180833817,
+          0.04349708557128906,
+          0.014035985805094242,
+          -0.035856880247592926,
+          0.02626321092247963,
+          -0.011170907877385616,
+          0.02059093490242958,
+          -0.009159565903246403,
+          0.0009902967140078545,
+          0.000590108276810497,
+          0.03637780249118805,
+          -0.02536606416106224,
+          0.010360583662986755,
+          -0.04254205897450447,
+          -0.008023664355278015,
+          -0.027377407997846603,
+          0.0032756661530584097,
+          -0.006352368742227554,
+          -0.001016523689031601,
+          0.009991596452891827,
+          -0.004362731706351042,
+          0.021502550691366196,
+          0.04236841946840286,
+          0.0016324068419635296,
+          0.02977944165468216,
+          -0.03883771598339081,
+          -0.020214714109897614,
+          0.013312481343746185,
+          -0.008595232851803303,
+          -0.013146075420081615,
+          -0.019114986062049866,
+          -0.012538331560790539,
+          -0.021213149651885033,
+          0.00700352294370532,
+          0.013710408471524715,
+          -0.01200293842703104,
+          0.001774394535459578,
+          0.016698481515049934,
+          -0.017291756346821785,
+          -0.0018847290193662047,
+          -0.03371530771255493,
+          -0.011633951216936111,
+          0.01899922639131546,
+          -0.026350030675530434,
+          0.029504509642720222,
+          -0.0045580780133605,
+          0.027970680966973305,
+          -0.02420845814049244,
+          -0.010751275345683098,
+          -0.0021415730006992817,
+          -0.02805750072002411,
+          -0.0020221949089318514,
+          0.016003917902708054,
+          0.03889559581875801,
+          -0.01761009730398655,
+          0.04714354872703552,
+          0.0003796137170866132,
+          0.013565707951784134,
+          -0.024555740877985954,
+          0.005249024368822575,
+          -0.052439600229263306,
+          0.008942514657974243,
+          -0.008660348132252693,
+          -0.033570606261491776,
+          -0.01581580750644207,
+          -0.004829391837120056,
+          0.013189485296607018,
+          -0.022370757535099983,
+          -0.00500665046274662,
+          0.01044740341603756,
+          -0.012560036964714527,
+          0.00022609512961935252,
+          -0.023021910339593887,
+          -0.020185774192214012,
+          -0.007065020501613617,
+          0.012639622204005718,
+          -0.03440986946225166,
+          0.016669541597366333,
+          -0.0352780744433403,
+          -0.0049704755656421185,
+          -0.027464227750897408,
+          -0.021271029487252235,
+          0.02911381796002388,
+          0.027232706546783447,
+          -0.04508879408240318,
+          0.029895203188061714,
+          0.018926875665783882,
+          0.011576070450246334,
+          -0.03976380452513695,
+          0.03840361535549164,
+          0.00019421572505962104,
+          0.010273762978613377,
+          0.003168949391692877,
+          -0.008645878173410892,
+          0.02228393591940403,
+          -0.03047400526702404,
+          -0.013052019290626049,
+          0.00472448393702507,
+          -0.006377691403031349,
+          0.012654092162847519,
+          -0.0010617427760735154,
+          -0.023586243391036987,
+          0.016307789832353592,
+          0.011482015252113342,
+          0.012871143408119678,
+          0.03339696303009987,
+          -0.025583116337656975,
+          0.0352780744433403,
+          0.0045182849280536175,
+          -0.018550653010606766,
+          0.006793706677854061,
+          -0.013507827185094357,
+          0.0034764388110488653,
+          0.029157228767871857,
+          0.03930075839161873,
+          0.008587997406721115,
+          -0.036956604570150375,
+          0.02397693693637848,
+          0.022226056084036827,
+          -0.0035867730621248484,
+          0.02867971546947956,
+          0.011474779807031155,
+          0.024613620713353157,
+          0.03545171767473221,
+          -0.0023441542871296406,
+          0.003197889542207122,
+          0.026017218828201294,
+          -0.023094261065125465,
+          0.007596796378493309,
+          -0.01246598083525896,
+          0.03186313435435295,
+          0.011149203404784203,
+          -0.021401260048151016,
+          -0.004189090803265572,
+          -0.010577634908258915,
+          0.009304266422986984,
+          0.007122900802642107,
+          -0.014101101085543633,
+          -0.02857842482626438,
+          -0.014933131635189056,
+          -0.004286763723939657,
+          0.010360583662986755,
+          -0.06262654066085815,
+          -0.01440497301518917,
+          -0.018883464857935905,
+          -0.010816390626132488,
+          0.0036555060651153326,
+          0.01889793574810028,
+          -0.009123390540480614,
+          -0.0029772205743938684,
+          0.0153382932767272,
+          0.05191867798566818,
+          -0.008566292002797127,
+          0.00704693328589201,
+          0.000250739511102438,
+          0.01034611277282238,
+          -0.01640908047556877,
+          0.03507549315690994,
+          -0.013544002547860146,
+          -0.048561617732048035,
+          0.008768873289227486,
+          -0.025293715298175812,
+          0.01863747462630272,
+          0.047230370342731476,
+          -0.023296842351555824,
+          -0.03733282908797264,
+          0.020171303302049637,
+          0.009297031909227371,
+          -0.02934533916413784,
+          -0.00019738104310818017,
+          -0.008226245641708374,
+          -0.008132189512252808,
+          -0.021068448200821877,
+          -0.004785981960594654,
+          0.04636216163635254,
+          -0.008537352085113525,
+          0.025742286816239357,
+          0.03394682705402374,
+          0.017016824334859848,
+          -0.03134221211075783,
+          0.02562652714550495,
+          0.005017503164708614,
+          -0.00505729578435421,
+          0.002235628664493561,
+          0.04031366482377052,
+          -0.02941768988966942,
+          -0.012328515760600567,
+          -0.030358245596289635,
+          -0.016032857820391655,
+          -0.03883771598339081,
+          -0.0005697597516700625,
+          0.0024888552725315094,
+          0.008725463412702084,
+          -0.03160267323255539,
+          -0.032904982566833496,
+          0.031081749126315117,
+          0.03704342618584633,
+          -0.01427474245429039,
+          -0.005390108097344637,
+          -0.010953856632113457,
+          -0.005194761790335178,
+          0.006739443633705378,
+          -0.013175015337765217,
+          -0.001295977272093296,
+          -0.00739421509206295,
+          0.0142240971326828,
+          0.0035343191120773554,
+          0.0071120485663414,
+          -0.05504421517252922,
+          -0.018116550520062447,
+          0.018623003736138344,
+          0.0025792932137846947,
+          0.00119197356980294,
+          -0.004453169647604227,
+          -0.0051332637667655945,
+          0.00782831758260727,
+          -0.026104038581252098,
+          0.02318108081817627,
+          0.022211585193872452,
+          0.005509485956281424,
+          -0.002067413879558444,
+          0.007075873203575611,
+          0.03273133933544159,
+          0.030184604227542877,
+          -0.014079395681619644,
+          -0.015468524768948555,
+          0.03154479339718819,
+          -0.0019100516801699996,
+          0.014614789746701717,
+          -0.0023495806381106377,
+          0.01568557508289814,
+          -1.6335372492903844e-05,
+          -0.01853618398308754,
+          -0.008204540237784386,
+          -0.01622096821665764,
+          0.005824210587888956,
+          0.03131327033042908,
+          -0.00011587375774979591,
+          0.01953461952507496,
+          0.015092302113771439,
+          0.017089175060391426,
+          0.013847874477505684,
+          -0.009528553113341331,
+          -0.018926875665783882,
+          0.01794290915131569,
+          0.0029537067748606205,
+          0.026104038581252098,
+          0.007093960884958506,
+          -0.0039394814521074295,
+          -0.008906339295208454,
+          -0.011409664526581764,
+          -0.005035590846091509,
+          0.009246386587619781,
+          0.02125656045973301,
+          0.03380212560296059,
+          -0.008117719553411007,
+          -0.02716035582125187,
+          -0.01501995138823986,
+          0.021430201828479767,
+          0.02138679102063179,
+          -0.01907157711684704,
+          -0.021459141746163368,
+          0.01740751601755619,
+          -0.014130041003227234,
+          0.015931567177176476,
+          0.0019932547584176064,
+          0.003203315893188119,
+          -0.010903211310505867,
+          0.07107707113027573,
+          -0.02759445831179619,
+          -0.004398907069116831,
+          -0.02798515185713768,
+          -0.003205124521628022,
+          0.006652622949331999,
+          -0.014781195670366287,
+          -0.0013737539993599057,
+          0.018623003736138344,
+          -0.003024248406291008,
+          0.022399697452783585,
+          0.01601838879287243,
+          -0.012017408385872841,
+          -0.00755338603630662,
+          -0.028071971610188484,
+          0.027174826711416245,
+          -0.008660348132252693,
+          0.0037405178882181644,
+          -0.0038309558294713497,
+          0.011359019204974174,
+          0.03713024780154228,
+          0.024989843368530273,
+          0.013363126665353775,
+          0.01740751601755619,
+          -0.010671690106391907,
+          0.013442711904644966,
+          0.012856673449277878,
+          0.014665434136986732,
+          -0.026017218828201294,
+          -0.015092302113771439,
+          0.001866641454398632,
+          0.006428336724638939,
+          0.008146659471094608,
+          0.005661421921104193,
+          -0.001539255608804524,
+          0.012762618251144886,
+          -0.006446424406021833,
+          0.00756062101572752,
+          -0.0009215637692250311,
+          -0.0204896442592144,
+          -0.009854130446910858,
+          0.025091134011745453,
+          0.0003529344976413995,
+          0.024179518222808838,
+          0.0011567026376724243,
+          -0.005542044062167406,
+          0.002711332868784666,
+          -0.02061987668275833,
+          0.00019195476488675922,
+          -0.01714705489575863,
+          -0.0063849263824522495,
+          0.0036790198646485806,
+          -0.01648143120110035,
+          -0.01195229310542345,
+          -0.01170630194246769,
+          0.000591917079873383,
+          0.0003979274188168347,
+          0.0056288642808794975,
+          0.00756062101572752,
+          -0.0036844462156295776,
+          0.009058275260031223,
+          -0.01016523689031601,
+          0.01011459156870842,
+          -0.007850022986531258,
+          -0.012871143408119678,
+          -0.007705322001129389,
+          0.011402430012822151,
+          0.009427262470126152,
+          -0.004492962267249823,
+          0.01160501129925251,
+          -0.016365669667720795,
+          0.003754987847059965,
+          -0.010924916714429855,
+          -0.025915928184986115,
+          -0.006124464794993401,
+          -0.02739187702536583,
+          0.004521902650594711,
+          -0.013131605461239815,
+          0.022269466891884804,
+          -0.024266337975859642,
+          0.027232706546783447,
+          0.007770437281578779,
+          -0.009297031909227371,
+          -0.00021286855917423964,
+          -0.03620416298508644,
+          0.011395194567739964,
+          -0.01846383325755596,
+          0.0028940176125615835,
+          0.012299574911594391,
+          -0.003438454819843173,
+          0.0056071593426167965,
+          -0.008645878173410892,
+          -0.010121827013790607,
+          -0.003353442996740341,
+          -0.021531492471694946,
+          0.004098652396351099,
+          0.006768384017050266,
+          -0.0031906545627862215,
+          -0.01499101147055626,
+          -0.01727728545665741,
+          -0.023528363555669785,
+          0.017812678590416908,
+          0.022891679778695107,
+          -0.021010568365454674,
+          -0.020518586039543152,
+          -0.005314140114933252,
+          0.0030712762381881475,
+          -0.02708800509572029,
+          0.0033064151648432016,
+          -0.008710993453860283,
+          0.01016523689031601,
+          -0.009065510705113411,
+          0.010606574825942516,
+          -0.003870748681947589,
+          0.020070012658834457,
+          0.018377011641860008,
+          0.02351389452815056,
+          -0.03620416298508644,
+          0.0017409324645996094,
+          0.010599339380860329,
+          -0.009246386587619781,
+          -0.014014280401170254,
+          0.021242089569568634,
+          -0.010403993539512157,
+          -0.0020999715197831392,
+          0.03090810962021351,
+          0.000624926935415715,
+          -0.008964220061898232,
+          0.008472236804664135,
+          -0.0017662551254034042,
+          -0.021531492471694946,
+          0.026436852291226387,
+          0.010830861516296864,
+          0.00905104074627161,
+          0.022573338821530342,
+          -0.003820103360339999,
+          0.005397343076765537,
+          -0.017494337633252144,
+          0.020981628447771072,
+          -0.0036772112362086773,
+          -0.009072745218873024,
+          0.019679320976138115,
+          0.026089569553732872,
+          -0.008233480155467987,
+          0.011510955169796944,
+          -0.03249981999397278,
+          0.0012028261553496122,
+          0.03892453759908676,
+          0.01255280151963234,
+          0.0009378425893373787,
+          -0.014281976968050003,
+          -0.012075289152562618,
+          0.013449947349727154,
+          0.021198678761720657,
+          -0.004442316945642233,
+          -0.02875206619501114,
+          0.008009194396436214,
+          -0.011257728561758995,
+          0.03322332352399826,
+          0.018594063818454742,
+          -0.00976730976253748,
+          0.027464227750897408,
+          -0.004619575571268797,
+          -0.01830466091632843,
+          0.01681424304842949,
+          -0.005379255395382643,
+          -0.004243353381752968,
+          0.011843767017126083,
+          0.0036482710856944323,
+          0.03044506534934044,
+          0.006837116554379463,
+          -0.0055709839798510075,
+          -0.0014605745673179626,
+          0.01661166176199913,
+          -0.002420122269541025,
+          -0.024165047332644463,
+          0.0048510972410440445,
+          0.010288232937455177,
+          -0.0007777672726660967,
+          -0.012588976882398129,
+          0.011966763064265251,
+          -0.014694374985992908,
+          -0.0030423360876739025,
+          0.0008921714033931494,
+          0.006099142134189606,
+          -0.00047615633229725063,
+          -0.0233113132417202,
+          -0.006576654966920614,
+          -0.03449669107794762,
+          0.028708655387163162,
+          0.009441732428967953,
+          0.007538916077464819,
+          0.013218426145613194,
+          0.005419048015028238,
+          0.008624172769486904,
+          -0.0034185582771897316,
+          -0.007850022986531258,
+          0.004485727287828922,
+          -0.009919245727360249,
+          -0.02831796370446682,
+          -0.03163161501288414,
+          0.016394609585404396,
+          -0.021299969404935837,
+          -0.0066236830316483974,
+          0.0028596511110663414,
+          -0.02433868870139122,
+          -0.02205241471529007,
+          0.003123730421066284,
+          0.01661166176199913,
+          -0.022139234468340874,
+          0.01664060167968273,
+          -0.001658633933402598,
+          -0.007350804749876261,
+          0.023730944842100143,
+          -0.008023664355278015,
+          -0.001483184052631259,
+          -0.0116701265797019,
+          -0.02669731341302395,
+          0.013666998594999313,
+          0.01283496804535389,
+          -0.0002509655896574259,
+          0.009608138352632523,
+          0.0012118698796257377,
+          0.010707865469157696,
+          0.009116156026721,
+          -0.01830466091632843,
+          0.01273367740213871,
+          -0.02716035582125187,
+          0.03345484286546707,
+          0.01910051703453064,
+          -0.028129851445555687,
+          0.0352780744433403,
+          -0.008710993453860283,
+          0.015859216451644897,
+          -0.0035288927610963583,
+          0.01400704588741064,
+          0.010845331475138664,
+          0.017393046990036964,
+          0.01601838879287243,
+          0.01399257592856884,
+          0.0041529154404997826,
+          0.01920180767774582,
+          -0.006088289897888899,
+          0.023065321147441864,
+          0.013949165120720863,
+          -3.0748939025215805e-05,
+          0.013580177910625935,
+          0.003338972805067897,
+          -0.018449362367391586,
+          -0.0015220724744722247,
+          -0.021068448200821877,
+          -0.007654676679521799,
+          0.0030622323974967003,
+          -0.0204317644238472,
+          -0.004561695270240307,
+          -0.030821288004517555,
+          0.006598360370844603,
+          0.0016595382476225495,
+          -0.012458746321499348,
+          -0.009362147189676762,
+          0.010765746235847473,
+          0.002649834845215082,
+          0.022775918245315552,
+          -0.011908882297575474,
+          0.02241416648030281,
+          0.008935279212892056,
+          -0.002969985594972968,
+          0.01049804873764515,
+          -0.02457020990550518,
+          0.014499028213322163,
+          0.0017535937950015068,
+          0.022472048178315163,
+          0.021632781252264977,
+          0.0005114271771162748,
+          0.010150766931474209,
+          0.0016197455115616322,
+          0.015352764166891575,
+          -0.015164652839303017,
+          -0.012429806403815746,
+          -0.00489088986068964,
+          -0.015584285371005535,
+          0.05814081430435181,
+          -0.02171960286796093,
+          -0.012314044870436192,
+          -0.017682448029518127,
+          -0.002385755768045783,
+          0.009311501868069172,
+          -0.030126724392175674,
+          -0.018651943653821945,
+          -0.022168176248669624,
+          -0.006797323934733868,
+          -0.02038835547864437,
+          -0.004579782951623201,
+          0.021603841334581375,
+          -0.0011901648249477148,
+          0.007850022986531258,
+          -0.01953461952507496,
+          -0.0068009416572749615,
+          -0.01409386657178402,
+          -0.003257578704506159,
+          -0.014535203576087952,
+          -0.012024643830955029,
+          -0.0042144134640693665,
+          0.021545961499214172,
+          0.01426027249544859,
+          -0.010859801433980465,
+          0.006486217025667429,
+          0.013580177910625935,
+          -0.012010172940790653,
+          0.003698916407302022,
+          0.01714705489575863,
+          0.011691831052303314,
+          -0.00944896787405014,
+          0.01007841620594263,
+          -0.0026842013467103243,
+          -0.005419048015028238,
+          0.004963240120559931,
+          0.0023749032989144325,
+          0.03244193643331528,
+          0.010512519627809525,
+          -0.04520455747842789,
+          -0.00811048410832882,
+          0.010968326590955257,
+          0.006844351999461651,
+          0.0019697407260537148,
+          0.005400960333645344,
+          -0.015309353359043598,
+          0.004250588361173868,
+          -0.03093704953789711,
+          -0.0001339613663731143,
+          -0.0026263210456818342,
+          -0.008645878173410892,
+          0.006634535733610392,
+          -0.006591125391423702,
+          0.01177141722291708,
+          0.01923074759542942,
+          -0.003143626730889082,
+          0.01457137893885374,
+          0.0030712762381881475,
+          0.013218426145613194,
+          -0.03281816095113754,
+          0.008739933371543884,
+          -0.011380724608898163,
+          0.009029335342347622,
+          -0.012328515760600567,
+          -0.035856880247592926,
+          -0.010867035947740078,
+          0.015728985890746117,
+          -0.002018577419221401,
+          -0.005976146552711725,
+          -0.0021922183223068714,
+          0.02626321092247963,
+          0.019780611619353294,
+          -0.0014280168106779456,
+          -0.0065621850080788136,
+          0.030589766800403595,
+          -0.0006945642526261508,
+          0.0002321997017133981,
+          0.0003368817560840398,
+          0.004934300202876329,
+          -0.011829297058284283,
+          0.005697597283869982,
+          -0.01740751601755619,
+          0.0067177386954426765,
+          -0.008052604272961617,
+          0.026899894699454308,
+          -0.0019064341904595494,
+          -0.014614789746701717,
+          0.0193609781563282,
+          -0.032876040786504745,
+          -0.004398907069116831,
+          -0.006478982046246529,
+          -0.00466660363599658,
+          0.021806422621011734,
+          -0.0019480356713756919,
+          0.025076663121581078,
+          -0.05145563557744026,
+          -0.0014379650820046663,
+          -0.004051624797284603,
+          -0.016365669667720795,
+          0.004283146001398563,
+          -0.02135785110294819,
+          0.014130041003227234,
+          -0.0030459535773843527,
+          -0.015757925808429718,
+          0.002360433107241988,
+          -0.008291360922157764,
+          -0.008595232851803303,
+          0.013175015337765217,
+          0.01578686572611332,
+          -0.003349825507029891,
+          0.014253037050366402,
+          0.0039539518766105175,
+          0.015265943482518196,
+          0.017494337633252144,
+          -0.013066490180790424,
+          -0.013558472506701946,
+          0.01178588718175888,
+          0.0006185962702147663,
+          0.01697341352701187,
+          0.008320300839841366,
+          -0.009991596452891827,
+          -0.02749316766858101,
+          0.005447988398373127,
+          -0.006714120972901583,
+          -0.0036301834043115377,
+          -0.018492773175239563,
+          -0.016626130789518356,
+          -0.001278794021345675,
+          -0.01668401248753071,
+          0.00461595831438899,
+          -0.023036381229758263,
+          -0.0010617427760735154,
+          0.008812284097075462,
+          -0.02338366210460663,
+          -0.004847479518502951,
+          0.013580177910625935,
+          0.016235439106822014,
+          -0.03247087821364403,
+          0.003870748681947589,
+          -0.0037369003985077143,
+          -0.028129851445555687,
+          -0.0035126139409840107,
+          0.0025720582343637943,
+          -0.013732113875448704,
+          0.01500548142939806,
+          -0.011344549246132374,
+          -0.017103644087910652,
+          -0.03397576883435249,
+          -0.022298406809568405,
+          -0.007010757923126221,
+          0.019158396869897842,
+          0.005462458357214928,
+          0.004178238101303577,
+          0.0027185678482055664,
+          0.0003034196561202407,
+          0.0076691466383636,
+          0.0284481942653656,
+          -0.013724878430366516,
+          -6.6804045673052315e-06,
+          -0.001633311272598803,
+          0.02061987668275833,
+          -0.004373584408313036,
+          0.004113122820854187,
+          0.004605105612426996,
+          0.000946886430028826,
+          0.0033425905276089907,
+          -0.005317757371813059,
+          -0.006938407197594643,
+          -0.010874271392822266,
+          0.010483578778803349,
+          0.024888552725315094,
+          -0.01910051703453064,
+          -0.0012679415522143245,
+          -0.02028706483542919,
+          0.029461100697517395,
+          0.030126724392175674,
+          0.00784278754144907,
+          -0.0409214086830616,
+          -0.0032340646721422672,
+          -0.0030170134268701077,
+          -0.0014325387310236692,
+          -0.02420845814049244,
+          0.01203911378979683,
+          0.006196815520524979,
+          0.009702194482088089,
+          0.018753234297037125,
+          -0.006583890412002802,
+          0.017002353444695473,
+          -0.008573527447879314,
+          -0.005968911573290825,
+          0.001627884921617806,
+          -0.007878962904214859,
+          0.015077832154929638,
+          0.04115293174982071,
+          -0.028838885948061943,
+          0.00650068698450923,
+          -0.007857258431613445,
+          0.022674627602100372,
+          -0.009543023072183132,
+          0.009195741266012192,
+          0.02061987668275833,
+          0.017161523923277855,
+          -0.0051151760853827,
+          0.016958944499492645,
+          0.07582326233386993,
+          -0.003993744496256113,
+          0.022009003907442093,
+          0.008768873289227486,
+          -0.000894884520675987,
+          0.0041167400777339935,
+          -0.019187336787581444,
+          -0.02305085025727749,
+          0.017233874648809433,
+          0.0039756568148732185,
+          -0.004492962267249823,
+          0.011496485210955143,
+          -0.015164652839303017,
+          0.006319811102002859,
+          0.017827149480581284,
+          0.005386490374803543,
+          -0.012639622204005718,
+          -0.006927554961293936,
+          0.022906150668859482,
+          0.019158396869897842,
+          0.0056288642808794975,
+          -0.010823626071214676,
+          0.01454243902117014,
+          -0.005646951962262392,
+          -0.01661166176199913,
+          0.016192028298974037,
+          -0.01691553369164467,
+          0.018782174214720726,
+          -0.0024978991132229567,
+          0.024787262082099915,
+          -0.007640206720679998,
+          -0.010172472335398197,
+          -0.0016088929260149598,
+          0.00032648135675117373,
+          0.01684318296611309,
+          0.019635910168290138,
+          0.0036573149263858795,
+          6.426754407584667e-05,
+          0.002027621027082205,
+          0.012364690192043781,
+          0.0006294488557614386,
+          0.0032431085128337145,
+          0.00500665046274662,
+          0.003143626730889082,
+          -0.012965199537575245,
+          0.023962466046214104,
+          0.006048496812582016,
+          0.008132189512252808,
+          -0.00616064015775919,
+          -0.005618012044578791,
+          0.004894507583230734,
+          -0.009904775768518448,
+          0.00650068698450923,
+          0.001803334802389145,
+          -0.003291945206001401,
+          -0.01261791680008173,
+          0.029721561819314957,
+          -0.021198678761720657,
+          0.014281976968050003,
+          0.01923074759542942,
+          0.008219010196626186,
+          0.00978177972137928,
+          -0.013189485296607018,
+          -0.018362542614340782,
+          0.00761850131675601,
+          -0.00027221854543313384,
+          0.005864003207534552,
+          0.0051115588285028934,
+          0.001834083697758615,
+          0.016597190871834755,
+          -0.00244182744063437,
+          0.005495015997439623,
+          0.013225660659372807,
+          0.007864492945373058,
+          0.005976146552711725,
+          0.016134148463606834,
+          -0.030329305678606033,
+          -0.018753234297037125,
+          0.004315703641623259,
+          0.00516582140699029,
+          -0.0005634290864691138,
+          0.0043554967269301414,
+          0.01006394624710083,
+          0.011279433965682983,
+          -0.008399886079132557,
+          -0.015497464686632156,
+          -0.014817370101809502,
+          0.04184749722480774,
+          0.014347092248499393,
+          -0.020750107243657112,
+          -0.011959527619183064,
+          -0.0021650870330631733,
+          0.003226829692721367,
+          -0.007198868785053492,
+          0.014933131635189056,
+          0.014962071552872658,
+          -0.018955815583467484,
+          -0.02590145915746689,
+          -0.027102475985884666,
+          -0.012704737484455109,
+          0.014310917817056179,
+          0.013688703998923302,
+          0.010288232937455177,
+          0.005809740629047155,
+          0.02112632989883423,
+          -0.014846310950815678,
+          0.008428826928138733,
+          0.012089759111404419,
+          0.0034782474394887686,
+          -0.009427262470126152,
+          -0.00600146921351552,
+          -0.014477323740720749,
+          0.015873687341809273,
+          0.008276890963315964,
+          -0.005802505649626255,
+          -0.01611967757344246,
+          0.0034438809379935265,
+          -0.014918660745024681,
+          -0.010411228984594345,
+          -0.006714120972901583,
+          -0.0021994535345584154,
+          -0.012082523666322231,
+          0.004927065223455429,
+          -0.009745604358613491,
+          -0.007090343162417412,
+          0.0033896181266754866,
+          -0.004171003121882677,
+          -0.005140498746186495,
+          -0.02685648389160633,
+          -0.026914363726973534,
+          -0.003196080680936575,
+          -0.0024038434494286776,
+          -0.00600146921351552,
+          -0.003590390784665942,
+          -0.029359810054302216,
+          -0.0065404800698161125,
+          0.009022099897265434,
+          -0.011851002462208271,
+          0.021068448200821877,
+          0.0029573242645710707,
+          -0.008906339295208454,
+          0.0033896181266754866,
+          -0.003604860743507743,
+          -0.029866263270378113,
+          0.009977125562727451,
+          0.005639716982841492,
+          0.02208135463297367,
+          -0.0182033721357584,
+          0.0015962315956130624,
+          0.0061497874557971954,
+          0.023759884759783745,
+          -0.031718432903289795,
+          -0.015902627259492874,
+          -0.007654676679521799,
+          -0.010975562036037445,
+          -0.006674328353255987,
+          -0.014354327693581581,
+          0.005060913506895304,
+          -0.008117719553411007,
+          -0.025423945859074593,
+          -0.02281932905316353,
+          -0.00650068698450923,
+          -0.008038134314119816,
+          0.022124765440821648,
+          1.9359395082574338e-05,
+          0.03684084489941597,
+          0.008660348132252693,
+          0.003986509516835213,
+          0.010852565988898277,
+          -0.0030133959371596575,
+          -0.007546151056885719,
+          0.011851002462208271,
+          -0.017581157386302948,
+          -0.006873291917145252,
+          0.003925011493265629,
+          0.01943332888185978,
+          -0.007654676679521799,
+          -0.002360433107241988,
+          0.011040677316486835,
+          0.016857653856277466,
+          0.008776108734309673,
+          0.0027438905090093613,
+          -0.007289307191967964,
+          -0.005610776599496603,
+          0.02215370535850525,
+          0.027073536068201065,
+          0.004728101193904877,
+          0.003436645958572626,
+          0.007806612644344568,
+          -0.002843372290953994,
+          -0.006674328353255987,
+          0.001284220372326672,
+          -0.006337898783385754,
+          -0.02085139788687229,
+          -0.0038635137025266886,
+          -0.004225265700370073,
+          -0.0025720582343637943,
+          -0.02205241471529007,
+          -0.01661166176199913,
+          -0.014875250868499279,
+          0.008956984616816044,
+          0.014470088295638561,
+          -0.019578030332922935,
+          0.01211869902908802,
+          0.006062967237085104,
+          0.006569419987499714,
+          -0.006435571704059839,
+          0.01753774657845497,
+          0.042223718017339706,
+          0.00023073007469065487,
+          -0.03906923905014992,
+          -0.01707470417022705,
+          0.0009111633989959955,
+          -0.012494921684265137,
+          -0.022935090586543083,
+          -0.00756062101572752,
+          0.0031617144122719765,
+          -0.024411039426922798,
+          0.023398132994771004,
+          -0.001627884921617806,
+          -0.0009658784256316721,
+          0.004330174066126347,
+          0.017060235142707825,
+          -0.002376711927354336,
+          0.005332227796316147,
+          -0.014535203576087952,
+          0.003442072309553623,
+          -0.01417345181107521,
+          0.010548694059252739,
+          -0.016857653856277466,
+          -0.018984755501151085,
+          0.006804558914154768,
+          -0.002042091218754649,
+          0.008920809254050255,
+          0.022341815754771233,
+          0.02652367204427719,
+          0.01028099749237299,
+          -0.027869390323758125,
+          -0.01523700263351202,
+          -0.011807592585682869,
+          0.0026950540486723185,
+          -0.014021515846252441,
+          -0.007321864832192659,
+          -0.00761850131675601,
+          0.003545171581208706,
+          0.015323823317885399,
+          0.015251473523676395,
+          0.005151351448148489,
+          0.030647646635770798,
+          -3.1540272175334394e-05,
+          0.006804558914154768,
+          -0.003615713445469737,
+          -0.02801409177482128,
+          -0.005939971189945936,
+          0.005708449985831976,
+          0.009991596452891827,
+          -0.013254600577056408,
+          0.007936843670904636,
+          0.014209627173841,
+          -0.016872122883796692,
+          -0.011279433965682983,
+          -0.02716035582125187,
+          0.010440168902277946,
+          -0.002913914155215025,
+          0.03093704953789711,
+          0.020070012658834457,
+          -0.02138679102063179,
+          -0.015439583919942379,
+          -0.02112632989883423,
+          0.018984755501151085,
+          -0.010953856632113457,
+          -0.0019154779147356749,
+          0.007973019033670425,
+          0.019983192905783653,
+          -0.007712556980550289,
+          -0.009094450622797012,
+          -0.0077921426855027676,
+          0.013876814395189285,
+          0.018753234297037125,
+          7.57983943913132e-05,
+          0.0010300894500687718,
+          -0.0096515491604805,
+          0.010780216194689274,
+          -0.009977125562727451,
+          -0.010592104867100716,
+          0.0051115588285028934,
+          -0.004529137630015612,
+          0.019968722015619278,
+          -0.01638014055788517,
+          0.022529928013682365,
+          0.010360583662986755,
+          -0.009202975779771805,
+          -0.0398506224155426,
+          -0.006522392388433218,
+          -0.006005086470395327,
+          -0.001984210917726159,
+          0.011995702981948853,
+          -0.0027131414972245693,
+          -0.01771138794720173,
+          0.008971454575657845,
+          0.000733000400941819,
+          -0.003031483618542552,
+          0.007604031357914209,
+          0.008096014149487019,
+          0.0007470183190889657,
+          0.011684596538543701,
+          -0.005046443548053503,
+          -0.009702194482088089,
+          0.02195112407207489,
+          0.0014090248150750995,
+          -7.969853322720155e-05,
+          0.02002660185098648,
+          0.02251545712351799,
+          0.010982797481119633,
+          -0.00988307036459446,
+          0.0034547336399555206,
+          -0.009825189597904682,
+          -0.007538916077464819,
+          -0.030850227922201157,
+          0.027579989284276962,
+          -0.007929608225822449,
+          0.03258663788437843,
+          0.013232896104454994,
+          -0.004037154838442802,
+          -0.00075018365168944,
+          0.0074593303725123405,
+          -0.005458840634673834,
+          -0.0006244747783057392,
+          0.017450926825404167,
+          -0.0009233725140802562,
+          0.006261930800974369,
+          -0.0032340646721422672,
+          -0.02028706483542919,
+          0.025915928184986115,
+          -0.0022754215169698,
+          0.005220084451138973,
+          0.011966763064265251,
+          0.009087215177714825,
+          0.007057785522192717,
+          -0.01607626862823963,
+          -0.0028071971610188484,
+          -0.03608839958906174,
+          -0.008016428910195827,
+          -0.0021108242217451334,
+          -0.005697597283869982,
+          -0.003261196194216609,
+          -0.0070831081829965115,
+          -0.04691202566027641,
+          -0.004189090803265572,
+          0.008544587530195713,
+          -0.009868600405752659,
+          0.01853618398308754,
+          -0.016828712075948715,
+          -0.017957380041480064,
+          -0.02115526981651783,
+          0.007195251528173685,
+          0.02002660185098648,
+          0.013587413355708122,
+          -0.003053188556805253,
+          -0.017986319959163666,
+          0.018550653010606766,
+          -0.011800357140600681,
+          -0.008211774751543999,
+          -0.01635120064020157,
+          -0.020055541768670082,
+          0.014549673534929752,
+          0.016105208545923233,
+          -0.011908882297575474,
+          -0.0032431085128337145,
+          -0.007604031357914209,
+          0.013797229155898094,
+          -0.01720493473112583,
+          -0.008327536284923553,
+          0.008096014149487019,
+          0.0018738764338195324,
+          0.012342985719442368,
+          -0.019172867760062218,
+          0.007108430843800306,
+          -0.015801336616277695,
+          0.009043805301189423,
+          -0.01789950020611286,
+          -0.013218426145613194,
+          0.002179556991904974,
+          0.0010237587848678231,
+          0.039705920964479446,
+          0.022920619696378708,
+          -0.027406347915530205,
+          0.01846383325755596,
+          0.004572547972202301,
+          0.016336729750037193,
+          -0.0031038338784128428,
+          0.01467266958206892,
+          -0.01720493473112583,
+          0.016655072569847107,
+          -0.007216956466436386,
+          0.01761009730398655,
+          0.003404088318347931,
+          0.03293392062187195,
+          -0.017566686496138573,
+          0.010432933457195759,
+          0.0015681958757340908,
+          -0.004634045995771885,
+          -0.00705416826531291,
+          0.016785303130745888,
+          0.010411228984594345,
+          0.014520733617246151,
+          0.00616787513718009,
+          -0.023600714281201363,
+          -0.009608138352632523,
+          0.004091417416930199,
+          0.020475175231695175,
+          0.014781195670366287,
+          -0.00349814398214221,
+          -0.0036573149263858795,
+          0.002850607503205538,
+          -0.01028099749237299,
+          0.0096660191193223,
+          -0.013334186747670174,
+          0.007596796378493309,
+          -0.008493942208588123,
+          0.0077921426855027676,
+          -0.008407121524214745,
+          0.0007913329754956067,
+          0.021473610773682594,
+          -0.03510443493723869,
+          0.01510677207261324,
+          -0.003747752867639065,
+          -0.014231331646442413,
+          0.027044596150517464,
+          -0.017393046990036964,
+          -0.005845915526151657,
+          -0.008964220061898232,
+          -0.0008143946761265397,
+          0.014216861687600613,
+          -0.000501479022204876,
+          0.0035704942420125008,
+          -0.006768384017050266,
+          -0.006949259899556637,
+          0.001438869396224618,
+          0.0091016860678792,
+          -0.011851002462208271,
+          0.006677945610135794,
+          -0.004026302136480808,
+          0.021444670855998993,
+          0.008732697926461697,
+          0.006938407197594643,
+          0.00020653790852520615,
+          -0.002971794456243515,
+          0.006062967237085104,
+          0.02182089351117611,
+          0.0048149218782782555,
+          0.023166611790657043,
+          -0.023658594116568565,
+          0.013486122712492943,
+          -0.00016154497279785573,
+          -0.008096014149487019,
+          -0.01694447360932827,
+          -0.0017870558658614755,
+          0.02543841488659382,
+          0.0020475175697356462,
+          -0.02400587685406208,
+          0.00033281202195212245,
+          0.010143532417714596,
+          -0.0014524351572617888,
+          0.01500548142939806,
+          0.01784161850810051,
+          -0.0014343474758788943,
+          0.007282072212547064,
+          -0.002738464158028364,
+          -0.024092696607112885,
+          -0.01807313971221447,
+          -0.003783928230404854,
+          -0.016785303130745888,
+          0.02271803840994835,
+          0.01691553369164467,
+          0.0051694391295313835,
+          -0.0038526610005646944,
+          0.0017517850501462817,
+          0.01522253267467022,
+          0.020041072741150856,
+          0.0056288642808794975,
+          0.00939832255244255,
+          0.0011594158131629229,
+          -0.010309938341379166,
+          0.0016504944069311023,
+          -0.008052604272961617,
+          -0.008797814138233662,
+          -0.012357455678284168,
+          0.009275326505303383,
+          -0.006240225397050381,
+          -0.013449947349727154,
+          -0.01437603309750557,
+          -0.004619575571268797,
+          0.016090737655758858,
+          -0.007546151056885719,
+          -0.026870954781770706,
+          0.0036265659146010876,
+          0.018377011641860008,
+          0.011612245813012123,
+          0.015642166137695312,
+          -0.009087215177714825,
+          -0.011185378767549992,
+          0.007929608225822449,
+          -0.019476739689707756,
+          -0.0029193402733653784,
+          -0.010729570873081684,
+          0.003085746429860592,
+          -0.021545961499214172,
+          0.02937427908182144,
+          0.008472236804664135,
+          0.027739159762859344,
+          0.024729380384087563,
+          -0.01150372065603733,
+          0.004478492308408022,
+          -0.003693490056321025,
+          -0.0002373998868279159,
+          -0.008956984616816044,
+          3.804050720646046e-05,
+          0.0006158831529319286,
+          0.021545961499214172,
+          0.013175015337765217,
+          0.01245151087641716,
+          0.001984210917726159,
+          -0.0009134243591688573,
+          0.002418313641101122,
+          -7.246348832268268e-05,
+          -0.010042241774499416,
+          -0.017856089398264885,
+          0.005277964752167463,
+          0.017002353444695473,
+          -0.004178238101303577,
+          -0.005639716982841492,
+          0.009969891048967838,
+          0.0003594912705011666,
+          -0.0029555156361311674,
+          -0.011612245813012123,
+          -0.011858237907290459,
+          0.018159961327910423,
+          0.026509201154112816,
+          -0.008609702810645103,
+          -0.006547715049237013,
+          0.02115526981651783,
+          -0.001689382828772068,
+          0.03215253725647926,
+          -0.011098558083176613,
+          -0.010621044784784317,
+          -0.0014596701366826892,
+          -0.005212849471718073,
+          -0.01730622537434101,
+          -0.0238756462931633,
+          0.01255280151963234,
+          0.0035397454630583525,
+          -0.0035361277405172586,
+          -0.03047400526702404,
+          0.015034422278404236,
+          0.006135317496955395,
+          0.032876040786504745,
+          -0.001438869396224618,
+          -0.006460894364863634,
+          -0.000922015926335007,
+          -0.0010653602657839656,
+          -0.0013864153297618032,
+          -0.013811699114739895,
+          0.00905104074627161,
+          -0.0032684311736375093,
+          0.0034746299497783184,
+          -0.0020312387496232986,
+          -0.025554176419973373,
+          -0.005693979561328888,
+          0.002465341240167618,
+          -0.00727121951058507,
+          -0.00756062101572752,
+          -0.013131605461239815,
+          -0.0012145830551162362,
+          0.017335165292024612,
+          0.004416994284838438,
+          -0.0010997267672792077,
+          0.008674818091094494,
+          -0.0077559673227369785,
+          -0.008493942208588123,
+          -0.0060665844939649105,
+          -0.01720493473112583,
+          0.01830466091632843,
+          -0.02105397917330265,
+          0.0017563069704920053,
+          0.026350030675530434,
+          -0.004218030720949173,
+          -0.003295562695711851,
+          0.018695354461669922,
+          0.0091016860678792,
+          0.008587997406721115,
+          -0.0006027696072123945,
+          -0.002096354030072689,
+          0.00361028709448874,
+          0.01177141722291708,
+          0.011590540409088135,
+          -0.002790918340906501,
+          -0.00232968432828784,
+          -0.02089480683207512,
+          0.008602467365562916,
+          -0.01889793574810028,
+          0.024657031521201134,
+          0.02061987668275833,
+          -0.0015374468639492989,
+          0.01187270786613226,
+          0.02646579220890999,
+          0.008349240757524967,
+          0.01488972082734108,
+          0.000975826580543071,
+          0.005737389903515577,
+          -0.0025431180838495493,
+          -0.000960452132858336,
+          -0.023253431543707848,
+          0.006135317496955395,
+          0.005831445567309856,
+          0.004308468662202358,
+          -0.0035940082743763924,
+          -0.01976614072918892,
+          0.017421986907720566,
+          0.014151746407151222,
+          -0.018217841163277626,
+          0.021936653181910515,
+          -0.005737389903515577,
+          0.019838491454720497,
+          0.017103644087910652,
+          0.011959527619183064,
+          -0.007311012130230665,
+          -0.014166216365993023,
+          0.013558472506701946,
+          -0.02085139788687229,
+          -0.0024092698004096746,
+          0.005632482003420591,
+          -0.01727728545665741,
+          0.002420122269541025,
+          -0.02025812305510044,
+          -0.004076947458088398,
+          -0.008334770798683167,
+          0.006956494878977537,
+          -0.008327536284923553,
+          0.00628363573923707,
+          0.00583506328985095,
+          -0.0022844651248306036,
+          -0.001457861391827464,
+          -0.03637780249118805,
+          -0.005845915526151657,
+          0.021777482703328133,
+          0.0026028072461485863,
+          -0.04057412967085838,
+          0.021097388118505478,
+          0.004811304621398449,
+          -0.01684318296611309,
+          -0.015280413441359997,
+          -0.021936653181910515,
+          -0.014064925722777843,
+          -0.007625736761838198,
+          0.0037405178882181644,
+          -0.02102503925561905,
+          -0.0029048703145235777,
+          0.011807592585682869,
+          -0.01902816630899906,
+          -0.0102158822119236,
+          -0.008124954998493195,
+          -0.0031056427396833897,
+          0.019086046144366264,
+          -0.011923353187739849,
+          -0.00951408315449953,
+          0.025814637541770935,
+          0.006916702259331942,
+          -0.0058061229065060616,
+          -0.015323823317885399,
+          -0.016134148463606834,
+          -0.0071409884840250015,
+          0.013804464600980282,
+          -0.0025467355735599995,
+          -0.003921393770724535,
+          -0.030155664309859276,
+          -0.0028705038130283356,
+          0.006949259899556637,
+          0.019476739689707756,
+          -0.00043726799776777625,
+          0.002132529392838478,
+          0.009072745218873024,
+          0.00794407818466425,
+          -0.014535203576087952,
+          -7.447008556482615e-06,
+          0.009072745218873024,
+          0.012494921684265137,
+          -0.00015498821449000388,
+          0.019722729921340942,
+          -0.01833360269665718,
+          0.004937917459756136,
+          -0.008718227967619896,
+          0.005220084451138973,
+          -0.01860853284597397,
+          -0.006439189426600933,
+          -0.01627884991466999,
+          0.008052604272961617,
+          0.009275326505303383,
+          -0.018319131806492805,
+          -0.0019172866595909,
+          -0.006866056937724352,
+          0.017856089398264885,
+          -0.027174826711416245,
+          0.001350240083411336,
+          -0.017494337633252144,
+          -0.0015817615203559399,
+          -0.026350030675530434,
+          0.00311287771910429,
+          0.006471747066825628,
+          0.004561695270240307,
+          0.0003723786794580519,
+          -0.007850022986531258,
+          -0.02457020990550518,
+          0.01198846846818924,
+          -0.002181365853175521,
+          -0.001573622110299766,
+          -0.010526989586651325,
+          -0.0017327930545434356,
+          0.011966763064265251,
+          0.016163088381290436,
+          -0.0042361184023320675,
+          -0.014665434136986732,
+          0.01843489333987236,
+          -0.016105208545923233,
+          -0.019549088552594185,
+          -0.015179122798144817,
+          -0.011185378767549992,
+          -0.004644898232072592,
+          0.012639622204005718,
+          0.013804464600980282,
+          0.020779047161340714,
+          0.00961537379771471,
+          0.010765746235847473,
+          -0.007358039729297161,
+          0.00016572773165535182,
+          -0.006638152990490198,
+          -0.0015085067134350538,
+          0.012574506923556328,
+          -0.0054299007169902325,
+          0.02613298036158085,
+          -0.015410644002258778,
+          0.0015274987090379,
+          -0.00025797454873099923,
+          0.0069637298583984375,
+          -0.007354422472417355,
+          0.00015397078823298216,
+          -0.010411228984594345,
+          0.008392651565372944,
+          -0.03455457091331482,
+          -0.006258313078433275,
+          -0.0033263114746659994,
+          -0.011185378767549992,
+          0.03157373145222664,
+          0.0003567781241144985,
+          0.00756062101572752,
+          0.0048149218782782555,
+          0.011395194567739964,
+          -0.008298595435917377,
+          0.016061797738075256,
+          -0.017783738672733307,
+          0.001122336252592504,
+          0.015526404604315758,
+          0.007770437281578779,
+          -0.0012932640966027975,
+          0.006775618996471167,
+          -0.006041261833161116,
+          0.009123390540480614,
+          -0.0014108336763456464,
+          -0.014361563138663769,
+          -0.01450626365840435,
+          0.00017024963744916022,
+          0.013146075420081615,
+          -0.01727728545665741,
+          -0.006670710630714893,
+          -0.006511539686471224,
+          0.01394193060696125,
+          -0.01409386657178402,
+          0.026914363726973534,
+          -0.001209156820550561,
+          -0.017465395852923393,
+          0.01238639559596777,
+          0.030589766800403595,
+          -0.014383267611265182,
+          0.02059093490242958,
+          -0.007546151056885719,
+          -0.003906923811882734,
+          0.004988562781363726,
+          -0.0015482994494959712,
+          -0.0004268675984349102,
+          0.018217841163277626,
+          -0.008204540237784386,
+          -0.007777672726660967,
+          0.011207083240151405,
+          -0.02218264527618885,
+          0.016090737655758858,
+          -0.01584474742412567,
+          0.0010300894500687718,
+          0.004858332220464945,
+          0.0031110690906643867,
+          0.02028706483542919,
+          0.037622228264808655,
+          -0.00014944888243917376,
+          -0.007878962904214859,
+          -0.012234459631145,
+          0.02089480683207512,
+          0.005871238186955452,
+          0.005853150971233845,
+          -0.004388054367154837,
+          0.002447253791615367,
+          -0.01740751601755619,
+          -0.013681468553841114,
+          0.0076691466383636,
+          -0.010606574825942516,
+          -0.03313650190830231,
+          -0.015077832154929638,
+          0.0051115588285028934,
+          0.0063993968069553375,
+          0.018579592928290367,
+          0.006204050499945879,
+          -0.003986509516835213,
+          0.020967157557606697,
+          -0.003581346943974495,
+          0.011496485210955143,
+          -0.008414356037974358,
+          -0.002881356282159686,
+          -0.004876419901847839,
+          -0.023354722186923027,
+          0.024729380384087563,
+          -0.027637869119644165,
+          -0.003433028468862176,
+          -0.011337313801050186,
+          -0.000320150691550225,
+          0.005914648529142141,
+          -0.0031291565392166376,
+          0.01470884494483471,
+          0.007466565817594528,
+          0.016958944499492645,
+          0.01737857609987259,
+          -0.03166055306792259,
+          0.01707470417022705,
+          0.019303098320961,
+          -0.0020149596966803074,
+          -0.007538916077464819,
+          -0.0032449173741042614,
+          -0.012017408385872841,
+          -0.012762618251144886,
+          0.014296446926891804,
+          -0.0004528685531113297,
+          -0.006073819473385811,
+          -0.005654186941683292,
+          0.02287720888853073,
+          -0.007784907706081867,
+          0.02351389452815056,
+          -0.0046955435536801815,
+          0.0027800656389445066,
+          0.02652367204427719,
+          0.007325482089072466,
+          0.020417295396327972,
+          -0.016452491283416748,
+          -0.02271803840994835,
+          -0.007159076165407896,
+          -0.005249024368822575,
+          0.006916702259331942,
+          -0.005715684965252876,
+          0.00944896787405014,
+          -0.021647252142429352,
+          -0.020402824506163597,
+          0.01850724220275879,
+          -0.0199253112077713,
+          0.010736805386841297,
+          0.0061027598567306995,
+          0.01899922639131546,
+          -0.004000979475677013,
+          0.012531096115708351,
+          -0.010382288135588169,
+          -0.01604732871055603,
+          0.011424134485423565,
+          0.0062221381813287735,
+          -0.006652622949331999,
+          0.003928629215806723,
+          -0.011192613281309605,
+          -0.008790578693151474,
+          -0.01638014055788517,
+          -0.0010120017686858773,
+          -0.010961092077195644,
+          0.008161129429936409,
+          0.01787055842578411,
+          0.004959622863680124,
+          0.014745020307600498,
+          0.0032412998843938112,
+          -0.01250215619802475,
+          -0.010273762978613377,
+          0.0035940082743763924,
+          -0.010932152159512043,
+          -0.003747752867639065,
+          -0.022920619696378708,
+          0.0007212434429675341,
+          -0.014361563138663769,
+          -0.01175694726407528,
+          -0.009289796464145184,
+          0.0048510972410440445,
+          0.015497464686632156,
+          -0.018825585022568703,
+          -0.008457766845822334,
+          -0.008703758008778095,
+          0.0023170229978859425,
+          0.007057785522192717,
+          -0.016322258859872818,
+          0.008682052604854107,
+          0.02931639924645424,
+          -0.0034402634482830763,
+          -0.014795665629208088,
+          0.004442316945642233,
+          0.005614394322037697,
+          -0.008378181606531143,
+          0.018058670684695244,
+          0.011395194567739964,
+          -0.013826169073581696,
+          -0.004275911021977663,
+          0.013399302028119564,
+          -0.00019625057757366449,
+          -0.0005494111683219671,
+          0.0009396513924002647,
+          -0.014412208460271358,
+          0.03157373145222664,
+          0.0022808476351201534,
+          0.021082919090986252,
+          -8.054639329202473e-05,
+          -0.006601977627724409,
+          -0.006920319981873035,
+          -0.002239246154204011,
+          -0.006894997321069241,
+          -0.0034493072889745235,
+          -0.0011214318219572306,
+          -0.005668656900525093,
+          0.016307789832353592,
+          0.00704693328589201,
+          -0.002043900080025196,
+          0.0006629109266214073,
+          -0.016655072569847107,
+          0.00517305638641119,
+          -0.005249024368822575,
+          -0.005151351448148489,
+          -0.01804419979453087,
+          0.031284332275390625,
+          0.01261791680008173,
+          0.006728590931743383,
+          -0.006370456423610449,
+          -0.020301533862948418,
+          -0.00013983984536025673,
+          0.030358245596289635,
+          -0.011236024089157581,
+          0.004091417416930199,
+          0.01488972082734108,
+          -0.005162204150110483,
+          -0.002150616841390729,
+          -0.014281976968050003,
+          -0.0028053882997483015,
+          -0.027768099680542946,
+          0.0023061702959239483,
+          -0.006775618996471167,
+          0.006323428824543953,
+          -0.002731229178607464,
+          -0.013630823232233524,
+          -0.015902627259492874,
+          -0.0204607043415308,
+          -0.0062438431195914745,
+          -0.009275326505303383,
+          -0.009702194482088089,
+          -0.010411228984594345,
+          -0.025062192231416702,
+          0.021314440295100212,
+          -0.011843767017126083,
+          -0.0004286763723939657,
+          -0.024599149823188782,
+          0.0182033721357584,
+          -0.01034611277282238,
+          -0.015642166137695312,
+          0.0066453879699110985,
+          0.009789015166461468,
+          0.016192028298974037,
+          0.006124464794993401,
+          -0.004977710545063019,
+          -0.004181855358183384,
+          0.014064925722777843,
+          -0.0066236830316483974,
+          -0.005328610073775053,
+          -0.002212114864960313,
+          -0.021893244236707687,
+          -0.007202486507594585,
+          -0.010367818176746368,
+          -0.01177141722291708,
+          -0.005853150971233845,
+          0.010801920667290688,
+          0.015150182880461216,
+          0.0013104473473504186,
+          -0.01761009730398655,
+          -0.0054877810180187225,
+          0.0034113232977688313,
+          0.010085651651024818,
+          -0.023195551708340645,
+          0.008378181606531143,
+          -0.008356476202607155,
+          0.013732113875448704,
+          0.030792348086833954,
+          0.00577718298882246,
+          -0.004720866214483976,
+          0.0008682053303346038,
+          0.02102503925561905,
+          0.0030423360876739025,
+          0.00938385259360075,
+          -0.030358245596289635,
+          -0.022906150668859482,
+          -0.003950334154069424,
+          0.004923447500914335,
+          -0.005932736210525036,
+          0.030358245596289635,
+          0.008573527447879314,
+          0.01840595155954361,
+          0.006612830329686403,
+          0.023484952747821808,
+          0.006620065309107304,
+          0.022558867931365967,
+          0.011568835936486721,
+          0.003633800894021988,
+          -0.007517211139202118,
+          -0.024251868948340416,
+          -0.02944662980735302,
+          0.007690852042287588,
+          0.016568250954151154,
+          0.02115526981651783,
+          -0.03553853556513786,
+          0.023600714281201363,
+          -0.0015284031396731734,
+          0.0003922750474885106,
+          -0.014679905027151108,
+          0.007929608225822449,
+          0.005599924363195896,
+          0.022139234468340874,
+          0.0003649175341706723,
+          0.006775618996471167,
+          0.01039675809442997,
+          0.024512330070137978,
+          -0.017798209562897682,
+          0.01902816630899906,
+          0.017335165292024612,
+          0.0329628624022007,
+          0.017725858837366104,
+          -0.010461874306201935,
+          0.02095268853008747,
+          0.014202391728758812,
+          -0.004351879004389048,
+          -0.011807592585682869,
+          0.0037658405490219593,
+          -0.007799377664923668,
+          0.006319811102002859,
+          0.006493452005088329,
+          0.004464022349566221,
+          -0.010375053621828556,
+          0.007368892431259155,
+          -0.00728568946942687,
+          0.005216466728597879,
+          0.003527084132656455,
+          0.012096993625164032,
+          0.0013782759197056293,
+          0.02785491943359375,
+          -0.04306298494338989,
+          0.02161831222474575,
+          -0.02652367204427719,
+          -0.013529532589018345,
+          -0.005382873117923737,
+          0.005158586427569389,
+          0.0030748937278985977,
+          -0.009904775768518448,
+          -0.011532660573720932,
+          -0.01472331490367651,
+          0.014846310950815678,
+          0.0198674313724041,
+          0.0018286574631929398,
+          -0.0071120485663414,
+          0.006938407197594643,
+          -0.00183227495290339,
+          -0.01747986674308777,
+          0.0019010078394785523,
+          -0.008124954998493195,
+          -0.011236024089157581,
+          -0.02208135463297367,
+          0.013059254735708237,
+          0.0027420816477388144,
+          0.006710503716021776,
+          0.00017431934247724712,
+          -0.025858048349618912,
+          -0.01399257592856884,
+          0.009875834919512272,
+          0.0010328025091439486,
+          -0.013102664612233639,
+          -0.013276305980980396,
+          0.006750296335667372,
+          -0.0027583607006818056,
+          0.003986509516835213,
+          0.011199848726391792,
+          0.0102158822119236,
+          -0.006641770713031292,
+          0.0036247570533305407,
+          0.01910051703453064,
+          0.005267112050205469,
+          -0.016452491283416748,
+          0.01648143120110035,
+          -0.016061797738075256,
+          0.0020312387496232986,
+          0.00035768249654211104,
+          -0.02028706483542919,
+          -0.011973998509347439,
+          0.008182834833860397,
+          -0.009079980663955212,
+          -0.0016794346738606691,
+          0.016061797738075256,
+          0.011735241860151291,
+          0.019375449046492577,
+          0.022037943825125694,
+          0.004084182437509298,
+          0.016452491283416748,
+          -0.02310873195528984,
+          0.014130041003227234,
+          -0.021878773346543312,
+          -0.010888741351664066,
+          0.01753774657845497,
+          0.007032462861388922,
+          0.006573037710040808,
+          -0.0012263399548828602,
+          -0.004275911021977663,
+          -0.0060448795557022095,
+          0.014745020307600498,
+          0.0329628624022007,
+          -0.0022247761953622103,
+          -0.01235022023320198,
+          -0.017364105209708214,
+          -0.00755338603630662,
+          0.03666720539331436,
+          -0.01522253267467022,
+          -0.014578614383935928,
+          0.021213149651885033,
+          0.02742081694304943,
+          0.00489812484011054,
+          -0.009593668393790722,
+          -0.0063993968069553375,
+          0.0032539612147957087,
+          0.015265943482518196,
+          0.013464417308568954,
+          -0.010353348217904568,
+          -0.003545171581208706,
+          0.006891379598528147,
+          0.02600274793803692,
+          0.00716992886736989,
+          -0.013695938512682915,
+          -0.025351595133543015,
+          0.0063849263824522495,
+          -0.0023369193077087402,
+          0.012661327607929707,
+          0.0018241355428472161,
+          0.028592893853783607,
+          0.02516348287463188,
+          0.0012191049754619598,
+          0.018854524940252304,
+          -0.014238567091524601,
+          0.04338132590055466,
+          -0.019303098320961,
+          -0.00472448393702507,
+          -0.02801409177482128,
+          0.0010409420356154442,
+          -0.008291360922157764,
+          0.007980253547430038,
+          -0.006544097326695919,
+          0.002765595680102706,
+          -0.020750107243657112,
+          0.01979508064687252,
+          0.02079351618885994,
+          -0.023730944842100143,
+          0.006739443633705378,
+          -0.024859612807631493,
+          0.01417345181107521,
+          -0.029258519411087036,
+          -0.00961537379771471,
+          -0.017190465703606606,
+          -0.006269165780395269,
+          -0.025886988267302513,
+          0.0021433818619698286,
+          0.0009595477604307234,
+          0.02238522656261921,
+          -0.0051332637667655945,
+          0.003635609755292535,
+          0.01187270786613226,
+          -0.01437603309750557,
+          -0.011981233023107052,
+          0.019245218485593796,
+          0.02559758722782135,
+          0.007929608225822449,
+          0.018319131806492805,
+          0.02089480683207512,
+          -0.01440497301518917,
+          -0.04427846893668175,
+          0.0009749222081154585,
+          -0.003903306322172284,
+          -0.001428921241313219,
+          0.004565312992781401,
+          0.012972434051334858,
+          -0.02593039907515049,
+          -0.001539255608804524,
+          -0.004160150419920683,
+          0.0033823831472545862,
+          -0.004536372609436512,
+          0.015902627259492874,
+          0.025481825694441795,
+          -0.013218426145613194,
+          -0.013804464600980282,
+          -0.011366254650056362,
+          -0.005361167713999748,
+          0.008436061441898346,
+          -0.015396174043416977,
+          0.0008849363657645881,
+          -0.0034891001414507627,
+          -0.015511934645473957,
+          -0.00976730976253748,
+          0.002360433107241988,
+          -0.012943494133651257,
+          -0.008566292002797127,
+          0.006844351999461651,
+          -0.027637869119644165,
+          -0.01640908047556877,
+          0.019158396869897842,
+          -0.015164652839303017,
+          0.005118793807923794,
+          0.013746583834290504,
+          0.008812284097075462,
+          0.011149203404784203,
+          0.030097784474492073,
+          -0.00984689500182867,
+          0.0038562784902751446,
+          -0.021763013675808907,
+          -0.016770832240581512,
+          -0.0096804890781641,
+          0.01797184906899929,
+          0.003168949391692877,
+          -0.011829297058284283,
+          0.007647441700100899,
+          -0.012379161082208157,
+          -0.013232896104454994,
+          -0.006949259899556637,
+          0.001656825072132051,
+          -0.008645878173410892,
+          0.015265943482518196,
+          -0.01039675809442997,
+          -0.004131210036575794,
+          0.012075289152562618,
+          0.036696143448352814,
+          -0.0035415540914982557,
+          -0.00988307036459446,
+          -0.0014334431616589427,
+          0.01501995138823986,
+          0.008602467365562916,
+          -0.004351879004389048,
+          -0.020634345710277557,
+          -0.015700045973062515,
+          -0.0022302023135125637,
+          -0.010780216194689274,
+          -0.015627695247530937,
+          0.02271803840994835,
+          -0.021632781252264977,
+          -0.018854524940252304,
+          -0.018782174214720726,
+          -0.0158881563693285,
+          0.00048113043885678053,
+          0.03270240128040314,
+          0.0147739602252841,
+          0.0024780025705695152,
+          0.0038020156789571047,
+          -0.010056711733341217,
+          -0.027478698641061783,
+          0.024584680795669556,
+          0.011380724608898163,
+          -0.030097784474492073,
+          -0.009275326505303383,
+          -0.0187966451048851,
+          -0.004066094756126404,
+          0.007821083068847656,
+          -0.0022591424640268087,
+          0.05417601019144058,
+          -0.0007131040329113603,
+          0.006478982046246529,
+          -0.026682842522859573,
+          0.008298595435917377,
+          -0.015555345453321934,
+          -0.02192218415439129,
+          -0.010990031994879246,
+          -0.012299574911594391,
+          -0.009470673277974129,
+          -0.016336729750037193,
+          -0.002657070057466626,
+          -0.011438604444265366,
+          0.006471747066825628,
+          -0.015092302113771439,
+          -0.0034456897992640734,
+          -0.02407822757959366,
+          -0.004507432691752911,
+          0.005683127325028181,
+          -0.0007497314363718033,
+          0.008964220061898232,
+          -0.01863747462630272,
+          -0.006486217025667429,
+          0.0014054073253646493,
+          0.010794686153531075,
+          0.007886198349297047,
+          -0.0038490435108542442,
+          -0.014115571044385433,
+          -0.013167780824005604,
+          -0.008616937324404716,
+          0.01001330092549324,
+          -0.00699628796428442,
+          0.008096014149487019,
+          -0.01664060167968273,
+          -0.014643729664385319,
+          0.004062477499246597,
+          0.03328120335936546,
+          0.01490419078618288,
+          -0.009065510705113411,
+          -0.00961537379771471,
+          -0.014296446926891804,
+          0.034004706889390945,
+          -0.012777088209986687,
+          -0.014585848897695541,
+          0.012625152245163918,
+          0.022443106397986412,
+          -0.0007632971392013133,
+          0.0045508430339396,
+          0.010440168902277946,
+          -0.00010264718730468303,
+          -0.0007316438714042306,
+          0.0005679509486071765,
+          0.005328610073775053,
+          0.00020608570775948465,
+          0.028144322335720062,
+          -0.009065510705113411,
+          0.014419442974030972,
+          -0.004594252910465002,
+          0.006887761875987053,
+          -0.028100911527872086,
+          -0.0024906639009714127,
+          -0.006522392388433218,
+          -0.005965293850749731,
+          0.004771511536091566,
+          -0.0015076022827997804,
+          0.01774032786488533,
+          -0.03044506534934044,
+          -0.027305057272315025,
+          0.016669541597366333,
+          -0.02112632989883423,
+          -0.007958549074828625,
+          0.005433517973870039,
+          -0.014072161167860031,
+          0.008710993453860283,
+          0.006507922429591417,
+          -0.022327346727252007,
+          -0.01607626862823963,
+          0.0019588882569223642,
+          -0.0023206404875963926,
+          0.0027221853379160166,
+          0.01661166176199913,
+          -0.010548694059252739,
+          -0.012545567005872726,
+          -0.00761850131675601,
+          0.012914554215967655,
+          -0.005212849471718073,
+          0.024165047332644463,
+          -0.013261836022138596,
+          -0.0005720206536352634,
+          0.0009541214676573873,
+          -0.005976146552711725,
+          0.015613225288689137,
+          0.015468524768948555,
+          -0.0035071875900030136,
+          -0.0010002448689192533,
+          0.0003527084190864116,
+          -0.006847969256341457,
+          0.011033442802727222,
+          0.010635514743626118,
+          0.01460755430161953,
+          -0.004102270118892193,
+          0.012531096115708351,
+          -0.004297616425901651,
+          0.0027203767094761133,
+          -0.02079351618885994,
+          0.0040588597767055035,
+          0.002065605018287897,
+          0.018232312053442,
+          -0.020214714109897614,
+          -0.011800357140600681,
+          -0.031110690906643867,
+          -0.004529137630015612,
+          -0.01691553369164467,
+          0.0022012621629983187,
+          0.030155664309859276,
+          0.0015817615203559399,
+          -0.010541459545493126,
+          0.014629259705543518,
+          0.031226450577378273,
+          0.009506847709417343,
+          0.0004411115951370448,
+          0.010830861516296864,
+          -0.011532660573720932,
+          0.010592104867100716,
+          0.012480450794100761,
+          -0.01846383325755596,
+          -0.0015636739553883672,
+          -0.0007429486140608788,
+          -0.003494526259601116,
+          0.009825189597904682,
+          0.007517211139202118,
+          -0.004322939086705446,
+          0.017682448029518127,
+          -0.027015654370188713,
+          0.0036591235548257828,
+          -0.0030115870758891106,
+          -0.004681073594838381,
+          -0.027305057272315025,
+          -0.006352368742227554,
+          0.01747986674308777,
+          -0.010519754141569138,
+          0.058574918657541275,
+          0.002271803794428706,
+          0.007922373712062836,
+          -0.027536578476428986,
+          -0.005230937153100967,
+          -0.008906339295208454,
+          0.0034493072889745235,
+          -0.007184398826211691,
+          0.00761126633733511,
+          0.004008214455097914,
+          0.00022835608979221433,
+          -0.0035885819233953953,
+          0.0020583700388669968,
+          0.0022917003370821476,
+          0.016872122883796692,
+          0.00506453076377511,
+          0.03524913638830185,
+          -0.007727027405053377,
+          -0.005151351448148489,
+          -0.011062382720410824,
+          0.005932736210525036,
+          0.019042637199163437,
+          0.0002742533979471773,
+          0.006956494878977537,
+          0.011156437918543816,
+          0.011395194567739964,
+          0.013985340483486652,
+          0.018261251971125603,
+          -0.0033950444776564837,
+          0.00500665046274662,
+          0.01622096821665764,
+          0.00494153518229723,
+          -0.01410833653062582,
+          -0.011127498000860214,
+          -0.004297616425901651,
+          0.018290191888809204,
+          -0.007466565817594528,
+          0.011207083240151405,
+          0.006761149037629366,
+          0.00582782831043005,
+          0.01016523689031601,
+          -0.01178588718175888,
+          -0.00023604331363458186,
+          0.012777088209986687,
+          -0.030097784474492073,
+          0.012574506923556328,
+          0.0029301929753273726,
+          -0.00789343286305666,
+          0.00244906242005527,
+          -0.0076980870217084885,
+          -0.009065510705113411,
+          -0.0020836926996707916,
+          -0.009145095944404602,
+          0.0005069052567705512,
+          0.015251473523676395,
+          -0.012133168987929821,
+          0.0023658594582229853,
+          -0.003762223059311509,
+          -0.017103644087910652,
+          0.02595933899283409,
+          -0.018290191888809204,
+          0.005183909088373184,
+          0.014368797652423382
+        ],
+        "title": "Introduction And Related Work",
+        "keyphrases": [
+          "conditional computation",
+          "model capacity",
+          "Sparsely-Gated Mixture-of-Experts Layer",
+          "language modeling",
+          "machine translation"
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "bec4b8f2-cf26-434f-a587-d63e318fec0c",
+      "properties": {
+        "page_content": "2 The Structure Of The Mixture-Of-Experts Layer\n\nThe Mixture-of-Experts (MoE) layer consists of a set of n \"expert networks\" E1, · · · , En, and a \"gating network\" G whose output is a sparse n-dimensional vector. Figure 1 shows an overview of the MoE module. The experts are themselves neural networks, each with their own parameters.\n\nAlthough in principle we only require that the experts accept the same sized inputs and produce the same-sized outputs, in our initial investigations in this paper, we restrict ourselves to the case where the models are feed-forward networks with identical architectures, but with separate parameters.\n\nLet us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows:\n\n$$y=\\sum_{i=1}^{n}G(x){i}E{i}(x)$$ $$(\\mathbf{l})$$ G(x)iEi(x) (1) We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example. If the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network. In the following we focus on ordinary MoEs. We provide more details on hierarchical MoEs in Appendix B. Our implementation is related to other models of conditional computation. A MoE whose experts are simple weight matrices is similar to the parameterized weight matrix proposed in (Cho & Bengio, 2014). A MoE whose experts have one hidden layer is similar to the block-wise dropout described in (Bengio et al., 2015), where the dropped-out layer is sandwiched between fully-activated layers.\n\n2.1 Gating Network\n\nSoftmax Gating: A simple choice of non-sparse gating function (Jordan & Jacobs, 1994) is to multiply the input by a trainable weight matrix Wg and then apply the Sof tmax function.\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ $\\eqref{eq:walpha}$. Gσ(x) = Sof tmax(x · Wg) (2) Noisy Top-K Gating: We add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to −∞ (which causes the corresponding gate values to equal 0). The sparsity serves to save computation, as described above. While this form of sparsity creates some theoretically scary discontinuities in the output of gating function, we have not yet observed this to be a problem in practice. The noise term helps with load balancing, as will be discussed in Appendix A. The amount of noise per component is controlled by a second trainable weight matrix Wnoise.\n\n$$G(x)=S o f t m a x(K e e p T o p K(H(x),k))$$ $$(4)$$ $$(S)$$ G(x) = Sof tmax(KeepT opK(H(x), k)) (3) $$H(x){i}=(x\\cdot W{g}){i}+S t a n d a r d N o r m a l()\\cdot S o f t p l u s((x\\cdot W{n o i s e}){i})$$ $KeepTopK(v,k){i}=\\begin{cases}v_{i}&\\text{if}v_{i}\\text{is in the top}k\\text{elements of}v.\\ -\\infty&\\text{otherwise.}\\end{cases}$ Training the Gating Network We train the gating network by simple back-propagation, along with the rest of the model. If we choose k > 1, the gate values for the top k experts have nonzero derivatives with respect to the weights of the gating network. This type of occasionally-sensitive behavior is described in (Bengio et al., 2013) with respect to noisy rectifiers. Gradients also backpropagate through the gating network to its inputs. Our method differs here from (Bengio et al., 2015) who use boolean gates and a REINFORCE-style approach to train the gating network.\n\n",
+        "embedding": [
+          0.00986587256193161,
+          -0.03260384127497673,
+          -0.016219932585954666,
+          -0.022328028455376625,
+          0.017736708745360374,
+          0.029624948278069496,
+          -0.004546910524368286,
+          0.026386426761746407,
+          -0.00606368575245142,
+          0.009852208197116852,
+          -0.018788885325193405,
+          -0.02659139782190323,
+          0.0016850214451551437,
+          -0.01340501569211483,
+          -0.004304362926632166,
+          -0.00787083525210619,
+          -0.007932325825095177,
+          0.022382687777280807,
+          -0.00348277622833848,
+          -0.029105691239237785,
+          0.03632062301039696,
+          -0.0069143096916377544,
+          -0.021795107051730156,
+          0.0010393669363111258,
+          0.011690102517604828,
+          0.010815565474331379,
+          0.058430016040802,
+          -0.025744188576936722,
+          -0.05607970058917999,
+          0.010234818793833256,
+          0.020223673433065414,
+          0.003211191389709711,
+          -0.022328028455376625,
+          -0.008465247228741646,
+          -0.07581144571304321,
+          -0.04277033731341362,
+          0.026851026341319084,
+          0.04208710417151451,
+          0.0028746994212269783,
+          0.024746669456362724,
+          0.064114511013031,
+          0.0007968195714056492,
+          -0.00416088430210948,
+          -0.0368945375084877,
+          -0.019854728132486343,
+          -0.012175197713077068,
+          -0.025949159637093544,
+          -0.0034093286376446486,
+          -0.01524974312633276,
+          -0.03134669363498688,
+          4.102061939192936e-05,
+          -0.008957174606621265,
+          0.005318962503224611,
+          0.04080262780189514,
+          -0.05337410047650337,
+          0.022874614223837852,
+          0.01740875653922558,
+          0.021439826115965843,
+          0.03134669363498688,
+          0.02275163307785988,
+          0.04361754283308983,
+          0.00478262547403574,
+          -0.05853933468461037,
+          0.008123630657792091,
+          0.019444787874817848,
+          -0.00873170793056488,
+          0.01642490178346634,
+          0.0034161610528826714,
+          -0.00102143210824579,
+          0.009339784272015095,
+          -0.039490822702646255,
+          0.014552845619618893,
+          -0.02160380221903324,
+          0.0027124318294227123,
+          -0.0009351740591228008,
+          -0.006668346468359232,
+          0.02001870423555374,
+          0.04028337076306343,
+          0.028039850294589996,
+          0.009920531883835793,
+          -0.01302923820912838,
+          0.023366540670394897,
+          0.008595061488449574,
+          0.006244742311537266,
+          0.007324249017983675,
+          -0.02907836250960827,
+          -0.01475781574845314,
+          -0.04747097194194794,
+          -0.010487614199519157,
+          0.006070517934858799,
+          -0.006986049469560385,
+          0.02747959829866886,
+          0.011464636772871017,
+          0.002150473417714238,
+          0.017531737685203552,
+          0.02048330195248127,
+          0.045612581074237823,
+          -0.0037577771581709385,
+          -0.006627352442592382,
+          -0.01876155659556389,
+          -0.0019147582352161407,
+          0.026154128834605217,
+          -0.001622676500119269,
+          0.032795146107673645,
+          0.0015825366135686636,
+          -0.0020411561708897352,
+          0.018816214054822922,
+          0.03131936490535736,
+          0.03260384127497673,
+          0.08685247600078583,
+          -0.0060021947138011456,
+          0.00043107682722620666,
+          -0.019267147406935692,
+          0.03517279401421547,
+          -0.02644108608365059,
+          -0.009086987935006618,
+          -0.04812687635421753,
+          -0.054248638451099396,
+          0.010118668898940086,
+          -0.03875293210148811,
+          0.028039850294589996,
+          0.020893240347504616,
+          -0.023626169189810753,
+          0.0012520230375230312,
+          -0.06941638886928558,
+          0.06351326406002045,
+          0.0015082351164892316,
+          0.007604374550282955,
+          -0.010378297418355942,
+          -0.030854765325784683,
+          0.005377037450671196,
+          0.0025945743545889854,
+          -0.01956777088344097,
+          -0.038889575749635696,
+          0.03028085082769394,
+          0.017299439758062363,
+          0.041540518403053284,
+          0.028504448011517525,
+          -0.005674243438988924,
+          -0.029870912432670593,
+          0.029706936329603195,
+          -0.022778961807489395,
+          -0.007385740056633949,
+          -0.028067179024219513,
+          0.011123020201921463,
+          0.012059048749506474,
+          0.04036535695195198,
+          0.06263872981071472,
+          -0.038670942187309265,
+          0.027834879234433174,
+          -0.04525730013847351,
+          0.04080262780189514,
+          -0.012468988075852394,
+          0.036566589027643204,
+          -0.001954043982550502,
+          0.0030711288563907146,
+          -0.01716279238462448,
+          -0.020578954368829727,
+          -0.014580175280570984,
+          -0.02582617662847042,
+          -0.010754074901342392,
+          -0.028559105470776558,
+          0.005889461375772953,
+          0.049356695264577866,
+          0.01159445010125637,
+          0.05118775740265846,
+          0.009421772323548794,
+          0.019977709278464317,
+          -0.05427596718072891,
+          0.014019925147294998,
+          -0.0017371178837493062,
+          0.008916180580854416,
+          0.04585854709148407,
+          0.008383259177207947,
+          0.0039115045219659805,
+          0.002119727898389101,
+          0.02943364344537258,
+          0.031893279403448105,
+          -0.006326730363070965,
+          0.04028337076306343,
+          -0.03375167027115822,
+          -0.017217451706528664,
+          0.01627459190785885,
+          0.006500954274088144,
+          0.009838543832302094,
+          -0.0024391389451920986,
+          0.03284980356693268,
+          -0.01389694306999445,
+          0.005595671944320202,
+          -0.00036745081888511777,
+          0.042743004858493805,
+          0.039572808891534805,
+          0.04610450938344002,
+          -0.022614985704421997,
+          0.017067139968276024,
+          -0.019854728132486343,
+          0.0023076168727129698,
+          0.015755334869027138,
+          -0.0200323686003685,
+          0.018638573586940765,
+          -0.012223023921251297,
+          -0.012421161867678165,
+          0.011491965502500534,
+          -0.025484561920166016,
+          0.01572800613939762,
+          0.015536700375378132,
+          -0.014853468164801598,
+          -0.019239818677306175,
+          0.013917439617216587,
+          -0.040256042033433914,
+          0.019021185114979744,
+          -0.01000935211777687,
+          -0.01973174512386322,
+          -0.02899637445807457,
+          -0.010453452356159687,
+          -0.005004676058888435,
+          -0.00741990189999342,
+          0.017190122976899147,
+          0.03260384127497673,
+          0.003147992305457592,
+          0.0021487653721123934,
+          0.038370318710803986,
+          0.009387610480189323,
+          0.009585747495293617,
+          -0.0007908412953838706,
+          -0.02303859032690525,
+          -0.016821177676320076,
+          -0.025347914546728134,
+          0.012475820258259773,
+          -0.022396352142095566,
+          -0.030772779136896133,
+          0.02335287630558014,
+          0.007727356161922216,
+          0.01336402166634798,
+          0.015331730246543884,
+          -0.013732966966927052,
+          0.013172716833651066,
+          0.014785144478082657,
+          0.02817649580538273,
+          -0.017531737685203552,
+          -0.02198641188442707,
+          0.020893240347504616,
+          -0.06673812121152878,
+          -0.0074745602905750275,
+          0.006716172676533461,
+          0.02793053165078163,
+          -0.013507500290870667,
+          -0.012926753610372543,
+          -0.007481392472982407,
+          -0.004116474185138941,
+          0.012981412000954151,
+          -0.015277071855962276,
+          -0.010501278564333916,
+          0.019021185114979744,
+          -0.015072102658450603,
+          0.046131838113069534,
+          0.011840414255857468,
+          -0.014484522864222527,
+          -0.015454712323844433,
+          -0.00046801407006569207,
+          0.0370585136115551,
+          0.031292036175727844,
+          -0.0269603431224823,
+          0.009852208197116852,
+          0.008533569984138012,
+          -0.025634871795773506,
+          0.0616002157330513,
+          0.015564029105007648,
+          0.01341184787452221,
+          -0.000460754701634869,
+          0.024131761863827705,
+          -0.04222375154495239,
+          0.013036070391535759,
+          -0.021125540137290955,
+          0.05400267243385315,
+          -0.02295660227537155,
+          0.011539791710674763,
+          -0.04708836227655411,
+          -0.0032009428832679987,
+          0.047170352190732956,
+          0.013418680056929588,
+          0.010965877212584019,
+          0.004526413511484861,
+          0.01973174512386322,
+          0.029215008020401,
+          0.015441047959029675,
+          -0.009121149778366089,
+          -0.017572732642292976,
+          -0.05310080572962761,
+          0.008567731827497482,
+          -0.015482041984796524,
+          0.007495057303458452,
+          -0.03342371806502342,
+          0.0015953471884131432,
+          0.02981625311076641,
+          0.014621169306337833,
+          -0.014361540786921978,
+          -0.002075317781418562,
+          0.027807550504803658,
+          -0.042879652231931686,
+          -0.04968464747071266,
+          -0.007317416835576296,
+          -0.011368983425199986,
+          0.007693194784224033,
+          -0.03361502289772034,
+          0.02048330195248127,
+          0.009394442662596703,
+          0.024828657507896423,
+          -0.005103744566440582,
+          -0.011027367785573006,
+          0.01386278122663498,
+          0.002191467210650444,
+          -0.009209970012307167,
+          -0.03366968408226967,
+          -0.029269667342305183,
+          -0.010166495107114315,
+          0.008137295953929424,
+          0.027848543599247932,
+          -0.016219932585954666,
+          -0.012011222541332245,
+          0.027684569358825684,
+          0.022164052352309227,
+          -0.007153441198170185,
+          0.0025689529720693827,
+          -0.024651017040014267,
+          0.005295049399137497,
+          0.015755334869027138,
+          0.028340471908450127,
+          -0.03960013762116432,
+          0.011321157217025757,
+          -0.00564691424369812,
+          -0.022341692820191383,
+          -0.03312309831380844,
+          0.02380380965769291,
+          -0.0134391775354743,
+          -0.03621130809187889,
+          -0.014279552735388279,
+          0.014812474139034748,
+          -0.0044615063816308975,
+          -0.03172930330038071,
+          -0.013650978915393353,
+          0.006398469675332308,
+          -0.014416199177503586,
+          0.01612428016960621,
+          -0.015085767023265362,
+          -0.0035254780668765306,
+          -0.03880758956074715,
+          0.013036070391535759,
+          0.014006259851157665,
+          0.010938547551631927,
+          0.0009035745752044022,
+          0.006791328079998493,
+          0.017572732642292976,
+          -0.016055956482887268,
+          -0.026755372062325478,
+          -0.024159090593457222,
+          0.0017046643188223243,
+          0.006490706000477076,
+          0.006415550597012043,
+          0.010200656950473785,
+          0.016916830092668533,
+          -0.026454750448465347,
+          -0.011573953554034233,
+          0.007986984215676785,
+          -0.054631248116493225,
+          -0.012298179790377617,
+          -0.03878026083111763,
+          -0.019526775926351547,
+          -0.0013092437293380499,
+          0.028941715136170387,
+          -0.0064462958835065365,
+          0.03427092730998993,
+          -0.0056708273477852345,
+          0.00754971569404006,
+          0.017203787341713905,
+          0.018993854522705078,
+          0.018173975870013237,
+          -0.011806252412497997,
+          -0.050340548157691956,
+          0.028887057676911354,
+          -0.03593801334500313,
+          -0.0045229969546198845,
+          0.00282345712184906,
+          -0.002869575284421444,
+          -0.0057049887254834175,
+          0.0338883176445961,
+          0.0015304401749745011,
+          0.023544181138277054,
+          -0.00381243578158319,
+          0.04353555664420128,
+          -0.027055995538830757,
+          -0.049821291118860245,
+          0.00873170793056488,
+          0.027657238766551018,
+          0.024814993143081665,
+          -0.006521451286971569,
+          -0.026536738499999046,
+          -0.02018267847597599,
+          -0.013514332473278046,
+          0.007966487668454647,
+          0.002003578469157219,
+          -0.00044239286216907203,
+          -0.0013263245346024632,
+          0.006548780482262373,
+          -0.034462232142686844,
+          -0.03645727038383484,
+          0.0040549831464886665,
+          0.011813084594905376,
+          0.03213924169540405,
+          -0.02590816468000412,
+          0.017067139968276024,
+          -0.015454712323844433,
+          -0.038588955998420715,
+          0.0006661514053121209,
+          0.023598840460181236,
+          0.035965342074632645,
+          -0.03921752795577049,
+          0.018419940024614334,
+          0.015850987285375595,
+          -0.005602504126727581,
+          0.004188213497400284,
+          0.025265926495194435,
+          0.025347914546728134,
+          0.022273369133472443,
+          -0.00858822837471962,
+          0.04020138084888458,
+          0.014457193203270435,
+          -0.005564926192164421,
+          0.029706936329603195,
+          -0.016766518354415894,
+          0.02998022921383381,
+          -0.0002463906130287796,
+          0.021426161751151085,
+          -0.027916867285966873,
+          0.019007520750164986,
+          -0.0382336750626564,
+          -0.008642887696623802,
+          0.03304110839962959,
+          -0.026085805147886276,
+          0.0034537387546151876,
+          0.022915607318282127,
+          0.01851559244096279,
+          -0.010631092824041843,
+          0.01725844480097294,
+          -0.04451940953731537,
+          -0.029160350561141968,
+          -0.026195121929049492,
+          -0.006822073366492987,
+          -0.02500629797577858,
+          -0.03552807494997978,
+          -0.056134358048439026,
+          0.008649719879031181,
+          -0.047935571521520615,
+          0.004386350512504578,
+          0.03533677011728287,
+          0.010760907083749771,
+          0.010549104772508144,
+          -0.024432383477687836,
+          0.02809450775384903,
+          -0.018351616337895393,
+          -0.00914847943931818,
+          0.021043552085757256,
+          0.024883316829800606,
+          -0.012346005998551846,
+          -0.013842284679412842,
+          0.01612428016960621,
+          -0.03804237022995949,
+          0.006350643467158079,
+          -0.0020326157100498676,
+          -0.024446047842502594,
+          0.019991373643279076,
+          -0.04684239998459816,
+          0.02380380965769291,
+          0.013876445591449738,
+          0.02877774089574814,
+          0.06039772555232048,
+          -0.024651017040014267,
+          -0.06285735964775085,
+          0.020128021016716957,
+          -0.004143803380429745,
+          0.01336402166634798,
+          0.003747528651729226,
+          -0.006169586908072233,
+          0.002331529976800084,
+          -0.01679384708404541,
+          -0.026031147688627243,
+          0.052526891231536865,
+          0.023475859314203262,
+          0.015263407491147518,
+          -0.010023016482591629,
+          0.0013707346515730023,
+          0.0368945375084877,
+          0.023817474022507668,
+          0.01378079317510128,
+          0.016998816281557083,
+          -0.03929951786994934,
+          0.0037782741710543633,
+          0.0064087179489433765,
+          0.022246040403842926,
+          0.027206305414438248,
+          -0.02711065299808979,
+          -0.027083324268460274,
+          0.020920569077134132,
+          0.04528462886810303,
+          0.011908737011253834,
+          -0.0139379370957613,
+          0.016055956482887268,
+          0.03383366018533707,
+          -0.035965342074632645,
+          -0.003212899435311556,
+          0.009551586583256721,
+          0.0011298952158540487,
+          -0.010719913057982922,
+          -0.02041497826576233,
+          0.009961525909602642,
+          0.014402534812688828,
+          -0.02010069042444229,
+          0.01906217820942402,
+          0.010972709394991398,
+          0.028449788689613342,
+          0.0052267261780798435,
+          -0.006750334054231644,
+          -0.029570288956165314,
+          0.04902874305844307,
+          -0.022027406841516495,
+          0.022997595369815826,
+          0.03774174675345421,
+          -0.0027431773487478495,
+          0.024487042799592018,
+          0.002572369296103716,
+          -0.03812435641884804,
+          0.00813046284019947,
+          0.04451940953731537,
+          0.005575174931436777,
+          -0.028039850294589996,
+          -0.016670865938067436,
+          -0.03145601227879524,
+          0.004348773043602705,
+          -0.01738142780959606,
+          -0.02793053165078163,
+          -0.0014125825837254524,
+          0.005551261827349663,
+          0.016684530302882195,
+          -0.002896904479712248,
+          -0.005985114257782698,
+          -0.006132008973509073,
+          0.02507462166249752,
+          -0.049821291118860245,
+          -0.005663995165377855,
+          0.014443528838455677,
+          -0.029406314715743065,
+          -0.028504448011517525,
+          -0.03711317479610443,
+          -0.004229207523167133,
+          -0.013842284679412842,
+          0.0010385129135102034,
+          -0.001186261884868145,
+          -0.0134391775354743,
+          0.011082026176154613,
+          -0.03940883278846741,
+          0.02508828602731228,
+          -0.012100041843950748,
+          -0.010139165446162224,
+          0.0030813771300017834,
+          0.009230467490851879,
+          -0.010788236744701862,
+          -0.008226115256547928,
+          0.04178648069500923,
+          0.02198641188442707,
+          0.024418719112873077,
+          -0.006641016807407141,
+          -0.007071453146636486,
+          0.005745982751250267,
+          0.000374923663912341,
+          0.03809702768921852,
+          -0.001469803275540471,
+          0.03561006113886833,
+          -0.009121149778366089,
+          -0.012660292908549309,
+          0.006883564405143261,
+          0.03752311319112778,
+          -0.0069962977431714535,
+          0.010207489132881165,
+          0.03088209591805935,
+          0.027015000581741333,
+          -0.017121799290180206,
+          -0.015195083804428577,
+          -0.0025894499849528074,
+          -0.004434177186340094,
+          -0.021057216450572014,
+          0.02198641188442707,
+          0.03350570797920227,
+          0.01171059999614954,
+          -0.009353448636829853,
+          0.022573992609977722,
+          -0.04348089545965195,
+          -0.0011435599299147725,
+          0.004113057628273964,
+          0.012455322779715061,
+          -0.05001259595155716,
+          0.016247261315584183,
+          0.014416199177503586,
+          0.017367763444781303,
+          0.00129045476205647,
+          0.006395053584128618,
+          0.023175235837697983,
+          0.02176777832210064,
+          0.024951640516519547,
+          -0.015850987285375595,
+          2.2431901015806943e-05,
+          0.019841063767671585,
+          0.015618688426911831,
+          0.024268407374620438,
+          0.0021436410024762154,
+          0.024978969246149063,
+          -0.017586397007107735,
+          -0.0138559490442276,
+          0.00282345712184906,
+          -0.023380205035209656,
+          -0.010330471210181713,
+          0.05468590557575226,
+          0.011553456075489521,
+          0.051242414861917496,
+          0.01426588837057352,
+          0.00024169338576029986,
+          0.018993854522705078,
+          -0.020510630682110786,
+          -0.02026466652750969,
+          -0.0007050101994536817,
+          0.0027824630960822105,
+          -0.020387649536132812,
+          -0.00945593323558569,
+          -0.010610596276819706,
+          -0.018187642097473145,
+          -0.018939197063446045,
+          -0.006746917963027954,
+          -0.006326730363070965,
+          0.016848506405949593,
+          -0.002319573424756527,
+          -0.013541662134230137,
+          -0.0320572555065155,
+          0.022041071206331253,
+          0.0199503805488348,
+          0.040556661784648895,
+          -0.013575823977589607,
+          0.011239170096814632,
+          -0.0033819994423538446,
+          -0.004560574889183044,
+          0.014621169306337833,
+          -0.01604229211807251,
+          0.017982671037316322,
+          0.009196305647492409,
+          0.00139293959364295,
+          0.011075193993747234,
+          -0.01077457144856453,
+          -0.04684239998459816,
+          -0.010200656950473785,
+          -0.023557845503091812,
+          0.03284980356693268,
+          0.00672300485894084,
+          0.0018566835206001997,
+          0.0039012557826936245,
+          0.0035459750797599554,
+          -0.008116798475384712,
+          0.016821177676320076,
+          0.013275201432406902,
+          -0.03274048864841461,
+          0.017436085268855095,
+          -0.011382648721337318,
+          0.010255315341055393,
+          -0.034598879516124725,
+          -0.013691972941160202,
+          -0.002481841016560793,
+          0.017586397007107735,
+          0.026482079178094864,
+          -0.02869575284421444,
+          0.010610596276819706,
+          0.02131684496998787,
+          -0.002876407466828823,
+          -0.005698156543076038,
+          0.0021231439895927906,
+          0.0014151446521282196,
+          -0.026304440572857857,
+          -0.012024886906147003,
+          0.005301882047206163,
+          0.004659643396735191,
+          -0.03386098891496658,
+          0.005448776762932539,
+          -0.0293789841234684,
+          -0.005595671944320202,
+          -0.011908737011253834,
+          0.003928584977984428,
+          0.0068152411840856075,
+          0.002565536880865693,
+          0.009134814143180847,
+          -0.01831062324345112,
+          -0.007638535927981138,
+          0.024145426228642464,
+          0.004222374875098467,
+          0.015195083804428577,
+          -0.012414328753948212,
+          0.029542960226535797,
+          -0.016684530302882195,
+          0.021959083154797554,
+          0.014402534812688828,
+          0.007857169955968857,
+          -0.002591158030554652,
+          -0.002437430899590254,
+          -0.04471071437001228,
+          -0.01046711765229702,
+          0.010111836716532707,
+          0.02559387870132923,
+          -0.0005986822070553899,
+          0.014539181254804134,
+          0.02477400004863739,
+          -0.008882018737494946,
+          0.015495706349611282,
+          -0.01701248250901699,
+          -0.030144205316901207,
+          0.020305661484599113,
+          -0.0027380529791116714,
+          0.015318065881729126,
+          0.02425474300980568,
+          -0.03148334100842476,
+          -0.02432306669652462,
+          -0.002088982379063964,
+          0.011813084594905376,
+          0.05137906223535538,
+          0.029706936329603195,
+          0.012182029895484447,
+          0.020906904712319374,
+          -0.008157792501151562,
+          -0.02869575284421444,
+          0.012455322779715061,
+          0.01265346072614193,
+          -0.03110072948038578,
+          0.00634722737595439,
+          0.026755372062325478,
+          -0.008601893670856953,
+          -0.02853177674114704,
+          0.02599015273153782,
+          -0.00689381267875433,
+          0.002553580328822136,
+          -0.004796289838850498,
+          -0.020947899669408798,
+          0.004386350512504578,
+          -0.01602862775325775,
+          0.027903202921152115,
+          -0.024418719112873077,
+          0.008595061488449574,
+          0.010084507055580616,
+          -0.009414940141141415,
+          0.032931793481111526,
+          -0.0005137051921337843,
+          -0.01505843736231327,
+          0.010173327289521694,
+          0.021344173699617386,
+          -0.028887057676911354,
+          0.0005017486400902271,
+          0.002992557128891349,
+          0.01339818350970745,
+          0.03968212753534317,
+          0.018939197063446045,
+          0.016698194667696953,
+          0.03295912221074104,
+          0.020169014111161232,
+          -0.01294725015759468,
+          0.025170274078845978,
+          0.01039879396557808,
+          -0.020237337797880173,
+          0.008615558035671711,
+          -0.0060056112706661224,
+          0.014580175280570984,
+          -0.026851026341319084,
+          -0.007050956133753061,
+          -0.002191467210650444,
+          0.0020087026059627533,
+          -0.020373983308672905,
+          -0.005223310086876154,
+          -0.002256374340504408,
+          -0.023680828511714935,
+          0.008150960318744183,
+          0.028668422251939774,
+          0.023393871262669563,
+          -0.00436585396528244,
+          -0.021508149802684784,
+          0.026673385873436928,
+          -0.02410443127155304,
+          -0.012714951299130917,
+          -0.0037782741710543633,
+          0.015796327963471413,
+          -0.022983931005001068,
+          -0.032494522631168365,
+          0.0019437955925241113,
+          0.03509080410003662,
+          0.02823115512728691,
+          -0.004167716484516859,
+          0.044546738266944885,
+          0.02333921194076538,
+          -0.021617466583848,
+          0.02033299021422863,
+          -0.038288332521915436,
+          0.018939197063446045,
+          0.017531737685203552,
+          -0.04684239998459816,
+          -0.030253522098064423,
+          -0.006258407142013311,
+          -0.011895072646439075,
+          0.008123630657792091,
+          -0.017231116071343422,
+          0.003781690262258053,
+          -0.00754971569404006,
+          -0.023147907108068466,
+          0.03470819443464279,
+          -0.016506889835000038,
+          0.03402496501803398,
+          -0.008369594812393188,
+          0.003199234837666154,
+          -0.015632351860404015,
+          -0.016315585002303123,
+          -0.012468988075852394,
+          -0.020770259201526642,
+          -0.01027581188827753,
+          -0.02681003138422966,
+          -0.021521814167499542,
+          -0.011341654695570469,
+          0.004509332589805126,
+          -0.008458415046334267,
+          0.02363983355462551,
+          0.0268783550709486,
+          0.01133482251316309,
+          0.015331730246543884,
+          -0.0207839235663414,
+          0.043644871562719345,
+          -0.026304440572857857,
+          0.022942937910556793,
+          -0.013186381198465824,
+          -0.015372724272310734,
+          0.009681399911642075,
+          0.020879575982689857,
+          -0.04072063788771629,
+          0.0034229932352900505,
+          -0.01611061580479145,
+          -0.007385740056633949,
+          -0.0215354785323143,
+          0.00495001720264554,
+          0.017818694934248924,
+          -0.008793198503553867,
+          0.00791182927787304,
+          0.00041847972897812724,
+          -0.004543493967503309,
+          0.05099645256996155,
+          -0.0027175559662282467,
+          0.02642742171883583,
+          -0.016301920637488365,
+          -0.019171494990587234,
+          0.013261537067592144,
+          -0.029870912432670593,
+          0.013698806054890156,
+          -0.00832176860421896,
+          -0.037085842341184616,
+          -0.00865655206143856,
+          0.01171059999614954,
+          0.018392611294984818,
+          0.0116422763094306,
+          -0.0038875911850482225,
+          -0.009845376014709473,
+          -0.011389480903744698,
+          -0.00627207150682807,
+          -0.03582869842648506,
+          -0.015919310972094536,
+          0.0021436410024762154,
+          -0.019690752029418945,
+          0.03394297510385513,
+          0.0034434902481734753,
+          0.03741379454731941,
+          -0.011409977450966835,
+          -0.014211229979991913,
+          0.0052643041126430035,
+          -0.017518073320388794,
+          0.0038226840551942587,
+          0.0029771842528134584,
+          0.008458415046334267,
+          -0.018993854522705078,
+          0.03962746635079384,
+          0.009052827022969723,
+          0.014511851593852043,
+          -0.03462620824575424,
+          -0.013295698910951614,
+          -0.04501133784651756,
+          0.006654681637883186,
+          -0.017873354256153107,
+          -0.032357875257730484,
+          -0.0004112203896511346,
+          -0.005698156543076038,
+          0.005882629193365574,
+          -0.016534218564629555,
+          -0.013609985820949078,
+          0.006702507846057415,
+          -0.008226115256547928,
+          0.009223634377121925,
+          -0.01896652579307556,
+          0.010180159471929073,
+          0.004000324755907059,
+          0.03192060813307762,
+          -0.021658461540937424,
+          0.034598879516124725,
+          -0.012011222541332245,
+          0.003979827743023634,
+          -0.00013408431550487876,
+          -0.012632963247597218,
+          0.019239818677306175,
+          0.016766518354415894,
+          -0.0372224897146225,
+          0.011464636772871017,
+          0.0477442666888237,
+          0.03563738986849785,
+          -0.036484599113464355,
+          0.012530478648841381,
+          0.007208099588751793,
+          -0.003062588395550847,
+          0.03186595067381859,
+          0.0013647563755512238,
+          0.01671185903251171,
+          -0.0007998087094165385,
+          -0.02582617662847042,
+          -0.024514371529221535,
+          0.0037201992236077785,
+          0.01378762535750866,
+          0.007392572239041328,
+          -0.031073400750756264,
+          0.01941745914518833,
+          0.002908861031755805,
+          0.029925569891929626,
+          0.025812512263655663,
+          -0.012598801404237747,
+          0.031210046261548996,
+          0.01423855870962143,
+          -0.029242338612675667,
+          0.027985190972685814,
+          0.010747242718935013,
+          0.014060918241739273,
+          0.029652277007699013,
+          0.04749830439686775,
+          0.009326119907200336,
+          -0.022027406841516495,
+          0.001334864879027009,
+          0.012960914522409439,
+          -0.009954692795872688,
+          0.016288256272673607,
+          0.014183900319039822,
+          0.026031147688627243,
+          0.010965877212584019,
+          -0.014019925147294998,
+          -0.010863391682505608,
+          0.04430077597498894,
+          -0.01574167050421238,
+          -0.00499101122841239,
+          -0.016766518354415894,
+          0.035200122743844986,
+          0.027452269569039345,
+          -0.02402244508266449,
+          -0.02335287630558014,
+          -0.005202813073992729,
+          -0.006750334054231644,
+          0.016329249367117882,
+          -0.01687583513557911,
+          -0.01378762535750866,
+          -0.02891438640654087,
+          -0.01596030406653881,
+          0.0018754723714664578,
+          -0.03929951786994934,
+          -0.02839512936770916,
+          -0.0008578834240324795,
+          0.011437307111918926,
+          0.0008672778494656086,
+          0.02206839993596077,
+          -0.012714951299130917,
+          -0.019021185114979744,
+          -0.005202813073992729,
+          0.02847711741924286,
+          -0.01125283446162939,
+          0.021508149802684784,
+          -0.004833867773413658,
+          0.005014924332499504,
+          0.005206229165196419,
+          0.041239894926548004,
+          0.0016380491433665156,
+          -0.02847711741924286,
+          0.02794419787824154,
+          -0.008554067462682724,
+          0.009688233025372028,
+          0.03607466071844101,
+          0.0026663136668503284,
+          -0.02235535718500614,
+          0.02418641932308674,
+          -0.0026321520563215017,
+          -0.02025100216269493,
+          -0.008936677128076553,
+          0.008007481694221497,
+          -0.0027500095311552286,
+          -0.019390130415558815,
+          0.009319286793470383,
+          0.042141761630773544,
+          -0.0028405378106981516,
+          0.02387213334441185,
+          0.053046148270368576,
+          -0.02235535718500614,
+          -0.02387213334441185,
+          0.015372724272310734,
+          -0.028504448011517525,
+          -0.007044123951345682,
+          -0.006832322105765343,
+          0.03826100379228592,
+          -0.03041749820113182,
+          -0.010863391682505608,
+          -0.025361578911542892,
+          7.216640369733796e-05,
+          -0.05495919659733772,
+          0.01904851384460926,
+          0.004266784992069006,
+          0.01505843736231327,
+          -0.041895799338817596,
+          -0.0353914275765419,
+          0.028559105470776558,
+          0.029269667342305183,
+          -0.007433566264808178,
+          -0.00429069809615612,
+          -0.011608115397393703,
+          -0.009763387963175774,
+          0.021371502429246902,
+          0.0008967422763817012,
+          -0.01565968245267868,
+          0.014101912267506123,
+          0.03170197457075119,
+          -0.0014373497106134892,
+          -0.015577694401144981,
+          -0.06045238673686981,
+          -0.017299439758062363,
+          0.014908126555383205,
+          0.0023178651463240385,
+          -0.0020394481252878904,
+          -3.75510826415848e-05,
+          0.0011683270568028092,
+          -0.0024288904387503862,
+          -0.025197602808475494,
+          0.02726096473634243,
+          0.008793198503553867,
+          0.014525516889989376,
+          -0.004587904084473848,
+          -0.011416809633374214,
+          0.014894462190568447,
+          0.017682049423456192,
+          -0.011881408281624317,
+          -0.017121799290180206,
+          0.03894423693418503,
+          0.01973174512386322,
+          -0.001998454099521041,
+          -0.0005862986436113715,
+          0.0010333886602893472,
+          -0.004136971198022366,
+          -0.030062217265367508,
+          0.001253731083124876,
+          -0.0009095528512261808,
+          0.00581772206351161,
+          0.02823115512728691,
+          -0.008724874816834927,
+          0.029652277007699013,
+          0.023462193086743355,
+          0.021357838064432144,
+          -0.0032590175978839397,
+          -0.0043897670693695545,
+          -0.0018618077738210559,
+          0.02538890950381756,
+          0.01208637747913599,
+          0.008670216426253319,
+          0.002106063300743699,
+          -0.0038602619897574186,
+          -0.017354097217321396,
+          0.010781404562294483,
+          -0.013391351327300072,
+          -0.015482041984796524,
+          0.016001299023628235,
+          0.027001336216926575,
+          -0.007604374550282955,
+          -0.004222374875098467,
+          0.0025962824001908302,
+          0.01076773926615715,
+          0.033478379249572754,
+          -0.019267147406935692,
+          -0.007652200758457184,
+          0.021795107051730156,
+          -0.00502858916297555,
+          0.01265346072614193,
+          0.010528608225286007,
+          0.00910748541355133,
+          0.027616245672106743,
+          0.06580892205238342,
+          -0.030909424647688866,
+          0.0016628163866698742,
+          -0.01934913545846939,
+          -0.0011239169398322701,
+          0.011512462981045246,
+          -0.023516852408647537,
+          0.034598879516124725,
+          0.003549391170963645,
+          -0.017969006672501564,
+          0.034680865705013275,
+          0.015564029105007648,
+          0.010945379734039307,
+          0.010296309366822243,
+          -0.01657521352171898,
+          -0.004909023176878691,
+          -0.011088858358561993,
+          0.008328600786626339,
+          0.020606283098459244,
+          0.023065919056534767,
+          0.01679384708404541,
+          0.018474599346518517,
+          0.002625319641083479,
+          0.01770937815308571,
+          -0.0200323686003685,
+          0.021029887720942497,
+          -0.003908087965101004,
+          0.005117409396916628,
+          -0.02251933328807354,
+          0.0028319975826889277,
+          0.0025552883744239807,
+          -0.001964292488992214,
+          0.013698806054890156,
+          0.000991540728136897,
+          -0.011000038124620914,
+          0.003528894390910864,
+          0.0021248520351946354,
+          0.01132799033075571,
+          0.009312454611063004,
+          -0.0032931792084127665,
+          0.004249704070389271,
+          0.018652239814400673,
+          0.022874614223837852,
+          0.012530478648841381,
+          0.0013177840737625957,
+          0.012305011972784996,
+          0.019923051819205284,
+          -0.022273369133472443,
+          0.01088388916105032,
+          -0.030854765325784683,
+          -0.003627962898463011,
+          0.0036689569242298603,
+          -0.010330471210181713,
+          -0.005964617244899273,
+          -0.011505629867315292,
+          -0.008301271125674248,
+          0.01295408234000206,
+          -0.0074472310952842236,
+          0.021726783365011215,
+          -0.001455284538678825,
+          -0.006538532208651304,
+          0.02176777832210064,
+          0.0013391351094469428,
+          -0.0044512576423585415,
+          -0.028559105470776558,
+          -0.0099751902744174,
+          0.021043552085757256,
+          -0.002442555036395788,
+          -0.0005747691029682755,
+          0.02559387870132923,
+          -0.015495706349611282,
+          0.01133482251316309,
+          -0.01686217077076435,
+          -0.015304401516914368,
+          -0.0028217490762472153,
+          -0.015468376688659191,
+          0.016452230513095856,
+          -0.004406847525388002,
+          0.00627207150682807,
+          -0.03448956087231636,
+          0.03142867982387543,
+          0.006627352442592382,
+          -0.0016611083410680294,
+          -0.013664644211530685,
+          -0.030253522098064423,
+          0.0095584187656641,
+          -0.021207528188824654,
+          -0.00017358368495479226,
+          0.02176777832210064,
+          0.011471468955278397,
+          -0.0004603276902344078,
+          -0.009066491387784481,
+          0.0024835490621626377,
+          0.0059680333361029625,
+          -0.0009180932538583875,
+          0.005862132180482149,
+          -0.00028332785586826503,
+          0.015304401516914368,
+          -0.016957823187112808,
+          -0.028449788689613342,
+          -0.02303859032690525,
+          0.01306340005248785,
+          0.00862239021807909,
+          -0.004871445707976818,
+          -0.015523036010563374,
+          0.0057767280377447605,
+          0.004181380849331617,
+          -0.0018020248971879482,
+          -0.0034810679499059916,
+          0.007488225121051073,
+          -0.0010103295790031552,
+          0.001913050189614296,
+          0.002036032034084201,
+          -0.012796939350664616,
+          0.020537959411740303,
+          0.035118136554956436,
+          0.006186667364090681,
+          -0.03320508450269699,
+          0.0021624299697577953,
+          0.00015885148604866117,
+          0.0011461219983175397,
+          -0.0069587198086082935,
+          0.026974007487297058,
+          0.008519905619323254,
+          0.01550937071442604,
+          0.030827436596155167,
+          0.0036347953137010336,
+          -0.0031821539159864187,
+          0.012475820258259773,
+          -0.023434864357113838,
+          -0.02425474300980568,
+          0.0027858791872859,
+          0.009244131855666637,
+          0.010405626147985458,
+          0.019226154312491417,
+          -0.012188862077891827,
+          0.004003740847110748,
+          0.01679384708404541,
+          0.019294477999210358,
+          -0.009920531883835793,
+          -0.01816031150519848,
+          0.01516775507479906,
+          0.054084662348032,
+          0.01590564474463463,
+          -0.006565861403942108,
+          -0.01035779993981123,
+          -0.013377686031162739,
+          0.02801251970231533,
+          0.01092488318681717,
+          0.01596030406653881,
+          -0.0003646751865744591,
+          -0.011840414255857468,
+          0.015919310972094536,
+          0.005981697700917721,
+          -0.0014271012041717768,
+          -0.03910820931196213,
+          0.020073361694812775,
+          -0.0058245547115802765,
+          0.02139883302152157,
+          -0.0005619584699161351,
+          -0.0027397610247135162,
+          0.03899889439344406,
+          0.03683988004922867,
+          -0.011491965502500534,
+          0.04443742334842682,
+          -0.007385740056633949,
+          -0.0011409977450966835,
+          -0.0008967422763817012,
+          -0.0026270279195159674,
+          0.01822863519191742,
+          -0.012899423949420452,
+          0.0008950341725721955,
+          -0.0024271823931485415,
+          0.026413757354021072,
+          0.0042702010832726955,
+          -0.014785144478082657,
+          0.0075087216682732105,
+          -0.006442879792302847,
+          -0.02674170769751072,
+          -0.005899710115045309,
+          0.013760296627879143,
+          -0.010685751214623451,
+          0.005783560685813427,
+          0.006521451286971569,
+          0.023065919056534767,
+          0.01738142780959606,
+          -0.019977709278464317,
+          -0.007495057303458452,
+          -0.008526737801730633,
+          0.01261929888278246,
+          0.004813370760530233,
+          -0.012366502545773983,
+          0.022874614223837852,
+          -0.01964975893497467,
+          0.0030284267850220203,
+          0.016165273264050484,
+          -0.007310584653168917,
+          -0.0011854078620672226,
+          -0.02175411395728588,
+          -0.012305011972784996,
+          -0.04001007601618767,
+          0.013849116861820221,
+          -0.016315585002303123,
+          -0.013773960992693901,
+          0.019786404445767403,
+          -0.027848543599247932,
+          0.025197602808475494,
+          -0.009401274845004082,
+          0.024200083687901497,
+          -0.019923051819205284,
+          0.013849116861820221,
+          0.005708404816687107,
+          -0.017531737685203552,
+          0.005339459516108036,
+          -0.012243521399796009,
+          -0.007228596601635218,
+          -0.0020821501966565847,
+          -0.014921790920197964,
+          0.03593801334500313,
+          0.0323852077126503,
+          -0.02568953111767769,
+          0.03142867982387543,
+          0.02034665457904339,
+          0.02071559987962246,
+          -0.005240391008555889,
+          -0.010836062952876091,
+          0.0199503805488348,
+          -0.016903163865208626,
+          0.03323241323232651,
+          0.0027841711416840553,
+          -0.0008284190553240478,
+          0.0015620396006852388,
+          -0.007529218681156635,
+          0.020988892763853073,
+          -0.007761518005281687,
+          0.008806862868368626,
+          0.021275850012898445,
+          -0.003580136690288782,
+          0.010549104772508144,
+          -0.006357475649565458,
+          0.015768999233841896,
+          0.026536738499999046,
+          -0.009968358092010021,
+          0.03733180835843086,
+          0.00813046284019947,
+          -0.00014177068078424782,
+          0.011983892880380154,
+          0.0018310622544959188,
+          -0.019308142364025116,
+          0.006945055443793535,
+          -0.021426161751151085,
+          0.0030403833370655775,
+          0.01434787642210722,
+          -0.004833867773413658,
+          -0.021808771416544914,
+          0.005752814933657646,
+          -0.004150635562837124,
+          -0.010248483158648014,
+          -0.0074745602905750275,
+          -0.01816031150519848,
+          0.011075193993747234,
+          0.018351616337895393,
+          0.028641093522310257,
+          -0.0030335509218275547,
+          -0.002674854127690196,
+          0.007228596601635218,
+          -0.008567731827497482,
+          0.030007557943463326,
+          -0.029187679290771484,
+          0.008308103308081627,
+          -0.0033905399031937122,
+          0.0017012482276186347,
+          0.02139883302152157,
+          -8.753912698011845e-05,
+          -0.010494446381926537,
+          -0.012188862077891827,
+          0.006832322105765343,
+          -0.009818046353757381,
+          -0.007228596601635218,
+          -0.002570661250501871,
+          0.006729837041348219,
+          0.024268407374620438,
+          0.005804057698696852,
+          0.012714951299130917,
+          -0.03402496501803398,
+          0.000319197541102767,
+          -0.003631379222497344,
+          -0.015154089778661728,
+          -0.0037714417558163404,
+          -0.012646627612411976,
+          -0.002927649999037385,
+          -0.030007557943463326,
+          0.008779534138739109,
+          0.001211883150972426,
+          0.006593190599232912,
+          0.004864613059908152,
+          -0.022642314434051514,
+          0.004488835576921701,
+          -0.006641016807407141,
+          -0.0018600996118038893,
+          0.013452841900289059,
+          -0.0033580863382667303,
+          -0.012236688286066055,
+          0.0002893061318900436,
+          0.014019925147294998,
+          -0.014156571589410305,
+          0.006248158402740955,
+          0.005066166631877422,
+          -0.003921752795577049,
+          0.028504448011517525,
+          0.022806290537118912,
+          0.006429214961826801,
+          -0.013678308576345444,
+          0.005694740451872349,
+          -0.002972060116007924,
+          -0.003522061975672841,
+          0.001583390636369586,
+          0.006979216821491718,
+          0.026195121929049492,
+          0.01956777088344097,
+          -0.03629329428076744,
+          -0.01716279238462448,
+          0.004922688007354736,
+          -0.005459025502204895,
+          -0.000369158893590793,
+          0.010077674873173237,
+          -0.012325508520007133,
+          0.001945503638125956,
+          -0.012455322779715061,
+          0.005592255387455225,
+          -0.002287119859829545,
+          -0.010624260641634464,
+          0.01471682172268629,
+          -0.017190122976899147,
+          0.02967960759997368,
+          0.015454712323844433,
+          0.004905607085675001,
+          0.01520874910056591,
+          0.003979827743023634,
+          0.008308103308081627,
+          -0.027602581307291985,
+          0.01479880977421999,
+          -0.008513073436915874,
+          0.007638535927981138,
+          0.004492251668125391,
+          -0.03315042704343796,
+          -0.014443528838455677,
+          0.0014971325872465968,
+          -0.003959330730140209,
+          -0.007399404887109995,
+          0.001276790164411068,
+          0.014662163332104683,
+          0.014375205151736736,
+          -0.004922688007354736,
+          -0.012011222541332245,
+          0.03533677011728287,
+          -0.002997681265696883,
+          0.015386389568448067,
+          -0.001395501778461039,
+          0.013603152707219124,
+          -0.014429864473640919,
+          0.004847532603889704,
+          -0.013275201432406902,
+          -0.001998454099521041,
+          -0.01212053932249546,
+          0.039053551852703094,
+          0.00012116695870645344,
+          -0.00454007787629962,
+          0.0022768713533878326,
+          -0.019267147406935692,
+          -0.006586358416825533,
+          -0.001919882488436997,
+          -0.01647956110537052,
+          0.004181380849331617,
+          0.0200323686003685,
+          0.011758426204323769,
+          -0.03243986517190933,
+          0.024801328778266907,
+          -0.011157182045280933,
+          0.01702614687383175,
+          0.003424701513722539,
+          0.006548780482262373,
+          0.02093423530459404,
+          -0.00820561870932579,
+          -0.033095769584178925,
+          -0.0020052865147590637,
+          -0.002015535021200776,
+          0.004266784992069006,
+          0.03675789386034012,
+          0.02312057837843895,
+          0.020196344703435898,
+          0.011860910803079605,
+          0.007255925796926022,
+          0.01687583513557911,
+          0.018870873376727104,
+          0.008800030685961246,
+          -0.013268369249999523,
+          -0.006603439338505268,
+          -0.030772779136896133,
+          0.015646018087863922,
+          0.023216230794787407,
+          0.015714339911937714,
+          -0.02168579027056694,
+          0.006391637027263641,
+          0.00696555245667696,
+          0.015113096684217453,
+          0.009886370040476322,
+          -0.00903916172683239,
+          -0.010685751214623451,
+          -0.016096951439976692,
+          0.02847711741924286,
+          -0.027602581307291985,
+          0.006569277495145798,
+          0.0008339703199453652,
+          -0.01647956110537052,
+          -0.015577694401144981,
+          0.02545723132789135,
+          0.04391816630959511,
+          -0.02634543366730213,
+          0.003522061975672841,
+          0.009353448636829853,
+          -0.010521776042878628,
+          0.014279552735388279,
+          0.002939606551080942,
+          -0.012673957273364067,
+          0.013145387172698975,
+          -0.019144166260957718,
+          -0.012632963247597218,
+          -0.012359670363366604,
+          -0.0320572555065155,
+          0.0034349497873336077,
+          0.02913302183151245,
+          0.022314364090561867,
+          0.006381388753652573,
+          -0.00011241305037401617,
+          0.016096951439976692,
+          0.0004517872876022011,
+          0.019718080759048462,
+          -0.01627459190785885,
+          0.009954692795872688,
+          -0.0008621536544524133,
+          0.014730486087501049,
+          -0.008465247228741646,
+          0.01596030406653881,
+          -0.0003898693830706179,
+          0.0007088534184731543,
+          0.00986587256193161,
+          -0.0005717799649573863,
+          -0.014812474139034748,
+          0.00993419624865055,
+          0.005144738592207432,
+          0.014949120581150055,
+          -0.006207164376974106,
+          0.0075565483421087265,
+          -0.03011687472462654,
+          0.01627459190785885,
+          0.02462368831038475,
+          0.009066491387784481,
+          -0.026851026341319084,
+          -0.000849770032800734,
+          0.00523355882614851,
+          -0.010740410536527634,
+          -0.0012486068299040198,
+          0.01837894693017006,
+          0.024159090593457222,
+          0.0029771842528134584,
+          0.028750410303473473,
+          0.001836186507716775,
+          0.019021185114979744,
+          -0.012072713114321232,
+          -0.003141160123050213,
+          0.013254704885184765,
+          0.0012725199339911342,
+          0.0027448853943496943,
+          0.02749326452612877,
+          -0.016971487551927567,
+          0.015482041984796524,
+          -0.00021991536777932197,
+          0.020291997119784355,
+          -0.0064804572612047195,
+          0.00903916172683239,
+          -0.00048039763350971043,
+          0.026564067229628563,
+          0.005732318386435509,
+          0.0065897745080292225,
+          0.0582660436630249,
+          -0.021111875772476196,
+          0.019977709278464317,
+          0.006856235209852457,
+          -0.006576110143214464,
+          -0.015400053933262825,
+          -0.0014894461492076516,
+          -0.01989572122693062,
+          0.026482079178094864,
+          0.014293218031525612,
+          0.0025433318223804235,
+          -0.0026663136668503284,
+          -0.007392572239041328,
+          0.011027367785573006,
+          -0.0012528770603239536,
+          0.0039729950949549675,
+          0.004680140409618616,
+          -0.012933585792779922,
+          0.02394045703113079,
+          0.007932325825095177,
+          -0.004051567055284977,
+          -0.010337303392589092,
+          -0.006463376805186272,
+          0.009018665179610252,
+          -0.016629870980978012,
+          0.001118792686611414,
+          -0.00013034789299126714,
+          0.02250566892325878,
+          -0.017572732642292976,
+          0.035282112658023834,
+          -0.014006259851157665,
+          -0.017449749633669853,
+          -0.010754074901342392,
+          0.011655941605567932,
+          0.024719340726733208,
+          0.01381495501846075,
+          0.020373983308672905,
+          0.01596030406653881,
+          -0.0034264095593243837,
+          0.019526775926351547,
+          -0.0067810798063874245,
+          -0.014033589512109756,
+          0.007064620964229107,
+          0.008000649511814117,
+          -0.0039115045219659805,
+          0.021412497386336327,
+          0.009408107027411461,
+          0.0008587374468334019,
+          0.002954979194328189,
+          -0.0030608803499490023,
+          -0.018419940024614334,
+          -0.005691324360668659,
+          -0.0003687318821903318,
+          0.0046767243184149265,
+          0.0023229895159602165,
+          -0.01505843736231327,
+          0.020988892763853073,
+          -0.0016773350071161985,
+          -0.005534180905669928,
+          0.025429902598261833,
+          0.011123020201921463,
+          -0.0039832438342273235,
+          -0.012468988075852394,
+          -0.012018054723739624,
+          -0.004984179046005011,
+          0.0009881245205178857,
+          0.005383869633078575,
+          0.012175197713077068,
+          0.003908087965101004,
+          -0.006350643467158079,
+          0.005735734477639198,
+          -0.0017746955854818225,
+          0.00561275240033865,
+          0.016383908689022064,
+          0.009838543832302094,
+          0.008007481694221497,
+          -0.021180197596549988,
+          -0.015017443336546421,
+          0.015331730246543884,
+          0.005636665504425764,
+          0.005257471930235624,
+          0.00204286421649158,
+          0.024664683267474174,
+          0.02026466652750969,
+          -0.029734265059232712,
+          -0.011444139294326305,
+          -0.012284515425562859,
+          0.03503614664077759,
+          0.015946639701724052,
+          -0.016588877886533737,
+          0.01174476183950901,
+          0.0011068361345678568,
+          0.004492251668125391,
+          -0.0038500134833157063,
+          0.009660903364419937,
+          0.011006870307028294,
+          -0.021822435781359673,
+          -0.028285812586545944,
+          -0.02064727619290352,
+          -0.013323027640581131,
+          0.032658498734235764,
+          0.02529325522482395,
+          -0.02410443127155304,
+          0.004909023176878691,
+          0.03413427993655205,
+          -0.00540095055475831,
+          -0.009654071182012558,
+          0.006104679778218269,
+          -0.00011123873991891742,
+          -0.01479880977421999,
+          -0.008793198503553867,
+          -0.021330509334802628,
+          0.014525516889989376,
+          0.0034691113978624344,
+          0.005090079735964537,
+          -0.004167716484516859,
+          -0.011997557245194912,
+          -0.027465933933854103,
+          -0.009025497362017632,
+          -0.002867867238819599,
+          -0.0003439646970946342,
+          -0.004475170746445656,
+          0.012967746704816818,
+          -0.0011307492386549711,
+          0.004960265941917896,
+          0.001984789501875639,
+          -0.015031108632683754,
+          -0.007823009043931961,
+          -0.027301957830786705,
+          -0.016329249367117882,
+          0.0034810679499059916,
+          -0.013186381198465824,
+          0.004543493967503309,
+          -0.008929844945669174,
+          -0.014484522864222527,
+          -0.010610596276819706,
+          0.011286996304988861,
+          -0.006425798870623112,
+          0.030772779136896133,
+          0.008314935490489006,
+          -0.011464636772871017,
+          0.0015919309807941318,
+          -0.0022051320411264896,
+          -0.023011259734630585,
+          0.019540440291166306,
+          0.005541013088077307,
+          0.009633573703467846,
+          -0.031374022364616394,
+          0.004471754655241966,
+          0.012933585792779922,
+          0.006531700026243925,
+          -0.023065919056534767,
+          -0.010808733291924,
+          -0.0007246531313285232,
+          -0.0035323104821145535,
+          -0.018542921170592308,
+          -0.014675827696919441,
+          -0.009373946115374565,
+          -0.012065880931913853,
+          -0.020729264244437218,
+          -0.019376466050744057,
+          0.008984503336250782,
+          0.00585529999807477,
+          0.009592579677700996,
+          -0.0030420913826674223,
+          0.007645368110388517,
+          0.013159052468836308,
+          0.0050012595020234585,
+          0.022614985704421997,
+          0.004707470070570707,
+          0.0037270316388458014,
+          0.012899423949420452,
+          -0.018433604389429092,
+          -0.004516164772212505,
+          -0.0016730648931115866,
+          0.02138516865670681,
+          0.007003129925578833,
+          -0.0001629081816645339,
+          0.0004842408234253526,
+          0.02305225469172001,
+          0.0030403833370655775,
+          0.01574167050421238,
+          0.013343525119125843,
+          -0.00589287793263793,
+          0.03216657042503357,
+          -0.00024382848641835153,
+          -0.0004850948753301054,
+          -0.002987432759255171,
+          0.021795107051730156,
+          -0.005845051258802414,
+          0.0004086582630407065,
+          0.013760296627879143,
+          -0.01205221563577652,
+          -0.033095769584178925,
+          -0.005387285724282265,
+          0.026919348165392876,
+          0.015236077830195427,
+          -0.020169014111161232,
+          -0.01000935211777687,
+          0.0016312168445438147,
+          -0.005800641607493162,
+          0.015181419439613819,
+          -0.0019164662808179855,
+          0.00870437826961279,
+          0.017723042517900467,
+          -0.008048475719988346,
+          -0.020155349746346474,
+          0.027834879234433174,
+          0.02732928842306137,
+          0.010193824768066406,
+          -0.038452308624982834,
+          -0.014853468164801598,
+          0.010938547551631927,
+          -0.016670865938067436,
+          -0.019704416394233704,
+          -0.014006259851157665,
+          -0.014730486087501049,
+          -0.007536051329225302,
+          0.01694415882229805,
+          0.008328600786626339,
+          -0.004136971198022366,
+          0.020510630682110786,
+          0.026646055281162262,
+          0.004587904084473848,
+          -0.0035767205990850925,
+          -0.017422420904040337,
+          -0.006593190599232912,
+          -0.0208659116178751,
+          0.0065112030133605,
+          -0.022423680871725082,
+          -0.012906256131827831,
+          -0.010111836716532707,
+          0.002377648139372468,
+          0.02809450775384903,
+          0.005482938606292009,
+          0.03110072948038578,
+          0.0063643078319728374,
+          -0.009353448636829853,
+          -0.008526737801730633,
+          -0.0039729950949549675,
+          -0.0022717469837516546,
+          -0.013548494316637516,
+          -0.007986984215676785,
+          -0.0010163078550249338,
+          -0.005725485738366842,
+          0.012359670363366604,
+          0.016206268221139908,
+          0.004543493967503309,
+          0.025197602808475494,
+          -2.132431836798787e-05,
+          -0.002203423762694001,
+          0.025402573868632317,
+          -0.01590564474463463,
+          0.003627962898463011,
+          0.012475820258259773,
+          -0.0021282683592289686,
+          -0.014880796894431114,
+          0.011198176071047783,
+          0.0325491838157177,
+          -0.014566510915756226,
+          -0.011983892880380154,
+          -0.011983892880380154,
+          0.012332341633737087,
+          -0.004229207523167133,
+          0.03517279401421547,
+          0.021426161751151085,
+          -0.03533677011728287,
+          0.0027226803358644247,
+          -0.019034849479794502,
+          -0.0017951925983652472,
+          -0.023817474022507668,
+          0.0020240754820406437,
+          0.015687011182308197,
+          0.021494485437870026,
+          -0.002029199618846178,
+          0.006524867378175259,
+          -0.02130318060517311,
+          0.01299507636576891,
+          0.008677048608660698,
+          -0.0003403350419830531,
+          0.010665254667401314,
+          -0.0038500134833157063,
+          0.007358410861343145,
+          -0.011109355837106705,
+          -0.013152220286428928,
+          -0.002973768161609769,
+          0.01612428016960621,
+          0.01989572122693062,
+          -0.007713691331446171,
+          0.03028085082769394,
+          -0.002451095497235656,
+          -0.006360891740769148,
+          -0.0368945375084877,
+          -0.0032180235721170902,
+          -0.003962746821343899,
+          0.006705923937261105,
+          -0.0013434053398668766,
+          -0.011164014227688313,
+          -0.011034199967980385,
+          -0.015946639701724052,
+          -0.0030079297721385956,
+          0.004885110072791576,
+          -0.0021026472095400095,
+          -0.0053804535418748856,
+          0.002242709742859006,
+          -0.006029524374753237,
+          0.0054897707886993885,
+          -0.015003778971731663,
+          0.0053087142296135426,
+          0.014334211125969887,
+          -0.006705923937261105,
+          0.0207019355148077,
+          0.022683309391140938,
+          0.01859758049249649,
+          0.0016030335100367665,
+          0.019745411351323128,
+          -0.011020535603165627,
+          -0.017436085268855095,
+          -0.01046711765229702,
+          0.01740875653922558,
+          -0.008314935490489006,
+          0.01852925680577755,
+          0.013685140758752823,
+          0.008888850919902325,
+          0.0005380453658290207,
+          -0.013760296627879143,
+          -0.00783667340874672,
+          0.0009069907246157527,
+          0.01572800613939762,
+          0.01889820210635662,
+          0.004393183160573244,
+          0.0009753139456734061,
+          -0.018624909222126007,
+          0.02200007624924183,
+          -0.015618688426911831,
+          0.002056528814136982,
+          -0.004727966617792845,
+          -0.001228109933435917,
+          0.014088247902691364,
+          -0.007727356161922216,
+          0.008950341492891312,
+          -0.02034665457904339,
+          0.011840414255857468,
+          -0.01117767859250307,
+          -0.0033444215077906847,
+          -0.00602610781788826,
+          -0.012844765558838844,
+          -0.01980006881058216,
+          -0.02508828602731228,
+          0.007085117977112532,
+          -0.0023878966458141804,
+          0.019554104655981064,
+          -0.007631703745573759,
+          -0.020811252295970917,
+          -0.014675827696919441,
+          -0.007986984215676785,
+          0.012223023921251297,
+          0.01747708022594452,
+          -0.00873854011297226,
+          -0.018010001629590988,
+          0.029843581840395927,
+          -0.013992595486342907,
+          -0.008970838971436024,
+          -0.01438887044787407,
+          -0.01672552339732647,
+          -0.0017798198387026787,
+          0.006186667364090681,
+          -0.012585137039422989,
+          -0.014675827696919441,
+          0.014279552735388279,
+          0.0023246975615620613,
+          -0.01649322547018528,
+          -0.016698194667696953,
+          0.005281385034322739,
+          -0.011246002279222012,
+          0.02205473557114601,
+          -0.015700675547122955,
+          0.016520554199814796,
+          -0.011450971476733685,
+          0.01596030406653881,
+          -0.02093423530459404,
+          -0.01973174512386322,
+          0.005834802985191345,
+          -0.011799420230090618,
+          0.029488300904631615,
+          0.001263125566765666,
+          -0.017887018620967865,
+          0.008991335518658161,
+          -0.015290736220777035,
+          0.023544181138277054,
+          -0.0016824592603370547,
+          0.01516775507479906,
+          -0.013220543041825294,
+          0.014511851593852043,
+          0.0012964331544935703,
+          0.014853468164801598,
+          0.008198786526918411,
+          0.038588955998420715,
+          -0.006572693586349487,
+          -0.008943509310483932,
+          0.0008578834240324795,
+          0.008157792501151562,
+          0.0027551339007914066,
+          0.007358410861343145,
+          0.024077102541923523,
+          -0.0012938709696754813,
+          0.005575174931436777,
+          -0.01904851384460926,
+          -0.015290736220777035,
+          -0.007317416835576296,
+          0.015359059907495975,
+          0.017818694934248924,
+          -0.011116188019514084,
+          -0.006784495897591114,
+          0.0055273487232625484,
+          -0.023380205035209656,
+          0.010221153497695923,
+          -0.016247261315584183,
+          -0.013589488342404366,
+          -0.010173327289521694,
+          -0.005964617244899273,
+          -0.008540402166545391,
+          0.011047864332795143,
+          0.014252224005758762,
+          -0.025402573868632317,
+          0.01204538345336914,
+          0.016301920637488365,
+          -0.00626865541562438,
+          0.025129280984401703,
+          -0.027547921985387802,
+          -0.01121867261826992,
+          -0.0023520267568528652,
+          0.011512462981045246,
+          -0.0012725199339911342,
+          -0.006432631053030491,
+          0.009592579677700996,
+          -0.012332341633737087,
+          -0.013691972941160202,
+          -0.002029199618846178,
+          -0.006248158402740955,
+          -0.011170846410095692,
+          0.00023870424774941057,
+          -0.0006435193354263902,
+          0.0065521965734660625,
+          0.012414328753948212,
+          0.00544194458052516,
+          -0.00429411418735981,
+          -0.00956525094807148,
+          0.019485782831907272,
+          0.02463735267519951,
+          0.01596030406653881,
+          0.005510267801582813,
+          -0.01604229211807251,
+          -0.00433169212192297,
+          -0.008547235280275345,
+          -0.02545723132789135,
+          -0.014539181254804134,
+          0.01171059999614954,
+          0.0200323686003685,
+          0.0028217490762472153,
+          -0.02839512936770916,
+          -0.004932936280965805,
+          0.0008907640003599226,
+          0.020291997119784355,
+          0.0019677088130265474,
+          -0.004700637422502041,
+          0.004078896250575781,
+          0.008232948370277882,
+          0.0100571783259511,
+          -0.027069659903645515,
+          -0.015850987285375595,
+          -0.010344135574996471,
+          -0.025511890649795532,
+          0.022164052352309227,
+          0.02817649580538273,
+          0.0134391775354743,
+          0.004106225445866585,
+          0.013500668108463287,
+          0.013773960992693901,
+          0.031975265592336655,
+          -0.00647704117000103,
+          0.01754540205001831,
+          6.079698869143613e-05,
+          -0.007802511565387249,
+          0.018952861428260803,
+          -0.00436585396528244,
+          -0.011758426204323769,
+          -0.013111226260662079,
+          0.010152830742299557,
+          -0.0066615138202905655,
+          -0.008403755724430084,
+          -0.019321806728839874,
+          0.012257185764610767,
+          0.014443528838455677,
+          -0.00865655206143856,
+          -0.010590098798274994,
+          0.004956849385052919,
+          0.007392572239041328,
+          0.004024237859994173,
+          -0.0017217451240867376,
+          -0.0067810798063874245,
+          -0.017067139968276024,
+          -0.005404366645962,
+          -0.0146894920617342,
+          -0.009127981960773468,
+          -0.004174548666924238,
+          0.015113096684217453,
+          -0.017067139968276024,
+          0.018884537741541862,
+          0.014593839645385742,
+          0.014470857568085194,
+          0.019977709278464317,
+          -0.008048475719988346,
+          0.010050345212221146,
+          0.01754540205001831,
+          -0.0014373497106134892,
+          0.0019113420275971293,
+          -0.005605920217931271,
+          0.0007951114675961435,
+          0.008055307902395725,
+          0.0027841711416840553,
+          0.0011162306182086468,
+          -0.003083085408434272,
+          -0.025853507220745087,
+          -0.01876155659556389,
+          -0.0037782741710543633,
+          -0.020196344703435898,
+          -0.011430474929511547,
+          0.003417869098484516,
+          0.0028012520633637905,
+          -0.007167105562984943,
+          -0.012530478648841381,
+          0.0022512502036988735,
+          -0.004495667759329081,
+          0.006658097729086876,
+          -0.0031138306949287653,
+          -0.009052827022969723,
+          0.010672086849808693,
+          0.013138554990291595,
+          -0.016534218564629555,
+          -0.015796327963471413,
+          0.005110576748847961,
+          0.0030386752914637327,
+          0.018870873376727104,
+          -0.0013519456842914224,
+          -0.003747528651729226,
+          -0.008601893670856953,
+          -0.001989913871511817,
+          -0.010521776042878628,
+          -0.007003129925578833,
+          0.008759036660194397,
+          0.016288256272673607,
+          0.005568342283368111,
+          -0.024077102541923523,
+          0.030007557943463326,
+          -0.006559029221534729,
+          0.038534294813871384,
+          0.005564926192164421,
+          -0.005674243438988924,
+          -0.0014313714345917106,
+          0.00956525094807148,
+          -0.011170846410095692,
+          0.012387000024318695,
+          0.009168975986540318,
+          0.010023016482591629,
+          -0.006572693586349487,
+          -0.007057788781821728,
+          0.002792711602523923,
+          0.009715561755001545,
+          -0.0008459268719889224,
+          9.453157690586522e-05,
+          -0.016684530302882195,
+          -0.014320546761155128,
+          -0.008000649511814117,
+          0.026085805147886276,
+          0.005978281609714031,
+          0.004188213497400284,
+          -0.006299400702118874,
+          -0.007986984215676785,
+          -0.0112255048006773,
+          -0.0046152337454259396,
+          -0.03418894112110138,
+          0.011157182045280933,
+          -0.028258483856916428,
+          0.004587904084473848,
+          0.007023626938462257,
+          -5.860851160832681e-05,
+          0.002843954134732485,
+          0.0063028172589838505,
+          0.010091339237987995,
+          0.014101912267506123,
+          0.00741990189999342,
+          -0.013534829951822758,
+          0.0015987633960321546,
+          0.017490744590759277,
+          -0.0004169851599726826,
+          0.0012921629240736365,
+          -0.009175808168947697,
+          -0.016657201573252678,
+          -0.0012981412000954151,
+          -0.016069620847702026,
+          0.002094106748700142,
+          0.010979541577398777,
+          -0.000233366503380239,
+          0.005250639282166958,
+          0.01027581188827753,
+          0.004550326615571976,
+          -0.007235428784042597,
+          6.223818490980193e-05,
+          0.010227985680103302,
+          0.012557808309793472,
+          -0.00457765581086278,
+          -0.016602542251348495,
+          0.015687011182308197,
+          0.016383908689022064,
+          0.02040131390094757,
+          0.0016371951205655932,
+          -0.019718080759048462,
+          -0.0025484561920166016,
+          0.009326119907200336,
+          -0.009346616454422474,
+          0.01966342329978943,
+          0.0013510916614905,
+          0.006012443453073502,
+          0.02146715484559536,
+          0.015482041984796524,
+          -0.009585747495293617,
+          -0.014470857568085194,
+          0.005093495827168226,
+          -0.02327088825404644,
+          -0.0005871526664122939,
+          0.011150349862873554,
+          -0.031073400750756264,
+          0.01627459190785885,
+          -0.020975228399038315,
+          0.012209359556436539,
+          -1.091570175049128e-05,
+          -0.022847285494208336,
+          -0.020087026059627533,
+          0.012837933376431465,
+          0.022095728665590286,
+          0.021193861961364746,
+          -0.0077341883443295956,
+          -0.024719340726733208,
+          -0.007119279354810715,
+          0.02674170769751072,
+          0.006972384639084339,
+          -0.016233596950769424,
+          0.025798847898840904,
+          -0.0075087216682732105,
+          -0.030827436596155167,
+          -0.006125176791101694,
+          -0.02764357440173626,
+          -0.007201267406344414,
+          -0.009025497362017632,
+          0.018706897273659706,
+          -0.027315624058246613,
+          -0.009442268870770931,
+          0.014402534812688828,
+          -0.005076415371149778,
+          -0.00630623335018754,
+          -0.00134169717784971,
+          0.014006259851157665,
+          0.003243644954636693,
+          -0.006808409001678228,
+          -0.0030557559803128242,
+          0.014170235954225063,
+          0.015714339911937714,
+          -0.01559135876595974,
+          -0.009018665179610252,
+          -0.013637314550578594,
+          -0.009237299673259258,
+          0.025716859847307205,
+          -0.00025087432004511356,
+          0.0022205046843737364,
+          -0.011717432178556919,
+          -0.008888850919902325,
+          0.007515554316341877,
+          0.02025100216269493,
+          0.00602269172668457,
+          0.001989913871511817,
+          0.014101912267506123,
+          -0.0017798198387026787,
+          -0.011539791710674763,
+          -1.084230825654231e-05,
+          0.002377648139372468,
+          0.01269445475190878,
+          -0.009770220145583153,
+          0.020455971360206604,
+          -0.007167105562984943,
+          0.014006259851157665,
+          -0.022614985704421997,
+          0.007276422809809446,
+          -0.019185159355401993,
+          -0.008431085385382175,
+          -0.0012964331544935703,
+          -0.004444425459951162,
+          0.00139293959364295,
+          -0.011389480903744698,
+          -0.0032077752985060215,
+          0.0099751902744174,
+          0.008123630657792091,
+          -0.025019962340593338,
+          -0.0012938709696754813,
+          -0.01597396843135357,
+          -0.015932975336909294,
+          -0.013514332473278046,
+          0.014662163332104683,
+          0.015632351860404015,
+          0.0028337056282907724,
+          0.0020462805405259132,
+          -0.020291997119784355,
+          -0.011457803659141064,
+          0.009749723598361015,
+          -0.011587617918848991,
+          0.006319897715002298,
+          -0.0191168375313282,
+          -0.022492004558444023,
+          0.01035096775740385,
+          0.04465605691075325,
+          -0.0030096378177404404,
+          -0.015523036010563374,
+          0.013828619383275509,
+          -0.018843544647097588,
+          -0.012632963247597218,
+          -0.009824879467487335,
+          -0.010111836716532707,
+          -0.005769895855337381,
+          -0.0016149900620803237,
+          0.0033000116236507893,
+          0.02522493340075016,
+          -0.0005888607702217996,
+          0.010665254667401314,
+          0.004195045679807663,
+          0.01762739010155201,
+          -0.005947536323219538,
+          0.0009744599228724837,
+          -0.002644108608365059,
+          0.00280466815456748,
+          0.01597396843135357,
+          0.00010648814350133762,
+          0.005674243438988924,
+          0.008376426994800568,
+          0.0011708892416208982,
+          0.015345395542681217,
+          -0.0068972292356193066,
+          -0.025416238233447075,
+          0.02101622335612774,
+          -0.009018665179610252,
+          -0.004635730292648077,
+          -0.011778922751545906,
+          -0.01550937071442604,
+          0.035965342074632645,
+          0.003768025664612651,
+          -0.008964006789028645,
+          0.018570251762866974,
+          0.002572369296103716,
+          0.0058860452845692635,
+          0.007050956133753061,
+          -0.01876155659556389,
+          -0.010173327289521694,
+          0.018392611294984818,
+          -0.011949731037020683,
+          -0.013541662134230137,
+          0.01257830485701561,
+          -0.0016303628217428923,
+          0.007604374550282955,
+          0.0015031108632683754,
+          -0.013015573844313622,
+          -0.009005000814795494,
+          -0.006261823233217001,
+          0.013193213380873203,
+          -0.0048680296167731285,
+          0.018010001629590988,
+          0.00786400306969881,
+          0.008888850919902325,
+          -0.007727356161922216,
+          0.022259704768657684,
+          -0.020510630682110786,
+          -0.007399404887109995,
+          0.0033375893253833055,
+          0.021070880815386772,
+          -0.014019925147294998,
+          0.013008740730583668,
+          -0.0021880511194467545,
+          -0.010631092824041843,
+          0.0033973720856010914,
+          0.0004987595020793378,
+          0.0009172392310574651,
+          -0.0007302043959498405,
+          0.012762777507305145,
+          -0.0029310660902410746,
+          0.002444263081997633,
+          -0.018802549690008163,
+          0.025921829044818878,
+          -0.008724874816834927,
+          0.013384519144892693,
+          -0.007891331799328327,
+          0.007003129925578833,
+          0.02363983355462551,
+          0.03304110839962959,
+          0.0029242339078336954,
+          0.008444749750196934,
+          -0.0029037368949502707,
+          0.01602862775325775,
+          0.004024237859994173,
+          0.00609784759581089,
+          0.007454063277691603,
+          0.014675827696919441,
+          -0.017203787341713905,
+          -0.020660942420363426,
+          0.013316195458173752,
+          -0.015605023130774498,
+          -0.012188862077891827,
+          -0.016165273264050484,
+          -0.00873854011297226,
+          0.018802549690008163,
+          0.0215354785323143,
+          -0.0015210456913337111,
+          0.006142257712781429,
+          0.016465896740555763,
+          -0.007536051329225302,
+          -0.00108206900767982,
+          -0.0013177840737625957,
+          0.006869899574667215,
+          -0.00034887544461525977,
+          -0.016438566148281097,
+          0.01694415882229805,
+          -0.02273796685039997,
+          0.00948326289653778,
+          -0.007146609015762806,
+          0.0061935000121593475,
+          0.014224894344806671,
+          0.01295408234000206,
+          0.015523036010563374,
+          0.015181419439613819,
+          0.0055615101009607315,
+          0.006753750145435333,
+          -0.005653746426105499,
+          0.012981412000954151,
+          0.009162143804132938,
+          0.00820561870932579,
+          0.014129241928458214,
+          -0.0027380529791116714,
+          -0.01934913545846939,
+          -0.005749398842453957,
+          0.01882988028228283,
+          -0.007966487668454647,
+          -0.009244131855666637,
+          0.0139379370957613,
+          0.023680828511714935,
+          -0.021658461540937424,
+          0.02649574540555477,
+          -0.0031428681686520576,
+          -0.014484522864222527,
+          0.035063475370407104,
+          -0.00564691424369812,
+          0.01949944719672203,
+          -0.021439826115965843,
+          -0.037003856152296066,
+          -0.00477920938283205,
+          0.0028337056282907724,
+          0.016752853989601135,
+          0.005113992840051651,
+          0.0012195694725960493,
+          -0.018037330359220505,
+          -0.006849403027445078,
+          0.01672552339732647,
+          -0.01587831601500511,
+          0.01434787642210722,
+          -0.0039729950949549675,
+          0.0207019355148077,
+          0.0014740735059604049,
+          0.015331730246543884,
+          -0.012188862077891827,
+          -0.007570212706923485,
+          0.030253522098064423,
+          0.005178899969905615,
+          -0.010788236744701862,
+          0.017367763444781303,
+          -0.0043795183300971985,
+          -0.002268330892547965,
+          -0.013275201432406902,
+          0.006572693586349487,
+          -0.0007169668097048998,
+          0.0020992308855056763,
+          0.004652811214327812,
+          -0.003959330730140209,
+          0.02230069972574711,
+          -0.005998778622597456,
+          0.010002519004046917,
+          0.005100328475236893,
+          -0.006104679778218269,
+          0.0055717583745718,
+          0.01589198037981987,
+          -0.02115286886692047,
+          0.010528608225286007,
+          -0.0064087179489433765,
+          0.006289152428507805,
+          -0.01175159402191639,
+          0.00022632066975347698,
+          -0.0070304591208696365,
+          0.004755296278744936,
+          0.01178575586527586,
+          -0.013275201432406902,
+          -0.0032795146107673645,
+          0.007522386498749256,
+          -0.004611817188560963,
+          0.025197602808475494,
+          0.021658461540937424,
+          -0.013124890625476837,
+          -0.013664644211530685,
+          0.01382178720086813,
+          -0.003284638747572899,
+          0.006371140480041504,
+          0.005684491712599993,
+          0.004382934421300888,
+          -0.034899499267339706,
+          -0.012018054723739624,
+          0.004099393263459206,
+          -0.016957823187112808,
+          0.001854975474998355,
+          -0.009223634377121925,
+          -0.024282071739435196,
+          0.022396352142095566,
+          -0.004266784992069006,
+          0.021125540137290955,
+          0.0074472310952842236,
+          -0.011840414255857468,
+          -0.016780182719230652,
+          0.014880796894431114,
+          -0.01649322547018528,
+          -0.0020701936446130276,
+          -0.001913050189614296,
+          -0.020223673433065414,
+          0.015345395542681217,
+          -0.004150635562837124,
+          0.010863391682505608,
+          0.001276790164411068,
+          -0.01269445475190878,
+          -0.0036177143920212984,
+          -0.0077341883443295956,
+          0.008301271125674248,
+          -0.018283294513821602,
+          0.0071397763676941395,
+          0.011526127345860004,
+          -0.002435722853988409,
+          -0.019267147406935692,
+          -0.00782984122633934,
+          -2.091064197884407e-05,
+          0.03670323267579079,
+          -0.0261268001049757,
+          -0.0024015612434595823,
+          -7.245997949212324e-06,
+          -0.012373334728181362,
+          0.0035015649627894163,
+          -0.009271460585296154,
+          -0.0037987709511071444,
+          -0.01964975893497467,
+          -0.006617103703320026,
+          0.007501889485865831,
+          0.008000649511814117,
+          -0.013288866728544235,
+          -0.015482041984796524,
+          -0.010453452356159687,
+          -0.025033628568053246,
+          -0.008718042634427547,
+          -0.02967960759997368,
+          -0.017518073320388794,
+          0.0018652238650247455,
+          -0.002572369296103716,
+          0.02235535718500614,
+          -0.019485782831907272,
+          0.010098172351717949,
+          -0.00495001720264554,
+          0.018939197063446045,
+          -0.008595061488449574,
+          -0.004031070042401552,
+          0.006863067392259836,
+          0.0048680296167731285,
+          0.0009257796336896718,
+          -0.008232948370277882,
+          -0.003805603366345167,
+          -0.00786400306969881,
+          -0.00960624497383833,
+          0.013985763303935528,
+          -0.006319897715002298,
+          0.008977671153843403,
+          -0.017190122976899147,
+          -0.000306386937154457,
+          -0.012585137039422989,
+          -0.008048475719988346,
+          -0.009442268870770931,
+          0.003347837831825018,
+          0.014279552735388279,
+          -0.009537921287119389,
+          -0.03774174675345421,
+          -0.0019010936375707388,
+          -0.0051549868658185005,
+          0.0034298256505280733,
+          -0.01874789223074913,
+          0.015782663598656654,
+          -0.011423642747104168,
+          0.011170846410095692,
+          0.011614947579801083,
+          0.005086663644760847,
+          0.008116798475384712,
+          0.006422382779419422,
+          0.015386389568448067,
+          -0.0011025660205632448,
+          -0.004611817188560963,
+          -0.02318890020251274,
+          0.018460934981703758,
+          -0.01249631680548191,
+          -0.008014313876628876,
+          -0.008403755724430084,
+          0.041841138154268265,
+          0.02266964502632618,
+          0.01212053932249546,
+          0.002522834809496999,
+          0.02891438640654087,
+          0.0031240792013704777,
+          0.018132982775568962,
+          0.011813084594905376,
+          -0.004973930306732655,
+          -0.006418966688215733,
+          -0.01295408234000206,
+          -0.029460972175002098,
+          0.016957823187112808,
+          0.008232948370277882,
+          0.029925569891929626,
+          -0.028504448011517525,
+          0.015031108632683754,
+          -0.005462441593408585,
+          -0.01092488318681717,
+          -0.010023016482591629,
+          0.008519905619323254,
+          -0.0024596359580755234,
+          0.031592655926942825,
+          0.00911431759595871,
+          -0.0029515631031244993,
+          0.008191954344511032,
+          0.013145387172698975,
+          -0.012885759584605694,
+          0.01077457144856453,
+          0.01574167050421238,
+          0.03960013762116432,
+          0.009886370040476322,
+          2.0523655621218495e-05,
+          0.035118136554956436,
+          0.016014963388442993,
+          -0.0036792054306715727,
+          -0.0074745602905750275,
+          -0.016206268221139908,
+          -0.012325508520007133,
+          -0.006439463701099157,
+          0.01988205686211586,
+          -0.005602504126727581,
+          0.009162143804132938,
+          0.007563380524516106,
+          0.00783667340874672,
+          -0.02899637445807457,
+          0.013903775252401829,
+          0.003149700351059437,
+          0.013808122836053371,
+          0.014730486087501049,
+          -0.023475859314203262,
+          0.011977060697972775,
+          -0.02176777832210064,
+          -0.0035903851967304945,
+          -0.001383545226417482,
+          -0.0008151814108714461,
+          0.01792801357805729,
+          -0.023079583421349525,
+          -0.013773960992693901,
+          -0.004673308227211237,
+          0.014525516889989376,
+          0.008007481694221497,
+          0.0103919617831707,
+          0.012188862077891827,
+          0.002838829765096307,
+          -0.005377037450671196,
+          -0.0062549905851483345,
+          0.0018788885790854692,
+          -0.003733864054083824,
+          -0.002818332752212882,
+          -0.006497538182884455,
+          0.009018665179610252,
+          0.006091014947742224,
+          -0.004977346397936344,
+          0.009503760375082493,
+          -0.01627459190785885,
+          -0.013596320524811745,
+          0.02582617662847042,
+          0.002526251133531332,
+          -0.0013135139597579837,
+          0.005295049399137497,
+          0.010952211916446686,
+          0.01475781574845314,
+          0.009770220145583153,
+          0.004013989120721817,
+          0.013241039589047432,
+          -0.005595671944320202,
+          -0.007009962573647499,
+          0.019690752029418945,
+          0.0031052904669195414,
+          -0.018406275659799576,
+          0.0026031145825982094,
+          -0.010870223864912987,
+          -0.0012272557942196727,
+          0.010139165446162224,
+          -0.010405626147985458,
+          -0.015796327963471413,
+          0.0011102523421868682,
+          -0.02275163307785988,
+          -0.004645979031920433,
+          0.01709447056055069,
+          0.0073310816660523415,
+          0.024268407374620438,
+          0.002584325848147273,
+          -0.005985114257782698,
+          -0.0006883564637973905,
+          -0.024159090593457222,
+          0.005284801125526428,
+          -0.003614298300817609,
+          -0.008027978241443634,
+          0.014402534812688828,
+          -0.001230672001838684,
+          0.00241010170429945,
+          -0.011123020201921463,
+          -0.00862239021807909,
+          0.006412134040147066,
+          0.012284515425562859,
+          0.03172930330038071,
+          -0.004266784992069006,
+          -0.008718042634427547,
+          -0.029570288956165314,
+          -0.003781690262258053,
+          0.024364059790968895,
+          -0.003819267963990569,
+          -0.020756594836711884,
+          0.01951311156153679,
+          0.02537524327635765,
+          -0.0008514781366102397,
+          -0.0011316033778712153,
+          -0.007153441198170185,
+          -0.004891942720860243,
+          0.007918661460280418,
+          0.019403794780373573,
+          -0.018337951973080635,
+          0.010439787991344929,
+          0.014976450242102146,
+          0.03817901387810707,
+          0.0025006297510117292,
+          -0.004543493967503309,
+          -0.015523036010563374,
+          -0.0014843220124021173,
+          -0.018556587398052216,
+          0.02861376479268074,
+          -0.000738317787181586,
+          0.03304110839962959,
+          0.020811252295970917,
+          0.008492575958371162,
+          0.0028132086154073477,
+          -0.010808733291924,
+          0.059468530118465424,
+          -0.019089506939053535,
+          -0.007782014552503824,
+          -0.011983892880380154,
+          -0.023257223889231682,
+          -0.010419291444122791,
+          -0.0028217490762472153,
+          -0.012878927402198315,
+          0.015318065881729126,
+          -0.01649322547018528,
+          0.00585529999807477,
+          0.005981697700917721,
+          0.0011879700468853116,
+          0.008451581932604313,
+          -0.022232376039028168,
+          0.021180197596549988,
+          -0.02275163307785988,
+          -0.004400015342980623,
+          0.002869575284421444,
+          -0.0003832505608443171,
+          -0.009312454611063004,
+          0.0031104146037250757,
+          0.004693805240094662,
+          0.005455608945339918,
+          0.001044491189531982,
+          -0.01299507636576891,
+          0.024077102541923523,
+          -0.010576434433460236,
+          -0.017203787341713905,
+          0.013247872702777386,
+          0.037304479628801346,
+          0.00696896854788065,
+          0.008355929516255856,
+          0.022710638120770454,
+          0.003508397378027439,
+          -0.03910820931196213,
+          0.004618649836629629,
+          0.005144738592207432,
+          -0.012762777507305145,
+          0.023079583421349525,
+          0.003781690262258053,
+          -0.00325560150668025,
+          -0.01434787642210722,
+          -0.00412330636754632,
+          -0.0012554391287267208,
+          -0.02312057837843895,
+          0.014525516889989376,
+          0.0016021794872358441,
+          -0.018406275659799576,
+          -0.0081782890483737,
+          0.01175159402191639,
+          0.015468376688659191,
+          0.002534791361540556,
+          -0.004943185020238161,
+          0.003178737824782729,
+          0.004232623614370823,
+          -0.013951601460576057,
+          -0.00436927005648613,
+          -0.00046630596625618637,
+          -0.018488263711333275,
+          0.01306340005248785,
+          0.004837283864617348,
+          -0.01559135876595974,
+          -0.007009962573647499,
+          0.020128021016716957,
+          -0.001632070867344737,
+          0.01611061580479145,
+          -0.004058399237692356,
+          0.002451095497235656,
+          0.01777770183980465,
+          0.026318104937672615,
+          -0.005062750540673733,
+          -0.009093821048736572,
+          -0.016014963388442993,
+          -0.013726134784519672,
+          -0.006825489457696676,
+          0.02432306669652462,
+          -0.0011948023457080126,
+          -0.01129382848739624,
+          0.021877095103263855,
+          -0.0011034200433641672,
+          -0.011416809633374214,
+          -0.004659643396735191,
+          -0.0020326157100498676,
+          -0.021262185648083687,
+          0.008157792501151562,
+          -0.002565536880865693,
+          0.009893202222883701,
+          0.020087026059627533,
+          0.031292036175727844,
+          -0.004512748681008816,
+          -0.003405912546440959,
+          -0.006104679778218269,
+          0.005838219076395035,
+          0.00873854011297226,
+          -0.03033551014959812,
+          -0.012407496571540833,
+          -0.014949120581150055,
+          0.006053437478840351,
+          0.018419940024614334,
+          0.007392572239041328,
+          0.015687011182308197,
+          -0.011000038124620914,
+          -0.014921790920197964,
+          -0.02131684496998787,
+          -0.022765297442674637,
+          0.004406847525388002,
+          0.010596930980682373,
+          0.03500881791114807,
+          0.0009266336564905941,
+          -0.010521776042878628,
+          -0.009701897390186787,
+          -0.010419291444122791,
+          0.0015483750030398369,
+          0.006087598856538534,
+          -0.02605847641825676,
+          -0.012721783481538296,
+          -0.016506889835000038,
+          0.004396599251776934,
+          -0.012892591767013073,
+          -0.013357189483940601,
+          0.04971197620034218,
+          -0.006873316131532192,
+          0.0223690215498209,
+          -0.0340522937476635,
+          -0.00457423971965909,
+          -0.0007259342237375677,
+          -0.02250566892325878,
+          -0.001954043982550502,
+          -0.016397573053836823,
+          0.00914847943931818,
+          -0.022123059257864952,
+          -0.0049295201897621155,
+          -0.006466792896389961,
+          -0.005305298138409853,
+          -0.019253483042120934,
+          0.0038807590026408434,
+          -0.020141685381531715,
+          0.01641123741865158,
+          0.0034981488715857267,
+          0.004526413511484861,
+          0.00953108910471201,
+          -0.030390167608857155,
+          0.011929234489798546,
+          -0.006716172676533461,
+          0.007823009043931961,
+          0.003932001534849405,
+          -0.01249631680548191,
+          -0.012216191738843918,
+          -0.004806538578122854,
+          -0.009278293699026108,
+          0.003733864054083824,
+          -0.0032197318505495787,
+          0.018050994724035263,
+          -0.018255963921546936,
+          -0.008895683102309704,
+          -0.004762128461152315,
+          0.019185159355401993,
+          0.01671185903251171,
+          -0.010043513029813766,
+          -0.005127657670527697,
+          -0.02063361182808876,
+          0.025101950392127037,
+          0.005277968943119049,
+          -0.007570212706923485,
+          0.02567586675286293,
+          0.007980152033269405,
+          0.0001476422039559111,
+          0.014142906293272972,
+          0.01739509217441082,
+          -0.017299439758062363,
+          -0.0012588553363457322,
+          0.00824661273509264,
+          0.00758387753739953,
+          -0.00033094058744609356,
+          0.0018088571960106492,
+          0.010412458330392838,
+          0.008232948370277882,
+          -0.004078896250575781,
+          -0.007249093614518642,
+          -0.029324326664209366,
+          -0.008191954344511032,
+          -0.0005337751354090869,
+          0.00011561570136109367,
+          0.016055956482887268,
+          -0.0020906904246658087,
+          0.0054077827371656895,
+          -0.024432383477687836,
+          -0.015236077830195427,
+          0.013732966966927052,
+          -0.022874614223837852,
+          -0.018816214054822922,
+          0.005950952414423227,
+          0.003788522444665432,
+          -0.0107267452403903,
+          0.0036074661184102297,
+          -0.02297026664018631,
+          -0.02109820954501629,
+          -0.00565033033490181,
+          0.007071453146636486,
+          -0.019677087664604187,
+          0.013186381198465824,
+          -0.019294477999210358,
+          -0.030089545994997025,
+          -0.017449749633669853,
+          0.005482938606292009,
+          0.00630623335018754,
+          0.002254666294902563,
+          -0.0006567569798789918,
+          -0.009326119907200336,
+          0.002830289304256439,
+          -0.01656154915690422,
+          0.01209320966154337,
+          0.009271460585296154,
+          0.004806538578122854,
+          -0.01001618430018425,
+          0.002121435943990946,
+          -0.007126112002879381,
+          0.0031326196622103453,
+          0.010979541577398777,
+          0.026195121929049492,
+          0.009920531883835793,
+          -0.006186667364090681,
+          -0.020988892763853073,
+          -0.002758549991995096,
+          -0.013036070391535759,
+          -0.0031582408118993044,
+          0.009783885441720486,
+          0.004208710510283709,
+          -0.019540440291166306,
+          -0.016438566148281097,
+          -0.026031147688627243,
+          -0.011792588047683239,
+          -0.02327088825404644,
+          -0.004283865913748741,
+          0.025730524212121964,
+          -0.010125501081347466,
+          0.0013254705118015409,
+          0.00015564884233754128,
+          0.01626092568039894,
+          -0.0021436410024762154,
+          -0.026987671852111816,
+          0.015044772997498512,
+          -0.008519905619323254,
+          0.0028729913756251335,
+          0.006073934491723776,
+          -0.00412330636754632,
+          0.0008907640003599226,
+          -0.004242871887981892,
+          0.004140387289226055,
+          -0.014320546761155128,
+          0.005506851710379124,
+          -0.007925493642687798,
+          0.019021185114979744,
+          -0.014033589512109756,
+          -0.000921509403269738,
+          0.009059659205377102,
+          -0.0047860415652394295,
+          -0.014402534812688828,
+          -0.017941677942872047,
+          0.024118097499012947,
+          -0.023981450125575066,
+          0.049520671367645264,
+          -0.004136971198022366,
+          0.006138841155916452,
+          -0.02183610200881958,
+          0.003580136690288782,
+          -0.015714339911937714,
+          -0.013992595486342907,
+          -0.01647956110537052,
+          -0.002818332752212882,
+          0.035118136554956436,
+          0.005759647581726313,
+          0.0014843220124021173,
+          0.00045563047751784325,
+          0.004960265941917896,
+          0.03110072948038578,
+          -0.0013271785574033856,
+          0.01956777088344097,
+          -0.0056708273477852345,
+          -0.007686362136155367,
+          -0.007802511565387249,
+          0.010023016482591629,
+          0.016096951439976692,
+          -0.016069620847702026,
+          0.0019677088130265474,
+          -0.0009052826790139079,
+          0.009196305647492409,
+          0.009421772323548794,
+          0.005277968943119049,
+          -0.0048202034085989,
+          -0.009168975986540318,
+          0.00828077457845211,
+          -0.009995686821639538,
+          -0.00832176860421896,
+          -0.017723042517900467,
+          0.005124241579324007,
+          0.01027581188827753,
+          -0.0025484561920166016,
+          0.014525516889989376,
+          -0.005021756514906883,
+          0.004208710510283709,
+          0.005940704140812159,
+          -0.013985763303935528,
+          0.005318962503224611,
+          0.006019275635480881,
+          -0.030554143711924553,
+          0.02727462910115719,
+          -0.01076773926615715,
+          -0.017299439758062363,
+          -0.003011345863342285,
+          0.003069420577958226,
+          -0.004355605226010084,
+          -0.005421447567641735,
+          -0.008984503336250782,
+          -0.018884537741541862,
+          0.006429214961826801,
+          -0.007467728108167648,
+          -0.01844726875424385,
+          -0.001386107294820249,
+          -0.0024579279124736786,
+          0.035883355885744095,
+          -0.003424701513722539,
+          0.010904385708272457,
+          0.016288256272673607
+        ],
+        "title": "The Structure Of The Mixture-Of-Experts Layer",
+        "keyphrases": [
+          "Mixture-of-Experts layer",
+          "gating network",
+          "neural networks",
+          "sparsity",
+          "hierarchical MoE"
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "44767af8-7eba-44fe-a7ef-fd48aa105f77",
+      "properties": {
+        "page_content": "3 Addressing Performance Challenges 3.1 The Shrinking Batch Problem\n\nOn modern CPUs and GPUs, large batch sizes are necessary for computational efficiency, so as to amortize the overhead of parameter loads and updates. If the gating network chooses k out of n experts for each example, then for a batch of b examples, each expert receives a much smaller batch of approximately kb n b examples. This causes a naive MoE implementation to become very inefficient as the number of experts increases. The solution to this shrinking batch problem is to make the original batch size as large as possible. However, batch size tends to be limited by the memory necessary to store activations between the forwards and backwards passes. We propose the following techniques for increasing the batch size: Mixing Data Parallelism and Model Parallelism: In a conventional distributed training setting, multiple copies of the model on different devices asynchronously process distinct batches of data, and parameters are synchronized through a set of parameter servers. In our technique, these different batches run synchronously so that they can be combined for the MoE layer. We distribute the standard layers of the model and the gating network according to conventional data-parallel schemes, but keep only one shared copy of each expert. Each expert in the MoE layer receives a combined batch consisting of the relevant examples from all of the data-parallel input batches. The same set of devices function as data-parallel replicas (for the standard layers and the gating networks) and as model-parallel shards (each hosting a subset of the experts). If the model is distributed over d devices, and each device processes a batch of size b, each expert receives a batch of approximately kbd nexamples. Thus, we achieve a factor of d improvement in expert batch size.\n\nIn the case of a hierarchical MoE (Section B), the primary gating network employs data parallelism, and the secondary MoEs employ model parallelism. Each secondary MoE resides on one device.\n\nThis technique allows us to increase the number of experts (and hence the number of parameters) by proportionally increasing the number of devices in the training cluster. The total batch size increases, keeping the batch size per expert constant. The memory and bandwidth requirements per device also remain constant, as do the step times, as does the amount of time necessary to process a number of training examples equal to the number of parameters in the model. It is our goal to train a trillionparameter model on a trillion-word corpus. We have not scaled our systems this far as of the writing of this paper, but it should be possible by adding more hardware.\n\nTaking Advantage of Convolutionality: In our language models, we apply the same MoE to each time step of the previous layer. If we wait for the previous layer to finish, we can apply the MoE to all the time steps together as one big batch. Doing so increases the size of the input batch to the MoE layer by a factor of the number of unrolled time steps.\n\nIncreasing Batch Size for a Recurrent MoE: We suspect that even more powerful models may involve applying a MoE recurrently. For example, the weight matrices of a LSTM or other RNN could be replaced by a MoE. Sadly, such models break the convolutional trick from the last paragraph, since the input to the MoE at one timestep depends on the output of the MoE at the previous timestep. Gruslys et al. (2016) describe a technique for drastically reducing the number of stored activations in an unrolled RNN, at the cost of recomputing forward activations. This would allow for a large increase in batch size.\n\n3.2 Network Bandwidth\n\nAnother major performance concern in distributed computing is network bandwidth. Since the experts are stationary (see above) and the number of gating parameters is small, most of the communication involves sending the inputs and outputs of the experts across the network. To maintain computational efficiency, the ratio of an expert's computation to the size of its input and output must exceed the ratio of computational to network capacity of the computing device. For GPUs, this may be thousands to one. In our experiments, we use experts with one hidden layer containing thousands of RELU-activated units. Since the weight matrices in the expert have sizes input_size×hidden_size and hidden_size × output_size, the ratio of computation to input and output is equal to the size of the hidden layer. Conveniently, we can increase computational efficiency simply by using a larger hidden layer, or more hidden layers.\n\n",
+        "embedding": [
+          0.0014724871143698692,
+          -0.0002654724521562457,
+          -0.015004266053438187,
+          -0.010133732110261917,
+          0.02161158062517643,
+          0.014150978997349739,
+          -0.020856458693742752,
+          0.05349281430244446,
+          -0.016355933621525764,
+          0.02585536427795887,
+          0.004545832052826881,
+          -0.0036283594090491533,
+          0.015313866548240185,
+          0.006528026424348354,
+          -0.01309380866587162,
+          0.013184423558413982,
+          0.010216795839369297,
+          0.007860816083848476,
+          0.012504814192652702,
+          0.009748619981110096,
+          0.023076515644788742,
+          -0.03379924222826958,
+          -0.03869242966175079,
+          -0.013516676612198353,
+          -0.0013828164665028453,
+          0.015766939148306847,
+          0.0215813759714365,
+          0.00212189182639122,
+          -0.04113902524113655,
+          0.013191974721848965,
+          0.018107816576957703,
+          -0.0014149091439321637,
+          -0.03023506887257099,
+          -0.02502473071217537,
+          -0.05300953611731529,
+          -0.029328923672437668,
+          0.05705698952078819,
+          0.01131927315145731,
+          -0.023907151073217392,
+          0.005855968222022057,
+          0.03201715648174286,
+          0.013720559887588024,
+          -0.042709678411483765,
+          -0.008555527776479721,
+          0.0013847042573615909,
+          0.009076561778783798,
+          -0.02502473071217537,
+          0.017443308606743813,
+          0.015933066606521606,
+          -0.04283049702644348,
+          0.0038530079182237387,
+          -0.06312816590070724,
+          0.014974061399698257,
+          0.007317128591239452,
+          -0.04415951296687126,
+          0.0005852192407473922,
+          0.025674134492874146,
+          0.01730738766491413,
+          0.05485203489661217,
+          0.0018349455203860998,
+          0.036547884345054626,
+          0.0009023703169077635,
+          -0.05071396753191948,
+          0.006059851031750441,
+          0.00730957742780447,
+          -0.010163936764001846,
+          -0.05470100790262222,
+          0.03787690028548241,
+          -0.01833435334265232,
+          -0.007256718818098307,
+          -0.04116922989487648,
+          0.0024220526684075594,
+          0.0015055236872285604,
+          0.008336542174220085,
+          0.018711913377046585,
+          -0.010873750783503056,
+          0.04974741116166115,
+          0.04283049702644348,
+          0.01686941646039486,
+          0.0027278768830001354,
+          -0.016099192202091217,
+          0.009688210673630238,
+          -0.012217868119478226,
+          0.01082089263945818,
+          0.038269564509391785,
+          -0.012059291824698448,
+          -0.02571944147348404,
+          -0.0495963878929615,
+          -0.01418873481452465,
+          -0.005553919356316328,
+          -0.021234020590782166,
+          0.014196285977959633,
+          0.013894237577915192,
+          -0.005689841229468584,
+          -0.003618920221924782,
+          0.036245837807655334,
+          -0.012089497409760952,
+          -0.01176479458808899,
+          -0.0005918265669606626,
+          -0.008502669632434845,
+          0.00020046118879690766,
+          0.011228658258914948,
+          0.019451932981610298,
+          -0.000972219102550298,
+          0.02058461494743824,
+          0.00044788149534724653,
+          -0.009514532051980495,
+          0.00590127520263195,
+          0.02843788079917431,
+          0.10390473157167435,
+          0.015555505640804768,
+          -0.015072227455675602,
+          -0.012814413756132126,
+          0.004606241825968027,
+          -0.0477236844599247,
+          -0.010496190749108791,
+          -0.010481088422238827,
+          -0.038209155201911926,
+          -0.004406134597957134,
+          -0.001600857824087143,
+          0.019874799996614456,
+          0.020040927454829216,
+          -0.014906100928783417,
+          -0.0016169041628018022,
+          -0.07424356043338776,
+          0.03325555473566055,
+          -0.01044333167374134,
+          -0.01090395636856556,
+          -0.01703554391860962,
+          -0.03902468457818031,
+          0.005587900057435036,
+          0.017971893772482872,
+          -0.015223251655697823,
+          -0.05627166107296944,
+          0.007860816083848476,
+          0.004719510208815336,
+          0.055607154965400696,
+          0.05231482535600662,
+          0.014755076728761196,
+          0.0007310521323233843,
+          0.017911484465003014,
+          -0.040021445602178574,
+          -0.01766984537243843,
+          -0.005716270767152309,
+          -0.013614842668175697,
+          0.01789638213813305,
+          -0.002208730671554804,
+          0.04739143326878548,
+          -0.03530948609113693,
+          0.027259889990091324,
+          -0.02774316817522049,
+          0.017745357006788254,
+          -8.854980842443183e-05,
+          0.05470100790262222,
+          -0.012142355553805828,
+          -0.016386138275265694,
+          -0.02513044700026512,
+          0.005916377995163202,
+          -0.037937309592962265,
+          -0.011938472278416157,
+          -0.013388305902481079,
+          -0.02419409714639187,
+          -0.014332207851111889,
+          0.023121824488043785,
+          -0.001210082438774407,
+          0.038239359855651855,
+          0.04859962686896324,
+          -0.019980518147349358,
+          -0.08771492540836334,
+          0.021203814074397087,
+          0.005720046348869801,
+          0.007721118628978729,
+          0.01756412908434868,
+          0.026761509478092194,
+          0.017382899299263954,
+          0.02369571663439274,
+          0.016914723441004753,
+          0.01736779697239399,
+          -0.029344025999307632,
+          0.0517711378633976,
+          -0.023182233795523643,
+          -0.016189808025956154,
+          0.021128302440047264,
+          0.012663389556109905,
+          0.033134736120700836,
+          0.0027448670007288456,
+          0.006973547860980034,
+          0.028332162648439407,
+          0.011832755990326405,
+          0.011696834117174149,
+          0.03301391750574112,
+          0.030839167535305023,
+          0.018802527338266373,
+          -0.01302584819495678,
+          0.024405529722571373,
+          -0.019874799996614456,
+          -0.012504814192652702,
+          0.0034074862487614155,
+          0.016612675040960312,
+          -0.00446654437109828,
+          -0.014022608287632465,
+          -0.02960076741874218,
+          -0.021883424371480942,
+          -0.02241200953722,
+          0.02618761733174324,
+          0.033195145428180695,
+          0.019149884581565857,
+          0.026942739263176918,
+          0.03990062698721886,
+          -0.031111011281609535,
+          0.03476579859852791,
+          -0.009250239469110966,
+          -0.046122826635837555,
+          -0.04047451913356781,
+          -0.032168179750442505,
+          0.003192276693880558,
+          -0.011794999241828918,
+          0.039598576724529266,
+          -0.012376443482935429,
+          -0.03467518463730812,
+          -0.016809007152915,
+          0.017488617449998856,
+          0.009854337200522423,
+          -0.016068987548351288,
+          -0.005350036546587944,
+          -0.02041848748922348,
+          -0.00730957742780447,
+          -0.017715152353048325,
+          0.021928731352090836,
+          -0.020735640078783035,
+          -0.013373203575611115,
+          0.035853173583745956,
+          -0.0014498335076496005,
+          0.020327873528003693,
+          0.015238353982567787,
+          -0.016416342929005623,
+          0.021989140659570694,
+          0.015736734494566917,
+          -0.007807957474142313,
+          -0.010586804710328579,
+          -0.024284711107611656,
+          0.006648845970630646,
+          -0.05276789888739586,
+          -0.03703116253018379,
+          0.0047950223088264465,
+          0.012988091446459293,
+          -0.01902906410396099,
+          0.04277008771896362,
+          -0.003701983718201518,
+          0.0010835995199158788,
+          -0.0025844036135822535,
+          -0.05162011459469795,
+          0.005735148675739765,
+          0.0007291642832569778,
+          -0.003358403453603387,
+          0.03237961605191231,
+          0.012134804390370846,
+          -0.02399776503443718,
+          -0.004719510208815336,
+          -0.01756412908434868,
+          0.030023634433746338,
+          -0.004172047134488821,
+          -0.029253412038087845,
+          0.02963097207248211,
+          -0.023574896156787872,
+          -0.00843470823019743,
+          0.03497723489999771,
+          0.04434074088931084,
+          0.016718393191695213,
+          0.006101382430642843,
+          -0.005893724039196968,
+          -0.0357927642762661,
+          0.0015734846238046885,
+          -0.031171420589089394,
+          0.03947775810956955,
+          -0.02907218225300312,
+          0.0018104041228070855,
+          -0.008215723559260368,
+          0.02121891640126705,
+          0.035218872129917145,
+          0.015442237257957458,
+          0.009605146944522858,
+          -0.00013780969311483204,
+          0.01214990671724081,
+          -0.004715734627097845,
+          -0.018847836181521416,
+          -0.02094707265496254,
+          0.00012164537474745885,
+          -0.03198695182800293,
+          -0.015117534436285496,
+          -0.005882397294044495,
+          -0.017020441591739655,
+          -0.01464935950934887,
+          0.005670963320881128,
+          0.04319295659661293,
+          0.005935255903750658,
+          -0.016522061079740524,
+          0.012255623936653137,
+          0.030552219599485397,
+          -0.015125085599720478,
+          -0.044401150196790695,
+          -0.003303657053038478,
+          -0.0013507237890735269,
+          -0.021671989932656288,
+          -0.03639686107635498,
+          -0.015283661894500256,
+          -0.017549026757478714,
+          0.011606219224631786,
+          0.008646142669022083,
+          -0.007611625827848911,
+          0.0310203954577446,
+          0.012044189497828484,
+          0.014807934872806072,
+          -0.03974959999322891,
+          -0.03370862826704979,
+          0.011606219224631786,
+          0.025447597727179527,
+          0.01756412908434868,
+          -0.04557913914322853,
+          -0.04491463303565979,
+          0.03869242966175079,
+          0.017609436064958572,
+          -0.03325555473566055,
+          0.013146666809916496,
+          -0.027561938390135765,
+          -0.00012601091293618083,
+          0.020403385162353516,
+          0.06324898451566696,
+          -0.040927592664957047,
+          -0.008827371522784233,
+          -0.01830414868891239,
+          -0.021762605756521225,
+          -0.034161701798439026,
+          -0.0007310521323233843,
+          0.005889948457479477,
+          -0.04657590016722679,
+          -0.026399051770567894,
+          -0.009363507851958275,
+          -0.004451442044228315,
+          0.013131564483046532,
+          -0.01047353632748127,
+          -0.009627800434827805,
+          -0.012519916519522667,
+          0.03905488923192024,
+          0.0010958702769130468,
+          0.010564151220023632,
+          -0.05920153483748436,
+          0.015721632167696953,
+          0.014989163726568222,
+          -0.006520475260913372,
+          0.0011581677244976163,
+          0.01686941646039486,
+          0.023257745429873466,
+          0.03678952530026436,
+          0.006999977398663759,
+          -0.03023506887257099,
+          -0.015797143802046776,
+          0.025764750316739082,
+          -0.02585536427795887,
+          -0.004817675799131393,
+          0.009129419922828674,
+          -0.013879135251045227,
+          -0.026142310351133347,
+          0.02574964612722397,
+          -0.046787336468696594,
+          -0.0031431936658918858,
+          -0.02078094705939293,
+          -0.008714103139936924,
+          0.012572774663567543,
+          -0.00676966505125165,
+          -0.01786617748439312,
+          0.017684947699308395,
+          -0.006697928532958031,
+          0.00021202399511821568,
+          0.017881279811263084,
+          0.014928754419088364,
+          0.03259104862809181,
+          -0.0055388170294463634,
+          -0.04838819429278374,
+          0.033829446882009506,
+          -0.022396907210350037,
+          0.019074371084570885,
+          0.02105279080569744,
+          -0.015797143802046776,
+          0.013569535687565804,
+          0.035218872129917145,
+          -0.0023125598672777414,
+          0.008268581703305244,
+          -0.02061481960117817,
+          0.04684774577617645,
+          -0.022094858810305595,
+          -0.03301391750574112,
+          -0.010488639585673809,
+          0.004224905278533697,
+          -0.015857553109526634,
+          0.01720166951417923,
+          0.0022748038172721863,
+          -0.04603221267461777,
+          -0.006040973123162985,
+          0.006441187579184771,
+          -0.024813296273350716,
+          0.006543128751218319,
+          0.007698464673012495,
+          0.005440651439130306,
+          -0.010617010295391083,
+          -0.008744308724999428,
+          0.0021577600855380297,
+          -0.007981635630130768,
+          0.04956618323922157,
+          -0.03639686107635498,
+          -0.0022011795081198215,
+          -0.04603221267461777,
+          -0.01892334781587124,
+          -0.008427157066762447,
+          0.03811853751540184,
+          0.016129396855831146,
+          0.0040323492139577866,
+          0.018047407269477844,
+          -0.018349455669522285,
+          -0.006403431296348572,
+          -0.02990281581878662,
+          0.03376903757452965,
+          0.025961080566048622,
+          0.010043117217719555,
+          -0.01055660005658865,
+          0.0031262035481631756,
+          0.008049596101045609,
+          -0.006932016462087631,
+          0.0002463584241922945,
+          -0.021626682952046394,
+          0.020765844732522964,
+          -0.0029147693421691656,
+          0.019119679927825928,
+          -0.027984807267785072,
+          -0.012127253226935863,
+          -0.008464912883937359,
+          0.004938495345413685,
+          0.041984762996435165,
+          -0.03234941139817238,
+          -0.012157457880675793,
+          -0.00476481718942523,
+          0.002593842800706625,
+          -0.0003669419093057513,
+          0.0014762626960873604,
+          -0.03920591250061989,
+          -0.03754464536905289,
+          -0.0429513156414032,
+          -0.015268558636307716,
+          -0.01199888251721859,
+          -0.05337199568748474,
+          -0.07581420987844467,
+          0.00016659870743751526,
+          -0.0720083937048912,
+          -0.009174727834761143,
+          0.0407765656709671,
+          -0.015087329782545567,
+          -0.018319251015782356,
+          -0.03292330354452133,
+          -0.0076833623461425304,
+          -0.01736779697239399,
+          -0.01206684298813343,
+          0.004776144400238991,
+          0.016914723441004753,
+          0.006233528722077608,
+          -0.024571657180786133,
+          0.018636401742696762,
+          -0.027003148570656776,
+          -0.028422776609659195,
+          0.007158552762120962,
+          0.0037246374413371086,
+          -0.038964275270700455,
+          -0.029872611165046692,
+          0.013629944995045662,
+          0.012036638334393501,
+          0.046153031289577484,
+          0.025598622858524323,
+          -0.0014101896667852998,
+          -0.05908071622252464,
+          0.01919519156217575,
+          -0.001417740830220282,
+          0.02940443530678749,
+          0.015631018206477165,
+          -0.005384017247706652,
+          0.01677880249917507,
+          -0.016023680567741394,
+          -0.010511293075978756,
+          0.012542570009827614,
+          -0.006025870330631733,
+          -0.0011996994726359844,
+          -0.02926851436495781,
+          0.017186567187309265,
+          0.03049181029200554,
+          0.005806885194033384,
+          0.025568418204784393,
+          0.004036124795675278,
+          -0.012542570009827614,
+          -0.021128302440047264,
+          0.013645047321915627,
+          0.008540425449609756,
+          0.03159428760409355,
+          -0.03908509388566017,
+          -0.023756125941872597,
+          0.012361341156065464,
+          0.010481088422238827,
+          -0.016295524314045906,
+          -0.0004523650277405977,
+          0.01578204147517681,
+          0.021490762010216713,
+          -0.034222111105918884,
+          -0.03292330354452133,
+          0.008208171464502811,
+          0.028317060321569443,
+          0.019602956250309944,
+          -0.001834001624956727,
+          0.011674179695546627,
+          0.026731304824352264,
+          -0.0360041968524456,
+          0.0064713922329247,
+          0.012731350027024746,
+          0.034161701798439026,
+          1.8229107809020206e-05,
+          0.02393735572695732,
+          -0.02910238690674305,
+          0.03700095787644386,
+          -0.020660126581788063,
+          0.03201715648174286,
+          1.8627315512276255e-05,
+          0.011613770388066769,
+          0.03370862826704979,
+          -0.009733517654240131,
+          -0.026066798716783524,
+          0.004496749024838209,
+          0.005078192800283432,
+          -0.0035887155681848526,
+          -0.04228680953383446,
+          -0.014166081324219704,
+          0.01886293850839138,
+          0.02217037044465542,
+          -0.038632020354270935,
+          -0.011077634058892727,
+          0.006516699679195881,
+          -0.0002836425555869937,
+          0.0028675743378698826,
+          -0.003426364390179515,
+          -0.009831683710217476,
+          0.0036170324310660362,
+          0.034886617213487625,
+          -0.034252315759658813,
+          0.0032753399573266506,
+          0.018047407269477844,
+          -0.013199525885283947,
+          -0.037605054676532745,
+          -0.01055660005658865,
+          -0.0042098029516637325,
+          -0.012429301626980305,
+          -0.016189808025956154,
+          0.019497239962220192,
+          -0.013977301307022572,
+          0.011991331353783607,
+          -0.03361801430583,
+          -0.013305243104696274,
+          0.017820870503783226,
+          -0.03669891133904457,
+          0.009801478125154972,
+          -0.017292285338044167,
+          -0.007494581863284111,
+          0.009990259073674679,
+          0.03872263804078102,
+          -0.0028184913098812103,
+          0.015261007472872734,
+          0.000569172902032733,
+          -0.007853264920413494,
+          0.01453609112650156,
+          -0.013365652412176132,
+          -0.0004693552909884602,
+          0.005002680700272322,
+          0.031171420589089394,
+          -0.01244440395385027,
+          -0.0050215586088597775,
+          0.018153123557567596,
+          0.05237523466348648,
+          -0.031050600111484528,
+          0.007120796944946051,
+          0.014974061399698257,
+          0.028483187779784203,
+          -0.007471928372979164,
+          -0.003031813306733966,
+          -0.01955764926970005,
+          0.013267486356198788,
+          -0.01936131902039051,
+          0.0006673387251794338,
+          0.028226446360349655,
+          -0.02449614554643631,
+          -0.01902906410396099,
+          -0.0015527188079431653,
+          -0.02579495497047901,
+          0.0029411986470222473,
+          0.005127275828272104,
+          0.014468129724264145,
+          -0.030522014945745468,
+          -0.0007551216403953731,
+          0.044370945543050766,
+          0.014702217653393745,
+          -0.01147029735147953,
+          -0.008110006339848042,
+          0.0065997629426419735,
+          0.024239404127001762,
+          0.008125108666718006,
+          -0.0030771205201745033,
+          -0.010941712185740471,
+          0.0032621254213154316,
+          -0.011666628532111645,
+          0.020478898659348488,
+          0.004470319952815771,
+          0.030476707965135574,
+          0.008827371522784233,
+          -0.008351645432412624,
+          -0.004213578533381224,
+          -0.021762605756521225,
+          0.011485399678349495,
+          0.05926194414496422,
+          0.006750787142664194,
+          0.016084089875221252,
+          -0.016763700172305107,
+          0.009076561778783798,
+          0.0341314971446991,
+          -0.0019293357618153095,
+          -0.034886617213487625,
+          0.006263733841478825,
+          -0.008276132866740227,
+          -0.01889314316213131,
+          -0.012686043046414852,
+          -0.004190925043076277,
+          -0.025674134492874146,
+          -0.01329014077782631,
+          -0.007513460237532854,
+          -0.011862960644066334,
+          0.006833850406110287,
+          -0.002359755104407668,
+          -0.005278300028294325,
+          -0.029026875272393227,
+          0.024209199473261833,
+          -0.0003412205842323601,
+          0.05303974077105522,
+          0.00045637661241926253,
+          0.012210316024720669,
+          -0.012293379753828049,
+          0.0024352672044187784,
+          0.0018037967383861542,
+          0.0040323492139577866,
+          0.025734543800354004,
+          0.019134782254695892,
+          0.011794999241828918,
+          0.03684993460774422,
+          -0.012044189497828484,
+          -0.03189633786678314,
+          -0.01123620942234993,
+          -0.0020482672844082117,
+          0.03307432681322098,
+          0.021037688478827477,
+          -0.003971939906477928,
+          0.015600812621414661,
+          -0.011107838712632656,
+          -0.008812269195914268,
+          0.041018206626176834,
+          -0.008804718032479286,
+          -0.025779852643609047,
+          0.043374184519052505,
+          -0.012867271900177002,
+          0.04657590016722679,
+          0.00700375298038125,
+          -0.00858573243021965,
+          0.004292866215109825,
+          0.010133732110261917,
+          -0.003817139659076929,
+          -0.018289046362042427,
+          0.011394784785807133,
+          0.04479381442070007,
+          0.012399096973240376,
+          0.0061240363866090775,
+          0.02041848748922348,
+          0.02224588207900524,
+          -0.027622347697615623,
+          -0.023589998483657837,
+          -0.0034622326493263245,
+          -0.0025466475635766983,
+          -0.015933066606521606,
+          0.016824109479784966,
+          -0.0400516502559185,
+          -0.009393712505698204,
+          0.026942739263176918,
+          -0.0011732701677829027,
+          0.017292285338044167,
+          0.003854895941913128,
+          0.014543642289936543,
+          0.008245928213000298,
+          -0.01803230494260788,
+          0.026006387546658516,
+          0.030204864218831062,
+          0.013365652412176132,
+          -0.02519085630774498,
+          0.031080804765224457,
+          -0.0065997629426419735,
+          0.019089475274086,
+          0.01677880249917507,
+          0.005014007445424795,
+          -0.006010768003761768,
+          -0.00208035996183753,
+          -0.026127208024263382,
+          -0.006814972497522831,
+          -0.007396416272968054,
+          0.027758270502090454,
+          -0.00439103227108717,
+          0.01564612053334713,
+          0.0159783735871315,
+          -0.012708696536719799,
+          -0.0006475168047472835,
+          -0.0010156385833397508,
+          -0.031775519251823425,
+          0.025613725185394287,
+          0.007891020737588406,
+          0.006999977398663759,
+          0.03802792355418205,
+          -0.042045172303915024,
+          -0.01842496730387211,
+          -0.003711422672495246,
+          0.029147693887352943,
+          0.05134826898574829,
+          0.0031431936658918858,
+          -0.0003223425301257521,
+          0.031775519251823425,
+          -0.03479600325226784,
+          -0.032530639320611954,
+          0.011349477805197239,
+          0.03292330354452133,
+          -0.0005526546156033874,
+          -0.003741627559065819,
+          0.04379705339670181,
+          -0.010254551656544209,
+          -0.021762605756521225,
+          0.021762605756521225,
+          0.01839476265013218,
+          -0.002372969640418887,
+          -0.005648309830576181,
+          0.0003556150768417865,
+          0.014166081324219704,
+          -0.0045005246065557,
+          0.04310234263539314,
+          -0.02419409714639187,
+          0.01495140790939331,
+          -0.01603878289461136,
+          0.013501574285328388,
+          0.015049573965370655,
+          0.00010677891259547323,
+          -0.04283049702644348,
+          0.013040950521826744,
+          0.022351600229740143,
+          0.005474631674587727,
+          0.00958249345421791,
+          0.017322489991784096,
+          0.021838117390871048,
+          0.03832997381687164,
+          0.02960076741874218,
+          0.005923929158598185,
+          0.040655747056007385,
+          0.01453609112650156,
+          -0.0005847472930327058,
+          0.02784888446331024,
+          -0.0019302796572446823,
+          -0.006105158012360334,
+          -0.0011307946406304836,
+          -0.0041418420150876045,
+          0.02343897521495819,
+          -0.021883424371480942,
+          -0.015223251655697823,
+          0.02124912291765213,
+          0.0028468084055930376,
+          -0.00019196607172489166,
+          -0.03343678638339043,
+          0.006290162913501263,
+          -0.02790929563343525,
+          0.0220495518296957,
+          -0.0024560331366956234,
+          0.03364821895956993,
+          -0.023227540776133537,
+          -0.0056747389025986195,
+          0.022260984405875206,
+          -0.019482137635350227,
+          0.0033055448438972235,
+          0.01392444223165512,
+          0.008631040342152119,
+          -0.005965460557490587,
+          -0.011492950841784477,
+          -0.013569535687565804,
+          0.047089383006095886,
+          -0.005350036546587944,
+          0.018183328211307526,
+          0.018138021230697632,
+          -0.007709791883826256,
+          -0.019300907850265503,
+          0.028392571955919266,
+          -0.040716156363487244,
+          0.010662317276000977,
+          0.03201715648174286,
+          -0.06475923210382462,
+          -0.034161701798439026,
+          -0.006006992422044277,
+          -0.028996670618653297,
+          0.005489734001457691,
+          0.00033579315640963614,
+          0.008064698427915573,
+          0.010095976293087006,
+          -0.018802527338266373,
+          0.018455171957612038,
+          0.0006678106728941202,
+          0.02973668836057186,
+          -0.0031356425024569035,
+          0.0069131385535001755,
+          -0.021883424371480942,
+          -0.025930875912308693,
+          -0.01952744461596012,
+          -0.0008792447042651474,
+          -0.0036566765047609806,
+          0.00772489421069622,
+          -0.02741091512143612,
+          -0.015721632167696953,
+          0.01936131902039051,
+          -0.009861888363957405,
+          0.04026308283209801,
+          0.0405651330947876,
+          0.022381804883480072,
+          0.005980563350021839,
+          -0.025009628385305405,
+          0.03923611715435982,
+          -0.042014967650175095,
+          0.022019345313310623,
+          -0.0015904748579487205,
+          -0.017262080684304237,
+          -0.01252746768295765,
+          0.031111011281609535,
+          -0.007845713756978512,
+          0.005504836328327656,
+          0.0011883726110681891,
+          0.021234020590782166,
+          -0.02294059470295906,
+          -0.0030922230798751116,
+          -0.00885002501308918,
+          -0.011900716461241245,
+          0.015359173528850079,
+          0.0003539632598403841,
+          0.01958785392343998,
+          0.02549290470778942,
+          0.006343021523207426,
+          0.0027448670007288456,
+          -0.032107770442962646,
+          -0.031201625242829323,
+          0.026353744789958,
+          0.012308482080698013,
+          -0.028452983126044273,
+          0.013146666809916496,
+          -0.045428115874528885,
+          -0.013652598485350609,
+          -0.011213555932044983,
+          0.0075059086084365845,
+          0.013199525885283947,
+          -0.002378633012995124,
+          -0.0008320496417582035,
+          -0.02777337282896042,
+          -5.3418953029904515e-05,
+          -0.05001925677061081,
+          -0.017594333738088608,
+          0.015276109799742699,
+          -0.01102477591484785,
+          0.01866660639643669,
+          0.005478407256305218,
+          0.0107378289103508,
+          -0.01919519156217575,
+          -0.012187662534415722,
+          0.004666651599109173,
+          -0.030869372189044952,
+          0.01581224612891674,
+          -0.018440069630742073,
+          0.013433613814413548,
+          -0.013033399358391762,
+          0.03981001302599907,
+          -0.009174727834761143,
+          0.016748597845435143,
+          -0.025613725185394287,
+          -0.029887713491916656,
+          -0.02565903216600418,
+          0.02294059470295906,
+          -0.004485422279685736,
+          -0.018787425011396408,
+          0.012029087170958519,
+          -0.007875918410718441,
+          0.011719487607479095,
+          -0.02214016579091549,
+          -0.00916717667132616,
+          0.023287950083613396,
+          -0.026263128966093063,
+          0.0023842963855713606,
+          -0.04902249574661255,
+          -0.03709157183766365,
+          -0.00450807623565197,
+          0.02114340476691723,
+          -0.019300907850265503,
+          0.030688142403960228,
+          -0.0005021558608859777,
+          -0.004379705525934696,
+          -0.02399776503443718,
+          -0.028694620355963707,
+          0.01112294103950262,
+          -0.0002989809727296233,
+          -0.02880033850669861,
+          0.05781210958957672,
+          0.025009628385305405,
+          0.04008185490965843,
+          -0.04482401907444,
+          0.026867227628827095,
+          0.0066261920146644115,
+          -0.029283616691827774,
+          0.01291257981210947,
+          0.01079068798571825,
+          0.016401240602135658,
+          -0.009484327398240566,
+          -0.007468152791261673,
+          0.0034414667170494795,
+          -0.009371059015393257,
+          0.007687137927860022,
+          0.005297177936881781,
+          -0.03222858905792236,
+          0.005784231703728437,
+          0.025704339146614075,
+          -0.001376209082081914,
+          0.04014226421713829,
+          0.007196309044957161,
+          0.011568463407456875,
+          -0.004674202762544155,
+          -0.04011205956339836,
+          0.01730738766491413,
+          -0.027622347697615623,
+          -0.03277227655053139,
+          0.03887366130948067,
+          0.020992381498217583,
+          0.011711936444044113,
+          -0.02416389249265194,
+          0.00416071992367506,
+          0.014641808345913887,
+          -0.016431447118520737,
+          0.0081704156473279,
+          0.006524250842630863,
+          0.027863986790180206,
+          0.014709768816828728,
+          -0.007860816083848476,
+          0.00044740954763256013,
+          0.02303120866417885,
+          -0.01564612053334713,
+          0.0028373694512993097,
+          0.0020407161209732294,
+          0.02813583053648472,
+          0.010330063290894032,
+          -0.024586759507656097,
+          -0.017986996099352837,
+          -0.017549026757478714,
+          -0.020040927454829216,
+          0.0009608922409825027,
+          -0.013713008724153042,
+          -0.02177770808339119,
+          -0.04358562082052231,
+          -0.004798797890543938,
+          -0.019497239962220192,
+          -0.03138285502791405,
+          -0.015026920475065708,
+          -0.008125108666718006,
+          -0.02821134403347969,
+          0.011198453605175018,
+          0.001511187176220119,
+          -0.02038828283548355,
+          0.002797725610435009,
+          -0.004813900217413902,
+          0.050140075385570526,
+          -0.019542546942830086,
+          0.013259935192763805,
+          -0.0038643348962068558,
+          -0.006052299868315458,
+          -0.00875185988843441,
+          0.03168490156531334,
+          -0.026610486209392548,
+          -0.04977761581540108,
+          0.026625588536262512,
+          -0.022683853283524513,
+          0.000796181324403733,
+          0.04497504234313965,
+          -0.024254506453871727,
+          -0.01753392443060875,
+          0.02005602978169918,
+          -0.020297668874263763,
+          -0.032440025359392166,
+          0.003915305715054274,
+          -0.010647214949131012,
+          -0.016068987548351288,
+          -0.02704845555126667,
+          -0.010118629783391953,
+          0.052163802087306976,
+          0.011092736385762691,
+          0.027335401624441147,
+          0.03364821895956993,
+          0.00480634905397892,
+          -0.016552265733480453,
+          0.010881302878260612,
+          -0.015540402382612228,
+          -0.026202719658613205,
+          0.010095976293087006,
+          0.02562882751226425,
+          0.01739800162613392,
+          0.007317128591239452,
+          -0.030310582369565964,
+          -0.005980563350021839,
+          -0.03204736113548279,
+          -0.0051763588562607765,
+          0.019044166430830956,
+          -0.003532081376761198,
+          -0.031956747174263,
+          -0.011115389876067638,
+          0.025840261951088905,
+          0.021928731352090836,
+          -0.018606197088956833,
+          -0.016748597845435143,
+          -0.0028807888738811016,
+          0.0007291642832569778,
+          0.016914723441004753,
+          -0.01327503751963377,
+          -0.00476481718942523,
+          0.004330622497946024,
+          0.019814390689134598,
+          -0.013916891068220139,
+          0.0013129677390679717,
+          -0.05950358137488365,
+          -0.01620491035282612,
+          0.0211132001131773,
+          0.002933647483587265,
+          -0.0017735918518155813,
+          -0.010088425129652023,
+          0.007668260019272566,
+          -0.0031054376158863306,
+          -0.030008532106876373,
+          0.010201692581176758,
+          0.0023238868452608585,
+          0.02214016579091549,
+          -0.02602148987352848,
+          0.008842473849654198,
+          0.01686941646039486,
+          0.0015206261305138469,
+          -0.013546881265938282,
+          -0.0036113690584897995,
+          0.0343429297208786,
+          0.010465985164046288,
+          0.003277227748185396,
+          0.004440115299075842,
+          0.014543642289936543,
+          0.004224905278533697,
+          -0.0034320277627557516,
+          0.002644813386723399,
+          -0.0012959775049239397,
+          -0.006856504362076521,
+          0.004021022468805313,
+          0.006724358070641756,
+          0.02363530732691288,
+          0.024058174341917038,
+          0.005059314891695976,
+          0.0008070362382568419,
+          -0.00976372230798006,
+          -0.0057426998391747475,
+          0.01700533926486969,
+          -0.012557672336697578,
+          0.015827348455786705,
+          0.011168248951435089,
+          -0.02072053775191307,
+          -0.023892046883702278,
+          0.0018859162228181958,
+          0.001278043375350535,
+          0.01830414868891239,
+          0.02217037044465542,
+          -0.00047100710798986256,
+          0.0005309449043124914,
+          -0.017443308606743813,
+          0.00964290276169777,
+          0.007664484437555075,
+          0.03597399219870567,
+          -0.0365176796913147,
+          -0.01480038370937109,
+          0.021823015064001083,
+          -0.009673107415437698,
+          0.020342975854873657,
+          0.0003593906876631081,
+          0.006977323908358812,
+          0.0004969644360244274,
+          0.06935036927461624,
+          -0.017986996099352837,
+          -0.00620709965005517,
+          -0.029872611165046692,
+          0.009605146944522858,
+          -0.0056332070380449295,
+          -0.02052420563995838,
+          0.011198453605175018,
+          0.0328930988907814,
+          -0.015283661894500256,
+          0.042347218841314316,
+          -0.0018575991271063685,
+          0.009303098544478416,
+          -0.005417997483164072,
+          -0.019738879054784775,
+          0.013305243104696274,
+          -0.0014536090893670917,
+          0.004942270927131176,
+          0.0032847789116203785,
+          0.012436852790415287,
+          0.02069033309817314,
+          0.01020924374461174,
+          0.014498335309326649,
+          0.004172047134488821,
+          -0.02008623443543911,
+          0.01978418603539467,
+          -0.0019897455349564552,
+          0.000477614434203133,
+          -0.015993475914001465,
+          -0.013335447758436203,
+          0.0067092557437717915,
+          0.01155336108058691,
+          -0.0033546278718858957,
+          -0.0017207333585247397,
+          0.003413149621337652,
+          0.025070037692785263,
+          -0.016416342929005623,
+          0.014694666489958763,
+          -0.012346237897872925,
+          -0.012308482080698013,
+          -0.012640736065804958,
+          0.005251870956271887,
+          0.005417997483164072,
+          -0.0055388170294463634,
+          0.004927168600261211,
+          -0.006184446159750223,
+          0.011953575536608696,
+          -0.03739362210035324,
+          0.014241593889892101,
+          -0.006633743178099394,
+          -0.003290442517027259,
+          0.010201692581176758,
+          -0.012867271900177002,
+          0.007517235819250345,
+          -0.007403967436403036,
+          0.01653716340661049,
+          0.006977323908358812,
+          -0.007528562564402819,
+          0.0017282846383750439,
+          -0.014853241853415966,
+          0.004519402980804443,
+          0.014792832545936108,
+          -0.0018085162155330181,
+          0.01495140790939331,
+          -0.01972377672791481,
+          -0.01474752463400364,
+          0.0006059851148165762,
+          -0.01853068359196186,
+          0.002325774636119604,
+          0.00454205647110939,
+          -0.009710864163935184,
+          0.023952458053827286,
+          -0.0034565692767500877,
+          -0.01026210281997919,
+          -0.021928731352090836,
+          -0.015495095402002335,
+          -0.015661222860217094,
+          -0.020629921928048134,
+          0.006093831267207861,
+          -0.03307432681322098,
+          0.022623443976044655,
+          0.016235115006566048,
+          -0.01733759231865406,
+          -0.01972377672791481,
+          -0.032440025359392166,
+          0.0030676815658807755,
+          -0.006093831267207861,
+          -0.005455753766000271,
+          0.012097048573195934,
+          -0.004153168760240078,
+          0.011893165297806263,
+          0.0031299791298806667,
+          -0.010956814512610435,
+          0.011485399678349495,
+          -0.012285828590393066,
+          -0.006494045723229647,
+          -0.0018009650520980358,
+          -0.003647237317636609,
+          -0.015042022801935673,
+          -0.023741023615002632,
+          -0.03769567236304283,
+          0.020071132108569145,
+          0.013426062650978565,
+          -0.014422822743654251,
+          -0.01529876422137022,
+          -0.002776959678158164,
+          0.01223297044634819,
+          -0.013108910992741585,
+          -0.01264828722923994,
+          0.008940639905631542,
+          -0.008464912883937359,
+          0.01461915485560894,
+          -0.0014413383323699236,
+          -0.0034018228761851788,
+          0.023710818961262703,
+          0.00937861017882824,
+          0.005440651439130306,
+          -0.02479819394648075,
+          -0.012240521609783173,
+          0.01370545756071806,
+          -0.013931993395090103,
+          -0.0017226211493834853,
+          0.0219438336789608,
+          -0.00154139194637537,
+          0.000530472956597805,
+          0.011145595461130142,
+          0.0019265039591118693,
+          -0.02058461494743824,
+          0.0054028951562941074,
+          0.008283684030175209,
+          -0.009854337200522423,
+          0.046092621982097626,
+          0.014098120853304863,
+          0.01603878289461136,
+          0.0025655257049947977,
+          -0.006184446159750223,
+          0.013342998921871185,
+          0.010669868439435959,
+          0.02787908911705017,
+          -0.015094880945980549,
+          -0.007366211153566837,
+          0.013388305902481079,
+          0.04080677032470703,
+          0.00013851761468686163,
+          0.011409887112677097,
+          -0.027592143043875694,
+          -0.02164178527891636,
+          0.028815440833568573,
+          0.004787471145391464,
+          0.01302584819495678,
+          -0.011138043366372585,
+          -0.024012867361307144,
+          0.015079778619110584,
+          -0.018545785918831825,
+          -0.015011817216873169,
+          -0.028558699414134026,
+          0.009627800434827805,
+          -0.027471324428915977,
+          0.02840767428278923,
+          0.0069810994900763035,
+          -0.0006479887524619699,
+          0.02519085630774498,
+          0.000645628955680877,
+          -0.025311676785349846,
+          0.014860793016850948,
+          -0.013607291504740715,
+          0.006562006659805775,
+          0.016688188537955284,
+          -0.03092978149652481,
+          0.0014130213530734181,
+          -0.006569557823240757,
+          -0.004107861779630184,
+          -0.0016848650993779302,
+          0.02436022274196148,
+          -0.016612675040960312,
+          -0.02599128521978855,
+          -0.005682290066033602,
+          -0.006331694778054953,
+          0.002212506253272295,
+          -0.007747547701001167,
+          0.022215677425265312,
+          0.005950358230620623,
+          0.003154520643875003,
+          -0.01789638213813305,
+          -0.009287996217608452,
+          0.020297668874263763,
+          -0.020750742405653,
+          0.0014715432189404964,
+          -0.021958936005830765,
+          0.010133732110261917,
+          0.006629967596381903,
+          0.010186590254306793,
+          0.014762627892196178,
+          6.300545646809042e-05,
+          0.003545295912772417,
+          0.0007820228347554803,
+          0.0026183840818703175,
+          0.013063604012131691,
+          0.01524590514600277,
+          -0.0446125864982605,
+          -0.02257813699543476,
+          0.028679518029093742,
+          -0.016461651772260666,
+          0.00308278389275074,
+          0.008563078939914703,
+          -0.03343678638339043,
+          -0.011779896914958954,
+          -0.005614329129457474,
+          0.035520922392606735,
+          -0.01614449918270111,
+          0.0015149627579376101,
+          -0.001464935950934887,
+          -0.024224301800131798,
+          0.0026882330421358347,
+          -0.020131541416049004,
+          0.008842473849654198,
+          -0.01830414868891239,
+          -0.043434593826532364,
+          -0.0008244984201155603,
+          0.02535698376595974,
+          -0.007449274882674217,
+          0.003824691055342555,
+          0.021354839205741882,
+          -0.018138021230697632,
+          0.003739739768207073,
+          -0.019089475274086,
+          0.013780969195067883,
+          0.003877549432218075,
+          0.028196241706609726,
+          0.008736756630241871,
+          0.004685529507696629,
+          0.006380777806043625,
+          -0.00368688115850091,
+          0.030476707965135574,
+          -0.009227585978806019,
+          -0.012036638334393501,
+          0.021022586151957512,
+          0.021520966663956642,
+          0.02127932757139206,
+          0.002335213590413332,
+          0.010835994966328144,
+          0.035520922392606735,
+          -0.005210339091718197,
+          0.019874799996614456,
+          0.007815509103238583,
+          -6.365439185174182e-05,
+          0.008940639905631542,
+          -0.007698464673012495,
+          -0.03443354368209839,
+          -0.0022653648629784584,
+          -0.022865083068609238,
+          -0.01697513461112976,
+          0.011772345751523972,
+          -0.023454077541828156,
+          -0.02217037044465542,
+          -0.028679518029093742,
+          0.013660149648785591,
+          0.004783695563673973,
+          0.004749714862555265,
+          -0.021234020590782166,
+          0.02777337282896042,
+          0.024647168815135956,
+          0.015585710294544697,
+          0.017715152353048325,
+          0.012754004448652267,
+          0.030386094003915787,
+          0.008336542174220085,
+          0.02641415409743786,
+          0.0065997629426419735,
+          0.012406648136675358,
+          -0.007747547701001167,
+          0.031956747174263,
+          0.017594333738088608,
+          0.006267509423196316,
+          0.007509684655815363,
+          -0.01733759231865406,
+          -0.0032338083256036043,
+          0.0007013191934674978,
+          -0.020478898659348488,
+          -0.011039878241717815,
+          -0.006788543425500393,
+          0.032802481204271317,
+          0.02200424298644066,
+          -0.016325728967785835,
+          -0.014211388304829597,
+          -0.005191461183130741,
+          0.0060560754500329494,
+          -0.023046310991048813,
+          -0.007543664891272783,
+          -0.012285828590393066,
+          -0.0023238868452608585,
+          -0.009340854361653328,
+          0.02343897521495819,
+          0.021626682952046394,
+          0.010760483331978321,
+          0.01833435334265232,
+          0.001749050454236567,
+          0.003915305715054274,
+          -0.010760483331978321,
+          -0.007317128591239452,
+          0.011379682458937168,
+          -0.012421750463545322,
+          -0.025341881439089775,
+          0.008087352849543095,
+          -0.006256182678043842,
+          -0.020267464220523834,
+          0.009567391127347946,
+          0.018379660323262215,
+          0.001985969953238964,
+          0.01252746768295765,
+          0.017216771841049194,
+          0.0014998603146523237,
+          0.0065997629426419735,
+          0.02214016579091549,
+          -0.0028298182878643274,
+          -0.006543128751218319,
+          0.004236232023686171,
+          -0.007981635630130768,
+          0.02230629324913025,
+          0.016461651772260666,
+          -0.03207756578922272,
+          -0.01529876422137022,
+          -0.010798239149153233,
+          0.02960076741874218,
+          -0.0070754894986748695,
+          0.008653693832457066,
+          -0.03265145793557167,
+          -0.006267509423196316,
+          -0.026308437809348106,
+          0.02002582512795925,
+          -0.003362179035320878,
+          -0.006331694778054953,
+          0.004734612535685301,
+          -0.01578204147517681,
+          0.017413103953003883,
+          -0.003756730118766427,
+          -0.009287996217608452,
+          0.012429301626980305,
+          -0.004991353955119848,
+          0.01617470569908619,
+          -0.007441723719239235,
+          0.004323071334511042,
+          0.015525300055742264,
+          0.00722651369869709,
+          0.004723285790532827,
+          -0.03201715648174286,
+          -0.0019878577440977097,
+          -0.014528539963066578,
+          0.0062863873317837715,
+          -0.006701704114675522,
+          0.0009490934899076819,
+          0.024178994819521904,
+          0.027728065848350525,
+          -0.010133732110261917,
+          -0.008011840283870697,
+          0.029419537633657455,
+          0.004070105496793985,
+          0.012029087170958519,
+          -0.009333303198218346,
+          -0.013584638014435768,
+          -0.0012251847656443715,
+          -0.012504814192652702,
+          -0.028890952467918396,
+          -0.014437925070524216,
+          -0.027108866721391678,
+          0.016718393191695213,
+          -0.009310649707913399,
+          -0.02019195258617401,
+          0.0328930988907814,
+          -0.041712917387485504,
+          -0.008412054739892483,
+          -0.00329421809874475,
+          0.0015347846783697605,
+          0.012587876990437508,
+          0.003345188684761524,
+          0.0028260427061468363,
+          -0.028951361775398254,
+          -0.013984852470457554,
+          -0.007026406470686197,
+          0.004081432241946459,
+          -0.000620143604464829,
+          -0.02496432140469551,
+          0.007256718818098307,
+          -0.0038058129139244556,
+          -0.004273988306522369,
+          0.010314960964024067,
+          -0.00484788091853261,
+          -0.024481043219566345,
+          0.02602148987352848,
+          -0.007120796944946051,
+          0.02052420563995838,
+          0.010171487927436829,
+          0.016914723441004753,
+          0.010730277746915817,
+          0.03023506887257099,
+          -0.0007919337949715555,
+          -0.002229496603831649,
+          -0.004357051569968462,
+          -0.009703313000500202,
+          0.01165152620524168,
+          0.014279349707067013,
+          -0.01128906849771738,
+          -0.03706136718392372,
+          0.006018319167196751,
+          -0.02343897521495819,
+          0.019738879054784775,
+          0.0010118630016222596,
+          -0.02999342978000641,
+          -0.007037733215838671,
+          -0.0422264002263546,
+          0.02386184222996235,
+          -0.005300953518599272,
+          -0.016386138275265694,
+          0.017050646245479584,
+          -0.006577109452337027,
+          -0.005557694938033819,
+          0.027199480682611465,
+          0.005131051409989595,
+          -0.030084045603871346,
+          0.013743213377892971,
+          0.010156385600566864,
+          -0.012104599736630917,
+          0.0018141797045245767,
+          -0.0006838570116087794,
+          -0.034252315759658813,
+          0.0071170213632285595,
+          0.0008518715621903539,
+          -0.0034660082310438156,
+          -0.02058461494743824,
+          -0.019633162766695023,
+          -0.00881982035934925,
+          0.018379660323262215,
+          0.018349455669522285,
+          0.01836455799639225,
+          -0.007762650027871132,
+          0.00036127850762568414,
+          0.006799870170652866,
+          0.005331158638000488,
+          0.010911507532000542,
+          0.005501060746610165,
+          -0.00045118515845388174,
+          0.007928776554763317,
+          -0.014150978997349739,
+          0.017413103953003883,
+          -0.020901765674352646,
+          -0.003990817815065384,
+          -0.00019208405865356326,
+          -0.008555527776479721,
+          0.00019845539645757526,
+          -0.016809007152915,
+          0.011681731790304184,
+          0.016461651772260666,
+          -0.006822523660957813,
+          -0.009491878561675549,
+          -0.020705435425043106,
+          0.009997810237109661,
+          0.029525255784392357,
+          0.01020924374461174,
+          -0.041320253163576126,
+          0.021898526698350906,
+          -0.01739800162613392,
+          0.0018160674953833222,
+          -0.017941689118742943,
+          0.025961080566048622,
+          0.014332207851111889,
+          0.0035736130084842443,
+          0.018983757123351097,
+          -0.0049989051185548306,
+          0.0003516034921631217,
+          -0.00704528484493494,
+          0.001961428439244628,
+          0.00045614063856191933,
+          -0.009522083215415478,
+          -0.0040738810785114765,
+          0.02704845555126667,
+          -0.02416389249265194,
+          -0.004334398079663515,
+          0.001532896887511015,
+          0.023121824488043785,
+          -0.008261030539870262,
+          0.012535018846392632,
+          0.017911484465003014,
+          0.018017200753092766,
+          -0.002399398945271969,
+          0.011447643861174583,
+          0.07291454076766968,
+          -0.019089475274086,
+          0.017624538391828537,
+          0.019331112504005432,
+          -0.008245928213000298,
+          -0.01637103594839573,
+          -0.010851097293198109,
+          -0.01988990418612957,
+          0.02144545316696167,
+          -0.0025711890775710344,
+          0.01445302739739418,
+          -0.011296619661152363,
+          -0.009257790632545948,
+          0.007539889309555292,
+          0.0037869347725063562,
+          0.0015168505487963557,
+          0.005285851191729307,
+          0.011742141097784042,
+          0.023876944556832314,
+          0.010118629783391953,
+          -0.005165031645447016,
+          0.0012619970366358757,
+          -0.0011996994726359844,
+          -0.0023805208038538694,
+          -0.01184785831719637,
+          -0.0013554432662203908,
+          0.009386161342263222,
+          0.02340877056121826,
+          0.0019878577440977097,
+          0.035218872129917145,
+          -0.015011817216873169,
+          -0.009121868759393692,
+          0.00916717667132616,
+          0.0006692265160381794,
+          0.004606241825968027,
+          0.014724871143698692,
+          0.012837067246437073,
+          -0.011772345751523972,
+          -0.006176894996315241,
+          0.0076154014095664024,
+          0.0052934023551642895,
+          0.006384553387761116,
+          0.0107378289103508,
+          -0.01112294103950262,
+          -0.016960032284259796,
+          0.03709157183766365,
+          0.009884541854262352,
+          0.0023238868452608585,
+          0.0049309441819787025,
+          0.0009769386379048228,
+          -0.011062531732022762,
+          0.004832778126001358,
+          0.004443890880793333,
+          0.005327383056282997,
+          -0.000515842461027205,
+          -0.0009325752034783363,
+          0.003205491229891777,
+          -0.0056332070380449295,
+          0.0014847578713670373,
+          0.008049596101045609,
+          0.005319831892848015,
+          0.010911507532000542,
+          -0.008396952413022518,
+          -0.019210293889045715,
+          0.0010005361400544643,
+          -0.016657982021570206,
+          0.006679050624370575,
+          0.002169086830690503,
+          -0.0063279191963374615,
+          0.007536113727837801,
+          0.009189830161631107,
+          -0.0073473332449793816,
+          0.014332207851111889,
+          -0.001468711532652378,
+          0.001872701570391655,
+          0.0107378289103508,
+          -0.020327873528003693,
+          -0.007672035600990057,
+          0.0024767988361418247,
+          0.006894260179251432,
+          0.003726525232195854,
+          0.0018321138340979815,
+          -0.008615938015282154,
+          0.014936305582523346,
+          -0.023544691503047943,
+          -0.00908411294221878,
+          -0.0032375839073210955,
+          0.026595383882522583,
+          0.013071155175566673,
+          -0.013003193773329258,
+          0.004406134597957134,
+          0.00012789871834684163,
+          0.0066941529512405396,
+          0.0038058129139244556,
+          -0.004383481107652187,
+          0.02735050395131111,
+          -0.003715198254212737,
+          -0.024420632049441338,
+          -0.008238377049565315,
+          -0.002325774636119604,
+          0.009257790632545948,
+          0.023982662707567215,
+          0.0054028951562941074,
+          0.006497821304947138,
+          0.006988650653511286,
+          0.00502910977229476,
+          0.009892093017697334,
+          0.013629944995045662,
+          -0.003990817815065384,
+          -0.008646142669022083,
+          -0.004002144560217857,
+          -0.011402335949242115,
+          0.015283661894500256,
+          0.016386138275265694,
+          -0.012701145373284817,
+          -0.017881279811263084,
+          -0.0073020257987082005,
+          -0.01456629578024149,
+          -0.0075059086084365845,
+          0.003930408041924238,
+          -0.006048524286597967,
+          0.0011279629543423653,
+          -0.0009250239818356931,
+          -0.009597595781087875,
+          0.01324483286589384,
+          -0.012618082575500011,
+          -0.008087352849543095,
+          -0.008547976613044739,
+          -0.034947026520967484,
+          -0.024375325068831444,
+          0.005727597512304783,
+          -0.014966510236263275,
+          -0.001957652857527137,
+          0.014958959072828293,
+          -0.007358659990131855,
+          0.003715198254212737,
+          0.017624538391828537,
+          0.004987578373402357,
+          0.0060560754500329494,
+          -0.005006456281989813,
+          -0.0038114762865006924,
+          -0.010186590254306793,
+          -0.011938472278416157,
+          -0.01464935950934887,
+          0.0040059201419353485,
+          0.0012421749997884035,
+          0.011228658258914948,
+          -0.01544978842139244,
+          -0.0020444917026907206,
+          -0.003060130402445793,
+          0.019482137635350227,
+          -0.028845645487308502,
+          -0.019497239962220192,
+          0.008706551976501942,
+          -0.0004950765869580209,
+          0.01047353632748127,
+          -0.021369941532611847,
+          -0.0029732913244515657,
+          0.0017046870198100805,
+          -0.008691449649631977,
+          -0.019799288362264633,
+          0.02291039004921913,
+          -0.015125085599720478,
+          0.004534505307674408,
+          -0.008396952413022518,
+          0.027592143043875694,
+          0.012693594209849834,
+          0.013894237577915192,
+          0.0064902701415121555,
+          0.00590127520263195,
+          -0.01647675409913063,
+          -0.0026693549007177353,
+          0.0017905820859596133,
+          -0.006528026424348354,
+          0.0015810359036549926,
+          0.0075059086084365845,
+          -0.002389959990978241,
+          -0.007441723719239235,
+          0.018122918903827667,
+          0.01524590514600277,
+          0.005444427020847797,
+          0.020297668874263763,
+          0.0011619434226304293,
+          0.002886452479287982,
+          -0.00011043653648812324,
+          0.0027656329330056906,
+          0.003494325326755643,
+          0.010919058695435524,
+          0.004489197861403227,
+          -0.019874799996614456,
+          0.0013894237345084548,
+          -0.002399398945271969,
+          -0.007604074664413929,
+          -0.026988046243786812,
+          0.00120158726349473,
+          0.007709791883826256,
+          -0.001681089517660439,
+          -0.01471731998026371,
+          -0.015004266053438187,
+          0.0054972851648926735,
+          -0.001405470073223114,
+          0.013848930597305298,
+          -0.02105279080569744,
+          0.001889691804535687,
+          0.0013582749525085092,
+          -0.01925560086965561,
+          -0.005670963320881128,
+          0.02191362902522087,
+          0.019572751596570015,
+          -0.006550679914653301,
+          -0.03953816741704941,
+          -0.005980563350021839,
+          0.01301074493676424,
+          -0.0058370898477733135,
+          -0.01753392443060875,
+          0.006509148515760899,
+          -0.009053908288478851,
+          -0.006101382430642843,
+          0.018243737518787384,
+          0.0010448995744809508,
+          -0.01544978842139244,
+          0.006188221741467714,
+          0.008283684030175209,
+          0.02399776503443718,
+          0.004685529507696629,
+          -0.009439020417630672,
+          -0.005391568411141634,
+          -0.022200575098395348,
+          0.008427157066762447,
+          0.00046392783406190574,
+          -0.026625588536262512,
+          -0.002437154995277524,
+          -0.004662876017391682,
+          0.03165469691157341,
+          0.01040557585656643,
+          0.021505864337086678,
+          0.011092736385762691,
+          -0.018742118030786514,
+          -0.008593283593654633,
+          -0.008480016142129898,
+          -0.012980540283024311,
+          -0.0034225888084620237,
+          -0.0016697626560926437,
+          0.008615938015282154,
+          0.01581224612891674,
+          0.028030114248394966,
+          0.00020706850045826286,
+          0.011742141097784042,
+          0.01468711532652378,
+          -2.842029971361626e-05,
+          0.011636423878371716,
+          0.001077936147339642,
+          -0.016235115006566048,
+          0.00047147905570454895,
+          0.00476481718942523,
+          -0.008985946886241436,
+          -0.006584660615772009,
+          0.006203324068337679,
+          0.018183328211307526,
+          -0.006116485223174095,
+          -0.005081968382000923,
+          -0.010012912563979626,
+          -0.00026452852762304246,
+          -0.012225419282913208,
+          0.04503545165061951,
+          0.009673107415437698,
+          -0.033497195690870285,
+          -0.001490421243943274,
+          -0.021536068990826607,
+          0.01856089010834694,
+          -0.0037982617504894733,
+          -0.004036124795675278,
+          0.007018855307251215,
+          0.02701825089752674,
+          0.025568418204784393,
+          -0.009046357125043869,
+          -0.011168248951435089,
+          0.0007848545210435987,
+          0.011062531732022762,
+          -0.0004979083314538002,
+          -0.008706551976501942,
+          -0.0012431190116330981,
+          0.0028732377104461193,
+          -0.018711913377046585,
+          0.0030808961018919945,
+          0.008155313320457935,
+          0.00044363393681123853,
+          0.0074379476718604565,
+          -0.014551193453371525,
+          0.026172515004873276,
+          -0.0049309441819787025,
+          -0.01385648176074028,
+          -0.03772587701678276,
+          -0.01995031349360943,
+          0.020237259566783905,
+          0.0068716066889464855,
+          0.002169086830690503,
+          -0.015389378182590008,
+          -0.02011643908917904,
+          0.020433591678738594,
+          -0.011183351278305054,
+          -0.0056520854122936726,
+          -0.010934161022305489,
+          -0.010851097293198109,
+          0.0214756578207016,
+          0.007430396508425474,
+          -0.007875918410718441,
+          -0.012821964919567108,
+          0.012383994646370411,
+          -0.0005762521759606898,
+          0.006826299242675304,
+          0.02369571663439274,
+          0.026761509478092194,
+          0.01819843053817749,
+          -0.009899644181132317,
+          -0.010579253546893597,
+          -0.020509103313088417,
+          -0.003562286263331771,
+          -0.022593239322304726,
+          0.007430396508425474,
+          0.0016357821878045797,
+          0.01427179854363203,
+          0.00722651369869709,
+          -0.0016631553880870342,
+          -0.005603002384305,
+          0.009506980888545513,
+          0.0070490604266524315,
+          -0.012980540283024311,
+          0.018047407269477844,
+          -0.01108518522232771,
+          0.018772322684526443,
+          -0.0030582426115870476,
+          -0.013494023121893406,
+          0.019316010177135468,
+          -0.013750764541327953,
+          0.004409910179674625,
+          -0.004039900843054056,
+          0.008412054739892483,
+          0.00442501250654459,
+          -0.010133732110261917,
+          0.016461651772260666,
+          -0.016612675040960312,
+          0.0002716078015509993,
+          -0.010919058695435524,
+          -0.00476481718942523,
+          -0.014558744616806507,
+          -0.022094858810305595,
+          -0.044672995805740356,
+          -0.005297177936881781,
+          0.019965415820479393,
+          0.0008900996181182563,
+          0.01044333167374134,
+          -0.013826277107000351,
+          -0.0006555399741046131,
+          -0.005950358230620623,
+          -0.008525323122739792,
+          0.012383994646370411,
+          0.01842496730387211,
+          -0.009831683710217476,
+          -0.003926632460206747,
+          0.021264225244522095,
+          0.0021917405538260937,
+          -0.019965415820479393,
+          -0.004651549272239208,
+          -0.006562006659805775,
+          0.0031979400664567947,
+          0.014815486036241055,
+          -0.005376466084271669,
+          -0.023574896156787872,
+          0.0013894237345084548,
+          0.006879157852381468,
+          -0.02724478766322136,
+          -0.0013903676299378276,
+          -0.01720166951417923,
+          -0.01667308434844017,
+          0.006407206878066063,
+          -0.028679518029093742,
+          0.009809029288589954,
+          -0.022955697029829025,
+          -0.010095976293087006,
+          -0.005051763728260994,
+          -0.020796049386262894,
+          0.004610017407685518,
+          -0.008744308724999428,
+          0.031805723905563354,
+          -0.001901962561532855,
+          -0.008691449649631977,
+          0.01158356573432684,
+          0.01753392443060875,
+          0.02200424298644066,
+          0.0051574804820120335,
+          0.025583520531654358,
+          -0.004179598297923803,
+          -0.014974061399698257,
+          -0.0027561939787119627,
+          0.01611429452896118,
+          0.002474911045283079,
+          0.024919012561440468,
+          -0.0158726554363966,
+          0.00037850471562705934,
+          0.00934840552508831,
+          0.01670329086482525,
+          0.0053689144551754,
+          -0.0031733985524624586,
+          0.018409864977002144,
+          0.0007839106256142259,
+          -0.003658564295619726,
+          -0.02771296352148056,
+          -0.0016933601582422853,
+          0.0002349136193515733,
+          0.003620808245614171,
+          0.01032251212745905,
+          -0.0015857553808018565,
+          0.006399655714631081,
+          0.016688188537955284,
+          0.0010722726583480835,
+          -0.007690913509577513,
+          -0.01620491035282612,
+          0.004934719763696194,
+          -0.01955764926970005,
+          0.014702217653393745,
+          -0.004738388117402792,
+          0.015691427513957024,
+          0.002267252653837204,
+          -0.038632020354270935,
+          0.032107770442962646,
+          0.004394807852804661,
+          -0.008842473849654198,
+          0.019814390689134598,
+          -0.02990281581878662,
+          -0.018213532865047455,
+          0.0011175799882039428,
+          0.0022559259086847305,
+          0.018319251015782356,
+          -0.011772345751523972,
+          0.0018283382523804903,
+          0.003532081376761198,
+          -0.0037472909316420555,
+          -0.0039379592053592205,
+          0.009053908288478851,
+          -0.025115344673395157,
+          0.00822327472269535,
+          0.018983757123351097,
+          -0.001961428439244628,
+          0.003881325013935566,
+          0.010858648456633091,
+          -0.0067356848157942295,
+          0.0009245520341210067,
+          0.007690913509577513,
+          0.024647168815135956,
+          -0.005316056311130524,
+          0.017262080684304237,
+          -0.03159428760409355,
+          0.004145617596805096,
+          -0.016235115006566048,
+          -0.017277183011174202,
+          -0.015155291184782982,
+          0.001227072672918439,
+          0.01650695875287056,
+          0.008042044937610626,
+          -0.014369964599609375,
+          -0.00018417887622490525,
+          0.0029147693421691656,
+          0.0049724760465323925,
+          -0.0012723798863589764,
+          0.021203814074397087,
+          1.1548051588761155e-05,
+          0.024420632049441338,
+          -0.00772489421069622,
+          -0.027531733736395836,
+          -0.012089497409760952,
+          0.017881279811263084,
+          -0.00651292409747839,
+          0.010533946566283703,
+          0.01495140790939331,
+          -0.00295441341586411,
+          -0.011070082895457745,
+          -0.002305008703842759,
+          0.018681708723306656,
+          0.00208035996183753,
+          0.0003353211795911193,
+          0.009673107415437698,
+          0.009476776234805584,
+          -0.0017669845838099718,
+          0.022230779752135277,
+          -0.00790612306445837,
+          -0.018107816576957703,
+          0.001507411478087306,
+          0.009907195344567299,
+          0.01988990418612957,
+          -0.0159783735871315,
+          -0.013984852470457554,
+          0.003769944654777646,
+          0.01241419930011034,
+          -0.0033433008939027786,
+          -0.01773025467991829,
+          0.0017169577768072486,
+          0.028785236179828644,
+          0.00374540314078331,
+          -0.0084045035764575,
+          -0.0017112944042310119,
+          -0.017186567187309265,
+          -0.0002583931782282889,
+          -0.011175800114870071,
+          0.010511293075978756,
+          -0.004813900217413902,
+          0.0036226960364729166,
+          -0.017095953226089478,
+          0.014702217653393745,
+          0.013728111051023006,
+          0.021717296913266182,
+          0.02343897521495819,
+          -0.022155268117785454,
+          -0.015963271260261536,
+          0.005372690502554178,
+          0.0052443197928369045,
+          0.004515627399086952,
+          -0.005323607474565506,
+          0.0013752651866525412,
+          0.021264225244522095,
+          0.008555527776479721,
+          -0.0038152518682181835,
+          0.0107378289103508,
+          -0.011115389876067638,
+          -0.014513437636196613,
+          -0.008978395722806454,
+          -0.003983266651630402,
+          -0.018575992435216904,
+          -0.0028241549152880907,
+          0.012142355553805828,
+          0.011077634058892727,
+          -0.01480038370937109,
+          0.009295547381043434,
+          0.015917964279651642,
+          -0.004217354115098715,
+          -0.007706015836447477,
+          -0.010163936764001846,
+          -0.0008150594076141715,
+          0.030023634433746338,
+          0.00019161211093887687,
+          -0.006826299242675304,
+          0.014324656687676907,
+          -0.011930921114981174,
+          0.008208171464502811,
+          -0.02019195258617401,
+          -0.01150050200521946,
+          0.00011356321192579344,
+          -0.01032251212745905,
+          -0.016884518787264824,
+          -0.016748597845435143,
+          0.01916498690843582,
+          0.0055161635391414165,
+          -0.008449810557067394,
+          -0.035581331700086594,
+          0.025432495400309563,
+          -0.0018585430225357413,
+          0.040988001972436905,
+          0.006154241040349007,
+          0.007377538364380598,
+          -0.0005111229256726801,
+          0.015185495838522911,
+          0.011334375478327274,
+          -0.020841356366872787,
+          0.015140187926590443,
+          0.007173655088990927,
+          0.0011477848747745156,
+          0.007222738116979599,
+          -0.0025617501232773066,
+          0.0068036457523703575,
+          -0.0103753712028265,
+          -0.0005738924373872578,
+          -0.014415271580219269,
+          -0.027093762531876564,
+          -0.015412031672894955,
+          0.013222179375588894,
+          -0.00041154128848575056,
+          -0.0036283594090491533,
+          -0.018319251015782356,
+          -0.01789638213813305,
+          -0.01919519156217575,
+          -0.0075927479192614555,
+          -0.024209199473261833,
+          -0.008132659830152988,
+          -0.00958249345421791,
+          0.0043004173785448074,
+          0.015117534436285496,
+          0.010065770708024502,
+          -0.003264013212174177,
+          0.027697861194610596,
+          0.019316010177135468,
+          0.013599740341305733,
+          0.00011427114077378064,
+          -0.01158356573432684,
+          0.013063604012131691,
+          0.02665579319000244,
+          -0.0025277696549892426,
+          0.003892651991918683,
+          0.00730957742780447,
+          -0.006309041287750006,
+          0.014083018526434898,
+          -0.008336542174220085,
+          0.0031318669207394123,
+          0.009091664105653763,
+          -0.016099192202091217,
+          0.0015602700877934694,
+          0.02038828283548355,
+          0.013471369631588459,
+          0.00420225178822875,
+          0.0009920409647747874,
+          0.008985946886241436,
+          0.008110006339848042,
+          -0.006271285004913807,
+          -0.012240521609783173,
+          -0.005006456281989813,
+          0.006441187579184771,
+          0.0109719168394804,
+          0.012165009044110775,
+          -0.020765844732522964,
+          0.017322489991784096,
+          0.016265319660305977,
+          -0.013577086851000786,
+          0.012512365356087685,
+          -0.0038662226870656013,
+          0.003658564295619726,
+          0.015125085599720478,
+          0.008985946886241436,
+          -0.0002326718531548977,
+          -0.013939544558525085,
+          0.012097048573195934,
+          -0.02522106096148491,
+          0.004364602733403444,
+          -0.008729205466806889,
+          -0.0310203954577446,
+          0.012572774663567543,
+          -0.02396756038069725,
+          -0.0006475168047472835,
+          0.005704943556338549,
+          -0.013750764541327953,
+          0.001948213786818087,
+          -0.010752932168543339,
+          0.005131051409989595,
+          0.006071177776902914,
+          -0.007777752820402384,
+          -0.016718393191695213,
+          -0.00514615373685956,
+          1.4519867363560479e-05,
+          -0.000538968073669821,
+          -0.023423872888088226,
+          0.02291039004921913,
+          -0.006173118948936462,
+          -0.017986996099352837,
+          0.0004884693189524114,
+          -0.02291039004921913,
+          -0.018606197088956833,
+          -0.006078728940337896,
+          -0.005923929158598185,
+          -0.021853219717741013,
+          -0.00916717667132616,
+          0.0206450242549181,
+          -0.005478407256305218,
+          -0.012535018846392632,
+          -0.024103481322526932,
+          -0.0015527188079431653,
+          0.000700375298038125,
+          -0.008638591505587101,
+          -0.015268558636307716,
+          0.015155291184782982,
+          0.016960032284259796,
+          -0.019572751596570015,
+          -0.010662317276000977,
+          0.006414758041501045,
+          -0.0009122812771238387,
+          0.01859109476208687,
+          0.0063958801329135895,
+          -0.012361341156065464,
+          -0.011560912244021893,
+          0.014468129724264145,
+          0.00952963437885046,
+          0.021626682952046394,
+          0.006214650813490152,
+          0.01090395636856556,
+          0.0057011679746210575,
+          -0.0025787402410060167,
+          -0.009250239469110966,
+          -9.225167559634428e-06,
+          0.010987019166350365,
+          -0.005384017247706652,
+          -0.009605146944522858,
+          0.0023201112635433674,
+          -0.0031941644847393036,
+          0.00692824088037014,
+          -0.010156385600566864,
+          -0.0064223092049360275,
+          -0.027169276028871536,
+          -0.019270703196525574,
+          -0.019708674401044846,
+          -0.009091664105653763,
+          0.00814021099358797,
+          -0.008653693832457066,
+          -0.014966510236263275,
+          0.0041154129430651665,
+          0.014279349707067013,
+          -0.03543030470609665,
+          -0.009522083215415478,
+          -0.004753490444272757,
+          -0.0029355352744460106,
+          -0.036215633153915405,
+          0.019844595342874527,
+          0.025115344673395157,
+          -0.0005441594985313714,
+          -0.000711230153683573,
+          -0.0030582426115870476,
+          -0.012655838392674923,
+          -0.008978395722806454,
+          0.002072808798402548,
+          0.0020067356526851654,
+          -0.0270786602050066,
+          0.005285851191729307,
+          0.01388668641448021,
+          0.021490762010216713,
+          0.008593283593654633,
+          -0.015495095402002335,
+          0.0036510128993541002,
+          -0.024677373468875885,
+          -0.025402290746569633,
+          -0.0220495518296957,
+          -0.004557158797979355,
+          -0.006018319167196751,
+          0.004606241825968027,
+          0.0056332070380449295,
+          0.01842496730387211,
+          0.00719253346323967,
+          0.005493509583175182,
+          -0.01972377672791481,
+          0.010911507532000542,
+          0.0009377666283398867,
+          0.0005111229256726801,
+          -0.012897477485239506,
+          -0.01412077434360981,
+          0.02343897521495819,
+          -0.00041319310548715293,
+          0.014551193453371525,
+          0.006029645912349224,
+          -0.009650453925132751,
+          -0.007966533303260803,
+          -0.0012940897140651941,
+          -0.026066798716783524,
+          0.016325728967785835,
+          -0.011168248951435089,
+          0.0006970716640353203,
+          -0.015555505640804768,
+          -0.00976372230798006,
+          0.022155268117785454,
+          0.012640736065804958,
+          -0.0025881794281303883,
+          0.015933066606521606,
+          0.018349455669522285,
+          0.0008315776940435171,
+          0.005474631674587727,
+          -0.017971893772482872,
+          -0.007230289280414581,
+          0.008396952413022518,
+          -0.0035188666079193354,
+          0.008615938015282154,
+          -0.023650409653782845,
+          -0.005969236139208078,
+          0.016612675040960312,
+          0.0214756578207016,
+          -0.021430350840091705,
+          0.001019414165057242,
+          0.0068036457523703575,
+          0.009325752034783363,
+          -0.006879157852381468,
+          -0.009733517654240131,
+          -0.015902861952781677,
+          -0.00893308874219656,
+          -0.005923929158598185,
+          0.021173609420657158,
+          -0.01853068359196186,
+          -0.004341949243098497,
+          -0.0025617501232773066,
+          0.005999441258609295,
+          -0.013962198980152607,
+          -0.0039379592053592205,
+          -0.016159603372216225,
+          -0.0009269117726944387,
+          0.013871584087610245,
+          -0.008744308724999428,
+          -0.024752886965870857,
+          0.010201692581176758,
+          0.002148320898413658,
+          -0.005055539309978485,
+          -0.0012166897067800164,
+          -0.018938450142741203,
+          0.025311676785349846,
+          -0.011628872714936733,
+          0.011190902441740036,
+          -0.012950335629284382,
+          -0.0034037106670439243,
+          0.01889314316213131,
+          0.030416298657655716,
+          0.01131927315145731,
+          -0.005587900057435036,
+          -0.018908245489001274,
+          0.015419583767652512,
+          0.0063543482683598995,
+          -0.005610553547739983,
+          -0.004553383216261864,
+          0.0215813759714365,
+          -0.017745357006788254,
+          -0.03461477532982826,
+          0.012361341156065464,
+          -0.0003303656994830817,
+          -0.0306579377502203,
+          -0.010231898166239262,
+          -0.006067402195185423,
+          -0.0015272335149347782,
+          0.016235115006566048,
+          0.006603538524359465,
+          -0.004357051569968462,
+          0.001753769931383431,
+          0.0021879649721086025,
+          0.002920432947576046,
+          -0.008457361720502377,
+          -0.0013611067552119493,
+          0.0006659228820353746,
+          0.0007739996653981507,
+          0.020871561020612717,
+          -0.016235115006566048,
+          -0.004409910179674625,
+          -0.0065997629426419735,
+          0.004776144400238991,
+          0.018047407269477844,
+          0.009801478125154972,
+          0.03265145793557167,
+          0.007396416272968054,
+          0.00351697881706059,
+          -0.0040738810785114765,
+          -0.010020463727414608,
+          0.00741529418155551,
+          0.011893165297806263,
+          -0.0017131821950897574,
+          -0.01047353632748127,
+          -0.011870511807501316,
+          -0.014241593889892101,
+          0.007626728154718876,
+          0.013199525885283947,
+          0.00045307297841645777,
+          -0.008464912883937359,
+          -0.011213555932044983,
+          0.025039833039045334,
+          -0.025900671258568764,
+          0.01892334781587124,
+          -0.006724358070641756,
+          -0.002399398945271969,
+          0.023710818961262703,
+          0.015072227455675602,
+          0.015019369311630726,
+          -0.010684970766305923,
+          -0.026957841590046883,
+          0.01670329086482525,
+          -0.0034414667170494795,
+          -0.002335213590413332,
+          -0.001184597029350698,
+          0.01063966378569603,
+          -0.03259104862809181,
+          -0.012897477485239506,
+          0.012429301626980305,
+          -0.0008688617963343859,
+          0.012225419282913208,
+          -0.01611429452896118,
+          0.02363530732691288,
+          0.010979468002915382,
+          0.01697513461112976,
+          -0.012731350027024746,
+          0.00012223531666677445,
+          0.023227540776133537,
+          -0.00040918152080848813,
+          -0.010231898166239262,
+          -0.004474095534533262,
+          -0.007222738116979599,
+          -0.0006470448570325971,
+          -0.013577086851000786,
+          0.012165009044110775,
+          -0.00468930508941412,
+          0.011221107095479965,
+          0.018273942172527313,
+          0.019738879054784775,
+          0.021264225244522095,
+          0.0031998278573155403,
+          -0.0008315776940435171,
+          -0.006905586924403906,
+          0.005625655874609947,
+          -0.0019595406483858824,
+          -0.00024801024119369686,
+          -0.02582515962421894,
+          0.021369941532611847,
+          -0.008797166869044304,
+          0.009408814832568169,
+          0.004055003169924021,
+          1.6061083442764357e-05,
+          0.015721632167696953,
+          0.0012761554680764675,
+          -0.0018764771521091461,
+          0.004530729725956917,
+          -0.012368892319500446,
+          0.00458358833566308,
+          -0.011636423878371716,
+          0.010465985164046288,
+          0.018455171957612038,
+          -0.007702240254729986,
+          -0.009053908288478851,
+          0.009295547381043434,
+          0.0027146621141582727,
+          -0.006724358070641756,
+          0.0071849822998046875,
+          0.010858648456633091,
+          -0.00734355766326189,
+          -0.005274524446576834,
+          0.006418533623218536,
+          -0.01090395636856556,
+          -0.0007268045446835458,
+          0.006645070388913155,
+          -0.015661222860217094,
+          0.01961805857717991,
+          0.003146969247609377,
+          0.01816822588443756,
+          -0.009635351598262787,
+          0.008827371522784233,
+          -0.006592211779206991,
+          0.013954647816717625,
+          -0.02127932757139206,
+          -0.015585710294544697,
+          0.00866879615932703,
+          -0.008744308724999428,
+          0.017277183011174202,
+          0.008593283593654633,
+          0.011175800114870071,
+          -0.004334398079663515,
+          -0.013796071521937847,
+          0.004768592771142721,
+          -0.0010128068970516324,
+          0.0040474520064890385,
+          -0.01958785392343998,
+          0.026942739263176918,
+          0.01955764926970005,
+          0.020358078181743622,
+          -0.016491856426000595,
+          -0.016295524314045906,
+          0.004104085732251406,
+          0.02685212530195713,
+          -0.01547999307513237,
+          0.0008485679281875491,
+          0.01474752463400364,
+          -0.007577645592391491,
+          -0.004270212724804878,
+          0.02052420563995838,
+          -0.0026561403647065163,
+          -0.0376654677093029,
+          -0.013388305902481079,
+          0.00564453424885869,
+          0.003026149934157729,
+          0.003375393571332097,
+          -0.0374540314078331,
+          0.003987042233347893,
+          -0.025447597727179527,
+          0.004677978344261646,
+          -0.008268581703305244,
+          -0.015631018206477165,
+          0.0003327254671603441,
+          -0.001685808994807303,
+          0.009650453925132751,
+          0.005436875391751528,
+          0.00408520782366395,
+          -0.006392104551196098,
+          0.012210316024720669,
+          -0.006720582488924265,
+          -0.002958188997581601,
+          -0.005546368192881346,
+          0.018485376611351967,
+          0.005127275828272104,
+          0.014958959072828293,
+          0.0043872566893696785,
+          -0.01647675409913063,
+          0.00783816259354353,
+          0.010269653983414173,
+          -0.008162864483892918,
+          -0.00756631838157773,
+          0.002348428126424551,
+          -0.004610017407685518,
+          -0.004081432241946459,
+          -0.02061481960117817,
+          -0.0037888227961957455,
+          0.008291235193610191,
+          0.021958936005830765,
+          -0.0025541989598423243,
+          -0.008646142669022083,
+          -0.0029279841110110283,
+          -0.0034716716036200523,
+          0.010465985164046288,
+          -0.015419583767652512,
+          0.021792810410261154,
+          -0.002814715728163719,
+          0.011092736385762691,
+          0.010805790312588215,
+          0.00801939144730568,
+          0.0055388170294463634,
+          0.008442259393632412,
+          0.004217354115098715,
+          0.00964290276169777,
+          0.005595451220870018,
+          -0.02069033309817314,
+          -0.009038805961608887,
+          0.012429301626980305,
+          -0.004934719763696194,
+          -0.015434686094522476,
+          0.03231920674443245,
+          0.008087352849543095,
+          0.0006078729056753218,
+          0.007672035600990057,
+          0.03159428760409355,
+          0.0015848114853724837,
+          0.005772904958575964,
+          0.01123620942234993,
+          0.010828443802893162,
+          -0.0029997206293046474,
+          -0.012429301626980305,
+          -0.044038690626621246,
+          0.0008792447042651474,
+          0.0015404480509459972,
+          0.020750742405653,
+          -0.033225350081920624,
+          0.009590044617652893,
+          -0.013048501685261726,
+          0.008087352849543095,
+          -0.016990236937999725,
+          0.007830611430108547,
+          0.006331694778054953,
+          0.01822863519191742,
+          0.010609458200633526,
+          -0.011032327078282833,
+          0.008865127339959145,
+          0.0015904748579487205,
+          -0.011334375478327274,
+          0.0014083017595112324,
+          0.02810562588274479,
+          0.020599717274308205,
+          0.011168248951435089,
+          -0.0018028528429567814,
+          0.01853068359196186,
+          0.014807934872806072,
+          -0.009129419922828674,
+          -0.027093762531876564,
+          0.003180949715897441,
+          -0.0031564084347337484,
+          -0.004666651599109173,
+          0.01128906849771738,
+          -0.0043457248248159885,
+          0.006361899431794882,
+          0.01750371977686882,
+          -0.006222201976925135,
+          -0.0006158960750326514,
+          0.010360268875956535,
+          0.008117557503283024,
+          -0.0023371013812720776,
+          0.016250217333436012,
+          -0.015615914948284626,
+          0.021158507093787193,
+          -0.025900671258568764,
+          -0.0014696554280817509,
+          0.012625633738934994,
+          0.006958445534110069,
+          0.017820870503783226,
+          -0.01218011137098074,
+          -0.02519085630774498,
+          -0.00976372230798006,
+          0.01249726302921772,
+          0.012957886792719364,
+          0.00420225178822875,
+          0.009031254798173904,
+          0.01196867786347866,
+          -0.004927168600261211,
+          -0.0032451350707560778,
+          0.0012676604092121124,
+          -0.013237281702458858,
+          -0.00722651369869709,
+          -0.009741068817675114,
+          0.010065770708024502,
+          0.0007268045446835458,
+          -0.01079068798571825,
+          0.007283147890120745,
+          -0.02197403833270073,
+          -0.008351645432412624,
+          0.004602466244250536,
+          -0.0024900136049836874,
+          0.0018670381978154182,
+          -0.015313866548240185,
+          0.0074568260461091995,
+          0.00040257422369904816,
+          -0.003881325013935566,
+          -0.010828443802893162,
+          0.022260984405875206,
+          0.00450807623565197,
+          0.003953061532229185,
+          0.013071155175566673,
+          -0.00681119691580534,
+          -0.011538257822394371,
+          0.01544978842139244,
+          -0.009242688305675983,
+          0.0013658262323588133,
+          -0.005939031485468149,
+          -0.012240521609783173,
+          -0.008736756630241871,
+          -0.010488639585673809,
+          -0.032500434666872025,
+          -0.024420632049441338,
+          0.016386138275265694,
+          0.0015574382850900292,
+          0.012338686734437943,
+          0.017216771841049194,
+          -0.012165009044110775,
+          0.008132659830152988,
+          -0.03636665642261505,
+          0.00128653843421489,
+          -0.02155117131769657,
+          -0.01047353632748127,
+          0.0008839642396196723,
+          0.00810245517641306,
+          0.014143427833914757,
+          -0.008011840283870697,
+          0.00790612306445837,
+          -0.0018962991889566183,
+          0.018787425011396408,
+          0.015615914948284626,
+          0.003647237317636609,
+          0.0012893701205030084,
+          -0.007173655088990927,
+          -0.00011834171164082363,
+          0.022064654156565666,
+          -0.002271028235554695,
+          -0.017443308606743813,
+          0.0176396407186985,
+          0.022215677425265312,
+          0.0025202184915542603,
+          -0.02496432140469551,
+          0.005542592611163855,
+          0.0025277696549892426,
+          0.017020441591739655,
+          -0.0040738810785114765,
+          -0.010843546129763126,
+          -0.004610017407685518,
+          -0.0027580817695707083,
+          0.03881325200200081,
+          0.018409864977002144,
+          -0.005047988146543503,
+          -0.03231920674443245,
+          0.0021709746215492487,
+          -0.0028807888738811016,
+          1.456411337130703e-05,
+          0.01988990418612957,
+          0.02393735572695732,
+          0.023559793829917908,
+          -0.0032470228616148233,
+          0.007739996537566185,
+          -0.019980518147349358,
+          0.04331377521157265,
+          -0.007853264920413494,
+          -0.016884518787264824,
+          -0.015102432109415531,
+          -0.009046357125043869,
+          0.003919081296771765,
+          0.008532874286174774,
+          0.006611089687794447,
+          -0.008261030539870262,
+          -0.004704407416284084,
+          0.010428229346871376,
+          0.008631040342152119,
+          -0.017956791445612907,
+          0.013003193773329258,
+          -0.02393735572695732,
+          0.016718393191695213,
+          -0.01603878289461136,
+          0.00035797484451904893,
+          0.0037982617504894733,
+          -0.012202764861285686,
+          -0.029842406511306763,
+          -0.01176479458808899,
+          0.017186567187309265,
+          0.013448716141283512,
+          0.008547976613044739,
+          0.0035679496359080076,
+          0.005757802166044712,
+          -0.009544737637043,
+          -0.00016801456513348967,
+          0.01191581878811121,
+          0.027365608140826225,
+          0.01703554391860962,
+          0.0129427844658494,
+          0.025432495400309563,
+          -0.00737376231700182,
+          -0.039598576724529266,
+          -0.0035056520719081163,
+          0.00943146925419569,
+          0.007883469574153423,
+          0.002072808798402548,
+          0.002338989172130823,
+          -0.019980518147349358,
+          0.0062184263952076435,
+          -0.0003690656740218401,
+          -0.00681119691580534,
+          -0.020901765674352646,
+          0.007966533303260803,
+          -0.013214628212153912,
+          -0.0005021558608859777,
+          0.0007466264651156962,
+          0.00331120821647346,
+          0.007453050464391708,
+          0.012383994646370411,
+          0.0020916869398206472,
+          -0.0013309018686413765,
+          -0.006826299242675304,
+          -0.008079801686108112,
+          -0.016582470387220383,
+          0.01170438528060913,
+          -0.01362239383161068,
+          -0.005195236764848232,
+          -0.004077656660228968,
+          -0.003949285950511694,
+          0.002650476759299636,
+          0.02088666334748268,
+          0.0042098029516637325,
+          0.0027486425824463367,
+          0.01301074493676424,
+          0.008442259393632412,
+          0.0039115301333367825,
+          0.03476579859852791,
+          -0.01836455799639225,
+          -0.0063279191963374615,
+          -0.014150978997349739,
+          -0.01544978842139244,
+          -0.019013961777091026,
+          0.022019345313310623,
+          -0.004296641796827316,
+          -0.010133732110261917,
+          0.007200084626674652,
+          0.014679564163088799,
+          -0.013788520358502865,
+          3.648771235020831e-05,
+          0.010964365676045418,
+          -0.015117534436285496,
+          0.010896405205130577,
+          -0.009544737637043,
+          0.010609458200633526,
+          0.00810245517641306,
+          0.03956837207078934,
+          -0.009295547381043434,
+          -0.014468129724264145,
+          0.0026523645501583815,
+          0.009861888363957405,
+          0.018470274284482002,
+          -0.024677373468875885,
+          0.0013148555299267173,
+          -0.0012733237817883492,
+          -0.000611648487392813,
+          -0.00662241643294692,
+          -0.006252407096326351,
+          0.014415271580219269,
+          -0.006520475260913372,
+          -0.01055660005658865,
+          -0.024949217215180397,
+          -0.017971893772482872,
+          -0.00961269810795784,
+          0.022653648629784584,
+          0.024284711107611656,
+          -0.005399119574576616,
+          -0.005803109612315893,
+          0.0011978116817772388,
+          -0.01567632518708706,
+          0.01700533926486969,
+          0.013629944995045662,
+          -0.02535698376595974,
+          -0.0019000747706741095,
+          -0.009446571581065655,
+          -0.006365675013512373,
+          -0.014543642289936543,
+          0.00878206454217434,
+          0.05584879592061043,
+          -0.02655007503926754,
+          -0.0003390967904124409,
+          -0.01733759231865406,
+          0.00642986036837101,
+          -0.010722726583480835,
+          -0.005074417218565941,
+          -0.01647675409913063,
+          -0.01040557585656643,
+          -0.006656397134065628,
+          -0.006686601787805557,
+          -0.004311744589358568,
+          0.00869900081306696,
+          -0.007294474635273218,
+          -0.02227608673274517,
+          0.016250217333436012,
+          -0.031050600111484528,
+          0.002148320898413658,
+          0.010382922366261482,
+          -0.0017037431243807077,
+          0.019104577600955963,
+          -0.028030114248394966,
+          0.01165152620524168,
+          0.004927168600261211,
+          0.00137148960493505,
+          0.007083040662109852,
+          -0.014641808345913887,
+          -0.003188501112163067,
+          -0.01683921180665493,
+          -0.003941734787076712,
+          0.009748619981110096,
+          -0.010692521929740906,
+          0.023212438449263573,
+          -0.014483232982456684,
+          -0.011070082895457745,
+          0.015827348455786705,
+          0.013977301307022572,
+          0.014672012999653816,
+          -0.013546881265938282,
+          -0.0030412522610276937,
+          -0.010496190749108791,
+          0.016688188537955284,
+          -0.0021974039264023304,
+          -0.021883424371480942,
+          -0.014234042726457119,
+          0.02055441029369831,
+          -0.008042044937610626,
+          0.022381804883480072,
+          -0.002335213590413332,
+          -0.001269548200070858,
+          -0.008472464047372341,
+          0.005191461183130741,
+          -0.0006215595058165491,
+          -0.0025674134958535433,
+          0.017277183011174202,
+          -0.0025070037227123976,
+          0.026776611804962158,
+          -0.01498161256313324,
+          0.015585710294544697,
+          -0.029162796214222908,
+          -0.009204932488501072,
+          -0.00783816259354353,
+          0.0014545529847964644,
+          -0.00017084626597352326,
+          0.0008608386269770563,
+          0.030340787023305893,
+          -0.01291257981210947,
+          -0.02241200953722,
+          0.023741023615002632,
+          -0.015933066606521606,
+          -0.0328930988907814,
+          0.005587900057435036,
+          -0.001396975014358759,
+          0.004768592771142721,
+          0.008917986415326595,
+          -0.007083040662109852,
+          -0.025734543800354004,
+          0.012844618409872055,
+          0.01806250959634781,
+          0.0030374766793102026,
+          4.574975173454732e-05,
+          -0.015004266053438187,
+          -0.019316010177135468,
+          -0.011281517334282398,
+          0.008366747759282589,
+          -0.0040738810785114765,
+          0.020463796332478523,
+          -0.017262080684304237,
+          -0.00405122758820653,
+          -0.006361899431794882,
+          -0.008495118468999863,
+          0.011424990370869637,
+          0.028966465964913368,
+          0.010601907037198544,
+          0.004417461343109608,
+          -0.00123084825463593,
+          0.0015225139213725924,
+          0.0021728624124079943,
+          0.008064698427915573,
+          0.009325752034783363,
+          -0.006444963160902262,
+          -0.0016839212039485574,
+          -0.009287996217608452,
+          -0.0024088378995656967,
+          -0.01335810124874115,
+          0.003643461735919118,
+          -0.003515091026201844,
+          0.005678514484316111,
+          -0.01975398138165474,
+          0.01020924374461174,
+          -0.010284756310284138,
+          0.0031564084347337484,
+          -0.02088666334748268,
+          -0.002552310936152935,
+          0.017473513260483742,
+          -0.010277205146849155,
+          0.018621299415826797,
+          0.0010618898086249828,
+          0.026097003370523453,
+          0.008042044937610626,
+          -0.0030242621432989836,
+          0.003998368978500366,
+          0.0055652461014688015,
+          0.00476481718942523,
+          0.014173632487654686,
+          0.004153168760240078,
+          -0.012618082575500011,
+          0.006241080351173878,
+          0.012550121173262596,
+          -0.008532874286174774,
+          0.011674179695546627,
+          -0.0009844898013398051,
+          0.01700533926486969,
+          -0.010775585658848286,
+          0.010911507532000542,
+          0.0079363277181983,
+          0.007891020737588406,
+          -0.023378565907478333,
+          -0.0038020373322069645,
+          0.023650409653782845,
+          -0.014543642289936543,
+          0.07092101871967316,
+          -0.005893724039196968,
+          0.025508008897304535,
+          -0.008570630103349686,
+          -0.012104599736630917,
+          -0.008034493774175644,
+          -0.004323071334511042,
+          0.007898571901023388,
+          -0.010601907037198544,
+          0.027456222102046013,
+          -0.011606219224631786,
+          0.00439103227108717,
+          -0.006788543425500393,
+          0.003760505700483918,
+          0.034916821867227554,
+          0.011198453605175018,
+          0.021369941532611847,
+          0.012210316024720669,
+          0.010662317276000977,
+          0.00040918152080848813,
+          -0.004693080671131611,
+          0.010118629783391953,
+          0.006414758041501045,
+          0.012776657938957214,
+          -0.014996714890003204,
+          0.012217868119478226,
+          0.01020924374461174,
+          0.01683921180665493,
+          -0.014611602760851383,
+          -0.004719510208815336,
+          0.011266414076089859,
+          0.0012761554680764675,
+          -0.012482160702347755,
+          -0.011643975041806698,
+          0.015079778619110584,
+          0.015042022801935673,
+          0.008563078939914703,
+          0.00810245517641306,
+          -0.00987699069082737,
+          0.0012563335476443172,
+          0.008744308724999428,
+          -0.01188561413437128,
+          0.005723821930587292,
+          -0.006875382270663977,
+          -0.006346797104924917,
+          0.007951430976390839,
+          0.003971939906477928,
+          -0.002012399025261402,
+          -0.013765866868197918,
+          0.010994570329785347,
+          -0.010254551656544209,
+          -0.01567632518708706,
+          -0.012044189497828484,
+          0.000821666675619781,
+          0.012202764861285686,
+          -0.0076607088558375835,
+          -0.002223833231255412,
+          -0.012587876990437508,
+          0.0036453495267778635,
+          0.017292285338044167,
+          0.007588972337543964,
+          0.020660126581788063,
+          0.023227540776133537
+        ],
+        "title": "Addressing Performance Challenges",
+        "keyphrases": [
+          "Shrinking Batch Problem",
+          "Mixing Data Parallelism and Model Parallelism",
+          "hierarchical MoE",
+          "network bandwidth",
+          "computational efficiency"
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "ff7dd6f5-6560-42e5-923e-f4091e278202",
+      "properties": {
+        "page_content": "4 Balancing Expert Utilization\n\nWe have observed that the gating network tends to converge to a state where it always produces large weights for the same few experts. This imbalance is self-reinforcing, as the favored experts are trained more rapidly and thus are selected even more by the gating network. Eigen et al. (2013) describe the same phenomenon, and use a hard constraint at the beginning of training to avoid this local minimum. Bengio et al. (2015) include a soft constraint on the batch-wise average of each gate.1 We take a soft constraint approach. We define the importance of an expert relative to a batch of training examples to be the batchwise sum of the gate values for that expert. We define an additional loss Limportance, which is added to the overall loss function for the model. This loss is equal to the square of the coefficient of variation of the set of importance values, multiplied by a hand-tuned scaling factor wimportance. This additional loss encourages all experts to have equal importance.\n\n$$I m p o r t a n c e(X)=\\sum_{x\\in X}G(x)$$ $$L_{importance}(X)=w_{importance}\\cdot CV(Importance(X))^{2}\\tag{7}$$ $$(6)$$\n\nWhile this loss function can ensure equal importance, experts may still receive very different numbers of examples. For example, one expert may receive a few examples with large weights, and another may receive many examples with small weights. This can cause memory and performance problems on distributed hardware. To solve this problem, we introduce a second loss function, Lload , which ensures balanced loads. Appendix A contains the definition of this function, along with experimental results.\n\n",
+        "embedding": [
+          -0.013197094202041626,
+          -0.002106932457536459,
+          -0.020997902378439903,
+          -0.01955360174179077,
+          -0.0042971898801624775,
+          0.013943051919341087,
+          -0.01379227265715599,
+          0.05729586258530617,
+          -0.02623547427356243,
+          0.009284786880016327,
+          -0.014958823099732399,
+          0.013482780195772648,
+          -0.0004461359931156039,
+          -0.0026227538473904133,
+          0.012546366080641747,
+          0.013776401989161968,
+          0.013578008860349655,
+          -0.0015841671265661716,
+          0.0022577110212296247,
+          -0.008999100886285305,
+          0.036821719259023666,
+          -0.03621860593557358,
+          -0.02348971739411354,
+          0.012839986942708492,
+          -0.011721051298081875,
+          0.029409760609269142,
+          0.025187959894537926,
+          -0.015276252292096615,
+          -0.048630062490701675,
+          0.009760930202901363,
+          0.025886302813887596,
+          -0.002469991333782673,
+          -0.04361468926072121,
+          -0.004309093579649925,
+          -0.04536054655909538,
+          -0.018458474427461624,
+          0.04875703528523445,
+          0.02812417410314083,
+          -0.012403522618114948,
+          0.030568374320864677,
+          0.02133120223879814,
+          0.006888200994580984,
+          -0.03971032053232193,
+          0.01788710243999958,
+          0.009380015544593334,
+          -0.032631661742925644,
+          -0.03777400404214859,
+          -0.0011179439024999738,
+          -0.0026505289133638144,
+          -0.03621860593557358,
+          0.013538329862058163,
+          -0.05221700668334961,
+          0.005737522151321173,
+          -0.018934616819024086,
+          -0.011316330172121525,
+          0.02893361821770668,
+          -0.010998900979757309,
+          0.014236672781407833,
+          0.042598918080329895,
+          0.01918855868279934,
+          0.006015272345393896,
+          0.03012397512793541,
+          -0.055962663143873215,
+          0.021616889163851738,
+          0.023997602984309196,
+          -0.04751906171441078,
+          -0.0064636399038136005,
+          0.01644280180335045,
+          -0.001340143964625895,
+          -0.012998702004551888,
+          -0.06805669516324997,
+          0.02564823254942894,
+          -0.0355837456882,
+          0.025187959894537926,
+          0.005527225788682699,
+          -0.006130340043455362,
+          0.042027547955513,
+          0.03217139095067978,
+          -0.0028231306932866573,
+          0.013181222602725029,
+          -0.026933817192912102,
+          0.009205429814755917,
+          0.004031343385577202,
+          0.022108903154730797,
+          0.04294808954000473,
+          -0.010895737446844578,
+          -0.00926891528069973,
+          -0.05367717891931534,
+          0.01799820177257061,
+          -0.022077159956097603,
+          -0.015966659411787987,
+          0.02117248810827732,
+          0.01120522990822792,
+          -0.012308293953537941,
+          -0.02818766050040722,
+          0.03694869205355644,
+          -0.000555996026378125,
+          -0.004745557904243469,
+          -0.010451337322592735,
+          -0.008872129954397678,
+          -0.02236284501850605,
+          0.006181922275573015,
+          -0.02958434633910656,
+          -0.029520859941840172,
+          0.030203333124518394,
+          -0.022378716617822647,
+          -0.01826801709830761,
+          0.012459073215723038,
+          0.030250946059823036,
+          0.10710041224956512,
+          -0.009221301414072514,
+          -0.009784736670553684,
+          -0.030838189646601677,
+          0.013371679931879044,
+          -0.03891674801707268,
+          -0.037043917924165726,
+          -0.027632160112261772,
+          -0.06120023503899574,
+          0.025362545624375343,
+          -0.023584945127367973,
+          0.023188160732388496,
+          0.04977280646562576,
+          0.012966958805918694,
+          0.01961708813905716,
+          -0.044916149228811264,
+          0.04409083351492882,
+          -0.023997602984309196,
+          0.027775002643465996,
+          0.008102364838123322,
+          0.0019363145111128688,
+          0.0032278522849082947,
+          0.02650528959929943,
+          -0.02269614487886429,
+          -0.044376518577337265,
+          0.006757261697202921,
+          0.013300258666276932,
+          0.06339049339294434,
+          0.02499750256538391,
+          0.01441919431090355,
+          -0.034345775842666626,
+          0.01129252277314663,
+          -0.053645435720682144,
+          -0.019220301881432533,
+          0.015982531011104584,
+          -0.00685645779594779,
+          0.016347574070096016,
+          -0.00514631113037467,
+          0.020442401990294456,
+          -0.016887202858924866,
+          0.010364044457674026,
+          -0.005693875718861818,
+          0.04843960702419281,
+          -0.006166050676256418,
+          0.024108702316880226,
+          0.019061587750911713,
+          0.016506288200616837,
+          -0.028949489817023277,
+          0.025553002953529358,
+          -0.015427030622959137,
+          0.0011467108270153403,
+          -0.0006834634696133435,
+          -0.001254834933206439,
+          -0.015847623348236084,
+          0.02344210259616375,
+          0.024965759366750717,
+          0.06729486584663391,
+          0.01676023006439209,
+          0.013903372921049595,
+          -0.07834137976169586,
+          0.026759231463074684,
+          -0.0036087664775550365,
+          -0.03891674801707268,
+          0.03996426239609718,
+          0.03631383180618286,
+          2.3481657990487292e-05,
+          0.02656877413392067,
+          -0.007566704414784908,
+          0.017760131508111954,
+          -0.014720751903951168,
+          0.03980554640293121,
+          -0.03479017689824104,
+          -0.014744559302926064,
+          0.008058718405663967,
+          0.007725419010967016,
+          0.01649041660130024,
+          0.008308693766593933,
+          0.026743359863758087,
+          0.012879665940999985,
+          -0.019442502409219742,
+          0.004567004274576902,
+          0.028378117829561234,
+          0.01788710243999958,
+          0.014355708844959736,
+          -0.003918259404599667,
+          0.016887202858924866,
+          -0.03574246168136597,
+          0.010935415513813496,
+          -0.03307606279850006,
+          0.0299017746001482,
+          0.0007930755382403731,
+          -0.005070921964943409,
+          -0.02004561759531498,
+          0.013974794186651707,
+          -0.015831751748919487,
+          -0.023997602984309196,
+          0.012403522618114948,
+          0.03732960298657417,
+          0.004404321778565645,
+          0.013189158402383327,
+          -0.03539329022169113,
+          0.009078457951545715,
+          -0.02053763158619404,
+          -0.018093431368470192,
+          0.004916175734251738,
+          -0.05082032084465027,
+          -0.01330819446593523,
+          0.0177760012447834,
+          0.058819521218538284,
+          0.042376719415187836,
+          -0.007765097543597221,
+          -0.026965560391545296,
+          0.02139468863606453,
+          0.031076261773705482,
+          0.00046944714267738163,
+          -0.0034480681642889977,
+          -0.014585844241082668,
+          -0.004142443183809519,
+          -0.013808144256472588,
+          0.008927679620683193,
+          -0.014593780040740967,
+          0.007959522306919098,
+          0.015442902222275734,
+          0.03767877444624901,
+          0.021902574226260185,
+          -0.0007375255227088928,
+          -0.0022081127390265465,
+          -0.012522558681666851,
+          -0.005023307632654905,
+          0.0010033720172941685,
+          -0.03552026301622391,
+          -0.026600517332553864,
+          -0.017474444583058357,
+          -0.03840886056423187,
+          0.0005227652145549655,
+          -0.011403623037040234,
+          0.011356008239090443,
+          -0.00880864355713129,
+          0.03821840509772301,
+          -0.0026088664308190346,
+          -0.011022708378732204,
+          -0.02785436064004898,
+          -0.027394089847803116,
+          -0.008149979636073112,
+          0.021109001711010933,
+          -0.011363944038748741,
+          0.014196994714438915,
+          0.02785436064004898,
+          -0.047614291310310364,
+          0.011840087361633778,
+          -0.0018361260881647468,
+          0.030949288979172707,
+          -0.004781268537044525,
+          -0.022267617285251617,
+          0.01896636001765728,
+          -0.01371291559189558,
+          -0.03599640354514122,
+          0.047233376652002335,
+          0.0371708907186985,
+          0.024235675111413002,
+          -0.005400254391133785,
+          0.020616987720131874,
+          -0.027552803978323936,
+          0.006423961371183395,
+          -0.0549786351621151,
+          0.03675823286175728,
+          -0.03640906140208244,
+          -0.025505388155579567,
+          -0.007392118684947491,
+          -0.014133509248495102,
+          0.050026748329401016,
+          -0.007046915125101805,
+          0.034028347581624985,
+          0.008967358618974686,
+          -0.007400054484605789,
+          0.03644080460071564,
+          -0.012355908751487732,
+          -0.02187083102762699,
+          -0.031330205500125885,
+          -0.047614291310310364,
+          -0.011848023161292076,
+          -0.0003290841996204108,
+          -0.010014872997999191,
+          -0.001375854597426951,
+          0.006765197031199932,
+          0.04707466438412666,
+          -0.00778493657708168,
+          -0.02625134587287903,
+          0.0028211467433720827,
+          0.01669674552977085,
+          -0.019315531477332115,
+          0.005578808020800352,
+          0.0038865164387971163,
+          0.01633170247077942,
+          -0.026759231463074684,
+          -0.02537841722369194,
+          0.006816779263317585,
+          -0.033488716930150986,
+          0.01680784486234188,
+          -0.03137781843543053,
+          0.0004920143401250243,
+          0.015982531011104584,
+          0.0218232162296772,
+          -0.0049836291000247,
+          -0.05637532100081444,
+          -0.04078957810997963,
+          0.01653803139925003,
+          0.015696844086050987,
+          0.03191744536161423,
+          0.0010425546206533909,
+          0.0014323965879157186,
+          0.016633259132504463,
+          0.01586349494755268,
+          -0.03358394652605057,
+          0.013173287734389305,
+          -0.026743359863758087,
+          -0.010165651328861713,
+          0.01029262226074934,
+          0.04564623534679413,
+          -0.04551926255226135,
+          -0.011030644178390503,
+          -0.012752694077789783,
+          -0.0072770509868860245,
+          -0.03983728960156441,
+          -0.013578008860349655,
+          -0.0009036795818246901,
+          -0.015046115964651108,
+          -0.015276252292096615,
+          0.012951087206602097,
+          -0.014054151251912117,
+          -0.047550804913043976,
+          -0.016839588060975075,
+          0.014530294574797153,
+          -0.023505588993430138,
+          0.014696944504976273,
+          0.02780674584209919,
+          0.01058624405413866,
+          -0.020664602518081665,
+          0.0055311936885118484,
+          0.01597459428012371,
+          0.008673736825585365,
+          0.0010921527864411473,
+          0.016014274209737778,
+          2.0304270947235636e-05,
+          0.03221900388598442,
+          -0.0020751894917339087,
+          -0.010633858852088451,
+          -0.0009309586021117866,
+          0.03212377429008484,
+          -0.015807945281267166,
+          -0.016299959272146225,
+          0.014863594435155392,
+          -0.02042653039097786,
+          -0.06027969345450401,
+          -0.007530993781983852,
+          -0.027775002643465996,
+          0.00897529348731041,
+          -0.011641694232821465,
+          -0.020029745995998383,
+          -0.02758454531431198,
+          0.0034659237135201693,
+          -0.012705080211162567,
+          0.0013252644566819072,
+          0.029028845950961113,
+          0.022394588217139244,
+          0.011038579978048801,
+          0.015585744753479958,
+          -0.001685347524471581,
+          0.0027318699285387993,
+          -0.055264320224523544,
+          0.03237771987915039,
+          -0.04529706388711929,
+          0.01939488761126995,
+          0.007729386445134878,
+          -0.010848122648894787,
+          0.02053763158619404,
+          0.027013175189495087,
+          -0.0009607175015844405,
+          0.0074794115498661995,
+          0.019410759210586548,
+          0.04355120658874512,
+          -0.015800008550286293,
+          -0.05570872128009796,
+          -0.0387897752225399,
+          -0.0019601215608417988,
+          -0.011974994093179703,
+          -0.00724927568808198,
+          0.01158614456653595,
+          -0.024426132440567017,
+          -0.004781268537044525,
+          -0.009602215141057968,
+          0.006662033032625914,
+          0.018442602828145027,
+          0.0009369103936478496,
+          0.009308594278991222,
+          -0.020934415981173515,
+          -0.003948018420487642,
+          -0.030742960050702095,
+          0.008443601429462433,
+          0.021521659567952156,
+          -0.00889593642205,
+          0.00948318000882864,
+          -0.002079157391563058,
+          -0.030362047255039215,
+          0.010792572982609272,
+          0.02791784703731537,
+          0.030187461525201797,
+          -0.01660151593387127,
+          0.021902574226260185,
+          -0.010324365459382534,
+          -0.008261079899966717,
+          -0.007530993781983852,
+          0.029647832736372948,
+          0.0678662359714508,
+          0.022029545158147812,
+          0.007546865381300449,
+          -0.02063285931944847,
+          0.0073683117516338825,
+          -0.02155340276658535,
+          0.012657465413212776,
+          -0.028171788901090622,
+          0.03431403264403343,
+          0.03621860593557358,
+          0.012800308875739574,
+          -0.0014046216383576393,
+          0.017283987253904343,
+          -0.029568474739789963,
+          -0.020823316648602486,
+          0.03093341737985611,
+          -0.03282211720943451,
+          -0.029870031401515007,
+          0.014101766049861908,
+          -0.002797339577227831,
+          0.010943351313471794,
+          0.016204729676246643,
+          -0.02920343168079853,
+          -0.042979832738637924,
+          -0.002227952005341649,
+          -0.031155617907643318,
+          -0.02052175998687744,
+          -0.012720951810479164,
+          -0.0625016912817955,
+          -0.002642593113705516,
+          -0.046630263328552246,
+          -0.014720751903951168,
+          0.013133608736097813,
+          -0.008003168739378452,
+          0.011482980102300644,
+          -0.03517109155654907,
+          0.005483579356223345,
+          -0.019966259598731995,
+          -0.026632260531187057,
+          0.015188959427177906,
+          0.045233577489852905,
+          -0.004166250582784414,
+          -0.0059676580131053925,
+          0.027187760919332504,
+          -0.039361149072647095,
+          -0.01750618778169155,
+          0.009784736670553684,
+          0.00778493657708168,
+          -0.014641394838690758,
+          -0.04009123519062996,
+          -0.021839087828993797,
+          0.014244608581066132,
+          0.019315531477332115,
+          0.021632758900523186,
+          -0.004257511347532272,
+          -0.04989977926015854,
+          -0.03520283102989197,
+          -0.030250946059823036,
+          0.039361149072647095,
+          0.0005202853353694081,
+          0.004436064977198839,
+          -0.023616688326001167,
+          -0.009062586352229118,
+          0.004531293176114559,
+          0.022712016478180885,
+          0.009253043681383133,
+          -0.028203532099723816,
+          -0.0049677579663693905,
+          0.0006284094415605068,
+          0.005114568397402763,
+          -0.030139846727252007,
+          0.025441903620958328,
+          0.0013649429893121123,
+          -0.05961309373378754,
+          0.008149979636073112,
+          -0.02926691807806492,
+          0.007344504352658987,
+          0.01409383025020361,
+          -0.010149779729545116,
+          -0.03093341737985611,
+          0.01676023006439209,
+          0.04085306078195572,
+          0.020759830251336098,
+          -0.026759231463074684,
+          0.012713016010820866,
+          0.012927279807627201,
+          -0.030917545780539513,
+          -0.04104351997375488,
+          -0.002900503808632493,
+          0.004801107570528984,
+          -0.009713315404951572,
+          -0.011221101507544518,
+          -0.002594978781417012,
+          0.03475843369960785,
+          -0.009689508005976677,
+          0.0036841558758169413,
+          0.022712016478180885,
+          0.021854959428310394,
+          -0.021680373698472977,
+          0.014577909372746944,
+          -0.01153059396892786,
+          0.033266518265008926,
+          -0.027029046788811684,
+          0.03593291714787483,
+          0.009118136949837208,
+          -0.0022636628709733486,
+          0.031441304832696915,
+          -0.00964189413934946,
+          -0.02042653039097786,
+          0.02321990206837654,
+          -0.0007915875758044422,
+          -0.02026781626045704,
+          -0.029489118605852127,
+          -0.01058624405413866,
+          -0.0069159758277237415,
+          0.026632260531187057,
+          -0.022267617285251617,
+          -0.023315131664276123,
+          0.01298283040523529,
+          0.042027547955513,
+          0.009300658479332924,
+          0.026600517332553864,
+          -0.0004915183526463807,
+          -0.00017086586740333587,
+          0.04964583367109299,
+          -0.03659952059388161,
+          0.02312467433512211,
+          0.018823517486453056,
+          -0.021743860095739365,
+          -0.01182421576231718,
+          -0.0014056136133149266,
+          0.011236973106861115,
+          -0.010284687392413616,
+          -0.006328732706606388,
+          0.003662332659587264,
+          0.005570872221142054,
+          0.0020394788589328527,
+          -0.02112487331032753,
+          0.010252944193780422,
+          0.0006199777708388865,
+          0.005812911316752434,
+          0.017204631119966507,
+          -0.02229936048388481,
+          -0.026473546400666237,
+          0.025394288823008537,
+          0.027092531323432922,
+          -0.010197394527494907,
+          0.014379516243934631,
+          -0.0027001271955668926,
+          -0.01072115171700716,
+          0.01644280180335045,
+          -0.005150279030203819,
+          0.002200176939368248,
+          0.0002386914420640096,
+          0.02650528959929943,
+          -0.022013673558831215,
+          -0.017220502719283104,
+          0.015871429815888405,
+          0.016617387533187866,
+          -0.031758733093738556,
+          0.005511354189366102,
+          0.01745857298374176,
+          0.014030344784259796,
+          -0.04240846261382103,
+          -0.0076976437121629715,
+          -0.023854760453104973,
+          -0.00869754422456026,
+          -0.01462552323937416,
+          -0.03939289227128029,
+          0.03291734680533409,
+          -0.003497666446492076,
+          -0.026917945593595505,
+          -0.010975094512104988,
+          -0.032361846417188644,
+          0.011514722369611263,
+          0.014435065910220146,
+          0.025346674025058746,
+          -0.073262520134449,
+          0.011348072439432144,
+          -0.010729086585342884,
+          0.01441125851124525,
+          -0.007975393906235695,
+          -0.008903872221708298,
+          0.02510860376060009,
+          -0.00015301050734706223,
+          0.021854959428310394,
+          0.009260979481041431,
+          0.0016089662676677108,
+          0.04285286366939545,
+          0.008134108036756516,
+          0.028140045702457428,
+          -0.016823716461658478,
+          0.038123175501823425,
+          0.013276451267302036,
+          0.0025275254156440496,
+          -0.00956253707408905,
+          -0.0061700185760855675,
+          0.03980554640293121,
+          0.06275563687086105,
+          0.00560261495411396,
+          0.03374266251921654,
+          0.0013659349642693996,
+          0.014482680708169937,
+          0.004221800714731216,
+          -0.01107825804501772,
+          -0.02980654686689377,
+          0.0040452308021485806,
+          -0.030917545780539513,
+          -0.01955360174179077,
+          -0.011752794496715069,
+          -0.002979861106723547,
+          -0.021410560235381126,
+          -0.0062454077415168285,
+          0.010768765583634377,
+          -0.01018152292817831,
+          0.044693946838378906,
+          0.009816479869186878,
+          -0.02215651609003544,
+          -0.0037297862581908703,
+          0.061136748641729355,
+          0.0048249149695038795,
+          0.03466320410370827,
+          -0.014593780040740967,
+          0.017125273123383522,
+          -0.00552325788885355,
+          -0.0011695260182023048,
+          0.006328732706606388,
+          -0.010729086585342884,
+          -0.025045117363333702,
+          0.03698043152689934,
+          0.002549348631873727,
+          0.033647432923316956,
+          0.011117937043309212,
+          -0.025362545624375343,
+          0.009554601274430752,
+          -0.012617787346243858,
+          0.015688909217715263,
+          0.01417318731546402,
+          0.0067136152647435665,
+          -0.011363944038748741,
+          0.008657865226268768,
+          -0.025299061089754105,
+          0.015300058759748936,
+          -0.0014631474623456597,
+          0.0001887212274596095,
+          0.02306118793785572,
+          -0.017093529924750328,
+          0.0060589187778532505,
+          -0.01376846618950367,
+          -0.0280130747705698,
+          0.015696844086050987,
+          0.024616587907075882,
+          0.021204231306910515,
+          -0.023077059537172318,
+          -0.011578208766877651,
+          0.04355120658874512,
+          0.013943051919341087,
+          -0.002037494909018278,
+          -0.006757261697202921,
+          -0.0017160983989015222,
+          -0.008221400901675224,
+          -0.01087986584752798,
+          -0.012554301880300045,
+          -0.006435865070670843,
+          -0.011998801492154598,
+          0.0490427203476429,
+          -0.014903273433446884,
+          0.00878483708947897,
+          0.011038579978048801,
+          -0.015276252292096615,
+          -0.002704094862565398,
+          0.0030215235892683268,
+          -0.007769065443426371,
+          -0.0056224544532597065,
+          -0.0039757932536304,
+          0.030489018186926842,
+          0.016506288200616837,
+          0.024600718170404434,
+          -0.037900976836681366,
+          -0.010705280117690563,
+          0.002364843152463436,
+          0.02166450209915638,
+          0.021029645577073097,
+          0.005872429348528385,
+          -0.0029600218404084444,
+          -0.0015623439103364944,
+          -0.02888600341975689,
+          -0.015736524015665054,
+          -0.017093529924750328,
+          0.03459971770644188,
+          -0.025394288823008537,
+          0.03313954547047615,
+          0.02123597450554371,
+          0.008530894294381142,
+          0.0052812183275818825,
+          -0.01511753723025322,
+          -0.034726690500974655,
+          0.015379415825009346,
+          -0.01519689429551363,
+          0.0006452728412114084,
+          0.0290923323482275,
+          -0.04374166205525398,
+          -0.04567797854542732,
+          -0.003692091442644596,
+          0.016665002331137657,
+          0.024553103372454643,
+          0.001200276892632246,
+          0.0011070322943851352,
+          0.012625723145902157,
+          -0.012157515622675419,
+          -0.011808344163000584,
+          0.004654297139495611,
+          0.01880764588713646,
+          -0.005860525649040937,
+          -0.0061501795426011086,
+          0.025981532409787178,
+          0.00932446587830782,
+          -0.03926591947674751,
+          -0.002416425384581089,
+          0.006193825975060463,
+          0.022553302347660065,
+          0.02301357500255108,
+          -0.03142543137073517,
+          0.015855558216571808,
+          0.013189158402383327,
+          0.04459872096776962,
+          -0.012990766204893589,
+          0.008388050831854343,
+          0.038662806153297424,
+          -0.005558968521654606,
+          0.03082231804728508,
+          -0.004634457640349865,
+          -0.03345697373151779,
+          0.02328338846564293,
+          0.0011100082192569971,
+          -0.01831563003361225,
+          -0.012411458417773247,
+          0.007070722058415413,
+          0.004777300637215376,
+          0.027886103838682175,
+          0.025457775220274925,
+          0.0033230807166546583,
+          0.028965361416339874,
+          0.038885004818439484,
+          0.0057176826521754265,
+          0.013236773200333118,
+          0.007237372454255819,
+          -0.015069923363626003,
+          -0.00490427203476429,
+          -0.006729486398398876,
+          0.015323866158723831,
+          -0.04002774879336357,
+          0.020061487331986427,
+          -0.003616702277213335,
+          0.01982341706752777,
+          -0.00529312202706933,
+          -0.012435265816748142,
+          -0.0007657965179532766,
+          -0.02220413088798523,
+          0.02391824498772621,
+          0.0025989466812461615,
+          0.001375854597426951,
+          -0.021077260375022888,
+          -0.024426132440567017,
+          0.00475746113806963,
+          -0.02328338846564293,
+          0.011260779574513435,
+          0.026489417999982834,
+          0.010356108658015728,
+          0.003874612972140312,
+          -0.03713914752006531,
+          0.02123597450554371,
+          0.02979067526757717,
+          -0.0021386751905083656,
+          0.017125273123383522,
+          0.030044617131352425,
+          -0.021886702626943588,
+          -0.015942851081490517,
+          0.015934916213154793,
+          -0.02715601772069931,
+          -0.0019750010687857866,
+          -0.0002596466802060604,
+          -0.05082032084465027,
+          -0.04856657609343529,
+          0.013125672936439514,
+          -0.003858741605654359,
+          0.015379415825009346,
+          -0.010483079589903355,
+          -0.005793072283267975,
+          0.004912207834422588,
+          -0.0005872429464943707,
+          0.006527125835418701,
+          -0.0059676580131053925,
+          0.0056264218874275684,
+          -0.013498651795089245,
+          -0.008515022695064545,
+          -0.02790197543799877,
+          -0.005308993626385927,
+          -0.018712416291236877,
+          -0.013514523394405842,
+          -0.027441702783107758,
+          0.020172588527202606,
+          -0.00964189413934946,
+          -0.012419394217431545,
+          0.019871031865477562,
+          0.013101865537464619,
+          0.020347174257040024,
+          0.010197394527494907,
+          -0.003904371988028288,
+          0.028092432767152786,
+          -0.03409183397889137,
+          0.03615511953830719,
+          0.006328732706606388,
+          0.02677510306239128,
+          -0.002079157391563058,
+          -0.020680474117398262,
+          0.009673637337982655,
+          0.03437751904129982,
+          0.0016109502175822854,
+          -0.013808144256472588,
+          -0.015260380692780018,
+          -0.004376546945422888,
+          -0.005059018265455961,
+          -0.007955554872751236,
+          -0.008872129954397678,
+          0.010744958184659481,
+          -0.0036841558758169413,
+          0.001297489390708506,
+          -0.006884233094751835,
+          0.04193231835961342,
+          0.011713115498423576,
+          -0.004539228975772858,
+          -0.023267516866326332,
+          -0.02758454531431198,
+          0.01999800279736519,
+          -0.00280130747705698,
+          -0.024616587907075882,
+          0.026378316804766655,
+          -0.014054151251912117,
+          -0.004654297139495611,
+          0.00723340455442667,
+          -0.018077559769153595,
+          0.013816080056130886,
+          -0.021362945437431335,
+          0.005979561246931553,
+          -0.041837092489004135,
+          -0.011959122493863106,
+          -0.03253643214702606,
+          -0.005856557749211788,
+          0.0177760012447834,
+          -0.007447668816894293,
+          0.02441026084125042,
+          -0.016728486865758896,
+          -0.0015425046440213919,
+          -0.007332601118832827,
+          -0.013887501321732998,
+          -0.0088880006223917,
+          -0.0050947293639183044,
+          -0.005245507694780827,
+          0.0191250741481781,
+          -0.002251759171485901,
+          -0.02144230343401432,
+          0.027870232239365578,
+          -0.024648331105709076,
+          1.071011593012372e-05,
+          -0.04659852012991905,
+          -0.03640906140208244,
+          -0.04085306078195572,
+          0.02375953085720539,
+          -0.0169665589928627,
+          -0.04942363500595093,
+          -0.009657765738666058,
+          0.00434877211228013,
+          0.0159031730145216,
+          -0.02490227483212948,
+          -0.02434677444398403,
+          0.022220002487301826,
+          -0.011086193844676018,
+          0.0034659237135201693,
+          -0.01642693020403385,
+          -0.011760730296373367,
+          -0.017157016322016716,
+          0.04091654717922211,
+          -0.02348971739411354,
+          0.022870730608701706,
+          -0.01492708083242178,
+          0.007296890020370483,
+          -0.007273083087056875,
+          -0.015450838021934032,
+          0.03383788838982582,
+          0.025299061089754105,
+          -0.04713814705610275,
+          0.04018646106123924,
+          0.020982030779123306,
+          0.03086993284523487,
+          -0.04459872096776962,
+          0.016617387533187866,
+          -0.030187461525201797,
+          8.307702228194103e-05,
+          0.04063086211681366,
+          0.004412257578223944,
+          0.009856157936155796,
+          -0.011498851701617241,
+          -0.020172588527202606,
+          0.003795255906879902,
+          0.010221200995147228,
+          0.000911119335796684,
+          -0.02791784703731537,
+          -0.007130240090191364,
+          0.007455604616552591,
+          0.02080744504928589,
+          0.00272790202870965,
+          0.02537841722369194,
+          0.03475843369960785,
+          -0.014935015700757504,
+          0.006967558059841394,
+          -0.0458049476146698,
+          0.011903572827577591,
+          -0.005582775454968214,
+          0.01741096004843712,
+          0.025314930826425552,
+          0.02214064635336399,
+          0.030044617131352425,
+          -0.03748831897974014,
+          0.024648331105709076,
+          0.0037516094744205475,
+          -0.005562936421483755,
+          0.0323459766805172,
+          0.0088880006223917,
+          0.03183808922767639,
+          -0.015355609357357025,
+          -0.011276651173830032,
+          0.007273083087056875,
+          0.02375953085720539,
+          -0.017442701384425163,
+          0.01669674552977085,
+          -0.005305025726556778,
+          0.04945537820458412,
+          0.013831951655447483,
+          -0.03488540276885033,
+          -0.027298860251903534,
+          -0.020918546244502068,
+          -0.00918955821543932,
+          0.012863794341683388,
+          0.01112587284296751,
+          -0.01129252277314663,
+          -0.0312667191028595,
+          0.02382301725447178,
+          -0.010998900979757309,
+          -0.014530294574797153,
+          -0.04132920503616333,
+          -0.007054850924760103,
+          -0.017823616042733192,
+          0.0024322967510670424,
+          -0.003579007461667061,
+          -0.01676023006439209,
+          -0.0052970899268984795,
+          0.006427929271012545,
+          0.025314930826425552,
+          -0.025743460282683372,
+          -0.008935615420341492,
+          -0.038662806153297424,
+          -0.005039179231971502,
+          -0.0023509557358920574,
+          0.01729985885322094,
+          -0.002940182574093342,
+          -0.038726288825273514,
+          0.019490117207169533,
+          0.010649729520082474,
+          -0.01433190144598484,
+          0.043932121247053146,
+          -0.036345575004816055,
+          0.005872429348528385,
+          0.03215551748871803,
+          -0.00173097790684551,
+          -0.03086993284523487,
+          -0.0017270100070163608,
+          0.007273083087056875,
+          0.0021049485076218843,
+          0.002737821778282523,
+          0.018077559769153595,
+          0.031123874709010124,
+          0.01276856567710638,
+          0.012633658945560455,
+          0.06977080553770065,
+          -0.00043224848923273385,
+          -0.03086993284523487,
+          0.02285485900938511,
+          -0.001691299257799983,
+          -0.01067353691905737,
+          -0.01376846618950367,
+          0.023045316338539124,
+          -0.02139468863606453,
+          0.011522658169269562,
+          -0.019109202548861504,
+          -0.0004731670196633786,
+          -0.0344727486371994,
+          -0.030108103528618813,
+          -0.01239558681845665,
+          0.022029545158147812,
+          -0.04104351997375488,
+          -0.029822418466210365,
+          0.018283888697624207,
+          0.017712516710162163,
+          -0.004582875408232212,
+          0.0018589412793517113,
+          -0.001100088469684124,
+          0.009340336546301842,
+          0.01761728711426258,
+          -0.0020989966578781605,
+          -0.007519090082496405,
+          0.009872029535472393,
+          -0.0074952831491827965,
+          -0.0066064829006791115,
+          0.02123597450554371,
+          -0.07034218311309814,
+          -0.04409083351492882,
+          0.024045217782258987,
+          -0.008626122027635574,
+          0.002261678921058774,
+          0.022886602208018303,
+          -6.004484384902753e-05,
+          -0.01525244489312172,
+          -0.016887202858924866,
+          0.03463146090507507,
+          -0.012101965956389904,
+          0.022759631276130676,
+          -0.0015266332775354385,
+          -0.0020989966578781605,
+          0.01314948033541441,
+          0.011879765428602695,
+          -0.014736623503267765,
+          0.0018877083202823997,
+          0.015284187160432339,
+          0.014760429970920086,
+          0.00878483708947897,
+          0.017283987253904343,
+          -0.008967358618974686,
+          -0.004094828851521015,
+          -0.0009175670566037297,
+          0.001588135026395321,
+          -0.0019204431446269155,
+          0.012839986942708492,
+          0.031980931758880615,
+          0.0013272484065964818,
+          0.03498063236474991,
+          0.029060589149594307,
+          0.02085505984723568,
+          0.0036960593424737453,
+          -0.004614618606865406,
+          -0.009451436810195446,
+          0.023584945127367973,
+          0.016585644334554672,
+          0.010054551064968109,
+          0.0099672582000494,
+          0.005459771957248449,
+          -0.022505687549710274,
+          0.006971525959670544,
+          -0.0042813182808458805,
+          -0.008380115032196045,
+          0.0071500795893371105,
+          0.007320697419345379,
+          -0.018775902688503265,
+          0.012800308875739574,
+          0.013117737136781216,
+          0.0022041448391973972,
+          0.02229936048388481,
+          -0.01833150163292885,
+          -0.012697144411504269,
+          0.004618586506694555,
+          -0.012705080211162567,
+          0.025679973885416985,
+          -0.0030631860718131065,
+          0.0019531778525561094,
+          0.011387751437723637,
+          0.07459572702646255,
+          -0.026854461058974266,
+          -0.0014829868450760841,
+          -0.015522259287536144,
+          -0.02026781626045704,
+          0.0090229082852602,
+          -0.023156417533755302,
+          0.022553302347660065,
+          0.01822040230035782,
+          -0.004110700450837612,
+          0.04574146121740341,
+          0.0020712215919047594,
+          0.019966259598731995,
+          0.024315031245350838,
+          -0.009197494015097618,
+          -0.02053763158619404,
+          0.0045908112078905106,
+          0.012617787346243858,
+          0.00024241130449809134,
+          0.010252944193780422,
+          0.023902375251054764,
+          0.013943051919341087,
+          -0.003702011192217469,
+          -0.003840886289253831,
+          -0.021299459040164948,
+          0.025949789211153984,
+          0.010078358463943005,
+          -0.025124475359916687,
+          -0.038599319756031036,
+          0.0003905859775841236,
+          0.01239558681845665,
+          0.01236384455114603,
+          0.023632559925317764,
+          0.0033786308486014605,
+          -0.011554401367902756,
+          0.012379716150462627,
+          -0.014101766049861908,
+          0.003928178921341896,
+          0.0042813182808458805,
+          -0.007185790222138166,
+          -0.010245008394122124,
+          0.011348072439432144,
+          -0.004729686304926872,
+          0.003174286102876067,
+          0.00020422067609615624,
+          0.0031484949868172407,
+          -0.010038679465651512,
+          -0.010602115653455257,
+          -0.012276551686227322,
+          -0.0029540699906647205,
+          -0.005678004119545221,
+          0.032409459352493286,
+          -0.013109801337122917,
+          -0.003598846960812807,
+          -0.01166550163179636,
+          0.030250946059823036,
+          -0.001897627953439951,
+          -0.006412058137357235,
+          -0.018553702160716057,
+          0.004301157779991627,
+          -0.000162558164447546,
+          -0.009760930202901363,
+          -0.011117937043309212,
+          0.006979461759328842,
+          -0.01998213119804859,
+          -0.008459472097456455,
+          -0.0019710331689566374,
+          0.009173686616122723,
+          0.0026842555962502956,
+          -0.014958823099732399,
+          -0.014149379916489124,
+          0.021966060623526573,
+          -0.011562337167561054,
+          -0.014244608581066132,
+          -0.0010862010531127453,
+          -0.013490715995430946,
+          0.013800208456814289,
+          -0.007900004275143147,
+          -0.0008501135162077844,
+          -0.02144230343401432,
+          0.021093131974339485,
+          0.02333100326359272,
+          -0.017109401524066925,
+          -0.010959222912788391,
+          -0.031060390174388885,
+          -0.0005088777397759259,
+          0.0037278023082762957,
+          0.009443501010537148,
+          0.03628208860754967,
+          0.013744658790528774,
+          0.004245607648044825,
+          0.011459172703325748,
+          -0.007487347349524498,
+          0.008142043836414814,
+          0.009721251204609871,
+          -0.010014872997999191,
+          0.0011248876107856631,
+          2.0846751795033924e-05,
+          -0.03409183397889137,
+          -0.007054850924760103,
+          -0.036186862736940384,
+          -0.0048606256023049355,
+          0.005896236281841993,
+          -0.009816479869186878,
+          -0.0013966858386993408,
+          0.007975393906235695,
+          -0.006138275843113661,
+          -0.015482580289244652,
+          0.000179917537025176,
+          0.008348372764885426,
+          -0.008300757966935635,
+          -0.0010207314044237137,
+          -0.008792772889137268,
+          0.016299959272146225,
+          0.022870730608701706,
+          0.02295008860528469,
+          0.007828583009541035,
+          0.0005460763932205737,
+          -0.004082925617694855,
+          0.007693675812333822,
+          -0.01360181625932455,
+          -0.013466908596456051,
+          0.004305125679820776,
+          0.00950698647648096,
+          -0.0055311936885118484,
+          0.007423861417919397,
+          0.013490715995430946,
+          -0.0029421665240079165,
+          0.018093431368470192,
+          0.00622160080820322,
+          -0.004051182419061661,
+          0.008824515156447887,
+          -0.004019439686089754,
+          0.005578808020800352,
+          -0.008300757966935635,
+          -0.004253543447703123,
+          0.007943650707602501,
+          0.004217832814902067,
+          0.0149508873000741,
+          -0.0032238843850791454,
+          -0.020601116120815277,
+          -0.0033369683660566807,
+          0.04586843401193619,
+          -0.00028419779846444726,
+          0.013958923518657684,
+          -0.020680474117398262,
+          -0.013316130265593529,
+          0.01080050878226757,
+          0.014030344784259796,
+          0.010919543914496899,
+          -0.015125473029911518,
+          -0.016680873930454254,
+          0.02360081672668457,
+          0.00435274001210928,
+          -0.005451836157590151,
+          -0.035329803824424744,
+          0.0036782040260732174,
+          -0.03786923363804817,
+          0.02215651609003544,
+          0.017855359241366386,
+          -0.0035036182962358,
+          -0.004701911471784115,
+          0.017474444583058357,
+          -0.013197094202041626,
+          0.020299559459090233,
+          -0.01637931540608406,
+          0.019632959738373756,
+          0.0002527029428165406,
+          -0.006864393595606089,
+          -0.01676023006439209,
+          -0.02053763158619404,
+          0.004059118218719959,
+          -0.013578008860349655,
+          0.04326551780104637,
+          0.024981630966067314,
+          0.0067334542982280254,
+          0.0017498251982033253,
+          -0.027790874242782593,
+          -0.013141544535756111,
+          0.00934827234596014,
+          0.019109202548861504,
+          -0.010943351313471794,
+          -0.01592698134481907,
+          -0.00505108293145895,
+          0.013895437121391296,
+          -0.0049518863670527935,
+          -0.025886302813887596,
+          0.005253443494439125,
+          -0.014784237369894981,
+          0.013427230529487133,
+          -0.010919543914496899,
+          0.008118236437439919,
+          0.016014274209737778,
+          -0.00552325788885355,
+          -0.0014839787036180496,
+          0.03517109155654907,
+          0.012308293953537941,
+          -0.01322883740067482,
+          0.006781068630516529,
+          -0.018791774287819862,
+          -0.01880764588713646,
+          0.03250468894839287,
+          -0.01503024436533451,
+          0.01988690160214901,
+          0.008991165086627007,
+          -0.0242515467107296,
+          0.017109401524066925,
+          -0.021696245297789574,
+          0.008610251359641552,
+          -0.008681672625243664,
+          0.0042813182808458805,
+          -0.016728486865758896,
+          -0.02268027327954769,
+          -0.01619679480791092,
+          -0.02526731789112091,
+          0.01734747365117073,
+          -0.013125672936439514,
+          -0.0023588913027197123,
+          -0.013514523394405842,
+          0.025584746152162552,
+          0.0040392791852355,
+          0.024203931912779808,
+          0.01622060127556324,
+          0.003977777436375618,
+          0.00544390082359314,
+          -0.033171288669109344,
+          0.028520960360765457,
+          0.002400553785264492,
+          0.020616987720131874,
+          0.013919244520366192,
+          0.006296989973634481,
+          0.0019809529185295105,
+          0.007892068475484848,
+          0.031108003109693527,
+          0.004602714907377958,
+          0.0011982930591329932,
+          0.03158414736390114,
+          -0.00918955821543932,
+          0.013665301725268364,
+          -2.7558011424844153e-05,
+          0.00886419415473938,
+          -0.003112784354016185,
+          -0.009538729675114155,
+          0.02866380289196968,
+          0.0010326349874958396,
+          -0.00019132513261865824,
+          0.02355320379137993,
+          0.0034500521142035723,
+          -0.04018646106123924,
+          -0.003662332659587264,
+          -0.020950287580490112,
+          -0.016522159799933434,
+          0.014554101973772049,
+          0.005908139981329441,
+          0.002900503808632493,
+          0.021966060623526573,
+          0.0004959821817465127,
+          0.01075289398431778,
+          -0.008269015699625015,
+          -0.016617387533187866,
+          -0.021839087828993797,
+          -0.0036881237756460905,
+          0.02839398942887783,
+          0.011308394372463226,
+          -0.011268715374171734,
+          0.015712715685367584,
+          -0.006050982978194952,
+          -0.00435274001210928,
+          0.007110401056706905,
+          -0.0006695759948343039,
+          -0.010951287113130093,
+          0.0014512438792735338,
+          0.0062454077415168285,
+          -0.02175973169505596,
+          0.005872429348528385,
+          -0.0076222545467317104,
+          -0.0003261083038523793,
+          -0.008459472097456455,
+          -0.020013874396681786,
+          -0.011165550909936428,
+          -0.0035809914115816355,
+          0.03685346245765686,
+          0.013347873464226723,
+          -0.020601116120815277,
+          -0.014323965646326542,
+          0.0003261083038523793,
+          -0.0031385754700750113,
+          -0.0008193625835701823,
+          -0.0020930448081344366,
+          -0.0008977277902886271,
+          -0.014569973573088646,
+          0.006265247240662575,
+          -0.0006269214791245759,
+          0.010871930047869682,
+          0.011776601895689964,
+          0.007189758121967316,
+          -0.01606188714504242,
+          -0.0005068937898613513,
+          -0.02434677444398403,
+          -0.023077059537172318,
+          -0.004110700450837612,
+          -0.0026505289133638144,
+          -0.017490316182374954,
+          -0.0022894537542015314,
+          0.014601715840399265,
+          -0.0138795655220747,
+          0.029774803668260574,
+          -0.0006576724117621779,
+          -0.002549348631873727,
+          0.044757433235645294,
+          0.012998702004551888,
+          -0.00030775697086937726,
+          -0.0021724021062254906,
+          0.01917268894612789,
+          -0.005797040183097124,
+          -0.0029203433077782393,
+          0.005642293486744165,
+          0.006047015078365803,
+          0.013704979792237282,
+          0.026584645733237267,
+          -0.02090267464518547,
+          0.0024203930515795946,
+          -0.006047015078365803,
+          0.015387351624667645,
+          -0.014244608581066132,
+          0.0002603906614240259,
+          -0.02155340276658535,
+          -0.01613330841064453,
+          -0.017696645110845566,
+          0.006626322399824858,
+          -0.0072770509868860245,
+          -0.010467207990586758,
+          -0.0016526127001270652,
+          -0.012173387221992016,
+          0.012776501476764679,
+          0.020870931446552277,
+          -0.02936214581131935,
+          0.0199345164000988,
+          -0.006419993471354246,
+          0.0149508873000741,
+          -0.00755876861512661,
+          0.015149280428886414,
+          -0.0020315430592745543,
+          -0.013212965801358223,
+          0.006820747163146734,
+          -0.03752006217837334,
+          -0.017633158713579178,
+          -0.0064953831024467945,
+          0.002751709194853902,
+          -0.03152066096663475,
+          -0.0048764972016215324,
+          0.030901676043868065,
+          0.0035809914115816355,
+          -0.012578108347952366,
+          0.002473959233611822,
+          0.025299061089754105,
+          0.002876696875318885,
+          0.015268316492438316,
+          0.002487846650183201,
+          -0.016887202858924866,
+          -0.004340836312621832,
+          0.02521970309317112,
+          -0.011848023161292076,
+          -0.01026881579309702,
+          -0.024807045236229897,
+          0.03605988994240761,
+          -0.008872129954397678,
+          -0.00140065373852849,
+          0.023949988186359406,
+          -0.006562836468219757,
+          0.005705779418349266,
+          0.0021803376730531454,
+          -0.020553503185510635,
+          0.0231722891330719,
+          0.029520859941840172,
+          0.004197993315756321,
+          -0.02936214581131935,
+          -0.004622553940862417,
+          -0.003910323604941368,
+          0.003116752253845334,
+          0.00015648239059373736,
+          0.002900503808632493,
+          0.002198193222284317,
+          -0.009864093735814095,
+          -0.02079157344996929,
+          -0.00028742168797180057,
+          -0.015720652416348457,
+          -0.010903672315180302,
+          0.018553702160716057,
+          -0.006102565210312605,
+          0.015577808953821659,
+          0.01376846618950367,
+          0.009610150940716267,
+          0.0057494258508086205,
+          0.020886803045868874,
+          0.02893361821770668,
+          -0.014958823099732399,
+          0.00951492227613926,
+          -0.017490316182374954,
+          0.01245113741606474,
+          0.007070722058415413,
+          -0.006669968832284212,
+          -0.03834537789225578,
+          0.008951487019658089,
+          0.007423861417919397,
+          0.014276351779699326,
+          -0.013395487330853939,
+          -0.024775303900241852,
+          -0.036885205656290054,
+          -0.016665002331137657,
+          0.003930163104087114,
+          0.005725618451833725,
+          0.005801007617264986,
+          -0.008522958494722843,
+          -0.005269315093755722,
+          -0.004471775609999895,
+          0.004209897015243769,
+          0.013435166329145432,
+          -0.049106206744909286,
+          -0.019490117207169533,
+          0.006205729208886623,
+          -0.01676023006439209,
+          0.016363445669412613,
+          0.007431797217577696,
+          -0.013236773200333118,
+          0.009737122803926468,
+          -0.016117436811327934,
+          0.000434232410043478,
+          -0.00926891528069973,
+          0.001266738516278565,
+          -0.004194025415927172,
+          0.008967358618974686,
+          0.019585344940423965,
+          0.034948889166116714,
+          -0.007753193844109774,
+          0.003061202121898532,
+          -0.008848322555422783,
+          0.0007211581105366349,
+          0.01029262226074934,
+          0.006332700606435537,
+          -0.005340736359357834,
+          -0.00027403017156757414,
+          -0.02015671692788601,
+          0.015450838021934032,
+          -0.020140845328569412,
+          0.00280130747705698,
+          0.00023273965052794665,
+          -0.0029143914580345154,
+          -0.0019859126769006252,
+          -0.008102364838123322,
+          -0.004928078968077898,
+          -0.007935714907944202,
+          -0.0019630976021289825,
+          0.02042653039097786,
+          -0.02187083102762699,
+          0.00217637000605464,
+          0.0018648931290954351,
+          0.004745557904243469,
+          -0.027870232239365578,
+          -0.0051859901286661625,
+          -0.009705379605293274,
+          -0.014157315716147423,
+          -0.023204030469059944,
+          0.026219602674245834,
+          0.016228538006544113,
+          0.007566704414784908,
+          0.004678104072809219,
+          0.0022954056039452553,
+          -0.0010653698118403554,
+          -0.013657365925610065,
+          0.003325064666569233,
+          -0.001118935877457261,
+          -0.0026683842297643423,
+          -0.0033111772499978542,
+          0.024743560701608658,
+          -0.014823916368186474,
+          0.008634057827293873,
+          -0.0014790189452469349,
+          0.0073683117516338825,
+          0.005900204181671143,
+          0.0121257733553648,
+          0.012030544690787792,
+          0.019966259598731995,
+          0.012117837555706501,
+          0.014593780040740967,
+          0.06742183864116669,
+          -0.0058248150162398815,
+          -0.0032457076013088226,
+          0.0031544468365609646,
+          -0.02080744504928589,
+          -0.01902984455227852,
+          0.004638425540179014,
+          -0.019791673868894577,
+          0.029504990205168724,
+          -0.00918955821543932,
+          0.009713315404951572,
+          0.005725618451833725,
+          -0.007066754624247551,
+          0.017982330173254013,
+          -0.002755677094683051,
+          -0.002307309303432703,
+          0.007927779108285904,
+          0.01438745204359293,
+          0.034028347581624985,
+          0.014681072905659676,
+          -0.016395187005400658,
+          0.0026842555962502956,
+          -0.02252155914902687,
+          0.0028588413260877132,
+          -0.011482980102300644,
+          0.005328832659870386,
+          -0.013744658790528774,
+          0.026584645733237267,
+          0.0011318314354866743,
+          0.02661638893187046,
+          -0.016395187005400658,
+          0.0015365529106929898,
+          -0.014522358775138855,
+          0.0007335576810874045,
+          0.015593680553138256,
+          0.017982330173254013,
+          -0.0038567576557397842,
+          0.006729486398398876,
+          -0.013657365925610065,
+          0.00803094357252121,
+          0.014189058914780617,
+          0.015014373697340488,
+          0.001624837750568986,
+          -0.011340136639773846,
+          -0.011895637027919292,
+          0.02215651609003544,
+          0.0004414241702761501,
+          0.006118436343967915,
+          0.012459073215723038,
+          -0.014474744908511639,
+          0.00011543985601747409,
+          -0.005269315093755722,
+          0.01599840261042118,
+          -0.008026976138353348,
+          0.004376546945422888,
+          -0.006118436343967915,
+          -0.019379016011953354,
+          -0.005495482590049505,
+          -0.02252155914902687,
+          0.01718875952064991,
+          0.00046721522812731564,
+          0.00801110453903675,
+          -0.021839087828993797,
+          -0.01739508844912052,
+          -0.006876297295093536,
+          0.015379415825009346,
+          2.8162488888483495e-05,
+          0.010919543914496899,
+          0.012086094357073307,
+          0.01042752992361784,
+          -0.007400054484605789,
+          0.0004890384734608233,
+          0.02215651609003544,
+          -0.010316429659724236,
+          0.010395786724984646,
+          0.00739608658477664,
+          -0.005435965023934841,
+          -0.032250747084617615,
+          -0.004301157779991627,
+          -0.006820747163146734,
+          0.020236073061823845,
+          0.002410473534837365,
+          -0.01492708083242178,
+          0.009768865071237087,
+          -0.0409482903778553,
+          -0.009451436810195446,
+          -0.0027140146121382713,
+          0.025092732161283493,
+          0.011165550909936428,
+          -0.0069159758277237415,
+          0.01950598880648613,
+          0.008753093890845776,
+          0.013117737136781216,
+          -0.0013619671808555722,
+          -0.007324665319174528,
+          0.0036206701770424843,
+          -0.002962005790323019,
+          0.0028588413260877132,
+          -0.009959322400391102,
+          0.002946134191006422,
+          0.015561937354505062,
+          0.004174186382442713,
+          -0.011411558836698532,
+          -0.005983529146760702,
+          -0.015363545157015324,
+          -0.016117436811327934,
+          0.0026862395461648703,
+          0.00037248263834044337,
+          0.011919444426894188,
+          -0.007903972640633583,
+          -0.004209897015243769,
+          -0.015887301415205002,
+          0.019632959738373756,
+          0.0012320197420194745,
+          0.0009061594610102475,
+          -0.00824520830065012,
+          0.003061202121898532,
+          -0.012657465413212776,
+          -0.007273083087056875,
+          0.008745158091187477,
+          -0.00021451229986269027,
+          -0.0080864941701293,
+          0.024600718170404434,
+          -0.02445787377655506,
+          0.0049677579663693905,
+          -0.010149779729545116,
+          -0.028108302503824234,
+          -0.00940382294356823,
+          -0.03282211720943451,
+          -0.013133608736097813,
+          0.0039202431216835976,
+          -0.011863894760608673,
+          -0.007820647209882736,
+          0.01556987315416336,
+          -0.008074590004980564,
+          -0.005023307632654905,
+          0.015982531011104584,
+          0.003579007461667061,
+          0.0160380806773901,
+          -0.005761329084634781,
+          0.0027715484611690044,
+          -0.01750618778169155,
+          -0.006003368645906448,
+          -0.01463345903903246,
+          0.009824415668845177,
+          0.0016952671576291323,
+          0.013078059069812298,
+          -0.017093529924750328,
+          0.02106138877570629,
+          -0.010006937198340893,
+          0.00474159000441432,
+          -0.03507586196064949,
+          -0.01231622975319624,
+          0.017172887921333313,
+          0.016165051609277725,
+          -0.006003368645906448,
+          -0.00700326869264245,
+          -0.020759830251336098,
+          -0.010744958184659481,
+          -0.014681072905659676,
+          -0.02753693237900734,
+          0.011903572827577591,
+          0.00038339424645528197,
+          0.021188359707593918,
+          0.004142443183809519,
+          0.030457274988293648,
+          0.013046315871179104,
+          0.019839288666844368,
+          -0.0034302128478884697,
+          0.0011020724195986986,
+          -0.0038785808719694614,
+          0.015561937354505062,
+          -0.005804975517094135,
+          -0.012292423285543919,
+          -0.005741490051150322,
+          0.0231722891330719,
+          -0.009308594278991222,
+          0.002333100186660886,
+          0.005908139981329441,
+          0.024553103372454643,
+          0.0038726290222257376,
+          0.01714114472270012,
+          0.004392418544739485,
+          -0.002789404010400176,
+          0.03647254779934883,
+          -0.0027021111454814672,
+          -0.021854959428310394,
+          -0.0009011996444314718,
+          0.011705179698765278,
+          -0.021950189024209976,
+          -0.008856258355081081,
+          0.001166550093330443,
+          -0.022172387689352036,
+          -0.015942851081490517,
+          -2.3326663722400554e-05,
+          -0.00801904033869505,
+          0.027330603450536728,
+          0.0055311936885118484,
+          -0.008594379760324955,
+          -0.016522159799933434,
+          0.010705280117690563,
+          0.025997404009103775,
+          -0.0008962398278526962,
+          0.015173087827861309,
+          -0.014776301570236683,
+          -0.01761728711426258,
+          -0.0054994504898786545,
+          0.013681173324584961,
+          0.004182121716439724,
+          -0.0047336542047560215,
+          -0.05977180600166321,
+          -0.01815691590309143,
+          0.006086693610996008,
+          0.002291437704116106,
+          -0.026394188404083252,
+          -0.008388050831854343,
+          -0.017823616042733192,
+          -0.008911808021366596,
+          0.0029639897402375937,
+          0.02285485900938511,
+          -0.008538829162716866,
+          0.011689308099448681,
+          0.019744059070944786,
+          -0.01745857298374176,
+          -0.0007796840509399772,
+          0.00395992211997509,
+          -0.019379016011953354,
+          -0.005431997124105692,
+          0.0030393789056688547,
+          -0.018188659101724625,
+          -0.005614518653601408,
+          -0.0011754778679460287,
+          -0.007602415047585964,
+          0.012046415358781815,
+          -0.017601415514945984,
+          0.022394588217139244,
+          0.004194025415927172,
+          -0.011681373231112957,
+          -0.017744259908795357,
+          -0.03169524669647217,
+          -0.007741290144622326,
+          -0.002170418156310916,
+          -0.008713414892554283,
+          0.011149680241942406,
+          0.019267916679382324,
+          0.013760530389845371,
+          0.006173986475914717,
+          0.002852889709174633,
+          0.004761429037898779,
+          -2.9216451366664842e-05,
+          0.0005525241722352803,
+          0.003467907663434744,
+          -0.008530894294381142,
+          -0.0003127167874481529,
+          0.0077889044769108295,
+          -0.013506587594747543,
+          -0.021537531167268753,
+          0.009467308409512043,
+          0.03605988994240761,
+          -0.011411558836698532,
+          0.0038865164387971163,
+          -0.005618486553430557,
+          0.007566704414784908,
+          0.006630290299654007,
+          0.03012397512793541,
+          0.01655390113592148,
+          -0.022220002487301826,
+          0.00940382294356823,
+          -0.010959222912788391,
+          0.01425254438072443,
+          -0.0047495258040726185,
+          0.01961708813905716,
+          0.016395187005400658,
+          0.01586349494755268,
+          0.008181722834706306,
+          0.004813011270016432,
+          -0.0101021658629179,
+          -0.0033210967667400837,
+          0.014879466034471989,
+          -0.00872928649187088,
+          -0.011236973106861115,
+          0.0004543196992017329,
+          0.006316829472780228,
+          -0.01276856567710638,
+          -0.0160380806773901,
+          0.0034202933311462402,
+          0.009364143945276737,
+          0.03304431959986687,
+          -0.0036543968599289656,
+          0.003364743199199438,
+          -0.00654696486890316,
+          -0.013355808332562447,
+          -0.02166450209915638,
+          -0.02133120223879814,
+          0.012347972951829433,
+          -0.00739608658477664,
+          -0.0011437350185588002,
+          -0.014363644644618034,
+          -0.020997902378439903,
+          0.0035809914115816355,
+          -0.01441125851124525,
+          -0.00034569960553199053,
+          -0.002469991333782673,
+          -0.0007295898394659162,
+          -0.0028747129254043102,
+          0.0038369183894246817,
+          0.02133120223879814,
+          -0.016299959272146225,
+          0.013141544535756111,
+          0.005614518653601408,
+          0.0039698416367173195,
+          0.02542603202164173,
+          0.007455604616552591,
+          0.009118136949837208,
+          -0.016569772735238075,
+          0.025251446291804314,
+          -0.027600416913628578,
+          -0.014617587439715862,
+          -0.010030743665993214,
+          0.021188359707593918,
+          -0.0049836291000247,
+          0.011030644178390503,
+          0.016665002331137657,
+          0.005384382791817188,
+          -0.001797439530491829,
+          0.026108503341674805,
+          0.02069634571671486,
+          -0.001594086759723723,
+          0.008681672625243664,
+          -0.0005287170060910285,
+          0.02274375967681408,
+          -0.005920043680816889,
+          -0.025156216695904732,
+          0.011522658169269562,
+          0.009332400746643543,
+          0.0051701185293495655,
+          -0.0014075974468141794,
+          -0.013887501321732998,
+          -0.012213066220283508,
+          -0.009499051608145237,
+          -0.0177918728441,
+          -0.019696446135640144,
+          -0.007566704414784908,
+          0.0013074091402813792,
+          -0.01107825804501772,
+          -0.0019333385862410069,
+          -0.01783948764204979,
+          -0.02312467433512211,
+          -0.016474545001983643,
+          0.027568675577640533,
+          -0.0001265994505956769,
+          0.01592698134481907,
+          0.004400353878736496,
+          0.006217632908374071,
+          -0.019490117207169533,
+          -0.005035211332142353,
+          0.023156417533755302,
+          0.01761728711426258,
+          -0.003640509443357587,
+          -0.003360775299370289,
+          0.021886702626943588,
+          0.0017250260571017861,
+          -0.007011204492300749,
+          -0.0017418895149603486,
+          -0.0010385867208242416,
+          0.012990766204893589,
+          0.01207022275775671,
+          -0.008126172237098217,
+          -0.030790574848651886,
+          0.007836518809199333,
+          0.0007940675131976604,
+          -0.010895737446844578,
+          0.012086094357073307,
+          -0.0037416897248476744,
+          -0.013800208456814289,
+          0.005447868723422289,
+          -0.024013474583625793,
+          0.00014643874601460993,
+          -0.005812911316752434,
+          0.008181722834706306,
+          -0.03834537789225578,
+          0.003967857453972101,
+          0.014157315716147423,
+          -0.004495582543313503,
+          -0.007765097543597221,
+          -0.01128458697348833,
+          -0.021045517176389694,
+          0.002924310974776745,
+          0.009999001398682594,
+          0.009459372609853745,
+          -0.0026941753458231688,
+          0.010070422664284706,
+          -0.008840386755764484,
+          0.0012379715917631984,
+          0.006423961371183395,
+          0.012006737291812897,
+          0.012482879683375359,
+          0.020680474117398262,
+          0.0008382099331356585,
+          -0.00832456536591053,
+          0.005876397248357534,
+          -0.002870745025575161,
+          0.02080744504928589,
+          -0.01822040230035782,
+          0.00022976375475991517,
+          8.462696132482961e-05,
+          -0.003707963041961193,
+          -0.022807246074080467,
+          0.0020355109591037035,
+          5.387606506701559e-05,
+          0.010205329395830631,
+          0.014379516243934631,
+          0.005701811518520117,
+          0.002110900357365608,
+          -0.000986508559435606,
+          0.007769065443426371,
+          0.013673237524926662,
+          -0.009149880148470402,
+          -0.015307994559407234,
+          -0.02085505984723568,
+          -0.014070022851228714,
+          0.010451337322592735,
+          -0.0037000272423028946,
+          -0.0029997003730386496,
+          -0.003422277281060815,
+          0.012720951810479164,
+          0.011030644178390503,
+          -0.009165750816464424,
+          0.02236284501850605,
+          -0.016069823876023293,
+          -0.004396386444568634,
+          -0.01087986584752798,
+          0.01203848049044609,
+          0.0015434966189786792,
+          -0.01301457267254591,
+          0.006558868568390608,
+          0.008364243432879448,
+          -0.0008783844532445073,
+          -0.009983129799365997,
+          -0.0023807145189493895,
+          -0.003507586196064949,
+          0.003283402184024453,
+          0.007451636716723442,
+          -0.0007315737311728299,
+          0.002406505635008216,
+          0.017268115654587746,
+          0.003957937937229872,
+          0.005543096922338009,
+          0.013728787191212177,
+          -0.0017895037308335304,
+          -0.013657365925610065,
+          0.00436464324593544,
+          -0.028108302503824234,
+          -0.0047177826054394245,
+          0.011578208766877651,
+          -0.004630489740520716,
+          -0.000541116576641798,
+          -0.006880265194922686,
+          0.0571054071187973,
+          0.00369804329238832,
+          -0.015720652416348457,
+          0.0011873814510181546,
+          0.008507086895406246,
+          0.023410359397530556,
+          -0.0003945538483094424,
+          0.011967058293521404,
+          0.001203252817504108,
+          0.022870730608701706,
+          0.01977580226957798,
+          -0.02560061775147915,
+          -0.005570872221142054,
+          -0.003928178921341896,
+          0.00583275081589818,
+          0.0301715899258852,
+          0.02866380289196968,
+          0.0062612793408334255,
+          0.0006834634696133435,
+          0.005424061324447393,
+          0.021156616508960724,
+          0.003944050520658493,
+          -0.017760131508111954,
+          0.0026306896470487118,
+          0.0024203930515795946,
+          -0.006582675967365503,
+          0.005888300947844982,
+          -0.016887202858924866,
+          -0.029949389398097992,
+          0.018093431368470192,
+          7.718722918070853e-05,
+          -0.002864793175831437,
+          -0.003094929037615657,
+          -0.018839387223124504,
+          0.01222893688827753,
+          0.013593880459666252,
+          0.0017547850729897618,
+          -0.021410560235381126,
+          -0.002325164619833231,
+          0.006689807865768671,
+          -0.012601915746927261,
+          0.013966858386993408,
+          -0.014022408984601498,
+          -0.031869832426309586,
+          0.0034579879138618708,
+          -0.005535161588340998,
+          0.013236773200333118,
+          -0.0015147296944633126,
+          -0.015379415825009346,
+          -0.009356208145618439,
+          0.024219803512096405,
+          -0.01285585854202509,
+          0.0017041948158293962,
+          -0.005943850614130497,
+          -0.01150678750127554,
+          -0.001055450178682804,
+          0.0001287073828279972,
+          0.008292822167277336,
+          7.19174204277806e-05,
+          -0.01253049448132515,
+          0.01282411627471447,
+          0.024442002177238464,
+          0.0009131032275035977,
+          -0.010784637182950974,
+          -0.01107825804501772,
+          -0.01083225104957819,
+          0.002688223496079445,
+          -0.010491015389561653,
+          0.0017954555805772543,
+          -0.021886702626943588,
+          -0.00956253707408905,
+          0.01236384455114603,
+          -0.002618785947561264,
+          -0.0034500521142035723,
+          0.010657665319740772,
+          -0.0010023800423368812,
+          -0.0006889192736707628,
+          0.016903072595596313,
+          -0.013784337788820267,
+          0.0030413628555834293,
+          0.024267416447401047,
+          -0.0010931447613984346,
+          -0.003392518265172839,
+          -0.006400154437869787,
+          0.0050272755324840546,
+          -0.0002998212294187397,
+          -0.01655390113592148,
+          -0.006293022073805332,
+          -0.017522059381008148,
+          -0.009356208145618439,
+          -0.006848522461950779,
+          -0.018633060157299042,
+          -0.002815194893628359,
+          0.001860925229266286,
+          0.012340037152171135,
+          0.014744559302926064,
+          -0.0019144912948831916,
+          0.006126372143626213,
+          0.02053763158619404,
+          -0.01541115902364254,
+          0.014823916368186474,
+          0.00276956451125443,
+          0.007800808176398277,
+          0.005793072283267975,
+          0.017426829785108566,
+          0.009141944348812103,
+          0.0015028261113911867,
+          0.0036186862271279097,
+          -0.006967558059841394,
+          0.002551332348957658,
+          0.00646760780364275,
+          -0.02247394621372223,
+          0.006352540105581284,
+          0.00021810817997902632,
+          -0.00420196121558547,
+          -0.017744259908795357,
+          -0.004257511347532272,
+          0.002378730569034815,
+          0.007737322244793177,
+          -0.004305125679820776,
+          -0.00040571345016360283,
+          -0.02058524452149868,
+          -0.005320896860212088,
+          -0.0020394788589328527,
+          -0.026330703869462013,
+          -0.005785136483609676,
+          0.0008957438403740525,
+          0.013300258666276932,
+          -0.007661933079361916,
+          -0.005551032721996307,
+          0.022489817813038826,
+          0.011967058293521404,
+          -0.013070123270154,
+          -0.0010693375952541828,
+          -0.0029818450566381216,
+          -0.005098697263747454,
+          0.01644280180335045,
+          0.013220901601016521,
+          -0.00030130919185467064,
+          0.011935316026210785,
+          -0.008792772889137268,
+          0.00692391162738204,
+          -0.007939683273434639,
+          -0.0040611024014651775,
+          0.012586044147610664,
+          -0.011808344163000584,
+          0.00404919870197773,
+          0.0140065373852849,
+          0.012792373076081276,
+          -0.017268115654587746,
+          -0.0070270756259560585,
+          -0.01793471723794937,
+          0.0028409860096871853,
+          0.002198193222284317,
+          -0.008713414892554283,
+          0.008356308564543724,
+          0.0033151451498270035,
+          0.007661933079361916,
+          0.012657465413212776,
+          -0.007106433156877756,
+          0.023949988186359406,
+          0.007292922120541334,
+          -0.0019085395615547895,
+          0.01191150862723589,
+          -0.014673137106001377,
+          0.014728687703609467,
+          0.011490915901958942,
+          -0.0009711331804282963,
+          -0.016903072595596313,
+          -0.0016992350574582815,
+          0.0280289463698864,
+          -0.021696245297789574,
+          0.007392118684947491,
+          -0.003473859280347824,
+          -0.01756967417895794,
+          -0.013887501321732998,
+          -0.009792672470211983,
+          0.007098497357219458,
+          -0.010792572982609272,
+          0.0005416125641204417,
+          -0.019363144412636757,
+          -0.009578408673405647,
+          -0.00011240196909056976,
+          0.01745857298374176,
+          -0.001312368898652494,
+          -0.007808743976056576,
+          0.00809839740395546,
+          0.008721350692212582,
+          -0.010689408518373966,
+          -0.03012397512793541,
+          0.02650528959929943,
+          -0.0025810913648456335,
+          -0.008467407897114754,
+          -0.0013748626224696636,
+          -0.0036365415435284376,
+          0.0048606256023049355,
+          -0.018601316958665848,
+          -0.00811030063778162,
+          -0.007812711410224438,
+          -0.007979361340403557,
+          0.005558968521654606,
+          0.00436464324593544,
+          0.0019591297022998333,
+          -0.008721350692212582,
+          0.006217632908374071,
+          -0.005483579356223345,
+          -0.01622060127556324,
+          -0.01691894419491291,
+          0.010737022385001183,
+          -0.006892168894410133,
+          -0.013538329862058163,
+          -0.008050783537328243,
+          0.0038865164387971163,
+          -0.003188173519447446,
+          -0.006102565210312605,
+          0.01691894419491291,
+          -0.005332800559699535,
+          -0.016569772735238075,
+          0.001228051958605647,
+          0.01004661526530981,
+          0.017522059381008148,
+          0.016569772735238075,
+          0.03367917612195015,
+          0.0015682957600802183,
+          -0.017601415514945984,
+          0.010649729520082474,
+          -8.43944690132048e-06,
+          0.0031961093191057444,
+          0.013831951655447483,
+          -0.01849021576344967,
+          0.010697344318032265,
+          -0.019521860405802727,
+          0.010094230063259602,
+          -0.013736722990870476,
+          -0.005162182729691267,
+          -0.018998103216290474,
+          -0.01772838830947876,
+          -0.0008223385084420443,
+          -0.003146511036902666,
+          0.01355420146137476,
+          0.0005530201597139239,
+          -0.012713016010820866,
+          0.008364243432879448,
+          0.017363345250487328,
+          -0.038726288825273514,
+          -0.011133808642625809,
+          -0.004729686304926872,
+          0.0036484450101852417,
+          -0.017268115654587746,
+          0.02042653039097786,
+          0.023838888853788376,
+          -0.003993648570030928,
+          0.003088977187871933,
+          0.00404919870197773,
+          0.005471675656735897,
+          0.010562436655163765,
+          -0.01810930296778679,
+          0.005118536297231913,
+          -0.036885205656290054,
+          0.0010891768615692854,
+          0.016728486865758896,
+          0.02418806031346321,
+          -0.006281118839979172,
+          -0.01642693020403385,
+          -0.012347972951829433,
+          -0.009602215141057968,
+          -0.031012775376439095,
+          -0.006550932768732309,
+          -0.01368910912424326,
+          -0.0026326735969632864,
+          -0.007844454608857632,
+          -0.01236384455114603,
+          0.024600718170404434,
+          0.008427729830145836,
+          -0.0019283788278698921,
+          -0.004686039872467518,
+          0.003116752253845334,
+          -0.024330902844667435,
+          -0.01622060127556324,
+          -0.014204930514097214,
+          0.005801007617264986,
+          0.021426431834697723,
+          -0.0033786308486014605,
+          0.012593979947268963,
+          -0.011395687237381935,
+          0.001903579686768353,
+          0.0024719752836972475,
+          0.004182121716439724,
+          -0.013649430125951767,
+          0.032044418156147,
+          -0.0012171403504908085,
+          -0.0002213320549344644,
+          0.0037238344084471464,
+          -0.004602714907377958,
+          0.014974694699048996,
+          0.006717583164572716,
+          -0.013324066065251827,
+          -0.010657665319740772,
+          0.01741096004843712,
+          0.01549051608890295,
+          0.0019174672197550535,
+          -0.01174485869705677,
+          -0.0005376447224989533,
+          0.008126172237098217,
+          -0.006023207679390907,
+          -0.003297289600595832,
+          -0.00013267523900140077,
+          0.0005698835593648255,
+          0.02769564650952816,
+          0.005487547256052494,
+          0.005090761464089155,
+          -0.011340136639773846,
+          0.015101666562259197,
+          0.017014173790812492,
+          0.005535161588340998,
+          -0.0013084009988233447,
+          -0.020664602518081665,
+          0.004186089616268873,
+          -0.021220102906227112,
+          -0.003416325431317091,
+          -0.008951487019658089,
+          -0.0034778271801769733,
+          0.004559068474918604,
+          -0.0025394288823008537,
+          -0.016791973263025284,
+          0.00607478991150856,
+          -0.009848223067820072,
+          -4.6684326662216336e-05,
+          0.009181622415781021,
+          -0.013347873464226723,
+          0.013022508472204208,
+          0.029552603140473366,
+          0.018823517486453056,
+          0.015514323487877846,
+          -0.0025275254156440496,
+          -0.02241045981645584,
+          0.02321990206837654,
+          0.008761029690504074,
+          0.006114468444138765,
+          0.0010078358463943005,
+          -0.007392118684947491,
+          0.025505388155579567,
+          0.029822418466210365,
+          0.002785436110571027,
+          0.006741390097886324,
+          -0.006332700606435537,
+          0.013514523394405842,
+          0.00864992942661047,
+          -0.006717583164572716,
+          0.0006824714946560562,
+          0.018521958962082863,
+          -0.0066223544999957085,
+          -0.003807159373536706,
+          0.031076261773705482,
+          -0.023156417533755302,
+          -0.012419394217431545,
+          -0.013458972796797752,
+          -0.0021168519742786884,
+          0.00428528618067503,
+          0.017522059381008148,
+          -0.021839087828993797,
+          -0.008459472097456455,
+          0.013141544535756111,
+          0.008792772889137268,
+          0.00013689108891412616,
+          -0.008713414892554283,
+          0.0006621362408623099,
+          0.010713215917348862,
+          -0.006991364993155003,
+          0.0005802991800010204,
+          -0.029917646199464798,
+          -0.0027576610445976257,
+          -0.0023410359863191843,
+          0.0027775003109127283,
+          0.04339249059557915,
+          -0.002136691240593791,
+          0.00878483708947897,
+          0.003140559419989586,
+          0.008078558370471,
+          -0.007185790222138166,
+          -0.014435065910220146,
+          0.0011625823099166155,
+          0.01642693020403385,
+          -0.00584862194955349,
+          0.010435465723276138,
+          -0.0008883041446097195,
+          -0.027346475049853325,
+          0.017331602051854134,
+          0.03659952059388161,
+          0.006082725711166859,
+          -0.005428029224276543,
+          -0.005900204181671143,
+          -0.005150279030203819,
+          -0.012371780350804329,
+          -0.007328633219003677,
+          0.013506587594747543,
+          0.0012657465413212776,
+          0.027235373854637146,
+          -0.015061987563967705,
+          0.005150279030203819,
+          -0.005816879216581583,
+          -0.025632360950112343,
+          0.010824315249919891,
+          0.004309093579649925,
+          0.00013974298781249672,
+          0.01563335955142975,
+          0.020616987720131874,
+          -0.03082231804728508,
+          -0.011030644178390503,
+          -0.005364543292671442,
+          -0.021696245297789574,
+          0.016188859939575195,
+          -0.006864393595606089,
+          0.01503024436533451,
+          -0.002779484260827303,
+          0.007907940074801445,
+          -0.011236973106861115,
+          0.004293221980333328,
+          0.019712315872311592,
+          -0.004086893517524004,
+          0.018982231616973877,
+          -0.009983129799365997,
+          -0.007618286646902561,
+          0.013355808332562447,
+          -0.016633259132504463,
+          0.020775701850652695,
+          -0.005678004119545221,
+          -0.002283502137288451,
+          0.021918445825576782,
+          -0.01347484439611435,
+          0.012355908751487732,
+          0.0030314433388412,
+          -0.01309392973780632,
+          -0.00044415207230485976,
+          -0.0033389520831406116,
+          0.0013252644566819072,
+          -0.009610150940716267,
+          -0.019474245607852936,
+          0.021029645577073097,
+          -0.0180458165705204,
+          0.023807145655155182,
+          0.0015028261113911867,
+          -0.011348072439432144,
+          0.001473067095503211,
+          -0.007971425540745258,
+          -0.023521460592746735,
+          0.0015752394683659077,
+          -0.00280130747705698,
+          0.011816279962658882,
+          -0.009475244209170341,
+          0.020601116120815277,
+          0.0057335542514920235,
+          -0.016165051609277725,
+          0.00047986279241740704,
+          0.011570272967219353,
+          -0.0006963590276427567,
+          0.005158214829862118,
+          0.027711518108844757,
+          -0.001079257228411734,
+          0.0029957324732095003,
+          0.006054950878024101,
+          0.008745158091187477,
+          -0.03352046012878418,
+          -0.009673637337982655,
+          0.004047214984893799,
+          -0.0004498558701016009,
+          0.010149779729545116,
+          0.005221700761467218,
+          -0.010221200995147228,
+          0.022712016478180885,
+          -0.00013354320253711194,
+          -0.017744259908795357,
+          0.008475343696773052,
+          -0.009205429814755917,
+          -0.011602015234529972,
+          -0.009284786880016327,
+          -0.002348971785977483,
+          0.021029645577073097,
+          -0.005820847116410732,
+          0.010030743665993214,
+          0.0017418895149603486,
+          0.003398470114916563,
+          -0.002497766399756074,
+          -0.017966458573937416,
+          0.002624737797304988,
+          -0.004618586506694555,
+          0.020188460126519203,
+          0.015919044613838196,
+          0.015823816880583763,
+          -0.00911020115017891,
+          -0.015593680553138256,
+          0.004725718405097723,
+          0.02796545997262001,
+          -0.009895836934447289,
+          0.0071699186228215694,
+          -0.004011503886431456,
+          0.0013937100302428007,
+          -0.006539029534906149,
+          -0.013133608736097813,
+          -0.015228637494146824,
+          -0.03482192009687424,
+          -0.0019690494518727064,
+          0.002146610990166664,
+          -0.00044415207230485976,
+          -0.012919344007968903,
+          -0.03685346245765686,
+          -0.007911908440291882,
+          -0.02052175998687744,
+          -0.019585344940423965,
+          -0.02807656116783619,
+          -0.02445787377655506,
+          -0.004880465101450682,
+          -0.014784237369894981,
+          0.011244908906519413,
+          -0.0034460844472050667,
+          -0.009792672470211983,
+          0.0037754166405647993,
+          0.023045316338539124,
+          0.00047267103218473494,
+          0.02042653039097786,
+          -0.013760530389845371,
+          0.008880065754055977,
+          -0.007530993781983852,
+          0.007308793719857931,
+          -0.006360475905239582,
+          -0.01649041660130024,
+          0.00654696486890316,
+          0.008070622570812702,
+          0.021140744909644127,
+          -0.01129252277314663,
+          0.0018837404204532504,
+          -0.006285086274147034,
+          0.0013431197730824351,
+          -0.00023447559215128422,
+          0.000928974652197212,
+          0.006709647364914417,
+          0.020188460126519203,
+          -0.02625134587287903,
+          -0.018775902688503265,
+          0.008126172237098217,
+          0.009760930202901363,
+          0.021807344630360603,
+          -0.021632758900523186,
+          0.013284387066960335,
+          -2.7294519895804115e-05,
+          -0.00575339375063777,
+          0.013220901601016521,
+          0.012078158557415009,
+          -0.005277250427752733,
+          0.0024997503496706486,
+          0.02301357500255108,
+          0.012467009015381336,
+          -0.009673637337982655,
+          -0.020601116120815277,
+          0.0088880006223917,
+          0.0006874313112348318,
+          -0.007733354344964027,
+          0.004868561401963234,
+          0.03361568972468376,
+          0.023156417533755302,
+          0.01982341706752777,
+          -0.00039802573155611753,
+          0.031552404165267944,
+          0.0011506787268444896,
+          -0.016744358465075493,
+          0.010705280117690563,
+          -0.0036087664775550365,
+          0.004928078968077898,
+          -0.016315830871462822,
+          -0.04532880708575249,
+          0.004880465101450682,
+          0.002618785947561264,
+          0.009999001398682594,
+          -0.027457574382424355,
+          0.01891874521970749,
+          0.0006244415999390185,
+          -0.0024660234339535236,
+          -0.014958823099732399,
+          0.008126172237098217,
+          -0.010387850925326347,
+          0.00818965770304203,
+          -0.004559068474918604,
+          -0.01463345903903246,
+          0.021474044770002365,
+          0.00630492577329278,
+          -0.004531293176114559,
+          0.011340136639773846,
+          0.022537430748343468,
+          0.01847434602677822,
+          0.012078158557415009,
+          0.010467207990586758,
+          0.04247194901108742,
+          0.011784536764025688,
+          0.0025632360484451056,
+          0.006626322399824858,
+          -0.00334688788279891,
+          -0.0032774503342807293,
+          -0.004908239934593439,
+          0.000227779833949171,
+          -0.004071021918207407,
+          -0.018299760296940804,
+          0.016855459660291672,
+          0.010118036530911922,
+          -0.010729086585342884,
+          0.01714114472270012,
+          -0.01619679480791092,
+          0.012030544690787792,
+          0.03485365957021713,
+          0.00010012640996137634,
+          0.017696645110845566,
+          -0.01901397295296192,
+          -0.009760930202901363,
+          -0.0020573341753333807,
+          -0.011546465568244457,
+          0.0036206701770424843,
+          -0.015823816880583763,
+          -0.007455604616552591,
+          -0.008213465102016926,
+          0.00770557951182127,
+          0.01595078781247139,
+          0.015347673557698727,
+          0.013530394062399864,
+          0.01338755153119564,
+          0.0056065828539431095,
+          -0.009681572206318378,
+          -0.001512745744548738,
+          -0.015379415825009346,
+          -0.012522558681666851,
+          -0.01606188714504242,
+          0.004943950567394495,
+          -0.0025394288823008537,
+          -0.017966458573937416,
+          0.0269496887922287,
+          -0.013054251670837402,
+          0.005487547256052494,
+          0.009959322400391102,
+          0.01876003108918667,
+          -0.0005031739710830152,
+          0.00045853553456254303,
+          -0.005035211332142353,
+          0.0008818563655950129,
+          0.005150279030203819,
+          -0.001076281419955194,
+          0.027394089847803116,
+          -0.004225768148899078,
+          0.004277350381016731,
+          -0.0017498251982033253,
+          -0.014974694699048996,
+          -0.0007028068066574633,
+          0.00840392243117094,
+          -0.020870931446552277,
+          -0.02263266034424305,
+          -0.018283888697624207,
+          -0.005253443494439125,
+          -0.011157616041600704,
+          0.0067334542982280254,
+          -0.011300458572804928,
+          -0.008880065754055977,
+          0.023838888853788376,
+          0.00661441870033741,
+          0.014744559302926064,
+          0.0058446540497243404,
+          -0.0009671652805991471,
+          0.017712516710162163,
+          -0.02306118793785572,
+          -0.014379516243934631,
+          0.0024263449013233185,
+          -0.010744958184659481,
+          0.032568175345659256,
+          0.00097311707213521,
+          0.01301457267254591,
+          -0.010983030311763287,
+          0.010435465723276138,
+          -0.014117637649178505,
+          0.02418806031346321,
+          0.012705080211162567,
+          0.00584862194955349,
+          0.004186089616268873,
+          -0.003910323604941368,
+          -0.011308394372463226,
+          0.03821840509772301,
+          0.0002916375233326107,
+          -0.007225468754768372,
+          0.02004561759531498,
+          0.00832456536591053,
+          0.008721350692212582,
+          -0.008657865226268768,
+          0.012022608891129494,
+          0.016665002331137657,
+          0.0002353435556869954,
+          0.006741390097886324,
+          -0.01826801709830761,
+          -0.0040452308021485806,
+          0.023204030469059944,
+          0.025092732161283493,
+          0.011467108502984047,
+          -0.003124687820672989,
+          -0.002600930631160736,
+          0.012720951810479164,
+          -0.0017408975400030613,
+          0.019410759210586548,
+          0.009300658479332924,
+          0.03863106295466423,
+          0.019601216539740562,
+          -0.014046216383576393,
+          -0.0013530394062399864,
+          0.005110600497573614,
+          0.05183609202504158,
+          -0.010657665319740772,
+          -0.007161982823163271,
+          -0.008269015699625015,
+          -0.014752495102584362,
+          0.015157216228544712,
+          -0.010173587128520012,
+          0.008082525804638863,
+          -0.008657865226268768,
+          0.010863994248211384,
+          0.017823616042733192,
+          0.007892068475484848,
+          -0.009792672470211983,
+          -0.007376247551292181,
+          -0.011117937043309212,
+          0.012562237679958344,
+          0.001430412638001144,
+          -0.00338855036534369,
+          0.011855958960950375,
+          0.01026087999343872,
+          -0.016014274209737778,
+          -0.007677804678678513,
+          0.017855359241366386,
+          -0.022346973419189453,
+          0.0013738706475123763,
+          0.007284986786544323,
+          0.0036008309107273817,
+          -0.00108818500302732,
+          -6.959250185900601e-06,
+          0.00910226535052061,
+          0.028600318357348442,
+          0.0016545966500416398,
+          0.02942563220858574,
+          0.01799820177257061,
+          0.00014445481065195054,
+          -0.024965759366750717,
+          0.005789104383438826,
+          0.01739508844912052,
+          0.004412257578223944,
+          0.024695945903658867,
+          -0.005420093424618244,
+          -0.012459073215723038,
+          -0.0191250741481781,
+          -0.0088880006223917,
+          0.012094030156731606,
+          -0.018394988030195236,
+          0.013197094202041626,
+          -0.005189957562834024,
+          0.007915875874459743,
+          0.005777200683951378,
+          0.015093730762600899,
+          -0.010530694387853146,
+          0.007685740012675524,
+          0.0021902574226260185,
+          -0.011086193844676018,
+          -0.011784536764025688,
+          -0.00032883620588108897,
+          -0.007384182885289192,
+          0.003209996735677123,
+          -0.0035849593114107847,
+          0.011649630032479763,
+          0.023727789521217346,
+          -0.004773332737386227,
+          -0.006947718560695648,
+          0.009991065599024296,
+          0.004654297139495611,
+          -0.011562337167561054,
+          0.02193431742489338,
+          0.009705379605293274,
+          0.004110700450837612,
+          0.007856357842683792,
+          -0.01042752992361784,
+          -0.016069823876023293,
+          -0.023045316338539124,
+          -0.007181822322309017,
+          -0.0231722891330719,
+          0.014974694699048996,
+          -0.006658065132796764,
+          -0.015800008550286293,
+          0.01858544535934925,
+          0.002934230724349618,
+          0.0018480296712368727,
+          -0.00848327949643135,
+          -0.026330703869462013,
+          -0.022331101819872856,
+          0.01026087999343872,
+          -0.00654696486890316,
+          0.005201861262321472,
+          0.02706078812479973,
+          0.030155718326568604,
+          -0.007019140291959047,
+          0.004261479247361422,
+          0.0034500521142035723,
+          0.003930163104087114,
+          -0.0002421633107587695,
+          -0.024981630966067314,
+          -0.0064041223376989365,
+          0.007947619073092937,
+          0.006789004430174828,
+          -0.00544390082359314,
+          -0.00926891528069973,
+          0.011943251825869083,
+          -0.019299659878015518,
+          -0.005920043680816889,
+          -0.009173686616122723,
+          -0.002819162793457508,
+          -0.00032015651231631637,
+          0.017982330173254013,
+          0.015268316492438316,
+          -0.005551032721996307,
+          -0.0025870432145893574,
+          -0.023775402456521988,
+          -0.012181323021650314,
+          0.01939488761126995,
+          0.004884432535618544,
+          0.006574740167707205,
+          -0.01471281610429287,
+          -0.0012141644256189466,
+          -0.0033389520831406116,
+          -0.012498751282691956,
+          0.015173087827861309,
+          0.06513635069131851,
+          -0.016006337478756905,
+          0.02483878843486309,
+          -0.005007436498999596,
+          0.004701911471784115,
+          -0.003416325431317091,
+          -0.021426431834697723,
+          -0.011951187625527382,
+          -0.0060271755792200565,
+          0.0017756163142621517,
+          -0.0188552588224411,
+          -0.0023965861182659864,
+          -0.0006715598865412176,
+          0.002325164619833231,
+          -0.007019140291959047,
+          0.0011953171342611313,
+          -0.02085505984723568,
+          0.016887202858924866,
+          0.006431897170841694,
+          -0.003862709505483508,
+          -0.006447768770158291,
+          -0.018823517486453056,
+          0.00022120805806480348,
+          -0.0016357493586838245,
+          -2.8115991881350055e-05,
+          0.0016784038161858916,
+          -0.025902174413204193,
+          -0.010657665319740772,
+          -0.00964189413934946,
+          -0.015069923363626003,
+          0.0002531989011913538,
+          -0.006380314938724041,
+          0.01463345903903246,
+          -0.0220930315554142,
+          -0.004872529301792383,
+          0.01822040230035782,
+          0.028362246230244637,
+          0.020728088915348053,
+          -0.011213165707886219,
+          0.001291537657380104,
+          -0.02160101756453514,
+          0.02533080242574215,
+          0.01880764588713646,
+          -0.017442701384425163,
+          0.005102664697915316,
+          0.008768965490162373,
+          0.009951386600732803,
+          -0.0019998003263026476,
+          0.030139846727252007,
+          -0.00038463418604806066,
+          -0.015427030622959137,
+          0.005090761464089155,
+          0.018934616819024086,
+          0.002618785947561264,
+          0.002612834330648184,
+          -0.006209697108715773,
+          0.021585145965218544,
+          -0.005582775454968214,
+          0.009633958339691162,
+          -0.044852662831544876,
+          -0.01815691590309143,
+          -0.005519289989024401,
+          -0.004213864915072918,
+          0.007812711410224438,
+          0.004011503886431456,
+          0.008673736825585365,
+          -0.018093431368470192,
+          -0.0021208198741078377,
+          0.025283189490437508,
+          -0.026965560391545296,
+          -0.02823527529835701,
+          -0.005225668661296368,
+          -0.02441026084125042,
+          -0.0007444692892022431,
+          0.0011546466266736388,
+          -0.007959522306919098,
+          -0.026473546400666237,
+          0.008284886367619038,
+          -0.006122404243797064,
+          -0.027679774910211563,
+          0.01768077351152897,
+          -0.005681972019374371,
+          0.000294117460725829,
+          -0.010300558060407639,
+          -0.003963889554142952,
+          -0.005356607958674431,
+          0.016569772735238075,
+          -0.01330819446593523,
+          0.008062686771154404,
+          0.0010266831377521157,
+          -0.0032357878517359495,
+          0.0016873314743861556,
+          0.016093630343675613,
+          0.002900503808632493,
+          0.000488046498503536,
+          -0.006039079278707504,
+          -0.003088977187871933,
+          -0.017871230840682983,
+          0.000921534956432879,
+          0.01891874521970749,
+          0.0036325736436992884,
+          0.02672748826444149,
+          -0.03469494730234146,
+          -0.01642693020403385,
+          -0.005725618451833725,
+          -0.01725224405527115,
+          0.011403623037040234,
+          -0.0036821719259023666,
+          -0.008499151095747948,
+          -0.0019075475865975022,
+          -0.02548951655626297,
+          0.03304431959986687,
+          -0.01939488761126995,
+          -0.00506695406511426,
+          0.004428129177540541,
+          -0.019839288666844368,
+          -0.00044935988262295723,
+          0.008157915435731411,
+          0.01998213119804859,
+          0.010308493860065937,
+          -0.015617487952113152,
+          0.009070522151887417,
+          -0.008538829162716866,
+          -0.008054750971496105,
+          0.0041543468832969666,
+          0.008126172237098217,
+          -0.009681572206318378,
+          -0.0029362146742641926,
+          0.00816585123538971,
+          -0.0030731058213859797,
+          -0.014284287579357624,
+          -0.00878483708947897,
+          0.027219504117965698,
+          -0.008515022695064545,
+          0.00575339375063777,
+          -0.0026386252138763666,
+          -0.001564327860251069,
+          -0.0020394788589328527,
+          0.00064130499958992,
+          0.0062612793408334255,
+          -0.006253343541175127,
+          0.05850209295749664,
+          -0.0003680188092403114,
+          -0.00022405995696317405,
+          -0.01869654469192028,
+          -0.011855958960950375,
+          0.005035211332142353,
+          0.006880265194922686,
+          -0.02220413088798523,
+          -0.006959622260183096,
+          0.025092732161283493,
+          -0.0020533662755042315,
+          0.010808444581925869,
+          0.005408189725130796,
+          -0.001736929640173912,
+          0.0328538604080677,
+          0.006852489896118641,
+          0.0280130747705698,
+          -0.010760829783976078,
+          0.004015471786260605,
+          0.0001645420998102054,
+          0.00755876861512661,
+          0.0069516864605247974,
+          -0.009118136949837208,
+          -0.005003468599170446,
+          0.0040611024014651775,
+          0.014300159178674221,
+          -0.0004451440181583166,
+          -0.008292822167277336,
+          0.0030135877896100283,
+          -0.012109901756048203,
+          -0.0017528011230751872,
+          -0.0008218425209634006,
+          -0.0140065373852849,
+          -0.00576529698446393,
+          0.009173686616122723,
+          0.015458772890269756,
+          0.007324665319174528,
+          0.0010683456202968955,
+          0.004400353878736496,
+          0.004999500699341297,
+          0.005269315093755722,
+          -0.012094030156731606,
+          0.009570472873747349,
+          -0.02667987532913685,
+          -0.009062586352229118,
+          0.01214957982301712,
+          -0.0011348072439432144,
+          -0.00886419415473938,
+          0.023505588993430138,
+          0.0046265218406915665,
+          -0.012332101352512836,
+          -0.011022708378732204,
+          0.006796940229833126,
+          -0.0023172288201749325,
+          0.008991165086627007,
+          -0.007860326208174229,
+          0.0027636128943413496,
+          -0.0007955554756335914,
+          0.01622060127556324,
+          0.01653803139925003,
+          -0.004253543447703123,
+          0.012117837555706501,
+          -0.0004079453647136688
+        ],
+        "title": "Balancing Expert Utilization",
+        "keyphrases": [
+          "gating network",
+          "expert utilization",
+          "soft constraint",
+          "importance of an expert",
+          "balanced loads"
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "87b82083-05f3-4d69-9ea0-97d4f551aa37",
+      "properties": {
+        "page_content": "5 Experiments 5.1 1 Billion Word Language Modeling Benchmark\n\nDataset: This dataset, introduced by (Chelba et al., 2013) consists of shuffled unique sentences from news articles, totaling approximately 829 million words, with a vocabulary of 793,471 words.\n\nPrevious State-of-the-Art: The best previously published results (Jozefowicz et al., 2016) use models consisting of one or more stacked Long Short-Term Memory (LSTM) layers (Hochreiter & Schmidhuber, 1997; Gers et al., 2000). The number of parameters in the LSTM layers of these models vary from 2 million to 151 million. Quality increases greatly with parameter count, as do computational costs. Results for these models form the top line of Figure 2-right. MoE Models: Our models consist of two stacked LSTM layers with a MoE layer between them (see Figure 1). We vary the sizes of the layers and the number of experts. For full details on model architecture, training regimen, additional baselines and results, see Appendix C.\n\nLow Computation, Varied Capacity: To investigate the effects of adding capacity, we trained\n\na series of MoE models all with roughly equal computational costs: about 8 million multiply-andadds per training example per timestep in the forwards pass, excluding the softmax layer. We call this metric (ops/timestep). We trained models with flat MoEs containing 4, 32, and 256 experts, and models with hierarchical MoEs containing 256, 1024, and 4096 experts. Each expert had about 1 million parameters. For all the MoE layers, 4 experts were active per input.\n\nThe results of these models are shown in Figure 2-left. The model with 4 always-active experts performed (unsurprisingly) similarly to the computationally-matched baseline models, while the largest of the models (4096 experts) achieved an impressive 24% lower perplexity on the test set. Figure 2: Model comparison on 1-Billion-Word Language-Modeling Benchmark. On the left, we plot test perplexity as a function of model capacity for models with similar computational budgets of approximately 8-million-ops-per-timestep. On the right, we plot test perplexity as a function of computational budget. The top line represents the LSTM models from (Jozefowicz et al., 2016).\n\nThe bottom line represents 4-billion parameter MoE models with different computational budgets.\n\nVaried Computation, High Capacity: In addition to the largest model from the previous section, we trained two more MoE models with similarly high capacity (4 billion parameters), but higher computation budgets. These models had larger LSTMs, and fewer but larger and experts. Details\n\nTest Test #Parameters ops/timestep Training TFLOPS Perplexity Perplexity excluding embedding Time /GPU 10 epochs 100 epochs and softmax layers 10 epochs Best Published Results 34.7 30.6 151 million 151 million 59 hours, 32 k40s 1.09 Low-Budget MoE Model 34.1 4303 million 8.9 million 15 hours, 16 k40s 0.74 Medium-Budget MoE Model 31.3 4313 million 33.8 million 17 hours, 32 k40s 1.22 High-Budget MoE Model 28.0 4371 million 142.7 million 47 hours, 32 k40s 1.56\n\nTable 1: Summary of high-capacity MoE-augmented models with varying computational budgets, vs. best previously published results (Jozefowicz et al., 2016). Details in Appendix C. can be found in Appendix C.2. Results of these three models form the bottom line of Figure 2-right.\n\nTable 1 compares the results of these models to the best previously-published result on this dataset .\n\nEven the fastest of these models beats the best published result (when controlling for the number of training epochs), despite requiring only 6% of the computation.\n\nComputational Efficiency: We trained our models using TensorFlow (Abadi et al., 2016) on clusters containing 16-32 Tesla K40 GPUs. For each of our models, we determine computational efficiency in TFLOPS/GPU by dividing the number of floating point operations required to process one training batch by the observed step time and the number of GPUs in the cluster. The operation counts used here are higher than the ones we report in our ops/timestep numbers in that we include the backwards pass, we include the importance-sampling-based training of the softmax layer, and we count a multiply-and-add as two separate operations. For all of our MoE models, the floating point operations involved in the experts represent between 37% and 46% of the total.\n\nFor our baseline models wtih no MoE, observed computational efficiency ranged from 1.07-1.29 TFLOPS/GPU. For our low-computation MoE models, computation efficiency ranged from 0.740.90 TFLOPS/GPU, except for the 4-expert model which did not make full use of the available parallelism. Our highest-computation MoE model was more efficient at 1.56 TFLOPS/GPU, likely due to the larger matrices. These numbers represent a significant fraction of the theoretical maximum of 4.29 TFLOPS/GPU claimed by NVIDIA. Detailed results are in Appendix C, Table 7.\n\n5.2 100 BILLION WORD GOOGLE NEWS CORPUS\n\nOn the 1-billion-word corpus, adding additional capacity seems to produce diminishing returns as the number of parameters in the MoE layer exceeds 1 billion, as can be seen in Figure 2-left. We hypothesized that for a larger training set, even higher capacities would produce significant quality improvements. We constructed a similar training set consisting of shuffled unique sentences from Google's internal news corpus, totalling roughly 100 billion words. Similarly to the previous section, we tested a series of models with similar computational costs of about 8 million ops/timestep. In addition to a baseline LSTM model, we trained models augmented with MoE layers containing 32, 256, 1024, 4096, 16384, 65536, and 131072 experts. This corresponds to up to 137 billion parameters in the MoE layer. Details on architecture, training, and results are given in Appendix D.\n\nResults: Figure 3 shows test perplexity as a function of capacity after training on 10 billion words (top line) and 100 billion words (bottom line). When training over the full 100 billion words, test perplexity improves significantly up to 65536 experts (68 billion parameters), dropping 39% lower than the computationally matched baseline, but degrades at 131072 experts, possibly a result of too much sparsity. The widening gap between the two lines demonstrates (unsurprisingly) that increased model capacity helps more on larger training sets. Even at 65536 experts (99.994% layer sparsity), computational efficiency for the model stays at a respectable 0.72 TFLOPS/GPU.\n\n5.3 Machine Translation (Single Language Pair)\n\nModel Architecture: Our model was a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decreased the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We inserted MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). Each MoE layer contained up to 2048 experts each with about two million parameters, adding a total of about 8 billion parameters to the models. Further details on model architecture, testing procedure and results can be found in Appendix E.\n\nDatasets: We benchmarked our method on the WMT'14 En→Fr and En→De corpora, whose training sets have 36M sentence pairs and 5M sentence pairs, respectively. The experimental protocols were also similar to those in (Wu et al., 2016): newstest2014 was used as the test set to compare against previous work (Luong et al., 2015a; Zhou et al., 2016; Wu et al., 2016), while the combination of newstest2012 and newstest2013 was used as the development set. We also tested the same model on a Google's Production English to French data.\n\nTable 2: Results on WMT'14 En→ Fr newstest2014 (bold values represent best results). Model Test Test ops/timenstep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.69 40.35 85M 8.7B 3 days/64 k40s MoE with 2048 Experts (longer training) 2.63 40.56 85M 8.7B 6 days/64 k40s GNMT (Wu et al., 2016) 2.79 39.22 214M 278M 6 days/96 k80s GNMT+RL (Wu et al., 2016) 2.96 39.92 214M 278M 6 days/96 k80s PBMT (Durrani et al., 2014) 37.0 LSTM (6-layer) (Luong et al., 2015b) 31.5 LSTM (6-layer+PosUnk) (Luong et al., 2015b) 33.1 DeepAtt (Zhou et al., 2016) 37.7 DeepAtt+PosUnk (Zhou et al., 2016) 39.2\n\nTable 3: Results on WMT'14 En → De newstest2014 (bold values represent best results). Model Test Test ops/timestep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 4.64 26.03 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 5.25 24.91 214M 278M 1 day/96 k80s GNMT +RL (Wu et al., 2016) 8.08 24.66 214M 278M 1 day/96 k80s PBMT (Durrani et al., 2014) 20.7 DeepAtt (Zhou et al., 2016) 20.6\n\nTable 4: Results on the Google Production En→ Fr dataset (bold values represent best results). Model Eval Eval Test Test ops/timestep Total Training Perplexity BLEU Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.60 37.27 2.69 36.57 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 2.78 35.80 2.87 35.56 214M 278M 6 days/96 k80s\n\nResults: Tables 2, 3, and 4 show the results of our largest models, compared with published results. Our approach achieved BLEU scores of 40.56 and 26.03 on the WMT'14 En→Fr and En→De benchmarks. As our models did not use RL refinement, these results constitute significant gains of 1.34 and 1.12 BLEU score on top of the strong baselines in (Wu et al., 2016). The perplexity scores are also better.2 On the Google Production dataset, our model achieved 1.01 higher test BLEU score even after training for only one sixth of the time.\n\n5.4 Multilingual Machine Translation\n\nDataset: (Johnson et al., 2016) train a single GNMT (Wu et al., 2016) model on a very large combined dataset of twelve language pairs. Results are somewhat worse than those for 12 separately trained single-pair GNMT models. This is not surprising, given that the twelve models have 12 times the capacity and twelve times the aggregate training of the one model. We repeat this experiment with a single MoE-augmented model. See Appendix E for details on model architecture.\n\nWe train our model on the same dataset as (Johnson et al., 2016) and process the same number of training examples (about 3 billion sentence pairs). Our training time was shorter due to the lower computational budget of our model.\n\nResults: Results for the single-pair GNMT models, the multilingual GNMT model and the multilingual MoE model are given in Table 5. The MoE model achieves 19% lower perplexity on the dev set than the multilingual GNMT model. On BLEU score, the MoE model significantly beats the multilingual GNMT model on 11 of the 12 language pairs (by as much as 5.84 points), and even beats the monolingual GNMT models on 8 of 12 language pairs. The poor performance on English → Korean seems to be a result of severe overtraining, as for the rarer language pairs a small number of real examples were highly oversampled in the training corpus.\n\nTable 5: Multilingual Machine Translation (bold values represent best results). GNMT-Mono GNMT-Multi MoE-Multi MoE-Multi vs. GNMT-Multi Parameters 278M / model 278M 8.7B ops/timestep 212M 212M 102M training time, hardware various 21 days, 96 k20s 12 days, 64 k40s Perplexity (dev) 4.14 3.35 -19% French → English Test BLEU 36.47 34.40 37.46 +3.06 German → English Test BLEU 31.77 31.17 34.80 +3.63 Japanese → English Test BLEU 23.41 21.62 25.91 +4.29 Korean → English Test BLEU 25.42 22.87 28.71 +5.84 Portuguese → English Test BLEU 44.40 42.53 46.13 +3.60 Spanish → English Test BLEU 38.00 36.04 39.39 +3.35 English → French Test BLEU 35.37 34.00 36.59 +2.59 English → German Test BLEU 26.43 23.15 24.53 +1.38 English → Japanese Test BLEU 23.66 21.10 22.78 +1.68 English → Korean Test BLEU 19.75 18.41 16.62 -1.79 English → Portuguese Test BLEU 38.40 37.35 37.90 +0.55 English → Spanish Test BLEU 34.50 34.25 36.21 +1.96\n\n",
+        "embedding": [
+          -0.009427943266928196,
+          0.0009052514214999974,
+          -0.031882621347904205,
+          0.02155720256268978,
+          -0.0010102161904796958,
+          0.017834007740020752,
+          -0.019673390313982964,
+          0.04869475215673447,
+          -0.022961175069212914,
+          0.03497491776943207,
+          0.036041226238012314,
+          -0.002160385251045227,
+          0.013355513103306293,
+          -0.01921132393181324,
+          -0.013746492564678192,
+          0.007433058228343725,
+          -0.00785069540143013,
+          0.004056415054947138,
+          0.003800945123657584,
+          -0.0150793781504035,
+          -0.03134946897625923,
+          -0.014439593069255352,
+          -0.04698865860700607,
+          -0.015088263899087906,
+          -0.016625525429844856,
+          0.005407072138041258,
+          0.02432960458099842,
+          -0.0009768940508365631,
+          -0.040199827402830124,
+          0.01593242585659027,
+          0.004467388149350882,
+          0.0005736961611546576,
+          0.009525688365101814,
+          -0.008481594733893871,
+          -0.0005428731674328446,
+          -0.017078706994652748,
+          0.07435724139213562,
+          -0.004718414973467588,
+          -0.03138501197099686,
+          0.03760514408349991,
+          0.028381574898958206,
+          -0.006029085721820593,
+          -0.0007786272908560932,
+          0.0035210391506552696,
+          0.0162967462092638,
+          0.003805388230830431,
+          -0.01993996649980545,
+          0.007339756470173597,
+          -0.01571027748286724,
+          -0.037356339395046234,
+          -0.006606669165194035,
+          -0.03152718394994736,
+          0.0029923280235379934,
+          -0.02411634288728237,
+          -0.0463133305311203,
+          0.014768371358513832,
+          0.028292717412114143,
+          0.02548477053642273,
+          0.06902569532394409,
+          0.034797199070453644,
+          0.01878480054438114,
+          0.0006881021545268595,
+          -0.024222973734140396,
+          -0.02600015327334404,
+          0.03351762890815735,
+          -0.012244774959981441,
+          -0.02527150884270668,
+          0.005011649336665869,
+          0.016243431717157364,
+          -0.049334537237882614,
+          -0.03643220290541649,
+          0.037498511373996735,
+          0.009730064310133457,
+          -0.005127166397869587,
+          0.015026062726974487,
+          -0.022961175069212914,
+          0.04986768960952759,
+          0.045104846358299255,
+          -0.012777929194271564,
+          -0.0024103012401610613,
+          0.013133365660905838,
+          0.017505230382084846,
+          -0.008437165059149265,
+          -0.004611784126609564,
+          0.0228723157197237,
+          -0.022374704480171204,
+          -0.014164130203425884,
+          -0.07983095943927765,
+          0.0133999427780509,
+          -0.028097227215766907,
+          -0.03525926545262337,
+          0.015585875138640404,
+          0.015976853668689728,
+          -0.0007325316546484828,
+          -0.002772401785477996,
+          0.031171750277280807,
+          -0.024080798029899597,
+          -0.003567690262570977,
+          0.001345103606581688,
+          0.04762844368815422,
+          -0.01612791419029236,
+          2.6570933187031187e-05,
+          0.04848149046301842,
+          0.01478614378720522,
+          0.027048690244555473,
+          -0.012058171443641186,
+          -0.032824527472257614,
+          0.006117944605648518,
+          0.023796450346708298,
+          0.12625092267990112,
+          0.007517474237829447,
+          -0.004080851096659899,
+          -0.03270012512803078,
+          -0.01029876247048378,
+          -0.03732079640030861,
+          0.002994549460709095,
+          -0.030158756300807,
+          -0.02031317539513111,
+          0.021361712366342545,
+          -0.017620746046304703,
+          0.03206034004688263,
+          0.02752852812409401,
+          0.019246866926550865,
+          -0.020757470279932022,
+          -0.03646774962544441,
+          0.04862366244196892,
+          -0.004369643051177263,
+          0.012031513266265392,
+          0.0041452739387750626,
+          -0.024436235427856445,
+          -0.01317779440432787,
+          0.014368506148457527,
+          -0.004580683074891567,
+          -0.02411634288728237,
+          0.03417518362402916,
+          -0.020064370706677437,
+          0.026195643469691277,
+          0.04286559671163559,
+          -0.01928240992128849,
+          -0.015434814617037773,
+          -0.002169271232560277,
+          -0.013426600024104118,
+          0.0041008442640304565,
+          -0.018535993993282318,
+          -0.05061410740017891,
+          -0.006962105166167021,
+          -0.007939554750919342,
+          0.0924844816327095,
+          -0.012831244617700577,
+          0.023245522752404213,
+          -0.02607124112546444,
+          3.4328742913203314e-05,
+          -0.015274868346750736,
+          0.0298033207654953,
+          -0.013568774797022343,
+          -0.0069754342548549175,
+          -0.01673215627670288,
+          0.02038426324725151,
+          -0.04506930336356163,
+          0.015665847808122635,
+          -0.01008550077676773,
+          -0.010938547551631927,
+          -0.008441608399152756,
+          0.02219698764383793,
+          0.0029367911629378796,
+          0.035170406103134155,
+          0.05530586466193199,
+          -0.003258905140683055,
+          -0.06625329703092575,
+          0.04009319469332695,
+          0.004449616186320782,
+          0.009005863219499588,
+          -0.00478505901992321,
+          0.001390643883496523,
+          0.002912354888394475,
+          0.027155321091413498,
+          -0.004562911577522755,
+          0.016341175884008408,
+          -0.046846482902765274,
+          0.04155048355460167,
+          0.001919355127029121,
+          0.02104181982576847,
+          0.03643220290541649,
+          0.037143077701330185,
+          0.017149792984128,
+          0.006753286812454462,
+          0.03156273066997528,
+          0.02264128252863884,
+          0.006522253155708313,
+          0.015203780494630337,
+          0.006531138904392719,
+          0.011631648056209087,
+          0.0467398539185524,
+          0.004736186470836401,
+          0.0028012811671942472,
+          -0.008152816444635391,
+          -0.014608425088226795,
+          0.060957297682762146,
+          -0.0039808847941458225,
+          0.0034966031089425087,
+          0.011924882419407368,
+          -0.04453614726662636,
+          0.006513367407023907,
+          -0.010334305465221405,
+          0.02148611471056938,
+          0.017647404223680496,
+          0.015585875138640404,
+          0.032895613461732864,
+          0.021237310022115707,
+          -0.01701650582253933,
+          -0.030798541381955147,
+          -0.02534259669482708,
+          -0.02900358848273754,
+          -0.018011726438999176,
+          -0.004891689866781235,
+          0.023831993341445923,
+          -0.008308319374918938,
+          0.020490894094109535,
+          -0.008143930695950985,
+          -0.05004540830850601,
+          0.010680856183171272,
+          0.024347376078367233,
+          0.0011462816037237644,
+          -0.03884917125105858,
+          -0.03179376199841499,
+          -0.01540815643966198,
+          -0.025360368192195892,
+          -0.009223568253219128,
+          0.029110219329595566,
+          -0.01116958074271679,
+          0.02555585838854313,
+          0.02571580372750759,
+          -0.02038426324725151,
+          0.012555781751871109,
+          0.04592235013842583,
+          -0.016270089894533157,
+          0.0072997696697711945,
+          -0.00013884223881177604,
+          -0.025147106498479843,
+          0.009321312420070171,
+          -0.0197977926582098,
+          -0.019673390313982964,
+          -0.0338730625808239,
+          -0.017140908166766167,
+          -0.016909874975681305,
+          -0.02219698764383793,
+          -0.012466922402381897,
+          0.010494251735508442,
+          0.013702062889933586,
+          0.022694597020745277,
+          0.012031513266265392,
+          -0.03536589443683624,
+          0.015665847808122635,
+          0.007130937650799751,
+          -0.022730141878128052,
+          0.06554242223501205,
+          0.045460283756256104,
+          -0.004136388190090656,
+          -0.01041427906602621,
+          -0.046704307198524475,
+          0.0391690619289875,
+          0.032095883041620255,
+          -0.057296305894851685,
+          0.03236246109008789,
+          -0.017913982272148132,
+          0.042403530329465866,
+          0.02841711975634098,
+          0.025680260732769966,
+          -0.014377391897141933,
+          0.01803838461637497,
+          -0.005087179597467184,
+          -0.04777061566710472,
+          -0.034441761672496796,
+          -0.03433513268828392,
+          0.0076729776337742805,
+          -0.003112287726253271,
+          -0.012760157696902752,
+          -0.02900358848273754,
+          0.04133722186088562,
+          0.03214919939637184,
+          0.03563247248530388,
+          0.02104181982576847,
+          0.0030700797215104103,
+          0.00974783580750227,
+          0.016616638749837875,
+          -0.01890920288860798,
+          -0.014812801033258438,
+          -0.00617126002907753,
+          -0.03870699554681778,
+          -0.01540815643966198,
+          -0.017345283180475235,
+          0.002443623496219516,
+          -0.03756960108876228,
+          0.014155244454741478,
+          0.006144602317363024,
+          0.0014572881627827883,
+          0.011214010417461395,
+          0.002150388667359948,
+          0.03664546459913254,
+          -0.0016416707076132298,
+          -0.026906516402959824,
+          0.009943326003849506,
+          0.010405393317341805,
+          0.00527822645381093,
+          -0.029892180114984512,
+          -0.013799807988107204,
+          -0.006024642381817102,
+          0.027812877669930458,
+          -0.026266731321811676,
+          -0.009436829946935177,
+          -0.0020393149461597204,
+          0.0025813549291342497,
+          0.0016538887284696102,
+          -0.02024208754301071,
+          -0.016554437577724457,
+          0.014626197516918182,
+          0.0052115824073553085,
+          0.006637769751250744,
+          -0.04396745190024376,
+          -0.0198688805103302,
+          0.05573238804936409,
+          0.01586133800446987,
+          -0.06284110993146896,
+          0.023725362494587898,
+          -0.01658109575510025,
+          -0.011853795498609543,
+          0.006913233082741499,
+          0.02504047565162182,
+          -0.008406064473092556,
+          -0.01062754075974226,
+          0.0030611937399953604,
+          -0.052995529025793076,
+          -0.019389040768146515,
+          0.0035099319647997618,
+          0.004220804199576378,
+          -0.049263447523117065,
+          -0.019833335652947426,
+          -0.007597447372972965,
+          -0.015719164162874222,
+          0.003709864802658558,
+          0.009658977389335632,
+          -0.017416371032595634,
+          -0.010423164814710617,
+          0.012928989715874195,
+          0.018020613119006157,
+          -0.02299671806395054,
+          -0.03636111691594124,
+          0.027350811287760735,
+          -0.0019337947014719248,
+          -0.008099501021206379,
+          0.0016805464401841164,
+          0.012573553249239922,
+          0.0016838787123560905,
+          0.017771806567907333,
+          -0.005153824109584093,
+          -0.032682351768016815,
+          -0.02818608656525612,
+          0.02752852812409401,
+          -0.021734919399023056,
+          0.004147495608776808,
+          0.02278345637023449,
+          0.008303876966238022,
+          -0.011187352240085602,
+          0.020046597346663475,
+          -0.0527467243373394,
+          -0.008135044947266579,
+          0.0006581122288480401,
+          -0.006269005127251148,
+          0.025307053700089455,
+          -0.016829900443553925,
+          -0.014821686781942844,
+          0.019833335652947426,
+          0.009005863219499588,
+          0.0021626066882163286,
+          0.004014207050204277,
+          0.005966884084045887,
+          0.05512814596295357,
+          0.016154572367668152,
+          -0.01197819784283638,
+          0.017860665917396545,
+          -0.024596180766820908,
+          0.03102957457304001,
+          0.021290624514222145,
+          0.00529599841684103,
+          0.001194043317809701,
+          0.04581571742892265,
+          0.01008550077676773,
+          0.008757057599723339,
+          -0.011507244780659676,
+          0.011516130529344082,
+          -0.007401957642287016,
+          -0.05594564974308014,
+          -0.00806395709514618,
+          -0.022534651681780815,
+          0.002414744347333908,
+          0.0233699269592762,
+          0.031438328325748444,
+          -0.018376048654317856,
+          -0.00924133975058794,
+          -0.0009363520657643676,
+          -0.012102600187063217,
+          0.0010907446267083287,
+          -0.004687313921749592,
+          0.014359620399773121,
+          -0.0027457440737634897,
+          0.007686306722462177,
+          0.0018615968292579055,
+          -0.003058972302824259,
+          0.021343940868973732,
+          -0.03257572278380394,
+          -0.011853795498609543,
+          -0.03788949176669121,
+          -0.059784360229969025,
+          -0.03449507802724838,
+          0.030709682032465935,
+          0.023725362494587898,
+          -0.002539146924391389,
+          -0.004589569289237261,
+          -0.014324076473712921,
+          -0.031438328325748444,
+          -0.021592745557427406,
+          0.0006164595251902938,
+          0.04894355684518814,
+          -0.007868467830121517,
+          -0.02578689157962799,
+          0.01761186122894287,
+          -0.006939890794456005,
+          -0.004389636218547821,
+          -0.004927233327180147,
+          -0.03636111691594124,
+          0.023405469954013824,
+          0.010174359194934368,
+          0.03053196519613266,
+          -0.022143671289086342,
+          0.005575904622673988,
+          -0.006095729768276215,
+          -0.014377391897141933,
+          0.029963266104459763,
+          -0.008859246037900448,
+          0.004727300722151995,
+          0.0058869109489023685,
+          -0.003043422009795904,
+          0.0001384951319778338,
+          -0.005260454956442118,
+          -0.032540179789066315,
+          -0.026017924770712852,
+          -0.041230592876672745,
+          0.002370314672589302,
+          -0.02825717255473137,
+          -0.027137549594044685,
+          -0.06650210171937943,
+          0.024880530312657356,
+          -0.07098059356212616,
+          -0.02498716115951538,
+          0.038529276847839355,
+          -0.008885903283953667,
+          0.01768294721841812,
+          -0.04279451072216034,
+          -0.0006664427346549928,
+          -0.05107617378234863,
+          -0.0009580114856362343,
+          0.035241492092609406,
+          0.0029234623070806265,
+          -0.006393407471477985,
+          0.014368506148457527,
+          -0.00013120591756887734,
+          -0.03389083594083786,
+          -0.04229690134525299,
+          0.0038276028353720903,
+          0.005207139533013105,
+          -0.02301448956131935,
+          -0.014288532547652721,
+          0.04339875280857086,
+          0.00908583588898182,
+          -0.0006070183007977903,
+          0.0300876684486866,
+          0.007939554750919342,
+          -0.027137549594044685,
+          0.019246866926550865,
+          0.02564471773803234,
+          0.038884714245796204,
+          -0.02768847532570362,
+          0.021130679175257683,
+          -0.03200702369213104,
+          -0.005629220046103001,
+          0.011569445952773094,
+          0.0023636503610759974,
+          -0.006011313758790493,
+          0.014457364566624165,
+          -0.0015683620003983378,
+          0.0034077439922839403,
+          0.006842145696282387,
+          -0.006304548587650061,
+          0.032984472811222076,
+          -0.0004387414955999702,
+          -0.016821015626192093,
+          -0.021148450672626495,
+          0.009694520384073257,
+          0.031082890927791595,
+          0.028737012296915054,
+          -0.018304960802197456,
+          -0.014635083265602589,
+          -0.003976441919803619,
+          0.029056904837489128,
+          0.016083484515547752,
+          -0.01384423766285181,
+          0.0067444005981087685,
+          0.02431183122098446,
+          -0.023618731647729874,
+          -0.01586133800446987,
+          0.017398599535226822,
+          -0.006424508057534695,
+          0.012564667500555515,
+          -0.012644641101360321,
+          0.00200932496227324,
+          0.017931753769516945,
+          -0.005367085803300142,
+          0.026693254709243774,
+          -0.017034277319908142,
+          0.010529795661568642,
+          -0.007459715940058231,
+          -0.008681527338922024,
+          0.007917339913547039,
+          0.06948776543140411,
+          -0.020615296438336372,
+          0.022659054026007652,
+          0.018464908003807068,
+          -0.0018671504221856594,
+          0.034726109355688095,
+          -0.04457169026136398,
+          -0.046704307198524475,
+          0.012182573787868023,
+          -0.04314994812011719,
+          -0.0020493115298449993,
+          -0.022410249337553978,
+          -0.023032261058688164,
+          0.008321648463606834,
+          0.011915996670722961,
+          -0.024880530312657356,
+          -0.008717071264982224,
+          0.020864101126790047,
+          0.01577247865498066,
+          -0.02169937640428543,
+          0.019922195002436638,
+          0.00920579582452774,
+          -0.018749255686998367,
+          0.032682351768016815,
+          -0.02205481193959713,
+          -0.0003795946831814945,
+          -0.007099837064743042,
+          -0.0024947174824774265,
+          -0.00668219942599535,
+          -0.004285227041691542,
+          0.008868131786584854,
+          -0.02893250249326229,
+          -0.01622566021978855,
+          0.01680324412882328,
+          -0.00772185018286109,
+          -0.010058842599391937,
+          -0.042403530329465866,
+          0.0032611265778541565,
+          0.013782036490738392,
+          -0.023281067609786987,
+          0.01132064126431942,
+          -0.019477900117635727,
+          -0.032895613461732864,
+          0.01993996649980545,
+          0.03508154675364494,
+          -0.032984472811222076,
+          0.0016561102820560336,
+          0.00873040035367012,
+          -0.009028078056871891,
+          -0.0024502878077328205,
+          -0.01766517572104931,
+          0.007157595362514257,
+          0.03271789476275444,
+          0.014741714112460613,
+          -0.013257768005132675,
+          -0.024222973734140396,
+          -0.017807351425290108,
+          0.021379483863711357,
+          -0.0059313406236469746,
+          0.005162709858268499,
+          0.02562694624066353,
+          0.041514940559864044,
+          0.012173688039183617,
+          0.0008830366423353553,
+          -0.013666519895195961,
+          0.014803915284574032,
+          -0.015887996181845665,
+          0.021646060049533844,
+          0.022072583436965942,
+          -0.01746080070734024,
+          -0.03803166747093201,
+          0.018429363146424294,
+          0.00022339716088026762,
+          0.004554025363177061,
+          0.008112830109894276,
+          0.02768847532570362,
+          -0.016616638749837875,
+          -0.00549593148753047,
+          0.03934678062796593,
+          0.016314519569277763,
+          -0.014892774634063244,
+          -0.01622566021978855,
+          0.016838787123560905,
+          0.0074419439770281315,
+          0.006060186307877302,
+          -0.0018293854082003236,
+          -0.010369849391281605,
+          -0.0016127914423123002,
+          0.009365742094814777,
+          -0.008481594733893871,
+          0.008992534130811691,
+          0.014839459210634232,
+          0.005438172724097967,
+          0.0011035181814804673,
+          -0.01737194135785103,
+          -0.01263575442135334,
+          0.015799136832356453,
+          0.010538681410253048,
+          -0.010023298673331738,
+          0.00927688367664814,
+          0.007961769588291645,
+          0.014013069681823254,
+          0.032166969031095505,
+          -0.0031278380192816257,
+          -0.01671438477933407,
+          -0.0026657709386199713,
+          -0.0006958773010410368,
+          -0.025769120082259178,
+          0.007601890247315168,
+          -0.014892774634063244,
+          -0.02898581698536873,
+          0.018429363146424294,
+          0.02249910682439804,
+          -0.011418386362493038,
+          -0.01802949793636799,
+          -0.0030878514517098665,
+          -0.0043563139624893665,
+          -0.006264561787247658,
+          0.007406400516629219,
+          -0.011276211589574814,
+          0.042616792023181915,
+          -0.001704982714727521,
+          0.0023614289239048958,
+          -0.02703091874718666,
+          -0.0023503215052187443,
+          0.04332766681909561,
+          0.025253737345337868,
+          0.0030545294284820557,
+          0.009116937406361103,
+          0.008010641671717167,
+          0.0467398539185524,
+          -0.013764264062047005,
+          -0.04133722186088562,
+          0.008601554669439793,
+          -0.015319298021495342,
+          0.012173688039183617,
+          0.03266458213329315,
+          -0.009436829946935177,
+          0.010014412924647331,
+          0.010547567158937454,
+          -0.03147387132048607,
+          0.027350811287760735,
+          -0.023796450346708298,
+          -0.022605737671256065,
+          0.030994031578302383,
+          -0.01723865233361721,
+          0.01074305735528469,
+          0.028737012296915054,
+          0.010911889374256134,
+          -0.029838863760232925,
+          0.0013817580183967948,
+          -0.010423164814710617,
+          -0.025218194350600243,
+          0.025911293923854828,
+          0.02863038145005703,
+          0.018642624840140343,
+          0.0063001057133078575,
+          0.0015339291421696544,
+          0.004896132741123438,
+          0.0033699788618832827,
+          -0.01007661409676075,
+          0.013204452581703663,
+          -0.010165473446249962,
+          -0.01768294721841812,
+          -0.0015283754328265786,
+          -0.013568774797022343,
+          -0.00927688367664814,
+          0.0021748249419033527,
+          -0.02607124112546444,
+          0.009134708903729916,
+          0.012351405806839466,
+          0.010458708740770817,
+          -0.018962517380714417,
+          -0.01862485334277153,
+          0.05534140765666962,
+          0.013799807988107204,
+          -0.0032700125593692064,
+          0.004087515641003847,
+          -0.017585203051567078,
+          -0.04318549111485481,
+          0.004571797326207161,
+          0.011418386362493038,
+          0.005562575533986092,
+          0.00020270967797841877,
+          -0.030496420338749886,
+          0.008108386769890785,
+          0.032326918095350266,
+          0.008046185597777367,
+          0.04446506127715111,
+          -0.0012251439038664103,
+          0.030123213306069374,
+          -0.008414950221776962,
+          0.003574354574084282,
+          0.019513443112373352,
+          0.0041230591014027596,
+          -0.019033605232834816,
+          0.03038978949189186,
+          0.01834050565958023,
+          0.018304960802197456,
+          0.006979877129197121,
+          -0.04514038935303688,
+          -0.009872239083051682,
+          0.010236560367047787,
+          0.03182930499315262,
+          0.020259859040379524,
+          -0.017176451161503792,
+          -0.007437501102685928,
+          0.03296670317649841,
+          -0.022232530638575554,
+          -0.004431844223290682,
+          0.0116138756275177,
+          -0.008597111329436302,
+          0.002483610063791275,
+          -0.004687313921749592,
+          0.02249910682439804,
+          -0.039844390004873276,
+          -0.004687313921749592,
+          0.0019349055364727974,
+          0.0057225218042731285,
+          -0.002215922111645341,
+          -0.006939890794456005,
+          0.010405393317341805,
+          0.014261875301599503,
+          0.009076950140297413,
+          0.02943011187016964,
+          -0.0201532281935215,
+          0.015034948475658894,
+          -0.015168237499892712,
+          -0.003134502563625574,
+          0.005749179515987635,
+          -0.03138501197099686,
+          -0.030283158645033836,
+          0.007619662210345268,
+          0.03175821900367737,
+          0.013080050237476826,
+          0.0014817244373261929,
+          -0.004589569289237261,
+          -0.011516130529344082,
+          0.026853200048208237,
+          0.02109513431787491,
+          0.02111290581524372,
+          0.023831993341445923,
+          -0.005087179597467184,
+          -0.0004323547473177314,
+          0.012475808151066303,
+          -0.008650426752865314,
+          -0.024791670963168144,
+          0.008930332958698273,
+          -0.011427272111177444,
+          0.0270842332392931,
+          -0.01724753901362419,
+          -0.003258905140683055,
+          0.005646991543471813,
+          -0.012404721230268478,
+          -0.01128509733825922,
+          -0.039702218025922775,
+          0.025893522426486015,
+          -0.02548477053642273,
+          0.0036343345418572426,
+          -0.019193550571799278,
+          0.01490166038274765,
+          -0.03870699554681778,
+          0.000813615508377552,
+          -0.0018393819918856025,
+          0.010538681410253048,
+          -0.010529795661568642,
+          0.0021770463790744543,
+          -0.03193593770265579,
+          -0.03788949176669121,
+          -0.0116849634796381,
+          0.005073850974440575,
+          0.007899568416178226,
+          -0.0009330198518000543,
+          0.006913233082741499,
+          0.015967968851327896,
+          -0.025093792006373405,
+          -0.03369534760713577,
+          0.018820343539118767,
+          -0.03440621867775917,
+          0.020419806241989136,
+          0.0004870585980825126,
+          -0.02804391086101532,
+          -0.021965952590107918,
+          -0.016492236405611038,
+          -0.015248210169374943,
+          0.021006274968385696,
+          0.022552423179149628,
+          0.017798464745283127,
+          -0.016847673803567886,
+          0.0025124892126768827,
+          0.014039727859199047,
+          0.008121715858578682,
+          0.011880452744662762,
+          0.008481594733893871,
+          0.01825164631009102,
+          -0.016847673803567886,
+          -0.015887996181845665,
+          -0.012244774959981441,
+          -0.002794616622850299,
+          -0.013799807988107204,
+          0.0011179577559232712,
+          -0.038813624531030655,
+          -0.00737529993057251,
+          0.01116958074271679,
+          -0.0020048818551003933,
+          0.06038859859108925,
+          0.0527467243373394,
+          0.013160022906959057,
+          0.007384185679256916,
+          -0.01876702904701233,
+          0.0461711548268795,
+          -0.025804663076996803,
+          0.04986768960952759,
+          0.0034521734341979027,
+          -0.013142251409590244,
+          -0.013462143950164318,
+          0.013426600024104118,
+          -0.029199078679084778,
+          -0.00719313882291317,
+          -0.005424844101071358,
+          -0.009005863219499588,
+          0.018002839758992195,
+          0.00739307189360261,
+          -0.009347970597445965,
+          0.003696535946801305,
+          0.009383514523506165,
+          -0.011258440092206001,
+          0.013222224079072475,
+          0.01612791419029236,
+          0.0017349726986140013,
+          0.015070492401719093,
+          -0.04208363965153694,
+          -0.011231781914830208,
+          -0.00041291682282462716,
+          0.009685634635388851,
+          -0.006220132578164339,
+          0.013168908655643463,
+          -0.006428950931876898,
+          -0.0044229584746062756,
+          -0.007015420589596033,
+          -0.02169937640428543,
+          0.022534651681780815,
+          0.036041226238012314,
+          -0.0016938752960413694,
+          -0.048979099839925766,
+          -0.02614232711493969,
+          -0.050223127007484436,
+          -0.012928989715874195,
+          0.024063026532530785,
+          -0.0050249784253537655,
+          0.015976853668689728,
+          0.004087515641003847,
+          0.0048828041180968285,
+          -0.0030456434469670057,
+          -0.017567431554198265,
+          0.01613680087029934,
+          -0.0232277512550354,
+          0.002739079762250185,
+          0.0003854260721709579,
+          0.007144266273826361,
+          -0.03543698415160179,
+          0.007757393643260002,
+          0.00023422685626428574,
+          0.012093714438378811,
+          -0.03660992160439491,
+          -0.038244929164648056,
+          -0.017398599535226822,
+          0.015941310673952103,
+          -0.002383643528446555,
+          -0.012244774959981441,
+          0.0029501200187951326,
+          0.005033864174038172,
+          0.014741714112460613,
+          -0.017567431554198265,
+          -0.006229018326848745,
+          0.02162828855216503,
+          -0.0032433548476547003,
+          0.007206467911601067,
+          -0.035099320113658905,
+          -0.01555921696126461,
+          -0.01278681494295597,
+          0.0116494195535779,
+          -0.0133999427780509,
+          0.004905018489807844,
+          0.00017799575289245695,
+          0.014848344959318638,
+          -0.015505901537835598,
+          -0.027013147249817848,
+          -0.011231781914830208,
+          0.020099913701415062,
+          -0.03511708974838257,
+          0.036112312227487564,
+          0.002739079762250185,
+          0.02118399366736412,
+          -0.014404049143195152,
+          0.025609172880649567,
+          -0.004036421421915293,
+          -0.015301525592803955,
+          -0.002151499269530177,
+          0.022001497447490692,
+          0.03332213684916496,
+          -0.021361712366342545,
+          0.0013228888856247067,
+          -0.019015833735466003,
+          0.003327770857140422,
+          -0.02351210080087185,
+          0.017345283180475235,
+          -0.004007542505860329,
+          0.01234252005815506,
+          0.017105363309383392,
+          0.013204452581703663,
+          0.05996207520365715,
+          -0.017194222658872604,
+          0.02557362988591194,
+          0.012324748560786247,
+          -0.017265310510993004,
+          -0.007935111410915852,
+          -0.032095883041620255,
+          -0.027333039790391922,
+          0.03133169561624527,
+          0.002228140365332365,
+          0.0015561438631266356,
+          -0.03104734607040882,
+          0.015879109501838684,
+          0.01832273229956627,
+          0.019460128620266914,
+          -0.014288532547652721,
+          -0.015274868346750736,
+          0.016030170023441315,
+          0.026248957961797714,
+          -0.008135044947266579,
+          0.003865367965772748,
+          0.012769043445587158,
+          -0.04076852649450302,
+          0.005282669793814421,
+          0.0029634488746523857,
+          0.012857901863753796,
+          -0.0021548315417021513,
+          0.010183244943618774,
+          -0.03214919939637184,
+          -0.0055137029848992825,
+          -0.01457288209348917,
+          0.02315666526556015,
+          -0.015665847808122635,
+          -0.047806162387132645,
+          -0.03161604329943657,
+          0.025147106498479843,
+          -0.030194299295544624,
+          -0.020988503471016884,
+          0.01311559323221445,
+          0.0022170329466462135,
+          0.015337069518864155,
+          -0.01471505593508482,
+          -0.02747521363198757,
+          -0.038955800235271454,
+          -0.021965952590107918,
+          -0.0006886575138196349,
+          0.0394534096121788,
+          -0.03010544180870056,
+          0.015159351751208305,
+          0.005309327505528927,
+          0.00686436053365469,
+          -0.004869475029408932,
+          0.002365871798247099,
+          -0.029447883367538452,
+          -0.04133722186088562,
+          0.02351210080087185,
+          -0.00494944816455245,
+          -0.0015150465769693255,
+          0.014803915284574032,
+          -0.033055562525987625,
+          -0.023103348910808563,
+          0.003154495730996132,
+          0.00737529993057251,
+          -0.00785069540143013,
+          -0.002352542942389846,
+          0.0013395498972386122,
+          -0.01215591561049223,
+          -0.038600362837314606,
+          -0.004416293930262327,
+          0.03397969529032707,
+          0.02177046425640583,
+          0.031740445643663406,
+          0.011525017209351063,
+          0.00010468705295352265,
+          -0.02886141464114189,
+          -0.013595432043075562,
+          -0.006895461119711399,
+          -0.0009818923426792026,
+          -0.0025769120547920465,
+          0.01848267950117588,
+          0.005291555542498827,
+          0.0034077439922839403,
+          -0.02169937640428543,
+          -0.02783064916729927,
+          -0.03511708974838257,
+          -0.01928240992128849,
+          -0.010049956850707531,
+          -0.00949014537036419,
+          -0.03899134323000908,
+          -0.016741042956709862,
+          0.04485604166984558,
+          0.026995373889803886,
+          -0.009152480401098728,
+          -0.00018729818111751229,
+          -0.0012240331852808595,
+          -0.012973419390618801,
+          0.019886652007699013,
+          -0.013826466165482998,
+          0.0059091257862746716,
+          0.00031656029750593007,
+          0.018607081845402718,
+          0.0017538551473990083,
+          0.0010013302089646459,
+          -0.03419295698404312,
+          -0.009401286020874977,
+          0.031171750277280807,
+          0.00701097771525383,
+          -0.019531216472387314,
+          0.0163678340613842,
+          0.009259111247956753,
+          -0.014466251246631145,
+          -0.047806162387132645,
+          0.026782112196087837,
+          0.04567354544997215,
+          0.02212589979171753,
+          -0.005940226372331381,
+          0.022179214283823967,
+          0.024933844804763794,
+          -0.0012462479062378407,
+          -0.0007541910745203495,
+          -0.004945005290210247,
+          0.026977602392435074,
+          0.006535581778734922,
+          -0.000127387756947428,
+          0.00939240027219057,
+          0.01810947060585022,
+          0.005838038399815559,
+          -0.0007697413675487041,
+          0.007077622227370739,
+          -0.020793013274669647,
+          -0.004913904704153538,
+          -0.013853123411536217,
+          0.010974090546369553,
+          0.00920579582452774,
+          0.041088417172431946,
+          0.008774830028414726,
+          0.013559889048337936,
+          -0.0043563139624893665,
+          -0.010174359194934368,
+          0.028648152947425842,
+          0.014039727859199047,
+          0.001577247865498066,
+          0.008499366231262684,
+          -0.030834084376692772,
+          -0.01586133800446987,
+          -0.013950868509709835,
+          0.01384423766285181,
+          0.0037520728074014187,
+          0.04286559671163559,
+          0.006873246282339096,
+          0.0016794357215985656,
+          -0.02943011187016964,
+          0.0026235629338771105,
+          0.008232789114117622,
+          0.02285454422235489,
+          -0.013266653753817081,
+          -2.6171763238380663e-05,
+          0.0073664141818881035,
+          0.005638105794787407,
+          0.026497764512896538,
+          -0.026195643469691277,
+          -0.017052048817276955,
+          0.019300181418657303,
+          0.09511470794677734,
+          -0.009330199100077152,
+          -0.0233699269592762,
+          -0.026053469628095627,
+          -0.02104181982576847,
+          0.008241675794124603,
+          0.017585203051567078,
+          0.005429286975413561,
+          0.017443029209971428,
+          0.010218788869678974,
+          0.01788732409477234,
+          0.015603646636009216,
+          -0.015976853668689728,
+          -0.007419729605317116,
+          -0.03225582838058472,
+          0.0327356681227684,
+          0.00014911657490301877,
+          -0.0047228578478097916,
+          0.00739307189360261,
+          0.007135380525141954,
+          0.019744478166103363,
+          0.018091699108481407,
+          0.015887996181845665,
+          -0.007979541085660458,
+          -0.011924882419407368,
+          0.020135456696152687,
+          -0.013142251409590244,
+          -0.0034810525830835104,
+          -0.01899806223809719,
+          -0.021592745557427406,
+          -0.005651434417814016,
+          0.014643969014286995,
+          -0.006468937732279301,
+          0.014848344959318638,
+          -0.01132064126431942,
+          0.028310488909482956,
+          -0.021965952590107918,
+          0.011160694994032383,
+          -0.00737529993057251,
+          -0.01526598259806633,
+          0.005216025281697512,
+          0.010245447047054768,
+          -0.007677420508116484,
+          0.010716399177908897,
+          -0.014324076473712921,
+          0.014821686781942844,
+          0.019477900117635727,
+          -0.03650329262018204,
+          0.03328659385442734,
+          0.008126158267259598,
+          -0.009836695156991482,
+          0.00924133975058794,
+          -0.00582470977678895,
+          0.012529123574495316,
+          0.007268669083714485,
+          0.00806395709514618,
+          0.008352749049663544,
+          0.00817947369068861,
+          -0.007659649010747671,
+          0.014883887954056263,
+          0.012138144113123417,
+          0.0007353085093200207,
+          0.010636426508426666,
+          0.014279646798968315,
+          -0.009730064310133457,
+          -0.004696200136095285,
+          0.0038364888168871403,
+          -0.010325419716536999,
+          0.007317541632801294,
+          -0.009108050726354122,
+          0.0060912868939340115,
+          0.024507321417331696,
+          -0.0017083149868994951,
+          -0.016492236405611038,
+          -0.016625525429844856,
+          -0.007935111410915852,
+          -0.021237310022115707,
+          -0.015434814617037773,
+          0.00514938123524189,
+          -0.030834084376692772,
+          0.013053392060101032,
+          0.028665924444794655,
+          0.0233699269592762,
+          -0.006180145777761936,
+          -0.024720583111047745,
+          -0.012413606978952885,
+          -0.0018582645570859313,
+          0.011276211589574814,
+          0.005162709858268499,
+          -0.02454286627471447,
+          0.002643556334078312,
+          -0.011356184259057045,
+          -0.021432798355817795,
+          -0.008277218788862228,
+          -0.01651000790297985,
+          0.003458837978541851,
+          0.0030456434469670057,
+          -0.008441608399152756,
+          -0.010707513429224491,
+          0.003541032550856471,
+          -0.03685872629284859,
+          0.014048613607883453,
+          0.03207810968160629,
+          -0.000934130628593266,
+          -0.03255794942378998,
+          -0.015221552923321724,
+          -0.003250019159168005,
+          -0.017194222658872604,
+          -0.02863038145005703,
+          0.008934776298701763,
+          0.006104615516960621,
+          -0.008454937487840652,
+          0.01252023782581091,
+          -0.004036421421915293,
+          0.02001105435192585,
+          0.0009130265680141747,
+          0.01869594119489193,
+          -0.0408751554787159,
+          0.012564667500555515,
+          0.019158007577061653,
+          -0.019122464582324028,
+          0.006904346868395805,
+          0.023458784446120262,
+          -0.008383849635720253,
+          0.0034699453972280025,
+          0.035756874829530716,
+          0.010369849391281605,
+          -0.042474620044231415,
+          0.01905137673020363,
+          0.006082401145249605,
+          -0.00668219942599535,
+          0.016323404386639595,
+          0.019673390313982964,
+          -0.000788623932749033,
+          0.020757470279932022,
+          -0.006446722894906998,
+          -0.018233874812722206,
+          0.00218260008841753,
+          0.020846329629421234,
+          -0.009330199100077152,
+          -0.015345955267548561,
+          0.01092077512294054,
+          0.0027235294692218304,
+          0.004758401308208704,
+          0.01738082803785801,
+          -0.02607124112546444,
+          -0.018642624840140343,
+          0.019513443112373352,
+          0.012831244617700577,
+          0.017336398363113403,
+          -0.011214010417461395,
+          -0.01402195543050766,
+          0.02219698764383793,
+          -0.005464830435812473,
+          0.010707513429224491,
+          -0.031082890927791595,
+          0.008748171851038933,
+          -0.0019904423970729113,
+          0.01848267950117588,
+          -0.004918347578495741,
+          0.008752615191042423,
+          0.03255794942378998,
+          -0.015514787286520004,
+          -0.025964610278606415,
+          0.009543460793793201,
+          -0.033624257892370224,
+          -0.010094386525452137,
+          -0.029607830569148064,
+          -0.03920460492372513,
+          0.03170490264892578,
+          -0.008046185597777367,
+          -0.0049583339132368565,
+          0.026746569201350212,
+          0.007353085093200207,
+          0.0020270966924726963,
+          -0.03213142603635788,
+          0.021574974060058594,
+          0.006642212625592947,
+          0.007784051354974508,
+          -0.018873659893870354,
+          0.025147106498479843,
+          -0.0030922943260520697,
+          -0.021166222169995308,
+          0.013462143950164318,
+          0.012182573787868023,
+          0.008126158267259598,
+          -0.004534032195806503,
+          0.001586133730597794,
+          -0.03682318329811096,
+          -0.005775837227702141,
+          0.008246118202805519,
+          0.005522589199244976,
+          -0.003236690303310752,
+          0.018802572041749954,
+          -0.016909874975681305,
+          -0.002843489171937108,
+          -0.005629220046103001,
+          -0.022747913375496864,
+          0.0033855291549116373,
+          -0.03248686343431473,
+          -0.020686382427811623,
+          0.019780021160840988,
+          -0.003989770542830229,
+          -0.01825164631009102,
+          0.014581767842173576,
+          -0.03980884701013565,
+          -0.011178466491401196,
+          0.007304212544113398,
+          0.027777334675192833,
+          -0.03447730466723442,
+          0.015719164162874222,
+          -0.01958453096449375,
+          -0.0023769792169332504,
+          0.023618731647729874,
+          -0.011809365823864937,
+          -0.00010725563333835453,
+          -0.012564667500555515,
+          -0.031882621347904205,
+          0.003963112831115723,
+          0.028239401057362556,
+          -0.029234621673822403,
+          0.0009446826297789812,
+          0.031136205419898033,
+          -0.000653669296298176,
+          0.004616227000951767,
+          0.0007586340070702136,
+          0.01113403681665659,
+          -0.015186008997261524,
+          0.015106036327779293,
+          0.005838038399815559,
+          0.00582470977678895,
+          -0.01372872106730938,
+          -0.005202696658670902,
+          0.02710200473666191,
+          0.001230697613209486,
+          -0.008819258771836758,
+          0.0047006430104374886,
+          0.004058636259287596,
+          0.010174359194934368,
+          -0.0016627747099846601,
+          0.014457364566624165,
+          0.018100585788488388,
+          -0.028168313205242157,
+          0.012884560041129589,
+          0.009836695156991482,
+          7.844586798455566e-05,
+          0.013995298184454441,
+          -0.017514115199446678,
+          -0.034352902323007584,
+          0.02161051705479622,
+          -0.022019268944859505,
+          -0.016234545037150383,
+          0.0270842332392931,
+          -0.010867459699511528,
+          -0.006064629182219505,
+          -0.009774493984878063,
+          -0.001900472561828792,
+          0.010609768331050873,
+          -0.009978869929909706,
+          -0.009463487192988396,
+          0.023760905489325523,
+          0.018145015463232994,
+          0.020917417481541634,
+          0.029856635257601738,
+          0.0012118150480091572,
+          0.01570139080286026,
+          -0.0005223245243541896,
+          0.008974762633442879,
+          0.005171595606952906,
+          -0.0018627075478434563,
+          -0.01738971285521984,
+          0.025982381775975227,
+          -0.001804949133656919,
+          0.006264561787247658,
+          0.0063223205506801605,
+          -0.014368506148457527,
+          -0.010867459699511528,
+          -0.026479993015527725,
+          -0.004887246992439032,
+          -0.006580011453479528,
+          -0.016234545037150383,
+          0.0789068192243576,
+          -0.0099344402551651,
+          -0.016554437577724457,
+          0.010494251735508442,
+          -0.004856146406382322,
+          0.0015528115909546614,
+          -0.016847673803567886,
+          0.003736522514373064,
+          -0.008783715777099133,
+          -0.000988556770607829,
+          -0.030176527798175812,
+          -0.0015616975724697113,
+          0.0167943574488163,
+          0.015283754095435143,
+          -0.006557796616107225,
+          -0.018731484189629555,
+          0.008223903365433216,
+          -0.023209979757666588,
+          0.02036648988723755,
+          0.0012418050318956375,
+          0.018447136506438255,
+          0.005087179597467184,
+          0.005082736723124981,
+          -0.014946090057492256,
+          -0.016749927774071693,
+          0.004434065893292427,
+          0.012315861880779266,
+          0.002247022930532694,
+          -0.0090236347168684,
+          -0.009152480401098728,
+          -0.011009634472429752,
+          -0.021859321743249893,
+          0.020544208586215973,
+          -0.01296453271061182,
+          -0.0039142402820289135,
+          -0.003983106464147568,
+          -0.0020726369693875313,
+          0.006331206299364567,
+          0.011356184259057045,
+          -0.02358318865299225,
+          0.0018849222688004375,
+          -0.016972076147794724,
+          0.006913233082741499,
+          0.02504047565162182,
+          0.00981892365962267,
+          -0.04720192030072212,
+          -0.00651781028136611,
+          -0.02754630148410797,
+          0.013746492564678192,
+          -0.011747164651751518,
+          -0.009445715695619583,
+          0.0133999427780509,
+          0.008072842843830585,
+          0.028719240799546242,
+          0.010280990041792393,
+          0.009881124831736088,
+          0.015372613444924355,
+          -0.00032850075513124466,
+          -0.004440730437636375,
+          -0.017647404223680496,
+          0.013568774797022343,
+          -0.023316610604524612,
+          0.020348718389868736,
+          -0.0006103505147621036,
+          -0.017034277319908142,
+          0.004194146487861872,
+          0.002821274334564805,
+          0.013853123411536217,
+          -0.021806007251143456,
+          -0.022481335327029228,
+          0.022730141878128052,
+          0.018216101452708244,
+          0.012600211426615715,
+          0.015123807825148106,
+          0.03182930499315262,
+          0.02139725536108017,
+          0.024294059723615646,
+          -0.025324825197458267,
+          0.0014428485883399844,
+          -0.033855292946100235,
+          -0.005087179597467184,
+          -0.026551079005002975,
+          0.004038643091917038,
+          -0.012973419390618801,
+          0.005638105794787407,
+          -0.022001497447490692,
+          0.001768294838257134,
+          0.017549660056829453,
+          -0.030691910535097122,
+          0.010965204797685146,
+          0.015754707157611847,
+          0.00788623932749033,
+          0.017993954941630363,
+          0.006944333668798208,
+          -0.012831244617700577,
+          -0.01737194135785103,
+          -0.008717071264982224,
+          -0.013782036490738392,
+          0.014812801033258438,
+          0.0020082141272723675,
+          -0.0027679589111357927,
+          -0.00027990597300231457,
+          0.0001258604897884652,
+          -0.014466251246631145,
+          -0.001746080000884831,
+          -0.005180481821298599,
+          -0.013710948638617992,
+          0.005926897749304771,
+          0.0014928317395970225,
+          0.013222224079072475,
+          0.008917003870010376,
+          -0.007286441046744585,
+          0.011293983086943626,
+          0.02578689157962799,
+          0.0005528698093257844,
+          -0.018358277156949043,
+          0.00721091078594327,
+          0.008290547877550125,
+          0.007215353660285473,
+          0.006073514930903912,
+          0.0015172680141404271,
+          -0.012253660708665848,
+          0.027333039790391922,
+          -0.023938624188303947,
+          0.02205481193959713,
+          0.004856146406382322,
+          -0.0008075064979493618,
+          0.0003068413643632084,
+          -0.03518817946314812,
+          0.002401415491476655,
+          -0.024222973734140396,
+          -0.04112396016716957,
+          -0.02411634288728237,
+          -0.01547924429178238,
+          0.0007264225860126317,
+          0.022765684872865677,
+          0.0012862344738095999,
+          -0.027297494933009148,
+          0.022907858714461327,
+          -0.001922687399201095,
+          -0.015959082171320915,
+          0.00420081103220582,
+          -0.001362875453196466,
+          0.0019515665480867028,
+          0.0028745897579938173,
+          -0.0029567843303084373,
+          -0.0081839170306921,
+          -0.017407484352588654,
+          -0.02635558880865574,
+          0.003163381712511182,
+          0.009223568253219128,
+          -0.019815564155578613,
+          0.02249910682439804,
+          -0.009978869929909706,
+          -0.015879109501838684,
+          0.026497764512896538,
+          -0.009312426671385765,
+          -0.021343940868973732,
+          -0.021308396011590958,
+          0.02095296047627926,
+          0.004669542424380779,
+          -0.024418462067842484,
+          0.01643003523349762,
+          0.005562575533986092,
+          -0.017638517543673515,
+          -0.002251465804874897,
+          -0.012040399014949799,
+          0.012351405806839466,
+          -0.01782512292265892,
+          0.0016472243005409837,
+          0.010503137484192848,
+          0.0017738484311848879,
+          0.0030278717167675495,
+          -0.01095631904900074,
+          0.019993282854557037,
+          0.01825164631009102,
+          0.008046185597777367,
+          -0.03200702369213104,
+          0.008819258771836758,
+          -0.007588561624288559,
+          -0.0007630769396200776,
+          -0.020419806241989136,
+          0.029821092262864113,
+          0.013435485772788525,
+          0.007975098676979542,
+          0.01055645290762186,
+          -0.006491152569651604,
+          0.006282333750277758,
+          -0.009685634635388851,
+          -0.002785730641335249,
+          -0.013319969177246094,
+          -0.029678918421268463,
+          -0.007593004498630762,
+          0.023760905489325523,
+          -0.006451165769249201,
+          0.01095631904900074,
+          -0.004316327627748251,
+          0.029270166531205177,
+          -0.015106036327779293,
+          0.000624234729912132,
+          0.02477389946579933,
+          -0.0022370261140167713,
+          0.0058646961115300655,
+          0.011533902958035469,
+          0.06394296139478683,
+          -0.003012321190908551,
+          0.013239996507763863,
+          0.0013817580183967948,
+          0.002856818027794361,
+          0.007948440499603748,
+          -0.015070492401719093,
+          -0.03190039470791817,
+          0.001449513016268611,
+          0.0043718647211790085,
+          -0.013435485772788525,
+          -0.0016072377329692245,
+          -0.0018793685594573617,
+          0.001987110124900937,
+          -0.004909461829811335,
+          0.018153900280594826,
+          -0.020935188978910446,
+          0.028737012296915054,
+          0.019851109012961388,
+          0.012884560041129589,
+          -0.010387620888650417,
+          0.015941310673952103,
+          0.025253737345337868,
+          -0.008121715858578682,
+          -0.0038498176727443933,
+          0.007401957642287016,
+          0.0053937435150146484,
+          0.023760905489325523,
+          0.0027079791761934757,
+          0.029981037601828575,
+          -0.0035343680065125227,
+          0.003134502563625574,
+          -0.014270761050283909,
+          -0.001956009538844228,
+          -0.0033988580107688904,
+          0.005904682911932468,
+          0.003414408303797245,
+          0.0014061941765248775,
+          -0.016403377056121826,
+          0.013648747466504574,
+          0.004176374524831772,
+          0.003367757424712181,
+          0.01456399541348219,
+          -0.0006292330217547715,
+          -0.002503603231161833,
+          0.013444371521472931,
+          -0.002197039546445012,
+          -0.0013151137391105294,
+          0.008143930695950985,
+          -0.012315861880779266,
+          -0.016332291066646576,
+          -0.008255003951489925,
+          0.000503719667904079,
+          -0.006344534922391176,
+          -0.01958453096449375,
+          -0.004847260192036629,
+          0.022232530638575554,
+          0.0022259189281612635,
+          0.031011803075671196,
+          -0.003445509122684598,
+          0.008277218788862228,
+          0.016980960965156555,
+          -0.011489473283290863,
+          -0.0005217691650614142,
+          0.009312426671385765,
+          -0.0034566165413707495,
+          -0.0016960968496277928,
+          -0.004984991624951363,
+          0.00048372638411819935,
+          -0.009001419879496098,
+          3.720972017617896e-05,
+          0.0167232695966959,
+          0.011036291718482971,
+          0.0025458112359046936,
+          -0.005913568660616875,
+          -0.0017027612775564194,
+          -0.011844909749925137,
+          -0.006979877129197121,
+          -0.004487381316721439,
+          0.0017105364240705967,
+          -0.0011201791930943727,
+          0.01182713732123375,
+          0.00040319786057807505,
+          0.010218788869678974,
+          -0.010982976295053959,
+          -0.0011357294861227274,
+          -0.006548910867422819,
+          0.020419806241989136,
+          0.0053937435150146484,
+          -0.0043496498838067055,
+          0.007566346786916256,
+          -0.011027405969798565,
+          0.021379483863711357,
+          0.005291555542498827,
+          -0.00226479466073215,
+          0.0039075762033462524,
+          -0.0017560767009854317,
+          0.01149835903197527,
+          -0.0010962983360514045,
+          -0.013239996507763863,
+          0.02891472913324833,
+          0.017638517543673515,
+          0.027404125779867172,
+          0.0017405262915417552,
+          0.005735850892961025,
+          -0.00888146087527275,
+          0.027510756626725197,
+          0.01585245132446289,
+          -0.0017427478451281786,
+          -0.028150541707873344,
+          0.004243019036948681,
+          -0.012067057192325592,
+          -0.001978224376216531,
+          0.009614547714591026,
+          -0.01095631904900074,
+          -0.009552346542477608,
+          -0.01607459969818592,
+          -0.03138501197099686,
+          -0.026782112196087837,
+          0.0038164956495165825,
+          0.007068736013025045,
+          -0.008312762714922428,
+          0.0008847027784213424,
+          -0.015301525592803955,
+          -0.009543460793793201,
+          -0.025591401383280754,
+          0.002565804636105895,
+          -0.007313098758459091,
+          -0.037356339395046234,
+          -0.02381422184407711,
+          -0.009330199100077152,
+          -0.007166481111198664,
+          0.006611112039536238,
+          -0.0036898714024573565,
+          -0.02461395226418972,
+          0.011729392223060131,
+          -0.00974783580750227,
+          0.011542788706719875,
+          0.017700720578432083,
+          0.007535246200859547,
+          -0.010547567158937454,
+          -0.013559889048337936,
+          -0.009481258690357208,
+          -0.01628786139190197,
+          0.02052643708884716,
+          -0.010360963642597198,
+          0.006215689703822136,
+          -0.0058646961115300655,
+          -0.0018138349987566471,
+          0.0014184123137965798,
+          0.01832273229956627,
+          -0.03796058148145676,
+          -0.027190864086151123,
+          -0.00651781028136611,
+          0.011382842436432838,
+          0.0004651214985642582,
+          -0.012600211426615715,
+          0.014261875301599503,
+          -0.006406736560165882,
+          -0.02031317539513111,
+          -0.014519566670060158,
+          -0.001221811748109758,
+          -0.011054064147174358,
+          0.00840162206441164,
+          -0.011569445952773094,
+          0.01555921696126461,
+          0.0035299251321703196,
+          0.005349313840270042,
+          -0.00651781028136611,
+          0.015967968851327896,
+          0.008921447210013866,
+          -0.021343940868973732,
+          -0.003747629700228572,
+          -0.009463487192988396,
+          0.0017660732846707106,
+          0.01622566021978855,
+          0.007966212928295135,
+          -0.019815564155578613,
+          0.01738082803785801,
+          0.02687097154557705,
+          0.01315113715827465,
+          0.02118399366736412,
+          0.002925683744251728,
+          0.00752636045217514,
+          -0.001412858604453504,
+          0.011062949895858765,
+          -0.004811716731637716,
+          0.0002893472264986485,
+          0.00702874967828393,
+          -0.01628786139190197,
+          -0.011604989878833294,
+          0.0359523668885231,
+          0.000578694452997297,
+          -0.021219536662101746,
+          0.018153900280594826,
+          -0.005753622390329838,
+          -0.009076950140297413,
+          0.005900240037590265,
+          -0.0020581972785294056,
+          0.002081522950902581,
+          -0.010156587697565556,
+          0.009854466654360294,
+          -0.007961769588291645,
+          0.0057003069669008255,
+          0.006557796616107225,
+          0.004651770461350679,
+          -0.016190115362405777,
+          0.01658109575510025,
+          0.029092447832226753,
+          0.02616010047495365,
+          -0.042616792023181915,
+          -0.006753286812454462,
+          0.005424844101071358,
+          -0.018944745883345604,
+          -0.013675405643880367,
+          0.026017924770712852,
+          -0.0022559086792171,
+          -0.011658305302262306,
+          0.014039727859199047,
+          -0.009472372941672802,
+          -0.015434814617037773,
+          -0.0014961640117689967,
+          0.023938624188303947,
+          0.008623769506812096,
+          0.006402293220162392,
+          -0.007939554750919342,
+          0.005220468156039715,
+          -0.021077362820506096,
+          0.006242347415536642,
+          -0.006095729768276215,
+          0.009481258690357208,
+          -0.005349313840270042,
+          -0.005487045273184776,
+          0.012946761213243008,
+          0.005442616064101458,
+          -0.004847260192036629,
+          0.018589310348033905,
+          -0.00038292689714580774,
+          0.001768294838257134,
+          -0.012422492727637291,
+          0.0201532281935215,
+          -0.000900253071449697,
+          -0.009952211752533913,
+          -0.005424844101071358,
+          0.0197267048060894,
+          0.012511352077126503,
+          -0.0030878514517098665,
+          -0.0029145763255655766,
+          0.006055743433535099,
+          -3.049322731385473e-05,
+          0.000538985594175756,
+          -0.003632113104686141,
+          -0.008348306640982628,
+          -0.004402965307235718,
+          -0.007779608480632305,
+          0.015461471863090992,
+          -0.0028101669158786535,
+          -0.012742385268211365,
+          0.011427272111177444,
+          -0.003472166834399104,
+          -0.00031905947253108025,
+          -0.016607753932476044,
+          0.0011729393154382706,
+          0.005815824028104544,
+          0.026017924770712852,
+          0.012760157696902752,
+          -0.014519566670060158,
+          -0.025164877995848656,
+          -0.014484022743999958,
+          0.03270012512803078,
+          0.014377391897141933,
+          -0.004289669916033745,
+          0.01694541797041893,
+          0.005913568660616875,
+          0.021059591323137283,
+          0.005020535551011562,
+          -0.003063415177166462,
+          0.012102600187063217,
+          0.028132770210504532,
+          0.00480283098295331,
+          -0.01577247865498066,
+          -0.0042830053716897964,
+          -0.0017694055568426847,
+          -0.014128586277365685,
+          -0.008930332958698273,
+          -0.006495595443993807,
+          -0.02438291907310486,
+          0.01620788872241974,
+          -0.013497686944901943,
+          0.013666519895195961,
+          -0.014555109664797783,
+          -0.004509596154093742,
+          -0.027261951938271523,
+          0.002392529509961605,
+          0.021450571715831757,
+          -0.0035899050999432802,
+          0.012689069844782352,
+          -0.02738635428249836,
+          -0.020846329629421234,
+          0.019371269270777702,
+          -0.019922195002436638,
+          -0.00027823986602015793,
+          -0.0022559086792171,
+          -0.0034988245461136103,
+          -0.0051449378952383995,
+          -0.0010701960418373346,
+          -0.005375971551984549,
+          -0.0002896249061450362,
+          0.021432798355817795,
+          -0.004505153279751539,
+          0.0051671527326107025,
+          0.00011628037464106455,
+          0.015168237499892712,
+          0.014119700528681278,
+          -0.006544467993080616,
+          -0.007726293057203293,
+          -0.019762249663472176,
+          0.0037720659747719765,
+          -0.02249910682439804,
+          0.022321389988064766,
+          0.005011649336665869,
+          0.016403377056121826,
+          -0.007153152488172054,
+          -0.004263012204319239,
+          -0.006388964597135782,
+          -0.01047648023813963,
+          0.018660398200154305,
+          -0.010325419716536999,
+          0.007059850264340639,
+          -0.008277218788862228,
+          0.03403301164507866,
+          0.005269340705126524,
+          -0.013355513103306293,
+          0.02454286627471447,
+          0.005264897830784321,
+          0.011054064147174358,
+          -0.016741042956709862,
+          0.009427943266928196,
+          0.01022767461836338,
+          -0.016829900443553925,
+          0.01296453271061182,
+          -0.02921685017645359,
+          0.006251233164221048,
+          -0.024720583111047745,
+          -0.013071163557469845,
+          -0.017505230382084846,
+          -0.01914023607969284,
+          -0.04243907332420349,
+          -0.001449513016268611,
+          0.030691910535097122,
+          -0.005420401226729155,
+          0.017505230382084846,
+          -0.009738950058817863,
+          0.011373956687748432,
+          0.00821501761674881,
+          -0.005100508686155081,
+          0.020864101126790047,
+          0.02790173701941967,
+          -0.02052643708884716,
+          -0.006357864011079073,
+          0.0006603337242268026,
+          0.008983648382127285,
+          -0.006073514930903912,
+          -0.02921685017645359,
+          0.0050694081000983715,
+          -0.002747965743765235,
+          0.012644641101360321,
+          -0.01702539063990116,
+          0.0062778908759355545,
+          0.0007464158697985113,
+          0.011427272111177444,
+          -0.008339419960975647,
+          -0.014252989552915096,
+          -0.0180472694337368,
+          0.0016783250030130148,
+          0.0069976490922272205,
+          -0.022232530638575554,
+          0.016039054840803146,
+          -0.023903081193566322,
+          0.0014472915790975094,
+          0.015941310673952103,
+          -0.01956675946712494,
+          -0.006993206217885017,
+          -0.005438172724097967,
+          0.036929816007614136,
+          -0.0023414355237036943,
+          -0.002870146883651614,
+          0.011951540596783161,
+          0.0082194609567523,
+          0.010494251735508442,
+          -0.002934569725766778,
+          0.0051449378952383995,
+          0.0026280060410499573,
+          -0.004336320795118809,
+          0.0017427478451281786,
+          0.02761738747358322,
+          0.002912354888394475,
+          0.02125508151948452,
+          -0.0032966702710837126,
+          -0.002921240869909525,
+          0.0019349055364727974,
+          -0.007170923985540867,
+          0.009810036979615688,
+          7.4350020440761e-05,
+          -0.004203032236546278,
+          -0.017327511683106422,
+          0.015159351751208305,
+          -0.015168237499892712,
+          0.010121043771505356,
+          0.020473120734095573,
+          0.01788732409477234,
+          0.006771058309823275,
+          -0.004878360778093338,
+          -0.010547567158937454,
+          0.014928317628800869,
+          0.017078706994652748,
+          -0.00018424366135150194,
+          -0.020579751580953598,
+          0.0010801926255226135,
+          -0.00869485642760992,
+          0.003989770542830229,
+          -0.0116849634796381,
+          0.009516802616417408,
+          0.008805930614471436,
+          -0.003201146610081196,
+          0.015425928868353367,
+          0.008779272437095642,
+          0.014706170186400414,
+          0.01321333833038807,
+          -0.03317996487021446,
+          -0.0030145428609102964,
+          -0.008459379896521568,
+          -0.0020393149461597204,
+          -0.006384521722793579,
+          -0.017558544874191284,
+          -0.0004920568899251521,
+          0.016927646473050117,
+          -0.005993541795760393,
+          -0.006086844019591808,
+          0.01942458562552929,
+          -0.01420855987817049,
+          0.013853123411536217,
+          0.006180145777761936,
+          0.016305632889270782,
+          -0.02017100155353546,
+          0.003740965388715267,
+          -0.0012007077457383275,
+          -0.0037654016632586718,
+          -0.004905018489807844,
+          0.015665847808122635,
+          0.006224575452506542,
+          0.02475612796843052,
+          -0.02651553601026535,
+          0.006060186307877302,
+          -0.007779608480632305,
+          -0.011542788706719875,
+          -0.018011726438999176,
+          0.011009634472429752,
+          0.00021798232046421617,
+          -0.015372613444924355,
+          -0.01315113715827465,
+          0.006851031444966793,
+          0.011525017209351063,
+          -0.0014006405835971236,
+          0.013195566833019257,
+          0.009685634635388851,
+          0.0018615968292579055,
+          0.009738950058817863,
+          0.008605997078120708,
+          -0.02242802083492279,
+          -0.008059514686465263,
+          0.016554437577724457,
+          -0.0029056903440505266,
+          -0.010014412924647331,
+          0.018873659893870354,
+          0.006548910867422819,
+          -0.011880452744662762,
+          0.006891018245369196,
+          0.01701650582253933,
+          0.007988426834344864,
+          0.006011313758790493,
+          -0.007126494776457548,
+          0.003241133177652955,
+          0.002292563207447529,
+          0.0004895577440038323,
+          -0.020615296438336372,
+          -0.008530467748641968,
+          0.008921447210013866,
+          -0.01092077512294054,
+          0.008570454083383083,
+          -0.005282669793814421,
+          -0.023352153599262238,
+          0.01570139080286026,
+          0.0024325160775333643,
+          -0.008268333040177822,
+          -0.008015085011720657,
+          0.008090615272521973,
+          0.004487381316721439,
+          -0.008281662128865719,
+          0.0009963319171220064,
+          -0.0007880685734562576,
+          -0.012360291555523872,
+          0.022676825523376465,
+          0.00800175592303276,
+          0.0010085500543937087,
+          -0.006428950931876898,
+          -0.009116937406361103,
+          -0.0133643988519907,
+          0.031438328325748444,
+          0.013355513103306293,
+          0.023067805916070938,
+          0.01201374176889658,
+          -0.0033699788618832827,
+          -0.0062556760385632515,
+          -0.002807945478707552,
+          0.01559476088732481,
+          0.012111486867070198,
+          -0.00020451462478376925,
+          0.010831916704773903,
+          0.003245576284825802,
+          0.0037298579700291157,
+          0.006033528596162796,
+          0.005238240119069815,
+          -0.003241133177652955,
+          0.008139487355947495,
+          0.0016305632889270782,
+          0.004811716731637716,
+          0.002556918654590845,
+          0.011729392223060131,
+          0.02541368454694748,
+          -0.006544467993080616,
+          -0.008086171932518482,
+          0.01029876247048378,
+          0.012609097175300121,
+          0.004651770461350679,
+          -0.01569250598549843,
+          0.0066022262908518314,
+          -0.011391728185117245,
+          0.011276211589574814,
+          -0.02877255529165268,
+          -0.006877689156681299,
+          0.009952211752533913,
+          -0.021148450672626495,
+          0.03163381665945053,
+          -0.007419729605317116,
+          -0.024596180766820908,
+          0.007766279857605696,
+          -0.00941905751824379,
+          -0.013302197679877281,
+          -0.018429363146424294,
+          0.020402034744620323,
+          0.017505230382084846,
+          0.020419806241989136,
+          -0.012466922402381897,
+          0.01862485334277153,
+          0.004851703066378832,
+          0.022587966173887253,
+          -0.0037542942445725203,
+          -0.014439593069255352,
+          -0.010574225336313248,
+          0.013391056098043919,
+          0.009614547714591026,
+          -0.007495259400457144,
+          0.029465656727552414,
+          0.014368506148457527,
+          0.005838038399815559,
+          -0.0013339963043108582,
+          -0.00737529993057251,
+          0.004143052268773317,
+          -0.002876811195164919,
+          0.004416293930262327,
+          -0.009161366149783134,
+          -0.02111290581524372,
+          -0.0008302765781991184,
+          0.017265310510993004,
+          0.008619326166808605,
+          0.024933844804763794,
+          -0.015106036327779293,
+          0.0081839170306921,
+          0.002747965743765235,
+          -0.01044093631207943,
+          -0.02095296047627926,
+          0.002816831460222602,
+          -0.010201017372310162,
+          0.023192208260297775,
+          0.007468601688742638,
+          -0.007490816526114941,
+          -0.012022627517580986,
+          0.004451837856322527,
+          0.0034943814389407635,
+          0.024454006925225258,
+          -0.005944669246673584,
+          -0.01029876247048378,
+          0.01420855987817049,
+          0.004776173271238804,
+          0.0015372612979263067,
+          0.004038643091917038,
+          0.013506573624908924,
+          0.00668219942599535,
+          -0.0011362849036231637,
+          -0.001868261257186532,
+          0.0037298579700291157,
+          0.007544131949543953,
+          -0.0010435383301228285,
+          0.0011285097571089864,
+          0.001358432462438941,
+          0.004580683074891567,
+          0.012475808151066303,
+          -0.004098623059689999,
+          0.0027768448926508427,
+          -0.0041230591014027596,
+          -0.00912582315504551,
+          0.0014894995838403702,
+          0.0005156600964255631,
+          -0.012626868672668934,
+          -0.0009224678506143391,
+          -0.00912582315504551,
+          -0.0020126572344452143,
+          0.01417301595211029,
+          0.03367757424712181,
+          -0.010538681410253048,
+          0.022321389988064766,
+          -0.011720506474375725,
+          0.02146834321320057,
+          0.013373284600675106,
+          0.005469273775815964,
+          -0.02228584513068199,
+          -0.015088263899087906,
+          0.01405749935656786,
+          -0.017940638586878777,
+          0.004209696780890226,
+          -0.0052337972447276115,
+          -0.03024761565029621,
+          0.013053392060101032,
+          -0.03170490264892578,
+          -0.0030478648841381073,
+          0.005287112668156624,
+          -0.014332962222397327,
+          -0.009259111247956753,
+          -0.0031789320055395365,
+          -0.002585797803476453,
+          -0.0027368583250790834,
+          0.0064200651831924915,
+          -0.028026139363646507,
+          -0.0133999427780509,
+          0.005540360696613789,
+          0.009996641427278519,
+          -0.01620788872241974,
+          0.010538681410253048,
+          0.004589569289237261,
+          -0.0012617983156815171,
+          0.00462066987529397,
+          -0.018153900280594826,
+          -0.007033192552626133,
+          -0.02512933500111103,
+          -0.003809831105172634,
+          -0.00803729984909296,
+          0.015417042188346386,
+          -0.00045679096365347505,
+          -0.005953555461019278,
+          -0.0009407950565218925,
+          -0.011187352240085602,
+          0.0030834085773676634,
+          0.01055645290762186,
+          0.002639113226905465,
+          -0.008170587942004204,
+          0.01723865233361721,
+          0.02278345637023449,
+          -0.01300007663667202,
+          -0.025662489235401154,
+          0.01921132393181324,
+          -0.01577247865498066,
+          0.021574974060058594,
+          0.0005645325290970504,
+          -0.001508382149040699,
+          -0.0150793781504035,
+          0.015150465071201324,
+          0.0010563117684796453,
+          0.01216480229049921,
+          -0.014795029535889626,
+          0.02825717255473137,
+          -0.0010374292032793164,
+          0.003809831105172634,
+          -0.0073664141818881035,
+          -8.213386536226608e-06,
+          0.00888146087527275,
+          0.0019904423970729113,
+          -0.010369849391281605,
+          0.020508665591478348,
+          -0.012182573787868023,
+          0.0045473612844944,
+          -0.008446050807833672,
+          0.00020104357099626213,
+          -0.008974762633442879,
+          0.0034543948713690042,
+          -0.0017838451312854886,
+          0.012573553249239922,
+          0.012546896003186703,
+          0.015274868346750736,
+          -0.010023298673331738,
+          -0.008948104456067085,
+          0.013782036490738392,
+          -0.02498716115951538,
+          0.0013828687369823456,
+          0.0006231239531189203,
+          0.011951540596783161,
+          -0.02118399366736412,
+          0.006237904075533152,
+          0.026391133666038513,
+          0.005318213254213333,
+          0.009836695156991482,
+          -0.010716399177908897,
+          -0.018002839758992195,
+          -0.007815152406692505,
+          0.007917339913547039,
+          -0.014457364566624165,
+          -0.02059752494096756,
+          -0.013702062889933586,
+          -0.00020479230443015695,
+          -0.0021315061021596193,
+          0.004918347578495741,
+          -0.013630975969135761,
+          -0.002365871798247099,
+          -0.01664329692721367,
+          -0.0024502878077328205,
+          -0.014039727859199047,
+          -0.007042078301310539,
+          0.008090615272521973,
+          0.02855929359793663,
+          0.008077286183834076,
+          0.02402748353779316,
+          0.01237806398421526,
+          0.020917417481541634,
+          0.017834007740020752,
+          -0.004567354451864958,
+          -0.005695864092558622,
+          -0.0002508879406377673,
+          0.018660398200154305,
+          -0.03321550786495209,
+          0.01438627764582634,
+          -0.0077174073085188866,
+          -0.004138609394431114,
+          -0.0054914881475269794,
+          -0.0019304625457152724,
+          0.010796372778713703,
+          0.003976441919803619,
+          -0.004167488776147366,
+          0.010929660871624947,
+          -0.024596180766820908,
+          -0.007135380525141954,
+          -0.00735752796754241,
+          -0.01723865233361721,
+          -0.0013362177414819598,
+          0.02409856952726841,
+          -0.005424844101071358,
+          0.002994549460709095,
+          0.006984320003539324,
+          0.008437165059149265,
+          0.016492236405611038,
+          0.015808021649718285,
+          -0.0016438921447843313,
+          -0.0057669514790177345,
+          0.005398186389356852,
+          -0.004887246992439032,
+          -0.03081631287932396,
+          -0.006899903994053602,
+          0.008379407227039337,
+          0.011729392223060131,
+          -0.025289282202720642,
+          -0.011116265319287777,
+          0.0017338618636131287,
+          0.01540815643966198,
+          0.02081078663468361,
+          -0.0039297910407185555,
+          -0.016083484515547752,
+          -0.004147495608776808,
+          -0.020579751580953598,
+          0.021806007251143456,
+          -0.012804586440324783,
+          0.0005670317332260311,
+          -0.009232454001903534,
+          0.00939240027219057,
+          -0.012564667500555515,
+          0.03317996487021446,
+          -0.014590653590857983,
+          0.002401415491476655,
+          -0.0010685299057513475,
+          0.011098493821918964,
+          0.00476728705689311,
+          0.024134114384651184,
+          -0.01722976751625538,
+          -0.003796502249315381,
+          0.028683695942163467,
+          -0.033624257892370224,
+          0.016696613281965256,
+          0.008743729442358017,
+          0.01084080245345831,
+          -0.009303540922701359,
+          -0.008743729442358017,
+          0.027955051511526108,
+          0.020793013274669647,
+          0.006535581778734922,
+          -0.001412858604453504,
+          -0.019264638423919678,
+          0.023689819499850273,
+          -0.0011829358991235495,
+          0.002683542901650071,
+          -0.002794616622850299,
+          0.013453258201479912,
+          -0.034886058419942856,
+          -0.01657220907509327,
+          0.005562575533986092,
+          -0.008663755841553211,
+          -0.01892697438597679,
+          0.0005864696577191353,
+          -0.0116494195535779,
+          -0.02132616750895977,
+          -0.0032655694521963596,
+          0.01965561881661415,
+          -0.002612455515190959,
+          0.003072301158681512,
+          0.006686642300337553,
+          0.004411851055920124,
+          -0.008605997078120708,
+          -0.024525092914700508,
+          -0.01099186297506094,
+          0.003709864802658558,
+          0.007348642218858004,
+          -0.013017848134040833,
+          -0.012982305139303207,
+          -0.01077860128134489,
+          -0.0030100997537374496,
+          0.013471029698848724,
+          0.002305892063304782,
+          0.02168160490691662,
+          -0.004474052228033543,
+          0.01435073371976614,
+          0.013906438834965229,
+          -0.006851031444966793,
+          -0.008685970678925514,
+          0.013248882256448269,
+          -0.0027457440737634897,
+          -0.007504145614802837,
+          -0.007384185679256916,
+          -0.0016205665888264775,
+          -0.006584454327821732,
+          0.002288120100274682,
+          0.01956675946712494,
+          -0.01970893330872059,
+          0.004820602480322123,
+          0.0017249759985134006,
+          -0.01212925836443901,
+          0.005629220046103001,
+          0.008308319374918938,
+          -0.0011884896084666252,
+          0.011196237988770008,
+          0.003774287411943078,
+          -0.002202593255788088,
+          0.00226479466073215,
+          -0.013071163557469845,
+          0.006575568579137325,
+          0.015417042188346386,
+          0.014190787449479103,
+          0.01077860128134489,
+          0.006366749759763479,
+          -0.034299585968256,
+          -0.006415622308850288,
+          -0.003118952037766576,
+          -0.02374313399195671,
+          0.0014106371672824025,
+          0.0026280060410499573,
+          0.020259859040379524,
+          -0.003076744033023715,
+          0.014937203377485275,
+          -0.005882468074560165,
+          0.0006458940915763378,
+          0.016821015626192093,
+          0.009907782077789307,
+          -0.03573910519480705,
+          -0.010663083754479885,
+          -0.01523043867200613,
+          -0.007979541085660458,
+          -0.015177123248577118,
+          -0.014377391897141933,
+          0.008077286183834076,
+          0.017291968688368797,
+          -0.013648747466504574,
+          -0.012200345285236835,
+          0.011711620725691319,
+          0.013888667337596416,
+          0.007370857056230307,
+          0.006144602317363024,
+          0.016314519569277763,
+          -0.0018127242801710963,
+          0.0006347867310978472,
+          0.00026338372845202684,
+          0.018873659893870354,
+          -0.013026734814047813,
+          -0.0023392140865325928,
+          -0.011418386362493038,
+          0.011676076799631119,
+          0.020864101126790047,
+          -0.01076971460133791,
+          0.014448478817939758,
+          -0.007059850264340639,
+          -0.006779944524168968,
+          -0.047379638999700546,
+          -0.01321333833038807,
+          -0.009436829946935177,
+          0.028132770210504532,
+          -0.004807273857295513,
+          -0.005833595525473356,
+          0.0055803474970161915,
+          0.019602302461862564,
+          0.0063001057133078575,
+          0.008166145533323288,
+          0.01818055845797062,
+          -0.00839273538440466,
+          -0.012804586440324783,
+          0.004160824231803417,
+          0.007859582081437111,
+          -0.00927688367664814,
+          0.0027546300552785397,
+          0.0099344402551651,
+          0.032753441482782364,
+          -0.0057669514790177345,
+          0.0075085884891450405,
+          -0.009108050726354122,
+          0.0022303618025034666,
+          -0.016758814454078674,
+          0.001700539723969996,
+          -0.005473716650158167,
+          -0.006815487984567881,
+          0.014439593069255352,
+          -0.021788235753774643,
+          -0.0014795028837397695,
+          -0.0028968045953661203,
+          0.008406064473092556,
+          -0.01318668108433485,
+          0.011045178398489952,
+          -0.0041230591014027596,
+          -0.004271897953003645,
+          0.014128586277365685,
+          -0.011915996670722961,
+          0.027581844478845596,
+          -0.002074858406558633,
+          0.009969983249902725,
+          -0.007970655336976051,
+          -0.012866788543760777,
+          -0.017807351425290108,
+          0.0025369254872202873,
+          -0.0021348383743315935,
+          -0.008534910157322884,
+          0.008850360289216042,
+          -0.022463563829660416,
+          0.0036543277092278004,
+          -0.004873917903751135,
+          0.0015339291421696544,
+          -0.02592906542122364,
+          0.00015036614786367863,
+          -0.002412522677332163,
+          0.011080721393227577,
+          -0.008810373023152351,
+          -0.013160022906959057,
+          0.01176493614912033,
+          -0.010734171606600285,
+          -0.013888667337596416,
+          -0.01318668108433485,
+          0.008410507813096046,
+          0.004038643091917038,
+          -0.002434737514704466,
+          0.014457364566624165,
+          -0.011009634472429752,
+          0.0008480484248138964,
+          -0.006184588652104139,
+          0.00924133975058794,
+          -0.00047900574281811714,
+          -0.016989847645163536,
+          -4.682454527937807e-05,
+          -0.0002667159424163401,
+          0.012440265156328678,
+          0.012982305139303207,
+          0.018447136506438255,
+          -0.008441608399152756,
+          0.0026168986223638058,
+          0.0034766097087413073,
+          -0.0037809519562870264,
+          0.011187352240085602,
+          0.005775837227702141,
+          -0.013888667337596416,
+          0.0013650968903675675,
+          -0.013684291392564774,
+          0.00854379590600729,
+          -0.00891256146132946,
+          0.010209903120994568,
+          0.009108050726354122,
+          -0.0024769455194473267,
+          -0.010343191213905811,
+          0.011729392223060131,
+          0.01657220907509327,
+          -0.012333634309470654,
+          0.00032655693939886987,
+          -0.007228682283312082,
+          0.025431456044316292,
+          0.016749927774071693,
+          0.010583111084997654,
+          -0.013355513103306293,
+          0.013826466165482998,
+          -0.00025310940691269934,
+          0.0037254150956869125,
+          0.00957011803984642,
+          -0.026408905163407326,
+          -0.02491607330739498,
+          0.003736522514373064,
+          -0.0069310045801103115,
+          -0.0060468572191894054,
+          0.023316610604524612,
+          -0.012138144113123417,
+          0.006184588652104139,
+          -0.011569445952773094,
+          0.028168313205242157,
+          0.01621677353978157,
+          -0.0009307984146289527,
+          -0.015834679827094078,
+          0.0002776844776235521,
+          0.01011215802282095,
+          -0.0013417714508250356,
+          -0.007766279857605696,
+          -0.0022681269329041243,
+          -0.012538010254502296,
+          0.004398522432893515,
+          -0.003783173393458128,
+          0.004776173271238804,
+          0.004402965307235718,
+          -0.012697956524789333,
+          -0.011311755515635014,
+          0.028221629559993744,
+          0.0008724846411496401,
+          0.012973419390618801,
+          -0.004971663001924753,
+          0.0017494122730568051,
+          0.005957998335361481,
+          0.02233916148543358,
+          -0.024276288226246834,
+          0.01905137673020363,
+          0.006739957723766565,
+          0.017771806567907333,
+          0.008903675712645054,
+          -0.00839717872440815,
+          0.01249358057975769,
+          -0.006157930940389633,
+          -0.007415286265313625,
+          -0.010280990041792393,
+          -0.0014750600093975663,
+          -0.0016050162957981229,
+          -0.0076285479590296745,
+          0.009579003788530827,
+          -0.0011290650581941009,
+          -0.004905018489807844,
+          0.01420855987817049,
+          -0.022890087217092514,
+          0.014723941683769226,
+          -0.0013239996042102575,
+          0.01124955341219902,
+          -0.0021781569812446833,
+          0.011125151067972183,
+          -0.02278345637023449,
+          0.0038675894029438496,
+          -0.01797618344426155,
+          -0.0012118150480091572,
+          -0.0040941801853477955,
+          -0.001176271471194923,
+          -0.0030323145911097527,
+          0.007979541085660458,
+          -0.034139640629291534,
+          -0.010067728348076344,
+          0.015488130040466785,
+          0.013933097012341022,
+          0.0026768783573061228,
+          0.013026734814047813,
+          0.006246790289878845,
+          0.0012362513225525618,
+          -0.00615348806604743,
+          -0.009294655174016953,
+          -0.020259859040379524,
+          -0.02219698764383793,
+          -0.013053392060101032,
+          -0.0022725698072463274,
+          -0.0008230567909777164,
+          -0.005242682993412018,
+          0.008286104537546635,
+          -0.018287189304828644,
+          -0.0007325316546484828,
+          -0.0005164931644685566,
+          0.0004479050694499165,
+          0.001358432462438941,
+          0.0018538215663284063,
+          0.016341175884008408,
+          0.008765943348407745,
+          -0.0042363544926047325,
+          -0.007370857056230307,
+          0.025182651355862617,
+          -0.0030345360282808542,
+          0.012147029861807823,
+          0.020028825849294662,
+          0.0025080463383346796,
+          -0.014688398689031601,
+          0.013835351914167404,
+          -0.008290547877550125,
+          0.018642624840140343,
+          -0.012333634309470654,
+          -0.025680260732769966,
+          -0.014857230708003044,
+          0.007610776461660862,
+          -0.011871566995978355,
+          -0.009596775285899639,
+          0.01365763321518898,
+          -0.002821274334564805,
+          0.0001021878924802877,
+          0.007135380525141954,
+          -0.005762508604675531,
+          0.0025369254872202873,
+          -0.03497491776943207,
+          -0.000194934502360411,
+          -0.03001658245921135,
+          -0.030567508190870285,
+          0.014670626260340214,
+          0.006855474319308996,
+          0.021077362820506096,
+          -0.006948776543140411,
+          0.013977525755763054,
+          -0.009099164977669716,
+          0.026053469628095627,
+          0.014946090057492256,
+          -0.0006936558638699353,
+          0.0031789320055395365,
+          -0.007250897120684385,
+          -0.006371192634105682,
+          0.027155321091413498,
+          -0.008081729523837566,
+          -0.0059313406236469746,
+          0.02475612796843052,
+          0.026657709851861,
+          0.01738082803785801,
+          -0.018944745883345604,
+          0.019460128620266914,
+          0.0013017848832532763,
+          0.01789620891213417,
+          -0.016199002042412758,
+          -0.00032850075513124466,
+          -0.015665847808122635,
+          -0.008161702193319798,
+          0.05530586466193199,
+          0.01723865233361721,
+          -0.0007297547999769449,
+          -0.0202243160456419,
+          0.015017176978290081,
+          -0.0029789991676807404,
+          -0.002028207527473569,
+          0.00981892365962267,
+          0.0302298441529274,
+          -0.00920579582452774,
+          -0.013799807988107204,
+          0.013293311931192875,
+          -0.005247125867754221,
+          0.026746569201350212,
+          -0.002260351786389947,
+          0.0020904086995869875,
+          -0.015221552923321724,
+          -0.013133365660905838,
+          0.0013395498972386122,
+          0.009756721556186676,
+          -0.007939554750919342,
+          -0.0011018520453944802,
+          -0.006944333668798208,
+          0.012769043445587158,
+          0.006815487984567881,
+          -0.020277632400393486,
+          -0.003112287726253271,
+          -0.030922943726181984,
+          -0.00480283098295331,
+          -0.035525843501091,
+          -0.005944669246673584,
+          -0.005269340705126524,
+          -0.014626197516918182,
+          -0.032611265778541565,
+          -0.02073969878256321,
+          -0.009996641427278519,
+          0.004660656210035086,
+          0.002425851533189416,
+          0.010041071102023125,
+          0.002474724082276225,
+          0.0005764730158261955,
+          0.0010274326195940375,
+          0.0041230591014027596,
+          0.01303562056273222,
+          0.001622788142412901,
+          0.011729392223060131,
+          0.02233916148543358,
+          -0.018198329955339432,
+          -0.033410996198654175,
+          -0.014484022743999958,
+          0.0050249784253537655,
+          0.005686978343874216,
+          -0.0018749256851151586,
+          0.008601554669439793,
+          0.0015561438631266356,
+          0.011231781914830208,
+          -0.014324076473712921,
+          -0.005020535551011562,
+          -0.009774493984878063,
+          -0.0011773821897804737,
+          0.0026191200595349073,
+          0.003896468784660101,
+          0.0026946503203362226,
+          0.0029412340372800827,
+          -0.011711620725691319,
+          0.01014770194888115,
+          -0.005380414426326752,
+          -0.0009291322785429657,
+          -0.0009024745668284595,
+          -0.012538010254502296,
+          0.004927233327180147,
+          -0.007935111410915852,
+          -0.00753968907520175,
+          -0.008534910157322884,
+          0.0034521734341979027,
+          -0.01600351184606552,
+          -0.009036963805556297,
+          0.01716756634414196,
+          -0.006424508057534695,
+          0.007757393643260002,
+          0.012831244617700577,
+          -0.0013650968903675675,
+          -0.004562911577522755,
+          0.019229095429182053,
+          -0.012742385268211365,
+          -0.01417301595211029,
+          -0.024454006925225258,
+          -0.006446722894906998,
+          0.0005456500221043825,
+          0.015452586114406586,
+          0.0042830053716897964,
+          -0.010903003625571728,
+          -0.0034810525830835104,
+          -0.002579133491963148,
+          0.003554361406713724,
+          -0.012324748560786247,
+          0.005966884084045887,
+          -0.02388530783355236,
+          0.015959082171320915,
+          -0.006677756551653147,
+          -0.009303540922701359,
+          0.018713712692260742,
+          0.04535365104675293,
+          -0.007606333587318659,
+          -0.0177629217505455,
+          0.009014748968183994,
+          0.022836772724986076,
+          0.022676825523376465,
+          -0.007086507976055145,
+          0.013053392060101032,
+          -0.02001105435192585,
+          -0.011880452744662762,
+          -0.005082736723124981,
+          -0.002314777811989188,
+          0.022516880184412003,
+          -0.03104734607040882,
+          -0.019246866926550865,
+          -0.007784051354974508,
+          -0.03241577744483948,
+          -0.006948776543140411,
+          0.028470434248447418,
+          0.004611784126609564,
+          -0.005620333831757307,
+          -0.004143052268773317,
+          -0.002716864924877882,
+          -0.01753188669681549,
+          0.020259859040379524,
+          0.08686859160661697,
+          -0.014635083265602589,
+          -0.008294991217553616,
+          -0.004131945315748453,
+          0.00635342113673687,
+          0.005784722976386547,
+          -0.005056079011410475,
+          0.06181034445762634,
+          -0.023121120408177376,
+          0.00033433211501687765,
+          0.003609898267313838,
+          0.015816908329725266,
+          -0.007170923985540867,
+          -0.0002644944761414081,
+          0.0012107043294236064,
+          -0.005953555461019278,
+          -0.010129929520189762,
+          -0.017105363309383392,
+          -0.0045295893214643,
+          -0.0017949525499716401,
+          -0.00701097771525383,
+          -0.01620788872241974,
+          0.019335726276040077,
+          -0.023849764838814735,
+          -0.006615554913878441,
+          -0.0023347712121903896,
+          -0.006513367407023907,
+          0.009188024327158928,
+          -0.019833335652947426,
+          0.002570247510448098,
+          -0.0011773821897804737,
+          0.005380414426326752,
+          -0.005158266983926296,
+          -0.005575904622673988,
+          -0.0028234957717359066,
+          0.0009313537739217281,
+          -0.0098988963291049,
+          0.01826941780745983,
+          0.013790922239422798,
+          0.009152480401098728,
+          0.010058842599391937,
+          -0.0055803474970161915,
+          0.023121120408177376,
+          -0.005784722976386547,
+          0.0057891663163900375,
+          -0.0067888302728533745,
+          0.0014161908766254783,
+          -0.004789501894265413,
+          0.01839382015168667,
+          -0.020633067935705185,
+          -0.013071163557469845,
+          -0.014875002205371857,
+          0.035170406103134155,
+          -0.02097073197364807,
+          0.014972747303545475,
+          0.0010307647753506899,
+          0.011356184259057045,
+          0.010485365986824036,
+          0.0013051170390099287,
+          0.01702539063990116,
+          -0.012093714438378811,
+          0.016776585951447487,
+          0.002699093194678426,
+          0.015363726764917374,
+          -0.008255003951489925,
+          0.012573553249239922,
+          0.00010274325904902071,
+          0.003809831105172634,
+          -0.008637098595499992,
+          0.0044696093536913395,
+          0.011676076799631119,
+          -0.005780280102044344,
+          0.03541921079158783,
+          -0.0013728720368817449,
+          -0.02226807363331318,
+          0.016741042956709862,
+          -0.015106036327779293,
+          -0.02082855813205242,
+          0.0011740500340238214,
+          -0.014652854762971401,
+          -0.0030278717167675495,
+          0.012315861880779266,
+          -0.028381574898958206,
+          -0.020988503471016884,
+          0.005247125867754221,
+          0.001098519773222506,
+          0.0016572210006415844,
+          0.0006486709462478757,
+          -0.014439593069255352,
+          -0.0018138349987566471,
+          -0.0030656366143375635,
+          0.007957326248288155,
+          -0.014777257107198238,
+          0.006437837146222591,
+          -0.026266731321811676,
+          -0.0031389454379677773,
+          -0.003132280893623829,
+          0.023689819499850273,
+          -0.002907912014052272,
+          0.010432050563395023,
+          -0.005993541795760393,
+          -0.0025502543430775404,
+          -0.016252316534519196,
+          -0.0033522071316838264,
+          -0.003258905140683055,
+          0.018518222495913506,
+          0.026746569201350212,
+          -0.0016761035658419132,
+          -0.0025235966313630342,
+          -6.3693878473714e-05,
+          0.015461471863090992,
+          -0.00858378317207098,
+          -0.01022767461836338,
+          0.01666106842458248,
+          0.014404049143195152,
+          -0.016554437577724457,
+          0.011862681247293949,
+          -0.0022125900723040104,
+          0.011658305302262306,
+          -0.012253660708665848,
+          0.016536666080355644,
+          0.020490894094109535,
+          -0.005282669793814421,
+          0.019193550571799278,
+          -0.01095631904900074,
+          0.015390384942293167,
+          0.018731484189629555,
+          0.006948776543140411,
+          0.014155244454741478,
+          0.012662412598729134,
+          -0.01180048007518053,
+          0.0005836928030475974,
+          -0.005758065264672041,
+          -0.005322656128555536,
+          0.005127166397869587,
+          -0.002612455515190959,
+          -0.019406812265515327,
+          0.0036121197044849396,
+          0.0012462479062378407,
+          0.008508252911269665,
+          0.0034899385645985603,
+          -0.00617126002907753,
+          -0.0008730400004424155,
+          0.013462143950164318,
+          -0.02095296047627926,
+          -0.006077957805246115,
+          0.023938624188303947,
+          -0.006015756633132696,
+          -0.005673649255186319,
+          0.0011407278943806887,
+          0.012093714438378811,
+          -0.01092077512294054,
+          -0.01651889458298683,
+          -0.020508665591478348,
+          0.02841711975634098,
+          0.004509596154093742,
+          0.00887701753526926,
+          0.006762172561138868,
+          -0.008610440418124199,
+          -0.002639113226905465,
+          -0.015950197353959084,
+          -0.001235140603967011,
+          0.018233874812722206,
+          0.01723865233361721,
+          0.011027405969798565,
+          -0.019780021160840988,
+          0.008317206054925919,
+          -0.008486038073897362,
+          3.512708644848317e-05,
+          0.014004183933138847,
+          0.01622566021978855,
+          0.0063223205506801605,
+          -0.0027079791761934757,
+          0.016989847645163536,
+          0.001412858604453504,
+          0.029678918421268463,
+          -0.0006458940915763378,
+          -0.01767406240105629,
+          0.0003993102873209864,
+          0.001091300044208765,
+          -0.01620788872241974,
+          -0.003932012245059013,
+          0.0024769455194473267,
+          0.038813624531030655,
+          -0.008139487355947495,
+          0.007148709148168564,
+          -0.003058972302824259,
+          -0.012422492727637291,
+          0.0025724689476191998,
+          -0.0015883552841842175,
+          -0.002055975841358304,
+          0.02177046425640583,
+          -0.017567431554198265,
+          0.012004856020212173,
+          -0.012591325677931309,
+          0.0056780921295285225,
+          0.0012740164529532194,
+          0.007761836517602205,
+          -0.011356184259057045,
+          -0.03067413903772831,
+          -0.013790922239422798,
+          0.02635558880865574,
+          0.018642624840140343,
+          -0.007135380525141954,
+          0.018145015463232994,
+          -0.014821686781942844,
+          -0.00941905751824379,
+          0.008303876966238022,
+          -0.010982976295053959,
+          0.0009352413471788168,
+          0.007233125623315573
+        ],
+        "keyphrases": [
+          "1 Billion Word Language Modeling Benchmark",
+          "Long Short-Term Memory (LSTM) layers",
+          "Mixture of Experts (MoE) models",
+          "computational efficiency",
+          "Machine Translation"
+        ],
+        "title": "5 Experiments 5.1 1 Billion Word Language Modeling Benchmark"
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "a004def7-e6ff-4a78-ad92-424e0eff3d92",
+      "properties": {
+        "page_content": "6 Conclusion\n\nThis work is the first to demonstrate major wins from conditional computation in deep networks.\n\nWe carefully identified the design considerations and challenges of conditional computing and addressed them with a combination of algorithmic and engineering solutions. While we focused on text, conditional computation may help in other domains as well, provided sufficiently large training sets. We look forward to seeing many novel implementations and applications of conditional computation in the years to come.\n\nAcknowledgments\n\nWe would like to thank all of the members of the Google Brain and Google Translate teams who helped us with this project, in particular Zhifeng Chen, Yonghui Wu, and Melvin Johnson. Thanks also to our anonymous ICLR reviewers for the helpful suggestions on making this paper better.\n\n2Reported perplexities relative to the tokenization used by both our models and GNMT.\n\nReferences\n\nMartín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Gordon Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda B. Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. CoRR, abs/1603.04467, 2016. URL http://arxiv.org/abs/1603.04467.\n\nRahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. CoRR, abs/1611.06194, 2016. URL http://arxiv.org/abs/1611.\n\n06194.\n\nA. Almahairi, N. Ballas, T. Cooijmans, Y. Zheng, H. Larochelle, and A. Courville. Dynamic Capacity Networks. ArXiv e-prints, November 2015.\n\nDario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan, Christopher Fougner, Tony Han, Awni Y. Hannun, Billy Jun, Patrick LeGresley, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595, 2015. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.\n\nEmmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297, 2015.\n\nYoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.\n\nCiprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling.\n\narXiv preprint arXiv:1312.3005, 2013.\n\nK. Cho and Y. Bengio. Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning. ArXiv e-prints, June 2014. Ronan Collobert, Samy Bengio, and Yoshua Bengio. A parallel mixture of SVMs for very large scale problems. Neural Computing, 2002.\n\nAndrew Davis and Itamar Arel. Low-rank approximations for conditional feedforward computation in deep neural networks. arXiv preprint arXiv:1312.4461, 2013. Marc Peter Deisenroth and Jun Wei Ng. Distributed Gaussian processes. In ICML, 2015.\n\nJohn Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization, 2010.\n\nNadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. Edinburgh's phrase-based machine translation systems for wmt-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation, 2014.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013. Ekaterina Garmash and Christof Monz. Ensemble learning for multi-source neural machine translation. In staff.science.uva.nl/c.monz, 2016.\n\nFelix A. Gers, Jürgen A. Schmidhuber, and Fred A. Cummins. Learning to forget: Continual prediction with lstm. Neural Computation, 2000.\n\nAudrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. Memory-efficient backpropagation through time. CoRR, abs/1606.03401, 2016. URL http://arxiv.org/ abs/1606.03401. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2015.\n\nGeoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 2012. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.\n\nSergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.\n\nRobert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computing, 1991.\n\nMelvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation.\n\nCoRR, abs/1611.04558, 2016. URL http://arxiv.org/abs/1611.04558.\n\nMichael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm.\n\nNeural Computing, 1994. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.\n\nDiederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.\n\nReinhard Kneser and Hermann. Ney. Improved backingoff for m-gram language modeling., 1995.\n\nAlex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.\n\nQuoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeffrey Dean, and Andrew Y. Ng. Building high-level features using large scale unsupervised learning. In ICML, 2012. Patrick Gallinari Ludovic Denoyer. Deep sequential neural network. arXiv preprint arXiv:1410.0510, 2014.\n\nMinh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attentionbased neural machine translation. EMNLP, 2015a.\n\nMinh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the rare word problem in neural machine translation. ACL, 2015b.\n\nCarl Edward Rasmussen and Zoubin Ghahramani. Infinite mixtures of Gaussian process experts.\n\nNIPS, 2002.\n\nHasim Sak, Andrew W Senior, and Françoise Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH, pp. 338–342, 2014.\n\nMike Schuster and Kaisuke Nakajima. Japanese and Korean voice search. ICASSP, 2012.\n\nBabak Shahbaba and Radford Neal. Nonlinear models using dirichlet process mixtures. JMLR, 2009.\n\nIlya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks.\n\nIn NIPS, 2014.\n\nLucas Theis and Matthias Bethge. Generative image modeling using spatial LSTMs. In NIPS, 2015. Volker Tresp. Mixtures of Gaussian Processes. In NIPS, 2001.\n\nYonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.\n\nBangpeng Yao, Dirk Walther, Diane Beck, and Li Fei-fei. Hierarchical mixture of classification experts uncovers interactions between brain regions. In NIPS. 2009.\n\nWojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization.\n\narXiv preprint arXiv:1409.2329, 2014.\n\nJie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with fast-forward connections for neural machine translation. arXiv preprint arXiv:1606.04199, 2016.\n\n",
+        "embedding": [
+          0.012222016230225563,
+          0.0018153652781620622,
+          -0.018968041986227036,
+          -0.007018175441771746,
+          0.012403449974954128,
+          0.003484346903860569,
+          0.006766642909497023,
+          0.03404351323843002,
+          -0.030365360900759697,
+          0.03598979860544205,
+          0.007698550820350647,
+          -0.006844989024102688,
+          0.017252670601010323,
+          -0.018192825838923454,
+          0.0035915574990212917,
+          0.014209537766873837,
+          0.014687862247228622,
+          0.004014214966446161,
+          -0.011182897724211216,
+          -0.040674079209566116,
+          -0.03125603497028351,
+          -0.017648525536060333,
+          -0.03161890059709549,
+          -0.009582984261214733,
+          0.016172317788004875,
+          0.03310335800051689,
+          0.022712169215083122,
+          0.017153708264231682,
+          -0.045556288212537766,
+          0.015842437744140625,
+          0.028155172243714333,
+          0.003665780182927847,
+          -0.02747892029583454,
+          -0.023569853976368904,
+          -0.015380607917904854,
+          -0.03267451375722885,
+          0.06508512794971466,
+          0.03876078128814697,
+          -0.014654873870313168,
+          0.019149474799633026,
+          0.04321414604783058,
+          -0.011900384910404682,
+          -0.007348054554313421,
+          -0.015990883111953735,
+          -0.004210080485790968,
+          -0.002177201444283128,
+          -0.014085832983255386,
+          -0.015174433588981628,
+          -0.030051976442337036,
+          -0.007912972010672092,
+          0.028105689212679863,
+          -0.028353098779916763,
+          0.04743659868836403,
+          -0.003750311676412821,
+          -0.016485702246427536,
+          0.04671086370944977,
+          0.010762302204966545,
+          0.008997449651360512,
+          0.06310585141181946,
+          0.04139981418848038,
+          0.003208073088899255,
+          0.018192825838923454,
+          -0.023899732157588005,
+          -0.006424393504858017,
+          0.0322621650993824,
+          -0.02089783363044262,
+          0.0017978404648602009,
+          0.025433670729398727,
+          0.039783407002687454,
+          -0.015347619540989399,
+          -0.03783712163567543,
+          0.029045844450592995,
+          -0.010498398914933205,
+          -0.015957895666360855,
+          -0.00029766425723209977,
+          -0.0036699038464576006,
+          0.029029350727796555,
+          0.05367131158709526,
+          0.005014160647988319,
+          0.004304920788854361,
+          0.012782811187207699,
+          0.017219683155417442,
+          -0.004572947509586811,
+          -0.006494492758065462,
+          0.023157505318522453,
+          0.002239053603261709,
+          -0.005183223634958267,
+          -0.05383625254034996,
+          0.03328479081392288,
+          -0.04740361124277115,
+          -0.039816394448280334,
+          0.018225813284516335,
+          0.006473875138908625,
+          -0.01018501352518797,
+          -0.009624219499528408,
+          0.029656121507287025,
+          -0.00020127772586420178,
+          -0.000498168810736388,
+          -0.018918560817837715,
+          0.014357983134686947,
+          -0.023784274235367775,
+          0.00528631079941988,
+          0.013285876251757145,
+          -0.013665237464010715,
+          0.027577882632613182,
+          0.009541749954223633,
+          -0.013541532680392265,
+          0.004121425561606884,
+          0.01654343120753765,
+          0.09295989573001862,
+          -0.0011030328460037708,
+          -0.022530734539031982,
+          -0.05314350500702858,
+          0.03267451375722885,
+          -0.02602745220065117,
+          -0.019512342289090157,
+          -0.051791001111269,
+          -0.039882369339466095,
+          0.0015978513984009624,
+          -0.00780988484621048,
+          0.03721034899353981,
+          0.03533003851771355,
+          0.034241437911987305,
+          -0.011974607594311237,
+          -0.029342735186219215,
+          0.055914487689733505,
+          -0.002408116590231657,
+          -0.0425214022397995,
+          -0.0034740380942821503,
+          -0.023965708911418915,
+          -0.007587216794490814,
+          0.008593347854912281,
+          -0.040575116872787476,
+          0.003964733332395554,
+          0.03285594657063484,
+          -0.018638163805007935,
+          0.029342735186219215,
+          0.03582485765218735,
+          -0.025417177006602287,
+          -0.0004953339230269194,
+          0.011232379823923111,
+          -0.0033709509298205376,
+          0.016733111813664436,
+          -0.01296424400061369,
+          -0.034208450466394424,
+          -0.0008540773415006697,
+          -0.015990883111953735,
+          0.0541001558303833,
+          -0.009459280408918858,
+          0.030942648649215698,
+          -0.03885974362492561,
+          -0.0009102598414756358,
+          -0.014267266727983952,
+          0.034538328647613525,
+          0.029606638476252556,
+          -0.009046930819749832,
+          0.0019669034518301487,
+          0.009615972638130188,
+          -0.009541749954223633,
+          0.010201508179306984,
+          -0.00514198886230588,
+          -0.03315283730626106,
+          -0.007966578006744385,
+          0.03757321834564209,
+          0.009756171144545078,
+          0.04097096994519234,
+          0.04737062379717827,
+          0.008750040084123611,
+          -0.04882209002971649,
+          0.04700775444507599,
+          0.034439366310834885,
+          0.01601562462747097,
+          0.001114372513256967,
+          -0.018291790038347244,
+          0.01057262159883976,
+          0.01364049594849348,
+          0.033878572285175323,
+          0.009921110235154629,
+          -0.040179260075092316,
+          0.024790406227111816,
+          -0.0025256359949707985,
+          0.010119037702679634,
+          0.04885507747530937,
+          0.02596147730946541,
+          0.040179260075092316,
+          0.003092615632340312,
+          0.029161302372813225,
+          0.028204653412103653,
+          0.029260266572237015,
+          -0.007599587086588144,
+          0.008585100993514061,
+          0.02132667601108551,
+          0.045655250549316406,
+          -0.034769244492053986,
+          0.029903531074523926,
+          -0.039354562759399414,
+          0.0009916987037286162,
+          0.028649989515542984,
+          -0.024526502937078476,
+          0.004700775723904371,
+          0.00834593828767538,
+          -0.055947475135326385,
+          -0.009632466360926628,
+          0.005344039760529995,
+          0.00844490248709917,
+          0.007591340225189924,
+          -0.012667353264987469,
+          0.004255439154803753,
+          0.0053234221413731575,
+          -0.040674079209566116,
+          -0.006354294251650572,
+          0.004333785269409418,
+          -0.003735879436135292,
+          -0.017829960212111473,
+          0.012560142204165459,
+          -0.004383266903460026,
+          -0.003599804360419512,
+          -0.005364656914025545,
+          -0.025631597265601158,
+          -0.009005696512758732,
+          -0.01664239540696144,
+          0.020403016358613968,
+          0.04215853661298752,
+          -0.024856381118297577,
+          -0.016320763155817986,
+          -0.017928922548890114,
+          -0.03556095436215401,
+          -0.030117951333522797,
+          0.017566056922078133,
+          0.00014135829405859113,
+          0.03688047081232071,
+          0.02172253094613552,
+          -0.0031750851776450872,
+          0.004581194370985031,
+          0.02591199427843094,
+          -0.017269164323806763,
+          -0.0021524603944271803,
+          0.004577070940285921,
+          -0.007756279781460762,
+          0.0136487428098917,
+          -0.003177147125825286,
+          0.01680733449757099,
+          -0.00515848258510232,
+          -0.00132157769985497,
+          -0.001624654047191143,
+          -0.01583419181406498,
+          -0.017599044367671013,
+          0.01364049594849348,
+          -0.0008695403812453151,
+          0.010069556534290314,
+          -0.018935054540634155,
+          -0.04262036457657814,
+          0.013896152377128601,
+          -0.010003580711781979,
+          -0.05927925556898117,
+          0.06967044621706009,
+          0.02676968090236187,
+          -0.014621886424720287,
+          -0.00333796301856637,
+          -0.050471484661102295,
+          0.05802571401000023,
+          0.004453366156667471,
+          -0.02221735008060932,
+          0.05353936180472374,
+          -0.007117139175534248,
+          0.03829894959926605,
+          0.05241777002811432,
+          0.022827625274658203,
+          0.010044815018773079,
+          -0.02714904025197029,
+          0.05268167331814766,
+          -0.04205957055091858,
+          -0.00666767917573452,
+          -0.034670282155275345,
+          0.016881557181477547,
+          -0.04278530552983284,
+          -0.004498724825680256,
+          0.018324777483940125,
+          0.010877760127186775,
+          0.03592382371425629,
+          0.023619335144758224,
+          0.015908414497971535,
+          -0.03145396336913109,
+          -0.004395637661218643,
+          -0.00642026960849762,
+          -0.013879658654332161,
+          -0.03737528994679451,
+          -0.0037626822013407946,
+          -0.06696543842554092,
+          -0.004630676470696926,
+          -0.014811567030847073,
+          0.01346730999648571,
+          -0.05264868587255478,
+          0.029458193108439445,
+          0.018291790038347244,
+          0.020419510081410408,
+          0.007104768883436918,
+          -0.024180129170417786,
+          0.019281426444649696,
+          0.011446801014244556,
+          -0.017186695709824562,
+          -0.040113285183906555,
+          0.004412131384015083,
+          -0.01324464101344347,
+          -0.022745156660676003,
+          0.0018205196829512715,
+          0.019842220470309258,
+          0.029606638476252556,
+          0.005950192455202341,
+          0.004560577217489481,
+          -0.0032225053291767836,
+          0.017516573891043663,
+          0.03180033341050148,
+          -0.019693775102496147,
+          -0.03373012691736221,
+          0.040014322847127914,
+          0.025747055187821388,
+          0.003550322726368904,
+          -0.017120718955993652,
+          -0.03123953938484192,
+          0.051164232194423676,
+          -0.0006412022630684078,
+          -0.06333676725625992,
+          0.012329227291047573,
+          -0.006073896773159504,
+          0.002081330167129636,
+          -0.0012401387793943286,
+          0.05159307271242142,
+          0.002733872039243579,
+          0.014935270883142948,
+          0.0019091746071353555,
+          -0.050471484661102295,
+          -0.018522705882787704,
+          -0.007360424846410751,
+          -0.023091528564691544,
+          -0.039156634360551834,
+          -0.008061417378485203,
+          0.03816699981689453,
+          -0.006135749164968729,
+          -0.017401115968823433,
+          -0.03602278605103493,
+          -0.02089783363044262,
+          -0.009211870841681957,
+          -0.001520535908639431,
+          0.004915196914225817,
+          -0.030563287436962128,
+          -0.01624654047191143,
+          -0.0009813900105655193,
+          0.0069480761885643005,
+          0.001200965722091496,
+          0.0013545656111091375,
+          0.015339372679591179,
+          0.005059518851339817,
+          0.006865606643259525,
+          0.010613856837153435,
+          -0.018902065232396126,
+          -0.02114524319767952,
+          0.012180781923234463,
+          -0.007558352313935757,
+          0.012222016230225563,
+          -0.007113015744835138,
+          -0.0006963539053685963,
+          -0.029870541766285896,
+          0.007587216794490814,
+          -0.024411045014858246,
+          -0.03589083254337311,
+          -0.023701805621385574,
+          -0.0026101674884557724,
+          0.023157505318522453,
+          0.001592696993611753,
+          -0.003183332271873951,
+          0.03905767202377319,
+          -0.023982202634215355,
+          -0.011438554152846336,
+          -0.0059172045439481735,
+          -0.010754055343568325,
+          0.03849687799811363,
+          0.0035400139167904854,
+          -0.011158157140016556,
+          0.017928922548890114,
+          -0.025730561465024948,
+          -0.00416884571313858,
+          0.05838857963681221,
+          0.0018669088603928685,
+          0.0066429381258785725,
+          0.02089783363044262,
+          -0.0025297594256699085,
+          -0.0050224075093865395,
+          -0.02139265276491642,
+          0.017945416271686554,
+          0.0020297865848988295,
+          -0.040212247520685196,
+          -0.023239973932504654,
+          -0.039849381893873215,
+          -0.006725407671183348,
+          0.028715966269373894,
+          0.024724429473280907,
+          0.004436872433871031,
+          0.015199174173176289,
+          -0.005067766178399324,
+          -0.002641093684360385,
+          -0.018093861639499664,
+          0.003655471606180072,
+          0.009756171144545078,
+          -0.01081178430467844,
+          0.0016401171451434493,
+          -0.003805978922173381,
+          -0.0052945576608181,
+          0.04126786068081856,
+          0.004010091535747051,
+          -0.009079919196665287,
+          -0.018869077786803246,
+          -0.055287718772888184,
+          -0.034670282155275345,
+          0.034538328647613525,
+          0.014630133286118507,
+          -0.010622103698551655,
+          -0.01332711149007082,
+          0.0016164070693776011,
+          -0.04354402795433998,
+          -0.01296424400061369,
+          -0.009838640689849854,
+          0.02579653635621071,
+          0.014861048199236393,
+          -0.0028287123423069715,
+          0.020551461726427078,
+          0.022959576919674873,
+          0.013558026403188705,
+          -0.007141880225390196,
+          -0.03278997167944908,
+          0.011372578330338001,
+          0.006919211708009243,
+          0.04591915383934975,
+          0.008626335300505161,
+          -0.0047914921306073666,
+          -0.04921794682741165,
+          -0.024575984105467796,
+          0.018159838393330574,
+          -0.03819998726248741,
+          0.001192718744277954,
+          0.013541532680392265,
+          -0.01623004674911499,
+          0.0058677224442362785,
+          0.0003208588750567287,
+          -0.02205241098999977,
+          -0.0161310825496912,
+          -0.004461613483726978,
+          -0.034208450466394424,
+          -0.009929358027875423,
+          -0.04773348942399025,
+          -0.06749323755502701,
+          0.02655525878071785,
+          -0.040278226137161255,
+          -0.0648542121052742,
+          0.03592382371425629,
+          -0.03655059263110161,
+          0.006510986480861902,
+          -0.040278226137161255,
+          0.0021019477862864733,
+          -0.060268890112638474,
+          0.017961911857128143,
+          0.034175463020801544,
+          0.015710486099123955,
+          -0.0021132873371243477,
+          0.022398782894015312,
+          -0.02736346237361431,
+          -0.040443163365125656,
+          -0.06752622872591019,
+          -0.017780477181077003,
+          -0.01351679116487503,
+          -0.0031503443606197834,
+          0.002323585096746683,
+          0.05284661427140236,
+          0.0004986842395737767,
+          0.024674948304891586,
+          0.022547228261828423,
+          -0.0012978676240891218,
+          -0.04885507747530937,
+          -0.024724429473280907,
+          0.011611740104854107,
+          0.03876078128814697,
+          -0.030530299991369247,
+          0.051395148038864136,
+          -0.05258271098136902,
+          0.009665454737842083,
+          -0.009409798309206963,
+          0.006036785431206226,
+          0.019116487354040146,
+          0.007092398125678301,
+          -0.02203591726720333,
+          0.003894633846357465,
+          0.023404914885759354,
+          -0.011595246382057667,
+          0.03585784509778023,
+          -0.012692093849182129,
+          -0.034538328647613525,
+          -0.009442785754799843,
+          -0.007661439478397369,
+          0.03750723972916603,
+          0.011182897724211216,
+          -0.016172317788004875,
+          -0.028798434883356094,
+          0.011784926988184452,
+          0.02111225575208664,
+          0.03866181895136833,
+          -0.02229982055723667,
+          -0.022761650383472443,
+          0.009393304586410522,
+          -0.035692907869815826,
+          0.018027886748313904,
+          -0.017170201987028122,
+          0.005900710355490446,
+          -0.009442785754799843,
+          -0.045226410031318665,
+          -0.0008643860346637666,
+          -0.00010920798376901075,
+          -0.015891920775175095,
+          0.01045716367661953,
+          0.0061604902148246765,
+          0.018638163805007935,
+          0.04113591089844704,
+          -0.02221735008060932,
+          -0.002874070778489113,
+          0.0772576555609703,
+          -0.015537300147116184,
+          0.018819596618413925,
+          0.01659291237592697,
+          -0.01568574644625187,
+          0.02652227133512497,
+          -0.025829525664448738,
+          -0.028419075533747673,
+          0.017467092722654343,
+          -0.01558678224682808,
+          -0.030398348346352577,
+          -0.007467635441571474,
+          -0.003577125258743763,
+          0.010910747572779655,
+          0.02193695306777954,
+          -0.03293841704726219,
+          -0.004898703191429377,
+          -0.019825726747512817,
+          0.03311984986066818,
+          -0.03366415202617645,
+          0.007327436935156584,
+          9.535564458929002e-05,
+          -0.010259236209094524,
+          0.040542129427194595,
+          -0.017763983458280563,
+          -0.006127502303570509,
+          0.006292441859841347,
+          -0.010020074434578419,
+          -0.003263740334659815,
+          0.008411914110183716,
+          0.01359101478010416,
+          -0.02037002705037594,
+          0.0034328033216297626,
+          -0.0009437631815671921,
+          -0.0008262437768280506,
+          -0.004927567206323147,
+          -0.03671552985906601,
+          -0.014687862247228622,
+          0.006238836329430342,
+          -0.023800769820809364,
+          -0.0015256903134286404,
+          0.004939937964081764,
+          -0.03876078128814697,
+          0.0065893325954675674,
+          0.02144213393330574,
+          -0.0028431445825845003,
+          0.006045032292604446,
+          -0.0032431227155029774,
+          -0.005500731989741325,
+          -0.0009633497102186084,
+          -0.0018246431136503816,
+          0.017994899302721024,
+          0.04644696041941643,
+          0.008593347854912281,
+          0.011479788459837437,
+          0.007508870679885149,
+          -0.012469425797462463,
+          0.017516573891043663,
+          -0.022547228261828423,
+          0.007009928580373526,
+          0.03754022717475891,
+          0.02627486176788807,
+          0.011826162226498127,
+          0.003166838316246867,
+          0.0007674840744584799,
+          0.018951548263430595,
+          -0.034208450466394424,
+          0.024872876703739166,
+          0.014234278351068497,
+          -0.000996337621472776,
+          -0.008865498006343842,
+          -0.01110042817890644,
+          2.0150322598055936e-05,
+          0.017945416271686554,
+          0.008939720690250397,
+          -0.004106993321329355,
+          -0.03536302596330643,
+          0.002777168760076165,
+          0.018489716574549675,
+          0.010119037702679634,
+          -0.010894253849983215,
+          -0.024064673110842705,
+          -0.024180129170417786,
+          9.25851782085374e-05,
+          0.02180500142276287,
+          0.014432205818593502,
+          0.019231945276260376,
+          0.0269676074385643,
+          0.029260266572237015,
+          0.02693461999297142,
+          0.008733546361327171,
+          0.012114806100726128,
+          0.011521023698151112,
+          0.006424393504858017,
+          -0.027858281508088112,
+          -0.007657316047698259,
+          0.025466658174991608,
+          0.019594812765717506,
+          0.002787477569654584,
+          0.025450164452195168,
+          0.00653572753071785,
+          -0.005187347065657377,
+          0.020534968003630638,
+          -0.0012937441933900118,
+          -0.0324271060526371,
+          -0.0074758827686309814,
+          -0.015265149995684624,
+          0.008428407832980156,
+          -0.003253431525081396,
+          -0.0032266287598758936,
+          -0.017285659909248352,
+          0.014209537766873837,
+          0.012857033871114254,
+          0.0014988876646384597,
+          0.00264315539970994,
+          -0.0010659215040504932,
+          -0.015017741359770298,
+          -0.013788941316306591,
+          0.00541001558303833,
+          -0.013887905515730381,
+          0.056013453751802444,
+          -0.0214256402105093,
+          -0.016469208523631096,
+          -0.017648525536060333,
+          -0.006787260062992573,
+          0.017747489735484123,
+          0.01654343120753765,
+          0.014102326706051826,
+          0.018308283761143684,
+          0.00041441048961132765,
+          0.04370896518230438,
+          0.00416884571313858,
+          -0.03625370189547539,
+          0.002042157109826803,
+          -0.01590016670525074,
+          0.011512776836752892,
+          0.017302153632044792,
+          0.0042471918277442455,
+          0.002052465919405222,
+          0.01359926164150238,
+          -0.006432640366256237,
+          0.023239973932504654,
+          -0.008758286945521832,
+          -0.0023792521096765995,
+          0.008980955928564072,
+          0.004750257357954979,
+          0.012733329087495804,
+          0.005174976773560047,
+          -0.028748953714966774,
+          -0.014019857160747051,
+          0.01084477175027132,
+          0.005038901697844267,
+          -0.011281860992312431,
+          0.030480818822979927,
+          0.01591666042804718,
+          0.018176332116127014,
+          0.004350278992205858,
+          -0.010077803395688534,
+          -0.012287992052733898,
+          0.0014751775888726115,
+          0.008003689348697662,
+          0.023767780512571335,
+          0.019594812765717506,
+          -0.013162171468138695,
+          0.0056244367733597755,
+          -0.04189463332295418,
+          0.012518907897174358,
+          0.009112906642258167,
+          -0.015875427052378654,
+          -0.0036719655618071556,
+          0.01636199839413166,
+          0.0045440830290317535,
+          0.00045848023728467524,
+          -0.008791275322437286,
+          0.039915356785058975,
+          0.024806899949908257,
+          0.008914980106055737,
+          -0.0063955290243029594,
+          -0.014753838069736958,
+          -0.002406054874882102,
+          0.016634147614240646,
+          0.028946882113814354,
+          0.011570505797863007,
+          0.014061091467738152,
+          0.010795289650559425,
+          -0.023338938131928444,
+          0.012230263091623783,
+          0.028072701767086983,
+          0.044929519295692444,
+          0.016485702246427536,
+          0.03340024873614311,
+          0.027858281508088112,
+          0.0027379957027733326,
+          0.009327328763902187,
+          -0.02068341337144375,
+          -0.014778578653931618,
+          0.003181270556524396,
+          -0.006197601556777954,
+          0.04684281721711159,
+          -0.0071872384287416935,
+          -0.05287960171699524,
+          -0.016502195969223976,
+          0.0025668710004538298,
+          0.02164006233215332,
+          0.020007161423563957,
+          0.004424502141773701,
+          0.0006118224118836224,
+          0.02221735008060932,
+          -0.012065324001014233,
+          -0.013986868783831596,
+          0.011578752659261227,
+          0.0071789915673434734,
+          -0.018588680773973465,
+          0.013945634476840496,
+          0.019067006185650826,
+          -0.003026639809831977,
+          -0.009937604889273643,
+          -0.0036080514546483755,
+          0.024592477828264236,
+          0.006725407671183348,
+          -2.0440254957065918e-05,
+          0.0013968313578516245,
+          0.009269599802792072,
+          0.0027627365197986364,
+          0.04176267981529236,
+          -0.005789376329630613,
+          0.009764418005943298,
+          -0.012271498329937458,
+          -0.004201833624392748,
+          0.03377960994839668,
+          -0.03509912267327309,
+          -0.015496065840125084,
+          0.019248438999056816,
+          0.013665237464010715,
+          0.002488524653017521,
+          0.0017256794963032007,
+          -0.004972925875335932,
+          -0.012213769368827343,
+          0.0316024087369442,
+          -0.0056120664812624454,
+          -0.006605826783925295,
+          0.039651453495025635,
+          -0.006985187530517578,
+          -0.005702782887965441,
+          0.028204653412103653,
+          -0.0015782647533342242,
+          -0.03241061046719551,
+          0.014226031489670277,
+          -0.008552112616598606,
+          0.03552796691656113,
+          0.015792956575751305,
+          -0.012172535061836243,
+          -0.011554011143743992,
+          0.013929140754044056,
+          -0.009879875928163528,
+          -0.023075034841895103,
+          0.0029586022719740868,
+          -0.022580217570066452,
+          0.013986868783831596,
+          -0.009797406382858753,
+          0.022365795448422432,
+          -0.006737778428941965,
+          0.017615538090467453,
+          0.018407247960567474,
+          0.008440778590738773,
+          -0.017417611554265022,
+          0.011306602507829666,
+          -0.02121121808886528,
+          -0.05462796241044998,
+          0.010135532356798649,
+          -0.002533883089199662,
+          0.017153708264231682,
+          -0.011158157140016556,
+          -0.009541749954223633,
+          0.003568878397345543,
+          -0.008659323677420616,
+          -0.05185697600245476,
+          0.0067006670869886875,
+          -0.023718299344182014,
+          0.014308501034975052,
+          -0.02104627899825573,
+          -0.027957243844866753,
+          -0.015776462852954865,
+          -0.04361000284552574,
+          -0.014762084931135178,
+          -0.0130714550614357,
+          0.015289891511201859,
+          0.016238292679190636,
+          -0.011677715927362442,
+          -0.018275296315550804,
+          0.030744722113013268,
+          0.012436438351869583,
+          -0.006754272151738405,
+          -0.016972273588180542,
+          0.017120718955993652,
+          -0.03117356449365616,
+          -0.0440058559179306,
+          -0.013846670277416706,
+          0.02079886943101883,
+          0.007970700971782207,
+          -0.02083185873925686,
+          -0.028353098779916763,
+          -0.005772882141172886,
+          -0.012131299823522568,
+          -0.002649340545758605,
+          0.03356518596410751,
+          0.017335141077637672,
+          0.01619705744087696,
+          0.029227279126644135,
+          -0.03336726129055023,
+          0.03142097219824791,
+          -0.015570288524031639,
+          0.023982202634215355,
+          0.0035894957836717367,
+          -0.011916878633201122,
+          -0.006515109911561012,
+          0.04403884336352348,
+          -0.012716835364699364,
+          0.018110357224941254,
+          -0.022266831248998642,
+          -0.0028967498801648617,
+          0.006172860506922007,
+          0.02137615904211998,
+          -0.006915088277310133,
+          -0.007273831870406866,
+          0.00403689406812191,
+          -0.024196624755859375,
+          0.022547228261828423,
+          0.04295024275779724,
+          0.00791709590703249,
+          0.01558678224682808,
+          -0.019314413890242577,
+          -0.012914762832224369,
+          0.018935054540634155,
+          0.008849004283547401,
+          -0.014085832983255386,
+          -0.012898268178105354,
+          0.01030871830880642,
+          -0.018044380471110344,
+          -0.019330909475684166,
+          -0.019677281379699707,
+          -0.017401115968823433,
+          0.009929358027875423,
+          0.011314849369227886,
+          -0.040212247520685196,
+          -0.011603493243455887,
+          -0.04288426786661148,
+          -0.011842655949294567,
+          0.008766533806920052,
+          -0.006832618732005358,
+          0.03579187020659447,
+          -0.0036740272771567106,
+          0.004214203916490078,
+          -0.015562041662633419,
+          -0.007418153807520866,
+          -0.002801909577101469,
+          -0.035264063626527786,
+          0.0005896586808376014,
+          0.015075469389557838,
+          0.0003703407128341496,
+          -0.04710672050714493,
+          0.017829960212111473,
+          -0.016617653891444206,
+          0.016865063458681107,
+          -0.02708306536078453,
+          -0.008185122162103653,
+          -0.024312080815434456,
+          0.011611740104854107,
+          0.008997449651360512,
+          -0.03605577349662781,
+          -0.014423958957195282,
+          -0.002556562190875411,
+          0.029293254017829895,
+          -0.017351634800434113,
+          0.0006633660523220897,
+          0.017302153632044792,
+          -0.00019715423695743084,
+          -0.010894253849983215,
+          -0.022497747093439102,
+          -0.0022266830783337355,
+          -0.007937713526189327,
+          0.020106123760342598,
+          -0.04156475141644478,
+          -0.005496608559042215,
+          -0.01082003116607666,
+          0.004001844674348831,
+          -0.030150938779115677,
+          -0.009723182767629623,
+          0.006432640366256237,
+          0.02229982055723667,
+          -0.04882209002971649,
+          0.009253106079995632,
+          0.04374195262789726,
+          -0.018489716574549675,
+          -0.01057262159883976,
+          0.025120284408330917,
+          -0.0007772773969918489,
+          0.009459280408918858,
+          -0.009063425473868847,
+          0.016683630645275116,
+          0.02579653635621071,
+          -0.02061743661761284,
+          -0.008020183071494102,
+          0.0003203434171155095,
+          0.002498833229765296,
+          -0.027973737567663193,
+          0.018473222851753235,
+          -0.013261135667562485,
+          0.023289456963539124,
+          0.0077768974006175995,
+          0.010943735018372536,
+          0.05429808050394058,
+          -0.015248656272888184,
+          0.014399217441678047,
+          0.015562041662633419,
+          -0.0074800061993300915,
+          -0.007974824868142605,
+          -0.013772447593510151,
+          0.022563721984624863,
+          0.03704541176557541,
+          0.013582766987383366,
+          0.005649177823215723,
+          -0.039717432111501694,
+          0.01293950341641903,
+          0.0074758827686309814,
+          0.0038637076504528522,
+          0.02096381038427353,
+          0.017763983458280563,
+          0.02196994051337242,
+          0.024163635447621346,
+          -0.02594498172402382,
+          -0.006098637823015451,
+          0.005513102747499943,
+          -0.02762736566364765,
+          -0.003422494512051344,
+          -0.019512342289090157,
+          0.023619335144758224,
+          0.015223915688693523,
+          -0.02660473994910717,
+          0.0008654168923385441,
+          -0.007954207248985767,
+          -0.0006757365190424025,
+          -0.003993597347289324,
+          -0.012667353264987469,
+          -0.012263251468539238,
+          -0.01576821506023407,
+          0.02759437821805477,
+          -0.014753838069736958,
+          -0.024378057569265366,
+          -0.015990883111953735,
+          0.014489934779703617,
+          -0.009088166058063507,
+          0.011224132962524891,
+          -0.030992131680250168,
+          -0.04964678734540939,
+          -0.0216730497777462,
+          0.028419075533747673,
+          0.03302088752388954,
+          -0.025417177006602287,
+          -1.3602676517621148e-05,
+          -0.007042916491627693,
+          0.029755085706710815,
+          -0.03622071444988251,
+          -0.008568606339395046,
+          -0.01332711149007082,
+          -0.04733763635158539,
+          0.0130714550614357,
+          -0.028567520901560783,
+          0.024922357872128487,
+          0.02708306536078453,
+          -0.018308283761143684,
+          -0.03148695081472397,
+          0.0210957620292902,
+          0.020155606791377068,
+          -0.004109055269509554,
+          0.010234495624899864,
+          -0.0036822741385549307,
+          -0.02658824622631073,
+          -0.028270630165934563,
+          -0.003356518689543009,
+          0.04163073003292084,
+          0.014902283437550068,
+          0.03536302596330643,
+          0.04212554544210434,
+          -0.009921110235154629,
+          -0.034241437911987305,
+          0.019099993631243706,
+          0.009945851750671864,
+          -0.007653192616999149,
+          0.00516672944650054,
+          0.003502902574837208,
+          -0.018423741683363914,
+          -0.009137648157775402,
+          -0.008840756490826607,
+          -0.028765447437763214,
+          -0.0441707968711853,
+          0.0012638488551601768,
+          0.0013360099401324987,
+          0.0167083702981472,
+          -0.012518907897174358,
+          -0.006374911405146122,
+          0.035132113844156265,
+          0.03331777825951576,
+          -0.009789158590137959,
+          -2.2904683646629564e-05,
+          -0.017994899302721024,
+          -0.0103004714474082,
+          -0.008131517097353935,
+          -0.002187510021030903,
+          0.023833757266402245,
+          -0.014300254173576832,
+          0.030051976442337036,
+          -0.0015452768420800567,
+          0.015512559562921524,
+          -0.06749323755502701,
+          -0.02066691778600216,
+          0.03155292570590973,
+          0.008725299499928951,
+          -0.009970592334866524,
+          0.002399869728833437,
+          0.005480114836245775,
+          0.0001164240893558599,
+          -0.0220029279589653,
+          0.011694210581481457,
+          0.020023655146360397,
+          0.019974172115325928,
+          -0.020534968003630638,
+          0.022497747093439102,
+          0.01632900908589363,
+          0.019314413890242577,
+          -0.004210080485790968,
+          -0.025268729776144028,
+          0.012444685213267803,
+          -0.007867613807320595,
+          -0.0019627800211310387,
+          0.002861700253561139,
+          0.007463512010872364,
+          -0.008923226967453957,
+          0.01017676666378975,
+          -0.02205241098999977,
+          -0.02726449817419052,
+          -0.011529270559549332,
+          0.017813464626669884,
+          0.010473658330738544,
+          0.007113015744835138,
+          0.008436654694378376,
+          0.0018452606163918972,
+          0.00416884571313858,
+          0.0013267321046441793,
+          -0.019099993631243706,
+          0.028237640857696533,
+          0.001028294675052166,
+          0.013929140754044056,
+          0.013145677745342255,
+          -0.023635828867554665,
+          -0.023058541119098663,
+          -0.008626335300505161,
+          0.01072106696665287,
+          0.0035647547338157892,
+          0.019017523154616356,
+          0.03879376873373985,
+          -0.014918777160346508,
+          -0.014992999844253063,
+          -0.0008478920790366828,
+          0.019759751856327057,
+          0.0056780423037707806,
+          -0.010943735018372536,
+          -0.0028369592037051916,
+          -0.00416884571313858,
+          -0.005925451405346394,
+          0.025747055187821388,
+          -0.008890238590538502,
+          -0.03373012691736221,
+          0.010943735018372536,
+          0.05756388232111931,
+          -0.007059410214424133,
+          -0.0011659160954877734,
+          -0.022514240816235542,
+          -0.0019122672965750098,
+          0.006906841415911913,
+          0.0080655412748456,
+          0.006993434391915798,
+          0.015281643718481064,
+          0.01581769809126854,
+          0.012147793546319008,
+          -0.010102543979883194,
+          -0.012007595039904118,
+          -0.010374694131314754,
+          -0.03787010908126831,
+          0.030018987134099007,
+          -0.018258802592754364,
+          -0.0035358904860913754,
+          0.011331343092024326,
+          0.02188747003674507,
+          0.00010637308878358454,
+          0.016518689692020416,
+          0.013409581035375595,
+          0.004692528396844864,
+          0.0005932667409069836,
+          0.011652975343167782,
+          0.0015494003891944885,
+          -0.013896152377128601,
+          -0.022761650383472443,
+          -0.009533503092825413,
+          0.018918560817837715,
+          0.02705007791519165,
+          -0.004643046762794256,
+          -0.009978839196264744,
+          -0.005673918407410383,
+          0.014687862247228622,
+          -0.004878085572272539,
+          0.011916878633201122,
+          -0.0045317127369344234,
+          0.00045564534957520664,
+          0.030728228390216827,
+          0.013953881338238716,
+          -0.007298572454601526,
+          0.016972273588180542,
+          0.004556453321129084,
+          -0.003414247650653124,
+          0.019479354843497276,
+          -0.04915196821093559,
+          0.01590016670525074,
+          0.0028348974883556366,
+          0.0018029948696494102,
+          0.0036699038464576006,
+          -0.0020679289009422064,
+          -0.013277629390358925,
+          0.012626118026673794,
+          0.007636698428541422,
+          0.003036948386579752,
+          0.010152026079595089,
+          0.010110790841281414,
+          0.00022125086979940534,
+          0.012279745191335678,
+          -0.019776245579123497,
+          0.011512776836752892,
+          -0.015908414497971535,
+          -0.009467527270317078,
+          -0.008477889932692051,
+          -0.0013019911712035537,
+          -0.0037647439166903496,
+          -0.009962345473468304,
+          0.000947886670473963,
+          -0.022398782894015312,
+          -0.006758395582437515,
+          -0.017434105277061462,
+          -0.008799522183835506,
+          -0.0025359448045492172,
+          -0.024081166833639145,
+          0.0076738097704946995,
+          0.004960555117577314,
+          0.011232379823923111,
+          -0.03658358007669449,
+          0.0028328357730060816,
+          0.03272399678826332,
+          0.004506971687078476,
+          -0.018077367916703224,
+          -0.024378057569265366,
+          -0.011702457442879677,
+          -0.02657175250351429,
+          0.013285876251757145,
+          0.011075686663389206,
+          -0.020040148869156837,
+          0.015463077463209629,
+          -0.001200965722091496,
+          -0.023784274235367775,
+          -0.02098030410706997,
+          -0.02073289453983307,
+          -0.0007360425079241395,
+          0.0013566273264586926,
+          0.001249416614882648,
+          0.0007391351391561329,
+          0.004869838710874319,
+          -0.022877108305692673,
+          0.019611306488513947,
+          0.03354869410395622,
+          -0.008486136794090271,
+          -0.0030472571961581707,
+          0.008816015906631947,
+          0.0220029279589653,
+          -0.01660940796136856,
+          0.00010611536708893254,
+          -0.015042481943964958,
+          0.02150811068713665,
+          -0.0027565513737499714,
+          0.012626118026673794,
+          -0.0026946989819407463,
+          0.029639627784490585,
+          -0.004688404966145754,
+          0.0272974856197834,
+          -0.008947967551648617,
+          0.0220029279589653,
+          0.0013782756868749857,
+          -0.034769244492053986,
+          -0.009401551447808743,
+          0.03277347609400749,
+          -0.002628723159432411,
+          -0.013178665190935135,
+          0.022926589474081993,
+          -0.0036967063788324594,
+          -0.024048177525401115,
+          0.00832944456487894,
+          0.0015875425888225436,
+          -0.01375595387071371,
+          0.024955345317721367,
+          0.006370787974447012,
+          0.009549996815621853,
+          0.004742010496556759,
+          0.015529053285717964,
+          0.024031683802604675,
+          -0.000978297321125865,
+          0.03213021531701088,
+          0.009508761577308178,
+          0.0070016817189753056,
+          -0.004346155561506748,
+          0.012403449974954128,
+          -0.005100754089653492,
+          0.024806899949908257,
+          -0.03272399678826332,
+          -0.014118820428848267,
+          0.033746618777513504,
+          0.015232162550091743,
+          0.018159838393330574,
+          -0.022563721984624863,
+          -0.0047873686999082565,
+          0.016386738047003746,
+          0.012733329087495804,
+          -0.01585068553686142,
+          -0.02081536501646042,
+          -0.001431880984455347,
+          0.008510878309607506,
+          0.004280179738998413,
+          -0.017599044367671013,
+          0.00668417289853096,
+          0.03816699981689453,
+          -0.017978405579924583,
+          -0.040047310292720795,
+          0.02596147730946541,
+          -0.015471324324607849,
+          -0.003115294734016061,
+          0.016007378697395325,
+          -0.005389397963881493,
+          0.014473441056907177,
+          -0.008857251144945621,
+          -0.0033029133919626474,
+          0.03628668934106827,
+          0.008914980106055737,
+          0.010481905192136765,
+          -0.030018987134099007,
+          0.02101329155266285,
+          0.002962725702673197,
+          0.007653192616999149,
+          -0.023668818175792694,
+          0.002189571736380458,
+          0.012411696836352348,
+          -0.030051976442337036,
+          -0.003247246379032731,
+          0.03816699981689453,
+          0.008148010820150375,
+          -0.017615538090467453,
+          0.0008721175836399198,
+          -0.02579653635621071,
+          0.02038652077317238,
+          -0.0050801364704966545,
+          0.014580651186406612,
+          0.019462861120700836,
+          -0.012114806100726128,
+          0.018060874193906784,
+          -0.0056161899119615555,
+          -0.011562258936464787,
+          -0.004931691102683544,
+          -0.010943735018372536,
+          -0.02665422298014164,
+          -0.013986868783831596,
+          0.0020029840525239706,
+          -0.01059736218303442,
+          -0.01629602164030075,
+          0.007158373948186636,
+          -0.03641863912343979,
+          -0.014910530298948288,
+          -0.006391405593603849,
+          0.007632574997842312,
+          -0.023569853976368904,
+          0.017450598999857903,
+          -0.024641960859298706,
+          0.0029874665196985006,
+          0.02182149514555931,
+          -0.003570940112695098,
+          0.0024204871151596308,
+          -0.019001029431819916,
+          -0.029276760295033455,
+          0.008366555906832218,
+          0.006506863050162792,
+          -0.00681200111284852,
+          -0.015908414497971535,
+          0.008857251144945621,
+          0.0017669143853709102,
+          -0.0037771144416183233,
+          -0.006255330517888069,
+          0.01293950341641903,
+          -0.023685311898589134,
+          0.017483586445450783,
+          0.019545329734683037,
+          -0.024246105924248695,
+          0.010762302204966545,
+          -0.0069975582882761955,
+          0.017483586445450783,
+          0.01655992493033409,
+          0.0051337420009076595,
+          0.0164609607309103,
+          0.02695111371576786,
+          0.008259344846010208,
+          0.022728662937879562,
+          0.000705116312019527,
+          -0.004383266903460026,
+          -0.019512342289090157,
+          0.027792304754257202,
+          -0.009525256231427193,
+          0.0001148777810158208,
+          0.012568389065563679,
+          -0.009731430560350418,
+          -0.013945634476840496,
+          0.0019308229675516486,
+          4.95140629936941e-05,
+          -0.016172317788004875,
+          -0.007875860668718815,
+          -0.02150811068713665,
+          -0.005768758710473776,
+          -0.018242308869957924,
+          -0.01110042817890644,
+          0.02220085635781288,
+          -0.020023655146360397,
+          -0.0209308210760355,
+          0.00045384131954051554,
+          -0.00031982798827812076,
+          0.019974172115325928,
+          0.020155606791377068,
+          0.009302587248384953,
+          0.0021297812927514315,
+          -0.0025462533812969923,
+          -0.00341012398712337,
+          -0.005686289165169001,
+          0.008366555906832218,
+          0.006510986480861902,
+          0.02220085635781288,
+          -0.0216730497777462,
+          -0.01576821506023407,
+          0.007080027833580971,
+          -0.006840865593403578,
+          0.009764418005943298,
+          -0.019974172115325928,
+          -0.011776680126786232,
+          -0.004704899154603481,
+          -0.017005261033773422,
+          0.0552547313272953,
+          -0.02716553397476673,
+          -0.035297051072120667,
+          -0.01058086846023798,
+          0.002946231747046113,
+          0.006741901859641075,
+          -0.018522705882787704,
+          0.010366447269916534,
+          -0.01323639415204525,
+          -0.014869295060634613,
+          -0.023586347699165344,
+          0.00514611229300499,
+          0.0026081057731062174,
+          0.005933698266744614,
+          -0.009772664867341518,
+          -0.022910095751285553,
+          0.0036121748853474855,
+          -0.017236176878213882,
+          -0.017434105277061462,
+          -0.0020060765091329813,
+          0.01031696517020464,
+          0.015858933329582214,
+          0.018027886748313904,
+          0.007277955301105976,
+          0.002725625177845359,
+          0.0009215994505211711,
+          -0.002183386590331793,
+          0.009145895019173622,
+          -0.00333383958786726,
+          0.003041071817278862,
+          0.008585100993514061,
+          -0.027808798477053642,
+          0.009582984261214733,
+          -0.013574520125985146,
+          0.02705007791519165,
+          0.029161302372813225,
+          0.01603211835026741,
+          0.013137430883944035,
+          0.009244858287274837,
+          -0.040179260075092316,
+          0.011364331468939781,
+          -0.000893765885848552,
+          0.00807378813624382,
+          0.007220226339995861,
+          0.0038637076504528522,
+          -0.007298572454601526,
+          0.0013040528865531087,
+          -0.019116487354040146,
+          -0.004408007953315973,
+          0.000873663870152086,
+          -0.024559490382671356,
+          0.02187097631394863,
+          0.013533285818994045,
+          0.015545547008514404,
+          0.020403016358613968,
+          -0.011801420710980892,
+          0.010886006988584995,
+          0.01297249086201191,
+          0.003876078175380826,
+          -0.020419510081410408,
+          0.005830611102283001,
+          -0.023437902331352234,
+          0.0065893325954675674,
+          -0.010086050257086754,
+          -0.0340765006840229,
+          -0.020122617483139038,
+          0.017846453934907913,
+          0.008432531729340553,
+          -0.009261352941393852,
+          -0.028946882113814354,
+          0.025235742330551147,
+          0.027858281508088112,
+          -0.004498724825680256,
+          0.008956214413046837,
+          0.027792304754257202,
+          0.010358200408518314,
+          0.0446656160056591,
+          -0.0080655412748456,
+          0.009723182767629623,
+          -0.004630676470696926,
+          0.007088274694979191,
+          -0.0051378654316067696,
+          -0.006869730073958635,
+          0.011050946079194546,
+          0.014762084931135178,
+          0.006069773342460394,
+          0.018836090341210365,
+          0.029359230771660805,
+          -0.019281426444649696,
+          -0.007459388580173254,
+          0.013137430883944035,
+          -0.010770549066364765,
+          0.02088133990764618,
+          -0.0001592696935404092,
+          -0.006358417682349682,
+          -0.028303617611527443,
+          0.006976940669119358,
+          0.008774780668318272,
+          -0.007401659619063139,
+          -0.003787423251196742,
+          -0.019842220470309258,
+          -0.005504855420440435,
+          -0.0012762193800881505,
+          -0.020320545881986618,
+          -0.0009262383682653308,
+          -0.004993543028831482,
+          -0.0026699581649154425,
+          0.0047914921306073666,
+          0.018159838393330574,
+          0.008659323677420616,
+          0.009912863373756409,
+          -0.0017308337846770883,
+          0.014861048199236393,
+          0.03381259739398956,
+          -0.005636807065457106,
+          -0.000506158045027405,
+          0.019231945276260376,
+          0.0011978730326518416,
+          0.013937387615442276,
+          -0.004259562585502863,
+          -0.005785252898931503,
+          -0.014894036576151848,
+          0.01571873389184475,
+          -0.004614182282239199,
+          -0.004201833624392748,
+          -0.012774563394486904,
+          -0.020205087959766388,
+          0.012494166381657124,
+          -0.008857251144945621,
+          -0.008923226967453957,
+          -0.023685311898589134,
+          -0.03130551427602768,
+          -0.017450598999857903,
+          -0.0038224728778004646,
+          -0.0009566490771248937,
+          0.014654873870313168,
+          0.02124420739710331,
+          -0.007232596632093191,
+          0.022250337526202202,
+          -0.005962562747299671,
+          -0.03320232033729553,
+          0.009096412919461727,
+          0.014992999844253063,
+          -0.010275730863213539,
+          0.008267591707408428,
+          -0.02221735008060932,
+          -0.009277846664190292,
+          -0.03206423670053482,
+          -0.019776245579123497,
+          -0.0034781615249812603,
+          0.02149161696434021,
+          -0.01636199839413166,
+          0.02688513696193695,
+          -0.013277629390358925,
+          -0.00779339112341404,
+          0.01333535835146904,
+          0.003752373391762376,
+          -0.01606510765850544,
+          -0.004440995864570141,
+          0.011479788459837437,
+          0.01674135960638523,
+          -0.013021972961723804,
+          -0.004560577217489481,
+          -0.00333796301856637,
+          -0.009706689044833183,
+          0.008914980106055737,
+          -0.006857359316200018,
+          0.007640821859240532,
+          -0.019644293934106827,
+          0.010918994434177876,
+          0.005055395420640707,
+          -0.0017143398290500045,
+          -0.0019339155405759811,
+          -0.013673484325408936,
+          0.005711030215024948,
+          0.013656990602612495,
+          0.020551461726427078,
+          -0.03866181895136833,
+          0.004572947509586811,
+          -0.002818403532728553,
+          -0.005224458407610655,
+          -0.018605174496769905,
+          0.012452932074666023,
+          0.016584666445851326,
+          0.015463077463209629,
+          0.013722965493798256,
+          -0.01041592936962843,
+          0.022514240816235542,
+          -0.012395203113555908,
+          -0.013038466684520245,
+          0.00780988484621048,
+          -0.01306320820003748,
+          0.0005324453231878579,
+          0.029045844450592995,
+          -0.007574846502393484,
+          -0.004626553039997816,
+          0.005574955139309168,
+          0.039552491158246994,
+          0.005385274533182383,
+          0.004985296167433262,
+          0.029375724494457245,
+          -0.0013391025131568313,
+          0.006844989024102688,
+          0.03262503072619438,
+          0.017928922548890114,
+          0.007896478287875652,
+          -0.007096521556377411,
+          0.013962128199636936,
+          0.011496283113956451,
+          0.013904399238526821,
+          -0.012510661035776138,
+          -0.04077304154634476,
+          0.019231945276260376,
+          0.004651293624192476,
+          -0.011974607594311237,
+          0.010152026079595089,
+          -0.0024328576400876045,
+          0.007257337681949139,
+          0.017054744064807892,
+          0.00026699580485001206,
+          -0.022811131551861763,
+          0.013426074758172035,
+          0.004170907661318779,
+          0.011298355646431446,
+          -0.005055395420640707,
+          0.000341476290486753,
+          0.02233280800282955,
+          0.007108892314136028,
+          0.0014689923264086246,
+          0.012469425797462463,
+          -0.0128240454941988,
+          0.02145862765610218,
+          0.019825726747512817,
+          0.02716553397476673,
+          -0.0031503443606197834,
+          -0.0031482826452702284,
+          0.000705116312019527,
+          -0.0070016817189753056,
+          -0.012890021316707134,
+          0.009492267854511738,
+          0.0003778145182877779,
+          0.009401551447808743,
+          -0.005851228721439838,
+          0.011685962788760662,
+          0.007570722606033087,
+          0.004849221091717482,
+          0.019974172115325928,
+          0.005253322888165712,
+          -0.0068037542514503,
+          0.028946882113814354,
+          0.0020277248695492744,
+          0.0009226303081959486,
+          -0.011050946079194546,
+          0.0006973848212510347,
+          0.0014669306110590696,
+          -0.001361781731247902,
+          0.007611957844346762,
+          -0.008110899478197098,
+          -0.0130714550614357,
+          -0.01086126547306776,
+          0.028369592502713203,
+          0.00025436762371100485,
+          0.023503877222537994,
+          -0.0053316690027713776,
+          -0.002236991887912154,
+          0.002789539285004139,
+          -0.01618056371808052,
+          -0.005669794976711273,
+          0.015710486099123955,
+          0.0033214690629392862,
+          0.006432640366256237,
+          0.013285876251757145,
+          -0.0053770276717841625,
+          0.016353750601410866,
+          -0.0056203133426606655,
+          0.0036987680941820145,
+          0.013030219823122025,
+          0.005987303797155619,
+          0.008733546361327171,
+          0.0006375942612066865,
+          -0.017796970903873444,
+          -0.019611306488513947,
+          0.01085301861166954,
+          -0.006865606643259525,
+          0.000996337621472776,
+          0.003098800778388977,
+          -0.001747327740304172,
+          0.011248873546719551,
+          -0.015009493567049503,
+          -0.003950301092118025,
+          -0.008007812313735485,
+          0.034967172890901566,
+          -0.006127502303570509,
+          -0.002942108316347003,
+          -0.0063996524550020695,
+          -0.01334360521286726,
+          0.003667841898277402,
+          -0.008585100993514061,
+          -0.00026235688710585237,
+          0.015710486099123955,
+          0.004915196914225817,
+          -0.01703825034201145,
+          -0.013269382528960705,
+          -0.014696109108626842,
+          0.006469751708209515,
+          0.02078237570822239,
+          0.02706657163798809,
+          -0.008783028461039066,
+          0.007108892314136028,
+          -0.008135640993714333,
+          0.017054744064807892,
+          -0.007706797681748867,
+          0.013541532680392265,
+          -0.022695673629641533,
+          0.008898485451936722,
+          -0.01362400222569704,
+          0.016353750601410866,
+          -0.007438771426677704,
+          0.0038905104156583548,
+          -0.01557853538542986,
+          -0.009690195322036743,
+          0.0010999402729794383,
+          -0.011273614130914211,
+          0.009879875928163528,
+          -0.01030871830880642,
+          -0.008535618893802166,
+          0.0043007973581552505,
+          -0.02159057930111885,
+          -0.0021297812927514315,
+          -0.007682057097554207,
+          -0.005636807065457106,
+          -0.009698442183434963,
+          -0.02116173692047596,
+          -0.02140914648771286,
+          0.005694536026567221,
+          -0.009492267854511738,
+          -0.012147793546319008,
+          -0.028979869559407234,
+          -0.03326829522848129,
+          -0.001901958603411913,
+          -0.02180500142276287,
+          -0.014259018935263157,
+          -0.005814117379486561,
+          0.019215451553463936,
+          -0.011166404001414776,
+          -0.012914762832224369,
+          -0.00795008335262537,
+          -0.011125168763101101,
+          0.019462861120700836,
+          0.014234278351068497,
+          0.009954098612070084,
+          -0.001399923930875957,
+          0.016667135059833527,
+          -0.008700557984411716,
+          0.009756171144545078,
+          -0.040443163365125656,
+          -0.017450598999857903,
+          -0.0028369592037051916,
+          0.0026720198802649975,
+          0.008890238590538502,
+          -0.000427811814006418,
+          0.019924690946936607,
+          -0.010102543979883194,
+          -0.04374195262789726,
+          -0.02116173692047596,
+          -0.002880255924537778,
+          -0.011438554152846336,
+          0.022794637829065323,
+          0.003810102352872491,
+          0.039288587868213654,
+          0.022893602028489113,
+          0.015858933329582214,
+          -0.006708913948386908,
+          0.007401659619063139,
+          -0.019001029431819916,
+          0.0015380608383566141,
+          0.016881557181477547,
+          0.0032369375694543123,
+          -0.011174650862812996,
+          0.029359230771660805,
+          -0.009442785754799843,
+          -0.015075469389557838,
+          0.027858281508088112,
+          0.0011803482193499804,
+          0.013285876251757145,
+          0.0005705875810235739,
+          -0.0050760130397975445,
+          -0.010399434715509415,
+          0.0038451519794762135,
+          0.002278226660564542,
+          -0.012700340710580349,
+          0.01596614345908165,
+          0.00808203499764204,
+          4.94818486913573e-05,
+          0.0007891323766671121,
+          0.025153273716568947,
+          -0.0071295094676315784,
+          -0.017681512981653214,
+          0.014894036576151848,
+          0.0032018879428505898,
+          -0.01084477175027132,
+          -0.011215885169804096,
+          -0.009558243677020073,
+          -0.01349205058068037,
+          -0.010927241295576096,
+          0.0027689216658473015,
+          -0.002622538013383746,
+          0.023784274235367775,
+          -0.004022461827844381,
+          0.006370787974447012,
+          -0.00390906585380435,
+          -0.002867885399609804,
+          0.028699472546577454,
+          0.019149474799633026,
+          -0.02643980085849762,
+          -0.017335141077637672,
+          0.0007241874700412154,
+          -0.005508979316800833,
+          -0.011372578330338001,
+          0.02091432735323906,
+          -0.006024415139108896,
+          -0.024724429473280907,
+          0.025070803239941597,
+          -0.0004649232141673565,
+          0.004102869890630245,
+          0.005739894229918718,
+          0.0264727883040905,
+          -0.0025668710004538298,
+          -0.0004610574396792799,
+          -0.003036948386579752,
+          0.011323096230626106,
+          2.6738238375401124e-05,
+          0.007018175441771746,
+          -0.023998696357011795,
+          -0.003014269284904003,
+          0.010803536511957645,
+          0.002383375773206353,
+          0.02747892029583454,
+          0.005459497217088938,
+          0.00514611229300499,
+          0.01324464101344347,
+          -0.016955779865384102,
+          0.008890238590538502,
+          -0.017895935103297234,
+          0.017170201987028122,
+          -0.008750040084123611,
+          -0.025582116097211838,
+          -0.010275730863213539,
+          0.0034740380942821503,
+          0.008675817400217056,
+          0.006898594554513693,
+          0.0043007973581552505,
+          0.019825726747512817,
+          -3.6467092286329716e-05,
+          0.006354294251650572,
+          0.002397808013483882,
+          0.0009432477527298033,
+          -0.0003213743038941175,
+          0.0037709292955696583,
+          0.02660473994910717,
+          -0.008659323677420616,
+          -0.006296565290540457,
+          0.0009757201769389212,
+          -0.0033276542089879513,
+          -0.01324464101344347,
+          0.003115294734016061,
+          -0.003919374663382769,
+          -0.0034596058540046215,
+          0.045787204056978226,
+          0.017104225233197212,
+          -0.007962454110383987,
+          -0.015265149995684624,
+          -0.01598263718187809,
+          0.027759317308664322,
+          -0.001940100803039968,
+          0.011578752659261227,
+          0.019759751856327057,
+          0.0030637511517852545,
+          -0.002773045329377055,
+          -0.0007468666299246252,
+          -0.0016091909492388368,
+          0.001787531771697104,
+          0.024889370426535606,
+          0.008964461274445057,
+          -0.006964570377022028,
+          -0.008469643071293831,
+          0.019215451553463936,
+          -0.01362400222569704,
+          -0.009426292032003403,
+          0.02119472436606884,
+          -0.003034886671230197,
+          0.02079886943101883,
+          -0.007521240971982479,
+          0.01606510765850544,
+          -0.005443003494292498,
+          -0.006964570377022028,
+          -0.02162356860935688,
+          0.00404101749882102,
+          -0.002410178305581212,
+          0.003016331000253558,
+          0.028254134580492973,
+          -0.014465193264186382,
+          -0.002249362412840128,
+          0.0036080514546483755,
+          0.011182897724211216,
+          -0.015108457766473293,
+          0.01031696517020464,
+          0.001118495943956077,
+          0.0007360425079241395,
+          0.025285225361585617,
+          -0.004133795853704214,
+          -0.004692528396844864,
+          0.03130551427602768,
+          -0.007459388580173254,
+          -0.011422060430049896,
+          -0.001281373668462038,
+          0.008069665171205997,
+          0.005661548115313053,
+          -0.02126070111989975,
+          -0.002247300697490573,
+          -0.01019326038658619,
+          0.0021194724831730127,
+          -0.045391347259283066,
+          0.02159057930111885,
+          -0.02165655605494976,
+          0.025367693975567818,
+          -0.005723400507122278,
+          -0.002488524653017521,
+          -0.008593347854912281,
+          0.006201724987477064,
+          0.005574955139309168,
+          -0.012114806100726128,
+          0.007665562909096479,
+          -0.019198957830667496,
+          0.013302369974553585,
+          0.006473875138908625,
+          -0.019231945276260376,
+          0.015693992376327515,
+          -0.007529487833380699,
+          0.006568715441972017,
+          -0.015438336879014969,
+          -0.003880201606079936,
+          0.019347403198480606,
+          -0.0026967606972903013,
+          -0.005772882141172886,
+          -0.03783712163567543,
+          -0.00418121600523591,
+          -0.011389072053134441,
+          0.00163496274035424,
+          0.002814280102029443,
+          -0.015743475407361984,
+          -0.03572589531540871,
+          0.005113124381750822,
+          0.01305496133863926,
+          -0.0009911833330988884,
+          0.022481253370642662,
+          -0.005781129468232393,
+          -0.005925451405346394,
+          0.008024306036531925,
+          0.0025297594256699085,
+          0.018968041986227036,
+          0.014267266727983952,
+          -0.003490532049909234,
+          -0.0045399595983326435,
+          0.0026740815956145525,
+          -0.007698550820350647,
+          -0.017813464626669884,
+          -0.018836090341210365,
+          -0.013021972961723804,
+          0.008832509629428387,
+          0.018489716574549675,
+          -0.018159838393330574,
+          -0.00040281316614709795,
+          -0.004263686016201973,
+          0.015529053285717964,
+          -0.011356084607541561,
+          -0.022398782894015312,
+          -0.009879875928163528,
+          -0.0034451736137270927,
+          0.012329227291047573,
+          -0.024411045014858246,
+          0.014663120731711388,
+          -0.009615972638130188,
+          -0.004276056308299303,
+          0.015809450298547745,
+          -0.0053316690027713776,
+          0.0014968259492889047,
+          0.004915196914225817,
+          0.028897399082779884,
+          0.0019370082300156355,
+          -0.022596711292862892,
+          0.018638163805007935,
+          0.009426292032003403,
+          -0.00026879983488470316,
+          -0.005001790355890989,
+          -0.004214203916490078,
+          -0.018984535709023476,
+          -0.00652748066931963,
+          0.008626335300505161,
+          0.01351679116487503,
+          0.013162171468138695,
+          0.016106341034173965,
+          -0.00528218736872077,
+          0.0013452877756208181,
+          0.004313167650252581,
+          0.0011349898995831609,
+          0.009912863373756409,
+          0.007455265149474144,
+          -0.002552438760176301,
+          -0.007467635441571474,
+          0.022629698738455772,
+          -0.023223480209708214,
+          -0.0032328141387552023,
+          0.023734793066978455,
+          -0.0030451954808086157,
+          0.013788941316306591,
+          -0.00817275233566761,
+          0.002777168760076165,
+          0.015001246705651283,
+          0.007405783515423536,
+          0.009607725776731968,
+          -0.024196624755859375,
+          0.020023655146360397,
+          -0.019858716055750847,
+          0.005723400507122278,
+          0.005038901697844267,
+          0.006774889770895243,
+          0.008494383655488491,
+          -0.010020074434578419,
+          0.0034554824233055115,
+          0.01334360521286726,
+          0.016939286142587662,
+          0.017120718955993652,
+          -0.015562041662633419,
+          -0.001777223078534007,
+          -0.022514240816235542,
+          -0.017236176878213882,
+          0.02137615904211998,
+          -0.013277629390358925,
+          -0.001867939718067646,
+          0.015281643718481064,
+          -0.009640713222324848,
+          -0.009888122789561749,
+          0.023701805621385574,
+          -0.009797406382858753,
+          0.016840321943163872,
+          0.0068862237967550755,
+          0.017994899302721024,
+          0.008011936210095882,
+          -0.0012184905353933573,
+          0.0026534642092883587,
+          -0.010267484001815319,
+          -0.0012442623265087605,
+          -0.004208019003272057,
+          0.011883890256285667,
+          0.0061522433534264565,
+          -0.01041592936962843,
+          -0.001817427109926939,
+          -0.009657206945121288,
+          -0.009780911728739738,
+          -0.017087731510400772,
+          0.006791383493691683,
+          0.005991427227854729,
+          -0.007385165896266699,
+          -0.016700124368071556,
+          0.00011146302131237462,
+          0.01386316493153572,
+          -0.0029359229374676943,
+          -0.00011751939018722624,
+          0.011644728481769562,
+          -0.0076696863397955894,
+          -0.011586999520659447,
+          -0.007937713526189327,
+          -0.040113285183906555,
+          -0.012057077139616013,
+          0.012799304910004139,
+          -0.0007819163147360086,
+          0.014588898047804832,
+          0.034307416528463364,
+          -0.005591448862105608,
+          0.0017401116201654077,
+          -0.007661439478397369,
+          0.014143561944365501,
+          0.022728662937879562,
+          -0.004816233180463314,
+          -0.004465736914426088,
+          -0.001779284793883562,
+          -0.012353967875242233,
+          -0.000558217114303261,
+          -0.016073353588581085,
+          0.0009757201769389212,
+          -0.003036948386579752,
+          -0.007983071729540825,
+          -0.00339775369502604,
+          -0.010382940992712975,
+          -0.014960012398660183,
+          -0.014300254173576832,
+          0.008568606339395046,
+          0.002474092412739992,
+          -0.007748032920062542,
+          -0.0032967282459139824,
+          -0.012642612680792809,
+          0.013937387615442276,
+          0.006197601556777954,
+          0.006869730073958635,
+          0.0037132003344595432,
+          0.009030437096953392,
+          0.006263577379286289,
+          -0.005455373786389828,
+          0.003102924209088087,
+          -0.0023751286789774895,
+          -0.030662251636385918,
+          0.011661222204566002,
+          -0.016551679000258446,
+          0.020436003804206848,
+          0.009937604889273643,
+          -0.0053729042410850525,
+          0.0053729042410850525,
+          0.0024575984571129084,
+          0.02121121808886528,
+          -0.009393304586410522,
+          0.014489934779703617,
+          0.0035894957836717367,
+          0.00821398664265871,
+          -0.00529868109151721,
+          0.004271932877600193,
+          -0.004688404966145754,
+          -0.014992999844253063,
+          0.01641147956252098,
+          0.00795833021402359,
+          0.022448265925049782,
+          0.00416884571313858,
+          0.015438336879014969,
+          0.0035276433918625116,
+          -0.001785470056347549,
+          0.00021802939590997994,
+          0.016551679000258446,
+          0.006078020203858614,
+          -0.015990883111953735,
+          -0.004395637661218643,
+          -0.006201724987477064,
+          0.0128240454941988,
+          0.02630784921348095,
+          -0.01629602164030075,
+          -0.01350029744207859,
+          0.023503877222537994,
+          -0.006981064099818468,
+          0.02629135549068451,
+          -0.004589441232383251,
+          -0.017665019258856773,
+          -0.008832509629428387,
+          -0.0018473223317414522,
+          -0.001506103784777224,
+          -0.0208483524620533,
+          0.019446365535259247,
+          0.01334360521286726,
+          -0.002018447034060955,
+          -0.0035874340683221817,
+          0.010622103698551655,
+          -0.0018493840470910072,
+          0.020073136314749718,
+          -0.024064673110842705,
+          -0.007100645452737808,
+          0.0059130811132490635,
+          0.011125168763101101,
+          0.0071913618594408035,
+          -0.015487818978726864,
+          0.015990883111953735,
+          0.018902065232396126,
+          -0.0021318430081009865,
+          0.020172100514173508,
+          -0.015339372679591179,
+          -0.02091432735323906,
+          0.0014576527755707502,
+          -0.007306819781661034,
+          -0.0026720198802649975,
+          -0.0015751721803098917,
+          -0.0009288155124522746,
+          0.009310834109783173,
+          0.011207638308405876,
+          0.0005079620750620961,
+          -0.012337474152445793,
+          0.00038992727058939636,
+          0.00340600055642426,
+          -0.014382723718881607,
+          -0.019215451553463936,
+          0.009838640689849854,
+          -0.015430090017616749,
+          0.011455047875642776,
+          0.018885571509599686,
+          0.0038245345931500196,
+          -0.008807769045233727,
+          0.015174433588981628,
+          0.017516573891043663,
+          0.0065893325954675674,
+          -0.013384840451180935,
+          0.004230698104947805,
+          -0.0010097390040755272,
+          -0.004737887065857649,
+          0.008824262768030167,
+          -0.004812109749764204,
+          0.020023655146360397,
+          -0.0133600989356637,
+          0.02691812627017498,
+          0.004577070940285921,
+          0.030101457610726357,
+          0.013162171468138695,
+          0.005636807065457106,
+          0.001126742921769619,
+          0.0077232918702065945,
+          0.00780163798481226,
+          0.011982854455709457,
+          -0.0164609607309103,
+          0.013896152377128601,
+          -0.0006267700809985399,
+          0.009813900105655193,
+          -0.019908197224140167,
+          -0.003797731827944517,
+          0.013384840451180935,
+          -0.0033812597393989563,
+          -0.02615940384566784,
+          0.014193043112754822,
+          0.02126070111989975,
+          0.01655992493033409,
+          -0.030530299991369247,
+          0.0164609607309103,
+          -0.013401334173977375,
+          0.015347619540989399,
+          0.0030678745824843645,
+          -0.001799902180209756,
+          -0.02126070111989975,
+          -0.0210957620292902,
+          0.015941401943564415,
+          -0.025301719084382057,
+          -0.009244858287274837,
+          0.008807769045233727,
+          0.0038492754101753235,
+          0.014061091467738152,
+          -0.02617589756846428,
+          -0.008028429932892323,
+          -0.008069665171205997,
+          -0.008296456187963486,
+          0.0036740272771567106,
+          0.002236991887912154,
+          0.006069773342460394,
+          -0.006292441859841347,
+          -0.02149161696434021,
+          -0.018110357224941254,
+          0.0011813790770247579,
+          0.01375595387071371,
+          0.011595246382057667,
+          -0.029441699385643005,
+          0.005748141556978226,
+          0.010886006988584995,
+          -0.0033049751073122025,
+          0.0007545981789007783,
+          -0.024328574538230896,
+          -0.016254786401987076,
+          -0.013549779541790485,
+          -0.006762519478797913,
+          -0.009310834109783173,
+          0.004676034674048424,
+          -0.008424284867942333,
+          -0.004581194370985031,
+          -0.01639498583972454,
+          -0.009665454737842083,
+          0.01297249086201191,
+          0.018687644973397255,
+          -0.005657424684613943,
+          -0.029821060597896576,
+          0.02681916207075119,
+          0.018291790038347244,
+          -0.008824262768030167,
+          -0.01641147956252098,
+          -0.0003654440806712955,
+          -0.016955779865384102,
+          0.008947967551648617,
+          0.004803862888365984,
+          -0.014580651186406612,
+          -0.009112906642258167,
+          -0.010877760127186775,
+          0.013310616835951805,
+          0.020122617483139038,
+          0.004820356611162424,
+          0.0074882530607283115,
+          -0.0035317668225616217,
+          0.023042047396302223,
+          -0.006915088277310133,
+          -2.363756948398077e-06,
+          0.0029833430889993906,
+          -0.00680787768214941,
+          0.007756279781460762,
+          0.023718299344182014,
+          -0.0335816815495491,
+          -0.006853235885500908,
+          -0.002346264198422432,
+          -0.002056589350104332,
+          -0.011949866078794003,
+          -0.011644728481769562,
+          -0.0077356621623039246,
+          0.010778795927762985,
+          0.006750148721039295,
+          -0.0018473223317414522,
+          0.002083391882479191,
+          -0.0133188646286726,
+          0.009129401296377182,
+          -0.024889370426535606,
+          0.002865823684260249,
+          -0.018473222851753235,
+          0.010028321295976639,
+          -0.009657206945121288,
+          -0.00012628179683815688,
+          0.013533285818994045,
+          -0.000562340603210032,
+          0.0015019802376627922,
+          -0.019182464107871056,
+          -0.009805653244256973,
+          0.00791709590703249,
+          0.008436654694378376,
+          0.009294340386986732,
+          -0.02586251311004162,
+          -0.0012298300862312317,
+          0.004601811990141869,
+          0.004170907661318779,
+          -0.0018267049454152584,
+          -0.02726449817419052,
+          -0.0074758827686309814,
+          -0.014894036576151848,
+          -0.02172253094613552,
+          0.0017421734519302845,
+          -0.015149693004786968,
+          -0.013986868783831596,
+          0.024130648002028465,
+          0.02216786891222,
+          0.016551679000258446,
+          -0.004886332433670759,
+          0.01571873389184475,
+          0.006601703353226185,
+          -0.023652324452996254,
+          0.0010679832194000483,
+          -0.014168302528560162,
+          0.01098497025668621,
+          -0.008906732313334942,
+          0.022629698738455772,
+          -0.001510227215476334,
+          -0.010020074434578419,
+          -0.003008084138855338,
+          0.013137430883944035,
+          0.008956214413046837,
+          -0.022382289171218872,
+          -0.009978839196264744,
+          0.003024578094482422,
+          -0.03224566951394081,
+          -0.006622320506721735,
+          0.0025276977103203535,
+          0.0014566219178959727,
+          0.007401659619063139,
+          0.007059410214424133,
+          -0.014539416879415512,
+          -0.0037152620498090982,
+          0.0021524603944271803,
+          -0.01086951233446598,
+          0.005983180366456509,
+          0.015083717182278633,
+          0.017912428826093674,
+          0.011595246382057667,
+          0.007356301415711641,
+          -0.014028104022145271,
+          0.004222451243549585,
+          -0.015174433588981628,
+          -0.0053811511024832726,
+          0.005010037217289209,
+          -0.025252236053347588,
+          -0.012386956252157688,
+          0.02162356860935688,
+          0.02168954350054264,
+          -0.0005592479719780385,
+          -0.023718299344182014,
+          -0.01675785332918167,
+          0.012923009693622589,
+          -0.01309619564563036,
+          0.018885571509599686,
+          -0.0033936300314962864,
+          -0.014259018935263157,
+          -0.0016184687847271562,
+          0.017928922548890114,
+          -0.018275296315550804,
+          0.01598263718187809,
+          -0.02071640081703663,
+          0.009129401296377182,
+          0.002925614360719919,
+          0.0025957352481782436,
+          0.0014875481138005853,
+          0.050273556262254715,
+          -0.00046518092858605087,
+          0.0009396396926604211,
+          0.018737126141786575,
+          -0.033680643886327744,
+          0.02132667601108551,
+          0.003560631303116679,
+          0.00822223350405693,
+          0.0012885897886008024,
+          -0.01319515984505415,
+          0.0105891153216362,
+          0.019677281379699707,
+          0.027874775230884552,
+          -0.010094297118484974,
+          -0.007121262606233358,
+          0.018143344670534134,
+          0.003245184663683176,
+          -0.001950409496203065,
+          0.0029606639873236418,
+          0.004185339901596308,
+          -0.019330909475684166,
+          -0.0036822741385549307,
+          0.011166404001414776,
+          -0.014506428502500057,
+          -0.009624219499528408,
+          -0.014374476857483387,
+          -0.009409798309206963,
+          -0.008964461274445057,
+          -0.010382940992712975,
+          0.0005654331762343645,
+          -0.0026534642092883587,
+          0.014423958957195282,
+          -0.013013726100325584,
+          0.029590144753456116,
+          -0.013302369974553585,
+          -0.024526502937078476,
+          -0.0037544353399425745,
+          -0.007422277238219976,
+          0.02074938826262951,
+          -0.015446583740413189,
+          -0.010127284564077854,
+          -0.0034348650369793177,
+          -0.012535401619970798,
+          0.022448265925049782,
+          0.008931473828852177,
+          -0.006222342606633902,
+          -0.0061604902148246765,
+          0.020172100514173508,
+          0.024295587092638016,
+          -0.0256480909883976,
+          0.02144213393330574,
+          0.020139113068580627,
+          -0.008552112616598606,
+          -0.0033874448854476213,
+          0.002342140767723322,
+          0.005974933039397001,
+          -0.0015050729271024466,
+          0.010646844282746315,
+          0.006226466037333012,
+          -0.013830176554620266,
+          -0.0163784921169281,
+          0.01692279241979122,
+          0.018473222851753235,
+          -0.008750040084123611,
+          0.009063425473868847,
+          -0.003573001828044653,
+          0.002962725702673197,
+          -0.00333796301856637,
+          0.007682057097554207,
+          -0.009987086057662964,
+          -0.017961911857128143,
+          0.011042699217796326,
+          0.0029070586897432804,
+          -0.005715153645724058,
+          0.0037647439166903496,
+          0.023718299344182014,
+          -0.024510009214282036,
+          -0.013566273264586926,
+          0.005488361697643995,
+          -0.02159057930111885,
+          0.014687862247228622,
+          -0.0006107915542088449,
+          0.008824262768030167,
+          0.001291682361625135,
+          0.003952362574636936,
+          -0.024955345317721367,
+          0.009393304586410522,
+          0.008906732313334942,
+          0.0006061526364646852,
+          -0.000622646592091769,
+          -0.009022190235555172,
+          -0.01100146397948265,
+          -0.007381042465567589,
+          -0.006964570377022028,
+          -0.0108365248888731,
+          -0.02061743661761284,
+          0.023883238434791565,
+          0.0022864737547934055,
+          -0.006840865593403578,
+          0.0022947206161916256,
+          0.0136487428098917,
+          -0.0032163201831281185,
+          -0.004317291080951691,
+          0.010077803395688534,
+          -0.017219683155417442,
+          -0.006073896773159504,
+          0.005174976773560047,
+          0.007628451567143202,
+          -0.019710268825292587,
+          -0.009517008438706398,
+          -0.012436438351869583,
+          0.012362214736640453,
+          0.007199608720839024,
+          -0.015512559562921524,
+          -0.01100146397948265,
+          0.00257099443115294,
+          0.005034778267145157,
+          -0.05881742388010025,
+          -0.008543865755200386,
+          0.003925560042262077,
+          0.03879376873373985,
+          -0.003125603310763836,
+          -0.0158836729824543,
+          -0.002467907266691327,
+          0.02736346237361431,
+          0.00016519721248187125,
+          0.02058444917201996,
+          0.006271824240684509,
+          -0.009714935906231403,
+          -0.008477889932692051,
+          0.007591340225189924,
+          -0.005335792433470488,
+          -0.013929140754044056,
+          0.004696652293205261,
+          0.005880093201994896,
+          0.02703358232975006,
+          -0.010514892637729645,
+          0.0005762573564425111,
+          -0.0031379738356918097,
+          -0.001628777477890253,
+          -0.003711138619109988,
+          -0.005253322888165712,
+          0.013162171468138695,
+          -0.015529053285717964,
+          0.014333241619169712,
+          -0.01386316493153572,
+          0.004416254814714193,
+          0.007442894857376814,
+          -0.004511095117777586,
+          -0.002079268451780081,
+          -0.0031585912220180035,
+          0.002257609274238348,
+          0.0071954852901399136,
+          0.004911073483526707,
+          -0.024180129170417786,
+          0.012543648481369019,
+          0.0036987680941820145,
+          0.011784926988184452,
+          0.00779339112341404,
+          -0.023965708911418915,
+          -0.0026163526345044374,
+          0.014168302528560162,
+          0.011438554152846336,
+          0.004100808408111334,
+          0.008494383655488491,
+          -0.008102652616798878,
+          0.015124951489269733,
+          -0.04209255799651146,
+          0.00844490248709917,
+          -0.005789376329630613,
+          0.008275839500129223,
+          -0.011586999520659447,
+          0.009343822486698627,
+          0.007884107530117035,
+          -0.023668818175792694,
+          0.009888122789561749,
+          -0.002232868457213044,
+          -0.014044597744941711,
+          -0.015289891511201859,
+          -0.01055612787604332,
+          -0.00011519993131514639,
+          -0.012551895342767239,
+          0.0017648525536060333,
+          0.0026926372665911913,
+          -0.001965872710570693,
+          -0.024526502937078476,
+          -0.0007612988702021539,
+          -0.009517008438706398,
+          -0.008618088439106941,
+          -0.00555021408945322,
+          -0.009805653244256973,
+          0.012197275646030903,
+          0.008218110539019108,
+          -0.0030039604753255844,
+          -0.01086126547306776,
+          0.02173902466893196,
+          0.0005819271318614483,
+          0.000386576954042539,
+          -0.007987194694578648,
+          -0.004729640204459429,
+          -0.02686864323914051,
+          0.009055178612470627,
+          -0.02206890471279621,
+          -0.0005499701364897192,
+          0.002034940989688039,
+          0.0050760130397975445,
+          -0.014094079844653606,
+          0.0025421299505978823,
+          -0.00138961523771286,
+          0.024163635447621346,
+          0.0031544677913188934,
+          -0.013698224909603596,
+          0.009805653244256973,
+          -0.015496065840125084,
+          0.0108365248888731,
+          0.03278997167944908,
+          0.001166946953162551,
+          0.0011659160954877734,
+          -0.001972057856619358,
+          -0.003702891757711768,
+          0.008424284867942333,
+          0.0019442243501543999,
+          -0.03354869410395622,
+          -0.02655525878071785,
+          -0.009863382205367088,
+          0.011826162226498127,
+          0.0056780423037707806,
+          0.0011174650862812996,
+          -0.010119037702679634,
+          0.01593315601348877,
+          0.006601703353226185,
+          0.02578004263341427,
+          0.014720849692821503,
+          0.013038466684520245,
+          0.005525473039597273,
+          0.0042925504967570305,
+          -0.006275947671383619,
+          -0.009178882464766502,
+          -0.012362214736640453,
+          0.011883890256285667,
+          0.007987194694578648,
+          0.012123052962124348,
+          0.0010772610548883677,
+          -0.00163083930965513,
+          0.010110790841281414,
+          -0.018885571509599686,
+          -0.024114154279232025,
+          0.020007161423563957,
+          0.0020638054702430964,
+          0.002554500475525856,
+          -0.013013726100325584,
+          0.004717269446700811,
+          9.213417069986463e-05,
+          0.008898485451936722,
+          -0.005900710355490446,
+          0.00265140226110816,
+          0.01677434705197811,
+          0.0051502357237041,
+          0.007463512010872364,
+          -0.0012937441933900118,
+          -0.01692279241979122,
+          0.01108393445611,
+          -0.004750257357954979,
+          -0.015487818978726864,
+          -0.014151808805763721,
+          0.01095198281109333,
+          -0.0008185122278518975,
+          0.006226466037333012,
+          -0.0011803482193499804,
+          -0.011133415624499321,
+          0.017368128523230553,
+          -0.024575984105467796,
+          0.006510986480861902,
+          0.001821550540626049,
+          0.024872876703739166,
+          0.0012267375132068992,
+          0.0038616459351032972,
+          -0.020188594236969948,
+          0.028435569256544113,
+          -0.023916225880384445,
+          -0.007945960387587547,
+          -0.011347836814820766,
+          6.86818384565413e-05,
+          -0.0007463512010872364,
+          0.0037853613030165434,
+          -0.008251097984611988,
+          -0.02144213393330574,
+          0.008750040084123611,
+          0.024213118478655815,
+          0.0017081546830013394,
+          0.007748032920062542,
+          0.016386738047003746,
+          0.0016184687847271562,
+          -0.012667353264987469,
+          -0.0105478810146451,
+          -0.011248873546719551,
+          -0.011207638308405876,
+          -0.012255004607141018,
+          0.003812164068222046,
+          -0.018852584064006805,
+          0.015372361056506634,
+          0.007265584543347359,
+          -0.020337039604783058,
+          -0.0015050729271024466,
+          0.006374911405146122,
+          0.013566273264586926,
+          -0.007517117541283369,
+          0.012123052962124348,
+          0.019875209778547287,
+          0.002107101958245039,
+          -0.0006345016299746931,
+          0.00041080242954194546,
+          0.006746025290340185,
+          -0.015661004930734634,
+          0.030018987134099007,
+          0.006869730073958635,
+          0.0063955290243029594,
+          -0.01060560904443264,
+          0.018819596618413925,
+          -0.007261461112648249,
+          0.014481687918305397,
+          -0.018671151250600815,
+          -0.01337659265846014,
+          -0.014745591208338737,
+          0.011166404001414776,
+          -0.010630350559949875,
+          0.003573001828044653,
+          0.03582485765218735,
+          0.008576854132115841,
+          -0.002403993159532547,
+          -0.006016168277710676,
+          -0.015050728805363178,
+          -0.004333785269409418,
+          -0.013953881338238716,
+          -0.000127505962154828,
+          -0.025169767439365387,
+          -0.013409581035375595,
+          0.00807378813624382,
+          0.02122771367430687,
+          0.014836307615041733,
+          0.004671911243349314,
+          0.015067222528159618,
+          -0.007133633363991976,
+          0.01682382822036743,
+          0.018687644973397255,
+          0.01069632638245821,
+          -0.00012067643547197804,
+          -0.009813900105655193,
+          0.0010020073968917131,
+          0.025235742330551147,
+          -0.011199391447007656,
+          -0.00541001558303833,
+          0.02229982055723667,
+          0.02046899124979973,
+          0.0011102489661425352,
+          0.0058594755828380585,
+          0.010127284564077854,
+          0.009079919196665287,
+          0.014168302528560162,
+          -0.004523465409874916,
+          0.012584883719682693,
+          -0.01082003116607666,
+          -0.006506863050162792,
+          0.04816233366727829,
+          0.014448699541389942,
+          -0.0036059897392988205,
+          -0.012197275646030903,
+          -0.0020772067364305258,
+          0.0023895609192550182,
+          0.02107926830649376,
+          0.010069556534290314,
+          0.015289891511201859,
+          0.024608973413705826,
+          -0.007661439478397369,
+          0.014423958957195282,
+          -0.0059172045439481735,
+          0.04410482197999954,
+          -0.017236176878213882,
+          -0.004972925875335932,
+          -0.012535401619970798,
+          0.006840865593403578,
+          -0.00527394050732255,
+          0.018011393025517464,
+          -0.0011545764282345772,
+          0.0007509901188313961,
+          -0.028386086225509644,
+          0.019380390644073486,
+          0.012692093849182129,
+          -0.03829894959926605,
+          0.006234712898731232,
+          -0.009599478915333748,
+          -0.0019493786385282874,
+          -0.02645629458129406,
+          -0.004109055269509554,
+          -0.009987086057662964,
+          -0.0029194289818406105,
+          -0.025351200252771378,
+          0.007302696350961924,
+          -0.025318212807178497,
+          0.009426292032003403,
+          -0.0077851442620158195,
+          -0.0058718458749353886,
+          -0.005661548115313053,
+          -0.009269599802792072,
+          -0.0033523952588438988,
+          0.019495848566293716,
+          0.017203189432621002,
+          0.004878085572272539,
+          0.006783136632293463,
+          0.008914980106055737,
+          -0.011529270559549332,
+          -0.033944547176361084,
+          -0.008519125171005726,
+          -0.01657641865313053,
+          0.00834593828767538,
+          0.014324994757771492,
+          0.02597797103226185,
+          -0.02150811068713665,
+          -0.008799522183835506,
+          -0.009904616512358189,
+          -0.01571873389184475,
+          -0.010663338005542755,
+          0.010102543979883194,
+          -0.004564700648188591,
+          -0.023437902331352234,
+          0.0010468503460288048,
+          -0.005038901697844267,
+          0.004968802444636822,
+          -0.003812164068222046,
+          0.007273831870406866,
+          0.008956214413046837,
+          -0.011652975343167782,
+          -0.03125603497028351,
+          -0.0035297051072120667,
+          -0.001355596468783915,
+          -0.0028452062979340553,
+          -0.025417177006602287,
+          0.007756279781460762,
+          -0.015990883111953735,
+          -0.02112874947488308,
+          0.013294123113155365,
+          -0.007405783515423536,
+          0.006292441859841347,
+          0.018638163805007935,
+          0.013310616835951805,
+          0.010490152053534985,
+          0.033977534621953964,
+          -0.0011525147128850222,
+          -0.014382723718881607,
+          -0.011314849369227886,
+          0.014531169086694717,
+          -0.017005261033773422,
+          0.014011610299348831,
+          -0.007442894857376814,
+          -0.005884216632694006,
+          -0.0002360696526011452,
+          -0.0017968096071854234,
+          0.008857251144945621,
+          -0.009203623980283737,
+          0.00258336472325027,
+          -0.024575984105467796,
+          0.015413595363497734,
+          -0.016155824065208435,
+          -0.009970592334866524,
+          0.010226248763501644,
+          0.031025119125843048,
+          -0.015339372679591179,
+          -0.012634364888072014,
+          -0.0017504204297438264,
+          0.000949948385823518,
+          0.003818349214270711,
+          -0.006242959760129452,
+          -0.008123270235955715,
+          0.0051378654316067696,
+          0.0007159404922276735,
+          -0.016304269433021545,
+          0.0004020400228910148,
+          0.01111692190170288,
+          -0.028485050424933434,
+          -0.017500080168247223,
+          -0.007150127086788416,
+          -0.01566925272345543,
+          0.008267591707408428,
+          0.030101457610726357,
+          0.007084151264280081,
+          -0.006506863050162792,
+          -0.022794637829065323,
+          0.008246975019574165,
+          -0.027643859386444092,
+          0.02098030410706997,
+          0.07903900742530823,
+          -0.004820356611162424,
+          -0.0025441916659474373,
+          -0.018918560817837715,
+          -0.0014267265796661377,
+          0.0032946662977337837,
+          -0.017384622246026993,
+          0.05429808050394058,
+          -0.0014287884114310145,
+          0.004280179738998413,
+          -0.0076738097704946995,
+          0.010886006988584995,
+          -0.0267201978713274,
+          -0.02134316973388195,
+          -0.020205087959766388,
+          -0.010366447269916534,
+          -0.00543475616723299,
+          -0.0036245454102754593,
+          -0.013541532680392265,
+          0.0018225813983008265,
+          0.011958112940192223,
+          -0.00833769142627716,
+          0.0064078993164002895,
+          -0.024114154279232025,
+          -0.007166620809584856,
+          0.011067439801990986,
+          0.014885789714753628,
+          0.027990233153104782,
+          -0.030134445056319237,
+          -0.014588898047804832,
+          -0.007146003656089306,
+          0.008110899478197098,
+          -0.009302587248384953,
+          0.007644945755600929,
+          -0.0208483524620533,
+          -0.005908957216888666,
+          -0.010036568157374859,
+          0.014094079844653606,
+          0.014836307615041733,
+          -0.0022204979322850704,
+          -0.01058086846023798,
+          0.012090064585208893,
+          0.029227279126644135,
+          0.0015421842690557241,
+          0.009690195322036743,
+          -0.011677715927362442,
+          -0.011347836814820766,
+          0.0036925829481333494,
+          0.03151993826031685,
+          0.00652335723862052,
+          -0.00780576141551137,
+          -0.005785252898931503,
+          0.024641960859298706,
+          -0.004191524814814329,
+          0.01100146397948265,
+          -0.008939720690250397,
+          0.023157505318522453,
+          0.0016442405758425593,
+          0.009978839196264744,
+          0.011422060430049896,
+          -0.011933372355997562,
+          0.02076588198542595,
+          -0.0007344961632043123,
+          0.004589441232383251,
+          -0.0011009711306542158,
+          0.0017730995314195752,
+          -0.004737887065857649,
+          0.0040533882565796375,
+          0.003193640848621726,
+          -0.0013184851268306375,
+          0.010770549066364765,
+          3.614494562498294e-05,
+          0.005777005571871996,
+          -0.018951548263430595,
+          -0.012527154758572578,
+          0.013854917138814926,
+          -0.034406378865242004,
+          0.000833459896966815,
+          0.011191144585609436,
+          -0.02094731666147709,
+          0.02206890471279621,
+          -0.0077810208313167095,
+          -0.023256469517946243,
+          -0.007851120084524155,
+          0.00833769142627716,
+          0.004799739457666874,
+          -0.004968802444636822,
+          0.007884107530117035,
+          -0.007063533645123243,
+          0.005946068558841944,
+          -0.0024184253998100758,
+          0.02037002705037594,
+          -0.022431770339608192,
+          0.020534968003630638,
+          0.0005891432520002127,
+          0.01593315601348877,
+          -0.008849004283547401,
+          0.011273614130914211,
+          0.011908631771802902,
+          0.004498724825680256,
+          0.013722965493798256,
+          0.006003797519952059,
+          -0.014976506121456623,
+          0.014432205818593502,
+          0.008378926664590836,
+          0.009145895019173622,
+          0.02211838588118553,
+          0.014704355970025063,
+          0.011240626685321331,
+          -0.009517008438706398,
+          0.008947967551648617,
+          -0.011504529975354671,
+          -0.0015153816202655435,
+          0.01682382822036743,
+          0.02196994051337242,
+          -0.013945634476840496,
+          -0.004902826622128487,
+          -0.019034016877412796,
+          0.015240409411489964,
+          -0.0006375942612066865,
+          0.012692093849182129,
+          0.018390754237771034,
+          0.004882209002971649,
+          0.008980955928564072,
+          0.019743258133530617,
+          0.04215853661298752,
+          0.004799739457666874,
+          -0.0062182191759347916,
+          0.013978621922433376,
+          0.00795833021402359,
+          -0.005785252898931503,
+          0.00679963082075119,
+          -0.01674135960638523,
+          0.0003015300026163459,
+          0.009500514715909958,
+          -0.0025318211410194635,
+          -0.010020074434578419,
+          0.005983180366456509,
+          0.008741793222725391,
+          0.011413812637329102,
+          -0.012205522507429123,
+          0.010086050257086754,
+          -0.000819543085526675,
+          -0.004807986319065094,
+          -0.02063393034040928,
+          -0.013252888806164265,
+          0.03153643012046814,
+          -0.01348380371928215,
+          -0.008102652616798878,
+          0.02162356860935688,
+          0.017054744064807892,
+          0.004585317801684141,
+          -0.017582550644874573,
+          -0.005851228721439838,
+          0.012461178936064243,
+          -0.0011504529975354671,
+          0.023899732157588005,
+          -0.011240626685321331,
+          0.008634582161903381,
+          -0.014357983134686947,
+          -0.009376809932291508,
+          0.0011514838552102447,
+          0.012370462529361248,
+          0.011471541598439217,
+          0.02749541401863098,
+          0.014720849692821503,
+          -0.004086376167833805,
+          -0.0010834463173523545,
+          0.00791709590703249,
+          0.013533285818994045,
+          0.023108022287487984,
+          -0.0026122292038053274,
+          0.005047148559242487,
+          0.020089630037546158,
+          -0.00554609065875411,
+          0.012535401619970798,
+          -0.022382289171218872,
+          -0.014770331792533398,
+          -0.007574846502393484,
+          0.01099321711808443,
+          -0.011974607594311237,
+          0.017467092722654343,
+          0.0013102381490170956,
+          0.02691812627017498,
+          -0.0026823284570127726,
+          0.0028988115955144167,
+          0.0037647439166903496,
+          0.003985350485891104,
+          -0.0012257066555321217,
+          0.00805729441344738,
+          -0.02101329155266285,
+          0.01046541053801775,
+          -0.008519125171005726,
+          0.003183332271873951,
+          -0.01616406999528408,
+          0.0025895501021295786,
+          -0.007954207248985767,
+          -0.024312080815434456,
+          9.664423851063475e-05,
+          -0.014646627008914948,
+          -0.00529043423011899,
+          0.0018431989010423422,
+          0.004704899154603481,
+          -0.00820573978126049,
+          0.007966578006744385,
+          0.007071780972182751,
+          -0.014927024021744728,
+          0.0009128370438702404,
+          -0.00514198886230588,
+          -0.0001724133180687204,
+          0.0067047905176877975
+        ],
+        "keyphrases": [
+          "conditional computation",
+          "deep networks",
+          "Google Brain",
+          "neural machine translation",
+          "algorithmic and engineering solutions"
+        ],
+        "title": "6 Conclusion"
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "7f92225a-9c2d-4745-b7b1-2273bcee2b14",
+      "properties": {
+        "page_content": "Appendices A Load-Balancing Loss\n\nAs discussed in section 4, for load-balancing purposes, we want to define an additional loss function to encourage experts to receive roughly equal numbers of training examples. Unfortunately, the number of examples received by an expert is a discrete quantity, so it can not be used in backpropagation. Instead, we define a smooth estimator Load(X) of the number of examples assigned to each expert for a batch X of inputs. The smoothness allows us to back-propagate gradients through the estimator. This is the purpose of the noise term in the gating function. We define P(x, i) as the probability that G(x)iis nonzero, given a new random choice of noise on element i, but keeping the already-sampled choices of noise on the other elements. To compute P(x, i), we note that the G(x)iis nonzero if and only if H(x)iis greater than the k th-greatest element of H(x) excluding itself. The probability works out to be:\n\n$$P(x,i)=P r\\Big((x\\cdot W_{g}){i}+S t a n d x N o r a l()\\cdot S o f t p u s((x\\cdot W{n o i s e})_{i})$$ $$>k t h_e x c l u d i n(H(x),k,i)\\Big)$$ $$({\\boldsymbol{8}})$$\n\n$$(9)$$ $$(10)$$ (8) Where kth_excluding(v, k, i) means the kth highest component of v, excluding component i. Simplifying, we get:\n\n$$P(x,i)=\\Phi{\\Big(}{\\frac{(x\\cdot W_{g}){i}-k t h_c x c l u d i n g(H(x),k,i)}{S o f t p l u s((x\\cdot W{n o i s e})_{i})}}{\\Big)}$$\n\nWhere Φ is the CDF of the standard normal distribution.\n\n$$L o a d(X){i}=\\sum{x\\in X}P(x,i)$$\n\n$$(11)$$ P(x, i) (10) We can now define the load loss to be the square of the coefficient of variation of the load vector, multiplied by a hand-tuned scaling factor wload.\n\n$$L_{l o a d}(X)=w_{l o a d}\\cdot C V(L o a d(X))^{2}$$ Lload(X) = wload · CV (Load(X))2(11) Initial Load Imbalance: To avoid out-of-memory errors, we need to initialize the network in a state of approximately equal expert load (since the soft constraints need some time to work). To accomplish this, we initialize the matrices Wg and Wnoise to all zeros, which yields no signal and some noise.\n\nExperiments: We trained a set of models with identical architecture (the MoE-256 model described in Appendix C), using different values of wimportance and wload. We trained each model for 10 epochs, then measured perplexity on the test set. We also measured the coefficients of variation in Importance and Load, as well as ratio of the load on the most overloaded expert to the average load. This last value is significant for load balancing purposes on distributed hardware. All of these metrics were averaged over several training batches.\n\nwimportance wload Test Perplexity CV (Importance(X)) CV (Load(X)) max(Load(X)) mean(Load(X)) 0.0 0.0 39.8 3.04 3.01 17.80 0.2 0.0 35.6 0.06 0.17 1.47 0.0 0.2 35.7 0.22 0.04 1.15 0.1 0.1 35.6 0.06 0.05 1.14 0.01 0.01 35.7 0.48 0.11 1.37 1.0 1.0 35.7 0.03 0.02 1.07\n\nTable 6: Experiments with different combinations of losses.\n\n13 Results: Results are reported in Table 6. All the combinations containing at least one the two losses led to very similar model quality, where having no loss was much worse. Models with higher values of wload had lower loads on the most overloaded expert.\n\nB Hierachical Mixture Of Experts\n\nIf the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network.3If the hierarchical MoE consists of a groups of b experts each, we denote the primary gating network by Gprimary, the secondary gating networks by (G1, G2..Ga), and the expert networks by (E0,0, E0,1..Ea,b). The output of the MoE is given by:\n\n$$y_{H}=\\sum_{i=1}^{a}\\sum_{j=1}^{b}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}\\cdot E{i,j}(x)$$ $$(12)$$ $$(13)$$ $$(14)$$\n\nOur metrics of expert utilization change to the following:\n\n$$I m p o r t a n c e_{H}(X){i,j}=\\sum{x\\in X}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}$$ $$L o a d{H}(X){i,j}={\\frac{L o a d{p r i m a r y}(X){i}\\cdot L o a d{i}(X^{(i)})_{j}}{|X^{(i)}|}}$$\n\nLoadprimary and Loadi deonte the Load functions for the primary gating network and i th secondary gating network respectively. X(i) denotes the subset of X for which Gprimary(x)i > 0.\n\nIt would seem simpler to let LoadH(X)i,j = Loadi(Xi)j , but this would not have a gradient with respect to the primary gating network, so we use the formulation above.\n\nC 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS C.1 8-MILLION-OPERATIONS-PER-TIMESTEP MODELS Model Architecture: Our model consists of five layers: a word embedding layer, a recurrent Long Short-Term Memory (LSTM) layer (Hochreiter & Schmidhuber, 1997; Gers et al., 2000), a MoE layer, a second LSTM layer, and a softmax layer. The dimensionality of the embedding layer, the number of units in each LSTM layer, and the input and output dimensionality of the MoE layer are all equal to 512. For every layer other than the softmax, we apply drouput (Zaremba et al., 2014) to the layer output, dropping each activation with probability DropP rob, otherwise dividing by (1 − DropP rob). After dropout, the output of the previous layer is added to the layer output.\n\nThis residual connection encourages gradient flow (He et al., 2015).\n\nMoE Layer Architecture: Each expert in the MoE layer is a feed forward network with one ReLU-activated hidden layer of size 1024 and an output layer of size 512. Thus, each expert contains [512 ∗ 1024] + [1024 ∗ 512] = 1M parameters. The output of the MoE layer is passed through a sigmoid function before dropout. We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts.\n\nWe call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers. Thus, each example is processed by exactly 4 experts for a total of 4M ops/timestep. The two LSTM layers contribute 2M ops/timestep each for the desired total of 8M.\n\n3 We have not found the need for deeper hierarchies.\n\nComputationally-Matched Baselines: The MoE-4 model does not employ sparsity, since all 4 experts are always used. In addition, we trained four more computationally-matched baseline models with no sparsity:\n\nMoE-1-Wide: The MoE layer consists of a single \"expert\" containing one ReLU-activated hidden layer of size 4096.\n\nMoE-1-Deep: The MoE layer consists of a single \"expert\" containing four ReLU-activated hidden layers, each with size 1024.\n\n4xLSTM-512: We replace the MoE layer with two additional 512-unit LSTM layers.\n\nLSTM-2048-512: The model contains one 2048-unit LSTM layer (and no MoE). The output of the LSTM is projected down to 512 dimensions (Sak et al., 2014). The next timestep of the LSTM receives the projected output. This is identical to one of the models published in (Jozefowicz et al., 2016). We re-ran it to account for differences in training regimen, and obtained results very similar to the published ones. Training: The models were trained on a cluster of 16 K40 GPUs using the synchronous method described in Section 3. Each batch consisted of a set of sentences totaling roughly 300,000 words. In the interest of time, we limited training to 10 epochs, (27,000 steps). Training took 12-16 hours for all models, except for MoE-4, which took 18 hours (since all the expert computation was performed on only 4 of 16 GPUs). We used the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 1000 training steps, and decreased after that so as to be proportional to the inverse square root of the step number. The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016). For each model, we performed a hyper-parmeter search to find the best dropout probability, in increments of 0.1.\n\nTo ensure balanced expert utilization we set wimportance = 0.1 and wload = 0.1, as described in Section 4 and Appendix A.\n\nResults: We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol. Results are reported in Table 7. For each model, we report the test perplexity, the computational budget, the parameter counts, the value of DropP rob, and the computational efficiency.\n\nModel Test Test ops/timestep #Params excluding Total Drop- TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params P rob per GPU 10 epochs (final) (millions) (billions) (observed) Kneser-Ney 5-gram* 67.6 0.00001 1.8 LSTM-512-512* 54.1 2.4 2.4 0.8 0.1 LSTM-1024-512* 48.2 4.7 4.7 0.8 0.1 LSTM-2048-512* 45.0 43.7 9.4 9.4 0.8 0.1 0.61 LSTM-2048-512 44.7 9.4 9.4 0.8 0.1 1.21 4xLSTM-512 46.0 8.4 8.4 0.8 0.1 1.07 MoE-1-Wide 46.1 8.4 8.4 0.8 0.1 1.29 MoE-1-Deep 45.7 8.4 8.4 0.8 0.1 1.29 MoE-4 45.0 8.4 8.4 0.8 0.1 0.52 MoE-32 39.7 8.4 37.8 0.9 0.1 0.87 MoE-256 35.7 8.6 272.9 1.1 0.1 0.81 MoE-256-h 36.0 8.4 272.9 1.1 0.1 0.89 MoE-1024-h 34.6 8.5 1079.0 1.9 0.2 0.90 MoE-4096-h 34.1 8.9 4303.4 5.1 0.2 0.74 2xLSTM-8192-1024* 34.7 30.6 151.0 151.0 1.8 0.25 1.09 MoE-34M 31.3 33.8 4313.9 6.0 0.3 1.22 MoE-143M 28.0 142.7 4371.1 6.0 0.4 1.56\n\nC.2 More Expensive Models\n\nWe ran two additional models (MoE-34M and MoE-143M) to investigate the effects of adding more computation in the presence of a large MoE layer. These models have computation budgets of 34M and 143M ops/timestep. Similar to the models above, these models use a MoE layer between two LSTM layers. The dimensionality of the embedding layer, and the input and output dimensionality of the MoE layer are set to 1024 instead of 512. For MoE-34M, the LSTM layers have 1024 units.\n\nFor MoE-143M, the LSTM layers have 4096 units and an output projection of size 1024 (Sak et al., 2014). MoE-34M uses a hierarchical MoE layer with 1024 experts, each with a hidden layer of size 2048. MoE-143M uses a hierarchical MoE layer with 256 experts, each with a hidden layer of size 8192. Both models have 4B parameters in the MoE layers. We searched for the best DropP rob for each model, and trained each model for 10 epochs.\n\nThe two models achieved test perplexity of 31.3 and 28.0 respectively, showing that even in the presence of a large MoE, more computation is still useful. Results are reported at the bottom of Table 7. The larger of the two models has a similar computational budget to the best published model from the literature, and training times are similar. Comparing after 10 epochs, our model has a lower test perplexity by 18%.\n\nD 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS Model Architecture: The models are similar in structure to the 8-million-operations-per-timestep models described in the previous section. We vary the number of experts between models, using an ordinary MoE layer with 32 experts and hierarchical MoE layers with 256, 1024, 4096, 16384, 65536 and 131072 experts. For the hierarchical MoE layers, the first level branching factors are 32, 32, 64, 128, 256 and 256, respectively.\n\nTraining: Models are trained on a cluster of 32 Tesla K40 GPUs, except for the last two models, which are trained on clusters of 64 and 128 GPUs so as to have enough memory for all the parameters. For all models, training batch sizes are approximately 2.5 million words. Models are trained once-through over about 100 billion words. We implement several memory optimizations in order to fit up to 1 billion parameters per GPU. First, we do not store the activations of the hidden layers of the experts, but instead recompute them on the backwards pass. Secondly, we modify the optimizer on the expert parameters to require less auxiliary storage: The Adam optimizer (Kingma & Ba, 2015) keeps first and second moment estimates of the perparameter gradients. This triples the required memory. To avoid keeping a first-moment estimator, we set β1 = 0. To reduce the size of the second moment estimator, we replace it with a factored approximation. For a matrix of parameters, instead of maintaining a full matrix of second-moment estimators, we maintain vectors of row-wise and column-wise averages of that matrix. At each step, the matrix of estimators is taken to be the outer product of those two vectors divided by the mean of either one. This technique could similarly be applied to Adagrad (Duchi et al., 2010).\n\nModel Test Test ops/timestep #Params excluding Total TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params per GPU .1 epochs 1 epoch (millions) (billions) (observed) Kneser-Ney 5-gram 67.1 45.3 0.00001 76.0 4xLSTM-512 54.5 47.0 8.4 8.4 0.1 1.23 MoE-32 48.5 40.4 8.4 37.8 0.1 0.83 MoE-256-h 42.8 35.3 8.4 272.9 0.4 1.11 MoE-1024-h 40.3 32.7 8.5 1079.0 1.2 1.14 MoE-4096-h 38.9 30.9 8.6 4303.4 4.4 1.07 MoE-16384-h 38.2 29.7 8.8 17201.0 17.3 0.96 MoE-65536-h 38.2 28.9 9.2 68791.0 68.9 0.72 MoE-131072-h 39.8 29.2 9.7 137577.6 137.7 0.30\n\nResults: We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8. Perplexity after 100 billion training words is 39% lower for the 68-billion-parameter MoE model than for the baseline model. It is notable that the measured computational efficiency of the largest model (0.30 TFLOPS/GPU) is very low compared to the other models. This is likely a result of the fact that, for purposes of comparison to the other models, we did not increase the training batch size proportionally to the number of GPUs. For comparison, we include results for a computationally matched baseline model consisting of 4 LSTMs, and for an unpruned 5-gram model with Kneser-Ney smoothing (Kneser & Ney, 1995).4\n\nE Machine Translation - Experimental Details\n\nModel Architecture for Single Language Pair MoE Models: Our model is a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decrease the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We insert MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). We use an attention mechanism between the encoder and decoder, with the first decoder LSTM receiving output from and providing input for the attention 5. All of the layers in our model have input and output dimensionality of 512. Our LSTM layers have 2048 hidden units, with a 512-dimensional output projection. We add residual connections around all LSTM and MoE layers to encourage gradient flow (He et al., 2015). Similar to GNMT, to effectively deal with rare words, we used subword units (also known as \"wordpieces\") (Schuster & Nakajima, 2012) for inputs and outputs in our system.\n\nWe use a shared source and target vocabulary of 32K wordpieces. We also used the same beam search technique as proposed in (Wu et al., 2016).\n\nWe train models with different numbers of experts in the MoE layers. In addition to a baseline model with no MoE layers, we train models with flat MoE layers containing 32 experts, and models with hierarchical MoE layers containing 512 and 2048 experts. The flat MoE layers use k = 4 and the hierarchical MoE models use k = 2 at each level of the gating network. Thus, each input is processed by exactly 4 experts in each MoE layer. Each expert in the MoE layer is a feed forward network with one hidden layer of size 2048 and ReLU activation. Thus, each expert contains [512 ∗ 2048] + [2048 ∗ 512] = 2M parameters. The output of the MoE layer is passed through a sigmoid function. We use the strictly-balanced gating function described in Appendix F.\n\nModel Architecture for Multilingual MoE Model: We used the same model architecture as for the single-language-pair models, with the following exceptions: We used noisy-top-k gating as described in Section 2.1, not the scheme from Appendix F. The MoE layers in the encoder and decoder are non-hierarchical MoEs with n = 512 experts, and k = 2. Each expert has a larger hidden layer of size 8192. This doubles the amount of computation in the MoE layers, raising the computational budget of the entire model from 85M to 102M ops/timestep. Training: We trained our networks using the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 2000 training steps, held constant for an additional 8000 steps, and decreased after that so as to be proportional to the inverse square root of the step number. For the single-language-pair models, similarly to (Wu et al., 2016), we applied dropout (Zaremba et al., 2014) to the output of all embedding, LSTM and MoE layers, using DropP rob = 0.4. Training was done synchronously on a cluster of up to 64 GPUs as described in section 3. Each training batch consisted of a set of sentence pairs containing roughly 16000 words per GPU.\n\nTo ensure balanced expert utilization we set wimportance = 0.01 and wload = 0.01, as described in Section 4 and Appendix A. Metrics: We evaluated our models using the perplexity and the standard BLEU score metric. We reported tokenized BLEU score as computed by the multi-bleu.pl script, downloaded from the public implementation of Moses (on Github), which was also used in (Luong et al., 2015a). Results: Tables 2, 3 and 4 in Section 5.3 show comparisons of our results to other published methods. Figure 4 shows test perplexity as a function of number of words in the (training data's) source sentences processed for models with different numbers of experts. As can be seen from the Figure, as we increased the number of experts to approach 2048, the test perplexity of our model continued to improve.\n\nWe found that the experts indeed become highly specialized by syntax and/or semantics, as can be seen in Table 9. For example, one expert is used when the indefinite article \"a\" introduces the direct object in a verb phrase indicating importance or leadership.\n\nExpert 381 Expert 752 Expert 2004 ... with researchers , ... ... plays a core ... ... with rapidly growing ... ... to innovation . ... plays a critical ... ... under static conditions ... ... tics researchers . ... provides a legislative ... ... to swift ly ... ... the generation of ... ... play a leading ... ... to dras tically ... ... technology innovations is ... ... assume a leadership ... ... the rapid and ... ... technological innovations , ... ... plays a central ... ... the fast est ... ... support innovation throughout ... ... taken a leading ... ... the Quick Method ... ... role innovation will ... ... established a reconciliation ... ... rec urrent ) ... ... research scienti st ... ... played a vital ... ... provides quick access ... ... promoting innovation where ... ... have a central ... ... of volatile organic ... ... ... ...\n\nDue to some peculiarities in our infrastructure which have since been fixed, at the time we ran some of the machine translation experiments, our models ran faster if every expert received exactly the same batch size. To accommodate this, we used a different gating function which we describe below. Recall that we define the softmax gating function to be:\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ Gσ(x) = Sof tmax(x · Wg) (15) Sparse Gating (alternate formulation): To obtain a sparse gating vector, we multiply Gσ(x) component-wise with a sparse mask M(Gσ(x)) and normalize the output. The mask itself is a function of Gσ(x) and specifies which experts are assigned to each input example:\n\n$$(15)$$ $$G(x){i}=\\frac{G{\\sigma}(x){i}M(G{\\sigma}(x)){i}}{\\sum{j=1}^{n}G_{\\sigma}(x){j}M(G{\\sigma}(x))_{j}}\\tag{1}$$ $$(17)$$\n\n$$(16)$$\n\nTop-K Mask: To implement top-k gating in this formulation, we would let M(v) = T opK(v, k), where:\n\n$$T o p K(v,k){i}=\\begin{cases}1&{\\mathrm{if~}}v{i}{\\mathrm{~is~in~the~top~}}k{\\mathrm{~elements~of~}}v.\\ 0&{\\mathrm{otherwise.}}\\end{cases}$$ 0 otherwise. (17) Batchwise Mask: To force each expert to receive the exact same number of examples, we introduce an alternative mask function, Mbatchwise(X, m), which operates over batches of input vectors.\n\nInstead of keeping the top k values per example, we keep the top m values per expert across the training batch, where m = k|X| n, so that each example is sent to an average of k experts.\n\n$$M_{b a t c h w i s e}(X,m){j,i}=\\begin{cases}1&\\text{if}X{j,i}\\text{is in the top}m\\text{values for to expert}i\\ 0&\\text{otherwise}\\end{cases}$$\n\nAs our experiments suggest and also observed in (Ioffe & Szegedy, 2015), using a batchwise function during training (such as Mbatchwise) requires modifications to the inference when we may not have a large batch of examples. Our solution to this is to train a vector T of per-expert threshold values to approximate the effects of the batchwise mask. We use the following mask at inference time:\n\n$$(18)$$ $$M_{threshold}(x,T){i}=\\begin{cases}1&\\text{if}x{i}>T_{i}\\ 0&\\text{otherwise}\\end{cases}\\tag{1}$$ $$(19)$$\n\nTo learn the threshold values, we apply an additional loss at training time which is minimized when the batchwise mask and the threshold mask are identical.\n\n$$L_{batchwise}(X,T,m)=\\sum_{j=1}^{|X|}\\sum_{i=1}^{n}(M_{threshold}(x,T){i}-M{batchwise}(X,m){j,i})(X{j,i}-T_{i})\\tag{20}$$\n\nG ATTENTION FUNCTION The attention mechanism described in GNMT (Wu et al., 2016) involves a learned \"Attention Function\" A(xi, yj ) which takes a \"source vector\" xi and a \"target vector\" yj , and must be computed for every source time step i and target time step j. In GNMT, the attention function is implemented as a feed forward neural network with a hidden layer of size n. It can be expressed as:\n\n$$A_{G N M T}(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}t a n h((x_{i}U){d}+(y{j}W)_{d})$$\n\nWhere U and W are trainable weight matrices and V is a trainable weight vector.\n\nFor performance reasons, in our models, we used a slightly different attention function:\n\n$$A(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}tanh((x_{i}U){d})tanh((y{j}W)_{d})\\tag{22}$$\n\nWith our attention function, we can simultaneously compute the attention function on multiple source time steps and multiple target time steps using optimized matrix multiplications. We found little difference in quality between the two functions.\n\n$$(21)$$",
+        "embedding": [
+          0.0011764472583308816,
+          -0.018546752631664276,
+          -0.01668562740087509,
+          -0.003858148353174329,
+          0.009172027930617332,
+          0.015837986022233963,
+          -0.013212142512202263,
+          0.06935913860797882,
+          -0.011065399274230003,
+          0.035029683262109756,
+          0.016455290839076042,
+          0.002998990472406149,
+          0.020509224385023117,
+          -0.022867878898978233,
+          0.0126132657751441,
+          -0.002063821302726865,
+          0.02303372137248516,
+          0.015008772723376751,
+          -0.015994615852832794,
+          -0.019071921706199646,
+          -0.008485622704029083,
+          -0.004452418070286512,
+          -0.029298892244696617,
+          -0.004053934942930937,
+          -0.01113910786807537,
+          0.010531017556786537,
+          0.020085405558347702,
+          0.015220683068037033,
+          -0.0586714930832386,
+          -0.0015317423967644572,
+          -0.002752529690042138,
+          -0.0007854498689994216,
+          -0.015248322859406471,
+          -0.012097310274839401,
+          -0.027824735268950462,
+          -0.03810698911547661,
+          0.047799576073884964,
+          -0.00880348775535822,
+          -0.03169440105557442,
+          0.020896190777420998,
+          0.0326526015996933,
+          0.009411578066647053,
+          -0.009347083978354931,
+          -0.00776697089895606,
+          -0.009227308444678783,
+          0.001612360472790897,
+          0.012263152748346329,
+          0.010208545252680779,
+          -0.01241978257894516,
+          -0.031841814517974854,
+          -0.0024001137353479862,
+          -0.020306527614593506,
+          0.015008772723376751,
+          0.0039019123651087284,
+          -0.015156188048422337,
+          -0.008172363974153996,
+          0.0126132657751441,
+          0.048573512583971024,
+          0.05200092867016792,
+          0.04765216261148453,
+          0.038364965468645096,
+          0.01730293035507202,
+          -0.0017505628056824207,
+          -0.00107452308293432,
+          0.031086310744285583,
+          -0.020195966586470604,
+          -0.006030227988958359,
+          0.006214498076587915,
+          -0.016768548637628555,
+          -0.034440018236637115,
+          -0.06047733873128891,
+          0.02084091119468212,
+          0.007868318818509579,
+          -0.015764279291033745,
+          -0.00650011608377099,
+          -0.028635522350668907,
+          0.01830720156431198,
+          0.07252857834100723,
+          0.013009445741772652,
+          0.005095059052109718,
+          0.017883380874991417,
+          0.010816635563969612,
+          -0.015036413446068764,
+          0.01986428163945675,
+          0.024415744468569756,
+          -0.005647868383675814,
+          -0.032431479543447495,
+          -0.06198835000395775,
+          0.008844949305057526,
+          -0.011083826422691345,
+          -0.039323169738054276,
+          0.03440316393971443,
+          0.0019210122991353273,
+          -0.01265933271497488,
+          -0.0008666437352076173,
+          0.03983912244439125,
+          0.03114159032702446,
+          0.0012944950722157955,
+          -0.011793265119194984,
+          0.022093946114182472,
+          -0.002939102705568075,
+          0.0032569682225584984,
+          0.005164160393178463,
+          0.025171250104904175,
+          0.02139372006058693,
+          -0.005099665839225054,
+          -0.007439891807734966,
+          -0.007467532530426979,
+          0.039212606847286224,
+          0.11468950659036636,
+          -0.019606303423643112,
+          -0.0036969122011214495,
+          -0.03038608469069004,
+          -0.0014557312242686749,
+          -0.023678665980696678,
+          -0.015764279291033745,
+          -0.056386549025774,
+          -0.022204507142305374,
+          0.02006697840988636,
+          0.0007134695188142359,
+          0.030109679326415062,
+          0.051153287291526794,
+          0.031473275274038315,
+          -0.0012956467689946294,
+          -0.02800900489091873,
+          0.06519464403390884,
+          -0.02168855257332325,
+          0.025853049010038376,
+          -0.006477082148194313,
+          -0.04636227339506149,
+          -0.0012069669319316745,
+          0.018279561772942543,
+          -0.013875513337552547,
+          -0.031510129570961,
+          0.014760008081793785,
+          0.006771914195269346,
+          0.02799057774245739,
+          0.05008452385663986,
+          -0.029685860499739647,
+          -0.010733714327216148,
+          -0.0009679921204224229,
+          -0.02004855126142502,
+          -0.003448148025199771,
+          -0.012640906497836113,
+          -0.03899148106575012,
+          0.0012334557250142097,
+          0.015543155372142792,
+          0.07444498687982559,
+          -0.022296641021966934,
+          0.029354173690080643,
+          -0.01696203276515007,
+          -0.016519784927368164,
+          0.01671326719224453,
+          0.018887652084231377,
+          -0.008181577548384666,
+          -0.01625259406864643,
+          -0.022130798548460007,
+          0.009333264082670212,
+          -0.04415103793144226,
+          0.0015858716797083616,
+          -0.00802494864910841,
+          -0.008881802670657635,
+          -0.018344055861234665,
+          0.01060472521930933,
+          0.0071035996079444885,
+          0.033481817692518234,
+          0.04308227077126503,
+          -0.006698206067085266,
+          -0.058413516730070114,
+          0.05148497223854065,
+          0.005809104535728693,
+          0.04463013634085655,
+          0.030072826892137527,
+          0.03891777619719505,
+          -0.0029068556614220142,
+          0.01007034257054329,
+          0.010687646456062794,
+          0.008918656967580318,
+          -0.03639327734708786,
+          0.04595687985420227,
+          -0.023181136697530746,
+          0.0117564108222723,
+          0.016556639224290848,
+          0.04440901428461075,
+          0.03666968271136284,
+          0.01958787627518177,
+          0.009793938137590885,
+          0.019716864451766014,
+          -0.011719557456672192,
+          0.010770567692816257,
+          0.02760361135005951,
+          0.017579335719347,
+          0.04927373677492142,
+          0.017081808298826218,
+          0.00880348775535822,
+          -0.00950832013040781,
+          -0.015091693960130215,
+          0.020214393734931946,
+          -0.010365174151957035,
+          0.0126132657751441,
+          0.026387430727481842,
+          -0.03106788359582424,
+          -0.004744946490973234,
+          -0.014068996533751488,
+          -0.003989440388977528,
+          0.00300820404663682,
+          -0.011876186355948448,
+          -0.016694841906428337,
+          0.03066249005496502,
+          -0.01771753840148449,
+          -0.021854395046830177,
+          -0.0038051707670092583,
+          0.00054647505749017,
+          -0.017653044313192368,
+          -0.0017010403098538518,
+          0.01538652554154396,
+          -0.009517533704638481,
+          0.019035067409276962,
+          0.01853753812611103,
+          -0.0024876417592167854,
+          -0.0067212400026619434,
+          0.03226563706994057,
+          0.012631692923605442,
+          -0.003680788679048419,
+          -0.022978439927101135,
+          -0.010236185975372791,
+          -0.03145484998822212,
+          -0.04444586858153343,
+          0.017643829807639122,
+          -0.027179790660738945,
+          0.017579335719347,
+          0.024968553334474564,
+          -0.011102253571152687,
+          -0.012677759863436222,
+          0.02036180905997753,
+          -0.03563777357339859,
+          0.02331012487411499,
+          0.0023482879623770714,
+          -0.023862935602664948,
+          -0.009600454941391945,
+          -0.024913271889090538,
+          -0.01956944912672043,
+          -0.027437768876552582,
+          -0.002099523553624749,
+          -0.0045491596683859825,
+          -0.0019878100138157606,
+          -0.01796630211174488,
+          -0.015856413170695305,
+          -0.011968321166932583,
+          0.013064726255834103,
+          0.024397317320108414,
+          -0.04297171160578728,
+          0.012576411478221416,
+          -0.0023770800326019526,
+          -0.030883613973855972,
+          0.06814295798540115,
+          0.0442984513938427,
+          0.00950832013040781,
+          0.015469446778297424,
+          -0.047541599720716476,
+          0.03315013274550438,
+          0.024765856564044952,
+          -0.05155868083238602,
+          0.047799576073884964,
+          -0.001612360472790897,
+          -0.005776857025921345,
+          0.052885424345731735,
+          -0.004975283518433571,
+          0.0018519111908972263,
+          0.012475063093006611,
+          0.02846967987716198,
+          -0.014833716675639153,
+          -0.03521395102143288,
+          -0.03162069246172905,
+          0.02769574522972107,
+          -0.017312144860625267,
+          -0.03217350319027901,
+          0.0005597193958237767,
+          0.04440901428461075,
+          0.01614203117787838,
+          0.002754833083599806,
+          0.013654389418661594,
+          0.015174615196883678,
+          0.02120945043861866,
+          0.024986980482935905,
+          0.01589326746761799,
+          -0.014004502445459366,
+          9.407835750607774e-05,
+          -0.05056362599134445,
+          0.008273712359368801,
+          -0.020417090505361557,
+          -0.008563937619328499,
+          -0.014704727567732334,
+          -0.005085845477879047,
+          0.015672143548727036,
+          -0.003065788187086582,
+          -0.008149330504238605,
+          0.011949894018471241,
+          0.03455058112740517,
+          -0.03066249005496502,
+          -0.026590127497911453,
+          -0.008250678889453411,
+          -0.0036001706030219793,
+          0.0102730393409729,
+          -0.025558216497302055,
+          0.0033145523630082607,
+          0.004162193275988102,
+          0.029777994379401207,
+          -0.004226687829941511,
+          -0.0012323040282353759,
+          0.02084091119468212,
+          0.006725846324115992,
+          0.008872589096426964,
+          -0.0073661841452121735,
+          -0.016556639224290848,
+          0.008872589096426964,
+          0.0007664470467716455,
+          0.020877765491604805,
+          -0.02331012487411499,
+          -0.024231474846601486,
+          0.037222493439912796,
+          0.015027199871838093,
+          -0.04470384493470192,
+          0.02091461792588234,
+          -0.014336188323795795,
+          0.010328320786356926,
+          -0.012180231511592865,
+          0.0410921573638916,
+          -0.014907424338161945,
+          -0.024028778076171875,
+          0.003388260258361697,
+          -0.04142384231090546,
+          -0.02463686838746071,
+          -0.027253499254584312,
+          0.006085508968681097,
+          -0.02397349663078785,
+          0.005845958366990089,
+          0.0005499300896190107,
+          0.0009127111989073455,
+          -0.004892362281680107,
+          -0.01694360561668873,
+          0.0160959642380476,
+          0.016833042725920677,
+          -0.009052252396941185,
+          -0.013608322478830814,
+          0.009056858718395233,
+          -0.024913271889090538,
+          0.03388721123337746,
+          0.035306088626384735,
+          0.008370454423129559,
+          0.001825422397814691,
+          0.003524159314110875,
+          0.0060578687116503716,
+          -0.01534045860171318,
+          -0.010678432881832123,
+          -0.03687237948179245,
+          -0.04193979874253273,
+          0.013396412134170532,
+          -0.02837754413485527,
+          0.02817484736442566,
+          0.01318450178951025,
+          0.03751732409000397,
+          -0.020675068721175194,
+          0.0005908149178139865,
+          -0.050121378153562546,
+          -0.01739506609737873,
+          -0.0131292212754488,
+          -0.026295294985175133,
+          0.01566293090581894,
+          -0.031491704285144806,
+          -0.017247650772333145,
+          0.026442712172865868,
+          -0.007485959213227034,
+          -0.013479333370923996,
+          -0.01728450506925583,
+          0.003307642415165901,
+          0.05376992002129555,
+          -0.017588550224900246,
+          -0.025650352239608765,
+          0.02463686838746071,
+          -0.03305799514055252,
+          0.0005672053666785359,
+          0.02301529422402382,
+          0.008762027136981487,
+          0.0011442000977694988,
+          0.03316855803132057,
+          -0.0034619681537151337,
+          -0.018758662045001984,
+          0.00160084362141788,
+          0.027751026675105095,
+          -0.00611314969137311,
+          -0.036540694534778595,
+          -0.0026650014333426952,
+          -0.03226563706994057,
+          0.004079272039234638,
+          0.00807101558893919,
+          0.013110794126987457,
+          -0.019090348854660988,
+          -0.005288542248308659,
+          0.008084836415946484,
+          -0.01021775882691145,
+          -0.0018288774881511927,
+          0.011968321166932583,
+          0.032910581678152084,
+          -0.004533036146312952,
+          6.323476554825902e-05,
+          -0.02714293636381626,
+          0.016446076333522797,
+          0.04153440520167351,
+          -0.017183156684041023,
+          0.011636636219918728,
+          -0.012171017937362194,
+          -0.059371720999479294,
+          -0.0513744130730629,
+          0.04098159819841385,
+          0.014833716675639153,
+          -0.017708323895931244,
+          -0.011645849794149399,
+          -0.004203653894364834,
+          -0.037222493439912796,
+          -0.006730453111231327,
+          0.004611351061612368,
+          0.00632966635748744,
+          0.01519304234534502,
+          -0.012788321822881699,
+          0.028230128809809685,
+          0.017026526853442192,
+          0.018251920118927956,
+          0.008001914247870445,
+          -0.05491239205002785,
+          0.016648773103952408,
+          -0.003132586134597659,
+          0.024618441238999367,
+          -0.019809000194072723,
+          -0.005979553796350956,
+          -0.03237619996070862,
+          -0.034716423600912094,
+          0.022757316008210182,
+          -0.026111025363206863,
+          -0.04289800301194191,
+          -0.008757420815527439,
+          0.0358404703438282,
+          -0.0010618545347824693,
+          0.004005563911050558,
+          -0.01921933703124523,
+          -0.021467428654432297,
+          -0.019164055585861206,
+          -0.010503376834094524,
+          -0.013654389418661594,
+          -0.0235128216445446,
+          -0.06497351825237274,
+          -0.008808095008134842,
+          -0.0442984513938427,
+          -0.01725686341524124,
+          0.029114622622728348,
+          -0.010383601300418377,
+          0.028598668053746223,
+          -0.052111491560935974,
+          -0.00685483543202281,
+          -0.04153440520167351,
+          0.008084836415946484,
+          0.025908328592777252,
+          0.023715518414974213,
+          -0.0075274198316037655,
+          0.04319283366203308,
+          0.012134164571762085,
+          -0.030644062906503677,
+          -0.031215298920869827,
+          -0.00568472221493721,
+          -0.013626749627292156,
+          -0.00014914335042703897,
+          -0.024784283712506294,
+          0.06335194408893585,
+          0.02883821912109852,
+          0.05122699588537216,
+          0.02876451052725315,
+          -0.018076865002512932,
+          -0.02616630680859089,
+          -0.028340689837932587,
+          0.006449441891163588,
+          0.05447014421224594,
+          -0.021559562534093857,
+          -0.003945676609873772,
+          -0.013608322478830814,
+          0.0029874735046178102,
+          -0.012327647767961025,
+          0.0146125927567482,
+          0.03449530154466629,
+          -0.012253939174115658,
+          -0.017266077920794487,
+          0.004417867865413427,
+          0.01947731524705887,
+          -0.020232820883393288,
+          0.03545350208878517,
+          -0.005740003194659948,
+          -0.04488811641931534,
+          -0.009545174427330494,
+          0.020011696964502335,
+          0.028193274512887,
+          0.01910877414047718,
+          -0.0073892176151275635,
+          -0.026037318632006645,
+          0.011959107592701912,
+          0.023346979171037674,
+          0.004510002676397562,
+          -0.022775743156671524,
+          0.013774164952337742,
+          0.017597762867808342,
+          -0.006799554452300072,
+          -0.04956856742501259,
+          0.022886306047439575,
+          -0.02894878014922142,
+          0.02600046433508396,
+          -0.003966406919062138,
+          0.01899821311235428,
+          0.001621573930606246,
+          -0.01853753812611103,
+          0.022996867075562477,
+          -0.006675172131508589,
+          0.00989528652280569,
+          -0.0034919120371341705,
+          -0.012134164571762085,
+          -0.0012472759699448943,
+          0.05664452537894249,
+          0.0038535415660589933,
+          0.0047910138964653015,
+          0.0410921573638916,
+          0.013488546945154667,
+          0.013811019249260426,
+          -0.015460233204066753,
+          -0.039028335362672806,
+          0.025576643645763397,
+          -0.019053494557738304,
+          -0.013811019249260426,
+          -0.026903385296463966,
+          -0.028635522350668907,
+          -0.011415512301027775,
+          0.021320011466741562,
+          -0.027106082066893578,
+          -0.014842930249869823,
+          -0.0005844806437380612,
+          0.0168975368142128,
+          -0.0363011434674263,
+          0.009282589890062809,
+          0.019827427342534065,
+          -0.0022642146795988083,
+          0.02426832914352417,
+          -0.03217350319027901,
+          0.0007566577405668795,
+          0.02139372006058693,
+          0.008656072430312634,
+          -0.0001934832544066012,
+          -0.001268006395548582,
+          0.020896190777420998,
+          -0.02893035300076008,
+          -0.00601640809327364,
+          0.0059426999650895596,
+          0.010549443773925304,
+          -0.009379331022500992,
+          -0.01637236960232258,
+          0.009388544596731663,
+          0.0020960685797035694,
+          -0.01022697240114212,
+          0.010190118104219437,
+          -0.020619787275791168,
+          -0.03895463049411774,
+          0.0053944974206388,
+          0.008011127822101116,
+          -0.018979785963892937,
+          0.0069976444356143475,
+          0.0018887651385739446,
+          -0.0028469678945839405,
+          0.026866532862186432,
+          -0.005458991974592209,
+          0.032247208058834076,
+          0.04249260947108269,
+          0.01611439138650894,
+          0.008301353082060814,
+          0.00578146381303668,
+          -0.0067442734725773335,
+          0.02963057905435562,
+          -0.008292139507830143,
+          0.0016273324145004153,
+          0.026018891483545303,
+          0.010881129652261734,
+          -0.006384947337210178,
+          -0.007025284692645073,
+          -0.018251920118927956,
+          -0.0035287661012262106,
+          -0.014750794507563114,
+          0.017892595380544662,
+          0.041129011660814285,
+          -0.00030836393125355244,
+          -0.022867878898978233,
+          0.013470119796693325,
+          -0.02205709181725979,
+          -0.00439483392983675,
+          0.012309220619499683,
+          0.01847304403781891,
+          -0.039875976741313934,
+          -0.010577084496617317,
+          0.023623384535312653,
+          0.021320011466741562,
+          -0.012585625052452087,
+          -0.02876451052725315,
+          0.014022929593920708,
+          0.009185847826302052,
+          0.0167777631431818,
+          -0.002157107926905155,
+          -0.007485959213227034,
+          0.008038768544793129,
+          0.006380340550094843,
+          -0.007928206585347652,
+          -0.011820905841886997,
+          0.018251920118927956,
+          0.01979057304561138,
+          0.0063204532489180565,
+          -0.015644503757357597,
+          -0.023531248793005943,
+          0.024010350927710533,
+          0.03497440367937088,
+          -0.016888324171304703,
+          0.0194036066532135,
+          0.0036347212735563517,
+          0.011056185699999332,
+          0.00874360091984272,
+          -0.009821578860282898,
+          -0.009968994185328484,
+          -0.0004307305789552629,
+          -0.0037061257753521204,
+          -0.011194388382136822,
+          -0.005302362609654665,
+          -0.016860684379935265,
+          -0.012263152748346329,
+          0.053069692105054855,
+          0.010392814874649048,
+          0.014824503101408482,
+          -0.0009841157589107752,
+          -0.019237764179706573,
+          -0.007559667341411114,
+          0.011784051544964314,
+          0.02647956646978855,
+          0.004072362091392279,
+          0.03162069246172905,
+          -0.005058205220848322,
+          0.013479333370923996,
+          -0.024600014090538025,
+          0.0019382876344025135,
+          0.020011696964502335,
+          0.016409222036600113,
+          -0.0023678664583712816,
+          -0.0027502262964844704,
+          0.004544553346931934,
+          0.020896190777420998,
+          -0.007297082804143429,
+          -0.046583399176597595,
+          0.018316414207220078,
+          -0.015745852142572403,
+          0.02664540894329548,
+          0.04234519228339195,
+          0.01513776183128357,
+          -0.000702528515830636,
+          0.030883613973855972,
+          -0.021559562534093857,
+          0.015856413170695305,
+          -0.02577934041619301,
+          -0.027880016714334488,
+          0.014824503101408482,
+          -0.0067442734725773335,
+          0.013921581208705902,
+          0.01074292790144682,
+          -0.018758662045001984,
+          -0.02559507079422474,
+          0.015128548257052898,
+          0.007707083132117987,
+          -0.029685860499739647,
+          0.008421128615736961,
+          0.025060689076781273,
+          0.0011384416138753295,
+          0.01026382576674223,
+          0.004095395561307669,
+          0.01680540293455124,
+          -0.01189461350440979,
+          -0.007601127959787846,
+          0.029114622622728348,
+          -0.010973264463245869,
+          -0.013230569660663605,
+          0.000723258825019002,
+          -0.04853665828704834,
+          -0.007988094352185726,
+          -0.0020649731159210205,
+          -0.02091461792588234,
+          -0.001933680847287178,
+          0.011857759207487106,
+          0.006200677715241909,
+          0.010180904529988766,
+          0.015294390730559826,
+          0.043708790093660355,
+          -0.0009547477238811553,
+          -0.002020057290792465,
+          -0.0018231190042570233,
+          -0.030736196786165237,
+          -0.01744113303720951,
+          -0.011572141200304031,
+          -0.002694945316761732,
+          0.019348325207829475,
+          -0.0029183723963797092,
+          -0.011341804638504982,
+          -0.02292315848171711,
+          0.016657987609505653,
+          0.029372600838541985,
+          0.056902505457401276,
+          0.0031602266244590282,
+          0.043524518609046936,
+          0.0007048318511806428,
+          0.02828541025519371,
+          0.019274618476629257,
+          0.008660678751766682,
+          -0.046104297041893005,
+          0.01223551295697689,
+          0.022130798548460007,
+          0.03217350319027901,
+          0.01805843785405159,
+          -0.047357331961393356,
+          -0.004242811352014542,
+          0.006366520654410124,
+          0.03735148161649704,
+          0.027179790660738945,
+          0.013967648148536682,
+          -0.0034619681537151337,
+          0.022757316008210182,
+          -0.029593724757432938,
+          -0.021080462262034416,
+          -0.0005450354074127972,
+          -0.011931467801332474,
+          0.00437180045992136,
+          -0.018638886511325836,
+          0.029022488743066788,
+          -0.02111731469631195,
+          -0.0356193445622921,
+          -0.006255958694964647,
+          -0.008702139370143414,
+          -0.001766686444170773,
+          -0.002782473573461175,
+          0.009066072292625904,
+          0.028782937675714493,
+          0.004316519480198622,
+          0.027013948187232018,
+          -0.01118517480790615,
+          0.02559507079422474,
+          0.004715002607554197,
+          -0.00549123901873827,
+          0.008296746760606766,
+          -0.018325628712773323,
+          -0.006610678043216467,
+          -0.009351690299808979,
+          -0.007937420159578323,
+          0.00712202675640583,
+          0.006864049006253481,
+          -0.009254949167370796,
+          -0.010927197523415089,
+          0.019551021978259087,
+          0.0032892152667045593,
+          0.010005848482251167,
+          0.026332149282097816,
+          -0.010475736111402512,
+          -0.0011493826750665903,
+          0.024028778076171875,
+          0.002002781955525279,
+          -0.02349439635872841,
+          0.005924272816628218,
+          0.0013843266060575843,
+          0.037222493439912796,
+          -0.021246304735541344,
+          -0.008338207378983498,
+          0.020951472222805023,
+          0.000956475269049406,
+          -0.0028561814688146114,
+          -0.03913889825344086,
+          0.0062052845023572445,
+          -0.013285850174725056,
+          0.01022697240114212,
+          0.007974274456501007,
+          0.016400009393692017,
+          -0.01389394048601389,
+          0.006186857353895903,
+          0.004413261078298092,
+          0.00858236476778984,
+          -0.0075043863616883755,
+          0.018178213387727737,
+          -0.0030151139944791794,
+          -0.05074789375066757,
+          -0.006656745448708534,
+          -0.014971918426454067,
+          0.020509224385023117,
+          0.012825176119804382,
+          -0.004035507794469595,
+          0.020988326519727707,
+          -0.02531866542994976,
+          -0.030128106474876404,
+          0.03499282896518707,
+          -0.03283687308430672,
+          0.014916637912392616,
+          -0.0005496421945281327,
+          -0.032818444073200226,
+          -0.040834181010723114,
+          -0.018961358815431595,
+          -0.00040280219400301576,
+          -0.0052977558225393295,
+          0.004065451677888632,
+          -0.013718884438276291,
+          -0.017109448090195656,
+          -0.02616630680859089,
+          0.02109888754785061,
+          -0.0020039337687194347,
+          0.013682030141353607,
+          0.002022360684350133,
+          -0.012898883782327175,
+          -0.014962704852223396,
+          -0.01801236905157566,
+          -0.012382928282022476,
+          0.01810450479388237,
+          -0.016427649185061455,
+          -0.014262479729950428,
+          -0.017864953726530075,
+          -0.004896969068795443,
+          -0.012935738079249859,
+          0.0010215454967692494,
+          0.03637485206127167,
+          0.03565619885921478,
+          0.009079893119633198,
+          0.018942931666970253,
+          -0.020232820883393288,
+          0.045993734151124954,
+          -0.023936642333865166,
+          0.034532155841588974,
+          0.0001733287499519065,
+          -0.020859338343143463,
+          -0.005606407765299082,
+          0.024194620549678802,
+          -0.016519784927368164,
+          -0.01357146818190813,
+          0.004461631644517183,
+          -0.002443877747282386,
+          0.023549675941467285,
+          0.02159641683101654,
+          0.01447439007461071,
+          -0.004507699050009251,
+          0.02577934041619301,
+          0.004634384531527758,
+          -0.0006092418916523457,
+          0.0303676575422287,
+          0.02493169903755188,
+          0.008642252534627914,
+          -0.030920466408133507,
+          -0.013811019249260426,
+          -0.01584720052778721,
+          0.0071358466520905495,
+          0.012143378145992756,
+          0.001317528891377151,
+          0.0056985425762832165,
+          -0.014354614540934563,
+          0.004228990990668535,
+          -0.008315172977745533,
+          0.008633038960397243,
+          0.02244405820965767,
+          -0.010171690955758095,
+          -0.03093889355659485,
+          -0.028395971283316612,
+          -0.01851911097764969,
+          -0.003362923162057996,
+          -0.00018139055464416742,
+          -0.01666720025241375,
+          0.020582932978868484,
+          -0.0003740100364666432,
+          0.005620228126645088,
+          -0.014032143168151379,
+          0.0027225858066231012,
+          0.005993374157696962,
+          -0.012447422370314598,
+          0.0015064053004607558,
+          0.0055833738297224045,
+          -0.0015156188746914268,
+          -0.041239574551582336,
+          -0.004615957848727703,
+          0.004438598174601793,
+          0.004832474514842033,
+          -0.03455058112740517,
+          -0.03331597521901131,
+          -0.03270788490772247,
+          0.006767307408154011,
+          -0.003961800131946802,
+          -0.023936642333865166,
+          0.0131292212754488,
+          -0.01156292762607336,
+          -0.007416858337819576,
+          -0.023641811683773994,
+          -0.01151686068624258,
+          0.01938517950475216,
+          0.0014292424311861396,
+          0.01685146987438202,
+          -0.008200004696846008,
+          -0.00261663063429296,
+          -0.0204355176538229,
+          0.006993037648499012,
+          -0.03068091720342636,
+          0.0012023602612316608,
+          -0.009913713671267033,
+          -0.0005608710926026106,
+          -0.026921812444925308,
+          0.0022503945510834455,
+          0.0044109574519097805,
+          0.013746525160968304,
+          -0.03534294292330742,
+          0.0065646106377244,
+          0.011194388382136822,
+          0.016630345955491066,
+          -0.00720955478027463,
+          0.03244990482926369,
+          0.01094562467187643,
+          -0.005482025444507599,
+          0.03132586181163788,
+          0.05682879686355591,
+          0.022480912506580353,
+          -0.03364766016602516,
+          -0.01394922100007534,
+          -0.012198658660054207,
+          0.013811019249260426,
+          -0.014723154716193676,
+          -0.010770567692816257,
+          -0.008771240711212158,
+          0.023457542061805725,
+          -0.005362250376492739,
+          0.0075274198316037655,
+          0.04020766168832779,
+          -0.00880348775535822,
+          0.009591241367161274,
+          0.006417194847017527,
+          -0.012327647767961025,
+          0.00601640809327364,
+          -0.00658764410763979,
+          0.01710023358464241,
+          0.030920466408133507,
+          0.013534614816308022,
+          0.023328552022576332,
+          -0.04205036163330078,
+          0.0204355176538229,
+          0.010503376834094524,
+          0.006960790604352951,
+          0.022407203912734985,
+          0.028801364824175835,
+          0.016086751595139503,
+          -0.007195734418928623,
+          -0.026442712172865868,
+          -0.036337997764348984,
+          0.0020718832965940237,
+          -0.03373979404568672,
+          -0.0034435412380844355,
+          -0.014262479729950428,
+          0.023144282400608063,
+          0.004795620683580637,
+          -0.019458888098597527,
+          -0.013534614816308022,
+          0.006960790604352951,
+          -0.02091461792588234,
+          -0.004961463622748852,
+          -0.009867645800113678,
+          -0.021522708237171173,
+          -0.024102484807372093,
+          0.02120945043861866,
+          -0.009351690299808979,
+          -0.029298892244696617,
+          0.010125624015927315,
+          0.001757472869940102,
+          0.0003642207011580467,
+          -0.0007860257173888385,
+          -0.020509224385023117,
+          -0.03943372890353203,
+          -0.02082248404622078,
+          0.0025935969315469265,
+          0.032910581678152084,
+          -0.026313722133636475,
+          0.03554563969373703,
+          -0.007559667341411114,
+          0.007508993148803711,
+          -0.0023701698519289494,
+          0.009185847826302052,
+          -0.01634472794830799,
+          -0.038549236953258514,
+          0.01785574108362198,
+          -0.0010491859866306186,
+          0.026111025363206863,
+          0.006334273144602776,
+          -0.027474623173475266,
+          -0.0058275312185287476,
+          0.025742486119270325,
+          0.005048991646617651,
+          -0.013737311586737633,
+          -0.007375397253781557,
+          -0.005320789758116007,
+          0.011526074260473251,
+          -0.04157125949859619,
+          -0.003858148353174329,
+          0.06368362903594971,
+          -0.013276636600494385,
+          0.02732720598578453,
+          0.019090348854660988,
+          -0.03283687308430672,
+          -0.04824182391166687,
+          -0.021946528926491737,
+          -0.001739045954309404,
+          -0.008467195555567741,
+          -0.024342035874724388,
+          0.019661584869027138,
+          -0.002150197746232152,
+          -0.001446517650038004,
+          0.005910452920943499,
+          -0.0042727552354335785,
+          -0.026240015402436256,
+          0.0004845719086006284,
+          -0.015275963582098484,
+          -0.0006691296002827585,
+          -0.03466114401817322,
+          -0.02874608337879181,
+          0.02319956384599209,
+          0.014935065060853958,
+          -0.0055833738297224045,
+          -0.0008309414843097329,
+          -0.013995288871228695,
+          -0.0037521931808441877,
+          0.017579335719347,
+          -0.0010486101964488626,
+          0.012014389038085938,
+          0.00955438707023859,
+          0.031768109649419785,
+          0.0035517998039722443,
+          0.003777530277147889,
+          -0.049236882477998734,
+          -0.011673489585518837,
+          0.02760361135005951,
+          0.003952586557716131,
+          -0.017957089468836784,
+          0.0008326690294779837,
+          -0.00437180045992136,
+          -0.009517533704638481,
+          -0.038070134818553925,
+          0.030146533623337746,
+          0.046583399176597595,
+          -0.0022319676354527473,
+          -0.003123372560366988,
+          0.019256191328167915,
+          5.466009952215245e-06,
+          0.010300680063664913,
+          -0.005417531356215477,
+          -0.012336861342191696,
+          0.030902039259672165,
+          0.005905846133828163,
+          -0.00028720172122120857,
+          0.014778435230255127,
+          -0.009402364492416382,
+          -0.0024576981086283922,
+          -0.02561349794268608,
+          -0.0004474300367292017,
+          -0.028340689837932587,
+          0.006601464468985796,
+          0.0006265172269195318,
+          0.006467869039624929,
+          0.010660005733370781,
+          0.029391027987003326,
+          0.03668811172246933,
+          0.016768548637628555,
+          -0.00218590022996068,
+          -0.010917983949184418,
+          0.029980691149830818,
+          0.022794170305132866,
+          0.009922927245497704,
+          0.022812597453594208,
+          -0.031970806419849396,
+          -0.010825849138200283,
+          0.01065079215914011,
+          0.0015294391196221113,
+          -0.0008384274551644921,
+          0.008324386551976204,
+          0.009812365286052227,
+          0.0049061826430261135,
+          -0.005882812198251486,
+          0.016105176880955696,
+          0.007347756996750832,
+          0.014953491277992725,
+          -0.0020557595416903496,
+          -0.004404047504067421,
+          0.009738657623529434,
+          -0.012336861342191696,
+          0.03517710044980049,
+          -0.018712595105171204,
+          -0.014253266155719757,
+          0.031123163178563118,
+          0.10658163577318192,
+          0.0036278110928833485,
+          -0.014741581864655018,
+          -0.02034338191151619,
+          -0.0010428517125546932,
+          0.0235128216445446,
+          0.009922927245497704,
+          0.02972271293401718,
+          0.009204274974763393,
+          0.0026488779112696648,
+          0.02426832914352417,
+          0.029649006202816963,
+          0.0031072490382939577,
+          0.0006656745099462569,
+          -0.024950126186013222,
+          0.008715960197150707,
+          0.002158259740099311,
+          -0.005979553796350956,
+          -0.005380677059292793,
+          0.010365174151957035,
+          0.017551695927977562,
+          0.005431351251900196,
+          -0.004035507794469595,
+          -0.011820905841886997,
+          -0.016353942453861237,
+          0.005458991974592209,
+          -0.01881394349038601,
+          -0.009411578066647053,
+          -0.017984729260206223,
+          -0.009784724563360214,
+          0.00013971391308587044,
+          -0.003088821889832616,
+          -0.005947306752204895,
+          0.013580681756138802,
+          -0.019698437303304672,
+          0.01977214589715004,
+          -0.008522477000951767,
+          -0.00026863077073358,
+          0.0012587928213179111,
+          -0.010051915422081947,
+          0.017772819846868515,
+          0.0021167988888919353,
+          -0.029925409704446793,
+          0.013635963201522827,
+          0.008204611018300056,
+          -0.005772250704467297,
+          0.02052765153348446,
+          -0.03344496339559555,
+          0.019809000194072723,
+          -0.0013981468509882689,
+          -0.015561581589281559,
+          0.0040309010073542595,
+          -0.018408549949526787,
+          0.005997980944812298,
+          0.006831801496446133,
+          0.006361913867294788,
+          -0.0014879783848300576,
+          0.009075285866856575,
+          0.005634048022329807,
+          -0.0016054504085332155,
+          -0.002223905874416232,
+          0.006767307408154011,
+          -0.0011056186631321907,
+          -0.00016728240007068962,
+          -0.014824503101408482,
+          -0.011811692267656326,
+          -0.0002376792108407244,
+          -0.00412073265761137,
+          -0.0037452830001711845,
+          -0.01739506609737873,
+          0.003547193016856909,
+          0.00501674460247159,
+          0.013488546945154667,
+          -0.002655788091942668,
+          -0.010733714327216148,
+          0.0019117988413199782,
+          0.007799217943102121,
+          -0.014216412790119648,
+          -0.005099665839225054,
+          -0.030902039259672165,
+          -0.008646858856081963,
+          0.02196495607495308,
+          0.005882812198251486,
+          -0.025152822956442833,
+          -0.021062035113573074,
+          0.004450114909559488,
+          -0.016446076333522797,
+          0.006725846324115992,
+          0.007163487374782562,
+          -0.013046300038695335,
+          0.005578767042607069,
+          -0.010881129652261734,
+          -0.0031602266244590282,
+          -0.0029644398018717766,
+          -0.00865146517753601,
+          0.000947261753026396,
+          -0.010770567692816257,
+          -0.0020465462002903223,
+          -0.012410569004714489,
+          -0.014437536709010601,
+          -0.05399104207754135,
+          0.008223038166761398,
+          0.01753326877951622,
+          -0.0027686532121151686,
+          -0.024802710860967636,
+          -0.013718884438276291,
+          0.013820232823491096,
+          -0.0038788786623626947,
+          -0.01867574080824852,
+          0.005067418795078993,
+          0.01657506637275219,
+          -0.008780454285442829,
+          -0.006108542904257774,
+          -0.014013716019690037,
+          0.027179790660738945,
+          0.018076865002512932,
+          0.02874608337879181,
+          -0.018795516341924667,
+          0.021909674629569054,
+          0.02284945175051689,
+          -0.026866532862186432,
+          -0.010291466489434242,
+          0.03385035693645477,
+          0.01045730896294117,
+          0.004989103879779577,
+          0.006753487046808004,
+          0.0011989051708951592,
+          -0.0346979983150959,
+          -0.007002251222729683,
+          0.0020373326260596514,
+          -0.031196871772408485,
+          0.022407203912734985,
+          0.014898210763931274,
+          0.006449441891163588,
+          0.01332270447164774,
+          -0.014511244371533394,
+          0.010531017556786537,
+          0.015368098393082619,
+          0.021191023290157318,
+          -0.0071266330778598785,
+          -0.010696860030293465,
+          0.013101580552756786,
+          0.01737663894891739,
+          -0.013672816567122936,
+          0.004544553346931934,
+          0.014750794507563114,
+          -0.018482258543372154,
+          0.03764631226658821,
+          -0.010153263807296753,
+          0.006923936307430267,
+          -0.013460906222462654,
+          -0.019698437303304672,
+          0.03283687308430672,
+          -0.0032362379133701324,
+          0.000820000481326133,
+          -0.024507878348231316,
+          0.009241129271686077,
+          0.005002924241125584,
+          0.019753718748688698,
+          0.014603379182517529,
+          0.02646113932132721,
+          0.023162709549069405,
+          0.004090788774192333,
+          -0.02867237664759159,
+          -0.0013992985477671027,
+          -0.020232820883393288,
+          0.00740303797647357,
+          -0.06250430643558502,
+          -0.026516418904066086,
+          0.017708323895931244,
+          -0.01956944912672043,
+          -0.006094722542911768,
+          -0.008955510333180428,
+          0.0022780350409448147,
+          0.025373946875333786,
+          -0.017127875238656998,
+          0.014962704852223396,
+          6.658185156993568e-05,
+          0.011719557456672192,
+          -0.013202928937971592,
+          0.011093039996922016,
+          -0.014326974749565125,
+          -0.04050249606370926,
+          0.01332270447164774,
+          0.035582493990659714,
+          0.02963057905435562,
+          -0.016565851867198944,
+          0.02360495738685131,
+          -0.01899821311235428,
+          0.013276636600494385,
+          0.005283935461193323,
+          -0.009033825248479843,
+          -0.0034020806197077036,
+          0.0004123036051169038,
+          -0.009545174427330494,
+          -0.013046300038695335,
+          -0.006099329330027103,
+          -0.027179790660738945,
+          0.003427417716011405,
+          -0.034145187586545944,
+          -0.019090348854660988,
+          0.018942931666970253,
+          0.0012230905704200268,
+          -0.009683376178145409,
+          0.027345633134245872,
+          -0.04164496809244156,
+          0.005288542248308659,
+          -0.026424285024404526,
+          0.02657170034945011,
+          -0.025834621861577034,
+          0.008877196349203587,
+          -0.016750121489167213,
+          -0.011480006389319897,
+          0.020656641572713852,
+          -0.0010278797708451748,
+          0.0063434867188334465,
+          -0.011618209071457386,
+          -0.03274473920464516,
+          -0.0045583732426166534,
+          0.02979642152786255,
+          -0.010917983949184418,
+          0.005915059708058834,
+          0.021835967898368835,
+          -0.005283935461193323,
+          -0.01428090687841177,
+          -0.00246000150218606,
+          0.007011464796960354,
+          -0.04186609014868736,
+          0.012769894674420357,
+          0.01899821311235428,
+          0.01204202976077795,
+          0.008642252534627914,
+          0.0056985425762832165,
+          0.013626749627292156,
+          0.006085508968681097,
+          -0.006219104863703251,
+          0.048684071749448776,
+          0.0002988625201396644,
+          -0.00037055497523397207,
+          -0.004710395820438862,
+          0.0043902271427214146,
+          0.0002775563334580511,
+          -0.009102926589548588,
+          0.016427649185061455,
+          0.0021617147140204906,
+          0.00016742636216804385,
+          0.007117419969290495,
+          0.006062475498765707,
+          -0.027953723445534706,
+          0.023254845291376114,
+          -0.029980691149830818,
+          0.00029051280580461025,
+          0.03611687570810318,
+          -0.016105176880955696,
+          0.000605211011134088,
+          0.007200341206043959,
+          -0.0022573047317564487,
+          0.016722481697797775,
+          -0.017616190016269684,
+          -0.024673722684383392,
+          0.02120945043861866,
+          0.022186079993844032,
+          0.013663602992892265,
+          0.03326069191098213,
+          -0.020656641572713852,
+          0.02082248404622078,
+          -0.011940681375563145,
+          -0.009420791640877724,
+          0.004238204564899206,
+          0.0032753951381891966,
+          -0.01591169461607933,
+          0.018270347267389297,
+          -0.017708323895931244,
+          -0.0023909001611173153,
+          -0.004590620752424002,
+          -0.016040682792663574,
+          -0.0009904500329867005,
+          -0.010319107212126255,
+          -0.009370117448270321,
+          -0.0026028105057775974,
+          -0.013921581208705902,
+          0.10665534436702728,
+          0.0005424441187642515,
+          -0.0137557378038764,
+          0.014824503101408482,
+          -0.0033352829050272703,
+          -0.014492817223072052,
+          -0.004192137159407139,
+          0.001406208612024784,
+          -0.00640337448567152,
+          -0.007882139645516872,
+          -0.013608322478830814,
+          -0.0004266996693331748,
+          0.001249579363502562,
+          0.01026382576674223,
+          -0.009139780886471272,
+          -0.029188331216573715,
+          0.009420791640877724,
+          -0.02970428578555584,
+          0.01204202976077795,
+          0.005081238690763712,
+          0.015091693960130215,
+          0.006647531874477863,
+          0.009978207759559155,
+          -0.007177307270467281,
+          -0.013737311586737633,
+          0.031178444623947144,
+          -0.007098992820829153,
+          0.0022791866213083267,
+          -0.0024254508316516876,
+          -0.010835062712430954,
+          -0.012640906497836113,
+          -0.02196495607495308,
+          0.024950126186013222,
+          -0.01616045832633972,
+          0.007181914057582617,
+          0.006338879931718111,
+          0.00218590022996068,
+          0.030164960771799088,
+          0.0008608853095211089,
+          -0.02006697840988636,
+          0.00620989128947258,
+          -0.035766761749982834,
+          -0.005260901991277933,
+          0.019937988370656967,
+          -0.014345401898026466,
+          -0.04079732671380043,
+          -0.008923263289034367,
+          -0.009116746485233307,
+          0.0030335411429405212,
+          0.00068352569360286,
+          -0.014686300419270992,
+          0.003503429004922509,
+          -0.003699215594679117,
+          0.018657313659787178,
+          0.008826522156596184,
+          -0.0077485437504947186,
+          0.002402417128905654,
+          -0.0002808674471452832,
+          0.006454048678278923,
+          -0.013313490897417068,
+          0.005776857025921345,
+          -0.019827427342534065,
+          0.018509898334741592,
+          -0.008144724182784557,
+          -0.0204355176538229,
+          0.005095059052109718,
+          0.014253266155719757,
+          0.006592250894755125,
+          -0.003673878498375416,
+          -0.008200004696846008,
+          0.014068996533751488,
+          0.023678665980696678,
+          0.014124277979135513,
+          0.018657313659787178,
+          0.03781215474009514,
+          0.001152837765403092,
+          0.032247208058834076,
+          -0.02255461923778057,
+          0.018592819571495056,
+          -0.04116586595773697,
+          -0.0009167420794256032,
+          -0.009932140819728374,
+          -0.005638654809445143,
+          -0.0016238773241639137,
+          0.02301529422402382,
+          -0.014013716019690037,
+          0.0029644398018717766,
+          0.015985403209924698,
+          -0.010503376834094524,
+          0.012410569004714489,
+          0.01132337749004364,
+          -0.01828877441585064,
+          0.013479333370923996,
+          -0.0068502286449074745,
+          0.004818654619157314,
+          -0.027916869148612022,
+          0.005832138005644083,
+          -0.004450114909559488,
+          -0.005108879413455725,
+          -0.0015386525774374604,
+          -0.0016803100006654859,
+          -0.008273712359368801,
+          -0.009282589890062809,
+          -0.035011254251003265,
+          -0.005270115565508604,
+          -5.72963799641002e-05,
+          -0.006970004178583622,
+          0.024821138009428978,
+          0.029096195474267006,
+          0.011811692267656326,
+          0.00695157703012228,
+          0.000879312283359468,
+          0.012815962545573711,
+          0.005578767042607069,
+          0.0234022606164217,
+          -0.02579776756465435,
+          0.00412764260545373,
+          -0.007140453439205885,
+          -0.013672816567122936,
+          0.013728098012506962,
+          -0.004323429428040981,
+          -0.013350344263017178,
+          0.027769453823566437,
+          -0.02026967518031597,
+          0.007605734746903181,
+          -0.00029713500407524407,
+          -0.002948316279798746,
+          -0.0031302827410399914,
+          -0.01794787496328354,
+          0.0015881750732660294,
+          -0.00242314743809402,
+          -0.026387430727481842,
+          -0.0030128106009215117,
+          -0.008540904149413109,
+          -0.02244405820965767,
+          0.02310742810368538,
+          0.01703573949635029,
+          -0.01623416692018509,
+          0.0262584425508976,
+          -0.006808768026530743,
+          -0.013700457289814949,
+          -0.01228157989680767,
+          -0.0017805066891014576,
+          -0.004086181987076998,
+          0.009720230475068092,
+          -0.019937988370656967,
+          -0.011461579240858555,
+          0.0015121637843549252,
+          -0.02426832914352417,
+          0.015598435886204243,
+          0.017588550224900246,
+          -0.013433266431093216,
+          0.029483163729310036,
+          -0.0046320813708007336,
+          0.008287533186376095,
+          0.017800459638237953,
+          0.009738657623529434,
+          -0.019735291600227356,
+          -0.022609900683164597,
+          0.012788321822881699,
+          0.0051687671802937984,
+          -0.0065646106377244,
+          0.00027035828679800034,
+          -0.02472900226712227,
+          0.0009691438172012568,
+          0.007711689919233322,
+          0.001163778710179031,
+          0.0027133722323924303,
+          0.0036485414020717144,
+          0.0014119670959189534,
+          0.0021525011397898197,
+          -0.007587307598441839,
+          0.006131576374173164,
+          -0.02826698310673237,
+          0.013414839282631874,
+          0.004325732588768005,
+          0.002100675366818905,
+          -0.03475327789783478,
+          -0.001816208939999342,
+          -0.0011989051708951592,
+          -0.00821843184530735,
+          -0.012106523849070072,
+          0.023899788036942482,
+          0.021264731884002686,
+          0.015036413446068764,
+          0.025871476158499718,
+          -0.010816635563969612,
+          0.013746525160968304,
+          -0.011212815530598164,
+          0.018500683829188347,
+          -0.0041253394447267056,
+          -0.010328320786356926,
+          -0.008292139507830143,
+          0.03777530416846275,
+          -0.010171690955758095,
+          0.01260405220091343,
+          0.0024830352049320936,
+          0.023733945563435555,
+          0.0015098603907972574,
+          0.0005044384743086994,
+          0.002237726002931595,
+          -0.008697533048689365,
+          -0.0059196664951741695,
+          0.007513599935919046,
+          0.06983824074268341,
+          -0.011028545908629894,
+          0.005744609981775284,
+          0.010448095388710499,
+          -0.007637981791049242,
+          -0.006034834776073694,
+          -0.005763037130236626,
+          -0.030422938987612724,
+          0.01685146987438202,
+          -0.0017344391671940684,
+          -0.01920090988278389,
+          0.013580681756138802,
+          0.017505627125501633,
+          0.018030796200037003,
+          0.000927107292227447,
+          0.017726751044392586,
+          -0.005472812335938215,
+          0.0155800087377429,
+          0.02703237533569336,
+          0.009411578066647053,
+          -0.008153936825692654,
+          -0.009545174427330494,
+          0.007016071584075689,
+          0.0010779781732708216,
+          -0.01228157989680767,
+          0.003489608643576503,
+          -0.015229896642267704,
+          0.026866532862186432,
+          -0.006025621201843023,
+          0.04182923585176468,
+          -0.012097310274839401,
+          -0.00623292475938797,
+          -0.0011344107333570719,
+          -0.007260228972882032,
+          0.011885399930179119,
+          0.005629441235214472,
+          0.008748207241296768,
+          -0.0021835968364030123,
+          -0.014262479729950428,
+          0.018040010705590248,
+          0.002409327309578657,
+          -0.00040021090535447,
+          0.004966070409864187,
+          -0.00466893520206213,
+          -0.00474955327808857,
+          0.012622479349374771,
+          0.009913713671267033,
+          -0.008614611811935902,
+          0.005399104207754135,
+          -0.007149667013436556,
+          -0.015561581589281559,
+          -0.01065079215914011,
+          0.002605113899335265,
+          -0.008734387345612049,
+          -0.007370790932327509,
+          0.0038604517467319965,
+          0.020472371950745583,
+          0.007168094161897898,
+          0.012134164571762085,
+          -0.020122257992625237,
+          -0.0006063627079129219,
+          0.017266077920794487,
+          -0.010448095388710499,
+          0.001591630163602531,
+          0.004058541730046272,
+          0.011820905841886997,
+          -0.013147647492587566,
+          0.006200677715241909,
+          0.010144051164388657,
+          0.0037153391167521477,
+          0.01618809998035431,
+          0.006154610309749842,
+          0.003945676609873772,
+          0.004721913021057844,
+          -0.0014246356440708041,
+          0.0003927249344997108,
+          -0.009264162741601467,
+          1.6843408957356587e-05,
+          -0.005025957711040974,
+          -0.0023102823179215193,
+          -0.015451019629836082,
+          0.009738657623529434,
+          0.012217085808515549,
+          0.01680540293455124,
+          -0.026626981794834137,
+          0.011959107592701912,
+          -0.010503376834094524,
+          0.02741934172809124,
+          0.018021583557128906,
+          -0.02618473395705223,
+          0.016105176880955696,
+          -0.01079820841550827,
+          0.015626076608896255,
+          0.0018208156106993556,
+          -0.01675933599472046,
+          0.004528429359197617,
+          -0.004590620752424002,
+          0.004249721299856901,
+          -0.01833484135568142,
+          -0.024305181577801704,
+          0.021651698276400566,
+          0.0011275006690993905,
+          0.0155800087377429,
+          0.005205621011555195,
+          0.013700457289814949,
+          0.0007929358398541808,
+          0.01787416823208332,
+          0.007333936635404825,
+          0.005366857163608074,
+          -0.011397085152566433,
+          0.008909443393349648,
+          -0.0047910138964653015,
+          -0.007969667203724384,
+          -0.006785734090954065,
+          -0.010567870922386646,
+          -0.0005983008886687458,
+          -0.01065079215914011,
+          -0.024986980482935905,
+          -0.018565179780125618,
+          -0.007826858200132847,
+          0.02378922700881958,
+          -0.000312682765070349,
+          -0.0018092987593263388,
+          -0.03547193109989166,
+          -0.001534045790322125,
+          -0.003911125939339399,
+          -0.01997484266757965,
+          -0.003017417388036847,
+          -0.0205460786819458,
+          -0.017570123076438904,
+          -0.00611314969137311,
+          -0.02399192377924919,
+          -0.005035171285271645,
+          -0.018122931942343712,
+          -0.01428090687841177,
+          0.006435621529817581,
+          0.005002924241125584,
+          0.01265011914074421,
+          0.028617095202207565,
+          0.01150764711201191,
+          -0.007347756996750832,
+          -0.014686300419270992,
+          -0.01299101859331131,
+          -0.014197985641658306,
+          0.011258882470428944,
+          0.0096465228125453,
+          0.013995288871228695,
+          -1.3226394912635442e-05,
+          0.010844276286661625,
+          0.00018714898033067584,
+          0.02026967518031597,
+          -0.02924361266195774,
+          -0.03038608469069004,
+          0.0026511813048273325,
+          -0.000183118085260503,
+          -0.004219777882099152,
+          -0.0033053390216082335,
+          -0.005500452592968941,
+          -0.00932405050843954,
+          -0.015165401622653008,
+          -0.006292812526226044,
+          2.4365357603528537e-05,
+          -0.006638318300247192,
+          0.024526305496692657,
+          0.020103830844163895,
+          0.006007194519042969,
+          0.0016423043562099338,
+          0.01881394349038601,
+          0.006804161239415407,
+          0.00805719569325447,
+          -0.00289073190651834,
+          0.0043050022795796394,
+          0.0062283179722726345,
+          0.005131912883371115,
+          -0.004648204892873764,
+          0.02905934303998947,
+          0.0014799166237935424,
+          0.011995961889624596,
+          0.014446749351918697,
+          0.03256046772003174,
+          -0.0013359558070078492,
+          0.025963610038161278,
+          0.012161804363131523,
+          -0.009655735455453396,
+          -0.01611439138650894,
+          0.008301353082060814,
+          -0.008135510608553886,
+          0.006251351907849312,
+          0.01632630079984665,
+          -0.02071192115545273,
+          -0.007200341206043959,
+          0.03394249081611633,
+          -0.01389394048601389,
+          -0.012788321822881699,
+          0.017745178192853928,
+          0.007370790932327509,
+          -0.003231631126254797,
+          -0.0008505201549269259,
+          0.004609047435224056,
+          0.001758624566718936,
+          -0.004152979701757431,
+          0.019348325207829475,
+          0.004309609066694975,
+          0.003989440388977528,
+          0.012530344538390636,
+          -0.0019970235880464315,
+          -0.0076103415340185165,
+          -0.0016745515167713165,
+          0.01662113331258297,
+          0.004222081042826176,
+          -0.017791246995329857,
+          -0.006887082476168871,
+          0.013811019249260426,
+          -0.0041575864888727665,
+          -0.004590620752424002,
+          0.025152822956442833,
+          -0.002501462120562792,
+          -0.012576411478221416,
+          0.0011338349431753159,
+          0.006624497938901186,
+          -0.003643934614956379,
+          0.011986748315393925,
+          0.023439114913344383,
+          -0.003038147697225213,
+          -0.0018116021528840065,
+          -0.009056858718395233,
+          0.007923600263893604,
+          -0.008513263426721096,
+          0.013424052856862545,
+          -0.029372600838541985,
+          0.01002427563071251,
+          -0.012327647767961025,
+          0.005095059052109718,
+          0.012318434193730354,
+          0.01768989861011505,
+          0.0204355176538229,
+          0.013801805675029755,
+          -0.007043711841106415,
+          -0.011737984605133533,
+          -0.012917310930788517,
+          -0.0013843266060575843,
+          -0.009172027930617332,
+          -0.01901664026081562,
+          0.004970677196979523,
+          0.019937988370656967,
+          0.017127875238656998,
+          0.009259555488824844,
+          -0.015027199871838093,
+          0.019993269816040993,
+          -2.9781880584778264e-05,
+          0.007711689919233322,
+          0.0028976420871913433,
+          -0.01280674897134304,
+          -0.0020476977806538343,
+          -0.015828773379325867,
+          0.016400009393692017,
+          -0.014326974749565125,
+          0.01628023386001587,
+          0.030349230393767357,
+          -0.01337798498570919,
+          -0.013967648148536682,
+          -0.014262479729950428,
+          -0.00429118238389492,
+          -0.019035067409276962,
+          0.026442712172865868,
+          0.018076865002512932,
+          -0.016980459913611412,
+          -0.014741581864655018,
+          -0.019937988370656967,
+          0.01920090988278389,
+          0.015045627020299435,
+          0.007108206395059824,
+          0.01921933703124523,
+          0.02026967518031597,
+          0.006182250566780567,
+          0.0036761818919330835,
+          -0.01094562467187643,
+          0.0010889191180467606,
+          0.017339784651994705,
+          0.0016388492658734322,
+          -0.011111467145383358,
+          -0.017044954001903534,
+          0.0039963508024811745,
+          -0.02109888754785061,
+          0.008831128478050232,
+          0.006366520654410124,
+          -0.011480006389319897,
+          0.016040682792663574,
+          -0.024544732645154,
+          0.0073892176151275635,
+          -0.012023602612316608,
+          0.0029736533761024475,
+          -0.036724966019392014,
+          0.0002545226307120174,
+          0.003602473996579647,
+          0.006246745120733976,
+          0.023365406319499016,
+          -0.02990698255598545,
+          -0.01733057200908661,
+          0.003339889459311962,
+          -0.006435621529817581,
+          0.003643934614956379,
+          -0.014916637912392616,
+          0.003385957097634673,
+          -0.00484629487618804,
+          -0.0011177113046869636,
+          0.0036900020204484463,
+          -0.003945676609873772,
+          0.009319443255662918,
+          0.015128548257052898,
+          -0.017026526853442192,
+          -0.00016339545254595578,
+          0.017643829807639122,
+          0.014446749351918697,
+          -0.02255461923778057,
+          0.006688992492854595,
+          -0.013626749627292156,
+          0.006454048678278923,
+          -0.027069229632616043,
+          0.011222029104828835,
+          -0.005302362609654665,
+          0.01481528952717781,
+          -0.0031302827410399914,
+          -0.004351069685071707,
+          -0.0076425885781645775,
+          0.014713941141963005,
+          0.004574496764689684,
+          0.007568880915641785,
+          0.017201581969857216,
+          -0.018297987058758736,
+          0.012364501133561134,
+          -0.007158880587667227,
+          -0.016473717987537384,
+          0.006665959022939205,
+          -0.00675809383392334,
+          0.002443877747282386,
+          -0.02331012487411499,
+          -0.005440564826130867,
+          -0.0011453517945483327,
+          -0.019643157720565796,
+          0.008849555626511574,
+          -0.02999911829829216,
+          0.0001314361725235358,
+          -0.01700809970498085,
+          -0.0031072490382939577,
+          -0.007591914385557175,
+          -0.011655062437057495,
+          -0.02915147691965103,
+          -0.009406971745193005,
+          0.023328552022576332,
+          -0.009729444049298763,
+          0.011774837970733643,
+          0.0009916017297655344,
+          0.007914386689662933,
+          0.00585517194122076,
+          -0.0007952392334118485,
+          0.008522477000951767,
+          0.016704054549336433,
+          -0.014400682412087917,
+          -0.0012645513052120805,
+          0.005353036802262068,
+          0.024600014090538025,
+          -0.019422033801674843,
+          -0.01318450178951025,
+          -0.00865146517753601,
+          0.0019198606023564935,
+          0.017643829807639122,
+          -0.02102518081665039,
+          -0.002324102446436882,
+          -0.001895675202831626,
+          0.005486632231622934,
+          -0.006841015070676804,
+          -0.01441910956054926,
+          -0.009609668515622616,
+          -0.00790056586265564,
+          -0.0009622336947359145,
+          -0.019329898059368134,
+          0.01910877414047718,
+          -0.020490799099206924,
+          0.003613990731537342,
+          0.008821914903819561,
+          -0.010890343226492405,
+          -0.005348430015146732,
+          -0.016676414757966995,
+          0.024120911955833435,
+          0.011885399930179119,
+          -0.002489945152774453,
+          0.005002924241125584,
+          -0.00041950165177695453,
+          0.013221356086432934,
+          -0.006338879931718111,
+          0.003779833670705557,
+          0.0003184411907568574,
+          -0.01150764711201191,
+          -0.005707756150513887,
+          0.023328552022576332,
+          -0.007587307598441839,
+          0.021172596141695976,
+          -0.0018023886950686574,
+          0.005417531356215477,
+          0.004599833860993385,
+          -0.004627474583685398,
+          0.0080479821190238,
+          -0.003604777390137315,
+          -0.008577757515013218,
+          -0.019716864451766014,
+          0.02472900226712227,
+          -0.022573046386241913,
+          -0.0025820801965892315,
+          0.025171250104904175,
+          0.0016941301291808486,
+          -0.001815057243220508,
+          -0.012263152748346329,
+          -0.011949894018471241,
+          0.025337092578411102,
+          0.011378658004105091,
+          0.004256631713360548,
+          -0.014502030797302723,
+          -0.007269442547112703,
+          0.0017148605547845364,
+          -0.013506974093616009,
+          0.01222629938274622,
+          -0.012152590788900852,
+          0.021670125424861908,
+          -0.013875513337552547,
+          0.024231474846601486,
+          0.010641579516232014,
+          0.01586562767624855,
+          0.016031470149755478,
+          -0.019053494557738304,
+          0.0022711248602718115,
+          -0.006421801634132862,
+          0.008181577548384666,
+          0.0009587786626070738,
+          -0.026866532862186432,
+          -0.0029229791834950447,
+          0.009453038685023785,
+          0.0037268560845404863,
+          -0.01805843785405159,
+          0.00776697089895606,
+          -0.015322031453251839,
+          0.019716864451766014,
+          0.007301689591258764,
+          -0.002683428581804037,
+          -0.010917983949184418,
+          0.0028653948102146387,
+          -0.001710253767669201,
+          0.008568543940782547,
+          0.0005744034424424171,
+          0.008738993667066097,
+          0.013230569660663605,
+          0.01890607737004757,
+          -0.02883821912109852,
+          -0.0006230621365830302,
+          -0.005790677387267351,
+          -0.027364060282707214,
+          -0.008305959403514862,
+          0.009471465833485126,
+          0.016704054549336433,
+          -0.0007359273731708527,
+          -0.0004033780423924327,
+          -0.004583710338920355,
+          0.0028193274047225714,
+          0.0007566577405668795,
+          0.0004030901181977242,
+          0.013396412134170532,
+          -0.010954837314784527,
+          0.011102253571152687,
+          0.00021853242651559412,
+          -0.0061361831612885,
+          -0.016169672831892967,
+          0.02588990330696106,
+          -0.006122363265603781,
+          0.019993269816040993,
+          0.02447102591395378,
+          0.012576411478221416,
+          0.004205957520753145,
+          -0.014704727567732334,
+          0.019274618476629257,
+          0.012742254883050919,
+          -0.0029943836852908134,
+          -0.017339784651994705,
+          0.0024669114500284195,
+          0.0042612385004758835,
+          0.0005887994775548577,
+          -0.019993269816040993,
+          0.007016071584075689,
+          0.0008320931810885668,
+          -0.008278319612145424,
+          -0.020030124112963676,
+          -0.016842257231473923,
+          -0.024047205224633217,
+          0.02244405820965767,
+          0.0001376984582748264,
+          -0.014575738459825516,
+          -0.012244726531207561,
+          0.00013928202679380774,
+          0.006444835104048252,
+          -0.006887082476168871,
+          -0.005532699637115002,
+          0.009075285866856575,
+          -0.013442480005323887,
+          -0.00310494564473629,
+          0.0066291047260165215,
+          0.009070679545402527,
+          -0.0072418018244206905,
+          -0.01098247803747654,
+          -0.01785574108362198,
+          0.025742486119270325,
+          -0.009259555488824844,
+          0.01648293063044548,
+          0.0009034976828843355,
+          -0.0011361383367329836,
+          0.00893708411604166,
+          0.0072418018244206905,
+          0.005592587403953075,
+          0.021430574357509613,
+          0.010521803982555866,
+          0.01150764711201191,
+          0.0020384842064231634,
+          -0.022112371399998665,
+          0.014115064404904842,
+          -0.007269442547112703,
+          -0.009218094870448112,
+          -0.006182250566780567,
+          -0.0015225289389491081,
+          0.004470845218747854,
+          0.011046973057091236,
+          0.018979785963892937,
+          0.025355519726872444,
+          -0.006652138661593199,
+          -0.004754160065203905,
+          0.01228157989680767,
+          -0.0004039538907818496,
+          0.008660678751766682,
+          -0.0034435412380844355,
+          -0.002821630798280239,
+          -0.004199047107249498,
+          0.0061960709281265736,
+          -0.019127201288938522,
+          -0.005514272954314947,
+          0.008835735730826855,
+          -0.007297082804143429,
+          0.03232091665267944,
+          -0.0140966372564435,
+          -0.02417619340121746,
+          -0.00136705138720572,
+          0.002978260163217783,
+          -0.017496414482593536,
+          -0.006970004178583622,
+          0.016132818534970284,
+          0.015607649460434914,
+          0.01041124202311039,
+          -0.0027133722323924303,
+          0.024286756291985512,
+          0.005790677387267351,
+          0.025926755741238594,
+          -0.00041460696957074106,
+          -0.004364890046417713,
+          -0.022573046386241913,
+          0.017431920394301414,
+          -0.0003259271616116166,
+          0.01357146818190813,
+          0.027069229632616043,
+          0.008292139507830143,
+          0.006357307080179453,
+          0.00783146545290947,
+          0.012253939174115658,
+          -0.000859157822560519,
+          0.00014316897431854159,
+          -0.01663956046104431,
+          -0.013460906222462654,
+          -0.01568135805428028,
+          -0.002111040521413088,
+          0.010908770374953747,
+          0.011820905841886997,
+          0.004827867727726698,
+          0.0028976420871913433,
+          0.012143378145992756,
+          0.008550116792321205,
+          -0.007955847308039665,
+          -0.024802710860967636,
+          -0.00040740895201452076,
+          -0.026221588253974915,
+          0.009950567036867142,
+          0.007476745638996363,
+          -0.0007117419736459851,
+          0.0013705063611268997,
+          0.0015248323325067759,
+          0.003427417716011405,
+          -0.0005240171449258924,
+          -0.0034135973546653986,
+          -0.004279665183275938,
+          0.011102253571152687,
+          -0.006887082476168871,
+          -0.003547193016856909,
+          -0.00171946722548455,
+          0.03639327734708786,
+          -0.019422033801674843,
+          0.006960790604352951,
+          -0.011286523193120956,
+          0.009904500097036362,
+          0.005081238690763712,
+          0.014704727567732334,
+          -0.004265845287591219,
+          -0.009277982637286186,
+          -0.016022255644202232,
+          0.003584047080948949,
+          -0.016860684379935265,
+          0.014299334026873112,
+          0.0008084836299531162,
+          -0.018804728984832764,
+          0.01742270588874817,
+          -0.003095732070505619,
+          0.007720903493463993,
+          -0.0006224862881936133,
+          -0.01003348920494318,
+          0.013838659971952438,
+          0.008144724182784557,
+          0.03552721068263054,
+          -0.01714630238711834,
+          0.023715518414974213,
+          -0.015220683068037033,
+          0.020011696964502335,
+          0.020030124112963676,
+          -0.0013014052528887987,
+          -0.006504722870886326,
+          0.0013877816963940859,
+          0.013940008357167244,
+          -0.03057035431265831,
+          -0.008319780230522156,
+          0.010208545252680779,
+          -0.015091693960130215,
+          -0.011065399274230003,
+          -0.015837986022233963,
+          0.0012138771126046777,
+          0.00650011608377099,
+          -0.011028545908629894,
+          -0.0007267139153555036,
+          0.004288878757506609,
+          0.01742270588874817,
+          -0.008992364630103111,
+          0.003961800131946802,
+          -0.024194620549678802,
+          0.005085845477879047,
+          0.02283102460205555,
+          0.007011464796960354,
+          -0.006868655793368816,
+          0.019827427342534065,
+          0.012954164296388626,
+          -0.007154273800551891,
+          -0.006278992164880037,
+          -0.02273888885974884,
+          -0.015460233204066753,
+          -0.0240840595215559,
+          -0.0005948458565399051,
+          -0.01744113303720951,
+          -0.003581743687391281,
+          0.00019549870921764523,
+          0.0033721367362886667,
+          -0.0007203796412795782,
+          -0.01997484266757965,
+          0.018159786239266396,
+          0.01552472822368145,
+          0.002715675625950098,
+          -0.012557984329760075,
+          0.015626076608896255,
+          0.0234022606164217,
+          -0.01796630211174488,
+          -0.0004338977159932256,
+          0.0029874735046178102,
+          -0.017984729260206223,
+          0.01313843484967947,
+          0.004966070409864187,
+          -0.001163778710179031,
+          -0.022683609277009964,
+          -0.0006858290289528668,
+          0.009821578860282898,
+          0.015358884818851948,
+          0.004360283259302378,
+          0.005187193863093853,
+          -0.007168094161897898,
+          -0.000966840423643589,
+          -0.00818618480116129,
+          -5.551486538024619e-06,
+          -0.013553041033446789,
+          0.007025284692645073,
+          -0.008844949305057526,
+          0.022130798548460007,
+          -0.01958787627518177,
+          0.004318822640925646,
+          -0.02600046433508396,
+          -0.0011966017773374915,
+          0.003358316607773304,
+          0.009913713671267033,
+          -0.009738657623529434,
+          -0.009655735455453396,
+          -0.003991744015365839,
+          0.004171406850218773,
+          0.011599781922996044,
+          -0.009148993529379368,
+          0.010319107212126255,
+          -0.02915147691965103,
+          0.005025957711040974,
+          -0.0040078675374388695,
+          0.008762027136981487,
+          -0.023955069482326508,
+          0.009453038685023785,
+          -0.0010797056602314115,
+          -3.174334779032506e-05,
+          0.008932476863265038,
+          -0.01332270447164774,
+          -0.023770799860358238,
+          0.0043994407169520855,
+          -0.0016849166713654995,
+          -0.013589895330369473,
+          -0.03777530416846275,
+          -0.010871916078031063,
+          0.0010428517125546932,
+          0.0075043863616883755,
+          0.01170113030821085,
+          -0.028709229081869125,
+          -0.0003699791559483856,
+          -0.026332149282097816,
+          -0.006288205739110708,
+          -0.0187218077480793,
+          -0.011461579240858555,
+          0.0037452830001711845,
+          0.01931147091090679,
+          0.004984497092664242,
+          0.011922254227101803,
+          0.009959780611097813,
+          0.03009125404059887,
+          0.008444162085652351,
+          -0.0043994407169520855,
+          0.004109215922653675,
+          -0.005744609981775284,
+          -0.004947643261402845,
+          -0.013645176775753498,
+          0.003895002184435725,
+          -0.005145733244717121,
+          -0.0016238773241639137,
+          -0.003293822053819895,
+          0.01074292790144682,
+          0.011083826422691345,
+          0.00013122022210154682,
+          -0.01441910956054926,
+          0.019698437303304672,
+          -0.014225626364350319,
+          -0.010577084496617317,
+          -0.0021340742241591215,
+          -0.013820232823491096,
+          0.015110121108591557,
+          0.015469446778297424,
+          -0.026055745780467987,
+          -0.005730789620429277,
+          0.013543828390538692,
+          0.01279753539711237,
+          0.0054129245691001415,
+          0.009287196211516857,
+          0.01204202976077795,
+          -0.014068996533751488,
+          -0.0045053958892822266,
+          -0.01537731196731329,
+          -0.006527756340801716,
+          -0.0034435412380844355,
+          -0.0004523246898315847,
+          0.013838659971952438,
+          -0.02102518081665039,
+          -0.004643598105758429,
+          0.003116462379693985,
+          0.008296746760606766,
+          -0.0031878668814897537,
+          -0.0023379225749522448,
+          -0.02025124803185463,
+          0.007591914385557175,
+          -0.012640906497836113,
+          0.016906751319766045,
+          -0.0007146212155930698,
+          -0.0019267707830294967,
+          -0.01189461350440979,
+          0.013672816567122936,
+          -0.021154168993234634,
+          0.030035972595214844,
+          -0.011802478693425655,
+          0.002077641664072871,
+          -0.002655788091942668,
+          -0.001952107879333198,
+          0.00790056586265564,
+          0.03256046772003174,
+          -0.004786407109349966,
+          -0.004164496902376413,
+          0.015497087500989437,
+          -0.023549675941467285,
+          0.02541080117225647,
+          0.007974274456501007,
+          0.013433266431093216,
+          0.0006869807257317007,
+          -0.016013043001294136,
+          0.018942931666970253,
+          0.017883380874991417,
+          0.005288542248308659,
+          0.011046973057091236,
+          -0.01570899784564972,
+          0.016105176880955696,
+          0.007158880587667227,
+          -0.001377416541799903,
+          0.011903827078640461,
+          0.019532594829797745,
+          -0.02848810702562332,
+          -0.016842257231473923,
+          0.014391468837857246,
+          -0.018758662045001984,
+          -0.01995641551911831,
+          -0.012576411478221416,
+          -0.011968321166932583,
+          -0.028635522350668907,
+          0.012567197903990746,
+          0.015607649460434914,
+          0.0002802915987558663,
+          0.015229896642267704,
+          0.0030035972595214844,
+          0.017579335719347,
+          -0.01685146987438202,
+          -0.012244726531207561,
+          -0.0032085974235087633,
+          0.008301353082060814,
+          0.006771914195269346,
+          -0.015220683068037033,
+          -0.00785449892282486,
+          0.00334219285286963,
+          -0.027198217809200287,
+          0.008333600126206875,
+          -0.009664949029684067,
+          0.01910877414047718,
+          0.003918035887181759,
+          0.00503977807238698,
+          0.012742254883050919,
+          0.018620459362864494,
+          0.018500683829188347,
+          0.0050443848595023155,
+          -0.0021363776177167892,
+          0.004026294220238924,
+          -0.0035402828361839056,
+          -0.00210873712785542,
+          0.004005563911050558,
+          0.009236522018909454,
+          0.023918215185403824,
+          -0.0328737273812294,
+          -0.0026903385296463966,
+          0.0072418018244206905,
+          -0.0013992985477671027,
+          -0.0042842719703912735,
+          0.018417762592434883,
+          -0.002002781955525279,
+          0.010448095388710499,
+          0.005177980288863182,
+          0.012880456633865833,
+          -0.009231915697455406,
+          0.003026630962267518,
+          0.015589222311973572,
+          0.014769221656024456,
+          0.014548097737133503,
+          0.011802478693425655,
+          0.0009990877006202936,
+          -0.017597762867808342,
+          0.0075043863616883755,
+          0.004067755304276943,
+          -0.009245735593140125,
+          0.004666632041335106,
+          0.010899556800723076,
+          0.014824503101408482,
+          0.004139159806072712,
+          0.010964050889015198,
+          -0.01890607737004757,
+          -0.009010791778564453,
+          0.01429012045264244,
+          0.005970340687781572,
+          -0.0102730393409729,
+          -0.009775510989129543,
+          -0.004189833998680115,
+          0.0031072490382939577,
+          -0.0028354511596262455,
+          -0.009029218927025795,
+          0.005652475170791149,
+          0.002283793408423662,
+          -0.003855844959616661,
+          -0.015810346230864525,
+          0.014981132000684738,
+          0.004643598105758429,
+          0.006186857353895903,
+          0.007803824730217457,
+          0.007684049196541309,
+          -0.007536633405834436,
+          0.018546752631664276,
+          0.0038166875019669533,
+          0.004613654222339392,
+          -0.012447422370314598,
+          -0.00558798061683774,
+          -0.011572141200304031,
+          0.011747198179364204,
+          0.006647531874477863,
+          0.008964723907411098,
+          0.0024047205224633217,
+          -0.001717163948342204,
+          0.007688655983656645,
+          -0.07857263088226318,
+          -0.014713941141963005,
+          0.006583037320524454,
+          0.03526923432946205,
+          -0.0012610962148755789,
+          -0.005118092987686396,
+          0.0045146094635128975,
+          0.009406971745193005,
+          -0.004072362091392279,
+          -0.0013808716321364045,
+          0.02045394480228424,
+          -0.009305623359978199,
+          -0.013442480005323887,
+          -0.002022360684350133,
+          -0.000278132181847468,
+          -0.00224578776396811,
+          0.002814720617607236,
+          0.0028746083844453096,
+          0.028156420215964317,
+          7.521949737565592e-05,
+          0.017220009118318558,
+          -0.006675172131508589,
+          -0.004781800322234631,
+          -0.007559667341411114,
+          -0.008600790984928608,
+          -0.016261806711554527,
+          0.003406687406823039,
+          0.019845854490995407,
+          -0.011240456253290176,
+          0.017026526853442192,
+          0.0056893290020525455,
+          0.0036830920726060867,
+          -0.0019106471445411444,
+          -0.008863375522196293,
+          -0.011793265119194984,
+          -0.005090452264994383,
+          -0.009379331022500992,
+          -0.031104736030101776,
+          0.02073034830391407,
+          -0.002413933863863349,
+          0.010475736111402512,
+          -0.015358884818851948,
+          0.0033928670454770327,
+          -0.024010350927710533,
+          0.025392374023795128,
+          -0.019827427342534065,
+          -0.019182482734322548,
+          0.0067212400026619434,
+          -0.0019866584334522486,
+          0.006730453111231327,
+          -0.01839933544397354,
+          -0.0006184554076753557,
+          -0.02216765284538269,
+          -0.009245735593140125,
+          -0.0126132657751441,
+          0.012862029485404491,
+          -0.013488546945154667,
+          -0.01968001201748848,
+          0.01968001201748848,
+          -0.014594165608286858,
+          -0.015257536433637142,
+          -0.025816194713115692,
+          -0.016123604029417038,
+          0.0055603403598070145,
+          -0.008421128615736961,
+          0.0023183440789580345,
+          -0.008794275112450123,
+          0.005643261596560478,
+          -0.0036692717112600803,
+          -0.002842361107468605,
+          -0.004910789430141449,
+          -0.018482258543372154,
+          -0.0075274198316037655,
+          -0.00802494864910841,
+          -0.008550116792321205,
+          0.010687646456062794,
+          -0.005196407437324524,
+          -0.003959496505558491,
+          -0.004166800063103437,
+          0.010245399549603462,
+          0.002411630703136325,
+          -0.004613654222339392,
+          -0.009411578066647053,
+          -0.012438209727406502,
+          0.006656745448708534,
+          -0.011111467145383358,
+          -0.015699783340096474,
+          -0.018159786239266396,
+          0.015874840319156647,
+          0.016178885474801064,
+          -0.008001914247870445,
+          -0.005426744464784861,
+          0.008808095008134842,
+          0.0013532311422750354,
+          -0.029649006202816963,
+          0.0215779896825552,
+          -0.02073034830391407,
+          0.019661584869027138,
+          0.021338438615202904,
+          0.0036001706030219793,
+          -0.002823934191837907,
+          0.03268945589661598,
+          -0.008172363974153996,
+          0.0012380625121295452,
+          0.013046300038695335,
+          -0.03620900958776474,
+          -0.011608995497226715,
+          -0.0006720087840221822,
+          -0.027087656781077385,
+          -0.015598435886204243,
+          0.018095292150974274,
+          0.0006242138333618641,
+          0.004360283259302378,
+          0.003075001761317253,
+          0.03136271610856056,
+          0.0036462380085140467,
+          -0.006219104863703251,
+          0.005353036802262068,
+          0.00365314818918705,
+          0.010319107212126255,
+          -0.02120945043861866,
+          -0.02972271293401718,
+          -0.007315509952604771,
+          0.0019290741765871644,
+          -0.0026926419232040644,
+          -0.011848545633256435,
+          -0.0004975283518433571,
+          0.008338207378983498,
+          -0.022038664668798447,
+          -0.012539558112621307,
+          0.02474742941558361,
+          -0.01336877141147852,
+          0.018049223348498344,
+          -0.009784724563360214,
+          -0.009968994185328484,
+          0.006472475826740265,
+          0.023918215185403824,
+          -0.016317088156938553,
+          0.001357837812975049,
+          0.0019071920542046428,
+          0.03027552366256714,
+          0.0013267423491925001,
+          -0.017699111253023148,
+          0.013083153404295444,
+          0.012714614160358906,
+          -0.010623152367770672,
+          -0.015966976061463356,
+          -0.008213824592530727,
+          0.017127875238656998,
+          -0.015165401622653008,
+          0.006366520654410124,
+          -0.0054451716132462025,
+          0.0020269674714654684,
+          0.017044954001903534,
+          -0.03068091720342636,
+          0.005629441235214472,
+          0.010254612192511559,
+          -0.00041950165177695453,
+          0.002570563228800893,
+          0.006923936307430267,
+          -0.028045859187841415,
+          0.005403710994869471,
+          -0.035858895629644394,
+          -0.004445508122444153,
+          0.008886409923434258,
+          -0.009618882089853287,
+          0.01842697709798813,
+          -0.00916742067784071,
+          -0.020122257992625237,
+          -0.005201014224439859,
+          0.008757420815527439,
+          0.013248995877802372,
+          -0.006817981600761414,
+          0.014032143168151379,
+          0.011913040652871132,
+          0.0025912935379892588,
+          -0.003731462871655822,
+          -0.012290793471038342,
+          -0.005205621011555195,
+          -0.02227821573615074,
+          -0.014216412790119648,
+          0.006679778918623924,
+          0.006034834776073694,
+          -0.0010117562487721443,
+          0.009563600644469261,
+          -0.022609900683164597,
+          -0.00016785824846010655,
+          0.01988270878791809,
+          0.005901239346712828,
+          0.0011401692172512412,
+          -0.0016676414525136352,
+          0.028985634446144104,
+          0.008734387345612049,
+          -0.008135510608553886,
+          0.0010992843890562654,
+          0.012751467525959015,
+          0.0017206189222633839,
+          0.01447439007461071,
+          0.01424405351281166,
+          0.006444835104048252,
+          -0.009033825248479843,
+          0.0034803953021764755,
+          -0.003321462543681264,
+          0.00697461050003767,
+          0.002355197910219431,
+          -0.01538652554154396,
+          -0.013497760519385338,
+          0.012917310930788517,
+          -0.009462252259254456,
+          0.002360956510528922,
+          0.01862967386841774,
+          -0.006546183489263058,
+          -0.0028193274047225714,
+          0.0020684280898422003,
+          -0.025926755741238594,
+          -0.0016492144204676151,
+          -0.03191552311182022,
+          -0.010125624015927315,
+          -0.014511244371533394,
+          -0.026534846052527428,
+          0.03335282951593399,
+          -0.0021525011397898197,
+          0.02905934303998947,
+          -0.017524054273962975,
+          0.007739330176264048,
+          -0.015220683068037033,
+          0.024950126186013222,
+          0.005067418795078993,
+          0.00015432592772413045,
+          0.01604989729821682,
+          -0.027916869148612022,
+          -0.01566293090581894,
+          0.02550293505191803,
+          0.009609668515622616,
+          -0.01002427563071251,
+          0.036172155290842056,
+          0.02082248404622078,
+          0.0077485437504947186,
+          -0.006168430671095848,
+          0.023439114913344383,
+          0.0010319106513634324,
+          0.007260228972882032,
+          -0.010660005733370781,
+          0.013847872614860535,
+          -0.012686973437666893,
+          -0.0030151139944791794,
+          0.046767666935920715,
+          0.004477755166590214,
+          -0.0008827673736959696,
+          -0.013801805675029755,
+          0.010844276286661625,
+          -0.0060578687116503716,
+          0.015976188704371452,
+          -0.00862843170762062,
+          0.031583838164806366,
+          -0.005288542248308659,
+          -0.016980459913611412,
+          0.019753718748688698,
+          -0.01442832313477993,
+          0.04374564439058304,
+          -0.02561349794268608,
+          0.003968710079789162,
+          -0.012493490241467953,
+          -0.024544732645154,
+          0.013672816567122936,
+          -0.007707083132117987,
+          -0.003770620096474886,
+          0.008338207378983498,
+          -0.010733714327216148,
+          -0.010715287178754807,
+          0.011341804638504982,
+          -0.010116410441696644,
+          -0.006675172131508589,
+          -0.017275290563702583,
+          -0.009858432225883007,
+          -0.018528325483202934,
+          -0.002324102446436882,
+          -0.007725509814918041,
+          -0.00027582881739363074,
+          -0.00900618452578783,
+          -0.008490229956805706,
+          0.004222081042826176,
+          -0.00027122205938212574,
+          -0.004804834257811308,
+          0.0006040593143552542,
+          -0.003789047012105584,
+          0.00511348620057106,
+          -0.01486135646700859,
+          0.01639079675078392,
+          0.015423379838466644,
+          0.017477987334132195,
+          0.005680115427821875,
+          0.010577084496617317,
+          -0.006103936117142439,
+          -0.016409222036600113,
+          0.0006558852037414908,
+          0.01698967255651951,
+          0.01151686068624258,
+          0.012014389038085938,
+          0.019237764179706573,
+          -0.001894523506052792,
+          0.0007935116882435977,
+          -0.004219777882099152,
+          -0.005832138005644083,
+          -0.01223551295697689,
+          -0.009563600644469261,
+          -0.007587307598441839,
+          -0.02319956384599209,
+          0.0036969122011214495,
+          -0.0010906467214226723,
+          0.0013405625941231847,
+          0.0030427544843405485,
+          0.004763373639434576,
+          0.0072187683545053005,
+          0.001698736916296184,
+          -0.01685146987438202,
+          -0.003547193016856909,
+          -0.002957529854029417,
+          -0.019514167681336403,
+          -0.004247418139129877,
+          0.008702139370143414,
+          -0.010107196867465973,
+          -0.0016250290209427476,
+          0.012382928282022476,
+          0.0025521363131701946,
+          0.016934391111135483,
+          0.012640906497836113,
+          -0.006513936445116997,
+          -0.0006904358160682023,
+          0.009208881296217442,
+          -0.01462180633097887,
+          -0.012309220619499683,
+          -0.025355519726872444,
+          -0.017063381150364876,
+          -0.002549832919612527,
+          0.018749449402093887,
+          -0.00759652117267251,
+          -0.00880348775535822,
+          0.003574833506718278,
+          -0.009425398893654346,
+          0.00511348620057106,
+          -0.021559562534093857,
+          -0.0007624161662533879,
+          -0.020785629749298096,
+          0.005634048022329807,
+          -0.007052925415337086,
+          0.0036393278278410435,
+          0.009821578860282898,
+          0.034347884356975555,
+          0.0023678664583712816,
+          -0.013470119796693325,
+          -0.01265011914074421,
+          0.010881129652261734,
+          0.013212142512202263,
+          -0.013811019249260426,
+          -0.0020949169993400574,
+          -0.008324386551976204,
+          -0.002998990472406149,
+          0.0010509134735912085,
+          0.014492817223072052,
+          0.026203161105513573,
+          -0.01947731524705887,
+          -0.014713941141963005,
+          -0.018270347267389297,
+          -0.024323608726263046,
+          0.003708429168909788,
+          0.009637309238314629,
+          0.02082248404622078,
+          -0.00788674596697092,
+          -0.009148993529379368,
+          -0.005707756150513887,
+          -0.009047646075487137,
+          0.007545846980065107,
+          0.11542658507823944,
+          0.005772250704467297,
+          -0.001601995318196714,
+          -0.01462180633097887,
+          -0.004254328086972237,
+          0.010595511645078659,
+          -0.012760681100189686,
+          0.0664476752281189,
+          -0.013064726255834103,
+          0.005730789620429277,
+          -0.007467532530426979,
+          -0.0006184554076753557,
+          -0.004480058792978525,
+          -0.010973264463245869,
+          0.00079120829468593,
+          -0.018482258543372154,
+          0.000761840317863971,
+          -0.011295736767351627,
+          -0.0063204532489180565,
+          -0.004611351061612368,
+          -0.011885399930179119,
+          -0.016786975786089897,
+          0.01614203117787838,
+          -0.03038608469069004,
+          -0.001484523294493556,
+          0.0005410045268945396,
+          -0.020804056897759438,
+          0.003203990636393428,
+          -0.027013948187232018,
+          0.005726182833313942,
+          -0.0008816156769171357,
+          0.013811019249260426,
+          -0.005647868383675814,
+          -0.0204355176538229,
+          -0.013212142512202263,
+          0.008946297690272331,
+          0.009978207759559155,
+          0.01668562740087509,
+          0.017800459638237953,
+          0.005620228126645088,
+          -0.000780843140091747,
+          -0.011295736767351627,
+          0.010558657348155975,
+          -0.01065079215914011,
+          0.0003250633890274912,
+          -0.010890343226492405,
+          0.0018588212551549077,
+          -0.0021755348425358534,
+          0.02342068776488304,
+          -0.011811692267656326,
+          -0.021670125424861908,
+          0.001484523294493556,
+          0.02940945513546467,
+          -0.0034020806197077036,
+          0.017091020941734314,
+          7.133255712687969e-05,
+          0.009231915697455406,
+          -0.003399777226150036,
+          0.0004618261009454727,
+          0.026829678565263748,
+          -0.018454616889357567,
+          0.00623292475938797,
+          0.004625170957297087,
+          0.013672816567122936,
+          -0.008361240848898888,
+          0.01332270447164774,
+          -0.029943836852908134,
+          -0.005095059052109718,
+          -0.020195966586470604,
+          0.009089106693863869,
+          0.027566757053136826,
+          0.0051825870759785175,
+          0.012198658660054207,
+          -0.01899821311235428,
+          -0.012115737423300743,
+          0.0337950736284256,
+          -0.02006697840988636,
+          -0.010558657348155975,
+          -0.010871916078031063,
+          -0.024342035874724388,
+          0.00790977943688631,
+          0.011019332334399223,
+          -0.023457542061805725,
+          -0.015884054824709892,
+          0.004841688089072704,
+          -0.009664949029684067,
+          -0.017597762867808342,
+          0.0017160122515633702,
+          -0.015073266811668873,
+          -0.014409895986318588,
+          -0.005850565154105425,
+          -0.001894523506052792,
+          -0.011940681375563145,
+          0.00858236476778984,
+          -0.02159641683101654,
+          -0.0016446076333522797,
+          -0.0008729780092835426,
+          0.0024553947150707245,
+          0.0030634847935289145,
+          0.00623292475938797,
+          -0.0023770800326019526,
+          -0.005979553796350956,
+          -0.025171250104904175,
+          0.013064726255834103,
+          -0.0036577549763023853,
+          0.034937549382448196,
+          0.019256191328167915,
+          0.004056238103657961,
+          -0.004758766852319241,
+          -0.00841191504150629,
+          0.019993269816040993,
+          -0.02139372006058693,
+          -0.019145628437399864,
+          0.007776184007525444,
+          0.020398663356900215,
+          -0.02331012487411499,
+          0.00395028293132782,
+          -0.009024611674249172,
+          0.008665286004543304,
+          -0.010236185975372791,
+          -0.005247081629931927,
+          0.007587307598441839,
+          0.00011257731239311397,
+          0.0029045522678643465,
+          -0.006647531874477863,
+          0.01785574108362198,
+          0.013193715363740921,
+          -0.01242899615317583,
+          0.02310742810368538,
+          -0.0021064337342977524,
+          0.0019325291505083442,
+          0.0072510153986513615,
+          -0.009351690299808979,
+          -0.00769326277077198,
+          0.012521130964159966,
+          0.00484629487618804,
+          -0.02045394480228424,
+          0.018777089193463326,
+          0.007803824730217457,
+          0.02828541025519371,
+          -0.0014016019413247705,
+          -0.01662113331258297,
+          -0.0068594422191381454,
+          0.004173710476607084,
+          -0.02168855257332325,
+          -0.027953723445534706,
+          0.017072593793272972,
+          -0.012382928282022476,
+          -0.019182482734322548,
+          -0.00895090401172638,
+          0.010558657348155975,
+          -0.020472371950745583,
+          -0.0068502286449074745,
+          0.0029344961512833834,
+          0.0337950736284256,
+          -0.0038650583010166883,
+          0.002782473573461175,
+          0.011673489585518837,
+          0.004118429496884346,
+          0.001210422022268176,
+          -0.021430574357509613,
+          0.0016549729043617845,
+          0.02905934303998947,
+          0.025631925091147423,
+          0.018565179780125618,
+          -0.018841583281755447,
+          0.0005994525854475796,
+          0.009959780611097813,
+          0.009775510989129543,
+          0.006983824074268341,
+          0.008066409267485142,
+          -0.013405625708401203,
+          -0.0031187657732516527,
+          0.009411578066647053,
+          -0.011922254227101803,
+          0.014170344918966293,
+          -0.0013014052528887987,
+          -0.012180231511592865,
+          -0.004152979701757431,
+          -0.020656641572713852,
+          -0.02187282219529152,
+          -0.001980900065973401,
+          -0.005744609981775284,
+          0.0035978672094643116,
+          3.9805148844607174e-05,
+          0.005362250376492739,
+          0.01593933440744877,
+          -0.013884726911783218,
+          0.018076865002512932,
+          0.009932140819728374,
+          0.0033053390216082335,
+          -0.010917983949184418,
+          -0.0363011434674263,
+          0.01785574108362198,
+          -0.01513776183128357,
+          -0.012336861342191696,
+          0.021743832156062126,
+          -0.006679778918623924,
+          0.013783378526568413,
+          -0.021246304735541344,
+          0.0003751617332454771,
+          0.00769326277077198,
+          0.024581586942076683,
+          -0.017643829807639122,
+          0.00568472221493721,
+          0.0062283179722726345,
+          0.0030012938659638166,
+          0.00974787026643753,
+          0.008706746622920036,
+          0.0035287661012262106,
+          0.015515514649450779
+        ],
+        "title": "Appendices A Load-Balancing Loss",
+        "keyphrases": [
+          "Load-Balancing Loss",
+          "hierarchical MoE",
+          "experts",
+          "perplexity",
+          "attention mechanism"
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "4ef9096a-b51d-4ed6-b15e-5f4caff268e6",
+      "properties": {
+        "page_content": "1. Introduction\n\nRecent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023a). It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017) has emerged as a popular solution. It can\n\nenable parameter scaling, while concurrently keeping computational costs at a modest level.\n\nRecent applications of MoE architectures in Transformers (Vaswani et al., 2017) have yielded successful attempts at scaling language models to a substantial size (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022), accompanied with remarkable performance. These achievements underscore the considerable potential and promise of MoE language models.\n\nDespite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts.\n\nThis architecture manifests two potential issues: (1) Knowledge Hybridity: existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. (2) Knowledge Redundancy: tokens assigned to different experts may require common knowledge. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters.\n\nThese issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models.\n\nIn response to the aforementioned issues, we introduce DeepSeekMoE, an innovative MoE architecture specifically designed towards ultimate expert specialization. Our architecture involves two principal strategies: (1) Fine-Grained Expert Segmentation: while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. (2) Shared Expert Isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.\n\nThrough compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized.\n\nStarting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B (Lepikhin et al., 2021) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5× expert parameters and computation. Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE.\n\nThese studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization.\n\nLeveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B (DeepSeek-AI, 2024), a dense model trained on the same 2T corpus. We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B (Touvron et al., 2023b), which has approximately 2.5 times the activated parameters. Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard1. Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. The experimental results still validate its substantial advantages over the GShard architecture consistently. In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nOur contributions are summarized as follows: - Architectural Innovation. We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation.\n\nEmpirical Validation. We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. Experimental results validate the high 1https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models\n\nScalability. We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B.\n\nAlignment for MoE. We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B.\n\nPublic Release. In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization.\n\n",
+        "embedding": [
+          -0.007097613997757435,
+          -0.024122508242726326,
+          -0.01228793989866972,
+          -0.023231398314237595,
+          0.03164222836494446,
+          -0.012272306717932224,
+          -0.00579612422734499,
+          0.04649406298995018,
+          -0.012303573079407215,
+          0.0454309843480587,
+          -0.02185564860701561,
+          -0.010310300625860691,
+          0.03736409172415733,
+          -0.01593836396932602,
+          0.0027690858114510775,
+          0.013272850774228573,
+          0.008512446656823158,
+          0.021105241030454636,
+          0.017853470519185066,
+          -0.029391001909971237,
+          0.004928464535623789,
+          -0.04117866978049278,
+          -0.02241845615208149,
+          -0.022606058046221733,
+          0.011209228076040745,
+          0.016493353992700577,
+          0.025873461738228798,
+          0.010279033333063126,
+          -0.04374256357550621,
+          0.036988887935876846,
+          0.026092329993844032,
+          0.026905274018645287,
+          0.012608426623046398,
+          0.006147878244519234,
+          -0.01596963219344616,
+          -0.022027617320418358,
+          0.03842717036604881,
+          0.020776936784386635,
+          -0.013640238903462887,
+          0.023888004943728447,
+          0.03273657336831093,
+          0.01783783733844757,
+          -0.028609326109290123,
+          -0.008363928645849228,
+          0.020730035379529,
+          0.000511997495777905,
+          -0.039646584540605545,
+          0.003970911726355553,
+          0.006726318504661322,
+          -0.012373924255371094,
+          -0.017962904646992683,
+          0.0026518343947827816,
+          -0.0023176681715995073,
+          0.005772673990577459,
+          -0.04880782216787338,
+          0.008121608756482601,
+          0.03786436468362808,
+          0.007570527959614992,
+          0.06381599605083466,
+          -0.00819195993244648,
+          0.02632683329284191,
+          -0.028968896716833115,
+          -0.011349929496645927,
+          0.005518629681318998,
+          0.012342656962573528,
+          -0.03111068718135357,
+          -0.009755311533808708,
+          0.004252315033227205,
+          -0.001601457828655839,
+          0.008856384083628654,
+          -0.03999052196741104,
+          0.02268422581255436,
+          -0.021433543413877487,
+          0.01341355312615633,
+          0.022637324407696724,
+          0.0019063112558797002,
+          0.03542553633451462,
+          0.012115971185266972,
+          0.00941919069737196,
+          -0.028968896716833115,
+          0.010451002046465874,
+          0.03836463764309883,
+          0.0045532602816820145,
+          0.023387733846902847,
+          0.046181391924619675,
+          -0.033017974346876144,
+          -0.01883838139474392,
+          -0.07066347450017929,
+          0.019416820257902145,
+          -0.0036641042679548264,
+          -0.036301013082265854,
+          -0.0024915907997637987,
+          0.014390647411346436,
+          -0.017790935933589935,
+          -0.015133239328861237,
+          0.030250845476984978,
+          -0.009114337153732777,
+          -0.009606792591512203,
+          -0.04149134084582329,
+          0.0034120138734579086,
+          0.0041936892084777355,
+          0.012100338004529476,
+          0.0006917828577570617,
+          -0.01863514631986618,
+          -0.017353197559714317,
+          0.007664328906685114,
+          -0.01122486125677824,
+          0.027843283489346504,
+          -0.009442640468478203,
+          0.07210175693035126,
+          0.011662599630653858,
+          -0.013014898635447025,
+          -0.00774249667301774,
+          0.01697799377143383,
+          -0.01190491858869791,
+          -0.024028707295656204,
+          -0.042460616677999496,
+          -0.03342444822192192,
+          0.007097613997757435,
+          -0.03351824730634689,
+          0.0031052061822265387,
+          -0.007050713524222374,
+          0.022011984139680862,
+          -0.02262169122695923,
+          -0.07103867828845978,
+          0.041272468864917755,
+          -0.008317028172314167,
+          -0.01979202590882778,
+          -0.002986000617966056,
+          -0.007121064234524965,
+          0.0018261895747855306,
+          0.004967548418790102,
+          -0.03423738852143288,
+          -0.027171043679118156,
+          0.033737119287252426,
+          0.028562424704432487,
+          0.04386763274669647,
+          0.06197123974561691,
+          -0.014328112825751305,
+          0.018463177606463432,
+          0.016243217512965202,
+          -0.01963569037616253,
+          0.007547077722847462,
+          -0.013358835130929947,
+          -0.029656771570444107,
+          -0.01833810843527317,
+          0.01499253697693348,
+          0.06312812119722366,
+          -0.00027554063126444817,
+          0.005131700076162815,
+          -0.03043844737112522,
+          0.05531136319041252,
+          -0.02162114717066288,
+          0.02626429870724678,
+          -0.001939532463438809,
+          0.002087073866277933,
+          -0.018291208893060684,
+          0.014077977277338505,
+          -0.019745124503970146,
+          0.003943553194403648,
+          0.01255370955914259,
+          -0.0338309183716774,
+          0.00569450668990612,
+          0.01824430748820305,
+          0.004725228995084763,
+          0.03333064541220665,
+          0.006179145537316799,
+          0.0007611566106788814,
+          -0.06187743693590164,
+          0.02789018489420414,
+          0.007937915623188019,
+          0.02185564860701561,
+          -0.011428097262978554,
+          0.008035624399781227,
+          0.0017568158218637109,
+          0.0012887875782325864,
+          0.02079256996512413,
+          0.0060306270606815815,
+          -0.042304281145334244,
+          0.016868559643626213,
+          -0.028703127056360245,
+          0.024951083585619926,
+          0.027937084436416626,
+          0.01647772081196308,
+          0.029172131791710854,
+          0.01813487336039543,
+          0.04702560231089592,
+          0.028796928003430367,
+          0.017259396612644196,
+          0.01966695673763752,
+          0.01883838139474392,
+          0.007789397146552801,
+          0.03207996487617493,
+          0.011084159836173058,
+          0.010638604871928692,
+          -0.026076696813106537,
+          -0.0023919271770864725,
+          0.025123052299022675,
+          0.022371554747223854,
+          0.00022302180877886713,
+          -0.0031208398286253214,
+          -0.027405545115470886,
+          0.0038243476301431656,
+          -0.006437098141759634,
+          0.011725133284926414,
+          -0.0005163944442756474,
+          -0.003394426079466939,
+          0.02990690805017948,
+          0.02507615275681019,
+          -0.02055806666612625,
+          -0.013718406669795513,
+          -0.013202500529587269,
+          -0.03495652973651886,
+          -0.05943860858678818,
+          0.00731257488951087,
+          0.005628064274787903,
+          -0.002409514971077442,
+          0.012209772132337093,
+          0.024497712031006813,
+          -0.03864603862166405,
+          -0.006269038189202547,
+          -0.005651514511555433,
+          0.015563160181045532,
+          -0.02762441523373127,
+          -0.0013708635233342648,
+          -0.0062494962476193905,
+          0.012584976851940155,
+          0.0018232582369819283,
+          0.0441177673637867,
+          -0.0175720676779747,
+          -0.0032107324805110693,
+          0.018682045862078667,
+          0.00900490302592516,
+          0.02573275938630104,
+          0.03989672288298607,
+          -0.02431011013686657,
+          0.012209772132337093,
+          -0.0007171873585321009,
+          0.011154510080814362,
+          0.017368830740451813,
+          0.00593682611361146,
+          -0.002163287252187729,
+          -0.030500980094075203,
+          -0.018181772902607918,
+          0.035269200801849365,
+          0.02026103064417839,
+          0.00014326647215057164,
+          0.04396143555641174,
+          0.0065035405568778515,
+          0.021496077999472618,
+          0.013390102423727512,
+          -0.016165051609277725,
+          -0.008066891692578793,
+          0.006788852158933878,
+          -0.02623303234577179,
+          0.04271075502038002,
+          0.034831464290618896,
+          -0.0005466843722388148,
+          -0.010341567918658257,
+          -0.03958405181765556,
+          0.019682591781020164,
+          -0.028265388682484627,
+          -0.016071248799562454,
+          0.04943316429853439,
+          -0.01714996248483658,
+          0.02401307411491871,
+          0.04552478343248367,
+          0.01860387809574604,
+          -0.009575525298714638,
+          -0.0058977422304451466,
+          0.0014216724084690213,
+          -0.011396829970180988,
+          -0.02925029955804348,
+          0.013546437956392765,
+          0.003519494319334626,
+          -0.03176729381084442,
+          0.024169407784938812,
+          -0.0020089060999453068,
+          0.01830684207379818,
+          0.01532084122300148,
+          0.02158987894654274,
+          -0.009817845188081264,
+          0.02543572336435318,
+          0.010239950381219387,
+          -0.019651323556900024,
+          -0.0199014600366354,
+          -0.008270127698779106,
+          -0.04264821857213974,
+          -0.01573512889444828,
+          -0.013640238903462887,
+          -0.012413008138537407,
+          0.012037803418934345,
+          -0.05046497657895088,
+          0.004490726161748171,
+          0.00925503857433796,
+          -0.003300625132396817,
+          -0.017165595665574074,
+          0.020089061930775642,
+          0.004693961702287197,
+          -0.0008246677462011576,
+          -0.06691142916679382,
+          0.008473362773656845,
+          0.011529714800417423,
+          0.001870158826932311,
+          -0.02576402761042118,
+          0.023622235283255577,
+          -0.012209772132337093,
+          0.047901079058647156,
+          -0.01425776258111,
+          -0.03664495050907135,
+          0.0108574740588665,
+          -0.014296846464276314,
+          -0.0075079938396811485,
+          -0.007574436254799366,
+          0.0036054786760360003,
+          0.058719467371702194,
+          0.04042826220393181,
+          0.013007081113755703,
+          -0.03176729381084442,
+          -0.020683135837316513,
+          0.022699858993291855,
+          0.0027397728990763426,
+          -0.04705686867237091,
+          0.019057249650359154,
+          0.015461542643606663,
+          -0.010826206766068935,
+          0.024450812488794327,
+          0.04302342236042023,
+          -0.023794203996658325,
+          0.017024893313646317,
+          -0.011193593963980675,
+          -0.032580237835645676,
+          -0.031251389533281326,
+          0.012358291074633598,
+          -0.005334935616701841,
+          -0.04430537298321724,
+          -0.0290157962590456,
+          -0.0036191579420119524,
+          -0.007433734834194183,
+          0.009817845188081264,
+          -0.015180139802396297,
+          -0.039052512496709824,
+          -0.036394815891981125,
+          0.031517159193754196,
+          0.0017431365558877587,
+          0.0038946985732764006,
+          -0.03601961210370064,
+          0.0033651133999228477,
+          0.005702323280274868,
+          0.00542873702943325,
+          0.001134406658820808,
+          0.009090886451303959,
+          0.009294122457504272,
+          0.0013835657155141234,
+          -0.01187365222722292,
+          -0.038614772260189056,
+          -0.029203400015830994,
+          -0.0011451547034084797,
+          0.011209228076040745,
+          0.04055332764983177,
+          0.03014140948653221,
+          -0.05312267318367958,
+          -0.04149134084582329,
+          0.02957860380411148,
+          -0.03705142065882683,
+          -0.017228130251169205,
+          0.0033592507243156433,
+          0.00022241112310439348,
+          0.002487682504579425,
+          0.010724589228630066,
+          0.00532321073114872,
+          0.04427410289645195,
+          -0.0007357521099038422,
+          0.012710045091807842,
+          0.01567259430885315,
+          0.013460453599691391,
+          0.02424757555127144,
+          -0.015680411830544472,
+          -0.020354831591248512,
+          0.04755714163184166,
+          -0.023669136688113213,
+          0.01641518622636795,
+          0.011365562677383423,
+          0.011037259362637997,
+          0.013906008563935757,
+          0.017290662974119186,
+          0.001894586137495935,
+          0.011865834705531597,
+          -0.058844536542892456,
+          0.014375013299286366,
+          0.004736953880637884,
+          -0.031954895704984665,
+          -0.024482078850269318,
+          -0.014695500954985619,
+          0.02729611098766327,
+          0.03207996487617493,
+          0.00952080823481083,
+          -0.024466445669531822,
+          -0.022668590769171715,
+          -0.00555380480363965,
+          0.007304758299142122,
+          0.04596252366900444,
+          -0.0008617973071523011,
+          0.02049553394317627,
+          -0.015492809936404228,
+          0.010279033333063126,
+          0.01527394074946642,
+          0.01324940100312233,
+          0.026655137538909912,
+          -0.00941919069737196,
+          -3.502273102640174e-05,
+          -0.043930165469646454,
+          -0.027139775454998016,
+          -0.013937275856733322,
+          0.006112703122198582,
+          0.02712414227426052,
+          0.026389367878437042,
+          0.006374564487487078,
+          -0.003955278545618057,
+          0.022011984139680862,
+          -0.030500980094075203,
+          0.016165051609277725,
+          0.03608214482665062,
+          0.026670770719647408,
+          -0.016665322706103325,
+          0.0023587059695273638,
+          0.01933865435421467,
+          0.0032713122200220823,
+          0.01016178261488676,
+          -0.0252012200653553,
+          0.028859462589025497,
+          -0.022996895015239716,
+          0.029422268271446228,
+          -0.0237160362303257,
+          -0.027217943221330643,
+          -0.030954353511333466,
+          -0.003388563636690378,
+          0.03458132594823837,
+          -0.013046164996922016,
+          -0.01112324371933937,
+          -0.008449913002550602,
+          0.02380983717739582,
+          -0.014961270615458488,
+          -0.0068513862788677216,
+          -0.03702015429735184,
+          -0.017900370061397552,
+          -0.05071511119604111,
+          0.0032380910124629736,
+          0.003027038648724556,
+          -0.022199586033821106,
+          -0.044680576771497726,
+          0.014445364475250244,
+          -0.09092450141906738,
+          -0.004307032562792301,
+          0.032580237835645676,
+          -0.020683135837316513,
+          0.006538716144859791,
+          0.0038204393349587917,
+          0.011740767396986485,
+          -0.03877110779285431,
+          0.0010366971837356687,
+          0.03226756677031517,
+          0.0514029860496521,
+          -0.008637514896690845,
+          -0.01996399462223053,
+          0.00903616938740015,
+          -0.0172437634319067,
+          -0.01904161646962166,
+          0.008434279821813107,
+          0.0228874608874321,
+          0.020135963335633278,
+          0.0036289289128035307,
+          0.026358099654316902,
+          0.01718122884631157,
+          0.030860550701618195,
+          0.03336191177368164,
+          -0.01271786168217659,
+          -0.04918302595615387,
+          0.02768694795668125,
+          0.039083778858184814,
+          0.009739677421748638,
+          -0.015281757339835167,
+          -0.01543809287250042,
+          -0.0034159221686422825,
+          -0.0024661864154040813,
+          0.0010112927993759513,
+          0.0008114769589155912,
+          -0.00029141842969693244,
+          0.01521922368556261,
+          0.028109053149819374,
+          -0.009481724351644516,
+          0.026780204847455025,
+          0.003990453667938709,
+          0.02194944955408573,
+          0.004506359808146954,
+          -0.02513868547976017,
+          -0.010255583561956882,
+          -0.0037227298598736525,
+          0.02481038309633732,
+          0.005276310257613659,
+          -0.03505033254623413,
+          0.006155695300549269,
+          0.029406635090708733,
+          -0.01286637969315052,
+          0.008465546183288097,
+          0.013452637009322643,
+          -0.00037764699663966894,
+          0.016337020322680473,
+          -0.01910415105521679,
+          -0.0001468084374209866,
+          0.02330956608057022,
+          -0.01797853782773018,
+          0.008160692639648914,
+          -0.028593692928552628,
+          -0.012061254121363163,
+          0.007492360193282366,
+          -0.011396829970180988,
+          0.017556432634592056,
+          -0.001251657959073782,
+          0.015680411830544472,
+          -0.005045715719461441,
+          0.0027983984909951687,
+          -0.010654238052666187,
+          0.03501906618475914,
+          -0.00401390390470624,
+          0.03150152415037155,
+          0.021214675158262253,
+          -0.015250490047037601,
+          0.044524241238832474,
+          -0.03295544162392616,
+          -0.025873461738228798,
+          0.01159224845468998,
+          0.006386289373040199,
+          -0.011537531390786171,
+          -0.015336474403738976,
+          -0.013616788201034069,
+          -0.00037911266554147005,
+          0.01718122884631157,
+          -0.02209015190601349,
+          -0.024153774604201317,
+          -0.020010894164443016,
+          -0.010318117216229439,
+          -0.0031638317741453648,
+          0.000499783840496093,
+          -0.0059446427039802074,
+          -0.014593882486224174,
+          0.04880782216787338,
+          -0.05543643236160278,
+          0.01943245530128479,
+          0.047838546335697174,
+          0.013171233236789703,
+          -0.04252314940094948,
+          -0.023997440934181213,
+          -0.014539165422320366,
+          -0.0075509860180318356,
+          -0.041835278272628784,
+          0.014109243638813496,
+          0.002925420878455043,
+          0.005772673990577459,
+          -0.048995424062013626,
+          -0.0009712319006212056,
+          0.005784399341791868,
+          -0.022996895015239716,
+          0.014015442691743374,
+          -0.022934360429644585,
+          -0.022606058046221733,
+          0.01424212846904993,
+          0.019416820257902145,
+          -0.006104886066168547,
+          0.013741856440901756,
+          -0.0035761657636612654,
+          -0.007980907335877419,
+          -0.0021456994581967592,
+          -0.0009365450241602957,
+          0.0011207273928448558,
+          0.019229218363761902,
+          0.004365657921880484,
+          -0.00630812207236886,
+          -0.0013132150052115321,
+          -0.005002723541110754,
+          0.030688581988215446,
+          -0.008864200673997402,
+          -0.009661509655416012,
+          0.03958405181765556,
+          0.035800740122795105,
+          -0.010724589228630066,
+          -0.02218395285308361,
+          -0.023825472220778465,
+          0.014797118492424488,
+          -0.02948480285704136,
+          0.028265388682484627,
+          0.036426082253456116,
+          -0.011670416221022606,
+          -0.03162659332156181,
+          0.005577255040407181,
+          -0.022105785086750984,
+          -0.003996316343545914,
+          0.01704052835702896,
+          0.01804107241332531,
+          -0.04418030381202698,
+          -0.006003268528729677,
+          0.01688419282436371,
+          0.01238955743610859,
+          -0.009137787856161594,
+          -0.03064168244600296,
+          0.008653149008750916,
+          0.01904161646962166,
+          0.013335385359823704,
+          -0.0027808109298348427,
+          0.00702335499227047,
+          -0.0165871549397707,
+          -0.0011353837326169014,
+          -0.000332700670696795,
+          -0.013429186306893826,
+          0.03573820739984512,
+          0.0010484223021194339,
+          0.010294667445123196,
+          -0.00013398408191278577,
+          -0.02005779556930065,
+          -0.012944547459483147,
+          0.03551933914422989,
+          0.01399199292063713,
+          0.038614772260189056,
+          0.0019502805080264807,
+          0.015578794293105602,
+          0.03817703574895859,
+          -0.002221912844106555,
+          -0.0273586455732584,
+          0.021668046712875366,
+          0.0010064073139801621,
+          -0.0441177673637867,
+          0.00657389173284173,
+          -0.03128265589475632,
+          -0.04593125730752945,
+          0.023653503507375717,
+          -0.013624604791402817,
+          0.022793659940361977,
+          -0.005432645324617624,
+          -0.004381291568279266,
+          -0.0318298302590847,
+          0.0035038606729358435,
+          0.02188691683113575,
+          0.015664778649806976,
+          0.04424283653497696,
+          -0.0051629673689603806,
+          -0.005643697455525398,
+          0.011217044666409492,
+          -0.001060147536918521,
+          0.037989433854818344,
+          0.01185801811516285,
+          -0.014281212352216244,
+          0.013859108090400696,
+          0.030266478657722473,
+          0.04761967435479164,
+          -0.02928156778216362,
+          -0.03539426997303963,
+          0.008676598779857159,
+          0.0044633676297962666,
+          0.041303738951683044,
+          0.006831844337284565,
+          0.006155695300549269,
+          0.016790391877293587,
+          0.014695500954985619,
+          -0.008567164652049541,
+          0.024028707295656204,
+          -0.03014140948653221,
+          -0.034768927842378616,
+          0.03670748323202133,
+          -0.008145059458911419,
+          0.012858563102781773,
+          0.01800980605185032,
+          -0.01425776258111,
+          -0.016821658238768578,
+          0.005639789160341024,
+          0.01091219112277031,
+          -0.015344290994107723,
+          0.030798017978668213,
+          0.03389345109462738,
+          0.02398180589079857,
+          -0.0048581138253211975,
+          0.012694410979747772,
+          0.0009091863757930696,
+          0.002092936309054494,
+          -0.01744699850678444,
+          0.007003813050687313,
+          -0.0008837819332256913,
+          -0.01618068479001522,
+          0.043992701917886734,
+          -0.0290157962590456,
+          -0.012928914278745651,
+          -0.0022844469640403986,
+          -0.002614704892039299,
+          0.005502996034920216,
+          0.003972866106778383,
+          0.018619511276483536,
+          -0.012788211926817894,
+          -0.018916549161076546,
+          0.017368830740451813,
+          0.024747848510742188,
+          0.002688963897526264,
+          -0.0072461324743926525,
+          0.023966172710061073,
+          -0.007937915623188019,
+          0.009278489276766777,
+          0.02304379642009735,
+          0.004572802223265171,
+          0.012366107665002346,
+          -0.021542979404330254,
+          -0.006890470162034035,
+          -0.0059876348823308945,
+          0.0075079938396811485,
+          0.044680576771497726,
+          0.0025365371257066727,
+          0.027186676859855652,
+          0.028296655043959618,
+          -0.004799488000571728,
+          -0.009028352797031403,
+          0.0014529394684359431,
+          -0.013859108090400696,
+          0.046806734055280685,
+          0.0006092183757573366,
+          0.007265674415975809,
+          0.013468270190060139,
+          -0.05134045332670212,
+          -0.010779306292533875,
+          0.016024349257349968,
+          0.0012506808852776885,
+          0.03339318186044693,
+          -0.003781355684623122,
+          -0.0033846551086753607,
+          0.014804935082793236,
+          -0.035863276571035385,
+          -0.039677850902080536,
+          0.012366107665002346,
+          0.006018901709467173,
+          -0.011959636583924294,
+          -0.004299215506762266,
+          0.013749673031270504,
+          -0.005788307636976242,
+          -0.004346115980297327,
+          0.009184688329696655,
+          -0.006624700501561165,
+          0.008965819142758846,
+          0.009215954691171646,
+          -0.008739132434129715,
+          0.03986545279622078,
+          -0.009606792591512203,
+          0.03727029263973236,
+          0.0007645764271728694,
+          0.0010112927993759513,
+          0.006867019925266504,
+          -0.01979202590882778,
+          0.03345571458339691,
+          -0.0003559066681191325,
+          -0.04533718153834343,
+          0.0027827650774270296,
+          0.02082383632659912,
+          -0.008231043815612793,
+          0.009825661778450012,
+          -0.006233862601220608,
+          0.015711678192019463,
+          0.03952151536941528,
+          0.0055928886868059635,
+          0.01132647879421711,
+          0.017259396612644196,
+          0.0048581138253211975,
+          0.0018779755337163806,
+          0.031188854947686195,
+          0.0011090021580457687,
+          -0.009231588803231716,
+          0.029672404751181602,
+          -0.018713314086198807,
+          0.037457894533872604,
+          -0.047807276248931885,
+          -0.007414192892611027,
+          0.03561313822865486,
+          0.0005046692676842213,
+          -0.015383374877274036,
+          -0.04196034371852875,
+          0.025029251351952553,
+          -0.03229883313179016,
+          0.021183408796787262,
+          -0.003157969331368804,
+          0.033612050116062164,
+          -0.023825472220778465,
+          0.002618613187223673,
+          0.01883838139474392,
+          -0.013077432289719582,
+          0.02477911487221718,
+          0.014875286258757114,
+          -0.01064642146229744,
+          0.0021417911630123854,
+          -0.013225950300693512,
+          0.006738043390214443,
+          0.04646279662847519,
+          -0.009888196364045143,
+          0.004029537551105022,
+          0.007511902134865522,
+          -0.02224648743867874,
+          -0.018353741616010666,
+          0.04987090080976486,
+          0.0013239630497992039,
+          0.023293932899832726,
+          0.03661368414759636,
+          -0.041897810995578766,
+          -0.02168367989361286,
+          0.02135537751019001,
+          0.004912830889225006,
+          0.0037325008306652308,
+          0.014031076803803444,
+          0.016258852556347847,
+          0.014867469668388367,
+          -0.020386097952723503,
+          0.03411232307553291,
+          0.011131060309708118,
+          0.00669895950704813,
+          -0.019651323556900024,
+          -0.024169407784938812,
+          -0.020776936784386635,
+          -0.03398725390434265,
+          -0.02102707326412201,
+          -0.00927067268639803,
+          0.010200866498053074,
+          -0.008473362773656845,
+          -0.04883908852934837,
+          -0.017790935933589935,
+          0.026358099654316902,
+          -0.0069412789307534695,
+          0.04583745449781418,
+          0.0015369695611298084,
+          0.036957621574401855,
+          0.0012604518560692668,
+          -0.016993626952171326,
+          0.050590042024850845,
+          -0.027937084436416626,
+          0.04158513993024826,
+          0.022840559482574463,
+          -0.004025629255920649,
+          -0.008410829119384289,
+          0.04752587527036667,
+          -0.024122508242726326,
+          0.0032537246588617563,
+          -0.029688037931919098,
+          -0.003392471931874752,
+          -0.024951083585619926,
+          -0.0011197502026334405,
+          -0.0028746118769049644,
+          -0.008684415370225906,
+          0.024357011541724205,
+          -0.015242673456668854,
+          0.020401732996106148,
+          0.019385553896427155,
+          -0.0021437453106045723,
+          0.008739132434129715,
+          -0.020089061930775642,
+          0.012561526149511337,
+          -0.016399553045630455,
+          0.004568893928080797,
+          -0.009919462725520134,
+          0.001229184796102345,
+          -0.03279910609126091,
+          -0.00606189388781786,
+          -0.0067224097438156605,
+          -0.011193593963980675,
+          0.015367741696536541,
+          0.038552239537239075,
+          -0.0045884354040026665,
+          -0.035206668078899384,
+          0.007781580090522766,
+          -0.04008432477712631,
+          0.010169599205255508,
+          0.007144514936953783,
+          0.002470094710588455,
+          -0.009161237627267838,
+          0.01877584680914879,
+          0.022011984139680862,
+          -0.012772578746080399,
+          -0.033737119287252426,
+          0.009724044241011143,
+          -0.006038443651050329,
+          0.016524622216820717,
+          0.0029469169676303864,
+          0.0334869809448719,
+          -0.004009995609521866,
+          0.01880711503326893,
+          -0.012358291074633598,
+          -0.004693961702287197,
+          -0.017556432634592056,
+          0.0023293932899832726,
+          -0.02380983717739582,
+          -0.00526458490639925,
+          -0.00030778476502746344,
+          -0.013358835130929947,
+          -0.00802780780941248,
+          0.0027280477806925774,
+          0.011146693490445614,
+          -0.014156145043671131,
+          0.010732405818998814,
+          -0.019057249650359154,
+          -0.010044530965387821,
+          0.0072109573520720005,
+          -0.03226756677031517,
+          -0.014070160686969757,
+          -0.038552239537239075,
+          -0.003859523218125105,
+          0.013499537482857704,
+          0.03358078375458717,
+          -0.007562711369246244,
+          0.010724589228630066,
+          -0.030250845476984978,
+          -0.0007328208303079009,
+          0.023590968921780586,
+          0.023856738582253456,
+          -0.03545680269598961,
+          0.003523402614519,
+          0.025123052299022675,
+          0.0293441005051136,
+          -0.035206668078899384,
+          0.015625694766640663,
+          0.0351441353559494,
+          0.007093705702573061,
+          0.021371010690927505,
+          -0.0105291698127985,
+          0.042992155998945236,
+          -0.03786436468362808,
+          0.01160788256675005,
+          -0.004826846532523632,
+          0.024294476956129074,
+          0.007840205915272236,
+          0.024591512978076935,
+          -0.02049553394317627,
+          0.0328303724527359,
+          0.033612050116062164,
+          0.02032356522977352,
+          0.04418030381202698,
+          -0.03658241778612137,
+          -0.017415732145309448,
+          0.012538076378405094,
+          -0.01966695673763752,
+          0.01245209202170372,
+          -0.0023606601171195507,
+          -0.02481038309633732,
+          0.009755311533808708,
+          0.024325743317604065,
+          0.008637514896690845,
+          -0.059376075863838196,
+          0.023528434336185455,
+          0.02165241353213787,
+          0.0131008829921484,
+          0.02374730445444584,
+          -0.0371452234685421,
+          0.0004274788370821625,
+          0.015930548310279846,
+          0.015242673456668854,
+          0.020933272317051888,
+          0.017337564378976822,
+          -0.010349384509027004,
+          -0.0008661942556500435,
+          -0.023137597367167473,
+          0.006769310217350721,
+          -0.0035722574684768915,
+          0.010052347555756569,
+          -0.02646753564476967,
+          -0.028187220916152,
+          0.00872349925339222,
+          0.004994906950742006,
+          -0.0021535162813961506,
+          -0.03723902255296707,
+          -0.03933391347527504,
+          0.013749673031270504,
+          0.00819195993244648,
+          -0.044618040323257446,
+          -0.012921096757054329,
+          -0.03823956847190857,
+          -0.01447663176804781,
+          -0.021636780351400375,
+          0.0018261895747855306,
+          -0.010185232385993004,
+          0.0026928724255412817,
+          -0.005202050786465406,
+          0.025513891130685806,
+          -0.010451002046465874,
+          0.04552478343248367,
+          0.006566074676811695,
+          -0.003556623822078109,
+          -0.030125776305794716,
+          0.04802614822983742,
+          -0.007433734834194183,
+          -0.03933391347527504,
+          0.02454461343586445,
+          -0.017790935933589935,
+          0.011662599630653858,
+          0.03187672793865204,
+          -0.007949640974402428,
+          -0.014883102849125862,
+          0.013851291500031948,
+          -0.023262664675712585,
+          -0.042398083955049515,
+          0.007074163760989904,
+          -0.016337020322680473,
+          -0.0293441005051136,
+          -0.041460070759058,
+          -0.004834663588553667,
+          0.018713314086198807,
+          -0.025779660791158676,
+          0.004701778758317232,
+          0.01824430748820305,
+          0.019354287534952164,
+          -0.038520973175764084,
+          0.008551530539989471,
+          -0.00254044565372169,
+          -0.0017480220412835479,
+          -0.010787122882902622,
+          0.009591159410774708,
+          -0.0032380910124629736,
+          -0.004393016919493675,
+          -0.021449178457260132,
+          -0.005784399341791868,
+          -0.029203400015830994,
+          -0.010279033333063126,
+          -0.012006537057459354,
+          -0.027905818074941635,
+          -0.04589999094605446,
+          -0.04483691230416298,
+          0.025482622906565666,
+          0.03126702457666397,
+          -0.003568348940461874,
+          0.0028277114033699036,
+          -0.008379561826586723,
+          -0.012647510506212711,
+          0.019526256248354912,
+          0.00017880828818306327,
+          0.010771489702165127,
+          -0.0133510185405612,
+          0.016634056344628334,
+          -0.010279033333063126,
+          0.00540528679266572,
+          -0.04336735978722572,
+          -0.020370464771986008,
+          0.032986707985401154,
+          0.007062438875436783,
+          -0.0007914465386420488,
+          -0.004721320234239101,
+          0.013468270190060139,
+          -0.02434137649834156,
+          -0.01286637969315052,
+          0.03392472118139267,
+          0.01721249520778656,
+          0.01452353224158287,
+          -0.015000354498624802,
+          -0.008817300200462341,
+          0.003703187918290496,
+          -0.011412463150918484,
+          0.0049362811259925365,
+          -0.004615794401615858,
+          0.009075253270566463,
+          0.013804391026496887,
+          0.013569887727499008,
+          0.013898191973567009,
+          0.02235592156648636,
+          -0.009043985977768898,
+          -0.00649181567132473,
+          -0.003738363506272435,
+          0.008465546183288097,
+          -0.006272946484386921,
+          -0.006894378457218409,
+          -0.025873461738228798,
+          0.033674582839012146,
+          0.01890091598033905,
+          0.010998175479471684,
+          -0.015227040275931358,
+          -0.003757905215024948,
+          0.009356657043099403,
+          -0.009958546608686447,
+          -0.005823483224958181,
+          0.0037930808030068874,
+          0.025029251351952553,
+          -0.02194944955408573,
+          -0.013061799108982086,
+          -0.010865290649235249,
+          -0.0026479260995984077,
+          0.006870928220450878,
+          0.03773929551243782,
+          0.018431909382343292,
+          -0.008528080768883228,
+          -0.008488996885716915,
+          0.002384110586717725,
+          0.017337564378976822,
+          0.023919273167848587,
+          -0.03539426997303963,
+          -0.030751116573810577,
+          0.008105975575745106,
+          -0.0038829734548926353,
+          0.021136507391929626,
+          0.00532321073114872,
+          0.003744225949048996,
+          0.003537081880494952,
+          0.0722893550992012,
+          -0.0008554462110623717,
+          0.010067981667816639,
+          -0.017900370061397552,
+          -0.01520358957350254,
+          -0.0013493674341589212,
+          -0.005565530154854059,
+          -0.01308524888008833,
+          0.036957621574401855,
+          0.001335688168182969,
+          0.044618040323257446,
+          0.011865834705531597,
+          -0.006296396721154451,
+          -0.01190491858869791,
+          -0.0006805462762713432,
+          0.012084703892469406,
+          -0.0005608522333204746,
+          -0.0018662504153326154,
+          -0.0025658500380814075,
+          0.008973635733127594,
+          0.006499632261693478,
+          0.02570149302482605,
+          0.0005916306981816888,
+          0.017384463921189308,
+          -0.02424757555127144,
+          0.03601961210370064,
+          0.01711869426071644,
+          0.012842929922044277,
+          -0.019416820257902145,
+          -0.008207593113183975,
+          0.0028003526385873556,
+          0.0044633676297962666,
+          -0.007707321085035801,
+          -0.0006507448852062225,
+          -0.008660965599119663,
+          0.030454080551862717,
+          -0.03007887676358223,
+          0.01668095588684082,
+          0.011725133284926414,
+          -0.018197407945990562,
+          -0.005940734408795834,
+          0.015352108515799046,
+          -0.002077302895486355,
+          0.004459459334611893,
+          -0.006112703122198582,
+          0.0052450429648160934,
+          0.024357011541724205,
+          -0.028187220916152,
+          -0.006128336302936077,
+          0.010458819568157196,
+          -0.01493781991302967,
+          -0.0036445623263716698,
+          0.004221048206090927,
+          0.00956770870834589,
+          -0.014875286258757114,
+          0.011428097262978554,
+          0.02434137649834156,
+          -0.027937084436416626,
+          -0.009450457990169525,
+          0.011310845613479614,
+          -0.00010235064837615937,
+          0.01866641268134117,
+          0.023403367027640343,
+          0.008739132434129715,
+          -0.006448823492974043,
+          -0.0031423356849700212,
+          0.010341567918658257,
+          -0.009677143767476082,
+          -0.007468909956514835,
+          -0.0069842711091041565,
+          -0.010716771706938744,
+          0.01940118707716465,
+          -0.01591491512954235,
+          -0.0026928724255412817,
+          -0.003937690518796444,
+          -0.0106933219358325,
+          0.001270222826860845,
+          -0.013069615699350834,
+          0.017071794718503952,
+          0.0013249401235952973,
+          0.017931638285517693,
+          0.010951275005936623,
+          0.004846388474106789,
+          0.023590968921780586,
+          -0.012655327096581459,
+          0.012248856015503407,
+          -0.032017432153224945,
+          -0.0018027393380180001,
+          0.009145604446530342,
+          0.006022810470312834,
+          -0.01635265350341797,
+          -0.019948361441493034,
+          -0.013194683939218521,
+          0.0032244117464870214,
+          -0.009372290223836899,
+          0.0065465327352285385,
+          0.014296846464276314,
+          0.037926897406578064,
+          -0.016133783385157585,
+          -0.030469713732600212,
+          -0.015094155445694923,
+          0.012905463576316833,
+          0.016243217512965202,
+          -0.023888004943728447,
+          -0.009669327177107334,
+          -0.007269582711160183,
+          0.0034432809334248304,
+          -0.013171233236789703,
+          -0.01833810843527317,
+          -0.01668095588684082,
+          -0.003011405235156417,
+          -0.025889094918966293,
+          0.0013493674341589212,
+          -0.0043773832730948925,
+          0.0067224097438156605,
+          0.019479354843497276,
+          -0.0034725936129689217,
+          -0.02179311402142048,
+          -0.02490418404340744,
+          0.0175720676779747,
+          -0.02434137649834156,
+          0.008231043815612793,
+          0.025623325258493423,
+          -0.017853470519185066,
+          0.004928464535623789,
+          0.026670770719647408,
+          0.004948006477206945,
+          -0.009333206340670586,
+          0.012670961208641529,
+          0.009778761304914951,
+          0.006362839136272669,
+          0.008700049482285976,
+          0.0190728846937418,
+          0.012061254121363163,
+          0.020933272317051888,
+          -0.021605512127280235,
+          -0.013593338429927826,
+          -0.004025629255920649,
+          0.03626974672079086,
+          -0.0010327888885512948,
+          -0.00878603383898735,
+          0.014617333188652992,
+          0.0361446775496006,
+          0.005663239397108555,
+          -0.005585072096437216,
+          -0.01786910369992256,
+          0.0029586420860141516,
+          0.014601700007915497,
+          0.005291943438351154,
+          0.002679192926734686,
+          -0.00567105645313859,
+          -0.01048226933926344,
+          -0.00022790727962274104,
+          -0.0009438732522539794,
+          0.004697869997471571,
+          -0.026389367878437042,
+          0.018697679042816162,
+          -0.023419000208377838,
+          0.014171778224408627,
+          -0.0013063752558082342,
+          0.0007059507770463824,
+          0.03093871846795082,
+          -0.0022023709025233984,
+          -0.022340288385748863,
+          0.002501361770555377,
+          0.0019502805080264807,
+          0.013601155020296574,
+          0.022230852395296097,
+          -0.010943458415567875,
+          0.01261624414473772,
+          -0.003331892192363739,
+          0.0014382830122485757,
+          0.007461093366146088,
+          0.030469713732600212,
+          0.002923466730862856,
+          -0.011045075953006744,
+          0.020354831591248512,
+          0.009638059884309769,
+          0.0007425918010994792,
+          -0.007703412789851427,
+          0.020886370912194252,
+          -0.015602244064211845,
+          0.012491175904870033,
+          -0.012663144618272781,
+          -0.006323755253106356,
+          3.625936528806051e-07,
+          0.0010777352144941688,
+          -0.0023626144975423813,
+          -0.030688581988215446,
+          0.022965628653764725,
+          0.0005041807307861745,
+          0.0177440345287323,
+          -0.009505175054073334,
+          -0.019307386130094528,
+          0.002489636652171612,
+          0.020933272317051888,
+          -0.006855294574052095,
+          -0.01261624414473772,
+          -0.014304663054645061,
+          -0.014335930347442627,
+          -0.020073428750038147,
+          0.006695051211863756,
+          0.004060804378241301,
+          -0.015039438381791115,
+          0.0053740194998681545,
+          -0.003746180096641183,
+          -0.019119784235954285,
+          0.005280218552798033,
+          0.02410687506198883,
+          -0.027765115723013878,
+          0.03162659332156181,
+          -0.015860198065638542,
+          -0.015297390520572662,
+          0.009075253270566463,
+          0.0035175401717424393,
+          0.0005569438217207789,
+          0.005502996034920216,
+          -0.02135537751019001,
+          0.0020694860722869635,
+          0.020417366176843643,
+          -0.01704052835702896,
+          -0.0019932726863771677,
+          0.010075798258185387,
+          0.017493899911642075,
+          0.0007289124769158661,
+          -0.018025439232587814,
+          0.02490418404340744,
+          -0.016227584332227707,
+          0.02454461343586445,
+          0.02126157470047474,
+          -0.041804008185863495,
+          0.042116679251194,
+          -0.024654047563672066,
+          0.015828929841518402,
+          0.002237546257674694,
+          -0.006437098141759634,
+          0.013069615699350834,
+          0.018885280936956406,
+          -0.005256768316030502,
+          -0.0012780395336449146,
+          0.002804261166602373,
+          0.03167349472641945,
+          -0.023403367027640343,
+          0.024528980255126953,
+          0.02132410928606987,
+          -0.00010387735528638586,
+          0.02481038309633732,
+          -0.021230308338999748,
+          -0.029766205698251724,
+          0.019620057195425034,
+          -0.023465899750590324,
+          -0.034174855798482895,
+          0.020042162388563156,
+          -0.008418645709753036,
+          -0.008184143342077732,
+          -0.020339198410511017,
+          0.006386289373040199,
+          0.0054639121517539024,
+          -0.013007081113755703,
+          0.002849207492545247,
+          0.0027397728990763426,
+          0.02606106363236904,
+          0.022574789822101593,
+          0.01520358957350254,
+          -0.003320167073979974,
+          0.018166139721870422,
+          -0.0015731221064925194,
+          0.032580237835645676,
+          -0.0020304021891206503,
+          0.0004528832796495408,
+          -0.011052892543375492,
+          -0.0021456994581967592,
+          0.0034198306966573,
+          0.0039220573380589485,
+          0.017384463921189308,
+          -0.02069876901805401,
+          -0.015868013724684715,
+          -0.0351441353559494,
+          -0.0025971170980483294,
+          -0.003539036260917783,
+          0.010708955116569996,
+          0.03508159890770912,
+          -0.00903616938740015,
+          -0.013999809511005878,
+          0.009192504920065403,
+          -0.0068083941005170345,
+          -0.0002442736295051873,
+          -0.02330956608057022,
+          -0.020354831591248512,
+          -0.01493781991302967,
+          -0.013272850774228573,
+          -0.008856384083628654,
+          -0.020042162388563156,
+          0.012960180640220642,
+          -0.010599520988762379,
+          0.01048226933926344,
+          -0.02795271761715412,
+          0.014406280592083931,
+          -0.012280123308300972,
+          0.009059620089828968,
+          -0.005456095561385155,
+          -0.024200676009058952,
+          -0.004357841331511736,
+          0.02653006836771965,
+          -0.0034745479933917522,
+          -0.021449178457260132,
+          -0.01786910369992256,
+          0.005202050786465406,
+          -0.014844018965959549,
+          -0.0016180684324353933,
+          -0.015101972036063671,
+          -0.009614609181880951,
+          0.0008300417684949934,
+          0.0009849112248048186,
+          0.005475637502968311,
+          0.010443185456097126,
+          -0.0029332374688237906,
+          -0.018150506541132927,
+          0.014015442691743374,
+          0.04139753803610802,
+          -0.04386763274669647,
+          -0.017134329304099083,
+          0.012076887302100658,
+          0.010404101572930813,
+          -0.0036308830603957176,
+          0.010474452748894691,
+          -0.013554254546761513,
+          -0.003349479753524065,
+          -0.024388277903199196,
+          -0.01308524888008833,
+          -0.02513868547976017,
+          0.00749626848846674,
+          0.017931638285517693,
+          -0.008137242868542671,
+          0.03273657336831093,
+          0.030391545966267586,
+          -0.01877584680914879,
+          -0.0013884512009099126,
+          0.010583886876702309,
+          0.015281757339835167,
+          -0.029031431302428246,
+          0.016368286684155464,
+          -0.009911646135151386,
+          0.021167773753404617,
+          0.00968496035784483,
+          -0.021152140572667122,
+          -0.0146720502525568,
+          -0.016524622216820717,
+          0.007011629641056061,
+          -0.013874741271138191,
+          -3.935858694603667e-05,
+          0.027264844626188278,
+          0.015500626526772976,
+          -0.007484543602913618,
+          -0.014109243638813496,
+          0.018650779500603676,
+          0.013335385359823704,
+          0.019573155790567398,
+          -0.00540528679266572,
+          -0.0038399812765419483,
+          -0.007801122032105923,
+          -0.00946609117090702,
+          -0.018947815522551537,
+          0.00927067268639803,
+          -0.02984437346458435,
+          0.019057249650359154,
+          -0.01854134537279606,
+          -0.007437643129378557,
+          0.03273657336831093,
+          -0.04652532935142517,
+          -0.012913280166685581,
+          -0.013280668295919895,
+          0.02895326353609562,
+          0.015977447852492332,
+          0.004623610991984606,
+          0.009919462725520134,
+          -0.021449178457260132,
+          0.0033729299902915955,
+          -0.00321659492328763,
+          0.003099343739449978,
+          -0.016837291419506073,
+          -0.016399553045630455,
+          0.012413008138537407,
+          0.002137882634997368,
+          -0.004228864796459675,
+          0.0069061038084328175,
+          -0.005420919973403215,
+          -0.013538620434701443,
+          0.002679192926734686,
+          -0.0007513856398873031,
+          0.0015975494170561433,
+          0.008817300200462341,
+          0.002515041036531329,
+          0.01495345402508974,
+          0.01112324371933937,
+          -0.008660965599119663,
+          -0.005647606216371059,
+          -0.005647606216371059,
+          0.00088085065362975,
+          0.010153966024518013,
+          0.005284126847982407,
+          -0.006398014724254608,
+          -0.011631332337856293,
+          0.003376838518306613,
+          -0.007996541447937489,
+          0.026186130940914154,
+          -0.008418645709753036,
+          -0.042335547506809235,
+          0.002751498017460108,
+          -0.042366817593574524,
+          0.02354406751692295,
+          -0.02573275938630104,
+          -0.020651869475841522,
+          -0.009974180720746517,
+          -0.003695371327921748,
+          -0.019979627802968025,
+          0.009380106814205647,
+          0.007121064234524965,
+          -0.007941823452711105,
+          0.00792619027197361,
+          0.001134406658820808,
+          -0.015023804269731045,
+          0.0105291698127985,
+          -0.0101148821413517,
+          -0.010552620515227318,
+          0.005710139870643616,
+          0.0073868343606591225,
+          0.008848567493259907,
+          -0.006272946484386921,
+          -0.007472818251699209,
+          0.009505175054073334,
+          0.014734584838151932,
+          0.005229409784078598,
+          0.0053740194998681545,
+          0.020667502656579018,
+          0.013319752179086208,
+          -0.022496622055768967,
+          0.0023606601171195507,
+          -0.008019991219043732,
+          0.0038380271289497614,
+          0.003476502140983939,
+          -0.005932917352765799,
+          -0.009192504920065403,
+          0.003963095135986805,
+          -0.0003766699228435755,
+          4.7236411774065346e-05,
+          -0.0018340062815696,
+          -0.01614941656589508,
+          -0.005491270683705807,
+          -0.012006537057459354,
+          0.004221048206090927,
+          0.0300163421779871,
+          -0.006233862601220608,
+          -0.006398014724254608,
+          -0.019682591781020164,
+          0.018494443967938423,
+          0.03404978662729263,
+          0.017556432634592056,
+          -0.028671860694885254,
+          0.0005080891423858702,
+          -0.006417556665837765,
+          0.003300625132396817,
+          -0.025513891130685806,
+          0.022856194525957108,
+          0.018744580447673798,
+          0.009458274580538273,
+          0.011834568344056606,
+          -0.0025502166245132685,
+          0.005721865221858025,
+          -0.01548499334603548,
+          0.012858563102781773,
+          -0.004557168576866388,
+          -0.012014353647828102,
+          0.005006631836295128,
+          0.03942771628499031,
+          -0.015993082895874977,
+          -0.009677143767476082,
+          0.0030192218255251646,
+          0.02424757555127144,
+          -0.011889285407960415,
+          -0.007238315884023905,
+          0.011717316694557667,
+          0.009700593538582325,
+          0.004580618813633919,
+          0.00878603383898735,
+          0.06472273916006088,
+          0.0021867374889552593,
+          0.01638391986489296,
+          0.01283511333167553,
+          -0.021136507391929626,
+          -0.01919795200228691,
+          -0.005780491046607494,
+          -0.020354831591248512,
+          0.009012719616293907,
+          0.00900490302592516,
+          -0.011576615273952484,
+          -0.004068621434271336,
+          -0.020214131101965904,
+          0.0060736192390322685,
+          0.00731257488951087,
+          0.011936185881495476,
+          -0.0024192859418690205,
+          0.006870928220450878,
+          0.04274202138185501,
+          0.019448088482022285,
+          -0.0017196862027049065,
+          0.008105975575745106,
+          0.01419522799551487,
+          -0.009489541873335838,
+          -0.012420824728906155,
+          -0.002659651217982173,
+          0.0026107963640242815,
+          0.00877039972692728,
+          -0.010927824303507805,
+          0.033205579966306686,
+          -0.010419735684990883,
+          -0.015774212777614594,
+          0.011005992069840431,
+          -0.008066891692578793,
+          0.011013808660209179,
+          -0.012311390601098537,
+          0.017916003242135048,
+          -0.016665322706103325,
+          -0.0033221212215721607,
+          -0.0002916626981459558,
+          0.00019004486966878176,
+          0.017603334039449692,
+          0.0018095789710059762,
+          -0.010497902520000935,
+          0.0016815796261653304,
+          0.013851291500031948,
+          -0.008590614423155785,
+          0.003435464110225439,
+          -0.006261221133172512,
+          0.003959186840802431,
+          -0.011420279741287231,
+          -0.015555343590676785,
+          0.007191415410488844,
+          0.018822748214006424,
+          -0.026154864579439163,
+          -0.013210317119956017,
+          0.026576969772577286,
+          0.0071601481176912785,
+          0.002671376336365938,
+          0.013593338429927826,
+          -0.0058117578737437725,
+          0.0030954352114349604,
+          -0.016243217512965202,
+          -0.01777530275285244,
+          0.0061635118909180164,
+          -0.0048932889476418495,
+          0.019557522609829903,
+          -0.0024056066758930683,
+          0.004514176398515701,
+          -0.002837482374161482,
+          -0.008535897359251976,
+          0.019510623067617416,
+          0.015430275350809097,
+          -0.005170783959329128,
+          -0.006269038189202547,
+          0.011459363624453545,
+          -0.01827557571232319,
+          -0.020948905497789383,
+          0.003243953688070178,
+          0.020401732996106148,
+          -0.0065113576129078865,
+          0.004994906950742006,
+          -0.00030558629077859223,
+          -0.0012457953998818994,
+          -0.014664233662188053,
+          -0.009974180720746517,
+          -0.020902004092931747,
+          0.01919795200228691,
+          0.01336665265262127,
+          -0.02251225709915161,
+          0.0030192218255251646,
+          -0.005932917352765799,
+          0.02052680030465126,
+          -0.017587700858712196,
+          -0.003423738991841674,
+          0.033080510795116425,
+          -0.017759669572114944,
+          -0.004557168576866388,
+          -0.008231043815612793,
+          -0.014930003322660923,
+          0.016790391877293587,
+          0.03867730870842934,
+          0.0012037804117426276,
+          0.01014614850282669,
+          0.0029156499076634645,
+          -0.00452590174973011,
+          0.03558187186717987,
+          0.012647510506212711,
+          0.0004885472590103745,
+          -0.0032771748956292868,
+          -0.013835657387971878,
+          -0.02626429870724678,
+          0.0049440981820225716,
+          0.0026518343947827816,
+          -0.002016722923144698,
+          -5.20302819495555e-05,
+          -0.006014993414282799,
+          -0.007906648330390453,
+          0.006659876089543104,
+          -0.021668046712875366,
+          0.007797213736921549,
+          -0.016931092366576195,
+          -0.00276517728343606,
+          -0.00231375964358449,
+          -0.012796029448509216,
+          -0.005303668789565563,
+          0.010802756063640118,
+          -0.0016786482883617282,
+          -0.026123598217964172,
+          -0.03404978662729263,
+          -0.011670416221022606,
+          -0.012311390601098537,
+          -0.0007938892231322825,
+          0.00760179478675127,
+          -0.0267020370811224,
+          0.010224316269159317,
+          -0.0095364423468709,
+          -0.0014138557016849518,
+          0.0013679323019459844,
+          0.011983086355030537,
+          -0.006968637928366661,
+          0.0020362648647278547,
+          -0.02925029955804348,
+          -0.023434633389115334,
+          0.02135537751019001,
+          -0.003925965633243322,
+          0.012303573079407215,
+          -0.012538076378405094,
+          -0.004846388474106789,
+          -0.012592793442308903,
+          0.0177440345287323,
+          -0.017603334039449692,
+          -0.014007626101374626,
+          -0.014617333188652992,
+          -0.0005115089588798583,
+          -0.005366202909499407,
+          -0.009231588803231716,
+          -0.010083614848554134,
+          -0.010638604871928692,
+          -0.021152140572667122,
+          -0.014812751673161983,
+          -0.008442096412181854,
+          0.001930738682858646,
+          0.020151596516370773,
+          -0.006237770896404982,
+          0.026311200112104416,
+          0.02126157470047474,
+          0.00823886040598154,
+          -0.00520595908164978,
+          0.01425776258111,
+          -0.006937370635569096,
+          0.007679962553083897,
+          -0.00952080823481083,
+          0.0003239068028051406,
+          0.01671222411096096,
+          0.012850746512413025,
+          0.008410829119384289,
+          -0.004197597969323397,
+          0.002706551691517234,
+          0.01916668564081192,
+          0.015078521333634853,
+          0.01664968952536583,
+          0.005291943438351154,
+          0.008731315843760967,
+          0.010615154169499874,
+          0.0015399008989334106,
+          -0.005377927795052528,
+          0.0008955070516094565,
+          0.011701683513820171,
+          -0.011568798683583736,
+          -0.02005779556930065,
+          0.01329630147665739,
+          0.00930193904787302,
+          -0.018916549161076546,
+          0.009215954691171646,
+          0.0006575845764018595,
+          -0.009645876474678516,
+          -0.01836937665939331,
+          -0.014586065895855427,
+          -0.007805030792951584,
+          0.015430275350809097,
+          0.007918373681604862,
+          -0.022637324407696724,
+          0.01567259430885315,
+          0.010153966024518013,
+          -0.0015311070019379258,
+          -0.02712414227426052,
+          0.018619511276483536,
+          0.014054526574909687,
+          0.010638604871928692,
+          -0.04821375012397766,
+          -0.006069710943847895,
+          0.020730035379529,
+          -0.01329630147665739,
+          -0.026076696813106537,
+          -0.007625245023518801,
+          -0.008879834786057472,
+          -0.03902124613523483,
+          -0.0030758932698518038,
+          0.0029586420860141516,
+          -0.014844018965959549,
+          -0.003974820487201214,
+          0.015539710409939289,
+          0.00391228636726737,
+          0.018979083746671677,
+          -0.007781580090522766,
+          -0.01336665265262127,
+          -0.019213585183024406,
+          0.005202050786465406,
+          -0.010787122882902622,
+          -0.024263210594654083,
+          0.0027554063126444817,
+          -0.008465546183288097,
+          0.0022180043160915375,
+          0.01700926013290882,
+          0.042835820466279984,
+          0.01786910369992256,
+          -0.019917093217372894,
+          -0.025357555598020554,
+          -0.019307386130094528,
+          -0.007953548803925514,
+          0.009137787856161594,
+          0.007179690059274435,
+          -0.0021456994581967592,
+          0.003595707705244422,
+          0.006593433208763599,
+          0.031923629343509674,
+          0.0009536442230455577,
+          0.020886370912194252,
+          -3.169450064888224e-05,
+          -0.003814576892182231,
+          -0.002132020192220807,
+          -0.018979083746671677,
+          -0.0016053662402555346,
+          0.0029664586763828993,
+          0.013139965943992138,
+          0.004826846532523632,
+          0.005385744851082563,
+          0.016665322706103325,
+          -0.005628064274787903,
+          -0.0036035242956131697,
+          -0.017056161537766457,
+          -0.004428192041814327,
+          -0.004330482799559832,
+          0.03717648983001709,
+          0.020636234432458878,
+          -0.0199014600366354,
+          -0.0165871549397707,
+          -0.017431365326046944,
+          0.02194944955408573,
+          -0.00830921158194542,
+          -0.008997085504233837,
+          0.02099580504000187,
+          0.010239950381219387,
+          0.0015311070019379258,
+          0.0027280477806925774,
+          -0.006612975150346756,
+          0.011357746087014675,
+          0.02570149302482605,
+          0.012561526149511337,
+          -0.001964936964213848,
+          -0.004002179019153118,
+          -0.00016915948071982712,
+          -0.019323019310832024,
+          -0.01641518622636795,
+          0.00010302239388693124,
+          -0.012928914278745651,
+          0.025982895866036415,
+          -0.023106329143047333,
+          0.004314849153161049,
+          -0.01424212846904993,
+          -0.0024075608234852552,
+          -0.021996350958943367,
+          -0.011514080688357353,
+          0.014758034609258175,
+          -0.0237160362303257,
+          0.0054170116782188416,
+          -0.016759123653173447,
+          0.0010660100961104035,
+          0.011209228076040745,
+          -0.005159058608114719,
+          0.012561526149511337,
+          -0.00763697037473321,
+          0.004084255080670118,
+          0.004287490621209145,
+          0.00872349925339222,
+          -0.009145604446530342,
+          0.0015164506621658802,
+          0.005538171622902155,
+          -0.014578249305486679,
+          0.009239405393600464,
+          0.00925503857433796,
+          -0.004748679231852293,
+          0.03917757794260979,
+          0.015281757339835167,
+          0.00014949544856790453,
+          -0.009395739994943142,
+          -0.0038673398084938526,
+          -0.03140772506594658,
+          0.027014708146452904,
+          -0.012663144618272781,
+          0.021386643871665,
+          0.005983726587146521,
+          -0.01490655355155468,
+          -0.0016161142848432064,
+          -0.002688963897526264,
+          0.013194683939218521,
+          -0.009935096837580204,
+          0.0008969727205112576,
+          -0.013624604791402817,
+          0.011990902945399284,
+          0.00476822117343545,
+          -0.01016178261488676,
+          0.012295756489038467,
+          -0.015305208042263985,
+          0.0016571521991863847,
+          -0.007183598354458809,
+          0.0022844469640403986,
+          0.01588364690542221,
+          -0.0033416631631553173,
+          0.0028589784633368254,
+          -0.034675128757953644,
+          -0.015570977702736855,
+          -0.011529714800417423,
+          -8.244234777521342e-05,
+          0.007976999506354332,
+          -0.01754079945385456,
+          -0.028077786788344383,
+          -0.015977447852492332,
+          0.0099507300183177,
+          -0.0005926077719777822,
+          0.00028848715010099113,
+          -0.004658786579966545,
+          0.005225501023232937,
+          0.0009604838560335338,
+          -0.007468909956514835,
+          0.017525166273117065,
+          0.014875286258757114,
+          -0.026358099654316902,
+          -0.03081365115940571,
+          0.004564985167235136,
+          -0.0003722730034496635,
+          -0.027608782052993774,
+          -0.014077977277338505,
+          0.001066987169906497,
+          -0.0006072642281651497,
+          0.021199041977524757,
+          -0.020776936784386635,
+          0.0014588020276278257,
+          -0.00043260856182314456,
+          0.0030055425595492125,
+          -0.036394815891981125,
+          0.0030954352114349604,
+          -0.003144290065392852,
+          -0.0029273750260472298,
+          0.005565530154854059,
+          -0.01542245876044035,
+          0.0013278713449835777,
+          -0.008864200673997402,
+          -0.011428097262978554,
+          -0.025748392567038536,
+          -0.004592344164848328,
+          0.001534038339741528,
+          -8.647285721963271e-05,
+          0.023121964186429977,
+          -0.00622604601085186,
+          -0.019776392728090286,
+          0.002987954765558243,
+          -0.0019424638012424111,
+          -0.0030758932698518038,
+          0.018447542563080788,
+          0.020010894164443016,
+          -0.00651526590809226,
+          -0.0015105879865586758,
+          -0.0016258851392194629,
+          -0.001716754981316626,
+          -0.0004516619083005935,
+          0.039146311581134796,
+          -0.008551530539989471,
+          -0.006882653571665287,
+          0.0050261737778782845,
+          0.0016591063467785716,
+          -0.005444370210170746,
+          0.012100338004529476,
+          0.0015594427241012454,
+          0.007918373681604862,
+          0.0018388917669653893,
+          -0.010130515322089195,
+          -0.01824430748820305,
+          0.002528720535337925,
+          0.013851291500031948,
+          0.00581957446411252,
+          0.004197597969323397,
+          -0.014562616124749184,
+          0.008082525804638863,
+          -0.0075978864915668964,
+          -0.008778216317296028,
+          -0.010771489702165127,
+          0.007816756144165993,
+          -0.00444382568821311,
+          0.019526256248354912,
+          0.0003461357264313847,
+          0.021480444818735123,
+          0.022340288385748863,
+          -0.008442096412181854,
+          0.017024893313646317,
+          0.031485892832279205,
+          8.806063851807266e-05,
+          0.004744770936667919,
+          -0.011365562677383423,
+          -0.014679866842925549,
+          -0.013233767822384834,
+          0.03389345109462738,
+          0.004541534930467606,
+          -0.004267948679625988,
+          0.0018193499417975545,
+          -0.007316483184695244,
+          -0.01283511333167553,
+          0.012413008138537407,
+          0.003392471931874752,
+          -0.018556978553533554,
+          -0.004740862175822258,
+          0.009239405393600464,
+          0.006878744810819626,
+          0.012928914278745651,
+          0.02158987894654274,
+          -0.003869294188916683,
+          0.0028316196985542774,
+          0.004959731362760067,
+          0.029953807592391968,
+          -0.006374564487487078,
+          0.028593692928552628,
+          -0.0062494962476193905,
+          -0.0034823645837605,
+          -0.01107634324580431,
+          -0.015750762075185776,
+          -0.008426462300121784,
+          0.0029312833212316036,
+          0.0030602598562836647,
+          0.006186962127685547,
+          -0.01833810843527317,
+          0.006257312837988138,
+          0.02809341996908188,
+          0.0030055425595492125,
+          0.014734584838151932,
+          0.009278489276766777,
+          0.012741311453282833,
+          0.017321931198239326,
+          -0.00479167141020298,
+          -0.014851835556328297,
+          -0.003451097756624222,
+          0.020808203145861626,
+          -0.006683326326310635,
+          0.024654047563672066,
+          -0.0034745479933917522,
+          0.004350024741142988,
+          -0.00021972411195747554,
+          -0.0038810193073004484,
+          0.020479900762438774,
+          0.01500817108899355,
+          0.01664968952536583,
+          -0.001603411976248026,
+          0.00850463006645441,
+          0.0018349833553656936,
+          0.0017480220412835479,
+          -0.012913280166685581,
+          -0.016508987173438072,
+          0.01786910369992256,
+          0.0038477980997413397,
+          0.023168863728642464,
+          -0.022199586033821106,
+          -0.0007113247411325574,
+          0.006018901709467173,
+          0.016430821269750595,
+          0.004131155554205179,
+          0.010552620515227318,
+          -0.005444370210170746,
+          0.009544258937239647,
+          -0.005905558820813894,
+          0.0016913504805415869,
+          -0.01824430748820305,
+          -0.011600065045058727,
+          0.011631332337856293,
+          -0.00819195993244648,
+          0.010685505345463753,
+          0.001274131122045219,
+          0.009622426703572273,
+          -0.013843473978340626,
+          0.029672404751181602,
+          0.021402277052402496,
+          0.015234856866300106,
+          0.01499253697693348,
+          -0.0024779115337878466,
+          -0.007910557091236115,
+          -0.012420824728906155,
+          0.024560246616601944,
+          0.006800577510148287,
+          0.0029156499076634645,
+          -0.004475092515349388,
+          0.024153774604201317,
+          -0.0004384711501188576,
+          0.005151242017745972,
+          -0.0005886994185857475,
+          0.012967998161911964,
+          -0.01638391986489296,
+          0.008825116790831089,
+          -0.023559702560305595,
+          -0.013210317119956017,
+          -0.01913541741669178,
+          0.019588788971304893,
+          0.012741311453282833,
+          -0.0047721294686198235,
+          0.01543809287250042,
+          0.017603334039449692,
+          -0.0027319560758769512,
+          -0.027280477806925774,
+          -0.009786577895283699,
+          1.4351075151353143e-05,
+          -0.0032380910124629736,
+          -0.015727313235402107,
+          -0.0099507300183177,
+          0.002714368514716625,
+          -0.004885472357273102,
+          0.015547527000308037,
+          -0.0037637678906321526,
+          -0.024591512978076935,
+          -0.005686689633876085,
+          -0.007175781764090061,
+          -0.016555888578295708,
+          -0.013577704317867756,
+          0.008184143342077732,
+          0.005749223753809929,
+          -0.011506264097988605,
+          -0.018916549161076546,
+          0.018494443967938423,
+          -0.004693961702287197,
+          0.03214249759912491,
+          -0.015664778649806976,
+          -0.002808169461786747,
+          0.0053740194998681545,
+          0.01940118707716465,
+          0.0002865329443011433,
+          -0.0021183406934142113,
+          0.009638059884309769,
+          0.010615154169499874,
+          0.003232228569686413,
+          -0.007734679616987705,
+          0.0012917189160361886,
+          -0.0002904413267970085,
+          -0.019385553896427155,
+          0.005893833935260773,
+          -0.0005007609142921865,
+          -0.014070160686969757,
+          0.003351433901116252,
+          0.014335930347442627,
+          0.009622426703572273,
+          0.005452187266200781,
+          -0.02789018489420414,
+          0.012897646985948086,
+          -0.008582797832787037,
+          -0.011256128549575806,
+          -0.0016356561100110412,
+          0.01851007714867592,
+          -0.021965082734823227,
+          0.027937084436416626,
+          0.01419522799551487,
+          -0.00941919069737196,
+          0.005166875664144754,
+          0.019948361441493034,
+          0.017024893313646317,
+          0.011474997736513615,
+          -0.023794203996658325,
+          -0.004221048206090927,
+          0.01425776258111,
+          0.006304213311523199,
+          -0.0006717524374835193,
+          -0.0020206314511597157,
+          0.02463841438293457,
+          0.0034139680210500956,
+          -0.0018310750601813197,
+          -0.0070272632874548435,
+          -0.0018193499417975545,
+          0.009309755638241768,
+          0.021183408796787262,
+          0.009927279315888882,
+          0.018947815522551537,
+          -0.00131028366740793,
+          0.01578984595835209,
+          -0.004295307211577892,
+          -0.009669327177107334,
+          0.014844018965959549,
+          -0.0005882108816877007,
+          -0.009841295890510082,
+          0.01668095588684082,
+          0.008418645709753036,
+          0.012373924255371094,
+          -0.0008901330293156207,
+          -0.009872562251985073,
+          0.02871876023709774,
+          0.015109788626432419,
+          -0.0008701026090420783,
+          0.02046426571905613,
+          -0.02685837261378765,
+          0.024951083585619926,
+          0.012772578746080399,
+          0.0160868838429451,
+          -0.022340288385748863,
+          -0.004475092515349388,
+          0.00048488311585970223,
+          -0.018259940668940544,
+          -0.008700049482285976,
+          0.0037950349505990744,
+          -0.009989813901484013,
+          -0.0020264938939362764,
+          -0.037395358085632324,
+          0.01303053181618452,
+          -0.009427007287740707,
+          0.003249816130846739,
+          -0.015227040275931358,
+          -0.001274131122045219,
+          0.00532321073114872,
+          0.008543713949620724,
+          -0.007746404968202114,
+          -0.030469713732600212,
+          -0.004744770936667919,
+          0.016759123653173447,
+          -0.00835611205548048,
+          -0.019354287534952164,
+          0.024716582149267197,
+          -0.0004824404022656381,
+          -0.021699313074350357,
+          0.0041155219078063965,
+          -0.010544802993535995,
+          -0.007363383658230305,
+          -0.005518629681318998,
+          -0.008105975575745106,
+          -0.02424757555127144,
+          0.013843473978340626,
+          0.0019102196674793959,
+          -0.02046426571905613,
+          0.010232132859528065,
+          -0.01378094032406807,
+          0.004111613612622023,
+          0.008551530539989471,
+          -0.008895467966794968,
+          -0.011748583987355232,
+          0.027749482542276382,
+          0.022324655205011368,
+          -0.017618967220187187,
+          -0.015742946416139603,
+          9.386213787365705e-05,
+          -0.025060519576072693,
+          0.023700403049588203,
+          -0.00470959534868598,
+          0.0019825247582048178,
+          -0.011131060309708118,
+          0.011600065045058727,
+          0.010826206766068935,
+          0.012624060735106468,
+          -0.003554669674485922,
+          -0.0005579209537245333,
+          0.026514435186982155,
+          -0.009090886451303959,
+          0.003509723348543048,
+          -8.564843483327422e-06,
+          -0.012413008138537407,
+          0.02384110540151596,
+          -0.004486817866563797,
+          0.023950539529323578,
+          -0.006237770896404982,
+          -0.0006483022007159889,
+          -0.014961270615458488,
+          0.0026440175715833902,
+          -0.0031462442129850388,
+          -0.0069842711091041565,
+          -0.010286850854754448,
+          0.01283511333167553,
+          0.01213942188769579,
+          0.006835752632468939,
+          -0.010255583561956882,
+          0.0012184367515146732,
+          0.00038790650432929397,
+          -0.016696590930223465,
+          0.0023489349987357855,
+          -0.019713858142495155,
+          0.0009829570772126317,
+          -0.03329937905073166,
+          0.009708410128951073,
+          0.04430537298321724,
+          0.016634056344628334,
+          -0.008129426278173923,
+          -0.003537081880494952,
+          -0.01611815020442009,
+          0.005225501023232937,
+          -0.005706231575459242,
+          -0.011725133284926414,
+          -0.0136793227866292,
+          -0.01605561561882496,
+          0.0035312194377183914,
+          0.017916003242135048,
+          -0.0026518343947827816,
+          -0.020745670422911644,
+          0.02298126183450222,
+          -0.014500081539154053,
+          -0.026655137538909912,
+          -0.01754079945385456,
+          -0.02096453867852688,
+          0.007324300240725279,
+          0.020745670422911644,
+          -0.00579612422734499,
+          0.026576969772577286,
+          0.012428641319274902,
+          0.0028140321373939514,
+          -0.0010904374066740274,
+          0.01813487336039543,
+          -0.011349929496645927,
+          0.00452590174973011,
+          -0.0013816115679219365,
+          -0.020354831591248512,
+          0.0014783439692109823,
+          -0.008715682663023472,
+          -0.005600705277174711,
+          0.001242864178493619,
+          -0.005624155513942242,
+          -0.00260297954082489,
+          -0.0075978864915668964,
+          -0.017728401347994804,
+          0.02327829785645008,
+          -0.02348153479397297,
+          -0.002851161640137434,
+          -0.0005007609142921865,
+          -0.009591159410774708,
+          0.0111076096072793,
+          0.023950539529323578,
+          0.01261624414473772,
+          -0.015430275350809097,
+          0.01833810843527317,
+          -0.00037154016899876297,
+          0.026545701548457146,
+          -0.004662694875150919,
+          0.005585072096437216,
+          0.012694410979747772,
+          0.0008129425696097314,
+          0.01685292460024357,
+          -0.00710543105378747,
+          -0.011029441840946674,
+          0.0042913989163935184,
+          -0.00018686932162381709,
+          -0.021402277052402496,
+          -0.009755311533808708,
+          -0.0023782479111105204,
+          0.012217588722705841,
+          -0.017165595665574074,
+          0.015727313235402107,
+          -0.012741311453282833,
+          -0.001535992487333715,
+          -0.00387124833650887,
+          0.01397635880857706,
+          -0.04855768755078316,
+          -0.004963639657944441,
+          -0.008199776522815228,
+          0.03489399701356888,
+          -0.012444275431334972,
+          0.022434089332818985,
+          0.0040002246387302876,
+          -0.007621336728334427,
+          0.011834568344056606,
+          -0.010130515322089195,
+          -0.0004521504743024707,
+          0.015070704743266106,
+          -0.007347750477492809,
+          0.0017431365558877587,
+          0.016993626952171326,
+          -0.01836937665939331,
+          0.01674349047243595,
+          -0.019541889429092407,
+          0.0031071603298187256,
+          -0.012061254121363163,
+          -0.00640973960980773,
+          0.021386643871665,
+          0.0267020370811224,
+          -0.005815666168928146,
+          0.007398559246212244,
+          -0.009184688329696655,
+          0.007691687438637018,
+          -0.004252315033227205,
+          0.011545347981154919,
+          0.0010748038766905665,
+          -0.010232132859528065,
+          -0.0019317157566547394,
+          -0.009661509655416012,
+          0.020651869475841522,
+          -0.013382285833358765,
+          -0.012491175904870033,
+          -0.009559892117977142,
+          -0.004803396295756102,
+          -0.03351824730634689,
+          -0.005983726587146521,
+          -0.008137242868542671,
+          -0.012483359314501286,
+          -0.008426462300121784,
+          0.012733494862914085,
+          0.0010884832590818405,
+          0.0024290569126605988,
+          -0.009544258937239647,
+          -0.01144373044371605,
+          -0.013866924680769444,
+          0.015977447852492332,
+          0.004052987787872553,
+          -0.004111613612622023,
+          0.0038380271289497614,
+          0.00037056306609883904,
+          0.004795579705387354,
+          -0.00015438091941177845,
+          -0.00807470828294754,
+          -0.0032830373384058475,
+          0.012381740845739841,
+          0.006398014724254608,
+          -0.021011440083384514,
+          0.010388468392193317,
+          0.01893218234181404,
+          -0.0033572965767234564,
+          -0.006057985592633486,
+          -0.0064605483785271645,
+          -0.013382285833358765,
+          -0.004557168576866388,
+          0.0017665867926552892,
+          0.020917639136314392,
+          -0.01661842316389084,
+          -0.0013845429057255387,
+          0.019541889429092407,
+          -0.0027749482542276382,
+          0.023028161376714706,
+          0.0025384915061295033,
+          0.009935096837580204,
+          0.00988037884235382,
+          0.011740767396986485,
+          0.0062924884259700775,
+          -0.010951275005936623,
+          -0.02162114717066288,
+          0.001587778446264565,
+          -0.022434089332818985,
+          0.01588364690542221,
+          0.0023333015851676464,
+          -0.005100433249026537,
+          -0.021840015426278114,
+          -0.012506809085607529,
+          0.009512991644442081,
+          -0.02971930429339409,
+          -0.0007259811973199248,
+          -0.00737120071426034,
+          0.014265579171478748,
+          -0.007879289798438549,
+          0.0033866094890981913,
+          0.006042351946234703,
+          -0.007566619664430618,
+          0.00028799858409911394,
+          -0.0024505530018359423,
+          -0.01800980605185032,
+          -0.0045962524600327015,
+          0.00026381551288068295,
+          0.010372835211455822,
+          0.0038575688377022743,
+          0.015539710409939289,
+          0.023231398314237595,
+          0.031689126044511795,
+          -0.009653693065047264,
+          0.0018340062815696,
+          0.0009956592693924904,
+          0.016806025058031082,
+          0.005616338923573494,
+          0.0006727295112796128,
+          0.001513519324362278,
+          0.02510741911828518,
+          -0.011670416221022606,
+          -0.015899281948804855,
+          0.01596963219344616,
+          -0.020182862877845764,
+          -0.00010515979374758899,
+          -0.02029229700565338,
+          0.012084703892469406,
+          0.0011334295850247145,
+          0.013335385359823704,
+          0.013796573504805565,
+          -0.001073826802894473,
+          0.00819195993244648,
+          0.024794749915599823,
+          -0.005995451472699642,
+          0.0027358646038919687,
+          0.027233576402068138,
+          -0.00872349925339222,
+          -0.004779946058988571,
+          -0.000544730166438967,
+          0.002515041036531329,
+          0.00041257814154960215,
+          0.010599520988762379,
+          0.017822202295064926,
+          -0.027608782052993774,
+          -0.006034535355865955,
+          0.01240519154816866,
+          0.001964936964213848,
+          -0.006558258086442947,
+          0.004705687053501606,
+          0.0033553424291312695,
+          0.014851835556328297,
+          0.0003659218782559037,
+          0.01688419282436371,
+          -0.004819029942154884,
+          -0.02543572336435318,
+          -0.00930193904787302,
+          0.005835208110511303,
+          -0.012506809085607529,
+          -0.0032263658940792084,
+          -0.0034999523777514696,
+          -0.009512991644442081,
+          0.014265579171478748,
+          -0.008285760879516602,
+          0.0012340702814981341,
+          0.009395739994943142,
+          0.003802851540967822,
+          -0.007418101187795401,
+          -0.00215156190097332,
+          0.011881468817591667,
+          -0.012678777799010277,
+          0.004752587527036667,
+          0.022309020161628723,
+          0.003931828308850527,
+          -0.016821658238768578,
+          -0.018025439232587814,
+          0.006323755253106356,
+          0.03017267771065235,
+          -0.012194138951599598,
+          -0.0028765660244971514,
+          0.018916549161076546,
+          -0.0006644242093898356,
+          0.008215410634875298,
+          -0.002986000617966056,
+          -0.015023804269731045,
+          -0.03773929551243782,
+          -0.0019072884460911155,
+          -0.0018975174752995372,
+          0.005268493201583624,
+          -0.0003336777735967189,
+          -0.02374730445444584,
+          -0.007199232000857592,
+          -0.0103650176897645,
+          -0.002759314840659499,
+          -0.011021625250577927,
+          0.002632292453199625,
+          0.0008007289143279195,
+          -0.023684769868850708,
+          0.014031076803803444,
+          -0.012319207191467285,
+          0.010888740420341492,
+          -0.006100977770984173,
+          0.0190728846937418,
+          -0.0025169954169541597,
+          -0.0015457634581252933,
+          -0.008950185030698776,
+          0.006175236776471138,
+          0.013788756914436817,
+          0.0068513862788677216,
+          -0.011998720467090607,
+          0.0012467725900933146,
+          0.022856194525957108,
+          0.011818934231996536,
+          0.006179145537316799,
+          0.006683326326310635,
+          -0.011310845613479614,
+          -0.017884736880660057,
+          0.0020108604803681374,
+          -0.010451002046465874,
+          0.0033260295167565346,
+          0.007781580090522766,
+          0.014851835556328297,
+          0.005663239397108555,
+          -0.013694955967366695,
+          0.004760404117405415,
+          0.0103650176897645,
+          0.01694672554731369,
+          -0.014593882486224174,
+          -0.0034725936129689217,
+          -0.00925503857433796,
+          0.012483359314501286,
+          0.01916668564081192,
+          0.0042366813868284225,
+          0.006147878244519234,
+          0.004092071671038866,
+          0.015774212777614594,
+          -0.004228864796459675,
+          0.008332661353051662,
+          -0.013741856440901756,
+          -0.016493353992700577,
+          -0.019182318821549416,
+          0.004932372830808163,
+          0.00021899129205849022,
+          0.02745244652032852,
+          0.004557168576866388,
+          0.01976075768470764,
+          0.007457185070961714,
+          0.012749128974974155,
+          0.0020812111906707287,
+          0.022434089332818985,
+          -0.0009277511853724718,
+          -0.024153774604201317,
+          0.01206907071173191,
+          -0.021871281787753105,
+          -0.03070421703159809,
+          0.006843569688498974,
+          0.006612975150346756,
+          0.0381457693874836,
+          -0.022590424865484238,
+          0.011834568344056606,
+          -0.003156015183776617,
+          -0.023387733846902847,
+          -0.0022844469640403986,
+          -0.001389428274706006,
+          -0.0038380271289497614,
+          0.026358099654316902,
+          -0.0005198142607696354,
+          0.006315938662737608,
+          0.003521448466926813,
+          0.009810028597712517,
+          -0.00983347836881876,
+          0.021761847659945488,
+          0.01632138527929783,
+          0.018150506541132927,
+          0.005788307636976242,
+          -0.01112324371933937,
+          0.016993626952171326,
+          0.0010904374066740274,
+          -0.004955823067575693,
+          0.004568893928080797,
+          -0.005022265482693911,
+          0.0028257572557777166,
+          -0.0005564552848227322,
+          0.01474240142852068,
+          0.008707866072654724,
+          -0.015133239328861237,
+          0.010036714375019073,
+          -0.004568893928080797,
+          0.007898831740021706,
+          0.01949498802423477,
+          0.011975269764661789,
+          -0.002401698147878051,
+          0.009614609181880951,
+          -0.04430537298321724,
+          0.006140061654150486,
+          -0.03339318186044693,
+          -0.01233484037220478,
+          0.00270264339633286,
+          -0.017400098964571953,
+          0.0021105241030454636,
+          -0.01042755227535963,
+          -0.0175720676779747,
+          -0.006859203334897757,
+          0.019307386130094528,
+          0.01877584680914879,
+          0.004092071671038866,
+          0.00378917227499187,
+          0.010443185456097126,
+          -0.010873107239603996,
+          -0.013812207616865635,
+          -0.01394509244710207,
+          -0.012944547459483147,
+          -0.02049553394317627,
+          -0.0032556788064539433,
+          0.0009736746433191001,
+          -0.006441006902605295,
+          -0.002018677070736885,
+          0.009129970334470272,
+          -0.022043250501155853,
+          -0.010974724777042866,
+          0.014109243638813496,
+          -0.003392471931874752,
+          0.0006732181063853204,
+          -0.019119784235954285,
+          0.004256223328411579,
+          -0.003243953688070178,
+          0.00798481609672308,
+          -0.00581957446411252,
+          0.02431011013686657,
+          -0.01382784079760313,
+          -0.005053532775491476,
+          0.016821658238768578,
+          0.005491270683705807,
+          -0.007097613997757435,
+          -0.013546437956392765,
+          -0.03439372405409813,
+          0.025826560333371162,
+          -0.009770944714546204,
+          -0.01727502979338169,
+          -0.013288484886288643,
+          0.014984720386564732,
+          -0.009333206340670586,
+          -0.0018574565183371305,
+          0.011005992069840431,
+          0.0014988628681749105,
+          0.019713858142495155,
+          0.024826016277074814,
+          -0.010349384509027004,
+          0.0014206953346729279,
+          -0.01542245876044035,
+          0.012569342739880085,
+          -0.024700947105884552,
+          -0.009966363199055195,
+          0.013874741271138191,
+          0.012029986828565598,
+          0.002149607753381133,
+          -0.01661842316389084,
+          0.009661509655416012,
+          0.008363928645849228,
+          0.013906008563935757,
+          0.016993626952171326,
+          0.005467820446938276,
+          -0.010873107239603996,
+          -0.007914464920759201,
+          -0.024951083585619926,
+          0.0240443404763937,
+          0.005909467115998268,
+          -0.010974724777042866,
+          0.04368003085255623,
+          0.01993272639811039,
+          0.011998720467090607,
+          -0.01783783733844757,
+          -0.010552620515227318,
+          0.01664968952536583,
+          0.01218632236123085,
+          0.010802756063640118,
+          -0.00024036523245740682,
+          0.004932372830808163,
+          -0.0024603239726275206,
+          0.022168319672346115,
+          0.018213041126728058,
+          0.010185232385993004,
+          -0.05137171968817711,
+          0.014976903796195984,
+          0.002907833084464073,
+          -0.00520595908164978,
+          0.01685292460024357,
+          0.019948361441493034,
+          0.01217068824917078,
+          0.009606792591512203,
+          0.019854558631777763,
+          -0.00018735785852186382,
+          0.02194944955408573,
+          -0.009083069860935211,
+          -0.015758579596877098,
+          -0.00031950988341122866,
+          -0.013804391026496887,
+          -0.002776902401819825,
+          -0.019291752949357033,
+          0.009903829544782639,
+          0.00450245151296258,
+          -0.005237226374447346,
+          0.0240443404763937,
+          0.004209322854876518,
+          -0.029078330844640732,
+          -0.007656512316316366,
+          -0.029031431302428246,
+          0.0013796574203297496,
+          -0.019854558631777763,
+          -0.003818485187366605,
+          -0.014859652146697044,
+          -0.014421914704144001,
+          -0.026842739433050156,
+          -0.012241039425134659,
+          -0.00030363211408257484,
+          0.015023804269731045,
+          -0.001865273341536522,
+          -0.015633512288331985,
+          0.004529810044914484,
+          -0.03839590400457382,
+          -0.008582797832787037,
+          0.03645734861493111,
+          0.00819195993244648,
+          0.031954895704984665,
+          -0.008614065125584602,
+          0.01446099765598774,
+          -0.0010142240207642317,
+          -0.04261695221066475,
+          -0.0031657861545681953,
+          -0.003800897393375635,
+          0.00893455184996128,
+          0.007488451898097992,
+          0.0048932889476418495,
+          -0.012147238478064537,
+          0.016368286684155464,
+          -0.0014314433792605996,
+          0.00898926891386509,
+          0.0030524430330842733,
+          0.005151242017745972,
+          -0.010966908186674118,
+          0.010521353222429752,
+          -0.016368286684155464,
+          0.004248406738042831,
+          0.0019346470944583416,
+          0.004174147732555866,
+          -0.016868559643626213,
+          -0.016430821269750595,
+          -0.0037051422987133265,
+          -0.004576710518449545,
+          -0.012889830395579338,
+          -0.002970367204397917,
+          -0.011670416221022606,
+          -0.006655967328697443,
+          -0.008856384083628654,
+          -0.01431247964501381,
+          -0.011146693490445614,
+          0.0049362811259925365,
+          0.0027573604602366686,
+          0.008840750902891159,
+          0.014234311878681183,
+          0.003171648597344756,
+          0.007961365394294262,
+          0.02573275938630104,
+          0.001182284322567284,
+          -0.001272176974453032,
+          -0.01671222411096096,
+          0.001556511502712965,
+          -0.030157042667269707,
+          0.014453181065618992,
+          0.011748583987355232,
+          -0.025748392567038536,
+          0.006093161180615425,
+          -1.3885428415960632e-05,
+          0.0028570243157446384,
+          0.002175012370571494,
+          -0.0007220727857202291,
+          -0.014062343165278435,
+          0.003333846339955926,
+          -0.0180723387748003,
+          -0.00032488390570506454,
+          0.012467725202441216,
+          0.0401155911386013,
+          3.362579036547686e-06,
+          -0.0008657057187519968,
+          0.014429731294512749,
+          0.007707321085035801,
+          0.022074518725275993,
+          -0.01596963219344616,
+          -0.007015538401901722,
+          -0.03129829093813896,
+          -0.005342752672731876,
+          -0.005925100762397051,
+          -0.02198071777820587,
+          0.012241039425134659,
+          -0.0012545892968773842,
+          -0.010341567918658257,
+          -0.01366368867456913,
+          -0.02238718792796135,
+          0.012217588722705841,
+          0.02241845615208149,
+          0.008825116790831089,
+          0.007183598354458809,
+          0.008707866072654724,
+          -0.024450812488794327,
+          -0.020808203145861626,
+          0.029562970623373985,
+          0.013616788201034069,
+          -0.009231588803231716,
+          -0.01664968952536583,
+          -0.013585521839559078,
+          0.0031247481238096952,
+          -0.0006238748319447041,
+          0.0004951425944454968,
+          0.06219010800123215,
+          -0.018760213628411293,
+          0.0021847831085324287,
+          -0.034768927842378616,
+          -0.000998102012090385,
+          -0.009208138100802898,
+          0.013179049827158451,
+          0.005135608371347189,
+          -0.030188310891389847,
+          0.01618068479001522,
+          -0.030626049265265465,
+          -0.007257857825607061,
+          0.0015750762540847063,
+          -0.008137242868542671,
+          -0.010615154169499874,
+          -0.0023919271770864725,
+          -0.01827557571232319,
+          0.00013215202488936484,
+          -0.013327568769454956,
+          0.0021144323982298374,
+          0.002075348747894168,
+          -0.024700947105884552,
+          0.0015144963981583714,
+          0.006120519712567329,
+          -0.0046314275823533535,
+          -0.004350024741142988,
+          -0.019510623067617416,
+          -0.02019849605858326,
+          -0.014156145043671131,
+          -0.002032356569543481,
+          0.018025439232587814,
+          -0.02424757555127144,
+          0.01697799377143383,
+          -0.0032087783329188824,
+          0.004721320234239101,
+          0.003218549070879817,
+          0.008598431013524532,
+          0.018119240179657936,
+          0.01336665265262127,
+          0.0007044851081445813,
+          -0.010466636158525944,
+          0.01797853782773018,
+          -0.017947271466255188,
+          -0.02188691683113575,
+          0.001725548878312111,
+          0.0034081055782735348,
+          4.7083740355446935e-05,
+          0.0025932088028639555,
+          0.0045962524600327015,
+          0.013569887727499008,
+          0.007500177249312401,
+          -0.016196317970752716,
+          -0.002585391979664564,
+          -0.0027827650774270296,
+          0.029828740283846855,
+          0.0024603239726275206,
+          0.03276783972978592,
+          0.0012545892968773842,
+          0.015782030299305916,
+          -0.0014167869230732322,
+          -0.014570432715117931,
+          -0.017431365326046944,
+          -0.01314778346568346,
+          0.018728947266936302,
+          0.0016972130397334695,
+          0.007629153784364462,
+          -0.023700403049588203,
+          -0.01596963219344616,
+          0.003118885448202491,
+          -0.004049079492688179,
+          -0.01704052835702896,
+          0.0180723387748003,
+          -0.0177440345287323,
+          0.011521898210048676,
+          0.00032415110035799444,
+          -0.015023804269731045,
+          -0.017525166273117065,
+          0.014367196708917618,
+          0.0016923275543376803,
+          -0.000899904000107199,
+          0.012076887302100658,
+          -0.013257217593491077,
+          -0.011826751753687859,
+          0.002767131431028247,
+          0.00219259993173182,
+          -0.006824027746915817,
+          0.013616788201034069,
+          -0.016430821269750595,
+          0.005604614038020372,
+          -0.006824027746915817,
+          -0.0017802660586312413,
+          -0.010849656537175179,
+          0.013921641744673252,
+          0.012921096757054329,
+          -0.0015057026175782084,
+          0.012108154594898224,
+          0.012053437530994415,
+          0.003245907835662365,
+          0.014703317545354366,
+          0.005835208110511303,
+          -0.006378472782671452,
+          -0.010028897784650326,
+          0.0021124782506376505,
+          0.008676598779857159,
+          -0.004088163375854492,
+          -0.00383021030575037,
+          0.033080510795116425,
+          0.01668095588684082,
+          -0.017759669572114944,
+          0.021496077999472618,
+          -0.0038946985732764006,
+          0.007179690059274435,
+          -0.03186109662055969,
+          0.0004763335455209017,
+          0.013655872084200382,
+          -0.010873107239603996,
+          0.008223227225244045,
+          0.007816756144165993,
+          0.028171587735414505,
+          0.012194138951599598,
+          0.00819195993244648,
+          -0.0050769830122590065,
+          0.0021105241030454636,
+          0.013186866417527199,
+          0.0027964443434029818,
+          -0.002470094710588455,
+          -0.0026967807207256556,
+          0.01996399462223053,
+          -0.016337020322680473,
+          0.020370464771986008,
+          0.013335385359823704,
+          -0.029375368729233742,
+          0.005217684432864189,
+          -0.019010350108146667,
+          0.003961140755563974,
+          -0.006319846957921982,
+          -0.0037168674170970917,
+          -0.021386643871665,
+          0.007339933421462774,
+          0.02945353463292122,
+          -0.011772033758461475,
+          0.029734939336776733,
+          0.004326574504375458,
+          0.02016722969710827,
+          -0.009450457990169525,
+          -0.006198687478899956,
+          -0.009208138100802898,
+          0.0010054302401840687,
+          -0.008371745236217976,
+          0.011154510080814362,
+          0.0002304721565451473,
+          0.005846933461725712,
+          0.01424212846904993,
+          0.004060804378241301,
+          -0.003523402614519,
+          0.028828194364905357,
+          0.005925100762397051,
+          0.008231043815612793,
+          -0.013007081113755703,
+          0.005159058608114719,
+          -0.007863656617701054,
+          0.0038243476301431656,
+          0.0019854558631777763,
+          -0.008637514896690845,
+          9.306825086241588e-05,
+          -0.0021730579901486635,
+          0.013585521839559078,
+          0.009614609181880951,
+          0.013960725627839565,
+          -0.004678328521549702,
+          -0.008121608756482601,
+          0.025685859844088554,
+          0.001968845259398222,
+          -0.004310940857976675,
+          0.004729137290269136,
+          0.004881564062088728,
+          0.016931092366576195,
+          0.012960180640220642,
+          0.0006043329485692084,
+          -0.0021574245765805244,
+          -0.025185586884617805,
+          0.01212378777563572,
+          -0.018353741616010666,
+          -0.00039841028046794236,
+          -0.005698414985090494,
+          -0.022731125354766846,
+          0.013053981587290764,
+          -0.005248951260000467,
+          -0.003607432823628187,
+          -0.005542079918086529,
+          0.022965628653764725,
+          0.004064713139086962,
+          -0.014343746937811375,
+          -0.00399240804836154,
+          0.007468909956514835,
+          0.02765568159520626,
+          -0.0022121418733149767,
+          -0.00042381472303532064,
+          -0.010622970759868622,
+          -0.015289573930203915,
+          0.012749128974974155,
+          -0.015742946416139603,
+          -0.013624604791402817,
+          0.003101297887042165
+        ],
+        "title": "DeepSeekMoE: Achieving Ultimate Expert Specialization in Mixture-of-Experts Architectures",
+        "keyphrases": [
+          "DeepSeekMoE architecture",
+          "Mixture-of-Experts (MoE)",
+          "expert specialization",
+          "fine-grained expert segmentation",
+          "shared expert isolation"
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "767dcc8f-1e79-439d-92f3-8f20092ad68e",
+      "properties": {
+        "page_content": "2. Preliminaries: Mixture-Of-Experts For Transformers\n\nWe first introduce a generic MoE architecture commonly used in Transformer language models. A standard Transformer language model is constructed by stacking layers of standard Transformer blocks, where each block can be represented as follows:\n\n$$\\mathbf{u}{1:T}^{l}=\\text{Self-Att}\\left(\\mathbf{h}{1:T}^{l-1}\\right)+\\mathbf{h}{1:T}^{l-1},\\tag{1}$$ $$\\mathbf{h}{t}^{l}=\\text{FFN}\\left(\\mathbf{u}{t}^{l}\\right)+\\mathbf{u}{t}^{l},\\tag{2}$$\n\nwhere denotes the sequence length, Self-Att(·) denotes the self-attention module, FFN(·) denotes the Feed-Forward Network (FFN), u 1: ∈ R× are the hidden states of all tokens after the -th attention module, and h\n\n∈ Ris the output hidden state of the -th token after the -th Transformer block. For brevity, we omit the layer normalization in the above formulations.\n\nA typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022). An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. Then, each token will be assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts. If the -th FFN is substituted with an MoE layer, the computation for its output hidden state h\n\nis expressed as: where denotes the total number of experts, FFN(·) is the -th expert FFN, , denotes the gate value for the -th expert, , denotes the token-to-expert affinity, Topk(·, ) denotes the set comprising highest affinity scores among those calculated for the -th token and all experts, and e\n\nis the centroid of the -th expert in the -th layer. Note that ,is sparse, indicating that only out of gate values are nonzero. This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only experts. Also, in the above formulations, we omit the layer normalization operation for brevity.\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{N}\\left(g_{i,t}\\operatorname{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\operatorname{Topk}({s_{j,t}|1\\leqslant j\\leqslant N},K),\\ 0,&\\text{otherwise},\\end{cases}$$ $$s_{i,t}=\\operatorname{Softmax}{i}\\left(\\mathbf{u}{t}^{l}\\,\\mathbf{e}_{i}^{l}\\right),$$\n\n$$(3)$$\n\n(4) $\\binom{4}{5}$ .\n\n",
+        "title": "Preliminaries: Mixture-Of-Experts For Transformers",
+        "embedding": [
+          0.0012184951920062304,
+          -0.02724841982126236,
+          -0.02786005660891533,
+          -0.019281849265098572,
+          0.028028255328536034,
+          0.023196324706077576,
+          0.0037118708714842796,
+          0.03721809759736061,
+          -0.0036163025069981813,
+          0.03367060422897339,
+          -0.006009331438690424,
+          0.0015032885130494833,
+          -0.013440718874335289,
+          0.0020107559394091368,
+          -0.02350214309990406,
+          0.012936118058860302,
+          0.0041056121699512005,
+          0.036545298993587494,
+          0.003880070988088846,
+          -0.017630429938435555,
+          0.0016657545929774642,
+          -0.0020566286984831095,
+          -0.0359642431139946,
+          0.003516911529004574,
+          0.014763383194804192,
+          -0.0010483836522325873,
+          0.04584217816591263,
+          -0.011781653389334679,
+          -0.016330702230334282,
+          0.026667363941669464,
+          0.02402203530073166,
+          -0.003767300397157669,
+          0.002329953946173191,
+          -0.03529144451022148,
+          -0.05529196560382843,
+          -0.002939679427072406,
+          0.046514976769685745,
+          0.04406842961907387,
+          -0.03510795161128044,
+          0.027737729251384735,
+          0.019327722489833832,
+          0.008410005830228329,
+          -0.016453029587864876,
+          -0.02949618361890316,
+          -0.015283274464309216,
+          -0.009480370208621025,
+          -0.03651471808552742,
+          0.004227939527481794,
+          0.014755737967789173,
+          -0.027324873954057693,
+          -0.015206819400191307,
+          -0.008906960487365723,
+          0.009029287844896317,
+          0.025061817839741707,
+          -0.05318181961774826,
+          0.03244733065366745,
+          0.0012739247176796198,
+          0.0359642431139946,
+          0.053120654076337814,
+          0.006900027394294739,
+          0.025627581402659416,
+          -0.04003162682056427,
+          -0.020000522956252098,
+          -0.0015720976516604424,
+          0.040092792361974716,
+          0.0058067268691957,
+          0.019817031919956207,
+          -0.02495478093624115,
+          0.0116134537383914,
+          0.033640023320913315,
+          -0.03782973438501358,
+          0.008088896982371807,
+          -0.015275629237294197,
+          -0.018976030871272087,
+          -0.004931321833282709,
+          0.008203578181564808,
+          0.015994302928447723,
+          0.029465602710843086,
+          0.035230278968811035,
+          0.004912207834422588,
+          -0.02949618361890316,
+          0.027080219238996506,
+          -0.0006469970685429871,
+          0.021086178719997406,
+          0.028288202360272408,
+          -0.03168278560042381,
+          -0.01886899583041668,
+          -0.051530398428440094,
+          0.005971104372292757,
+          0.02536763623356819,
+          -0.023884417489171028,
+          0.01834910362958908,
+          -0.003622036660090089,
+          0.015084492042660713,
+          0.0006288390723057091,
+          0.038043808192014694,
+          0.055597785860300064,
+          -0.0010588961886242032,
+          -0.02565816417336464,
+          -0.011582871899008751,
+          0.010038488544523716,
+          0.010871844366192818,
+          0.005653817672282457,
+          0.023624470457434654,
+          0.017385775223374367,
+          -0.011712844483554363,
+          -0.006502463482320309,
+          0.042111191898584366,
+          0.034435153007507324,
+          0.08899315446615219,
+          0.007599587086588144,
+          0.0032454978208988905,
+          0.0005939566763117909,
+          0.043517958372831345,
+          -0.01177400816231966,
+          -0.012309190817177296,
+          -0.04550577700138092,
+          -0.057188041508197784,
+          0.003386938711628318,
+          -0.030214857310056686,
+          0.021223796531558037,
+          -0.003618213813751936,
+          0.009954389184713364,
+          -0.01877724938094616,
+          -0.05923702195286751,
+          0.037890899926424026,
+          -0.0064260088838636875,
+          0.012798500247299671,
+          -0.011085917241871357,
+          0.005256253760308027,
+          -0.001563496538437903,
+          0.020765069872140884,
+          -0.022125961259007454,
+          -0.03244733065366745,
+          0.036300644278526306,
+          0.02734016440808773,
+          0.023578599095344543,
+          0.020581578835844994,
+          0.0029587929602712393,
+          0.0038514004554599524,
+          0.016131920740008354,
+          -0.013822991400957108,
+          -0.0076875099912285805,
+          0.0016886909725144506,
+          -0.009725024923682213,
+          0.003293281886726618,
+          0.031147602945566177,
+          0.0689314678311348,
+          -0.0224776528775692,
+          0.011712844483554363,
+          -0.0219577606767416,
+          0.020520415157079697,
+          -0.02931269258260727,
+          0.01315783616155386,
+          -0.00795892346650362,
+          0.01703408546745777,
+          -0.007068227510899305,
+          0.009182197041809559,
+          -0.016208374872803688,
+          0.0015692306915298104,
+          -0.003788325469940901,
+          -0.030964111909270287,
+          0.0006613322766497731,
+          0.025168854743242264,
+          -0.009121033363044262,
+          0.018318522721529007,
+          0.04755476117134094,
+          0.024220816791057587,
+          -0.04868628829717636,
+          0.024419598281383514,
+          0.0011879133526235819,
+          0.03076533041894436,
+          0.040245700627565384,
+          0.0068847364746034145,
+          -0.015573801472783089,
+          0.012347417883574963,
+          0.007928341627120972,
+          0.021835433319211006,
+          0.011468189768493176,
+          0.025031235069036484,
+          -0.03929766267538071,
+          -0.00842529721558094,
+          0.032600242644548416,
+          0.007763964589685202,
+          0.03489387780427933,
+          0.018333813175559044,
+          0.037340424954891205,
+          0.019602959975600243,
+          0.010619543492794037,
+          0.015260337851941586,
+          0.03189685940742493,
+          0.011980435810983181,
+          0.03969522938132286,
+          -0.009633279405534267,
+          0.0167129747569561,
+          -0.03156045824289322,
+          -0.003220649901777506,
+          0.020994432270526886,
+          0.01137644425034523,
+          0.02050512284040451,
+          -0.0067088911309838295,
+          -0.01113943476229906,
+          0.003945057280361652,
+          -0.013287809677422047,
+          0.004407607484608889,
+          0.010000261478126049,
+          -0.0019381240708753467,
+          0.020902687683701515,
+          0.00837942399084568,
+          -0.04324272274971008,
+          0.02116263285279274,
+          -0.03422107920050621,
+          0.012278608977794647,
+          -0.009029287844896317,
+          0.006785345729440451,
+          0.0038552230689674616,
+          -0.017171703279018402,
+          0.0024656609166413546,
+          0.029664384201169014,
+          -0.017339903861284256,
+          -0.0012356974184513092,
+          0.00038083948311395943,
+          0.013058445416390896,
+          -0.0007425653166137636,
+          -0.004059739410877228,
+          -0.013196064159274101,
+          -0.017446938902139664,
+          -0.014763383194804192,
+          -0.0004931321600452065,
+          -0.006059026811271906,
+          -0.018456140533089638,
+          0.041407812386751175,
+          0.011903980746865273,
+          0.007400805130600929,
+          0.01877724938094616,
+          -0.010329016484320164,
+          0.021651942282915115,
+          0.008371778763830662,
+          0.028961002826690674,
+          -0.004357912112027407,
+          -0.02227886952459812,
+          0.02552054449915886,
+          -0.044404830783605576,
+          -0.03917533531785011,
+          0.025734618306159973,
+          0.026009853929281235,
+          -0.027798892930150032,
+          -0.0033391546458005905,
+          -0.011743426322937012,
+          0.0074275643564760685,
+          0.012714399956166744,
+          0.003622036660090089,
+          0.0030142224859446287,
+          0.005325062666088343,
+          -0.007278477773070335,
+          0.047585342079401016,
+          0.018456140533089638,
+          -0.017905667424201965,
+          -0.019924068823456764,
+          -0.01984761469066143,
+          0.028012964874505997,
+          0.018884286284446716,
+          0.001693469355814159,
+          0.017875084653496742,
+          0.02238590642809868,
+          -0.0010416939621791244,
+          0.05654582008719444,
+          -0.018945449963212013,
+          -0.01573435589671135,
+          -0.006972659379243851,
+          -0.002561229048296809,
+          -0.04853338003158569,
+          -0.02200363390147686,
+          0.007935987785458565,
+          0.05214203521609306,
+          -0.02327278070151806,
+          -0.0057073356583714485,
+          -0.0008553358493372798,
+          0.043426211923360825,
+          0.023869125172495842,
+          0.03519969806075096,
+          -2.6818839614861645e-05,
+          -0.004858689848333597,
+          0.020719196647405624,
+          0.03149929642677307,
+          -0.002643417799845338,
+          -0.016468320041894913,
+          -0.005168330855667591,
+          -0.03773799166083336,
+          -0.031346384435892105,
+          -0.017431648448109627,
+          -0.009686797857284546,
+          -0.03474096953868866,
+          0.002180867362767458,
+          0.017920957878232002,
+          0.02163665182888508,
+          -0.003543670754879713,
+          0.008333551697432995,
+          0.03177453204989433,
+          -0.02597927302122116,
+          -0.05841131508350372,
+          0.0032875477336347103,
+          0.004744007717818022,
+          0.0005552515503950417,
+          -0.04905327036976814,
+          0.0002306778624188155,
+          0.012561490759253502,
+          0.017446938902139664,
+          -0.02107088826596737,
+          -0.0047975261695683,
+          0.03090294823050499,
+          -0.007175263948738575,
+          -0.0014297009911388159,
+          -0.01163639035075903,
+          -0.03029131144285202,
+          -0.0015510726952925324,
+          0.011689907871186733,
+          -0.008494106121361256,
+          -0.026957891881465912,
+          -0.028823383152484894,
+          -0.007851887494325638,
+          0.009801479987800121,
+          -0.027645982801914215,
+          -0.015497347339987755,
+          0.006441299803555012,
+          0.012737336568534374,
+          0.021896596997976303,
+          0.02967967465519905,
+          -0.015520283952355385,
+          0.017064666375517845,
+          -0.0179821215569973,
+          -0.040704429149627686,
+          -0.028853965923190117,
+          0.0022038037423044443,
+          -0.008348842151463032,
+          -0.032508496195077896,
+          -0.004514643922448158,
+          0.0007502107764594257,
+          -0.01689646579325199,
+          -0.03498562425374985,
+          -0.023655053228139877,
+          -0.00943449791520834,
+          -0.014289364218711853,
+          0.011062980629503727,
+          -0.0020470719318836927,
+          -0.007836596108973026,
+          -0.01605546660721302,
+          0.011322925798594952,
+          0.007974214851856232,
+          0.010092006996273994,
+          0.0009609387489035726,
+          -0.0016925137024372816,
+          0.0062272269278764725,
+          -0.0016389954835176468,
+          -0.024465471506118774,
+          -0.03431282564997673,
+          -0.00899106077849865,
+          -0.015856683254241943,
+          0.007660750765353441,
+          0.0349244624376297,
+          -0.007974214851856232,
+          -0.03718751668930054,
+          -0.021988343447446823,
+          -0.0025115336757153273,
+          -0.0648946613073349,
+          -0.030214857310056686,
+          -0.0349244624376297,
+          -0.030092529952526093,
+          -0.0007473436999134719,
+          -0.0029301224276423454,
+          -0.007836596108973026,
+          0.024832453578710556,
+          -0.0006197600741870701,
+          -0.01240858156234026,
+          0.014778673648834229,
+          0.01587197557091713,
+          0.021560197696089745,
+          -0.011873398907482624,
+          -0.01793624833226204,
+          0.044404830783605576,
+          -0.022951669991016388,
+          -0.001025447272695601,
+          0.01998523250222206,
+          0.003316218266263604,
+          0.0026204814203083515,
+          0.021468451246619225,
+          0.0033601797185838223,
+          -0.007030000444501638,
+          -0.04639264941215515,
+          0.040704429149627686,
+          -0.00032756017753854394,
+          -0.02893042005598545,
+          0.009212778881192207,
+          0.007259364239871502,
+          0.011621098965406418,
+          0.02833407372236252,
+          -0.034343406558036804,
+          -0.014801610261201859,
+          -0.012018662877380848,
+          0.0012089383089914918,
+          -0.021713105961680412,
+          0.012125699780881405,
+          -0.015436183661222458,
+          -0.013134900480508804,
+          -0.023150453343987465,
+          -0.013815346173942089,
+          0.0033468001056462526,
+          0.011720489710569382,
+          0.03247791528701782,
+          -0.018379686400294304,
+          0.0033028386533260345,
+          -0.017676303163170815,
+          -0.0368817001581192,
+          -0.03029131144285202,
+          0.019113650545477867,
+          0.006926786620169878,
+          -0.02734016440808773,
+          0.024939490482211113,
+          -0.003333420492708683,
+          0.007385514210909605,
+          0.02293637953698635,
+          0.038563698530197144,
+          0.033028386533260345,
+          0.02935856580734253,
+          0.0001604352000867948,
+          0.02902216650545597,
+          -0.010665416717529297,
+          -0.010191397741436958,
+          -0.00784424226731062,
+          -0.0010264030424878001,
+          0.02425139956176281,
+          -0.007282300386577845,
+          0.05874771252274513,
+          -0.021789560094475746,
+          -0.009189843200147152,
+          -0.02541350945830345,
+          -0.006376313511282206,
+          0.01276027224957943,
+          -0.0184408500790596,
+          0.002058540005236864,
+          0.029832584783434868,
+          0.02082623355090618,
+          -0.01839497685432434,
+          0.015795519575476646,
+          -0.02223299816250801,
+          -0.02382325381040573,
+          -0.02786005660891533,
+          -0.0015548954252153635,
+          -0.011903980746865273,
+          -0.025153562426567078,
+          -0.03001607581973076,
+          -0.004839576315134764,
+          -0.09296879172325134,
+          -0.004101789090782404,
+          0.04321213811635971,
+          -0.005795258563011885,
+          0.032325003296136856,
+          -0.008937542326748371,
+          0.004151484929025173,
+          -0.03935882821679115,
+          0.00418971199542284,
+          0.03605598956346512,
+          0.039634063839912415,
+          0.0006541646434925497,
+          -0.005692044738680124,
+          -0.0009050313383340836,
+          -0.04675963148474693,
+          0.020428668707609177,
+          -0.005944345146417618,
+          -0.004289103206247091,
+          0.0002034409117186442,
+          -0.030964111909270287,
+          0.02293637953698635,
+          0.02620863728225231,
+          0.03749333694577217,
+          0.03908359259366989,
+          -0.017049375921487808,
+          -0.06532280892133713,
+          -0.0015376930823549628,
+          -0.000322303909342736,
+          0.025811072438955307,
+          0.009556825272738934,
+          -0.004442012403160334,
+          -0.009801479987800121,
+          -0.005252431146800518,
+          -0.01998523250222206,
+          0.016040174290537834,
+          0.01282908208668232,
+          0.013356618583202362,
+          0.0304289311170578,
+          -0.011491126380860806,
+          0.002372003858909011,
+          0.024312563240528107,
+          0.031071148812770844,
+          0.0031212589237838984,
+          -0.050582364201545715,
+          -0.017125830054283142,
+          -0.009319815784692764,
+          0.04883919656276703,
+          0.011361153796315193,
+          -0.021850723773241043,
+          -0.011116499081254005,
+          0.02256939746439457,
+          0.026147473603487015,
+          0.0020967673044651747,
+          -0.018853703513741493,
+          -0.015482055954635143,
+          0.011177662760019302,
+          -0.018700795248150826,
+          0.007309059612452984,
+          -0.0019916424062103033,
+          -0.010497216135263443,
+          -0.002899540588259697,
+          -0.0003841843572445214,
+          -0.018043285235762596,
+          0.01909835822880268,
+          -0.00835648737847805,
+          0.04859454184770584,
+          -0.01074187085032463,
+          0.02055099606513977,
+          -0.003369736485183239,
+          -0.0021827789023518562,
+          -0.0023758267052471638,
+          0.05113283544778824,
+          -0.00784424226731062,
+          0.01582610234618187,
+          0.04097966477274895,
+          -0.002408319851383567,
+          0.04657614231109619,
+          -0.011452899314463139,
+          -0.033120132982730865,
+          -0.0026663541793823242,
+          0.012225090526044369,
+          0.006150772329419851,
+          0.004617857746779919,
+          -0.022829342633485794,
+          -0.00800479669123888,
+          0.004526112228631973,
+          -0.029236238449811935,
+          -0.01591784693300724,
+          -0.013089027255773544,
+          -0.012125699780881405,
+          0.006842686794698238,
+          0.004338798578828573,
+          0.009212778881192207,
+          -0.021009724587202072,
+          0.044496577233076096,
+          -0.028395237401127815,
+          0.02266114391386509,
+          0.02659090980887413,
+          -0.008287678472697735,
+          -0.0008639369625598192,
+          -0.023945581167936325,
+          0.012852017767727375,
+          -0.009870288893580437,
+          -0.022309452295303345,
+          -0.01694233901798725,
+          0.006475704722106457,
+          0.0011238825973123312,
+          -0.045995086431503296,
+          0.03464922308921814,
+          0.01666710339486599,
+          -0.016300121322274208,
+          0.004560516681522131,
+          -0.0048013487830758095,
+          0.016743557527661324,
+          -0.002679733792319894,
+          0.02411377988755703,
+          0.013861218467354774,
+          0.02574990876019001,
+          -0.010458989068865776,
+          0.018884286284446716,
+          0.003998575732111931,
+          0.022997543215751648,
+          0.015053910203278065,
+          0.03253907710313797,
+          0.05663756653666496,
+          -0.005600299220532179,
+          -0.007989505305886269,
+          -0.012507972307503223,
+          0.019419468939304352,
+          -0.016315411776304245,
+          -0.00784424226731062,
+          0.02977142110466957,
+          0.015994302928447723,
+          -0.006575095467269421,
+          -0.0032990160398185253,
+          -0.0159637201577425,
+          0.013165482319891453,
+          -0.017768049612641335,
+          0.006120190490037203,
+          0.035750169306993484,
+          0.003279902273789048,
+          -0.02162136137485504,
+          0.03061242215335369,
+          -0.0040329801850020885,
+          0.034007005393505096,
+          0.00630750460550189,
+          0.02168252505362034,
+          -0.038838937878608704,
+          -0.013517173007130623,
+          0.017661012709140778,
+          0.021040305495262146,
+          -0.009052224457263947,
+          -0.01782921329140663,
+          0.028899839147925377,
+          0.018135031685233116,
+          -0.002549760974943638,
+          -0.01418232824653387,
+          -0.004430544096976519,
+          0.02851756475865841,
+          -0.020673323422670364,
+          0.023532725870609283,
+          0.005581185687333345,
+          0.02781418338418007,
+          -0.0010273586958646774,
+          -0.0013360441662371159,
+          0.004040625412017107,
+          -0.016544776037335396,
+          0.004805171396583319,
+          0.03825788199901581,
+          0.014503437094390392,
+          0.0266826543956995,
+          -4.62012758362107e-05,
+          0.011942208744585514,
+          0.03149929642677307,
+          -0.010657771490514278,
+          -0.02139199711382389,
+          0.0038838936015963554,
+          -0.0033487114124000072,
+          -0.045016467571258545,
+          0.00017799586930777878,
+          -0.007114100269973278,
+          -0.03090294823050499,
+          -0.013945318758487701,
+          0.0004625503206625581,
+          0.01500803790986538,
+          0.008654660545289516,
+          0.03685111552476883,
+          -0.022080088034272194,
+          -0.03180511295795441,
+          0.016774138435721397,
+          0.019281849265098572,
+          0.0334259495139122,
+          -0.008341196924448013,
+          0.005841131322085857,
+          -0.0055200220085680485,
+          0.007026177830994129,
+          0.024939490482211113,
+          0.0031308159232139587,
+          -0.0046255034394562244,
+          0.008585851639509201,
+          0.013899446465075016,
+          0.035750169306993484,
+          -0.024052616208791733,
+          -0.01591784693300724,
+          -0.03263082355260849,
+          -0.032233260571956635,
+          0.029389148578047752,
+          0.003199625061824918,
+          -0.014327592216432095,
+          0.0159637201577425,
+          0.020092269405722618,
+          -0.016590647399425507,
+          0.012890245765447617,
+          -0.014923937618732452,
+          -0.04293690249323845,
+          0.015130365267395973,
+          -0.02200363390147686,
+          0.007351109758019447,
+          -0.003146106842905283,
+          -0.004487885162234306,
+          -0.010504862293601036,
+          0.006452768109738827,
+          0.012248027138411999,
+          -0.03467980772256851,
+          0.012018662877380848,
+          0.03510795161128044,
+          -0.006406895350664854,
+          -0.01221744529902935,
+          -0.01402177382260561,
+          0.011598162353038788,
+          -0.019067777320742607,
+          0.00490456260740757,
+          0.00750401895493269,
+          0.004866335075348616,
+          -0.007018532138317823,
+          -0.003495886456221342,
+          -0.02050512284040451,
+          -0.007557536941021681,
+          0.013456009328365326,
+          -0.011491126380860806,
+          -0.00448406208306551,
+          0.001542471582069993,
+          0.007890114560723305,
+          0.0008448233129456639,
+          -0.008310615085065365,
+          0.013517173007130623,
+          0.019862905144691467,
+          0.016544776037335396,
+          0.0024580154567956924,
+          0.01474809180945158,
+          -0.027141382917761803,
+          -0.00588700408115983,
+          0.03168278560042381,
+          -0.0051989126950502396,
+          0.005764676723629236,
+          -0.011605808511376381,
+          -0.02752365544438362,
+          -0.003889627754688263,
+          -0.0031480181496590376,
+          0.029557347297668457,
+          0.0014134544180706143,
+          0.019740577787160873,
+          0.025015944615006447,
+          -0.0036832003388553858,
+          0.012553845532238483,
+          0.01121588982641697,
+          -0.037340424954891205,
+          0.034159913659095764,
+          -0.00941156130284071,
+          0.014151746407151222,
+          0.03431282564997673,
+          -0.015650255605578423,
+          -0.011743426322937012,
+          -0.002968349726870656,
+          0.03061242215335369,
+          0.04914501681923866,
+          0.013662436977028847,
+          0.005523844622075558,
+          0.04357912018895149,
+          -0.018716085702180862,
+          -0.04461890459060669,
+          0.03056654892861843,
+          0.020398087799549103,
+          -0.007645459845662117,
+          -0.00011969923070864752,
+          0.02659090980887413,
+          -0.017676303163170815,
+          0.003742452710866928,
+          0.04119373857975006,
+          -0.0024121426977217197,
+          0.010535444132983685,
+          -0.01909835822880268,
+          -0.024419598281383514,
+          0.04452715814113617,
+          -0.027263710275292397,
+          0.032508496195077896,
+          -0.015000392682850361,
+          0.007851887494325638,
+          0.04657614231109619,
+          -0.004671376198530197,
+          0.023471562191843987,
+          0.03113231249153614,
+          -0.03287547826766968,
+          0.0006670663715340197,
+          0.018899576738476753,
+          -0.011491126380860806,
+          -0.003715693484991789,
+          0.003880070988088846,
+          0.021667232736945152,
+          0.012064536102116108,
+          0.0036774661857634783,
+          0.021407287567853928,
+          0.0028632248286157846,
+          -0.0029836406465619802,
+          -0.014128809794783592,
+          0.018012704327702522,
+          -0.003092588623985648,
+          -0.015237401239573956,
+          0.014641055837273598,
+          0.00230510625988245,
+          0.022966962307691574,
+          -0.03437398746609688,
+          -0.0035723410546779633,
+          0.0010627189185470343,
+          0.0034404569305479527,
+          -0.044037848711013794,
+          -0.005202735308557749,
+          -0.015176237560808659,
+          -0.023762090131640434,
+          -0.013004926964640617,
+          0.006922964006662369,
+          0.016911758109927177,
+          -0.023471562191843987,
+          -0.007897759787738323,
+          0.02906803786754608,
+          -0.03376235067844391,
+          -0.004052093718200922,
+          0.00697648199275136,
+          0.005879358854144812,
+          -0.019419468939304352,
+          -0.032233260571956635,
+          0.020489832386374474,
+          0.05275367200374603,
+          0.0014975544763728976,
+          -0.0008338329498656094,
+          0.029832584783434868,
+          -0.012377999722957611,
+          -0.027370747178792953,
+          0.029939621686935425,
+          -0.01775275729596615,
+          0.022768178954720497,
+          0.03159103915095329,
+          -0.0449553057551384,
+          -0.013081382028758526,
+          -0.003134638536721468,
+          -0.03052067570388317,
+          0.005351821891963482,
+          -0.014847483485937119,
+          -0.0031289043836295605,
+          -0.01062718965113163,
+          -0.02438901737332344,
+          0.02536763623356819,
+          -0.009289233945310116,
+          0.01784450374543667,
+          -0.021239086985588074,
+          -8.111355418805033e-05,
+          -0.019327722489833832,
+          -0.022401196882128716,
+          0.010092006996273994,
+          -0.011414671316742897,
+          0.01159051712602377,
+          0.012095117941498756,
+          -0.024465471506118774,
+          -0.021009724587202072,
+          0.012270962819457054,
+          0.008708178997039795,
+          0.013066090643405914,
+          0.003167131682857871,
+          0.01639186590909958,
+          -0.010917716659605503,
+          -0.0099696796387434,
+          0.03174395114183426,
+          0.00028455446590669453,
+          0.022905798628926277,
+          -0.017446938902139664,
+          -0.030979404225945473,
+          0.0025574064347893,
+          0.01077245268970728,
+          -0.03847195580601692,
+          -0.004040625412017107,
+          -0.02986316569149494,
+          0.00022434646962210536,
+          -0.034343406558036804,
+          0.007461968809366226,
+          0.015390310436487198,
+          -0.007182909641414881,
+          -0.0007105499389581382,
+          -0.00035431928699836135,
+          0.02266114391386509,
+          0.021132051944732666,
+          0.000617370882537216,
+          0.014541665092110634,
+          -0.005787613335996866,
+          -0.01156758051365614,
+          -0.015673192217946053,
+          0.0003980417677666992,
+          0.021927179768681526,
+          -0.003945057280361652,
+          -0.04709603264927864,
+          -0.016682393848896027,
+          -0.004958080593496561,
+          -0.01769159361720085,
+          0.01261500921100378,
+          0.007599587086588144,
+          0.01102475356310606,
+          0.0032340295147150755,
+          0.009335106238722801,
+          -0.06055204197764397,
+          -0.030000785365700722,
+          0.02204950712621212,
+          -0.018135031685233116,
+          0.023563306778669357,
+          0.010420762002468109,
+          0.023089289665222168,
+          -0.01243151817470789,
+          0.00023462004901375622,
+          0.023379815742373466,
+          -0.005107167176902294,
+          0.018746668472886086,
+          0.0024866857565939426,
+          0.02434314414858818,
+          -0.0259333997964859,
+          0.030321894213557243,
+          -0.009862643666565418,
+          0.019954649731516838,
+          -0.02097914181649685,
+          -0.038838937878608704,
+          -0.033548276871442795,
+          0.018761958926916122,
+          0.00335253425873816,
+          -0.02697318233549595,
+          -0.0017575001111254096,
+          0.0068962047807872295,
+          0.02547467313706875,
+          -0.021560197696089745,
+          0.004575807601213455,
+          -0.0033104841131716967,
+          0.005833486095070839,
+          0.015887266024947166,
+          -0.015948429703712463,
+          -0.01966412365436554,
+          -0.01680472120642662,
+          0.050918761640787125,
+          0.009006352163851261,
+          0.02565816417336464,
+          -0.011468189768493176,
+          0.007091164123266935,
+          0.008868733420968056,
+          -0.020749777555465698,
+          0.005867890547960997,
+          0.02692730911076069,
+          -0.04807465150952339,
+          0.008081250824034214,
+          0.032569658011198044,
+          0.04663730412721634,
+          -0.02536763623356819,
+          0.02458779886364937,
+          0.0029702612664550543,
+          -0.003264611354097724,
+          0.031300511211156845,
+          0.02541350945830345,
+          0.030643003061413765,
+          -0.0167129747569561,
+          -0.009625634178519249,
+          -0.01957237720489502,
+          0.01975586824119091,
+          0.01998523250222206,
+          0.003597188973799348,
+          -0.044588323682546616,
+          0.022034214809536934,
+          -0.010611898265779018,
+          0.013471300713717937,
+          0.01848672144114971,
+          -0.005340353585779667,
+          0.032416749745607376,
+          0.013731245882809162,
+          -0.028762219473719597,
+          0.007549891714006662,
+          0.001983996946364641,
+          -0.0006933476543053985,
+          0.019817031919956207,
+          0.040184538811445236,
+          -0.005653817672282457,
+          -0.02163665182888508,
+          -0.010092006996273994,
+          0.014511083252727985,
+          -0.0045910985209047794,
+          0.022309452295303345,
+          -0.0043540894985198975,
+          0.019725287333130836,
+          0.004334975965321064,
+          -0.006598032079637051,
+          0.009526243433356285,
+          0.03498562425374985,
+          -0.006857977714389563,
+          0.015115073882043362,
+          -0.014251137152314186,
+          0.01821148581802845,
+          0.022814052179455757,
+          -0.02607101760804653,
+          -0.026835564523935318,
+          -0.011903980746865273,
+          -0.024450181052088737,
+          0.01058131642639637,
+          -0.003994752652943134,
+          -0.04143839329481125,
+          -0.03608657047152519,
+          -0.0056767538189888,
+          -0.018517304211854935,
+          -0.03718751668930054,
+          -0.021835433319211006,
+          -0.027508364990353584,
+          0.0008424341212958097,
+          0.004162952769547701,
+          0.011101207695901394,
+          -0.016177793964743614,
+          -0.015459120273590088,
+          0.009656216017901897,
+          0.031101731583476067,
+          -0.027508364990353584,
+          0.043884940445423126,
+          0.012492681853473186,
+          -0.02139199711382389,
+          0.009228070266544819,
+          0.033915258944034576,
+          0.0002752365544438362,
+          -0.0244348905980587,
+          0.019878195598721504,
+          0.0028288201428949833,
+          0.017294030636548996,
+          0.022645851597189903,
+          -0.011277053505182266,
+          0.0030008431058377028,
+          0.004526112228631973,
+          -0.005290658213198185,
+          -0.04269224777817726,
+          -0.0023203971795737743,
+          0.003195802215486765,
+          -0.025015944615006447,
+          0.0007052936707623303,
+          0.005164508242160082,
+          0.026804981753230095,
+          -0.0018005057936534286,
+          0.030031366273760796,
+          0.05813607573509216,
+          -0.020703906193375587,
+          -0.04452715814113617,
+          0.005088053643703461,
+          -0.010137880221009254,
+          0.0021942469757050276,
+          -0.029465602710843086,
+          0.03801322728395462,
+          -0.013868864625692368,
+          0.0033926728647202253,
+          -0.005003953352570534,
+          -0.0036239479668438435,
+          -0.029649093747138977,
+          0.021223796531558037,
+          0.0019161433447152376,
+          -0.006575095467269421,
+          -0.036117151379585266,
+          -0.0338846780359745,
+          0.02472541667521,
+          0.019511213526129723,
+          -0.022814052179455757,
+          -0.021284960210323334,
+          -0.0010426496155560017,
+          -0.011804590001702309,
+          0.026330964639782906,
+          -0.01834910362958908,
+          -0.01675884798169136,
+          0.0116134537383914,
+          0.023578599095344543,
+          0.018104448914527893,
+          -0.015451474115252495,
+          -0.06046029552817345,
+          -0.02191188745200634,
+          0.02125437930226326,
+          -0.00647188164293766,
+          -0.003503531916067004,
+          -0.016774138435721397,
+          0.007286123465746641,
+          -0.019220685586333275,
+          -0.025826362892985344,
+          0.038563698530197144,
+          -0.004239407368004322,
+          0.0227070152759552,
+          0.0004496486217249185,
+          0.0018645365489646792,
+          0.04786057770252228,
+          0.008784633129835129,
+          -0.020612159743905067,
+          -0.010619543492794037,
+          0.01376182772219181,
+          0.041744209825992584,
+          -0.01966412365436554,
+          0.005634704139083624,
+          0.01296669989824295,
+          -0.01102475356310606,
+          -0.013639500364661217,
+          0.0007310971268452704,
+          0.0099696796387434,
+          -0.002119703684002161,
+          0.004755476024001837,
+          -0.020765069872140884,
+          0.015115073882043362,
+          0.02088739722967148,
+          0.012118053622543812,
+          -0.003795970929786563,
+          -0.0037692117039114237,
+          -0.00027021922869607806,
+          0.01418232824653387,
+          0.016269538551568985,
+          0.02411377988755703,
+          0.024786580353975296,
+          -0.001051250728778541,
+          -0.01896074041724205,
+          -0.004182066768407822,
+          -0.003694668412208557,
+          0.001217539538629353,
+          0.02992432937026024,
+          0.032936640083789825,
+          -0.008738760836422443,
+          0.006357199978083372,
+          -0.018792539834976196,
+          0.012844372540712357,
+          0.021973051130771637,
+          -0.018884286284446716,
+          -0.018196195363998413,
+          0.025168854743242264,
+          -0.003528379835188389,
+          0.01441933773458004,
+          0.002433167537674308,
+          0.006364845205098391,
+          0.0206886138767004,
+          0.06813634186983109,
+          -0.015979010611772537,
+          -0.008050668984651566,
+          -0.022018924355506897,
+          0.0031136134639382362,
+          0.025444090366363525,
+          -0.026147473603487015,
+          0.013119609095156193,
+          0.00559265399351716,
+          0.02350214309990406,
+          0.016605939716100693,
+          0.004923676140606403,
+          0.0023280426394194365,
+          0.01214863546192646,
+          0.0025153562892228365,
+          0.016254248097538948,
+          0.0004979106015525758,
+          -0.00819593295454979,
+          0.0035207343753427267,
+          0.01591784693300724,
+          0.015160947106778622,
+          0.027982383966445923,
+          -0.0007554669864475727,
+          0.025734618306159973,
+          -0.03076533041894436,
+          0.022064797580242157,
+          -0.007599587086588144,
+          0.0020031104795634747,
+          -0.01802799478173256,
+          -0.008937542326748371,
+          0.011552290059626102,
+          -0.001057940535247326,
+          -0.0001701114815659821,
+          0.0025669632013887167,
+          -0.007010886911302805,
+          0.03090294823050499,
+          3.2761989132268354e-05,
+          0.011445253156125546,
+          0.008478814736008644,
+          -0.007419918663799763,
+          0.004308216739445925,
+          -0.001004422316327691,
+          0.01773746684193611,
+          -0.029832584783434868,
+          -0.010825971141457558,
+          0.012936118058860302,
+          0.016361285001039505,
+          -0.009059869684278965,
+          0.005130103789269924,
+          -0.023425688967108727,
+          0.0007463880465365946,
+          0.012951409444212914,
+          -0.010321371257305145,
+          0.006773877423256636,
+          -0.024037325754761696,
+          -0.011277053505182266,
+          0.005703513044863939,
+          -0.034526895731687546,
+          -0.006678309291601181,
+          0.008761697448790073,
+          -0.003845666302368045,
+          0.020749777555465698,
+          0.0036832003388553858,
+          0.01116237137466669,
+          -0.022783471271395683,
+          -0.011934562586247921,
+          -0.0022745244204998016,
+          0.0032951931934803724,
+          0.0010808769147843122,
+          0.006987950298935175,
+          -0.01605546660721302,
+          0.028961002826690674,
+          -0.024924200028181076,
+          -0.015252692624926567,
+          -0.004101789090782404,
+          -0.0024637493770569563,
+          0.0008410005830228329,
+          -0.01463341061025858,
+          0.004380848724395037,
+          -0.007710446137934923,
+          0.034007005393505096,
+          0.035322025418281555,
+          -0.005003953352570534,
+          -0.01077245268970728,
+          -0.03217209503054619,
+          0.015619674697518349,
+          -0.018563177436590195,
+          -0.0034939751494675875,
+          -0.0050498261116445065,
+          -0.006815927568823099,
+          0.0005284924409352243,
+          -0.02533705346286297,
+          0.0113535076379776,
+          0.002578431274741888,
+          0.0018893842352554202,
+          0.0011420405935496092,
+          0.011147080920636654,
+          0.015092138200998306,
+          -0.02003110572695732,
+          -0.040184538811445236,
+          -0.024878326803445816,
+          -0.002654885873198509,
+          0.009801479987800121,
+          -0.016254248097538948,
+          -0.008310615085065365,
+          -0.004820462316274643,
+          0.0116134537383914,
+          -0.006716536357998848,
+          0.006942077539861202,
+          0.003516911529004574,
+          0.0025841654278337955,
+          -0.010092006996273994,
+          -0.019358305260539055,
+          -0.010474280454218388,
+          0.01568848453462124,
+          0.020306341350078583,
+          0.007492550648748875,
+          -0.04672905057668686,
+          -0.0023643583990633488,
+          0.002708404092118144,
+          0.007473437115550041,
+          -0.006215759087353945,
+          0.03529144451022148,
+          0.0007846153457649052,
+          2.241374022560194e-05,
+          0.03737100958824158,
+          0.011062980629503727,
+          -0.016743557527661324,
+          0.012309190817177296,
+          -0.015459120273590088,
+          -0.002947324886918068,
+          0.00021455071691889316,
+          0.03168278560042381,
+          0.005290658213198185,
+          0.028074128553271294,
+          -0.0264532919973135,
+          -8.410005830228329e-05,
+          0.003337243339046836,
+          0.02827291004359722,
+          -0.015841392800211906,
+          -0.023762090131640434,
+          0.024006744846701622,
+          0.03929766267538071,
+          -0.007905405946075916,
+          -0.010634834878146648,
+          -0.03330362215638161,
+          -0.025918109342455864,
+          0.047829996794462204,
+          0.025872236117720604,
+          0.004598744213581085,
+          0.02102501504123211,
+          -0.021804852411150932,
+          0.015023328363895416,
+          0.015099783428013325,
+          0.0008868733420968056,
+          -0.030918240547180176,
+          0.03932824730873108,
+          -0.013478945940732956,
+          0.03526085987687111,
+          0.002372003858909011,
+          -0.003767300397157669,
+          0.020948560908436775,
+          0.023884417489171028,
+          -0.029893748462200165,
+          0.009579760953783989,
+          -0.0045643397606909275,
+          -0.013402490876615047,
+          0.015436183661222458,
+          0.007798369042575359,
+          0.02331865206360817,
+          -0.01479396503418684,
+          0.00012184951629023999,
+          -0.0013341327430680394,
+          0.03819671645760536,
+          0.008509396575391293,
+          -0.02350214309990406,
+          0.03263082355260849,
+          -0.012729690410196781,
+          0.00769133260473609,
+          -0.015046264976263046,
+          0.027049636468291283,
+          -0.011292343959212303,
+          -0.004055916331708431,
+          0.008348842151463032,
+          0.01568848453462124,
+          0.01784450374543667,
+          -0.015076846815645695,
+          -0.008318260312080383,
+          -0.03905300796031952,
+          0.01582610234618187,
+          0.012225090526044369,
+          -0.0003877681738231331,
+          0.02448076196014881,
+          -0.0072173140943050385,
+          -0.007851887494325638,
+          -0.0036832003388553858,
+          -0.007324350532144308,
+          -0.012653236277401447,
+          -0.003279902273789048,
+          -0.03510795161128044,
+          -0.03305896744132042,
+          0.011429962702095509,
+          -0.021697815507650375,
+          -0.01587197557091713,
+          0.011651680804789066,
+          -0.02672852762043476,
+          0.02214125171303749,
+          -0.007733382750302553,
+          0.03730984404683113,
+          -0.02289050631225109,
+          0.012683818116784096,
+          0.010374888777732849,
+          -0.004059739410877228,
+          -0.001281570177525282,
+          -0.012706754729151726,
+          -0.004006220959126949,
+          0.0061660632491111755,
+          0.01044369861483574,
+          0.02831878326833248,
+          0.03211093321442604,
+          -0.03458806127309799,
+          0.026346255093812943,
+          -0.007913051173090935,
+          0.010543089359998703,
+          0.00279632699675858,
+          0.001090433681383729,
+          0.03128522261977196,
+          -0.03379293158650398,
+          0.009166906587779522,
+          0.015497347339987755,
+          -0.0015682749217376113,
+          0.022584687918424606,
+          0.008043023757636547,
+          0.0003877681738231331,
+          -0.013853573240339756,
+          0.005068940110504627,
+          0.021269669756293297,
+          -0.0066591957584023476,
+          0.01503097452223301,
+          -0.01439640112221241,
+          0.026881437748670578,
+          0.03507737070322037,
+          -0.009296879172325134,
+          0.02266114391386509,
+          0.008486460894346237,
+          -0.0002391595480730757,
+          0.013249581679701805,
+          -0.007297591306269169,
+          -0.0349244624376297,
+          -0.001130572403781116,
+          -0.02715667337179184,
+          -0.010267852805554867,
+          -0.0009332239278592169,
+          -0.00590994069352746,
+          -0.023578599095344543,
+          -0.0033926728647202253,
+          0.016820011660456657,
+          -0.0007244073203764856,
+          -0.0059787495993077755,
+          -0.011536998674273491,
+          0.020076977089047432,
+          0.015749648213386536,
+          0.025061817839741707,
+          0.020245177671313286,
+          0.0020413377787917852,
+          0.014725156128406525,
+          0.0014421248342841864,
+          0.02438901737332344,
+          -0.006116367876529694,
+          -0.0017288295784965158,
+          -0.009059869684278965,
+          0.00859349686652422,
+          0.010711289010941982,
+          0.008501751348376274,
+          0.011582871899008751,
+          -0.021514324471354485,
+          -0.010902426205575466,
+          -0.012423872016370296,
+          0.004472594242542982,
+          -0.00540534034371376,
+          0.02036750502884388,
+          0.03529144451022148,
+          0.0011860019294545054,
+          -0.005305949132889509,
+          -0.016131920740008354,
+          -0.004671376198530197,
+          0.009747961536049843,
+          -0.032508496195077896,
+          0.0031919796019792557,
+          -0.015933139249682426,
+          0.0009059869917109609,
+          -0.020994432270526886,
+          0.0007736249826848507,
+          0.009579760953783989,
+          0.0030103998724371195,
+          0.016361285001039505,
+          -0.020443959161639214,
+          0.022447070106863976,
+          -0.0005710202967748046,
+          -0.008203578181564808,
+          0.013593628071248531,
+          -0.02093326859176159,
+          0.009304524399340153,
+          0.022737598046660423,
+          0.024985363706946373,
+          -0.016590647399425507,
+          -0.0008175863767974079,
+          0.00418971199542284,
+          0.006326618138700724,
+          0.00769133260473609,
+          0.0038265527691692114,
+          0.013310745358467102,
+          -0.016590647399425507,
+          0.013196064159274101,
+          0.013540109619498253,
+          0.004690489731729031,
+          5.689297177013941e-05,
+          -0.004786057863384485,
+          0.01854788511991501,
+          0.03776857256889343,
+          -0.02827291004359722,
+          -0.017095249146223068,
+          -0.0012796588707715273,
+          -0.009587407112121582,
+          0.000816152838524431,
+          0.015856683254241943,
+          -0.029847875237464905,
+          0.00437702564522624,
+          0.003228295361623168,
+          0.004445835016667843,
+          -0.021988343447446823,
+          0.017339903861284256,
+          0.008371778763830662,
+          -0.008020087145268917,
+          0.03137696906924248,
+          0.023288071155548096,
+          -0.0016514193266630173,
+          0.04088791832327843,
+          0.002578431274741888,
+          0.007087341509759426,
+          -0.026942601427435875,
+          0.031346384435892105,
+          -0.010971235111355782,
+          0.005864067934453487,
+          0.005367112811654806,
+          -0.02631567232310772,
+          -0.007068227510899305,
+          -0.010237270966172218,
+          0.004078852944076061,
+          -0.007595764473080635,
+          0.011842817068099976,
+          0.02552054449915886,
+          0.024465471506118774,
+          0.002127349143847823,
+          -0.019358305260539055,
+          0.0224776528775692,
+          0.018043285235762596,
+          0.013272518292069435,
+          -0.0011640213197097182,
+          -0.004350266885012388,
+          -0.017401067540049553,
+          -0.0032512317411601543,
+          -0.020719196647405624,
+          -0.01680472120642662,
+          -0.012278608977794647,
+          0.02279876172542572,
+          -0.010642480105161667,
+          -0.0004214559739921242,
+          -0.005183621775358915,
+          -0.02579578198492527,
+          -0.004629326052963734,
+          0.007660750765353441,
+          -0.00488162599503994,
+          0.010451343841850758,
+          0.026621490716934204,
+          0.010359598323702812,
+          -0.03269198536872864,
+          0.0068962047807872295,
+          -0.011017107404768467,
+          -0.007351109758019447,
+          0.0008854398038238287,
+          0.0025593177415430546,
+          0.01923597790300846,
+          -0.014663992449641228,
+          -0.022538816556334496,
+          0.020872104912996292,
+          0.005955813452601433,
+          -0.016590647399425507,
+          0.016376575455069542,
+          0.020765069872140884,
+          -0.008845796808600426,
+          0.010986525565385818,
+          -0.003868602681905031,
+          0.0204592514783144,
+          0.02761540189385414,
+          -0.024710126221179962,
+          9.509040683042258e-05,
+          -0.00469431234523654,
+          -0.005095698870718479,
+          0.0060016862116754055,
+          0.010734225623309612,
+          -0.008073605597019196,
+          -0.027447201311588287,
+          -0.0002871826000045985,
+          0.006991772912442684,
+          0.02561229094862938,
+          0.015275629237294197,
+          -0.03229442238807678,
+          -0.008731115609407425,
+          -0.016743557527661324,
+          0.03590308129787445,
+          -0.0404597744345665,
+          0.0001355874555883929,
+          0.0043311528861522675,
+          -0.010038488544523716,
+          -0.018471430987119675,
+          0.012316836044192314,
+          0.026483872905373573,
+          -0.008501751348376274,
+          -0.006555981934070587,
+          0.003174777142703533,
+          -0.014855128712952137,
+          0.027233127504587173,
+          0.033640023320913315,
+          -0.018471430987119675,
+          0.01775275729596615,
+          0.0037118708714842796,
+          0.006953545846045017,
+          -0.003455747850239277,
+          -0.040796175599098206,
+          0.011483481153845787,
+          0.028991583734750748,
+          0.0013828725786879659,
+          -0.004919853527098894,
+          0.00777160981670022,
+          0.009105742909014225,
+          -0.0037367185577750206,
+          0.010023198090493679,
+          -0.015933139249682426,
+          -0.0035666071344166994,
+          -0.003482507076114416,
+          0.01177400816231966,
+          -0.008417651057243347,
+          0.005936699453741312,
+          -0.008272388018667698,
+          -0.003377381945028901,
+          0.007202023174613714,
+          -0.00863937009125948,
+          -0.00014406914124265313,
+          -0.0025593177415430546,
+          0.0030428930185735226,
+          0.013563046231865883,
+          -0.009725024923682213,
+          -0.011644035577774048,
+          -0.03529144451022148,
+          0.016086047515273094,
+          0.028028255328536034,
+          0.0014908646699041128,
+          -0.030168984085321426,
+          0.006716536357998848,
+          -0.014128809794783592,
+          -0.01018375251442194,
+          -0.0060628498904407024,
+          0.020902687683701515,
+          0.03208034858107567,
+          0.020291050896048546,
+          0.01782921329140663,
+          0.007890114560723305,
+          0.019251268357038498,
+          -0.0029951089527457952,
+          0.008226514793932438,
+          -0.0038552230689674616,
+          -0.018716085702180862,
+          0.0015740090748295188,
+          0.017523394897580147,
+          -0.010351953096687794,
+          0.007056759670376778,
+          0.000946125655900687,
+          0.012561490759253502,
+          0.002589899580925703,
+          -0.0024064085446298122,
+          0.005053649190813303,
+          0.015489702112972736,
+          -0.0021827789023518562,
+          0.008853442966938019,
+          0.07370223104953766,
+          -0.01219450868666172,
+          0.010833616368472576,
+          0.008685242384672165,
+          -0.00578379025682807,
+          -0.017309321090579033,
+          -0.005535312928259373,
+          -0.022492943331599236,
+          0.027967091649770737,
+          0.00353984790854156,
+          -0.02116263285279274,
+          -0.004514643922448158,
+          0.00234715617261827,
+          0.016957629472017288,
+          0.015657901763916016,
+          -0.0027905928436666727,
+          -0.0026701767928898335,
+          -0.0023490674793720245,
+          0.017217576503753662,
+          0.0076263463124632835,
+          -0.025597000494599342,
+          -0.00450699869543314,
+          -0.0018196194432675838,
+          0.012324481271207333,
+          0.002180867362767458,
+          -0.004686667118221521,
+          0.0016428182134404778,
+          0.024603089317679405,
+          -0.005455035716295242,
+          0.029572639614343643,
+          -0.014243491925299168,
+          -0.017538685351610184,
+          0.014381109736859798,
+          -0.0012557668378576636,
+          0.021758979186415672,
+          0.00488162599503994,
+          0.018287939950823784,
+          -0.013654791750013828,
+          -0.005581185687333345,
+          0.01322664599865675,
+          -0.012622654438018799,
+          -0.0026625313330441713,
+          0.007935987785458565,
+          -0.00046756764641031623,
+          -0.002867047442123294,
+          0.006269277073442936,
+          -0.014426982961595058,
+          0.005894649773836136,
+          -0.01573435589671135,
+          -0.004537580534815788,
+          -0.007806014735251665,
+          -0.008188287727534771,
+          0.01463341061025858,
+          -0.0013886066153645515,
+          -0.03403758630156517,
+          -0.0024733063764870167,
+          0.034618642181158066,
+          0.003358268178999424,
+          0.00620429078117013,
+          0.01957237720489502,
+          0.005760854110121727,
+          0.014503437094390392,
+          -0.021132051944732666,
+          -0.0002107279869960621,
+          -0.0041056121699512005,
+          0.005554426461458206,
+          -0.006995595991611481,
+          0.0109788803383708,
+          0.005462680943310261,
+          -0.00017082825070247054,
+          -0.0028402884490787983,
+          -0.006330440752208233,
+          0.0032416749745607376,
+          0.008165351115167141,
+          0.008058315142989159,
+          0.01900661364197731,
+          -0.02266114391386509,
+          -0.01081068068742752,
+          0.009725024923682213,
+          0.009839707054197788,
+          0.01315783616155386,
+          0.008585851639509201,
+          0.0199393592774868,
+          -0.003065829398110509,
+          -0.019832322373986244,
+          -0.006127836182713509,
+          -0.0028880725149065256,
+          0.032325003296136856,
+          0.018150322139263153,
+          -0.025015944615006447,
+          -0.008647015318274498,
+          0.0018100626766681671,
+          0.002362447092309594,
+          -0.004247053060680628,
+          0.008708178997039795,
+          0.007874824106693268,
+          -0.02018401399254799,
+          -0.0008080295519903302,
+          -0.007014709524810314,
+          -0.009319815784692764,
+          0.009648570790886879,
+          0.019205395132303238,
+          -0.02223299816250801,
+          0.01435052789747715,
+          0.006819750182330608,
+          -0.02561229094862938,
+          0.024755999445915222,
+          0.008440587669610977,
+          -0.0028039724566042423,
+          -0.006804459262639284,
+          0.006173708941787481,
+          -0.03608657047152519,
+          0.01811973936855793,
+          0.003975639119744301,
+          -0.0035666071344166994,
+          0.0003598144685383886,
+          0.0019954650197178125,
+          -0.001083743991330266,
+          -0.010329016484320164,
+          0.00010416939039714634,
+          0.018425557762384415,
+          -0.008394715376198292,
+          0.01079538930207491,
+          -0.0035818980541080236,
+          0.008111832663416862,
+          -0.0066477274522185326,
+          -0.006953545846045017,
+          -0.004285280127078295,
+          -0.03324246034026146,
+          -0.01542853843420744,
+          -0.012041599489748478,
+          -0.02561229094862938,
+          0.005500908475369215,
+          0.001043605268932879,
+          -0.013234291225671768,
+          0.005669108591973782,
+          0.0013236202066764235,
+          0.0018177081365138292,
+          -0.004545225761830807,
+          0.010894780047237873,
+          -0.0040826755575835705,
+          0.016865884885191917,
+          0.008884024806320667,
+          -0.013662436977028847,
+          0.020168723538517952,
+          0.0009609387489035726,
+          0.002477128989994526,
+          -0.012660881504416466,
+          0.0002587510389275849,
+          0.003929766360670328,
+          0.009923807345330715,
+          -0.01934301294386387,
+          -0.005229494534432888,
+          -0.00601697713136673,
+          -0.011942208744585514,
+          -0.0022707015741616488,
+          -0.027645982801914215,
+          -0.007905405946075916,
+          -0.010466634295880795,
+          -0.0145798921585083,
+          -0.013700664043426514,
+          0.0022879038006067276,
+          -0.0006871357327327132,
+          0.01821148581802845,
+          -0.015130365267395973,
+          0.011108852922916412,
+          0.01280614547431469,
+          -0.0007903494406491518,
+          0.005657640285789967,
+          -0.007611055392771959,
+          -0.013004926964640617,
+          0.01854788511991501,
+          -0.017079956829547882,
+          -0.008394715376198292,
+          0.0206886138767004,
+          0.012225090526044369,
+          0.002939679427072406,
+          -0.014434628188610077,
+          -0.004201180301606655,
+          0.029052747413516045,
+          0.007733382750302553,
+          0.010864198207855225,
+          0.0032856364268809557,
+          0.003398407017812133,
+          0.02756952866911888,
+          0.0013809611555188894,
+          -0.014144101180136204,
+          0.006231050007045269,
+          0.004686667118221521,
+          -0.01102475356310606,
+          -0.013142545707523823,
+          0.017156412824988365,
+          -0.006926786620169878,
+          -0.015887266024947166,
+          0.024098489433526993,
+          0.023700926452875137,
+          0.0112388264387846,
+          -0.011926917359232903,
+          -0.020902687683701515,
+          0.0003727161674760282,
+          0.01001555286347866,
+          0.011812235228717327,
+          -0.01179694477468729,
+          -0.004224116448312998,
+          0.02457250840961933,
+          0.00901399739086628,
+          -0.022355325520038605,
+          0.008723469451069832,
+          0.030780620872974396,
+          0.015604383312165737,
+          -0.04544461518526077,
+          -0.01694233901798725,
+          -0.0007138948421925306,
+          -0.019251268357038498,
+          -0.02036750502884388,
+          -0.023609180003404617,
+          -0.011942208744585514,
+          0.0009485148475505412,
+          0.005615590140223503,
+          -0.0007291857618838549,
+          -0.017951540648937225,
+          0.01556615624576807,
+          0.017951540648937225,
+          0.011697554029524326,
+          -0.007760141976177692,
+          -0.018456140533089638,
+          -0.012530908919870853,
+          -0.01830323040485382,
+          0.01078774407505989,
+          -0.018089158460497856,
+          -0.012890245765447617,
+          0.006712713744491339,
+          -0.006854154635220766,
+          0.013983545824885368,
+          0.01554321963340044,
+          0.03076533041894436,
+          0.016040174290537834,
+          -0.021376706659793854,
+          -0.019373595714569092,
+          -0.005841131322085857,
+          -0.01074187085032463,
+          -0.004170598462224007,
+          -0.0013045065570622683,
+          0.005370935425162315,
+          0.009579760953783989,
+          0.005191267468035221,
+          0.02082623355090618,
+          -0.009610342793166637,
+          0.02776831015944481,
+          -2.6236470148433e-05,
+          -0.007362578064203262,
+          0.004201180301606655,
+          -0.018930159509181976,
+          0.009266297332942486,
+          -0.008302969858050346,
+          0.01120824459940195,
+          0.012263317592442036,
+          0.007599587086588144,
+          0.036392390727996826,
+          -0.011491126380860806,
+          -0.0013226645532995462,
+          -0.028012964874505997,
+          -0.0013417782029137015,
+          -0.003987107425928116,
+          0.021804852411150932,
+          0.005997863132506609,
+          -0.035841915756464005,
+          -0.020566286519169807,
+          -0.012959054671227932,
+          0.019281849265098572,
+          0.0035742525942623615,
+          -0.0053327083587646484,
+          0.020520415157079697,
+          0.025719327852129936,
+          -0.004679021425545216,
+          -0.0018263092497363687,
+          -0.027462491765618324,
+          -0.009686797857284546,
+          0.03287547826766968,
+          -0.005088053643703461,
+          0.001989730866625905,
+          0.0010636745719239116,
+          -0.015115073882043362,
+          -0.017309321090579033,
+          -0.012263317592442036,
+          0.003694668412208557,
+          0.007190554868429899,
+          0.027095509693026543,
+          -0.004766944330185652,
+          0.022768178954720497,
+          -0.01538266520947218,
+          0.008677597157657146,
+          -0.026850854977965355,
+          -0.012538554146885872,
+          0.018976030871272087,
+          -0.0009489927324466407,
+          0.014060000889003277,
+          -0.018058575689792633,
+          -0.006747118197381496,
+          -0.006024622358381748,
+          0.007419918663799763,
+          -0.005340353585779667,
+          -0.0017030262388288975,
+          -0.007744851056486368,
+          0.01479396503418684,
+          0.0009509040974080563,
+          0.007565182633697987,
+          -0.02275288850069046,
+          0.008318260312080383,
+          0.00087206024909392,
+          0.00670506851747632,
+          0.01882312260568142,
+          -0.00034858519211411476,
+          0.03951173648238182,
+          0.0072287824004888535,
+          0.011644035577774048,
+          -0.012003372423350811,
+          0.004560516681522131,
+          -0.008440587669610977,
+          0.012561490759253502,
+          0.002387294778600335,
+          0.010168462060391903,
+          0.007534600794315338,
+          -0.0029205656610429287,
+          -0.008975770324468613,
+          -0.0013245759764686227,
+          0.0014430806040763855,
+          -0.007706623524427414,
+          0.005703513044863939,
+          0.005493262782692909,
+          0.013570691458880901,
+          -0.01181988138705492,
+          -0.012653236277401447,
+          0.019037194550037384,
+          -0.021040305495262146,
+          -0.006479527335613966,
+          -0.00015768760931678116,
+          -0.006253986153751612,
+          0.004101789090782404,
+          0.0073664006777107716,
+          0.013066090643405914,
+          -0.03608657047152519,
+          0.005493262782692909,
+          -0.0164836123585701,
+          0.006987950298935175,
+          0.0005366157274693251,
+          -0.027829473838210106,
+          -0.024266690015792847,
+          0.0002518223482184112,
+          0.009725024923682213,
+          -0.00527918990701437,
+          0.005615590140223503,
+          -0.017920957878232002,
+          -0.0040329801850020885,
+          -0.004950435366481543,
+          -0.019878195598721504,
+          0.008348842151463032,
+          0.027584819123148918,
+          -0.018242066726088524,
+          -0.010657771490514278,
+          0.008723469451069832,
+          -0.005240962840616703,
+          -0.01989348605275154,
+          -0.006322795525193214,
+          -0.012821435928344727,
+          0.0035341137554496527,
+          0.019862905144691467,
+          -0.0028441110625863075,
+          -0.0113535076379776,
+          0.017248157411813736,
+          0.0066859545186161995,
+          -0.015367374755442142,
+          0.001213716808706522,
+          -0.0012194508453831077,
+          -0.012714399956166744,
+          0.01519152894616127,
+          -0.024266690015792847,
+          0.01578022912144661,
+          -0.0065101091749966145,
+          -0.003471038769930601,
+          -0.031621623784303665,
+          -0.006506286561489105,
+          0.008624078705906868,
+          0.0025688745081424713,
+          0.01793624833226204,
+          0.012393290176987648,
+          -0.020336924120783806,
+          0.01441933773458004,
+          -0.0058029042556881905,
+          0.005286835599690676,
+          -0.006357199978083372,
+          0.042386431246995926,
+          -0.010290789417922497,
+          -0.006517754402011633,
+          -0.00033520563738420606,
+          0.0015338703524321318,
+          0.0033850274048745632,
+          0.02761540189385414,
+          -0.004487885162234306,
+          -0.004308216739445925,
+          -0.0032493204344063997,
+          0.004434366710484028,
+          0.005611767526715994,
+          -0.0006713669281452894,
+          0.015160947106778622,
+          -0.006032268051058054,
+          0.008891670033335686,
+          -0.0224164891988039,
+          -0.0030428930185735226,
+          0.0159637201577425,
+          0.0032130044419318438,
+          -0.0012060713488608599,
+          -0.007672219071537256,
+          -0.03177453204989433,
+          -0.000630272610578686,
+          -0.023073997348546982,
+          0.006269277073442936,
+          -0.010351953096687794,
+          0.0017431648448109627,
+          -0.01415939163416624,
+          0.01081068068742752,
+          0.012744981795549393,
+          0.003799793543294072,
+          0.01460282877087593,
+          -0.0219577606767416,
+          0.0159637201577425,
+          0.007408450823277235,
+          -0.005890826694667339,
+          0.011865753680467606,
+          -0.015176237560808659,
+          -0.009717379696667194,
+          0.0017173613887280226,
+          0.02761540189385414,
+          -0.0012834816006943583,
+          -0.006047558970749378,
+          -0.00348824099637568,
+          -0.013532464392483234,
+          -0.0005409162840805948,
+          0.0010703643783926964,
+          -0.010451343841850758,
+          -0.001181223546154797,
+          -0.006655372679233551,
+          0.009396269917488098,
+          0.010543089359998703,
+          0.000875882979016751,
+          0.006093431729823351,
+          -0.009931452572345734,
+          0.00037367184995673597,
+          0.015099783428013325,
+          0.038563698530197144,
+          -0.0012959054438397288,
+          0.011659326031804085,
+          -0.01952650398015976,
+          -0.004407607484608889,
+          -0.0021789560560137033,
+          -0.029878458008170128,
+          -0.010703643783926964,
+          0.00819593295454979,
+          0.014105873182415962,
+          0.006506286561489105,
+          -0.015764938667416573,
+          0.005003953352570534,
+          0.01778334006667137,
+          0.0095950523391366,
+          0.0047898804768919945,
+          0.0071790870279073715,
+          0.003386938711628318,
+          0.006234872620552778,
+          0.00641454104334116,
+          -0.025306472554802895,
+          -0.011154726147651672,
+          -0.010084361769258976,
+          -0.021177923306822777,
+          0.013769472949206829,
+          0.01718699373304844,
+          0.01573435589671135,
+          0.013524818234145641,
+          -0.010382534936070442,
+          0.03299780562520027,
+          0.023853834718465805,
+          -0.00400239834561944,
+          0.003767300397157669,
+          0.004025334492325783,
+          0.005229494534432888,
+          0.013326036743819714,
+          -0.0028517565224319696,
+          -0.00418971199542284,
+          -0.00036602639011107385,
+          0.013165482319891453,
+          -0.0014535930240526795,
+          -0.004197357688099146,
+          -0.010199043899774551,
+          0.0032493204344063997,
+          0.0005714981234632432,
+          -0.004698134958744049,
+          -0.00477458955720067,
+          -0.006383959203958511,
+          -0.0007339641451835632,
+          -0.00025230017490684986,
+          0.013570691458880901,
+          -0.005523844622075558,
+          -0.023119870573282242,
+          -0.002480951836332679,
+          -0.0017555886879563332,
+          -0.003763477550819516,
+          -0.005088053643703461,
+          -0.005944345146417618,
+          -0.0006622879882343113,
+          0.019908778369426727,
+          0.03217209503054619,
+          0.01966412365436554,
+          0.01675884798169136,
+          0.016070757061243057,
+          0.008249451406300068,
+          -0.011070625856518745,
+          0.010803034529089928,
+          0.002651063259691,
+          -0.0030543613247573376,
+          0.005030712578445673,
+          0.008669951930642128,
+          0.006024622358381748,
+          0.009740316309034824,
+          -0.01782921329140663,
+          0.005485617555677891,
+          -0.017355194315314293,
+          0.00641454104334116,
+          -0.02481716312468052,
+          -0.007851887494325638,
+          -0.006533045321702957,
+          -0.010887134820222855,
+          -0.008914606645703316,
+          0.004407607484608889,
+          0.024327853694558144,
+          -0.006655372679233551,
+          0.01479396503418684,
+          -0.02243177965283394,
+          0.013509527780115604,
+          0.007312882225960493,
+          -0.006399250123649836,
+          -0.014694574289023876,
+          -0.006540691014379263,
+          -0.004847221542149782,
+          -0.008692887611687183,
+          0.012202153913676739,
+          -0.0036640865728259087,
+          -0.016498902812600136,
+          -4.0691170397622045e-06,
+          -0.012163926847279072,
+          -0.013249581679701805,
+          -0.005795258563011885,
+          0.020153433084487915,
+          0.013165482319891453,
+          0.005951990373432636,
+          -0.020627450197935104,
+          0.004174421075731516,
+          -0.0014937317464500666,
+          0.02827291004359722,
+          -0.004950435366481543,
+          -0.0038017048500478268,
+          0.0022343855816870928,
+          0.01402177382260561,
+          -0.007882469333708286,
+          -0.0025765199679881334,
+          0.00270267017185688,
+          0.02130025066435337,
+          0.01016081590205431,
+          0.004438189323991537,
+          -0.007966569624841213,
+          0.021514324471354485,
+          -0.012852017767727375,
+          -0.00269311317242682,
+          -0.003715693484991789,
+          -0.005867890547960997,
+          -0.013647145591676235,
+          0.015573801472783089,
+          0.010145525448024273,
+          0.0027657451573759317,
+          -0.005558249540627003,
+          0.001599812530912459,
+          -0.009327461011707783,
+          -0.010688353329896927,
+          -0.018379686400294304,
+          0.008899315260350704,
+          -0.024358434602618217,
+          0.006598032079637051,
+          0.003727161791175604,
+          0.00022745243040844798,
+          -0.00588700408115983,
+          -0.003430900163948536,
+          0.0027389859315007925,
+          0.01116237137466669,
+          -0.005879358854144812,
+          -0.017003502696752548,
+          -0.002113969763740897,
+          0.02940443903207779,
+          -0.011911626905202866,
+          -0.0015376930823549628,
+          -0.003889627754688263,
+          -0.0037443640176206827,
+          0.002712226938456297,
+          -0.01591784693300724,
+          -0.0048242853954434395,
+          0.0013618475059047341,
+          -6.134287104941905e-05,
+          0.0076034097000956535,
+          0.01675884798169136,
+          -0.007056759670376778,
+          0.016743557527661324,
+          -0.009709734469652176,
+          -0.0015262248925864697,
+          0.017920957878232002,
+          -0.0009126767981797457,
+          -0.01789037697017193,
+          0.008142414502799511,
+          -0.007905405946075916,
+          0.030367767438292503,
+          -0.009189843200147152,
+          -0.027462491765618324,
+          0.025811072438955307,
+          0.02986316569149494,
+          -0.010107298381626606,
+          0.01857846789062023,
+          -0.006590386386960745,
+          0.017997412011027336,
+          0.028915129601955414,
+          -0.0016552420565858483,
+          -0.014319946058094501,
+          0.004927498754113913,
+          -0.007943633012473583,
+          -0.014702219516038895,
+          -0.0034481023903936148,
+          0.01643773913383484,
+          -0.026376836001873016,
+          0.0071446821093559265,
+          -0.025581708177924156,
+          0.0191901046782732,
+          -0.016728267073631287,
+          -0.025994563475251198,
+          -0.008371778763830662,
+          0.012576781213283539,
+          0.007305236998945475,
+          0.029985493049025536,
+          -0.0035455820616334677,
+          -0.034098751842975616,
+          -0.0011793122394010425,
+          0.015329146757721901,
+          -0.008685242384672165,
+          -0.01095594372600317,
+          0.03165220469236374,
+          0.01081832591444254,
+          -0.032325003296136856,
+          0.008188287727534771,
+          -0.026239218190312386,
+          0.000751166429836303,
+          -0.01943475939333439,
+          0.012248027138411999,
+          -0.03984813764691353,
+          -0.0024025856982916594,
+          -0.0013006838271394372,
+          -0.0009853086667135358,
+          -0.004078852944076061,
+          0.005267722066491842,
+          0.0034328114707022905,
+          0.014495791867375374,
+          -0.017676303163170815,
+          -0.027401328086853027,
+          0.011246471665799618,
+          0.008700533770024776,
+          -0.0232269074767828,
+          -0.01737048476934433,
+          0.0035474933683872223,
+          -0.019602959975600243,
+          0.026835564523935318,
+          -0.007037645671516657,
+          -0.001758455764502287,
+          -0.018379686400294304,
+          -0.0043540894985198975,
+          -0.0019314342644065619,
+          -0.01564261130988598,
+          0.014304655604064465,
+          0.008287678472697735,
+          0.01179694477468729,
+          -0.0037214276380836964,
+          -0.01341013703495264,
+          -9.332837180409115e-06,
+          0.011911626905202866,
+          0.011429962702095509,
+          -0.005504731088876724,
+          0.009610342793166637,
+          -0.007943633012473583,
+          -0.0021464629098773003,
+          -0.015390310436487198,
+          0.013777119107544422,
+          -0.0026204814203083515,
+          -0.008914606645703316,
+          -0.0019381240708753467,
+          0.012110408395528793,
+          0.008127124048769474,
+          0.003438545623794198,
+          -0.01018375251442194,
+          0.005302126519382,
+          -0.006835041102021933,
+          -0.01540560182183981,
+          -0.01156758051365614,
+          -0.019083067774772644,
+          -0.007786900736391544,
+          -0.020306341350078583,
+          0.01119295321404934,
+          0.032844897359609604,
+          0.005347999278455973,
+          -0.009228070266544819,
+          -0.025107691064476967,
+          -0.027355454862117767,
+          0.017951540648937225,
+          0.0026529745664447546,
+          -0.002366269938647747,
+          -0.02354801632463932,
+          -0.019159521907567978,
+          0.015596738085150719,
+          0.03529144451022148,
+          -0.0109788803383708,
+          -0.016177793964743614,
+          0.004090321250259876,
+          -0.013027863577008247,
+          -0.014281718991696835,
+          -0.00896812416613102,
+          -0.013035508804023266,
+          -0.0005246697110123932,
+          0.0010464723454788327,
+          -0.007572827860713005,
+          0.028731638565659523,
+          0.009877934120595455,
+          0.009342752397060394,
+          0.011200598441064358,
+          0.0031652203761041164,
+          0.005604122299700975,
+          -0.007278477773070335,
+          -0.0009117210865952075,
+          -0.007033823058009148,
+          0.023884417489171028,
+          -0.003960348200052977,
+          -0.00041811109986156225,
+          0.001845422899350524,
+          -0.004705780651420355,
+          0.012446808628737926,
+          0.00029530588653869927,
+          -0.0329672247171402,
+          0.03501620516180992,
+          -0.006395427044481039,
+          0.0016045909142121673,
+          -0.02597927302122116,
+          -0.011605808511376381,
+          0.021177923306822777,
+          0.006280745379626751,
+          0.001812929636798799,
+          -0.006762409117072821,
+          0.0037290730979293585,
+          0.009121033363044262,
+          0.020658032968640327,
+          -0.01079538930207491,
+          0.002070008311420679,
+          0.015252692624926567,
+          0.012270962819457054,
+          0.00031131357536651194,
+          0.0043540894985198975,
+          0.0057417405769228935,
+          -0.005328885745257139,
+          0.007542246021330357,
+          -0.022492943331599236,
+          -0.01142231747508049,
+          -0.011582871899008751,
+          0.009472724981606007,
+          -0.001574964728206396,
+          0.006112545263022184,
+          0.004935144446790218,
+          0.0034595706965774298,
+          -0.010535444132983685,
+          0.008899315260350704,
+          -0.020795650780200958,
+          -0.004147661849856377,
+          0.00032182608265429735,
+          0.027875347062945366,
+          -0.013004926964640617,
+          0.005856422241777182,
+          -0.009656216017901897,
+          0.009342752397060394,
+          0.020917978137731552,
+          -0.0006183265941217542,
+          0.0026778222527354956,
+          0.0006971703842282295,
+          -0.004254698287695646,
+          0.005615590140223503,
+          -0.003937412053346634,
+          -0.017263447865843773,
+          0.017477521672844887,
+          0.00015051999071147293,
+          0.004549048840999603,
+          -0.002289815340191126,
+          0.0010311814257875085,
+          0.01802799478173256,
+          0.020352214574813843,
+          -0.00835648737847805,
+          -0.002268790267407894,
+          -0.010688353329896927,
+          0.03819671645760536,
+          -0.004491707775741816,
+          0.01793624833226204,
+          0.004938967060297728,
+          -0.004533757921308279,
+          0.0009389580227434635,
+          -0.005340353585779667,
+          0.012936118058860302,
+          -0.00842529721558094,
+          -0.015634965151548386,
+          -0.010703643783926964,
+          -0.008234160020947456,
+          -0.013517173007130623,
+          0.019786451011896133,
+          -0.023853834718465805,
+          0.016957629472017288,
+          0.0017708796076476574,
+          0.009923807345330715,
+          0.010351953096687794,
+          -0.00010811158426804468,
+          0.0049848398193717,
+          0.0003978028253186494,
+          -0.018410267308354378,
+          0.026529746130108833,
+          -0.016407156363129616,
+          0.01877724938094616,
+          -0.01175107154995203,
+          -0.0026032791938632727,
+          0.013585981912910938,
+          0.006995595991611481,
+          0.026330964639782906,
+          -0.004461125936359167,
+          -0.0050115990452468395,
+          0.015657901763916016,
+          0.008494106121361256,
+          0.029694966971874237,
+          0.015795519575476646,
+          0.0059787495993077755,
+          -0.0009169773547910154,
+          0.013257227838039398,
+          -0.01666710339486599,
+          -0.015573801472783089,
+          0.008104187436401844,
+          -0.0007229737821035087,
+          -0.004545225761830807,
+          0.0022305629681795835,
+          0.0349244624376297,
+          -0.0010416939621791244,
+          0.011942208744585514,
+          0.009258652105927467,
+          -0.019205395132303238,
+          0.04177479445934296,
+          0.009977324865758419,
+          0.014969810843467712,
+          -0.0199393592774868,
+          -0.04474123194813728,
+          -0.015466765500605106,
+          -0.008876378647983074,
+          0.005795258563011885,
+          0.0005676753935404122,
+          0.0032665226608514786,
+          -0.028716348111629486,
+          -0.019067777320742607,
+          0.006857977714389563,
+          -0.01675884798169136,
+          0.015887266024947166,
+          -0.0061813541688025,
+          0.0047287167981266975,
+          0.005799081176519394,
+          0.0017326523084193468,
+          -0.0065215774811804295,
+          -0.007569005247205496,
+          0.017079956829547882,
+          0.0033468001056462526,
+          -0.006146949715912342,
+          0.021040305495262146,
+          -0.01058131642639637,
+          0.005669108591973782,
+          -0.0009160216432064772,
+          0.007358754985034466,
+          0.016453029587864876,
+          0.01301257312297821,
+          0.0028001496102660894,
+          0.004327330272644758,
+          0.014381109736859798,
+          0.02992432937026024,
+          0.0014918203232809901,
+          -0.010244916193187237,
+          -0.003870513988658786,
+          0.016223665326833725,
+          0.001654286403208971,
+          -0.019373595714569092,
+          0.014686928130686283,
+          -0.027049636468291283,
+          -0.003406052477657795,
+          -0.01811973936855793,
+          0.002033692318946123,
+          -0.0018301319796591997,
+          -0.007630168925970793,
+          2.1741776436101645e-05,
+          0.007905405946075916,
+          -0.000541871995665133,
+          0.012959054671227932,
+          -0.003409875091165304,
+          0.009916161186993122,
+          0.021651942282915115,
+          -0.008478814736008644,
+          -0.012974345125257969,
+          0.017966831102967262,
+          0.010848907753825188,
+          -0.015038619749248028,
+          0.0008300102199427783,
+          -0.000931790447793901,
+          -0.03137696906924248,
+          -0.0037462753243744373,
+          0.012706754729151726,
+          -0.0012557668378576636,
+          0.00488162599503994,
+          -0.0092204250395298,
+          -0.018058575689792633,
+          0.024220816791057587,
+          0.008784633129835129,
+          0.01456460077315569,
+          0.009044579230248928,
+          -0.010244916193187237,
+          0.0001916143373819068,
+          0.015810811892151833,
+          -0.009885579347610474,
+          -0.0052600763738155365,
+          0.01498510129749775,
+          -0.022676434367895126,
+          0.0113535076379776,
+          -0.008769342675805092,
+          0.0023242197930812836,
+          -0.002817352069541812,
+          -0.01121588982641697,
+          -0.008868733420968056,
+          -0.010925361886620522,
+          0.013333681970834732,
+          -0.023914998397231102,
+          -0.0005160685395821929,
+          0.008081250824034214,
+          0.0029205656610429287,
+          -0.035138532519340515,
+          -0.0033391546458005905,
+          0.014946874231100082,
+          0.029572639614343643,
+          -0.01782921329140663,
+          -0.01102475356310606,
+          0.01582610234618187,
+          -0.007859532721340656,
+          -0.000128658750327304,
+          0.0019199660746380687,
+          -0.006915318313986063,
+          -0.02223299816250801,
+          -0.010566025972366333,
+          -0.021651942282915115,
+          0.007867177948355675,
+          0.003247409127652645,
+          -0.008700533770024776,
+          -0.007492550648748875,
+          -0.014319946058094501,
+          -0.011689907871186733,
+          -0.024129072204232216,
+          -0.0031919796019792557,
+          -0.0010770541848614812,
+          -0.017095249146223068,
+          0.02682027406990528,
+          -0.004036802798509598,
+          0.014908647164702415,
+          -0.020336924120783806,
+          0.04064326360821724,
+          -0.015764938667416573,
+          -0.0016514193266630173,
+          -0.010107298381626606,
+          0.01278320886194706,
+          0.0053441766649484634,
+          -2.7714791940525174e-05,
+          -0.010321371257305145,
+          -0.015979010611772537,
+          0.015451474115252495,
+          0.008631723932921886,
+          0.0029090975876897573,
+          -0.008624078705906868,
+          -0.020168723538517952,
+          -0.009977324865758419,
+          -0.009273942559957504,
+          -0.031193476170301437,
+          -0.0016017238376662135,
+          -0.005787613335996866,
+          0.014648701064288616,
+          -0.008960478939116001,
+          -0.01652948372066021,
+          0.01240858156234026,
+          0.021468451246619225,
+          0.009029287844896317,
+          -0.01639186590909958,
+          0.01118530798703432,
+          -0.010864198207855225,
+          0.004808994475752115,
+          0.017248157411813736,
+          -0.004250875674188137,
+          0.009342752397060394,
+          -0.0038380208425223827,
+          -0.0023127517197281122,
+          -0.004464948549866676,
+          -0.007882469333708286,
+          -0.01217157207429409,
+          -0.007798369042575359,
+          -0.029832584783434868,
+          0.007974214851856232,
+          -0.008792279288172722,
+          0.03351769596338272,
+          0.01240858156234026,
+          0.024496054276823997,
+          -0.01119295321404934,
+          0.023899707943201065,
+          -0.0164836123585701,
+          0.023838544264435768,
+          0.005974926985800266,
+          -0.02570403553545475,
+          0.016835302114486694,
+          -0.010642480105161667,
+          -0.03501620516180992,
+          0.0005934788496233523,
+          -0.011766362935304642,
+          0.03458806127309799,
+          -0.0359642431139946,
+          0.011414671316742897,
+          0.0043540894985198975,
+          -0.020443959161639214,
+          -0.004323507659137249,
+          0.006028444971889257,
+          0.00038275084807537496,
+          0.02139199711382389,
+          -0.009640924632549286,
+          0.004961903672665358,
+          0.014327592216432095,
+          0.022156542167067528,
+          -0.012278608977794647,
+          0.008302969858050346,
+          0.023838544264435768,
+          0.021514324471354485,
+          0.010856552980840206,
+          -0.01680472120642662,
+          0.02883867546916008,
+          -0.0037080480251461267,
+          0.0008797057089395821,
+          0.004235584754496813,
+          -0.02706492878496647,
+          -0.006311327219009399,
+          -0.002654885873198509,
+          0.011621098965406418,
+          0.008845796808600426,
+          0.0015701863449066877,
+          0.013249581679701805,
+          0.01652948372066021,
+          -0.01315783616155386,
+          0.004300571046769619,
+          0.005042180884629488,
+          0.006104899570345879,
+          0.01764572225511074,
+          -0.030750039964914322,
+          0.006028444971889257,
+          -0.021361414343118668,
+          -0.005156862549483776,
+          0.0071561504155397415,
+          0.011957499198615551,
+          0.015856683254241943,
+          -0.03694286197423935,
+          0.0011821791995316744,
+          -0.003293281886726618,
+          0.013096672482788563,
+          0.020841524004936218,
+          0.01479396503418684,
+          -0.01296669989824295,
+          0.008555269800126553,
+          -0.02186601608991623,
+          -0.00314228399656713,
+          -0.020963851362466812,
+          0.002131171990185976,
+          -0.019358305260539055,
+          0.006861800327897072,
+          0.006865622941404581,
+          -0.00559647660702467,
+          -0.0013341327430680394,
+          0.01685059443116188,
+          -0.022921089082956314,
+          -0.008302969858050346,
+          0.012377999722957611,
+          -0.0036507071927189827,
+          -0.004495530389249325,
+          -0.008624078705906868,
+          0.0012376088416203856,
+          0.012997281737625599,
+          0.0069688367657363415,
+          -0.006735650356858969,
+          0.010244916193187237,
+          0.0014268339145928621,
+          0.0025057995226234198,
+          0.006318972446024418,
+          0.004063562024384737,
+          -0.01652948372066021,
+          -0.0014956430532038212,
+          -5.632553802570328e-05,
+          0.012240380980074406,
+          0.008104187436401844,
+          -0.017309321090579033,
+          -0.02111675962805748,
+          0.019281849265098572,
+          -0.028578728437423706,
+          -0.003803616389632225,
+          -0.0008457790245302022,
+          -0.0021445516031235456,
+          0.014518728479743004,
+          0.012821435928344727,
+          -0.01058131642639637,
+          -0.0017498546512797475,
+          -0.01825735904276371,
+          0.0037080480251461267,
+          0.0009833972435444593,
+          -0.010650125332176685,
+          0.004701958037912846,
+          0.010527797974646091,
+          0.011888690292835236,
+          0.0025516722816973925,
+          -0.018242066726088524,
+          0.00994674302637577,
+          0.02152961492538452,
+          0.014411691576242447,
+          -0.004629326052963734,
+          -0.014725156128406525,
+          -0.03614773601293564,
+          -0.023012833669781685,
+          0.026193344965577126,
+          -0.009839707054197788,
+          -0.02157548815011978,
+          0.017523394897580147,
+          0.011644035577774048,
+          0.022034214809536934,
+          -0.009380979463458061,
+          -0.006796814035624266,
+          -0.0019228331511840224,
+          0.017615139484405518,
+          0.004403784871101379,
+          -0.0027389859315007925,
+          0.0017890376038849354,
+          -0.0025153562892228365,
+          0.04116315767168999,
+          0.0021005901508033276,
+          0.005336530972272158,
+          -0.0187313761562109,
+          0.008257096633315086,
+          -0.02477128989994526,
+          0.009442143142223358,
+          -0.011368799023330212,
+          0.029419729486107826,
+          0.019465340301394463,
+          0.0071676187217235565,
+          0.015657901763916016,
+          -0.00901399739086628,
+          0.03669820725917816,
+          -0.014847483485937119,
+          -0.013310745358467102,
+          -0.0024140540044754744,
+          -0.018761958926916122,
+          -0.009029287844896317,
+          -0.0037940593902021646,
+          -0.005546781234443188,
+          0.006953545846045017,
+          -0.0046255034394562244,
+          0.006563627161085606,
+          0.002496242756024003,
+          -0.017584558576345444,
+          0.011284698732197285,
+          -0.022263579070568085,
+          0.01779863052070141,
+          -0.01243151817470789,
+          -0.021269669756293297,
+          0.0014669726369902492,
+          0.01830323040485382,
+          -0.017966831102967262,
+          0.00038824600051157176,
+          -0.0021292606834322214,
+          0.02552054449915886,
+          0.008547624573111534,
+          -0.005642349366098642,
+          0.008272388018667698,
+          -0.004067384637892246,
+          -0.006173708941787481,
+          0.01905248686671257,
+          0.028945710510015488,
+          0.019419468939304352,
+          -0.007282300386577845,
+          0.014992746524512768,
+          -0.004350266885012388,
+          -0.02977142110466957,
+          0.012202153913676739,
+          -0.010803034529089928,
+          0.006586563773453236,
+          0.007301414385437965,
+          -0.009556825272738934,
+          -0.015887266024947166,
+          0.008562915027141571,
+          -0.0149545194581151,
+          0.006387781817466021,
+          0.0014077202649787068,
+          0.0035264682956039906,
+          -0.003073474857956171,
+          0.001033092732541263,
+          -0.008792279288172722,
+          0.027493074536323547,
+          0.023379815742373466,
+          0.012951409444212914,
+          -0.023807961493730545,
+          0.005435922183096409,
+          -0.02448076196014881,
+          -0.0031250817701220512,
+          -0.00686944555491209,
+          -0.0036239479668438435,
+          -0.008432942442595959,
+          -0.0071561504155397415,
+          -0.007159973029047251,
+          -0.01732461154460907,
+          -0.0010178018128499389,
+          0.013777119107544422,
+          -0.001823442173190415,
+          0.006880913861095905,
+          0.0031441953033208847,
+          -0.003029513405635953,
+          -0.0006799680995754898,
+          0.029511475935578346,
+          0.00861643347889185,
+          -0.008792279288172722,
+          -0.012018662877380848,
+          -0.01078774407505989,
+          -0.013035508804023266,
+          0.027003765106201172,
+          0.006032268051058054,
+          -0.013754182495176792,
+          0.012225090526044369,
+          -0.0071790870279073715,
+          -0.021606069058179855,
+          -0.003543670754879713,
+          0.009709734469652176,
+          -0.03403758630156517,
+          0.034007005393505096,
+          -0.004113257396966219,
+          0.0057417405769228935,
+          0.0036602639593183994,
+          0.05174447223544121,
+          -0.005890826694667339,
+          -0.007068227510899305,
+          0.017095249146223068,
+          0.014595182612538338,
+          0.011911626905202866,
+          -0.007756318897008896,
+          0.00559265399351716,
+          -0.023395108059048653,
+          0.007171441335231066,
+          -0.01764572225511074,
+          -0.012003372423350811,
+          0.02822703868150711,
+          0.0010244916193187237,
+          -0.017171703279018402,
+          -0.01117001660168171,
+          -0.015466765500605106,
+          0.007882469333708286,
+          0.029740838333964348,
+          0.022538816556334496,
+          -0.018364394083619118,
+          -0.00609725434333086,
+          -0.01458753738552332,
+          -0.016972921788692474,
+          -0.010604253038764,
+          0.00385713460855186,
+          -0.012423872016370296,
+          -0.0027313404716551304,
+          -0.006158418022096157,
+          0.0006900027510710061,
+          -0.0016848682425916195,
+          -0.000514635001309216,
+          0.06348790228366852,
+          -0.007935987785458565,
+          0.006219581700861454,
+          -0.016070757061243057,
+          0.0007707579061388969,
+          -0.0034538365434855223,
+          -0.008150060661137104,
+          0.0007822260959073901,
+          -0.01759984903037548,
+          0.0287010557949543,
+          -0.024939490482211113,
+          0.0010627189185470343,
+          -0.014992746524512768,
+          0.006877091247588396,
+          -0.029465602710843086,
+          -0.0013752271188423038,
+          -0.01943475939333439,
+          0.004598744213581085,
+          -0.0011038132943212986,
+          -0.01016081590205431,
+          0.015313856303691864,
+          -0.0311781857162714,
+          -0.014113519340753555,
+          0.007572827860713005,
+          0.010436052456498146,
+          0.015084492042660713,
+          -0.020627450197935104,
+          -0.007278477773070335,
+          -0.01877724938094616,
+          -0.023670343682169914,
+          0.008792279288172722,
+          -0.0037500981707125902,
+          0.02481716312468052,
+          -0.01923597790300846,
+          0.0037711230106651783,
+          -0.016820011660456657,
+          0.011147080920636654,
+          0.01402177382260561,
+          0.004487885162234306,
+          -0.0029129202011972666,
+          -0.010428407229483128,
+          0.023838544264435768,
+          -0.00758429616689682,
+          -0.034618642181158066,
+          0.007094986736774445,
+          0.004606389440596104,
+          -0.00779454642906785,
+          -0.011598162353038788,
+          0.0113535076379776,
+          -0.0028613132890313864,
+          0.006292213685810566,
+          -0.02050512284040451,
+          -0.0011028576409444213,
+          -0.014312300831079483,
+          0.012454453855752945,
+          0.013754182495176792,
+          0.01357833668589592,
+          3.586795719456859e-05,
+          -0.010749517008662224,
+          -0.0264532919973135,
+          -0.0004510821308940649,
+          -0.004908385220915079,
+          0.0013647145824506879,
+          0.008700533770024776,
+          0.0031652203761041164,
+          0.022599980235099792,
+          -0.016024883836507797,
+          -0.019495923072099686,
+          0.0002515834057703614,
+          -0.01853259466588497,
+          -0.02804354764521122,
+          0.005290658213198185,
+          -0.004120903089642525,
+          -0.004178243689239025,
+          0.003673643572255969,
+          -0.011108852922916412,
+          -0.010420762002468109,
+          0.007282300386577845,
+          0.0024943312164396048,
+          -0.01639186590909958,
+          0.026835564523935318,
+          -0.0076378146186470985,
+          -0.014060000889003277,
+          -0.006322795525193214,
+          0.008364133536815643,
+          -0.009725024923682213,
+          0.0057302722707390785,
+          -0.03038305789232254,
+          0.0033276863396167755,
+          -0.007159973029047251,
+          -0.012874954380095005,
+          -0.0040100435726344585,
+          0.01591784693300724,
+          0.010260207578539848,
+          -0.007030000444501638,
+          -0.004143839236348867,
+          -0.010130234062671661,
+          -0.0053212400525808334,
+          0.007446677889674902,
+          0.03052067570388317,
+          0.001286348677240312,
+          -0.010458989068865776,
+          -0.014809255488216877,
+          0.017875084653496742,
+          -0.010038488544523716,
+          -0.0287010557949543,
+          0.014281718991696835,
+          0.014732801355421543,
+          -0.03394584357738495,
+          0.0059175859205424786,
+          -0.02073448710143566,
+          -0.011666972190141678,
+          -0.01948063261806965,
+          -0.009625634178519249,
+          0.02393028885126114,
+          -0.02134612388908863,
+          0.009877934120595455,
+          0.005068940110504627,
+          0.03324246034026146,
+          0.01639186590909958,
+          -0.01938888616859913,
+          0.00936568807810545,
+          -0.0037787684705108404,
+          0.006441299803555012,
+          0.008906960487365723,
+          -0.014251137152314186,
+          -0.0002635294513311237,
+          0.016269538551568985,
+          -0.0013704487355425954,
+          0.0018177081365138292,
+          -0.003918298054486513,
+          -0.03211093321442604,
+          0.012630299665033817,
+          -0.02837994694709778,
+          -0.0037787684705108404,
+          0.010573671199381351,
+          -0.01859375834465027,
+          -0.02617805451154709,
+          -0.0024580154567956924,
+          0.012339772656559944,
+          -0.029373856261372566,
+          0.05997098609805107,
+          0.0014268339145928621,
+          0.021147342398762703,
+          -0.01456460077315569,
+          -0.008539978414773941,
+          -0.0047287167981266975,
+          0.0018224865198135376,
+          -0.001679134089499712,
+          0.00024632716667838395,
+          0.014686928130686283,
+          -0.008998706005513668,
+          0.017951540648937225,
+          0.010046134702861309,
+          -0.001892251311801374,
+          0.02477128989994526,
+          0.00678916834294796,
+          0.00739698251709342,
+          0.005034535191953182,
+          0.021055595949292183,
+          -0.0023452448658645153,
+          0.016820011660456657,
+          0.009327461011707783,
+          -0.0109788803383708,
+          0.001719272811897099,
+          -0.008792279288172722,
+          0.014281718991696835,
+          0.011040044017136097,
+          0.002257321961224079,
+          -0.008211224339902401,
+          -0.0032550545874983072,
+          0.021407287567853928,
+          -0.010757162235677242,
+          -0.022034214809536934,
+          -0.010267852805554867,
+          0.0059175859205424786,
+          0.007718091830611229,
+          0.01718699373304844,
+          0.01718699373304844,
+          -0.007458146195858717,
+          -0.011888690292835236,
+          0.022492943331599236,
+          -0.012645591050386429,
+          0.014725156128406525,
+          0.0036602639593183994,
+          -0.017706885933876038,
+          0.02200363390147686,
+          -0.011040044017136097,
+          -0.008295323699712753,
+          -0.0006183265941217542,
+          2.241374022560194e-05,
+          -0.006938254926353693,
+          -0.01587197557091713,
+          -0.020658032968640327,
+          -0.006762409117072821,
+          0.009136324748396873,
+          -0.014037064276635647,
+          0.0032187385950237513,
+          -0.0060016862116754055,
+          0.009755606763064861,
+          0.013272518292069435,
+          0.012668526731431484,
+          -0.003195802215486765,
+          -0.007297591306269169
+        ],
+        "keyphrases": [
+          "Mixture-Of-Experts",
+          "Transformer language models",
+          "self-attention module",
+          "Feed-Forward Network",
+          "computational efficiency"
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "317d6fdf-f473-4de1-9258-7ec217a9b4ca",
+      "properties": {
+        "page_content": "3. Deepseekmoe Architecture\n\nOn top of the generic MoE architecture outlined in Section 2, we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. As illustrated in Figure 2, our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. Both of these strategies are designed to elevate the level of expert specialization.\n\n3.1. Fine-Grained Expert Segmentation\n\nIn scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts.\n\nIn pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. The finer expert segmentation enables a more flexible and adaptable combination of activated experts. To be specific, on top of a typical MoE architecture shown in Figure 2(a), we segment each expert FFN into smaller experts by reducing the FFN intermediate hidden dimension to 1 times its original size. Since each expert becomes smaller, in response, we also increase the number of activated experts to times to keep the same computation cost, as illustrated in Figure 2(b). With the fine-grained expert segmentation, the output of an MoE layer can be expressed as:\n\n$$\\mathbf{h}{t}^{i}=\\sum{i=1}^{\\text{mN}}\\left(g_{i,t}\\text{FFN}{i}\\left(\\mathbf{u}{t}^{i}\\right)\\right)+\\mathbf{u}{t}^{i},\\tag{6}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\text{Tokr}(s_{i,t}|1\\leqslant j\\leqslant m\\text{N}),m\\text{K}),\\ 0,&\\text{otherwise},\\end{cases}$$ (7) $$s_{i,t}=\\text{Softmax}{i}\\left(\\mathbf{u}{t}^{T}\\mathbf{e}_{t}^{i}\\right),\\tag{8}$$ where the total number of expert parameters is equal to $N$ times the number of parameters in a\n\n$$(6)$$\n\n(7) (8) $\\frac{1}{2}$\n\nstandard FFN, and denotes the total number of fine-grained experts. With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to .\n\nFrom a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. As an illustrative example, we consider the case where = 16. A typical top-2 routing strategy can yield 16 2 = 120 possible combinations. By contrast, if each expert is split into 4 smaller experts, the fine-grained routing strategy can yield 64 8 = 4, 426, 165, 368 potential combinations. The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition.\n\n3.2. Shared Expert Isolation\n\nWith a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. This alleviation of redundancy will contribute to a more parameterefficient model with more specialized experts.\n\nTowards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate experts to serve as shared experts. Regardless of the router module, each token will be deterministically assigned to these shared experts. In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by , as depicted in Figure 2(c). With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows:\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{K_{s}}\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)+\\sum_{i=K_{s}+1}^{mN}\\left(g_{i,t}\\;\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},\\tag{9}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\mathrm{Topk}{s_{j,t}|K_{s}+1\\leqslant j\\leqslant mN},mK-K_{s}},\\ 0,&\\text{otherwise},\\end{cases}$$ (10) $$s_{i,t}=\\mathrm{Softmax}{i}\\left(\\mathbf{u}{t}^{l,T}e_{i}^{l}\\right).\\tag{11}$$ $\\mathbf{h}\\mathbf{M}\\mathbf{E}$ the number of closed-compact is $K_{s}$ the total number of routed currents.\n\nFinally, in DeepSeekMoE, the number of shared expert is , the total number of routed experts is − , and the number of nonzero gates is − .\n\nIt is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022). The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint.\n\n3.3. Load Balance Consideration\n\nAutomatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. Firstly, there is a risk of routing collapse (Shazeer et al., 2017), i.e., the model always selects only a few experts, preventing other experts from sufficient training.\n\nSecondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.\n\nExpert-Level Balance Loss. In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. The computation of the balance loss is as follows:\n\n$$\\begin{array}{l}{{{\\mathcal L}{\\mathrm{ExpBal}}=\\alpha{1}\\sum_{i=1}^{N^{\\prime}}f_{i}P_{i},}}\\ {{f_{i}=\\frac{N^{\\prime}}{K^{T}}\\sum_{t=1}^{T}\\mathds{1}\\,(\\mathrm{Token}\\;t\\;\\mathrm{s}\\mathrm{lects}\\;\\mathrm{Expert}\\;i),}}\\ {{P_{i}=\\frac{1}{T}\\sum_{t=1}^{T}s_{i,t},}}\\end{array}$$\n\n(12) $$\\begin{array}{l}\\small\\text{(13)}\\end{array}$$ = (14) $$\\begin{array}{l}\\small\\text{(14)}\\end{array}$$ .\n\n7 where 1 is a hyper-parameter called expert-level balance factor, ′is equal to ( − ) and ′ is equal to ( − ) for brevity. 1(·) denotes the indicator function.\n\nDevice-Level Balance Loss. In addition to the expert-level balance loss, we introduce a devicelevel balance loss. When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. Instead, our primary objective is to ensure balanced computation across the devices. If we partition all routed experts into groups {E1, E2, ..., E}, and deploy each group on a single device, the device-level balance loss is computed as follows:\n\n$$\\mathcal{L}{\\text{DevBal}}=\\alpha{2}\\sum_{i=1}^{D}f_{i}^{\\prime}P_{i}^{\\prime},$$ $$f_{i}^{\\prime}=\\frac{1}{|\\mathcal{E}{i}|}\\sum{j\\in\\mathcal{E}{i}}f{j},$$ $$P_{i}^{\\prime}=\\sum_{j\\in\\mathcal{E}{i}}P{j},$$\n\n(15) $$\\begin{array}{l}\\small\\text{(16)}\\end{array}$$ = (17) .\n\nwhere 2 is a hyper-parameter called device-level balance factor. In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices.\n\n",
+        "title": "Deepseekmoe Architecture",
+        "keyphrases": [
+          "DeepSeekMoE architecture",
+          "expert specialization",
+          "fine-grained expert segmentation",
+          "shared expert isolation",
+          "load balance consideration"
+        ],
+        "embedding": [
+          -0.004791774787008762,
+          -0.04079068824648857,
+          -0.01656515710055828,
+          -0.01955496519804001,
+          -0.006609901320189238,
+          0.034003015607595444,
+          -0.01574093848466873,
+          0.050455041229724884,
+          -0.04541277140378952,
+          0.02537297084927559,
+          -0.010747150518000126,
+          -0.015199541114270687,
+          0.008557317778468132,
+          -0.01236326340585947,
+          -0.001814086688682437,
+          0.03117481619119644,
+          0.022674063220620155,
+          0.022867996245622635,
+          0.041534099727869034,
+          -0.032096002250909805,
+          0.02517903782427311,
+          -0.04825713112950325,
+          -0.03211216256022453,
+          0.018375203013420105,
+          -0.020395344123244286,
+          0.01574093848466873,
+          0.023368991911411285,
+          0.00403220159932971,
+          -0.06251124292612076,
+          0.021995296701788902,
+          0.029623348265886307,
+          0.015635890886187553,
+          0.006242235656827688,
+          -0.0028645601123571396,
+          -0.034003015607595444,
+          -0.022593257948756218,
+          0.029300125315785408,
+          0.025227520614862442,
+          -0.026358800008893013,
+          0.03578073903918266,
+          0.03626557067036629,
+          0.021866006776690483,
+          -0.0449279360473156,
+          0.007236145436763763,
+          -0.0028686001896858215,
+          0.004626123234629631,
+          -0.015256104990839958,
+          0.008306819945573807,
+          0.01797117479145527,
+          -0.036459505558013916,
+          -0.015344991348683834,
+          -0.02821733057498932,
+          0.012444068677723408,
+          0.003559488570317626,
+          -0.028928419575095177,
+          0.009276487864553928,
+          0.031077850610017776,
+          0.05061665549874306,
+          0.02391847036778927,
+          0.005931134335696697,
+          0.03561912849545479,
+          -0.007898751646280289,
+          -0.036136284470558167,
+          0.013381414115428925,
+          0.02915467508137226,
+          -0.021332690492272377,
+          0.01580558344721794,
+          -0.009599709883332253,
+          -0.006294759456068277,
+          0.009599709883332253,
+          -0.05798612907528877,
+          0.030916238203644753,
+          -0.034746427088975906,
+          0.000505792791955173,
+          -0.006007899530231953,
+          0.00270294863730669,
+          0.03772007301449776,
+          0.03183742240071297,
+          0.013324850238859653,
+          -0.028508231043815613,
+          -0.001757522695697844,
+          0.011183501221239567,
+          -0.014771271497011185,
+          0.0015050050569698215,
+          0.01929638721048832,
+          -0.04334414750337601,
+          -0.0023595248349010944,
+          -0.06988071650266647,
+          0.048612672835588455,
+          -0.0075997705571353436,
+          -0.029332447797060013,
+          0.003993819002062082,
+          0.004078664816915989,
+          -0.0046018813736736774,
+          -0.0031049568206071854,
+          0.059246696531772614,
+          -0.00016565156693104655,
+          -0.017599469050765038,
+          -0.03529590368270874,
+          0.0033534341491758823,
+          -0.0022443768102675676,
+          0.024177048355340958,
+          -0.029865765944123268,
+          0.009147198870778084,
+          0.028863774612545967,
+          -5.586952465819195e-05,
+          0.01226629689335823,
+          0.02770017459988594,
+          -0.006545256823301315,
+          0.09709606319665909,
+          -0.008993667550384998,
+          -0.005486703012138605,
+          -0.007801784668117762,
+          0.002919103717431426,
+          -0.014003617689013481,
+          -0.02333666943013668,
+          -0.03920689597725868,
+          -0.06483844667673111,
+          0.02320738136768341,
+          -0.044055234640836716,
+          -0.0021756920032203197,
+          0.0017140896525233984,
+          0.02535681053996086,
+          -0.022754868492484093,
+          -0.06299608200788498,
+          0.04172803461551666,
+          0.0025979015044867992,
+          0.003816046519204974,
+          0.013276367448270321,
+          0.013324850238859653,
+          0.014674304984509945,
+          -0.0171954408288002,
+          -0.006525055505335331,
+          -0.04761068522930145,
+          0.021736718714237213,
+          0.036459505558013916,
+          0.04476632550358772,
+          0.05426906794309616,
+          -0.0033837363589555025,
+          0.01847217045724392,
+          0.022932641208171844,
+          -0.035522159188985825,
+          -0.010456250049173832,
+          0.002995869144797325,
+          -0.023562924936413765,
+          0.0130581920966506,
+          0.00017979255062527955,
+          0.05426906794309616,
+          -0.011749139986932278,
+          0.026843633502721786,
+          -0.017793402075767517,
+          0.06380413472652435,
+          0.0016544954851269722,
+          0.021461978554725647,
+          0.0027574924752116203,
+          0.011143097653985023,
+          -0.022609418258070946,
+          -0.019118614494800568,
+          -0.0020393324084579945,
+          0.005167520605027676,
+          0.007381595205515623,
+          -0.03267780318856239,
+          -0.01442380715161562,
+          0.04104926809668541,
+          0.0012221853248775005,
+          0.05110148712992668,
+          0.007939154282212257,
+          0.008072483353316784,
+          -0.08242175728082657,
+          0.025227520614862442,
+          -0.006226074881851673,
+          0.022124584764242172,
+          0.022221552208065987,
+          0.040952298790216446,
+          -0.012985466979444027,
+          -0.022237712517380714,
+          0.026277994737029076,
+          0.0034786828327924013,
+          -0.022738708183169365,
+          0.0346817821264267,
+          -0.031982872635126114,
+          -0.009979496710002422,
+          -0.008250256069004536,
+          -0.02094482257962227,
+          0.03316263481974602,
+          0.02181752398610115,
+          0.02005596086382866,
+          0.0104966526851058,
+          -0.004832177422940731,
+          -0.007809865288436413,
+          0.031223300844430923,
+          0.02233467996120453,
+          0.011741059832274914,
+          0.006807875353842974,
+          0.006339202634990215,
+          -0.02702140621840954,
+          0.010456250049173832,
+          -0.008452270179986954,
+          0.0032544471323490143,
+          0.01527226623147726,
+          -0.00561599200591445,
+          -0.017550986260175705,
+          -0.0024928541388362646,
+          -0.019894348457455635,
+          0.022867996245622635,
+          -0.01039968617260456,
+          0.023433636873960495,
+          0.011312790215015411,
+          -0.004403907340019941,
+          -0.042794667184352875,
+          0.006254356820136309,
+          -0.01075523067265749,
+          -0.0016181330429390073,
+          -0.04415220394730568,
+          -0.01727624610066414,
+          -0.003593830857425928,
+          0.00556750874966383,
+          0.033615145832300186,
+          0.024500271305441856,
+          -0.0036200927570462227,
+          -0.00013534944446291775,
+          -0.007656334433704615,
+          0.008395706303417683,
+          0.0056079113855957985,
+          -0.0066947476007044315,
+          0.00873508956283331,
+          0.0026787070091813803,
+          0.01797117479145527,
+          0.018197430297732353,
+          -0.03406766057014465,
+          0.010957244783639908,
+          0.026471927762031555,
+          0.0039574564434587955,
+          0.019457997754216194,
+          0.02151046134531498,
+          -0.035522159188985825,
+          -0.013098594732582569,
+          -0.00875933188945055,
+          0.0058543686755001545,
+          -0.010302719660103321,
+          -0.020847855135798454,
+          0.0041029066778719425,
+          -0.056014470756053925,
+          -0.005272568203508854,
+          0.00509075541049242,
+          0.010165349580347538,
+          -0.008371464908123016,
+          0.05213579908013344,
+          0.00807652436196804,
+          0.007874510250985622,
+          -0.017130795866250992,
+          -0.010205752216279507,
+          0.028540553525090218,
+          -0.003670596284791827,
+          -0.03355050086975098,
+          0.03626557067036629,
+          0.01229861844331026,
+          -0.034003015607595444,
+          -0.004674606490880251,
+          -0.02656889520585537,
+          0.04421684890985489,
+          -0.020459989085793495,
+          -0.018245914950966835,
+          0.044055234640836716,
+          0.0020302417688071728,
+          -0.016985345631837845,
+          0.057598263025283813,
+          0.029122352600097656,
+          0.0038483687676489353,
+          -0.00885629840195179,
+          -0.004844298120588064,
+          -0.04082300886511803,
+          -0.015288427472114563,
+          -0.0186661034822464,
+          0.011280467733740807,
+          -0.016839895397424698,
+          -0.0046584452502429485,
+          -0.008104805834591389,
+          0.02079937234520912,
+          0.02902538701891899,
+          0.02327202446758747,
+          0.012629921548068523,
+          0.02144581824541092,
+          0.004185732454061508,
+          0.002262558089569211,
+          -0.006379605270922184,
+          -0.033744435757398605,
+          -0.02320738136768341,
+          -0.035263583064079285,
+          0.006092745345085859,
+          -0.01562781073153019,
+          -0.0018423686269670725,
+          -0.020540794357657433,
+          -0.01864994317293167,
+          0.05585286021232605,
+          0.006173551082611084,
+          -0.016161128878593445,
+          0.009737079963088036,
+          0.019393354654312134,
+          -0.031918227672576904,
+          -0.023934630677103996,
+          0.0034180786460638046,
+          0.009672435000538826,
+          0.007486642803996801,
+          -0.019619610160589218,
+          -0.0017534823855385184,
+          -0.04282699152827263,
+          0.030576854944229126,
+          -0.02176903933286667,
+          -0.010698666796088219,
+          0.0290900319814682,
+          0.008201772347092628,
+          0.006771512795239687,
+          -0.04340879246592522,
+          -0.015450038947165012,
+          0.021995296701788902,
+          0.020912500098347664,
+          0.03093240037560463,
+          -0.024710364639759064,
+          -0.022237712517380714,
+          0.016548994928598404,
+          0.007874510250985622,
+          -0.04638243839144707,
+          0.002288819756358862,
+          -0.009510824456810951,
+          -0.013147078454494476,
+          0.050455041229724884,
+          0.04243912175297737,
+          -0.03555448353290558,
+          -0.0016373243415728211,
+          -0.015094494447112083,
+          -0.014819755218923092,
+          -0.03726756200194359,
+          -0.013599589467048645,
+          -0.03694434091448784,
+          -0.014294518157839775,
+          -0.01682373508810997,
+          -0.008452270179986954,
+          -0.013648073188960552,
+          -0.018197430297732353,
+          -0.0077896639704704285,
+          0.006824036594480276,
+          -0.032047517597675323,
+          0.008137128315865993,
+          0.004391786642372608,
+          0.007894711568951607,
+          -0.0403381772339344,
+          -0.006444249767810106,
+          0.023320509120821953,
+          0.03144955635070801,
+          0.001097946660593152,
+          0.012759210541844368,
+          0.01808430254459381,
+          -0.0037130191922187805,
+          0.007979556918144226,
+          -0.052523668855428696,
+          -0.006383645813912153,
+          0.005078634712845087,
+          0.0022282155696302652,
+          0.000984818791039288,
+          0.021930651739239693,
+          -0.048160161823034286,
+          -0.038107942789793015,
+          -0.005874569993466139,
+          -0.04366736859083176,
+          0.01483591552823782,
+          0.004767532926052809,
+          0.0033695953898131847,
+          -0.005712958984076977,
+          -0.006573539227247238,
+          -0.023175058886408806,
+          0.022916480898857117,
+          0.008727009408175945,
+          -0.008670445531606674,
+          0.007894711568951607,
+          0.031013205647468567,
+          0.0419866107404232,
+          -0.02086401730775833,
+          -0.025130555033683777,
+          0.054301392287015915,
+          -0.011361273005604744,
+          0.01543387770652771,
+          0.028815291821956635,
+          0.014496532268822193,
+          0.012945064343512058,
+          0.032984863966703415,
+          0.010642102919518948,
+          0.007413917686790228,
+          -0.023255864158272743,
+          0.04308556765317917,
+          -0.01871458627283573,
+          -0.028233490884304047,
+          -0.019958993420004845,
+          0.012007718905806541,
+          0.006981607526540756,
+          -0.02600325644016266,
+          -0.029558703303337097,
+          -0.027183018624782562,
+          -0.0005520035629160702,
+          -0.008613881655037403,
+          -0.016379304230213165,
+          0.020702404901385307,
+          -0.0050059095956385136,
+          0.0006893731188029051,
+          -0.036847371608018875,
+          -0.004957426339387894,
+          -0.0010787553619593382,
+          -0.006965446285903454,
+          0.0363302156329155,
+          0.0006999789038673043,
+          0.003561508608981967,
+          -0.04024121165275574,
+          -0.022738708183169365,
+          -0.004181691911071539,
+          0.022318517789244652,
+          0.056079115718603134,
+          -0.004957426339387894,
+          0.039627086371183395,
+          -0.014528854750096798,
+          0.012427907437086105,
+          -0.025082072243094444,
+          0.01738937385380268,
+          0.0272638238966465,
+          0.00945426058024168,
+          0.006266477517783642,
+          0.015046010725200176,
+          0.014924801886081696,
+          -0.019781220704317093,
+          0.02669818513095379,
+          -0.0033271722495555878,
+          0.01675909012556076,
+          0.0015726798446848989,
+          0.01154712587594986,
+          -0.02574467845261097,
+          -0.01490056049078703,
+          -0.022544773295521736,
+          -0.024629559367895126,
+          0.054301392287015915,
+          -0.02600325644016266,
+          0.013745039701461792,
+          -0.0016757070552557707,
+          0.018504492938518524,
+          -0.03180510178208351,
+          -0.01599951647222042,
+          -0.04043514281511307,
+          -0.04473400488495827,
+          -0.04686727374792099,
+          -0.019765060395002365,
+          -0.00938961561769247,
+          -5.8962868934031576e-05,
+          -0.03995031118392944,
+          0.021607428789138794,
+          -0.05627305060625076,
+          -0.008929023519158363,
+          0.03772007301449776,
+          -0.022415485233068466,
+          0.0011908732121810317,
+          -0.021365011110901833,
+          0.009421938098967075,
+          -0.04900054261088371,
+          0.002274678787216544,
+          0.027344629168510437,
+          0.0555296391248703,
+          -0.0013706657337024808,
+          -0.022981123998761177,
+          0.004375625401735306,
+          -0.013405656442046165,
+          0.0018797412049025297,
+          0.0010565337724983692,
+          0.009114876389503479,
+          0.02815268561244011,
+          -0.017114635556936264,
+          0.053105469793081284,
+          0.012177410535514355,
+          0.030463727191090584,
+          0.026455767452716827,
+          -0.007078574039041996,
+          -0.06548488885164261,
+          -0.0006020020227879286,
+          0.021720556542277336,
+          0.010165349580347538,
+          0.0018544894410297275,
+          -0.004945305176079273,
+          0.004094826057553291,
+          0.0049049025401473045,
+          -0.0015716697089374065,
+          0.01625809445977211,
+          0.012015799060463905,
+          0.00763613311573863,
+          -0.004678646568208933,
+          0.013300608843564987,
+          0.015134897083044052,
+          -0.035457514226436615,
+          0.022221552208065987,
+          0.011805704794824123,
+          -0.048612672835588455,
+          -0.008727009408175945,
+          -0.02155894599854946,
+          0.02181752398610115,
+          -0.003492823801934719,
+          -0.016338901594281197,
+          -0.004569558892399073,
+          0.033938370645046234,
+          0.01973273791372776,
+          -0.0040645236149430275,
+          -0.014650062657892704,
+          0.013332931324839592,
+          0.04250376671552658,
+          -0.022997286170721054,
+          -0.030560694634914398,
+          0.011741059832274914,
+          -0.00397159717977047,
+          -0.0015454079257324338,
+          0.0017241904279217124,
+          -0.016613639891147614,
+          0.04470168054103851,
+          -0.009316890500485897,
+          0.035134293138980865,
+          0.004115027375519276,
+          0.014633901417255402,
+          -0.019021648913621902,
+          0.009179521352052689,
+          -0.018068142235279083,
+          0.02689211815595627,
+          -0.0157813411206007,
+          0.044119879603385925,
+          0.026972923427820206,
+          0.012646082788705826,
+          0.024694204330444336,
+          -0.035134293138980865,
+          -0.012702646665275097,
+          0.014221793040633202,
+          0.01884387619793415,
+          -0.010900680907070637,
+          -0.010213833302259445,
+          -0.014609660021960735,
+          -0.008751250803470612,
+          0.03772007301449776,
+          -0.02162358909845352,
+          -0.024112403392791748,
+          0.001966607291251421,
+          -0.0013797563733533025,
+          0.0006626062677241862,
+          -0.01588638871908188,
+          -0.016468189656734467,
+          0.017631791532039642,
+          0.050519686192274094,
+          -0.02244780771434307,
+          0.022415485233068466,
+          0.040144242346286774,
+          -0.0048038954846560955,
+          -0.020670082420110703,
+          -0.025728516280651093,
+          -0.009098715148866177,
+          0.00039493758231401443,
+          -0.019247904419898987,
+          0.016379304230213165,
+          -0.008565397933125496,
+          -0.00945426058024168,
+          -0.04612386226654053,
+          -0.0010247165337204933,
+          0.02516287751495838,
+          0.0033312125597149134,
+          0.015579327940940857,
+          -0.016985345631837845,
+          -0.005684677045792341,
+          -0.004876620601862669,
+          0.024257853627204895,
+          -0.016920700669288635,
+          0.007672495674341917,
+          0.0024605318903923035,
+          0.0003421613946557045,
+          0.0009858288103714585,
+          -0.003064553951844573,
+          0.00032120241667144,
+          0.007405837066471577,
+          0.027086051180958748,
+          -0.008355303667485714,
+          -0.023449797183275223,
+          0.007983597926795483,
+          0.02079937234520912,
+          -0.02598709426820278,
+          0.004917023237794638,
+          0.049873244017362595,
+          0.023869987577199936,
+          -0.013518784195184708,
+          0.005963456351310015,
+          -0.0010444129584357142,
+          0.017049990594387054,
+          0.0040604835376143456,
+          0.01505409087985754,
+          0.03494035825133324,
+          0.0016686365706846118,
+          -0.03193439170718193,
+          -0.00795531552284956,
+          0.0017322709318250418,
+          0.010569377802312374,
+          0.016662122681736946,
+          0.0008716908632777631,
+          -0.04573599249124527,
+          0.0013757160631939769,
+          0.005797804798930883,
+          0.010884519666433334,
+          -0.014625821262598038,
+          -0.02214074693620205,
+          0.022948801517486572,
+          0.026213349774479866,
+          0.001036332338117063,
+          0.0035170656628906727,
+          0.005914973095059395,
+          0.017066150903701782,
+          0.02307809144258499,
+          0.022092262282967567,
+          -0.014367243275046349,
+          0.019183259457349777,
+          -0.004755412228405476,
+          -0.012355182319879532,
+          -0.001959536923095584,
+          -0.020459989085793495,
+          0.021348850801587105,
+          0.044378459453582764,
+          0.02094482257962227,
+          0.018423685804009438,
+          0.017825724557042122,
+          0.010270397178828716,
+          0.026229511946439743,
+          -0.02732846885919571,
+          -0.03600699454545975,
+          0.0025938611943274736,
+          -0.0003156470484100282,
+          -0.025502260774374008,
+          -0.002603961853310466,
+          -0.018439847975969315,
+          -0.039756376296281815,
+          0.009623952209949493,
+          -0.0026160827837884426,
+          0.010472411289811134,
+          0.03442320227622986,
+          -0.01077947299927473,
+          -0.03224145248532295,
+          -0.0004121087840758264,
+          0.033421214669942856,
+          0.01797117479145527,
+          0.039497796446084976,
+          -0.01877923123538494,
+          0.0062462761998176575,
+          -0.0014656123239547014,
+          0.007991678081452847,
+          0.01372887846082449,
+          -0.007676536217331886,
+          -0.013389495201408863,
+          0.030124343931674957,
+          0.023482119664549828,
+          0.04777229577302933,
+          -0.006484652869403362,
+          -0.043473437428474426,
+          -0.012233974412083626,
+          -0.01531266886740923,
+          0.015062171965837479,
+          0.008605800569057465,
+          0.0050947959534823895,
+          0.0008646203787066042,
+          0.008880539797246456,
+          -0.007583609316498041,
+          0.006254356820136309,
+          0.007098775822669268,
+          -0.01879539154469967,
+          0.03332424536347389,
+          -0.008096725679934025,
+          0.0302374716848135,
+          -0.019942833110690117,
+          -0.02548610046505928,
+          -0.01864994317293167,
+          0.0024241693317890167,
+          0.0023150816559791565,
+          -0.026229511946439743,
+          0.03049604967236519,
+          0.03471410274505615,
+          -0.0018666103715077043,
+          0.008331061340868473,
+          -0.010432008653879166,
+          0.003490803763270378,
+          0.002375685842707753,
+          -0.006213953718543053,
+          0.007935114204883575,
+          0.0003030211664736271,
+          -0.026035577058792114,
+          0.035198938101530075,
+          -0.03542519360780716,
+          0.0018777211662381887,
+          -0.020718567073345184,
+          0.021979134529829025,
+          0.008452270179986954,
+          0.008387625217437744,
+          0.011676414869725704,
+          -0.023304346948862076,
+          -0.0030059698037803173,
+          0.009777482599020004,
+          0.015603569336235523,
+          0.008710848167538643,
+          -0.015749020501971245,
+          0.012476391158998013,
+          0.0019060031045228243,
+          0.020960982888936996,
+          0.03030211664736271,
+          0.00044241087744012475,
+          0.008250256069004536,
+          -0.01638738438487053,
+          -0.019215581938624382,
+          -0.0061533497646451,
+          0.010310799814760685,
+          0.030512209981679916,
+          -0.012169329449534416,
+          0.04192196577787399,
+          0.037849362939596176,
+          -0.003840288147330284,
+          -0.006965446285903454,
+          0.010254235938191414,
+          -0.0551740936934948,
+          0.05071362107992172,
+          -0.003490803763270378,
+          -0.0007388665690086782,
+          0.020088283345103264,
+          -0.0442814938724041,
+          -0.023175058886408806,
+          0.005098836030811071,
+          0.010827955789864063,
+          0.01732472889125347,
+          0.009842127561569214,
+          0.002264578128233552,
+          0.014504612423479557,
+          -0.03136875107884407,
+          -0.039562441408634186,
+          0.02763552963733673,
+          0.00970475748181343,
+          0.004123107995837927,
+          0.012815774418413639,
+          0.02018524892628193,
+          -0.0017716636648401618,
+          0.008840137161314487,
+          0.01309051364660263,
+          -0.0021150875836610794,
+          0.01732472889125347,
+          0.02600325644016266,
+          -0.010706747882068157,
+          0.026035577058792114,
+          0.011215822771191597,
+          0.03781704232096672,
+          -0.0050543928518891335,
+          -0.00488470122218132,
+          0.005474582314491272,
+          -0.021785201504826546,
+          0.03195055201649666,
+          0.0031392991077154875,
+          -0.03846348449587822,
+          0.01682373508810997,
+          0.022367002442479134,
+          -0.017793402075767517,
+          0.021284205839037895,
+          0.0014858137583360076,
+          0.020912500098347664,
+          0.026956763118505478,
+          0.0040039196610450745,
+          0.0020585237070918083,
+          0.028896097093820572,
+          0.01886003650724888,
+          -0.008387625217437744,
+          0.050261110067367554,
+          0.010674425400793552,
+          -0.007664415054023266,
+          0.01511873584240675,
+          0.0013605649583041668,
+          0.02624567225575447,
+          -0.047028884291648865,
+          8.831804007058963e-05,
+          -0.012710727751255035,
+          -6.256124470382929e-05,
+          0.011821865104138851,
+          -0.01173297967761755,
+          0.003684737253934145,
+          -0.033744435757398605,
+          0.023643730208277702,
+          0.005102876108139753,
+          0.022560935467481613,
+          -0.008428028784692287,
+          0.006916963029652834,
+          0.016662122681736946,
+          -0.0009782533161342144,
+          0.007042211946099997,
+          0.009777482599020004,
+          -0.006193752400577068,
+          0.017486341297626495,
+          -0.008395706303417683,
+          0.016807572916150093,
+          0.046641018241643906,
+          -0.01636314205825329,
+          -0.0038039255887269974,
+          0.039691731333732605,
+          -0.009114876389503479,
+          -0.025195199996232986,
+          0.020459989085793495,
+          -0.007632093038409948,
+          -0.0006838177214376628,
+          0.04230983555316925,
+          -0.043796658515930176,
+          -0.025825483724474907,
+          0.004016040358692408,
+          -0.023562924936413765,
+          -0.019635770469903946,
+          0.009365374222397804,
+          0.016314659267663956,
+          0.003751402022317052,
+          -0.04421684890985489,
+          0.005947295110672712,
+          -0.000348221801687032,
+          0.01258143875747919,
+          -0.00016312638763338327,
+          -0.01688838005065918,
+          -0.036394860595464706,
+          0.011757221072912216,
+          -0.004436229821294546,
+          -0.025890128687024117,
+          4.217044624965638e-05,
+          -0.010294638574123383,
+          -0.05242669954895973,
+          -0.026665862649679184,
+          0.040758367627859116,
+          0.0012009738711640239,
+          0.033485859632492065,
+          -5.463850175146945e-05,
+          0.03262931853532791,
+          -0.00822601467370987,
+          -0.02075088955461979,
+          0.01517529971897602,
+          -0.0073371524922549725,
+          0.016330819576978683,
+          0.010019899345934391,
+          0.001137339393608272,
+          -0.00023951297043822706,
+          0.03490803763270378,
+          -0.033421214669942856,
+          0.004030181560665369,
+          0.0038726103957742453,
+          -0.012112765572965145,
+          -0.045186515897512436,
+          0.010213833302259445,
+          -0.00844419002532959,
+          -0.009955255314707756,
+          0.022059939801692963,
+          -0.011821865104138851,
+          0.001617122907191515,
+          0.025276005268096924,
+          0.006484652869403362,
+          -0.006488692946732044,
+          -0.0363302156329155,
+          -0.013462220318615437,
+          0.0090098287910223,
+          -0.008508834056556225,
+          -0.0008686606306582689,
+          -0.010318880900740623,
+          -0.04176035523414612,
+          0.014884399250149727,
+          -0.025502260774374008,
+          -0.006116987206041813,
+          0.014811674132943153,
+          0.019878188148140907,
+          -0.010092624463140965,
+          -0.04205125570297241,
+          -0.007038171403110027,
+          -0.03290405869483948,
+          -0.006929083727300167,
+          0.003482723142951727,
+          -0.01044008880853653,
+          0.008904782123863697,
+          0.016112644225358963,
+          -0.0186661034822464,
+          0.010165349580347538,
+          -0.02663354016840458,
+          0.021203400567173958,
+          -0.014254115521907806,
+          0.01593487337231636,
+          0.0021130675449967384,
+          0.003232225775718689,
+          -0.021962974220514297,
+          0.025954771786928177,
+          -0.0299142487347126,
+          -0.004816016182303429,
+          -0.0315626822412014,
+          -0.02624567225575447,
+          -0.030140504240989685,
+          0.006848277989774942,
+          0.0010555237531661987,
+          -0.04091997817158699,
+          -0.005284688901156187,
+          0.015320749953389168,
+          0.0020373123697936535,
+          -0.028928419575095177,
+          -0.020136766135692596,
+          0.0009827986359596252,
+          0.007349273189902306,
+          0.0017928752349689603,
+          -0.031724296510219574,
+          -0.008678525686264038,
+          -0.032483868300914764,
+          0.020201411098241806,
+          -0.0017706536455079913,
+          0.027344629168510437,
+          0.0011262285988777876,
+          0.003840288147330284,
+          -0.02029837667942047,
+          0.002234275918453932,
+          0.0036907976027578115,
+          0.022302357479929924,
+          -0.03484339267015457,
+          0.016548994928598404,
+          0.035392872989177704,
+          0.022221552208065987,
+          -0.049356088042259216,
+          0.0016342941671609879,
+          0.01385008729994297,
+          -0.011021889746189117,
+          0.01871458627283573,
+          -0.005733160302042961,
+          0.025082072243094444,
+          -0.01277537178248167,
+          0.0005641243769787252,
+          0.005684677045792341,
+          0.006327081937342882,
+          -0.0022564975079149008,
+          0.004335222765803337,
+          -0.024435626342892647,
+          0.02320738136768341,
+          0.02687595598399639,
+          0.007551287300884724,
+          0.010634022764861584,
+          -0.0031069768592715263,
+          -0.012387504801154137,
+          0.011805704794824123,
+          -0.026488089933991432,
+          -0.0005691747646778822,
+          0.011021889746189117,
+          -0.022544773295521736,
+          -0.006815955974161625,
+          0.03959476575255394,
+          0.024952782317996025,
+          -0.05313779041171074,
+          0.017680274322628975,
+          0.018423685804009438,
+          -0.005058433394879103,
+          0.01747017912566662,
+          -0.0403381772339344,
+          -0.00844419002532959,
+          -0.0254376158118248,
+          0.0005666495417244732,
+          0.016807572916150093,
+          0.011005728505551815,
+          -0.015377313829958439,
+          0.024936622008681297,
+          0.0003035262052435428,
+          0.03201519697904587,
+          -0.002511035418137908,
+          -0.020411504432559013,
+          -0.02650425024330616,
+          -0.020605439320206642,
+          -0.00743007892742753,
+          0.009858288802206516,
+          -0.0005403877585195005,
+          -0.03161116689443588,
+          -0.03746149688959122,
+          0.007385635748505592,
+          0.00307667488232255,
+          -0.04790158569812775,
+          -0.013171319849789143,
+          -0.02845974639058113,
+          -0.03060917742550373,
+          -0.009163360111415386,
+          -0.009058312512934208,
+          -0.03775239735841751,
+          0.0006009919452480972,
+          0.003133238758891821,
+          0.021672073751688004,
+          -0.010868359357118607,
+          0.013720798306167126,
+          -0.02333666943013668,
+          -0.007838147692382336,
+          0.0016706566093489528,
+          0.03985334187746048,
+          -0.009615871123969555,
+          -0.035069648176431656,
+          0.011021889746189117,
+          0.019377192482352257,
+          -0.015902550891041756,
+          0.012217813171446323,
+          0.005082674790173769,
+          -0.006678586360067129,
+          0.017793402075767517,
+          -0.013955134898424149,
+          -0.03762310743331909,
+          0.004549357574433088,
+          -0.013147078454494476,
+          -0.01442380715161562,
+          -0.017179278656840324,
+          0.010367363691329956,
+          0.02214074693620205,
+          -0.021930651739239693,
+          0.008880539797246456,
+          0.04486329108476639,
+          -0.02227003499865532,
+          -0.029364770278334618,
+          0.01053705532103777,
+          -0.018375203013420105,
+          0.01401169877499342,
+          -0.004214014392346144,
+          0.03062533773481846,
+          -0.009163360111415386,
+          0.01053705532103777,
+          -0.015425797551870346,
+          0.010092624463140965,
+          -0.03384140133857727,
+          -0.019005486741662025,
+          -0.0034079779870808125,
+          -0.007672495674341917,
+          -0.03778471797704697,
+          -0.03616860508918762,
+          0.022916480898857117,
+          0.019150936976075172,
+          -0.009906771592795849,
+          0.0058947717770934105,
+          -0.010836036875844002,
+          9.204267553286627e-05,
+          0.026520412415266037,
+          0.016613639891147614,
+          -0.008177530951797962,
+          0.0012292558094486594,
+          -0.0047594523057341576,
+          0.0065088942646980286,
+          0.019312549382448196,
+          -0.06493541598320007,
+          -0.029203159734606743,
+          0.02504974976181984,
+          0.01789036951959133,
+          0.012330940924584866,
+          0.015207622200250626,
+          0.02535681053996086,
+          -0.02383766509592533,
+          -0.020912500098347664,
+          0.037849362939596176,
+          -0.003381716087460518,
+          0.01024615578353405,
+          -0.007947235368192196,
+          0.0019302447326481342,
+          0.00030453625367954373,
+          0.002240336500108242,
+          -0.0010797653812915087,
+          0.005757402162998915,
+          0.036394860595464706,
+          0.02239932492375374,
+          0.013122836127877235,
+          0.015078333206474781,
+          -0.0013625851133838296,
+          -0.004973587114363909,
+          -0.007951275445520878,
+          -0.0037796839606016874,
+          0.025405293330550194,
+          -0.017292408272624016,
+          0.0017150997882708907,
+          -0.026277994737029076,
+          0.02492045983672142,
+          0.025130555033683777,
+          0.01708231307566166,
+          -0.005369534716010094,
+          -0.007034131325781345,
+          0.013421816751360893,
+          0.0034766627941280603,
+          -0.009276487864553928,
+          0.016807572916150093,
+          0.0068442379124462605,
+          -0.020379183813929558,
+          -0.011789543554186821,
+          0.008205813355743885,
+          -0.007207863498479128,
+          -0.022948801517486572,
+          0.031158655881881714,
+          0.012831935659050941,
+          0.012662244029343128,
+          0.004795814864337444,
+          -0.0005444280104711652,
+          0.011522884480655193,
+          0.03484339267015457,
+          -0.036588795483112335,
+          -0.01771259680390358,
+          0.005434179678559303,
+          -0.02138117328286171,
+          0.00929264910519123,
+          -0.014908640645444393,
+          0.006193752400577068,
+          0.015983356162905693,
+          0.07408261299133301,
+          -0.027247661724686623,
+          -4.744175021187402e-05,
+          -0.016274256631731987,
+          -0.017098473384976387,
+          0.018326720222830772,
+          -0.004230175167322159,
+          0.0051109567284584045,
+          0.02765168994665146,
+          0.008395706303417683,
+          0.03636253997683525,
+          -0.007013929542154074,
+          -0.005373575259000063,
+          0.023385152220726013,
+          -0.0018645902164280415,
+          -0.010852198116481304,
+          -0.006723029538989067,
+          0.001137339393608272,
+          0.012815774418413639,
+          0.00790279172360897,
+          0.03258083388209343,
+          0.017567146569490433,
+          0.0012019838904961944,
+          -0.00039493758231401443,
+          -0.0044564311392605305,
+          0.028556713834404945,
+          0.012848096899688244,
+          0.0017423717072233558,
+          -0.01953880488872528,
+          -0.003040312323719263,
+          0.027473919093608856,
+          -0.021672073751688004,
+          0.009922932833433151,
+          0.015328830108046532,
+          0.005034191533923149,
+          0.04809551686048508,
+          -0.03351818025112152,
+          -0.008298739790916443,
+          0.012064282782375813,
+          0.0018625700613483787,
+          -0.008068443275988102,
+          0.01480359397828579,
+          0.006581619381904602,
+          -0.01056129764765501,
+          -0.00926840677857399,
+          0.012613760307431221,
+          0.013575348071753979,
+          -0.0063917264342308044,
+          -0.00901790987700224,
+          0.0066543444991111755,
+          -0.010771391913294792,
+          0.019958993420004845,
+          -0.00035781748010776937,
+          0.008254296146333218,
+          0.004193812608718872,
+          0.008718929253518581,
+          -0.0007681586430408061,
+          -0.04111390933394432,
+          -0.00789067056030035,
+          0.016177289187908173,
+          -0.003333232831209898,
+          0.021348850801587105,
+          -0.001368645578622818,
+          -0.008565397933125496,
+          -0.014876319095492363,
+          -0.004876620601862669,
+          -0.0036019114777445793,
+          -0.00519984308630228,
+          0.0005641243769787252,
+          -0.00471500912681222,
+          -0.01738937385380268,
+          0.011126937344670296,
+          -0.028766809031367302,
+          -0.007680576294660568,
+          -0.0033473738003522158,
+          -0.014246034435927868,
+          -0.004525116179138422,
+          -0.007094735279679298,
+          0.005102876108139753,
+          -0.007595730479806662,
+          0.019134776666760445,
+          0.0050543928518891335,
+          -0.029057709500193596,
+          -0.008710848167538643,
+          -0.027296146377921104,
+          0.017631791532039642,
+          -0.027215341106057167,
+          0.01192691270262003,
+          0.025340650230646133,
+          0.002365585183724761,
+          -0.0012100645108148456,
+          0.004131188616156578,
+          -0.029817281290888786,
+          0.0036685762461274862,
+          0.010609780438244343,
+          -0.009728998877108097,
+          0.00901790987700224,
+          0.023627569898962975,
+          -0.01816510781645775,
+          -0.025841644033789635,
+          -0.03207983821630478,
+          0.01565205305814743,
+          0.011110776104032993,
+          -0.014157148078083992,
+          -0.011692576110363007,
+          -0.0012080443557351828,
+          0.004929144401103258,
+          -0.01378544233739376,
+          -0.00010245903104078025,
+          -0.00785834901034832,
+          -0.010326961055397987,
+          -0.008088644593954086,
+          -0.008985587395727634,
+          -0.002513055456802249,
+          -0.0038907916750758886,
+          0.008355303667485714,
+          0.006933124270290136,
+          -0.03494035825133324,
+          -0.025841644033789635,
+          0.012856177985668182,
+          0.007288668770343065,
+          -0.01331677008420229,
+          0.009567388333380222,
+          0.014989446848630905,
+          0.004965506494045258,
+          -0.002377706114202738,
+          0.009890610352158546,
+          -0.018456008285284042,
+          0.0127188079059124,
+          0.0046665258705616,
+          0.004084725398570299,
+          0.016920700669288635,
+          0.015151058323681355,
+          0.012339022010564804,
+          0.01293698325753212,
+          -0.03290405869483948,
+          0.0007151299505494535,
+          0.027991075068712234,
+          0.048580352216959,
+          0.01636314205825329,
+          -0.022754868492484093,
+          0.021962974220514297,
+          0.04625314846634865,
+          0.02024989388883114,
+          -0.00060301210032776,
+          -0.016985345631837845,
+          0.0050543928518891335,
+          0.026261834427714348,
+          0.010351202450692654,
+          0.00418977253139019,
+          -0.006415967829525471,
+          -0.013551106676459312,
+          0.014173309318721294,
+          0.009462340734899044,
+          0.00678363349288702,
+          -0.037914007902145386,
+          0.020023638382554054,
+          -0.023579087108373642,
+          0.020653922110795975,
+          0.0027756737545132637,
+          -0.012427907437086105,
+          0.016314659267663956,
+          0.022156907245516777,
+          -0.0048725800588727,
+          0.020960982888936996,
+          0.0008767411927692592,
+          0.011296628974378109,
+          0.00837954506278038,
+          -0.011934993788599968,
+          0.003896852023899555,
+          -0.005975577048957348,
+          0.003793824929744005,
+          0.006189712323248386,
+          0.02302960865199566,
+          0.00397159717977047,
+          0.0186661034822464,
+          0.017696436494588852,
+          0.0010045150993391871,
+          0.008104805834591389,
+          -0.019748898223042488,
+          -0.00022486694797407836,
+          -0.009696677327156067,
+          0.007349273189902306,
+          0.009462340734899044,
+          0.0072927093133330345,
+          0.02479117177426815,
+          -0.012597599998116493,
+          -0.010585539042949677,
+          -0.01816510781645775,
+          0.014561176300048828,
+          -0.014149067923426628,
+          0.01634698174893856,
+          0.017761079594492912,
+          0.0012363262940198183,
+          0.008565397933125496,
+          0.02446794882416725,
+          0.002816076623275876,
+          -0.01656515710055828,
+          0.011078453622758389,
+          -0.023740697652101517,
+          -0.017308568581938744,
+          0.013478381559252739,
+          -0.01315515860915184,
+          -0.010270397178828716,
+          0.020524634048342705,
+          -0.015967195853590965,
+          0.0017565126763656735,
+          0.011151178739964962,
+          0.03062533773481846,
+          -0.017373213544487953,
+          0.022318517789244652,
+          -0.01075523067265749,
+          -0.023110413923859596,
+          -0.002626183442771435,
+          0.016662122681736946,
+          0.0269406009465456,
+          1.085036728909472e-06,
+          -0.005729120224714279,
+          -0.005207923706620932,
+          0.02309425175189972,
+          -0.008573479019105434,
+          0.022867996245622635,
+          0.007034131325781345,
+          0.012419827282428741,
+          0.00020618064445443451,
+          -0.018003497272729874,
+          0.017437858507037163,
+          0.0020544833969324827,
+          0.02214074693620205,
+          0.014407645910978317,
+          -0.015749020501971245,
+          0.02904154732823372,
+          -0.0030241510830819607,
+          0.028556713834404945,
+          0.0007206853479146957,
+          -0.009583549574017525,
+          0.02845974639058113,
+          -0.0016807573847472668,
+          -0.0004017555620521307,
+          0.006165470462292433,
+          0.01505409087985754,
+          0.016443947330117226,
+          0.00392311392351985,
+          0.020605439320206642,
+          0.016306579113006592,
+          -0.00015100554446689785,
+          0.014318759553134441,
+          -0.00879165343940258,
+          -0.022819513455033302,
+          0.022043779492378235,
+          -0.005034191533923149,
+          -0.01745401881635189,
+          0.01734089106321335,
+          -0.007696737535297871,
+          -0.004058463498950005,
+          -0.0007075543981045485,
+          0.023320509120821953,
+          0.016209611669182777,
+          -0.007575529161840677,
+          0.003365555079653859,
+          -0.004282698966562748,
+          -0.001307031256146729,
+          0.02789410762488842,
+          0.008613881655037403,
+          -0.0034968641120940447,
+          0.01511873584240675,
+          0.005236205644905567,
+          0.01024615578353405,
+          -0.00025580034707672894,
+          -0.007466441486030817,
+          -0.01955496519804001,
+          -0.0035796898882836103,
+          0.004763492848724127,
+          0.005494783632457256,
+          0.0005767502589151263,
+          -0.0060563827864825726,
+          -0.020734727382659912,
+          -0.04476632550358772,
+          -0.00019128210260532796,
+          0.0014726828085258603,
+          0.025922449305653572,
+          0.04169571027159691,
+          -0.005902852397412062,
+          -0.004747331608086824,
+          -0.012452149763703346,
+          0.0014009678270667791,
+          -0.008533075451850891,
+          -0.019700415432453156,
+          -0.006064463406801224,
+          -0.013817764818668365,
+          -0.01916709914803505,
+          0.01615304872393608,
+          -0.012654163874685764,
+          0.006048302166163921,
+          0.002583760302513838,
+          0.019215581938624382,
+          -0.016742929816246033,
+          0.02744159661233425,
+          -0.01634698174893856,
+          0.004747331608086824,
+          0.001796915428712964,
+          -0.002872640499845147,
+          -0.03322727978229523,
+          0.02068624459207058,
+          -0.003304950660094619,
+          -0.036685761064291,
+          -0.0037716033402830362,
+          -0.007219984196126461,
+          -0.009955255314707756,
+          0.03429391607642174,
+          0.0014171289512887597,
+          -0.007805825211107731,
+          -0.012096604332327843,
+          0.01119158137589693,
+          0.011328951455652714,
+          0.008197732269763947,
+          -0.006020020227879286,
+          -0.015029849484562874,
+          0.007405837066471577,
+          0.03267780318856239,
+          -0.019765060395002365,
+          -0.015207622200250626,
+          -0.0014858137583360076,
+          0.011692576110363007,
+          0.002739311195909977,
+          0.005927093792706728,
+          -0.030156666412949562,
+          -0.014302598312497139,
+          -0.009607790969312191,
+          -0.01135319285094738,
+          -0.022674063220620155,
+          -0.007979556918144226,
+          0.00907447375357151,
+          -0.02125188335776329,
+          0.02138117328286171,
+          0.029881926253437996,
+          -0.027991075068712234,
+          0.0340353362262249,
+          -0.011345111764967442,
+          0.009599709883332253,
+          -0.01034312229603529,
+          0.0007449270342476666,
+          -0.011102695018053055,
+          0.0007964406395331025,
+          0.011304709129035473,
+          -0.03581305965781212,
+          -0.04237448051571846,
+          -0.008945184759795666,
+          0.0010959265055134892,
+          -0.01795501448214054,
+          -0.006925043649971485,
+          0.03959476575255394,
+          0.0100845443084836,
+          -0.0031837422866374254,
+          -0.016225773841142654,
+          0.01864994317293167,
+          0.02233467996120453,
+          0.03262931853532791,
+          0.01555508654564619,
+          0.00450895493850112,
+          0.0008186621707864106,
+          0.011789543554186821,
+          -0.027490079402923584,
+          0.014795512892305851,
+          -0.0013696555979549885,
+          0.02676282823085785,
+          -0.0015746999997645617,
+          -0.0003719584783539176,
+          0.019748898223042488,
+          -0.024257853627204895,
+          -0.0024847735185176134,
+          0.00732099125161767,
+          -0.014722787775099277,
+          0.004965506494045258,
+          0.019021648913621902,
+          0.007757341489195824,
+          -0.036588795483112335,
+          0.0041655306704342365,
+          -0.010771391913294792,
+          0.01986202597618103,
+          0.00917144026607275,
+          0.010423927567899227,
+          -0.0031291984487324953,
+          -0.011102695018053055,
+          -0.011910751461982727,
+          0.007817945443093777,
+          -0.0025979015044867992,
+          -0.008003799244761467,
+          0.03041524440050125,
+          -0.003412018297240138,
+          -0.0032382861245423555,
+          0.00837954506278038,
+          0.006032141391187906,
+          0.006036181468516588,
+          0.0085977204144001,
+          -0.009163360111415386,
+          -0.024435626342892647,
+          -0.01490056049078703,
+          -0.018746908754110336,
+          0.005353373941034079,
+          0.010262317024171352,
+          -0.003717059502378106,
+          -0.02416088618338108,
+          0.005114997271448374,
+          -0.007203822955489159,
+          0.036782726645469666,
+          -0.008347222581505775,
+          -0.035457514226436615,
+          -0.02377302013337612,
+          -0.04667333886027336,
+          0.009211842902004719,
+          6.436042895074934e-05,
+          -0.021979134529829025,
+          -0.0039614965207874775,
+          -0.011490561999380589,
+          -0.028007235378026962,
+          0.012056201696395874,
+          0.019651932641863823,
+          -0.026019416749477386,
+          0.021461978554725647,
+          0.00763613311573863,
+          -0.015102574601769447,
+          0.00567659642547369,
+          -0.008743170648813248,
+          -0.0269406009465456,
+          0.002250437159091234,
+          -0.006278598215430975,
+          0.017114635556936264,
+          -0.01173297967761755,
+          -0.0001382534101139754,
+          -0.020912500098347664,
+          0.01860145851969719,
+          0.021219560876488686,
+          -0.011070373468101025,
+          0.0014433907344937325,
+          -0.0037210998125374317,
+          -0.02568003349006176,
+          0.005555388052016497,
+          0.0007545226835645735,
+          -0.004614002071321011,
+          -0.0004100886289961636,
+          0.006791714113205671,
+          -0.009122957475483418,
+          0.00086159014608711,
+          -0.0051513598300516605,
+          0.008274497464299202,
+          0.0017393414163962007,
+          -0.01034312229603529,
+          -0.002296900376677513,
+          -0.003636253997683525,
+          0.0024484109599143267,
+          0.019134776666760445,
+          -0.015183379873633385,
+          0.007777543272823095,
+          -0.03157884627580643,
+          0.006832117214798927,
+          0.009623952209949493,
+          -0.0006408897461369634,
+          -0.02813652530312538,
+          0.0010327971540391445,
+          -0.0007090694853104651,
+          0.005531146191060543,
+          -0.026342639699578285,
+          0.029267802834510803,
+          0.0216397512704134,
+          0.011458240449428558,
+          0.015862148255109787,
+          0.001985798589885235,
+          -0.013712717220187187,
+          -0.016678284853696823,
+          0.003785744309425354,
+          -0.016936862841248512,
+          -0.009284568019211292,
+          0.00036589804221875966,
+          0.027490079402923584,
+          -0.024257853627204895,
+          -0.009720918722450733,
+          -0.00028054710128344595,
+          0.0005009949672967196,
+          -0.005680636502802372,
+          -0.010019899345934391,
+          0.004836217500269413,
+          0.013971295207738876,
+          0.017179278656840324,
+          -0.010698666796088219,
+          0.07395332306623459,
+          0.002173671731725335,
+          -0.006787674035876989,
+          0.0013817765284329653,
+          -0.011918832547962666,
+          -0.019910510629415512,
+          0.02086401730775833,
+          -0.022124584764242172,
+          0.013736959546804428,
+          0.007288668770343065,
+          -0.01567629538476467,
+          -0.00758764985948801,
+          -0.015991436317563057,
+          0.024904299527406693,
+          0.0017201501177623868,
+          0.0034867634531110525,
+          0.00689272116869688,
+          0.020734727382659912,
+          0.027296146377921104,
+          0.012274377048015594,
+          0.0041069467552006245,
+          -0.0027978953439742327,
+          -0.004840258043259382,
+          0.016726767644286156,
+          -0.01910245418548584,
+          0.002250437159091234,
+          0.005886691156774759,
+          0.022415485233068466,
+          -0.004969547037035227,
+          0.034261591732501984,
+          0.002882741391658783,
+          0.003143339417874813,
+          -0.006415967829525471,
+          0.003878670744597912,
+          0.020589277148246765,
+          -0.002850419143214822,
+          0.01664596237242222,
+          -0.006367484573274851,
+          -0.0033069709315896034,
+          0.0004800360184162855,
+          0.0010196662042289972,
+          0.002692847978323698,
+          0.0072967493906617165,
+          -0.019344870001077652,
+          -0.0002853449259418994,
+          0.00017448968719691038,
+          0.007264427375048399,
+          0.0029898087959736586,
+          0.0034180786460638046,
+          -0.004375625401735306,
+          0.00046437993296422064,
+          0.006613941863179207,
+          0.010003738105297089,
+          0.013074353337287903,
+          -0.0013696555979549885,
+          -0.005171561148017645,
+          0.006876560393720865,
+          -0.008169450797140598,
+          0.0013272326905280352,
+          0.025793161243200302,
+          0.0013504642993211746,
+          0.0062018330208957195,
+          -0.024871977046132088,
+          0.001361575094051659,
+          0.008686606772243977,
+          0.007595730479806662,
+          0.010674425400793552,
+          4.075003380421549e-05,
+          -0.0029372849967330694,
+          0.0012050141813233495,
+          -0.014019778929650784,
+          0.009632032364606857,
+          0.021785201504826546,
+          -0.0037675630301237106,
+          0.009042151272296906,
+          0.01050473377108574,
+          0.0023494239430874586,
+          -0.029687993228435516,
+          -0.015110654756426811,
+          0.013639992102980614,
+          0.001351474318653345,
+          0.008775493130087852,
+          4.4506232370622456e-05,
+          0.009543146006762981,
+          -0.02214074693620205,
+          -0.011175420135259628,
+          -0.0049089426174759865,
+          0.012807694263756275,
+          0.018229752779006958,
+          -0.021720556542277336,
+          0.013882409781217575,
+          0.006504854187369347,
+          0.004331182222813368,
+          -0.027279984205961227,
+          -0.005454380996525288,
+          0.013187481090426445,
+          -0.01128854788839817,
+          0.002939305268228054,
+          -0.01072290912270546,
+          0.005684677045792341,
+          0.003739281091839075,
+          0.02307809144258499,
+          -0.008840137161314487,
+          0.007761382032185793,
+          0.0037352407816797495,
+          -0.023045768961310387,
+          0.011078453622758389,
+          0.020411504432559013,
+          -0.00525236688554287,
+          -0.0013262225547805429,
+          -0.00577356293797493,
+          -0.00828257855027914,
+          0.015797503292560577,
+          -0.0021433697547763586,
+          -0.011660254560410976,
+          0.01658131740987301,
+          -0.006908882409334183,
+          -0.011369354091584682,
+          -0.022609418258070946,
+          -0.006177591159939766,
+          -0.006658385042101145,
+          -0.0026968882884830236,
+          0.009753241203725338,
+          -0.011442079208791256,
+          0.0034322196152061224,
+          -0.0017565126763656735,
+          -0.0078462278470397,
+          -0.004605921451002359,
+          -0.03358282521367073,
+          -0.018423685804009438,
+          -0.00811692699790001,
+          -0.02012060582637787,
+          0.002057513687759638,
+          0.003706958843395114,
+          -0.00716342031955719,
+          0.017922692000865936,
+          -0.010569377802312374,
+          -0.007575529161840677,
+          0.003414038335904479,
+          0.012912741862237453,
+          -0.00810884591192007,
+          -0.009890610352158546,
+          -0.019651932641863823,
+          0.0003696858184412122,
+          0.01591063104569912,
+          0.0183428805321455,
+          0.02537297084927559,
+          -0.016144966706633568,
+          0.0032948500011116266,
+          -0.024177048355340958,
+          0.008125007152557373,
+          -0.030382921919226646,
+          -0.012751130387187004,
+          -0.004840258043259382,
+          -0.0007378565496765077,
+          -0.008888620883226395,
+          -0.012040040455758572,
+          -0.04124319925904274,
+          -0.009785563684999943,
+          -0.015102574601769447,
+          -0.02322354167699814,
+          0.015134897083044052,
+          -0.005818006116896868,
+          0.017049990594387054,
+          -0.009567388333380222,
+          0.016856057569384575,
+          0.016904540359973907,
+          0.014553096145391464,
+          -0.007341192569583654,
+          0.00732099125161767,
+          -0.004387746565043926,
+          0.014407645910978317,
+          0.004123107995837927,
+          -0.0038928117137402296,
+          0.014973285607993603,
+          0.009995657950639725,
+          0.0050947959534823895,
+          0.00045478425454348326,
+          0.0017191400984302163,
+          0.010941083543002605,
+          0.0007924003293737769,
+          0.015635890886187553,
+          0.007975516840815544,
+          -0.0016837875591591,
+          0.034003015607595444,
+          -0.021785201504826546,
+          -0.010634022764861584,
+          0.010060302913188934,
+          0.01258143875747919,
+          -0.01873074844479561,
+          -0.0017837845953181386,
+          0.010512813925743103,
+          -0.0013201622059568763,
+          -0.030964722856879234,
+          0.013171319849789143,
+          0.011361273005604744,
+          0.0029999094549566507,
+          -0.005114997271448374,
+          -0.016791412606835365,
+          -0.005062473472207785,
+          0.007062413264065981,
+          0.020314538851380348,
+          -0.018245914950966835,
+          0.009939094074070454,
+          -0.007401796989142895,
+          -0.0061452691443264484,
+          -0.02689211815595627,
+          0.012532955035567284,
+          0.022948801517486572,
+          0.009195681661367416,
+          -0.059052761644124985,
+          -0.010375444777309895,
+          0.017858047038316727,
+          -0.0064927334897220135,
+          -0.03755846247076988,
+          -0.002294880338013172,
+          -0.01517529971897602,
+          -0.02307809144258499,
+          0.008767412044107914,
+          0.015732858330011368,
+          -0.006832117214798927,
+          0.015724778175354004,
+          0.006609901320189238,
+          -0.007482602261006832,
+          0.0003903922624886036,
+          -0.004973587114363909,
+          -0.03458481654524803,
+          -0.006678586360067129,
+          0.016548994928598404,
+          -0.03561912849545479,
+          -0.00721190357580781,
+          -0.00827045738697052,
+          -0.014553096145391464,
+          0.006052342709153891,
+          0.011248145252466202,
+          0.03972405195236206,
+          0.007405837066471577,
+          -0.01229861844331026,
+          -0.014003617689013481,
+          -0.014528854750096798,
+          -0.013389495201408863,
+          0.0017100494587793946,
+          -0.01176530122756958,
+          0.011240065097808838,
+          0.008331061340868473,
+          -0.009373454377055168,
+          0.023417474702000618,
+          -0.003403937676921487,
+          0.009559307247400284,
+          -3.494212796795182e-05,
+          -0.004710969049483538,
+          -0.005830127280205488,
+          -0.014722787775099277,
+          0.0008090665214695036,
+          0.003913013264536858,
+          -0.0025191158056259155,
+          -0.0005232165567576885,
+          0.0038221068680286407,
+          0.02706989087164402,
+          -0.010472411289811134,
+          -0.005442259833216667,
+          -0.010286558419466019,
+          -0.028362780809402466,
+          0.0003393836959730834,
+          0.02845974639058113,
+          0.030948560684919357,
+          -0.015684375539422035,
+          -0.01309051364660263,
+          -0.01789036951959133,
+          0.0067351502366364,
+          -0.0038604894652962685,
+          -0.0033392931800335646,
+          0.029397092759609222,
+          0.020977145060896873,
+          0.004137248732149601,
+          0.01732472889125347,
+          -0.009357293136417866,
+          0.006169511005282402,
+          0.028443586081266403,
+          -0.0013302628649398685,
+          0.009939094074070454,
+          0.010149188339710236,
+          0.013963215053081512,
+          -0.00986636895686388,
+          -0.000141536133014597,
+          0.005288729444146156,
+          -0.004646324552595615,
+          0.014084423892199993,
+          0.010666345246136189,
+          -0.008864378556609154,
+          -0.021203400567173958,
+          -0.005066513549536467,
+          -0.020896339789032936,
+          -0.0052483263425529,
+          0.006112946663051844,
+          -0.010836036875844002,
+          -0.001267638523131609,
+          -0.029397092759609222,
+          -0.004383706022053957,
+          0.002311041345819831,
+          -0.013211722485721111,
+          0.01027847733348608,
+          -0.011078453622758389,
+          0.0025251763872802258,
+          0.006290719378739595,
+          -0.02088017761707306,
+          -0.003717059502378106,
+          -0.007377555128186941,
+          0.003652415005490184,
+          0.005325092002749443,
+          0.00614930922165513,
+          -0.0016534854657948017,
+          0.005874569993466139,
+          0.020395344123244286,
+          0.0020787252578884363,
+          0.01486015785485506,
+          -0.02079937234520912,
+          -0.028508231043815613,
+          -0.029073869809508324,
+          0.015611650422215462,
+          -0.0031251581385731697,
+          0.0031837422866374254,
+          -0.0033291925210505724,
+          -0.010844117030501366,
+          0.004828137345612049,
+          0.003913013264536858,
+          0.0014322800561785698,
+          -0.002852439181879163,
+          0.00493318447843194,
+          -0.0031716213561594486,
+          0.010585539042949677,
+          0.006036181468516588,
+          -0.02036302164196968,
+          -0.006024060770869255,
+          -0.009623952209949493,
+          0.0073533132672309875,
+          -0.018375203013420105,
+          -0.009357293136417866,
+          0.00031968732946552336,
+          -0.0018777211662381887,
+          0.0033493938390165567,
+          -0.04812784120440483,
+          -0.019522642716765404,
+          -0.014334920793771744,
+          0.008516915142536163,
+          -0.0019443858182057738,
+          -0.03128794580698013,
+          -0.020039798691868782,
+          0.008133088238537312,
+          0.01580558344721794,
+          0.002410028362646699,
+          0.008920943364501,
+          0.010189591906964779,
+          0.00128480966668576,
+          -0.006436169613152742,
+          -0.0020251914393156767,
+          0.02081553265452385,
+          0.01714695803821087,
+          -0.030593017116189003,
+          -0.019150936976075172,
+          0.010642102919518948,
+          -0.008331061340868473,
+          -0.013615750707685947,
+          -0.01002798043191433,
+          0.007199782878160477,
+          0.0006338192615658045,
+          0.021041790023446083,
+          -0.0015514683909714222,
+          -0.019183259457349777,
+          0.004084725398570299,
+          0.00578568410128355,
+          -0.018892358988523483,
+          0.013219802640378475,
+          -0.007975516840815544,
+          -0.008743170648813248,
+          0.0058988118544220924,
+          -0.007264427375048399,
+          0.0032362660858780146,
+          -0.005183681845664978,
+          0.02580932155251503,
+          -0.020039798691868782,
+          0.0011969335610046983,
+          0.005535186734050512,
+          -0.014092504046857357,
+          0.016969185322523117,
+          -0.008605800569057465,
+          -0.0047594523057341576,
+          0.005013990215957165,
+          0.0007782593602314591,
+          0.009575468488037586,
+          0.009971416555345058,
+          0.008238134905695915,
+          -0.02023373357951641,
+          -0.0012837996473535895,
+          0.005381655879318714,
+          0.007357353810220957,
+          -0.008702768012881279,
+          0.03004353865981102,
+          -0.0054018571972846985,
+          -0.022722546011209488,
+          -0.0009408806799910963,
+          -0.008165409788489342,
+          0.0004906417452730238,
+          -0.00450491439551115,
+          0.007276548072695732,
+          0.015635890886187553,
+          0.006169511005282402,
+          -0.017599469050765038,
+          -0.012831935659050941,
+          0.003323131939396262,
+          0.005147319287061691,
+          0.012767291627824306,
+          0.015167218632996082,
+          -0.011555206961929798,
+          0.010068383067846298,
+          -0.020152926445007324,
+          0.0213165283203125,
+          -0.009753241203725338,
+          0.0038019055500626564,
+          -0.005591750610619783,
+          0.003121117828413844,
+          0.008670445531606674,
+          0.0175186637789011,
+          0.0019575166516005993,
+          -0.0012433967785909772,
+          -0.006133148446679115,
+          0.0350373275578022,
+          -0.00604022154584527,
+          0.016936862841248512,
+          -0.020734727382659912,
+          -0.010060302913188934,
+          -0.024063920602202415,
+          0.03662111610174179,
+          -0.004117047414183617,
+          -0.004589760676026344,
+          0.012007718905806541,
+          0.021930651739239693,
+          -0.012896580621600151,
+          0.009753241203725338,
+          0.003682717215269804,
+          -0.006230114959180355,
+          0.005591750610619783,
+          0.0013767260825261474,
+          0.0008519944967702031,
+          0.02079937234520912,
+          0.017163118347525597,
+          0.008141168393194675,
+          0.007805825211107731,
+          0.011377434246242046,
+          -0.0005098331021144986,
+          -0.012104685418307781,
+          0.01182994619011879,
+          -0.011999637819826603,
+          -0.004094826057553291,
+          -0.0032302055042237043,
+          -0.014415726996958256,
+          -0.009947174228727818,
+          -0.00828257855027914,
+          0.012128926813602448,
+          -0.0035453476011753082,
+          -0.01966809295117855,
+          0.0037251401226967573,
+          0.020266054198145866,
+          0.003032231703400612,
+          0.0059957788325846195,
+          0.014052101410925388,
+          0.01886003650724888,
+          0.01986202597618103,
+          0.0070462520234286785,
+          -0.02941325306892395,
+          -0.0009024980245158076,
+          -0.007070493884384632,
+          -0.006036181468516588,
+          0.009219923987984657,
+          0.0011949134059250355,
+          0.00810884591192007,
+          -0.0035776698496192694,
+          0.012799614109098911,
+          0.03713827207684517,
+          0.013583428226411343,
+          -0.0011353192385286093,
+          0.002395887393504381,
+          -0.002739311195909977,
+          -0.01986202597618103,
+          0.018116625025868416,
+          -0.01623385399580002,
+          -0.027183018624782562,
+          0.021268045529723167,
+          -0.0002911528281401843,
+          -0.006185671780258417,
+          -0.021962974220514297,
+          -0.006884640548378229,
+          0.016080323606729507,
+          0.018116625025868416,
+          -0.0008767411927692592,
+          -0.0018676203908398747,
+          -0.0033190918620675802,
+          0.0031473797280341387,
+          -0.006161430384963751,
+          0.005583669990301132,
+          -0.0020221611484885216,
+          -0.023708375170826912,
+          -0.006815955974161625,
+          -0.011296628974378109,
+          0.006803835276514292,
+          0.0033069709315896034,
+          0.013696555979549885,
+          -0.004117047414183617,
+          0.020346861332654953,
+          0.005325092002749443,
+          -0.004234215710312128,
+          0.007013929542154074,
+          -0.009858288802206516,
+          -0.0030504129827022552,
+          -0.01997515559196472,
+          0.015223783440887928,
+          0.0055028642527759075,
+          0.0032847493421286345,
+          -0.001361575094051659,
+          0.02910619229078293,
+          -0.0005600841250270605,
+          0.006864439230412245,
+          -0.003783724270761013,
+          0.011248145252466202,
+          -0.002197913359850645,
+          -0.008751250803470612,
+          -0.03183742240071297,
+          -0.01039968617260456,
+          -0.02682747319340706,
+          0.023320509120821953,
+          -0.009219923987984657,
+          0.0012353162746876478,
+          0.008589639328420162,
+          0.003672616556286812,
+          -0.004315021447837353,
+          0.010924923233687878,
+          -0.013591509312391281,
+          -0.004234215710312128,
+          0.012896580621600151,
+          -0.0004992273752577603,
+          0.009720918722450733,
+          -0.0073937163688242435,
+          -0.003852409077808261,
+          -0.011369354091584682,
+          -0.008500753901898861,
+          -0.03010818175971508,
+          -0.013308688998222351,
+          -0.01452077366411686,
+          -0.013551106676459312,
+          -0.024500271305441856,
+          -0.0008459340897388756,
+          0.004197853151708841,
+          -0.003761502681300044,
+          -0.010900680907070637,
+          0.017486341297626495,
+          0.01416522916406393,
+          0.02770017459988594,
+          -0.026714345440268517,
+          0.0006358394166454673,
+          -0.007648254279047251,
+          0.018520653247833252,
+          -0.01072290912270546,
+          0.0050947959534823895,
+          0.0015726798446848989,
+          0.01677525043487549,
+          0.0033837363589555025,
+          0.0012464270694181323,
+          0.004517035558819771,
+          0.0018858016701415181,
+          -0.01027847733348608,
+          0.009179521352052689,
+          0.0008439139346592128,
+          -0.011644093319773674,
+          -0.003921093884855509,
+          0.016872217878699303,
+          0.0201690886169672,
+          0.01293698325753212,
+          -0.007219984196126461,
+          0.0034403002355247736,
+          -0.009381535463035107,
+          -0.011846107430756092,
+          -0.011789543554186821,
+          0.008314901031553745,
+          -0.025259843096137047,
+          0.01814894750714302,
+          0.008937103673815727,
+          -0.0008227024227380753,
+          0.007785623427480459,
+          0.0023514442145824432,
+          0.015967195853590965,
+          0.005523065570741892,
+          -0.019393354654312134,
+          -0.014448048546910286,
+          0.0027978953439742327,
+          0.012104685418307781,
+          0.0010252215433865786,
+          -0.0023817464243620634,
+          0.00710281589999795,
+          0.0002638809382915497,
+          -0.011474400758743286,
+          -0.017534824088215828,
+          0.0003237276105210185,
+          0.00938961561769247,
+          0.02290031872689724,
+          0.005757402162998915,
+          0.013446059077978134,
+          0.0022282155696302652,
+          -0.0057493215426802635,
+          -0.007143219001591206,
+          -0.0161045640707016,
+          0.015959113836288452,
+          0.0038928117137402296,
+          -0.025453777983784676,
+          0.024888137355446815,
+          0.002057513687759638,
+          0.01688838005065918,
+          -0.004686727188527584,
+          -0.005838207434862852,
+          0.023756857961416245,
+          0.027780979871749878,
+          0.0043109809048473835,
+          0.024839654564857483,
+          -0.013494541868567467,
+          0.0072523062117397785,
+          0.02516287751495838,
+          0.020459989085793495,
+          -0.009688596241176128,
+          0.0012908701319247484,
+          0.011967315338551998,
+          0.001570659689605236,
+          -0.0005828107241541147,
+          0.009971416555345058,
+          -0.010593620128929615,
+          0.010690586641430855,
+          -0.02175287902355194,
+          0.017437858507037163,
+          -0.0014777331380173564,
+          -0.005183681845664978,
+          -0.006395766511559486,
+          0.0050947959534823895,
+          0.006011939607560635,
+          0.011102695018053055,
+          -0.009203762747347355,
+          -0.021542783826589584,
+          -0.0015292468015104532,
+          -0.003304950660094619,
+          -0.012395585887134075,
+          -0.028120363131165504,
+          0.012177410535514355,
+          -0.013575348071753979,
+          -0.008112886920571327,
+          0.009882530197501183,
+          -0.01721160113811493,
+          0.00241204840131104,
+          -0.02335282973945141,
+          0.010108785703778267,
+          -0.0216397512704134,
+          0.0052564069628715515,
+          0.011854187585413456,
+          -0.005846288055181503,
+          0.004282698966562748,
+          0.006165470462292433,
+          0.010415847413241863,
+          8.724484359845519e-05,
+          -0.013195561245083809,
+          -0.007450280245393515,
+          0.028605196624994278,
+          0.006290719378739595,
+          -0.003737261053174734,
+          -0.01821359246969223,
+          -0.0008454290218651295,
+          -0.028492068871855736,
+          0.018310558050870895,
+          -0.00041185625013895333,
+          0.0008888620650395751,
+          -0.019651932641863823,
+          -0.011636012233793736,
+          -0.0008120966958813369,
+          0.014391484670341015,
+          -0.00019607994181569666,
+          0.003793824929744005,
+          0.0317404568195343,
+          -0.013599589467048645,
+          0.001599951763637364,
+          -6.699608547933167e-06,
+          -0.01303394977003336,
+          0.02005596086382866,
+          -0.005757402162998915,
+          0.006925043649971485,
+          -0.0007333112298510969,
+          -0.008165409788489342,
+          -0.014666223898530006,
+          0.022108424454927444,
+          -0.0033190918620675802,
+          -0.012258215807378292,
+          -0.006763432174921036,
+          0.002161551034078002,
+          0.013551106676459312,
+          -0.012379424646496773,
+          -0.008621961809694767,
+          0.010924923233687878,
+          0.00488470122218132,
+          -0.011110776104032993,
+          -0.010044141672551632,
+          -0.013478381559252739,
+          0.0037433214019984007,
+          -0.03494035825133324,
+          -0.006929083727300167,
+          0.04586528241634369,
+          0.0056038713082671165,
+          -0.008395706303417683,
+          -0.0058503285981714725,
+          0.0009530015522614121,
+          0.006973526906222105,
+          -0.01847217045724392,
+          -0.007619971875101328,
+          -0.027780979871749878,
+          0.005983657669275999,
+          0.002139329444617033,
+          0.03243538364768028,
+          0.0003406462783459574,
+          -0.018488330766558647,
+          0.003549387911334634,
+          -0.021300368010997772,
+          -0.0074421996250748634,
+          -0.018391363322734833,
+          -0.016209611669182777,
+          -0.008920943364501,
+          0.016904540359973907,
+          -0.015167218632996082,
+          0.02663354016840458,
+          0.016597479581832886,
+          0.007603811100125313,
+          0.003117077751085162,
+          -0.0037998852785676718,
+          -0.006415967829525471,
+          0.006432129070162773,
+          -0.0019383253529667854,
+          0.004953385796397924,
+          0.011296628974378109,
+          -0.01075523067265749,
+          0.009922932833433151,
+          0.013688475824892521,
+          -0.007624012418091297,
+          0.01840752549469471,
+          -0.00837954506278038,
+          -0.020459989085793495,
+          0.01452077366411686,
+          -0.014706626534461975,
+          0.00152621662709862,
+          0.0037574623711407185,
+          -0.00832298118621111,
+          0.00472713029012084,
+          0.018989326432347298,
+          0.0073977564461529255,
+          -0.00572507968172431,
+          0.017373213544487953,
+          0.020670082420110703,
+          0.012629921548068523,
+          -0.01050473377108574,
+          0.009591629728674889,
+          0.014439968392252922,
+          -0.0001660303387325257,
+          -0.0003701908572111279,
+          0.004407947883009911,
+          -0.014407645910978317,
+          -0.0006307890289463103,
+          0.003212024224922061,
+          -0.011094614863395691,
+          -0.0115713682025671,
+          0.016839895397424698,
+          0.005244286265224218,
+          -0.004517035558819771,
+          0.011700657196342945,
+          -0.01284001674503088,
+          -0.012751130387187004,
+          -0.009825966320931911,
+          0.0066987876780331135,
+          -0.040758367627859116,
+          0.004840258043259382,
+          -0.006464451551437378,
+          0.018100464716553688,
+          -0.014456129632890224,
+          0.005236205644905567,
+          -0.005919013172388077,
+          -0.010949164628982544,
+          0.015070252120494843,
+          -0.0009762331610545516,
+          -0.002218114910647273,
+          0.03188590705394745,
+          0.011934993788599968,
+          0.01077947299927473,
+          0.004646324552595615,
+          -0.016371222212910652,
+          0.034261591732501984,
+          -0.009882530197501183,
+          0.00012966780923306942,
+          -0.012856177985668182,
+          -0.0017342910869047046,
+          0.03186974674463272,
+          0.029332447797060013,
+          -0.005046312231570482,
+          0.004480673000216484,
+          -0.00795531552284956,
+          0.01363191194832325,
+          0.015991436317563057,
+          0.008133088238537312,
+          0.00599173828959465,
+          0.0032605077140033245,
+          0.004383706022053957,
+          0.0011009768350049853,
+          0.038495808839797974,
+          -0.012613760307431221,
+          -0.011943073943257332,
+          -0.013955134898424149,
+          -0.016662122681736946,
+          -0.022738708183169365,
+          0.02138117328286171,
+          -0.00747856218367815,
+          -0.014916721731424332,
+          -0.007769462652504444,
+          0.018989326432347298,
+          0.00705433264374733,
+          0.006714948918670416,
+          -0.0012928902870044112,
+          0.005539226811379194,
+          -0.024451786652207375,
+          -0.00499378889799118,
+          -0.01738937385380268,
+          0.002729210536926985,
+          -0.0007913902518339455,
+          0.015951033681631088,
+          0.018456008285284042,
+          0.017437858507037163,
+          0.019005486741662025,
+          -0.007426038384437561,
+          0.006169511005282402,
+          -0.0055069043301045895,
+          -0.01226629689335823,
+          0.019199419766664505,
+          0.02086401730775833,
+          -0.01404402032494545,
+          -0.004274618346244097,
+          -0.016710607334971428,
+          -0.021171078085899353,
+          0.002983748447149992,
+          0.009543146006762981,
+          0.010884519666433334,
+          -0.013219802640378475,
+          0.002816076623275876,
+          0.015264186076819897,
+          -0.007042211946099997,
+          0.002769613405689597,
+          0.0048483386635780334,
+          0.017373213544487953,
+          0.01224205456674099,
+          0.0031675812788307667,
+          0.01803581975400448,
+          -0.018957003951072693,
+          -0.02776481956243515,
+          0.006589700002223253,
+          -0.014650062657892704,
+          0.017502501606941223,
+          0.009025990031659603,
+          0.005672555882483721,
+          -0.03149804100394249,
+          -0.010827955789864063,
+          0.01814894750714302,
+          -0.00046715763164684176,
+          -0.001248447224497795,
+          0.001733281067572534,
+          0.01077947299927473,
+          -0.009228004142642021,
+          -0.012670325115323067,
+          -0.01886003650724888,
+          0.0010595639469102025,
+          -0.0051513598300516605,
+          -0.0030584936030209064,
+          -0.01293698325753212,
+          -0.00844419002532959,
+          -0.013486461713910103,
+          -0.004420068580657244,
+          0.0008504794095642865,
+          0.0234659593552351,
+          0.01871458627283573,
+          0.01936103217303753,
+          0.002983748447149992,
+          0.004500874318182468,
+          0.017744919285178185,
+          0.013025869615375996,
+          0.007922993041574955,
+          0.00160500209312886,
+          -0.0019474159926176071,
+          0.028572876006364822,
+          -0.019199419766664505,
+          -0.012031960301101208,
+          0.018569136038422585,
+          -0.01331677008420229,
+          0.017761079594492912,
+          -0.02118724025785923,
+          0.001272688852623105,
+          0.005692757666110992,
+          0.003387776669114828,
+          -0.009809805080294609,
+          -0.008686606772243977,
+          0.019247904419898987,
+          0.009122957475483418,
+          -0.009413857012987137,
+          0.01979738287627697,
+          0.005769522860646248,
+          -0.006658385042101145,
+          -0.009696677327156067,
+          -0.0019383253529667854,
+          -0.0041089667938649654,
+          -0.015732858330011368,
+          0.016856057569384575,
+          0.018132785335183144,
+          -0.015353072434663773,
+          0.0042665377259254456,
+          0.01879539154469967,
+          -0.01973273791372776,
+          0.011248145252466202,
+          0.01378544233739376,
+          -0.010092624463140965,
+          0.007809865288436413,
+          -0.0010186561848968267,
+          0.013446059077978134,
+          0.003761502681300044,
+          -0.004145329352468252,
+          -0.0005929113831371069,
+          0.011967315338551998,
+          -0.017243923619389534,
+          -0.012112765572965145,
+          -0.012427907437086105,
+          -0.008157329633831978,
+          0.007123017217963934,
+          0.002535277046263218,
+          -0.0056483144871890545,
+          0.009025990031659603,
+          -0.011337031610310078,
+          -0.015991436317563057,
+          -0.028378941118717194,
+          0.0005499834078364074,
+          -0.00971283856779337,
+          -0.001211074530147016,
+          0.014698546379804611,
+          0.015749020501971245,
+          -0.0009449209901504219,
+          -0.016969185322523117,
+          0.01816510781645775,
+          0.04408755898475647,
+          -0.007809865288436413,
+          0.003777663689106703,
+          0.010601700283586979,
+          0.002488813828676939,
+          -0.0026302237529307604,
+          -0.0022908400278538465,
+          -0.016427787020802498,
+          -0.03765542805194855,
+          -0.012912741862237453,
+          0.01077947299927473,
+          -0.0012787493178620934,
+          -0.004638243932276964,
+          -0.021914489567279816,
+          -0.005842247977852821,
+          -0.016201531514525414,
+          -0.004921063780784607,
+          -0.019651932641863823,
+          -0.004799854941666126,
+          0.013979376293718815,
+          -0.015959113836288452,
+          0.019878188148140907,
+          -0.02024989388883114,
+          0.020540794357657433,
+          0.01385008729994297,
+          0.02277103066444397,
+          -0.01641162671148777,
+          0.009341131895780563,
+          -0.0014120786217972636,
+          0.014666223898530006,
+          0.01034312229603529,
+          0.007308870553970337,
+          -0.010617861524224281,
+          -0.006634143181145191,
+          0.01928022690117359,
+          0.009470420889556408,
+          0.013082433491945267,
+          0.009567388333380222,
+          -0.002262558089569211,
+          -0.0073371524922549725,
+          -0.014464209787547588,
+          -0.022156907245516777,
+          0.003163540968671441,
+          -0.0021676113829016685,
+          0.01688838005065918,
+          -0.020379183813929558,
+          -0.021607428789138794,
+          0.0002142612065654248,
+          0.014722787775099277,
+          0.035069648176431656,
+          -0.008573479019105434,
+          0.004371585324406624,
+          -0.012330940924584866,
+          0.01202387921512127,
+          0.018310558050870895,
+          0.005236205644905567,
+          0.0012221853248775005,
+          -0.020346861332654953,
+          0.011579448357224464,
+          0.011086533777415752,
+          -0.002422149060294032,
+          -0.0035089850425720215,
+          -0.0029029427096247673,
+          -0.02290031872689724,
+          0.0064078872092068195,
+          0.004399867262691259,
+          0.038043297827243805,
+          0.013761200942099094,
+          -0.0014807634288445115,
+          0.0072523062117397785,
+          0.029073869809508324,
+          -0.0032362660858780146,
+          0.009276487864553928,
+          0.0007363414042629302,
+          -0.003854429116472602,
+          0.008060363121330738,
+          -0.00811692699790001,
+          -0.0459299273788929,
+          0.010577458888292313,
+          -0.004492793697863817,
+          0.038043297827243805,
+          -0.0032746486831456423,
+          0.03043140470981598,
+          -0.006533136125653982,
+          -0.00810884591192007,
+          -0.005531146191060543,
+          0.017373213544487953,
+          -0.012508713640272617,
+          0.012258215807378292,
+          -0.008044201880693436,
+          -0.010625941678881645,
+          -0.0016342941671609879,
+          0.006779593415558338,
+          -0.012371343560516834,
+          -0.007333111949265003,
+          0.022156907245516777,
+          0.024387143552303314,
+          0.013761200942099094,
+          -0.009834046475589275,
+          0.0315626822412014,
+          -0.007510884664952755,
+          -0.014472290873527527,
+          0.019005486741662025,
+          -0.0037109991535544395,
+          -0.006428088992834091,
+          -0.001322182361036539,
+          -0.0023029607255011797,
+          -0.0024322497192770243,
+          0.017357051372528076,
+          0.0175186637789011,
+          -0.00429886020720005,
+          0.0015373273054137826,
+          0.00429886020720005,
+          -0.002468612277880311,
+          0.019150936976075172,
+          0.011749139986932278,
+          -0.03636253997683525,
+          0.014221793040633202,
+          -0.029364770278334618,
+          -0.01821359246969223,
+          -0.007579569239169359,
+          -0.004157450515776873,
+          0.011708737351000309,
+          -0.02390230819582939,
+          -0.011490561999380589,
+          -0.023417474702000618,
+          0.009753241203725338,
+          0.016969185322523117,
+          0.0065290960483253,
+          0.012137007899582386,
+          0.014302598312497139,
+          0.0012252154992893338,
+          -0.017421696335077286,
+          -0.013680395670235157,
+          -0.02765168994665146,
+          -0.014706626534461975,
+          0.012185490690171719,
+          0.0037493817508220673,
+          -0.013446059077978134,
+          -0.0023453838657587767,
+          0.026649700477719307,
+          -0.0037918046582490206,
+          -0.00313121872022748,
+          0.009753241203725338,
+          0.003694837912917137,
+          0.005220044404268265,
+          -0.005886691156774759,
+          0.014246034435927868,
+          0.00716342031955719,
+          0.01012494694441557,
+          -0.02984960377216339,
+          0.01738937385380268,
+          -0.003737261053174734,
+          -0.0008519944967702031,
+          0.0013363233301788568,
+          -0.004262497648596764,
+          -0.011660254560410976,
+          -0.0017757039749994874,
+          -0.02548610046505928,
+          -0.00689272116869688,
+          -0.0031918229069560766,
+          -0.011716818436980247,
+          -0.002658505691215396,
+          -0.00741795776411891,
+          -0.016807572916150093,
+          -0.010593620128929615,
+          0.01847217045724392,
+          -0.011062292382121086,
+          0.01293698325753212,
+          0.0011060272809118032,
+          -0.017550986260175705,
+          0.007361393887549639,
+          -0.012654163874685764,
+          0.0016403545159846544,
+          -0.009591629728674889,
+          0.0001431775017408654,
+          0.019134776666760445,
+          -0.008767412044107914,
+          0.006581619381904602,
+          -0.004278658889234066,
+          0.011151178739964962,
+          -0.00102168635930866,
+          0.016710607334971428,
+          0.015951033681631088,
+          0.011046131141483784,
+          0.00010163834667764604,
+          -0.015732858330011368,
+          -0.019053971394896507,
+          0.022108424454927444,
+          0.004214014392346144,
+          -0.01046433113515377,
+          0.02404775843024254,
+          0.002684767358005047,
+          0.01992667093873024,
+          -0.011652173474431038,
+          0.003864529775455594,
+          0.024516431614756584,
+          0.010060302913188934,
+          0.008993667550384998,
+          -0.0035776698496192694,
+          -0.0003694332845043391,
+          0.00482409680262208,
+          0.03416462615132332,
+          0.019247904419898987,
+          -0.005927093792706728,
+          -0.024306336417794228,
+          0.018262075260281563,
+          0.0019191340543329716,
+          0.001435310230590403,
+          0.02024989388883114,
+          0.024322498589754105,
+          0.01808430254459381,
+          0.007143219001591206,
+          0.008654284290969372,
+          -0.012452149763703346,
+          0.010747150518000126,
+          -0.0007651284104213119,
+          -0.0055473074316978455,
+          0.0003335757937747985,
+          -0.029591025784611702,
+          0.012258215807378292,
+          -0.00270294863730669,
+          -0.014229873195290565,
+          0.0021817523520439863,
+          -0.013130917213857174,
+          0.022867996245622635,
+          -0.0011575408279895782,
+          -0.01645202934741974,
+          -0.004694807808846235,
+          -0.024063920602202415,
+          0.0028201169334352016,
+          -0.01923174224793911,
+          -0.005369534716010094,
+          -0.012015799060463905,
+          -0.0013918771874159575,
+          -0.022237712517380714,
+          -0.009591629728674889,
+          0.011539045721292496,
+          -0.002353464253246784,
+          -0.007450280245393515,
+          -0.005914973095059395,
+          0.00763613311573863,
+          -0.018569136038422585,
+          -0.0028423385228961706,
+          0.005716999061405659,
+          0.019150936976075172,
+          0.0025191158056259155,
+          0.003535246942192316,
+          0.01871458627283573,
+          0.013963215053081512,
+          -0.03445552662014961,
+          0.003650394966825843,
+          0.00254537770524621,
+          0.00223023584112525,
+          0.025195199996232986,
+          0.007462400943040848,
+          -0.00732099125161767,
+          0.014795512892305851,
+          -0.0003237276105210185,
+          0.01871458627283573,
+          0.002375685842707753,
+          -0.014682385139167309,
+          -0.00716342031955719,
+          0.005672555882483721,
+          -0.007737140171229839,
+          0.021106433123350143,
+          -0.002543357666581869,
+          0.007813905365765095,
+          -0.009995657950639725,
+          -0.011078453622758389,
+          -0.00678363349288702,
+          0.00013863218191545457,
+          -0.006011939607560635,
+          0.00885629840195179,
+          -0.009809805080294609,
+          0.005813966039568186,
+          0.006133148446679115,
+          0.004022100940346718,
+          -0.013938973657786846,
+          0.016056081280112267,
+          -0.003199903527274728,
+          0.008573479019105434,
+          0.02605173923075199,
+          0.00572507968172431,
+          -0.009438099339604378,
+          0.017550986260175705,
+          -8.59191277413629e-05,
+          -0.01128854788839817,
+          -0.02244780771434307,
+          -0.003878670744597912,
+          -0.03907760977745056,
+          0.028378941118717194,
+          0.00023193744709715247,
+          -0.018520653247833252,
+          0.02517903782427311,
+          -0.00816136971116066,
+          -0.008274497464299202,
+          -0.00018143391935154796,
+          -0.0100845443084836,
+          -0.02340131439268589,
+          0.002715069567784667,
+          -0.010569377802312374,
+          -0.006108906585723162,
+          0.011934993788599968,
+          0.030334437265992165,
+          -0.008815895766019821,
+          -0.0008913872297853231,
+          0.006937164347618818,
+          0.003607971826568246,
+          0.01879539154469967,
+          -0.028071880340576172,
+          -0.0032948500011116266,
+          -0.018326720222830772,
+          0.019199419766664505,
+          -0.0007105846307240427,
+          -0.01682373508810997,
+          0.012339022010564804,
+          -0.014949044212698936,
+          -0.02175287902355194,
+          -0.0003398887347429991,
+          -0.0060563827864825726,
+          0.005664475727826357,
+          0.0063594039529562,
+          0.02498510479927063,
+          0.0033756557386368513,
+          0.007963395677506924,
+          -0.02580932155251503,
+          -0.023708375170826912,
+          0.02383766509592533,
+          0.01103805098682642,
+          -0.014932882972061634,
+          -0.0011555206729099154,
+          -0.005106916651129723,
+          -0.007219984196126461,
+          -0.019910510629415512,
+          0.007175541017204523,
+          0.06651920825242996,
+          -0.007361393887549639,
+          0.003096876200288534,
+          -0.015643972903490067,
+          -0.005632153246551752,
+          -0.007098775822669268,
+          -0.01929638721048832,
+          -0.006363444495946169,
+          0.00024822482373565435,
+          0.005733160302042961,
+          -0.025405293330550194,
+          0.004054422955960035,
+          -0.006706868298351765,
+          0.007001808844506741,
+          -0.012137007899582386,
+          0.010270397178828716,
+          -0.015490441583096981,
+          -0.006860399153083563,
+          0.0028686001896858215,
+          -0.009817885234951973,
+          -0.0009388605831190944,
+          -0.017243923619389534,
+          0.001279759337194264,
+          0.016112644225358963,
+          0.003616052446886897,
+          -0.012767291627824306,
+          -0.016460109502077103,
+          -0.01947415992617607,
+          -0.01214508805423975,
+          -0.005070554092526436,
+          -0.008339142426848412,
+          -0.023320509120821953,
+          0.012169329449534416,
+          -0.007741180714219809,
+          0.016856057569384575,
+          0.011054212227463722,
+          0.013429897837340832,
+          0.020573116838932037,
+          -0.003389796707779169,
+          0.009308810345828533,
+          -0.013421816751360893,
+          0.03067382238805294,
+          -0.0022787190973758698,
+          -0.023449797183275223,
+          -0.0007312910747714341,
+          0.00033913119114004076,
+          -0.007931074127554893,
+          -0.021849846467375755,
+          0.007555327378213406,
+          -0.002583760302513838,
+          0.011934993788599968,
+          -0.005959416273981333,
+          0.003995839040726423,
+          0.0015686395345255733,
+          0.006294759456068277,
+          0.004343303386121988,
+          0.017841886729002,
+          -0.0059957788325846195,
+          0.01997515559196472,
+          -0.02815268561244011,
+          -0.01738937385380268,
+          -0.008230054751038551,
+          -0.007490682881325483,
+          0.012613760307431221,
+          0.006044262088835239,
+          -0.0032685883343219757,
+          -0.016492431983351707,
+          -0.010076463222503662,
+          0.008880539797246456,
+          -0.016807572916150093,
+          -0.026601217687129974,
+          -0.0016080322675406933,
+          -0.0325646735727787,
+          0.007110896520316601,
+          0.004214014392346144,
+          -0.005700838286429644,
+          -0.005575589369982481,
+          0.02010444365441799,
+          0.00948658213019371,
+          0.006646263878792524,
+          0.005692757666110992,
+          -0.0017070191679522395,
+          -0.011312790215015411,
+          -0.0009595670271664858,
+          0.0015565187204629183,
+          0.011749139986932278,
+          -0.002377706114202738,
+          -0.01571669802069664,
+          0.015353072434663773,
+          0.009034071117639542,
+          -0.0108198756352067,
+          0.0004469561972655356,
+          0.019700415432453156,
+          0.0039574564434587955,
+          -0.0180196575820446,
+          0.004597840830683708,
+          0.018197430297732353,
+          -0.005979617591947317,
+          0.013163238763809204,
+          0.026552734896540642,
+          -1.8986169379786588e-05,
+          -0.00078280468005687,
+          -0.017599469050765038,
+          -0.01255719643086195,
+          -0.0035574682988226414,
+          0.023611409589648247,
+          0.029251642525196075,
+          0.002240336500108242,
+          -0.027344629168510437,
+          0.009567388333380222,
+          -0.016969185322523117,
+          0.012961224652826786,
+          -0.015231863595545292,
+          0.0021474098321050406,
+          0.02092866227030754,
+          -0.013761200942099094,
+          0.004989748355001211,
+          0.009898691438138485,
+          0.028750646859407425,
+          0.004561478737741709,
+          -0.015264186076819897,
+          0.0006994738359935582,
+          -0.02081553265452385,
+          0.002008020179346204,
+          -0.006395766511559486,
+          0.005733160302042961,
+          -0.011369354091584682,
+          0.00646849162876606,
+          -0.011110776104032993,
+          0.0006439199787564576,
+          0.0012141048209741712,
+          -0.02118724025785923,
+          0.007583609316498041,
+          -0.018003497272729874,
+          0.0037453414406627417,
+          0.0010767352068796754,
+          -0.004048362839967012,
+          -0.023643730208277702,
+          -0.002511035418137908,
+          0.028896097093820572,
+          -0.007805825211107731,
+          0.05788916349411011,
+          -0.006290719378739595,
+          0.0014403605600818992,
+          -0.009034071117639542,
+          0.0063068801537156105,
+          -0.0009706777636893094,
+          -0.009349212981760502,
+          -0.005413977894932032,
+          0.002723150188103318,
+          0.0029130433686077595,
+          0.004492793697863817,
+          0.009914852678775787,
+          0.004836217500269413,
+          0.0005153884994797409,
+          0.02984960377216339,
+          0.0008681556209921837,
+          0.0085977204144001,
+          -0.004484713077545166,
+          0.0022908400278538465,
+          -0.016524754464626312,
+          0.01803581975400448,
+          0.005801845341920853,
+          -0.022609418258070946,
+          -0.016662122681736946,
+          -0.017049990594387054,
+          0.013462220318615437,
+          -0.0031009165104478598,
+          0.013898570090532303,
+          -0.005846288055181503,
+          -0.014553096145391464,
+          0.017664114013314247,
+          0.008557317778468132,
+          0.009405776858329773,
+          -0.0002760017814580351,
+          0.005874569993466139,
+          0.005462461616843939,
+          0.008823975920677185,
+          -0.005102876108139753,
+          -0.005317011382430792,
+          -0.0026766869705170393,
+          0.014981365762650967,
+          -0.011029969900846481,
+          -0.0011454200139269233,
+          -0.016112644225358963,
+          -0.006791714113205671,
+          0.022932641208171844,
+          0.000245573406573385,
+          -0.003708978882059455,
+          0.02650425024330616,
+          0.011692576110363007,
+          -0.00010833006672328338,
+          -0.009696677327156067,
+          -0.0013646052684634924,
+          0.007725019473582506,
+          0.020718567073345184,
+          -0.004028161056339741,
+          -0.008500753901898861,
+          -0.017615629360079765,
+          -0.0011454200139269233,
+          0.030059698969125748,
+          -0.007535126060247421,
+          -0.019894348457455635,
+          0.01284001674503088
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "1ea6fc21-365a-41fe-9b07-1e5411ca0bc9",
+      "properties": {
+        "page_content": "4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n",
+        "title": "Validation Experiments",
+        "embedding": [
+          -0.010754532180726528,
+          -0.023697586730122566,
+          -0.02424471639096737,
+          -0.004748920910060406,
+          0.0029044931288808584,
+          0.014695581048727036,
+          -0.013233717530965805,
+          0.03884626179933548,
+          -0.0167387705296278,
+          0.032195206731557846,
+          -0.028296900913119316,
+          0.003022040706127882,
+          0.033374954015016556,
+          -0.032622650265693665,
+          -0.007941939868032932,
+          0.013165325857698917,
+          0.035289913415908813,
+          0.03330656513571739,
+          0.011327309533953667,
+          -0.037751998752355576,
+          0.04247099906206131,
+          -0.036520954221487045,
+          -0.060047563165426254,
+          -0.011079391464591026,
+          -0.019577009603381157,
+          0.02089354395866394,
+          0.00764272827655077,
+          0.016584891825914383,
+          -0.0636381059885025,
+          0.03330656513571739,
+          0.00491989916190505,
+          0.023150455206632614,
+          0.0002941360289696604,
+          0.008754086680710316,
+          -0.011498288251459599,
+          -0.03205842152237892,
+          0.022381054237484932,
+          -0.0009312971960753202,
+          -0.015362396836280823,
+          0.042607780545949936,
+          0.03381950035691261,
+          0.01822628267109394,
+          -0.023372728377580643,
+          -0.004340710584074259,
+          0.016832809895277023,
+          0.00703148078173399,
+          -0.013173874467611313,
+          -0.010062070563435555,
+          0.015704352408647537,
+          -0.04240260645747185,
+          -0.03546088933944702,
+          -0.011600874364376068,
+          0.008172760717570782,
+          -0.01200267393141985,
+          -0.03761521726846695,
+          0.007856450974941254,
+          0.016140347346663475,
+          0.026159673929214478,
+          0.04093219339847565,
+          0.015439336188137531,
+          0.027305228635668755,
+          -0.018602434545755386,
+          -0.019765086472034454,
+          0.010164657607674599,
+          0.0010231980122625828,
+          -0.03443502262234688,
+          0.0016926847165450454,
+          -0.011720559559762478,
+          0.017046531662344933,
+          -0.034383729100227356,
+          -0.04520665109157562,
+          0.02380017377436161,
+          -0.009292668662965298,
+          -0.010993901640176773,
+          -0.0009270227164961398,
+          -0.010198852978646755,
+          0.031203532591462135,
+          0.027544597163796425,
+          -0.0026907704304903746,
+          0.000909390626475215,
+          0.014806716702878475,
+          0.02405664138495922,
+          -0.023714683949947357,
+          0.0028040434699505568,
+          0.04373623803257942,
+          -0.0030434129294008017,
+          -0.02814302034676075,
+          -0.06527949869632721,
+          0.06415104120969772,
+          -0.0167815163731575,
+          -0.013644064776599407,
+          0.0002978761913254857,
+          0.023646293208003044,
+          -0.019577009603381157,
+          -0.029237281531095505,
+          0.05844036862254143,
+          -0.007394809741526842,
+          0.0007074225577525795,
+          -0.018619531765580177,
+          0.014601543545722961,
+          -0.009771407581865788,
+          0.010865667834877968,
+          -0.008617304265499115,
+          0.014422016218304634,
+          0.032622650265693665,
+          -0.024877335876226425,
+          -0.014447662979364395,
+          0.012831917963922024,
+          0.013712456449866295,
+          0.10628008842468262,
+          -0.016217287629842758,
+          -0.005920122377574444,
+          -0.03257135674357414,
+          0.0011006725253537297,
+          -0.009745760820806026,
+          -0.05006243288516998,
+          -0.026826487854123116,
+          -0.025373173877596855,
+          0.031237727031111717,
+          -0.008035978302359581,
+          0.033836595714092255,
+          0.032434575259685516,
+          0.045035671442747116,
+          -0.0132422661408782,
+          -0.04804489016532898,
+          0.046950630843639374,
+          -0.0051549943163990974,
+          0.000823367154225707,
+          -0.0025347527116537094,
+          -0.003041275776922703,
+          -0.0017194000538438559,
+          -0.007373437285423279,
+          -0.01822628267109394,
+          -0.01443056482821703,
+          0.011831695213913918,
+          0.0043449848890304565,
+          0.034298237413167953,
+          0.053003259003162384,
+          -0.04373623803257942,
+          -0.009685918688774109,
+          0.026792293414473534,
+          -0.029938293620944023,
+          0.010985353030264378,
+          0.006091100629419088,
+          -0.02403954230248928,
+          0.015011890791356564,
+          0.0010638053063303232,
+          0.055465348064899445,
+          0.0003606572572607547,
+          0.02374888025224209,
+          0.0011113586369901896,
+          0.04274456575512886,
+          -0.008360836654901505,
+          0.04363365098834038,
+          0.010592103004455566,
+          0.0010258695110678673,
+          -0.020927738398313522,
+          0.018602434545755386,
+          -0.039427585899829865,
+          -0.000900841667316854,
+          0.03171646595001221,
+          -0.0211500097066164,
+          -0.009899641387164593,
+          0.012660940177738667,
+          0.004398415796458721,
+          0.06869906187057495,
+          -0.01395182590931654,
+          -0.0074888477101922035,
+          -0.0781712606549263,
+          0.04982306435704231,
+          0.023971151560544968,
+          0.00814711395651102,
+          0.005907298997044563,
+          0.04182128235697746,
+          0.003832050133496523,
+          0.01309693418443203,
+          0.012438667938113213,
+          -0.0098397983238101,
+          -0.05026760697364807,
+          0.03207552060484886,
+          -0.0026587119791656733,
+          0.012908858247101307,
+          -0.009130238555371761,
+          0.026860684156417847,
+          0.05505499988794327,
+          0.022021999582648277,
+          0.01165216788649559,
+          0.027476206421852112,
+          -0.020671270787715912,
+          0.01749962382018566,
+          0.037033889442682266,
+          -0.0011412798194214702,
+          0.03744423761963844,
+          0.026980368420481682,
+          -0.005979964509606361,
+          -0.013105483725667,
+          -0.028348194435238838,
+          0.013797945342957973,
+          0.006086825858801603,
+          0.03405886888504028,
+          -0.02034641243517399,
+          -0.03525571525096893,
+          0.007664100266993046,
+          -0.0034281141124665737,
+          0.04079541191458702,
+          -0.00934396218508482,
+          0.007822255603969097,
+          0.023099161684513092,
+          0.010942608118057251,
+          -0.004847233649343252,
+          -0.024210521951317787,
+          -0.008540363982319832,
+          -0.01188298873603344,
+          -0.0402824766933918,
+          -0.011891537345945835,
+          0.03157968446612358,
+          -0.015353847295045853,
+          0.01938893459737301,
+          -0.005915847606956959,
+          -0.007800883147865534,
+          -0.023543706163764,
+          0.018072402104735374,
+          0.0032464496325701475,
+          -0.008027428761124611,
+          -0.0069075217470526695,
+          0.003357585519552231,
+          -0.0019352601375430822,
+          0.003267821855843067,
+          0.048181671649217606,
+          -0.02402244508266449,
+          0.012113809585571289,
+          0.008527540601789951,
+          0.0011316622840240598,
+          0.02380017377436161,
+          0.03346044570207596,
+          -0.035563476383686066,
+          -0.012934505008161068,
+          -0.0014618640998378396,
+          0.009514939971268177,
+          0.020876444876194,
+          -0.009873994626104832,
+          -0.0097030159085989,
+          -0.03378530219197273,
+          -0.0005765173118561506,
+          0.00013043770741205662,
+          -0.015712901949882507,
+          -0.008467698469758034,
+          0.017166217789053917,
+          0.042641978710889816,
+          0.032434575259685516,
+          0.02034641243517399,
+          -0.035289913415908813,
+          -0.00907894503325224,
+          0.027082955464720726,
+          -0.014610092155635357,
+          0.034315336495637894,
+          0.0504043884575367,
+          -0.029459552839398384,
+          -0.018773412331938744,
+          -0.03980373963713646,
+          0.07112695276737213,
+          -0.023731783032417297,
+          -0.02964762970805168,
+          0.042368412017822266,
+          -0.01408860832452774,
+          0.0021778354421257973,
+          0.03187034651637077,
+          0.035324107855558395,
+          -0.0035307009238749743,
+          0.012053967453539371,
+          0.03499924764037132,
+          -0.0194060318171978,
+          -0.04691643267869949,
+          -0.02523639053106308,
+          -0.025116706266999245,
+          -0.016619086265563965,
+          -0.010275793261826038,
+          -0.017311548814177513,
+          0.010087717324495316,
+          0.020278021693229675,
+          0.013635516166687012,
+          -0.010882765986025333,
+          0.007022931706160307,
+          0.009985130280256271,
+          -0.009976581670343876,
+          -0.000917939527425915,
+          -0.027561694383621216,
+          -0.0071939099580049515,
+          -0.028467880561947823,
+          -0.012053967453539371,
+          -0.014977695420384407,
+          -0.006685249973088503,
+          -0.025783522054553032,
+          0.01889309659600258,
+          0.023167554289102554,
+          -0.002190658822655678,
+          -0.008459148928523064,
+          0.01238737441599369,
+          0.03472568467259407,
+          -0.028108825907111168,
+          -0.03812815248966217,
+          0.008916515856981277,
+          -0.0034558980260044336,
+          0.009762858040630817,
+          -0.015037537552416325,
+          0.010677591897547245,
+          -0.011395701207220554,
+          0.010540809482336044,
+          -0.010703238658607006,
+          -0.010002228431403637,
+          -0.0006646779947914183,
+          0.021953608840703964,
+          0.00801460538059473,
+          -0.014592994004487991,
+          -0.017593663185834885,
+          0.027715574949979782,
+          0.014293782413005829,
+          0.0334775410592556,
+          -0.029545042663812637,
+          -0.023099161684513092,
+          0.042573586106300354,
+          0.004015851765871048,
+          -0.043086521327495575,
+          0.016148896887898445,
+          0.01671312376856804,
+          0.011797499842941761,
+          0.01942313089966774,
+          0.048181671649217606,
+          -0.033870793879032135,
+          -0.038196541368961334,
+          0.005420010536909103,
+          -0.031938739120960236,
+          -0.030929965898394585,
+          0.014883656986057758,
+          -0.006227883044630289,
+          -0.05125928297638893,
+          -0.031203532591462135,
+          -0.014020216651260853,
+          -0.019337641075253487,
+          0.0022098938934504986,
+          -0.015046087093651295,
+          0.004941271618008614,
+          -0.03741004317998886,
+          0.0025689483154565096,
+          0.0004648471367545426,
+          -0.018328869715332985,
+          -0.028724348172545433,
+          0.0027548871003091335,
+          0.004488179460167885,
+          0.03542669489979744,
+          0.0017097825184464455,
+          0.01128456462174654,
+          -0.012951603159308434,
+          -0.000579188868869096,
+          0.003056236309930682,
+          -0.05016501992940903,
+          -0.019474424421787262,
+          0.038983043283224106,
+          0.009027652442455292,
+          0.006048355717211962,
+          0.011874440126121044,
+          -0.028211412951350212,
+          -0.0592268668115139,
+          0.029716020449995995,
+          -0.04643769562244415,
+          0.015268358401954174,
+          0.004415513481944799,
+          -0.0010344184702262282,
+          -0.001160514890216291,
+          -0.0007758138235658407,
+          -0.0067707388661801815,
+          0.023167554289102554,
+          -0.007407633122056723,
+          0.012498511001467705,
+          -0.0028788463678210974,
+          0.03669193387031555,
+          0.0358712375164032,
+          -0.00673226872459054,
+          -0.038948848843574524,
+          0.02956213988363743,
+          -0.008668597787618637,
+          0.025612542405724525,
+          0.01889309659600258,
+          0.02961343340575695,
+          -0.017918521538376808,
+          0.032417476177215576,
+          0.002891669748350978,
+          0.024125032126903534,
+          -0.03877786919474602,
+          0.023167554289102554,
+          0.0052191112190485,
+          -0.054268497973680496,
+          -0.00860020611435175,
+          -0.0073777115903794765,
+          0.022244270890951157,
+          0.008822478353977203,
+          -0.02477474883198738,
+          -0.021970706060528755,
+          0.0009409147314727306,
+          -0.02817721664905548,
+          -0.007574336603283882,
+          0.04568539187312126,
+          -0.012558353133499622,
+          0.0014875108608976007,
+          -0.017414135858416557,
+          0.014174097217619419,
+          0.02886112965643406,
+          0.027510400861501694,
+          0.03354593366384506,
+          -0.013986021280288696,
+          0.020021554082632065,
+          -0.022278467193245888,
+          -0.037033889442682266,
+          -0.02595449984073639,
+          0.0073050460778176785,
+          0.03939339146018028,
+          0.009728662669658661,
+          0.006039807107299566,
+          -0.0013913356233388186,
+          0.00790346972644329,
+          -0.015370945446193218,
+          0.027989141643047333,
+          0.044796302914619446,
+          -0.002494145417585969,
+          -0.011464091949164867,
+          0.007505945395678282,
+          0.015046087093651295,
+          -0.018619531765580177,
+          0.027373619377613068,
+          -0.025424467399716377,
+          0.02255203202366829,
+          0.004744646605104208,
+          0.02065417356789112,
+          -0.047702934592962265,
+          -0.004556570667773485,
+          -0.0023082063999027014,
+          -0.03445211797952652,
+          0.05946623906493187,
+          -0.028690151870250702,
+          -0.013524380512535572,
+          0.0029386887326836586,
+          -0.0011316622840240598,
+          -0.01313113048672676,
+          -0.026980368420481682,
+          -0.014849461615085602,
+          -0.0061894129030406475,
+          -0.04554860666394234,
+          -0.029801510274410248,
+          -0.014815266244113445,
+          -0.01213945634663105,
+          -0.06240706518292427,
+          0.004058596212416887,
+          -0.0741361752152443,
+          0.007206733338534832,
+          0.027715574949979782,
+          -0.016661832109093666,
+          0.03963275998830795,
+          -0.022637521848082542,
+          0.02374888025224209,
+          -0.04076121747493744,
+          -0.0157299991697073,
+          0.03258845582604408,
+          0.047463566064834595,
+          0.010848570615053177,
+          0.011498288251459599,
+          0.021047422662377357,
+          -0.03199003264307976,
+          -0.0026758096646517515,
+          -0.00860875565558672,
+          -0.0018219870980829,
+          0.03332366421818733,
+          -0.008553187362849712,
+          0.021765531972050667,
+          0.0051549943163990974,
+          0.024142129346728325,
+          0.031442902982234955,
+          -0.002782671246677637,
+          -0.023868564516305923,
+          0.012857564724981785,
+          0.025544151663780212,
+          0.013054190203547478,
+          -0.014251037500798702,
+          0.012686586938798428,
+          -0.01579839177429676,
+          0.007719668094068766,
+          0.008484795689582825,
+          0.009908189997076988,
+          0.034315336495637894,
+          0.024142129346728325,
+          -0.013105483725667,
+          0.0029814334120601416,
+          0.009865445084869862,
+          -0.03645256534218788,
+          0.042641978710889816,
+          0.0051849158480763435,
+          -0.03953017294406891,
+          -0.0005487333401106298,
+          -0.007617081515491009,
+          0.020551586523652077,
+          0.02156035788357258,
+          0.00936105940490961,
+          -0.022671716287732124,
+          -0.00020944836433045566,
+          0.016841357573866844,
+          0.008660049177706242,
+          -0.012182201258838177,
+          0.02450118400156498,
+          0.03833332657814026,
+          -0.020260922610759735,
+          -0.030639303848147392,
+          0.00922427698969841,
+          -0.010489515960216522,
+          0.0007736766128800809,
+          -0.02715134806931019,
+          -0.004394141025841236,
+          0.018072402104735374,
+          -0.044762108474969864,
+          0.05177221819758415,
+          -0.013575674034655094,
+          0.008685695938766003,
+          -0.012943054549396038,
+          -0.002703593811020255,
+          -0.02138938009738922,
+          0.034588903188705444,
+          0.014131353236734867,
+          0.02867305465042591,
+          0.016106151044368744,
+          0.010318538174033165,
+          0.028040433302521706,
+          -0.020278021693229675,
+          -0.023099161684513092,
+          0.014422016218304634,
+          -0.007980410009622574,
+          -0.00467198109254241,
+          -0.02959633618593216,
+          -0.024107934907078743,
+          -0.01892729289829731,
+          0.02527058683335781,
+          -0.01530255377292633,
+          -0.008566010743379593,
+          0.010951157659292221,
+          0.0014981969725340605,
+          -0.002131885150447488,
+          0.013943277299404144,
+          -0.007638453505933285,
+          0.008707067929208279,
+          0.04192386940121651,
+          -0.013934727758169174,
+          0.02159455418586731,
+          0.03611060976982117,
+          0.011472641490399837,
+          -0.018516944721341133,
+          -0.0064800758846104145,
+          -0.0007293291273526847,
+          -0.01066904328763485,
+          -0.03570026159286499,
+          0.035289913415908813,
+          0.020996129140257835,
+          -0.0021842471323907375,
+          -0.037239063531160355,
+          -0.012583999894559383,
+          0.014635738916695118,
+          -0.014037314802408218,
+          0.027732674032449722,
+          -0.015943722799420357,
+          -0.015653058886528015,
+          0.00474037230014801,
+          0.021970706060528755,
+          -0.049857258796691895,
+          -0.012977249920368195,
+          0.010899864137172699,
+          0.002707868115976453,
+          0.008057350292801857,
+          -0.006779287941753864,
+          0.013455988839268684,
+          0.020568683743476868,
+          0.004945545922964811,
+          0.0036439741961658,
+          -0.0038085407577455044,
+          0.01647375524044037,
+          0.013455988839268684,
+          -0.017593663185834885,
+          -0.009241375140845776,
+          0.02210748940706253,
+          0.0230820644646883,
+          -0.030211858451366425,
+          -0.023919858038425446,
+          -0.00583035871386528,
+          0.021902315318584442,
+          -0.004753195680677891,
+          0.03689710795879364,
+          0.0388120636343956,
+          0.011438445188105106,
+          -0.04318910837173462,
+          -0.014516053721308708,
+          0.0017290175892412663,
+          -0.010660494677722454,
+          0.016431011259555817,
+          -0.0012438668636605144,
+          -0.042812954634428024,
+          0.002143639838322997,
+          -0.0036055040545761585,
+          0.028450781479477882,
+          -0.040214087814092636,
+          -0.011959929019212723,
+          0.009617527015507221,
+          -0.012951603159308434,
+          0.003302017692476511,
+          0.012182201258838177,
+          0.008484795689582825,
+          0.021269695833325386,
+          0.012703684158623219,
+          -0.005526872351765633,
+          -0.0014607955235987902,
+          0.016550695523619652,
+          0.004535198211669922,
+          0.026604216545820236,
+          -0.018328869715332985,
+          -0.017345745116472244,
+          0.01844855397939682,
+          0.03860688954591751,
+          -0.012327532283961773,
+          0.04076121747493744,
+          0.007514494471251965,
+          0.012062516063451767,
+          0.012088162824511528,
+          -0.02453538030385971,
+          -0.01598646678030491,
+          -0.011951380409300327,
+          0.006257804110646248,
+          -0.037512630224227905,
+          0.014858010224997997,
+          -0.02380017377436161,
+          -0.027134248986840248,
+          0.02619386836886406,
+          -0.006014160346239805,
+          0.010686141438782215,
+          0.022244270890951157,
+          -0.012891761027276516,
+          -0.027271032333374023,
+          0.03369981423020363,
+          0.034349530935287476,
+          0.008433502167463303,
+          0.04677965119481087,
+          -0.010395478457212448,
+          0.006779287941753864,
+          0.005702124908566475,
+          0.0024920080322772264,
+          0.044762108474969864,
+          -0.008236877620220184,
+          -0.01200267393141985,
+          0.007535866927355528,
+          0.009779956191778183,
+          0.04206065088510513,
+          -0.015413690358400345,
+          -0.049412716180086136,
+          0.019508618861436844,
+          -0.014507505111396313,
+          0.009856896474957466,
+          -0.006655328441411257,
+          0.008198407478630543,
+          -0.005890200845897198,
+          0.010053521953523159,
+          -0.004522374831140041,
+          0.010113364085555077,
+          -0.005843182094395161,
+          -0.014327977783977985,
+          0.015174319967627525,
+          -0.01018175482749939,
+          0.030280249193310738,
+          0.009557684883475304,
+          -0.02503121644258499,
+          -0.026501629501581192,
+          0.01746542938053608,
+          0.0019277798710390925,
+          -0.028724348172545433,
+          0.009882543236017227,
+          0.0023872838355600834,
+          0.010104815475642681,
+          0.004381317645311356,
+          -0.012242043390870094,
+          0.015926625579595566,
+          0.01676441729068756,
+          -0.022210076451301575,
+          0.03911982476711273,
+          -0.014285233803093433,
+          -0.014644287526607513,
+          0.019235054031014442,
+          -0.03730745613574982,
+          0.008424953557550907,
+          -0.012524157762527466,
+          0.010831472463905811,
+          0.00491989916190505,
+          0.004667706321924925,
+          -0.011352956295013428,
+          -0.0026715353596955538,
+          -0.02352660894393921,
+          0.03281072899699211,
+          0.0272197388112545,
+          -0.0012834055814892054,
+          0.005958592053502798,
+          -0.007535866927355528,
+          -0.012900309637188911,
+          0.0024984197225421667,
+          0.026142574846744537,
+          -0.024381499737501144,
+          -0.0017899286467581987,
+          -0.0006310166209004819,
+          0.007694021798670292,
+          0.00551832327619195,
+          0.022432347759604454,
+          0.04900236800312996,
+          -0.0038192267529666424,
+          0.028980813920497894,
+          0.035050541162490845,
+          0.024142129346728325,
+          -0.003259273013100028,
+          0.004761744290590286,
+          -0.025150902569293976,
+          0.03645256534218788,
+          0.008950712159276009,
+          0.016670379787683487,
+          0.017798837274312973,
+          -0.04312071576714516,
+          -0.009634625166654587,
+          0.019525716081261635,
+          0.02205619588494301,
+          0.020089944824576378,
+          -0.003881206503137946,
+          -0.0035884061362594366,
+          0.03298170492053032,
+          -0.025920303538441658,
+          -0.019354738295078278,
+          0.004154771566390991,
+          -0.0053174239583313465,
+          -0.005732045974582434,
+          -0.0069032469764351845,
+          0.01865372806787491,
+          -0.014584445394575596,
+          -0.004458257928490639,
+          0.016867004334926605,
+          0.001722605898976326,
+          0.012113809585571289,
+          0.016165994107723236,
+          -0.00024137321452144533,
+          0.022415248677134514,
+          -0.004932722542434931,
+          0.021509064361453056,
+          0.005330247338861227,
+          -0.003663209266960621,
+          -0.006911796052008867,
+          -0.010275793261826038,
+          0.029015010222792625,
+          -0.006980187259614468,
+          -0.034640196710824966,
+          -0.01360986940562725,
+          -0.009284119121730328,
+          0.004156908951699734,
+          0.0011391426669433713,
+          0.0050524077378213406,
+          0.01744833029806614,
+          0.026501629501581192,
+          -0.00046751866466365755,
+          0.004280867986381054,
+          0.014336527325212955,
+          -0.0024193422868847847,
+          -0.010506614111363888,
+          0.019559912383556366,
+          0.008779733441770077,
+          -0.01624293439090252,
+          0.018756315112113953,
+          0.0030861576087772846,
+          0.04688223823904991,
+          -0.044522739946842194,
+          0.018209183588624,
+          0.007805157452821732,
+          0.0006956678116694093,
+          -0.014960597269237041,
+          -0.03327237069606781,
+          0.0036097783595323563,
+          -0.014122803695499897,
+          -0.0033597226720303297,
+          -0.024689260870218277,
+          0.00786499958485365,
+          -0.018089499324560165,
+          0.014045863412320614,
+          0.011600874364376068,
+          -0.0016349796205759048,
+          -0.002028229646384716,
+          0.017542369663715363,
+          -0.001940603251568973,
+          0.00787782296538353,
+          -0.017362842336297035,
+          0.010942608118057251,
+          0.022381054237484932,
+          -0.010677591897547245,
+          0.008638676255941391,
+          0.0073306928388774395,
+          -0.02793784812092781,
+          -0.028536271303892136,
+          0.013336303643882275,
+          -0.03546088933944702,
+          0.007681198418140411,
+          0.031203532591462135,
+          -0.032434575259685516,
+          -0.028519174084067345,
+          0.0019085448002442718,
+          -0.021406477317214012,
+          -0.017867228016257286,
+          0.011446994729340076,
+          0.018021108582615852,
+          -0.004216751083731651,
+          -0.030451226979494095,
+          0.013173874467611313,
+          -0.007809431757777929,
+          0.00698873633518815,
+          0.00015388043539132923,
+          -0.02569803223013878,
+          -0.02569803223013878,
+          0.004984016064554453,
+          -0.028211412951350212,
+          -0.007138342130929232,
+          -0.0005297654424794018,
+          -0.024466989561915398,
+          -0.043804630637168884,
+          -0.015242711640894413,
+          -0.013507282361388206,
+          -0.010156108066439629,
+          0.0480106957256794,
+          0.0196795966476202,
+          0.026894880458712578,
+          -0.0003211185394320637,
+          -0.018106596544384956,
+          0.04773712903261185,
+          0.0009954140987247229,
+          0.041752889752388,
+          0.013387597166001797,
+          -0.008335189893841743,
+          -0.006693798583000898,
+          0.03392208740115166,
+          -0.015584668144583702,
+          -0.007775236386805773,
+          -0.010651945136487484,
+          -0.0015836860984563828,
+          0.0025091059505939484,
+          0.022449444979429245,
+          0.0019085448002442718,
+          0.004037224221974611,
+          0.027100054547190666,
+          -0.023201748728752136,
+          0.002739926567301154,
+          0.01238737441599369,
+          -0.006937442813068628,
+          0.008215505629777908,
+          -0.02838239073753357,
+          -0.008300994522869587,
+          0.019081173464655876,
+          0.006044081412255764,
+          -0.007437554188072681,
+          -0.0022440897300839424,
+          -0.027134248986840248,
+          -0.0014052275801077485,
+          0.0038106779102236032,
+          -0.005826083943247795,
+          0.005744869355112314,
+          0.04459112882614136,
+          -0.004817312583327293,
+          -0.039940521121025085,
+          -0.00029493749025277793,
+          -0.03634997829794884,
+          0.0001610935723874718,
+          0.00010799681331263855,
+          -0.0033191153779625893,
+          0.005488402210175991,
+          0.0018422907451167703,
+          -0.012472864240407944,
+          0.012130907736718655,
+          -0.028741445392370224,
+          0.019782183691859245,
+          0.004817312583327293,
+          -0.007574336603283882,
+          -0.005056682042777538,
+          0.01988477073609829,
+          0.008151388727128506,
+          -0.013148227706551552,
+          -0.0014960597036406398,
+          0.0028852580580860376,
+          -0.026125477626919746,
+          -0.020773857831954956,
+          -0.03369981423020363,
+          -0.0022505014203488827,
+          -0.009583331644535065,
+          -0.03010927140712738,
+          -0.006937442813068628,
+          -0.0003617258626036346,
+          0.007775236386805773,
+          -0.020295118913054466,
+          0.0006956678116694093,
+          0.005552519112825394,
+          0.012199298478662968,
+          0.00624070642516017,
+          -0.007339241914451122,
+          -0.004263770300894976,
+          -0.03033154271543026,
+          0.003584131831303239,
+          -0.0015345298452302814,
+          0.0038619714323431253,
+          -0.016063407063484192,
+          0.006351842079311609,
+          -0.016183091327548027,
+          -0.0035264266189187765,
+          0.011335858143866062,
+          0.01800401136279106,
+          -0.028741445392370224,
+          0.036726128309965134,
+          0.02889532595872879,
+          0.017320098355412483,
+          -0.038914650678634644,
+          0.028587564826011658,
+          -0.0012331806356087327,
+          0.001522775157354772,
+          0.0030177661683410406,
+          0.021731335669755936,
+          0.026039987802505493,
+          -0.016431011259555817,
+          -0.006531369406729937,
+          0.010224499739706516,
+          0.028194313868880272,
+          -0.031545490026474,
+          -0.007266575936228037,
+          1.9886239897459745e-05,
+          0.02378307469189167,
+          0.027783967554569244,
+          0.023116260766983032,
+          0.04271036759018898,
+          -0.0053131491877138615,
+          0.002891669748350978,
+          -0.005082328803837299,
+          -0.013926179148256779,
+          0.0024727729614824057,
+          -0.0028767092153429985,
+          -0.0017952717607840896,
+          0.011489738710224628,
+          0.012695135548710823,
+          0.017593663185834885,
+          -0.07126373797655106,
+          0.038743674755096436,
+          0.026570022106170654,
+          0.012105260975658894,
+          0.008399306796491146,
+          -0.024706358090043068,
+          -0.0017909972229972482,
+          -0.0022205801215022802,
+          0.0018604571232572198,
+          0.013438890688121319,
+          -0.010874217376112938,
+          -0.02475765161216259,
+          0.016567792743444443,
+          -0.019457325339317322,
+          0.017149118706583977,
+          0.0050267609767615795,
+          0.00503958435729146,
+          -0.024603771045804024,
+          -0.014721227809786797,
+          0.006830581463873386,
+          0.008065898902714252,
+          0.008100095205008984,
+          -0.03857269510626793,
+          -0.044727910310029984,
+          0.01750817336142063,
+          0.008035978302359581,
+          -0.04862621799111366,
+          -0.006586937233805656,
+          -0.017388489097356796,
+          -0.004210339393466711,
+          -0.006326195318251848,
+          -0.01767915114760399,
+          -0.040897998958826065,
+          0.002064562402665615,
+          0.017302999272942543,
+          0.018534043803811073,
+          5.349749335437082e-05,
+          0.028826933354139328,
+          0.0017183314776048064,
+          -0.0022355406545102596,
+          -0.030434129759669304,
+          0.026860684156417847,
+          -0.007326418533921242,
+          -0.022278467193245888,
+          0.024142129346728325,
+          0.01628567837178707,
+          -0.019970260560512543,
+          -0.0038769319653511047,
+          0.0014928538585081697,
+          -0.008446325547993183,
+          0.034554705023765564,
+          -0.0017129883635789156,
+          -0.008933614008128643,
+          0.0026907704304903746,
+          -0.0020998266991227865,
+          -0.00959188025444746,
+          -0.02479184791445732,
+          0.00145438383333385,
+          0.025869010016322136,
+          -0.007749589625746012,
+          0.006437331438064575,
+          0.04202645644545555,
+          -0.012353179045021534,
+          -0.019850576296448708,
+          0.00043145293602719903,
+          -0.02839948795735836,
+          0.010318538174033165,
+          -0.006685249973088503,
+          0.030280249193310738,
+          0.004206065088510513,
+          0.013002896681427956,
+          -0.016644733026623726,
+          -0.01066904328763485,
+          -0.021970706060528755,
+          -0.0641852393746376,
+          -0.012122358195483685,
+          -0.005561067722737789,
+          -0.024894434958696365,
+          -0.02398824878036976,
+          0.03234908729791641,
+          0.022398151457309723,
+          0.010609201155602932,
+          -0.002107307082042098,
+          -0.01891019567847252,
+          -0.004458257928490639,
+          0.03009217418730259,
+          -0.005458480678498745,
+          0.017371391877532005,
+          -0.011617972515523434,
+          0.01627713069319725,
+          -0.006578388623893261,
+          0.0051336223259568214,
+          -0.053960736840963364,
+          -0.005167817696928978,
+          0.02983570657670498,
+          0.01993606425821781,
+          0.007240929175168276,
+          0.010019325651228428,
+          0.013293559662997723,
+          -0.007501671090722084,
+          -0.02764718420803547,
+          0.02525348775088787,
+          0.017132021486759186,
+          0.01991896703839302,
+          -0.015935173258185387,
+          0.007587159983813763,
+          0.011788951233029366,
+          0.00288098375312984,
+          0.000775279535446316,
+          0.0008281759219244123,
+          0.025133803486824036,
+          0.003549935994669795,
+          -0.009275570511817932,
+          0.015721451491117477,
+          -0.007163988891988993,
+          -0.021269695833325386,
+          -0.004233849234879017,
+          0.008959260769188404,
+          0.004035086836665869,
+          0.001731154858134687,
+          -0.0015922350576147437,
+          -0.008984907530248165,
+          0.02886112965643406,
+          0.03539250046014786,
+          0.03183615207672119,
+          -0.0057876138016581535,
+          -0.005137896630913019,
+          0.0016456657322123647,
+          0.003513603238388896,
+          -0.007659825962036848,
+          -0.0018294673645868897,
+          0.019799282774329185,
+          -0.030006684362888336,
+          -0.011276016011834145,
+          -0.009318315424025059,
+          0.016362618654966354,
+          0.003522152081131935,
+          0.04411239176988602,
+          0.02111581526696682,
+          0.006522820331156254,
+          -0.00557389110326767,
+          0.0037209142465144396,
+          0.004389866720885038,
+          0.029237281531095505,
+          -0.02282559685409069,
+          9.023110033012927e-05,
+          -0.0034217024222016335,
+          -0.017901424318552017,
+          0.019046977162361145,
+          -0.007300771772861481,
+          0.009540586732327938,
+          0.02205619588494301,
+          0.09212308377027512,
+          -0.0065527418628335,
+          0.022449444979429245,
+          -0.027852358296513557,
+          -0.024090835824608803,
+          0.00291517935693264,
+          0.006313371937721968,
+          0.002265461953356862,
+          0.022039096802473068,
+          0.0054499320685863495,
+          0.02525348775088787,
+          0.0272197388112545,
+          0.0009067190694622695,
+          0.006698073353618383,
+          -0.010292891412973404,
+          0.004118438810110092,
+          -0.013182424008846283,
+          -0.009549135342240334,
+          -0.009437999688088894,
+          0.008779733441770077,
+          0.019012782722711563,
+          0.00911314133554697,
+          0.007531592156738043,
+          -0.004084242973476648,
+          -0.015482081100344658,
+          0.026518728584051132,
+          0.010292891412973404,
+          -0.004475356079638004,
+          -0.015601766295731068,
+          -0.015114477835595608,
+          0.012404472567141056,
+          -0.006886149290949106,
+          -0.005894475616514683,
+          0.01721751131117344,
+          0.005069505423307419,
+          0.02865595556795597,
+          -0.03369981423020363,
+          0.008890869095921516,
+          0.014259587042033672,
+          -0.0010857119923457503,
+          0.007245203480124474,
+          0.021423576399683952,
+          -0.0022633245680481195,
+          0.0242789126932621,
+          -0.01468703243881464,
+          0.00874553807079792,
+          0.010959706269204617,
+          -0.021201303228735924,
+          0.0036055040545761585,
+          0.01990186981856823,
+          -0.003381095128133893,
+          0.015413690358400345,
+          -0.0052191112190485,
+          -0.001976936124265194,
+          0.0025903205387294292,
+          0.006582662928849459,
+          0.00363756250590086,
+          -0.01988477073609829,
+          0.01188298873603344,
+          -0.01872211880981922,
+          0.009172983467578888,
+          -0.004565119277685881,
+          0.012190749868750572,
+          -0.008424953557550907,
+          0.011370054446160793,
+          -0.006121021695435047,
+          0.006847679149359465,
+          -0.0105750048533082,
+          -0.016559245064854622,
+          -0.012515608221292496,
+          0.012960151769220829,
+          0.007249478250741959,
+          -0.013558575883507729,
+          0.008309543132781982,
+          -0.008228329010307789,
+          -0.002067768247798085,
+          0.014960597269237041,
+          -0.011746206320822239,
+          0.00407355697825551,
+          -0.039461780339479446,
+          -0.004535198211669922,
+          0.018089499324560165,
+          -0.001959838205948472,
+          0.0027719850186258554,
+          -0.007352065294981003,
+          0.011199075728654861,
+          -0.008288171142339706,
+          -0.0024086562916636467,
+          0.0254757609218359,
+          -0.019782183691859245,
+          -0.017328646034002304,
+          -0.032913316041231155,
+          -0.05009663105010986,
+          -0.002131885150447488,
+          -0.009326864033937454,
+          0.005864554084837437,
+          -0.00790346972644329,
+          0.017815934494137764,
+          -0.0040179891511797905,
+          -0.007095597684383392,
+          -0.024125032126903534,
+          0.010395478457212448,
+          0.028211412951350212,
+          0.0008682489860802889,
+          -0.03079318441450596,
+          -0.012344630435109138,
+          0.010463869199156761,
+          0.0006646779947914183,
+          -0.029049206525087357,
+          0.0083693852648139,
+          0.0006181932403706014,
+          -0.014473309740424156,
+          -0.008882320486009121,
+          -0.005971415434032679,
+          0.021440673619508743,
+          0.024193422868847847,
+          0.005753418430685997,
+          -0.019782183691859245,
+          -0.004225300159305334,
+          0.03258845582604408,
+          -0.018294673413038254,
+          0.003994479309767485,
+          0.023868564516305923,
+          -0.00264375121332705,
+          0.00982270110398531,
+          0.01054935809224844,
+          0.018072402104735374,
+          -0.008352288044989109,
+          0.012660940177738667,
+          0.007561513222754002,
+          -0.015328200533986092,
+          0.008548912592232227,
+          0.006668151821941137,
+          0.016405364498496056,
+          0.006890423595905304,
+          -0.022244270890951157,
+          -0.004411239176988602,
+          0.015832586213946342,
+          0.053276825696229935,
+          -0.006706621963530779,
+          -0.008506168611347675,
+          0.03617899864912033,
+          0.024415696039795876,
+          0.007416181731969118,
+          0.005937220063060522,
+          -0.03231488913297653,
+          -0.004646334331482649,
+          0.0008709204848855734,
+          0.005236208904534578,
+          0.0132422661408782,
+          0.0035029170103371143,
+          -0.007770961616188288,
+          0.018978586420416832,
+          -0.001043501659296453,
+          0.014293782413005829,
+          -0.020107043907046318,
+          0.009514939971268177,
+          -0.00618513859808445,
+          0.03161387890577316,
+          0.01042112521827221,
+          -0.0010632710764184594,
+          0.03318687900900841,
+          0.006719445344060659,
+          -0.03344334661960602,
+          0.006125296000391245,
+          -0.007638453505933285,
+          0.011352956295013428,
+          -0.02260332554578781,
+          -0.007894921116530895,
+          0.009899641387164593,
+          0.0012289062142372131,
+          -0.018123695626854897,
+          0.011900086887180805,
+          0.007142616901546717,
+          0.009378157556056976,
+          -0.0005217508296482265,
+          0.006356116849929094,
+          0.008553187362849712,
+          -0.004522374831140041,
+          -0.008168485946953297,
+          0.005599537864327431,
+          -0.016336971893906593,
+          -0.0040179891511797905,
+          0.029425358399748802,
+          -0.0006726926076225936,
+          0.010309988632798195,
+          0.0004135536728426814,
+          -0.0004116835771128535,
+          -0.018978586420416832,
+          0.009207178838551044,
+          -0.024569574743509293,
+          -0.006415958981961012,
+          0.005595263559371233,
+          -0.01225059200078249,
+          -0.006142393685877323,
+          0.014413466677069664,
+          0.0022633245680481195,
+          -0.021047422662377357,
+          -0.008061625063419342,
+          -0.030485423281788826,
+          -0.021218402311205864,
+          0.031904544681310654,
+          0.006518546026200056,
+          -0.005646557081490755,
+          0.015379494056105614,
+          -0.022876890376210213,
+          -0.0054969508200883865,
+          -0.005035309586673975,
+          0.016636185348033905,
+          -0.020021554082632065,
+          0.004398415796458721,
+          -0.010634847916662693,
+          -0.016431011259555817,
+          0.023099161684513092,
+          0.008326641283929348,
+          0.0011220448650419712,
+          0.006133845075964928,
+          -0.02497992292046547,
+          -0.014285233803093433,
+          0.01798691228032112,
+          0.005702124908566475,
+          -0.008228329010307789,
+          0.01795271784067154,
+          -0.0012609646655619144,
+          -0.007215282414108515,
+          -0.013780847191810608,
+          0.01841435767710209,
+          -0.02183392271399498,
+          0.026535825803875923,
+          0.012156554497778416,
+          -0.010472417809069157,
+          0.008822478353977203,
+          0.004992565140128136,
+          0.01287466287612915,
+          -0.0003889755462296307,
+          -0.015370945446193218,
+          0.013421793468296528,
+          0.013712456449866295,
+          0.002194933360442519,
+          -0.0021778354421257973,
+          0.0037572472356259823,
+          0.005826083943247795,
+          -0.0054456572979688644,
+          0.02333853207528591,
+          0.023594999685883522,
+          0.0002186651690863073,
+          0.015080282464623451,
+          -0.02084225043654442,
+          -0.019611205905675888,
+          0.02378307469189167,
+          -0.004595040809363127,
+          -0.0314599983394146,
+          0.038709476590156555,
+          -0.010292891412973404,
+          -0.007236654870212078,
+          -0.01993606425821781,
+          0.00771111948415637,
+          0.020448999479413033,
+          -0.021885216236114502,
+          -0.010002228431403637,
+          -0.00110708421561867,
+          0.008848125115036964,
+          0.027852358296513557,
+          0.020568683743476868,
+          -0.015507727861404419,
+          0.02788655459880829,
+          0.009668820537626743,
+          0.008044526912271976,
+          0.011874440126121044,
+          0.005561067722737789,
+          -0.010840021073818207,
+          0.010044972412288189,
+          -0.0005241552134975791,
+          0.005296051502227783,
+          -0.00516354339197278,
+          -0.02231266349554062,
+          -0.015644511207938194,
+          -0.04178708791732788,
+          -0.0036589347291737795,
+          -0.008604480884969234,
+          0.01553337462246418,
+          0.08624143153429031,
+          -0.014037314802408218,
+          -0.009309765882790089,
+          -0.0020271609537303448,
+          0.009976581670343876,
+          -0.01720896176993847,
+          0.012566901743412018,
+          0.024466989561915398,
+          -0.013626967556774616,
+          -0.0329304113984108,
+          0.0025518503971397877,
+          -0.012994347140192986,
+          0.007805157452821732,
+          0.017542369663715363,
+          0.0019245740259066224,
+          -0.023680489510297775,
+          -0.0018433593213558197,
+          -0.020004456862807274,
+          0.001239592325873673,
+          0.010369831696152687,
+          -0.016131797805428505,
+          -0.010771630331873894,
+          0.023902760818600655,
+          -0.02017543464899063,
+          -0.025167999789118767,
+          -0.012925956398248672,
+          -0.007745314855128527,
+          -0.011891537345945835,
+          0.017815934494137764,
+          0.008647225797176361,
+          -0.0186708252876997,
+          -0.008595932275056839,
+          0.005915847606956959,
+          -0.011378603056073189,
+          8.963001164374873e-05,
+          0.011925733648240566,
+          -0.0033169782254844904,
+          0.008788282051682472,
+          0.01482381485402584,
+          -0.03400757536292076,
+          -0.003128902055323124,
+          -0.014704130589962006,
+          0.01360986940562725,
+          0.006557016167789698,
+          0.028057532384991646,
+          -0.028279803693294525,
+          -0.021765531972050667,
+          -0.00981415156275034,
+          -0.01892729289829731,
+          -0.03082738071680069,
+          -0.02692907489836216,
+          0.0058175353333354,
+          0.00636466545984149,
+          0.021526163443922997,
+          0.014729776419699192,
+          -0.014396369457244873,
+          0.010096265934407711,
+          -0.0033981928136199713,
+          0.008984907530248165,
+          -0.014404918067157269,
+          -0.0018454965902492404,
+          -0.015550472773611546,
+          0.015131575986742973,
+          0.017662053927779198,
+          -0.002583908848464489,
+          -0.018380163237452507,
+          -0.019730890169739723,
+          0.010429673828184605,
+          -0.007604258134961128,
+          -0.004637785255908966,
+          0.02668970637023449,
+          0.003182332729920745,
+          0.0019096133764833212,
+          0.007527317851781845,
+          0.023406922817230225,
+          0.008664323017001152,
+          0.03546088933944702,
+          -0.023406922817230225,
+          0.001999377040192485,
+          -0.0189443901181221,
+          0.0031866072677075863,
+          -0.023218847811222076,
+          0.005112249869853258,
+          -0.01942313089966774,
+          0.03417855501174927,
+          -0.006133845075964928,
+          -0.007852176204323769,
+          0.030468326061964035,
+          -0.02624516189098358,
+          0.002979296026751399,
+          -0.009754309430718422,
+          0.004650608636438847,
+          0.022141683846712112,
+          -0.0038256384432315826,
+          -0.017285902053117752,
+          -0.03925660625100136,
+          0.003522152081131935,
+          -0.009899641387164593,
+          -0.006091100629419088,
+          -0.011318760924041271,
+          0.01773044466972351,
+          0.005065230652689934,
+          0.005569616798311472,
+          -0.0015633824514225125,
+          0.016892651095986366,
+          0.0013913356233388186,
+          0.0035456616897135973,
+          0.029955390840768814,
+          -0.001200053608044982,
+          0.012558353133499622,
+          0.0033533109817653894,
+          0.005689301528036594,
+          0.023492412641644478,
+          -0.007745314855128527,
+          -0.0015206378884613514,
+          -0.017362842336297035,
+          0.0067237201146781445,
+          -0.0007555101765319705,
+          0.02520219422876835,
+          0.01539659220725298,
+          -0.01771334744989872,
+          -0.040658630430698395,
+          0.02159455418586731,
+          -0.011352956295013428,
+          0.027373619377613068,
+          -0.009497841820120811,
+          -0.006569839548319578,
+          -0.0003144397051073611,
+          -0.03754682466387749,
+          0.003182332729920745,
+          -0.009720114059746265,
+          -0.052798084914684296,
+          -0.018089499324560165,
+          -0.007548689842224121,
+          -0.026296455413103104,
+          0.03342624753713608,
+          -0.0014650699449703097,
+          -0.026775194332003593,
+          0.024706358090043068,
+          -0.0034516234882175922,
+          -0.00698873633518815,
+          0.012233494780957699,
+          -0.016704576089978218,
+          -0.009378157556056976,
+          -0.0032443124800920486,
+          -0.005116524174809456,
+          0.0006069728406146169,
+          -0.012421570718288422,
+          -0.019235054031014442,
+          0.015866782516241074,
+          -0.006980187259614468,
+          0.0008538226829841733,
+          0.015866782516241074,
+          -0.0020496018696576357,
+          0.025065412744879723,
+          -0.014122803695499897,
+          0.005385815165936947,
+          0.007612806744873524,
+          -0.013883434236049652,
+          0.012301885522902012,
+          0.003847010899335146,
+          -0.015430787578225136,
+          0.01580693945288658,
+          0.009215728379786015,
+          0.00848907046020031,
+          0.0041611832566559315,
+          -0.014533151872456074,
+          0.0022889713291078806,
+          -0.011421347968280315,
+          0.006646779831498861,
+          0.013909080997109413,
+          0.005304600577801466,
+          -0.011276016011834145,
+          -0.016653282567858696,
+          0.013857787474989891,
+          0.007501671090722084,
+          0.020004456862807274,
+          -0.03180195763707161,
+          0.007386260665953159,
+          -0.004676255397498608,
+          0.006941717118024826,
+          -0.01227623876184225,
+          0.025287684053182602,
+          0.022620422765612602,
+          0.02111581526696682,
+          0.006356116849929094,
+          0.0028767092153429985,
+          0.010515162721276283,
+          -0.01966249942779541,
+          -0.0033618600573390722,
+          -0.0204148031771183,
+          -0.0006417027907446027,
+          -0.0008324504015035927,
+          0.01743123307824135,
+          -0.012318983674049377,
+          0.013105483725667,
+          0.006522820331156254,
+          0.015447885729372501,
+          -0.02887822687625885,
+          0.0039047158788889647,
+          0.0070400298573076725,
+          0.0013421793701127172,
+          0.0066211330704391,
+          0.0031224903650581837,
+          0.08371095359325409,
+          0.010472417809069157,
+          -0.0070870486088097095,
+          0.003968833014369011,
+          -0.009437999688088894,
+          -0.0077538639307022095,
+          0.015251260250806808,
+          -0.01081437524408102,
+          0.012233494780957699,
+          0.007347790524363518,
+          -0.013148227706551552,
+          0.004582217428833246,
+          0.0073563395999372005,
+          0.008108643814921379,
+          -0.01623438484966755,
+          0.011190527118742466,
+          -0.005253307055681944,
+          0.0038961670361459255,
+          0.025133803486824036,
+          0.01652504876255989,
+          -0.005723497364670038,
+          0.00971156544983387,
+          0.006467252504080534,
+          0.0009863307932391763,
+          -0.012404472567141056,
+          0.006390312220901251,
+          -0.004152634646743536,
+          0.023013673722743988,
+          -0.016208738088607788,
+          0.0344863161444664,
+          -0.0005706399097107351,
+          -0.003855559742078185,
+          -0.010882765986025333,
+          -0.003383232280611992,
+          0.007501671090722084,
+          -0.008322366513311863,
+          0.01114778220653534,
+          -0.013712456449866295,
+          0.0005973553052172065,
+          -0.013045640662312508,
+          -0.008189858868718147,
+          0.004058596212416887,
+          -0.007540141232311726,
+          -0.0073777115903794765,
+          -0.0009211453725583851,
+          0.001756801619194448,
+          -4.190970867057331e-05,
+          -0.0012331806356087327,
+          -0.0012919544242322445,
+          -0.009437999688088894,
+          -0.004513826221227646,
+          -0.013601320795714855,
+          -1.5010688912298065e-05,
+          0.02137228287756443,
+          -0.003682444104924798,
+          -0.006809209007769823,
+          0.01044677197933197,
+          0.003128902055323124,
+          0.02429600991308689,
+          -0.006095374934375286,
+          -0.014558798633515835,
+          -0.0021564632188528776,
+          -0.007005834020674229,
+          -0.007390534970909357,
+          0.00033928497578017414,
+          0.0038790691178292036,
+          0.006180863827466965,
+          0.003287056926637888,
+          0.012438667938113213,
+          0.006018434651196003,
+          -0.008831026963889599,
+          0.011763304471969604,
+          0.003120353212580085,
+          -0.0009190081618726254,
+          -0.01891019567847252,
+          -0.003242175094783306,
+          -0.013550027273595333,
+          -0.01199412439018488,
+          -0.011455543339252472,
+          0.017345745116472244,
+          -0.01723460853099823,
+          0.019765086472034454,
+          -0.006339018698781729,
+          0.00765555165708065,
+          -0.02549285814166069,
+          0.002449263585731387,
+          0.008899418637156487,
+          0.011250369250774384,
+          0.006702347658574581,
+          -0.012857564724981785,
+          -0.002013268880546093,
+          -0.0010862462222576141,
+          0.02963053248822689,
+          -0.008702793158590794,
+          -0.010010777041316032,
+          0.02816011942923069,
+          -0.014413466677069664,
+          -0.01019030436873436,
+          0.0014362173387780786,
+          -0.0013090522261336446,
+          0.02552705444395542,
+          0.011455543339252472,
+          0.017328646034002304,
+          0.0054456572979688644,
+          0.002840376226231456,
+          0.0013379048323258758,
+          0.023936955258250237,
+          0.016071956604719162,
+          0.010908412747085094,
+          -0.01418264675885439,
+          -0.008446325547993183,
+          -0.01942313089966774,
+          0.014216842129826546,
+          0.0033746834378689528,
+          -0.01530255377292633,
+          0.011224722489714622,
+          -0.021953608840703964,
+          -0.019064076244831085,
+          -0.023697586730122566,
+          -0.010386928915977478,
+          0.013438890688121319,
+          -0.011686364188790321,
+          -0.005505499895662069,
+          -0.017089277505874634,
+          -0.005454206373542547,
+          -0.010455320589244366,
+          -0.015148673206567764,
+          -0.01673022285103798,
+          -0.02598869428038597,
+          -0.022688815370202065,
+          -0.0098397983238101,
+          -0.017294451594352722,
+          -0.008300994522869587,
+          0.012797722592949867,
+          -0.020448999479413033,
+          0.017662053927779198,
+          -0.005065230652689934,
+          0.0037893056869506836,
+          0.01188298873603344,
+          0.0216116514056921,
+          -0.01748252660036087,
+          -0.031904544681310654,
+          -0.02402244508266449,
+          -0.02135518379509449,
+          0.022705912590026855,
+          0.0021927962079644203,
+          0.024142129346728325,
+          -0.007125518750399351,
+          0.016157444566488266,
+          -0.018363064154982567,
+          0.019303444772958755,
+          -0.04202645644545555,
+          -0.03222940117120743,
+          -0.007168263662606478,
+          0.00034836819395422935,
+          -0.004362082574516535,
+          -0.0010199921671301126,
+          -0.015661608427762985,
+          -0.010993901640176773,
+          -0.007168263662606478,
+          -0.006210785359144211,
+          -0.009685918688774109,
+          -0.01516577135771513,
+          0.01007916871458292,
+          0.004370631650090218,
+          0.016165994107723236,
+          0.001985484967008233,
+          0.015559021383523941,
+          -0.01516577135771513,
+          0.006945991888642311,
+          -0.0022269918117672205,
+          -0.005753418430685997,
+          0.004646334331482649,
+          0.003804266219958663,
+          0.0044924537651240826,
+          0.0002427089784760028,
+          0.0011017411015927792,
+          -0.00029386888490989804,
+          0.017362842336297035,
+          0.013361950404942036,
+          -0.007146891206502914,
+          0.01773044466972351,
+          6.485151970991865e-05,
+          0.0016787927597761154,
+          -0.010429673828184605,
+          -0.003214391181245446,
+          -0.00911314133554697,
+          -0.008882320486009121,
+          0.015362396836280823,
+          -0.019730890169739723,
+          -0.010626298375427723,
+          0.026843586936593056,
+          -0.0018348104786127806,
+          -0.03443502262234688,
+          0.02742491289973259,
+          7.86015789344674e-06,
+          -0.0024364402052015066,
+          -0.01369535829871893,
+          -0.007078499998897314,
+          -0.0006363596767187119,
+          -0.0005978895933367312,
+          0.017337195575237274,
+          -0.018038205802440643,
+          -0.005249032285064459,
+          0.00630909763276577,
+          -0.0040436359122395515,
+          -0.0021254734601825476,
+          0.006497173570096493,
+          0.014028766192495823,
+          -0.00047820480540394783,
+          -0.03501634672284126,
+          -0.012190749868750572,
+          0.020568683743476868,
+          -0.017106374725699425,
+          -0.014994793571531773,
+          0.011233271099627018,
+          -0.001976936124265194,
+          -0.012601097114384174,
+          -0.003911127801984549,
+          -0.005223385989665985,
+          -0.010993901640176773,
+          0.008442051708698273,
+          0.010224499739706516,
+          0.008339464664459229,
+          0.02210748940706253,
+          -0.014447662979364395,
+          -0.005757692735642195,
+          -0.015251260250806808,
+          0.0031075298320502043,
+          -0.022962380200624466,
+          -0.00015214394079521298,
+          -0.009745760820806026,
+          0.0016766554908826947,
+          0.012575451284646988,
+          0.01749962382018566,
+          0.028724348172545433,
+          0.0089678093791008,
+          0.0072879483923316,
+          -0.010848570615053177,
+          -0.01456734724342823,
+          0.004680529702454805,
+          0.0031417254358530045,
+          -0.00558671448379755,
+          0.006655328441411257,
+          0.019286347553133965,
+          0.0013261501444503665,
+          0.019303444772958755,
+          0.0025133804883807898,
+          0.0037978545296937227,
+          -2.8652215405600145e-05,
+          -0.010455320589244366,
+          -0.006706621963530779,
+          -0.02374888025224209,
+          0.0007453583530150354,
+          0.005779065191745758,
+          0.01889309659600258,
+          0.0005540763959288597,
+          0.0030370012391358614,
+          0.026416141539812088,
+          -0.0029771588742733,
+          -0.014345075935125351,
+          -0.012592548504471779,
+          -0.011942830868065357,
+          -0.00874553807079792,
+          0.034332435578107834,
+          0.01454170048236847,
+          -0.01454170048236847,
+          -0.015234163030982018,
+          -0.011566678993403912,
+          0.03033154271543026,
+          0.00442833686247468,
+          -0.0004928982816636562,
+          0.021902315318584442,
+          0.018294673413038254,
+          0.017345745116472244,
+          0.016362618654966354,
+          0.009420901536941528,
+          0.0027912200894206762,
+          0.020466096699237823,
+          0.0019480835180729628,
+          0.015088831074535847,
+          -0.008172760717570782,
+          0.008642951026558876,
+          -0.005864554084837437,
+          0.012712233699858189,
+          -0.010677591897547245,
+          -0.011797499842941761,
+          0.008433502167463303,
+          -0.014003119431436062,
+          0.002117993077263236,
+          -0.007373437285423279,
+          0.00948074460029602,
+          -0.02017543464899063,
+          -0.007599983364343643,
+          0.012190749868750572,
+          -0.014533151872456074,
+          0.0024043817538768053,
+          -0.022893989458680153,
+          -0.0009761789697222412,
+          0.014695581048727036,
+          -0.0025368898641318083,
+          0.0066125839948654175,
+          0.005321698263287544,
+          0.00012549536768347025,
+          0.010104815475642681,
+          -0.014789619483053684,
+          0.004030812531709671,
+          -0.005377266090363264,
+          -0.004834410268813372,
+          -0.00637748884037137,
+          -0.0011712011182680726,
+          0.0009542724001221359,
+          -0.005556793417781591,
+          0.01991896703839302,
+          0.008091545663774014,
+          0.00435994565486908,
+          -0.005270404741168022,
+          -0.012831917963922024,
+          -0.026159673929214478,
+          0.0145758967846632,
+          -0.015447885729372501,
+          0.01528545655310154,
+          0.0013550026342272758,
+          -0.003092569299042225,
+          -0.014370722696185112,
+          0.010643396526575089,
+          0.0056337337009608746,
+          0.00474037230014801,
+          0.015114477835595608,
+          -0.011481190100312233,
+          0.009583331644535065,
+          0.009147336706519127,
+          -0.009267021901905537,
+          -0.0067493668757379055,
+          0.0005936151137575507,
+          0.006279176566749811,
+          -0.0232701413333416,
+          -0.0016862730262801051,
+          0.008292445912957191,
+          -0.01962830312550068,
+          0.008762635290622711,
+          -0.03730745613574982,
+          0.0020068571902811527,
+          -0.012088162824511528,
+          0.011258917860686779,
+          -0.002776259556412697,
+          -0.033374954015016556,
+          -0.026518728584051132,
+          0.0016745183384045959,
+          0.0040179891511797905,
+          -0.009993678890168667,
+          0.009010554291307926,
+          0.009275570511817932,
+          0.011532483622431755,
+          0.01443056482821703,
+          0.0098397983238101,
+          0.010113364085555077,
+          0.04223163053393364,
+          -0.04151352122426033,
+          -0.007146891206502914,
+          0.0070656766183674335,
+          0.02209039032459259,
+          -0.010267244651913643,
+          -0.011361504904925823,
+          0.008685695938766003,
+          -0.003859834047034383,
+          0.034349530935287476,
+          -0.024877335876226425,
+          0.014994793571531773,
+          -0.0040543219074606895,
+          -0.00010605995339574292,
+          -0.024159228429198265,
+          -0.0020325039513409138,
+          -0.01969669573009014,
+          0.0032357636373490095,
+          -0.002079523168504238,
+          -0.017191864550113678,
+          0.018021108582615852,
+          -0.004595040809363127,
+          0.012789173983037472,
+          0.007813706062734127,
+          -0.01625148393213749,
+          0.004526649136096239,
+          -0.015593216754496098,
+          0.032896216958761215,
+          -0.012105260975658894,
+          0.005911573302000761,
+          0.00873698852956295,
+          -0.006779287941753864,
+          0.016439558938145638,
+          0.009352510794997215,
+          -0.006044081412255764,
+          -0.0051592690870165825,
+          -0.002759161638095975,
+          -0.0023680487647652626,
+          0.008493345230817795,
+          -0.021474869921803474,
+          0.027493303641676903,
+          -0.006894698366522789,
+          -0.014037314802408218,
+          0.006266353186219931,
+          -0.001722605898976326,
+          0.002626653527840972,
+          0.004406964406371117,
+          -0.0018059578724205494,
+          -0.0007939802599139512,
+          -0.004958369303494692,
+          -0.017798837274312973,
+          -0.010130461305379868,
+          0.0010467075044289231,
+          -0.0016114701284095645,
+          0.013105483725667,
+          -0.010540809482336044,
+          -0.016601989045739174,
+          0.005112249869853258,
+          0.00026929075829684734,
+          -0.003975244704633951,
+          -0.016807163134217262,
+          0.007215282414108515,
+          -0.007544415537267923,
+          0.005372991785407066,
+          -0.008168485946953297,
+          0.016448108479380608,
+          0.008313817903399467,
+          -0.0003630616411101073,
+          0.01767915114760399,
+          0.024706358090043068,
+          0.02061997726559639,
+          0.01152393501251936,
+          -0.040214087814092636,
+          -0.0189443901181221,
+          -0.0025091059505939484,
+          0.018807608634233475,
+          -0.0008981701685115695,
+          -0.006420233752578497,
+          -0.0017632133094593883,
+          0.0029835705645382404,
+          -0.003084020223468542,
+          -0.0007474955637007952,
+          0.016918297857046127,
+          -0.020192531868815422,
+          0.0032742335461080074,
+          -0.012720782309770584,
+          0.014404918067157269,
+          0.0006860502762719989,
+          -0.0006737611838616431,
+          5.767176844528876e-05,
+          0.00945509783923626,
+          0.014900755137205124,
+          0.014661385677754879,
+          0.0014159136917442083,
+          0.019064076244831085,
+          -0.003663209266960621,
+          -0.0016510088462382555,
+          -0.018636630848050117,
+          -0.01696104370057583,
+          -0.0034494863357394934,
+          0.01298579853028059,
+          -0.0015601766062900424,
+          0.011575227603316307,
+          -0.014191195368766785,
+          0.01649085246026516,
+          0.025680935010313988,
+          -0.012840467505156994,
+          0.005069505423307419,
+          0.002564673777669668,
+          -0.001211808412335813,
+          0.0064800758846104145,
+          -0.007381986360996962,
+          -0.004030812531709671,
+          -0.012438667938113213,
+          0.01868792437016964,
+          -0.0032208028715103865,
+          0.018277576193213463,
+          0.013635516166687012,
+          0.011968477629125118,
+          -0.00032993461354635656,
+          0.008305269293487072,
+          0.018636630848050117,
+          0.014413466677069664,
+          -0.0005158734857104719,
+          0.0011615835828706622,
+          -0.01284901611506939,
+          0.0023594999220222235,
+          0.016311325132846832,
+          -0.02715134806931019,
+          -0.007685472723096609,
+          0.011840244755148888,
+          -0.016567792743444443,
+          0.0025603994727134705,
+          -0.02597159706056118,
+          -0.011472641490399837,
+          0.012053967453539371,
+          0.010233048349618912,
+          -0.011053744703531265,
+          0.001883966731838882,
+          0.001705508097074926,
+          -0.0008377934573218226,
+          -0.00936105940490961,
+          0.0167815163731575,
+          -0.005868828855454922,
+          -0.011258917860686779,
+          0.011686364188790321,
+          -0.00981415156275034,
+          0.012695135548710823,
+          0.004603589419275522,
+          0.011746206320822239,
+          -0.0024578124284744263,
+          0.026877781376242638,
+          0.01287466287612915,
+          0.007236654870212078,
+          0.0004386660875752568,
+          0.005937220063060522,
+          -0.004847233649343252,
+          0.007407633122056723,
+          0.026570022106170654,
+          -0.00048728802357800305,
+          -0.0116350706666708,
+          -0.0013197384541854262,
+          0.010951157659292221,
+          -0.01988477073609829,
+          0.0023124809376895428,
+          0.005676478147506714,
+          0.028211412951350212,
+          0.014327977783977985,
+          0.0025796345435082912,
+          -0.0204148031771183,
+          -0.00023709874949418008,
+          -0.00041595802758820355,
+          0.04106897860765457,
+          0.0026116929948329926,
+          -0.012532706372439861,
+          0.0021489830687642097,
+          0.013849238865077496,
+          -0.005590988788753748,
+          -0.004834410268813372,
+          -0.014310880564153194,
+          -0.008027428761124611,
+          -0.006018434651196003,
+          -0.0198676735162735,
+          0.001279131043702364,
+          0.005800437182188034,
+          -0.013319206424057484,
+          0.007471750024706125,
+          -0.006403135601431131,
+          -0.03031444549560547,
+          -0.005932945758104324,
+          -0.0034131535794585943,
+          -0.023731783032417297,
+          -0.007125518750399351,
+          0.0089678093791008,
+          0.005915847606956959,
+          -0.014122803695499897,
+          -0.025338977575302124,
+          0.020996129140257835,
+          0.007668375037610531,
+          0.03392208740115166,
+          -0.012669488787651062,
+          -0.007185361348092556,
+          -0.00948074460029602,
+          0.008557462133467197,
+          -0.023150455206632614,
+          -0.008715616539120674,
+          0.009061847813427448,
+          0.01767915114760399,
+          0.012404472567141056,
+          -0.011370054446160793,
+          0.0015163634670898318,
+          0.012122358195483685,
+          -0.00010085045505547896,
+          0.02764718420803547,
+          -0.009104591794312,
+          -0.014994793571531773,
+          0.010754532180726528,
+          -0.003259273013100028,
+          0.01916666328907013,
+          -0.0006203305092640221,
+          -0.0025924579240381718,
+          0.012917407788336277,
+          0.0038341875188052654,
+          -0.012934505008161068,
+          -0.005240483675152063,
+          0.008647225797176361,
+          -0.028553368523716927,
+          0.02792074903845787,
+          0.015909526497125626,
+          -0.02816011942923069,
+          -0.008305269293487072,
+          -0.0025091059505939484,
+          0.017140571027994156,
+          0.020089944824576378,
+          -0.011446994729340076,
+          -0.009728662669658661,
+          0.014960597269237041,
+          0.0034217024222016335,
+          0.0077538639307022095,
+          -0.0026565745938569307,
+          0.01896148920059204,
+          -0.004370631650090218,
+          -0.007800883147865534,
+          -0.003408879041671753,
+          -0.012789173983037472,
+          -0.0003505054337438196,
+          0.015618863515555859,
+          0.012977249920368195,
+          -0.005984238814562559,
+          -0.008771184831857681,
+          0.015208516269922256,
+          -0.022432347759604454,
+          0.0010841090697795153,
+          0.010908412747085094,
+          -0.01745687983930111,
+          -0.007561513222754002,
+          0.01767915114760399,
+          -0.0031395882833749056,
+          0.011258917860686779,
+          -0.007381986360996962,
+          0.007856450974941254,
+          0.0067237201146781445,
+          0.031135139986872673,
+          -0.007476024329662323,
+          0.03525571525096893,
+          -0.009309765882790089,
+          0.01991896703839302,
+          0.037991367280483246,
+          0.01455025002360344,
+          -0.0031118043698370457,
+          -0.008335189893841743,
+          0.009198630228638649,
+          -0.010096265934407711,
+          0.002415067981928587,
+          0.013524380512535572,
+          -0.026809390634298325,
+          -0.01418264675885439,
+          -0.011378603056073189,
+          0.013233717530965805,
+          0.0034580351784825325,
+          -0.013661162927746773,
+          -0.011575227603316307,
+          0.019508618861436844,
+          0.010352733545005322,
+          -0.0038705202750861645,
+          -0.007869274355471134,
+          -0.02157745696604252,
+          0.0020688369404524565,
+          0.012088162824511528,
+          -0.009164434857666492,
+          -0.011532483622431755,
+          0.016670379787683487,
+          -0.01261819526553154,
+          0.0027228286489844322,
+          -0.008300994522869587,
+          -0.027989141643047333,
+          -0.005800437182188034,
+          -0.00491989916190505,
+          -0.0019844165071845055,
+          -0.005565342493355274,
+          0.006471527274698019,
+          0.005227660294622183,
+          -0.01824337989091873,
+          0.0024278913624584675,
+          -0.008788282051682472,
+          0.00861302949488163,
+          0.00436635734513402,
+          0.006249255500733852,
+          -0.0019384659826755524,
+          0.014225390739738941,
+          0.013661162927746773,
+          -0.005377266090363264,
+          -0.020773857831954956,
+          0.012772075831890106,
+          -0.03634997829794884,
+          0.020722564309835434,
+          0.008694244548678398,
+          0.0025411644019186497,
+          -0.03086157515645027,
+          0.01018175482749939,
+          0.01115633174777031,
+          0.0160035640001297,
+          0.011370054446160793,
+          0.0016766554908826947,
+          0.013190972618758678,
+          -0.01114778220653534,
+          0.006766464561223984,
+          -7.192273642431246e-06,
+          -0.01482381485402584,
+          0.020688369870185852,
+          0.011814597994089127,
+          0.02913469448685646,
+          0.002951512113213539,
+          -0.0011209761723876,
+          0.001963044051080942,
+          0.00899345614016056,
+          -0.007685472723096609,
+          0.0030904319137334824,
+          0.004954094998538494,
+          0.0058175353333354,
+          0.013190972618758678,
+          0.007745314855128527,
+          -0.00860875565558672,
+          0.007839352823793888,
+          0.0026736725121736526,
+          -0.014020216651260853,
+          -0.000775279535446316,
+          -0.006997285410761833,
+          0.007552964612841606,
+          -0.01505463570356369,
+          0.004560844972729683,
+          0.04342847689986229,
+          0.012413021177053452,
+          -0.0034323884174227715,
+          -0.017354292795062065,
+          -0.011079391464591026,
+          -0.01273788046091795,
+          -0.01272933091968298,
+          -0.0053644427098333836,
+          -0.02303077094256878,
+          -0.003434525802731514,
+          0.0065912120044231415,
+          0.02453538030385971,
+          0.01092551089823246,
+          -0.017191864550113678,
+          -0.005813260562717915,
+          -0.02380017377436161,
+          -0.010566456243395805,
+          -0.016191640868782997,
+          -0.016319874674081802,
+          -0.008061625063419342,
+          0.018978586420416832,
+          -0.008839575573801994,
+          0.01408860832452774,
+          0.002968610031530261,
+          -0.0002393695613136515,
+          -0.0020431901793926954,
+          -0.000852754048537463,
+          -0.0051549943163990974,
+          0.0015911663649603724,
+          0.003152411663904786,
+          -0.0005375128821469843,
+          0.020722564309835434,
+          -0.007800883147865534,
+          0.003866245737299323,
+          -0.0026929075829684734,
+          0.01542223896831274,
+          0.007108421064913273,
+          0.0028681603725999594,
+          -0.0238172709941864,
+          0.02304787002503872,
+          -0.024381499737501144,
+          -0.014156999997794628,
+          0.02284269593656063,
+          -0.008108643814921379,
+          0.004804489202797413,
+          0.024107934907078743,
+          0.00029493749025277793,
+          -0.0009697672794573009,
+          0.015123026445508003,
+          0.004045772831887007,
+          0.011258917860686779,
+          0.015242711640894413,
+          0.007082774303853512,
+          -0.0052875024266541,
+          0.019098270684480667,
+          0.0073306928388774395,
+          -0.006403135601431131,
+          -0.01626003161072731,
+          0.0014244626509025693,
+          0.004238123539835215,
+          -0.010044972412288189,
+          -0.004137673880904913,
+          0.008630127646028996,
+          0.004453983623534441,
+          -0.007843627594411373,
+          -0.0012972975382581353,
+          -0.024706358090043068,
+          -0.005821809638291597,
+          -0.01530255377292633,
+          -0.006039807107299566,
+          -0.023406922817230225,
+          0.005676478147506714,
+          -0.009053298272192478,
+          0.01672167330980301,
+          0.00016403225890826434,
+          0.024347303435206413,
+          0.0020154062658548355,
+          -0.0007266575703397393,
+          -0.0051592690870165825,
+          0.009318315424025059,
+          -0.0007266575703397393,
+          0.027236836031079292,
+          -0.00011026761785615236,
+          -0.005911573302000761,
+          0.01032708678394556,
+          -0.01769625023007393,
+          0.020876444876194,
+          0.006356116849929094,
+          0.01454170048236847,
+          -0.015610314905643463,
+          -0.0073563395999372005,
+          0.035289913415908813,
+          0.024843141436576843,
+          0.0035029170103371143,
+          0.019816379994153976,
+          -0.010455320589244366,
+          0.016311325132846832,
+          0.003013491863384843,
+          -0.002389421220868826,
+          0.011010999791324139,
+          -0.005420010536909103,
+          -0.014627190306782722,
+          -0.006997285410761833,
+          0.03911982476711273,
+          -0.014832363463938236,
+          -6.972707342356443e-05,
+          -0.005060956347733736,
+          -0.010104815475642681,
+          -0.030929965898394585,
+          0.011609423905611038,
+          0.011951380409300327,
+          -0.024706358090043068,
+          -0.011566678993403912,
+          0.0032699592411518097,
+          -0.00026448199059814215,
+          -0.0030348640866577625,
+          -0.005706399213522673,
+          0.006509996950626373,
+          -0.0017375665483996272,
+          0.009275570511817932,
+          0.005569616798311472,
+          -0.0009617527248337865,
+          0.00013798479631077498,
+          0.005539695732295513,
+          0.011780401691794395,
+          0.006206510588526726,
+          0.0005498019745573401,
+          -0.007381986360996962,
+          0.00558671448379755,
+          0.010745983570814133,
+          -0.015088831074535847,
+          0.020739663392305374,
+          0.013823592104017735,
+          -0.016576342284679413,
+          0.0038149524480104446,
+          -0.019132466986775398,
+          -0.0329304113984108,
+          0.004787391051650047,
+          -0.0009029789362102747,
+          0.021748434752225876,
+          -0.018311770632863045,
+          -0.00995948351919651,
+          0.00442833686247468,
+          0.003908990416675806,
+          -0.011421347968280315,
+          0.019371837377548218,
+          0.007215282414108515,
+          -0.0017664191545918584,
+          0.01745687983930111,
+          0.016593439504504204,
+          -0.006403135601431131,
+          -0.00907894503325224,
+          0.018824705854058266,
+          -0.00437276903539896,
+          0.02744201011955738,
+          0.026621313765645027,
+          0.009207178838551044,
+          -0.018602434545755386,
+          0.0018262615194544196,
+          0.0080231549218297,
+          -0.02036350965499878,
+          -0.005590988788753748,
+          0.015670157968997955,
+          0.005402912851423025,
+          -0.0003841667785309255,
+          -0.0050737797282636166,
+          -0.022175880149006844,
+          0.007407633122056723,
+          0.011840244755148888,
+          -0.005189190153032541,
+          -0.028809836134314537,
+          -0.0011199075961485505,
+          -0.01431942917406559,
+          -0.0007694021333009005,
+          0.0010648739989846945,
+          0.01236172765493393,
+          0.009036201052367687,
+          0.028074629604816437,
+          -0.009865445084869862,
+          -0.00885667372494936,
+          0.010173206217586994,
+          0.009797054342925549,
+          0.012583999894559383,
+          0.012327532283961773,
+          0.013986021280288696,
+          0.007292222697287798,
+          -0.015960820019245148,
+          -0.004966918379068375,
+          0.007527317851781845,
+          -0.00899345614016056,
+          0.003985930699855089,
+          -0.01358422264456749,
+          0.020278021693229675,
+          0.015259809792041779,
+          0.0008260387112386525,
+          0.004804489202797413,
+          -0.004753195680677891,
+          -0.0012684449320659041,
+          -0.06483495235443115,
+          -0.013532929122447968,
+          0.00861302949488163,
+          0.021252596750855446,
+          -0.016396814957261086,
+          0.00046244275290519,
+          0.0014287370722740889,
+          0.008788282051682472,
+          -0.011481190100312233,
+          0.01020740158855915,
+          0.014524603262543678,
+          -0.005223385989665985,
+          -0.0035349754616618156,
+          0.005099426489323378,
+          -0.0016403226181864738,
+          -0.00013371034583542496,
+          0.004958369303494692,
+          0.014900755137205124,
+          0.021406477317214012,
+          -0.004385592415928841,
+          0.018585337325930595,
+          0.004112027119845152,
+          -0.015080282464623451,
+          -0.0027014564257115126,
+          0.013575674034655094,
+          -0.02328723855316639,
+          -0.004607864189893007,
+          0.007480298634618521,
+          -0.006980187259614468,
+          0.010908412747085094,
+          -0.004069282673299313,
+          -0.004406964406371117,
+          0.008677146397531033,
+          0.01420829351991415,
+          0.002276147948578,
+          -0.026775194332003593,
+          0.0034665842540562153,
+          -0.011233271099627018,
+          0.0035093287006020546,
+          0.01020740158855915,
+          -0.008219779469072819,
+          1.5344630810432136e-05,
+          -0.015328200533986092,
+          -0.002739926567301154,
+          0.024860238656401634,
+          -0.004289417061954737,
+          0.0067707388661801815,
+          0.017302999272942543,
+          -0.007018657401204109,
+          0.002389421220868826,
+          -0.02862176112830639,
+          0.007796608377248049,
+          -0.0369655005633831,
+          -0.0004579011583700776,
+          0.004984016064554453,
+          0.004097066354006529,
+          -0.010104815475642681,
+          -0.030211858451366425,
+          0.008412130177021027,
+          -0.00474037230014801,
+          -0.006753641180694103,
+          -0.022723009809851646,
+          -0.007937666028738022,
+          0.008429228328168392,
+          -0.006941717118024826,
+          0.010147559456527233,
+          -0.017662053927779198,
+          0.017815934494137764,
+          0.015439336188137531,
+          0.013302108272910118,
+          0.003968833014369011,
+          0.0032379007898271084,
+          0.005129347555339336,
+          0.007946214638650417,
+          0.004701902158558369,
+          0.004697627853602171,
+          -0.01019030436873436,
+          -0.011464091949164867,
+          0.01575564593076706,
+          0.004255221225321293,
+          0.00860875565558672,
+          -0.0018604571232572198,
+          0.002479184651747346,
+          -0.0073050460778176785,
+          -0.00045629823580384254,
+          -0.014396369457244873,
+          0.0021596690639853477,
+          0.01199412439018488,
+          0.015499179251492023,
+          0.0014650699449703097,
+          -0.00625352980569005,
+          -0.007634179200977087,
+          -0.00667670089751482,
+          0.020209630951285362,
+          -0.017149118706583977,
+          -0.006698073353618383,
+          -0.012635293416678905,
+          0.021526163443922997,
+          0.01599501632153988,
+          0.0035563476849347353,
+          -0.006826307158917189,
+          -0.004092792049050331,
+          0.0097030159085989,
+          0.008570285513997078,
+          0.006766464561223984,
+          -0.023663390427827835,
+          -0.005377266090363264,
+          -0.019081173464655876,
+          -0.01539659220725298,
+          -0.01406296156346798,
+          0.021218402311205864,
+          -0.004924173932522535,
+          -0.007471750024706125,
+          0.008454875089228153,
+          0.024655064567923546,
+          0.007916293106973171,
+          0.007715393789112568,
+          0.00503958435729146,
+          -0.01726025529205799,
+          0.012071064673364162,
+          -0.010139010846614838,
+          -0.03665773943066597,
+          0.0065912120044231415,
+          -0.00971156544983387,
+          0.02598869428038597,
+          -0.017183315008878708,
+          -0.0011167017510160804,
+          -0.002635202370584011,
+          -0.030878672376275063,
+          -0.005638008005917072,
+          0.02961343340575695,
+          -0.010737434960901737,
+          0.011669266037642956,
+          -0.010062070563435555,
+          -0.009950934909284115,
+          0.01019030436873436,
+          0.013062738813459873,
+          -0.011720559559762478,
+          -0.002940826117992401,
+          0.014721227809786797,
+          0.02453538030385971,
+          0.0035093287006020546,
+          -0.004548021592199802,
+          0.009429451078176498,
+          0.0123702771961689,
+          -0.006804934702813625,
+          0.019559912383556366,
+          -0.007561513222754002,
+          -0.0015078145079314709,
+          -0.00656556524336338,
+          0.0012235631002113223,
+          0.006193687207996845,
+          -0.019577009603381157,
+          0.017320098355412483,
+          -0.009916738606989384,
+          0.008873771876096725,
+          0.0028702975250780582,
+          0.01726025529205799,
+          0.011267467401921749,
+          0.01896148920059204,
+          -0.02891242317855358,
+          -0.001095329411327839,
+          -0.0276813805103302,
+          -0.015772745013237,
+          -0.010703238658607006,
+          -0.00873698852956295,
+          -0.002485596342012286,
+          -0.007796608377248049,
+          -0.023868564516305923,
+          -0.005625184625387192,
+          0.0007939802599139512,
+          0.0036781697999686003,
+          -0.017362842336297035,
+          0.01007061917334795,
+          1.5286190318875015e-05,
+          -0.016593439504504204,
+          -0.020825151354074478,
+          -0.002087003318592906,
+          -0.022021999582648277,
+          -0.012464314699172974,
+          -0.005005388520658016,
+          -0.0022248546592891216,
+          -0.0012139456812292337,
+          -0.0005172092351131141,
+          0.027544597163796425,
+          -0.019457325339317322,
+          0.004381317645311356,
+          0.02744201011955738,
+          -0.0019352601375430822,
+          0.005035309586673975,
+          -0.006668151821941137,
+          0.012532706372439861,
+          0.0032913314644247293,
+          -0.003973107319325209,
+          -0.022688815370202065,
+          0.02521929331123829,
+          0.0005311011918820441,
+          0.013669711537659168,
+          0.010139010846614838,
+          -0.008065898902714252,
+          -0.005783339496701956,
+          0.00485150795429945,
+          -0.03253716230392456,
+          0.0028531996067613363,
+          -0.013011445291340351,
+          -0.01865372806787491,
+          -0.005539695732295513,
+          -0.004586491733789444,
+          -0.006561290472745895,
+          -0.0071939099580049515,
+          0.017285902053117752,
+          -0.004377043340355158,
+          0.02376597747206688,
+          -0.007146891206502914,
+          -0.00453947251662612,
+          0.010908412747085094,
+          -0.04301812872290611,
+          -0.007240929175168276,
+          -0.006129570305347443,
+          -0.01843145675957203,
+          0.019799282774329185,
+          0.012173651717603207,
+          -0.001380649395287037,
+          -0.020243825390934944,
+          0.019816379994153976,
+          -0.0030028056353330612,
+          0.02552705444395542,
+          0.010634847916662693,
+          0.010173206217586994,
+          0.004364219959825277,
+          -0.010840021073818207,
+          -0.010113364085555077,
+          0.018311770632863045,
+          0.006304823327809572,
+          -0.011609423905611038,
+          0.01873921789228916,
+          0.004445434547960758,
+          -0.00186686881352216,
+          -0.003041275776922703,
+          0.018756315112113953,
+          0.024603771045804024,
+          0.013002896681427956,
+          0.002274010796099901,
+          -0.016174543648958206,
+          -0.006475801579654217,
+          0.0037038165610283613,
+          0.036418367177248,
+          0.012036869302392006,
+          0.012994347140192986,
+          -0.008202682249248028,
+          0.027373619377613068,
+          -0.009181532077491283,
+          0.009044749662280083,
+          0.00863440241664648,
+          0.017174765467643738,
+          0.0035328383091837168,
+          0.020329315215349197,
+          0.010857119224965572,
+          -0.00764272827655077,
+          0.02405664138495922,
+          0.007625630125403404,
+          0.003767933463677764,
+          0.006625407375395298,
+          -0.01844855397939682,
+          0.012626743875443935,
+          -0.011609423905611038,
+          -0.005950043443590403,
+          0.004505277145653963,
+          0.002058150712400675,
+          0.02374888025224209,
+          0.008027428761124611,
+          -0.03135741129517555,
+          -0.0017867228016257286,
+          -0.023612096905708313,
+          0.004340710584074259,
+          -0.0167387705296278,
+          0.012447217479348183,
+          0.004092792049050331,
+          -0.01774754375219345,
+          -0.018790511414408684,
+          -0.010728885419666767,
+          -0.01942313089966774,
+          -0.004924173932522535,
+          -0.012173651717603207,
+          -0.010840021073818207,
+          0.006954540498554707,
+          -0.0025069687981158495,
+          0.0024813220370560884,
+          0.006963089574128389,
+          0.019046977162361145,
+          0.020260922610759735,
+          0.0035819944459944963,
+          0.018824705854058266,
+          -0.0009019103017635643,
+          -0.055431149899959564,
+          0.0036589347291737795,
+          0.009309765882790089,
+          -0.0002863885893020779,
+          0.026108380407094955,
+          -0.004041498526930809,
+          0.00042557556298561394,
+          0.007223831489682198,
+          0.006937442813068628,
+          0.013276461511850357,
+          -0.004210339393466711,
+          -0.025886109098792076,
+          -0.019337641075253487,
+          0.0040393611416220665,
+          -0.006155217066407204,
+          0.008100095205008984,
+          -0.006685249973088503,
+          0.004761744290590286,
+          0.002353088231757283,
+          -0.009745760820806026,
+          -0.006595486309379339,
+          0.009617527015507221,
+          0.0014469034504145384,
+          -0.010386928915977478,
+          -0.013173874467611313,
+          -0.0018690060824155807,
+          0.01824337989091873,
+          -0.009019102901220322,
+          -0.024364402517676353,
+          0.009660271927714348,
+          -0.010010777041316032,
+          -0.0014821677468717098,
+          0.025869010016322136,
+          0.02429600991308689,
+          -0.003970969934016466,
+          0.019371837377548218,
+          -0.0021190617699176073,
+          -0.016405364498496056,
+          -0.03029734641313553,
+          -0.006873325910419226,
+          -0.028997913002967834,
+          0.027253935113549232,
+          -0.006424508057534695,
+          -0.0013176011852920055,
+          0.0035285637713968754,
+          -0.00413553649559617,
+          0.003434525802731514,
+          -0.009788504801690578,
+          -0.007672649342566729,
+          -0.02716844528913498,
+          0.007527317851781845,
+          -0.0069075217470526695,
+          -0.008172760717570782,
+          0.00031550831045024097,
+          0.039701152592897415,
+          -0.004556570667773485,
+          0.00850189384073019,
+          0.006232157349586487,
+          0.011865891516208649,
+          0.024193422868847847,
+          -0.008463423699140549,
+          -0.012096711434423923,
+          -0.015046087093651295,
+          -0.0020154062658548355,
+          0.00014052275219000876,
+          -0.003975244704633951,
+          0.022876890376210213,
+          -0.013019993901252747,
+          -0.015191418118774891,
+          -0.00034489520476199687,
+          -0.023389825597405434,
+          -0.011737657710909843,
+          0.009968032129108906,
+          0.012908858247101307,
+          0.007535866927355528,
+          -0.014003119431436062,
+          -0.014268135651946068,
+          -0.022945282980799675,
+          0.01800401136279106,
+          0.0976627841591835,
+          0.0012545529752969742,
+          -0.009070396423339844,
+          0.0026758096646517515,
+          0.010771630331873894,
+          -0.0008372591692022979,
+          -0.017559466883540154,
+          0.05919267237186432,
+          -0.013567124493420124,
+          0.005856005474925041,
+          -0.009010554291307926,
+          0.01007916871458292,
+          0.005031035281717777,
+          -0.00013257493264973164,
+          0.005928670987486839,
+          -0.008796831592917442,
+          0.006129570305347443,
+          -0.03645256534218788,
+          -0.00186686881352216,
+          -0.011344407685101032,
+          -0.006381763610988855,
+          -0.016824260354042053,
+          0.009275570511817932,
+          -0.02911759726703167,
+          -0.00922427698969841,
+          0.0020880720112472773,
+          -0.0059543177485466,
+          0.012336080893874168,
+          -0.01628567837178707,
+          -0.0017995461821556091,
+          0.005368717480450869,
+          0.005120798945426941,
+          -0.0036867186427116394,
+          -0.0074632009491324425,
+          -0.013310656882822514,
+          0.005065230652689934,
+          0.015576119534671307,
+          0.005548244342207909,
+          -0.008228329010307789,
+          0.00959188025444746,
+          0.016388265416026115,
+          0.0009510665549896657,
+          0.013661162927746773,
+          0.009412352927029133,
+          0.0007950488943606615,
+          0.0007875685696490109,
+          0.019611205905675888,
+          -0.016345521435141563,
+          0.02137228287756443,
+          -0.015883879736065865,
+          -0.02843368425965309,
+          -0.006236432120203972,
+          0.006411684677004814,
+          -0.014413466677069664,
+          0.0010910549899563193,
+          0.006582662928849459,
+          0.04100058600306511,
+          0.00863440241664648,
+          -0.0034281141124665737,
+          -0.0009259541402570903,
+          -0.013669711537659168,
+          0.023954054340720177,
+          0.01091696135699749,
+          0.030502520501613617,
+          0.007604258134961128,
+          0.00373373762704432,
+          -0.007001559715718031,
+          -0.009557684883475304,
+          -0.003487956477329135,
+          0.00474037230014801,
+          0.0025069687981158495,
+          0.004415513481944799,
+          0.008523265831172466,
+          -0.014498956501483917,
+          -0.02036350965499878,
+          0.017610760405659676,
+          -0.00474037230014801,
+          -0.023663390427827835,
+          0.012609646655619144,
+          -0.0316309779882431,
+          0.0047831167466938496,
+          0.0010344184702262282,
+          -0.005877377465367317,
+          -0.013960374519228935,
+          0.006300548557192087,
+          0.01151538547128439,
+          -0.009378157556056976,
+          -0.015482081100344658,
+          -0.008454875089228153,
+          -0.010592103004455566,
+          0.0001227570464834571,
+          0.0037657960783690214,
+          -0.011583777144551277,
+          -0.00394318625330925,
+          -0.01796981506049633,
+          0.007142616901546717,
+          -0.003013491863384843,
+          0.004654882941395044,
+          -0.011122135445475578,
+          0.011558130383491516,
+          0.020226728171110153,
+          -0.009788504801690578,
+          -0.007069950923323631,
+          0.013105483725667,
+          0.003400330198928714,
+          0.016089053824543953,
+          0.03368271514773369,
+          -0.0033704089000821114,
+          0.0033511738292872906,
+          0.003849148051813245,
+          0.0058731031604111195,
+          0.0008874839986674488,
+          0.015909526497125626,
+          0.03621319308876991,
+          0.01942313089966774,
+          -0.0250141192227602,
+          0.02186811901628971,
+          -0.019713792949914932,
+          0.021936509758234024,
+          -0.02231266349554062,
+          -0.0030797459185123444,
+          0.007005834020674229,
+          -0.007394809741526842,
+          0.004146222956478596,
+          0.00624070642516017,
+          0.017627857625484467,
+          0.022363955155014992,
+          0.000617124664131552,
+          -0.0003163097717333585,
+          0.004860057029873133,
+          0.021953608840703964,
+          -0.0025411644019186497,
+          0.007125518750399351,
+          -0.013421793468296528,
+          0.004411239176988602,
+          -0.007202459033578634,
+          0.0021767669823020697,
+          0.008395032025873661,
+          -0.014952048659324646,
+          0.013344853185117245,
+          0.012891761027276516,
+          -0.017302999272942543,
+          0.0043342988938093185,
+          0.0025945950765162706,
+          -0.018568238243460655,
+          -0.018978586420416832,
+          0.023372728377580643,
+          -0.013635516166687012,
+          -0.012660940177738667,
+          -0.009891091845929623,
+          -0.01152393501251936,
+          -0.0201412383466959,
+          -0.005056682042777538,
+          -0.0017471840837970376,
+          0.005753418430685997,
+          -0.0024449890479445457,
+          0.006467252504080534,
+          0.01573854871094227,
+          0.005612361244857311,
+          -0.006437331438064575,
+          -0.005561067722737789,
+          -0.0015163634670898318,
+          0.020107043907046318,
+          0.005013937596231699,
+          0.0026458885986357927,
+          -0.04031667485833168,
+          0.0013143953401595354,
+          -0.010540809482336044,
+          0.006428782362490892,
+          0.0031417254358530045,
+          -0.004971192684024572,
+          -0.01942313089966774,
+          -0.014755423180758953,
+          0.01872211880981922,
+          -0.0103014400228858,
+          0.024637967348098755,
+          4.564986011246219e-05,
+          -0.012053967453539371,
+          -0.0018999958410859108,
+          0.003960283938795328,
+          0.0016029211692512035,
+          0.004415513481944799,
+          -0.00800178200006485,
+          0.016174543648958206,
+          -0.011694912798702717,
+          0.010908412747085094,
+          -0.00401157746091485,
+          -0.024706358090043068,
+          0.010891314595937729,
+          -0.01744833029806614,
+          0.0004752661334350705,
+          -0.00666387751698494,
+          -0.030040880665183067,
+          0.018551141023635864,
+          -0.023731783032417297,
+          -0.0033233899157494307,
+          0.009643173776566982,
+          0.018841803073883057,
+          0.0017461155075579882,
+          -0.010976804420351982,
+          0.00630909763276577,
+          0.017362842336297035,
+          0.013481635600328445,
+          -0.004062870983034372,
+          0.012438667938113213,
+          -0.004971192684024572,
+          0.0025603994727134705,
+          0.005172092467546463,
+          0.008399306796491146,
+          -0.006804934702813625,
+          0.007805157452821732
+        ],
+        "keyphrases": [
+          "DeepSeekMoE",
+          "multilingual corpus",
+          "HuggingFace Tokenizer",
+          "MoE architectures",
+          "evaluation benchmarks"
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "95f27e84-0fb9-44ee-a783-45f42551401c",
+      "properties": {
+        "page_content": "5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n",
+        "title": "Scaling Up To Deepseekmoe 16B",
+        "keyphrases": [
+          "DeepSeekMoE 16B",
+          "MoE model",
+          "language modeling",
+          "evaluation benchmarks",
+          "Open LLM Leaderboard"
+        ],
+        "embedding": [
+          0.005968157202005386,
+          -0.015490485355257988,
+          -0.027613861486315727,
+          0.012274806387722492,
+          0.008261889219284058,
+          -0.01577553153038025,
+          -0.016158562153577805,
+          0.06238951161503792,
+          -0.002393943490460515,
+          0.048564765602350235,
+          0.008756265975534916,
+          -0.009914266876876354,
+          0.03532792627811432,
+          -0.018332039937376976,
+          -0.006921280641108751,
+          0.023961706086993217,
+          0.018617086112499237,
+          0.02068367227911949,
+          -0.00015922509192023426,
+          -0.02930632419884205,
+          -0.021538810804486275,
+          -0.03865940496325493,
+          -0.037447959184646606,
+          -0.023801365867257118,
+          -0.021129056811332703,
+          0.012862714938819408,
+          0.01521434634923935,
+          0.015909146517515182,
+          -0.041616760194301605,
+          0.008319789543747902,
+          0.016042763367295265,
+          0.021449735388159752,
+          0.004554060287773609,
+          0.009063581936061382,
+          0.012675653211772442,
+          -0.015294515527784824,
+          0.07097653299570084,
+          -0.0016189739108085632,
+          -0.009361989796161652,
+          0.010226036421954632,
+          0.018777426332235336,
+          0.010083513334393501,
+          -0.009317451156675816,
+          -0.010787221603095531,
+          0.019899794831871986,
+          0.012176821939647198,
+          -0.00043619878124445677,
+          -0.0028460093308240175,
+          0.00021837776876054704,
+          -0.021592257544398308,
+          -0.027720753103494644,
+          -0.013682222925126553,
+          0.0055272262543439865,
+          -0.006075049284845591,
+          -0.017797578126192093,
+          -0.005594033747911453,
+          0.03819620609283447,
+          0.0277029387652874,
+          0.03762611374258995,
+          -0.01240842230618,
+          0.03308318555355072,
+          -0.017494717612862587,
+          -0.006288834381848574,
+          -2.058165227936115e-05,
+          -0.006248749326914549,
+          -0.034597497433423996,
+          -0.005117471795529127,
+          0.001308317994698882,
+          0.005865718703716993,
+          -0.024068597704172134,
+          -0.04464537650346756,
+          0.02754260040819645,
+          -0.003171140095219016,
+          0.00048101565334945917,
+          0.02171696536242962,
+          0.0010444275103509426,
+          0.01798463985323906,
+          0.007918942719697952,
+          -0.02321345917880535,
+          -0.014279038645327091,
+          0.020505519583821297,
+          0.010386374779045582,
+          -0.006529341917484999,
+          0.025565091520547867,
+          0.06046544760465622,
+          -0.01011914387345314,
+          -0.024959366768598557,
+          -0.08444496989250183,
+          7.453237049048766e-05,
+          -0.0283442921936512,
+          -0.011490928940474987,
+          0.007825411856174469,
+          -0.004527336917817593,
+          -0.019062472507357597,
+          -0.0022169030271470547,
+          0.025547275319695473,
+          -0.005602941382676363,
+          -0.006342280190438032,
+          -0.017957916483283043,
+          0.03791115805506706,
+          -0.002776974579319358,
+          -0.004046321380883455,
+          0.030054569244384766,
+          0.0013561969390138984,
+          0.016888992860913277,
+          -0.0027858822140842676,
+          -0.02565416693687439,
+          0.029965493828058243,
+          -0.0067653958685696125,
+          0.08929075300693512,
+          -0.014564084820449352,
+          -0.009691574610769749,
+          -0.050453197211027145,
+          -0.011063359677791595,
+          -0.014083068817853928,
+          -0.003999555949121714,
+          -0.03424118831753731,
+          0.001817170181311667,
+          0.011588914319872856,
+          -0.014368114992976189,
+          0.013682222925126553,
+          0.021859489381313324,
+          0.03887319192290306,
+          -0.026990322396159172,
+          -0.04343393072485924,
+          0.06506182253360748,
+          0.021503180265426636,
+          -0.014804592356085777,
+          0.013824746012687683,
+          0.003099878551438451,
+          -0.01950785703957081,
+          0.011366221122443676,
+          -0.022393951192498207,
+          -0.013334822840988636,
+          0.017254209145903587,
+          -0.00778978131711483,
+          0.026152998208999634,
+          0.0797417014837265,
+          -0.018332039937376976,
+          0.014973838813602924,
+          0.022108905017375946,
+          -0.029217246919870377,
+          0.012372790835797787,
+          0.0028148323763161898,
+          -0.034989435225725174,
+          0.0058924416080117226,
+          0.008894335478544235,
+          0.05825633928179741,
+          0.003538582706823945,
+          0.010929744690656662,
+          0.010974282398819923,
+          0.011098990216851234,
+          -0.0009859707206487656,
+          0.023997336626052856,
+          0.008497943170368671,
+          0.0018372124759480357,
+          -0.026527121663093567,
+          0.01851019449532032,
+          -0.018813056871294975,
+          0.00869391206651926,
+          0.012167913839221,
+          -0.022661181166768074,
+          -0.012417329475283623,
+          -0.002204654971137643,
+          0.004934864118695259,
+          0.05184279754757881,
+          0.020024502649903297,
+          -0.010555621236562729,
+          -0.0979846715927124,
+          0.050132520496845245,
+          0.014145422726869583,
+          0.024923736229538918,
+          -0.04197306931018829,
+          0.017476901412010193,
+          0.009647035971283913,
+          0.028878754004836082,
+          0.0057677337899804115,
+          0.0020532242488116026,
+          -0.06816169619560242,
+          0.03545263409614563,
+          0.011419667862355709,
+          0.024994997307658195,
+          0.02485247515141964,
+          0.008840889669954777,
+          0.014065253548324108,
+          0.0004027949180454016,
+          0.04058346897363663,
+          0.029573554173111916,
+          -0.005389156751334667,
+          0.0357198640704155,
+          0.036236513406038284,
+          -0.0020799473859369755,
+          0.04428907111287117,
+          0.003500724909827113,
+          0.015089638531208038,
+          -0.006609511096030474,
+          -0.027328815311193466,
+          0.04963368922472,
+          -0.01895557902753353,
+          0.024157674983143806,
+          0.012069929391145706,
+          -0.03119475580751896,
+          -0.011125713586807251,
+          -0.01026166696101427,
+          0.028611524030566216,
+          -0.01782430149614811,
+          0.0019407644867897034,
+          0.02168133482336998,
+          0.009451067075133324,
+          -0.008163904771208763,
+          -0.026919061318039894,
+          -0.0066406880505383015,
+          -0.028967831283807755,
+          -0.04175928235054016,
+          0.0026010475121438503,
+          0.02702595293521881,
+          -0.012755822390317917,
+          -0.00048185072955675423,
+          -0.010742682963609695,
+          -0.030820632353425026,
+          -0.009932082146406174,
+          0.03388487920165062,
+          0.007901127450168133,
+          -0.011802698485553265,
+          -0.02971607819199562,
+          -0.014403746463358402,
+          0.01066251378506422,
+          -0.0017403413075953722,
+          0.05105891823768616,
+          -0.01533905416727066,
+          0.020648041740059853,
+          0.011437483131885529,
+          0.0065605188719928265,
+          0.02424675226211548,
+          0.02437146008014679,
+          -0.016559408977627754,
+          0.005843449383974075,
+          -0.003349294187501073,
+          -0.0055183181539177895,
+          0.043505191802978516,
+          -0.003075382439419627,
+          -0.029128169640898705,
+          -0.031586695462465286,
+          -0.012203545309603214,
+          0.010181497782468796,
+          -0.005081841256469488,
+          -0.01433248445391655,
+          0.02123595029115677,
+          0.02882530726492405,
+          0.04842223972082138,
+          0.01276472955942154,
+          -0.05725867673754692,
+          -0.013896007090806961,
+          0.0020732665434479713,
+          -0.02260773442685604,
+          0.032424017786979675,
+          0.060322925448417664,
+          -0.00506847957149148,
+          -0.020612411201000214,
+          -0.027150660753250122,
+          0.0659882202744484,
+          0.012328253127634525,
+          -0.04464537650346756,
+          0.04938427358865738,
+          -0.013655499555170536,
+          0.022340504452586174,
+          0.04517984017729759,
+          0.021859489381313324,
+          -0.0304286926984787,
+          0.017735224217176437,
+          0.024157674983143806,
+          -0.004836879670619965,
+          -0.06196194142103195,
+          -0.02260773442685604,
+          -0.002491928171366453,
+          -0.002779201604425907,
+          -0.013397176750004292,
+          -0.024585243314504623,
+          0.02485247515141964,
+          0.014537361450493336,
+          0.019098103046417236,
+          0.02120031975209713,
+          0.02324908971786499,
+          0.0164970550686121,
+          0.005259995348751545,
+          -0.007331034634262323,
+          -0.005153102800250053,
+          -0.023355981335043907,
+          -0.0269012451171875,
+          -0.02184167318046093,
+          -0.01982853375375271,
+          0.017574885860085487,
+          -0.04371897876262665,
+          0.016568316146731377,
+          0.0024251204449683428,
+          0.024389274418354034,
+          -0.023712290450930595,
+          0.004099767655134201,
+          0.028682785108685493,
+          -0.010591251775622368,
+          -0.04079725220799446,
+          0.025084074586629868,
+          0.011677990667521954,
+          -0.007241957820951939,
+          -0.007504734676331282,
+          0.0006981407641433179,
+          -0.008867612108588219,
+          0.031622324138879776,
+          -0.025280045345425606,
+          -0.01902684196829796,
+          -0.005375795066356659,
+          -0.02011357992887497,
+          0.0058924416080117226,
+          -0.03436589613556862,
+          -0.007464650087058544,
+          0.048600394278764725,
+          0.02132502757012844,
+          0.015098546631634235,
+          -0.04240063950419426,
+          -0.006631780415773392,
+          0.06142747774720192,
+          0.015392500907182693,
+          -0.07354194670915604,
+          0.044787902384996414,
+          0.0010861823102459311,
+          0.006698588375002146,
+          0.004685448482632637,
+          0.030054569244384766,
+          -0.015454854816198349,
+          -0.0159358698874712,
+          0.007673981133848429,
+          -0.030250540003180504,
+          -0.022091088816523552,
+          -0.005175372119992971,
+          0.021449735388159752,
+          -0.04521546885371208,
+          -0.03887319192290306,
+          -0.016791008412837982,
+          -0.0053802491165697575,
+          0.017129501327872276,
+          -0.011758159846067429,
+          -0.024923736229538918,
+          -0.030054569244384766,
+          -0.0005670305690728128,
+          -0.00188175099901855,
+          -0.04101103916764259,
+          -0.03067810833454132,
+          0.0068455650471150875,
+          0.021930750459432602,
+          0.02886093780398369,
+          0.0017392279114574194,
+          0.009459974244236946,
+          0.02504844404757023,
+          -0.00419552531093359,
+          0.029520107433199883,
+          -0.02766730636358261,
+          -0.032584354281425476,
+          0.012782545760273933,
+          -0.010911929421126842,
+          0.02116468921303749,
+          0.001672420185059309,
+          -0.021022165194153786,
+          -0.04689011722803116,
+          0.02706158347427845,
+          -0.0457855649292469,
+          -0.012087744660675526,
+          0.010974282398819923,
+          -0.0022937320172786713,
+          0.010288390330970287,
+          -0.00252533215098083,
+          0.0066451421007514,
+          0.02526222914457321,
+          -0.017601609230041504,
+          0.011909591034054756,
+          0.008012473583221436,
+          0.006306649651378393,
+          0.033991772681474686,
+          -0.008444496430456638,
+          -0.020505519583821297,
+          0.026242075487971306,
+          -0.03416992723941803,
+          0.01401180773973465,
+          0.0360405407845974,
+          0.013726761564612389,
+          0.010181497782468796,
+          0.011998668313026428,
+          0.011820513755083084,
+          0.0013862603809684515,
+          -0.02376573532819748,
+          0.01915154792368412,
+          0.01711168698966503,
+          -0.06866052746772766,
+          -0.020327365025877953,
+          -0.012702375650405884,
+          0.024068597704172134,
+          0.014412653632462025,
+          0.03340386599302292,
+          0.0033092093653976917,
+          -0.032459646463394165,
+          -0.025636352598667145,
+          -0.0075002810917794704,
+          0.04624876379966736,
+          -0.007437927182763815,
+          0.015962593257427216,
+          -0.00821289699524641,
+          0.021859489381313324,
+          0.012675653211772442,
+          -0.0016623989213258028,
+          0.026847798377275467,
+          -0.02597484551370144,
+          0.01240842230618,
+          -0.018688349053263664,
+          -0.04375460743904114,
+          -0.009771743789315224,
+          0.02360539697110653,
+          0.01886650174856186,
+          0.00418661767616868,
+          0.0078075965866446495,
+          -0.01634562388062477,
+          -0.0024986090138554573,
+          -0.03823183476924896,
+          0.010448728688061237,
+          0.027649492025375366,
+          0.0016223143320530653,
+          -0.005638572387397289,
+          -0.0038815289735794067,
+          0.0159358698874712,
+          0.0004142079269513488,
+          0.007469104137271643,
+          -0.034918174147605896,
+          0.040334053337574005,
+          0.0054158796556293964,
+          0.04582119360566139,
+          -0.04724642634391785,
+          -0.017450178042054176,
+          -0.0038904366083443165,
+          -0.007380026858299971,
+          0.042863838374614716,
+          -0.03573768213391304,
+          -0.02830866165459156,
+          -0.011464206501841545,
+          0.01947222650051117,
+          0.015668639913201332,
+          -0.02155662700533867,
+          -0.03976396098732948,
+          -0.002783655421808362,
+          -0.035274479538202286,
+          -0.010947559960186481,
+          -0.014412653632462025,
+          -0.0032802594359964132,
+          -0.049847472459077835,
+          -0.0017592702060937881,
+          -0.0835898295044899,
+          0.005081841256469488,
+          0.03755485266447067,
+          -0.009807374328374863,
+          0.04111792892217636,
+          -0.002641132101416588,
+          0.002984078601002693,
+          -0.047959040850400925,
+          0.01689790189266205,
+          0.026242075487971306,
+          0.04678322374820709,
+          -0.022180166095495224,
+          0.005242179613560438,
+          0.014679884538054466,
+          -0.011214790865778923,
+          -0.017610516399145126,
+          -0.010511082597076893,
+          0.02004231885075569,
+          -0.0051976414397358894,
+          0.0012392833596095443,
+          0.04528673365712166,
+          0.029537923634052277,
+          0.014697699807584286,
+          0.031943000853061676,
+          0.00030648044776171446,
+          -0.01814497821033001,
+          0.0025320129934698343,
+          0.043148886412382126,
+          0.04093977436423302,
+          -0.04375460743904114,
+          0.015543931163847446,
+          -0.03905134275555611,
+          0.025636352598667145,
+          0.013985084369778633,
+          0.007442380767315626,
+          0.03343949466943741,
+          0.006466988008469343,
+          -0.005803364794701338,
+          0.00020348520774859935,
+          0.024674320593476295,
+          -0.01662176288664341,
+          0.017922285944223404,
+          -0.004836879670619965,
+          -0.022661181166768074,
+          0.000857365841511637,
+          -0.015232161618769169,
+          0.017191855236887932,
+          0.02187730371952057,
+          -0.007188511546701193,
+          -0.0010806149803102016,
+          -0.0003092641127295792,
+          0.021948564797639847,
+          0.004395948722958565,
+          -0.0005812272429466248,
+          0.011428575031459332,
+          0.03500724956393242,
+          -0.03215678781270981,
+          -0.010644698515534401,
+          0.03212115541100502,
+          -0.033101003617048264,
+          0.009566866792738438,
+          -0.04104666784405708,
+          0.010609067045152187,
+          0.0022324915044009686,
+          -0.020986534655094147,
+          0.04076162353157997,
+          -0.010911929421126842,
+          0.001672420185059309,
+          -0.013317007571458817,
+          0.00856920424848795,
+          0.015561746433377266,
+          0.03762611374258995,
+          -0.0039015712682157755,
+          0.02882530726492405,
+          0.01489366963505745,
+          -0.020220471546053886,
+          0.04008463770151138,
+          -0.04268568381667137,
+          -0.010208221152424812,
+          0.015249977819621563,
+          -0.005981518421322107,
+          -0.012461868114769459,
+          -0.020576780661940575,
+          -0.021663518622517586,
+          -0.007001449819654226,
+          0.015143085271120071,
+          -0.02686561457812786,
+          0.006760942284017801,
+          0.006801026873290539,
+          -0.017842117697000504,
+          -0.007651711814105511,
+          0.006609511096030474,
+          -0.008016927167773247,
+          -0.0056296647526323795,
+          0.028201768174767494,
+          -0.017975732684135437,
+          0.01260439120233059,
+          0.02321345917880535,
+          0.028843123465776443,
+          -0.018162794411182404,
+          -0.0016189739108085632,
+          -0.0032490824814885855,
+          -0.0058479029685258865,
+          -0.03285158798098564,
+          0.026509307324886322,
+          0.009620312601327896,
+          -0.010056789964437485,
+          -0.024709951132535934,
+          -0.008836435154080391,
+          0.01879524067044258,
+          -0.0035185404121875763,
+          -0.005549495108425617,
+          -0.021984197199344635,
+          -0.02878967672586441,
+          0.0257788747549057,
+          0.01863490231335163,
+          -0.046711962670087814,
+          -0.0032112249173223972,
+          0.005411426071077585,
+          0.004783433396369219,
+          0.0027213015127927065,
+          -0.014849130995571613,
+          0.007388934958726168,
+          0.026972506195306778,
+          0.0054826876148581505,
+          -0.015543931163847446,
+          -0.012622206471860409,
+          -0.005099656525999308,
+          0.008613742887973785,
+          0.0069925421848893166,
+          0.009932082146406174,
+          0.02975170873105526,
+          0.04072599112987518,
+          0.0018628222169354558,
+          -0.01288943737745285,
+          -0.02561853639781475,
+          0.025315675884485245,
+          -0.0019630338065326214,
+          0.04165239259600639,
+          0.04564303904771805,
+          0.0025876860599964857,
+          -0.060572341084480286,
+          0.012479683384299278,
+          -0.02517315186560154,
+          -0.009843004867434502,
+          0.0066406880505383015,
+          0.0036209789104759693,
+          -0.04660507291555405,
+          -0.0031956364400684834,
+          0.0022536474280059338,
+          0.034152109175920486,
+          -0.007299857679754496,
+          -0.04436033219099045,
+          -0.008929966017603874,
+          -0.007406750228255987,
+          0.02574324421584606,
+          0.010110235773026943,
+          -0.007892219349741936,
+          -0.015294515527784824,
+          0.013361545279622078,
+          -0.0026567205786705017,
+          -0.006190849468111992,
+          0.017013700678944588,
+          -0.007691796403378248,
+          0.011170252226293087,
+          -0.012096652761101723,
+          -0.019739456474781036,
+          -0.017655055969953537,
+          0.03393832594156265,
+          -0.02706158347427845,
+          0.025565091520547867,
+          0.014947115443646908,
+          0.0026923513505607843,
+          0.039407651871442795,
+          -0.01535686943680048,
+          -0.027364445850253105,
+          -0.002897228579968214,
+          -0.0037078289315104485,
+          -0.03766174241900444,
+          0.01815388724207878,
+          -0.023516319692134857,
+          -0.022091088816523552,
+          0.013495161198079586,
+          0.00951342098414898,
+          0.0002751643187366426,
+          -0.009308543056249619,
+          -0.00851130485534668,
+          -0.0015465988544747233,
+          0.01956130377948284,
+          0.019258441403508186,
+          -0.007451288867741823,
+          0.043505191802978516,
+          -0.0004167132137808949,
+          0.0038369905669242144,
+          0.0005995993851684034,
+          0.018100440502166748,
+          0.060287293046712875,
+          0.020452072843909264,
+          0.0012481911107897758,
+          0.030482139438390732,
+          0.022500842809677124,
+          0.037162911146879196,
+          -0.03019709326326847,
+          -0.04564303904771805,
+          0.017895564436912537,
+          -0.007250865455716848,
+          0.009522328153252602,
+          0.022429581731557846,
+          0.0011813833843916655,
+          0.01418996136635542,
+          0.007063803728669882,
+          -0.010243851691484451,
+          0.023801365867257118,
+          -0.032905034720897675,
+          -0.02789890766143799,
+          0.027115030214190483,
+          0.010056789964437485,
+          0.019971057772636414,
+          0.0360405407845974,
+          -0.024870289489626884,
+          -0.033671095967292786,
+          0.004881417844444513,
+          0.006680772639811039,
+          -0.0283442921936512,
+          0.012942884117364883,
+          -0.013521884568035603,
+          0.015160900540649891,
+          0.0064001805149018764,
+          -0.012648929841816425,
+          -0.0027279823552817106,
+          0.013780207373201847,
+          -0.0141721460968256,
+          0.008698366582393646,
+          -0.015160900540649891,
+          -0.012452960945665836,
+          0.010626882314682007,
+          -0.030642477795481682,
+          0.0008445610292255878,
+          0.00507738720625639,
+          -0.0015131949912756681,
+          0.006230934057384729,
+          -0.0067297653295099735,
+          -0.004431579262018204,
+          -0.005865718703716993,
+          -0.03185392543673515,
+          0.02795235440135002,
+          0.02766730636358261,
+          -0.025119705125689507,
+          -0.0030553401447832584,
+          -0.0036900134291499853,
+          -0.014528454281389713,
+          -0.006346734240651131,
+          0.010876297950744629,
+          -0.014448285102844238,
+          0.0021411876659840345,
+          -0.016087301075458527,
+          -0.018367672339081764,
+          0.023961706086993217,
+          0.022518659010529518,
+          0.042863838374614716,
+          -0.0010037861065939069,
+          0.022750258445739746,
+          0.021307211369276047,
+          0.0011028843000531197,
+          0.0017982413992285728,
+          0.017494717612862587,
+          -0.017557071521878242,
+          0.028985645622015,
+          0.025707613676786423,
+          0.03099878504872322,
+          0.010849575512111187,
+          -0.03221023455262184,
+          -0.0024674320593476295,
+          0.018848687410354614,
+          0.017076054587960243,
+          0.0008317562169395387,
+          -0.013486253097653389,
+          -0.009771743789315224,
+          0.012533130124211311,
+          -0.02152099646627903,
+          -0.0065248883329331875,
+          0.013512976467609406,
+          -0.016514871269464493,
+          0.0032802594359964132,
+          -0.007825411856174469,
+          0.02472776733338833,
+          0.0028838668949902058,
+          -0.005193187389522791,
+          0.012773637659847736,
+          -0.00048185072955675423,
+          0.005237726029008627,
+          -0.005037302616983652,
+          0.02324908971786499,
+          0.03493598848581314,
+          0.012684560380876064,
+          0.04407528415322304,
+          0.01998887211084366,
+          -0.013423899188637733,
+          -0.009531236253678799,
+          -0.020612411201000214,
+          0.022358320653438568,
+          -0.030286170542240143,
+          -0.024709951132535934,
+          -0.022073272615671158,
+          0.01535686943680048,
+          0.00679657282307744,
+          0.015606285072863102,
+          -0.0033893787767738104,
+          -0.004447167739272118,
+          0.031372908502817154,
+          0.004179936833679676,
+          0.025511644780635834,
+          0.006907918956130743,
+          -0.007905581034719944,
+          -0.004012917634099722,
+          0.0024139860179275274,
+          0.008150543086230755,
+          -0.015241069719195366,
+          0.03826746717095375,
+          -0.0035630788188427687,
+          0.04813719540834427,
+          -0.052733566612005234,
+          -0.002003118395805359,
+          0.00635564187541604,
+          -0.01918718032538891,
+          -0.005669749341905117,
+          -0.03915823623538017,
+          0.02030954882502556,
+          -0.03377798572182655,
+          0.026669645681977272,
+          -0.02027391828596592,
+          0.026491491124033928,
+          -0.029929861426353455,
+          0.003576440503820777,
+          -0.0015243296511471272,
+          -0.0038035865873098373,
+          0.0164970550686121,
+          0.024282382801175117,
+          -0.002409531967714429,
+          -0.02590358257293701,
+          -0.0008083735010586679,
+          0.0037635019980371,
+          0.0066273268312215805,
+          -0.013717853464186192,
+          0.0054158796556293964,
+          -0.0012582122581079602,
+          -0.029680447652935982,
+          -0.033101003617048264,
+          0.012132283300161362,
+          -0.010911929421126842,
+          0.0337958037853241,
+          0.030375247821211815,
+          -0.026794353500008583,
+          -0.027043769136071205,
+          0.0034561865031719208,
+          -0.008676096796989441,
+          -0.009090305306017399,
+          0.020737119019031525,
+          0.04532236233353615,
+          0.00925509724766016,
+          -0.020790565758943558,
+          0.023943889886140823,
+          0.004734441172331572,
+          0.008582565933465958,
+          0.004422671627253294,
+          -0.021734781563282013,
+          -0.018492378294467926,
+          -0.01689790189266205,
+          -0.01726311631500721,
+          0.008177265524864197,
+          -0.01972164213657379,
+          -0.04012026637792587,
+          -0.04742458090186119,
+          -0.011179160326719284,
+          0.019383149221539497,
+          0.004144306294620037,
+          0.05052445828914642,
+          0.012452960945665836,
+          0.02478121407330036,
+          -0.0017191855004057288,
+          -0.005469325929880142,
+          0.04646254703402519,
+          -0.019899794831871986,
+          0.051700275391340256,
+          0.009700481779873371,
+          -0.0021634569857269526,
+          -0.01578443869948387,
+          0.03188955411314964,
+          -0.024905921891331673,
+          -0.0019240626133978367,
+          0.006342280190438032,
+          -0.0034984981175512075,
+          0.017298748716711998,
+          0.008644919842481613,
+          0.010128051973879337,
+          -0.02360539697110653,
+          0.028130507096648216,
+          -0.023641027510166168,
+          0.014590808190405369,
+          0.015926962718367577,
+          -0.002009799238294363,
+          0.0036098442506045103,
+          -0.0054826876148581505,
+          0.004574102349579334,
+          -0.002970716916024685,
+          0.011597821488976479,
+          -0.02510189078748226,
+          0.015009469352662563,
+          -0.02228705771267414,
+          -0.013317007571458817,
+          0.0004178266681265086,
+          -0.030945340171456337,
+          0.012399514205753803,
+          0.04108230024576187,
+          0.01037746760994196,
+          -0.02232268825173378,
+          0.005371341481804848,
+          -0.051450859755277634,
+          -0.008199535310268402,
+          0.0014809046406298876,
+          0.01271128375083208,
+          0.01346843782812357,
+          0.0007504734676331282,
+          0.009727205149829388,
+          -0.00955795869231224,
+          -0.03055340051651001,
+          0.033421680331230164,
+          -0.001038860180415213,
+          -0.0020298415329307318,
+          0.003928294405341148,
+          0.01851019449532032,
+          0.0025854590348899364,
+          0.004188844468444586,
+          0.012773637659847736,
+          -0.007709611672908068,
+          -0.03224586322903633,
+          -0.021449735388159752,
+          -0.0038926636334508657,
+          0.0019318568520247936,
+          -0.003694467479363084,
+          0.003318117232993245,
+          0.0055851261131465435,
+          -0.015535023994743824,
+          0.01082285214215517,
+          -0.011980853043496609,
+          0.010707051493227482,
+          -0.019169364124536514,
+          -0.004329140763729811,
+          0.006658503785729408,
+          -0.020808380097150803,
+          -0.018296409398317337,
+          -0.051450859755277634,
+          -0.010466543957591057,
+          0.002101103076711297,
+          0.021146873012185097,
+          0.006676319055259228,
+          0.008489035069942474,
+          -0.014394838362932205,
+          0.0068322038277983665,
+          0.003656609682366252,
+          0.02520878240466118,
+          -0.026829984039068222,
+          0.03119475580751896,
+          0.018723979592323303,
+          0.016951346769928932,
+          -0.03862377628684044,
+          0.03249527886509895,
+          0.016443608328700066,
+          -0.003759048180654645,
+          0.017521440982818604,
+          0.014002899639308453,
+          0.04753147065639496,
+          -0.030606847256422043,
+          0.004810156300663948,
+          -0.02754260040819645,
+          0.025084074586629868,
+          -0.03805368393659592,
+          0.028237400576472282,
+          0.0009163793292827904,
+          0.023943889886140823,
+          0.04428907111287117,
+          9.728875738801435e-05,
+          0.03851688280701637,
+          -0.022019827738404274,
+          -0.009045766666531563,
+          0.025636352598667145,
+          -0.017476901412010193,
+          0.015882423147559166,
+          0.00778978131711483,
+          -0.024282382801175117,
+          0.01838548667728901,
+          0.007709611672908068,
+          -0.0031956364400684834,
+          -0.04788777977228165,
+          0.02795235440135002,
+          0.02260773442685604,
+          0.02120031975209713,
+          0.0067297653295099735,
+          -0.012150098569691181,
+          -0.010234943591058254,
+          0.005611849017441273,
+          -0.009246189147233963,
+          0.016755377873778343,
+          -0.015846792608499527,
+          -0.030909709632396698,
+          0.00849348958581686,
+          -0.01927625574171543,
+          0.012648929841816425,
+          -0.0038904366083443165,
+          0.029092539101839066,
+          -0.017770856618881226,
+          -0.01068032905459404,
+          0.01101882103830576,
+          -0.0028237400110810995,
+          -0.016069484874606133,
+          -0.04842223972082138,
+          -0.038801927119493484,
+          0.008453404530882835,
+          -0.0013105449033901095,
+          -0.020184841006994247,
+          0.011909591034054756,
+          -0.010591251775622368,
+          -0.004792341031134129,
+          -0.01972164213657379,
+          -0.010742682963609695,
+          -0.023035304620862007,
+          -0.006760942284017801,
+          0.010074605233967304,
+          0.026277706027030945,
+          -0.018848687410354614,
+          0.03798241913318634,
+          0.029413215816020966,
+          0.013985084369778633,
+          -0.012506406754255295,
+          0.0257788747549057,
+          -0.009353081695735455,
+          -0.01489366963505745,
+          0.033671095967292786,
+          -0.0031466439832001925,
+          -0.009192743338644505,
+          0.00017439600196667016,
+          -0.024799028411507607,
+          -0.023070935159921646,
+          0.019579118117690086,
+          -0.022393951192498207,
+          -0.025280045345425606,
+          -0.004921502433717251,
+          -0.006141857244074345,
+          -0.02975170873105526,
+          -0.04753147065639496,
+          -0.011214790865778923,
+          0.03645029664039612,
+          0.0057632802054286,
+          0.018492378294467926,
+          0.01711168698966503,
+          -0.012230267748236656,
+          -0.03039306215941906,
+          0.004721079487353563,
+          -0.012105559930205345,
+          -0.008738450706005096,
+          -0.01287162210792303,
+          0.007700704038143158,
+          0.016951346769928932,
+          0.01101882103830576,
+          0.0011669083032757044,
+          -0.0052778106182813644,
+          -0.02123595029115677,
+          -0.007861042395234108,
+          -0.0176995936781168,
+          -0.028985645622015,
+          -0.0031466439832001925,
+          -0.033029742538928986,
+          0.03823183476924896,
+          0.018777426332235336,
+          0.005273356568068266,
+          -0.01982853375375271,
+          -0.012720191851258278,
+          -0.020790565758943558,
+          0.00927291251718998,
+          -0.00707716541364789,
+          0.009718297980725765,
+          -0.004614186938852072,
+          0.01726311631500721,
+          -0.006057234015315771,
+          0.00047684015589766204,
+          -0.04436033219099045,
+          0.009023496881127357,
+          0.01934751868247986,
+          0.003930521197617054,
+          -0.009878636337816715,
+          0.008880973793566227,
+          0.011606729589402676,
+          -0.021022165194153786,
+          -0.0236944742500782,
+          0.019222810864448547,
+          0.020576780661940575,
+          0.015909146517515182,
+          -0.009299635887145996,
+          -0.009299635887145996,
+          0.007130611687898636,
+          0.0007682888535782695,
+          -0.006315557286143303,
+          0.016550501808524132,
+          0.023623213171958923,
+          0.01375348400324583,
+          -0.00018177268793806434,
+          0.026206444948911667,
+          0.009174928069114685,
+          -0.012310437858104706,
+          -0.012114468030631542,
+          -0.007620534859597683,
+          0.0023694473784416914,
+          -0.01670193299651146,
+          -0.011989760212600231,
+          -0.010083513334393501,
+          0.025226598605513573,
+          0.009370896965265274,
+          0.011294960044324398,
+          -0.003090970916673541,
+          -0.006173034198582172,
+          0.005411426071077585,
+          0.009949897415935993,
+          0.0001479512866353616,
+          -0.001145752496086061,
+          0.007219688501209021,
+          -0.038766298443078995,
+          -0.028682785108685493,
+          0.0014196642441675067,
+          -0.002750251442193985,
+          0.0033515209797769785,
+          0.03506069630384445,
+          0.027596045285463333,
+          -0.007014811504632235,
+          -0.01918718032538891,
+          -0.0018227375112473965,
+          0.024282382801175117,
+          0.0001636789384065196,
+          -0.02308875136077404,
+          -0.009477789513766766,
+          0.009090305306017399,
+          0.00032874970929697156,
+          0.02927069365978241,
+          0.005700926296412945,
+          -0.008235165849328041,
+          0.019383149221539497,
+          0.10076387226581573,
+          0.007776419632136822,
+          0.006885650102049112,
+          -0.012150098569691181,
+          -0.031016601249575615,
+          0.0021033298689872026,
+          0.021699151024222374,
+          -0.019080286845564842,
+          0.04485916346311569,
+          0.011553282849490643,
+          0.013263560831546783,
+          0.018973395228385925,
+          -0.0021144645288586617,
+          -0.010974282398819923,
+          -0.016131838783621788,
+          0.030606847256422043,
+          0.0016323354793712497,
+          0.02184167318046093,
+          -0.020897457376122475,
+          0.0017414548201486468,
+          0.004591918084770441,
+          0.012800361029803753,
+          0.00606168806552887,
+          -0.005055117886513472,
+          -0.02510189078748226,
+          0.018777426332235336,
+          -0.018180610612034798,
+          0.007224142551422119,
+          -0.008858704939484596,
+          -0.022874966263771057,
+          0.008987866342067719,
+          -0.00014502844715025276,
+          -0.01244405284523964,
+          0.005513864569365978,
+          -0.014109792187809944,
+          0.0326734334230423,
+          -0.012738007120788097,
+          0.0015354643110185862,
+          0.003663290524855256,
+          -0.016390163451433182,
+          0.01202539075165987,
+          0.01746799424290657,
+          -0.01200757548213005,
+          0.030090201646089554,
+          -0.016710840165615082,
+          -0.004318006336688995,
+          0.012061022222042084,
+          -0.018002456054091454,
+          0.011214790865778923,
+          0.028486816212534904,
+          -0.009335266426205635,
+          0.005683110561221838,
+          -0.0075581809505820274,
+          0.009905358776450157,
+          0.009335266426205635,
+          0.026402413845062256,
+          0.023142196238040924,
+          -0.0005219354061409831,
+          0.004251198377460241,
+          -0.008880973793566227,
+          -0.0019708280451595783,
+          -0.0009464428294450045,
+          0.034312449395656586,
+          -0.018162794411182404,
+          -0.0043246871791779995,
+          -0.009326359257102013,
+          0.0006680772639811039,
+          -0.014724423177540302,
+          -0.002120031975209713,
+          -0.007437927182763815,
+          -0.0019351971568539739,
+          0.0076606194488704205,
+          -0.00036493720836006105,
+          0.016158562153577805,
+          -0.008257435634732246,
+          -0.004556287080049515,
+          0.004997218027710915,
+          -0.005224364344030619,
+          0.01657722517848015,
+          -0.02574324421584606,
+          0.0037612752057611942,
+          0.018456747755408287,
+          0.00869391206651926,
+          0.011508745141327381,
+          -0.01642579399049282,
+          0.0025854590348899364,
+          -0.002992986235767603,
+          -0.012577668763697147,
+          0.012577668763697147,
+          -0.01831422559916973,
+          -0.011499837040901184,
+          -0.021129056811332703,
+          -0.0297873392701149,
+          -0.002863824600353837,
+          -0.01851019449532032,
+          0.008146088570356369,
+          0.01918718032538891,
+          0.028647154569625854,
+          0.00018024168093688786,
+          -0.004970495123416185,
+          -0.01743236370384693,
+          0.015962593257427216,
+          0.045037318021059036,
+          -0.016292177140712738,
+          -0.02520878240466118,
+          -0.009914266876876354,
+          0.0127290990203619,
+          -0.008622650988399982,
+          -0.026046106591820717,
+          -0.003943882882595062,
+          0.00577218784019351,
+          -0.02911035344004631,
+          0.009379805065691471,
+          0.0005712060956284404,
+          0.024442721158266068,
+          0.004672087263315916,
+          0.020665857940912247,
+          -0.017521440982818604,
+          -0.0019708280451595783,
+          0.025476014241576195,
+          -0.029520107433199883,
+          0.0045874640345573425,
+          0.01802917942404747,
+          -0.01346843782812357,
+          0.010769405402243137,
+          0.030820632353425026,
+          0.025226598605513573,
+          -0.01963256485760212,
+          0.024335827678442,
+          0.014626438729465008,
+          -0.008720635436475277,
+          0.003284713253378868,
+          0.011660175397992134,
+          0.012622206471860409,
+          0.006698588375002146,
+          -0.0034584132954478264,
+          -0.005861264653503895,
+          0.009045766666531563,
+          0.031337279826402664,
+          0.0016623989213258028,
+          -0.012657837942242622,
+          0.028415553271770477,
+          0.02497718296945095,
+          -0.005429241340607405,
+          0.008426681160926819,
+          -0.02039862610399723,
+          -0.016443608328700066,
+          0.01730765588581562,
+          0.022465212270617485,
+          0.005910256877541542,
+          0.0032401748467236757,
+          -0.011366221122443676,
+          0.022839335724711418,
+          -0.008377688936889172,
+          0.01154437568038702,
+          -0.027168476954102516,
+          -0.002518651308491826,
+          -0.0036900134291499853,
+          0.02068367227911949,
+          -0.0008106004097498953,
+          -0.005032849032431841,
+          0.019062472507357597,
+          -0.01228371448814869,
+          -0.019775087013840675,
+          -0.006462534423917532,
+          -0.018118256703019142,
+          0.017637239769101143,
+          -0.004774525761604309,
+          -0.02030954882502556,
+          0.007549273315817118,
+          0.0005678656743839383,
+          -0.009041312150657177,
+          0.006801026873290539,
+          0.004144306294620037,
+          0.008057012222707272,
+          -0.02264336682856083,
+          0.010751590132713318,
+          -0.005464872345328331,
+          0.007901127450168133,
+          -0.008382143452763557,
+          0.014368114992976189,
+          -0.006738672964274883,
+          0.0015599604230374098,
+          0.0006012695375829935,
+          -0.006208664737641811,
+          0.000636900367680937,
+          0.013210115022957325,
+          0.0032624441664665937,
+          -0.016550501808524132,
+          0.015312331728637218,
+          -0.008662735112011433,
+          -0.004925956483930349,
+          -0.005028394982218742,
+          -0.004106448497623205,
+          -0.0011357313487678766,
+          0.012738007120788097,
+          0.00865382794290781,
+          -0.031123492866754532,
+          -0.006689680740237236,
+          -0.024959366768598557,
+          -0.009326359257102013,
+          0.010145867243409157,
+          0.012052114121615887,
+          -0.020576780661940575,
+          0.009001228027045727,
+          -0.027079399675130844,
+          -0.01226589921861887,
+          0.0269012451171875,
+          0.023961706086993217,
+          -0.02068367227911949,
+          0.012738007120788097,
+          -0.014804592356085777,
+          -0.02014921046793461,
+          0.015009469352662563,
+          0.021467549726366997,
+          0.01224808394908905,
+          0.005028394982218742,
+          -0.04400402307510376,
+          -0.010920836590230465,
+          0.019864164292812347,
+          -0.03140854090452194,
+          -0.009504512883722782,
+          0.027417892590165138,
+          -0.006827749777585268,
+          -0.01726311631500721,
+          -0.003516313387081027,
+          0.03522103279829025,
+          -0.02123595029115677,
+          0.015526115894317627,
+          0.011428575031459332,
+          -0.02152099646627903,
+          0.014760053716599941,
+          -0.012853806838393211,
+          0.008618197403848171,
+          0.0019296299433335662,
+          -0.022536473348736763,
+          -0.0006502618780359626,
+          0.018011363223195076,
+          -0.0005444829585030675,
+          -0.004050775431096554,
+          0.011455298401415348,
+          0.016639579087495804,
+          -0.018421117216348648,
+          0.013922730460762978,
+          0.006088410969823599,
+          9.90285407169722e-05,
+          0.016283269971609116,
+          -0.02052333392202854,
+          -0.01810934767127037,
+          0.01979290321469307,
+          -0.004179936833679676,
+          -0.02421111986041069,
+          0.02344505861401558,
+          -0.02376573532819748,
+          -0.020950904116034508,
+          -0.02340942807495594,
+          0.00032652277150191367,
+          0.00849348958581686,
+          -0.01734328642487526,
+          -0.026562752202153206,
+          0.007233050186187029,
+          0.01950785703957081,
+          0.04689011722803116,
+          0.018038086593151093,
+          -0.0023048666771501303,
+          0.023819182068109512,
+          -0.016595039516687393,
+          0.013263560831546783,
+          0.012105559930205345,
+          -0.0036543826572597027,
+          -0.015757715329527855,
+          0.0025743243750184774,
+          0.009602497331798077,
+          0.007152881007641554,
+          0.0030263899825513363,
+          -0.022251427173614502,
+          -0.02120031975209713,
+          -0.04101103916764259,
+          0.00957577396184206,
+          -0.007771965581923723,
+          0.0008968937327153981,
+          0.05790003016591072,
+          0.00126711989287287,
+          -0.033350419253110886,
+          0.01754816249012947,
+          0.00047767526120878756,
+          0.0007610513712279499,
+          -0.011633452028036118,
+          0.002986305393278599,
+          -0.02389044314622879,
+          -0.013949453830718994,
+          -0.0353991873562336,
+          -0.0036877866368740797,
+          0.01027948223054409,
+          0.005162010435014963,
+          0.005700926296412945,
+          -0.02805924601852894,
+          -0.002245853189378977,
+          -0.025867952033877373,
+          0.022304873913526535,
+          0.006475895643234253,
+          -0.0001588075392646715,
+          0.0023649935610592365,
+          0.027400076389312744,
+          -0.008622650988399982,
+          -0.01854582503437996,
+          -0.020790565758943558,
+          0.0011902910191565752,
+          -0.004248971585184336,
+          0.001448614289984107,
+          0.0009458860731683671,
+          -0.025832321494817734,
+          0.009486697614192963,
+          0.00028532466967590153,
+          -0.012595484033226967,
+          0.014100884087383747,
+          -0.013352638110518456,
+          -0.007001449819654226,
+          0.009157112799584866,
+          0.023160012438893318,
+          -0.027524784207344055,
+          -0.008506850339472294,
+          -0.024870289489626884,
+          0.009985528886318207,
+          0.012675653211772442,
+          0.0029484478291124105,
+          -0.019775087013840675,
+          -0.012925068847835064,
+          -0.02574324421584606,
+          -0.009851912967860699,
+          -0.02059459500014782,
+          -0.008012473583221436,
+          0.005041756667196751,
+          0.01657722517848015,
+          0.03244183212518692,
+          0.016799917444586754,
+          -0.005157556850463152,
+          0.0017548163887113333,
+          -0.002420666627585888,
+          0.008751812390983105,
+          -0.020060133188962936,
+          2.0077113731531426e-05,
+          -0.0159358698874712,
+          0.017957916483283043,
+          0.00999443605542183,
+          -0.013887099921703339,
+          -0.017761947587132454,
+          -0.02276807464659214,
+          0.019525671377778053,
+          -0.022429581731557846,
+          -0.017886655405163765,
+          0.01918718032538891,
+          -0.011036636307835579,
+          0.006409088149666786,
+          -0.013691130094230175,
+          0.037198543548583984,
+          0.010021159425377846,
+          0.025476014241576195,
+          -0.016612855717539787,
+          -0.005567310843616724,
+          -0.0272219218313694,
+          -0.004400402307510376,
+          -0.01682663895189762,
+          -0.00434695603325963,
+          -0.0012337160296738148,
+          0.009753928519785404,
+          -0.02526222914457321,
+          -0.005567310843616724,
+          0.02321345917880535,
+          -0.027560414746403694,
+          -0.0022124492097646,
+          -0.0021757050417363644,
+          0.006337826605886221,
+          0.007121704053133726,
+          0.0014397065388038754,
+          -0.002219130052253604,
+          -0.022180166095495224,
+          -0.008448950946331024,
+          -0.028807492926716805,
+          0.022091088816523552,
+          -0.010315113700926304,
+          -0.014653162099421024,
+          0.0001746743801049888,
+          0.021182503551244736,
+          0.0046899025328457355,
+          -0.010172589682042599,
+          0.0017403413075953722,
+          -0.007598265539854765,
+          0.0021645703818649054,
+          0.0038147212471812963,
+          0.012096652761101723,
+          -0.0020988760516047478,
+          0.014083068817853928,
+          0.02020265720784664,
+          -0.008560297079384327,
+          -0.0006975840078666806,
+          -0.0058122724294662476,
+          0.0037790904752910137,
+          0.011375129222869873,
+          0.01037746760994196,
+          0.016203101724386215,
+          0.003641021205112338,
+          -0.009486697614192963,
+          0.01670193299651146,
+          -0.014083068817853928,
+          0.02184167318046093,
+          -0.001927402918227017,
+          -0.01943659596145153,
+          0.002552055288106203,
+          -0.047923412173986435,
+          0.0009069149382412434,
+          -0.012310437858104706,
+          -0.022750258445739746,
+          0.006293287966400385,
+          -0.011838329024612904,
+          -0.008190627209842205,
+          0.019098103046417236,
+          -0.009103666059672832,
+          -0.02219798043370247,
+          0.011571098119020462,
+          0.0027591590769588947,
+          -0.009036858566105366,
+          0.006613965146243572,
+          -0.0016679662512615323,
+          0.008658281527459621,
+          0.004482798743993044,
+          0.005366887431591749,
+          0.001175816054455936,
+          -0.01565973088145256,
+          -0.018563641235232353,
+          0.020095765590667725,
+          0.008333150297403336,
+          -0.0016278816619887948,
+          0.010003344155848026,
+          0.002652266761288047,
+          0.011428575031459332,
+          -0.003694467479363084,
+          -0.004810156300663948,
+          0.00019415996212046593,
+          -0.014679884538054466,
+          0.014145422726869583,
+          -0.013085407204926014,
+          -0.032228048890829086,
+          0.013664407655596733,
+          0.006008241791278124,
+          0.00681438809260726,
+          -0.008119366131722927,
+          -0.011891775764524937,
+          -0.007161788642406464,
+          -0.012078837491571903,
+          0.007455742452293634,
+          0.015294515527784824,
+          -0.0038147212471812963,
+          -0.0008879860397428274,
+          -0.01418996136635542,
+          0.007575996220111847,
+          0.020184841006994247,
+          0.029965493828058243,
+          -0.02488810569047928,
+          0.00724641140550375,
+          -0.005344618111848831,
+          -0.011873960494995117,
+          -0.020291734486818314,
+          0.024264566600322723,
+          0.01457299292087555,
+          0.01626545563340187,
+          0.004206659737974405,
+          0.0036165250930935144,
+          0.000802249473053962,
+          -0.019365333020687103,
+          0.003191182389855385,
+          0.008137181401252747,
+          -0.013317007571458817,
+          -0.0020688127260655165,
+          0.02478121407330036,
+          -0.013780207373201847,
+          -0.0013484027003869414,
+          0.001965260598808527,
+          0.024032967165112495,
+          -0.0138514693826437,
+          0.001873956760391593,
+          0.004729987122118473,
+          0.0006508186343125999,
+          0.005522772204130888,
+          0.013219022192060947,
+          0.07496717572212219,
+          0.019579118117690086,
+          0.005393610801547766,
+          0.007446834817528725,
+          -0.006311103235930204,
+          -0.006008241791278124,
+          -0.009264005348086357,
+          -0.011553282849490643,
+          0.002538693603128195,
+          -0.0017325470689684153,
+          -0.02308875136077404,
+          -0.002113351132720709,
+          -0.005825634114444256,
+          0.00045429254532791674,
+          -0.0012982968473806977,
+          0.01734328642487526,
+          -0.012666745111346245,
+          0.013913823291659355,
+          0.03204989433288574,
+          0.007188511546701193,
+          -0.021770412102341652,
+          0.013504069298505783,
+          0.0025854590348899364,
+          -0.006484803277999163,
+          -0.003402740228921175,
+          -0.006609511096030474,
+          0.006996996235102415,
+          0.011669083498418331,
+          -0.010600159876048565,
+          0.03851688280701637,
+          0.0013951680157333612,
+          0.0014764508232474327,
+          -0.01298742275685072,
+          -0.008008019998669624,
+          -0.0038035865873098373,
+          0.0024852475617080927,
+          0.012354975566267967,
+          -0.0010978736681863666,
+          -0.01851019449532032,
+          0.003068701596930623,
+          0.004166575148701668,
+          0.021182503551244736,
+          0.0044182175770401955,
+          0.002165683778002858,
+          -0.009246189147233963,
+          0.01069814432412386,
+          -0.00923728197813034,
+          -0.004021825268864632,
+          0.009860821068286896,
+          -0.0033760173246264458,
+          -0.010457636788487434,
+          -0.016452517360448837,
+          -0.0003234607574995607,
+          0.008742904290556908,
+          -0.013691130094230175,
+          -0.014742238447070122,
+          0.022429581731557846,
+          -0.0014385931426659226,
+          0.023035304620862007,
+          -0.005406972020864487,
+          0.008622650988399982,
+          0.011473113670945168,
+          -0.0012159006437286735,
+          -0.00782986544072628,
+          0.019703825935721397,
+          -0.00027196313021704555,
+          0.004277921747416258,
+          -0.014679884538054466,
+          0.009531236253678799,
+          -0.01114352885633707,
+          -0.0023605397436767817,
+          0.016363440081477165,
+          0.0001629830221645534,
+          0.007037080824375153,
+          -0.002538693603128195,
+          -0.009032404981553555,
+          -0.031782664358615875,
+          -0.02353413589298725,
+          -0.0022057686001062393,
+          0.013094314374029636,
+          -0.008578112348914146,
+          0.0022592146415263414,
+          -0.0027480246499180794,
+          -0.004053002223372459,
+          -0.016194192692637444,
+          0.006507072597742081,
+          5.518596663023345e-05,
+          0.015971500426530838,
+          0.0011435255873948336,
+          -0.015276700258255005,
+          0.003774636657908559,
+          -0.0031399631407111883,
+          0.018937764689326286,
+          -0.013922730460762978,
+          -0.0029573554638773203,
+          0.0236944742500782,
+          -0.01618528552353382,
+          -0.0011691352119669318,
+          0.003202317049726844,
+          -0.013682222925126553,
+          0.031016601249575615,
+          0.013094314374029636,
+          0.023943889886140823,
+          0.015401408076286316,
+          0.01212337613105774,
+          0.001612293184734881,
+          0.01630108617246151,
+          0.013619869016110897,
+          0.012925068847835064,
+          -0.015989316627383232,
+          -0.007348849903792143,
+          -0.017280932515859604,
+          -0.004551833029836416,
+          0.012782545760273933,
+          -0.017031516879796982,
+          0.0053579797968268394,
+          -0.01559737790375948,
+          -0.013014145195484161,
+          -0.0167375635355711,
+          -0.006079503335058689,
+          0.005148648750036955,
+          -0.02216234989464283,
+          -0.012942884117364883,
+          -0.012862714938819408,
+          -0.009860821068286896,
+          -0.01879524067044258,
+          -0.011339498683810234,
+          -0.012016483582556248,
+          -0.019204994663596153,
+          -0.035559527575969696,
+          -0.014626438729465008,
+          -0.0037456867285072803,
+          -0.011615636758506298,
+          -0.0038414443843066692,
+          -0.03634340316057205,
+          0.012916160747408867,
+          6.475478585343808e-05,
+          -0.004371452145278454,
+          0.003963925410062075,
+          0.013397176750004292,
+          -0.00856920424848795,
+          -0.03237057104706764,
+          -0.028077062219381332,
+          -0.01972164213657379,
+          -0.00024579677847214043,
+          -0.00031984198722057045,
+          0.011259329505264759,
+          0.0004286829207558185,
+          0.017058240249753,
+          -0.01053780596703291,
+          0.017672870308160782,
+          -0.03586238995194435,
+          -0.03196081891655922,
+          -0.003759048180654645,
+          -0.008729543536901474,
+          -0.012043206952512264,
+          0.006409088149666786,
+          0.007037080824375153,
+          -0.0055851261131465435,
+          -0.012354975566267967,
+          -0.007117250002920628,
+          -0.016407977789640427,
+          -0.008925512433052063,
+          0.009210558608174324,
+          0.003333705710247159,
+          0.012809268198907375,
+          0.0150540079921484,
+          0.00868945848196745,
+          -0.01657722517848015,
+          0.01975727267563343,
+          -0.005362433847039938,
+          -0.003008574713021517,
+          0.013512976467609406,
+          0.010110235773026943,
+          0.0167375635355711,
+          0.01130386721342802,
+          0.004306871443986893,
+          -0.014964930713176727,
+          0.02911035344004631,
+          0.015846792608499527,
+          0.009174928069114685,
+          0.01543703954666853,
+          0.015151992440223694,
+          0.009081397205591202,
+          -0.01202539075165987,
+          -0.0007176263607107103,
+          -0.017655055969953537,
+          0.0008055898360908031,
+          0.007353303954005241,
+          -0.010992098599672318,
+          -0.006070595700293779,
+          0.033350419253110886,
+          0.0022703493013978004,
+          -0.015730993822216988,
+          0.013138853013515472,
+          -0.011250421404838562,
+          -0.006720857694745064,
+          -0.011214790865778923,
+          -0.0118472371250391,
+          -0.003019709372892976,
+          0.015303423628211021,
+          0.011366221122443676,
+          -0.02103998139500618,
+          0.011437483131885529,
+          0.01303196046501398,
+          -0.0057098339311778545,
+          -0.005335710477083921,
+          0.014528454281389713,
+          0.01870616339147091,
+          0.01433248445391655,
+          -0.03628995642066002,
+          -0.0023271357640624046,
+          0.012078837491571903,
+          -0.026527121663093567,
+          -0.019383149221539497,
+          0.013940545730292797,
+          0.0012548718368634582,
+          -0.019044656306505203,
+          0.01642579399049282,
+          -0.002663401421159506,
+          -0.022874966263771057,
+          -0.008542481809854507,
+          0.0034851364325731993,
+          0.019935425370931625,
+          0.021129056811332703,
+          -0.012542037293314934,
+          0.008217350579798222,
+          -0.014697699807584286,
+          0.004286829382181168,
+          -0.023355981335043907,
+          -0.0037456867285072803,
+          -0.008560297079384327,
+          -0.0021345068234950304,
+          0.017672870308160782,
+          0.008729543536901474,
+          0.02312438189983368,
+          0.019971057772636414,
+          0.0006380138220265508,
+          -0.016167471185326576,
+          -0.003306982573121786,
+          0.011232606135308743,
+          -0.008208442479372025,
+          -0.0002967376494780183,
+          -0.001585570047609508,
+          0.012907253578305244,
+          0.005393610801547766,
+          0.030642477795481682,
+          -0.002772520761936903,
+          0.0012994103599339724,
+          -3.0011284252395853e-05,
+          0.002788109239190817,
+          0.0012392833596095443,
+          -0.005972610786557198,
+          -0.0018917721463367343,
+          0.005153102800250053,
+          0.02552945911884308,
+          0.004939318168908358,
+          -0.0069123730063438416,
+          0.007976843044161797,
+          -0.0022169030271470547,
+          -0.011642360128462315,
+          -0.0038102674297988415,
+          -0.014840223826467991,
+          -0.01746799424290657,
+          0.01979290321469307,
+          0.0277029387652874,
+          -0.013762392103672028,
+          -0.026954691857099533,
+          -0.00011412987078074366,
+          0.04044094309210777,
+          0.002268122276291251,
+          0.0028727322351187468,
+          0.022946227341890335,
+          -0.0020465434063225985,
+          0.018563641235232353,
+          0.021823858842253685,
+          0.011027729138731956,
+          0.009442158974707127,
+          0.03221023455262184,
+          0.0017158451955765486,
+          -0.004694356583058834,
+          -0.0030286170076578856,
+          -0.0025320129934698343,
+          -0.012497498653829098,
+          0.0009325245628133416,
+          0.002859370782971382,
+          -0.02068367227911949,
+          0.00792785082012415,
+          -0.01959693431854248,
+          -0.0006819955888204277,
+          -0.003652155864983797,
+          0.0065382495522499084,
+          -0.025476014241576195,
+          -0.010047882795333862,
+          0.009967712685465813,
+          -0.01043982058763504,
+          0.0023739011958241463,
+          -0.01915154792368412,
+          -0.004184390883892775,
+          0.022055458277463913,
+          -0.004451621789485216,
+          0.013887099921703339,
+          -0.00841331947594881,
+          -0.0011318342294543982,
+          -0.008404412306845188,
+          -0.0066451421007514,
+          0.011196975596249104,
+          -0.0058122724294662476,
+          0.001120699685998261,
+          -0.010876297950744629,
+          0.008065919391810894,
+          -0.005634118337184191,
+          0.004805702716112137,
+          0.023712290450930595,
+          -0.0030442054849117994,
+          -0.0068901036866009235,
+          0.007424565497785807,
+          0.010038974694907665,
+          -0.022411765530705452,
+          0.02702595293521881,
+          -0.011348405852913857,
+          0.02059459500014782,
+          0.00296626309864223,
+          -0.019703825935721397,
+          -0.0027168476954102516,
+          -0.0010488813277333975,
+          0.010867390781641006,
+          -0.006649595685303211,
+          0.002970716916024685,
+          -0.01831422559916973,
+          0.015276700258255005,
+          0.005945887882262468,
+          -0.015330146998167038,
+          0.01979290321469307,
+          -0.006988088600337505,
+          0.0019663742277771235,
+          -0.017485808581113815,
+          0.00578109547495842,
+          0.007130611687898636,
+          -0.013664407655596733,
+          -0.003739005886018276,
+          -0.019703825935721397,
+          -0.0005723195499740541,
+          -0.01565973088145256,
+          0.0036098442506045103,
+          -0.009718297980725765,
+          -0.013646592386066914,
+          -0.036913495510816574,
+          -0.0001862265489762649,
+          0.004685448482632637,
+          -0.007197419181466103,
+          -0.001399621949531138,
+          0.00012637795589398593,
+          0.013958360999822617,
+          0.009290727786719799,
+          -0.0039572445675730705,
+          0.004979402758181095,
+          0.034348081797361374,
+          -0.04421781003475189,
+          -0.01138403732329607,
+          -0.00033654391882009804,
+          0.004008463583886623,
+          -0.020737119019031525,
+          -0.008275250904262066,
+          0.013121037743985653,
+          -0.0007081619114615023,
+          0.016274362802505493,
+          -0.016033854335546494,
+          0.010947559960186481,
+          0.003079836256802082,
+          0.008177265524864197,
+          -0.007135065272450447,
+          -0.0053579797968268394,
+          -0.012034298852086067,
+          -0.000576773367356509,
+          -0.001016590977087617,
+          -0.02827303111553192,
+          0.01734328642487526,
+          0.0016033854335546494,
+          -0.0056296647526323795,
+          0.0069480035454034805,
+          -0.02453179843723774,
+          -0.009148204699158669,
+          -0.012292621657252312,
+          0.023658843711018562,
+          -0.01738782413303852,
+          -0.00026625662576407194,
+          -0.004698810167610645,
+          -0.006444718688726425,
+          0.014724423177540302,
+          -0.0037234174087643623,
+          0.007322126999497414,
+          0.00037635021726600826,
+          -0.011339498683810234,
+          -0.002014253055676818,
+          0.0024785667192190886,
+          -0.010190405882894993,
+          0.031693585216999054,
+          -0.005959249567240477,
+          -0.0027702937368303537,
+          0.007433473132550716,
+          0.0010666967136785388,
+          -0.002318228129297495,
+          0.010270575061440468,
+          0.0017882202519103885,
+          0.0039572445675730705,
+          0.008894335478544235,
+          -0.0036989212967455387,
+          -0.006021603476256132,
+          0.016007130965590477,
+          0.020541150122880936,
+          0.005616303067654371,
+          -0.016033854335546494,
+          -0.011116806417703629,
+          0.011758159846067429,
+          0.0015733219916000962,
+          0.0006396839744411409,
+          -0.008057012222707272,
+          0.006324464920908213,
+          -0.002256987849250436,
+          0.012043206952512264,
+          0.011838329024612904,
+          0.017298748716711998,
+          0.012898345477879047,
+          -0.005723195616155863,
+          0.03144416958093643,
+          0.01614074781537056,
+          0.0009007908520288765,
+          0.015134177170693874,
+          -0.04368334636092186,
+          -0.01027948223054409,
+          -0.012078837491571903,
+          0.01754816249012947,
+          -0.014751146547496319,
+          -0.016488147899508476,
+          -0.013183391653001308,
+          0.010653605684638023,
+          -0.0032446286641061306,
+          0.005473779980093241,
+          0.0251375213265419,
+          -0.013414992019534111,
+          0.004393721465021372,
+          -0.002739116782322526,
+          0.023623213171958923,
+          0.0030263899825513363,
+          0.00907694362103939,
+          0.007495827041566372,
+          0.005104110576212406,
+          -0.0027057130355387926,
+          0.01963256485760212,
+          -0.009219466708600521,
+          0.036485929042100906,
+          -0.01053780596703291,
+          -0.009317451156675816,
+          -0.01582006923854351,
+          -0.017637239769101143,
+          -0.012354975566267967,
+          0.004752256441861391,
+          -0.000690346525516361,
+          0.0012570987455546856,
+          -0.00821289699524641,
+          0.013637684285640717,
+          0.03140854090452194,
+          -0.0013417218578979373,
+          0.015160900540649891,
+          0.007856588810682297,
+          -0.0006513753323815763,
+          0.01543703954666853,
+          0.003883755998685956,
+          -0.004489479120820761,
+          0.004729987122118473,
+          0.024513982236385345,
+          0.0004924286040477455,
+          0.00767843471840024,
+          -0.0010744909523054957,
+          -0.009602497331798077,
+          -0.01156219094991684,
+          0.010831759311258793,
+          0.027115030214190483,
+          0.017334379255771637,
+          0.001331700710579753,
+          -0.0118472371250391,
+          -0.00017815394676290452,
+          -0.0009375351364724338,
+          0.016791008412837982,
+          -0.010965375229716301,
+          -0.006689680740237236,
+          0.013121037743985653,
+          -0.009085850790143013,
+          0.007611627224832773,
+          -0.01662176288664341,
+          0.0013907141983509064,
+          0.00722859613597393,
+          0.014270130544900894,
+          -0.0045206560753285885,
+          0.006070595700293779,
+          -0.00707716541364789,
+          0.010555621236562729,
+          -0.0127290990203619,
+          0.011277144774794579,
+          -0.005723195616155863,
+          0.00046709735761396587,
+          0.028807492926716805,
+          -0.006475895643234253,
+          0.008195081725716591,
+          0.00011621761223068461,
+          -0.0034717749804258347,
+          -0.017494717612862587,
+          0.025761060416698456,
+          0.004084179177880287,
+          0.005914710927754641,
+          0.00020014481560792774,
+          0.010288390330970287,
+          -0.008301973342895508,
+          -0.001823851023800671,
+          0.022304873913526535,
+          0.011927406303584576,
+          0.0038703943137079477,
+          -0.0005873512709513307,
+          0.008003565482795238,
+          -0.012221360579133034,
+          0.015134177170693874,
+          0.01899120956659317,
+          0.008644919842481613,
+          0.0036543826572597027,
+          0.014599715359508991,
+          -0.013094314374029636,
+          0.008466766215860844,
+          -0.009072489105165005,
+          0.029163800179958344,
+          0.012640022672712803,
+          -0.016523778438568115,
+          0.019294071942567825,
+          0.030767185613512993,
+          -0.004930410534143448,
+          -0.010430913418531418,
+          0.0015844566514715552,
+          -0.017414547502994537,
+          -0.0021567761432379484,
+          -0.014920393005013466,
+          -0.014768961817026138,
+          0.005126379895955324,
+          -0.015312331728637218,
+          0.02497718296945095,
+          -0.017280932515859604,
+          -0.031657956540584564,
+          -0.004309098701924086,
+          -0.00825298111885786,
+          -0.02818395383656025,
+          -0.008301973342895508,
+          0.006190849468111992,
+          0.014849130995571613,
+          -0.0019930971320718527,
+          -0.0144928228110075,
+          0.013664407655596733,
+          0.009085850790143013,
+          0.01359314564615488,
+          -0.017316563054919243,
+          -0.01606057770550251,
+          -0.015615193173289299,
+          0.017316563054919243,
+          -0.010956467129290104,
+          -0.016470331698656082,
+          0.014270130544900894,
+          0.003759048180654645,
+          0.005464872345328331,
+          0.0017503624549135566,
+          -0.014519546180963516,
+          -0.016354531049728394,
+          -0.027845460921525955,
+          0.010626882314682007,
+          -0.0027970168739557266,
+          -0.010974282398819923,
+          0.0029194976668804884,
+          0.011437483131885529,
+          0.00939762033522129,
+          0.005749918520450592,
+          -0.018242964521050453,
+          0.011268236674368382,
+          0.01311213057488203,
+          -0.018528010696172714,
+          -0.007429019547998905,
+          0.005451510660350323,
+          -0.009727205149829388,
+          0.03798241913318634,
+          0.01345062255859375,
+          -0.015205439180135727,
+          -0.0030397516675293446,
+          0.004930410534143448,
+          0.01686227135360241,
+          0.010056789964437485,
+          -0.018848687410354614,
+          -0.001327246893197298,
+          0.021895119920372963,
+          -0.004549606237560511,
+          0.004366998560726643,
+          -3.483918590063695e-06,
+          0.042614422738552094,
+          0.0033426133450120687,
+          -0.0056875646114349365,
+          -0.013138853013515472,
+          0.002210222417488694,
+          0.00997662078589201,
+          0.013023053295910358,
+          0.012239175848662853,
+          -0.007326581049710512,
+          0.0025008360389620066,
+          0.02020265720784664,
+          -0.010395282879471779,
+          0.0020955358631908894,
+          0.0030352978501468897,
+          0.005936980247497559,
+          -0.016880085691809654,
+          0.026242075487971306,
+          -0.0009653716697357595,
+          0.00692573469132185,
+          0.0004620867839548737,
+          -0.009923174977302551,
+          0.014670977368950844,
+          0.024122044444084167,
+          0.0008417773642577231,
+          0.015624100342392921,
+          -0.012435144744813442,
+          0.01831422559916973,
+          0.007901127450168133,
+          0.00782986544072628,
+          -0.015330146998167038,
+          -0.010092420503497124,
+          0.006952457595616579,
+          -0.0028816401027143,
+          -0.002538693603128195,
+          -0.0039839674718678,
+          -0.012372790835797787,
+          -0.011936314404010773,
+          -0.021129056811332703,
+          0.004373679403215647,
+          0.00836878176778555,
+          -0.006132949609309435,
+          0.006850019097328186,
+          0.004309098701924086,
+          -0.0033671094570308924,
+          0.0005984858726151288,
+          -0.013682222925126553,
+          -0.008707273751497269,
+          0.0033470671623945236,
+          0.002696805400773883,
+          0.004191071726381779,
+          -0.007856588810682297,
+          0.020184841006994247,
+          0.006609511096030474,
+          -0.006110680289566517,
+          0.007874404080212116,
+          -0.019062472507357597,
+          -0.0028660516254603863,
+          -0.012105559930205345,
+          0.00011127662583021447,
+          0.0042200214229524136,
+          0.01638125441968441,
+          0.0031243746634572744,
+          -0.023747920989990234,
+          0.0068322038277983665,
+          -0.01202539075165987,
+          0.008965597487986088,
+          -0.003741232678294182,
+          -0.0005160897271707654,
+          -0.012693468481302261,
+          0.01594477705657482,
+          0.01535686943680048,
+          -0.011419667862355709,
+          -0.021093426272273064,
+          0.017637239769101143,
+          -0.03103441745042801,
+          0.022108905017375946,
+          0.010671420954167843,
+          0.0036855596117675304,
+          -0.019810717552900314,
+          0.017120594158768654,
+          0.012399514205753803,
+          0.01101882103830576,
+          -0.0032936211209744215,
+          0.011294960044324398,
+          0.021627888083457947,
+          -0.007941211573779583,
+          -0.0010410870891064405,
+          -9.525320820102934e-06,
+          -0.010430913418531418,
+          -0.0002900568943005055,
+          -0.007175149861723185,
+          0.014421561732888222,
+          0.00836878176778555,
+          -0.008836435154080391,
+          -0.013548607006669044,
+          0.01228371448814869,
+          -0.011482021771371365,
+          0.007117250002920628,
+          -0.005985972471535206,
+          0.007219688501209021,
+          0.004342502448707819,
+          0.01770850270986557,
+          -0.021057795733213425,
+          0.0005670305690728128,
+          0.005237726029008627,
+          -0.022946227341890335,
+          0.007041534408926964,
+          -0.024585243314504623,
+          0.014902577735483646,
+          -0.025351306423544884,
+          -0.005549495108425617,
+          0.02046988718211651,
+          0.014599715359508991,
+          0.0033092093653976917,
+          -0.018011363223195076,
+          -0.007059350144118071,
+          -0.013619869016110897,
+          0.002101103076711297,
+          -0.022429581731557846,
+          -0.013388268649578094,
+          -0.015303423628211021,
+          0.009531236253678799,
+          0.011027729138731956,
+          0.0024028513580560684,
+          -0.010582344606518745,
+          0.007535911630839109,
+          -0.02168133482336998,
+          -0.010617975145578384,
+          -0.014653162099421024,
+          -0.017681779339909554,
+          0.001418550731614232,
+          0.02661619894206524,
+          -0.0027413438074290752,
+          0.01879524067044258,
+          0.011954129673540592,
+          0.01471551600843668,
+          -0.0031733671203255653,
+          -0.010938651859760284,
+          -0.00923728197813034,
+          -0.0027925630565732718,
+          -0.003554171184077859,
+          -0.018777426332235336,
+          -0.012559852562844753,
+          0.00489923357963562,
+          -0.0019853028934448957,
+          0.003981740679591894,
+          0.007874404080212116,
+          0.00841331947594881,
+          -0.01100100576877594,
+          -0.010751590132713318,
+          0.016630670055747032,
+          -0.025671983137726784,
+          -0.017405640333890915,
+          0.018278595060110092,
+          -0.011232606135308743,
+          -0.0012348295422270894,
+          0.019864164292812347,
+          -0.010956467129290104,
+          -0.007491373457014561,
+          0.010493267327547073,
+          -0.003865940496325493,
+          0.010528897866606712,
+          0.022180166095495224,
+          0.006484803277999163,
+          -0.016087301075458527,
+          0.008595927618443966,
+          0.010609067045152187,
+          -0.014163237996399403,
+          -0.02734662964940071,
+          -0.0026010475121438503,
+          0.004560741130262613,
+          -0.020701488479971886,
+          -0.0026945783756673336,
+          0.0004078055208083242,
+          0.00722859613597393,
+          -0.0003855362592730671,
+          0.004925956483930349,
+          -0.018100440502166748,
+          -0.007139519322663546,
+          0.003429463366046548,
+          0.00868945848196745,
+          -0.022215796634554863,
+          0.0013873737771064043,
+          -0.013227930292487144,
+          0.02216234989464283,
+          -0.003257990116253495,
+          0.007415657863020897,
+          -0.0012548718368634582,
+          -0.0074201119132339954,
+          0.008769627660512924,
+          0.012069929391145706,
+          0.005972610786557198,
+          0.03295847773551941,
+          -0.003894890658557415,
+          -0.015009469352662563,
+          0.014234500005841255,
+          -0.025885768234729767,
+          -0.0007337715360336006,
+          -0.0038325367495417595,
+          0.010626882314682007,
+          -0.011036636307835579,
+          -0.011188067495822906,
+          0.02203764207661152,
+          0.02100434899330139,
+          -0.006930188275873661,
+          -0.0002297907485626638,
+          -0.01750362478196621,
+          0.01662176288664341,
+          -0.00824852753430605,
+          0.0015833431389182806,
+          0.007976843044161797,
+          -0.023320350795984268,
+          -0.012631114572286606,
+          -0.006021603476256132,
+          0.024870289489626884,
+          -0.01934751868247986,
+          -0.016559408977627754,
+          -0.012515314854681492,
+          0.00012282878742553294,
+          -0.02709721401333809,
+          -0.0075225504115223885,
+          0.007063803728669882,
+          -0.012925068847835064,
+          -0.007553727366030216,
+          0.007954573258757591,
+          0.009477789513766766,
+          -0.0021033298689872026,
+          -0.02520878240466118,
+          -0.011188067495822906,
+          0.013210115022957325,
+          0.003939428832381964,
+          0.006190849468111992,
+          -0.004364771768450737,
+          0.006168580148369074,
+          -0.006279926281422377,
+          0.0032513095065951347,
+          0.00899231992661953,
+          -0.0016490374691784382,
+          -0.006342280190438032,
+          0.02219798043370247,
+          0.005522772204130888,
+          -0.018180610612034798,
+          0.01433248445391655,
+          0.01570427045226097,
+          -0.012185730040073395,
+          -0.008489035069942474,
+          3.824046507361345e-05,
+          -0.013931638561189175,
+          -0.0008918831590563059,
+          -0.006475895643234253,
+          0.03196081891655922,
+          -0.022874966263771057,
+          0.003857032861560583,
+          0.008685004897415638,
+          -0.0002842112153302878,
+          -0.004903687164187431,
+          0.002912817057222128,
+          0.0054826876148581505,
+          -0.0012793679488822818,
+          0.00577218784019351,
+          0.00812827330082655,
+          -0.014216684736311436,
+          -0.006337826605886221,
+          0.013691130094230175,
+          0.002291504992172122,
+          0.017806487157940865,
+          0.016452517360448837,
+          1.9311608411953785e-05,
+          -0.03313663229346275,
+          0.004193298518657684,
+          0.0007877744501456618,
+          -0.021627888083457947,
+          -0.0038169482722878456,
+          -0.006106226239353418,
+          0.00967375934123993,
+          0.010324020870029926,
+          0.01614074781537056,
+          0.0029194976668804884,
+          0.007143972907215357,
+          0.006141857244074345,
+          0.010288390330970287,
+          -0.030927523970603943,
+          -0.006444718688726425,
+          -0.008333150297403336,
+          -0.006453626789152622,
+          0.012622206471860409,
+          0.012631114572286606,
+          0.029983308166265488,
+          0.028736229985952377,
+          -0.014751146547496319,
+          -0.019204994663596153,
+          0.004242290742695332,
+          0.011027729138731956,
+          0.008640466257929802,
+          -0.007290950044989586,
+          0.006462534423917532,
+          8.504067227477208e-05,
+          -0.02526222914457321,
+          0.0017392279114574194,
+          0.01375348400324583,
+          -0.017530348151922226,
+          -0.010617975145578384,
+          -0.018652718514204025,
+          0.014626438729465008,
+          0.02116468921303749,
+          -0.00392384035512805,
+          0.007188511546701193,
+          -0.010591251775622368,
+          -0.009317451156675816,
+          -0.04500168561935425,
+          0.0053000799380242825,
+          0.009054673835635185,
+          0.022215796634554863,
+          0.001312771812081337,
+          -0.004001782741397619,
+          -6.158837641123682e-05,
+          0.007001449819654226,
+          -0.0033960596192628145,
+          0.011464206501841545,
+          0.013210115022957325,
+          -0.018759610131382942,
+          -0.016042763367295265,
+          0.004150986671447754,
+          0.0023115472868084908,
+          -0.010172589682042599,
+          0.005968157202005386,
+          0.021253764629364014,
+          0.01586460880935192,
+          -0.007241957820951939,
+          0.004431579262018204,
+          0.005625210702419281,
+          -0.014225591905415058,
+          -0.012034298852086067,
+          -0.0045429253950715065,
+          -0.011517652310431004,
+          -0.024193305522203445,
+          0.017191855236887932,
+          -0.013824746012687683,
+          0.006395726464688778,
+          0.009869728237390518,
+          0.011223698034882545,
+          0.0018450067145749927,
+          0.023177828639745712,
+          0.00811045803129673,
+          -0.008738450706005096,
+          0.011036636307835579,
+          -0.026170814409852028,
+          0.012684560380876064,
+          -0.00023368786787614226,
+          0.011259329505264759,
+          0.0001613128260942176,
+          -0.03301192447543144,
+          -0.013512976467609406,
+          0.008172811940312386,
+          -0.006622872781008482,
+          0.009896451607346535,
+          0.014626438729465008,
+          -0.008288612589240074,
+          -0.00100879673846066,
+          -0.022340504452586174,
+          0.00445830263197422,
+          -0.03147980198264122,
+          0.00011357313633197919,
+          -0.004716625437140465,
+          0.010929744690656662,
+          -0.005362433847039938,
+          -0.0240151509642601,
+          0.0026789898984134197,
+          -0.012720191851258278,
+          -0.005825634114444256,
+          0.0010616861982271075,
+          -0.0005828974535688758,
+          0.010876297950744629,
+          -0.00593252619728446,
+          0.01375348400324583,
+          -0.007348849903792143,
+          0.005215456709265709,
+          -0.0007688456098549068,
+          0.006792119238525629,
+          -0.006658503785729408,
+          -0.012907253578305244,
+          0.00238948967307806,
+          -0.004845787305384874,
+          0.021342841908335686,
+          0.010243851691484451,
+          -0.0006380138220265508,
+          0.004362544510513544,
+          0.020291734486818314,
+          0.004672087263315916,
+          0.013165576383471489,
+          0.006676319055259228,
+          0.0032000902574509382,
+          -0.015285608358681202,
+          0.009736113250255585,
+          -0.007371119223535061,
+          0.009477789513766766,
+          0.002870505442842841,
+          -0.0010310659417882562,
+          0.014599715359508991,
+          0.008364327251911163,
+          -0.009905358776450157,
+          0.011767067946493626,
+          0.018581455573439598,
+          -0.008408865891397,
+          -0.003220132552087307,
+          -0.0127290990203619,
+          0.011294960044324398,
+          0.027204107493162155,
+          0.004260106012225151,
+          -0.01831422559916973,
+          0.014145422726869583,
+          0.00010097710037371144,
+          -0.011713622137904167,
+          0.026046106591820717,
+          -0.014751146547496319,
+          -0.025689799338579178,
+          -0.013379361480474472,
+          -0.007273134775459766,
+          -0.01750362478196621,
+          0.021271580830216408,
+          -0.011473113670945168,
+          0.012212452478706837,
+          0.004193298518657684,
+          0.03119475580751896,
+          0.010689236223697662,
+          0.01212337613105774,
+          -0.003015255555510521,
+          -0.025351306423544884,
+          0.009379805065691471,
+          0.0045874640345573425,
+          -0.04044094309210777,
+          0.0005979291745461524,
+          -0.003004120895639062,
+          0.029377585276961327,
+          0.0010550053557381034,
+          -0.0014898123918101192,
+          -0.0038971174508333206,
+          -0.020380811765789986,
+          -0.015686454251408577,
+          0.007322126999497414,
+          -0.028682785108685493,
+          0.013397176750004292,
+          -0.00507738720625639,
+          -0.012150098569691181,
+          0.002214676234871149,
+          0.010831759311258793,
+          -0.019044656306505203,
+          0.01879524067044258,
+          0.015846792608499527,
+          0.025030629709362984,
+          0.01069814432412386,
+          -0.008885428309440613,
+          0.020006688311696053,
+          0.0010210447944700718,
+          -0.006591695826500654,
+          0.0020688127260655165,
+          -1.8998447558260523e-05,
+          0.0015410316409543157,
+          -0.0033671094570308924,
+          0.003015255555510521,
+          0.01598040945827961,
+          -0.01738782413303852,
+          0.019418779760599136,
+          -0.01702260971069336,
+          0.016007130965590477,
+          0.0005347402184270322,
+          0.012925068847835064,
+          4.071095827384852e-05,
+          0.0011958583490923047,
+          -0.029840786010026932,
+          -0.0009653716697357595,
+          -0.03915823623538017,
+          -0.011312775313854218,
+          -0.0016679662512615323,
+          -0.018091533333063126,
+          -0.0037679558154195547,
+          0.009032404981553555,
+          -0.01487585436552763,
+          -0.011588914319872856,
+          0.016069484874606133,
+          0.007598265539854765,
+          0.007375573273748159,
+          0.001965260598808527,
+          -8.03084549261257e-05,
+          -0.005162010435014963,
+          -0.020808380097150803,
+          -0.012951791286468506,
+          -0.025155337527394295,
+          -0.025190968066453934,
+          -0.0035742134787142277,
+          0.007584904320538044,
+          -0.004832425620406866,
+          -0.008564750663936138,
+          0.017761947587132454,
+          -0.019044656306505203,
+          -0.00040140311466529965,
+          0.004242290742695332,
+          0.008172811940312386,
+          0.013103222474455833,
+          -0.007063803728669882,
+          0.007384480908513069,
+          0.0030709286220371723,
+          -0.0013361545279622078,
+          -0.0005183166358619928,
+          0.024104228243231773,
+          -0.008836435154080391,
+          0.011009913869202137,
+          0.007722973357886076,
+          0.011775976046919823,
+          -0.006217572372406721,
+          0.009045766666531563,
+          -0.024104228243231773,
+          0.007687342818826437,
+          -0.011045544408261776,
+          -0.026794353500008583,
+          -0.007148426957428455,
+          0.02071930281817913,
+          -0.0006536022992804646,
+          0.0008662735344842076,
+          -0.0018193970900028944,
+          -0.0004406526277307421,
+          0.023676658049225807,
+          0.01042200531810522,
+          0.0012147872475907207,
+          -0.007700704038143158,
+          -0.029769523069262505,
+          0.005985972471535206,
+          -0.021271580830216408,
+          -0.021699151024222374,
+          0.03288721665740013,
+          0.010368559509515762,
+          0.0027213015127927065,
+          -0.013691130094230175,
+          0.010038974694907665,
+          0.009442158974707127,
+          6.134480645414442e-05,
+          0.014804592356085777,
+          0.003088743891566992,
+          0.001969714416190982,
+          -0.009932082146406174,
+          -0.01918718032538891,
+          0.01614074781537056,
+          0.014581900089979172,
+          -0.01559737790375948,
+          0.036877866834402084,
+          0.016773194074630737,
+          0.012809268198907375,
+          -0.010288390330970287,
+          0.007714065723121166,
+          0.016995886340737343,
+          0.020060133188962936,
+          -0.0070860730484128,
+          0.0034339171834290028,
+          -0.010742682963609695,
+          -0.00023466214770451188,
+          0.03894445300102234,
+          0.03516758978366852,
+          0.008444496430456638,
+          -0.022785888984799385,
+          0.014581900089979172,
+          -0.009335266426205635,
+          -0.01287162210792303,
+          0.010306205600500107,
+          0.01606057770550251,
+          -0.0005923618446104228,
+          0.0014686565846204758,
+          0.009451067075133324,
+          0.0076606194488704205,
+          0.017441270872950554,
+          0.005353525746613741,
+          -0.002378355246037245,
+          0.003402740228921175,
+          -0.0034339171834290028,
+          0.00965594407171011,
+          -0.016808824613690376,
+          -0.005126379895955324,
+          -0.00339828641153872,
+          -0.0016969162970781326,
+          0.005251087248325348,
+          0.009540143422782421,
+          -0.036361221224069595,
+          -0.007415657863020897,
+          -0.033350419253110886,
+          0.004021825268864632,
+          -0.01995324157178402,
+          -0.0035430365242064,
+          -0.010965375229716301,
+          -0.01598040945827961,
+          -0.026562752202153206,
+          -0.016933532431721687,
+          -0.0031266016885638237,
+          0.008903243578970432,
+          -0.003888209816068411,
+          0.003215678734704852,
+          0.003436144208535552,
+          -0.021378472447395325,
+          0.005660841707140207,
+          0.013174484483897686,
+          0.009370896965265274,
+          0.021057795733213425,
+          0.0001496214681537822,
+          0.014368114992976189,
+          -0.014822407625615597,
+          -0.04029842093586922,
+          -0.0007649484905414283,
+          0.00031009921804070473,
+          -0.005175372119992971,
+          -0.00424674479290843,
+          0.004859148990362883,
+          -0.009379805065691471,
+          0.011098990216851234,
+          -0.0021745916455984116,
+          0.0009681553347036242,
+          -0.015722084790468216,
+          -0.010707051493227482,
+          -0.023587582632899284,
+          -0.00505066430196166,
+          -0.013121037743985653,
+          -2.176470479753334e-05,
+          -0.01288943737745285,
+          -0.004952679388225079,
+          -0.005398064386099577,
+          -0.0013461756752803922,
+          -0.006426903419196606,
+          -0.002723528305068612,
+          -0.001202539075165987,
+          -0.013263560831546783,
+          -0.012559852562844753,
+          -0.0013172257458791137,
+          0.004832425620406866,
+          0.0006942436448298395,
+          -0.01098319049924612,
+          0.013414992019534111,
+          -0.014911484904587269,
+          0.0018038086127489805,
+          0.02574324421584606,
+          0.013325914740562439,
+          -0.010484359227120876,
+          0.016479238867759705,
+          0.002199087757617235,
+          0.010128051973879337,
+          -0.02152099646627903,
+          -0.015722084790468216,
+          -0.010110235773026943,
+          0.026990322396159172,
+          0.0012682334054261446,
+          -0.013557515107095242,
+          -0.007206326816231012,
+          -0.01706714741885662,
+          0.004680994898080826,
+          -0.0077407886274158955,
+          0.011375129222869873,
+          -0.020630227401852608,
+          -0.007215234450995922,
+          -0.011473113670945168,
+          -0.007531458046287298,
+          0.0065827881917357445,
+          0.03794679045677185,
+          -0.00289054773747921,
+          -0.007852135226130486,
+          0.005963703151792288,
+          0.010849575512111187,
+          0.028237400576472282,
+          0.0010516649344936013,
+          -0.0014307989040389657,
+          -0.012043206952512264,
+          0.0034717749804258347,
+          0.004551833029836416,
+          -0.01051999069750309,
+          0.017031516879796982,
+          -0.009148204699158669,
+          -0.01934751868247986,
+          -0.004500614013522863,
+          -0.00868945848196745,
+          -0.007353303954005241,
+          0.01753925532102585,
+          0.0023226819466799498,
+          0.003208997892215848,
+          0.0008540254784747958,
+          -0.007250865455716848,
+          -0.008333150297403336,
+          0.011009913869202137,
+          0.08451622724533081,
+          0.00943325087428093,
+          -0.02228705771267414,
+          0.0012526449281722307,
+          0.004943771753460169,
+          0.001479791128076613,
+          -0.014795685186982155,
+          0.06591695547103882,
+          -0.014368114992976189,
+          0.005451510660350323,
+          0.0004428795655258,
+          0.004810156300663948,
+          -0.004447167739272118,
+          0.02136065810918808,
+          0.010199313051998615,
+          -0.01244405284523964,
+          0.002025387715548277,
+          -0.03541700541973114,
+          -0.017280932515859604,
+          0.0034984981175512075,
+          -0.0044404868967831135,
+          -0.004393721465021372,
+          0.013121037743985653,
+          -0.023106565698981285,
+          -0.008163904771208763,
+          0.010956467129290104,
+          -0.012052114121615887,
+          0.011473113670945168,
+          -0.019383149221539497,
+          -0.001312771812081337,
+          -0.0020198202691972256,
+          0.0014519545948132873,
+          -0.009727205149829388,
+          0.0036766519770026207,
+          -0.0027123938780277967,
+          0.0010705938329920173,
+          0.009477789513766766,
+          0.0077853272669017315,
+          -0.003741232678294182,
+          0.017877748236060143,
+          0.013441715389490128,
+          0.0035563979763537645,
+          0.017485808581113815,
+          0.0027480246499180794,
+          0.007651711814105511,
+          0.0003485136548988521,
+          0.01785993203520775,
+          -0.014483915641903877,
+          0.021022165194153786,
+          -0.01834985613822937,
+          -0.02059459500014782,
+          -0.013691130094230175,
+          0.009063581936061382,
+          -0.015561746433377266,
+          0.001760383602231741,
+          -0.011571098119020462,
+          0.029858600348234177,
+          0.010885206051170826,
+          -0.003088743891566992,
+          0.004380360245704651,
+          -0.007045988459140062,
+          0.03930075839161873,
+          0.007856588810682297,
+          0.033225711435079575,
+          0.003155551617965102,
+          0.017886655405163765,
+          -0.002788109239190817,
+          0.004001782741397619,
+          -0.013254653662443161,
+          -0.005821180064231157,
+          0.018278595060110092,
+          -0.009905358776450157,
+          0.019062472507357597,
+          0.004863602574914694,
+          -0.014439377002418041,
+          0.01770850270986557,
+          -0.015410316176712513,
+          -0.02292841300368309,
+          0.013842561282217503,
+          -0.02408641390502453,
+          0.0002925621811300516,
+          0.006230934057384729,
+          -0.0036276597529649734,
+          -0.01051999069750309,
+          0.021752595901489258,
+          -0.0005116358515806496,
+          -0.006907918956130743,
+          -0.010644698515534401,
+          -0.013245745562016964,
+          -0.014279038645327091,
+          0.00895668938755989,
+          -0.004574102349579334,
+          -0.01298742275685072,
+          0.011009913869202137,
+          -0.02526222914457321,
+          0.012862714938819408,
+          -0.012043206952512264,
+          0.023017490282654762,
+          -0.00795011967420578,
+          0.02075493521988392,
+          0.0022825973574072123,
+          -0.0002908919705078006,
+          0.0019853028934448957,
+          0.008863158524036407,
+          -0.003224586369469762,
+          0.016327809542417526,
+          0.004819063935428858,
+          0.004997218027710915,
+          0.004583009984344244,
+          -0.005041756667196751,
+          0.015472670085728168,
+          -0.011749252676963806,
+          -0.0009202764485962689,
+          0.01418996136635542,
+          0.009290727786719799,
+          -0.018207332119345665,
+          0.025850137695670128,
+          0.0038748481310904026,
+          0.015303423628211021,
+          -0.023266904056072235,
+          0.0058924416080117226,
+          0.012622206471860409,
+          0.002320455154404044,
+          0.01288943737745285,
+          -0.004729987122118473,
+          0.022946227341890335,
+          0.018902134150266647,
+          0.004734441172331572,
+          -0.0025854590348899364,
+          0.013040868565440178,
+          -0.002086627995595336,
+          -0.0016456970479339361,
+          0.011891775764524937,
+          -0.0034940443001687527,
+          0.004077498335391283,
+          0.001429685391485691,
+          -0.0006591695710085332,
+          0.016755377873778343,
+          -0.01399399247020483,
+          0.014216684736311436,
+          0.006030511111021042,
+          -0.011063359677791595,
+          -0.013896007090806961,
+          0.007099434733390808,
+          -0.015080731362104416,
+          0.0025097436737269163,
+          0.023177828639745712,
+          -0.0014775642193853855,
+          -0.00865382794290781,
+          0.0048502408899366856,
+          0.012381698936223984,
+          -0.0058479029685258865,
+          -0.010484359227120876,
+          0.006181941833347082,
+          0.008221804164350033,
+          0.00027766963467001915,
+          0.006596149876713753,
+          0.0068099345080554485,
+          0.007028173189610243,
+          -0.0007922283257357776,
+          -0.008195081725716591,
+          0.0030508863274008036,
+          0.020933087915182114,
+          -0.0008629331714473665,
+          -0.009210558608174324,
+          -0.03643248230218887,
+          -0.003191182389855385,
+          -0.014679884538054466,
+          0.003086517099291086,
+          0.00030870738555677235,
+          -0.0008350965799763799,
+          -0.004493933171033859,
+          -0.009638127870857716,
+          0.009099212475121021,
+          0.0008907697047106922,
+          0.024745581671595573,
+          -0.012693468481302261,
+          -0.015365777537226677,
+          0.003257990116253495,
+          -0.0055405874736607075,
+          -0.010769405402243137,
+          0.007451288867741823,
+          -0.004213340580463409,
+          0.03269124776124954,
+          0.00901013519614935,
+          0.0025052898563444614,
+          -0.014786777086555958,
+          -0.018813056871294975,
+          0.010208221152424812,
+          -0.00245852442458272,
+          -0.005598487798124552,
+          -0.003939428832381964,
+          -0.02994767762720585,
+          0.010324020870029926,
+          -0.00967375934123993,
+          0.0036744249518960714,
+          -0.0044182175770401955,
+          0.009522328153252602,
+          0.003327024867758155,
+          -0.01487585436552763,
+          -0.0030375246424227953,
+          0.013566422276198864,
+          0.015258884988725185,
+          -0.012435144744813442,
+          0.004451621789485216,
+          -0.005999334156513214,
+          -0.012185730040073395,
+          0.009326359257102013,
+          -0.00967375934123993,
+          -0.017655055969953537,
+          0.002447389764711261
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "611ac443-aa6f-4416-a1f9-dade7d45888c",
+      "properties": {
+        "page_content": "6. Alignment For Deepseekmoe 16B\n\nPrevious research indicates that MoE models typically do not emerge significant gains from fine-tuning (Artetxe et al., 2022; Fedus et al., 2021). However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B.\n\n6.1. Experimental Setup\n\nTraining Data. For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios.\n\nHyper-Parameters. During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).\n\nWe employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10−5 without incorporating any learning rate scheduling strategy.\n\nEvaluation Benchmarks. For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3, with the following adjustments: (1) We exclude Pile (Gao et al., 2020) since chat models are seldom employed for pure language modeling. (2) We exclude\n\nMetric # Shot LLaMA2 DeepSeek DeepSeekMoE SFT 7B Chat 7B Chat 16B # Total Params N/A 6.7B 6.9B 16.4B # Activated Params N/A 6.7B 6.9B 2.8B FLOPs per 4K Tokens N/A 187.9T 183.5T 74.4T HellaSwag (Acc.) 0-shot 67.9 71.0 72.2 PIQA (Acc.) 0-shot 76.9 78.4 79.7 ARC-easy (Acc.) 0-shot 69.7 70.2 69.9 ARC-challenge (Acc.) 0-shot 50.8 50.2 50.0 BBH (EM) 3-shot 39.3 43.1 42.2 RACE-middle (Acc.) 5-shot 63.9 66.1 64.8 RACE-high (Acc.) 5-shot 49.6 50.8 50.6 DROP (EM) 1-shot 40.0 41.7 33.8 GSM8K (EM) 0-shot 63.4 62.6 62.2 MATH (EM) 4-shot 13.5 14.7 15.2 HumanEval (Pass@1) 0-shot 35.4 45.1 45.7 MBPP (Pass@1) 3-shot 27.8 39.0 46.2 TriviaQA (EM) 5-shot 60.1 59.5 63.3 NaturalQuestions (EM) 0-shot 35.2 32.7 35.1 MMLU (Acc.) 0-shot 50.0 49.7 47.2 WinoGrande (Acc.) 0-shot 65.1 68.4 69.0 CLUEWSC (EM) 5-shot 48.4 66.2 68.2 CEval (Acc.) 0-shot 35.1 44.7 40.0 CMMLU (Acc.) 0-shot 36.9 51.2 49.3\n\nCHID (Zheng et al., 2019) due to the observed instability of results, hindering the derivation of solid conclusions. (3) We additionally include BBH (Suzgun et al., 2022) to provide a more comprehensive assessment of the reasoning ability of the chat models.\n\nTable 5 | Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations.\n\n6.2. Evaluations\n\nBaselines. In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. Correspondingly, we construct three chat models, including LLaMA2 SFT 7B3, DeepSeek Chat 7B, and DeepSeekMoE Chat 16B.\n\nSubsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks.\n\nResults. The evaluation results are presented in Table 5. Our key observations include: (1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intensive tasks (TriviaQA, NaturalQuestions). (2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. In addition, it also surpasses DeepSeek Chat 7B. (3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1). However, it is worth noting that, after supervised finetuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. (4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations.\n\n",
+        "keyphrases": [
+          "DeepSeekMoE 16B",
+          "supervised fine-tuning",
+          "chat model",
+          "bilingual scenarios",
+          "evaluation benchmarks"
+        ],
+        "title": "Alignment For Deepseekmoe 16B",
+        "embedding": [
+          0.0034426983911544085,
+          -0.007639183662831783,
+          -0.011979537084698677,
+          0.00880228541791439,
+          0.011939010582864285,
+          0.013551952317357063,
+          -0.013365532271564007,
+          0.05112782120704651,
+          -0.01517299935221672,
+          0.02819000743329525,
+          -0.013219637796282768,
+          0.0074000791646540165,
+          -0.008145760744810104,
+          0.00818223413079977,
+          0.01559447217732668,
+          -0.004850983154028654,
+          0.02065213769674301,
+          0.04480573907494545,
+          0.0024822282139211893,
+          -0.021673398092389107,
+          -0.019825404509902,
+          -0.0646149292588234,
+          -0.037348922342061996,
+          0.011112277396023273,
+          0.0042836167849600315,
+          0.004996877163648605,
+          0.019955087453126907,
+          0.009685755707323551,
+          -0.005908716004341841,
+          0.017928779125213623,
+          -0.009920807555317879,
+          0.01728036068379879,
+          0.021122241392731667,
+          -0.012060589157044888,
+          -0.010982593521475792,
+          -0.028498005121946335,
+          0.05323518067598343,
+          0.011598590761423111,
+          -0.01838267222046852,
+          0.021511293947696686,
+          -0.009807334281504154,
+          0.016680574044585228,
+          -0.01202006358653307,
+          -0.014605632983148098,
+          0.0015673497691750526,
+          -0.021851712837815285,
+          -0.03183736279606819,
+          -0.01961466856300831,
+          -0.0014396923361346126,
+          -0.032809991389513016,
+          -0.012806271202862263,
+          -0.0026564907748252153,
+          -0.007140711881220341,
+          0.010480068624019623,
+          -0.033263884484767914,
+          0.034593142569065094,
+          0.045778367668390274,
+          0.03880786523222923,
+          0.04791814833879471,
+          -0.014200371690094471,
+          0.028481794521212578,
+          -0.0202792976051569,
+          0.006269399076700211,
+          -0.01276574470102787,
+          -0.017247939482331276,
+          -0.04185543209314346,
+          -0.018755514174699783,
+          -0.01043954212218523,
+          -0.004026275593787432,
+          0.014378686435520649,
+          -0.025904331356287003,
+          0.03783523663878441,
+          -0.018204357475042343,
+          -0.008129550144076347,
+          0.0036027769092470407,
+          0.0015926785999909043,
+          0.02387802116572857,
+          0.028433164581656456,
+          -0.003852012800052762,
+          -0.03744618594646454,
+          0.015083841979503632,
+          0.006763818673789501,
+          -0.014313844963908195,
+          0.028838425874710083,
+          0.06814881414175034,
+          -0.034366194158792496,
+          -0.024753388017416,
+          -0.07748604565858841,
+          0.006536871660500765,
+          0.0009523651096969843,
+          -0.027719903737306595,
+          -0.019079722464084625,
+          -0.01313048042356968,
+          -0.028757372871041298,
+          -0.025061385706067085,
+          0.0002973607915919274,
+          -0.033263884484767914,
+          -0.007444658316671848,
+          -0.045065104961395264,
+          0.016680574044585228,
+          0.025353174656629562,
+          0.017150677740573883,
+          0.010893435217440128,
+          -0.011955221183598042,
+          0.004008038435131311,
+          0.010763752274215221,
+          -0.020749401301145554,
+          -0.015805207192897797,
+          -0.004713193979114294,
+          0.08585064858198166,
+          -0.03151315450668335,
+          -0.012919744476675987,
+          -0.02702285349369049,
+          0.017572149634361267,
+          0.009450703859329224,
+          -0.010455752722918987,
+          -0.026390643790364265,
+          -0.03867818042635918,
+          0.0368950292468071,
+          -0.02243529073894024,
+          0.015707945451140404,
+          0.003924959804862738,
+          0.03371777758002281,
+          -0.0426335334777832,
+          -0.029713790863752365,
+          0.030427051708102226,
+          0.0018024016171693802,
+          -0.010585436597466469,
+          0.014921736903488636,
+          0.00511035043746233,
+          -0.017588360235095024,
+          0.0023201233707368374,
+          -0.011841747909784317,
+          0.0060262419283390045,
+          0.02198139764368534,
+          0.02209486998617649,
+          0.011525643989443779,
+          0.022613605484366417,
+          -0.01522973645478487,
+          -0.0017942963168025017,
+          0.0014224686892703176,
+          -0.03297209367156029,
+          0.015278367325663567,
+          0.0035744085907936096,
+          -0.012530692853033543,
+          0.008761758916079998,
+          0.025223489850759506,
+          0.06182673200964928,
+          0.020246876403689384,
+          0.004871245939284563,
+          -0.012619850225746632,
+          0.018868986517190933,
+          -0.028449373319745064,
+          0.024169810116291046,
+          0.0041985115967690945,
+          -0.03540366515517235,
+          0.01993887685239315,
+          0.0004736496484838426,
+          -0.012895428575575352,
+          -0.020311718806624413,
+          0.032647885382175446,
+          -0.031269997358322144,
+          -0.009588493034243584,
+          -0.0018652171129360795,
+          0.027266010642051697,
+          0.03141589090228081,
+          0.0100829117000103,
+          0.00386214442551136,
+          -0.04636194184422493,
+          0.037413764744997025,
+          -0.010585436597466469,
+          0.039650809019804,
+          -0.009531755931675434,
+          0.0054710335098207,
+          0.019792983308434486,
+          -0.006848923396319151,
+          0.008648285642266273,
+          0.007845867425203323,
+          -0.06597661226987839,
+          0.015432367101311684,
+          -0.029324740171432495,
+          0.00394724914804101,
+          0.02901674062013626,
+          -0.004611878655850887,
+          0.026828328147530556,
+          -0.0014569159829989076,
+          0.017491096630692482,
+          -0.0015399946132674813,
+          0.01872309297323227,
+          0.011639117263257504,
+          0.022013816982507706,
+          0.02780095487833023,
+          0.022954024374485016,
+          -0.005102245137095451,
+          0.011679643765091896,
+          -0.0150108952075243,
+          -0.012814376503229141,
+          -0.0010678647086024284,
+          0.008826600387692451,
+          0.010609752498567104,
+          0.003880381118506193,
+          -0.01966330036520958,
+          -0.01102311909198761,
+          -0.0032015678007155657,
+          0.018706882372498512,
+          0.005750664044171572,
+          -0.0019341117003932595,
+          0.021397819742560387,
+          0.011760695837438107,
+          -0.029308529570698738,
+          -0.02637443318963051,
+          -0.021738240495324135,
+          -0.011323013342916965,
+          -0.0371219739317894,
+          0.0024761492386460304,
+          0.017183097079396248,
+          0.016445521265268326,
+          0.001868256600573659,
+          0.013762688264250755,
+          -0.027006642892956734,
+          0.007493289653211832,
+          0.0007132606697268784,
+          -0.013008901849389076,
+          -0.04227690398693085,
+          -0.03841881453990936,
+          0.005268402863293886,
+          0.008648285642266273,
+          -0.0009052534005604684,
+          0.03841881453990936,
+          -0.009393966756761074,
+          0.01789635792374611,
+          0.031043048948049545,
+          0.006435556337237358,
+          0.012109220959246159,
+          0.014184161089360714,
+          -0.035144299268722534,
+          0.023829391226172447,
+          -0.007262290455400944,
+          0.021446451544761658,
+          0.033652935177087784,
+          -0.02037656120955944,
+          0.00043211033334955573,
+          -0.027930639684200287,
+          -0.00955607183277607,
+          0.026617590337991714,
+          -0.006220767740160227,
+          0.0009508453658781946,
+          0.017102045938372612,
+          0.025726014748215675,
+          0.033977143466472626,
+          0.009069757536053658,
+          -0.037640709429979324,
+          0.007237974554300308,
+          0.021187083795666695,
+          -0.00024974255939014256,
+          0.04269837588071823,
+          0.05255434289574623,
+          -0.03050810471177101,
+          -0.018739303573966026,
+          -0.054142966866493225,
+          0.01916077546775341,
+          -0.014848790131509304,
+          -0.011898485012352467,
+          0.014727211557328701,
+          0.007477079052478075,
+          0.009977543726563454,
+          0.02397528477013111,
+          0.017134467139840126,
+          0.004692931193858385,
+          0.012644166126847267,
+          -0.011306802742183208,
+          0.012003852985799313,
+          -0.049993086606264114,
+          0.007148817181587219,
+          0.005118455737829208,
+          -0.027557797729969025,
+          0.003432566998526454,
+          -0.010690804570913315,
+          0.0005987742333672941,
+          0.018804144114255905,
+          0.027557797729969025,
+          0.023780759423971176,
+          0.03151315450668335,
+          0.008931968361139297,
+          -0.046005312353372574,
+          -0.01732899248600006,
+          -0.021170873194932938,
+          -0.03909965232014656,
+          -0.005957347806543112,
+          -0.029875895008444786,
+          0.0010040359338745475,
+          -0.005418349523097277,
+          -0.036311451345682144,
+          -0.0011904563289135695,
+          -0.01503521017730236,
+          0.030443262308835983,
+          -0.017815306782722473,
+          0.04136911779642105,
+          0.056801486760377884,
+          -0.004449774045497179,
+          -0.06717618554830551,
+          0.009321019984781742,
+          -0.0016950072022154927,
+          -0.012392903678119183,
+          -0.018917618319392204,
+          0.014443528838455677,
+          0.020846663042902946,
+          0.03050810471177101,
+          -0.043541319668293,
+          -0.04386553168296814,
+          0.013949109241366386,
+          -0.0058844005689024925,
+          -0.01599973440170288,
+          -0.0034467510413378477,
+          -0.000564326997846365,
+          0.03206430748105049,
+          0.03540366515517235,
+          0.017572149634361267,
+          -0.038580916821956635,
+          -2.4299873985000886e-05,
+          0.01407068781554699,
+          0.02404012717306614,
+          -0.055407386273145676,
+          0.023343076929450035,
+          0.010544910095632076,
+          0.028660111129283905,
+          0.02919505536556244,
+          0.034204091876745224,
+          -0.015578261576592922,
+          5.2050803788006306e-05,
+          -0.017134467139840126,
+          -0.018009832128882408,
+          -0.01895003952085972,
+          0.0024700702633708715,
+          -0.027557797729969025,
+          -0.028676321730017662,
+          -0.041012488305568695,
+          -0.01684267818927765,
+          -0.03013526275753975,
+          0.006500398274511099,
+          -0.018868986517190933,
+          -0.0022836499847471714,
+          -0.010309858247637749,
+          0.01783151738345623,
+          -0.01559447217732668,
+          -0.0027334904298186302,
+          -0.023067498579621315,
+          -0.006881344132125378,
+          0.0018307699356228113,
+          -0.0031286205630749464,
+          0.0011549959890544415,
+          0.0025815172120928764,
+          0.030329788103699684,
+          0.030702630057930946,
+          -0.014362475834786892,
+          -0.02569359354674816,
+          -0.0497337207198143,
+          0.00334948836825788,
+          -0.005219771061092615,
+          0.018982460722327232,
+          0.012806271202862263,
+          -0.03747860714793205,
+          -0.039132073521614075,
+          0.010601647198200226,
+          -0.0371219739317894,
+          -0.034041985869407654,
+          0.01977677270770073,
+          0.012141642160713673,
+          -0.005207613576203585,
+          0.0029644896276295185,
+          -0.005458875559270382,
+          0.03174009919166565,
+          0.0023403866216540337,
+          -0.0022897287271916866,
+          0.024883070960640907,
+          0.02076561190187931,
+          0.008413233794271946,
+          -0.019760562106966972,
+          -0.005292718298733234,
+          0.0460701547563076,
+          -0.025077596306800842,
+          0.0008312931749969721,
+          -0.015659313648939133,
+          -0.029276108369231224,
+          0.017523517832159996,
+          0.021916555240750313,
+          0.000971615023445338,
+          -0.014005845412611961,
+          -0.03838639333844185,
+          0.027233589440584183,
+          0.015278367325663567,
+          -0.046556469053030014,
+          -0.0268121175467968,
+          0.0055723488330841064,
+          0.02869253046810627,
+          0.014848790131509304,
+          -0.010285543277859688,
+          0.006245083641260862,
+          0.02929231896996498,
+          -0.0056777168065309525,
+          -0.014905527234077454,
+          0.01545668300241232,
+          -0.02094392664730549,
+          -0.002857095329090953,
+          0.015278367325663567,
+          -6.775469955755398e-05,
+          0.008745548315346241,
+          -0.005000929813832045,
+          0.0182205680757761,
+          -0.0007907669642008841,
+          -0.006431503687053919,
+          -0.037737973034381866,
+          -0.025223489850759506,
+          -0.046102575957775116,
+          0.03439861536026001,
+          0.037089552730321884,
+          0.008737443014979362,
+          0.017491096630692482,
+          -0.01631583832204342,
+          -0.00244372827000916,
+          -0.033490829169750214,
+          0.01182553730905056,
+          0.02536938525736332,
+          0.005013087764382362,
+          0.004587562754750252,
+          0.0069948178716003895,
+          0.009726282209157944,
+          0.014500265009701252,
+          0.0004141268436796963,
+          -0.04347648099064827,
+          0.004790193866938353,
+          0.008003919385373592,
+          0.034982193261384964,
+          -0.031205154955387115,
+          -0.03125378489494324,
+          -0.011939010582864285,
+          0.010309858247637749,
+          0.036473557353019714,
+          -0.034528300166130066,
+          -0.0182205680757761,
+          -0.021835502237081528,
+          0.016048364341259003,
+          -0.006030294578522444,
+          -0.012441535480320454,
+          -0.0003432060475461185,
+          0.008429444395005703,
+          -0.012490167282521725,
+          0.01711825653910637,
+          -0.026115067303180695,
+          -8.656137651996687e-05,
+          -0.04493542015552521,
+          0.004437616094946861,
+          -0.06639808416366577,
+          -0.019760562106966972,
+          0.03543608635663986,
+          -0.024121178314089775,
+          0.0214302409440279,
+          -0.006366661749780178,
+          0.012247010134160519,
+          -0.05654211714863777,
+          0.002972594927996397,
+          -0.008510496467351913,
+          0.04182301089167595,
+          -0.0083078658208251,
+          -0.005134666338562965,
+          -0.034301355481147766,
+          0.002052650786936283,
+          -0.007704025600105524,
+          0.009985649026930332,
+          0.03146452084183693,
+          -0.014216582290828228,
+          -0.0016554942121729255,
+          0.012530692853033543,
+          0.00012430136848706752,
+          0.011955221183598042,
+          0.03073505125939846,
+          -0.017523517832159996,
+          -0.04227690398693085,
+          0.02071698009967804,
+          0.036797765642404556,
+          -0.021057400852441788,
+          -0.016080785542726517,
+          -0.02530454285442829,
+          0.0042836167849600315,
+          0.006447714287787676,
+          0.004769930616021156,
+          0.017685621976852417,
+          -0.0047739832662045956,
+          0.02065213769674301,
+          0.015059526078402996,
+          0.021900344640016556,
+          0.030054211616516113,
+          0.01636446826159954,
+          0.017636990174651146,
+          -0.017458675429224968,
+          -0.03783523663878441,
+          0.010966382920742035,
+          -0.03543608635663986,
+          0.0035926452837884426,
+          0.02747674658894539,
+          -0.03079989179968834,
+          -0.009645229205489159,
+          -0.006646292749792337,
+          4.45787918579299e-05,
+          0.013138585723936558,
+          0.0061478205025196075,
+          0.0366680808365345,
+          0.011031224392354488,
+          0.005341349635273218,
+          -0.0007411224069073796,
+          0.028887057676911354,
+          -0.015391840599477291,
+          0.035144299268722534,
+          -0.015853838995099068,
+          -0.01553773507475853,
+          0.006549029611051083,
+          -0.010520595125854015,
+          0.019258037209510803,
+          -0.008206550031900406,
+          -0.0043200901709496975,
+          -0.022662237286567688,
+          -0.008769864216446877,
+          0.0071204486303031445,
+          0.016542784869670868,
+          -0.019484983757138252,
+          0.02603401429951191,
+          -0.0005334257730282843,
+          -0.014865000732243061,
+          0.01950119435787201,
+          -0.03024873696267605,
+          -0.023294445127248764,
+          0.0031002522446215153,
+          -0.003789197187870741,
+          0.003740565851330757,
+          -0.005426454823464155,
+          -0.0024072546511888504,
+          -0.015772787854075432,
+          -0.002682832768186927,
+          -0.02120329439640045,
+          -0.014305739663541317,
+          -0.006678713485598564,
+          -0.005000929813832045,
+          -0.01509194727987051,
+          0.002642306499183178,
+          -0.023861810564994812,
+          0.007683762814849615,
+          0.0323074646294117,
+          -0.03922933712601662,
+          0.019857825711369514,
+          0.03627903014421463,
+          0.003473093034699559,
+          -0.018917618319392204,
+          -0.022581184282898903,
+          -0.010917751118540764,
+          -0.010042386129498482,
+          -0.05044697970151901,
+          0.001210719463415444,
+          0.0015308762667700648,
+          -0.0021843607537448406,
+          -0.022921603173017502,
+          0.0013393900590017438,
+          0.016583310440182686,
+          -0.012676587328314781,
+          0.012247010134160519,
+          -0.005045508500188589,
+          -0.014727211557328701,
+          0.014832579530775547,
+          0.027768533676862717,
+          -0.02679590694606304,
+          -0.0029462529346346855,
+          -0.0036088558845221996,
+          0.0033839354291558266,
+          0.0045835101045668125,
+          -0.008931968361139297,
+          -0.007704025600105524,
+          0.033750198781490326,
+          0.001635231077671051,
+          -0.034917350858449936,
+          -0.012911639176309109,
+          -0.002362675964832306,
+          0.027444325387477875,
+          -0.024769596755504608,
+          -0.004899614490568638,
+          0.03180494159460068,
+          0.026293382048606873,
+          -0.014881211332976818,
+          -0.03130241855978966,
+          -0.028060322627425194,
+          0.02021445520222187,
+          0.00838891789317131,
+          0.025888120755553246,
+          0.00861586444079876,
+          0.0009341283002868295,
+          -0.0538511797785759,
+          0.01899867132306099,
+          -0.018690671771764755,
+          -0.007529763039201498,
+          -0.003687881864607334,
+          0.02055487595498562,
+          -0.07592984288930893,
+          -0.03456072136759758,
+          0.03462556377053261,
+          0.033652935177087784,
+          -0.011104172095656395,
+          -0.00919133611023426,
+          0.012263220734894276,
+          -0.002164097735658288,
+          0.034982193261384964,
+          0.006196451839059591,
+          0.023521391674876213,
+          0.009596598334610462,
+          -0.0013880215119570494,
+          0.011841747909784317,
+          -0.01700478233397007,
+          0.0011874169576913118,
+          -0.02775232493877411,
+          0.016032153740525246,
+          -0.005941137205809355,
+          -0.004397090058773756,
+          -0.02215971238911152,
+          0.051646556705236435,
+          -0.014297634363174438,
+          0.034366194158792496,
+          -0.004624036606401205,
+          0.02083045430481434,
+          0.05394844338297844,
+          0.016972361132502556,
+          -0.03683018684387207,
+          0.0030739102512598038,
+          -0.018301621079444885,
+          -0.041174594312906265,
+          0.012368588708341122,
+          -0.022192133590579033,
+          -0.02598538249731064,
+          0.029227476567029953,
+          0.0009467927156947553,
+          0.006654397584497929,
+          -0.0027719903737306595,
+          0.0004359096637926996,
+          -0.008567233569920063,
+          -0.013673530891537666,
+          0.017523517832159996,
+          -0.006642240099608898,
+          0.04192027449607849,
+          -0.009977543726563454,
+          -0.008364601992070675,
+          0.002861147979274392,
+          0.013357426971197128,
+          0.045000262558460236,
+          0.019582247361540794,
+          -0.016923731192946434,
+          0.03146452084183693,
+          0.022062448784708977,
+          0.03715439513325691,
+          -0.018804144114255905,
+          -0.05135476961731911,
+          0.006358556915074587,
+          -0.0009148783865384758,
+          0.016080785542726517,
+          0.0018115199636667967,
+          0.022564973682165146,
+          0.001599770737811923,
+          0.01789635792374611,
+          -0.0019016907317563891,
+          0.04386553168296814,
+          -0.009167020209133625,
+          -0.009653334505856037,
+          0.027930639684200287,
+          -0.0005764848319813609,
+          0.0336853563785553,
+          0.02039276994764805,
+          -0.03511187806725502,
+          -0.034755248576402664,
+          0.01838267222046852,
+          0.026439275592565536,
+          -0.02237044833600521,
+          0.009102178737521172,
+          0.003746644826605916,
+          0.023991495370864868,
+          -0.014654264785349369,
+          0.012530692853033543,
+          0.0036919345147907734,
+          -0.013195321895182133,
+          -0.024964123964309692,
+          0.015043315477669239,
+          0.004052617587149143,
+          -0.0059006111696362495,
+          0.035533349961042404,
+          -0.03462556377053261,
+          -0.03362051397562027,
+          -0.00743655301630497,
+          -0.007071817293763161,
+          -0.0012198378099128604,
+          0.0371219739317894,
+          -0.005462928209453821,
+          0.01877172477543354,
+          -0.03118894435465336,
+          0.040850382298231125,
+          0.00765539426356554,
+          0.004688878543674946,
+          -0.0071731326170265675,
+          0.0016960203647613525,
+          -0.009977543726563454,
+          0.014257107861340046,
+          0.007736446801573038,
+          -0.007716183550655842,
+          0.0018084804760292172,
+          -0.011006908491253853,
+          -0.007290658541023731,
+          0.01642931066453457,
+          -0.027541587129235268,
+          -0.008518601767718792,
+          0.02259739488363266,
+          0.03465798497200012,
+          0.0007927932892926037,
+          0.001413350342772901,
+          -0.00983165018260479,
+          0.006362609565258026,
+          -0.02232181653380394,
+          0.04299016669392586,
+          -0.02138160914182663,
+          0.014751527458429337,
+          0.01590247079730034,
+          -0.042179640382528305,
+          -0.021446451544761658,
+          0.026050224900245667,
+          0.020198244601488113,
+          -0.018415093421936035,
+          -0.011225750669836998,
+          -0.014962263405323029,
+          0.016810256987810135,
+          -0.03068641945719719,
+          -0.04292532429099083,
+          0.010982593521475792,
+          -0.0015845734160393476,
+          -0.0047050886787474155,
+          -0.01595110259950161,
+          0.018690671771764755,
+          0.01085290964692831,
+          -0.054467178881168365,
+          -0.00502524571493268,
+          0.010009964928030968,
+          0.006107294466346502,
+          0.018706882372498512,
+          0.004672667942941189,
+          0.03362051397562027,
+          0.0006463924655690789,
+          0.01838267222046852,
+          -0.007898551411926746,
+          0.01307374332100153,
+          -0.010950172320008278,
+          -0.019371511414647102,
+          0.03279377892613411,
+          0.011420276015996933,
+          -0.03266409784555435,
+          -0.0044254581443965435,
+          0.0021742291282862425,
+          -0.016631942242383957,
+          0.002182334428653121,
+          -0.015878155827522278,
+          -0.006184294354170561,
+          0.028984319418668747,
+          0.00685297604650259,
+          0.025239700451493263,
+          0.029616527259349823,
+          -0.0035865663085132837,
+          -0.01581331342458725,
+          0.012133536860346794,
+          0.009718176908791065,
+          -0.01160669606178999,
+          0.022013816982507706,
+          -0.031091680750250816,
+          0.035371243953704834,
+          -0.06166462600231171,
+          0.030929576605558395,
+          0.027671271935105324,
+          -0.016794046387076378,
+          0.003426488023251295,
+          -0.021867923438549042,
+          0.01766941137611866,
+          -0.01883656531572342,
+          0.029827265068888664,
+          -0.020360350608825684,
+          0.02913021482527256,
+          -0.021170873194932938,
+          0.021576134487986565,
+          0.002658517099916935,
+          -0.018674461171030998,
+          0.008060656487941742,
+          0.0008348391856998205,
+          -0.010998803190886974,
+          -0.013600584119558334,
+          -0.029454423114657402,
+          0.012425324879586697,
+          0.058292847126722336,
+          -0.033490829169750214,
+          -0.0017031125025823712,
+          0.018917618319392204,
+          -0.02259739488363266,
+          -0.03152936324477196,
+          0.0100829117000103,
+          0.004534878768026829,
+          0.030864734202623367,
+          0.014240897260606289,
+          -0.04075312241911888,
+          -0.04172574728727341,
+          0.008324076421558857,
+          0.010350384749472141,
+          -0.004243090283125639,
+          0.046264681965112686,
+          0.01040712185204029,
+          0.014840684831142426,
+          0.01041522715240717,
+          0.0040566702373325825,
+          0.001815572613850236,
+          0.007890446111559868,
+          -0.007570289541035891,
+          -0.012587429955601692,
+          -0.001791256945580244,
+          -0.028498005121946335,
+          -0.005240034312009811,
+          0.017361413687467575,
+          -0.011598590761423111,
+          -0.010204490274190903,
+          -0.04503268375992775,
+          -0.01572415605187416,
+          0.017069624736905098,
+          -0.003262357087805867,
+          0.04668615385890007,
+          0.013171006925404072,
+          -0.004676720593124628,
+          -0.01933909021317959,
+          -0.02076561190187931,
+          0.04619983956217766,
+          -0.017636990174651146,
+          0.04000743851065636,
+          -0.0015734287444502115,
+          -0.013535741716623306,
+          -0.025110017508268356,
+          0.033879879862070084,
+          -0.03728407993912697,
+          -0.013957214541733265,
+          -0.008117392659187317,
+          -0.0035359086468815804,
+          -0.005458875559270382,
+          0.0018409014446660876,
+          -0.0008019116939976811,
+          -0.030167683959007263,
+          0.024396756663918495,
+          -0.025336964055895805,
+          0.031983256340026855,
+          0.02055487595498562,
+          0.010123438201844692,
+          0.00048200818127952516,
+          -0.03222641348838806,
+          -0.005969505291432142,
+          0.0005481165135279298,
+          0.008761758916079998,
+          -0.0011539828265085816,
+          0.00394724914804101,
+          -0.015934891998767853,
+          -0.01356816291809082,
+          -0.012287535704672337,
+          -0.04253627359867096,
+          -0.015480998903512955,
+          0.021089820191264153,
+          0.010374700650572777,
+          -0.03311799094080925,
+          -0.017636990174651146,
+          -0.03653839975595474,
+          0.006840818095952272,
+          -0.010625963099300861,
+          -0.008680706843733788,
+          0.012473956681787968,
+          0.001798348966985941,
+          0.015019000507891178,
+          -0.005479138810187578,
+          -0.023245813325047493,
+          0.026487907394766808,
+          -0.03219399228692055,
+          -0.029486844316124916,
+          -0.01043954212218523,
+          -0.003635197877883911,
+          -0.010042386129498482,
+          0.031059259548783302,
+          0.021251926198601723,
+          0.0007943130331113935,
+          -0.050479400902986526,
+          -0.03050810471177101,
+          -0.051873501390218735,
+          0.024396756663918495,
+          0.0009315954521298409,
+          -0.00938586238771677,
+          0.003252225462347269,
+          0.02039276994764805,
+          0.008178181946277618,
+          -0.01313048042356968,
+          -0.02603401429951191,
+          -0.024023916572332382,
+          -0.013884266838431358,
+          -0.005195455625653267,
+          -0.029065372422337532,
+          -0.033263884484767914,
+          -0.037089552730321884,
+          -0.01334932167083025,
+          0.005916821304708719,
+          0.011776906438171864,
+          -0.003000963246449828,
+          0.0008049511234275997,
+          -0.006759766023606062,
+          -0.02459128201007843,
+          0.004125564359128475,
+          0.015205420553684235,
+          -0.01998750865459442,
+          0.02708769403398037,
+          0.009629018604755402,
+          0.0141517398878932,
+          -0.029843473806977272,
+          0.013859951868653297,
+          0.009118389338254929,
+          0.014338160865008831,
+          -0.004976614378392696,
+          -0.003906723111867905,
+          0.02060350775718689,
+          -0.01972814090549946,
+          0.026017803698778152,
+          -0.006350451614707708,
+          -0.0024700702633708715,
+          -0.010066702030599117,
+          0.020360350608825684,
+          0.003473093034699559,
+          0.012725219130516052,
+          0.022224554792046547,
+          -2.4996417778311297e-05,
+          0.037089552730321884,
+          -0.005090087652206421,
+          -0.01326016429811716,
+          0.00575876934453845,
+          -0.02121950499713421,
+          -0.005702032707631588,
+          0.005993821192532778,
+          -0.029000530019402504,
+          0.0012887322809547186,
+          0.014792053960263729,
+          0.009880281053483486,
+          -0.0318535715341568,
+          0.023634864017367363,
+          0.02603401429951191,
+          0.006848923396319151,
+          0.010399016551673412,
+          -0.033815041184425354,
+          0.0006266359705477953,
+          0.011274381540715694,
+          -0.004778035916388035,
+          0.036700502038002014,
+          0.0201658234000206,
+          -0.03614934906363487,
+          0.010998803190886974,
+          -0.01043954212218523,
+          0.00396548630669713,
+          -0.004765877965837717,
+          0.029616527259349823,
+          -0.022338027134537697,
+          -0.011006908491253853,
+          -0.029324740171432495,
+          0.010180175304412842,
+          -0.021948976442217827,
+          -0.015497208572924137,
+          -0.03524155914783478,
+          0.03579271584749222,
+          0.005811453331261873,
+          -0.013551952317357063,
+          0.0029989369213581085,
+          -0.02039276994764805,
+          -0.0071690804325044155,
+          -0.02951926551759243,
+          -0.012660376727581024,
+          -0.012230799533426762,
+          -0.0007537868223153055,
+          0.016810256987810135,
+          0.02491549216210842,
+          0.006439608987420797,
+          0.029389580711722374,
+          0.03728407993912697,
+          0.00218841340392828,
+          -0.014167950488626957,
+          0.06169704720377922,
+          -0.0007922867080196738,
+          -0.024607492610812187,
+          0.04075312241911888,
+          -0.024964123964309692,
+          0.0016544810496270657,
+          0.01625099591910839,
+          0.014743422158062458,
+          -0.03211294114589691,
+          0.016923731192946434,
+          -0.006739502772688866,
+          -0.033815041184425354,
+          0.003924959804862738,
+          -0.005171139724552631,
+          -0.03195083513855934,
+          -0.04185543209314346,
+          -0.010115332901477814,
+          0.004879351239651442,
+          -0.02160855568945408,
+          0.017020992934703827,
+          0.020473822951316833,
+          -0.002723358804360032,
+          -0.014500265009701252,
+          0.02275949902832508,
+          0.001914861728437245,
+          0.006970501970499754,
+          -0.006771923508495092,
+          0.007833709008991718,
+          -0.0027253851294517517,
+          -0.0010035294108092785,
+          -0.024834439158439636,
+          -0.015140579082071781,
+          -0.041239432990550995,
+          0.0159267857670784,
+          -0.023780759423971176,
+          -0.046880677342414856,
+          0.0056412434205412865,
+          -0.03744618594646454,
+          0.03355567157268524,
+          0.017474886029958725,
+          0.0005683795898221433,
+          -0.0033434093929827213,
+          -0.007716183550655842,
+          -0.021689608693122864,
+          0.010917751118540764,
+          0.022856762632727623,
+          0.004316037520766258,
+          -0.0059897685423493385,
+          0.014192266389727592,
+          -0.022305605933070183,
+          0.000483021343825385,
+          -0.060335367918014526,
+          -0.02381318062543869,
+          0.025628753006458282,
+          0.007983656600117683,
+          0.0005856032366864383,
+          -0.014824474230408669,
+          0.006808397360146046,
+          -0.03266409784555435,
+          -0.019420143216848373,
+          0.025126228109002113,
+          0.019744351506233215,
+          0.014086898416280746,
+          -0.004960403777658939,
+          0.0007396026630885899,
+          0.0182205680757761,
+          -0.008850916288793087,
+          0.008980600163340569,
+          0.0030556735582649708,
+          0.025045175105333328,
+          0.00021225583623163402,
+          0.0077931834384799,
+          0.01210111565887928,
+          0.028822215273976326,
+          0.0034710667096078396,
+          0.010544910095632076,
+          0.006038399878889322,
+          0.0009300757083110511,
+          -0.0134060587733984,
+          -0.013276374898850918,
+          -0.008153866045176983,
+          0.024137388914823532,
+          0.018852775916457176,
+          0.00955607183277607,
+          -0.009580387733876705,
+          -0.009880281053483486,
+          0.008956284262239933,
+          0.009004916064441204,
+          -0.001745664980262518,
+          0.01223890483379364,
+          0.012117326259613037,
+          -0.029778633266687393,
+          -0.03355567157268524,
+          -0.0013799162115901709,
+          -0.004984719213098288,
+          0.005795242730528116,
+          0.00802012998610735,
+          0.021948976442217827,
+          0.0013181137619540095,
+          -0.013479005545377731,
+          -0.004980667028576136,
+          0.02530454285442829,
+          0.005742558743804693,
+          -0.01317911222577095,
+          -0.015716049820184708,
+          -0.009766807779669762,
+          0.00772834150120616,
+          0.011193329468369484,
+          -0.003473093034699559,
+          0.0033596197608858347,
+          0.012741428799927235,
+          0.08163592219352722,
+          -0.007533815689384937,
+          0.0036818028893321753,
+          -0.03854849562048912,
+          -0.01284679677337408,
+          -0.008640180341899395,
+          0.019484983757138252,
+          -0.0019867955707013607,
+          0.032323677092790604,
+          0.015051420778036118,
+          0.01849614642560482,
+          0.0059208739548921585,
+          0.015326999127864838,
+          0.005365665536373854,
+          -0.02929231896996498,
+          0.02227318473160267,
+          -0.0005982676520943642,
+          0.011817432008683681,
+          0.0019857825245708227,
+          -0.011979537084698677,
+          -0.011006908491253853,
+          0.02032792940735817,
+          0.003671671263873577,
+          0.02397528477013111,
+          -0.02159234508872032,
+          0.019744351506233215,
+          0.01849614642560482,
+          0.011979537084698677,
+          -0.010301753878593445,
+          -0.005004982464015484,
+          0.0014913631603121758,
+          0.00517924502491951,
+          -0.017037203535437584,
+          -0.0016139548970386386,
+          -0.019582247361540794,
+          0.06652776896953583,
+          -0.030556734651327133,
+          0.0007244053413160145,
+          0.0031346995383501053,
+          -0.018577197566628456,
+          0.004713193979114294,
+          0.001658533699810505,
+          0.01659141480922699,
+          0.0057263486087322235,
+          -0.006581450812518597,
+          0.005531822796911001,
+          0.024688545614480972,
+          -0.00560882268473506,
+          0.01762077957391739,
+          0.005523717496544123,
+          -0.012263220734894276,
+          -0.014638054184615612,
+          0.012076799757778645,
+          0.011939010582864285,
+          0.009142705239355564,
+          0.025904331356287003,
+          0.008012024685740471,
+          -0.009880281053483486,
+          -0.020895294845104218,
+          -0.011582380160689354,
+          0.006151873152703047,
+          0.005876295268535614,
+          0.03815944492816925,
+          0.030216315761208534,
+          -0.0006884383619762957,
+          -0.013592478819191456,
+          0.016259100288152695,
+          -0.005365665536373854,
+          0.024169810116291046,
+          -0.0029259896837174892,
+          -0.013608689419925213,
+          0.008980600163340569,
+          0.009564177133142948,
+          -0.008178181946277618,
+          -0.02747674658894539,
+          -0.011582380160689354,
+          -0.01013964880257845,
+          -0.01581331342458725,
+          0.010099122300744057,
+          -0.027071483433246613,
+          -0.002474122913554311,
+          0.0011884300038218498,
+          0.027039064094424248,
+          0.03738134354352951,
+          -0.011768801137804985,
+          -0.00539808627218008,
+          -0.0008814443135634065,
+          -0.025223489850759506,
+          -0.005000929813832045,
+          -0.020749401301145554,
+          -0.024964123964309692,
+          -0.016810256987810135,
+          -0.016105102375149727,
+          -0.005527770146727562,
+          -0.0029178846161812544,
+          0.004474089480936527,
+          0.0015045341569930315,
+          0.058130744844675064,
+          -0.01102311909198761,
+          -0.012084905058145523,
+          -0.02076561190187931,
+          0.006942133419215679,
+          0.019128354266285896,
+          -0.011387854814529419,
+          0.0027274114545434713,
+          -0.01393289864063263,
+          0.004010064993053675,
+          -0.014881211332976818,
+          -0.014362475834786892,
+          -0.023002656176686287,
+          0.002506543882191181,
+          -0.006455819588154554,
+          0.0028084637597203255,
+          -0.00627345172688365,
+          0.009726282209157944,
+          0.019857825711369514,
+          -3.7233421608107165e-05,
+          0.001175259007140994,
+          -0.008810389786958694,
+          0.015043315477669239,
+          -0.030491894111037254,
+          0.02392665296792984,
+          0.019144564867019653,
+          0.0033373304177075624,
+          -0.0053778234869241714,
+          0.02752537652850151,
+          0.00364127685315907,
+          -0.011720169335603714,
+          0.04568110406398773,
+          0.010399016551673412,
+          0.026179907843470573,
+          -0.009872175753116608,
+          0.005774979945272207,
+          -0.0181070938706398,
+          0.009231862612068653,
+          -0.006484187673777342,
+          0.0072420272044837475,
+          -0.006131610367447138,
+          0.02780095487833023,
+          -0.008777969516813755,
+          0.0013464820804074407,
+          0.002717280061915517,
+          0.04098006710410118,
+          -0.0019452563719823956,
+          0.015853838995099068,
+          -0.026617590337991714,
+          -0.00035536388168111444,
+          0.02126813679933548,
+          0.014362475834786892,
+          0.017183097079396248,
+          -0.000494925887323916,
+          -0.01950119435787201,
+          -0.0045997207053005695,
+          -0.014321950264275074,
+          0.0009179178159683943,
+          -0.030005579814314842,
+          0.002401175908744335,
+          -0.00473345723003149,
+          0.013187216594815254,
+          0.013876162469387054,
+          -0.011695853434503078,
+          0.02044140174984932,
+          -0.017701832577586174,
+          -0.02376454882323742,
+          0.008964389562606812,
+          0.0018135462887585163,
+          0.01700478233397007,
+          0.017020992934703827,
+          -0.022030027583241463,
+          -0.006982659921050072,
+          -0.01883656531572342,
+          -0.01849614642560482,
+          0.0032562781125307083,
+          0.034041985869407654,
+          -0.007096133194863796,
+          -0.010828593745827675,
+          0.018188146874308586,
+          -0.01750730723142624,
+          0.01732899248600006,
+          -0.007250132504850626,
+          0.01954982616007328,
+          -0.001856098766438663,
+          -0.006609818898141384,
+          0.005240034312009811,
+          -0.02358623407781124,
+          0.0012897454435005784,
+          0.00999375432729721,
+          0.001181337982416153,
+          -0.03229125589132309,
+          0.012036273255944252,
+          -0.0020678481087088585,
+          0.001809493638575077,
+          -0.006484187673777342,
+          0.014767738059163094,
+          0.006788134109228849,
+          0.00869691651314497,
+          -0.002474122913554311,
+          0.010074807330965996,
+          0.003671671263873577,
+          -0.015083841979503632,
+          -0.003112410195171833,
+          0.018593408167362213,
+          0.008599653840065002,
+          -0.011614801362156868,
+          0.00575876934453845,
+          -0.014946052804589272,
+          -0.016080785542726517,
+          0.019258037209510803,
+          0.019760562106966972,
+          -0.031432099640369415,
+          0.022791920229792595,
+          -0.024461599066853523,
+          -0.0035521192476153374,
+          0.017977410927414894,
+          -0.0020972294732928276,
+          -0.0008490233449265361,
+          -0.004474089480936527,
+          -0.027557797729969025,
+          -0.01428952906280756,
+          0.005924926605075598,
+          -0.011525643989443779,
+          0.00452677346765995,
+          0.025288332253694534,
+          0.015691734850406647,
+          -0.0056250328198075294,
+          -0.0016858888557180762,
+          0.030832313001155853,
+          -0.003071883926168084,
+          0.0054831914603710175,
+          0.005126561038196087,
+          -0.028870847076177597,
+          0.02995694801211357,
+          -0.024056337773799896,
+          0.007550026290118694,
+          -0.000363722414476797,
+          -0.00247817556373775,
+          -0.006358556915074587,
+          0.014411107636988163,
+          -0.0005481165135279298,
+          0.0016200337558984756,
+          -0.014475949108600616,
+          0.006269399076700211,
+          -0.019355300813913345,
+          0.030864734202623367,
+          -0.0016088890843093395,
+          -0.00017768194084055722,
+          0.015197315253317356,
+          -0.015205420553684235,
+          -0.017863936722278595,
+          0.01013964880257845,
+          0.008453760296106339,
+          -0.0230350773781538,
+          0.004778035916388035,
+          -0.01933909021317959,
+          -0.03871060162782669,
+          -0.026115067303180695,
+          0.003706118557602167,
+          0.008915758691728115,
+          0.0165022574365139,
+          -0.009426387958228588,
+          -0.0061113471165299416,
+          0.02692558988928795,
+          0.02358623407781124,
+          0.03378261998295784,
+          -0.013949109241366386,
+          0.009094073437154293,
+          0.0027963060420006514,
+          0.01317911222577095,
+          0.001355600543320179,
+          -0.004384932108223438,
+          0.0033839354291558266,
+          -0.004247142933309078,
+          -0.012587429955601692,
+          -0.0018044279422610998,
+          0.027606429532170296,
+          -0.006650345399975777,
+          -0.024380546063184738,
+          -0.03151315450668335,
+          0.003161041531711817,
+          -0.005231929011642933,
+          0.0040222229436039925,
+          0.03971565142273903,
+          -0.024510230869054794,
+          -0.02587191015481949,
+          0.02026308700442314,
+          -0.003205620450899005,
+          0.004567299969494343,
+          -0.0019847694784402847,
+          0.01027743797749281,
+          -0.014775843359529972,
+          -0.028643900528550148,
+          -0.022192133590579033,
+          -0.0027193063870072365,
+          0.0037425921764224768,
+          -0.007128553930670023,
+          -0.016972361132502556,
+          -0.007886393927037716,
+          0.007622973527759314,
+          -0.016510363668203354,
+          0.013762688264250755,
+          -0.02237044833600521,
+          0.010147754102945328,
+          -0.0012299694353714585,
+          0.030653998255729675,
+          -0.015513419173657894,
+          -0.01608889177441597,
+          -0.023894231766462326,
+          0.007254185155034065,
+          0.02099255844950676,
+          -0.0009913715766742826,
+          0.0012026141630485654,
+          0.006780028808861971,
+          0.005292718298733234,
+          0.007197448518127203,
+          -0.0069948178716003895,
+          0.019063511863350868,
+          -0.034820087254047394,
+          -0.0024356229696422815,
+          0.0019047302193939686,
+          0.039521124213933945,
+          -0.027168747037649155,
+          -0.003321120049804449,
+          -0.007817499339580536,
+          0.010228806175291538,
+          0.013519532047212124,
+          0.013162901625037193,
+          -0.0027679377235472202,
+          -0.01677783578634262,
+          -0.022451501339673996,
+          0.0005597677663899958,
+          -0.038516074419021606,
+          0.0026544644497334957,
+          0.015197315253317356,
+          -0.0013991661835461855,
+          0.03751102834939957,
+          0.0469779409468174,
+          -0.03738134354352951,
+          -0.006269399076700211,
+          0.009799228981137276,
+          0.03410682827234268,
+          -0.020911505445837975,
+          0.020408980548381805,
+          -0.011063645593822002,
+          0.01620236411690712,
+          0.0035602243151515722,
+          -0.009085968136787415,
+          -0.02786579728126526,
+          -0.014905527234077454,
+          0.021089820191264153,
+          0.008145760744810104,
+          -0.021478872746229172,
+          0.009734386578202248,
+          0.011558065190911293,
+          -0.008364601992070675,
+          -0.025061385706067085,
+          0.02598538249731064,
+          0.01883656531572342,
+          0.01157427579164505,
+          -0.018204357475042343,
+          -0.011768801137804985,
+          -0.02674727514386177,
+          0.008996810764074326,
+          -0.020846663042902946,
+          0.01723172888159752,
+          -0.023861810564994812,
+          0.0008839771617203951,
+          -0.01844751462340355,
+          -0.007225817069411278,
+          0.006532819475978613,
+          -0.04370342567563057,
+          -0.006476082839071751,
+          -0.005924926605075598,
+          0.012960270047187805,
+          0.028951898217201233,
+          0.021414030343294144,
+          -0.014394897036254406,
+          -0.02890326827764511,
+          -0.003379883011803031,
+          -0.0073960269801318645,
+          0.013446584343910217,
+          -0.0059208739548921585,
+          -0.008397023193538189,
+          -0.0022654130589216948,
+          0.008332181721925735,
+          -0.010026175528764725,
+          0.007622973527759314,
+          0.006082978565245867,
+          0.001210719463415444,
+          0.018593408167362213,
+          -0.0008186287013813853,
+          -0.013316900469362736,
+          -0.0033900144044309855,
+          -0.008542917668819427,
+          0.02083045430481434,
+          0.03355567157268524,
+          0.003979670349508524,
+          -0.007383869029581547,
+          -0.00466456264257431,
+          0.004648352041840553,
+          0.020684558898210526,
+          -0.015416156500577927,
+          -0.0041620382107794285,
+          -0.003440672066062689,
+          0.01307374332100153,
+          -0.011736379936337471,
+          0.027039064094424248,
+          -0.013235848397016525,
+          -0.023521391674876213,
+          -0.022030027583241463,
+          -0.0559261217713356,
+          0.006605766247957945,
+          -0.028400743380188942,
+          -0.03096199780702591,
+          -0.01581331342458725,
+          -0.012287535704672337,
+          -0.01760457083582878,
+          0.011849853210151196,
+          -0.005697980057448149,
+          -0.0032603307627141476,
+          0.005965452641248703,
+          0.013219637796282768,
+          -0.016380678862333298,
+          0.0008576351683586836,
+          0.00511035043746233,
+          0.0010156872449442744,
+          -0.01922561787068844,
+          0.016599521040916443,
+          0.009361546486616135,
+          -0.01223890483379364,
+          -0.013892372138798237,
+          0.0021661240607500076,
+          0.013187216594815254,
+          -0.004591615404933691,
+          0.006301820278167725,
+          0.015562050975859165,
+          0.010180175304412842,
+          -0.0214302409440279,
+          -0.0075540789403021336,
+          0.005065771751105785,
+          -0.007059659343212843,
+          0.01487310603260994,
+          -0.015667418017983437,
+          -0.006885396782308817,
+          0.0026321751065552235,
+          0.012076799757778645,
+          -0.018463725224137306,
+          -0.007343342527747154,
+          -0.00350348767824471,
+          3.1661072625865927e-06,
+          -0.018739303573966026,
+          0.0003079989110119641,
+          0.012919744476675987,
+          0.0008267339435406029,
+          0.008129550144076347,
+          -0.01317911222577095,
+          0.003142804838716984,
+          0.028384532779455185,
+          0.019760562106966972,
+          -0.037024714052677155,
+          0.0013272322248667479,
+          0.014362475834786892,
+          -0.009150810539722443,
+          -0.01989024691283703,
+          0.019063511863350868,
+          0.02298644557595253,
+          0.002540991175919771,
+          0.0017254018457606435,
+          -0.0023464655969291925,
+          -0.003270462155342102,
+          -0.01865825057029724,
+          0.011566170491278172,
+          0.01116090826690197,
+          -0.005843874532729387,
+          -0.0012887322809547186,
+          0.006366661749780178,
+          -0.01642931066453457,
+          -0.006326135713607073,
+          0.007789130788296461,
+          0.014856895431876183,
+          0.007489237003028393,
+          0.009499335661530495,
+          0.010561120696365833,
+          0.004575405269861221,
+          0.0015967312501743436,
+          0.024104967713356018,
+          0.07385490089654922,
+          0.0013819425366818905,
+          0.008380812592804432,
+          0.018852775916457176,
+          -0.006504450924694538,
+          -0.013187216594815254,
+          0.005860084667801857,
+          -0.008016076870262623,
+          -3.7233421608107165e-05,
+          0.005240034312009811,
+          -0.04072070121765137,
+          -0.02165718749165535,
+          -0.010877225548028946,
+          -0.013381742872297764,
+          0.004591615404933691,
+          0.018042253330349922,
+          -0.01799362152814865,
+          0.01590247079730034,
+          0.03793250024318695,
+          0.015148683451116085,
+          -0.009223757311701775,
+          0.01437058113515377,
+          0.0071893432177603245,
+          -0.011087961494922638,
+          -0.006646292749792337,
+          -0.01227132510393858,
+          0.008255181834101677,
+          0.015027105808258057,
+          0.0005825637490488589,
+          0.02741190418601036,
+          -0.0026200171560049057,
+          -0.00540213892236352,
+          0.003217778168618679,
+          -0.02491549216210842,
+          -0.015942996367812157,
+          0.004798299167305231,
+          0.008198444731533527,
+          -0.015083841979503632,
+          -0.013859951868653297,
+          -0.006881344132125378,
+          0.011087961494922638,
+          -0.0017872042953968048,
+          -0.006553082261234522,
+          -0.0013981530210003257,
+          -0.002670674817636609,
+          -0.004782088566571474,
+          -0.024023916572332382,
+          0.0024579125456511974,
+          0.013381742872297764,
+          0.006046505179256201,
+          0.014727211557328701,
+          0.003460935316979885,
+          0.007108291145414114,
+          -0.006739502772688866,
+          -0.012935955077409744,
+          -0.00822681374847889,
+          0.012563114054501057,
+          0.021365398541092873,
+          0.013414164073765278,
+          0.010553015395998955,
+          -0.005568296182900667,
+          0.007371711079031229,
+          -0.008113340474665165,
+          -0.015148683451116085,
+          -0.00663008214905858,
+          -0.00176896748598665,
+          0.0019006775692105293,
+          -0.004453826695680618,
+          0.002962463302537799,
+          -0.002729437779635191,
+          0.006982659921050072,
+          0.004842877853661776,
+          0.008559128269553185,
+          0.0007877275347709656,
+          -0.002368754940107465,
+          -0.012149747461080551,
+          -0.014719106256961823,
+          -0.019193196669220924,
+          0.017815306782722473,
+          0.010034280829131603,
+          0.013633005321025848,
+          0.018917618319392204,
+          -0.0024822282139211893,
+          -0.012749534100294113,
+          -0.0197119303047657,
+          0.00038297235732898116,
+          0.003321120049804449,
+          0.016056470572948456,
+          0.026228539645671844,
+          -0.01789635792374611,
+          0.00382972345687449,
+          -0.006168083753436804,
+          0.01179311703890562,
+          -0.022678447887301445,
+          -0.010293648578226566,
+          0.015878155827522278,
+          -0.030281158164143562,
+          -5.6609998864587396e-05,
+          -0.010723225772380829,
+          -0.00444572139531374,
+          0.018593408167362213,
+          0.023602444678544998,
+          0.017247939482331276,
+          0.012976480647921562,
+          -0.010471963323652744,
+          -0.005009035114198923,
+          0.01920940726995468,
+          0.008559128269553185,
+          -0.005795242730528116,
+          0.0002556948165874928,
+          -0.007732394151389599,
+          -0.02104119025170803,
+          0.005146824289113283,
+          0.00969386100769043,
+          -0.024737177416682243,
+          0.005122508388012648,
+          -0.017701832577586174,
+          -0.0039350916631519794,
+          -0.018431304022669792,
+          -0.00305972620844841,
+          0.00611945241689682,
+          0.00244372827000916,
+          0.0024457545951008797,
+          0.0004257781256455928,
+          -0.003122541820630431,
+          -0.0005324126104824245,
+          0.01284679677337408,
+          -0.007489237003028393,
+          -0.01972814090549946,
+          -0.027379482984542847,
+          -0.010844804346561432,
+          -0.0018165857763960958,
+          0.002555175218731165,
+          -0.007335237693041563,
+          -0.02254876308143139,
+          0.0029644896276295185,
+          -0.009896491654217243,
+          -0.000849529926199466,
+          -0.001954374834895134,
+          0.02630959264934063,
+          0.002792253391817212,
+          -0.015245947055518627,
+          -0.034301355481147766,
+          -0.03123757429420948,
+          0.011622906662523746,
+          -0.0008160958532243967,
+          0.004672667942941189,
+          -0.011631011962890625,
+          0.001885480247437954,
+          -0.016169942915439606,
+          0.01464615948498249,
+          -0.022532552480697632,
+          -0.008012024685740471,
+          0.009337230585515499,
+          -0.0003305416030343622,
+          0.001216798322275281,
+          0.000767970981542021,
+          -0.0074325003661215305,
+          0.003718276508152485,
+          -0.01728036068379879,
+          -0.009961334057152271,
+          -0.02087908424437046,
+          -0.00027431154740042984,
+          0.004352510906755924,
+          -0.013422268442809582,
+          0.019031090661883354,
+          0.01987403631210327,
+          0.0038884864188730717,
+          -0.012109220959246159,
+          0.017863936722278595,
+          0.01188227441161871,
+          -0.009547966532409191,
+          -0.014613738283514977,
+          -0.021786870434880257,
+          0.011266276240348816,
+          0.026325803250074387,
+          0.018415093421936035,
+          0.0075540789403021336,
+          0.01588626019656658,
+          0.012287535704672337,
+          0.016007838770747185,
+          0.001041522715240717,
+          0.0017831516452133656,
+          0.0009766807779669762,
+          0.011979537084698677,
+          -0.015910575166344643,
+          0.0035359086468815804,
+          0.0090535469353199,
+          0.00985596515238285,
+          -0.016072681173682213,
+          -0.01290353387594223,
+          0.024007705971598625,
+          -0.0010516542242839932,
+          -0.012028168886899948,
+          0.0028348059859126806,
+          0.00540213892236352,
+          -0.007177185267210007,
+          0.000767970981542021,
+          -0.005462928209453821,
+          -0.002014150843024254,
+          0.01590247079730034,
+          0.017766674980521202,
+          -0.009758702479302883,
+          -0.004401142708957195,
+          0.011995747685432434,
+          -0.0028773583471775055,
+          -0.03190220519900322,
+          0.006184294354170561,
+          0.028822215273976326,
+          0.006018136627972126,
+          -0.03790007904171944,
+          -0.0013454690342769027,
+          0.005219771061092615,
+          -0.032048098742961884,
+          -0.03449587896466255,
+          0.0021965187042951584,
+          -0.010771856643259525,
+          -0.015294577926397324,
+          -0.0035521192476153374,
+          0.0075905523262917995,
+          -0.028984319418668747,
+          0.009410177357494831,
+          0.014281423762440681,
+          -0.0006276491330936551,
+          0.022240763530135155,
+          -0.004944193176925182,
+          -0.012352378107607365,
+          -0.027266010642051697,
+          0.02237044833600521,
+          -0.013470900245010853,
+          -0.008461865596473217,
+          -0.0012380746193230152,
+          -0.009945123456418514,
+          0.005240034312009811,
+          0.003071883926168084,
+          0.02890326827764511,
+          0.018479935824871063,
+          -0.0029259896837174892,
+          -0.04201753810048103,
+          -0.022338027134537697,
+          0.003971565049141645,
+          -0.025045175105333328,
+          0.013389848172664642,
+          -0.006054610479623079,
+          -0.00321169919334352,
+          0.005969505291432142,
+          0.03164283558726311,
+          0.011274381540715694,
+          0.019355300813913345,
+          -3.3022501156665385e-05,
+          0.0042674061842262745,
+          -0.03384746238589287,
+          -0.02415359951555729,
+          0.00947501976042986,
+          -0.0024639912880957127,
+          0.032647885382175446,
+          0.0036270925775170326,
+          0.004619983956217766,
+          0.014475949108600616,
+          -0.006719239521771669,
+          -0.004054643679410219,
+          -0.018804144114255905,
+          0.011922799982130527,
+          -0.004761825315654278,
+          0.01932287961244583,
+          0.01625099591910839,
+          -0.014378686435520649,
+          -0.022354237735271454,
+          0.005223823711276054,
+          0.04986340552568436,
+          0.013803214766085148,
+          -0.0006757739465683699,
+          0.005839821882545948,
+          0.0021195188164711,
+          0.0021073610987514257,
+          0.004340353421866894,
+          0.004046538379043341,
+          0.015002789907157421,
+          0.026212329044938087,
+          -0.004745615180581808,
+          0.0036736975889652967,
+          0.00350348767824471,
+          -0.005098192486912012,
+          -0.0028307533357292414,
+          -0.013033217750489712,
+          0.004684825893491507,
+          -0.010925856418907642,
+          0.008125497959554195,
+          -0.02053866535425186,
+          -0.011995747685432434,
+          -0.011055540293455124,
+          -0.010609752498567104,
+          -0.010009964928030968,
+          -0.018787933513522148,
+          0.01927424781024456,
+          -0.029730001464486122,
+          0.002589622512459755,
+          -0.013227743096649647,
+          0.013689741492271423,
+          0.009620913304388523,
+          -0.0006119452300481498,
+          0.02353760227560997,
+          -0.023407917469739914,
+          -0.007935024797916412,
+          0.004465984180569649,
+          0.00266662216745317,
+          -0.0022654130589216948,
+          0.003124567912891507,
+          -0.013057533651590347,
+          -0.025612542405724525,
+          0.009547966532409191,
+          -0.00540213892236352,
+          0.0034305406734347343,
+          0.006747608073055744,
+          -0.0010405095526948571,
+          -0.005843874532729387,
+          -0.003625066252425313,
+          0.01826919987797737,
+          -0.020425191149115562,
+          0.04114217311143875,
+          -0.03018389455974102,
+          -0.00480235181748867,
+          0.020311718806624413,
+          -0.01723172888159752,
+          -0.0001227183238370344,
+          -0.0020648085046559572,
+          0.0073636057786643505,
+          0.0035014613531529903,
+          -0.0022188080474734306,
+          -0.014386791735887527,
+          0.03161041438579559,
+          3.1233648769557476e-05,
+          -0.016745414584875107,
+          0.00889144279062748,
+          -0.013308795168995857,
+          0.017734253779053688,
+          -0.028108954429626465,
+          -1.29652098621591e-05,
+          0.015221631154417992,
+          0.0016281390562653542,
+          0.009783018380403519,
+          -0.01121764536947012,
+          -0.010958277620375156,
+          -0.01260364055633545,
+          0.015797102823853493,
+          0.01933909021317959,
+          -0.011525643989443779,
+          -0.022840552031993866,
+          0.0040242490358650684,
+          0.01666436344385147,
+          -0.021867923438549042,
+          -0.0017132440116256475,
+          -0.004397090058773756,
+          -0.009531755931675434,
+          0.0003996893938165158,
+          -0.012587429955601692,
+          0.011801222339272499,
+          0.014467843808233738,
+          -0.0046686152927577496,
+          -0.03316662088036537,
+          0.008084971457719803,
+          0.010625963099300861,
+          -0.03169146925210953,
+          -0.013049428351223469,
+          0.012555008754134178,
+          -0.011898485012352467,
+          0.010180175304412842,
+          -0.006775976158678532,
+          0.00986407045274973,
+          -0.00928049348294735,
+          -0.001244153594598174,
+          -0.00031737060635350645,
+          -0.0007380829192698002,
+          -0.022856762632727623,
+          -0.008648285642266273,
+          -0.0009584440267644823,
+          -0.03079989179968834,
+          0.011695853434503078,
+          0.003937117755413055,
+          -0.009734386578202248,
+          -0.0006560174515470862,
+          -0.017085835337638855,
+          -0.004368721507489681,
+          0.0045105633325874805,
+          0.00685297604650259,
+          -0.011493222787976265,
+          -0.013187216594815254,
+          -0.002350518014281988,
+          -0.016007838770747185,
+          0.018236778676509857,
+          -0.0004701036086771637,
+          0.036635659635066986,
+          -0.0023829389829188585,
+          -0.0013900477206334472,
+          -0.0021418083924800158,
+          -0.00280441134236753,
+          0.005791190080344677,
+          0.03195083513855934,
+          -0.0013971398584544659,
+          0.006390977650880814,
+          0.0018054411048069596,
+          -0.003677750239148736,
+          0.006001926492899656,
+          0.01576468162238598,
+          0.0005633138353005052,
+          0.003975617699325085,
+          -0.009726282209157944,
+          -0.010893435217440128,
+          0.006942133419215679,
+          -0.002822648035362363,
+          0.02021445520222187,
+          -0.0014812316512688994,
+          -0.00932912528514862,
+          -0.009256178513169289,
+          0.0012887322809547186,
+          0.008153866045176983,
+          -0.005977610591799021,
+          -0.002030361210927367,
+          0.02243529073894024,
+          -0.005122508388012648,
+          0.014905527234077454,
+          -0.015424261800944805,
+          0.015805207192897797,
+          0.014394897036254406,
+          -0.013284480199217796,
+          0.00781344622373581,
+          0.018804144114255905,
+          -0.007667552214115858,
+          0.018706882372498512,
+          -0.016550889238715172,
+          0.007602710276842117,
+          -0.00983165018260479,
+          0.03884028643369675,
+          -0.009450703859329224,
+          -0.022013816982507706,
+          -0.0029766475781798363,
+          0.015383735299110413,
+          -0.02005235105752945,
+          0.010569225996732712,
+          0.013114269822835922,
+          -0.02536938525736332,
+          0.002944226609542966,
+          0.014751527458429337,
+          0.003997907042503357,
+          -0.004206616897135973,
+          0.006265346426516771,
+          0.005406191572546959,
+          0.009499335661530495,
+          0.005811453331261873,
+          0.020360350608825684,
+          -0.016048364341259003,
+          0.01972814090549946,
+          -0.012174062430858612,
+          -0.0032035941258072853,
+          -0.022889183834195137,
+          -0.0030131209641695023,
+          -0.009572282433509827,
+          0.011250065639615059,
+          -0.012230799533426762,
+          0.006828660145401955,
+          -0.010042386129498482,
+          -0.0001888266415335238,
+          0.030702630057930946,
+          0.018706882372498512,
+          0.008372707292437553,
+          0.003292751731351018,
+          0.01196332648396492,
+          0.03355567157268524,
+          0.012798165902495384,
+          -0.009742491878569126,
+          0.0025531488936394453,
+          0.023440338671207428,
+          0.00649634562432766,
+          0.0169561505317688,
+          0.008988705463707447,
+          3.9164748159237206e-05,
+          -0.013973425142467022,
+          0.011509433388710022,
+          0.033101778477430344,
+          0.024996543303132057,
+          0.025952961295843124,
+          0.005353507585823536,
+          0.0010577331995591521,
+          -0.01282248180359602,
+          -0.01343037374317646,
+          -0.016680574044585228,
+          -0.011736379936337471,
+          0.022678447887301445,
+          -0.009547966532409191,
+          0.002751727122813463,
+          -0.01872309297323227,
+          0.008632075041532516,
+          0.020019929856061935,
+          0.009507440030574799,
+          0.017653200775384903,
+          0.010228806175291538,
+          0.00451461598277092,
+          -0.012287535704672337,
+          0.0007613854832015932,
+          0.0060059791430830956,
+          -0.024964123964309692,
+          0.016558995470404625,
+          0.031756311655044556,
+          -0.0003996893938165158,
+          0.01539994589984417,
+          -0.0002955877862405032,
+          -0.003209673101082444,
+          -0.012190273031592369,
+          0.02392665296792984,
+          0.010625963099300861,
+          0.012449640780687332,
+          0.0023201233707368374,
+          -0.008713127113878727,
+          -0.011201434768736362,
+          -0.024218441918492317,
+          0.026568960398435593,
+          0.02092771604657173,
+          -0.00196551950648427,
+          0.00386417075060308,
+          -0.001570389256812632,
+          -0.012174062430858612,
+          0.014970368705689907,
+          0.015732260420918465,
+          0.019468773156404495,
+          -0.0028672267217189074,
+          0.012911639176309109,
+          -0.018139515072107315,
+          0.000802418275270611,
+          -0.012781955301761627,
+          0.01700478233397007,
+          0.004275511484593153,
+          -0.021462662145495415,
+          0.016696784645318985,
+          0.013770793564617634,
+          0.0005202547763474286,
+          -0.019955087453126907,
+          0.005531822796911001,
+          -0.0055196648463606834,
+          0.001069891033694148,
+          0.0007168066804297268,
+          -0.0012826534220948815,
+          -0.006747608073055744,
+          0.000229859389946796,
+          0.011703958734869957,
+          -0.0069097126834094524,
+          -0.022954024374485016,
+          -0.010447647422552109,
+          -0.001273535075597465,
+          -0.00656524021178484,
+          -0.020133404061198235,
+          0.015286472626030445,
+          0.004826667252928019,
+          -0.016907520592212677,
+          -0.020036140456795692,
+          -0.0004072880547028035,
+          0.008559128269553185,
+          0.03410682827234268,
+          -0.0284169539809227,
+          0.0032745148055255413,
+          -0.012352378107607365,
+          0.020798033103346825,
+          0.00357238226570189,
+          -0.023359287530183792,
+          0.022775709629058838,
+          0.0008277471060864627,
+          0.0056412434205412865,
+          -0.01179311703890562,
+          -0.0045835101045668125,
+          0.009766807779669762,
+          -0.022743288427591324,
+          0.005353507585823536,
+          0.01049627922475338,
+          -0.0038378287572413683,
+          0.008518601767718792,
+          9.124721691478044e-05,
+          0.005183297675102949,
+          0.0005379849462769926,
+          -0.009920807555317879,
+          0.018787933513522148,
+          -0.0014619816793128848,
+          -0.014605632983148098,
+          0.0034589089918881655,
+          0.013908582739531994,
+          -0.02564496360719204,
+          0.03595482185482979,
+          -0.0038256708066910505,
+          -0.014548896811902523,
+          -0.004433563444763422,
+          0.0010293647646903992,
+          0.019484983757138252,
+          0.030880944803357124,
+          -0.020911505445837975,
+          -0.010763752274215221,
+          0.005090087652206421,
+          0.01137974951416254,
+          -0.009815439581871033,
+          0.004206616897135973,
+          0.017572149634361267,
+          0.010553015395998955,
+          0.011209540069103241,
+          -0.010707015171647072,
+          -0.01595110259950161,
+          0.030427051708102226,
+          0.019922668114304543,
+          0.003683829214423895,
+          0.009701966308057308,
+          -0.002788200741633773,
+          0.020425191149115562,
+          0.0025531488936394453,
+          -0.01124196033924818,
+          0.00452677346765995,
+          0.016413100063800812,
+          -0.010763752274215221,
+          -0.0010749567300081253,
+          0.008247076533734798,
+          -0.011671538464725018,
+          0.008429444395005703,
+          -0.027395693585276604,
+          0.0263582244515419,
+          0.007906656712293625,
+          0.011955221183598042,
+          0.020133404061198235,
+          -0.01977677270770073,
+          -0.0011631011730059981,
+          0.014305739663541317,
+          0.012344272807240486,
+          -0.02353760227560997,
+          -0.008542917668819427,
+          0.008826600387692451,
+          -0.00020794992451556027,
+          -0.0008216681890189648,
+          -0.003635197877883911,
+          -0.013519532047212124,
+          -0.005000929813832045,
+          -0.04130427539348602,
+          0.011047434993088245,
+          -0.004786141216754913,
+          0.011922799982130527,
+          0.005985715892165899,
+          0.00911028403788805,
+          -0.02613127790391445,
+          0.003578461240977049,
+          -0.01000185962766409,
+          -0.015480998903512955,
+          0.005495349410921335,
+          0.006054610479623079,
+          -0.013162901625037193,
+          -0.0035642769653350115,
+          -0.0018358356319367886,
+          0.002589622512459755,
+          0.0021134400740265846,
+          -0.004790193866938353,
+          -0.029178844764828682,
+          -0.00823897123336792,
+          -0.013227743096649647,
+          0.006938081234693527,
+          -0.014840684831142426,
+          0.03052431344985962,
+          0.0005192416138015687,
+          -0.009207546710968018,
+          0.0036574872210621834,
+          -0.013859951868653297,
+          -0.00411948561668396,
+          -0.0038216181565076113,
+          0.01895003952085972,
+          -0.0038074341136962175,
+          0.028643900528550148,
+          0.00883470568805933,
+          -0.010958277620375156,
+          -0.018852775916457176,
+          0.003193462500348687,
+          -0.014119318686425686,
+          -0.0019422168843448162,
+          0.03133483603596687,
+          0.007720236200839281,
+          -0.008186287246644497,
+          0.013900477439165115,
+          -0.008996810764074326,
+          0.022078659385442734,
+          -0.01298458594828844,
+          -0.01027743797749281,
+          0.012887323275208473,
+          0.013738373294472694,
+          0.004384932108223438,
+          -1.1429647202021442e-05,
+          -0.005548033397644758,
+          0.00883470568805933,
+          -0.019922668114304543,
+          -0.001798348966985941,
+          0.012749534100294113,
+          -0.007079922594130039,
+          0.0036230399273335934,
+          0.017701832577586174,
+          0.013689741492271423,
+          -0.009766807779669762,
+          -0.00019199274538550526,
+          0.012076799757778645,
+          0.01583762839436531,
+          0.017020992934703827,
+          -0.01838267222046852,
+          0.0028874899726361036,
+          0.014751527458429337,
+          -0.02498033456504345,
+          -0.005803348030894995,
+          -0.012709008529782295,
+          0.0037000395823270082,
+          -0.036473557353019714,
+          0.00016045832308009267,
+          0.04493542015552521,
+          0.0035946716088801622,
+          0.0008135629468597472,
+          -0.029551686719059944,
+          -0.023359287530183792,
+          -0.005458875559270382,
+          -0.0011134566739201546,
+          -0.003997907042503357,
+          -0.019809193909168243,
+          -0.012263220734894276,
+          0.014702895656228065,
+          0.016631942242383957,
+          -0.018188146874308586,
+          -0.009620913304388523,
+          0.0013707978650927544,
+          -0.006192399188876152,
+          -0.005892505869269371,
+          0.0008166024344973266,
+          -0.004068827722221613,
+          -0.0010911672143265605,
+          0.01137974951416254,
+          0.015059526078402996,
+          0.007481131702661514,
+          0.016680574044585228,
+          0.011987642385065556,
+          -0.008048498071730137,
+          0.014176055788993835,
+          -0.003600750584155321,
+          -0.004555142018944025,
+          -0.004287669435143471,
+          -0.023570023477077484,
+          -0.0038540391251444817,
+          0.003408251330256462,
+          -0.014135529287159443,
+          0.008948178961873055,
+          0.00389861804433167,
+          0.003991828300058842,
+          -0.001961466856300831,
+          -0.010690804570913315,
+          0.0016960203647613525,
+          -0.02287297323346138,
+          -0.006504450924694538,
+          0.0027942797169089317,
+          -0.017815306782722473,
+          0.019306669011712074,
+          0.01922561787068844,
+          -0.0059532951563596725,
+          -0.016648152843117714,
+          0.014224686659872532,
+          0.001204640488140285,
+          0.015942996367812157,
+          0.015237841755151749,
+          0.000640313548501581,
+          0.0047739832662045956,
+          0.01204437855631113,
+          -0.0019857825245708227,
+          -0.021624766290187836,
+          -0.018301621079444885,
+          0.0053251395002007484,
+          -0.003165094181895256,
+          -0.018674461171030998,
+          0.0026443328242748976,
+          -0.0029118056409060955,
+          0.018155725672841072,
+          0.0022836499847471714,
+          0.006694924086332321,
+          -0.0030475682578980923,
+          -0.0218841340392828,
+          -0.0015795076033100486,
+          -0.004506510682404041,
+          -0.02853042632341385,
+          -0.00620455713942647,
+          -0.009888386353850365,
+          0.025466646999120712,
+          -0.021738240495324135,
+          0.015164894051849842,
+          -0.003852012800052762,
+          0.0042147221975028515,
+          0.02049003355205059,
+          0.008680706843733788,
+          0.002648385474458337,
+          0.012433430179953575,
+          -0.008591548539698124,
+          -0.00533324433490634,
+          0.01088532991707325,
+          -0.008461865596473217,
+          0.013495216146111488,
+          -0.024623703211545944,
+          0.01559447217732668,
+          -0.014694790355861187,
+          -0.005394033621996641,
+          0.010074807330965996,
+          0.013187216594815254,
+          0.0074325003661215305,
+          -0.01531078852713108,
+          -0.012376694008708,
+          0.022532552480697632,
+          -0.0077931834384799,
+          0.00331504107452929,
+          -0.003886460093781352,
+          -0.011250065639615059,
+          -0.01672920398414135,
+          -0.0006707081920467317,
+          0.013551952317357063,
+          -0.017020992934703827,
+          -0.00985596515238285,
+          -0.01595110259950161,
+          -0.0071690804325044155,
+          -0.04947435483336449,
+          -0.02369970642030239,
+          0.013025112450122833,
+          -0.009815439581871033,
+          -0.018706882372498512,
+          0.02614748664200306,
+          -0.006472030188888311,
+          -0.00591276865452528,
+          -0.008076866157352924,
+          -0.008283549919724464,
+          -0.014143634587526321,
+          0.01961466856300831,
+          0.010317963548004627,
+          -0.01365732029080391,
+          0.01922561787068844,
+          0.000424764963099733,
+          0.015051420778036118,
+          -0.012611744925379753,
+          -0.018204357475042343,
+          -0.0032542517874389887,
+          0.022743288427591324,
+          0.022030027583241463,
+          -0.009928912855684757,
+          0.0014619816793128848,
+          0.0021316767670214176,
+          0.018074674531817436,
+          0.01174448523670435,
+          -0.010739436373114586,
+          -0.010561120696365833,
+          0.013008901849389076,
+          -0.00985596515238285,
+          0.003858091775327921,
+          0.009661439806222916,
+          0.0042349849827587605,
+          0.002431570552289486,
+          -0.009037337265908718,
+          -0.001908782753162086,
+          0.006091083865612745,
+          0.0217706598341465,
+          -0.0007553065661340952,
+          -0.0061113471165299416,
+          -0.004867193289101124,
+          -0.009783018380403519,
+          -0.0025693594943732023,
+          -0.020862873643636703,
+          -0.013203427195549011,
+          -0.0023201233707368374,
+          0.018317829817533493,
+          0.004660509992390871,
+          -0.020798033103346825,
+          -0.02702285349369049,
+          -0.008988705463707447,
+          -0.023861810564994812,
+          -0.0055358754470944405,
+          -0.011250065639615059,
+          -0.008745548315346241,
+          0.008036340586841106,
+          0.009815439581871033,
+          -0.0017213491955772042,
+          -0.00199996680021286,
+          -0.006504450924694538,
+          0.005017140414565802,
+          -0.024947913363575935,
+          -0.0013272322248667479,
+          -0.0015896392287686467,
+          -0.021803081035614014,
+          -0.012733323499560356,
+          0.005916821304708719,
+          0.031059259548783302,
+          0.04224448278546333,
+          -0.008940073661506176,
+          0.0009270362206734717,
+          -0.0038216181565076113,
+          0.010674593970179558,
+          0.017847727984189987,
+          0.0090535469353199,
+          0.007019133307039738,
+          0.017345203086733818,
+          -0.03203188627958298,
+          0.0043200901709496975,
+          0.012392903678119183,
+          -0.0004067814734298736,
+          -0.007647288963198662,
+          -0.015489104203879833,
+          0.005000929813832045,
+          -0.008153866045176983,
+          0.005568296182900667,
+          0.012660376727581024,
+          -0.011039329692721367,
+          -0.00946691446006298,
+          0.0283034797757864,
+          0.009572282433509827,
+          0.011857958510518074,
+          0.01522973645478487,
+          -0.01407068781554699,
+          -0.022289395332336426,
+          0.0008231879328377545,
+          0.006549029611051083,
+          -0.0037446185015141964,
+          0.008575338870286942,
+          -0.010650279000401497,
+          -0.03595482185482979,
+          0.0002515155647415668,
+          -0.0006661489605903625,
+          -0.012336167506873608,
+          -0.016332048922777176,
+          0.0035055140033364296,
+          0.01932287961244583,
+          0.025904331356287003,
+          -0.01531889382749796,
+          0.017410043627023697,
+          0.007663499563932419,
+          -0.011720169335603714,
+          -0.009426387958228588,
+          -0.009572282433509827,
+          -0.015164894051849842,
+          -0.02037656120955944,
+          0.001938164234161377,
+          -0.014929842203855515,
+          -0.011833642609417439,
+          0.008234918117523193,
+          0.03263167664408684,
+          -0.0009467927156947553,
+          -0.004838825203478336,
+          -0.0011590485228225589,
+          0.0070839752443134785,
+          0.004518668632954359,
+          -0.010698909871280193,
+          0.01450837031006813,
+          0.019290458410978317,
+          0.010966382920742035,
+          0.001982743153348565,
+          -0.021235715597867966,
+          -0.008948178961873055,
+          0.017572149634361267,
+          -0.01223890483379364,
+          0.004575405269861221,
+          0.020198244601488113,
+          -0.005949242506176233,
+          -0.0069097126834094524,
+          0.0015339156379923224,
+          -0.007720236200839281,
+          -0.03268030658364296,
+          0.008315971121191978,
+          -0.007108291145414114,
+          0.011063645593822002,
+          0.0071528698317706585,
+          -0.019792983308434486,
+          0.003991828300058842,
+          -0.007233921904116869,
+          0.002139782067388296,
+          -0.0060789259150624275,
+          -0.0018024016171693802,
+          -0.00044198858086019754,
+          -0.01723172888159752,
+          0.024785807356238365,
+          -0.003612908534705639,
+          2.2194411940290593e-05,
+          0.0016372574027627707,
+          0.022532552480697632,
+          -0.0011428381549194455,
+          0.006334241013973951,
+          0.0033089620992541313,
+          -0.012725219130516052,
+          0.006654397584497929,
+          -0.009118389338254929,
+          -0.004393037408590317,
+          -0.00787423551082611,
+          0.012944060377776623,
+          0.013876162469387054,
+          0.02425086311995983,
+          0.00017109644250012934,
+          0.005527770146727562,
+          -0.019371511414647102,
+          -0.0035237509291619062,
+          -0.01553773507475853,
+          0.0031205154955387115,
+          0.010617857798933983,
+          0.0181070938706398,
+          -0.003770960494875908,
+          -0.004587562754750252,
+          -0.0010258187539875507,
+          0.009069757536053658,
+          0.016404995694756508,
+          -0.004064775072038174,
+          -0.022062448784708977,
+          0.0016382705653086305,
+          -0.004066801629960537,
+          0.009685755707323551,
+          0.022143501788377762,
+          -0.01634015329182148,
+          0.02032792940735817,
+          0.001971598481759429,
+          -0.015408051200211048,
+          0.013527637347579002,
+          -0.0074000791646540165,
+          -0.027233589440584183,
+          -0.020911505445837975,
+          0.0018753487383946776,
+          0.004879351239651442,
+          0.013033217750489712,
+          0.012433430179953575,
+          0.014840684831142426,
+          0.004725351929664612,
+          0.015610681846737862,
+          -0.0007117409259080887,
+          -0.0083078658208251,
+          -0.017085835337638855,
+          -0.0006367675377987325,
+          -0.007906656712293625,
+          -0.00822681374847889,
+          -0.038353972136974335,
+          -0.017426254227757454,
+          -0.007785078138113022,
+          0.02541801705956459,
+          0.013730267994105816,
+          0.005770927295088768,
+          -0.004571352619677782,
+          -0.01826919987797737,
+          -0.007290658541023731,
+          0.019484983757138252,
+          -0.0026564907748252153,
+          0.029308529570698738,
+          -0.006318030413240194,
+          0.00218841340392828,
+          0.003229936119168997,
+          -0.00817007664591074,
+          -0.020360350608825684,
+          0.022970234975218773,
+          0.01298458594828844,
+          0.007732394151389599,
+          0.014038266614079475,
+          -0.02010098285973072,
+          0.02404012717306614,
+          -0.014143634587526321,
+          -0.023164760321378708,
+          0.012668482027947903,
+          0.004534878768026829,
+          0.01961466856300831,
+          -0.009434493258595467,
+          0.005296770948916674,
+          0.015116263180971146,
+          -0.02287297323346138,
+          0.0263582244515419,
+          -0.005361612886190414,
+          0.023683495819568634,
+          -0.006390977650880814,
+          0.00852670706808567,
+          -0.015148683451116085,
+          0.005146824289113283,
+          -0.04454636946320534,
+          0.0020546771120280027,
+          -0.03024873696267605,
+          -8.561154390918091e-05,
+          0.017961200326681137,
+          -0.025612542405724525,
+          -0.0026504117995500565,
+          0.005211665760725737,
+          -0.011671538464725018,
+          -0.007456816267222166,
+          0.02254876308143139,
+          0.009783018380403519,
+          0.0008839771617203951,
+          -0.0003571369161363691,
+          0.011857958510518074,
+          -0.013770793564617634,
+          -0.024412967264652252,
+          -0.01838267222046852,
+          -0.017474886029958725,
+          -0.033750198781490326,
+          0.0003259824006818235,
+          0.00023669817892368883,
+          0.0045470367185771465,
+          -0.008915758691728115,
+          -0.004170143511146307,
+          -0.009710071608424187,
+          -0.0014579291455447674,
+          0.016048364341259003,
+          0.008198444731533527,
+          -0.014800158329308033,
+          -0.004506510682404041,
+          -0.006034347228705883,
+          -0.0016676520463079214,
+          0.00022276731033343822,
+          0.004769930616021156,
+          0.018852775916457176,
+          -0.011387854814529419,
+          -0.008591548539698124,
+          0.025612542405724525,
+          0.022954024374485016,
+          -0.016323942691087723,
+          0.00389861804433167,
+          -0.03751102834939957,
+          0.01865825057029724,
+          -0.02293781377375126,
+          -5.395046900957823e-05,
+          -0.012571219354867935,
+          0.022662237286567688,
+          -0.003939144313335419,
+          0.0035217246040701866,
+          0.013422268442809582,
+          -0.010034280829131603,
+          0.020895294845104218,
+          0.011922799982130527,
+          -0.0031326732132583857,
+          0.0026504117995500565,
+          -0.010674593970179558,
+          0.01356816291809082,
+          -0.02974621206521988,
+          0.00013918208424001932,
+          0.01893382892012596,
+          0.006285609677433968,
+          0.011314908042550087,
+          -0.015894364565610886,
+          0.004498405382037163,
+          0.0028814109973609447,
+          0.030491894111037254,
+          0.004680773243308067,
+          0.004103275015950203,
+          0.0035602243151515722,
+          0.0005020179669372737,
+          -0.029000530019402504,
+          0.009499335661530495,
+          0.007849919609725475,
+          -0.007699972949922085,
+          0.021576134487986565,
+          0.012133536860346794,
+          0.018366461619734764,
+          -0.006788134109228849,
+          -0.009742491878569126,
+          0.025661174207925797,
+          0.01744246482849121,
+          -0.006192399188876152,
+          0.0025146491825580597,
+          0.023683495819568634,
+          -0.0074000791646540165,
+          0.040234386920928955,
+          0.027655061334371567,
+          -0.009264283813536167,
+          -0.028611479327082634,
+          0.010836699046194553,
+          -0.01509194727987051,
+          -0.004131643567234278,
+          0.01922561787068844,
+          0.022678447887301445,
+          -0.011258170939981937,
+          0.01531889382749796,
+          0.015440472401678562,
+          0.012652271427214146,
+          0.01656709983944893,
+          -0.008996810764074326,
+          -0.0035338823217898607,
+          0.00047618255484849215,
+          -0.0013201400870457292,
+          -0.0002829233417287469,
+          -0.02608264610171318,
+          0.006151873152703047,
+          -0.0023748339153826237,
+          0.007193395867943764,
+          0.03007042035460472,
+          0.0073960269801318645,
+          -0.023359287530183792,
+          -0.0150108952075243,
+          -0.01998750865459442,
+          0.022662237286567688,
+          -0.022451501339673996,
+          -0.017345203086733818,
+          -0.00869691651314497,
+          -0.01260364055633545,
+          -0.032988306134939194,
+          0.009904596954584122,
+          0.00013740906433667988,
+          -0.005215718410909176,
+          0.0006104254862293601,
+          0.0013515478931367397,
+          -0.005491296760737896,
+          -0.0015268236165866256,
+          0.0016717046964913607,
+          0.008202497847378254,
+          0.010091017000377178,
+          0.016648152843117714,
+          -0.005365665536373854,
+          0.0166157316416502,
+          -0.012571219354867935,
+          -0.03738134354352951,
+          -0.0037000395823270082,
+          -0.003385961754247546,
+          0.017102045938372612,
+          0.0005400112713687122,
+          0.006480135023593903,
+          -0.019063511863350868,
+          0.003440672066062689,
+          0.00040146239916794,
+          0.005292718298733234,
+          0.01304132305085659,
+          -0.0013525610556825995,
+          -0.004474089480936527,
+          0.027071483433246613,
+          -0.004786141216754913,
+          0.007104238495230675,
+          -0.00986407045274973,
+          0.016648152843117714,
+          -0.016826467588543892,
+          -0.003088094526901841,
+          0.007027238607406616,
+          -0.010828593745827675,
+          -0.0022613604087382555,
+          -0.011509433388710022,
+          -0.017912568524479866,
+          -0.010212595574557781,
+          0.011395960114896297,
+          -0.020960137248039246,
+          -0.013106164522469044,
+          -0.0004678240220528096,
+          -0.007193395867943764,
+          0.008502391166985035,
+          0.01601594313979149,
+          0.005219771061092615,
+          -0.010763752274215221,
+          0.023229602724313736,
+          0.013284480199217796,
+          0.023732127621769905,
+          -0.006767870858311653,
+          -0.013989635743200779,
+          -0.004174196161329746,
+          -0.006301820278167725,
+          0.015983523800969124,
+          0.0070839752443134785,
+          0.005276507697999477,
+          -0.003495382610708475,
+          0.02160855568945408,
+          -0.009369651786983013,
+          0.0006732410402037203,
+          -0.002932068658992648,
+          0.010504384525120258,
+          -0.010974488221108913,
+          -0.009239967912435532,
+          0.015140579082071781,
+          0.026941800490021706,
+          0.0059006111696362495,
+          0.005673664156347513,
+          0.009021126665174961,
+          0.00020896308706142008,
+          0.009069757536053658,
+          0.0029158582910895348,
+          -0.006362609565258026,
+          -0.03465798497200012,
+          -0.00875365361571312,
+          -0.01005049142986536,
+          -0.019825404509902,
+          0.019582247361540794,
+          -0.0002821634989231825,
+          -0.009677650406956673,
+          -0.006638187449425459,
+          -0.00928049348294735,
+          0.004316037520766258,
+          0.0013900477206334472,
+          -0.000849529926199466,
+          -0.008429444395005703,
+          0.008003919385373592,
+          -0.023667285218834877,
+          -0.015027105808258057,
+          0.028708741068840027,
+          0.008915758691728115,
+          0.013292585499584675,
+          -0.008672601543366909,
+          -0.0021033084485679865,
+          0.006087031215429306,
+          -0.02442917786538601,
+          -0.00883470568805933,
+          0.06539303809404373,
+          -0.016242889687418938,
+          0.019695719704031944,
+          -0.024331914260983467,
+          -0.001989835174754262,
+          -0.002711201086640358,
+          0.018204357475042343,
+          0.021803081035614014,
+          -0.025515278801321983,
+          -0.0009123454801738262,
+          -0.02076561190187931,
+          0.0005531822680495679,
+          0.0067030293866992,
+          0.0026747274678200483,
+          -0.015124368481338024,
+          0.010350384749472141,
+          -0.03222641348838806,
+          -0.016794046387076378,
+          0.009726282209157944,
+          -0.0075378683395683765,
+          0.003035410540178418,
+          -0.016161838546395302,
+          -0.005811453331261873,
+          0.007541920989751816,
+          0.010690804570913315,
+          -0.006224820390343666,
+          -0.027849586680531502,
+          -0.02575843594968319,
+          -0.009013021364808083,
+          -0.02026308700442314,
+          0.017701832577586174,
+          -0.015780892223119736,
+          0.0166157316416502,
+          -0.009491230361163616,
+          -0.003108357544988394,
+          0.006982659921050072,
+          0.01689130999147892,
+          0.012538798153400421,
+          0.0017274281708523631,
+          -0.005787137895822525,
+          -0.008291655220091343,
+          0.01700478233397007,
+          -0.013633005321025848,
+          -0.0359872430562973,
+          0.0003687881981022656,
+          -0.005191402975469828,
+          0.0020789927802979946,
+          0.008769864216446877,
+          0.018285410478711128,
+          0.01590247079730034,
+          0.01531889382749796,
+          -0.015359420329332352,
+          -0.01379510946571827,
+          0.009118389338254929,
+          0.031205154955387115,
+          -0.001978690503165126,
+          0.03637629374861717,
+          0.011874169111251831,
+          0.004964456427842379,
+          -0.003852012800052762,
+          -0.008380812592804432,
+          -0.018285410478711128,
+          -0.003509566653519869,
+          0.02049003355205059,
+          -0.0155863668769598,
+          0.02381318062543869,
+          0.0010394963901489973,
+          -0.010609752498567104,
+          -0.001762888627126813,
+          -0.0011357460170984268,
+          -0.018285410478711128,
+          0.013503321446478367,
+          -0.01599973440170288,
+          0.0023768600076436996,
+          -0.016145627945661545,
+          0.001996927196159959,
+          -0.004855035804212093,
+          0.01933909021317959,
+          0.0019766641780734062,
+          -0.007371711079031229,
+          0.005357560236006975,
+          -0.01202006358653307,
+          0.002869253046810627,
+          -3.042629214178305e-05,
+          0.01638878509402275,
+          -0.0003361139679327607,
+          0.0010253122309222817,
+          -0.02026308700442314,
+          0.008259234018623829,
+          0.0022147553972899914,
+          -0.015562050975859165,
+          -0.01760457083582878,
+          0.007991761900484562,
+          0.002368754940107465,
+          0.002439675619825721,
+          0.006115399766713381,
+          0.003961433656513691,
+          -0.004393037408590317,
+          0.004676720593124628,
+          0.007464921101927757,
+          0.005389980971813202,
+          0.005081982351839542,
+          -0.013235848397016525,
+          0.002636227523908019,
+          0.0018642040668055415,
+          0.004311984870582819,
+          0.01771804317831993,
+          -0.0001176525474875234,
+          -0.004206616897135973,
+          0.00875365361571312,
+          0.009442598558962345,
+          0.008534812368452549,
+          -0.026682432740926743,
+          -0.018188146874308586,
+          0.012717113830149174,
+          -0.001926006400026381,
+          0.006168083753436804,
+          0.00018452072981745005,
+          0.018690671771764755,
+          0.005645296070724726,
+          0.004328195471316576,
+          0.006974554620683193,
+          0.008255181834101677,
+          -0.005134666338562965,
+          -0.0018571119289845228,
+          0.013511426746845245,
+          -0.012417219579219818,
+          0.010228806175291538,
+          -0.02901674062013626,
+          0.010326068848371506,
+          0.009167020209133625,
+          -0.0023647022899240255,
+          0.0014001793460920453,
+          -0.027849586680531502,
+          -0.005803348030894995,
+          -0.01572415605187416,
+          0.0059532951563596725,
+          -0.012068694457411766,
+          0.017572149634361267,
+          0.01783151738345623,
+          -0.018123304471373558,
+          0.04282806068658829,
+          0.0035602243151515722,
+          0.02674727514386177,
+          -0.009094073437154293,
+          -0.003929012455046177,
+          -0.00244372827000916,
+          0.008923863992094994,
+          -0.01063406839966774,
+          0.012571219354867935,
+          0.006804344709962606,
+          0.02065213769674301,
+          -0.00605055782943964,
+          0.011809326708316803,
+          0.00012461798905860633,
+          0.027849586680531502,
+          -0.006662502884864807,
+          0.013543847016990185,
+          -0.011663433164358139,
+          -0.024412967264652252,
+          -0.010366595350205898,
+          0.0037912235129624605,
+          0.006557134911417961,
+          0.005150876939296722,
+          0.008109287358820438,
+          0.001973624574020505,
+          0.009807334281504154,
+          0.007837762124836445,
+          0.04039648920297623,
+          -0.012700903229415417,
+          -0.010488173924386501,
+          0.011371644213795662,
+          -0.0032542517874389887,
+          0.0025572015438228846,
+          0.005037403665482998,
+          0.011120381765067577,
+          0.01143648661673069,
+          7.332704990403727e-05,
+          -0.014127423986792564,
+          -0.009150810539722443,
+          -0.009791123680770397,
+          0.002575438469648361,
+          -0.017134467139840126,
+          0.007254185155034065,
+          -0.001919927541166544,
+          -0.00815791916102171,
+          0.006727344822138548,
+          -0.006881344132125378,
+          0.0044781421311199665,
+          0.012417219579219818,
+          0.01223890483379364,
+          0.00765539426356554,
+          -0.014240897260606289,
+          0.00401614373549819,
+          0.004453826695680618,
+          0.022662237286567688,
+          -0.004830719903111458,
+          0.017766674980521202,
+          -0.020425191149115562,
+          -0.0043565635569393635,
+          0.010026175528764725,
+          -0.012660376727581024,
+          -0.012749534100294113,
+          0.0057790325954556465
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "3080f374-964f-4b6f-bb07-d9220ba99e16",
+      "properties": {
+        "page_content": "7. Deepseekmoe 145B Ongoing\n\nEncouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available.\n\n7.1. Experimental Setup\n\nTraining Data and Tokenization. For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study.\n\nModel Settings. For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps.\n\nSubsequently, the learning rate keeps constant during the remaining training process. The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. Also, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse.\n\nEvaluation Benchmarks. We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3).\n\n7.2. Evaluations\n\nBaselines. Apart from DeepSeekMoE 145B, we consider three additional models for comparison. DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters.\n\nResults. From the evaluation results presented in Table 6, we have the following observations: (1) Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. (2) On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. (3) At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5.\n\n",
+        "keyphrases": [
+          "DeepSeekMoE 145B",
+          "GShard architecture",
+          "Transformer layers",
+          "multi-head attention mechanism",
+          "expert parallelism"
+        ],
+        "title": "Deepseekmoe 145B Ongoing",
+        "embedding": [
+          0.011145060881972313,
+          0.0028371023945510387,
+          -0.01864549145102501,
+          -0.008063549175858498,
+          0.004066969733685255,
+          0.007203228771686554,
+          0.0034158634953200817,
+          0.061411239206790924,
+          -0.013921549543738365,
+          0.03988758474588394,
+          0.002668948844075203,
+          -0.009103755466639996,
+          0.020194068551063538,
+          -0.01502432394772768,
+          -0.007441772148013115,
+          0.021367233246564865,
+          0.03610217571258545,
+          0.012208729982376099,
+          0.007637299597263336,
+          -0.028171585872769356,
+          -0.0256375502794981,
+          -0.05227620154619217,
+          -0.012240014038980007,
+          -0.0378853864967823,
+          0.011450083926320076,
+          0.01580643467605114,
+          0.029986079782247543,
+          0.0068551902659237385,
+          -0.014132719486951828,
+          0.018520353361964226,
+          -0.006624467670917511,
+          0.03178493306040764,
+          -0.013851160183548927,
+          -0.0013999760849401355,
+          0.0002678725286386907,
+          -0.01470365934073925,
+          0.05349629372358322,
+          0.010644510388374329,
+          -0.018989618867635727,
+          0.0074926093220710754,
+          0.04495565593242645,
+          0.005001590587198734,
+          -0.02932910807430744,
+          0.016236593946814537,
+          0.029907869175076485,
+          -0.009471346624195576,
+          -0.025668835267424583,
+          6.886230403324589e-05,
+          -0.0020589034538716078,
+          -0.024151543155312538,
+          -0.01281095389276743,
+          -0.013452284038066864,
+          0.004950753413140774,
+          -0.008759627118706703,
+          -0.03494465351104736,
+          -0.008282540366053581,
+          0.038104377686977386,
+          0.06757426261901855,
+          0.05612418055534363,
+          -0.026216311380267143,
+          0.031925711780786514,
+          -0.015743864700198174,
+          -0.030079932883381844,
+          0.011356230825185776,
+          0.0009101799805648625,
+          -0.01856728084385395,
+          -0.017018703743815422,
+          0.0004423806967679411,
+          -0.00746132479980588,
+          0.02050691284239292,
+          -0.046363454312086105,
+          0.026998421177268028,
+          -0.016596363857388496,
+          0.01866113394498825,
+          0.030392777174711227,
+          -0.00615520216524601,
+          0.03419382870197296,
+          0.007711600046604872,
+          -0.0013344744220376015,
+          -0.026623008772730827,
+          0.02580961398780346,
+          0.04971088096499443,
+          -0.015008682385087013,
+          0.010143960826098919,
+          0.06569720059633255,
+          -0.01866113394498825,
+          -0.027326907962560654,
+          -0.06444582343101501,
+          0.011090313084423542,
+          -0.012138339690864086,
+          -0.02261860854923725,
+          -0.0006212882581166923,
+          0.012756206095218658,
+          -0.022383974865078926,
+          -0.00839985627681017,
+          0.0512751005589962,
+          -0.031206170096993446,
+          -0.009651231579482555,
+          -0.017691317945718765,
+          0.019365031272172928,
+          0.008118296973407269,
+          -0.015853360295295715,
+          0.015962855890393257,
+          -0.0033376524224877357,
+          0.01507125049829483,
+          -0.005983137991279364,
+          -0.030721262097358704,
+          0.032942455261945724,
+          -0.006495419889688492,
+          0.08321645110845566,
+          -0.020741544663906097,
+          -0.001445924979634583,
+          -0.0033435183577239513,
+          0.01900526136159897,
+          -0.008524994365870953,
+          -0.007121107541024685,
+          -0.03472566232085228,
+          -0.01020652987062931,
+          0.021836498752236366,
+          -0.021429801359772682,
+          0.05143152177333832,
+          -0.00858756247907877,
+          0.016690216958522797,
+          -0.036352451890707016,
+          -0.054278403520584106,
+          0.050023727118968964,
+          0.008141760714352131,
+          0.002275938866659999,
+          0.000169253398780711,
+          -0.0024245395325124264,
+          -0.016393015161156654,
+          0.0008818284841254354,
+          -0.03838593512773514,
+          -0.00615520216524601,
+          0.003464745357632637,
+          0.013491389341652393,
+          0.06172408536076546,
+          0.062068212777376175,
+          -0.01295173354446888,
+          0.01028474047780037,
+          -0.019724803045392036,
+          -0.020991820842027664,
+          0.019803013652563095,
+          -0.002023708540946245,
+          -0.01618966832756996,
+          0.011457905173301697,
+          0.0026865461841225624,
+          0.0422339141368866,
+          -0.011700358241796494,
+          0.003171454183757305,
+          -0.00040083113708533347,
+          0.02399512007832527,
+          0.01683099754154682,
+          0.04063841328024864,
+          -0.00606525968760252,
+          0.00877526868134737,
+          -0.019224252551794052,
+          -0.006995969917625189,
+          -0.052057210355997086,
+          0.015915928408503532,
+          -0.003816694486886263,
+          -0.008337287232279778,
+          -0.01634608954191208,
+          0.002739338669925928,
+          -0.004301602486521006,
+          0.035977039486169815,
+          0.02426103875041008,
+          0.012505931779742241,
+          -0.0752076506614685,
+          0.023228652775287628,
+          0.020553838461637497,
+          0.017347188666462898,
+          -0.048177946358919144,
+          0.03757254034280777,
+          0.0016443852800875902,
+          0.015681296586990356,
+          0.008329466916620731,
+          0.0012396436650305986,
+          -0.044236116111278534,
+          0.04088868573307991,
+          -0.009948433376848698,
+          0.006557988468557596,
+          0.06360115110874176,
+          0.009455704130232334,
+          0.021648792549967766,
+          0.0022270570043474436,
+          0.052057210355997086,
+          0.012826596386730671,
+          -0.0038694869726896286,
+          0.033755846321582794,
+          -0.0005205916822887957,
+          0.005908837541937828,
+          0.028234153985977173,
+          0.0011770748533308506,
+          0.009181966073811054,
+          -0.02388562448322773,
+          0.007046807091683149,
+          0.014625448733568192,
+          -0.00258291675709188,
+          -0.018535995855927467,
+          -0.01011267676949501,
+          -0.031378235667943954,
+          -0.01828572154045105,
+          0.009486989118158817,
+          0.0248867254704237,
+          0.00891604833304882,
+          0.003511671908199787,
+          0.029157044366002083,
+          0.024511313065886497,
+          -0.017613107338547707,
+          0.009033365175127983,
+          -0.0015779059613123536,
+          -0.02821851149201393,
+          -0.048553358763456345,
+          0.00512672821059823,
+          0.022290121763944626,
+          -0.010988638736307621,
+          0.015423200093209743,
+          0.019505811855196953,
+          -0.015321525745093822,
+          -0.020194068551063538,
+          0.016142740845680237,
+          0.010542836971580982,
+          -0.02518392726778984,
+          -0.033411718904972076,
+          -0.014359530992805958,
+          0.014758407138288021,
+          -0.003525358624756336,
+          0.05446610972285271,
+          -0.025574982166290283,
+          -0.006084812339395285,
+          0.026795072481036186,
+          0.00516583351418376,
+          0.026138100773096085,
+          0.023197369650006294,
+          -0.04229648411273956,
+          0.024495670571923256,
+          0.0007200295804068446,
+          0.002708054380491376,
+          0.009463525377213955,
+          0.003918368835002184,
+          -0.01892705075442791,
+          -0.03281731531023979,
+          -0.015368452295660973,
+          0.02729562297463417,
+          -0.03203520551323891,
+          -0.025105716660618782,
+          0.02580961398780346,
+          0.010190887376666069,
+          0.014633269049227238,
+          0.006882563699036837,
+          -0.03963731229305267,
+          -0.007930590771138668,
+          0.002023708540946245,
+          -0.007789811119437218,
+          0.05969059839844704,
+          0.04886620491743088,
+          0.0029661504086107016,
+          -0.01457070093601942,
+          -0.030580483376979828,
+          0.00900990143418312,
+          -0.016095813363790512,
+          -0.03538263589143753,
+          0.029454244300723076,
+          -0.026341449469327927,
+          0.04339143633842468,
+          0.025668835267424583,
+          0.017049988731741905,
+          -0.017706960439682007,
+          -0.013577421195805073,
+          -0.012983018532395363,
+          -0.03191006928682327,
+          -0.03363071009516716,
+          0.008008801378309727,
+          0.001456678961403668,
+          -0.022947093471884727,
+          -0.011410977691411972,
+          -0.018332647159695625,
+          0.012099234387278557,
+          0.03813566267490387,
+          0.024667734280228615,
+          0.009948433376848698,
+          0.004305513110011816,
+          0.024307964369654655,
+          -0.035883184522390366,
+          -0.0030443614814430475,
+          -0.0018663088558241725,
+          -0.009846759028732777,
+          -0.015032145194709301,
+          -0.017534896731376648,
+          0.001207381603308022,
+          -0.011551758274435997,
+          -0.035069793462753296,
+          0.008712700568139553,
+          -0.0028762079309672117,
+          0.010229992680251598,
+          -0.012709279544651508,
+          0.015274599194526672,
+          0.022743744775652885,
+          -0.023823056370019913,
+          -0.049022626131772995,
+          0.014437741599977016,
+          0.020710259675979614,
+          -0.02784309908747673,
+          -0.015751685947179794,
+          -0.009862401522696018,
+          -0.0019249670440331101,
+          0.02407333068549633,
+          -0.04539363831281662,
+          -0.021257737651467323,
+          0.02471466176211834,
+          -0.019646592438220978,
+          -0.011465725488960743,
+          -0.026310164481401443,
+          -0.005298791918903589,
+          0.04192107170820236,
+          0.033755846321582794,
+          0.03907419368624687,
+          -0.06350729614496231,
+          0.002970061032101512,
+          0.019521454349160194,
+          0.03366199508309364,
+          -0.058752067387104034,
+          0.027702320367097855,
+          -0.00732445577159524,
+          -0.011583042331039906,
+          0.021914709359407425,
+          0.043766848742961884,
+          -0.023948194459080696,
+          -0.0025340348947793245,
+          0.017941592261195183,
+          -0.03485080227255821,
+          -0.02701406367123127,
+          -0.0019709160551428795,
+          0.005580351687967777,
+          -0.044236116111278534,
+          -0.03519492968916893,
+          0.00030037894612178206,
+          0.014343888498842716,
+          0.0040161325596272945,
+          -0.008650131523609161,
+          -0.03538263589143753,
+          -0.050899688154459,
+          0.02390126697719097,
+          0.0010548701975494623,
+          -0.004340707790106535,
+          -0.04160822555422783,
+          -0.014828796498477459,
+          -0.0064132981933653355,
+          0.01023781392723322,
+          0.0010734453098848462,
+          0.023275580257177353,
+          0.013155082240700722,
+          0.0010812664404511452,
+          -0.00877526868134737,
+          -0.04132666811347008,
+          -0.032754749059677124,
+          -0.01065233163535595,
+          0.007742884568870068,
+          0.029907869175076485,
+          0.024229753762483597,
+          -0.048021525144577026,
+          -0.018989618867635727,
+          0.03123745508491993,
+          -0.03869878128170967,
+          -0.014547237195074558,
+          0.04045070707798004,
+          0.018520353361964226,
+          0.0262006688863039,
+          0.00544739281758666,
+          0.00310106435790658,
+          0.043891988694667816,
+          0.0001796407887013629,
+          0.006757426541298628,
+          0.021523654460906982,
+          0.0043133338913321495,
+          0.01830136403441429,
+          -0.0014752540737390518,
+          -0.0040943436324596405,
+          0.0452372170984745,
+          -0.0049585746601223946,
+          0.03050227276980877,
+          -0.000946841377299279,
+          -0.020084572955965996,
+          0.020929250866174698,
+          0.025309065356850624,
+          0.00022534531308338046,
+          0.009064649231731892,
+          -0.011833317577838898,
+          0.0026200669817626476,
+          -0.004583161789923906,
+          -0.03566419333219528,
+          -0.013913728296756744,
+          -0.011794212274253368,
+          0.013561779633164406,
+          0.02444874495267868,
+          -0.0017939637182280421,
+          -0.015196388587355614,
+          -0.034318964928388596,
+          -0.0126779954880476,
+          -0.007817184552550316,
+          0.04564391076564789,
+          -0.004066969733685255,
+          0.012803133577108383,
+          -0.008978617377579212,
+          0.02424539625644684,
+          0.01015178207308054,
+          -0.003208604408428073,
+          0.03181621432304382,
+          -0.02047562785446644,
+          0.008110475726425648,
+          -0.020632049068808556,
+          -0.026716861873865128,
+          -0.021007461473345757,
+          -0.0005797386984340847,
+          0.03747868910431862,
+          0.023213010281324387,
+          0.009494809433817863,
+          -0.048303086310625076,
+          0.02471466176211834,
+          -0.047646112740039825,
+          0.01894269324839115,
+          0.01892705075442791,
+          -0.00530661316588521,
+          -0.013381893746554852,
+          0.008462425321340561,
+          -0.016127098351716995,
+          0.006890384946018457,
+          0.01928682066500187,
+          -0.025121359154582024,
+          0.00408652238547802,
+          0.000701943296007812,
+          0.03628988191485405,
+          -0.014719301834702492,
+          -0.04070097953081131,
+          -0.012107055634260178,
+          0.003024808596819639,
+          0.029188327491283417,
+          0.009635589085519314,
+          -0.026388375088572502,
+          0.0011487234150990844,
+          0.002221191069111228,
+          -0.009135039523243904,
+          -0.019599664956331253,
+          -0.026044247671961784,
+          -0.009995359927415848,
+          -0.054528675973415375,
+          0.010276919230818748,
+          -0.016220951452851295,
+          -0.011082491837441921,
+          -0.05396555736660957,
+          0.03325529769062996,
+          -0.06601004302501678,
+          -0.01033166702836752,
+          0.04949188977479935,
+          -0.004141270183026791,
+          -0.006733963266015053,
+          -0.007735063321888447,
+          -0.014351709745824337,
+          -0.039293184876441956,
+          0.022649891674518585,
+          0.0009991449769586325,
+          0.04877234995365143,
+          -0.02178957127034664,
+          -0.003578151110559702,
+          0.032473187893629074,
+          -0.018348289653658867,
+          -0.016009781509637833,
+          0.0018839063122868538,
+          0.028046447783708572,
+          -0.016877923160791397,
+          0.025496771559119225,
+          0.0018457785481587052,
+          0.018864482641220093,
+          -0.001967005431652069,
+          0.025293422862887383,
+          -0.0033298314083367586,
+          -0.026591723784804344,
+          -0.0008060616673901677,
+          0.04273446649312973,
+          0.018066730350255966,
+          -0.023228652775287628,
+          -0.018082372844219208,
+          -0.022915810346603394,
+          -0.02546548657119274,
+          -0.00638201367110014,
+          -0.008861301466822624,
+          -0.0019112802110612392,
+          0.0251682847738266,
+          0.030705619603395462,
+          -0.011066850274801254,
+          0.007918858900666237,
+          0.00011420021473895758,
+          0.022853240370750427,
+          0.0001724307076074183,
+          -0.017315905541181564,
+          -0.009666874073445797,
+          -0.007879753597080708,
+          0.016971776261925697,
+          0.0254498440772295,
+          -0.039199329912662506,
+          -0.021727003157138824,
+          0.017753886058926582,
+          -0.012153982184827328,
+          0.0022505202796310186,
+          0.02277502976357937,
+          0.011684716679155827,
+          0.02279067225754261,
+          -0.019177325069904327,
+          -0.01700306124985218,
+          0.024323606863617897,
+          -0.018129298463463783,
+          0.00502896448597312,
+          -0.019224252551794052,
+          -0.017128199338912964,
+          0.023744845762848854,
+          -0.004293781239539385,
+          0.0253559909760952,
+          0.01231822557747364,
+          0.016299162060022354,
+          -0.016408657655119896,
+          0.006925580091774464,
+          -0.015344989486038685,
+          0.029172684997320175,
+          -0.04179593548178673,
+          0.014085792936384678,
+          0.029626309871673584,
+          -0.00041891742148436606,
+          0.049585744738578796,
+          -0.043766848742961884,
+          -0.012513753026723862,
+          0.0074339513666927814,
+          0.019161684438586235,
+          -0.012419898994266987,
+          0.0007356717833317816,
+          -0.006999880541115999,
+          0.007672494743019342,
+          -0.00431724451482296,
+          -0.03679043427109718,
+          -0.004954664036631584,
+          0.01919296756386757,
+          -0.008376393467187881,
+          -0.0028586103580892086,
+          -0.0051971180364489555,
+          -0.007871932350099087,
+          0.002614201046526432,
+          0.05578005313873291,
+          -0.04630088433623314,
+          0.03391227126121521,
+          0.00905682798475027,
+          0.013186367228627205,
+          -0.031206170096993446,
+          -0.011246735230088234,
+          -0.003703288733959198,
+          0.00615911278873682,
+          -0.028797272592782974,
+          0.011614326387643814,
+          0.01033948827534914,
+          0.00864231027662754,
+          -0.01892705075442791,
+          -0.03253575786948204,
+          0.006624467670917511,
+          -0.028969338163733482,
+          0.0020784561056643724,
+          -0.0056742047891020775,
+          -0.007390934973955154,
+          -0.0034354161471128464,
+          0.025011863559484482,
+          -0.010667974129319191,
+          0.0014820975484326482,
+          -0.004305513110011816,
+          0.008955154567956924,
+          -0.00822779256850481,
+          -0.002438226481899619,
+          0.00794623326510191,
+          0.005959674715995789,
+          0.016862282529473305,
+          -0.007109375670552254,
+          -0.018129298463463783,
+          -0.004399366211146116,
+          0.01466455403715372,
+          -0.022540396079421043,
+          0.0036817805375903845,
+          0.03391227126121521,
+          0.0363837368786335,
+          0.0018242704682052135,
+          -0.010441162623465061,
+          -0.04254676029086113,
+          0.03409997746348381,
+          -0.013765127398073673,
+          0.01883319765329361,
+          0.027514614164829254,
+          -0.009776369668543339,
+          -0.04201492294669151,
+          -0.0014908963348716497,
+          -0.01442210003733635,
+          -0.011199808679521084,
+          0.017034346237778664,
+          -0.00836857222020626,
+          -0.02435489185154438,
+          -0.03162850812077522,
+          0.026341449469327927,
+          0.02727998048067093,
+          -0.00629207119345665,
+          -0.00733227701857686,
+          0.013452284038066864,
+          0.022118058055639267,
+          0.017112556844949722,
+          -0.016205308958888054,
+          0.01300648134201765,
+          -0.01231040433049202,
+          0.014304783195257187,
+          0.007840648293495178,
+          -0.009971896186470985,
+          0.004379813559353352,
+          -0.0043915449641644955,
+          0.006726142019033432,
+          -0.0004186730075161904,
+          -0.012232192791998386,
+          -0.01474276464432478,
+          0.03510107472538948,
+          0.004325065761804581,
+          0.006601004395633936,
+          0.005240133963525295,
+          0.028093375265598297,
+          0.035977039486169815,
+          -0.023635350167751312,
+          -0.0032281570602208376,
+          0.01866113394498825,
+          0.009189787320792675,
+          -0.03209777548909187,
+          -0.0107931112870574,
+          -0.020428700372576714,
+          -0.03014250099658966,
+          0.006796531844884157,
+          -0.007692047394812107,
+          0.013264577835798264,
+          0.009619947522878647,
+          0.026263238862156868,
+          -0.004149090964347124,
+          -0.0174410417675972,
+          0.03076818957924843,
+          -0.002999390009790659,
+          0.03907419368624687,
+          -0.003232067683711648,
+          -0.0027647572569549084,
+          -0.01093389093875885,
+          0.020147141069173813,
+          0.04066969454288483,
+          0.009041186422109604,
+          0.0033884895965456963,
+          0.011403157375752926,
+          0.022947093471884727,
+          0.044611528515815735,
+          -0.03632116690278053,
+          -0.016330447047948837,
+          -0.0011741418857127428,
+          -0.018410857766866684,
+          0.029172684997320175,
+          0.009158502332866192,
+          0.015032145194709301,
+          0.00863448902964592,
+          0.001260173972696066,
+          -0.0147505858913064,
+          0.03503850847482681,
+          -0.008783089928328991,
+          -0.015462305396795273,
+          0.02987658418715,
+          -0.02030356414616108,
+          0.03325529769062996,
+          0.009682515636086464,
+          -0.024558238685131073,
+          -0.024386174976825714,
+          0.0011389469727873802,
+          -0.0038851292338222265,
+          -0.027983879670500755,
+          0.031096674501895905,
+          0.029250897467136383,
+          -0.0049155582673847675,
+          0.006808263715356588,
+          0.009095934219658375,
+          0.02114824205636978,
+          -0.020240994170308113,
+          -0.016518153250217438,
+          0.017128199338912964,
+          -0.007171944249421358,
+          -0.022196268662810326,
+          0.03976244851946831,
+          -0.01856728084385395,
+          -0.011723821982741356,
+          -0.012302583083510399,
+          0.0032692179083824158,
+          0.011465725488960743,
+          0.017800813540816307,
+          0.03494465351104736,
+          -0.001723573892377317,
+          -0.028703419491648674,
+          0.021617507562041283,
+          0.026435302570462227,
+          -0.013170724734663963,
+          -0.019349390640854836,
+          0.025387275964021683,
+          -0.01855163834989071,
+          0.013358430936932564,
+          0.05161922797560692,
+          0.008251255378127098,
+          0.008955154567956924,
+          -0.024745944887399673,
+          -0.02178957127034664,
+          0.025387275964021683,
+          -0.017300263047218323,
+          0.04307859390974045,
+          0.013131619431078434,
+          0.02379177138209343,
+          0.007864111103117466,
+          -0.023040946573019028,
+          0.0011037520598620176,
+          0.0037756338715553284,
+          0.004927290137857199,
+          0.02369791828095913,
+          0.016846640035510063,
+          0.0032203360460698605,
+          0.005208849441260099,
+          -0.04514336213469505,
+          -0.014117076992988586,
+          -0.014062329195439816,
+          0.009768548421561718,
+          0.01747232675552368,
+          0.00155053217895329,
+          0.0033337417989969254,
+          0.005451303441077471,
+          -0.01506342925131321,
+          -0.033505573868751526,
+          0.01698741875588894,
+          -0.0031284382566809654,
+          -0.003015032270923257,
+          0.0047317626886069775,
+          0.011770748533308506,
+          -0.006909937597811222,
+          -0.0012777714291587472,
+          0.005025053862482309,
+          -0.011278019286692142,
+          -0.02269681915640831,
+          0.0009082247270271182,
+          -0.009698158130049706,
+          0.0358206182718277,
+          0.012552858330309391,
+          0.024964936077594757,
+          -0.002391299931332469,
+          0.01024563517421484,
+          -0.003580106422305107,
+          -0.016017602756619453,
+          0.012709279544651508,
+          -0.0076764049008488655,
+          -0.007246244698762894,
+          0.0022954915184527636,
+          0.027233054861426353,
+          -0.016393015161156654,
+          0.007261887192726135,
+          -0.013233293779194355,
+          0.020632049068808556,
+          0.046895287930965424,
+          0.002715875394642353,
+          0.0034667004365473986,
+          0.0422339141368866,
+          -0.009221071377396584,
+          0.0026591725181788206,
+          0.025793973356485367,
+          0.0017382384976372123,
+          -0.016705859452486038,
+          0.04282831773161888,
+          -0.0063350871205329895,
+          0.042390335351228714,
+          -0.044517673552036285,
+          -0.0013246979797258973,
+          0.0358206182718277,
+          -0.01025345642119646,
+          -0.008235613815486431,
+          -0.02599732019007206,
+          0.002567274495959282,
+          -0.020538195967674255,
+          0.021977277472615242,
+          -0.007359650917351246,
+          0.038730062544345856,
+          -0.03944960609078407,
+          0.004434560891240835,
+          0.015399736352264881,
+          -0.01216962467879057,
+          0.036164745688438416,
+          0.01737847365438938,
+          -0.031096674501895905,
+          -0.034600526094436646,
+          0.006663573440164328,
+          0.010175244882702827,
+          0.05446610972285271,
+          -0.01272492203861475,
+          0.008306003175675869,
+          0.027154842391610146,
+          -0.009979717433452606,
+          -0.00019626061839517206,
+          0.043516576290130615,
+          -0.010511551983654499,
+          0.02637273445725441,
+          0.03916804492473602,
+          -0.053621429949998856,
+          -0.02756153978407383,
+          0.019552739337086678,
+          -0.0008725409279577434,
+          0.011301483027637005,
+          0.010941712185740471,
+          0.014085792936384678,
+          -0.0028801183216273785,
+          -0.027342548593878746,
+          0.01938067376613617,
+          0.013585242442786694,
+          0.0015671519795432687,
+          -0.004997679963707924,
+          -0.016752786934375763,
+          -0.012380793690681458,
+          -0.01939631626009941,
+          -0.009979717433452606,
+          0.005025053862482309,
+          0.0025281691923737526,
+          -0.017425401136279106,
+          -0.04326630011200905,
+          -0.02929782308638096,
+          0.027045348659157753,
+          -0.0074691460467875,
+          0.03300502151250839,
+          0.023400716483592987,
+          0.023025304079055786,
+          0.0026024694088846445,
+          -0.025856541469693184,
+          0.03841722011566162,
+          -0.011817675083875656,
+          0.03372456505894661,
+          -0.0022251016926020384,
+          -0.015962855890393257,
+          -0.027968237176537514,
+          0.04310987889766693,
+          -0.01254503708332777,
+          0.006952953990548849,
+          -0.011739464476704597,
+          -0.0036153013352304697,
+          -0.004387634340673685,
+          -0.01066015288233757,
+          0.011184166185557842,
+          0.004974216688424349,
+          0.016877923160791397,
+          0.0008544546435587108,
+          0.021163884550333023,
+          0.017206409946084023,
+          -0.0030033006332814693,
+          0.018144940957427025,
+          -0.027858741581439972,
+          -0.02452695555984974,
+          0.014304783195257187,
+          -0.02684199996292591,
+          -0.027717962861061096,
+          -0.008783089928328991,
+          -0.04051327332854271,
+          -0.006941222120076418,
+          -0.0010157647775486112,
+          -0.010503730736672878,
+          0.035069793462753296,
+          0.03907419368624687,
+          0.015642190352082253,
+          -0.03575804829597473,
+          0.022446542978286743,
+          -0.04407969489693642,
+          -0.008798732422292233,
+          0.0030912880320101976,
+          0.011184166185557842,
+          -0.022305764257907867,
+          0.03278603032231331,
+          0.00886912178248167,
+          -0.008259076625108719,
+          -0.028500070795416832,
+          0.004899916239082813,
+          -0.01030038297176361,
+          -0.0003243310493417084,
+          -0.00799707043915987,
+          0.011794212274253368,
+          0.0257314033806324,
+          0.008118296973407269,
+          0.0005758281331509352,
+          0.002829281147569418,
+          -0.03779153153300285,
+          0.00258291675709188,
+          -0.00422339141368866,
+          0.0023639260325580835,
+          -0.009878043085336685,
+          -0.020147141069173813,
+          -0.005146280862390995,
+          0.002923134481534362,
+          0.003040450857952237,
+          -0.004020043183118105,
+          0.01873934455215931,
+          -0.015657832846045494,
+          0.009838937781751156,
+          0.005220581311732531,
+          -0.04154565930366516,
+          -0.01994379237294197,
+          -0.04526849836111069,
+          -0.0022075041197240353,
+          0.0002268117677886039,
+          0.022008562460541725,
+          -0.007723331451416016,
+          0.009596483781933784,
+          -0.020882325246930122,
+          -0.010832217521965504,
+          -0.0006887452327646315,
+          0.017315905541181564,
+          -0.028922410681843758,
+          0.00019540518405847251,
+          0.006601004395633936,
+          0.02986094169318676,
+          -0.04066969454288483,
+          0.007336187642067671,
+          0.029078831896185875,
+          -0.017988519743084908,
+          -0.004544056486338377,
+          -0.0060965437442064285,
+          0.028875483199954033,
+          -0.024745944887399673,
+          0.0034803873859345913,
+          -0.0065384358167648315,
+          0.006511061917990446,
+          0.01836393214762211,
+          0.01501650270074606,
+          -0.0008935601217672229,
+          0.015524874441325665,
+          0.02288452535867691,
+          -0.0020765007939189672,
+          0.03832336887717247,
+          -0.03963731229305267,
+          -0.03832336887717247,
+          0.028984978795051575,
+          -0.017597464844584465,
+          -0.0031382145825773478,
+          0.009408777579665184,
+          -0.052964456379413605,
+          -0.0026806804817169905,
+          0.011786391027271748,
+          -0.002029574243351817,
+          -0.05671858415007591,
+          0.027968237176537514,
+          0.03406869247555733,
+          0.02527778036892414,
+          0.0013012347044423223,
+          -0.012388614937663078,
+          -0.0006960774771869183,
+          0.02444874495267868,
+          0.012568499892950058,
+          0.0294073186814785,
+          0.014250035397708416,
+          -0.013749485835433006,
+          -0.0008559211273677647,
+          -0.014860081486403942,
+          -0.014719301834702492,
+          -0.017128199338912964,
+          0.016956135630607605,
+          -0.009893685579299927,
+          -0.01563436910510063,
+          0.006186486221849918,
+          -0.005009411368519068,
+          -0.0148444389924407,
+          -0.03935575112700462,
+          -0.03613346070051193,
+          0.018317004665732384,
+          -0.015313704498112202,
+          -0.04861592873930931,
+          0.0065384358167648315,
+          -0.009932790882885456,
+          -0.004446292761713266,
+          -0.007062449119985104,
+          0.010941712185740471,
+          -0.01598631963133812,
+          0.013436641544103622,
+          0.005990959238260984,
+          0.0004746427293866873,
+          0.0003773678618017584,
+          0.018410857766866684,
+          0.01873934455215931,
+          0.013843338936567307,
+          -0.02480851486325264,
+          0.03106539137661457,
+          -0.016612006351351738,
+          -0.05155666172504425,
+          0.020647691562771797,
+          -0.015352809801697731,
+          0.003449103096500039,
+          0.027702320367097855,
+          -0.010511551983654499,
+          -0.02914140187203884,
+          0.028155943378806114,
+          -0.031925711780786514,
+          -0.02343200147151947,
+          -0.022853240370750427,
+          -0.00033044128213077784,
+          -0.03876134753227234,
+          -0.0450495108962059,
+          -0.007042896468192339,
+          0.011262377724051476,
+          -0.00849370937794447,
+          0.0004577784857247025,
+          0.03463181108236313,
+          -0.00357228540815413,
+          -0.03475694730877876,
+          -0.007488698698580265,
+          -0.005251865368336439,
+          -0.009885864332318306,
+          -0.0026865461841225624,
+          0.00029206901672296226,
+          0.004720031283795834,
+          -0.006663573440164328,
+          -0.0262006688863039,
+          -0.022352689877152443,
+          -0.016064530238509178,
+          -0.007344008423388004,
+          -0.01956837996840477,
+          -0.02003764547407627,
+          -0.011098134331405163,
+          -0.03306759148836136,
+          0.03076818957924843,
+          0.026075532659888268,
+          -0.006542346440255642,
+          -0.009174144826829433,
+          -0.009526094421744347,
+          0.0020706350915133953,
+          0.02993915230035782,
+          0.006053527817130089,
+          0.005658562295138836,
+          -0.020272279158234596,
+          0.004829526413232088,
+          -0.011317124590277672,
+          0.011567399837076664,
+          -0.051775652915239334,
+          -0.017534896731376648,
+          0.039199329912662506,
+          0.02271246165037155,
+          -0.0005782722146250308,
+          -0.006409387569874525,
+          0.018535995855927467,
+          -0.029907869175076485,
+          -0.028343649581074715,
+          0.03876134753227234,
+          0.015376273542642593,
+          0.013905907981097698,
+          -0.005506051238626242,
+          0.003539045574143529,
+          0.012787491083145142,
+          0.0019386539934203029,
+          -0.01042552012950182,
+          -0.006022243294864893,
+          0.03363071009516716,
+          0.0017558359540998936,
+          0.0022172804456204176,
+          -0.00822779256850481,
+          0.023478928953409195,
+          0.01023781392723322,
+          -0.0019738490227609873,
+          0.01773824356496334,
+          0.01043334137648344,
+          -0.01709691435098648,
+          -0.011137239634990692,
+          -0.02473030425608158,
+          0.0035527325235307217,
+          0.02260296605527401,
+          0.010034465231001377,
+          -0.02379177138209343,
+          -0.0061982180923223495,
+          -0.006718320772051811,
+          0.00267676985822618,
+          -0.012052307836711407,
+          0.012302583083510399,
+          0.025105716660618782,
+          -0.030940253287553787,
+          -0.014867902733385563,
+          -0.008259076625108719,
+          0.005032874643802643,
+          0.01033948827534914,
+          0.02086668275296688,
+          0.019693518057465553,
+          -0.0020765007939189672,
+          -0.004626177717000246,
+          0.0063116238452494144,
+          0.02756153978407383,
+          0.016580721363425255,
+          -0.02555933967232704,
+          -0.020694619044661522,
+          0.012333867140114307,
+          0.009111575782299042,
+          0.034788232296705246,
+          -0.012279119342565536,
+          -0.00044995738426223397,
+          -0.01496957615017891,
+          0.09547993540763855,
+          -0.01636173203587532,
+          0.026998421177268028,
+          -0.016048887744545937,
+          -0.02857828326523304,
+          0.00028815848054364324,
+          0.003701333422213793,
+          0.011223271489143372,
+          0.05559234693646431,
+          -0.00151631492190063,
+          0.03076818957924843,
+          0.012044486589729786,
+          0.0022075041197240353,
+          0.0029837479814887047,
+          -0.00746132479980588,
+          0.016205308958888054,
+          -0.006421119440346956,
+          0.02351021207869053,
+          0.0110746705904603,
+          0.012920449487864971,
+          -0.009784189984202385,
+          0.02335379086434841,
+          -0.0007136749336495996,
+          0.020725902169942856,
+          -0.023213010281324387,
+          0.04016914591193199,
+          0.00014676773571409285,
+          0.0036368092987686396,
+          -0.004501040559262037,
+          -0.025215212255716324,
+          -0.0044150082394480705,
+          0.011090313084423542,
+          -0.014914829283952713,
+          0.011317124590277672,
+          -0.019318105652928352,
+          0.05318344756960869,
+          -0.02793695218861103,
+          0.022164983674883842,
+          -0.005373092368245125,
+          -0.03603960573673248,
+          0.014406457543373108,
+          0.028187228366732597,
+          0.008016622625291348,
+          0.0028957605827599764,
+          -0.013467926532030106,
+          0.012599784880876541,
+          0.022305764257907867,
+          -0.02030356414616108,
+          2.63045440078713e-05,
+          0.025934752076864243,
+          -0.016877923160791397,
+          0.02307223156094551,
+          0.009674695320427418,
+          -0.02279067225754261,
+          -0.005322255194187164,
+          -0.006843458395451307,
+          0.009095934219658375,
+          -0.005658562295138836,
+          -0.008618847467005253,
+          -0.01038641482591629,
+          0.010456804186105728,
+          0.022759387269616127,
+          0.027702320367097855,
+          0.019271178171038628,
+          -0.01037859357893467,
+          -0.0015642191283404827,
+          0.0007234513177536428,
+          -0.014711480587720871,
+          -0.002647440880537033,
+          -0.012240014038980007,
+          -0.01819186843931675,
+          0.02682635746896267,
+          0.0035058059729635715,
+          -0.003580106422305107,
+          -0.015251136384904385,
+          -0.013413178734481335,
+          -0.02105438895523548,
+          -0.0006515950080938637,
+          -0.0029641950968652964,
+          -0.015696939080953598,
+          -0.003108885372057557,
+          0.015501410700380802,
+          -0.015337168239057064,
+          0.018410857766866684,
+          -0.02482415735721588,
+          0.013718200847506523,
+          -0.027326907962560654,
+          -0.0035234035458415747,
+          0.012599784880876541,
+          -0.012466826476156712,
+          -0.0032574862707406282,
+          -0.01884884014725685,
+          -0.011723821982741356,
+          0.010190887376666069,
+          -0.01836393214762211,
+          -0.007449593394994736,
+          0.007738973945379257,
+          0.005408287514001131,
+          -0.017972877249121666,
+          -0.027170484885573387,
+          -0.022118058055639267,
+          0.005865821614861488,
+          0.04070097953081131,
+          -0.02020971104502678,
+          0.00018428456678520888,
+          -0.007211050018668175,
+          0.006569720339030027,
+          0.0013706469908356667,
+          -0.013772948645055294,
+          -0.0007038986077532172,
+          0.0034354161471128464,
+          -0.004211660008877516,
+          0.024057690054178238,
+          0.006565809715539217,
+          0.023948194459080696,
+          0.005728952586650848,
+          0.0012005381286144257,
+          -0.02066333405673504,
+          -0.016565080732107162,
+          0.016752786934375763,
+          -0.010097034275531769,
+          0.014351709745824337,
+          0.02435489185154438,
+          -0.021429801359772682,
+          0.0012924360344186425,
+          0.01424221508204937,
+          0.014038866385817528,
+          -0.02663865126669407,
+          0.024949293583631516,
+          0.006421119440346956,
+          0.014414278790354729,
+          -0.0021195169538259506,
+          0.02325993776321411,
+          -0.007848469540476799,
+          0.033505573868751526,
+          -0.011504831723868847,
+          -0.00804008636623621,
+          0.0010754006216302514,
+          0.030189428478479385,
+          -0.006370282266288996,
+          -0.00020762563508469611,
+          0.019224252551794052,
+          0.043516576290130615,
+          0.0035664194729179144,
+          -0.0016551392618566751,
+          -0.013452284038066864,
+          0.010362951084971428,
+          0.008259076625108719,
+          0.021382873877882957,
+          0.00450495071709156,
+          0.001804717699997127,
+          -0.008243434131145477,
+          0.017597464844584465,
+          0.006671394221484661,
+          0.011512652039527893,
+          -0.026325806975364685,
+          0.0015642191283404827,
+          0.0070389858447015285,
+          0.010073570534586906,
+          0.002948553068563342,
+          -0.0026885014958679676,
+          0.020632049068808556,
+          -0.010769648477435112,
+          -0.0360083244740963,
+          0.024464385583996773,
+          0.002373702358454466,
+          0.024182826280593872,
+          -0.007054627873003483,
+          -0.015869002789258957,
+          0.02103874646127224,
+          0.002291580894961953,
+          0.006503240671008825,
+          0.025574982166290283,
+          0.0017206409247592092,
+          -0.00269827782176435,
+          0.005623367615044117,
+          0.01005792897194624,
+          0.0031049747485667467,
+          0.0009292439208365977,
+          -0.003840157762169838,
+          0.007257976569235325,
+          0.010448983870446682,
+          0.006080901715904474,
+          -0.002289625583216548,
+          -0.028328007087111473,
+          -0.0009043141617439687,
+          0.009768548421561718,
+          0.0063585503958165646,
+          -0.039825018495321274,
+          0.01497739739716053,
+          -0.006827816367149353,
+          0.004035685211420059,
+          0.0074378615245223045,
+          -0.008478067815303802,
+          -0.012083591893315315,
+          0.01245900522917509,
+          0.0018506667111068964,
+          -0.006898206193000078,
+          0.0043602604418993,
+          -0.03660272806882858,
+          -0.01745668426156044,
+          0.023275580257177353,
+          0.003564464161172509,
+          -0.021429801359772682,
+          0.014117076992988586,
+          -0.007707689423114061,
+          -0.010143960826098919,
+          0.005322255194187164,
+          0.02757718227803707,
+          -0.044705379754304886,
+          0.013694738037884235,
+          -0.02352585457265377,
+          -0.012552858330309391,
+          0.021617507562041283,
+          0.010003181174397469,
+          0.0041295383125543594,
+          -0.00101087661460042,
+          -0.011426620185375214,
+          0.005904926918447018,
+          0.012889165431261063,
+          -0.011895885691046715,
+          -0.020162783563137054,
+          0.0217739287763834,
+          0.012928270734846592,
+          0.005944032687693834,
+          -0.0074339513666927814,
+          0.0215862225741148,
+          -0.010644510388374329,
+          1.3480696452461416e-06,
+          0.009275819174945354,
+          -0.0454249233007431,
+          0.030721262097358704,
+          -0.0043328870087862015,
+          0.02360406517982483,
+          -0.008024443872272968,
+          -0.028328007087111473,
+          -0.008978617377579212,
+          0.01645558513700962,
+          0.018692417070269585,
+          0.005717220716178417,
+          0.0061982180923223495,
+          0.032019563019275665,
+          -0.0129595547914505,
+          0.013819875195622444,
+          0.0057993424125015736,
+          -0.00016289875202346593,
+          0.0261068157851696,
+          -0.016752786934375763,
+          -0.03130002319812775,
+          0.01428914163261652,
+          -0.012513753026723862,
+          -0.03663400933146477,
+          0.015337168239057064,
+          -0.02103874646127224,
+          -0.03397483751177788,
+          -0.029986079782247543,
+          -0.0035136269871145487,
+          -0.007344008423388004,
+          -0.0014830752043053508,
+          -0.003116706619039178,
+          0.016924850642681122,
+          0.023932551965117455,
+          0.012005381286144257,
+          0.01525895670056343,
+          -0.006511061917990446,
+          0.01419528853148222,
+          -0.00397116132080555,
+          0.033224012702703476,
+          0.0012435541721060872,
+          0.004325065761804581,
+          0.001327630947344005,
+          0.017957234755158424,
+          0.013389714993536472,
+          0.020084572955965996,
+          0.009612126275897026,
+          -0.015368452295660973,
+          -0.01900526136159897,
+          -0.046738866716623306,
+          0.009760727174580097,
+          -0.004692657385021448,
+          0.006256876513361931,
+          0.03813566267490387,
+          -0.013186367228627205,
+          -0.010816575028002262,
+          0.009228892624378204,
+          0.004966395441442728,
+          0.0032437993213534355,
+          -0.013108155690133572,
+          -0.006100454367697239,
+          -0.0031499462202191353,
+          -0.02233704924583435,
+          -0.0253559909760952,
+          -0.018442142754793167,
+          0.020819755271077156,
+          -0.0017597464611753821,
+          0.013108155690133572,
+          -0.02077282965183258,
+          -0.0031577672343701124,
+          -0.028093375265598297,
+          -0.011191987432539463,
+          0.0013872667914256454,
+          0.013084692880511284,
+          0.004821705166250467,
+          0.016971776261925697,
+          -0.005330076441168785,
+          -0.020256636664271355,
+          -0.017800813540816307,
+          0.012075771577656269,
+          -0.010761827230453491,
+          -0.005947942845523357,
+          -0.004978127311915159,
+          0.01938067376613617,
+          0.007954053580760956,
+          0.004278139211237431,
+          -0.001422461704351008,
+          0.021899066865444183,
+          -0.021085673943161964,
+          -0.015955034643411636,
+          0.01747232675552368,
+          0.03262960910797119,
+          -0.03170672059059143,
+          -0.005552977789193392,
+          0.0009018700802698731,
+          0.014688016846776009,
+          0.015861181542277336,
+          0.012693637982010841,
+          -0.006557988468557596,
+          -0.008055727928876877,
+          -0.028046447783708572,
+          -0.01240425743162632,
+          -0.04095125570893288,
+          0.0029114028438925743,
+          0.029720162972807884,
+          -0.005025053862482309,
+          0.01911475695669651,
+          0.017988519743084908,
+          -0.027170484885573387,
+          6.660151848336682e-05,
+          0.004106075037270784,
+          0.017628749832510948,
+          -0.033849701285362244,
+          0.0145628796890378,
+          -0.009361851029098034,
+          0.02169571816921234,
+          0.0005733840516768396,
+          -0.01909911446273327,
+          -0.017753886058926582,
+          -0.011888065375387669,
+          0.021742645651102066,
+          -0.00253990083001554,
+          -0.029751446098089218,
+          0.021633150056004524,
+          0.009565199725329876,
+          -0.007136749569326639,
+          -0.024949293583631516,
+          0.012013202533125877,
+          0.0129595547914505,
+          0.017613107338547707,
+          -0.01074618473649025,
+          -0.002436271170154214,
+          -0.012568499892950058,
+          -0.010214350186288357,
+          -0.022211911156773567,
+          -0.00043211551383137703,
+          -0.017628749832510948,
+          0.0015945257619023323,
+          -0.011512652039527893,
+          -0.0022778939455747604,
+          0.014351709745824337,
+          -0.041483089327812195,
+          -0.0018457785481587052,
+          -0.008955154567956924,
+          0.028531355783343315,
+          0.02473030425608158,
+          0.008478067815303802,
+          0.0027960415463894606,
+          -0.02113259956240654,
+          -0.010816575028002262,
+          -0.011786391027271748,
+          0.01820751093327999,
+          -0.02261860854923725,
+          0.0003284859994892031,
+          0.010018822737038136,
+          0.008611026220023632,
+          0.0003871442168019712,
+          0.005114996340125799,
+          -0.002453868743032217,
+          -0.02094489336013794,
+          0.014641090296208858,
+          -0.0005171699449419975,
+          0.007003790698945522,
+          -0.012138339690864086,
+          0.0032262017484754324,
+          -0.0036778701469302177,
+          0.025668835267424583,
+          -0.014774048700928688,
+          -0.004387634340673685,
+          -0.010355129837989807,
+          0.0013110111467540264,
+          0.0070937336422502995,
+          -1.234266619576374e-05,
+          -0.0006423074519261718,
+          -0.010777469724416733,
+          0.01240425743162632,
+          -0.011160703375935555,
+          0.030439702793955803,
+          9.531959949526936e-05,
+          -0.03172236308455467,
+          -0.0036074803210794926,
+          -0.03766639530658722,
+          -0.007168034091591835,
+          -0.017800813540816307,
+          -0.033943552523851395,
+          -0.007430040743201971,
+          0.0017089092871174216,
+          -0.01493047084659338,
+          0.016283521428704262,
+          0.004559698514640331,
+          -0.02473030425608158,
+          0.013905907981097698,
+          0.006831726990640163,
+          -0.017894666641950607,
+          -0.0038440683856606483,
+          -0.01920861005783081,
+          0.0049116481095552444,
+          0.005142370238900185,
+          0.009987538680434227,
+          0.018348289653658867,
+          -0.007394845597445965,
+          -0.013022123835980892,
+          0.003024808596819639,
+          0.02096053585410118,
+          0.01483661774545908,
+          0.010824396274983883,
+          0.0068356371484696865,
+          0.00377758895047009,
+          -0.025043146684765816,
+          -0.003355249995365739,
+          -0.008149581030011177,
+          -0.0070389858447015285,
+          -0.0009810586925595999,
+          0.0034295502118766308,
+          -0.008337287232279778,
+          -0.0006525726639665663,
+          0.000982525059953332,
+          0.0018956379499286413,
+          0.0030072112567722797,
+          -0.013546137139201164,
+          0.0012865702155977488,
+          -0.008517173118889332,
+          0.002373702358454466,
+          0.024026405066251755,
+          -0.012185266241431236,
+          0.008650131523609161,
+          -0.0027510703075677156,
+          0.00634290836751461,
+          0.038636211305856705,
+          0.015352809801697731,
+          -0.044423822313547134,
+          0.003670048899948597,
+          0.007926680147647858,
+          0.0008823173120617867,
+          -0.017972877249121666,
+          0.017675675451755524,
+          0.004837347660213709,
+          -0.008650131523609161,
+          -0.002266162307932973,
+          -0.007844558916985989,
+          0.009385314770042896,
+          -0.015251136384904385,
+          0.012302583083510399,
+          0.013420999981462955,
+          -0.01438299473375082,
+          -0.004923379514366388,
+          0.038730062544345856,
+          -0.010941712185740471,
+          -0.011285840533673763,
+          0.0009121352341026068,
+          0.013319325633347034,
+          0.011262377724051476,
+          -0.004798241890966892,
+          0.014500310644507408,
+          0.0037365283351391554,
+          0.006186486221849918,
+          0.007218870799988508,
+          0.0597531683743,
+          0.0067769791930913925,
+          0.008251255378127098,
+          0.011364051140844822,
+          -0.007445682771503925,
+          -0.014500310644507408,
+          -0.018958335742354393,
+          -0.026357091963291168,
+          -0.004207749385386705,
+          0.01590810716152191,
+          -0.018270079046487808,
+          -0.0032653072848916054,
+          -0.02252475544810295,
+          0.01249811053276062,
+          0.013061229139566422,
+          0.005709399469196796,
+          -0.01717512495815754,
+          0.02066333405673504,
+          0.02793695218861103,
+          0.010308203287422657,
+          -0.0016746920300647616,
+          0.017034346237778664,
+          0.005036785267293453,
+          -0.015548337250947952,
+          -0.0023404627572745085,
+          -0.0023717472795397043,
+          0.004720031283795834,
+          0.021382873877882957,
+          -0.008251255378127098,
+          0.01221655122935772,
+          -0.0045166825875639915,
+          0.009221071377396584,
+          0.002291580894961953,
+          -0.013115976937115192,
+          -0.008306003175675869,
+          -0.0008432118338532746,
+          0.01792594976723194,
+          0.00013894664880353957,
+          -0.025011863559484482,
+          0.0030365402344614267,
+          0.0045948936603963375,
+          0.002862520981580019,
+          0.003812784096226096,
+          -0.014727122150361538,
+          -0.007938412018120289,
+          0.021914709359407425,
+          -0.016111455857753754,
+          0.0032066490966826677,
+          -0.012568499892950058,
+          0.0011369917774572968,
+          -0.01285788044333458,
+          0.0006750582833774388,
+          0.00397116132080555,
+          0.016612006351351738,
+          -0.027608467265963554,
+          -0.015352809801697731,
+          0.0217739287763834,
+          0.00836075097322464,
+          0.009049007669091225,
+          -0.0027256517205387354,
+          -0.005658562295138836,
+          0.016424300149083138,
+          -0.01783209666609764,
+          -0.013827696442604065,
+          0.005904926918447018,
+          -0.008024443872272968,
+          0.0052323127165436745,
+          -0.02296273596584797,
+          0.007969696074724197,
+          -0.008071370422840118,
+          -0.008165223523974419,
+          0.008212150074541569,
+          0.015829896554350853,
+          0.003789320820942521,
+          0.007918858900666237,
+          -0.0008104610024020076,
+          -0.01801980286836624,
+          -0.029516814276576042,
+          0.027311265468597412,
+          0.0043915449641644955,
+          -0.00404350645840168,
+          0.004219480790197849,
+          -0.0038049628492444754,
+          -0.004888184834271669,
+          -0.0021449353080242872,
+          -0.014641090296208858,
+          -0.02271246165037155,
+          0.013647811487317085,
+          0.012247835285961628,
+          -0.012294761836528778,
+          0.021289020776748657,
+          -0.008532814681529999,
+          0.007715510670095682,
+          -0.01791030913591385,
+          0.00850153062492609,
+          0.028703419491648674,
+          -0.006823905743658543,
+          -0.014359530992805958,
+          -0.0016248325118795037,
+          -0.008000980131328106,
+          0.0035370904952287674,
+          0.027264337986707687,
+          0.0062451446428895,
+          0.014531595632433891,
+          -0.01728462055325508,
+          -0.011176344938576221,
+          0.02243090234696865,
+          0.004434560891240835,
+          0.011207629926502705,
+          -0.0010001225164160132,
+          -0.009095934219658375,
+          -0.012099234387278557,
+          -0.00790321733802557,
+          0.010581942275166512,
+          0.012326045893132687,
+          -0.013288040645420551,
+          -0.0005117929540574551,
+          -0.007871932350099087,
+          0.005588172469288111,
+          -0.012505931779742241,
+          -0.011434441432356834,
+          -0.022008562460541725,
+          -0.008509351871907711,
+          -0.015532695688307285,
+          -0.026513513177633286,
+          -0.007168034091591835,
+          -0.0008265920332632959,
+          -0.012021023780107498,
+          -0.0369468554854393,
+          -0.022477827966213226,
+          -0.007547357119619846,
+          -0.019740445539355278,
+          0.008532814681529999,
+          0.0005748505354858935,
+          -0.021070031449198723,
+          0.006831726990640163,
+          -0.012349509634077549,
+          -0.009354029782116413,
+          0.00251448224298656,
+          0.01244336273521185,
+          -0.010065749287605286,
+          -0.014774048700928688,
+          -0.01240425743162632,
+          -0.01994379237294197,
+          0.0146567327901721,
+          -0.0027667125687003136,
+          0.003470611060038209,
+          -0.006299892440438271,
+          -0.004297691863030195,
+          -0.015454484149813652,
+          0.01994379237294197,
+          -0.02473030425608158,
+          0.0013139439979568124,
+          -0.0015867047477513552,
+          0.006929490715265274,
+          -0.003302457509562373,
+          -0.01745668426156044,
+          -0.009088112972676754,
+          -0.0010773559333756566,
+          -0.015313704498112202,
+          -0.02003764547407627,
+          -0.013014302588999271,
+          -0.005795431789010763,
+          0.015337168239057064,
+          -0.008564099669456482,
+          0.01273274328559637,
+          0.007922769524157047,
+          0.0035194929223507643,
+          -0.004422829486429691,
+          0.0035527325235307217,
+          -0.010738363489508629,
+          -0.0034803873859345913,
+          0.0017558359540998936,
+          0.005474766716361046,
+          0.006022243294864893,
+          0.0014263722114264965,
+          0.015939392149448395,
+          -0.009080291725695133,
+          0.012013202533125877,
+          0.020287921652197838,
+          0.02150801196694374,
+          0.006194307468831539,
+          0.00629207119345665,
+          0.004141270183026791,
+          -9.611393033992499e-05,
+          0.002846878720447421,
+          0.0023873893078416586,
+          -0.010980817489326,
+          0.012185266241431236,
+          -0.015587443485856056,
+          -0.008603204973042011,
+          0.013890265487134457,
+          -0.002537945518270135,
+          -0.015603085048496723,
+          0.015923749655485153,
+          0.0016746920300647616,
+          -0.02369791828095913,
+          -0.01062104757875204,
+          -0.007156302221119404,
+          0.004262497182935476,
+          0.020553838461637497,
+          0.007293171249330044,
+          -0.027811815962195396,
+          0.01708127185702324,
+          -0.0002189906663261354,
+          -0.013147260993719101,
+          -0.009103755466639996,
+          0.01837957464158535,
+          0.024042047560214996,
+          -0.0006647931295447052,
+          -0.04633216932415962,
+          -0.014226572588086128,
+          0.00808701291680336,
+          -0.026685576885938644,
+          -0.0285626407712698,
+          0.011684716679155827,
+          -0.00808701291680336,
+          -0.031565941870212555,
+          -0.005279239267110825,
+          -0.0007855312433093786,
+          -0.03078383207321167,
+          0.009494809433817863,
+          -0.0008705856744199991,
+          0.0032868152484297752,
+          0.007770258467644453,
+          -0.013647811487317085,
+          -0.00544348219409585,
+          -0.02471466176211834,
+          0.012286940589547157,
+          -0.007035075221210718,
+          -0.0038597106467932463,
+          -0.0036368092987686396,
+          0.009815474972128868,
+          -0.002762801945209503,
+          0.02279067225754261,
+          0.026701219379901886,
+          0.01966223306953907,
+          -0.003132348647341132,
+          -0.023588424548506737,
+          -0.009377493523061275,
+          0.002107785316184163,
+          0.012889165431261063,
+          0.006831726990640163,
+          -0.017018703743815422,
+          0.005678115412592888,
+          0.0014605895848944783,
+          0.013436641544103622,
+          0.000748869904782623,
+          0.02609117329120636,
+          -3.1039973691804335e-05,
+          0.002481242408975959,
+          0.002082366729155183,
+          -0.02580961398780346,
+          0.004680925514549017,
+          -0.0058110738173127174,
+          0.022477827966213226,
+          0.016784070059657097,
+          -0.0003084444615524262,
+          0.007011611945927143,
+          -0.0127718485891819,
+          -0.004727852065116167,
+          -0.028234153985977173,
+          -0.01709691435098648,
+          -0.00015947702922858298,
+          0.031002821400761604,
+          0.033130161464214325,
+          -0.02673250436782837,
+          -0.010136139579117298,
+          -0.0009385314187966287,
+          0.029735805466771126,
+          -0.006022243294864893,
+          -0.00218013022094965,
+          0.029000621289014816,
+          0.004485398065298796,
+          0.0030717351473867893,
+          0.012200908735394478,
+          0.0012767937732860446,
+          0.008861301466822624,
+          0.033568140119314194,
+          0.015915928408503532,
+          0.009627768769860268,
+          0.008861301466822624,
+          -0.002152756555005908,
+          -0.015837717801332474,
+          -0.021992919966578484,
+          -0.0020589034538716078,
+          -0.00652279332280159,
+          0.009486989118158817,
+          -0.018144940957427025,
+          0.006022243294864893,
+          -0.011786391027271748,
+          0.004125627689063549,
+          -0.011817675083875656,
+          -0.0022876705043017864,
+          0.0017450818559154868,
+          -0.01700306124985218,
+          -0.0025985590182244778,
+          -0.024980578571558,
+          -0.0027510703075677156,
+          0.0108869643881917,
+          -0.001176097197458148,
+          0.016127098351716995,
+          -0.016471227630972862,
+          0.0051541016437113285,
+          0.019177325069904327,
+          0.021289020776748657,
+          -0.01474276464432478,
+          6.71514353598468e-05,
+          0.003977026790380478,
+          -0.025981679558753967,
+          0.01579861342906952,
+          0.005158012267202139,
+          0.010175244882702827,
+          0.025684477761387825,
+          -0.0009443972376175225,
+          -0.0031831858213990927,
+          -0.012826596386730671,
+          -0.004532324615865946,
+          -0.03491336852312088,
+          0.024182826280593872,
+          -0.00034339496050961316,
+          0.009111575782299042,
+          0.004931200761348009,
+          -0.009627768769860268,
+          0.007574731018394232,
+          -0.010808753781020641,
+          0.019834298640489578,
+          0.005904926918447018,
+          0.003060003509745002,
+          -0.01492264959961176,
+          0.01653379574418068,
+          0.0025731404311954975,
+          -0.01894269324839115,
+          0.02463645115494728,
+          -0.01055847853422165,
+          0.0044932193122804165,
+          -0.0030580481979995966,
+          0.011160703375935555,
+          0.009690336883068085,
+          -0.006968596018850803,
+          0.004149090964347124,
+          -0.0296888779848814,
+          -0.02307223156094551,
+          -0.013037766329944134,
+          -0.0011389469727873802,
+          -0.007742884568870068,
+          -0.026435302570462227,
+          -0.022493470460176468,
+          0.004743494559079409,
+          -0.005294881761074066,
+          0.004688746761530638,
+          0.012208729982376099,
+          0.012654532678425312,
+          0.005728952586650848,
+          0.006331176497042179,
+          -0.009354029782116413,
+          0.006565809715539217,
+          0.026904568076133728,
+          -0.006565809715539217,
+          -0.03197263926267624,
+          -0.006933400873094797,
+          0.003554687835276127,
+          -0.018582923337817192,
+          -0.009948433376848698,
+          0.009486989118158817,
+          -0.004027863964438438,
+          0.0288442000746727,
+          -0.02380741387605667,
+          0.0038460236974060535,
+          -0.0038831739220768213,
+          0.004950753413140774,
+          -0.004806063137948513,
+          1.4840223229839467e-05,
+          -0.01911475695669651,
+          0.00017340834892820567,
+          0.004946842789649963,
+          -0.016846640035510063,
+          0.015696939080953598,
+          -0.007230602670460939,
+          -0.011019923724234104,
+          -0.007699868176132441,
+          0.0012846149038523436,
+          -0.007211050018668175,
+          -0.001168276066891849,
+          0.02627888135612011,
+          -0.007973606698215008,
+          -0.017394116148352623,
+          -0.004547967109829187,
+          0.0053574503399431705,
+          0.015149462036788464,
+          0.011473546735942364,
+          0.024057690054178238,
+          -0.013389714993536472,
+          0.003515582298859954,
+          -0.004059148486703634,
+          -0.0031734094955027103,
+          -4.744594116345979e-05,
+          0.039105478674173355,
+          -0.011434441432356834,
+          -0.0004949286812916398,
+          -0.00844678282737732,
+          0.009479167871177197,
+          0.006495419889688492,
+          0.020616406574845314,
+          0.006346818991005421,
+          0.002668948844075203,
+          0.007160212844610214,
+          -0.0006931445677764714,
+          0.0006760359392501414,
+          0.01460198499262333,
+          0.011184166185557842,
+          0.00723842391744256,
+          -0.0072814398445189,
+          -0.023025304079055786,
+          0.014234393835067749,
+          -0.0010841992916539311,
+          -0.0031675435602664948,
+          0.010261276736855507,
+          0.01212269812822342,
+          -0.009346208535134792,
+          0.012435541488230228,
+          -0.007703778799623251,
+          0.024761587381362915,
+          0.013702559284865856,
+          -0.01005792897194624,
+          0.009346208535134792,
+          0.021633150056004524,
+          -0.00032139813993126154,
+          0.015235493890941143,
+          -0.015423200093209743,
+          -0.021945994347333908,
+          0.0012210685526952147,
+          0.021539296954870224,
+          0.0005015277420170605,
+          -0.015344989486038685,
+          0.009322745725512505,
+          -0.007856289856135845,
+          -0.007805453147739172,
+          0.017847739160060883,
+          0.0125841423869133,
+          -0.009463525377213955,
+          -0.009682515636086464,
+          0.0010314069222658873,
+          0.004872542340308428,
+          0.012943913228809834,
+          0.02066333405673504,
+          -0.003447147784754634,
+          0.004239033907651901,
+          -0.00251448224298656,
+          0.03860492631793022,
+          -0.005748505238443613,
+          0.03125309571623802,
+          -0.015642190352082253,
+          -0.011528294533491135,
+          -0.009205428883433342,
+          -0.0063389977440238,
+          -0.005509961862117052,
+          -0.0024421371053904295,
+          0.0012347553856670856,
+          -0.005482587963342667,
+          -0.023103516548871994,
+          0.012717100791633129,
+          0.022634249180555344,
+          0.002792131155729294,
+          0.007031164597719908,
+          0.005013321992009878,
+          0.013710380531847477,
+          0.030924610793590546,
+          -0.005615546368062496,
+          -0.0147505858913064,
+          0.009002081118524075,
+          0.010034465231001377,
+          -0.016424300149083138,
+          0.021179527044296265,
+          0.006765247322618961,
+          0.006710499990731478,
+          -0.014320425689220428,
+          -0.0017147751059383154,
+          0.02022535353899002,
+          0.0071758548729121685,
+          0.009792011231184006,
+          -0.012294761836528778,
+          -0.0025966037064790726,
+          0.0012679951032623649,
+          0.0072814398445189,
+          -0.0034764769952744246,
+          -0.015008682385087013,
+          0.0159784983843565,
+          -0.014390815049409866,
+          0.02261860854923725,
+          -0.021335948258638382,
+          0.012513753026723862,
+          -0.009252355434000492,
+          0.020694619044661522,
+          -0.0011545892339199781,
+          -0.007727242074906826,
+          -0.00534962909296155,
+          0.013147260993719101,
+          -0.005955764092504978,
+          -0.00256140879355371,
+          -0.02250911295413971,
+          -0.013499210588634014,
+          0.017879024147987366,
+          -0.0216800756752491,
+          -0.0012826595921069384,
+          0.0019005262292921543,
+          0.010714900679886341,
+          -0.028687776997685432,
+          0.022821955382823944,
+          0.031174886971712112,
+          0.012294761836528778,
+          0.012474646791815758,
+          -0.0045166825875639915,
+          -0.016674574464559555,
+          -0.009432241320610046,
+          0.03344300389289856,
+          0.008345108479261398,
+          -0.01801980286836624,
+          0.006952953990548849,
+          0.01830136403441429,
+          -0.009119397029280663,
+          0.004661372862756252,
+          -0.0017685452476143837,
+          0.016064530238509178,
+          0.0020178426057100296,
+          0.0035898827482014894,
+          -0.01673714444041252,
+          -0.0365714430809021,
+          -0.012302583083510399,
+          0.02060076594352722,
+          -0.005666383542120457,
+          -0.008861301466822624,
+          0.009033365175127983,
+          0.0020002450328320265,
+          -0.003146035596728325,
+          -0.011387514881789684,
+          0.003810828784480691,
+          0.0074339513666927814,
+          0.006080901715904474,
+          -0.01967787556350231,
+          0.012021023780107498,
+          -0.009361851029098034,
+          -0.010276919230818748,
+          0.01075400598347187,
+          -0.013858981430530548,
+          -0.026216311380267143,
+          -0.0022172804456204176,
+          -0.015563979744911194,
+          -0.015689117833971977,
+          -0.0144690265879035,
+          0.006816084496676922,
+          -0.0012279120273888111,
+          -0.017347188666462898,
+          -0.028593923896551132,
+          0.014077971689403057,
+          0.0010186976287513971,
+          0.028265438973903656,
+          -0.029078831896185875,
+          0.022649891674518585,
+          0.00035341575858183205,
+          0.012920449487864971,
+          0.0017216185806319118,
+          -0.011848959140479565,
+          0.0019220341928303242,
+          0.005021143238991499,
+          0.003394355298951268,
+          0.001639497117139399,
+          -0.010456804186105728,
+          -0.0010890875710174441,
+          -0.018160583451390266,
+          0.006773068569600582,
+          -0.005236223340034485,
+          -0.014187467284500599,
+          -0.0026415749453008175,
+          0.015431021340191364,
+          0.00606134906411171,
+          0.022290121763944626,
+          -0.019599664956331253,
+          -0.002825370756909251,
+          -0.007817184552550316,
+          -0.012161803431808949,
+          -0.006788710597902536,
+          0.02031920664012432,
+          -0.007222781423479319,
+          0.03363071009516716,
+          0.008923869580030441,
+          -0.01024563517421484,
+          -0.011450083926320076,
+          0.0012562634656205773,
+          0.02066333405673504,
+          0.018504710868000984,
+          -0.02333814837038517,
+          -0.012114876881241798,
+          -0.0013892221031710505,
+          0.013311504386365414,
+          0.006690946873277426,
+          -0.0016717591788619757,
+          0.01736283116042614,
+          0.0066166468895971775,
+          -0.01679971255362034,
+          -0.01254503708332777,
+          -0.002807773184031248,
+          0.015650011599063873,
+          0.013217651285231113,
+          0.013960654847323895,
+          0.005337897688150406,
+          0.013248935341835022,
+          0.01902090385556221,
+          -0.01745668426156044,
+          -0.005369181744754314,
+          0.019255537539720535,
+          0.0029270448721945286,
+          -0.017879024147987366,
+          0.0028195048216730356,
+          0.0023033125326037407,
+          0.00807919166982174,
+          0.003292681183665991,
+          -0.004196017514914274,
+          0.03155029937624931,
+          -0.0009263110114261508,
+          0.005025053862482309,
+          0.018051087856292725,
+          -0.006655752193182707,
+          0.022305764257907867,
+          -0.0017421490047127008,
+          0.028906768187880516,
+          -0.02003764547407627,
+          0.003496029647067189,
+          0.0004572896577883512,
+          -0.0027979968581348658,
+          -0.0026357092428952456,
+          0.0016150561859831214,
+          -0.018317004665732384,
+          0.009275819174945354,
+          -0.04567519575357437,
+          0.008345108479261398,
+          -0.007602104917168617,
+          -0.0021410249173641205,
+          -0.0248867254704237,
+          -0.01015960332006216,
+          -0.009150682017207146,
+          -0.010581942275166512,
+          -0.018786270171403885,
+          -0.021899066865444183,
+          -0.00789930671453476,
+          0.004751315340399742,
+          0.01470365934073925,
+          -0.019896866753697395,
+          0.01939631626009941,
+          0.007070270366966724,
+          -0.026982778683304787,
+          -0.016877923160791397,
+          -0.019959434866905212,
+          -0.0010538926580920815,
+          -0.010065749287605286,
+          0.0028781630098819733,
+          -0.027952594682574272,
+          0.022024204954504967,
+          0.016674574464559555,
+          -0.022258836776018143,
+          0.0058580003678798676,
+          -0.01428914163261652,
+          -0.023776130750775337,
+          0.0017499701352789998,
+          0.0006931445677764714,
+          -0.00864231027662754,
+          0.04273446649312973,
+          0.021101314574480057,
+          8.737629832467064e-05,
+          -0.028640851378440857,
+          0.005728952586650848,
+          -0.02727998048067093,
+          0.013921549543738365,
+          0.004747404716908932,
+          0.00015752174658700824,
+          -0.0045166825875639915,
+          0.016236593946814537,
+          0.0019738490227609873,
+          0.01698741875588894,
+          0.006057438440620899,
+          0.0007400711765512824,
+          0.02473030425608158,
+          -0.011301483027637005,
+          0.006792621221393347,
+          -8.447394066024572e-06,
+          -0.0005313456640578806,
+          0.015696939080953598,
+          -0.004966395441442728,
+          -0.002283759880810976,
+          0.012279119342565536,
+          0.000982525059953332,
+          -0.009721621870994568,
+          -0.004371992312371731,
+          -0.0030424061696976423,
+          0.009135039523243904,
+          -0.015493590384721756,
+          0.01001100242137909,
+          0.0026259326841682196,
+          0.01928682066500187,
+          -0.018254436552524567,
+          -0.005803252570331097,
+          0.002670904155820608,
+          -0.009940612129867077,
+          -0.004606625065207481,
+          -0.015032145194709301,
+          -0.009338388219475746,
+          -0.02020971104502678,
+          0.01617402583360672,
+          0.04320373013615608,
+          0.02252475544810295,
+          0.000653550261631608,
+          -0.004055237863212824,
+          -0.01433606818318367,
+          0.003934010863304138,
+          -0.003478432074189186,
+          -0.011927170678973198,
+          -0.022540396079421043,
+          -0.005189296789467335,
+          0.012036665342748165,
+          -0.004078701138496399,
+          0.0019132354063913226,
+          -0.011653431691229343,
+          0.021101314574480057,
+          -0.01273274328559637,
+          -0.0004919957718811929,
+          -0.018723702058196068,
+          -0.015470126643776894,
+          0.009393135085701942,
+          0.032848600298166275,
+          -0.0023326417431235313,
+          0.013420999981462955,
+          0.013022123835980892,
+          0.004023953340947628,
+          -0.01478969119489193,
+          0.010675795376300812,
+          -0.006010511890053749,
+          0.004106075037270784,
+          -0.0025477218441665173,
+          -0.02380741387605667,
+          -0.009267997927963734,
+          -0.011176344938576221,
+          0.006909937597811222,
+          -0.009768548421561718,
+          -0.0006555055733770132,
+          -0.0015495545230805874,
+          -0.009909328073263168,
+          -0.018160583451390266,
+          0.007586462423205376,
+          -0.007774168625473976,
+          0.0032437993213534355,
+          0.00014090191689319909,
+          -0.0018741299863904715,
+          0.0087205208837986,
+          0.02020971104502678,
+          -0.0067535159178078175,
+          0.0030130769591778517,
+          0.004931200761348009,
+          -0.016846640035510063,
+          0.016393015161156654,
+          0.0019533184822648764,
+          -0.0006476844428107142,
+          0.03237933665513992,
+          0.016236593946814537,
+          0.012795312330126762,
+          -0.0017841873923316598,
+          -0.00305413780733943,
+          -0.001330563798546791,
+          0.014672375284135342,
+          -0.03388098627328873,
+          -0.009111575782299042,
+          0.014985218644142151,
+          0.02067897655069828,
+          -0.015829896554350853,
+          0.0127718485891819,
+          -0.015032145194709301,
+          -0.009690336883068085,
+          -0.007297081872820854,
+          0.00303262984380126,
+          -0.05340243875980377,
+          0.011043386533856392,
+          -0.006366371642798185,
+          0.02712355926632881,
+          0.0006178665207698941,
+          0.0108869643881917,
+          0.010120497085154057,
+          -0.006100454367697239,
+          0.025606267154216766,
+          0.0015045831678435206,
+          -0.0041764648631215096,
+          0.012850060127675533,
+          -0.011778569780290127,
+          -0.011661252938210964,
+          0.008564099669456482,
+          -0.0288442000746727,
+          0.008994259871542454,
+          -0.024933652952313423,
+          0.0022270570043474436,
+          -0.016220951452851295,
+          -0.012396436184644699,
+          0.022649891674518585,
+          0.03287988528609276,
+          0.00202175322920084,
+          0.0022329227067530155,
+          -0.010816575028002262,
+          0.025700120255351067,
+          -0.010550657287240028,
+          0.019443243741989136,
+          -0.008876943029463291,
+          -0.011207629926502705,
+          -0.017112556844949722,
+          -0.010808753781020641,
+          0.008783089928328991,
+          -0.003986803349107504,
+          -0.004375902935862541,
+          -0.021461086347699165,
+          0.005287060514092445,
+          -0.022211911156773567,
+          0.008837837725877762,
+          0.002567274495959282,
+          -0.006518883164972067,
+          0.001965050119906664,
+          0.0029876583721488714,
+          -0.0010167424334213138,
+          -0.00016729811613913625,
+          -0.024088973179459572,
+          -0.008743984624743462,
+          -0.008118296973407269,
+          0.009049007669091225,
+          0.001364781172014773,
+          -0.020538195967674255,
+          0.010581942275166512,
+          -0.009095934219658375,
+          -0.0036153013352304697,
+          -0.008657952770590782,
+          0.0028038627933710814,
+          0.0013872667914256454,
+          0.02067897655069828,
+          0.003855800023302436,
+          -0.020194068551063538,
+          0.009494809433817863,
+          0.017394116148352623,
+          -0.00253990083001554,
+          0.006331176497042179,
+          -0.00019063919899053872,
+          -0.01028474047780037,
+          -0.010589763522148132,
+          -0.0012865702155977488,
+          0.006299892440438271,
+          -0.009831116534769535,
+          0.005744594614952803,
+          0.006992059294134378,
+          0.0058345370925962925,
+          0.021070031449198723,
+          0.008900406770408154,
+          0.004242944065481424,
+          -0.0015603085048496723,
+          0.006733963266015053,
+          -0.0011614327086135745,
+          -0.0014928515302017331,
+          -0.012748385779559612,
+          0.009862401522696018,
+          -0.019630949944257736,
+          0.009651231579482555,
+          0.017675675451755524,
+          0.008986438624560833,
+          -0.016971776261925697,
+          -0.01755053736269474,
+          -0.007363561075180769,
+          -0.023478928953409195,
+          0.009095934219658375,
+          0.0007459369953721762,
+          0.018535995855927467,
+          -0.0046535516157746315,
+          0.008196507580578327,
+          0.006147380918264389,
+          -0.013804233632981777,
+          -0.0020549928303807974,
+          0.009502630680799484,
+          -0.029657592996954918,
+          -0.012521573342382908,
+          -0.000881339656189084,
+          -0.0010548701975494623,
+          0.018176225945353508,
+          0.01919296756386757,
+          0.019146041944622993,
+          0.03206649050116539,
+          -0.009705979377031326,
+          0.01023781392723322,
+          -0.0028371023945510387,
+          0.018864482641220093,
+          0.010042286477982998,
+          0.00733227701857686,
+          0.020632049068808556,
+          0.006972506642341614,
+          -0.02103874646127224,
+          -0.010832217521965504,
+          0.019818656146526337,
+          -0.010417698882520199,
+          -0.005959674715995789,
+          -0.012552858330309391,
+          0.0037756338715553284,
+          0.0033904449082911015,
+          0.011958454735577106,
+          0.018708059564232826,
+          0.008884764276444912,
+          -0.017315905541181564,
+          0.032754749059677124,
+          0.0007058538612909615,
+          0.00647977739572525,
+          0.004199928138405085,
+          -0.003887084312736988,
+          -0.005502140615135431,
+          0.013968476094305515,
+          0.012466826476156712,
+          -0.0014938291860744357,
+          0.0034745216835290194,
+          0.012623247690498829,
+          -0.01447684783488512,
+          -0.018598563969135284,
+          0.007527804467827082,
+          0.0017558359540998936,
+          0.00037883431650698185,
+          0.006589272990822792,
+          0.0025496771559119225,
+          0.017613107338547707,
+          -0.006831726990640163,
+          0.028187228366732597,
+          -0.005521693266928196,
+          -0.016142740845680237,
+          -0.013139440678060055,
+          0.0011555668897926807,
+          -0.011903706938028336,
+          -0.018144940957427025,
+          0.007504341192543507,
+          -0.0015104489866644144,
+          0.011934991925954819,
+          -0.002289625583216548,
+          0.004250765312463045,
+          0.013381893746554852,
+          0.017706960439682007,
+          0.014492489397525787,
+          0.002645485568791628,
+          -0.004508861340582371,
+          -0.015438842587172985,
+          0.008032265119254589,
+          -0.009698158130049706,
+          0.02132030576467514,
+          -0.014085792936384678,
+          -0.021977277472615242,
+          0.0031362592708319426,
+          0.01474276464432478,
+          -0.0020589034538716078,
+          0.0001868508552433923,
+          0.012748385779559612,
+          -0.0062451446428895,
+          0.00031675436184741557,
+          0.012568499892950058,
+          0.00445802416652441,
+          -0.02810901589691639,
+          -0.009612126275897026,
+          0.007273618597537279,
+          -0.0021625328809022903,
+          -0.003701333422213793,
+          -0.02388562448322773,
+          -0.004430650733411312,
+          -0.011324945837259293,
+          -0.005944032687693834,
+          -0.004371992312371731,
+          -8.847613935358822e-05,
+          0.009361851029098034,
+          -0.02269681915640831,
+          0.014461205340921879,
+          -0.013327146880328655,
+          -0.004555787891149521,
+          -0.017660032957792282,
+          0.022290121763944626,
+          0.00530661316588521,
+          0.0007527804118581116,
+          0.00507589103654027,
+          0.0012396436650305986,
+          0.008665774017572403,
+          0.01452377438545227,
+          -0.002537945518270135,
+          0.004778689239174128,
+          0.015665654093027115,
+          0.005345718469470739,
+          0.005760236643254757,
+          0.01938067376613617,
+          -0.003339607734233141,
+          -0.017769528552889824,
+          -0.003564464161172509,
+          0.016424300149083138,
+          0.011231092736124992,
+          0.015493590384721756,
+          0.004645730834454298,
+          0.01033166702836752,
+          -0.003564464161172509,
+          -0.006909937597811222,
+          0.018129298463463783,
+          0.0125841423869133,
+          -0.016205308958888054,
+          -0.011332767084240913,
+          0.012294761836528778,
+          0.01033166702836752,
+          0.011786391027271748,
+          0.003580106422305107,
+          -0.0027060990687459707,
+          0.014578522183001041,
+          0.017018703743815422,
+          0.006647930946201086,
+          0.020162783563137054,
+          -0.001665893360041082,
+          -0.03572676330804825,
+          -0.004684836138039827,
+          0.0011809853604063392,
+          -0.00016595386841800064,
+          0.009526094421744347,
+          0.00445802416652441,
+          0.008290360681712627,
+          0.005795431789010763,
+          0.01958402246236801,
+          -0.006628378294408321,
+          0.010589763522148132,
+          0.00020762563508469611,
+          -0.02056948095560074,
+          -0.0020862771198153496,
+          -0.0040708803571760654,
+          -0.032191626727581024,
+          -0.00019369431538507342,
+          0.006080901715904474,
+          0.023087874054908752,
+          -0.009479167871177197,
+          0.0029055369086563587,
+          -0.006057438440620899,
+          -0.013162903487682343,
+          -0.00220554880797863,
+          0.014101434499025345,
+          -0.0012249790597707033,
+          0.01037859357893467,
+          0.0030678247567266226,
+          0.0005034829955548048,
+          0.0036387646105140448,
+          0.01966223306953907,
+          -0.00021385807485785335,
+          0.00868923682719469,
+          0.016877923160791397,
+          0.020256636664271355,
+          8.627645729575306e-05,
+          -0.01222437247633934,
+          0.027983879670500755,
+          -0.01428914163261652,
+          -0.008697058074176311,
+          -0.005603814963251352,
+          -0.005290971137583256,
+          0.015305883251130581,
+          -0.008478067815303802,
+          0.010589763522148132,
+          0.006628378294408321,
+          -0.018035445362329483,
+          0.01102774403989315,
+          -0.011176344938576221,
+          0.02361970767378807,
+          0.01689356565475464,
+          -0.0017186857294291258,
+          0.004282049834728241,
+          0.007293171249330044,
+          -0.03466309607028961,
+          0.009792011231184006,
+          -0.010957354679703712,
+          -0.0010734453098848462,
+          -0.0030111216474324465,
+          -0.004939021542668343,
+          0.016768427565693855,
+          -0.00014383482630364597,
+          -0.022086773067712784,
+          -0.00312257232144475,
+          0.021429801359772682,
+          -0.0013638035161420703,
+          0.01010485552251339,
+          0.006448493339121342,
+          -0.0020256636198610067,
+          -0.01618966832756996,
+          -0.020788472145795822,
+          -0.018129298463463783,
+          -0.020240994170308113,
+          -0.014367352239787579,
+          -0.00849370937794447,
+          -0.004794331267476082,
+          0.0045948936603963375,
+          0.0026572172064334154,
+          -0.008923869580030441,
+          -0.020929250866174698,
+          -0.014320425689220428,
+          0.003934010863304138,
+          0.0020354401785880327,
+          0.007723331451416016,
+          -0.0071993181481957436,
+          -0.007954053580760956,
+          -0.002113651018589735,
+          -0.0005504096043296158,
+          -0.006972506642341614,
+          0.01295173354446888,
+          -0.02463645115494728,
+          -0.005658562295138836,
+          0.01947452686727047,
+          0.011434441432356834,
+          -0.004536235239356756,
+          0.010316024534404278,
+          -0.026263238862156868,
+          0.02286888286471367,
+          -0.018410857766866684,
+          -0.015470126643776894,
+          -0.0079423226416111,
+          0.013632168993353844,
+          -0.015313704498112202,
+          0.005427840165793896,
+          0.006769157946109772,
+          -0.004395455587655306,
+          0.016408657655119896,
+          0.009181966073811054,
+          -0.011066850274801254,
+          0.009987538680434227,
+          -0.009416598826646805,
+          0.0015475992113351822,
+          -0.01645558513700962,
+          0.0009624835220165551,
+          0.015423200093209743,
+          0.010495910421013832,
+          -0.005267507862299681,
+          -0.010988638736307621,
+          -0.005705488845705986,
+          0.00047635359806008637,
+          0.0004220947448629886,
+          0.007500430569052696,
+          0.004008311312645674,
+          -0.002915313234552741,
+          -0.016064530238509178,
+          -0.017706960439682007,
+          0.0262006688863039,
+          -0.0051541016437113285,
+          -0.009103755466639996,
+          0.02047562785446644,
+          0.02150801196694374,
+          0.01615838333964348,
+          -0.017534896731376648,
+          -0.0005655629793182015,
+          0.009612126275897026,
+          0.014031045138835907,
+          0.007351829670369625,
+          -0.003501895349472761,
+          -0.00719149736687541,
+          -0.021523654460906982,
+          0.030095575377345085,
+          0.007261887192726135,
+          0.0075942836701869965,
+          -0.031284380704164505,
+          0.023385075852274895,
+          0.00021129178639966995,
+          -0.010409877635538578,
+          0.0026787251699715853,
+          0.01966223306953907,
+          0.005388734862208366,
+          0.0008539658738300204,
+          0.02297837845981121,
+          0.015086892992258072,
+          0.004931200761348009,
+          -0.007680315524339676,
+          -0.013217651285231113,
+          0.004551877733319998,
+          -0.005392645485699177,
+          0.009557378478348255,
+          -0.012028845027089119,
+          -0.009033365175127983,
+          0.0037658573128283024,
+          -0.003056093119084835,
+          0.008212150074541569,
+          -0.0002551632351242006,
+          -0.0211169570684433,
+          -0.006487598642706871,
+          -0.0363837368786335,
+          0.0050172326155006886,
+          -0.024464385583996773,
+          -0.005693757440894842,
+          -0.014234393835067749,
+          -0.013069050386548042,
+          -0.03932446613907814,
+          -0.010667974129319191,
+          0.008282540366053581,
+          0.015548337250947952,
+          0.0036446305457502604,
+          -0.011543937027454376,
+          0.0037287073209881783,
+          -0.040763549506664276,
+          0.0003023342287633568,
+          0.02848443016409874,
+          0.014492489397525787,
+          0.023494569584727287,
+          -0.015321525745093822,
+          0.022665534168481827,
+          0.0005333009175956249,
+          -0.02848443016409874,
+          -0.0029896136838942766,
+          -0.0006017355481162667,
+          0.011950633488595486,
+          0.00093902024673298,
+          -0.009721621870994568,
+          -0.017675675451755524,
+          0.020741544663906097,
+          0.006264697294682264,
+          0.009502630680799484,
+          -0.005185386165976524,
+          0.001369669334962964,
+          0.00380300753749907,
+          0.005204938817769289,
+          -0.014953934587538242,
+          0.0005670294049195945,
+          0.010581942275166512,
+          0.016095813363790512,
+          -0.006808263715356588,
+          -0.01263889018446207,
+          -0.008955154567956924,
+          -0.008196507580578327,
+          -0.00213711429387331,
+          -0.012576321139931679,
+          -0.010370772331953049,
+          -0.011606505140662193,
+          -0.0028957605827599764,
+          -0.01469583809375763,
+          -0.01618966832756996,
+          0.008188687264919281,
+          -0.0009399979026056826,
+          0.02316608466207981,
+          0.013984118588268757,
+          0.018676776438951492,
+          0.008141760714352131,
+          0.01681535504758358,
+          0.005142370238900185,
+          0.004305513110011816,
+          -0.014343888498842716,
+          -0.002422584220767021,
+          -0.030533555895090103,
+          0.013647811487317085,
+          0.02984529919922352,
+          -0.007003790698945522,
+          -0.00305413780733943,
+          0.010050107724964619,
+          0.0022466096561402082,
+          -0.002281804569065571,
+          0.0019415869610384107,
+          -0.01645558513700962,
+          -0.005236223340034485,
+          -0.01010485552251339,
+          0.006483688019216061,
+          0.009346208535134792,
+          0.028906768187880516,
+          -0.005552977789193392,
+          -0.0019083472434431314,
+          0.02241525985300541,
+          0.014359530992805958,
+          0.02537163347005844,
+          0.009471346624195576,
+          -0.006604915019124746,
+          0.00042062829015776515,
+          -0.0065384358167648315,
+          -0.009095934219658375,
+          -0.026122458279132843,
+          0.031034106388688087,
+          -0.004477577283978462,
+          -0.00877526868134737,
+          -0.009267997927963734,
+          -0.011590863578021526,
+          0.0025105716194957495,
+          0.02020971104502678,
+          -0.015947213396430016,
+          0.0011233048280701041,
+          0.0001070512444130145,
+          -0.016064530238509178,
+          -0.02712355926632881,
+          0.03189442679286003,
+          0.005979227367788553,
+          -0.002801907481625676,
+          -3.3239655749639496e-05,
+          -0.007426130119711161,
+          -0.011387514881789684,
+          -0.009346208535134792,
+          -0.0056976680643856525,
+          0.06563463062047958,
+          -0.01466455403715372,
+          0.002569229807704687,
+          -0.02305658906698227,
+          0.0031499462202191353,
+          -0.007766347844153643,
+          -0.00024954183027148247,
+          0.0075395358726382256,
+          -0.011004281230270863,
+          -0.0030736904591321945,
+          -0.01310033444315195,
+          -0.002369791967794299,
+          0.004160822834819555,
+          -0.019771728664636612,
+          -0.009236713871359825,
+          -0.0034745216835290194,
+          -0.011254556477069855,
+          -0.005283149890601635,
+          0.012153982184827328,
+          0.011309304274618626,
+          0.013890265487134457,
+          -0.02537163347005844,
+          -0.003361115697771311,
+          0.004852989688515663,
+          -0.008900406770408154,
+          -0.004012221936136484,
+          -0.017769528552889824,
+          -0.010065749287605286,
+          -0.009158502332866192,
+          -0.011066850274801254,
+          0.014774048700928688,
+          -0.009526094421744347,
+          0.012200908735394478,
+          0.006956864148378372,
+          0.0065970937721431255,
+          0.01958402246236801,
+          0.014492489397525787,
+          0.011692537926137447,
+          0.013037766329944134,
+          0.005658562295138836,
+          -0.01442992128431797,
+          0.019709160551428795,
+          -0.02213370054960251,
+          -0.022383974865078926,
+          -0.010018822737038136,
+          0.0033063681330531836,
+          -0.009893685579299927,
+          -0.007926680147647858,
+          -0.011332767084240913,
+          0.015650011599063873,
+          0.0026161563582718372,
+          -0.007649031467735767,
+          -0.012388614937663078,
+          -0.0024890636559575796,
+          0.0291257593780756,
+          -0.007649031467735767,
+          0.02039741724729538,
+          -0.012153982184827328,
+          0.009533915668725967,
+          -0.015086892992258072,
+          -0.00813393946737051,
+          -0.009713800624012947,
+          -0.008282540366053581,
+          0.014907008036971092,
+          -0.006319445092231035,
+          0.005298791918903589,
+          -0.017394116148352623,
+          -0.02692021057009697,
+          -0.0003842113073915243,
+          0.00380300753749907,
+          -0.028140300884842873,
+          0.007578641641885042,
+          -0.02316608466207981,
+          0.004802152514457703,
+          -0.011692537926137447,
+          -0.014672375284135342,
+          -0.01654943823814392,
+          0.024057690054178238,
+          0.002991568995639682,
+          -0.004188196733593941,
+          0.00126995041500777,
+          -0.00521667068824172,
+          -0.006354639772325754,
+          0.009330566972494125,
+          0.006964685395359993,
+          0.0049585746601223946,
+          0.015000861138105392,
+          -0.006980327423661947,
+          -0.011450083926320076,
+          -0.00256140879355371,
+          -0.0035879274364560843,
+          -0.017597464844584465,
+          0.0015700849471613765,
+          0.002289625583216548,
+          -0.006741784047335386,
+          0.005201028194278479,
+          0.013530494645237923,
+          -0.0049703060649335384,
+          0.006530614569783211,
+          0.018692417070269585,
+          0.0022270570043474436,
+          -0.005728952586650848,
+          -0.0055177826434373856,
+          -0.003934010863304138,
+          -0.002438226481899619,
+          0.014062329195439816,
+          0.010464625433087349,
+          0.015266777947545052,
+          -0.007273618597537279,
+          0.009385314770042896,
+          -0.00436417106539011,
+          0.001194672309793532,
+          -0.028140300884842873,
+          0.00799315981566906,
+          0.005052427761256695,
+          -0.0028351470828056335,
+          0.00530661316588521,
+          0.0017069540917873383,
+          0.04270318150520325,
+          0.008814374916255474,
+          0.013327146880328655,
+          -0.003323965473100543,
+          0.009674695320427418,
+          0.0027354280464351177,
+          0.012615426443517208,
+          0.0006882564048282802,
+          -0.02105438895523548,
+          0.011567399837076664,
+          -0.015047787688672543,
+          0.0027021884452551603,
+          0.017034346237778664,
+          -0.019177325069904327,
+          0.015438842587172985,
+          -0.01025345642119646,
+          -0.0001107173811760731,
+          -0.004002445377409458,
+          -0.022743744775652885,
+          -0.01001100242137909,
+          -0.00019137242634315044,
+          0.023932551965117455,
+          -0.0017978743417188525,
+          0.025747045874595642,
+          0.006417208816856146,
+          0.02491801045835018,
+          -0.00804008636623621,
+          -0.017253335565328598,
+          -0.005607725586742163,
+          0.005103264935314655,
+          -0.003548822132870555,
+          0.0057524158619344234,
+          0.008282540366053581,
+          0.0060183326713740826,
+          -0.001099841552786529,
+          -0.0007962852832861245,
+          -0.000347061111824587,
+          0.022900167852640152,
+          0.009291461668908596,
+          0.01056629978120327,
+          -0.01037859357893467,
+          0.013084692880511284,
+          -0.018051087856292725,
+          0.00021336926147341728,
+          0.0010089213028550148,
+          -0.022649891674518585,
+          0.0014077972155064344,
+          -0.004852989688515663,
+          -0.004371992312371731,
+          0.0026669935323297977,
+          0.019505811855196953,
+          -0.019427601248025894,
+          -0.014555058442056179,
+          0.02590346708893776,
+          0.007426130119711161,
+          0.0025555428583174944,
+          -0.002262251917272806,
+          0.009831116534769535,
+          0.00873616337776184,
+          0.011614326387643814,
+          -0.004720031283795834,
+          -0.005846268963068724,
+          -0.017753886058926582,
+          0.014164003543555737,
+          -0.006534525193274021,
+          -0.010308203287422657,
+          -0.008923869580030441,
+          -0.008345108479261398,
+          0.01617402583360672,
+          0.0007835759897716343,
+          0.007668584119528532,
+          0.002807773184031248,
+          0.02637273445725441,
+          -0.01281877513974905,
+          -0.011332767084240913,
+          -0.009408777579665184,
+          0.008556278422474861,
+          0.024464385583996773,
+          0.007957964204251766,
+          0.013202008791267872,
+          -0.01618966832756996,
+          -0.0036993781104683876,
+          0.019271178171038628,
+          -0.011872422881424427,
+          -0.033224012702703476,
+          -0.007500430569052696
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "c6bf0f92-84c8-44b9-8b28-7fcab6058203",
+      "properties": {
+        "page_content": "8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n",
+        "keyphrases": [
+          "Mixture of Experts (MoE)",
+          "language model training",
+          "DeepSeekMoE architecture",
+          "expert-choice routing strategy",
+          "training instability"
+        ],
+        "title": "Related Work",
+        "embedding": [
+          -0.0004454151203390211,
+          -0.01349730882793665,
+          -0.018033159896731377,
+          0.005232766270637512,
+          0.01874188706278801,
+          0.010268663987517357,
+          -0.011331754736602306,
+          0.03266443312168121,
+          -0.018993878737092018,
+          0.018789134919643402,
+          -0.005878495052456856,
+          -0.009221323765814304,
+          0.007355009205639362,
+          -0.008567719720304012,
+          0.0011782583314925432,
+          0.02784508839249611,
+          0.018474144861102104,
+          0.015410871244966984,
+          0.002397859003394842,
+          -0.03512135148048401,
+          0.019734105095267296,
+          -0.02189178392291069,
+          -0.044067058712244034,
+          -0.013772924430668354,
+          0.014253283850848675,
+          0.01693069562315941,
+          0.03858623653650284,
+          -0.006386416032910347,
+          -0.05959605053067207,
+          0.03386139124631882,
+          0.020521579310297966,
+          0.02291550114750862,
+          -0.01582823134958744,
+          -0.014930511824786663,
+          -0.02899480052292347,
+          -0.013481559231877327,
+          0.05077634006738663,
+          0.021734289824962616,
+          -0.016410963609814644,
+          0.027687594294548035,
+          0.0467129722237587,
+          0.013268941082060337,
+          -0.02280525304377079,
+          -0.0008283244678750634,
+          0.005512319505214691,
+          0.0011831800220534205,
+          -0.019009627401828766,
+          -0.016332216560840607,
+          0.010497031733393669,
+          -0.028175828978419304,
+          -0.018883632495999336,
+          -0.027278108522295952,
+          -0.006665969267487526,
+          0.0006486818892881274,
+          -0.0642578974366188,
+          0.007378633599728346,
+          0.01752917654812336,
+          0.03389289230108261,
+          0.05058734491467476,
+          0.0015080131124705076,
+          0.012308222241699696,
+          -0.019419115036725998,
+          -0.01686769910156727,
+          -0.008260604925453663,
+          -0.00022885970247443765,
+          -0.01097739115357399,
+          0.012016857042908669,
+          -0.013324064202606678,
+          0.01982860080897808,
+          0.016221968457102776,
+          -0.040854163467884064,
+          0.027026114985346794,
+          -0.023876218125224113,
+          0.009426066651940346,
+          -0.006969146896153688,
+          0.007095142733305693,
+          0.019135624170303345,
+          0.026600880548357964,
+          -0.0019509674748405814,
+          -0.007252637762576342,
+          0.0025514166336506605,
+          0.017324432730674744,
+          -0.007122704293578863,
+          0.0022108338307589293,
+          0.05014635995030403,
+          -0.04687046632170677,
+          -0.02584490366280079,
+          -0.07049469649791718,
+          0.00882758665829897,
+          0.00037503460771404207,
+          -0.024789689108729362,
+          -0.003289673710241914,
+          0.028490817174315453,
+          0.008835460990667343,
+          -0.040602169930934906,
+          0.06312393397092819,
+          -0.014056415297091007,
+          -0.015662861987948418,
+          -0.03512135148048401,
+          -0.008355101570487022,
+          0.0005320372874848545,
+          -0.0049768369644880295,
+          0.011575871147215366,
+          0.010709649883210659,
+          0.012197976000607014,
+          -0.012812205590307713,
+          -0.033199913799762726,
+          0.04687046632170677,
+          -0.005713125690817833,
+          0.08523621410131454,
+          0.0021379925310611725,
+          -0.009945799596607685,
+          -0.003252268536016345,
+          0.03537334129214287,
+          -0.021435048431158066,
+          -0.008386600762605667,
+          -0.028349071741104126,
+          -0.04935888573527336,
+          0.024852687492966652,
+          -0.02348248101770878,
+          0.018631640821695328,
+          -0.0007800916791893542,
+          0.016395213082432747,
+          -0.019214371219277382,
+          -0.0721326395869255,
+          0.05739112198352814,
+          -0.0026774124708026648,
+          -0.015111630782485008,
+          -0.003021932439878583,
+          0.007201451808214188,
+          -0.0008194653782993555,
+          0.0037345965392887592,
+          -0.01735593192279339,
+          -0.019088376313447952,
+          0.029136545956134796,
+          0.023513980209827423,
+          0.03861773759126663,
+          0.046240486204624176,
+          -0.020301084965467453,
+          -0.01452889945358038,
+          0.023734472692012787,
+          -0.029514534398913383,
+          0.006630532909184694,
+          0.011567996814846992,
+          -0.01746617816388607,
+          0.011599495075643063,
+          0.011662493459880352,
+          0.06967572122812271,
+          -0.019860100001096725,
+          0.017009442672133446,
+          0.0022895813453942537,
+          0.0643208920955658,
+          0.0025435418356209993,
+          0.030475253239274025,
+          -0.014765141531825066,
+          0.02178153768181801,
+          -0.028160078451037407,
+          0.0025061366613954306,
+          -0.020222337916493416,
+          -0.003464886685833335,
+          -0.002510074060410261,
+          -0.04094865918159485,
+          -0.002821126487106085,
+          0.028853056952357292,
+          0.0015119505114853382,
+          0.01776541955769062,
+          0.033199913799762726,
+          0.003850748995319009,
+          -0.05962754786014557,
+          0.023198990151286125,
+          0.03452287241816521,
+          0.010985265485942364,
+          -0.009103202261030674,
+          0.021624041721224785,
+          0.011056138202548027,
+          0.010575778782367706,
+          0.02067907340824604,
+          0.008189732208848,
+          -0.022710757330060005,
+          0.03972020000219345,
+          -0.02886880561709404,
+          0.0018830477492883801,
+          0.03228644281625748,
+          0.0016743671149015427,
+          0.044791534543037415,
+          0.012906703166663647,
+          0.026852872222661972,
+          0.01693069562315941,
+          0.012316097505390644,
+          0.029939770698547363,
+          0.007166015449911356,
+          0.010536405257880688,
+          0.03540484234690666,
+          -0.015095881186425686,
+          0.008158233016729355,
+          -0.02255326136946678,
+          -0.012867328710854053,
+          0.006488787941634655,
+          0.00988280214369297,
+          -0.005815497133880854,
+          -0.0021753974724560976,
+          -0.04201962426304817,
+          0.02212802693247795,
+          -0.0055831922218203545,
+          0.021986281499266624,
+          0.0026144145522266626,
+          0.012615337036550045,
+          0.007630625274032354,
+          0.03559383377432823,
+          -0.03852323815226555,
+          -0.011788489297032356,
+          -0.007878679782152176,
+          0.005535943899303675,
+          -0.044791534543037415,
+          -0.020033344626426697,
+          0.016725953668355942,
+          -0.004776031244546175,
+          0.021135808899998665,
+          0.009276446886360645,
+          -0.02159254439175129,
+          -0.006626595743000507,
+          -0.006445476785302162,
+          0.018363898620009422,
+          -0.01533999852836132,
+          -0.030491001904010773,
+          -0.0351843498647213,
+          0.01300907414406538,
+          0.011536497622728348,
+          0.03590882569551468,
+          -0.0009178996551781893,
+          -0.020600326359272003,
+          0.014788766391575336,
+          0.001092620543204248,
+          0.0039216214790940285,
+          0.02274225652217865,
+          -0.04646097868680954,
+          0.00845747347921133,
+          0.0026537880767136812,
+          0.011520748026669025,
+          0.02085231803357601,
+          0.007394383195787668,
+          0.017182687297463417,
+          -0.035751331597566605,
+          0.004086991306394339,
+          0.03688529506325722,
+          0.011638869531452656,
+          -0.016103848814964294,
+          0.028065582737326622,
+          4.574118065647781e-05,
+          -0.012883078306913376,
+          -0.0033290472347289324,
+          -0.03222344443202019,
+          -0.012623212300240993,
+          0.0008617921266704798,
+          -0.016757450997829437,
+          0.05269777774810791,
+          0.043090589344501495,
+          -0.019324617460370064,
+          -0.014631271362304688,
+          -0.05918656289577484,
+          0.039688702672719955,
+          -0.02408096194267273,
+          -0.025262173265218735,
+          0.04416155442595482,
+          -0.016725953668355942,
+          0.020159341394901276,
+          0.032475437968969345,
+          0.012820080853998661,
+          -0.00011873635230585933,
+          -0.03527884557843208,
+          -0.004051554948091507,
+          -0.04523251950740814,
+          -0.03801925480365753,
+          0.005126457195729017,
+          0.03170371428132057,
+          -0.03559383377432823,
+          -0.0055792550556361675,
+          -0.009481189772486687,
+          0.030191762372851372,
+          0.028837306424975395,
+          0.02049008011817932,
+          -0.0016379464650526643,
+          0.007969239726662636,
+          0.016379464417696,
+          -0.03208170086145401,
+          0.013394936919212341,
+          -0.0046460977755486965,
+          -0.026522131636738777,
+          -0.010308037512004375,
+          -0.015749484300613403,
+          -0.006760466378182173,
+          -4.174228524789214e-05,
+          -0.04693346470594406,
+          0.00440985569730401,
+          0.03540484234690666,
+          -0.0037050663959234953,
+          -0.01795441284775734,
+          0.03323141112923622,
+          0.033451907336711884,
+          -0.027860838919878006,
+          -0.06167498230934143,
+          -0.001828908920288086,
+          0.021450798958539963,
+          -0.0024529823567718267,
+          -0.05616266280412674,
+          -0.003445199690759182,
+          -0.008276354521512985,
+          0.030554000288248062,
+          -0.01710394024848938,
+          -0.04201962426304817,
+          0.0032680181320756674,
+          -0.0005128426128067076,
+          -0.013481559231877327,
+          -0.03433387726545334,
+          -0.00764243770390749,
+          0.034365374594926834,
+          0.01437140442430973,
+          0.025813406333327293,
+          -0.04384656623005867,
+          -0.02929404191672802,
+          0.01270983461290598,
+          0.0394052118062973,
+          -0.06284044682979584,
+          0.010308037512004375,
+          -0.014292657375335693,
+          -0.001888953847810626,
+          0.004728782922029495,
+          0.043878063559532166,
+          -0.03291642293334007,
+          0.009016579948365688,
+          -0.00609505083411932,
+          -0.04167313501238823,
+          -0.034963857382535934,
+          -0.010827770456671715,
+          -0.00495715020224452,
+          -0.016332216560840607,
+          -0.02297849766910076,
+          0.002760097151622176,
+          -0.014269033446907997,
+          0.003340859431773424,
+          -0.02165554091334343,
+          0.004846903961151838,
+          -0.017859915271401405,
+          0.021734289824962616,
+          0.010056045837700367,
+          -0.013938293792307377,
+          -0.025703158229589462,
+          -0.004941401071846485,
+          0.02097831293940544,
+          0.012670460157096386,
+          0.00115561846178025,
+          -0.0034688240848481655,
+          0.002754191169515252,
+          -0.003376295790076256,
+          -0.009693807922303677,
+          -0.06545485556125641,
+          -0.017261436209082603,
+          -0.017607923597097397,
+          0.015552615746855736,
+          0.016056599095463753,
+          0.010725399479269981,
+          -0.046901967376470566,
+          -0.05118582397699356,
+          0.028538066893815994,
+          -0.03549933806061745,
+          -0.001561167649924755,
+          -0.0025927589740604162,
+          -0.0015926666092127562,
+          -0.004386231303215027,
+          0.016568457707762718,
+          -0.005606816615909338,
+          0.03360940143465996,
+          -0.012016857042908669,
+          0.007650312036275864,
+          0.004480728413909674,
+          0.020269587635993958,
+          0.0333574078977108,
+          -0.011103386990725994,
+          -0.011048263870179653,
+          0.049106892198324203,
+          -0.013631178997457027,
+          0.025088928639888763,
+          0.005638315342366695,
+          0.001570026739500463,
+          0.0033861391711980104,
+          0.02382897026836872,
+          0.014198160730302334,
+          0.005350887309759855,
+          -0.03713728487491608,
+          0.027089113369584084,
+          0.010363160632550716,
+          -0.060919005423784256,
+          -0.015088006854057312,
+          0.006595096550881863,
+          0.017418930307030678,
+          0.016536958515644073,
+          -0.003559383563697338,
+          -0.014662770554423332,
+          -0.013457934372127056,
+          -0.008481097407639027,
+          -0.018521394580602646,
+          0.036412809044122696,
+          -0.0033054230734705925,
+          0.010725399479269981,
+          -0.027215110138058662,
+          0.0045752255246043205,
+          0.01328469067811966,
+          0.016206219792366028,
+          0.02942003682255745,
+          -0.015922728925943375,
+          -0.015017134137451649,
+          -0.04718545451760292,
+          -0.03288492560386658,
+          -0.00928432121872902,
+          0.013245316222310066,
+          0.030128763988614082,
+          0.01600935123860836,
+          0.046429481357336044,
+          -0.012757082469761372,
+          0.00840235035866499,
+          -0.027624595910310745,
+          0.03795625641942024,
+          0.049043893814086914,
+          0.009985173121094704,
+          -0.020631825551390648,
+          0.02171853929758072,
+          0.009985173121094704,
+          -0.01995459757745266,
+          0.012308222241699696,
+          -0.020631825551390648,
+          0.040413178503513336,
+          -0.022458765655755997,
+          0.03874373435974121,
+          -0.04337408021092415,
+          -0.010709649883210659,
+          -0.03414488211274147,
+          0.003462917869910598,
+          0.02856956608593464,
+          -0.022395767271518707,
+          -0.021608293056488037,
+          -0.0017294903518632054,
+          0.0272938571870327,
+          0.012410594150424004,
+          -0.0009070718660950661,
+          -0.02293124981224537,
+          -0.021198807284235954,
+          -0.03534184396266937,
+          0.006890399381518364,
+          -0.014465902000665665,
+          -0.017371682450175285,
+          -0.0533907525241375,
+          0.014245408587157726,
+          -0.07143966108560562,
+          0.010993140749633312,
+          0.0485084131360054,
+          -0.012276723049581051,
+          0.016662955284118652,
+          0.0007077424670569599,
+          -0.004256298299878836,
+          -0.02935704030096531,
+          0.00967018399387598,
+          0.049831368029117584,
+          0.04028718173503876,
+          -0.021482296288013458,
+          -0.017670921981334686,
+          0.009150451049208641,
+          -0.028601063415408134,
+          -0.01589122973382473,
+          -0.005209141876548529,
+          0.01704094186425209,
+          0.014481651596724987,
+          -0.02589215338230133,
+          0.02638038620352745,
+          0.008465347811579704,
+          0.004953213036060333,
+          0.028396321460604668,
+          0.01331618893891573,
+          -0.0460829921066761,
+          0.014347780495882034,
+          0.029372788965702057,
+          0.001872219960205257,
+          -0.010378910228610039,
+          -0.004106678068637848,
+          0.00592968100681901,
+          0.01819065399467945,
+          -0.017576424404978752,
+          0.007114829961210489,
+          0.005591067019850016,
+          -0.01583610661327839,
+          0.020694822072982788,
+          0.011930234730243683,
+          0.01128450594842434,
+          -0.01067027635872364,
+          0.0218287855386734,
+          -0.00031375925755128264,
+          -0.019907347857952118,
+          -0.009426066651940346,
+          -0.00981980375945568,
+          0.02061607502400875,
+          0.00038364759529940784,
+          -0.03445987403392792,
+          0.011111261323094368,
+          0.04372056946158409,
+          -0.001538527780212462,
+          0.01003242190927267,
+          0.010812020860612392,
+          0.030223261564970016,
+          0.028538066893815994,
+          -0.022710757330060005,
+          -0.02825457602739334,
+          0.01691494695842266,
+          -0.009394567459821701,
+          0.004547663498669863,
+          -0.022632010281085968,
+          -0.014237534254789352,
+          0.03408188372850418,
+          -0.016725953668355942,
+          0.033136915415525436,
+          -0.002256113803014159,
+          0.015969976782798767,
+          -0.004846903961151838,
+          0.0010483250953257084,
+          -0.006433664355427027,
+          0.035215847194194794,
+          -0.026396136730909348,
+          0.014930511824786663,
+          0.007110892329365015,
+          0.002866406226530671,
+          0.05723362788558006,
+          -0.03814525157213211,
+          -0.03625531494617462,
+          0.00262228911742568,
+          0.030853239819407463,
+          -0.017796916887164116,
+          -0.005913931410759687,
+          -0.009599311277270317,
+          0.0031813958194106817,
+          0.018757635727524757,
+          -0.0351843498647213,
+          -0.01855289377272129,
+          0.01331618893891573,
+          -0.01801741123199463,
+          0.003470792667940259,
+          -0.0006501584430225194,
+          0.0018436741083860397,
+          0.00524457823485136,
+          0.0618639774620533,
+          -0.05175280570983887,
+          0.016946446150541306,
+          0.03603482246398926,
+          -0.0007520379149354994,
+          -0.03282192721962929,
+          -0.02625439129769802,
+          -0.005736749619245529,
+          -0.003931465093046427,
+          -0.03067999705672264,
+          -1.9440769392531365e-05,
+          -0.004043680150061846,
+          0.007024270016700029,
+          -0.03669629991054535,
+          0.014623397029936314,
+          0.011355378665030003,
+          -0.002449044957756996,
+          0.003744439920410514,
+          -0.01391466986387968,
+          -0.002553385216742754,
+          -0.003896028734743595,
+          0.03367239981889725,
+          -0.011064013466238976,
+          -0.002878218423575163,
+          -0.00967018399387598,
+          -0.003775938879698515,
+          -0.007197514642030001,
+          -0.0027738779317587614,
+          -0.00845747347921133,
+          0.0031873020343482494,
+          0.012237349525094032,
+          0.001938170986250043,
+          -0.012426343746483326,
+          -0.0028171890880912542,
+          0.03657030314207077,
+          -0.02510467916727066,
+          -0.021183056756854057,
+          0.051280323415994644,
+          0.040476176887750626,
+          0.004374419339001179,
+          -0.021907532587647438,
+          -0.010260789655148983,
+          0.021498046815395355,
+          -0.019749853760004044,
+          0.013670552521944046,
+          0.032176196575164795,
+          -0.007547940593212843,
+          -0.028364822268486023,
+          -0.00964655913412571,
+          -0.014410778880119324,
+          0.0019509674748405814,
+          -0.00562256621196866,
+          0.024065213277935982,
+          -0.044539544731378555,
+          0.008488972671329975,
+          0.019938847050070763,
+          0.002590790158137679,
+          -0.012394844554364681,
+          -0.011040388606488705,
+          -0.003295579692348838,
+          0.02178153768181801,
+          0.0009813897777348757,
+          -0.01893088035285473,
+          0.00861496850848198,
+          0.013418560847640038,
+          0.006158048752695322,
+          0.01893088035285473,
+          -0.005732812453061342,
+          0.02352973073720932,
+          0.009733181446790695,
+          -0.0005241625476628542,
+          0.014056415297091007,
+          -0.019970346242189407,
+          -0.00488234031945467,
+          0.061517488211393356,
+          0.011623119935393333,
+          0.025734657421708107,
+          -0.01431628130376339,
+          0.020631825551390648,
+          0.03814525157213211,
+          -0.0016999599756672978,
+          -0.020820818841457367,
+          0.008362976834177971,
+          0.00283490726724267,
+          -0.044665537774562836,
+          -0.0134894335642457,
+          -0.023923467844724655,
+          -0.037326280027627945,
+          0.0035357594024389982,
+          -0.021214555948972702,
+          0.01801741123199463,
+          0.025199174880981445,
+          0.0065832845866680145,
+          -0.030884739011526108,
+          -0.01197748351842165,
+          0.022710757330060005,
+          0.02741985209286213,
+          0.03449137136340141,
+          -0.01233972143381834,
+          0.010103294625878334,
+          -0.003295579692348838,
+          0.013694177381694317,
+          0.037987757474184036,
+          0.006925835739821196,
+          -0.0031006797216832638,
+          0.008059798739850521,
+          0.029703527688980103,
+          0.03675929829478264,
+          -0.024584945291280746,
+          -0.052130796015262604,
+          -0.0028703436255455017,
+          -0.01686769910156727,
+          0.018600141629576683,
+          -0.001023716526106,
+          0.003746408736333251,
+          0.0011428219731897116,
+          0.006654157303273678,
+          -0.008740964345633984,
+          0.02893180400133133,
+          0.008028299547731876,
+          -0.021797286346554756,
+          0.05159531161189079,
+          -0.015481743961572647,
+          0.024112461134791374,
+          -0.001423359732143581,
+          -0.02590790204703808,
+          -0.028711311519145966,
+          0.013111446052789688,
+          0.007390445563942194,
+          -0.03546784073114395,
+          0.03742077574133873,
+          0.033451907336711884,
+          0.0030731181614100933,
+          0.0012127102818340063,
+          0.0166787039488554,
+          0.016962194815278053,
+          -0.014780891127884388,
+          -0.023876218125224113,
+          0.02286825142800808,
+          -0.008677965961396694,
+          -0.021939031779766083,
+          0.01982860080897808,
+          -0.018521394580602646,
+          -0.008268479257822037,
+          0.0042484235018491745,
+          -0.001532621681690216,
+          -0.010182041674852371,
+          0.014395029284060001,
+          0.016521209850907326,
+          -0.015332123264670372,
+          -0.02527792379260063,
+          0.031215479597449303,
+          0.021513795480132103,
+          0.0038231874350458384,
+          -0.02165554091334343,
+          0.013473683968186378,
+          -0.022537512704730034,
+          -0.0010286382166668773,
+          0.043752070516347885,
+          -0.002433295361697674,
+          0.012946076691150665,
+          -0.010237164795398712,
+          -0.029719278216362,
+          0.002671506255865097,
+          0.011213633231818676,
+          0.03789326176047325,
+          0.01885213330388069,
+          0.01795441284775734,
+          0.019450614228844643,
+          -0.01631646603345871,
+          0.009528438560664654,
+          -0.014560398645699024,
+          -0.03294792026281357,
+          0.03179820999503136,
+          -0.006024178117513657,
+          0.025577163323760033,
+          0.008465347811579704,
+          -0.0321604460477829,
+          -0.004728782922029495,
+          0.011040388606488705,
+          0.013001199811697006,
+          0.023120243102312088,
+          0.013867421075701714,
+          0.003108554519712925,
+          0.031388722360134125,
+          -0.014513149857521057,
+          -0.051091328263282776,
+          0.02710486389696598,
+          0.0063076685182750225,
+          -0.006701405625790358,
+          -0.0016143221873790026,
+          0.020364083349704742,
+          -0.003027838421985507,
+          0.00017139868577942252,
+          0.01270195934921503,
+          -0.010961641557514668,
+          0.00882758665829897,
+          -0.006992771290242672,
+          -0.005469008814543486,
+          0.02414396032691002,
+          0.011827862821519375,
+          0.024065213277935982,
+          -0.019497862085700035,
+          0.006347042508423328,
+          0.017324432730674744,
+          -0.009748931042850018,
+          0.014568272978067398,
+          0.0034314189106225967,
+          -0.05011485889554024,
+          0.01607234962284565,
+          0.017576424404978752,
+          -0.005173705518245697,
+          0.007103017531335354,
+          -0.013402811251580715,
+          0.013804423622786999,
+          0.04627198725938797,
+          0.0016330246580764651,
+          0.0066856564953923225,
+          0.013205942697823048,
+          -0.005012273322790861,
+          -0.009804054163396358,
+          0.03112098202109337,
+          0.00845747347921133,
+          -0.008796087466180325,
+          0.009079578332602978,
+          0.001405641552992165,
+          0.03499535471200943,
+          -0.04828792065382004,
+          0.010717524215579033,
+          0.012520840391516685,
+          -0.018584391102194786,
+          -0.012812205590307713,
+          -0.02886880561709404,
+          0.009520563296973705,
+          -0.03044375404715538,
+          0.01814340613782406,
+          -0.022789504379034042,
+          0.021608293056488037,
+          -0.035688333213329315,
+          -0.0018732043681666255,
+          0.012623212300240993,
+          -0.0058076223358511925,
+          0.003575132926926017,
+          0.012473591603338718,
+          -0.007575502153486013,
+          -0.009323694743216038,
+          -0.018096158280968666,
+          0.023671476170420647,
+          0.05575317516922951,
+          -0.0003388600016478449,
+          0.00024116398708429188,
+          0.015930604189634323,
+          -0.01443440280854702,
+          -0.02620714344084263,
+          0.02953028306365013,
+          -0.016773201525211334,
+          0.015623488463461399,
+          0.03723178058862686,
+          -0.03468036651611328,
+          -0.027262357994914055,
+          0.01543449517339468,
+          -0.013725675642490387,
+          0.008347227238118649,
+          0.0020651512313634157,
+          0.02159254439175129,
+          -0.00283490726724267,
+          -0.026096895337104797,
+          0.028538066893815994,
+          0.006197422277182341,
+          0.009418192319571972,
+          -0.021671291440725327,
+          -0.011095511727035046,
+          -0.027325356379151344,
+          -0.023797471076250076,
+          -0.011867236346006393,
+          -0.010174167342483997,
+          0.006508474703878164,
+          -0.016820449382066727,
+          -0.057832106947898865,
+          -0.015229751355946064,
+          0.023907717317342758,
+          -0.01412728801369667,
+          0.05263477936387062,
+          0.01844264753162861,
+          0.016269218176603317,
+          0.0048035928048193455,
+          -0.04022418335080147,
+          0.030176013708114624,
+          -0.017560675740242004,
+          0.03143597021698952,
+          0.016088098287582397,
+          0.006839213892817497,
+          -0.0059611801989376545,
+          0.05058734491467476,
+          -0.02389196865260601,
+          0.007126641925424337,
+          -0.019686855375766754,
+          0.006473038345575333,
+          -0.02499443292617798,
+          -0.004508289974182844,
+          -0.00521701667457819,
+          -0.011465624906122684,
+          0.010063921101391315,
+          -0.011426251381635666,
+          0.024049462750554085,
+          0.02541966922581196,
+          0.0009026423213072121,
+          0.020301084965467453,
+          -0.030900489538908005,
+          -0.022206773981451988,
+          -0.004740594886243343,
+          0.0018899381393566728,
+          -0.02080507017672062,
+          -0.016332216560840607,
+          -0.040066689252853394,
+          -0.012788581661880016,
+          -0.0002187701902585104,
+          0.01212710328400135,
+          0.0040909284725785255,
+          0.02838057093322277,
+          -0.0017038973746821284,
+          -0.03209745138883591,
+          0.0071896398440003395,
+          -0.029514534398913383,
+          -0.0005906056612730026,
+          0.01391466986387968,
+          -0.017859915271401405,
+          -0.005945430602878332,
+          0.030648497864603996,
+          0.0016812575049698353,
+          -0.0150092588737607,
+          -0.016032975167036057,
+          0.005567443091422319,
+          -0.016237718984484673,
+          -0.0035554461646825075,
+          -0.01028441358357668,
+          0.03364089876413345,
+          -0.015332123264670372,
+          0.007780245505273342,
+          0.0005694422870874405,
+          4.2634346755221486e-05,
+          -0.03439687564969063,
+          -0.012890953570604324,
+          -0.019686855375766754,
+          0.013371312990784645,
+          0.0017777230823412538,
+          -0.027073364704847336,
+          0.007902303710579872,
+          0.0009838506812229753,
+          -0.004531914368271828,
+          -0.021324802190065384,
+          0.0032621119171380997,
+          0.005339075345546007,
+          -0.01771816983819008,
+          0.001696022693067789,
+          -0.033388908952474594,
+          -0.02831757441163063,
+          -0.029719278216362,
+          0.019419115036725998,
+          -0.0030435880180448294,
+          0.021183056756854057,
+          -0.013205942697823048,
+          0.012969700619578362,
+          -0.017261436209082603,
+          0.004590974655002356,
+          0.01570223644375801,
+          0.024663692340254784,
+          -0.03160921484231949,
+          0.013898920267820358,
+          0.01582035794854164,
+          0.021561045199632645,
+          -0.04523251950740814,
+          0.010709649883210659,
+          0.013631178997457027,
+          -0.0015227783005684614,
+          0.016332216560840607,
+          -0.010835645720362663,
+          0.03354640305042267,
+          -0.02093106508255005,
+          -0.0010749023640528321,
+          -0.005587129853665829,
+          0.013520932756364346,
+          0.01861589029431343,
+          0.012883078306913376,
+          -0.030412254855036736,
+          0.03110523149371147,
+          0.02352973073720932,
+          -0.012662585824728012,
+          0.04614599049091339,
+          -0.02656938135623932,
+          0.016458211466670036,
+          0.018600141629576683,
+          -0.02827032469213009,
+          0.0015660893404856324,
+          0.0045791626907885075,
+          -0.015631362795829773,
+          -0.014206035062670708,
+          0.025340920314192772,
+          0.00750462943688035,
+          -0.04346857964992523,
+          0.015001384541392326,
+          0.018237903714179993,
+          0.01119000930339098,
+          0.00433504581451416,
+          -0.023939216509461403,
+          0.009174074977636337,
+          0.028128579258918762,
+          0.0006875634426251054,
+          0.030475253239274025,
+          0.015064381994307041,
+          -0.02323048934340477,
+          0.011930234730243683,
+          -0.005795810371637344,
+          0.007406195160001516,
+          -0.01831665076315403,
+          0.00392752792686224,
+          -0.00747706787660718,
+          -0.014654895290732384,
+          -0.007232951000332832,
+          0.029640531167387962,
+          -0.003964932635426521,
+          -0.018883632495999336,
+          -0.03833424672484398,
+          0.015804607421159744,
+          0.009252822026610374,
+          -0.06709280610084534,
+          -0.00876458827406168,
+          -0.03386139124631882,
+          -0.02140354923903942,
+          -0.008906333707273006,
+          0.012749208137392998,
+          -0.015851857140660286,
+          -0.002389984205365181,
+          0.006154111120849848,
+          0.015599864535033703,
+          -0.01564711332321167,
+          0.02085231803357601,
+          0.006024178117513657,
+          0.005291827023029327,
+          -0.013993416912853718,
+          0.042051125317811966,
+          -0.007563690189272165,
+          -0.03814525157213211,
+          0.032176196575164795,
+          -0.012820080853998661,
+          0.003575132926926017,
+          0.024065213277935982,
+          -0.0019421083852648735,
+          -0.022002030164003372,
+          0.028963303193449974,
+          -0.026285890489816666,
+          -0.03512135148048401,
+          0.01522187702357769,
+          -0.006547848228365183,
+          -0.020285336300730705,
+          -0.02056882716715336,
+          -0.004681534133851528,
+          0.012733458541333675,
+          -0.02050582878291607,
+          0.018489895388484,
+          0.03266443312168121,
+          0.0052524530328810215,
+          -0.016206219792366028,
+          0.012457842007279396,
+          -0.0032247069757431746,
+          -0.0015030914219096303,
+          -0.005173705518245697,
+          0.024364452809095383,
+          -0.004055492114275694,
+          0.0033625150099396706,
+          -0.019119873642921448,
+          0.0030239010229706764,
+          -0.016088098287582397,
+          -0.012670460157096386,
+          -0.013457934372127056,
+          -0.025214925408363342,
+          -0.02493143454194069,
+          -0.04088566079735756,
+          0.03531034663319588,
+          0.024458950385451317,
+          -0.017324432730674744,
+          -0.010599403642117977,
+          -0.00861496850848198,
+          0.0013957980554550886,
+          0.028396321460604668,
+          -0.002578978193923831,
+          -0.0036597864236682653,
+          -0.02231702022254467,
+          0.011686117388308048,
+          -0.010583654046058655,
+          0.0014322188217192888,
+          -0.06296644359827042,
+          -0.01806465908885002,
+          0.034428372979164124,
+          0.007181765045970678,
+          -0.007996801286935806,
+          0.005917869042605162,
+          0.02376597188413143,
+          -0.020222337916493416,
+          -0.012481466867029667,
+          0.04390956461429596,
+          0.0027305670082569122,
+          0.0021458673290908337,
+          0.0070203328505158424,
+          -0.008347227238118649,
+          0.010961641557514668,
+          -0.004445292055606842,
+          -0.007729059550911188,
+          0.0005728874821215868,
+          0.010969515889883041,
+          0.009654434397816658,
+          0.00638247886672616,
+          -0.00442954245954752,
+          0.020710572600364685,
+          -0.011402626521885395,
+          -0.0043665445409715176,
+          0.006543911062180996,
+          0.019749853760004044,
+          -0.004783906042575836,
+          0.00025014611310325563,
+          -0.020301084965467453,
+          0.023986464366316795,
+          0.020773570984601974,
+          0.016158971935510635,
+          -0.0004537820350378752,
+          -0.008733089081943035,
+          -0.008059798739850521,
+          0.004401980899274349,
+          0.0030317758210003376,
+          0.017497677356004715,
+          0.028112830594182014,
+          -0.019482113420963287,
+          -0.016536958515644073,
+          -0.0012373188510537148,
+          0.0025336984544992447,
+          -0.004968962632119656,
+          0.03990919515490532,
+          0.02431720495223999,
+          0.005598941817879677,
+          -0.013969792984426022,
+          -0.0007269371417351067,
+          0.00891420803964138,
+          0.03505835309624672,
+          -0.026112645864486694,
+          -0.01571011170744896,
+          0.019907347857952118,
+          -0.0016330246580764651,
+          0.03433387726545334,
+          -0.021387800574302673,
+          0.014261158183217049,
+          -0.0026006337720900774,
+          0.08655916899442673,
+          -0.023372234776616096,
+          -0.005098895635455847,
+          -0.016174720600247383,
+          -0.014450152404606342,
+          0.004220861941576004,
+          -0.00459491228684783,
+          -0.003289673710241914,
+          0.031624965369701385,
+          -0.002029714873060584,
+          0.017182687297463417,
+          -0.0004545202828012407,
+          -0.008898458443582058,
+          0.008020425215363503,
+          -0.0035180412232875824,
+          0.018206404522061348,
+          -0.006681718863546848,
+          0.008969331160187721,
+          -0.003004214260727167,
+          0.0008957519312389195,
+          0.01322169229388237,
+          0.03291642293334007,
+          0.003939339891076088,
+          0.022537512704730034,
+          -0.02419120818376541,
+          0.018631640821695328,
+          0.007677873596549034,
+          0.016647204756736755,
+          -0.019986096769571304,
+          -0.017418930307030678,
+          -0.0007298901909962296,
+          0.013709926046431065,
+          -0.008575594052672386,
+          0.010906518436968327,
+          -0.003833030816167593,
+          0.039216216653585434,
+          -0.026396136730909348,
+          0.007047894410789013,
+          0.010835645720362663,
+          -0.004051554948091507,
+          0.009118951857089996,
+          0.02208077721297741,
+          0.017985912039875984,
+          0.005118582397699356,
+          -0.02297849766910076,
+          0.014135162346065044,
+          0.02293124981224537,
+          -0.01734018325805664,
+          0.0019470300758257508,
+          0.006539973430335522,
+          -0.0015119505114853382,
+          0.0004030883719678968,
+          0.005835183896124363,
+          -0.00024300962104462087,
+          -0.008331477642059326,
+          0.004512227606028318,
+          -0.007961364462971687,
+          -0.035562336444854736,
+          -0.0027384415734559298,
+          0.012851579114794731,
+          -0.00626435736194253,
+          0.031451720744371414,
+          0.029735026881098747,
+          0.019671106711030006,
+          -0.009811929427087307,
+          -0.006874650251120329,
+          0.004177550785243511,
+          -0.0040751793421804905,
+          -0.0011890861205756664,
+          -0.013607555069029331,
+          -0.014308406971395016,
+          0.014111538417637348,
+          -0.022474514320492744,
+          -0.00888270977884531,
+          0.0018338306108489633,
+          -0.005532006733119488,
+          -0.00459491228684783,
+          -0.0025002306792885065,
+          0.017481928691267967,
+          -0.005724937655031681,
+          0.02037983387708664,
+          0.01048915646970272,
+          -0.011528623290359974,
+          0.011308129876852036,
+          -0.026900120079517365,
+          0.009567812085151672,
+          -0.026963118463754654,
+          -0.010756897740066051,
+          0.009213448502123356,
+          -0.003992494195699692,
+          -0.017985912039875984,
+          -0.010473407804965973,
+          -0.016095973551273346,
+          -0.005220954306423664,
+          -0.00745738111436367,
+          -0.0006659079226665199,
+          0.013607555069029331,
+          0.008150358684360981,
+          -0.01601722650229931,
+          -0.016599956899881363,
+          -0.03694828972220421,
+          0.007843242958188057,
+          0.012993324548006058,
+          -0.025246424600481987,
+          -0.014119412750005722,
+          -0.007567627355456352,
+          0.008433849550783634,
+          -0.01233972143381834,
+          -0.0014351718127727509,
+          -0.0033802331890910864,
+          0.014883263036608696,
+          -0.02493143454194069,
+          0.006827401462942362,
+          -0.01929311826825142,
+          0.01558411493897438,
+          0.009583561681210995,
+          0.009260697290301323,
+          -0.03612931817770004,
+          -0.01885213330388069,
+          0.023561229929327965,
+          -0.008205481804907322,
+          -0.009504813700914383,
+          0.024049462750554085,
+          -0.012969700619578362,
+          0.011134886182844639,
+          0.019875850528478622,
+          0.01145775057375431,
+          -0.0026537880767136812,
+          0.0218287855386734,
+          -0.006213171873241663,
+          7.259527774294838e-05,
+          -0.004512227606028318,
+          0.024332953616976738,
+          0.0063588544726371765,
+          0.032790426164865494,
+          -0.015300624072551727,
+          -0.018521394580602646,
+          -0.005760374013334513,
+          0.0442875511944294,
+          0.007559752557426691,
+          -0.02723085880279541,
+          0.018174905329942703,
+          0.029136545956134796,
+          -0.0032916422933340073,
+          0.005445384420454502,
+          -0.038239747285842896,
+          0.0004163770063314587,
+          0.015505367890000343,
+          0.021923283115029335,
+          0.011292380280792713,
+          -0.009922175668179989,
+          -0.018363898620009422,
+          0.007181765045970678,
+          0.002352579263970256,
+          0.006039927247911692,
+          -0.03990919515490532,
+          0.011449875310063362,
+          -0.020946815609931946,
+          0.01970260590314865,
+          0.00861496850848198,
+          -0.009378817863762379,
+          0.022458765655755997,
+          0.0028152205049991608,
+          -0.017088191583752632,
+          0.014710018411278725,
+          -0.005980866961181164,
+          0.0006673844181932509,
+          0.03329440951347351,
+          -0.0036184440832585096,
+          0.012717708945274353,
+          -0.007815681397914886,
+          0.0061383615247905254,
+          0.0031046171206980944,
+          0.03505835309624672,
+          -0.0008091297931969166,
+          -0.014458026736974716,
+          0.0068628378212451935,
+          0.005417822860181332,
+          -0.0005571380024775863,
+          -0.005059522110968828,
+          0.0077448091469705105,
+          -0.012670460157096386,
+          0.0014400935033336282,
+          -0.003053431399166584,
+          -0.006315543316304684,
+          -0.0030396506190299988,
+          0.002159648109227419,
+          0.00047543758410029113,
+          -0.033136915415525436,
+          0.02178153768181801,
+          -0.0042484235018491745,
+          0.013843797147274017,
+          -0.005913931410759687,
+          -0.01531637366861105,
+          0.0023643914610147476,
+          0.02074207179248333,
+          -0.005492632742971182,
+          -0.011898735538125038,
+          -0.01595422811806202,
+          -0.02238001860678196,
+          -0.026301639154553413,
+          0.028522316366434097,
+          0.0024254205636680126,
+          -0.01607234962284565,
+          0.008481097407639027,
+          -0.009591436013579369,
+          -0.025545664131641388,
+          0.006520286668092012,
+          0.018395397812128067,
+          -0.041641637682914734,
+          0.016048725694417953,
+          -0.006953397300094366,
+          -0.0025573226157575846,
+          0.028648313134908676,
+          0.006732904817909002,
+          -0.003504260443150997,
+          0.014450152404606342,
+          -0.0027384415734559298,
+          0.011945984326303005,
+          0.02165554091334343,
+          0.0023072995245456696,
+          0.0066384077072143555,
+          0.014198160730302334,
+          0.015521117486059666,
+          -0.002368328860029578,
+          -0.01837964914739132,
+          0.027624595910310745,
+          -0.021545294672250748,
+          0.02025383710861206,
+          0.021261803805828094,
+          -0.026427635923027992,
+          0.029388537630438805,
+          -0.00997729878872633,
+          0.019182872027158737,
+          0.0034491370897740126,
+          -0.00592968100681901,
+          -0.01233972143381834,
+          0.004949275404214859,
+          0.007484942674636841,
+          0.009300070814788342,
+          0.010331662371754646,
+          0.044004060328006744,
+          -0.01364692859351635,
+          0.028601063415408134,
+          0.03401888534426689,
+          -0.00010723184095695615,
+          0.02255326136946678,
+          -0.018584391102194786,
+          -0.02123030461370945,
+          0.007488879840821028,
+          -0.015780983492732048,
+          -0.03025476075708866,
+          0.0032483311370015144,
+          -0.01452889945358038,
+          -0.008284228853881359,
+          -0.016269218176603317,
+          -0.0019539203494787216,
+          -0.0022324894089251757,
+          -0.0009164231596514583,
+          0.004220861941576004,
+          0.020348334684967995,
+          0.026049647480249405,
+          0.02376597188413143,
+          0.01024504005908966,
+          0.00747706787660718,
+          0.01148924883455038,
+          -0.009930050000548363,
+          0.026175644248723984,
+          -0.0011713679414242506,
+          -0.0028624688275158405,
+          -0.0172929335385561,
+          0.007055769208818674,
+          -0.00029013503808528185,
+          -0.005110707599669695,
+          0.007425881922245026,
+          -0.022175274789333344,
+          -0.0015906979097053409,
+          -0.04687046632170677,
+          -0.002504168078303337,
+          -0.0007677873945795,
+          0.010394659824669361,
+          0.03965720161795616,
+          -0.007449506316334009,
+          -0.017922913655638695,
+          0.008284228853881359,
+          -0.01685194857418537,
+          0.004098803270608187,
+          -0.01814340613782406,
+          -0.015591990202665329,
+          -0.010756897740066051,
+          -0.02414396032691002,
+          -0.009315820410847664,
+          -0.013812297955155373,
+          0.015143129974603653,
+          -0.00876458827406168,
+          0.011229382827877998,
+          -0.018710387870669365,
+          0.00041736135608516634,
+          -0.006221046671271324,
+          -0.008433849550783634,
+          -0.002559291198849678,
+          -0.020348334684967995,
+          -0.012379094958305359,
+          0.022710757330060005,
+          -0.008780337870121002,
+          -0.029609031975269318,
+          -0.006311606150120497,
+          0.004232673905789852,
+          -0.014859639108181,
+          0.0037365653552114964,
+          0.00592968100681901,
+          -0.005917869042605162,
+          -0.008985080756247044,
+          0.00961506087332964,
+          0.018726136535406113,
+          0.012457842007279396,
+          -0.0028644376434385777,
+          0.00395705783739686,
+          0.02291550114750862,
+          0.02348248101770878,
+          -0.0454845130443573,
+          -0.005102833267301321,
+          0.017686670646071434,
+          0.016347965225577354,
+          -0.0070321448147296906,
+          0.018411148339509964,
+          -0.01507225725799799,
+          0.011827862821519375,
+          -0.02250601351261139,
+          -0.00612261239439249,
+          -0.026915868744254112,
+          0.00845747347921133,
+          0.01734018325805664,
+          -0.006768341176211834,
+          0.035215847194194794,
+          0.020600326359272003,
+          -0.016788950189948082,
+          0.020994063466787338,
+          -0.0013150819577276707,
+          0.011174259707331657,
+          -0.027089113369584084,
+          0.01364692859351635,
+          -0.009229198098182678,
+          0.008236980065703392,
+          0.014206035062670708,
+          -0.020049093291163445,
+          -0.025340920314192772,
+          -0.01222159992903471,
+          0.012174352072179317,
+          0.003065243363380432,
+          -0.0027896275278180838,
+          0.022222522646188736,
+          0.015143129974603653,
+          -0.01197748351842165,
+          -0.01740317977964878,
+          0.023561229929327965,
+          0.01276495773345232,
+          0.015418745577335358,
+          -0.0032581747509539127,
+          -0.0033290472347289324,
+          -0.017009442672133446,
+          0.0005093974177725613,
+          -0.02450619824230671,
+          -0.001176289631985128,
+          -0.002449044957756996,
+          0.021529546007514,
+          -0.007681811228394508,
+          -0.006224983837455511,
+          -0.007177827879786491,
+          -0.02808133140206337,
+          -0.000330000912072137,
+          0.00023365837114397436,
+          0.01252871472388506,
+          0.007658186834305525,
+          0.009473315440118313,
+          0.017481928691267967,
+          -0.026364637538790703,
+          0.002179334871470928,
+          -0.019482113420963287,
+          0.004126364830881357,
+          -0.00928432121872902,
+          -0.014292657375335693,
+          0.01033953670412302,
+          0.003470792667940259,
+          -0.00480753043666482,
+          0.01012691855430603,
+          0.003260143334046006,
+          -0.017387431114912033,
+          0.029939770698547363,
+          -0.004006274975836277,
+          0.002279737964272499,
+          0.003663723822683096,
+          0.005736749619245529,
+          0.0019736073445528746,
+          0.02827032469213009,
+          -0.025388170033693314,
+          -0.016773201525211334,
+          -0.018600141629576683,
+          0.0024372327607125044,
+          0.004988649394363165,
+          0.005386323668062687,
+          -0.0010079670464619994,
+          -0.00013190193567425013,
+          0.002657725475728512,
+          -0.0070321448147296906,
+          0.027876587584614754,
+          0.0040909284725785255,
+          -0.03382989391684532,
+          -0.024395952001214027,
+          -0.04038167744874954,
+          0.008307852782309055,
+          -0.01982860080897808,
+          -0.029766526073217392,
+          -0.010008797980844975,
+          -0.02274225652217865,
+          -0.035940323024988174,
+          0.018474144861102104,
+          0.020757820457220078,
+          -0.02025383710861206,
+          -0.008205481804907322,
+          -0.0028447506483644247,
+          -0.013142945244908333,
+          0.010048171505331993,
+          -0.0029805900994688272,
+          -0.016269218176603317,
+          0.010355286300182343,
+          0.014765141531825066,
+          0.0068628378212451935,
+          -0.00867009162902832,
+          -0.016347965225577354,
+          0.00028521331842057407,
+          0.023813221603631973,
+          0.0064651635475456715,
+          -0.01107188779860735,
+          0.007925928570330143,
+          0.006398227997124195,
+          -0.015111630782485008,
+          0.007437694352120161,
+          -0.023088743910193443,
+          -0.000766310840845108,
+          0.0025021994952112436,
+          0.014056415297091007,
+          -0.01644246280193329,
+          0.003551508765667677,
+          0.005618628580123186,
+          -0.0036873482167720795,
+          0.003545602783560753,
+          -0.018757635727524757,
+          0.00671715522184968,
+          -0.009748931042850018,
+          0.0019913255237042904,
+          0.028128579258918762,
+          -0.007232951000332832,
+          -0.005161893554031849,
+          -0.032412439584732056,
+          0.01642671227455139,
+          0.03360940143465996,
+          0.013678427785634995,
+          -0.021513795480132103,
+          -0.015985727310180664,
+          0.004283859860152006,
+          -0.0016468055546283722,
+          -0.019198622554540634,
+          0.015473868697881699,
+          0.019450614228844643,
+          0.012197976000607014,
+          0.015505367890000343,
+          0.013166569173336029,
+          0.014379279688000679,
+          -0.025451166555285454,
+          0.001903718919493258,
+          -0.0020454642362892628,
+          -0.02170279063284397,
+          0.012024731375277042,
+          0.02061607502400875,
+          -0.02378172241151333,
+          -0.003651911858469248,
+          0.0016153064789250493,
+          0.02165554091334343,
+          -0.003827124834060669,
+          -0.002248239004984498,
+          0.01808040775358677,
+          0.010252914391458035,
+          0.015119505114853382,
+          0.004795718006789684,
+          0.061045002192258835,
+          -0.003384170588105917,
+          0.007122704293578863,
+          0.008575594052672386,
+          -0.005106770433485508,
+          -0.01965535618364811,
+          -0.0010197791270911694,
+          -0.026537882164120674,
+          0.006654157303273678,
+          0.007752683945000172,
+          -0.02778209187090397,
+          -0.005902119446545839,
+          -0.009008705615997314,
+          -0.006595096550881863,
+          0.018048910424113274,
+          0.0047484696842730045,
+          -0.0020179026760160923,
+          0.006260420195758343,
+          0.037987757474184036,
+          0.014686394482851028,
+          -0.009504813700914383,
+          0.0023072995245456696,
+          0.010150542482733727,
+          0.007760558743029833,
+          -0.008268479257822037,
+          0.009040203876793385,
+          0.0004434464208316058,
+          0.004461041651666164,
+          -0.00800467561930418,
+          0.039625704288482666,
+          -0.009040203876793385,
+          0.0009188840049318969,
+          0.0023801408242434263,
+          -0.014426528476178646,
+          0.010938017629086971,
+          0.002990433480590582,
+          0.00729201128706336,
+          -0.007095142733305693,
+          0.0013308314373716712,
+          0.002222646027803421,
+          -0.0020001844968646765,
+          -0.0012343658600002527,
+          -0.009559936821460724,
+          -0.01655270904302597,
+          -0.005587129853665829,
+          0.013016949407756329,
+          0.0005856839707121253,
+          0.0052367039024829865,
+          -0.01812765747308731,
+          -0.0072447629645466805,
+          -0.0007126641576178372,
+          -0.011134886182844639,
+          0.007221138570457697,
+          0.024899935349822044,
+          -0.027498601004481316,
+          -0.004283859860152006,
+          0.014166661538183689,
+          -0.00015171183622442186,
+          0.007327447645366192,
+          0.014607647433876991,
+          -0.013119320385158062,
+          0.008000737987458706,
+          -0.016040850430727005,
+          -0.014347780495882034,
+          0.0007456396706402302,
+          -0.006492725107818842,
+          0.0023506106808781624,
+          -0.01307207252830267,
+          -0.0012058198917657137,
+          0.01279645599424839,
+          0.0006412993534468114,
+          0.012079854495823383,
+          0.014544649049639702,
+          -0.00679196510463953,
+          -0.00038364759529940784,
+          0.012694085016846657,
+          -0.012780707329511642,
+          -0.024128209799528122,
+          0.0007146328571252525,
+          0.023876218125224113,
+          -0.00492958864197135,
+          0.00383499963209033,
+          -0.0028506566304713488,
+          0.0024825125001370907,
+          -0.012662585824728012,
+          -0.013001199811697006,
+          -0.012213725596666336,
+          0.0209153164178133,
+          0.004571287892758846,
+          -0.022758005186915398,
+          0.0008258636225946248,
+          0.0055831922218203545,
+          0.009103202261030674,
+          -0.017497677356004715,
+          0.004000368993729353,
+          0.02995551936328411,
+          -0.01479664072394371,
+          -0.0041814884170889854,
+          -0.0008224184275604784,
+          -0.013812297955155373,
+          0.008044049143791199,
+          0.017088191583752632,
+          0.005484757944941521,
+          0.016363713890314102,
+          -0.003575132926926017,
+          -0.009441816247999668,
+          0.03038075566291809,
+          0.015308499336242676,
+          0.004185425583273172,
+          -0.006536036264151335,
+          -0.012788581661880016,
+          -0.028333323076367378,
+          0.004374419339001179,
+          0.010623027570545673,
+          -0.014292657375335693,
+          -0.0005876526702195406,
+          -0.012276723049581051,
+          -0.018001660704612732,
+          -0.004488603211939335,
+          -0.006524223834276199,
+          0.012300347909331322,
+          -0.00976468063890934,
+          0.010804146528244019,
+          -0.0003120366600342095,
+          -0.0035377279855310917,
+          -0.002352579263970256,
+          0.0002889046154450625,
+          -0.009410317055881023,
+          -0.03291642293334007,
+          -0.028412070125341415,
+          -0.008898458443582058,
+          -0.008142483420670033,
+          0.006528161466121674,
+          0.0009907410712912679,
+          -0.030664246529340744,
+          0.010300163179636002,
+          -0.00012433479423634708,
+          -0.014843889512121677,
+          0.015190377831459045,
+          0.009898550808429718,
+          -0.002703005215153098,
+          -0.007528253830969334,
+          -0.007807807065546513,
+          -0.02874280884861946,
+          0.01589910499751568,
+          0.0021872096695005894,
+          0.009260697290301323,
+          -0.014237534254789352,
+          0.0010424189968034625,
+          -0.021009812131524086,
+          0.01727718487381935,
+          -0.026474883779883385,
+          -0.015095881186425686,
+          -0.01522187702357769,
+          -0.0022128026466816664,
+          -0.0060320524498820305,
+          -0.017781168222427368,
+          -0.021907532587647438,
+          -0.006161985918879509,
+          -0.025640161707997322,
+          -0.022899750620126724,
+          -0.0017609893111512065,
+          -0.0034353563096374273,
+          0.010583654046058655,
+          -0.007040019612759352,
+          0.027986833825707436,
+          0.01619047112762928,
+          0.006579347420483828,
+          -0.011560121551156044,
+          -0.013442184776067734,
+          -0.0020198714919388294,
+          0.0025179488584399223,
+          -0.009158325381577015,
+          0.002384078223258257,
+          0.019797101616859436,
+          -0.0036184440832585096,
+          0.0045240395702421665,
+          0.004106678068637848,
+          -0.004035805352032185,
+          0.016836199909448624,
+          0.007988926023244858,
+          0.011788489297032356,
+          0.006315543316304684,
+          0.0016094004968181252,
+          0.03379839286208153,
+          0.016473960131406784,
+          -0.0035869451239705086,
+          0.0022324894089251757,
+          0.020962564274668694,
+          -0.02170279063284397,
+          -0.01571011170744896,
+          0.015450244769454002,
+          0.001383001683279872,
+          -0.03546784073114395,
+          0.02067907340824604,
+          0.01661570556461811,
+          -0.009796179831027985,
+          -0.010764773003757,
+          -0.007961364462971687,
+          -0.00021323325927369297,
+          -0.005181580316275358,
+          0.008544095791876316,
+          -0.011560121551156044,
+          0.013426436111330986,
+          0.006496662274003029,
+          -0.0064651635475456715,
+          -0.014269033446907997,
+          0.01422178465873003,
+          0.027971085160970688,
+          0.0035790703259408474,
+          -0.04819342494010925,
+          -0.012276723049581051,
+          0.01704094186425209,
+          -0.02140354923903942,
+          -0.025797655805945396,
+          0.00011172290396643803,
+          -0.0075715649873018265,
+          -0.023608477786183357,
+          -0.007063644006848335,
+          -0.00392752792686224,
+          -0.016836199909448624,
+          0.012883078306913376,
+          0.01064665149897337,
+          0.012426343746483326,
+          0.008205481804907322,
+          -0.015867605805397034,
+          -0.012064104899764061,
+          -0.042366113513708115,
+          0.00788655411452055,
+          -0.024836936965584755,
+          -0.021939031779766083,
+          0.004902027081698179,
+          -0.006827401462942362,
+          0.008740964345633984,
+          0.01452889945358038,
+          0.039153218269348145,
+          0.01255233958363533,
+          -0.015080131590366364,
+          -0.025687409564852715,
+          -0.017907164990901947,
+          -0.007961364462971687,
+          0.0022758005652576685,
+          -0.0074180071242153645,
+          -0.0035003230441361666,
+          0.0033192038536071777,
+          0.002246270189061761,
+          0.029309790581464767,
+          0.001104432623833418,
+          0.018962379544973373,
+          -3.389830453670584e-05,
+          0.00928432121872902,
+          -0.0038920913357287645,
+          -0.025025932118296623,
+          0.006213171873241663,
+          0.004106678068637848,
+          0.00794167723506689,
+          0.0068628378212451935,
+          0.005106770433485508,
+          0.016584208235144615,
+          -0.015867605805397034,
+          0.005394198466092348,
+          -0.026222892105579376,
+          -0.006469100713729858,
+          -0.014418653212487698,
+          0.0294042881578207,
+          0.014733643271028996,
+          -0.02327773906290531,
+          -0.013174444437026978,
+          -0.018993878737092018,
+          0.017970161512494087,
+          -0.008685841225087643,
+          0.009756805375218391,
+          0.01355243194848299,
+          0.01040253508836031,
+          0.006205297075212002,
+          0.0053548249416053295,
+          -0.005012273322790861,
+          0.007736934348940849,
+          0.02310449443757534,
+          -0.007106955163180828,
+          0.008000737987458706,
+          -0.0013849702663719654,
+          -0.0041814884170889854,
+          -0.014812390320003033,
+          -0.01722993701696396,
+          -0.0012570057297125459,
+          -0.00373853393830359,
+          0.01935611665248871,
+          -0.017623674124479294,
+          0.020001845434308052,
+          -0.011906610801815987,
+          0.005181580316275358,
+          -0.026490632444620132,
+          -0.008481097407639027,
+          0.01644246280193329,
+          -0.015505367890000343,
+          0.0024195145815610886,
+          -0.015458119101822376,
+          -0.003754283534362912,
+          0.006665969267487526,
+          0.001975575927644968,
+          0.001650742837227881,
+          0.005575317423790693,
+          0.005201267544180155,
+          0.016662955284118652,
+          0.008110984228551388,
+          -0.017450429499149323,
+          -0.0032562059350311756,
+          0.008370851166546345,
+          -0.015135254710912704,
+          0.006961272098124027,
+          0.005862745922058821,
+          0.0009361099801026285,
+          0.04082266613841057,
+          0.0066384077072143555,
+          0.002084837993606925,
+          -0.009898550808429718,
+          0.0037798762787133455,
+          -0.036727797240018845,
+          0.02904205024242401,
+          -0.0016576332272961736,
+          0.016221968457102776,
+          0.001489310641773045,
+          -0.014465902000665665,
+          0.0045791626907885075,
+          -0.005154018756002188,
+          0.014828139916062355,
+          -0.006287981756031513,
+          0.006969146896153688,
+          -0.00131114455871284,
+          0.017387431114912033,
+          -0.007339260075241327,
+          -0.018899381160736084,
+          0.009142575785517693,
+          -0.011316005140542984,
+          0.0007323510362766683,
+          -0.009693807922303677,
+          0.0055280691012740135,
+          0.01352880708873272,
+          0.0070203328505158424,
+          0.005480820778757334,
+          -0.03808225318789482,
+          -0.008622842840850353,
+          -0.003271955531090498,
+          0.002777815330773592,
+          -0.0024687317200005054,
+          -0.023970715701580048,
+          -0.024584945291280746,
+          -0.0002655264688655734,
+          0.02170279063284397,
+          -0.006587222218513489,
+          -0.0011831800220534205,
+          0.009118951857089996,
+          0.0015503398608416319,
+          -0.011749115772545338,
+          -0.006980959326028824,
+          0.007421944756060839,
+          0.021246055141091347,
+          -0.025514164939522743,
+          -0.020883817225694656,
+          0.012048356235027313,
+          0.0029274355620145798,
+          -0.019875850528478622,
+          -0.01033953670412302,
+          -0.0020651512313634157,
+          0.008512596599757671,
+          0.03439687564969063,
+          -0.017686670646071434,
+          0.001204835600219667,
+          0.0008947675814852118,
+          0.014080039225518703,
+          -0.026301639154553413,
+          -0.005043772514909506,
+          -0.01112701091915369,
+          -0.010418283753097057,
+          0.01148924883455038,
+          -0.01631646603345871,
+          0.0070518315769732,
+          -0.002840813249349594,
+          -0.002578978193923831,
+          -0.014064289629459381,
+          0.0009021501755341887,
+          0.0013810329837724566,
+          0.0016064473893493414,
+          0.03316841647028923,
+          -0.009567812085151672,
+          -0.013505183160305023,
+          0.0018968285294249654,
+          0.00041391613194718957,
+          0.014922636561095715,
+          0.010465532541275024,
+          0.01976560428738594,
+          -0.008331477642059326,
+          0.001558214658871293,
+          0.00961506087332964,
+          0.014623397029936314,
+          0.0005468024173751473,
+          0.042240116745233536,
+          -0.005795810371637344,
+          -0.006725030019879341,
+          -0.011788489297032356,
+          -0.005390261299908161,
+          -0.01233972143381834,
+          0.01491476222872734,
+          0.00269119325093925,
+          0.010371035896241665,
+          -0.0020828694105148315,
+          -0.008945707231760025,
+          0.012111353687942028,
+          0.007445569150149822,
+          0.010560029186308384,
+          0.013205942697823048,
+          -0.004014149773865938,
+          -0.02001759596168995,
+          0.010063921101391315,
+          -0.010048171505331993,
+          0.006059614475816488,
+          -0.001052262494340539,
+          0.008866960182785988,
+          -0.007996801286935806,
+          0.018631640821695328,
+          -0.0028703436255455017,
+          0.017607923597097397,
+          0.015560491010546684,
+          -0.019545109942555428,
+          0.017072441056370735,
+          0.01782841607928276,
+          -0.010788396932184696,
+          0.012780707329511642,
+          -0.01648971065878868,
+          -0.010237164795398712,
+          -0.0052524530328810215,
+          0.02176578715443611,
+          -0.013142945244908333,
+          -0.013324064202606678,
+          -0.010922268033027649,
+          0.00395508948713541,
+          -0.007988926023244858,
+          0.017324432730674744,
+          0.00588243268430233,
+          -0.009890676476061344,
+          -0.019686855375766754,
+          0.015505367890000343,
+          0.004433480091392994,
+          0.005213079508394003,
+          0.022474514320492744,
+          -0.007898367010056973,
+          0.005878495052456856,
+          0.014717893674969673,
+          0.030349256470799446,
+          -0.008323602378368378,
+          0.02831757441163063,
+          -0.011111261323094368,
+          0.00266756908968091,
+          -0.0037601895164698362,
+          -0.016158971935510635,
+          -0.012402718886733055,
+          -8.889845776138827e-05,
+          0.0050280229188501835,
+          0.012930327095091343,
+          -0.028601063415408134,
+          0.00861496850848198,
+          0.026222892105579376,
+          0.0027502537705004215,
+          0.008937832899391651,
+          0.0026537880767136812,
+          0.006890399381518364,
+          0.005386323668062687,
+          0.0029943708796054125,
+          -0.022364268079400063,
+          -0.007126641925424337,
+          0.00323848775587976,
+          -0.010851395316421986,
+          0.02104131132364273,
+          0.014355655759572983,
+          0.01067027635872364,
+          -0.002246270189061761,
+          -0.0012619274202734232,
+          0.025151927024126053,
+          0.01959235966205597,
+          0.004051554948091507,
+          -0.001164477551355958,
+          0.006854963023215532,
+          0.0023191117215901613,
+          0.011875111609697342,
+          -0.014190285466611385,
+          -0.014166661538183689,
+          0.007331385277211666,
+          0.00262228911742568,
+          0.010890768840909004,
+          -0.01680470071732998,
+          -0.006449413951486349,
+          -0.003303454490378499,
+          0.01746617816388607,
+          -0.009993048384785652,
+          -0.0018879694398492575,
+          0.0012599588371813297,
+          0.015253376215696335,
+          0.00039250668487511575,
+          -0.0032680181320756674,
+          -0.004724845290184021,
+          -0.014119412750005722,
+          0.0047996556386351585,
+          -0.009725307114422321,
+          0.004894152283668518,
+          -0.005661939736455679,
+          0.00881971139460802,
+          -0.007311698514968157,
+          0.04378356784582138,
+          0.027876587584614754,
+          0.008512596599757671,
+          0.017371682450175285,
+          -0.003986588213592768,
+          0.0017127564642578363,
+          -0.016119597479701042,
+          0.020222337916493416,
+          0.005598941817879677,
+          4.361868923297152e-05,
+          0.003966901451349258,
+          0.024915684014558792,
+          0.004728782922029495,
+          0.009733181446790695,
+          0.0006260420195758343,
+          0.006394290830940008,
+          0.0026439446955919266,
+          0.009410317055881023,
+          -0.018474144861102104,
+          -0.01507225725799799,
+          -0.014497400261461735,
+          0.02316749282181263,
+          -0.005512319505214691,
+          -0.008567719720304012,
+          0.013733550906181335,
+          0.011111261323094368,
+          -0.002498262096196413,
+          -0.014670644886791706,
+          -0.005693438928574324,
+          -0.0008736042072996497,
+          -0.005161893554031849,
+          -0.016269218176603317,
+          0.00297862128354609,
+          0.0020316834561526775,
+          -0.012182226404547691,
+          0.021372050046920776,
+          -0.012166476808488369,
+          -0.020395582541823387,
+          -0.003490479663014412,
+          -0.006843151059001684,
+          -0.04022418335080147,
+          -0.019056877121329308,
+          0.011906610801815987,
+          0.010788396932184696,
+          -0.008591343648731709,
+          -0.03480635955929756,
+          0.021261803805828094,
+          0.0060832384042441845,
+          0.03808225318789482,
+          -0.026065398007631302,
+          0.0023762034252285957,
+          0.0006777200032956898,
+          0.00397477624937892,
+          -0.011268756352365017,
+          -4.0081209590425715e-05,
+          0.019419115036725998,
+          -0.0007800916791893542,
+          0.00812673382461071,
+          -0.00542963482439518,
+          -0.010922268033027649,
+          0.014686394482851028,
+          -0.007209326606243849,
+          0.012906703166663647,
+          -0.006161985918879509,
+          -0.014442277140915394,
+          0.0032424251548945904,
+          0.016631456092000008,
+          0.012607462704181671,
+          -0.0017993786605075002,
+          -0.014670644886791706,
+          -0.009567812085151672,
+          0.00433504581451416,
+          -0.013331938534975052,
+          -0.012890953570604324,
+          0.02378172241151333,
+          -0.0166787039488554,
+          0.018111906945705414,
+          0.013205942697823048,
+          -0.017182687297463417,
+          0.004839029163122177,
+          0.006035990081727505,
+          0.006008428521454334,
+          0.007567627355456352,
+          -0.015064381994307041,
+          -0.019434863701462746,
+          0.004476791247725487,
+          0.026128394529223442,
+          -0.00240376521833241,
+          -0.006020240485668182,
+          0.012544464319944382,
+          -0.0026951306499540806,
+          0.008512596599757671,
+          -0.023387985303997993,
+          -0.0023821096401661634,
+          0.010993140749633312,
+          0.015568365342915058,
+          0.015686485916376114,
+          0.02335648611187935,
+          0.009181949310004711,
+          0.012402718886733055,
+          -0.012323971837759018,
+          0.005177643150091171,
+          0.022726505994796753,
+          -0.009630809538066387,
+          -0.018883632495999336,
+          0.011379002593457699,
+          0.01763942278921604,
+          0.023687224835157394,
+          0.0018456426914781332,
+          -0.010268663987517357,
+          0.030916238203644753,
+          0.012284598313272,
+          -0.009063828736543655,
+          0.014135162346065044,
+          -0.010906518436968327,
+          0.024962933734059334,
+          0.010237164795398712,
+          0.016174720600247383,
+          -0.01219010166823864,
+          -0.010835645720362663,
+          0.006646282505244017,
+          -0.011567996814846992,
+          -0.007410132791846991,
+          0.01249721646308899,
+          -0.020175090059638023,
+          -0.003384170588105917,
+          -0.031325723975896835,
+          0.00638247886672616,
+          0.003254237351939082,
+          0.0035790703259408474,
+          -0.02461644448339939,
+          0.005658002570271492,
+          -0.006146236322820187,
+          0.008079485967755318,
+          -0.011048263870179653,
+          -0.031262725591659546,
+          0.0039058721158653498,
+          -0.006496662274003029,
+          0.002059245016425848,
+          -0.01433990616351366,
+          0.01589122973382473,
+          0.007114829961210489,
+          -0.021088559180498123,
+          0.0010315912077203393,
+          -0.01904112659394741,
+          -0.0072565749287605286,
+          0.0003491956158541143,
+          -0.0023880156222730875,
+          -0.016347965225577354,
+          0.011363252997398376,
+          0.005886369850486517,
+          -0.02527792379260063,
+          -0.006937648169696331,
+          -0.0024687317200005054,
+          0.0007170937024056911,
+          0.016048725694417953,
+          -0.006055676843971014,
+          0.0022324894089251757,
+          0.033388908952474594,
+          0.015867605805397034,
+          -0.008488972671329975,
+          -0.025624411180615425,
+          -0.0028644376434385777,
+          -0.02359272725880146,
+          0.015631362795829773,
+          -0.0007953489548526704,
+          0.005650127772241831,
+          -0.029971269890666008,
+          0.005150081589818001,
+          0.006709280423820019,
+          0.01746617816388607,
+          0.006488787941634655,
+          0.01033953670412302,
+          0.026837121695280075,
+          -0.005106770433485508,
+          -0.001383001683279872,
+          -1.0535543879086617e-05,
+          0.00679196510463953,
+          0.018411148339509964,
+          -0.0036184440832585096,
+          0.01812765747308731,
+          -0.009607185609638691,
+          0.0029136547818779945,
+          -0.012654710561037064,
+          0.0035554461646825075,
+          -0.003462917869910598,
+          -0.013095696456730366,
+          -9.954166307579726e-05,
+          0.012686209753155708,
+          0.003395982552319765,
+          -0.0012727552093565464,
+          -0.018836384639143944,
+          -0.003833030816167593,
+          -0.0012530683306977153,
+          -0.021261803805828094,
+          0.0029628719203174114,
+          -0.01970260590314865,
+          -0.01124513242393732,
+          -0.02778209187090397,
+          -0.0025435418356209993,
+          0.04438204690814018,
+          0.00955206248909235,
+          -0.012331846170127392,
+          0.002647882094606757,
+          -0.02067907340824604,
+          0.0009956627618521452,
+          -0.012213725596666336,
+          -0.0036479744594544172,
+          -0.006886462215334177,
+          -0.013820173218846321,
+          0.014631271362304688,
+          0.01735593192279339,
+          -0.00404761778190732,
+          -0.006701405625790358,
+          0.012481466867029667,
+          -0.01070177461951971,
+          -0.009504813700914383,
+          -0.017576424404978752,
+          -0.017922913655638695,
+          -0.00785505585372448,
+          0.018883632495999336,
+          -0.005795810371637344,
+          0.02127755433320999,
+          0.008544095791876316,
+          0.0034353563096374273,
+          -0.00692977337166667,
+          0.0015168722020462155,
+          -0.021498046815395355,
+          -0.0006855947431176901,
+          0.006287981756031513,
+          -0.01433990616351366,
+          0.016599956899881363,
+          -0.006815589498728514,
+          -0.0031361160799860954,
+          0.008725214749574661,
+          0.004413793329149485,
+          -0.008725214749574661,
+          -0.005516257137060165,
+          -0.017135439440608025,
+          0.018820634111762047,
+          -0.018332399427890778,
+          -0.0027502537705004215,
+          0.00022246147273108363,
+          -0.009969423525035381,
+          0.012158602476119995,
+          0.015591990202665329,
+          0.007481005042791367,
+          -0.010378910228610039,
+          0.006610846146941185,
+          -0.010292287915945053,
+          0.0221437755972147,
+          -0.001872219960205257,
+          0.01612747274339199,
+          0.013024823740124702,
+          0.005433572456240654,
+          0.006284044589847326,
+          0.001713740755803883,
+          -0.012544464319944382,
+          -0.006949460133910179,
+          0.011064013466238976,
+          -0.02571890875697136,
+          -0.01349730882793665,
+          0.005917869042605162,
+          0.002801439492031932,
+          -0.01148924883455038,
+          -0.0007451475248672068,
+          -0.009205574169754982,
+          0.00037798762787133455,
+          -0.008481097407639027,
+          0.016710203140974045,
+          -0.03433387726545334,
+          -0.00861496850848198,
+          0.0017580362036824226,
+          0.031136730685830116,
+          -0.008449598215520382,
+          0.01619047112762928,
+          0.007532190997153521,
+          0.006488787941634655,
+          0.023687224835157394,
+          0.001509981811977923,
+          0.007662124466150999,
+          0.02116730809211731,
+          -0.011095511727035046,
+          0.00785505585372448,
+          0.0058036851696670055,
+          -0.01995459757745266,
+          0.025120427832007408,
+          -0.0061856103129684925,
+          0.005906056612730026,
+          0.002665600273758173,
+          0.001731458934955299,
+          0.02571890875697136,
+          0.027624595910310745,
+          -0.00870158988982439,
+          -0.005665877368301153,
+          -0.013379187323153019,
+          0.027262357994914055,
+          -0.004945338238030672,
+          0.007610938511788845,
+          -0.010197791270911694,
+          0.002689224435016513,
+          0.0007668030448257923,
+          -0.01024504005908966,
+          0.021482296288013458,
+          -0.016032975167036057,
+          -0.011875111609697342,
+          -0.017560675740242004,
+          -0.002472669119015336,
+          -0.024222707375884056,
+          0.013536682352423668,
+          0.0025829155929386616,
+          -0.003352671628817916,
+          0.0023821096401661634,
+          0.006484850309789181,
+          -0.003496385645121336,
+          -0.00940244272351265,
+          -0.0020887753926217556,
+          -0.01170186698436737,
+          -0.020348334684967995,
+          0.0166787039488554,
+          -0.007248700130730867,
+          -0.002116336952894926,
+          -0.0012127102818340063,
+          0.009811929427087307,
+          0.014985634945333004,
+          0.02129330299794674,
+          0.002915623364970088,
+          0.0066344705410301685,
+          0.007311698514968157,
+          -0.0019332492956891656,
+          -0.012032606638967991,
+          0.020427081733942032,
+          0.0197183545678854,
+          0.0003445199690759182,
+          -0.0031695838551968336,
+          -0.0109143927693367,
+          -0.035562336444854736,
+          -0.009733181446790695,
+          0.0008125749882310629,
+          0.011938109062612057,
+          -0.012442093342542648,
+          -0.007103017531335354,
+          0.039027221500873566,
+          -0.003220769576728344,
+          0.02110430970788002,
+          -0.004189362749457359,
+          3.842013029498048e-05,
+          0.01674170233309269,
+          0.004606724251061678,
+          0.012142852880060673,
+          -0.01264683622866869,
+          -0.02327773906290531,
+          0.002559291198849678,
+          -0.012961826287209988,
+          0.002697099233046174,
+          0.0004523055104073137,
+          -0.0002670029934961349,
+          -0.019497862085700035,
+          -0.008504721336066723,
+          0.005189455114305019,
+          -0.01128450594842434,
+          0.010040296241641045,
+          -0.004157864023000002,
+          0.017387431114912033,
+          -0.0030179950408637524,
+          0.0037897196598351,
+          -0.008583469316363335,
+          -0.006811652332544327,
+          0.0011575871612876654,
+          -0.0017304746434092522,
+          -0.017198437824845314,
+          -8.342305227415636e-05,
+          -0.0020887753926217556,
+          -0.0003410747740417719,
+          0.008410224691033363,
+          0.008378725498914719,
+          0.01328469067811966,
+          0.01385954674333334,
+          -0.0009119936148636043,
+          0.006996708456426859,
+          0.009008705615997314,
+          0.02735685557126999,
+          -0.007185702212154865,
+          0.0068077147006988525,
+          0.002291549928486347,
+          0.008012549951672554,
+          -0.012946076691150665,
+          -0.03285342454910278,
+          0.015426619909703732,
+          -0.015780983492732048,
+          -0.0048035928048193455,
+          -0.014717893674969673,
+          0.01040253508836031,
+          -0.004445292055606842,
+          0.0010876987362280488,
+          0.006098988000303507,
+          0.00845747347921133,
+          0.004937463440001011,
+          0.019308868795633316,
+          -0.007213264238089323,
+          0.0071935770101845264,
+          0.021309053525328636,
+          -0.003295579692348838,
+          -0.002515980275347829,
+          0.006563597824424505,
+          0.009323694743216038,
+          -0.01644246280193329,
+          0.01103251427412033,
+          0.024647943675518036,
+          -0.023198990151286125,
+          0.004441354889422655,
+          0.002891999203711748,
+          -0.01070177461951971,
+          0.004126364830881357,
+          0.0065320986323058605,
+          -0.00033024701406247914,
+          0.023009996861219406,
+          0.004587037488818169,
+          0.019576609134674072,
+          0.004022024571895599,
+          -0.014670644886791706,
+          -0.015599864535033703,
+          0.006646282505244017,
+          -0.010890768840909004,
+          -0.011016764678061008,
+          -0.008063736371695995,
+          -0.010693900287151337,
+          0.0054611340165138245,
+          -0.004787843208760023,
+          -0.004022024571895599,
+          -0.0037739702966064215,
+          -0.011253006756305695,
+          0.006827401462942362,
+          -0.01148924883455038,
+          0.001115260412916541,
+          -0.00638247886672616,
+          -0.005280014593154192,
+          0.00945756584405899,
+          0.011851487681269646,
+          -0.010008797980844975,
+          -0.02674262598156929,
+          0.005555630661547184,
+          0.031199729070067406,
+          -0.006728967186063528,
+          0.007929865270853043,
+          0.012355471029877663,
+          -0.00745738111436367,
+          0.000588144815992564,
+          -0.006732904817909002,
+          -0.004650035407394171,
+          -0.03105798363685608,
+          -0.0010227321181446314,
+          -0.01067027635872364,
+          0.0003351687337271869,
+          -0.014245408587157726,
+          -0.015017134137451649,
+          -0.013166569173336029,
+          0.004968962632119656,
+          -0.013142945244908333,
+          -0.005043772514909506,
+          0.004149989224970341,
+          0.006988833658397198,
+          -0.016379464417696,
+          0.017576424404978752,
+          -0.00943394098430872,
+          -0.003933433908969164,
+          -0.00592968100681901,
+          0.033451907336711884,
+          -0.005406010430306196,
+          -0.003504260443150997,
+          -0.00492958864197135,
+          0.008874834515154362,
+          0.006776215974241495,
+          0.002053339034318924,
+          -0.0029038111679255962,
+          0.0021379925310611725,
+          0.02323048934340477,
+          0.006232858635485172,
+          0.003397951368242502,
+          0.015536867082118988,
+          -0.013473683968186378,
+          -0.008044049143791199,
+          -0.012654710561037064,
+          -0.01067027635872364,
+          0.0020454642362892628,
+          0.010985265485942364,
+          0.02365572564303875,
+          -0.005083146039396524,
+          -0.014985634945333004,
+          0.011473499238491058,
+          0.006772278342396021,
+          0.02140354923903942,
+          -0.011843612417578697,
+          5.6138298532459885e-05,
+          0.00812673382461071,
+          0.02376597188413143,
+          0.031813960522413254,
+          -0.000510381767526269,
+          7.437940075760707e-05,
+          0.0083393519744277,
+          0.010615152306854725,
+          -0.004980774596333504,
+          0.00664234533905983,
+          -0.01704094186425209,
+          -0.02795533463358879,
+          -0.008898458443582058,
+          0.006646282505244017,
+          -0.001707834773696959,
+          0.018647389486432076,
+          0.01007179543375969,
+          0.016757450997829437,
+          0.02382897026836872,
+          0.023025747388601303,
+          0.00039545970503240824,
+          0.02037983387708664,
+          -0.004425605293363333,
+          -0.004417730495333672,
+          0.0025691348128020763,
+          -0.015048632398247719,
+          -0.03471186384558678,
+          0.006118674762547016,
+          0.00442954245954752,
+          0.024773940443992615,
+          -0.015757359564304352,
+          0.023561229929327965,
+          -0.004146052058786154,
+          -0.016158971935510635,
+          -0.005898181814700365,
+          0.020789319649338722,
+          -0.008772462606430054,
+          0.020789319649338722,
+          0.0013239410473033786,
+          -0.0010945892427116632,
+          0.00988280214369297,
+          -0.0011201821034774184,
+          -0.01379654835909605,
+          0.011253006756305695,
+          0.01763942278921604,
+          0.03770426660776138,
+          0.009630809538066387,
+          -0.008118859492242336,
+          0.01752917654812336,
+          0.001776738790795207,
+          -0.008394475094974041,
+          -0.004433480091392994,
+          -0.013820173218846321,
+          0.0076936231926083565,
+          -0.0074180071242153645,
+          0.01697794534265995,
+          0.002640007296577096,
+          -0.005260327830910683,
+          0.014725768007338047,
+          -0.012481466867029667,
+          0.01219010166823864,
+          0.021246055141091347,
+          0.0011949921026825905,
+          0.005886369850486517,
+          0.020001845434308052,
+          -0.033136915415525436,
+          0.011528623290359974,
+          -0.034837860614061356,
+          -0.005138269625604153,
+          0.012064104899764061,
+          -0.01861589029431343,
+          0.007732997182756662,
+          -0.018584391102194786,
+          -0.013253191486001015,
+          -0.002553385216742754,
+          0.014599772170186043,
+          0.02497868239879608,
+          0.006555723026394844,
+          -0.017670921981334686,
+          0.01179636362940073,
+          -0.00876458827406168,
+          -0.01825365237891674,
+          -0.008874834515154362,
+          -0.019419115036725998,
+          -0.011693992651998997,
+          -0.003850748995319009,
+          0.011938109062612057,
+          -0.008811837062239647,
+          0.0037011289969086647,
+          0.002248239004984498,
+          -0.0221437755972147,
+          -0.013883170671761036,
+          0.015914853662252426,
+          0.003413700731471181,
+          -0.00664234533905983,
+          -0.007343197241425514,
+          0.013016949407756329,
+          -0.010756897740066051,
+          0.007323510479182005,
+          -0.008181856945157051,
+          0.014253283850848675,
+          -0.012142852880060673,
+          -0.00319123943336308,
+          0.007079393602907658,
+          0.0035869451239705086,
+          -0.018033159896731377,
+          -0.004732720088213682,
+          -0.02493143454194069,
+          0.015355747193098068,
+          0.0029766527004539967,
+          -0.017481928691267967,
+          -0.00692977337166667,
+          0.007732997182756662,
+          -0.017922913655638695,
+          -0.004894152283668518,
+          0.008717339485883713,
+          -0.003990525845438242,
+          0.025340920314192772,
+          0.022301269695162773,
+          -0.01248934119939804,
+          0.00016143222455866635,
+          -0.009213448502123356,
+          0.009985173121094704,
+          -0.01856864243745804,
+          -0.004484665580093861,
+          0.02650638297200203,
+          0.007315635681152344,
+          -0.00550444470718503,
+          -0.009937925264239311,
+          0.001173336640931666,
+          0.002771909348666668,
+          0.017607923597097397,
+          0.02493143454194069,
+          0.00621710903942585,
+          -0.010985265485942364,
+          -0.022458765655755997,
+          -0.027388354763388634,
+          0.027372604236006737,
+          0.0007485927199013531,
+          -0.004118490032851696,
+          0.013276815414428711,
+          0.02529367245733738,
+          0.023939216509461403,
+          -0.0062013594433665276,
+          -0.00483509199693799,
+          0.018300902098417282,
+          0.00997729878872633,
+          -0.005402073264122009,
+          -0.010166292078793049,
+          0.003392045386135578,
+          -0.0071345167234539986,
+          0.03414488211274147,
+          0.010111168958246708,
+          0.004417730495333672,
+          -0.046366482973098755,
+          0.009993048384785652,
+          -0.01619047112762928,
+          -0.003758220700547099,
+          0.008236980065703392,
+          0.021734289824962616,
+          0.010095419362187386,
+          0.013032699003815651,
+          0.007984988391399384,
+          -0.007236888166517019,
+          0.02861681394279003,
+          -0.0028703436255455017,
+          -0.010623027570545673,
+          -0.007272324524819851,
+          -0.025687409564852715,
+          0.004342920612543821,
+          -0.016410963609814644,
+          0.0021517733111977577,
+          0.005677689332515001,
+          -0.00918982457369566,
+          0.02590790204703808,
+          0.00794167723506689,
+          -0.02382897026836872,
+          0.00588243268430233,
+          -0.03682229667901993,
+          0.019482113420963287,
+          -0.02619139291346073,
+          -0.009323694743216038,
+          -0.004862653557211161,
+          -0.003758220700547099,
+          -0.025923652574419975,
+          -0.0037365653552114964,
+          -0.004783906042575836,
+          0.016883447766304016,
+          0.0024017964024096727,
+          -0.00013805407797917724,
+          -0.00013399365707300603,
+          -0.007103017531335354,
+          0.002041527070105076,
+          0.012410594150424004,
+          0.022521764039993286,
+          0.01571798510849476,
+          -0.0037897196598351,
+          0.018521394580602646,
+          -0.0005768248811364174,
+          -0.03704278916120529,
+          0.0037168783601373434,
+          -0.011662493459880352,
+          0.008103109896183014,
+          0.015332123264670372,
+          0.009591436013579369,
+          -0.01370205171406269,
+          0.013190193101763725,
+          -0.012150727212429047,
+          0.015395121648907661,
+          0.01619047112762928,
+          0.004894152283668518,
+          -0.0009976314613595605,
+          0.0157652348279953,
+          -0.02389196865260601,
+          0.00020941892580594867,
+          -0.00626435736194253,
+          0.006713218055665493,
+          -0.009930050000548363,
+          -0.0033251100685447454,
+          0.002041527070105076,
+          0.0006855947431176901,
+          -0.0034038573503494263,
+          -0.012064104899764061,
+          -0.017387431114912033,
+          -0.021498046815395355,
+          -0.01428478304296732,
+          -0.020820818841457367,
+          -0.011001015082001686,
+          0.030711494386196136,
+          -0.008622842840850353,
+          0.004559475928544998,
+          0.015025008469820023,
+          0.018710387870669365,
+          0.002596696373075247,
+          0.01856864243745804,
+          0.008260604925453663,
+          -0.007299886085093021,
+          -0.01727718487381935,
+          0.005543818697333336,
+          -0.025435417890548706,
+          0.03310541808605194,
+          0.00933944433927536,
+          -0.014032791368663311,
+          0.012969700619578362,
+          -0.01070177461951971,
+          -0.004346857778728008,
+          -0.007784182671457529,
+          0.0060163033194839954,
+          -0.025498416274785995,
+          0.01455252431333065,
+          -0.011749115772545338,
+          0.006961272098124027,
+          0.002305330941453576,
+          0.03505835309624672,
+          -0.014489525929093361,
+          -0.008929957635700703,
+          0.012032606638967991,
+          0.019119873642921448,
+          0.026553630828857422,
+          -0.009441816247999668,
+          -0.004937463440001011,
+          -0.01814340613782406,
+          -0.00357119576074183,
+          -0.005906056612730026,
+          -0.018631640821695328,
+          0.03590882569551468,
+          -0.007847180590033531,
+          -0.021387800574302673,
+          -0.00824485532939434,
+          -0.02006484381854534,
+          0.0031262726988643408,
+          0.0315462164580822,
+          0.01040253508836031,
+          0.006890399381518364,
+          -8.92060634214431e-05,
+          -0.020285336300730705,
+          -0.021324802190065384,
+          0.019797101616859436,
+          0.012166476808488369,
+          -0.019844351336359978,
+          -0.0009695776388980448,
+          -0.009394567459821701,
+          -0.00216752290725708,
+          0.0006609861738979816,
+          0.0021773662883788347,
+          0.05780060961842537,
+          -0.008087360300123692,
+          0.0013229567557573318,
+          -0.024065213277935982,
+          -0.001320003648288548,
+          -0.003332984633743763,
+          -0.0035318220034241676,
+          0.005646190140396357,
+          -0.00626435736194253,
+          0.01128450594842434,
+          -0.03480635955929756,
+          0.006784090772271156,
+          -0.010607277974486351,
+          -0.009654434397816658,
+          -0.016111722216010094,
+          0.00166747672483325,
+          -0.018285151571035385,
+          -4.6664001274621114e-05,
+          0.00021680150530301034,
+          -0.012134977616369724,
+          0.004083054140210152,
+          -0.013883170671761036,
+          -0.0003961979818996042,
+          0.024553446099162102,
+          0.011686117388308048,
+          -0.0013268940383568406,
+          -0.022222522646188736,
+          -0.022395767271518707,
+          0.0015562459593638778,
+          -0.010394659824669361,
+          0.01801741123199463,
+          -0.013229567557573318,
+          0.017182687297463417,
+          -0.02165554091334343,
+          -0.00046780891716480255,
+          0.0019765603356063366,
+          0.006232858635485172,
+          0.01861589029431343,
+          -0.00010969270078931004,
+          0.009693807922303677,
+          -0.014993509277701378,
+          0.02535667084157467,
+          -0.006949460133910179,
+          -0.02019083872437477,
+          0.0007328432402573526,
+          0.005181580316275358,
+          -0.0044216676615178585,
+          -0.0003430434735491872,
+          0.011056138202548027,
+          0.019419115036725998,
+          -0.0024766065180301666,
+          -0.007839306257665157,
+          -0.014064289629459381,
+          -0.009930050000548363,
+          0.0194033645093441,
+          0.0009297117940150201,
+          0.02055307850241661,
+          -0.011536497622728348,
+          0.00918982457369566,
+          -0.017623674124479294,
+          -0.007662124466150999,
+          -0.00907170306891203,
+          -0.004022024571895599,
+          0.016725953668355942,
+          -0.0007599126547574997,
+          0.01512738037854433,
+          -0.018001660704612732,
+          -0.021907532587647438,
+          0.0006098003359511495,
+          -0.02348248101770878,
+          -0.018411148339509964,
+          0.009205574169754982,
+          -0.01855289377272129,
+          -0.0025986649561673403,
+          0.001046356395818293,
+          -0.020883817225694656,
+          -0.013473683968186378,
+          0.018694639205932617,
+          -0.001501122722402215,
+          -0.0029628719203174114,
+          0.008496847003698349,
+          -0.006343104876577854,
+          -0.021813036873936653,
+          -0.0039688702672719955,
+          0.009512688964605331,
+          0.009748931042850018,
+          0.012993324548006058,
+          -0.022222522646188736,
+          0.004724845290184021,
+          0.0015719954390078783,
+          -0.005350887309759855,
+          -0.007095142733305693,
+          0.018174905329942703,
+          0.008047986775636673,
+          0.004531914368271828,
+          0.007768433541059494,
+          0.010008797980844975,
+          -0.0009479221189394593,
+          0.008370851166546345,
+          0.02146654762327671,
+          0.00108376145362854,
+          -0.002827032469213009,
+          -0.016347965225577354,
+          0.00612261239439249,
+          -0.012292472645640373,
+          0.015473868697881699,
+          0.02589215338230133,
+          0.008355101570487022,
+          -0.016332216560840607,
+          0.020883817225694656,
+          -0.014481651596724987,
+          0.015151004306972027,
+          -0.0282073263078928,
+          -0.00988280214369297,
+          0.018159156665205956,
+          -0.0011546340538188815,
+          0.01842689700424671,
+          -0.003108554519712925,
+          0.024065213277935982,
+          0.012481466867029667,
+          0.006417915225028992,
+          1.7518224922241643e-05,
+          -0.0007170937024056911,
+          0.010063921101391315,
+          -0.005189455114305019,
+          0.00014851271407678723,
+          -0.005740687251091003,
+          0.009134701453149319,
+          -0.006551785860210657,
+          0.00027832292835228145,
+          0.010725399479269981,
+          -0.028065582737326622,
+          0.014332030899822712,
+          -0.02274225652217865,
+          -0.0036735672038048506,
+          -0.0024431387428194284,
+          -0.007146328687667847,
+          -0.02031683549284935,
+          -0.003079024376347661,
+          0.019088376313447952,
+          -0.018489895388484,
+          0.04819342494010925,
+          0.0012028669007122517,
+          0.018300902098417282,
+          -0.007453443482518196,
+          0.002006090711802244,
+          -0.015143129974603653,
+          -0.00035928512807004154,
+          -0.02674262598156929,
+          0.00807161070406437,
+          0.003821218851953745,
+          0.0024254205636680126,
+          0.0025238548405468464,
+          0.006902211811393499,
+          -0.002173428889364004,
+          0.026648128405213356,
+          0.0002606047492008656,
+          0.01127663068473339,
+          -0.004295671824365854,
+          0.010717524215579033,
+          -0.019261619076132774,
+          0.004327171016484499,
+          0.0034491370897740126,
+          -0.00797711405903101,
+          -0.003933433908969164,
+          -0.007091205567121506,
+          0.0221437755972147,
+          0.016631456092000008,
+          0.009662308730185032,
+          -0.008020425215363503,
+          -0.0013042541686445475,
+          0.006803777534514666,
+          -0.0030396506190299988,
+          -0.011756990104913712,
+          -0.0038724045734852552,
+          0.005969054531306028,
+          0.012512966059148312,
+          -0.0009011658257804811,
+          0.011961733922362328,
+          0.005362699739634991,
+          -0.008008613251149654,
+          0.01297757588326931,
+          -0.020096343010663986,
+          -0.006390353664755821,
+          -0.009685933589935303,
+          -0.033987388014793396,
+          0.009292195551097393,
+          -0.0017304746434092522,
+          0.003090836340561509,
+          -0.007229013368487358,
+          0.010260789655148983,
+          -0.009520563296973705,
+          -0.01322169229388237,
+          -0.006512411870062351,
+          0.002208865247666836,
+          0.023639976978302002,
+          -0.006193485110998154,
+          0.00240376521833241,
+          -0.013355563394725323,
+          -0.013804423622786999,
+          0.016899196431040764,
+          -0.01170186698436737,
+          -0.007343197241425514,
+          0.009827678091824055
+        ]
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "396d9412-17a9-4b7e-831f-84f4e5006484",
+      "properties": {
+        "page_content": "9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\n",
+        "keyphrases": [
+          "DeepSeekMoE architecture",
+          "expert specialization",
+          "MoE language models",
+          "supervised fine-tuning",
+          "large-scale language models"
+        ],
+        "embedding": [
+          -0.003971853759139776,
+          -0.012639197520911694,
+          -0.032538022845983505,
+          0.01764647848904133,
+          0.028833750635385513,
+          0.017460335046052933,
+          0.0033017343375831842,
+          0.0448235459625721,
+          -0.002440816955640912,
+          0.05025895684957504,
+          -0.011708475649356842,
+          -0.010033177211880684,
+          0.026748934760689735,
+          -0.023044662550091743,
+          -0.0071293264627456665,
+          0.011140735819935799,
+          0.007008332759141922,
+          0.022765446454286575,
+          0.005472642369568348,
+          -0.03823403641581535,
+          0.014482025988399982,
+          -0.014649556018412113,
+          -0.05766750127077103,
+          -0.008972154930233955,
+          0.008334610611200333,
+          0.017088046297430992,
+          0.018679579719901085,
+          0.0209412332624197,
+          -0.06805435568094254,
+          0.020010510459542274,
+          0.045344747602939606,
+          0.012024921365082264,
+          -0.0010371727403253317,
+          -0.006017114035785198,
+          -0.006594161503016949,
+          -0.03143977001309395,
+          0.04270150139927864,
+          0.015459283255040646,
+          -0.024254601448774338,
+          0.02741905488073826,
+          0.02278406172990799,
+          -0.006980410777032375,
+          -0.02874067798256874,
+          -0.009809804148972034,
+          0.00014906085561960936,
+          0.007208437658846378,
+          -0.006026421207934618,
+          0.005379570182412863,
+          -0.0071293264627456665,
+          -0.005030549596995115,
+          -0.012173837050795555,
+          -0.014119044877588749,
+          0.02192779816687107,
+          -0.01348615437746048,
+          -0.025278395041823387,
+          0.010154170915484428,
+          0.014649556018412113,
+          0.0407283715903759,
+          0.04463740065693855,
+          0.026693090796470642,
+          0.021778881549835205,
+          -0.030192604288458824,
+          0.003183067310601473,
+          -0.000935375050175935,
+          0.02533423714339733,
+          0.013030100613832474,
+          0.004416273441165686,
+          0.03518127277493477,
+          0.023658938705921173,
+          -0.02598574385046959,
+          -0.049328237771987915,
+          0.02230008691549301,
+          -0.007487654220312834,
+          -0.012797419913113117,
+          0.004241763148456812,
+          -0.00017014751210808754,
+          0.0030667271930724382,
+          0.0407283715903759,
+          -0.023510023951530457,
+          0.003876454895362258,
+          0.010749832727015018,
+          0.013662991113960743,
+          0.010498537681996822,
+          -0.0014414548641070724,
+          0.04325993359088898,
+          -0.014845007099211216,
+          -0.010051791556179523,
+          -0.09188082069158554,
+          0.0304159764200449,
+          -0.010126249864697456,
+          -0.020475871860980988,
+          0.006919913925230503,
+          0.013616454787552357,
+          -0.011280343867838383,
+          -0.009781882166862488,
+          0.027605198323726654,
+          0.01356991846114397,
+          0.006398709956556559,
+          -0.0263208020478487,
+          0.01287187822163105,
+          -0.013030100613832474,
+          0.0022663066629320383,
+          0.012918413616716862,
+          0.003997448831796646,
+          -0.00012244803656358272,
+          -0.008799971081316471,
+          -0.029466642066836357,
+          0.030267061665654182,
+          0.014156273566186428,
+          0.10290056467056274,
+          0.00873482134193182,
+          -0.0014821739168837667,
+          -0.035088200122117996,
+          0.015133530832827091,
+          -0.028722064569592476,
+          -0.01276949793100357,
+          -0.0439300537109375,
+          -0.03203543275594711,
+          0.005868198815733194,
+          0.0019475346198305488,
+          0.03531157225370407,
+          0.01991743966937065,
+          0.02561345510184765,
+          -0.0014717032900080085,
+          -0.06310291588306427,
+          0.046982817351818085,
+          -0.015608198940753937,
+          -0.02019665576517582,
+          0.010358929634094238,
+          -0.004600090906023979,
+          -0.016101481392979622,
+          -0.018028074875473976,
+          -0.033115070313215256,
+          -0.04102620109915733,
+          0.0028037982992827892,
+          -0.010991820134222507,
+          0.014584405347704887,
+          0.06392195075750351,
+          -0.040951743721961975,
+          -0.01026585791260004,
+          0.015989795327186584,
+          0.010935977101325989,
+          0.009302561171352863,
+          -0.020866775885224342,
+          -0.029299112036824226,
+          -0.0041649783961474895,
+          0.0076691447757184505,
+          0.08391384780406952,
+          -0.007845981977880001,
+          0.02118322066962719,
+          -0.024831648916006088,
+          0.011205886490643024,
+          0.011969078332185745,
+          0.004658260848373175,
+          0.0013623435515910387,
+          -0.013523383066058159,
+          -0.030490435659885406,
+          0.005826316308230162,
+          -0.021034304052591324,
+          0.02980170026421547,
+          0.0025920593179762363,
+          -0.025632068514823914,
+          -0.0009185057133436203,
+          0.01166194025427103,
+          -0.0058030481450259686,
+          0.017702322453260422,
+          0.021127376705408096,
+          0.0077249882742762566,
+          -0.060087379068136215,
+          0.053758472204208374,
+          0.01866096444427967,
+          0.0218347255140543,
+          -0.024329058825969696,
+          0.004721084609627724,
+          0.00043831163202412426,
+          0.017972230911254883,
+          0.0070595224387943745,
+          0.011410645209252834,
+          -0.005281844176352024,
+          0.033561814576387405,
+          -0.015245217829942703,
+          0.023863697424530983,
+          0.021648580208420753,
+          -0.0014717032900080085,
+          0.030453206971287727,
+          0.013495461083948612,
+          0.018474821001291275,
+          0.02894543670117855,
+          -0.0181862972676754,
+          0.005430759862065315,
+          0.014044586569070816,
+          0.00012295703345444053,
+          0.029448026791214943,
+          -0.006738423369824886,
+          -0.007389928214251995,
+          -0.037340544164180756,
+          -0.018484128639101982,
+          0.02455243282020092,
+          -0.0004868836549576372,
+          0.009372364729642868,
+          -0.00881858542561531,
+          -0.03901584446430206,
+          -0.004251070320606232,
+          -0.012946335598826408,
+          0.028870979323983192,
+          0.005100353620946407,
+          -0.011150043457746506,
+          0.005807701963931322,
+          0.022058097645640373,
+          -0.013672297820448875,
+          -0.012434438802301884,
+          -0.007520229555666447,
+          0.002373339608311653,
+          -0.02220701426267624,
+          -0.0041021546348929405,
+          0.0037182322703301907,
+          -0.01921009086072445,
+          0.0027642427012324333,
+          0.008585905656218529,
+          -0.00825549941509962,
+          -0.01627831906080246,
+          0.015040459111332893,
+          0.03441808000206947,
+          -0.016203859820961952,
+          -0.03585138916969299,
+          -0.024310443550348282,
+          -0.022560687735676765,
+          0.018996024504303932,
+          0.028498690575361252,
+          -0.009511973708868027,
+          0.03091856651008129,
+          0.008367185480892658,
+          0.014295881614089012,
+          0.004711777437478304,
+          0.021127376705408096,
+          -0.008074008859694004,
+          -0.011354802176356316,
+          -0.018791265785694122,
+          0.014212116599082947,
+          0.026488332077860832,
+          0.00021682900842279196,
+          0.018260754644870758,
+          -0.04556812345981598,
+          -0.011540945619344711,
+          0.0020394434686750174,
+          -0.0006038055289536715,
+          -0.012406516820192337,
+          -0.016864672303199768,
+          0.01268573384732008,
+          0.029652785509824753,
+          0.008381146937608719,
+          -0.016557535156607628,
+          -0.002878256142139435,
+          0.00434181559830904,
+          -0.04084005579352379,
+          0.052455462515354156,
+          0.03990933671593666,
+          -0.012648504227399826,
+          -0.024366287514567375,
+          -0.033282600343227386,
+          0.05710906907916069,
+          -0.02380785532295704,
+          -0.025222551077604294,
+          0.06961796432733536,
+          -0.013849135488271713,
+          0.019768523052334785,
+          0.05707183852791786,
+          0.0026944386772811413,
+          0.004486077465116978,
+          -0.01794430986046791,
+          0.026227731257677078,
+          -0.021369364112615585,
+          -0.01190392766147852,
+          0.006394056603312492,
+          0.03030429035425186,
+          -0.012332059442996979,
+          -0.01941484957933426,
+          -0.002408241853117943,
+          0.0011081402190029621,
+          0.015059073455631733,
+          0.002245365409180522,
+          0.009539894759654999,
+          0.013514075428247452,
+          0.012536818161606789,
+          0.00968881044536829,
+          -0.0047303917817771435,
+          -0.01110350713133812,
+          -0.04325993359088898,
+          -0.03162591531872749,
+          -0.01740449108183384,
+          -0.035572174936532974,
+          0.0013146440032869577,
+          -0.021909182891249657,
+          0.011205886490643024,
+          0.03786174952983856,
+          -0.009027997963130474,
+          0.003348270431160927,
+          0.018884338438510895,
+          -0.0023140062112361193,
+          -0.0005747204995714128,
+          -0.031718987971544266,
+          -0.018577199429273605,
+          -0.00893958006054163,
+          0.00128555903211236,
+          -0.05126413702964783,
+          -0.01171778328716755,
+          0.008418375626206398,
+          0.03103025257587433,
+          -0.02250484563410282,
+          -0.023621710017323494,
+          0.015878107398748398,
+          -0.019805751740932465,
+          -0.028647607192397118,
+          -0.02931772544980049,
+          -0.019191475585103035,
+          0.05465196445584297,
+          0.016706449910998344,
+          0.025967128574848175,
+          -0.03147700056433678,
+          -0.03423193469643593,
+          0.05025895684957504,
+          0.008474218659102917,
+          -0.07550012320280075,
+          0.036595966666936874,
+          0.0014507620362564921,
+          0.0010063425870612264,
+          -0.005495910067111254,
+          0.05837484821677208,
+          -0.0032575251534581184,
+          -0.0010057608596980572,
+          0.012536818161606789,
+          -0.05748135596513748,
+          -0.00575651228427887,
+          -0.006412670947611332,
+          0.013253473676741123,
+          -0.04642438516020775,
+          -0.024943334981799126,
+          0.004027697257697582,
+          -0.012974257580935955,
+          -0.005700668785721064,
+          -0.042180296033620834,
+          -0.03145838528871536,
+          -0.0028526613023132086,
+          -0.0022756140679121017,
+          -0.007943707518279552,
+          -0.03078826516866684,
+          -0.025929899886250496,
+          0.021164605394005775,
+          0.045344747602939606,
+          0.007641223259270191,
+          0.0017695341957733035,
+          0.023621710017323494,
+          0.0197871383279562,
+          0.005598289426416159,
+          0.020345570519566536,
+          -0.01645515486598015,
+          -0.0375266894698143,
+          -0.005495910067111254,
+          -0.00010630584438331425,
+          0.02680477872490883,
+          -0.00926067866384983,
+          -0.049700524657964706,
+          -0.033599045127630234,
+          0.0193403922021389,
+          -0.03687518462538719,
+          -0.015487205237150192,
+          -0.004313894081860781,
+          -0.033636271953582764,
+          0.013141786679625511,
+          0.00029579491820186377,
+          -0.01730211265385151,
+          0.049328237771987915,
+          -0.023454179987311363,
+          -0.0019975609611719847,
+          -0.023752011358737946,
+          0.003443669294938445,
+          0.05312558263540268,
+          -0.018502742052078247,
+          -0.018028074875473976,
+          0.022672373801469803,
+          -0.01730211265385151,
+          0.01849343627691269,
+          0.04314824566245079,
+          0.009837726131081581,
+          0.004304586909711361,
+          0.014724013395607471,
+          0.006226526573300362,
+          -0.008483526296913624,
+          -0.0386807844042778,
+          0.021871954202651978,
+          0.012220372445881367,
+          -0.045307520776987076,
+          -0.022597916424274445,
+          -0.02479442022740841,
+          0.0007329431246034801,
+          0.011801548302173615,
+          0.016557535156607628,
+          -0.00968881044536829,
+          -0.028387004509568214,
+          0.006421978119760752,
+          -0.016017716377973557,
+          0.010163478553295135,
+          0.00786459632217884,
+          0.014845007099211216,
+          -0.005519178230315447,
+          0.0067756520584225655,
+          0.004006756003946066,
+          0.012788113206624985,
+          0.0362422950565815,
+          -0.02721429616212845,
+          -0.013216244988143444,
+          -0.020271113142371178,
+          -0.05506148189306259,
+          0.0014589058700948954,
+          0.015059073455631733,
+          0.004544247407466173,
+          -0.02455243282020092,
+          0.005709975957870483,
+          0.012546124868094921,
+          -0.04180800914764404,
+          -0.003641447750851512,
+          0.012099378742277622,
+          0.034678682684898376,
+          0.028238089755177498,
+          -0.011596789583563805,
+          0.003436689032241702,
+          0.0011959770927205682,
+          -0.0032435643952339888,
+          -0.0007631916087120771,
+          -0.023100506514310837,
+          0.027493512257933617,
+          0.014649556018412113,
+          0.038941387087106705,
+          -0.023472795262932777,
+          -0.005998499691486359,
+          -0.048211369663476944,
+          -0.003466937458142638,
+          0.030248448252677917,
+          -0.020680630579590797,
+          -0.011736397631466389,
+          -0.0005956617533229291,
+          0.005044510122388601,
+          -0.014733321033418179,
+          -0.006338213104754686,
+          -0.03194236010313034,
+          -0.026674477383494377,
+          -0.04143571853637695,
+          -0.01702289469540119,
+          -0.00934909749776125,
+          -0.01416558027267456,
+          -0.05748135596513748,
+          -0.011364108882844448,
+          -0.09463575482368469,
+          0.009218796156346798,
+          0.04117511585354805,
+          -0.0257437564432621,
+          0.028238089755177498,
+          -0.028796521946787834,
+          -0.0035576827358454466,
+          -0.03501374274492264,
+          0.012648504227399826,
+          0.040616683661937714,
+          0.033115070313215256,
+          -0.015291753225028515,
+          -0.006966450251638889,
+          0.017516177147626877,
+          -0.030732423067092896,
+          -0.03802927955985069,
+          -0.019228706136345863,
+          -0.00846025813370943,
+          0.00572859076783061,
+          0.0018474820535629988,
+          0.03789897635579109,
+          0.04504691809415817,
+          0.04631270095705986,
+          0.03592584654688835,
+          0.00934909749776125,
+          -0.035367414355278015,
+          -0.010740526020526886,
+          0.005216693971306086,
+          0.03071380779147148,
+          -0.03305922448635101,
+          0.01166194025427103,
+          -0.0311233252286911,
+          0.01853066496551037,
+          0.002405914943665266,
+          -0.023454179987311363,
+          0.021853338927030563,
+          0.005947310011833906,
+          0.0013507094699889421,
+          -0.005919388495385647,
+          0.027325982227921486,
+          -0.01828867755830288,
+          0.035125426948070526,
+          -0.028442848473787308,
+          -0.024105684831738472,
+          -0.028684835880994797,
+          -0.011122121475636959,
+          0.031104711815714836,
+          0.019154246896505356,
+          -0.0005947891622781754,
+          -0.002433836692944169,
+          -0.015273138880729675,
+          0.009172259829938412,
+          0.004853712394833565,
+          -0.02032695710659027,
+          -0.016334161162376404,
+          0.026879236102104187,
+          -0.06943181902170181,
+          -0.009763267822563648,
+          0.008306688629090786,
+          0.0005555243697017431,
+          0.00929790735244751,
+          -0.0386807844042778,
+          0.010377543978393078,
+          0.004590783733874559,
+          -0.03990933671593666,
+          0.022877132520079613,
+          0.00926067866384983,
+          0.01971268095076084,
+          -0.004362756852060556,
+          -0.008837200701236725,
+          0.007022293284535408,
+          0.06451760977506638,
+          -0.010563688352704048,
+          0.0022791041992604733,
+          0.007455078884959221,
+          -0.00812519807368517,
+          0.03845741227269173,
+          -0.025222551077604294,
+          -0.028033331036567688,
+          0.02935495413839817,
+          -0.008455604314804077,
+          -0.007650530431419611,
+          -0.015310367569327354,
+          -0.006696540862321854,
+          0.007827367633581161,
+          0.02870344929397106,
+          -0.02373339794576168,
+          -0.005212040152400732,
+          -0.02676755003631115,
+          0.006524357479065657,
+          0.0022500192280858755,
+          0.002908504568040371,
+          0.0061334543861448765,
+          -0.0009958719601854682,
+          0.039685964584350586,
+          -0.0225420743227005,
+          0.03452976793050766,
+          0.02959694154560566,
+          -0.004090521018952131,
+          -0.0045582083985209465,
+          -0.0005939166294410825,
+          0.013179016299545765,
+          -0.01289049256592989,
+          -0.01808391883969307,
+          0.01736726239323616,
+          -0.018679579719901085,
+          -0.013244166038930416,
+          -0.054949793964624405,
+          -0.002410568529739976,
+          0.002610673662275076,
+          -0.010051791556179523,
+          0.006784959230571985,
+          -0.001237859483808279,
+          0.002903850981965661,
+          0.005393530707806349,
+          0.03949981927871704,
+          -0.03827126696705818,
+          0.023714782670140266,
+          0.00015429616905748844,
+          -0.0026525561697781086,
+          0.008716206066310406,
+          0.0035018394701182842,
+          0.007106058299541473,
+          0.05554545670747757,
+          0.012899799272418022,
+          -0.007994897663593292,
+          -0.005063124932348728,
+          0.005002627614885569,
+          0.016296932473778725,
+          -0.016027024015784264,
+          0.0052073863334953785,
+          0.03668903931975365,
+          0.025297008454799652,
+          0.000632890616543591,
+          -0.0017951290355995297,
+          -0.010182092897593975,
+          0.02751212567090988,
+          -0.021034304052591324,
+          0.029820315539836884,
+          0.04307378828525543,
+          0.009470091201364994,
+          -0.0439300537109375,
+          0.0069571430794894695,
+          -0.015431362204253674,
+          -0.009251371026039124,
+          0.0031528188847005367,
+          0.019600993022322655,
+          -0.04381836578249931,
+          0.003245891071856022,
+          0.01145718153566122,
+          0.008320650085806847,
+          -0.021052919328212738,
+          -0.04675944522023201,
+          -0.008674323558807373,
+          -0.0035274343099445105,
+          0.012974257580935955,
+          -0.016846058890223503,
+          -0.006510396488010883,
+          0.005742551293224096,
+          0.016259703785181046,
+          -0.004988667089492083,
+          0.012034228071570396,
+          0.033394284546375275,
+          -0.016408618539571762,
+          0.013783984817564487,
+          -0.007054868619889021,
+          -0.02189056947827339,
+          0.0110476640984416,
+          0.03095579519867897,
+          -0.01307663694024086,
+          0.03546048700809479,
+          -0.0015345270512625575,
+          0.012415824458003044,
+          0.012964949943125248,
+          0.003639120841398835,
+          -0.014817086048424244,
+          0.002092959824949503,
+          0.018335212022066116,
+          -0.021443821489810944,
+          0.013020792976021767,
+          -0.013541997410356998,
+          -0.037992049008607864,
+          0.010051791556179523,
+          -0.0038950692396610975,
+          0.0016752986703068018,
+          0.009041959419846535,
+          0.0062591019086539745,
+          -0.027195680886507034,
+          -0.01611078903079033,
+          0.023342493921518326,
+          0.024105684831738472,
+          0.06098087131977081,
+          -0.014081815257668495,
+          -0.000535164843313396,
+          -0.007617955096065998,
+          0.009493358433246613,
+          0.04460017383098602,
+          0.024459360167384148,
+          0.0038694743998348713,
+          0.004290625918656588,
+          0.027102608233690262,
+          0.046945590525865555,
+          -0.005230654496699572,
+          -0.010377543978393078,
+          0.006394056603312492,
+          0.003197028301656246,
+          0.023789240047335625,
+          0.021574122831225395,
+          -0.0005409818259067833,
+          0.014072508551180363,
+          0.014444797299802303,
+          -0.016538919880986214,
+          0.012937027961015701,
+          -0.04556812345981598,
+          -0.040169939398765564,
+          0.007115365471690893,
+          0.00435112277045846,
+          0.019359005615115166,
+          0.002450124127790332,
+          -0.005039856769144535,
+          -0.0027851839549839497,
+          0.01750687137246132,
+          -0.010535767301917076,
+          -0.0122948307543993,
+          0.02475719153881073,
+          -0.00873482134193182,
+          0.006747730541974306,
+          0.00030801063985563815,
+          -0.009716732427477837,
+          0.0035181271377950907,
+          -0.0014972981298342347,
+          -0.005077085457742214,
+          0.0023989344481378794,
+          -0.008381146937608719,
+          -0.017516177147626877,
+          0.012229680083692074,
+          -0.04396728053689003,
+          0.013886364176869392,
+          0.001635742955841124,
+          -0.008911658078432083,
+          0.002878256142139435,
+          0.0013146440032869577,
+          0.0114944102242589,
+          -0.018381748348474503,
+          -0.005309765692800283,
+          0.012453053146600723,
+          0.026413874700665474,
+          -0.007403889205306768,
+          0.0038904156535863876,
+          0.015142838470637798,
+          -0.007534190081059933,
+          -0.00987495481967926,
+          0.028070559725165367,
+          -0.007822713814675808,
+          0.006859417073428631,
+          -0.008181042037904263,
+          -0.02540869638323784,
+          0.0053888773545622826,
+          0.03793620690703392,
+          0.04865811765193939,
+          0.007538843899965286,
+          0.01648307777941227,
+          0.03123501129448414,
+          0.014835700392723083,
+          0.00822292361408472,
+          0.015422054566442966,
+          -0.04437679797410965,
+          0.01541274692863226,
+          -0.018037382513284683,
+          0.040542226284742355,
+          0.030844109132885933,
+          -0.05044510215520859,
+          -0.02812640182673931,
+          0.019284548237919807,
+          0.023100506514310837,
+          0.017562713474035263,
+          0.013681605458259583,
+          -0.005314419511705637,
+          0.02053171582520008,
+          -0.023249421268701553,
+          -0.01235998049378395,
+          0.01316040102392435,
+          -0.0032435643952339888,
+          -0.004816483706235886,
+          0.005067778285592794,
+          0.03912752866744995,
+          -0.00436741067096591,
+          -0.016911208629608154,
+          0.02131352201104164,
+          -0.005984539166092873,
+          0.005928695667535067,
+          -0.013123172335326672,
+          0.0009958719601854682,
+          0.03916475921869278,
+          0.005393530707806349,
+          0.03761976212263107,
+          -0.021499665454030037,
+          0.012145915068686008,
+          -0.028182245790958405,
+          -0.014733321033418179,
+          0.029243268072605133,
+          -0.013216244988143444,
+          -0.021201834082603455,
+          0.016976360231637955,
+          0.020513100549578667,
+          0.0032156426459550858,
+          0.011168657802045345,
+          -0.013681605458259583,
+          -0.010600917041301727,
+          0.038159579038619995,
+          -0.0029527137521654367,
+          -0.004144037142395973,
+          0.01712527498602867,
+          -0.01812114752829075,
+          -0.011922542005777359,
+          0.016883287578821182,
+          -0.0019580053631216288,
+          -0.032742779701948166,
+          0.02241177298128605,
+          -0.0004022461944259703,
+          0.03620506450533867,
+          -0.010237935930490494,
+          -0.0056634400971233845,
+          0.010535767301917076,
+          -0.006654658354818821,
+          -0.029987845569849014,
+          -0.034176092594861984,
+          0.007455078884959221,
+          -0.027437668293714523,
+          0.010963899083435535,
+          -0.006650005001574755,
+          0.03255663812160492,
+          -0.030248448252677917,
+          0.008669670671224594,
+          0.03030429035425186,
+          -0.00825549941509962,
+          0.022690989077091217,
+          0.032947540283203125,
+          -0.019079789519309998,
+          -0.03525573015213013,
+          -0.003245891071856022,
+          -0.0003592003195080906,
+          0.029578328132629395,
+          0.002880582818761468,
+          0.0026246344204992056,
+          0.004960745573043823,
+          -0.016222475096583366,
+          -0.03510681167244911,
+          0.022430386394262314,
+          -0.04474908858537674,
+          0.007068829610943794,
+          0.026618633419275284,
+          -0.0430365614593029,
+          -0.036800727248191833,
+          -0.018446899950504303,
+          -0.0263208020478487,
+          -0.02159273810684681,
+          0.001237859483808279,
+          0.02639526128768921,
+          -0.017581328749656677,
+          -0.030602121725678444,
+          0.028089173138141632,
+          0.0034948589745908976,
+          -0.012695040553808212,
+          -0.011056970804929733,
+          -0.010126249864697456,
+          -0.027083994820713997,
+          -0.022262856364250183,
+          -0.014035279862582684,
+          0.0009807477472350001,
+          0.004881633911281824,
+          -0.010368237271904945,
+          -0.0225420743227005,
+          -0.013616454787552357,
+          -0.009032651782035828,
+          -0.004825790878385305,
+          0.050147272646427155,
+          0.021090148016810417,
+          0.023863697424530983,
+          0.0009185057133436203,
+          -0.015719885006546974,
+          0.04601486772298813,
+          -0.0007253810181282461,
+          0.02591128461062908,
+          0.005612250417470932,
+          -0.022058097645640373,
+          -0.005844930652529001,
+          0.03715439885854721,
+          -0.018279369920492172,
+          -0.004395332187414169,
+          -0.014724013395607471,
+          -0.00374150020070374,
+          -0.008609173819422722,
+          0.026693090796470642,
+          0.012024921365082264,
+          -0.004769947379827499,
+          0.013458232395350933,
+          -0.01270434819161892,
+          0.02200225554406643,
+          0.005821662489324808,
+          -0.003676349762827158,
+          -0.0010331007651984692,
+          -0.028684835880994797,
+          -0.02114599198102951,
+          0.001068002893589437,
+          0.022765446454286575,
+          -0.016334161162376404,
+          -0.013197630643844604,
+          -0.015291753225028515,
+          -0.021574122831225395,
+          -0.005198079161345959,
+          0.0004831026017200202,
+          0.005407491698861122,
+          0.015850186347961426,
+          -0.003311041509732604,
+          -0.0430365614593029,
+          -0.005272537004202604,
+          -0.03763837739825249,
+          0.00048455686192028224,
+          0.01961960829794407,
+          -0.011243115179240704,
+          0.01128965150564909,
+          0.001805599662475288,
+          -0.0038648208137601614,
+          -0.01832590624690056,
+          0.007636569440364838,
+          0.01982436701655388,
+          -0.006789613049477339,
+          -0.0022197705693542957,
+          0.018986716866493225,
+          0.0021569470409303904,
+          -0.007645876612514257,
+          0.004849058575928211,
+          -0.001855625887401402,
+          -0.006845456548035145,
+          -0.023379722610116005,
+          -0.029280496761202812,
+          -0.013607148081064224,
+          0.0018090897938236594,
+          -0.001733468729071319,
+          -0.024906106293201447,
+          0.018875030800700188,
+          -0.006082264706492424,
+          0.02336110919713974,
+          -0.011326880194246769,
+          0.01316040102392435,
+          -0.0022697970271110535,
+          -0.0090931486338377,
+          0.018214218318462372,
+          -0.010554381646215916,
+          -0.03264970704913139,
+          -0.01316040102392435,
+          0.028517305850982666,
+          0.00034872969263233244,
+          -0.008097277022898197,
+          0.010042484849691391,
+          0.00922810286283493,
+          -0.027437668293714523,
+          -0.0077017201110720634,
+          0.010619531385600567,
+          0.006626736838370562,
+          -0.032333262264728546,
+          0.026506947353482246,
+          0.03266832232475281,
+          0.01235998049378395,
+          -0.03222157806158066,
+          0.048620887100696564,
+          0.008651056326925755,
+          -0.02475719153881073,
+          0.005677401088178158,
+          0.020624786615371704,
+          0.0363725945353508,
+          -0.0299692302942276,
+          -0.007413196377456188,
+          -0.007422503549605608,
+          0.009009383618831635,
+          -0.03473452478647232,
+          0.015235910192131996,
+          -0.02537146769464016,
+          0.01961960829794407,
+          0.032333262264728546,
+          0.0032831199932843447,
+          0.040579456835985184,
+          -0.009511973708868027,
+          0.017693014815449715,
+          0.0060310750268399715,
+          -0.0076691447757184505,
+          0.006459206808358431,
+          -0.02162996679544449,
+          0.01624108850955963,
+          0.019768523052334785,
+          0.02162996679544449,
+          0.009902875870466232,
+          -0.024682732298970222,
+          0.01941484957933426,
+          0.015133530832827091,
+          0.023826468735933304,
+          0.027605198323726654,
+          -0.014658862724900246,
+          0.026842007413506508,
+          0.005812355317175388,
+          -0.013635069131851196,
+          0.004430233966559172,
+          0.009386326186358929,
+          -0.040616683661937714,
+          -0.0012948662042617798,
+          -0.022523459047079086,
+          0.01459371205419302,
+          -0.0005066614830866456,
+          -0.0028224128764122725,
+          -0.02377062663435936,
+          -0.013653683476150036,
+          -7.831148832337931e-05,
+          0.005146889481693506,
+          -0.004358103033155203,
+          -0.04087728634476662,
+          -0.045754265040159225,
+          0.0234355665743351,
+          -0.009800496511161327,
+          -0.040542226284742355,
+          -0.012648504227399826,
+          -0.009311867877840996,
+          -0.0015752461040392518,
+          -0.014109737239778042,
+          -0.01289049256592989,
+          -0.04385559633374214,
+          -0.014435489661991596,
+          0.01958237960934639,
+          0.030043689534068108,
+          -0.02203948423266411,
+          0.030248448252677917,
+          0.006873378064483404,
+          0.015198681503534317,
+          -0.014630941674113274,
+          0.013560611754655838,
+          0.00614741537719965,
+          -0.04638715833425522,
+          0.02894543670117855,
+          -0.0028898902237415314,
+          0.021276293322443962,
+          0.0384201817214489,
+          -0.005272537004202604,
+          -0.010023870505392551,
+          0.017246268689632416,
+          -0.02213255688548088,
+          -0.013551304116845131,
+          -0.008092623203992844,
+          -0.01586880162358284,
+          -0.037005484104156494,
+          -0.04884426295757294,
+          0.016129402443766594,
+          0.04080282896757126,
+          0.00911176297813654,
+          0.014426182955503464,
+          0.018679579719901085,
+          0.004311567172408104,
+          -0.033282600343227386,
+          0.0225420743227005,
+          0.0047932155430316925,
+          -0.012387902475893497,
+          -0.023826468735933304,
+          0.013653683476150036,
+          0.012778805568814278,
+          -0.006198605056852102,
+          -0.006384748965501785,
+          -0.026655862107872963,
+          -0.024477973580360413,
+          0.0021476398687809706,
+          -0.013886364176869392,
+          0.0033250025007873774,
+          -0.022765446454286575,
+          -0.02857314795255661,
+          0.032128505408763885,
+          0.03546048700809479,
+          -0.01235998049378395,
+          -0.00393229816108942,
+          -0.013579226098954678,
+          -0.008404415100812912,
+          0.008920964784920216,
+          0.005230654496699572,
+          0.017953617498278618,
+          -0.019526535645127296,
+          0.01128965150564909,
+          -0.010442694649100304,
+          0.012099378742277622,
+          -0.0762074738740921,
+          -0.01679021492600441,
+          0.025874055922031403,
+          0.022672373801469803,
+          0.0009662052034400403,
+          0.011438566260039806,
+          0.008893043734133244,
+          0.0006619756459258497,
+          -0.035088200122117996,
+          0.01921009086072445,
+          0.013793292455375195,
+          0.016222475096583366,
+          -0.01151302456855774,
+          0.005235308315604925,
+          0.023845084011554718,
+          -0.0006683743558824062,
+          -0.006319598760455847,
+          -0.01760924980044365,
+          0.011540945619344711,
+          0.020103583112359047,
+          0.0026851315051317215,
+          0.013877056539058685,
+          0.010572995990514755,
+          -0.01961960829794407,
+          -0.013011486269533634,
+          -0.0005930440966039896,
+          -0.005668093450367451,
+          -0.010963899083435535,
+          0.008706899359822273,
+          0.0007399235619232059,
+          0.03763837739825249,
+          0.024887491017580032,
+          0.02172303944826126,
+          0.015980487689375877,
+          0.0069571430794894695,
+          0.007469039876013994,
+          -0.0020917963702231646,
+          0.0018323578406125307,
+          0.01746964268386364,
+          0.03272416815161705,
+          -0.022932976484298706,
+          -0.005156196653842926,
+          0.0042533972300589085,
+          0.024831648916006088,
+          -0.008827893063426018,
+          0.025892671197652817,
+          0.030062302947044373,
+          -0.007650530431419611,
+          -0.029503870755434036,
+          0.0032924271654337645,
+          0.018502742052078247,
+          0.009302561171352863,
+          -0.01634346880018711,
+          -0.0027921644505113363,
+          0.013728141784667969,
+          0.0075434972532093525,
+          0.020922617986798286,
+          -0.004241763148456812,
+          -0.014053894206881523,
+          0.01880057342350483,
+          0.09076395630836487,
+          -0.005691361613571644,
+          -0.0012017941335216165,
+          -0.017106659710407257,
+          -0.01921009086072445,
+          0.0014670497039332986,
+          0.0098935691639781,
+          0.016613377258181572,
+          0.017209040001034737,
+          0.00374615378677845,
+          0.004586129914969206,
+          0.01455648336559534,
+          -0.011531638912856579,
+          0.009479397907853127,
+          -0.02421737276017666,
+          0.01804668828845024,
+          -0.014249345287680626,
+          -0.0067756520584225655,
+          -0.011233807541429996,
+          0.023621710017323494,
+          -0.0019568419083952904,
+          0.007720334455370903,
+          0.019154246896505356,
+          0.003871801309287548,
+          -0.0247199609875679,
+          0.006412670947611332,
+          -0.006454553455114365,
+          0.0007160738459788263,
+          -0.038866929709911346,
+          -0.018363134935498238,
+          0.01127103716135025,
+          -0.0014123697765171528,
+          -0.0011232644319534302,
+          -0.004672221839427948,
+          -0.007804099470376968,
+          -0.00025580296642147005,
+          -0.02455243282020092,
+          0.016836751252412796,
+          0.008855815045535564,
+          0.015403440222144127,
+          0.022393157705664635,
+          -0.0010522969532757998,
+          -0.0009138521272689104,
+          0.025315623730421066,
+          -0.0071293264627456665,
+          -0.004043984692543745,
+          0.027232909575104713,
+          -0.023621710017323494,
+          -0.005188771989196539,
+          0.005300458520650864,
+          -0.01692982390522957,
+          -0.002913158154115081,
+          -0.00414636405184865,
+          -0.021034304052591324,
+          -0.013206937350332737,
+          0.0014914810890331864,
+          0.020717859268188477,
+          -0.015468590892851353,
+          -0.0029178117401897907,
+          0.013048714958131313,
+          -0.005286497995257378,
+          0.007585379760712385,
+          0.0032901002559810877,
+          -0.005067778285592794,
+          -0.02114599198102951,
+          -0.026860620826482773,
+          -0.007999550551176071,
+          0.0011936502996832132,
+          2.35407078434946e-05,
+          -0.0037321930285543203,
+          -0.009372364729642868,
+          -0.004830444231629372,
+          -0.01089874841272831,
+          0.007855288684368134,
+          -0.007413196377456188,
+          -0.014928772114217281,
+          0.013988743536174297,
+          0.005226001143455505,
+          0.008436989970505238,
+          -0.031086096540093422,
+          0.013830521143972874,
+          0.020587557926774025,
+          -0.006961796432733536,
+          0.009237410500645638,
+          -0.015198681503534317,
+          -0.00025216734502464533,
+          -0.01685536466538906,
+          0.004043984692543745,
+          0.004632666241377592,
+          -0.01169916894286871,
+          0.005379570182412863,
+          0.007403889205306768,
+          -0.01913563348352909,
+          -0.006929221097379923,
+          -0.02060617320239544,
+          0.00492817023769021,
+          0.011578175239264965,
+          0.009000076912343502,
+          -0.01559889130294323,
+          -0.004886287730187178,
+          -0.03449253737926483,
+          0.016101481392979622,
+          0.017078738659620285,
+          -0.029001280665397644,
+          -0.014584405347704887,
+          0.009046612307429314,
+          0.013588533736765385,
+          -0.01771163009107113,
+          -0.012834648601710796,
+          -0.0009981987532228231,
+          0.005938002839684486,
+          -0.018744729459285736,
+          0.00533768767490983,
+          -0.0037647683639079332,
+          0.01832590624690056,
+          0.0014740300830453634,
+          0.024850262328982353,
+          -0.026581404730677605,
+          0.010107634589076042,
+          0.017004281282424927,
+          -0.027474896982312202,
+          0.0005968251498416066,
+          0.014658862724900246,
+          0.00042202402255497873,
+          0.00020286819199100137,
+          0.0004857202584389597,
+          -0.007594686932861805,
+          -0.008483526296913624,
+          0.004513998981565237,
+          -0.0014658862492069602,
+          -0.026432489976286888,
+          0.01950792223215103,
+          0.01410043053328991,
+          0.013104557991027832,
+          0.008799971081316471,
+          -0.011652632616460323,
+          0.0006020604632794857,
+          0.01573850028216839,
+          0.032742779701948166,
+          -0.0014891542959958315,
+          -0.015654735267162323,
+          0.010014562867581844,
+          0.026637248694896698,
+          0.0003402950242161751,
+          0.0008190348744392395,
+          -0.01603632979094982,
+          -0.026469718664884567,
+          0.038978613913059235,
+          0.018270062282681465,
+          0.01958237960934639,
+          -0.0018253774615004659,
+          -0.01781400851905346,
+          0.01455648336559534,
+          -0.004437214694917202,
+          0.0027595891151577234,
+          -0.015301060862839222,
+          -0.009418901056051254,
+          0.0026572097558528185,
+          0.026655862107872963,
+          -0.005374916363507509,
+          0.03382241725921631,
+          0.012052842415869236,
+          0.009423554874956608,
+          -0.035609401762485504,
+          0.0031528188847005367,
+          -0.02462689019739628,
+          0.011559560894966125,
+          -0.0094142472371459,
+          -0.0026944386772811413,
+          0.008590559475123882,
+          -0.02935495413839817,
+          -0.009032651782035828,
+          0.01011694222688675,
+          0.023063277825713158,
+          0.007366660516709089,
+          -0.023007433861494064,
+          0.020178040489554405,
+          0.0013134806649759412,
+          0.02151828072965145,
+          -0.012052842415869236,
+          0.012266908772289753,
+          -0.008744128048419952,
+          -0.017804700881242752,
+          -0.005170157644897699,
+          0.03374795988202095,
+          -0.008530061691999435,
+          -0.0036577354185283184,
+          0.006254448089748621,
+          -0.0279216431081295,
+          0.032612480223178864,
+          -0.0037182322703301907,
+          -0.0029154850635677576,
+          0.0041975537315011024,
+          -0.012006307020783424,
+          0.00991218350827694,
+          0.0094142472371459,
+          0.008115891367197037,
+          -0.018707500770688057,
+          -0.023714782670140266,
+          -0.04184523597359657,
+          -0.024533817544579506,
+          0.014733321033418179,
+          0.011503716930747032,
+          -0.015114916488528252,
+          0.03387825936079025,
+          -0.02192779816687107,
+          -0.016306240111589432,
+          -0.011447873897850513,
+          0.028684835880994797,
+          -0.03722885996103287,
+          0.00875343568623066,
+          -0.01350476872175932,
+          -0.020587557926774025,
+          -0.00148566416464746,
+          0.004900248255580664,
+          -0.002478045877069235,
+          -0.009176913648843765,
+          -0.02939218282699585,
+          0.0060636503621935844,
+          0.04999835789203644,
+          -0.0055936360731720924,
+          -0.0016415599966421723,
+          0.00418126629665494,
+          0.003774075536057353,
+          -0.014342417940497398,
+          -0.016501691192388535,
+          0.03486482426524162,
+          -0.03806650638580322,
+          0.0017520831897854805,
+          0.01845620572566986,
+          -0.003583277575671673,
+          0.00554244639351964,
+          0.004523306153714657,
+          0.003578623989596963,
+          0.012629889883100986,
+          -0.00727824168279767,
+          0.013579226098954678,
+          0.011056970804929733,
+          0.011624710634350777,
+          0.005291151348501444,
+          0.014910157769918442,
+          0.025185322389006615,
+          -0.01706012338399887,
+          0.024868877604603767,
+          0.025017792358994484,
+          0.00014222587924450636,
+          0.03581416234374046,
+          -0.019433464854955673,
+          -0.026004357263445854,
+          0.005626211408525705,
+          -0.011745704337954521,
+          -0.004711777437478304,
+          0.001653193961828947,
+          -0.0220953281968832,
+          -0.0031993549782782793,
+          -0.011401337571442127,
+          0.0011895783245563507,
+          0.014053894206881523,
+          -0.014724013395607471,
+          -0.0204944871366024,
+          0.015422054566442966,
+          0.013430310413241386,
+          0.02861037850379944,
+          0.01518937386572361,
+          -0.0034971858840435743,
+          0.02881513722240925,
+          -0.011996999382972717,
+          0.014789164066314697,
+          0.008190348744392395,
+          0.011913234367966652,
+          -0.0069571430794894695,
+          0.024329058825969696,
+          0.0019963975064456463,
+          0.009456129744648933,
+          0.001500788377597928,
+          -0.01692051626741886,
+          -0.0034297085367143154,
+          -0.034213319420814514,
+          -0.018344519659876823,
+          0.003041132353246212,
+          -0.00020417702035047114,
+          0.08748781681060791,
+          0.006980410777032375,
+          -0.02090400457382202,
+          0.003069054102525115,
+          0.0011866698041558266,
+          0.005212040152400732,
+          -0.018279369920492172,
+          0.015180067159235477,
+          -0.027456283569335938,
+          -0.020140811800956726,
+          -0.017832623794674873,
+          -0.024980563670396805,
+          0.0030969756189733744,
+          0.004339488688856363,
+          0.0102472435683012,
+          -0.027474896982312202,
+          0.0019824367482215166,
+          -0.006701194681227207,
+          -0.020717859268188477,
+          0.009027997963130474,
+          -0.01924731954932213,
+          0.004816483706235886,
+          0.011419951915740967,
+          -0.00728754885494709,
+          -0.01791638880968094,
+          0.007603994105011225,
+          0.005314419511705637,
+          0.0033878260292112827,
+          -0.01682744361460209,
+          -5.409818550106138e-05,
+          -0.00768775912001729,
+          0.004092847462743521,
+          0.009595737792551517,
+          -0.013821213506162167,
+          0.015505819581449032,
+          0.006366134621202946,
+          -0.008790664374828339,
+          0.024515202268958092,
+          0.00873482134193182,
+          -0.02241177298128605,
+          -0.0018463187152519822,
+          -0.011689861305058002,
+          0.005658786278218031,
+          -0.0016799522563815117,
+          0.004206860903650522,
+          -0.011354802176356316,
+          -0.016287624835968018,
+          -0.02693507820367813,
+          -0.0008167080814018846,
+          -0.006645351182669401,
+          -0.005374916363507509,
+          0.0036554085090756416,
+          -0.006380095612257719,
+          0.011764319613575935,
+          0.00553313922137022,
+          -0.01396082155406475,
+          0.01045200228691101,
+          -0.0005985702155157924,
+          0.013169708661735058,
+          -0.026879236102104187,
+          0.005314419511705637,
+          -0.026655862107872963,
+          0.009642274118959904,
+          0.01330001000314951,
+          -0.038866929709911346,
+          0.015533741563558578,
+          -0.007101404946297407,
+          0.015570970252156258,
+          -0.0014088796451687813,
+          -0.01828867755830288,
+          0.020550329238176346,
+          0.007887864485383034,
+          -0.008841853588819504,
+          0.012332059442996979,
+          0.025073636323213577,
+          0.00967019610106945,
+          0.03830849379301071,
+          -0.016259703785181046,
+          -0.0003152818826492876,
+          -0.0064312852919101715,
+          0.007036254275590181,
+          -0.005449374206364155,
+          -0.0021301887463778257,
+          -0.020475871860980988,
+          0.029243268072605133,
+          -0.008450950495898724,
+          0.007762216962873936,
+          0.02282129041850567,
+          -0.0051050069741904736,
+          -0.009418901056051254,
+          0.004507018718868494,
+          0.012099378742277622,
+          0.018167681992053986,
+          -0.0032342569902539253,
+          0.015096302144229412,
+          -0.015487205237150192,
+          0.012211065739393234,
+          0.008250845596194267,
+          0.0015240564243867993,
+          -0.027903029695153236,
+          -0.022970205172896385,
+          -0.0017858217470347881,
+          0.0012750884052366018,
+          0.002101103775203228,
+          0.012434438802301884,
+          0.00748300040140748,
+          -0.006822188384830952,
+          0.029652785509824753,
+          -0.005798394791781902,
+          0.012099378742277622,
+          -0.0002384973777225241,
+          0.021481052041053772,
+          0.01862373575568199,
+          0.025017792358994484,
+          -0.0010325191542506218,
+          -0.01009832788258791,
+          0.005226001143455505,
+          -0.006859417073428631,
+          0.01991743966937065,
+          0.010759140364825726,
+          -0.01328139565885067,
+          -0.023752011358737946,
+          0.015049765817821026,
+          0.00011735816224245355,
+          -0.0018963449401780963,
+          0.012695040553808212,
+          -0.02336110919713974,
+          0.0009510809904895723,
+          -0.035404644906520844,
+          0.015077687799930573,
+          -0.04240366816520691,
+          -0.01672506518661976,
+          0.005221347324550152,
+          -0.014258652925491333,
+          -0.01026585791260004,
+          0.017795395106077194,
+          0.017748858779668808,
+          -0.008213616907596588,
+          0.014975308440625668,
+          -0.008697591722011566,
+          -0.009753961116075516,
+          0.013811906799674034,
+          0.012387902475893497,
+          -0.02315635047852993,
+          0.015301060862839222,
+          0.005212040152400732,
+          0.0058914669789373875,
+          -0.021871954202651978,
+          -0.02012219838798046,
+          0.006840802729129791,
+          0.01706012338399887,
+          -0.0003562917991075665,
+          0.0016194552881643176,
+          0.006631390191614628,
+          0.003539068391546607,
+          0.00614741537719965,
+          0.003271485911682248,
+          0.0003123733913525939,
+          -0.000290705036604777,
+          0.02274683304131031,
+          0.011140735819935799,
+          -0.015524433925747871,
+          0.0071293264627456665,
+          -0.009335136041045189,
+          0.0005508707836270332,
+          0.009377018548548222,
+          -1.0870535334106535e-05,
+          0.014649556018412113,
+          -0.008176388218998909,
+          0.013653683476150036,
+          0.02043864317238331,
+          -0.007445771712809801,
+          -0.003076034365221858,
+          -0.008948886767029762,
+          0.006891992408782244,
+          0.016082866117358208,
+          0.025148093700408936,
+          -0.017748858779668808,
+          -0.0002715089067351073,
+          -0.009032651782035828,
+          -0.009437515400350094,
+          -0.026842007413506508,
+          0.017050817608833313,
+          0.020289728417992592,
+          0.0073573533445596695,
+          0.017795395106077194,
+          -0.00849748682230711,
+          0.011326880194246769,
+          -0.010191399604082108,
+          -0.0023244768381118774,
+          -0.00374150020070374,
+          -0.012648504227399826,
+          0.002643248997628689,
+          0.023826468735933304,
+          -0.013914286158978939,
+          0.00790182501077652,
+          -0.0013646703446283937,
+          0.03681934252381325,
+          -0.011233807541429996,
+          0.005612250417470932,
+          0.02019665576517582,
+          -0.004979359917342663,
+          -0.011773626320064068,
+          0.02295159175992012,
+          0.008841853588819504,
+          0.0035530291497707367,
+          0.005635518580675125,
+          0.024031227454543114,
+          0.011224500834941864,
+          0.0016043310752138495,
+          0.000546508003026247,
+          -0.04716896265745163,
+          0.00888838991522789,
+          -0.001733468729071319,
+          -0.010638146661221981,
+          0.017525484785437584,
+          0.010619531385600567,
+          0.009395632892847061,
+          -0.003269159235060215,
+          0.019191475585103035,
+          -0.012266908772289753,
+          0.012508896179497242,
+          0.033636271953582764,
+          0.0034157477784901857,
+          -0.019014639779925346,
+          0.0021848685573786497,
+          0.018009459599852562,
+          -0.0024222026113420725,
+          0.00217090779915452,
+          0.006705848034471273,
+          -0.017190424725413322,
+          0.0004281318688299507,
+          0.0036903105210512877,
+          0.029224654659628868,
+          -0.015180067159235477,
+          -0.027828572317957878,
+          0.0004447103419806808,
+          -0.005872852168977261,
+          -0.003073707688599825,
+          0.008841853588819504,
+          0.0032924271654337645,
+          -0.0016217821976169944,
+          -0.012722962535917759,
+          0.0017323052743449807,
+          0.007743602618575096,
+          0.010787061415612698,
+          0.003583277575671673,
+          -0.005965924356132746,
+          -0.008916311897337437,
+          0.021462436765432358,
+          -0.006045036017894745,
+          -0.0038834353908896446,
+          -0.0035925847478210926,
+          -0.006980410777032375,
+          -0.010172785259783268,
+          -0.012778805568814278,
+          0.011122121475636959,
+          -0.001270434819161892,
+          -0.01437033899128437,
+          -0.017004281282424927,
+          0.010461308993399143,
+          -0.0067756520584225655,
+          0.025725141167640686,
+          0.0001714563404675573,
+          -0.008032126352190971,
+          0.003369211684912443,
+          -0.015319675207138062,
+          -0.010535767301917076,
+          0.007752909790724516,
+          -0.002316333120688796,
+          -0.00495609175413847,
+          0.01091736275702715,
+          -0.0032575251534581184,
+          0.018958795815706253,
+          -0.0012913760729134083,
+          -0.008385800756514072,
+          0.002873602556064725,
+          0.0025571573060005903,
+          -0.012806727550923824,
+          -0.0006579037290066481,
+          -0.02131352201104164,
+          -0.013476846739649773,
+          -0.0014519254909828305,
+          0.014091122895479202,
+          -0.010777754709124565,
+          0.005868198815733194,
+          0.0010674211662262678,
+          0.006896646227687597,
+          -0.016371389850974083,
+          -0.010237935930490494,
+          -0.0012890491634607315,
+          0.024198757484555244,
+          0.011001127772033215,
+          -0.02319357916712761,
+          -0.010824290104210377,
+          -0.01007971353828907,
+          0.018055995926260948,
+          -0.013635069131851196,
+          0.004490730818361044,
+          0.013234859332442284,
+          -0.016194554045796394,
+          -0.0009208325645886362,
+          -0.012601968832314014,
+          -0.02220701426267624,
+          0.0003839225973933935,
+          0.032705552875995636,
+          0.012015613727271557,
+          0.017953617498278618,
+          -0.008078662678599358,
+          -0.00011510407057357952,
+          0.03168175742030144,
+          0.0032505446579307318,
+          0.0018253774615004659,
+          -0.01437033899128437,
+          0.009381672367453575,
+          -0.009493358433246613,
+          0.018875030800700188,
+          0.006612775847315788,
+          -0.00827411375939846,
+          0.003231930313631892,
+          -0.02233731560409069,
+          -0.003841552883386612,
+          -0.005244615487754345,
+          -0.009698117151856422,
+          0.012825341895222664,
+          -0.007040907628834248,
+          -0.015543048270046711,
+          0.0005316746537573636,
+          -0.007124672643840313,
+          -0.015980487689375877,
+          -0.008758088573813438,
+          -0.011373416520655155,
+          -0.030062302947044373,
+          -0.027567969635128975,
+          0.0010226302547380328,
+          -0.005868198815733194,
+          -0.028461461886763573,
+          -0.019359005615115166,
+          -0.022393157705664635,
+          0.009097802452743053,
+          -0.012369288131594658,
+          -0.02101569063961506,
+          0.007157247979193926,
+          0.005095699802041054,
+          -0.01842828467488289,
+          -0.026469718664884567,
+          -0.014435489661991596,
+          -0.016334161162376404,
+          0.025278395041823387,
+          0.012713654898107052,
+          -0.0008283421047963202,
+          -0.005174811463803053,
+          0.012220372445881367,
+          0.0032784664072096348,
+          0.01127103716135025,
+          -0.03268693760037422,
+          -0.03129085525870323,
+          -0.011187272146344185,
+          -0.00430691335350275,
+          0.0007451588753610849,
+          -0.03300338238477707,
+          0.002910831244662404,
+          0.003069054102525115,
+          -0.03827126696705818,
+          -0.008706899359822273,
+          0.015040459111332893,
+          -0.0009749307064339519,
+          0.011261729523539543,
+          0.00635217409580946,
+          0.026618633419275284,
+          0.02332388050854206,
+          0.0014437816571444273,
+          4.0937200537882745e-05,
+          0.012211065739393234,
+          -0.014705399051308632,
+          0.0014961347915232182,
+          0.005775126628577709,
+          0.008585905656218529,
+          -0.0004086449043825269,
+          0.015031151473522186,
+          -0.001642723334953189,
+          -0.015403440222144127,
+          0.017311418429017067,
+          0.01541274692863226,
+          0.00934909749776125,
+          0.002934099407866597,
+          0.0009121070033870637,
+          0.0038019970525056124,
+          0.0037880362942814827,
+          0.006770998705178499,
+          -0.01227621641010046,
+          -0.006417324300855398,
+          0.016901900991797447,
+          -0.011829469352960587,
+          -0.013942207209765911,
+          0.019805751740932465,
+          -0.00024489607312716544,
+          -0.023826468735933304,
+          0.02758658491075039,
+          0.006328905932605267,
+          -0.004479097202420235,
+          -0.02537146769464016,
+          -0.0018323578406125307,
+          -0.024664118885993958,
+          0.015543048270046711,
+          0.005407491698861122,
+          0.0015449975617229939,
+          0.022597916424274445,
+          0.01108489278703928,
+          0.0038206116296350956,
+          -0.013206937350332737,
+          0.006231179926544428,
+          0.01106627844274044,
+          0.015896722674369812,
+          -0.03704271465539932,
+          -0.02455243282020092,
+          0.012285523116588593,
+          -0.01841897703707218,
+          -0.005244615487754345,
+          0.012788113206624985,
+          0.0009766757721081376,
+          -0.02019665576517582,
+          0.0031667796429246664,
+          0.003608872415497899,
+          0.004758313298225403,
+          0.012192451395094395,
+          0.016631992533802986,
+          0.0122948307543993,
+          -0.0018521357560530305,
+          -0.022970205172896385,
+          0.006007806863635778,
+          -0.008241538889706135,
+          0.0007137469947338104,
+          -0.023342493921518326,
+          -0.0018102532485499978,
+          0.010610224679112434,
+          0.00767845194786787,
+          0.025874055922031403,
+          0.030676579102873802,
+          0.027903029695153236,
+          0.013244166038930416,
+          -0.004942130763083696,
+          -0.014416875317692757,
+          -0.020959846675395966,
+          0.0019917439203709364,
+          0.002347744768485427,
+          -0.011568867601454258,
+          -0.001557794981636107,
+          0.017488256096839905,
+          0.004011409357190132,
+          0.021816110238432884,
+          -0.005793740972876549,
+          0.017460335046052933,
+          -3.364776057424024e-05,
+          -0.005212040152400732,
+          -0.0008463748381473124,
+          -0.003173760138452053,
+          0.013244166038930416,
+          0.006650005001574755,
+          0.026004357263445854,
+          0.00022031922708265483,
+          0.003708925098180771,
+          0.005481949541717768,
+          -0.0007905315142124891,
+          -0.0008289237739518285,
+          -0.007380621042102575,
+          -0.003539068391546607,
+          -0.009847032837569714,
+          0.034678682684898376,
+          0.025278395041823387,
+          -0.02250484563410282,
+          -0.015124224126338959,
+          -0.012713654898107052,
+          0.02639526128768921,
+          -0.010833597742021084,
+          0.006524357479065657,
+          0.010796369053423405,
+          0.008660363033413887,
+          0.01065676100552082,
+          0.000908035144675523,
+          0.005719283130019903,
+          0.0028898902237415314,
+          0.022672373801469803,
+          -0.012611275538802147,
+          0.008669670671224594,
+          0.003920664079487324,
+          0.008357878774404526,
+          -0.015077687799930573,
+          -0.001618291949853301,
+          0.005872852168977261,
+          -0.014919465407729149,
+          0.006170683074742556,
+          -0.004118442535400391,
+          0.006207912229001522,
+          0.011559560894966125,
+          0.0015775728970766068,
+          -0.01607356034219265,
+          -0.017935002222657204,
+          0.012927721254527569,
+          -0.009781882166862488,
+          0.00916295312345028,
+          -0.018028074875473976,
+          -0.011606096290051937,
+          0.017897773534059525,
+          -0.006119493395090103,
+          -0.011931848712265491,
+          0.0026292880065739155,
+          0.00411146180704236,
+          -5.733389480155893e-05,
+          0.004009082447737455,
+          0.007752909790724516,
+          0.0020592212677001953,
+          0.00535630201920867,
+          0.005947310011833906,
+          -0.002478045877069235,
+          0.002440816955640912,
+          -0.0019789463840425014,
+          0.035739704966545105,
+          -0.009847032837569714,
+          0.006617429666221142,
+          -0.0022372216917574406,
+          -0.0012099378509446979,
+          -0.04154740646481514,
+          0.027772728353738785,
+          -0.006263755261898041,
+          0.015319675207138062,
+          0.00801816489547491,
+          -0.003401787020266056,
+          -0.010768447071313858,
+          0.008353224955499172,
+          0.01050784531980753,
+          -0.02135075069963932,
+          0.015142838470637798,
+          -0.024198757484555244,
+          -0.004218494985252619,
+          -0.002433836692944169,
+          -0.01108489278703928,
+          0.015496511943638325,
+          -0.01459371205419302,
+          0.008092623203992844,
+          -0.01091736275702715,
+          -0.0073713138699531555,
+          0.010144864208996296,
+          -0.0072549739852547646,
+          0.011010434478521347,
+          -0.042924873530864716,
+          0.0067756520584225655,
+          -0.004821137059479952,
+          -0.0010517152259126306,
+          -0.0006689560250379145,
+          -0.02270960435271263,
+          -0.031067483127117157,
+          -0.000392648100387305,
+          0.004853712394833565,
+          -0.005384223535656929,
+          0.00748300040140748,
+          0.005407491698861122,
+          0.004355776589363813,
+          0.0032621787395328283,
+          -0.01954515092074871,
+          0.025390081107616425,
+          0.03665181249380112,
+          -0.011829469352960587,
+          0.013383775018155575,
+          0.007161901798099279,
+          0.008148466236889362,
+          -0.03622367978096008,
+          -0.02352863922715187,
+          -0.008450950495898724,
+          0.009181567467749119,
+          0.02591128461062908,
+          -0.004679202102124691,
+          0.01559889130294323,
+          0.009437515400350094,
+          0.0050910464487969875,
+          -0.02870344929397106,
+          -0.015403440222144127,
+          -0.008046086877584457,
+          -0.015654735267162323,
+          -0.0022442021872848272,
+          -0.0033017343375831842,
+          0.014696091413497925,
+          -0.00473271869122982,
+          -0.018260754644870758,
+          -0.011941156350076199,
+          -0.006864070892333984,
+          -0.0037368466146290302,
+          -0.010154170915484428,
+          0.023547252640128136,
+          -0.010833597742021084,
+          -0.015589584596455097,
+          0.02512947842478752,
+          -0.001938227447681129,
+          0.005793740972876549,
+          0.005235308315604925,
+          0.00750161474570632,
+          -0.003304061247035861,
+          -0.004590783733874559,
+          -0.007673798594623804,
+          0.017627865076065063,
+          0.004202207550406456,
+          0.02874067798256874,
+          -0.004532613325864077,
+          0.00770637346431613,
+          0.0016764620086178184,
+          0.004355776589363813,
+          -0.002505967626348138,
+          0.008599866181612015,
+          0.006003153510391712,
+          -0.003648428013548255,
+          0.0071991304866969585,
+          -0.01350476872175932,
+          0.002410568529739976,
+          0.024943334981799126,
+          0.005984539166092873,
+          0.010414772666990757,
+          -0.012629889883100986,
+          -0.012667118571698666,
+          -0.003418074455112219,
+          -0.016203859820961952,
+          0.015924643725156784,
+          -0.020662017166614532,
+          0.018670272082090378,
+          -0.017460335046052933,
+          -0.001818397082388401,
+          0.0043930052779614925,
+          0.015198681503534317,
+          0.021090148016810417,
+          -0.004346469417214394,
+          0.026469718664884567,
+          0.013728141784667969,
+          0.0035227807238698006,
+          0.004970052745193243,
+          -0.022653760388493538,
+          -0.005514524411410093,
+          -0.011094199493527412,
+          0.003913683816790581,
+          -0.008581251837313175,
+          -0.016734370961785316,
+          -0.0010197217343375087,
+          0.007315470837056637,
+          -0.005398184526711702,
+          0.003701944602653384,
+          0.012657811865210533,
+          0.0015484878094866872,
+          0.0028293931391090155,
+          0.014230730943381786,
+          0.01651099883019924,
+          0.007510922383517027,
+          -0.007622608914971352,
+          0.0040323506109416485,
+          0.006622083019465208,
+          -0.0046280124224722385,
+          0.0009504992631264031,
+          0.017395183444023132,
+          0.01438895333558321,
+          -0.004060272127389908,
+          0.0053656091913580894,
+          -0.0027805303689092398,
+          -0.02676755003631115,
+          -0.014267959631979465,
+          -0.0003990468103438616,
+          -0.0005968251498416066,
+          -0.003506493056192994,
+          -0.015161452814936638,
+          0.004483750555664301,
+          0.012155221775174141,
+          -0.0122948307543993,
+          0.0035181271377950907,
+          0.008399761281907558,
+          -0.009949412196874619,
+          0.0021976660937070847,
+          -0.0034785715397447348,
+          -0.014249345287680626,
+          -0.009791189804673195,
+          0.027623813599348068,
+          -0.01011694222688675,
+          0.013746756128966808,
+          0.013635069131851196,
+          0.00039875597576610744,
+          -0.006440592464059591,
+          -0.0013041734928265214,
+          0.019061176106333733,
+          0.01746964268386364,
+          -0.001688096090219915,
+          -0.00036327220732346177,
+          -0.007552804425358772,
+          -0.006161375902593136,
+          0.01350476872175932,
+          -0.01634346880018711,
+          -0.004311567172408104,
+          0.009577123448252678,
+          -0.00846025813370943,
+          -0.004593110177665949,
+          -0.024533817544579506,
+          0.0021522934548556805,
+          0.009135031141340733,
+          0.0035227807238698006,
+          -0.0068314955569803715,
+          -0.001905652228742838,
+          -0.002934099407866597,
+          0.0058030481450259686,
+          -0.00118550646584481,
+          -0.00687803141772747,
+          -0.016836751252412796,
+          -0.0024175490252673626,
+          0.0070920973084867,
+          -0.006184644065797329,
+          -0.001833521295338869,
+          0.0036251600831747055,
+          -0.006082264706492424,
+          -0.015673348680138588,
+          0.03244495019316673,
+          9.452639642404392e-05,
+          0.021332135424017906,
+          0.004132403060793877,
+          0.0061334543861448765,
+          0.0027246871031820774,
+          0.007222398649901152,
+          0.02561345510184765,
+          0.009614353068172932,
+          0.006747730541974306,
+          0.01621316745877266,
+          0.014975308440625668,
+          -0.014240038581192493,
+          0.0072549739852547646,
+          0.012853262946009636,
+          -0.010163478553295135,
+          0.00417195912450552,
+          0.009251371026039124,
+          -0.0007649366743862629,
+          -0.003988141659647226,
+          -0.0037228858564049006,
+          0.016799522563815117,
+          0.003071380779147148,
+          -0.0008132178918458521,
+          0.012183143757283688,
+          0.01166194025427103,
+          -0.018754037097096443,
+          -0.00033796823117882013,
+          -0.0015135857975110412,
+          0.002343091182410717,
+          -0.005882159806787968,
+          -0.026581404730677605,
+          -0.008320650085806847,
+          0.0033854993525892496,
+          -0.0030434592626988888,
+          0.03091856651008129,
+          -0.00950266607105732,
+          -0.03354320302605629,
+          -0.01563611999154091,
+          -0.001225062063895166,
+          -0.01692982390522957,
+          -0.007966975681483746,
+          0.01276949793100357,
+          0.0295224841684103,
+          0.012415824458003044,
+          -0.013802599161863327,
+          0.024664118885993958,
+          -0.0028131057042628527,
+          0.01750687137246132,
+          -0.00870224554091692,
+          -0.009083840996026993,
+          -0.004509345628321171,
+          0.022113941609859467,
+          -0.008744128048419952,
+          -0.009605045430362225,
+          0.013579226098954678,
+          0.016985666006803513,
+          0.00842768233269453,
+          0.02323080785572529,
+          0.013644376769661903,
+          -0.007022293284535408,
+          -0.014891543425619602,
+          0.003808977548032999,
+          -0.008595212362706661,
+          -0.020252499729394913,
+          0.007324778009206057,
+          -0.014900850132107735,
+          0.005416798871010542,
+          -0.0021348423324525356,
+          -0.023696167394518852,
+          0.005537792574614286,
+          0.0053656091913580894,
+          -0.012043535709381104,
+          -0.012546124868094921,
+          0.000147243044921197,
+          -0.009507319889962673,
+          0.014463411644101143,
+          0.0068314955569803715,
+          -0.00023253493418451399,
+          -0.009251371026039124,
+          0.017832623794674873,
+          0.011001127772033215,
+          0.009968026541173458,
+          -0.00792043935507536,
+          0.0033924796152859926,
+          0.008934926241636276,
+          0.007590033579617739,
+          -0.0014495986979454756,
+          -0.0056634400971233845,
+          0.037880364805459976,
+          0.0030620736069977283,
+          0.012527510523796082,
+          -0.0017451026942580938,
+          0.009865647181868553,
+          0.01086151972413063,
+          0.015291753225028515,
+          0.010256550274789333,
+          -0.003646101336926222,
+          -0.0038275918923318386,
+          0.013458232395350933,
+          -0.012024921365082264,
+          0.0055052172392606735,
+          0.010275164619088173,
+          -0.0046675680205225945,
+          -0.005384223535656929,
+          0.010647453367710114,
+          0.00014309842663351446,
+          0.01812114752829075,
+          -0.005900774151086807,
+          0.009288599714636803,
+          0.018335212022066116,
+          0.009577123448252678,
+          -0.019098404794931412,
+          0.013728141784667969,
+          -0.01556166261434555,
+          0.029950616881251335,
+          0.01917286217212677,
+          -0.01812114752829075,
+          -0.0018975083949044347,
+          -0.011019742116332054,
+          0.009763267822563648,
+          -0.04221752658486366,
+          -0.01190392766147852,
+          0.0023919541854411364,
+          -0.004090521018952131,
+          -0.018884338438510895,
+          -0.027028150856494904,
+          0.0044558290392160416,
+          -0.004430233966559172,
+          -0.009353750385344028,
+          -0.008278767578303814,
+          0.0006509232916869223,
+          0.015924643725156784,
+          -0.005221347324550152,
+          -0.0077715241350233555,
+          -0.024310443550348282,
+          0.014993922784924507,
+          0.00886977557092905,
+          -0.004176612477749586,
+          -0.024403516203165054,
+          0.01455648336559534,
+          0.01603632979094982,
+          -0.00687803141772747,
+          0.01190392766147852,
+          -0.024049842730164528,
+          0.00197545625269413,
+          -0.005700668785721064,
+          -0.01654822751879692,
+          -0.015263832174241543,
+          0.0010999963851645589,
+          -0.012164529412984848,
+          -0.003313368419185281,
+          -0.014119044877588749,
+          -0.009083840996026993,
+          -0.0028945438098162413,
+          0.020475871860980988,
+          0.0025641375686973333,
+          -0.01541274692863226,
+          0.016390005126595497,
+          0.007869250141084194,
+          -0.0229888204485178,
+          -0.01631554774940014,
+          0.0046605877578258514,
+          -0.029876159504055977,
+          0.010805675759911537,
+          0.007869250141084194,
+          0.006826841738075018,
+          -0.02189056947827339,
+          0.007683105766773224,
+          0.02241177298128605,
+          0.007994897663593292,
+          0.0017660439480096102,
+          0.01913563348352909,
+          0.008185694925487041,
+          0.012667118571698666,
+          0.0019696392118930817,
+          -3.335691189931822e-06,
+          -0.005849584471434355,
+          -0.004148690961301327,
+          -0.004714104346930981,
+          0.03802927955985069,
+          -0.01617593877017498,
+          0.009498012252151966,
+          -0.022560687735676765,
+          0.0002395153569523245,
+          -0.01921009086072445,
+          -0.011298958212137222,
+          -0.015980487689375877,
+          0.0008010021410882473,
+          0.02151828072965145,
+          -0.007892518304288387,
+          -0.0002912867348641157,
+          -0.01583157107234001,
+          -0.0058356234803795815,
+          -0.006542971823364496,
+          0.02036418579518795,
+          -0.012741576880216599,
+          0.0018626062665134668,
+          -0.017311418429017067,
+          0.0010249570477753878,
+          0.026693090796470642,
+          0.005556406918913126,
+          -0.013225551694631577,
+          -0.026618633419275284,
+          -0.016194554045796394,
+          -0.009158299304544926,
+          -0.01316040102392435,
+          -0.005239961668848991,
+          -0.019228706136345863,
+          -0.011364108882844448,
+          -0.006473167799413204,
+          0.00654762564226985,
+          0.004039331339299679,
+          -0.015012537129223347,
+          -0.004997974261641502,
+          -0.024664118885993958,
+          -0.0202338844537735,
+          -0.010014562867581844,
+          -0.009470091201364994,
+          0.0013844481436535716,
+          0.02853591926395893,
+          0.0028549879789352417,
+          0.030397363007068634,
+          -0.00415567122399807,
+          0.008790664374828339,
+          0.009828418493270874,
+          0.007203784305602312,
+          -0.01520798821002245,
+          0.007487654220312834,
+          0.014584405347704887,
+          -0.008530061691999435,
+          0.0057890876196324825,
+          -0.0122948307543993,
+          -0.0007463222718797624,
+          0.005249268840998411,
+          0.004970052745193243,
+          0.011261729523539543,
+          -0.007548151072114706,
+          -0.01580365002155304,
+          0.013495461083948612,
+          -0.02040141448378563,
+          0.0011971404310315847,
+          0.00031295508961193264,
+          -0.003236583899706602,
+          0.003632140578702092,
+          0.018102532252669334,
+          -0.01336515974253416,
+          0.001435637823306024,
+          0.01999189704656601,
+          -0.012536818161606789,
+          0.01316040102392435,
+          0.003013210603967309,
+          0.0010214668000116944,
+          0.009511973708868027,
+          0.0011215193662792444,
+          0.005156196653842926,
+          0.0005630864761769772,
+          -0.019228706136345863,
+          -0.006966450251638889,
+          0.00831134244799614,
+          -0.024608274921774864,
+          -0.01128965150564909,
+          0.011177964508533478,
+          0.006254448089748621,
+          -0.017832623794674873,
+          -0.012788113206624985,
+          -0.023863697424530983,
+          0.01166194025427103,
+          -0.005467988550662994,
+          0.023993998765945435,
+          -0.015105608850717545,
+          0.009204835630953312,
+          -0.007738948799669743,
+          0.021164605394005775,
+          -0.0122948307543993,
+          0.005212040152400732,
+          0.0013576899655163288,
+          -0.01665060594677925,
+          0.009567816741764545,
+          -0.012285523116588593,
+          -0.0017567367758601904,
+          0.042775958776474,
+          0.006952489260584116,
+          0.0074271573685109615,
+          0.015068380162119865,
+          -0.022020868957042694,
+          0.01852135732769966,
+          0.002450124127790332,
+          0.018502742052078247,
+          0.010572995990514755,
+          -0.019265934824943542,
+          0.01695774495601654,
+          0.026823392137885094,
+          0.01982436701655388,
+          0.006221872754395008,
+          -0.01147579587996006,
+          0.027903029695153236,
+          0.012862570583820343,
+          0.0006660475628450513,
+          0.0022569994907826185,
+          0.0016846058424562216,
+          -0.011038356460630894,
+          -0.010842905379831791,
+          0.009018691256642342,
+          -0.017730243504047394,
+          -0.013272088021039963,
+          -0.0030923220328986645,
+          0.003080687951296568,
+          -0.010219321586191654,
+          -0.0036996176932007074,
+          0.006719809025526047,
+          0.007399235386401415,
+          0.011419951915740967,
+          0.0016020042821764946,
+          0.01611078903079033,
+          -0.004774601198732853,
+          -0.01893087476491928,
+          0.0002504222502466291,
+          -0.0032598518300801516,
+          0.014947386458516121,
+          -0.029299112036824226,
+          -0.007911132648587227,
+          -0.0004516907501965761,
+          0.005267883650958538,
+          0.006035728845745325,
+          0.007832021452486515,
+          -0.015096302144229412,
+          -0.005682054441422224,
+          -0.003113263286650181,
+          0.016120094805955887,
+          0.0016811155946925282,
+          0.023212192580103874,
+          0.015608198940753937,
+          -0.009968026541173458,
+          -0.006328905932605267,
+          0.0032668323256075382,
+          -0.010982513427734375,
+          0.00017320144979748875,
+          0.0025943859945982695,
+          0.011122121475636959,
+          -0.018847109749913216,
+          -0.008208963088691235,
+          0.025594839826226234,
+          0.02015942707657814,
+          0.00357629731297493,
+          0.026748934760689735,
+          -0.014174887910485268,
+          0.008227577432990074,
+          0.019842982292175293,
+          0.014407568611204624,
+          -0.010572995990514755,
+          -0.010731218382716179,
+          -0.0046675680205225945,
+          0.001287885825149715,
+          -0.0027549355290830135,
+          -0.005179464817047119,
+          0.006161375902593136,
+          -0.022467615082859993,
+          -0.016259703785181046,
+          0.0012157548917457461,
+          -0.013793292455375195,
+          0.0032342569902539253,
+          0.009358404204249382,
+          0.018502742052078247,
+          -0.011736397631466389,
+          0.01787916012108326,
+          -0.026879236102104187,
+          -0.003480898216366768,
+          0.010256550274789333,
+          0.0019545149989426136,
+          -0.016808830201625824,
+          -0.003911356907337904,
+          -0.002536216052249074,
+          -0.009218796156346798,
+          -0.006989717949181795,
+          0.01671575754880905,
+          0.012397210113704205,
+          0.015059073455631733,
+          -0.008195002563297749,
+          0.004423253703862429,
+          0.0023547252640128136,
+          0.009395632892847061,
+          -0.0018847109749913216,
+          -0.009442169219255447,
+          0.006696540862321854,
+          0.0017509197350591421,
+          -0.005863544996827841,
+          -0.005281844176352024,
+          0.023063277825713158,
+          -0.026134658604860306,
+          0.003641447750851512,
+          -0.016501691192388535,
+          0.011410645209252834,
+          0.012257601134479046,
+          0.002092959824949503,
+          -0.005598289426416159,
+          -0.006994371768087149,
+          0.013458232395350933,
+          -0.06496435403823853,
+          -0.008432336151599884,
+          -0.0015938604483380914,
+          0.027623813599348068,
+          -0.01794430986046791,
+          0.003169106552377343,
+          0.008404415100812912,
+          0.015422054566442966,
+          -0.007189823314547539,
+          0.007101404946297407,
+          0.016324853524565697,
+          -0.02261653169989586,
+          -0.011326880194246769,
+          0.00187075010035187,
+          -0.004974706098437309,
+          -0.011838776990771294,
+          0.0010470616398379207,
+          0.004500038456171751,
+          0.018940182402729988,
+          0.0015496512642130256,
+          0.011401337571442127,
+          -0.023510023951530457,
+          -0.013393081724643707,
+          -0.00436042994260788,
+          0.009060573764145374,
+          -0.005910081323236227,
+          0.0040323506109416485,
+          0.00931652169674635,
+          -0.03136531263589859,
+          0.011224500834941864,
+          0.008450950495898724,
+          -0.007241012994199991,
+          0.009605045430362225,
+          0.011326880194246769,
+          0.008474218659102917,
+          0.005267883650958538,
+          0.0011447874130681157,
+          -0.03210989013314247,
+          0.0043162209913134575,
+          0.013337238691747189,
+          -0.009200181812047958,
+          -0.013206937350332737,
+          -0.009977334178984165,
+          -0.0012366961454972625,
+          0.019973281770944595,
+          -0.004530286882072687,
+          -0.0018742403481155634,
+          0.014640248380601406,
+          0.006021767854690552,
+          0.01455648336559534,
+          -0.013337238691747189,
+          0.004942130763083696,
+          -0.010666067712008953,
+          -0.00854867696762085,
+          -0.0014100429834797978,
+          -0.0001836720621213317,
+          -0.0028968704864382744,
+          -0.038159579038619995,
+          0.00965158175677061,
+          -0.0036693692672997713,
+          -0.0197871383279562,
+          -0.01248097512871027,
+          -0.013411696068942547,
+          -0.004532613325864077,
+          -0.010870826430618763,
+          0.006854763720184565,
+          0.0031877208966761827,
+          0.009135031141340733,
+          -0.01336515974253416,
+          0.02496195025742054,
+          -0.014044586569070816,
+          -0.009977334178984165,
+          -0.009107109159231186,
+          -0.0005622139433398843,
+          0.002806125208735466,
+          0.0149846151471138,
+          -0.0038624941371381283,
+          -0.003204008564352989,
+          0.021853338927030563,
+          0.008013512007892132,
+          0.006114840041846037,
+          -0.01188531331717968,
+          -0.018875030800700188,
+          -0.023137735202908516,
+          -0.0029015240725129843,
+          -0.021946411579847336,
+          0.0007562111713923514,
+          0.004853712394833565,
+          0.02377062663435936,
+          0.007720334455370903,
+          -9.13997573661618e-05,
+          0.00436042994260788,
+          0.0038206116296350956,
+          0.019452078267931938,
+          -0.008371839299798012,
+          -0.004046311601996422,
+          -0.0071432869881391525,
+          -0.002633941825479269,
+          0.03482759743928909,
+          -0.008423029445111752,
+          0.00037955984589643776,
+          0.0013239512918516994,
+          -0.008879082277417183,
+          -0.007590033579617739,
+          0.010703296400606632,
+          -0.02881513722240925,
+          -0.014919465407729149,
+          -0.00036181797622703016,
+          0.0024873530492186546,
+          -0.010749832727015018,
+          0.02073647454380989,
+          -0.008502140641212463,
+          0.0023023721296340227,
+          0.009256024844944477,
+          0.024161528795957565,
+          0.022430386394262314,
+          0.023137735202908516,
+          0.007315470837056637,
+          -0.003671696176752448,
+          -0.002410568529739976,
+          -0.0067616915330290794,
+          -0.02496195025742054,
+          0.0161387100815773,
+          -0.0040951743721961975,
+          0.03404578939080238,
+          -0.03428777679800987,
+          0.0012413497315719724,
+          -0.007380621042102575,
+          -0.023472795262932777,
+          -0.009986640885472298,
+          0.023174963891506195,
+          -0.01045200228691101,
+          0.023900926113128662,
+          -0.0003123733913525939,
+          0.01556166261434555,
+          0.007906478829681873,
+          0.008441643789410591,
+          -0.01930316351354122,
+          0.011810855008661747,
+          -0.0023372743744403124,
+          0.01822352595627308,
+          0.009307214990258217,
+          -0.004607071168720722,
+          0.008832546882331371,
+          0.011689861305058002,
+          -0.01518937386572361,
+          -0.011196578852832317,
+          -0.0029015240725129843,
+          0.0018474820535629988,
+          -0.009847032837569714,
+          0.008553329855203629,
+          -0.002062711399048567,
+          -0.01600840874016285,
+          0.02537146769464016,
+          -0.01593395136296749,
+          0.010368237271904945,
+          0.004904902074486017,
+          0.00647782115265727,
+          0.01736726239323616,
+          -0.001187833258882165,
+          -0.027400439605116844,
+          0.0002552212681621313,
+          -0.03222157806158066,
+          -0.012341366149485111,
+          0.001407716190442443,
+          0.006566239986568689,
+          -0.0030434592626988888,
+          -0.010666067712008953,
+          -0.01883780211210251,
+          -0.0060310750268399715,
+          0.024571046233177185,
+          0.03439946472644806,
+          0.007743602618575096,
+          0.011745704337954521,
+          -0.011122121475636959,
+          -0.00967019610106945,
+          -0.006207912229001522,
+          -0.01917286217212677,
+          -0.009004729799926281,
+          -0.005863544996827841,
+          -0.00813915953040123,
+          0.011531638912856579,
+          -0.011568867601454258,
+          0.016538919880986214,
+          0.030118146911263466,
+          -0.031830672174692154,
+          -0.008530061691999435,
+          0.01570127159357071,
+          0.008651056326925755,
+          0.013253473676741123,
+          0.008804624900221825,
+          0.007631916087120771,
+          -0.011233807541429996,
+          -0.005817009136080742,
+          0.000602642132434994,
+          0.0062032584100961685,
+          -0.013802599161863327,
+          0.02162996679544449,
+          0.0010011072736233473,
+          0.002340764505788684,
+          0.004704797174781561,
+          0.01128965150564909,
+          -0.012760191224515438,
+          0.00357629731297493,
+          -0.007613301742821932,
+          -0.02537146769464016,
+          -0.026674477383494377,
+          -0.008134505711495876,
+          -0.006724462378770113,
+          -0.020680630579590797,
+          0.002073182025924325,
+          0.006333559285849333,
+          0.01950792223215103,
+          0.014724013395607471,
+          -0.009246718138456345,
+          -0.007250320166349411,
+          -0.024868877604603767,
+          0.010777754709124565,
+          -0.026376646012067795,
+          -0.02295159175992012,
+          0.027735499665141106,
+          0.013541997410356998,
+          0.014333110302686691,
+          -0.0016043310752138495,
+          0.0024315097834914923,
+          -0.010526459664106369,
+          0.013430310413241386,
+          0.02213255688548088,
+          0.004597763996571302,
+          0.007403889205306768,
+          -0.015105608850717545,
+          -0.002538542728871107,
+          0.020289728417992592,
+          -0.00022119177447166294,
+          -0.005002627614885569,
+          0.022486230358481407,
+          0.015422054566442966,
+          0.01886572316288948,
+          -0.004904902074486017,
+          0.015943259000778198,
+          0.01648307777941227,
+          0.008920964784920216,
+          -0.004106808453798294,
+          0.00020301362383179367,
+          -0.020997075363993645,
+          -0.004835098050534725,
+          0.03337567299604416,
+          0.012341366149485111,
+          0.025762369856238365,
+          -0.0263208020478487,
+          0.004825790878385305,
+          -0.004141710698604584,
+          0.013020792976021767,
+          0.01047061663120985,
+          0.019638223573565483,
+          0.028442848473787308,
+          0.009395632892847061,
+          0.009902875870466232,
+          -0.0012238987255841494,
+          0.029224654659628868,
+          -0.012611275538802147,
+          -0.009902875870466232,
+          -0.0006788449827581644,
+          -0.006622083019465208,
+          -0.004211514722555876,
+          0.00010339733853470534,
+          0.01476124208420515,
+          0.002815432380884886,
+          -0.019061176106333733,
+          0.014146965928375721,
+          0.007054868619889021,
+          -0.018819188699126244,
+          0.012127300724387169,
+          -0.02151828072965145,
+          0.011373416520655155,
+          -0.02738182619214058,
+          -0.0053656091913580894,
+          -0.01377467717975378,
+          -0.009358404204249382,
+          -0.023174963891506195,
+          -0.006933874916285276,
+          0.004034677520394325,
+          0.0082927281036973,
+          -0.0029317724984139204,
+          0.0043301815167069435,
+          -0.0030550931114703417,
+          -0.01600840874016285,
+          0.007929746992886066,
+          0.030937181785702705,
+          0.008716206066310406,
+          0.007450425066053867,
+          -0.0035530291497707367,
+          0.011224500834941864,
+          -0.008972154930233955,
+          -0.02213255688548088,
+          0.0002355888718739152,
+          -0.013467539101839066,
+          -0.008562637493014336,
+          0.013532689772546291,
+          0.006515050306916237,
+          -0.014202809892594814,
+          0.00905591994524002,
+          -0.024440744891762733,
+          -0.0002466411970090121,
+          -0.017786087468266487,
+          -0.00875343568623066,
+          -0.0053656091913580894,
+          -0.015021844767034054,
+          0.0013483826769515872,
+          0.007617955096065998,
+          -0.00890700425952673,
+          -0.006277716252952814,
+          0.00968881044536829,
+          -0.0037926898803561926,
+          -0.014333110302686691,
+          -0.0038880889769643545,
+          -0.008916311897337437,
+          -0.002316333120688796,
+          0.0018905280157923698,
+          -0.00985634047538042,
+          -0.005095699802041054,
+          -0.005407491698861122,
+          -0.026506947353482246,
+          0.006812881212681532,
+          0.019600993022322655,
+          0.004602417815476656,
+          0.020829545333981514,
+          0.0029573673382401466,
+          0.010526459664106369,
+          0.02931772544980049,
+          0.0055238320492208,
+          -0.008748781867325306,
+          -0.024887491017580032,
+          0.02159273810684681,
+          -0.010312393307685852,
+          0.024887491017580032,
+          -0.011447873897850513,
+          -0.014472718350589275,
+          0.0026548828464001417,
+          -0.00554244639351964,
+          -0.005267883650958538,
+          0.00035076565109193325,
+          0.006026421207934618,
+          -0.017497563734650612,
+          0.018009459599852562,
+          -0.011857391335070133,
+          0.008897697553038597,
+          0.010600917041301727,
+          0.041733551770448685,
+          -0.0013134806649759412,
+          -0.007375967688858509,
+          0.004462809301912785,
+          -0.001483337371610105,
+          0.01958237960934639,
+          -0.004497711546719074,
+          0.007282895501703024,
+          -0.009716732427477837,
+          -0.0007445771479979157,
+          -0.009265332482755184,
+          0.0062730624340474606,
+          0.015133530832827091,
+          -0.016771601513028145,
+          -0.01328139565885067,
+          0.0055052172392606735,
+          -0.026506947353482246,
+          0.0086138267070055,
+          0.030248448252677917,
+          0.017562713474035263,
+          -0.007324778009206057,
+          -0.014379646629095078,
+          0.007422503549605608,
+          -0.030080918222665787,
+          -0.0028480077162384987,
+          0.09649720042943954,
+          -0.009339789859950542,
+          -0.005128275137394667,
+          -0.02336110919713974,
+          -0.006524357479065657,
+          0.019228706136345863,
+          -0.014342417940497398,
+          0.07330362498760223,
+          -0.008748781867325306,
+          0.00046128881513141096,
+          -0.0073015098460018635,
+          0.001322787837125361,
+          -0.02053171582520008,
+          0.003978834021836519,
+          -0.02196502685546875,
+          -0.024868877604603767,
+          0.0021266986150294542,
+          -0.030118146911263466,
+          -0.007366660516709089,
+          -0.0034971858840435743,
+          0.008413721807301044,
+          -0.0049188630655407906,
+          0.011857391335070133,
+          -0.009865647181868553,
+          -0.012462359853088856,
+          -0.0011639834847301245,
+          0.005677401088178158,
+          0.015654735267162323,
+          -0.026264959946274757,
+          -0.01991743966937065,
+          0.010545074008405209,
+          0.01900533214211464,
+          0.004469790030270815,
+          -0.008609173819422722,
+          0.001268108026124537,
+          -0.00514223612844944,
+          0.008590559475123882,
+          0.029708629474043846,
+          0.01089874841272831,
+          0.01108489278703928,
+          0.0037531342823058367,
+          -0.011075585149228573,
+          0.015328982844948769,
+          0.007785485126078129,
+          0.013662991113960743,
+          -0.006021767854690552,
+          -0.004816483706235886,
+          -0.0021022669970989227,
+          0.009558509103953838,
+          -0.02220701426267624,
+          -0.014119044877588749,
+          -0.012238986790180206,
+          0.016967052593827248,
+          -0.006938528269529343,
+          -0.0005441812099888921,
+          -0.009791189804673195,
+          0.013579226098954678,
+          -0.008874429389834404,
+          -0.0051236217841506,
+          0.005184118635952473,
+          -0.021276293322443962,
+          0.022467615082859993,
+          0.006864070892333984,
+          0.021983640268445015,
+          -0.002318659797310829,
+          0.003573970403522253,
+          -0.014714706689119339,
+          -0.006580200511962175,
+          0.0106939896941185,
+          -0.004718757700175047,
+          0.00822292361408472,
+          0.002288411371409893,
+          0.02894543670117855,
+          0.0005770472926087677,
+          -0.013523383066058159,
+          0.008250845596194267,
+          -0.011587481945753098,
+          -0.01791638880968094,
+          0.010182092897593975,
+          -0.020382799208164215,
+          0.009060573764145374,
+          -6.11876675975509e-05,
+          -0.025445925071835518,
+          -0.010005255229771137,
+          0.004932823590934277,
+          -0.006924567744135857,
+          -0.013141786679625511,
+          0.013718834146857262,
+          -0.01853066496551037,
+          -0.01071260403841734,
+          -0.007194476667791605,
+          0.02101569063961506,
+          -0.016846058890223503,
+          0.023379722610116005,
+          -0.020680630579590797,
+          0.00634286692366004,
+          -0.00875343568623066,
+          0.02015942707657814,
+          -0.004453502129763365,
+          0.018195604905486107,
+          0.0122948307543993,
+          0.00674307718873024,
+          0.002838700544089079,
+          0.007469039876013994,
+          0.009698117151856422,
+          0.02581821382045746,
+          0.01982436701655388,
+          -0.003245891071856022,
+          -0.008534715510904789,
+          -0.011615403927862644,
+          0.013839827850461006,
+          -0.005793740972876549,
+          -0.00922810286283493,
+          0.011782933957874775,
+          0.016027024015784264,
+          -0.014947386458516121,
+          0.014854314737021923,
+          -0.0028666220605373383,
+          0.01086151972413063,
+          -0.013904978521168232,
+          0.004013736266642809,
+          0.0009057082934305072,
+          0.012369288131594658,
+          0.0075993407517671585,
+          -0.00016491221322212368,
+          0.01541274692863226,
+          0.02040141448378563,
+          -0.0004836842999793589,
+          0.01651099883019924,
+          0.0046210321597754955,
+          0.010377543978393078,
+          0.0021104109473526478,
+          -0.0037926898803561926,
+          -0.00514223612844944,
+          0.0021871954668313265,
+          -0.007892518304288387,
+          0.008786010555922985,
+          0.011056970804929733,
+          -0.010768447071313858,
+          0.002801471622660756,
+          0.012611275538802147,
+          -0.011913234367966652,
+          -0.008511447347700596,
+          0.0038275918923318386,
+          -0.026879236102104187,
+          -0.006850109901279211,
+          0.028089173138141632,
+          -0.018214218318462372,
+          -0.007273588329553604,
+          0.010033177211880684,
+          0.012425131164491177,
+          -0.0010011072736233473,
+          -0.014472718350589275,
+          0.009079188108444214,
+          0.005216693971306086,
+          -0.02557622641324997,
+          0.0024128954391926527,
+          0.011606096290051937,
+          0.018577199429273605,
+          -0.0004990993766114116,
+          -0.010768447071313858,
+          0.0013367487117648125,
+          0.025352852419018745,
+          0.016492383554577827,
+          0.013067329302430153,
+          -0.0050910464487969875,
+          -0.003990468103438616,
+          0.0051375823095440865,
+          0.0012064477195963264,
+          0.002601366490125656,
+          0.00668257987126708,
+          -0.0012983564520254731,
+          -0.0059054275043308735,
+          0.01348615437746048,
+          -0.0010458981851115823,
+          0.012322751805186272,
+          -0.013700219802558422,
+          -0.0186981949955225,
+          -0.0013134806649759412,
+          -0.003874127985909581,
+          -0.012387902475893497,
+          0.019470693543553352,
+          -0.003285446669906378,
+          0.03709855675697327,
+          0.013458232395350933,
+          0.005086392629891634,
+          -0.002601366490125656,
+          -0.008404415100812912,
+          0.016631992533802986,
+          0.0053656091913580894,
+          -0.01520798821002245,
+          0.003927644342184067,
+          -0.029168810695409775,
+          0.021071534603834152,
+          -0.009865647181868553,
+          0.010554381646215916,
+          -0.002633941825479269,
+          0.007906478829681873,
+          0.004876980558037758,
+          -0.011466488242149353,
+          -0.003373865270987153,
+          0.009567816741764545,
+          0.0022220974788069725,
+          -0.012127300724387169,
+          -0.0015612852293998003,
+          0.008678977377712727,
+          -0.006049689371138811,
+          0.012211065739393234,
+          0.005272537004202604,
+          -0.008520754985511303,
+          0.00436741067096591
+        ],
+        "title": "Conclusion"
+      },
+      "type": "chunk"
+    },
+    {
+      "id": "1084fc41-4c97-458c-a6c9-2140fb99d5a9",
+      "properties": {
+        "page_content": "Appendices A. Overview Of Hyper-Parameters\n\nWe present the overview of hyper-parameters for DeepSeekMoE across various sizes in Table 7.\n\n# Params # Layers Hidden # Attn # Shared # Routed Relative Sequence Batch Size Learning Size Heads Experts Experts Expert Size Length (Sequence) Rate 2.0B 9 1280 10 1 63 (7 activated) 0.25 2048 2048 1.08e-3 16.4B 28 2048 16 2 64 (6 activated) 0.25 4096 4608 4.2e-4 144.6B 62 4096 32 4 128 (12 activated) 0.125 4096 4608 3.0e-4\n\nTable 7 | Overview of hyper-parameters for DeepSeekMoE across various sizes. The relative expert size is in comparison to a standard FFN.\n\nB. Comparing Deepseekmoe With Larger Models\n\nComparisons among DeepSeekMoE, GShard×1.2, and GShard×1.5 are shown in Table 8. Comparisons among DeepSeekMoE, Dense×4, and Dense×16 are shown in Table 9.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 2.3B 2.8B 1.9B # Activated Expert Params N/A 0.28B 0.35B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.824 1.808 1.808 HellaSwag (Acc.) 0-shot 53.7 54.4 54.8 PIQA (Acc.) 0-shot 71.8 71.1 72.3 ARC-easy (Acc.) 0-shot 46.8 47.3 49.4 ARC-challenge (Acc.) 0-shot 31.7 34.1 34.3 RACE-middle (Acc.) 5-shot 43.7 46.4 44.0 RACE-high (Acc.) 5-shot 31.9 32.4 31.7 HumanEval (Pass@1) 0-shot 3.7 3.0 4.9 MBPP (Pass@1) 3-shot 2.4 2.6 2.2 TriviaQA (EM) 5-shot 15.2 15.7 16.6 NaturalQuestions (EM) 5-shot 4.5 4.7 5.7\n\nTable 8 | Comparison between DeepSeekMoE and larger GShard models.\n\nAt a larger scale of 13B total parameters, we also compare DeepSeekMoE with GShard×1.2 and GShard×1.5, and show results in Table 10. At a larger scale, DeepSeekMoE even outperforms GShard×1.5 distinctly.\n\nMetric # Shot Dense×4 Dense×16 DeepSeekMoE Relative Expert Size N/A 1 1 0.25 # Experts N/A 4 + 0 16 + 0 1 + 63 # Activated Experts N/A 4 + 0 16 + 0 1 + 7 # Total Expert Params N/A 0.47B 1.89B 1.89B # Activated Expert Params N/A 0.47B 1.89B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.908 1.806 1.808 HellaSwag (Acc.) 0-shot 47.6 55.1 54.8 PIQA (Acc.) 0-shot 70.0 71.9 72.3 ARC-easy (Acc.) 0-shot 43.9 51.9 49.4 ARC-challenge (Acc.) 0-shot 30.5 33.8 34.3 RACE-middle (Acc.) 5-shot 42.4 46.3 44.0 RACE-high (Acc.) 5-shot 30.7 33.0 31.7 HumanEval (Pass@1) 0-shot 1.8 4.3 4.9 MBPP (Pass@1) 3-shot 0.2 2.2 2.2 TriviaQA (EM) 5-shot 9.9 16.5 16.6 NaturalQuestions (EM) 5-shot 3.0 6.3 5.7\n\nTable 9 | Comparison between DeepSeekMoE and larger dense baselines.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 15.9B 19.8B 13.3B # Activated Expert Params N/A 2.37B 2.82B 2.05B # Training Tokens N/A 100B 100B 100B HellaSwag (Acc.) 0-shot 66.6 67.7 69.1 PIQA (Acc.) 0-shot 75.6 76.0 75.7 ARC-easy (Acc.) 0-shot 56.8 56.8 58.8 ARC-challenge (Acc.) 0-shot 39.9 37.6 38.5 RACE-middle (Acc.) 5-shot 51.6 50.6 52.4 RACE-high (Acc.) 5-shot 37.4 36.3 38.5 HumanEval (Pass@1) 0-shot 6.1 6.1 9.8 MBPP (Pass@1) 3-shot 7.0 11.6 10.6 TriviaQA (EM) 5-shot 36.5 36.7 38.2 NaturalQuestions (EM) 5-shot 12.6 12.1 13.7\n\nTable 10 | Comparison between DeepSeekMoE and larger GShard models at a larger scale.\n\nC. Training Benchmark Curves Of Deepseekmoe 16B\n\nWe present the benchmark curves during training of DeepSeekMoE 16B and DeepSeek 7B (Dense) in Figure 7 for reference.",
+        "keyphrases": [
+          "DeepSeekMoE",
+          "hyper-parameters",
+          "GShard models",
+          "Dense baselines",
+          "training benchmark curves"
+        ],
+        "embedding": [
+          0.020979074761271477,
+          -0.0013986049452796578,
+          -0.019327709451317787,
+          0.0138933714479208,
+          0.0006998290773481131,
+          -0.02645554021000862,
+          0.0001438888575648889,
+          0.05624751001596451,
+          -0.006693925708532333,
+          0.024147000163793564,
+          -0.016437821090221405,
+          -0.0021316087804734707,
+          0.04061009734869003,
+          0.005270044784992933,
+          0.00756173487752676,
+          0.03029749169945717,
+          0.026859955862164497,
+          0.02480417490005493,
+          0.0019567832350730896,
+          -0.041183020919561386,
+          0.017339332029223442,
+          -0.04290178790688515,
+          -0.025309694930911064,
+          0.008383204229176044,
+          -0.016185062006115913,
+          0.009588027372956276,
+          0.022377679124474525,
+          0.014769605360925198,
+          -0.037071458995342255,
+          0.022091219201683998,
+          0.01640412025153637,
+          0.023371869698166847,
+          0.0032205828465521336,
+          -0.005885093938559294,
+          -0.02074316516518593,
+          -0.004515977576375008,
+          0.06709934026002884,
+          0.027870995923876762,
+          -0.010388433001935482,
+          0.018485177308321,
+          0.029404405504465103,
+          0.011500577442348003,
+          -0.013918647542595863,
+          -0.00502149760723114,
+          0.005880881100893021,
+          -0.009082507342100143,
+          -0.018165014684200287,
+          0.014019750989973545,
+          0.011744911782443523,
+          -0.05675303190946579,
+          -0.0348808728158474,
+          -0.017726896330714226,
+          -0.008109381422400475,
+          -0.013952348381280899,
+          -0.0411493182182312,
+          -0.03166239336133003,
+          0.028376515954732895,
+          0.061841931194067,
+          0.04182334616780281,
+          0.008939276449382305,
+          0.02399534359574318,
+          -0.006129428278654814,
+          -0.03993606939911842,
+          0.0007677582907490432,
+          0.008855023421347141,
+          -0.015452057123184204,
+          0.022579887881875038,
+          -0.005105750635266304,
+          -0.006647586356848478,
+          -0.020608359947800636,
+          -0.037745483219623566,
+          0.022377679124474525,
+          -0.003243752522394061,
+          0.006015686318278313,
+          0.02106332778930664,
+          -0.01328674703836441,
+          0.027803592383861542,
+          0.02632073499262333,
+          0.00384195102378726,
+          -0.013918647542595863,
+          0.017726896330714226,
+          0.0043074507266283035,
+          -0.01196397002786398,
+          0.03747587278485298,
+          0.04842880368232727,
+          -0.026287034153938293,
+          -0.040374185889959335,
+          -0.06318998336791992,
+          0.010321030393242836,
+          -0.008244186639785767,
+          -0.018384072929620743,
+          -0.008678090758621693,
+          0.03508307784795761,
+          -0.02180475741624832,
+          -0.03354967013001442,
+          0.06864959746599197,
+          -0.012494766153395176,
+          0.007115192245692015,
+          -0.03442590311169624,
+          -0.0069087715819478035,
+          0.004235835280269384,
+          0.00045523117296397686,
+          0.008661240339279175,
+          0.0043685343116521835,
+          0.005177366081625223,
+          -0.00362710515037179,
+          -0.01878848858177662,
+          0.0002004965499509126,
+          -0.004111561458557844,
+          0.0718512237071991,
+          0.012410513125360012,
+          -0.01227570790797472,
+          -0.048327699303627014,
+          -0.013927072286605835,
+          -0.02032189816236496,
+          -0.038655418902635574,
+          -0.029775120317935944,
+          -0.021568847820162773,
+          0.024281805381178856,
+          -0.002023132750764489,
+          0.033263206481933594,
+          0.020642060786485672,
+          0.030044730752706528,
+          -0.01792910508811474,
+          -0.043070293962955475,
+          0.04421613737940788,
+          0.0006208415725268424,
+          -0.0033933022059500217,
+          -0.0019631022587418556,
+          -0.029842523857951164,
+          0.02118128351867199,
+          -0.0036713380832225084,
+          -0.03408889099955559,
+          -0.030954666435718536,
+          0.00232539139688015,
+          -0.008694942109286785,
+          0.05213594809174538,
+          0.03949795290827751,
+          -0.01417140755802393,
+          -0.0003188461414538324,
+          0.0034438541624695063,
+          -0.04034048691391945,
+          0.029337003827095032,
+          -0.017246652394533157,
+          -0.03181404992938042,
+          0.022327126935124397,
+          0.017895404249429703,
+          0.06558278203010559,
+          -0.0028098479378968477,
+          0.041351526975631714,
+          0.0015386760933324695,
+          0.04003717377781868,
+          0.02350667491555214,
+          0.015763795003294945,
+          -0.009074081666767597,
+          -0.004326407797634602,
+          -0.03693665191531181,
+          0.00613785395398736,
+          -0.03360022231936455,
+          -0.015098193660378456,
+          -0.001018938492052257,
+          -0.004907755646854639,
+          -0.017288779839873314,
+          -0.0007556469063274562,
+          -0.002022079424932599,
+          0.047788478434085846,
+          0.023355018347501755,
+          -0.013244620524346828,
+          -0.06804297864437103,
+          0.03263973444700241,
+          0.04620451480150223,
+          0.007477481383830309,
+          -0.006196831353008747,
+          0.03788029029965401,
+          0.003601829055696726,
+          0.02229342609643936,
+          0.020709464326500893,
+          0.01904124952852726,
+          -0.06531316787004471,
+          0.04522717744112015,
+          -0.015300401486456394,
+          0.003485980909317732,
+          0.01658105105161667,
+          0.011264667846262455,
+          0.0310389194637537,
+          -0.0018630513222888112,
+          0.029033690690994263,
+          0.026859955862164497,
+          -0.012604295276105404,
+          0.02136664092540741,
+          0.031932003796100616,
+          0.008248399011790752,
+          0.04553049057722092,
+          -0.001733511919155717,
+          0.0005845073610544205,
+          -0.018687384203076363,
+          0.008838172070682049,
+          0.03353281691670418,
+          -0.017423585057258606,
+          0.018215566873550415,
+          0.0006161023629829288,
+          -0.027685638517141342,
+          -0.0010357891442254186,
+          -0.013337299227714539,
+          0.027651937678456306,
+          0.0262701828032732,
+          -0.008509584702551365,
+          0.0066855004988610744,
+          0.008404267951846123,
+          -0.016067106276750565,
+          -0.031190576031804085,
+          -0.013935497961938381,
+          -0.017069721594452858,
+          -0.0532480925321579,
+          -0.0025634069461375475,
+          0.03521788492798805,
+          -0.007785005960613489,
+          0.022512484341859818,
+          0.010860252194106579,
+          -0.0323195718228817,
+          0.003799824509769678,
+          0.009933465160429478,
+          0.014179832302033901,
+          0.006390613969415426,
+          -0.017229802906513214,
+          -0.02350667491555214,
+          -0.0032016257755458355,
+          0.008753919042646885,
+          0.032841939479112625,
+          -0.010919229127466679,
+          0.001601866097189486,
+          -0.001936773071065545,
+          -0.01959731988608837,
+          0.0043348330073058605,
+          0.03257232904434204,
+          -0.01013567391782999,
+          -0.00016798004799056798,
+          0.007005663122981787,
+          0.02662404626607895,
+          0.018485177308321,
+          0.01279807835817337,
+          -0.015266700647771358,
+          -0.04192444682121277,
+          -0.016564201563596725,
+          0.01824926771223545,
+          -0.00505098607391119,
+          -0.026742001995444298,
+          -0.01732248067855835,
+          0.057629264891147614,
+          0.024197550490498543,
+          0.023304466158151627,
+          -0.06281927227973938,
+          -0.02283264696598053,
+          0.005535442847758532,
+          -0.01727192848920822,
+          0.05668563023209572,
+          0.059415437281131744,
+          0.005628121085464954,
+          -0.00452861562371254,
+          -0.04121671989560127,
+          0.04023938253521919,
+          -0.01184601616114378,
+          -0.049979064613580704,
+          0.018704235553741455,
+          -0.01627773977816105,
+          0.01627773977816105,
+          0.033633921295404434,
+          0.027685638517141342,
+          0.0044443621300160885,
+          0.008231548592448235,
+          0.04266587644815445,
+          -0.023152809590101242,
+          -0.0458001010119915,
+          -0.02374258264899254,
+          0.021349789574742317,
+          -0.018721086904406548,
+          -0.007696540094912052,
+          -0.02852817252278328,
+          0.02633758634328842,
+          0.05055198818445206,
+          0.02662404626607895,
+          0.018350372090935707,
+          0.016142934560775757,
+          0.025056935846805573,
+          -0.024096447974443436,
+          0.0011490045581012964,
+          -0.023675180971622467,
+          -0.043306201696395874,
+          -0.017710046842694283,
+          -0.015199298039078712,
+          0.0068076676689088345,
+          0.006015686318278313,
+          -0.022849498316645622,
+          0.005493315868079662,
+          0.02606797404587269,
+          0.014407316222786903,
+          -0.03140963613986969,
+          0.011702785268425941,
+          0.05119231343269348,
+          -0.035116780549287796,
+          -0.03194885700941086,
+          0.0317297987639904,
+          0.017844852060079575,
+          -0.0067992424592375755,
+          -0.01885589212179184,
+          0.01466007623821497,
+          -9.174066690320615e-06,
+          0.01113828830420971,
+          -0.009284715168178082,
+          0.0007667051395401359,
+          -0.011938693933188915,
+          -0.0016303015872836113,
+          0.007801856845617294,
+          -0.03386983275413513,
+          -0.008905574679374695,
+          0.019176054745912552,
+          0.022124920040369034,
+          0.034510158002376556,
+          -0.06474024802446365,
+          -0.01166065875440836,
+          0.06969434022903442,
+          0.01836722157895565,
+          -0.04610341414809227,
+          0.025478200986981392,
+          0.019799528643488884,
+          -0.007351101376116276,
+          0.004520190414041281,
+          0.04559789225459099,
+          -0.04684484004974365,
+          -0.014002900570631027,
+          0.029050542041659355,
+          -0.04040788859128952,
+          -0.03139278292655945,
+          0.01892329379916191,
+          0.025023233145475388,
+          -0.06376291066408157,
+          -0.03761067986488342,
+          0.005404850002378225,
+          0.012258856557309628,
+          -0.007397440727800131,
+          -0.021956413984298706,
+          0.0029320153407752514,
+          -0.036094117909669876,
+          -0.01426408626139164,
+          0.0003749272436834872,
+          -0.005135239567607641,
+          -0.028545022010803223,
+          0.002346454653888941,
+          0.020793717354536057,
+          0.01627773977816105,
+          0.00159554707352072,
+          0.012191453948616982,
+          0.03424054756760597,
+          -0.006175767630338669,
+          0.0027550833765417337,
+          -0.028443917632102966,
+          -0.02645554021000862,
+          0.01579749584197998,
+          0.0032079447992146015,
+          0.00201260088942945,
+          -0.017162399366497993,
+          -0.04377802088856697,
+          -0.03366762399673462,
+          0.028545022010803223,
+          -0.020557807758450508,
+          -0.013623761013150215,
+          0.021821608766913414,
+          -0.004482276272028685,
+          -0.011862866580486298,
+          0.012292558327317238,
+          0.002847762079909444,
+          0.030819861218333244,
+          -0.03619522228837013,
+          0.006053600460290909,
+          0.017844852060079575,
+          -0.001491283648647368,
+          0.035116780549287796,
+          -0.010422134771943092,
+          -0.02392794005572796,
+          0.021838458254933357,
+          -0.049069128930568695,
+          0.024871578440070152,
+          0.03848691284656525,
+          0.012831779196858406,
+          0.03066820651292801,
+          0.031055770814418793,
+          -0.009326841682195663,
+          0.011610106565058231,
+          -0.012385237030684948,
+          -0.0024496649857610464,
+          0.033768728375434875,
+          -0.059482838958501816,
+          -0.023034855723381042,
+          0.010801274329423904,
+          0.018872741609811783,
+          0.03943055123090744,
+          0.017221376299858093,
+          -0.010834976099431515,
+          -0.022546187043190002,
+          -0.008138869889080524,
+          0.013640611432492733,
+          0.04927133768796921,
+          -0.0011468982556834817,
+          -0.003104734467342496,
+          -0.005493315868079662,
+          0.026371287181973457,
+          0.006803454831242561,
+          0.012233581393957138,
+          0.023708881810307503,
+          0.0004207399906590581,
+          0.022074367851018906,
+          -0.0037366345059126616,
+          -0.0342910997569561,
+          -0.008581199683248997,
+          0.007616499438881874,
+          0.03515048325061798,
+          -0.0021610974799841642,
+          0.021097028627991676,
+          -0.02473677322268486,
+          0.01978267729282379,
+          -0.039767563343048096,
+          0.020490406081080437,
+          0.031679246574640274,
+          -0.0028561872895807028,
+          -0.0045328279957175255,
+          0.01227570790797472,
+          -0.015157170593738556,
+          0.0019136033952236176,
+          0.029185347259044647,
+          -0.03471236303448677,
+          0.014778031036257744,
+          0.012115626595914364,
+          0.02945495769381523,
+          -0.04320509731769562,
+          -0.022984303534030914,
+          -0.013775416649878025,
+          -0.011685934849083424,
+          0.04320509731769562,
+          -0.04023938253521919,
+          -0.02259673923254013,
+          -0.0018862209981307387,
+          0.0372062623500824,
+          0.019850080832839012,
+          -0.013177217915654182,
+          -0.04327250272035599,
+          -0.033515967428684235,
+          -0.05008016899228096,
+          -0.012284132651984692,
+          -0.03922834247350693,
+          0.002550769131630659,
+          -0.07731083780527115,
+          0.01941196247935295,
+          -0.050113871693611145,
+          0.020642060786485672,
+          0.046811141073703766,
+          -0.008572774939239025,
+          0.030836712568998337,
+          -0.0196478720754385,
+          0.014205108396708965,
+          -0.06251595914363861,
+          -0.0068076676689088345,
+          0.025747811421751976,
+          0.02945495769381523,
+          -0.003494406118988991,
+          -0.003734528087079525,
+          0.019445665180683136,
+          -0.01279807835817337,
+          -0.0065675457008183,
+          -0.008930850774049759,
+          0.02608482539653778,
+          0.008496946655213833,
+          -0.010674894787371159,
+          0.024601968005299568,
+          0.008265250362455845,
+          0.015502609312534332,
+          0.03761067986488342,
+          -0.011365772224962711,
+          -0.04023938253521919,
+          -0.0034417477436363697,
+          0.029117945581674576,
+          0.029707718640565872,
+          -0.031072622165083885,
+          0.014339913614094257,
+          -0.003869333304464817,
+          0.005906157195568085,
+          0.006512781139463186,
+          0.004033627454191446,
+          0.029758270829916,
+          -0.006394826341420412,
+          0.026708299294114113,
+          -0.014811732806265354,
+          0.019850080832839012,
+          -0.026421839371323586,
+          0.03147703781723976,
+          0.012132477015256882,
+          -0.030162686482071877,
+          0.005910370033234358,
+          -0.01628616452217102,
+          0.016918065026402473,
+          0.019361412152647972,
+          -0.013143517076969147,
+          -0.027500281110405922,
+          0.013446828350424767,
+          0.013480530120432377,
+          0.012225155718624592,
+          0.004494914319366217,
+          0.027399176731705666,
+          0.03006158210337162,
+          -0.02227657660841942,
+          -0.03545379266142845,
+          0.0174404364079237,
+          -0.021872159093618393,
+          0.015039216727018356,
+          -0.03427424654364586,
+          0.01426408626139164,
+          0.00647486699745059,
+          -0.03595931455492973,
+          0.04010457545518875,
+          0.020642060786485672,
+          0.0019451983971521258,
+          0.004128412343561649,
+          -0.005990410689264536,
+          -0.009326841682195663,
+          0.05129341781139374,
+          -0.015957577154040337,
+          0.019816379994153976,
+          0.015637414529919624,
+          0.012882331386208534,
+          0.03683554753661156,
+          -0.03322950750589371,
+          -0.014575823210179806,
+          0.01347210444509983,
+          0.014499994926154613,
+          -0.03445960581302643,
+          -0.01365746185183525,
+          -0.02148459479212761,
+          -0.011450025252997875,
+          0.040374185889959335,
+          -0.014525271020829678,
+          -0.02131608873605728,
+          0.007658625952899456,
+          0.0023359230253845453,
+          -0.016934916377067566,
+          0.008092531003057957,
+          -0.015047641471028328,
+          -0.005194216500967741,
+          0.04064379632472992,
+          -0.023978492245078087,
+          0.03622892498970032,
+          0.031561292707920074,
+          0.016859088093042374,
+          -0.042699579149484634,
+          0.0006656011682935059,
+          0.006112577859312296,
+          -0.014188257977366447,
+          -0.01597442850470543,
+          0.034442752599716187,
+          0.014887560158967972,
+          0.01450842060148716,
+          -0.023152809590101242,
+          -0.006205256562680006,
+          0.02362462878227234,
+          -0.023304466158151627,
+          0.02578151412308216,
+          -0.00396411819383502,
+          -0.02332131750881672,
+          -0.03213421255350113,
+          0.007650200743228197,
+          -0.022680992260575294,
+          0.012755951844155788,
+          -0.003985181916505098,
+          0.012292558327317238,
+          0.01236838661134243,
+          -0.025259142741560936,
+          0.0036587000358849764,
+          0.019209755584597588,
+          0.000130460990476422,
+          -0.0003262182872276753,
+          -0.009048805572092533,
+          0.004604443442076445,
+          0.018198715522885323,
+          -0.001285916194319725,
+          0.005215280223637819,
+          0.022006966173648834,
+          0.031005218625068665,
+          0.004381172358989716,
+          -0.003250071546062827,
+          -0.020793717354536057,
+          0.02797210030257702,
+          -0.02357407659292221,
+          0.02773619070649147,
+          0.028174307197332382,
+          0.014921261928975582,
+          -0.03855431452393532,
+          -0.0021800545509904623,
+          -0.016496798023581505,
+          -0.008353715762495995,
+          0.01776059903204441,
+          0.022883199155330658,
+          -0.024281805381178856,
+          -0.0024075384717434645,
+          -0.004147369414567947,
+          0.0215014461427927,
+          0.003218476427718997,
+          -0.030769309028983116,
+          -0.010843401774764061,
+          0.009866062551736832,
+          0.00896455254405737,
+          -0.01456739753484726,
+          0.0019831124227494,
+          0.011888142675161362,
+          0.006150492001324892,
+          -0.0016060788184404373,
+          -0.00011077993985963985,
+          0.01897384598851204,
+          -0.002618171740323305,
+          0.01567954197525978,
+          -0.013750140555202961,
+          -0.013446828350424767,
+          -0.011087736114859581,
+          0.05247296392917633,
+          -0.0032205828465521336,
+          0.03899243474006653,
+          0.014188257977366447,
+          0.009722832590341568,
+          0.04431724175810814,
+          -0.013253046199679375,
+          -0.009309991262853146,
+          0.01721295155584812,
+          -0.01750783808529377,
+          -0.03568970412015915,
+          0.004103136248886585,
+          -0.005025709979236126,
+          -0.028983140364289284,
+          0.01071702130138874,
+          0.005038348026573658,
+          -0.004067328758537769,
+          0.0009341586264781654,
+          0.008846597746014595,
+          -0.026843104511499405,
+          0.03247122839093208,
+          0.005944071337580681,
+          0.011879716999828815,
+          0.034746065735816956,
+          -0.016201911494135857,
+          -0.007974576205015182,
+          0.013792267069220543,
+          -0.01518244668841362,
+          0.028545022010803223,
+          -0.01347210444509983,
+          0.01047268696129322,
+          0.01707814633846283,
+          0.030381744727492332,
+          0.03622892498970032,
+          -0.02492213062942028,
+          -0.06261706352233887,
+          0.011711210012435913,
+          -0.004638144746422768,
+          0.011559554375708103,
+          0.016260888427495956,
+          0.01233468484133482,
+          -0.005640759132802486,
+          0.01811446249485016,
+          -0.007540671620517969,
+          0.028999989852309227,
+          -0.006121003068983555,
+          -0.0326734334230423,
+          0.020962223410606384,
+          0.006007261108607054,
+          0.023034855723381042,
+          0.014339913614094257,
+          -0.03895873203873634,
+          -0.009596452116966248,
+          -0.004596018232405186,
+          0.010270479135215282,
+          -0.03383613005280495,
+          0.00483614020049572,
+          0.010759147815406322,
+          0.017912253737449646,
+          0.002041036495938897,
+          -0.007595436181873083,
+          0.010556939989328384,
+          -0.005227918270975351,
+          -0.016303015872836113,
+          0.0131350914016366,
+          -0.0022664139978587627,
+          -0.013059263117611408,
+          0.026303883641958237,
+          -0.03545379266142845,
+          -0.016993893310427666,
+          -4.235703818267211e-05,
+          0.0013027668464928865,
+          0.013244620524346828,
+          0.003953586798161268,
+          0.0015492078382521868,
+          -0.02896628901362419,
+          -0.03848691284656525,
+          0.04475535824894905,
+          0.022967452183365822,
+          -0.0033153677359223366,
+          -0.0003277980431448668,
+          -0.0004360108869150281,
+          -0.005792415235191584,
+          0.023894239217042923,
+          0.03164554387331009,
+          -0.020658912137150764,
+          0.005004646722227335,
+          -0.027466580271720886,
+          -0.019698424264788628,
+          0.015283551067113876,
+          0.009031955152750015,
+          0.033701326698064804,
+          0.006217894610017538,
+          0.006912984419614077,
+          0.018384072929620743,
+          0.008610688149929047,
+          -0.004507552366703749,
+          -0.003808249719440937,
+          -0.022697841748595238,
+          0.03230271860957146,
+          0.0390598364174366,
+          0.0149381123483181,
+          0.014070303179323673,
+          -0.034746065735816956,
+          -0.003814568743109703,
+          0.010885528288781643,
+          0.02492213062942028,
+          0.02424810267984867,
+          0.006748690269887447,
+          0.001917816000059247,
+          0.015157170593738556,
+          -0.03703775629401207,
+          -0.013573208823800087,
+          0.009798659943044186,
+          -0.014129280112683773,
+          -0.027988949790596962,
+          -0.006087301764637232,
+          0.022916901856660843,
+          -0.005897731985896826,
+          0.0021526720374822617,
+          -0.004659208003431559,
+          -0.011542703956365585,
+          -0.015527885407209396,
+          -0.004541253671050072,
+          0.006693925708532333,
+          0.02380998618900776,
+          0.0044317240826785564,
+          0.024770474061369896,
+          -0.009343692101538181,
+          0.011087736114859581,
+          0.006546482443809509,
+          -0.014162981882691383,
+          0.016488373279571533,
+          -0.013059263117611408,
+          -0.026708299294114113,
+          -0.0092425886541605,
+          0.01781114935874939,
+          -0.00644537853077054,
+          -0.0016945448005571961,
+          -0.01647152192890644,
+          0.005004646722227335,
+          0.036397431045770645,
+          0.008265250362455845,
+          0.017558390274643898,
+          0.023523524403572083,
+          -0.002397006843239069,
+          -0.009318416006863117,
+          0.0131350914016366,
+          0.0055902074091136456,
+          0.0055859945714473724,
+          0.021585699170827866,
+          -0.001512346905656159,
+          0.04387912526726723,
+          -0.05058568716049194,
+          0.008383204229176044,
+          0.027803592383861542,
+          -0.0057081617414951324,
+          -0.005939858499914408,
+          -0.032841939479112625,
+          0.01653892546892166,
+          -0.01678325980901718,
+          0.009798659943044186,
+          -0.011155138723552227,
+          0.028427068144083023,
+          -0.03723996505141258,
+          -0.008378991857171059,
+          -3.804563675657846e-05,
+          0.00811359379440546,
+          0.016083957627415657,
+          0.018653683364391327,
+          -0.006580183748155832,
+          -0.02497268095612526,
+          -0.003673444502055645,
+          0.012882331386208534,
+          0.04290178790688515,
+          -0.003953586798161268,
+          -0.014685352332890034,
+          0.0215014461427927,
+          -0.011171989142894745,
+          -0.020119691267609596,
+          0.0027635085862129927,
+          -0.024821026250720024,
+          0.029471809044480324,
+          0.03612782061100006,
+          -0.041958149522542953,
+          -0.024450311437249184,
+          0.002029451774433255,
+          -0.010936079546809196,
+          0.002504429779946804,
+          0.021012775599956512,
+          0.02743287943303585,
+          -0.004170538857579231,
+          -0.007629137486219406,
+          0.0206926129758358,
+          -0.005316384136676788,
+          0.015949152410030365,
+          0.016665305942296982,
+          -0.011348920874297619,
+          -0.0069551109336316586,
+          -0.017844852060079575,
+          -0.03270713612437248,
+          -0.00037361079012043774,
+          -0.0186031311750412,
+          -0.03481346741318703,
+          -0.0346449613571167,
+          -0.01398605015128851,
+          -0.010278903879225254,
+          -0.003949373960494995,
+          0.0364985354244709,
+          0.02394479140639305,
+          0.0251074880361557,
+          0.015746943652629852,
+          -0.013278322294354439,
+          0.05405692383646965,
+          -0.006816092878580093,
+          0.0460360087454319,
+          -0.020187092944979668,
+          0.0008951914496719837,
+          -0.019816379994153976,
+          0.03226901963353157,
+          -0.014390465803444386,
+          0.011525853537023067,
+          -0.009899764321744442,
+          0.006533844396471977,
+          0.0009504826739430428,
+          0.007574372924864292,
+          -0.0019451983971521258,
+          -0.009419520385563374,
+          0.027079014107584953,
+          -0.0023274978157132864,
+          0.029471809044480324,
+          0.020406153053045273,
+          0.004600231070071459,
+          0.005076262168586254,
+          -0.029707718640565872,
+          -0.00035149429459124804,
+          0.005067836958914995,
+          -0.0066855004988610744,
+          -0.018148163333535194,
+          0.0024201765190809965,
+          -0.007001450285315514,
+          0.007999852299690247,
+          -0.0039051410276442766,
+          -0.009225737303495407,
+          0.014104004949331284,
+          0.025629857555031776,
+          0.0069845994003117085,
+          -0.04391282796859741,
+          0.0035702341701835394,
+          -0.03936314582824707,
+          0.002685574349015951,
+          0.010818125680088997,
+          0.010093546472489834,
+          -0.002889888593927026,
+          0.0057123745791614056,
+          0.009613302536308765,
+          -0.004202133975923061,
+          -0.020962223410606384,
+          0.002352773677557707,
+          -0.015047641471028328,
+          -0.006146279163658619,
+          -0.011281518265604973,
+          3.797981480602175e-05,
+          -0.003614467103034258,
+          -0.0019262413261458278,
+          -0.005986197851598263,
+          -0.0067318398505449295,
+          -0.027146417647600174,
+          -0.02822485938668251,
+          -0.006108365021646023,
+          0.0002985726750921458,
+          -0.00706885289400816,
+          -0.03024693951010704,
+          0.03134223073720932,
+          -0.008539073169231415,
+          0.007136255502700806,
+          -0.0093015655875206,
+          0.009882913902401924,
+          -0.002344348467886448,
+          -0.005960921756923199,
+          -0.004553891718387604,
+          -0.026421839371323586,
+          -0.013994474895298481,
+          -0.03895873203873634,
+          0.016741132363677025,
+          -0.008471670560538769,
+          0.007106767036020756,
+          -0.010506387799978256,
+          0.001101612113416195,
+          -0.018148163333535194,
+          -0.0009278396028093994,
+          -0.002559194341301918,
+          0.027651937678456306,
+          -0.024163849651813507,
+          0.009074081666767597,
+          0.01616821065545082,
+          0.015485758893191814,
+          -0.0346449613571167,
+          0.02008599042892456,
+          0.012191453948616982,
+          -0.019243456423282623,
+          -0.0025423436891287565,
+          0.018350372090935707,
+          0.02576466277241707,
+          -0.013901796191930771,
+          -0.014162981882691383,
+          -0.004326407797634602,
+          0.009630153886973858,
+          -0.025006383657455444,
+          0.00954589992761612,
+          -0.010203076526522636,
+          0.017356181517243385,
+          0.025174889713525772,
+          -0.011256242170929909,
+          0.04023938253521919,
+          -0.03501567617058754,
+          -0.012831779196858406,
+          0.013497380539774895,
+          -0.023675180971622467,
+          0.013177217915654182,
+          -0.0003612360742408782,
+          -0.016690582036972046,
+          0.021754205226898193,
+          -0.006428527645766735,
+          0.01841777376830578,
+          -0.03693665191531181,
+          0.030078433454036713,
+          0.03012898378074169,
+          0.006656011566519737,
+          -0.011079310439527035,
+          -0.019513066858053207,
+          -0.01056536566466093,
+          0.0057418630458414555,
+          -0.0015397293027490377,
+          0.025730961933732033,
+          -0.014331488870084286,
+          -0.02466936968266964,
+          0.004633932374417782,
+          -0.0030478634871542454,
+          0.004692909307777882,
+          -0.02643868885934353,
+          0.004105242434889078,
+          -0.009832361713051796,
+          -0.0017924892017617822,
+          0.0006161023629829288,
+          -0.005990410689264536,
+          -0.009200462140142918,
+          -0.028696678578853607,
+          -0.056416019797325134,
+          0.0014891773462295532,
+          -0.0037913990672677755,
+          -0.023725733160972595,
+          -0.017743747681379318,
+          -0.012402087450027466,
+          -0.014449442736804485,
+          0.018518878147006035,
+          -0.0017208738718181849,
+          -0.023793134838342667,
+          -0.011862866580486298,
+          0.002586576621979475,
+          0.0017630006186664104,
+          0.0024412397760897875,
+          0.02362462878227234,
+          0.011163563467562199,
+          0.031493887305259705,
+          -0.018097611144185066,
+          0.02694420889019966,
+          -0.017238227650523186,
+          -0.03831840679049492,
+          0.03980126604437828,
+          -0.0009425839525647461,
+          -0.022697841748595238,
+          0.010801274329423904,
+          0.0009594346047379076,
+          -0.006483292207121849,
+          0.044115033000707626,
+          -0.0205746591091156,
+          -0.013429977931082249,
+          0.010177800431847572,
+          -0.006441165693104267,
+          -0.016631603240966797,
+          -0.03299359604716301,
+          -0.015393080189824104,
+          0.03240382298827171,
+          -0.006373763084411621,
+          0.013783842325210571,
+          0.035487495362758636,
+          -0.023776285350322723,
+          -0.029353853315114975,
+          0.012073500081896782,
+          -0.020675763487815857,
+          -0.01615135930478573,
+          0.0027782530523836613,
+          0.018636832013726234,
+          0.012098775245249271,
+          -0.0012248324928805232,
+          0.0006171555141918361,
+          -0.004945669323205948,
+          -0.012250431813299656,
+          0.003391195787116885,
+          -0.023776285350322723,
+          -0.035656001418828964,
+          -0.024012194946408272,
+          -0.01162695698440075,
+          0.03280824050307274,
+          0.03457755967974663,
+          -0.00030146888457238674,
+          -0.022495634853839874,
+          -0.02204066701233387,
+          -0.0029256963171064854,
+          0.039329446852207184,
+          0.013758566230535507,
+          0.012410513125360012,
+          -0.005333234556019306,
+          0.010396858677268028,
+          0.0005781883373856544,
+          0.006281084381043911,
+          -0.039093535393476486,
+          -0.008197847753763199,
+          0.019058099016547203,
+          0.001322777010500431,
+          -0.01629459112882614,
+          0.018771637231111526,
+          0.00517315324395895,
+          -0.013497380539774895,
+          -0.022259725257754326,
+          0.018518878147006035,
+          0.013674312271177769,
+          0.008004064671695232,
+          0.0044570001773536205,
+          -0.01873793639242649,
+          0.0005866136634722352,
+          0.009192036464810371,
+          -0.01285705529153347,
+          -0.0011142500443384051,
+          0.021400341764092445,
+          0.012739100493490696,
+          -0.008383204229176044,
+          0.012452639639377594,
+          0.013969198800623417,
+          -0.007115192245692015,
+          -0.008172571659088135,
+          8.287103264592588e-05,
+          0.014778031036257744,
+          -0.007148893550038338,
+          -2.983422928082291e-05,
+          -0.011525853537023067,
+          0.010700170882046223,
+          0.004709760192781687,
+          0.011778613552451134,
+          -0.02443346008658409,
+          -0.006592821795493364,
+          0.0055902074091136456,
+          0.02160254865884781,
+          -0.0079324496909976,
+          -0.013708014041185379,
+          0.009908189997076988,
+          -0.024450311437249184,
+          -0.02185530960559845,
+          0.00659703416749835,
+          0.0009615409071557224,
+          -0.009402669966220856,
+          0.033751875162124634,
+          0.0323195718228817,
+          -0.0010084068635478616,
+          -0.0011511108605191112,
+          0.00995031651109457,
+          0.008180996403098106,
+          0.017608942463994026,
+          -0.022883199155330658,
+          -0.006003048736602068,
+          -0.0013817542931064963,
+          0.016067106276750565,
+          0.04856361076235771,
+          -0.005826116539537907,
+          -0.0022011178079992533,
+          0.015898600220680237,
+          0.0920720174908638,
+          -0.00039783361717127264,
+          0.0067866044119000435,
+          -0.02394479140639305,
+          -0.030769309028983116,
+          0.005017284769564867,
+          0.001989431446418166,
+          0.019058099016547203,
+          0.05786517634987831,
+          0.003043650882318616,
+          0.0014407316921278834,
+          -0.0022179684601724148,
+          0.003943054936826229,
+          -0.0085432855412364,
+          -0.022512484341859818,
+          0.015839623287320137,
+          -0.0067866044119000435,
+          0.010683320462703705,
+          -6.480264710262418e-05,
+          0.01659790240228176,
+          0.004101030062884092,
+          0.008193634450435638,
+          0.006146279163658619,
+          0.017406733706593513,
+          -0.021535146981477737,
+          0.026421839371323586,
+          -0.010219926945865154,
+          0.011525853537023067,
+          -0.014702202752232552,
+          -0.027517132461071014,
+          0.019799528643488884,
+          -4.670134876505472e-05,
+          -0.007490119431167841,
+          -0.004002032335847616,
+          -0.011348920874297619,
+          0.02606797404587269,
+          -0.021400341764092445,
+          0.022849498316645622,
+          0.01261272095143795,
+          -0.020119691267609596,
+          0.010497963055968285,
+          0.008585412986576557,
+          0.0010257840622216463,
+          0.015763795003294945,
+          -0.008821321651339531,
+          0.0161008071154356,
+          0.01242736354470253,
+          -0.027112716808915138,
+          0.009958741255104542,
+          0.014550547115504742,
+          -0.009461646899580956,
+          0.01546048279851675,
+          0.008753919042646885,
+          0.001712448545731604,
+          -0.0010089334100484848,
+          -0.01075072307139635,
+          -0.0043769595213234425,
+          -0.009748108685016632,
+          0.007641775533556938,
+          -0.005552293267101049,
+          0.00820627249777317,
+          0.016134509816765785,
+          0.04782218113541603,
+          0.008320014923810959,
+          0.003953586798161268,
+          3.060764720430598e-05,
+          0.00025091689894907176,
+          -0.015966001898050308,
+          -0.022192321717739105,
+          -0.013649036176502705,
+          -0.009832361713051796,
+          0.024821026250720024,
+          -0.005792415235191584,
+          0.006390613969415426,
+          -0.017179250717163086,
+          0.005463827401399612,
+          0.0017177143599838018,
+          0.010843401774764061,
+          0.0033722389489412308,
+          -0.042396266013383865,
+          -0.0045791673474013805,
+          0.014213534072041512,
+          -0.011104586534202099,
+          -0.001104771508835256,
+          -0.031611841171979904,
+          0.003911459818482399,
+          -0.0158648993819952,
+          -0.0044907014816999435,
+          0.016749558970332146,
+          -0.020153392106294632,
+          0.0006840315763838589,
+          0.0032816664315760136,
+          -0.015940727666020393,
+          0.005122601520270109,
+          -0.008737068623304367,
+          0.0016840130556374788,
+          0.018215566873550415,
+          0.002660298254340887,
+          -0.01298343576490879,
+          -0.002645554021000862,
+          -0.024568265303969383,
+          0.01750783808529377,
+          0.029758270829916,
+          -0.008008277043700218,
+          -0.01941196247935295,
+          -0.005177366081625223,
+          0.014247234910726547,
+          -0.004145262762904167,
+          -0.00814308226108551,
+          0.0031405421905219555,
+          0.003479661885648966,
+          -0.020052287727594376,
+          0.00356602156534791,
+          0.003753485158085823,
+          0.01585647277534008,
+          0.010489537380635738,
+          0.014769605360925198,
+          -0.029286451637744904,
+          -0.023472972214221954,
+          0.007102554198354483,
+          -0.019378261640667915,
+          0.005463827401399612,
+          0.03986866772174835,
+          -0.005404850002378225,
+          0.014348339289426804,
+          0.011011907830834389,
+          0.022006966173648834,
+          -0.023658329620957375,
+          0.017524689435958862,
+          0.0020642061717808247,
+          -0.017710046842694283,
+          -0.010321030393242836,
+          0.016673730686306953,
+          0.00954589992761612,
+          0.02766878716647625,
+          -0.019260307773947716,
+          -0.004570742137730122,
+          0.007696540094912052,
+          0.03398778662085533,
+          0.003955692984163761,
+          -0.014617949724197388,
+          0.027517132461071014,
+          0.047114454209804535,
+          0.008897149935364723,
+          0.01652207411825657,
+          -0.01678325980901718,
+          -0.005101538263261318,
+          0.007785005960613489,
+          0.009849212132394314,
+          0.013337299227714539,
+          -0.0019746869802474976,
+          -0.020962223410606384,
+          0.033027298748493195,
+          -0.009899764321744442,
+          0.002022079424932599,
+          -0.027904696762561798,
+          0.018569430336356163,
+          0.009689130820333958,
+          0.005093112587928772,
+          0.004452787339687347,
+          0.001162695698440075,
+          0.04340730607509613,
+          0.004701334983110428,
+          -0.029303302988409996,
+          0.012065074406564236,
+          0.002390687819570303,
+          0.020153392106294632,
+          -0.013615335337817669,
+          -0.013059263117611408,
+          0.004739248659461737,
+          -0.0018978058360517025,
+          -0.010877102613449097,
+          0.008998253382742405,
+          -0.0005845073610544205,
+          0.019243456423282623,
+          -0.009933465160429478,
+          -0.001282756682485342,
+          -0.0138933714479208,
+          0.0014723266940563917,
+          -0.016067106276750565,
+          0.025360247120261192,
+          -0.014407316222786903,
+          0.004398022778332233,
+          0.009630153886973858,
+          -0.00914990995079279,
+          -0.0034712364431470633,
+          0.007953512482345104,
+          0.009933465160429478,
+          -0.02369203232228756,
+          0.018822189420461655,
+          0.0015323570696637034,
+          0.016977041959762573,
+          0.003606041893362999,
+          -0.01224200613796711,
+          0.007047789636999369,
+          0.005362723488360643,
+          0.017153974622488022,
+          -0.00942794606089592,
+          0.005817691329866648,
+          -0.030904114246368408,
+          -0.02374258264899254,
+          0.038284704089164734,
+          0.015350953675806522,
+          -0.022630440071225166,
+          0.015839623287320137,
+          -0.02148459479212761,
+          -0.006656011566519737,
+          0.01285705529153347,
+          0.01365746185183525,
+          -0.03110632300376892,
+          0.00914990995079279,
+          -0.009040380828082561,
+          0.009166760370135307,
+          0.02241138182580471,
+          0.023119108751416206,
+          0.0038103561382740736,
+          -0.00668971287086606,
+          -0.021400341764092445,
+          -0.006091514602303505,
+          0.007907173596322536,
+          0.006314785685390234,
+          -0.007203658111393452,
+          0.02106332778930664,
+          0.01426408626139164,
+          -0.0055017415434122086,
+          -0.005396424792706966,
+          0.02413014881312847,
+          -0.03043229691684246,
+          0.019816379994153976,
+          0.0010763360187411308,
+          -0.03301044926047325,
+          0.01959731988608837,
+          -0.007325825747102499,
+          0.030196387320756912,
+          -0.0035428518895059824,
+          -0.008762344717979431,
+          0.0012722250539809465,
+          0.01162695698440075,
+          0.0028161669615656137,
+          -0.00014138758706394583,
+          0.007843983359634876,
+          0.020962223410606384,
+          -0.022512484341859818,
+          0.023220213130116463,
+          0.011921843513846397,
+          3.429373100516386e-05,
+          0.014676926657557487,
+          -0.023186512291431427,
+          -0.03011213429272175,
+          0.02743287943303585,
+          -0.011610106565058231,
+          -0.02123183384537697,
+          0.015359378419816494,
+          -0.013615335337817669,
+          -0.012814928777515888,
+          -0.02278209663927555,
+          -0.0092425886541605,
+          0.004014670383185148,
+          -0.006554907653480768,
+          -0.016252463683485985,
+          -0.0033427502494305372,
+          0.021518295630812645,
+          0.01806391030550003,
+          0.022310277447104454,
+          -0.00462129432708025,
+          0.012873905710875988,
+          0.001359637826681137,
+          0.019563619047403336,
+          0.019883781671524048,
+          -0.00939424429088831,
+          -0.0029467595741152763,
+          0.013488955795764923,
+          -0.007363739423453808,
+          0.022192321717739105,
+          0.0008404267719015479,
+          -0.02950550988316536,
+          -0.010969781316816807,
+          -0.03690294921398163,
+          0.007911385968327522,
+          -0.00792823638767004,
+          -0.006487505044788122,
+          0.05395582318305969,
+          0.006175767630338669,
+          -0.01248634047806263,
+          0.007704965304583311,
+          0.0054554021917283535,
+          -0.005341660231351852,
+          -0.00786925945430994,
+          0.017094997689127922,
+          -0.01380911748856306,
+          -0.015266700647771358,
+          -0.013118240982294083,
+          -0.010843401774764061,
+          0.009714406915009022,
+          0.020642060786485672,
+          -0.0033617070876061916,
+          -0.026169078424572945,
+          -0.013210919685661793,
+          -0.020305048674345016,
+          -0.013151941820979118,
+          0.007886109873652458,
+          -0.0042716427706182,
+          -0.0007840823964215815,
+          0.037071458995342255,
+          -0.019496217370033264,
+          -0.013404701836407185,
+          -0.010683320462703705,
+          -0.002651873044669628,
+          -0.01658947765827179,
+          -0.005939858499914408,
+          0.016201911494135857,
+          -0.016867512837052345,
+          -0.0034480667673051357,
+          -0.0056533971801400185,
+          -0.010245203040540218,
+          0.020187092944979668,
+          0.008522222749888897,
+          -0.00668971287086606,
+          0.01156798005104065,
+          0.04886692017316818,
+          -0.03798139467835426,
+          -0.01603340543806553,
+          -0.02515803836286068,
+          0.026489241048693657,
+          0.009259439073503017,
+          0.01670743152499199,
+          -0.02092852257192135,
+          -0.028494469821453094,
+          -0.023725733160972595,
+          -0.012292558327317238,
+          -0.019732125103473663,
+          -0.003188987961038947,
+          0.023102257400751114,
+          0.014803307130932808,
+          0.04283438250422478,
+          0.00532902218401432,
+          0.0035428518895059824,
+          -0.0028772505465894938,
+          0.0022474571596831083,
+          0.011197265237569809,
+          -0.02178790606558323,
+          0.0026834679301828146,
+          0.00024749411386437714,
+          0.026371287181973457,
+          0.01105403434485197,
+          -0.006024111993610859,
+          -0.028865184634923935,
+          0.0021610974799841642,
+          0.012418937869369984,
+          -0.007549096830189228,
+          -0.017069721594452858,
+          0.027837295085191727,
+          0.004549678880721331,
+          -0.0034017274156212807,
+          -0.011281518265604973,
+          0.028511321172118187,
+          0.0027550833765417337,
+          0.028123755007982254,
+          -0.020406153053045273,
+          -0.009040380828082561,
+          -0.022798946127295494,
+          0.009436370804905891,
+          -0.009107783436775208,
+          0.002430708147585392,
+          -0.01025362778455019,
+          0.015696391463279724,
+          -0.01508976798504591,
+          -0.010245203040540218,
+          0.030465997755527496,
+          -0.021198133006691933,
+          -0.0025739388074725866,
+          -0.015578437596559525,
+          -0.028022652491927147,
+          0.00967228040099144,
+          -0.0011005589039996266,
+          0.007852409034967422,
+          -0.006656011566519737,
+          0.003707145806401968,
+          -0.023034855723381042,
+          0.009166760370135307,
+          -0.00963857863098383,
+          -0.011222541332244873,
+          -0.001816712087020278,
+          0.013649036176502705,
+          0.0008383204694837332,
+          0.010337881743907928,
+          0.005034135654568672,
+          -0.03892502933740616,
+          0.01836722157895565,
+          0.0036776571068912745,
+          0.010278903879225254,
+          0.003707145806401968,
+          0.008155720308423042,
+          0.007924024015665054,
+          0.029337003827095032,
+          0.00982393603771925,
+          -0.009874488227069378,
+          -0.011854440905153751,
+          -0.0011216222774237394,
+          0.011163563467562199,
+          0.018653683364391327,
+          -0.004899329971522093,
+          -0.01616821065545082,
+          0.006399039179086685,
+          3.333929998916574e-05,
+          0.03491457179188728,
+          0.008509584702551365,
+          -0.020372450351715088,
+          -0.005337447393685579,
+          -0.04917023330926895,
+          0.020490406081080437,
+          -0.015376229770481586,
+          -0.0339372344315052,
+          0.01075072307139635,
+          -0.0043074507266283035,
+          -0.022428231313824654,
+          0.029353853315114975,
+          -0.004945669323205948,
+          -0.013219344429671764,
+          0.007001450285315514,
+          -0.009343692101538181,
+          -0.006879283115267754,
+          0.0014428379945456982,
+          -0.016623178496956825,
+          6.6851384872279596e-06,
+          0.00356602156534791,
+          0.019176054745912552,
+          -0.0034775554668158293,
+          -0.01549418456852436,
+          0.001500762184150517,
+          0.00623053265735507,
+          0.017861701548099518,
+          0.0021221302449703217,
+          0.003989394288510084,
+          0.0006424314924515784,
+          0.014828583225607872,
+          -0.02664089761674404,
+          0.005366935860365629,
+          -0.0012501084711402655,
+          -0.026792554184794426,
+          0.004553891718387604,
+          -0.006024111993610859,
+          -0.01983322948217392,
+          0.012494766153395176,
+          -0.0009241535444743931,
+          -0.0016408332157880068,
+          -0.00585981784388423,
+          -0.009478497318923473,
+          -0.009697556495666504,
+          -0.008972978219389915,
+          0.008956126868724823,
+          0.016016554087400436,
+          0.007304762024432421,
+          -0.010118822567164898,
+          -0.02431550621986389,
+          0.001289075706154108,
+          0.02529284358024597,
+          0.02357407659292221,
+          -0.035285286605358124,
+          0.014735904522240162,
+          0.005935645662248135,
+          -0.00851379707455635,
+          -0.027163267135620117,
+          0.0310389194637537,
+          0.02197326347231865,
+          0.0038061433006078005,
+          0.007397440727800131,
+          0.01162695698440075,
+          0.007359527051448822,
+          -0.03949795290827751,
+          0.011079310439527035,
+          -0.011020333506166935,
+          -0.009849212132394314,
+          -3.0377268558368087e-05,
+          0.009031955152750015,
+          -0.009882913902401924,
+          -0.011803888715803623,
+          0.007481694221496582,
+          0.02204066701233387,
+          -0.009368968196213245,
+          -0.003660806454718113,
+          0.01678325980901718,
+          0.002243244554847479,
+          0.001279597170650959,
+          0.016808535903692245,
+          0.05621381103992462,
+          0.003601829055696726,
+          -0.0019662617705762386,
+          0.02192271128296852,
+          -0.0018651577411219478,
+          -0.022428231313824654,
+          -0.006196831353008747,
+          -0.01792910508811474,
+          0.008105169050395489,
+          0.010304179973900318,
+          -0.01456739753484726,
+          -0.005088900215923786,
+          -0.0035786593798547983,
+          0.009048805572092533,
+          -0.0046507827937603,
+          0.017828000709414482,
+          -0.00638218829408288,
+          0.01909179985523224,
+          0.03152759000658989,
+          0.01407872885465622,
+          -0.013345724903047085,
+          0.014162981882691383,
+          -0.007473269011825323,
+          -0.0033954083919525146,
+          -0.010666469112038612,
+          0.001936773071065545,
+          0.0015576331643387675,
+          0.005000433884561062,
+          0.0009467966156080365,
+          0.025680409744381905,
+          -0.0012048223288729787,
+          -0.01978267729282379,
+          -0.005805053282529116,
+          -0.0057334378361701965,
+          -0.01224200613796711,
+          0.007755517493933439,
+          0.0014059771783649921,
+          0.0034480667673051357,
+          -0.02081056870520115,
+          0.0031005218625068665,
+          0.005261619575321674,
+          0.005927220452576876,
+          0.006112577859312296,
+          0.0030268002301454544,
+          -0.0038166751619428396,
+          0.02013654261827469,
+          -0.00623053265735507,
+          0.001733511919155717,
+          0.013151941820979118,
+          -0.006415889598429203,
+          -0.010877102613449097,
+          -0.013143517076969147,
+          -0.0007919811177998781,
+          0.019698424264788628,
+          -0.014719054102897644,
+          -0.00436432147398591,
+          0.013556358404457569,
+          0.00863596424460411,
+          0.006710776127874851,
+          -0.0019072843715548515,
+          -0.00604517525061965,
+          0.0026876807678490877,
+          -0.02343927137553692,
+          -0.020557807758450508,
+          -0.008905574679374695,
+          -0.008496946655213833,
+          0.024264954030513763,
+          -0.006681287661194801,
+          0.005956709384918213,
+          -0.00756173487752676,
+          0.006466441787779331,
+          0.01824926771223545,
+          0.005413275212049484,
+          -0.003639743197709322,
+          0.006757115479558706,
+          -0.00021734721667598933,
+          0.002392794005572796,
+          -0.009318416006863117,
+          0.0031658182851970196,
+          0.015291975811123848,
+          -0.010995057411491871,
+          0.0022263936698436737,
+          -0.007738666608929634,
+          -0.0036629128735512495,
+          -0.022866349667310715,
+          0.010430560447275639,
+          -0.001800914527848363,
+          0.009613302536308765,
+          0.007873471826314926,
+          -0.008058829233050346,
+          0.012789652682840824,
+          -0.008694942109286785,
+          0.015713242813944817,
+          -0.012250431813299656,
+          -0.0004007298266515136,
+          0.03017953597009182,
+          -0.011155138723552227,
+          -0.019007546827197075,
+          -0.0034733428619802,
+          -0.016176635399460793,
+          0.02308540791273117,
+          0.027230670675635338,
+          0.015435206703841686,
+          0.016075532883405685,
+          -0.007456418126821518,
+          0.0066517991945147514,
+          0.025680409744381905,
+          0.003513363189995289,
+          0.020962223410606384,
+          -0.01162695698440075,
+          0.011803888715803623,
+          -0.03017953597009182,
+          0.0003238486824557185,
+          -0.0029994179494678974,
+          -0.004992008674889803,
+          0.007843983359634876,
+          -0.02131608873605728,
+          -0.008711792528629303,
+          -0.004992008674889803,
+          -0.00719102006405592,
+          0.0041389442048966885,
+          -0.019732125103473663,
+          0.00952062476426363,
+          -0.018451474606990814,
+          -0.01658947765827179,
+          -0.012006097473204136,
+          -0.01806391030550003,
+          -0.013480530120432377,
+          -0.01721295155584812,
+          -0.030819861218333244,
+          -0.00356602156534791,
+          -0.02062521129846573,
+          -0.025073785334825516,
+          0.018721086904406548,
+          -0.023860538378357887,
+          0.01199767179787159,
+          -0.0008514850633218884,
+          -0.006458016578108072,
+          0.025680409744381905,
+          0.029471809044480324,
+          -0.009866062551736832,
+          -0.023708881810307503,
+          -0.009655429981648922,
+          -0.029117945581674576,
+          0.009714406915009022,
+          0.011028758250176907,
+          0.0092425886541605,
+          -0.01090237870812416,
+          0.0015397293027490377,
+          -0.014516845345497131,
+          0.01732248067855835,
+          -0.032117363065481186,
+          -0.029522361233830452,
+          -0.008029340766370296,
+          -0.006879283115267754,
+          -0.001448103808797896,
+          -0.015350953675806522,
+          0.005969346966594458,
+          -0.018080761656165123,
+          -0.02748342975974083,
+          -0.0030331192538142204,
+          -0.006163129583001137,
+          -0.01304241269826889,
+          0.005219492595642805,
+          -0.00557335652410984,
+          0.022394530475139618,
+          0.010666469112038612,
+          0.015780646353960037,
+          -0.004848778247833252,
+          -0.009192036464810371,
+          -0.014306212775409222,
+          -0.01573851890861988,
+          0.004777162801474333,
+          0.009748108685016632,
+          0.012621146626770496,
+          0.0004154741473030299,
+          0.010076696053147316,
+          0.000991029548458755,
+          0.013210919685661793,
+          0.010270479135215282,
+          0.010910804383456707,
+          0.021164432168006897,
+          -0.0023801561910659075,
+          0.014188257977366447,
+          0.00402730843052268,
+          0.005459614563733339,
+          -0.0029383343644440174,
+          -0.008357929065823555,
+          0.022360829636454582,
+          -0.007658625952899456,
+          -0.013960774056613445,
+          0.029337003827095032,
+          0.004697122145444155,
+          -0.02320336177945137,
+          0.012629571370780468,
+          0.0003151600540149957,
+          -0.018721086904406548,
+          -0.008652815595269203,
+          -0.0009815511293709278,
+          -0.004473851062357426,
+          -0.00765441358089447,
+          0.0014112429926171899,
+          -0.008686516433954239,
+          0.01621876284480095,
+          0.016875937581062317,
+          -0.008930850774049759,
+          -0.008429544046521187,
+          0.017844852060079575,
+          0.016614753752946854,
+          -0.001709289033897221,
+          -0.022209173068404198,
+          -0.008694942109286785,
+          0.02504008449614048,
+          -0.03629632666707039,
+          -0.026893656700849533,
+          0.007700752932578325,
+          0.006420102436095476,
+          -0.009756533429026604,
+          6.574062354047783e-06,
+          0.003983075264841318,
+          -0.007578585296869278,
+          0.0032479651272296906,
+          0.0016208230517804623,
+          0.012680123560130596,
+          0.01484543364495039,
+          -0.011096160858869553,
+          0.010556939989328384,
+          -0.01573851890861988,
+          0.0018672640435397625,
+          -0.020035438239574432,
+          -0.013126665726304054,
+          -0.01380069274455309,
+          0.010371582582592964,
+          0.020591510459780693,
+          0.011829164810478687,
+          0.03889133036136627,
+          0.01466007623821497,
+          0.004014670383185148,
+          -0.006917196791619062,
+          -0.017288779839873314,
+          0.004802438896149397,
+          -0.002948865992948413,
+          -0.0037766548339277506,
+          -0.0021158112213015556,
+          0.014912836253643036,
+          -0.0021779481321573257,
+          0.028123755007982254,
+          0.006158917210996151,
+          0.01236838661134243,
+          -2.6213969249511138e-05,
+          -0.007022513542324305,
+          -0.003983075264841318,
+          0.013101390562951565,
+          5.410642552305944e-05,
+          -0.0023675181437283754,
+          0.0067192018032073975,
+          -0.0018483070889487863,
+          -0.005055198911577463,
+          0.0027656150050461292,
+          -0.011365772224962711,
+          -0.0055101667530834675,
+          -0.017356181517243385,
+          -0.008787620812654495,
+          -0.009175186045467854,
+          0.02485472708940506,
+          0.028865184634923935,
+          -0.015013940632343292,
+          -0.010430560447275639,
+          -0.013455254025757313,
+          0.03589191287755966,
+          -0.004840353038161993,
+          0.0034291099291294813,
+          0.014811732806265354,
+          -0.008753919042646885,
+          0.0024222827050834894,
+          0.012418937869369984,
+          0.0042210910469293594,
+          0.003555489704012871,
+          0.029926776885986328,
+          -0.0031721373088657856,
+          -0.01013567391782999,
+          -0.0035028315614908934,
+          0.003907247446477413,
+          -0.026101676747202873,
+          0.0003712411562446505,
+          0.004840353038161993,
+          -0.010514813475310802,
+          0.013092964887619019,
+          -0.019513066858053207,
+          0.006795029621571302,
+          -0.010632768273353577,
+          -0.0008367407135665417,
+          -0.01976582780480385,
+          -0.0007946140831336379,
+          0.007633350323885679,
+          -0.012587444856762886,
+          -0.003197413170710206,
+          -0.017356181517243385,
+          -0.0020062820985913277,
+          0.014643225818872452,
+          0.007001450285315514,
+          0.006040962412953377,
+          -0.0007472215802408755,
+          0.0032016257755458355,
+          0.022478783503174782,
+          0.0043474710546433926,
+          0.004718185402452946,
+          -0.004541253671050072,
+          0.0042526861652731895,
+          -0.024264954030513763,
+          0.006453803740441799,
+          -0.005762926768511534,
+          -0.0026497666258364916,
+          0.024955831468105316,
+          -0.005842967424541712,
+          0.0009678599308244884,
+          -0.006238957867026329,
+          0.005059411283582449,
+          -0.024568265303969383,
+          0.01508976798504591,
+          -0.01762579381465912,
+          0.010700170882046223,
+          -0.008564349263906479,
+          -0.002706637606024742,
+          -0.00251074880361557,
+          0.009444796480238438,
+          0.012292558327317238,
+          -0.013084539212286472,
+          0.007397440727800131,
+          -0.015822771936655045,
+          0.021686803549528122,
+          -0.003867227118462324,
+          -0.01878848858177662,
+          0.0018314564367756248,
+          -0.008046191185712814,
+          0.008762344717979431,
+          -0.014702202752232552,
+          0.0010515867033973336,
+          0.010396858677268028,
+          -0.007216296158730984,
+          0.005809266120195389,
+          -0.023119108751416206,
+          0.006828730925917625,
+          -0.017490988597273827,
+          0.008796045556664467,
+          -0.012966584414243698,
+          -0.012966584414243698,
+          -0.02118128351867199,
+          -0.012006097473204136,
+          -0.00024551941896788776,
+          0.0051436647772789,
+          0.015713242813944817,
+          -0.006934047676622868,
+          0.005838754586875439,
+          0.021872159093618393,
+          -0.010026143863797188,
+          0.01719610020518303,
+          0.053113289177417755,
+          -0.025444500148296356,
+          -0.016934916377067566,
+          0.00136279733851552,
+          0.010784423910081387,
+          -0.03434165194630623,
+          -0.008581199683248997,
+          0.019395112991333008,
+          -0.016361992806196213,
+          0.029606614261865616,
+          -0.025663558393716812,
+          0.01156798005104065,
+          -0.010708595626056194,
+          0.00796193815767765,
+          -0.017052870243787766,
+          -0.01929400861263275,
+          -0.00991661474108696,
+          -0.003568127751350403,
+          -0.0018462006701156497,
+          -0.02394479140639305,
+          0.009495348669588566,
+          -0.014390465803444386,
+          -0.011045609600841999,
+          0.006150492001324892,
+          -0.0057882023975253105,
+          0.007460630964487791,
+          0.0016608433797955513,
+          0.013758566230535507,
+          -0.020776866003870964,
+          0.0008767610415816307,
+          -0.011045609600841999,
+          0.004747674334794283,
+          -0.0067739663645625114,
+          0.011298369616270065,
+          0.01337942574173212,
+          -0.013092964887619019,
+          -0.002439133357256651,
+          -0.002023132750764489,
+          0.015511034987866879,
+          -0.006504355929791927,
+          0.03683554753661156,
+          -0.008075679652392864,
+          -0.004339045379310846,
+          -0.008871873840689659,
+          -0.0056870984844863415,
+          0.011829164810478687,
+          0.01328674703836441,
+          0.008568561635911465,
+          0.005834541749209166,
+          0.00914990995079279,
+          -0.014112429693341255,
+          0.01536780409514904,
+          0.007949300110340118,
+          0.017659494653344154,
+          0.005307958461344242,
+          -0.013851244933903217,
+          -0.024837875738739967,
+          -0.0066855004988610744,
+          -0.007262635510414839,
+          -0.0029446533881127834,
+          -0.01665687933564186,
+          0.0035512770991772413,
+          0.0067781792022287846,
+          0.007827132940292358,
+          -0.003900928422808647,
+          0.03257232904434204,
+          0.005729225464165211,
+          0.0012753845658153296,
+          0.027635086327791214,
+          0.007793431635946035,
+          0.0015112938126549125,
+          0.017524689435958862,
+          -0.031072622165083885,
+          -0.01404502708464861,
+          -0.021889010444283485,
+          0.011761762201786041,
+          0.0004615501675289124,
+          -0.0318477526307106,
+          -0.0024833662901073694,
+          0.0011121437419205904,
+          -0.01025362778455019,
+          0.012629571370780468,
+          0.0206926129758358,
+          -0.016378844156861305,
+          -0.011947119608521461,
+          0.008037766441702843,
+          0.011070885695517063,
+          -0.005190004128962755,
+          0.027635086327791214,
+          -0.002001016167923808,
+          0.009630153886973858,
+          0.007405866403132677,
+          0.019428813830018044,
+          0.0006540163303725421,
+          0.023523524403572083,
+          -0.006643373519182205,
+          -0.00483614020049572,
+          -0.020490406081080437,
+          -0.008412693627178669,
+          -0.01175333745777607,
+          0.011610106565058231,
+          -0.005206854548305273,
+          0.003789292648434639,
+          -0.013952348381280899,
+          0.007915598340332508,
+          0.014963388442993164,
+          0.011028758250176907,
+          0.0044443621300160885,
+          0.012124051339924335,
+          0.010455836541950703,
+          0.013590059243142605,
+          0.0012353642378002405,
+          -0.020338749513030052,
+          0.008914000354707241,
+          0.017608942463994026,
+          -0.006934047676622868,
+          0.02141719125211239,
+          0.023540375754237175,
+          0.001537623000331223,
+          -0.003188987961038947,
+          0.008518009446561337,
+          0.01652207411825657,
+          0.01976582780480385,
+          -0.01298343576490879,
+          -0.0032332208938896656,
+          0.0036629128735512495,
+          -0.0028983140364289284,
+          -0.0016408332157880068,
+          -0.019327709451317787,
+          -0.000656649237498641,
+          0.008614901453256607,
+          0.0010921335779130459,
+          0.030230088159441948,
+          -0.01823241636157036,
+          -0.01428936142474413,
+          0.01915920339524746,
+          0.006622310262173414,
+          -0.01880533993244171,
+          0.012503191828727722,
+          0.00015968635852914304,
+          0.013960774056613445,
+          -0.011104586534202099,
+          0.005215280223637819,
+          -0.01004299521446228,
+          -0.004503339529037476,
+          0.017524689435958862,
+          -0.009267864748835564,
+          0.011778613552451134,
+          -0.008383204229176044,
+          -0.001989431446418166,
+          -0.02074316516518593,
+          0.02032189816236496,
+          0.02613537758588791,
+          0.01086867693811655,
+          0.008412693627178669,
+          0.004751886706799269,
+          0.007271060720086098,
+          0.00436432147398591,
+          0.01927715726196766,
+          0.020658912137150764,
+          -0.011424749158322811,
+          0.007494332268834114,
+          0.018148163333535194,
+          -0.007249997463077307,
+          0.012840204872190952,
+          0.004827714990824461,
+          0.004970945417881012,
+          -0.007094128988683224,
+          0.006015686318278313,
+          -0.017162399366497993,
+          -0.013033987022936344,
+          0.0012627465184777975,
+          0.032606031745672226,
+          0.008454820141196251,
+          -0.023304466158151627,
+          0.001874636160209775,
+          0.024349207058548927,
+          -0.020894821733236313,
+          -0.0034754490479826927,
+          0.008315801620483398,
+          -0.0019157096976414323,
+          0.012899181805551052,
+          -0.02461881749331951,
+          -0.00786925945430994,
+          0.004810864105820656,
+          -0.006348486989736557,
+          0.016193486750125885,
+          -0.012098775245249271,
+          -0.0390598364174366,
+          -0.01199767179787159,
+          -0.020153392106294632,
+          -0.017861701548099518,
+          -0.019664723426103592,
+          0.0008930850890465081,
+          0.01585647277534008,
+          -0.019024398177862167,
+          -0.018333520740270615,
+          0.016741132363677025,
+          -0.001975740073248744,
+          0.03764437884092331,
+          -0.017373032867908478,
+          0.007536458782851696,
+          -0.01727192848920822,
+          0.01646309718489647,
+          -0.016800111159682274,
+          -0.018502026796340942,
+          0.01475275494158268,
+          -0.0031679244711995125,
+          -0.003721890039741993,
+          0.0023422420490533113,
+          0.00235698651522398,
+          -0.009714406915009022,
+          -0.003024693811312318,
+          0.011652233079075813,
+          0.0006982493214309216,
+          -0.0161008071154356,
+          -0.002645554021000862,
+          0.003528107423335314,
+          0.012882331386208534,
+          0.01578907109797001,
+          -0.0069635361433029175,
+          -0.003808249719440937,
+          0.010211501270532608,
+          -0.01762579381465912,
+          -0.010363157838582993,
+          0.015300401486456394,
+          -0.01732248067855835,
+          0.03191515430808067,
+          0.006592821795493364,
+          -0.01776059903204441,
+          -0.00716153159737587,
+          0.003629211336374283,
+          0.023287614807486534,
+          0.01332044880837202,
+          -0.020726315677165985,
+          -0.008004064671695232,
+          0.007810282055288553,
+          0.012739100493490696,
+          0.005438551306724548,
+          -0.007186807692050934,
+          0.027820443734526634,
+          -0.0007530139992013574,
+          0.00015547369548585266,
+          0.001172174233943224,
+          -0.01056536566466093,
+          -0.0021737355273216963,
+          0.008804471231997013,
+          0.015839623287320137,
+          0.004292706493288279,
+          0.00985763780772686,
+          0.005333234556019306,
+          -0.01635356806218624,
+          0.015713242813944817,
+          0.0068076676689088345,
+          -0.0035449580755084753,
+          -0.016370419412851334,
+          0.011273093521595001,
+          0.007119405083358288,
+          0.016513649374246597,
+          -0.012772802263498306,
+          -0.004882479552179575,
+          -1.0696221579564735e-05,
+          0.00737637747079134,
+          0.0022221810650080442,
+          0.008067254908382893,
+          0.0010705436579883099,
+          0.02938755601644516,
+          0.03282508999109268,
+          0.03312840312719345,
+          -0.025326546281576157,
+          -0.010321030393242836,
+          0.008231548592448235,
+          -0.003157392842695117,
+          -0.004654995631426573,
+          -0.0003354334912728518,
+          -0.01720452681183815,
+          0.006971961818635464,
+          -0.02620278112590313,
+          0.01199767179787159,
+          -0.001448103808797896,
+          0.005147877614945173,
+          -0.010860252194106579,
+          0.0066517991945147514,
+          -0.0013680631527677178,
+          0.0001929927384480834,
+          -0.011424749158322811,
+          -0.019209755584597588,
+          -0.0014586354373022914,
+          0.014002900570631027,
+          -0.0012374705402180552,
+          -0.008122019469738007,
+          0.014061877503991127,
+          -0.006057813297957182,
+          -0.023237064480781555,
+          0.006087301764637232,
+          -0.028882035985589027,
+          0.0021010669879615307,
+          -0.013092964887619019,
+          0.003909353632479906,
+          -0.022680992260575294,
+          0.013421552255749702,
+          -0.0013554252218455076,
+          -0.01267169788479805,
+          0.004941456951200962,
+          -0.017288779839873314,
+          0.0025676197838038206,
+          -0.003197413170710206,
+          0.006272659171372652,
+          -0.010708595626056194,
+          0.028376515954732895,
+          0.014575823210179806,
+          -0.015140320174396038,
+          -0.017052870243787766,
+          0.008446394465863705,
+          -0.02809005416929722,
+          0.021703653037548065,
+          0.010885528288781643,
+          -0.0034733428619802,
+          -0.02406274527311325,
+          0.009748108685016632,
+          0.0036966141778975725,
+          0.02050725556910038,
+          0.010860252194106579,
+          0.009293140843510628,
+          0.018822189420461655,
+          -0.015527885407209396,
+          0.0021800545509904623,
+          -1.19962232929538e-05,
+          -0.00251074880361557,
+          0.014921261928975582,
+          -0.008071467280387878,
+          0.014584247954189777,
+          0.007359527051448822,
+          -0.0044232988730072975,
+          -0.006268446333706379,
+          -0.0030520763248205185,
+          -0.015165596269071102,
+          0.0004289020143914968,
+          -0.0020241858437657356,
+          -0.0008130444912239909,
+          0.006647586356848478,
+          0.010110397823154926,
+          -0.017305631190538406,
+          0.00914990995079279,
+          -0.0012248324928805232,
+          -0.025848915800452232,
+          0.0014038708759471774,
+          -0.016741132363677025,
+          0.002437027171254158,
+          -0.021383490413427353,
+          0.01252004224807024,
+          0.025208590552210808,
+          0.011037183925509453,
+          -0.007073065731674433,
+          -0.006234745029360056,
+          -0.011879716999828815,
+          -0.007485906593501568,
+          -0.010514813475310802,
+          -0.006125215906649828,
+          -0.019209755584597588,
+          -0.003728209063410759,
+          0.011121436953544617,
+          0.011728061363101006,
+          0.00034227909054607153,
+          -0.018350372090935707,
+          -0.005135239567607641,
+          -0.013109815306961536,
+          -0.005400637164711952,
+          -0.008547498844563961,
+          -0.013084539212286472,
+          0.0011974502122029662,
+          0.012545318342745304,
+          -0.011669083498418331,
+          0.029421256855130196,
+          0.005779777187854052,
+          0.006243170239031315,
+          -0.0022158620413392782,
+          -0.007881897501647472,
+          -0.004162113647907972,
+          -0.012216730043292046,
+          -0.0022074368316680193,
+          -0.005341660231351852,
+          -0.0020757908932864666,
+          -0.00905723124742508,
+          0.005813478492200375,
+          -0.0015913344686850905,
+          0.01555316150188446,
+          0.006062025669962168,
+          -0.008126231841742992,
+          -0.002955185016617179,
+          0.031679246574640274,
+          -0.007140468340367079,
+          -0.016614753752946854,
+          -0.00037861333112232387,
+          -0.00638218829408288,
+          0.0037303154822438955,
+          0.007414291612803936,
+          -0.01769319549202919,
+          0.0043895975686609745,
+          0.010851826518774033,
+          0.004794013686478138,
+          -0.0033827705774456263,
+          0.02081056870520115,
+          0.008779195137321949,
+          -0.002984673483297229,
+          0.010329456068575382,
+          0.0032269018702208996,
+          -0.010068271309137344,
+          -0.0044232988730072975,
+          -0.0026476604398339987,
+          0.028814632445573807,
+          -0.03484717011451721,
+          -0.0020547276362776756,
+          0.019732125103473663,
+          0.0044780634343624115,
+          -0.021147580817341805,
+          -0.002312753349542618,
+          -0.026977911591529846,
+          -0.0020673656836152077,
+          -0.009891338646411896,
+          0.013404701836407185,
+          -0.036026716232299805,
+          -0.002512854989618063,
+          -0.00905723124742508,
+          0.0044991266913712025,
+          -0.006373763084411621,
+          0.026893656700849533,
+          -0.0008936116937547922,
+          -0.005505953915417194,
+          0.011011907830834389,
+          0.009689130820333958,
+          0.011947119608521461,
+          0.026590345427393913,
+          -0.006934047676622868,
+          -0.0067529031075537205,
+          0.003083671210333705,
+          -0.028073204681277275,
+          0.02276524528861046,
+          0.00362710515037179,
+          0.0032669221982359886,
+          -0.0004115247866138816,
+          -0.006740265060216188,
+          0.02625333145260811,
+          0.02866297774016857,
+          -0.005223705433309078,
+          -0.005282682832330465,
+          -0.016387268900871277,
+          0.029101094231009483,
+          -0.011441599577665329,
+          0.0011205690680071712,
+          -0.002015760401263833,
+          -0.0121493274345994,
+          -0.01732248067855835,
+          -0.013446828350424767,
+          0.019984886050224304,
+          -0.02008599042892456,
+          -0.0017724790377542377,
+          -0.012882331386208534,
+          -0.001390179619193077,
+          -0.02497268095612526,
+          -0.008467458188533783,
+          0.008572774939239025,
+          -0.014070303179323673,
+          -0.0032416461035609245,
+          0.0003148967516608536,
+          -0.006436952855437994,
+          0.0011311006965115666,
+          -0.0067866044119000435,
+          -0.00031437017605639994,
+          -0.003799824509769678,
+          0.011129862628877163,
+          0.0002694789436645806,
+          -0.008855023421347141,
+          0.019496217370033264,
+          -0.010919229127466679,
+          0.006795029621571302,
+          0.0043221949599683285,
+          -0.004010457545518875,
+          -0.0003080511814914644,
+          0.009697556495666504,
+          0.014188257977366447,
+          -0.013708014041185379,
+          0.021939562633633614,
+          0.009377393871545792,
+          -0.01484543364495039,
+          0.010405284352600574,
+          -0.005354297813028097,
+          -0.019866930320858955,
+          -0.0026687236968427896,
+          0.015022365376353264,
+          0.019850080832839012,
+          -0.022445082664489746,
+          0.0011195158585906029,
+          0.01873793639242649,
+          -0.00021194973669480532,
+          0.0021379278041422367,
+          0.007338463328778744,
+          0.006142066325992346,
+          0.007485906593501568,
+          0.010026143863797188,
+          0.009739683009684086,
+          -0.0037576977629214525,
+          -0.0206926129758358,
+          0.015199298039078712,
+          -0.012140902690589428,
+          0.01561213843524456,
+          0.023776285350322723,
+          -0.010607492178678513,
+          -0.021467743441462517,
+          0.015106619335711002,
+          -0.007186807692050934,
+          -0.02827541157603264,
+          -0.0015039215795695782,
+          -0.0017292991979047656,
+          0.01978267729282379,
+          0.0030815647915005684,
+          -2.3564598450320773e-05,
+          -0.018282968550920486,
+          -1.969750337593723e-05,
+          0.01080970000475645,
+          -0.0017524688737466931,
+          -0.0411493182182312,
+          -0.006217894610017538,
+          0.004263217560946941,
+          -0.00486562866717577,
+          0.014255660586059093,
+          0.0017651069210842252,
+          0.005527017172425985,
+          0.026000572368502617,
+          -0.019698424264788628,
+          -0.010118822567164898,
+          0.006521206349134445,
+          0.016370419412851334,
+          0.010152524337172508,
+          0.011104586534202099,
+          0.006508568301796913,
+          0.015224573202431202,
+          -0.012039798311889172,
+          -0.007102554198354483,
+          0.018266119062900543,
+          -0.013387851417064667,
+          0.014811732806265354,
+          -0.01811446249485016,
+          0.004461213015019894,
+          -0.006310573313385248,
+          0.004739248659461737,
+          0.010969781316816807,
+          -0.00768390204757452,
+          -0.0010357891442254186,
+          -0.040744900703430176,
+          0.004570742137730122,
+          -0.0027213820721954107,
+          0.014710628427565098,
+          -0.01105403434485197,
+          0.004608656279742718,
+          -0.0001983902184292674,
+          0.005834541749209166,
+          0.009689130820333958,
+          -0.0008893990307115018,
+          0.01615135930478573,
+          -0.02601742371916771,
+          -0.025141188874840736,
+          7.082543743308634e-05,
+          -0.006917196791619062,
+          0.008762344717979431,
+          0.020423002541065216,
+          0.012444213964045048,
+          0.023489823564887047,
+          -0.01205664873123169,
+          0.008509584702551365,
+          0.011188839562237263,
+          -0.016875937581062317,
+          -0.003881971351802349,
+          0.0011953439097851515,
+          -0.008400055579841137,
+          -0.02332131750881672,
+          -0.0018177651800215244,
+          -0.004671846050769091,
+          0.008623326197266579,
+          0.0035639151465147734,
+          0.006377975922077894,
+          -0.025427650660276413,
+          0.004160007461905479,
+          0.004265323746949434,
+          -0.0298762246966362,
+          0.00018074968829751015,
+          -0.017423585057258606,
+          0.008627539500594139,
+          -0.0016650561010465026,
+          0.021282386034727097,
+          -0.013126665726304054,
+          -0.022849498316645622,
+          -0.005716587416827679,
+          0.005425913259387016,
+          -0.007389015518128872,
+          -0.01025362778455019,
+          0.015814347192645073,
+          -0.017069721594452858,
+          -0.008008277043700218,
+          -0.021585699170827866,
+          -0.006601247005164623,
+          -0.036026716232299805,
+          -0.006344274617731571,
+          0.01056536566466093,
+          0.005548080429434776,
+          -0.010068271309137344,
+          -0.029657166451215744,
+          0.005889306776225567,
+          -0.0067781792022287846,
+          -0.004549678880721331,
+          -0.01096135564148426,
+          0.0014923367416486144,
+          0.022006966173648834,
+          -0.006411677226424217,
+          0.02148459479212761,
+          -0.0174404364079237,
+          -0.004705547355115414,
+          -0.017061294987797737,
+          0.0044570001773536205,
+          -0.0022179684601724148,
+          -0.004583380185067654,
+          -0.00666022440418601,
+          -0.011736486107110977,
+          0.0024875791277736425,
+          0.002984673483297229,
+          -0.007435354869812727,
+          -0.007532245945185423,
+          0.012267282232642174,
+          0.009141484275460243,
+          0.007831345312297344,
+          0.007843983359634876,
+          0.004840353038161993,
+          -0.012621146626770496,
+          -0.00032700816518627107,
+          -0.006024111993610859,
+          0.005194216500967741,
+          0.009933465160429478,
+          0.0057207997888326645,
+          0.005004646722227335,
+          -0.008897149935364723,
+          0.012081924825906754,
+          0.01899069733917713,
+          0.033263206481933594,
+          -0.01304241269826889,
+          -0.005168940871953964,
+          -0.014011326245963573,
+          0.02166995219886303,
+          0.019024398177862167,
+          0.003317474154755473,
+          0.005088900215923786,
+          0.006925622466951609,
+          -0.0019315071403980255,
+          0.0013238302199169993,
+          0.012098775245249271,
+          -0.00872021820396185,
+          -0.009293140843510628,
+          -0.011323644779622555,
+          0.0013733289670199156,
+          -0.01781114935874939,
+          0.008058829233050346,
+          0.006441165693104267,
+          0.002065259264782071,
+          0.01025362778455019,
+          0.03936314582824707,
+          0.0026139591354876757,
+          -0.0016471522394567728,
+          -0.00238647498190403,
+          -0.01915920339524746,
+          0.02124868519604206,
+          -0.007915598340332508,
+          -0.039295744150877,
+          -0.0024960043374449015,
+          0.002679255325347185,
+          0.01347210444509983,
+          -0.0079324496909976,
+          -0.000739322800654918,
+          -0.011787038296461105,
+          -0.02074316516518593,
+          0.0014459975063800812,
+          0.015039216727018356,
+          -0.0005660769529640675,
+          0.011955545283854008,
+          0.003568127751350403,
+          0.0026350223924964666,
+          0.0034164718817919493,
+          0.009537475183606148,
+          -0.02148459479212761,
+          0.002752976957708597,
+          0.007671264000236988,
+          0.028747230768203735,
+          0.0007682848954573274,
+          -0.007233147043734789,
+          0.017828000709414482,
+          -0.0034164718817919493,
+          -0.007157318759709597,
+          0.008041978813707829,
+          -0.011256242170929909,
+          0.010893953032791615,
+          -0.013143517076969147,
+          0.011795463971793652,
+          0.02290005050599575,
+          -0.0206926129758358,
+          0.005383786745369434,
+          -0.007325825747102499,
+          -0.003957799170166254,
+          0.015696391463279724,
+          0.020102839916944504,
+          0.006255808286368847,
+          0.0010484271915629506,
+          -0.02106332778930664,
+          0.017490988597273827,
+          -0.02081056870520115,
+          -0.0102873295545578,
+          0.005581781733781099,
+          -0.014794881455600262,
+          0.010978206992149353,
+          -0.0038988220039755106,
+          -0.026539793238043785,
+          0.003875652328133583,
+          0.009360543452203274,
+          0.015115044079720974,
+          0.006340061780065298,
+          0.005560718476772308,
+          0.0012048223288729787,
+          -0.012022947892546654,
+          -0.023708881810307503,
+          -0.0058219037018716335,
+          -0.019007546827197075,
+          -0.021889010444283485,
+          -0.006424315273761749,
+          0.007110979408025742,
+          -0.008058829233050346,
+          0.0018704235553741455,
+          0.023001154884696007,
+          -0.018535729497671127,
+          -0.0065675457008183,
+          0.022748393937945366,
+          0.003507044166326523,
+          0.013017136603593826,
+          -0.007460630964487791,
+          0.017423585057258606,
+          0.00954589992761612,
+          -0.012494766153395176,
+          -0.01959731988608837,
+          0.027045313268899918,
+          -0.00495830737054348,
+          0.007321612909436226,
+          0.008846597746014595,
+          0.0012195666786283255,
+          -0.004726610612124205,
+          0.00814308226108551,
+          -0.030465997755527496,
+          0.013151941820979118,
+          -0.00734688900411129,
+          -0.013244620524346828,
+          -0.012208305299282074,
+          -0.00628950959071517,
+          0.001043687923811376,
+          -0.0015302507672458887,
+          0.01603340543806553,
+          -0.006116790696978569,
+          0.01841777376830578,
+          0.003210051218047738,
+          -0.0013785947812721133,
+          0.004608656279742718,
+          -0.0437106192111969,
+          -0.002169522689655423,
+          -0.010851826518774033,
+          -0.022798946127295494,
+          0.023422420024871826,
+          0.010952930897474289,
+          0.00039098801789805293,
+          -0.023557227104902267,
+          0.003938842564821243,
+          -0.004195814952254295,
+          -0.00025855234707705677,
+          0.020406153053045273,
+          0.0026961059775203466,
+          0.0010089334100484848,
+          -0.022731544449925423,
+          -0.0029783546924591064,
+          0.010068271309137344,
+          -0.0028182733803987503,
+          0.0010178852826356888,
+          0.01567954197525978,
+          0.010632768273353577,
+          -0.012393662706017494,
+          -0.018704235553741455,
+          0.005691311322152615,
+          0.016825387254357338,
+          0.008821321651339531,
+          0.0032416461035609245,
+          -0.008071467280387878,
+          0.0044443621300160885,
+          -0.003835632000118494,
+          0.038419511169195175,
+          0.020894821733236313,
+          0.006272659171372652,
+          -0.02504008449614048,
+          0.008475882932543755,
+          -0.019176054745912552,
+          0.004574954975396395,
+          0.010556939989328384,
+          0.00872021820396185,
+          -0.0032269018702208996,
+          -0.007587010972201824,
+          0.017052870243787766,
+          -0.005291108042001724,
+          0.018451474606990814,
+          -0.005286895204335451,
+          -0.00538799911737442,
+          0.018839040771126747,
+          -0.023540375754237175,
+          0.007650200743228197,
+          -0.018586281687021255,
+          -0.011787038296461105,
+          -0.013185643590986729,
+          -0.004248473327606916,
+          0.014087153598666191,
+          0.0013543720124289393,
+          -0.0326734334230423,
+          0.007321612909436226,
+          -0.026775702834129333,
+          0.00860647577792406,
+          -0.011323644779622555,
+          -4.169880776316859e-05,
+          -0.002578151412308216,
+          -0.008496946655213833,
+          -0.02431550621986389,
+          -0.021754205226898193,
+          0.005404850002378225,
+          -0.004570742137730122,
+          -0.0030520763248205185,
+          0.004633932374417782,
+          -0.0018082867609336972,
+          -0.01579749584197998,
+          0.008063041605055332,
+          0.028814632445573807,
+          0.02111387997865677,
+          0.008539073169231415,
+          0.0056196958757936954,
+          0.014727478846907616,
+          0.0011542703723534942,
+          -0.037745483219623566,
+          0.008404267951846123,
+          0.004684484098106623,
+          0.006529631558805704,
+          0.006175767630338669,
+          -0.005539655219763517,
+          0.0017608941998332739,
+          -0.010203076526522636,
+          -0.007553309667855501,
+          0.0029130582697689533,
+          -0.010607492178678513,
+          -0.007169956807047129,
+          -0.031241128221154213,
+          0.003083671210333705,
+          -0.010826550424098969,
+          0.007325825747102499,
+          -0.006971961818635464,
+          0.01641254499554634,
+          0.0016945448005571961,
+          0.00020286618382669985,
+          -0.013817543163895607,
+          -0.013497380539774895,
+          -0.0029320153407752514,
+          5.486338704940863e-05,
+          -0.01885589212179184,
+          -0.008416905999183655,
+          0.005105750635266304,
+          -0.002458090428262949,
+          -0.026657748967409134,
+          0.023489823564887047,
+          -0.000226299132918939,
+          0.011289943940937519,
+          0.023355018347501755,
+          0.020726315677165985,
+          4.5582357415696606e-05,
+          0.004815076943486929,
+          0.0025971082504838705,
+          -0.008404267951846123,
+          -0.01861998252570629,
+          0.007587010972201824,
+          -0.029859373345971107,
+          0.02136664092540741,
+          -0.006146279163658619,
+          -0.018636832013726234,
+          -0.006436952855437994,
+          -0.009082507342100143,
+          0.011011907830834389,
+          0.00015270912263076752,
+          0.01664845459163189,
+          -0.011348920874297619,
+          -0.003734528087079525,
+          -0.005607057828456163,
+          0.0018830614862963557,
+          0.008404267951846123,
+          0.029589762911200523,
+          -0.0032079447992146015,
+          -0.010413709096610546,
+          0.0131940683349967,
+          0.007679689209908247,
+          0.02896628901362419,
+          0.0013954454334452748,
+          -0.0008451660396531224,
+          -0.013868095353245735,
+          -0.0010378954466432333,
+          -0.01075072307139635,
+          -0.007220508996397257,
+          0.02136664092540741,
+          -0.010481111705303192,
+          -0.01995118521153927,
+          -0.00817678403109312,
+          -0.02050725556910038,
+          -0.007717603351920843,
+          0.013025562278926373,
+          -0.004954094998538494,
+          -0.004326407797634602,
+          -0.007182594854384661,
+          -0.01508976798504591,
+          -0.017962805926799774,
+          0.02606797404587269,
+          0.07434512674808502,
+          -0.021265536546707153,
+          -0.009596452116966248,
+          -0.011281518265604973,
+          0.0033954083919525146,
+          -0.004903542809188366,
+          -0.006748690269887447,
+          0.05661822482943535,
+          -0.019984886050224304,
+          0.005868243053555489,
+          -0.020406153053045273,
+          -0.006108365021646023,
+          -8.392419840674847e-05,
+          0.0068877083249390125,
+          0.002439133357256651,
+          -0.008105169050395489,
+          0.009899764321744442,
+          -0.018350372090935707,
+          -0.0057334378361701965,
+          -0.016201911494135857,
+          -0.013977624475955963,
+          -0.0169686172157526,
+          0.011643807403743267,
+          -0.019917482510209084,
+          -0.008956126868724823,
+          -0.0014091366901993752,
+          -0.009588027372956276,
+          0.015317251905798912,
+          -0.009267864748835564,
+          0.009958741255104542,
+          0.0009815511293709278,
+          0.003102628281340003,
+          -0.00939424429088831,
+          -0.010026143863797188,
+          -0.007144680712372065,
+          -0.0023822623770684004,
+          0.005703949369490147,
+          0.0007082544034346938,
+          0.0014491570182144642,
+          0.007292123977094889,
+          -0.0067739663645625114,
+          0.0054764654487371445,
+          0.020355600863695145,
+          0.0009315256611444056,
+          0.004899329971522093,
+          -0.017255079001188278,
+          0.019799528643488884,
+          -0.00517315324395895,
+          0.0289494376629591,
+          0.0057123745791614056,
+          -0.009689130820333958,
+          -0.016640029847621918,
+          0.014247234910726547,
+          -0.013573208823800087,
+          -0.0034227909054607153,
+          0.01983322948217392,
+          0.029168495908379555,
+          0.00789874792098999,
+          -0.012536892667412758,
+          0.0010415816213935614,
+          -4.456210444914177e-05,
+          0.01959731988608837,
+          -0.011348920874297619,
+          0.03230271860957146,
+          -0.011896567419171333,
+          0.004448574967682362,
+          -0.026118526235222816,
+          -0.009402669966220856,
+          -0.006323210895061493,
+          -0.003669231664389372,
+          0.022546187043190002,
+          -0.0044907014816999435,
+          0.007999852299690247,
+          -0.01267169788479805,
+          -0.021046478301286697,
+          0.004642357584089041,
+          -0.0043685343116521835,
+          -0.006723414175212383,
+          0.01579749584197998,
+          -0.029488658532500267,
+          0.004250579513609409,
+          0.009023529477417469,
+          -0.013101390562951565,
+          -0.017356181517243385,
+          0.012814928777515888,
+          0.008202060125768185,
+          -0.009461646899580956,
+          -0.008821321651339531,
+          -0.019395112991333008,
+          -0.02760138548910618,
+          0.010144098661839962,
+          0.0018925400217995048,
+          0.006192618515342474,
+          0.0022221810650080442,
+          -0.011728061363101006,
+          0.020709464326500893,
+          -0.005055198911577463,
+          0.009048805572092533,
+          -0.017726896330714226,
+          0.006281084381043911,
+          0.011155138723552227,
+          0.003974650055170059,
+          -0.0004581273824442178,
+          0.01404502708464861,
+          0.007338463328778744,
+          0.033027298748493195,
+          0.01270539965480566,
+          0.008189422078430653,
+          -0.009992443025112152,
+          -0.009882913902401924,
+          0.012140902690589428,
+          -0.021080179139971733,
+          -0.0007645988371223211,
+          0.016008129343390465,
+          0.010422134771943092,
+          -0.016260888427495956,
+          0.012680123560130596,
+          -0.01720452681183815,
+          0.010818125680088997,
+          -0.03378557786345482,
+          -0.012494766153395176,
+          0.006592821795493364,
+          0.014011326245963573,
+          0.008871873840689659,
+          0.0002376206684857607,
+          0.017592091113328934,
+          0.015376229770481586,
+          0.0031342231668531895,
+          0.006845581810921431,
+          0.022815797477960587,
+          0.004697122145444155,
+          -0.004671846050769091,
+          0.0035091505851596594,
+          -0.005244768690317869,
+          0.0016776940319687128,
+          -0.010919229127466679,
+          0.011483726091682911,
+          0.008897149935364723,
+          -0.006575970910489559,
+          0.01365746185183525,
+          -0.0009689130820333958,
+          -0.012250431813299656,
+          -0.007325825747102499,
+          -0.004587593022733927,
+          -0.008981402963399887,
+          -0.005649184808135033,
+          0.019513066858053207,
+          -0.015696391463279724,
+          -0.008939276449382305,
+          0.0024496649857610464,
+          0.014685352332890034,
+          -0.0025634069461375475,
+          -0.006571758538484573,
+          -0.008678090758621693,
+          -0.00019167628488503397,
+          -0.01205664873123169,
+          0.01450842060148716,
+          0.026742001995444298,
+          0.023894239217042923,
+          -0.004225303418934345,
+          -0.0032858792692422867,
+          -0.0013448934769257903,
+          0.013632185757160187,
+          0.0010505334939807653,
+          0.00133225554600358,
+          -0.007709178142249584,
+          -0.005725012626498938,
+          -0.018653683364391327,
+          -0.0014038708759471774,
+          0.0017956487135961652,
+          0.00014099264808464795,
+          -0.005973559804260731,
+          -0.013421552255749702,
+          0.011003483086824417,
+          0.008846597746014595,
+          0.0174404364079237,
+          -0.014668501913547516,
+          -0.02980882115662098,
+          0.0195299182087183,
+          -0.003911459818482399,
+          -0.004372747149318457,
+          0.003587084822356701,
+          -0.004574954975396395,
+          0.04333990439772606,
+          0.0051436647772789,
+          0.005939858499914408,
+          -0.011441599577665329,
+          -0.008787620812654495,
+          0.015637414529919624,
+          -0.005219492595642805,
+          -0.002031557960435748,
+          -0.019732125103473663,
+          -0.02466936968266964,
+          0.005131026729941368,
+          -0.024332357570528984,
+          -0.001645045937038958,
+          0.002439133357256651,
+          -0.0013027668464928865,
+          -0.013303598389029503,
+          -0.015949152410030365,
+          -0.003936735913157463,
+          0.011045609600841999,
+          0.01719610020518303,
+          -7.589380402350798e-05,
+          0.006116790696978569,
+          -0.00746905617415905,
+          0.003593403846025467,
+          0.01233468484133482,
+          -0.003881971351802349,
+          -0.01647152192890644,
+          -0.007511182688176632
+        ],
+        "title": "Appendices A. Overview Of Hyper-Parameters"
+      },
+      "type": "chunk"
+    }
+  ],
+  "relationships": [
+    {
+      "id": "5d2a4e39-ad33-4004-9074-5994e76a5dd8",
+      "type": "child",
+      "source": {
+        "id": "b570d71e-4f83-486e-b70f-b0c6217db455",
+        "properties": {
+          "page_content": "Mixture-Of-Experts With Expert Choice Routing\n\nYanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon Google, Mountain View, CA, USA {yanqiz, taole, hanxiaol, dunan, huangyp, vzhao, adai, zhifengc, qvl, jlaudon}@google.com\n\nAbstract\n\nSparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choice method. Instead of letting tokens select the top-k experts, we have experts selecting the top-k tokens. As a result, each token can be routed to a variable number of experts and each expert can have a fixed bucket size. We systematically study pre-training speedups using the same computational resources of the Switch Transformer top-1 and GShard top-2 gating of prior work and find that our method improves training convergence time by more than 2×. For the same computational cost, our method demonstrates higher performance in fine-tuning 11 selected tasks in the GLUE and SuperGLUE benchmarks. For a smaller activation cost, our method outperforms the T5 dense model in 7 out of the 11 tasks.\n\n1 Introduction\n\nScaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27]. The final model quality has been found to have a power-law relationship with the amount of data, model size, and compute time [16, 20]. However, training efficiency, which is defined as the total amount of computation used to achieve superior model quality than the state of the art system [21], should receive greater attention as we increase our efforts towards green AI [29].\n\nSparsely gated mixture-of-experts [31] (MoE) provides an effective way to scale model capacity given a fixed computational cost, and has recently played an important role in increasing the training efficiency of large-scale language models [10, 21]. MoE operate by adopting a number of experts, each as a sub-network, and by activating only one or a few experts for each input token. A gating network must be chosen and optimized in order to route each token to the most suited expert(s). For example, recent work has implemented sparse routing via k-means clustering [12], linear assignment to maximize token-expert affinities [22], or hashing [8, 28]. Many of the prior work use a routing strategy concerning the token choice, where each token selects the best one or two experts.\n\nWe argue that the independent token choice of prior work often leads to an imbalanced load of experts, which causes training inefficiency and sub-optimal training of the model. In order to mitigate this\n\nissue, previous sparsely gated networks introduce additional auxiliary losses as regularization to prevent too many tokens being routed to a single expert, but the effectiveness is still limited. Recent approaches [8, 22, 28] explore alternative strategies for routing, but they focus on pre-training only and do not demonstrate performance gain on downstream tasks. Moreover, none of the previous methods consider allocating a variable number of experts to each token based on importance, which can be beneficial.\n\nWe propose a very simple yet effective routing method we are calling expert choice. Unlike conventional MoE where tokens select one or two top-scoring experts, our method lets each expert pick the top-k tokens. Our method guarantees perfect load balancing, allows a variable number of experts for each token, and achieves substantial gains in training efficiency and downstream performance as demonstrated in our experiments. Our major contributions include:\n\nWe identify common pitfalls in conventional MoE such as load imbalance as described in Section 3.1. We then propose a heterogeneous, expert choice method to provide a fluid allocation of model parameters based on a learnt token-to-expert importance. This method intrinsically guarantees load balance without imposing an auxiliary loss.\n\nWe show our method provides over 2× faster training convergence in a 8B/64E (8 billion activated parameters, 64 experts) model, compared to the top-1 and top-2 gating counterparts in Switch Transformer [10] and GShard [21].\n\nWe show our method demonstrates strong scaling when increasing the number of experts from 16 to 128, evaluated in training perplexity.\n\nWe show our method demonstrates strong performance on downstream tasks selected from GLUE and SuperGLUE at all the evaluated scales. More specifically, our 8B/64E model outperforms a T5 11B dense model in 7 out of 11 tasks evaluated.\n\n2 Related Work\n\nScaling: Various approaches have been proposed to scale up neural network capacity to improve performance. Recent works have successfully scaled models to billions of parameters via various forms of model parallelism [2, 21, 26, 27, 33]. Model parallelism [30] splits weights and tensors across multiple cores while pipeline parallelism [18, 24] splits different layers across devices with micro-batches pipelined to the different layers. To enable continued scaling of neural networks, improving model training and serving efficiency has become a critical research area.\n\nConditional Computation: Computation decisions can be made dynamically based on the input [23, 25]. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation, by activating certain parameters and computation on demand, on a per-example or per-token basis [3]. Conditional convolution layers [1] with task-specific gating has been used to combat catastrophic forgetting when a sequence of learning problems are optimized. The gating decisions may be binary or sparse and continuous, stochastic or deterministic.\n\nMixture of Experts: Sparsely-gated MoE [31] is the first model to demonstrate massive improvements in model capacity, training time, or model quality with gating. Switch Transformer [10] simplifies the gating by selecting only the top expert per token using a softmax over the hidden state and demonstrates better scaling than previous work. All the prior work requires an auxiliary loss to explicitly encourage balancing. This loss term has to be carefully weighted to not overwhelm the primary loss. However, auxiliary loss does not guarantee balancing and a hard capacity factor has to be imposed. As a result, many tokens can still be unprocessed by the MoE layer. Hard MoE [12] with a single decoding layer can be efficiently trained to good effect on large scale hashtag prediction tasks.\n\nBase Layers [22] formulate a linear assignment that maximizes token-expert affinities while ensuring each expert receives an equal number of tokens. Hash layers [8, 28] devise hashing techniques on input tokens. However, the evaluations are limited to pre-training perplexity. THOR [? ] randomly activates experts during training and inference and is trained with a consistency regularization loss.\n\nTHOR has demonstrated strong performance on translation tasks. Different from these prior works, our method is a learnt method that enables heterogeneous MoE and effectively improves downstream fine-tuning performance.\n\n3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n5 Conclusion\n\nWe propose a new routing method for sparsely activated mixture-of-experts (MoE) models. This method addresses load imbalance and under-utilization of experts in conventional MoE methods, and enables selecting different numbers of experts for each token. Our model demonstrates more than 2x training efficiency improvements when compared to the state-of-the-art GShard and Switch Transformer models, and also achieves strong gains when finetuning on 11 datasets in the GLUE and SuperGLUE benchmark.\n\n6 Limitations\n\nThe expert choice method might not immediately apply to auto-regressive text generation as our current implementation takes in the past and future tokens to perform the top-k selection. One possible solution is to collect a large batch of input sequences, dispatch tokens of the same sequence into separate groups, and perform expert choice routing for each group. Another scenario where the expert choice method does not immediately apply is when the batch size becomes very small during serving or inference. A global top-k can be selected instead and we can cap the number of times each expert or token gets selected. We leave these possible improvements for future work.\n\nAnother long-standing issue with MoE has been the large memory footprint. Even though computational cost can be reduced using sparsely gated networks, the total number of parameters increases linearly or sub-linearly with the number of experts. Increasing the number of experts requires reservation of a large number of hardware devices. Therefore, dynamic (used) power is saved while static (reserved) power is not. Power saving techniques such as the ability to put hardware devices into low power states while not in use [17] can help with reducing the reserved power requirements.\n\nReferences\n\n[1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pages 3930–3939. Computer Vision Foundation / IEEE, 2020.\n\n[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. In Advances in Neural Information Processing Systems.\n\n[3] Kyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning, 2014.\n\n[4] Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems, 2021.\n\n[5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.\n\nTransformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.\n\n[6] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 933–941. JMLR.org, 2017.\n\n[7] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixtureof-experts, 2021.\n\n[8] Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan.\n\nTricks for training sparse translation models, 2021.\n\n[9] Richard L Dykstra. An iterative procedure for obtaining i-projections onto the intersection of convex sets. The annals of Probability, pages 975–984, 1985.\n\n[10] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.\n\n[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. NAS-FPN: learning scalable feature pyramid architecture for object detection. In CVPR, pages 7036–7045. Computer Vision Foundation / IEEE, 2019.\n\n[12] Sam Gross, Marc'Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision, 2017.\n\n[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.\n\n[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 630–645, Cham, 2016. Springer International Publishing.\n\n[15] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2016.\n\n[16] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.\n\n[17] Ping Huang, Zuocheng Xing, Tianran Wang, Qiang Wei, Hongyan Wang, and Guitao Fu. A brief survey on power gating design. In 2010 10th IEEE International Conference on Solid-State and Integrated Circuit Technology, pages 788–790, 2010.\n\n[18] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 103–112, 2019.\n\n[19] Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David A. Patterson. A domain-specific supercomputer for training deep neural networks. Commun. ACM, 63(7):67–78, 2020.\n\n[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.\n\n[21] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.\n\n[22] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6265–6274. PMLR, 18–24 Jul 2021.\n\n[24] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. New York, NY, USA, 2019. Association for Computing Machinery.\n\n[25] Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, Cédric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. In ICLR. OpenReview.net, 2021.\n\n[26] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.\n\n[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.\n\n[28] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models, 2021.\n\n[30] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 10435–10444, Red Hook, NY, USA, 2018. Curran Associates Inc.\n\n[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E.\n\nHinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In ICLR (Poster). OpenReview.net, 2017.\n\n[32] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 10–15 Jul 2018. [23] Min Lin, Jie Fu, and Yoshua Bengio. Conditional computation for continual learning, 2019.\n\n[29] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.\n\n[33] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.\n\n[34] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc.\n\n[35] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.\n\nGLUE: A multi-task benchmark and analysis platform for natural language understanding.\n\nIn Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, November 2018. Association for Computational Linguistics.\n\n[36] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake A. Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen. GSPMD: general and scalable parallelization for ML computation graphs. CoRR, abs/2105.04663, 2021.\n\n7 Checklist\n\n(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Yes (b) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes (c) Did you discuss any potential negative societal impacts of your work? N/A. Not any. (d) Did you describe the limitations of your work? Yes (a) Did you include the code, data, and instructions needed to reproduce the main experimental results? Yes. We include details in the experiment setup to help reproduce the main results. (b) Did you specify all the training details? Yes (c) Did you report error bars? Yes (d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? Yes (a) If your work uses existing assets, did you cite the creators? Yes (b) Did you mention the license of the assets? No. The used dataset is not released yet. (c) Did you include any new assets either in the supplemental material or as a URL? No. The dataset is not released yet.\n\n(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? No. Not using persons' data. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes. The dataset does not contain any personally identifiable information or offensive content.\n\nA Comparison On Fine-Tuning With A Dense Model\n\nOur 8B MoE model achieves stronger pre-training perplexity than its dense counterpart. However, a better perplexity does not always directly translate to downstream performance as demonstrated in Section 4.4. To this end, we compare fine-tuning performance of the 8B dense model and MoE model in Table 1. As shown in the table, our MoE model using expert choice routing consistently outperforms the dense model across the 11 tasks in GLUE and SuperGLUE.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 1: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nB Capacity Factor\n\nWe evaluate the downstream task fine-tuning performance by varying the capacity factors. Note that a capacity factor of n indicates on average how many experts each token can be received. EC-CF2 is our baseline expert choice, which matches GShard top-2 gating computational footprint. EC-CF1, however, matches Switch Transformer top-1 gating computational footprint. EC-CF0.5 further verifies that an aggressively lowered capacity factor can provide strong enough performance, that almost matches the top-2 gating baseline.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Top-2 78.1 87.0 88.3 85.0 82.6 90.1 90.7 81.6 94.7 68.2 67.2 83.0±0.3 EC-CAP2 78.2 88.0 88.5 85.7 83.0 90.8 91.1 80.0 95.4 70.4 64.1 83.2±0.4 EC-CAP3 78.5 91.7 89.3 86.3 83.5 90.9 91.1 81.8 94.9 70.0 65.6 84.0±0.4 EC-CF2 79.1 89.6 89.3 86.8 84.3 91.3 91.2 81.1 95.2 68.1 68.0 84.0±0.2 EC-CF1 77.4 90.6 88.0 85.5 83.6 90.3 91.2 79.8 95.3 66.5 64.9 83.0±0.2 EC-CF0.5 77.4 89.6 86.3 85.2 82.7 91.7 91.0 79.6 94.9 67.3 63.5 83.0 ±0.05 Hash Layers 76.1 85.2 86.7 83.4 82.5 90.0 90.3 75.7 94.0 67.4 63.3 81.3±1.0\n\nTable 2: Comparison between different routing methods in fine-tuning of 100M/64E models. We perform 3 independent fine-tuning runs for each method and report the average results. This gives more accurate difference between the variants of expert choice method, since they achieve close fine-tuning results. We do not report averaged results in other experiments.\n\nC Capped Expert Choice\n\nAs described in Section 4.5, the maximum number of experts each token is assigned can be capped by an entropy-regularized linear programming. Figure 1 compares the validation perplexity when training the 100M/64E models using the base expert choice method (EC-BASE), expert choice capped by two experts per token (EC-CAP2), expert choice capped by three experts per token (EC-CAP3), and GShard top-2 gating.\n\nAs shown in the figure, restricting the number of experts to 2 degrades the perplexity compared to the base expert choice method. This suggests that a more flexible allocation of experts (e.g. more than 2 experts for a token) can enhance model expressiveness. On the other hand, our EC-CAP2 and EC-CAP3 methods still outperform the top-2 gating method by a clear margin. We believe this confirms the effectiveness of a load balanced training, provided by our method. Finally, EC-CAP3 obtains comparable perplexity to EC-BASE. As indicated by Figure 3, only a little fraction of tokens use more than 3 experts therefore we see little or no difference between EC-BASE and EC-CAP3 variants. We present the fine-tuning results of these methods in Table 2.\n\nD Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [? ]. We use mod x to map a token ID to an expert ID. This in some way ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 2. Hashing based routing performs much worse than expert choice in terms of average scores and variance.\n\nE Fine-Tuning Details\n\nWe did a hyperparameter search for both baseline models and expert choice method. For fine-tuning of the 8B dense model, we use a constant learning rate of 0.0001 and a dropout rate of 0.1. We freeze the attention layer and feed-forward layer while leaving the embedding and layer normalization trainable. This setting has been found optimal for the 8B dense model. For MoE 8B/64E models including GShard top-2 gating and expert choice, we found continuing the learning rate from the pre-trained model while using a square root learning rate decay works better. In addition, we do not apply parameter freezing for fine-tuning MoE models. For models with 100M expert size, we use a constant learning rate of 0.0001 and no dropout is used.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/2202.09368v2.md"
+          },
+          "headlines": {
+            "1 Introduction": [],
+            "2 Related Work": [],
+            "3 Method": [
+              "3.1 Pitfalls Of Token-Choice Routing",
+              "3.2 Heterogeneous Moe Via Expert Choice",
+              "3.3 Expert Choice With Additional Constraint",
+              "3.4 Model Architecture"
+            ],
+            "4 Experiments": [
+              "4.1 Setup",
+              "4.2 Training Efficiency",
+              "4.3 Scaling The Number Of Experts",
+              "4.4 Fine-Tuning On Glue And Superglue",
+              "4.5 Heterogeneity Matters",
+              "4.6 Comparison With Hash Layer",
+              "4.7 Ablation"
+            ],
+            "5 Conclusion": [],
+            "6 Limitations": [],
+            "A Comparison On Fine-Tuning With A Dense Model": [],
+            "B Capacity Factor": [],
+            "C Capped Expert Choice": [],
+            "D Comparison With Hash Layer": [],
+            "E Fine-Tuning Details": []
+          },
+          "summary": "The paper introduces a new routing method for sparsely-activated Mixture-of-Experts (MoE) models called 'expert choice,' which addresses load imbalance and under-utilization issues in conventional MoE methods. Unlike traditional token-choice routing, expert choice allows experts to select the top-k tokens, ensuring perfect load balancing and enabling variable expert allocation per token. This method improves training efficiency, achieving over 2x faster convergence compared to existing models like Switch Transformer and GShard. It also demonstrates superior performance on downstream tasks in the GLUE and SuperGLUE benchmarks. The study highlights the benefits of flexible expert allocation and shows that expert choice outperforms hashing-based routing methods. The paper also discusses potential limitations and future improvements for the expert choice method.",
+          "summary_embedding": [
+            -0.009726986289024353,
+            -0.0337262786924839,
+            -0.023302342742681503,
+            -0.019878201186656952,
+            -0.025953780859708786,
+            0.04560471698641777,
+            -0.006874797400087118,
+            0.039665497839450836,
+            -0.006973279174417257,
+            0.034544438123703,
+            -0.025787118822336197,
+            -0.00363057479262352,
+            -0.01052999310195446,
+            0.0054657477885484695,
+            -0.024499278515577316,
+            0.036786794662475586,
+            0.010007280856370926,
+            0.003272630739957094,
+            0.006102093029767275,
+            -0.04048365354537964,
+            -0.018666116520762444,
+            -0.013522328808903694,
+            -0.03230207785964012,
+            0.008969432674348354,
+            0.02484775334596634,
+            0.0016713523073121905,
+            0.0026230288203805685,
+            0.003624893259257078,
+            -0.05311964824795723,
+            0.04339266195893288,
+            0.018681267276406288,
+            0.023711422458291054,
+            -0.016332851722836494,
+            -0.012832955457270145,
+            -0.039120059460401535,
+            -0.0197418425232172,
+            0.030786970630288124,
+            0.039968520402908325,
+            -0.019287310540676117,
+            0.03357476741075516,
+            0.018060073256492615,
+            0.02454473078250885,
+            -0.021272100508213043,
+            -0.029817303642630577,
+            -0.0061892117373645306,
+            -0.007840678095817566,
+            -0.008355814032256603,
+            0.01242387667298317,
+            0.016984347254037857,
+            -0.013007191941142082,
+            0.0212569497525692,
+            -0.01371171697974205,
+            0.03384748846292496,
+            0.008916404098272324,
+            -0.042332086712121964,
+            0.03984731063246727,
+            -0.008007340133190155,
+            0.012105703353881836,
+            0.03281721472740173,
+            0.010333028621971607,
+            -0.005287723150104284,
+            -0.021544819697737694,
+            -0.03378688171505928,
+            0.0010529992869123816,
+            0.00014014738553669304,
+            0.005950582213699818,
+            -0.003069985192269087,
+            -0.009742137044668198,
+            0.03227177634835243,
+            -0.003939277958124876,
+            -0.0740584209561348,
+            0.050089433789253235,
+            -0.026590125635266304,
+            -0.007628562860190868,
+            -0.028938541188836098,
+            -0.01736312545835972,
+            0.037362534552812576,
+            0.005662711802870035,
+            0.0098709212616086,
+            0.0106284748762846,
+            -0.0007892759749665856,
+            0.028923390433192253,
+            0.00910579226911068,
+            -0.024923507124185562,
+            0.04478655755519867,
+            -0.04790768027305603,
+            0.01801462098956108,
+            -0.04336235672235489,
+            0.022756904363632202,
+            1.6393929399782792e-05,
+            -0.020726662129163742,
+            -0.0013304532039910555,
+            -0.016666175797581673,
+            0.0218023881316185,
+            -0.01730252057313919,
+            0.026802239939570427,
+            0.013825350441038609,
+            -0.015802564099431038,
+            -0.029817303642630577,
+            0.025378039106726646,
+            0.003367325058206916,
+            0.010499690659344196,
+            -0.02075696364045143,
+            0.014332910999655724,
+            0.04393810033798218,
+            -0.02593863010406494,
+            -0.02271145209670067,
+            0.027681002393364906,
+            0.014408666640520096,
+            0.0633314698934555,
+            0.036544378846883774,
+            0.006624804809689522,
+            0.022362977266311646,
+            0.020575150847434998,
+            -0.028968842700123787,
+            -0.01963578537106514,
+            -0.046362269669771194,
+            -0.04757435619831085,
+            0.019878201186656952,
+            -0.04566532000899315,
+            0.0050869714468717575,
+            0.038544319570064545,
+            -0.013704141601920128,
+            -0.014446543529629707,
+            -0.04021093621850014,
+            0.017847958952188492,
+            0.0003908502112608403,
+            0.011976920068264008,
+            -0.0022082682698965073,
+            0.00033285003155469894,
+            -0.011825408786535263,
+            0.008878526277840137,
+            -0.03778676688671112,
+            -0.025196226313710213,
+            0.025181075558066368,
+            0.022862963378429413,
+            0.024120500311255455,
+            0.028802180662751198,
+            -0.03077181987464428,
+            -0.009984553791582584,
+            0.021378157660365105,
+            -0.04242299124598503,
+            -0.006556625012308359,
+            0.011332998983561993,
+            0.0026703758630901575,
+            0.003872992005199194,
+            0.032786913216114044,
+            0.052998438477516174,
+            -0.03172633796930313,
+            0.02065090648829937,
+            -0.010431510396301746,
+            0.04957429692149162,
+            0.00024691506405360997,
+            0.028968842700123787,
+            -0.019514575600624084,
+            -0.0134389977902174,
+            -0.029529431834816933,
+            -0.009795165620744228,
+            -0.01356020662933588,
+            -0.008484598249197006,
+            -0.002998017705976963,
+            -0.009575475007295609,
+            0.0040453351102769375,
+            0.0331808403134346,
+            0.00273855566047132,
+            0.030680913478136063,
+            0.014325335621833801,
+            0.0051475754007697105,
+            -0.08745196461677551,
+            0.026059838011860847,
+            -0.0033616432920098305,
+            0.005859675817191601,
+            0.0062460280023515224,
+            0.03013547509908676,
+            -0.0312718041241169,
+            0.020938776433467865,
+            0.026620427146553993,
+            0.0011145505122840405,
+            0.016969196498394012,
+            0.036847397685050964,
+            -0.04181694984436035,
+            0.009643655270338058,
+            0.012590537779033184,
+            -0.0033881578128784895,
+            0.04584713280200958,
+            0.011628445237874985,
+            0.019514575600624084,
+            0.018529755994677544,
+            0.003569970605894923,
+            0.00297907879576087,
+            0.006533898413181305,
+            0.03724132478237152,
+            0.003462019143626094,
+            -0.009833043441176414,
+            0.014976831153035164,
+            -0.010757259093225002,
+            -0.011219366453588009,
+            -0.037998881191015244,
+            0.014454119838774204,
+            0.008742166683077812,
+            -0.017454031854867935,
+            -0.02959003672003746,
+            0.0026949963066726923,
+            -0.031908150762319565,
+            0.026559824123978615,
+            -0.002445003716275096,
+            0.005738467443734407,
+            -0.010454237461090088,
+            -0.006757376715540886,
+            -0.035423196852207184,
+            0.028453707695007324,
+            -0.02095392718911171,
+            0.0053142374381423,
+            0.0007258309051394463,
+            -0.029438525438308716,
+            0.0060073984786868095,
+            0.017863109707832336,
+            -0.0025965142995119095,
+            0.0025302283465862274,
+            -0.02349930815398693,
+            0.024468975141644478,
+            0.025635607540607452,
+            0.004518806003034115,
+            -0.014037464745342731,
+            -0.013113250024616718,
+            -0.003605954349040985,
+            -0.009742137044668198,
+            -0.016196493059396744,
+            0.024468975141644478,
+            -0.014810169115662575,
+            0.00308892410248518,
+            0.021847840398550034,
+            0.014264730736613274,
+            -0.007045246660709381,
+            -0.004007457755506039,
+            -0.04066546633839607,
+            -0.01218903437256813,
+            -0.04760465770959854,
+            -0.011628445237874985,
+            -0.0034601252991706133,
+            -0.028696123510599136,
+            0.0061513339169323444,
+            -0.04481685906648636,
+            -0.01235569640994072,
+            0.02309022843837738,
+            0.018135828897356987,
+            0.0019488062243908644,
+            -0.009173971600830555,
+            -0.022605394944548607,
+            8.486965816700831e-05,
+            -0.01763584464788437,
+            -0.017044952139258385,
+            0.015393485315144062,
+            0.031059689819812775,
+            -0.03101423755288124,
+            0.04024123772978783,
+            0.003893824527040124,
+            -0.08066429197788239,
+            0.010393633507192135,
+            -0.026362858712673187,
+            0.07011914253234863,
+            -0.0025453795678913593,
+            0.014075342565774918,
+            0.06781618297100067,
+            -0.012431452050805092,
+            -0.02007516659796238,
+            0.03142331540584564,
+            -0.008249756880104542,
+            -1.1311515663692262e-05,
+            -0.02884763479232788,
+            -0.02739313244819641,
+            -0.053271155804395676,
+            -0.009704259224236012,
+            -0.010787560604512691,
+            0.03360506892204285,
+            -0.024317465722560883,
+            -0.0007656024536117911,
+            0.012620840221643448,
+            0.02983245439827442,
+            0.016635872423648834,
+            -0.0015681355725973845,
+            0.008272483013570309,
+            0.02881733328104019,
+            0.0274234339594841,
+            -0.0036154238041490316,
+            0.00814369972795248,
+            -0.034211114048957825,
+            -0.0068066176027059555,
+            -0.0493621826171875,
+            0.011302697472274303,
+            -0.02596893161535263,
+            -0.02596893161535263,
+            0.007643714081496,
+            -0.011476934887468815,
+            0.03748374432325363,
+            -0.024696242064237595,
+            -0.007503566797822714,
+            0.005037730094045401,
+            0.012855681590735912,
+            -0.017757052555680275,
+            -0.03612014651298523,
+            -0.0028502948116511106,
+            0.0026381798088550568,
+            0.007560383062809706,
+            -0.053240854293107986,
+            0.004499867558479309,
+            -0.018469152972102165,
+            0.0279688723385334,
+            -0.02139330841600895,
+            -0.013688990846276283,
+            0.042029064148664474,
+            0.011363301426172256,
+            0.027499189600348473,
+            -0.026468917727470398,
+            0.0017849853029474616,
+            0.012825379148125648,
+            -0.0020056227222085,
+            0.032514192163944244,
+            -0.030574856325984,
+            -0.03257479891180992,
+            0.001548249856568873,
+            -0.007840678095817566,
+            -0.0625436082482338,
+            -0.011386028490960598,
+            -0.00981789268553257,
+            0.017317671328783035,
+            0.0446956530213356,
+            0.04557441547513008,
+            -0.01068150345236063,
+            0.002174178371205926,
+            -0.02213571034371853,
+            -0.04542290419340134,
+            -0.02643861435353756,
+            0.0016486257081851363,
+            -0.019999410957098007,
+            -0.02789311669766903,
+            -0.0018313854234293103,
+            0.011211791075766087,
+            -0.013651113025844097,
+            0.01134057529270649,
+            -0.018560059368610382,
+            -0.01862066239118576,
+            -0.012332970276474953,
+            0.018575210124254227,
+            0.009901223704218864,
+            -0.0018796793883666396,
+            -0.015037435106933117,
+            -0.00980274099856615,
+            0.018969137221574783,
+            -0.011279970407485962,
+            0.0009388927719555795,
+            0.01188601367175579,
+            0.024347767233848572,
+            -0.007083124481141567,
+            0.010961798019707203,
+            -0.034756552428007126,
+            -0.01966608688235283,
+            0.007984613068401814,
+            0.002719616750255227,
+            0.020711511373519897,
+            0.008742166683077812,
+            -0.04275631532073021,
+            -0.044301725924015045,
+            0.016044981777668,
+            -0.0562104657292366,
+            0.028120383620262146,
+            -0.03327174484729767,
+            -0.012514783069491386,
+            -0.02671133354306221,
+            0.0013418165035545826,
+            -0.01828734017908573,
+            0.022438732907176018,
+            0.03778676688671112,
+            -0.0011486404109746218,
+            0.005503625608980656,
+            0.014658658765256405,
+            0.04384719207882881,
+            -0.05021063983440399,
+            -0.041180603206157684,
+            0.03754434734582901,
+            -0.01601468026638031,
+            0.0185146052390337,
+            0.00021744151308666915,
+            0.010348179377615452,
+            -0.014431392773985863,
+            0.022772056981921196,
+            0.013757170177996159,
+            -0.008764892816543579,
+            -0.030574856325984,
+            0.04945308715105057,
+            -0.023650817573070526,
+            -0.049513693898916245,
+            0.0018323323456570506,
+            0.018878230825066566,
+            0.018635815009474754,
+            -0.014870774000883102,
+            -0.02907489985227585,
+            -0.024317465722560883,
+            -0.006060427520424128,
+            -0.00947699323296547,
+            -0.03402930125594139,
+            0.011363301426172256,
+            0.0048445542342960835,
+            0.016666175797581673,
+            -0.00645056739449501,
+            -0.0073141781613230705,
+            -0.009370936080813408,
+            0.022559940814971924,
+            0.012635990977287292,
+            -0.03175663948059082,
+            -0.017484333366155624,
+            -0.02528713271021843,
+            0.004359719809144735,
+            0.010136065073311329,
+            0.022166013717651367,
+            0.024287162348628044,
+            -0.0323626808822155,
+            0.04218057543039322,
+            0.01550711877644062,
+            -0.008727015927433968,
+            -0.0019317612750455737,
+            0.02478714846074581,
+            0.04324115067720413,
+            0.0218023881316185,
+            0.026514369994401932,
+            0.002107892418280244,
+            8.705945947440341e-05,
+            -0.015620751306414604,
+            -0.0011391709558665752,
+            0.01862066239118576,
+            0.04911976307630539,
+            -0.012151156552135944,
+            0.01716616004705429,
+            -0.03578682243824005,
+            0.00396200455725193,
+            -0.02580226957798004,
+            -0.030786970630288124,
+            0.03745344281196594,
+            -0.015060162171721458,
+            0.0191964041441679,
+            0.025044716894626617,
+            -0.022469034418463707,
+            -0.011075431481003761,
+            -0.019423669204115868,
+            -0.026181045919656754,
+            -0.01702980138361454,
+            -0.03281721472740173,
+            -0.018135828897356987,
+            -0.002653330797329545,
+            -0.035635314881801605,
+            -0.04987731948494911,
+            -0.004329417832195759,
+            -0.046392571181058884,
+            -0.010507266037166119,
+            0.017378276214003563,
+            -0.011166337877511978,
+            -0.025711363181471825,
+            -0.005336964037269354,
+            0.003973367623984814,
+            -0.022196315228939056,
+            -0.0066588944755494595,
+            0.008439145050942898,
+            0.035938333719968796,
+            -0.02298417128622532,
+            -0.013060221448540688,
+            -0.0024317465722560883,
+            -0.02887793630361557,
+            -0.007878555916249752,
+            0.008999735116958618,
+            0.007723257411271334,
+            -0.0012357590021565557,
+            -0.05196816474199295,
+            -0.006916462909430265,
+            -0.012992041185498238,
+            0.029196109622716904,
+            0.026468917727470398,
+            0.0012168202083557844,
+            -0.02783251367509365,
+            0.008749742060899734,
+            -0.0072535742074251175,
+            0.02339324913918972,
+            -0.007992188446223736,
+            -0.022908415645360947,
+            -0.02034788578748703,
+            0.030786970630288124,
+            -0.0159086212515831,
+            0.009113367646932602,
+            0.009976978413760662,
+            0.0121360057964921,
+            -0.03899884968996048,
+            -0.001445980160497129,
+            0.0005648507503792644,
+            -0.010128489695489407,
+            0.00423093605786562,
+            -0.006738437805324793,
+            -0.007079336792230606,
+            -0.022090258076786995,
+            -0.0218023881316185,
+            0.03818069398403168,
+            -0.009113367646932602,
+            -0.0035661826841533184,
+            -0.01088604237884283,
+            0.03436262533068657,
+            -0.00595815759152174,
+            -0.01993880607187748,
+            -0.019347915425896645,
+            0.014431392773985863,
+            0.05105910077691078,
+            -0.028529461473226547,
+            -0.03984731063246727,
+            0.0011136035900563002,
+            0.030150625854730606,
+            0.004140029661357403,
+            -0.016908591613173485,
+            0.0035851215943694115,
+            0.056392278522253036,
+            -0.011742077767848969,
+            0.028362801298499107,
+            0.027771908789873123,
+            0.02295386977493763,
+            -0.02021152526140213,
+            -0.01678738370537758,
+            -0.020999381318688393,
+            0.011098157614469528,
+            -0.03027183562517166,
+            0.027059808373451233,
+            0.018969137221574783,
+            0.01538590993732214,
+            0.05054396390914917,
+            -0.03178694099187851,
+            -0.02515077404677868,
+            0.008499749936163425,
+            0.0021722845267504454,
+            0.011007251217961311,
+            -0.007727045100182295,
+            -0.01801462098956108,
+            -0.001950700068846345,
+            -0.014022313989698887,
+            -0.019817598164081573,
+            -0.020938776433467865,
+            0.00954517349600792,
+            0.015696506947278976,
+            0.002649543108418584,
+            0.010863316245377064,
+            0.0016590419691056013,
+            0.01686313934624195,
+            0.01321930717676878,
+            -0.0444229319691658,
+            -7.936555630294606e-05,
+            0.03378688171505928,
+            -0.0293476190418005,
+            -0.012264790013432503,
+            -0.016060132533311844,
+            -0.004439263138920069,
+            -0.01706010289490223,
+            -0.02762039750814438,
+            -0.006874797400087118,
+            -0.01889338344335556,
+            -0.00814369972795248,
+            -0.028196139261126518,
+            -0.009992130100727081,
+            0.01949942484498024,
+            0.022332675755023956,
+            0.001196934375911951,
+            -0.006783891003578901,
+            -0.005901341326534748,
+            0.02092362567782402,
+            0.032514192163944244,
+            0.01952972821891308,
+            0.0058331615291535854,
+            -0.015158643946051598,
+            -0.004295328166335821,
+            -0.012370847165584564,
+            0.016060132533311844,
+            0.012537509202957153,
+            6.344509893096983e-05,
+            0.03493836522102356,
+            0.004170331638306379,
+            -0.022726602852344513,
+            -0.001230077352374792,
+            0.024923507124185562,
+            -0.021590271964669228,
+            -0.0108330138027668,
+            0.040029123425483704,
+            0.0085679292678833,
+            -0.0024506852496415377,
+            -0.019287310540676117,
+            -0.019075196236371994,
+            0.02630225569009781,
+            -0.02133270539343357,
+            0.002083271974697709,
+            0.0482713058590889,
+            0.00035344602656550705,
+            -0.029802151024341583,
+            0.017954016104340553,
+            -0.013242034241557121,
+            -0.018560059368610382,
+            0.01848430372774601,
+            0.0012859469279646873,
+            -0.039968520402908325,
+            0.019014591351151466,
+            0.018408548086881638,
+            0.022635696455836296,
+            -0.02918095886707306,
+            0.0002127068000845611,
+            0.03233237937092781,
+            0.004037759732455015,
+            0.001595596899278462,
+            -0.008530051447451115,
+            -0.0006065162015147507,
+            0.010431510396301746,
+            0.001746160676702857,
+            0.02854461409151554,
+            -0.0030434709042310715,
+            0.027771908789873123,
+            -0.006727074272930622,
+            -0.008863375522196293,
+            0.019181253388524055,
+            -0.02109028771519661,
+            0.013166278600692749,
+            0.0685434341430664,
+            0.0021950111258774996,
+            0.025847723707556725,
+            0.007181606721132994,
+            0.024347767233848572,
+            0.02018122375011444,
+            -0.027271922677755356,
+            -0.02021152526140213,
+            -0.005397567991167307,
+            -0.0014033677289262414,
+            -0.03787767142057419,
+            0.011128460057079792,
+            -0.021302402019500732,
+            -0.013249609619379044,
+            -0.0022347825579345226,
+            -0.002316219499334693,
+            -0.006405114196240902,
+            0.034211114048957825,
+            -0.0014630251098424196,
+            -0.018560059368610382,
+            -0.035423196852207184,
+            0.037726160138845444,
+            0.04048365354537964,
+            0.0389382466673851,
+            -0.005984671879559755,
+            0.03548380360007286,
+            0.0030756669584661722,
+            0.0025302283465862274,
+            0.041453324258327484,
+            0.004791525658220053,
+            -0.011567841283977032,
+            0.02295386977493763,
+            0.012416300363838673,
+            0.03351416438817978,
+            -0.02045394293963909,
+            -0.02630225569009781,
+            -0.003299145260825753,
+            -0.026423463597893715,
+            0.012492056004703045,
+            -0.019726691767573357,
+            0.0018768386216834188,
+            0.007117214612662792,
+            0.015196521766483784,
+            -0.016044981777668,
+            0.014719262719154358,
+            -0.006102093029767275,
+            -0.021014532074332237,
+            0.011719351634383202,
+            -0.007780073676258326,
+            0.011961768381297588,
+            -0.02262054570019245,
+            0.001138224033638835,
+            -0.04063516482710838,
+            0.008280059322714806,
+            0.014022313989698887,
+            -0.01598437689244747,
+            0.03205966204404831,
+            0.020969079807400703,
+            -0.01801462098956108,
+            -0.0009682480012997985,
+            0.0034714885987341404,
+            -0.010393633507192135,
+            -0.011067855171859264,
+            -0.019923655316233635,
+            0.02786281518638134,
+            -0.015802564099431038,
+            -0.025908326730132103,
+            0.02454473078250885,
+            -0.01743887923657894,
+            0.015151068568229675,
+            -0.004299115855246782,
+            0.00888610165566206,
+            -0.019120648503303528,
+            0.02284781076014042,
+            -0.023741723969578743,
+            -0.0009128518868237734,
+            -0.020287280902266502,
+            0.0028389315120875835,
+            0.014135947450995445,
+            -0.018272189423441887,
+            -0.02139330841600895,
+            0.019150950014591217,
+            -0.005420294590294361,
+            -0.003931702114641666,
+            0.010014856234192848,
+            0.0030983935575932264,
+            -0.0029715034179389477,
+            -0.004549108445644379,
+            -0.03281721472740173,
+            -0.02915065549314022,
+            0.0033086147159337997,
+            0.011386028490960598,
+            0.015257125720381737,
+            0.027574945241212845,
+            0.03715042024850845,
+            0.006052851676940918,
+            0.0036949669010937214,
+            0.0009327376610599458,
+            -0.020726662129163742,
+            0.042635105550289154,
+            -0.020469093695282936,
+            0.040847279131412506,
+            0.032544493675231934,
+            -0.0197418425232172,
+            -0.023908386006951332,
+            0.0029052174650132656,
+            0.03115059621632099,
+            0.03151422366499901,
+            0.025211378931999207,
+            0.012310243211686611,
+            0.03690800443291664,
+            -0.004359719809144735,
+            -0.032211173325777054,
+            0.015802564099431038,
+            0.022256920114159584,
+            -0.008105821907520294,
+            0.009673957712948322,
+            0.027438584715127945,
+            0.013279912061989307,
+            -0.007757347077131271,
+            0.030150625854730606,
+            0.0012234487803652883,
+            0.02396899089217186,
+            0.02498411200940609,
+            -0.02190844528377056,
+            0.01743887923657894,
+            -0.0001154677156591788,
+            0.006727074272930622,
+            -0.009840618818998337,
+            0.004348356742411852,
+            0.011810258030891418,
+            0.018272189423441887,
+            0.018908534198999405,
+            0.003246116451919079,
+            -0.07060398161411285,
+            0.019726691767573357,
+            0.004507442936301231,
+            -0.02160542458295822,
+            0.00390140013769269,
+            -0.0024866692256182432,
+            0.012704171240329742,
+            0.02105998620390892,
+            0.012461753562092781,
+            -0.01790856383740902,
+            0.0013626492582261562,
+            0.011961768381297588,
+            -0.00595815759152174,
+            0.04730163514614105,
+            0.0019791084341704845,
+            -0.01437078882008791,
+            -0.016105586662888527,
+            0.013726867735385895,
+            0.02525683119893074,
+            -0.02355991117656231,
+            0.006325571332126856,
+            0.005621046293526888,
+            -0.004147605039179325,
+            -0.010742107406258583,
+            -0.030377892777323723,
+            -0.005280147306621075,
+            -0.03436262533068657,
+            0.023150833323597908,
+            0.015491968020796776,
+            0.014764715917408466,
+            -0.025044716894626617,
+            -0.004280176945030689,
+            0.0046400148421525955,
+            -0.021287251263856888,
+            0.003261267440393567,
+            0.01161329448223114,
+            0.0005104016163386405,
+            -0.020984230563044548,
+            0.00959820207208395,
+            0.013469300232827663,
+            0.06848283112049103,
+            0.0007442962378263474,
+            0.029771849513053894,
+            0.0422411784529686,
+            -0.0012243957025930285,
+            -0.006371024530380964,
+            0.0036286809481680393,
+            -0.019953956827521324,
+            0.003473382443189621,
+            0.025878025218844414,
+            -0.03224147483706474,
+            -0.055058982223272324,
+            -0.007545232307165861,
+            -0.04624105989933014,
+            0.001489539397880435,
+            0.014711687341332436,
+            0.023044776171445847,
+            -0.0037782976869493723,
+            -0.034817155450582504,
+            0.016908591613173485,
+            -0.003071879269555211,
+            0.007014944683760405,
+            -0.02593863010406494,
+            -0.0294839795678854,
+            -0.026014383882284164,
+            -0.02339324913918972,
+            -0.012908710166811943,
+            -0.022726602852344513,
+            0.012590537779033184,
+            -0.0015103721525520086,
+            -0.06587684899568558,
+            -0.009780014865100384,
+            0.016166189685463905,
+            0.001200722181238234,
+            0.011939042247831821,
+            0.011083006858825684,
+            -0.007083124481141567,
+            -0.0018275976181030273,
+            -0.022559940814971924,
+            0.03124150261282921,
+            0.005177877843379974,
+            0.03584742918610573,
+            -0.006503595970571041,
+            -0.009128518402576447,
+            0.01073453202843666,
+            0.04527139291167259,
+            0.0017916138749569654,
+            0.010226971469819546,
+            -0.020560000091791153,
+            0.0015065843472257257,
+            -0.02284781076014042,
+            -0.0031135445460677147,
+            -0.008863375522196293,
+            -0.0021211495622992516,
+            0.015893470495939255,
+            0.0019753205124288797,
+            -0.010204244405031204,
+            0.025438643991947174,
+            0.02251448854804039,
+            -0.009605777449905872,
+            -0.015969226136803627,
+            -0.02437806874513626,
+            -0.014545026235282421,
+            -0.0002627763315103948,
+            -0.010007280856370926,
+            -0.022453883662819862,
+            -0.020878173410892487,
+            -0.002566212322562933,
+            -0.02559015527367592,
+            0.022529639303684235,
+            0.01220418605953455,
+            -4.882668508798815e-05,
+            -0.006723286584019661,
+            -0.038544319570064545,
+            -0.003727162955328822,
+            -0.03118089959025383,
+            -0.005374841392040253,
+            0.03287781774997711,
+            -0.03745344281196594,
+            -0.0002231231628684327,
+            0.014491996727883816,
+            -0.01993880607187748,
+            0.0028938541654497385,
+            0.008961857296526432,
+            0.02769615314900875,
+            -0.027408283203840256,
+            0.0001795638381736353,
+            0.006954340264201164,
+            0.016908591613173485,
+            -0.009878496639430523,
+            0.027196168899536133,
+            -0.04275631532073021,
+            0.011037553660571575,
+            -0.037120118737220764,
+            -0.020423639565706253,
+            -0.01523439958691597,
+            0.008363389410078526,
+            -0.005018791649490595,
+            -0.02637801133096218,
+            -0.010454237461090088,
+            0.006026337388902903,
+            -0.00693918950855732,
+            -0.01902974210679531,
+            -0.010620899498462677,
+            0.027438584715127945,
+            0.00931033119559288,
+            0.007977037690579891,
+            -0.02955973520874977,
+            -0.01631770096719265,
+            0.006280118133872747,
+            0.052089374512434006,
+            -0.01584801822900772,
+            0.014938953332602978,
+            -0.006639956030994654,
+            0.02342355251312256,
+            -0.028484009206295013,
+            -0.01872672140598297,
+            0.0315445251762867,
+            0.013946558348834515,
+            0.007064185570925474,
+            0.028953691944479942,
+            0.027196168899536133,
+            0.027105262503027916,
+            -0.04654408246278763,
+            0.009886072017252445,
+            -0.002613559365272522,
+            -0.008757317438721657,
+            0.03402930125594139,
+            0.01916610263288021,
+            0.022150862962007523,
+            0.0009744031121954322,
+            -0.01730252057313919,
+            0.0139844361692667,
+            0.014885924756526947,
+            0.007681591901928186,
+            -0.014242004603147507,
+            -0.02660527639091015,
+            0.003367325058206916,
+            -0.007992188446223736,
+            0.00047276070108637214,
+            0.020726662129163742,
+            -0.009052763693034649,
+            0.030999086797237396,
+            -0.014605630189180374,
+            -0.02593863010406494,
+            0.008916404098272324,
+            -0.02268115058541298,
+            -0.0041741193272173405,
+            0.0004476667381823063,
+            0.010083036497235298,
+            -0.023847781121730804,
+            -0.04669559374451637,
+            0.019408518448472023,
+            0.028393102809786797,
+            0.005579380784183741,
+            0.01328748743981123,
+            -0.035332292318344116,
+            0.008992158807814121,
+            0.012310243211686611,
+            0.011348150670528412,
+            0.0019847899675369263,
+            0.035362593829631805,
+            -0.00028834378463216126,
+            0.012885984033346176,
+            -0.008620957843959332,
+            0.028362801298499107,
+            0.00667025800794363,
+            -0.0323626808822155,
+            -0.0378473699092865,
+            -0.01513591781258583,
+            0.015257125720381737,
+            0.006692984607070684,
+            -0.011840560473501682,
+            -0.02278720773756504,
+            -0.038817036896944046,
+            0.015787413343787193,
+            -0.012916285544633865,
+            -0.038271598517894745,
+            -0.010158791206777096,
+            -0.009742137044668198,
+            0.016635872423648834,
+            -0.006916462909430265,
+            -0.016802534461021423,
+            -0.016090434044599533,
+            0.005821797996759415,
+            7.161247049225494e-05,
+            0.0244083721190691,
+            -0.001251857029274106,
+            -0.006855858489871025,
+            -0.016393456608057022,
+            -0.01188601367175579,
+            -0.0035074723418802023,
+            0.03602924197912216,
+            -0.014219277538359165,
+            -0.02383263036608696,
+            0.014984406530857086,
+            0.02657497487962246,
+            0.011636020615696907,
+            0.03227177634835243,
+            0.007181606721132994,
+            -0.014022313989698887,
+            0.003016956616193056,
+            -0.033029329031705856,
+            -0.0046400148421525955,
+            0.005196816753596067,
+            -0.006427840795367956,
+            -0.015651054680347443,
+            -0.01815097965300083,
+            -0.015832867473363876,
+            0.030756669119000435,
+            -0.02487805485725403,
+            0.0218023881316185,
+            0.030347591266036034,
+            0.003174148965626955,
+            -0.02915065549314022,
+            0.013598084449768066,
+            -0.0072005451656877995,
+            0.01862066239118576,
+            0.0019014590652659535,
+            0.028014326468110085,
+            -0.007461901288479567,
+            -0.004818039946258068,
+            -0.02501441352069378,
+            -0.03696860745549202,
+            -0.023484155535697937,
+            -0.008817922323942184,
+            0.0027537066489458084,
+            0.020984230563044548,
+            -0.0375746488571167,
+            -0.017757052555680275,
+            0.009855770505964756,
+            0.015757111832499504,
+            -0.03027183562517166,
+            -0.0025908327661454678,
+            -0.010158791206777096,
+            -0.0004055278259329498,
+            0.005140000022947788,
+            -0.02227207086980343,
+            -0.008977008052170277,
+            -0.015726808458566666,
+            -0.0010435298318043351,
+            0.006166485138237476,
+            -0.0007409819518215954,
+            -0.05805889517068863,
+            -0.013969285413622856,
+            0.018635815009474754,
+            0.0026400736533105373,
+            0.008908828720450401,
+            0.015938924625515938,
+            -0.0111739132553339,
+            0.0031533162109553814,
+            -0.026756787672638893,
+            0.023893235251307487,
+            -0.009492144919931889,
+            0.009355784393846989,
+            -0.013742019422352314,
+            0.012514783069491386,
+            0.014014738611876965,
+            0.004795313347131014,
+            0.0010681503918021917,
+            -0.0054051438346505165,
+            0.01601468026638031,
+            0.028377952054142952,
+            -0.023756874725222588,
+            0.011454207822680473,
+            -0.0002514130319468677,
+            -0.0032158142421394587,
+            -0.029196109622716904,
+            0.00509454682469368,
+            0.003931702114641666,
+            -0.005109698045998812,
+            0.016651025041937828,
+            -0.036544378846883774,
+            0.01662072166800499,
+            0.02177208475768566,
+            0.032544493675231934,
+            0.014832896180450916,
+            -0.017287369817495346,
+            -0.011658747680485249,
+            -0.012385998852550983,
+            -0.002624922664836049,
+            0.022181164473295212,
+            0.012120855040848255,
+            -0.01815097965300083,
+            -0.024287162348628044,
+            0.011249667964875698,
+            0.0005283934879116714,
+            -0.01203752402216196,
+            0.010795135982334614,
+            -0.013166278600692749,
+            0.01349202636629343,
+            0.012711746618151665,
+            0.00866641104221344,
+            0.00888610165566206,
+            0.022862963378429413,
+            -0.011878437362611294,
+            -0.017711598426103592,
+            0.00656041270121932,
+            -0.008916404098272324,
+            -0.0034089903347194195,
+            -0.012598113156855106,
+            0.010795135982334614,
+            0.014878349378705025,
+            0.06490717828273773,
+            -0.013143551535904408,
+            0.0009526234352961183,
+            -0.020438792183995247,
+            -0.031938452273607254,
+            -0.0017944546416401863,
+            -0.020908474922180176,
+            0.0111739132553339,
+            0.008370965719223022,
+            0.01598437689244747,
+            0.012022373266518116,
+            -0.0021912232041358948,
+            0.00795431062579155,
+            0.019317612051963806,
+            -0.010423935018479824,
+            7.255941454786807e-05,
+            -0.006503595970571041,
+            0.0013683309080079198,
+            0.0035851215943694115,
+            -0.011719351634383202,
+            -0.0026949963066726923,
+            0.035695917904376984,
+            0.01068150345236063,
+            0.011651171371340752,
+            -0.03445352986454964,
+            0.0020435003098100424,
+            0.020150920376181602,
+            0.007602048572152853,
+            -0.008302785456180573,
+            -0.01563590206205845,
+            0.01706010289490223,
+            -0.015317730605602264,
+            -0.0022499337792396545,
+            0.002888172399252653,
+            -0.010151215828955173,
+            0.04581683129072189,
+            -0.019787294790148735,
+            -0.0011742077767848969,
+            0.031847547739744186,
+            0.0030207443051040173,
+            -0.0040983641520142555,
+            0.005408931523561478,
+            -0.003473382443189621,
+            -0.0069013116881251335,
+            -0.02139330841600895,
+            0.0031381649896502495,
+            0.00667025800794363,
+            -0.019287310540676117,
+            -0.0072005451656877995,
+            -0.007719469256699085,
+            0.006897523999214172,
+            0.023105379194021225,
+            -0.009492144919931889,
+            -0.00016145357221830636,
+            -0.008575504645705223,
+            0.009848194196820259,
+            -0.0011912527261301875,
+            -0.044301725924015045,
+            -0.006147546228021383,
+            0.021181194111704826,
+            0.000973456131760031,
+            0.0005757405888289213,
+            -0.007450537756085396,
+            0.005007428117096424,
+            -0.017423728480935097,
+            -0.008454295806586742,
+            0.02369627170264721,
+            0.01736312545835972,
+            -0.009113367646932602,
+            0.010863316245377064,
+            -0.0035680767614394426,
+            0.00433320552110672,
+            -0.020014561712741852,
+            -0.01095422264188528,
+            0.003462019143626094,
+            0.015272277407348156,
+            -0.0009819786064326763,
+            -0.029029447585344315,
+            -0.0012272365856915712,
+            -0.00024194362049456686,
+            0.02918095886707306,
+            0.01486319862306118,
+            -0.008961857296526432,
+            -0.021211495622992516,
+            -0.013242034241557121,
+            -0.001237652963027358,
+            -0.015726808458566666,
+            -0.009620928205549717,
+            0.011855711229145527,
+            0.0009137988090515137,
+            0.008992158807814121,
+            -0.018135828897356987,
+            -0.027347678318619728,
+            -0.016166189685463905,
+            0.0018465365283191204,
+            -0.012870832346379757,
+            0.003340810537338257,
+            0.015514694154262543,
+            -0.03430201858282089,
+            -0.015060162171721458,
+            -0.03805948421359062,
+            0.008280059322714806,
+            -0.0029658216517418623,
+            -0.02024182677268982,
+            -0.023605365306138992,
+            0.009484568610787392,
+            0.019181253388524055,
+            -0.0015548784285783768,
+            0.019847899675369263,
+            -0.000889651826582849,
+            0.020060013979673386,
+            -0.0073141781613230705,
+            0.0031703610438853502,
+            -0.013007191941142082,
+            0.0030245319940149784,
+            0.008333087898790836,
+            0.02890823967754841,
+            -0.023075077682733536,
+            -0.006503595970571041,
+            -0.004685468040406704,
+            0.004746072459965944,
+            0.012658718042075634,
+            0.014613205567002296,
+            0.020741812884807587,
+            -0.008272483013570309,
+            -0.0027612822595983744,
+            0.012484480626881123,
+            -0.003799130441620946,
+            0.007295239716768265,
+            0.002390081062912941,
+            0.0019677451346069574,
+            0.004162756260484457,
+            0.008477022871375084,
+            0.0053710537031292915,
+            0.027650700882077217,
+            -0.0028275682125240564,
+            0.0031703610438853502,
+            0.024620486423373222,
+            0.02322658896446228,
+            -0.018878230825066566,
+            -0.028074929490685463,
+            0.002024561632424593,
+            0.050695475190877914,
+            0.007245998363941908,
+            -0.006810405291616917,
+            -0.03927157074213028,
+            -0.0031760428100824356,
+            0.028635520488023758,
+            0.023650817573070526,
+            0.01584801822900772,
+            -0.010567869991064072,
+            -0.012098127976059914,
+            0.02772645466029644,
+            0.013295062817633152,
+            0.02616589516401291,
+            -0.025302285328507423,
+            0.010704229585826397,
+            -0.03869583085179329,
+            0.0449683703482151,
+            0.004439263138920069,
+            -0.014476845972239971,
+            0.0011884119594469666,
+            0.010386057198047638,
+            -0.004537744913250208,
+            0.006431628484278917,
+            -5.8059369621332735e-05,
+            -0.0008086882880888879,
+            0.011923890560865402,
+            0.012007221579551697,
+            0.004265025723725557,
+            -0.02399929240345955,
+            0.007359631359577179,
+            0.004272601567208767,
+            0.03005971945822239,
+            0.005431658122688532,
+            -0.0013863227795809507,
+            0.012385998852550983,
+            -0.005810434930026531,
+            0.008855799213051796,
+            -0.010719381272792816,
+            0.020469093695282936,
+            -0.02918095886707306,
+            1.0986003871948924e-05,
+            0.02051454596221447,
+            -0.0015785519499331713,
+            0.0038199631962925196,
+            -0.023741723969578743,
+            0.01186328660696745,
+            -0.052968136966228485,
+            0.021166043356060982,
+            -0.018741872161626816,
+            0.021741783246397972,
+            0.019590331241488457,
+            -0.009734561666846275,
+            -0.012052674777805805,
+            0.03545350208878517,
+            0.0026760573964565992,
+            -0.014264730736613274,
+            -0.010643625631928444,
+            -0.017999470233917236,
+            -0.02830219641327858,
+            0.041968461126089096,
+            -0.0006117243901826441,
+            0.0028275682125240564,
+            0.0024222771171480417,
+            -0.010666352696716785,
+            -0.008923979476094246,
+            0.001954487757757306,
+            0.020499395206570625,
+            -0.03369597718119621,
+            -0.011128460057079792,
+            -0.008689138107001781,
+            -0.02566591091454029,
+            -0.00020300064352340996,
+            -0.011961768381297588,
+            0.026211349293589592,
+            0.01240872498601675,
+            -0.03696860745549202,
+            0.003780191531404853,
+            0.023650817573070526,
+            0.001603172393515706,
+            0.018120678141713142,
+            0.002130619017407298,
+            0.0010795136913657188,
+            0.009742137044668198,
+            -0.010476963594555855,
+            0.026559824123978615,
+            -0.009204274043440819,
+            0.008658835664391518,
+            -0.001950700068846345,
+            -0.017287369817495346,
+            0.017969166859984398,
+            0.004083212930709124,
+            0.024120500311255455,
+            0.011954193003475666,
+            0.011605718173086643,
+            0.01112088467925787,
+            -0.019878201186656952,
+            0.020090317353606224,
+            -0.0018891488434746861,
+            0.009075489826500416,
+            0.024726543575525284,
+            -0.005863463506102562,
+            0.020832719281315804,
+            0.015257125720381737,
+            -0.00020229043730068952,
+            0.011295121163129807,
+            -0.020120618864893913,
+            -0.011779955588281155,
+            0.0085679292678833,
+            -0.009605777449905872,
+            -0.010325453244149685,
+            0.009901223704218864,
+            -0.010105762630701065,
+            -0.015052586793899536,
+            -0.0071247899904847145,
+            0.016635872423648834,
+            0.0028976418543606997,
+            -0.02610529027879238,
+            -1.7326075976598077e-05,
+            0.008901252411305904,
+            0.006166485138237476,
+            0.018741872161626816,
+            0.011317848227918148,
+            -0.006692984607070684,
+            0.010363331064581871,
+            0.011476934887468815,
+            0.0053104497492313385,
+            -0.005079395603388548,
+            -0.013431422412395477,
+            -0.0024241709616035223,
+            0.018090376630425453,
+            0.009810317307710648,
+            -0.01922670565545559,
+            0.004564259201288223,
+            0.0008877579239197075,
+            -0.01420412678271532,
+            -0.019347915425896645,
+            -0.0038483713287860155,
+            -0.004170331638306379,
+            0.0027915844693779945,
+            0.0509984977543354,
+            -0.0006216673064045608,
+            0.007893706671893597,
+            -0.025741664692759514,
+            -0.006299056578427553,
+            -0.00027556007262319326,
+            -0.0036097420379519463,
+            0.01272689737379551,
+            -0.01824188604950905,
+            -0.03724132478237152,
+            0.012711746618151665,
+            -0.0012584856012836099,
+            0.014923802576959133,
+            -0.00297907879576087,
+            0.014226853847503662,
+            -0.017347972840070724,
+            0.01916610263288021,
+            -0.0011117096291854978,
+            -0.016029831022024155,
+            0.0029374135192483664,
+            -0.011143610812723637,
+            -0.021317552775144577,
+            0.005916492082178593,
+            -0.006848283112049103,
+            -0.03896854817867279,
+            0.0029885482508689165,
+            -0.010423935018479824,
+            0.0011798894265666604,
+            -0.003912763670086861,
+            0.020166072994470596,
+            -0.0061513339169323444,
+            0.0008697660523466766,
+            0.02177208475768566,
+            0.0017026013229042292,
+            0.01601468026638031,
+            -0.014499573037028313,
+            0.010999675840139389,
+            0.012598113156855106,
+            0.02210540883243084,
+            -0.032180868089199066,
+            -0.01393140759319067,
+            0.014211702160537243,
+            0.0026116655208170414,
+            -0.005382417235523462,
+            0.020635755732655525,
+            -0.01294658798724413,
+            0.019650936126708984,
+            -0.0005577487172558904,
+            -0.011704200878739357,
+            -0.03524138405919075,
+            -0.016529815271496773,
+            -3.343888238305226e-05,
+            -0.008439145050942898,
+            0.025317436084151268,
+            0.01760554127395153,
+            -0.0108330138027668,
+            0.026468917727470398,
+            0.0037574649322777987,
+            0.014598054811358452,
+            -0.014257155358791351,
+            -0.005999823100864887,
+            -0.0003773563075810671,
+            -0.0119314668700099,
+            0.015264702029526234,
+            -0.03512017801403999,
+            0.0008342557121068239,
+            -0.00839369185268879,
+            0.005242269951850176,
+            -0.0254992488771677,
+            0.018257036805152893,
+            0.025817420333623886,
+            -0.0031419529113918543,
+            -0.005037730094045401,
+            -0.022802358493208885,
+            0.028968842700123787,
+            0.03049910068511963,
+            0.0024828813038766384,
+            -0.0013825349742546678,
+            -0.00352072948589921,
+            0.0033218718599528074,
+            0.0211205892264843,
+            -0.037998881191015244,
+            -0.009098216891288757,
+            -0.028696123510599136,
+            0.042332086712121964,
+            -0.008098246529698372,
+            -0.013431422412395477,
+            0.01862066239118576,
+            -0.0024715180043131113,
+            0.02355991117656231,
+            -0.004689255729317665,
+            -0.004855917301028967,
+            0.03148392215371132,
+            0.011514811776578426,
+            0.02772645466029644,
+            -0.04015033319592476,
+            0.014704111963510513,
+            -0.006867222022265196,
+            0.004867280833423138,
+            -0.013484450988471508,
+            0.005302873905748129,
+            0.02580226957798004,
+            -0.018060073256492615,
+            0.0068066176027059555,
+            0.030665762722492218,
+            0.00748084019869566,
+            0.002448791405186057,
+            0.0165904201567173,
+            0.003890036838129163,
+            0.010855740867555141,
+            0.000841831264551729,
+            0.0052574207074940205,
+            0.022423582151532173,
+            0.014454119838774204,
+            -0.016029831022024155,
+            -0.015120766125619411,
+            -0.00044790346873924136,
+            -0.004450626205652952,
+            0.005261208396404982,
+            0.013242034241557121,
+            0.0061172437854111195,
+            -0.015681356191635132,
+            0.01052999310195446,
+            0.014158673584461212,
+            0.013128400780260563,
+            -0.0039847311563789845,
+            -0.03412020578980446,
+            -0.02443867363035679,
+            -0.021438762545585632,
+            0.003104075090959668,
+            -0.0008281006012111902,
+            -0.0021211495622992516,
+            0.0017954016802832484,
+            0.017347972840070724,
+            -0.0008352026343345642,
+            0.00379155483096838,
+            0.01686313934624195,
+            -0.01415109820663929,
+            -0.022090258076786995,
+            -0.0035964848939329386,
+            -0.027529491111636162,
+            0.0053710537031292915,
+            0.001844642567448318,
+            -0.00965123064815998,
+            0.008530051447451115,
+            -0.0046059247106313705,
+            -0.007223271764814854,
+            -0.012196609750390053,
+            -0.02007516659796238,
+            0.005780132487416267,
+            0.016878290101885796,
+            0.005730891600251198,
+            -0.009272454306483269,
+            -0.003939277958124876,
+            0.011575416661798954,
+            0.017484333366155624,
+            0.017317671328783035,
+            -0.009295180439949036,
+            0.009628504514694214,
+            0.0004022135108243674,
+            0.004670316819101572,
+            -0.01834794320166111,
+            0.0045869858004152775,
+            -0.0027953721582889557,
+            -0.008658835664391518,
+            0.01105270441621542,
+            -0.00031201730598695576,
+            0.01628739945590496,
+            -0.013090522959828377,
+            0.005624833982437849,
+            0.010333028621971607,
+            -0.020590301603078842,
+            -0.0067308624275028706,
+            -0.01692374423146248,
+            0.026484068483114243,
+            -0.00504530593752861,
+            0.00045405857963487506,
+            -0.02978700026869774,
+            0.0033616432920098305,
+            0.000848933297675103,
+            -0.000845619011670351,
+            -0.012764775194227695,
+            0.010749682784080505,
+            0.03242328763008118,
+            0.0005179771687835455,
+            0.0216660276055336,
+            -0.0036154238041490316,
+            0.0007480839849449694,
+            -0.00785582885146141,
+            0.004977126140147448,
+            -0.004916521720588207,
+            -0.009901223704218864,
+            -0.010863316245377064,
+            0.021726632490754128,
+            -0.0033029329497367144,
+            0.0003356908564455807,
+            0.011332998983561993,
+            -0.012067826464772224,
+            0.0014620780711993575,
+            -0.0047687990590929985,
+            0.00971941091120243,
+            0.0061702728271484375,
+            0.009113367646932602,
+            -0.01176480483263731,
+            0.05451354384422302,
+            -0.0106284748762846,
+            0.005537715740501881,
+            -0.00439380994066596,
+            -0.022226616740226746,
+            -0.017317671328783035,
+            0.012711746618151665,
+            -0.017544938251376152,
+            0.03375658020377159,
+            0.024150803685188293,
+            0.0006467612111009657,
+            0.010393633507192135,
+            -0.006958128418773413,
+            0.023923536762595177,
+            -0.002602196065708995,
+            0.0010795136913657188,
+            -0.004855917301028967,
+            0.015802564099431038,
+            0.04511988162994385,
+            0.017393426969647408,
+            -0.001152428099885583,
+            0.00829521007835865,
+            0.020575150847434998,
+            0.00814369972795248,
+            -0.0020075165666639805,
+            0.014219277538359165,
+            -0.01024212222546339,
+            0.00900731049478054,
+            -0.004458202049136162,
+            0.027438584715127945,
+            0.00814369972795248,
+            -0.006310420110821724,
+            -0.0036836036015301943,
+            -0.0008209985098801553,
+            0.014173824340105057,
+            0.006950552575290203,
+            0.01963578537106514,
+            -0.00461350055411458,
+            -0.0012575386790558696,
+            0.013870803639292717,
+            0.014491996727883816,
+            -0.00552635220810771,
+            0.001724380999803543,
+            -0.01435563713312149,
+            -0.010204244405031204,
+            0.013575357384979725,
+            0.006086941808462143,
+            0.006310420110821724,
+            -0.014514723792672157,
+            -0.00980274099856615,
+            -0.004670316819101572,
+            0.0015435151290148497,
+            0.015052586793899536,
+            0.011060279794037342,
+            -0.004814252257347107,
+            0.006052851676940918,
+            0.013825350441038609,
+            -0.014014738611876965,
+            0.006147546228021383,
+            0.007208121009171009,
+            0.008855799213051796,
+            0.004787737503647804,
+            -0.010817863047122955,
+            -0.00428396463394165,
+            -0.0063937511295080185,
+            0.0012670081341639161,
+            0.016832837834954262,
+            0.013181429356336594,
+            0.0049581872299313545,
+            0.027529491111636162,
+            -0.00839369185268879,
+            -0.012302667833864689,
+            0.02339324913918972,
+            0.011257244274020195,
+            0.0031703610438853502,
+            0.014651083387434483,
+            -0.012635990977287292,
+            -0.012446602806448936,
+            0.006655106786638498,
+            0.004848341923207045,
+            0.006049063988029957,
+            0.009151245467364788,
+            0.00450365524739027,
+            0.007333117071539164,
+            -0.007098275702446699,
+            -0.007427811156958342,
+            0.0014810169814154506,
+            0.01642375811934471,
+            0.020393338054418564,
+            -0.024968961253762245,
+            -0.0018806264270097017,
+            0.0009052763343788683,
+            0.010878467001020908,
+            -0.003316190093755722,
+            0.012310243211686611,
+            5.65205882594455e-05,
+            -0.0074240234680473804,
+            0.010158791206777096,
+            -0.02119634486734867,
+            -0.012681444175541401,
+            0.0012717428617179394,
+            0.013037494383752346,
+            -0.019590331241488457,
+            0.018226735293865204,
+            -0.008893677033483982,
+            -0.013348091393709183,
+            0.013893529772758484,
+            0.020014561712741852,
+            0.004090788308531046,
+            -0.007738408166915178,
+            -0.021862991154193878,
+            -0.01127239502966404,
+            0.016060132533311844,
+            0.020272130146622658,
+            0.000947888707742095,
+            -0.0006472347304224968,
+            0.010173942893743515,
+            -0.013113250024616718,
+            -0.023605365306138992,
+            -0.005230906419456005,
+            0.010386057198047638,
+            0.01746918261051178,
+            0.018832778558135033,
+            -0.006685409229248762,
+            0.02443867363035679,
+            -0.0059127043932676315,
+            -0.015499543398618698,
+            2.86597842205083e-05,
+            -0.02802947722375393,
+            -0.017423728480935097,
+            0.00013778003631159663,
+            0.00246773031540215,
+            0.008598231710493565,
+            0.0008555619278922677,
+            0.01943882182240486,
+            0.000904329412151128,
+            0.017590390518307686,
+            -0.000659071491099894,
+            -0.002009410411119461,
+            -0.0013995799235999584,
+            -0.0015340456739068031,
+            0.02254479005932808,
+            -0.012166308239102364,
+            -0.01176480483263731,
+            0.022302372381091118,
+            0.006852070800960064,
+            0.014719262719154358,
+            -0.027635548263788223,
+            0.013757170177996159,
+            -0.015499543398618698,
+            0.017423728480935097,
+            -0.03408990427851677,
+            0.006420265417546034,
+            0.01489350013434887,
+            -0.010348179377615452,
+            -0.020499395206570625,
+            -0.029650641605257988,
+            -0.024150803685188293,
+            -0.0038710979279130697,
+            -0.010855740867555141,
+            -0.016847988590598106,
+            0.006121031939983368,
+            -0.005802859086543322,
+            0.018499454483389854,
+            -0.007079336792230606,
+            0.033877789974212646,
+            0.006969491485506296,
+            -0.004056698642671108,
+            0.003835114184767008,
+            0.022469034418463707,
+            -0.006102093029767275,
+            0.02109028771519661,
+            0.004408961161971092,
+            -0.001467759720981121,
+            0.013681414537131786,
+            -0.006552837323397398,
+            -0.002361672930419445,
+            0.005821797996759415,
+            -0.010484539903700352,
+            0.022014502435922623,
+            0.005924067925661802,
+            -0.007708106189966202,
+            -0.0006775368237867951,
+            -0.011454207822680473,
+            0.014726839028298855,
+            0.006530110724270344,
+            0.0022480397019535303,
+            -0.011295121163129807,
+            0.011249667964875698,
+            -0.004859705455601215,
+            0.0040642740204930305,
+            0.009484568610787392,
+            -0.01774190180003643,
+            -0.028862785547971725,
+            0.011416330002248287,
+            0.019120648503303528,
+            0.006382387597113848,
+            -0.0055755930952727795,
+            -0.0025681061670184135,
+            0.00839369185268879,
+            0.010651201009750366,
+            0.022150862962007523,
+            -0.012984465807676315,
+            -8.889415767043829e-05,
+            0.013204156421124935,
+            -0.007170243188738823,
+            -0.01056029461324215,
+            0.00487485621124506,
+            0.023438703268766403,
+            0.0068823727779090405,
+            -0.04327145218849182,
+            -0.018469152972102165,
+            0.01909034699201584,
+            0.007136153522878885,
+            -0.04154422879219055,
+            0.0026381798088550568,
+            -0.003689285134896636,
+            -0.023787178099155426,
+            0.016105586662888527,
+            -0.0011817833874374628,
+            -0.01828734017908573,
+            0.028756728395819664,
+            0.02830219641327858,
+            0.01834794320166111,
+            2.6987840101355687e-05,
+            -0.013332940638065338,
+            -0.0318172425031662,
+            -0.023590214550495148,
+            0.008280059322714806,
+            -0.02660527639091015,
+            -0.0072725131176412106,
+            0.011242092587053776,
+            -0.017454031854867935,
+            0.02972639724612236,
+            -0.008507325313985348,
+            0.026938600465655327,
+            0.011446632444858551,
+            0.001691238023340702,
+            -0.013446573168039322,
+            -0.018802475184202194,
+            -0.01350717805325985,
+            -0.008196728304028511,
+            0.014090493321418762,
+            0.017923714593052864,
+            0.011007251217961311,
+            0.011332998983561993,
+            0.02251448854804039,
+            -0.012158732861280441,
+            0.007969462312757969,
+            -2.888172457460314e-05,
+            -0.008961857296526432,
+            0.0026343921199440956,
+            -0.018393397331237793,
+            0.00868156272917986,
+            0.0062308767810463905,
+            0.005458172410726547,
+            -0.007208121009171009,
+            0.010742107406258583,
+            0.02712041325867176,
+            -0.018847929313778877,
+            0.011401179246604443,
+            -0.031362712383270264,
+            -0.014787442982196808,
+            -0.01056029461324215,
+            0.026862844824790955,
+            0.007757347077131271,
+            -0.015726808458566666,
+            -0.007514929864555597,
+            -0.01545409020036459,
+            0.012885984033346176,
+            0.0032442226074635983,
+            0.005764981731772423,
+            0.015817714855074883,
+            0.021893294528126717,
+            0.01746918261051178,
+            -0.00795431062579155,
+            0.006852070800960064,
+            0.004674104508012533,
+            0.028090080246329308,
+            -0.01777220331132412,
+            0.020802417770028114,
+            0.009954252280294895,
+            -0.003935490269213915,
+            0.005685438401997089,
+            -0.014257155358791351,
+            0.008530051447451115,
+            -0.007325541693717241,
+            0.030514251440763474,
+            0.0033067206386476755,
+            0.014075342565774918,
+            -0.02062060497701168,
+            0.006575563922524452,
+            -0.03172633796930313,
+            0.009022461250424385,
+            0.013704141601920128,
+            0.012976890429854393,
+            -0.007677804213017225,
+            -0.024181105196475983,
+            -0.0119314668700099,
+            0.0057990713976323605,
+            -0.01279507763683796,
+            -0.01046181283891201,
+            -0.0034828518982976675,
+            0.012446602806448936,
+            0.018847929313778877,
+            -0.01848430372774601,
+            -0.019923655316233635,
+            -0.009583051316440105,
+            0.02898399345576763,
+            -0.015787413343787193,
+            -0.001471547526307404,
+            0.02413565292954445,
+            0.021241798996925354,
+            0.015022284351289272,
+            0.002155239460989833,
+            0.020090317353606224,
+            -0.0007111532613635063,
+            -0.0237265732139349,
+            -0.02813553437590599,
+            0.007302815094590187,
+            -0.006912675220519304,
+            0.01943882182240486,
+            -0.0035548193845897913,
+            -0.005658924113959074,
+            -0.0034033088013529778,
+            -0.007503566797822714,
+            -0.0165904201567173,
+            -0.01787826046347618,
+            0.0021647089160978794,
+            -0.008454295806586742,
+            0.005863463506102562,
+            -0.0002795845502987504,
+            -0.013105674646794796,
+            0.013401119969785213,
+            -0.014075342565774918,
+            0.005772557109594345,
+            -0.002653330797329545,
+            0.001962063368409872,
+            0.0035926972050219774,
+            0.009113367646932602,
+            -0.007336904760450125,
+            -0.06381630152463913,
+            -0.013105674646794796,
+            -0.003704436356201768,
+            0.006071790587157011,
+            -0.00019578021601773798,
+            -0.016635872423648834,
+            -0.013196581043303013,
+            0.01706010289490223,
+            0.008863375522196293,
+            -0.013870803639292717,
+            0.016954045742750168,
+            0.009961827658116817,
+            0.003622999181970954,
+            -0.024044746533036232,
+            -0.0008148433989845216,
+            0.008757317438721657,
+            0.032756611704826355,
+            -0.008643684908747673,
+            -0.0003839849086944014,
+            0.0036684523802250624,
+            -0.0033332351595163345,
+            -0.010052734054625034,
+            -0.014582903124392033,
+            -0.011848135851323605,
+            0.011908739805221558,
+            0.01790856383740902,
+            -0.014264730736613274,
+            -0.02007516659796238,
+            -0.0007106798002496362,
+            0.012931437231600285,
+            -0.026590125635266304,
+            0.0007244104635901749,
+            -0.007787649054080248,
+            -0.01052999310195446,
+            0.018302490934729576,
+            -0.019014591351151466,
+            0.005249845329672098,
+            0.00048294031876139343,
+            0.008507325313985348,
+            -0.023544760420918465,
+            -0.024893205612897873,
+            0.0035131541080772877,
+            -0.023847781121730804,
+            0.03612014651298523,
+            -0.021590271964669228,
+            -0.012082977220416069,
+            0.034483831375837326,
+            -0.0037423139438033104,
+            0.01702980138361454,
+            0.0023995505180209875,
+            0.02739313244819641,
+            -0.011439057067036629,
+            -0.00617784820497036,
+            0.012499631382524967,
+            0.009575475007295609,
+            0.0032593735959380865,
+            0.03775646165013313,
+            -0.01257538702338934,
+            -0.013901105150580406,
+            -0.005556654185056686,
+            -0.012954163365066051,
+            0.0006775368237867951,
+            0.014431392773985863,
+            0.028393102809786797,
+            0.004257450345903635,
+            -0.00628769351169467,
+            -0.006143758539110422,
+            -0.0014649189542979002,
+            -0.004242299124598503,
+            -0.00947699323296547,
+            -0.00715130427852273,
+            0.017287369817495346,
+            -0.025241680443286896,
+            0.016711628064513206,
+            -0.006420265417546034,
+            0.021135739982128143,
+            -0.011757229454815388,
+            0.007007369305938482,
+            -0.008961857296526432,
+            0.003492321353405714,
+            0.0023635667748749256,
+            -0.005109698045998812,
+            0.009287605062127113,
+            -0.014772292226552963,
+            0.008454295806586742,
+            0.013742019422352314,
+            0.0011183382011950016,
+            -0.010590597055852413,
+            -0.019272159785032272,
+            0.0022972808219492435,
+            -0.000710206339135766,
+            0.01181783340871334,
+            -0.0047498601488769054,
+            -0.000981031684204936,
+            0.0002921315317507833,
+            -0.014476845972239971,
+            -0.009575475007295609,
+            0.0006050958181731403,
+            0.005302873905748129,
+            -0.008961857296526432,
+            0.010378481820225716,
+            -0.005693014245480299,
+            -0.00034160923678427935,
+            0.02627195231616497,
+            0.006526322569698095,
+            -0.007632351014763117,
+            -0.014098069630563259,
+            0.01563590206205845,
+            0.015370759181678295,
+            -0.007454325910657644,
+            0.01933276280760765,
+            -0.016196493059396744,
+            -0.0008366230758838356,
+            0.013916256837546825,
+            -0.009219424799084663,
+            0.0017698341980576515,
+            -0.008196728304028511,
+            0.03554440662264824,
+            -0.0008787619881331921,
+            -0.0035415622405707836,
+            -0.0006401326390914619,
+            0.012976890429854393,
+            -0.0026779514737427235,
+            0.001182730309665203,
+            0.01132542360574007,
+            0.005321812815964222,
+            0.0185146052390337,
+            0.00328020635060966,
+            -0.03378688171505928,
+            -0.007780073676258326,
+            -0.0065490491688251495,
+            -0.008090670220553875,
+            0.009022461250424385,
+            0.025196226313710213,
+            0.01990850456058979,
+            0.006408901885151863,
+            0.001881573349237442,
+            0.018423698842525482,
+            0.015757111832499504,
+            -0.01489350013434887,
+            0.016363153234124184,
+            0.02802947722375393,
+            -0.010030006989836693,
+            0.022741753607988358,
+            -0.005253633018583059,
+            -0.008893677033483982,
+            0.008946705609560013,
+            -0.0015378334792330861,
+            -0.017393426969647408,
+            -0.015681356191635132,
+            0.005022579338401556,
+            -0.008999735116958618,
+            -0.0022897052112966776,
+            -0.018605511635541916,
+            0.004670316819101572,
+            0.017105557024478912,
+            -0.004965762607753277,
+            -3.5243989259470254e-05,
+            0.019484274089336395,
+            -0.006965703796595335,
+            -0.016908591613173485,
+            0.006325571332126856,
+            -0.015007133595645428,
+            0.007946735247969627,
+            -0.013666263781487942,
+            0.010492115281522274,
+            -0.006098304875195026,
+            0.018817627802491188,
+            0.005446809343993664,
+            -0.008090670220553875,
+            -0.0008958069374784827,
+            -0.018984289839863777,
+            0.02065090648829937,
+            -0.033089932054281235,
+            0.00839369185268879,
+            -0.026226500049233437,
+            -0.0009781909175217152,
+            0.0008849170990288258,
+            0.014605630189180374,
+            -0.001218714052811265,
+            0.019696388393640518,
+            -0.002751812804490328,
+            -0.020090317353606224,
+            -0.007086912170052528,
+            -0.0010634156642481685,
+            -0.01618134044110775,
+            -0.030923331156373024,
+            -0.004272601567208767,
+            0.0036267871037125587,
+            -0.004711982328444719,
+            0.0021135739516466856,
+            0.017984317615628242,
+            -0.005613470915704966,
+            0.009173971600830555,
+            -0.02690829709172249,
+            -0.007416448090225458,
+            -0.00355103169567883,
+            0.0010776197304949164,
+            -0.011242092587053776,
+            0.0010596278589218855,
+            0.011226941831409931,
+            -0.019044892862439156,
+            0.013461724855005741,
+            -0.021378157660365105,
+            -0.004371083341538906,
+            -0.008507325313985348,
+            -0.00135412672534585,
+            -0.013446573168039322,
+            -0.028120383620262146,
+            0.025029566138982773,
+            -0.00021495578403118998,
+            -0.003528305096551776,
+            -0.011999646201729774,
+            0.023181134834885597,
+            0.002914686920121312,
+            0.027817361056804657,
+            -0.03157482668757439,
+            0.0013427634257823229,
+            0.011552689597010612,
+            0.0025984083767980337,
+            -0.017287369817495346,
+            0.004746072459965944,
+            -0.005234694108366966,
+            0.02210540883243084,
+            -0.0011694730492308736,
+            -0.0005340751959010959,
+            -0.008234606124460697,
+            0.030999086797237396,
+            0.009136094711720943,
+            0.0012944694608449936,
+            -0.013226882554590702,
+            -0.008060368709266186,
+            -0.002759388415142894,
+            -0.0011069749016314745,
+            0.012855681590735912,
+            0.01290113478899002,
+            -0.012105703353881836,
+            0.0013607552973553538,
+            -0.006878585088998079,
+            0.0011694730492308736,
+            -0.002935519441962242,
+            -0.0004966709529981017,
+            -0.01149208564311266,
+            0.003257479751482606,
+            0.016272246837615967,
+            -0.00780280027538538,
+            -0.006480869371443987,
+            0.0007750718505121768,
+            0.009181547909975052,
+            0.012461753562092781,
+            -0.012742049060761929,
+            -0.018226735293865204,
+            -0.011908739805221558,
+            0.016029831022024155,
+            0.010522416792809963,
+            -0.008204303681850433,
+            -0.013234458863735199,
+            -0.02583257108926773,
+            0.01169662456959486,
+            -0.03293842449784279,
+            -0.012242063879966736,
+            0.005821797996759415,
+            0.028559764847159386,
+            0.019999410957098007,
+            0.012226912193000317,
+            0.001332347048446536,
+            0.002950670663267374,
+            0.00023342114582192153,
+            -0.01916610263288021,
+            0.014408666640520096,
+            -0.011143610812723637,
+            -0.018166130408644676,
+            0.0075906855054199696,
+            0.011939042247831821,
+            0.03148392215371132,
+            0.006942977197468281,
+            -0.0010160685051232576,
+            0.01686313934624195,
+            -0.001218714052811265,
+            -0.013582932762801647,
+            0.01420412678271532,
+            -0.024256860837340355,
+            0.016438908874988556,
+            0.02860521711409092,
+            -0.004780162125825882,
+            0.0008531945641152561,
+            -0.020938776433467865,
+            0.00482561532407999,
+            -0.011529963463544846,
+            -0.002177966060116887,
+            -0.003287781961262226,
+            -0.03460504114627838,
+            0.007946735247969627,
+            -0.006825556512922049,
+            0.01046181283891201,
+            -0.00814369972795248,
+            0.006655106786638498,
+            -0.012431452050805092,
+            0.004253662656992674,
+            0.01513591781258583,
+            0.016105586662888527,
+            0.007060397882014513,
+            -0.025211378931999207,
+            -0.009196698665618896,
+            0.008105821907520294,
+            -0.01349202636629343,
+            -0.02295386977493763,
+            -0.006852070800960064,
+            0.009840618818998337,
+            -0.01061332318931818,
+            -0.00488243205472827,
+            -0.028180986642837524,
+            -0.013681414537131786,
+            -0.0014327229000627995,
+            -0.017817657440900803,
+            -0.005556654185056686,
+            -0.003100287402048707,
+            0.01571165770292282,
+            -0.0049733384512364864,
+            -0.006185423582792282,
+            0.004742284305393696,
+            -0.023878084495663643,
+            0.0037423139438033104,
+            -0.008280059322714806,
+            0.0008266801596619189,
+            -0.0032517979852855206,
+            -0.0014762822538614273,
+            -0.007969462312757969,
+            -0.014332910999655724,
+            0.006893736310303211,
+            -0.016469212248921394,
+            0.002954458352178335,
+            -0.015173795633018017,
+            -0.016499513760209084,
+            -0.00682934420183301,
+            0.00775355938822031,
+            0.021575121209025383,
+            0.006541473791003227,
+            0.021484214812517166,
+            0.005469535943120718,
+            0.028680972754955292,
+            0.0003953482082579285,
+            0.003041577059775591,
+            -8.958956641436089e-06,
+            0.003232859307900071,
+            0.019347915425896645,
+            0.006571775767952204,
+            0.018984289839863777,
+            -0.008742166683077812,
+            0.010514841414988041,
+            -0.010355755686759949,
+            0.010014856234192848,
+            0.006333146709948778,
+            -0.006473293993622065,
+            -0.003799130441620946,
+            -0.010348179377615452,
+            -0.0031419529113918543,
+            -0.016544966027140617,
+            0.0027366618160158396,
+            0.00693161366507411,
+            0.0071588801220059395,
+            -0.011582992039620876,
+            -0.005230906419456005,
+            -0.014704111963510513,
+            -0.0064354161731898785,
+            -0.01831764169037342,
+            0.014385939575731754,
+            0.029226411134004593,
+            -0.016332851722836494,
+            -0.01977214403450489,
+            -0.007772498298436403,
+            0.0007296186522580683,
+            -0.006636167876422405,
+            -0.01240872498601675,
+            -0.009151245467364788,
+            -0.035059571266174316,
+            -0.017666146159172058,
+            0.007586897816509008,
+            0.02484775334596634,
+            0.01387837901711464,
+            -0.013628385961055756,
+            -0.0006983695784583688,
+            -0.016984347254037857,
+            -0.010787560604512691,
+            -0.01535560842603445,
+            -0.02054484933614731,
+            -0.02498411200940609,
+            0.0077952248975634575,
+            -0.004033972043544054,
+            0.014931377954781055,
+            0.008113397285342216,
+            0.0032631612848490477,
+            -0.004371083341538906,
+            -0.0024109138175845146,
+            -0.013196581043303013,
+            0.0012925754999741912,
+            -0.004787737503647804,
+            -0.0017291157273575664,
+            0.03554440662264824,
+            -0.008098246529698372,
+            -0.005871038883924484,
+            0.011060279794037342,
+            0.011128460057079792,
+            -0.013764745555818081,
+            0.0006642796797677875,
+            -0.011249667964875698,
+            0.021847840398550034,
+            -0.021620575338602066,
+            0.019257009029388428,
+            0.0014507147716358304,
+            -0.010840589180588722,
+            0.028999146074056625,
+            0.01892368495464325,
+            0.00743159931153059,
+            0.006579351611435413,
+            0.01672677882015705,
+            0.0044316877610981464,
+            0.016196493059396744,
+            -0.0026552248746156693,
+            -0.004632438998669386,
+            0.009295180439949036,
+            0.015681356191635132,
+            -0.000702630786690861,
+            -0.006942977197468281,
+            0.016438908874988556,
+            0.004056698642671108,
+            0.0075414446182549,
+            -0.01356020662933588,
+            -0.008961857296526432,
+            0.001507531269453466,
+            0.006257391534745693,
+            -0.005939218681305647,
+            -0.0026003022212535143,
+            0.005530139897018671,
+            0.003943065647035837,
+            -0.013302638195455074,
+            0.01689344085752964,
+            -0.016166189685463905,
+            0.008742166683077812,
+            -0.0037972365971654654,
+            0.0121360057964921,
+            -0.006799042224884033,
+            0.0013153020991012454,
+            -0.01818128302693367,
+            -0.003869204083457589,
+            0.0038597346283495426,
+            0.012173883616924286,
+            -0.01262841559946537,
+            0.025484098121523857,
+            0.008166425861418247,
+            0.006602078210562468,
+            0.0059467945247888565,
+            -0.012855681590735912,
+            0.027211319655179977,
+            0.005969521123915911,
+            0.009886072017252445,
+            -0.008893677033483982,
+            0.011136035434901714,
+            0.035362593829631805,
+            0.03060515783727169,
+            0.014135947450995445,
+            0.004469565115869045,
+            -0.018999440595507622,
+            0.015120766125619411,
+            0.0018427487229928374,
+            0.0023806116078048944,
+            0.00863610953092575,
+            -0.011348150670528412,
+            0.01305264513939619,
+            -0.009484568610787392,
+            0.015241974964737892,
+            -0.021272100508213043,
+            -0.011749654076993465,
+            -0.014408666640520096,
+            -0.021075136959552765,
+            -0.01430260855704546,
+            0.034180812537670135,
+            -0.002346521709114313,
+            -0.010393633507192135,
+            0.0045112306252121925,
+            0.008477022871375084,
+            -0.006079366430640221,
+            0.0053142374381423,
+            -0.008204303681850433,
+            -0.013749594800174236,
+            -0.01889338344335556,
+            0.006628592498600483,
+            -0.01550711877644062,
+            0.007651289459317923,
+            -0.003304826794192195,
+            0.0037688282318413258,
+            0.007014944683760405,
+            0.018105527386069298,
+            0.00530666159465909,
+            -0.00135412672534585,
+            0.0014033677289262414,
+            0.0016865032957866788,
+            -0.00931033119559288,
+            0.0108330138027668,
+            0.015408637002110481,
+            -0.0011921997647732496,
+            -0.011279970407485962,
+            -0.009136094711720943,
+            -0.042877525091171265,
+            0.0006264019757509232,
+            -0.013128400780260563,
+            0.0064354161731898785,
+            -0.011196639388799667,
+            0.0022234192583709955,
+            0.015863168984651566,
+            -0.0054051438346505165,
+            -0.0010918239131569862,
+            0.02193874679505825,
+            0.013014768250286579,
+            0.021075136959552765,
+            0.02065090648829937,
+            0.014188976027071476,
+            -0.015302578918635845,
+            -0.03469594568014145,
+            0.002221525413915515,
+            -0.010030006989836693,
+            0.013772320933640003,
+            0.004238511435687542,
+            0.016060132533311844,
+            -0.023893235251307487,
+            -0.011878437362611294,
+            -0.006662682630121708,
+            0.014431392773985863,
+            0.01491622719913721,
+            -0.0032271775417029858,
+            0.02690829709172249,
+            0.009280029684305191,
+            0.0020510759204626083,
+            -0.02281750924885273,
+            0.003312402404844761,
+            0.005321812815964222,
+            -0.008370965719223022,
+            -0.016681326553225517,
+            -0.0004552422615233809,
+            -0.0012168202083557844,
+            -0.0007817004225216806,
+            -0.021650876849889755,
+            0.018832778558135033,
+            -0.004018820822238922,
+            0.0037574649322777987,
+            0.013840501196682453,
+            0.01516621932387352,
+            0.017984317615628242,
+            0.008370965719223022,
+            0.001497115008533001,
+            -0.006605865899473429,
+            -0.0007973249885253608,
+            -0.0005828426801599562,
+            -0.007389933802187443,
+            -0.005598319694399834,
+            0.006382387597113848,
+            -0.01134057529270649,
+            0.008810346014797688,
+            0.000896753859706223,
+            0.006113456096500158,
+            0.010605747811496258,
+            -0.004836978856474161,
+            -0.011628445237874985,
+            0.014628356322646141,
+            -0.0010548931313678622,
+            -0.0055755930952727795,
+            -0.011060279794037342,
+            0.003624893259257078,
+            0.013575357384979725,
+            -0.003096499713137746,
+            -0.013499601744115353,
+            0.00910579226911068,
+            0.002009410411119461,
+            -0.027438584715127945,
+            0.015499543398618698,
+            0.010295150801539421,
+            -0.013370818458497524,
+            0.004980913829058409,
+            -0.003371112747117877,
+            -0.03027183562517166,
+            0.004791525658220053,
+            0.002660906407982111,
+            -0.010643625631928444,
+            0.028938541188836098,
+            -0.003821857040748,
+            0.029256712645292282,
+            -0.004980913829058409,
+            -0.006905099377036095,
+            -0.011355726048350334,
+            0.009416389279067516,
+            -0.018681267276406288,
+            -0.0037953427527099848,
+            -0.008727015927433968,
+            -0.00915882084518671,
+            -0.0009838725673034787,
+            -0.005185453221201897,
+            0.006984642706811428,
+            0.005121061112731695,
+            -0.012704171240329742,
+            0.006757376715540886,
+            -0.010257272981107235,
+            -0.00020856392802670598,
+            0.0006718551740050316,
+            -0.008408842608332634,
+            0.013885954394936562,
+            0.005996035411953926,
+            -0.008340663276612759,
+            -0.0005032995832152665,
+            0.027817361056804657,
+            0.05051366239786148,
+            -0.018863080069422722,
+            0.0009270560112781823,
+            0.01452229917049408,
+            -0.0028824908658862114,
+            -0.0021590273827314377,
+            -0.011976920068264008,
+            0.005821797996759415,
+            -0.019423669204115868,
+            0.013317788951098919,
+            0.006473293993622065,
+            -0.008833073079586029,
+            0.003202557098120451,
+            -0.016711628064513206,
+            0.002621134975925088,
+            -0.0018058179412037134,
+            -0.018681267276406288,
+            -0.015529844909906387,
+            -0.012060250155627728,
+            -0.024181105196475983,
+            0.0004235197266098112,
+            -0.0008546149474568665,
+            -0.013870803639292717,
+            0.006579351611435413,
+            0.008431569673120975,
+            0.032483890652656555,
+            0.0011912527261301875,
+            0.010128489695489407,
+            -2.1513333194889128e-05,
+            -0.0010416359873488545,
+            -0.005988460034132004,
+            -0.01523439958691597,
+            0.00015387803432531655,
+            -0.002342734020203352,
+            0.022590242326259613,
+            0.005549078807234764,
+            0.017560089007019997,
+            0.0017887729918584228,
+            -0.016514664515852928,
+            -0.00596573343500495,
+            -0.007408872712403536,
+            -0.018060073256492615,
+            0.0062649669125676155,
+            -0.005473323632031679,
+            0.009605777449905872,
+            -0.015863168984651566,
+            -0.02731737680733204,
+            0.00792400911450386,
+            0.012234487570822239,
+            0.028514310717582703,
+            -0.0035623949952423573,
+            0.001716805505566299,
+            -0.005533927585929632,
+            0.016408607363700867,
+            0.032180868089199066,
+            -0.0008550884085707366,
+            0.006950552575290203,
+            -0.019999410957098007,
+            0.0017404790269210935,
+            0.013704141601920128,
+            -0.008401267230510712,
+            -0.01821158453822136,
+            0.003939277958124876,
+            -0.020635755732655525,
+            0.004583198111504316,
+            -0.020393338054418564,
+            0.030408194288611412,
+            -0.0023522034753113985,
+            0.008105821907520294,
+            0.004310478921979666,
+            0.01760554127395153,
+            -0.0031646795105189085,
+            0.010545143857598305,
+            0.013916256837546825,
+            0.014908651821315289,
+            0.0009980766335502267,
+            -0.011189064010977745,
+            -0.03790797293186188,
+            0.001859793672338128,
+            0.00817400123924017,
+            0.018635815009474754,
+            -0.01980244740843773,
+            0.011681473813951015,
+            -0.01296931505203247,
+            -0.020363036543130875,
+            -0.01003758329898119,
+            0.011575416661798954,
+            -0.006139970384538174,
+            0.03378688171505928,
+            -0.005874827038496733,
+            -0.008204303681850433,
+            0.005136212334036827,
+            0.010454237461090088,
+            0.0066778333857655525,
+            0.014704111963510513,
+            0.0326354019343853,
+            0.010196669027209282,
+            0.014173824340105057,
+            -0.004859705455601215,
+            0.02399929240345955,
+            -0.003931702114641666,
+            0.003954428713768721,
+            0.00509454682469368,
+            -0.022832660004496574,
+            -0.0040983641520142555,
+            -0.003931702114641666,
+            -0.016711628064513206,
+            -0.007458113599568605,
+            0.0032366469968110323,
+            0.008802770636975765,
+            -0.014045040123164654,
+            -0.00649980828166008,
+            -0.01350717805325985,
+            0.01218903437256813,
+            0.002730980049818754,
+            0.018302490934729576,
+            -0.029802151024341583,
+            0.009825468063354492,
+            -0.017484333366155624,
+            -0.017044952139258385,
+            0.006511171814054251,
+            -0.0038332203403115273,
+            0.006855858489871025,
+            -0.018272189423441887,
+            -0.01318900566548109,
+            0.012817803770303726,
+            0.0007897494360804558,
+            0.013060221448540688,
+            -0.009825468063354492,
+            0.0009706153068691492,
+            0.006064215209335089,
+            -0.009833043441176414,
+            -0.010961798019707203,
+            -0.007977037690579891,
+            -0.009681533090770245,
+            -0.008848223835229874,
+            0.005621046293526888,
+            0.01656011864542961,
+            -0.013742019422352314,
+            -0.003685497445985675,
+            0.016135888174176216,
+            0.006416477728635073,
+            -0.008166425861418247,
+            0.004075637552887201,
+            -0.003793448908254504,
+            -9.948806837201118e-05,
+            0.0013740125577896833,
+            0.012439027428627014,
+            -0.0017632056260481477,
+            0.024665940552949905,
+            -0.02654467150568962,
+            0.007598260883241892,
+            -0.0054051438346505165,
+            0.005446809343993664,
+            0.018226735293865204,
+            -0.02366596832871437,
+            -0.013082947582006454,
+            0.006276329979300499,
+            -0.01966608688235283,
+            -0.008060368709266186,
+            0.0054051438346505165,
+            -0.00016405766655225307,
+            -0.019923655316233635,
+            -0.012923861853778362,
+            -0.034786853939294815,
+            -0.02309022843837738,
+            0.024665940552949905,
+            0.008401267230510712,
+            -0.0073672072030603886,
+            0.018060073256492615,
+            -0.018438849598169327,
+            0.0172722190618515,
+            -0.002225313102826476,
+            -0.006613441277295351,
+            0.003935490269213915,
+            0.007674016058444977,
+            0.025362888351082802,
+            0.002653330797329545,
+            0.008264907635748386,
+            -0.0042082094587385654,
+            -0.014931377954781055,
+            -0.006321783177554607,
+            0.01980244740843773,
+            0.02471139281988144,
+            0.018605511635541916,
+            -0.018575210124254227,
+            0.010742107406258583,
+            -0.011749654076993465,
+            0.01818128302693367,
+            -0.006537686102092266,
+            0.00039037675014697015,
+            0.01859036087989807,
+            0.007969462312757969,
+            0.025241680443286896,
+            0.019923655316233635,
+            0.006852070800960064,
+            0.014128371141850948,
+            0.011908739805221558,
+            0.001204509986564517,
+            -0.012832955457270145,
+            -0.01154511421918869,
+            -0.02840825356543064,
+            0.042847223579883575,
+            0.0025434857234358788,
+            0.008317936211824417,
+            -0.027635548263788223,
+            -0.00021389048197306693,
+            -0.0034942151978611946,
+            -0.018832778558135033,
+            0.005939218681305647,
+            0.03260510042309761,
+            0.03284751623868942,
+            0.021438762545585632,
+            -2.888172457460314e-05,
+            0.0012234487803652883,
+            0.0329081192612648,
+            -0.013673839159309864,
+            -0.00937851145863533,
+            -0.01034060399979353,
+            -0.018393397331237793,
+            -0.006852070800960064,
+            -0.0032423287630081177,
+            -0.01031030248850584,
+            -0.0061172437854111195,
+            -0.009272454306483269,
+            0.0238629337400198,
+            0.0001374249259242788,
+            -0.0022594030015170574,
+            0.006605865899473429,
+            -0.020878173410892487,
+            0.011537538841366768,
+            -0.02813553437590599,
+            -0.013431422412395477,
+            -0.008651260286569595,
+            -0.006950552575290203,
+            -0.018802475184202194,
+            0.008643684908747673,
+            -0.006942977197468281,
+            0.021438762545585632,
+            0.002914686920121312,
+            -0.023590214550495148,
+            0.007530081085860729,
+            0.015317730605602264,
+            -0.0003825644962489605,
+            0.014037464745342731,
+            -0.003890036838129163,
+            0.009060339070856571,
+            0.0045112306252121925,
+            0.011681473813951015,
+            0.016954045742750168,
+            0.010651201009750366,
+            0.009030036628246307,
+            0.003437398700043559,
+            -0.001167579204775393,
+            0.01878732442855835,
+            -0.0007163614500313997,
+            -0.03405960276722908,
+            2.9236827685963362e-05,
+            -0.014628356322646141,
+            0.029499130323529243,
+            -0.009795165620744228,
+            0.009514871053397655,
+            0.008105821907520294,
+            0.007098275702446699,
+            -0.0067119235172867775,
+            0.007916433736681938,
+            -0.008742166683077812,
+            0.0052763596177101135,
+            0.015832867473363876,
+            -0.0042877523228526115,
+            -0.0014980619307607412,
+            0.007840678095817566,
+            0.00407184986397624,
+            -0.005454384721815586,
+            -0.019711541011929512,
+            -0.012287517078220844,
+            0.00726114958524704,
+            -0.02065090648829937,
+            -0.004367295652627945,
+            0.01161329448223114,
+            -0.005454384721815586,
+            -0.009620928205549717,
+            0.01393140759319067,
+            0.02136300690472126,
+            -0.002382505452260375,
+            0.022893264889717102,
+            -0.01349202636629343,
+            -0.009833043441176414,
+            -0.007984613068401814,
+            -0.004374871030449867,
+            -0.041120000183582306,
+            0.025984082370996475,
+            0.01068150345236063,
+            -0.006871009711176157,
+            0.01895398646593094,
+            0.0022499337792396545,
+            -0.002954458352178335,
+            -0.0011041341349482536,
+            0.00010499217023607343,
+            -0.03281721472740173,
+            0.03660498186945915,
+            -0.007007369305938482,
+            0.0005449649761430919,
+            0.010272424668073654,
+            0.042029064148664474,
+            -0.004590773954987526,
+            -0.005859675817191601,
+            0.01225721463561058,
+            0.005336964037269354,
+            0.031938452273607254,
+            -0.010416359640657902,
+            -0.014340486377477646,
+            -0.01815097965300083,
+            -0.004833190701901913,
+            -0.01939336769282818,
+            -0.0038313264958560467,
+            0.014037464745342731,
+            -0.019650936126708984,
+            -0.011242092587053776,
+            -0.001287840772420168,
+            -0.012537509202957153,
+            -0.0010747789638116956,
+            0.008598231710493565,
+            0.017832808196544647,
+            -0.011348150670528412,
+            -0.004450626205652952,
+            -2.2904154320713133e-05,
+            -0.005003640428185463,
+            0.0185146052390337,
+            0.0020056227222085,
+            -0.01702980138361454,
+            0.01321930717676878,
+            -0.00526499655097723,
+            0.006749800872057676,
+            0.0074429623782634735,
+            0.007939159870147705,
+            0.05145302787423134,
+            0.010317877866327763,
+            -0.001339922659099102,
+            -0.01607528328895569,
+            0.023196285590529442,
+            -0.01749948412179947,
+            -0.014181400649249554,
+            -0.02119634486734867,
+            -0.0029923359397798777,
+            -0.007060397882014513,
+            -0.028014326468110085,
+            -9.137987944995984e-05,
+            0.0016221113037317991,
+            -0.0024980325251817703,
+            -0.010045158676803112,
+            0.0005208179936744273,
+            -0.011529963463544846,
+            -0.006954340264201164,
+            -0.008545203134417534,
+            -0.016044981777668,
+            0.02240842953324318,
+            -0.021817538887262344,
+            -0.008855799213051796,
+            0.011514811776578426,
+            0.03118089959025383,
+            -0.005011215806007385,
+            -0.014173824340105057,
+            -0.019817598164081573,
+            0.0025643182452768087,
+            -0.01763584464788437,
+            0.012181458994746208,
+            -0.01225721463561058,
+            0.015045011416077614,
+            -0.03142331540584564,
+            -0.006079366430640221,
+            -0.008992158807814121,
+            0.012840530835092068,
+            0.019817598164081573,
+            -0.0025093958247452974,
+            0.00606800289824605,
+            -0.010295150801539421,
+            0.030211230739951134,
+            -0.00012286569108255208,
+            -0.018605511635541916,
+            0.00873459130525589,
+            0.007977037690579891,
+            0.012537509202957153,
+            -0.004435475450009108,
+            0.00257757562212646,
+            0.011567841283977032,
+            0.012742049060761929,
+            -0.01634800247848034,
+            -0.003174148965626955,
+            -0.017969166859984398,
+            0.005996035411953926,
+            0.006999793462455273,
+            0.021711481735110283,
+            -0.0013550736475735903,
+            0.018272189423441887,
+            -0.01563590206205845,
+            0.00019412307301536202,
+            -0.011749654076993465,
+            -0.0082270298153162,
+            -0.0028237802907824516,
+            -0.003958216868340969,
+            0.006685409229248762,
+            -0.028211290016770363,
+            -0.01274962443858385,
+            0.005287723150104284,
+            -0.004992276895791292,
+            -0.03405960276722908,
+            0.0053710537031292915,
+            -0.02725677192211151,
+            0.013128400780260563,
+            -0.002397656673565507,
+            -0.017847958952188492,
+            -0.00030846628942526877,
+            0.026332557201385498,
+            -0.003677921835333109,
+            -0.012537509202957153,
+            0.015544996596872807,
+            -0.006757376715540886,
+            -0.02119634486734867,
+            -0.005420294590294361,
+            0.014658658765256405,
+            -0.0010719380807131529,
+            -0.012393574230372906,
+            -0.029635490849614143,
+            0.012992041185498238,
+            0.0026874206960201263,
+            -0.007386146113276482,
+            0.010893618687987328,
+            0.025953780859708786,
+            0.008151275105774403,
+            -0.005924067925661802,
+            0.0015757111832499504,
+            0.0049581872299313545,
+            0.00300938100554049,
+            0.020287280902266502,
+            0.02728707529604435,
+            -0.008014915511012077,
+            -0.0011145505122840405,
+            -0.014954105019569397,
+            0.015802564099431038,
+            0.0013020449550822377,
+            0.010507266037166119,
+            0.021893294528126717,
+            0.007045246660709381,
+            -0.005738467443734407,
+            0.003905187826603651,
+            -0.023135682567954063,
+            0.01420412678271532,
+            -0.012272365391254425,
+            0.000622140767518431,
+            0.008749742060899734,
+            -0.007283876184374094,
+            0.01437078882008791,
+            -0.0015245762187987566,
+            0.0339989997446537,
+            0.025211378931999207,
+            0.0043218424543738365,
+            0.0007362472242675722,
+            -0.028559764847159386,
+            0.003547244006767869,
+            0.008651260286569595,
+            0.014499573037028313,
+            -0.0076664406806230545,
+            0.006261179223656654,
+            -0.004355932120233774,
+            0.00959820207208395,
+            -0.0065490491688251495,
+            -0.015893470495939255,
+            0.006113456096500158,
+            0.0012689019786193967,
+            0.006795254070311785,
+            0.0023408401757478714,
+            -0.008863375522196293,
+            -0.024362917989492416,
+            -0.025984082370996475,
+            0.021984200924634933,
+            -0.03918066248297691,
+            0.0073293293826282024,
+            0.011234517209231853,
+            0.03324144333600998,
+            -0.008158850483596325,
+            -0.004980913829058409,
+            0.006276329979300499,
+            -0.016711628064513206,
+            -0.004147605039179325,
+            0.008204303681850433,
+            0.012385998852550983,
+            -0.002420383272692561,
+            0.00974971242249012,
+            0.01318900566548109,
+            -0.012378423474729061,
+            0.03660498186945915,
+            0.016514664515852928,
+            0.01467380952090025,
+            -0.004007457755506039,
+            0.01312082540243864,
+            0.0066172294318675995,
+            0.019044892862439156,
+            0.025165924802422523,
+            -0.017454031854867935,
+            -0.002365460619330406,
+            -0.02292356640100479,
+            0.024165954440832138,
+            0.00726114958524704,
+            0.004556683823466301,
+            -0.008310360834002495,
+            0.014870774000883102,
+            0.0017954016802832484,
+            0.0028199926018714905,
+            -0.006696772295981646,
+            -0.010173942893743515,
+            0.019241856411099434,
+            0.0034317169338464737,
+            0.014870774000883102,
+            4.8501173296244815e-05,
+            -0.005503625608980656,
+            0.006052851676940918,
+            0.014832896180450916,
+            -0.023590214550495148,
+            -0.00203592493198812,
+            -0.019484274089336395,
+            -0.03202936053276062,
+            0.009022461250424385,
+            0.0016135887708514929,
+            -0.007052822504192591,
+            0.03230207785964012,
+            -0.010083036497235298,
+            -0.019787294790148735,
+            -0.008067944087088108,
+            -0.013113250024616718,
+            -0.010469388216733932,
+            0.0008271536789834499,
+            0.003908975515514612,
+            -0.00012428610352799296,
+            -0.009317907504737377,
+            -0.005749830510467291,
+            0.01990850456058979,
+            -0.0030358952935785055,
+            -0.00753386877477169,
+            0.00868156272917986
+          ],
+          "embedding": [
+            0.0040707582550429606,
+            -0.010230625151989194,
+            -0.021520242101903186,
+            -0.010115756270640508,
+            0.026738734586159035,
+            0.01796370233223267,
+            -0.012245199029994153,
+            0.06506924494147726,
+            -0.0014087587222033442,
+            0.009300068719271429,
+            0.003787396973047584,
+            0.002818772639754174,
+            0.022129077062176306,
+            -0.021526709808445683,
+            -0.0034388135367225824,
+            0.02763761747879405,
+            0.013210013805395276,
+            0.025255872672296602,
+            0.00029011480909986703,
+            -0.05438974762290585,
+            0.016671640187490865,
+            -0.016426252259893328,
+            -0.056591180651012554,
+            0.009335648425523075,
+            0.006141624459418827,
+            0.02568943956694641,
+            0.019377132054042116,
+            0.017946464337078746,
+            -0.057184776985927686,
+            0.03208527139096687,
+            0.030132392063755547,
+            -0.003846032371791839,
+            -0.008376524427531415,
+            -0.0010513963058713435,
+            -0.02734868388724371,
+            -0.014879400370545479,
+            0.04383920959748825,
+            0.02563613659516289,
+            -0.020117467074438997,
+            0.0398016699155716,
+            0.029896497884197775,
+            -0.0004195844530067313,
+            -0.023302720788253625,
+            -0.001168337916577517,
+            0.009149021369661628,
+            -0.02508145721900992,
+            -0.004307594138480351,
+            0.00550940415564303,
+            0.014016998918718476,
+            -0.028607379616884317,
+            -0.0018317656587526954,
+            -0.01973464940415756,
+            0.03762738392056255,
+            0.015358106058583114,
+            -0.03931907682104936,
+            0.011519637298832444,
+            0.004482080558226775,
+            0.0747867063383242,
+            0.05550679018263077,
+            0.023243129601931176,
+            0.02671631686665238,
+            -0.009686146566573536,
+            -0.04088582189725495,
+            -0.0065344623594924085,
+            0.019617295999325712,
+            -0.004722087402575777,
+            -0.002871844114437583,
+            0.003935087265300519,
+            0.03229890413393201,
+            -0.02813577047904649,
+            -0.05124172132597234,
+            0.03325235407841968,
+            -0.008687655557217192,
+            -0.011027174838246948,
+            -0.003653553063547375,
+            -0.02350737945182988,
+            0.016223087939183468,
+            0.03789252356462714,
+            0.0034350623378603083,
+            0.00195158657888739,
+            0.011789712862232108,
+            -0.0004435439660769551,
+            -0.0037330456918125894,
+            0.0031552713243866858,
+            0.04465432992298031,
+            -0.0032465421571915712,
+            -0.0075366743861922745,
+            -0.059958752731410256,
+            0.016784616861008125,
+            -0.028734506065772967,
+            -0.03035575736273943,
+            0.02680390892611741,
+            0.02338368719038086,
+            -0.0030658375437784277,
+            -0.03754896050794851,
+            0.07005659523248504,
+            0.0020711575553570695,
+            0.013034425083076923,
+            -0.02316832262132332,
+            0.021332142030732894,
+            -0.0318756233020585,
+            0.02270782509507388,
+            0.01570810309454529,
+            0.004788124787612736,
+            0.026727250200142456,
+            -0.022380142778058276,
+            -0.008593910379436904,
+            0.014439754382387034,
+            0.015007697617383174,
+            0.07662531963612977,
+            0.011195998137723795,
+            -0.010809283538470098,
+            -0.042227822120219305,
+            0.003824320933275175,
+            -0.03797629427607094,
+            -0.02347390684453348,
+            -0.0526876440280276,
+            -0.017799861680287913,
+            0.01701482796266917,
+            0.002730395144917155,
+            0.03789451759899825,
+            0.038338545350313946,
+            0.028463789278406586,
+            0.015494583705011921,
+            -0.05700237266350479,
+            0.047895170359075166,
+            -0.0064406894534929406,
+            -0.0007954721295065988,
+            0.012543029839729358,
+            -0.004703334868741216,
+            -0.0060569679284068,
+            0.012324901254711192,
+            -0.019883935809784068,
+            -0.04253565942154215,
+            0.02164409276211115,
+            -0.0075367555415808635,
+            0.03617159147722157,
+            0.07567745988686005,
+            -0.04225977020036783,
+            -0.022548837589622003,
+            0.029811602145850144,
+            -0.02175792196819533,
+            0.02069861458115363,
+            -0.00941331829664397,
+            -0.035479639885943264,
+            0.020352552580119105,
+            0.003678588588550461,
+            0.06776105274336729,
+            -0.02346047721661687,
+            0.02721075076349443,
+            -0.029386781707644253,
+            0.023530831921117993,
+            -0.02077660533100372,
+            0.020022475323372906,
+            0.024195118676513715,
+            -0.028378518720494666,
+            -0.012900969023913307,
+            0.019264089970719624,
+            -0.0374983624629265,
+            -0.019726171557384355,
+            0.00979636085642416,
+            -0.014479528435832063,
+            -0.014533384583777662,
+            0.005904623758767224,
+            0.010344475500263744,
+            0.05659646358026115,
+            0.04163866113482093,
+            0.007152494819169405,
+            -0.07584500130974763,
+            0.05471858597878099,
+            0.03580211013427615,
+            0.044161429444475306,
+            0.02258180566046248,
+            0.00980779221334715,
+            0.0011193505603397395,
+            0.018086566768892507,
+            0.015605066702191871,
+            -0.00954727399396313,
+            -0.027950079146919375,
+            0.03028514025485407,
+            -0.006351669761257431,
+            0.005095804801686436,
+            0.009620040085743162,
+            0.019748001227987134,
+            0.033709535059510916,
+            0.008203559956940196,
+            0.007389752602218563,
+            0.04197771878351742,
+            -0.02067363758471765,
+            0.007543739188471913,
+            0.011207941261799052,
+            0.013964827251172813,
+            0.03839562075248879,
+            0.0009775244543292637,
+            0.006469507611492216,
+            -0.022808235395746654,
+            -0.024795004398633292,
+            0.006249024864814684,
+            0.011292877109066925,
+            0.01596839619826819,
+            -0.0014105396434174883,
+            -0.04152075438277094,
+            0.008267788411079337,
+            -0.028834191212891387,
+            0.030940526405027906,
+            -0.0018271056434014425,
+            0.006185475047820603,
+            -0.007920574674727123,
+            0.012907560753043616,
+            -0.015165704767187252,
+            -0.021962244496450727,
+            0.0037071053072833147,
+            -0.013508376064336042,
+            -0.015178823540513864,
+            -0.02518557053061311,
+            0.011693256916760944,
+            -0.01716015674592331,
+            0.02048897513447166,
+            -0.003412078966361745,
+            -0.001807298511378529,
+            -0.0012575655363779904,
+            0.03116103961636643,
+            0.011668533361641842,
+            -0.011637010548030312,
+            -0.040260195687925236,
+            -0.011129376691522074,
+            -0.0034493180842871908,
+            -0.02098244789711689,
+            0.026658743116016938,
+            -0.01923736325953796,
+            -0.009723937181300192,
+            -0.013134679989565051,
+            -0.011974548914818263,
+            -0.004028712301052159,
+            0.019628329175867303,
+            -0.028192034966932684,
+            -0.012638520793012082,
+            -0.011568652932013111,
+            -0.0025919910519445737,
+            0.012497908463405111,
+            0.015113186712897145,
+            0.014880607940212641,
+            -0.03059227859452334,
+            -0.021899716217105084,
+            -0.0008513506358489523,
+            0.011985931118464679,
+            -0.01321584595585435,
+            -0.011735288366213981,
+            0.010710683541435907,
+            0.01961722172525993,
+            -0.023571946726192585,
+            -0.03364281880125354,
+            0.0014460076743756465,
+            0.016563167828467187,
+            -0.02891589059260944,
+            0.04595166331401113,
+            0.034271553243501465,
+            -0.027557971577291153,
+            -0.007138821520412525,
+            -0.039508407807651844,
+            0.06814329636830853,
+            0.00022136123564675344,
+            -0.039718016705753735,
+            0.0503457594963753,
+            -0.003580581891576583,
+            0.013820796398521455,
+            0.02054050419088757,
+            0.035682661890720495,
+            -0.0013220739026852943,
+            -0.0052759394697328675,
+            0.04733018719162478,
+            -0.0187845855131196,
+            -0.03982699468608443,
+            -0.023905460465070062,
+            0.002631233133039351,
+            -0.03090995554989513,
+            -0.012617612051488022,
+            -0.013605787969473513,
+            0.039672783793764745,
+            0.008669582242097126,
+            0.009974277556736362,
+            0.005535636914843583,
+            0.014180668493093584,
+            0.012042659947850441,
+            0.011870115226306776,
+            0.00451259454348581,
+            -0.014023048592193627,
+            -0.007769033996392623,
+            -0.05715987527668358,
+            0.003374564518384728,
+            -0.028801158523506985,
+            -0.00756331369170698,
+            -0.02163054894055572,
+            0.005367181573233089,
+            0.02308705505078254,
+            -0.005438909398653553,
+            -0.00873620909426313,
+            -0.0036530652237719077,
+            0.013920407174296543,
+            -0.01539613379172909,
+            -0.03908211213269472,
+            -0.016810726548844992,
+            0.007460173854386443,
+            -0.007069189426742819,
+            -0.041197222498539215,
+            -0.003843233633335066,
+            0.004373367480223065,
+            0.016194121660459908,
+            -0.029165150066068787,
+            -0.00735139702658342,
+            0.017257895082337448,
+            0.014153313280536095,
+            -0.002376178042741926,
+            -0.023223700836427403,
+            -0.0319877537205969,
+            0.02088060774052704,
+            0.0012829290529515294,
+            0.03251550040316258,
+            -0.04128453069131847,
+            -0.023376114412063578,
+            0.04083022735076889,
+            0.005690491480259003,
+            -0.05810403800689653,
+            0.016582434816685452,
+            0.010402038964417569,
+            0.016795644054313566,
+            -0.00908590043098552,
+            0.05003656634510042,
+            -0.037216197954986735,
+            -0.013027960261454162,
+            -0.007090943677204207,
+            -0.05175371787508879,
+            -0.0020065660448063322,
+            -0.014908489552053589,
+            0.010627472758306387,
+            -0.03275473161158777,
+            -0.017334608434133866,
+            0.025321047439048318,
+            -0.0033515195870701753,
+            0.011526068468647207,
+            -0.026783879816504975,
+            -0.0012830935419413407,
+            -0.017506908439998053,
+            -0.02114853058440477,
+            0.004878884893669383,
+            -0.020936900837115968,
+            -0.0025587027781733642,
+            0.01557850246191957,
+            0.023652335794299834,
+            0.011489428313303988,
+            0.0019089137782552768,
+            0.02054907941022243,
+            0.014361232983780595,
+            -0.0056413109308506085,
+            -0.0010300050480036684,
+            -0.03413262505887213,
+            -0.018821160485010483,
+            0.03361989907065425,
+            0.013125889701529858,
+            -0.003742167496379789,
+            0.008813100942466033,
+            -0.0383265293638935,
+            -0.035316957263713485,
+            0.03020499323039515,
+            -0.05001623971243151,
+            -0.00855479667673597,
+            -0.005939674940109814,
+            -0.026451831943555135,
+            -0.008032873049527808,
+            0.0009431571685654594,
+            -0.014387605128858954,
+            0.016382559541755014,
+            -0.024909510232038502,
+            0.003715736505444385,
+            -0.006016712902409688,
+            -0.009110197692342108,
+            0.05507263160354303,
+            -0.022103031032598353,
+            -0.0475415595313825,
+            0.021958434235402655,
+            -0.016670809659931243,
+            0.0323519363801999,
+            0.03867494951405658,
+            0.011124508964363016,
+            -0.008002445714431074,
+            0.034041838454818577,
+            0.018400123235267234,
+            -0.01161754789471199,
+            -0.025901844576206774,
+            0.021800298132870245,
+            0.00943488425697727,
+            -0.05906691123008531,
+            -0.00782549274387971,
+            -0.019357446956797447,
+            0.008682376344965976,
+            0.02863077694455766,
+            0.0024925977373894447,
+            -0.002682170968419412,
+            -0.013590235283325244,
+            -0.021511149507762284,
+            -0.03255453916518719,
+            0.009039307015853905,
+            0.008305286476500875,
+            0.009312625027668458,
+            -0.0014144165766983945,
+            -0.007853869334376857,
+            0.009067065760942163,
+            0.022485201611182105,
+            0.03966189080765722,
+            -0.041627289519629344,
+            0.002707086456239785,
+            -0.022678178050044222,
+            -0.03653753142622053,
+            0.0036574646778835588,
+            0.029996160786576367,
+            0.02901163765559891,
+            -0.01971777193183787,
+            -0.0036589934841109884,
+            0.009181234938554578,
+            -0.025979483002227535,
+            -0.00773645103995767,
+            0.004280770859737866,
+            0.04116668301017644,
+            0.006201085381597264,
+            0.01402728258112097,
+            0.001060407317110527,
+            -0.00920801784000087,
+            0.02109880889040528,
+            0.025211022083982727,
+            -0.029866678264942617,
+            0.014439702646820226,
+            0.009379206296908577,
+            0.04156435812161122,
+            -0.037309655469525846,
+            -0.011819731348299495,
+            -0.02766498020409985,
+            -0.010454143230794476,
+            0.027911532296566614,
+            -0.04127890742215716,
+            -0.006948536183086887,
+            0.010207784113909938,
+            -0.0016584043356136647,
+            0.016437495428919636,
+            -0.014607861750686952,
+            -0.03222311919805017,
+            -0.000610723203118213,
+            -0.02867445198803213,
+            -0.024772816792038133,
+            -0.016510309753766612,
+            -0.018581201775667577,
+            -0.07450654873867177,
+            0.00026832248829154255,
+            -0.06724696990816072,
+            -0.02447002541264633,
+            0.04885569249258513,
+            -0.010195604207134053,
+            0.03332905935704749,
+            -0.02815272112271314,
+            -0.005296340304911926,
+            -0.05908835042526019,
+            0.0038476172058970774,
+            0.025753576863447866,
+            0.02386003857017594,
+            0.005206447371391696,
+            0.024799057871126032,
+            0.012060738268265606,
+            -0.04146503002980051,
+            -0.024949907556862643,
+            -0.014503475223770958,
+            -0.0016340466065744286,
+            -0.0034009167013795906,
+            -0.031298432995411005,
+            0.036241272149880994,
+            0.008902644391348814,
+            0.03550223922667804,
+            0.04463301409716232,
+            -0.010385984413759852,
+            -0.03305004479734889,
+            -0.006952739338820577,
+            -0.0175981588092108,
+            0.042668533202572856,
+            -0.017112184620086308,
+            0.005149173014526686,
+            -0.03219589107997635,
+            0.024181808150698054,
+            0.0014339790422945406,
+            0.01254721190611376,
+            0.011099458649404119,
+            0.0027626836767364253,
+            -0.026883640446090158,
+            0.0024160770766463607,
+            0.023874514847956222,
+            -0.022558433617669373,
+            0.00917667715473093,
+            -0.006754091166465801,
+            -0.033686935718776144,
+            -0.01078389638118685,
+            -0.013595327486167114,
+            0.024712781935748277,
+            0.013168464807451798,
+            0.005875864604862434,
+            -0.02417814420558463,
+            -0.0023978800459714144,
+            0.028650463366555304,
+            0.020176980768301288,
+            -0.02118184201861616,
+            0.019082340279671355,
+            0.019734925740100277,
+            -0.04209645071102001,
+            -0.04217310001316092,
+            0.006553913719098956,
+            -0.003735719227468833,
+            0.0061793909220459,
+            -0.028285889794908054,
+            0.008181788076484304,
+            0.004432488437404309,
+            -0.01743905248466261,
+            0.041221269621866216,
+            -0.014892354419172107,
+            0.005087102208335201,
+            -0.022198383716186407,
+            -0.011936388003605221,
+            -0.00027487042024290816,
+            0.044599215801158845,
+            0.008068975171827777,
+            0.003496950921841073,
+            0.02238779276447619,
+            0.0054839808262270315,
+            0.04816723616939378,
+            -0.02685221260466647,
+            -0.0190356946337995,
+            0.016464136459560113,
+            -0.0176092780782677,
+            -0.012513274095432353,
+            -0.010695421445334002,
+            -0.004784348908457072,
+            -0.022203041334932165,
+            0.010788155380415583,
+            -0.018353897272137543,
+            -0.02277431189907027,
+            -0.003945465850796918,
+            0.016024639792926797,
+            -0.006248368166794934,
+            -0.0038769803892794875,
+            0.0016292556540434277,
+            0.02412864830338344,
+            0.02892955453819876,
+            -0.02408022470555968,
+            0.019190282372105023,
+            0.018941584034503536,
+            -0.006709516380602965,
+            -0.020956377933747532,
+            -0.01964885206488053,
+            0.01434623424964373,
+            -0.01352968586212152,
+            -0.013872556892707697,
+            0.004052798542613835,
+            0.009201147374299997,
+            -0.004086067985859617,
+            -0.04185829685685201,
+            -0.014668698123513235,
+            0.003179403563664022,
+            -0.0006415014901450319,
+            0.019360184344240947,
+            -0.0150817538567364,
+            -0.029771999278536083,
+            0.014221776097046751,
+            0.022944884879589678,
+            -0.04493425220030206,
+            0.02117577719533236,
+            -0.0030648751497261983,
+            -0.00016251533643103604,
+            -3.817296018601414e-05,
+            0.008727767382077138,
+            0.026914779443266063,
+            0.03456565094655899,
+            0.01974806772696219,
+            0.00450800007152441,
+            -0.008952599700759673,
+            0.025184190461155245,
+            0.009846044083064704,
+            -0.025168735183134738,
+            0.010983353565382029,
+            0.016740721743326522,
+            0.0070838930012654296,
+            0.009173879407277282,
+            -0.006078288608766152,
+            -0.018269347893097643,
+            0.013181494188066667,
+            -0.01435693747022231,
+            0.01678907949532146,
+            0.05154463552784974,
+            -0.00039632596424461964,
+            -0.02461439550772063,
+            0.008418147037164186,
+            -0.006544319842335593,
+            -0.010065129286193278,
+            0.02214628550460128,
+            0.0029515574751370417,
+            -0.030862152033847717,
+            0.007509556025223583,
+            0.00675485304374519,
+            0.01894936231559203,
+            -0.02355525295639599,
+            -0.01387885858544803,
+            -0.00431398020087544,
+            0.0021208056468763523,
+            0.01580766504149989,
+            -0.004739477121073512,
+            -0.013204823447960706,
+            0.02994098229517376,
+            0.025496728234662373,
+            -0.00026124946738213036,
+            0.015060064980768175,
+            0.022262345898533903,
+            0.009968408664790368,
+            0.007588431141088797,
+            -0.0083063413609779,
+            -0.031336272171937095,
+            0.0018957858383561667,
+            0.05800811417728381,
+            -0.014937085296865802,
+            0.04313875432235027,
+            0.02018476425923236,
+            0.015657644644987107,
+            0.017880263053253607,
+            -0.01662268082870562,
+            -0.008740053682875323,
+            -0.0017630394204179638,
+            -0.008294417990024647,
+            -0.027613464270562948,
+            -0.005953209556351086,
+            -0.02379961210881074,
+            -0.013878676554701543,
+            0.015418315245979704,
+            -0.014764728065382074,
+            0.0027420640870858457,
+            0.010395538988699663,
+            -0.006590633833835581,
+            -0.00841537209927409,
+            -0.003943772700797381,
+            0.03376338017637296,
+            0.011980354721458463,
+            0.027761929822019422,
+            -0.00011128539412582973,
+            0.009342430952107655,
+            -0.018419526564396063,
+            -0.006839752805318664,
+            0.03271190887864607,
+            0.006011520427944665,
+            0.004652835471939905,
+            0.01485920920301398,
+            0.014305626381398871,
+            0.038015677219487023,
+            -0.005102005401565164,
+            -0.018204460895989052,
+            0.02065495133597456,
+            -0.0049699013356374775,
+            0.019149036693562712,
+            0.008825981577388056,
+            -0.002660055848523568,
+            -0.004292379561953203,
+            0.03090270710860331,
+            -0.020243440437217503,
+            0.018735727616321387,
+            -0.02238236089688497,
+            -0.0276584413929402,
+            0.023028408906176946,
+            0.008320436733620496,
+            0.04047641118026783,
+            0.011463851066078041,
+            -0.02247143742166039,
+            -0.034588725413285185,
+            0.009827713930789184,
+            0.00337071450636392,
+            -0.024446280199879062,
+            0.017542777160283324,
+            0.00780836483490212,
+            0.008453886667283516,
+            0.016624116261219637,
+            -0.004175682824655864,
+            -0.0042784362933758835,
+            -0.001546656717896957,
+            -0.008819739495525477,
+            0.01406790007187259,
+            -0.015487658289541583,
+            -0.021457182907140035,
+            0.024687169583245905,
+            -0.03915911962859764,
+            0.01771290139419629,
+            -0.006510046907173026,
+            -0.00997450880725535,
+            -0.02059560682587115,
+            1.5939836949198004e-05,
+            -0.007159493752862143,
+            -0.0004782363814453761,
+            -0.017503181957042913,
+            0.016643971600145645,
+            0.024056442566869802,
+            -0.013209211443909658,
+            -0.00445521990927108,
+            0.0020313626821877526,
+            -0.029235692455820056,
+            0.006662209097255912,
+            0.016993027053110635,
+            0.00529940612845993,
+            0.0026104743851393716,
+            0.0001707934714983891,
+            -0.031289446611163674,
+            -0.005782208620284939,
+            0.03053516889069683,
+            0.05590704747438521,
+            0.021628254147289437,
+            0.02403314981042378,
+            0.02908606663381421,
+            0.02717206823892633,
+            -0.003198844932040613,
+            0.0016217508103442348,
+            -0.01620660606431983,
+            0.027098323143010743,
+            -0.011664168290676943,
+            0.04322175021589083,
+            0.008652323184522541,
+            -0.049493020736445056,
+            0.004537958157696794,
+            0.00943239454147692,
+            0.034918603888916544,
+            0.016678716235690318,
+            0.0005887414054803805,
+            -0.005894338664998101,
+            0.03279165663336323,
+            -0.02735726709032382,
+            -0.011962070182991011,
+            -0.00036014407560325284,
+            -0.019079003730607717,
+            -0.015938295169860884,
+            0.010357611943935459,
+            0.04278924678845588,
+            0.0009957483598272451,
+            -0.0071422790468850215,
+            0.01571201177743999,
+            -0.009445307674383,
+            0.02286338492075306,
+            0.007336440384050498,
+            0.0042809497342459904,
+            0.025581467690095977,
+            -0.0025007871391868105,
+            0.02300539823997411,
+            -0.000628898771317254,
+            0.004582863909134464,
+            -0.015960578501883316,
+            -0.00930649396114688,
+            0.027838455754803444,
+            -0.011112268986721,
+            -0.029253746504396393,
+            -0.008734785561831333,
+            0.00472579547270089,
+            -0.0003805825712986138,
+            0.0021854358149324227,
+            -0.01183885533661493,
+            -0.010840951201591629,
+            0.019965691788904057,
+            -0.004922943301424112,
+            0.003800048754664761,
+            0.012664049429916252,
+            -0.0030124219674414897,
+            -0.01910844996750822,
+            0.01889983984736845,
+            -0.003858283778519141,
+            -0.024168821338992813,
+            0.020083955711766945,
+            0.01850497889545421,
+            0.045156567155119454,
+            -0.01586824055785609,
+            0.004612757419198795,
+            0.017497774574602505,
+            -0.002172088240279357,
+            -0.019617327081566827,
+            -0.014944795772093997,
+            -0.018212462639041492,
+            -0.015975283672064702,
+            0.012837956594691596,
+            -0.01243789877725244,
+            0.019500571353639314,
+            -0.0380735226388175,
+            0.012590900534826689,
+            0.004832168601136412,
+            0.007505385730121269,
+            0.016011983211212992,
+            0.025994560026860358,
+            0.002157436199771672,
+            -0.033465476168065784,
+            -0.01014599891204166,
+            0.007845373376916211,
+            0.010093432683262838,
+            -0.002849185178237036,
+            -0.018147004183434554,
+            0.003633790611809425,
+            -0.0266147420637404,
+            -0.018644422659309035,
+            0.022926737299761104,
+            -0.04798772015599113,
+            0.017908415301694906,
+            0.024859125158586466,
+            -0.04343699525707025,
+            -0.04886699607556352,
+            -0.011459188432595229,
+            -0.036774381316312464,
+            0.0014128005215813578,
+            -0.0005256067584181517,
+            0.011194992852264102,
+            -0.009342140079829378,
+            -0.018598671981710183,
+            0.01584789414301952,
+            -0.0036412641242200418,
+            -0.004225840239935549,
+            -0.00616696736018045,
+            -0.0025598631463178384,
+            -0.010188803157718543,
+            -0.03244694662411977,
+            -0.03290891136780909,
+            -0.00033742479296272127,
+            -0.009743927051414455,
+            -0.02334288359945272,
+            -0.03447062782872289,
+            -0.008728968107967028,
+            -0.037753659749578956,
+            -0.00515561961442376,
+            0.04070890150714597,
+            0.027517147608335957,
+            0.009449414978133334,
+            0.008086900089929912,
+            -0.013989034604490105,
+            0.05691047488419298,
+            -0.020425883381389086,
+            0.039466620642512155,
+            0.005301493569940605,
+            -0.0037222353899918703,
+            0.013075425214878277,
+            0.027710826422351564,
+            -0.006253665106995653,
+            -0.0017412928073238833,
+            -0.01928096630475438,
+            -0.012920476859996416,
+            0.009444408011467955,
+            0.015799884767184885,
+            -0.0025083061619197524,
+            0.001274457251739847,
+            0.00930533161700852,
+            -0.0006652478540264343,
+            0.017258038491335247,
+            0.03996268817486439,
+            0.003833844612473671,
+            -0.00826404141706295,
+            -0.018822002570963614,
+            -0.003188844900317347,
+            -0.011807006674248829,
+            0.010482058734260632,
+            -0.015410635475145028,
+            -0.011540803927522013,
+            -0.01249428777269578,
+            -0.017081856786754605,
+            -0.002183034976477535,
+            0.003114834030705039,
+            0.0027475411722881747,
+            0.02261627635192778,
+            -0.015872027027080824,
+            -0.022487581027490516,
+            -0.0035823117885461506,
+            -0.02398778449664045,
+            8.13716819067491e-05,
+            -0.01050060782407259,
+            -0.015616181283381218,
+            0.013676480044963827,
+            -0.021513173575512936,
+            -0.01091399089076612,
+            -0.008699176177861256,
+            -0.0006986953850759474,
+            0.005853723710671249,
+            -0.00993355748621621,
+            -0.00010241843582159751,
+            0.01150872694850241,
+            0.005559552266827916,
+            -0.012336449703762907,
+            0.0039431718621504,
+            -0.012740550272925562,
+            -0.004432059174951158,
+            -0.03592309122711106,
+            -0.019665686718735042,
+            -0.01846795637714379,
+            0.005642094448288269,
+            -0.005108964028543505,
+            -0.020586678054151854,
+            0.00668364061048941,
+            0.0034775127050453774,
+            0.004713169526661792,
+            -0.010723667875209928,
+            0.013944735241748791,
+            0.010882420168586074,
+            -0.003518245916227596,
+            0.013827334494230662,
+            -0.001330399936559009,
+            -0.020286697504699764,
+            -0.011139922402071095,
+            0.03538258948179278,
+            -0.024961826464903997,
+            -0.0018019328351763667,
+            -0.0026423695690168066,
+            0.00597003516870726,
+            -0.03071835946501977,
+            -0.013634212232744325,
+            0.02164400349828422,
+            -0.005846552894831441,
+            -0.026367014426434226,
+            0.03399116480234033,
+            0.03473258231629753,
+            0.010041605592929843,
+            -0.013423384340589794,
+            0.04730054254696893,
+            -0.012968978954938308,
+            -0.018274800841400665,
+            0.026826644936918647,
+            0.0278389611094493,
+            0.02525164095926168,
+            -0.029782550447890466,
+            -0.003010713086487275,
+            -0.02393629084357885,
+            0.017993366287762866,
+            -0.0263476663180971,
+            0.01864011499179424,
+            -0.02332263152537849,
+            0.016014908881756034,
+            0.01868464825773604,
+            -0.003104041044182249,
+            0.026313491086684425,
+            -0.0052945248919367815,
+            0.015253731232325074,
+            0.0012371308759955573,
+            -0.02119765517013431,
+            0.007795466385020642,
+            -0.006469500272298062,
+            0.025762169721880456,
+            0.00833992768202466,
+            0.01801901639978976,
+            0.022019940825351116,
+            -0.0409522298213741,
+            0.02003698451081499,
+            -0.001367916864679543,
+            0.01782984895971748,
+            0.0038516134262842373,
+            -0.007885432294782304,
+            0.014221728513400065,
+            0.010991352188016265,
+            -0.012851099261924757,
+            0.003840539873231741,
+            0.0008923482291826014,
+            -0.028928104542716937,
+            0.007066004768771198,
+            -0.026914574065060926,
+            0.025866907400807087,
+            -0.005526774784152086,
+            -0.00998122210956923,
+            -0.020744532673097317,
+            -0.0071425207262770865,
+            0.003754468896273156,
+            -0.012747874997429746,
+            -0.011385805601999295,
+            -0.024700457669046855,
+            -0.02401291382882952,
+            0.029591582802488962,
+            -0.014054332096344003,
+            -0.04015978298687865,
+            0.001234440863385445,
+            -0.023301255402129767,
+            0.01193125007773239,
+            -0.005793124129267758,
+            -0.024440100649464773,
+            -0.042563399082257154,
+            -0.017548888269389457,
+            -0.0009394887961485922,
+            0.022739763644266633,
+            -0.00114156949394707,
+            0.041791395064755515,
+            0.0008816696205437911,
+            0.022207132857533028,
+            -0.015547580543606755,
+            0.005319731264840819,
+            -0.02071229859314312,
+            -0.041495579742912285,
+            0.03660432378985971,
+            -0.014637031447181937,
+            0.025854844497435178,
+            0.03324325296704186,
+            -0.013009347371118843,
+            0.001354504885302625,
+            0.02652570709474851,
+            -0.014618100923440427,
+            -0.010782232747863238,
+            -0.0034936699515378212,
+            0.003461670978569965,
+            -0.006468116285021298,
+            -0.027793221545159466,
+            -0.007752458229193391,
+            0.04193706715422074,
+            0.017894069367616637,
+            0.021928994684423043,
+            0.022606401934291136,
+            -0.017972404868085358,
+            -0.026402048654300563,
+            0.003656646576244765,
+            -0.013129722990153284,
+            -0.018090360146368677,
+            -0.01624598856366761,
+            0.012397150789705746,
+            0.011741002980073786,
+            -0.004052422390675261,
+            -0.01836750423615859,
+            -0.020929335029248472,
+            -0.016327412417122416,
+            -0.0042615345226698015,
+            -0.008458117095535,
+            -0.005002582649405199,
+            -0.012158173174049797,
+            -0.020409616620022143,
+            0.03706368540614453,
+            0.0285565642978937,
+            -0.0041732933884846566,
+            -0.00607154864704767,
+            -0.011142150176334957,
+            0.008834431806990914,
+            0.006021232046087652,
+            -0.005028870471297621,
+            -2.3396043292139426e-05,
+            0.005112033372713154,
+            0.022822312435625314,
+            0.007271235299348001,
+            0.0035791789459533856,
+            -0.06340305811365254,
+            -0.005392000517722054,
+            0.022560201096078862,
+            0.012581401789917604,
+            -0.0032248219865223496,
+            0.025180718735801837,
+            0.0006451724964970927,
+            -0.009714514747463593,
+            -0.018389422219956937,
+            0.024728862150150004,
+            0.008760329201960964,
+            0.020988700992999768,
+            -0.014974763313686516,
+            0.003845297486923296,
+            0.0006445983847138896,
+            0.00750394530065325,
+            -0.0019792992847026793,
+            -0.008578254004439607,
+            0.025776677078006516,
+            0.016593438298233645,
+            -0.005695578040595484,
+            0.01604018433477423,
+            -0.0017497260488899657,
+            -0.021460543443428924,
+            -0.010265213095609554,
+            0.00042017209251926397,
+            -0.010570864288876387,
+            -0.018015790888295944,
+            0.017569414447505878,
+            -0.002601036282535863,
+            0.0072399377790164435,
+            0.030281707151249355,
+            0.0423183067569767,
+            0.009566493398534713,
+            -0.005928603293907002,
+            -0.0037936911251953654,
+            0.015477873304924939,
+            0.01513127995244932,
+            0.010741256034652754,
+            0.03015858543221727,
+            -0.029247492875021843,
+            -0.006690306960497428,
+            0.0046653727410352576,
+            0.004610405633429778,
+            -0.011692179099382171,
+            0.030055897497741527,
+            0.04569730592994177,
+            0.006736813244286893,
+            -0.004799978513117371,
+            0.002121067028052434,
+            0.011299889107780231,
+            0.007768137218140627,
+            -0.018927080714961783,
+            -0.0012222962555201746,
+            -0.006009911026251445,
+            -0.014001766533604134,
+            0.026941585318261085,
+            -0.003960837039351648,
+            -0.011848232223811548,
+            0.02903357274500332,
+            0.1105267435019781,
+            -0.006011029338781496,
+            -0.0053646998281195206,
+            -0.03242321704621517,
+            -0.015194102811332289,
+            -0.008860590919147765,
+            0.0056988205877963076,
+            0.01635957585050017,
+            0.020327661913559888,
+            -0.0008943408274661007,
+            0.016820860249955754,
+            0.02822175614376368,
+            -0.013854827839129885,
+            0.007906244895327178,
+            -0.030891381690403466,
+            0.015702117924362928,
+            -0.005383974926093977,
+            -0.012032955400313954,
+            -0.0026169511828034054,
+            -0.008618307035330928,
+            -0.0037356680273096105,
+            0.0009666430093455632,
+            0.011296307527684261,
+            0.004427516814276546,
+            -0.014484198313369681,
+            0.002829529156309784,
+            -0.008595300985863424,
+            -0.005122996529848575,
+            -0.02393571073523279,
+            -0.024119245543788293,
+            0.005223147047205755,
+            0.0019389305548629142,
+            6.344308292376579e-05,
+            0.006848965054539454,
+            -0.003810712095343454,
+            0.030149081317333418,
+            -0.01700322008272963,
+            0.0012288912810468446,
+            0.01193226302726573,
+            0.012332749933413344,
+            0.01688602002694633,
+            0.0028600984823761366,
+            -1.9882692552530084e-06,
+            0.021172977047420227,
+            -0.010514036509402561,
+            0.01460334896316005,
+            0.013091417303145243,
+            -0.04103572381458182,
+            0.0089746401463863,
+            -0.0027841289484449873,
+            -0.024252447218794344,
+            0.004446472209246425,
+            -0.018987867343500102,
+            -0.00881974688349601,
+            0.0005365625696166774,
+            0.012054719659248512,
+            0.01960300421041832,
+            -0.008553943699846696,
+            0.02111201386480852,
+            -0.0003501657740808554,
+            0.004177088753111459,
+            -0.012879944670340026,
+            0.0065222126975051685,
+            -0.007563354540402343,
+            0.0008168128607271199,
+            -0.018213241937597355,
+            0.012526561612175514,
+            -0.0008285725179894544,
+            -0.015737003159436623,
+            -0.011337688275309921,
+            0.004737346137976519,
+            -0.0027993545140114705,
+            0.0030843163114700313,
+            0.010153740340626154,
+            -0.008815302484351509,
+            0.004919275910399053,
+            -0.0017541042538165297,
+            0.002562816687714991,
+            0.014285249186421434,
+            -0.029816964593980453,
+            0.005357227738099325,
+            0.009154842495811638,
+            -0.00048782303639333943,
+            -0.013074976152495576,
+            -0.004054505559072866,
+            0.0028322102449485015,
+            -0.016877328079971877,
+            -0.012128199639779175,
+            0.023421330164419888,
+            -0.012565145848820333,
+            0.008609070188894315,
+            -0.010936598722317274,
+            -0.018401801734378692,
+            -0.01066119413133089,
+            -0.005254229780560808,
+            0.01234969508270852,
+            -0.0038427767204741877,
+            -0.006947594635952202,
+            -0.01756894543880097,
+            -0.007890897373987378,
+            -0.047246698568222466,
+            0.009617876521097624,
+            0.027157555263862723,
+            -0.011355758309667893,
+            -0.04050311381480088,
+            -0.01017774338301804,
+            0.02437257208967103,
+            -0.004056329295095018,
+            -0.008345450168215274,
+            -0.001599171409135998,
+            0.018358386463243014,
+            0.0027207445181251406,
+            -0.007292967749349572,
+            -0.011267861428599221,
+            0.01657507035043879,
+            0.015105061280816214,
+            0.021419854579631276,
+            -0.02088952706825817,
+            0.006295984081511286,
+            0.020973020776936996,
+            -0.03073246543555915,
+            0.0023720435876790876,
+            0.03363966216709148,
+            0.010412588613998134,
+            0.013023311559810034,
+            -0.0010217662933572685,
+            -0.0075567381078977215,
+            -0.008248818794295818,
+            0.011848200323549865,
+            0.005066463524567732,
+            -0.023174489297821385,
+            0.021823825355858556,
+            0.020368868756448753,
+            0.0026042856858509524,
+            0.003457067317891926,
+            0.0005539745115410357,
+            0.01062670844108589,
+            0.009948889803890108,
+            0.05100741556350918,
+            -0.006493714124814164,
+            -0.014021751055930108,
+            0.015063988102064609,
+            0.02488059237697814,
+            -0.0023499941119596228,
+            0.01087282522970853,
+            -0.02612218935931224,
+            -0.021423880914461062,
+            0.023389294741183817,
+            0.008967763132667892,
+            0.009637806238448432,
+            -0.010815706635624337,
+            -0.011390999462441815,
+            0.031750675158698435,
+            -0.01307829462211914,
+            0.00759749911213318,
+            -0.02169895957209634,
+            0.01387879409130081,
+            -0.008606362268863441,
+            0.032229658009209815,
+            -0.0009066257436946125,
+            0.011062436424260079,
+            0.028010486866468023,
+            -0.007835805879241447,
+            -0.019657486763625547,
+            0.003967480939987674,
+            -0.015577952617624695,
+            0.02188429826317828,
+            -0.020502758155233298,
+            -0.0002362500702658591,
+            0.0040935406015014265,
+            -0.010056462958039452,
+            -0.0057038083589231615,
+            0.011475170331848666,
+            0.0015345076181482267,
+            0.006359262977587827,
+            -0.020293014401106,
+            0.01369706440475074,
+            -0.003989482215141903,
+            0.007887824197823247,
+            -0.015925992045349828,
+            0.014370223626942766,
+            -0.02309139937510304,
+            -0.021808736033141787,
+            0.008142621179624918,
+            0.0108996933426044,
+            -0.007589050511458233,
+            -0.025971539510030145,
+            0.02250992080022228,
+            -0.024980833805349545,
+            0.024979008145354243,
+            -0.006622339940663538,
+            0.002566378488088469,
+            0.004913874740363343,
+            -0.008102103508732738,
+            -0.0028098905427027783,
+            0.0057865587191235565,
+            -0.0073509382400023995,
+            -0.014123756810537343,
+            -0.019453380788786282,
+            -0.04083629135586275,
+            -0.03177927047231389,
+            0.022927581786088044,
+            0.011557232599801421,
+            0.0017073832658476193,
+            0.02339552660545296,
+            -0.02247471449131386,
+            -0.017890866287500053,
+            -0.009775068417716798,
+            0.025184899522229628,
+            -0.04408541669037361,
+            0.012104255392615549,
+            -0.014669458760022245,
+            -0.0245258242481677,
+            0.006274252098230249,
+            -0.004161960147481513,
+            0.009299171724026455,
+            -0.010779421730862708,
+            -0.04713627671791054,
+            -0.005192684784377356,
+            0.02330975631183273,
+            -0.0036802876364168935,
+            -0.006197548344287773,
+            0.03048284650641829,
+            -0.0019503797069087845,
+            -0.005038254331399784,
+            -0.005524875459245342,
+            0.027783596017224226,
+            -0.024950083604372984,
+            0.010988398531346805,
+            0.008304937912856357,
+            -0.008485065895193593,
+            0.006572665697403556,
+            0.0011518846966102158,
+            0.011927348472282199,
+            0.015632588644277183,
+            -0.015559413716826264,
+            0.02262990861928957,
+            0.011446850286896198,
+            0.006506075777765303,
+            0.008015225053785824,
+            0.00010054430757843335,
+            0.0022975742216194894,
+            -0.024948925449668525,
+            0.026047924967851216,
+            0.01888211557921096,
+            0.0001632181017876392,
+            0.012553094476616919,
+            -0.015042621225242777,
+            -0.02782009986290425,
+            0.02606646297992026,
+            -0.016847653820092335,
+            -0.00045272379465939274,
+            0.020053091486344633,
+            -0.016210036358455773,
+            -0.007930550661303182,
+            -0.01123514849773852,
+            -0.0162675150307978,
+            0.01803076559921804,
+            -0.012160217543572069,
+            -0.02216594536057653,
+            0.002594675636753233,
+            0.016443123828746582,
+            0.02134480656850119,
+            0.026694575953313482,
+            -0.011470930697351533,
+            0.017768927048191052,
+            0.0008653408391607816,
+            0.001865765599795092,
+            0.025745954171649543,
+            0.01702677011968492,
+            -0.008474889109168202,
+            0.025347176686722313,
+            -0.0016686223698600755,
+            -0.002421921060670558,
+            -0.012265504084322075,
+            -0.0165214436458303,
+            -0.007545471138114208,
+            -0.015333235556146196,
+            -0.0051830106590765685,
+            0.000762746782664525,
+            -0.0018118022696726346,
+            0.10876576838038368,
+            0.0034948044540582626,
+            -0.011608460689239234,
+            -0.005870263500417206,
+            0.007510774308250375,
+            0.0023525201199670765,
+            -0.006846154764389575,
+            0.016082721103476375,
+            -0.018616626204086575,
+            -0.023593744079394,
+            0.004743481974389998,
+            -0.0009262286319469741,
+            0.00870122878944883,
+            0.02029523251993269,
+            -0.008608837818169835,
+            -0.0200567711498164,
+            0.007089477722191928,
+            0.005535149644041221,
+            0.0023640746516071774,
+            0.005256431824732927,
+            -0.0006118671321824933,
+            0.008209920240361308,
+            0.02448422122961942,
+            -0.015622892878860465,
+            -0.029770836607612903,
+            0.010535371601298532,
+            -0.00015994156470228762,
+            -0.007193425789067776,
+            -0.011046316463797128,
+            0.014728841492009627,
+            -0.0011488451990980268,
+            -0.02751404133172324,
+            0.017663132712536528,
+            -0.019055335178900415,
+            0.014643246200933506,
+            0.0025348345163347464,
+            -0.00016387173410118788,
+            0.022565953069005714,
+            0.013101158251834834,
+            -0.04272085076882881,
+            -0.0034442020309914823,
+            0.01041205537746844,
+            0.011059422927946552,
+            0.0116446016862942,
+            0.0012992535381387887,
+            -0.018670801241345514,
+            -0.011788984490283379,
+            -0.009109849103208542,
+            -0.021268488414296117,
+            -0.012523033785938503,
+            -0.011393853408628214,
+            0.010743646386439633,
+            0.0012964030814325153,
+            0.023809566099779113,
+            -0.003030535542277875,
+            -0.006526351867141901,
+            -0.000430699447948508,
+            0.00880561288237257,
+            0.009694646009259077,
+            -0.02079011138439714,
+            0.008294338719512953,
+            -0.01823922789220214,
+            0.011819769685519291,
+            0.006416044253693268,
+            -0.022799568832306067,
+            -0.009911619276012527,
+            0.007152786695893951,
+            0.008393677195358034,
+            -0.013569953300860946,
+            -0.006503988731872938,
+            0.02299498754558268,
+            0.018870619152710703,
+            0.012233964644738905,
+            0.024994616203196145,
+            0.03838969722235787,
+            0.005744255912479976,
+            0.027682123067526515,
+            -0.016791471092091604,
+            0.008459745383680297,
+            -0.022493868174622695,
+            0.012902262638928338,
+            -0.00460399519266689,
+            -0.0050431343566427886,
+            -0.018188192498411616,
+            0.033765105245412236,
+            -0.009159751849801773,
+            -0.0027150644587327763,
+            0.034778224828120714,
+            -0.0066502021793471075,
+            0.004525230911930458,
+            0.005096748531775903,
+            -0.016067049494073893,
+            0.033274672789154185,
+            0.01686831040872425,
+            0.016353966592947262,
+            -0.021631941883929817,
+            0.011748317746828848,
+            -0.005675701689448899,
+            0.0001273442591116645,
+            -0.020738819037390263,
+            -0.0037528323797385733,
+            0.0027725541343763104,
+            0.0059860509282538546,
+            -0.016556134518370437,
+            0.02033623476757896,
+            0.00019647023567641008,
+            0.0052895744489259875,
+            0.03253462247073863,
+            0.01043311189804924,
+            0.019182938878855655,
+            0.010051051068017807,
+            0.019523739618278902,
+            0.0281939391326836,
+            0.025685787482623124,
+            0.0029932688462254147,
+            -0.018668945726139946,
+            -0.0009326901784812825,
+            -0.007880352863625262,
+            0.010492063055849727,
+            0.022849902214818293,
+            -0.022619228461640748,
+            -0.02772644874135741,
+            0.018566981184526675,
+            -0.0022608185345048414,
+            0.020031103834122595,
+            0.007544464625538984,
+            -0.020935181884939796,
+            0.003407355515103286,
+            -0.023178230166337178,
+            -0.0005418155871620593,
+            -0.031039187720144967,
+            -0.028591174077243604,
+            -0.0007659241904525392,
+            0.011945344534674635,
+            -0.01688153017778311,
+            0.03357788163306365,
+            0.017667778671101954,
+            -0.014289109343082812,
+            0.01604684528817524,
+            -0.011054587325539086,
+            -0.0036959091138078683,
+            -0.005819853870004682,
+            -0.00203755010992417,
+            -0.02157944474979592,
+            0.005138614763345542,
+            -0.0033509138150859593,
+            -0.0024871060745482883,
+            -0.016936575573598154,
+            -0.021901830366872595,
+            0.016375173843894617,
+            0.016742770709641697,
+            -0.006606252542838758,
+            0.039771905404651496,
+            -0.004238071340064403,
+            0.00018364315349545657,
+            0.011690521707403059,
+            0.0010018392078563795,
+            -0.013651921068215563,
+            -0.010829994512513501,
+            0.012362647379788833,
+            0.011321403447882543,
+            -0.018942043512252663,
+            0.005763773608775982,
+            -0.009131565038297889,
+            0.002089216671850269,
+            0.005438887787329238,
+            -0.008183835275629478,
+            0.006690280329947951,
+            -0.01673229763262922,
+            0.015833397876364723,
+            0.00295474085300386,
+            0.003186554859191664,
+            -0.011797859525926772,
+            -0.020991927615680524,
+            0.01576413532776261,
+            0.0032403514518527287,
+            0.018819824905201635,
+            -0.026755218746060662,
+            -0.00012946467986739473,
+            -0.009552681500884924,
+            -0.0028941357661144826,
+            -0.012533800273548772,
+            0.013975640628890588,
+            0.016864739319953755,
+            0.0063941926578529,
+            0.01781016436903586,
+            -0.0026268224264609724,
+            0.015368284773773678,
+            -0.018122416427266218,
+            0.008686009180697955,
+            -0.006801074032721299,
+            -0.018537521789181048,
+            0.0013237338264105415,
+            0.027863741654435434,
+            -0.004670012662973029,
+            0.011066853115455966,
+            0.011704884649534624,
+            0.018954056680143452,
+            -0.0012626215734593707,
+            -0.0030850559460101866,
+            0.015610404665171367,
+            -0.005838226405658572,
+            -0.006861883833171456,
+            0.020872966233916948,
+            0.032951653763679145,
+            -0.006716215648910268,
+            -0.0021292599951407863,
+            0.010886596327481677,
+            -0.004901684487088035,
+            -0.008644584974417106,
+            -0.009587037829603014,
+            -0.03158137900717491,
+            0.024369089446706894,
+            0.012092412439477474,
+            -0.009601042951586109,
+            0.022690862146548786,
+            0.020752792552278052,
+            0.011412743968296708,
+            -0.010471860708195726,
+            0.02291536038589634,
+            -0.01268035142112569,
+            0.024168869865226136,
+            0.032482831775271157,
+            0.004400973556358811,
+            0.0022312906286559167,
+            0.0021175489661843985,
+            0.01691348460546639,
+            0.006050357346231155,
+            -0.007068133214675996,
+            0.011656003476409986,
+            -0.018487923180009552,
+            0.01598530891381001,
+            -0.004432366596808033,
+            0.02808371007118948,
+            -0.01337145216499331,
+            -0.010473700033368562,
+            -0.00537325376270318,
+            0.00010726431372328729,
+            0.006445701087575359,
+            -0.003324002570256215,
+            0.005255443653813165,
+            0.0007072577303252132,
+            -0.017825649146944237,
+            0.003050128130393781,
+            0.004825967170196209,
+            0.009018799597337101,
+            0.004103653497834134,
+            0.007249957401824204,
+            -0.012360355929483,
+            0.01441885664789707,
+            0.004401820363035808,
+            -0.0026190938717927086,
+            -0.0007163225760398784,
+            0.011124538295057815,
+            -0.013232822471919533,
+            -0.004895624967029031,
+            0.005281228035321008,
+            0.0059275753353475414,
+            -0.007507924998741734,
+            -0.004484140989405051,
+            0.011908076549392969,
+            -0.0037917291321648483,
+            0.006351543504726163,
+            -0.004409464890606014,
+            -0.0037760571637986827,
+            0.005303661313434895,
+            -0.00974511544836468,
+            -0.004204167898342997,
+            -0.0007093125445559502,
+            -0.006869250680233214,
+            0.00786202375969793,
+            0.005971033227117194,
+            0.010606085272785863,
+            0.006969437776542303,
+            0.004293129058563288,
+            0.004114452739123398,
+            0.004071017374828283,
+            -0.002445681623750424,
+            -0.00592480769363354,
+            -0.004617501328052899,
+            -0.012067458701076778,
+            -0.008335239153829813,
+            0.009025028998949398,
+            0.008653779421856954,
+            -0.019363018792788574,
+            0.02048011578221327,
+            -0.001188320960771659,
+            0.00772165044232104,
+            -0.01602086976290928,
+            -0.000907553574811279,
+            0.005464880629734941,
+            0.008267702703609528,
+            0.016136762227127184,
+            -0.03654209526492064,
+            0.005179791231187316,
+            -0.0037321883493524766,
+            0.016026013709955444,
+            -0.008215703411373182,
+            -0.006979816286354862,
+            0.02564060526656963,
+            -0.01450763388985563,
+            -0.0009847758228920171,
+            -0.016142304341196283,
+            -0.01407778890808498,
+            0.010815725252959506,
+            0.02050940280233426,
+            0.020684808835848383,
+            -0.0004965802897137407,
+            -0.0005781445948759235,
+            0.00036179841583004193,
+            0.022775804864342608,
+            0.015545681427566235,
+            0.018697154644607316,
+            -0.004543500782944068,
+            0.007028436664826706,
+            -0.023057215501463973,
+            0.009912581556496658,
+            0.0011795417129232596,
+            -0.008215027850908477,
+            -0.008554137076267468,
+            -0.008703378564520125,
+            -0.010448538943182258,
+            -0.011773811327190638,
+            0.004106616839960258,
+            0.013895013678116032,
+            0.004190549543250594,
+            0.0031571190600981074,
+            -0.0225385788570314,
+            -0.006243193914542516,
+            -0.011229641377928767,
+            -0.01674214043453308,
+            -0.007367389908398082,
+            -0.02386334969150212,
+            -0.028021339684039734,
+            0.0002794207370355175,
+            -0.02411861258809571,
+            -0.015958336406924612,
+            0.005268815864773021,
+            -0.01546610225139129,
+            0.00898600137695074,
+            -0.008396481690803911,
+            -0.011873486990732417,
+            0.0074743033679134105,
+            0.015336444788692,
+            -0.017676977079590463,
+            -0.010165317702640217,
+            -0.01690524990506033,
+            -0.013460451917691695,
+            0.010918550283871637,
+            0.00595960977280955,
+            0.009250157926438213,
+            -0.012447979337850305,
+            0.0062932435120863365,
+            -0.012673905639312696,
+            0.015073013552721701,
+            -0.03769166067890086,
+            -0.029992249463405965,
+            -0.00014990754540999537,
+            0.007288036630677436,
+            -0.0020735742042505426,
+            -0.018695120921334806,
+            0.0003860116505754868,
+            -0.003608995261360232,
+            -0.014900191335144973,
+            -0.013866216738266176,
+            0.025251475305524053,
+            -0.006788483107616133,
+            0.01223227424986397,
+            0.0034937277040015626,
+            0.03438333560343413,
+            0.008803852645562129,
+            0.014666206816302367,
+            -0.005709421955160623,
+            0.003742433643393657,
+            -0.0033706258426092744,
+            0.006906464031352315,
+            0.0076578888497945185,
+            0.009276592191710524,
+            0.008025321004305927,
+            0.004576470294889512,
+            0.002887346543823031,
+            -0.001925468214331082,
+            0.02125241545487307,
+            0.025671028960818562,
+            -0.00037467906648160044,
+            0.008981063192139038,
+            0.0009387900084090577,
+            0.0002735789978680719,
+            -0.008374290340790827,
+            0.006547013284767913,
+            -0.008274637814401893,
+            0.004041066530254948,
+            0.012897549078838394,
+            -0.009150863940242148,
+            -0.010211568689323908,
+            0.016575088843292566,
+            -0.012380099999574783,
+            -0.03420455578618604,
+            0.026550654680719987,
+            0.005324512466403649,
+            -0.0035670106106754245,
+            -0.010657928468872608,
+            0.0037176027853158167,
+            -0.01103891334887315,
+            0.008096915671982816,
+            0.022587659771132487,
+            -0.0013527917944322358,
+            -0.0018085171207746743,
+            0.0038503794877980586,
+            -0.011870484733648887,
+            -0.007881489342351292,
+            0.006462772034381968,
+            0.0315696514747912,
+            0.01326417653779577,
+            -0.020400369539140532,
+            -0.015223808971473513,
+            0.020406862349006738,
+            -0.023142176712111575,
+            -0.006254959806032159,
+            0.009039506093607052,
+            0.0009087520492922078,
+            -0.022766046838415165,
+            0.0009662037093312435,
+            0.004701966191595815,
+            0.005946458676746661,
+            -0.006358986495019584,
+            0.03237532341275415,
+            0.001075992257265679,
+            0.015892020271402903,
+            -0.017189453434945788,
+            0.0017514632185789299,
+            -0.014189278671121677,
+            -0.014265447388465805,
+            -0.029809460132405256,
+            0.0005557491671071693,
+            -0.0030324198417091704,
+            0.011038984948790744,
+            0.024808067315720335,
+            0.01145103533194245,
+            0.014432246996007246,
+            0.010552852947578788,
+            0.01513689173494211,
+            -0.013653722180565311,
+            -0.029009757850432732,
+            -0.005171522291984226,
+            -0.018001048743500243,
+            -0.02297482628212251,
+            0.003962613844063587,
+            0.013645125285548414,
+            0.006278662690346082,
+            0.023786381386575035,
+            0.0021738516691483163,
+            0.0009580591750940011,
+            -2.690805459518147e-05,
+            0.008894091070322921,
+            -0.004952203441302471,
+            -0.006553986197076102,
+            0.003570189040872711,
+            -0.000652714137223501,
+            0.021228360881461853,
+            -0.005758946428212041,
+            7.897153237591187e-05,
+            0.014242175492137011,
+            -0.005622256348934008,
+            -0.0035009192529791146,
+            -0.017339926143652377,
+            0.006248842389703505,
+            -0.01622011586976866,
+            0.025783086900871728,
+            0.018450783573486916,
+            -0.009109295293546267,
+            -0.013469521573468915,
+            -0.011259555201313116,
+            0.02773097452736661,
+            0.004631282554715908,
+            0.010787055220617197,
+            0.018228322680582464,
+            0.01461254259249465,
+            0.01784830193964943,
+            0.004664725683522323,
+            -0.0034339781146649384,
+            -0.0007692550982545399,
+            0.021176646860433974,
+            -0.014700178590249292,
+            -0.002701208084390198,
+            -0.008842788249773623,
+            0.0021799005046816444,
+            -0.008431299000066363,
+            0.006086447848362669,
+            -0.010793016137040172,
+            -0.015251381840165954,
+            0.0189503457652375,
+            -0.02660767353735015,
+            0.016423753600066336,
+            -0.00593016170680532,
+            -0.00482891500342298,
+            -0.027459214027897378,
+            0.010728039528784636,
+            0.0027866316191608297,
+            0.0022519974195000687,
+            0.010770472550461752,
+            -0.021427634123401408,
+            -0.01867717541189363,
+            0.008532444554010673,
+            -0.011457117350255678,
+            -0.007152301670244725,
+            -0.00348619742388509,
+            0.0154715607296319,
+            -0.00017898955187849443,
+            -0.009925365923862246,
+            0.02085093731060963,
+            -0.007175309371250024,
+            0.01223979370120031,
+            0.005256127464980655,
+            -0.008756077893734508,
+            -0.0028438320920811362,
+            0.021022708898898203,
+            0.027557382862284808,
+            -0.008342206702778956,
+            0.014526066580760486,
+            -0.011883797998224506,
+            -0.01088335352319397,
+            -0.03479444009382274,
+            0.030104838505937447,
+            -0.013305732257226255,
+            0.026773747582411633,
+            -0.0006968265979630097,
+            -0.017037729111584094,
+            -0.016206207453228583,
+            0.01625225876515654,
+            -0.0024290584023212254,
+            -0.009366287391027166,
+            0.01465885301870365,
+            -0.02428628495473643,
+            0.0058336571592899735,
+            -0.0006627935593670669,
+            -0.011531762838191602,
+            0.013275508677482622,
+            0.0019223252930710324,
+            0.010738622785461513,
+            -0.02030405805119001,
+            0.0009661595529028194,
+            0.009263716373456188,
+            -0.016791855691574455,
+            0.006057830565306032,
+            -0.03730952786847912,
+            0.0029341721588188943,
+            -0.014833895684348656,
+            0.005725615593914644,
+            0.001498755253963699,
+            -0.01820708352937343,
+            -0.03652692232407375,
+            0.00043665701245599497,
+            0.01913883585802903,
+            -0.007975988799790905,
+            0.020538595326175857,
+            0.008243569241768276,
+            0.008421404577931517,
+            0.002314738067443065,
+            0.0016702403734608475,
+            0.008279538988821465,
+            0.03679456861213399,
+            -0.01775927634148114,
+            -0.004836606118958108,
+            -0.0012151329859941935,
+            0.010644195708319258,
+            -0.02028470571116305,
+            -0.009287515726906219,
+            -0.0036506912311261536,
+            -0.013134467570201307,
+            0.030615862647585222,
+            -0.020265555441523482,
+            0.025634449009756734,
+            -0.009300770899023007,
+            0.009274072905190077,
+            -0.02097685009125915,
+            -0.009797008690754956,
+            -0.010661540100131183,
+            0.0019532291898233895,
+            0.013465005282849381,
+            -0.009151430285942593,
+            0.0071467934925125145,
+            -0.006655608635025676,
+            0.01020327561163374,
+            -0.001472597695707201,
+            -0.02443508800166349,
+            -0.0024998811551575563,
+            -0.01107321258077183,
+            0.02456164421154801,
+            -0.0019619696941303388,
+            0.0005043179699158694,
+            0.010951580163871948,
+            -0.00853593956432799,
+            0.023927336608555987,
+            -0.008855702280766362,
+            0.0010093679746665581,
+            -0.004863269000020324,
+            -0.004005691415117686,
+            0.001698212044518236,
+            0.01822024022650613,
+            -0.005707404715849356,
+            0.03722587302228139,
+            -0.01708925706528118,
+            -0.0065500772743641925,
+            -0.00015538686795361824,
+            0.010852925330001283,
+            0.010035359503147549,
+            0.008754719437760327,
+            0.004067820167375348,
+            -0.014861417626433965,
+            0.009346669946626484,
+            -0.020136029447262147,
+            0.007501669351583119,
+            0.023178490169347605,
+            -0.006840359541885465,
+            0.0006676450965357345,
+            -0.01283320739092204,
+            -0.015610538827742276,
+            0.006308543164022682,
+            0.002329420005586303,
+            -0.002295739637504597,
+            -0.02181189272979859,
+            0.006298804075737111,
+            -0.014398470126723037,
+            0.004007419942728446,
+            -0.0007497656993578918,
+            0.013863867106289847,
+            0.016451166273826513,
+            -0.006740209707090658,
+            0.039292260347522444,
+            0.01871723939316731,
+            0.009109005310505462,
+            0.005674782595856756,
+            -0.040588444759736254,
+            -0.008993850998154852,
+            -0.0010197185016393919,
+            0.004866150984083949,
+            -0.01141354216871663,
+            -0.018944881712855702,
+            -0.002646110151132781,
+            0.000308585036997592,
+            -0.010681954707313127,
+            -0.009091585604191968,
+            0.01751599002710486,
+            -0.006034424764302979,
+            0.00872057856926618,
+            0.004146961781109887,
+            0.023758065635807138,
+            -0.009727802672285544,
+            0.01176257119851645,
+            -0.0071822083765114165,
+            -0.005456197751568526,
+            0.0006726765703478756,
+            0.011011628907288813,
+            -0.008031792262886802,
+            0.009763340867550022,
+            -0.004279891138185881,
+            0.008355720233269893,
+            -0.008552596384067825,
+            -0.023952129458400563,
+            -0.014972827087541732,
+            0.014347535404251018,
+            0.011651260496709961,
+            0.008137738462704715,
+            -0.013624435569539285,
+            0.0031728721674035352,
+            0.00675830277744976,
+            -0.009184784787581942,
+            0.009161771311910335,
+            0.017733082848622708,
+            -0.001411542568472906,
+            -0.0017830167499636113,
+            -0.0016130003349297027,
+            -0.012173528776899308,
+            0.0011196437753239162,
+            0.02284887358189995,
+            -0.003054479643937998,
+            0.014614283913976716,
+            0.010689486748036806,
+            0.009643292262102645,
+            -0.00810930837603355,
+            0.001915955511174678,
+            0.02341863048264366,
+            0.015022291783105098,
+            -0.004625715884342598,
+            -0.011422973894623295,
+            0.004259757717597968,
+            -0.010332744704689279,
+            0.007373328473086509,
+            -0.02117940113144938,
+            -0.004672698999328004,
+            0.008896311380763048,
+            -0.007028862917603513,
+            -0.01732432285099198,
+            -0.028026173555055017,
+            -0.016765805892242715,
+            0.019819329380181,
+            0.0027213760678977965,
+            -0.013291251876048186,
+            -0.0032919084721662363,
+            0.0031214883337300926,
+            0.002553005168048115,
+            -0.0071104806975682885,
+            0.005310583052470149,
+            0.007359183453835734,
+            -0.008674133980486269,
+            0.017665958608281126,
+            -0.000521269431423141,
+            -0.0008020251454616109,
+            -0.0005442374261144659,
+            0.0006127687598711624,
+            -0.016387752806126442,
+            0.01666161912459431,
+            -0.010949228376419892,
+            0.015198110119816624,
+            -0.008966726991404666,
+            0.008600499511690635,
+            -0.007617043351740469,
+            0.009458641139070436,
+            0.011597957400445055,
+            0.012755037944648426,
+            -0.004403680820063604,
+            0.008519241473418196,
+            0.004807816191718704,
+            -0.008147125240985166,
+            0.007017601015398136,
+            0.01296082929962557,
+            0.00582353932160707,
+            0.0013634237970372698,
+            -0.0008246894322793264,
+            0.0022035670958705576,
+            -0.002480976219285683,
+            0.0056293125704658135,
+            0.012714850523393719,
+            -0.002497639449055006,
+            -0.007135683201188999,
+            0.014988249771676699,
+            0.014955895630183352,
+            -0.022994948923833993,
+            0.007121634434082677,
+            0.005835971825639792,
+            -0.0030507120821557008,
+            0.002145266953387315,
+            -0.027334525318320587,
+            -0.020359474070022213,
+            0.01901995970350808,
+            -0.015260454474605018,
+            0.02850460517419317,
+            -0.011624138792780823,
+            -0.02284242513728605,
+            -0.01277489645734181,
+            -0.004209529633534193,
+            -0.022684500849916897,
+            -0.004894414098943742,
+            0.01919165657366256,
+            0.013401560972810082,
+            -0.004313549020265677,
+            -0.019089369171102138,
+            0.03863248408555474,
+            0.010497951840980463,
+            0.03508157919687289,
+            -0.008597970944278045,
+            0.0023354831464448313,
+            -0.01563401808449416,
+            0.015551176593436512,
+            0.001075882802133782,
+            -0.011575692834796734,
+            0.01505716108583101,
+            0.031836226915668404,
+            0.0034740889237207494,
+            -0.00617206855371193,
+            0.0018420207148235309,
+            0.00418433325800744,
+            -0.0008167766650900381,
+            0.011160989178055499,
+            -0.018497472318034093,
+            -0.011951262348399922,
+            0.0013093376768780122,
+            0.00216347903176653,
+            0.0059233460419514045,
+            -0.001778091863744448,
+            -0.013317639704511549,
+            0.011433308892643204,
+            0.011596810919469926,
+            -0.01518143105525228,
+            -0.007497373860338335,
+            0.0050957665991362405,
+            -0.018974405317819656,
+            0.01872472902494418,
+            0.011859522398789263,
+            0.000792413325332956,
+            -0.0006503947129906383,
+            0.00808635755624928,
+            0.003749813548995223,
+            0.0033657799335872368,
+            -0.007862215303617144,
+            -0.0028014669075116876,
+            0.0009892310856576799,
+            3.834379772674841e-06,
+            0.015402211364447726,
+            -0.015264893397161102,
+            0.02488855366696587,
+            -0.0025643639525691083,
+            0.00953993868921819,
+            -0.017460590918773188,
+            -0.0017911712931080155,
+            -0.00402374787529868,
+            0.01838051670627292,
+            0.006753072393532114,
+            -0.010130676216186492,
+            0.004921510943896829,
+            0.007975201081296845,
+            -0.012589786869771252,
+            0.00018319054705558588,
+            -0.0022606180909706688,
+            -0.014791528072476771,
+            -0.0031171076370692706,
+            0.017873825561816544,
+            0.007593688317551089,
+            0.0037375070591255375,
+            -0.015863143964928172,
+            0.007192324815779268,
+            0.007522990667369163,
+            0.024504875417877527,
+            -0.026309816223960274,
+            0.024779042752840836,
+            -0.022104087334088543,
+            0.02165781034613893,
+            0.0319051741666687,
+            0.007538146369110045,
+            -0.01690181186708103,
+            -0.006595947249169521,
+            0.006373991305923002,
+            -0.023410814842488123,
+            -0.005494017994983188,
+            0.016903736810723716,
+            -0.01792400152372258,
+            0.000330865061689199,
+            -0.014660413003063089,
+            0.0040825880154611895,
+            0.008248037865838212,
+            0.011540333451799686,
+            -0.0033578830043738085,
+            0.018327525495676525,
+            0.020171745352159914,
+            -0.01138759854371235,
+            -0.0004221149053107702,
+            -0.022321409200941777,
+            0.005046865873981027,
+            0.022509643273449936,
+            0.001111272240506322,
+            -0.009540467231190513,
+            0.018376261423745193,
+            0.008480532791922674,
+            -0.020801891594934313,
+            0.008436374227753589,
+            -0.036175982655998785,
+            -0.010454118523313122,
+            -0.009328507118928412,
+            -0.001218633166683384,
+            -0.004142773602181115,
+            -0.0002494371580772723,
+            -0.011365548842513265,
+            -7.699548642315061e-05,
+            -0.006889339420877534,
+            -0.0037173070609413474,
+            0.014953978004522174,
+            0.011678957032329096,
+            -0.003501047405385249,
+            -0.0073984932581048915,
+            0.008493128567102267,
+            0.006163937693564917,
+            -0.017034696006318337,
+            -0.013268418502042576,
+            0.0037122128689059085,
+            -0.020067656401141467,
+            0.005148329575092478,
+            0.00459729178556505,
+            0.014031669598917171,
+            -0.01849235806598133,
+            -0.009298127665365717,
+            0.019716512208448042,
+            0.0094466405158553,
+            -0.0008988916054855498,
+            0.004264260983102956,
+            -0.00032312548009733906,
+            0.002279987803645912,
+            -0.0008902345374346296,
+            -6.145528806973222e-06,
+            0.003602229465148649,
+            0.012503366799381277,
+            0.007759393855482671,
+            0.03694402922447051,
+            -0.016692148708641812,
+            0.005873765863350649,
+            -0.014187609702966002,
+            -0.0037592517335595654,
+            -0.00749939010396784,
+            -0.0004308157309481927,
+            -0.0022282234382577843,
+            0.0021999996319642854,
+            0.0052176773733353925,
+            0.009353171029508977,
+            0.002155405440765206,
+            -0.00823565584692709,
+            0.009691447250255694,
+            -0.015228173095533029,
+            0.003853858061067531,
+            -0.008207910291425473,
+            0.01017972550354623,
+            -0.009695280827515048,
+            0.0028705587799331246,
+            0.03849358164875786,
+            0.01130021300937487,
+            -0.004934175853966098,
+            -0.014300147534361459,
+            -0.0007796051618993811,
+            -0.01737877501558196,
+            -0.008579119859817895,
+            -0.005434792488602325,
+            -0.02617338653902575,
+            -0.00838916510137031,
+            0.015344260464413648,
+            0.011202567142310492,
+            0.014124853863170132,
+            -0.023894182185855824,
+            0.008897943489322673,
+            -0.03001670823127986,
+            -0.011951507895140876,
+            -0.001530967140156163,
+            -0.015467812535428609,
+            0.0010110999991167703,
+            0.023623196504289838,
+            -0.007762735863648054,
+            0.012654652822750325,
+            0.012867958170258448,
+            0.018777522196571934,
+            0.006390986057157016,
+            0.00038045736974183937,
+            -0.009444911783760569,
+            0.009534310165417939,
+            0.008806031651319704,
+            0.0080954543713255,
+            0.013161833638467553,
+            -0.0022840529693446553,
+            -0.018060162439023572,
+            0.005469288590000856,
+            0.014354475778235131,
+            -0.0033504276914600585,
+            -0.001615191819057537,
+            -0.011088493180661976,
+            0.016039040565630432,
+            -0.023559046333872156,
+            -0.009482398794557168,
+            0.009918576960319548,
+            -0.0027308309155029842,
+            0.009212063547988036,
+            0.0160181200081126,
+            -0.01881373001297218,
+            -0.0024168618343039703,
+            0.016244889977796682,
+            0.0024104546308670025,
+            -0.00022606425578303413,
+            0.016792244238218245,
+            -0.0007616857360728623,
+            -0.004279120886828465,
+            0.00997764266310123,
+            -0.0075631917138318655,
+            7.3878137053128405e-06,
+            -0.002416364915008295,
+            -0.0028253498168087964,
+            0.02054128727706496,
+            -0.021281801678871736,
+            -0.005379259148621041,
+            0.02257306103533458,
+            0.015687105622685975,
+            -0.007843745111084413,
+            6.258498042855072e-05,
+            -0.0219506172079994,
+            0.0019925886885244454,
+            -0.007768133999661569,
+            0.016524850592507592,
+            -0.007957949830060354,
+            0.0030085044673901665,
+            -0.01361024620290394,
+            0.011563781350927133,
+            -0.01277407169098418,
+            0.00873716215529134,
+            -0.002986943948333218,
+            -0.017567646684567492,
+            -0.0053109390854215125,
+            0.004561236136619266,
+            0.0014045175130801308,
+            0.044958041347040185,
+            -0.005539136232703934,
+            0.003729517522976541,
+            0.018086335927383804,
+            -0.014199453349727933,
+            0.02079719360289355,
+            0.016663293302592568,
+            0.00991445855219406,
+            -0.00033967370741635794,
+            -0.0018726748849858836,
+            0.023373354806639515,
+            0.013258226441642464,
+            0.00861986998150065,
+            -0.0014340218811281975,
+            -0.008516541227326329,
+            0.018239230461345617,
+            0.005458629943667832,
+            -0.012421896028975013,
+            0.012732721349758832,
+            0.013559877132508506,
+            -0.013720939472355347,
+            -0.011545210834395917,
+            0.01616358223872008,
+            -0.019904594319155373,
+            -0.018260016571763644,
+            -0.0036124537591368864,
+            -0.01614510591132948,
+            -0.03864699384205461,
+            0.01619401261888991,
+            0.018351607973242735,
+            0.006666313000123845,
+            0.013647410298896132,
+            0.002597680061427196,
+            0.004545530267244157,
+            -0.018531082430861854,
+            -0.0077618625909995275,
+            -0.008085144524135461,
+            0.003744735923004248,
+            0.011059457364156674,
+            -0.011335432904114519,
+            -0.009514379703581616,
+            -0.00034500702695418285,
+            -0.007433132060989644,
+            0.01971420270994951,
+            0.0020753610290475642,
+            0.0034506706065161024,
+            -0.00628437298309269,
+            0.017966485854173167,
+            0.026789678964152908,
+            -0.007532446825077476,
+            0.021886182869092073,
+            -0.002864230253056434,
+            -0.0015870360120997492,
+            -0.014393519359298496,
+            -0.0014439412842964876,
+            -0.011527182213472351,
+            0.0005453493437517964,
+            -0.0006791481533636673,
+            0.014093767997869645,
+            -0.03288824642147967,
+            -0.003957301296796813,
+            0.014444592778851293,
+            0.007538052312876599,
+            -0.005057755472988024,
+            0.011357919882949903,
+            -0.002964451028327082,
+            0.016977317604413924,
+            0.0075201055190542146,
+            0.023391646079994784,
+            -0.016588823248397513,
+            -0.004504858318770565,
+            0.006660877064954993,
+            0.0034631255931793638,
+            0.0014484203909961972,
+            0.01912983041187014,
+            0.023439941258581887,
+            -0.014715886385460541,
+            -0.0013596123366001458,
+            -0.0029020372473879614,
+            -0.021414221343490594,
+            0.014976806123253727,
+            0.0074729481304182865,
+            0.010967351226114886,
+            0.009806663105020302,
+            0.015465076881663691,
+            -0.02109587527354609,
+            0.0011671121739822202,
+            0.020567458365051382,
+            -0.008618333549105003,
+            -0.018815706906171112,
+            -0.009804829158883123,
+            -0.00966828714719191,
+            -0.006078893678150302,
+            -0.00803783313451146,
+            0.005523023167028105,
+            0.0022647506733877573,
+            0.01652997604669998,
+            -0.008204200434441975,
+            -0.008404609147866811,
+            0.008489247125986078,
+            0.007546785673285454,
+            -0.0033093451283570316,
+            0.010614050963656496,
+            0.0030633179859393587,
+            -0.006940540318011839,
+            0.00894884343338777,
+            -0.002787816848430143,
+            0.014347598920245487,
+            -0.02441769157218451,
+            0.004717742147185283,
+            -0.012096876825337124,
+            0.006405151160675699,
+            0.0041568790527989905,
+            0.0016702066682613428,
+            -0.0041159211459004,
+            0.0035217362856754353,
+            0.0008893479210743541,
+            -0.07433312151926533,
+            -0.012629681414004524,
+            -0.0035797012285864486,
+            0.033657838713146884,
+            -0.019695806809053914,
+            0.004477927056336216,
+            0.011966065644382396,
+            0.0096717896523898,
+            0.003070713232538593,
+            0.006136726101157844,
+            0.006112385569942848,
+            -0.0245258242481677,
+            -0.00015284418977092615,
+            0.004821040694679944,
+            0.0020347004045943454,
+            -0.007532408630122127,
+            0.00750565079228405,
+            0.015327710245879296,
+            0.020989926895388523,
+            -0.010970305087289488,
+            0.017227663358885748,
+            -0.003352718496981928,
+            -0.006068892126250978,
+            0.0014285750322333686,
+            -0.010153799722483554,
+            -0.017307415843726394,
+            -0.006495008086641125,
+            0.0068983699213297075,
+            -0.001982568363667046,
+            0.0044032687939897855,
+            0.003333596495330213,
+            -0.0032171586993948696,
+            0.0072365127663633925,
+            0.013920326463272646,
+            0.006415916101287133,
+            -0.013675562796300214,
+            0.003003782758807665,
+            -0.02500375035312058,
+            0.011945988879567646,
+            -0.012708975181025296,
+            -0.0009488161508767279,
+            -0.0047093877027523014,
+            -0.0035750724825159503,
+            -0.002213435677949806,
+            0.02043567538125188,
+            -0.014304009033474186,
+            -0.01802899775650995,
+            0.008187227768196673,
+            -0.011217013195839762,
+            0.008300740149622635,
+            -0.03382831122769974,
+            0.012386335491333264,
+            -0.0276029934382802,
+            -0.0002243752790856292,
+            -0.004521387639042107,
+            -0.0010631721066129095,
+            -0.009491184157278669,
+            -0.03274559774619223,
+            0.015845705808609598,
+            -0.007204751554079551,
+            -0.020338337341085228,
+            0.0007143424225187802,
+            -0.021267094083674304,
+            0.014150458196361147,
+            -0.01851165553224022,
+            0.0033450353123933058,
+            -0.018775439467431685,
+            0.002782289821140401,
+            -0.0018711350216089556,
+            0.010502195116579441,
+            -0.00557814442809019,
+            -0.0007130962242755068,
+            -0.013072988887382269,
+            -0.002121841212381393,
+            -0.006653444103331213,
+            0.009394907690516121,
+            0.0028678074167728367,
+            -0.009601577734902932,
+            0.011153571864462314,
+            0.013046116892519414,
+            0.019812860521973938,
+            -0.00539047787915588,
+            -0.00555139958411929,
+            -0.011553689090179406,
+            0.009852063270798534,
+            -0.016326212706634428,
+            -0.00827402913960903,
+            0.008982303148984887,
+            0.015216947130173639,
+            0.01026889105233137,
+            0.003352028462252994,
+            0.003150801750574548,
+            0.002531620731961765,
+            0.026744691368233237,
+            -0.0015120532859309638,
+            0.0008324701960729416,
+            -0.014508988496009931,
+            0.022127525293249926,
+            0.02772430884375744,
+            -0.0012206102581726523,
+            -0.00018859639594030037,
+            0.017437360169132967,
+            -0.004697052522250805,
+            -0.0038111184371524645,
+            0.024015246959315593,
+            -0.038396108183338655,
+            -0.008416825232533771,
+            -0.006788816882617399,
+            -0.006043428198944214,
+            -0.009587366247924467,
+            0.007294190273465812,
+            0.00041966080769004275,
+            0.0024709692753547823,
+            -0.0005250904505954556,
+            0.03452858797518386,
+            0.003403515654095309,
+            -0.0003392060101496309,
+            -0.0032980864446307116,
+            0.0019125121221204029,
+            0.0032987517853214346,
+            -0.009116761755779728,
+            -0.038030085122772624,
+            0.006006334000396066,
+            0.007932785488877407,
+            0.00975009749364395,
+            -0.008120121513849434,
+            -0.008864461019882246,
+            0.004909348738798099,
+            -0.02273129283865203,
+            -0.01372636307167243,
+            0.02625229144362647,
+            -0.008523028986127237,
+            0.029005877324974557,
+            -0.022435059061182174,
+            -0.013135836801818153,
+            -0.004126433322528265,
+            0.010283975013795537,
+            -0.019732856462444506,
+            0.004343860338939202,
+            -0.004022387065239562,
+            0.023733721026005954,
+            0.0077578144626961245,
+            -0.008371631446146718,
+            0.012386735614153616,
+            0.0064789380436815714,
+            -0.0026638642827013837,
+            0.00136333052963848,
+            -0.007209517067748103,
+            0.016916505654836485,
+            -0.011819784026452944,
+            -0.005833909403097991,
+            0.008977527508585844,
+            -0.0024556434204852946,
+            0.02509293948918158,
+            -0.0237762245071987,
+            0.005288171058938098,
+            -0.00019047961064753207,
+            0.01822542471604037,
+            0.008127347078975925,
+            0.008361588667386882,
+            -0.011835407590060994,
+            0.010004482616697591,
+            -0.03378107067882622,
+            -0.015433906677849963,
+            -0.007702910250795191,
+            -0.0013877702264578865,
+            -0.002059152251387601,
+            -0.015745218548968338,
+            -0.026393221735795527,
+            0.006294583740809379,
+            0.016267525575472445,
+            0.023291528020980265,
+            -0.004683197930892445,
+            0.012912830736650997,
+            -0.002831583991769044,
+            -0.008904271394710672,
+            -0.011441882849296418,
+            -0.012202629605208118,
+            -0.012470773094383122,
+            -0.014560694782340029,
+            -0.017045701603711442,
+            0.012139665577738337,
+            -0.020079061996873537,
+            0.013911518328924777,
+            0.0347210775522539,
+            -0.017176114991595492,
+            -0.0018941772168839354,
+            0.019702722555622765,
+            -0.004221469612316137,
+            0.01212088353057276,
+            0.0063707489618353725,
+            0.015298014532351866,
+            0.0038643174573634725,
+            -0.012181479717826009,
+            -0.007374005740809251,
+            0.019411457664902376,
+            -0.004902591844710158,
+            0.014591446725717338,
+            0.015069599884557399,
+            -0.008675672164385891,
+            0.002544198867387345,
+            0.01951865904496276,
+            -0.01762101777798757,
+            0.01665654707715257,
+            -0.00458402229480694,
+            -0.009936206685538294,
+            -0.013345740581684638,
+            -0.005336416724426228,
+            -0.019469457091169776,
+            0.0061608454109892425,
+            0.018931676738452464,
+            -0.00533013594306834,
+            0.009310871143727683,
+            -0.0037670793158045,
+            -0.005045151384034014,
+            0.0002041909886640482,
+            -0.04152751596227068,
+            -0.008340911336217609,
+            -0.0006651584370005978,
+            -0.02505748815737022,
+            0.025954102004571416,
+            0.007205602377144382,
+            0.022602344303400163,
+            -0.017168610245491406,
+            0.004459735690564842,
+            -0.005882024209102714,
+            0.013217517324383834,
+            0.02255689669623969,
+            0.0017984057469785134,
+            0.005699128497215882,
+            -0.02353849721739205,
+            -0.0034711576718386185,
+            0.015924913374553738,
+            -0.003604478141616307,
+            -0.0034993903155063817,
+            0.012592308877056896,
+            0.012864938348003884,
+            0.01567418642605425,
+            0.008801753550506258,
+            0.015606443428689292,
+            0.010589933994371628,
+            0.008928721556432368,
+            -0.005805129999077126,
+            -0.001025103064751441,
+            -0.015907755521365925,
+            0.0045874330999415985,
+            0.031701049450012876,
+            0.012594483199773746,
+            0.016783858873835704,
+            -0.01778536268185936,
+            0.00789523134933153,
+            -0.0012706411671644011,
+            0.008325108613530198,
+            -0.001677173669891956,
+            0.029540136288412976,
+            0.011739999605699826,
+            0.015670966746894652,
+            0.008732334090105053,
+            -0.016237731209426752,
+            0.045334690225148,
+            -0.016911928942220152,
+            -0.0021139311364009113,
+            -0.01058970056572469,
+            -0.021705338531028618,
+            0.007324792557651635,
+            -0.006556582104686972,
+            0.0018243188317438144,
+            0.010780292738415671,
+            -0.006471850255151019,
+            0.0006997395548596146,
+            0.003907433192252573,
+            -0.023129313269763224,
+            -0.002550875642832235,
+            -0.02237917712764273,
+            -0.003107775639788617,
+            -0.02621608419628401,
+            -0.00012482032027919036,
+            -0.016207377503696517,
+            -0.0038712739995960753,
+            -0.023315508569288476,
+            -0.014963211090562272,
+            -0.01633110043829169,
+            -0.004241081909202262,
+            -0.008295402942465353,
+            0.004976396647108597,
+            -0.007402107755970892,
+            -0.020321297548927468,
+            0.0020986169430507757,
+            0.016803273183070004,
+            0.01434729320047051,
+            0.018612970083178984,
+            0.00603868809307069,
+            0.014278001198851581,
+            -0.0026350854954985117,
+            0.017171016209107656,
+            -0.006968968261909459,
+            0.006881847166565678,
+            0.011454001418121107,
+            0.02140935294963631,
+            0.011121833270235448,
+            -0.012801756637470525,
+            0.0010870041670931706,
+            -0.009655388034828094,
+            0.002094601818569396,
+            -0.00828486034340067,
+            -0.009190975158350585,
+            0.0017762581946855365,
+            -0.007261996868906865,
+            0.004674405202069334,
+            0.0030526253833799426,
+            -0.0014082232541965316,
+            0.010419302200840906,
+            -0.0024200053377751805,
+            -0.012208685875688886,
+            -0.0033037330470612695,
+            -0.017975940046938498,
+            -0.006958042903103732,
+            -0.0014496522238488239,
+            -0.011711502944602638,
+            -0.01016130418405875,
+            0.003321436189059608,
+            -0.00538282506552848,
+            -0.009353581612407435,
+            0.013884147453136417,
+            0.0001309738696853529,
+            0.0005878330075918333,
+            0.023495280602974504,
+            0.006035028203642483,
+            0.008087594627921345,
+            0.013144288945385904,
+            -0.01023995000994454,
+            -0.022574958230635715,
+            -0.0243435727055662,
+            0.007867022215096418,
+            -0.01573478573423379,
+            0.02324699908478915,
+            -0.007929809922043611,
+            -0.00561630876431978,
+            0.0020982723335057625,
+            -0.013699317926847062,
+            0.008578431496569186,
+            -0.011271381061801356,
+            0.0014450645788554738,
+            -0.019581451799757368,
+            0.018059311800267413,
+            -0.010917911869373693,
+            -0.004302687883018328,
+            0.009466103226841532,
+            0.034049086327052606,
+            -0.0005689605174080737,
+            -0.007410970644432269,
+            -0.006594150877740108,
+            -0.00022810271313786974,
+            0.018766765773409123,
+            -0.0029041815756733333,
+            0.005758781908059541,
+            -0.01563519107805783,
+            0.006571628287171105,
+            -0.006132372252825422,
+            0.0012423261344297225,
+            0.01513074831647933,
+            -0.026306320707079913,
+            -0.015491710256868008,
+            -0.007857352164317122,
+            -0.024893430640551818,
+            0.002253574511878983,
+            0.017579477884206534,
+            0.020505606775521504,
+            -0.020324505029926956,
+            -0.02513332780748072,
+            0.005150966733930881,
+            -0.026580994694344065,
+            0.0037390577082486594,
+            0.1241031878517593,
+            -0.002138404898309707,
+            0.002720723037908521,
+            -0.011425036886011179,
+            0.0022399461009316216,
+            0.005389407121650262,
+            -0.019552543955456567,
+            0.05950746473186985,
+            -0.00638023587296763,
+            -0.004023798021748334,
+            0.002170545188776296,
+            0.001785855390676113,
+            -0.005831534307818096,
+            0.008643501093778434,
+            -0.017602787281866816,
+            -0.02184792441891879,
+            -0.0052215705748838905,
+            -0.021252169339074325,
+            -0.008509621422071373,
+            -0.013000137524907565,
+            0.002893880203681282,
+            -0.015821609145923173,
+            0.010397692426614709,
+            -0.01638916243796692,
+            -0.010283706937701179,
+            -0.01103621454761223,
+            -0.00373552043529251,
+            0.01563553822905027,
+            -0.03188533919653949,
+            -0.004873550301425776,
+            -0.005183590054152726,
+            0.018218971950313273,
+            0.006004328765748325,
+            0.004357734370247252,
+            -0.014230318971458851,
+            0.0064792070801456235,
+            0.0215589173714926,
+            0.0055272493764401305,
+            0.014643335833970546,
+            0.010618070641357003,
+            -0.0022177303813575846,
+            -0.0024402761885758007,
+            0.017541799738501007,
+            -0.005445774414228888,
+            -0.00023502936119626386,
+            -0.00537569046685066,
+            -0.005974703247446444,
+            -0.010712244203425036,
+            0.01843613166635215,
+            -0.009938564643456672,
+            -0.017388592763277214,
+            -0.0015797412875366156,
+            0.02028434083164951,
+            -0.012761534382003973,
+            0.0031906812114950982,
+            0.000788436210275669,
+            0.020589196433948156,
+            0.0007726855314648301,
+            -0.005481260996289263,
+            0.007165014826294827,
+            -0.02956334397512344,
+            0.004095715232997753,
+            0.0003568512047626187,
+            0.027290768799658912,
+            -0.0017631129386615335,
+            0.01552542586826711,
+            -0.020871286597113327,
+            0.004388125824324094,
+            -0.016926008631435403,
+            0.005099899758322398,
+            0.012560923968841436,
+            0.002781049890810361,
+            0.010889021753593997,
+            -0.008500358897706648,
+            -0.021353460542554444,
+            0.012663934009161902,
+            -0.018657094130860155,
+            -0.029156017587920958,
+            0.0018889388761887053,
+            -0.028158076315864772,
+            -0.003048854609750186,
+            0.006812072971079798,
+            -0.035217412591558014,
+            -0.02251675180883659,
+            0.007396524512996277,
+            -0.00995571600201846,
+            -0.011912753666073882,
+            0.010675240017836214,
+            -0.02645155743041798,
+            -0.018137648527851668,
+            0.0010666944556449141,
+            0.012588800077165337,
+            -0.004561825939541096,
+            0.007555558090958502,
+            -0.028558603070876605,
+            0.014816816621445352,
+            -0.01313764261744744,
+            0.016036065063831813,
+            -0.0037797584923144134,
+            0.022217516257304868,
+            0.011131597210722573,
+            0.008795376259031849,
+            -0.004950080558710245,
+            0.021409590863550987,
+            0.005574351961953293,
+            0.051358937509801736,
+            0.031972009383746405,
+            0.0019821003238211276,
+            -0.003251257665378234,
+            0.003313045083311649,
+            0.008294776423730887,
+            -0.00940948257229233,
+            0.005114631360856872,
+            0.028053227377164806,
+            0.016738013668710083,
+            -0.018755924318109216,
+            0.006606711923035118,
+            -0.02702692774986498,
+            0.017950884766781056,
+            -0.022054653845163,
+            0.0003755944703013671,
+            0.007766253634286795,
+            0.0016206993755471613,
+            0.010682347575070117,
+            0.0013842930079185415,
+            0.01705234255261425,
+            0.023975191051210525,
+            -0.002976402980885077,
+            0.017466027107477605,
+            0.012734909275073793,
+            0.018143161773162015,
+            -0.0038399373197265827,
+            0.006929440921265761,
+            -0.012003095293274786,
+            -0.0008883470597690258,
+            0.0020490527756307267,
+            -0.004231091383919565,
+            0.012128848448049833,
+            0.008340302519160299,
+            0.012989849936937843,
+            0.002974488532907779,
+            0.003198854381461613,
+            -0.005309753760578859,
+            -0.0017024982049936384,
+            -0.01763395035509857,
+            -0.01819680226734474,
+            0.028417724274372468,
+            -0.020217879775485576,
+            -0.050118184195774004,
+            0.0025509548991664425,
+            0.014185761006969567,
+            -0.0012021669304388173,
+            -0.0051260576846881515,
+            -0.0005774778872927327,
+            0.005718864494530703,
+            -0.016918749210038116,
+            -0.0015472638835434902,
+            0.008125845907627804,
+            0.020347547062899797,
+            -0.002036218493206914,
+            -0.01079739518866248,
+            0.0038112958713600906,
+            0.02169339303105242,
+            0.01667594322285835,
+            0.012338193092567552,
+            -0.009798409233733167,
+            0.0003198215558702285,
+            -1.6308740825163337e-05,
+            0.0076870827874737325,
+            0.004356644356656038,
+            0.003500023005601613,
+            -0.019968549869439228,
+            -0.008634740687755112,
+            0.012781565963545847,
+            -0.0025637758121512927,
+            0.010243567793632335,
+            -0.00042296044205132426,
+            -0.003069251553118065,
+            -0.009344199449486244,
+            -0.004694953215957636,
+            -0.009583993744231791,
+            0.013783878036521561,
+            -0.007705982042252044,
+            -0.01549567538665377,
+            0.00020666298495823114,
+            0.0005515775451348472,
+            0.001667367278489879,
+            -0.004518431526236256,
+            0.03302324454129141,
+            0.009531990824506103,
+            -0.0004937348108340977,
+            -0.004047613286672229,
+            -0.0327383112522095,
+            0.012456150771224677,
+            -0.009677533214778067,
+            0.0006455811519981003,
+            0.020012976827426605,
+            -0.00020277620481825933,
+            0.002168432791818704,
+            -0.015657551629104767,
+            0.0032924357559238502,
+            0.006161295735841883,
+            0.013814165416788555,
+            0.0005164426482972658,
+            0.01076512534105537,
+            -0.006498977875968572,
+            -0.010325144017898257,
+            0.004804158067334203,
+            0.0104064855510473,
+            -0.0012188100892986933,
+            0.005171197790296134
+          ],
+          "title": "Mixture-Of-Experts With Expert Choice Routing",
+          "keyphrases": [
+            "Mixture-Of-Experts",
+            "Expert Choice Routing",
+            "sparsely-activated models",
+            "training efficiency",
+            "downstream performance"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "8bd19de5-c2d4-458d-833b-5962c425f915",
+        "properties": {
+          "page_content": "1 Introduction\n\nScaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27]. The final model quality has been found to have a power-law relationship with the amount of data, model size, and compute time [16, 20]. However, training efficiency, which is defined as the total amount of computation used to achieve superior model quality than the state of the art system [21], should receive greater attention as we increase our efforts towards green AI [29].\n\nSparsely gated mixture-of-experts [31] (MoE) provides an effective way to scale model capacity given a fixed computational cost, and has recently played an important role in increasing the training efficiency of large-scale language models [10, 21]. MoE operate by adopting a number of experts, each as a sub-network, and by activating only one or a few experts for each input token. A gating network must be chosen and optimized in order to route each token to the most suited expert(s). For example, recent work has implemented sparse routing via k-means clustering [12], linear assignment to maximize token-expert affinities [22], or hashing [8, 28]. Many of the prior work use a routing strategy concerning the token choice, where each token selects the best one or two experts.\n\nWe argue that the independent token choice of prior work often leads to an imbalanced load of experts, which causes training inefficiency and sub-optimal training of the model. In order to mitigate this\n\nissue, previous sparsely gated networks introduce additional auxiliary losses as regularization to prevent too many tokens being routed to a single expert, but the effectiveness is still limited. Recent approaches [8, 22, 28] explore alternative strategies for routing, but they focus on pre-training only and do not demonstrate performance gain on downstream tasks. Moreover, none of the previous methods consider allocating a variable number of experts to each token based on importance, which can be beneficial.\n\nWe propose a very simple yet effective routing method we are calling expert choice. Unlike conventional MoE where tokens select one or two top-scoring experts, our method lets each expert pick the top-k tokens. Our method guarantees perfect load balancing, allows a variable number of experts for each token, and achieves substantial gains in training efficiency and downstream performance as demonstrated in our experiments. Our major contributions include:\n\nWe identify common pitfalls in conventional MoE such as load imbalance as described in Section 3.1. We then propose a heterogeneous, expert choice method to provide a fluid allocation of model parameters based on a learnt token-to-expert importance. This method intrinsically guarantees load balance without imposing an auxiliary loss.\n\nWe show our method provides over 2× faster training convergence in a 8B/64E (8 billion activated parameters, 64 experts) model, compared to the top-1 and top-2 gating counterparts in Switch Transformer [10] and GShard [21].\n\nWe show our method demonstrates strong scaling when increasing the number of experts from 16 to 128, evaluated in training perplexity.\n\nWe show our method demonstrates strong performance on downstream tasks selected from GLUE and SuperGLUE at all the evaluated scales. More specifically, our 8B/64E model outperforms a T5 11B dense model in 7 out of 11 tasks evaluated.\n\n",
+          "title": "Introduction",
+          "embedding": [
+            0.0024422307033091784,
+            -0.014919951558113098,
+            -0.01768755167722702,
+            -0.02871091105043888,
+            0.020494353026151657,
+            0.03838575258851051,
+            -0.01718577742576599,
+            0.0458810068666935,
+            -0.00809895433485508,
+            0.02212512120604515,
+            -0.01569613441824913,
+            -0.011674097739160061,
+            0.024147899821400642,
+            -0.016072465106844902,
+            -0.003586903912946582,
+            0.029040200635790825,
+            0.00016084715025499463,
+            0.0012985375942662358,
+            0.021921275183558464,
+            -0.021889913827180862,
+            -0.0052411919459700584,
+            -0.014426017180085182,
+            -0.01895766891539097,
+            -0.002695078030228615,
+            0.025621861219406128,
+            0.010239336639642715,
+            0.029526295140385628,
+            -0.00020776601741090417,
+            -0.05397212132811546,
+            0.04054965451359749,
+            0.015868619084358215,
+            0.025010323151946068,
+            -0.020776601508259773,
+            -0.014253532513976097,
+            -0.047417692840099335,
+            -0.029228366911411285,
+            0.05165141448378563,
+            0.018314769491553307,
+            -0.027127185836434364,
+            0.030765051022171974,
+            0.0260295532643795,
+            0.018973348662257195,
+            -0.037350840866565704,
+            -0.009784603491425514,
+            0.0013112779706716537,
+            0.00785982795059681,
+            -0.029761500656604767,
+            0.00613497756421566,
+            0.022031037136912346,
+            -0.01818932592868805,
+            0.02342659793794155,
+            -0.01548444852232933,
+            0.015492288395762444,
+            0.006064415443688631,
+            -0.06635968387126923,
+            0.04058101400732994,
+            0.0031008098740130663,
+            0.008036232553422451,
+            0.05513247475028038,
+            0.019961217418313026,
+            0.018205007538199425,
+            -0.0099727688357234,
+            -0.04525379091501236,
+            0.008130315691232681,
+            0.006036974955350161,
+            -0.019506484270095825,
+            -0.008647770620882511,
+            0.015061075799167156,
+            0.023410918191075325,
+            0.01010605227202177,
+            -0.055038392543792725,
+            0.029275408014655113,
+            -0.019929856061935425,
+            0.005562641192227602,
+            0.006781796459108591,
+            -0.01776595413684845,
+            0.01925559714436531,
+            0.01078031212091446,
+            0.005041265860199928,
+            0.0031556913163512945,
+            -0.01123504526913166,
+            0.038072142750024796,
+            0.004821739625185728,
+            0.014802348800003529,
+            0.04224314168095589,
+            -0.03951474279165268,
+            0.0034300992265343666,
+            -0.06153010204434395,
+            0.01611950621008873,
+            -0.012019068002700806,
+            -0.030733689665794373,
+            -0.003298775525763631,
+            -0.01593918167054653,
+            0.017828676849603653,
+            -0.003900512820109725,
+            0.041521843522787094,
+            0.011423210613429546,
+            -0.006993482355028391,
+            -0.02982422336935997,
+            -0.0009119163732975721,
+            -0.005151029210537672,
+            0.017985479906201363,
+            0.0025931550189852715,
+            -0.023175710812211037,
+            0.019396722316741943,
+            -0.03333664685487747,
+            -0.033023037016391754,
+            0.035281021147966385,
+            0.017907077446579933,
+            0.05739046260714531,
+            0.024100856855511665,
+            -0.011509452946484089,
+            0.011870103888213634,
+            0.025621861219406128,
+            -0.02924404665827751,
+            -0.017279859632253647,
+            -0.03810350224375725,
+            -0.06316086649894714,
+            0.019020389765501022,
+            -0.013657675124704838,
+            0.01680944673717022,
+            0.006762195844203234,
+            -0.012716848403215408,
+            -0.008624250069260597,
+            -0.06369400024414062,
+            0.045347873121500015,
+            -0.014026165939867496,
+            -0.011352648958563805,
+            -0.0018924347823485732,
+            -0.02450854890048504,
+            -0.003341896692290902,
+            0.028397301211953163,
+            -0.03418339043855667,
+            -0.05814312398433685,
+            0.040330126881599426,
+            0.013148060999810696,
+            0.03534374386072159,
+            0.04365438222885132,
+            -0.015782376751303673,
+            -0.022234883159399033,
+            0.026217719539999962,
+            -0.035124216228723526,
+            0.007946070283651352,
+            -0.0005517559475265443,
+            -0.02668813243508339,
+            0.01322646252810955,
+            0.010011970065534115,
+            0.06447802484035492,
+            -0.02052571438252926,
+            0.021889913827180862,
+            -0.02918132394552231,
+            0.057108212262392044,
+            -0.016260631382465363,
+            0.02201535739004612,
+            -0.023018905892968178,
+            -0.0031008098740130663,
+            -0.031219784170389175,
+            0.0018375532235950232,
+            -0.019584886729717255,
+            -0.011485932394862175,
+            -0.02443014644086361,
+            -0.01301477663218975,
+            -0.007569739129394293,
+            0.01749938726425171,
+            0.010686228983104229,
+            0.024931922554969788,
+            0.030827771872282028,
+            0.002036498859524727,
+            -0.07018571346998215,
+            0.026468606665730476,
+            0.018205007538199425,
+            0.01676240563392639,
+            0.01731122098863125,
+            0.019004710018634796,
+            0.006217299960553646,
+            0.026656771078705788,
+            0.011148802936077118,
+            0.011195844039320946,
+            -0.008059753105044365,
+            0.03985971212387085,
+            -0.03716267645359039,
+            -0.007585419807583094,
+            0.01726417988538742,
+            0.0046766954474151134,
+            0.026249079033732414,
+            0.018079563975334167,
+            0.01726417988538742,
+            0.012748208828270435,
+            0.007448215503245592,
+            0.01121936459094286,
+            0.00966699980199337,
+            0.03672362491488457,
+            0.01779731549322605,
+            -0.01638607494533062,
+            0.016433116048574448,
+            -0.02817777544260025,
+            -0.01050590444356203,
+            -0.004990304354578257,
+            0.02722126804292202,
+            0.00673867529258132,
+            -0.02339523658156395,
+            -0.0031027698423713446,
+            0.009196585975587368,
+            -0.022532811388373375,
+            0.008702651597559452,
+            0.006174178794026375,
+            -6.774446228519082e-05,
+            0.009525875560939312,
+            0.007271810434758663,
+            -0.030043749138712883,
+            0.023630443960428238,
+            -0.01404968649148941,
+            -0.008726172149181366,
+            -0.01792275905609131,
+            -0.007032683584839106,
+            -0.000910936389118433,
+            0.004002436064183712,
+            0.006323142908513546,
+            0.006664192769676447,
+            -0.022501451894640923,
+            -0.004837419837713242,
+            0.011454571969807148,
+            0.02488487958908081,
+            0.002355988137423992,
+            -0.0009266168344765902,
+            -0.020353229716420174,
+            -0.006664192769676447,
+            -0.01840885356068611,
+            0.04258811101317406,
+            -0.005386236123740673,
+            -0.03223901242017746,
+            0.009902206249535084,
+            0.014849389903247356,
+            0.010200135409832,
+            0.010286377742886543,
+            -0.042556751519441605,
+            0.00047947888378985226,
+            0.0019267357420176268,
+            0.005268632434308529,
+            -0.01114096213132143,
+            -0.019114473834633827,
+            0.019882814958691597,
+            -0.051431890577077866,
+            -0.014347615651786327,
+            0.029306767508387566,
+            0.02737807296216488,
+            -0.020070981234312057,
+            0.01941240206360817,
+            -0.007851987145841122,
+            -0.004453248810023069,
+            -0.0246967151761055,
+            -0.024336064234375954,
+            0.01234835758805275,
+            0.014378976076841354,
+            -0.0033595373388379812,
+            0.03434019535779953,
+            0.016213588416576385,
+            -0.034089308232069016,
+            -0.017091695219278336,
+            -0.0149669935926795,
+            0.050052009522914886,
+            -0.013006936758756638,
+            -0.015194360166788101,
+            0.06372536718845367,
+            -0.0255748201161623,
+            0.012622765265405178,
+            0.03515557944774628,
+            0.014684745110571384,
+            -0.002234464744105935,
+            -0.024947602301836014,
+            -0.0025970751885324717,
+            -0.05425437167286873,
+            -0.0051588695496320724,
+            -0.013720396906137466,
+            0.04224314168095589,
+            -0.04142775759100914,
+            -0.004076918121427298,
+            -0.005880170036107302,
+            0.01577453687787056,
+            0.03024759516119957,
+            0.006785716395825148,
+            0.00034153988235630095,
+            0.01049022376537323,
+            0.02308162860572338,
+            -0.0034457796718925238,
+            -0.014465218409895897,
+            -0.005151029210537672,
+            -0.018471574410796165,
+            -0.04838988184928894,
+            -0.011046879924833775,
+            -0.01742098480463028,
+            -0.004813899286091328,
+            -0.031454991549253464,
+            -0.00391815323382616,
+            0.03531238064169884,
+            -0.010647027753293514,
+            -0.015421726740896702,
+            0.008522327058017254,
+            0.00809895433485508,
+            -0.022877782583236694,
+            -0.04719816520810127,
+            0.004974624142050743,
+            0.00666027283295989,
+            -0.011321287602186203,
+            -0.0572650171816349,
+            0.006750435568392277,
+            -0.02461831271648407,
+            0.01837749220430851,
+            -0.021999677643179893,
+            -0.0099727688357234,
+            0.025872748345136642,
+            0.0016327272169291973,
+            0.0045120506547391415,
+            -0.02573162503540516,
+            0.0011064520804211497,
+            0.022595534101128578,
+            0.006699474062770605,
+            0.021105891093611717,
+            -0.027958249673247337,
+            -0.03418339043855667,
+            0.020839322358369827,
+            -0.0055077592842280865,
+            -0.06350583583116531,
+            -0.0026676373090595007,
+            -0.02251713164150715,
+            0.0027009581681340933,
+            0.019161514937877655,
+            0.04694727808237076,
+            -0.030184874311089516,
+            0.008663450367748737,
+            -0.023646123707294464,
+            -0.04650822654366493,
+            -0.030467122793197632,
+            -0.0027303590904921293,
+            -0.007957830093801022,
+            -0.026547009125351906,
+            -0.012991256080567837,
+            0.008467445150017738,
+            -0.03901296854019165,
+            0.009274988435208797,
+            -0.014245692640542984,
+            -0.026233399286866188,
+            -0.028648190200328827,
+            0.010427501983940601,
+            0.00468061538413167,
+            -0.002107060980051756,
+            -0.03603368252515793,
+            0.0060722557827830315,
+            0.004939342848956585,
+            -0.005162789486348629,
+            0.0010476503521203995,
+            0.002673517446964979,
+            0.004296444356441498,
+            0.005880170036107302,
+            -0.0027538796421140432,
+            -0.04183544963598251,
+            -0.026750855147838593,
+            0.016072465106844902,
+            0.02090204507112503,
+            -0.001751310657709837,
+            0.029197005555033684,
+            -0.050898753106594086,
+            -0.03230173513293266,
+            0.025951150804758072,
+            -0.06002477928996086,
+            0.006272181402891874,
+            -0.011995547451078892,
+            -0.022376008331775665,
+            -0.00047531374730169773,
+            -0.014073207043111324,
+            -0.02063547819852829,
+            0.026249079033732414,
+            0.02626476064324379,
+            0.017624830827116966,
+            0.017938438802957535,
+            0.013336226344108582,
+            0.02342659793794155,
+            -0.02220352366566658,
+            -0.06391353160142899,
+            0.035625990480184555,
+            -0.01566477306187153,
+            0.007781425025314093,
+            0.017860036343336105,
+            -0.012536522932350636,
+            -0.004010275937616825,
+            0.0303573589771986,
+            0.011713298968970776,
+            0.004022036213427782,
+            -0.018346130847930908,
+            0.05221591144800186,
+            -0.005986013449728489,
+            -0.04886029288172722,
+            -0.0021325417328625917,
+            -0.012120991013944149,
+            0.023489320650696754,
+            0.023348195478320122,
+            -0.00301064713858068,
+            -0.028742272406816483,
+            -0.022830741479992867,
+            -0.007197328377515078,
+            -0.015194360166788101,
+            0.02798961102962494,
+            -0.00789902824908495,
+            0.01466122455894947,
+            -0.0194908045232296,
+            -0.001328918500803411,
+            -0.004104358609765768,
+            0.01139185018837452,
+            0.04287036135792732,
+            -0.040643736720085144,
+            -0.009235787205398083,
+            -0.03361889347434044,
+            -0.03261534497141838,
+            0.00725613022223115,
+            0.017562108114361763,
+            0.03844847157597542,
+            -0.02182719111442566,
+            0.01078031212091446,
+            0.01853429712355137,
+            -0.010349099524319172,
+            -0.012418919242918491,
+            0.03393250331282616,
+            0.04873485118150711,
+            0.03452835977077484,
+            0.011956346221268177,
+            -0.00046306339208967984,
+            0.008992739953100681,
+            0.011415370739996433,
+            0.01152513362467289,
+            0.012967735528945923,
+            0.029338128864765167,
+            -0.008067593909800053,
+            0.006969961803406477,
+            -0.018126605078577995,
+            -0.020118022337555885,
+            -0.01611950621008873,
+            -0.011713298968970776,
+            0.030529843643307686,
+            -0.011768180876970291,
+            -0.005782167427241802,
+            0.0294792540371418,
+            -0.008498805575072765,
+            0.00907114241272211,
+            -0.009392592124640942,
+            -0.023755887523293495,
+            -0.03085913322865963,
+            -0.04202361777424812,
+            -0.005292152985930443,
+            -0.007753984536975622,
+            -0.035092856734991074,
+            -0.0529058501124382,
+            0.010380460880696774,
+            -0.07514073699712753,
+            -0.002481431933119893,
+            0.029965346679091454,
+            0.0010692109353840351,
+            0.0008305740193463862,
+            -0.0010888115502893925,
+            0.008208717219531536,
+            -0.03763309121131897,
+            0.0031478512100875378,
+            0.02243872918188572,
+            0.03089049458503723,
+            -0.0294792540371418,
+            -0.010991998016834259,
+            0.007103245705366135,
+            -0.03879344463348389,
+            -0.0038828724063932896,
+            0.0039887153543531895,
+            0.005190230440348387,
+            0.013312705792486668,
+            -0.03227037563920021,
+            -0.0069033196195960045,
+            0.004982464015483856,
+            0.01630767248570919,
+            0.03835438936948776,
+            -0.0023677486460655928,
+            -0.04512834548950195,
+            0.019004710018634796,
+            -0.0007732423837296665,
+            0.01531196292489767,
+            -0.00893785897642374,
+            -0.020839322358369827,
+            -0.0015778456581756473,
+            -0.00539799639955163,
+            0.0006272181635722518,
+            0.023332515731453896,
+            -0.008530166931450367,
+            0.01787571795284748,
+            -0.01707601360976696,
+            -0.015210039913654327,
+            -0.002385389059782028,
+            -0.006679873447865248,
+            0.018863586708903313,
+            0.0031125701498240232,
+            -0.021121570840477943,
+            -0.026092275977134705,
+            -0.017217138782143593,
+            0.027503516525030136,
+            0.0035927840508520603,
+            -0.040298767387866974,
+            -0.012795249931514263,
+            0.024289023131132126,
+            0.019726011902093887,
+            0.006613231264054775,
+            -0.019616248086094856,
+            0.008208717219531536,
+            0.03024759516119957,
+            -0.044657934457063675,
+            -0.02522985078394413,
+            0.0026931180618703365,
+            0.0033066156320273876,
+            -0.004888381343334913,
+            -0.011148802936077118,
+            0.0006757295341230929,
+            0.04365438222885132,
+            -0.010576466098427773,
+            0.021717429161071777,
+            0.019631927832961082,
+            0.01684080809354782,
+            -0.01898903027176857,
+            -0.013649835251271725,
+            -0.011454571969807148,
+            0.025010323151946068,
+            -0.014669064432382584,
+            0.027205588296055794,
+            0.022219203412532806,
+            0.002944005187600851,
+            0.06447802484035492,
+            -0.026672452688217163,
+            -0.025559140369296074,
+            0.011705459095537663,
+            -0.013579272665083408,
+            0.0009697380592115223,
+            -0.019475122913718224,
+            -0.007812785916030407,
+            -0.004092598333954811,
+            0.022109439596533775,
+            -0.028914757072925568,
+            -0.028507065027952194,
+            0.0017552308272570372,
+            0.010129572823643684,
+            -0.009925727732479572,
+            0.01257572416216135,
+            0.0027950408402830362,
+            0.004539491608738899,
+            0.0212470144033432,
+            -0.04149048030376434,
+            0.011681938543915749,
+            0.035469185560941696,
+            -0.02477511763572693,
+            -0.034402914345264435,
+            -0.031219784170389175,
+            0.011862263083457947,
+            0.0029890865553170443,
+            -0.04434432461857796,
+            -0.015037555247545242,
+            -0.029996708035469055,
+            -0.0015660853823646903,
+            -0.03572007268667221,
+            -0.00958859734237194,
+            0.010082531720399857,
+            -0.0037143074441701174,
+            0.01183090265840292,
+            -0.022987544536590576,
+            -0.005445037502795458,
+            0.028852034360170364,
+            0.04478337615728378,
+            0.012168032117187977,
+            0.020337548106908798,
+            -0.001616066787391901,
+            -0.024978963658213615,
+            -0.015492288395762444,
+            0.012599244713783264,
+            -0.0007776524871587753,
+            0.0031968525145202875,
+            0.0294478926807642,
+            -0.004907981958240271,
+            -0.03763309121131897,
+            0.00632706331089139,
+            0.04095734655857086,
+            -0.020572755485773087,
+            0.007146366871893406,
+            0.04192953556776047,
+            0.025935471057891846,
+            -0.019239917397499084,
+            -0.020008258521556854,
+            -0.031298186630010605,
+            0.004951103124767542,
+            -0.02817777544260025,
+            -0.005201990716159344,
+            0.03688042610883713,
+            -0.006185939069837332,
+            -0.026390204206109047,
+            0.012262115254998207,
+            -0.02297186478972435,
+            -0.009337710216641426,
+            0.01278741005808115,
+            0.01100767869502306,
+            -0.03625321015715599,
+            0.013328385539352894,
+            0.019537845626473427,
+            0.023332515731453896,
+            -0.008882977068424225,
+            -0.018973348662257195,
+            0.021074529737234116,
+            0.006166338454931974,
+            0.0032301736064255238,
+            -0.004202361684292555,
+            0.0075501385144889355,
+            0.019773053005337715,
+            -0.0036045443266630173,
+            0.011305606923997402,
+            -0.0034300992265343666,
+            0.03317984193563461,
+            -0.014473059214651585,
+            -0.002887163544073701,
+            0.01188578363507986,
+            -0.014622023329138756,
+            -0.009314189665019512,
+            0.059523001313209534,
+            0.007213008590042591,
+            0.040486931800842285,
+            -0.003974995110183954,
+            0.018847905099391937,
+            0.024633992463350296,
+            -0.01627631112933159,
+            -0.027205588296055794,
+            0.0006580890621989965,
+            -0.013869361020624638,
+            -0.03085913322865963,
+            -0.01699761115014553,
+            -0.03534374386072159,
+            -0.018471574410796165,
+            -0.00014565671153832227,
+            -0.01404968649148941,
+            0.0007394314161501825,
+            0.028428662568330765,
+            0.010239336639642715,
+            -0.012168032117187977,
+            -0.033901140093803406,
+            0.03885616362094879,
+            0.030874812975525856,
+            0.03261534497141838,
+            -0.007236529607325792,
+            0.019271278753876686,
+            -0.010074691846966743,
+            -0.0058135283179581165,
+            0.020807962864637375,
+            0.015437406487762928,
+            -0.00238342909142375,
+            0.014825869351625443,
+            0.01577453687787056,
+            0.0459437295794487,
+            -0.025621861219406128,
+            -0.03024759516119957,
+            -0.010176614858210087,
+            -0.0260295532643795,
+            0.027817124500870705,
+            0.012983416207134724,
+            -0.015445247292518616,
+            0.015727495774626732,
+            0.017044654116034508,
+            -0.029620377346873283,
+            0.02010234259068966,
+            -0.01374391745775938,
+            -0.03195676580071449,
+            0.030325997620821,
+            -0.016825126484036446,
+            0.013242143206298351,
+            -0.007013082969933748,
+            -0.004033796954900026,
+            -0.020086660981178284,
+            0.009298508986830711,
+            0.01131344772875309,
+            -0.028554106131196022,
+            0.038072142750024796,
+            0.04368574544787407,
+            0.003083169227465987,
+            0.009604278020560741,
+            0.014574982225894928,
+            -0.0007943129749037325,
+            -0.020353229716420174,
+            -0.012724688276648521,
+            0.006679873447865248,
+            -0.001151533331722021,
+            -0.028836354613304138,
+            0.0307807307690382,
+            -0.016103826463222504,
+            0.007495257072150707,
+            -0.01154081430286169,
+            -0.006087936460971832,
+            0.0063192229717969894,
+            -0.0007805926143191755,
+            0.006558349821716547,
+            0.0007997031207196414,
+            -0.03170587867498398,
+            0.014888591133058071,
+            0.031110020354390144,
+            -0.005758646875619888,
+            -0.01079599279910326,
+            0.01737394370138645,
+            -0.02090204507112503,
+            0.009180906228721142,
+            0.028005290776491165,
+            0.015186519362032413,
+            0.004206281621009111,
+            -0.0051980703137815,
+            -0.030310317873954773,
+            -0.020243465900421143,
+            -0.001436721533536911,
+            0.024304702877998352,
+            0.014841550029814243,
+            0.026437245309352875,
+            0.026735173538327217,
+            -0.00780886597931385,
+            0.0013436188455671072,
+            -0.0022677856031805277,
+            -0.03396386280655861,
+            0.04440704733133316,
+            -1.7808952179620974e-05,
+            0.01779731549322605,
+            0.021152932196855545,
+            -0.027095824480056763,
+            -0.027268309146165848,
+            -0.001135852886363864,
+            0.032333094626665115,
+            0.03484196960926056,
+            0.01699761115014553,
+            0.004590453114360571,
+            0.04202361777424812,
+            -0.02464967407286167,
+            -0.03741356357932091,
+            0.017891397699713707,
+            0.030043749138712883,
+            -0.009188746102154255,
+            0.0016738884150981903,
+            0.027017422020435333,
+            -0.011838742531836033,
+            -0.005574401468038559,
+            0.024367425590753555,
+            0.019083112478256226,
+            0.008914338424801826,
+            0.004104358609765768,
+            -0.009659159928560257,
+            0.02369316667318344,
+            0.002107060980051756,
+            0.024336064234375954,
+            -0.01960056647658348,
+            0.007318852003663778,
+            0.00945531390607357,
+            -0.026891978457570076,
+            0.033023037016391754,
+            0.009941407479345798,
+            -0.0502401739358902,
+            0.025668902322649956,
+            0.0032517341896891594,
+            -0.004304284695535898,
+            0.003394818166270852,
+            0.0059076109901070595,
+            0.007789265364408493,
+            0.023363877087831497,
+            0.00446108914911747,
+            -0.004096518736332655,
+            0.030106471851468086,
+            0.023912692442536354,
+            -0.0003197342448402196,
+            0.020666837692260742,
+            -0.0019110552966594696,
+            -0.02074524015188217,
+            -0.002781320596113801,
+            -0.0063780248165130615,
+            0.04105142876505852,
+            -0.0390443317592144,
+            0.014825869351625443,
+            0.02374020777642727,
+            0.003420298919081688,
+            -0.002857762621715665,
+            -0.02320707216858864,
+            -0.008443924598395824,
+            -0.011180163361132145,
+            0.024257661774754524,
+            -0.010145253501832485,
+            0.023301154375076294,
+            -0.02833458036184311,
+            -0.0038926727138459682,
+            0.01353223156183958,
+            -0.030498482286930084,
+            0.011031199246644974,
+            0.02722126804292202,
+            0.0010897915344685316,
+            -0.02010234259068966,
+            -0.005852729547768831,
+            0.036849066615104675,
+            0.055226556956768036,
+            0.0054959990084171295,
+            0.01630767248570919,
+            0.02867954969406128,
+            -0.02140381932258606,
+            -0.0106235072016716,
+            0.019537845626473427,
+            -0.018126605078577995,
+            0.014582822099328041,
+            0.04302716627717018,
+            -0.036347292363643646,
+            -0.03782125562429428,
+            -0.008130315691232681,
+            -0.032333094626665115,
+            0.008898657746613026,
+            -0.012003387324512005,
+            0.012889333069324493,
+            -0.009204426780343056,
+            -0.02841298282146454,
+            0.02798961102962494,
+            0.015170839615166187,
+            0.009384751319885254,
+            -0.014622023329138756,
+            -0.02182719111442566,
+            -0.02411653846502304,
+            -0.031078658998012543,
+            -0.008898657746613026,
+            -0.01765619032084942,
+            0.01975737139582634,
+            -0.0006556389853358269,
+            -0.04224314168095589,
+            -0.007848067209124565,
+            0.0194908045232296,
+            -0.012120991013944149,
+            0.040675096213817596,
+            0.02416357956826687,
+            0.016558559611439705,
+            -0.009267148561775684,
+            -0.03760172799229622,
+            0.05425437167286873,
+            -2.794612191792112e-05,
+            0.03027895651757717,
+            -0.013179421424865723,
+            -0.007655981462448835,
+            0.0026107956655323505,
+            0.027958249673247337,
+            0.005201990716159344,
+            0.009470994584262371,
+            -0.031909722834825516,
+            0.011078240349888802,
+            -0.024304702877998352,
+            -0.011932825669646263,
+            -0.006899399682879448,
+            -0.0034673402551561594,
+            0.02450854890048504,
+            0.00416316045448184,
+            0.006217299960553646,
+            0.02875795215368271,
+            0.012599244713783264,
+            -0.0017483705887570977,
+            -0.023834289982914925,
+            -0.013493030332028866,
+            -0.0015964661724865437,
+            0.003849551547318697,
+            -0.01676240563392639,
+            -0.013053977862000465,
+            -0.03835438936948776,
+            -0.010748950764536858,
+            0.0034810607321560383,
+            0.021152932196855545,
+            0.02151358313858509,
+            0.005688084755092859,
+            -0.010819513350725174,
+            -0.03085913322865963,
+            -0.007405094336718321,
+            -0.04735497012734413,
+            -0.016001902520656586,
+            0.018628379330039024,
+            -0.021043168380856514,
+            0.010356939397752285,
+            0.00260295532643795,
+            0.01590782031416893,
+            -0.0012289555743336678,
+            -0.01925559714436531,
+            0.003136090701445937,
+            -0.009086823090910912,
+            0.011548654176294804,
+            0.0010986118577420712,
+            0.03932657837867737,
+            -0.004084758460521698,
+            0.027660321444272995,
+            -0.029604697600007057,
+            0.008585048839449883,
+            -0.044124796986579895,
+            -0.0097610829398036,
+            -0.03866799920797348,
+            0.009855165146291256,
+            -0.008161676116287708,
+            -0.0338384211063385,
+            0.004523810930550098,
+            0.008357682265341282,
+            -0.002804841147735715,
+            -0.015892140567302704,
+            -0.006468187086284161,
+            0.02294050343334675,
+            -0.0047668581828475,
+            0.008953538723289967,
+            -0.03938930109143257,
+            -0.016166547313332558,
+            -0.007510937284678221,
+            0.03675498440861702,
+            -0.0036241449415683746,
+            0.017436664551496506,
+            -0.015727495774626732,
+            0.016213588416576385,
+            -0.04443840682506561,
+            -0.025825707241892815,
+            0.034057945013046265,
+            0.017389623448252678,
+            -0.021874234080314636,
+            0.026625411584973335,
+            0.019710330292582512,
+            0.04042420908808708,
+            -0.0580490417778492,
+            0.023802928626537323,
+            -0.0018473535310477018,
+            -0.009008420631289482,
+            0.019098792225122452,
+            0.010756791569292545,
+            0.03289759159088135,
+            -0.028961798176169395,
+            -0.014441697858273983,
+            0.006017374340444803,
+            0.020666837692260742,
+            8.042357512749732e-05,
+            0.006691633723676205,
+            -0.02358340285718441,
+            0.014496579766273499,
+            0.017562108114361763,
+            0.005190230440348387,
+            0.02703310176730156,
+            -0.026390204206109047,
+            0.028428662568330765,
+            0.014324095100164413,
+            -0.027770083397626877,
+            0.02132541686296463,
+            -0.027127185836434364,
+            -0.01390072237700224,
+            -0.001531784306280315,
+            0.02780144475400448,
+            0.011360488831996918,
+            -0.060244303196668625,
+            0.02063547819852829,
+            0.03173723816871643,
+            -0.006189859006553888,
+            0.03134522587060928,
+            -0.024053815752267838,
+            0.024665353819727898,
+            0.021372457966208458,
+            0.022799380123615265,
+            0.018440213054418564,
+            0.046037811785936356,
+            -0.00785982795059681,
+            0.01184658333659172,
+            -0.020118022337555885,
+            0.022579854354262352,
+            0.011595695279538631,
+            -0.009541556239128113,
+            -0.037570368498563766,
+            -0.021152932196855545,
+            0.007648141589015722,
+            0.013469509780406952,
+            -0.00831064023077488,
+            -0.03317984193563461,
+            -0.0458810068666935,
+            0.010090372525155544,
+            0.00569592509418726,
+            -0.057484544813632965,
+            -0.013555752113461494,
+            -0.009110343642532825,
+            0.005229431204497814,
+            -0.008459605276584625,
+            0.004061237443238497,
+            -0.028397301211953163,
+            0.015335483476519585,
+            -0.014457378536462784,
+            0.03716267645359039,
+            -0.011485932394862175,
+            0.022140800952911377,
+            -0.015837257727980614,
+            -0.004719816613942385,
+            -0.025292571634054184,
+            0.02488487958908081,
+            -0.021466542035341263,
+            -0.04199225455522537,
+            0.010835194028913975,
+            0.004480689764022827,
+            0.01622927002608776,
+            0.0537525974214077,
+            -0.005884090438485146,
+            -0.010317739099264145,
+            0.020321868360042572,
+            -0.019506484270095825,
+            -0.014002645388245583,
+            0.007679502479732037,
+            -0.005844889208674431,
+            -0.022031037136912346,
+            -0.01665264181792736,
+            -0.012262115254998207,
+            0.0247280765324831,
+            -0.00041602205601520836,
+            0.022611213847994804,
+            0.026390204206109047,
+            0.0195221658796072,
+            -0.02814641408622265,
+            0.009855165146291256,
+            -0.005127508658915758,
+            -0.011031199246644974,
+            0.004245482850819826,
+            0.021356778219342232,
+            -0.004755097441375256,
+            0.009000580757856369,
+            -0.01646447740495205,
+            -0.021121570840477943,
+            -0.03731948137283325,
+            -0.01249732170253992,
+            -0.003145891008898616,
+            0.0031478512100875378,
+            -0.035625990480184555,
+            -0.018126605078577995,
+            0.026060914620757103,
+            0.03070232830941677,
+            -0.028303219005465508,
+            0.0019051751587539911,
+            -0.002246225019916892,
+            -0.01132912840694189,
+            -0.003904432989656925,
+            -0.011791701428592205,
+            0.007040523923933506,
+            -0.015225720591843128,
+            8.691626862855628e-05,
+            -0.002042379230260849,
+            0.0023736287839710712,
+            -0.05792359635233879,
+            -0.017781633883714676,
+            0.02913428284227848,
+            0.002930284710600972,
+            0.00550383934751153,
+            -0.008279279805719852,
+            -0.01322646252810955,
+            -0.007009163033217192,
+            -0.02679789625108242,
+            0.024367425590753555,
+            0.015437406487762928,
+            0.019647609442472458,
+            -0.003559462958946824,
+            0.0017885518027469516,
+            0.018659740686416626,
+            0.0014171210350468755,
+            0.003351696999743581,
+            -0.007279650773853064,
+            0.006507388316094875,
+            0.02182719111442566,
+            -0.012795249931514263,
+            0.0158294178545475,
+            0.005778247490525246,
+            0.002685277722775936,
+            -0.02679789625108242,
+            -0.003682946553453803,
+            0.016793766990303993,
+            0.002657837001606822,
+            0.005296073388308287,
+            -0.0255434587597847,
+            0.038072142750024796,
+            0.04039284959435463,
+            0.025872748345136642,
+            0.020337548106908798,
+            -0.03220765292644501,
+            -0.004206281621009111,
+            -0.00758149940520525,
+            0.01121936459094286,
+            0.021811511367559433,
+            0.020055299624800682,
+            -0.010427501983940601,
+            -0.015743175521492958,
+            0.011062560603022575,
+            -0.02013370208442211,
+            -0.0019473163411021233,
+            0.018894946202635765,
+            0.011266405694186687,
+            -0.006201619748026133,
+            0.009988449513912201,
+            0.0004351325915195048,
+            0.005268632434308529,
+            0.02140381932258606,
+            -0.030529843643307686,
+            -0.025167128071188927,
+            0.0212470144033432,
+            -0.017123056575655937,
+            0.01154081430286169,
+            0.004876621067523956,
+            0.0032517341896891594,
+            0.020776601508259773,
+            0.07445079833269119,
+            -0.020070981234312057,
+            0.0043591661378741264,
+            -0.022768018767237663,
+            -0.02576298639178276,
+            -0.012614925391972065,
+            -0.022109439596533775,
+            0.005378395784646273,
+            0.013571432791650295,
+            -0.00024022944853641093,
+            0.020651157945394516,
+            0.01056862622499466,
+            -0.00644074659794569,
+            0.007216928992420435,
+            -0.01173681952059269,
+            -0.0007139506633393466,
+            -0.006585790775716305,
+            -0.0027087985072284937,
+            0.002058059675619006,
+            0.0015229640994220972,
+            0.002697037998586893,
+            0.02836594171822071,
+            0.012371878139674664,
+            0.013116699643433094,
+            -0.03327392414212227,
+            0.009815963916480541,
+            0.02151358313858509,
+            0.018095243722200394,
+            -0.03317984193563461,
+            -0.021388139575719833,
+            0.004543411545455456,
+            0.005017745308578014,
+            0.012928534299135208,
+            0.009196585975587368,
+            -0.006977802142500877,
+            0.02504168450832367,
+            -0.0355319082736969,
+            0.01814228482544422,
+            0.016527198255062103,
+            -0.011705459095537663,
+            -0.004931502975523472,
+            0.01986713521182537,
+            -0.0020090581383556128,
+            0.006860198453068733,
+            -0.005946812219917774,
+            0.0025794347748160362,
+            0.00653874920681119,
+            -0.021356778219342232,
+            -0.003559462958946824,
+            -0.012983416207134724,
+            -0.001520024030469358,
+            0.021419500932097435,
+            -0.017170097678899765,
+            0.008726172149181366,
+            -0.008655610494315624,
+            0.028836354613304138,
+            0.013414627872407436,
+            -0.029933987185359,
+            0.0007183607667684555,
+            0.023065946996212006,
+            0.01173681952059269,
+            -0.002052179304882884,
+            0.00922794733196497,
+            0.008851616643369198,
+            -0.016825126484036446,
+            6.958201265661046e-05,
+            0.012520842254161835,
+            0.018079563975334167,
+            0.007957830093801022,
+            0.006879799067974091,
+            0.006417225580662489,
+            0.019616248086094856,
+            -0.018785184249281883,
+            -0.02010234259068966,
+            0.00456693209707737,
+            0.010521584190428257,
+            -0.0065426696091890335,
+            -0.020996127277612686,
+            0.007060124538838863,
+            -0.016401754692196846,
+            0.037350840866565704,
+            0.009690520353615284,
+            -0.010803832672536373,
+            -0.0035790635738521814,
+            -0.031141381710767746,
+            -0.0027479995042085648,
+            -0.021983996033668518,
+            -0.010153094306588173,
+            0.019004710018634796,
+            -0.003353656968101859,
+            -0.00965131912380457,
+            -0.01906743273139,
+            -0.011015518568456173,
+            -0.008734012953937054,
+            0.0017326901433989406,
+            -0.008984900079667568,
+            0.01944376341998577,
+            0.01144673116505146,
+            -0.02844434417784214,
+            -0.031799960881471634,
+            -0.03361889347434044,
+            0.011054719798266888,
+            0.0025559139903634787,
+            -0.02913428284227848,
+            -0.011721138842403889,
+            -0.004555171821266413,
+            -0.004080838058143854,
+            -0.008000951260328293,
+            0.008145995438098907,
+            -0.003831910900771618,
+            0.006017374340444803,
+            -0.013830160722136497,
+            -0.005554800853133202,
+            -0.008130315691232681,
+            0.00788726843893528,
+            0.0149513129144907,
+            -0.005488158669322729,
+            -0.03093753568828106,
+            -0.017781633883714676,
+            0.00965131912380457,
+            -0.012826611287891865,
+            0.008145995438098907,
+            0.01684080809354782,
+            -0.004923662636429071,
+            0.00594289181753993,
+            0.01848725602030754,
+            0.018095243722200394,
+            -0.0030753291212022305,
+            0.014363295398652554,
+            -0.0011162522714585066,
+            -0.006385864689946175,
+            0.012607084587216377,
+            0.008953538723289967,
+            0.01487291045486927,
+            0.02507304586470127,
+            -0.010294217616319656,
+            -0.006087936460971832,
+            -0.0016954491147771478,
+            0.03274078667163849,
+            -0.023599082604050636,
+            -0.019804412499070168,
+            0.015688294544816017,
+            0.05425437167286873,
+            -0.0010662708664312959,
+            -0.0020913805346935987,
+            -0.022344646975398064,
+            -0.01102335937321186,
+            0.023489320650696754,
+            0.016793766990303993,
+            0.008036232553422451,
+            -0.010843033902347088,
+            -0.014386816881597042,
+            0.010545105673372746,
+            0.014418177306652069,
+            0.01922423578798771,
+            -0.024053815752267838,
+            0.021278375759720802,
+            -0.026750855147838593,
+            0.036127764731645584,
+            0.0075501385144889355,
+            -0.005974252708256245,
+            0.015539329499006271,
+            0.0036378654185682535,
+            -0.006526988931000233,
+            0.014324095100164413,
+            -0.01070190966129303,
+            0.011893624439835548,
+            0.006868038792163134,
+            -0.0015004234155640006,
+            0.021043168380856514,
+            -0.01108608115464449,
+            -0.0010045290691778064,
+            -0.0027048783376812935,
+            0.02090204507112503,
+            0.0031184502877295017,
+            -0.0038671919610351324,
+            0.01464554388076067,
+            0.0028773632366210222,
+            -0.004351325798779726,
+            -0.010074691846966743,
+            0.007350212894380093,
+            -0.03173723816871643,
+            -0.008138155564665794,
+            0.013924242928624153,
+            -0.005735125858336687,
+            0.002120781457051635,
+            -0.011015518568456173,
+            -0.002642156556248665,
+            -0.03609640523791313,
+            0.01787571795284748,
+            -0.000642898608930409,
+            0.017091695219278336,
+            0.0028656029608100653,
+            -0.00936123076826334,
+            -0.004809979349374771,
+            0.02297186478972435,
+            -0.009565076790750027,
+            -0.006930760573595762,
+            -0.018769502639770508,
+            -0.021497901529073715,
+            -0.025135766714811325,
+            0.021121570840477943,
+            -0.009580757468938828,
+            0.0018189327092841268,
+            -0.007706942968070507,
+            -0.0029126442968845367,
+            -0.00791470892727375,
+            0.012222914025187492,
+            0.018706781789660454,
+            -0.030576884746551514,
+            0.006617151666432619,
+            -0.006170258391648531,
+            -0.022109439596533775,
+            0.006456426810473204,
+            -0.009518035687506199,
+            0.01684080809354782,
+            -0.0009344570571556687,
+            -0.04123959317803383,
+            0.006303542293608189,
+            0.026484286412596703,
+            -0.017969800159335136,
+            -0.0015141437761485577,
+            0.010043330490589142,
+            0.008718332275748253,
+            -0.0026499966625124216,
+            -0.018894946202635765,
+            0.01811092346906662,
+            -0.017436664551496506,
+            0.018675420433282852,
+            0.026844937354326248,
+            -0.008757533505558968,
+            0.028993159532546997,
+            0.0001089056531782262,
+            0.01864405907690525,
+            0.0029832064174115658,
+            0.005641043186187744,
+            0.010082531720399857,
+            0.0038985528517514467,
+            0.020070981234312057,
+            0.00295772566460073,
+            0.007949990220367908,
+            0.038072142750024796,
+            -0.010788151994347572,
+            0.02182719111442566,
+            0.011305606923997402,
+            -0.00019184054690413177,
+            0.01615086756646633,
+            -0.007628540974110365,
+            -0.032489899545907974,
+            0.01566477306187153,
+            -0.027770083397626877,
+            -0.004668855108320713,
+            0.00922794733196497,
+            -0.0147396270185709,
+            -0.00934555009007454,
+            -0.014418177306652069,
+            0.009235787205398083,
+            0.012779570184648037,
+            -0.013908562250435352,
+            -0.0005326454411260784,
+            0.018973348662257195,
+            0.019381040707230568,
+            0.014527940191328526,
+            0.005982093047350645,
+            -0.005182390101253986,
+            0.013830160722136497,
+            0.0030204474460333586,
+            0.023646123707294464,
+            0.0016895688604563475,
+            0.005633202847093344,
+            -0.010592146776616573,
+            0.0145122604444623,
+            0.01508459635078907,
+            -0.0017875717021524906,
+            0.01092927623540163,
+            -0.004845260176807642,
+            -0.005817448254674673,
+            -0.01960056647658348,
+            -0.007636380847543478,
+            -0.009110343642532825,
+            0.006170258391648531,
+            0.04719816520810127,
+            0.0004782538453582674,
+            -0.002846002345904708,
+            -0.01765619032084942,
+            0.004559091757982969,
+            0.012685487046837807,
+            -0.0097454022616148,
+            -0.0034987013787031174,
+            -0.019365360960364342,
+            -0.022720977663993835,
+            0.0034144187811762094,
+            -0.0050843870267271996,
+            0.027001742273569107,
+            -0.003822110593318939,
+            0.02767600119113922,
+            -0.029102923348546028,
+            0.01956920698285103,
+            -0.016135187819600105,
+            -0.0097454022616148,
+            -0.0022560253273695707,
+            -0.012160192243754864,
+            -0.014331934973597527,
+            0.02300322614610195,
+            0.00936123076826334,
+            -0.032678067684173584,
+            -0.0016190068563446403,
+            0.01010605227202177,
+            -0.009306349791586399,
+            0.009008420631289482,
+            0.006860198453068733,
+            0.010803832672536373,
+            -0.0015798057429492474,
+            0.01190930511802435,
+            0.0013592992909252644,
+            -0.001451421994715929,
+            -9.536901052342728e-05,
+            -0.02294050343334675,
+            0.018346130847930908,
+            0.030482802540063858,
+            -0.05265496298670769,
+            -0.01914583332836628,
+            0.020118022337555885,
+            0.00569592509418726,
+            0.0005865469574928284,
+            0.013061817735433578,
+            -0.024367425590753555,
+            0.01027853786945343,
+            -0.018675420433282852,
+            -0.014316254295408726,
+            -0.022140800952911377,
+            -0.005460718180984259,
+            0.0151786794885993,
+            -0.027581918984651566,
+            0.025794345885515213,
+            0.017170097678899765,
+            -0.0015023835003376007,
+            0.006452506873756647,
+            0.0007472716388292611,
+            0.008851616643369198,
+            -0.03255262225866318,
+            -0.006617151666432619,
+            -0.0059664128348231316,
+            0.006009534001350403,
+            0.003655505832284689,
+            -0.02897747792303562,
+            0.014527940191328526,
+            -0.02074524015188217,
+            0.008263599127531052,
+            -0.029949666932225227,
+            0.011321287602186203,
+            0.03622184693813324,
+            -2.7471420253277756e-05,
+            0.009157384745776653,
+            -0.0031282505951821804,
+            0.04218042269349098,
+            0.026625411584973335,
+            0.0015709854196757078,
+            -0.008024472743272781,
+            -0.019694650545716286,
+            -0.002522593131288886,
+            0.00673867529258132,
+            -0.02918132394552231,
+            0.0012113150442019105,
+            -0.035092856734991074,
+            0.02209375984966755,
+            -0.006099696736782789,
+            -0.025512099266052246,
+            0.033556170761585236,
+            -0.020384589210152626,
+            0.002808761317282915,
+            -0.005347034893929958,
+            0.0027068383060395718,
+            0.030561204999685287,
+            0.0024990723468363285,
+            0.03005943074822426,
+            -0.03826030716300011,
+            -0.0013504790840670466,
+            0.0028244417626410723,
+            0.0038162304554134607,
+            -0.010435341857373714,
+            -0.015711814165115356,
+            0.016903528943657875,
+            -0.010200135409832,
+            0.004292524419724941,
+            0.02522985078394413,
+            -0.003655505832284689,
+            -0.013132380321621895,
+            0.021278375759720802,
+            -0.002808761317282915,
+            0.005417597014456987,
+            -0.005460718180984259,
+            0.000567926443181932,
+            0.023222751915454865,
+            0.02568458393216133,
+            -0.01136832870543003,
+            -0.011336968280375004,
+            -0.002781320596113801,
+            -0.00031654915073886514,
+            0.014104568399488926,
+            0.015264921821653843,
+            -0.008200877346098423,
+            -0.020541394129395485,
+            0.006817077286541462,
+            0.009808124043047428,
+            0.018549976870417595,
+            0.0012946174247190356,
+            -0.0255748201161623,
+            -0.008459605276584625,
+            -0.03027895651757717,
+            0.010153094306588173,
+            -0.016558559611439705,
+            -0.005601841956377029,
+            0.0013112779706716537,
+            -0.001341658877208829,
+            -0.009549396112561226,
+            0.014394656755030155,
+            0.015194360166788101,
+            -0.03901296854019165,
+            -0.015813738107681274,
+            -0.0007222809363156557,
+            -0.018502935767173767,
+            0.0024539909791201353,
+            -0.0011662337929010391,
+            0.001961036818102002,
+            0.0021658625919371843,
+            -0.003165491623803973,
+            0.004739417228847742,
+            -0.020070981234312057,
+            -0.0024324303958564997,
+            0.005629282910376787,
+            0.02113725244998932,
+            0.004535571206361055,
+            -0.0015014035161584616,
+            0.0022442650515586138,
+            0.0038907125126570463,
+            0.0017464105039834976,
+            0.0063897850923240185,
+            -0.00673867529258132,
+            0.015108116902410984,
+            -0.00020262086763978004,
+            0.015327643603086472,
+            -0.01906743273139,
+            0.007557978853583336,
+            -0.002369708614423871,
+            -0.005123588256537914,
+            0.015782376751303673,
+            -0.00037020572926849127,
+            -0.0020482593681663275,
+            -0.010294217616319656,
+            0.007605019956827164,
+            0.015586371533572674,
+            -0.022501451894640923,
+            -0.009063302539288998,
+            -0.014614183455705643,
+            0.02711150422692299,
+            0.0153746847063303,
+            0.008122474886476994,
+            -0.03223901242017746,
+            -0.0056488835252821445,
+            -0.012120991013944149,
+            -0.0059193712659180164,
+            -0.015296283178031445,
+            0.018863586708903313,
+            0.019349679350852966,
+            -0.0051784696988761425,
+            0.01619790866971016,
+            -0.0015592251438647509,
+            0.020855003967881203,
+            -0.00924362801015377,
+            0.006476027425378561,
+            -0.0007590319728478789,
+            -0.006315303035080433,
+            0.010168774053454399,
+            0.03521829843521118,
+            -0.017389623448252678,
+            0.008240078575909138,
+            -0.000719830859452486,
+            -0.002530433237552643,
+            -0.00306356861256063,
+            -0.004022036213427782,
+            0.006358424201607704,
+            0.014316254295408726,
+            0.009369071573019028,
+            0.0029342048801481724,
+            0.07056204229593277,
+            -0.003506541484966874,
+            0.020243465900421143,
+            0.006323142908513546,
+            -0.014574982225894928,
+            -0.005891930777579546,
+            0.0022109439596533775,
+            -0.028397301211953163,
+            0.031266823410987854,
+            0.020086660981178284,
+            0.0048413402400910854,
+            0.003782909596338868,
+            -0.014692584984004498,
+            0.023034587502479553,
+            0.004453248810023069,
+            0.0030616086442023516,
+            -0.0049119023606181145,
+            0.005864489823579788,
+            0.047762662172317505,
+            0.012403239496052265,
+            -0.007420775014907122,
+            -0.0022658256348222494,
+            0.014292733743786812,
+            0.0054254368878901005,
+            -0.011493772268295288,
+            -0.0003150791162624955,
+            -0.000643878651317209,
+            0.015037555247545242,
+            -0.001587645965628326,
+            0.021952634677290916,
+            -0.01114096213132143,
+            -0.01826772838830948,
+            0.0018375532235950232,
+            0.004394447430968285,
+            0.01571965590119362,
+            0.009486674331128597,
+            0.008318481035530567,
+            -0.005884090438485146,
+            0.004700215999037027,
+            0.006397624965757132,
+            0.005664564203470945,
+            0.00790294911712408,
+            -0.005852729547768831,
+            -0.008655610494315624,
+            -0.01826772838830948,
+            0.02637452259659767,
+            -0.002616675803437829,
+            0.003259574295952916,
+            -0.016856487840414047,
+            0.002438310533761978,
+            -0.0034712604247033596,
+            -0.007769664749503136,
+            0.010937117040157318,
+            0.017750274389982224,
+            -0.009392592124640942,
+            -0.008185196667909622,
+            0.0147239463403821,
+            -0.017028972506523132,
+            0.004457169212400913,
+            0.012622765265405178,
+            -0.004778618458658457,
+            0.006311382632702589,
+            0.003381097922101617,
+            -0.008898657746613026,
+            0.0012073949910700321,
+            0.0013671396300196648,
+            0.015539329499006271,
+            0.015249241143465042,
+            0.007075804751366377,
+            0.022031037136912346,
+            0.0016395874554291368,
+            0.0009217166807502508,
+            0.01654287800192833,
+            0.015609892085194588,
+            0.0064721074886620045,
+            0.022078078240156174,
+            -0.015445247292518616,
+            -0.016668323427438736,
+            0.003367377445101738,
+            0.006895479746162891,
+            0.0025441537145525217,
+            0.0030243676155805588,
+            0.0036751064471900463,
+            0.010545105673372746,
+            -0.014394656755030155,
+            -0.006946441251784563,
+            -0.009322029538452625,
+            0.018973348662257195,
+            0.022689616307616234,
+            -0.020588435232639313,
+            0.0003922563628293574,
+            0.004958943463861942,
+            0.017091695219278336,
+            -0.001889494713395834,
+            0.006558349821716547,
+            0.01760914921760559,
+            -0.006574030499905348,
+            -0.008389042690396309,
+            -0.02316003106534481,
+            -0.012426760047674179,
+            -0.0006556389853358269,
+            0.019898496568202972,
+            -0.011407529935240746,
+            0.01286581251770258,
+            -0.015633411705493927,
+            -0.0042768437415361404,
+            0.029918305575847626,
+            0.011877943761646748,
+            0.005739046260714531,
+            -0.01330486498773098,
+            -0.011674097739160061,
+            -0.013673355802893639,
+            0.014324095100164413,
+            0.010317739099264145,
+            0.003122370457276702,
+            -0.010121732950210571,
+            -0.0017013292526826262,
+            -0.005515599623322487,
+            -0.008961379528045654,
+            -0.00685235857963562,
+            0.011462411843240261,
+            -0.0027617199812084436,
+            0.003949514124542475,
+            -0.011987706646323204,
+            0.0009903187165036798,
+            -0.010748950764536858,
+            -0.006495628040283918,
+            -0.0015415846137329936,
+            -0.03481060639023781,
+            -0.0299026258289814,
+            -0.001531784306280315,
+            -0.005107908044010401,
+            0.005652803461998701,
+            0.004123959224671125,
+            0.011611375957727432,
+            0.0029185244347900152,
+            0.00937691144645214,
+            0.009251467883586884,
+            0.011062560603022575,
+            3.1728417525300756e-05,
+            -0.004202361684292555,
+            0.011995547451078892,
+            -0.013602794148027897,
+            -0.016244949772953987,
+            0.024916240945458412,
+            0.004751177504658699,
+            0.013461669906973839,
+            -0.028397301211953163,
+            0.011783861555159092,
+            -0.002575514605268836,
+            0.010654868558049202,
+            -0.031141381710767746,
+            -0.0039769550785422325,
+            0.0020443391986191273,
+            -0.0033242562785744667,
+            -0.0101844547316432,
+            -0.028052331879734993,
+            -0.0075383782386779785,
+            -0.017812995240092278,
+            -0.01630767248570919,
+            -0.024759436026215553,
+            0.0050138249062001705,
+            -0.012034748680889606,
+            0.021341098472476006,
+            -0.010286377742886543,
+            0.03725675866007805,
+            0.006040894892066717,
+            -0.0015719655202701688,
+            0.01361063402146101,
+            0.0030478881672024727,
+            -0.0020090581383556128,
+            0.016636962071061134,
+            -0.006942520849406719,
+            -0.0013749798526987433,
+            0.017969800159335136,
+            0.01102335937321186,
+            0.003243893850594759,
+            -0.003602584358304739,
+            0.0029949666932225227,
+            0.026562688872218132,
+            0.01814228482544422,
+            0.00426900340244174,
+            0.005840969271957874,
+            0.002998886862769723,
+            0.016417434439063072,
+            0.009808124043047428,
+            -0.014778828248381615,
+            -0.008145995438098907,
+            0.011611375957727432,
+            -0.01173681952059269,
+            -0.006832757964730263,
+            0.004845260176807642,
+            -0.0097454022616148,
+            -0.023097308352589607,
+            0.012748208828270435,
+            0.006256501190364361,
+            -0.0005919371615163982,
+            -0.019239917397499084,
+            -0.012920694425702095,
+            0.0017620910657569766,
+            0.01225427445024252,
+            0.016323352232575417,
+            -0.01914583332836628,
+            0.002738199196755886,
+            0.008153836242854595,
+            -0.006417225580662489,
+            -0.011580015532672405,
+            0.016950570046901703,
+            0.022313285619020462,
+            0.014888591133058071,
+            -0.04788810759782791,
+            -0.008945698849856853,
+            0.017546428367495537,
+            -0.012183712795376778,
+            -0.043967992067337036,
+            -0.0015455047832801938,
+            -0.0018395131919533014,
+            -0.025982512161135674,
+            0.0052490318194031715,
+            0.006182019133120775,
+            -0.014700425788760185,
+            0.016527198255062103,
+            0.03200380504131317,
+            0.00019159555085934699,
+            0.009102503769099712,
+            -0.011109601706266403,
+            -0.014426017180085182,
+            -0.009047621861100197,
+            0.008726172149181366,
+            -0.01175250019878149,
+            -0.009847325272858143,
+            0.00248927203938365,
+            -0.015555010177195072,
+            0.01848725602030754,
+            0.007569739129394293,
+            0.03150203078985214,
+            0.01132912840694189,
+            -0.00248927203938365,
+            -0.01760914921760559,
+            -0.020964767783880234,
+            -0.009533716365695,
+            0.003741748398169875,
+            0.007997031323611736,
+            0.007005242630839348,
+            0.010160934180021286,
+            0.013328385539352894,
+            0.022673936560750008,
+            -0.006252580787986517,
+            0.014825869351625443,
+            -3.0059307391638868e-05,
+            -0.0014680825406685472,
+            0.0025676742661744356,
+            -0.020149383693933487,
+            0.005143188871443272,
+            0.0018228527624160051,
+            -0.004904062021523714,
+            -0.007013082969933748,
+            -0.003083169227465987,
+            0.032066527754068375,
+            -0.016480157151818275,
+            0.015970543026924133,
+            -0.031078658998012543,
+            -0.009337710216641426,
+            0.001246596104465425,
+            0.02814641408622265,
+            0.012701167725026608,
+            -0.02841298282146454,
+            -0.011297767050564289,
+            -0.026515647768974304,
+            0.02278370037674904,
+            -0.014300573617219925,
+            0.004723736550658941,
+            0.015845099464058876,
+            0.01444953866302967,
+            0.01194850541651249,
+            -0.01165841706097126,
+            -0.00878889486193657,
+            0.011776020750403404,
+            0.021999677643179893,
+            0.007679502479732037,
+            0.0022266244050115347,
+            -0.0009261267841793597,
+            -0.001644487609155476,
+            -0.004304284695535898,
+            -0.017985479906201363,
+            0.006711234338581562,
+            -0.00819303747266531,
+            0.027127185836434364,
+            -0.007597180083394051,
+            0.016746724024415016,
+            -0.0062957024201750755,
+            -0.00582920853048563,
+            -0.02932244911789894,
+            -0.0203689094632864,
+            0.007605019956827164,
+            -0.006703393999487162,
+            0.012959894724190235,
+            -0.019365360960364342,
+            -0.0164958368986845,
+            0.010733271017670631,
+            -0.02010234259068966,
+            -0.002083540428429842,
+            -0.004719816613942385,
+            0.0022442650515586138,
+            0.00676611578091979,
+            0.00736197317019105,
+            -0.012826611287891865,
+            -0.006883719470351934,
+            0.016072465106844902,
+            -0.007663821801543236,
+            0.007640301249921322,
+            0.029871264472603798,
+            0.01768755167722702,
+            0.026844937354326248,
+            0.007048364263027906,
+            0.013751758262515068,
+            -0.010364780202507973,
+            -0.017546428367495537,
+            -0.023301154375076294,
+            0.01278741005808115,
+            -0.0025519938208162785,
+            0.03368161618709564,
+            0.00478645833209157,
+            -0.007820626720786095,
+            -0.0031027698423713446,
+            0.005417597014456987,
+            0.0008408643188886344,
+            -0.01414376962929964,
+            0.0029518455266952515,
+            -0.00551167968660593,
+            0.012826611287891865,
+            0.016033263877034187,
+            -0.01883222535252571,
+            0.02522985078394413,
+            -0.016856487840414047,
+            0.012779570184648037,
+            -0.003602584358304739,
+            0.01630767248570919,
+            0.009753242135047913,
+            -0.0003626105026341975,
+            -0.0015415846137329936,
+            -0.05394076183438301,
+            -0.010458862408995628,
+            0.0038456313777714968,
+            -0.004970703739672899,
+            0.000352075177943334,
+            -0.012638445943593979,
+            -0.038573917001485825,
+            0.0034810607321560383,
+            0.013391107320785522,
+            -0.011556494981050491,
+            0.013422468677163124,
+            -0.002016898477450013,
+            0.006617151666432619,
+            -0.007922549732029438,
+            -0.006260421127080917,
+            0.016480157151818275,
+            0.02573162503540516,
+            -0.021121570840477943,
+            -0.022344646975398064,
+            0.005778247490525246,
+            -0.005543040577322245,
+            -0.014684745110571384,
+            -0.005021665245294571,
+            -0.01598622277379036,
+            0.0025069124531000853,
+            0.023567723110318184,
+            -0.01426137238740921,
+            -0.01776595413684845,
+            0.010145253501832485,
+            0.007757904473692179,
+            -0.024853520095348358,
+            -0.002355988137423992,
+            -0.009815963916480541,
+            -0.010803832672536373,
+            0.018393171951174736,
+            -0.01362631469964981,
+            0.008906497620046139,
+            -0.013124539516866207,
+            0.010396140627563,
+            -0.02825617790222168,
+            -0.01226995512843132,
+            -0.0019502564100548625,
+            -0.008882977068424225,
+            0.028083693236112595,
+            -0.010772472247481346,
+            -0.010200135409832,
+            0.020070981234312057,
+            -0.008663450367748737,
+            0.012567883357405663,
+            0.007855907082557678,
+            0.028648190200328827,
+            -0.0025853149127215147,
+            0.009141704998910427,
+            0.0028930436819791794,
+            0.01390072237700224,
+            0.0011936745140701532,
+            0.035437826067209244,
+            -0.00957291666418314,
+            -0.00030307378619909286,
+            -0.008835935965180397,
+            -0.0017140696290880442,
+            0.0010427501983940601,
+            0.010991998016834259,
+            0.014151609502732754,
+            0.012693327851593494,
+            -0.006244740914553404,
+            -0.0077853454276919365,
+            -0.015500128269195557,
+            -0.004527730867266655,
+            -0.004888381343334913,
+            0.007428614888340235,
+            0.012442439794540405,
+            -0.020039619877934456,
+            0.005358795169740915,
+            -0.012144511565566063,
+            0.0011946546146646142,
+            -0.01587645895779133,
+            -0.0022560253273695707,
+            -0.013634154573082924,
+            -0.004080838058143854,
+            -0.004331725183874369,
+            0.003790749702602625,
+            0.008749693632125854,
+            -0.016401754692196846,
+            0.0199455376714468,
+            0.008153836242854595,
+            -0.0004074467869941145,
+            -0.0028107212856411934,
+            -0.024931922554969788,
+            -0.014010485261678696,
+            0.00014859679504297674,
+            0.016668323427438736,
+            0.0050138249062001705,
+            -0.0051784696988761425,
+            0.002026698552072048,
+            -0.012826611287891865,
+            0.005280392710119486,
+            -0.0009040761506184936,
+            0.0022285846062004566,
+            -0.0030067269690334797,
+            0.0057312059216201305,
+            0.006762195844203234,
+            0.003700587200000882,
+            0.012959894724190235,
+            0.01487291045486927,
+            -0.011760340072214603,
+            -0.00966699980199337,
+            0.012285635806620121,
+            0.015217880718410015,
+            -0.008914338424801826,
+            0.01941240206360817,
+            -0.02878931351006031,
+            0.002820521593093872,
+            0.005233351606875658,
+            -0.0042533231899142265,
+            -0.0021854632068425417,
+            7.748349162284285e-05,
+            0.028475703671574593,
+            -0.0010221695993095636,
+            -0.010388300754129887,
+            -0.0009854185627773404,
+            0.01707601360976696,
+            -0.0006600490887649357,
+            0.006123217288404703,
+            0.023724526166915894,
+            0.00023790188424754888,
+            0.009768922813236713,
+            0.009776762686669827,
+            -0.019506484270095825,
+            -0.003145891008898616,
+            0.004633574280887842,
+            -0.012442439794540405,
+            0.018393171951174736,
+            0.014841550029814243,
+            0.008389042690396309,
+            0.0038005500100553036,
+            0.007510937284678221,
+            0.020086660981178284,
+            0.006374104414135218,
+            -0.002336387522518635,
+            -0.0002511322672944516,
+            0.0066445921547710896,
+            -0.011721138842403889,
+            0.017405303195118904,
+            -0.004363086074590683,
+            -0.015437406487762928,
+            0.006679873447865248,
+            0.011674097739160061,
+            -0.006773956120014191,
+            -0.004033796954900026,
+            0.0022540653590112925,
+            0.005711605306714773,
+            0.005464638117700815,
+            -0.003888752544298768,
+            -0.006574030499905348,
+            0.0012505161575973034,
+            0.017812995240092278,
+            0.010639187879860401,
+            0.016433116048574448,
+            -0.014990514144301414,
+            -0.011697618290781975,
+            0.0023050266318023205,
+            -0.018016841262578964,
+            0.002996926661580801,
+            -0.011517293751239777,
+            0.006299622356891632,
+            -0.007420775014907122,
+            0.02568458393216133,
+            0.019616248086094856,
+            0.007573659066110849,
+            0.011932825669646263,
+            -0.015539329499006271,
+            0.016715364530682564,
+            -0.00913386419415474,
+            0.012120991013944149,
+            -0.018847905099391937,
+            -0.0017346502281725407,
+            0.005296073388308287,
+            0.027628960087895393,
+            0.010302058421075344,
+            0.006064415443688631,
+            -0.001778751495294273,
+            -0.005182390101253986,
+            -0.008318481035530567,
+            -0.009619958698749542,
+            -0.014755306765437126,
+            -0.026233399286866188,
+            -0.00022810160589870065,
+            0.016339033842086792,
+            -0.00195907661691308,
+            0.0022521051578223705,
+            0.014590661972761154,
+            0.013187261298298836,
+            -0.0024128297809511423,
+            -0.013908562250435352,
+            -0.007252209819853306,
+            0.010835194028913975,
+            0.005119668319821358,
+            -0.008357682265341282,
+            -0.009141704998910427,
+            0.008138155564665794,
+            -0.013273504562675953,
+            0.016433116048574448,
+            -0.016182228922843933,
+            -0.009949248284101486,
+            -0.013649835251271725,
+            -0.0005929171456955373,
+            -0.008263599127531052,
+            -0.031454991549253464,
+            0.012152351438999176,
+            0.005707685369998217,
+            -0.0006365284207277,
+            -0.025715945288538933,
+            0.02369316667318344,
+            -0.008569368161261082,
+            0.03656681999564171,
+            -0.01593918167054653,
+            0.015633411705493927,
+            0.0010623506968840957,
+            0.011493772268295288,
+            -0.003392858197912574,
+            0.0029479253571480513,
+            0.007953910157084465,
+            0.002550033852458,
+            0.003055728506296873,
+            -0.00539799639955163,
+            -0.0033066156320273876,
+            0.008898657746613026,
+            -0.006448586471378803,
+            0.008490965701639652,
+            -0.014849389903247356,
+            -0.02320707216858864,
+            -0.01144673116505146,
+            0.009502355009317398,
+            0.014104568399488926,
+            0.00861640926450491,
+            -0.014473059214651585,
+            0.00882809516042471,
+            -0.02435174584388733,
+            -0.0023461878299713135,
+            -0.0031282505951821804,
+            0.004782538395375013,
+            -0.017201457172632217,
+            0.016605600714683533,
+            0.02193695493042469,
+            -3.518914309097454e-05,
+            -0.0007879427867010236,
+            0.01382231991738081,
+            0.010160934180021286,
+            0.009423952549695969,
+            -0.010011970065534115,
+            0.0017826715484261513,
+            -8.403743413509801e-05,
+            0.02044731192290783,
+            -0.002397149335592985,
+            -0.009831644594669342,
+            -0.0035379023756831884,
+            -0.01399480551481247,
+            0.009698361158370972,
+            -0.029275408014655113,
+            -0.0052490318194031715,
+            -0.006523068994283676,
+            0.017170097678899765,
+            0.00684451824054122,
+            0.021639026701450348,
+            0.0065622697584331036,
+            0.005523439962416887,
+            -8.844755939207971e-05,
+            -0.008240078575909138,
+            0.012097470462322235,
+            -0.00809895433485508,
+            -0.007201248314231634,
+            0.006013453938066959,
+            0.004492450039833784,
+            0.016856487840414047,
+            0.008067593909800053,
+            -0.009180906228721142,
+            0.030106471851468086,
+            0.014904271811246872,
+            -0.013939923606812954,
+            0.01814228482544422,
+            -0.01826772838830948,
+            0.020776601508259773,
+            0.01853429712355137,
+            0.010239336639642715,
+            -0.015304123051464558,
+            -0.015680454671382904,
+            0.01784435659646988,
+            -0.010466703213751316,
+            -0.004657094832509756,
+            0.0018052122322842479,
+            -0.017938438802957535,
+            8.092890311672818e-06,
+            -0.011972026899456978,
+            0.015429566614329815,
+            -0.0021991836838424206,
+            -0.00012042098387610167,
+            -0.020776601508259773,
+            -0.00043831768562085927,
+            0.016903528943657875,
+            0.01615086756646633,
+            -0.0033340565860271454,
+            -0.04478337615728378,
+            -0.006178098730742931,
+            0.009855165146291256,
+            -0.00706404447555542,
+            -0.025982512161135674,
+            0.01803252287209034,
+            0.0034104986116290092,
+            -0.019428081810474396,
+            -0.01184658333659172,
+            -0.018502935767173767,
+            -0.019396722316741943,
+            -0.008820255286991596,
+            -0.004563012160360813,
+            -0.015272761695086956,
+            -0.003884832374751568,
+            0.008224397897720337,
+            -0.007213008590042591,
+            -0.006064415443688631,
+            0.004061237443238497,
+            -0.007354132831096649,
+            -0.0003518301818985492,
+            -0.010866554453969002,
+            -0.005629282910376787,
+            0.011133122257888317,
+            0.002452031010761857,
+            -0.01508459635078907,
+            -0.015366844832897186,
+            -0.011564334854483604,
+            -0.010537264868617058,
+            0.004139639902859926,
+            -0.009212266653776169,
+            -0.005080467090010643,
+            -0.020149383693933487,
+            0.0006615191232413054,
+            0.013469509780406952,
+            0.0104118213057518,
+            0.0006154578295536339,
+            0.004931502975523472,
+            0.022313285619020462,
+            -0.006189859006553888,
+            -0.0006242780946195126,
+            -9.363864592160098e-06,
+            -0.002955765463411808,
+            0.02687629871070385,
+            0.004312125034630299,
+            0.01286581251770258,
+            -0.005562641192227602,
+            0.009494515135884285,
+            -0.01975737139582634,
+            -0.0023755887523293495,
+            -0.010850873775780201,
+            -0.019773053005337715,
+            -0.011979866772890091,
+            -5.2125258662272245e-05,
+            0.004743337165564299,
+            -0.013069658540189266,
+            0.0021541023161262274,
+            -0.00467277504503727,
+            0.006734754890203476,
+            -0.01508459635078907,
+            -0.005782167427241802,
+            -0.007134606596082449,
+            -0.010662708431482315,
+            -0.02416357956826687,
+            0.018283409997820854,
+            0.02889907732605934,
+            0.001601366326212883,
+            -0.007554058451205492,
+            -0.01391640305519104,
+            -0.008177356794476509,
+            -0.004041636828333139,
+            -0.018048202618956566,
+            -0.008702651597559452,
+            -0.027926888316869736,
+            -0.013641994446516037,
+            -0.0003746158326976001,
+            0.010772472247481346,
+            0.0036065042950212955,
+            -0.018126605078577995,
+            0.005107908044010401,
+            -0.025527779012918472,
+            -0.025668902322649956,
+            -0.019428081810474396,
+            -0.017295541241765022,
+            0.0025676742661744356,
+            0.003888752544298768,
+            0.0021149013191461563,
+            0.017358262091875076,
+            0.020227786153554916,
+            0.004813899286091328,
+            -0.002424590289592743,
+            0.008592888712882996,
+            -0.014974833466112614,
+            0.0046766954474151134,
+            0.0023305073846131563,
+            -0.013422468677163124,
+            0.02358340285718441,
+            -0.011917144991457462,
+            -0.00353594240732491,
+            0.0008570348145440221,
+            0.0034575401805341244,
+            -0.028193457052111626,
+            -0.002152142347767949,
+            -0.0164958368986845,
+            0.02270529791712761,
+            -0.029651738703250885,
+            -0.0021443020086735487,
+            -0.00416316045448184,
+            -0.007895108312368393,
+            0.026578368619084358,
+            0.021356778219342232,
+            -0.0004645334556698799,
+            0.008984900079667568,
+            0.010709750466048717,
+            -0.0013063778169453144,
+            0.01869110018014908,
+            -0.017483705654740334,
+            -0.014151609502732754,
+            0.022234883159399033,
+            0.008757533505558968,
+            0.0036163046024739742,
+            -0.011242885142564774,
+            0.017562108114361763,
+            0.017468025907874107,
+            0.0059193712659180164,
+            -0.011485932394862175,
+            -0.016244949772953987,
+            0.0007933329907245934,
+            0.019773053005337715,
+            -0.012403239496052265,
+            0.0027440793346613646,
+            -0.010631348006427288,
+            0.008091114461421967,
+            -0.014708265662193298,
+            0.017750274389982224,
+            -0.027503516525030136,
+            -0.005060866475105286,
+            -0.0016092065488919616,
+            0.01795412041246891,
+            -0.008012712001800537,
+            0.015711814165115356,
+            -0.00696604186668992,
+            -0.009470994584262371,
+            0.014888591133058071,
+            -0.0005556760588660836,
+            -0.004739417228847742,
+            0.022909143939614296,
+            0.00653874920681119,
+            -0.0009614078444428742,
+            0.004884461406618357,
+            -0.018424533307552338,
+            0.025010323151946068,
+            -0.010074691846966743,
+            0.003000846831128001,
+            0.0012524762423709035,
+            0.0013710596831515431,
+            0.026844937354326248,
+            0.02634316310286522,
+            0.0057429661974310875,
+            0.008585048839449883,
+            -0.01351655088365078,
+            0.02519848942756653,
+            0.00467277504503727,
+            0.0012720768572762609,
+            -0.0009893386159092188,
+            -0.0024167499504983425,
+            -0.0017454305198043585,
+            -0.009180906228721142,
+            0.015743175521492958,
+            -0.01213667169213295,
+            -0.025935471057891846,
+            -0.009322029538452625,
+            -0.004492450039833784,
+            -0.017107374966144562,
+            0.020055299624800682,
+            0.0004699236014857888,
+            -0.003992635756731033,
+            0.011681938543915749,
+            0.007428614888340235,
+            0.0032360537443310022,
+            -0.008655610494315624,
+            0.0038671919610351324,
+            -0.007130686193704605,
+            -0.013046137988567352,
+            0.012450280599296093,
+            -0.019208556041121483,
+            -0.008984900079667568,
+            -0.008553687483072281,
+            0.009063302539288998,
+            0.010443182662129402,
+            0.00928282830864191,
+            0.015656933188438416,
+            0.004441488534212112,
+            0.00819303747266531,
+            0.012489481829106808,
+            -0.024931922554969788,
+            0.01707601360976696,
+            0.021764470264315605,
+            -0.0005096147651784122,
+            0.001671928446739912,
+            0.00045350813888944685,
+            -0.03446563705801964,
+            0.00027293790481053293,
+            0.0014034006744623184,
+            0.0077853454276919365,
+            -0.013406787998974323,
+            0.0022677856031805277,
+            0.022673936560750008,
+            -0.011109601706266403,
+            0.020996127277612686,
+            0.005970332771539688,
+            0.002479471731930971,
+            0.03343072906136513,
+            0.012230753898620605,
+            0.016339033842086792,
+            -0.0068915593437850475,
+            -0.026625411584973335,
+            0.0054371971637010574,
+            -0.018518615514039993,
+            0.009714040905237198,
+            0.0014386816183105111,
+            0.013148060999810696,
+            -0.01792275905609131,
+            -0.006025214213877916,
+            0.0027715202886611223,
+            -0.011039039120078087,
+            0.01654287800192833,
+            0.0014043806586414576,
+            0.019631927832961082,
+            0.001000609016045928,
+            -0.0028264017309993505,
+            -0.012967735528945923,
+            -0.01404968649148941,
+            0.0029773262795060873,
+            -0.009659159928560257,
+            -0.01699761115014553,
+            0.004966783802956343,
+            -0.008342001587152481,
+            0.007091485429555178,
+            -0.01587645895779133,
+            0.02044731192290783,
+            0.0005806668195873499,
+            0.00853800680488348,
+            0.015523649752140045,
+            0.0010319699067622423,
+            0.017750274389982224,
+            0.010341259650886059,
+            -0.0075501385144889355,
+            0.0062957024201750755,
+            0.0002594625111669302,
+            0.0029636058025062084,
+            0.0017895317869260907,
+            -0.01219155266880989,
+            0.0058135283179581165,
+            -0.004159240517765284,
+            0.00015165939112193882,
+            -0.006479947362095118,
+            -0.004092598333954811,
+            0.0034046184737235308,
+            -0.013406787998974323,
+            -0.00539015606045723,
+            0.002242304850369692,
+            0.001080971327610314,
+            0.02148222178220749,
+            -0.0017905118875205517,
+            0.00878889486193657,
+            0.02228192426264286,
+            -0.015570690855383873,
+            0.006703393999487162,
+            0.010059011168777943,
+            -0.004892301745712757,
+            -0.011156642809510231,
+            0.004661014769226313,
+            0.016166547313332558,
+            -0.01587645895779133,
+            -0.01131344772875309,
+            0.011501613073050976,
+            -0.021921275183558464,
+            0.004390527028590441,
+            -0.003963234834372997,
+            -0.007338452618569136,
+            0.02143518067896366,
+            -0.0012436560355126858,
+            0.032333094626665115,
+            -0.004770778119564056,
+            -0.015460927970707417,
+            -0.01930263824760914,
+            0.0012554163113236427,
+            -0.025512099266052246,
+            -0.005637123249471188,
+            -0.009188746102154255,
+            -0.006848438177257776,
+            0.00958859734237194,
+            0.0006865098839625716,
+            0.004919742234051228,
+            0.005950732156634331,
+            -0.004390527028590441,
+            0.004012236371636391,
+            -0.0031400108709931374,
+            0.007883348502218723,
+            -0.02143518067896366,
+            0.018628379330039024,
+            0.014120249077677727,
+            0.0005228451336733997,
+            -0.019741691648960114,
+            -0.008945698849856853,
+            0.020086660981178284,
+            0.04537923261523247,
+            -0.023222751915454865,
+            -0.0033085758332163095,
+            0.018330451101064682,
+            0.0038691519293934107,
+            0.0014475019415840507,
+            -0.00924362801015377,
+            -0.005570481065660715,
+            -0.04202361777424812,
+            0.0016415475402027369,
+            0.001603326410986483,
+            -0.009306349791586399,
+            0.007048364263027906,
+            -0.017812995240092278,
+            -0.007275730837136507,
+            -0.012638445943593979,
+            -0.020274827256798744,
+            -0.004610053263604641,
+            -0.010341259650886059,
+            -0.006503468379378319,
+            -0.011415370739996433,
+            0.013046137988567352,
+            -0.005774327088147402,
+            0.010082531720399857,
+            -0.004104358609765768,
+            0.01768755167722702,
+            -0.0013240183470770717,
+            0.012544362805783749,
+            -0.0005821368540637195,
+            0.009768922813236713,
+            0.00665243249386549,
+            -0.006789636332541704,
+            -0.008255759254097939,
+            -0.007667741738259792,
+            0.018659740686416626,
+            0.009768922813236713,
+            0.010498063638806343,
+            0.0014337814645841718,
+            -0.012552203610539436,
+            -0.011674097739160061,
+            5.987360782455653e-05,
+            -0.010333418846130371,
+            0.0006835698150098324,
+            0.006742595229297876,
+            0.026970380917191505,
+            -0.0033340565860271454,
+            -0.025527779012918472,
+            0.004649254493415356,
+            0.009847325272858143,
+            0.020917724817991257,
+            -0.013148060999810696,
+            0.007354132831096649,
+            -0.010498063638806343,
+            0.013108859769999981,
+            0.026311801746487617,
+            -0.009212266653776169,
+            0.018283409997820854,
+            -0.006205539684742689,
+            0.013947763480246067,
+            0.006836677901446819,
+            -0.00416316045448184,
+            -0.02963605709373951,
+            -0.008302800357341766,
+            0.0016131267184391618,
+            0.0036319850478321314,
+            -0.009369071573019028,
+            0.030874812975525856,
+            0.010482383891940117,
+            0.016088144853711128,
+            -0.009580757468938828,
+            0.010403981432318687,
+            0.00539015606045723,
+            0.019694650545716286,
+            0.012238594703376293,
+            0.00021413620561361313,
+            0.0019806374330073595,
+            -0.008498805575072765,
+            -0.045504678040742874,
+            0.00957291666418314,
+            0.006668113172054291,
+            0.02538665570318699,
+            -0.04196089506149292,
+            0.01569613441824913,
+            -0.010200135409832,
+            -0.0059899333864450455,
+            -0.003700587200000882,
+            0.007667741738259792,
+            -0.003696667030453682,
+            0.0290245208889246,
+            -0.005790007766336203,
+            0.0019355560652911663,
+            0.016684003174304962,
+            0.014073207043111324,
+            -0.007683422416448593,
+            0.01567261293530464,
+            0.023018905892968178,
+            0.01898903027176857,
+            0.013979124836623669,
+            -0.00748349679633975,
+            0.025621861219406128,
+            0.0006811197381466627,
+            -0.003218413097783923,
+            -0.008240078575909138,
+            -0.016166547313332558,
+            0.0024128297809511423,
+            -0.013116699643433094,
+            0.003659426001831889,
+            -0.00561752263456583,
+            -0.004633574280887842,
+            0.010905755683779716,
+            -0.010991998016834259,
+            0.0017454305198043585,
+            0.0018081523012369871,
+            0.007040523923933506,
+            0.0003800060076173395,
+            0.02703310176730156,
+            -0.032521262764930725,
+            0.014081047847867012,
+            -0.03258398175239563,
+            -0.01657423935830593,
+            0.006942520849406719,
+            -0.0044179679825901985,
+            0.00687195872887969,
+            -0.0195221658796072,
+            -0.017358262091875076,
+            0.00591545132920146,
+            0.01829908974468708,
+            0.01502971537411213,
+            0.00716988742351532,
+            0.006091856397688389,
+            0.004884461406618357,
+            -0.010059011168777943,
+            -0.006299622356891632,
+            -0.009557236917316914,
+            -0.00884377583861351,
+            -0.005049106199294329,
+            -0.02010234259068966,
+            0.015453087165951729,
+            -0.0070562041364610195,
+            -0.007922549732029438,
+            0.016746724024415016,
+            -0.01026285719126463,
+            -0.018424533307552338,
+            0.005539120174944401,
+            -0.0038926727138459682,
+            -0.005025585647672415,
+            -0.016213588416576385,
+            0.00922794733196497,
+            -0.0032752547413110733,
+            0.0039240336045622826,
+            -0.01309317909181118,
+            0.026672452688217163,
+            -0.006785716395825148,
+            0.001506303553469479,
+            0.011924984864890575,
+            -0.009408272802829742,
+            -0.024853520095348358,
+            -0.0021991836838424206,
+            -0.014896431006491184,
+            0.0027323190588504076,
+            -0.0028773632366210222,
+            -0.01673104427754879,
+            -0.008443924598395824,
+            0.0035967042203992605,
+            -0.02698606066405773,
+            -0.011384009383618832,
+            0.0099570881575346,
+            0.011783861555159092,
+            0.009431793354451656,
+            0.011932825669646263,
+            -0.0021011808421462774,
+            0.013148060999810696,
+            -0.029008839279413223,
+            -0.014496579766273499,
+            -0.01362631469964981,
+            -0.009549396112561226,
+            0.01412808895111084,
+            0.0017954119248315692,
+            0.007416854612529278,
+            -0.004480689764022827,
+            -0.005252952221781015,
+            -0.0011201724410057068,
+            0.011776020750403404,
+            0.028804993256926537,
+            0.0003814760420937091,
+            -0.003884832374751568,
+            -0.001314218039624393,
+            -0.011423210613429546,
+            0.028381621465086937,
+            -0.0041121989488601685,
+            -0.0072090886533260345,
+            0.03449700027704239,
+            0.01933399960398674,
+            0.027362391352653503,
+            -0.0001465142413508147,
+            -0.006025214213877916,
+            -0.0005091247148811817,
+            0.004931502975523472,
+            0.004974624142050743,
+            -0.01226995512843132,
+            -0.00842040404677391,
+            -0.004751177504658699,
+            0.03465380147099495,
+            0.0022952265571802855,
+            0.011133122257888317,
+            -0.0307493694126606,
+            0.015225720591843128,
+            -0.003577103605493903,
+            -0.007546218577772379,
+            0.003269374603405595,
+            0.036692261695861816,
+            0.02576298639178276,
+            0.0069738817401230335,
+            0.009447473101317883,
+            8.630374941276386e-05,
+            0.0398910753428936,
+            -0.02438310533761978,
+            -0.020196424797177315,
+            -0.02273665741086006,
+            -0.021717429161071777,
+            0.0016601680545136333,
+            0.007271810434758663,
+            0.003628065111115575,
+            -0.0009770882315933704,
+            0.0015259041683748364,
+            0.028585467487573624,
+            0.0015788257587701082,
+            -0.010341259650886059,
+            -0.0014308413956314325,
+            -0.034089308232069016,
+            0.013618473894894123,
+            -0.017389623448252678,
+            -0.010200135409832,
+            -0.016511518508195877,
+            -0.013006936758756638,
+            -0.025982512161135674,
+            -0.010607827454805374,
+            -0.0009599378099665046,
+            0.015970543026924133,
+            0.010004129260778427,
+            -0.0019120353972539306,
+            0.005613602697849274,
+            -0.023834289982914925,
+            -0.006703393999487162,
+            0.020917724817991257,
+            0.01567261293530464,
+            0.015790216624736786,
+            0.0046531748957931995,
+            0.01622927002608776,
+            -0.0003020937438122928,
+            -0.015923500061035156,
+            -0.0020913805346935987,
+            0.002767600119113922,
+            0.006413305643945932,
+            0.017091695219278336,
+            0.012222914025187492,
+            -0.018769502639770508,
+            0.0071267662569880486,
+            -0.012795249931514263,
+            0.03139226883649826,
+            -0.006197699345648289,
+            0.01205826923251152,
+            0.009682680480182171,
+            0.0032830950804054737,
+            -0.012489481829106808,
+            -0.006158498115837574,
+            -0.00479429867118597,
+            0.01569613441824913,
+            -0.00831064023077488,
+            -0.011062560603022575,
+            -0.005676324479281902,
+            0.009729721583425999,
+            -0.005535200238227844,
+            -0.0010349099757149816,
+            -0.015609892085194588,
+            -0.014347615651786327,
+            0.0030753291212022305,
+            -0.012826611287891865,
+            -0.0010711710201576352,
+            0.019036071375012398,
+            -0.005033425521105528,
+            -0.0004390527028590441,
+            0.014684745110571384,
+            0.016589920967817307,
+            0.004649254493415356,
+            0.018926307559013367,
+            -0.009549396112561226,
+            0.0022932663559913635,
+            -0.021623345091938972,
+            -0.0024716316256672144,
+            -0.032364457845687866,
+            0.019647609442472458,
+            0.0017836516490206122,
+            -0.020070981234312057,
+            0.010944956913590431,
+            -0.0016239070100709796,
+            -0.009165225550532341,
+            -0.0099727688357234,
+            0.002452031010761857,
+            -0.024179259315133095,
+            0.024147899821400642,
+            -0.016048945486545563,
+            0.008906497620046139,
+            0.01161921676248312,
+            0.041333675384521484,
+            0.0038573916535824537,
+            -0.00747565645724535,
+            0.021152932196855545,
+            -0.0002717128663789481,
+            0.02982422336935997,
+            -0.001121152425184846,
+            -0.012081789784133434,
+            -0.022689616307616234,
+            -0.005237271543592215,
+            -0.015202200040221214,
+            -0.014112408272922039,
+            0.0147239463403821,
+            -0.0195221658796072,
+            -0.015217880718410015,
+            -0.010004129260778427,
+            -0.019741691648960114,
+            -0.011509452946484089,
+            0.025982512161135674,
+            0.015758855268359184,
+            -0.011705459095537663,
+            -0.006425065919756889,
+            -0.0055077592842280865,
+            -0.02711150422692299,
+            0.021952634677290916,
+            0.008012712001800537,
+            -0.01983577385544777,
+            -0.0032772149424999952,
+            -0.009768922813236713,
+            0.004378766752779484,
+            0.01588430069386959,
+            -0.0001214010117109865,
+            0.05789223685860634,
+            -0.008804574608802795,
+            -0.0015945062041282654,
+            -0.020008258521556854,
+            0.009298508986830711,
+            -0.01575101539492607,
+            -0.006374104414135218,
+            -0.026672452688217163,
+            -0.008255759254097939,
+            -0.005119668319821358,
+            -0.021105891093611717,
+            -0.0022481849882751703,
+            -0.008287119679152966,
+            0.0057547264732420444,
+            -0.01280309073626995,
+            0.0008050933247432113,
+            -0.01665264181792736,
+            -0.00624866085126996,
+            -0.012089629657566547,
+            -0.012371878139674664,
+            0.01925559714436531,
+            -0.028303219005465508,
+            -0.0033869780600070953,
+            0.008153836242854595,
+            0.01353223156183958,
+            0.006268261466175318,
+            -0.021043168380856514,
+            -0.012614925391972065,
+            -0.011917144991457462,
+            -0.018894946202635765,
+            0.010733271017670631,
+            -0.010741110891103745,
+            0.013140220195055008,
+            -0.03186268359422684,
+            -0.012607084587216377,
+            -0.0019522164948284626,
+            0.019976899027824402,
+            0.01753074675798416,
+            -0.0004733536916319281,
+            -0.0059193712659180164,
+            -0.014002645388245583,
+            0.02278370037674904,
+            -0.013877201825380325,
+            -0.032678067684173584,
+            -0.0013494990998879075,
+            0.002218784298747778,
+            -0.00104079011362046,
+            -0.007816705852746964,
+            0.0027538796421140432,
+            0.004214121960103512,
+            0.004092598333954811,
+            -0.007546218577772379,
+            -0.0016464476939290762,
+            -0.01508459635078907,
+            0.0047668581828475,
+            -3.659793583210558e-05,
+            0.029730141162872314,
+            -0.0035339822061359882,
+            0.008992739953100681,
+            -0.02182719111442566,
+            0.0007546218694187701,
+            -0.01317158155143261,
+            -0.005782167427241802,
+            0.000176037588971667,
+            0.003330136416479945,
+            0.01956920698285103,
+            -0.025794345885515213,
+            -0.013038297183811665,
+            0.01205826923251152,
+            -0.019929856061935425,
+            -0.03572007268667221,
+            0.002736239228397608,
+            -0.029463572427630424,
+            -0.0013053978327661753,
+            -0.003777029225602746,
+            -0.015703974291682243,
+            -0.026217719539999962,
+            0.0060722557827830315,
+            -0.005366635508835316,
+            -0.005786087363958359,
+            0.003516341792419553,
+            -0.015233561396598816,
+            -0.015531489625573158,
+            0.005151029210537672,
+            0.007346292492002249,
+            -0.006503468379378319,
+            0.008561528287827969,
+            -0.0238186102360487,
+            0.011274246498942375,
+            -0.008145995438098907,
+            0.0037358682602643967,
+            0.005135348532348871,
+            0.026123635470867157,
+            0.008561528287827969,
+            -0.004000475630164146,
+            0.009306349791586399,
+            0.005472478456795216,
+            -0.010333418846130371,
+            0.011070400476455688,
+            0.02400677464902401,
+            0.0005184350302442908,
+            0.0032340935431420803,
+            -0.002973406109958887,
+            0.005746886134147644,
+            -0.001685648807324469,
+            -0.007612860295921564,
+            0.020933406427502632,
+            0.01593918167054653,
+            -0.019239917397499084,
+            0.0062369005754590034,
+            -0.020039619877934456,
+            0.010976317338645458,
+            -0.028742272406816483,
+            0.0005703765200451016,
+            0.005840969271957874,
+            -0.0050020646303892136,
+            0.011979866772890091,
+            0.006860198453068733,
+            0.03151771053671837,
+            0.026280440390110016,
+            -0.0018875346286222339,
+            0.0048178196884691715,
+            -0.02002394013106823,
+            0.01280309073626995,
+            0.009118183515965939,
+            0.010066851042211056,
+            -0.010694069787859917,
+            0.00791470892727375,
+            -0.00878889486193657,
+            0.014606342650949955,
+            -0.00260295532643795,
+            -0.014214331284165382,
+            0.006350583862513304,
+            -0.006287862081080675,
+            0.014441697858273983,
+            0.014081047847867012,
+            -0.013148060999810696,
+            -0.026923339813947678,
+            -0.006162418518215418,
+            0.018095243722200394,
+            -0.023912692442536354,
+            0.02443014644086361,
+            0.006464267149567604,
+            0.02265825681388378,
+            -0.009988449513912201,
+            -0.0030655288137495518,
+            -0.001984557369723916,
+            0.002657837001606822,
+            -0.00685235857963562,
+            0.00014345165982376784,
+            0.02243872918188572,
+            -0.009510194882750511,
+            0.005601841956377029,
+            0.012175872921943665,
+            -0.0016052864957600832,
+            0.03622184693813324,
+            0.018769502639770508,
+            0.024978963658213615,
+            -0.0017366103129461408,
+            0.010584305971860886,
+            0.0008266539080068469,
+            0.010537264868617058,
+            0.013845840469002724,
+            -0.010984158143401146,
+            0.0008913357742130756,
+            -0.014347615651786327,
+            0.017028972506523132,
+            0.024665353819727898,
+            0.009463153779506683,
+            0.0002709778491407633,
+            0.019506484270095825,
+            0.023191390559077263,
+            0.0031419708393514156,
+            -0.00707188481464982,
+            -0.007753984536975622,
+            0.017091695219278336,
+            0.01420649141073227,
+            0.007240449544042349,
+            0.0054254368878901005,
+            -0.007189488038420677,
+            -0.006036974955350161,
+            0.019192876294255257,
+            -0.019804412499070168,
+            -0.0018150125397369266,
+            0.0015288442373275757,
+            -0.04042420908808708,
+            0.010286377742886543,
+            -0.00031115900492295623,
+            -0.014669064432382584,
+            0.011180163361132145,
+            0.004072997719049454,
+            -0.010976317338645458,
+            -0.010176614858210087,
+            -0.016448795795440674,
+            -0.006879799067974091,
+            0.007593259681016207,
+            0.0024324303958564997,
+            0.008506646379828453,
+            -0.01050590444356203,
+            -0.005884090438485146,
+            0.019961217418313026,
+            0.003396778367459774,
+            0.0062369005754590034,
+            -0.0001754250843077898
+          ],
+          "keyphrases": [
+            "scaling up model capacity",
+            "training efficiency",
+            "sparsely gated mixture-of-experts",
+            "expert choice method",
+            "downstream performance"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "5d2725e3-01f4-4afd-b439-b42f131445a8",
+      "type": "child",
+      "source": {
+        "id": "b570d71e-4f83-486e-b70f-b0c6217db455",
+        "properties": {
+          "page_content": "Mixture-Of-Experts With Expert Choice Routing\n\nYanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon Google, Mountain View, CA, USA {yanqiz, taole, hanxiaol, dunan, huangyp, vzhao, adai, zhifengc, qvl, jlaudon}@google.com\n\nAbstract\n\nSparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choice method. Instead of letting tokens select the top-k experts, we have experts selecting the top-k tokens. As a result, each token can be routed to a variable number of experts and each expert can have a fixed bucket size. We systematically study pre-training speedups using the same computational resources of the Switch Transformer top-1 and GShard top-2 gating of prior work and find that our method improves training convergence time by more than 2×. For the same computational cost, our method demonstrates higher performance in fine-tuning 11 selected tasks in the GLUE and SuperGLUE benchmarks. For a smaller activation cost, our method outperforms the T5 dense model in 7 out of the 11 tasks.\n\n1 Introduction\n\nScaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27]. The final model quality has been found to have a power-law relationship with the amount of data, model size, and compute time [16, 20]. However, training efficiency, which is defined as the total amount of computation used to achieve superior model quality than the state of the art system [21], should receive greater attention as we increase our efforts towards green AI [29].\n\nSparsely gated mixture-of-experts [31] (MoE) provides an effective way to scale model capacity given a fixed computational cost, and has recently played an important role in increasing the training efficiency of large-scale language models [10, 21]. MoE operate by adopting a number of experts, each as a sub-network, and by activating only one or a few experts for each input token. A gating network must be chosen and optimized in order to route each token to the most suited expert(s). For example, recent work has implemented sparse routing via k-means clustering [12], linear assignment to maximize token-expert affinities [22], or hashing [8, 28]. Many of the prior work use a routing strategy concerning the token choice, where each token selects the best one or two experts.\n\nWe argue that the independent token choice of prior work often leads to an imbalanced load of experts, which causes training inefficiency and sub-optimal training of the model. In order to mitigate this\n\nissue, previous sparsely gated networks introduce additional auxiliary losses as regularization to prevent too many tokens being routed to a single expert, but the effectiveness is still limited. Recent approaches [8, 22, 28] explore alternative strategies for routing, but they focus on pre-training only and do not demonstrate performance gain on downstream tasks. Moreover, none of the previous methods consider allocating a variable number of experts to each token based on importance, which can be beneficial.\n\nWe propose a very simple yet effective routing method we are calling expert choice. Unlike conventional MoE where tokens select one or two top-scoring experts, our method lets each expert pick the top-k tokens. Our method guarantees perfect load balancing, allows a variable number of experts for each token, and achieves substantial gains in training efficiency and downstream performance as demonstrated in our experiments. Our major contributions include:\n\nWe identify common pitfalls in conventional MoE such as load imbalance as described in Section 3.1. We then propose a heterogeneous, expert choice method to provide a fluid allocation of model parameters based on a learnt token-to-expert importance. This method intrinsically guarantees load balance without imposing an auxiliary loss.\n\nWe show our method provides over 2× faster training convergence in a 8B/64E (8 billion activated parameters, 64 experts) model, compared to the top-1 and top-2 gating counterparts in Switch Transformer [10] and GShard [21].\n\nWe show our method demonstrates strong scaling when increasing the number of experts from 16 to 128, evaluated in training perplexity.\n\nWe show our method demonstrates strong performance on downstream tasks selected from GLUE and SuperGLUE at all the evaluated scales. More specifically, our 8B/64E model outperforms a T5 11B dense model in 7 out of 11 tasks evaluated.\n\n2 Related Work\n\nScaling: Various approaches have been proposed to scale up neural network capacity to improve performance. Recent works have successfully scaled models to billions of parameters via various forms of model parallelism [2, 21, 26, 27, 33]. Model parallelism [30] splits weights and tensors across multiple cores while pipeline parallelism [18, 24] splits different layers across devices with micro-batches pipelined to the different layers. To enable continued scaling of neural networks, improving model training and serving efficiency has become a critical research area.\n\nConditional Computation: Computation decisions can be made dynamically based on the input [23, 25]. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation, by activating certain parameters and computation on demand, on a per-example or per-token basis [3]. Conditional convolution layers [1] with task-specific gating has been used to combat catastrophic forgetting when a sequence of learning problems are optimized. The gating decisions may be binary or sparse and continuous, stochastic or deterministic.\n\nMixture of Experts: Sparsely-gated MoE [31] is the first model to demonstrate massive improvements in model capacity, training time, or model quality with gating. Switch Transformer [10] simplifies the gating by selecting only the top expert per token using a softmax over the hidden state and demonstrates better scaling than previous work. All the prior work requires an auxiliary loss to explicitly encourage balancing. This loss term has to be carefully weighted to not overwhelm the primary loss. However, auxiliary loss does not guarantee balancing and a hard capacity factor has to be imposed. As a result, many tokens can still be unprocessed by the MoE layer. Hard MoE [12] with a single decoding layer can be efficiently trained to good effect on large scale hashtag prediction tasks.\n\nBase Layers [22] formulate a linear assignment that maximizes token-expert affinities while ensuring each expert receives an equal number of tokens. Hash layers [8, 28] devise hashing techniques on input tokens. However, the evaluations are limited to pre-training perplexity. THOR [? ] randomly activates experts during training and inference and is trained with a consistency regularization loss.\n\nTHOR has demonstrated strong performance on translation tasks. Different from these prior works, our method is a learnt method that enables heterogeneous MoE and effectively improves downstream fine-tuning performance.\n\n3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n5 Conclusion\n\nWe propose a new routing method for sparsely activated mixture-of-experts (MoE) models. This method addresses load imbalance and under-utilization of experts in conventional MoE methods, and enables selecting different numbers of experts for each token. Our model demonstrates more than 2x training efficiency improvements when compared to the state-of-the-art GShard and Switch Transformer models, and also achieves strong gains when finetuning on 11 datasets in the GLUE and SuperGLUE benchmark.\n\n6 Limitations\n\nThe expert choice method might not immediately apply to auto-regressive text generation as our current implementation takes in the past and future tokens to perform the top-k selection. One possible solution is to collect a large batch of input sequences, dispatch tokens of the same sequence into separate groups, and perform expert choice routing for each group. Another scenario where the expert choice method does not immediately apply is when the batch size becomes very small during serving or inference. A global top-k can be selected instead and we can cap the number of times each expert or token gets selected. We leave these possible improvements for future work.\n\nAnother long-standing issue with MoE has been the large memory footprint. Even though computational cost can be reduced using sparsely gated networks, the total number of parameters increases linearly or sub-linearly with the number of experts. Increasing the number of experts requires reservation of a large number of hardware devices. Therefore, dynamic (used) power is saved while static (reserved) power is not. Power saving techniques such as the ability to put hardware devices into low power states while not in use [17] can help with reducing the reserved power requirements.\n\nReferences\n\n[1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pages 3930–3939. Computer Vision Foundation / IEEE, 2020.\n\n[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. In Advances in Neural Information Processing Systems.\n\n[3] Kyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning, 2014.\n\n[4] Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems, 2021.\n\n[5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.\n\nTransformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.\n\n[6] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 933–941. JMLR.org, 2017.\n\n[7] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixtureof-experts, 2021.\n\n[8] Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan.\n\nTricks for training sparse translation models, 2021.\n\n[9] Richard L Dykstra. An iterative procedure for obtaining i-projections onto the intersection of convex sets. The annals of Probability, pages 975–984, 1985.\n\n[10] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.\n\n[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. NAS-FPN: learning scalable feature pyramid architecture for object detection. In CVPR, pages 7036–7045. Computer Vision Foundation / IEEE, 2019.\n\n[12] Sam Gross, Marc'Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision, 2017.\n\n[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.\n\n[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 630–645, Cham, 2016. Springer International Publishing.\n\n[15] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2016.\n\n[16] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.\n\n[17] Ping Huang, Zuocheng Xing, Tianran Wang, Qiang Wei, Hongyan Wang, and Guitao Fu. A brief survey on power gating design. In 2010 10th IEEE International Conference on Solid-State and Integrated Circuit Technology, pages 788–790, 2010.\n\n[18] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 103–112, 2019.\n\n[19] Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David A. Patterson. A domain-specific supercomputer for training deep neural networks. Commun. ACM, 63(7):67–78, 2020.\n\n[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.\n\n[21] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.\n\n[22] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6265–6274. PMLR, 18–24 Jul 2021.\n\n[24] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. New York, NY, USA, 2019. Association for Computing Machinery.\n\n[25] Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, Cédric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. In ICLR. OpenReview.net, 2021.\n\n[26] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.\n\n[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.\n\n[28] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models, 2021.\n\n[30] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 10435–10444, Red Hook, NY, USA, 2018. Curran Associates Inc.\n\n[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E.\n\nHinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In ICLR (Poster). OpenReview.net, 2017.\n\n[32] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 10–15 Jul 2018. [23] Min Lin, Jie Fu, and Yoshua Bengio. Conditional computation for continual learning, 2019.\n\n[29] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.\n\n[33] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.\n\n[34] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc.\n\n[35] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.\n\nGLUE: A multi-task benchmark and analysis platform for natural language understanding.\n\nIn Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, November 2018. Association for Computational Linguistics.\n\n[36] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake A. Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen. GSPMD: general and scalable parallelization for ML computation graphs. CoRR, abs/2105.04663, 2021.\n\n7 Checklist\n\n(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Yes (b) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes (c) Did you discuss any potential negative societal impacts of your work? N/A. Not any. (d) Did you describe the limitations of your work? Yes (a) Did you include the code, data, and instructions needed to reproduce the main experimental results? Yes. We include details in the experiment setup to help reproduce the main results. (b) Did you specify all the training details? Yes (c) Did you report error bars? Yes (d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? Yes (a) If your work uses existing assets, did you cite the creators? Yes (b) Did you mention the license of the assets? No. The used dataset is not released yet. (c) Did you include any new assets either in the supplemental material or as a URL? No. The dataset is not released yet.\n\n(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? No. Not using persons' data. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes. The dataset does not contain any personally identifiable information or offensive content.\n\nA Comparison On Fine-Tuning With A Dense Model\n\nOur 8B MoE model achieves stronger pre-training perplexity than its dense counterpart. However, a better perplexity does not always directly translate to downstream performance as demonstrated in Section 4.4. To this end, we compare fine-tuning performance of the 8B dense model and MoE model in Table 1. As shown in the table, our MoE model using expert choice routing consistently outperforms the dense model across the 11 tasks in GLUE and SuperGLUE.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 1: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nB Capacity Factor\n\nWe evaluate the downstream task fine-tuning performance by varying the capacity factors. Note that a capacity factor of n indicates on average how many experts each token can be received. EC-CF2 is our baseline expert choice, which matches GShard top-2 gating computational footprint. EC-CF1, however, matches Switch Transformer top-1 gating computational footprint. EC-CF0.5 further verifies that an aggressively lowered capacity factor can provide strong enough performance, that almost matches the top-2 gating baseline.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Top-2 78.1 87.0 88.3 85.0 82.6 90.1 90.7 81.6 94.7 68.2 67.2 83.0±0.3 EC-CAP2 78.2 88.0 88.5 85.7 83.0 90.8 91.1 80.0 95.4 70.4 64.1 83.2±0.4 EC-CAP3 78.5 91.7 89.3 86.3 83.5 90.9 91.1 81.8 94.9 70.0 65.6 84.0±0.4 EC-CF2 79.1 89.6 89.3 86.8 84.3 91.3 91.2 81.1 95.2 68.1 68.0 84.0±0.2 EC-CF1 77.4 90.6 88.0 85.5 83.6 90.3 91.2 79.8 95.3 66.5 64.9 83.0±0.2 EC-CF0.5 77.4 89.6 86.3 85.2 82.7 91.7 91.0 79.6 94.9 67.3 63.5 83.0 ±0.05 Hash Layers 76.1 85.2 86.7 83.4 82.5 90.0 90.3 75.7 94.0 67.4 63.3 81.3±1.0\n\nTable 2: Comparison between different routing methods in fine-tuning of 100M/64E models. We perform 3 independent fine-tuning runs for each method and report the average results. This gives more accurate difference between the variants of expert choice method, since they achieve close fine-tuning results. We do not report averaged results in other experiments.\n\nC Capped Expert Choice\n\nAs described in Section 4.5, the maximum number of experts each token is assigned can be capped by an entropy-regularized linear programming. Figure 1 compares the validation perplexity when training the 100M/64E models using the base expert choice method (EC-BASE), expert choice capped by two experts per token (EC-CAP2), expert choice capped by three experts per token (EC-CAP3), and GShard top-2 gating.\n\nAs shown in the figure, restricting the number of experts to 2 degrades the perplexity compared to the base expert choice method. This suggests that a more flexible allocation of experts (e.g. more than 2 experts for a token) can enhance model expressiveness. On the other hand, our EC-CAP2 and EC-CAP3 methods still outperform the top-2 gating method by a clear margin. We believe this confirms the effectiveness of a load balanced training, provided by our method. Finally, EC-CAP3 obtains comparable perplexity to EC-BASE. As indicated by Figure 3, only a little fraction of tokens use more than 3 experts therefore we see little or no difference between EC-BASE and EC-CAP3 variants. We present the fine-tuning results of these methods in Table 2.\n\nD Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [? ]. We use mod x to map a token ID to an expert ID. This in some way ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 2. Hashing based routing performs much worse than expert choice in terms of average scores and variance.\n\nE Fine-Tuning Details\n\nWe did a hyperparameter search for both baseline models and expert choice method. For fine-tuning of the 8B dense model, we use a constant learning rate of 0.0001 and a dropout rate of 0.1. We freeze the attention layer and feed-forward layer while leaving the embedding and layer normalization trainable. This setting has been found optimal for the 8B dense model. For MoE 8B/64E models including GShard top-2 gating and expert choice, we found continuing the learning rate from the pre-trained model while using a square root learning rate decay works better. In addition, we do not apply parameter freezing for fine-tuning MoE models. For models with 100M expert size, we use a constant learning rate of 0.0001 and no dropout is used.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/2202.09368v2.md"
+          },
+          "headlines": {
+            "1 Introduction": [],
+            "2 Related Work": [],
+            "3 Method": [
+              "3.1 Pitfalls Of Token-Choice Routing",
+              "3.2 Heterogeneous Moe Via Expert Choice",
+              "3.3 Expert Choice With Additional Constraint",
+              "3.4 Model Architecture"
+            ],
+            "4 Experiments": [
+              "4.1 Setup",
+              "4.2 Training Efficiency",
+              "4.3 Scaling The Number Of Experts",
+              "4.4 Fine-Tuning On Glue And Superglue",
+              "4.5 Heterogeneity Matters",
+              "4.6 Comparison With Hash Layer",
+              "4.7 Ablation"
+            ],
+            "5 Conclusion": [],
+            "6 Limitations": [],
+            "A Comparison On Fine-Tuning With A Dense Model": [],
+            "B Capacity Factor": [],
+            "C Capped Expert Choice": [],
+            "D Comparison With Hash Layer": [],
+            "E Fine-Tuning Details": []
+          },
+          "summary": "The paper introduces a new routing method for sparsely-activated Mixture-of-Experts (MoE) models called 'expert choice,' which addresses load imbalance and under-utilization issues in conventional MoE methods. Unlike traditional token-choice routing, expert choice allows experts to select the top-k tokens, ensuring perfect load balancing and enabling variable expert allocation per token. This method improves training efficiency, achieving over 2x faster convergence compared to existing models like Switch Transformer and GShard. It also demonstrates superior performance on downstream tasks in the GLUE and SuperGLUE benchmarks. The study highlights the benefits of flexible expert allocation and shows that expert choice outperforms hashing-based routing methods. The paper also discusses potential limitations and future improvements for the expert choice method.",
+          "summary_embedding": [
+            -0.009726986289024353,
+            -0.0337262786924839,
+            -0.023302342742681503,
+            -0.019878201186656952,
+            -0.025953780859708786,
+            0.04560471698641777,
+            -0.006874797400087118,
+            0.039665497839450836,
+            -0.006973279174417257,
+            0.034544438123703,
+            -0.025787118822336197,
+            -0.00363057479262352,
+            -0.01052999310195446,
+            0.0054657477885484695,
+            -0.024499278515577316,
+            0.036786794662475586,
+            0.010007280856370926,
+            0.003272630739957094,
+            0.006102093029767275,
+            -0.04048365354537964,
+            -0.018666116520762444,
+            -0.013522328808903694,
+            -0.03230207785964012,
+            0.008969432674348354,
+            0.02484775334596634,
+            0.0016713523073121905,
+            0.0026230288203805685,
+            0.003624893259257078,
+            -0.05311964824795723,
+            0.04339266195893288,
+            0.018681267276406288,
+            0.023711422458291054,
+            -0.016332851722836494,
+            -0.012832955457270145,
+            -0.039120059460401535,
+            -0.0197418425232172,
+            0.030786970630288124,
+            0.039968520402908325,
+            -0.019287310540676117,
+            0.03357476741075516,
+            0.018060073256492615,
+            0.02454473078250885,
+            -0.021272100508213043,
+            -0.029817303642630577,
+            -0.0061892117373645306,
+            -0.007840678095817566,
+            -0.008355814032256603,
+            0.01242387667298317,
+            0.016984347254037857,
+            -0.013007191941142082,
+            0.0212569497525692,
+            -0.01371171697974205,
+            0.03384748846292496,
+            0.008916404098272324,
+            -0.042332086712121964,
+            0.03984731063246727,
+            -0.008007340133190155,
+            0.012105703353881836,
+            0.03281721472740173,
+            0.010333028621971607,
+            -0.005287723150104284,
+            -0.021544819697737694,
+            -0.03378688171505928,
+            0.0010529992869123816,
+            0.00014014738553669304,
+            0.005950582213699818,
+            -0.003069985192269087,
+            -0.009742137044668198,
+            0.03227177634835243,
+            -0.003939277958124876,
+            -0.0740584209561348,
+            0.050089433789253235,
+            -0.026590125635266304,
+            -0.007628562860190868,
+            -0.028938541188836098,
+            -0.01736312545835972,
+            0.037362534552812576,
+            0.005662711802870035,
+            0.0098709212616086,
+            0.0106284748762846,
+            -0.0007892759749665856,
+            0.028923390433192253,
+            0.00910579226911068,
+            -0.024923507124185562,
+            0.04478655755519867,
+            -0.04790768027305603,
+            0.01801462098956108,
+            -0.04336235672235489,
+            0.022756904363632202,
+            1.6393929399782792e-05,
+            -0.020726662129163742,
+            -0.0013304532039910555,
+            -0.016666175797581673,
+            0.0218023881316185,
+            -0.01730252057313919,
+            0.026802239939570427,
+            0.013825350441038609,
+            -0.015802564099431038,
+            -0.029817303642630577,
+            0.025378039106726646,
+            0.003367325058206916,
+            0.010499690659344196,
+            -0.02075696364045143,
+            0.014332910999655724,
+            0.04393810033798218,
+            -0.02593863010406494,
+            -0.02271145209670067,
+            0.027681002393364906,
+            0.014408666640520096,
+            0.0633314698934555,
+            0.036544378846883774,
+            0.006624804809689522,
+            0.022362977266311646,
+            0.020575150847434998,
+            -0.028968842700123787,
+            -0.01963578537106514,
+            -0.046362269669771194,
+            -0.04757435619831085,
+            0.019878201186656952,
+            -0.04566532000899315,
+            0.0050869714468717575,
+            0.038544319570064545,
+            -0.013704141601920128,
+            -0.014446543529629707,
+            -0.04021093621850014,
+            0.017847958952188492,
+            0.0003908502112608403,
+            0.011976920068264008,
+            -0.0022082682698965073,
+            0.00033285003155469894,
+            -0.011825408786535263,
+            0.008878526277840137,
+            -0.03778676688671112,
+            -0.025196226313710213,
+            0.025181075558066368,
+            0.022862963378429413,
+            0.024120500311255455,
+            0.028802180662751198,
+            -0.03077181987464428,
+            -0.009984553791582584,
+            0.021378157660365105,
+            -0.04242299124598503,
+            -0.006556625012308359,
+            0.011332998983561993,
+            0.0026703758630901575,
+            0.003872992005199194,
+            0.032786913216114044,
+            0.052998438477516174,
+            -0.03172633796930313,
+            0.02065090648829937,
+            -0.010431510396301746,
+            0.04957429692149162,
+            0.00024691506405360997,
+            0.028968842700123787,
+            -0.019514575600624084,
+            -0.0134389977902174,
+            -0.029529431834816933,
+            -0.009795165620744228,
+            -0.01356020662933588,
+            -0.008484598249197006,
+            -0.002998017705976963,
+            -0.009575475007295609,
+            0.0040453351102769375,
+            0.0331808403134346,
+            0.00273855566047132,
+            0.030680913478136063,
+            0.014325335621833801,
+            0.0051475754007697105,
+            -0.08745196461677551,
+            0.026059838011860847,
+            -0.0033616432920098305,
+            0.005859675817191601,
+            0.0062460280023515224,
+            0.03013547509908676,
+            -0.0312718041241169,
+            0.020938776433467865,
+            0.026620427146553993,
+            0.0011145505122840405,
+            0.016969196498394012,
+            0.036847397685050964,
+            -0.04181694984436035,
+            0.009643655270338058,
+            0.012590537779033184,
+            -0.0033881578128784895,
+            0.04584713280200958,
+            0.011628445237874985,
+            0.019514575600624084,
+            0.018529755994677544,
+            0.003569970605894923,
+            0.00297907879576087,
+            0.006533898413181305,
+            0.03724132478237152,
+            0.003462019143626094,
+            -0.009833043441176414,
+            0.014976831153035164,
+            -0.010757259093225002,
+            -0.011219366453588009,
+            -0.037998881191015244,
+            0.014454119838774204,
+            0.008742166683077812,
+            -0.017454031854867935,
+            -0.02959003672003746,
+            0.0026949963066726923,
+            -0.031908150762319565,
+            0.026559824123978615,
+            -0.002445003716275096,
+            0.005738467443734407,
+            -0.010454237461090088,
+            -0.006757376715540886,
+            -0.035423196852207184,
+            0.028453707695007324,
+            -0.02095392718911171,
+            0.0053142374381423,
+            0.0007258309051394463,
+            -0.029438525438308716,
+            0.0060073984786868095,
+            0.017863109707832336,
+            -0.0025965142995119095,
+            0.0025302283465862274,
+            -0.02349930815398693,
+            0.024468975141644478,
+            0.025635607540607452,
+            0.004518806003034115,
+            -0.014037464745342731,
+            -0.013113250024616718,
+            -0.003605954349040985,
+            -0.009742137044668198,
+            -0.016196493059396744,
+            0.024468975141644478,
+            -0.014810169115662575,
+            0.00308892410248518,
+            0.021847840398550034,
+            0.014264730736613274,
+            -0.007045246660709381,
+            -0.004007457755506039,
+            -0.04066546633839607,
+            -0.01218903437256813,
+            -0.04760465770959854,
+            -0.011628445237874985,
+            -0.0034601252991706133,
+            -0.028696123510599136,
+            0.0061513339169323444,
+            -0.04481685906648636,
+            -0.01235569640994072,
+            0.02309022843837738,
+            0.018135828897356987,
+            0.0019488062243908644,
+            -0.009173971600830555,
+            -0.022605394944548607,
+            8.486965816700831e-05,
+            -0.01763584464788437,
+            -0.017044952139258385,
+            0.015393485315144062,
+            0.031059689819812775,
+            -0.03101423755288124,
+            0.04024123772978783,
+            0.003893824527040124,
+            -0.08066429197788239,
+            0.010393633507192135,
+            -0.026362858712673187,
+            0.07011914253234863,
+            -0.0025453795678913593,
+            0.014075342565774918,
+            0.06781618297100067,
+            -0.012431452050805092,
+            -0.02007516659796238,
+            0.03142331540584564,
+            -0.008249756880104542,
+            -1.1311515663692262e-05,
+            -0.02884763479232788,
+            -0.02739313244819641,
+            -0.053271155804395676,
+            -0.009704259224236012,
+            -0.010787560604512691,
+            0.03360506892204285,
+            -0.024317465722560883,
+            -0.0007656024536117911,
+            0.012620840221643448,
+            0.02983245439827442,
+            0.016635872423648834,
+            -0.0015681355725973845,
+            0.008272483013570309,
+            0.02881733328104019,
+            0.0274234339594841,
+            -0.0036154238041490316,
+            0.00814369972795248,
+            -0.034211114048957825,
+            -0.0068066176027059555,
+            -0.0493621826171875,
+            0.011302697472274303,
+            -0.02596893161535263,
+            -0.02596893161535263,
+            0.007643714081496,
+            -0.011476934887468815,
+            0.03748374432325363,
+            -0.024696242064237595,
+            -0.007503566797822714,
+            0.005037730094045401,
+            0.012855681590735912,
+            -0.017757052555680275,
+            -0.03612014651298523,
+            -0.0028502948116511106,
+            0.0026381798088550568,
+            0.007560383062809706,
+            -0.053240854293107986,
+            0.004499867558479309,
+            -0.018469152972102165,
+            0.0279688723385334,
+            -0.02139330841600895,
+            -0.013688990846276283,
+            0.042029064148664474,
+            0.011363301426172256,
+            0.027499189600348473,
+            -0.026468917727470398,
+            0.0017849853029474616,
+            0.012825379148125648,
+            -0.0020056227222085,
+            0.032514192163944244,
+            -0.030574856325984,
+            -0.03257479891180992,
+            0.001548249856568873,
+            -0.007840678095817566,
+            -0.0625436082482338,
+            -0.011386028490960598,
+            -0.00981789268553257,
+            0.017317671328783035,
+            0.0446956530213356,
+            0.04557441547513008,
+            -0.01068150345236063,
+            0.002174178371205926,
+            -0.02213571034371853,
+            -0.04542290419340134,
+            -0.02643861435353756,
+            0.0016486257081851363,
+            -0.019999410957098007,
+            -0.02789311669766903,
+            -0.0018313854234293103,
+            0.011211791075766087,
+            -0.013651113025844097,
+            0.01134057529270649,
+            -0.018560059368610382,
+            -0.01862066239118576,
+            -0.012332970276474953,
+            0.018575210124254227,
+            0.009901223704218864,
+            -0.0018796793883666396,
+            -0.015037435106933117,
+            -0.00980274099856615,
+            0.018969137221574783,
+            -0.011279970407485962,
+            0.0009388927719555795,
+            0.01188601367175579,
+            0.024347767233848572,
+            -0.007083124481141567,
+            0.010961798019707203,
+            -0.034756552428007126,
+            -0.01966608688235283,
+            0.007984613068401814,
+            0.002719616750255227,
+            0.020711511373519897,
+            0.008742166683077812,
+            -0.04275631532073021,
+            -0.044301725924015045,
+            0.016044981777668,
+            -0.0562104657292366,
+            0.028120383620262146,
+            -0.03327174484729767,
+            -0.012514783069491386,
+            -0.02671133354306221,
+            0.0013418165035545826,
+            -0.01828734017908573,
+            0.022438732907176018,
+            0.03778676688671112,
+            -0.0011486404109746218,
+            0.005503625608980656,
+            0.014658658765256405,
+            0.04384719207882881,
+            -0.05021063983440399,
+            -0.041180603206157684,
+            0.03754434734582901,
+            -0.01601468026638031,
+            0.0185146052390337,
+            0.00021744151308666915,
+            0.010348179377615452,
+            -0.014431392773985863,
+            0.022772056981921196,
+            0.013757170177996159,
+            -0.008764892816543579,
+            -0.030574856325984,
+            0.04945308715105057,
+            -0.023650817573070526,
+            -0.049513693898916245,
+            0.0018323323456570506,
+            0.018878230825066566,
+            0.018635815009474754,
+            -0.014870774000883102,
+            -0.02907489985227585,
+            -0.024317465722560883,
+            -0.006060427520424128,
+            -0.00947699323296547,
+            -0.03402930125594139,
+            0.011363301426172256,
+            0.0048445542342960835,
+            0.016666175797581673,
+            -0.00645056739449501,
+            -0.0073141781613230705,
+            -0.009370936080813408,
+            0.022559940814971924,
+            0.012635990977287292,
+            -0.03175663948059082,
+            -0.017484333366155624,
+            -0.02528713271021843,
+            0.004359719809144735,
+            0.010136065073311329,
+            0.022166013717651367,
+            0.024287162348628044,
+            -0.0323626808822155,
+            0.04218057543039322,
+            0.01550711877644062,
+            -0.008727015927433968,
+            -0.0019317612750455737,
+            0.02478714846074581,
+            0.04324115067720413,
+            0.0218023881316185,
+            0.026514369994401932,
+            0.002107892418280244,
+            8.705945947440341e-05,
+            -0.015620751306414604,
+            -0.0011391709558665752,
+            0.01862066239118576,
+            0.04911976307630539,
+            -0.012151156552135944,
+            0.01716616004705429,
+            -0.03578682243824005,
+            0.00396200455725193,
+            -0.02580226957798004,
+            -0.030786970630288124,
+            0.03745344281196594,
+            -0.015060162171721458,
+            0.0191964041441679,
+            0.025044716894626617,
+            -0.022469034418463707,
+            -0.011075431481003761,
+            -0.019423669204115868,
+            -0.026181045919656754,
+            -0.01702980138361454,
+            -0.03281721472740173,
+            -0.018135828897356987,
+            -0.002653330797329545,
+            -0.035635314881801605,
+            -0.04987731948494911,
+            -0.004329417832195759,
+            -0.046392571181058884,
+            -0.010507266037166119,
+            0.017378276214003563,
+            -0.011166337877511978,
+            -0.025711363181471825,
+            -0.005336964037269354,
+            0.003973367623984814,
+            -0.022196315228939056,
+            -0.0066588944755494595,
+            0.008439145050942898,
+            0.035938333719968796,
+            -0.02298417128622532,
+            -0.013060221448540688,
+            -0.0024317465722560883,
+            -0.02887793630361557,
+            -0.007878555916249752,
+            0.008999735116958618,
+            0.007723257411271334,
+            -0.0012357590021565557,
+            -0.05196816474199295,
+            -0.006916462909430265,
+            -0.012992041185498238,
+            0.029196109622716904,
+            0.026468917727470398,
+            0.0012168202083557844,
+            -0.02783251367509365,
+            0.008749742060899734,
+            -0.0072535742074251175,
+            0.02339324913918972,
+            -0.007992188446223736,
+            -0.022908415645360947,
+            -0.02034788578748703,
+            0.030786970630288124,
+            -0.0159086212515831,
+            0.009113367646932602,
+            0.009976978413760662,
+            0.0121360057964921,
+            -0.03899884968996048,
+            -0.001445980160497129,
+            0.0005648507503792644,
+            -0.010128489695489407,
+            0.00423093605786562,
+            -0.006738437805324793,
+            -0.007079336792230606,
+            -0.022090258076786995,
+            -0.0218023881316185,
+            0.03818069398403168,
+            -0.009113367646932602,
+            -0.0035661826841533184,
+            -0.01088604237884283,
+            0.03436262533068657,
+            -0.00595815759152174,
+            -0.01993880607187748,
+            -0.019347915425896645,
+            0.014431392773985863,
+            0.05105910077691078,
+            -0.028529461473226547,
+            -0.03984731063246727,
+            0.0011136035900563002,
+            0.030150625854730606,
+            0.004140029661357403,
+            -0.016908591613173485,
+            0.0035851215943694115,
+            0.056392278522253036,
+            -0.011742077767848969,
+            0.028362801298499107,
+            0.027771908789873123,
+            0.02295386977493763,
+            -0.02021152526140213,
+            -0.01678738370537758,
+            -0.020999381318688393,
+            0.011098157614469528,
+            -0.03027183562517166,
+            0.027059808373451233,
+            0.018969137221574783,
+            0.01538590993732214,
+            0.05054396390914917,
+            -0.03178694099187851,
+            -0.02515077404677868,
+            0.008499749936163425,
+            0.0021722845267504454,
+            0.011007251217961311,
+            -0.007727045100182295,
+            -0.01801462098956108,
+            -0.001950700068846345,
+            -0.014022313989698887,
+            -0.019817598164081573,
+            -0.020938776433467865,
+            0.00954517349600792,
+            0.015696506947278976,
+            0.002649543108418584,
+            0.010863316245377064,
+            0.0016590419691056013,
+            0.01686313934624195,
+            0.01321930717676878,
+            -0.0444229319691658,
+            -7.936555630294606e-05,
+            0.03378688171505928,
+            -0.0293476190418005,
+            -0.012264790013432503,
+            -0.016060132533311844,
+            -0.004439263138920069,
+            -0.01706010289490223,
+            -0.02762039750814438,
+            -0.006874797400087118,
+            -0.01889338344335556,
+            -0.00814369972795248,
+            -0.028196139261126518,
+            -0.009992130100727081,
+            0.01949942484498024,
+            0.022332675755023956,
+            0.001196934375911951,
+            -0.006783891003578901,
+            -0.005901341326534748,
+            0.02092362567782402,
+            0.032514192163944244,
+            0.01952972821891308,
+            0.0058331615291535854,
+            -0.015158643946051598,
+            -0.004295328166335821,
+            -0.012370847165584564,
+            0.016060132533311844,
+            0.012537509202957153,
+            6.344509893096983e-05,
+            0.03493836522102356,
+            0.004170331638306379,
+            -0.022726602852344513,
+            -0.001230077352374792,
+            0.024923507124185562,
+            -0.021590271964669228,
+            -0.0108330138027668,
+            0.040029123425483704,
+            0.0085679292678833,
+            -0.0024506852496415377,
+            -0.019287310540676117,
+            -0.019075196236371994,
+            0.02630225569009781,
+            -0.02133270539343357,
+            0.002083271974697709,
+            0.0482713058590889,
+            0.00035344602656550705,
+            -0.029802151024341583,
+            0.017954016104340553,
+            -0.013242034241557121,
+            -0.018560059368610382,
+            0.01848430372774601,
+            0.0012859469279646873,
+            -0.039968520402908325,
+            0.019014591351151466,
+            0.018408548086881638,
+            0.022635696455836296,
+            -0.02918095886707306,
+            0.0002127068000845611,
+            0.03233237937092781,
+            0.004037759732455015,
+            0.001595596899278462,
+            -0.008530051447451115,
+            -0.0006065162015147507,
+            0.010431510396301746,
+            0.001746160676702857,
+            0.02854461409151554,
+            -0.0030434709042310715,
+            0.027771908789873123,
+            -0.006727074272930622,
+            -0.008863375522196293,
+            0.019181253388524055,
+            -0.02109028771519661,
+            0.013166278600692749,
+            0.0685434341430664,
+            0.0021950111258774996,
+            0.025847723707556725,
+            0.007181606721132994,
+            0.024347767233848572,
+            0.02018122375011444,
+            -0.027271922677755356,
+            -0.02021152526140213,
+            -0.005397567991167307,
+            -0.0014033677289262414,
+            -0.03787767142057419,
+            0.011128460057079792,
+            -0.021302402019500732,
+            -0.013249609619379044,
+            -0.0022347825579345226,
+            -0.002316219499334693,
+            -0.006405114196240902,
+            0.034211114048957825,
+            -0.0014630251098424196,
+            -0.018560059368610382,
+            -0.035423196852207184,
+            0.037726160138845444,
+            0.04048365354537964,
+            0.0389382466673851,
+            -0.005984671879559755,
+            0.03548380360007286,
+            0.0030756669584661722,
+            0.0025302283465862274,
+            0.041453324258327484,
+            0.004791525658220053,
+            -0.011567841283977032,
+            0.02295386977493763,
+            0.012416300363838673,
+            0.03351416438817978,
+            -0.02045394293963909,
+            -0.02630225569009781,
+            -0.003299145260825753,
+            -0.026423463597893715,
+            0.012492056004703045,
+            -0.019726691767573357,
+            0.0018768386216834188,
+            0.007117214612662792,
+            0.015196521766483784,
+            -0.016044981777668,
+            0.014719262719154358,
+            -0.006102093029767275,
+            -0.021014532074332237,
+            0.011719351634383202,
+            -0.007780073676258326,
+            0.011961768381297588,
+            -0.02262054570019245,
+            0.001138224033638835,
+            -0.04063516482710838,
+            0.008280059322714806,
+            0.014022313989698887,
+            -0.01598437689244747,
+            0.03205966204404831,
+            0.020969079807400703,
+            -0.01801462098956108,
+            -0.0009682480012997985,
+            0.0034714885987341404,
+            -0.010393633507192135,
+            -0.011067855171859264,
+            -0.019923655316233635,
+            0.02786281518638134,
+            -0.015802564099431038,
+            -0.025908326730132103,
+            0.02454473078250885,
+            -0.01743887923657894,
+            0.015151068568229675,
+            -0.004299115855246782,
+            0.00888610165566206,
+            -0.019120648503303528,
+            0.02284781076014042,
+            -0.023741723969578743,
+            -0.0009128518868237734,
+            -0.020287280902266502,
+            0.0028389315120875835,
+            0.014135947450995445,
+            -0.018272189423441887,
+            -0.02139330841600895,
+            0.019150950014591217,
+            -0.005420294590294361,
+            -0.003931702114641666,
+            0.010014856234192848,
+            0.0030983935575932264,
+            -0.0029715034179389477,
+            -0.004549108445644379,
+            -0.03281721472740173,
+            -0.02915065549314022,
+            0.0033086147159337997,
+            0.011386028490960598,
+            0.015257125720381737,
+            0.027574945241212845,
+            0.03715042024850845,
+            0.006052851676940918,
+            0.0036949669010937214,
+            0.0009327376610599458,
+            -0.020726662129163742,
+            0.042635105550289154,
+            -0.020469093695282936,
+            0.040847279131412506,
+            0.032544493675231934,
+            -0.0197418425232172,
+            -0.023908386006951332,
+            0.0029052174650132656,
+            0.03115059621632099,
+            0.03151422366499901,
+            0.025211378931999207,
+            0.012310243211686611,
+            0.03690800443291664,
+            -0.004359719809144735,
+            -0.032211173325777054,
+            0.015802564099431038,
+            0.022256920114159584,
+            -0.008105821907520294,
+            0.009673957712948322,
+            0.027438584715127945,
+            0.013279912061989307,
+            -0.007757347077131271,
+            0.030150625854730606,
+            0.0012234487803652883,
+            0.02396899089217186,
+            0.02498411200940609,
+            -0.02190844528377056,
+            0.01743887923657894,
+            -0.0001154677156591788,
+            0.006727074272930622,
+            -0.009840618818998337,
+            0.004348356742411852,
+            0.011810258030891418,
+            0.018272189423441887,
+            0.018908534198999405,
+            0.003246116451919079,
+            -0.07060398161411285,
+            0.019726691767573357,
+            0.004507442936301231,
+            -0.02160542458295822,
+            0.00390140013769269,
+            -0.0024866692256182432,
+            0.012704171240329742,
+            0.02105998620390892,
+            0.012461753562092781,
+            -0.01790856383740902,
+            0.0013626492582261562,
+            0.011961768381297588,
+            -0.00595815759152174,
+            0.04730163514614105,
+            0.0019791084341704845,
+            -0.01437078882008791,
+            -0.016105586662888527,
+            0.013726867735385895,
+            0.02525683119893074,
+            -0.02355991117656231,
+            0.006325571332126856,
+            0.005621046293526888,
+            -0.004147605039179325,
+            -0.010742107406258583,
+            -0.030377892777323723,
+            -0.005280147306621075,
+            -0.03436262533068657,
+            0.023150833323597908,
+            0.015491968020796776,
+            0.014764715917408466,
+            -0.025044716894626617,
+            -0.004280176945030689,
+            0.0046400148421525955,
+            -0.021287251263856888,
+            0.003261267440393567,
+            0.01161329448223114,
+            0.0005104016163386405,
+            -0.020984230563044548,
+            0.00959820207208395,
+            0.013469300232827663,
+            0.06848283112049103,
+            0.0007442962378263474,
+            0.029771849513053894,
+            0.0422411784529686,
+            -0.0012243957025930285,
+            -0.006371024530380964,
+            0.0036286809481680393,
+            -0.019953956827521324,
+            0.003473382443189621,
+            0.025878025218844414,
+            -0.03224147483706474,
+            -0.055058982223272324,
+            -0.007545232307165861,
+            -0.04624105989933014,
+            0.001489539397880435,
+            0.014711687341332436,
+            0.023044776171445847,
+            -0.0037782976869493723,
+            -0.034817155450582504,
+            0.016908591613173485,
+            -0.003071879269555211,
+            0.007014944683760405,
+            -0.02593863010406494,
+            -0.0294839795678854,
+            -0.026014383882284164,
+            -0.02339324913918972,
+            -0.012908710166811943,
+            -0.022726602852344513,
+            0.012590537779033184,
+            -0.0015103721525520086,
+            -0.06587684899568558,
+            -0.009780014865100384,
+            0.016166189685463905,
+            0.001200722181238234,
+            0.011939042247831821,
+            0.011083006858825684,
+            -0.007083124481141567,
+            -0.0018275976181030273,
+            -0.022559940814971924,
+            0.03124150261282921,
+            0.005177877843379974,
+            0.03584742918610573,
+            -0.006503595970571041,
+            -0.009128518402576447,
+            0.01073453202843666,
+            0.04527139291167259,
+            0.0017916138749569654,
+            0.010226971469819546,
+            -0.020560000091791153,
+            0.0015065843472257257,
+            -0.02284781076014042,
+            -0.0031135445460677147,
+            -0.008863375522196293,
+            -0.0021211495622992516,
+            0.015893470495939255,
+            0.0019753205124288797,
+            -0.010204244405031204,
+            0.025438643991947174,
+            0.02251448854804039,
+            -0.009605777449905872,
+            -0.015969226136803627,
+            -0.02437806874513626,
+            -0.014545026235282421,
+            -0.0002627763315103948,
+            -0.010007280856370926,
+            -0.022453883662819862,
+            -0.020878173410892487,
+            -0.002566212322562933,
+            -0.02559015527367592,
+            0.022529639303684235,
+            0.01220418605953455,
+            -4.882668508798815e-05,
+            -0.006723286584019661,
+            -0.038544319570064545,
+            -0.003727162955328822,
+            -0.03118089959025383,
+            -0.005374841392040253,
+            0.03287781774997711,
+            -0.03745344281196594,
+            -0.0002231231628684327,
+            0.014491996727883816,
+            -0.01993880607187748,
+            0.0028938541654497385,
+            0.008961857296526432,
+            0.02769615314900875,
+            -0.027408283203840256,
+            0.0001795638381736353,
+            0.006954340264201164,
+            0.016908591613173485,
+            -0.009878496639430523,
+            0.027196168899536133,
+            -0.04275631532073021,
+            0.011037553660571575,
+            -0.037120118737220764,
+            -0.020423639565706253,
+            -0.01523439958691597,
+            0.008363389410078526,
+            -0.005018791649490595,
+            -0.02637801133096218,
+            -0.010454237461090088,
+            0.006026337388902903,
+            -0.00693918950855732,
+            -0.01902974210679531,
+            -0.010620899498462677,
+            0.027438584715127945,
+            0.00931033119559288,
+            0.007977037690579891,
+            -0.02955973520874977,
+            -0.01631770096719265,
+            0.006280118133872747,
+            0.052089374512434006,
+            -0.01584801822900772,
+            0.014938953332602978,
+            -0.006639956030994654,
+            0.02342355251312256,
+            -0.028484009206295013,
+            -0.01872672140598297,
+            0.0315445251762867,
+            0.013946558348834515,
+            0.007064185570925474,
+            0.028953691944479942,
+            0.027196168899536133,
+            0.027105262503027916,
+            -0.04654408246278763,
+            0.009886072017252445,
+            -0.002613559365272522,
+            -0.008757317438721657,
+            0.03402930125594139,
+            0.01916610263288021,
+            0.022150862962007523,
+            0.0009744031121954322,
+            -0.01730252057313919,
+            0.0139844361692667,
+            0.014885924756526947,
+            0.007681591901928186,
+            -0.014242004603147507,
+            -0.02660527639091015,
+            0.003367325058206916,
+            -0.007992188446223736,
+            0.00047276070108637214,
+            0.020726662129163742,
+            -0.009052763693034649,
+            0.030999086797237396,
+            -0.014605630189180374,
+            -0.02593863010406494,
+            0.008916404098272324,
+            -0.02268115058541298,
+            -0.0041741193272173405,
+            0.0004476667381823063,
+            0.010083036497235298,
+            -0.023847781121730804,
+            -0.04669559374451637,
+            0.019408518448472023,
+            0.028393102809786797,
+            0.005579380784183741,
+            0.01328748743981123,
+            -0.035332292318344116,
+            0.008992158807814121,
+            0.012310243211686611,
+            0.011348150670528412,
+            0.0019847899675369263,
+            0.035362593829631805,
+            -0.00028834378463216126,
+            0.012885984033346176,
+            -0.008620957843959332,
+            0.028362801298499107,
+            0.00667025800794363,
+            -0.0323626808822155,
+            -0.0378473699092865,
+            -0.01513591781258583,
+            0.015257125720381737,
+            0.006692984607070684,
+            -0.011840560473501682,
+            -0.02278720773756504,
+            -0.038817036896944046,
+            0.015787413343787193,
+            -0.012916285544633865,
+            -0.038271598517894745,
+            -0.010158791206777096,
+            -0.009742137044668198,
+            0.016635872423648834,
+            -0.006916462909430265,
+            -0.016802534461021423,
+            -0.016090434044599533,
+            0.005821797996759415,
+            7.161247049225494e-05,
+            0.0244083721190691,
+            -0.001251857029274106,
+            -0.006855858489871025,
+            -0.016393456608057022,
+            -0.01188601367175579,
+            -0.0035074723418802023,
+            0.03602924197912216,
+            -0.014219277538359165,
+            -0.02383263036608696,
+            0.014984406530857086,
+            0.02657497487962246,
+            0.011636020615696907,
+            0.03227177634835243,
+            0.007181606721132994,
+            -0.014022313989698887,
+            0.003016956616193056,
+            -0.033029329031705856,
+            -0.0046400148421525955,
+            0.005196816753596067,
+            -0.006427840795367956,
+            -0.015651054680347443,
+            -0.01815097965300083,
+            -0.015832867473363876,
+            0.030756669119000435,
+            -0.02487805485725403,
+            0.0218023881316185,
+            0.030347591266036034,
+            0.003174148965626955,
+            -0.02915065549314022,
+            0.013598084449768066,
+            -0.0072005451656877995,
+            0.01862066239118576,
+            0.0019014590652659535,
+            0.028014326468110085,
+            -0.007461901288479567,
+            -0.004818039946258068,
+            -0.02501441352069378,
+            -0.03696860745549202,
+            -0.023484155535697937,
+            -0.008817922323942184,
+            0.0027537066489458084,
+            0.020984230563044548,
+            -0.0375746488571167,
+            -0.017757052555680275,
+            0.009855770505964756,
+            0.015757111832499504,
+            -0.03027183562517166,
+            -0.0025908327661454678,
+            -0.010158791206777096,
+            -0.0004055278259329498,
+            0.005140000022947788,
+            -0.02227207086980343,
+            -0.008977008052170277,
+            -0.015726808458566666,
+            -0.0010435298318043351,
+            0.006166485138237476,
+            -0.0007409819518215954,
+            -0.05805889517068863,
+            -0.013969285413622856,
+            0.018635815009474754,
+            0.0026400736533105373,
+            0.008908828720450401,
+            0.015938924625515938,
+            -0.0111739132553339,
+            0.0031533162109553814,
+            -0.026756787672638893,
+            0.023893235251307487,
+            -0.009492144919931889,
+            0.009355784393846989,
+            -0.013742019422352314,
+            0.012514783069491386,
+            0.014014738611876965,
+            0.004795313347131014,
+            0.0010681503918021917,
+            -0.0054051438346505165,
+            0.01601468026638031,
+            0.028377952054142952,
+            -0.023756874725222588,
+            0.011454207822680473,
+            -0.0002514130319468677,
+            -0.0032158142421394587,
+            -0.029196109622716904,
+            0.00509454682469368,
+            0.003931702114641666,
+            -0.005109698045998812,
+            0.016651025041937828,
+            -0.036544378846883774,
+            0.01662072166800499,
+            0.02177208475768566,
+            0.032544493675231934,
+            0.014832896180450916,
+            -0.017287369817495346,
+            -0.011658747680485249,
+            -0.012385998852550983,
+            -0.002624922664836049,
+            0.022181164473295212,
+            0.012120855040848255,
+            -0.01815097965300083,
+            -0.024287162348628044,
+            0.011249667964875698,
+            0.0005283934879116714,
+            -0.01203752402216196,
+            0.010795135982334614,
+            -0.013166278600692749,
+            0.01349202636629343,
+            0.012711746618151665,
+            0.00866641104221344,
+            0.00888610165566206,
+            0.022862963378429413,
+            -0.011878437362611294,
+            -0.017711598426103592,
+            0.00656041270121932,
+            -0.008916404098272324,
+            -0.0034089903347194195,
+            -0.012598113156855106,
+            0.010795135982334614,
+            0.014878349378705025,
+            0.06490717828273773,
+            -0.013143551535904408,
+            0.0009526234352961183,
+            -0.020438792183995247,
+            -0.031938452273607254,
+            -0.0017944546416401863,
+            -0.020908474922180176,
+            0.0111739132553339,
+            0.008370965719223022,
+            0.01598437689244747,
+            0.012022373266518116,
+            -0.0021912232041358948,
+            0.00795431062579155,
+            0.019317612051963806,
+            -0.010423935018479824,
+            7.255941454786807e-05,
+            -0.006503595970571041,
+            0.0013683309080079198,
+            0.0035851215943694115,
+            -0.011719351634383202,
+            -0.0026949963066726923,
+            0.035695917904376984,
+            0.01068150345236063,
+            0.011651171371340752,
+            -0.03445352986454964,
+            0.0020435003098100424,
+            0.020150920376181602,
+            0.007602048572152853,
+            -0.008302785456180573,
+            -0.01563590206205845,
+            0.01706010289490223,
+            -0.015317730605602264,
+            -0.0022499337792396545,
+            0.002888172399252653,
+            -0.010151215828955173,
+            0.04581683129072189,
+            -0.019787294790148735,
+            -0.0011742077767848969,
+            0.031847547739744186,
+            0.0030207443051040173,
+            -0.0040983641520142555,
+            0.005408931523561478,
+            -0.003473382443189621,
+            -0.0069013116881251335,
+            -0.02139330841600895,
+            0.0031381649896502495,
+            0.00667025800794363,
+            -0.019287310540676117,
+            -0.0072005451656877995,
+            -0.007719469256699085,
+            0.006897523999214172,
+            0.023105379194021225,
+            -0.009492144919931889,
+            -0.00016145357221830636,
+            -0.008575504645705223,
+            0.009848194196820259,
+            -0.0011912527261301875,
+            -0.044301725924015045,
+            -0.006147546228021383,
+            0.021181194111704826,
+            0.000973456131760031,
+            0.0005757405888289213,
+            -0.007450537756085396,
+            0.005007428117096424,
+            -0.017423728480935097,
+            -0.008454295806586742,
+            0.02369627170264721,
+            0.01736312545835972,
+            -0.009113367646932602,
+            0.010863316245377064,
+            -0.0035680767614394426,
+            0.00433320552110672,
+            -0.020014561712741852,
+            -0.01095422264188528,
+            0.003462019143626094,
+            0.015272277407348156,
+            -0.0009819786064326763,
+            -0.029029447585344315,
+            -0.0012272365856915712,
+            -0.00024194362049456686,
+            0.02918095886707306,
+            0.01486319862306118,
+            -0.008961857296526432,
+            -0.021211495622992516,
+            -0.013242034241557121,
+            -0.001237652963027358,
+            -0.015726808458566666,
+            -0.009620928205549717,
+            0.011855711229145527,
+            0.0009137988090515137,
+            0.008992158807814121,
+            -0.018135828897356987,
+            -0.027347678318619728,
+            -0.016166189685463905,
+            0.0018465365283191204,
+            -0.012870832346379757,
+            0.003340810537338257,
+            0.015514694154262543,
+            -0.03430201858282089,
+            -0.015060162171721458,
+            -0.03805948421359062,
+            0.008280059322714806,
+            -0.0029658216517418623,
+            -0.02024182677268982,
+            -0.023605365306138992,
+            0.009484568610787392,
+            0.019181253388524055,
+            -0.0015548784285783768,
+            0.019847899675369263,
+            -0.000889651826582849,
+            0.020060013979673386,
+            -0.0073141781613230705,
+            0.0031703610438853502,
+            -0.013007191941142082,
+            0.0030245319940149784,
+            0.008333087898790836,
+            0.02890823967754841,
+            -0.023075077682733536,
+            -0.006503595970571041,
+            -0.004685468040406704,
+            0.004746072459965944,
+            0.012658718042075634,
+            0.014613205567002296,
+            0.020741812884807587,
+            -0.008272483013570309,
+            -0.0027612822595983744,
+            0.012484480626881123,
+            -0.003799130441620946,
+            0.007295239716768265,
+            0.002390081062912941,
+            0.0019677451346069574,
+            0.004162756260484457,
+            0.008477022871375084,
+            0.0053710537031292915,
+            0.027650700882077217,
+            -0.0028275682125240564,
+            0.0031703610438853502,
+            0.024620486423373222,
+            0.02322658896446228,
+            -0.018878230825066566,
+            -0.028074929490685463,
+            0.002024561632424593,
+            0.050695475190877914,
+            0.007245998363941908,
+            -0.006810405291616917,
+            -0.03927157074213028,
+            -0.0031760428100824356,
+            0.028635520488023758,
+            0.023650817573070526,
+            0.01584801822900772,
+            -0.010567869991064072,
+            -0.012098127976059914,
+            0.02772645466029644,
+            0.013295062817633152,
+            0.02616589516401291,
+            -0.025302285328507423,
+            0.010704229585826397,
+            -0.03869583085179329,
+            0.0449683703482151,
+            0.004439263138920069,
+            -0.014476845972239971,
+            0.0011884119594469666,
+            0.010386057198047638,
+            -0.004537744913250208,
+            0.006431628484278917,
+            -5.8059369621332735e-05,
+            -0.0008086882880888879,
+            0.011923890560865402,
+            0.012007221579551697,
+            0.004265025723725557,
+            -0.02399929240345955,
+            0.007359631359577179,
+            0.004272601567208767,
+            0.03005971945822239,
+            0.005431658122688532,
+            -0.0013863227795809507,
+            0.012385998852550983,
+            -0.005810434930026531,
+            0.008855799213051796,
+            -0.010719381272792816,
+            0.020469093695282936,
+            -0.02918095886707306,
+            1.0986003871948924e-05,
+            0.02051454596221447,
+            -0.0015785519499331713,
+            0.0038199631962925196,
+            -0.023741723969578743,
+            0.01186328660696745,
+            -0.052968136966228485,
+            0.021166043356060982,
+            -0.018741872161626816,
+            0.021741783246397972,
+            0.019590331241488457,
+            -0.009734561666846275,
+            -0.012052674777805805,
+            0.03545350208878517,
+            0.0026760573964565992,
+            -0.014264730736613274,
+            -0.010643625631928444,
+            -0.017999470233917236,
+            -0.02830219641327858,
+            0.041968461126089096,
+            -0.0006117243901826441,
+            0.0028275682125240564,
+            0.0024222771171480417,
+            -0.010666352696716785,
+            -0.008923979476094246,
+            0.001954487757757306,
+            0.020499395206570625,
+            -0.03369597718119621,
+            -0.011128460057079792,
+            -0.008689138107001781,
+            -0.02566591091454029,
+            -0.00020300064352340996,
+            -0.011961768381297588,
+            0.026211349293589592,
+            0.01240872498601675,
+            -0.03696860745549202,
+            0.003780191531404853,
+            0.023650817573070526,
+            0.001603172393515706,
+            0.018120678141713142,
+            0.002130619017407298,
+            0.0010795136913657188,
+            0.009742137044668198,
+            -0.010476963594555855,
+            0.026559824123978615,
+            -0.009204274043440819,
+            0.008658835664391518,
+            -0.001950700068846345,
+            -0.017287369817495346,
+            0.017969166859984398,
+            0.004083212930709124,
+            0.024120500311255455,
+            0.011954193003475666,
+            0.011605718173086643,
+            0.01112088467925787,
+            -0.019878201186656952,
+            0.020090317353606224,
+            -0.0018891488434746861,
+            0.009075489826500416,
+            0.024726543575525284,
+            -0.005863463506102562,
+            0.020832719281315804,
+            0.015257125720381737,
+            -0.00020229043730068952,
+            0.011295121163129807,
+            -0.020120618864893913,
+            -0.011779955588281155,
+            0.0085679292678833,
+            -0.009605777449905872,
+            -0.010325453244149685,
+            0.009901223704218864,
+            -0.010105762630701065,
+            -0.015052586793899536,
+            -0.0071247899904847145,
+            0.016635872423648834,
+            0.0028976418543606997,
+            -0.02610529027879238,
+            -1.7326075976598077e-05,
+            0.008901252411305904,
+            0.006166485138237476,
+            0.018741872161626816,
+            0.011317848227918148,
+            -0.006692984607070684,
+            0.010363331064581871,
+            0.011476934887468815,
+            0.0053104497492313385,
+            -0.005079395603388548,
+            -0.013431422412395477,
+            -0.0024241709616035223,
+            0.018090376630425453,
+            0.009810317307710648,
+            -0.01922670565545559,
+            0.004564259201288223,
+            0.0008877579239197075,
+            -0.01420412678271532,
+            -0.019347915425896645,
+            -0.0038483713287860155,
+            -0.004170331638306379,
+            0.0027915844693779945,
+            0.0509984977543354,
+            -0.0006216673064045608,
+            0.007893706671893597,
+            -0.025741664692759514,
+            -0.006299056578427553,
+            -0.00027556007262319326,
+            -0.0036097420379519463,
+            0.01272689737379551,
+            -0.01824188604950905,
+            -0.03724132478237152,
+            0.012711746618151665,
+            -0.0012584856012836099,
+            0.014923802576959133,
+            -0.00297907879576087,
+            0.014226853847503662,
+            -0.017347972840070724,
+            0.01916610263288021,
+            -0.0011117096291854978,
+            -0.016029831022024155,
+            0.0029374135192483664,
+            -0.011143610812723637,
+            -0.021317552775144577,
+            0.005916492082178593,
+            -0.006848283112049103,
+            -0.03896854817867279,
+            0.0029885482508689165,
+            -0.010423935018479824,
+            0.0011798894265666604,
+            -0.003912763670086861,
+            0.020166072994470596,
+            -0.0061513339169323444,
+            0.0008697660523466766,
+            0.02177208475768566,
+            0.0017026013229042292,
+            0.01601468026638031,
+            -0.014499573037028313,
+            0.010999675840139389,
+            0.012598113156855106,
+            0.02210540883243084,
+            -0.032180868089199066,
+            -0.01393140759319067,
+            0.014211702160537243,
+            0.0026116655208170414,
+            -0.005382417235523462,
+            0.020635755732655525,
+            -0.01294658798724413,
+            0.019650936126708984,
+            -0.0005577487172558904,
+            -0.011704200878739357,
+            -0.03524138405919075,
+            -0.016529815271496773,
+            -3.343888238305226e-05,
+            -0.008439145050942898,
+            0.025317436084151268,
+            0.01760554127395153,
+            -0.0108330138027668,
+            0.026468917727470398,
+            0.0037574649322777987,
+            0.014598054811358452,
+            -0.014257155358791351,
+            -0.005999823100864887,
+            -0.0003773563075810671,
+            -0.0119314668700099,
+            0.015264702029526234,
+            -0.03512017801403999,
+            0.0008342557121068239,
+            -0.00839369185268879,
+            0.005242269951850176,
+            -0.0254992488771677,
+            0.018257036805152893,
+            0.025817420333623886,
+            -0.0031419529113918543,
+            -0.005037730094045401,
+            -0.022802358493208885,
+            0.028968842700123787,
+            0.03049910068511963,
+            0.0024828813038766384,
+            -0.0013825349742546678,
+            -0.00352072948589921,
+            0.0033218718599528074,
+            0.0211205892264843,
+            -0.037998881191015244,
+            -0.009098216891288757,
+            -0.028696123510599136,
+            0.042332086712121964,
+            -0.008098246529698372,
+            -0.013431422412395477,
+            0.01862066239118576,
+            -0.0024715180043131113,
+            0.02355991117656231,
+            -0.004689255729317665,
+            -0.004855917301028967,
+            0.03148392215371132,
+            0.011514811776578426,
+            0.02772645466029644,
+            -0.04015033319592476,
+            0.014704111963510513,
+            -0.006867222022265196,
+            0.004867280833423138,
+            -0.013484450988471508,
+            0.005302873905748129,
+            0.02580226957798004,
+            -0.018060073256492615,
+            0.0068066176027059555,
+            0.030665762722492218,
+            0.00748084019869566,
+            0.002448791405186057,
+            0.0165904201567173,
+            0.003890036838129163,
+            0.010855740867555141,
+            0.000841831264551729,
+            0.0052574207074940205,
+            0.022423582151532173,
+            0.014454119838774204,
+            -0.016029831022024155,
+            -0.015120766125619411,
+            -0.00044790346873924136,
+            -0.004450626205652952,
+            0.005261208396404982,
+            0.013242034241557121,
+            0.0061172437854111195,
+            -0.015681356191635132,
+            0.01052999310195446,
+            0.014158673584461212,
+            0.013128400780260563,
+            -0.0039847311563789845,
+            -0.03412020578980446,
+            -0.02443867363035679,
+            -0.021438762545585632,
+            0.003104075090959668,
+            -0.0008281006012111902,
+            -0.0021211495622992516,
+            0.0017954016802832484,
+            0.017347972840070724,
+            -0.0008352026343345642,
+            0.00379155483096838,
+            0.01686313934624195,
+            -0.01415109820663929,
+            -0.022090258076786995,
+            -0.0035964848939329386,
+            -0.027529491111636162,
+            0.0053710537031292915,
+            0.001844642567448318,
+            -0.00965123064815998,
+            0.008530051447451115,
+            -0.0046059247106313705,
+            -0.007223271764814854,
+            -0.012196609750390053,
+            -0.02007516659796238,
+            0.005780132487416267,
+            0.016878290101885796,
+            0.005730891600251198,
+            -0.009272454306483269,
+            -0.003939277958124876,
+            0.011575416661798954,
+            0.017484333366155624,
+            0.017317671328783035,
+            -0.009295180439949036,
+            0.009628504514694214,
+            0.0004022135108243674,
+            0.004670316819101572,
+            -0.01834794320166111,
+            0.0045869858004152775,
+            -0.0027953721582889557,
+            -0.008658835664391518,
+            0.01105270441621542,
+            -0.00031201730598695576,
+            0.01628739945590496,
+            -0.013090522959828377,
+            0.005624833982437849,
+            0.010333028621971607,
+            -0.020590301603078842,
+            -0.0067308624275028706,
+            -0.01692374423146248,
+            0.026484068483114243,
+            -0.00504530593752861,
+            0.00045405857963487506,
+            -0.02978700026869774,
+            0.0033616432920098305,
+            0.000848933297675103,
+            -0.000845619011670351,
+            -0.012764775194227695,
+            0.010749682784080505,
+            0.03242328763008118,
+            0.0005179771687835455,
+            0.0216660276055336,
+            -0.0036154238041490316,
+            0.0007480839849449694,
+            -0.00785582885146141,
+            0.004977126140147448,
+            -0.004916521720588207,
+            -0.009901223704218864,
+            -0.010863316245377064,
+            0.021726632490754128,
+            -0.0033029329497367144,
+            0.0003356908564455807,
+            0.011332998983561993,
+            -0.012067826464772224,
+            0.0014620780711993575,
+            -0.0047687990590929985,
+            0.00971941091120243,
+            0.0061702728271484375,
+            0.009113367646932602,
+            -0.01176480483263731,
+            0.05451354384422302,
+            -0.0106284748762846,
+            0.005537715740501881,
+            -0.00439380994066596,
+            -0.022226616740226746,
+            -0.017317671328783035,
+            0.012711746618151665,
+            -0.017544938251376152,
+            0.03375658020377159,
+            0.024150803685188293,
+            0.0006467612111009657,
+            0.010393633507192135,
+            -0.006958128418773413,
+            0.023923536762595177,
+            -0.002602196065708995,
+            0.0010795136913657188,
+            -0.004855917301028967,
+            0.015802564099431038,
+            0.04511988162994385,
+            0.017393426969647408,
+            -0.001152428099885583,
+            0.00829521007835865,
+            0.020575150847434998,
+            0.00814369972795248,
+            -0.0020075165666639805,
+            0.014219277538359165,
+            -0.01024212222546339,
+            0.00900731049478054,
+            -0.004458202049136162,
+            0.027438584715127945,
+            0.00814369972795248,
+            -0.006310420110821724,
+            -0.0036836036015301943,
+            -0.0008209985098801553,
+            0.014173824340105057,
+            0.006950552575290203,
+            0.01963578537106514,
+            -0.00461350055411458,
+            -0.0012575386790558696,
+            0.013870803639292717,
+            0.014491996727883816,
+            -0.00552635220810771,
+            0.001724380999803543,
+            -0.01435563713312149,
+            -0.010204244405031204,
+            0.013575357384979725,
+            0.006086941808462143,
+            0.006310420110821724,
+            -0.014514723792672157,
+            -0.00980274099856615,
+            -0.004670316819101572,
+            0.0015435151290148497,
+            0.015052586793899536,
+            0.011060279794037342,
+            -0.004814252257347107,
+            0.006052851676940918,
+            0.013825350441038609,
+            -0.014014738611876965,
+            0.006147546228021383,
+            0.007208121009171009,
+            0.008855799213051796,
+            0.004787737503647804,
+            -0.010817863047122955,
+            -0.00428396463394165,
+            -0.0063937511295080185,
+            0.0012670081341639161,
+            0.016832837834954262,
+            0.013181429356336594,
+            0.0049581872299313545,
+            0.027529491111636162,
+            -0.00839369185268879,
+            -0.012302667833864689,
+            0.02339324913918972,
+            0.011257244274020195,
+            0.0031703610438853502,
+            0.014651083387434483,
+            -0.012635990977287292,
+            -0.012446602806448936,
+            0.006655106786638498,
+            0.004848341923207045,
+            0.006049063988029957,
+            0.009151245467364788,
+            0.00450365524739027,
+            0.007333117071539164,
+            -0.007098275702446699,
+            -0.007427811156958342,
+            0.0014810169814154506,
+            0.01642375811934471,
+            0.020393338054418564,
+            -0.024968961253762245,
+            -0.0018806264270097017,
+            0.0009052763343788683,
+            0.010878467001020908,
+            -0.003316190093755722,
+            0.012310243211686611,
+            5.65205882594455e-05,
+            -0.0074240234680473804,
+            0.010158791206777096,
+            -0.02119634486734867,
+            -0.012681444175541401,
+            0.0012717428617179394,
+            0.013037494383752346,
+            -0.019590331241488457,
+            0.018226735293865204,
+            -0.008893677033483982,
+            -0.013348091393709183,
+            0.013893529772758484,
+            0.020014561712741852,
+            0.004090788308531046,
+            -0.007738408166915178,
+            -0.021862991154193878,
+            -0.01127239502966404,
+            0.016060132533311844,
+            0.020272130146622658,
+            0.000947888707742095,
+            -0.0006472347304224968,
+            0.010173942893743515,
+            -0.013113250024616718,
+            -0.023605365306138992,
+            -0.005230906419456005,
+            0.010386057198047638,
+            0.01746918261051178,
+            0.018832778558135033,
+            -0.006685409229248762,
+            0.02443867363035679,
+            -0.0059127043932676315,
+            -0.015499543398618698,
+            2.86597842205083e-05,
+            -0.02802947722375393,
+            -0.017423728480935097,
+            0.00013778003631159663,
+            0.00246773031540215,
+            0.008598231710493565,
+            0.0008555619278922677,
+            0.01943882182240486,
+            0.000904329412151128,
+            0.017590390518307686,
+            -0.000659071491099894,
+            -0.002009410411119461,
+            -0.0013995799235999584,
+            -0.0015340456739068031,
+            0.02254479005932808,
+            -0.012166308239102364,
+            -0.01176480483263731,
+            0.022302372381091118,
+            0.006852070800960064,
+            0.014719262719154358,
+            -0.027635548263788223,
+            0.013757170177996159,
+            -0.015499543398618698,
+            0.017423728480935097,
+            -0.03408990427851677,
+            0.006420265417546034,
+            0.01489350013434887,
+            -0.010348179377615452,
+            -0.020499395206570625,
+            -0.029650641605257988,
+            -0.024150803685188293,
+            -0.0038710979279130697,
+            -0.010855740867555141,
+            -0.016847988590598106,
+            0.006121031939983368,
+            -0.005802859086543322,
+            0.018499454483389854,
+            -0.007079336792230606,
+            0.033877789974212646,
+            0.006969491485506296,
+            -0.004056698642671108,
+            0.003835114184767008,
+            0.022469034418463707,
+            -0.006102093029767275,
+            0.02109028771519661,
+            0.004408961161971092,
+            -0.001467759720981121,
+            0.013681414537131786,
+            -0.006552837323397398,
+            -0.002361672930419445,
+            0.005821797996759415,
+            -0.010484539903700352,
+            0.022014502435922623,
+            0.005924067925661802,
+            -0.007708106189966202,
+            -0.0006775368237867951,
+            -0.011454207822680473,
+            0.014726839028298855,
+            0.006530110724270344,
+            0.0022480397019535303,
+            -0.011295121163129807,
+            0.011249667964875698,
+            -0.004859705455601215,
+            0.0040642740204930305,
+            0.009484568610787392,
+            -0.01774190180003643,
+            -0.028862785547971725,
+            0.011416330002248287,
+            0.019120648503303528,
+            0.006382387597113848,
+            -0.0055755930952727795,
+            -0.0025681061670184135,
+            0.00839369185268879,
+            0.010651201009750366,
+            0.022150862962007523,
+            -0.012984465807676315,
+            -8.889415767043829e-05,
+            0.013204156421124935,
+            -0.007170243188738823,
+            -0.01056029461324215,
+            0.00487485621124506,
+            0.023438703268766403,
+            0.0068823727779090405,
+            -0.04327145218849182,
+            -0.018469152972102165,
+            0.01909034699201584,
+            0.007136153522878885,
+            -0.04154422879219055,
+            0.0026381798088550568,
+            -0.003689285134896636,
+            -0.023787178099155426,
+            0.016105586662888527,
+            -0.0011817833874374628,
+            -0.01828734017908573,
+            0.028756728395819664,
+            0.02830219641327858,
+            0.01834794320166111,
+            2.6987840101355687e-05,
+            -0.013332940638065338,
+            -0.0318172425031662,
+            -0.023590214550495148,
+            0.008280059322714806,
+            -0.02660527639091015,
+            -0.0072725131176412106,
+            0.011242092587053776,
+            -0.017454031854867935,
+            0.02972639724612236,
+            -0.008507325313985348,
+            0.026938600465655327,
+            0.011446632444858551,
+            0.001691238023340702,
+            -0.013446573168039322,
+            -0.018802475184202194,
+            -0.01350717805325985,
+            -0.008196728304028511,
+            0.014090493321418762,
+            0.017923714593052864,
+            0.011007251217961311,
+            0.011332998983561993,
+            0.02251448854804039,
+            -0.012158732861280441,
+            0.007969462312757969,
+            -2.888172457460314e-05,
+            -0.008961857296526432,
+            0.0026343921199440956,
+            -0.018393397331237793,
+            0.00868156272917986,
+            0.0062308767810463905,
+            0.005458172410726547,
+            -0.007208121009171009,
+            0.010742107406258583,
+            0.02712041325867176,
+            -0.018847929313778877,
+            0.011401179246604443,
+            -0.031362712383270264,
+            -0.014787442982196808,
+            -0.01056029461324215,
+            0.026862844824790955,
+            0.007757347077131271,
+            -0.015726808458566666,
+            -0.007514929864555597,
+            -0.01545409020036459,
+            0.012885984033346176,
+            0.0032442226074635983,
+            0.005764981731772423,
+            0.015817714855074883,
+            0.021893294528126717,
+            0.01746918261051178,
+            -0.00795431062579155,
+            0.006852070800960064,
+            0.004674104508012533,
+            0.028090080246329308,
+            -0.01777220331132412,
+            0.020802417770028114,
+            0.009954252280294895,
+            -0.003935490269213915,
+            0.005685438401997089,
+            -0.014257155358791351,
+            0.008530051447451115,
+            -0.007325541693717241,
+            0.030514251440763474,
+            0.0033067206386476755,
+            0.014075342565774918,
+            -0.02062060497701168,
+            0.006575563922524452,
+            -0.03172633796930313,
+            0.009022461250424385,
+            0.013704141601920128,
+            0.012976890429854393,
+            -0.007677804213017225,
+            -0.024181105196475983,
+            -0.0119314668700099,
+            0.0057990713976323605,
+            -0.01279507763683796,
+            -0.01046181283891201,
+            -0.0034828518982976675,
+            0.012446602806448936,
+            0.018847929313778877,
+            -0.01848430372774601,
+            -0.019923655316233635,
+            -0.009583051316440105,
+            0.02898399345576763,
+            -0.015787413343787193,
+            -0.001471547526307404,
+            0.02413565292954445,
+            0.021241798996925354,
+            0.015022284351289272,
+            0.002155239460989833,
+            0.020090317353606224,
+            -0.0007111532613635063,
+            -0.0237265732139349,
+            -0.02813553437590599,
+            0.007302815094590187,
+            -0.006912675220519304,
+            0.01943882182240486,
+            -0.0035548193845897913,
+            -0.005658924113959074,
+            -0.0034033088013529778,
+            -0.007503566797822714,
+            -0.0165904201567173,
+            -0.01787826046347618,
+            0.0021647089160978794,
+            -0.008454295806586742,
+            0.005863463506102562,
+            -0.0002795845502987504,
+            -0.013105674646794796,
+            0.013401119969785213,
+            -0.014075342565774918,
+            0.005772557109594345,
+            -0.002653330797329545,
+            0.001962063368409872,
+            0.0035926972050219774,
+            0.009113367646932602,
+            -0.007336904760450125,
+            -0.06381630152463913,
+            -0.013105674646794796,
+            -0.003704436356201768,
+            0.006071790587157011,
+            -0.00019578021601773798,
+            -0.016635872423648834,
+            -0.013196581043303013,
+            0.01706010289490223,
+            0.008863375522196293,
+            -0.013870803639292717,
+            0.016954045742750168,
+            0.009961827658116817,
+            0.003622999181970954,
+            -0.024044746533036232,
+            -0.0008148433989845216,
+            0.008757317438721657,
+            0.032756611704826355,
+            -0.008643684908747673,
+            -0.0003839849086944014,
+            0.0036684523802250624,
+            -0.0033332351595163345,
+            -0.010052734054625034,
+            -0.014582903124392033,
+            -0.011848135851323605,
+            0.011908739805221558,
+            0.01790856383740902,
+            -0.014264730736613274,
+            -0.02007516659796238,
+            -0.0007106798002496362,
+            0.012931437231600285,
+            -0.026590125635266304,
+            0.0007244104635901749,
+            -0.007787649054080248,
+            -0.01052999310195446,
+            0.018302490934729576,
+            -0.019014591351151466,
+            0.005249845329672098,
+            0.00048294031876139343,
+            0.008507325313985348,
+            -0.023544760420918465,
+            -0.024893205612897873,
+            0.0035131541080772877,
+            -0.023847781121730804,
+            0.03612014651298523,
+            -0.021590271964669228,
+            -0.012082977220416069,
+            0.034483831375837326,
+            -0.0037423139438033104,
+            0.01702980138361454,
+            0.0023995505180209875,
+            0.02739313244819641,
+            -0.011439057067036629,
+            -0.00617784820497036,
+            0.012499631382524967,
+            0.009575475007295609,
+            0.0032593735959380865,
+            0.03775646165013313,
+            -0.01257538702338934,
+            -0.013901105150580406,
+            -0.005556654185056686,
+            -0.012954163365066051,
+            0.0006775368237867951,
+            0.014431392773985863,
+            0.028393102809786797,
+            0.004257450345903635,
+            -0.00628769351169467,
+            -0.006143758539110422,
+            -0.0014649189542979002,
+            -0.004242299124598503,
+            -0.00947699323296547,
+            -0.00715130427852273,
+            0.017287369817495346,
+            -0.025241680443286896,
+            0.016711628064513206,
+            -0.006420265417546034,
+            0.021135739982128143,
+            -0.011757229454815388,
+            0.007007369305938482,
+            -0.008961857296526432,
+            0.003492321353405714,
+            0.0023635667748749256,
+            -0.005109698045998812,
+            0.009287605062127113,
+            -0.014772292226552963,
+            0.008454295806586742,
+            0.013742019422352314,
+            0.0011183382011950016,
+            -0.010590597055852413,
+            -0.019272159785032272,
+            0.0022972808219492435,
+            -0.000710206339135766,
+            0.01181783340871334,
+            -0.0047498601488769054,
+            -0.000981031684204936,
+            0.0002921315317507833,
+            -0.014476845972239971,
+            -0.009575475007295609,
+            0.0006050958181731403,
+            0.005302873905748129,
+            -0.008961857296526432,
+            0.010378481820225716,
+            -0.005693014245480299,
+            -0.00034160923678427935,
+            0.02627195231616497,
+            0.006526322569698095,
+            -0.007632351014763117,
+            -0.014098069630563259,
+            0.01563590206205845,
+            0.015370759181678295,
+            -0.007454325910657644,
+            0.01933276280760765,
+            -0.016196493059396744,
+            -0.0008366230758838356,
+            0.013916256837546825,
+            -0.009219424799084663,
+            0.0017698341980576515,
+            -0.008196728304028511,
+            0.03554440662264824,
+            -0.0008787619881331921,
+            -0.0035415622405707836,
+            -0.0006401326390914619,
+            0.012976890429854393,
+            -0.0026779514737427235,
+            0.001182730309665203,
+            0.01132542360574007,
+            0.005321812815964222,
+            0.0185146052390337,
+            0.00328020635060966,
+            -0.03378688171505928,
+            -0.007780073676258326,
+            -0.0065490491688251495,
+            -0.008090670220553875,
+            0.009022461250424385,
+            0.025196226313710213,
+            0.01990850456058979,
+            0.006408901885151863,
+            0.001881573349237442,
+            0.018423698842525482,
+            0.015757111832499504,
+            -0.01489350013434887,
+            0.016363153234124184,
+            0.02802947722375393,
+            -0.010030006989836693,
+            0.022741753607988358,
+            -0.005253633018583059,
+            -0.008893677033483982,
+            0.008946705609560013,
+            -0.0015378334792330861,
+            -0.017393426969647408,
+            -0.015681356191635132,
+            0.005022579338401556,
+            -0.008999735116958618,
+            -0.0022897052112966776,
+            -0.018605511635541916,
+            0.004670316819101572,
+            0.017105557024478912,
+            -0.004965762607753277,
+            -3.5243989259470254e-05,
+            0.019484274089336395,
+            -0.006965703796595335,
+            -0.016908591613173485,
+            0.006325571332126856,
+            -0.015007133595645428,
+            0.007946735247969627,
+            -0.013666263781487942,
+            0.010492115281522274,
+            -0.006098304875195026,
+            0.018817627802491188,
+            0.005446809343993664,
+            -0.008090670220553875,
+            -0.0008958069374784827,
+            -0.018984289839863777,
+            0.02065090648829937,
+            -0.033089932054281235,
+            0.00839369185268879,
+            -0.026226500049233437,
+            -0.0009781909175217152,
+            0.0008849170990288258,
+            0.014605630189180374,
+            -0.001218714052811265,
+            0.019696388393640518,
+            -0.002751812804490328,
+            -0.020090317353606224,
+            -0.007086912170052528,
+            -0.0010634156642481685,
+            -0.01618134044110775,
+            -0.030923331156373024,
+            -0.004272601567208767,
+            0.0036267871037125587,
+            -0.004711982328444719,
+            0.0021135739516466856,
+            0.017984317615628242,
+            -0.005613470915704966,
+            0.009173971600830555,
+            -0.02690829709172249,
+            -0.007416448090225458,
+            -0.00355103169567883,
+            0.0010776197304949164,
+            -0.011242092587053776,
+            0.0010596278589218855,
+            0.011226941831409931,
+            -0.019044892862439156,
+            0.013461724855005741,
+            -0.021378157660365105,
+            -0.004371083341538906,
+            -0.008507325313985348,
+            -0.00135412672534585,
+            -0.013446573168039322,
+            -0.028120383620262146,
+            0.025029566138982773,
+            -0.00021495578403118998,
+            -0.003528305096551776,
+            -0.011999646201729774,
+            0.023181134834885597,
+            0.002914686920121312,
+            0.027817361056804657,
+            -0.03157482668757439,
+            0.0013427634257823229,
+            0.011552689597010612,
+            0.0025984083767980337,
+            -0.017287369817495346,
+            0.004746072459965944,
+            -0.005234694108366966,
+            0.02210540883243084,
+            -0.0011694730492308736,
+            -0.0005340751959010959,
+            -0.008234606124460697,
+            0.030999086797237396,
+            0.009136094711720943,
+            0.0012944694608449936,
+            -0.013226882554590702,
+            -0.008060368709266186,
+            -0.002759388415142894,
+            -0.0011069749016314745,
+            0.012855681590735912,
+            0.01290113478899002,
+            -0.012105703353881836,
+            0.0013607552973553538,
+            -0.006878585088998079,
+            0.0011694730492308736,
+            -0.002935519441962242,
+            -0.0004966709529981017,
+            -0.01149208564311266,
+            0.003257479751482606,
+            0.016272246837615967,
+            -0.00780280027538538,
+            -0.006480869371443987,
+            0.0007750718505121768,
+            0.009181547909975052,
+            0.012461753562092781,
+            -0.012742049060761929,
+            -0.018226735293865204,
+            -0.011908739805221558,
+            0.016029831022024155,
+            0.010522416792809963,
+            -0.008204303681850433,
+            -0.013234458863735199,
+            -0.02583257108926773,
+            0.01169662456959486,
+            -0.03293842449784279,
+            -0.012242063879966736,
+            0.005821797996759415,
+            0.028559764847159386,
+            0.019999410957098007,
+            0.012226912193000317,
+            0.001332347048446536,
+            0.002950670663267374,
+            0.00023342114582192153,
+            -0.01916610263288021,
+            0.014408666640520096,
+            -0.011143610812723637,
+            -0.018166130408644676,
+            0.0075906855054199696,
+            0.011939042247831821,
+            0.03148392215371132,
+            0.006942977197468281,
+            -0.0010160685051232576,
+            0.01686313934624195,
+            -0.001218714052811265,
+            -0.013582932762801647,
+            0.01420412678271532,
+            -0.024256860837340355,
+            0.016438908874988556,
+            0.02860521711409092,
+            -0.004780162125825882,
+            0.0008531945641152561,
+            -0.020938776433467865,
+            0.00482561532407999,
+            -0.011529963463544846,
+            -0.002177966060116887,
+            -0.003287781961262226,
+            -0.03460504114627838,
+            0.007946735247969627,
+            -0.006825556512922049,
+            0.01046181283891201,
+            -0.00814369972795248,
+            0.006655106786638498,
+            -0.012431452050805092,
+            0.004253662656992674,
+            0.01513591781258583,
+            0.016105586662888527,
+            0.007060397882014513,
+            -0.025211378931999207,
+            -0.009196698665618896,
+            0.008105821907520294,
+            -0.01349202636629343,
+            -0.02295386977493763,
+            -0.006852070800960064,
+            0.009840618818998337,
+            -0.01061332318931818,
+            -0.00488243205472827,
+            -0.028180986642837524,
+            -0.013681414537131786,
+            -0.0014327229000627995,
+            -0.017817657440900803,
+            -0.005556654185056686,
+            -0.003100287402048707,
+            0.01571165770292282,
+            -0.0049733384512364864,
+            -0.006185423582792282,
+            0.004742284305393696,
+            -0.023878084495663643,
+            0.0037423139438033104,
+            -0.008280059322714806,
+            0.0008266801596619189,
+            -0.0032517979852855206,
+            -0.0014762822538614273,
+            -0.007969462312757969,
+            -0.014332910999655724,
+            0.006893736310303211,
+            -0.016469212248921394,
+            0.002954458352178335,
+            -0.015173795633018017,
+            -0.016499513760209084,
+            -0.00682934420183301,
+            0.00775355938822031,
+            0.021575121209025383,
+            0.006541473791003227,
+            0.021484214812517166,
+            0.005469535943120718,
+            0.028680972754955292,
+            0.0003953482082579285,
+            0.003041577059775591,
+            -8.958956641436089e-06,
+            0.003232859307900071,
+            0.019347915425896645,
+            0.006571775767952204,
+            0.018984289839863777,
+            -0.008742166683077812,
+            0.010514841414988041,
+            -0.010355755686759949,
+            0.010014856234192848,
+            0.006333146709948778,
+            -0.006473293993622065,
+            -0.003799130441620946,
+            -0.010348179377615452,
+            -0.0031419529113918543,
+            -0.016544966027140617,
+            0.0027366618160158396,
+            0.00693161366507411,
+            0.0071588801220059395,
+            -0.011582992039620876,
+            -0.005230906419456005,
+            -0.014704111963510513,
+            -0.0064354161731898785,
+            -0.01831764169037342,
+            0.014385939575731754,
+            0.029226411134004593,
+            -0.016332851722836494,
+            -0.01977214403450489,
+            -0.007772498298436403,
+            0.0007296186522580683,
+            -0.006636167876422405,
+            -0.01240872498601675,
+            -0.009151245467364788,
+            -0.035059571266174316,
+            -0.017666146159172058,
+            0.007586897816509008,
+            0.02484775334596634,
+            0.01387837901711464,
+            -0.013628385961055756,
+            -0.0006983695784583688,
+            -0.016984347254037857,
+            -0.010787560604512691,
+            -0.01535560842603445,
+            -0.02054484933614731,
+            -0.02498411200940609,
+            0.0077952248975634575,
+            -0.004033972043544054,
+            0.014931377954781055,
+            0.008113397285342216,
+            0.0032631612848490477,
+            -0.004371083341538906,
+            -0.0024109138175845146,
+            -0.013196581043303013,
+            0.0012925754999741912,
+            -0.004787737503647804,
+            -0.0017291157273575664,
+            0.03554440662264824,
+            -0.008098246529698372,
+            -0.005871038883924484,
+            0.011060279794037342,
+            0.011128460057079792,
+            -0.013764745555818081,
+            0.0006642796797677875,
+            -0.011249667964875698,
+            0.021847840398550034,
+            -0.021620575338602066,
+            0.019257009029388428,
+            0.0014507147716358304,
+            -0.010840589180588722,
+            0.028999146074056625,
+            0.01892368495464325,
+            0.00743159931153059,
+            0.006579351611435413,
+            0.01672677882015705,
+            0.0044316877610981464,
+            0.016196493059396744,
+            -0.0026552248746156693,
+            -0.004632438998669386,
+            0.009295180439949036,
+            0.015681356191635132,
+            -0.000702630786690861,
+            -0.006942977197468281,
+            0.016438908874988556,
+            0.004056698642671108,
+            0.0075414446182549,
+            -0.01356020662933588,
+            -0.008961857296526432,
+            0.001507531269453466,
+            0.006257391534745693,
+            -0.005939218681305647,
+            -0.0026003022212535143,
+            0.005530139897018671,
+            0.003943065647035837,
+            -0.013302638195455074,
+            0.01689344085752964,
+            -0.016166189685463905,
+            0.008742166683077812,
+            -0.0037972365971654654,
+            0.0121360057964921,
+            -0.006799042224884033,
+            0.0013153020991012454,
+            -0.01818128302693367,
+            -0.003869204083457589,
+            0.0038597346283495426,
+            0.012173883616924286,
+            -0.01262841559946537,
+            0.025484098121523857,
+            0.008166425861418247,
+            0.006602078210562468,
+            0.0059467945247888565,
+            -0.012855681590735912,
+            0.027211319655179977,
+            0.005969521123915911,
+            0.009886072017252445,
+            -0.008893677033483982,
+            0.011136035434901714,
+            0.035362593829631805,
+            0.03060515783727169,
+            0.014135947450995445,
+            0.004469565115869045,
+            -0.018999440595507622,
+            0.015120766125619411,
+            0.0018427487229928374,
+            0.0023806116078048944,
+            0.00863610953092575,
+            -0.011348150670528412,
+            0.01305264513939619,
+            -0.009484568610787392,
+            0.015241974964737892,
+            -0.021272100508213043,
+            -0.011749654076993465,
+            -0.014408666640520096,
+            -0.021075136959552765,
+            -0.01430260855704546,
+            0.034180812537670135,
+            -0.002346521709114313,
+            -0.010393633507192135,
+            0.0045112306252121925,
+            0.008477022871375084,
+            -0.006079366430640221,
+            0.0053142374381423,
+            -0.008204303681850433,
+            -0.013749594800174236,
+            -0.01889338344335556,
+            0.006628592498600483,
+            -0.01550711877644062,
+            0.007651289459317923,
+            -0.003304826794192195,
+            0.0037688282318413258,
+            0.007014944683760405,
+            0.018105527386069298,
+            0.00530666159465909,
+            -0.00135412672534585,
+            0.0014033677289262414,
+            0.0016865032957866788,
+            -0.00931033119559288,
+            0.0108330138027668,
+            0.015408637002110481,
+            -0.0011921997647732496,
+            -0.011279970407485962,
+            -0.009136094711720943,
+            -0.042877525091171265,
+            0.0006264019757509232,
+            -0.013128400780260563,
+            0.0064354161731898785,
+            -0.011196639388799667,
+            0.0022234192583709955,
+            0.015863168984651566,
+            -0.0054051438346505165,
+            -0.0010918239131569862,
+            0.02193874679505825,
+            0.013014768250286579,
+            0.021075136959552765,
+            0.02065090648829937,
+            0.014188976027071476,
+            -0.015302578918635845,
+            -0.03469594568014145,
+            0.002221525413915515,
+            -0.010030006989836693,
+            0.013772320933640003,
+            0.004238511435687542,
+            0.016060132533311844,
+            -0.023893235251307487,
+            -0.011878437362611294,
+            -0.006662682630121708,
+            0.014431392773985863,
+            0.01491622719913721,
+            -0.0032271775417029858,
+            0.02690829709172249,
+            0.009280029684305191,
+            0.0020510759204626083,
+            -0.02281750924885273,
+            0.003312402404844761,
+            0.005321812815964222,
+            -0.008370965719223022,
+            -0.016681326553225517,
+            -0.0004552422615233809,
+            -0.0012168202083557844,
+            -0.0007817004225216806,
+            -0.021650876849889755,
+            0.018832778558135033,
+            -0.004018820822238922,
+            0.0037574649322777987,
+            0.013840501196682453,
+            0.01516621932387352,
+            0.017984317615628242,
+            0.008370965719223022,
+            0.001497115008533001,
+            -0.006605865899473429,
+            -0.0007973249885253608,
+            -0.0005828426801599562,
+            -0.007389933802187443,
+            -0.005598319694399834,
+            0.006382387597113848,
+            -0.01134057529270649,
+            0.008810346014797688,
+            0.000896753859706223,
+            0.006113456096500158,
+            0.010605747811496258,
+            -0.004836978856474161,
+            -0.011628445237874985,
+            0.014628356322646141,
+            -0.0010548931313678622,
+            -0.0055755930952727795,
+            -0.011060279794037342,
+            0.003624893259257078,
+            0.013575357384979725,
+            -0.003096499713137746,
+            -0.013499601744115353,
+            0.00910579226911068,
+            0.002009410411119461,
+            -0.027438584715127945,
+            0.015499543398618698,
+            0.010295150801539421,
+            -0.013370818458497524,
+            0.004980913829058409,
+            -0.003371112747117877,
+            -0.03027183562517166,
+            0.004791525658220053,
+            0.002660906407982111,
+            -0.010643625631928444,
+            0.028938541188836098,
+            -0.003821857040748,
+            0.029256712645292282,
+            -0.004980913829058409,
+            -0.006905099377036095,
+            -0.011355726048350334,
+            0.009416389279067516,
+            -0.018681267276406288,
+            -0.0037953427527099848,
+            -0.008727015927433968,
+            -0.00915882084518671,
+            -0.0009838725673034787,
+            -0.005185453221201897,
+            0.006984642706811428,
+            0.005121061112731695,
+            -0.012704171240329742,
+            0.006757376715540886,
+            -0.010257272981107235,
+            -0.00020856392802670598,
+            0.0006718551740050316,
+            -0.008408842608332634,
+            0.013885954394936562,
+            0.005996035411953926,
+            -0.008340663276612759,
+            -0.0005032995832152665,
+            0.027817361056804657,
+            0.05051366239786148,
+            -0.018863080069422722,
+            0.0009270560112781823,
+            0.01452229917049408,
+            -0.0028824908658862114,
+            -0.0021590273827314377,
+            -0.011976920068264008,
+            0.005821797996759415,
+            -0.019423669204115868,
+            0.013317788951098919,
+            0.006473293993622065,
+            -0.008833073079586029,
+            0.003202557098120451,
+            -0.016711628064513206,
+            0.002621134975925088,
+            -0.0018058179412037134,
+            -0.018681267276406288,
+            -0.015529844909906387,
+            -0.012060250155627728,
+            -0.024181105196475983,
+            0.0004235197266098112,
+            -0.0008546149474568665,
+            -0.013870803639292717,
+            0.006579351611435413,
+            0.008431569673120975,
+            0.032483890652656555,
+            0.0011912527261301875,
+            0.010128489695489407,
+            -2.1513333194889128e-05,
+            -0.0010416359873488545,
+            -0.005988460034132004,
+            -0.01523439958691597,
+            0.00015387803432531655,
+            -0.002342734020203352,
+            0.022590242326259613,
+            0.005549078807234764,
+            0.017560089007019997,
+            0.0017887729918584228,
+            -0.016514664515852928,
+            -0.00596573343500495,
+            -0.007408872712403536,
+            -0.018060073256492615,
+            0.0062649669125676155,
+            -0.005473323632031679,
+            0.009605777449905872,
+            -0.015863168984651566,
+            -0.02731737680733204,
+            0.00792400911450386,
+            0.012234487570822239,
+            0.028514310717582703,
+            -0.0035623949952423573,
+            0.001716805505566299,
+            -0.005533927585929632,
+            0.016408607363700867,
+            0.032180868089199066,
+            -0.0008550884085707366,
+            0.006950552575290203,
+            -0.019999410957098007,
+            0.0017404790269210935,
+            0.013704141601920128,
+            -0.008401267230510712,
+            -0.01821158453822136,
+            0.003939277958124876,
+            -0.020635755732655525,
+            0.004583198111504316,
+            -0.020393338054418564,
+            0.030408194288611412,
+            -0.0023522034753113985,
+            0.008105821907520294,
+            0.004310478921979666,
+            0.01760554127395153,
+            -0.0031646795105189085,
+            0.010545143857598305,
+            0.013916256837546825,
+            0.014908651821315289,
+            0.0009980766335502267,
+            -0.011189064010977745,
+            -0.03790797293186188,
+            0.001859793672338128,
+            0.00817400123924017,
+            0.018635815009474754,
+            -0.01980244740843773,
+            0.011681473813951015,
+            -0.01296931505203247,
+            -0.020363036543130875,
+            -0.01003758329898119,
+            0.011575416661798954,
+            -0.006139970384538174,
+            0.03378688171505928,
+            -0.005874827038496733,
+            -0.008204303681850433,
+            0.005136212334036827,
+            0.010454237461090088,
+            0.0066778333857655525,
+            0.014704111963510513,
+            0.0326354019343853,
+            0.010196669027209282,
+            0.014173824340105057,
+            -0.004859705455601215,
+            0.02399929240345955,
+            -0.003931702114641666,
+            0.003954428713768721,
+            0.00509454682469368,
+            -0.022832660004496574,
+            -0.0040983641520142555,
+            -0.003931702114641666,
+            -0.016711628064513206,
+            -0.007458113599568605,
+            0.0032366469968110323,
+            0.008802770636975765,
+            -0.014045040123164654,
+            -0.00649980828166008,
+            -0.01350717805325985,
+            0.01218903437256813,
+            0.002730980049818754,
+            0.018302490934729576,
+            -0.029802151024341583,
+            0.009825468063354492,
+            -0.017484333366155624,
+            -0.017044952139258385,
+            0.006511171814054251,
+            -0.0038332203403115273,
+            0.006855858489871025,
+            -0.018272189423441887,
+            -0.01318900566548109,
+            0.012817803770303726,
+            0.0007897494360804558,
+            0.013060221448540688,
+            -0.009825468063354492,
+            0.0009706153068691492,
+            0.006064215209335089,
+            -0.009833043441176414,
+            -0.010961798019707203,
+            -0.007977037690579891,
+            -0.009681533090770245,
+            -0.008848223835229874,
+            0.005621046293526888,
+            0.01656011864542961,
+            -0.013742019422352314,
+            -0.003685497445985675,
+            0.016135888174176216,
+            0.006416477728635073,
+            -0.008166425861418247,
+            0.004075637552887201,
+            -0.003793448908254504,
+            -9.948806837201118e-05,
+            0.0013740125577896833,
+            0.012439027428627014,
+            -0.0017632056260481477,
+            0.024665940552949905,
+            -0.02654467150568962,
+            0.007598260883241892,
+            -0.0054051438346505165,
+            0.005446809343993664,
+            0.018226735293865204,
+            -0.02366596832871437,
+            -0.013082947582006454,
+            0.006276329979300499,
+            -0.01966608688235283,
+            -0.008060368709266186,
+            0.0054051438346505165,
+            -0.00016405766655225307,
+            -0.019923655316233635,
+            -0.012923861853778362,
+            -0.034786853939294815,
+            -0.02309022843837738,
+            0.024665940552949905,
+            0.008401267230510712,
+            -0.0073672072030603886,
+            0.018060073256492615,
+            -0.018438849598169327,
+            0.0172722190618515,
+            -0.002225313102826476,
+            -0.006613441277295351,
+            0.003935490269213915,
+            0.007674016058444977,
+            0.025362888351082802,
+            0.002653330797329545,
+            0.008264907635748386,
+            -0.0042082094587385654,
+            -0.014931377954781055,
+            -0.006321783177554607,
+            0.01980244740843773,
+            0.02471139281988144,
+            0.018605511635541916,
+            -0.018575210124254227,
+            0.010742107406258583,
+            -0.011749654076993465,
+            0.01818128302693367,
+            -0.006537686102092266,
+            0.00039037675014697015,
+            0.01859036087989807,
+            0.007969462312757969,
+            0.025241680443286896,
+            0.019923655316233635,
+            0.006852070800960064,
+            0.014128371141850948,
+            0.011908739805221558,
+            0.001204509986564517,
+            -0.012832955457270145,
+            -0.01154511421918869,
+            -0.02840825356543064,
+            0.042847223579883575,
+            0.0025434857234358788,
+            0.008317936211824417,
+            -0.027635548263788223,
+            -0.00021389048197306693,
+            -0.0034942151978611946,
+            -0.018832778558135033,
+            0.005939218681305647,
+            0.03260510042309761,
+            0.03284751623868942,
+            0.021438762545585632,
+            -2.888172457460314e-05,
+            0.0012234487803652883,
+            0.0329081192612648,
+            -0.013673839159309864,
+            -0.00937851145863533,
+            -0.01034060399979353,
+            -0.018393397331237793,
+            -0.006852070800960064,
+            -0.0032423287630081177,
+            -0.01031030248850584,
+            -0.0061172437854111195,
+            -0.009272454306483269,
+            0.0238629337400198,
+            0.0001374249259242788,
+            -0.0022594030015170574,
+            0.006605865899473429,
+            -0.020878173410892487,
+            0.011537538841366768,
+            -0.02813553437590599,
+            -0.013431422412395477,
+            -0.008651260286569595,
+            -0.006950552575290203,
+            -0.018802475184202194,
+            0.008643684908747673,
+            -0.006942977197468281,
+            0.021438762545585632,
+            0.002914686920121312,
+            -0.023590214550495148,
+            0.007530081085860729,
+            0.015317730605602264,
+            -0.0003825644962489605,
+            0.014037464745342731,
+            -0.003890036838129163,
+            0.009060339070856571,
+            0.0045112306252121925,
+            0.011681473813951015,
+            0.016954045742750168,
+            0.010651201009750366,
+            0.009030036628246307,
+            0.003437398700043559,
+            -0.001167579204775393,
+            0.01878732442855835,
+            -0.0007163614500313997,
+            -0.03405960276722908,
+            2.9236827685963362e-05,
+            -0.014628356322646141,
+            0.029499130323529243,
+            -0.009795165620744228,
+            0.009514871053397655,
+            0.008105821907520294,
+            0.007098275702446699,
+            -0.0067119235172867775,
+            0.007916433736681938,
+            -0.008742166683077812,
+            0.0052763596177101135,
+            0.015832867473363876,
+            -0.0042877523228526115,
+            -0.0014980619307607412,
+            0.007840678095817566,
+            0.00407184986397624,
+            -0.005454384721815586,
+            -0.019711541011929512,
+            -0.012287517078220844,
+            0.00726114958524704,
+            -0.02065090648829937,
+            -0.004367295652627945,
+            0.01161329448223114,
+            -0.005454384721815586,
+            -0.009620928205549717,
+            0.01393140759319067,
+            0.02136300690472126,
+            -0.002382505452260375,
+            0.022893264889717102,
+            -0.01349202636629343,
+            -0.009833043441176414,
+            -0.007984613068401814,
+            -0.004374871030449867,
+            -0.041120000183582306,
+            0.025984082370996475,
+            0.01068150345236063,
+            -0.006871009711176157,
+            0.01895398646593094,
+            0.0022499337792396545,
+            -0.002954458352178335,
+            -0.0011041341349482536,
+            0.00010499217023607343,
+            -0.03281721472740173,
+            0.03660498186945915,
+            -0.007007369305938482,
+            0.0005449649761430919,
+            0.010272424668073654,
+            0.042029064148664474,
+            -0.004590773954987526,
+            -0.005859675817191601,
+            0.01225721463561058,
+            0.005336964037269354,
+            0.031938452273607254,
+            -0.010416359640657902,
+            -0.014340486377477646,
+            -0.01815097965300083,
+            -0.004833190701901913,
+            -0.01939336769282818,
+            -0.0038313264958560467,
+            0.014037464745342731,
+            -0.019650936126708984,
+            -0.011242092587053776,
+            -0.001287840772420168,
+            -0.012537509202957153,
+            -0.0010747789638116956,
+            0.008598231710493565,
+            0.017832808196544647,
+            -0.011348150670528412,
+            -0.004450626205652952,
+            -2.2904154320713133e-05,
+            -0.005003640428185463,
+            0.0185146052390337,
+            0.0020056227222085,
+            -0.01702980138361454,
+            0.01321930717676878,
+            -0.00526499655097723,
+            0.006749800872057676,
+            0.0074429623782634735,
+            0.007939159870147705,
+            0.05145302787423134,
+            0.010317877866327763,
+            -0.001339922659099102,
+            -0.01607528328895569,
+            0.023196285590529442,
+            -0.01749948412179947,
+            -0.014181400649249554,
+            -0.02119634486734867,
+            -0.0029923359397798777,
+            -0.007060397882014513,
+            -0.028014326468110085,
+            -9.137987944995984e-05,
+            0.0016221113037317991,
+            -0.0024980325251817703,
+            -0.010045158676803112,
+            0.0005208179936744273,
+            -0.011529963463544846,
+            -0.006954340264201164,
+            -0.008545203134417534,
+            -0.016044981777668,
+            0.02240842953324318,
+            -0.021817538887262344,
+            -0.008855799213051796,
+            0.011514811776578426,
+            0.03118089959025383,
+            -0.005011215806007385,
+            -0.014173824340105057,
+            -0.019817598164081573,
+            0.0025643182452768087,
+            -0.01763584464788437,
+            0.012181458994746208,
+            -0.01225721463561058,
+            0.015045011416077614,
+            -0.03142331540584564,
+            -0.006079366430640221,
+            -0.008992158807814121,
+            0.012840530835092068,
+            0.019817598164081573,
+            -0.0025093958247452974,
+            0.00606800289824605,
+            -0.010295150801539421,
+            0.030211230739951134,
+            -0.00012286569108255208,
+            -0.018605511635541916,
+            0.00873459130525589,
+            0.007977037690579891,
+            0.012537509202957153,
+            -0.004435475450009108,
+            0.00257757562212646,
+            0.011567841283977032,
+            0.012742049060761929,
+            -0.01634800247848034,
+            -0.003174148965626955,
+            -0.017969166859984398,
+            0.005996035411953926,
+            0.006999793462455273,
+            0.021711481735110283,
+            -0.0013550736475735903,
+            0.018272189423441887,
+            -0.01563590206205845,
+            0.00019412307301536202,
+            -0.011749654076993465,
+            -0.0082270298153162,
+            -0.0028237802907824516,
+            -0.003958216868340969,
+            0.006685409229248762,
+            -0.028211290016770363,
+            -0.01274962443858385,
+            0.005287723150104284,
+            -0.004992276895791292,
+            -0.03405960276722908,
+            0.0053710537031292915,
+            -0.02725677192211151,
+            0.013128400780260563,
+            -0.002397656673565507,
+            -0.017847958952188492,
+            -0.00030846628942526877,
+            0.026332557201385498,
+            -0.003677921835333109,
+            -0.012537509202957153,
+            0.015544996596872807,
+            -0.006757376715540886,
+            -0.02119634486734867,
+            -0.005420294590294361,
+            0.014658658765256405,
+            -0.0010719380807131529,
+            -0.012393574230372906,
+            -0.029635490849614143,
+            0.012992041185498238,
+            0.0026874206960201263,
+            -0.007386146113276482,
+            0.010893618687987328,
+            0.025953780859708786,
+            0.008151275105774403,
+            -0.005924067925661802,
+            0.0015757111832499504,
+            0.0049581872299313545,
+            0.00300938100554049,
+            0.020287280902266502,
+            0.02728707529604435,
+            -0.008014915511012077,
+            -0.0011145505122840405,
+            -0.014954105019569397,
+            0.015802564099431038,
+            0.0013020449550822377,
+            0.010507266037166119,
+            0.021893294528126717,
+            0.007045246660709381,
+            -0.005738467443734407,
+            0.003905187826603651,
+            -0.023135682567954063,
+            0.01420412678271532,
+            -0.012272365391254425,
+            0.000622140767518431,
+            0.008749742060899734,
+            -0.007283876184374094,
+            0.01437078882008791,
+            -0.0015245762187987566,
+            0.0339989997446537,
+            0.025211378931999207,
+            0.0043218424543738365,
+            0.0007362472242675722,
+            -0.028559764847159386,
+            0.003547244006767869,
+            0.008651260286569595,
+            0.014499573037028313,
+            -0.0076664406806230545,
+            0.006261179223656654,
+            -0.004355932120233774,
+            0.00959820207208395,
+            -0.0065490491688251495,
+            -0.015893470495939255,
+            0.006113456096500158,
+            0.0012689019786193967,
+            0.006795254070311785,
+            0.0023408401757478714,
+            -0.008863375522196293,
+            -0.024362917989492416,
+            -0.025984082370996475,
+            0.021984200924634933,
+            -0.03918066248297691,
+            0.0073293293826282024,
+            0.011234517209231853,
+            0.03324144333600998,
+            -0.008158850483596325,
+            -0.004980913829058409,
+            0.006276329979300499,
+            -0.016711628064513206,
+            -0.004147605039179325,
+            0.008204303681850433,
+            0.012385998852550983,
+            -0.002420383272692561,
+            0.00974971242249012,
+            0.01318900566548109,
+            -0.012378423474729061,
+            0.03660498186945915,
+            0.016514664515852928,
+            0.01467380952090025,
+            -0.004007457755506039,
+            0.01312082540243864,
+            0.0066172294318675995,
+            0.019044892862439156,
+            0.025165924802422523,
+            -0.017454031854867935,
+            -0.002365460619330406,
+            -0.02292356640100479,
+            0.024165954440832138,
+            0.00726114958524704,
+            0.004556683823466301,
+            -0.008310360834002495,
+            0.014870774000883102,
+            0.0017954016802832484,
+            0.0028199926018714905,
+            -0.006696772295981646,
+            -0.010173942893743515,
+            0.019241856411099434,
+            0.0034317169338464737,
+            0.014870774000883102,
+            4.8501173296244815e-05,
+            -0.005503625608980656,
+            0.006052851676940918,
+            0.014832896180450916,
+            -0.023590214550495148,
+            -0.00203592493198812,
+            -0.019484274089336395,
+            -0.03202936053276062,
+            0.009022461250424385,
+            0.0016135887708514929,
+            -0.007052822504192591,
+            0.03230207785964012,
+            -0.010083036497235298,
+            -0.019787294790148735,
+            -0.008067944087088108,
+            -0.013113250024616718,
+            -0.010469388216733932,
+            0.0008271536789834499,
+            0.003908975515514612,
+            -0.00012428610352799296,
+            -0.009317907504737377,
+            -0.005749830510467291,
+            0.01990850456058979,
+            -0.0030358952935785055,
+            -0.00753386877477169,
+            0.00868156272917986
+          ],
+          "embedding": [
+            0.0040707582550429606,
+            -0.010230625151989194,
+            -0.021520242101903186,
+            -0.010115756270640508,
+            0.026738734586159035,
+            0.01796370233223267,
+            -0.012245199029994153,
+            0.06506924494147726,
+            -0.0014087587222033442,
+            0.009300068719271429,
+            0.003787396973047584,
+            0.002818772639754174,
+            0.022129077062176306,
+            -0.021526709808445683,
+            -0.0034388135367225824,
+            0.02763761747879405,
+            0.013210013805395276,
+            0.025255872672296602,
+            0.00029011480909986703,
+            -0.05438974762290585,
+            0.016671640187490865,
+            -0.016426252259893328,
+            -0.056591180651012554,
+            0.009335648425523075,
+            0.006141624459418827,
+            0.02568943956694641,
+            0.019377132054042116,
+            0.017946464337078746,
+            -0.057184776985927686,
+            0.03208527139096687,
+            0.030132392063755547,
+            -0.003846032371791839,
+            -0.008376524427531415,
+            -0.0010513963058713435,
+            -0.02734868388724371,
+            -0.014879400370545479,
+            0.04383920959748825,
+            0.02563613659516289,
+            -0.020117467074438997,
+            0.0398016699155716,
+            0.029896497884197775,
+            -0.0004195844530067313,
+            -0.023302720788253625,
+            -0.001168337916577517,
+            0.009149021369661628,
+            -0.02508145721900992,
+            -0.004307594138480351,
+            0.00550940415564303,
+            0.014016998918718476,
+            -0.028607379616884317,
+            -0.0018317656587526954,
+            -0.01973464940415756,
+            0.03762738392056255,
+            0.015358106058583114,
+            -0.03931907682104936,
+            0.011519637298832444,
+            0.004482080558226775,
+            0.0747867063383242,
+            0.05550679018263077,
+            0.023243129601931176,
+            0.02671631686665238,
+            -0.009686146566573536,
+            -0.04088582189725495,
+            -0.0065344623594924085,
+            0.019617295999325712,
+            -0.004722087402575777,
+            -0.002871844114437583,
+            0.003935087265300519,
+            0.03229890413393201,
+            -0.02813577047904649,
+            -0.05124172132597234,
+            0.03325235407841968,
+            -0.008687655557217192,
+            -0.011027174838246948,
+            -0.003653553063547375,
+            -0.02350737945182988,
+            0.016223087939183468,
+            0.03789252356462714,
+            0.0034350623378603083,
+            0.00195158657888739,
+            0.011789712862232108,
+            -0.0004435439660769551,
+            -0.0037330456918125894,
+            0.0031552713243866858,
+            0.04465432992298031,
+            -0.0032465421571915712,
+            -0.0075366743861922745,
+            -0.059958752731410256,
+            0.016784616861008125,
+            -0.028734506065772967,
+            -0.03035575736273943,
+            0.02680390892611741,
+            0.02338368719038086,
+            -0.0030658375437784277,
+            -0.03754896050794851,
+            0.07005659523248504,
+            0.0020711575553570695,
+            0.013034425083076923,
+            -0.02316832262132332,
+            0.021332142030732894,
+            -0.0318756233020585,
+            0.02270782509507388,
+            0.01570810309454529,
+            0.004788124787612736,
+            0.026727250200142456,
+            -0.022380142778058276,
+            -0.008593910379436904,
+            0.014439754382387034,
+            0.015007697617383174,
+            0.07662531963612977,
+            0.011195998137723795,
+            -0.010809283538470098,
+            -0.042227822120219305,
+            0.003824320933275175,
+            -0.03797629427607094,
+            -0.02347390684453348,
+            -0.0526876440280276,
+            -0.017799861680287913,
+            0.01701482796266917,
+            0.002730395144917155,
+            0.03789451759899825,
+            0.038338545350313946,
+            0.028463789278406586,
+            0.015494583705011921,
+            -0.05700237266350479,
+            0.047895170359075166,
+            -0.0064406894534929406,
+            -0.0007954721295065988,
+            0.012543029839729358,
+            -0.004703334868741216,
+            -0.0060569679284068,
+            0.012324901254711192,
+            -0.019883935809784068,
+            -0.04253565942154215,
+            0.02164409276211115,
+            -0.0075367555415808635,
+            0.03617159147722157,
+            0.07567745988686005,
+            -0.04225977020036783,
+            -0.022548837589622003,
+            0.029811602145850144,
+            -0.02175792196819533,
+            0.02069861458115363,
+            -0.00941331829664397,
+            -0.035479639885943264,
+            0.020352552580119105,
+            0.003678588588550461,
+            0.06776105274336729,
+            -0.02346047721661687,
+            0.02721075076349443,
+            -0.029386781707644253,
+            0.023530831921117993,
+            -0.02077660533100372,
+            0.020022475323372906,
+            0.024195118676513715,
+            -0.028378518720494666,
+            -0.012900969023913307,
+            0.019264089970719624,
+            -0.0374983624629265,
+            -0.019726171557384355,
+            0.00979636085642416,
+            -0.014479528435832063,
+            -0.014533384583777662,
+            0.005904623758767224,
+            0.010344475500263744,
+            0.05659646358026115,
+            0.04163866113482093,
+            0.007152494819169405,
+            -0.07584500130974763,
+            0.05471858597878099,
+            0.03580211013427615,
+            0.044161429444475306,
+            0.02258180566046248,
+            0.00980779221334715,
+            0.0011193505603397395,
+            0.018086566768892507,
+            0.015605066702191871,
+            -0.00954727399396313,
+            -0.027950079146919375,
+            0.03028514025485407,
+            -0.006351669761257431,
+            0.005095804801686436,
+            0.009620040085743162,
+            0.019748001227987134,
+            0.033709535059510916,
+            0.008203559956940196,
+            0.007389752602218563,
+            0.04197771878351742,
+            -0.02067363758471765,
+            0.007543739188471913,
+            0.011207941261799052,
+            0.013964827251172813,
+            0.03839562075248879,
+            0.0009775244543292637,
+            0.006469507611492216,
+            -0.022808235395746654,
+            -0.024795004398633292,
+            0.006249024864814684,
+            0.011292877109066925,
+            0.01596839619826819,
+            -0.0014105396434174883,
+            -0.04152075438277094,
+            0.008267788411079337,
+            -0.028834191212891387,
+            0.030940526405027906,
+            -0.0018271056434014425,
+            0.006185475047820603,
+            -0.007920574674727123,
+            0.012907560753043616,
+            -0.015165704767187252,
+            -0.021962244496450727,
+            0.0037071053072833147,
+            -0.013508376064336042,
+            -0.015178823540513864,
+            -0.02518557053061311,
+            0.011693256916760944,
+            -0.01716015674592331,
+            0.02048897513447166,
+            -0.003412078966361745,
+            -0.001807298511378529,
+            -0.0012575655363779904,
+            0.03116103961636643,
+            0.011668533361641842,
+            -0.011637010548030312,
+            -0.040260195687925236,
+            -0.011129376691522074,
+            -0.0034493180842871908,
+            -0.02098244789711689,
+            0.026658743116016938,
+            -0.01923736325953796,
+            -0.009723937181300192,
+            -0.013134679989565051,
+            -0.011974548914818263,
+            -0.004028712301052159,
+            0.019628329175867303,
+            -0.028192034966932684,
+            -0.012638520793012082,
+            -0.011568652932013111,
+            -0.0025919910519445737,
+            0.012497908463405111,
+            0.015113186712897145,
+            0.014880607940212641,
+            -0.03059227859452334,
+            -0.021899716217105084,
+            -0.0008513506358489523,
+            0.011985931118464679,
+            -0.01321584595585435,
+            -0.011735288366213981,
+            0.010710683541435907,
+            0.01961722172525993,
+            -0.023571946726192585,
+            -0.03364281880125354,
+            0.0014460076743756465,
+            0.016563167828467187,
+            -0.02891589059260944,
+            0.04595166331401113,
+            0.034271553243501465,
+            -0.027557971577291153,
+            -0.007138821520412525,
+            -0.039508407807651844,
+            0.06814329636830853,
+            0.00022136123564675344,
+            -0.039718016705753735,
+            0.0503457594963753,
+            -0.003580581891576583,
+            0.013820796398521455,
+            0.02054050419088757,
+            0.035682661890720495,
+            -0.0013220739026852943,
+            -0.0052759394697328675,
+            0.04733018719162478,
+            -0.0187845855131196,
+            -0.03982699468608443,
+            -0.023905460465070062,
+            0.002631233133039351,
+            -0.03090995554989513,
+            -0.012617612051488022,
+            -0.013605787969473513,
+            0.039672783793764745,
+            0.008669582242097126,
+            0.009974277556736362,
+            0.005535636914843583,
+            0.014180668493093584,
+            0.012042659947850441,
+            0.011870115226306776,
+            0.00451259454348581,
+            -0.014023048592193627,
+            -0.007769033996392623,
+            -0.05715987527668358,
+            0.003374564518384728,
+            -0.028801158523506985,
+            -0.00756331369170698,
+            -0.02163054894055572,
+            0.005367181573233089,
+            0.02308705505078254,
+            -0.005438909398653553,
+            -0.00873620909426313,
+            -0.0036530652237719077,
+            0.013920407174296543,
+            -0.01539613379172909,
+            -0.03908211213269472,
+            -0.016810726548844992,
+            0.007460173854386443,
+            -0.007069189426742819,
+            -0.041197222498539215,
+            -0.003843233633335066,
+            0.004373367480223065,
+            0.016194121660459908,
+            -0.029165150066068787,
+            -0.00735139702658342,
+            0.017257895082337448,
+            0.014153313280536095,
+            -0.002376178042741926,
+            -0.023223700836427403,
+            -0.0319877537205969,
+            0.02088060774052704,
+            0.0012829290529515294,
+            0.03251550040316258,
+            -0.04128453069131847,
+            -0.023376114412063578,
+            0.04083022735076889,
+            0.005690491480259003,
+            -0.05810403800689653,
+            0.016582434816685452,
+            0.010402038964417569,
+            0.016795644054313566,
+            -0.00908590043098552,
+            0.05003656634510042,
+            -0.037216197954986735,
+            -0.013027960261454162,
+            -0.007090943677204207,
+            -0.05175371787508879,
+            -0.0020065660448063322,
+            -0.014908489552053589,
+            0.010627472758306387,
+            -0.03275473161158777,
+            -0.017334608434133866,
+            0.025321047439048318,
+            -0.0033515195870701753,
+            0.011526068468647207,
+            -0.026783879816504975,
+            -0.0012830935419413407,
+            -0.017506908439998053,
+            -0.02114853058440477,
+            0.004878884893669383,
+            -0.020936900837115968,
+            -0.0025587027781733642,
+            0.01557850246191957,
+            0.023652335794299834,
+            0.011489428313303988,
+            0.0019089137782552768,
+            0.02054907941022243,
+            0.014361232983780595,
+            -0.0056413109308506085,
+            -0.0010300050480036684,
+            -0.03413262505887213,
+            -0.018821160485010483,
+            0.03361989907065425,
+            0.013125889701529858,
+            -0.003742167496379789,
+            0.008813100942466033,
+            -0.0383265293638935,
+            -0.035316957263713485,
+            0.03020499323039515,
+            -0.05001623971243151,
+            -0.00855479667673597,
+            -0.005939674940109814,
+            -0.026451831943555135,
+            -0.008032873049527808,
+            0.0009431571685654594,
+            -0.014387605128858954,
+            0.016382559541755014,
+            -0.024909510232038502,
+            0.003715736505444385,
+            -0.006016712902409688,
+            -0.009110197692342108,
+            0.05507263160354303,
+            -0.022103031032598353,
+            -0.0475415595313825,
+            0.021958434235402655,
+            -0.016670809659931243,
+            0.0323519363801999,
+            0.03867494951405658,
+            0.011124508964363016,
+            -0.008002445714431074,
+            0.034041838454818577,
+            0.018400123235267234,
+            -0.01161754789471199,
+            -0.025901844576206774,
+            0.021800298132870245,
+            0.00943488425697727,
+            -0.05906691123008531,
+            -0.00782549274387971,
+            -0.019357446956797447,
+            0.008682376344965976,
+            0.02863077694455766,
+            0.0024925977373894447,
+            -0.002682170968419412,
+            -0.013590235283325244,
+            -0.021511149507762284,
+            -0.03255453916518719,
+            0.009039307015853905,
+            0.008305286476500875,
+            0.009312625027668458,
+            -0.0014144165766983945,
+            -0.007853869334376857,
+            0.009067065760942163,
+            0.022485201611182105,
+            0.03966189080765722,
+            -0.041627289519629344,
+            0.002707086456239785,
+            -0.022678178050044222,
+            -0.03653753142622053,
+            0.0036574646778835588,
+            0.029996160786576367,
+            0.02901163765559891,
+            -0.01971777193183787,
+            -0.0036589934841109884,
+            0.009181234938554578,
+            -0.025979483002227535,
+            -0.00773645103995767,
+            0.004280770859737866,
+            0.04116668301017644,
+            0.006201085381597264,
+            0.01402728258112097,
+            0.001060407317110527,
+            -0.00920801784000087,
+            0.02109880889040528,
+            0.025211022083982727,
+            -0.029866678264942617,
+            0.014439702646820226,
+            0.009379206296908577,
+            0.04156435812161122,
+            -0.037309655469525846,
+            -0.011819731348299495,
+            -0.02766498020409985,
+            -0.010454143230794476,
+            0.027911532296566614,
+            -0.04127890742215716,
+            -0.006948536183086887,
+            0.010207784113909938,
+            -0.0016584043356136647,
+            0.016437495428919636,
+            -0.014607861750686952,
+            -0.03222311919805017,
+            -0.000610723203118213,
+            -0.02867445198803213,
+            -0.024772816792038133,
+            -0.016510309753766612,
+            -0.018581201775667577,
+            -0.07450654873867177,
+            0.00026832248829154255,
+            -0.06724696990816072,
+            -0.02447002541264633,
+            0.04885569249258513,
+            -0.010195604207134053,
+            0.03332905935704749,
+            -0.02815272112271314,
+            -0.005296340304911926,
+            -0.05908835042526019,
+            0.0038476172058970774,
+            0.025753576863447866,
+            0.02386003857017594,
+            0.005206447371391696,
+            0.024799057871126032,
+            0.012060738268265606,
+            -0.04146503002980051,
+            -0.024949907556862643,
+            -0.014503475223770958,
+            -0.0016340466065744286,
+            -0.0034009167013795906,
+            -0.031298432995411005,
+            0.036241272149880994,
+            0.008902644391348814,
+            0.03550223922667804,
+            0.04463301409716232,
+            -0.010385984413759852,
+            -0.03305004479734889,
+            -0.006952739338820577,
+            -0.0175981588092108,
+            0.042668533202572856,
+            -0.017112184620086308,
+            0.005149173014526686,
+            -0.03219589107997635,
+            0.024181808150698054,
+            0.0014339790422945406,
+            0.01254721190611376,
+            0.011099458649404119,
+            0.0027626836767364253,
+            -0.026883640446090158,
+            0.0024160770766463607,
+            0.023874514847956222,
+            -0.022558433617669373,
+            0.00917667715473093,
+            -0.006754091166465801,
+            -0.033686935718776144,
+            -0.01078389638118685,
+            -0.013595327486167114,
+            0.024712781935748277,
+            0.013168464807451798,
+            0.005875864604862434,
+            -0.02417814420558463,
+            -0.0023978800459714144,
+            0.028650463366555304,
+            0.020176980768301288,
+            -0.02118184201861616,
+            0.019082340279671355,
+            0.019734925740100277,
+            -0.04209645071102001,
+            -0.04217310001316092,
+            0.006553913719098956,
+            -0.003735719227468833,
+            0.0061793909220459,
+            -0.028285889794908054,
+            0.008181788076484304,
+            0.004432488437404309,
+            -0.01743905248466261,
+            0.041221269621866216,
+            -0.014892354419172107,
+            0.005087102208335201,
+            -0.022198383716186407,
+            -0.011936388003605221,
+            -0.00027487042024290816,
+            0.044599215801158845,
+            0.008068975171827777,
+            0.003496950921841073,
+            0.02238779276447619,
+            0.0054839808262270315,
+            0.04816723616939378,
+            -0.02685221260466647,
+            -0.0190356946337995,
+            0.016464136459560113,
+            -0.0176092780782677,
+            -0.012513274095432353,
+            -0.010695421445334002,
+            -0.004784348908457072,
+            -0.022203041334932165,
+            0.010788155380415583,
+            -0.018353897272137543,
+            -0.02277431189907027,
+            -0.003945465850796918,
+            0.016024639792926797,
+            -0.006248368166794934,
+            -0.0038769803892794875,
+            0.0016292556540434277,
+            0.02412864830338344,
+            0.02892955453819876,
+            -0.02408022470555968,
+            0.019190282372105023,
+            0.018941584034503536,
+            -0.006709516380602965,
+            -0.020956377933747532,
+            -0.01964885206488053,
+            0.01434623424964373,
+            -0.01352968586212152,
+            -0.013872556892707697,
+            0.004052798542613835,
+            0.009201147374299997,
+            -0.004086067985859617,
+            -0.04185829685685201,
+            -0.014668698123513235,
+            0.003179403563664022,
+            -0.0006415014901450319,
+            0.019360184344240947,
+            -0.0150817538567364,
+            -0.029771999278536083,
+            0.014221776097046751,
+            0.022944884879589678,
+            -0.04493425220030206,
+            0.02117577719533236,
+            -0.0030648751497261983,
+            -0.00016251533643103604,
+            -3.817296018601414e-05,
+            0.008727767382077138,
+            0.026914779443266063,
+            0.03456565094655899,
+            0.01974806772696219,
+            0.00450800007152441,
+            -0.008952599700759673,
+            0.025184190461155245,
+            0.009846044083064704,
+            -0.025168735183134738,
+            0.010983353565382029,
+            0.016740721743326522,
+            0.0070838930012654296,
+            0.009173879407277282,
+            -0.006078288608766152,
+            -0.018269347893097643,
+            0.013181494188066667,
+            -0.01435693747022231,
+            0.01678907949532146,
+            0.05154463552784974,
+            -0.00039632596424461964,
+            -0.02461439550772063,
+            0.008418147037164186,
+            -0.006544319842335593,
+            -0.010065129286193278,
+            0.02214628550460128,
+            0.0029515574751370417,
+            -0.030862152033847717,
+            0.007509556025223583,
+            0.00675485304374519,
+            0.01894936231559203,
+            -0.02355525295639599,
+            -0.01387885858544803,
+            -0.00431398020087544,
+            0.0021208056468763523,
+            0.01580766504149989,
+            -0.004739477121073512,
+            -0.013204823447960706,
+            0.02994098229517376,
+            0.025496728234662373,
+            -0.00026124946738213036,
+            0.015060064980768175,
+            0.022262345898533903,
+            0.009968408664790368,
+            0.007588431141088797,
+            -0.0083063413609779,
+            -0.031336272171937095,
+            0.0018957858383561667,
+            0.05800811417728381,
+            -0.014937085296865802,
+            0.04313875432235027,
+            0.02018476425923236,
+            0.015657644644987107,
+            0.017880263053253607,
+            -0.01662268082870562,
+            -0.008740053682875323,
+            -0.0017630394204179638,
+            -0.008294417990024647,
+            -0.027613464270562948,
+            -0.005953209556351086,
+            -0.02379961210881074,
+            -0.013878676554701543,
+            0.015418315245979704,
+            -0.014764728065382074,
+            0.0027420640870858457,
+            0.010395538988699663,
+            -0.006590633833835581,
+            -0.00841537209927409,
+            -0.003943772700797381,
+            0.03376338017637296,
+            0.011980354721458463,
+            0.027761929822019422,
+            -0.00011128539412582973,
+            0.009342430952107655,
+            -0.018419526564396063,
+            -0.006839752805318664,
+            0.03271190887864607,
+            0.006011520427944665,
+            0.004652835471939905,
+            0.01485920920301398,
+            0.014305626381398871,
+            0.038015677219487023,
+            -0.005102005401565164,
+            -0.018204460895989052,
+            0.02065495133597456,
+            -0.0049699013356374775,
+            0.019149036693562712,
+            0.008825981577388056,
+            -0.002660055848523568,
+            -0.004292379561953203,
+            0.03090270710860331,
+            -0.020243440437217503,
+            0.018735727616321387,
+            -0.02238236089688497,
+            -0.0276584413929402,
+            0.023028408906176946,
+            0.008320436733620496,
+            0.04047641118026783,
+            0.011463851066078041,
+            -0.02247143742166039,
+            -0.034588725413285185,
+            0.009827713930789184,
+            0.00337071450636392,
+            -0.024446280199879062,
+            0.017542777160283324,
+            0.00780836483490212,
+            0.008453886667283516,
+            0.016624116261219637,
+            -0.004175682824655864,
+            -0.0042784362933758835,
+            -0.001546656717896957,
+            -0.008819739495525477,
+            0.01406790007187259,
+            -0.015487658289541583,
+            -0.021457182907140035,
+            0.024687169583245905,
+            -0.03915911962859764,
+            0.01771290139419629,
+            -0.006510046907173026,
+            -0.00997450880725535,
+            -0.02059560682587115,
+            1.5939836949198004e-05,
+            -0.007159493752862143,
+            -0.0004782363814453761,
+            -0.017503181957042913,
+            0.016643971600145645,
+            0.024056442566869802,
+            -0.013209211443909658,
+            -0.00445521990927108,
+            0.0020313626821877526,
+            -0.029235692455820056,
+            0.006662209097255912,
+            0.016993027053110635,
+            0.00529940612845993,
+            0.0026104743851393716,
+            0.0001707934714983891,
+            -0.031289446611163674,
+            -0.005782208620284939,
+            0.03053516889069683,
+            0.05590704747438521,
+            0.021628254147289437,
+            0.02403314981042378,
+            0.02908606663381421,
+            0.02717206823892633,
+            -0.003198844932040613,
+            0.0016217508103442348,
+            -0.01620660606431983,
+            0.027098323143010743,
+            -0.011664168290676943,
+            0.04322175021589083,
+            0.008652323184522541,
+            -0.049493020736445056,
+            0.004537958157696794,
+            0.00943239454147692,
+            0.034918603888916544,
+            0.016678716235690318,
+            0.0005887414054803805,
+            -0.005894338664998101,
+            0.03279165663336323,
+            -0.02735726709032382,
+            -0.011962070182991011,
+            -0.00036014407560325284,
+            -0.019079003730607717,
+            -0.015938295169860884,
+            0.010357611943935459,
+            0.04278924678845588,
+            0.0009957483598272451,
+            -0.0071422790468850215,
+            0.01571201177743999,
+            -0.009445307674383,
+            0.02286338492075306,
+            0.007336440384050498,
+            0.0042809497342459904,
+            0.025581467690095977,
+            -0.0025007871391868105,
+            0.02300539823997411,
+            -0.000628898771317254,
+            0.004582863909134464,
+            -0.015960578501883316,
+            -0.00930649396114688,
+            0.027838455754803444,
+            -0.011112268986721,
+            -0.029253746504396393,
+            -0.008734785561831333,
+            0.00472579547270089,
+            -0.0003805825712986138,
+            0.0021854358149324227,
+            -0.01183885533661493,
+            -0.010840951201591629,
+            0.019965691788904057,
+            -0.004922943301424112,
+            0.003800048754664761,
+            0.012664049429916252,
+            -0.0030124219674414897,
+            -0.01910844996750822,
+            0.01889983984736845,
+            -0.003858283778519141,
+            -0.024168821338992813,
+            0.020083955711766945,
+            0.01850497889545421,
+            0.045156567155119454,
+            -0.01586824055785609,
+            0.004612757419198795,
+            0.017497774574602505,
+            -0.002172088240279357,
+            -0.019617327081566827,
+            -0.014944795772093997,
+            -0.018212462639041492,
+            -0.015975283672064702,
+            0.012837956594691596,
+            -0.01243789877725244,
+            0.019500571353639314,
+            -0.0380735226388175,
+            0.012590900534826689,
+            0.004832168601136412,
+            0.007505385730121269,
+            0.016011983211212992,
+            0.025994560026860358,
+            0.002157436199771672,
+            -0.033465476168065784,
+            -0.01014599891204166,
+            0.007845373376916211,
+            0.010093432683262838,
+            -0.002849185178237036,
+            -0.018147004183434554,
+            0.003633790611809425,
+            -0.0266147420637404,
+            -0.018644422659309035,
+            0.022926737299761104,
+            -0.04798772015599113,
+            0.017908415301694906,
+            0.024859125158586466,
+            -0.04343699525707025,
+            -0.04886699607556352,
+            -0.011459188432595229,
+            -0.036774381316312464,
+            0.0014128005215813578,
+            -0.0005256067584181517,
+            0.011194992852264102,
+            -0.009342140079829378,
+            -0.018598671981710183,
+            0.01584789414301952,
+            -0.0036412641242200418,
+            -0.004225840239935549,
+            -0.00616696736018045,
+            -0.0025598631463178384,
+            -0.010188803157718543,
+            -0.03244694662411977,
+            -0.03290891136780909,
+            -0.00033742479296272127,
+            -0.009743927051414455,
+            -0.02334288359945272,
+            -0.03447062782872289,
+            -0.008728968107967028,
+            -0.037753659749578956,
+            -0.00515561961442376,
+            0.04070890150714597,
+            0.027517147608335957,
+            0.009449414978133334,
+            0.008086900089929912,
+            -0.013989034604490105,
+            0.05691047488419298,
+            -0.020425883381389086,
+            0.039466620642512155,
+            0.005301493569940605,
+            -0.0037222353899918703,
+            0.013075425214878277,
+            0.027710826422351564,
+            -0.006253665106995653,
+            -0.0017412928073238833,
+            -0.01928096630475438,
+            -0.012920476859996416,
+            0.009444408011467955,
+            0.015799884767184885,
+            -0.0025083061619197524,
+            0.001274457251739847,
+            0.00930533161700852,
+            -0.0006652478540264343,
+            0.017258038491335247,
+            0.03996268817486439,
+            0.003833844612473671,
+            -0.00826404141706295,
+            -0.018822002570963614,
+            -0.003188844900317347,
+            -0.011807006674248829,
+            0.010482058734260632,
+            -0.015410635475145028,
+            -0.011540803927522013,
+            -0.01249428777269578,
+            -0.017081856786754605,
+            -0.002183034976477535,
+            0.003114834030705039,
+            0.0027475411722881747,
+            0.02261627635192778,
+            -0.015872027027080824,
+            -0.022487581027490516,
+            -0.0035823117885461506,
+            -0.02398778449664045,
+            8.13716819067491e-05,
+            -0.01050060782407259,
+            -0.015616181283381218,
+            0.013676480044963827,
+            -0.021513173575512936,
+            -0.01091399089076612,
+            -0.008699176177861256,
+            -0.0006986953850759474,
+            0.005853723710671249,
+            -0.00993355748621621,
+            -0.00010241843582159751,
+            0.01150872694850241,
+            0.005559552266827916,
+            -0.012336449703762907,
+            0.0039431718621504,
+            -0.012740550272925562,
+            -0.004432059174951158,
+            -0.03592309122711106,
+            -0.019665686718735042,
+            -0.01846795637714379,
+            0.005642094448288269,
+            -0.005108964028543505,
+            -0.020586678054151854,
+            0.00668364061048941,
+            0.0034775127050453774,
+            0.004713169526661792,
+            -0.010723667875209928,
+            0.013944735241748791,
+            0.010882420168586074,
+            -0.003518245916227596,
+            0.013827334494230662,
+            -0.001330399936559009,
+            -0.020286697504699764,
+            -0.011139922402071095,
+            0.03538258948179278,
+            -0.024961826464903997,
+            -0.0018019328351763667,
+            -0.0026423695690168066,
+            0.00597003516870726,
+            -0.03071835946501977,
+            -0.013634212232744325,
+            0.02164400349828422,
+            -0.005846552894831441,
+            -0.026367014426434226,
+            0.03399116480234033,
+            0.03473258231629753,
+            0.010041605592929843,
+            -0.013423384340589794,
+            0.04730054254696893,
+            -0.012968978954938308,
+            -0.018274800841400665,
+            0.026826644936918647,
+            0.0278389611094493,
+            0.02525164095926168,
+            -0.029782550447890466,
+            -0.003010713086487275,
+            -0.02393629084357885,
+            0.017993366287762866,
+            -0.0263476663180971,
+            0.01864011499179424,
+            -0.02332263152537849,
+            0.016014908881756034,
+            0.01868464825773604,
+            -0.003104041044182249,
+            0.026313491086684425,
+            -0.0052945248919367815,
+            0.015253731232325074,
+            0.0012371308759955573,
+            -0.02119765517013431,
+            0.007795466385020642,
+            -0.006469500272298062,
+            0.025762169721880456,
+            0.00833992768202466,
+            0.01801901639978976,
+            0.022019940825351116,
+            -0.0409522298213741,
+            0.02003698451081499,
+            -0.001367916864679543,
+            0.01782984895971748,
+            0.0038516134262842373,
+            -0.007885432294782304,
+            0.014221728513400065,
+            0.010991352188016265,
+            -0.012851099261924757,
+            0.003840539873231741,
+            0.0008923482291826014,
+            -0.028928104542716937,
+            0.007066004768771198,
+            -0.026914574065060926,
+            0.025866907400807087,
+            -0.005526774784152086,
+            -0.00998122210956923,
+            -0.020744532673097317,
+            -0.0071425207262770865,
+            0.003754468896273156,
+            -0.012747874997429746,
+            -0.011385805601999295,
+            -0.024700457669046855,
+            -0.02401291382882952,
+            0.029591582802488962,
+            -0.014054332096344003,
+            -0.04015978298687865,
+            0.001234440863385445,
+            -0.023301255402129767,
+            0.01193125007773239,
+            -0.005793124129267758,
+            -0.024440100649464773,
+            -0.042563399082257154,
+            -0.017548888269389457,
+            -0.0009394887961485922,
+            0.022739763644266633,
+            -0.00114156949394707,
+            0.041791395064755515,
+            0.0008816696205437911,
+            0.022207132857533028,
+            -0.015547580543606755,
+            0.005319731264840819,
+            -0.02071229859314312,
+            -0.041495579742912285,
+            0.03660432378985971,
+            -0.014637031447181937,
+            0.025854844497435178,
+            0.03324325296704186,
+            -0.013009347371118843,
+            0.001354504885302625,
+            0.02652570709474851,
+            -0.014618100923440427,
+            -0.010782232747863238,
+            -0.0034936699515378212,
+            0.003461670978569965,
+            -0.006468116285021298,
+            -0.027793221545159466,
+            -0.007752458229193391,
+            0.04193706715422074,
+            0.017894069367616637,
+            0.021928994684423043,
+            0.022606401934291136,
+            -0.017972404868085358,
+            -0.026402048654300563,
+            0.003656646576244765,
+            -0.013129722990153284,
+            -0.018090360146368677,
+            -0.01624598856366761,
+            0.012397150789705746,
+            0.011741002980073786,
+            -0.004052422390675261,
+            -0.01836750423615859,
+            -0.020929335029248472,
+            -0.016327412417122416,
+            -0.0042615345226698015,
+            -0.008458117095535,
+            -0.005002582649405199,
+            -0.012158173174049797,
+            -0.020409616620022143,
+            0.03706368540614453,
+            0.0285565642978937,
+            -0.0041732933884846566,
+            -0.00607154864704767,
+            -0.011142150176334957,
+            0.008834431806990914,
+            0.006021232046087652,
+            -0.005028870471297621,
+            -2.3396043292139426e-05,
+            0.005112033372713154,
+            0.022822312435625314,
+            0.007271235299348001,
+            0.0035791789459533856,
+            -0.06340305811365254,
+            -0.005392000517722054,
+            0.022560201096078862,
+            0.012581401789917604,
+            -0.0032248219865223496,
+            0.025180718735801837,
+            0.0006451724964970927,
+            -0.009714514747463593,
+            -0.018389422219956937,
+            0.024728862150150004,
+            0.008760329201960964,
+            0.020988700992999768,
+            -0.014974763313686516,
+            0.003845297486923296,
+            0.0006445983847138896,
+            0.00750394530065325,
+            -0.0019792992847026793,
+            -0.008578254004439607,
+            0.025776677078006516,
+            0.016593438298233645,
+            -0.005695578040595484,
+            0.01604018433477423,
+            -0.0017497260488899657,
+            -0.021460543443428924,
+            -0.010265213095609554,
+            0.00042017209251926397,
+            -0.010570864288876387,
+            -0.018015790888295944,
+            0.017569414447505878,
+            -0.002601036282535863,
+            0.0072399377790164435,
+            0.030281707151249355,
+            0.0423183067569767,
+            0.009566493398534713,
+            -0.005928603293907002,
+            -0.0037936911251953654,
+            0.015477873304924939,
+            0.01513127995244932,
+            0.010741256034652754,
+            0.03015858543221727,
+            -0.029247492875021843,
+            -0.006690306960497428,
+            0.0046653727410352576,
+            0.004610405633429778,
+            -0.011692179099382171,
+            0.030055897497741527,
+            0.04569730592994177,
+            0.006736813244286893,
+            -0.004799978513117371,
+            0.002121067028052434,
+            0.011299889107780231,
+            0.007768137218140627,
+            -0.018927080714961783,
+            -0.0012222962555201746,
+            -0.006009911026251445,
+            -0.014001766533604134,
+            0.026941585318261085,
+            -0.003960837039351648,
+            -0.011848232223811548,
+            0.02903357274500332,
+            0.1105267435019781,
+            -0.006011029338781496,
+            -0.0053646998281195206,
+            -0.03242321704621517,
+            -0.015194102811332289,
+            -0.008860590919147765,
+            0.0056988205877963076,
+            0.01635957585050017,
+            0.020327661913559888,
+            -0.0008943408274661007,
+            0.016820860249955754,
+            0.02822175614376368,
+            -0.013854827839129885,
+            0.007906244895327178,
+            -0.030891381690403466,
+            0.015702117924362928,
+            -0.005383974926093977,
+            -0.012032955400313954,
+            -0.0026169511828034054,
+            -0.008618307035330928,
+            -0.0037356680273096105,
+            0.0009666430093455632,
+            0.011296307527684261,
+            0.004427516814276546,
+            -0.014484198313369681,
+            0.002829529156309784,
+            -0.008595300985863424,
+            -0.005122996529848575,
+            -0.02393571073523279,
+            -0.024119245543788293,
+            0.005223147047205755,
+            0.0019389305548629142,
+            6.344308292376579e-05,
+            0.006848965054539454,
+            -0.003810712095343454,
+            0.030149081317333418,
+            -0.01700322008272963,
+            0.0012288912810468446,
+            0.01193226302726573,
+            0.012332749933413344,
+            0.01688602002694633,
+            0.0028600984823761366,
+            -1.9882692552530084e-06,
+            0.021172977047420227,
+            -0.010514036509402561,
+            0.01460334896316005,
+            0.013091417303145243,
+            -0.04103572381458182,
+            0.0089746401463863,
+            -0.0027841289484449873,
+            -0.024252447218794344,
+            0.004446472209246425,
+            -0.018987867343500102,
+            -0.00881974688349601,
+            0.0005365625696166774,
+            0.012054719659248512,
+            0.01960300421041832,
+            -0.008553943699846696,
+            0.02111201386480852,
+            -0.0003501657740808554,
+            0.004177088753111459,
+            -0.012879944670340026,
+            0.0065222126975051685,
+            -0.007563354540402343,
+            0.0008168128607271199,
+            -0.018213241937597355,
+            0.012526561612175514,
+            -0.0008285725179894544,
+            -0.015737003159436623,
+            -0.011337688275309921,
+            0.004737346137976519,
+            -0.0027993545140114705,
+            0.0030843163114700313,
+            0.010153740340626154,
+            -0.008815302484351509,
+            0.004919275910399053,
+            -0.0017541042538165297,
+            0.002562816687714991,
+            0.014285249186421434,
+            -0.029816964593980453,
+            0.005357227738099325,
+            0.009154842495811638,
+            -0.00048782303639333943,
+            -0.013074976152495576,
+            -0.004054505559072866,
+            0.0028322102449485015,
+            -0.016877328079971877,
+            -0.012128199639779175,
+            0.023421330164419888,
+            -0.012565145848820333,
+            0.008609070188894315,
+            -0.010936598722317274,
+            -0.018401801734378692,
+            -0.01066119413133089,
+            -0.005254229780560808,
+            0.01234969508270852,
+            -0.0038427767204741877,
+            -0.006947594635952202,
+            -0.01756894543880097,
+            -0.007890897373987378,
+            -0.047246698568222466,
+            0.009617876521097624,
+            0.027157555263862723,
+            -0.011355758309667893,
+            -0.04050311381480088,
+            -0.01017774338301804,
+            0.02437257208967103,
+            -0.004056329295095018,
+            -0.008345450168215274,
+            -0.001599171409135998,
+            0.018358386463243014,
+            0.0027207445181251406,
+            -0.007292967749349572,
+            -0.011267861428599221,
+            0.01657507035043879,
+            0.015105061280816214,
+            0.021419854579631276,
+            -0.02088952706825817,
+            0.006295984081511286,
+            0.020973020776936996,
+            -0.03073246543555915,
+            0.0023720435876790876,
+            0.03363966216709148,
+            0.010412588613998134,
+            0.013023311559810034,
+            -0.0010217662933572685,
+            -0.0075567381078977215,
+            -0.008248818794295818,
+            0.011848200323549865,
+            0.005066463524567732,
+            -0.023174489297821385,
+            0.021823825355858556,
+            0.020368868756448753,
+            0.0026042856858509524,
+            0.003457067317891926,
+            0.0005539745115410357,
+            0.01062670844108589,
+            0.009948889803890108,
+            0.05100741556350918,
+            -0.006493714124814164,
+            -0.014021751055930108,
+            0.015063988102064609,
+            0.02488059237697814,
+            -0.0023499941119596228,
+            0.01087282522970853,
+            -0.02612218935931224,
+            -0.021423880914461062,
+            0.023389294741183817,
+            0.008967763132667892,
+            0.009637806238448432,
+            -0.010815706635624337,
+            -0.011390999462441815,
+            0.031750675158698435,
+            -0.01307829462211914,
+            0.00759749911213318,
+            -0.02169895957209634,
+            0.01387879409130081,
+            -0.008606362268863441,
+            0.032229658009209815,
+            -0.0009066257436946125,
+            0.011062436424260079,
+            0.028010486866468023,
+            -0.007835805879241447,
+            -0.019657486763625547,
+            0.003967480939987674,
+            -0.015577952617624695,
+            0.02188429826317828,
+            -0.020502758155233298,
+            -0.0002362500702658591,
+            0.0040935406015014265,
+            -0.010056462958039452,
+            -0.0057038083589231615,
+            0.011475170331848666,
+            0.0015345076181482267,
+            0.006359262977587827,
+            -0.020293014401106,
+            0.01369706440475074,
+            -0.003989482215141903,
+            0.007887824197823247,
+            -0.015925992045349828,
+            0.014370223626942766,
+            -0.02309139937510304,
+            -0.021808736033141787,
+            0.008142621179624918,
+            0.0108996933426044,
+            -0.007589050511458233,
+            -0.025971539510030145,
+            0.02250992080022228,
+            -0.024980833805349545,
+            0.024979008145354243,
+            -0.006622339940663538,
+            0.002566378488088469,
+            0.004913874740363343,
+            -0.008102103508732738,
+            -0.0028098905427027783,
+            0.0057865587191235565,
+            -0.0073509382400023995,
+            -0.014123756810537343,
+            -0.019453380788786282,
+            -0.04083629135586275,
+            -0.03177927047231389,
+            0.022927581786088044,
+            0.011557232599801421,
+            0.0017073832658476193,
+            0.02339552660545296,
+            -0.02247471449131386,
+            -0.017890866287500053,
+            -0.009775068417716798,
+            0.025184899522229628,
+            -0.04408541669037361,
+            0.012104255392615549,
+            -0.014669458760022245,
+            -0.0245258242481677,
+            0.006274252098230249,
+            -0.004161960147481513,
+            0.009299171724026455,
+            -0.010779421730862708,
+            -0.04713627671791054,
+            -0.005192684784377356,
+            0.02330975631183273,
+            -0.0036802876364168935,
+            -0.006197548344287773,
+            0.03048284650641829,
+            -0.0019503797069087845,
+            -0.005038254331399784,
+            -0.005524875459245342,
+            0.027783596017224226,
+            -0.024950083604372984,
+            0.010988398531346805,
+            0.008304937912856357,
+            -0.008485065895193593,
+            0.006572665697403556,
+            0.0011518846966102158,
+            0.011927348472282199,
+            0.015632588644277183,
+            -0.015559413716826264,
+            0.02262990861928957,
+            0.011446850286896198,
+            0.006506075777765303,
+            0.008015225053785824,
+            0.00010054430757843335,
+            0.0022975742216194894,
+            -0.024948925449668525,
+            0.026047924967851216,
+            0.01888211557921096,
+            0.0001632181017876392,
+            0.012553094476616919,
+            -0.015042621225242777,
+            -0.02782009986290425,
+            0.02606646297992026,
+            -0.016847653820092335,
+            -0.00045272379465939274,
+            0.020053091486344633,
+            -0.016210036358455773,
+            -0.007930550661303182,
+            -0.01123514849773852,
+            -0.0162675150307978,
+            0.01803076559921804,
+            -0.012160217543572069,
+            -0.02216594536057653,
+            0.002594675636753233,
+            0.016443123828746582,
+            0.02134480656850119,
+            0.026694575953313482,
+            -0.011470930697351533,
+            0.017768927048191052,
+            0.0008653408391607816,
+            0.001865765599795092,
+            0.025745954171649543,
+            0.01702677011968492,
+            -0.008474889109168202,
+            0.025347176686722313,
+            -0.0016686223698600755,
+            -0.002421921060670558,
+            -0.012265504084322075,
+            -0.0165214436458303,
+            -0.007545471138114208,
+            -0.015333235556146196,
+            -0.0051830106590765685,
+            0.000762746782664525,
+            -0.0018118022696726346,
+            0.10876576838038368,
+            0.0034948044540582626,
+            -0.011608460689239234,
+            -0.005870263500417206,
+            0.007510774308250375,
+            0.0023525201199670765,
+            -0.006846154764389575,
+            0.016082721103476375,
+            -0.018616626204086575,
+            -0.023593744079394,
+            0.004743481974389998,
+            -0.0009262286319469741,
+            0.00870122878944883,
+            0.02029523251993269,
+            -0.008608837818169835,
+            -0.0200567711498164,
+            0.007089477722191928,
+            0.005535149644041221,
+            0.0023640746516071774,
+            0.005256431824732927,
+            -0.0006118671321824933,
+            0.008209920240361308,
+            0.02448422122961942,
+            -0.015622892878860465,
+            -0.029770836607612903,
+            0.010535371601298532,
+            -0.00015994156470228762,
+            -0.007193425789067776,
+            -0.011046316463797128,
+            0.014728841492009627,
+            -0.0011488451990980268,
+            -0.02751404133172324,
+            0.017663132712536528,
+            -0.019055335178900415,
+            0.014643246200933506,
+            0.0025348345163347464,
+            -0.00016387173410118788,
+            0.022565953069005714,
+            0.013101158251834834,
+            -0.04272085076882881,
+            -0.0034442020309914823,
+            0.01041205537746844,
+            0.011059422927946552,
+            0.0116446016862942,
+            0.0012992535381387887,
+            -0.018670801241345514,
+            -0.011788984490283379,
+            -0.009109849103208542,
+            -0.021268488414296117,
+            -0.012523033785938503,
+            -0.011393853408628214,
+            0.010743646386439633,
+            0.0012964030814325153,
+            0.023809566099779113,
+            -0.003030535542277875,
+            -0.006526351867141901,
+            -0.000430699447948508,
+            0.00880561288237257,
+            0.009694646009259077,
+            -0.02079011138439714,
+            0.008294338719512953,
+            -0.01823922789220214,
+            0.011819769685519291,
+            0.006416044253693268,
+            -0.022799568832306067,
+            -0.009911619276012527,
+            0.007152786695893951,
+            0.008393677195358034,
+            -0.013569953300860946,
+            -0.006503988731872938,
+            0.02299498754558268,
+            0.018870619152710703,
+            0.012233964644738905,
+            0.024994616203196145,
+            0.03838969722235787,
+            0.005744255912479976,
+            0.027682123067526515,
+            -0.016791471092091604,
+            0.008459745383680297,
+            -0.022493868174622695,
+            0.012902262638928338,
+            -0.00460399519266689,
+            -0.0050431343566427886,
+            -0.018188192498411616,
+            0.033765105245412236,
+            -0.009159751849801773,
+            -0.0027150644587327763,
+            0.034778224828120714,
+            -0.0066502021793471075,
+            0.004525230911930458,
+            0.005096748531775903,
+            -0.016067049494073893,
+            0.033274672789154185,
+            0.01686831040872425,
+            0.016353966592947262,
+            -0.021631941883929817,
+            0.011748317746828848,
+            -0.005675701689448899,
+            0.0001273442591116645,
+            -0.020738819037390263,
+            -0.0037528323797385733,
+            0.0027725541343763104,
+            0.0059860509282538546,
+            -0.016556134518370437,
+            0.02033623476757896,
+            0.00019647023567641008,
+            0.0052895744489259875,
+            0.03253462247073863,
+            0.01043311189804924,
+            0.019182938878855655,
+            0.010051051068017807,
+            0.019523739618278902,
+            0.0281939391326836,
+            0.025685787482623124,
+            0.0029932688462254147,
+            -0.018668945726139946,
+            -0.0009326901784812825,
+            -0.007880352863625262,
+            0.010492063055849727,
+            0.022849902214818293,
+            -0.022619228461640748,
+            -0.02772644874135741,
+            0.018566981184526675,
+            -0.0022608185345048414,
+            0.020031103834122595,
+            0.007544464625538984,
+            -0.020935181884939796,
+            0.003407355515103286,
+            -0.023178230166337178,
+            -0.0005418155871620593,
+            -0.031039187720144967,
+            -0.028591174077243604,
+            -0.0007659241904525392,
+            0.011945344534674635,
+            -0.01688153017778311,
+            0.03357788163306365,
+            0.017667778671101954,
+            -0.014289109343082812,
+            0.01604684528817524,
+            -0.011054587325539086,
+            -0.0036959091138078683,
+            -0.005819853870004682,
+            -0.00203755010992417,
+            -0.02157944474979592,
+            0.005138614763345542,
+            -0.0033509138150859593,
+            -0.0024871060745482883,
+            -0.016936575573598154,
+            -0.021901830366872595,
+            0.016375173843894617,
+            0.016742770709641697,
+            -0.006606252542838758,
+            0.039771905404651496,
+            -0.004238071340064403,
+            0.00018364315349545657,
+            0.011690521707403059,
+            0.0010018392078563795,
+            -0.013651921068215563,
+            -0.010829994512513501,
+            0.012362647379788833,
+            0.011321403447882543,
+            -0.018942043512252663,
+            0.005763773608775982,
+            -0.009131565038297889,
+            0.002089216671850269,
+            0.005438887787329238,
+            -0.008183835275629478,
+            0.006690280329947951,
+            -0.01673229763262922,
+            0.015833397876364723,
+            0.00295474085300386,
+            0.003186554859191664,
+            -0.011797859525926772,
+            -0.020991927615680524,
+            0.01576413532776261,
+            0.0032403514518527287,
+            0.018819824905201635,
+            -0.026755218746060662,
+            -0.00012946467986739473,
+            -0.009552681500884924,
+            -0.0028941357661144826,
+            -0.012533800273548772,
+            0.013975640628890588,
+            0.016864739319953755,
+            0.0063941926578529,
+            0.01781016436903586,
+            -0.0026268224264609724,
+            0.015368284773773678,
+            -0.018122416427266218,
+            0.008686009180697955,
+            -0.006801074032721299,
+            -0.018537521789181048,
+            0.0013237338264105415,
+            0.027863741654435434,
+            -0.004670012662973029,
+            0.011066853115455966,
+            0.011704884649534624,
+            0.018954056680143452,
+            -0.0012626215734593707,
+            -0.0030850559460101866,
+            0.015610404665171367,
+            -0.005838226405658572,
+            -0.006861883833171456,
+            0.020872966233916948,
+            0.032951653763679145,
+            -0.006716215648910268,
+            -0.0021292599951407863,
+            0.010886596327481677,
+            -0.004901684487088035,
+            -0.008644584974417106,
+            -0.009587037829603014,
+            -0.03158137900717491,
+            0.024369089446706894,
+            0.012092412439477474,
+            -0.009601042951586109,
+            0.022690862146548786,
+            0.020752792552278052,
+            0.011412743968296708,
+            -0.010471860708195726,
+            0.02291536038589634,
+            -0.01268035142112569,
+            0.024168869865226136,
+            0.032482831775271157,
+            0.004400973556358811,
+            0.0022312906286559167,
+            0.0021175489661843985,
+            0.01691348460546639,
+            0.006050357346231155,
+            -0.007068133214675996,
+            0.011656003476409986,
+            -0.018487923180009552,
+            0.01598530891381001,
+            -0.004432366596808033,
+            0.02808371007118948,
+            -0.01337145216499331,
+            -0.010473700033368562,
+            -0.00537325376270318,
+            0.00010726431372328729,
+            0.006445701087575359,
+            -0.003324002570256215,
+            0.005255443653813165,
+            0.0007072577303252132,
+            -0.017825649146944237,
+            0.003050128130393781,
+            0.004825967170196209,
+            0.009018799597337101,
+            0.004103653497834134,
+            0.007249957401824204,
+            -0.012360355929483,
+            0.01441885664789707,
+            0.004401820363035808,
+            -0.0026190938717927086,
+            -0.0007163225760398784,
+            0.011124538295057815,
+            -0.013232822471919533,
+            -0.004895624967029031,
+            0.005281228035321008,
+            0.0059275753353475414,
+            -0.007507924998741734,
+            -0.004484140989405051,
+            0.011908076549392969,
+            -0.0037917291321648483,
+            0.006351543504726163,
+            -0.004409464890606014,
+            -0.0037760571637986827,
+            0.005303661313434895,
+            -0.00974511544836468,
+            -0.004204167898342997,
+            -0.0007093125445559502,
+            -0.006869250680233214,
+            0.00786202375969793,
+            0.005971033227117194,
+            0.010606085272785863,
+            0.006969437776542303,
+            0.004293129058563288,
+            0.004114452739123398,
+            0.004071017374828283,
+            -0.002445681623750424,
+            -0.00592480769363354,
+            -0.004617501328052899,
+            -0.012067458701076778,
+            -0.008335239153829813,
+            0.009025028998949398,
+            0.008653779421856954,
+            -0.019363018792788574,
+            0.02048011578221327,
+            -0.001188320960771659,
+            0.00772165044232104,
+            -0.01602086976290928,
+            -0.000907553574811279,
+            0.005464880629734941,
+            0.008267702703609528,
+            0.016136762227127184,
+            -0.03654209526492064,
+            0.005179791231187316,
+            -0.0037321883493524766,
+            0.016026013709955444,
+            -0.008215703411373182,
+            -0.006979816286354862,
+            0.02564060526656963,
+            -0.01450763388985563,
+            -0.0009847758228920171,
+            -0.016142304341196283,
+            -0.01407778890808498,
+            0.010815725252959506,
+            0.02050940280233426,
+            0.020684808835848383,
+            -0.0004965802897137407,
+            -0.0005781445948759235,
+            0.00036179841583004193,
+            0.022775804864342608,
+            0.015545681427566235,
+            0.018697154644607316,
+            -0.004543500782944068,
+            0.007028436664826706,
+            -0.023057215501463973,
+            0.009912581556496658,
+            0.0011795417129232596,
+            -0.008215027850908477,
+            -0.008554137076267468,
+            -0.008703378564520125,
+            -0.010448538943182258,
+            -0.011773811327190638,
+            0.004106616839960258,
+            0.013895013678116032,
+            0.004190549543250594,
+            0.0031571190600981074,
+            -0.0225385788570314,
+            -0.006243193914542516,
+            -0.011229641377928767,
+            -0.01674214043453308,
+            -0.007367389908398082,
+            -0.02386334969150212,
+            -0.028021339684039734,
+            0.0002794207370355175,
+            -0.02411861258809571,
+            -0.015958336406924612,
+            0.005268815864773021,
+            -0.01546610225139129,
+            0.00898600137695074,
+            -0.008396481690803911,
+            -0.011873486990732417,
+            0.0074743033679134105,
+            0.015336444788692,
+            -0.017676977079590463,
+            -0.010165317702640217,
+            -0.01690524990506033,
+            -0.013460451917691695,
+            0.010918550283871637,
+            0.00595960977280955,
+            0.009250157926438213,
+            -0.012447979337850305,
+            0.0062932435120863365,
+            -0.012673905639312696,
+            0.015073013552721701,
+            -0.03769166067890086,
+            -0.029992249463405965,
+            -0.00014990754540999537,
+            0.007288036630677436,
+            -0.0020735742042505426,
+            -0.018695120921334806,
+            0.0003860116505754868,
+            -0.003608995261360232,
+            -0.014900191335144973,
+            -0.013866216738266176,
+            0.025251475305524053,
+            -0.006788483107616133,
+            0.01223227424986397,
+            0.0034937277040015626,
+            0.03438333560343413,
+            0.008803852645562129,
+            0.014666206816302367,
+            -0.005709421955160623,
+            0.003742433643393657,
+            -0.0033706258426092744,
+            0.006906464031352315,
+            0.0076578888497945185,
+            0.009276592191710524,
+            0.008025321004305927,
+            0.004576470294889512,
+            0.002887346543823031,
+            -0.001925468214331082,
+            0.02125241545487307,
+            0.025671028960818562,
+            -0.00037467906648160044,
+            0.008981063192139038,
+            0.0009387900084090577,
+            0.0002735789978680719,
+            -0.008374290340790827,
+            0.006547013284767913,
+            -0.008274637814401893,
+            0.004041066530254948,
+            0.012897549078838394,
+            -0.009150863940242148,
+            -0.010211568689323908,
+            0.016575088843292566,
+            -0.012380099999574783,
+            -0.03420455578618604,
+            0.026550654680719987,
+            0.005324512466403649,
+            -0.0035670106106754245,
+            -0.010657928468872608,
+            0.0037176027853158167,
+            -0.01103891334887315,
+            0.008096915671982816,
+            0.022587659771132487,
+            -0.0013527917944322358,
+            -0.0018085171207746743,
+            0.0038503794877980586,
+            -0.011870484733648887,
+            -0.007881489342351292,
+            0.006462772034381968,
+            0.0315696514747912,
+            0.01326417653779577,
+            -0.020400369539140532,
+            -0.015223808971473513,
+            0.020406862349006738,
+            -0.023142176712111575,
+            -0.006254959806032159,
+            0.009039506093607052,
+            0.0009087520492922078,
+            -0.022766046838415165,
+            0.0009662037093312435,
+            0.004701966191595815,
+            0.005946458676746661,
+            -0.006358986495019584,
+            0.03237532341275415,
+            0.001075992257265679,
+            0.015892020271402903,
+            -0.017189453434945788,
+            0.0017514632185789299,
+            -0.014189278671121677,
+            -0.014265447388465805,
+            -0.029809460132405256,
+            0.0005557491671071693,
+            -0.0030324198417091704,
+            0.011038984948790744,
+            0.024808067315720335,
+            0.01145103533194245,
+            0.014432246996007246,
+            0.010552852947578788,
+            0.01513689173494211,
+            -0.013653722180565311,
+            -0.029009757850432732,
+            -0.005171522291984226,
+            -0.018001048743500243,
+            -0.02297482628212251,
+            0.003962613844063587,
+            0.013645125285548414,
+            0.006278662690346082,
+            0.023786381386575035,
+            0.0021738516691483163,
+            0.0009580591750940011,
+            -2.690805459518147e-05,
+            0.008894091070322921,
+            -0.004952203441302471,
+            -0.006553986197076102,
+            0.003570189040872711,
+            -0.000652714137223501,
+            0.021228360881461853,
+            -0.005758946428212041,
+            7.897153237591187e-05,
+            0.014242175492137011,
+            -0.005622256348934008,
+            -0.0035009192529791146,
+            -0.017339926143652377,
+            0.006248842389703505,
+            -0.01622011586976866,
+            0.025783086900871728,
+            0.018450783573486916,
+            -0.009109295293546267,
+            -0.013469521573468915,
+            -0.011259555201313116,
+            0.02773097452736661,
+            0.004631282554715908,
+            0.010787055220617197,
+            0.018228322680582464,
+            0.01461254259249465,
+            0.01784830193964943,
+            0.004664725683522323,
+            -0.0034339781146649384,
+            -0.0007692550982545399,
+            0.021176646860433974,
+            -0.014700178590249292,
+            -0.002701208084390198,
+            -0.008842788249773623,
+            0.0021799005046816444,
+            -0.008431299000066363,
+            0.006086447848362669,
+            -0.010793016137040172,
+            -0.015251381840165954,
+            0.0189503457652375,
+            -0.02660767353735015,
+            0.016423753600066336,
+            -0.00593016170680532,
+            -0.00482891500342298,
+            -0.027459214027897378,
+            0.010728039528784636,
+            0.0027866316191608297,
+            0.0022519974195000687,
+            0.010770472550461752,
+            -0.021427634123401408,
+            -0.01867717541189363,
+            0.008532444554010673,
+            -0.011457117350255678,
+            -0.007152301670244725,
+            -0.00348619742388509,
+            0.0154715607296319,
+            -0.00017898955187849443,
+            -0.009925365923862246,
+            0.02085093731060963,
+            -0.007175309371250024,
+            0.01223979370120031,
+            0.005256127464980655,
+            -0.008756077893734508,
+            -0.0028438320920811362,
+            0.021022708898898203,
+            0.027557382862284808,
+            -0.008342206702778956,
+            0.014526066580760486,
+            -0.011883797998224506,
+            -0.01088335352319397,
+            -0.03479444009382274,
+            0.030104838505937447,
+            -0.013305732257226255,
+            0.026773747582411633,
+            -0.0006968265979630097,
+            -0.017037729111584094,
+            -0.016206207453228583,
+            0.01625225876515654,
+            -0.0024290584023212254,
+            -0.009366287391027166,
+            0.01465885301870365,
+            -0.02428628495473643,
+            0.0058336571592899735,
+            -0.0006627935593670669,
+            -0.011531762838191602,
+            0.013275508677482622,
+            0.0019223252930710324,
+            0.010738622785461513,
+            -0.02030405805119001,
+            0.0009661595529028194,
+            0.009263716373456188,
+            -0.016791855691574455,
+            0.006057830565306032,
+            -0.03730952786847912,
+            0.0029341721588188943,
+            -0.014833895684348656,
+            0.005725615593914644,
+            0.001498755253963699,
+            -0.01820708352937343,
+            -0.03652692232407375,
+            0.00043665701245599497,
+            0.01913883585802903,
+            -0.007975988799790905,
+            0.020538595326175857,
+            0.008243569241768276,
+            0.008421404577931517,
+            0.002314738067443065,
+            0.0016702403734608475,
+            0.008279538988821465,
+            0.03679456861213399,
+            -0.01775927634148114,
+            -0.004836606118958108,
+            -0.0012151329859941935,
+            0.010644195708319258,
+            -0.02028470571116305,
+            -0.009287515726906219,
+            -0.0036506912311261536,
+            -0.013134467570201307,
+            0.030615862647585222,
+            -0.020265555441523482,
+            0.025634449009756734,
+            -0.009300770899023007,
+            0.009274072905190077,
+            -0.02097685009125915,
+            -0.009797008690754956,
+            -0.010661540100131183,
+            0.0019532291898233895,
+            0.013465005282849381,
+            -0.009151430285942593,
+            0.0071467934925125145,
+            -0.006655608635025676,
+            0.01020327561163374,
+            -0.001472597695707201,
+            -0.02443508800166349,
+            -0.0024998811551575563,
+            -0.01107321258077183,
+            0.02456164421154801,
+            -0.0019619696941303388,
+            0.0005043179699158694,
+            0.010951580163871948,
+            -0.00853593956432799,
+            0.023927336608555987,
+            -0.008855702280766362,
+            0.0010093679746665581,
+            -0.004863269000020324,
+            -0.004005691415117686,
+            0.001698212044518236,
+            0.01822024022650613,
+            -0.005707404715849356,
+            0.03722587302228139,
+            -0.01708925706528118,
+            -0.0065500772743641925,
+            -0.00015538686795361824,
+            0.010852925330001283,
+            0.010035359503147549,
+            0.008754719437760327,
+            0.004067820167375348,
+            -0.014861417626433965,
+            0.009346669946626484,
+            -0.020136029447262147,
+            0.007501669351583119,
+            0.023178490169347605,
+            -0.006840359541885465,
+            0.0006676450965357345,
+            -0.01283320739092204,
+            -0.015610538827742276,
+            0.006308543164022682,
+            0.002329420005586303,
+            -0.002295739637504597,
+            -0.02181189272979859,
+            0.006298804075737111,
+            -0.014398470126723037,
+            0.004007419942728446,
+            -0.0007497656993578918,
+            0.013863867106289847,
+            0.016451166273826513,
+            -0.006740209707090658,
+            0.039292260347522444,
+            0.01871723939316731,
+            0.009109005310505462,
+            0.005674782595856756,
+            -0.040588444759736254,
+            -0.008993850998154852,
+            -0.0010197185016393919,
+            0.004866150984083949,
+            -0.01141354216871663,
+            -0.018944881712855702,
+            -0.002646110151132781,
+            0.000308585036997592,
+            -0.010681954707313127,
+            -0.009091585604191968,
+            0.01751599002710486,
+            -0.006034424764302979,
+            0.00872057856926618,
+            0.004146961781109887,
+            0.023758065635807138,
+            -0.009727802672285544,
+            0.01176257119851645,
+            -0.0071822083765114165,
+            -0.005456197751568526,
+            0.0006726765703478756,
+            0.011011628907288813,
+            -0.008031792262886802,
+            0.009763340867550022,
+            -0.004279891138185881,
+            0.008355720233269893,
+            -0.008552596384067825,
+            -0.023952129458400563,
+            -0.014972827087541732,
+            0.014347535404251018,
+            0.011651260496709961,
+            0.008137738462704715,
+            -0.013624435569539285,
+            0.0031728721674035352,
+            0.00675830277744976,
+            -0.009184784787581942,
+            0.009161771311910335,
+            0.017733082848622708,
+            -0.001411542568472906,
+            -0.0017830167499636113,
+            -0.0016130003349297027,
+            -0.012173528776899308,
+            0.0011196437753239162,
+            0.02284887358189995,
+            -0.003054479643937998,
+            0.014614283913976716,
+            0.010689486748036806,
+            0.009643292262102645,
+            -0.00810930837603355,
+            0.001915955511174678,
+            0.02341863048264366,
+            0.015022291783105098,
+            -0.004625715884342598,
+            -0.011422973894623295,
+            0.004259757717597968,
+            -0.010332744704689279,
+            0.007373328473086509,
+            -0.02117940113144938,
+            -0.004672698999328004,
+            0.008896311380763048,
+            -0.007028862917603513,
+            -0.01732432285099198,
+            -0.028026173555055017,
+            -0.016765805892242715,
+            0.019819329380181,
+            0.0027213760678977965,
+            -0.013291251876048186,
+            -0.0032919084721662363,
+            0.0031214883337300926,
+            0.002553005168048115,
+            -0.0071104806975682885,
+            0.005310583052470149,
+            0.007359183453835734,
+            -0.008674133980486269,
+            0.017665958608281126,
+            -0.000521269431423141,
+            -0.0008020251454616109,
+            -0.0005442374261144659,
+            0.0006127687598711624,
+            -0.016387752806126442,
+            0.01666161912459431,
+            -0.010949228376419892,
+            0.015198110119816624,
+            -0.008966726991404666,
+            0.008600499511690635,
+            -0.007617043351740469,
+            0.009458641139070436,
+            0.011597957400445055,
+            0.012755037944648426,
+            -0.004403680820063604,
+            0.008519241473418196,
+            0.004807816191718704,
+            -0.008147125240985166,
+            0.007017601015398136,
+            0.01296082929962557,
+            0.00582353932160707,
+            0.0013634237970372698,
+            -0.0008246894322793264,
+            0.0022035670958705576,
+            -0.002480976219285683,
+            0.0056293125704658135,
+            0.012714850523393719,
+            -0.002497639449055006,
+            -0.007135683201188999,
+            0.014988249771676699,
+            0.014955895630183352,
+            -0.022994948923833993,
+            0.007121634434082677,
+            0.005835971825639792,
+            -0.0030507120821557008,
+            0.002145266953387315,
+            -0.027334525318320587,
+            -0.020359474070022213,
+            0.01901995970350808,
+            -0.015260454474605018,
+            0.02850460517419317,
+            -0.011624138792780823,
+            -0.02284242513728605,
+            -0.01277489645734181,
+            -0.004209529633534193,
+            -0.022684500849916897,
+            -0.004894414098943742,
+            0.01919165657366256,
+            0.013401560972810082,
+            -0.004313549020265677,
+            -0.019089369171102138,
+            0.03863248408555474,
+            0.010497951840980463,
+            0.03508157919687289,
+            -0.008597970944278045,
+            0.0023354831464448313,
+            -0.01563401808449416,
+            0.015551176593436512,
+            0.001075882802133782,
+            -0.011575692834796734,
+            0.01505716108583101,
+            0.031836226915668404,
+            0.0034740889237207494,
+            -0.00617206855371193,
+            0.0018420207148235309,
+            0.00418433325800744,
+            -0.0008167766650900381,
+            0.011160989178055499,
+            -0.018497472318034093,
+            -0.011951262348399922,
+            0.0013093376768780122,
+            0.00216347903176653,
+            0.0059233460419514045,
+            -0.001778091863744448,
+            -0.013317639704511549,
+            0.011433308892643204,
+            0.011596810919469926,
+            -0.01518143105525228,
+            -0.007497373860338335,
+            0.0050957665991362405,
+            -0.018974405317819656,
+            0.01872472902494418,
+            0.011859522398789263,
+            0.000792413325332956,
+            -0.0006503947129906383,
+            0.00808635755624928,
+            0.003749813548995223,
+            0.0033657799335872368,
+            -0.007862215303617144,
+            -0.0028014669075116876,
+            0.0009892310856576799,
+            3.834379772674841e-06,
+            0.015402211364447726,
+            -0.015264893397161102,
+            0.02488855366696587,
+            -0.0025643639525691083,
+            0.00953993868921819,
+            -0.017460590918773188,
+            -0.0017911712931080155,
+            -0.00402374787529868,
+            0.01838051670627292,
+            0.006753072393532114,
+            -0.010130676216186492,
+            0.004921510943896829,
+            0.007975201081296845,
+            -0.012589786869771252,
+            0.00018319054705558588,
+            -0.0022606180909706688,
+            -0.014791528072476771,
+            -0.0031171076370692706,
+            0.017873825561816544,
+            0.007593688317551089,
+            0.0037375070591255375,
+            -0.015863143964928172,
+            0.007192324815779268,
+            0.007522990667369163,
+            0.024504875417877527,
+            -0.026309816223960274,
+            0.024779042752840836,
+            -0.022104087334088543,
+            0.02165781034613893,
+            0.0319051741666687,
+            0.007538146369110045,
+            -0.01690181186708103,
+            -0.006595947249169521,
+            0.006373991305923002,
+            -0.023410814842488123,
+            -0.005494017994983188,
+            0.016903736810723716,
+            -0.01792400152372258,
+            0.000330865061689199,
+            -0.014660413003063089,
+            0.0040825880154611895,
+            0.008248037865838212,
+            0.011540333451799686,
+            -0.0033578830043738085,
+            0.018327525495676525,
+            0.020171745352159914,
+            -0.01138759854371235,
+            -0.0004221149053107702,
+            -0.022321409200941777,
+            0.005046865873981027,
+            0.022509643273449936,
+            0.001111272240506322,
+            -0.009540467231190513,
+            0.018376261423745193,
+            0.008480532791922674,
+            -0.020801891594934313,
+            0.008436374227753589,
+            -0.036175982655998785,
+            -0.010454118523313122,
+            -0.009328507118928412,
+            -0.001218633166683384,
+            -0.004142773602181115,
+            -0.0002494371580772723,
+            -0.011365548842513265,
+            -7.699548642315061e-05,
+            -0.006889339420877534,
+            -0.0037173070609413474,
+            0.014953978004522174,
+            0.011678957032329096,
+            -0.003501047405385249,
+            -0.0073984932581048915,
+            0.008493128567102267,
+            0.006163937693564917,
+            -0.017034696006318337,
+            -0.013268418502042576,
+            0.0037122128689059085,
+            -0.020067656401141467,
+            0.005148329575092478,
+            0.00459729178556505,
+            0.014031669598917171,
+            -0.01849235806598133,
+            -0.009298127665365717,
+            0.019716512208448042,
+            0.0094466405158553,
+            -0.0008988916054855498,
+            0.004264260983102956,
+            -0.00032312548009733906,
+            0.002279987803645912,
+            -0.0008902345374346296,
+            -6.145528806973222e-06,
+            0.003602229465148649,
+            0.012503366799381277,
+            0.007759393855482671,
+            0.03694402922447051,
+            -0.016692148708641812,
+            0.005873765863350649,
+            -0.014187609702966002,
+            -0.0037592517335595654,
+            -0.00749939010396784,
+            -0.0004308157309481927,
+            -0.0022282234382577843,
+            0.0021999996319642854,
+            0.0052176773733353925,
+            0.009353171029508977,
+            0.002155405440765206,
+            -0.00823565584692709,
+            0.009691447250255694,
+            -0.015228173095533029,
+            0.003853858061067531,
+            -0.008207910291425473,
+            0.01017972550354623,
+            -0.009695280827515048,
+            0.0028705587799331246,
+            0.03849358164875786,
+            0.01130021300937487,
+            -0.004934175853966098,
+            -0.014300147534361459,
+            -0.0007796051618993811,
+            -0.01737877501558196,
+            -0.008579119859817895,
+            -0.005434792488602325,
+            -0.02617338653902575,
+            -0.00838916510137031,
+            0.015344260464413648,
+            0.011202567142310492,
+            0.014124853863170132,
+            -0.023894182185855824,
+            0.008897943489322673,
+            -0.03001670823127986,
+            -0.011951507895140876,
+            -0.001530967140156163,
+            -0.015467812535428609,
+            0.0010110999991167703,
+            0.023623196504289838,
+            -0.007762735863648054,
+            0.012654652822750325,
+            0.012867958170258448,
+            0.018777522196571934,
+            0.006390986057157016,
+            0.00038045736974183937,
+            -0.009444911783760569,
+            0.009534310165417939,
+            0.008806031651319704,
+            0.0080954543713255,
+            0.013161833638467553,
+            -0.0022840529693446553,
+            -0.018060162439023572,
+            0.005469288590000856,
+            0.014354475778235131,
+            -0.0033504276914600585,
+            -0.001615191819057537,
+            -0.011088493180661976,
+            0.016039040565630432,
+            -0.023559046333872156,
+            -0.009482398794557168,
+            0.009918576960319548,
+            -0.0027308309155029842,
+            0.009212063547988036,
+            0.0160181200081126,
+            -0.01881373001297218,
+            -0.0024168618343039703,
+            0.016244889977796682,
+            0.0024104546308670025,
+            -0.00022606425578303413,
+            0.016792244238218245,
+            -0.0007616857360728623,
+            -0.004279120886828465,
+            0.00997764266310123,
+            -0.0075631917138318655,
+            7.3878137053128405e-06,
+            -0.002416364915008295,
+            -0.0028253498168087964,
+            0.02054128727706496,
+            -0.021281801678871736,
+            -0.005379259148621041,
+            0.02257306103533458,
+            0.015687105622685975,
+            -0.007843745111084413,
+            6.258498042855072e-05,
+            -0.0219506172079994,
+            0.0019925886885244454,
+            -0.007768133999661569,
+            0.016524850592507592,
+            -0.007957949830060354,
+            0.0030085044673901665,
+            -0.01361024620290394,
+            0.011563781350927133,
+            -0.01277407169098418,
+            0.00873716215529134,
+            -0.002986943948333218,
+            -0.017567646684567492,
+            -0.0053109390854215125,
+            0.004561236136619266,
+            0.0014045175130801308,
+            0.044958041347040185,
+            -0.005539136232703934,
+            0.003729517522976541,
+            0.018086335927383804,
+            -0.014199453349727933,
+            0.02079719360289355,
+            0.016663293302592568,
+            0.00991445855219406,
+            -0.00033967370741635794,
+            -0.0018726748849858836,
+            0.023373354806639515,
+            0.013258226441642464,
+            0.00861986998150065,
+            -0.0014340218811281975,
+            -0.008516541227326329,
+            0.018239230461345617,
+            0.005458629943667832,
+            -0.012421896028975013,
+            0.012732721349758832,
+            0.013559877132508506,
+            -0.013720939472355347,
+            -0.011545210834395917,
+            0.01616358223872008,
+            -0.019904594319155373,
+            -0.018260016571763644,
+            -0.0036124537591368864,
+            -0.01614510591132948,
+            -0.03864699384205461,
+            0.01619401261888991,
+            0.018351607973242735,
+            0.006666313000123845,
+            0.013647410298896132,
+            0.002597680061427196,
+            0.004545530267244157,
+            -0.018531082430861854,
+            -0.0077618625909995275,
+            -0.008085144524135461,
+            0.003744735923004248,
+            0.011059457364156674,
+            -0.011335432904114519,
+            -0.009514379703581616,
+            -0.00034500702695418285,
+            -0.007433132060989644,
+            0.01971420270994951,
+            0.0020753610290475642,
+            0.0034506706065161024,
+            -0.00628437298309269,
+            0.017966485854173167,
+            0.026789678964152908,
+            -0.007532446825077476,
+            0.021886182869092073,
+            -0.002864230253056434,
+            -0.0015870360120997492,
+            -0.014393519359298496,
+            -0.0014439412842964876,
+            -0.011527182213472351,
+            0.0005453493437517964,
+            -0.0006791481533636673,
+            0.014093767997869645,
+            -0.03288824642147967,
+            -0.003957301296796813,
+            0.014444592778851293,
+            0.007538052312876599,
+            -0.005057755472988024,
+            0.011357919882949903,
+            -0.002964451028327082,
+            0.016977317604413924,
+            0.0075201055190542146,
+            0.023391646079994784,
+            -0.016588823248397513,
+            -0.004504858318770565,
+            0.006660877064954993,
+            0.0034631255931793638,
+            0.0014484203909961972,
+            0.01912983041187014,
+            0.023439941258581887,
+            -0.014715886385460541,
+            -0.0013596123366001458,
+            -0.0029020372473879614,
+            -0.021414221343490594,
+            0.014976806123253727,
+            0.0074729481304182865,
+            0.010967351226114886,
+            0.009806663105020302,
+            0.015465076881663691,
+            -0.02109587527354609,
+            0.0011671121739822202,
+            0.020567458365051382,
+            -0.008618333549105003,
+            -0.018815706906171112,
+            -0.009804829158883123,
+            -0.00966828714719191,
+            -0.006078893678150302,
+            -0.00803783313451146,
+            0.005523023167028105,
+            0.0022647506733877573,
+            0.01652997604669998,
+            -0.008204200434441975,
+            -0.008404609147866811,
+            0.008489247125986078,
+            0.007546785673285454,
+            -0.0033093451283570316,
+            0.010614050963656496,
+            0.0030633179859393587,
+            -0.006940540318011839,
+            0.00894884343338777,
+            -0.002787816848430143,
+            0.014347598920245487,
+            -0.02441769157218451,
+            0.004717742147185283,
+            -0.012096876825337124,
+            0.006405151160675699,
+            0.0041568790527989905,
+            0.0016702066682613428,
+            -0.0041159211459004,
+            0.0035217362856754353,
+            0.0008893479210743541,
+            -0.07433312151926533,
+            -0.012629681414004524,
+            -0.0035797012285864486,
+            0.033657838713146884,
+            -0.019695806809053914,
+            0.004477927056336216,
+            0.011966065644382396,
+            0.0096717896523898,
+            0.003070713232538593,
+            0.006136726101157844,
+            0.006112385569942848,
+            -0.0245258242481677,
+            -0.00015284418977092615,
+            0.004821040694679944,
+            0.0020347004045943454,
+            -0.007532408630122127,
+            0.00750565079228405,
+            0.015327710245879296,
+            0.020989926895388523,
+            -0.010970305087289488,
+            0.017227663358885748,
+            -0.003352718496981928,
+            -0.006068892126250978,
+            0.0014285750322333686,
+            -0.010153799722483554,
+            -0.017307415843726394,
+            -0.006495008086641125,
+            0.0068983699213297075,
+            -0.001982568363667046,
+            0.0044032687939897855,
+            0.003333596495330213,
+            -0.0032171586993948696,
+            0.0072365127663633925,
+            0.013920326463272646,
+            0.006415916101287133,
+            -0.013675562796300214,
+            0.003003782758807665,
+            -0.02500375035312058,
+            0.011945988879567646,
+            -0.012708975181025296,
+            -0.0009488161508767279,
+            -0.0047093877027523014,
+            -0.0035750724825159503,
+            -0.002213435677949806,
+            0.02043567538125188,
+            -0.014304009033474186,
+            -0.01802899775650995,
+            0.008187227768196673,
+            -0.011217013195839762,
+            0.008300740149622635,
+            -0.03382831122769974,
+            0.012386335491333264,
+            -0.0276029934382802,
+            -0.0002243752790856292,
+            -0.004521387639042107,
+            -0.0010631721066129095,
+            -0.009491184157278669,
+            -0.03274559774619223,
+            0.015845705808609598,
+            -0.007204751554079551,
+            -0.020338337341085228,
+            0.0007143424225187802,
+            -0.021267094083674304,
+            0.014150458196361147,
+            -0.01851165553224022,
+            0.0033450353123933058,
+            -0.018775439467431685,
+            0.002782289821140401,
+            -0.0018711350216089556,
+            0.010502195116579441,
+            -0.00557814442809019,
+            -0.0007130962242755068,
+            -0.013072988887382269,
+            -0.002121841212381393,
+            -0.006653444103331213,
+            0.009394907690516121,
+            0.0028678074167728367,
+            -0.009601577734902932,
+            0.011153571864462314,
+            0.013046116892519414,
+            0.019812860521973938,
+            -0.00539047787915588,
+            -0.00555139958411929,
+            -0.011553689090179406,
+            0.009852063270798534,
+            -0.016326212706634428,
+            -0.00827402913960903,
+            0.008982303148984887,
+            0.015216947130173639,
+            0.01026889105233137,
+            0.003352028462252994,
+            0.003150801750574548,
+            0.002531620731961765,
+            0.026744691368233237,
+            -0.0015120532859309638,
+            0.0008324701960729416,
+            -0.014508988496009931,
+            0.022127525293249926,
+            0.02772430884375744,
+            -0.0012206102581726523,
+            -0.00018859639594030037,
+            0.017437360169132967,
+            -0.004697052522250805,
+            -0.0038111184371524645,
+            0.024015246959315593,
+            -0.038396108183338655,
+            -0.008416825232533771,
+            -0.006788816882617399,
+            -0.006043428198944214,
+            -0.009587366247924467,
+            0.007294190273465812,
+            0.00041966080769004275,
+            0.0024709692753547823,
+            -0.0005250904505954556,
+            0.03452858797518386,
+            0.003403515654095309,
+            -0.0003392060101496309,
+            -0.0032980864446307116,
+            0.0019125121221204029,
+            0.0032987517853214346,
+            -0.009116761755779728,
+            -0.038030085122772624,
+            0.006006334000396066,
+            0.007932785488877407,
+            0.00975009749364395,
+            -0.008120121513849434,
+            -0.008864461019882246,
+            0.004909348738798099,
+            -0.02273129283865203,
+            -0.01372636307167243,
+            0.02625229144362647,
+            -0.008523028986127237,
+            0.029005877324974557,
+            -0.022435059061182174,
+            -0.013135836801818153,
+            -0.004126433322528265,
+            0.010283975013795537,
+            -0.019732856462444506,
+            0.004343860338939202,
+            -0.004022387065239562,
+            0.023733721026005954,
+            0.0077578144626961245,
+            -0.008371631446146718,
+            0.012386735614153616,
+            0.0064789380436815714,
+            -0.0026638642827013837,
+            0.00136333052963848,
+            -0.007209517067748103,
+            0.016916505654836485,
+            -0.011819784026452944,
+            -0.005833909403097991,
+            0.008977527508585844,
+            -0.0024556434204852946,
+            0.02509293948918158,
+            -0.0237762245071987,
+            0.005288171058938098,
+            -0.00019047961064753207,
+            0.01822542471604037,
+            0.008127347078975925,
+            0.008361588667386882,
+            -0.011835407590060994,
+            0.010004482616697591,
+            -0.03378107067882622,
+            -0.015433906677849963,
+            -0.007702910250795191,
+            -0.0013877702264578865,
+            -0.002059152251387601,
+            -0.015745218548968338,
+            -0.026393221735795527,
+            0.006294583740809379,
+            0.016267525575472445,
+            0.023291528020980265,
+            -0.004683197930892445,
+            0.012912830736650997,
+            -0.002831583991769044,
+            -0.008904271394710672,
+            -0.011441882849296418,
+            -0.012202629605208118,
+            -0.012470773094383122,
+            -0.014560694782340029,
+            -0.017045701603711442,
+            0.012139665577738337,
+            -0.020079061996873537,
+            0.013911518328924777,
+            0.0347210775522539,
+            -0.017176114991595492,
+            -0.0018941772168839354,
+            0.019702722555622765,
+            -0.004221469612316137,
+            0.01212088353057276,
+            0.0063707489618353725,
+            0.015298014532351866,
+            0.0038643174573634725,
+            -0.012181479717826009,
+            -0.007374005740809251,
+            0.019411457664902376,
+            -0.004902591844710158,
+            0.014591446725717338,
+            0.015069599884557399,
+            -0.008675672164385891,
+            0.002544198867387345,
+            0.01951865904496276,
+            -0.01762101777798757,
+            0.01665654707715257,
+            -0.00458402229480694,
+            -0.009936206685538294,
+            -0.013345740581684638,
+            -0.005336416724426228,
+            -0.019469457091169776,
+            0.0061608454109892425,
+            0.018931676738452464,
+            -0.00533013594306834,
+            0.009310871143727683,
+            -0.0037670793158045,
+            -0.005045151384034014,
+            0.0002041909886640482,
+            -0.04152751596227068,
+            -0.008340911336217609,
+            -0.0006651584370005978,
+            -0.02505748815737022,
+            0.025954102004571416,
+            0.007205602377144382,
+            0.022602344303400163,
+            -0.017168610245491406,
+            0.004459735690564842,
+            -0.005882024209102714,
+            0.013217517324383834,
+            0.02255689669623969,
+            0.0017984057469785134,
+            0.005699128497215882,
+            -0.02353849721739205,
+            -0.0034711576718386185,
+            0.015924913374553738,
+            -0.003604478141616307,
+            -0.0034993903155063817,
+            0.012592308877056896,
+            0.012864938348003884,
+            0.01567418642605425,
+            0.008801753550506258,
+            0.015606443428689292,
+            0.010589933994371628,
+            0.008928721556432368,
+            -0.005805129999077126,
+            -0.001025103064751441,
+            -0.015907755521365925,
+            0.0045874330999415985,
+            0.031701049450012876,
+            0.012594483199773746,
+            0.016783858873835704,
+            -0.01778536268185936,
+            0.00789523134933153,
+            -0.0012706411671644011,
+            0.008325108613530198,
+            -0.001677173669891956,
+            0.029540136288412976,
+            0.011739999605699826,
+            0.015670966746894652,
+            0.008732334090105053,
+            -0.016237731209426752,
+            0.045334690225148,
+            -0.016911928942220152,
+            -0.0021139311364009113,
+            -0.01058970056572469,
+            -0.021705338531028618,
+            0.007324792557651635,
+            -0.006556582104686972,
+            0.0018243188317438144,
+            0.010780292738415671,
+            -0.006471850255151019,
+            0.0006997395548596146,
+            0.003907433192252573,
+            -0.023129313269763224,
+            -0.002550875642832235,
+            -0.02237917712764273,
+            -0.003107775639788617,
+            -0.02621608419628401,
+            -0.00012482032027919036,
+            -0.016207377503696517,
+            -0.0038712739995960753,
+            -0.023315508569288476,
+            -0.014963211090562272,
+            -0.01633110043829169,
+            -0.004241081909202262,
+            -0.008295402942465353,
+            0.004976396647108597,
+            -0.007402107755970892,
+            -0.020321297548927468,
+            0.0020986169430507757,
+            0.016803273183070004,
+            0.01434729320047051,
+            0.018612970083178984,
+            0.00603868809307069,
+            0.014278001198851581,
+            -0.0026350854954985117,
+            0.017171016209107656,
+            -0.006968968261909459,
+            0.006881847166565678,
+            0.011454001418121107,
+            0.02140935294963631,
+            0.011121833270235448,
+            -0.012801756637470525,
+            0.0010870041670931706,
+            -0.009655388034828094,
+            0.002094601818569396,
+            -0.00828486034340067,
+            -0.009190975158350585,
+            0.0017762581946855365,
+            -0.007261996868906865,
+            0.004674405202069334,
+            0.0030526253833799426,
+            -0.0014082232541965316,
+            0.010419302200840906,
+            -0.0024200053377751805,
+            -0.012208685875688886,
+            -0.0033037330470612695,
+            -0.017975940046938498,
+            -0.006958042903103732,
+            -0.0014496522238488239,
+            -0.011711502944602638,
+            -0.01016130418405875,
+            0.003321436189059608,
+            -0.00538282506552848,
+            -0.009353581612407435,
+            0.013884147453136417,
+            0.0001309738696853529,
+            0.0005878330075918333,
+            0.023495280602974504,
+            0.006035028203642483,
+            0.008087594627921345,
+            0.013144288945385904,
+            -0.01023995000994454,
+            -0.022574958230635715,
+            -0.0243435727055662,
+            0.007867022215096418,
+            -0.01573478573423379,
+            0.02324699908478915,
+            -0.007929809922043611,
+            -0.00561630876431978,
+            0.0020982723335057625,
+            -0.013699317926847062,
+            0.008578431496569186,
+            -0.011271381061801356,
+            0.0014450645788554738,
+            -0.019581451799757368,
+            0.018059311800267413,
+            -0.010917911869373693,
+            -0.004302687883018328,
+            0.009466103226841532,
+            0.034049086327052606,
+            -0.0005689605174080737,
+            -0.007410970644432269,
+            -0.006594150877740108,
+            -0.00022810271313786974,
+            0.018766765773409123,
+            -0.0029041815756733333,
+            0.005758781908059541,
+            -0.01563519107805783,
+            0.006571628287171105,
+            -0.006132372252825422,
+            0.0012423261344297225,
+            0.01513074831647933,
+            -0.026306320707079913,
+            -0.015491710256868008,
+            -0.007857352164317122,
+            -0.024893430640551818,
+            0.002253574511878983,
+            0.017579477884206534,
+            0.020505606775521504,
+            -0.020324505029926956,
+            -0.02513332780748072,
+            0.005150966733930881,
+            -0.026580994694344065,
+            0.0037390577082486594,
+            0.1241031878517593,
+            -0.002138404898309707,
+            0.002720723037908521,
+            -0.011425036886011179,
+            0.0022399461009316216,
+            0.005389407121650262,
+            -0.019552543955456567,
+            0.05950746473186985,
+            -0.00638023587296763,
+            -0.004023798021748334,
+            0.002170545188776296,
+            0.001785855390676113,
+            -0.005831534307818096,
+            0.008643501093778434,
+            -0.017602787281866816,
+            -0.02184792441891879,
+            -0.0052215705748838905,
+            -0.021252169339074325,
+            -0.008509621422071373,
+            -0.013000137524907565,
+            0.002893880203681282,
+            -0.015821609145923173,
+            0.010397692426614709,
+            -0.01638916243796692,
+            -0.010283706937701179,
+            -0.01103621454761223,
+            -0.00373552043529251,
+            0.01563553822905027,
+            -0.03188533919653949,
+            -0.004873550301425776,
+            -0.005183590054152726,
+            0.018218971950313273,
+            0.006004328765748325,
+            0.004357734370247252,
+            -0.014230318971458851,
+            0.0064792070801456235,
+            0.0215589173714926,
+            0.0055272493764401305,
+            0.014643335833970546,
+            0.010618070641357003,
+            -0.0022177303813575846,
+            -0.0024402761885758007,
+            0.017541799738501007,
+            -0.005445774414228888,
+            -0.00023502936119626386,
+            -0.00537569046685066,
+            -0.005974703247446444,
+            -0.010712244203425036,
+            0.01843613166635215,
+            -0.009938564643456672,
+            -0.017388592763277214,
+            -0.0015797412875366156,
+            0.02028434083164951,
+            -0.012761534382003973,
+            0.0031906812114950982,
+            0.000788436210275669,
+            0.020589196433948156,
+            0.0007726855314648301,
+            -0.005481260996289263,
+            0.007165014826294827,
+            -0.02956334397512344,
+            0.004095715232997753,
+            0.0003568512047626187,
+            0.027290768799658912,
+            -0.0017631129386615335,
+            0.01552542586826711,
+            -0.020871286597113327,
+            0.004388125824324094,
+            -0.016926008631435403,
+            0.005099899758322398,
+            0.012560923968841436,
+            0.002781049890810361,
+            0.010889021753593997,
+            -0.008500358897706648,
+            -0.021353460542554444,
+            0.012663934009161902,
+            -0.018657094130860155,
+            -0.029156017587920958,
+            0.0018889388761887053,
+            -0.028158076315864772,
+            -0.003048854609750186,
+            0.006812072971079798,
+            -0.035217412591558014,
+            -0.02251675180883659,
+            0.007396524512996277,
+            -0.00995571600201846,
+            -0.011912753666073882,
+            0.010675240017836214,
+            -0.02645155743041798,
+            -0.018137648527851668,
+            0.0010666944556449141,
+            0.012588800077165337,
+            -0.004561825939541096,
+            0.007555558090958502,
+            -0.028558603070876605,
+            0.014816816621445352,
+            -0.01313764261744744,
+            0.016036065063831813,
+            -0.0037797584923144134,
+            0.022217516257304868,
+            0.011131597210722573,
+            0.008795376259031849,
+            -0.004950080558710245,
+            0.021409590863550987,
+            0.005574351961953293,
+            0.051358937509801736,
+            0.031972009383746405,
+            0.0019821003238211276,
+            -0.003251257665378234,
+            0.003313045083311649,
+            0.008294776423730887,
+            -0.00940948257229233,
+            0.005114631360856872,
+            0.028053227377164806,
+            0.016738013668710083,
+            -0.018755924318109216,
+            0.006606711923035118,
+            -0.02702692774986498,
+            0.017950884766781056,
+            -0.022054653845163,
+            0.0003755944703013671,
+            0.007766253634286795,
+            0.0016206993755471613,
+            0.010682347575070117,
+            0.0013842930079185415,
+            0.01705234255261425,
+            0.023975191051210525,
+            -0.002976402980885077,
+            0.017466027107477605,
+            0.012734909275073793,
+            0.018143161773162015,
+            -0.0038399373197265827,
+            0.006929440921265761,
+            -0.012003095293274786,
+            -0.0008883470597690258,
+            0.0020490527756307267,
+            -0.004231091383919565,
+            0.012128848448049833,
+            0.008340302519160299,
+            0.012989849936937843,
+            0.002974488532907779,
+            0.003198854381461613,
+            -0.005309753760578859,
+            -0.0017024982049936384,
+            -0.01763395035509857,
+            -0.01819680226734474,
+            0.028417724274372468,
+            -0.020217879775485576,
+            -0.050118184195774004,
+            0.0025509548991664425,
+            0.014185761006969567,
+            -0.0012021669304388173,
+            -0.0051260576846881515,
+            -0.0005774778872927327,
+            0.005718864494530703,
+            -0.016918749210038116,
+            -0.0015472638835434902,
+            0.008125845907627804,
+            0.020347547062899797,
+            -0.002036218493206914,
+            -0.01079739518866248,
+            0.0038112958713600906,
+            0.02169339303105242,
+            0.01667594322285835,
+            0.012338193092567552,
+            -0.009798409233733167,
+            0.0003198215558702285,
+            -1.6308740825163337e-05,
+            0.0076870827874737325,
+            0.004356644356656038,
+            0.003500023005601613,
+            -0.019968549869439228,
+            -0.008634740687755112,
+            0.012781565963545847,
+            -0.0025637758121512927,
+            0.010243567793632335,
+            -0.00042296044205132426,
+            -0.003069251553118065,
+            -0.009344199449486244,
+            -0.004694953215957636,
+            -0.009583993744231791,
+            0.013783878036521561,
+            -0.007705982042252044,
+            -0.01549567538665377,
+            0.00020666298495823114,
+            0.0005515775451348472,
+            0.001667367278489879,
+            -0.004518431526236256,
+            0.03302324454129141,
+            0.009531990824506103,
+            -0.0004937348108340977,
+            -0.004047613286672229,
+            -0.0327383112522095,
+            0.012456150771224677,
+            -0.009677533214778067,
+            0.0006455811519981003,
+            0.020012976827426605,
+            -0.00020277620481825933,
+            0.002168432791818704,
+            -0.015657551629104767,
+            0.0032924357559238502,
+            0.006161295735841883,
+            0.013814165416788555,
+            0.0005164426482972658,
+            0.01076512534105537,
+            -0.006498977875968572,
+            -0.010325144017898257,
+            0.004804158067334203,
+            0.0104064855510473,
+            -0.0012188100892986933,
+            0.005171197790296134
+          ],
+          "title": "Mixture-Of-Experts With Expert Choice Routing",
+          "keyphrases": [
+            "Mixture-Of-Experts",
+            "Expert Choice Routing",
+            "sparsely-activated models",
+            "training efficiency",
+            "downstream performance"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "3639a96c-e9a4-45ed-a3ac-1e18b2cfe8d4",
+        "properties": {
+          "page_content": "2 Related Work\n\nScaling: Various approaches have been proposed to scale up neural network capacity to improve performance. Recent works have successfully scaled models to billions of parameters via various forms of model parallelism [2, 21, 26, 27, 33]. Model parallelism [30] splits weights and tensors across multiple cores while pipeline parallelism [18, 24] splits different layers across devices with micro-batches pipelined to the different layers. To enable continued scaling of neural networks, improving model training and serving efficiency has become a critical research area.\n\nConditional Computation: Computation decisions can be made dynamically based on the input [23, 25]. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation, by activating certain parameters and computation on demand, on a per-example or per-token basis [3]. Conditional convolution layers [1] with task-specific gating has been used to combat catastrophic forgetting when a sequence of learning problems are optimized. The gating decisions may be binary or sparse and continuous, stochastic or deterministic.\n\nMixture of Experts: Sparsely-gated MoE [31] is the first model to demonstrate massive improvements in model capacity, training time, or model quality with gating. Switch Transformer [10] simplifies the gating by selecting only the top expert per token using a softmax over the hidden state and demonstrates better scaling than previous work. All the prior work requires an auxiliary loss to explicitly encourage balancing. This loss term has to be carefully weighted to not overwhelm the primary loss. However, auxiliary loss does not guarantee balancing and a hard capacity factor has to be imposed. As a result, many tokens can still be unprocessed by the MoE layer. Hard MoE [12] with a single decoding layer can be efficiently trained to good effect on large scale hashtag prediction tasks.\n\nBase Layers [22] formulate a linear assignment that maximizes token-expert affinities while ensuring each expert receives an equal number of tokens. Hash layers [8, 28] devise hashing techniques on input tokens. However, the evaluations are limited to pre-training perplexity. THOR [? ] randomly activates experts during training and inference and is trained with a consistency regularization loss.\n\nTHOR has demonstrated strong performance on translation tasks. Different from these prior works, our method is a learnt method that enables heterogeneous MoE and effectively improves downstream fine-tuning performance.\n\n",
+          "embedding": [
+            -0.013259558007121086,
+            -0.007197607308626175,
+            -0.015822459012269974,
+            0.013052377849817276,
+            0.029680538922548294,
+            0.02572108618915081,
+            -0.01866159960627556,
+            0.040392544120550156,
+            -0.039103418588638306,
+            0.017602676525712013,
+            0.020380431786179543,
+            -0.012024147436022758,
+            0.0011442891554906964,
+            0.006088807247579098,
+            0.009645408019423485,
+            -0.014533335343003273,
+            -0.004703766666352749,
+            0.0049493140541017056,
+            0.02260570414364338,
+            -0.03012559376657009,
+            -0.008847379125654697,
+            -0.007220627274364233,
+            -0.03612615540623665,
+            -0.01026694942265749,
+            0.03443801775574684,
+            0.0267186239361763,
+            0.03146075829863548,
+            -0.011226118542253971,
+            -0.04496586322784424,
+            0.025629006326198578,
+            0.015055122785270214,
+            0.009300106205046177,
+            -0.016912074759602547,
+            -0.01592988520860672,
+            -0.0355122871696949,
+            -0.04048462212085724,
+            0.05073622614145279,
+            0.013781346380710602,
+            -0.01992003060877323,
+            0.02774685248732567,
+            0.021807676181197166,
+            0.0150397764518857,
+            -0.03502119332551956,
+            -0.02219134382903576,
+            0.00023379754566121846,
+            0.0007141016540117562,
+            -0.032136011868715286,
+            -0.009069905616343021,
+            -0.008317917585372925,
+            -0.00694822333753109,
+            0.028268640860915184,
+            -0.04450546205043793,
+            0.01573037914931774,
+            0.012584302574396133,
+            -0.038274697959423065,
+            0.039932142943143845,
+            0.013228864409029484,
+            0.005678283050656319,
+            0.053621407598257065,
+            0.03726181387901306,
+            0.011333545669913292,
+            0.006449454929679632,
+            -0.04613221064209938,
+            0.002123601036146283,
+            0.010443436913192272,
+            -0.008847379125654697,
+            -0.00621158117428422,
+            0.0015471403021365404,
+            0.0041819787584245205,
+            0.010289969854056835,
+            -0.0367400236427784,
+            -0.0014166932087391615,
+            -0.011893700808286667,
+            -0.003069342114031315,
+            0.010834777727723122,
+            0.001008087070658803,
+            0.019367549568414688,
+            0.039011336863040924,
+            0.03511327505111694,
+            -0.008678564801812172,
+            -0.01622147299349308,
+            0.03317958861589432,
+            -0.002503432100638747,
+            0.016574447974562645,
+            0.02119380794465542,
+            -0.03713903948664665,
+            -0.0017399332718923688,
+            -0.06629779189825058,
+            -0.018983881920576096,
+            -0.01323653757572174,
+            -0.02991073951125145,
+            0.0056015495210886,
+            -0.01734178327023983,
+            -0.0033398279920220375,
+            -0.020273005589842796,
+            0.04646984115242958,
+            -0.018446745350956917,
+            -0.013482085429131985,
+            -0.032872654497623444,
+            0.004730623681098223,
+            -0.01812446489930153,
+            0.03631031885743141,
+            -0.0027451429050415754,
+            -0.0076541719026863575,
+            0.01571503095328808,
+            0.009092926047742367,
+            -0.05006096884608269,
+            0.034622177481651306,
+            -0.00013188579760026187,
+            0.074830561876297,
+            0.012898909859359264,
+            -0.033210281282663345,
+            0.00029134770738892257,
+            0.04054601117968559,
+            -0.03152214363217354,
+            -0.02452404424548149,
+            -0.04149750620126724,
+            -0.03597268834710121,
+            0.001932726358063519,
+            -0.028406761586666107,
+            0.024416616186499596,
+            0.00759278517216444,
+            -0.0010752289090305567,
+            -0.012430835515260696,
+            -0.05288476496934891,
+            0.050889693200588226,
+            -0.013812039978802204,
+            -0.01692742109298706,
+            -0.005755016580224037,
+            -0.02243689075112343,
+            -0.011356566101312637,
+            0.023541854694485664,
+            -0.032872654497623444,
+            -0.05638381466269493,
+            0.01705019548535347,
+            0.016160085797309875,
+            0.051380787044763565,
+            0.02021161839365959,
+            -0.018738333135843277,
+            -0.010044422000646591,
+            0.03278057277202606,
+            -0.005781873129308224,
+            0.02360324189066887,
+            -0.022022530436515808,
+            -0.028821121901273727,
+            0.013942486606538296,
+            -0.033455830067396164,
+            0.06227695196866989,
+            -0.01651306077837944,
+            0.023787401616573334,
+            -0.013474412262439728,
+            0.04542626440525055,
+            -0.03324097394943237,
+            0.03235086426138878,
+            -0.029388951137661934,
+            0.021500742062926292,
+            0.005755016580224037,
+            -0.00864787120372057,
+            0.0028065296355634928,
+            0.00858648493885994,
+            -0.008279550820589066,
+            -0.020902220159769058,
+            0.005098944529891014,
+            0.030800849199295044,
+            0.0026933476328849792,
+            0.024278495460748672,
+            0.04907878115773201,
+            0.02077944576740265,
+            -0.052915457636117935,
+            0.029726577922701836,
+            0.029695885255932808,
+            0.020426472648978233,
+            0.0444747656583786,
+            0.007554418407380581,
+            0.010604577139019966,
+            0.006587575655430555,
+            0.014993736520409584,
+            0.0070403036661446095,
+            -0.01655910164117813,
+            0.03879648447036743,
+            -0.0241403765976429,
+            -0.0031268922612071037,
+            0.024784937500953674,
+            0.0015365893486887217,
+            0.036617252975702286,
+            0.005482612177729607,
+            0.013512779027223587,
+            0.02136262133717537,
+            0.00933079980313778,
+            0.027808239683508873,
+            0.02021161839365959,
+            0.030171632766723633,
+            0.020150231197476387,
+            -0.024401269853115082,
+            0.03198254480957985,
+            -0.056813523173332214,
+            -0.015945231541991234,
+            -0.009323126636445522,
+            0.028283987194299698,
+            -0.006169377826154232,
+            -0.01812446489930153,
+            -0.017894264310598373,
+            0.008049349300563335,
+            0.002048785798251629,
+            -0.009676100686192513,
+            0.008931785821914673,
+            0.033916231244802475,
+            0.026288915425539017,
+            0.024002255871891975,
+            -0.037937067449092865,
+            -0.0052025350742042065,
+            -0.010903838090598583,
+            -0.013374658301472664,
+            -0.034990500658750534,
+            -0.00218306970782578,
+            0.009461247362196445,
+            0.0012622670037671924,
+            0.009630060754716396,
+            -0.002929303329437971,
+            0.0027355512138456106,
+            0.03015628643333912,
+            -0.0046423799358308315,
+            0.045610424131155014,
+            -0.014425908215343952,
+            -0.0034817850682884455,
+            -0.020503206178545952,
+            -0.006576065439730883,
+            0.003627578727900982,
+            0.01622147299349308,
+            0.011824640445411205,
+            0.0031134639866650105,
+            0.012661036103963852,
+            0.004577156621962786,
+            -0.004465892910957336,
+            0.04435199499130249,
+            -0.016697220504283905,
+            0.01414966769516468,
+            -0.011064978316426277,
+            0.008195143193006516,
+            0.003395459847524762,
+            -0.005912320222705603,
+            0.041466813534498215,
+            -0.047851044684648514,
+            -0.0132749043405056,
+            0.020718058571219444,
+            0.0033820313401520252,
+            -0.03176769241690636,
+            0.012001127935945988,
+            -0.0009505368652753532,
+            -0.01609869860112667,
+            -0.01435684785246849,
+            -0.02364928089082241,
+            -0.005543999373912811,
+            0.00993699487298727,
+            0.00022012938279658556,
+            0.05666005611419678,
+            0.018078425899147987,
+            -0.01528532337397337,
+            -0.01978190988302231,
+            -0.02140866219997406,
+            0.03962520509958267,
+            -0.004730623681098223,
+            -0.035819221287965775,
+            0.013934813439846039,
+            -0.012039494700729847,
+            -0.01883041486144066,
+            0.030938968062400818,
+            0.01470982190221548,
+            -0.003180605825036764,
+            -0.019029920920729637,
+            0.008862725459039211,
+            -0.05260852351784706,
+            -0.005716649815440178,
+            0.0007275299867615104,
+            0.06727997958660126,
+            -0.024984445422887802,
+            -0.010934531688690186,
+            0.0050490680150687695,
+            0.0011596358381211758,
+            0.02364928089082241,
+            -0.00858648493885994,
+            -0.003474111668765545,
+            -0.021961143240332603,
+            0.03003351204097271,
+            -0.008010982535779476,
+            0.005578529555350542,
+            -0.026979517191648483,
+            -0.017955651506781578,
+            -0.04849560558795929,
+            -0.02978796511888504,
+            -0.014510314911603928,
+            0.012607323005795479,
+            -0.05273129791021347,
+            0.017648717388510704,
+            0.026856742799282074,
+            -0.021086379885673523,
+            0.009668427519500256,
+            0.012208308093249798,
+            0.027286451309919357,
+            -0.034407325088977814,
+            -0.05260852351784706,
+            -0.02941964380443096,
+            0.014157340861856937,
+            -0.016942769289016724,
+            -0.05512538552284241,
+            0.0036122319288551807,
+            0.012868217192590237,
+            0.015546217560768127,
+            -0.004849560558795929,
+            -0.0013322862796485424,
+            0.02133192867040634,
+            -0.006986590102314949,
+            0.0062768044881522655,
+            -0.01721900887787342,
+            -0.00873227883130312,
+            0.011793946847319603,
+            0.010282295756042004,
+            -0.000877640035469085,
+            -0.03738458827137947,
+            -0.013466738164424896,
+            0.0236953217536211,
+            0.018600212410092354,
+            -0.07317311316728592,
+            0.011487012729048729,
+            -0.027501305565238,
+            -0.007170750759541988,
+            -0.0042126718908548355,
+            0.03302612155675888,
+            -0.019060615450143814,
+            0.016083352267742157,
+            -0.0020583774894475937,
+            -0.02808448113501072,
+            -0.027639426290988922,
+            -0.006537698674947023,
+            -0.03410039097070694,
+            -0.03207462653517723,
+            -0.028468148782849312,
+            -0.004178142175078392,
+            -0.03143006190657616,
+            -0.029220137745141983,
+            -0.024508696049451828,
+            0.0023173533845692873,
+            -0.00552865257486701,
+            0.017357129603624344,
+            0.01899922825396061,
+            -0.03373207151889801,
+            -0.045364875346422195,
+            0.013190497644245625,
+            -0.006587575655430555,
+            0.01259197574108839,
+            0.0009654040331952274,
+            -0.012062514200806618,
+            0.016282860189676285,
+            -0.003094280604273081,
+            -0.03015628643333912,
+            -0.046193599700927734,
+            -0.013297924771904945,
+            0.014610068872570992,
+            0.008095390163362026,
+            -0.0002774397435132414,
+            0.010006055235862732,
+            -0.04686885327100754,
+            -0.024171069264411926,
+            0.01066596433520317,
+            -0.04008561000227928,
+            -0.024002255871891975,
+            -0.0045694829896092415,
+            -0.025444846600294113,
+            -0.006391904782503843,
+            -0.006844632793217897,
+            -0.0222373828291893,
+            0.03729250654578209,
+            0.018262585625052452,
+            -0.00933079980313778,
+            0.02941964380443096,
+            0.012162268161773682,
+            0.020672019571065903,
+            -0.01617543213069439,
+            -0.04640845209360123,
+            0.030048860237002373,
+            -0.019444283097982407,
+            -0.03098500892519951,
+            8.272836566902697e-05,
+            -0.022252731025218964,
+            0.007792292162775993,
+            0.00012229410640429705,
+            0.015945231541991234,
+            -0.006422598380595446,
+            -0.04573319852352142,
+            0.02932756394147873,
+            0.004872580524533987,
+            -0.04956987500190735,
+            -0.0022137630730867386,
+            -0.04791243001818657,
+            -0.0065607186406850815,
+            0.030340446159243584,
+            -0.004914783872663975,
+            -0.027977053076028824,
+            -0.03459148481488228,
+            -0.0015605686930939555,
+            -0.011433299630880356,
+            0.014625415205955505,
+            -0.013643225654959679,
+            0.013727632351219654,
+            -0.008041676133871078,
+            -0.0188764538615942,
+            -0.010719677433371544,
+            0.015776418149471283,
+            0.03759944066405296,
+            -0.024155722931027412,
+            -0.022682437673211098,
+            -0.03572714328765869,
+            -0.03953312709927559,
+            0.013221191242337227,
+            0.024201761931180954,
+            0.01867694780230522,
+            -0.010972898453474045,
+            0.018078425899147987,
+            0.0068369596265256405,
+            -0.014878636226058006,
+            -0.025153258815407753,
+            0.0029810986015945673,
+            0.05441943556070328,
+            0.00807236973196268,
+            -0.00013368423969950527,
+            0.002990690292790532,
+            0.009798875078558922,
+            0.017188316211104393,
+            0.005555509123951197,
+            -0.014280114322900772,
+            0.04373812675476074,
+            -0.014433581382036209,
+            0.039686594158411026,
+            -0.01065829023718834,
+            -0.02070271223783493,
+            -0.03634101152420044,
+            0.023679975420236588,
+            0.005313798785209656,
+            -0.03443801775574684,
+            -0.012361775152385235,
+            0.029358256608247757,
+            0.005720486398786306,
+            -0.010773390531539917,
+            0.012599649839103222,
+            -0.035358820110559464,
+            -0.029557764530181885,
+            -0.022713132202625275,
+            -0.037108346819877625,
+            -0.00947659369558096,
+            -0.03453009948134422,
+            -0.06365815550088882,
+            -0.004247202072292566,
+            -0.061294760555028915,
+            -0.014878636226058006,
+            0.02820725366473198,
+            -0.0006987549131736159,
+            0.0162982065230608,
+            -0.00065223517594859,
+            -0.007700212299823761,
+            -0.03833608329296112,
+            0.005206371657550335,
+            0.04174305126070976,
+            0.01746455766260624,
+            -0.0380905345082283,
+            -0.017280396074056625,
+            -0.009031538851559162,
+            -0.04263316094875336,
+            -0.029342910274863243,
+            0.009031538851559162,
+            -0.013643225654959679,
+            6.396460958058015e-05,
+            -0.03941035270690918,
+            0.003128810552880168,
+            0.006840796209871769,
+            0.018109118565917015,
+            0.04444407299160957,
+            0.022129956632852554,
+            -0.06040465459227562,
+            0.007987963035702705,
+            -0.003629497019574046,
+            0.027240412309765816,
+            -0.014072933234274387,
+            0.018017038702964783,
+            -0.0006474393303506076,
+            -0.007009610068053007,
+            -0.02662654221057892,
+            0.009054559282958508,
+            -0.0012507569044828415,
+            -0.011878354474902153,
+            -0.0038596978411078453,
+            0.004316262435168028,
+            0.00914663914591074,
+            0.03182907775044441,
+            0.010282295756042004,
+            0.009537980891764164,
+            -0.04082224890589714,
+            -0.012860543094575405,
+            -0.01130285207182169,
+            0.04042323678731918,
+            0.009246393106877804,
+            -0.034990500658750534,
+            0.007316544186323881,
+            0.03253502771258354,
+            0.028406761586666107,
+            0.028115173801779747,
+            -0.011072651483118534,
+            0.012730096466839314,
+            0.020349739119410515,
+            -0.07004238665103912,
+            -0.005758853163570166,
+            -0.0069904266856610775,
+            -0.0037503524217754602,
+            0.0037407607305794954,
+            -0.02289729192852974,
+            -0.00019531087309587747,
+            0.042541082948446274,
+            -0.016896728426218033,
+            -0.00151644682046026,
+            -0.011003591120243073,
+            0.029235484078526497,
+            0.002560982247814536,
+            0.004086061846464872,
+            -0.012607323005795479,
+            0.058470968157052994,
+            -0.017863571643829346,
+            0.013343964703381062,
+            0.006077297497540712,
+            -0.01306772418320179,
+            0.024861671030521393,
+            -0.021715596318244934,
+            -0.007051813416182995,
+            0.005056741181761026,
+            0.0094382269307971,
+            -0.017157621681690216,
+            -0.003107708878815174,
+            -0.0026991027407348156,
+            -0.01741851679980755,
+            0.010811757296323776,
+            -0.0489560067653656,
+            -0.03471425920724869,
+            -0.0022463747300207615,
+            0.01005976926535368,
+            -0.020027456805109978,
+            0.010305316187441349,
+            0.014326154254376888,
+            0.007934249006211758,
+            0.055739253759384155,
+            -0.03143006190657616,
+            -0.0016229146858677268,
+            -0.003309134393930435,
+            -0.023925522342324257,
+            -0.028636962175369263,
+            -0.023004719987511635,
+            -0.0056130592711269855,
+            -0.015822459012269974,
+            -0.04386089742183685,
+            -0.008693911135196686,
+            -0.021439354866743088,
+            0.021270541474223137,
+            -0.04094502329826355,
+            0.016912074759602547,
+            0.008724604733288288,
+            0.0006124296924099326,
+            -0.009008519351482391,
+            -0.01995072327554226,
+            -0.0042625488713383675,
+            0.029757272452116013,
+            0.032166704535484314,
+            0.018861107528209686,
+            0.034867726266384125,
+            -0.005636079702526331,
+            -0.0231888797134161,
+            -0.013581839390099049,
+            0.03572714328765869,
+            -0.00854811817407608,
+            0.020058151334524155,
+            0.004980007652193308,
+            -0.0006061950698494911,
+            -0.0224061980843544,
+            -0.0077270688489079475,
+            0.03646378591656685,
+            -0.02555227279663086,
+            0.014425908215343952,
+            0.05122731998562813,
+            0.032749880105257034,
+            0.0006071542156860232,
+            -0.008640198037028313,
+            -0.01472516916692257,
+            0.005919993855059147,
+            -0.021792329847812653,
+            0.03198254480957985,
+            0.01987398974597454,
+            -0.001803238526917994,
+            -0.010535516776144505,
+            0.01325188484042883,
+            -0.030217673629522324,
+            0.012469202280044556,
+            0.006668145768344402,
+            0.003606477053835988,
+            -0.034867726266384125,
+            -0.02136262133717537,
+            -0.002419025171548128,
+            -0.0007759680738672614,
+            -0.01929081603884697,
+            -0.015553891658782959,
+            -0.02286659926176071,
+            0.002154294401407242,
+            -0.006821612827479839,
+            0.004281732253730297,
+            0.016697220504283905,
+            0.02837606891989708,
+            -0.0006138684111647308,
+            0.03701626509428024,
+            0.00303289364092052,
+            0.023879481479525566,
+            0.004945477470755577,
+            -0.003512478433549404,
+            -0.0070403036661446095,
+            -0.011755580082535744,
+            -0.014487294480204582,
+            0.057273924350738525,
+            0.017019502818584442,
+            0.05957593023777008,
+            -0.011064978316426277,
+            -0.010819431394338608,
+            0.030938968062400818,
+            0.004316262435168028,
+            -0.020733406767249107,
+            0.008670891635119915,
+            -0.016160085797309875,
+            -0.02871369570493698,
+            -0.0027470611967146397,
+            -0.03799845650792122,
+            -0.0188764538615942,
+            -0.022621052339673042,
+            -0.018308626487851143,
+            0.004684583283960819,
+            0.0196591354906559,
+            0.016436327248811722,
+            -0.00016737506666686386,
+            -0.011763254180550575,
+            0.031583528965711594,
+            0.019306162372231483,
+            0.028928549960255623,
+            -0.011809294112026691,
+            0.007331890985369682,
+            -0.01576107181608677,
+            0.0052831051871180534,
+            0.019137348979711533,
+            -0.01253058947622776,
+            0.009453574195504189,
+            -0.0008905887953005731,
+            0.011180078610777855,
+            0.03821330890059471,
+            -0.010036748833954334,
+            -0.03953312709927559,
+            -0.006280641537159681,
+            -0.014778882265090942,
+            0.007278177421540022,
+            0.0067065125331282616,
+            -0.020932912826538086,
+            0.012254348024725914,
+            0.007730905432254076,
+            -0.01479422952979803,
+            0.02128588780760765,
+            -0.007205280475318432,
+            -0.027624079957604408,
+            0.03468356654047966,
+            -0.014134320430457592,
+            0.01787891797721386,
+            -0.017986344173550606,
+            -0.0028755899984389544,
+            -0.025828514248132706,
+            0.022559665143489838,
+            0.01751059666275978,
+            -0.027317145839333534,
+            0.022989371791481972,
+            0.04564111679792404,
+            0.020150231197476387,
+            0.0007615805370733142,
+            0.012415489181876183,
+            0.0057665263302624226,
+            -0.036494478583335876,
+            -0.02057993970811367,
+            -0.006944386754184961,
+            -0.0027950196526944637,
+            -0.016574447974562645,
+            0.030217673629522324,
+            -0.03824400156736374,
+            -0.0031518307514488697,
+            -0.0052025350742042065,
+            -0.024078989401459694,
+            0.0055823661386966705,
+            0.009069905616343021,
+            0.003951778169721365,
+            0.017142275348305702,
+            -0.014172687195241451,
+            0.03020232729613781,
+            0.03566575422883034,
+            0.006322844885289669,
+            -0.017848225310444832,
+            0.011256812140345573,
+            -0.024508696049451828,
+            0.0048457239754498005,
+            0.043001484125852585,
+            0.014072933234274387,
+            0.008494404144585133,
+            -0.020426472648978233,
+            -0.02584386058151722,
+            0.007235974073410034,
+            0.008709258399903774,
+            0.04198860004544258,
+            0.02630426175892353,
+            0.03606477007269859,
+            0.009369166567921638,
+            -0.01199345476925373,
+            -0.013819713145494461,
+            -0.0049493140541017056,
+            -0.023265613242983818,
+            0.021761635318398476,
+            -0.02228342369198799,
+            0.01717296987771988,
+            0.012039494700729847,
+            -0.028759736567735672,
+            0.00017528820899315178,
+            0.01576107181608677,
+            0.03575783595442772,
+            0.04361535236239433,
+            0.023925522342324257,
+            0.0032765227369964123,
+            0.039686594158411026,
+            -0.04493517056107521,
+            -0.03716973215341568,
+            0.00753907160833478,
+            0.017571983858942986,
+            0.009154313243925571,
+            0.0058164033107459545,
+            0.03290334716439247,
+            -0.00947659369558096,
+            -0.003376276232302189,
+            0.012024147436022758,
+            0.02074875310063362,
+            0.002031520940363407,
+            -0.00429324246942997,
+            -0.008095390163362026,
+            0.02289729192852974,
+            0.021086379885673523,
+            0.03937966004014015,
+            -0.020058151334524155,
+            0.01780218444764614,
+            0.0009936995338648558,
+            -0.02552158012986183,
+            0.02559831365942955,
+            0.00021701208606828004,
+            -0.04530349001288414,
+            0.03339444100856781,
+            0.017249703407287598,
+            0.015108836814761162,
+            0.011786273680627346,
+            -0.006131011061370373,
+            0.012553608976304531,
+            0.030524607747793198,
+            -0.0031019540037959814,
+            0.0011423707474023104,
+            0.03115382231771946,
+            -0.003311052918434143,
+            -0.004807357210665941,
+            0.020886873826384544,
+            0.004887927323579788,
+            -0.013167478144168854,
+            0.01867694780230522,
+            -0.012169941328465939,
+            0.044075753539800644,
+            -0.024692857638001442,
+            0.006760226096957922,
+            0.010205562226474285,
+            -0.01883041486144066,
+            -0.007293524220585823,
+            -0.0013188580051064491,
+            -0.022268077358603477,
+            0.004500423092395067,
+            0.018155159428715706,
+            -0.012331082485616207,
+            0.0295731108635664,
+            -0.028606269508600235,
+            0.0012430836213752627,
+            0.015945231541991234,
+            -0.026933476328849792,
+            0.02767011895775795,
+            0.013006336987018585,
+            0.02713298425078392,
+            -0.009576347656548023,
+            -0.014456601813435555,
+            0.04797381907701492,
+            0.0563531219959259,
+            0.0060504404827952385,
+            -0.008578810840845108,
+            0.013742979615926743,
+            0.0003174850717186928,
+            -0.02360324189066887,
+            0.024048294872045517,
+            -0.03572714328765869,
+            0.0162982065230608,
+            0.03107708878815174,
+            -0.014617742039263248,
+            -0.014095953665673733,
+            -0.034622177481651306,
+            -0.0052831051871180534,
+            0.00429324246942997,
+            0.004381485749036074,
+            0.0036966388579458,
+            -0.025659700855612755,
+            -0.015599931590259075,
+            0.022252731025218964,
+            0.005379022099077702,
+            0.0010157604701817036,
+            -0.025076525285840034,
+            0.015530871227383614,
+            -0.01617543213069439,
+            -0.04162028059363365,
+            -0.006330518051981926,
+            -0.012492222711443901,
+            0.025214646011590958,
+            -6.204627425177023e-05,
+            -0.03477564454078674,
+            -0.006718022748827934,
+            -0.005191024858504534,
+            -0.009054559282958508,
+            0.044536154717206955,
+            0.023741360753774643,
+            0.03812123090028763,
+            -0.02170024812221527,
+            -0.03141471743583679,
+            0.02460077777504921,
+            -0.03489841893315315,
+            0.030770154669880867,
+            -0.0032439110800623894,
+            0.007186097092926502,
+            -0.0052025350742042065,
+            0.039348967373371124,
+            -0.012146921828389168,
+            0.008571137674152851,
+            -0.03588061034679413,
+            -0.004980007652193308,
+            -0.01945962943136692,
+            -0.008517424575984478,
+            0.0031153822783380747,
+            0.011870680376887321,
+            -0.009568674489855766,
+            0.005659099668264389,
+            0.04720648378133774,
+            0.031583528965711594,
+            0.0018675028113648295,
+            0.026565156877040863,
+            -0.02862161584198475,
+            -0.017986344173550606,
+            0.01464076153934002,
+            -0.0064072515815496445,
+            -0.00785751547664404,
+            -0.013973180204629898,
+            -0.031798385083675385,
+            -0.012799156829714775,
+            0.011187751777470112,
+            0.013581839390099049,
+            -0.011602113023400307,
+            0.003735005622729659,
+            0.012906583957374096,
+            -0.014172687195241451,
+            -0.008057023398578167,
+            -0.0393182709813118,
+            0.0011222282191738486,
+            0.024370577186346054,
+            -0.027977053076028824,
+            0.013121437281370163,
+            0.002286659786477685,
+            0.007389441132545471,
+            -0.023618588224053383,
+            -0.02087152563035488,
+            0.008218163624405861,
+            -0.012154594995081425,
+            0.005248575005680323,
+            -0.002712531015276909,
+            0.034744951874017715,
+            -0.00800330936908722,
+            0.03351721540093422,
+            -0.012975644320249557,
+            0.01953636296093464,
+            -0.025506233796477318,
+            -0.02389482781291008,
+            -0.04272524267435074,
+            -0.0012977562146261334,
+            0.0029580784030258656,
+            -0.02820725366473198,
+            -0.006495495326817036,
+            -0.001958623994141817,
+            0.017249703407287598,
+            -0.012246674858033657,
+            -0.003274604445323348,
+            0.006622105836868286,
+            -0.01354347262531519,
+            -0.0020104190334677696,
+            -0.03987075388431549,
+            -0.02825329452753067,
+            -0.03406969830393791,
+            0.016835341230034828,
+            -0.022667091339826584,
+            0.017234355211257935,
+            -0.01995072327554226,
+            0.0005678283050656319,
+            -0.047359950840473175,
+            -0.01573037914931774,
+            0.02991073951125145,
+            0.008141430094838142,
+            -0.04204998537898064,
+            0.021470047533512115,
+            0.01755663752555847,
+            0.017479903995990753,
+            -0.02311214618384838,
+            0.018492786213755608,
+            0.002338455058634281,
+            0.004933967255055904,
+            0.024416616186499596,
+            -0.02535276673734188,
+            0.02177698165178299,
+            -0.024907711893320084,
+            -0.021838368847966194,
+            -0.004020838066935539,
+            0.013827386312186718,
+            0.01726504974067211,
+            0.0020104190334677696,
+            -0.01709623634815216,
+            0.016743261367082596,
+            0.018323972821235657,
+            0.003915329463779926,
+            0.022636398673057556,
+            -0.02664189040660858,
+            0.016620486974716187,
+            0.011809294112026691,
+            -0.02323492057621479,
+            0.0036256604362279177,
+            -0.01483259629458189,
+            -0.007174587342888117,
+            0.017525942996144295,
+            0.03146075829863548,
+            0.014617742039263248,
+            -0.03683210536837578,
+            0.012860543094575405,
+            0.011747906915843487,
+            -0.0014147749170660973,
+            0.038274697959423065,
+            0.0027240412309765816,
+            0.0239715613424778,
+            0.03397761657834053,
+            0.0066105956211686134,
+            0.017664063721895218,
+            0.03548159450292587,
+            -0.022912638261914253,
+            -0.009277086704969406,
+            -0.023802747949957848,
+            0.014111299999058247,
+            0.014310807920992374,
+            -0.012753116898238659,
+            -0.012001127935945988,
+            -0.015239283442497253,
+            -0.014118974097073078,
+            0.01879972033202648,
+            -0.0008503036806359887,
+            -0.015745725482702255,
+            -0.029711231589317322,
+            0.024692857638001442,
+            0.0006781327538192272,
+            -0.044658929109573364,
+            0.012323408387601376,
+            -0.014218727126717567,
+            -0.0037868008948862553,
+            -0.0027835096698254347,
+            0.007769272197037935,
+            -0.005605386104434729,
+            0.0033359911758452654,
+            0.004266385454684496,
+            0.05135009437799454,
+            -0.0014166932087391615,
+            0.017311088740825653,
+            -0.008839705027639866,
+            0.01970517635345459,
+            -0.010988244786858559,
+            0.022912638261914253,
+            -0.014556354843080044,
+            -0.051503561437129974,
+            0.02618148922920227,
+            -0.011041957885026932,
+            0.040883637964725494,
+            0.05098177120089531,
+            -0.01841605268418789,
+            -0.010750371031463146,
+            0.03575783595442772,
+            0.004680746700614691,
+            -0.025337418541312218,
+            0.007888209074735641,
+            -0.0012296552304178476,
+            -0.007504541426897049,
+            -0.019613096490502357,
+            -0.012077861465513706,
+            0.019720522686839104,
+            -0.0038712078239768744,
+            0.017372475937008858,
+            0.040638089179992676,
+            0.03161422535777092,
+            -0.021301234140992165,
+            0.009522633627057076,
+            0.01331327110528946,
+            -0.009215699508786201,
+            -0.007796129211783409,
+            0.0254295002669096,
+            -0.017034849151968956,
+            -0.0008320794440805912,
+            -0.012177614495158195,
+            -0.00400165468454361,
+            -0.035941995680332184,
+            0.001303511206060648,
+            -0.016605140641331673,
+            -0.0018761353567242622,
+            -0.007692538667470217,
+            -0.030923621729016304,
+            0.03228947892785072,
+            0.028698349371552467,
+            -0.0070633236318826675,
+            -0.009077579714357853,
+            -0.009468920528888702,
+            0.005037557799369097,
+            0.0036333338357508183,
+            -0.0019969907589256763,
+            0.0034357449039816856,
+            -0.01792495884001255,
+            0.007443154696375132,
+            -0.006775572896003723,
+            -0.01325188484042883,
+            -0.051135238260030746,
+            -0.01867694780230522,
+            0.025045832619071007,
+            0.004830377176403999,
+            -0.01692742109298706,
+            -0.003261175937950611,
+            0.008103063330054283,
+            -0.005862443707883358,
+            -0.02496909908950329,
+            0.03135332837700844,
+            0.01668187417089939,
+            0.01875368133187294,
+            -0.0017456883797422051,
+            -0.0022578847128897905,
+            0.01746455766260624,
+            0.006084970664232969,
+            -0.01547715812921524,
+            -0.009499614126980305,
+            0.021761635318398476,
+            0.004120592027902603,
+            0.020932912826538086,
+            -0.010735023766756058,
+            0.021086379885673523,
+            0.007957269437611103,
+            -0.012123901396989822,
+            -0.0027202044147998095,
+            -0.0012718586949631572,
+            0.004105245228856802,
+            0.016912074759602547,
+            -0.013980853371322155,
+            0.024953750893473625,
+            0.020088844001293182,
+            0.007109363563358784,
+            0.0044889128766953945,
+            -0.01668187417089939,
+            -0.02452404424548149,
+            0.016574447974562645,
+            0.002522615483030677,
+            0.027977053076028824,
+            0.012139247730374336,
+            -0.004013164900243282,
+            -0.02759338542819023,
+            -0.013052377849817276,
+            -0.024217110127210617,
+            0.01005976926535368,
+            0.023664627224206924,
+            0.030632033944129944,
+            -0.015461810864508152,
+            -0.013489758595824242,
+            -0.025276033207774162,
+            0.00469993008300662,
+            0.008908765390515327,
+            -0.027347838506102562,
+            -0.01992003060877323,
+            0.018032385036349297,
+            -0.025291379541158676,
+            0.026488423347473145,
+            -0.0019269713666290045,
+            -0.012852869927883148,
+            0.006675818935036659,
+            0.07071764022111893,
+            -0.013090744614601135,
+            -0.0012200635392218828,
+            -0.028652308508753777,
+            -0.017618024721741676,
+            -0.005931503605097532,
+            -0.008617177605628967,
+            0.0004412179405335337,
+            0.018492786213755608,
+            -0.008624851703643799,
+            0.020902220159769058,
+            0.01668187417089939,
+            0.0013006337685510516,
+            0.0032074626069515944,
+            -0.03526674211025238,
+            0.03032509982585907,
+            -0.004224182106554508,
+            0.0014838350471109152,
+            0.00244204537011683,
+            0.006307498086243868,
+            0.018400706350803375,
+            0.016896728426218033,
+            0.005931503605097532,
+            0.009200353175401688,
+            -0.005912320222705603,
+            -0.003738842438906431,
+            0.016114046797156334,
+            0.019858643412590027,
+            -0.01111869141459465,
+            -0.012944950722157955,
+            0.005728159565478563,
+            0.014901655726134777,
+            -0.004853397142142057,
+            -0.00414744857698679,
+            0.0075736017897725105,
+            0.020472511649131775,
+            -0.01617543213069439,
+            -0.002286659786477685,
+            0.004742133431136608,
+            -0.017372475937008858,
+            0.0007548663415946066,
+            0.01875368133187294,
+            0.007565928157418966,
+            0.015853151679039,
+            -0.015484831295907497,
+            -0.009161986410617828,
+            0.0021255193278193474,
+            -0.022421544417738914,
+            0.003121137386187911,
+            -0.012192961759865284,
+            -0.01058155670762062,
+            -0.00239600520581007,
+            -0.007416298147290945,
+            -0.006487822160124779,
+            -0.006568392273038626,
+            0.004630869720131159,
+            0.009430553764104843,
+            -0.004151285160332918,
+            0.01738782227039337,
+            0.005252411589026451,
+            0.013988526538014412,
+            -0.0053560021333396435,
+            0.01489398255944252,
+            0.02136262133717537,
+            -0.010213236324489117,
+            0.00075822341023013,
+            0.01775614358484745,
+            -0.0006344905705191195,
+            0.006741042714565992,
+            0.004013164900243282,
+            1.707021874608472e-05,
+            0.010842450894415379,
+            -0.004343118984252214,
+            -0.01992003060877323,
+            -0.0055094691924750805,
+            -0.011440972797572613,
+            0.002133192727342248,
+            0.002252129837870598,
+            0.03716973215341568,
+            -0.02513791248202324,
+            0.03235086426138878,
+            0.012914257124066353,
+            -0.014011546969413757,
+            -0.00569362984970212,
+            -0.047482721507549286,
+            0.01338233146816492,
+            -0.014694475568830967,
+            3.977555752499029e-05,
+            0.017970997840166092,
+            -0.00914663914591074,
+            -0.0014799984637647867,
+            0.00502988463267684,
+            -0.005482612177729607,
+            -0.0009457410196773708,
+            -0.01924477517604828,
+            -0.0067563895136117935,
+            0.010382049717009068,
+            0.013842732645571232,
+            -0.03471425920724869,
+            -0.02243689075112343,
+            -0.030263712629675865,
+            0.00392875773832202,
+            0.01883041486144066,
+            -0.010942204855382442,
+            -0.025828514248132706,
+            -0.009537980891764164,
+            0.013405351899564266,
+            -0.01697346195578575,
+            0.006388068199157715,
+            -0.003997818101197481,
+            0.014326154254376888,
+            -0.008832031860947609,
+            0.001874217065051198,
+            -0.01899922825396061,
+            0.01487096305936575,
+            0.0028314681258052588,
+            -0.00420116214081645,
+            -0.030340446159243584,
+            0.007120873779058456,
+            -0.00020574184600263834,
+            -0.01450264174491167,
+            -0.012269695289433002,
+            0.017280396074056625,
+            -0.011134038679301739,
+            -0.004745970480144024,
+            0.016743261367082596,
+            0.006695002317428589,
+            0.0006618269253522158,
+            0.018277931958436966,
+            0.005352165549993515,
+            -0.0005002068355679512,
+            0.005336818750947714,
+            0.013106090947985649,
+            0.0012517160503193736,
+            0.024048294872045517,
+            -0.01617543213069439,
+            0.00997536163777113,
+            -0.014333827421069145,
+            0.033701375126838684,
+            -0.01597592607140541,
+            -0.0016027720412239432,
+            0.012177614495158195,
+            0.040883637964725494,
+            -0.013497431762516499,
+            0.01712692901492119,
+            -0.0291280560195446,
+            -0.015354383736848831,
+            0.03465287387371063,
+            0.01688138209283352,
+            0.010988244786858559,
+            -0.010359029285609722,
+            -0.01332094520330429,
+            0.008801338262856007,
+            0.01470982190221548,
+            0.008824358694255352,
+            -0.0241403765976429,
+            0.02908201701939106,
+            -0.014571702107787132,
+            0.04162028059363365,
+            -0.0001700127759249881,
+            -0.003117300570011139,
+            0.023204226046800613,
+            -0.0009577306336723268,
+            -0.02265174500644207,
+            0.014947696588933468,
+            0.00268183765001595,
+            0.01203182153403759,
+            -0.01066596433520317,
+            0.010527843609452248,
+            0.023787401616573334,
+            0.006537698674947023,
+            -0.005018374416977167,
+            -7.163796544773504e-05,
+            0.016912074759602547,
+            0.0005409715231508017,
+            -0.005033721216022968,
+            0.027731506153941154,
+            -0.007481521461158991,
+            -0.010842450894415379,
+            -0.01875368133187294,
+            -0.0012171859852969646,
+            -0.018247239291667938,
+            -0.00043474353151395917,
+            -0.007700212299823761,
+            0.0008320794440805912,
+            -0.010589230805635452,
+            -0.015154876746237278,
+            -0.003944104537367821,
+            -0.025076525285840034,
+            0.03480634093284607,
+            0.009676100686192513,
+            0.013597185723483562,
+            0.009200353175401688,
+            0.00365059869363904,
+            -0.000640245561953634,
+            0.0214547012001276,
+            -0.0030175470747053623,
+            0.0005649508093483746,
+            -0.005685956217348576,
+            -0.0119857806712389,
+            -0.0122390016913414,
+            0.013980853371322155,
+            -0.008317917585372925,
+            -0.008640198037028313,
+            -0.00844069104641676,
+            -0.01821654476225376,
+            -0.03477564454078674,
+            -0.0013610613532364368,
+            0.008893419057130814,
+            -0.028360720723867416,
+            0.014909329824149609,
+            -0.006441781762987375,
+            -0.01600661873817444,
+            0.0008009064476937056,
+            0.006161704193800688,
+            0.018554173409938812,
+            -0.007972615770995617,
+            -0.03425385802984238,
+            0.031384024769067764,
+            0.01646701991558075,
+            0.003899982897564769,
+            0.0033820313401520252,
+            0.008010982535779476,
+            0.008064696565270424,
+            -0.01395783293992281,
+            -0.0099139753729105,
+            0.028115173801779747,
+            -0.026212181895971298,
+            0.0239715613424778,
+            0.012561283074319363,
+            0.0003963767667300999,
+            0.027240412309765816,
+            -0.019474975764751434,
+            0.02630426175892353,
+            -0.0076042949222028255,
+            -5.1765175157925114e-05,
+            0.015200916677713394,
+            0.02099430002272129,
+            0.020319044589996338,
+            0.02272847853600979,
+            -0.02265174500644207,
+            0.038643017411231995,
+            -0.0023787401150912046,
+            0.03201323747634888,
+            0.006307498086243868,
+            -0.00015094928676262498,
+            0.008486730977892876,
+            0.008172123692929745,
+            -0.017679410055279732,
+            -0.0010176787618547678,
+            -0.03069342114031315,
+            0.010604577139019966,
+            -0.005229391623288393,
+            -0.01867694780230522,
+            -0.020794792100787163,
+            -0.033916231244802475,
+            0.010888490825891495,
+            0.001221981830894947,
+            -0.013037030585110188,
+            -0.004742133431136608,
+            0.0078076389618217945,
+            0.01306772418320179,
+            0.022713132202625275,
+            -0.0037100673653185368,
+            0.0032688493374735117,
+            0.019643789157271385,
+            -0.005298451986163855,
+            0.004331609234213829,
+            -0.01617543213069439,
+            0.02211461029946804,
+            0.005555509123951197,
+            0.030754808336496353,
+            0.02476959116756916,
+            0.008540444076061249,
+            0.004611686337739229,
+            -0.004204998724162579,
+            0.017817530781030655,
+            -0.01600661873817444,
+            -0.010481803677976131,
+            -0.001481916755437851,
+            -0.0038366776425391436,
+            0.06037395820021629,
+            -0.024692857638001442,
+            -0.02506117895245552,
+            -0.005559345707297325,
+            -0.011487012729048729,
+            0.006675818935036659,
+            -0.03247363865375519,
+            -0.028928549960255623,
+            -0.023511160165071487,
+            -0.0175873301923275,
+            -0.0019624605774879456,
+            -0.009998382069170475,
+            0.014049913734197617,
+            -0.00569362984970212,
+            0.011778600513935089,
+            -0.0239715613424778,
+            0.000355372263584286,
+            -0.016651181504130363,
+            -0.005685956217348576,
+            -0.014026893302798271,
+            -0.0070403036661446095,
+            -0.000628735579084605,
+            0.02949637733399868,
+            0.011824640445411205,
+            -0.019229428842663765,
+            -0.0032918695360422134,
+            0.012745442800223827,
+            -0.008279550820589066,
+            -0.007362584583461285,
+            0.012246674858033657,
+            0.01916804164648056,
+            -0.012208308093249798,
+            0.037937067449092865,
+            -0.0037983108777552843,
+            -0.00038582590059377253,
+            -0.005904647056013346,
+            -0.012584302574396133,
+            0.024877017363905907,
+            0.028314681723713875,
+            -0.03984006121754646,
+            -0.0013303679879754782,
+            0.0018694212194532156,
+            0.007600458338856697,
+            -0.005321471951901913,
+            0.0030866072047501802,
+            -0.027117637917399406,
+            -0.0012776136863976717,
+            -0.026872090995311737,
+            -0.01573037914931774,
+            -0.021470047533512115,
+            -0.005923830438405275,
+            0.0032458293717354536,
+            -0.023050758987665176,
+            0.030094899237155914,
+            0.02077944576740265,
+            -0.014487294480204582,
+            -0.0038596978411078453,
+            -0.003500968450680375,
+            0.008126083761453629,
+            -0.03189046308398247,
+            -0.004362302366644144,
+            -0.009745161049067974,
+            -9.597688767826185e-05,
+            -0.00962238758802414,
+            -0.04198860004544258,
+            0.0012239001225680113,
+            -0.003167177317664027,
+            -0.0017955651273950934,
+            -0.008494404144585133,
+            -0.016727915033698082,
+            0.02177698165178299,
+            0.020303698256611824,
+            -0.010880817659199238,
+            -0.008532770909368992,
+            0.045610424131155014,
+            0.008425343781709671,
+            0.0016411389224231243,
+            -0.010888490825891495,
+            0.002762407995760441,
+            0.0028314681258052588,
+            0.008486730977892876,
+            -0.013528125360608101,
+            0.011463993228971958,
+            -0.005298451986163855,
+            0.014118974097073078,
+            -0.0067065125331282616,
+            -0.016114046797156334,
+            0.023204226046800613,
+            -0.02829933539032936,
+            0.000396856339648366,
+            -0.010174869559705257,
+            -0.001934644766151905,
+            0.01395783293992281,
+            0.010259276255965233,
+            0.01726504974067211,
+            -0.03953312709927559,
+            -0.004815030377358198,
+            -0.008386977016925812,
+            -0.0004716715484391898,
+            -0.0059468504041433334,
+            -0.026994863525032997,
+            0.013443718664348125,
+            0.0023557201493531466,
+            -0.010880817659199238,
+            0.018323972821235657,
+            -0.004515769425779581,
+            -0.009960015304386616,
+            0.022375503554940224,
+            -0.0019020329928025603,
+            0.015362057834863663,
+            0.029235484078526497,
+            -0.016789302229881287,
+            0.0183700118213892,
+            0.032504335045814514,
+            -0.009369166567921638,
+            -0.0037139039486646652,
+            -0.0028142030350863934,
+            0.0009936995338648558,
+            0.016328901052474976,
+            0.017525942996144295,
+            0.0007447950774803758,
+            -0.03228947892785072,
+            0.00829489715397358,
+            -0.00218306970782578,
+            0.007918902672827244,
+            0.009514960460364819,
+            -0.014318481087684631,
+            -0.000949098146520555,
+            -0.025997327640652657,
+            0.006952059920877218,
+            -0.0209482591599226,
+            -0.009422880597412586,
+            0.024171069264411926,
+            -0.017786838114261627,
+            -0.019152695313096046,
+            0.003688965691253543,
+            0.02837606891989708,
+            -0.029680538922548294,
+            0.01130285207182169,
+            0.004139775410294533,
+            -0.005812566727399826,
+            0.006361211650073528,
+            0.000759182614274323,
+            -0.01895318739116192,
+            0.004500423092395067,
+            -0.009507287293672562,
+            -0.012169941328465939,
+            -0.020257657393813133,
+            -0.013835059478878975,
+            -0.0006699798395857215,
+            0.014617742039263248,
+            0.015822459012269974,
+            0.028851816430687904,
+            0.0016516897594556212,
+            -0.0069405497051775455,
+            -0.014656108804047108,
+            0.006879162974655628,
+            -0.0064801485277712345,
+            0.012990990653634071,
+            -0.008709258399903774,
+            0.02535276673734188,
+            -0.00811073649674654,
+            -0.0006037971470504999,
+            -0.010497150011360645,
+            -0.013198170810937881,
+            0.008762971498072147,
+            -0.003247747663408518,
+            0.0017994018271565437,
+            -0.011732560582458973,
+            0.004354629199951887,
+            0.01692742109298706,
+            -0.01863090693950653,
+            -0.02177698165178299,
+            -0.023449774831533432,
+            0.021976489573717117,
+            0.03766082972288132,
+            0.0009236801415681839,
+            -0.050214435905218124,
+            -0.0021063359454274178,
+            -0.007677191868424416,
+            -0.012077861465513706,
+            -0.02265174500644207,
+            0.011487012729048729,
+            0.018139811232686043,
+            -0.0008004268747754395,
+            -0.0019893173594027758,
+            0.0031978709157556295,
+            0.015108836814761162,
+            -0.0013207762967795134,
+            0.0053866952657699585,
+            0.00251494231633842,
+            -0.0075007048435509205,
+            0.004155121743679047,
+            0.029941432178020477,
+            -0.027731506153941154,
+            0.00941520743072033,
+            0.002618532394990325,
+            0.011755580082535744,
+            0.004669236950576305,
+            0.009177332744002342,
+            0.01470982190221548,
+            0.0032400742638856173,
+            -0.00709401722997427,
+            0.0196591354906559,
+            0.07206815481185913,
+            -0.0028314681258052588,
+            0.015385077334940434,
+            0.00972981471568346,
+            -0.012822176329791546,
+            -0.004040021449327469,
+            -0.010312989354133606,
+            -0.03848955035209656,
+            0.025398805737495422,
+            0.008034002967178822,
+            -0.012077861465513706,
+            0.006483985111117363,
+            -0.014126647263765335,
+            0.008878071792423725,
+            0.029020629823207855,
+            0.010366703383624554,
+            -0.020794792100787163,
+            0.004281732253730297,
+            0.03453009948134422,
+            0.0214547012001276,
+            0.004542626440525055,
+            -0.01780218444764614,
+            0.006219254340976477,
+            -0.0014195707626640797,
+            -0.0038712078239768744,
+            0.009614714421331882,
+            -0.004465892910957336,
+            0.022375503554940224,
+            -0.004078388214111328,
+            0.014111299999058247,
+            -0.013535798527300358,
+            -0.013911793008446693,
+            0.006115664262324572,
+            -0.001624832977540791,
+            0.0014790392015129328,
+            0.020150231197476387,
+            0.0028314681258052588,
+            -0.006760226096957922,
+            0.0024861672427505255,
+            -0.0005850933375768363,
+            0.013681592419743538,
+            0.004818866960704327,
+            0.007423971313983202,
+            0.006614432204514742,
+            -0.023050758987665176,
+            0.01879972033202648,
+            0.006434108596295118,
+            0.00976818148046732,
+            -0.013037030585110188,
+            -0.005682119634002447,
+            -0.005720486398786306,
+            -0.004757480230182409,
+            0.0018454419914633036,
+            -0.00196437886916101,
+            -0.01573037914931774,
+            -0.004435199312865734,
+            0.009960015304386616,
+            -0.024662164971232414,
+            0.0048764171078801155,
+            0.010842450894415379,
+            0.004991517867892981,
+            0.0009601286146789789,
+            -0.01124146580696106,
+            -0.03256572037935257,
+            5.1435461500659585e-05,
+            -0.005647589452564716,
+            -0.0021255193278193474,
+            0.004853397142142057,
+            0.010919184423983097,
+            0.0222373828291893,
+            0.004396832548081875,
+            -0.0013831222895532846,
+            0.021715596318244934,
+            0.0007064282544888556,
+            -0.0033014612272381783,
+            0.016866035759449005,
+            -0.00735491095110774,
+            -0.013129111379384995,
+            0.00012517160212155432,
+            0.003560436889529228,
+            0.014755862765014172,
+            0.0057358331978321075,
+            0.01617543213069439,
+            0.010351356118917465,
+            -0.02555227279663086,
+            -0.020840832963585854,
+            9.196036990033463e-05,
+            0.026319608092308044,
+            -0.012653362937271595,
+            -0.01912200078368187,
+            -0.00412826519459486,
+            -0.005187188275158405,
+            0.006925203371793032,
+            -0.012231328524649143,
+            0.008977825753390789,
+            0.016451673582196236,
+            -0.011364239268004894,
+            -0.026273569092154503,
+            -0.020472511649131775,
+            -0.005847096908837557,
+            0.009775854647159576,
+            0.009814221411943436,
+            -0.006741042714565992,
+            0.012983317486941814,
+            0.002706776140257716,
+            -0.021761635318398476,
+            0.022835904732346535,
+            0.013489758595824242,
+            0.005689792800694704,
+            -0.005229391623288393,
+            -0.004523443058133125,
+            -0.011479339562356472,
+            0.0068753263913095,
+            0.004070715047419071,
+            -0.007293524220585823,
+            -0.030923621729016304,
+            -0.0024880855344235897,
+            -0.006537698674947023,
+            -0.00125747115816921,
+            -0.009514960460364819,
+            0.00602742051705718,
+            -0.029972126707434654,
+            0.004020838066935539,
+            -0.0060926442965865135,
+            -0.0041819787584245205,
+            -0.006449454929679632,
+            -0.0009423839510418475,
+            -0.009238719940185547,
+            -0.028237948194146156,
+            -0.027271104976534843,
+            0.0011807375121861696,
+            0.01199345476925373,
+            -0.0016392205143347383,
+            -0.002380658406764269,
+            -0.024247802793979645,
+            -0.0004181978583801538,
+            0.008640198037028313,
+            -0.00744699127972126,
+            0.01846209354698658,
+            -0.003890391206368804,
+            -0.006952059920877218,
+            -0.0004949314170517027,
+            -0.0012392469216138124,
+            -0.026749316602945328,
+            0.017080888152122498,
+            0.0031000354792922735,
+            0.014326154254376888,
+            -0.02435522899031639,
+            0.011279832571744919,
+            -0.013366985134780407,
+            0.007780782412737608,
+            -0.01841605268418789,
+            -0.008762971498072147,
+            -0.00786518957465887,
+            -0.012392468750476837,
+            -0.007826822809875011,
+            -0.029143404215574265,
+            0.00844069104641676,
+            -0.008801338262856007,
+            -0.025813167914748192,
+            -0.02908201701939106,
+            0.009353820234537125,
+            -0.015776418149471283,
+            0.015561564825475216,
+            -0.009215699508786201,
+            0.04527279734611511,
+            0.020027456805109978,
+            0.015354383736848831,
+            0.008824358694255352,
+            -0.017863571643829346,
+            -0.0032937878277152777,
+            0.012653362937271595,
+            -0.026273569092154503,
+            0.0003601681091822684,
+            0.012553608976304531,
+            0.01487096305936575,
+            -0.011318199336528778,
+            0.001353388070128858,
+            0.01651306077837944,
+            0.015177897177636623,
+            0.01622147299349308,
+            0.0024036786053329706,
+            -0.009131292812526226,
+            0.010305316187441349,
+            0.023633934557437897,
+            0.02567504718899727,
+            -0.004803520627319813,
+            0.0019662973936647177,
+            0.0011625132756307721,
+            -0.018477439880371094,
+            -0.009077579714357853,
+            -0.013205844908952713,
+            -0.009783527813851833,
+            -0.02659584954380989,
+            0.011525379493832588,
+            -0.0038002291694283485,
+            -0.010013728402554989,
+            -0.01580711267888546,
+            -0.014978389255702496,
+            -0.005133474711328745,
+            0.006599085405468941,
+            0.021546781063079834,
+            -0.010712004266679287,
+            0.010919184423983097,
+            -0.010052095167338848,
+            0.0020545409061014652,
+            -0.00196437886916101,
+            0.0081491032615304,
+            0.027225064113736153,
+            0.010274622589349747,
+            -0.055524397641420364,
+            -0.006499331910163164,
+            0.002762407995760441,
+            -0.010650617070496082,
+            -0.051503561437129974,
+            0.002656899392604828,
+            0.007324217818677425,
+            -0.03185977041721344,
+            0.008655544370412827,
+            -0.0023154348600655794,
+            -0.0079265758395195,
+            0.006595248822122812,
+            0.019935376942157745,
+            -0.016206126660108566,
+            0.013259558007121086,
+            -6.013542588334531e-06,
+            0.004320099018514156,
+            -0.014924676157534122,
+            0.0062230913899838924,
+            -0.015147203579545021,
+            -0.014471948146820068,
+            0.01734178327023983,
+            -0.001127024064771831,
+            -0.002089071087539196,
+            0.012008801102638245,
+            0.02506117895245552,
+            0.02579781971871853,
+            -0.012162268161773682,
+            -0.01592988520860672,
+            -0.01879972033202648,
+            0.0007073874585330486,
+            -0.0055401623249053955,
+            -0.010712004266679287,
+            -0.0019624605774879456,
+            0.008417670615017414,
+            0.01209320779889822,
+            0.01987398974597454,
+            0.008417670615017414,
+            0.006315171252936125,
+            -3.35109798470512e-05,
+            0.009898628108203411,
+            0.0023422916419804096,
+            -0.020303698256611824,
+            -0.0065530454739928246,
+            0.012983317486941814,
+            0.019797256216406822,
+            0.0017936468357220292,
+            0.004297079052776098,
+            0.018185852095484734,
+            -0.00773857906460762,
+            -0.009208026342093945,
+            -0.0336092971265316,
+            -0.0006901223678141832,
+            -0.017203662544488907,
+            0.016236819326877594,
+            0.01945962943136692,
+            -0.02136262133717537,
+            -0.01022858265787363,
+            -0.01866159960627556,
+            0.02394086867570877,
+            -0.010735023766756058,
+            0.005912320222705603,
+            0.0022252730559557676,
+            0.008739951997995377,
+            0.0019068288384005427,
+            -0.008517424575984478,
+            -0.014272441156208515,
+            0.009277086704969406,
+            0.00048318158951587975,
+            0.007661845069378614,
+            -0.0061195008456707,
+            -0.012039494700729847,
+            0.001056045526638627,
+            -0.009860261343419552,
+            -0.028283987194299698,
+            0.011579093523323536,
+            0.0008670891402289271,
+            0.01821654476225376,
+            -0.019597750157117844,
+            0.01675860770046711,
+            0.0007380808820016682,
+            -0.013520452193915844,
+            -0.02788497321307659,
+            -0.015277650207281113,
+            0.003919166047126055,
+            0.003792555769905448,
+            0.012661036103963852,
+            0.0034817850682884455,
+            -0.01658979430794716,
+            0.008739951997995377,
+            -0.005647589452564716,
+            -0.005751179996877909,
+            -0.001697729923762381,
+            -0.01170186698436737,
+            -0.001447386690415442,
+            0.011072651483118534,
+            -0.01037437655031681,
+            -0.005574692506343126,
+            0.01468680240213871,
+            0.006411088164895773,
+            -0.002988772001117468,
+            0.02742457203567028,
+            0.01821654476225376,
+            0.02153143472969532,
+            0.0006488781073130667,
+            0.01066596433520317,
+            -0.0018473602831363678,
+            0.003558518597856164,
+            -0.03505188599228859,
+            0.025122566148638725,
+            -0.01576107181608677,
+            0.023373039439320564,
+            0.008532770909368992,
+            -0.00694822333753109,
+            -0.005881627090275288,
+            0.01331327110528946,
+            0.006088807247579098,
+            -0.004803520627319813,
+            0.00602742051705718,
+            -0.003128810552880168,
+            0.009844915010035038,
+            -0.0013102254597470164,
+            -0.02493840456008911,
+            0.020012110471725464,
+            -0.024416616186499596,
+            0.009990708902478218,
+            -0.002990690292790532,
+            0.009123619645833969,
+            -0.006894509773701429,
+            -5.931863415753469e-05,
+            -0.00457331957295537,
+            -0.040638089179992676,
+            -0.001970133977010846,
+            -0.010773390531539917,
+            0.004450546111911535,
+            0.016819994896650314,
+            -0.00021497384295798838,
+            -0.04220345616340637,
+            -0.005916156806051731,
+            0.01958240196108818,
+            -0.00021737176575697958,
+            0.009883281774818897,
+            -0.008271876722574234,
+            -0.004086061846464872,
+            -0.004320099018514156,
+            -0.007792292162775993,
+            0.016543753445148468,
+            0.013136784546077251,
+            -0.01563829742372036,
+            -0.023618588224053383,
+            0.012507569044828415,
+            -0.01879972033202648,
+            -0.019567055627703667,
+            0.00530996173620224,
+            -0.01267638336867094,
+            0.001553854439407587,
+            0.022268077358603477,
+            -0.004715276882052422,
+            0.011164732277393341,
+            -0.0056130592711269855,
+            0.013282578438520432,
+            -0.01543111726641655,
+            -0.0019471138948574662,
+            -0.0047651538625359535,
+            -0.02348046749830246,
+            0.025567619130015373,
+            -0.018937841057777405,
+            0.0019931539427489042,
+            -0.016528407111763954,
+            0.02228342369198799,
+            -0.020088844001293182,
+            -0.002275149803608656,
+            0.008010982535779476,
+            0.010174869559705257,
+            0.029803311452269554,
+            0.01345906499773264,
+            -0.007408624514937401,
+            0.009169659577310085,
+            0.008824358694255352,
+            0.019827950745821,
+            0.00156632368452847,
+            0.029803311452269554,
+            -0.005271594971418381,
+            0.01041274331510067,
+            -0.001517405966296792,
+            0.018968533724546432,
+            0.0067065125331282616,
+            0.029926085844635963,
+            0.0003174850717186928,
+            0.00015430638450197875,
+            -0.0001380004978273064,
+            0.018600212410092354,
+            -0.004369975998997688,
+            -0.005313798785209656,
+            0.007765435613691807,
+            -0.0059698703698813915,
+            -0.0015797520754858851,
+            -0.020441818982362747,
+            -0.005877790041267872,
+            0.001410938217304647,
+            0.013789019547402859,
+            -0.0005241860635578632,
+            0.005390532314777374,
+            0.0008200898300856352,
+            0.005586202722042799,
+            -0.014617742039263248,
+            0.011402606032788754,
+            -0.015668991953134537,
+            0.01883041486144066,
+            -0.01005976926535368,
+            0.01008278876543045,
+            -0.0009692406747490168,
+            0.00811073649674654,
+            0.011072651483118534,
+            -0.013574165292084217,
+            0.01614473946392536,
+            0.012139247730374336,
+            -0.010589230805635452,
+            0.016313552856445312,
+            -0.007228300906717777,
+            0.005083597730845213,
+            0.0038616161327809095,
+            0.0005870116874575615,
+            0.003153749043121934,
+            -0.018109118565917015,
+            -0.009461247362196445,
+            -0.0001405183138558641,
+            -0.012576629407703876,
+            -0.0030501587316393852,
+            0.012331082485616207,
+            -0.005505632609128952,
+            0.015507850795984268,
+            0.010197889059782028,
+            0.012285041622817516,
+            0.004933967255055904,
+            0.011172405444085598,
+            -0.0014186116168275476,
+            -0.0034299897961318493,
+            0.004684583283960819,
+            0.032964736223220825,
+            0.0008824358810670674,
+            0.025905247777700424,
+            -0.014978389255702496,
+            0.006084970664232969,
+            0.001282409531995654,
+            -0.0016181188402697444,
+            -0.008701585233211517,
+            0.016236819326877594,
+            0.020088844001293182,
+            0.0018934004474431276,
+            -0.011893700808286667,
+            -0.003499049926176667,
+            0.029404297471046448,
+            -0.005501795560121536,
+            0.007454664912074804,
+            0.010696657001972198,
+            -0.006196234375238419,
+            0.016605140641331673,
+            0.004381485749036074,
+            -0.024692857638001442,
+            0.0013457146706059575,
+            0.004047695081681013,
+            -0.019152695313096046,
+            0.020917566493153572,
+            0.02104034088551998,
+            0.0076042949222028255,
+            0.002739387797191739,
+            0.01345906499773264,
+            0.010497150011360645,
+            0.021623514592647552,
+            0.0032400742638856173,
+            0.004580993205308914,
+            0.002643470885232091,
+            -0.012952623888850212,
+            -0.00123445107601583,
+            -0.020963607355952263,
+            -0.011533052660524845,
+            -0.0020583774894475937,
+            0.018983881920576096,
+            0.0026530625764280558,
+            -0.013290251605212688,
+            -0.003959451336413622,
+            0.008855052292346954,
+            0.004381485749036074,
+            -0.006299824919551611,
+            -0.022053223103284836,
+            -0.008202817291021347,
+            0.024094335734844208,
+            0.008095390163362026,
+            0.010550864040851593,
+            -0.0031729324255138636,
+            -0.002160049509257078,
+            -0.0006109909154474735,
+            -0.006614432204514742,
+            -0.0006450414075516164,
+            -0.01597592607140541,
+            -0.0021063359454274178,
+            -0.016666527837514877,
+            0.015346710570156574,
+            0.015791764482855797,
+            0.01763337105512619,
+            0.018139811232686043,
+            -0.02102499268949032,
+            0.017602676525712013,
+            -0.0071438937447965145,
+            0.001838727737776935,
+            -0.0034414997790008783,
+            0.00010826624929904938,
+            0.019275467842817307,
+            0.029388951137661934,
+            0.0056015495210886,
+            0.011149385012686253,
+            -0.0013831222895532846,
+            0.005927667021751404,
+            -0.006457128562033176,
+            -0.013980853371322155,
+            -0.0033935413230210543,
+            -0.01892249472439289,
+            0.004907110705971718,
+            0.017848225310444832,
+            -0.015139530412852764,
+            -0.004922457505017519,
+            0.014349174685776234,
+            0.009867934510111809,
+            0.008340937085449696,
+            -0.008931785821914673,
+            -0.004112918395549059,
+            0.0008714054129086435,
+            0.01557691115885973,
+            -0.007585111539810896,
+            -0.016605140641331673,
+            0.008394651114940643,
+            -0.006146357394754887,
+            0.02123984694480896,
+            -0.004542626440525055,
+            -0.01605265960097313,
+            -0.010282295756042004,
+            0.003186360700055957,
+            -0.021439354866743088,
+            -0.014272441156208515,
+            0.01002140250056982,
+            0.0021312744356691837,
+            0.006936713121831417,
+            -0.017326436936855316,
+            0.013060051016509533,
+            0.004834213759750128,
+            0.03351721540093422,
+            -0.008133756928145885,
+            -0.0005424103001132607,
+            0.0003779127437155694,
+            0.005916156806051731,
+            0.0002481850970070809,
+            -0.01755663752555847,
+            0.026933476328849792,
+            0.006587575655430555,
+            0.013781346380710602,
+            -0.0003863054735120386,
+            -0.012645689770579338,
+            0.006994263269007206,
+            -0.01564597152173519,
+            0.001863666228018701,
+            -0.010205562226474285,
+            -0.020088844001293182,
+            -0.01058155670762062,
+            0.02165420912206173,
+            0.0070403036661446095,
+            -0.00732038076967001,
+            -0.0001907548139570281,
+            -0.0077769458293914795,
+            -0.016359593719244003,
+            -0.009852588176727295,
+            -0.015492504462599754,
+            0.008386977016925812,
+            -0.01663583517074585,
+            0.0029005282558500767,
+            0.008126083761453629,
+            0.009806548245251179,
+            0.005586202722042799,
+            0.021623514592647552,
+            0.002338455058634281,
+            0.016405634582042694,
+            0.0001647373428568244,
+            -0.0019049104303121567,
+            0.005651426035910845,
+            0.014771209098398685,
+            0.0072973608039319515,
+            -0.008555791340768337,
+            -0.0046423799358308315,
+            -0.012691729702055454,
+            0.0005198698490858078,
+            -0.009676100686192513,
+            0.01066596433520317,
+            0.0064302715472877026,
+            -0.0009404656011611223,
+            -0.00678324606269598,
+            0.021715596318244934,
+            -0.0038980646058917046,
+            0.007711722049862146,
+            -0.007957269437611103,
+            -0.004300915636122227,
+            0.01065829023718834,
+            0.0010848206002265215,
+            -0.013765999116003513,
+            0.008210490457713604,
+            0.00919268000870943,
+            -0.002263639820739627,
+            -0.003238155972212553,
+            -0.020672019571065903,
+            0.02874438837170601,
+            0.008885745890438557,
+            -0.010182542726397514,
+            0.014425908215343952,
+            -0.0052831051871180534,
+            0.019306162372231483,
+            0.016282860189676285,
+            0.0185388270765543,
+            -0.018323972821235657,
+            -0.019643789157271385,
+            0.007251320872455835,
+            -0.015561564825475216,
+            -0.0027835096698254347,
+            0.0029715069103986025,
+            -0.0022809049114584923,
+            -0.0002340373321203515,
+            -0.016068005934357643,
+            0.0009150475962087512,
+            -0.0046500531025230885,
+            0.0024938404094427824,
+            0.000545287795830518,
+            0.0077347420156002045,
+            0.0009395064553245902,
+            0.012760790064930916,
+            -0.006196234375238419,
+            -0.01783287711441517,
+            -0.006606759037822485,
+            0.013512779027223587,
+            0.006906019989401102,
+            -0.03600338473916054,
+            0.004980007652193308,
+            -0.012499895878136158,
+            -0.01751059666275978,
+            -0.002232946455478668,
+            -0.02157747559249401,
+            -0.02116311341524124,
+            -0.001316939597018063,
+            0.0009831486968323588,
+            -0.011510033160448074,
+            0.0021869062911719084,
+            0.006648962385952473,
+            -0.009300106205046177,
+            -0.00919268000870943,
+            0.0054595922119915485,
+            -0.0018751762108877301,
+            0.012645689770579338,
+            -0.012392468750476837,
+            -0.010443436913192272,
+            0.035174660384655,
+            0.013635552488267422,
+            -0.00766568211838603,
+            -0.013812039978802204,
+            -0.010873144492506981,
+            -0.014280114322900772,
+            0.010980571620166302,
+            -0.0030827706214040518,
+            -0.00466539990156889,
+            -0.037814296782016754,
+            0.0013543472159653902,
+            0.0011884109117090702,
+            0.01524695660918951,
+            -0.008862725459039211,
+            0.014533335343003273,
+            0.005793383345007896,
+            -0.00058077706489712,
+            -0.012354101985692978,
+            -6.549328645633068e-06,
+            0.01080408412963152,
+            0.010136502794921398,
+            -0.007197607308626175,
+            0.012100880965590477,
+            -0.013006336987018585,
+            0.007527561392635107,
+            -0.014633088372647762,
+            -0.0077769458293914795,
+            -0.01933685503900051,
+            -0.01787891797721386,
+            -0.007516051642596722,
+            0.01634424738585949,
+            0.009890954941511154,
+            -0.01787891797721386,
+            -0.004815030377358198,
+            -0.014525661244988441,
+            0.013451391831040382,
+            -0.011533052660524845,
+            -0.006015910767018795,
+            -0.01306772418320179,
+            0.001044535543769598,
+            -0.018784373998641968,
+            0.01385807991027832,
+            0.01775614358484745,
+            0.01639028638601303,
+            0.0004093255556654185,
+            -0.0053866952657699585,
+            -0.02248293161392212,
+            -0.01870764046907425,
+            -0.004462056327611208,
+            0.002572492463514209,
+            -0.013781346380710602,
+            -0.003744597313925624,
+            0.011732560582458973,
+            0.021915102377533913,
+            -0.0017984426813200116,
+            -0.013129111379384995,
+            0.00854811817407608,
+            -0.016958115622401237,
+            -0.017311088740825653,
+            -0.018277931958436966,
+            0.007358747534453869,
+            0.008172123692929745,
+            0.01605265960097313,
+            0.0024746570270508528,
+            0.006058114115148783,
+            -0.004623196553438902,
+            -0.007730905432254076,
+            -0.024278495460748672,
+            0.005110454745590687,
+            -0.0112030990421772,
+            0.010075115598738194,
+            -0.0021408661268651485,
+            -0.004070715047419071,
+            0.023465121164917946,
+            0.009760508313775063,
+            -0.004995354451239109,
+            -0.00018631866259966046,
+            0.020840832963585854,
+            -0.008686237968504429,
+            -0.00602742051705718,
+            -0.008057023398578167,
+            0.017771491780877113,
+            -0.021961143240332603,
+            0.00016090067219920456,
+            -0.011187751777470112,
+            -0.0222373828291893,
+            0.003272686153650284,
+            0.012200634926557541,
+            0.00606578728184104,
+            -0.003355174558237195,
+            0.003453009994700551,
+            -0.008433017879724503,
+            0.024094335734844208,
+            -0.01435684785246849,
+            0.002348046749830246,
+            0.013620206154882908,
+            -0.0021638860926032066,
+            -0.002465065335854888,
+            0.013727632351219654,
+            -0.005417388863861561,
+            0.009530307725071907,
+            0.0035738651640713215,
+            0.00011857732170028612,
+            -0.009660754352807999,
+            0.016988808289170265,
+            0.02535276673734188,
+            -0.00860950443893671,
+            0.0008152939844876528,
+            -0.003782964078709483,
+            0.00602742051705718,
+            -0.0070403036661446095,
+            0.03201323747634888,
+            -0.008287223987281322,
+            0.0005500836414285004,
+            0.0010100053623318672,
+            0.03161422535777092,
+            -0.0014771209098398685,
+            0.010044422000646591,
+            0.0014541008276864886,
+            -0.0034184798132628202,
+            0.022927986457943916,
+            0.0025667373556643724,
+            0.004531116224825382,
+            0.027102291584014893,
+            -0.013574165292084217,
+            -0.00493780430406332,
+            0.005543999373912811,
+            -0.015200916677713394,
+            0.012392468750476837,
+            -0.02048785798251629,
+            0.0009188842959702015,
+            -0.01522393710911274,
+            0.014280114322900772,
+            0.019981417804956436,
+            0.030386487022042274,
+            -0.014418235048651695,
+            -0.00619239779189229,
+            -0.00728201400488615,
+            0.03143006190657616,
+            0.00044217711547389627,
+            0.006288314703851938,
+            -0.00030045982566662133,
+            0.011471666395664215,
+            -0.015177897177636623,
+            -0.01962844282388687,
+            0.012499895878136158,
+            -0.019551709294319153,
+            -0.03413108363747597,
+            -0.004634706769138575,
+            0.005670609418302774,
+            -0.0030175470747053623,
+            0.0037100673653185368,
+            0.012146921828389168,
+            0.004358465783298016,
+            0.022145302966237068,
+            0.007243647240102291,
+            0.009752834215760231,
+            -0.016482368111610413,
+            3.229703224860714e-06,
+            -0.006848469842225313,
+            -0.025398805737495422,
+            0.016037313267588615,
+            -0.021516088396310806,
+            0.0014138157712295651,
+            -0.008433017879724503,
+            -0.002647307701408863,
+            0.009967688471078873,
+            0.004780500195920467,
+            0.014778882265090942,
+            0.005789546761661768,
+            0.016282860189676285,
+            0.019551709294319153,
+            -0.01573037914931774,
+            0.011586766690015793,
+            0.027025558054447174,
+            0.00113853404764086,
+            -0.017188316211104393,
+            -0.007784618996083736,
+            -0.01975121721625328,
+            -0.01945962943136692,
+            0.011256812140345573,
+            0.02041112445294857,
+            -0.00846371054649353,
+            -0.001863666228018701,
+            0.03904203325510025,
+            -0.013113764114677906,
+            0.023956215009093285,
+            -0.00775776244699955,
+            0.005240901838988066,
+            0.03468356654047966,
+            0.010827104561030865,
+            0.018692294135689735,
+            -0.00613868422806263,
+            -0.034407325088977814,
+            -0.002121682744473219,
+            -0.00680242944508791,
+            0.0065607186406850815,
+            -0.011049631983041763,
+            0.018032385036349297,
+            -0.013673919253051281,
+            -0.019812602549791336,
+            0.02036508545279503,
+            -0.02243689075112343,
+            0.021377967670559883,
+            0.007562091574072838,
+            0.008578810840845108,
+            -0.006269131321460009,
+            -0.011479339562356472,
+            -3.974558421759866e-05,
+            -0.0077270688489079475,
+            0.011318199336528778,
+            0.008034002967178822,
+            -0.011832313612103462,
+            -0.006806266028434038,
+            0.0031729324255138636,
+            -0.004634706769138575,
+            -0.02311214618384838,
+            0.012860543094575405,
+            0.008640198037028313,
+            0.013428371399641037,
+            0.007381767965853214,
+            0.005075924564152956,
+            0.010635270737111568,
+            0.0034817850682884455,
+            -0.013113764114677906,
+            -0.01203182153403759,
+            0.00914663914591074,
+            -0.006127174012362957,
+            -0.0007332850364036858,
+            -0.026687929406762123,
+            0.014702148735523224,
+            -0.022743824869394302,
+            -0.0032573393546044827,
+            -0.0018195443553850055,
+            0.0039786347188055515,
+            0.005486449226737022,
+            -0.01866159960627556,
+            -0.005636079702526331,
+            -0.00028990895953029394,
+            -0.009392186999320984,
+            0.01188602764159441,
+            -0.013620206154882908,
+            0.01646701991558075,
+            0.027056250721216202,
+            -0.01487096305936575,
+            -0.0036199053283780813,
+            0.00554783595725894,
+            0.0077040488831698895,
+            -0.012522916309535503,
+            0.016988808289170265,
+            0.015108836814761162,
+            -0.01205484103411436,
+            0.008847379125654697,
+            0.03083154186606407,
+            -0.006579902023077011,
+            0.0005088393809273839,
+            -0.00012493181566242129,
+            0.0024995955172926188,
+            0.029726577922701836,
+            0.0016066087409853935,
+            0.024293843656778336,
+            0.005873953457921743,
+            -0.0008862725226208568,
+            -0.014610068872570992,
+            -0.014364521019160748,
+            -0.009522633627057076,
+            -0.009990708902478218,
+            -0.005520978942513466,
+            -0.010497150011360645,
+            0.009706794284284115,
+            0.012699402868747711,
+            -0.004676910117268562,
+            0.0038884729146957397,
+            -0.0019164205295965075,
+            0.02131658047437668,
+            -0.005052904598414898,
+            0.013113764114677906,
+            -0.022206690162420273,
+            0.01962844282388687,
+            0.020595286041498184,
+            0.011839987710118294,
+            -0.016313552856445312,
+            -0.005643752869218588,
+            0.016819994896650314,
+            0.025214646011590958,
+            -0.00538285868242383,
+            0.001968215685337782,
+            0.019444283097982407,
+            0.0031384022440761328,
+            0.005252411589026451,
+            -0.0026626542676240206,
+            -0.01987398974597454,
+            -0.027194371446967125,
+            -0.003568110289052129,
+            -0.006737206131219864,
+            0.002689511049538851,
+            0.004097571596503258,
+            -0.011839987710118294,
+            -0.011609786190092564,
+            0.0046500531025230885,
+            -0.005026047583669424,
+            0.0011020856909453869,
+            -0.0099139753729105,
+            -0.007849842309951782,
+            -0.038059841841459274,
+            0.020150231197476387,
+            0.0004304272879380733,
+            0.004396832548081875,
+            0.004239528905600309,
+            0.015108836814761162,
+            -0.007723232265561819,
+            -0.0016229146858677268,
+            0.015085816383361816,
+            0.006069623865187168,
+            -0.0018233810551464558,
+            -0.0003510560200084001,
+            -0.0035719468723982573,
+            -0.007197607308626175,
+            0.0104741295799613,
+            -0.004980007652193308,
+            -0.010274622589349747,
+            0.0084023242816329,
+            -0.012852869927883148,
+            -0.015062796883285046,
+            0.008225836791098118,
+            -0.025153258815407753,
+            0.00501070125028491,
+            -0.000793712679296732,
+            0.010159522294998169,
+            -0.004661563318222761,
+            -0.008809012360870838,
+            0.0010032912250608206,
+            0.013873426243662834,
+            0.007032630033791065,
+            -0.004726787097752094,
+            0.016083352267742157,
+            -0.007400951348245144,
+            0.007124710362404585,
+            0.029220137745141983,
+            -0.007949596270918846,
+            0.008601831272244453,
+            0.018477439880371094,
+            0.0013687347527593374,
+            0.008471384644508362,
+            0.01130285207182169,
+            -0.03643308952450752,
+            -0.016743261367082596,
+            -0.0008809971041046083,
+            0.0051603312604129314,
+            -0.0011174323735758662,
+            0.0188764538615942,
+            0.00919268000870943,
+            0.02725575864315033,
+            -0.006177050992846489,
+            0.012684056535363197,
+            0.0019509505946189165,
+            0.023572547361254692,
+            0.012108555063605309,
+            0.01199345476925373,
+            -0.005747342947870493,
+            -0.0021312744356691837,
+            -0.03588061034679413,
+            0.0035661919973790646,
+            0.022175997495651245,
+            0.019490322098135948,
+            -0.02579781971871853,
+            0.017203662544488907,
+            -0.0057435063645243645,
+            0.0033129712101072073,
+            -0.013428371399641037,
+            0.017694758251309395,
+            0.004392995964735746,
+            0.017571983858942986,
+            -0.008156776428222656,
+            -0.002417106879875064,
+            0.016866035759449005,
+            0.010466456413269043,
+            -0.012914257124066353,
+            0.021470047533512115,
+            0.012806829996407032,
+            0.02119380794465542,
+            0.019198734313249588,
+            -0.018308626487851143,
+            0.00908525288105011,
+            0.00875529833137989,
+            -0.012200634926557541,
+            -0.0081491032615304,
+            0.00457331957295537,
+            0.007232137490063906,
+            -0.016605140641331673,
+            -0.005018374416977167,
+            -0.0005678283050656319,
+            0.00023475672060158104,
+            0.012768463231623173,
+            -0.012668709270656109,
+            0.000462079857243225,
+            0.00412826519459486,
+            0.011080324649810791,
+            -0.0065607186406850815,
+            0.01734178327023983,
+            -0.028729042038321495,
+            0.012223655357956886,
+            -0.022022530436515808,
+            -0.0239715613424778,
+            0.0014195707626640797,
+            -0.004481239710003138,
+            -0.0036333338357508183,
+            -0.0051027811132371426,
+            0.0015154876746237278,
+            0.005647589452564716,
+            0.009983035735785961,
+            0.028391415253281593,
+            0.007715558633208275,
+            0.0013601022073999047,
+            -0.0004165193240623921,
+            -0.02302006632089615,
+            -0.010827104561030865,
+            -0.006054277531802654,
+            -0.0021389478351920843,
+            -0.006445618346333504,
+            -0.0336092971265316,
+            -0.00022852211259305477,
+            -0.0057358331978321075,
+            0.0029216299299150705,
+            0.0008435894851572812,
+            -0.02007349766790867,
+            -0.021500742062926292,
+            0.00926941353827715,
+            -0.008870398625731468,
+            -0.014026893302798271,
+            -0.0168046485632658,
+            0.006472475361078978,
+            0.00037431585951708257,
+            -0.0036563538014888763,
+            0.010151849128305912,
+            0.01846209354698658,
+            -0.004676910117268562,
+            -0.0006843673763796687,
+            0.010090461932122707,
+            0.0026396343018859625,
+            -0.02165420912206173,
+            0.008617177605628967,
+            -0.014111299999058247,
+            0.003951778169721365,
+            -0.01130285207182169,
+            -0.01709623634815216,
+            -0.004964660853147507,
+            -0.003967124503105879,
+            -0.028391415253281593,
+            0.0011673092376440763,
+            0.011832313612103462,
+            -0.007067160215228796,
+            0.013635552488267422,
+            0.020441818982362747,
+            0.010405070148408413,
+            0.00084119156235829,
+            -0.014863288961350918,
+            0.011425626464188099,
+            -0.01457937527447939,
+            -0.012998663820326328,
+            0.01970517635345459,
+            0.012929603457450867,
+            0.0112030990421772,
+            -0.0008503036806359887,
+            -0.014019220136106014,
+            0.011072651483118534,
+            0.015745725482702255,
+            0.025245338678359985,
+            -0.006272967904806137,
+            -0.0007697334513068199,
+            -0.0162982065230608,
+            -0.018446745350956917,
+            0.034161776304244995,
+            -0.011932067573070526,
+            -0.013681592419743538,
+            0.01655910164117813,
+            0.019613096490502357,
+            0.012668709270656109,
+            -0.005616896320134401,
+            -0.004684583283960819,
+            -0.0014684883644804358,
+            0.012346428819000721,
+            0.005820239894092083,
+            -0.025199299678206444,
+            -0.010681310668587685,
+            0.004400669131428003,
+            0.0325964130461216,
+            0.011747906915843487,
+            0.006069623865187168,
+            -0.043001484125852585,
+            0.014172687195241451,
+            0.0008853133767843246,
+            0.0009505368652753532,
+            0.013980853371322155,
+            0.029481031000614166,
+            0.020257657393813133,
+            0.0009706794517114758,
+            0.01962844282388687,
+            -0.015914538875222206,
+            0.02269778586924076,
+            -0.020978953689336777,
+            -0.010919184423983097,
+            -0.0241403765976429,
+            -0.005006864201277494,
+            -0.008479057811200619,
+            0.0028257130179554224,
+            0.009031538851559162,
+            0.009637733921408653,
+            -0.012262022122740746,
+            0.02021161839365959,
+            0.013221191242337227,
+            -0.010044422000646591,
+            -0.001552895293571055,
+            -0.02070271223783493,
+            0.02547553926706314,
+            0.003771454095840454,
+            -0.0008680483442731202,
+            -0.010612250305712223,
+            -0.005018374416977167,
+            -0.021132420748472214,
+            0.0018368094461038709,
+            -0.0028506515081971884,
+            0.012906583957374096,
+            -0.0016085271490737796,
+            0.008348610252141953,
+            0.016942769289016724,
+            -0.015208589844405651,
+            -0.003878881223499775,
+            0.007846006192266941,
+            0.02104034088551998,
+            0.006437945179641247,
+            0.01190904714167118,
+            0.02854488231241703,
+            -0.0013936731265857816,
+            -0.027485959231853485,
+            0.012008801102638245,
+            -0.006403414998203516,
+            -0.003366684541106224,
+            0.01111869141459465,
+            0.02131658047437668,
+            -0.021746288985013962,
+            0.0011557991383597255,
+            -0.007335727568715811,
+            0.020150231197476387,
+            3.1173007300822064e-05,
+            0.013198170810937881,
+            0.009138965979218483,
+            0.003936431370675564,
+            -0.01111869141459465,
+            -0.01421105396002531,
+            -0.008026329800486565,
+            0.01170186698436737,
+            -0.026979517191648483,
+            -0.0024036786053329706,
+            0.001851196982897818,
+            -0.003021383658051491,
+            0.006311334669589996,
+            0.006230764556676149,
+            -0.022099263966083527,
+            -0.010635270737111568,
+            -0.0028564066160470247,
+            -0.010182542726397514,
+            -0.005206371657550335,
+            0.012476875446736813,
+            -0.005409715697169304,
+            0.0005798179190605879,
+            0.00875529833137989,
+            0.009630060754716396,
+            0.004300915636122227,
+            0.025506233796477318,
+            -0.012584302574396133,
+            0.01924477517604828,
+            -0.030647380277514458,
+            0.001827217754907906,
+            -0.008578810840845108,
+            0.024171069264411926,
+            -0.0024497187696397305,
+            -0.019075961783528328,
+            0.012047167867422104,
+            -0.008195143193006516,
+            0.001614282140508294,
+            -0.002810366451740265,
+            -0.008433017879724503,
+            -0.0065300255082547665,
+            0.012545935809612274,
+            -0.018155159428715706,
+            0.004308588802814484,
+            0.025444846600294113,
+            0.0429094024002552,
+            -0.013643225654959679,
+            -0.013121437281370163,
+            0.0052255550399422646,
+            0.009983035735785961,
+            0.024094335734844208,
+            -0.01144864596426487,
+            -0.009307780303061008,
+            -0.01159443985670805,
+            0.0075812749564647675,
+            -0.016896728426218033,
+            -0.023833442479372025,
+            0.011563746258616447,
+            -0.015960579738020897,
+            -0.016850687563419342,
+            -0.018155159428715706,
+            -0.017955651506781578,
+            -0.0027662445791065693,
+            0.013773673214018345,
+            0.005904647056013346,
+            -0.0028046113438904285,
+            -0.0015615278389304876,
+            -0.006111827678978443,
+            -0.021270541474223137,
+            0.016896728426218033,
+            0.005133474711328745,
+            -0.01614473946392536,
+            -0.004373812582343817,
+            -0.012039494700729847,
+            0.002562900772318244,
+            0.0037522707134485245,
+            -0.003075097221881151,
+            0.05506399646401405,
+            0.003518233308568597,
+            0.009507287293672562,
+            -0.02107103355228901,
+            0.004266385454684496,
+            -0.010796410962939262,
+            -0.01074269786477089,
+            -0.007707885466516018,
+            -0.012116228230297565,
+            -0.003617987036705017,
+            -0.011479339562356472,
+            0.0019969907589256763,
+            -0.01543111726641655,
+            0.008133756928145885,
+            -0.013673919253051281,
+            -0.013474412262439728,
+            -0.017571983858942986,
+            -0.0015864662127569318,
+            -0.003451091470196843,
+            0.016114046797156334,
+            0.01325188484042883,
+            -0.016374940052628517,
+            -0.0030904437880963087,
+            0.022268077358603477,
+            0.007895882241427898,
+            0.018968533724546432,
+            -0.0035048050340265036,
+            -0.013129111379384995,
+            -0.020012110471725464,
+            0.0006565514486283064,
+            0.013075397349894047,
+            -0.0075812749564647675,
+            0.002915875054895878,
+            -0.029097363352775574,
+            -0.006748715881258249,
+            0.0014090199256315827,
+            0.01600661873817444,
+            0.01668187417089939,
+            0.008862725459039211,
+            -0.0018732579192146659,
+            -0.020856179296970367,
+            0.0332716703414917,
+            -0.007753925397992134,
+            -0.022743824869394302,
+            0.0019758890848606825,
+            0.007328054402023554,
+            -0.005144984927028418,
+            0.0024036786053329706,
+            0.013351637870073318,
+            0.004903274122625589,
+            -0.008340937085449696,
+            -0.004887927323579788,
+            0.0011836150661110878,
+            -0.021715596318244934,
+            0.01838536001741886,
+            -0.0033263994846493006,
+            0.00854811817407608,
+            -0.004838050343096256,
+            0.004696093499660492,
+            -0.023158187046647072,
+            0.014126647263765335,
+            -0.008164450526237488,
+            -0.0069098565727472305,
+            0.008878071792423725,
+            -0.0023653118405491114,
+            0.02231411822140217,
+            -0.020273005589842796,
+            -0.008908765390515327,
+            0.008855052292346954,
+            -0.022129956632852554,
+            -0.02206856943666935,
+            -0.0004939722712151706,
+            -0.019275467842817307,
+            -0.0014780800556764007,
+            -0.00021233613369986415,
+            -0.032964736223220825,
+            -0.026856742799282074,
+            0.003044403623789549,
+            -0.0073472377844154835,
+            -0.008287223987281322,
+            0.012753116898238659,
+            -7.697334513068199e-05,
+            -0.013850406743586063,
+            -0.003665945492684841,
+            0.01041274331510067,
+            0.00023139962286222726,
+            0.01800169236958027,
+            -0.021930448710918427,
+            0.0020372758153826,
+            0.004431362729519606,
+            -0.004327772185206413,
+            0.006725695915520191,
+            0.01712692901492119,
+            0.0005990013014525175,
+            0.0003234798787161708,
+            0.006084970664232969,
+            0.0034932950511574745,
+            0.010850124061107635,
+            0.015745725482702255,
+            0.015032103285193443,
+            -0.004174305126070976,
+            0.016068005934357643,
+            -0.001623873831704259,
+            2.7096535632153973e-05,
+            -0.02460077777504921,
+            2.6736846848507412e-05,
+            0.012415489181876183,
+            0.004450546111911535,
+            -0.02410968206822872,
+            -0.0031997892074286938,
+            -0.03278057277202606,
+            0.005133474711328745,
+            -0.019382895901799202,
+            0.0013802447356283665,
+            0.017571983858942986,
+            0.00728201400488615,
+            -0.004446709528565407,
+            0.014402887783944607,
+            0.022789865732192993,
+            -0.0010502905352041125,
+            0.0006623064982704818,
+            0.004515769425779581,
+            -0.003224727464839816,
+            0.002537962282076478,
+            0.0022080079652369022,
+            -0.01597592607140541,
+            -0.01414966769516468,
+            0.008156776428222656,
+            0.004611686337739229,
+            0.011310526169836521,
+            0.007385604549199343,
+            -0.015446464531123638,
+            0.012223655357956886,
+            -0.013934813439846039,
+            0.008164450526237488,
+            -0.00473829684779048,
+            -0.013988526538014412,
+            -0.013466738164424896,
+            0.005359838716685772,
+            0.017618024721741676,
+            -0.006012073718011379,
+            0.033425137400627136,
+            -0.00925406627357006,
+            0.01533136423677206,
+            -0.019306162372231483,
+            -0.009491940960288048,
+            -0.010282295756042004,
+            0.01184766087681055,
+            -0.01567666418850422,
+            -0.0009020988363772631,
+            0.0035508451983332634,
+            -0.010573883540928364,
+            -0.010351356118917465,
+            0.005294615402817726,
+            -0.0030309753492474556,
+            0.03956381976604462,
+            -0.0024631470441818237,
+            0.025705739855766296,
+            0.006031257100403309,
+            -0.011517706327140331,
+            -0.010075115598738194,
+            0.002643470885232091,
+            0.00800330936908722,
+            0.0180937722325325,
+            -0.006568392273038626,
+            -0.004354629199951887,
+            0.011041957885026932,
+            0.017955651506781578,
+            0.007251320872455835,
+            0.011563746258616447,
+            -0.004093735013157129,
+            0.0037311690393835306,
+            0.002374903531745076,
+            -0.013666246086359024,
+            -0.0031096271704882383,
+            0.011916721239686012,
+            0.017019502818584442,
+            0.00786518957465887,
+            0.012852869927883148,
+            -0.0003946982033085078,
+            0.004611686337739229,
+            0.013996199704706669,
+            -0.013121437281370163,
+            -0.005240901838988066,
+            0.008241184055805206,
+            -0.01721900887787342,
+            0.009223372675478458,
+            0.0044083427637815475,
+            -0.004638543352484703,
+            0.00893945898860693,
+            -0.005348328500986099,
+            -0.0018483194289729,
+            -0.02099430002272129,
+            -0.014134320430457592,
+            -0.0033494196832180023,
+            0.020932912826538086,
+            -0.006299824919551611,
+            0.0015452218940481544,
+            -0.006790919695049524,
+            -0.02567504718899727,
+            0.01705019548535347,
+            0.004550299607217312,
+            0.0065530454739928246,
+            0.005966033786535263
+          ],
+          "title": "2 Related Work",
+          "keyphrases": [
+            "neural network capacity",
+            "model parallelism",
+            "conditional computation",
+            "mixture of experts",
+            "Switch Transformer"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "3a64eee4-da9a-458d-b832-4f399202000c",
+      "type": "child",
+      "source": {
+        "id": "b570d71e-4f83-486e-b70f-b0c6217db455",
+        "properties": {
+          "page_content": "Mixture-Of-Experts With Expert Choice Routing\n\nYanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon Google, Mountain View, CA, USA {yanqiz, taole, hanxiaol, dunan, huangyp, vzhao, adai, zhifengc, qvl, jlaudon}@google.com\n\nAbstract\n\nSparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choice method. Instead of letting tokens select the top-k experts, we have experts selecting the top-k tokens. As a result, each token can be routed to a variable number of experts and each expert can have a fixed bucket size. We systematically study pre-training speedups using the same computational resources of the Switch Transformer top-1 and GShard top-2 gating of prior work and find that our method improves training convergence time by more than 2×. For the same computational cost, our method demonstrates higher performance in fine-tuning 11 selected tasks in the GLUE and SuperGLUE benchmarks. For a smaller activation cost, our method outperforms the T5 dense model in 7 out of the 11 tasks.\n\n1 Introduction\n\nScaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27]. The final model quality has been found to have a power-law relationship with the amount of data, model size, and compute time [16, 20]. However, training efficiency, which is defined as the total amount of computation used to achieve superior model quality than the state of the art system [21], should receive greater attention as we increase our efforts towards green AI [29].\n\nSparsely gated mixture-of-experts [31] (MoE) provides an effective way to scale model capacity given a fixed computational cost, and has recently played an important role in increasing the training efficiency of large-scale language models [10, 21]. MoE operate by adopting a number of experts, each as a sub-network, and by activating only one or a few experts for each input token. A gating network must be chosen and optimized in order to route each token to the most suited expert(s). For example, recent work has implemented sparse routing via k-means clustering [12], linear assignment to maximize token-expert affinities [22], or hashing [8, 28]. Many of the prior work use a routing strategy concerning the token choice, where each token selects the best one or two experts.\n\nWe argue that the independent token choice of prior work often leads to an imbalanced load of experts, which causes training inefficiency and sub-optimal training of the model. In order to mitigate this\n\nissue, previous sparsely gated networks introduce additional auxiliary losses as regularization to prevent too many tokens being routed to a single expert, but the effectiveness is still limited. Recent approaches [8, 22, 28] explore alternative strategies for routing, but they focus on pre-training only and do not demonstrate performance gain on downstream tasks. Moreover, none of the previous methods consider allocating a variable number of experts to each token based on importance, which can be beneficial.\n\nWe propose a very simple yet effective routing method we are calling expert choice. Unlike conventional MoE where tokens select one or two top-scoring experts, our method lets each expert pick the top-k tokens. Our method guarantees perfect load balancing, allows a variable number of experts for each token, and achieves substantial gains in training efficiency and downstream performance as demonstrated in our experiments. Our major contributions include:\n\nWe identify common pitfalls in conventional MoE such as load imbalance as described in Section 3.1. We then propose a heterogeneous, expert choice method to provide a fluid allocation of model parameters based on a learnt token-to-expert importance. This method intrinsically guarantees load balance without imposing an auxiliary loss.\n\nWe show our method provides over 2× faster training convergence in a 8B/64E (8 billion activated parameters, 64 experts) model, compared to the top-1 and top-2 gating counterparts in Switch Transformer [10] and GShard [21].\n\nWe show our method demonstrates strong scaling when increasing the number of experts from 16 to 128, evaluated in training perplexity.\n\nWe show our method demonstrates strong performance on downstream tasks selected from GLUE and SuperGLUE at all the evaluated scales. More specifically, our 8B/64E model outperforms a T5 11B dense model in 7 out of 11 tasks evaluated.\n\n2 Related Work\n\nScaling: Various approaches have been proposed to scale up neural network capacity to improve performance. Recent works have successfully scaled models to billions of parameters via various forms of model parallelism [2, 21, 26, 27, 33]. Model parallelism [30] splits weights and tensors across multiple cores while pipeline parallelism [18, 24] splits different layers across devices with micro-batches pipelined to the different layers. To enable continued scaling of neural networks, improving model training and serving efficiency has become a critical research area.\n\nConditional Computation: Computation decisions can be made dynamically based on the input [23, 25]. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation, by activating certain parameters and computation on demand, on a per-example or per-token basis [3]. Conditional convolution layers [1] with task-specific gating has been used to combat catastrophic forgetting when a sequence of learning problems are optimized. The gating decisions may be binary or sparse and continuous, stochastic or deterministic.\n\nMixture of Experts: Sparsely-gated MoE [31] is the first model to demonstrate massive improvements in model capacity, training time, or model quality with gating. Switch Transformer [10] simplifies the gating by selecting only the top expert per token using a softmax over the hidden state and demonstrates better scaling than previous work. All the prior work requires an auxiliary loss to explicitly encourage balancing. This loss term has to be carefully weighted to not overwhelm the primary loss. However, auxiliary loss does not guarantee balancing and a hard capacity factor has to be imposed. As a result, many tokens can still be unprocessed by the MoE layer. Hard MoE [12] with a single decoding layer can be efficiently trained to good effect on large scale hashtag prediction tasks.\n\nBase Layers [22] formulate a linear assignment that maximizes token-expert affinities while ensuring each expert receives an equal number of tokens. Hash layers [8, 28] devise hashing techniques on input tokens. However, the evaluations are limited to pre-training perplexity. THOR [? ] randomly activates experts during training and inference and is trained with a consistency regularization loss.\n\nTHOR has demonstrated strong performance on translation tasks. Different from these prior works, our method is a learnt method that enables heterogeneous MoE and effectively improves downstream fine-tuning performance.\n\n3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n5 Conclusion\n\nWe propose a new routing method for sparsely activated mixture-of-experts (MoE) models. This method addresses load imbalance and under-utilization of experts in conventional MoE methods, and enables selecting different numbers of experts for each token. Our model demonstrates more than 2x training efficiency improvements when compared to the state-of-the-art GShard and Switch Transformer models, and also achieves strong gains when finetuning on 11 datasets in the GLUE and SuperGLUE benchmark.\n\n6 Limitations\n\nThe expert choice method might not immediately apply to auto-regressive text generation as our current implementation takes in the past and future tokens to perform the top-k selection. One possible solution is to collect a large batch of input sequences, dispatch tokens of the same sequence into separate groups, and perform expert choice routing for each group. Another scenario where the expert choice method does not immediately apply is when the batch size becomes very small during serving or inference. A global top-k can be selected instead and we can cap the number of times each expert or token gets selected. We leave these possible improvements for future work.\n\nAnother long-standing issue with MoE has been the large memory footprint. Even though computational cost can be reduced using sparsely gated networks, the total number of parameters increases linearly or sub-linearly with the number of experts. Increasing the number of experts requires reservation of a large number of hardware devices. Therefore, dynamic (used) power is saved while static (reserved) power is not. Power saving techniques such as the ability to put hardware devices into low power states while not in use [17] can help with reducing the reserved power requirements.\n\nReferences\n\n[1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pages 3930–3939. Computer Vision Foundation / IEEE, 2020.\n\n[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. In Advances in Neural Information Processing Systems.\n\n[3] Kyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning, 2014.\n\n[4] Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems, 2021.\n\n[5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.\n\nTransformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.\n\n[6] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 933–941. JMLR.org, 2017.\n\n[7] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixtureof-experts, 2021.\n\n[8] Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan.\n\nTricks for training sparse translation models, 2021.\n\n[9] Richard L Dykstra. An iterative procedure for obtaining i-projections onto the intersection of convex sets. The annals of Probability, pages 975–984, 1985.\n\n[10] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.\n\n[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. NAS-FPN: learning scalable feature pyramid architecture for object detection. In CVPR, pages 7036–7045. Computer Vision Foundation / IEEE, 2019.\n\n[12] Sam Gross, Marc'Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision, 2017.\n\n[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.\n\n[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 630–645, Cham, 2016. Springer International Publishing.\n\n[15] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2016.\n\n[16] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.\n\n[17] Ping Huang, Zuocheng Xing, Tianran Wang, Qiang Wei, Hongyan Wang, and Guitao Fu. A brief survey on power gating design. In 2010 10th IEEE International Conference on Solid-State and Integrated Circuit Technology, pages 788–790, 2010.\n\n[18] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 103–112, 2019.\n\n[19] Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David A. Patterson. A domain-specific supercomputer for training deep neural networks. Commun. ACM, 63(7):67–78, 2020.\n\n[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.\n\n[21] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.\n\n[22] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6265–6274. PMLR, 18–24 Jul 2021.\n\n[24] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. New York, NY, USA, 2019. Association for Computing Machinery.\n\n[25] Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, Cédric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. In ICLR. OpenReview.net, 2021.\n\n[26] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.\n\n[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.\n\n[28] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models, 2021.\n\n[30] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 10435–10444, Red Hook, NY, USA, 2018. Curran Associates Inc.\n\n[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E.\n\nHinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In ICLR (Poster). OpenReview.net, 2017.\n\n[32] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 10–15 Jul 2018. [23] Min Lin, Jie Fu, and Yoshua Bengio. Conditional computation for continual learning, 2019.\n\n[29] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.\n\n[33] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.\n\n[34] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc.\n\n[35] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.\n\nGLUE: A multi-task benchmark and analysis platform for natural language understanding.\n\nIn Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, November 2018. Association for Computational Linguistics.\n\n[36] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake A. Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen. GSPMD: general and scalable parallelization for ML computation graphs. CoRR, abs/2105.04663, 2021.\n\n7 Checklist\n\n(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Yes (b) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes (c) Did you discuss any potential negative societal impacts of your work? N/A. Not any. (d) Did you describe the limitations of your work? Yes (a) Did you include the code, data, and instructions needed to reproduce the main experimental results? Yes. We include details in the experiment setup to help reproduce the main results. (b) Did you specify all the training details? Yes (c) Did you report error bars? Yes (d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? Yes (a) If your work uses existing assets, did you cite the creators? Yes (b) Did you mention the license of the assets? No. The used dataset is not released yet. (c) Did you include any new assets either in the supplemental material or as a URL? No. The dataset is not released yet.\n\n(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? No. Not using persons' data. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes. The dataset does not contain any personally identifiable information or offensive content.\n\nA Comparison On Fine-Tuning With A Dense Model\n\nOur 8B MoE model achieves stronger pre-training perplexity than its dense counterpart. However, a better perplexity does not always directly translate to downstream performance as demonstrated in Section 4.4. To this end, we compare fine-tuning performance of the 8B dense model and MoE model in Table 1. As shown in the table, our MoE model using expert choice routing consistently outperforms the dense model across the 11 tasks in GLUE and SuperGLUE.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 1: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nB Capacity Factor\n\nWe evaluate the downstream task fine-tuning performance by varying the capacity factors. Note that a capacity factor of n indicates on average how many experts each token can be received. EC-CF2 is our baseline expert choice, which matches GShard top-2 gating computational footprint. EC-CF1, however, matches Switch Transformer top-1 gating computational footprint. EC-CF0.5 further verifies that an aggressively lowered capacity factor can provide strong enough performance, that almost matches the top-2 gating baseline.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Top-2 78.1 87.0 88.3 85.0 82.6 90.1 90.7 81.6 94.7 68.2 67.2 83.0±0.3 EC-CAP2 78.2 88.0 88.5 85.7 83.0 90.8 91.1 80.0 95.4 70.4 64.1 83.2±0.4 EC-CAP3 78.5 91.7 89.3 86.3 83.5 90.9 91.1 81.8 94.9 70.0 65.6 84.0±0.4 EC-CF2 79.1 89.6 89.3 86.8 84.3 91.3 91.2 81.1 95.2 68.1 68.0 84.0±0.2 EC-CF1 77.4 90.6 88.0 85.5 83.6 90.3 91.2 79.8 95.3 66.5 64.9 83.0±0.2 EC-CF0.5 77.4 89.6 86.3 85.2 82.7 91.7 91.0 79.6 94.9 67.3 63.5 83.0 ±0.05 Hash Layers 76.1 85.2 86.7 83.4 82.5 90.0 90.3 75.7 94.0 67.4 63.3 81.3±1.0\n\nTable 2: Comparison between different routing methods in fine-tuning of 100M/64E models. We perform 3 independent fine-tuning runs for each method and report the average results. This gives more accurate difference between the variants of expert choice method, since they achieve close fine-tuning results. We do not report averaged results in other experiments.\n\nC Capped Expert Choice\n\nAs described in Section 4.5, the maximum number of experts each token is assigned can be capped by an entropy-regularized linear programming. Figure 1 compares the validation perplexity when training the 100M/64E models using the base expert choice method (EC-BASE), expert choice capped by two experts per token (EC-CAP2), expert choice capped by three experts per token (EC-CAP3), and GShard top-2 gating.\n\nAs shown in the figure, restricting the number of experts to 2 degrades the perplexity compared to the base expert choice method. This suggests that a more flexible allocation of experts (e.g. more than 2 experts for a token) can enhance model expressiveness. On the other hand, our EC-CAP2 and EC-CAP3 methods still outperform the top-2 gating method by a clear margin. We believe this confirms the effectiveness of a load balanced training, provided by our method. Finally, EC-CAP3 obtains comparable perplexity to EC-BASE. As indicated by Figure 3, only a little fraction of tokens use more than 3 experts therefore we see little or no difference between EC-BASE and EC-CAP3 variants. We present the fine-tuning results of these methods in Table 2.\n\nD Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [? ]. We use mod x to map a token ID to an expert ID. This in some way ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 2. Hashing based routing performs much worse than expert choice in terms of average scores and variance.\n\nE Fine-Tuning Details\n\nWe did a hyperparameter search for both baseline models and expert choice method. For fine-tuning of the 8B dense model, we use a constant learning rate of 0.0001 and a dropout rate of 0.1. We freeze the attention layer and feed-forward layer while leaving the embedding and layer normalization trainable. This setting has been found optimal for the 8B dense model. For MoE 8B/64E models including GShard top-2 gating and expert choice, we found continuing the learning rate from the pre-trained model while using a square root learning rate decay works better. In addition, we do not apply parameter freezing for fine-tuning MoE models. For models with 100M expert size, we use a constant learning rate of 0.0001 and no dropout is used.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/2202.09368v2.md"
+          },
+          "headlines": {
+            "1 Introduction": [],
+            "2 Related Work": [],
+            "3 Method": [
+              "3.1 Pitfalls Of Token-Choice Routing",
+              "3.2 Heterogeneous Moe Via Expert Choice",
+              "3.3 Expert Choice With Additional Constraint",
+              "3.4 Model Architecture"
+            ],
+            "4 Experiments": [
+              "4.1 Setup",
+              "4.2 Training Efficiency",
+              "4.3 Scaling The Number Of Experts",
+              "4.4 Fine-Tuning On Glue And Superglue",
+              "4.5 Heterogeneity Matters",
+              "4.6 Comparison With Hash Layer",
+              "4.7 Ablation"
+            ],
+            "5 Conclusion": [],
+            "6 Limitations": [],
+            "A Comparison On Fine-Tuning With A Dense Model": [],
+            "B Capacity Factor": [],
+            "C Capped Expert Choice": [],
+            "D Comparison With Hash Layer": [],
+            "E Fine-Tuning Details": []
+          },
+          "summary": "The paper introduces a new routing method for sparsely-activated Mixture-of-Experts (MoE) models called 'expert choice,' which addresses load imbalance and under-utilization issues in conventional MoE methods. Unlike traditional token-choice routing, expert choice allows experts to select the top-k tokens, ensuring perfect load balancing and enabling variable expert allocation per token. This method improves training efficiency, achieving over 2x faster convergence compared to existing models like Switch Transformer and GShard. It also demonstrates superior performance on downstream tasks in the GLUE and SuperGLUE benchmarks. The study highlights the benefits of flexible expert allocation and shows that expert choice outperforms hashing-based routing methods. The paper also discusses potential limitations and future improvements for the expert choice method.",
+          "summary_embedding": [
+            -0.009726986289024353,
+            -0.0337262786924839,
+            -0.023302342742681503,
+            -0.019878201186656952,
+            -0.025953780859708786,
+            0.04560471698641777,
+            -0.006874797400087118,
+            0.039665497839450836,
+            -0.006973279174417257,
+            0.034544438123703,
+            -0.025787118822336197,
+            -0.00363057479262352,
+            -0.01052999310195446,
+            0.0054657477885484695,
+            -0.024499278515577316,
+            0.036786794662475586,
+            0.010007280856370926,
+            0.003272630739957094,
+            0.006102093029767275,
+            -0.04048365354537964,
+            -0.018666116520762444,
+            -0.013522328808903694,
+            -0.03230207785964012,
+            0.008969432674348354,
+            0.02484775334596634,
+            0.0016713523073121905,
+            0.0026230288203805685,
+            0.003624893259257078,
+            -0.05311964824795723,
+            0.04339266195893288,
+            0.018681267276406288,
+            0.023711422458291054,
+            -0.016332851722836494,
+            -0.012832955457270145,
+            -0.039120059460401535,
+            -0.0197418425232172,
+            0.030786970630288124,
+            0.039968520402908325,
+            -0.019287310540676117,
+            0.03357476741075516,
+            0.018060073256492615,
+            0.02454473078250885,
+            -0.021272100508213043,
+            -0.029817303642630577,
+            -0.0061892117373645306,
+            -0.007840678095817566,
+            -0.008355814032256603,
+            0.01242387667298317,
+            0.016984347254037857,
+            -0.013007191941142082,
+            0.0212569497525692,
+            -0.01371171697974205,
+            0.03384748846292496,
+            0.008916404098272324,
+            -0.042332086712121964,
+            0.03984731063246727,
+            -0.008007340133190155,
+            0.012105703353881836,
+            0.03281721472740173,
+            0.010333028621971607,
+            -0.005287723150104284,
+            -0.021544819697737694,
+            -0.03378688171505928,
+            0.0010529992869123816,
+            0.00014014738553669304,
+            0.005950582213699818,
+            -0.003069985192269087,
+            -0.009742137044668198,
+            0.03227177634835243,
+            -0.003939277958124876,
+            -0.0740584209561348,
+            0.050089433789253235,
+            -0.026590125635266304,
+            -0.007628562860190868,
+            -0.028938541188836098,
+            -0.01736312545835972,
+            0.037362534552812576,
+            0.005662711802870035,
+            0.0098709212616086,
+            0.0106284748762846,
+            -0.0007892759749665856,
+            0.028923390433192253,
+            0.00910579226911068,
+            -0.024923507124185562,
+            0.04478655755519867,
+            -0.04790768027305603,
+            0.01801462098956108,
+            -0.04336235672235489,
+            0.022756904363632202,
+            1.6393929399782792e-05,
+            -0.020726662129163742,
+            -0.0013304532039910555,
+            -0.016666175797581673,
+            0.0218023881316185,
+            -0.01730252057313919,
+            0.026802239939570427,
+            0.013825350441038609,
+            -0.015802564099431038,
+            -0.029817303642630577,
+            0.025378039106726646,
+            0.003367325058206916,
+            0.010499690659344196,
+            -0.02075696364045143,
+            0.014332910999655724,
+            0.04393810033798218,
+            -0.02593863010406494,
+            -0.02271145209670067,
+            0.027681002393364906,
+            0.014408666640520096,
+            0.0633314698934555,
+            0.036544378846883774,
+            0.006624804809689522,
+            0.022362977266311646,
+            0.020575150847434998,
+            -0.028968842700123787,
+            -0.01963578537106514,
+            -0.046362269669771194,
+            -0.04757435619831085,
+            0.019878201186656952,
+            -0.04566532000899315,
+            0.0050869714468717575,
+            0.038544319570064545,
+            -0.013704141601920128,
+            -0.014446543529629707,
+            -0.04021093621850014,
+            0.017847958952188492,
+            0.0003908502112608403,
+            0.011976920068264008,
+            -0.0022082682698965073,
+            0.00033285003155469894,
+            -0.011825408786535263,
+            0.008878526277840137,
+            -0.03778676688671112,
+            -0.025196226313710213,
+            0.025181075558066368,
+            0.022862963378429413,
+            0.024120500311255455,
+            0.028802180662751198,
+            -0.03077181987464428,
+            -0.009984553791582584,
+            0.021378157660365105,
+            -0.04242299124598503,
+            -0.006556625012308359,
+            0.011332998983561993,
+            0.0026703758630901575,
+            0.003872992005199194,
+            0.032786913216114044,
+            0.052998438477516174,
+            -0.03172633796930313,
+            0.02065090648829937,
+            -0.010431510396301746,
+            0.04957429692149162,
+            0.00024691506405360997,
+            0.028968842700123787,
+            -0.019514575600624084,
+            -0.0134389977902174,
+            -0.029529431834816933,
+            -0.009795165620744228,
+            -0.01356020662933588,
+            -0.008484598249197006,
+            -0.002998017705976963,
+            -0.009575475007295609,
+            0.0040453351102769375,
+            0.0331808403134346,
+            0.00273855566047132,
+            0.030680913478136063,
+            0.014325335621833801,
+            0.0051475754007697105,
+            -0.08745196461677551,
+            0.026059838011860847,
+            -0.0033616432920098305,
+            0.005859675817191601,
+            0.0062460280023515224,
+            0.03013547509908676,
+            -0.0312718041241169,
+            0.020938776433467865,
+            0.026620427146553993,
+            0.0011145505122840405,
+            0.016969196498394012,
+            0.036847397685050964,
+            -0.04181694984436035,
+            0.009643655270338058,
+            0.012590537779033184,
+            -0.0033881578128784895,
+            0.04584713280200958,
+            0.011628445237874985,
+            0.019514575600624084,
+            0.018529755994677544,
+            0.003569970605894923,
+            0.00297907879576087,
+            0.006533898413181305,
+            0.03724132478237152,
+            0.003462019143626094,
+            -0.009833043441176414,
+            0.014976831153035164,
+            -0.010757259093225002,
+            -0.011219366453588009,
+            -0.037998881191015244,
+            0.014454119838774204,
+            0.008742166683077812,
+            -0.017454031854867935,
+            -0.02959003672003746,
+            0.0026949963066726923,
+            -0.031908150762319565,
+            0.026559824123978615,
+            -0.002445003716275096,
+            0.005738467443734407,
+            -0.010454237461090088,
+            -0.006757376715540886,
+            -0.035423196852207184,
+            0.028453707695007324,
+            -0.02095392718911171,
+            0.0053142374381423,
+            0.0007258309051394463,
+            -0.029438525438308716,
+            0.0060073984786868095,
+            0.017863109707832336,
+            -0.0025965142995119095,
+            0.0025302283465862274,
+            -0.02349930815398693,
+            0.024468975141644478,
+            0.025635607540607452,
+            0.004518806003034115,
+            -0.014037464745342731,
+            -0.013113250024616718,
+            -0.003605954349040985,
+            -0.009742137044668198,
+            -0.016196493059396744,
+            0.024468975141644478,
+            -0.014810169115662575,
+            0.00308892410248518,
+            0.021847840398550034,
+            0.014264730736613274,
+            -0.007045246660709381,
+            -0.004007457755506039,
+            -0.04066546633839607,
+            -0.01218903437256813,
+            -0.04760465770959854,
+            -0.011628445237874985,
+            -0.0034601252991706133,
+            -0.028696123510599136,
+            0.0061513339169323444,
+            -0.04481685906648636,
+            -0.01235569640994072,
+            0.02309022843837738,
+            0.018135828897356987,
+            0.0019488062243908644,
+            -0.009173971600830555,
+            -0.022605394944548607,
+            8.486965816700831e-05,
+            -0.01763584464788437,
+            -0.017044952139258385,
+            0.015393485315144062,
+            0.031059689819812775,
+            -0.03101423755288124,
+            0.04024123772978783,
+            0.003893824527040124,
+            -0.08066429197788239,
+            0.010393633507192135,
+            -0.026362858712673187,
+            0.07011914253234863,
+            -0.0025453795678913593,
+            0.014075342565774918,
+            0.06781618297100067,
+            -0.012431452050805092,
+            -0.02007516659796238,
+            0.03142331540584564,
+            -0.008249756880104542,
+            -1.1311515663692262e-05,
+            -0.02884763479232788,
+            -0.02739313244819641,
+            -0.053271155804395676,
+            -0.009704259224236012,
+            -0.010787560604512691,
+            0.03360506892204285,
+            -0.024317465722560883,
+            -0.0007656024536117911,
+            0.012620840221643448,
+            0.02983245439827442,
+            0.016635872423648834,
+            -0.0015681355725973845,
+            0.008272483013570309,
+            0.02881733328104019,
+            0.0274234339594841,
+            -0.0036154238041490316,
+            0.00814369972795248,
+            -0.034211114048957825,
+            -0.0068066176027059555,
+            -0.0493621826171875,
+            0.011302697472274303,
+            -0.02596893161535263,
+            -0.02596893161535263,
+            0.007643714081496,
+            -0.011476934887468815,
+            0.03748374432325363,
+            -0.024696242064237595,
+            -0.007503566797822714,
+            0.005037730094045401,
+            0.012855681590735912,
+            -0.017757052555680275,
+            -0.03612014651298523,
+            -0.0028502948116511106,
+            0.0026381798088550568,
+            0.007560383062809706,
+            -0.053240854293107986,
+            0.004499867558479309,
+            -0.018469152972102165,
+            0.0279688723385334,
+            -0.02139330841600895,
+            -0.013688990846276283,
+            0.042029064148664474,
+            0.011363301426172256,
+            0.027499189600348473,
+            -0.026468917727470398,
+            0.0017849853029474616,
+            0.012825379148125648,
+            -0.0020056227222085,
+            0.032514192163944244,
+            -0.030574856325984,
+            -0.03257479891180992,
+            0.001548249856568873,
+            -0.007840678095817566,
+            -0.0625436082482338,
+            -0.011386028490960598,
+            -0.00981789268553257,
+            0.017317671328783035,
+            0.0446956530213356,
+            0.04557441547513008,
+            -0.01068150345236063,
+            0.002174178371205926,
+            -0.02213571034371853,
+            -0.04542290419340134,
+            -0.02643861435353756,
+            0.0016486257081851363,
+            -0.019999410957098007,
+            -0.02789311669766903,
+            -0.0018313854234293103,
+            0.011211791075766087,
+            -0.013651113025844097,
+            0.01134057529270649,
+            -0.018560059368610382,
+            -0.01862066239118576,
+            -0.012332970276474953,
+            0.018575210124254227,
+            0.009901223704218864,
+            -0.0018796793883666396,
+            -0.015037435106933117,
+            -0.00980274099856615,
+            0.018969137221574783,
+            -0.011279970407485962,
+            0.0009388927719555795,
+            0.01188601367175579,
+            0.024347767233848572,
+            -0.007083124481141567,
+            0.010961798019707203,
+            -0.034756552428007126,
+            -0.01966608688235283,
+            0.007984613068401814,
+            0.002719616750255227,
+            0.020711511373519897,
+            0.008742166683077812,
+            -0.04275631532073021,
+            -0.044301725924015045,
+            0.016044981777668,
+            -0.0562104657292366,
+            0.028120383620262146,
+            -0.03327174484729767,
+            -0.012514783069491386,
+            -0.02671133354306221,
+            0.0013418165035545826,
+            -0.01828734017908573,
+            0.022438732907176018,
+            0.03778676688671112,
+            -0.0011486404109746218,
+            0.005503625608980656,
+            0.014658658765256405,
+            0.04384719207882881,
+            -0.05021063983440399,
+            -0.041180603206157684,
+            0.03754434734582901,
+            -0.01601468026638031,
+            0.0185146052390337,
+            0.00021744151308666915,
+            0.010348179377615452,
+            -0.014431392773985863,
+            0.022772056981921196,
+            0.013757170177996159,
+            -0.008764892816543579,
+            -0.030574856325984,
+            0.04945308715105057,
+            -0.023650817573070526,
+            -0.049513693898916245,
+            0.0018323323456570506,
+            0.018878230825066566,
+            0.018635815009474754,
+            -0.014870774000883102,
+            -0.02907489985227585,
+            -0.024317465722560883,
+            -0.006060427520424128,
+            -0.00947699323296547,
+            -0.03402930125594139,
+            0.011363301426172256,
+            0.0048445542342960835,
+            0.016666175797581673,
+            -0.00645056739449501,
+            -0.0073141781613230705,
+            -0.009370936080813408,
+            0.022559940814971924,
+            0.012635990977287292,
+            -0.03175663948059082,
+            -0.017484333366155624,
+            -0.02528713271021843,
+            0.004359719809144735,
+            0.010136065073311329,
+            0.022166013717651367,
+            0.024287162348628044,
+            -0.0323626808822155,
+            0.04218057543039322,
+            0.01550711877644062,
+            -0.008727015927433968,
+            -0.0019317612750455737,
+            0.02478714846074581,
+            0.04324115067720413,
+            0.0218023881316185,
+            0.026514369994401932,
+            0.002107892418280244,
+            8.705945947440341e-05,
+            -0.015620751306414604,
+            -0.0011391709558665752,
+            0.01862066239118576,
+            0.04911976307630539,
+            -0.012151156552135944,
+            0.01716616004705429,
+            -0.03578682243824005,
+            0.00396200455725193,
+            -0.02580226957798004,
+            -0.030786970630288124,
+            0.03745344281196594,
+            -0.015060162171721458,
+            0.0191964041441679,
+            0.025044716894626617,
+            -0.022469034418463707,
+            -0.011075431481003761,
+            -0.019423669204115868,
+            -0.026181045919656754,
+            -0.01702980138361454,
+            -0.03281721472740173,
+            -0.018135828897356987,
+            -0.002653330797329545,
+            -0.035635314881801605,
+            -0.04987731948494911,
+            -0.004329417832195759,
+            -0.046392571181058884,
+            -0.010507266037166119,
+            0.017378276214003563,
+            -0.011166337877511978,
+            -0.025711363181471825,
+            -0.005336964037269354,
+            0.003973367623984814,
+            -0.022196315228939056,
+            -0.0066588944755494595,
+            0.008439145050942898,
+            0.035938333719968796,
+            -0.02298417128622532,
+            -0.013060221448540688,
+            -0.0024317465722560883,
+            -0.02887793630361557,
+            -0.007878555916249752,
+            0.008999735116958618,
+            0.007723257411271334,
+            -0.0012357590021565557,
+            -0.05196816474199295,
+            -0.006916462909430265,
+            -0.012992041185498238,
+            0.029196109622716904,
+            0.026468917727470398,
+            0.0012168202083557844,
+            -0.02783251367509365,
+            0.008749742060899734,
+            -0.0072535742074251175,
+            0.02339324913918972,
+            -0.007992188446223736,
+            -0.022908415645360947,
+            -0.02034788578748703,
+            0.030786970630288124,
+            -0.0159086212515831,
+            0.009113367646932602,
+            0.009976978413760662,
+            0.0121360057964921,
+            -0.03899884968996048,
+            -0.001445980160497129,
+            0.0005648507503792644,
+            -0.010128489695489407,
+            0.00423093605786562,
+            -0.006738437805324793,
+            -0.007079336792230606,
+            -0.022090258076786995,
+            -0.0218023881316185,
+            0.03818069398403168,
+            -0.009113367646932602,
+            -0.0035661826841533184,
+            -0.01088604237884283,
+            0.03436262533068657,
+            -0.00595815759152174,
+            -0.01993880607187748,
+            -0.019347915425896645,
+            0.014431392773985863,
+            0.05105910077691078,
+            -0.028529461473226547,
+            -0.03984731063246727,
+            0.0011136035900563002,
+            0.030150625854730606,
+            0.004140029661357403,
+            -0.016908591613173485,
+            0.0035851215943694115,
+            0.056392278522253036,
+            -0.011742077767848969,
+            0.028362801298499107,
+            0.027771908789873123,
+            0.02295386977493763,
+            -0.02021152526140213,
+            -0.01678738370537758,
+            -0.020999381318688393,
+            0.011098157614469528,
+            -0.03027183562517166,
+            0.027059808373451233,
+            0.018969137221574783,
+            0.01538590993732214,
+            0.05054396390914917,
+            -0.03178694099187851,
+            -0.02515077404677868,
+            0.008499749936163425,
+            0.0021722845267504454,
+            0.011007251217961311,
+            -0.007727045100182295,
+            -0.01801462098956108,
+            -0.001950700068846345,
+            -0.014022313989698887,
+            -0.019817598164081573,
+            -0.020938776433467865,
+            0.00954517349600792,
+            0.015696506947278976,
+            0.002649543108418584,
+            0.010863316245377064,
+            0.0016590419691056013,
+            0.01686313934624195,
+            0.01321930717676878,
+            -0.0444229319691658,
+            -7.936555630294606e-05,
+            0.03378688171505928,
+            -0.0293476190418005,
+            -0.012264790013432503,
+            -0.016060132533311844,
+            -0.004439263138920069,
+            -0.01706010289490223,
+            -0.02762039750814438,
+            -0.006874797400087118,
+            -0.01889338344335556,
+            -0.00814369972795248,
+            -0.028196139261126518,
+            -0.009992130100727081,
+            0.01949942484498024,
+            0.022332675755023956,
+            0.001196934375911951,
+            -0.006783891003578901,
+            -0.005901341326534748,
+            0.02092362567782402,
+            0.032514192163944244,
+            0.01952972821891308,
+            0.0058331615291535854,
+            -0.015158643946051598,
+            -0.004295328166335821,
+            -0.012370847165584564,
+            0.016060132533311844,
+            0.012537509202957153,
+            6.344509893096983e-05,
+            0.03493836522102356,
+            0.004170331638306379,
+            -0.022726602852344513,
+            -0.001230077352374792,
+            0.024923507124185562,
+            -0.021590271964669228,
+            -0.0108330138027668,
+            0.040029123425483704,
+            0.0085679292678833,
+            -0.0024506852496415377,
+            -0.019287310540676117,
+            -0.019075196236371994,
+            0.02630225569009781,
+            -0.02133270539343357,
+            0.002083271974697709,
+            0.0482713058590889,
+            0.00035344602656550705,
+            -0.029802151024341583,
+            0.017954016104340553,
+            -0.013242034241557121,
+            -0.018560059368610382,
+            0.01848430372774601,
+            0.0012859469279646873,
+            -0.039968520402908325,
+            0.019014591351151466,
+            0.018408548086881638,
+            0.022635696455836296,
+            -0.02918095886707306,
+            0.0002127068000845611,
+            0.03233237937092781,
+            0.004037759732455015,
+            0.001595596899278462,
+            -0.008530051447451115,
+            -0.0006065162015147507,
+            0.010431510396301746,
+            0.001746160676702857,
+            0.02854461409151554,
+            -0.0030434709042310715,
+            0.027771908789873123,
+            -0.006727074272930622,
+            -0.008863375522196293,
+            0.019181253388524055,
+            -0.02109028771519661,
+            0.013166278600692749,
+            0.0685434341430664,
+            0.0021950111258774996,
+            0.025847723707556725,
+            0.007181606721132994,
+            0.024347767233848572,
+            0.02018122375011444,
+            -0.027271922677755356,
+            -0.02021152526140213,
+            -0.005397567991167307,
+            -0.0014033677289262414,
+            -0.03787767142057419,
+            0.011128460057079792,
+            -0.021302402019500732,
+            -0.013249609619379044,
+            -0.0022347825579345226,
+            -0.002316219499334693,
+            -0.006405114196240902,
+            0.034211114048957825,
+            -0.0014630251098424196,
+            -0.018560059368610382,
+            -0.035423196852207184,
+            0.037726160138845444,
+            0.04048365354537964,
+            0.0389382466673851,
+            -0.005984671879559755,
+            0.03548380360007286,
+            0.0030756669584661722,
+            0.0025302283465862274,
+            0.041453324258327484,
+            0.004791525658220053,
+            -0.011567841283977032,
+            0.02295386977493763,
+            0.012416300363838673,
+            0.03351416438817978,
+            -0.02045394293963909,
+            -0.02630225569009781,
+            -0.003299145260825753,
+            -0.026423463597893715,
+            0.012492056004703045,
+            -0.019726691767573357,
+            0.0018768386216834188,
+            0.007117214612662792,
+            0.015196521766483784,
+            -0.016044981777668,
+            0.014719262719154358,
+            -0.006102093029767275,
+            -0.021014532074332237,
+            0.011719351634383202,
+            -0.007780073676258326,
+            0.011961768381297588,
+            -0.02262054570019245,
+            0.001138224033638835,
+            -0.04063516482710838,
+            0.008280059322714806,
+            0.014022313989698887,
+            -0.01598437689244747,
+            0.03205966204404831,
+            0.020969079807400703,
+            -0.01801462098956108,
+            -0.0009682480012997985,
+            0.0034714885987341404,
+            -0.010393633507192135,
+            -0.011067855171859264,
+            -0.019923655316233635,
+            0.02786281518638134,
+            -0.015802564099431038,
+            -0.025908326730132103,
+            0.02454473078250885,
+            -0.01743887923657894,
+            0.015151068568229675,
+            -0.004299115855246782,
+            0.00888610165566206,
+            -0.019120648503303528,
+            0.02284781076014042,
+            -0.023741723969578743,
+            -0.0009128518868237734,
+            -0.020287280902266502,
+            0.0028389315120875835,
+            0.014135947450995445,
+            -0.018272189423441887,
+            -0.02139330841600895,
+            0.019150950014591217,
+            -0.005420294590294361,
+            -0.003931702114641666,
+            0.010014856234192848,
+            0.0030983935575932264,
+            -0.0029715034179389477,
+            -0.004549108445644379,
+            -0.03281721472740173,
+            -0.02915065549314022,
+            0.0033086147159337997,
+            0.011386028490960598,
+            0.015257125720381737,
+            0.027574945241212845,
+            0.03715042024850845,
+            0.006052851676940918,
+            0.0036949669010937214,
+            0.0009327376610599458,
+            -0.020726662129163742,
+            0.042635105550289154,
+            -0.020469093695282936,
+            0.040847279131412506,
+            0.032544493675231934,
+            -0.0197418425232172,
+            -0.023908386006951332,
+            0.0029052174650132656,
+            0.03115059621632099,
+            0.03151422366499901,
+            0.025211378931999207,
+            0.012310243211686611,
+            0.03690800443291664,
+            -0.004359719809144735,
+            -0.032211173325777054,
+            0.015802564099431038,
+            0.022256920114159584,
+            -0.008105821907520294,
+            0.009673957712948322,
+            0.027438584715127945,
+            0.013279912061989307,
+            -0.007757347077131271,
+            0.030150625854730606,
+            0.0012234487803652883,
+            0.02396899089217186,
+            0.02498411200940609,
+            -0.02190844528377056,
+            0.01743887923657894,
+            -0.0001154677156591788,
+            0.006727074272930622,
+            -0.009840618818998337,
+            0.004348356742411852,
+            0.011810258030891418,
+            0.018272189423441887,
+            0.018908534198999405,
+            0.003246116451919079,
+            -0.07060398161411285,
+            0.019726691767573357,
+            0.004507442936301231,
+            -0.02160542458295822,
+            0.00390140013769269,
+            -0.0024866692256182432,
+            0.012704171240329742,
+            0.02105998620390892,
+            0.012461753562092781,
+            -0.01790856383740902,
+            0.0013626492582261562,
+            0.011961768381297588,
+            -0.00595815759152174,
+            0.04730163514614105,
+            0.0019791084341704845,
+            -0.01437078882008791,
+            -0.016105586662888527,
+            0.013726867735385895,
+            0.02525683119893074,
+            -0.02355991117656231,
+            0.006325571332126856,
+            0.005621046293526888,
+            -0.004147605039179325,
+            -0.010742107406258583,
+            -0.030377892777323723,
+            -0.005280147306621075,
+            -0.03436262533068657,
+            0.023150833323597908,
+            0.015491968020796776,
+            0.014764715917408466,
+            -0.025044716894626617,
+            -0.004280176945030689,
+            0.0046400148421525955,
+            -0.021287251263856888,
+            0.003261267440393567,
+            0.01161329448223114,
+            0.0005104016163386405,
+            -0.020984230563044548,
+            0.00959820207208395,
+            0.013469300232827663,
+            0.06848283112049103,
+            0.0007442962378263474,
+            0.029771849513053894,
+            0.0422411784529686,
+            -0.0012243957025930285,
+            -0.006371024530380964,
+            0.0036286809481680393,
+            -0.019953956827521324,
+            0.003473382443189621,
+            0.025878025218844414,
+            -0.03224147483706474,
+            -0.055058982223272324,
+            -0.007545232307165861,
+            -0.04624105989933014,
+            0.001489539397880435,
+            0.014711687341332436,
+            0.023044776171445847,
+            -0.0037782976869493723,
+            -0.034817155450582504,
+            0.016908591613173485,
+            -0.003071879269555211,
+            0.007014944683760405,
+            -0.02593863010406494,
+            -0.0294839795678854,
+            -0.026014383882284164,
+            -0.02339324913918972,
+            -0.012908710166811943,
+            -0.022726602852344513,
+            0.012590537779033184,
+            -0.0015103721525520086,
+            -0.06587684899568558,
+            -0.009780014865100384,
+            0.016166189685463905,
+            0.001200722181238234,
+            0.011939042247831821,
+            0.011083006858825684,
+            -0.007083124481141567,
+            -0.0018275976181030273,
+            -0.022559940814971924,
+            0.03124150261282921,
+            0.005177877843379974,
+            0.03584742918610573,
+            -0.006503595970571041,
+            -0.009128518402576447,
+            0.01073453202843666,
+            0.04527139291167259,
+            0.0017916138749569654,
+            0.010226971469819546,
+            -0.020560000091791153,
+            0.0015065843472257257,
+            -0.02284781076014042,
+            -0.0031135445460677147,
+            -0.008863375522196293,
+            -0.0021211495622992516,
+            0.015893470495939255,
+            0.0019753205124288797,
+            -0.010204244405031204,
+            0.025438643991947174,
+            0.02251448854804039,
+            -0.009605777449905872,
+            -0.015969226136803627,
+            -0.02437806874513626,
+            -0.014545026235282421,
+            -0.0002627763315103948,
+            -0.010007280856370926,
+            -0.022453883662819862,
+            -0.020878173410892487,
+            -0.002566212322562933,
+            -0.02559015527367592,
+            0.022529639303684235,
+            0.01220418605953455,
+            -4.882668508798815e-05,
+            -0.006723286584019661,
+            -0.038544319570064545,
+            -0.003727162955328822,
+            -0.03118089959025383,
+            -0.005374841392040253,
+            0.03287781774997711,
+            -0.03745344281196594,
+            -0.0002231231628684327,
+            0.014491996727883816,
+            -0.01993880607187748,
+            0.0028938541654497385,
+            0.008961857296526432,
+            0.02769615314900875,
+            -0.027408283203840256,
+            0.0001795638381736353,
+            0.006954340264201164,
+            0.016908591613173485,
+            -0.009878496639430523,
+            0.027196168899536133,
+            -0.04275631532073021,
+            0.011037553660571575,
+            -0.037120118737220764,
+            -0.020423639565706253,
+            -0.01523439958691597,
+            0.008363389410078526,
+            -0.005018791649490595,
+            -0.02637801133096218,
+            -0.010454237461090088,
+            0.006026337388902903,
+            -0.00693918950855732,
+            -0.01902974210679531,
+            -0.010620899498462677,
+            0.027438584715127945,
+            0.00931033119559288,
+            0.007977037690579891,
+            -0.02955973520874977,
+            -0.01631770096719265,
+            0.006280118133872747,
+            0.052089374512434006,
+            -0.01584801822900772,
+            0.014938953332602978,
+            -0.006639956030994654,
+            0.02342355251312256,
+            -0.028484009206295013,
+            -0.01872672140598297,
+            0.0315445251762867,
+            0.013946558348834515,
+            0.007064185570925474,
+            0.028953691944479942,
+            0.027196168899536133,
+            0.027105262503027916,
+            -0.04654408246278763,
+            0.009886072017252445,
+            -0.002613559365272522,
+            -0.008757317438721657,
+            0.03402930125594139,
+            0.01916610263288021,
+            0.022150862962007523,
+            0.0009744031121954322,
+            -0.01730252057313919,
+            0.0139844361692667,
+            0.014885924756526947,
+            0.007681591901928186,
+            -0.014242004603147507,
+            -0.02660527639091015,
+            0.003367325058206916,
+            -0.007992188446223736,
+            0.00047276070108637214,
+            0.020726662129163742,
+            -0.009052763693034649,
+            0.030999086797237396,
+            -0.014605630189180374,
+            -0.02593863010406494,
+            0.008916404098272324,
+            -0.02268115058541298,
+            -0.0041741193272173405,
+            0.0004476667381823063,
+            0.010083036497235298,
+            -0.023847781121730804,
+            -0.04669559374451637,
+            0.019408518448472023,
+            0.028393102809786797,
+            0.005579380784183741,
+            0.01328748743981123,
+            -0.035332292318344116,
+            0.008992158807814121,
+            0.012310243211686611,
+            0.011348150670528412,
+            0.0019847899675369263,
+            0.035362593829631805,
+            -0.00028834378463216126,
+            0.012885984033346176,
+            -0.008620957843959332,
+            0.028362801298499107,
+            0.00667025800794363,
+            -0.0323626808822155,
+            -0.0378473699092865,
+            -0.01513591781258583,
+            0.015257125720381737,
+            0.006692984607070684,
+            -0.011840560473501682,
+            -0.02278720773756504,
+            -0.038817036896944046,
+            0.015787413343787193,
+            -0.012916285544633865,
+            -0.038271598517894745,
+            -0.010158791206777096,
+            -0.009742137044668198,
+            0.016635872423648834,
+            -0.006916462909430265,
+            -0.016802534461021423,
+            -0.016090434044599533,
+            0.005821797996759415,
+            7.161247049225494e-05,
+            0.0244083721190691,
+            -0.001251857029274106,
+            -0.006855858489871025,
+            -0.016393456608057022,
+            -0.01188601367175579,
+            -0.0035074723418802023,
+            0.03602924197912216,
+            -0.014219277538359165,
+            -0.02383263036608696,
+            0.014984406530857086,
+            0.02657497487962246,
+            0.011636020615696907,
+            0.03227177634835243,
+            0.007181606721132994,
+            -0.014022313989698887,
+            0.003016956616193056,
+            -0.033029329031705856,
+            -0.0046400148421525955,
+            0.005196816753596067,
+            -0.006427840795367956,
+            -0.015651054680347443,
+            -0.01815097965300083,
+            -0.015832867473363876,
+            0.030756669119000435,
+            -0.02487805485725403,
+            0.0218023881316185,
+            0.030347591266036034,
+            0.003174148965626955,
+            -0.02915065549314022,
+            0.013598084449768066,
+            -0.0072005451656877995,
+            0.01862066239118576,
+            0.0019014590652659535,
+            0.028014326468110085,
+            -0.007461901288479567,
+            -0.004818039946258068,
+            -0.02501441352069378,
+            -0.03696860745549202,
+            -0.023484155535697937,
+            -0.008817922323942184,
+            0.0027537066489458084,
+            0.020984230563044548,
+            -0.0375746488571167,
+            -0.017757052555680275,
+            0.009855770505964756,
+            0.015757111832499504,
+            -0.03027183562517166,
+            -0.0025908327661454678,
+            -0.010158791206777096,
+            -0.0004055278259329498,
+            0.005140000022947788,
+            -0.02227207086980343,
+            -0.008977008052170277,
+            -0.015726808458566666,
+            -0.0010435298318043351,
+            0.006166485138237476,
+            -0.0007409819518215954,
+            -0.05805889517068863,
+            -0.013969285413622856,
+            0.018635815009474754,
+            0.0026400736533105373,
+            0.008908828720450401,
+            0.015938924625515938,
+            -0.0111739132553339,
+            0.0031533162109553814,
+            -0.026756787672638893,
+            0.023893235251307487,
+            -0.009492144919931889,
+            0.009355784393846989,
+            -0.013742019422352314,
+            0.012514783069491386,
+            0.014014738611876965,
+            0.004795313347131014,
+            0.0010681503918021917,
+            -0.0054051438346505165,
+            0.01601468026638031,
+            0.028377952054142952,
+            -0.023756874725222588,
+            0.011454207822680473,
+            -0.0002514130319468677,
+            -0.0032158142421394587,
+            -0.029196109622716904,
+            0.00509454682469368,
+            0.003931702114641666,
+            -0.005109698045998812,
+            0.016651025041937828,
+            -0.036544378846883774,
+            0.01662072166800499,
+            0.02177208475768566,
+            0.032544493675231934,
+            0.014832896180450916,
+            -0.017287369817495346,
+            -0.011658747680485249,
+            -0.012385998852550983,
+            -0.002624922664836049,
+            0.022181164473295212,
+            0.012120855040848255,
+            -0.01815097965300083,
+            -0.024287162348628044,
+            0.011249667964875698,
+            0.0005283934879116714,
+            -0.01203752402216196,
+            0.010795135982334614,
+            -0.013166278600692749,
+            0.01349202636629343,
+            0.012711746618151665,
+            0.00866641104221344,
+            0.00888610165566206,
+            0.022862963378429413,
+            -0.011878437362611294,
+            -0.017711598426103592,
+            0.00656041270121932,
+            -0.008916404098272324,
+            -0.0034089903347194195,
+            -0.012598113156855106,
+            0.010795135982334614,
+            0.014878349378705025,
+            0.06490717828273773,
+            -0.013143551535904408,
+            0.0009526234352961183,
+            -0.020438792183995247,
+            -0.031938452273607254,
+            -0.0017944546416401863,
+            -0.020908474922180176,
+            0.0111739132553339,
+            0.008370965719223022,
+            0.01598437689244747,
+            0.012022373266518116,
+            -0.0021912232041358948,
+            0.00795431062579155,
+            0.019317612051963806,
+            -0.010423935018479824,
+            7.255941454786807e-05,
+            -0.006503595970571041,
+            0.0013683309080079198,
+            0.0035851215943694115,
+            -0.011719351634383202,
+            -0.0026949963066726923,
+            0.035695917904376984,
+            0.01068150345236063,
+            0.011651171371340752,
+            -0.03445352986454964,
+            0.0020435003098100424,
+            0.020150920376181602,
+            0.007602048572152853,
+            -0.008302785456180573,
+            -0.01563590206205845,
+            0.01706010289490223,
+            -0.015317730605602264,
+            -0.0022499337792396545,
+            0.002888172399252653,
+            -0.010151215828955173,
+            0.04581683129072189,
+            -0.019787294790148735,
+            -0.0011742077767848969,
+            0.031847547739744186,
+            0.0030207443051040173,
+            -0.0040983641520142555,
+            0.005408931523561478,
+            -0.003473382443189621,
+            -0.0069013116881251335,
+            -0.02139330841600895,
+            0.0031381649896502495,
+            0.00667025800794363,
+            -0.019287310540676117,
+            -0.0072005451656877995,
+            -0.007719469256699085,
+            0.006897523999214172,
+            0.023105379194021225,
+            -0.009492144919931889,
+            -0.00016145357221830636,
+            -0.008575504645705223,
+            0.009848194196820259,
+            -0.0011912527261301875,
+            -0.044301725924015045,
+            -0.006147546228021383,
+            0.021181194111704826,
+            0.000973456131760031,
+            0.0005757405888289213,
+            -0.007450537756085396,
+            0.005007428117096424,
+            -0.017423728480935097,
+            -0.008454295806586742,
+            0.02369627170264721,
+            0.01736312545835972,
+            -0.009113367646932602,
+            0.010863316245377064,
+            -0.0035680767614394426,
+            0.00433320552110672,
+            -0.020014561712741852,
+            -0.01095422264188528,
+            0.003462019143626094,
+            0.015272277407348156,
+            -0.0009819786064326763,
+            -0.029029447585344315,
+            -0.0012272365856915712,
+            -0.00024194362049456686,
+            0.02918095886707306,
+            0.01486319862306118,
+            -0.008961857296526432,
+            -0.021211495622992516,
+            -0.013242034241557121,
+            -0.001237652963027358,
+            -0.015726808458566666,
+            -0.009620928205549717,
+            0.011855711229145527,
+            0.0009137988090515137,
+            0.008992158807814121,
+            -0.018135828897356987,
+            -0.027347678318619728,
+            -0.016166189685463905,
+            0.0018465365283191204,
+            -0.012870832346379757,
+            0.003340810537338257,
+            0.015514694154262543,
+            -0.03430201858282089,
+            -0.015060162171721458,
+            -0.03805948421359062,
+            0.008280059322714806,
+            -0.0029658216517418623,
+            -0.02024182677268982,
+            -0.023605365306138992,
+            0.009484568610787392,
+            0.019181253388524055,
+            -0.0015548784285783768,
+            0.019847899675369263,
+            -0.000889651826582849,
+            0.020060013979673386,
+            -0.0073141781613230705,
+            0.0031703610438853502,
+            -0.013007191941142082,
+            0.0030245319940149784,
+            0.008333087898790836,
+            0.02890823967754841,
+            -0.023075077682733536,
+            -0.006503595970571041,
+            -0.004685468040406704,
+            0.004746072459965944,
+            0.012658718042075634,
+            0.014613205567002296,
+            0.020741812884807587,
+            -0.008272483013570309,
+            -0.0027612822595983744,
+            0.012484480626881123,
+            -0.003799130441620946,
+            0.007295239716768265,
+            0.002390081062912941,
+            0.0019677451346069574,
+            0.004162756260484457,
+            0.008477022871375084,
+            0.0053710537031292915,
+            0.027650700882077217,
+            -0.0028275682125240564,
+            0.0031703610438853502,
+            0.024620486423373222,
+            0.02322658896446228,
+            -0.018878230825066566,
+            -0.028074929490685463,
+            0.002024561632424593,
+            0.050695475190877914,
+            0.007245998363941908,
+            -0.006810405291616917,
+            -0.03927157074213028,
+            -0.0031760428100824356,
+            0.028635520488023758,
+            0.023650817573070526,
+            0.01584801822900772,
+            -0.010567869991064072,
+            -0.012098127976059914,
+            0.02772645466029644,
+            0.013295062817633152,
+            0.02616589516401291,
+            -0.025302285328507423,
+            0.010704229585826397,
+            -0.03869583085179329,
+            0.0449683703482151,
+            0.004439263138920069,
+            -0.014476845972239971,
+            0.0011884119594469666,
+            0.010386057198047638,
+            -0.004537744913250208,
+            0.006431628484278917,
+            -5.8059369621332735e-05,
+            -0.0008086882880888879,
+            0.011923890560865402,
+            0.012007221579551697,
+            0.004265025723725557,
+            -0.02399929240345955,
+            0.007359631359577179,
+            0.004272601567208767,
+            0.03005971945822239,
+            0.005431658122688532,
+            -0.0013863227795809507,
+            0.012385998852550983,
+            -0.005810434930026531,
+            0.008855799213051796,
+            -0.010719381272792816,
+            0.020469093695282936,
+            -0.02918095886707306,
+            1.0986003871948924e-05,
+            0.02051454596221447,
+            -0.0015785519499331713,
+            0.0038199631962925196,
+            -0.023741723969578743,
+            0.01186328660696745,
+            -0.052968136966228485,
+            0.021166043356060982,
+            -0.018741872161626816,
+            0.021741783246397972,
+            0.019590331241488457,
+            -0.009734561666846275,
+            -0.012052674777805805,
+            0.03545350208878517,
+            0.0026760573964565992,
+            -0.014264730736613274,
+            -0.010643625631928444,
+            -0.017999470233917236,
+            -0.02830219641327858,
+            0.041968461126089096,
+            -0.0006117243901826441,
+            0.0028275682125240564,
+            0.0024222771171480417,
+            -0.010666352696716785,
+            -0.008923979476094246,
+            0.001954487757757306,
+            0.020499395206570625,
+            -0.03369597718119621,
+            -0.011128460057079792,
+            -0.008689138107001781,
+            -0.02566591091454029,
+            -0.00020300064352340996,
+            -0.011961768381297588,
+            0.026211349293589592,
+            0.01240872498601675,
+            -0.03696860745549202,
+            0.003780191531404853,
+            0.023650817573070526,
+            0.001603172393515706,
+            0.018120678141713142,
+            0.002130619017407298,
+            0.0010795136913657188,
+            0.009742137044668198,
+            -0.010476963594555855,
+            0.026559824123978615,
+            -0.009204274043440819,
+            0.008658835664391518,
+            -0.001950700068846345,
+            -0.017287369817495346,
+            0.017969166859984398,
+            0.004083212930709124,
+            0.024120500311255455,
+            0.011954193003475666,
+            0.011605718173086643,
+            0.01112088467925787,
+            -0.019878201186656952,
+            0.020090317353606224,
+            -0.0018891488434746861,
+            0.009075489826500416,
+            0.024726543575525284,
+            -0.005863463506102562,
+            0.020832719281315804,
+            0.015257125720381737,
+            -0.00020229043730068952,
+            0.011295121163129807,
+            -0.020120618864893913,
+            -0.011779955588281155,
+            0.0085679292678833,
+            -0.009605777449905872,
+            -0.010325453244149685,
+            0.009901223704218864,
+            -0.010105762630701065,
+            -0.015052586793899536,
+            -0.0071247899904847145,
+            0.016635872423648834,
+            0.0028976418543606997,
+            -0.02610529027879238,
+            -1.7326075976598077e-05,
+            0.008901252411305904,
+            0.006166485138237476,
+            0.018741872161626816,
+            0.011317848227918148,
+            -0.006692984607070684,
+            0.010363331064581871,
+            0.011476934887468815,
+            0.0053104497492313385,
+            -0.005079395603388548,
+            -0.013431422412395477,
+            -0.0024241709616035223,
+            0.018090376630425453,
+            0.009810317307710648,
+            -0.01922670565545559,
+            0.004564259201288223,
+            0.0008877579239197075,
+            -0.01420412678271532,
+            -0.019347915425896645,
+            -0.0038483713287860155,
+            -0.004170331638306379,
+            0.0027915844693779945,
+            0.0509984977543354,
+            -0.0006216673064045608,
+            0.007893706671893597,
+            -0.025741664692759514,
+            -0.006299056578427553,
+            -0.00027556007262319326,
+            -0.0036097420379519463,
+            0.01272689737379551,
+            -0.01824188604950905,
+            -0.03724132478237152,
+            0.012711746618151665,
+            -0.0012584856012836099,
+            0.014923802576959133,
+            -0.00297907879576087,
+            0.014226853847503662,
+            -0.017347972840070724,
+            0.01916610263288021,
+            -0.0011117096291854978,
+            -0.016029831022024155,
+            0.0029374135192483664,
+            -0.011143610812723637,
+            -0.021317552775144577,
+            0.005916492082178593,
+            -0.006848283112049103,
+            -0.03896854817867279,
+            0.0029885482508689165,
+            -0.010423935018479824,
+            0.0011798894265666604,
+            -0.003912763670086861,
+            0.020166072994470596,
+            -0.0061513339169323444,
+            0.0008697660523466766,
+            0.02177208475768566,
+            0.0017026013229042292,
+            0.01601468026638031,
+            -0.014499573037028313,
+            0.010999675840139389,
+            0.012598113156855106,
+            0.02210540883243084,
+            -0.032180868089199066,
+            -0.01393140759319067,
+            0.014211702160537243,
+            0.0026116655208170414,
+            -0.005382417235523462,
+            0.020635755732655525,
+            -0.01294658798724413,
+            0.019650936126708984,
+            -0.0005577487172558904,
+            -0.011704200878739357,
+            -0.03524138405919075,
+            -0.016529815271496773,
+            -3.343888238305226e-05,
+            -0.008439145050942898,
+            0.025317436084151268,
+            0.01760554127395153,
+            -0.0108330138027668,
+            0.026468917727470398,
+            0.0037574649322777987,
+            0.014598054811358452,
+            -0.014257155358791351,
+            -0.005999823100864887,
+            -0.0003773563075810671,
+            -0.0119314668700099,
+            0.015264702029526234,
+            -0.03512017801403999,
+            0.0008342557121068239,
+            -0.00839369185268879,
+            0.005242269951850176,
+            -0.0254992488771677,
+            0.018257036805152893,
+            0.025817420333623886,
+            -0.0031419529113918543,
+            -0.005037730094045401,
+            -0.022802358493208885,
+            0.028968842700123787,
+            0.03049910068511963,
+            0.0024828813038766384,
+            -0.0013825349742546678,
+            -0.00352072948589921,
+            0.0033218718599528074,
+            0.0211205892264843,
+            -0.037998881191015244,
+            -0.009098216891288757,
+            -0.028696123510599136,
+            0.042332086712121964,
+            -0.008098246529698372,
+            -0.013431422412395477,
+            0.01862066239118576,
+            -0.0024715180043131113,
+            0.02355991117656231,
+            -0.004689255729317665,
+            -0.004855917301028967,
+            0.03148392215371132,
+            0.011514811776578426,
+            0.02772645466029644,
+            -0.04015033319592476,
+            0.014704111963510513,
+            -0.006867222022265196,
+            0.004867280833423138,
+            -0.013484450988471508,
+            0.005302873905748129,
+            0.02580226957798004,
+            -0.018060073256492615,
+            0.0068066176027059555,
+            0.030665762722492218,
+            0.00748084019869566,
+            0.002448791405186057,
+            0.0165904201567173,
+            0.003890036838129163,
+            0.010855740867555141,
+            0.000841831264551729,
+            0.0052574207074940205,
+            0.022423582151532173,
+            0.014454119838774204,
+            -0.016029831022024155,
+            -0.015120766125619411,
+            -0.00044790346873924136,
+            -0.004450626205652952,
+            0.005261208396404982,
+            0.013242034241557121,
+            0.0061172437854111195,
+            -0.015681356191635132,
+            0.01052999310195446,
+            0.014158673584461212,
+            0.013128400780260563,
+            -0.0039847311563789845,
+            -0.03412020578980446,
+            -0.02443867363035679,
+            -0.021438762545585632,
+            0.003104075090959668,
+            -0.0008281006012111902,
+            -0.0021211495622992516,
+            0.0017954016802832484,
+            0.017347972840070724,
+            -0.0008352026343345642,
+            0.00379155483096838,
+            0.01686313934624195,
+            -0.01415109820663929,
+            -0.022090258076786995,
+            -0.0035964848939329386,
+            -0.027529491111636162,
+            0.0053710537031292915,
+            0.001844642567448318,
+            -0.00965123064815998,
+            0.008530051447451115,
+            -0.0046059247106313705,
+            -0.007223271764814854,
+            -0.012196609750390053,
+            -0.02007516659796238,
+            0.005780132487416267,
+            0.016878290101885796,
+            0.005730891600251198,
+            -0.009272454306483269,
+            -0.003939277958124876,
+            0.011575416661798954,
+            0.017484333366155624,
+            0.017317671328783035,
+            -0.009295180439949036,
+            0.009628504514694214,
+            0.0004022135108243674,
+            0.004670316819101572,
+            -0.01834794320166111,
+            0.0045869858004152775,
+            -0.0027953721582889557,
+            -0.008658835664391518,
+            0.01105270441621542,
+            -0.00031201730598695576,
+            0.01628739945590496,
+            -0.013090522959828377,
+            0.005624833982437849,
+            0.010333028621971607,
+            -0.020590301603078842,
+            -0.0067308624275028706,
+            -0.01692374423146248,
+            0.026484068483114243,
+            -0.00504530593752861,
+            0.00045405857963487506,
+            -0.02978700026869774,
+            0.0033616432920098305,
+            0.000848933297675103,
+            -0.000845619011670351,
+            -0.012764775194227695,
+            0.010749682784080505,
+            0.03242328763008118,
+            0.0005179771687835455,
+            0.0216660276055336,
+            -0.0036154238041490316,
+            0.0007480839849449694,
+            -0.00785582885146141,
+            0.004977126140147448,
+            -0.004916521720588207,
+            -0.009901223704218864,
+            -0.010863316245377064,
+            0.021726632490754128,
+            -0.0033029329497367144,
+            0.0003356908564455807,
+            0.011332998983561993,
+            -0.012067826464772224,
+            0.0014620780711993575,
+            -0.0047687990590929985,
+            0.00971941091120243,
+            0.0061702728271484375,
+            0.009113367646932602,
+            -0.01176480483263731,
+            0.05451354384422302,
+            -0.0106284748762846,
+            0.005537715740501881,
+            -0.00439380994066596,
+            -0.022226616740226746,
+            -0.017317671328783035,
+            0.012711746618151665,
+            -0.017544938251376152,
+            0.03375658020377159,
+            0.024150803685188293,
+            0.0006467612111009657,
+            0.010393633507192135,
+            -0.006958128418773413,
+            0.023923536762595177,
+            -0.002602196065708995,
+            0.0010795136913657188,
+            -0.004855917301028967,
+            0.015802564099431038,
+            0.04511988162994385,
+            0.017393426969647408,
+            -0.001152428099885583,
+            0.00829521007835865,
+            0.020575150847434998,
+            0.00814369972795248,
+            -0.0020075165666639805,
+            0.014219277538359165,
+            -0.01024212222546339,
+            0.00900731049478054,
+            -0.004458202049136162,
+            0.027438584715127945,
+            0.00814369972795248,
+            -0.006310420110821724,
+            -0.0036836036015301943,
+            -0.0008209985098801553,
+            0.014173824340105057,
+            0.006950552575290203,
+            0.01963578537106514,
+            -0.00461350055411458,
+            -0.0012575386790558696,
+            0.013870803639292717,
+            0.014491996727883816,
+            -0.00552635220810771,
+            0.001724380999803543,
+            -0.01435563713312149,
+            -0.010204244405031204,
+            0.013575357384979725,
+            0.006086941808462143,
+            0.006310420110821724,
+            -0.014514723792672157,
+            -0.00980274099856615,
+            -0.004670316819101572,
+            0.0015435151290148497,
+            0.015052586793899536,
+            0.011060279794037342,
+            -0.004814252257347107,
+            0.006052851676940918,
+            0.013825350441038609,
+            -0.014014738611876965,
+            0.006147546228021383,
+            0.007208121009171009,
+            0.008855799213051796,
+            0.004787737503647804,
+            -0.010817863047122955,
+            -0.00428396463394165,
+            -0.0063937511295080185,
+            0.0012670081341639161,
+            0.016832837834954262,
+            0.013181429356336594,
+            0.0049581872299313545,
+            0.027529491111636162,
+            -0.00839369185268879,
+            -0.012302667833864689,
+            0.02339324913918972,
+            0.011257244274020195,
+            0.0031703610438853502,
+            0.014651083387434483,
+            -0.012635990977287292,
+            -0.012446602806448936,
+            0.006655106786638498,
+            0.004848341923207045,
+            0.006049063988029957,
+            0.009151245467364788,
+            0.00450365524739027,
+            0.007333117071539164,
+            -0.007098275702446699,
+            -0.007427811156958342,
+            0.0014810169814154506,
+            0.01642375811934471,
+            0.020393338054418564,
+            -0.024968961253762245,
+            -0.0018806264270097017,
+            0.0009052763343788683,
+            0.010878467001020908,
+            -0.003316190093755722,
+            0.012310243211686611,
+            5.65205882594455e-05,
+            -0.0074240234680473804,
+            0.010158791206777096,
+            -0.02119634486734867,
+            -0.012681444175541401,
+            0.0012717428617179394,
+            0.013037494383752346,
+            -0.019590331241488457,
+            0.018226735293865204,
+            -0.008893677033483982,
+            -0.013348091393709183,
+            0.013893529772758484,
+            0.020014561712741852,
+            0.004090788308531046,
+            -0.007738408166915178,
+            -0.021862991154193878,
+            -0.01127239502966404,
+            0.016060132533311844,
+            0.020272130146622658,
+            0.000947888707742095,
+            -0.0006472347304224968,
+            0.010173942893743515,
+            -0.013113250024616718,
+            -0.023605365306138992,
+            -0.005230906419456005,
+            0.010386057198047638,
+            0.01746918261051178,
+            0.018832778558135033,
+            -0.006685409229248762,
+            0.02443867363035679,
+            -0.0059127043932676315,
+            -0.015499543398618698,
+            2.86597842205083e-05,
+            -0.02802947722375393,
+            -0.017423728480935097,
+            0.00013778003631159663,
+            0.00246773031540215,
+            0.008598231710493565,
+            0.0008555619278922677,
+            0.01943882182240486,
+            0.000904329412151128,
+            0.017590390518307686,
+            -0.000659071491099894,
+            -0.002009410411119461,
+            -0.0013995799235999584,
+            -0.0015340456739068031,
+            0.02254479005932808,
+            -0.012166308239102364,
+            -0.01176480483263731,
+            0.022302372381091118,
+            0.006852070800960064,
+            0.014719262719154358,
+            -0.027635548263788223,
+            0.013757170177996159,
+            -0.015499543398618698,
+            0.017423728480935097,
+            -0.03408990427851677,
+            0.006420265417546034,
+            0.01489350013434887,
+            -0.010348179377615452,
+            -0.020499395206570625,
+            -0.029650641605257988,
+            -0.024150803685188293,
+            -0.0038710979279130697,
+            -0.010855740867555141,
+            -0.016847988590598106,
+            0.006121031939983368,
+            -0.005802859086543322,
+            0.018499454483389854,
+            -0.007079336792230606,
+            0.033877789974212646,
+            0.006969491485506296,
+            -0.004056698642671108,
+            0.003835114184767008,
+            0.022469034418463707,
+            -0.006102093029767275,
+            0.02109028771519661,
+            0.004408961161971092,
+            -0.001467759720981121,
+            0.013681414537131786,
+            -0.006552837323397398,
+            -0.002361672930419445,
+            0.005821797996759415,
+            -0.010484539903700352,
+            0.022014502435922623,
+            0.005924067925661802,
+            -0.007708106189966202,
+            -0.0006775368237867951,
+            -0.011454207822680473,
+            0.014726839028298855,
+            0.006530110724270344,
+            0.0022480397019535303,
+            -0.011295121163129807,
+            0.011249667964875698,
+            -0.004859705455601215,
+            0.0040642740204930305,
+            0.009484568610787392,
+            -0.01774190180003643,
+            -0.028862785547971725,
+            0.011416330002248287,
+            0.019120648503303528,
+            0.006382387597113848,
+            -0.0055755930952727795,
+            -0.0025681061670184135,
+            0.00839369185268879,
+            0.010651201009750366,
+            0.022150862962007523,
+            -0.012984465807676315,
+            -8.889415767043829e-05,
+            0.013204156421124935,
+            -0.007170243188738823,
+            -0.01056029461324215,
+            0.00487485621124506,
+            0.023438703268766403,
+            0.0068823727779090405,
+            -0.04327145218849182,
+            -0.018469152972102165,
+            0.01909034699201584,
+            0.007136153522878885,
+            -0.04154422879219055,
+            0.0026381798088550568,
+            -0.003689285134896636,
+            -0.023787178099155426,
+            0.016105586662888527,
+            -0.0011817833874374628,
+            -0.01828734017908573,
+            0.028756728395819664,
+            0.02830219641327858,
+            0.01834794320166111,
+            2.6987840101355687e-05,
+            -0.013332940638065338,
+            -0.0318172425031662,
+            -0.023590214550495148,
+            0.008280059322714806,
+            -0.02660527639091015,
+            -0.0072725131176412106,
+            0.011242092587053776,
+            -0.017454031854867935,
+            0.02972639724612236,
+            -0.008507325313985348,
+            0.026938600465655327,
+            0.011446632444858551,
+            0.001691238023340702,
+            -0.013446573168039322,
+            -0.018802475184202194,
+            -0.01350717805325985,
+            -0.008196728304028511,
+            0.014090493321418762,
+            0.017923714593052864,
+            0.011007251217961311,
+            0.011332998983561993,
+            0.02251448854804039,
+            -0.012158732861280441,
+            0.007969462312757969,
+            -2.888172457460314e-05,
+            -0.008961857296526432,
+            0.0026343921199440956,
+            -0.018393397331237793,
+            0.00868156272917986,
+            0.0062308767810463905,
+            0.005458172410726547,
+            -0.007208121009171009,
+            0.010742107406258583,
+            0.02712041325867176,
+            -0.018847929313778877,
+            0.011401179246604443,
+            -0.031362712383270264,
+            -0.014787442982196808,
+            -0.01056029461324215,
+            0.026862844824790955,
+            0.007757347077131271,
+            -0.015726808458566666,
+            -0.007514929864555597,
+            -0.01545409020036459,
+            0.012885984033346176,
+            0.0032442226074635983,
+            0.005764981731772423,
+            0.015817714855074883,
+            0.021893294528126717,
+            0.01746918261051178,
+            -0.00795431062579155,
+            0.006852070800960064,
+            0.004674104508012533,
+            0.028090080246329308,
+            -0.01777220331132412,
+            0.020802417770028114,
+            0.009954252280294895,
+            -0.003935490269213915,
+            0.005685438401997089,
+            -0.014257155358791351,
+            0.008530051447451115,
+            -0.007325541693717241,
+            0.030514251440763474,
+            0.0033067206386476755,
+            0.014075342565774918,
+            -0.02062060497701168,
+            0.006575563922524452,
+            -0.03172633796930313,
+            0.009022461250424385,
+            0.013704141601920128,
+            0.012976890429854393,
+            -0.007677804213017225,
+            -0.024181105196475983,
+            -0.0119314668700099,
+            0.0057990713976323605,
+            -0.01279507763683796,
+            -0.01046181283891201,
+            -0.0034828518982976675,
+            0.012446602806448936,
+            0.018847929313778877,
+            -0.01848430372774601,
+            -0.019923655316233635,
+            -0.009583051316440105,
+            0.02898399345576763,
+            -0.015787413343787193,
+            -0.001471547526307404,
+            0.02413565292954445,
+            0.021241798996925354,
+            0.015022284351289272,
+            0.002155239460989833,
+            0.020090317353606224,
+            -0.0007111532613635063,
+            -0.0237265732139349,
+            -0.02813553437590599,
+            0.007302815094590187,
+            -0.006912675220519304,
+            0.01943882182240486,
+            -0.0035548193845897913,
+            -0.005658924113959074,
+            -0.0034033088013529778,
+            -0.007503566797822714,
+            -0.0165904201567173,
+            -0.01787826046347618,
+            0.0021647089160978794,
+            -0.008454295806586742,
+            0.005863463506102562,
+            -0.0002795845502987504,
+            -0.013105674646794796,
+            0.013401119969785213,
+            -0.014075342565774918,
+            0.005772557109594345,
+            -0.002653330797329545,
+            0.001962063368409872,
+            0.0035926972050219774,
+            0.009113367646932602,
+            -0.007336904760450125,
+            -0.06381630152463913,
+            -0.013105674646794796,
+            -0.003704436356201768,
+            0.006071790587157011,
+            -0.00019578021601773798,
+            -0.016635872423648834,
+            -0.013196581043303013,
+            0.01706010289490223,
+            0.008863375522196293,
+            -0.013870803639292717,
+            0.016954045742750168,
+            0.009961827658116817,
+            0.003622999181970954,
+            -0.024044746533036232,
+            -0.0008148433989845216,
+            0.008757317438721657,
+            0.032756611704826355,
+            -0.008643684908747673,
+            -0.0003839849086944014,
+            0.0036684523802250624,
+            -0.0033332351595163345,
+            -0.010052734054625034,
+            -0.014582903124392033,
+            -0.011848135851323605,
+            0.011908739805221558,
+            0.01790856383740902,
+            -0.014264730736613274,
+            -0.02007516659796238,
+            -0.0007106798002496362,
+            0.012931437231600285,
+            -0.026590125635266304,
+            0.0007244104635901749,
+            -0.007787649054080248,
+            -0.01052999310195446,
+            0.018302490934729576,
+            -0.019014591351151466,
+            0.005249845329672098,
+            0.00048294031876139343,
+            0.008507325313985348,
+            -0.023544760420918465,
+            -0.024893205612897873,
+            0.0035131541080772877,
+            -0.023847781121730804,
+            0.03612014651298523,
+            -0.021590271964669228,
+            -0.012082977220416069,
+            0.034483831375837326,
+            -0.0037423139438033104,
+            0.01702980138361454,
+            0.0023995505180209875,
+            0.02739313244819641,
+            -0.011439057067036629,
+            -0.00617784820497036,
+            0.012499631382524967,
+            0.009575475007295609,
+            0.0032593735959380865,
+            0.03775646165013313,
+            -0.01257538702338934,
+            -0.013901105150580406,
+            -0.005556654185056686,
+            -0.012954163365066051,
+            0.0006775368237867951,
+            0.014431392773985863,
+            0.028393102809786797,
+            0.004257450345903635,
+            -0.00628769351169467,
+            -0.006143758539110422,
+            -0.0014649189542979002,
+            -0.004242299124598503,
+            -0.00947699323296547,
+            -0.00715130427852273,
+            0.017287369817495346,
+            -0.025241680443286896,
+            0.016711628064513206,
+            -0.006420265417546034,
+            0.021135739982128143,
+            -0.011757229454815388,
+            0.007007369305938482,
+            -0.008961857296526432,
+            0.003492321353405714,
+            0.0023635667748749256,
+            -0.005109698045998812,
+            0.009287605062127113,
+            -0.014772292226552963,
+            0.008454295806586742,
+            0.013742019422352314,
+            0.0011183382011950016,
+            -0.010590597055852413,
+            -0.019272159785032272,
+            0.0022972808219492435,
+            -0.000710206339135766,
+            0.01181783340871334,
+            -0.0047498601488769054,
+            -0.000981031684204936,
+            0.0002921315317507833,
+            -0.014476845972239971,
+            -0.009575475007295609,
+            0.0006050958181731403,
+            0.005302873905748129,
+            -0.008961857296526432,
+            0.010378481820225716,
+            -0.005693014245480299,
+            -0.00034160923678427935,
+            0.02627195231616497,
+            0.006526322569698095,
+            -0.007632351014763117,
+            -0.014098069630563259,
+            0.01563590206205845,
+            0.015370759181678295,
+            -0.007454325910657644,
+            0.01933276280760765,
+            -0.016196493059396744,
+            -0.0008366230758838356,
+            0.013916256837546825,
+            -0.009219424799084663,
+            0.0017698341980576515,
+            -0.008196728304028511,
+            0.03554440662264824,
+            -0.0008787619881331921,
+            -0.0035415622405707836,
+            -0.0006401326390914619,
+            0.012976890429854393,
+            -0.0026779514737427235,
+            0.001182730309665203,
+            0.01132542360574007,
+            0.005321812815964222,
+            0.0185146052390337,
+            0.00328020635060966,
+            -0.03378688171505928,
+            -0.007780073676258326,
+            -0.0065490491688251495,
+            -0.008090670220553875,
+            0.009022461250424385,
+            0.025196226313710213,
+            0.01990850456058979,
+            0.006408901885151863,
+            0.001881573349237442,
+            0.018423698842525482,
+            0.015757111832499504,
+            -0.01489350013434887,
+            0.016363153234124184,
+            0.02802947722375393,
+            -0.010030006989836693,
+            0.022741753607988358,
+            -0.005253633018583059,
+            -0.008893677033483982,
+            0.008946705609560013,
+            -0.0015378334792330861,
+            -0.017393426969647408,
+            -0.015681356191635132,
+            0.005022579338401556,
+            -0.008999735116958618,
+            -0.0022897052112966776,
+            -0.018605511635541916,
+            0.004670316819101572,
+            0.017105557024478912,
+            -0.004965762607753277,
+            -3.5243989259470254e-05,
+            0.019484274089336395,
+            -0.006965703796595335,
+            -0.016908591613173485,
+            0.006325571332126856,
+            -0.015007133595645428,
+            0.007946735247969627,
+            -0.013666263781487942,
+            0.010492115281522274,
+            -0.006098304875195026,
+            0.018817627802491188,
+            0.005446809343993664,
+            -0.008090670220553875,
+            -0.0008958069374784827,
+            -0.018984289839863777,
+            0.02065090648829937,
+            -0.033089932054281235,
+            0.00839369185268879,
+            -0.026226500049233437,
+            -0.0009781909175217152,
+            0.0008849170990288258,
+            0.014605630189180374,
+            -0.001218714052811265,
+            0.019696388393640518,
+            -0.002751812804490328,
+            -0.020090317353606224,
+            -0.007086912170052528,
+            -0.0010634156642481685,
+            -0.01618134044110775,
+            -0.030923331156373024,
+            -0.004272601567208767,
+            0.0036267871037125587,
+            -0.004711982328444719,
+            0.0021135739516466856,
+            0.017984317615628242,
+            -0.005613470915704966,
+            0.009173971600830555,
+            -0.02690829709172249,
+            -0.007416448090225458,
+            -0.00355103169567883,
+            0.0010776197304949164,
+            -0.011242092587053776,
+            0.0010596278589218855,
+            0.011226941831409931,
+            -0.019044892862439156,
+            0.013461724855005741,
+            -0.021378157660365105,
+            -0.004371083341538906,
+            -0.008507325313985348,
+            -0.00135412672534585,
+            -0.013446573168039322,
+            -0.028120383620262146,
+            0.025029566138982773,
+            -0.00021495578403118998,
+            -0.003528305096551776,
+            -0.011999646201729774,
+            0.023181134834885597,
+            0.002914686920121312,
+            0.027817361056804657,
+            -0.03157482668757439,
+            0.0013427634257823229,
+            0.011552689597010612,
+            0.0025984083767980337,
+            -0.017287369817495346,
+            0.004746072459965944,
+            -0.005234694108366966,
+            0.02210540883243084,
+            -0.0011694730492308736,
+            -0.0005340751959010959,
+            -0.008234606124460697,
+            0.030999086797237396,
+            0.009136094711720943,
+            0.0012944694608449936,
+            -0.013226882554590702,
+            -0.008060368709266186,
+            -0.002759388415142894,
+            -0.0011069749016314745,
+            0.012855681590735912,
+            0.01290113478899002,
+            -0.012105703353881836,
+            0.0013607552973553538,
+            -0.006878585088998079,
+            0.0011694730492308736,
+            -0.002935519441962242,
+            -0.0004966709529981017,
+            -0.01149208564311266,
+            0.003257479751482606,
+            0.016272246837615967,
+            -0.00780280027538538,
+            -0.006480869371443987,
+            0.0007750718505121768,
+            0.009181547909975052,
+            0.012461753562092781,
+            -0.012742049060761929,
+            -0.018226735293865204,
+            -0.011908739805221558,
+            0.016029831022024155,
+            0.010522416792809963,
+            -0.008204303681850433,
+            -0.013234458863735199,
+            -0.02583257108926773,
+            0.01169662456959486,
+            -0.03293842449784279,
+            -0.012242063879966736,
+            0.005821797996759415,
+            0.028559764847159386,
+            0.019999410957098007,
+            0.012226912193000317,
+            0.001332347048446536,
+            0.002950670663267374,
+            0.00023342114582192153,
+            -0.01916610263288021,
+            0.014408666640520096,
+            -0.011143610812723637,
+            -0.018166130408644676,
+            0.0075906855054199696,
+            0.011939042247831821,
+            0.03148392215371132,
+            0.006942977197468281,
+            -0.0010160685051232576,
+            0.01686313934624195,
+            -0.001218714052811265,
+            -0.013582932762801647,
+            0.01420412678271532,
+            -0.024256860837340355,
+            0.016438908874988556,
+            0.02860521711409092,
+            -0.004780162125825882,
+            0.0008531945641152561,
+            -0.020938776433467865,
+            0.00482561532407999,
+            -0.011529963463544846,
+            -0.002177966060116887,
+            -0.003287781961262226,
+            -0.03460504114627838,
+            0.007946735247969627,
+            -0.006825556512922049,
+            0.01046181283891201,
+            -0.00814369972795248,
+            0.006655106786638498,
+            -0.012431452050805092,
+            0.004253662656992674,
+            0.01513591781258583,
+            0.016105586662888527,
+            0.007060397882014513,
+            -0.025211378931999207,
+            -0.009196698665618896,
+            0.008105821907520294,
+            -0.01349202636629343,
+            -0.02295386977493763,
+            -0.006852070800960064,
+            0.009840618818998337,
+            -0.01061332318931818,
+            -0.00488243205472827,
+            -0.028180986642837524,
+            -0.013681414537131786,
+            -0.0014327229000627995,
+            -0.017817657440900803,
+            -0.005556654185056686,
+            -0.003100287402048707,
+            0.01571165770292282,
+            -0.0049733384512364864,
+            -0.006185423582792282,
+            0.004742284305393696,
+            -0.023878084495663643,
+            0.0037423139438033104,
+            -0.008280059322714806,
+            0.0008266801596619189,
+            -0.0032517979852855206,
+            -0.0014762822538614273,
+            -0.007969462312757969,
+            -0.014332910999655724,
+            0.006893736310303211,
+            -0.016469212248921394,
+            0.002954458352178335,
+            -0.015173795633018017,
+            -0.016499513760209084,
+            -0.00682934420183301,
+            0.00775355938822031,
+            0.021575121209025383,
+            0.006541473791003227,
+            0.021484214812517166,
+            0.005469535943120718,
+            0.028680972754955292,
+            0.0003953482082579285,
+            0.003041577059775591,
+            -8.958956641436089e-06,
+            0.003232859307900071,
+            0.019347915425896645,
+            0.006571775767952204,
+            0.018984289839863777,
+            -0.008742166683077812,
+            0.010514841414988041,
+            -0.010355755686759949,
+            0.010014856234192848,
+            0.006333146709948778,
+            -0.006473293993622065,
+            -0.003799130441620946,
+            -0.010348179377615452,
+            -0.0031419529113918543,
+            -0.016544966027140617,
+            0.0027366618160158396,
+            0.00693161366507411,
+            0.0071588801220059395,
+            -0.011582992039620876,
+            -0.005230906419456005,
+            -0.014704111963510513,
+            -0.0064354161731898785,
+            -0.01831764169037342,
+            0.014385939575731754,
+            0.029226411134004593,
+            -0.016332851722836494,
+            -0.01977214403450489,
+            -0.007772498298436403,
+            0.0007296186522580683,
+            -0.006636167876422405,
+            -0.01240872498601675,
+            -0.009151245467364788,
+            -0.035059571266174316,
+            -0.017666146159172058,
+            0.007586897816509008,
+            0.02484775334596634,
+            0.01387837901711464,
+            -0.013628385961055756,
+            -0.0006983695784583688,
+            -0.016984347254037857,
+            -0.010787560604512691,
+            -0.01535560842603445,
+            -0.02054484933614731,
+            -0.02498411200940609,
+            0.0077952248975634575,
+            -0.004033972043544054,
+            0.014931377954781055,
+            0.008113397285342216,
+            0.0032631612848490477,
+            -0.004371083341538906,
+            -0.0024109138175845146,
+            -0.013196581043303013,
+            0.0012925754999741912,
+            -0.004787737503647804,
+            -0.0017291157273575664,
+            0.03554440662264824,
+            -0.008098246529698372,
+            -0.005871038883924484,
+            0.011060279794037342,
+            0.011128460057079792,
+            -0.013764745555818081,
+            0.0006642796797677875,
+            -0.011249667964875698,
+            0.021847840398550034,
+            -0.021620575338602066,
+            0.019257009029388428,
+            0.0014507147716358304,
+            -0.010840589180588722,
+            0.028999146074056625,
+            0.01892368495464325,
+            0.00743159931153059,
+            0.006579351611435413,
+            0.01672677882015705,
+            0.0044316877610981464,
+            0.016196493059396744,
+            -0.0026552248746156693,
+            -0.004632438998669386,
+            0.009295180439949036,
+            0.015681356191635132,
+            -0.000702630786690861,
+            -0.006942977197468281,
+            0.016438908874988556,
+            0.004056698642671108,
+            0.0075414446182549,
+            -0.01356020662933588,
+            -0.008961857296526432,
+            0.001507531269453466,
+            0.006257391534745693,
+            -0.005939218681305647,
+            -0.0026003022212535143,
+            0.005530139897018671,
+            0.003943065647035837,
+            -0.013302638195455074,
+            0.01689344085752964,
+            -0.016166189685463905,
+            0.008742166683077812,
+            -0.0037972365971654654,
+            0.0121360057964921,
+            -0.006799042224884033,
+            0.0013153020991012454,
+            -0.01818128302693367,
+            -0.003869204083457589,
+            0.0038597346283495426,
+            0.012173883616924286,
+            -0.01262841559946537,
+            0.025484098121523857,
+            0.008166425861418247,
+            0.006602078210562468,
+            0.0059467945247888565,
+            -0.012855681590735912,
+            0.027211319655179977,
+            0.005969521123915911,
+            0.009886072017252445,
+            -0.008893677033483982,
+            0.011136035434901714,
+            0.035362593829631805,
+            0.03060515783727169,
+            0.014135947450995445,
+            0.004469565115869045,
+            -0.018999440595507622,
+            0.015120766125619411,
+            0.0018427487229928374,
+            0.0023806116078048944,
+            0.00863610953092575,
+            -0.011348150670528412,
+            0.01305264513939619,
+            -0.009484568610787392,
+            0.015241974964737892,
+            -0.021272100508213043,
+            -0.011749654076993465,
+            -0.014408666640520096,
+            -0.021075136959552765,
+            -0.01430260855704546,
+            0.034180812537670135,
+            -0.002346521709114313,
+            -0.010393633507192135,
+            0.0045112306252121925,
+            0.008477022871375084,
+            -0.006079366430640221,
+            0.0053142374381423,
+            -0.008204303681850433,
+            -0.013749594800174236,
+            -0.01889338344335556,
+            0.006628592498600483,
+            -0.01550711877644062,
+            0.007651289459317923,
+            -0.003304826794192195,
+            0.0037688282318413258,
+            0.007014944683760405,
+            0.018105527386069298,
+            0.00530666159465909,
+            -0.00135412672534585,
+            0.0014033677289262414,
+            0.0016865032957866788,
+            -0.00931033119559288,
+            0.0108330138027668,
+            0.015408637002110481,
+            -0.0011921997647732496,
+            -0.011279970407485962,
+            -0.009136094711720943,
+            -0.042877525091171265,
+            0.0006264019757509232,
+            -0.013128400780260563,
+            0.0064354161731898785,
+            -0.011196639388799667,
+            0.0022234192583709955,
+            0.015863168984651566,
+            -0.0054051438346505165,
+            -0.0010918239131569862,
+            0.02193874679505825,
+            0.013014768250286579,
+            0.021075136959552765,
+            0.02065090648829937,
+            0.014188976027071476,
+            -0.015302578918635845,
+            -0.03469594568014145,
+            0.002221525413915515,
+            -0.010030006989836693,
+            0.013772320933640003,
+            0.004238511435687542,
+            0.016060132533311844,
+            -0.023893235251307487,
+            -0.011878437362611294,
+            -0.006662682630121708,
+            0.014431392773985863,
+            0.01491622719913721,
+            -0.0032271775417029858,
+            0.02690829709172249,
+            0.009280029684305191,
+            0.0020510759204626083,
+            -0.02281750924885273,
+            0.003312402404844761,
+            0.005321812815964222,
+            -0.008370965719223022,
+            -0.016681326553225517,
+            -0.0004552422615233809,
+            -0.0012168202083557844,
+            -0.0007817004225216806,
+            -0.021650876849889755,
+            0.018832778558135033,
+            -0.004018820822238922,
+            0.0037574649322777987,
+            0.013840501196682453,
+            0.01516621932387352,
+            0.017984317615628242,
+            0.008370965719223022,
+            0.001497115008533001,
+            -0.006605865899473429,
+            -0.0007973249885253608,
+            -0.0005828426801599562,
+            -0.007389933802187443,
+            -0.005598319694399834,
+            0.006382387597113848,
+            -0.01134057529270649,
+            0.008810346014797688,
+            0.000896753859706223,
+            0.006113456096500158,
+            0.010605747811496258,
+            -0.004836978856474161,
+            -0.011628445237874985,
+            0.014628356322646141,
+            -0.0010548931313678622,
+            -0.0055755930952727795,
+            -0.011060279794037342,
+            0.003624893259257078,
+            0.013575357384979725,
+            -0.003096499713137746,
+            -0.013499601744115353,
+            0.00910579226911068,
+            0.002009410411119461,
+            -0.027438584715127945,
+            0.015499543398618698,
+            0.010295150801539421,
+            -0.013370818458497524,
+            0.004980913829058409,
+            -0.003371112747117877,
+            -0.03027183562517166,
+            0.004791525658220053,
+            0.002660906407982111,
+            -0.010643625631928444,
+            0.028938541188836098,
+            -0.003821857040748,
+            0.029256712645292282,
+            -0.004980913829058409,
+            -0.006905099377036095,
+            -0.011355726048350334,
+            0.009416389279067516,
+            -0.018681267276406288,
+            -0.0037953427527099848,
+            -0.008727015927433968,
+            -0.00915882084518671,
+            -0.0009838725673034787,
+            -0.005185453221201897,
+            0.006984642706811428,
+            0.005121061112731695,
+            -0.012704171240329742,
+            0.006757376715540886,
+            -0.010257272981107235,
+            -0.00020856392802670598,
+            0.0006718551740050316,
+            -0.008408842608332634,
+            0.013885954394936562,
+            0.005996035411953926,
+            -0.008340663276612759,
+            -0.0005032995832152665,
+            0.027817361056804657,
+            0.05051366239786148,
+            -0.018863080069422722,
+            0.0009270560112781823,
+            0.01452229917049408,
+            -0.0028824908658862114,
+            -0.0021590273827314377,
+            -0.011976920068264008,
+            0.005821797996759415,
+            -0.019423669204115868,
+            0.013317788951098919,
+            0.006473293993622065,
+            -0.008833073079586029,
+            0.003202557098120451,
+            -0.016711628064513206,
+            0.002621134975925088,
+            -0.0018058179412037134,
+            -0.018681267276406288,
+            -0.015529844909906387,
+            -0.012060250155627728,
+            -0.024181105196475983,
+            0.0004235197266098112,
+            -0.0008546149474568665,
+            -0.013870803639292717,
+            0.006579351611435413,
+            0.008431569673120975,
+            0.032483890652656555,
+            0.0011912527261301875,
+            0.010128489695489407,
+            -2.1513333194889128e-05,
+            -0.0010416359873488545,
+            -0.005988460034132004,
+            -0.01523439958691597,
+            0.00015387803432531655,
+            -0.002342734020203352,
+            0.022590242326259613,
+            0.005549078807234764,
+            0.017560089007019997,
+            0.0017887729918584228,
+            -0.016514664515852928,
+            -0.00596573343500495,
+            -0.007408872712403536,
+            -0.018060073256492615,
+            0.0062649669125676155,
+            -0.005473323632031679,
+            0.009605777449905872,
+            -0.015863168984651566,
+            -0.02731737680733204,
+            0.00792400911450386,
+            0.012234487570822239,
+            0.028514310717582703,
+            -0.0035623949952423573,
+            0.001716805505566299,
+            -0.005533927585929632,
+            0.016408607363700867,
+            0.032180868089199066,
+            -0.0008550884085707366,
+            0.006950552575290203,
+            -0.019999410957098007,
+            0.0017404790269210935,
+            0.013704141601920128,
+            -0.008401267230510712,
+            -0.01821158453822136,
+            0.003939277958124876,
+            -0.020635755732655525,
+            0.004583198111504316,
+            -0.020393338054418564,
+            0.030408194288611412,
+            -0.0023522034753113985,
+            0.008105821907520294,
+            0.004310478921979666,
+            0.01760554127395153,
+            -0.0031646795105189085,
+            0.010545143857598305,
+            0.013916256837546825,
+            0.014908651821315289,
+            0.0009980766335502267,
+            -0.011189064010977745,
+            -0.03790797293186188,
+            0.001859793672338128,
+            0.00817400123924017,
+            0.018635815009474754,
+            -0.01980244740843773,
+            0.011681473813951015,
+            -0.01296931505203247,
+            -0.020363036543130875,
+            -0.01003758329898119,
+            0.011575416661798954,
+            -0.006139970384538174,
+            0.03378688171505928,
+            -0.005874827038496733,
+            -0.008204303681850433,
+            0.005136212334036827,
+            0.010454237461090088,
+            0.0066778333857655525,
+            0.014704111963510513,
+            0.0326354019343853,
+            0.010196669027209282,
+            0.014173824340105057,
+            -0.004859705455601215,
+            0.02399929240345955,
+            -0.003931702114641666,
+            0.003954428713768721,
+            0.00509454682469368,
+            -0.022832660004496574,
+            -0.0040983641520142555,
+            -0.003931702114641666,
+            -0.016711628064513206,
+            -0.007458113599568605,
+            0.0032366469968110323,
+            0.008802770636975765,
+            -0.014045040123164654,
+            -0.00649980828166008,
+            -0.01350717805325985,
+            0.01218903437256813,
+            0.002730980049818754,
+            0.018302490934729576,
+            -0.029802151024341583,
+            0.009825468063354492,
+            -0.017484333366155624,
+            -0.017044952139258385,
+            0.006511171814054251,
+            -0.0038332203403115273,
+            0.006855858489871025,
+            -0.018272189423441887,
+            -0.01318900566548109,
+            0.012817803770303726,
+            0.0007897494360804558,
+            0.013060221448540688,
+            -0.009825468063354492,
+            0.0009706153068691492,
+            0.006064215209335089,
+            -0.009833043441176414,
+            -0.010961798019707203,
+            -0.007977037690579891,
+            -0.009681533090770245,
+            -0.008848223835229874,
+            0.005621046293526888,
+            0.01656011864542961,
+            -0.013742019422352314,
+            -0.003685497445985675,
+            0.016135888174176216,
+            0.006416477728635073,
+            -0.008166425861418247,
+            0.004075637552887201,
+            -0.003793448908254504,
+            -9.948806837201118e-05,
+            0.0013740125577896833,
+            0.012439027428627014,
+            -0.0017632056260481477,
+            0.024665940552949905,
+            -0.02654467150568962,
+            0.007598260883241892,
+            -0.0054051438346505165,
+            0.005446809343993664,
+            0.018226735293865204,
+            -0.02366596832871437,
+            -0.013082947582006454,
+            0.006276329979300499,
+            -0.01966608688235283,
+            -0.008060368709266186,
+            0.0054051438346505165,
+            -0.00016405766655225307,
+            -0.019923655316233635,
+            -0.012923861853778362,
+            -0.034786853939294815,
+            -0.02309022843837738,
+            0.024665940552949905,
+            0.008401267230510712,
+            -0.0073672072030603886,
+            0.018060073256492615,
+            -0.018438849598169327,
+            0.0172722190618515,
+            -0.002225313102826476,
+            -0.006613441277295351,
+            0.003935490269213915,
+            0.007674016058444977,
+            0.025362888351082802,
+            0.002653330797329545,
+            0.008264907635748386,
+            -0.0042082094587385654,
+            -0.014931377954781055,
+            -0.006321783177554607,
+            0.01980244740843773,
+            0.02471139281988144,
+            0.018605511635541916,
+            -0.018575210124254227,
+            0.010742107406258583,
+            -0.011749654076993465,
+            0.01818128302693367,
+            -0.006537686102092266,
+            0.00039037675014697015,
+            0.01859036087989807,
+            0.007969462312757969,
+            0.025241680443286896,
+            0.019923655316233635,
+            0.006852070800960064,
+            0.014128371141850948,
+            0.011908739805221558,
+            0.001204509986564517,
+            -0.012832955457270145,
+            -0.01154511421918869,
+            -0.02840825356543064,
+            0.042847223579883575,
+            0.0025434857234358788,
+            0.008317936211824417,
+            -0.027635548263788223,
+            -0.00021389048197306693,
+            -0.0034942151978611946,
+            -0.018832778558135033,
+            0.005939218681305647,
+            0.03260510042309761,
+            0.03284751623868942,
+            0.021438762545585632,
+            -2.888172457460314e-05,
+            0.0012234487803652883,
+            0.0329081192612648,
+            -0.013673839159309864,
+            -0.00937851145863533,
+            -0.01034060399979353,
+            -0.018393397331237793,
+            -0.006852070800960064,
+            -0.0032423287630081177,
+            -0.01031030248850584,
+            -0.0061172437854111195,
+            -0.009272454306483269,
+            0.0238629337400198,
+            0.0001374249259242788,
+            -0.0022594030015170574,
+            0.006605865899473429,
+            -0.020878173410892487,
+            0.011537538841366768,
+            -0.02813553437590599,
+            -0.013431422412395477,
+            -0.008651260286569595,
+            -0.006950552575290203,
+            -0.018802475184202194,
+            0.008643684908747673,
+            -0.006942977197468281,
+            0.021438762545585632,
+            0.002914686920121312,
+            -0.023590214550495148,
+            0.007530081085860729,
+            0.015317730605602264,
+            -0.0003825644962489605,
+            0.014037464745342731,
+            -0.003890036838129163,
+            0.009060339070856571,
+            0.0045112306252121925,
+            0.011681473813951015,
+            0.016954045742750168,
+            0.010651201009750366,
+            0.009030036628246307,
+            0.003437398700043559,
+            -0.001167579204775393,
+            0.01878732442855835,
+            -0.0007163614500313997,
+            -0.03405960276722908,
+            2.9236827685963362e-05,
+            -0.014628356322646141,
+            0.029499130323529243,
+            -0.009795165620744228,
+            0.009514871053397655,
+            0.008105821907520294,
+            0.007098275702446699,
+            -0.0067119235172867775,
+            0.007916433736681938,
+            -0.008742166683077812,
+            0.0052763596177101135,
+            0.015832867473363876,
+            -0.0042877523228526115,
+            -0.0014980619307607412,
+            0.007840678095817566,
+            0.00407184986397624,
+            -0.005454384721815586,
+            -0.019711541011929512,
+            -0.012287517078220844,
+            0.00726114958524704,
+            -0.02065090648829937,
+            -0.004367295652627945,
+            0.01161329448223114,
+            -0.005454384721815586,
+            -0.009620928205549717,
+            0.01393140759319067,
+            0.02136300690472126,
+            -0.002382505452260375,
+            0.022893264889717102,
+            -0.01349202636629343,
+            -0.009833043441176414,
+            -0.007984613068401814,
+            -0.004374871030449867,
+            -0.041120000183582306,
+            0.025984082370996475,
+            0.01068150345236063,
+            -0.006871009711176157,
+            0.01895398646593094,
+            0.0022499337792396545,
+            -0.002954458352178335,
+            -0.0011041341349482536,
+            0.00010499217023607343,
+            -0.03281721472740173,
+            0.03660498186945915,
+            -0.007007369305938482,
+            0.0005449649761430919,
+            0.010272424668073654,
+            0.042029064148664474,
+            -0.004590773954987526,
+            -0.005859675817191601,
+            0.01225721463561058,
+            0.005336964037269354,
+            0.031938452273607254,
+            -0.010416359640657902,
+            -0.014340486377477646,
+            -0.01815097965300083,
+            -0.004833190701901913,
+            -0.01939336769282818,
+            -0.0038313264958560467,
+            0.014037464745342731,
+            -0.019650936126708984,
+            -0.011242092587053776,
+            -0.001287840772420168,
+            -0.012537509202957153,
+            -0.0010747789638116956,
+            0.008598231710493565,
+            0.017832808196544647,
+            -0.011348150670528412,
+            -0.004450626205652952,
+            -2.2904154320713133e-05,
+            -0.005003640428185463,
+            0.0185146052390337,
+            0.0020056227222085,
+            -0.01702980138361454,
+            0.01321930717676878,
+            -0.00526499655097723,
+            0.006749800872057676,
+            0.0074429623782634735,
+            0.007939159870147705,
+            0.05145302787423134,
+            0.010317877866327763,
+            -0.001339922659099102,
+            -0.01607528328895569,
+            0.023196285590529442,
+            -0.01749948412179947,
+            -0.014181400649249554,
+            -0.02119634486734867,
+            -0.0029923359397798777,
+            -0.007060397882014513,
+            -0.028014326468110085,
+            -9.137987944995984e-05,
+            0.0016221113037317991,
+            -0.0024980325251817703,
+            -0.010045158676803112,
+            0.0005208179936744273,
+            -0.011529963463544846,
+            -0.006954340264201164,
+            -0.008545203134417534,
+            -0.016044981777668,
+            0.02240842953324318,
+            -0.021817538887262344,
+            -0.008855799213051796,
+            0.011514811776578426,
+            0.03118089959025383,
+            -0.005011215806007385,
+            -0.014173824340105057,
+            -0.019817598164081573,
+            0.0025643182452768087,
+            -0.01763584464788437,
+            0.012181458994746208,
+            -0.01225721463561058,
+            0.015045011416077614,
+            -0.03142331540584564,
+            -0.006079366430640221,
+            -0.008992158807814121,
+            0.012840530835092068,
+            0.019817598164081573,
+            -0.0025093958247452974,
+            0.00606800289824605,
+            -0.010295150801539421,
+            0.030211230739951134,
+            -0.00012286569108255208,
+            -0.018605511635541916,
+            0.00873459130525589,
+            0.007977037690579891,
+            0.012537509202957153,
+            -0.004435475450009108,
+            0.00257757562212646,
+            0.011567841283977032,
+            0.012742049060761929,
+            -0.01634800247848034,
+            -0.003174148965626955,
+            -0.017969166859984398,
+            0.005996035411953926,
+            0.006999793462455273,
+            0.021711481735110283,
+            -0.0013550736475735903,
+            0.018272189423441887,
+            -0.01563590206205845,
+            0.00019412307301536202,
+            -0.011749654076993465,
+            -0.0082270298153162,
+            -0.0028237802907824516,
+            -0.003958216868340969,
+            0.006685409229248762,
+            -0.028211290016770363,
+            -0.01274962443858385,
+            0.005287723150104284,
+            -0.004992276895791292,
+            -0.03405960276722908,
+            0.0053710537031292915,
+            -0.02725677192211151,
+            0.013128400780260563,
+            -0.002397656673565507,
+            -0.017847958952188492,
+            -0.00030846628942526877,
+            0.026332557201385498,
+            -0.003677921835333109,
+            -0.012537509202957153,
+            0.015544996596872807,
+            -0.006757376715540886,
+            -0.02119634486734867,
+            -0.005420294590294361,
+            0.014658658765256405,
+            -0.0010719380807131529,
+            -0.012393574230372906,
+            -0.029635490849614143,
+            0.012992041185498238,
+            0.0026874206960201263,
+            -0.007386146113276482,
+            0.010893618687987328,
+            0.025953780859708786,
+            0.008151275105774403,
+            -0.005924067925661802,
+            0.0015757111832499504,
+            0.0049581872299313545,
+            0.00300938100554049,
+            0.020287280902266502,
+            0.02728707529604435,
+            -0.008014915511012077,
+            -0.0011145505122840405,
+            -0.014954105019569397,
+            0.015802564099431038,
+            0.0013020449550822377,
+            0.010507266037166119,
+            0.021893294528126717,
+            0.007045246660709381,
+            -0.005738467443734407,
+            0.003905187826603651,
+            -0.023135682567954063,
+            0.01420412678271532,
+            -0.012272365391254425,
+            0.000622140767518431,
+            0.008749742060899734,
+            -0.007283876184374094,
+            0.01437078882008791,
+            -0.0015245762187987566,
+            0.0339989997446537,
+            0.025211378931999207,
+            0.0043218424543738365,
+            0.0007362472242675722,
+            -0.028559764847159386,
+            0.003547244006767869,
+            0.008651260286569595,
+            0.014499573037028313,
+            -0.0076664406806230545,
+            0.006261179223656654,
+            -0.004355932120233774,
+            0.00959820207208395,
+            -0.0065490491688251495,
+            -0.015893470495939255,
+            0.006113456096500158,
+            0.0012689019786193967,
+            0.006795254070311785,
+            0.0023408401757478714,
+            -0.008863375522196293,
+            -0.024362917989492416,
+            -0.025984082370996475,
+            0.021984200924634933,
+            -0.03918066248297691,
+            0.0073293293826282024,
+            0.011234517209231853,
+            0.03324144333600998,
+            -0.008158850483596325,
+            -0.004980913829058409,
+            0.006276329979300499,
+            -0.016711628064513206,
+            -0.004147605039179325,
+            0.008204303681850433,
+            0.012385998852550983,
+            -0.002420383272692561,
+            0.00974971242249012,
+            0.01318900566548109,
+            -0.012378423474729061,
+            0.03660498186945915,
+            0.016514664515852928,
+            0.01467380952090025,
+            -0.004007457755506039,
+            0.01312082540243864,
+            0.0066172294318675995,
+            0.019044892862439156,
+            0.025165924802422523,
+            -0.017454031854867935,
+            -0.002365460619330406,
+            -0.02292356640100479,
+            0.024165954440832138,
+            0.00726114958524704,
+            0.004556683823466301,
+            -0.008310360834002495,
+            0.014870774000883102,
+            0.0017954016802832484,
+            0.0028199926018714905,
+            -0.006696772295981646,
+            -0.010173942893743515,
+            0.019241856411099434,
+            0.0034317169338464737,
+            0.014870774000883102,
+            4.8501173296244815e-05,
+            -0.005503625608980656,
+            0.006052851676940918,
+            0.014832896180450916,
+            -0.023590214550495148,
+            -0.00203592493198812,
+            -0.019484274089336395,
+            -0.03202936053276062,
+            0.009022461250424385,
+            0.0016135887708514929,
+            -0.007052822504192591,
+            0.03230207785964012,
+            -0.010083036497235298,
+            -0.019787294790148735,
+            -0.008067944087088108,
+            -0.013113250024616718,
+            -0.010469388216733932,
+            0.0008271536789834499,
+            0.003908975515514612,
+            -0.00012428610352799296,
+            -0.009317907504737377,
+            -0.005749830510467291,
+            0.01990850456058979,
+            -0.0030358952935785055,
+            -0.00753386877477169,
+            0.00868156272917986
+          ],
+          "embedding": [
+            0.0040707582550429606,
+            -0.010230625151989194,
+            -0.021520242101903186,
+            -0.010115756270640508,
+            0.026738734586159035,
+            0.01796370233223267,
+            -0.012245199029994153,
+            0.06506924494147726,
+            -0.0014087587222033442,
+            0.009300068719271429,
+            0.003787396973047584,
+            0.002818772639754174,
+            0.022129077062176306,
+            -0.021526709808445683,
+            -0.0034388135367225824,
+            0.02763761747879405,
+            0.013210013805395276,
+            0.025255872672296602,
+            0.00029011480909986703,
+            -0.05438974762290585,
+            0.016671640187490865,
+            -0.016426252259893328,
+            -0.056591180651012554,
+            0.009335648425523075,
+            0.006141624459418827,
+            0.02568943956694641,
+            0.019377132054042116,
+            0.017946464337078746,
+            -0.057184776985927686,
+            0.03208527139096687,
+            0.030132392063755547,
+            -0.003846032371791839,
+            -0.008376524427531415,
+            -0.0010513963058713435,
+            -0.02734868388724371,
+            -0.014879400370545479,
+            0.04383920959748825,
+            0.02563613659516289,
+            -0.020117467074438997,
+            0.0398016699155716,
+            0.029896497884197775,
+            -0.0004195844530067313,
+            -0.023302720788253625,
+            -0.001168337916577517,
+            0.009149021369661628,
+            -0.02508145721900992,
+            -0.004307594138480351,
+            0.00550940415564303,
+            0.014016998918718476,
+            -0.028607379616884317,
+            -0.0018317656587526954,
+            -0.01973464940415756,
+            0.03762738392056255,
+            0.015358106058583114,
+            -0.03931907682104936,
+            0.011519637298832444,
+            0.004482080558226775,
+            0.0747867063383242,
+            0.05550679018263077,
+            0.023243129601931176,
+            0.02671631686665238,
+            -0.009686146566573536,
+            -0.04088582189725495,
+            -0.0065344623594924085,
+            0.019617295999325712,
+            -0.004722087402575777,
+            -0.002871844114437583,
+            0.003935087265300519,
+            0.03229890413393201,
+            -0.02813577047904649,
+            -0.05124172132597234,
+            0.03325235407841968,
+            -0.008687655557217192,
+            -0.011027174838246948,
+            -0.003653553063547375,
+            -0.02350737945182988,
+            0.016223087939183468,
+            0.03789252356462714,
+            0.0034350623378603083,
+            0.00195158657888739,
+            0.011789712862232108,
+            -0.0004435439660769551,
+            -0.0037330456918125894,
+            0.0031552713243866858,
+            0.04465432992298031,
+            -0.0032465421571915712,
+            -0.0075366743861922745,
+            -0.059958752731410256,
+            0.016784616861008125,
+            -0.028734506065772967,
+            -0.03035575736273943,
+            0.02680390892611741,
+            0.02338368719038086,
+            -0.0030658375437784277,
+            -0.03754896050794851,
+            0.07005659523248504,
+            0.0020711575553570695,
+            0.013034425083076923,
+            -0.02316832262132332,
+            0.021332142030732894,
+            -0.0318756233020585,
+            0.02270782509507388,
+            0.01570810309454529,
+            0.004788124787612736,
+            0.026727250200142456,
+            -0.022380142778058276,
+            -0.008593910379436904,
+            0.014439754382387034,
+            0.015007697617383174,
+            0.07662531963612977,
+            0.011195998137723795,
+            -0.010809283538470098,
+            -0.042227822120219305,
+            0.003824320933275175,
+            -0.03797629427607094,
+            -0.02347390684453348,
+            -0.0526876440280276,
+            -0.017799861680287913,
+            0.01701482796266917,
+            0.002730395144917155,
+            0.03789451759899825,
+            0.038338545350313946,
+            0.028463789278406586,
+            0.015494583705011921,
+            -0.05700237266350479,
+            0.047895170359075166,
+            -0.0064406894534929406,
+            -0.0007954721295065988,
+            0.012543029839729358,
+            -0.004703334868741216,
+            -0.0060569679284068,
+            0.012324901254711192,
+            -0.019883935809784068,
+            -0.04253565942154215,
+            0.02164409276211115,
+            -0.0075367555415808635,
+            0.03617159147722157,
+            0.07567745988686005,
+            -0.04225977020036783,
+            -0.022548837589622003,
+            0.029811602145850144,
+            -0.02175792196819533,
+            0.02069861458115363,
+            -0.00941331829664397,
+            -0.035479639885943264,
+            0.020352552580119105,
+            0.003678588588550461,
+            0.06776105274336729,
+            -0.02346047721661687,
+            0.02721075076349443,
+            -0.029386781707644253,
+            0.023530831921117993,
+            -0.02077660533100372,
+            0.020022475323372906,
+            0.024195118676513715,
+            -0.028378518720494666,
+            -0.012900969023913307,
+            0.019264089970719624,
+            -0.0374983624629265,
+            -0.019726171557384355,
+            0.00979636085642416,
+            -0.014479528435832063,
+            -0.014533384583777662,
+            0.005904623758767224,
+            0.010344475500263744,
+            0.05659646358026115,
+            0.04163866113482093,
+            0.007152494819169405,
+            -0.07584500130974763,
+            0.05471858597878099,
+            0.03580211013427615,
+            0.044161429444475306,
+            0.02258180566046248,
+            0.00980779221334715,
+            0.0011193505603397395,
+            0.018086566768892507,
+            0.015605066702191871,
+            -0.00954727399396313,
+            -0.027950079146919375,
+            0.03028514025485407,
+            -0.006351669761257431,
+            0.005095804801686436,
+            0.009620040085743162,
+            0.019748001227987134,
+            0.033709535059510916,
+            0.008203559956940196,
+            0.007389752602218563,
+            0.04197771878351742,
+            -0.02067363758471765,
+            0.007543739188471913,
+            0.011207941261799052,
+            0.013964827251172813,
+            0.03839562075248879,
+            0.0009775244543292637,
+            0.006469507611492216,
+            -0.022808235395746654,
+            -0.024795004398633292,
+            0.006249024864814684,
+            0.011292877109066925,
+            0.01596839619826819,
+            -0.0014105396434174883,
+            -0.04152075438277094,
+            0.008267788411079337,
+            -0.028834191212891387,
+            0.030940526405027906,
+            -0.0018271056434014425,
+            0.006185475047820603,
+            -0.007920574674727123,
+            0.012907560753043616,
+            -0.015165704767187252,
+            -0.021962244496450727,
+            0.0037071053072833147,
+            -0.013508376064336042,
+            -0.015178823540513864,
+            -0.02518557053061311,
+            0.011693256916760944,
+            -0.01716015674592331,
+            0.02048897513447166,
+            -0.003412078966361745,
+            -0.001807298511378529,
+            -0.0012575655363779904,
+            0.03116103961636643,
+            0.011668533361641842,
+            -0.011637010548030312,
+            -0.040260195687925236,
+            -0.011129376691522074,
+            -0.0034493180842871908,
+            -0.02098244789711689,
+            0.026658743116016938,
+            -0.01923736325953796,
+            -0.009723937181300192,
+            -0.013134679989565051,
+            -0.011974548914818263,
+            -0.004028712301052159,
+            0.019628329175867303,
+            -0.028192034966932684,
+            -0.012638520793012082,
+            -0.011568652932013111,
+            -0.0025919910519445737,
+            0.012497908463405111,
+            0.015113186712897145,
+            0.014880607940212641,
+            -0.03059227859452334,
+            -0.021899716217105084,
+            -0.0008513506358489523,
+            0.011985931118464679,
+            -0.01321584595585435,
+            -0.011735288366213981,
+            0.010710683541435907,
+            0.01961722172525993,
+            -0.023571946726192585,
+            -0.03364281880125354,
+            0.0014460076743756465,
+            0.016563167828467187,
+            -0.02891589059260944,
+            0.04595166331401113,
+            0.034271553243501465,
+            -0.027557971577291153,
+            -0.007138821520412525,
+            -0.039508407807651844,
+            0.06814329636830853,
+            0.00022136123564675344,
+            -0.039718016705753735,
+            0.0503457594963753,
+            -0.003580581891576583,
+            0.013820796398521455,
+            0.02054050419088757,
+            0.035682661890720495,
+            -0.0013220739026852943,
+            -0.0052759394697328675,
+            0.04733018719162478,
+            -0.0187845855131196,
+            -0.03982699468608443,
+            -0.023905460465070062,
+            0.002631233133039351,
+            -0.03090995554989513,
+            -0.012617612051488022,
+            -0.013605787969473513,
+            0.039672783793764745,
+            0.008669582242097126,
+            0.009974277556736362,
+            0.005535636914843583,
+            0.014180668493093584,
+            0.012042659947850441,
+            0.011870115226306776,
+            0.00451259454348581,
+            -0.014023048592193627,
+            -0.007769033996392623,
+            -0.05715987527668358,
+            0.003374564518384728,
+            -0.028801158523506985,
+            -0.00756331369170698,
+            -0.02163054894055572,
+            0.005367181573233089,
+            0.02308705505078254,
+            -0.005438909398653553,
+            -0.00873620909426313,
+            -0.0036530652237719077,
+            0.013920407174296543,
+            -0.01539613379172909,
+            -0.03908211213269472,
+            -0.016810726548844992,
+            0.007460173854386443,
+            -0.007069189426742819,
+            -0.041197222498539215,
+            -0.003843233633335066,
+            0.004373367480223065,
+            0.016194121660459908,
+            -0.029165150066068787,
+            -0.00735139702658342,
+            0.017257895082337448,
+            0.014153313280536095,
+            -0.002376178042741926,
+            -0.023223700836427403,
+            -0.0319877537205969,
+            0.02088060774052704,
+            0.0012829290529515294,
+            0.03251550040316258,
+            -0.04128453069131847,
+            -0.023376114412063578,
+            0.04083022735076889,
+            0.005690491480259003,
+            -0.05810403800689653,
+            0.016582434816685452,
+            0.010402038964417569,
+            0.016795644054313566,
+            -0.00908590043098552,
+            0.05003656634510042,
+            -0.037216197954986735,
+            -0.013027960261454162,
+            -0.007090943677204207,
+            -0.05175371787508879,
+            -0.0020065660448063322,
+            -0.014908489552053589,
+            0.010627472758306387,
+            -0.03275473161158777,
+            -0.017334608434133866,
+            0.025321047439048318,
+            -0.0033515195870701753,
+            0.011526068468647207,
+            -0.026783879816504975,
+            -0.0012830935419413407,
+            -0.017506908439998053,
+            -0.02114853058440477,
+            0.004878884893669383,
+            -0.020936900837115968,
+            -0.0025587027781733642,
+            0.01557850246191957,
+            0.023652335794299834,
+            0.011489428313303988,
+            0.0019089137782552768,
+            0.02054907941022243,
+            0.014361232983780595,
+            -0.0056413109308506085,
+            -0.0010300050480036684,
+            -0.03413262505887213,
+            -0.018821160485010483,
+            0.03361989907065425,
+            0.013125889701529858,
+            -0.003742167496379789,
+            0.008813100942466033,
+            -0.0383265293638935,
+            -0.035316957263713485,
+            0.03020499323039515,
+            -0.05001623971243151,
+            -0.00855479667673597,
+            -0.005939674940109814,
+            -0.026451831943555135,
+            -0.008032873049527808,
+            0.0009431571685654594,
+            -0.014387605128858954,
+            0.016382559541755014,
+            -0.024909510232038502,
+            0.003715736505444385,
+            -0.006016712902409688,
+            -0.009110197692342108,
+            0.05507263160354303,
+            -0.022103031032598353,
+            -0.0475415595313825,
+            0.021958434235402655,
+            -0.016670809659931243,
+            0.0323519363801999,
+            0.03867494951405658,
+            0.011124508964363016,
+            -0.008002445714431074,
+            0.034041838454818577,
+            0.018400123235267234,
+            -0.01161754789471199,
+            -0.025901844576206774,
+            0.021800298132870245,
+            0.00943488425697727,
+            -0.05906691123008531,
+            -0.00782549274387971,
+            -0.019357446956797447,
+            0.008682376344965976,
+            0.02863077694455766,
+            0.0024925977373894447,
+            -0.002682170968419412,
+            -0.013590235283325244,
+            -0.021511149507762284,
+            -0.03255453916518719,
+            0.009039307015853905,
+            0.008305286476500875,
+            0.009312625027668458,
+            -0.0014144165766983945,
+            -0.007853869334376857,
+            0.009067065760942163,
+            0.022485201611182105,
+            0.03966189080765722,
+            -0.041627289519629344,
+            0.002707086456239785,
+            -0.022678178050044222,
+            -0.03653753142622053,
+            0.0036574646778835588,
+            0.029996160786576367,
+            0.02901163765559891,
+            -0.01971777193183787,
+            -0.0036589934841109884,
+            0.009181234938554578,
+            -0.025979483002227535,
+            -0.00773645103995767,
+            0.004280770859737866,
+            0.04116668301017644,
+            0.006201085381597264,
+            0.01402728258112097,
+            0.001060407317110527,
+            -0.00920801784000087,
+            0.02109880889040528,
+            0.025211022083982727,
+            -0.029866678264942617,
+            0.014439702646820226,
+            0.009379206296908577,
+            0.04156435812161122,
+            -0.037309655469525846,
+            -0.011819731348299495,
+            -0.02766498020409985,
+            -0.010454143230794476,
+            0.027911532296566614,
+            -0.04127890742215716,
+            -0.006948536183086887,
+            0.010207784113909938,
+            -0.0016584043356136647,
+            0.016437495428919636,
+            -0.014607861750686952,
+            -0.03222311919805017,
+            -0.000610723203118213,
+            -0.02867445198803213,
+            -0.024772816792038133,
+            -0.016510309753766612,
+            -0.018581201775667577,
+            -0.07450654873867177,
+            0.00026832248829154255,
+            -0.06724696990816072,
+            -0.02447002541264633,
+            0.04885569249258513,
+            -0.010195604207134053,
+            0.03332905935704749,
+            -0.02815272112271314,
+            -0.005296340304911926,
+            -0.05908835042526019,
+            0.0038476172058970774,
+            0.025753576863447866,
+            0.02386003857017594,
+            0.005206447371391696,
+            0.024799057871126032,
+            0.012060738268265606,
+            -0.04146503002980051,
+            -0.024949907556862643,
+            -0.014503475223770958,
+            -0.0016340466065744286,
+            -0.0034009167013795906,
+            -0.031298432995411005,
+            0.036241272149880994,
+            0.008902644391348814,
+            0.03550223922667804,
+            0.04463301409716232,
+            -0.010385984413759852,
+            -0.03305004479734889,
+            -0.006952739338820577,
+            -0.0175981588092108,
+            0.042668533202572856,
+            -0.017112184620086308,
+            0.005149173014526686,
+            -0.03219589107997635,
+            0.024181808150698054,
+            0.0014339790422945406,
+            0.01254721190611376,
+            0.011099458649404119,
+            0.0027626836767364253,
+            -0.026883640446090158,
+            0.0024160770766463607,
+            0.023874514847956222,
+            -0.022558433617669373,
+            0.00917667715473093,
+            -0.006754091166465801,
+            -0.033686935718776144,
+            -0.01078389638118685,
+            -0.013595327486167114,
+            0.024712781935748277,
+            0.013168464807451798,
+            0.005875864604862434,
+            -0.02417814420558463,
+            -0.0023978800459714144,
+            0.028650463366555304,
+            0.020176980768301288,
+            -0.02118184201861616,
+            0.019082340279671355,
+            0.019734925740100277,
+            -0.04209645071102001,
+            -0.04217310001316092,
+            0.006553913719098956,
+            -0.003735719227468833,
+            0.0061793909220459,
+            -0.028285889794908054,
+            0.008181788076484304,
+            0.004432488437404309,
+            -0.01743905248466261,
+            0.041221269621866216,
+            -0.014892354419172107,
+            0.005087102208335201,
+            -0.022198383716186407,
+            -0.011936388003605221,
+            -0.00027487042024290816,
+            0.044599215801158845,
+            0.008068975171827777,
+            0.003496950921841073,
+            0.02238779276447619,
+            0.0054839808262270315,
+            0.04816723616939378,
+            -0.02685221260466647,
+            -0.0190356946337995,
+            0.016464136459560113,
+            -0.0176092780782677,
+            -0.012513274095432353,
+            -0.010695421445334002,
+            -0.004784348908457072,
+            -0.022203041334932165,
+            0.010788155380415583,
+            -0.018353897272137543,
+            -0.02277431189907027,
+            -0.003945465850796918,
+            0.016024639792926797,
+            -0.006248368166794934,
+            -0.0038769803892794875,
+            0.0016292556540434277,
+            0.02412864830338344,
+            0.02892955453819876,
+            -0.02408022470555968,
+            0.019190282372105023,
+            0.018941584034503536,
+            -0.006709516380602965,
+            -0.020956377933747532,
+            -0.01964885206488053,
+            0.01434623424964373,
+            -0.01352968586212152,
+            -0.013872556892707697,
+            0.004052798542613835,
+            0.009201147374299997,
+            -0.004086067985859617,
+            -0.04185829685685201,
+            -0.014668698123513235,
+            0.003179403563664022,
+            -0.0006415014901450319,
+            0.019360184344240947,
+            -0.0150817538567364,
+            -0.029771999278536083,
+            0.014221776097046751,
+            0.022944884879589678,
+            -0.04493425220030206,
+            0.02117577719533236,
+            -0.0030648751497261983,
+            -0.00016251533643103604,
+            -3.817296018601414e-05,
+            0.008727767382077138,
+            0.026914779443266063,
+            0.03456565094655899,
+            0.01974806772696219,
+            0.00450800007152441,
+            -0.008952599700759673,
+            0.025184190461155245,
+            0.009846044083064704,
+            -0.025168735183134738,
+            0.010983353565382029,
+            0.016740721743326522,
+            0.0070838930012654296,
+            0.009173879407277282,
+            -0.006078288608766152,
+            -0.018269347893097643,
+            0.013181494188066667,
+            -0.01435693747022231,
+            0.01678907949532146,
+            0.05154463552784974,
+            -0.00039632596424461964,
+            -0.02461439550772063,
+            0.008418147037164186,
+            -0.006544319842335593,
+            -0.010065129286193278,
+            0.02214628550460128,
+            0.0029515574751370417,
+            -0.030862152033847717,
+            0.007509556025223583,
+            0.00675485304374519,
+            0.01894936231559203,
+            -0.02355525295639599,
+            -0.01387885858544803,
+            -0.00431398020087544,
+            0.0021208056468763523,
+            0.01580766504149989,
+            -0.004739477121073512,
+            -0.013204823447960706,
+            0.02994098229517376,
+            0.025496728234662373,
+            -0.00026124946738213036,
+            0.015060064980768175,
+            0.022262345898533903,
+            0.009968408664790368,
+            0.007588431141088797,
+            -0.0083063413609779,
+            -0.031336272171937095,
+            0.0018957858383561667,
+            0.05800811417728381,
+            -0.014937085296865802,
+            0.04313875432235027,
+            0.02018476425923236,
+            0.015657644644987107,
+            0.017880263053253607,
+            -0.01662268082870562,
+            -0.008740053682875323,
+            -0.0017630394204179638,
+            -0.008294417990024647,
+            -0.027613464270562948,
+            -0.005953209556351086,
+            -0.02379961210881074,
+            -0.013878676554701543,
+            0.015418315245979704,
+            -0.014764728065382074,
+            0.0027420640870858457,
+            0.010395538988699663,
+            -0.006590633833835581,
+            -0.00841537209927409,
+            -0.003943772700797381,
+            0.03376338017637296,
+            0.011980354721458463,
+            0.027761929822019422,
+            -0.00011128539412582973,
+            0.009342430952107655,
+            -0.018419526564396063,
+            -0.006839752805318664,
+            0.03271190887864607,
+            0.006011520427944665,
+            0.004652835471939905,
+            0.01485920920301398,
+            0.014305626381398871,
+            0.038015677219487023,
+            -0.005102005401565164,
+            -0.018204460895989052,
+            0.02065495133597456,
+            -0.0049699013356374775,
+            0.019149036693562712,
+            0.008825981577388056,
+            -0.002660055848523568,
+            -0.004292379561953203,
+            0.03090270710860331,
+            -0.020243440437217503,
+            0.018735727616321387,
+            -0.02238236089688497,
+            -0.0276584413929402,
+            0.023028408906176946,
+            0.008320436733620496,
+            0.04047641118026783,
+            0.011463851066078041,
+            -0.02247143742166039,
+            -0.034588725413285185,
+            0.009827713930789184,
+            0.00337071450636392,
+            -0.024446280199879062,
+            0.017542777160283324,
+            0.00780836483490212,
+            0.008453886667283516,
+            0.016624116261219637,
+            -0.004175682824655864,
+            -0.0042784362933758835,
+            -0.001546656717896957,
+            -0.008819739495525477,
+            0.01406790007187259,
+            -0.015487658289541583,
+            -0.021457182907140035,
+            0.024687169583245905,
+            -0.03915911962859764,
+            0.01771290139419629,
+            -0.006510046907173026,
+            -0.00997450880725535,
+            -0.02059560682587115,
+            1.5939836949198004e-05,
+            -0.007159493752862143,
+            -0.0004782363814453761,
+            -0.017503181957042913,
+            0.016643971600145645,
+            0.024056442566869802,
+            -0.013209211443909658,
+            -0.00445521990927108,
+            0.0020313626821877526,
+            -0.029235692455820056,
+            0.006662209097255912,
+            0.016993027053110635,
+            0.00529940612845993,
+            0.0026104743851393716,
+            0.0001707934714983891,
+            -0.031289446611163674,
+            -0.005782208620284939,
+            0.03053516889069683,
+            0.05590704747438521,
+            0.021628254147289437,
+            0.02403314981042378,
+            0.02908606663381421,
+            0.02717206823892633,
+            -0.003198844932040613,
+            0.0016217508103442348,
+            -0.01620660606431983,
+            0.027098323143010743,
+            -0.011664168290676943,
+            0.04322175021589083,
+            0.008652323184522541,
+            -0.049493020736445056,
+            0.004537958157696794,
+            0.00943239454147692,
+            0.034918603888916544,
+            0.016678716235690318,
+            0.0005887414054803805,
+            -0.005894338664998101,
+            0.03279165663336323,
+            -0.02735726709032382,
+            -0.011962070182991011,
+            -0.00036014407560325284,
+            -0.019079003730607717,
+            -0.015938295169860884,
+            0.010357611943935459,
+            0.04278924678845588,
+            0.0009957483598272451,
+            -0.0071422790468850215,
+            0.01571201177743999,
+            -0.009445307674383,
+            0.02286338492075306,
+            0.007336440384050498,
+            0.0042809497342459904,
+            0.025581467690095977,
+            -0.0025007871391868105,
+            0.02300539823997411,
+            -0.000628898771317254,
+            0.004582863909134464,
+            -0.015960578501883316,
+            -0.00930649396114688,
+            0.027838455754803444,
+            -0.011112268986721,
+            -0.029253746504396393,
+            -0.008734785561831333,
+            0.00472579547270089,
+            -0.0003805825712986138,
+            0.0021854358149324227,
+            -0.01183885533661493,
+            -0.010840951201591629,
+            0.019965691788904057,
+            -0.004922943301424112,
+            0.003800048754664761,
+            0.012664049429916252,
+            -0.0030124219674414897,
+            -0.01910844996750822,
+            0.01889983984736845,
+            -0.003858283778519141,
+            -0.024168821338992813,
+            0.020083955711766945,
+            0.01850497889545421,
+            0.045156567155119454,
+            -0.01586824055785609,
+            0.004612757419198795,
+            0.017497774574602505,
+            -0.002172088240279357,
+            -0.019617327081566827,
+            -0.014944795772093997,
+            -0.018212462639041492,
+            -0.015975283672064702,
+            0.012837956594691596,
+            -0.01243789877725244,
+            0.019500571353639314,
+            -0.0380735226388175,
+            0.012590900534826689,
+            0.004832168601136412,
+            0.007505385730121269,
+            0.016011983211212992,
+            0.025994560026860358,
+            0.002157436199771672,
+            -0.033465476168065784,
+            -0.01014599891204166,
+            0.007845373376916211,
+            0.010093432683262838,
+            -0.002849185178237036,
+            -0.018147004183434554,
+            0.003633790611809425,
+            -0.0266147420637404,
+            -0.018644422659309035,
+            0.022926737299761104,
+            -0.04798772015599113,
+            0.017908415301694906,
+            0.024859125158586466,
+            -0.04343699525707025,
+            -0.04886699607556352,
+            -0.011459188432595229,
+            -0.036774381316312464,
+            0.0014128005215813578,
+            -0.0005256067584181517,
+            0.011194992852264102,
+            -0.009342140079829378,
+            -0.018598671981710183,
+            0.01584789414301952,
+            -0.0036412641242200418,
+            -0.004225840239935549,
+            -0.00616696736018045,
+            -0.0025598631463178384,
+            -0.010188803157718543,
+            -0.03244694662411977,
+            -0.03290891136780909,
+            -0.00033742479296272127,
+            -0.009743927051414455,
+            -0.02334288359945272,
+            -0.03447062782872289,
+            -0.008728968107967028,
+            -0.037753659749578956,
+            -0.00515561961442376,
+            0.04070890150714597,
+            0.027517147608335957,
+            0.009449414978133334,
+            0.008086900089929912,
+            -0.013989034604490105,
+            0.05691047488419298,
+            -0.020425883381389086,
+            0.039466620642512155,
+            0.005301493569940605,
+            -0.0037222353899918703,
+            0.013075425214878277,
+            0.027710826422351564,
+            -0.006253665106995653,
+            -0.0017412928073238833,
+            -0.01928096630475438,
+            -0.012920476859996416,
+            0.009444408011467955,
+            0.015799884767184885,
+            -0.0025083061619197524,
+            0.001274457251739847,
+            0.00930533161700852,
+            -0.0006652478540264343,
+            0.017258038491335247,
+            0.03996268817486439,
+            0.003833844612473671,
+            -0.00826404141706295,
+            -0.018822002570963614,
+            -0.003188844900317347,
+            -0.011807006674248829,
+            0.010482058734260632,
+            -0.015410635475145028,
+            -0.011540803927522013,
+            -0.01249428777269578,
+            -0.017081856786754605,
+            -0.002183034976477535,
+            0.003114834030705039,
+            0.0027475411722881747,
+            0.02261627635192778,
+            -0.015872027027080824,
+            -0.022487581027490516,
+            -0.0035823117885461506,
+            -0.02398778449664045,
+            8.13716819067491e-05,
+            -0.01050060782407259,
+            -0.015616181283381218,
+            0.013676480044963827,
+            -0.021513173575512936,
+            -0.01091399089076612,
+            -0.008699176177861256,
+            -0.0006986953850759474,
+            0.005853723710671249,
+            -0.00993355748621621,
+            -0.00010241843582159751,
+            0.01150872694850241,
+            0.005559552266827916,
+            -0.012336449703762907,
+            0.0039431718621504,
+            -0.012740550272925562,
+            -0.004432059174951158,
+            -0.03592309122711106,
+            -0.019665686718735042,
+            -0.01846795637714379,
+            0.005642094448288269,
+            -0.005108964028543505,
+            -0.020586678054151854,
+            0.00668364061048941,
+            0.0034775127050453774,
+            0.004713169526661792,
+            -0.010723667875209928,
+            0.013944735241748791,
+            0.010882420168586074,
+            -0.003518245916227596,
+            0.013827334494230662,
+            -0.001330399936559009,
+            -0.020286697504699764,
+            -0.011139922402071095,
+            0.03538258948179278,
+            -0.024961826464903997,
+            -0.0018019328351763667,
+            -0.0026423695690168066,
+            0.00597003516870726,
+            -0.03071835946501977,
+            -0.013634212232744325,
+            0.02164400349828422,
+            -0.005846552894831441,
+            -0.026367014426434226,
+            0.03399116480234033,
+            0.03473258231629753,
+            0.010041605592929843,
+            -0.013423384340589794,
+            0.04730054254696893,
+            -0.012968978954938308,
+            -0.018274800841400665,
+            0.026826644936918647,
+            0.0278389611094493,
+            0.02525164095926168,
+            -0.029782550447890466,
+            -0.003010713086487275,
+            -0.02393629084357885,
+            0.017993366287762866,
+            -0.0263476663180971,
+            0.01864011499179424,
+            -0.02332263152537849,
+            0.016014908881756034,
+            0.01868464825773604,
+            -0.003104041044182249,
+            0.026313491086684425,
+            -0.0052945248919367815,
+            0.015253731232325074,
+            0.0012371308759955573,
+            -0.02119765517013431,
+            0.007795466385020642,
+            -0.006469500272298062,
+            0.025762169721880456,
+            0.00833992768202466,
+            0.01801901639978976,
+            0.022019940825351116,
+            -0.0409522298213741,
+            0.02003698451081499,
+            -0.001367916864679543,
+            0.01782984895971748,
+            0.0038516134262842373,
+            -0.007885432294782304,
+            0.014221728513400065,
+            0.010991352188016265,
+            -0.012851099261924757,
+            0.003840539873231741,
+            0.0008923482291826014,
+            -0.028928104542716937,
+            0.007066004768771198,
+            -0.026914574065060926,
+            0.025866907400807087,
+            -0.005526774784152086,
+            -0.00998122210956923,
+            -0.020744532673097317,
+            -0.0071425207262770865,
+            0.003754468896273156,
+            -0.012747874997429746,
+            -0.011385805601999295,
+            -0.024700457669046855,
+            -0.02401291382882952,
+            0.029591582802488962,
+            -0.014054332096344003,
+            -0.04015978298687865,
+            0.001234440863385445,
+            -0.023301255402129767,
+            0.01193125007773239,
+            -0.005793124129267758,
+            -0.024440100649464773,
+            -0.042563399082257154,
+            -0.017548888269389457,
+            -0.0009394887961485922,
+            0.022739763644266633,
+            -0.00114156949394707,
+            0.041791395064755515,
+            0.0008816696205437911,
+            0.022207132857533028,
+            -0.015547580543606755,
+            0.005319731264840819,
+            -0.02071229859314312,
+            -0.041495579742912285,
+            0.03660432378985971,
+            -0.014637031447181937,
+            0.025854844497435178,
+            0.03324325296704186,
+            -0.013009347371118843,
+            0.001354504885302625,
+            0.02652570709474851,
+            -0.014618100923440427,
+            -0.010782232747863238,
+            -0.0034936699515378212,
+            0.003461670978569965,
+            -0.006468116285021298,
+            -0.027793221545159466,
+            -0.007752458229193391,
+            0.04193706715422074,
+            0.017894069367616637,
+            0.021928994684423043,
+            0.022606401934291136,
+            -0.017972404868085358,
+            -0.026402048654300563,
+            0.003656646576244765,
+            -0.013129722990153284,
+            -0.018090360146368677,
+            -0.01624598856366761,
+            0.012397150789705746,
+            0.011741002980073786,
+            -0.004052422390675261,
+            -0.01836750423615859,
+            -0.020929335029248472,
+            -0.016327412417122416,
+            -0.0042615345226698015,
+            -0.008458117095535,
+            -0.005002582649405199,
+            -0.012158173174049797,
+            -0.020409616620022143,
+            0.03706368540614453,
+            0.0285565642978937,
+            -0.0041732933884846566,
+            -0.00607154864704767,
+            -0.011142150176334957,
+            0.008834431806990914,
+            0.006021232046087652,
+            -0.005028870471297621,
+            -2.3396043292139426e-05,
+            0.005112033372713154,
+            0.022822312435625314,
+            0.007271235299348001,
+            0.0035791789459533856,
+            -0.06340305811365254,
+            -0.005392000517722054,
+            0.022560201096078862,
+            0.012581401789917604,
+            -0.0032248219865223496,
+            0.025180718735801837,
+            0.0006451724964970927,
+            -0.009714514747463593,
+            -0.018389422219956937,
+            0.024728862150150004,
+            0.008760329201960964,
+            0.020988700992999768,
+            -0.014974763313686516,
+            0.003845297486923296,
+            0.0006445983847138896,
+            0.00750394530065325,
+            -0.0019792992847026793,
+            -0.008578254004439607,
+            0.025776677078006516,
+            0.016593438298233645,
+            -0.005695578040595484,
+            0.01604018433477423,
+            -0.0017497260488899657,
+            -0.021460543443428924,
+            -0.010265213095609554,
+            0.00042017209251926397,
+            -0.010570864288876387,
+            -0.018015790888295944,
+            0.017569414447505878,
+            -0.002601036282535863,
+            0.0072399377790164435,
+            0.030281707151249355,
+            0.0423183067569767,
+            0.009566493398534713,
+            -0.005928603293907002,
+            -0.0037936911251953654,
+            0.015477873304924939,
+            0.01513127995244932,
+            0.010741256034652754,
+            0.03015858543221727,
+            -0.029247492875021843,
+            -0.006690306960497428,
+            0.0046653727410352576,
+            0.004610405633429778,
+            -0.011692179099382171,
+            0.030055897497741527,
+            0.04569730592994177,
+            0.006736813244286893,
+            -0.004799978513117371,
+            0.002121067028052434,
+            0.011299889107780231,
+            0.007768137218140627,
+            -0.018927080714961783,
+            -0.0012222962555201746,
+            -0.006009911026251445,
+            -0.014001766533604134,
+            0.026941585318261085,
+            -0.003960837039351648,
+            -0.011848232223811548,
+            0.02903357274500332,
+            0.1105267435019781,
+            -0.006011029338781496,
+            -0.0053646998281195206,
+            -0.03242321704621517,
+            -0.015194102811332289,
+            -0.008860590919147765,
+            0.0056988205877963076,
+            0.01635957585050017,
+            0.020327661913559888,
+            -0.0008943408274661007,
+            0.016820860249955754,
+            0.02822175614376368,
+            -0.013854827839129885,
+            0.007906244895327178,
+            -0.030891381690403466,
+            0.015702117924362928,
+            -0.005383974926093977,
+            -0.012032955400313954,
+            -0.0026169511828034054,
+            -0.008618307035330928,
+            -0.0037356680273096105,
+            0.0009666430093455632,
+            0.011296307527684261,
+            0.004427516814276546,
+            -0.014484198313369681,
+            0.002829529156309784,
+            -0.008595300985863424,
+            -0.005122996529848575,
+            -0.02393571073523279,
+            -0.024119245543788293,
+            0.005223147047205755,
+            0.0019389305548629142,
+            6.344308292376579e-05,
+            0.006848965054539454,
+            -0.003810712095343454,
+            0.030149081317333418,
+            -0.01700322008272963,
+            0.0012288912810468446,
+            0.01193226302726573,
+            0.012332749933413344,
+            0.01688602002694633,
+            0.0028600984823761366,
+            -1.9882692552530084e-06,
+            0.021172977047420227,
+            -0.010514036509402561,
+            0.01460334896316005,
+            0.013091417303145243,
+            -0.04103572381458182,
+            0.0089746401463863,
+            -0.0027841289484449873,
+            -0.024252447218794344,
+            0.004446472209246425,
+            -0.018987867343500102,
+            -0.00881974688349601,
+            0.0005365625696166774,
+            0.012054719659248512,
+            0.01960300421041832,
+            -0.008553943699846696,
+            0.02111201386480852,
+            -0.0003501657740808554,
+            0.004177088753111459,
+            -0.012879944670340026,
+            0.0065222126975051685,
+            -0.007563354540402343,
+            0.0008168128607271199,
+            -0.018213241937597355,
+            0.012526561612175514,
+            -0.0008285725179894544,
+            -0.015737003159436623,
+            -0.011337688275309921,
+            0.004737346137976519,
+            -0.0027993545140114705,
+            0.0030843163114700313,
+            0.010153740340626154,
+            -0.008815302484351509,
+            0.004919275910399053,
+            -0.0017541042538165297,
+            0.002562816687714991,
+            0.014285249186421434,
+            -0.029816964593980453,
+            0.005357227738099325,
+            0.009154842495811638,
+            -0.00048782303639333943,
+            -0.013074976152495576,
+            -0.004054505559072866,
+            0.0028322102449485015,
+            -0.016877328079971877,
+            -0.012128199639779175,
+            0.023421330164419888,
+            -0.012565145848820333,
+            0.008609070188894315,
+            -0.010936598722317274,
+            -0.018401801734378692,
+            -0.01066119413133089,
+            -0.005254229780560808,
+            0.01234969508270852,
+            -0.0038427767204741877,
+            -0.006947594635952202,
+            -0.01756894543880097,
+            -0.007890897373987378,
+            -0.047246698568222466,
+            0.009617876521097624,
+            0.027157555263862723,
+            -0.011355758309667893,
+            -0.04050311381480088,
+            -0.01017774338301804,
+            0.02437257208967103,
+            -0.004056329295095018,
+            -0.008345450168215274,
+            -0.001599171409135998,
+            0.018358386463243014,
+            0.0027207445181251406,
+            -0.007292967749349572,
+            -0.011267861428599221,
+            0.01657507035043879,
+            0.015105061280816214,
+            0.021419854579631276,
+            -0.02088952706825817,
+            0.006295984081511286,
+            0.020973020776936996,
+            -0.03073246543555915,
+            0.0023720435876790876,
+            0.03363966216709148,
+            0.010412588613998134,
+            0.013023311559810034,
+            -0.0010217662933572685,
+            -0.0075567381078977215,
+            -0.008248818794295818,
+            0.011848200323549865,
+            0.005066463524567732,
+            -0.023174489297821385,
+            0.021823825355858556,
+            0.020368868756448753,
+            0.0026042856858509524,
+            0.003457067317891926,
+            0.0005539745115410357,
+            0.01062670844108589,
+            0.009948889803890108,
+            0.05100741556350918,
+            -0.006493714124814164,
+            -0.014021751055930108,
+            0.015063988102064609,
+            0.02488059237697814,
+            -0.0023499941119596228,
+            0.01087282522970853,
+            -0.02612218935931224,
+            -0.021423880914461062,
+            0.023389294741183817,
+            0.008967763132667892,
+            0.009637806238448432,
+            -0.010815706635624337,
+            -0.011390999462441815,
+            0.031750675158698435,
+            -0.01307829462211914,
+            0.00759749911213318,
+            -0.02169895957209634,
+            0.01387879409130081,
+            -0.008606362268863441,
+            0.032229658009209815,
+            -0.0009066257436946125,
+            0.011062436424260079,
+            0.028010486866468023,
+            -0.007835805879241447,
+            -0.019657486763625547,
+            0.003967480939987674,
+            -0.015577952617624695,
+            0.02188429826317828,
+            -0.020502758155233298,
+            -0.0002362500702658591,
+            0.0040935406015014265,
+            -0.010056462958039452,
+            -0.0057038083589231615,
+            0.011475170331848666,
+            0.0015345076181482267,
+            0.006359262977587827,
+            -0.020293014401106,
+            0.01369706440475074,
+            -0.003989482215141903,
+            0.007887824197823247,
+            -0.015925992045349828,
+            0.014370223626942766,
+            -0.02309139937510304,
+            -0.021808736033141787,
+            0.008142621179624918,
+            0.0108996933426044,
+            -0.007589050511458233,
+            -0.025971539510030145,
+            0.02250992080022228,
+            -0.024980833805349545,
+            0.024979008145354243,
+            -0.006622339940663538,
+            0.002566378488088469,
+            0.004913874740363343,
+            -0.008102103508732738,
+            -0.0028098905427027783,
+            0.0057865587191235565,
+            -0.0073509382400023995,
+            -0.014123756810537343,
+            -0.019453380788786282,
+            -0.04083629135586275,
+            -0.03177927047231389,
+            0.022927581786088044,
+            0.011557232599801421,
+            0.0017073832658476193,
+            0.02339552660545296,
+            -0.02247471449131386,
+            -0.017890866287500053,
+            -0.009775068417716798,
+            0.025184899522229628,
+            -0.04408541669037361,
+            0.012104255392615549,
+            -0.014669458760022245,
+            -0.0245258242481677,
+            0.006274252098230249,
+            -0.004161960147481513,
+            0.009299171724026455,
+            -0.010779421730862708,
+            -0.04713627671791054,
+            -0.005192684784377356,
+            0.02330975631183273,
+            -0.0036802876364168935,
+            -0.006197548344287773,
+            0.03048284650641829,
+            -0.0019503797069087845,
+            -0.005038254331399784,
+            -0.005524875459245342,
+            0.027783596017224226,
+            -0.024950083604372984,
+            0.010988398531346805,
+            0.008304937912856357,
+            -0.008485065895193593,
+            0.006572665697403556,
+            0.0011518846966102158,
+            0.011927348472282199,
+            0.015632588644277183,
+            -0.015559413716826264,
+            0.02262990861928957,
+            0.011446850286896198,
+            0.006506075777765303,
+            0.008015225053785824,
+            0.00010054430757843335,
+            0.0022975742216194894,
+            -0.024948925449668525,
+            0.026047924967851216,
+            0.01888211557921096,
+            0.0001632181017876392,
+            0.012553094476616919,
+            -0.015042621225242777,
+            -0.02782009986290425,
+            0.02606646297992026,
+            -0.016847653820092335,
+            -0.00045272379465939274,
+            0.020053091486344633,
+            -0.016210036358455773,
+            -0.007930550661303182,
+            -0.01123514849773852,
+            -0.0162675150307978,
+            0.01803076559921804,
+            -0.012160217543572069,
+            -0.02216594536057653,
+            0.002594675636753233,
+            0.016443123828746582,
+            0.02134480656850119,
+            0.026694575953313482,
+            -0.011470930697351533,
+            0.017768927048191052,
+            0.0008653408391607816,
+            0.001865765599795092,
+            0.025745954171649543,
+            0.01702677011968492,
+            -0.008474889109168202,
+            0.025347176686722313,
+            -0.0016686223698600755,
+            -0.002421921060670558,
+            -0.012265504084322075,
+            -0.0165214436458303,
+            -0.007545471138114208,
+            -0.015333235556146196,
+            -0.0051830106590765685,
+            0.000762746782664525,
+            -0.0018118022696726346,
+            0.10876576838038368,
+            0.0034948044540582626,
+            -0.011608460689239234,
+            -0.005870263500417206,
+            0.007510774308250375,
+            0.0023525201199670765,
+            -0.006846154764389575,
+            0.016082721103476375,
+            -0.018616626204086575,
+            -0.023593744079394,
+            0.004743481974389998,
+            -0.0009262286319469741,
+            0.00870122878944883,
+            0.02029523251993269,
+            -0.008608837818169835,
+            -0.0200567711498164,
+            0.007089477722191928,
+            0.005535149644041221,
+            0.0023640746516071774,
+            0.005256431824732927,
+            -0.0006118671321824933,
+            0.008209920240361308,
+            0.02448422122961942,
+            -0.015622892878860465,
+            -0.029770836607612903,
+            0.010535371601298532,
+            -0.00015994156470228762,
+            -0.007193425789067776,
+            -0.011046316463797128,
+            0.014728841492009627,
+            -0.0011488451990980268,
+            -0.02751404133172324,
+            0.017663132712536528,
+            -0.019055335178900415,
+            0.014643246200933506,
+            0.0025348345163347464,
+            -0.00016387173410118788,
+            0.022565953069005714,
+            0.013101158251834834,
+            -0.04272085076882881,
+            -0.0034442020309914823,
+            0.01041205537746844,
+            0.011059422927946552,
+            0.0116446016862942,
+            0.0012992535381387887,
+            -0.018670801241345514,
+            -0.011788984490283379,
+            -0.009109849103208542,
+            -0.021268488414296117,
+            -0.012523033785938503,
+            -0.011393853408628214,
+            0.010743646386439633,
+            0.0012964030814325153,
+            0.023809566099779113,
+            -0.003030535542277875,
+            -0.006526351867141901,
+            -0.000430699447948508,
+            0.00880561288237257,
+            0.009694646009259077,
+            -0.02079011138439714,
+            0.008294338719512953,
+            -0.01823922789220214,
+            0.011819769685519291,
+            0.006416044253693268,
+            -0.022799568832306067,
+            -0.009911619276012527,
+            0.007152786695893951,
+            0.008393677195358034,
+            -0.013569953300860946,
+            -0.006503988731872938,
+            0.02299498754558268,
+            0.018870619152710703,
+            0.012233964644738905,
+            0.024994616203196145,
+            0.03838969722235787,
+            0.005744255912479976,
+            0.027682123067526515,
+            -0.016791471092091604,
+            0.008459745383680297,
+            -0.022493868174622695,
+            0.012902262638928338,
+            -0.00460399519266689,
+            -0.0050431343566427886,
+            -0.018188192498411616,
+            0.033765105245412236,
+            -0.009159751849801773,
+            -0.0027150644587327763,
+            0.034778224828120714,
+            -0.0066502021793471075,
+            0.004525230911930458,
+            0.005096748531775903,
+            -0.016067049494073893,
+            0.033274672789154185,
+            0.01686831040872425,
+            0.016353966592947262,
+            -0.021631941883929817,
+            0.011748317746828848,
+            -0.005675701689448899,
+            0.0001273442591116645,
+            -0.020738819037390263,
+            -0.0037528323797385733,
+            0.0027725541343763104,
+            0.0059860509282538546,
+            -0.016556134518370437,
+            0.02033623476757896,
+            0.00019647023567641008,
+            0.0052895744489259875,
+            0.03253462247073863,
+            0.01043311189804924,
+            0.019182938878855655,
+            0.010051051068017807,
+            0.019523739618278902,
+            0.0281939391326836,
+            0.025685787482623124,
+            0.0029932688462254147,
+            -0.018668945726139946,
+            -0.0009326901784812825,
+            -0.007880352863625262,
+            0.010492063055849727,
+            0.022849902214818293,
+            -0.022619228461640748,
+            -0.02772644874135741,
+            0.018566981184526675,
+            -0.0022608185345048414,
+            0.020031103834122595,
+            0.007544464625538984,
+            -0.020935181884939796,
+            0.003407355515103286,
+            -0.023178230166337178,
+            -0.0005418155871620593,
+            -0.031039187720144967,
+            -0.028591174077243604,
+            -0.0007659241904525392,
+            0.011945344534674635,
+            -0.01688153017778311,
+            0.03357788163306365,
+            0.017667778671101954,
+            -0.014289109343082812,
+            0.01604684528817524,
+            -0.011054587325539086,
+            -0.0036959091138078683,
+            -0.005819853870004682,
+            -0.00203755010992417,
+            -0.02157944474979592,
+            0.005138614763345542,
+            -0.0033509138150859593,
+            -0.0024871060745482883,
+            -0.016936575573598154,
+            -0.021901830366872595,
+            0.016375173843894617,
+            0.016742770709641697,
+            -0.006606252542838758,
+            0.039771905404651496,
+            -0.004238071340064403,
+            0.00018364315349545657,
+            0.011690521707403059,
+            0.0010018392078563795,
+            -0.013651921068215563,
+            -0.010829994512513501,
+            0.012362647379788833,
+            0.011321403447882543,
+            -0.018942043512252663,
+            0.005763773608775982,
+            -0.009131565038297889,
+            0.002089216671850269,
+            0.005438887787329238,
+            -0.008183835275629478,
+            0.006690280329947951,
+            -0.01673229763262922,
+            0.015833397876364723,
+            0.00295474085300386,
+            0.003186554859191664,
+            -0.011797859525926772,
+            -0.020991927615680524,
+            0.01576413532776261,
+            0.0032403514518527287,
+            0.018819824905201635,
+            -0.026755218746060662,
+            -0.00012946467986739473,
+            -0.009552681500884924,
+            -0.0028941357661144826,
+            -0.012533800273548772,
+            0.013975640628890588,
+            0.016864739319953755,
+            0.0063941926578529,
+            0.01781016436903586,
+            -0.0026268224264609724,
+            0.015368284773773678,
+            -0.018122416427266218,
+            0.008686009180697955,
+            -0.006801074032721299,
+            -0.018537521789181048,
+            0.0013237338264105415,
+            0.027863741654435434,
+            -0.004670012662973029,
+            0.011066853115455966,
+            0.011704884649534624,
+            0.018954056680143452,
+            -0.0012626215734593707,
+            -0.0030850559460101866,
+            0.015610404665171367,
+            -0.005838226405658572,
+            -0.006861883833171456,
+            0.020872966233916948,
+            0.032951653763679145,
+            -0.006716215648910268,
+            -0.0021292599951407863,
+            0.010886596327481677,
+            -0.004901684487088035,
+            -0.008644584974417106,
+            -0.009587037829603014,
+            -0.03158137900717491,
+            0.024369089446706894,
+            0.012092412439477474,
+            -0.009601042951586109,
+            0.022690862146548786,
+            0.020752792552278052,
+            0.011412743968296708,
+            -0.010471860708195726,
+            0.02291536038589634,
+            -0.01268035142112569,
+            0.024168869865226136,
+            0.032482831775271157,
+            0.004400973556358811,
+            0.0022312906286559167,
+            0.0021175489661843985,
+            0.01691348460546639,
+            0.006050357346231155,
+            -0.007068133214675996,
+            0.011656003476409986,
+            -0.018487923180009552,
+            0.01598530891381001,
+            -0.004432366596808033,
+            0.02808371007118948,
+            -0.01337145216499331,
+            -0.010473700033368562,
+            -0.00537325376270318,
+            0.00010726431372328729,
+            0.006445701087575359,
+            -0.003324002570256215,
+            0.005255443653813165,
+            0.0007072577303252132,
+            -0.017825649146944237,
+            0.003050128130393781,
+            0.004825967170196209,
+            0.009018799597337101,
+            0.004103653497834134,
+            0.007249957401824204,
+            -0.012360355929483,
+            0.01441885664789707,
+            0.004401820363035808,
+            -0.0026190938717927086,
+            -0.0007163225760398784,
+            0.011124538295057815,
+            -0.013232822471919533,
+            -0.004895624967029031,
+            0.005281228035321008,
+            0.0059275753353475414,
+            -0.007507924998741734,
+            -0.004484140989405051,
+            0.011908076549392969,
+            -0.0037917291321648483,
+            0.006351543504726163,
+            -0.004409464890606014,
+            -0.0037760571637986827,
+            0.005303661313434895,
+            -0.00974511544836468,
+            -0.004204167898342997,
+            -0.0007093125445559502,
+            -0.006869250680233214,
+            0.00786202375969793,
+            0.005971033227117194,
+            0.010606085272785863,
+            0.006969437776542303,
+            0.004293129058563288,
+            0.004114452739123398,
+            0.004071017374828283,
+            -0.002445681623750424,
+            -0.00592480769363354,
+            -0.004617501328052899,
+            -0.012067458701076778,
+            -0.008335239153829813,
+            0.009025028998949398,
+            0.008653779421856954,
+            -0.019363018792788574,
+            0.02048011578221327,
+            -0.001188320960771659,
+            0.00772165044232104,
+            -0.01602086976290928,
+            -0.000907553574811279,
+            0.005464880629734941,
+            0.008267702703609528,
+            0.016136762227127184,
+            -0.03654209526492064,
+            0.005179791231187316,
+            -0.0037321883493524766,
+            0.016026013709955444,
+            -0.008215703411373182,
+            -0.006979816286354862,
+            0.02564060526656963,
+            -0.01450763388985563,
+            -0.0009847758228920171,
+            -0.016142304341196283,
+            -0.01407778890808498,
+            0.010815725252959506,
+            0.02050940280233426,
+            0.020684808835848383,
+            -0.0004965802897137407,
+            -0.0005781445948759235,
+            0.00036179841583004193,
+            0.022775804864342608,
+            0.015545681427566235,
+            0.018697154644607316,
+            -0.004543500782944068,
+            0.007028436664826706,
+            -0.023057215501463973,
+            0.009912581556496658,
+            0.0011795417129232596,
+            -0.008215027850908477,
+            -0.008554137076267468,
+            -0.008703378564520125,
+            -0.010448538943182258,
+            -0.011773811327190638,
+            0.004106616839960258,
+            0.013895013678116032,
+            0.004190549543250594,
+            0.0031571190600981074,
+            -0.0225385788570314,
+            -0.006243193914542516,
+            -0.011229641377928767,
+            -0.01674214043453308,
+            -0.007367389908398082,
+            -0.02386334969150212,
+            -0.028021339684039734,
+            0.0002794207370355175,
+            -0.02411861258809571,
+            -0.015958336406924612,
+            0.005268815864773021,
+            -0.01546610225139129,
+            0.00898600137695074,
+            -0.008396481690803911,
+            -0.011873486990732417,
+            0.0074743033679134105,
+            0.015336444788692,
+            -0.017676977079590463,
+            -0.010165317702640217,
+            -0.01690524990506033,
+            -0.013460451917691695,
+            0.010918550283871637,
+            0.00595960977280955,
+            0.009250157926438213,
+            -0.012447979337850305,
+            0.0062932435120863365,
+            -0.012673905639312696,
+            0.015073013552721701,
+            -0.03769166067890086,
+            -0.029992249463405965,
+            -0.00014990754540999537,
+            0.007288036630677436,
+            -0.0020735742042505426,
+            -0.018695120921334806,
+            0.0003860116505754868,
+            -0.003608995261360232,
+            -0.014900191335144973,
+            -0.013866216738266176,
+            0.025251475305524053,
+            -0.006788483107616133,
+            0.01223227424986397,
+            0.0034937277040015626,
+            0.03438333560343413,
+            0.008803852645562129,
+            0.014666206816302367,
+            -0.005709421955160623,
+            0.003742433643393657,
+            -0.0033706258426092744,
+            0.006906464031352315,
+            0.0076578888497945185,
+            0.009276592191710524,
+            0.008025321004305927,
+            0.004576470294889512,
+            0.002887346543823031,
+            -0.001925468214331082,
+            0.02125241545487307,
+            0.025671028960818562,
+            -0.00037467906648160044,
+            0.008981063192139038,
+            0.0009387900084090577,
+            0.0002735789978680719,
+            -0.008374290340790827,
+            0.006547013284767913,
+            -0.008274637814401893,
+            0.004041066530254948,
+            0.012897549078838394,
+            -0.009150863940242148,
+            -0.010211568689323908,
+            0.016575088843292566,
+            -0.012380099999574783,
+            -0.03420455578618604,
+            0.026550654680719987,
+            0.005324512466403649,
+            -0.0035670106106754245,
+            -0.010657928468872608,
+            0.0037176027853158167,
+            -0.01103891334887315,
+            0.008096915671982816,
+            0.022587659771132487,
+            -0.0013527917944322358,
+            -0.0018085171207746743,
+            0.0038503794877980586,
+            -0.011870484733648887,
+            -0.007881489342351292,
+            0.006462772034381968,
+            0.0315696514747912,
+            0.01326417653779577,
+            -0.020400369539140532,
+            -0.015223808971473513,
+            0.020406862349006738,
+            -0.023142176712111575,
+            -0.006254959806032159,
+            0.009039506093607052,
+            0.0009087520492922078,
+            -0.022766046838415165,
+            0.0009662037093312435,
+            0.004701966191595815,
+            0.005946458676746661,
+            -0.006358986495019584,
+            0.03237532341275415,
+            0.001075992257265679,
+            0.015892020271402903,
+            -0.017189453434945788,
+            0.0017514632185789299,
+            -0.014189278671121677,
+            -0.014265447388465805,
+            -0.029809460132405256,
+            0.0005557491671071693,
+            -0.0030324198417091704,
+            0.011038984948790744,
+            0.024808067315720335,
+            0.01145103533194245,
+            0.014432246996007246,
+            0.010552852947578788,
+            0.01513689173494211,
+            -0.013653722180565311,
+            -0.029009757850432732,
+            -0.005171522291984226,
+            -0.018001048743500243,
+            -0.02297482628212251,
+            0.003962613844063587,
+            0.013645125285548414,
+            0.006278662690346082,
+            0.023786381386575035,
+            0.0021738516691483163,
+            0.0009580591750940011,
+            -2.690805459518147e-05,
+            0.008894091070322921,
+            -0.004952203441302471,
+            -0.006553986197076102,
+            0.003570189040872711,
+            -0.000652714137223501,
+            0.021228360881461853,
+            -0.005758946428212041,
+            7.897153237591187e-05,
+            0.014242175492137011,
+            -0.005622256348934008,
+            -0.0035009192529791146,
+            -0.017339926143652377,
+            0.006248842389703505,
+            -0.01622011586976866,
+            0.025783086900871728,
+            0.018450783573486916,
+            -0.009109295293546267,
+            -0.013469521573468915,
+            -0.011259555201313116,
+            0.02773097452736661,
+            0.004631282554715908,
+            0.010787055220617197,
+            0.018228322680582464,
+            0.01461254259249465,
+            0.01784830193964943,
+            0.004664725683522323,
+            -0.0034339781146649384,
+            -0.0007692550982545399,
+            0.021176646860433974,
+            -0.014700178590249292,
+            -0.002701208084390198,
+            -0.008842788249773623,
+            0.0021799005046816444,
+            -0.008431299000066363,
+            0.006086447848362669,
+            -0.010793016137040172,
+            -0.015251381840165954,
+            0.0189503457652375,
+            -0.02660767353735015,
+            0.016423753600066336,
+            -0.00593016170680532,
+            -0.00482891500342298,
+            -0.027459214027897378,
+            0.010728039528784636,
+            0.0027866316191608297,
+            0.0022519974195000687,
+            0.010770472550461752,
+            -0.021427634123401408,
+            -0.01867717541189363,
+            0.008532444554010673,
+            -0.011457117350255678,
+            -0.007152301670244725,
+            -0.00348619742388509,
+            0.0154715607296319,
+            -0.00017898955187849443,
+            -0.009925365923862246,
+            0.02085093731060963,
+            -0.007175309371250024,
+            0.01223979370120031,
+            0.005256127464980655,
+            -0.008756077893734508,
+            -0.0028438320920811362,
+            0.021022708898898203,
+            0.027557382862284808,
+            -0.008342206702778956,
+            0.014526066580760486,
+            -0.011883797998224506,
+            -0.01088335352319397,
+            -0.03479444009382274,
+            0.030104838505937447,
+            -0.013305732257226255,
+            0.026773747582411633,
+            -0.0006968265979630097,
+            -0.017037729111584094,
+            -0.016206207453228583,
+            0.01625225876515654,
+            -0.0024290584023212254,
+            -0.009366287391027166,
+            0.01465885301870365,
+            -0.02428628495473643,
+            0.0058336571592899735,
+            -0.0006627935593670669,
+            -0.011531762838191602,
+            0.013275508677482622,
+            0.0019223252930710324,
+            0.010738622785461513,
+            -0.02030405805119001,
+            0.0009661595529028194,
+            0.009263716373456188,
+            -0.016791855691574455,
+            0.006057830565306032,
+            -0.03730952786847912,
+            0.0029341721588188943,
+            -0.014833895684348656,
+            0.005725615593914644,
+            0.001498755253963699,
+            -0.01820708352937343,
+            -0.03652692232407375,
+            0.00043665701245599497,
+            0.01913883585802903,
+            -0.007975988799790905,
+            0.020538595326175857,
+            0.008243569241768276,
+            0.008421404577931517,
+            0.002314738067443065,
+            0.0016702403734608475,
+            0.008279538988821465,
+            0.03679456861213399,
+            -0.01775927634148114,
+            -0.004836606118958108,
+            -0.0012151329859941935,
+            0.010644195708319258,
+            -0.02028470571116305,
+            -0.009287515726906219,
+            -0.0036506912311261536,
+            -0.013134467570201307,
+            0.030615862647585222,
+            -0.020265555441523482,
+            0.025634449009756734,
+            -0.009300770899023007,
+            0.009274072905190077,
+            -0.02097685009125915,
+            -0.009797008690754956,
+            -0.010661540100131183,
+            0.0019532291898233895,
+            0.013465005282849381,
+            -0.009151430285942593,
+            0.0071467934925125145,
+            -0.006655608635025676,
+            0.01020327561163374,
+            -0.001472597695707201,
+            -0.02443508800166349,
+            -0.0024998811551575563,
+            -0.01107321258077183,
+            0.02456164421154801,
+            -0.0019619696941303388,
+            0.0005043179699158694,
+            0.010951580163871948,
+            -0.00853593956432799,
+            0.023927336608555987,
+            -0.008855702280766362,
+            0.0010093679746665581,
+            -0.004863269000020324,
+            -0.004005691415117686,
+            0.001698212044518236,
+            0.01822024022650613,
+            -0.005707404715849356,
+            0.03722587302228139,
+            -0.01708925706528118,
+            -0.0065500772743641925,
+            -0.00015538686795361824,
+            0.010852925330001283,
+            0.010035359503147549,
+            0.008754719437760327,
+            0.004067820167375348,
+            -0.014861417626433965,
+            0.009346669946626484,
+            -0.020136029447262147,
+            0.007501669351583119,
+            0.023178490169347605,
+            -0.006840359541885465,
+            0.0006676450965357345,
+            -0.01283320739092204,
+            -0.015610538827742276,
+            0.006308543164022682,
+            0.002329420005586303,
+            -0.002295739637504597,
+            -0.02181189272979859,
+            0.006298804075737111,
+            -0.014398470126723037,
+            0.004007419942728446,
+            -0.0007497656993578918,
+            0.013863867106289847,
+            0.016451166273826513,
+            -0.006740209707090658,
+            0.039292260347522444,
+            0.01871723939316731,
+            0.009109005310505462,
+            0.005674782595856756,
+            -0.040588444759736254,
+            -0.008993850998154852,
+            -0.0010197185016393919,
+            0.004866150984083949,
+            -0.01141354216871663,
+            -0.018944881712855702,
+            -0.002646110151132781,
+            0.000308585036997592,
+            -0.010681954707313127,
+            -0.009091585604191968,
+            0.01751599002710486,
+            -0.006034424764302979,
+            0.00872057856926618,
+            0.004146961781109887,
+            0.023758065635807138,
+            -0.009727802672285544,
+            0.01176257119851645,
+            -0.0071822083765114165,
+            -0.005456197751568526,
+            0.0006726765703478756,
+            0.011011628907288813,
+            -0.008031792262886802,
+            0.009763340867550022,
+            -0.004279891138185881,
+            0.008355720233269893,
+            -0.008552596384067825,
+            -0.023952129458400563,
+            -0.014972827087541732,
+            0.014347535404251018,
+            0.011651260496709961,
+            0.008137738462704715,
+            -0.013624435569539285,
+            0.0031728721674035352,
+            0.00675830277744976,
+            -0.009184784787581942,
+            0.009161771311910335,
+            0.017733082848622708,
+            -0.001411542568472906,
+            -0.0017830167499636113,
+            -0.0016130003349297027,
+            -0.012173528776899308,
+            0.0011196437753239162,
+            0.02284887358189995,
+            -0.003054479643937998,
+            0.014614283913976716,
+            0.010689486748036806,
+            0.009643292262102645,
+            -0.00810930837603355,
+            0.001915955511174678,
+            0.02341863048264366,
+            0.015022291783105098,
+            -0.004625715884342598,
+            -0.011422973894623295,
+            0.004259757717597968,
+            -0.010332744704689279,
+            0.007373328473086509,
+            -0.02117940113144938,
+            -0.004672698999328004,
+            0.008896311380763048,
+            -0.007028862917603513,
+            -0.01732432285099198,
+            -0.028026173555055017,
+            -0.016765805892242715,
+            0.019819329380181,
+            0.0027213760678977965,
+            -0.013291251876048186,
+            -0.0032919084721662363,
+            0.0031214883337300926,
+            0.002553005168048115,
+            -0.0071104806975682885,
+            0.005310583052470149,
+            0.007359183453835734,
+            -0.008674133980486269,
+            0.017665958608281126,
+            -0.000521269431423141,
+            -0.0008020251454616109,
+            -0.0005442374261144659,
+            0.0006127687598711624,
+            -0.016387752806126442,
+            0.01666161912459431,
+            -0.010949228376419892,
+            0.015198110119816624,
+            -0.008966726991404666,
+            0.008600499511690635,
+            -0.007617043351740469,
+            0.009458641139070436,
+            0.011597957400445055,
+            0.012755037944648426,
+            -0.004403680820063604,
+            0.008519241473418196,
+            0.004807816191718704,
+            -0.008147125240985166,
+            0.007017601015398136,
+            0.01296082929962557,
+            0.00582353932160707,
+            0.0013634237970372698,
+            -0.0008246894322793264,
+            0.0022035670958705576,
+            -0.002480976219285683,
+            0.0056293125704658135,
+            0.012714850523393719,
+            -0.002497639449055006,
+            -0.007135683201188999,
+            0.014988249771676699,
+            0.014955895630183352,
+            -0.022994948923833993,
+            0.007121634434082677,
+            0.005835971825639792,
+            -0.0030507120821557008,
+            0.002145266953387315,
+            -0.027334525318320587,
+            -0.020359474070022213,
+            0.01901995970350808,
+            -0.015260454474605018,
+            0.02850460517419317,
+            -0.011624138792780823,
+            -0.02284242513728605,
+            -0.01277489645734181,
+            -0.004209529633534193,
+            -0.022684500849916897,
+            -0.004894414098943742,
+            0.01919165657366256,
+            0.013401560972810082,
+            -0.004313549020265677,
+            -0.019089369171102138,
+            0.03863248408555474,
+            0.010497951840980463,
+            0.03508157919687289,
+            -0.008597970944278045,
+            0.0023354831464448313,
+            -0.01563401808449416,
+            0.015551176593436512,
+            0.001075882802133782,
+            -0.011575692834796734,
+            0.01505716108583101,
+            0.031836226915668404,
+            0.0034740889237207494,
+            -0.00617206855371193,
+            0.0018420207148235309,
+            0.00418433325800744,
+            -0.0008167766650900381,
+            0.011160989178055499,
+            -0.018497472318034093,
+            -0.011951262348399922,
+            0.0013093376768780122,
+            0.00216347903176653,
+            0.0059233460419514045,
+            -0.001778091863744448,
+            -0.013317639704511549,
+            0.011433308892643204,
+            0.011596810919469926,
+            -0.01518143105525228,
+            -0.007497373860338335,
+            0.0050957665991362405,
+            -0.018974405317819656,
+            0.01872472902494418,
+            0.011859522398789263,
+            0.000792413325332956,
+            -0.0006503947129906383,
+            0.00808635755624928,
+            0.003749813548995223,
+            0.0033657799335872368,
+            -0.007862215303617144,
+            -0.0028014669075116876,
+            0.0009892310856576799,
+            3.834379772674841e-06,
+            0.015402211364447726,
+            -0.015264893397161102,
+            0.02488855366696587,
+            -0.0025643639525691083,
+            0.00953993868921819,
+            -0.017460590918773188,
+            -0.0017911712931080155,
+            -0.00402374787529868,
+            0.01838051670627292,
+            0.006753072393532114,
+            -0.010130676216186492,
+            0.004921510943896829,
+            0.007975201081296845,
+            -0.012589786869771252,
+            0.00018319054705558588,
+            -0.0022606180909706688,
+            -0.014791528072476771,
+            -0.0031171076370692706,
+            0.017873825561816544,
+            0.007593688317551089,
+            0.0037375070591255375,
+            -0.015863143964928172,
+            0.007192324815779268,
+            0.007522990667369163,
+            0.024504875417877527,
+            -0.026309816223960274,
+            0.024779042752840836,
+            -0.022104087334088543,
+            0.02165781034613893,
+            0.0319051741666687,
+            0.007538146369110045,
+            -0.01690181186708103,
+            -0.006595947249169521,
+            0.006373991305923002,
+            -0.023410814842488123,
+            -0.005494017994983188,
+            0.016903736810723716,
+            -0.01792400152372258,
+            0.000330865061689199,
+            -0.014660413003063089,
+            0.0040825880154611895,
+            0.008248037865838212,
+            0.011540333451799686,
+            -0.0033578830043738085,
+            0.018327525495676525,
+            0.020171745352159914,
+            -0.01138759854371235,
+            -0.0004221149053107702,
+            -0.022321409200941777,
+            0.005046865873981027,
+            0.022509643273449936,
+            0.001111272240506322,
+            -0.009540467231190513,
+            0.018376261423745193,
+            0.008480532791922674,
+            -0.020801891594934313,
+            0.008436374227753589,
+            -0.036175982655998785,
+            -0.010454118523313122,
+            -0.009328507118928412,
+            -0.001218633166683384,
+            -0.004142773602181115,
+            -0.0002494371580772723,
+            -0.011365548842513265,
+            -7.699548642315061e-05,
+            -0.006889339420877534,
+            -0.0037173070609413474,
+            0.014953978004522174,
+            0.011678957032329096,
+            -0.003501047405385249,
+            -0.0073984932581048915,
+            0.008493128567102267,
+            0.006163937693564917,
+            -0.017034696006318337,
+            -0.013268418502042576,
+            0.0037122128689059085,
+            -0.020067656401141467,
+            0.005148329575092478,
+            0.00459729178556505,
+            0.014031669598917171,
+            -0.01849235806598133,
+            -0.009298127665365717,
+            0.019716512208448042,
+            0.0094466405158553,
+            -0.0008988916054855498,
+            0.004264260983102956,
+            -0.00032312548009733906,
+            0.002279987803645912,
+            -0.0008902345374346296,
+            -6.145528806973222e-06,
+            0.003602229465148649,
+            0.012503366799381277,
+            0.007759393855482671,
+            0.03694402922447051,
+            -0.016692148708641812,
+            0.005873765863350649,
+            -0.014187609702966002,
+            -0.0037592517335595654,
+            -0.00749939010396784,
+            -0.0004308157309481927,
+            -0.0022282234382577843,
+            0.0021999996319642854,
+            0.0052176773733353925,
+            0.009353171029508977,
+            0.002155405440765206,
+            -0.00823565584692709,
+            0.009691447250255694,
+            -0.015228173095533029,
+            0.003853858061067531,
+            -0.008207910291425473,
+            0.01017972550354623,
+            -0.009695280827515048,
+            0.0028705587799331246,
+            0.03849358164875786,
+            0.01130021300937487,
+            -0.004934175853966098,
+            -0.014300147534361459,
+            -0.0007796051618993811,
+            -0.01737877501558196,
+            -0.008579119859817895,
+            -0.005434792488602325,
+            -0.02617338653902575,
+            -0.00838916510137031,
+            0.015344260464413648,
+            0.011202567142310492,
+            0.014124853863170132,
+            -0.023894182185855824,
+            0.008897943489322673,
+            -0.03001670823127986,
+            -0.011951507895140876,
+            -0.001530967140156163,
+            -0.015467812535428609,
+            0.0010110999991167703,
+            0.023623196504289838,
+            -0.007762735863648054,
+            0.012654652822750325,
+            0.012867958170258448,
+            0.018777522196571934,
+            0.006390986057157016,
+            0.00038045736974183937,
+            -0.009444911783760569,
+            0.009534310165417939,
+            0.008806031651319704,
+            0.0080954543713255,
+            0.013161833638467553,
+            -0.0022840529693446553,
+            -0.018060162439023572,
+            0.005469288590000856,
+            0.014354475778235131,
+            -0.0033504276914600585,
+            -0.001615191819057537,
+            -0.011088493180661976,
+            0.016039040565630432,
+            -0.023559046333872156,
+            -0.009482398794557168,
+            0.009918576960319548,
+            -0.0027308309155029842,
+            0.009212063547988036,
+            0.0160181200081126,
+            -0.01881373001297218,
+            -0.0024168618343039703,
+            0.016244889977796682,
+            0.0024104546308670025,
+            -0.00022606425578303413,
+            0.016792244238218245,
+            -0.0007616857360728623,
+            -0.004279120886828465,
+            0.00997764266310123,
+            -0.0075631917138318655,
+            7.3878137053128405e-06,
+            -0.002416364915008295,
+            -0.0028253498168087964,
+            0.02054128727706496,
+            -0.021281801678871736,
+            -0.005379259148621041,
+            0.02257306103533458,
+            0.015687105622685975,
+            -0.007843745111084413,
+            6.258498042855072e-05,
+            -0.0219506172079994,
+            0.0019925886885244454,
+            -0.007768133999661569,
+            0.016524850592507592,
+            -0.007957949830060354,
+            0.0030085044673901665,
+            -0.01361024620290394,
+            0.011563781350927133,
+            -0.01277407169098418,
+            0.00873716215529134,
+            -0.002986943948333218,
+            -0.017567646684567492,
+            -0.0053109390854215125,
+            0.004561236136619266,
+            0.0014045175130801308,
+            0.044958041347040185,
+            -0.005539136232703934,
+            0.003729517522976541,
+            0.018086335927383804,
+            -0.014199453349727933,
+            0.02079719360289355,
+            0.016663293302592568,
+            0.00991445855219406,
+            -0.00033967370741635794,
+            -0.0018726748849858836,
+            0.023373354806639515,
+            0.013258226441642464,
+            0.00861986998150065,
+            -0.0014340218811281975,
+            -0.008516541227326329,
+            0.018239230461345617,
+            0.005458629943667832,
+            -0.012421896028975013,
+            0.012732721349758832,
+            0.013559877132508506,
+            -0.013720939472355347,
+            -0.011545210834395917,
+            0.01616358223872008,
+            -0.019904594319155373,
+            -0.018260016571763644,
+            -0.0036124537591368864,
+            -0.01614510591132948,
+            -0.03864699384205461,
+            0.01619401261888991,
+            0.018351607973242735,
+            0.006666313000123845,
+            0.013647410298896132,
+            0.002597680061427196,
+            0.004545530267244157,
+            -0.018531082430861854,
+            -0.0077618625909995275,
+            -0.008085144524135461,
+            0.003744735923004248,
+            0.011059457364156674,
+            -0.011335432904114519,
+            -0.009514379703581616,
+            -0.00034500702695418285,
+            -0.007433132060989644,
+            0.01971420270994951,
+            0.0020753610290475642,
+            0.0034506706065161024,
+            -0.00628437298309269,
+            0.017966485854173167,
+            0.026789678964152908,
+            -0.007532446825077476,
+            0.021886182869092073,
+            -0.002864230253056434,
+            -0.0015870360120997492,
+            -0.014393519359298496,
+            -0.0014439412842964876,
+            -0.011527182213472351,
+            0.0005453493437517964,
+            -0.0006791481533636673,
+            0.014093767997869645,
+            -0.03288824642147967,
+            -0.003957301296796813,
+            0.014444592778851293,
+            0.007538052312876599,
+            -0.005057755472988024,
+            0.011357919882949903,
+            -0.002964451028327082,
+            0.016977317604413924,
+            0.0075201055190542146,
+            0.023391646079994784,
+            -0.016588823248397513,
+            -0.004504858318770565,
+            0.006660877064954993,
+            0.0034631255931793638,
+            0.0014484203909961972,
+            0.01912983041187014,
+            0.023439941258581887,
+            -0.014715886385460541,
+            -0.0013596123366001458,
+            -0.0029020372473879614,
+            -0.021414221343490594,
+            0.014976806123253727,
+            0.0074729481304182865,
+            0.010967351226114886,
+            0.009806663105020302,
+            0.015465076881663691,
+            -0.02109587527354609,
+            0.0011671121739822202,
+            0.020567458365051382,
+            -0.008618333549105003,
+            -0.018815706906171112,
+            -0.009804829158883123,
+            -0.00966828714719191,
+            -0.006078893678150302,
+            -0.00803783313451146,
+            0.005523023167028105,
+            0.0022647506733877573,
+            0.01652997604669998,
+            -0.008204200434441975,
+            -0.008404609147866811,
+            0.008489247125986078,
+            0.007546785673285454,
+            -0.0033093451283570316,
+            0.010614050963656496,
+            0.0030633179859393587,
+            -0.006940540318011839,
+            0.00894884343338777,
+            -0.002787816848430143,
+            0.014347598920245487,
+            -0.02441769157218451,
+            0.004717742147185283,
+            -0.012096876825337124,
+            0.006405151160675699,
+            0.0041568790527989905,
+            0.0016702066682613428,
+            -0.0041159211459004,
+            0.0035217362856754353,
+            0.0008893479210743541,
+            -0.07433312151926533,
+            -0.012629681414004524,
+            -0.0035797012285864486,
+            0.033657838713146884,
+            -0.019695806809053914,
+            0.004477927056336216,
+            0.011966065644382396,
+            0.0096717896523898,
+            0.003070713232538593,
+            0.006136726101157844,
+            0.006112385569942848,
+            -0.0245258242481677,
+            -0.00015284418977092615,
+            0.004821040694679944,
+            0.0020347004045943454,
+            -0.007532408630122127,
+            0.00750565079228405,
+            0.015327710245879296,
+            0.020989926895388523,
+            -0.010970305087289488,
+            0.017227663358885748,
+            -0.003352718496981928,
+            -0.006068892126250978,
+            0.0014285750322333686,
+            -0.010153799722483554,
+            -0.017307415843726394,
+            -0.006495008086641125,
+            0.0068983699213297075,
+            -0.001982568363667046,
+            0.0044032687939897855,
+            0.003333596495330213,
+            -0.0032171586993948696,
+            0.0072365127663633925,
+            0.013920326463272646,
+            0.006415916101287133,
+            -0.013675562796300214,
+            0.003003782758807665,
+            -0.02500375035312058,
+            0.011945988879567646,
+            -0.012708975181025296,
+            -0.0009488161508767279,
+            -0.0047093877027523014,
+            -0.0035750724825159503,
+            -0.002213435677949806,
+            0.02043567538125188,
+            -0.014304009033474186,
+            -0.01802899775650995,
+            0.008187227768196673,
+            -0.011217013195839762,
+            0.008300740149622635,
+            -0.03382831122769974,
+            0.012386335491333264,
+            -0.0276029934382802,
+            -0.0002243752790856292,
+            -0.004521387639042107,
+            -0.0010631721066129095,
+            -0.009491184157278669,
+            -0.03274559774619223,
+            0.015845705808609598,
+            -0.007204751554079551,
+            -0.020338337341085228,
+            0.0007143424225187802,
+            -0.021267094083674304,
+            0.014150458196361147,
+            -0.01851165553224022,
+            0.0033450353123933058,
+            -0.018775439467431685,
+            0.002782289821140401,
+            -0.0018711350216089556,
+            0.010502195116579441,
+            -0.00557814442809019,
+            -0.0007130962242755068,
+            -0.013072988887382269,
+            -0.002121841212381393,
+            -0.006653444103331213,
+            0.009394907690516121,
+            0.0028678074167728367,
+            -0.009601577734902932,
+            0.011153571864462314,
+            0.013046116892519414,
+            0.019812860521973938,
+            -0.00539047787915588,
+            -0.00555139958411929,
+            -0.011553689090179406,
+            0.009852063270798534,
+            -0.016326212706634428,
+            -0.00827402913960903,
+            0.008982303148984887,
+            0.015216947130173639,
+            0.01026889105233137,
+            0.003352028462252994,
+            0.003150801750574548,
+            0.002531620731961765,
+            0.026744691368233237,
+            -0.0015120532859309638,
+            0.0008324701960729416,
+            -0.014508988496009931,
+            0.022127525293249926,
+            0.02772430884375744,
+            -0.0012206102581726523,
+            -0.00018859639594030037,
+            0.017437360169132967,
+            -0.004697052522250805,
+            -0.0038111184371524645,
+            0.024015246959315593,
+            -0.038396108183338655,
+            -0.008416825232533771,
+            -0.006788816882617399,
+            -0.006043428198944214,
+            -0.009587366247924467,
+            0.007294190273465812,
+            0.00041966080769004275,
+            0.0024709692753547823,
+            -0.0005250904505954556,
+            0.03452858797518386,
+            0.003403515654095309,
+            -0.0003392060101496309,
+            -0.0032980864446307116,
+            0.0019125121221204029,
+            0.0032987517853214346,
+            -0.009116761755779728,
+            -0.038030085122772624,
+            0.006006334000396066,
+            0.007932785488877407,
+            0.00975009749364395,
+            -0.008120121513849434,
+            -0.008864461019882246,
+            0.004909348738798099,
+            -0.02273129283865203,
+            -0.01372636307167243,
+            0.02625229144362647,
+            -0.008523028986127237,
+            0.029005877324974557,
+            -0.022435059061182174,
+            -0.013135836801818153,
+            -0.004126433322528265,
+            0.010283975013795537,
+            -0.019732856462444506,
+            0.004343860338939202,
+            -0.004022387065239562,
+            0.023733721026005954,
+            0.0077578144626961245,
+            -0.008371631446146718,
+            0.012386735614153616,
+            0.0064789380436815714,
+            -0.0026638642827013837,
+            0.00136333052963848,
+            -0.007209517067748103,
+            0.016916505654836485,
+            -0.011819784026452944,
+            -0.005833909403097991,
+            0.008977527508585844,
+            -0.0024556434204852946,
+            0.02509293948918158,
+            -0.0237762245071987,
+            0.005288171058938098,
+            -0.00019047961064753207,
+            0.01822542471604037,
+            0.008127347078975925,
+            0.008361588667386882,
+            -0.011835407590060994,
+            0.010004482616697591,
+            -0.03378107067882622,
+            -0.015433906677849963,
+            -0.007702910250795191,
+            -0.0013877702264578865,
+            -0.002059152251387601,
+            -0.015745218548968338,
+            -0.026393221735795527,
+            0.006294583740809379,
+            0.016267525575472445,
+            0.023291528020980265,
+            -0.004683197930892445,
+            0.012912830736650997,
+            -0.002831583991769044,
+            -0.008904271394710672,
+            -0.011441882849296418,
+            -0.012202629605208118,
+            -0.012470773094383122,
+            -0.014560694782340029,
+            -0.017045701603711442,
+            0.012139665577738337,
+            -0.020079061996873537,
+            0.013911518328924777,
+            0.0347210775522539,
+            -0.017176114991595492,
+            -0.0018941772168839354,
+            0.019702722555622765,
+            -0.004221469612316137,
+            0.01212088353057276,
+            0.0063707489618353725,
+            0.015298014532351866,
+            0.0038643174573634725,
+            -0.012181479717826009,
+            -0.007374005740809251,
+            0.019411457664902376,
+            -0.004902591844710158,
+            0.014591446725717338,
+            0.015069599884557399,
+            -0.008675672164385891,
+            0.002544198867387345,
+            0.01951865904496276,
+            -0.01762101777798757,
+            0.01665654707715257,
+            -0.00458402229480694,
+            -0.009936206685538294,
+            -0.013345740581684638,
+            -0.005336416724426228,
+            -0.019469457091169776,
+            0.0061608454109892425,
+            0.018931676738452464,
+            -0.00533013594306834,
+            0.009310871143727683,
+            -0.0037670793158045,
+            -0.005045151384034014,
+            0.0002041909886640482,
+            -0.04152751596227068,
+            -0.008340911336217609,
+            -0.0006651584370005978,
+            -0.02505748815737022,
+            0.025954102004571416,
+            0.007205602377144382,
+            0.022602344303400163,
+            -0.017168610245491406,
+            0.004459735690564842,
+            -0.005882024209102714,
+            0.013217517324383834,
+            0.02255689669623969,
+            0.0017984057469785134,
+            0.005699128497215882,
+            -0.02353849721739205,
+            -0.0034711576718386185,
+            0.015924913374553738,
+            -0.003604478141616307,
+            -0.0034993903155063817,
+            0.012592308877056896,
+            0.012864938348003884,
+            0.01567418642605425,
+            0.008801753550506258,
+            0.015606443428689292,
+            0.010589933994371628,
+            0.008928721556432368,
+            -0.005805129999077126,
+            -0.001025103064751441,
+            -0.015907755521365925,
+            0.0045874330999415985,
+            0.031701049450012876,
+            0.012594483199773746,
+            0.016783858873835704,
+            -0.01778536268185936,
+            0.00789523134933153,
+            -0.0012706411671644011,
+            0.008325108613530198,
+            -0.001677173669891956,
+            0.029540136288412976,
+            0.011739999605699826,
+            0.015670966746894652,
+            0.008732334090105053,
+            -0.016237731209426752,
+            0.045334690225148,
+            -0.016911928942220152,
+            -0.0021139311364009113,
+            -0.01058970056572469,
+            -0.021705338531028618,
+            0.007324792557651635,
+            -0.006556582104686972,
+            0.0018243188317438144,
+            0.010780292738415671,
+            -0.006471850255151019,
+            0.0006997395548596146,
+            0.003907433192252573,
+            -0.023129313269763224,
+            -0.002550875642832235,
+            -0.02237917712764273,
+            -0.003107775639788617,
+            -0.02621608419628401,
+            -0.00012482032027919036,
+            -0.016207377503696517,
+            -0.0038712739995960753,
+            -0.023315508569288476,
+            -0.014963211090562272,
+            -0.01633110043829169,
+            -0.004241081909202262,
+            -0.008295402942465353,
+            0.004976396647108597,
+            -0.007402107755970892,
+            -0.020321297548927468,
+            0.0020986169430507757,
+            0.016803273183070004,
+            0.01434729320047051,
+            0.018612970083178984,
+            0.00603868809307069,
+            0.014278001198851581,
+            -0.0026350854954985117,
+            0.017171016209107656,
+            -0.006968968261909459,
+            0.006881847166565678,
+            0.011454001418121107,
+            0.02140935294963631,
+            0.011121833270235448,
+            -0.012801756637470525,
+            0.0010870041670931706,
+            -0.009655388034828094,
+            0.002094601818569396,
+            -0.00828486034340067,
+            -0.009190975158350585,
+            0.0017762581946855365,
+            -0.007261996868906865,
+            0.004674405202069334,
+            0.0030526253833799426,
+            -0.0014082232541965316,
+            0.010419302200840906,
+            -0.0024200053377751805,
+            -0.012208685875688886,
+            -0.0033037330470612695,
+            -0.017975940046938498,
+            -0.006958042903103732,
+            -0.0014496522238488239,
+            -0.011711502944602638,
+            -0.01016130418405875,
+            0.003321436189059608,
+            -0.00538282506552848,
+            -0.009353581612407435,
+            0.013884147453136417,
+            0.0001309738696853529,
+            0.0005878330075918333,
+            0.023495280602974504,
+            0.006035028203642483,
+            0.008087594627921345,
+            0.013144288945385904,
+            -0.01023995000994454,
+            -0.022574958230635715,
+            -0.0243435727055662,
+            0.007867022215096418,
+            -0.01573478573423379,
+            0.02324699908478915,
+            -0.007929809922043611,
+            -0.00561630876431978,
+            0.0020982723335057625,
+            -0.013699317926847062,
+            0.008578431496569186,
+            -0.011271381061801356,
+            0.0014450645788554738,
+            -0.019581451799757368,
+            0.018059311800267413,
+            -0.010917911869373693,
+            -0.004302687883018328,
+            0.009466103226841532,
+            0.034049086327052606,
+            -0.0005689605174080737,
+            -0.007410970644432269,
+            -0.006594150877740108,
+            -0.00022810271313786974,
+            0.018766765773409123,
+            -0.0029041815756733333,
+            0.005758781908059541,
+            -0.01563519107805783,
+            0.006571628287171105,
+            -0.006132372252825422,
+            0.0012423261344297225,
+            0.01513074831647933,
+            -0.026306320707079913,
+            -0.015491710256868008,
+            -0.007857352164317122,
+            -0.024893430640551818,
+            0.002253574511878983,
+            0.017579477884206534,
+            0.020505606775521504,
+            -0.020324505029926956,
+            -0.02513332780748072,
+            0.005150966733930881,
+            -0.026580994694344065,
+            0.0037390577082486594,
+            0.1241031878517593,
+            -0.002138404898309707,
+            0.002720723037908521,
+            -0.011425036886011179,
+            0.0022399461009316216,
+            0.005389407121650262,
+            -0.019552543955456567,
+            0.05950746473186985,
+            -0.00638023587296763,
+            -0.004023798021748334,
+            0.002170545188776296,
+            0.001785855390676113,
+            -0.005831534307818096,
+            0.008643501093778434,
+            -0.017602787281866816,
+            -0.02184792441891879,
+            -0.0052215705748838905,
+            -0.021252169339074325,
+            -0.008509621422071373,
+            -0.013000137524907565,
+            0.002893880203681282,
+            -0.015821609145923173,
+            0.010397692426614709,
+            -0.01638916243796692,
+            -0.010283706937701179,
+            -0.01103621454761223,
+            -0.00373552043529251,
+            0.01563553822905027,
+            -0.03188533919653949,
+            -0.004873550301425776,
+            -0.005183590054152726,
+            0.018218971950313273,
+            0.006004328765748325,
+            0.004357734370247252,
+            -0.014230318971458851,
+            0.0064792070801456235,
+            0.0215589173714926,
+            0.0055272493764401305,
+            0.014643335833970546,
+            0.010618070641357003,
+            -0.0022177303813575846,
+            -0.0024402761885758007,
+            0.017541799738501007,
+            -0.005445774414228888,
+            -0.00023502936119626386,
+            -0.00537569046685066,
+            -0.005974703247446444,
+            -0.010712244203425036,
+            0.01843613166635215,
+            -0.009938564643456672,
+            -0.017388592763277214,
+            -0.0015797412875366156,
+            0.02028434083164951,
+            -0.012761534382003973,
+            0.0031906812114950982,
+            0.000788436210275669,
+            0.020589196433948156,
+            0.0007726855314648301,
+            -0.005481260996289263,
+            0.007165014826294827,
+            -0.02956334397512344,
+            0.004095715232997753,
+            0.0003568512047626187,
+            0.027290768799658912,
+            -0.0017631129386615335,
+            0.01552542586826711,
+            -0.020871286597113327,
+            0.004388125824324094,
+            -0.016926008631435403,
+            0.005099899758322398,
+            0.012560923968841436,
+            0.002781049890810361,
+            0.010889021753593997,
+            -0.008500358897706648,
+            -0.021353460542554444,
+            0.012663934009161902,
+            -0.018657094130860155,
+            -0.029156017587920958,
+            0.0018889388761887053,
+            -0.028158076315864772,
+            -0.003048854609750186,
+            0.006812072971079798,
+            -0.035217412591558014,
+            -0.02251675180883659,
+            0.007396524512996277,
+            -0.00995571600201846,
+            -0.011912753666073882,
+            0.010675240017836214,
+            -0.02645155743041798,
+            -0.018137648527851668,
+            0.0010666944556449141,
+            0.012588800077165337,
+            -0.004561825939541096,
+            0.007555558090958502,
+            -0.028558603070876605,
+            0.014816816621445352,
+            -0.01313764261744744,
+            0.016036065063831813,
+            -0.0037797584923144134,
+            0.022217516257304868,
+            0.011131597210722573,
+            0.008795376259031849,
+            -0.004950080558710245,
+            0.021409590863550987,
+            0.005574351961953293,
+            0.051358937509801736,
+            0.031972009383746405,
+            0.0019821003238211276,
+            -0.003251257665378234,
+            0.003313045083311649,
+            0.008294776423730887,
+            -0.00940948257229233,
+            0.005114631360856872,
+            0.028053227377164806,
+            0.016738013668710083,
+            -0.018755924318109216,
+            0.006606711923035118,
+            -0.02702692774986498,
+            0.017950884766781056,
+            -0.022054653845163,
+            0.0003755944703013671,
+            0.007766253634286795,
+            0.0016206993755471613,
+            0.010682347575070117,
+            0.0013842930079185415,
+            0.01705234255261425,
+            0.023975191051210525,
+            -0.002976402980885077,
+            0.017466027107477605,
+            0.012734909275073793,
+            0.018143161773162015,
+            -0.0038399373197265827,
+            0.006929440921265761,
+            -0.012003095293274786,
+            -0.0008883470597690258,
+            0.0020490527756307267,
+            -0.004231091383919565,
+            0.012128848448049833,
+            0.008340302519160299,
+            0.012989849936937843,
+            0.002974488532907779,
+            0.003198854381461613,
+            -0.005309753760578859,
+            -0.0017024982049936384,
+            -0.01763395035509857,
+            -0.01819680226734474,
+            0.028417724274372468,
+            -0.020217879775485576,
+            -0.050118184195774004,
+            0.0025509548991664425,
+            0.014185761006969567,
+            -0.0012021669304388173,
+            -0.0051260576846881515,
+            -0.0005774778872927327,
+            0.005718864494530703,
+            -0.016918749210038116,
+            -0.0015472638835434902,
+            0.008125845907627804,
+            0.020347547062899797,
+            -0.002036218493206914,
+            -0.01079739518866248,
+            0.0038112958713600906,
+            0.02169339303105242,
+            0.01667594322285835,
+            0.012338193092567552,
+            -0.009798409233733167,
+            0.0003198215558702285,
+            -1.6308740825163337e-05,
+            0.0076870827874737325,
+            0.004356644356656038,
+            0.003500023005601613,
+            -0.019968549869439228,
+            -0.008634740687755112,
+            0.012781565963545847,
+            -0.0025637758121512927,
+            0.010243567793632335,
+            -0.00042296044205132426,
+            -0.003069251553118065,
+            -0.009344199449486244,
+            -0.004694953215957636,
+            -0.009583993744231791,
+            0.013783878036521561,
+            -0.007705982042252044,
+            -0.01549567538665377,
+            0.00020666298495823114,
+            0.0005515775451348472,
+            0.001667367278489879,
+            -0.004518431526236256,
+            0.03302324454129141,
+            0.009531990824506103,
+            -0.0004937348108340977,
+            -0.004047613286672229,
+            -0.0327383112522095,
+            0.012456150771224677,
+            -0.009677533214778067,
+            0.0006455811519981003,
+            0.020012976827426605,
+            -0.00020277620481825933,
+            0.002168432791818704,
+            -0.015657551629104767,
+            0.0032924357559238502,
+            0.006161295735841883,
+            0.013814165416788555,
+            0.0005164426482972658,
+            0.01076512534105537,
+            -0.006498977875968572,
+            -0.010325144017898257,
+            0.004804158067334203,
+            0.0104064855510473,
+            -0.0012188100892986933,
+            0.005171197790296134
+          ],
+          "title": "Mixture-Of-Experts With Expert Choice Routing",
+          "keyphrases": [
+            "Mixture-Of-Experts",
+            "Expert Choice Routing",
+            "sparsely-activated models",
+            "training efficiency",
+            "downstream performance"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "86d0cb2e-1f6d-4e38-a077-7b7de2fe9e4a",
+        "properties": {
+          "page_content": "3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n",
+          "embedding": [
+            0.010104401037096977,
+            -0.020574966445565224,
+            -0.02167346142232418,
+            -0.031542472541332245,
+            0.03588413819670677,
+            0.03464615345001221,
+            -0.009755672886967659,
+            0.046869076788425446,
+            0.004803731571882963,
+            0.0364421010017395,
+            0.016547156497836113,
+            -0.007223033811897039,
+            0.009546436369419098,
+            -0.0017371026333421469,
+            0.005762734450399876,
+            0.04048734903335571,
+            0.011603932827711105,
+            0.01784616895020008,
+            0.011150586418807507,
+            -0.02915368229150772,
+            0.005383492447435856,
+            -0.01708768494427204,
+            -0.04317255690693855,
+            -0.006194285582751036,
+            0.009546436369419098,
+            0.005871712230145931,
+            0.027340294793248177,
+            0.0036812627222388983,
+            -0.055901139974594116,
+            0.023800702765583992,
+            0.021708333864808083,
+            0.010984940454363823,
+            -0.01007824670523405,
+            -0.010601338930428028,
+            -0.026573093608021736,
+            -0.021969880908727646,
+            0.04634598642587662,
+            0.007685098797082901,
+            -0.006399163510650396,
+            0.042475100606679916,
+            0.03703493997454643,
+            0.012728581205010414,
+            -0.012772172689437866,
+            -0.014175803400576115,
+            -0.003430614247918129,
+            -0.0012979229213669896,
+            -0.004712190479040146,
+            0.0065735275857150555,
+            0.009311044588685036,
+            -0.026172054931521416,
+            -0.001842810888774693,
+            -0.021900134161114693,
+            0.02798544242978096,
+            0.020609840750694275,
+            -0.049379922449588776,
+            0.028072625398635864,
+            -0.004890913609415293,
+            0.06461934745311737,
+            0.047461915761232376,
+            0.014602995477616787,
+            0.03794163465499878,
+            -0.034035876393318176,
+            -0.029275737702846527,
+            -0.003177786245942116,
+            0.03234454616904259,
+            0.007654585409909487,
+            -0.01033979281783104,
+            0.0010679803090170026,
+            0.011717269197106361,
+            -0.005706066265702248,
+            -0.044532597064971924,
+            0.010496720671653748,
+            -0.014925569295883179,
+            0.002881367225199938,
+            0.004034350160509348,
+            -0.020017001777887344,
+            0.03030448593199253,
+            0.0268520750105381,
+            0.014899414964020252,
+            0.010740830563008785,
+            0.010653648525476456,
+            0.03204812481999397,
+            -0.0026961052790284157,
+            0.009555154480040073,
+            0.0364421010017395,
+            -0.029415227472782135,
+            -0.01360040158033371,
+            -0.062178246676921844,
+            0.010540311224758625,
+            0.0028072623535990715,
+            -0.02815980650484562,
+            0.0018340926617383957,
+            -0.006569168530404568,
+            0.0036986989434808493,
+            -0.033094312995672226,
+            0.0727447122335434,
+            0.010653648525476456,
+            0.00023280335881281644,
+            -0.024044813588261604,
+            0.006394804455339909,
+            -0.015762517228722572,
+            0.021969880908727646,
+            0.0006560450419783592,
+            0.004008195362985134,
+            0.0229811929166317,
+            -0.02859571762382984,
+            -0.004690395202487707,
+            0.03815086930990219,
+            0.02148166112601757,
+            0.08676359057426453,
+            0.01007824670523405,
+            -0.017593340948224068,
+            -0.011211613193154335,
+            0.018482597544789314,
+            -0.04278895631432533,
+            -0.017689241096377373,
+            -0.03965040296316147,
+            -0.03215274587273598,
+            0.0114818774163723,
+            -0.025038689374923706,
+            0.01996469311416149,
+            0.02566639892756939,
+            -0.0011344566009938717,
+            -0.025422289967536926,
+            -0.07155904173851013,
+            0.04292844980955124,
+            -0.023696085438132286,
+            0.01780257746577263,
+            0.001439593848772347,
+            -0.0022144243121147156,
+            -0.00422179140150547,
+            0.0037183149252086878,
+            -0.020261110737919807,
+            -0.041882261633872986,
+            0.01856978051364422,
+            0.010479284450411797,
+            0.046834204345941544,
+            0.0633290484547615,
+            -0.021132932975888252,
+            -0.0072448295541107655,
+            0.017375385388731956,
+            -0.022894009947776794,
+            0.003243172774091363,
+            -0.0038142153061926365,
+            -0.029450101777911186,
+            0.013975284993648529,
+            0.028508534654974937,
+            0.08578714728355408,
+            -0.01956365443766117,
+            0.012022406794130802,
+            -0.033286113291978836,
+            0.037662651389837265,
+            -0.009799264371395111,
+            0.023417102172970772,
+            -0.025404853746294975,
+            -0.021708333864808083,
+            -0.025352545082569122,
+            0.010705957189202309,
+            -0.014995315112173557,
+            0.0059632533229887486,
+            -0.005889148451387882,
+            -0.0062945447862148285,
+            -0.018691834062337875,
+            0.010470565408468246,
+            0.007685098797082901,
+            0.047985006123781204,
+            0.029816266149282455,
+            0.005858634598553181,
+            -0.07769665867090225,
+            0.038325235247612,
+            0.017514877021312714,
+            0.036546722054481506,
+            0.01543994341045618,
+            0.033443041145801544,
+            -0.026538221165537834,
+            0.017680522054433823,
+            0.0032475318294018507,
+            0.01094134896993637,
+            -0.03197837993502617,
+            0.044916197657585144,
+            -0.00843050517141819,
+            0.002720080316066742,
+            0.01972058229148388,
+            0.013312701135873795,
+            0.04062684252858162,
+            0.027915697544813156,
+            0.034210242331027985,
+            0.03815086930990219,
+            -0.01332141924649477,
+            0.010453129187226295,
+            0.029380355030298233,
+            0.03515180945396423,
+            0.02796800620853901,
+            0.0027636713348329067,
+            0.0041890982538461685,
+            -0.026590529829263687,
+            -0.006721737328916788,
+            0.009964910335838795,
+            0.007039951626211405,
+            0.016390228644013405,
+            -0.012780890800058842,
+            -0.028124934062361717,
+            0.0220221895724535,
+            -0.028648026287555695,
+            0.030740395188331604,
+            0.008626664988696575,
+            -0.01083673071116209,
+            0.013208082877099514,
+            0.009895164519548416,
+            -0.029833702370524406,
+            0.004243587143719196,
+            -0.006250953767448664,
+            -0.004468080587685108,
+            -0.025021253153681755,
+            -0.01933698169887066,
+            0.00797280017286539,
+            0.0029532923363149166,
+            0.005021686665713787,
+            0.009380790404975414,
+            -0.022545281797647476,
+            0.004284998401999474,
+            0.02127242274582386,
+            0.022876573726534843,
+            -0.018325669690966606,
+            -0.018901070579886436,
+            -0.011438286863267422,
+            -0.01914518140256405,
+            -0.011865478940308094,
+            0.044393107295036316,
+            -0.012711144983768463,
+            -0.009694645181298256,
+            0.020574966445565224,
+            0.006774046458303928,
+            -0.005043482407927513,
+            0.017872322350740433,
+            -0.029484974220395088,
+            -0.010932630859315395,
+            -0.0035592077765613794,
+            0.011072122491896152,
+            0.010243892669677734,
+            -0.00997362844645977,
+            0.01550968922674656,
+            -0.050879452377557755,
+            -0.010932630859315395,
+            0.007881258614361286,
+            0.015082497149705887,
+            -0.0110285310074687,
+            0.02339966595172882,
+            0.012161897495388985,
+            0.016224581748247147,
+            -0.003081885864958167,
+            -0.014829669147729874,
+            0.008221268653869629,
+            0.017384104430675507,
+            0.007881258614361286,
+            0.05712168663740158,
+            0.02338222973048687,
+            -0.03947603702545166,
+            0.006538654677569866,
+            -0.029973194003105164,
+            0.05579652264714241,
+            0.007994594983756542,
+            -0.029833702370524406,
+            0.053215932101011276,
+            -0.023434538394212723,
+            -0.0024193022400140762,
+            0.060853078961372375,
+            -0.0007007258245721459,
+            -0.013548092916607857,
+            0.0013011923292651772,
+            0.02435866929590702,
+            -0.056075502187013626,
+            -0.04899631813168526,
+            -0.02148166112601757,
+            0.02857828140258789,
+            -0.02453303337097168,
+            0.0077156126499176025,
+            -0.015405070967972279,
+            0.04735729843378067,
+            0.036721084266901016,
+            0.018116433173418045,
+            -0.008731283247470856,
+            0.02008674666285515,
+            0.04749678820371628,
+            -0.007977158762514591,
+            -0.004742704331874847,
+            -0.02833417057991028,
+            0.005152460187673569,
+            -0.033216364681720734,
+            -0.009485408663749695,
+            -0.027462350204586983,
+            -0.011133149266242981,
+            -0.02125498652458191,
+            -0.0030317562632262707,
+            0.022336045280098915,
+            0.0041389684192836285,
+            5.1253518904559314e-05,
+            -0.0020389705896377563,
+            0.040522221475839615,
+            -0.024951506406068802,
+            -0.04470696300268173,
+            -0.00825178250670433,
+            0.01704409345984459,
+            0.0038556267973035574,
+            -0.05475033447146416,
+            -0.006250953767448664,
+            -0.0029707287903875113,
+            0.01304243691265583,
+            -0.023539157584309578,
+            -0.00863538309931755,
+            0.016573309898376465,
+            0.002870469354093075,
+            0.009807982482016087,
+            -0.02800287865102291,
+            -0.014629150740802288,
+            0.007406116463243961,
+            0.01841285265982151,
+            0.02817724272608757,
+            -0.0555872842669487,
+            -0.03225736320018768,
+            0.02067958563566208,
+            -0.009040780365467072,
+            -0.06409624963998795,
+            0.007833308540284634,
+            -0.001987751107662916,
+            0.01799437776207924,
+            0.016948193311691284,
+            0.04160328209400177,
+            -0.019581090658903122,
+            -0.022283734753727913,
+            -0.016599465161561966,
+            -0.0364072285592556,
+            -0.028700334951281548,
+            -0.009712082333862782,
+            -0.008299732580780983,
+            -0.028857262805104256,
+            -0.0017839629435911775,
+            0.01342603750526905,
+            -0.03759290650486946,
+            0.006176849361509085,
+            -0.021603714674711227,
+            0.0028486738447099924,
+            -0.03703493997454643,
+            0.007842026650905609,
+            -0.0030339357908815145,
+            -0.01613740064203739,
+            -0.02125498652458191,
+            -0.01561430748552084,
+            0.014725050888955593,
+            -0.0002036518562817946,
+            0.0017142173601314425,
+            0.0229637548327446,
+            0.007484580390155315,
+            0.009328480809926987,
+            -0.008535124361515045,
+            -0.029798829928040504,
+            -0.013382446952164173,
+            0.006586604751646519,
+            0.018534906208515167,
+            0.019999565556645393,
+            0.005501188337802887,
+            -0.04958915710449219,
+            -0.03584926575422287,
+            0.021621152758598328,
+            -0.05890020355582237,
+            0.006394804455339909,
+            -0.0017425514524802566,
+            -0.008622306399047375,
+            0.0033150978852063417,
+            -0.005832480266690254,
+            -0.026294110342860222,
+            0.015570716932415962,
+            0.0086135882884264,
+            0.012275234796106815,
+            0.007488939445465803,
+            0.015980472788214684,
+            0.02852597087621689,
+            -0.02451559714972973,
+            -0.05788889154791832,
+            0.03427998721599579,
+            -0.019075434654951096,
+            0.014210676774382591,
+            0.0440792515873909,
+            0.005475033540278673,
+            -0.006551732309162617,
+            0.025003815069794655,
+            0.013399883173406124,
+            -0.010034655220806599,
+            -0.03407074883580208,
+            0.04376539587974548,
+            -0.003430614247918129,
+            -0.07121030986309052,
+            0.005479392595589161,
+            0.0014810053398832679,
+            0.013103464618325233,
+            0.015283015556633472,
+            -0.003430614247918129,
+            -0.023643774911761284,
+            -0.01956365443766117,
+            -0.014088621363043785,
+            -0.018168741837143898,
+            0.029484974220395088,
+            0.003539591794833541,
+            0.005004250444471836,
+            -0.0006091846735216677,
+            -0.0014341449132189155,
+            0.004677317570894957,
+            0.014027593657374382,
+            0.04080120474100113,
+            -0.0364769771695137,
+            0.0025958458427339792,
+            -0.04275408387184143,
+            -0.03874370828270912,
+            -0.0046380856074392796,
+            0.018360542133450508,
+            0.026485910639166832,
+            -0.025910509750247,
+            0.02320786565542221,
+            0.0100608104839921,
+            -0.015675334259867668,
+            -0.005008609499782324,
+            0.028090061619877815,
+            0.025369981303811073,
+            0.01285063661634922,
+            -0.004533467348664999,
+            0.014690177515149117,
+            -0.012623962946236134,
+            0.0026045641861855984,
+            0.01574508100748062,
+            -0.00010305192699888721,
+            0.03490769863128662,
+            0.001937621389515698,
+            0.018918508663773537,
+            -0.04693882167339325,
+            -0.02699156664311886,
+            -0.027689022943377495,
+            -0.0172620490193367,
+            0.03874370828270912,
+            -0.006503781769424677,
+            -0.012728581205010414,
+            0.020261110737919807,
+            0.004282819107174873,
+            0.0017044093692675233,
+            -0.04066171497106552,
+            -0.028630590066313744,
+            -0.022283734753727913,
+            -0.044916197657585144,
+            -0.005618884228169918,
+            -0.02183038927614689,
+            -0.019057998433709145,
+            -0.05342517048120499,
+            0.0010527233825996518,
+            -0.07811512798070908,
+            -0.005370415281504393,
+            0.04048734903335571,
+            0.004686036147177219,
+            0.009424380958080292,
+            -0.028682898730039597,
+            0.01025261078029871,
+            -0.036337483674287796,
+            -0.02568383514881134,
+            0.017785141244530678,
+            0.04387001320719719,
+            -0.011813169345259666,
+            0.0019997386261820793,
+            0.019057998433709145,
+            -0.05056559666991234,
+            -0.005370415281504393,
+            -0.0031799657735973597,
+            0.007641507778316736,
+            0.012920381501317024,
+            -0.016974348574876785,
+            0.05457597225904465,
+            0.011054685339331627,
+            0.02223142609000206,
+            0.02606743760406971,
+            -0.00020024630066473037,
+            -0.038046251982450485,
+            0.017584621906280518,
+            0.008535124361515045,
+            0.02796800620853901,
+            0.0015681873774155974,
+            0.0073014977388083935,
+            -0.019807765260338783,
+            0.008979752659797668,
+            -0.009485408663749695,
+            0.01646869257092476,
+            0.018866198137402534,
+            0.02645103819668293,
+            -0.008644101209938526,
+            -0.002175192581489682,
+            0.0033434322103857994,
+            -0.0034785643219947815,
+            0.028700334951281548,
+            0.0076371487230062485,
+            -0.038848329335451126,
+            0.006813278421759605,
+            -0.002189359627664089,
+            0.023120682686567307,
+            0.030374230816960335,
+            -0.04108018800616264,
+            -0.02338222973048687,
+            -0.002796364715322852,
+            0.016974348574876785,
+            0.011019812896847725,
+            -0.020662149414420128,
+            -0.0031668883748352528,
+            0.044602345675230026,
+            -0.010479284450411797,
+            -0.032484035938978195,
+            0.01293781865388155,
+            -0.0018504393519833684,
+            0.0059806895442306995,
+            0.0024476363323628902,
+            0.012772172689437866,
+            0.013469628989696503,
+            -0.019598528742790222,
+            0.04087095335125923,
+            -0.004965018481016159,
+            0.013208082877099514,
+            -0.012031124904751778,
+            -0.010052092373371124,
+            -0.015195833519101143,
+            0.03926680237054825,
+            -0.0047906544059515,
+            0.02013905718922615,
+            0.024864325299859047,
+            0.006338136270642281,
+            0.05216974765062332,
+            -0.04125455394387245,
+            -0.03724417835474014,
+            0.005971971433609724,
+            0.0016215863870456815,
+            0.005871712230145931,
+            -0.011272640898823738,
+            -0.030932195484638214,
+            -0.012074715457856655,
+            0.004768858663737774,
+            -0.015535843558609486,
+            -0.015684053301811218,
+            0.00019043832435272634,
+            0.022144244983792305,
+            0.0011257383739575744,
+            0.005475033540278673,
+            0.00287700816988945,
+            0.014585559256374836,
+            0.03581439331173897,
+            -0.02589307352900505,
+            0.010758266784250736,
+            0.03147272393107414,
+            -0.009136680513620377,
+            -0.026764893904328346,
+            -0.027305422350764275,
+            0.015073779039084911,
+            -0.018918508663773537,
+            -0.028874699026346207,
+            3.807404209510423e-05,
+            -0.0034872826654464006,
+            0.008391273207962513,
+            -0.03584926575422287,
+            -0.0076763806864619255,
+            0.0021283321548253298,
+            -0.00196377607062459,
+            0.0007078094058670104,
+            -0.022318609058856964,
+            -0.013591683469712734,
+            0.01741897687315941,
+            0.024219177663326263,
+            0.005727861542254686,
+            0.03028704971075058,
+            -0.010243892669677734,
+            -0.00016768925706855953,
+            0.006525577511638403,
+            0.01589328981935978,
+            0.023050937801599503,
+            0.023312482982873917,
+            0.026573093608021736,
+            0.0009344827267341316,
+            -0.012327543459832668,
+            0.004252305254340172,
+            0.032484035938978195,
+            -0.016058936715126038,
+            -0.0030993223190307617,
+            0.014812232926487923,
+            0.03274558484554291,
+            0.010627493262290955,
+            -0.012188052758574486,
+            -0.02528279833495617,
+            0.015431225299835205,
+            -0.022283734753727913,
+            0.012510626576840878,
+            0.04777577146887779,
+            0.004393976181745529,
+            -0.029240863397717476,
+            0.02430635876953602,
+            -0.009546436369419098,
+            -0.024044813588261604,
+            0.027165930718183517,
+            0.003757546888664365,
+            -0.03271070867776871,
+            -6.98818766977638e-05,
+            -0.00018321855168323964,
+            0.014794795773923397,
+            -0.026015127077698708,
+            -0.015579435043036938,
+            0.02322530187666416,
+            0.009293608367443085,
+            0.0032627887558192015,
+            -0.021603714674711227,
+            0.001829733606427908,
+            0.015344043262302876,
+            0.0018177460879087448,
+            0.009424380958080292,
+            0.009145398624241352,
+            0.01505634281784296,
+            0.006939692422747612,
+            -0.013190646655857563,
+            -0.0032344546634703875,
+            -0.02720080316066742,
+            -0.006098385434597731,
+            0.0652819275856018,
+            -0.007292779628187418,
+            0.03686057776212692,
+            0.0015213270671665668,
+            0.012868072837591171,
+            0.025788454338908195,
+            -0.025160742923617363,
+            -0.02437610551714897,
+            -0.005047841463238001,
+            0.006961487699300051,
+            -0.04292844980955124,
+            -0.013269110582768917,
+            -0.023818138986825943,
+            -0.030217302963137627,
+            0.01933698169887066,
+            0.0005187332862988114,
+            0.0019267236348241568,
+            0.024829450994729996,
+            0.0051742554642260075,
+            -0.015038905665278435,
+            -0.017009221017360687,
+            0.0383601076900959,
+            0.016625620424747467,
+            0.028229551389813423,
+            -0.015989189967513084,
+            0.01625945419073105,
+            -0.008099214173853397,
+            0.003171247662976384,
+            0.03236198052763939,
+            -0.0026982848066836596,
+            -0.0030186788644641638,
+            0.009197707287967205,
+            0.01881388947367668,
+            0.040731459856033325,
+            -0.01745384931564331,
+            -0.030827578157186508,
+            -0.008914366364479065,
+            -0.014236831106245518,
+            0.021969880908727646,
+            -0.0009491947130300105,
+            -0.00041084547410719097,
+            -0.003934090491384268,
+            0.04352128505706787,
+            -0.029572155326604843,
+            0.014428631402552128,
+            -0.024445850402116776,
+            -0.039022691547870636,
+            0.017706677317619324,
+            -0.006102744489908218,
+            0.032693274319171906,
+            0.006883023772388697,
+            -0.007833308540284634,
+            -0.032867636531591415,
+            0.0016749853966757655,
+            0.0086179468780756,
+            -0.03194350749254227,
+            0.020418038591742516,
+            0.02472483366727829,
+            -0.00651250034570694,
+            0.0033063797745853662,
+            -0.016782546415925026,
+            0.008142804726958275,
+            -0.013844511471688747,
+            -0.022946318611502647,
+            0.022737082093954086,
+            -0.004298075567930937,
+            -0.028055187314748764,
+            0.006686864420771599,
+            -0.022458098828792572,
+            0.004707831423729658,
+            -0.0016771649243310094,
+            -0.00017014124023262411,
+            -0.00797280017286539,
+            0.010461847297847271,
+            -0.007571762427687645,
+            -0.01073211245238781,
+            -0.022353481501340866,
+            0.009991064667701721,
+            0.025631526485085487,
+            0.0017588981427252293,
+            -0.004672958515584469,
+            0.006848151329904795,
+            -0.03658159449696541,
+            0.00574093870818615,
+            0.02413199469447136,
+            -0.012205488979816437,
+            -0.008713847026228905,
+            -0.018255924805998802,
+            -0.029380355030298233,
+            -0.002366993110626936,
+            0.01130751334130764,
+            0.03584926575422287,
+            0.025788454338908195,
+            0.030409103259444237,
+            0.03391382470726967,
+            0.010810576379299164,
+            -0.00398204056546092,
+            0.009127962402999401,
+            -0.029031626880168915,
+            0.04006887599825859,
+            0.0040823002345860004,
+            0.02608487382531166,
+            0.016172273084521294,
+            -0.0459623858332634,
+            -0.00968592707067728,
+            0.014681459404528141,
+            0.027898261323571205,
+            0.044148996472358704,
+            0.025038689374923706,
+            -0.0010232995264232159,
+            0.027654150500893593,
+            -0.022196553647518158,
+            -0.02878751792013645,
+            0.01956365443766117,
+            -0.005235282704234123,
+            -0.00918898917734623,
+            -0.0013131798477843404,
+            0.021533969789743423,
+            -0.011385977268218994,
+            0.016058936715126038,
+            0.03415793180465698,
+            -0.008461019024252892,
+            0.012135743163526058,
+            0.0005590550135821104,
+            -0.008286654949188232,
+            0.026764893904328346,
+            -0.007214315701276064,
+            0.03485538810491562,
+            -0.01650356501340866,
+            -0.0018656961619853973,
+            0.000780279457103461,
+            -0.010766984894871712,
+            0.028020314872264862,
+            0.007270984351634979,
+            -0.06172490119934082,
+            0.00873564276844263,
+            0.011063403449952602,
+            0.013835793361067772,
+            -0.001278307056054473,
+            -0.0033957413397729397,
+            -0.025788454338908195,
+            0.019040562212467194,
+            0.008129727095365524,
+            0.02472483366727829,
+            0.010679802857339382,
+            0.019493909552693367,
+            -0.016643056645989418,
+            0.016599465161561966,
+            -0.0026590528432279825,
+            -0.02854340709745884,
+            0.007266624830663204,
+            0.013181927613914013,
+            0.0402083694934845,
+            -0.026538221165537834,
+            -0.002582768676802516,
+            0.010889039374887943,
+            -0.006970205809921026,
+            -0.011194176971912384,
+            -0.030443977564573288,
+            -0.0010756086558103561,
+            -0.028421353548765182,
+            0.02416686899960041,
+            -0.005139382556080818,
+            0.019406726583838463,
+            -0.027357731014490128,
+            0.0063337767496705055,
+            0.00043264098349027336,
+            -0.027654150500893593,
+            0.006390445400029421,
+            0.01782001368701458,
+            0.0031189383007586002,
+            -0.031176306307315826,
+            -0.006006844341754913,
+            0.023242738097906113,
+            0.03619799390435219,
+            0.006713018752634525,
+            -0.006848151329904795,
+            0.021533969789743423,
+            -0.015579435043036938,
+            -0.006150694563984871,
+            0.036546722054481506,
+            -0.025910509750247,
+            0.012231643311679363,
+            0.03759290650486946,
+            -0.03543078899383545,
+            -0.03196094557642937,
+            -0.0004021272761747241,
+            -0.04142891615629196,
+            0.001680434332229197,
+            -0.002975087845697999,
+            0.00835204217582941,
+            -0.0009715351043269038,
+            -0.027113622054457664,
+            0.024812014773488045,
+            -0.0003577189054340124,
+            0.013373728841543198,
+            -0.006848151329904795,
+            0.0021065366454422474,
+            -0.025143306702375412,
+            -0.01862208917737007,
+            -0.010793139226734638,
+            0.00015651904686819762,
+            0.0023059656377881765,
+            -0.008112290874123573,
+            -0.03944116458296776,
+            -0.020976005122065544,
+            0.007257906720042229,
+            0.00045770584256388247,
+            0.03584926575422287,
+            0.018482597544789314,
+            0.012188052758574486,
+            -0.02165602520108223,
+            -0.01898825354874134,
+            0.04756653308868408,
+            -0.026416165754199028,
+            0.0402432419359684,
+            -0.012135743163526058,
+            -0.017488721758127213,
+            0.002946753753349185,
+            0.02625923790037632,
+            -0.020923694595694542,
+            0.016817420721054077,
+            -0.01561430748552084,
+            0.005719143431633711,
+            -0.01409733947366476,
+            -0.004755781497806311,
+            0.009642336517572403,
+            -0.0017796038882806897,
+            0.018255924805998802,
+            0.0009072383400052786,
+            -0.0031123997177928686,
+            0.03183889016509056,
+            0.013652711175382137,
+            0.007763562723994255,
+            -0.016773829236626625,
+            -0.012222925201058388,
+            -0.010766984894871712,
+            0.007087901700288057,
+            -0.015073779039084911,
+            -0.007066106423735619,
+            -0.029659338295459747,
+            0.001694601378403604,
+            0.0007154378108680248,
+            -0.015256861224770546,
+            0.01235369872301817,
+            0.02706131339073181,
+            -0.013295264914631844,
+            -0.03157734498381615,
+            -0.006229158490896225,
+            -0.03944116458296776,
+            -0.021045750007033348,
+            0.019511345773935318,
+            -0.0005190056981518865,
+            0.009127962402999401,
+            0.004250125493854284,
+            0.00043509298120625317,
+            -0.022946318611502647,
+            -0.017976941540837288,
+            0.00714892940595746,
+            -0.009572590701282024,
+            0.00107724335975945,
+            0.00131862866692245,
+            0.01822105050086975,
+            -0.010156710632145405,
+            0.02800287865102291,
+            -0.017497440800070763,
+            0.015265579335391521,
+            -0.029851138591766357,
+            -0.02760184183716774,
+            -0.03199581801891327,
+            0.01304243691265583,
+            -0.014611713588237762,
+            -0.008195114322006702,
+            -0.010113119147717953,
+            0.0004999346565455198,
+            0.007567403372377157,
+            -0.009441817179322243,
+            0.00842614658176899,
+            0.011603932827711105,
+            0.013303983025252819,
+            0.007406116463243961,
+            -0.03244916349649429,
+            -0.02029598504304886,
+            -0.02397506684064865,
+            0.043974634259939194,
+            -0.030513722449541092,
+            0.01703537441790104,
+            -0.017392821609973907,
+            0.005823761690407991,
+            -0.0363723561167717,
+            -0.019075434654951096,
+            0.008953598327934742,
+            0.020749330520629883,
+            -0.025910509750247,
+            0.02741004154086113,
+            0.02934548258781433,
+            0.03013012185692787,
+            -0.036546722054481506,
+            0.025352545082569122,
+            -0.010505438782274723,
+            -0.02512587048113346,
+            0.027357731014490128,
+            0.03731392323970795,
+            0.03679082915186882,
+            -0.013460910879075527,
+            -0.012589089572429657,
+            -0.014690177515149117,
+            0.012318825349211693,
+            -0.0021555765997618437,
+            0.005801966413855553,
+            -0.037069812417030334,
+            0.01862208917737007,
+            0.009895164519548416,
+            0.005121946334838867,
+            0.03773239627480507,
+            -0.01898825354874134,
+            0.02054009400308132,
+            0.007471502758562565,
+            -0.01839541457593441,
+            0.006974565330892801,
+            0.000942111189942807,
+            -0.0004040343628730625,
+            0.003602798795327544,
+            0.017584621906280518,
+            0.005553497467190027,
+            -0.054541099816560745,
+            0.006582245696336031,
+            0.027898261323571205,
+            0.020418038591742516,
+            0.005235282704234123,
+            -0.026398729532957077,
+            0.014629150740802288,
+            0.015823544934391975,
+            0.002135960618034005,
+            0.007035592570900917,
+            0.016538437455892563,
+            -0.011080840602517128,
+            0.00042283302173018456,
+            0.0030339357908815145,
+            0.03720930591225624,
+            0.01475120522081852,
+            -0.013190646655857563,
+            -0.025056125596165657,
+            -0.016364073380827904,
+            0.0066519915126264095,
+            -0.005915303248912096,
+            -0.0004182014672551304,
+            -0.037488289177417755,
+            -0.03598875552415848,
+            0.013521937653422356,
+            -0.01818617805838585,
+            -0.03815086930990219,
+            -0.0042741005308926105,
+            -0.03222249075770378,
+            0.005200410261750221,
+            -0.014890696853399277,
+            -0.019319545477628708,
+            -0.03926680237054825,
+            -0.0038970382884144783,
+            0.00012893410166725516,
+            0.04045247659087181,
+            0.00037842465098947287,
+            0.03476820886135101,
+            -0.007253547664731741,
+            -0.0013676686212420464,
+            -0.004143327474594116,
+            0.022876573726534843,
+            -0.03147272393107414,
+            -0.04529980197548866,
+            0.026573093608021736,
+            0.005832480266690254,
+            0.016024062409996986,
+            0.035744644701480865,
+            -0.013870666734874249,
+            -0.010200301185250282,
+            0.024864325299859047,
+            -0.016747673973441124,
+            -0.010984940454363823,
+            -0.0022231426555663347,
+            0.0025980256032198668,
+            -0.012231643311679363,
+            -0.013521937653422356,
+            -0.011656241491436958,
+            0.05157690867781639,
+            0.005575293209403753,
+            0.027845950797200203,
+            0.04038273170590401,
+            0.0010271136416122317,
+            -0.04059197008609772,
+            0.0032889433205127716,
+            -0.0091018071398139,
+            -0.009642336517572403,
+            -0.004625008441507816,
+            0.033024564385414124,
+            0.001587803359143436,
+            0.017872322350740433,
+            -0.0042174323461949825,
+            -0.016416382044553757,
+            -0.020418038591742516,
+            -0.012318825349211693,
+            -0.00761535344645381,
+            -0.004145507235080004,
+            -0.03373945876955986,
+            -0.01256293524056673,
+            0.031716834753751755,
+            0.025335106998682022,
+            -0.022388353943824768,
+            -0.004925786517560482,
+            -0.01782001368701458,
+            -0.007672021631151438,
+            0.010479284450411797,
+            -0.002959831152111292,
+            -0.00416948227211833,
+            0.009250016883015633,
+            0.018133869394659996,
+            0.01007824670523405,
+            0.00766766257584095,
+            -0.05715656280517578,
+            -0.0019398009171709418,
+            0.011351104825735092,
+            0.010400820523500443,
+            -0.003968963399529457,
+            0.014541968703269958,
+            -0.012580371461808681,
+            -0.01265011727809906,
+            -0.01975545659661293,
+            0.0326235294342041,
+            0.006355572491884232,
+            0.030426539480686188,
+            -0.0005541510181501508,
+            0.009075652807950974,
+            0.02149909734725952,
+            0.004101916216313839,
+            0.012807045131921768,
+            0.00199102028273046,
+            0.019807765260338783,
+            0.00986029114574194,
+            -0.03142041712999344,
+            0.012414725497364998,
+            0.0010429153917357326,
+            -0.0024519956205040216,
+            -0.017872322350740433,
+            0.006922255735844374,
+            0.0023757112212479115,
+            -0.013949130661785603,
+            0.0041215321980416775,
+            -0.022370917722582817,
+            0.005954534746706486,
+            0.031089123338460922,
+            0.033233802765607834,
+            0.009206426329910755,
+            -0.015169679187238216,
+            -0.010810576379299164,
+            0.002057496691122651,
+            0.02008674666285515,
+            0.025404853746294975,
+            0.025003815069794655,
+            -0.0268172025680542,
+            -0.014393758960068226,
+            -0.003729212796315551,
+            -0.001253242138773203,
+            -0.010932630859315395,
+            0.03046141378581524,
+            0.02833417057991028,
+            -0.0008914365898817778,
+            0.0028835467528551817,
+            -0.0051916916854679585,
+            0.01523070689290762,
+            0.01898825354874134,
+            -0.017340512946248055,
+            0.0017665266059339046,
+            0.022283734753727913,
+            -0.020191365852952003,
+            0.022946318611502647,
+            -0.014672741293907166,
+            -0.011429568752646446,
+            0.02013905718922615,
+            0.09366840869188309,
+            -0.005248360335826874,
+            0.01120289508253336,
+            -0.028700334951281548,
+            -0.019214926287531853,
+            -0.008474096655845642,
+            -0.0035330532118678093,
+            0.016765110194683075,
+            0.026503346860408783,
+            0.009511562995612621,
+            0.021359605714678764,
+            0.015466097742319107,
+            -0.01666049286723137,
+            0.014411195181310177,
+            -0.009694645181298256,
+            0.016198428347706795,
+            -0.01362655684351921,
+            -0.004267562180757523,
+            0.010069528594613075,
+            -0.001332795829512179,
+            0.009816700592637062,
+            0.018674397841095924,
+            0.016799984499812126,
+            0.013190646655857563,
+            -0.024864325299859047,
+            0.00949412677437067,
+            0.002284170128405094,
+            -0.007763562723994255,
+            -0.02013905718922615,
+            -0.02322530187666416,
+            0.004999891389161348,
+            0.004428848624229431,
+            0.005296310409903526,
+            0.010331074707210064,
+            -0.0032889433205127716,
+            0.03811599686741829,
+            -0.03319893032312393,
+            0.01496044173836708,
+            0.006917896680533886,
+            -0.013103464618325233,
+            0.011935224756598473,
+            0.016198428347706795,
+            0.004049607086926699,
+            0.012249079532921314,
+            -0.011569059453904629,
+            0.014245549216866493,
+            0.02052265778183937,
+            -0.029781393706798553,
+            -0.00266123260371387,
+            -0.016372792422771454,
+            -0.008474096655845642,
+            0.0026677711866796017,
+            -0.016172273084521294,
+            -0.006006844341754913,
+            -0.00546195637434721,
+            0.01169983297586441,
+            0.0024825092405080795,
+            -0.0325886569917202,
+            0.005060918629169464,
+            0.00261110276915133,
+            0.005928380414843559,
+            0.018168741837143898,
+            0.01632048189640045,
+            0.01765436865389347,
+            -0.011525468900799751,
+            -0.01975545659661293,
+            0.006285826675593853,
+            0.0045029534958302975,
+            -0.01073211245238781,
+            0.0023473771288990974,
+            0.0024912275839596987,
+            0.01588457264006138,
+            -0.011185458861291409,
+            -0.005823761690407991,
+            -0.004141148179769516,
+            0.011664960533380508,
+            0.0031516316812485456,
+            -0.014541968703269958,
+            0.003940629307180643,
+            -0.021150369197130203,
+            0.020644713193178177,
+            0.011577778495848179,
+            -0.0019932000432163477,
+            -0.02088882215321064,
+            -0.02873520739376545,
+            0.01332141924649477,
+            -0.020836513489484787,
+            0.0031952227000147104,
+            0.015771236270666122,
+            -0.011673678644001484,
+            0.013260391540825367,
+            -0.018255924805998802,
+            -0.018866198137402534,
+            -0.0114818774163723,
+            -0.00473834527656436,
+            -0.005226564593613148,
+            -0.007576121482998133,
+            0.007502016611397266,
+            -0.01823848858475685,
+            -0.0306706503033638,
+            -0.033634841442108154,
+            0.019023125991225243,
+            0.01670408435165882,
+            -0.009467972442507744,
+            -0.015823544934391975,
+            -0.004642445128411055,
+            0.02319042943418026,
+            -0.0034415118861943483,
+            0.0007110786973498762,
+            -0.0026830281130969524,
+            0.013391165062785149,
+            -0.007610994391143322,
+            0.00016441992192994803,
+            -0.02284170128405094,
+            0.00044217653339728713,
+            0.0011725988006219268,
+            0.013391165062785149,
+            -0.027357731014490128,
+            -0.003380484413355589,
+            0.0003002332232426852,
+            -0.018098996952176094,
+            -0.004476799163967371,
+            0.04254484549164772,
+            -0.013417319394648075,
+            0.008979752659797668,
+            0.021551406010985374,
+            0.010104401037096977,
+            -0.00977310910820961,
+            0.010095682926476002,
+            0.007850744761526585,
+            -0.0028857262805104256,
+            0.026538221165537834,
+            0.022911446169018745,
+            0.01914518140256405,
+            0.01670408435165882,
+            -0.022161681205034256,
+            0.012528062798082829,
+            0.021969880908727646,
+            0.03752316161990166,
+            -0.017331793904304504,
+            -0.021952442824840546,
+            0.019860073924064636,
+            0.04676445946097374,
+            0.0021032672375440598,
+            -0.0006200824282132089,
+            -0.028648026287555695,
+            -0.017201021313667297,
+            0.024689961224794388,
+            0.020557530224323273,
+            0.016224581748247147,
+            -0.010121837258338928,
+            -0.031507596373558044,
+            0.014280421659350395,
+            -0.00309278373606503,
+            0.006865587551146746,
+            -0.01996469311416149,
+            0.01688716560602188,
+            -0.015004033222794533,
+            0.04257972165942192,
+            -0.012658835388720036,
+            0.004829886369407177,
+            0.017148712649941444,
+            0.011150586418807507,
+            -0.022318609058856964,
+            0.0009263094398193061,
+            -0.01666049286723137,
+            0.027933133766055107,
+            -0.018883634358644485,
+            0.003618055721744895,
+            0.01914518140256405,
+            -0.016608182340860367,
+            -0.014533249661326408,
+            0.004191277548670769,
+            0.019459037110209465,
+            0.006107103545218706,
+            -0.025247925892472267,
+            0.018098996952176094,
+            0.0019190951716154814,
+            0.009171552956104279,
+            -0.009930036962032318,
+            0.023870449513196945,
+            -0.027636714279651642,
+            -0.015387633815407753,
+            0.025003815069794655,
+            0.0041607641614973545,
+            0.013565529137849808,
+            -0.026311546564102173,
+            0.009380790404975414,
+            -0.029049063101410866,
+            0.02336479350924492,
+            0.0032584297005087137,
+            0.014280421659350395,
+            0.0015147883677855134,
+            0.002866110298782587,
+            -0.0015115190763026476,
+            0.014446067623794079,
+            -0.000550881668459624,
+            -0.022527845576405525,
+            -0.01569277234375477,
+            -0.03696519508957863,
+            -0.022737082093954086,
+            0.023818138986825943,
+            -0.002715721260756254,
+            0.003528693923726678,
+            0.02430635876953602,
+            -0.027462350204586983,
+            -0.01435888558626175,
+            0.009572590701282024,
+            0.02549203485250473,
+            -0.043625906109809875,
+            4.093470488442108e-05,
+            0.0013829254312440753,
+            -0.019232362508773804,
+            0.010627493262290955,
+            -0.005440160632133484,
+            0.015500971116125584,
+            0.0018776836805045605,
+            -0.037662651389837265,
+            0.011586496606469154,
+            0.03762777894735336,
+            -0.017924632877111435,
+            0.0028225192800164223,
+            0.017706677317619324,
+            0.012493189424276352,
+            -0.0012390750925987959,
+            -0.0017033196054399014,
+            0.01877901703119278,
+            -0.025945382192730904,
+            0.026102310046553612,
+            0.017026657238602638,
+            -0.014881978742778301,
+            0.007842026650905609,
+            -0.008753078989684582,
+            0.020731894299387932,
+            -0.0013208083109930158,
+            -0.013155773282051086,
+            0.017375385388731956,
+            0.002195898210629821,
+            0.01349578332155943,
+            -0.001912556472234428,
+            0.004304614383727312,
+            0.03107168711721897,
+            -0.01609380915760994,
+            0.02280682697892189,
+            0.01994725689291954,
+            3.6405323044164106e-05,
+            0.015405070967972279,
+            -0.019877510145306587,
+            -0.03312918543815613,
+            0.02221398986876011,
+            -0.013591683469712734,
+            -0.007310215849429369,
+            0.023347357288002968,
+            -0.010453129187226295,
+            -0.006250953767448664,
+            -0.004282819107174873,
+            0.004077941179275513,
+            0.012406007386744022,
+            -0.0230160653591156,
+            -0.015631744638085365,
+            0.022318609058856964,
+            0.017131276428699493,
+            0.022179117426276207,
+            0.023800702765583992,
+            -0.015378915704786777,
+            0.03182145208120346,
+            0.009223862551152706,
+            0.0065735275857150555,
+            0.0076371487230062485,
+            0.008395632728934288,
+            -0.005871712230145931,
+            0.01933698169887066,
+            0.014114775694906712,
+            0.007336370646953583,
+            -0.003565746359527111,
+            -0.02029598504304886,
+            -0.011447004973888397,
+            -0.027532095089554787,
+            -0.00045825072447769344,
+            -0.013914257287979126,
+            0.00704866973683238,
+            0.07797563821077347,
+            -0.006682505365461111,
+            0.0018144766800105572,
+            -0.007868180982768536,
+            0.000739412906114012,
+            0.0016771649243310094,
+            -0.005209128372371197,
+            0.016538437455892563,
+            -0.020662149414420128,
+            -0.020749330520629883,
+            -0.006163771729916334,
+            -0.010967503301799297,
+            0.013460910879075527,
+            0.01741897687315941,
+            0.0030034221708774567,
+            -0.02277195453643799,
+            0.011141867376863956,
+            -0.016355354338884354,
+            -0.011106994934380054,
+            0.010348510928452015,
+            -0.007231752388179302,
+            -0.011656241491436958,
+            0.03295481950044632,
+            -0.009206426329910755,
+            -0.03147272393107414,
+            0.015640461817383766,
+            -0.0024541751481592655,
+            -0.005976330488920212,
+            0.00470347236841917,
+            0.00853076484054327,
+            -0.004986814223229885,
+            0.0005544234300032258,
+            0.013504501432180405,
+            -0.002068394562229514,
+            0.005414006300270557,
+            0.005318105686455965,
+            -0.013565529137849808,
+            0.017296921461820602,
+            0.022527845576405525,
+            -0.033477913588285446,
+            -0.004755781497806311,
+            -0.013940411619842052,
+            0.010479284450411797,
+            0.018709270283579826,
+            0.015535843558609486,
+            -0.022580154240131378,
+            0.0025871277321130037,
+            -0.009154116734862328,
+            -0.012702426873147488,
+            -0.02741004154086113,
+            -0.018255924805998802,
+            0.00723611144348979,
+            -0.00851768720895052,
+            0.018656961619853973,
+            0.007907412946224213,
+            -0.005745298229157925,
+            0.005012968555092812,
+            -0.005178614519536495,
+            0.015518407337367535,
+            -0.0326235294342041,
+            0.010461847297847271,
+            -0.009337198920547962,
+            0.009581308811903,
+            0.01055774837732315,
+            -0.022388353943824768,
+            -0.014759923331439495,
+            -0.023417102172970772,
+            0.016555873677134514,
+            -0.010086964815855026,
+            -0.0020934592466801405,
+            0.02008674666285515,
+            0.0009088730439543724,
+            0.015413789078593254,
+            -0.001491903094574809,
+            0.023242738097906113,
+            0.014140930958092213,
+            0.016590746119618416,
+            -0.01215317938476801,
+            -0.001640112604945898,
+            -0.011124431155622005,
+            0.012719863094389439,
+            -0.02303350158035755,
+            -0.006813278421759605,
+            -0.026015127077698708,
+            0.03598875552415848,
+            -0.007602275814861059,
+            -0.01588457264006138,
+            0.02857828140258789,
+            -0.012414725497364998,
+            -0.0037052377592772245,
+            0.0058847893960773945,
+            -0.015431225299835205,
+            0.023451974615454674,
+            0.0004955755430273712,
+            0.011499314568936825,
+            -0.02549203485250473,
+            0.00863102450966835,
+            -0.007087901700288057,
+            0.006076589692384005,
+            -0.02128986082971096,
+            -0.002423661295324564,
+            0.012135743163526058,
+            -0.010426974855363369,
+            0.00411717314273119,
+            0.021185241639614105,
+            0.005331183318048716,
+            -0.01744513027369976,
+            0.03951091319322586,
+            -0.004025631584227085,
+            0.010653648525476456,
+            0.005475033540278673,
+            -0.0009028792264871299,
+            0.016799984499812126,
+            0.01742769405245781,
+            -0.004925786517560482,
+            -0.016547156497836113,
+            0.0032235567923635244,
+            -0.015021469444036484,
+            0.01561430748552084,
+            0.016233300790190697,
+            -0.02125498652458191,
+            -0.032658401876688004,
+            0.01779385842382908,
+            0.002061855746433139,
+            0.028072625398635864,
+            0.006983283441513777,
+            -0.025997690856456757,
+            -0.005296310409903526,
+            -0.03815086930990219,
+            0.012885509058833122,
+            -0.01801181398332119,
+            -0.02437610551714897,
+            0.002785466844215989,
+            -0.00986029114574194,
+            -0.01856978051364422,
+            0.02512587048113346,
+            0.013652711175382137,
+            -0.02552690915763378,
+            0.010889039374887943,
+            0.0013011923292651772,
+            -0.017549749463796616,
+            0.010052092373371124,
+            0.004350385162979364,
+            -0.012493189424276352,
+            0.007410475518554449,
+            0.005130664445459843,
+            -0.006978924386203289,
+            -0.02890957146883011,
+            -0.01670408435165882,
+            0.01972058229148388,
+            0.010993658564984798,
+            -0.0058455574326217175,
+            0.011290077120065689,
+            -0.00997362844645977,
+            0.0033456117380410433,
+            0.0035221553407609463,
+            0.00901462510228157,
+            -0.013051155023276806,
+            -0.007371243555098772,
+            -0.007162006571888924,
+            0.01973801851272583,
+            -0.016024062409996986,
+            0.007990236394107342,
+            -0.013443474657833576,
+            -0.007981518283486366,
+            0.011429568752646446,
+            -0.0013774766121059656,
+            0.014986597001552582,
+            -0.018744144588708878,
+            0.0073407297022640705,
+            0.01189163327217102,
+            -0.008883852511644363,
+            -0.010671084746718407,
+            -0.02437610551714897,
+            0.019110308960080147,
+            0.01438504084944725,
+            0.004694754257798195,
+            -0.03679082915186882,
+            0.003951527178287506,
+            -0.010453129187226295,
+            -0.0009775288635864854,
+            -0.008796670474112034,
+            0.019249800592660904,
+            0.018831325694918633,
+            0.007985876873135567,
+            0.013277828693389893,
+            -0.0023081451654434204,
+            0.013548092916607857,
+            -0.012414725497364998,
+            0.017593340948224068,
+            -0.004036529455333948,
+            -0.017575904726982117,
+            -0.005714784376323223,
+            0.019406726583838463,
+            -0.001491903094574809,
+            0.007903054356575012,
+            0.004372180439531803,
+            0.010226456448435783,
+            -0.012275234796106815,
+            -0.007493298500776291,
+            0.014045030809938908,
+            0.008378196507692337,
+            -0.0004288267809897661,
+            0.009250016883015633,
+            0.06758353859186172,
+            -0.00870512891560793,
+            0.023853013291954994,
+            0.006311981473118067,
+            -0.009345917031168938,
+            -0.008112290874123573,
+            0.0019932000432163477,
+            -0.02951984666287899,
+            0.023329921066761017,
+            0.011970097199082375,
+            -0.009180271066725254,
+            0.00036126066697761416,
+            -0.006726096384227276,
+            0.018325669690966606,
+            -0.009694645181298256,
+            0.016948193311691284,
+            -0.010052092373371124,
+            0.02435866929590702,
+            0.019162617623806,
+            0.005640679504722357,
+            0.005536061245948076,
+            0.0028203397523611784,
+            0.019459037110209465,
+            0.012868072837591171,
+            -0.002165384590625763,
+            -0.003123297356069088,
+            -0.009877728298306465,
+            0.019389290362596512,
+            -0.001910376944579184,
+            0.030792705714702606,
+            -0.0022144243121147156,
+            -0.005182973574846983,
+            -0.009842854924499989,
+            0.0041607641614973545,
+            0.011569059453904629,
+            0.009520281106233597,
+            0.012876790948212147,
+            -0.0013567708665505052,
+            -0.010679802857339382,
+            0.011909069493412971,
+            0.00863102450966835,
+            0.005326824262738228,
+            -0.0032518908847123384,
+            0.0045247492380440235,
+            -0.008404350839555264,
+            0.014001439325511456,
+            -0.0024781501851975918,
+            0.0015354941133409739,
+            -0.008774874731898308,
+            0.004426669329404831,
+            -0.01312961895018816,
+            0.0005348074482753873,
+            0.009607463143765926,
+            0.012868072837591171,
+            0.0015213270671665668,
+            -0.002273272257298231,
+            0.015405070967972279,
+            0.0025980256032198668,
+            0.0073014977388083935,
+            0.0009486498311161995,
+            -0.009537717327475548,
+            0.019057998433709145,
+            -0.01033979281783104,
+            -0.0019212746992707253,
+            0.010705957189202309,
+            0.006067871581763029,
+            0.008260500617325306,
+            0.01496916078031063,
+            0.009354635141789913,
+            0.003968963399529457,
+            0.0013545913388952613,
+            -0.00565375667065382,
+            -0.003535232739523053,
+            0.0036267738323658705,
+            0.0016771649243310094,
+            0.01646869257092476,
+            -0.019459037110209465,
+            -0.007039951626211405,
+            0.010679802857339382,
+            0.011438286863267422,
+            0.0017839629435911775,
+            0.015684053301811218,
+            0.014428631402552128,
+            0.01994725689291954,
+            -0.022562718018889427,
+            0.003831651760265231,
+            -0.002920599188655615,
+            0.019057998433709145,
+            0.008770515210926533,
+            -0.024114558473229408,
+            0.0003645300166681409,
+            -0.01063621137291193,
+            0.012676272541284561,
+            -0.0067086596973240376,
+            0.0023473771288990974,
+            0.01665177382528782,
+            -0.010121837258338928,
+            -0.009930036962032318,
+            -0.019424164667725563,
+            -0.016983065754175186,
+            0.010976221412420273,
+            0.014437349513173103,
+            0.0014188881032168865,
+            0.019232362508773804,
+            -0.0012412546202540398,
+            -0.008474096655845642,
+            0.030827578157186508,
+            0.016765110194683075,
+            -0.0004157494695391506,
+            -0.008962316438555717,
+            -0.0006544103380292654,
+            -0.015082497149705887,
+            0.01074954867362976,
+            0.004426669329404831,
+            0.005108869168907404,
+            0.003387023229151964,
+            9.174550359603018e-05,
+            -0.016712801530957222,
+            -0.014620431698858738,
+            -0.004668599460273981,
+            0.011045967228710651,
+            -0.003751008305698633,
+            0.009197707287967205,
+            -0.01609380915760994,
+            0.003121117828413844,
+            -0.008173318579792976,
+            -0.010714675299823284,
+            -0.008356400765478611,
+            -0.039406292140483856,
+            -0.02067958563566208,
+            0.0003173972072545439,
+            -0.00473834527656436,
+            -0.002358274767175317,
+            0.009223862551152706,
+            -0.003672544378787279,
+            0.013670147396624088,
+            -0.0015551100950688124,
+            0.0047906544059515,
+            0.008657178841531277,
+            0.01561430748552084,
+            -0.013504501432180405,
+            -0.0006631285650655627,
+            -0.014071185141801834,
+            -0.026782330125570297,
+            0.01707896590232849,
+            0.003199581755325198,
+            0.014446067623794079,
+            -0.015187115408480167,
+            0.009528999216854572,
+            -0.012589089572429657,
+            0.014463504776358604,
+            -0.0421612448990345,
+            -0.009520281106233597,
+            0.007031233515590429,
+            -0.0075979167595505714,
+            -0.005496829282492399,
+            -0.025369981303811073,
+            -0.0012205488746985793,
+            -0.011054685339331627,
+            -0.01666049286723137,
+            -0.019459037110209465,
+            0.00574093870818615,
+            -0.009311044588685036,
+            0.008591792546212673,
+            -0.006120180711150169,
+            0.02761927805840969,
+            0.00532246520742774,
+            -0.001346962875686586,
+            -0.006128899287432432,
+            0.009206426329910755,
+            -0.0007503106608055532,
+            0.012885509058833122,
+            0.008648460730910301,
+            0.005823761690407991,
+            0.023434538394212723,
+            0.0009339378448203206,
+            0.004690395202487707,
+            -0.006141976453363895,
+            0.014541968703269958,
+            0.024689961224794388,
+            0.011342386715114117,
+            0.011455723084509373,
+            0.005967612378299236,
+            0.0005950175691395998,
+            0.011429568752646446,
+            0.017392821609973907,
+            -0.003990759141743183,
+            -0.009171552956104279,
+            0.004869118332862854,
+            -0.01761949434876442,
+            -0.0033347138669341803,
+            0.012885509058833122,
+            -0.006128899287432432,
+            -0.02706131339073181,
+            0.01285063661634922,
+            0.004317691549658775,
+            -0.006673786789178848,
+            -0.016948193311691284,
+            -0.00661711860448122,
+            0.004646804183721542,
+            0.004729627165943384,
+            0.024236613884568214,
+            -0.022144244983792305,
+            -0.008644101209938526,
+            0.011917788535356522,
+            -0.014446067623794079,
+            -0.01063621137291193,
+            0.009790545329451561,
+            0.029484974220395088,
+            0.003496000776067376,
+            -0.05307644233107567,
+            -0.012467035092413425,
+            0.014236831106245518,
+            -0.01503018755465746,
+            -0.025980254635214806,
+            -0.0014428631402552128,
+            0.0038578063249588013,
+            -0.014010157436132431,
+            0.006625836715102196,
+            0.0011606112821027637,
+            -0.011708551086485386,
+            0.017139993607997894,
+            0.028874699026346207,
+            0.01312961895018816,
+            0.0114818774163723,
+            -0.025213053449988365,
+            -0.0008952508214861155,
+            -0.01914518140256405,
+            0.011752142570912838,
+            -0.030566031113266945,
+            -0.013984003104269505,
+            -0.0023342997301369905,
+            0.004141148179769516,
+            0.027444913983345032,
+            0.014376322738826275,
+            0.025962818413972855,
+            0.006778405513614416,
+            0.0037989583797752857,
+            -0.02125498652458191,
+            -0.018447725102305412,
+            0.0033456117380410433,
+            -0.0008304091752506793,
+            -0.002329940674826503,
+            -0.008216909132897854,
+            0.012275234796106815,
+            0.012039843015372753,
+            0.026172054931521416,
+            0.005087073426693678,
+            0.005553497467190027,
+            -2.538836997700855e-05,
+            -0.00494322320446372,
+            0.007593557704240084,
+            -0.019232362508773804,
+            -0.0029925242997705936,
+            -0.004559622146189213,
+            0.015405070967972279,
+            0.0045029534958302975,
+            0.0022275017108768225,
+            0.017357949167490005,
+            -0.0030099607538431883,
+            0.003729212796315551,
+            -0.019982129335403442,
+            -0.014672741293907166,
+            -0.0032518908847123384,
+            0.03525642678141594,
+            0.003971143160015345,
+            -0.02761927805840969,
+            -0.016224581748247147,
+            -0.0153353251516819,
+            0.030792705714702606,
+            -0.009581308811903,
+            0.004315512254834175,
+            0.02683463878929615,
+            0.01593688130378723,
+            0.012388571165502071,
+            0.006399163510650396,
+            -0.01822105050086975,
+            0.005021686665713787,
+            0.02931061014533043,
+            -0.0030317562632262707,
+            -0.010086964815855026,
+            -0.0066912234760820866,
+            0.007144570350646973,
+            -0.007798435632139444,
+            0.007632789667695761,
+            -0.0035068984143435955,
+            -0.013417319394648075,
+            0.029554719105362892,
+            -0.00968592707067728,
+            0.022370917722582817,
+            -0.010130555368959904,
+            0.00268738716840744,
+            -0.038848329335451126,
+            -0.002004097681492567,
+            0.0011453543556854129,
+            0.0019692247733473778,
+            0.004555262625217438,
+            -0.01881388947367668,
+            -0.008753078989684582,
+            0.011839324608445168,
+            -0.01276345457881689,
+            -0.007885618135333061,
+            -0.006137617398053408,
+            0.002983806189149618,
+            0.01977289281785488,
+            -0.00947669055312872,
+            0.0009857022669166327,
+            -0.0210980586707592,
+            0.01256293524056673,
+            -0.00282687833532691,
+            0.0017567186150699854,
+            0.010453129187226295,
+            0.007009437773376703,
+            0.03502975404262543,
+            0.01094134896993637,
+            0.005531701724976301,
+            -0.0032780456822365522,
+            -0.014053748920559883,
+            -0.03218761831521988,
+            0.013295264914631844,
+            -0.010453129187226295,
+            0.023800702765583992,
+            0.0036703648511320353,
+            -0.008103572763502598,
+            -0.0051960512064397335,
+            0.0037139558698982,
+            0.009590026922523975,
+            -0.0041106343269348145,
+            0.013478347100317478,
+            -0.010243892669677734,
+            0.021167805418372154,
+            0.0031298361718654633,
+            -0.005121946334838867,
+            0.012571653351187706,
+            -0.017689241096377373,
+            0.01055774837732315,
+            -0.01744513027369976,
+            0.0028225192800164223,
+            0.01360040158033371,
+            -0.010296201333403587,
+            0.0056276023387908936,
+            -0.052588220685720444,
+            -0.001197663601487875,
+            -0.016521001234650612,
+            -0.00675660977140069,
+            -0.008456660434603691,
+            -0.021394478157162666,
+            -0.03319893032312393,
+            -0.0036660057958215475,
+            0.015370197594165802,
+            0.006551732309162617,
+            0.023260174319148064,
+            0.0009895163821056485,
+            0.012257798574864864,
+            0.012807045131921768,
+            -0.0047906544059515,
+            0.012519344687461853,
+            0.033861514180898666,
+            -0.027741333469748497,
+            -0.019685709848999977,
+            0.005257078446447849,
+            -0.00028552126605063677,
+            -0.015788672491908073,
+            -0.013103464618325233,
+            -0.0017665266059339046,
+            0.0024955866392701864,
+            0.030443977564573288,
+            -0.020191365852952003,
+            -0.005579652264714241,
+            -0.007053028792142868,
+            0.004433208145201206,
+            -0.01918005384504795,
+            -0.0031450928654521704,
+            -0.011708551086485386,
+            -0.012780890800058842,
+            0.016041500493884087,
+            -0.011089558713138103,
+            0.011159304529428482,
+            -0.019511345773935318,
+            0.008626664988696575,
+            -0.015378915704786777,
+            -0.010627493262290955,
+            -0.0031690681353211403,
+            -0.010461847297847271,
+            0.030164994299411774,
+            -0.0041890982538461685,
+            0.0048386044800281525,
+            0.01670408435165882,
+            -0.00853076484054327,
+            0.011603932827711105,
+            -0.0006124540232121944,
+            0.007898694835603237,
+            -0.002704823622480035,
+            -0.0086135882884264,
+            0.0015278656501322985,
+            0.009537717327475548,
+            -0.010950067080557346,
+            0.03846472501754761,
+            -0.013731175102293491,
+            -0.008879492990672588,
+            -0.003859985852614045,
+            -0.004511671606451273,
+            0.002425840822979808,
+            0.0021762822289019823,
+            0.010984940454363823,
+            -0.0012870251666754484,
+            0.006835073698312044,
+            -0.024219177663326263,
+            -0.011717269197106361,
+            0.0013164491392672062,
+            0.006468909326940775,
+            0.003321636701002717,
+            0.008016390725970268,
+            -0.025579217821359634,
+            -0.0020782025530934334,
+            -0.012859354726970196,
+            0.005475033540278673,
+            -0.011499314568936825,
+            0.007332011591643095,
+            -0.01215317938476801,
+            -0.0013295264216139913,
+            0.009337198920547962,
+            0.012946536764502525,
+            0.013861947692930698,
+            -0.006608400493860245,
+            0.026573093608021736,
+            0.014899414964020252,
+            0.008709488436579704,
+            0.005405288189649582,
+            -0.033216364681720734,
+            -0.007802794687449932,
+            0.010365947149693966,
+            0.007519452832639217,
+            0.009232580661773682,
+            -0.017689241096377373,
+            -0.004176021087914705,
+            0.0029402151703834534,
+            -0.007039951626211405,
+            4.651980270864442e-05,
+            0.011080840602517128,
+            -0.010627493262290955,
+            0.005505547393113375,
+            -0.00031521765049546957,
+            0.0027549532242119312,
+            -0.006460190750658512,
+            0.012188052758574486,
+            -0.013286546804010868,
+            -0.008168959058821201,
+            0.0211154967546463,
+            0.02760184183716774,
+            -0.005531701724976301,
+            0.023696085438132286,
+            -0.021917570382356644,
+            0.008487174287438393,
+            -0.005043482407927513,
+            -0.014149649068713188,
+            -0.015300451777875423,
+            0.006883023772388697,
+            0.014777359552681446,
+            0.004760140553116798,
+            -0.013155773282051086,
+            -0.003027397207915783,
+            0.016983065754175186,
+            -0.0010952246375381947,
+            0.006216081324964762,
+            0.01593688130378723,
+            -0.00026971951592713594,
+            0.00017667989595793188,
+            -0.006002485286444426,
+            -0.01876158080995083,
+            -0.006290185730904341,
+            0.017924632877111435,
+            -0.0073799616657197475,
+            0.014262985438108444,
+            0.02320786565542221,
+            0.0005429807933978736,
+            0.011351104825735092,
+            -0.0038469084538519382,
+            0.01351321954280138,
+            0.01505634281784296,
+            -0.005235282704234123,
+            -0.0030600903555750847,
+            -0.0036965194158256054,
+            0.00011858122888952494,
+            0.012022406794130802,
+            -0.01741897687315941,
+            -0.011333668604493141,
+            0.0114818774163723,
+            -0.002121793571859598,
+            -0.008404350839555264,
+            -0.023486848920583725,
+            0.0016313943779096007,
+            0.0077156126499176025,
+            0.011447004973888397,
+            -0.016425101086497307,
+            -0.0005239096935838461,
+            0.000875089957844466,
+            0.010801857337355614,
+            -0.007497657556086779,
+            0.01159521471709013,
+            -0.007907412946224213,
+            -0.00863102450966835,
+            0.006381727289408445,
+            0.0015507509233430028,
+            0.007414834573864937,
+            -0.009999782778322697,
+            0.0015987011138349771,
+            -0.008025108836591244,
+            0.026590529829263687,
+            0.008317168802022934,
+            -0.0013011923292651772,
+            -0.005640679504722357,
+            0.004995532333850861,
+            0.008421787060797215,
+            -0.0100608104839921,
+            0.012815763242542744,
+            0.006107103545218706,
+            -0.002933676354587078,
+            0.010592620819807053,
+            0.015283015556633472,
+            0.0025631526950746775,
+            0.009328480809926987,
+            0.003443691413849592,
+            0.0059022256173193455,
+            -0.007022515404969454,
+            0.0017861425876617432,
+            -0.01550968922674656,
+            -0.020801641047000885,
+            -0.0035831828135997057,
+            0.016991784796118736,
+            -0.010583902709186077,
+            -0.001182406791485846,
+            0.017924632877111435,
+            0.01371373888105154,
+            -0.005034764297306538,
+            0.00551426550373435,
+            0.002217693720012903,
+            -0.0015529305674135685,
+            0.003323816228657961,
+            -0.014280421659350395,
+            -0.012493189424276352,
+            0.008726924657821655,
+            -0.02087138593196869,
+            0.01952878199517727,
+            -0.02165602520108223,
+            -0.035361044108867645,
+            -0.008831542916595936,
+            -0.009197707287967205,
+            -0.01707896590232849,
+            -0.013574247248470783,
+            0.023661212995648384,
+            0.016207145527005196,
+            -0.009354635141789913,
+            -0.034384604543447495,
+            0.03155990689992905,
+            0.00918898917734623,
+            0.027933133766055107,
+            -0.015544561669230461,
+            0.004343846347182989,
+            -0.0015213270671665668,
+            0.014707613736391068,
+            -0.006163771729916334,
+            -0.006686864420771599,
+            0.0091018071398139,
+            0.014620431698858738,
+            0.010897758416831493,
+            0.003245352301746607,
+            -0.003160349791869521,
+            0.007279702462255955,
+            0.002530459314584732,
+            0.0022285915911197662,
+            -0.008583074435591698,
+            -0.02282426506280899,
+            -0.009781827218830585,
+            0.017942069098353386,
+            0.022266298532485962,
+            0.012746017426252365,
+            -0.017131276428699493,
+            0.0020520477555692196,
+            -0.00484296353533864,
+            -0.014053748920559883,
+            -0.009991064667701721,
+            0.009642336517572403,
+            -0.033617403358221054,
+            0.02317299321293831,
+            0.004760140553116798,
+            -0.00627274950966239,
+            -0.0015006213216111064,
+            0.003836010815575719,
+            0.005679911468178034,
+            -0.004313332494348288,
+            0.0002961465797852725,
+            -0.002532638842239976,
+            0.010897758416831493,
+            0.016599465161561966,
+            0.005100150592625141,
+            -0.010400820523500443,
+            0.012388571165502071,
+            -0.0031407338101416826,
+            -0.002761491807177663,
+            -0.013914257287979126,
+            -0.006011203397065401,
+            -0.009502844884991646,
+            0.010261328890919685,
+            0.01483838725835085,
+            0.0054924702271819115,
+            0.006791482679545879,
+            0.004335128236562014,
+            -0.00013084120291750878,
+            -0.0002569146454334259,
+            0.008469737134873867,
+            -0.0005070181796327233,
+            -0.0045639812014997005,
+            0.0010821473551914096,
+            -0.0014679280575364828,
+            0.02320786565542221,
+            -0.00532246520742774,
+            -0.010950067080557346,
+            0.007196879480034113,
+            0.02033085748553276,
+            -0.022370917722582817,
+            0.022091934457421303,
+            -0.022179117426276207,
+            0.023853013291954994,
+            0.03482051566243172,
+            0.013975284993648529,
+            -0.011717269197106361,
+            -0.007096619810909033,
+            0.012815763242542744,
+            -0.012510626576840878,
+            -0.0009600924677215517,
+            0.0018079380970448256,
+            -0.022580154240131378,
+            0.015178397297859192,
+            -0.023347357288002968,
+            0.014280421659350395,
+            0.005121946334838867,
+            -0.00852640625089407,
+            -0.0073232934810221195,
+            -0.0009764390997588634,
+            0.018133869394659996,
+            -0.003990759141743183,
+            -0.0006255313055589795,
+            -0.03028704971075058,
+            -0.006930973846465349,
+            0.02163858897984028,
+            -0.0045247492380440235,
+            -0.013949130661785603,
+            0.007253547664731741,
+            -8.973622243502177e-06,
+            -0.016154836863279343,
+            -0.005876071285456419,
+            -0.03166452422738075,
+            -0.008142804726958275,
+            -0.02264990098774433,
+            -0.0075979167595505714,
+            -0.011577778495848179,
+            -0.012292671017348766,
+            -0.0038512677419930696,
+            -0.016276890411973,
+            0.001949608908034861,
+            -0.001063076313585043,
+            0.0033020207192748785,
+            0.0027091826777905226,
+            -0.009563872590661049,
+            -0.003968963399529457,
+            0.021324733272194862,
+            0.010705957189202309,
+            -0.016041500493884087,
+            -0.026206929236650467,
+            0.011638805270195007,
+            -0.012205488979816437,
+            -0.007593557704240084,
+            0.001547481631860137,
+            -0.005915303248912096,
+            -0.019267236813902855,
+            -0.01860465295612812,
+            0.005967612378299236,
+            0.0003718859807122499,
+            0.0051175872795283794,
+            0.013582965359091759,
+            0.014341449365019798,
+            -0.0019180054077878594,
+            -0.006734814494848251,
+            -1.330718441749923e-05,
+            -0.002195898210629821,
+            0.014027593657374382,
+            0.004348205402493477,
+            0.024445850402116776,
+            -0.008744360879063606,
+            0.0028944446239620447,
+            -0.017924632877111435,
+            0.00449859444051981,
+            -0.007393038831651211,
+            -0.005679911468178034,
+            -0.004064863547682762,
+            -0.011089558713138103,
+            0.005993766710162163,
+            -0.0011126610916107893,
+            -0.00247161160223186,
+            -0.007911772467195988,
+            -0.0020912797190248966,
+            -0.01707024872303009,
+            -0.0029794471338391304,
+            -0.020557530224323273,
+            -0.0013905538944527507,
+            -0.014916851185262203,
+            0.017000501975417137,
+            0.026172054931521416,
+            -0.00513502350077033,
+            -0.0006091846735216677,
+            -0.011229049414396286,
+            -4.08665910072159e-05,
+            -0.011909069493412971,
+            -0.00637736776843667,
+            -0.016416382044553757,
+            -0.0325189083814621,
+            -0.016904601827263832,
+            0.009049498476088047,
+            0.024288922548294067,
+            0.0019398009171709418,
+            -0.024986378848552704,
+            0.0029511128086596727,
+            -0.026607966050505638,
+            -0.014228112995624542,
+            -0.01496916078031063,
+            -0.006529936566948891,
+            -0.009136680513620377,
+            0.017392821609973907,
+            -0.0026350778061896563,
+            0.01571020856499672,
+            0.01747128553688526,
+            0.010906476527452469,
+            0.007497657556086779,
+            0.010880321264266968,
+            -0.009528999216854572,
+            -0.00016932391736190766,
+            0.009869009256362915,
+            -0.006311981473118067,
+            0.011839324608445168,
+            -0.00815152283757925,
+            -0.0011933045461773872,
+            0.01879645325243473,
+            0.008016390725970268,
+            -0.006255313288420439,
+            -0.0026699507143348455,
+            -0.02742747776210308,
+            0.017506157979369164,
+            -0.018552342429757118,
+            -0.00258494820445776,
+            -0.002944574225693941,
+            -0.016738956794142723,
+            0.02319042943418026,
+            0.017323076725006104,
+            -0.005021686665713787,
+            -0.0027549532242119312,
+            0.017889760434627533,
+            -0.006451472640037537,
+            0.00398204056546092,
+            0.0025762300938367844,
+            -0.0033281752839684486,
+            0.009790545329451561,
+            0.0027636713348329067,
+            -0.004577058367431164,
+            -0.0025587936397641897,
+            0.012196770869195461,
+            0.003901397343724966,
+            0.013530656695365906,
+            -0.027131058275699615,
+            -0.010566466487944126,
+            -0.0010303830495104194,
+            0.01935441792011261,
+            -0.005405288189649582,
+            -0.0007906323298811913,
+            -0.013312701135873795,
+            -0.0008854428306221962,
+            -0.005719143431633711,
+            0.023818138986825943,
+            -0.01992982067167759,
+            0.006120180711150169,
+            -0.008395632728934288,
+            0.013861947692930698,
+            -0.0032191977370530367,
+            0.010801857337355614,
+            -0.0007955363253131509,
+            -1.3375295566220302e-05,
+            0.009964910335838795,
+            0.012981409206986427,
+            -0.0022024367935955524,
+            0.024027377367019653,
+            -0.0004737800336442888,
+            -0.006403522565960884,
+            0.0027745692059397697,
+            -0.01895338110625744,
+            0.0229637548327446,
+            0.003402280155569315,
+            0.004315512254834175,
+            -0.0058629936538636684,
+            -0.008199472911655903,
+            0.039789892733097076,
+            0.021516533568501472,
+            0.010392102412879467,
+            -0.000592293159570545,
+            -0.01494300551712513,
+            0.028229551389813423,
+            0.007519452832639217,
+            0.00930232647806406,
+            -0.004012554418295622,
+            0.0027113622054457664,
+            -0.015178397297859192,
+            -0.010976221412420273,
+            0.019807765260338783,
+            -0.019999565556645393,
+            -0.008731283247470856,
+            -0.0026328982785344124,
+            -0.019703146070241928,
+            -0.029293173924088478,
+            0.015474816784262657,
+            0.009764390997588634,
+            0.0028443147893995047,
+            0.008495892398059368,
+            0.0037335718516260386,
+            -0.0015028008492663503,
+            -0.011647523380815983,
+            -0.003986400086432695,
+            -0.008574356324970722,
+            -0.01044441107660532,
+            0.006848151329904795,
+            -0.012004970572888851,
+            0.006717377807945013,
+            -0.008256141096353531,
+            -0.00024533577379770577,
+            0.01285063661634922,
+            0.005091432482004166,
+            0.0062771085649728775,
+            0.003443691413849592,
+            0.005078355316072702,
+            0.01591944508254528,
+            -0.013757329434156418,
+            0.016346637159585953,
+            0.017113838344812393,
+            0.0062945447862148285,
+            -0.0034131777938455343,
+            0.0023103246930986643,
+            -0.029188554733991623,
+            -0.00022381270537152886,
+            -0.0016869729151949286,
+            0.012632681056857109,
+            -0.02163858897984028,
+            0.004733986221253872,
+            0.021568842232227325,
+            -0.0016172273317351937,
+            0.009816700592637062,
+            0.014864541590213776,
+            -0.0001531135058030486,
+            0.028037751093506813,
+            0.011342386715114117,
+            0.02416686899960041,
+            -0.009459254331886768,
+            -0.02852597087621689,
+            0.006368649657815695,
+            -0.0038904997054487467,
+            0.014376322738826275,
+            0.0091018071398139,
+            0.0201216209679842,
+            -0.01369630265980959,
+            -0.006695582531392574,
+            0.007789717521518469,
+            0.0018918508430942893,
+            0.014306576922535896,
+            0.003217018209397793,
+            0.019302109256386757,
+            0.009119244292378426,
+            -0.0008625575574114919,
+            -0.007737408392131329,
+            8.588795753894374e-05,
+            0.006250953767448664,
+            -0.003500359831377864,
+            -0.03047885000705719,
+            0.0016390228411182761,
+            -0.004834245424717665,
+            0.004978095646947622,
+            -0.014245549216866493,
+            0.010409538634121418,
+            -0.0010015039006248116,
+            0.022196553647518158,
+            -0.0041498662903904915,
+            0.004064863547682762,
+            0.01708768494427204,
+            0.00938950851559639,
+            0.004991173278540373,
+            0.015300451777875423,
+            0.005658115725964308,
+            0.004511671606451273,
+            -0.008038186468183994,
+            -0.019493909552693367,
+            0.019162617623806,
+            -0.027462350204586983,
+            0.010653648525476456,
+            -0.00873564276844263,
+            0.004317691549658775,
+            0.003925372380763292,
+            -0.006887382827699184,
+            -0.002567511750385165,
+            0.003607157850638032,
+            0.001491903094574809,
+            -0.059144310653209686,
+            0.0037706242874264717,
+            0.0007388679659925401,
+            0.020452912896871567,
+            -0.00881846621632576,
+            0.0007590288296341896,
+            0.0026394370943307877,
+            0.00390357687138021,
+            -0.013347573578357697,
+            -0.0023604542948305607,
+            0.01571892574429512,
+            -0.01858721673488617,
+            -0.0059632533229887486,
+            0.004038709215819836,
+            -0.005557856522500515,
+            0.0035940806847065687,
+            -0.008779234252870083,
+            -0.002622000640258193,
+            0.02838647924363613,
+            -0.001655369414947927,
+            0.03140297904610634,
+            0.002506484277546406,
+            -0.0004756871494464576,
+            -0.01666049286723137,
+            0.007410475518554449,
+            -0.016738956794142723,
+            -0.00425884360447526,
+            0.008927443064749241,
+            -0.018726706504821777,
+            -0.00123580580111593,
+            0.001693511614575982,
+            -0.0013382446486502886,
+            -0.0077722808346152306,
+            -0.001763257198035717,
+            0.0010429153917357326,
+            -0.02048778533935547,
+            0.015997909009456635,
+            -0.023853013291954994,
+            0.01625945419073105,
+            -0.0034371528308838606,
+            -0.002946753753349185,
+            -0.0013033718569204211,
+            -0.0038098562508821487,
+            0.005666834302246571,
+            0.02531767077744007,
+            -0.015448661521077156,
+            -0.00522220553830266,
+            0.008365118876099586,
+            -0.003567925887182355,
+            0.002730978187173605,
+            -0.008761797100305557,
+            0.006656350567936897,
+            -0.0325886569917202,
+            -0.008112290874123573,
+            0.004413592163473368,
+            0.0031908636447042227,
+            -0.001236895564943552,
+            -0.02758440561592579,
+            0.003960245288908482,
+            -0.00854384247213602,
+            -0.024201741442084312,
+            -0.00027380615938454866,
+            -0.020592402666807175,
+            0.006717377807945013,
+            -0.00881846621632576,
+            0.01761949434876442,
+            -0.014568123035132885,
+            0.011098276823759079,
+            -0.014367603696882725,
+            0.016006626188755035,
+            -0.007406116463243961,
+            -0.009398226626217365,
+            -0.0069266147911548615,
+            -0.006338136270642281,
+            0.01044441107660532,
+            -0.014916851185262203,
+            -0.009511562995612621,
+            -0.007641507778316736,
+            0.005161178298294544,
+            0.011045967228710651,
+            0.013574247248470783,
+            0.0011333668371662498,
+            -0.00880538858473301,
+            -0.016547156497836113,
+            -0.005392210558056831,
+            -0.014881978742778301,
+            -0.0009301236714236438,
+            0.0042174323461949825,
+            0.020853949710726738,
+            0.008160240948200226,
+            -0.003753187833353877,
+            0.011586496606469154,
+            0.020801641047000885,
+            0.014132212847471237,
+            -0.014777359552681446,
+            0.0021697436459362507,
+            -0.010095682926476002,
+            0.013417319394648075,
+            0.022144244983792305,
+            0.002098908182233572,
+            0.002796364715322852,
+            0.006085308268666267,
+            0.007951004430651665,
+            -0.004396155476570129,
+            0.008291014470160007,
+            -0.025457162410020828,
+            -0.012667553499341011,
+            -0.006569168530404568,
+            0.005387851502746344,
+            -0.015378915704786777,
+            0.029973194003105164,
+            -0.003609337378293276,
+            0.004472440108656883,
+            -0.002182820811867714,
+            0.0326409637928009,
+            0.007196879480034113,
+            0.00473834527656436,
+            0.002874828642234206,
+            9.467427298659459e-05,
+            0.009136680513620377,
+            -0.01761077716946602,
+            -0.03888320177793503,
+            0.003254070645198226,
+            -0.000592293159570545,
+            0.01766308583319187,
+            -0.01514352485537529,
+            0.006878664717078209,
+            -0.0036899808328598738,
+            -0.01282448135316372,
+            -0.009633618406951427,
+            0.03138554468750954,
+            -0.005466315429657698,
+            0.03387894853949547,
+            -0.007558684796094894,
+            -0.010104401037096977,
+            0.021725770086050034,
+            0.021900134161114693,
+            -0.014210676774382591,
+            0.005531701724976301,
+            0.015762517228722572,
+            0.020278548821806908,
+            0.005209128372371197,
+            -0.00806434080004692,
+            0.02627667412161827,
+            0.012946536764502525,
+            -0.012807045131921768,
+            -0.0026350778061896563,
+            -0.009572590701282024,
+            0.006726096384227276,
+            -0.012301389127969742,
+            0.005387851502746344,
+            0.006434036418795586,
+            0.004986814223229885,
+            0.012963972985744476,
+            -0.01879645325243473,
+            -0.003831651760265231,
+            0.0073014977388083935,
+            0.013966566883027554,
+            0.00651250034570694,
+            0.017558468505740166,
+            -0.023643774911761284,
+            0.0028966241516172886,
+            -0.02493407018482685,
+            -0.008103572763502598,
+            0.0029968833550810814,
+            0.001748000388033688,
+            0.008875134401023388,
+            -0.01592816226184368,
+            -0.029484974220395088,
+            0.011516750790178776,
+            0.01708768494427204,
+            0.02568383514881134,
+            0.0011126610916107893,
+            0.008914366364479065,
+            0.00560144754126668,
+            -0.013739893212914467,
+            -0.00805998221039772,
+            -0.00997362844645977,
+            -0.01073211245238781,
+            -0.011159304529428482,
+            -0.002761491807177663,
+            0.009119244292378426,
+            -0.010592620819807053,
+            0.0006821996648795903,
+            0.016346637159585953,
+            -0.011717269197106361,
+            -0.0051960512064397335,
+            0.015535843558609486,
+            -0.005008609499782324,
+            0.0007977159111760557,
+            0.0016891524428501725,
+            0.013086027465760708,
+            0.0034480507019907236,
+            -0.0063730087131261826,
+            -0.01591944508254528,
+            0.01215317938476801,
+            -0.00417166156694293,
+            0.0016074193408712745,
+            0.00708354264497757,
+            -0.007933568209409714,
+            -0.0047819362953305244,
+            0.013373728841543198,
+            -0.00854384247213602,
+            0.0016259454423561692,
+            -0.007606634870171547,
+            -0.017549749463796616,
+            -0.0076938169077038765,
+            0.006394804455339909,
+            -0.026730021461844444,
+            -0.013478347100317478,
+            0.016233300790190697,
+            -0.003570105414837599,
+            0.001546391868032515,
+            -0.0005138292908668518,
+            -0.011124431155622005,
+            0.003160349791869521,
+            -0.019598528742790222,
+            -0.01064493041485548,
+            -0.004864759277552366,
+            -0.02662540227174759,
+            0.017549749463796616,
+            0.015091215260326862,
+            0.0033172776456922293,
+            -0.00699636060744524,
+            -0.005592729430645704,
+            0.002567511750385165,
+            0.010348510928452015,
+            0.01763693243265152,
+            0.009311044588685036,
+            -0.005592729430645704,
+            -0.019685709848999977,
+            -0.003443691413849592,
+            0.021324733272194862,
+            -6.375188240781426e-05,
+            -0.016590746119618416,
+            0.020609840750694275,
+            0.007109697442501783,
+            0.014306576922535896,
+            0.0055883703753352165,
+            0.0023451976012438536,
+            0.005383492447435856,
+            0.014114775694906712,
+            -0.0001486181718064472,
+            -0.01783744990825653,
+            -0.013984003104269505,
+            0.0004890369018539786,
+            0.03140297904610634,
+            0.00708354264497757,
+            0.017139993607997894,
+            -0.03623286634683609,
+            0.0006642183288931847,
+            -0.005335542373359203,
+            -0.0038490882143378258,
+            -0.0045988536439836025,
+            0.027671586722135544,
+            0.010766984894871712,
+            -0.0001184450084110722,
+            0.015605589374899864,
+            -0.010043373331427574,
+            0.0422309935092926,
+            -0.015378915704786777,
+            -0.00737560261040926,
+            -0.010052092373371124,
+            -0.01685229316353798,
+            -0.003940629307180643,
+            -0.0011769578559324145,
+            -0.01689588464796543,
+            -0.006647632457315922,
+            -0.006320699583739042,
+            0.021411914378404617,
+            0.0046380856074392796,
+            -0.01665177382528782,
+            0.0017065888969227672,
+            -0.023835577070713043,
+            -0.0010946798138320446,
+            -0.022475536912679672,
+            -0.006403522565960884,
+            -0.012327543459832668,
+            -0.01349578332155943,
+            -0.020993441343307495,
+            -0.02031342126429081,
+            -0.004908350296318531,
+            0.016547156497836113,
+            0.00532246520742774,
+            -0.0069091785699129105,
+            -0.0021043571177870035,
+            -0.014402477070689201,
+            -0.010278765112161636,
+            0.02144678682088852,
+            0.015605589374899864,
+            0.012432162649929523,
+            0.01332141924649477,
+            0.01685229316353798,
+            0.0037597264163196087,
+            -0.034593842923641205,
+            0.006128899287432432,
+            0.0041607641614973545,
+            -0.005435801576822996,
+            0.01782873272895813,
+            0.000511104823090136,
+            -0.010793139226734638,
+            0.013399883173406124,
+            -0.007593557704240084,
+            0.018098996952176094,
+            -0.003212659154087305,
+            -0.004184739198535681,
+            -0.005536061245948076,
+            -0.008740002289414406,
+            -0.007371243555098772,
+            0.00455090356990695,
+            0.0021043571177870035,
+            0.013643993064761162,
+            0.0005775811732746661,
+            -0.004625008441507816,
+            -0.011333668604493141,
+            0.0014602995943278074,
+            -0.0032257363200187683,
+            -0.001628125086426735,
+            -0.023835577070713043,
+            -0.012170616537332535,
+            0.0017120378324761987,
+            -0.0086179468780756,
+            -0.01235369872301817,
+            0.02856084331870079,
+            -0.0017011400777846575,
+            -0.0008009852026589215,
+            0.010688520967960358,
+            0.015675334259867668,
+            0.001266319421119988,
+            0.01837797835469246,
+            -0.012170616537332535,
+            -0.011185458861291409,
+            -0.01649484597146511,
+            -0.01282448135316372,
+            -0.01122033130377531,
+            0.02090625837445259,
+            0.0033979208674281836,
+            -0.00708354264497757,
+            0.010418256744742393,
+            -0.011106994934380054,
+            -0.005056559573858976,
+            -0.0091018071398139,
+            0.011708551086485386,
+            -0.02969421073794365,
+            0.02069702185690403,
+            -0.012711144983768463,
+            0.006024280562996864,
+            0.018639525398612022,
+            0.039231929928064346,
+            -0.006608400493860245,
+            -0.006869946606457233,
+            0.010034655220806599,
+            -0.003068808699026704,
+            0.029205990955233574,
+            -0.0059414575807750225,
+            0.004237048327922821,
+            -0.021551406010985374,
+            0.0018449904164299369,
+            -0.01074954867362976,
+            0.0012815763475373387,
+            0.01991238258779049,
+            -0.0191974900662899,
+            -0.013722456991672516,
+            -0.00675660977140069,
+            -0.016058936715126038,
+            -0.00823870487511158,
+            0.022283734753727913,
+            0.011377259157598019,
+            -0.006111462600529194,
+            -0.008273578248918056,
+            -0.005736579652875662,
+            -0.024201741442084312,
+            0.014716332778334618,
+            0.08174190670251846,
+            -0.009982346557080746,
+            -0.0001858067698776722,
+            0.0003367407189216465,
+            0.0017806936521083117,
+            0.007907412946224213,
+            -0.00031821453012526035,
+            0.05956278741359711,
+            -0.003509078174829483,
+            0.003899217816069722,
+            -0.012998845428228378,
+            0.01083673071116209,
+            -0.01438504084944725,
+            -0.005684270523488522,
+            -0.009712082333862782,
+            -0.01197881530970335,
+            -0.0008167869527824223,
+            -0.02203962579369545,
+            -0.001063076313585043,
+            -0.011996251530945301,
+            -0.005021686665713787,
+            -0.021324733272194862,
+            0.017514877021312714,
+            -0.02087138593196869,
+            -0.01648612879216671,
+            -0.004594494588673115,
+            -0.010705957189202309,
+            0.012118306942284107,
+            -0.016555873677134514,
+            0.002380070276558399,
+            0.007270984351634979,
+            0.014228112995624542,
+            0.007475861813873053,
+            -0.008199472911655903,
+            -0.008408710360527039,
+            -0.0076197125017642975,
+            0.004986814223229885,
+            0.0069876424968242645,
+            -0.0011088468600064516,
+            0.014637868851423264,
+            -0.02088882215321064,
+            -0.016058936715126038,
+            0.007589198648929596,
+            0.0017817834159359336,
+            0.009502844884991646,
+            -0.007820230908691883,
+            0.007449707482010126,
+            0.00020882829267065972,
+            0.022754518315196037,
+            -0.010156710632145405,
+            -0.0268520750105381,
+            -0.007933568209409714,
+            0.008644101209938526,
+            -0.013138337060809135,
+            -0.006102744489908218,
+            0.005331183318048716,
+            0.009807982482016087,
+            0.008081777021288872,
+            -0.009877728298306465,
+            0.0007110786973498762,
+            -0.018918508663773537,
+            0.013373728841543198,
+            0.003238813718780875,
+            0.022196553647518158,
+            -0.012667553499341011,
+            0.010880321264266968,
+            -0.02683463878929615,
+            -0.002574050333350897,
+            -0.014419913291931152,
+            0.0008592882077209651,
+            0.008434864692389965,
+            0.004742704331874847,
+            0.01684357412159443,
+            -0.017192302271723747,
+            -0.017096402123570442,
+            0.009311044588685036,
+            -0.01782873272895813,
+            -0.02376583032310009,
+            -0.0018471699440851808,
+            -0.02624180167913437,
+            -0.0038163948338478804,
+            0.007532530464231968,
+            -0.019389290362596512,
+            -0.01369630265980959,
+            0.0065953233279287815,
+            -0.0033412526827305555,
+            -0.014611713588237762,
+            0.017183585092425346,
+            -0.011970097199082375,
+            -0.03544822707772255,
+            0.003727033268660307,
+            0.008382555097341537,
+            -0.004729627165943384,
+            0.006747891660779715,
+            -0.024410977959632874,
+            0.016015345230698586,
+            -0.002397506730630994,
+            0.004293716512620449,
+            0.008029468357563019,
+            0.02606743760406971,
+            0.0010309278732165694,
+            0.0034545892849564552,
+            -0.005544779356569052,
+            0.014411195181310177,
+            0.005801966413855553,
+            0.023504285141825676,
+            0.026416165754199028,
+            0.0015529305674135685,
+            -0.010958785191178322,
+            -0.01081929448992014,
+            0.01592816226184368,
+            -0.011586496606469154,
+            -0.0048386044800281525,
+            0.022737082093954086,
+            0.013827075250446796,
+            -0.02203962579369545,
+            0.00017259323794860393,
+            -0.02149909734725952,
+            0.0069876424968242645,
+            -0.017148712649941444,
+            0.009467972442507744,
+            0.008731283247470856,
+            -0.005073996260762215,
+            0.02069702185690403,
+            0.010095682926476002,
+            0.030984506011009216,
+            0.01991238258779049,
+            0.002920599188655615,
+            0.011412132531404495,
+            -0.006451472640037537,
+            0.010470565408468246,
+            0.004537826403975487,
+            0.0022275017108768225,
+            -0.006608400493860245,
+            0.006930973846465349,
+            0.0023451976012438536,
+            -0.001170419156551361,
+            0.008583074435591698,
+            -0.007471502758562565,
+            0.009921318851411343,
+            0.0005939278053119779,
+            0.0055273426696658134,
+            -0.004287178162485361,
+            -0.010575184598565102,
+            -0.030339358374476433,
+            -0.003132015699520707,
+            0.020958568900823593,
+            -0.023539157584309578,
+            -0.008003313094377518,
+            0.005932739470154047,
+            0.016983065754175186,
+            -0.008282296359539032,
+            -0.012798327021300793,
+            -0.01708768494427204,
+            0.003105861134827137,
+            -0.0004893093137070537,
+            -0.010313638485968113,
+            0.01747128553688526,
+            0.0009546435903757811,
+            -0.004951941315084696,
+            0.008953598327934742,
+            -0.00680456031113863,
+            0.028212115168571472,
+            0.01505634281784296,
+            0.011150586418807507,
+            -0.012100870721042156,
+            0.014201958663761616,
+            -0.012711144983768463,
+            0.007680739741772413,
+            0.009799264371395111,
+            -0.006416599731892347,
+            -0.021220114082098007,
+            -0.017558468505740166,
+            0.023451974615454674,
+            0.006848151329904795,
+            0.012222925201058388,
+            -0.011987533420324326,
+            -0.0012815763475373387,
+            0.007475861813873053,
+            0.004834245424717665,
+            -0.01611996442079544,
+            -0.00048631246318109334,
+            0.0061986446380615234,
+            0.038604218512773514,
+            0.005087073426693678,
+            0.01072339341044426,
+            -0.0035984397400170565,
+            -0.0002066487359115854,
+            0.019162617623806,
+            -0.01952878199517727,
+            -0.006503781769424677,
+            -0.005832480266690254,
+            -0.02739260531961918,
+            0.016573309898376465,
+            -0.01685229316353798,
+            -0.016451256349682808,
+            0.010976221412420273,
+            -0.00041356991278007627,
+            -0.004422310274094343,
+            -0.01284191757440567,
+            -0.008766156621277332,
+            0.00794228632003069,
+            0.010810576379299164,
+            -0.009136680513620377,
+            0.00532246520742774,
+            0.00357446470297873,
+            -0.0035984397400170565,
+            0.02127242274582386,
+            0.0006467819330282509,
+            -0.002837776206433773,
+            0.007358166389167309
+          ],
+          "title": "Method",
+          "keyphrases": [
+            "mixture-of-experts (MoE) models",
+            "token-choice routing",
+            "load imbalance",
+            "expert choice method",
+            "Transformer architecture"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "4074001f-c9f1-47e9-8917-1f132c0cd526",
+      "type": "child",
+      "source": {
+        "id": "b570d71e-4f83-486e-b70f-b0c6217db455",
+        "properties": {
+          "page_content": "Mixture-Of-Experts With Expert Choice Routing\n\nYanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon Google, Mountain View, CA, USA {yanqiz, taole, hanxiaol, dunan, huangyp, vzhao, adai, zhifengc, qvl, jlaudon}@google.com\n\nAbstract\n\nSparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choice method. Instead of letting tokens select the top-k experts, we have experts selecting the top-k tokens. As a result, each token can be routed to a variable number of experts and each expert can have a fixed bucket size. We systematically study pre-training speedups using the same computational resources of the Switch Transformer top-1 and GShard top-2 gating of prior work and find that our method improves training convergence time by more than 2×. For the same computational cost, our method demonstrates higher performance in fine-tuning 11 selected tasks in the GLUE and SuperGLUE benchmarks. For a smaller activation cost, our method outperforms the T5 dense model in 7 out of the 11 tasks.\n\n1 Introduction\n\nScaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27]. The final model quality has been found to have a power-law relationship with the amount of data, model size, and compute time [16, 20]. However, training efficiency, which is defined as the total amount of computation used to achieve superior model quality than the state of the art system [21], should receive greater attention as we increase our efforts towards green AI [29].\n\nSparsely gated mixture-of-experts [31] (MoE) provides an effective way to scale model capacity given a fixed computational cost, and has recently played an important role in increasing the training efficiency of large-scale language models [10, 21]. MoE operate by adopting a number of experts, each as a sub-network, and by activating only one or a few experts for each input token. A gating network must be chosen and optimized in order to route each token to the most suited expert(s). For example, recent work has implemented sparse routing via k-means clustering [12], linear assignment to maximize token-expert affinities [22], or hashing [8, 28]. Many of the prior work use a routing strategy concerning the token choice, where each token selects the best one or two experts.\n\nWe argue that the independent token choice of prior work often leads to an imbalanced load of experts, which causes training inefficiency and sub-optimal training of the model. In order to mitigate this\n\nissue, previous sparsely gated networks introduce additional auxiliary losses as regularization to prevent too many tokens being routed to a single expert, but the effectiveness is still limited. Recent approaches [8, 22, 28] explore alternative strategies for routing, but they focus on pre-training only and do not demonstrate performance gain on downstream tasks. Moreover, none of the previous methods consider allocating a variable number of experts to each token based on importance, which can be beneficial.\n\nWe propose a very simple yet effective routing method we are calling expert choice. Unlike conventional MoE where tokens select one or two top-scoring experts, our method lets each expert pick the top-k tokens. Our method guarantees perfect load balancing, allows a variable number of experts for each token, and achieves substantial gains in training efficiency and downstream performance as demonstrated in our experiments. Our major contributions include:\n\nWe identify common pitfalls in conventional MoE such as load imbalance as described in Section 3.1. We then propose a heterogeneous, expert choice method to provide a fluid allocation of model parameters based on a learnt token-to-expert importance. This method intrinsically guarantees load balance without imposing an auxiliary loss.\n\nWe show our method provides over 2× faster training convergence in a 8B/64E (8 billion activated parameters, 64 experts) model, compared to the top-1 and top-2 gating counterparts in Switch Transformer [10] and GShard [21].\n\nWe show our method demonstrates strong scaling when increasing the number of experts from 16 to 128, evaluated in training perplexity.\n\nWe show our method demonstrates strong performance on downstream tasks selected from GLUE and SuperGLUE at all the evaluated scales. More specifically, our 8B/64E model outperforms a T5 11B dense model in 7 out of 11 tasks evaluated.\n\n2 Related Work\n\nScaling: Various approaches have been proposed to scale up neural network capacity to improve performance. Recent works have successfully scaled models to billions of parameters via various forms of model parallelism [2, 21, 26, 27, 33]. Model parallelism [30] splits weights and tensors across multiple cores while pipeline parallelism [18, 24] splits different layers across devices with micro-batches pipelined to the different layers. To enable continued scaling of neural networks, improving model training and serving efficiency has become a critical research area.\n\nConditional Computation: Computation decisions can be made dynamically based on the input [23, 25]. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation, by activating certain parameters and computation on demand, on a per-example or per-token basis [3]. Conditional convolution layers [1] with task-specific gating has been used to combat catastrophic forgetting when a sequence of learning problems are optimized. The gating decisions may be binary or sparse and continuous, stochastic or deterministic.\n\nMixture of Experts: Sparsely-gated MoE [31] is the first model to demonstrate massive improvements in model capacity, training time, or model quality with gating. Switch Transformer [10] simplifies the gating by selecting only the top expert per token using a softmax over the hidden state and demonstrates better scaling than previous work. All the prior work requires an auxiliary loss to explicitly encourage balancing. This loss term has to be carefully weighted to not overwhelm the primary loss. However, auxiliary loss does not guarantee balancing and a hard capacity factor has to be imposed. As a result, many tokens can still be unprocessed by the MoE layer. Hard MoE [12] with a single decoding layer can be efficiently trained to good effect on large scale hashtag prediction tasks.\n\nBase Layers [22] formulate a linear assignment that maximizes token-expert affinities while ensuring each expert receives an equal number of tokens. Hash layers [8, 28] devise hashing techniques on input tokens. However, the evaluations are limited to pre-training perplexity. THOR [? ] randomly activates experts during training and inference and is trained with a consistency regularization loss.\n\nTHOR has demonstrated strong performance on translation tasks. Different from these prior works, our method is a learnt method that enables heterogeneous MoE and effectively improves downstream fine-tuning performance.\n\n3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n5 Conclusion\n\nWe propose a new routing method for sparsely activated mixture-of-experts (MoE) models. This method addresses load imbalance and under-utilization of experts in conventional MoE methods, and enables selecting different numbers of experts for each token. Our model demonstrates more than 2x training efficiency improvements when compared to the state-of-the-art GShard and Switch Transformer models, and also achieves strong gains when finetuning on 11 datasets in the GLUE and SuperGLUE benchmark.\n\n6 Limitations\n\nThe expert choice method might not immediately apply to auto-regressive text generation as our current implementation takes in the past and future tokens to perform the top-k selection. One possible solution is to collect a large batch of input sequences, dispatch tokens of the same sequence into separate groups, and perform expert choice routing for each group. Another scenario where the expert choice method does not immediately apply is when the batch size becomes very small during serving or inference. A global top-k can be selected instead and we can cap the number of times each expert or token gets selected. We leave these possible improvements for future work.\n\nAnother long-standing issue with MoE has been the large memory footprint. Even though computational cost can be reduced using sparsely gated networks, the total number of parameters increases linearly or sub-linearly with the number of experts. Increasing the number of experts requires reservation of a large number of hardware devices. Therefore, dynamic (used) power is saved while static (reserved) power is not. Power saving techniques such as the ability to put hardware devices into low power states while not in use [17] can help with reducing the reserved power requirements.\n\nReferences\n\n[1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pages 3930–3939. Computer Vision Foundation / IEEE, 2020.\n\n[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. In Advances in Neural Information Processing Systems.\n\n[3] Kyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning, 2014.\n\n[4] Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems, 2021.\n\n[5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.\n\nTransformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.\n\n[6] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 933–941. JMLR.org, 2017.\n\n[7] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixtureof-experts, 2021.\n\n[8] Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan.\n\nTricks for training sparse translation models, 2021.\n\n[9] Richard L Dykstra. An iterative procedure for obtaining i-projections onto the intersection of convex sets. The annals of Probability, pages 975–984, 1985.\n\n[10] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.\n\n[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. NAS-FPN: learning scalable feature pyramid architecture for object detection. In CVPR, pages 7036–7045. Computer Vision Foundation / IEEE, 2019.\n\n[12] Sam Gross, Marc'Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision, 2017.\n\n[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.\n\n[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 630–645, Cham, 2016. Springer International Publishing.\n\n[15] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2016.\n\n[16] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.\n\n[17] Ping Huang, Zuocheng Xing, Tianran Wang, Qiang Wei, Hongyan Wang, and Guitao Fu. A brief survey on power gating design. In 2010 10th IEEE International Conference on Solid-State and Integrated Circuit Technology, pages 788–790, 2010.\n\n[18] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 103–112, 2019.\n\n[19] Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David A. Patterson. A domain-specific supercomputer for training deep neural networks. Commun. ACM, 63(7):67–78, 2020.\n\n[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.\n\n[21] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.\n\n[22] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6265–6274. PMLR, 18–24 Jul 2021.\n\n[24] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. New York, NY, USA, 2019. Association for Computing Machinery.\n\n[25] Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, Cédric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. In ICLR. OpenReview.net, 2021.\n\n[26] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.\n\n[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.\n\n[28] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models, 2021.\n\n[30] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 10435–10444, Red Hook, NY, USA, 2018. Curran Associates Inc.\n\n[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E.\n\nHinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In ICLR (Poster). OpenReview.net, 2017.\n\n[32] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 10–15 Jul 2018. [23] Min Lin, Jie Fu, and Yoshua Bengio. Conditional computation for continual learning, 2019.\n\n[29] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.\n\n[33] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.\n\n[34] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc.\n\n[35] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.\n\nGLUE: A multi-task benchmark and analysis platform for natural language understanding.\n\nIn Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, November 2018. Association for Computational Linguistics.\n\n[36] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake A. Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen. GSPMD: general and scalable parallelization for ML computation graphs. CoRR, abs/2105.04663, 2021.\n\n7 Checklist\n\n(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Yes (b) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes (c) Did you discuss any potential negative societal impacts of your work? N/A. Not any. (d) Did you describe the limitations of your work? Yes (a) Did you include the code, data, and instructions needed to reproduce the main experimental results? Yes. We include details in the experiment setup to help reproduce the main results. (b) Did you specify all the training details? Yes (c) Did you report error bars? Yes (d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? Yes (a) If your work uses existing assets, did you cite the creators? Yes (b) Did you mention the license of the assets? No. The used dataset is not released yet. (c) Did you include any new assets either in the supplemental material or as a URL? No. The dataset is not released yet.\n\n(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? No. Not using persons' data. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes. The dataset does not contain any personally identifiable information or offensive content.\n\nA Comparison On Fine-Tuning With A Dense Model\n\nOur 8B MoE model achieves stronger pre-training perplexity than its dense counterpart. However, a better perplexity does not always directly translate to downstream performance as demonstrated in Section 4.4. To this end, we compare fine-tuning performance of the 8B dense model and MoE model in Table 1. As shown in the table, our MoE model using expert choice routing consistently outperforms the dense model across the 11 tasks in GLUE and SuperGLUE.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 1: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nB Capacity Factor\n\nWe evaluate the downstream task fine-tuning performance by varying the capacity factors. Note that a capacity factor of n indicates on average how many experts each token can be received. EC-CF2 is our baseline expert choice, which matches GShard top-2 gating computational footprint. EC-CF1, however, matches Switch Transformer top-1 gating computational footprint. EC-CF0.5 further verifies that an aggressively lowered capacity factor can provide strong enough performance, that almost matches the top-2 gating baseline.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Top-2 78.1 87.0 88.3 85.0 82.6 90.1 90.7 81.6 94.7 68.2 67.2 83.0±0.3 EC-CAP2 78.2 88.0 88.5 85.7 83.0 90.8 91.1 80.0 95.4 70.4 64.1 83.2±0.4 EC-CAP3 78.5 91.7 89.3 86.3 83.5 90.9 91.1 81.8 94.9 70.0 65.6 84.0±0.4 EC-CF2 79.1 89.6 89.3 86.8 84.3 91.3 91.2 81.1 95.2 68.1 68.0 84.0±0.2 EC-CF1 77.4 90.6 88.0 85.5 83.6 90.3 91.2 79.8 95.3 66.5 64.9 83.0±0.2 EC-CF0.5 77.4 89.6 86.3 85.2 82.7 91.7 91.0 79.6 94.9 67.3 63.5 83.0 ±0.05 Hash Layers 76.1 85.2 86.7 83.4 82.5 90.0 90.3 75.7 94.0 67.4 63.3 81.3±1.0\n\nTable 2: Comparison between different routing methods in fine-tuning of 100M/64E models. We perform 3 independent fine-tuning runs for each method and report the average results. This gives more accurate difference between the variants of expert choice method, since they achieve close fine-tuning results. We do not report averaged results in other experiments.\n\nC Capped Expert Choice\n\nAs described in Section 4.5, the maximum number of experts each token is assigned can be capped by an entropy-regularized linear programming. Figure 1 compares the validation perplexity when training the 100M/64E models using the base expert choice method (EC-BASE), expert choice capped by two experts per token (EC-CAP2), expert choice capped by three experts per token (EC-CAP3), and GShard top-2 gating.\n\nAs shown in the figure, restricting the number of experts to 2 degrades the perplexity compared to the base expert choice method. This suggests that a more flexible allocation of experts (e.g. more than 2 experts for a token) can enhance model expressiveness. On the other hand, our EC-CAP2 and EC-CAP3 methods still outperform the top-2 gating method by a clear margin. We believe this confirms the effectiveness of a load balanced training, provided by our method. Finally, EC-CAP3 obtains comparable perplexity to EC-BASE. As indicated by Figure 3, only a little fraction of tokens use more than 3 experts therefore we see little or no difference between EC-BASE and EC-CAP3 variants. We present the fine-tuning results of these methods in Table 2.\n\nD Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [? ]. We use mod x to map a token ID to an expert ID. This in some way ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 2. Hashing based routing performs much worse than expert choice in terms of average scores and variance.\n\nE Fine-Tuning Details\n\nWe did a hyperparameter search for both baseline models and expert choice method. For fine-tuning of the 8B dense model, we use a constant learning rate of 0.0001 and a dropout rate of 0.1. We freeze the attention layer and feed-forward layer while leaving the embedding and layer normalization trainable. This setting has been found optimal for the 8B dense model. For MoE 8B/64E models including GShard top-2 gating and expert choice, we found continuing the learning rate from the pre-trained model while using a square root learning rate decay works better. In addition, we do not apply parameter freezing for fine-tuning MoE models. For models with 100M expert size, we use a constant learning rate of 0.0001 and no dropout is used.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/2202.09368v2.md"
+          },
+          "headlines": {
+            "1 Introduction": [],
+            "2 Related Work": [],
+            "3 Method": [
+              "3.1 Pitfalls Of Token-Choice Routing",
+              "3.2 Heterogeneous Moe Via Expert Choice",
+              "3.3 Expert Choice With Additional Constraint",
+              "3.4 Model Architecture"
+            ],
+            "4 Experiments": [
+              "4.1 Setup",
+              "4.2 Training Efficiency",
+              "4.3 Scaling The Number Of Experts",
+              "4.4 Fine-Tuning On Glue And Superglue",
+              "4.5 Heterogeneity Matters",
+              "4.6 Comparison With Hash Layer",
+              "4.7 Ablation"
+            ],
+            "5 Conclusion": [],
+            "6 Limitations": [],
+            "A Comparison On Fine-Tuning With A Dense Model": [],
+            "B Capacity Factor": [],
+            "C Capped Expert Choice": [],
+            "D Comparison With Hash Layer": [],
+            "E Fine-Tuning Details": []
+          },
+          "summary": "The paper introduces a new routing method for sparsely-activated Mixture-of-Experts (MoE) models called 'expert choice,' which addresses load imbalance and under-utilization issues in conventional MoE methods. Unlike traditional token-choice routing, expert choice allows experts to select the top-k tokens, ensuring perfect load balancing and enabling variable expert allocation per token. This method improves training efficiency, achieving over 2x faster convergence compared to existing models like Switch Transformer and GShard. It also demonstrates superior performance on downstream tasks in the GLUE and SuperGLUE benchmarks. The study highlights the benefits of flexible expert allocation and shows that expert choice outperforms hashing-based routing methods. The paper also discusses potential limitations and future improvements for the expert choice method.",
+          "summary_embedding": [
+            -0.009726986289024353,
+            -0.0337262786924839,
+            -0.023302342742681503,
+            -0.019878201186656952,
+            -0.025953780859708786,
+            0.04560471698641777,
+            -0.006874797400087118,
+            0.039665497839450836,
+            -0.006973279174417257,
+            0.034544438123703,
+            -0.025787118822336197,
+            -0.00363057479262352,
+            -0.01052999310195446,
+            0.0054657477885484695,
+            -0.024499278515577316,
+            0.036786794662475586,
+            0.010007280856370926,
+            0.003272630739957094,
+            0.006102093029767275,
+            -0.04048365354537964,
+            -0.018666116520762444,
+            -0.013522328808903694,
+            -0.03230207785964012,
+            0.008969432674348354,
+            0.02484775334596634,
+            0.0016713523073121905,
+            0.0026230288203805685,
+            0.003624893259257078,
+            -0.05311964824795723,
+            0.04339266195893288,
+            0.018681267276406288,
+            0.023711422458291054,
+            -0.016332851722836494,
+            -0.012832955457270145,
+            -0.039120059460401535,
+            -0.0197418425232172,
+            0.030786970630288124,
+            0.039968520402908325,
+            -0.019287310540676117,
+            0.03357476741075516,
+            0.018060073256492615,
+            0.02454473078250885,
+            -0.021272100508213043,
+            -0.029817303642630577,
+            -0.0061892117373645306,
+            -0.007840678095817566,
+            -0.008355814032256603,
+            0.01242387667298317,
+            0.016984347254037857,
+            -0.013007191941142082,
+            0.0212569497525692,
+            -0.01371171697974205,
+            0.03384748846292496,
+            0.008916404098272324,
+            -0.042332086712121964,
+            0.03984731063246727,
+            -0.008007340133190155,
+            0.012105703353881836,
+            0.03281721472740173,
+            0.010333028621971607,
+            -0.005287723150104284,
+            -0.021544819697737694,
+            -0.03378688171505928,
+            0.0010529992869123816,
+            0.00014014738553669304,
+            0.005950582213699818,
+            -0.003069985192269087,
+            -0.009742137044668198,
+            0.03227177634835243,
+            -0.003939277958124876,
+            -0.0740584209561348,
+            0.050089433789253235,
+            -0.026590125635266304,
+            -0.007628562860190868,
+            -0.028938541188836098,
+            -0.01736312545835972,
+            0.037362534552812576,
+            0.005662711802870035,
+            0.0098709212616086,
+            0.0106284748762846,
+            -0.0007892759749665856,
+            0.028923390433192253,
+            0.00910579226911068,
+            -0.024923507124185562,
+            0.04478655755519867,
+            -0.04790768027305603,
+            0.01801462098956108,
+            -0.04336235672235489,
+            0.022756904363632202,
+            1.6393929399782792e-05,
+            -0.020726662129163742,
+            -0.0013304532039910555,
+            -0.016666175797581673,
+            0.0218023881316185,
+            -0.01730252057313919,
+            0.026802239939570427,
+            0.013825350441038609,
+            -0.015802564099431038,
+            -0.029817303642630577,
+            0.025378039106726646,
+            0.003367325058206916,
+            0.010499690659344196,
+            -0.02075696364045143,
+            0.014332910999655724,
+            0.04393810033798218,
+            -0.02593863010406494,
+            -0.02271145209670067,
+            0.027681002393364906,
+            0.014408666640520096,
+            0.0633314698934555,
+            0.036544378846883774,
+            0.006624804809689522,
+            0.022362977266311646,
+            0.020575150847434998,
+            -0.028968842700123787,
+            -0.01963578537106514,
+            -0.046362269669771194,
+            -0.04757435619831085,
+            0.019878201186656952,
+            -0.04566532000899315,
+            0.0050869714468717575,
+            0.038544319570064545,
+            -0.013704141601920128,
+            -0.014446543529629707,
+            -0.04021093621850014,
+            0.017847958952188492,
+            0.0003908502112608403,
+            0.011976920068264008,
+            -0.0022082682698965073,
+            0.00033285003155469894,
+            -0.011825408786535263,
+            0.008878526277840137,
+            -0.03778676688671112,
+            -0.025196226313710213,
+            0.025181075558066368,
+            0.022862963378429413,
+            0.024120500311255455,
+            0.028802180662751198,
+            -0.03077181987464428,
+            -0.009984553791582584,
+            0.021378157660365105,
+            -0.04242299124598503,
+            -0.006556625012308359,
+            0.011332998983561993,
+            0.0026703758630901575,
+            0.003872992005199194,
+            0.032786913216114044,
+            0.052998438477516174,
+            -0.03172633796930313,
+            0.02065090648829937,
+            -0.010431510396301746,
+            0.04957429692149162,
+            0.00024691506405360997,
+            0.028968842700123787,
+            -0.019514575600624084,
+            -0.0134389977902174,
+            -0.029529431834816933,
+            -0.009795165620744228,
+            -0.01356020662933588,
+            -0.008484598249197006,
+            -0.002998017705976963,
+            -0.009575475007295609,
+            0.0040453351102769375,
+            0.0331808403134346,
+            0.00273855566047132,
+            0.030680913478136063,
+            0.014325335621833801,
+            0.0051475754007697105,
+            -0.08745196461677551,
+            0.026059838011860847,
+            -0.0033616432920098305,
+            0.005859675817191601,
+            0.0062460280023515224,
+            0.03013547509908676,
+            -0.0312718041241169,
+            0.020938776433467865,
+            0.026620427146553993,
+            0.0011145505122840405,
+            0.016969196498394012,
+            0.036847397685050964,
+            -0.04181694984436035,
+            0.009643655270338058,
+            0.012590537779033184,
+            -0.0033881578128784895,
+            0.04584713280200958,
+            0.011628445237874985,
+            0.019514575600624084,
+            0.018529755994677544,
+            0.003569970605894923,
+            0.00297907879576087,
+            0.006533898413181305,
+            0.03724132478237152,
+            0.003462019143626094,
+            -0.009833043441176414,
+            0.014976831153035164,
+            -0.010757259093225002,
+            -0.011219366453588009,
+            -0.037998881191015244,
+            0.014454119838774204,
+            0.008742166683077812,
+            -0.017454031854867935,
+            -0.02959003672003746,
+            0.0026949963066726923,
+            -0.031908150762319565,
+            0.026559824123978615,
+            -0.002445003716275096,
+            0.005738467443734407,
+            -0.010454237461090088,
+            -0.006757376715540886,
+            -0.035423196852207184,
+            0.028453707695007324,
+            -0.02095392718911171,
+            0.0053142374381423,
+            0.0007258309051394463,
+            -0.029438525438308716,
+            0.0060073984786868095,
+            0.017863109707832336,
+            -0.0025965142995119095,
+            0.0025302283465862274,
+            -0.02349930815398693,
+            0.024468975141644478,
+            0.025635607540607452,
+            0.004518806003034115,
+            -0.014037464745342731,
+            -0.013113250024616718,
+            -0.003605954349040985,
+            -0.009742137044668198,
+            -0.016196493059396744,
+            0.024468975141644478,
+            -0.014810169115662575,
+            0.00308892410248518,
+            0.021847840398550034,
+            0.014264730736613274,
+            -0.007045246660709381,
+            -0.004007457755506039,
+            -0.04066546633839607,
+            -0.01218903437256813,
+            -0.04760465770959854,
+            -0.011628445237874985,
+            -0.0034601252991706133,
+            -0.028696123510599136,
+            0.0061513339169323444,
+            -0.04481685906648636,
+            -0.01235569640994072,
+            0.02309022843837738,
+            0.018135828897356987,
+            0.0019488062243908644,
+            -0.009173971600830555,
+            -0.022605394944548607,
+            8.486965816700831e-05,
+            -0.01763584464788437,
+            -0.017044952139258385,
+            0.015393485315144062,
+            0.031059689819812775,
+            -0.03101423755288124,
+            0.04024123772978783,
+            0.003893824527040124,
+            -0.08066429197788239,
+            0.010393633507192135,
+            -0.026362858712673187,
+            0.07011914253234863,
+            -0.0025453795678913593,
+            0.014075342565774918,
+            0.06781618297100067,
+            -0.012431452050805092,
+            -0.02007516659796238,
+            0.03142331540584564,
+            -0.008249756880104542,
+            -1.1311515663692262e-05,
+            -0.02884763479232788,
+            -0.02739313244819641,
+            -0.053271155804395676,
+            -0.009704259224236012,
+            -0.010787560604512691,
+            0.03360506892204285,
+            -0.024317465722560883,
+            -0.0007656024536117911,
+            0.012620840221643448,
+            0.02983245439827442,
+            0.016635872423648834,
+            -0.0015681355725973845,
+            0.008272483013570309,
+            0.02881733328104019,
+            0.0274234339594841,
+            -0.0036154238041490316,
+            0.00814369972795248,
+            -0.034211114048957825,
+            -0.0068066176027059555,
+            -0.0493621826171875,
+            0.011302697472274303,
+            -0.02596893161535263,
+            -0.02596893161535263,
+            0.007643714081496,
+            -0.011476934887468815,
+            0.03748374432325363,
+            -0.024696242064237595,
+            -0.007503566797822714,
+            0.005037730094045401,
+            0.012855681590735912,
+            -0.017757052555680275,
+            -0.03612014651298523,
+            -0.0028502948116511106,
+            0.0026381798088550568,
+            0.007560383062809706,
+            -0.053240854293107986,
+            0.004499867558479309,
+            -0.018469152972102165,
+            0.0279688723385334,
+            -0.02139330841600895,
+            -0.013688990846276283,
+            0.042029064148664474,
+            0.011363301426172256,
+            0.027499189600348473,
+            -0.026468917727470398,
+            0.0017849853029474616,
+            0.012825379148125648,
+            -0.0020056227222085,
+            0.032514192163944244,
+            -0.030574856325984,
+            -0.03257479891180992,
+            0.001548249856568873,
+            -0.007840678095817566,
+            -0.0625436082482338,
+            -0.011386028490960598,
+            -0.00981789268553257,
+            0.017317671328783035,
+            0.0446956530213356,
+            0.04557441547513008,
+            -0.01068150345236063,
+            0.002174178371205926,
+            -0.02213571034371853,
+            -0.04542290419340134,
+            -0.02643861435353756,
+            0.0016486257081851363,
+            -0.019999410957098007,
+            -0.02789311669766903,
+            -0.0018313854234293103,
+            0.011211791075766087,
+            -0.013651113025844097,
+            0.01134057529270649,
+            -0.018560059368610382,
+            -0.01862066239118576,
+            -0.012332970276474953,
+            0.018575210124254227,
+            0.009901223704218864,
+            -0.0018796793883666396,
+            -0.015037435106933117,
+            -0.00980274099856615,
+            0.018969137221574783,
+            -0.011279970407485962,
+            0.0009388927719555795,
+            0.01188601367175579,
+            0.024347767233848572,
+            -0.007083124481141567,
+            0.010961798019707203,
+            -0.034756552428007126,
+            -0.01966608688235283,
+            0.007984613068401814,
+            0.002719616750255227,
+            0.020711511373519897,
+            0.008742166683077812,
+            -0.04275631532073021,
+            -0.044301725924015045,
+            0.016044981777668,
+            -0.0562104657292366,
+            0.028120383620262146,
+            -0.03327174484729767,
+            -0.012514783069491386,
+            -0.02671133354306221,
+            0.0013418165035545826,
+            -0.01828734017908573,
+            0.022438732907176018,
+            0.03778676688671112,
+            -0.0011486404109746218,
+            0.005503625608980656,
+            0.014658658765256405,
+            0.04384719207882881,
+            -0.05021063983440399,
+            -0.041180603206157684,
+            0.03754434734582901,
+            -0.01601468026638031,
+            0.0185146052390337,
+            0.00021744151308666915,
+            0.010348179377615452,
+            -0.014431392773985863,
+            0.022772056981921196,
+            0.013757170177996159,
+            -0.008764892816543579,
+            -0.030574856325984,
+            0.04945308715105057,
+            -0.023650817573070526,
+            -0.049513693898916245,
+            0.0018323323456570506,
+            0.018878230825066566,
+            0.018635815009474754,
+            -0.014870774000883102,
+            -0.02907489985227585,
+            -0.024317465722560883,
+            -0.006060427520424128,
+            -0.00947699323296547,
+            -0.03402930125594139,
+            0.011363301426172256,
+            0.0048445542342960835,
+            0.016666175797581673,
+            -0.00645056739449501,
+            -0.0073141781613230705,
+            -0.009370936080813408,
+            0.022559940814971924,
+            0.012635990977287292,
+            -0.03175663948059082,
+            -0.017484333366155624,
+            -0.02528713271021843,
+            0.004359719809144735,
+            0.010136065073311329,
+            0.022166013717651367,
+            0.024287162348628044,
+            -0.0323626808822155,
+            0.04218057543039322,
+            0.01550711877644062,
+            -0.008727015927433968,
+            -0.0019317612750455737,
+            0.02478714846074581,
+            0.04324115067720413,
+            0.0218023881316185,
+            0.026514369994401932,
+            0.002107892418280244,
+            8.705945947440341e-05,
+            -0.015620751306414604,
+            -0.0011391709558665752,
+            0.01862066239118576,
+            0.04911976307630539,
+            -0.012151156552135944,
+            0.01716616004705429,
+            -0.03578682243824005,
+            0.00396200455725193,
+            -0.02580226957798004,
+            -0.030786970630288124,
+            0.03745344281196594,
+            -0.015060162171721458,
+            0.0191964041441679,
+            0.025044716894626617,
+            -0.022469034418463707,
+            -0.011075431481003761,
+            -0.019423669204115868,
+            -0.026181045919656754,
+            -0.01702980138361454,
+            -0.03281721472740173,
+            -0.018135828897356987,
+            -0.002653330797329545,
+            -0.035635314881801605,
+            -0.04987731948494911,
+            -0.004329417832195759,
+            -0.046392571181058884,
+            -0.010507266037166119,
+            0.017378276214003563,
+            -0.011166337877511978,
+            -0.025711363181471825,
+            -0.005336964037269354,
+            0.003973367623984814,
+            -0.022196315228939056,
+            -0.0066588944755494595,
+            0.008439145050942898,
+            0.035938333719968796,
+            -0.02298417128622532,
+            -0.013060221448540688,
+            -0.0024317465722560883,
+            -0.02887793630361557,
+            -0.007878555916249752,
+            0.008999735116958618,
+            0.007723257411271334,
+            -0.0012357590021565557,
+            -0.05196816474199295,
+            -0.006916462909430265,
+            -0.012992041185498238,
+            0.029196109622716904,
+            0.026468917727470398,
+            0.0012168202083557844,
+            -0.02783251367509365,
+            0.008749742060899734,
+            -0.0072535742074251175,
+            0.02339324913918972,
+            -0.007992188446223736,
+            -0.022908415645360947,
+            -0.02034788578748703,
+            0.030786970630288124,
+            -0.0159086212515831,
+            0.009113367646932602,
+            0.009976978413760662,
+            0.0121360057964921,
+            -0.03899884968996048,
+            -0.001445980160497129,
+            0.0005648507503792644,
+            -0.010128489695489407,
+            0.00423093605786562,
+            -0.006738437805324793,
+            -0.007079336792230606,
+            -0.022090258076786995,
+            -0.0218023881316185,
+            0.03818069398403168,
+            -0.009113367646932602,
+            -0.0035661826841533184,
+            -0.01088604237884283,
+            0.03436262533068657,
+            -0.00595815759152174,
+            -0.01993880607187748,
+            -0.019347915425896645,
+            0.014431392773985863,
+            0.05105910077691078,
+            -0.028529461473226547,
+            -0.03984731063246727,
+            0.0011136035900563002,
+            0.030150625854730606,
+            0.004140029661357403,
+            -0.016908591613173485,
+            0.0035851215943694115,
+            0.056392278522253036,
+            -0.011742077767848969,
+            0.028362801298499107,
+            0.027771908789873123,
+            0.02295386977493763,
+            -0.02021152526140213,
+            -0.01678738370537758,
+            -0.020999381318688393,
+            0.011098157614469528,
+            -0.03027183562517166,
+            0.027059808373451233,
+            0.018969137221574783,
+            0.01538590993732214,
+            0.05054396390914917,
+            -0.03178694099187851,
+            -0.02515077404677868,
+            0.008499749936163425,
+            0.0021722845267504454,
+            0.011007251217961311,
+            -0.007727045100182295,
+            -0.01801462098956108,
+            -0.001950700068846345,
+            -0.014022313989698887,
+            -0.019817598164081573,
+            -0.020938776433467865,
+            0.00954517349600792,
+            0.015696506947278976,
+            0.002649543108418584,
+            0.010863316245377064,
+            0.0016590419691056013,
+            0.01686313934624195,
+            0.01321930717676878,
+            -0.0444229319691658,
+            -7.936555630294606e-05,
+            0.03378688171505928,
+            -0.0293476190418005,
+            -0.012264790013432503,
+            -0.016060132533311844,
+            -0.004439263138920069,
+            -0.01706010289490223,
+            -0.02762039750814438,
+            -0.006874797400087118,
+            -0.01889338344335556,
+            -0.00814369972795248,
+            -0.028196139261126518,
+            -0.009992130100727081,
+            0.01949942484498024,
+            0.022332675755023956,
+            0.001196934375911951,
+            -0.006783891003578901,
+            -0.005901341326534748,
+            0.02092362567782402,
+            0.032514192163944244,
+            0.01952972821891308,
+            0.0058331615291535854,
+            -0.015158643946051598,
+            -0.004295328166335821,
+            -0.012370847165584564,
+            0.016060132533311844,
+            0.012537509202957153,
+            6.344509893096983e-05,
+            0.03493836522102356,
+            0.004170331638306379,
+            -0.022726602852344513,
+            -0.001230077352374792,
+            0.024923507124185562,
+            -0.021590271964669228,
+            -0.0108330138027668,
+            0.040029123425483704,
+            0.0085679292678833,
+            -0.0024506852496415377,
+            -0.019287310540676117,
+            -0.019075196236371994,
+            0.02630225569009781,
+            -0.02133270539343357,
+            0.002083271974697709,
+            0.0482713058590889,
+            0.00035344602656550705,
+            -0.029802151024341583,
+            0.017954016104340553,
+            -0.013242034241557121,
+            -0.018560059368610382,
+            0.01848430372774601,
+            0.0012859469279646873,
+            -0.039968520402908325,
+            0.019014591351151466,
+            0.018408548086881638,
+            0.022635696455836296,
+            -0.02918095886707306,
+            0.0002127068000845611,
+            0.03233237937092781,
+            0.004037759732455015,
+            0.001595596899278462,
+            -0.008530051447451115,
+            -0.0006065162015147507,
+            0.010431510396301746,
+            0.001746160676702857,
+            0.02854461409151554,
+            -0.0030434709042310715,
+            0.027771908789873123,
+            -0.006727074272930622,
+            -0.008863375522196293,
+            0.019181253388524055,
+            -0.02109028771519661,
+            0.013166278600692749,
+            0.0685434341430664,
+            0.0021950111258774996,
+            0.025847723707556725,
+            0.007181606721132994,
+            0.024347767233848572,
+            0.02018122375011444,
+            -0.027271922677755356,
+            -0.02021152526140213,
+            -0.005397567991167307,
+            -0.0014033677289262414,
+            -0.03787767142057419,
+            0.011128460057079792,
+            -0.021302402019500732,
+            -0.013249609619379044,
+            -0.0022347825579345226,
+            -0.002316219499334693,
+            -0.006405114196240902,
+            0.034211114048957825,
+            -0.0014630251098424196,
+            -0.018560059368610382,
+            -0.035423196852207184,
+            0.037726160138845444,
+            0.04048365354537964,
+            0.0389382466673851,
+            -0.005984671879559755,
+            0.03548380360007286,
+            0.0030756669584661722,
+            0.0025302283465862274,
+            0.041453324258327484,
+            0.004791525658220053,
+            -0.011567841283977032,
+            0.02295386977493763,
+            0.012416300363838673,
+            0.03351416438817978,
+            -0.02045394293963909,
+            -0.02630225569009781,
+            -0.003299145260825753,
+            -0.026423463597893715,
+            0.012492056004703045,
+            -0.019726691767573357,
+            0.0018768386216834188,
+            0.007117214612662792,
+            0.015196521766483784,
+            -0.016044981777668,
+            0.014719262719154358,
+            -0.006102093029767275,
+            -0.021014532074332237,
+            0.011719351634383202,
+            -0.007780073676258326,
+            0.011961768381297588,
+            -0.02262054570019245,
+            0.001138224033638835,
+            -0.04063516482710838,
+            0.008280059322714806,
+            0.014022313989698887,
+            -0.01598437689244747,
+            0.03205966204404831,
+            0.020969079807400703,
+            -0.01801462098956108,
+            -0.0009682480012997985,
+            0.0034714885987341404,
+            -0.010393633507192135,
+            -0.011067855171859264,
+            -0.019923655316233635,
+            0.02786281518638134,
+            -0.015802564099431038,
+            -0.025908326730132103,
+            0.02454473078250885,
+            -0.01743887923657894,
+            0.015151068568229675,
+            -0.004299115855246782,
+            0.00888610165566206,
+            -0.019120648503303528,
+            0.02284781076014042,
+            -0.023741723969578743,
+            -0.0009128518868237734,
+            -0.020287280902266502,
+            0.0028389315120875835,
+            0.014135947450995445,
+            -0.018272189423441887,
+            -0.02139330841600895,
+            0.019150950014591217,
+            -0.005420294590294361,
+            -0.003931702114641666,
+            0.010014856234192848,
+            0.0030983935575932264,
+            -0.0029715034179389477,
+            -0.004549108445644379,
+            -0.03281721472740173,
+            -0.02915065549314022,
+            0.0033086147159337997,
+            0.011386028490960598,
+            0.015257125720381737,
+            0.027574945241212845,
+            0.03715042024850845,
+            0.006052851676940918,
+            0.0036949669010937214,
+            0.0009327376610599458,
+            -0.020726662129163742,
+            0.042635105550289154,
+            -0.020469093695282936,
+            0.040847279131412506,
+            0.032544493675231934,
+            -0.0197418425232172,
+            -0.023908386006951332,
+            0.0029052174650132656,
+            0.03115059621632099,
+            0.03151422366499901,
+            0.025211378931999207,
+            0.012310243211686611,
+            0.03690800443291664,
+            -0.004359719809144735,
+            -0.032211173325777054,
+            0.015802564099431038,
+            0.022256920114159584,
+            -0.008105821907520294,
+            0.009673957712948322,
+            0.027438584715127945,
+            0.013279912061989307,
+            -0.007757347077131271,
+            0.030150625854730606,
+            0.0012234487803652883,
+            0.02396899089217186,
+            0.02498411200940609,
+            -0.02190844528377056,
+            0.01743887923657894,
+            -0.0001154677156591788,
+            0.006727074272930622,
+            -0.009840618818998337,
+            0.004348356742411852,
+            0.011810258030891418,
+            0.018272189423441887,
+            0.018908534198999405,
+            0.003246116451919079,
+            -0.07060398161411285,
+            0.019726691767573357,
+            0.004507442936301231,
+            -0.02160542458295822,
+            0.00390140013769269,
+            -0.0024866692256182432,
+            0.012704171240329742,
+            0.02105998620390892,
+            0.012461753562092781,
+            -0.01790856383740902,
+            0.0013626492582261562,
+            0.011961768381297588,
+            -0.00595815759152174,
+            0.04730163514614105,
+            0.0019791084341704845,
+            -0.01437078882008791,
+            -0.016105586662888527,
+            0.013726867735385895,
+            0.02525683119893074,
+            -0.02355991117656231,
+            0.006325571332126856,
+            0.005621046293526888,
+            -0.004147605039179325,
+            -0.010742107406258583,
+            -0.030377892777323723,
+            -0.005280147306621075,
+            -0.03436262533068657,
+            0.023150833323597908,
+            0.015491968020796776,
+            0.014764715917408466,
+            -0.025044716894626617,
+            -0.004280176945030689,
+            0.0046400148421525955,
+            -0.021287251263856888,
+            0.003261267440393567,
+            0.01161329448223114,
+            0.0005104016163386405,
+            -0.020984230563044548,
+            0.00959820207208395,
+            0.013469300232827663,
+            0.06848283112049103,
+            0.0007442962378263474,
+            0.029771849513053894,
+            0.0422411784529686,
+            -0.0012243957025930285,
+            -0.006371024530380964,
+            0.0036286809481680393,
+            -0.019953956827521324,
+            0.003473382443189621,
+            0.025878025218844414,
+            -0.03224147483706474,
+            -0.055058982223272324,
+            -0.007545232307165861,
+            -0.04624105989933014,
+            0.001489539397880435,
+            0.014711687341332436,
+            0.023044776171445847,
+            -0.0037782976869493723,
+            -0.034817155450582504,
+            0.016908591613173485,
+            -0.003071879269555211,
+            0.007014944683760405,
+            -0.02593863010406494,
+            -0.0294839795678854,
+            -0.026014383882284164,
+            -0.02339324913918972,
+            -0.012908710166811943,
+            -0.022726602852344513,
+            0.012590537779033184,
+            -0.0015103721525520086,
+            -0.06587684899568558,
+            -0.009780014865100384,
+            0.016166189685463905,
+            0.001200722181238234,
+            0.011939042247831821,
+            0.011083006858825684,
+            -0.007083124481141567,
+            -0.0018275976181030273,
+            -0.022559940814971924,
+            0.03124150261282921,
+            0.005177877843379974,
+            0.03584742918610573,
+            -0.006503595970571041,
+            -0.009128518402576447,
+            0.01073453202843666,
+            0.04527139291167259,
+            0.0017916138749569654,
+            0.010226971469819546,
+            -0.020560000091791153,
+            0.0015065843472257257,
+            -0.02284781076014042,
+            -0.0031135445460677147,
+            -0.008863375522196293,
+            -0.0021211495622992516,
+            0.015893470495939255,
+            0.0019753205124288797,
+            -0.010204244405031204,
+            0.025438643991947174,
+            0.02251448854804039,
+            -0.009605777449905872,
+            -0.015969226136803627,
+            -0.02437806874513626,
+            -0.014545026235282421,
+            -0.0002627763315103948,
+            -0.010007280856370926,
+            -0.022453883662819862,
+            -0.020878173410892487,
+            -0.002566212322562933,
+            -0.02559015527367592,
+            0.022529639303684235,
+            0.01220418605953455,
+            -4.882668508798815e-05,
+            -0.006723286584019661,
+            -0.038544319570064545,
+            -0.003727162955328822,
+            -0.03118089959025383,
+            -0.005374841392040253,
+            0.03287781774997711,
+            -0.03745344281196594,
+            -0.0002231231628684327,
+            0.014491996727883816,
+            -0.01993880607187748,
+            0.0028938541654497385,
+            0.008961857296526432,
+            0.02769615314900875,
+            -0.027408283203840256,
+            0.0001795638381736353,
+            0.006954340264201164,
+            0.016908591613173485,
+            -0.009878496639430523,
+            0.027196168899536133,
+            -0.04275631532073021,
+            0.011037553660571575,
+            -0.037120118737220764,
+            -0.020423639565706253,
+            -0.01523439958691597,
+            0.008363389410078526,
+            -0.005018791649490595,
+            -0.02637801133096218,
+            -0.010454237461090088,
+            0.006026337388902903,
+            -0.00693918950855732,
+            -0.01902974210679531,
+            -0.010620899498462677,
+            0.027438584715127945,
+            0.00931033119559288,
+            0.007977037690579891,
+            -0.02955973520874977,
+            -0.01631770096719265,
+            0.006280118133872747,
+            0.052089374512434006,
+            -0.01584801822900772,
+            0.014938953332602978,
+            -0.006639956030994654,
+            0.02342355251312256,
+            -0.028484009206295013,
+            -0.01872672140598297,
+            0.0315445251762867,
+            0.013946558348834515,
+            0.007064185570925474,
+            0.028953691944479942,
+            0.027196168899536133,
+            0.027105262503027916,
+            -0.04654408246278763,
+            0.009886072017252445,
+            -0.002613559365272522,
+            -0.008757317438721657,
+            0.03402930125594139,
+            0.01916610263288021,
+            0.022150862962007523,
+            0.0009744031121954322,
+            -0.01730252057313919,
+            0.0139844361692667,
+            0.014885924756526947,
+            0.007681591901928186,
+            -0.014242004603147507,
+            -0.02660527639091015,
+            0.003367325058206916,
+            -0.007992188446223736,
+            0.00047276070108637214,
+            0.020726662129163742,
+            -0.009052763693034649,
+            0.030999086797237396,
+            -0.014605630189180374,
+            -0.02593863010406494,
+            0.008916404098272324,
+            -0.02268115058541298,
+            -0.0041741193272173405,
+            0.0004476667381823063,
+            0.010083036497235298,
+            -0.023847781121730804,
+            -0.04669559374451637,
+            0.019408518448472023,
+            0.028393102809786797,
+            0.005579380784183741,
+            0.01328748743981123,
+            -0.035332292318344116,
+            0.008992158807814121,
+            0.012310243211686611,
+            0.011348150670528412,
+            0.0019847899675369263,
+            0.035362593829631805,
+            -0.00028834378463216126,
+            0.012885984033346176,
+            -0.008620957843959332,
+            0.028362801298499107,
+            0.00667025800794363,
+            -0.0323626808822155,
+            -0.0378473699092865,
+            -0.01513591781258583,
+            0.015257125720381737,
+            0.006692984607070684,
+            -0.011840560473501682,
+            -0.02278720773756504,
+            -0.038817036896944046,
+            0.015787413343787193,
+            -0.012916285544633865,
+            -0.038271598517894745,
+            -0.010158791206777096,
+            -0.009742137044668198,
+            0.016635872423648834,
+            -0.006916462909430265,
+            -0.016802534461021423,
+            -0.016090434044599533,
+            0.005821797996759415,
+            7.161247049225494e-05,
+            0.0244083721190691,
+            -0.001251857029274106,
+            -0.006855858489871025,
+            -0.016393456608057022,
+            -0.01188601367175579,
+            -0.0035074723418802023,
+            0.03602924197912216,
+            -0.014219277538359165,
+            -0.02383263036608696,
+            0.014984406530857086,
+            0.02657497487962246,
+            0.011636020615696907,
+            0.03227177634835243,
+            0.007181606721132994,
+            -0.014022313989698887,
+            0.003016956616193056,
+            -0.033029329031705856,
+            -0.0046400148421525955,
+            0.005196816753596067,
+            -0.006427840795367956,
+            -0.015651054680347443,
+            -0.01815097965300083,
+            -0.015832867473363876,
+            0.030756669119000435,
+            -0.02487805485725403,
+            0.0218023881316185,
+            0.030347591266036034,
+            0.003174148965626955,
+            -0.02915065549314022,
+            0.013598084449768066,
+            -0.0072005451656877995,
+            0.01862066239118576,
+            0.0019014590652659535,
+            0.028014326468110085,
+            -0.007461901288479567,
+            -0.004818039946258068,
+            -0.02501441352069378,
+            -0.03696860745549202,
+            -0.023484155535697937,
+            -0.008817922323942184,
+            0.0027537066489458084,
+            0.020984230563044548,
+            -0.0375746488571167,
+            -0.017757052555680275,
+            0.009855770505964756,
+            0.015757111832499504,
+            -0.03027183562517166,
+            -0.0025908327661454678,
+            -0.010158791206777096,
+            -0.0004055278259329498,
+            0.005140000022947788,
+            -0.02227207086980343,
+            -0.008977008052170277,
+            -0.015726808458566666,
+            -0.0010435298318043351,
+            0.006166485138237476,
+            -0.0007409819518215954,
+            -0.05805889517068863,
+            -0.013969285413622856,
+            0.018635815009474754,
+            0.0026400736533105373,
+            0.008908828720450401,
+            0.015938924625515938,
+            -0.0111739132553339,
+            0.0031533162109553814,
+            -0.026756787672638893,
+            0.023893235251307487,
+            -0.009492144919931889,
+            0.009355784393846989,
+            -0.013742019422352314,
+            0.012514783069491386,
+            0.014014738611876965,
+            0.004795313347131014,
+            0.0010681503918021917,
+            -0.0054051438346505165,
+            0.01601468026638031,
+            0.028377952054142952,
+            -0.023756874725222588,
+            0.011454207822680473,
+            -0.0002514130319468677,
+            -0.0032158142421394587,
+            -0.029196109622716904,
+            0.00509454682469368,
+            0.003931702114641666,
+            -0.005109698045998812,
+            0.016651025041937828,
+            -0.036544378846883774,
+            0.01662072166800499,
+            0.02177208475768566,
+            0.032544493675231934,
+            0.014832896180450916,
+            -0.017287369817495346,
+            -0.011658747680485249,
+            -0.012385998852550983,
+            -0.002624922664836049,
+            0.022181164473295212,
+            0.012120855040848255,
+            -0.01815097965300083,
+            -0.024287162348628044,
+            0.011249667964875698,
+            0.0005283934879116714,
+            -0.01203752402216196,
+            0.010795135982334614,
+            -0.013166278600692749,
+            0.01349202636629343,
+            0.012711746618151665,
+            0.00866641104221344,
+            0.00888610165566206,
+            0.022862963378429413,
+            -0.011878437362611294,
+            -0.017711598426103592,
+            0.00656041270121932,
+            -0.008916404098272324,
+            -0.0034089903347194195,
+            -0.012598113156855106,
+            0.010795135982334614,
+            0.014878349378705025,
+            0.06490717828273773,
+            -0.013143551535904408,
+            0.0009526234352961183,
+            -0.020438792183995247,
+            -0.031938452273607254,
+            -0.0017944546416401863,
+            -0.020908474922180176,
+            0.0111739132553339,
+            0.008370965719223022,
+            0.01598437689244747,
+            0.012022373266518116,
+            -0.0021912232041358948,
+            0.00795431062579155,
+            0.019317612051963806,
+            -0.010423935018479824,
+            7.255941454786807e-05,
+            -0.006503595970571041,
+            0.0013683309080079198,
+            0.0035851215943694115,
+            -0.011719351634383202,
+            -0.0026949963066726923,
+            0.035695917904376984,
+            0.01068150345236063,
+            0.011651171371340752,
+            -0.03445352986454964,
+            0.0020435003098100424,
+            0.020150920376181602,
+            0.007602048572152853,
+            -0.008302785456180573,
+            -0.01563590206205845,
+            0.01706010289490223,
+            -0.015317730605602264,
+            -0.0022499337792396545,
+            0.002888172399252653,
+            -0.010151215828955173,
+            0.04581683129072189,
+            -0.019787294790148735,
+            -0.0011742077767848969,
+            0.031847547739744186,
+            0.0030207443051040173,
+            -0.0040983641520142555,
+            0.005408931523561478,
+            -0.003473382443189621,
+            -0.0069013116881251335,
+            -0.02139330841600895,
+            0.0031381649896502495,
+            0.00667025800794363,
+            -0.019287310540676117,
+            -0.0072005451656877995,
+            -0.007719469256699085,
+            0.006897523999214172,
+            0.023105379194021225,
+            -0.009492144919931889,
+            -0.00016145357221830636,
+            -0.008575504645705223,
+            0.009848194196820259,
+            -0.0011912527261301875,
+            -0.044301725924015045,
+            -0.006147546228021383,
+            0.021181194111704826,
+            0.000973456131760031,
+            0.0005757405888289213,
+            -0.007450537756085396,
+            0.005007428117096424,
+            -0.017423728480935097,
+            -0.008454295806586742,
+            0.02369627170264721,
+            0.01736312545835972,
+            -0.009113367646932602,
+            0.010863316245377064,
+            -0.0035680767614394426,
+            0.00433320552110672,
+            -0.020014561712741852,
+            -0.01095422264188528,
+            0.003462019143626094,
+            0.015272277407348156,
+            -0.0009819786064326763,
+            -0.029029447585344315,
+            -0.0012272365856915712,
+            -0.00024194362049456686,
+            0.02918095886707306,
+            0.01486319862306118,
+            -0.008961857296526432,
+            -0.021211495622992516,
+            -0.013242034241557121,
+            -0.001237652963027358,
+            -0.015726808458566666,
+            -0.009620928205549717,
+            0.011855711229145527,
+            0.0009137988090515137,
+            0.008992158807814121,
+            -0.018135828897356987,
+            -0.027347678318619728,
+            -0.016166189685463905,
+            0.0018465365283191204,
+            -0.012870832346379757,
+            0.003340810537338257,
+            0.015514694154262543,
+            -0.03430201858282089,
+            -0.015060162171721458,
+            -0.03805948421359062,
+            0.008280059322714806,
+            -0.0029658216517418623,
+            -0.02024182677268982,
+            -0.023605365306138992,
+            0.009484568610787392,
+            0.019181253388524055,
+            -0.0015548784285783768,
+            0.019847899675369263,
+            -0.000889651826582849,
+            0.020060013979673386,
+            -0.0073141781613230705,
+            0.0031703610438853502,
+            -0.013007191941142082,
+            0.0030245319940149784,
+            0.008333087898790836,
+            0.02890823967754841,
+            -0.023075077682733536,
+            -0.006503595970571041,
+            -0.004685468040406704,
+            0.004746072459965944,
+            0.012658718042075634,
+            0.014613205567002296,
+            0.020741812884807587,
+            -0.008272483013570309,
+            -0.0027612822595983744,
+            0.012484480626881123,
+            -0.003799130441620946,
+            0.007295239716768265,
+            0.002390081062912941,
+            0.0019677451346069574,
+            0.004162756260484457,
+            0.008477022871375084,
+            0.0053710537031292915,
+            0.027650700882077217,
+            -0.0028275682125240564,
+            0.0031703610438853502,
+            0.024620486423373222,
+            0.02322658896446228,
+            -0.018878230825066566,
+            -0.028074929490685463,
+            0.002024561632424593,
+            0.050695475190877914,
+            0.007245998363941908,
+            -0.006810405291616917,
+            -0.03927157074213028,
+            -0.0031760428100824356,
+            0.028635520488023758,
+            0.023650817573070526,
+            0.01584801822900772,
+            -0.010567869991064072,
+            -0.012098127976059914,
+            0.02772645466029644,
+            0.013295062817633152,
+            0.02616589516401291,
+            -0.025302285328507423,
+            0.010704229585826397,
+            -0.03869583085179329,
+            0.0449683703482151,
+            0.004439263138920069,
+            -0.014476845972239971,
+            0.0011884119594469666,
+            0.010386057198047638,
+            -0.004537744913250208,
+            0.006431628484278917,
+            -5.8059369621332735e-05,
+            -0.0008086882880888879,
+            0.011923890560865402,
+            0.012007221579551697,
+            0.004265025723725557,
+            -0.02399929240345955,
+            0.007359631359577179,
+            0.004272601567208767,
+            0.03005971945822239,
+            0.005431658122688532,
+            -0.0013863227795809507,
+            0.012385998852550983,
+            -0.005810434930026531,
+            0.008855799213051796,
+            -0.010719381272792816,
+            0.020469093695282936,
+            -0.02918095886707306,
+            1.0986003871948924e-05,
+            0.02051454596221447,
+            -0.0015785519499331713,
+            0.0038199631962925196,
+            -0.023741723969578743,
+            0.01186328660696745,
+            -0.052968136966228485,
+            0.021166043356060982,
+            -0.018741872161626816,
+            0.021741783246397972,
+            0.019590331241488457,
+            -0.009734561666846275,
+            -0.012052674777805805,
+            0.03545350208878517,
+            0.0026760573964565992,
+            -0.014264730736613274,
+            -0.010643625631928444,
+            -0.017999470233917236,
+            -0.02830219641327858,
+            0.041968461126089096,
+            -0.0006117243901826441,
+            0.0028275682125240564,
+            0.0024222771171480417,
+            -0.010666352696716785,
+            -0.008923979476094246,
+            0.001954487757757306,
+            0.020499395206570625,
+            -0.03369597718119621,
+            -0.011128460057079792,
+            -0.008689138107001781,
+            -0.02566591091454029,
+            -0.00020300064352340996,
+            -0.011961768381297588,
+            0.026211349293589592,
+            0.01240872498601675,
+            -0.03696860745549202,
+            0.003780191531404853,
+            0.023650817573070526,
+            0.001603172393515706,
+            0.018120678141713142,
+            0.002130619017407298,
+            0.0010795136913657188,
+            0.009742137044668198,
+            -0.010476963594555855,
+            0.026559824123978615,
+            -0.009204274043440819,
+            0.008658835664391518,
+            -0.001950700068846345,
+            -0.017287369817495346,
+            0.017969166859984398,
+            0.004083212930709124,
+            0.024120500311255455,
+            0.011954193003475666,
+            0.011605718173086643,
+            0.01112088467925787,
+            -0.019878201186656952,
+            0.020090317353606224,
+            -0.0018891488434746861,
+            0.009075489826500416,
+            0.024726543575525284,
+            -0.005863463506102562,
+            0.020832719281315804,
+            0.015257125720381737,
+            -0.00020229043730068952,
+            0.011295121163129807,
+            -0.020120618864893913,
+            -0.011779955588281155,
+            0.0085679292678833,
+            -0.009605777449905872,
+            -0.010325453244149685,
+            0.009901223704218864,
+            -0.010105762630701065,
+            -0.015052586793899536,
+            -0.0071247899904847145,
+            0.016635872423648834,
+            0.0028976418543606997,
+            -0.02610529027879238,
+            -1.7326075976598077e-05,
+            0.008901252411305904,
+            0.006166485138237476,
+            0.018741872161626816,
+            0.011317848227918148,
+            -0.006692984607070684,
+            0.010363331064581871,
+            0.011476934887468815,
+            0.0053104497492313385,
+            -0.005079395603388548,
+            -0.013431422412395477,
+            -0.0024241709616035223,
+            0.018090376630425453,
+            0.009810317307710648,
+            -0.01922670565545559,
+            0.004564259201288223,
+            0.0008877579239197075,
+            -0.01420412678271532,
+            -0.019347915425896645,
+            -0.0038483713287860155,
+            -0.004170331638306379,
+            0.0027915844693779945,
+            0.0509984977543354,
+            -0.0006216673064045608,
+            0.007893706671893597,
+            -0.025741664692759514,
+            -0.006299056578427553,
+            -0.00027556007262319326,
+            -0.0036097420379519463,
+            0.01272689737379551,
+            -0.01824188604950905,
+            -0.03724132478237152,
+            0.012711746618151665,
+            -0.0012584856012836099,
+            0.014923802576959133,
+            -0.00297907879576087,
+            0.014226853847503662,
+            -0.017347972840070724,
+            0.01916610263288021,
+            -0.0011117096291854978,
+            -0.016029831022024155,
+            0.0029374135192483664,
+            -0.011143610812723637,
+            -0.021317552775144577,
+            0.005916492082178593,
+            -0.006848283112049103,
+            -0.03896854817867279,
+            0.0029885482508689165,
+            -0.010423935018479824,
+            0.0011798894265666604,
+            -0.003912763670086861,
+            0.020166072994470596,
+            -0.0061513339169323444,
+            0.0008697660523466766,
+            0.02177208475768566,
+            0.0017026013229042292,
+            0.01601468026638031,
+            -0.014499573037028313,
+            0.010999675840139389,
+            0.012598113156855106,
+            0.02210540883243084,
+            -0.032180868089199066,
+            -0.01393140759319067,
+            0.014211702160537243,
+            0.0026116655208170414,
+            -0.005382417235523462,
+            0.020635755732655525,
+            -0.01294658798724413,
+            0.019650936126708984,
+            -0.0005577487172558904,
+            -0.011704200878739357,
+            -0.03524138405919075,
+            -0.016529815271496773,
+            -3.343888238305226e-05,
+            -0.008439145050942898,
+            0.025317436084151268,
+            0.01760554127395153,
+            -0.0108330138027668,
+            0.026468917727470398,
+            0.0037574649322777987,
+            0.014598054811358452,
+            -0.014257155358791351,
+            -0.005999823100864887,
+            -0.0003773563075810671,
+            -0.0119314668700099,
+            0.015264702029526234,
+            -0.03512017801403999,
+            0.0008342557121068239,
+            -0.00839369185268879,
+            0.005242269951850176,
+            -0.0254992488771677,
+            0.018257036805152893,
+            0.025817420333623886,
+            -0.0031419529113918543,
+            -0.005037730094045401,
+            -0.022802358493208885,
+            0.028968842700123787,
+            0.03049910068511963,
+            0.0024828813038766384,
+            -0.0013825349742546678,
+            -0.00352072948589921,
+            0.0033218718599528074,
+            0.0211205892264843,
+            -0.037998881191015244,
+            -0.009098216891288757,
+            -0.028696123510599136,
+            0.042332086712121964,
+            -0.008098246529698372,
+            -0.013431422412395477,
+            0.01862066239118576,
+            -0.0024715180043131113,
+            0.02355991117656231,
+            -0.004689255729317665,
+            -0.004855917301028967,
+            0.03148392215371132,
+            0.011514811776578426,
+            0.02772645466029644,
+            -0.04015033319592476,
+            0.014704111963510513,
+            -0.006867222022265196,
+            0.004867280833423138,
+            -0.013484450988471508,
+            0.005302873905748129,
+            0.02580226957798004,
+            -0.018060073256492615,
+            0.0068066176027059555,
+            0.030665762722492218,
+            0.00748084019869566,
+            0.002448791405186057,
+            0.0165904201567173,
+            0.003890036838129163,
+            0.010855740867555141,
+            0.000841831264551729,
+            0.0052574207074940205,
+            0.022423582151532173,
+            0.014454119838774204,
+            -0.016029831022024155,
+            -0.015120766125619411,
+            -0.00044790346873924136,
+            -0.004450626205652952,
+            0.005261208396404982,
+            0.013242034241557121,
+            0.0061172437854111195,
+            -0.015681356191635132,
+            0.01052999310195446,
+            0.014158673584461212,
+            0.013128400780260563,
+            -0.0039847311563789845,
+            -0.03412020578980446,
+            -0.02443867363035679,
+            -0.021438762545585632,
+            0.003104075090959668,
+            -0.0008281006012111902,
+            -0.0021211495622992516,
+            0.0017954016802832484,
+            0.017347972840070724,
+            -0.0008352026343345642,
+            0.00379155483096838,
+            0.01686313934624195,
+            -0.01415109820663929,
+            -0.022090258076786995,
+            -0.0035964848939329386,
+            -0.027529491111636162,
+            0.0053710537031292915,
+            0.001844642567448318,
+            -0.00965123064815998,
+            0.008530051447451115,
+            -0.0046059247106313705,
+            -0.007223271764814854,
+            -0.012196609750390053,
+            -0.02007516659796238,
+            0.005780132487416267,
+            0.016878290101885796,
+            0.005730891600251198,
+            -0.009272454306483269,
+            -0.003939277958124876,
+            0.011575416661798954,
+            0.017484333366155624,
+            0.017317671328783035,
+            -0.009295180439949036,
+            0.009628504514694214,
+            0.0004022135108243674,
+            0.004670316819101572,
+            -0.01834794320166111,
+            0.0045869858004152775,
+            -0.0027953721582889557,
+            -0.008658835664391518,
+            0.01105270441621542,
+            -0.00031201730598695576,
+            0.01628739945590496,
+            -0.013090522959828377,
+            0.005624833982437849,
+            0.010333028621971607,
+            -0.020590301603078842,
+            -0.0067308624275028706,
+            -0.01692374423146248,
+            0.026484068483114243,
+            -0.00504530593752861,
+            0.00045405857963487506,
+            -0.02978700026869774,
+            0.0033616432920098305,
+            0.000848933297675103,
+            -0.000845619011670351,
+            -0.012764775194227695,
+            0.010749682784080505,
+            0.03242328763008118,
+            0.0005179771687835455,
+            0.0216660276055336,
+            -0.0036154238041490316,
+            0.0007480839849449694,
+            -0.00785582885146141,
+            0.004977126140147448,
+            -0.004916521720588207,
+            -0.009901223704218864,
+            -0.010863316245377064,
+            0.021726632490754128,
+            -0.0033029329497367144,
+            0.0003356908564455807,
+            0.011332998983561993,
+            -0.012067826464772224,
+            0.0014620780711993575,
+            -0.0047687990590929985,
+            0.00971941091120243,
+            0.0061702728271484375,
+            0.009113367646932602,
+            -0.01176480483263731,
+            0.05451354384422302,
+            -0.0106284748762846,
+            0.005537715740501881,
+            -0.00439380994066596,
+            -0.022226616740226746,
+            -0.017317671328783035,
+            0.012711746618151665,
+            -0.017544938251376152,
+            0.03375658020377159,
+            0.024150803685188293,
+            0.0006467612111009657,
+            0.010393633507192135,
+            -0.006958128418773413,
+            0.023923536762595177,
+            -0.002602196065708995,
+            0.0010795136913657188,
+            -0.004855917301028967,
+            0.015802564099431038,
+            0.04511988162994385,
+            0.017393426969647408,
+            -0.001152428099885583,
+            0.00829521007835865,
+            0.020575150847434998,
+            0.00814369972795248,
+            -0.0020075165666639805,
+            0.014219277538359165,
+            -0.01024212222546339,
+            0.00900731049478054,
+            -0.004458202049136162,
+            0.027438584715127945,
+            0.00814369972795248,
+            -0.006310420110821724,
+            -0.0036836036015301943,
+            -0.0008209985098801553,
+            0.014173824340105057,
+            0.006950552575290203,
+            0.01963578537106514,
+            -0.00461350055411458,
+            -0.0012575386790558696,
+            0.013870803639292717,
+            0.014491996727883816,
+            -0.00552635220810771,
+            0.001724380999803543,
+            -0.01435563713312149,
+            -0.010204244405031204,
+            0.013575357384979725,
+            0.006086941808462143,
+            0.006310420110821724,
+            -0.014514723792672157,
+            -0.00980274099856615,
+            -0.004670316819101572,
+            0.0015435151290148497,
+            0.015052586793899536,
+            0.011060279794037342,
+            -0.004814252257347107,
+            0.006052851676940918,
+            0.013825350441038609,
+            -0.014014738611876965,
+            0.006147546228021383,
+            0.007208121009171009,
+            0.008855799213051796,
+            0.004787737503647804,
+            -0.010817863047122955,
+            -0.00428396463394165,
+            -0.0063937511295080185,
+            0.0012670081341639161,
+            0.016832837834954262,
+            0.013181429356336594,
+            0.0049581872299313545,
+            0.027529491111636162,
+            -0.00839369185268879,
+            -0.012302667833864689,
+            0.02339324913918972,
+            0.011257244274020195,
+            0.0031703610438853502,
+            0.014651083387434483,
+            -0.012635990977287292,
+            -0.012446602806448936,
+            0.006655106786638498,
+            0.004848341923207045,
+            0.006049063988029957,
+            0.009151245467364788,
+            0.00450365524739027,
+            0.007333117071539164,
+            -0.007098275702446699,
+            -0.007427811156958342,
+            0.0014810169814154506,
+            0.01642375811934471,
+            0.020393338054418564,
+            -0.024968961253762245,
+            -0.0018806264270097017,
+            0.0009052763343788683,
+            0.010878467001020908,
+            -0.003316190093755722,
+            0.012310243211686611,
+            5.65205882594455e-05,
+            -0.0074240234680473804,
+            0.010158791206777096,
+            -0.02119634486734867,
+            -0.012681444175541401,
+            0.0012717428617179394,
+            0.013037494383752346,
+            -0.019590331241488457,
+            0.018226735293865204,
+            -0.008893677033483982,
+            -0.013348091393709183,
+            0.013893529772758484,
+            0.020014561712741852,
+            0.004090788308531046,
+            -0.007738408166915178,
+            -0.021862991154193878,
+            -0.01127239502966404,
+            0.016060132533311844,
+            0.020272130146622658,
+            0.000947888707742095,
+            -0.0006472347304224968,
+            0.010173942893743515,
+            -0.013113250024616718,
+            -0.023605365306138992,
+            -0.005230906419456005,
+            0.010386057198047638,
+            0.01746918261051178,
+            0.018832778558135033,
+            -0.006685409229248762,
+            0.02443867363035679,
+            -0.0059127043932676315,
+            -0.015499543398618698,
+            2.86597842205083e-05,
+            -0.02802947722375393,
+            -0.017423728480935097,
+            0.00013778003631159663,
+            0.00246773031540215,
+            0.008598231710493565,
+            0.0008555619278922677,
+            0.01943882182240486,
+            0.000904329412151128,
+            0.017590390518307686,
+            -0.000659071491099894,
+            -0.002009410411119461,
+            -0.0013995799235999584,
+            -0.0015340456739068031,
+            0.02254479005932808,
+            -0.012166308239102364,
+            -0.01176480483263731,
+            0.022302372381091118,
+            0.006852070800960064,
+            0.014719262719154358,
+            -0.027635548263788223,
+            0.013757170177996159,
+            -0.015499543398618698,
+            0.017423728480935097,
+            -0.03408990427851677,
+            0.006420265417546034,
+            0.01489350013434887,
+            -0.010348179377615452,
+            -0.020499395206570625,
+            -0.029650641605257988,
+            -0.024150803685188293,
+            -0.0038710979279130697,
+            -0.010855740867555141,
+            -0.016847988590598106,
+            0.006121031939983368,
+            -0.005802859086543322,
+            0.018499454483389854,
+            -0.007079336792230606,
+            0.033877789974212646,
+            0.006969491485506296,
+            -0.004056698642671108,
+            0.003835114184767008,
+            0.022469034418463707,
+            -0.006102093029767275,
+            0.02109028771519661,
+            0.004408961161971092,
+            -0.001467759720981121,
+            0.013681414537131786,
+            -0.006552837323397398,
+            -0.002361672930419445,
+            0.005821797996759415,
+            -0.010484539903700352,
+            0.022014502435922623,
+            0.005924067925661802,
+            -0.007708106189966202,
+            -0.0006775368237867951,
+            -0.011454207822680473,
+            0.014726839028298855,
+            0.006530110724270344,
+            0.0022480397019535303,
+            -0.011295121163129807,
+            0.011249667964875698,
+            -0.004859705455601215,
+            0.0040642740204930305,
+            0.009484568610787392,
+            -0.01774190180003643,
+            -0.028862785547971725,
+            0.011416330002248287,
+            0.019120648503303528,
+            0.006382387597113848,
+            -0.0055755930952727795,
+            -0.0025681061670184135,
+            0.00839369185268879,
+            0.010651201009750366,
+            0.022150862962007523,
+            -0.012984465807676315,
+            -8.889415767043829e-05,
+            0.013204156421124935,
+            -0.007170243188738823,
+            -0.01056029461324215,
+            0.00487485621124506,
+            0.023438703268766403,
+            0.0068823727779090405,
+            -0.04327145218849182,
+            -0.018469152972102165,
+            0.01909034699201584,
+            0.007136153522878885,
+            -0.04154422879219055,
+            0.0026381798088550568,
+            -0.003689285134896636,
+            -0.023787178099155426,
+            0.016105586662888527,
+            -0.0011817833874374628,
+            -0.01828734017908573,
+            0.028756728395819664,
+            0.02830219641327858,
+            0.01834794320166111,
+            2.6987840101355687e-05,
+            -0.013332940638065338,
+            -0.0318172425031662,
+            -0.023590214550495148,
+            0.008280059322714806,
+            -0.02660527639091015,
+            -0.0072725131176412106,
+            0.011242092587053776,
+            -0.017454031854867935,
+            0.02972639724612236,
+            -0.008507325313985348,
+            0.026938600465655327,
+            0.011446632444858551,
+            0.001691238023340702,
+            -0.013446573168039322,
+            -0.018802475184202194,
+            -0.01350717805325985,
+            -0.008196728304028511,
+            0.014090493321418762,
+            0.017923714593052864,
+            0.011007251217961311,
+            0.011332998983561993,
+            0.02251448854804039,
+            -0.012158732861280441,
+            0.007969462312757969,
+            -2.888172457460314e-05,
+            -0.008961857296526432,
+            0.0026343921199440956,
+            -0.018393397331237793,
+            0.00868156272917986,
+            0.0062308767810463905,
+            0.005458172410726547,
+            -0.007208121009171009,
+            0.010742107406258583,
+            0.02712041325867176,
+            -0.018847929313778877,
+            0.011401179246604443,
+            -0.031362712383270264,
+            -0.014787442982196808,
+            -0.01056029461324215,
+            0.026862844824790955,
+            0.007757347077131271,
+            -0.015726808458566666,
+            -0.007514929864555597,
+            -0.01545409020036459,
+            0.012885984033346176,
+            0.0032442226074635983,
+            0.005764981731772423,
+            0.015817714855074883,
+            0.021893294528126717,
+            0.01746918261051178,
+            -0.00795431062579155,
+            0.006852070800960064,
+            0.004674104508012533,
+            0.028090080246329308,
+            -0.01777220331132412,
+            0.020802417770028114,
+            0.009954252280294895,
+            -0.003935490269213915,
+            0.005685438401997089,
+            -0.014257155358791351,
+            0.008530051447451115,
+            -0.007325541693717241,
+            0.030514251440763474,
+            0.0033067206386476755,
+            0.014075342565774918,
+            -0.02062060497701168,
+            0.006575563922524452,
+            -0.03172633796930313,
+            0.009022461250424385,
+            0.013704141601920128,
+            0.012976890429854393,
+            -0.007677804213017225,
+            -0.024181105196475983,
+            -0.0119314668700099,
+            0.0057990713976323605,
+            -0.01279507763683796,
+            -0.01046181283891201,
+            -0.0034828518982976675,
+            0.012446602806448936,
+            0.018847929313778877,
+            -0.01848430372774601,
+            -0.019923655316233635,
+            -0.009583051316440105,
+            0.02898399345576763,
+            -0.015787413343787193,
+            -0.001471547526307404,
+            0.02413565292954445,
+            0.021241798996925354,
+            0.015022284351289272,
+            0.002155239460989833,
+            0.020090317353606224,
+            -0.0007111532613635063,
+            -0.0237265732139349,
+            -0.02813553437590599,
+            0.007302815094590187,
+            -0.006912675220519304,
+            0.01943882182240486,
+            -0.0035548193845897913,
+            -0.005658924113959074,
+            -0.0034033088013529778,
+            -0.007503566797822714,
+            -0.0165904201567173,
+            -0.01787826046347618,
+            0.0021647089160978794,
+            -0.008454295806586742,
+            0.005863463506102562,
+            -0.0002795845502987504,
+            -0.013105674646794796,
+            0.013401119969785213,
+            -0.014075342565774918,
+            0.005772557109594345,
+            -0.002653330797329545,
+            0.001962063368409872,
+            0.0035926972050219774,
+            0.009113367646932602,
+            -0.007336904760450125,
+            -0.06381630152463913,
+            -0.013105674646794796,
+            -0.003704436356201768,
+            0.006071790587157011,
+            -0.00019578021601773798,
+            -0.016635872423648834,
+            -0.013196581043303013,
+            0.01706010289490223,
+            0.008863375522196293,
+            -0.013870803639292717,
+            0.016954045742750168,
+            0.009961827658116817,
+            0.003622999181970954,
+            -0.024044746533036232,
+            -0.0008148433989845216,
+            0.008757317438721657,
+            0.032756611704826355,
+            -0.008643684908747673,
+            -0.0003839849086944014,
+            0.0036684523802250624,
+            -0.0033332351595163345,
+            -0.010052734054625034,
+            -0.014582903124392033,
+            -0.011848135851323605,
+            0.011908739805221558,
+            0.01790856383740902,
+            -0.014264730736613274,
+            -0.02007516659796238,
+            -0.0007106798002496362,
+            0.012931437231600285,
+            -0.026590125635266304,
+            0.0007244104635901749,
+            -0.007787649054080248,
+            -0.01052999310195446,
+            0.018302490934729576,
+            -0.019014591351151466,
+            0.005249845329672098,
+            0.00048294031876139343,
+            0.008507325313985348,
+            -0.023544760420918465,
+            -0.024893205612897873,
+            0.0035131541080772877,
+            -0.023847781121730804,
+            0.03612014651298523,
+            -0.021590271964669228,
+            -0.012082977220416069,
+            0.034483831375837326,
+            -0.0037423139438033104,
+            0.01702980138361454,
+            0.0023995505180209875,
+            0.02739313244819641,
+            -0.011439057067036629,
+            -0.00617784820497036,
+            0.012499631382524967,
+            0.009575475007295609,
+            0.0032593735959380865,
+            0.03775646165013313,
+            -0.01257538702338934,
+            -0.013901105150580406,
+            -0.005556654185056686,
+            -0.012954163365066051,
+            0.0006775368237867951,
+            0.014431392773985863,
+            0.028393102809786797,
+            0.004257450345903635,
+            -0.00628769351169467,
+            -0.006143758539110422,
+            -0.0014649189542979002,
+            -0.004242299124598503,
+            -0.00947699323296547,
+            -0.00715130427852273,
+            0.017287369817495346,
+            -0.025241680443286896,
+            0.016711628064513206,
+            -0.006420265417546034,
+            0.021135739982128143,
+            -0.011757229454815388,
+            0.007007369305938482,
+            -0.008961857296526432,
+            0.003492321353405714,
+            0.0023635667748749256,
+            -0.005109698045998812,
+            0.009287605062127113,
+            -0.014772292226552963,
+            0.008454295806586742,
+            0.013742019422352314,
+            0.0011183382011950016,
+            -0.010590597055852413,
+            -0.019272159785032272,
+            0.0022972808219492435,
+            -0.000710206339135766,
+            0.01181783340871334,
+            -0.0047498601488769054,
+            -0.000981031684204936,
+            0.0002921315317507833,
+            -0.014476845972239971,
+            -0.009575475007295609,
+            0.0006050958181731403,
+            0.005302873905748129,
+            -0.008961857296526432,
+            0.010378481820225716,
+            -0.005693014245480299,
+            -0.00034160923678427935,
+            0.02627195231616497,
+            0.006526322569698095,
+            -0.007632351014763117,
+            -0.014098069630563259,
+            0.01563590206205845,
+            0.015370759181678295,
+            -0.007454325910657644,
+            0.01933276280760765,
+            -0.016196493059396744,
+            -0.0008366230758838356,
+            0.013916256837546825,
+            -0.009219424799084663,
+            0.0017698341980576515,
+            -0.008196728304028511,
+            0.03554440662264824,
+            -0.0008787619881331921,
+            -0.0035415622405707836,
+            -0.0006401326390914619,
+            0.012976890429854393,
+            -0.0026779514737427235,
+            0.001182730309665203,
+            0.01132542360574007,
+            0.005321812815964222,
+            0.0185146052390337,
+            0.00328020635060966,
+            -0.03378688171505928,
+            -0.007780073676258326,
+            -0.0065490491688251495,
+            -0.008090670220553875,
+            0.009022461250424385,
+            0.025196226313710213,
+            0.01990850456058979,
+            0.006408901885151863,
+            0.001881573349237442,
+            0.018423698842525482,
+            0.015757111832499504,
+            -0.01489350013434887,
+            0.016363153234124184,
+            0.02802947722375393,
+            -0.010030006989836693,
+            0.022741753607988358,
+            -0.005253633018583059,
+            -0.008893677033483982,
+            0.008946705609560013,
+            -0.0015378334792330861,
+            -0.017393426969647408,
+            -0.015681356191635132,
+            0.005022579338401556,
+            -0.008999735116958618,
+            -0.0022897052112966776,
+            -0.018605511635541916,
+            0.004670316819101572,
+            0.017105557024478912,
+            -0.004965762607753277,
+            -3.5243989259470254e-05,
+            0.019484274089336395,
+            -0.006965703796595335,
+            -0.016908591613173485,
+            0.006325571332126856,
+            -0.015007133595645428,
+            0.007946735247969627,
+            -0.013666263781487942,
+            0.010492115281522274,
+            -0.006098304875195026,
+            0.018817627802491188,
+            0.005446809343993664,
+            -0.008090670220553875,
+            -0.0008958069374784827,
+            -0.018984289839863777,
+            0.02065090648829937,
+            -0.033089932054281235,
+            0.00839369185268879,
+            -0.026226500049233437,
+            -0.0009781909175217152,
+            0.0008849170990288258,
+            0.014605630189180374,
+            -0.001218714052811265,
+            0.019696388393640518,
+            -0.002751812804490328,
+            -0.020090317353606224,
+            -0.007086912170052528,
+            -0.0010634156642481685,
+            -0.01618134044110775,
+            -0.030923331156373024,
+            -0.004272601567208767,
+            0.0036267871037125587,
+            -0.004711982328444719,
+            0.0021135739516466856,
+            0.017984317615628242,
+            -0.005613470915704966,
+            0.009173971600830555,
+            -0.02690829709172249,
+            -0.007416448090225458,
+            -0.00355103169567883,
+            0.0010776197304949164,
+            -0.011242092587053776,
+            0.0010596278589218855,
+            0.011226941831409931,
+            -0.019044892862439156,
+            0.013461724855005741,
+            -0.021378157660365105,
+            -0.004371083341538906,
+            -0.008507325313985348,
+            -0.00135412672534585,
+            -0.013446573168039322,
+            -0.028120383620262146,
+            0.025029566138982773,
+            -0.00021495578403118998,
+            -0.003528305096551776,
+            -0.011999646201729774,
+            0.023181134834885597,
+            0.002914686920121312,
+            0.027817361056804657,
+            -0.03157482668757439,
+            0.0013427634257823229,
+            0.011552689597010612,
+            0.0025984083767980337,
+            -0.017287369817495346,
+            0.004746072459965944,
+            -0.005234694108366966,
+            0.02210540883243084,
+            -0.0011694730492308736,
+            -0.0005340751959010959,
+            -0.008234606124460697,
+            0.030999086797237396,
+            0.009136094711720943,
+            0.0012944694608449936,
+            -0.013226882554590702,
+            -0.008060368709266186,
+            -0.002759388415142894,
+            -0.0011069749016314745,
+            0.012855681590735912,
+            0.01290113478899002,
+            -0.012105703353881836,
+            0.0013607552973553538,
+            -0.006878585088998079,
+            0.0011694730492308736,
+            -0.002935519441962242,
+            -0.0004966709529981017,
+            -0.01149208564311266,
+            0.003257479751482606,
+            0.016272246837615967,
+            -0.00780280027538538,
+            -0.006480869371443987,
+            0.0007750718505121768,
+            0.009181547909975052,
+            0.012461753562092781,
+            -0.012742049060761929,
+            -0.018226735293865204,
+            -0.011908739805221558,
+            0.016029831022024155,
+            0.010522416792809963,
+            -0.008204303681850433,
+            -0.013234458863735199,
+            -0.02583257108926773,
+            0.01169662456959486,
+            -0.03293842449784279,
+            -0.012242063879966736,
+            0.005821797996759415,
+            0.028559764847159386,
+            0.019999410957098007,
+            0.012226912193000317,
+            0.001332347048446536,
+            0.002950670663267374,
+            0.00023342114582192153,
+            -0.01916610263288021,
+            0.014408666640520096,
+            -0.011143610812723637,
+            -0.018166130408644676,
+            0.0075906855054199696,
+            0.011939042247831821,
+            0.03148392215371132,
+            0.006942977197468281,
+            -0.0010160685051232576,
+            0.01686313934624195,
+            -0.001218714052811265,
+            -0.013582932762801647,
+            0.01420412678271532,
+            -0.024256860837340355,
+            0.016438908874988556,
+            0.02860521711409092,
+            -0.004780162125825882,
+            0.0008531945641152561,
+            -0.020938776433467865,
+            0.00482561532407999,
+            -0.011529963463544846,
+            -0.002177966060116887,
+            -0.003287781961262226,
+            -0.03460504114627838,
+            0.007946735247969627,
+            -0.006825556512922049,
+            0.01046181283891201,
+            -0.00814369972795248,
+            0.006655106786638498,
+            -0.012431452050805092,
+            0.004253662656992674,
+            0.01513591781258583,
+            0.016105586662888527,
+            0.007060397882014513,
+            -0.025211378931999207,
+            -0.009196698665618896,
+            0.008105821907520294,
+            -0.01349202636629343,
+            -0.02295386977493763,
+            -0.006852070800960064,
+            0.009840618818998337,
+            -0.01061332318931818,
+            -0.00488243205472827,
+            -0.028180986642837524,
+            -0.013681414537131786,
+            -0.0014327229000627995,
+            -0.017817657440900803,
+            -0.005556654185056686,
+            -0.003100287402048707,
+            0.01571165770292282,
+            -0.0049733384512364864,
+            -0.006185423582792282,
+            0.004742284305393696,
+            -0.023878084495663643,
+            0.0037423139438033104,
+            -0.008280059322714806,
+            0.0008266801596619189,
+            -0.0032517979852855206,
+            -0.0014762822538614273,
+            -0.007969462312757969,
+            -0.014332910999655724,
+            0.006893736310303211,
+            -0.016469212248921394,
+            0.002954458352178335,
+            -0.015173795633018017,
+            -0.016499513760209084,
+            -0.00682934420183301,
+            0.00775355938822031,
+            0.021575121209025383,
+            0.006541473791003227,
+            0.021484214812517166,
+            0.005469535943120718,
+            0.028680972754955292,
+            0.0003953482082579285,
+            0.003041577059775591,
+            -8.958956641436089e-06,
+            0.003232859307900071,
+            0.019347915425896645,
+            0.006571775767952204,
+            0.018984289839863777,
+            -0.008742166683077812,
+            0.010514841414988041,
+            -0.010355755686759949,
+            0.010014856234192848,
+            0.006333146709948778,
+            -0.006473293993622065,
+            -0.003799130441620946,
+            -0.010348179377615452,
+            -0.0031419529113918543,
+            -0.016544966027140617,
+            0.0027366618160158396,
+            0.00693161366507411,
+            0.0071588801220059395,
+            -0.011582992039620876,
+            -0.005230906419456005,
+            -0.014704111963510513,
+            -0.0064354161731898785,
+            -0.01831764169037342,
+            0.014385939575731754,
+            0.029226411134004593,
+            -0.016332851722836494,
+            -0.01977214403450489,
+            -0.007772498298436403,
+            0.0007296186522580683,
+            -0.006636167876422405,
+            -0.01240872498601675,
+            -0.009151245467364788,
+            -0.035059571266174316,
+            -0.017666146159172058,
+            0.007586897816509008,
+            0.02484775334596634,
+            0.01387837901711464,
+            -0.013628385961055756,
+            -0.0006983695784583688,
+            -0.016984347254037857,
+            -0.010787560604512691,
+            -0.01535560842603445,
+            -0.02054484933614731,
+            -0.02498411200940609,
+            0.0077952248975634575,
+            -0.004033972043544054,
+            0.014931377954781055,
+            0.008113397285342216,
+            0.0032631612848490477,
+            -0.004371083341538906,
+            -0.0024109138175845146,
+            -0.013196581043303013,
+            0.0012925754999741912,
+            -0.004787737503647804,
+            -0.0017291157273575664,
+            0.03554440662264824,
+            -0.008098246529698372,
+            -0.005871038883924484,
+            0.011060279794037342,
+            0.011128460057079792,
+            -0.013764745555818081,
+            0.0006642796797677875,
+            -0.011249667964875698,
+            0.021847840398550034,
+            -0.021620575338602066,
+            0.019257009029388428,
+            0.0014507147716358304,
+            -0.010840589180588722,
+            0.028999146074056625,
+            0.01892368495464325,
+            0.00743159931153059,
+            0.006579351611435413,
+            0.01672677882015705,
+            0.0044316877610981464,
+            0.016196493059396744,
+            -0.0026552248746156693,
+            -0.004632438998669386,
+            0.009295180439949036,
+            0.015681356191635132,
+            -0.000702630786690861,
+            -0.006942977197468281,
+            0.016438908874988556,
+            0.004056698642671108,
+            0.0075414446182549,
+            -0.01356020662933588,
+            -0.008961857296526432,
+            0.001507531269453466,
+            0.006257391534745693,
+            -0.005939218681305647,
+            -0.0026003022212535143,
+            0.005530139897018671,
+            0.003943065647035837,
+            -0.013302638195455074,
+            0.01689344085752964,
+            -0.016166189685463905,
+            0.008742166683077812,
+            -0.0037972365971654654,
+            0.0121360057964921,
+            -0.006799042224884033,
+            0.0013153020991012454,
+            -0.01818128302693367,
+            -0.003869204083457589,
+            0.0038597346283495426,
+            0.012173883616924286,
+            -0.01262841559946537,
+            0.025484098121523857,
+            0.008166425861418247,
+            0.006602078210562468,
+            0.0059467945247888565,
+            -0.012855681590735912,
+            0.027211319655179977,
+            0.005969521123915911,
+            0.009886072017252445,
+            -0.008893677033483982,
+            0.011136035434901714,
+            0.035362593829631805,
+            0.03060515783727169,
+            0.014135947450995445,
+            0.004469565115869045,
+            -0.018999440595507622,
+            0.015120766125619411,
+            0.0018427487229928374,
+            0.0023806116078048944,
+            0.00863610953092575,
+            -0.011348150670528412,
+            0.01305264513939619,
+            -0.009484568610787392,
+            0.015241974964737892,
+            -0.021272100508213043,
+            -0.011749654076993465,
+            -0.014408666640520096,
+            -0.021075136959552765,
+            -0.01430260855704546,
+            0.034180812537670135,
+            -0.002346521709114313,
+            -0.010393633507192135,
+            0.0045112306252121925,
+            0.008477022871375084,
+            -0.006079366430640221,
+            0.0053142374381423,
+            -0.008204303681850433,
+            -0.013749594800174236,
+            -0.01889338344335556,
+            0.006628592498600483,
+            -0.01550711877644062,
+            0.007651289459317923,
+            -0.003304826794192195,
+            0.0037688282318413258,
+            0.007014944683760405,
+            0.018105527386069298,
+            0.00530666159465909,
+            -0.00135412672534585,
+            0.0014033677289262414,
+            0.0016865032957866788,
+            -0.00931033119559288,
+            0.0108330138027668,
+            0.015408637002110481,
+            -0.0011921997647732496,
+            -0.011279970407485962,
+            -0.009136094711720943,
+            -0.042877525091171265,
+            0.0006264019757509232,
+            -0.013128400780260563,
+            0.0064354161731898785,
+            -0.011196639388799667,
+            0.0022234192583709955,
+            0.015863168984651566,
+            -0.0054051438346505165,
+            -0.0010918239131569862,
+            0.02193874679505825,
+            0.013014768250286579,
+            0.021075136959552765,
+            0.02065090648829937,
+            0.014188976027071476,
+            -0.015302578918635845,
+            -0.03469594568014145,
+            0.002221525413915515,
+            -0.010030006989836693,
+            0.013772320933640003,
+            0.004238511435687542,
+            0.016060132533311844,
+            -0.023893235251307487,
+            -0.011878437362611294,
+            -0.006662682630121708,
+            0.014431392773985863,
+            0.01491622719913721,
+            -0.0032271775417029858,
+            0.02690829709172249,
+            0.009280029684305191,
+            0.0020510759204626083,
+            -0.02281750924885273,
+            0.003312402404844761,
+            0.005321812815964222,
+            -0.008370965719223022,
+            -0.016681326553225517,
+            -0.0004552422615233809,
+            -0.0012168202083557844,
+            -0.0007817004225216806,
+            -0.021650876849889755,
+            0.018832778558135033,
+            -0.004018820822238922,
+            0.0037574649322777987,
+            0.013840501196682453,
+            0.01516621932387352,
+            0.017984317615628242,
+            0.008370965719223022,
+            0.001497115008533001,
+            -0.006605865899473429,
+            -0.0007973249885253608,
+            -0.0005828426801599562,
+            -0.007389933802187443,
+            -0.005598319694399834,
+            0.006382387597113848,
+            -0.01134057529270649,
+            0.008810346014797688,
+            0.000896753859706223,
+            0.006113456096500158,
+            0.010605747811496258,
+            -0.004836978856474161,
+            -0.011628445237874985,
+            0.014628356322646141,
+            -0.0010548931313678622,
+            -0.0055755930952727795,
+            -0.011060279794037342,
+            0.003624893259257078,
+            0.013575357384979725,
+            -0.003096499713137746,
+            -0.013499601744115353,
+            0.00910579226911068,
+            0.002009410411119461,
+            -0.027438584715127945,
+            0.015499543398618698,
+            0.010295150801539421,
+            -0.013370818458497524,
+            0.004980913829058409,
+            -0.003371112747117877,
+            -0.03027183562517166,
+            0.004791525658220053,
+            0.002660906407982111,
+            -0.010643625631928444,
+            0.028938541188836098,
+            -0.003821857040748,
+            0.029256712645292282,
+            -0.004980913829058409,
+            -0.006905099377036095,
+            -0.011355726048350334,
+            0.009416389279067516,
+            -0.018681267276406288,
+            -0.0037953427527099848,
+            -0.008727015927433968,
+            -0.00915882084518671,
+            -0.0009838725673034787,
+            -0.005185453221201897,
+            0.006984642706811428,
+            0.005121061112731695,
+            -0.012704171240329742,
+            0.006757376715540886,
+            -0.010257272981107235,
+            -0.00020856392802670598,
+            0.0006718551740050316,
+            -0.008408842608332634,
+            0.013885954394936562,
+            0.005996035411953926,
+            -0.008340663276612759,
+            -0.0005032995832152665,
+            0.027817361056804657,
+            0.05051366239786148,
+            -0.018863080069422722,
+            0.0009270560112781823,
+            0.01452229917049408,
+            -0.0028824908658862114,
+            -0.0021590273827314377,
+            -0.011976920068264008,
+            0.005821797996759415,
+            -0.019423669204115868,
+            0.013317788951098919,
+            0.006473293993622065,
+            -0.008833073079586029,
+            0.003202557098120451,
+            -0.016711628064513206,
+            0.002621134975925088,
+            -0.0018058179412037134,
+            -0.018681267276406288,
+            -0.015529844909906387,
+            -0.012060250155627728,
+            -0.024181105196475983,
+            0.0004235197266098112,
+            -0.0008546149474568665,
+            -0.013870803639292717,
+            0.006579351611435413,
+            0.008431569673120975,
+            0.032483890652656555,
+            0.0011912527261301875,
+            0.010128489695489407,
+            -2.1513333194889128e-05,
+            -0.0010416359873488545,
+            -0.005988460034132004,
+            -0.01523439958691597,
+            0.00015387803432531655,
+            -0.002342734020203352,
+            0.022590242326259613,
+            0.005549078807234764,
+            0.017560089007019997,
+            0.0017887729918584228,
+            -0.016514664515852928,
+            -0.00596573343500495,
+            -0.007408872712403536,
+            -0.018060073256492615,
+            0.0062649669125676155,
+            -0.005473323632031679,
+            0.009605777449905872,
+            -0.015863168984651566,
+            -0.02731737680733204,
+            0.00792400911450386,
+            0.012234487570822239,
+            0.028514310717582703,
+            -0.0035623949952423573,
+            0.001716805505566299,
+            -0.005533927585929632,
+            0.016408607363700867,
+            0.032180868089199066,
+            -0.0008550884085707366,
+            0.006950552575290203,
+            -0.019999410957098007,
+            0.0017404790269210935,
+            0.013704141601920128,
+            -0.008401267230510712,
+            -0.01821158453822136,
+            0.003939277958124876,
+            -0.020635755732655525,
+            0.004583198111504316,
+            -0.020393338054418564,
+            0.030408194288611412,
+            -0.0023522034753113985,
+            0.008105821907520294,
+            0.004310478921979666,
+            0.01760554127395153,
+            -0.0031646795105189085,
+            0.010545143857598305,
+            0.013916256837546825,
+            0.014908651821315289,
+            0.0009980766335502267,
+            -0.011189064010977745,
+            -0.03790797293186188,
+            0.001859793672338128,
+            0.00817400123924017,
+            0.018635815009474754,
+            -0.01980244740843773,
+            0.011681473813951015,
+            -0.01296931505203247,
+            -0.020363036543130875,
+            -0.01003758329898119,
+            0.011575416661798954,
+            -0.006139970384538174,
+            0.03378688171505928,
+            -0.005874827038496733,
+            -0.008204303681850433,
+            0.005136212334036827,
+            0.010454237461090088,
+            0.0066778333857655525,
+            0.014704111963510513,
+            0.0326354019343853,
+            0.010196669027209282,
+            0.014173824340105057,
+            -0.004859705455601215,
+            0.02399929240345955,
+            -0.003931702114641666,
+            0.003954428713768721,
+            0.00509454682469368,
+            -0.022832660004496574,
+            -0.0040983641520142555,
+            -0.003931702114641666,
+            -0.016711628064513206,
+            -0.007458113599568605,
+            0.0032366469968110323,
+            0.008802770636975765,
+            -0.014045040123164654,
+            -0.00649980828166008,
+            -0.01350717805325985,
+            0.01218903437256813,
+            0.002730980049818754,
+            0.018302490934729576,
+            -0.029802151024341583,
+            0.009825468063354492,
+            -0.017484333366155624,
+            -0.017044952139258385,
+            0.006511171814054251,
+            -0.0038332203403115273,
+            0.006855858489871025,
+            -0.018272189423441887,
+            -0.01318900566548109,
+            0.012817803770303726,
+            0.0007897494360804558,
+            0.013060221448540688,
+            -0.009825468063354492,
+            0.0009706153068691492,
+            0.006064215209335089,
+            -0.009833043441176414,
+            -0.010961798019707203,
+            -0.007977037690579891,
+            -0.009681533090770245,
+            -0.008848223835229874,
+            0.005621046293526888,
+            0.01656011864542961,
+            -0.013742019422352314,
+            -0.003685497445985675,
+            0.016135888174176216,
+            0.006416477728635073,
+            -0.008166425861418247,
+            0.004075637552887201,
+            -0.003793448908254504,
+            -9.948806837201118e-05,
+            0.0013740125577896833,
+            0.012439027428627014,
+            -0.0017632056260481477,
+            0.024665940552949905,
+            -0.02654467150568962,
+            0.007598260883241892,
+            -0.0054051438346505165,
+            0.005446809343993664,
+            0.018226735293865204,
+            -0.02366596832871437,
+            -0.013082947582006454,
+            0.006276329979300499,
+            -0.01966608688235283,
+            -0.008060368709266186,
+            0.0054051438346505165,
+            -0.00016405766655225307,
+            -0.019923655316233635,
+            -0.012923861853778362,
+            -0.034786853939294815,
+            -0.02309022843837738,
+            0.024665940552949905,
+            0.008401267230510712,
+            -0.0073672072030603886,
+            0.018060073256492615,
+            -0.018438849598169327,
+            0.0172722190618515,
+            -0.002225313102826476,
+            -0.006613441277295351,
+            0.003935490269213915,
+            0.007674016058444977,
+            0.025362888351082802,
+            0.002653330797329545,
+            0.008264907635748386,
+            -0.0042082094587385654,
+            -0.014931377954781055,
+            -0.006321783177554607,
+            0.01980244740843773,
+            0.02471139281988144,
+            0.018605511635541916,
+            -0.018575210124254227,
+            0.010742107406258583,
+            -0.011749654076993465,
+            0.01818128302693367,
+            -0.006537686102092266,
+            0.00039037675014697015,
+            0.01859036087989807,
+            0.007969462312757969,
+            0.025241680443286896,
+            0.019923655316233635,
+            0.006852070800960064,
+            0.014128371141850948,
+            0.011908739805221558,
+            0.001204509986564517,
+            -0.012832955457270145,
+            -0.01154511421918869,
+            -0.02840825356543064,
+            0.042847223579883575,
+            0.0025434857234358788,
+            0.008317936211824417,
+            -0.027635548263788223,
+            -0.00021389048197306693,
+            -0.0034942151978611946,
+            -0.018832778558135033,
+            0.005939218681305647,
+            0.03260510042309761,
+            0.03284751623868942,
+            0.021438762545585632,
+            -2.888172457460314e-05,
+            0.0012234487803652883,
+            0.0329081192612648,
+            -0.013673839159309864,
+            -0.00937851145863533,
+            -0.01034060399979353,
+            -0.018393397331237793,
+            -0.006852070800960064,
+            -0.0032423287630081177,
+            -0.01031030248850584,
+            -0.0061172437854111195,
+            -0.009272454306483269,
+            0.0238629337400198,
+            0.0001374249259242788,
+            -0.0022594030015170574,
+            0.006605865899473429,
+            -0.020878173410892487,
+            0.011537538841366768,
+            -0.02813553437590599,
+            -0.013431422412395477,
+            -0.008651260286569595,
+            -0.006950552575290203,
+            -0.018802475184202194,
+            0.008643684908747673,
+            -0.006942977197468281,
+            0.021438762545585632,
+            0.002914686920121312,
+            -0.023590214550495148,
+            0.007530081085860729,
+            0.015317730605602264,
+            -0.0003825644962489605,
+            0.014037464745342731,
+            -0.003890036838129163,
+            0.009060339070856571,
+            0.0045112306252121925,
+            0.011681473813951015,
+            0.016954045742750168,
+            0.010651201009750366,
+            0.009030036628246307,
+            0.003437398700043559,
+            -0.001167579204775393,
+            0.01878732442855835,
+            -0.0007163614500313997,
+            -0.03405960276722908,
+            2.9236827685963362e-05,
+            -0.014628356322646141,
+            0.029499130323529243,
+            -0.009795165620744228,
+            0.009514871053397655,
+            0.008105821907520294,
+            0.007098275702446699,
+            -0.0067119235172867775,
+            0.007916433736681938,
+            -0.008742166683077812,
+            0.0052763596177101135,
+            0.015832867473363876,
+            -0.0042877523228526115,
+            -0.0014980619307607412,
+            0.007840678095817566,
+            0.00407184986397624,
+            -0.005454384721815586,
+            -0.019711541011929512,
+            -0.012287517078220844,
+            0.00726114958524704,
+            -0.02065090648829937,
+            -0.004367295652627945,
+            0.01161329448223114,
+            -0.005454384721815586,
+            -0.009620928205549717,
+            0.01393140759319067,
+            0.02136300690472126,
+            -0.002382505452260375,
+            0.022893264889717102,
+            -0.01349202636629343,
+            -0.009833043441176414,
+            -0.007984613068401814,
+            -0.004374871030449867,
+            -0.041120000183582306,
+            0.025984082370996475,
+            0.01068150345236063,
+            -0.006871009711176157,
+            0.01895398646593094,
+            0.0022499337792396545,
+            -0.002954458352178335,
+            -0.0011041341349482536,
+            0.00010499217023607343,
+            -0.03281721472740173,
+            0.03660498186945915,
+            -0.007007369305938482,
+            0.0005449649761430919,
+            0.010272424668073654,
+            0.042029064148664474,
+            -0.004590773954987526,
+            -0.005859675817191601,
+            0.01225721463561058,
+            0.005336964037269354,
+            0.031938452273607254,
+            -0.010416359640657902,
+            -0.014340486377477646,
+            -0.01815097965300083,
+            -0.004833190701901913,
+            -0.01939336769282818,
+            -0.0038313264958560467,
+            0.014037464745342731,
+            -0.019650936126708984,
+            -0.011242092587053776,
+            -0.001287840772420168,
+            -0.012537509202957153,
+            -0.0010747789638116956,
+            0.008598231710493565,
+            0.017832808196544647,
+            -0.011348150670528412,
+            -0.004450626205652952,
+            -2.2904154320713133e-05,
+            -0.005003640428185463,
+            0.0185146052390337,
+            0.0020056227222085,
+            -0.01702980138361454,
+            0.01321930717676878,
+            -0.00526499655097723,
+            0.006749800872057676,
+            0.0074429623782634735,
+            0.007939159870147705,
+            0.05145302787423134,
+            0.010317877866327763,
+            -0.001339922659099102,
+            -0.01607528328895569,
+            0.023196285590529442,
+            -0.01749948412179947,
+            -0.014181400649249554,
+            -0.02119634486734867,
+            -0.0029923359397798777,
+            -0.007060397882014513,
+            -0.028014326468110085,
+            -9.137987944995984e-05,
+            0.0016221113037317991,
+            -0.0024980325251817703,
+            -0.010045158676803112,
+            0.0005208179936744273,
+            -0.011529963463544846,
+            -0.006954340264201164,
+            -0.008545203134417534,
+            -0.016044981777668,
+            0.02240842953324318,
+            -0.021817538887262344,
+            -0.008855799213051796,
+            0.011514811776578426,
+            0.03118089959025383,
+            -0.005011215806007385,
+            -0.014173824340105057,
+            -0.019817598164081573,
+            0.0025643182452768087,
+            -0.01763584464788437,
+            0.012181458994746208,
+            -0.01225721463561058,
+            0.015045011416077614,
+            -0.03142331540584564,
+            -0.006079366430640221,
+            -0.008992158807814121,
+            0.012840530835092068,
+            0.019817598164081573,
+            -0.0025093958247452974,
+            0.00606800289824605,
+            -0.010295150801539421,
+            0.030211230739951134,
+            -0.00012286569108255208,
+            -0.018605511635541916,
+            0.00873459130525589,
+            0.007977037690579891,
+            0.012537509202957153,
+            -0.004435475450009108,
+            0.00257757562212646,
+            0.011567841283977032,
+            0.012742049060761929,
+            -0.01634800247848034,
+            -0.003174148965626955,
+            -0.017969166859984398,
+            0.005996035411953926,
+            0.006999793462455273,
+            0.021711481735110283,
+            -0.0013550736475735903,
+            0.018272189423441887,
+            -0.01563590206205845,
+            0.00019412307301536202,
+            -0.011749654076993465,
+            -0.0082270298153162,
+            -0.0028237802907824516,
+            -0.003958216868340969,
+            0.006685409229248762,
+            -0.028211290016770363,
+            -0.01274962443858385,
+            0.005287723150104284,
+            -0.004992276895791292,
+            -0.03405960276722908,
+            0.0053710537031292915,
+            -0.02725677192211151,
+            0.013128400780260563,
+            -0.002397656673565507,
+            -0.017847958952188492,
+            -0.00030846628942526877,
+            0.026332557201385498,
+            -0.003677921835333109,
+            -0.012537509202957153,
+            0.015544996596872807,
+            -0.006757376715540886,
+            -0.02119634486734867,
+            -0.005420294590294361,
+            0.014658658765256405,
+            -0.0010719380807131529,
+            -0.012393574230372906,
+            -0.029635490849614143,
+            0.012992041185498238,
+            0.0026874206960201263,
+            -0.007386146113276482,
+            0.010893618687987328,
+            0.025953780859708786,
+            0.008151275105774403,
+            -0.005924067925661802,
+            0.0015757111832499504,
+            0.0049581872299313545,
+            0.00300938100554049,
+            0.020287280902266502,
+            0.02728707529604435,
+            -0.008014915511012077,
+            -0.0011145505122840405,
+            -0.014954105019569397,
+            0.015802564099431038,
+            0.0013020449550822377,
+            0.010507266037166119,
+            0.021893294528126717,
+            0.007045246660709381,
+            -0.005738467443734407,
+            0.003905187826603651,
+            -0.023135682567954063,
+            0.01420412678271532,
+            -0.012272365391254425,
+            0.000622140767518431,
+            0.008749742060899734,
+            -0.007283876184374094,
+            0.01437078882008791,
+            -0.0015245762187987566,
+            0.0339989997446537,
+            0.025211378931999207,
+            0.0043218424543738365,
+            0.0007362472242675722,
+            -0.028559764847159386,
+            0.003547244006767869,
+            0.008651260286569595,
+            0.014499573037028313,
+            -0.0076664406806230545,
+            0.006261179223656654,
+            -0.004355932120233774,
+            0.00959820207208395,
+            -0.0065490491688251495,
+            -0.015893470495939255,
+            0.006113456096500158,
+            0.0012689019786193967,
+            0.006795254070311785,
+            0.0023408401757478714,
+            -0.008863375522196293,
+            -0.024362917989492416,
+            -0.025984082370996475,
+            0.021984200924634933,
+            -0.03918066248297691,
+            0.0073293293826282024,
+            0.011234517209231853,
+            0.03324144333600998,
+            -0.008158850483596325,
+            -0.004980913829058409,
+            0.006276329979300499,
+            -0.016711628064513206,
+            -0.004147605039179325,
+            0.008204303681850433,
+            0.012385998852550983,
+            -0.002420383272692561,
+            0.00974971242249012,
+            0.01318900566548109,
+            -0.012378423474729061,
+            0.03660498186945915,
+            0.016514664515852928,
+            0.01467380952090025,
+            -0.004007457755506039,
+            0.01312082540243864,
+            0.0066172294318675995,
+            0.019044892862439156,
+            0.025165924802422523,
+            -0.017454031854867935,
+            -0.002365460619330406,
+            -0.02292356640100479,
+            0.024165954440832138,
+            0.00726114958524704,
+            0.004556683823466301,
+            -0.008310360834002495,
+            0.014870774000883102,
+            0.0017954016802832484,
+            0.0028199926018714905,
+            -0.006696772295981646,
+            -0.010173942893743515,
+            0.019241856411099434,
+            0.0034317169338464737,
+            0.014870774000883102,
+            4.8501173296244815e-05,
+            -0.005503625608980656,
+            0.006052851676940918,
+            0.014832896180450916,
+            -0.023590214550495148,
+            -0.00203592493198812,
+            -0.019484274089336395,
+            -0.03202936053276062,
+            0.009022461250424385,
+            0.0016135887708514929,
+            -0.007052822504192591,
+            0.03230207785964012,
+            -0.010083036497235298,
+            -0.019787294790148735,
+            -0.008067944087088108,
+            -0.013113250024616718,
+            -0.010469388216733932,
+            0.0008271536789834499,
+            0.003908975515514612,
+            -0.00012428610352799296,
+            -0.009317907504737377,
+            -0.005749830510467291,
+            0.01990850456058979,
+            -0.0030358952935785055,
+            -0.00753386877477169,
+            0.00868156272917986
+          ],
+          "embedding": [
+            0.0040707582550429606,
+            -0.010230625151989194,
+            -0.021520242101903186,
+            -0.010115756270640508,
+            0.026738734586159035,
+            0.01796370233223267,
+            -0.012245199029994153,
+            0.06506924494147726,
+            -0.0014087587222033442,
+            0.009300068719271429,
+            0.003787396973047584,
+            0.002818772639754174,
+            0.022129077062176306,
+            -0.021526709808445683,
+            -0.0034388135367225824,
+            0.02763761747879405,
+            0.013210013805395276,
+            0.025255872672296602,
+            0.00029011480909986703,
+            -0.05438974762290585,
+            0.016671640187490865,
+            -0.016426252259893328,
+            -0.056591180651012554,
+            0.009335648425523075,
+            0.006141624459418827,
+            0.02568943956694641,
+            0.019377132054042116,
+            0.017946464337078746,
+            -0.057184776985927686,
+            0.03208527139096687,
+            0.030132392063755547,
+            -0.003846032371791839,
+            -0.008376524427531415,
+            -0.0010513963058713435,
+            -0.02734868388724371,
+            -0.014879400370545479,
+            0.04383920959748825,
+            0.02563613659516289,
+            -0.020117467074438997,
+            0.0398016699155716,
+            0.029896497884197775,
+            -0.0004195844530067313,
+            -0.023302720788253625,
+            -0.001168337916577517,
+            0.009149021369661628,
+            -0.02508145721900992,
+            -0.004307594138480351,
+            0.00550940415564303,
+            0.014016998918718476,
+            -0.028607379616884317,
+            -0.0018317656587526954,
+            -0.01973464940415756,
+            0.03762738392056255,
+            0.015358106058583114,
+            -0.03931907682104936,
+            0.011519637298832444,
+            0.004482080558226775,
+            0.0747867063383242,
+            0.05550679018263077,
+            0.023243129601931176,
+            0.02671631686665238,
+            -0.009686146566573536,
+            -0.04088582189725495,
+            -0.0065344623594924085,
+            0.019617295999325712,
+            -0.004722087402575777,
+            -0.002871844114437583,
+            0.003935087265300519,
+            0.03229890413393201,
+            -0.02813577047904649,
+            -0.05124172132597234,
+            0.03325235407841968,
+            -0.008687655557217192,
+            -0.011027174838246948,
+            -0.003653553063547375,
+            -0.02350737945182988,
+            0.016223087939183468,
+            0.03789252356462714,
+            0.0034350623378603083,
+            0.00195158657888739,
+            0.011789712862232108,
+            -0.0004435439660769551,
+            -0.0037330456918125894,
+            0.0031552713243866858,
+            0.04465432992298031,
+            -0.0032465421571915712,
+            -0.0075366743861922745,
+            -0.059958752731410256,
+            0.016784616861008125,
+            -0.028734506065772967,
+            -0.03035575736273943,
+            0.02680390892611741,
+            0.02338368719038086,
+            -0.0030658375437784277,
+            -0.03754896050794851,
+            0.07005659523248504,
+            0.0020711575553570695,
+            0.013034425083076923,
+            -0.02316832262132332,
+            0.021332142030732894,
+            -0.0318756233020585,
+            0.02270782509507388,
+            0.01570810309454529,
+            0.004788124787612736,
+            0.026727250200142456,
+            -0.022380142778058276,
+            -0.008593910379436904,
+            0.014439754382387034,
+            0.015007697617383174,
+            0.07662531963612977,
+            0.011195998137723795,
+            -0.010809283538470098,
+            -0.042227822120219305,
+            0.003824320933275175,
+            -0.03797629427607094,
+            -0.02347390684453348,
+            -0.0526876440280276,
+            -0.017799861680287913,
+            0.01701482796266917,
+            0.002730395144917155,
+            0.03789451759899825,
+            0.038338545350313946,
+            0.028463789278406586,
+            0.015494583705011921,
+            -0.05700237266350479,
+            0.047895170359075166,
+            -0.0064406894534929406,
+            -0.0007954721295065988,
+            0.012543029839729358,
+            -0.004703334868741216,
+            -0.0060569679284068,
+            0.012324901254711192,
+            -0.019883935809784068,
+            -0.04253565942154215,
+            0.02164409276211115,
+            -0.0075367555415808635,
+            0.03617159147722157,
+            0.07567745988686005,
+            -0.04225977020036783,
+            -0.022548837589622003,
+            0.029811602145850144,
+            -0.02175792196819533,
+            0.02069861458115363,
+            -0.00941331829664397,
+            -0.035479639885943264,
+            0.020352552580119105,
+            0.003678588588550461,
+            0.06776105274336729,
+            -0.02346047721661687,
+            0.02721075076349443,
+            -0.029386781707644253,
+            0.023530831921117993,
+            -0.02077660533100372,
+            0.020022475323372906,
+            0.024195118676513715,
+            -0.028378518720494666,
+            -0.012900969023913307,
+            0.019264089970719624,
+            -0.0374983624629265,
+            -0.019726171557384355,
+            0.00979636085642416,
+            -0.014479528435832063,
+            -0.014533384583777662,
+            0.005904623758767224,
+            0.010344475500263744,
+            0.05659646358026115,
+            0.04163866113482093,
+            0.007152494819169405,
+            -0.07584500130974763,
+            0.05471858597878099,
+            0.03580211013427615,
+            0.044161429444475306,
+            0.02258180566046248,
+            0.00980779221334715,
+            0.0011193505603397395,
+            0.018086566768892507,
+            0.015605066702191871,
+            -0.00954727399396313,
+            -0.027950079146919375,
+            0.03028514025485407,
+            -0.006351669761257431,
+            0.005095804801686436,
+            0.009620040085743162,
+            0.019748001227987134,
+            0.033709535059510916,
+            0.008203559956940196,
+            0.007389752602218563,
+            0.04197771878351742,
+            -0.02067363758471765,
+            0.007543739188471913,
+            0.011207941261799052,
+            0.013964827251172813,
+            0.03839562075248879,
+            0.0009775244543292637,
+            0.006469507611492216,
+            -0.022808235395746654,
+            -0.024795004398633292,
+            0.006249024864814684,
+            0.011292877109066925,
+            0.01596839619826819,
+            -0.0014105396434174883,
+            -0.04152075438277094,
+            0.008267788411079337,
+            -0.028834191212891387,
+            0.030940526405027906,
+            -0.0018271056434014425,
+            0.006185475047820603,
+            -0.007920574674727123,
+            0.012907560753043616,
+            -0.015165704767187252,
+            -0.021962244496450727,
+            0.0037071053072833147,
+            -0.013508376064336042,
+            -0.015178823540513864,
+            -0.02518557053061311,
+            0.011693256916760944,
+            -0.01716015674592331,
+            0.02048897513447166,
+            -0.003412078966361745,
+            -0.001807298511378529,
+            -0.0012575655363779904,
+            0.03116103961636643,
+            0.011668533361641842,
+            -0.011637010548030312,
+            -0.040260195687925236,
+            -0.011129376691522074,
+            -0.0034493180842871908,
+            -0.02098244789711689,
+            0.026658743116016938,
+            -0.01923736325953796,
+            -0.009723937181300192,
+            -0.013134679989565051,
+            -0.011974548914818263,
+            -0.004028712301052159,
+            0.019628329175867303,
+            -0.028192034966932684,
+            -0.012638520793012082,
+            -0.011568652932013111,
+            -0.0025919910519445737,
+            0.012497908463405111,
+            0.015113186712897145,
+            0.014880607940212641,
+            -0.03059227859452334,
+            -0.021899716217105084,
+            -0.0008513506358489523,
+            0.011985931118464679,
+            -0.01321584595585435,
+            -0.011735288366213981,
+            0.010710683541435907,
+            0.01961722172525993,
+            -0.023571946726192585,
+            -0.03364281880125354,
+            0.0014460076743756465,
+            0.016563167828467187,
+            -0.02891589059260944,
+            0.04595166331401113,
+            0.034271553243501465,
+            -0.027557971577291153,
+            -0.007138821520412525,
+            -0.039508407807651844,
+            0.06814329636830853,
+            0.00022136123564675344,
+            -0.039718016705753735,
+            0.0503457594963753,
+            -0.003580581891576583,
+            0.013820796398521455,
+            0.02054050419088757,
+            0.035682661890720495,
+            -0.0013220739026852943,
+            -0.0052759394697328675,
+            0.04733018719162478,
+            -0.0187845855131196,
+            -0.03982699468608443,
+            -0.023905460465070062,
+            0.002631233133039351,
+            -0.03090995554989513,
+            -0.012617612051488022,
+            -0.013605787969473513,
+            0.039672783793764745,
+            0.008669582242097126,
+            0.009974277556736362,
+            0.005535636914843583,
+            0.014180668493093584,
+            0.012042659947850441,
+            0.011870115226306776,
+            0.00451259454348581,
+            -0.014023048592193627,
+            -0.007769033996392623,
+            -0.05715987527668358,
+            0.003374564518384728,
+            -0.028801158523506985,
+            -0.00756331369170698,
+            -0.02163054894055572,
+            0.005367181573233089,
+            0.02308705505078254,
+            -0.005438909398653553,
+            -0.00873620909426313,
+            -0.0036530652237719077,
+            0.013920407174296543,
+            -0.01539613379172909,
+            -0.03908211213269472,
+            -0.016810726548844992,
+            0.007460173854386443,
+            -0.007069189426742819,
+            -0.041197222498539215,
+            -0.003843233633335066,
+            0.004373367480223065,
+            0.016194121660459908,
+            -0.029165150066068787,
+            -0.00735139702658342,
+            0.017257895082337448,
+            0.014153313280536095,
+            -0.002376178042741926,
+            -0.023223700836427403,
+            -0.0319877537205969,
+            0.02088060774052704,
+            0.0012829290529515294,
+            0.03251550040316258,
+            -0.04128453069131847,
+            -0.023376114412063578,
+            0.04083022735076889,
+            0.005690491480259003,
+            -0.05810403800689653,
+            0.016582434816685452,
+            0.010402038964417569,
+            0.016795644054313566,
+            -0.00908590043098552,
+            0.05003656634510042,
+            -0.037216197954986735,
+            -0.013027960261454162,
+            -0.007090943677204207,
+            -0.05175371787508879,
+            -0.0020065660448063322,
+            -0.014908489552053589,
+            0.010627472758306387,
+            -0.03275473161158777,
+            -0.017334608434133866,
+            0.025321047439048318,
+            -0.0033515195870701753,
+            0.011526068468647207,
+            -0.026783879816504975,
+            -0.0012830935419413407,
+            -0.017506908439998053,
+            -0.02114853058440477,
+            0.004878884893669383,
+            -0.020936900837115968,
+            -0.0025587027781733642,
+            0.01557850246191957,
+            0.023652335794299834,
+            0.011489428313303988,
+            0.0019089137782552768,
+            0.02054907941022243,
+            0.014361232983780595,
+            -0.0056413109308506085,
+            -0.0010300050480036684,
+            -0.03413262505887213,
+            -0.018821160485010483,
+            0.03361989907065425,
+            0.013125889701529858,
+            -0.003742167496379789,
+            0.008813100942466033,
+            -0.0383265293638935,
+            -0.035316957263713485,
+            0.03020499323039515,
+            -0.05001623971243151,
+            -0.00855479667673597,
+            -0.005939674940109814,
+            -0.026451831943555135,
+            -0.008032873049527808,
+            0.0009431571685654594,
+            -0.014387605128858954,
+            0.016382559541755014,
+            -0.024909510232038502,
+            0.003715736505444385,
+            -0.006016712902409688,
+            -0.009110197692342108,
+            0.05507263160354303,
+            -0.022103031032598353,
+            -0.0475415595313825,
+            0.021958434235402655,
+            -0.016670809659931243,
+            0.0323519363801999,
+            0.03867494951405658,
+            0.011124508964363016,
+            -0.008002445714431074,
+            0.034041838454818577,
+            0.018400123235267234,
+            -0.01161754789471199,
+            -0.025901844576206774,
+            0.021800298132870245,
+            0.00943488425697727,
+            -0.05906691123008531,
+            -0.00782549274387971,
+            -0.019357446956797447,
+            0.008682376344965976,
+            0.02863077694455766,
+            0.0024925977373894447,
+            -0.002682170968419412,
+            -0.013590235283325244,
+            -0.021511149507762284,
+            -0.03255453916518719,
+            0.009039307015853905,
+            0.008305286476500875,
+            0.009312625027668458,
+            -0.0014144165766983945,
+            -0.007853869334376857,
+            0.009067065760942163,
+            0.022485201611182105,
+            0.03966189080765722,
+            -0.041627289519629344,
+            0.002707086456239785,
+            -0.022678178050044222,
+            -0.03653753142622053,
+            0.0036574646778835588,
+            0.029996160786576367,
+            0.02901163765559891,
+            -0.01971777193183787,
+            -0.0036589934841109884,
+            0.009181234938554578,
+            -0.025979483002227535,
+            -0.00773645103995767,
+            0.004280770859737866,
+            0.04116668301017644,
+            0.006201085381597264,
+            0.01402728258112097,
+            0.001060407317110527,
+            -0.00920801784000087,
+            0.02109880889040528,
+            0.025211022083982727,
+            -0.029866678264942617,
+            0.014439702646820226,
+            0.009379206296908577,
+            0.04156435812161122,
+            -0.037309655469525846,
+            -0.011819731348299495,
+            -0.02766498020409985,
+            -0.010454143230794476,
+            0.027911532296566614,
+            -0.04127890742215716,
+            -0.006948536183086887,
+            0.010207784113909938,
+            -0.0016584043356136647,
+            0.016437495428919636,
+            -0.014607861750686952,
+            -0.03222311919805017,
+            -0.000610723203118213,
+            -0.02867445198803213,
+            -0.024772816792038133,
+            -0.016510309753766612,
+            -0.018581201775667577,
+            -0.07450654873867177,
+            0.00026832248829154255,
+            -0.06724696990816072,
+            -0.02447002541264633,
+            0.04885569249258513,
+            -0.010195604207134053,
+            0.03332905935704749,
+            -0.02815272112271314,
+            -0.005296340304911926,
+            -0.05908835042526019,
+            0.0038476172058970774,
+            0.025753576863447866,
+            0.02386003857017594,
+            0.005206447371391696,
+            0.024799057871126032,
+            0.012060738268265606,
+            -0.04146503002980051,
+            -0.024949907556862643,
+            -0.014503475223770958,
+            -0.0016340466065744286,
+            -0.0034009167013795906,
+            -0.031298432995411005,
+            0.036241272149880994,
+            0.008902644391348814,
+            0.03550223922667804,
+            0.04463301409716232,
+            -0.010385984413759852,
+            -0.03305004479734889,
+            -0.006952739338820577,
+            -0.0175981588092108,
+            0.042668533202572856,
+            -0.017112184620086308,
+            0.005149173014526686,
+            -0.03219589107997635,
+            0.024181808150698054,
+            0.0014339790422945406,
+            0.01254721190611376,
+            0.011099458649404119,
+            0.0027626836767364253,
+            -0.026883640446090158,
+            0.0024160770766463607,
+            0.023874514847956222,
+            -0.022558433617669373,
+            0.00917667715473093,
+            -0.006754091166465801,
+            -0.033686935718776144,
+            -0.01078389638118685,
+            -0.013595327486167114,
+            0.024712781935748277,
+            0.013168464807451798,
+            0.005875864604862434,
+            -0.02417814420558463,
+            -0.0023978800459714144,
+            0.028650463366555304,
+            0.020176980768301288,
+            -0.02118184201861616,
+            0.019082340279671355,
+            0.019734925740100277,
+            -0.04209645071102001,
+            -0.04217310001316092,
+            0.006553913719098956,
+            -0.003735719227468833,
+            0.0061793909220459,
+            -0.028285889794908054,
+            0.008181788076484304,
+            0.004432488437404309,
+            -0.01743905248466261,
+            0.041221269621866216,
+            -0.014892354419172107,
+            0.005087102208335201,
+            -0.022198383716186407,
+            -0.011936388003605221,
+            -0.00027487042024290816,
+            0.044599215801158845,
+            0.008068975171827777,
+            0.003496950921841073,
+            0.02238779276447619,
+            0.0054839808262270315,
+            0.04816723616939378,
+            -0.02685221260466647,
+            -0.0190356946337995,
+            0.016464136459560113,
+            -0.0176092780782677,
+            -0.012513274095432353,
+            -0.010695421445334002,
+            -0.004784348908457072,
+            -0.022203041334932165,
+            0.010788155380415583,
+            -0.018353897272137543,
+            -0.02277431189907027,
+            -0.003945465850796918,
+            0.016024639792926797,
+            -0.006248368166794934,
+            -0.0038769803892794875,
+            0.0016292556540434277,
+            0.02412864830338344,
+            0.02892955453819876,
+            -0.02408022470555968,
+            0.019190282372105023,
+            0.018941584034503536,
+            -0.006709516380602965,
+            -0.020956377933747532,
+            -0.01964885206488053,
+            0.01434623424964373,
+            -0.01352968586212152,
+            -0.013872556892707697,
+            0.004052798542613835,
+            0.009201147374299997,
+            -0.004086067985859617,
+            -0.04185829685685201,
+            -0.014668698123513235,
+            0.003179403563664022,
+            -0.0006415014901450319,
+            0.019360184344240947,
+            -0.0150817538567364,
+            -0.029771999278536083,
+            0.014221776097046751,
+            0.022944884879589678,
+            -0.04493425220030206,
+            0.02117577719533236,
+            -0.0030648751497261983,
+            -0.00016251533643103604,
+            -3.817296018601414e-05,
+            0.008727767382077138,
+            0.026914779443266063,
+            0.03456565094655899,
+            0.01974806772696219,
+            0.00450800007152441,
+            -0.008952599700759673,
+            0.025184190461155245,
+            0.009846044083064704,
+            -0.025168735183134738,
+            0.010983353565382029,
+            0.016740721743326522,
+            0.0070838930012654296,
+            0.009173879407277282,
+            -0.006078288608766152,
+            -0.018269347893097643,
+            0.013181494188066667,
+            -0.01435693747022231,
+            0.01678907949532146,
+            0.05154463552784974,
+            -0.00039632596424461964,
+            -0.02461439550772063,
+            0.008418147037164186,
+            -0.006544319842335593,
+            -0.010065129286193278,
+            0.02214628550460128,
+            0.0029515574751370417,
+            -0.030862152033847717,
+            0.007509556025223583,
+            0.00675485304374519,
+            0.01894936231559203,
+            -0.02355525295639599,
+            -0.01387885858544803,
+            -0.00431398020087544,
+            0.0021208056468763523,
+            0.01580766504149989,
+            -0.004739477121073512,
+            -0.013204823447960706,
+            0.02994098229517376,
+            0.025496728234662373,
+            -0.00026124946738213036,
+            0.015060064980768175,
+            0.022262345898533903,
+            0.009968408664790368,
+            0.007588431141088797,
+            -0.0083063413609779,
+            -0.031336272171937095,
+            0.0018957858383561667,
+            0.05800811417728381,
+            -0.014937085296865802,
+            0.04313875432235027,
+            0.02018476425923236,
+            0.015657644644987107,
+            0.017880263053253607,
+            -0.01662268082870562,
+            -0.008740053682875323,
+            -0.0017630394204179638,
+            -0.008294417990024647,
+            -0.027613464270562948,
+            -0.005953209556351086,
+            -0.02379961210881074,
+            -0.013878676554701543,
+            0.015418315245979704,
+            -0.014764728065382074,
+            0.0027420640870858457,
+            0.010395538988699663,
+            -0.006590633833835581,
+            -0.00841537209927409,
+            -0.003943772700797381,
+            0.03376338017637296,
+            0.011980354721458463,
+            0.027761929822019422,
+            -0.00011128539412582973,
+            0.009342430952107655,
+            -0.018419526564396063,
+            -0.006839752805318664,
+            0.03271190887864607,
+            0.006011520427944665,
+            0.004652835471939905,
+            0.01485920920301398,
+            0.014305626381398871,
+            0.038015677219487023,
+            -0.005102005401565164,
+            -0.018204460895989052,
+            0.02065495133597456,
+            -0.0049699013356374775,
+            0.019149036693562712,
+            0.008825981577388056,
+            -0.002660055848523568,
+            -0.004292379561953203,
+            0.03090270710860331,
+            -0.020243440437217503,
+            0.018735727616321387,
+            -0.02238236089688497,
+            -0.0276584413929402,
+            0.023028408906176946,
+            0.008320436733620496,
+            0.04047641118026783,
+            0.011463851066078041,
+            -0.02247143742166039,
+            -0.034588725413285185,
+            0.009827713930789184,
+            0.00337071450636392,
+            -0.024446280199879062,
+            0.017542777160283324,
+            0.00780836483490212,
+            0.008453886667283516,
+            0.016624116261219637,
+            -0.004175682824655864,
+            -0.0042784362933758835,
+            -0.001546656717896957,
+            -0.008819739495525477,
+            0.01406790007187259,
+            -0.015487658289541583,
+            -0.021457182907140035,
+            0.024687169583245905,
+            -0.03915911962859764,
+            0.01771290139419629,
+            -0.006510046907173026,
+            -0.00997450880725535,
+            -0.02059560682587115,
+            1.5939836949198004e-05,
+            -0.007159493752862143,
+            -0.0004782363814453761,
+            -0.017503181957042913,
+            0.016643971600145645,
+            0.024056442566869802,
+            -0.013209211443909658,
+            -0.00445521990927108,
+            0.0020313626821877526,
+            -0.029235692455820056,
+            0.006662209097255912,
+            0.016993027053110635,
+            0.00529940612845993,
+            0.0026104743851393716,
+            0.0001707934714983891,
+            -0.031289446611163674,
+            -0.005782208620284939,
+            0.03053516889069683,
+            0.05590704747438521,
+            0.021628254147289437,
+            0.02403314981042378,
+            0.02908606663381421,
+            0.02717206823892633,
+            -0.003198844932040613,
+            0.0016217508103442348,
+            -0.01620660606431983,
+            0.027098323143010743,
+            -0.011664168290676943,
+            0.04322175021589083,
+            0.008652323184522541,
+            -0.049493020736445056,
+            0.004537958157696794,
+            0.00943239454147692,
+            0.034918603888916544,
+            0.016678716235690318,
+            0.0005887414054803805,
+            -0.005894338664998101,
+            0.03279165663336323,
+            -0.02735726709032382,
+            -0.011962070182991011,
+            -0.00036014407560325284,
+            -0.019079003730607717,
+            -0.015938295169860884,
+            0.010357611943935459,
+            0.04278924678845588,
+            0.0009957483598272451,
+            -0.0071422790468850215,
+            0.01571201177743999,
+            -0.009445307674383,
+            0.02286338492075306,
+            0.007336440384050498,
+            0.0042809497342459904,
+            0.025581467690095977,
+            -0.0025007871391868105,
+            0.02300539823997411,
+            -0.000628898771317254,
+            0.004582863909134464,
+            -0.015960578501883316,
+            -0.00930649396114688,
+            0.027838455754803444,
+            -0.011112268986721,
+            -0.029253746504396393,
+            -0.008734785561831333,
+            0.00472579547270089,
+            -0.0003805825712986138,
+            0.0021854358149324227,
+            -0.01183885533661493,
+            -0.010840951201591629,
+            0.019965691788904057,
+            -0.004922943301424112,
+            0.003800048754664761,
+            0.012664049429916252,
+            -0.0030124219674414897,
+            -0.01910844996750822,
+            0.01889983984736845,
+            -0.003858283778519141,
+            -0.024168821338992813,
+            0.020083955711766945,
+            0.01850497889545421,
+            0.045156567155119454,
+            -0.01586824055785609,
+            0.004612757419198795,
+            0.017497774574602505,
+            -0.002172088240279357,
+            -0.019617327081566827,
+            -0.014944795772093997,
+            -0.018212462639041492,
+            -0.015975283672064702,
+            0.012837956594691596,
+            -0.01243789877725244,
+            0.019500571353639314,
+            -0.0380735226388175,
+            0.012590900534826689,
+            0.004832168601136412,
+            0.007505385730121269,
+            0.016011983211212992,
+            0.025994560026860358,
+            0.002157436199771672,
+            -0.033465476168065784,
+            -0.01014599891204166,
+            0.007845373376916211,
+            0.010093432683262838,
+            -0.002849185178237036,
+            -0.018147004183434554,
+            0.003633790611809425,
+            -0.0266147420637404,
+            -0.018644422659309035,
+            0.022926737299761104,
+            -0.04798772015599113,
+            0.017908415301694906,
+            0.024859125158586466,
+            -0.04343699525707025,
+            -0.04886699607556352,
+            -0.011459188432595229,
+            -0.036774381316312464,
+            0.0014128005215813578,
+            -0.0005256067584181517,
+            0.011194992852264102,
+            -0.009342140079829378,
+            -0.018598671981710183,
+            0.01584789414301952,
+            -0.0036412641242200418,
+            -0.004225840239935549,
+            -0.00616696736018045,
+            -0.0025598631463178384,
+            -0.010188803157718543,
+            -0.03244694662411977,
+            -0.03290891136780909,
+            -0.00033742479296272127,
+            -0.009743927051414455,
+            -0.02334288359945272,
+            -0.03447062782872289,
+            -0.008728968107967028,
+            -0.037753659749578956,
+            -0.00515561961442376,
+            0.04070890150714597,
+            0.027517147608335957,
+            0.009449414978133334,
+            0.008086900089929912,
+            -0.013989034604490105,
+            0.05691047488419298,
+            -0.020425883381389086,
+            0.039466620642512155,
+            0.005301493569940605,
+            -0.0037222353899918703,
+            0.013075425214878277,
+            0.027710826422351564,
+            -0.006253665106995653,
+            -0.0017412928073238833,
+            -0.01928096630475438,
+            -0.012920476859996416,
+            0.009444408011467955,
+            0.015799884767184885,
+            -0.0025083061619197524,
+            0.001274457251739847,
+            0.00930533161700852,
+            -0.0006652478540264343,
+            0.017258038491335247,
+            0.03996268817486439,
+            0.003833844612473671,
+            -0.00826404141706295,
+            -0.018822002570963614,
+            -0.003188844900317347,
+            -0.011807006674248829,
+            0.010482058734260632,
+            -0.015410635475145028,
+            -0.011540803927522013,
+            -0.01249428777269578,
+            -0.017081856786754605,
+            -0.002183034976477535,
+            0.003114834030705039,
+            0.0027475411722881747,
+            0.02261627635192778,
+            -0.015872027027080824,
+            -0.022487581027490516,
+            -0.0035823117885461506,
+            -0.02398778449664045,
+            8.13716819067491e-05,
+            -0.01050060782407259,
+            -0.015616181283381218,
+            0.013676480044963827,
+            -0.021513173575512936,
+            -0.01091399089076612,
+            -0.008699176177861256,
+            -0.0006986953850759474,
+            0.005853723710671249,
+            -0.00993355748621621,
+            -0.00010241843582159751,
+            0.01150872694850241,
+            0.005559552266827916,
+            -0.012336449703762907,
+            0.0039431718621504,
+            -0.012740550272925562,
+            -0.004432059174951158,
+            -0.03592309122711106,
+            -0.019665686718735042,
+            -0.01846795637714379,
+            0.005642094448288269,
+            -0.005108964028543505,
+            -0.020586678054151854,
+            0.00668364061048941,
+            0.0034775127050453774,
+            0.004713169526661792,
+            -0.010723667875209928,
+            0.013944735241748791,
+            0.010882420168586074,
+            -0.003518245916227596,
+            0.013827334494230662,
+            -0.001330399936559009,
+            -0.020286697504699764,
+            -0.011139922402071095,
+            0.03538258948179278,
+            -0.024961826464903997,
+            -0.0018019328351763667,
+            -0.0026423695690168066,
+            0.00597003516870726,
+            -0.03071835946501977,
+            -0.013634212232744325,
+            0.02164400349828422,
+            -0.005846552894831441,
+            -0.026367014426434226,
+            0.03399116480234033,
+            0.03473258231629753,
+            0.010041605592929843,
+            -0.013423384340589794,
+            0.04730054254696893,
+            -0.012968978954938308,
+            -0.018274800841400665,
+            0.026826644936918647,
+            0.0278389611094493,
+            0.02525164095926168,
+            -0.029782550447890466,
+            -0.003010713086487275,
+            -0.02393629084357885,
+            0.017993366287762866,
+            -0.0263476663180971,
+            0.01864011499179424,
+            -0.02332263152537849,
+            0.016014908881756034,
+            0.01868464825773604,
+            -0.003104041044182249,
+            0.026313491086684425,
+            -0.0052945248919367815,
+            0.015253731232325074,
+            0.0012371308759955573,
+            -0.02119765517013431,
+            0.007795466385020642,
+            -0.006469500272298062,
+            0.025762169721880456,
+            0.00833992768202466,
+            0.01801901639978976,
+            0.022019940825351116,
+            -0.0409522298213741,
+            0.02003698451081499,
+            -0.001367916864679543,
+            0.01782984895971748,
+            0.0038516134262842373,
+            -0.007885432294782304,
+            0.014221728513400065,
+            0.010991352188016265,
+            -0.012851099261924757,
+            0.003840539873231741,
+            0.0008923482291826014,
+            -0.028928104542716937,
+            0.007066004768771198,
+            -0.026914574065060926,
+            0.025866907400807087,
+            -0.005526774784152086,
+            -0.00998122210956923,
+            -0.020744532673097317,
+            -0.0071425207262770865,
+            0.003754468896273156,
+            -0.012747874997429746,
+            -0.011385805601999295,
+            -0.024700457669046855,
+            -0.02401291382882952,
+            0.029591582802488962,
+            -0.014054332096344003,
+            -0.04015978298687865,
+            0.001234440863385445,
+            -0.023301255402129767,
+            0.01193125007773239,
+            -0.005793124129267758,
+            -0.024440100649464773,
+            -0.042563399082257154,
+            -0.017548888269389457,
+            -0.0009394887961485922,
+            0.022739763644266633,
+            -0.00114156949394707,
+            0.041791395064755515,
+            0.0008816696205437911,
+            0.022207132857533028,
+            -0.015547580543606755,
+            0.005319731264840819,
+            -0.02071229859314312,
+            -0.041495579742912285,
+            0.03660432378985971,
+            -0.014637031447181937,
+            0.025854844497435178,
+            0.03324325296704186,
+            -0.013009347371118843,
+            0.001354504885302625,
+            0.02652570709474851,
+            -0.014618100923440427,
+            -0.010782232747863238,
+            -0.0034936699515378212,
+            0.003461670978569965,
+            -0.006468116285021298,
+            -0.027793221545159466,
+            -0.007752458229193391,
+            0.04193706715422074,
+            0.017894069367616637,
+            0.021928994684423043,
+            0.022606401934291136,
+            -0.017972404868085358,
+            -0.026402048654300563,
+            0.003656646576244765,
+            -0.013129722990153284,
+            -0.018090360146368677,
+            -0.01624598856366761,
+            0.012397150789705746,
+            0.011741002980073786,
+            -0.004052422390675261,
+            -0.01836750423615859,
+            -0.020929335029248472,
+            -0.016327412417122416,
+            -0.0042615345226698015,
+            -0.008458117095535,
+            -0.005002582649405199,
+            -0.012158173174049797,
+            -0.020409616620022143,
+            0.03706368540614453,
+            0.0285565642978937,
+            -0.0041732933884846566,
+            -0.00607154864704767,
+            -0.011142150176334957,
+            0.008834431806990914,
+            0.006021232046087652,
+            -0.005028870471297621,
+            -2.3396043292139426e-05,
+            0.005112033372713154,
+            0.022822312435625314,
+            0.007271235299348001,
+            0.0035791789459533856,
+            -0.06340305811365254,
+            -0.005392000517722054,
+            0.022560201096078862,
+            0.012581401789917604,
+            -0.0032248219865223496,
+            0.025180718735801837,
+            0.0006451724964970927,
+            -0.009714514747463593,
+            -0.018389422219956937,
+            0.024728862150150004,
+            0.008760329201960964,
+            0.020988700992999768,
+            -0.014974763313686516,
+            0.003845297486923296,
+            0.0006445983847138896,
+            0.00750394530065325,
+            -0.0019792992847026793,
+            -0.008578254004439607,
+            0.025776677078006516,
+            0.016593438298233645,
+            -0.005695578040595484,
+            0.01604018433477423,
+            -0.0017497260488899657,
+            -0.021460543443428924,
+            -0.010265213095609554,
+            0.00042017209251926397,
+            -0.010570864288876387,
+            -0.018015790888295944,
+            0.017569414447505878,
+            -0.002601036282535863,
+            0.0072399377790164435,
+            0.030281707151249355,
+            0.0423183067569767,
+            0.009566493398534713,
+            -0.005928603293907002,
+            -0.0037936911251953654,
+            0.015477873304924939,
+            0.01513127995244932,
+            0.010741256034652754,
+            0.03015858543221727,
+            -0.029247492875021843,
+            -0.006690306960497428,
+            0.0046653727410352576,
+            0.004610405633429778,
+            -0.011692179099382171,
+            0.030055897497741527,
+            0.04569730592994177,
+            0.006736813244286893,
+            -0.004799978513117371,
+            0.002121067028052434,
+            0.011299889107780231,
+            0.007768137218140627,
+            -0.018927080714961783,
+            -0.0012222962555201746,
+            -0.006009911026251445,
+            -0.014001766533604134,
+            0.026941585318261085,
+            -0.003960837039351648,
+            -0.011848232223811548,
+            0.02903357274500332,
+            0.1105267435019781,
+            -0.006011029338781496,
+            -0.0053646998281195206,
+            -0.03242321704621517,
+            -0.015194102811332289,
+            -0.008860590919147765,
+            0.0056988205877963076,
+            0.01635957585050017,
+            0.020327661913559888,
+            -0.0008943408274661007,
+            0.016820860249955754,
+            0.02822175614376368,
+            -0.013854827839129885,
+            0.007906244895327178,
+            -0.030891381690403466,
+            0.015702117924362928,
+            -0.005383974926093977,
+            -0.012032955400313954,
+            -0.0026169511828034054,
+            -0.008618307035330928,
+            -0.0037356680273096105,
+            0.0009666430093455632,
+            0.011296307527684261,
+            0.004427516814276546,
+            -0.014484198313369681,
+            0.002829529156309784,
+            -0.008595300985863424,
+            -0.005122996529848575,
+            -0.02393571073523279,
+            -0.024119245543788293,
+            0.005223147047205755,
+            0.0019389305548629142,
+            6.344308292376579e-05,
+            0.006848965054539454,
+            -0.003810712095343454,
+            0.030149081317333418,
+            -0.01700322008272963,
+            0.0012288912810468446,
+            0.01193226302726573,
+            0.012332749933413344,
+            0.01688602002694633,
+            0.0028600984823761366,
+            -1.9882692552530084e-06,
+            0.021172977047420227,
+            -0.010514036509402561,
+            0.01460334896316005,
+            0.013091417303145243,
+            -0.04103572381458182,
+            0.0089746401463863,
+            -0.0027841289484449873,
+            -0.024252447218794344,
+            0.004446472209246425,
+            -0.018987867343500102,
+            -0.00881974688349601,
+            0.0005365625696166774,
+            0.012054719659248512,
+            0.01960300421041832,
+            -0.008553943699846696,
+            0.02111201386480852,
+            -0.0003501657740808554,
+            0.004177088753111459,
+            -0.012879944670340026,
+            0.0065222126975051685,
+            -0.007563354540402343,
+            0.0008168128607271199,
+            -0.018213241937597355,
+            0.012526561612175514,
+            -0.0008285725179894544,
+            -0.015737003159436623,
+            -0.011337688275309921,
+            0.004737346137976519,
+            -0.0027993545140114705,
+            0.0030843163114700313,
+            0.010153740340626154,
+            -0.008815302484351509,
+            0.004919275910399053,
+            -0.0017541042538165297,
+            0.002562816687714991,
+            0.014285249186421434,
+            -0.029816964593980453,
+            0.005357227738099325,
+            0.009154842495811638,
+            -0.00048782303639333943,
+            -0.013074976152495576,
+            -0.004054505559072866,
+            0.0028322102449485015,
+            -0.016877328079971877,
+            -0.012128199639779175,
+            0.023421330164419888,
+            -0.012565145848820333,
+            0.008609070188894315,
+            -0.010936598722317274,
+            -0.018401801734378692,
+            -0.01066119413133089,
+            -0.005254229780560808,
+            0.01234969508270852,
+            -0.0038427767204741877,
+            -0.006947594635952202,
+            -0.01756894543880097,
+            -0.007890897373987378,
+            -0.047246698568222466,
+            0.009617876521097624,
+            0.027157555263862723,
+            -0.011355758309667893,
+            -0.04050311381480088,
+            -0.01017774338301804,
+            0.02437257208967103,
+            -0.004056329295095018,
+            -0.008345450168215274,
+            -0.001599171409135998,
+            0.018358386463243014,
+            0.0027207445181251406,
+            -0.007292967749349572,
+            -0.011267861428599221,
+            0.01657507035043879,
+            0.015105061280816214,
+            0.021419854579631276,
+            -0.02088952706825817,
+            0.006295984081511286,
+            0.020973020776936996,
+            -0.03073246543555915,
+            0.0023720435876790876,
+            0.03363966216709148,
+            0.010412588613998134,
+            0.013023311559810034,
+            -0.0010217662933572685,
+            -0.0075567381078977215,
+            -0.008248818794295818,
+            0.011848200323549865,
+            0.005066463524567732,
+            -0.023174489297821385,
+            0.021823825355858556,
+            0.020368868756448753,
+            0.0026042856858509524,
+            0.003457067317891926,
+            0.0005539745115410357,
+            0.01062670844108589,
+            0.009948889803890108,
+            0.05100741556350918,
+            -0.006493714124814164,
+            -0.014021751055930108,
+            0.015063988102064609,
+            0.02488059237697814,
+            -0.0023499941119596228,
+            0.01087282522970853,
+            -0.02612218935931224,
+            -0.021423880914461062,
+            0.023389294741183817,
+            0.008967763132667892,
+            0.009637806238448432,
+            -0.010815706635624337,
+            -0.011390999462441815,
+            0.031750675158698435,
+            -0.01307829462211914,
+            0.00759749911213318,
+            -0.02169895957209634,
+            0.01387879409130081,
+            -0.008606362268863441,
+            0.032229658009209815,
+            -0.0009066257436946125,
+            0.011062436424260079,
+            0.028010486866468023,
+            -0.007835805879241447,
+            -0.019657486763625547,
+            0.003967480939987674,
+            -0.015577952617624695,
+            0.02188429826317828,
+            -0.020502758155233298,
+            -0.0002362500702658591,
+            0.0040935406015014265,
+            -0.010056462958039452,
+            -0.0057038083589231615,
+            0.011475170331848666,
+            0.0015345076181482267,
+            0.006359262977587827,
+            -0.020293014401106,
+            0.01369706440475074,
+            -0.003989482215141903,
+            0.007887824197823247,
+            -0.015925992045349828,
+            0.014370223626942766,
+            -0.02309139937510304,
+            -0.021808736033141787,
+            0.008142621179624918,
+            0.0108996933426044,
+            -0.007589050511458233,
+            -0.025971539510030145,
+            0.02250992080022228,
+            -0.024980833805349545,
+            0.024979008145354243,
+            -0.006622339940663538,
+            0.002566378488088469,
+            0.004913874740363343,
+            -0.008102103508732738,
+            -0.0028098905427027783,
+            0.0057865587191235565,
+            -0.0073509382400023995,
+            -0.014123756810537343,
+            -0.019453380788786282,
+            -0.04083629135586275,
+            -0.03177927047231389,
+            0.022927581786088044,
+            0.011557232599801421,
+            0.0017073832658476193,
+            0.02339552660545296,
+            -0.02247471449131386,
+            -0.017890866287500053,
+            -0.009775068417716798,
+            0.025184899522229628,
+            -0.04408541669037361,
+            0.012104255392615549,
+            -0.014669458760022245,
+            -0.0245258242481677,
+            0.006274252098230249,
+            -0.004161960147481513,
+            0.009299171724026455,
+            -0.010779421730862708,
+            -0.04713627671791054,
+            -0.005192684784377356,
+            0.02330975631183273,
+            -0.0036802876364168935,
+            -0.006197548344287773,
+            0.03048284650641829,
+            -0.0019503797069087845,
+            -0.005038254331399784,
+            -0.005524875459245342,
+            0.027783596017224226,
+            -0.024950083604372984,
+            0.010988398531346805,
+            0.008304937912856357,
+            -0.008485065895193593,
+            0.006572665697403556,
+            0.0011518846966102158,
+            0.011927348472282199,
+            0.015632588644277183,
+            -0.015559413716826264,
+            0.02262990861928957,
+            0.011446850286896198,
+            0.006506075777765303,
+            0.008015225053785824,
+            0.00010054430757843335,
+            0.0022975742216194894,
+            -0.024948925449668525,
+            0.026047924967851216,
+            0.01888211557921096,
+            0.0001632181017876392,
+            0.012553094476616919,
+            -0.015042621225242777,
+            -0.02782009986290425,
+            0.02606646297992026,
+            -0.016847653820092335,
+            -0.00045272379465939274,
+            0.020053091486344633,
+            -0.016210036358455773,
+            -0.007930550661303182,
+            -0.01123514849773852,
+            -0.0162675150307978,
+            0.01803076559921804,
+            -0.012160217543572069,
+            -0.02216594536057653,
+            0.002594675636753233,
+            0.016443123828746582,
+            0.02134480656850119,
+            0.026694575953313482,
+            -0.011470930697351533,
+            0.017768927048191052,
+            0.0008653408391607816,
+            0.001865765599795092,
+            0.025745954171649543,
+            0.01702677011968492,
+            -0.008474889109168202,
+            0.025347176686722313,
+            -0.0016686223698600755,
+            -0.002421921060670558,
+            -0.012265504084322075,
+            -0.0165214436458303,
+            -0.007545471138114208,
+            -0.015333235556146196,
+            -0.0051830106590765685,
+            0.000762746782664525,
+            -0.0018118022696726346,
+            0.10876576838038368,
+            0.0034948044540582626,
+            -0.011608460689239234,
+            -0.005870263500417206,
+            0.007510774308250375,
+            0.0023525201199670765,
+            -0.006846154764389575,
+            0.016082721103476375,
+            -0.018616626204086575,
+            -0.023593744079394,
+            0.004743481974389998,
+            -0.0009262286319469741,
+            0.00870122878944883,
+            0.02029523251993269,
+            -0.008608837818169835,
+            -0.0200567711498164,
+            0.007089477722191928,
+            0.005535149644041221,
+            0.0023640746516071774,
+            0.005256431824732927,
+            -0.0006118671321824933,
+            0.008209920240361308,
+            0.02448422122961942,
+            -0.015622892878860465,
+            -0.029770836607612903,
+            0.010535371601298532,
+            -0.00015994156470228762,
+            -0.007193425789067776,
+            -0.011046316463797128,
+            0.014728841492009627,
+            -0.0011488451990980268,
+            -0.02751404133172324,
+            0.017663132712536528,
+            -0.019055335178900415,
+            0.014643246200933506,
+            0.0025348345163347464,
+            -0.00016387173410118788,
+            0.022565953069005714,
+            0.013101158251834834,
+            -0.04272085076882881,
+            -0.0034442020309914823,
+            0.01041205537746844,
+            0.011059422927946552,
+            0.0116446016862942,
+            0.0012992535381387887,
+            -0.018670801241345514,
+            -0.011788984490283379,
+            -0.009109849103208542,
+            -0.021268488414296117,
+            -0.012523033785938503,
+            -0.011393853408628214,
+            0.010743646386439633,
+            0.0012964030814325153,
+            0.023809566099779113,
+            -0.003030535542277875,
+            -0.006526351867141901,
+            -0.000430699447948508,
+            0.00880561288237257,
+            0.009694646009259077,
+            -0.02079011138439714,
+            0.008294338719512953,
+            -0.01823922789220214,
+            0.011819769685519291,
+            0.006416044253693268,
+            -0.022799568832306067,
+            -0.009911619276012527,
+            0.007152786695893951,
+            0.008393677195358034,
+            -0.013569953300860946,
+            -0.006503988731872938,
+            0.02299498754558268,
+            0.018870619152710703,
+            0.012233964644738905,
+            0.024994616203196145,
+            0.03838969722235787,
+            0.005744255912479976,
+            0.027682123067526515,
+            -0.016791471092091604,
+            0.008459745383680297,
+            -0.022493868174622695,
+            0.012902262638928338,
+            -0.00460399519266689,
+            -0.0050431343566427886,
+            -0.018188192498411616,
+            0.033765105245412236,
+            -0.009159751849801773,
+            -0.0027150644587327763,
+            0.034778224828120714,
+            -0.0066502021793471075,
+            0.004525230911930458,
+            0.005096748531775903,
+            -0.016067049494073893,
+            0.033274672789154185,
+            0.01686831040872425,
+            0.016353966592947262,
+            -0.021631941883929817,
+            0.011748317746828848,
+            -0.005675701689448899,
+            0.0001273442591116645,
+            -0.020738819037390263,
+            -0.0037528323797385733,
+            0.0027725541343763104,
+            0.0059860509282538546,
+            -0.016556134518370437,
+            0.02033623476757896,
+            0.00019647023567641008,
+            0.0052895744489259875,
+            0.03253462247073863,
+            0.01043311189804924,
+            0.019182938878855655,
+            0.010051051068017807,
+            0.019523739618278902,
+            0.0281939391326836,
+            0.025685787482623124,
+            0.0029932688462254147,
+            -0.018668945726139946,
+            -0.0009326901784812825,
+            -0.007880352863625262,
+            0.010492063055849727,
+            0.022849902214818293,
+            -0.022619228461640748,
+            -0.02772644874135741,
+            0.018566981184526675,
+            -0.0022608185345048414,
+            0.020031103834122595,
+            0.007544464625538984,
+            -0.020935181884939796,
+            0.003407355515103286,
+            -0.023178230166337178,
+            -0.0005418155871620593,
+            -0.031039187720144967,
+            -0.028591174077243604,
+            -0.0007659241904525392,
+            0.011945344534674635,
+            -0.01688153017778311,
+            0.03357788163306365,
+            0.017667778671101954,
+            -0.014289109343082812,
+            0.01604684528817524,
+            -0.011054587325539086,
+            -0.0036959091138078683,
+            -0.005819853870004682,
+            -0.00203755010992417,
+            -0.02157944474979592,
+            0.005138614763345542,
+            -0.0033509138150859593,
+            -0.0024871060745482883,
+            -0.016936575573598154,
+            -0.021901830366872595,
+            0.016375173843894617,
+            0.016742770709641697,
+            -0.006606252542838758,
+            0.039771905404651496,
+            -0.004238071340064403,
+            0.00018364315349545657,
+            0.011690521707403059,
+            0.0010018392078563795,
+            -0.013651921068215563,
+            -0.010829994512513501,
+            0.012362647379788833,
+            0.011321403447882543,
+            -0.018942043512252663,
+            0.005763773608775982,
+            -0.009131565038297889,
+            0.002089216671850269,
+            0.005438887787329238,
+            -0.008183835275629478,
+            0.006690280329947951,
+            -0.01673229763262922,
+            0.015833397876364723,
+            0.00295474085300386,
+            0.003186554859191664,
+            -0.011797859525926772,
+            -0.020991927615680524,
+            0.01576413532776261,
+            0.0032403514518527287,
+            0.018819824905201635,
+            -0.026755218746060662,
+            -0.00012946467986739473,
+            -0.009552681500884924,
+            -0.0028941357661144826,
+            -0.012533800273548772,
+            0.013975640628890588,
+            0.016864739319953755,
+            0.0063941926578529,
+            0.01781016436903586,
+            -0.0026268224264609724,
+            0.015368284773773678,
+            -0.018122416427266218,
+            0.008686009180697955,
+            -0.006801074032721299,
+            -0.018537521789181048,
+            0.0013237338264105415,
+            0.027863741654435434,
+            -0.004670012662973029,
+            0.011066853115455966,
+            0.011704884649534624,
+            0.018954056680143452,
+            -0.0012626215734593707,
+            -0.0030850559460101866,
+            0.015610404665171367,
+            -0.005838226405658572,
+            -0.006861883833171456,
+            0.020872966233916948,
+            0.032951653763679145,
+            -0.006716215648910268,
+            -0.0021292599951407863,
+            0.010886596327481677,
+            -0.004901684487088035,
+            -0.008644584974417106,
+            -0.009587037829603014,
+            -0.03158137900717491,
+            0.024369089446706894,
+            0.012092412439477474,
+            -0.009601042951586109,
+            0.022690862146548786,
+            0.020752792552278052,
+            0.011412743968296708,
+            -0.010471860708195726,
+            0.02291536038589634,
+            -0.01268035142112569,
+            0.024168869865226136,
+            0.032482831775271157,
+            0.004400973556358811,
+            0.0022312906286559167,
+            0.0021175489661843985,
+            0.01691348460546639,
+            0.006050357346231155,
+            -0.007068133214675996,
+            0.011656003476409986,
+            -0.018487923180009552,
+            0.01598530891381001,
+            -0.004432366596808033,
+            0.02808371007118948,
+            -0.01337145216499331,
+            -0.010473700033368562,
+            -0.00537325376270318,
+            0.00010726431372328729,
+            0.006445701087575359,
+            -0.003324002570256215,
+            0.005255443653813165,
+            0.0007072577303252132,
+            -0.017825649146944237,
+            0.003050128130393781,
+            0.004825967170196209,
+            0.009018799597337101,
+            0.004103653497834134,
+            0.007249957401824204,
+            -0.012360355929483,
+            0.01441885664789707,
+            0.004401820363035808,
+            -0.0026190938717927086,
+            -0.0007163225760398784,
+            0.011124538295057815,
+            -0.013232822471919533,
+            -0.004895624967029031,
+            0.005281228035321008,
+            0.0059275753353475414,
+            -0.007507924998741734,
+            -0.004484140989405051,
+            0.011908076549392969,
+            -0.0037917291321648483,
+            0.006351543504726163,
+            -0.004409464890606014,
+            -0.0037760571637986827,
+            0.005303661313434895,
+            -0.00974511544836468,
+            -0.004204167898342997,
+            -0.0007093125445559502,
+            -0.006869250680233214,
+            0.00786202375969793,
+            0.005971033227117194,
+            0.010606085272785863,
+            0.006969437776542303,
+            0.004293129058563288,
+            0.004114452739123398,
+            0.004071017374828283,
+            -0.002445681623750424,
+            -0.00592480769363354,
+            -0.004617501328052899,
+            -0.012067458701076778,
+            -0.008335239153829813,
+            0.009025028998949398,
+            0.008653779421856954,
+            -0.019363018792788574,
+            0.02048011578221327,
+            -0.001188320960771659,
+            0.00772165044232104,
+            -0.01602086976290928,
+            -0.000907553574811279,
+            0.005464880629734941,
+            0.008267702703609528,
+            0.016136762227127184,
+            -0.03654209526492064,
+            0.005179791231187316,
+            -0.0037321883493524766,
+            0.016026013709955444,
+            -0.008215703411373182,
+            -0.006979816286354862,
+            0.02564060526656963,
+            -0.01450763388985563,
+            -0.0009847758228920171,
+            -0.016142304341196283,
+            -0.01407778890808498,
+            0.010815725252959506,
+            0.02050940280233426,
+            0.020684808835848383,
+            -0.0004965802897137407,
+            -0.0005781445948759235,
+            0.00036179841583004193,
+            0.022775804864342608,
+            0.015545681427566235,
+            0.018697154644607316,
+            -0.004543500782944068,
+            0.007028436664826706,
+            -0.023057215501463973,
+            0.009912581556496658,
+            0.0011795417129232596,
+            -0.008215027850908477,
+            -0.008554137076267468,
+            -0.008703378564520125,
+            -0.010448538943182258,
+            -0.011773811327190638,
+            0.004106616839960258,
+            0.013895013678116032,
+            0.004190549543250594,
+            0.0031571190600981074,
+            -0.0225385788570314,
+            -0.006243193914542516,
+            -0.011229641377928767,
+            -0.01674214043453308,
+            -0.007367389908398082,
+            -0.02386334969150212,
+            -0.028021339684039734,
+            0.0002794207370355175,
+            -0.02411861258809571,
+            -0.015958336406924612,
+            0.005268815864773021,
+            -0.01546610225139129,
+            0.00898600137695074,
+            -0.008396481690803911,
+            -0.011873486990732417,
+            0.0074743033679134105,
+            0.015336444788692,
+            -0.017676977079590463,
+            -0.010165317702640217,
+            -0.01690524990506033,
+            -0.013460451917691695,
+            0.010918550283871637,
+            0.00595960977280955,
+            0.009250157926438213,
+            -0.012447979337850305,
+            0.0062932435120863365,
+            -0.012673905639312696,
+            0.015073013552721701,
+            -0.03769166067890086,
+            -0.029992249463405965,
+            -0.00014990754540999537,
+            0.007288036630677436,
+            -0.0020735742042505426,
+            -0.018695120921334806,
+            0.0003860116505754868,
+            -0.003608995261360232,
+            -0.014900191335144973,
+            -0.013866216738266176,
+            0.025251475305524053,
+            -0.006788483107616133,
+            0.01223227424986397,
+            0.0034937277040015626,
+            0.03438333560343413,
+            0.008803852645562129,
+            0.014666206816302367,
+            -0.005709421955160623,
+            0.003742433643393657,
+            -0.0033706258426092744,
+            0.006906464031352315,
+            0.0076578888497945185,
+            0.009276592191710524,
+            0.008025321004305927,
+            0.004576470294889512,
+            0.002887346543823031,
+            -0.001925468214331082,
+            0.02125241545487307,
+            0.025671028960818562,
+            -0.00037467906648160044,
+            0.008981063192139038,
+            0.0009387900084090577,
+            0.0002735789978680719,
+            -0.008374290340790827,
+            0.006547013284767913,
+            -0.008274637814401893,
+            0.004041066530254948,
+            0.012897549078838394,
+            -0.009150863940242148,
+            -0.010211568689323908,
+            0.016575088843292566,
+            -0.012380099999574783,
+            -0.03420455578618604,
+            0.026550654680719987,
+            0.005324512466403649,
+            -0.0035670106106754245,
+            -0.010657928468872608,
+            0.0037176027853158167,
+            -0.01103891334887315,
+            0.008096915671982816,
+            0.022587659771132487,
+            -0.0013527917944322358,
+            -0.0018085171207746743,
+            0.0038503794877980586,
+            -0.011870484733648887,
+            -0.007881489342351292,
+            0.006462772034381968,
+            0.0315696514747912,
+            0.01326417653779577,
+            -0.020400369539140532,
+            -0.015223808971473513,
+            0.020406862349006738,
+            -0.023142176712111575,
+            -0.006254959806032159,
+            0.009039506093607052,
+            0.0009087520492922078,
+            -0.022766046838415165,
+            0.0009662037093312435,
+            0.004701966191595815,
+            0.005946458676746661,
+            -0.006358986495019584,
+            0.03237532341275415,
+            0.001075992257265679,
+            0.015892020271402903,
+            -0.017189453434945788,
+            0.0017514632185789299,
+            -0.014189278671121677,
+            -0.014265447388465805,
+            -0.029809460132405256,
+            0.0005557491671071693,
+            -0.0030324198417091704,
+            0.011038984948790744,
+            0.024808067315720335,
+            0.01145103533194245,
+            0.014432246996007246,
+            0.010552852947578788,
+            0.01513689173494211,
+            -0.013653722180565311,
+            -0.029009757850432732,
+            -0.005171522291984226,
+            -0.018001048743500243,
+            -0.02297482628212251,
+            0.003962613844063587,
+            0.013645125285548414,
+            0.006278662690346082,
+            0.023786381386575035,
+            0.0021738516691483163,
+            0.0009580591750940011,
+            -2.690805459518147e-05,
+            0.008894091070322921,
+            -0.004952203441302471,
+            -0.006553986197076102,
+            0.003570189040872711,
+            -0.000652714137223501,
+            0.021228360881461853,
+            -0.005758946428212041,
+            7.897153237591187e-05,
+            0.014242175492137011,
+            -0.005622256348934008,
+            -0.0035009192529791146,
+            -0.017339926143652377,
+            0.006248842389703505,
+            -0.01622011586976866,
+            0.025783086900871728,
+            0.018450783573486916,
+            -0.009109295293546267,
+            -0.013469521573468915,
+            -0.011259555201313116,
+            0.02773097452736661,
+            0.004631282554715908,
+            0.010787055220617197,
+            0.018228322680582464,
+            0.01461254259249465,
+            0.01784830193964943,
+            0.004664725683522323,
+            -0.0034339781146649384,
+            -0.0007692550982545399,
+            0.021176646860433974,
+            -0.014700178590249292,
+            -0.002701208084390198,
+            -0.008842788249773623,
+            0.0021799005046816444,
+            -0.008431299000066363,
+            0.006086447848362669,
+            -0.010793016137040172,
+            -0.015251381840165954,
+            0.0189503457652375,
+            -0.02660767353735015,
+            0.016423753600066336,
+            -0.00593016170680532,
+            -0.00482891500342298,
+            -0.027459214027897378,
+            0.010728039528784636,
+            0.0027866316191608297,
+            0.0022519974195000687,
+            0.010770472550461752,
+            -0.021427634123401408,
+            -0.01867717541189363,
+            0.008532444554010673,
+            -0.011457117350255678,
+            -0.007152301670244725,
+            -0.00348619742388509,
+            0.0154715607296319,
+            -0.00017898955187849443,
+            -0.009925365923862246,
+            0.02085093731060963,
+            -0.007175309371250024,
+            0.01223979370120031,
+            0.005256127464980655,
+            -0.008756077893734508,
+            -0.0028438320920811362,
+            0.021022708898898203,
+            0.027557382862284808,
+            -0.008342206702778956,
+            0.014526066580760486,
+            -0.011883797998224506,
+            -0.01088335352319397,
+            -0.03479444009382274,
+            0.030104838505937447,
+            -0.013305732257226255,
+            0.026773747582411633,
+            -0.0006968265979630097,
+            -0.017037729111584094,
+            -0.016206207453228583,
+            0.01625225876515654,
+            -0.0024290584023212254,
+            -0.009366287391027166,
+            0.01465885301870365,
+            -0.02428628495473643,
+            0.0058336571592899735,
+            -0.0006627935593670669,
+            -0.011531762838191602,
+            0.013275508677482622,
+            0.0019223252930710324,
+            0.010738622785461513,
+            -0.02030405805119001,
+            0.0009661595529028194,
+            0.009263716373456188,
+            -0.016791855691574455,
+            0.006057830565306032,
+            -0.03730952786847912,
+            0.0029341721588188943,
+            -0.014833895684348656,
+            0.005725615593914644,
+            0.001498755253963699,
+            -0.01820708352937343,
+            -0.03652692232407375,
+            0.00043665701245599497,
+            0.01913883585802903,
+            -0.007975988799790905,
+            0.020538595326175857,
+            0.008243569241768276,
+            0.008421404577931517,
+            0.002314738067443065,
+            0.0016702403734608475,
+            0.008279538988821465,
+            0.03679456861213399,
+            -0.01775927634148114,
+            -0.004836606118958108,
+            -0.0012151329859941935,
+            0.010644195708319258,
+            -0.02028470571116305,
+            -0.009287515726906219,
+            -0.0036506912311261536,
+            -0.013134467570201307,
+            0.030615862647585222,
+            -0.020265555441523482,
+            0.025634449009756734,
+            -0.009300770899023007,
+            0.009274072905190077,
+            -0.02097685009125915,
+            -0.009797008690754956,
+            -0.010661540100131183,
+            0.0019532291898233895,
+            0.013465005282849381,
+            -0.009151430285942593,
+            0.0071467934925125145,
+            -0.006655608635025676,
+            0.01020327561163374,
+            -0.001472597695707201,
+            -0.02443508800166349,
+            -0.0024998811551575563,
+            -0.01107321258077183,
+            0.02456164421154801,
+            -0.0019619696941303388,
+            0.0005043179699158694,
+            0.010951580163871948,
+            -0.00853593956432799,
+            0.023927336608555987,
+            -0.008855702280766362,
+            0.0010093679746665581,
+            -0.004863269000020324,
+            -0.004005691415117686,
+            0.001698212044518236,
+            0.01822024022650613,
+            -0.005707404715849356,
+            0.03722587302228139,
+            -0.01708925706528118,
+            -0.0065500772743641925,
+            -0.00015538686795361824,
+            0.010852925330001283,
+            0.010035359503147549,
+            0.008754719437760327,
+            0.004067820167375348,
+            -0.014861417626433965,
+            0.009346669946626484,
+            -0.020136029447262147,
+            0.007501669351583119,
+            0.023178490169347605,
+            -0.006840359541885465,
+            0.0006676450965357345,
+            -0.01283320739092204,
+            -0.015610538827742276,
+            0.006308543164022682,
+            0.002329420005586303,
+            -0.002295739637504597,
+            -0.02181189272979859,
+            0.006298804075737111,
+            -0.014398470126723037,
+            0.004007419942728446,
+            -0.0007497656993578918,
+            0.013863867106289847,
+            0.016451166273826513,
+            -0.006740209707090658,
+            0.039292260347522444,
+            0.01871723939316731,
+            0.009109005310505462,
+            0.005674782595856756,
+            -0.040588444759736254,
+            -0.008993850998154852,
+            -0.0010197185016393919,
+            0.004866150984083949,
+            -0.01141354216871663,
+            -0.018944881712855702,
+            -0.002646110151132781,
+            0.000308585036997592,
+            -0.010681954707313127,
+            -0.009091585604191968,
+            0.01751599002710486,
+            -0.006034424764302979,
+            0.00872057856926618,
+            0.004146961781109887,
+            0.023758065635807138,
+            -0.009727802672285544,
+            0.01176257119851645,
+            -0.0071822083765114165,
+            -0.005456197751568526,
+            0.0006726765703478756,
+            0.011011628907288813,
+            -0.008031792262886802,
+            0.009763340867550022,
+            -0.004279891138185881,
+            0.008355720233269893,
+            -0.008552596384067825,
+            -0.023952129458400563,
+            -0.014972827087541732,
+            0.014347535404251018,
+            0.011651260496709961,
+            0.008137738462704715,
+            -0.013624435569539285,
+            0.0031728721674035352,
+            0.00675830277744976,
+            -0.009184784787581942,
+            0.009161771311910335,
+            0.017733082848622708,
+            -0.001411542568472906,
+            -0.0017830167499636113,
+            -0.0016130003349297027,
+            -0.012173528776899308,
+            0.0011196437753239162,
+            0.02284887358189995,
+            -0.003054479643937998,
+            0.014614283913976716,
+            0.010689486748036806,
+            0.009643292262102645,
+            -0.00810930837603355,
+            0.001915955511174678,
+            0.02341863048264366,
+            0.015022291783105098,
+            -0.004625715884342598,
+            -0.011422973894623295,
+            0.004259757717597968,
+            -0.010332744704689279,
+            0.007373328473086509,
+            -0.02117940113144938,
+            -0.004672698999328004,
+            0.008896311380763048,
+            -0.007028862917603513,
+            -0.01732432285099198,
+            -0.028026173555055017,
+            -0.016765805892242715,
+            0.019819329380181,
+            0.0027213760678977965,
+            -0.013291251876048186,
+            -0.0032919084721662363,
+            0.0031214883337300926,
+            0.002553005168048115,
+            -0.0071104806975682885,
+            0.005310583052470149,
+            0.007359183453835734,
+            -0.008674133980486269,
+            0.017665958608281126,
+            -0.000521269431423141,
+            -0.0008020251454616109,
+            -0.0005442374261144659,
+            0.0006127687598711624,
+            -0.016387752806126442,
+            0.01666161912459431,
+            -0.010949228376419892,
+            0.015198110119816624,
+            -0.008966726991404666,
+            0.008600499511690635,
+            -0.007617043351740469,
+            0.009458641139070436,
+            0.011597957400445055,
+            0.012755037944648426,
+            -0.004403680820063604,
+            0.008519241473418196,
+            0.004807816191718704,
+            -0.008147125240985166,
+            0.007017601015398136,
+            0.01296082929962557,
+            0.00582353932160707,
+            0.0013634237970372698,
+            -0.0008246894322793264,
+            0.0022035670958705576,
+            -0.002480976219285683,
+            0.0056293125704658135,
+            0.012714850523393719,
+            -0.002497639449055006,
+            -0.007135683201188999,
+            0.014988249771676699,
+            0.014955895630183352,
+            -0.022994948923833993,
+            0.007121634434082677,
+            0.005835971825639792,
+            -0.0030507120821557008,
+            0.002145266953387315,
+            -0.027334525318320587,
+            -0.020359474070022213,
+            0.01901995970350808,
+            -0.015260454474605018,
+            0.02850460517419317,
+            -0.011624138792780823,
+            -0.02284242513728605,
+            -0.01277489645734181,
+            -0.004209529633534193,
+            -0.022684500849916897,
+            -0.004894414098943742,
+            0.01919165657366256,
+            0.013401560972810082,
+            -0.004313549020265677,
+            -0.019089369171102138,
+            0.03863248408555474,
+            0.010497951840980463,
+            0.03508157919687289,
+            -0.008597970944278045,
+            0.0023354831464448313,
+            -0.01563401808449416,
+            0.015551176593436512,
+            0.001075882802133782,
+            -0.011575692834796734,
+            0.01505716108583101,
+            0.031836226915668404,
+            0.0034740889237207494,
+            -0.00617206855371193,
+            0.0018420207148235309,
+            0.00418433325800744,
+            -0.0008167766650900381,
+            0.011160989178055499,
+            -0.018497472318034093,
+            -0.011951262348399922,
+            0.0013093376768780122,
+            0.00216347903176653,
+            0.0059233460419514045,
+            -0.001778091863744448,
+            -0.013317639704511549,
+            0.011433308892643204,
+            0.011596810919469926,
+            -0.01518143105525228,
+            -0.007497373860338335,
+            0.0050957665991362405,
+            -0.018974405317819656,
+            0.01872472902494418,
+            0.011859522398789263,
+            0.000792413325332956,
+            -0.0006503947129906383,
+            0.00808635755624928,
+            0.003749813548995223,
+            0.0033657799335872368,
+            -0.007862215303617144,
+            -0.0028014669075116876,
+            0.0009892310856576799,
+            3.834379772674841e-06,
+            0.015402211364447726,
+            -0.015264893397161102,
+            0.02488855366696587,
+            -0.0025643639525691083,
+            0.00953993868921819,
+            -0.017460590918773188,
+            -0.0017911712931080155,
+            -0.00402374787529868,
+            0.01838051670627292,
+            0.006753072393532114,
+            -0.010130676216186492,
+            0.004921510943896829,
+            0.007975201081296845,
+            -0.012589786869771252,
+            0.00018319054705558588,
+            -0.0022606180909706688,
+            -0.014791528072476771,
+            -0.0031171076370692706,
+            0.017873825561816544,
+            0.007593688317551089,
+            0.0037375070591255375,
+            -0.015863143964928172,
+            0.007192324815779268,
+            0.007522990667369163,
+            0.024504875417877527,
+            -0.026309816223960274,
+            0.024779042752840836,
+            -0.022104087334088543,
+            0.02165781034613893,
+            0.0319051741666687,
+            0.007538146369110045,
+            -0.01690181186708103,
+            -0.006595947249169521,
+            0.006373991305923002,
+            -0.023410814842488123,
+            -0.005494017994983188,
+            0.016903736810723716,
+            -0.01792400152372258,
+            0.000330865061689199,
+            -0.014660413003063089,
+            0.0040825880154611895,
+            0.008248037865838212,
+            0.011540333451799686,
+            -0.0033578830043738085,
+            0.018327525495676525,
+            0.020171745352159914,
+            -0.01138759854371235,
+            -0.0004221149053107702,
+            -0.022321409200941777,
+            0.005046865873981027,
+            0.022509643273449936,
+            0.001111272240506322,
+            -0.009540467231190513,
+            0.018376261423745193,
+            0.008480532791922674,
+            -0.020801891594934313,
+            0.008436374227753589,
+            -0.036175982655998785,
+            -0.010454118523313122,
+            -0.009328507118928412,
+            -0.001218633166683384,
+            -0.004142773602181115,
+            -0.0002494371580772723,
+            -0.011365548842513265,
+            -7.699548642315061e-05,
+            -0.006889339420877534,
+            -0.0037173070609413474,
+            0.014953978004522174,
+            0.011678957032329096,
+            -0.003501047405385249,
+            -0.0073984932581048915,
+            0.008493128567102267,
+            0.006163937693564917,
+            -0.017034696006318337,
+            -0.013268418502042576,
+            0.0037122128689059085,
+            -0.020067656401141467,
+            0.005148329575092478,
+            0.00459729178556505,
+            0.014031669598917171,
+            -0.01849235806598133,
+            -0.009298127665365717,
+            0.019716512208448042,
+            0.0094466405158553,
+            -0.0008988916054855498,
+            0.004264260983102956,
+            -0.00032312548009733906,
+            0.002279987803645912,
+            -0.0008902345374346296,
+            -6.145528806973222e-06,
+            0.003602229465148649,
+            0.012503366799381277,
+            0.007759393855482671,
+            0.03694402922447051,
+            -0.016692148708641812,
+            0.005873765863350649,
+            -0.014187609702966002,
+            -0.0037592517335595654,
+            -0.00749939010396784,
+            -0.0004308157309481927,
+            -0.0022282234382577843,
+            0.0021999996319642854,
+            0.0052176773733353925,
+            0.009353171029508977,
+            0.002155405440765206,
+            -0.00823565584692709,
+            0.009691447250255694,
+            -0.015228173095533029,
+            0.003853858061067531,
+            -0.008207910291425473,
+            0.01017972550354623,
+            -0.009695280827515048,
+            0.0028705587799331246,
+            0.03849358164875786,
+            0.01130021300937487,
+            -0.004934175853966098,
+            -0.014300147534361459,
+            -0.0007796051618993811,
+            -0.01737877501558196,
+            -0.008579119859817895,
+            -0.005434792488602325,
+            -0.02617338653902575,
+            -0.00838916510137031,
+            0.015344260464413648,
+            0.011202567142310492,
+            0.014124853863170132,
+            -0.023894182185855824,
+            0.008897943489322673,
+            -0.03001670823127986,
+            -0.011951507895140876,
+            -0.001530967140156163,
+            -0.015467812535428609,
+            0.0010110999991167703,
+            0.023623196504289838,
+            -0.007762735863648054,
+            0.012654652822750325,
+            0.012867958170258448,
+            0.018777522196571934,
+            0.006390986057157016,
+            0.00038045736974183937,
+            -0.009444911783760569,
+            0.009534310165417939,
+            0.008806031651319704,
+            0.0080954543713255,
+            0.013161833638467553,
+            -0.0022840529693446553,
+            -0.018060162439023572,
+            0.005469288590000856,
+            0.014354475778235131,
+            -0.0033504276914600585,
+            -0.001615191819057537,
+            -0.011088493180661976,
+            0.016039040565630432,
+            -0.023559046333872156,
+            -0.009482398794557168,
+            0.009918576960319548,
+            -0.0027308309155029842,
+            0.009212063547988036,
+            0.0160181200081126,
+            -0.01881373001297218,
+            -0.0024168618343039703,
+            0.016244889977796682,
+            0.0024104546308670025,
+            -0.00022606425578303413,
+            0.016792244238218245,
+            -0.0007616857360728623,
+            -0.004279120886828465,
+            0.00997764266310123,
+            -0.0075631917138318655,
+            7.3878137053128405e-06,
+            -0.002416364915008295,
+            -0.0028253498168087964,
+            0.02054128727706496,
+            -0.021281801678871736,
+            -0.005379259148621041,
+            0.02257306103533458,
+            0.015687105622685975,
+            -0.007843745111084413,
+            6.258498042855072e-05,
+            -0.0219506172079994,
+            0.0019925886885244454,
+            -0.007768133999661569,
+            0.016524850592507592,
+            -0.007957949830060354,
+            0.0030085044673901665,
+            -0.01361024620290394,
+            0.011563781350927133,
+            -0.01277407169098418,
+            0.00873716215529134,
+            -0.002986943948333218,
+            -0.017567646684567492,
+            -0.0053109390854215125,
+            0.004561236136619266,
+            0.0014045175130801308,
+            0.044958041347040185,
+            -0.005539136232703934,
+            0.003729517522976541,
+            0.018086335927383804,
+            -0.014199453349727933,
+            0.02079719360289355,
+            0.016663293302592568,
+            0.00991445855219406,
+            -0.00033967370741635794,
+            -0.0018726748849858836,
+            0.023373354806639515,
+            0.013258226441642464,
+            0.00861986998150065,
+            -0.0014340218811281975,
+            -0.008516541227326329,
+            0.018239230461345617,
+            0.005458629943667832,
+            -0.012421896028975013,
+            0.012732721349758832,
+            0.013559877132508506,
+            -0.013720939472355347,
+            -0.011545210834395917,
+            0.01616358223872008,
+            -0.019904594319155373,
+            -0.018260016571763644,
+            -0.0036124537591368864,
+            -0.01614510591132948,
+            -0.03864699384205461,
+            0.01619401261888991,
+            0.018351607973242735,
+            0.006666313000123845,
+            0.013647410298896132,
+            0.002597680061427196,
+            0.004545530267244157,
+            -0.018531082430861854,
+            -0.0077618625909995275,
+            -0.008085144524135461,
+            0.003744735923004248,
+            0.011059457364156674,
+            -0.011335432904114519,
+            -0.009514379703581616,
+            -0.00034500702695418285,
+            -0.007433132060989644,
+            0.01971420270994951,
+            0.0020753610290475642,
+            0.0034506706065161024,
+            -0.00628437298309269,
+            0.017966485854173167,
+            0.026789678964152908,
+            -0.007532446825077476,
+            0.021886182869092073,
+            -0.002864230253056434,
+            -0.0015870360120997492,
+            -0.014393519359298496,
+            -0.0014439412842964876,
+            -0.011527182213472351,
+            0.0005453493437517964,
+            -0.0006791481533636673,
+            0.014093767997869645,
+            -0.03288824642147967,
+            -0.003957301296796813,
+            0.014444592778851293,
+            0.007538052312876599,
+            -0.005057755472988024,
+            0.011357919882949903,
+            -0.002964451028327082,
+            0.016977317604413924,
+            0.0075201055190542146,
+            0.023391646079994784,
+            -0.016588823248397513,
+            -0.004504858318770565,
+            0.006660877064954993,
+            0.0034631255931793638,
+            0.0014484203909961972,
+            0.01912983041187014,
+            0.023439941258581887,
+            -0.014715886385460541,
+            -0.0013596123366001458,
+            -0.0029020372473879614,
+            -0.021414221343490594,
+            0.014976806123253727,
+            0.0074729481304182865,
+            0.010967351226114886,
+            0.009806663105020302,
+            0.015465076881663691,
+            -0.02109587527354609,
+            0.0011671121739822202,
+            0.020567458365051382,
+            -0.008618333549105003,
+            -0.018815706906171112,
+            -0.009804829158883123,
+            -0.00966828714719191,
+            -0.006078893678150302,
+            -0.00803783313451146,
+            0.005523023167028105,
+            0.0022647506733877573,
+            0.01652997604669998,
+            -0.008204200434441975,
+            -0.008404609147866811,
+            0.008489247125986078,
+            0.007546785673285454,
+            -0.0033093451283570316,
+            0.010614050963656496,
+            0.0030633179859393587,
+            -0.006940540318011839,
+            0.00894884343338777,
+            -0.002787816848430143,
+            0.014347598920245487,
+            -0.02441769157218451,
+            0.004717742147185283,
+            -0.012096876825337124,
+            0.006405151160675699,
+            0.0041568790527989905,
+            0.0016702066682613428,
+            -0.0041159211459004,
+            0.0035217362856754353,
+            0.0008893479210743541,
+            -0.07433312151926533,
+            -0.012629681414004524,
+            -0.0035797012285864486,
+            0.033657838713146884,
+            -0.019695806809053914,
+            0.004477927056336216,
+            0.011966065644382396,
+            0.0096717896523898,
+            0.003070713232538593,
+            0.006136726101157844,
+            0.006112385569942848,
+            -0.0245258242481677,
+            -0.00015284418977092615,
+            0.004821040694679944,
+            0.0020347004045943454,
+            -0.007532408630122127,
+            0.00750565079228405,
+            0.015327710245879296,
+            0.020989926895388523,
+            -0.010970305087289488,
+            0.017227663358885748,
+            -0.003352718496981928,
+            -0.006068892126250978,
+            0.0014285750322333686,
+            -0.010153799722483554,
+            -0.017307415843726394,
+            -0.006495008086641125,
+            0.0068983699213297075,
+            -0.001982568363667046,
+            0.0044032687939897855,
+            0.003333596495330213,
+            -0.0032171586993948696,
+            0.0072365127663633925,
+            0.013920326463272646,
+            0.006415916101287133,
+            -0.013675562796300214,
+            0.003003782758807665,
+            -0.02500375035312058,
+            0.011945988879567646,
+            -0.012708975181025296,
+            -0.0009488161508767279,
+            -0.0047093877027523014,
+            -0.0035750724825159503,
+            -0.002213435677949806,
+            0.02043567538125188,
+            -0.014304009033474186,
+            -0.01802899775650995,
+            0.008187227768196673,
+            -0.011217013195839762,
+            0.008300740149622635,
+            -0.03382831122769974,
+            0.012386335491333264,
+            -0.0276029934382802,
+            -0.0002243752790856292,
+            -0.004521387639042107,
+            -0.0010631721066129095,
+            -0.009491184157278669,
+            -0.03274559774619223,
+            0.015845705808609598,
+            -0.007204751554079551,
+            -0.020338337341085228,
+            0.0007143424225187802,
+            -0.021267094083674304,
+            0.014150458196361147,
+            -0.01851165553224022,
+            0.0033450353123933058,
+            -0.018775439467431685,
+            0.002782289821140401,
+            -0.0018711350216089556,
+            0.010502195116579441,
+            -0.00557814442809019,
+            -0.0007130962242755068,
+            -0.013072988887382269,
+            -0.002121841212381393,
+            -0.006653444103331213,
+            0.009394907690516121,
+            0.0028678074167728367,
+            -0.009601577734902932,
+            0.011153571864462314,
+            0.013046116892519414,
+            0.019812860521973938,
+            -0.00539047787915588,
+            -0.00555139958411929,
+            -0.011553689090179406,
+            0.009852063270798534,
+            -0.016326212706634428,
+            -0.00827402913960903,
+            0.008982303148984887,
+            0.015216947130173639,
+            0.01026889105233137,
+            0.003352028462252994,
+            0.003150801750574548,
+            0.002531620731961765,
+            0.026744691368233237,
+            -0.0015120532859309638,
+            0.0008324701960729416,
+            -0.014508988496009931,
+            0.022127525293249926,
+            0.02772430884375744,
+            -0.0012206102581726523,
+            -0.00018859639594030037,
+            0.017437360169132967,
+            -0.004697052522250805,
+            -0.0038111184371524645,
+            0.024015246959315593,
+            -0.038396108183338655,
+            -0.008416825232533771,
+            -0.006788816882617399,
+            -0.006043428198944214,
+            -0.009587366247924467,
+            0.007294190273465812,
+            0.00041966080769004275,
+            0.0024709692753547823,
+            -0.0005250904505954556,
+            0.03452858797518386,
+            0.003403515654095309,
+            -0.0003392060101496309,
+            -0.0032980864446307116,
+            0.0019125121221204029,
+            0.0032987517853214346,
+            -0.009116761755779728,
+            -0.038030085122772624,
+            0.006006334000396066,
+            0.007932785488877407,
+            0.00975009749364395,
+            -0.008120121513849434,
+            -0.008864461019882246,
+            0.004909348738798099,
+            -0.02273129283865203,
+            -0.01372636307167243,
+            0.02625229144362647,
+            -0.008523028986127237,
+            0.029005877324974557,
+            -0.022435059061182174,
+            -0.013135836801818153,
+            -0.004126433322528265,
+            0.010283975013795537,
+            -0.019732856462444506,
+            0.004343860338939202,
+            -0.004022387065239562,
+            0.023733721026005954,
+            0.0077578144626961245,
+            -0.008371631446146718,
+            0.012386735614153616,
+            0.0064789380436815714,
+            -0.0026638642827013837,
+            0.00136333052963848,
+            -0.007209517067748103,
+            0.016916505654836485,
+            -0.011819784026452944,
+            -0.005833909403097991,
+            0.008977527508585844,
+            -0.0024556434204852946,
+            0.02509293948918158,
+            -0.0237762245071987,
+            0.005288171058938098,
+            -0.00019047961064753207,
+            0.01822542471604037,
+            0.008127347078975925,
+            0.008361588667386882,
+            -0.011835407590060994,
+            0.010004482616697591,
+            -0.03378107067882622,
+            -0.015433906677849963,
+            -0.007702910250795191,
+            -0.0013877702264578865,
+            -0.002059152251387601,
+            -0.015745218548968338,
+            -0.026393221735795527,
+            0.006294583740809379,
+            0.016267525575472445,
+            0.023291528020980265,
+            -0.004683197930892445,
+            0.012912830736650997,
+            -0.002831583991769044,
+            -0.008904271394710672,
+            -0.011441882849296418,
+            -0.012202629605208118,
+            -0.012470773094383122,
+            -0.014560694782340029,
+            -0.017045701603711442,
+            0.012139665577738337,
+            -0.020079061996873537,
+            0.013911518328924777,
+            0.0347210775522539,
+            -0.017176114991595492,
+            -0.0018941772168839354,
+            0.019702722555622765,
+            -0.004221469612316137,
+            0.01212088353057276,
+            0.0063707489618353725,
+            0.015298014532351866,
+            0.0038643174573634725,
+            -0.012181479717826009,
+            -0.007374005740809251,
+            0.019411457664902376,
+            -0.004902591844710158,
+            0.014591446725717338,
+            0.015069599884557399,
+            -0.008675672164385891,
+            0.002544198867387345,
+            0.01951865904496276,
+            -0.01762101777798757,
+            0.01665654707715257,
+            -0.00458402229480694,
+            -0.009936206685538294,
+            -0.013345740581684638,
+            -0.005336416724426228,
+            -0.019469457091169776,
+            0.0061608454109892425,
+            0.018931676738452464,
+            -0.00533013594306834,
+            0.009310871143727683,
+            -0.0037670793158045,
+            -0.005045151384034014,
+            0.0002041909886640482,
+            -0.04152751596227068,
+            -0.008340911336217609,
+            -0.0006651584370005978,
+            -0.02505748815737022,
+            0.025954102004571416,
+            0.007205602377144382,
+            0.022602344303400163,
+            -0.017168610245491406,
+            0.004459735690564842,
+            -0.005882024209102714,
+            0.013217517324383834,
+            0.02255689669623969,
+            0.0017984057469785134,
+            0.005699128497215882,
+            -0.02353849721739205,
+            -0.0034711576718386185,
+            0.015924913374553738,
+            -0.003604478141616307,
+            -0.0034993903155063817,
+            0.012592308877056896,
+            0.012864938348003884,
+            0.01567418642605425,
+            0.008801753550506258,
+            0.015606443428689292,
+            0.010589933994371628,
+            0.008928721556432368,
+            -0.005805129999077126,
+            -0.001025103064751441,
+            -0.015907755521365925,
+            0.0045874330999415985,
+            0.031701049450012876,
+            0.012594483199773746,
+            0.016783858873835704,
+            -0.01778536268185936,
+            0.00789523134933153,
+            -0.0012706411671644011,
+            0.008325108613530198,
+            -0.001677173669891956,
+            0.029540136288412976,
+            0.011739999605699826,
+            0.015670966746894652,
+            0.008732334090105053,
+            -0.016237731209426752,
+            0.045334690225148,
+            -0.016911928942220152,
+            -0.0021139311364009113,
+            -0.01058970056572469,
+            -0.021705338531028618,
+            0.007324792557651635,
+            -0.006556582104686972,
+            0.0018243188317438144,
+            0.010780292738415671,
+            -0.006471850255151019,
+            0.0006997395548596146,
+            0.003907433192252573,
+            -0.023129313269763224,
+            -0.002550875642832235,
+            -0.02237917712764273,
+            -0.003107775639788617,
+            -0.02621608419628401,
+            -0.00012482032027919036,
+            -0.016207377503696517,
+            -0.0038712739995960753,
+            -0.023315508569288476,
+            -0.014963211090562272,
+            -0.01633110043829169,
+            -0.004241081909202262,
+            -0.008295402942465353,
+            0.004976396647108597,
+            -0.007402107755970892,
+            -0.020321297548927468,
+            0.0020986169430507757,
+            0.016803273183070004,
+            0.01434729320047051,
+            0.018612970083178984,
+            0.00603868809307069,
+            0.014278001198851581,
+            -0.0026350854954985117,
+            0.017171016209107656,
+            -0.006968968261909459,
+            0.006881847166565678,
+            0.011454001418121107,
+            0.02140935294963631,
+            0.011121833270235448,
+            -0.012801756637470525,
+            0.0010870041670931706,
+            -0.009655388034828094,
+            0.002094601818569396,
+            -0.00828486034340067,
+            -0.009190975158350585,
+            0.0017762581946855365,
+            -0.007261996868906865,
+            0.004674405202069334,
+            0.0030526253833799426,
+            -0.0014082232541965316,
+            0.010419302200840906,
+            -0.0024200053377751805,
+            -0.012208685875688886,
+            -0.0033037330470612695,
+            -0.017975940046938498,
+            -0.006958042903103732,
+            -0.0014496522238488239,
+            -0.011711502944602638,
+            -0.01016130418405875,
+            0.003321436189059608,
+            -0.00538282506552848,
+            -0.009353581612407435,
+            0.013884147453136417,
+            0.0001309738696853529,
+            0.0005878330075918333,
+            0.023495280602974504,
+            0.006035028203642483,
+            0.008087594627921345,
+            0.013144288945385904,
+            -0.01023995000994454,
+            -0.022574958230635715,
+            -0.0243435727055662,
+            0.007867022215096418,
+            -0.01573478573423379,
+            0.02324699908478915,
+            -0.007929809922043611,
+            -0.00561630876431978,
+            0.0020982723335057625,
+            -0.013699317926847062,
+            0.008578431496569186,
+            -0.011271381061801356,
+            0.0014450645788554738,
+            -0.019581451799757368,
+            0.018059311800267413,
+            -0.010917911869373693,
+            -0.004302687883018328,
+            0.009466103226841532,
+            0.034049086327052606,
+            -0.0005689605174080737,
+            -0.007410970644432269,
+            -0.006594150877740108,
+            -0.00022810271313786974,
+            0.018766765773409123,
+            -0.0029041815756733333,
+            0.005758781908059541,
+            -0.01563519107805783,
+            0.006571628287171105,
+            -0.006132372252825422,
+            0.0012423261344297225,
+            0.01513074831647933,
+            -0.026306320707079913,
+            -0.015491710256868008,
+            -0.007857352164317122,
+            -0.024893430640551818,
+            0.002253574511878983,
+            0.017579477884206534,
+            0.020505606775521504,
+            -0.020324505029926956,
+            -0.02513332780748072,
+            0.005150966733930881,
+            -0.026580994694344065,
+            0.0037390577082486594,
+            0.1241031878517593,
+            -0.002138404898309707,
+            0.002720723037908521,
+            -0.011425036886011179,
+            0.0022399461009316216,
+            0.005389407121650262,
+            -0.019552543955456567,
+            0.05950746473186985,
+            -0.00638023587296763,
+            -0.004023798021748334,
+            0.002170545188776296,
+            0.001785855390676113,
+            -0.005831534307818096,
+            0.008643501093778434,
+            -0.017602787281866816,
+            -0.02184792441891879,
+            -0.0052215705748838905,
+            -0.021252169339074325,
+            -0.008509621422071373,
+            -0.013000137524907565,
+            0.002893880203681282,
+            -0.015821609145923173,
+            0.010397692426614709,
+            -0.01638916243796692,
+            -0.010283706937701179,
+            -0.01103621454761223,
+            -0.00373552043529251,
+            0.01563553822905027,
+            -0.03188533919653949,
+            -0.004873550301425776,
+            -0.005183590054152726,
+            0.018218971950313273,
+            0.006004328765748325,
+            0.004357734370247252,
+            -0.014230318971458851,
+            0.0064792070801456235,
+            0.0215589173714926,
+            0.0055272493764401305,
+            0.014643335833970546,
+            0.010618070641357003,
+            -0.0022177303813575846,
+            -0.0024402761885758007,
+            0.017541799738501007,
+            -0.005445774414228888,
+            -0.00023502936119626386,
+            -0.00537569046685066,
+            -0.005974703247446444,
+            -0.010712244203425036,
+            0.01843613166635215,
+            -0.009938564643456672,
+            -0.017388592763277214,
+            -0.0015797412875366156,
+            0.02028434083164951,
+            -0.012761534382003973,
+            0.0031906812114950982,
+            0.000788436210275669,
+            0.020589196433948156,
+            0.0007726855314648301,
+            -0.005481260996289263,
+            0.007165014826294827,
+            -0.02956334397512344,
+            0.004095715232997753,
+            0.0003568512047626187,
+            0.027290768799658912,
+            -0.0017631129386615335,
+            0.01552542586826711,
+            -0.020871286597113327,
+            0.004388125824324094,
+            -0.016926008631435403,
+            0.005099899758322398,
+            0.012560923968841436,
+            0.002781049890810361,
+            0.010889021753593997,
+            -0.008500358897706648,
+            -0.021353460542554444,
+            0.012663934009161902,
+            -0.018657094130860155,
+            -0.029156017587920958,
+            0.0018889388761887053,
+            -0.028158076315864772,
+            -0.003048854609750186,
+            0.006812072971079798,
+            -0.035217412591558014,
+            -0.02251675180883659,
+            0.007396524512996277,
+            -0.00995571600201846,
+            -0.011912753666073882,
+            0.010675240017836214,
+            -0.02645155743041798,
+            -0.018137648527851668,
+            0.0010666944556449141,
+            0.012588800077165337,
+            -0.004561825939541096,
+            0.007555558090958502,
+            -0.028558603070876605,
+            0.014816816621445352,
+            -0.01313764261744744,
+            0.016036065063831813,
+            -0.0037797584923144134,
+            0.022217516257304868,
+            0.011131597210722573,
+            0.008795376259031849,
+            -0.004950080558710245,
+            0.021409590863550987,
+            0.005574351961953293,
+            0.051358937509801736,
+            0.031972009383746405,
+            0.0019821003238211276,
+            -0.003251257665378234,
+            0.003313045083311649,
+            0.008294776423730887,
+            -0.00940948257229233,
+            0.005114631360856872,
+            0.028053227377164806,
+            0.016738013668710083,
+            -0.018755924318109216,
+            0.006606711923035118,
+            -0.02702692774986498,
+            0.017950884766781056,
+            -0.022054653845163,
+            0.0003755944703013671,
+            0.007766253634286795,
+            0.0016206993755471613,
+            0.010682347575070117,
+            0.0013842930079185415,
+            0.01705234255261425,
+            0.023975191051210525,
+            -0.002976402980885077,
+            0.017466027107477605,
+            0.012734909275073793,
+            0.018143161773162015,
+            -0.0038399373197265827,
+            0.006929440921265761,
+            -0.012003095293274786,
+            -0.0008883470597690258,
+            0.0020490527756307267,
+            -0.004231091383919565,
+            0.012128848448049833,
+            0.008340302519160299,
+            0.012989849936937843,
+            0.002974488532907779,
+            0.003198854381461613,
+            -0.005309753760578859,
+            -0.0017024982049936384,
+            -0.01763395035509857,
+            -0.01819680226734474,
+            0.028417724274372468,
+            -0.020217879775485576,
+            -0.050118184195774004,
+            0.0025509548991664425,
+            0.014185761006969567,
+            -0.0012021669304388173,
+            -0.0051260576846881515,
+            -0.0005774778872927327,
+            0.005718864494530703,
+            -0.016918749210038116,
+            -0.0015472638835434902,
+            0.008125845907627804,
+            0.020347547062899797,
+            -0.002036218493206914,
+            -0.01079739518866248,
+            0.0038112958713600906,
+            0.02169339303105242,
+            0.01667594322285835,
+            0.012338193092567552,
+            -0.009798409233733167,
+            0.0003198215558702285,
+            -1.6308740825163337e-05,
+            0.0076870827874737325,
+            0.004356644356656038,
+            0.003500023005601613,
+            -0.019968549869439228,
+            -0.008634740687755112,
+            0.012781565963545847,
+            -0.0025637758121512927,
+            0.010243567793632335,
+            -0.00042296044205132426,
+            -0.003069251553118065,
+            -0.009344199449486244,
+            -0.004694953215957636,
+            -0.009583993744231791,
+            0.013783878036521561,
+            -0.007705982042252044,
+            -0.01549567538665377,
+            0.00020666298495823114,
+            0.0005515775451348472,
+            0.001667367278489879,
+            -0.004518431526236256,
+            0.03302324454129141,
+            0.009531990824506103,
+            -0.0004937348108340977,
+            -0.004047613286672229,
+            -0.0327383112522095,
+            0.012456150771224677,
+            -0.009677533214778067,
+            0.0006455811519981003,
+            0.020012976827426605,
+            -0.00020277620481825933,
+            0.002168432791818704,
+            -0.015657551629104767,
+            0.0032924357559238502,
+            0.006161295735841883,
+            0.013814165416788555,
+            0.0005164426482972658,
+            0.01076512534105537,
+            -0.006498977875968572,
+            -0.010325144017898257,
+            0.004804158067334203,
+            0.0104064855510473,
+            -0.0012188100892986933,
+            0.005171197790296134
+          ],
+          "title": "Mixture-Of-Experts With Expert Choice Routing",
+          "keyphrases": [
+            "Mixture-Of-Experts",
+            "Expert Choice Routing",
+            "sparsely-activated models",
+            "training efficiency",
+            "downstream performance"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "33412f87-7418-4ce2-9116-377a27fd2732",
+        "properties": {
+          "page_content": "4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n",
+          "title": "4 Experiments",
+          "embedding": [
+            0.009938324801623821,
+            -0.009144282899796963,
+            -0.026211904361844063,
+            -0.022352691739797592,
+            0.016043050214648247,
+            0.016247965395450592,
+            -0.02783413790166378,
+            0.0537557490170002,
+            -0.02827811799943447,
+            -0.01755429059267044,
+            0.010237157344818115,
+            0.001297788112424314,
+            0.014762339182198048,
+            -0.01001516729593277,
+            0.007487895432859659,
+            0.015283161774277687,
+            0.019347287714481354,
+            0.019910801202058792,
+            0.02535809576511383,
+            -0.04251963645219803,
+            0.0020213902462273836,
+            -0.032034873962402344,
+            -0.058332156389951706,
+            0.0034600566141307354,
+            0.001006959704682231,
+            0.01755429059267044,
+            0.011645940132439137,
+            -0.0015795447397977114,
+            -0.053584985435009,
+            0.04794985428452492,
+            0.016555335372686386,
+            0.0044910297729074955,
+            -0.007833687588572502,
+            -0.004798400681465864,
+            -0.04204150289297104,
+            -0.028961164876818657,
+            0.05720513314008713,
+            0.005165537819266319,
+            -0.005519868340343237,
+            0.05481446906924248,
+            0.020252324640750885,
+            0.010245694778859615,
+            -0.03961668908596039,
+            -0.009904172271490097,
+            0.010134699754416943,
+            0.001490962109528482,
+            -0.0010427129454910755,
+            -0.007479357533156872,
+            0.02295035682618618,
+            -0.04146091267466545,
+            -0.005549751687794924,
+            -0.03811398521065712,
+            0.01863008923828602,
+            0.013780459761619568,
+            -0.04426140338182449,
+            0.023223575204610825,
+            -0.0052978782914578915,
+            0.03527934476733208,
+            0.057854026556015015,
+            0.02783413790166378,
+            0.024162763729691505,
+            -0.02590453252196312,
+            -0.04340759664773941,
+            -0.0036180110182613134,
+            0.010143238119781017,
+            0.008119713515043259,
+            -0.01344747468829155,
+            -0.010527451522648335,
+            0.046173933893442154,
+            -0.023701708763837814,
+            -0.05327761545777321,
+            0.027680452913045883,
+            -0.013584083877503872,
+            -0.008081291802227497,
+            0.0036372218746691942,
+            -0.03517688810825348,
+            0.012807118706405163,
+            0.03548425808548927,
+            0.01004078146070242,
+            0.015018481761217117,
+            0.001224147155880928,
+            0.008418546058237553,
+            -0.0014920294051989913,
+            0.017912890762090683,
+            0.028517184779047966,
+            -0.01417321152985096,
+            -0.006309640593826771,
+            -0.0686461552977562,
+            0.025767924264073372,
+            -0.007368362508714199,
+            -0.020115714520215988,
+            0.00026281276950612664,
+            -0.007099412847310305,
+            0.0033597343135625124,
+            -0.020781684666872025,
+            0.07472527027130127,
+            -0.013105952180922031,
+            0.014386664144694805,
+            -0.02096952311694622,
+            0.007300057914108038,
+            -0.031932417303323746,
+            0.02236976847052574,
+            0.01146664097905159,
+            0.005916888825595379,
+            0.038318898528814316,
+            -0.02834642305970192,
+            -0.005869929678738117,
+            0.0122009152546525,
+            0.0292856115847826,
+            0.08387809246778488,
+            0.01419028826057911,
+            -0.014352511614561081,
+            -0.025563009083271027,
+            -0.016538260504603386,
+            -0.046105626970529556,
+            -0.0292856115847826,
+            -0.025170257315039635,
+            -0.027099864557385445,
+            0.036338064819574356,
+            -0.0010613900376483798,
+            0.012388752773404121,
+            0.05338007211685181,
+            0.015863751992583275,
+            -0.00874299369752407,
+            -0.039241012185811996,
+            0.0438857264816761,
+            -0.006198645569384098,
+            0.010809208266437054,
+            0.002330895746126771,
+            -0.0146428057923913,
+            -0.006185838486999273,
+            0.011833777651190758,
+            -0.013763383962213993,
+            -0.029456373304128647,
+            0.031949494034051895,
+            -0.01727253384888172,
+            0.03944592550396919,
+            0.06239628419280052,
+            -0.03562086820602417,
+            -0.02001325786113739,
+            0.03189826384186745,
+            -0.035825781524181366,
+            0.018288565799593925,
+            -0.0005250918911769986,
+            -0.030993228778243065,
+            0.026246055960655212,
+            0.01562468521296978,
+            0.06092773377895355,
+            -0.024196917191147804,
+            0.028705023229122162,
+            -0.019176525995135307,
+            0.01628211699426174,
+            -0.020627999678254128,
+            0.03596239164471626,
+            0.0023351646959781647,
+            -0.024265222251415253,
+            -0.015752756968140602,
+            0.014694035053253174,
+            -0.014275669120252132,
+            -0.022830823436379433,
+            0.019740039482712746,
+            0.02100367471575737,
+            -0.010450609028339386,
+            -0.0049051265232264996,
+            0.012013077735900879,
+            0.04989653453230858,
+            0.016205275431275368,
+            -0.014130521565675735,
+            -0.07479357719421387,
+            0.04801815748214722,
+            0.021703798323869705,
+            0.01958635449409485,
+            0.023104043677449226,
+            0.035825781524181366,
+            -0.001056053675711155,
+            0.030566323548555374,
+            0.0019007899099960923,
+            -0.013601160608232021,
+            -0.02044016122817993,
+            0.043646663427352905,
+            -0.0039019023533910513,
+            -0.0019274713704362512,
+            0.0076373121701180935,
+            0.018288565799593925,
+            0.03123229369521141,
+            0.005430218297988176,
+            0.010399380698800087,
+            0.039719145745038986,
+            -0.011586173437535763,
+            0.01414759736508131,
+            0.009460192173719406,
+            0.02091829478740692,
+            0.04299776628613472,
+            0.02245514839887619,
+            -0.008670419454574585,
+            -0.02194286324083805,
+            0.00032551426556892693,
+            0.0219599399715662,
+            0.01811780408024788,
+            0.02199409157037735,
+            -0.012013077735900879,
+            -0.0220111683011055,
+            0.009383349679410458,
+            -0.02199409157037735,
+            0.01682855375111103,
+            -0.00167666538618505,
+            0.01859593763947487,
+            0.013865840621292591,
+            -0.006258412264287472,
+            -0.01677732542157173,
+            0.0002502724528312683,
+            -0.018186109140515327,
+            -0.008380125276744366,
+            -0.010945817455649376,
+            -0.016495568677783012,
+            0.03364003449678421,
+            -0.020747533068060875,
+            0.006599935237318277,
+            -0.01810072734951973,
+            -0.011885006912052631,
+            -0.005835777148604393,
+            0.03596239164471626,
+            0.012209453620016575,
+            -0.01851055584847927,
+            -0.026058217510581017,
+            -0.011108040809631348,
+            -0.018817927688360214,
+            -0.006463326048105955,
+            0.0342206209897995,
+            -0.01750306226313114,
+            0.006659701932221651,
+            -0.004211407620459795,
+            -0.002409872831776738,
+            0.004258367232978344,
+            0.033435117453336716,
+            -0.03082246705889702,
+            -0.01676025055348873,
+            0.023582175374031067,
+            0.008059946820139885,
+            0.029046546667814255,
+            -0.016888320446014404,
+            -0.006275488529354334,
+            -0.024231068789958954,
+            -0.016973702237010002,
+            -0.01643580198287964,
+            -0.005118578672409058,
+            -0.02199409157037735,
+            -0.016025975346565247,
+            0.02790244296193123,
+            0.013686541467905045,
+            -0.00790199264883995,
+            -0.019313136115670204,
+            -0.002333030104637146,
+            0.02542640082538128,
+            -0.013592622242867947,
+            0.04064125940203667,
+            0.050545431673526764,
+            -0.05713682621717453,
+            -0.0122521435841918,
+            -0.03661128506064415,
+            0.07684271782636642,
+            -0.015377080999314785,
+            -0.03032725863158703,
+            0.033964481204748154,
+            -0.029388070106506348,
+            -0.008990597911179066,
+            0.026997406035661697,
+            0.025272715836763382,
+            -0.014719649218022823,
+            0.01123611256480217,
+            0.03756754845380783,
+            -0.030139420181512833,
+            -0.0439881831407547,
+            -0.03777246177196503,
+            0.019859572872519493,
+            -0.025289790704846382,
+            -0.01419028826057911,
+            -0.0110311983153224,
+            0.023530947044491768,
+            0.01801534742116928,
+            0.010664060711860657,
+            0.007547662127763033,
+            0.006843270733952522,
+            0.021157361567020416,
+            0.02144765481352806,
+            0.007073798682540655,
+            -0.03237639740109444,
+            0.0003433908859733492,
+            -0.03944592550396919,
+            -0.009093054570257664,
+            -0.025289790704846382,
+            -0.02095244638621807,
+            -0.023821242153644562,
+            0.00949434470385313,
+            0.03275207430124283,
+            -0.0008143192972056568,
+            0.0067365444265306,
+            0.015044095925986767,
+            0.015488076023757458,
+            -0.03807983547449112,
+            -0.03555256128311157,
+            0.006348061840981245,
+            0.01682855375111103,
+            -0.0055625587701797485,
+            -0.059220120310783386,
+            -0.00042770442087203264,
+            -0.003165493020787835,
+            -0.005016121547669172,
+            -0.028909936547279358,
+            0.010476223193109035,
+            0.013345018029212952,
+            0.010237157344818115,
+            0.0110311983153224,
+            -0.027526767924427986,
+            -0.014651344157755375,
+            -0.0036073385272175074,
+            0.006019345950335264,
+            0.02100367471575737,
+            -0.040709562599658966,
+            -0.022830823436379433,
+            0.03818229213356972,
+            0.0003810117777902633,
+            -0.06598227471113205,
+            -0.005387527868151665,
+            -0.010757979936897755,
+            0.015436847694218159,
+            -0.0005656477296724916,
+            0.034476764500141144,
+            -0.03121521882712841,
+            -0.019296059384942055,
+            -0.020815838128328323,
+            -0.051979828625917435,
+            -0.03531349450349808,
+            -0.015436847694218159,
+            0.007389707490801811,
+            -0.038353051990270615,
+            -0.024265222251415253,
+            0.004111085087060928,
+            -0.0268949493765831,
+            0.007398245856165886,
+            -0.013567008078098297,
+            0.015633223578333855,
+            -0.03852381557226181,
+            -0.010092009790241718,
+            0.004262635949999094,
+            -0.01861301250755787,
+            -0.005507061257958412,
+            0.014369587413966656,
+            0.013925607316195965,
+            0.014719649218022823,
+            0.0017268265364691615,
+            0.015266085974872112,
+            0.0062071834690868855,
+            -0.00769280968233943,
+            0.007833687588572502,
+            -0.04091447591781616,
+            0.0014461372047662735,
+            0.018852079287171364,
+            0.011432488448917866,
+            -0.011082426644861698,
+            0.004151640925556421,
+            -0.010373766534030437,
+            -0.03034433349967003,
+            0.029917430132627487,
+            -0.05419972911477089,
+            0.0024141420144587755,
+            0.005554020404815674,
+            -0.016410188749432564,
+            -0.008384393528103828,
+            -0.009690719656646252,
+            -0.03708941861987114,
+            -0.01248267199844122,
+            0.0039019023533910513,
+            0.00019050588889513165,
+            0.009349197149276733,
+            0.002870929194614291,
+            0.05471201241016388,
+            -0.003165493020787835,
+            -0.04484199360013008,
+            0.023308956995606422,
+            -0.03671374171972275,
+            0.018749622628092766,
+            0.04245132952928543,
+            0.020713381469249725,
+            -0.017212767153978348,
+            0.04791570082306862,
+            0.0195692777633667,
+            0.000224524803343229,
+            -0.016999315470457077,
+            0.025153182446956635,
+            -0.007099412847310305,
+            -0.061303410679101944,
+            -0.004448339343070984,
+            -0.018749622628092766,
+            0.018664240837097168,
+            0.016674868762493134,
+            -0.002407738473266363,
+            -0.010032243095338345,
+            0.009426039643585682,
+            -0.01510386262089014,
+            -0.017443295568227768,
+            0.01705908216536045,
+            -0.011458102613687515,
+            -0.004525181837379932,
+            0.007526317145675421,
+            0.013148642145097256,
+            0.01271320041269064,
+            0.008853988721966743,
+            0.023992003872990608,
+            -0.045627497136592865,
+            0.009127207100391388,
+            -0.01997910626232624,
+            -0.036918655037879944,
+            -0.002990462351590395,
+            0.01318279467523098,
+            0.04183658957481384,
+            -0.02779998630285263,
+            0.0019253368955105543,
+            0.005007583647966385,
+            -0.013515779748558998,
+            -0.021106131374835968,
+            0.00341309723444283,
+            0.04600317031145096,
+            -0.0015763428527861834,
+            0.021618416532874107,
+            -0.011304416693747044,
+            -0.03562086820602417,
+            0.008811297826468945,
+            0.04142676293849945,
+            -0.005028928630053997,
+            0.01663217879831791,
+            0.009434578008949757,
+            0.03548425808548927,
+            -0.04590071365237236,
+            -0.007833687588572502,
+            -0.02344556525349617,
+            -0.01813488081097603,
+            0.04105108603835106,
+            -0.03075416199862957,
+            -0.01365238893777132,
+            0.03524519130587578,
+            -0.005306416191160679,
+            0.02441890724003315,
+            -0.018271489068865776,
+            -0.02588745579123497,
+            0.0014600115828216076,
+            -0.03712356835603714,
+            -0.00708660576492548,
+            -0.01806657575070858,
+            -0.006160224322229624,
+            -0.06557244807481766,
+            0.00010459146869834512,
+            -0.07431544363498688,
+            -0.002902946900576353,
+            0.025136105716228485,
+            -0.006552976090461016,
+            0.04429555684328079,
+            -0.026228979229927063,
+            0.02098659984767437,
+            -0.05553166940808296,
+            -0.01367800310254097,
+            0.00978463888168335,
+            0.03375956416130066,
+            0.007611697539687157,
+            -0.001995776081457734,
+            0.0015507286880165339,
+            -0.026997406035661697,
+            -0.0018303507240489125,
+            -0.007282981649041176,
+            -0.001387437921948731,
+            4.89605481561739e-05,
+            -0.018852079287171364,
+            0.00950288213789463,
+            0.000838332693092525,
+            0.01610281690955162,
+            0.031454283744096756,
+            0.022711291909217834,
+            -0.03070293366909027,
+            -0.008862526156008244,
+            0.005195421166718006,
+            0.02587038092315197,
+            -0.02978082001209259,
+            0.003897633170709014,
+            -0.03464752435684204,
+            0.014864795841276646,
+            5.636466448777355e-05,
+            0.016657792031764984,
+            0.008008718490600586,
+            0.017016392201185226,
+            -0.040197279304265976,
+            0.0030160765163600445,
+            -0.0016254368238151073,
+            -0.027202321216464043,
+            0.020867066457867622,
+            0.002305281348526478,
+            -0.022147778421640396,
+            0.0054472945630550385,
+            -0.004207138437777758,
+            0.024213993921875954,
+            0.04473953694105148,
+            -0.0010592554463073611,
+            -0.02930268831551075,
+            -0.012704662047326565,
+            0.042212262749671936,
+            0.02633143588900566,
+            -0.028483033180236816,
+            0.017118848860263824,
+            0.02103782817721367,
+            -0.02341141365468502,
+            -0.021089056506752968,
+            0.013925607316195965,
+            -0.016179660335183144,
+            -0.0007694943924434483,
+            -0.005468640010803938,
+            0.010399380698800087,
+            0.01240582950413227,
+            -0.029883278533816338,
+            0.05676115304231644,
+            -0.016820017248392105,
+            -0.006834732368588448,
+            -0.03034433349967003,
+            -0.006168762221932411,
+            -0.014531810767948627,
+            0.05553166940808296,
+            0.0028175662737339735,
+            0.02728770114481449,
+            0.01658095046877861,
+            0.002264725510030985,
+            0.05628301948308945,
+            -0.03678204491734505,
+            -0.03451091796159744,
+            0.018681317567825317,
+            -0.0055625587701797485,
+            -0.00671093026176095,
+            -0.022728366777300835,
+            -0.014207364059984684,
+            -0.013353556394577026,
+            0.016965163871645927,
+            -0.0268778745085001,
+            -0.012260681949555874,
+            0.019740039482712746,
+            0.017690900713205338,
+            -2.0469710761972237e-06,
+            0.02293328195810318,
+            0.012644895352423191,
+            0.014284206554293633,
+            0.017878737300634384,
+            -0.020867066457867622,
+            0.019142374396324158,
+            0.024111535400152206,
+            0.009946862235665321,
+            -0.01763967238366604,
+            -0.017878737300634384,
+            0.007347017526626587,
+            -0.0038613462820649147,
+            -0.007125027012079954,
+            0.01437812577933073,
+            0.007043915335088968,
+            -0.004047049675136805,
+            -0.027151092886924744,
+            -0.018356870859861374,
+            0.02093537151813507,
+            -0.0023757205344736576,
+            0.01539415679872036,
+            -0.00928943045437336,
+            -0.010612832382321358,
+            0.03763585537672043,
+            0.01861301250755787,
+            -0.046139780431985855,
+            0.026280207559466362,
+            0.01051037572324276,
+            -0.006228528916835785,
+            -0.013729231432080269,
+            0.005626594182103872,
+            0.009101592935621738,
+            0.010220080614089966,
+            0.016572412103414536,
+            -0.005097233224660158,
+            -0.03328143432736397,
+            0.01564176194369793,
+            0.026211904361844063,
+            -0.020798761397600174,
+            0.012021616101264954,
+            0.00017663151083979756,
+            0.022762520238757133,
+            -0.005028928630053997,
+            -0.012311910279095173,
+            -0.01562468521296978,
+            0.009741948917508125,
+            -0.012235067784786224,
+            0.01123611256480217,
+            0.037396788597106934,
+            0.015914980322122574,
+            -0.03565501794219017,
+            0.015735680237412453,
+            -0.01686270721256733,
+            -0.015684451907873154,
+            0.03261546418070793,
+            0.008264861069619656,
+            -0.02636558935046196,
+            -0.007142103277146816,
+            -0.006527361460030079,
+            0.02001325786113739,
+            -0.02102075144648552,
+            -0.008508196100592613,
+            0.004418455995619297,
+            0.0035646480973809958,
+            0.009938324801623821,
+            0.006745082791894674,
+            0.005016121547669172,
+            0.019159449264407158,
+            0.013806073926389217,
+            -0.0005186883499845862,
+            0.012730276212096214,
+            0.011577636003494263,
+            0.002902946900576353,
+            0.0075391242280602455,
+            -0.010997045785188675,
+            -0.02295035682618618,
+            0.004734364803880453,
+            0.05945918336510658,
+            -0.005852853413671255,
+            0.03876287862658501,
+            0.010407919064164162,
+            0.028961164876818657,
+            0.021686721593141556,
+            -0.024299373850226402,
+            -0.01681147888302803,
+            -0.018151957541704178,
+            -0.021737949922680855,
+            -0.04453462362289429,
+            -0.01510386262089014,
+            -0.01629919372498989,
+            -0.013404784724116325,
+            0.005694898776710033,
+            -0.00013474156730808318,
+            -0.0013116624904796481,
+            0.011355645954608917,
+            -0.002003246918320656,
+            -0.012491210363805294,
+            -0.003355465130880475,
+            0.05324346199631691,
+            -0.0011056812945753336,
+            0.017912890762090683,
+            -0.004576410632580519,
+            0.015539304353296757,
+            -0.007300057914108038,
+            -0.016111355274915695,
+            0.029131926596164703,
+            0.0035561099648475647,
+            -0.0007540191290900111,
+            0.012106996960937977,
+            0.004469684325158596,
+            0.04149506613612175,
+            -0.005887005943804979,
+            -0.026058217510581017,
+            0.011910621076822281,
+            -0.017187153920531273,
+            0.020371858030557632,
+            0.00769707839936018,
+            -0.007607428822666407,
+            0.001932807732373476,
+            0.012909576296806335,
+            -0.027116939425468445,
+            0.019705887883901596,
+            -0.020645076408982277,
+            -0.045149363577365875,
+            0.009280892089009285,
+            -0.01175693515688181,
+            0.023035738617181778,
+            0.001117421081289649,
+            -0.010800670832395554,
+            -0.03507443144917488,
+            0.0021387888118624687,
+            0.0039040367119014263,
+            -0.03372541442513466,
+            0.006548706907778978,
+            0.01863008923828602,
+            0.015539304353296757,
+            0.009485806338489056,
+            -0.012465596199035645,
+            0.003758889390155673,
+            0.0031847036443650723,
+            -0.012371676973998547,
+            0.02776583470404148,
+            -0.004742903169244528,
+            -0.031983643770217896,
+            0.035859934985637665,
+            -0.02733892947435379,
+            0.009468729607760906,
+            -0.020730456337332726,
+            -0.012346062809228897,
+            -0.01608574204146862,
+            0.00441418681293726,
+            -0.02148180827498436,
+            -0.002561423694714904,
+            -0.02983205020427704,
+            0.027577996253967285,
+            0.009383349679410458,
+            -0.015249010175466537,
+            -0.004862436093389988,
+            0.009101592935621738,
+            -0.025255639106035233,
+            0.007846495136618614,
+            0.012585128657519817,
+            -0.010646984912455082,
+            -0.01367800310254097,
+            -0.00706099160015583,
+            -0.0014685495989397168,
+            0.005972386337816715,
+            0.0158210601657629,
+            0.04672037065029144,
+            0.004431263078004122,
+            0.03712356835603714,
+            0.02337726205587387,
+            0.013532855547964573,
+            0.0037012575194239616,
+            0.0047898623161017895,
+            -0.0067322757095098495,
+            0.04012897238135338,
+            0.006249873898923397,
+            0.0219599399715662,
+            0.008247784338891506,
+            -0.044671230018138885,
+            0.008503926917910576,
+            0.03264961391687393,
+            0.026621730998158455,
+            0.018715469166636467,
+            0.009588262997567654,
+            -0.0024013349320739508,
+            0.021191513165831566,
+            -0.020252324640750885,
+            -0.01705908216536045,
+            0.0035902622621506453,
+            -0.007530585862696171,
+            -0.0056650154292583466,
+            -0.009716334752738476,
+            0.03387909755110741,
+            -0.003229528432711959,
+            -0.008862526156008244,
+            0.018186109140515327,
+            -0.007432397920638323,
+            0.00475997943431139,
+            0.006245605181902647,
+            0.0011163539020344615,
+            0.02739015780389309,
+            0.008683227002620697,
+            0.027680452913045883,
+            0.005391797050833702,
+            0.007573276292532682,
+            0.0058571225963532925,
+            -0.009451653808355331,
+            0.018920384347438812,
+            -0.0097590247169137,
+            -0.033366814255714417,
+            -0.009443115442991257,
+            0.0016393112018704414,
+            0.014070754870772362,
+            0.0004023569927085191,
+            -0.014062216505408287,
+            -0.011893544346094131,
+            0.007859301753342152,
+            0.012533900327980518,
+            0.02107197977602482,
+            0.018169032409787178,
+            -0.005016121547669172,
+            -0.024726277217268944,
+            0.012593667022883892,
+            0.011056812480092049,
+            -0.01861301250755787,
+            0.008465506136417389,
+            0.012542438693344593,
+            0.057410046458244324,
+            -0.044705383479595184,
+            0.016538260504603386,
+            0.02151595987379551,
+            -0.004896588623523712,
+            -0.015249010175466537,
+            -0.010851899161934853,
+            0.0031633584294468164,
+            -0.017477449029684067,
+            0.014975790865719318,
+            -0.01798119582235813,
+            0.011193421669304371,
+            -0.014745263382792473,
+            0.00743666710332036,
+            0.012866885401308537,
+            -0.009682182222604752,
+            0.003942457959055901,
+            0.020371858030557632,
+            0.0028068935498595238,
+            -0.0077867284417152405,
+            0.005387527868151665,
+            0.01390853151679039,
+            0.020354781299829483,
+            0.006066305562853813,
+            -0.0005933965439908206,
+            0.019876649603247643,
+            -0.028175661340355873,
+            -0.025170257315039635,
+            0.01909114606678486,
+            -0.03551841154694557,
+            0.005472908727824688,
+            0.02790244296193123,
+            -0.04296361654996872,
+            -0.05020390823483467,
+            0.0027407235465943813,
+            -0.039719145745038986,
+            0.0004191663465462625,
+            -0.0023287611547857523,
+            0.01536854263395071,
+            -0.004057722166180611,
+            -0.017212767153978348,
+            0.0146428057923913,
+            -0.012397291138768196,
+            -0.0067322757095098495,
+            0.009835867211222649,
+            0.009016212075948715,
+            -0.024572592228651047,
+            -0.0016222350532189012,
+            -0.030890770256519318,
+            0.009169897064566612,
+            -0.017844585701823235,
+            -0.019142374396324158,
+            -0.025528857484459877,
+            -0.0005656477296724916,
+            -0.010638446547091007,
+            -0.001638244022615254,
+            0.051501695066690445,
+            0.021652569994330406,
+            0.020166942849755287,
+            -0.007726961746811867,
+            -0.023650480434298515,
+            0.05457540228962898,
+            0.0018132745753973722,
+            0.03814813867211342,
+            -0.012380215339362621,
+            -0.0002006448630709201,
+            -0.0008852921309880912,
+            0.022113624960184097,
+            -0.017101772129535675,
+            -0.0011547752656042576,
+            -0.011022660881280899,
+            -0.00586566049605608,
+            -0.0070780678652226925,
+            0.014045140706002712,
+            0.015974747017025948,
+            -0.002610517665743828,
+            0.014156135730445385,
+            0.01221799198538065,
+            0.022694215178489685,
+            0.03130060061812401,
+            0.016230888664722443,
+            -0.0034664601553231478,
+            -0.022199006751179695,
+            -0.012918113730847836,
+            -0.008200825192034245,
+            0.008999135345220566,
+            -0.008525271899998188,
+            -0.0031847036443650723,
+            -0.02631436102092266,
+            -0.020730456337332726,
+            -0.011312955059111118,
+            -0.01292665209621191,
+            -0.0031462821643799543,
+            0.02641681768000126,
+            -0.002469639526680112,
+            -0.03763585537672043,
+            -0.01511240005493164,
+            -0.03524519130587578,
+            -0.015607609413564205,
+            -0.0017876604106277227,
+            -0.01748598739504814,
+            0.016538260504603386,
+            -0.003780234605073929,
+            -0.007833687588572502,
+            -0.0008964983280748129,
+            -0.018169032409787178,
+            0.009690719656646252,
+            -0.011048275046050549,
+            -0.000543768925126642,
+            -0.0017503062263131142,
+            0.013345018029212952,
+            -0.008205094374716282,
+            0.008973521180450916,
+            -0.01412198320031166,
+            0.006868884898722172,
+            -0.048120614141225815,
+            -0.027731681242585182,
+            -0.027099864557385445,
+            0.022557605057954788,
+            -0.015214857645332813,
+            -0.023189423605799675,
+            0.008734455332159996,
+            -0.013686541467905045,
+            0.008939368650317192,
+            -0.012021616101264954,
+            0.008388662710785866,
+            0.019210679456591606,
+            0.009024749509990215,
+            -0.002614786848425865,
+            -0.013669464737176895,
+            -0.02390662208199501,
+            -0.02585330419242382,
+            0.023650480434298515,
+            -0.026485122740268707,
+            0.01071528997272253,
+            -0.012021616101264954,
+            0.008585038594901562,
+            -0.029917430132627487,
+            -0.020730456337332726,
+            0.0044910297729074955,
+            0.011347107589244843,
+            -0.021140284836292267,
+            0.044671230018138885,
+            0.04043634235858917,
+            0.029985735192894936,
+            -0.020252324640750885,
+            0.05139923840761185,
+            -0.022796671837568283,
+            0.007312864996492863,
+            0.036440521478652954,
+            0.030446790158748627,
+            0.026092370972037315,
+            -0.015581995248794556,
+            -0.012294834479689598,
+            -0.015496614389121532,
+            0.013404784724116325,
+            -0.024333525449037552,
+            0.002821835223585367,
+            -0.02042308636009693,
+            0.013643850572407246,
+            0.03451091796159744,
+            -0.004917933605611324,
+            0.029934506863355637,
+            -0.013720693998038769,
+            0.030361410230398178,
+            0.0017652479000389576,
+            -0.026587579399347305,
+            0.015667375177145004,
+            -0.014232978224754333,
+            0.004418455995619297,
+            0.017964119091629982,
+            -0.006087650544941425,
+            0.013575546443462372,
+            -0.061337560415267944,
+            0.028141509741544724,
+            0.01758844405412674,
+            0.01123611256480217,
+            -0.002180411946028471,
+            -0.024726277217268944,
+            -0.0021142419427633286,
+            0.009340658783912659,
+            -0.0016745307948440313,
+            0.011859391815960407,
+            0.01193623524159193,
+            -0.033418044447898865,
+            0.010578680783510208,
+            -0.013635313138365746,
+            0.024299373850226402,
+            0.006258412264287472,
+            0.0007476155878975987,
+            -0.03961668908596039,
+            -0.00560524920001626,
+            0.014830644242465496,
+            0.01095435582101345,
+            -0.005989462602883577,
+            -0.04624223709106445,
+            -0.030105268582701683,
+            0.025563009083271027,
+            0.005110040307044983,
+            -0.05167245492339134,
+            0.014557424932718277,
+            -0.015334390103816986,
+            0.018766699358820915,
+            -0.008708841167390347,
+            -0.025972837582230568,
+            -0.029985735192894936,
+            -0.017187153920531273,
+            0.016726097092032433,
+            0.048564594238996506,
+            0.004207138437777758,
+            0.013345018029212952,
+            -0.0048538981936872005,
+            0.006651164032518864,
+            -0.024760430678725243,
+            0.012030153535306454,
+            -0.027151092886924744,
+            -0.02971251681447029,
+            0.029012393206357956,
+            0.009981014765799046,
+            -0.011330030858516693,
+            0.019330210983753204,
+            -0.017084697261452675,
+            -0.0014141193823888898,
+            0.02240392006933689,
+            -0.0006163425859995186,
+            -0.007295788731426001,
+            -0.010382304899394512,
+            0.013618236407637596,
+            -0.0014354645973071456,
+            -0.01762259565293789,
+            -0.009571187198162079,
+            0.04241717979311943,
+            0.010092009790241718,
+            0.018835002556443214,
+            0.03261546418070793,
+            -0.0018933190731331706,
+            -0.02400907874107361,
+            -0.0027215126901865005,
+            -0.026638807728886604,
+            -0.017033468931913376,
+            -0.014548887498676777,
+            0.016674868762493134,
+            0.023530947044491768,
+            0.015249010175466537,
+            -0.027526767924427986,
+            -0.030019886791706085,
+            -0.01950097270309925,
+            -0.022199006751179695,
+            -0.0034493841230869293,
+            -0.0004213008505757898,
+            -0.026502197608351707,
+            0.007005494087934494,
+            0.0341181643307209,
+            0.03357172757387161,
+            -0.0010870042024180293,
+            -0.009016212075948715,
+            -0.010638446547091007,
+            0.0007721625734120607,
+            0.015231933444738388,
+            -0.015018481761217117,
+            0.0012326851719990373,
+            0.005028928630053997,
+            0.005383259151130915,
+            0.0029285612981766462,
+            -0.007095144130289555,
+            -0.05747834965586662,
+            0.003961668815463781,
+            0.02544347569346428,
+            0.005084426142275333,
+            -0.0008377990452572703,
+            0.02629728429019451,
+            0.0008276600856333971,
+            0.008123982697725296,
+            -0.03142013028264046,
+            0.011108040809631348,
+            0.01803242415189743,
+            0.015163629315793514,
+            -0.018151957541704178,
+            0.0122521435841918,
+            0.0064206356182694435,
+            0.004811207763850689,
+            -0.00030950538348406553,
+            0.010143238119781017,
+            0.028636718168854713,
+            0.019620506092905998,
+            -0.014736725017428398,
+            0.023821242153644562,
+            0.0006435577524825931,
+            -0.012072844430804253,
+            -0.024333525449037552,
+            -0.001040044822730124,
+            0.0029883277602493763,
+            -0.011176345869898796,
+            0.014702572487294674,
+            -0.0023949311580508947,
+            0.0007956422632560134,
+            0.04139260947704315,
+            0.0437491200864315,
+            0.013481627218425274,
+            -0.011295879259705544,
+            0.003304236801341176,
+            0.026502197608351707,
+            0.020132791250944138,
+            0.004862436093389988,
+            0.011458102613687515,
+            -0.03331558406352997,
+            -0.019620506092905998,
+            -0.002981924219056964,
+            0.0014290610561147332,
+            -0.0026788224931806326,
+            0.03876287862658501,
+            0.029524678364396095,
+            0.009682182222604752,
+            -0.005741858389228582,
+            -0.011364183388650417,
+            0.013643850572407246,
+            0.020201096311211586,
+            -0.023599252104759216,
+            -0.0038015798199921846,
+            0.0016745307948440313,
+            -0.017170077189803123,
+            0.02288205362856388,
+            -0.0070311082527041435,
+            -0.00928943045437336,
+            0.02645096927881241,
+            0.09186973422765732,
+            -0.00464471522718668,
+            0.012141148559749126,
+            -0.03948007896542549,
+            -0.021823331713676453,
+            -0.006975610740482807,
+            -0.0036671049892902374,
+            0.015402695164084435,
+            0.02739015780389309,
+            0.009135744534432888,
+            0.007133565377444029,
+            0.017878737300634384,
+            -0.004606293980032206,
+            0.012508286163210869,
+            -0.02826104313135147,
+            0.020166942849755287,
+            -0.005007583647966385,
+            -0.0038335975259542465,
+            -0.011389797553420067,
+            -0.005989462602883577,
+            0.011765473522245884,
+            0.005468640010803938,
+            0.017016392201185226,
+            0.0019893725402653217,
+            -0.0024760430678725243,
+            0.0014333301223814487,
+            0.003031018190085888,
+            -0.010877513326704502,
+            -0.009383349679410458,
+            -0.021396426483988762,
+            0.003827193984761834,
+            0.0009914844995364547,
+            -0.007884915918111801,
+            0.012755890376865864,
+            0.005502792075276375,
+            0.024043232202529907,
+            -0.028175661340355873,
+            -0.0030609015375375748,
+            0.00633525475859642,
+            -0.008956445381045341,
+            0.010390842333436012,
+            0.012380215339362621,
+            -0.008640536107122898,
+            0.022830823436379433,
+            -0.01466841995716095,
+            0.014762339182198048,
+            0.01170570682734251,
+            -0.021601341664791107,
+            -0.004155910108238459,
+            -0.009938324801623821,
+            -0.02192578837275505,
+            0.020252324640750885,
+            -0.018715469166636467,
+            0.005033197812736034,
+            0.00073320756200701,
+            0.006774966139346361,
+            -0.0010496501345187426,
+            -0.01740914396941662,
+            0.010348152369260788,
+            -0.008785683661699295,
+            0.006839001551270485,
+            -0.005058811977505684,
+            0.018254414200782776,
+            0.012166762724518776,
+            0.0028090281412005424,
+            -0.0011622461024671793,
+            0.014600115828216076,
+            -0.00372900627553463,
+            -0.018886230885982513,
+            -0.0036713741719722748,
+            0.01633334532380104,
+            0.01051037572324276,
+            0.0002606782363727689,
+            -0.010305461473762989,
+            -0.0146342683583498,
+            0.014139058999717236,
+            0.000977076473645866,
+            -0.0018122073961421847,
+            -0.0005285604856908321,
+            -0.03760170191526413,
+            0.002606248715892434,
+            0.013532855547964573,
+            0.012704662047326565,
+            -0.005818700883537531,
+            -0.017477449029684067,
+            0.007983104325830936,
+            -0.004190062638372183,
+            -0.0003204447857569903,
+            0.01661510206758976,
+            -0.019774191081523895,
+            0.01002370472997427,
+            -0.014779414981603622,
+            -0.03278622403740883,
+            -0.0029776552692055702,
+            -0.008380125276744366,
+            0.005822970066219568,
+            -0.005592442117631435,
+            -0.0071805245243012905,
+            -0.01414759736508131,
+            -0.005741858389228582,
+            -0.02979789674282074,
+            0.008947907015681267,
+            0.0317445807158947,
+            0.004473953507840633,
+            -0.028978241607546806,
+            -0.012158225290477276,
+            0.025580085813999176,
+            -0.0024931191001087427,
+            -0.006915844511240721,
+            0.003009672975167632,
+            0.0012572321575134993,
+            -0.005784548819065094,
+            -0.009656568057835102,
+            -0.012935190461575985,
+            0.013063261285424232,
+            0.01320840884000063,
+            0.011244650930166245,
+            -0.03357172757387161,
+            0.010664060711860657,
+            0.01813488081097603,
+            -0.02042308636009693,
+            0.011748396791517735,
+            0.028585489839315414,
+            0.0029435029719024897,
+            0.010399380698800087,
+            0.007334209978580475,
+            0.008286206051707268,
+            -0.017153002321720123,
+            0.016683407127857208,
+            0.009186973795294762,
+            -0.012559514492750168,
+            0.004563603550195694,
+            0.007381169591099024,
+            0.008542348630726337,
+            0.01960342936217785,
+            -0.011603250168263912,
+            0.013609698973596096,
+            -0.0068816919811069965,
+            0.03321312740445137,
+            -0.011919158510863781,
+            -0.014881872572004795,
+            0.03229101747274399,
+            0.027560919523239136,
+            -0.004563603550195694,
+            0.017246920615434647,
+            -0.01943266950547695,
+            -0.01267050951719284,
+            0.019296059384942055,
+            0.0005907283630222082,
+            0.019313136115670204,
+            -0.011304416693747044,
+            -0.02595576085150242,
+            0.029012393206357956,
+            -0.008106905966997147,
+            -0.001259366748854518,
+            -0.015829598531126976,
+            0.0043458822183310986,
+            -0.0005789885181002319,
+            0.037328481674194336,
+            0.012653433717787266,
+            0.0011985328746959567,
+            0.026570502668619156,
+            -0.00769280968233943,
+            -0.025067800655961037,
+            0.001990439835935831,
+            -0.015214857645332813,
+            0.013336479663848877,
+            -0.027970748022198677,
+            -0.00560524920001626,
+            0.014506196603178978,
+            0.003026749240234494,
+            -0.002644670195877552,
+            0.01666633039712906,
+            -0.007513510063290596,
+            0.011791087687015533,
+            -0.019244831055402756,
+            0.016717558726668358,
+            -0.011517869308590889,
+            0.004473953507840633,
+            -0.011910621076822281,
+            0.026092370972037315,
+            -0.025563009083271027,
+            -0.00999809056520462,
+            0.013660927303135395,
+            -0.008354511111974716,
+            0.0006376878009177744,
+            -0.02252345345914364,
+            -0.008089830167591572,
+            -0.020371858030557632,
+            0.01679440215229988,
+            -0.005682091694325209,
+            0.011902082711458206,
+            -0.00306303589604795,
+            0.0025528857950121164,
+            -0.008585038594901562,
+            0.010322538204491138,
+            -0.0009199780761264265,
+            -0.017417682334780693,
+            -0.005968117620795965,
+            -0.03025895357131958,
+            -0.030361410230398178,
+            0.03072001039981842,
+            -0.00880275946110487,
+            0.003923247568309307,
+            0.02728770114481449,
+            -0.02540932409465313,
+            -0.005485715810209513,
+            0.001918933354318142,
+            0.022728366777300835,
+            -0.030651705339550972,
+            0.007914799265563488,
+            -0.009391887113451958,
+            -0.016452878713607788,
+            0.007184793706983328,
+            -0.0008874266059137881,
+            0.015880826860666275,
+            -0.0025635582860559225,
+            -0.04924764111638069,
+            -0.0007081269286572933,
+            0.017135925590991974,
+            -0.013806073926389217,
+            0.008081291802227497,
+            0.02151595987379551,
+            -0.009630953893065453,
+            -0.008439891040325165,
+            -0.0011334300506860018,
+            0.025989914312958717,
+            -0.022625910118222237,
+            0.030019886791706085,
+            0.005767472553998232,
+            0.004713019821792841,
+            0.005340568721294403,
+            0.002962713595479727,
+            0.01730668731033802,
+            0.002080089645460248,
+            0.004258367232978344,
+            0.025238562375307083,
+            0.013148642145097256,
+            0.0037418133579194546,
+            -0.0032978332601487637,
+            -0.0030352871399372816,
+            0.016538260504603386,
+            -0.018442250788211823,
+            0.02100367471575737,
+            -0.0004714620590675622,
+            0.00013560871593654156,
+            0.016256503760814667,
+            -0.01484772004187107,
+            -0.027219397947192192,
+            0.03075416199862957,
+            -0.022335615009069443,
+            0.01757136732339859,
+            0.023770013824105263,
+            -0.01339624635875225,
+            -0.004922202788293362,
+            -0.007073798682540655,
+            0.001534719718620181,
+            0.018339794129133224,
+            -0.0134218605235219,
+            -0.023308956995606422,
+            -0.00030790449818596244,
+            0.0220111683011055,
+            0.02293328195810318,
+            0.028175661340355873,
+            -0.021140284836292267,
+            0.01412198320031166,
+            0.007466550450772047,
+            -0.008311820216476917,
+            0.016469955444335938,
+            0.008452698588371277,
+            -0.0025272713974118233,
+            0.021293969824910164,
+            0.0075903525575995445,
+            0.004576410632580519,
+            -0.008896678686141968,
+            -0.013242561370134354,
+            -0.01267050951719284,
+            -0.006749351508915424,
+            -0.006164493504911661,
+            -0.008264861069619656,
+            -0.01709323562681675,
+            0.09548988193273544,
+            -0.011867930181324482,
+            -0.0028410458471626043,
+            -0.009195511229336262,
+            0.0026041141245514154,
+            -0.006949996575713158,
+            0.0013479492627084255,
+            0.023086966946721077,
+            -0.022728366777300835,
+            -0.03671374171972275,
+            0.011167807504534721,
+            -0.004070529248565435,
+            0.006288295611739159,
+            0.03712356835603714,
+            0.00038314631092362106,
+            -0.016589488834142685,
+            -6.773765198886395e-05,
+            -0.003372541395947337,
+            0.01295226626098156,
+            0.0028197006322443485,
+            -0.0036158766597509384,
+            0.0109799699857831,
+            0.029200231656432152,
+            -0.017810434103012085,
+            -0.02144765481352806,
+            -0.012525362893939018,
+            0.0032615463715046644,
+            -0.002578499959781766,
+            0.003707661060616374,
+            0.0051996903494000435,
+            -0.014053679071366787,
+            -0.0134218605235219,
+            0.018817927688360214,
+            -0.016094280406832695,
+            0.0008025794522836804,
+            -0.012627819553017616,
+            -0.008222170174121857,
+            0.01753721572458744,
+            0.01757136732339859,
+            -0.03406693786382675,
+            -0.009921248070895672,
+            -0.03855796530842781,
+            0.018749622628092766,
+            0.01643580198287964,
+            0.014540349133312702,
+            -0.020884141325950623,
+            -0.012644895352423191,
+            -0.004520913120359182,
+            -0.02981497347354889,
+            -0.02243807166814804,
+            -0.026536351069808006,
+            0.009605339728295803,
+            0.00365856708958745,
+            0.027595072984695435,
+            0.005259457044303417,
+            -0.006262681446969509,
+            0.00801725685596466,
+            -0.01025423314422369,
+            0.00755620002746582,
+            -0.02146473154425621,
+            0.007107951212674379,
+            -0.0011409008875489235,
+            0.0012529630912467837,
+            0.005592442117631435,
+            -0.008853988721966743,
+            -0.01735791563987732,
+            -0.0079276068136096,
+            0.01466841995716095,
+            -0.015752756968140602,
+            -0.00821790099143982,
+            0.020798761397600174,
+            0.009383349679410458,
+            0.00931504461914301,
+            0.015616146847605705,
+            0.045661646872758865,
+            0.008273398503661156,
+            0.00783795677125454,
+            -0.011056812480092049,
+            0.009648029692471027,
+            -0.01813488081097603,
+            0.00745801255106926,
+            -0.019330210983753204,
+            -0.006292564328759909,
+            -0.019722962751984596,
+            0.04012897238135338,
+            -0.009366272948682308,
+            -0.01635042205452919,
+            0.033435117453336716,
+            -0.011808163486421108,
+            0.01910822093486786,
+            0.0035475720651447773,
+            -0.022830823436379433,
+            0.03352050110697746,
+            0.003876287955790758,
+            0.012918113730847836,
+            -0.025682542473077774,
+            0.0035859933122992516,
+            -0.00877714529633522,
+            0.006032153032720089,
+            -0.011586173437535763,
+            0.003494208911433816,
+            0.0023885276168584824,
+            -0.004841091111302376,
+            -0.008760069496929646,
+            0.018698394298553467,
+            0.006595666520297527,
+            0.0002183880569646135,
+            0.028056127950549126,
+            -0.000954130373429507,
+            0.018220260739326477,
+            6.110063259257004e-05,
+            0.006010808050632477,
+            0.027680452913045883,
+            0.002821835223585367,
+            0.006809118203818798,
+            -0.01714446395635605,
+            -0.0009199780761264265,
+            -0.009485806338489056,
+            0.009596801362931728,
+            0.02639974094927311,
+            -0.010612832382321358,
+            -0.031078608706593513,
+            0.022574681788682938,
+            -0.007048184517771006,
+            0.015966208651661873,
+            0.002999000484123826,
+            -0.014062216505408287,
+            -0.0023437028285115957,
+            -0.027680452913045883,
+            0.014454968273639679,
+            -0.017349377274513245,
+            -0.024265222251415253,
+            -0.007231753319501877,
+            -0.01045914739370346,
+            -0.004845359828323126,
+            0.038353051990270615,
+            0.012474133633077145,
+            -0.024572592228651047,
+            0.006817656569182873,
+            -0.003974475897848606,
+            -0.005989462602883577,
+            0.0057119750417768955,
+            0.005976655520498753,
+            0.004098278004676104,
+            0.010945817455649376,
+            -0.0065700518898665905,
+            -0.01320840884000063,
+            -0.03324728086590767,
+            -0.02240392006933689,
+            0.014941639266908169,
+            0.00028068936080671847,
+            -0.01315718051046133,
+            0.028483033180236816,
+            -0.01002370472997427,
+            -0.005575365852564573,
+            0.0071292961947619915,
+            0.00013887719251215458,
+            -0.0022796671837568283,
+            -0.017844585701823235,
+            0.002273263642564416,
+            0.006245605181902647,
+            -0.014950176700949669,
+            0.009844405576586723,
+            0.0032124524004757404,
+            -0.014540349133312702,
+            0.005635132547467947,
+            -0.0015027020126581192,
+            -0.003562513506039977,
+            -0.02481165900826454,
+            0.0059595792554318905,
+            0.00708233704790473,
+            -0.0018036692636087537,
+            0.002390662208199501,
+            -0.018169032409787178,
+            0.01801534742116928,
+            0.001955220242962241,
+            0.0071762558072805405,
+            -0.029166078194975853,
+            0.007282981649041176,
+            -0.006740813609212637,
+            0.0015080382581800222,
+            -0.0195692777633667,
+            0.009161358699202538,
+            0.020798761397600174,
+            0.012491210363805294,
+            -0.0005250918911769986,
+            0.010442070662975311,
+            0.008175211027264595,
+            -0.012704662047326565,
+            0.003950996324419975,
+            -0.02050846628844738,
+            -0.007308595813810825,
+            -0.010057857260107994,
+            0.011372721754014492,
+            -0.004219945520162582,
+            0.015983285382390022,
+            0.009511420503258705,
+            0.005600980017334223,
+            -0.009699258022010326,
+            0.00743666710332036,
+            0.01292665209621191,
+            0.005874198395758867,
+            -0.0011099503608420491,
+            0.017827508971095085,
+            0.07363239675760269,
+            0.005336299538612366,
+            0.013618236407637596,
+            6.870485958643258e-05,
+            -0.013071799650788307,
+            0.0015624684747308493,
+            -0.004939279053360224,
+            -0.01748598739504814,
+            0.026126522570848465,
+            0.026587579399347305,
+            -0.006382214371114969,
+            0.0004917400074191391,
+            0.009016212075948715,
+            0.005558289587497711,
+            0.005485715810209513,
+            0.022216081619262695,
+            -0.0077867284417152405,
+            0.01753721572458744,
+            0.024179840460419655,
+            0.005541213322430849,
+            -0.0019093280425295234,
+            -0.0022305732127279043,
+            0.009152821265161037,
+            0.004209273029118776,
+            -0.009050363674759865,
+            0.01150079257786274,
+            -0.01437812577933073,
+            0.01909114606678486,
+            0.011389797553420067,
+            0.02146473154425621,
+            0.007073798682540655,
+            -0.01704200729727745,
+            -0.018356870859861374,
+            0.0041281613521277905,
+            -0.0013479492627084255,
+            0.0059083509258925915,
+            0.0063224476762115955,
+            0.005477177910506725,
+            -0.011611787602305412,
+            -0.0041793896816670895,
+            0.010356690734624863,
+            -0.002471773885190487,
+            -0.0008500725380145013,
+            0.01757136732339859,
+            -0.01268758624792099,
+            0.01168009266257286,
+            -0.001839956152252853,
+            -0.0009984216885641217,
+            0.009682182222604752,
+            0.0014354645973071456,
+            -0.011825240217149258,
+            -0.004166582599282265,
+            0.0021003675647079945,
+            0.01217530108988285,
+            -0.013259637169539928,
+            -0.01048476155847311,
+            0.011825240217149258,
+            -0.006168762221932411,
+            0.009169897064566612,
+            -0.009579724632203579,
+            -0.0025315405800938606,
+            -0.00039648707024753094,
+            -0.008085560984909534,
+            -0.0014578771078959107,
+            0.007820880971848965,
+            0.0008324627415277064,
+            0.010518914088606834,
+            0.011560559272766113,
+            0.01150079257786274,
+            0.003560379147529602,
+            0.004002224653959274,
+            -0.002964847953990102,
+            -0.0064206356182694435,
+            -0.011432488448917866,
+            8.15119783510454e-05,
+            0.005776010919362307,
+            -0.02242099680006504,
+            -0.00024800453684292734,
+            0.008465506136417389,
+            0.005046004895120859,
+            -0.0007710952777415514,
+            0.020849989727139473,
+            0.001264702994376421,
+            0.007820880971848965,
+            -0.025221485644578934,
+            0.0011238247388973832,
+            0.006890229880809784,
+            0.007718423847109079,
+            0.012798581272363663,
+            -0.028636718168854713,
+            -0.0038528083823621273,
+            -0.0038634808734059334,
+            0.0171273872256279,
+            -0.00011606451880652457,
+            -0.0033106403425335884,
+            0.021686721593141556,
+            -0.0007833687704987824,
+            -0.003430173499509692,
+            -0.00755620002746582,
+            -0.005579635035246611,
+            0.008004449307918549,
+            0.012994957156479359,
+            0.024623820558190346,
+            -0.0019968433771282434,
+            -0.0055967108346521854,
+            -0.00809409935027361,
+            0.0244530588388443,
+            0.015249010175466537,
+            0.01863008923828602,
+            -0.0219087116420269,
+            0.010638446547091007,
+            -0.013865840621292591,
+            0.009067440405488014,
+            0.011714245192706585,
+            -0.010911665856838226,
+            0.0004290384822525084,
+            -0.010757979936897755,
+            -0.017135925590991974,
+            -0.022711291909217834,
+            0.00502039073035121,
+            0.02839765138924122,
+            -0.0002676154254004359,
+            0.007193332072347403,
+            -0.016358960419893265,
+            -0.0018442251021042466,
+            -0.011312955059111118,
+            -0.025750847533345222,
+            -0.016222350299358368,
+            -0.021772101521492004,
+            -0.02141350321471691,
+            -0.0038592119235545397,
+            -0.002131317974999547,
+            -0.01907406933605671,
+            0.00013153978215996176,
+            -0.0012817791430279613,
+            0.01902284100651741,
+            -0.0014888276346027851,
+            0.0015624684747308493,
+            0.012337524443864822,
+            0.012943727895617485,
+            -0.012824195437133312,
+            -0.004228483885526657,
+            -0.019261907786130905,
+            -0.024179840460419655,
+            0.0032487392891198397,
+            -0.007726961746811867,
+            0.013874378986656666,
+            -0.008234977722167969,
+            0.004260501824319363,
+            -0.017187153920531273,
+            0.014702572487294674,
+            -0.03579162806272507,
+            -0.024128612130880356,
+            0.0006531630642712116,
+            -0.0040299734100699425,
+            -0.009605339728295803,
+            -0.02585330419242382,
+            0.01198746357113123,
+            -0.019620506092905998,
+            0.002104636514559388,
+            -0.018442250788211823,
+            0.011398335918784142,
+            -0.015283161774277687,
+            0.018681317567825317,
+            -0.00025520852068439126,
+            0.03179580718278885,
+            0.004405648913234472,
+            0.007978835143148899,
+            -0.004525181837379932,
+            0.003327716374769807,
+            -0.005468640010803938,
+            0.006365138106048107,
+            0.006988417822867632,
+            0.008055677637457848,
+            0.007825150154531002,
+            -0.004768517334014177,
+            0.004061991348862648,
+            0.005110040307044983,
+            0.030515095219016075,
+            0.028021976351737976,
+            0.0027706066612154245,
+            0.0014674824196845293,
+            0.0034685947466641665,
+            0.004576410632580519,
+            -0.00038021133514121175,
+            0.008862526156008244,
+            -0.011287340894341469,
+            -0.005391797050833702,
+            0.01417321152985096,
+            -0.012448519468307495,
+            -0.0037695621140301228,
+            0.015377080999314785,
+            -0.015086785890161991,
+            -0.026092370972037315,
+            0.021823331713676453,
+            -0.006552976090461016,
+            -0.002450428670272231,
+            -0.006689585279673338,
+            0.008115444332361221,
+            -0.005946772173047066,
+            0.004194331355392933,
+            0.028107358142733574,
+            -0.005737589206546545,
+            -0.010945817455649376,
+            0.012636357918381691,
+            -0.002330895746126771,
+            -0.007948951795697212,
+            0.02291620522737503,
+            0.035894084721803665,
+            0.00999809056520462,
+            -0.026041142642498016,
+            -0.007381169591099024,
+            0.02144765481352806,
+            -0.00973341055214405,
+            -0.021755026653409004,
+            0.008273398503661156,
+            0.007522047962993383,
+            -0.005421680398285389,
+            0.018937459215521812,
+            0.004239156376570463,
+            -0.010792132467031479,
+            0.009041826240718365,
+            0.022711291909217834,
+            0.021686721593141556,
+            0.008947907015681267,
+            -0.023548023775219917,
+            0.01025423314422369,
+            -0.026228979229927063,
+            -0.013831689022481441,
+            -0.023821242153644562,
+            0.003581724362447858,
+            0.003857077332213521,
+            -0.003257277188822627,
+            0.026536351069808006,
+            0.011876468546688557,
+            0.011645940132439137,
+            0.0030950538348406553,
+            0.023052813485264778,
+            -0.015223395079374313,
+            -0.028107358142733574,
+            -0.0009733410552144051,
+            -0.015479537658393383,
+            -0.0018282162491232157,
+            0.002166537567973137,
+            0.007150641642510891,
+            0.005852853413671255,
+            0.01953512616455555,
+            0.000376209121895954,
+            0.0019584218971431255,
+            -2.4263486920972355e-05,
+            0.010100548155605793,
+            -0.015727141872048378,
+            -0.019705887883901596,
+            -0.008922292850911617,
+            6.470263178925961e-05,
+            0.011637401767075062,
+            -0.005208228249102831,
+            -3.448583447607234e-05,
+            0.007846495136618614,
+            -0.015897903591394424,
+            -0.0097590247169137,
+            -0.014412278309464455,
+            -0.0029306956566870213,
+            -0.005071619059890509,
+            0.03136890381574631,
+            0.014437892474234104,
+            -0.004747172351926565,
+            -0.011543483473360538,
+            -0.005741858389228582,
+            0.03370833769440651,
+            0.0017204229952767491,
+            0.012422905303537846,
+            0.0146342683583498,
+            0.015684451907873154,
+            0.027014482766389847,
+            0.012807118706405163,
+            0.003018211107701063,
+            0.002516598906368017,
+            0.02346264198422432,
+            -0.013951221480965614,
+            0.008730186149477959,
+            -0.011645940132439137,
+            -0.003521957667544484,
+            -0.010937280021607876,
+            0.015078248456120491,
+            -0.013003494590520859,
+            -0.011278802528977394,
+            0.023069890215992928,
+            -0.0004381102044135332,
+            0.013276712968945503,
+            -0.0004242358263581991,
+            -0.003953130915760994,
+            -0.02534101903438568,
+            0.0015464596217498183,
+            0.0006531630642712116,
+            -0.004401379730552435,
+            0.010578680783510208,
+            -0.027475539594888687,
+            0.0013660927070304751,
+            0.015240471810102463,
+            -0.007116489112377167,
+            0.00038981667603366077,
+            -0.010826284997165203,
+            0.005904081743210554,
+            0.00011606451880652457,
+            -0.02385539375245571,
+            0.015488076023757458,
+            -0.006698123179376125,
+            0.01243144366890192,
+            -0.0018548977095633745,
+            -0.013788998126983643,
+            0.008687496185302734,
+            0.019296059384942055,
+            0.027560919523239136,
+            -0.007825150154531002,
+            0.0005528406472876668,
+            -0.0001855698210420087,
+            -0.006027883850038052,
+            -0.02737308293581009,
+            0.02054261974990368,
+            -0.007863570936024189,
+            0.01953512616455555,
+            -0.009682182222604752,
+            -0.018186109140515327,
+            -0.014856258407235146,
+            0.011927696876227856,
+            0.007603159639984369,
+            0.0010304395109415054,
+            0.01489894837141037,
+            -0.024077383801341057,
+            0.020867066457867622,
+            0.0033383890986442566,
+            -0.0064718639478087425,
+            0.008303281851112843,
+            0.004230618476867676,
+            0.004695943556725979,
+            -0.008384393528103828,
+            0.007658657152205706,
+            0.007569007575511932,
+            -0.007253098301589489,
+            0.004326671827584505,
+            -0.04057295247912407,
+            0.010732365772128105,
+            -0.011090965010225773,
+            0.005878467578440905,
+            -0.004742903169244528,
+            -0.022745443508028984,
+            -0.035825781524181366,
+            0.004567872267216444,
+            0.019313136115670204,
+            -0.020235247910022736,
+            0.018408099189400673,
+            0.012124072760343552,
+            0.014574501663446426,
+            0.01859593763947487,
+            0.008853988721966743,
+            0.01461719162762165,
+            0.0438857264816761,
+            -0.024231068789958954,
+            6.483604374807328e-05,
+            0.008734455332159996,
+            0.011381260119378567,
+            -0.013404784724116325,
+            -0.010126162320375443,
+            0.005784548819065094,
+            -0.0015197781613096595,
+            0.04067540913820267,
+            -0.019210679456591606,
+            0.010886050760746002,
+            -0.010962894186377525,
+            0.008371586911380291,
+            -0.009562648832798004,
+            -0.0036158766597509384,
+            -0.012559514492750168,
+            -0.011893544346094131,
+            0.014745263382792473,
+            -0.019671734422445297,
+            -0.005460101645439863,
+            -0.007282981649041176,
+            0.01048476155847311,
+            -0.006096188444644213,
+            -0.020730456337332726,
+            0.018647165969014168,
+            -0.0018079383298754692,
+            0.029968658462166786,
+            -0.012755890376865864,
+            0.00574612757191062,
+            0.022591758519411087,
+            -0.010604294948279858,
+            0.018732545897364616,
+            -0.004362958483397961,
+            0.008853988721966743,
+            0.0055625587701797485,
+            -0.0065188235603272915,
+            -0.0014685495989397168,
+            0.01389145478606224,
+            -0.0027385889552533627,
+            0.029934506863355637,
+            -0.017400605604052544,
+            -0.009818791411817074,
+            0.009707796387374401,
+            0.0073769004084169865,
+            0.0042370217852294445,
+            0.003878422547131777,
+            0.009579724632203579,
+            -0.01071528997272253,
+            0.0009728074073791504,
+            -0.021686721593141556,
+            -0.015453923493623734,
+            -0.0011483717244118452,
+            -0.004683136474341154,
+            -0.0018730411538854241,
+            -0.002467504935339093,
+            -0.02044016122817993,
+            0.004627638962119818,
+            -0.0021579996682703495,
+            -8.451364556094632e-05,
+            -0.024487212300300598,
+            -0.00769280968233943,
+            -0.010057857260107994,
+            -0.005161269102245569,
+            -0.003139878623187542,
+            0.018271489068865776,
+            0.0034856710117310286,
+            -0.004862436093389988,
+            0.0293197650462389,
+            0.005404604133218527,
+            0.01536854263395071,
+            0.00280262460000813,
+            -0.05334591865539551,
+            -0.011014122515916824,
+            -0.004563603550195694,
+            -0.009912709705531597,
+            -0.007261636666953564,
+            -0.008456967771053314,
+            -0.010681137442588806,
+            -0.0019061262719333172,
+            0.0032914294861257076,
+            -0.004170851781964302,
+            0.0122521435841918,
+            -0.01700785383582115,
+            0.025494705885648727,
+            -0.004017166327685118,
+            0.015744218602776527,
+            -0.011722782626748085,
+            0.011654478497803211,
+            -0.006194376386702061,
+            -0.010851899161934853,
+            0.010544528253376484,
+            0.0146342683583498,
+            -0.0030544979963451624,
+            0.020371858030557632,
+            -0.014736725017428398,
+            0.003197510726749897,
+            -0.008303281851112843,
+            -0.02143057994544506,
+            -0.015991821885108948,
+            0.011295879259705544,
+            0.03130060061812401,
+            0.010066395625472069,
+            -0.0037524858489632607,
+            0.004072663839906454,
+            0.021669644862413406,
+            -0.004352285992354155,
+            0.0063181789591908455,
+            0.03025895357131958,
+            -0.0010982104577124119,
+            0.017742129042744637,
+            0.0001935742620844394,
+            -0.005028928630053997,
+            -0.002362913452088833,
+            0.01808365248143673,
+            -0.002766337711364031,
+            0.0057119750417768955,
+            0.02441890724003315,
+            0.010997045785188675,
+            -0.0064718639478087425,
+            0.008230708539485931,
+            0.014309820719063282,
+            0.013874378986656666,
+            -0.015351466834545135,
+            -0.0122009152546525,
+            -0.011534945107996464,
+            -0.015778370201587677,
+            0.007291519548743963,
+            -0.019261907786130905,
+            -0.004049184266477823,
+            0.007829418405890465,
+            -0.015138015151023865,
+            -0.012807118706405163,
+            -0.023548023775219917,
+            -0.015914980322122574,
+            0.01863008923828602,
+            0.005357644986361265,
+            -0.01465988252311945,
+            -0.004369362257421017,
+            0.0013714289525523782,
+            0.006181569304317236,
+            -0.0054985228925943375,
+            0.021840406581759453,
+            -0.004098278004676104,
+            -0.011125117540359497,
+            0.019330210983753204,
+            0.0017897948855534196,
+            0.005097233224660158,
+            -0.0064206356182694435,
+            0.004550796467810869,
+            -0.011364183388650417,
+            0.022096550092101097,
+            -0.003470729338005185,
+            0.012636357918381691,
+            -0.0037033918779343367,
+            0.004631908144801855,
+            -0.001978699816390872,
+            0.00821790099143982,
+            0.004017166327685118,
+            0.010629909113049507,
+            -0.005656477529555559,
+            0.004525181837379932,
+            0.012491210363805294,
+            -0.01641872711479664,
+            0.007265905383974314,
+            0.004559334367513657,
+            -4.365760105429217e-05,
+            0.005780279636383057,
+            -0.007645850069820881,
+            -0.005417411215603352,
+            -0.015488076023757458,
+            0.010100548155605793,
+            0.030873695388436317,
+            -0.0016649254830554128,
+            -0.007240291219204664,
+            0.01704200729727745,
+            0.015735680237412453,
+            -0.003816521493718028,
+            0.0061218030750751495,
+            0.009280892089009285,
+            -0.003148416755720973,
+            0.01370361726731062,
+            -0.024487212300300598,
+            -0.029524678364396095,
+            0.01763967238366604,
+            -0.020867066457867622,
+            0.022779595106840134,
+            -0.019210679456591606,
+            -0.027731681242585182,
+            -0.01847640424966812,
+            -0.0032530082389712334,
+            -0.019842496141791344,
+            -0.01611989364027977,
+            0.02532394416630268,
+            0.00685180863365531,
+            -0.006949996575713158,
+            -0.031112760305404663,
+            0.04050464928150177,
+            0.014489120803773403,
+            0.038318898528814316,
+            -0.011603250168263912,
+            -2.554753518779762e-05,
+            -0.007504971697926521,
+            0.012064306065440178,
+            -0.01146664097905159,
+            -0.014258592389523983,
+            0.007509240880608559,
+            0.028175661340355873,
+            0.015231933444738388,
+            -0.015223395079374313,
+            -0.007944682613015175,
+            0.015991821885108948,
+            -0.007043915335088968,
+            -0.0005480379913933575,
+            -0.008836911991238594,
+            -0.014830644242465496,
+            -0.010066395625472069,
+            -0.0005312286084517837,
+            0.012243606150150299,
+            0.016179660335183144,
+            -0.0007257367251440883,
+            0.01390853151679039,
+            -0.009955400601029396,
+            -0.014420815743505955,
+            -0.004602024797350168,
+            0.005327761638909578,
+            -0.02252345345914364,
+            0.02105490304529667,
+            0.002292474266141653,
+            -0.009537034668028355,
+            0.0030865157023072243,
+            -0.004572141449898481,
+            0.012602205388247967,
+            0.009391887113451958,
+            -0.0038421356584876776,
+            -0.0036137420684099197,
+            0.00572051340714097,
+            -0.001715086749754846,
+            0.015001405030488968,
+            -0.011714245192706585,
+            0.011842316016554832,
+            -0.007432397920638323,
+            -0.00019931078713852912,
+            -0.009648029692471027,
+            -0.005238111596554518,
+            -0.010493299923837185,
+            0.0028794673271477222,
+            0.012943727895617485,
+            -0.014548887498676777,
+            0.012807118706405163,
+            -0.0008132520597428083,
+            -0.00949434470385313,
+            -0.015138015151023865,
+            -0.0055155991576612,
+            -0.011364183388650417,
+            -0.016683407127857208,
+            0.007069529499858618,
+            0.0007855033036321402,
+            0.019347287714481354,
+            0.0030054040253162384,
+            -0.004572141449898481,
+            -0.00017262929759453982,
+            0.018903307616710663,
+            -0.02580207586288452,
+            0.03403278440237045,
+            -0.00010045584349427372,
+            0.022711291909217834,
+            0.04251963645219803,
+            0.00954557303339243,
+            -0.006663971114903688,
+            -0.013754845596849918,
+            0.005801625084131956,
+            -0.013609698973596096,
+            0.009981014765799046,
+            0.014335434883832932,
+            -0.03235932067036629,
+            -0.003944592550396919,
+            -0.013976835645735264,
+            0.011586173437535763,
+            -0.007957490161061287,
+            0.0032850259449332952,
+            -0.010587218217551708,
+            0.011765473522245884,
+            0.01679440215229988,
+            -0.009075978770852089,
+            -0.004311730153858662,
+            -0.027492616325616837,
+            0.00032524747075513005,
+            0.016674868762493134,
+            -0.010911665856838226,
+            -0.01684563048183918,
+            0.010945817455649376,
+            -0.007218946237117052,
+            -0.015453923493623734,
+            -0.006898768246173859,
+            -0.03278622403740883,
+            -0.00020504731219261885,
+            -0.01076651830226183,
+            -0.003927516285330057,
+            -0.0028794673271477222,
+            0.0011024795239791274,
+            -0.0033917520195245743,
+            -0.0067066615447402,
+            -0.0025891726836562157,
+            0.0067664277739822865,
+            0.005011852364987135,
+            0.002080089645460248,
+            0.0005987327895127237,
+            -0.00023506399884354323,
+            0.009255277924239635,
+            0.010168852284550667,
+            -0.007419590838253498,
+            -0.017212767153978348,
+            0.010834822431206703,
+            -0.020730456337332726,
+            0.005938234273344278,
+            0.00783795677125454,
+            0.011039736680686474,
+            -0.0220111683011055,
+            -0.010868974961340427,
+            0.009818791411817074,
+            0.003835732117295265,
+            0.01902284100651741,
+            0.010886050760746002,
+            0.0022455148864537477,
+            -0.0035646480973809958,
+            0.0028495839796960354,
+            -1.0947752343781758e-05,
+            -0.0012508286163210869,
+            0.015308775939047337,
+            0.0006504949415102601,
+            0.02974666841328144,
+            -0.012055767700076103,
+            0.011201960034668446,
+            -0.009340658783912659,
+            0.0013842361513525248,
+            -0.009741948917508125,
+            -0.001973363570868969,
+            -0.004194331355392933,
+            -0.006484671495854855,
+            0.013148642145097256,
+            0.007714154664427042,
+            -0.004239156376570463,
+            -0.01416467409580946,
+            0.01194477267563343,
+            -0.023035738617181778,
+            -0.012269220314919949,
+            -0.00881983619183302,
+            0.012935190461575985,
+            -0.0053960662335157394,
+            0.01170570682734251,
+            0.03457922115921974,
+            -0.0018559650052338839,
+            0.0007860369514673948,
+            -0.013746308162808418,
+            -0.008311820216476917,
+            -0.02728770114481449,
+            -0.01510386262089014,
+            -0.008922292850911617,
+            -0.03454506769776344,
+            -0.011919158510863781,
+            0.008239246904850006,
+            0.008111175149679184,
+            0.0014237246941775084,
+            -0.019176525995135307,
+            -0.008888140320777893,
+            -0.01492456253618002,
+            -0.017460372298955917,
+            -0.00924673955887556,
+            -0.01175693515688181,
+            -0.0037119300104677677,
+            0.020884141325950623,
+            -0.004162313882261515,
+            0.014267130754888058,
+            0.016939548775553703,
+            1.3132299500284716e-05,
+            0.015334390103816986,
+            0.009665105491876602,
+            -0.009127207100391388,
+            -0.004725826904177666,
+            0.006885961163789034,
+            0.0027706066612154245,
+            0.018749622628092766,
+            -0.006702392362058163,
+            -0.014796491712331772,
+            0.004298923071473837,
+            0.005571096669882536,
+            -0.012346062809228897,
+            0.010655523277819157,
+            -0.00476424815133214,
+            0.02054261974990368,
+            -0.018254414200782776,
+            -0.011885006912052631,
+            0.014472044073045254,
+            -0.008862526156008244,
+            0.02394077368080616,
+            0.013951221480965614,
+            -0.015291700139641762,
+            0.007167717441916466,
+            0.013029108755290508,
+            0.007615966722369194,
+            0.0018495614640414715,
+            0.022147778421640396,
+            -0.0011846584966406226,
+            0.0031590894795954227,
+            0.010237157344818115,
+            -0.018835002556443214,
+            -0.02042308636009693,
+            -0.003118533408269286,
+            0.02247222512960434,
+            0.007961759343743324,
+            -0.016222350299358368,
+            0.0037930416874587536,
+            0.012849809601902962,
+            0.012072844430804253,
+            0.0039787450805306435,
+            -0.001204936415888369,
+            -0.02248930186033249,
+            0.0035155541263520718,
+            0.0001910395221784711,
+            0.018664240837097168,
+            -0.011116579174995422,
+            0.001461078878492117,
+            -0.011389797553420067,
+            0.012747352942824364,
+            -0.005221035331487656,
+            0.01193623524159193,
+            0.0011761204805225134,
+            -0.005353375803679228,
+            0.0001780989987310022,
+            0.014941639266908169,
+            0.014070754870772362,
+            0.037738312035799026,
+            -0.005916888825595379,
+            -0.006480402313172817,
+            0.017170077189803123,
+            -0.012388752773404121,
+            0.018715469166636467,
+            0.006497478578239679,
+            0.0170249305665493,
+            -0.011090965010225773,
+            0.0011344973463565111,
+            0.04115354269742966,
+            0.015949131920933723,
+            -6.583659705938771e-05,
+            -0.008760069496929646,
+            -0.00805140845477581,
+            0.0244530588388443,
+            0.009067440405488014,
+            -2.7882166250492446e-05,
+            0.015744218602776527,
+            0.006096188444644213,
+            -0.016324806958436966,
+            -0.012918113730847836,
+            0.03555256128311157,
+            -0.007718423847109079,
+            -0.022045321762561798,
+            -0.0009311842732131481,
+            -0.009383349679410458,
+            -0.02884163148701191,
+            0.013959759846329689,
+            0.013788998126983643,
+            -0.008162403479218483,
+            0.006992687005549669,
+            0.008439891040325165,
+            -0.004117488861083984,
+            -0.004435532260686159,
+            0.0005346972029656172,
+            -0.0007049251580610871,
+            0.006177300587296486,
+            0.005092964507639408,
+            -0.014463506639003754,
+            -0.006787773221731186,
+            -0.005669284611940384,
+            -0.004064125940203667,
+            0.009016212075948715,
+            0.005058811977505684,
+            0.004870974458754063,
+            -0.008482581935822964,
+            0.018339794129133224,
+            0.02298451028764248,
+            -0.0122009152546525,
+            0.012115534394979477,
+            0.005233842879533768,
+            -0.0146428057923913,
+            -0.004614831879734993,
+            -0.01121903583407402,
+            -0.026570502668619156,
+            -0.006113264709711075,
+            0.0033490615896880627,
+            0.015428309328854084,
+            -0.022796671837568283,
+            0.0008500725380145013,
+            0.007790997624397278,
+            2.7365211280994117e-05,
+            -0.008879602886736393,
+            0.007573276292532682,
+            -0.00044157879892736673,
+            0.022301463410258293,
+            0.019808344542980194,
+            0.025631314143538475,
+            -0.008414276875555515,
+            -0.00499904528260231,
+            0.014301283285021782,
+            0.00433947890996933,
+            0.009374811314046383,
+            0.027543844655156136,
+            0.022164853289723396,
+            -0.013857303187251091,
+            0.006429173983633518,
+            0.000640355923678726,
+            -0.014250054024159908,
+            0.0066682398319244385,
+            0.013054722920060158,
+            0.012414366938173771,
+            0.0028410458471626043,
+            -0.0013415457215160131,
+            -0.015957670286297798,
+            0.004824014846235514,
+            0.018408099189400673,
+            -0.017118848860263824,
+            -0.03471583127975464,
+            -0.011842316016554832,
+            -0.012158225290477276,
+            -0.00624133599922061,
+            -0.012875423766672611,
+            0.01440373994410038,
+            0.004418455995619297,
+            0.019705887883901596,
+            0.0002309283590875566,
+            -0.01022861897945404,
+            0.014250054024159908,
+            0.005110040307044983,
+            -0.0016638581873849034,
+            0.0005352308508008718,
+            0.007231753319501877,
+            -0.0030331527814269066,
+            -0.0022476494777947664,
+            -0.00027055040118284523,
+            0.016239427030086517,
+            -0.02532394416630268,
+            0.008947907015681267,
+            -0.017067620530724525,
+            0.0017951311310753226,
+            0.015249010175466537,
+            -0.003775965655222535,
+            -0.016025975346565247,
+            0.0011216901475563645,
+            -0.008619191125035286,
+            -0.07288104295730591,
+            -0.005882736761122942,
+            0.004512374754995108,
+            0.023992003872990608,
+            -0.013336479663848877,
+            -0.0024824466090649366,
+            0.0018474268727004528,
+            0.008657612837851048,
+            -0.010134699754416943,
+            0.003995821345597506,
+            0.010390842333436012,
+            -0.003319178242236376,
+            -0.009221125394105911,
+            0.002508060773834586,
+            -0.008106905966997147,
+            -0.0038805571384727955,
+            -0.00783795677125454,
+            0.003402424743399024,
+            0.025221485644578934,
+            -0.010868974961340427,
+            0.024657974019646645,
+            0.009699258022010326,
+            -0.0075433929450809956,
+            -0.0027428579051047564,
+            -0.007761114276945591,
+            -0.027543844655156136,
+            -0.012397291138768196,
+            -0.003396020969375968,
+            -0.01292665209621191,
+            0.0029264267068356276,
+            0.001692674239166081,
+            -0.007167717441916466,
+            -0.004348016809672117,
+            0.0021547977812588215,
+            0.00030336863710545003,
+            -0.022147778421640396,
+            -0.0009407896432094276,
+            -0.014267130754888058,
+            0.02105490304529667,
+            -0.012371676973998547,
+            -0.004461146425455809,
+            0.0030544979963451624,
+            -0.010109085589647293,
+            0.004580679349601269,
+            0.019313136115670204,
+            -0.007308595813810825,
+            -0.009682182222604752,
+            0.014045140706002712,
+            -0.003690584795549512,
+            0.00185916677583009,
+            -0.01798119582235813,
+            0.010032243095338345,
+            -0.03232516720890999,
+            -0.003583858720958233,
+            -0.009981014765799046,
+            -0.0031847036443650723,
+            -0.0067151994444429874,
+            -0.03162504732608795,
+            0.015138015151023865,
+            -0.007039646618068218,
+            -0.02346264198422432,
+            -0.0011793222511187196,
+            -0.01859593763947487,
+            0.01419028826057911,
+            -0.01487333420664072,
+            0.01393414568156004,
+            -0.020645076408982277,
+            0.012533900327980518,
+            0.012798581272363663,
+            0.021123208105564117,
+            0.006493209395557642,
+            -0.005302147474139929,
+            0.002676687901839614,
+            0.0018730411538854241,
+            0.014352511614561081,
+            -0.008905217051506042,
+            0.00462336977943778,
+            -0.002390662208199501,
+            0.005737589206546545,
+            0.009426039643585682,
+            0.02107197977602482,
+            -0.0005896611255593598,
+            -0.0037823691964149475,
+            -0.017468910664319992,
+            0.00757754547521472,
+            -0.01559907104820013,
+            0.005485715810209513,
+            0.0025251370389014482,
+            0.023787088692188263,
+            0.010886050760746002,
+            0.0035859933122992516,
+            0.004900857340544462,
+            0.004520913120359182,
+            0.01861301250755787,
+            -0.006890229880809784,
+            -0.004747172351926565,
+            -0.018186109140515327,
+            0.019688811153173447,
+            0.007705616764724255,
+            -0.005306416191160679,
+            -0.011902082711458206,
+            0.006010808050632477,
+            0.002134519862011075,
+            0.0017577770631760359,
+            0.02288205362856388,
+            -0.02974666841328144,
+            -0.008964983746409416,
+            -0.007808073423802853,
+            -0.009682182222604752,
+            -0.017844585701823235,
+            0.017084697261452675,
+            -0.008012987673282623,
+            0.00461910106241703,
+            -0.012790042906999588,
+            0.033418044447898865,
+            0.002969117136672139,
+            0.00513992365449667,
+            0.013729231432080269,
+            -0.00024373548512812704,
+            0.017844585701823235,
+            -0.008367317728698254,
+            -0.04036803916096687,
+            -0.0031911071855574846,
+            0.0034515184815973043,
+            0.007645850069820881,
+            -0.010903127491474152,
+            0.004862436093389988,
+            -0.006339523941278458,
+            -0.01854470930993557,
+            -0.014232978224754333,
+            0.037840768694877625,
+            -0.005225304514169693,
+            0.017947042360901833,
+            -0.016145508736371994,
+            -0.0033682722132653,
+            0.007107951212674379,
+            0.022762520238757133,
+            -0.012602205388247967,
+            0.00791053008288145,
+            0.009571187198162079,
+            0.02199409157037735,
+            0.00978463888168335,
+            -0.01565883681178093,
+            0.023735860362648964,
+            0.005741858389228582,
+            -0.0023565099108964205,
+            0.005276532843708992,
+            -0.0030352871399372816,
+            0.016000360250473022,
+            -0.014813567511737347,
+            -0.011876468546688557,
+            0.002303146990016103,
+            -0.010168852284550667,
+            0.0037055264692753553,
+            -0.024589668959379196,
+            0.00037540867924690247,
+            -0.009844405576586723,
+            0.021089056506752968,
+            0.01027130987495184,
+            0.00877714529633522,
+            -0.019637582823634148,
+            -0.0048026698641479015,
+            -0.02645096927881241,
+            -0.014574501663446426,
+            -0.01022861897945404,
+            -0.0011451698374003172,
+            -0.004431263078004122,
+            -0.007765382993966341,
+            -0.03133475035429001,
+            -0.004202869720757008,
+            0.013541393913328648,
+            0.014975790865719318,
+            -0.0013714289525523782,
+            0.012089920230209827,
+            -0.00019544197130016983,
+            -0.008106905966997147,
+            -0.02236976847052574,
+            -0.010407919064164162,
+            -0.02091829478740692,
+            -0.03123229369521141,
+            -0.022199006751179695,
+            0.008670419454574585,
+            -0.012021616101264954,
+            -0.002941368380561471,
+            0.028961164876818657,
+            -0.022130701690912247,
+            0.007513510063290596,
+            0.005400335416197777,
+            0.0028666602447628975,
+            0.014856258407235146,
+            0.0054472945630550385,
+            0.0003634020104072988,
+            0.0037652929313480854,
+            -0.008341703563928604,
+            -0.008081291802227497,
+            0.01799827069044113,
+            -0.003331985557451844,
+            0.005097233224660158,
+            0.010809208266437054,
+            -0.006749351508915424,
+            -0.005438756663352251,
+            0.02392369881272316,
+            -0.015718603506684303,
+            0.002044870052486658,
+            -0.008610652759671211,
+            -0.014079293236136436,
+            -0.00949434470385313,
+            -0.005793086718767881,
+            -0.01813488081097603,
+            -0.015240471810102463,
+            0.01689685881137848,
+            -0.0004028906114399433,
+            0.016025975346565247,
+            -0.005289340391755104,
+            -0.005007583647966385,
+            0.014437892474234104,
+            -0.03558671474456787,
+            -0.003402424743399024,
+            0.006377945188432932,
+            -0.026211904361844063,
+            0.02484581060707569,
+            0.005276532843708992,
+            0.01605158858001232,
+            -0.013780459761619568,
+            0.0026318628806620836,
+            -0.006365138106048107,
+            0.004358689300715923,
+            0.018903307616710663,
+            0.007308595813810825,
+            0.004858167376369238,
+            -0.006689585279673338,
+            -0.008047140203416348,
+            0.016077203676104546,
+            -0.007415322121232748,
+            0.0018719738582149148,
+            0.016785863786935806,
+            0.018425175920128822,
+            0.012004539370536804,
+            0.0059553105384111404,
+            0.013071799650788307,
+            0.01750306226313114,
+            -0.007043915335088968,
+            -0.005835777148604393,
+            -0.013481627218425274,
+            -0.00802579429000616,
+            -0.0013105951948091388,
+            0.03859211876988411,
+            0.018920384347438812,
+            0.007385438773781061,
+            -0.02054261974990368,
+            0.013456013053655624,
+            0.0043928418308496475,
+            6.841302820248529e-06,
+            -0.0027385889552533627,
+            0.02293328195810318,
+            0.0024952536914497614,
+            0.007125027012079954,
+            0.006258412264287472,
+            -0.017212767153978348,
+            0.03309359401464462,
+            -0.020713381469249725,
+            0.0008532743086107075,
+            -0.013319403864443302,
+            -0.012166762724518776,
+            0.015530765987932682,
+            0.002104636514559388,
+            -0.004164448007941246,
+            -0.006296833511441946,
+            -0.0033575997222214937,
+            0.013515779748558998,
+            0.008499657735228539,
+            -0.012875423766672611,
+            0.004516643937677145,
+            -0.02542640082538128,
+            0.004315999336540699,
+            -0.015334390103816986,
+            0.021755026653409004,
+            -0.004559334367513657,
+            -0.004674598574638367,
+            -0.019261907786130905,
+            -0.017067620530724525,
+            -0.00999809056520462,
+            0.002403469290584326,
+            -0.004755710251629353,
+            0.004422725178301334,
+            -0.009468729607760906,
+            -0.008499657735228539,
+            0.0027599341701716185,
+            0.0036628360394388437,
+            0.012465596199035645,
+            0.013669464737176895,
+            0.019893724471330643,
+            0.017759205773472786,
+            -0.01536000519990921,
+            -0.0341864712536335,
+            0.0006056699785403907,
+            0.005084426142275333,
+            -0.0036436254158616066,
+            0.018664240837097168,
+            -0.001938143977895379,
+            -0.0009968207450583577,
+            -0.0007150641758926213,
+            0.0005419012159109116,
+            0.02004740945994854,
+            -0.011014122515916824,
+            -0.016657792031764984,
+            0.00021545309573411942,
+            0.008853988721966743,
+            0.00502039073035121,
+            -0.0025934416335076094,
+            -0.010296924039721489,
+            -0.0009455922991037369,
+            0.0020341973286122084,
+            0.006032153032720089,
+            -0.009101592935621738,
+            -0.0010496501345187426,
+            -0.0013671600027009845,
+            -0.0030544979963451624,
+            -0.006254143081605434,
+            -0.009844405576586723,
+            0.008222170174121857,
+            -0.012602205388247967,
+            -0.01123611256480217,
+            0.016222350299358368,
+            -0.015086785890161991,
+            0.001136631821282208,
+            0.023821242153644562,
+            0.022557605057954788,
+            -0.0008842248353175819,
+            0.006774966139346361,
+            -0.007936145178973675,
+            -0.011517869308590889,
+            -0.02785121463239193,
+            -0.019637582823634148,
+            -0.00806848518550396,
+            0.017947042360901833,
+            -0.002053408185020089,
+            -0.004049184266477823,
+            -0.0018869155319407582,
+            -0.008580769412219524,
+            0.008444160223007202,
+            -0.013814612291753292,
+            0.011620325967669487,
+            -0.034425534307956696,
+            0.013088875450193882,
+            -0.006702392362058163,
+            0.0010485828388482332,
+            0.015009943395853043,
+            0.04351005330681801,
+            -0.007351286243647337,
+            0.0029541754629462957,
+            0.001540056080557406,
+            0.01389145478606224,
+            0.028209814801812172,
+            0.004369362257421017,
+            0.0006136744632385671,
+            -0.01765674725174904,
+            0.004091874696314335,
+            0.002264725510030985,
+            -0.013456013053655624,
+            0.009579724632203579,
+            -0.023035738617181778,
+            -0.023155272006988525,
+            -0.009067440405488014,
+            -0.021123208105564117,
+            -0.002608383307233453,
+            0.002011785050854087,
+            0.012883962132036686,
+            -0.009878558106720448,
+            -0.012098458595573902,
+            -0.0005656477296724916,
+            -0.032991137355566025,
+            0.016709022223949432,
+            0.09665105491876602,
+            0.00875153113156557,
+            -0.003908305894583464,
+            -0.0021515958942472935,
+            0.002499522641301155,
+            0.009562648832798004,
+            -0.004768517334014177,
+            0.055360905826091766,
+            -0.018852079287171364,
+            0.008311820216476917,
+            -0.0008105839369818568,
+            0.008064216002821922,
+            -0.007223214954137802,
+            0.015530765987932682,
+            -0.012209453620016575,
+            -0.005951041355729103,
+            -0.009460192173719406,
+            -0.01123611256480217,
+            -0.0017246920615434647,
+            -0.016640717163681984,
+            -0.004986238200217485,
+            -0.020645076408982277,
+            0.013336479663848877,
+            -0.03234224393963814,
+            -0.011509330943226814,
+            -0.012474133633077145,
+            -0.0063224476762115955,
+            0.006066305562853813,
+            -0.015684451907873154,
+            -0.002595576224848628,
+            -0.002099300269037485,
+            0.015402695164084435,
+            0.0013863706262782216,
+            -0.00232235761359334,
+            -0.008328896015882492,
+            0.0005405671545304358,
+            0.005024659913033247,
+            0.003940323833376169,
+            0.006557244807481766,
+            0.006792041938751936,
+            -0.005848584230989218,
+            -0.008811297826468945,
+            0.010655523277819157,
+            -0.002510195365175605,
+            -0.0027919518761336803,
+            -0.008871064521372318,
+            0.010382304899394512,
+            -0.013481627218425274,
+            0.018356870859861374,
+            -0.013840226456522942,
+            -0.029507601633667946,
+            -0.015547842718660831,
+            0.02100367471575737,
+            -0.01996202953159809,
+            -0.0039381892420351505,
+            0.010553065687417984,
+            0.02829519473016262,
+            0.004113219678401947,
+            -0.001529383473098278,
+            -0.004909395705908537,
+            -0.017204230651259422,
+            0.009511420503258705,
+            0.008196556009352207,
+            0.018442250788211823,
+            -0.012371676973998547,
+            0.016487032175064087,
+            -0.018288565799593925,
+            0.006177300587296486,
+            -0.01663217879831791,
+            0.0006083381595090032,
+            0.00011866596469189972,
+            -0.0041793896816670895,
+            0.01608574204146862,
+            -0.01534292846918106,
+            -0.026263132691383362,
+            0.01900576427578926,
+            -0.013635313138365746,
+            -0.03307652100920677,
+            0.003374675754457712,
+            -0.034937821328639984,
+            -0.0025827689096331596,
+            0.009716334752738476,
+            -0.02149888314306736,
+            -0.018305642530322075,
+            0.007487895432859659,
+            -0.010066395625472069,
+            -0.008871064521372318,
+            0.00502039073035121,
+            -0.01689685881137848,
+            -0.012149686925113201,
+            -0.001152640674263239,
+            0.006753620691597462,
+            -0.00499477656558156,
+            0.0057632033713161945,
+            -0.026075294241309166,
+            0.014010988175868988,
+            -0.0170249305665493,
+            0.01144102681428194,
+            -0.010587218217551708,
+            0.022762520238757133,
+            0.011748396791517735,
+            -0.010356690734624863,
+            -0.0008303282083943486,
+            0.006036422215402126,
+            0.0005264259525574744,
+            0.026126522570848465,
+            0.03237639740109444,
+            -0.0011889275629073381,
+            0.008469774387776852,
+            0.007586083374917507,
+            0.003921112976968288,
+            -0.011014122515916824,
+            0.0015923518221825361,
+            0.014830644242465496,
+            0.01992787793278694,
+            -0.013729231432080269,
+            0.008879602886736393,
+            -0.026672959327697754,
+            0.02103782817721367,
+            -0.022096550092101097,
+            -0.000982412719167769,
+            0.010066395625472069,
+            -0.004777055233716965,
+            0.007090874947607517,
+            0.0066682398319244385,
+            0.021362274885177612,
+            0.03375956416130066,
+            0.00807702261954546,
+            0.008879602886736393,
+            0.007257367484271526,
+            -0.004798400681465864,
+            0.002044870052486658,
+            0.009135744534432888,
+            -0.005869929678738117,
+            -0.0040940092876553535,
+            -0.010083471424877644,
+            -0.006685316096991301,
+            0.016205275431275368,
+            0.0008079157560132444,
+            0.012602205388247967,
+            0.007018301170319319,
+            -0.007428129203617573,
+            -0.005669284611940384,
+            0.007014031987637281,
+            -0.023787088692188263,
+            -0.00950288213789463,
+            0.025221485644578934,
+            -0.013814612291753292,
+            -0.043202683329582214,
+            -0.0024333526380360126,
+            0.008853988721966743,
+            -0.017964119091629982,
+            -0.01245705783367157,
+            -0.004772786516696215,
+            0.010117623955011368,
+            0.0007241358398459852,
+            -0.007569007575511932,
+            0.028175661340355873,
+            0.00502039073035121,
+            -0.005169807001948357,
+            0.003571051638573408,
+            -2.8232359909452498e-05,
+            0.019244831055402756,
+            0.01536000519990921,
+            0.008294744417071342,
+            -0.03266669064760208,
+            -0.007304327096790075,
+            -0.005780279636383057,
+            0.013114489614963531,
+            0.002813297091051936,
+            0.003423769725486636,
+            -0.019740039482712746,
+            -0.020798761397600174,
+            0.019210679456591606,
+            0.004935009870678186,
+            0.019825419411063194,
+            -0.00867895781993866,
+            -0.006548706907778978,
+            -0.004841091111302376,
+            -0.004742903169244528,
+            0.00499477656558156,
+            0.0076415808871388435,
+            0.0038933642208576202,
+            0.03377664089202881,
+            -0.001280711847357452,
+            0.004042780492454767,
+            -0.0036350872833281755,
+            -0.0002441357064526528,
+            0.014787953346967697,
+            -0.00790199264883995,
+            0.008905217051506042,
+            0.018800850957632065,
+            -0.02887578494846821,
+            0.011398335918784142,
+            -0.01679440215229988,
+            -0.00828193686902523,
+            0.016000360250473022,
+            -0.0005320290802046657,
+            -0.004922202788293362,
+            -0.014583039097487926,
+            0.0008127184119075537,
+            0.007752575911581516,
+            0.01996202953159809,
+            -0.004091874696314335,
+            0.011287340894341469,
+            -0.009648029692471027,
+            -0.004305326379835606,
+            0.010083471424877644,
+            0.006087650544941425,
+            -0.0033213128335773945,
+            0.006232798099517822
+          ],
+          "keyphrases": [
+            "MoE models",
+            "expert choice",
+            "training efficiency",
+            "fine-tuning on GLUE and SuperGLUE",
+            "capacity factor"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "f40423d6-544f-4b7f-8e6f-e18fde5a5649",
+      "type": "child",
+      "source": {
+        "id": "b570d71e-4f83-486e-b70f-b0c6217db455",
+        "properties": {
+          "page_content": "Mixture-Of-Experts With Expert Choice Routing\n\nYanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon Google, Mountain View, CA, USA {yanqiz, taole, hanxiaol, dunan, huangyp, vzhao, adai, zhifengc, qvl, jlaudon}@google.com\n\nAbstract\n\nSparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choice method. Instead of letting tokens select the top-k experts, we have experts selecting the top-k tokens. As a result, each token can be routed to a variable number of experts and each expert can have a fixed bucket size. We systematically study pre-training speedups using the same computational resources of the Switch Transformer top-1 and GShard top-2 gating of prior work and find that our method improves training convergence time by more than 2×. For the same computational cost, our method demonstrates higher performance in fine-tuning 11 selected tasks in the GLUE and SuperGLUE benchmarks. For a smaller activation cost, our method outperforms the T5 dense model in 7 out of the 11 tasks.\n\n1 Introduction\n\nScaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27]. The final model quality has been found to have a power-law relationship with the amount of data, model size, and compute time [16, 20]. However, training efficiency, which is defined as the total amount of computation used to achieve superior model quality than the state of the art system [21], should receive greater attention as we increase our efforts towards green AI [29].\n\nSparsely gated mixture-of-experts [31] (MoE) provides an effective way to scale model capacity given a fixed computational cost, and has recently played an important role in increasing the training efficiency of large-scale language models [10, 21]. MoE operate by adopting a number of experts, each as a sub-network, and by activating only one or a few experts for each input token. A gating network must be chosen and optimized in order to route each token to the most suited expert(s). For example, recent work has implemented sparse routing via k-means clustering [12], linear assignment to maximize token-expert affinities [22], or hashing [8, 28]. Many of the prior work use a routing strategy concerning the token choice, where each token selects the best one or two experts.\n\nWe argue that the independent token choice of prior work often leads to an imbalanced load of experts, which causes training inefficiency and sub-optimal training of the model. In order to mitigate this\n\nissue, previous sparsely gated networks introduce additional auxiliary losses as regularization to prevent too many tokens being routed to a single expert, but the effectiveness is still limited. Recent approaches [8, 22, 28] explore alternative strategies for routing, but they focus on pre-training only and do not demonstrate performance gain on downstream tasks. Moreover, none of the previous methods consider allocating a variable number of experts to each token based on importance, which can be beneficial.\n\nWe propose a very simple yet effective routing method we are calling expert choice. Unlike conventional MoE where tokens select one or two top-scoring experts, our method lets each expert pick the top-k tokens. Our method guarantees perfect load balancing, allows a variable number of experts for each token, and achieves substantial gains in training efficiency and downstream performance as demonstrated in our experiments. Our major contributions include:\n\nWe identify common pitfalls in conventional MoE such as load imbalance as described in Section 3.1. We then propose a heterogeneous, expert choice method to provide a fluid allocation of model parameters based on a learnt token-to-expert importance. This method intrinsically guarantees load balance without imposing an auxiliary loss.\n\nWe show our method provides over 2× faster training convergence in a 8B/64E (8 billion activated parameters, 64 experts) model, compared to the top-1 and top-2 gating counterparts in Switch Transformer [10] and GShard [21].\n\nWe show our method demonstrates strong scaling when increasing the number of experts from 16 to 128, evaluated in training perplexity.\n\nWe show our method demonstrates strong performance on downstream tasks selected from GLUE and SuperGLUE at all the evaluated scales. More specifically, our 8B/64E model outperforms a T5 11B dense model in 7 out of 11 tasks evaluated.\n\n2 Related Work\n\nScaling: Various approaches have been proposed to scale up neural network capacity to improve performance. Recent works have successfully scaled models to billions of parameters via various forms of model parallelism [2, 21, 26, 27, 33]. Model parallelism [30] splits weights and tensors across multiple cores while pipeline parallelism [18, 24] splits different layers across devices with micro-batches pipelined to the different layers. To enable continued scaling of neural networks, improving model training and serving efficiency has become a critical research area.\n\nConditional Computation: Computation decisions can be made dynamically based on the input [23, 25]. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation, by activating certain parameters and computation on demand, on a per-example or per-token basis [3]. Conditional convolution layers [1] with task-specific gating has been used to combat catastrophic forgetting when a sequence of learning problems are optimized. The gating decisions may be binary or sparse and continuous, stochastic or deterministic.\n\nMixture of Experts: Sparsely-gated MoE [31] is the first model to demonstrate massive improvements in model capacity, training time, or model quality with gating. Switch Transformer [10] simplifies the gating by selecting only the top expert per token using a softmax over the hidden state and demonstrates better scaling than previous work. All the prior work requires an auxiliary loss to explicitly encourage balancing. This loss term has to be carefully weighted to not overwhelm the primary loss. However, auxiliary loss does not guarantee balancing and a hard capacity factor has to be imposed. As a result, many tokens can still be unprocessed by the MoE layer. Hard MoE [12] with a single decoding layer can be efficiently trained to good effect on large scale hashtag prediction tasks.\n\nBase Layers [22] formulate a linear assignment that maximizes token-expert affinities while ensuring each expert receives an equal number of tokens. Hash layers [8, 28] devise hashing techniques on input tokens. However, the evaluations are limited to pre-training perplexity. THOR [? ] randomly activates experts during training and inference and is trained with a consistency regularization loss.\n\nTHOR has demonstrated strong performance on translation tasks. Different from these prior works, our method is a learnt method that enables heterogeneous MoE and effectively improves downstream fine-tuning performance.\n\n3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n5 Conclusion\n\nWe propose a new routing method for sparsely activated mixture-of-experts (MoE) models. This method addresses load imbalance and under-utilization of experts in conventional MoE methods, and enables selecting different numbers of experts for each token. Our model demonstrates more than 2x training efficiency improvements when compared to the state-of-the-art GShard and Switch Transformer models, and also achieves strong gains when finetuning on 11 datasets in the GLUE and SuperGLUE benchmark.\n\n6 Limitations\n\nThe expert choice method might not immediately apply to auto-regressive text generation as our current implementation takes in the past and future tokens to perform the top-k selection. One possible solution is to collect a large batch of input sequences, dispatch tokens of the same sequence into separate groups, and perform expert choice routing for each group. Another scenario where the expert choice method does not immediately apply is when the batch size becomes very small during serving or inference. A global top-k can be selected instead and we can cap the number of times each expert or token gets selected. We leave these possible improvements for future work.\n\nAnother long-standing issue with MoE has been the large memory footprint. Even though computational cost can be reduced using sparsely gated networks, the total number of parameters increases linearly or sub-linearly with the number of experts. Increasing the number of experts requires reservation of a large number of hardware devices. Therefore, dynamic (used) power is saved while static (reserved) power is not. Power saving techniques such as the ability to put hardware devices into low power states while not in use [17] can help with reducing the reserved power requirements.\n\nReferences\n\n[1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pages 3930–3939. Computer Vision Foundation / IEEE, 2020.\n\n[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. In Advances in Neural Information Processing Systems.\n\n[3] Kyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning, 2014.\n\n[4] Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems, 2021.\n\n[5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.\n\nTransformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.\n\n[6] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 933–941. JMLR.org, 2017.\n\n[7] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixtureof-experts, 2021.\n\n[8] Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan.\n\nTricks for training sparse translation models, 2021.\n\n[9] Richard L Dykstra. An iterative procedure for obtaining i-projections onto the intersection of convex sets. The annals of Probability, pages 975–984, 1985.\n\n[10] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.\n\n[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. NAS-FPN: learning scalable feature pyramid architecture for object detection. In CVPR, pages 7036–7045. Computer Vision Foundation / IEEE, 2019.\n\n[12] Sam Gross, Marc'Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision, 2017.\n\n[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.\n\n[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 630–645, Cham, 2016. Springer International Publishing.\n\n[15] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2016.\n\n[16] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.\n\n[17] Ping Huang, Zuocheng Xing, Tianran Wang, Qiang Wei, Hongyan Wang, and Guitao Fu. A brief survey on power gating design. In 2010 10th IEEE International Conference on Solid-State and Integrated Circuit Technology, pages 788–790, 2010.\n\n[18] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 103–112, 2019.\n\n[19] Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David A. Patterson. A domain-specific supercomputer for training deep neural networks. Commun. ACM, 63(7):67–78, 2020.\n\n[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.\n\n[21] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.\n\n[22] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6265–6274. PMLR, 18–24 Jul 2021.\n\n[24] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. New York, NY, USA, 2019. Association for Computing Machinery.\n\n[25] Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, Cédric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. In ICLR. OpenReview.net, 2021.\n\n[26] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.\n\n[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.\n\n[28] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models, 2021.\n\n[30] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 10435–10444, Red Hook, NY, USA, 2018. Curran Associates Inc.\n\n[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E.\n\nHinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In ICLR (Poster). OpenReview.net, 2017.\n\n[32] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 10–15 Jul 2018. [23] Min Lin, Jie Fu, and Yoshua Bengio. Conditional computation for continual learning, 2019.\n\n[29] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.\n\n[33] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.\n\n[34] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc.\n\n[35] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.\n\nGLUE: A multi-task benchmark and analysis platform for natural language understanding.\n\nIn Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, November 2018. Association for Computational Linguistics.\n\n[36] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake A. Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen. GSPMD: general and scalable parallelization for ML computation graphs. CoRR, abs/2105.04663, 2021.\n\n7 Checklist\n\n(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Yes (b) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes (c) Did you discuss any potential negative societal impacts of your work? N/A. Not any. (d) Did you describe the limitations of your work? Yes (a) Did you include the code, data, and instructions needed to reproduce the main experimental results? Yes. We include details in the experiment setup to help reproduce the main results. (b) Did you specify all the training details? Yes (c) Did you report error bars? Yes (d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? Yes (a) If your work uses existing assets, did you cite the creators? Yes (b) Did you mention the license of the assets? No. The used dataset is not released yet. (c) Did you include any new assets either in the supplemental material or as a URL? No. The dataset is not released yet.\n\n(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? No. Not using persons' data. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes. The dataset does not contain any personally identifiable information or offensive content.\n\nA Comparison On Fine-Tuning With A Dense Model\n\nOur 8B MoE model achieves stronger pre-training perplexity than its dense counterpart. However, a better perplexity does not always directly translate to downstream performance as demonstrated in Section 4.4. To this end, we compare fine-tuning performance of the 8B dense model and MoE model in Table 1. As shown in the table, our MoE model using expert choice routing consistently outperforms the dense model across the 11 tasks in GLUE and SuperGLUE.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 1: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nB Capacity Factor\n\nWe evaluate the downstream task fine-tuning performance by varying the capacity factors. Note that a capacity factor of n indicates on average how many experts each token can be received. EC-CF2 is our baseline expert choice, which matches GShard top-2 gating computational footprint. EC-CF1, however, matches Switch Transformer top-1 gating computational footprint. EC-CF0.5 further verifies that an aggressively lowered capacity factor can provide strong enough performance, that almost matches the top-2 gating baseline.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Top-2 78.1 87.0 88.3 85.0 82.6 90.1 90.7 81.6 94.7 68.2 67.2 83.0±0.3 EC-CAP2 78.2 88.0 88.5 85.7 83.0 90.8 91.1 80.0 95.4 70.4 64.1 83.2±0.4 EC-CAP3 78.5 91.7 89.3 86.3 83.5 90.9 91.1 81.8 94.9 70.0 65.6 84.0±0.4 EC-CF2 79.1 89.6 89.3 86.8 84.3 91.3 91.2 81.1 95.2 68.1 68.0 84.0±0.2 EC-CF1 77.4 90.6 88.0 85.5 83.6 90.3 91.2 79.8 95.3 66.5 64.9 83.0±0.2 EC-CF0.5 77.4 89.6 86.3 85.2 82.7 91.7 91.0 79.6 94.9 67.3 63.5 83.0 ±0.05 Hash Layers 76.1 85.2 86.7 83.4 82.5 90.0 90.3 75.7 94.0 67.4 63.3 81.3±1.0\n\nTable 2: Comparison between different routing methods in fine-tuning of 100M/64E models. We perform 3 independent fine-tuning runs for each method and report the average results. This gives more accurate difference between the variants of expert choice method, since they achieve close fine-tuning results. We do not report averaged results in other experiments.\n\nC Capped Expert Choice\n\nAs described in Section 4.5, the maximum number of experts each token is assigned can be capped by an entropy-regularized linear programming. Figure 1 compares the validation perplexity when training the 100M/64E models using the base expert choice method (EC-BASE), expert choice capped by two experts per token (EC-CAP2), expert choice capped by three experts per token (EC-CAP3), and GShard top-2 gating.\n\nAs shown in the figure, restricting the number of experts to 2 degrades the perplexity compared to the base expert choice method. This suggests that a more flexible allocation of experts (e.g. more than 2 experts for a token) can enhance model expressiveness. On the other hand, our EC-CAP2 and EC-CAP3 methods still outperform the top-2 gating method by a clear margin. We believe this confirms the effectiveness of a load balanced training, provided by our method. Finally, EC-CAP3 obtains comparable perplexity to EC-BASE. As indicated by Figure 3, only a little fraction of tokens use more than 3 experts therefore we see little or no difference between EC-BASE and EC-CAP3 variants. We present the fine-tuning results of these methods in Table 2.\n\nD Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [? ]. We use mod x to map a token ID to an expert ID. This in some way ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 2. Hashing based routing performs much worse than expert choice in terms of average scores and variance.\n\nE Fine-Tuning Details\n\nWe did a hyperparameter search for both baseline models and expert choice method. For fine-tuning of the 8B dense model, we use a constant learning rate of 0.0001 and a dropout rate of 0.1. We freeze the attention layer and feed-forward layer while leaving the embedding and layer normalization trainable. This setting has been found optimal for the 8B dense model. For MoE 8B/64E models including GShard top-2 gating and expert choice, we found continuing the learning rate from the pre-trained model while using a square root learning rate decay works better. In addition, we do not apply parameter freezing for fine-tuning MoE models. For models with 100M expert size, we use a constant learning rate of 0.0001 and no dropout is used.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/2202.09368v2.md"
+          },
+          "headlines": {
+            "1 Introduction": [],
+            "2 Related Work": [],
+            "3 Method": [
+              "3.1 Pitfalls Of Token-Choice Routing",
+              "3.2 Heterogeneous Moe Via Expert Choice",
+              "3.3 Expert Choice With Additional Constraint",
+              "3.4 Model Architecture"
+            ],
+            "4 Experiments": [
+              "4.1 Setup",
+              "4.2 Training Efficiency",
+              "4.3 Scaling The Number Of Experts",
+              "4.4 Fine-Tuning On Glue And Superglue",
+              "4.5 Heterogeneity Matters",
+              "4.6 Comparison With Hash Layer",
+              "4.7 Ablation"
+            ],
+            "5 Conclusion": [],
+            "6 Limitations": [],
+            "A Comparison On Fine-Tuning With A Dense Model": [],
+            "B Capacity Factor": [],
+            "C Capped Expert Choice": [],
+            "D Comparison With Hash Layer": [],
+            "E Fine-Tuning Details": []
+          },
+          "summary": "The paper introduces a new routing method for sparsely-activated Mixture-of-Experts (MoE) models called 'expert choice,' which addresses load imbalance and under-utilization issues in conventional MoE methods. Unlike traditional token-choice routing, expert choice allows experts to select the top-k tokens, ensuring perfect load balancing and enabling variable expert allocation per token. This method improves training efficiency, achieving over 2x faster convergence compared to existing models like Switch Transformer and GShard. It also demonstrates superior performance on downstream tasks in the GLUE and SuperGLUE benchmarks. The study highlights the benefits of flexible expert allocation and shows that expert choice outperforms hashing-based routing methods. The paper also discusses potential limitations and future improvements for the expert choice method.",
+          "summary_embedding": [
+            -0.009726986289024353,
+            -0.0337262786924839,
+            -0.023302342742681503,
+            -0.019878201186656952,
+            -0.025953780859708786,
+            0.04560471698641777,
+            -0.006874797400087118,
+            0.039665497839450836,
+            -0.006973279174417257,
+            0.034544438123703,
+            -0.025787118822336197,
+            -0.00363057479262352,
+            -0.01052999310195446,
+            0.0054657477885484695,
+            -0.024499278515577316,
+            0.036786794662475586,
+            0.010007280856370926,
+            0.003272630739957094,
+            0.006102093029767275,
+            -0.04048365354537964,
+            -0.018666116520762444,
+            -0.013522328808903694,
+            -0.03230207785964012,
+            0.008969432674348354,
+            0.02484775334596634,
+            0.0016713523073121905,
+            0.0026230288203805685,
+            0.003624893259257078,
+            -0.05311964824795723,
+            0.04339266195893288,
+            0.018681267276406288,
+            0.023711422458291054,
+            -0.016332851722836494,
+            -0.012832955457270145,
+            -0.039120059460401535,
+            -0.0197418425232172,
+            0.030786970630288124,
+            0.039968520402908325,
+            -0.019287310540676117,
+            0.03357476741075516,
+            0.018060073256492615,
+            0.02454473078250885,
+            -0.021272100508213043,
+            -0.029817303642630577,
+            -0.0061892117373645306,
+            -0.007840678095817566,
+            -0.008355814032256603,
+            0.01242387667298317,
+            0.016984347254037857,
+            -0.013007191941142082,
+            0.0212569497525692,
+            -0.01371171697974205,
+            0.03384748846292496,
+            0.008916404098272324,
+            -0.042332086712121964,
+            0.03984731063246727,
+            -0.008007340133190155,
+            0.012105703353881836,
+            0.03281721472740173,
+            0.010333028621971607,
+            -0.005287723150104284,
+            -0.021544819697737694,
+            -0.03378688171505928,
+            0.0010529992869123816,
+            0.00014014738553669304,
+            0.005950582213699818,
+            -0.003069985192269087,
+            -0.009742137044668198,
+            0.03227177634835243,
+            -0.003939277958124876,
+            -0.0740584209561348,
+            0.050089433789253235,
+            -0.026590125635266304,
+            -0.007628562860190868,
+            -0.028938541188836098,
+            -0.01736312545835972,
+            0.037362534552812576,
+            0.005662711802870035,
+            0.0098709212616086,
+            0.0106284748762846,
+            -0.0007892759749665856,
+            0.028923390433192253,
+            0.00910579226911068,
+            -0.024923507124185562,
+            0.04478655755519867,
+            -0.04790768027305603,
+            0.01801462098956108,
+            -0.04336235672235489,
+            0.022756904363632202,
+            1.6393929399782792e-05,
+            -0.020726662129163742,
+            -0.0013304532039910555,
+            -0.016666175797581673,
+            0.0218023881316185,
+            -0.01730252057313919,
+            0.026802239939570427,
+            0.013825350441038609,
+            -0.015802564099431038,
+            -0.029817303642630577,
+            0.025378039106726646,
+            0.003367325058206916,
+            0.010499690659344196,
+            -0.02075696364045143,
+            0.014332910999655724,
+            0.04393810033798218,
+            -0.02593863010406494,
+            -0.02271145209670067,
+            0.027681002393364906,
+            0.014408666640520096,
+            0.0633314698934555,
+            0.036544378846883774,
+            0.006624804809689522,
+            0.022362977266311646,
+            0.020575150847434998,
+            -0.028968842700123787,
+            -0.01963578537106514,
+            -0.046362269669771194,
+            -0.04757435619831085,
+            0.019878201186656952,
+            -0.04566532000899315,
+            0.0050869714468717575,
+            0.038544319570064545,
+            -0.013704141601920128,
+            -0.014446543529629707,
+            -0.04021093621850014,
+            0.017847958952188492,
+            0.0003908502112608403,
+            0.011976920068264008,
+            -0.0022082682698965073,
+            0.00033285003155469894,
+            -0.011825408786535263,
+            0.008878526277840137,
+            -0.03778676688671112,
+            -0.025196226313710213,
+            0.025181075558066368,
+            0.022862963378429413,
+            0.024120500311255455,
+            0.028802180662751198,
+            -0.03077181987464428,
+            -0.009984553791582584,
+            0.021378157660365105,
+            -0.04242299124598503,
+            -0.006556625012308359,
+            0.011332998983561993,
+            0.0026703758630901575,
+            0.003872992005199194,
+            0.032786913216114044,
+            0.052998438477516174,
+            -0.03172633796930313,
+            0.02065090648829937,
+            -0.010431510396301746,
+            0.04957429692149162,
+            0.00024691506405360997,
+            0.028968842700123787,
+            -0.019514575600624084,
+            -0.0134389977902174,
+            -0.029529431834816933,
+            -0.009795165620744228,
+            -0.01356020662933588,
+            -0.008484598249197006,
+            -0.002998017705976963,
+            -0.009575475007295609,
+            0.0040453351102769375,
+            0.0331808403134346,
+            0.00273855566047132,
+            0.030680913478136063,
+            0.014325335621833801,
+            0.0051475754007697105,
+            -0.08745196461677551,
+            0.026059838011860847,
+            -0.0033616432920098305,
+            0.005859675817191601,
+            0.0062460280023515224,
+            0.03013547509908676,
+            -0.0312718041241169,
+            0.020938776433467865,
+            0.026620427146553993,
+            0.0011145505122840405,
+            0.016969196498394012,
+            0.036847397685050964,
+            -0.04181694984436035,
+            0.009643655270338058,
+            0.012590537779033184,
+            -0.0033881578128784895,
+            0.04584713280200958,
+            0.011628445237874985,
+            0.019514575600624084,
+            0.018529755994677544,
+            0.003569970605894923,
+            0.00297907879576087,
+            0.006533898413181305,
+            0.03724132478237152,
+            0.003462019143626094,
+            -0.009833043441176414,
+            0.014976831153035164,
+            -0.010757259093225002,
+            -0.011219366453588009,
+            -0.037998881191015244,
+            0.014454119838774204,
+            0.008742166683077812,
+            -0.017454031854867935,
+            -0.02959003672003746,
+            0.0026949963066726923,
+            -0.031908150762319565,
+            0.026559824123978615,
+            -0.002445003716275096,
+            0.005738467443734407,
+            -0.010454237461090088,
+            -0.006757376715540886,
+            -0.035423196852207184,
+            0.028453707695007324,
+            -0.02095392718911171,
+            0.0053142374381423,
+            0.0007258309051394463,
+            -0.029438525438308716,
+            0.0060073984786868095,
+            0.017863109707832336,
+            -0.0025965142995119095,
+            0.0025302283465862274,
+            -0.02349930815398693,
+            0.024468975141644478,
+            0.025635607540607452,
+            0.004518806003034115,
+            -0.014037464745342731,
+            -0.013113250024616718,
+            -0.003605954349040985,
+            -0.009742137044668198,
+            -0.016196493059396744,
+            0.024468975141644478,
+            -0.014810169115662575,
+            0.00308892410248518,
+            0.021847840398550034,
+            0.014264730736613274,
+            -0.007045246660709381,
+            -0.004007457755506039,
+            -0.04066546633839607,
+            -0.01218903437256813,
+            -0.04760465770959854,
+            -0.011628445237874985,
+            -0.0034601252991706133,
+            -0.028696123510599136,
+            0.0061513339169323444,
+            -0.04481685906648636,
+            -0.01235569640994072,
+            0.02309022843837738,
+            0.018135828897356987,
+            0.0019488062243908644,
+            -0.009173971600830555,
+            -0.022605394944548607,
+            8.486965816700831e-05,
+            -0.01763584464788437,
+            -0.017044952139258385,
+            0.015393485315144062,
+            0.031059689819812775,
+            -0.03101423755288124,
+            0.04024123772978783,
+            0.003893824527040124,
+            -0.08066429197788239,
+            0.010393633507192135,
+            -0.026362858712673187,
+            0.07011914253234863,
+            -0.0025453795678913593,
+            0.014075342565774918,
+            0.06781618297100067,
+            -0.012431452050805092,
+            -0.02007516659796238,
+            0.03142331540584564,
+            -0.008249756880104542,
+            -1.1311515663692262e-05,
+            -0.02884763479232788,
+            -0.02739313244819641,
+            -0.053271155804395676,
+            -0.009704259224236012,
+            -0.010787560604512691,
+            0.03360506892204285,
+            -0.024317465722560883,
+            -0.0007656024536117911,
+            0.012620840221643448,
+            0.02983245439827442,
+            0.016635872423648834,
+            -0.0015681355725973845,
+            0.008272483013570309,
+            0.02881733328104019,
+            0.0274234339594841,
+            -0.0036154238041490316,
+            0.00814369972795248,
+            -0.034211114048957825,
+            -0.0068066176027059555,
+            -0.0493621826171875,
+            0.011302697472274303,
+            -0.02596893161535263,
+            -0.02596893161535263,
+            0.007643714081496,
+            -0.011476934887468815,
+            0.03748374432325363,
+            -0.024696242064237595,
+            -0.007503566797822714,
+            0.005037730094045401,
+            0.012855681590735912,
+            -0.017757052555680275,
+            -0.03612014651298523,
+            -0.0028502948116511106,
+            0.0026381798088550568,
+            0.007560383062809706,
+            -0.053240854293107986,
+            0.004499867558479309,
+            -0.018469152972102165,
+            0.0279688723385334,
+            -0.02139330841600895,
+            -0.013688990846276283,
+            0.042029064148664474,
+            0.011363301426172256,
+            0.027499189600348473,
+            -0.026468917727470398,
+            0.0017849853029474616,
+            0.012825379148125648,
+            -0.0020056227222085,
+            0.032514192163944244,
+            -0.030574856325984,
+            -0.03257479891180992,
+            0.001548249856568873,
+            -0.007840678095817566,
+            -0.0625436082482338,
+            -0.011386028490960598,
+            -0.00981789268553257,
+            0.017317671328783035,
+            0.0446956530213356,
+            0.04557441547513008,
+            -0.01068150345236063,
+            0.002174178371205926,
+            -0.02213571034371853,
+            -0.04542290419340134,
+            -0.02643861435353756,
+            0.0016486257081851363,
+            -0.019999410957098007,
+            -0.02789311669766903,
+            -0.0018313854234293103,
+            0.011211791075766087,
+            -0.013651113025844097,
+            0.01134057529270649,
+            -0.018560059368610382,
+            -0.01862066239118576,
+            -0.012332970276474953,
+            0.018575210124254227,
+            0.009901223704218864,
+            -0.0018796793883666396,
+            -0.015037435106933117,
+            -0.00980274099856615,
+            0.018969137221574783,
+            -0.011279970407485962,
+            0.0009388927719555795,
+            0.01188601367175579,
+            0.024347767233848572,
+            -0.007083124481141567,
+            0.010961798019707203,
+            -0.034756552428007126,
+            -0.01966608688235283,
+            0.007984613068401814,
+            0.002719616750255227,
+            0.020711511373519897,
+            0.008742166683077812,
+            -0.04275631532073021,
+            -0.044301725924015045,
+            0.016044981777668,
+            -0.0562104657292366,
+            0.028120383620262146,
+            -0.03327174484729767,
+            -0.012514783069491386,
+            -0.02671133354306221,
+            0.0013418165035545826,
+            -0.01828734017908573,
+            0.022438732907176018,
+            0.03778676688671112,
+            -0.0011486404109746218,
+            0.005503625608980656,
+            0.014658658765256405,
+            0.04384719207882881,
+            -0.05021063983440399,
+            -0.041180603206157684,
+            0.03754434734582901,
+            -0.01601468026638031,
+            0.0185146052390337,
+            0.00021744151308666915,
+            0.010348179377615452,
+            -0.014431392773985863,
+            0.022772056981921196,
+            0.013757170177996159,
+            -0.008764892816543579,
+            -0.030574856325984,
+            0.04945308715105057,
+            -0.023650817573070526,
+            -0.049513693898916245,
+            0.0018323323456570506,
+            0.018878230825066566,
+            0.018635815009474754,
+            -0.014870774000883102,
+            -0.02907489985227585,
+            -0.024317465722560883,
+            -0.006060427520424128,
+            -0.00947699323296547,
+            -0.03402930125594139,
+            0.011363301426172256,
+            0.0048445542342960835,
+            0.016666175797581673,
+            -0.00645056739449501,
+            -0.0073141781613230705,
+            -0.009370936080813408,
+            0.022559940814971924,
+            0.012635990977287292,
+            -0.03175663948059082,
+            -0.017484333366155624,
+            -0.02528713271021843,
+            0.004359719809144735,
+            0.010136065073311329,
+            0.022166013717651367,
+            0.024287162348628044,
+            -0.0323626808822155,
+            0.04218057543039322,
+            0.01550711877644062,
+            -0.008727015927433968,
+            -0.0019317612750455737,
+            0.02478714846074581,
+            0.04324115067720413,
+            0.0218023881316185,
+            0.026514369994401932,
+            0.002107892418280244,
+            8.705945947440341e-05,
+            -0.015620751306414604,
+            -0.0011391709558665752,
+            0.01862066239118576,
+            0.04911976307630539,
+            -0.012151156552135944,
+            0.01716616004705429,
+            -0.03578682243824005,
+            0.00396200455725193,
+            -0.02580226957798004,
+            -0.030786970630288124,
+            0.03745344281196594,
+            -0.015060162171721458,
+            0.0191964041441679,
+            0.025044716894626617,
+            -0.022469034418463707,
+            -0.011075431481003761,
+            -0.019423669204115868,
+            -0.026181045919656754,
+            -0.01702980138361454,
+            -0.03281721472740173,
+            -0.018135828897356987,
+            -0.002653330797329545,
+            -0.035635314881801605,
+            -0.04987731948494911,
+            -0.004329417832195759,
+            -0.046392571181058884,
+            -0.010507266037166119,
+            0.017378276214003563,
+            -0.011166337877511978,
+            -0.025711363181471825,
+            -0.005336964037269354,
+            0.003973367623984814,
+            -0.022196315228939056,
+            -0.0066588944755494595,
+            0.008439145050942898,
+            0.035938333719968796,
+            -0.02298417128622532,
+            -0.013060221448540688,
+            -0.0024317465722560883,
+            -0.02887793630361557,
+            -0.007878555916249752,
+            0.008999735116958618,
+            0.007723257411271334,
+            -0.0012357590021565557,
+            -0.05196816474199295,
+            -0.006916462909430265,
+            -0.012992041185498238,
+            0.029196109622716904,
+            0.026468917727470398,
+            0.0012168202083557844,
+            -0.02783251367509365,
+            0.008749742060899734,
+            -0.0072535742074251175,
+            0.02339324913918972,
+            -0.007992188446223736,
+            -0.022908415645360947,
+            -0.02034788578748703,
+            0.030786970630288124,
+            -0.0159086212515831,
+            0.009113367646932602,
+            0.009976978413760662,
+            0.0121360057964921,
+            -0.03899884968996048,
+            -0.001445980160497129,
+            0.0005648507503792644,
+            -0.010128489695489407,
+            0.00423093605786562,
+            -0.006738437805324793,
+            -0.007079336792230606,
+            -0.022090258076786995,
+            -0.0218023881316185,
+            0.03818069398403168,
+            -0.009113367646932602,
+            -0.0035661826841533184,
+            -0.01088604237884283,
+            0.03436262533068657,
+            -0.00595815759152174,
+            -0.01993880607187748,
+            -0.019347915425896645,
+            0.014431392773985863,
+            0.05105910077691078,
+            -0.028529461473226547,
+            -0.03984731063246727,
+            0.0011136035900563002,
+            0.030150625854730606,
+            0.004140029661357403,
+            -0.016908591613173485,
+            0.0035851215943694115,
+            0.056392278522253036,
+            -0.011742077767848969,
+            0.028362801298499107,
+            0.027771908789873123,
+            0.02295386977493763,
+            -0.02021152526140213,
+            -0.01678738370537758,
+            -0.020999381318688393,
+            0.011098157614469528,
+            -0.03027183562517166,
+            0.027059808373451233,
+            0.018969137221574783,
+            0.01538590993732214,
+            0.05054396390914917,
+            -0.03178694099187851,
+            -0.02515077404677868,
+            0.008499749936163425,
+            0.0021722845267504454,
+            0.011007251217961311,
+            -0.007727045100182295,
+            -0.01801462098956108,
+            -0.001950700068846345,
+            -0.014022313989698887,
+            -0.019817598164081573,
+            -0.020938776433467865,
+            0.00954517349600792,
+            0.015696506947278976,
+            0.002649543108418584,
+            0.010863316245377064,
+            0.0016590419691056013,
+            0.01686313934624195,
+            0.01321930717676878,
+            -0.0444229319691658,
+            -7.936555630294606e-05,
+            0.03378688171505928,
+            -0.0293476190418005,
+            -0.012264790013432503,
+            -0.016060132533311844,
+            -0.004439263138920069,
+            -0.01706010289490223,
+            -0.02762039750814438,
+            -0.006874797400087118,
+            -0.01889338344335556,
+            -0.00814369972795248,
+            -0.028196139261126518,
+            -0.009992130100727081,
+            0.01949942484498024,
+            0.022332675755023956,
+            0.001196934375911951,
+            -0.006783891003578901,
+            -0.005901341326534748,
+            0.02092362567782402,
+            0.032514192163944244,
+            0.01952972821891308,
+            0.0058331615291535854,
+            -0.015158643946051598,
+            -0.004295328166335821,
+            -0.012370847165584564,
+            0.016060132533311844,
+            0.012537509202957153,
+            6.344509893096983e-05,
+            0.03493836522102356,
+            0.004170331638306379,
+            -0.022726602852344513,
+            -0.001230077352374792,
+            0.024923507124185562,
+            -0.021590271964669228,
+            -0.0108330138027668,
+            0.040029123425483704,
+            0.0085679292678833,
+            -0.0024506852496415377,
+            -0.019287310540676117,
+            -0.019075196236371994,
+            0.02630225569009781,
+            -0.02133270539343357,
+            0.002083271974697709,
+            0.0482713058590889,
+            0.00035344602656550705,
+            -0.029802151024341583,
+            0.017954016104340553,
+            -0.013242034241557121,
+            -0.018560059368610382,
+            0.01848430372774601,
+            0.0012859469279646873,
+            -0.039968520402908325,
+            0.019014591351151466,
+            0.018408548086881638,
+            0.022635696455836296,
+            -0.02918095886707306,
+            0.0002127068000845611,
+            0.03233237937092781,
+            0.004037759732455015,
+            0.001595596899278462,
+            -0.008530051447451115,
+            -0.0006065162015147507,
+            0.010431510396301746,
+            0.001746160676702857,
+            0.02854461409151554,
+            -0.0030434709042310715,
+            0.027771908789873123,
+            -0.006727074272930622,
+            -0.008863375522196293,
+            0.019181253388524055,
+            -0.02109028771519661,
+            0.013166278600692749,
+            0.0685434341430664,
+            0.0021950111258774996,
+            0.025847723707556725,
+            0.007181606721132994,
+            0.024347767233848572,
+            0.02018122375011444,
+            -0.027271922677755356,
+            -0.02021152526140213,
+            -0.005397567991167307,
+            -0.0014033677289262414,
+            -0.03787767142057419,
+            0.011128460057079792,
+            -0.021302402019500732,
+            -0.013249609619379044,
+            -0.0022347825579345226,
+            -0.002316219499334693,
+            -0.006405114196240902,
+            0.034211114048957825,
+            -0.0014630251098424196,
+            -0.018560059368610382,
+            -0.035423196852207184,
+            0.037726160138845444,
+            0.04048365354537964,
+            0.0389382466673851,
+            -0.005984671879559755,
+            0.03548380360007286,
+            0.0030756669584661722,
+            0.0025302283465862274,
+            0.041453324258327484,
+            0.004791525658220053,
+            -0.011567841283977032,
+            0.02295386977493763,
+            0.012416300363838673,
+            0.03351416438817978,
+            -0.02045394293963909,
+            -0.02630225569009781,
+            -0.003299145260825753,
+            -0.026423463597893715,
+            0.012492056004703045,
+            -0.019726691767573357,
+            0.0018768386216834188,
+            0.007117214612662792,
+            0.015196521766483784,
+            -0.016044981777668,
+            0.014719262719154358,
+            -0.006102093029767275,
+            -0.021014532074332237,
+            0.011719351634383202,
+            -0.007780073676258326,
+            0.011961768381297588,
+            -0.02262054570019245,
+            0.001138224033638835,
+            -0.04063516482710838,
+            0.008280059322714806,
+            0.014022313989698887,
+            -0.01598437689244747,
+            0.03205966204404831,
+            0.020969079807400703,
+            -0.01801462098956108,
+            -0.0009682480012997985,
+            0.0034714885987341404,
+            -0.010393633507192135,
+            -0.011067855171859264,
+            -0.019923655316233635,
+            0.02786281518638134,
+            -0.015802564099431038,
+            -0.025908326730132103,
+            0.02454473078250885,
+            -0.01743887923657894,
+            0.015151068568229675,
+            -0.004299115855246782,
+            0.00888610165566206,
+            -0.019120648503303528,
+            0.02284781076014042,
+            -0.023741723969578743,
+            -0.0009128518868237734,
+            -0.020287280902266502,
+            0.0028389315120875835,
+            0.014135947450995445,
+            -0.018272189423441887,
+            -0.02139330841600895,
+            0.019150950014591217,
+            -0.005420294590294361,
+            -0.003931702114641666,
+            0.010014856234192848,
+            0.0030983935575932264,
+            -0.0029715034179389477,
+            -0.004549108445644379,
+            -0.03281721472740173,
+            -0.02915065549314022,
+            0.0033086147159337997,
+            0.011386028490960598,
+            0.015257125720381737,
+            0.027574945241212845,
+            0.03715042024850845,
+            0.006052851676940918,
+            0.0036949669010937214,
+            0.0009327376610599458,
+            -0.020726662129163742,
+            0.042635105550289154,
+            -0.020469093695282936,
+            0.040847279131412506,
+            0.032544493675231934,
+            -0.0197418425232172,
+            -0.023908386006951332,
+            0.0029052174650132656,
+            0.03115059621632099,
+            0.03151422366499901,
+            0.025211378931999207,
+            0.012310243211686611,
+            0.03690800443291664,
+            -0.004359719809144735,
+            -0.032211173325777054,
+            0.015802564099431038,
+            0.022256920114159584,
+            -0.008105821907520294,
+            0.009673957712948322,
+            0.027438584715127945,
+            0.013279912061989307,
+            -0.007757347077131271,
+            0.030150625854730606,
+            0.0012234487803652883,
+            0.02396899089217186,
+            0.02498411200940609,
+            -0.02190844528377056,
+            0.01743887923657894,
+            -0.0001154677156591788,
+            0.006727074272930622,
+            -0.009840618818998337,
+            0.004348356742411852,
+            0.011810258030891418,
+            0.018272189423441887,
+            0.018908534198999405,
+            0.003246116451919079,
+            -0.07060398161411285,
+            0.019726691767573357,
+            0.004507442936301231,
+            -0.02160542458295822,
+            0.00390140013769269,
+            -0.0024866692256182432,
+            0.012704171240329742,
+            0.02105998620390892,
+            0.012461753562092781,
+            -0.01790856383740902,
+            0.0013626492582261562,
+            0.011961768381297588,
+            -0.00595815759152174,
+            0.04730163514614105,
+            0.0019791084341704845,
+            -0.01437078882008791,
+            -0.016105586662888527,
+            0.013726867735385895,
+            0.02525683119893074,
+            -0.02355991117656231,
+            0.006325571332126856,
+            0.005621046293526888,
+            -0.004147605039179325,
+            -0.010742107406258583,
+            -0.030377892777323723,
+            -0.005280147306621075,
+            -0.03436262533068657,
+            0.023150833323597908,
+            0.015491968020796776,
+            0.014764715917408466,
+            -0.025044716894626617,
+            -0.004280176945030689,
+            0.0046400148421525955,
+            -0.021287251263856888,
+            0.003261267440393567,
+            0.01161329448223114,
+            0.0005104016163386405,
+            -0.020984230563044548,
+            0.00959820207208395,
+            0.013469300232827663,
+            0.06848283112049103,
+            0.0007442962378263474,
+            0.029771849513053894,
+            0.0422411784529686,
+            -0.0012243957025930285,
+            -0.006371024530380964,
+            0.0036286809481680393,
+            -0.019953956827521324,
+            0.003473382443189621,
+            0.025878025218844414,
+            -0.03224147483706474,
+            -0.055058982223272324,
+            -0.007545232307165861,
+            -0.04624105989933014,
+            0.001489539397880435,
+            0.014711687341332436,
+            0.023044776171445847,
+            -0.0037782976869493723,
+            -0.034817155450582504,
+            0.016908591613173485,
+            -0.003071879269555211,
+            0.007014944683760405,
+            -0.02593863010406494,
+            -0.0294839795678854,
+            -0.026014383882284164,
+            -0.02339324913918972,
+            -0.012908710166811943,
+            -0.022726602852344513,
+            0.012590537779033184,
+            -0.0015103721525520086,
+            -0.06587684899568558,
+            -0.009780014865100384,
+            0.016166189685463905,
+            0.001200722181238234,
+            0.011939042247831821,
+            0.011083006858825684,
+            -0.007083124481141567,
+            -0.0018275976181030273,
+            -0.022559940814971924,
+            0.03124150261282921,
+            0.005177877843379974,
+            0.03584742918610573,
+            -0.006503595970571041,
+            -0.009128518402576447,
+            0.01073453202843666,
+            0.04527139291167259,
+            0.0017916138749569654,
+            0.010226971469819546,
+            -0.020560000091791153,
+            0.0015065843472257257,
+            -0.02284781076014042,
+            -0.0031135445460677147,
+            -0.008863375522196293,
+            -0.0021211495622992516,
+            0.015893470495939255,
+            0.0019753205124288797,
+            -0.010204244405031204,
+            0.025438643991947174,
+            0.02251448854804039,
+            -0.009605777449905872,
+            -0.015969226136803627,
+            -0.02437806874513626,
+            -0.014545026235282421,
+            -0.0002627763315103948,
+            -0.010007280856370926,
+            -0.022453883662819862,
+            -0.020878173410892487,
+            -0.002566212322562933,
+            -0.02559015527367592,
+            0.022529639303684235,
+            0.01220418605953455,
+            -4.882668508798815e-05,
+            -0.006723286584019661,
+            -0.038544319570064545,
+            -0.003727162955328822,
+            -0.03118089959025383,
+            -0.005374841392040253,
+            0.03287781774997711,
+            -0.03745344281196594,
+            -0.0002231231628684327,
+            0.014491996727883816,
+            -0.01993880607187748,
+            0.0028938541654497385,
+            0.008961857296526432,
+            0.02769615314900875,
+            -0.027408283203840256,
+            0.0001795638381736353,
+            0.006954340264201164,
+            0.016908591613173485,
+            -0.009878496639430523,
+            0.027196168899536133,
+            -0.04275631532073021,
+            0.011037553660571575,
+            -0.037120118737220764,
+            -0.020423639565706253,
+            -0.01523439958691597,
+            0.008363389410078526,
+            -0.005018791649490595,
+            -0.02637801133096218,
+            -0.010454237461090088,
+            0.006026337388902903,
+            -0.00693918950855732,
+            -0.01902974210679531,
+            -0.010620899498462677,
+            0.027438584715127945,
+            0.00931033119559288,
+            0.007977037690579891,
+            -0.02955973520874977,
+            -0.01631770096719265,
+            0.006280118133872747,
+            0.052089374512434006,
+            -0.01584801822900772,
+            0.014938953332602978,
+            -0.006639956030994654,
+            0.02342355251312256,
+            -0.028484009206295013,
+            -0.01872672140598297,
+            0.0315445251762867,
+            0.013946558348834515,
+            0.007064185570925474,
+            0.028953691944479942,
+            0.027196168899536133,
+            0.027105262503027916,
+            -0.04654408246278763,
+            0.009886072017252445,
+            -0.002613559365272522,
+            -0.008757317438721657,
+            0.03402930125594139,
+            0.01916610263288021,
+            0.022150862962007523,
+            0.0009744031121954322,
+            -0.01730252057313919,
+            0.0139844361692667,
+            0.014885924756526947,
+            0.007681591901928186,
+            -0.014242004603147507,
+            -0.02660527639091015,
+            0.003367325058206916,
+            -0.007992188446223736,
+            0.00047276070108637214,
+            0.020726662129163742,
+            -0.009052763693034649,
+            0.030999086797237396,
+            -0.014605630189180374,
+            -0.02593863010406494,
+            0.008916404098272324,
+            -0.02268115058541298,
+            -0.0041741193272173405,
+            0.0004476667381823063,
+            0.010083036497235298,
+            -0.023847781121730804,
+            -0.04669559374451637,
+            0.019408518448472023,
+            0.028393102809786797,
+            0.005579380784183741,
+            0.01328748743981123,
+            -0.035332292318344116,
+            0.008992158807814121,
+            0.012310243211686611,
+            0.011348150670528412,
+            0.0019847899675369263,
+            0.035362593829631805,
+            -0.00028834378463216126,
+            0.012885984033346176,
+            -0.008620957843959332,
+            0.028362801298499107,
+            0.00667025800794363,
+            -0.0323626808822155,
+            -0.0378473699092865,
+            -0.01513591781258583,
+            0.015257125720381737,
+            0.006692984607070684,
+            -0.011840560473501682,
+            -0.02278720773756504,
+            -0.038817036896944046,
+            0.015787413343787193,
+            -0.012916285544633865,
+            -0.038271598517894745,
+            -0.010158791206777096,
+            -0.009742137044668198,
+            0.016635872423648834,
+            -0.006916462909430265,
+            -0.016802534461021423,
+            -0.016090434044599533,
+            0.005821797996759415,
+            7.161247049225494e-05,
+            0.0244083721190691,
+            -0.001251857029274106,
+            -0.006855858489871025,
+            -0.016393456608057022,
+            -0.01188601367175579,
+            -0.0035074723418802023,
+            0.03602924197912216,
+            -0.014219277538359165,
+            -0.02383263036608696,
+            0.014984406530857086,
+            0.02657497487962246,
+            0.011636020615696907,
+            0.03227177634835243,
+            0.007181606721132994,
+            -0.014022313989698887,
+            0.003016956616193056,
+            -0.033029329031705856,
+            -0.0046400148421525955,
+            0.005196816753596067,
+            -0.006427840795367956,
+            -0.015651054680347443,
+            -0.01815097965300083,
+            -0.015832867473363876,
+            0.030756669119000435,
+            -0.02487805485725403,
+            0.0218023881316185,
+            0.030347591266036034,
+            0.003174148965626955,
+            -0.02915065549314022,
+            0.013598084449768066,
+            -0.0072005451656877995,
+            0.01862066239118576,
+            0.0019014590652659535,
+            0.028014326468110085,
+            -0.007461901288479567,
+            -0.004818039946258068,
+            -0.02501441352069378,
+            -0.03696860745549202,
+            -0.023484155535697937,
+            -0.008817922323942184,
+            0.0027537066489458084,
+            0.020984230563044548,
+            -0.0375746488571167,
+            -0.017757052555680275,
+            0.009855770505964756,
+            0.015757111832499504,
+            -0.03027183562517166,
+            -0.0025908327661454678,
+            -0.010158791206777096,
+            -0.0004055278259329498,
+            0.005140000022947788,
+            -0.02227207086980343,
+            -0.008977008052170277,
+            -0.015726808458566666,
+            -0.0010435298318043351,
+            0.006166485138237476,
+            -0.0007409819518215954,
+            -0.05805889517068863,
+            -0.013969285413622856,
+            0.018635815009474754,
+            0.0026400736533105373,
+            0.008908828720450401,
+            0.015938924625515938,
+            -0.0111739132553339,
+            0.0031533162109553814,
+            -0.026756787672638893,
+            0.023893235251307487,
+            -0.009492144919931889,
+            0.009355784393846989,
+            -0.013742019422352314,
+            0.012514783069491386,
+            0.014014738611876965,
+            0.004795313347131014,
+            0.0010681503918021917,
+            -0.0054051438346505165,
+            0.01601468026638031,
+            0.028377952054142952,
+            -0.023756874725222588,
+            0.011454207822680473,
+            -0.0002514130319468677,
+            -0.0032158142421394587,
+            -0.029196109622716904,
+            0.00509454682469368,
+            0.003931702114641666,
+            -0.005109698045998812,
+            0.016651025041937828,
+            -0.036544378846883774,
+            0.01662072166800499,
+            0.02177208475768566,
+            0.032544493675231934,
+            0.014832896180450916,
+            -0.017287369817495346,
+            -0.011658747680485249,
+            -0.012385998852550983,
+            -0.002624922664836049,
+            0.022181164473295212,
+            0.012120855040848255,
+            -0.01815097965300083,
+            -0.024287162348628044,
+            0.011249667964875698,
+            0.0005283934879116714,
+            -0.01203752402216196,
+            0.010795135982334614,
+            -0.013166278600692749,
+            0.01349202636629343,
+            0.012711746618151665,
+            0.00866641104221344,
+            0.00888610165566206,
+            0.022862963378429413,
+            -0.011878437362611294,
+            -0.017711598426103592,
+            0.00656041270121932,
+            -0.008916404098272324,
+            -0.0034089903347194195,
+            -0.012598113156855106,
+            0.010795135982334614,
+            0.014878349378705025,
+            0.06490717828273773,
+            -0.013143551535904408,
+            0.0009526234352961183,
+            -0.020438792183995247,
+            -0.031938452273607254,
+            -0.0017944546416401863,
+            -0.020908474922180176,
+            0.0111739132553339,
+            0.008370965719223022,
+            0.01598437689244747,
+            0.012022373266518116,
+            -0.0021912232041358948,
+            0.00795431062579155,
+            0.019317612051963806,
+            -0.010423935018479824,
+            7.255941454786807e-05,
+            -0.006503595970571041,
+            0.0013683309080079198,
+            0.0035851215943694115,
+            -0.011719351634383202,
+            -0.0026949963066726923,
+            0.035695917904376984,
+            0.01068150345236063,
+            0.011651171371340752,
+            -0.03445352986454964,
+            0.0020435003098100424,
+            0.020150920376181602,
+            0.007602048572152853,
+            -0.008302785456180573,
+            -0.01563590206205845,
+            0.01706010289490223,
+            -0.015317730605602264,
+            -0.0022499337792396545,
+            0.002888172399252653,
+            -0.010151215828955173,
+            0.04581683129072189,
+            -0.019787294790148735,
+            -0.0011742077767848969,
+            0.031847547739744186,
+            0.0030207443051040173,
+            -0.0040983641520142555,
+            0.005408931523561478,
+            -0.003473382443189621,
+            -0.0069013116881251335,
+            -0.02139330841600895,
+            0.0031381649896502495,
+            0.00667025800794363,
+            -0.019287310540676117,
+            -0.0072005451656877995,
+            -0.007719469256699085,
+            0.006897523999214172,
+            0.023105379194021225,
+            -0.009492144919931889,
+            -0.00016145357221830636,
+            -0.008575504645705223,
+            0.009848194196820259,
+            -0.0011912527261301875,
+            -0.044301725924015045,
+            -0.006147546228021383,
+            0.021181194111704826,
+            0.000973456131760031,
+            0.0005757405888289213,
+            -0.007450537756085396,
+            0.005007428117096424,
+            -0.017423728480935097,
+            -0.008454295806586742,
+            0.02369627170264721,
+            0.01736312545835972,
+            -0.009113367646932602,
+            0.010863316245377064,
+            -0.0035680767614394426,
+            0.00433320552110672,
+            -0.020014561712741852,
+            -0.01095422264188528,
+            0.003462019143626094,
+            0.015272277407348156,
+            -0.0009819786064326763,
+            -0.029029447585344315,
+            -0.0012272365856915712,
+            -0.00024194362049456686,
+            0.02918095886707306,
+            0.01486319862306118,
+            -0.008961857296526432,
+            -0.021211495622992516,
+            -0.013242034241557121,
+            -0.001237652963027358,
+            -0.015726808458566666,
+            -0.009620928205549717,
+            0.011855711229145527,
+            0.0009137988090515137,
+            0.008992158807814121,
+            -0.018135828897356987,
+            -0.027347678318619728,
+            -0.016166189685463905,
+            0.0018465365283191204,
+            -0.012870832346379757,
+            0.003340810537338257,
+            0.015514694154262543,
+            -0.03430201858282089,
+            -0.015060162171721458,
+            -0.03805948421359062,
+            0.008280059322714806,
+            -0.0029658216517418623,
+            -0.02024182677268982,
+            -0.023605365306138992,
+            0.009484568610787392,
+            0.019181253388524055,
+            -0.0015548784285783768,
+            0.019847899675369263,
+            -0.000889651826582849,
+            0.020060013979673386,
+            -0.0073141781613230705,
+            0.0031703610438853502,
+            -0.013007191941142082,
+            0.0030245319940149784,
+            0.008333087898790836,
+            0.02890823967754841,
+            -0.023075077682733536,
+            -0.006503595970571041,
+            -0.004685468040406704,
+            0.004746072459965944,
+            0.012658718042075634,
+            0.014613205567002296,
+            0.020741812884807587,
+            -0.008272483013570309,
+            -0.0027612822595983744,
+            0.012484480626881123,
+            -0.003799130441620946,
+            0.007295239716768265,
+            0.002390081062912941,
+            0.0019677451346069574,
+            0.004162756260484457,
+            0.008477022871375084,
+            0.0053710537031292915,
+            0.027650700882077217,
+            -0.0028275682125240564,
+            0.0031703610438853502,
+            0.024620486423373222,
+            0.02322658896446228,
+            -0.018878230825066566,
+            -0.028074929490685463,
+            0.002024561632424593,
+            0.050695475190877914,
+            0.007245998363941908,
+            -0.006810405291616917,
+            -0.03927157074213028,
+            -0.0031760428100824356,
+            0.028635520488023758,
+            0.023650817573070526,
+            0.01584801822900772,
+            -0.010567869991064072,
+            -0.012098127976059914,
+            0.02772645466029644,
+            0.013295062817633152,
+            0.02616589516401291,
+            -0.025302285328507423,
+            0.010704229585826397,
+            -0.03869583085179329,
+            0.0449683703482151,
+            0.004439263138920069,
+            -0.014476845972239971,
+            0.0011884119594469666,
+            0.010386057198047638,
+            -0.004537744913250208,
+            0.006431628484278917,
+            -5.8059369621332735e-05,
+            -0.0008086882880888879,
+            0.011923890560865402,
+            0.012007221579551697,
+            0.004265025723725557,
+            -0.02399929240345955,
+            0.007359631359577179,
+            0.004272601567208767,
+            0.03005971945822239,
+            0.005431658122688532,
+            -0.0013863227795809507,
+            0.012385998852550983,
+            -0.005810434930026531,
+            0.008855799213051796,
+            -0.010719381272792816,
+            0.020469093695282936,
+            -0.02918095886707306,
+            1.0986003871948924e-05,
+            0.02051454596221447,
+            -0.0015785519499331713,
+            0.0038199631962925196,
+            -0.023741723969578743,
+            0.01186328660696745,
+            -0.052968136966228485,
+            0.021166043356060982,
+            -0.018741872161626816,
+            0.021741783246397972,
+            0.019590331241488457,
+            -0.009734561666846275,
+            -0.012052674777805805,
+            0.03545350208878517,
+            0.0026760573964565992,
+            -0.014264730736613274,
+            -0.010643625631928444,
+            -0.017999470233917236,
+            -0.02830219641327858,
+            0.041968461126089096,
+            -0.0006117243901826441,
+            0.0028275682125240564,
+            0.0024222771171480417,
+            -0.010666352696716785,
+            -0.008923979476094246,
+            0.001954487757757306,
+            0.020499395206570625,
+            -0.03369597718119621,
+            -0.011128460057079792,
+            -0.008689138107001781,
+            -0.02566591091454029,
+            -0.00020300064352340996,
+            -0.011961768381297588,
+            0.026211349293589592,
+            0.01240872498601675,
+            -0.03696860745549202,
+            0.003780191531404853,
+            0.023650817573070526,
+            0.001603172393515706,
+            0.018120678141713142,
+            0.002130619017407298,
+            0.0010795136913657188,
+            0.009742137044668198,
+            -0.010476963594555855,
+            0.026559824123978615,
+            -0.009204274043440819,
+            0.008658835664391518,
+            -0.001950700068846345,
+            -0.017287369817495346,
+            0.017969166859984398,
+            0.004083212930709124,
+            0.024120500311255455,
+            0.011954193003475666,
+            0.011605718173086643,
+            0.01112088467925787,
+            -0.019878201186656952,
+            0.020090317353606224,
+            -0.0018891488434746861,
+            0.009075489826500416,
+            0.024726543575525284,
+            -0.005863463506102562,
+            0.020832719281315804,
+            0.015257125720381737,
+            -0.00020229043730068952,
+            0.011295121163129807,
+            -0.020120618864893913,
+            -0.011779955588281155,
+            0.0085679292678833,
+            -0.009605777449905872,
+            -0.010325453244149685,
+            0.009901223704218864,
+            -0.010105762630701065,
+            -0.015052586793899536,
+            -0.0071247899904847145,
+            0.016635872423648834,
+            0.0028976418543606997,
+            -0.02610529027879238,
+            -1.7326075976598077e-05,
+            0.008901252411305904,
+            0.006166485138237476,
+            0.018741872161626816,
+            0.011317848227918148,
+            -0.006692984607070684,
+            0.010363331064581871,
+            0.011476934887468815,
+            0.0053104497492313385,
+            -0.005079395603388548,
+            -0.013431422412395477,
+            -0.0024241709616035223,
+            0.018090376630425453,
+            0.009810317307710648,
+            -0.01922670565545559,
+            0.004564259201288223,
+            0.0008877579239197075,
+            -0.01420412678271532,
+            -0.019347915425896645,
+            -0.0038483713287860155,
+            -0.004170331638306379,
+            0.0027915844693779945,
+            0.0509984977543354,
+            -0.0006216673064045608,
+            0.007893706671893597,
+            -0.025741664692759514,
+            -0.006299056578427553,
+            -0.00027556007262319326,
+            -0.0036097420379519463,
+            0.01272689737379551,
+            -0.01824188604950905,
+            -0.03724132478237152,
+            0.012711746618151665,
+            -0.0012584856012836099,
+            0.014923802576959133,
+            -0.00297907879576087,
+            0.014226853847503662,
+            -0.017347972840070724,
+            0.01916610263288021,
+            -0.0011117096291854978,
+            -0.016029831022024155,
+            0.0029374135192483664,
+            -0.011143610812723637,
+            -0.021317552775144577,
+            0.005916492082178593,
+            -0.006848283112049103,
+            -0.03896854817867279,
+            0.0029885482508689165,
+            -0.010423935018479824,
+            0.0011798894265666604,
+            -0.003912763670086861,
+            0.020166072994470596,
+            -0.0061513339169323444,
+            0.0008697660523466766,
+            0.02177208475768566,
+            0.0017026013229042292,
+            0.01601468026638031,
+            -0.014499573037028313,
+            0.010999675840139389,
+            0.012598113156855106,
+            0.02210540883243084,
+            -0.032180868089199066,
+            -0.01393140759319067,
+            0.014211702160537243,
+            0.0026116655208170414,
+            -0.005382417235523462,
+            0.020635755732655525,
+            -0.01294658798724413,
+            0.019650936126708984,
+            -0.0005577487172558904,
+            -0.011704200878739357,
+            -0.03524138405919075,
+            -0.016529815271496773,
+            -3.343888238305226e-05,
+            -0.008439145050942898,
+            0.025317436084151268,
+            0.01760554127395153,
+            -0.0108330138027668,
+            0.026468917727470398,
+            0.0037574649322777987,
+            0.014598054811358452,
+            -0.014257155358791351,
+            -0.005999823100864887,
+            -0.0003773563075810671,
+            -0.0119314668700099,
+            0.015264702029526234,
+            -0.03512017801403999,
+            0.0008342557121068239,
+            -0.00839369185268879,
+            0.005242269951850176,
+            -0.0254992488771677,
+            0.018257036805152893,
+            0.025817420333623886,
+            -0.0031419529113918543,
+            -0.005037730094045401,
+            -0.022802358493208885,
+            0.028968842700123787,
+            0.03049910068511963,
+            0.0024828813038766384,
+            -0.0013825349742546678,
+            -0.00352072948589921,
+            0.0033218718599528074,
+            0.0211205892264843,
+            -0.037998881191015244,
+            -0.009098216891288757,
+            -0.028696123510599136,
+            0.042332086712121964,
+            -0.008098246529698372,
+            -0.013431422412395477,
+            0.01862066239118576,
+            -0.0024715180043131113,
+            0.02355991117656231,
+            -0.004689255729317665,
+            -0.004855917301028967,
+            0.03148392215371132,
+            0.011514811776578426,
+            0.02772645466029644,
+            -0.04015033319592476,
+            0.014704111963510513,
+            -0.006867222022265196,
+            0.004867280833423138,
+            -0.013484450988471508,
+            0.005302873905748129,
+            0.02580226957798004,
+            -0.018060073256492615,
+            0.0068066176027059555,
+            0.030665762722492218,
+            0.00748084019869566,
+            0.002448791405186057,
+            0.0165904201567173,
+            0.003890036838129163,
+            0.010855740867555141,
+            0.000841831264551729,
+            0.0052574207074940205,
+            0.022423582151532173,
+            0.014454119838774204,
+            -0.016029831022024155,
+            -0.015120766125619411,
+            -0.00044790346873924136,
+            -0.004450626205652952,
+            0.005261208396404982,
+            0.013242034241557121,
+            0.0061172437854111195,
+            -0.015681356191635132,
+            0.01052999310195446,
+            0.014158673584461212,
+            0.013128400780260563,
+            -0.0039847311563789845,
+            -0.03412020578980446,
+            -0.02443867363035679,
+            -0.021438762545585632,
+            0.003104075090959668,
+            -0.0008281006012111902,
+            -0.0021211495622992516,
+            0.0017954016802832484,
+            0.017347972840070724,
+            -0.0008352026343345642,
+            0.00379155483096838,
+            0.01686313934624195,
+            -0.01415109820663929,
+            -0.022090258076786995,
+            -0.0035964848939329386,
+            -0.027529491111636162,
+            0.0053710537031292915,
+            0.001844642567448318,
+            -0.00965123064815998,
+            0.008530051447451115,
+            -0.0046059247106313705,
+            -0.007223271764814854,
+            -0.012196609750390053,
+            -0.02007516659796238,
+            0.005780132487416267,
+            0.016878290101885796,
+            0.005730891600251198,
+            -0.009272454306483269,
+            -0.003939277958124876,
+            0.011575416661798954,
+            0.017484333366155624,
+            0.017317671328783035,
+            -0.009295180439949036,
+            0.009628504514694214,
+            0.0004022135108243674,
+            0.004670316819101572,
+            -0.01834794320166111,
+            0.0045869858004152775,
+            -0.0027953721582889557,
+            -0.008658835664391518,
+            0.01105270441621542,
+            -0.00031201730598695576,
+            0.01628739945590496,
+            -0.013090522959828377,
+            0.005624833982437849,
+            0.010333028621971607,
+            -0.020590301603078842,
+            -0.0067308624275028706,
+            -0.01692374423146248,
+            0.026484068483114243,
+            -0.00504530593752861,
+            0.00045405857963487506,
+            -0.02978700026869774,
+            0.0033616432920098305,
+            0.000848933297675103,
+            -0.000845619011670351,
+            -0.012764775194227695,
+            0.010749682784080505,
+            0.03242328763008118,
+            0.0005179771687835455,
+            0.0216660276055336,
+            -0.0036154238041490316,
+            0.0007480839849449694,
+            -0.00785582885146141,
+            0.004977126140147448,
+            -0.004916521720588207,
+            -0.009901223704218864,
+            -0.010863316245377064,
+            0.021726632490754128,
+            -0.0033029329497367144,
+            0.0003356908564455807,
+            0.011332998983561993,
+            -0.012067826464772224,
+            0.0014620780711993575,
+            -0.0047687990590929985,
+            0.00971941091120243,
+            0.0061702728271484375,
+            0.009113367646932602,
+            -0.01176480483263731,
+            0.05451354384422302,
+            -0.0106284748762846,
+            0.005537715740501881,
+            -0.00439380994066596,
+            -0.022226616740226746,
+            -0.017317671328783035,
+            0.012711746618151665,
+            -0.017544938251376152,
+            0.03375658020377159,
+            0.024150803685188293,
+            0.0006467612111009657,
+            0.010393633507192135,
+            -0.006958128418773413,
+            0.023923536762595177,
+            -0.002602196065708995,
+            0.0010795136913657188,
+            -0.004855917301028967,
+            0.015802564099431038,
+            0.04511988162994385,
+            0.017393426969647408,
+            -0.001152428099885583,
+            0.00829521007835865,
+            0.020575150847434998,
+            0.00814369972795248,
+            -0.0020075165666639805,
+            0.014219277538359165,
+            -0.01024212222546339,
+            0.00900731049478054,
+            -0.004458202049136162,
+            0.027438584715127945,
+            0.00814369972795248,
+            -0.006310420110821724,
+            -0.0036836036015301943,
+            -0.0008209985098801553,
+            0.014173824340105057,
+            0.006950552575290203,
+            0.01963578537106514,
+            -0.00461350055411458,
+            -0.0012575386790558696,
+            0.013870803639292717,
+            0.014491996727883816,
+            -0.00552635220810771,
+            0.001724380999803543,
+            -0.01435563713312149,
+            -0.010204244405031204,
+            0.013575357384979725,
+            0.006086941808462143,
+            0.006310420110821724,
+            -0.014514723792672157,
+            -0.00980274099856615,
+            -0.004670316819101572,
+            0.0015435151290148497,
+            0.015052586793899536,
+            0.011060279794037342,
+            -0.004814252257347107,
+            0.006052851676940918,
+            0.013825350441038609,
+            -0.014014738611876965,
+            0.006147546228021383,
+            0.007208121009171009,
+            0.008855799213051796,
+            0.004787737503647804,
+            -0.010817863047122955,
+            -0.00428396463394165,
+            -0.0063937511295080185,
+            0.0012670081341639161,
+            0.016832837834954262,
+            0.013181429356336594,
+            0.0049581872299313545,
+            0.027529491111636162,
+            -0.00839369185268879,
+            -0.012302667833864689,
+            0.02339324913918972,
+            0.011257244274020195,
+            0.0031703610438853502,
+            0.014651083387434483,
+            -0.012635990977287292,
+            -0.012446602806448936,
+            0.006655106786638498,
+            0.004848341923207045,
+            0.006049063988029957,
+            0.009151245467364788,
+            0.00450365524739027,
+            0.007333117071539164,
+            -0.007098275702446699,
+            -0.007427811156958342,
+            0.0014810169814154506,
+            0.01642375811934471,
+            0.020393338054418564,
+            -0.024968961253762245,
+            -0.0018806264270097017,
+            0.0009052763343788683,
+            0.010878467001020908,
+            -0.003316190093755722,
+            0.012310243211686611,
+            5.65205882594455e-05,
+            -0.0074240234680473804,
+            0.010158791206777096,
+            -0.02119634486734867,
+            -0.012681444175541401,
+            0.0012717428617179394,
+            0.013037494383752346,
+            -0.019590331241488457,
+            0.018226735293865204,
+            -0.008893677033483982,
+            -0.013348091393709183,
+            0.013893529772758484,
+            0.020014561712741852,
+            0.004090788308531046,
+            -0.007738408166915178,
+            -0.021862991154193878,
+            -0.01127239502966404,
+            0.016060132533311844,
+            0.020272130146622658,
+            0.000947888707742095,
+            -0.0006472347304224968,
+            0.010173942893743515,
+            -0.013113250024616718,
+            -0.023605365306138992,
+            -0.005230906419456005,
+            0.010386057198047638,
+            0.01746918261051178,
+            0.018832778558135033,
+            -0.006685409229248762,
+            0.02443867363035679,
+            -0.0059127043932676315,
+            -0.015499543398618698,
+            2.86597842205083e-05,
+            -0.02802947722375393,
+            -0.017423728480935097,
+            0.00013778003631159663,
+            0.00246773031540215,
+            0.008598231710493565,
+            0.0008555619278922677,
+            0.01943882182240486,
+            0.000904329412151128,
+            0.017590390518307686,
+            -0.000659071491099894,
+            -0.002009410411119461,
+            -0.0013995799235999584,
+            -0.0015340456739068031,
+            0.02254479005932808,
+            -0.012166308239102364,
+            -0.01176480483263731,
+            0.022302372381091118,
+            0.006852070800960064,
+            0.014719262719154358,
+            -0.027635548263788223,
+            0.013757170177996159,
+            -0.015499543398618698,
+            0.017423728480935097,
+            -0.03408990427851677,
+            0.006420265417546034,
+            0.01489350013434887,
+            -0.010348179377615452,
+            -0.020499395206570625,
+            -0.029650641605257988,
+            -0.024150803685188293,
+            -0.0038710979279130697,
+            -0.010855740867555141,
+            -0.016847988590598106,
+            0.006121031939983368,
+            -0.005802859086543322,
+            0.018499454483389854,
+            -0.007079336792230606,
+            0.033877789974212646,
+            0.006969491485506296,
+            -0.004056698642671108,
+            0.003835114184767008,
+            0.022469034418463707,
+            -0.006102093029767275,
+            0.02109028771519661,
+            0.004408961161971092,
+            -0.001467759720981121,
+            0.013681414537131786,
+            -0.006552837323397398,
+            -0.002361672930419445,
+            0.005821797996759415,
+            -0.010484539903700352,
+            0.022014502435922623,
+            0.005924067925661802,
+            -0.007708106189966202,
+            -0.0006775368237867951,
+            -0.011454207822680473,
+            0.014726839028298855,
+            0.006530110724270344,
+            0.0022480397019535303,
+            -0.011295121163129807,
+            0.011249667964875698,
+            -0.004859705455601215,
+            0.0040642740204930305,
+            0.009484568610787392,
+            -0.01774190180003643,
+            -0.028862785547971725,
+            0.011416330002248287,
+            0.019120648503303528,
+            0.006382387597113848,
+            -0.0055755930952727795,
+            -0.0025681061670184135,
+            0.00839369185268879,
+            0.010651201009750366,
+            0.022150862962007523,
+            -0.012984465807676315,
+            -8.889415767043829e-05,
+            0.013204156421124935,
+            -0.007170243188738823,
+            -0.01056029461324215,
+            0.00487485621124506,
+            0.023438703268766403,
+            0.0068823727779090405,
+            -0.04327145218849182,
+            -0.018469152972102165,
+            0.01909034699201584,
+            0.007136153522878885,
+            -0.04154422879219055,
+            0.0026381798088550568,
+            -0.003689285134896636,
+            -0.023787178099155426,
+            0.016105586662888527,
+            -0.0011817833874374628,
+            -0.01828734017908573,
+            0.028756728395819664,
+            0.02830219641327858,
+            0.01834794320166111,
+            2.6987840101355687e-05,
+            -0.013332940638065338,
+            -0.0318172425031662,
+            -0.023590214550495148,
+            0.008280059322714806,
+            -0.02660527639091015,
+            -0.0072725131176412106,
+            0.011242092587053776,
+            -0.017454031854867935,
+            0.02972639724612236,
+            -0.008507325313985348,
+            0.026938600465655327,
+            0.011446632444858551,
+            0.001691238023340702,
+            -0.013446573168039322,
+            -0.018802475184202194,
+            -0.01350717805325985,
+            -0.008196728304028511,
+            0.014090493321418762,
+            0.017923714593052864,
+            0.011007251217961311,
+            0.011332998983561993,
+            0.02251448854804039,
+            -0.012158732861280441,
+            0.007969462312757969,
+            -2.888172457460314e-05,
+            -0.008961857296526432,
+            0.0026343921199440956,
+            -0.018393397331237793,
+            0.00868156272917986,
+            0.0062308767810463905,
+            0.005458172410726547,
+            -0.007208121009171009,
+            0.010742107406258583,
+            0.02712041325867176,
+            -0.018847929313778877,
+            0.011401179246604443,
+            -0.031362712383270264,
+            -0.014787442982196808,
+            -0.01056029461324215,
+            0.026862844824790955,
+            0.007757347077131271,
+            -0.015726808458566666,
+            -0.007514929864555597,
+            -0.01545409020036459,
+            0.012885984033346176,
+            0.0032442226074635983,
+            0.005764981731772423,
+            0.015817714855074883,
+            0.021893294528126717,
+            0.01746918261051178,
+            -0.00795431062579155,
+            0.006852070800960064,
+            0.004674104508012533,
+            0.028090080246329308,
+            -0.01777220331132412,
+            0.020802417770028114,
+            0.009954252280294895,
+            -0.003935490269213915,
+            0.005685438401997089,
+            -0.014257155358791351,
+            0.008530051447451115,
+            -0.007325541693717241,
+            0.030514251440763474,
+            0.0033067206386476755,
+            0.014075342565774918,
+            -0.02062060497701168,
+            0.006575563922524452,
+            -0.03172633796930313,
+            0.009022461250424385,
+            0.013704141601920128,
+            0.012976890429854393,
+            -0.007677804213017225,
+            -0.024181105196475983,
+            -0.0119314668700099,
+            0.0057990713976323605,
+            -0.01279507763683796,
+            -0.01046181283891201,
+            -0.0034828518982976675,
+            0.012446602806448936,
+            0.018847929313778877,
+            -0.01848430372774601,
+            -0.019923655316233635,
+            -0.009583051316440105,
+            0.02898399345576763,
+            -0.015787413343787193,
+            -0.001471547526307404,
+            0.02413565292954445,
+            0.021241798996925354,
+            0.015022284351289272,
+            0.002155239460989833,
+            0.020090317353606224,
+            -0.0007111532613635063,
+            -0.0237265732139349,
+            -0.02813553437590599,
+            0.007302815094590187,
+            -0.006912675220519304,
+            0.01943882182240486,
+            -0.0035548193845897913,
+            -0.005658924113959074,
+            -0.0034033088013529778,
+            -0.007503566797822714,
+            -0.0165904201567173,
+            -0.01787826046347618,
+            0.0021647089160978794,
+            -0.008454295806586742,
+            0.005863463506102562,
+            -0.0002795845502987504,
+            -0.013105674646794796,
+            0.013401119969785213,
+            -0.014075342565774918,
+            0.005772557109594345,
+            -0.002653330797329545,
+            0.001962063368409872,
+            0.0035926972050219774,
+            0.009113367646932602,
+            -0.007336904760450125,
+            -0.06381630152463913,
+            -0.013105674646794796,
+            -0.003704436356201768,
+            0.006071790587157011,
+            -0.00019578021601773798,
+            -0.016635872423648834,
+            -0.013196581043303013,
+            0.01706010289490223,
+            0.008863375522196293,
+            -0.013870803639292717,
+            0.016954045742750168,
+            0.009961827658116817,
+            0.003622999181970954,
+            -0.024044746533036232,
+            -0.0008148433989845216,
+            0.008757317438721657,
+            0.032756611704826355,
+            -0.008643684908747673,
+            -0.0003839849086944014,
+            0.0036684523802250624,
+            -0.0033332351595163345,
+            -0.010052734054625034,
+            -0.014582903124392033,
+            -0.011848135851323605,
+            0.011908739805221558,
+            0.01790856383740902,
+            -0.014264730736613274,
+            -0.02007516659796238,
+            -0.0007106798002496362,
+            0.012931437231600285,
+            -0.026590125635266304,
+            0.0007244104635901749,
+            -0.007787649054080248,
+            -0.01052999310195446,
+            0.018302490934729576,
+            -0.019014591351151466,
+            0.005249845329672098,
+            0.00048294031876139343,
+            0.008507325313985348,
+            -0.023544760420918465,
+            -0.024893205612897873,
+            0.0035131541080772877,
+            -0.023847781121730804,
+            0.03612014651298523,
+            -0.021590271964669228,
+            -0.012082977220416069,
+            0.034483831375837326,
+            -0.0037423139438033104,
+            0.01702980138361454,
+            0.0023995505180209875,
+            0.02739313244819641,
+            -0.011439057067036629,
+            -0.00617784820497036,
+            0.012499631382524967,
+            0.009575475007295609,
+            0.0032593735959380865,
+            0.03775646165013313,
+            -0.01257538702338934,
+            -0.013901105150580406,
+            -0.005556654185056686,
+            -0.012954163365066051,
+            0.0006775368237867951,
+            0.014431392773985863,
+            0.028393102809786797,
+            0.004257450345903635,
+            -0.00628769351169467,
+            -0.006143758539110422,
+            -0.0014649189542979002,
+            -0.004242299124598503,
+            -0.00947699323296547,
+            -0.00715130427852273,
+            0.017287369817495346,
+            -0.025241680443286896,
+            0.016711628064513206,
+            -0.006420265417546034,
+            0.021135739982128143,
+            -0.011757229454815388,
+            0.007007369305938482,
+            -0.008961857296526432,
+            0.003492321353405714,
+            0.0023635667748749256,
+            -0.005109698045998812,
+            0.009287605062127113,
+            -0.014772292226552963,
+            0.008454295806586742,
+            0.013742019422352314,
+            0.0011183382011950016,
+            -0.010590597055852413,
+            -0.019272159785032272,
+            0.0022972808219492435,
+            -0.000710206339135766,
+            0.01181783340871334,
+            -0.0047498601488769054,
+            -0.000981031684204936,
+            0.0002921315317507833,
+            -0.014476845972239971,
+            -0.009575475007295609,
+            0.0006050958181731403,
+            0.005302873905748129,
+            -0.008961857296526432,
+            0.010378481820225716,
+            -0.005693014245480299,
+            -0.00034160923678427935,
+            0.02627195231616497,
+            0.006526322569698095,
+            -0.007632351014763117,
+            -0.014098069630563259,
+            0.01563590206205845,
+            0.015370759181678295,
+            -0.007454325910657644,
+            0.01933276280760765,
+            -0.016196493059396744,
+            -0.0008366230758838356,
+            0.013916256837546825,
+            -0.009219424799084663,
+            0.0017698341980576515,
+            -0.008196728304028511,
+            0.03554440662264824,
+            -0.0008787619881331921,
+            -0.0035415622405707836,
+            -0.0006401326390914619,
+            0.012976890429854393,
+            -0.0026779514737427235,
+            0.001182730309665203,
+            0.01132542360574007,
+            0.005321812815964222,
+            0.0185146052390337,
+            0.00328020635060966,
+            -0.03378688171505928,
+            -0.007780073676258326,
+            -0.0065490491688251495,
+            -0.008090670220553875,
+            0.009022461250424385,
+            0.025196226313710213,
+            0.01990850456058979,
+            0.006408901885151863,
+            0.001881573349237442,
+            0.018423698842525482,
+            0.015757111832499504,
+            -0.01489350013434887,
+            0.016363153234124184,
+            0.02802947722375393,
+            -0.010030006989836693,
+            0.022741753607988358,
+            -0.005253633018583059,
+            -0.008893677033483982,
+            0.008946705609560013,
+            -0.0015378334792330861,
+            -0.017393426969647408,
+            -0.015681356191635132,
+            0.005022579338401556,
+            -0.008999735116958618,
+            -0.0022897052112966776,
+            -0.018605511635541916,
+            0.004670316819101572,
+            0.017105557024478912,
+            -0.004965762607753277,
+            -3.5243989259470254e-05,
+            0.019484274089336395,
+            -0.006965703796595335,
+            -0.016908591613173485,
+            0.006325571332126856,
+            -0.015007133595645428,
+            0.007946735247969627,
+            -0.013666263781487942,
+            0.010492115281522274,
+            -0.006098304875195026,
+            0.018817627802491188,
+            0.005446809343993664,
+            -0.008090670220553875,
+            -0.0008958069374784827,
+            -0.018984289839863777,
+            0.02065090648829937,
+            -0.033089932054281235,
+            0.00839369185268879,
+            -0.026226500049233437,
+            -0.0009781909175217152,
+            0.0008849170990288258,
+            0.014605630189180374,
+            -0.001218714052811265,
+            0.019696388393640518,
+            -0.002751812804490328,
+            -0.020090317353606224,
+            -0.007086912170052528,
+            -0.0010634156642481685,
+            -0.01618134044110775,
+            -0.030923331156373024,
+            -0.004272601567208767,
+            0.0036267871037125587,
+            -0.004711982328444719,
+            0.0021135739516466856,
+            0.017984317615628242,
+            -0.005613470915704966,
+            0.009173971600830555,
+            -0.02690829709172249,
+            -0.007416448090225458,
+            -0.00355103169567883,
+            0.0010776197304949164,
+            -0.011242092587053776,
+            0.0010596278589218855,
+            0.011226941831409931,
+            -0.019044892862439156,
+            0.013461724855005741,
+            -0.021378157660365105,
+            -0.004371083341538906,
+            -0.008507325313985348,
+            -0.00135412672534585,
+            -0.013446573168039322,
+            -0.028120383620262146,
+            0.025029566138982773,
+            -0.00021495578403118998,
+            -0.003528305096551776,
+            -0.011999646201729774,
+            0.023181134834885597,
+            0.002914686920121312,
+            0.027817361056804657,
+            -0.03157482668757439,
+            0.0013427634257823229,
+            0.011552689597010612,
+            0.0025984083767980337,
+            -0.017287369817495346,
+            0.004746072459965944,
+            -0.005234694108366966,
+            0.02210540883243084,
+            -0.0011694730492308736,
+            -0.0005340751959010959,
+            -0.008234606124460697,
+            0.030999086797237396,
+            0.009136094711720943,
+            0.0012944694608449936,
+            -0.013226882554590702,
+            -0.008060368709266186,
+            -0.002759388415142894,
+            -0.0011069749016314745,
+            0.012855681590735912,
+            0.01290113478899002,
+            -0.012105703353881836,
+            0.0013607552973553538,
+            -0.006878585088998079,
+            0.0011694730492308736,
+            -0.002935519441962242,
+            -0.0004966709529981017,
+            -0.01149208564311266,
+            0.003257479751482606,
+            0.016272246837615967,
+            -0.00780280027538538,
+            -0.006480869371443987,
+            0.0007750718505121768,
+            0.009181547909975052,
+            0.012461753562092781,
+            -0.012742049060761929,
+            -0.018226735293865204,
+            -0.011908739805221558,
+            0.016029831022024155,
+            0.010522416792809963,
+            -0.008204303681850433,
+            -0.013234458863735199,
+            -0.02583257108926773,
+            0.01169662456959486,
+            -0.03293842449784279,
+            -0.012242063879966736,
+            0.005821797996759415,
+            0.028559764847159386,
+            0.019999410957098007,
+            0.012226912193000317,
+            0.001332347048446536,
+            0.002950670663267374,
+            0.00023342114582192153,
+            -0.01916610263288021,
+            0.014408666640520096,
+            -0.011143610812723637,
+            -0.018166130408644676,
+            0.0075906855054199696,
+            0.011939042247831821,
+            0.03148392215371132,
+            0.006942977197468281,
+            -0.0010160685051232576,
+            0.01686313934624195,
+            -0.001218714052811265,
+            -0.013582932762801647,
+            0.01420412678271532,
+            -0.024256860837340355,
+            0.016438908874988556,
+            0.02860521711409092,
+            -0.004780162125825882,
+            0.0008531945641152561,
+            -0.020938776433467865,
+            0.00482561532407999,
+            -0.011529963463544846,
+            -0.002177966060116887,
+            -0.003287781961262226,
+            -0.03460504114627838,
+            0.007946735247969627,
+            -0.006825556512922049,
+            0.01046181283891201,
+            -0.00814369972795248,
+            0.006655106786638498,
+            -0.012431452050805092,
+            0.004253662656992674,
+            0.01513591781258583,
+            0.016105586662888527,
+            0.007060397882014513,
+            -0.025211378931999207,
+            -0.009196698665618896,
+            0.008105821907520294,
+            -0.01349202636629343,
+            -0.02295386977493763,
+            -0.006852070800960064,
+            0.009840618818998337,
+            -0.01061332318931818,
+            -0.00488243205472827,
+            -0.028180986642837524,
+            -0.013681414537131786,
+            -0.0014327229000627995,
+            -0.017817657440900803,
+            -0.005556654185056686,
+            -0.003100287402048707,
+            0.01571165770292282,
+            -0.0049733384512364864,
+            -0.006185423582792282,
+            0.004742284305393696,
+            -0.023878084495663643,
+            0.0037423139438033104,
+            -0.008280059322714806,
+            0.0008266801596619189,
+            -0.0032517979852855206,
+            -0.0014762822538614273,
+            -0.007969462312757969,
+            -0.014332910999655724,
+            0.006893736310303211,
+            -0.016469212248921394,
+            0.002954458352178335,
+            -0.015173795633018017,
+            -0.016499513760209084,
+            -0.00682934420183301,
+            0.00775355938822031,
+            0.021575121209025383,
+            0.006541473791003227,
+            0.021484214812517166,
+            0.005469535943120718,
+            0.028680972754955292,
+            0.0003953482082579285,
+            0.003041577059775591,
+            -8.958956641436089e-06,
+            0.003232859307900071,
+            0.019347915425896645,
+            0.006571775767952204,
+            0.018984289839863777,
+            -0.008742166683077812,
+            0.010514841414988041,
+            -0.010355755686759949,
+            0.010014856234192848,
+            0.006333146709948778,
+            -0.006473293993622065,
+            -0.003799130441620946,
+            -0.010348179377615452,
+            -0.0031419529113918543,
+            -0.016544966027140617,
+            0.0027366618160158396,
+            0.00693161366507411,
+            0.0071588801220059395,
+            -0.011582992039620876,
+            -0.005230906419456005,
+            -0.014704111963510513,
+            -0.0064354161731898785,
+            -0.01831764169037342,
+            0.014385939575731754,
+            0.029226411134004593,
+            -0.016332851722836494,
+            -0.01977214403450489,
+            -0.007772498298436403,
+            0.0007296186522580683,
+            -0.006636167876422405,
+            -0.01240872498601675,
+            -0.009151245467364788,
+            -0.035059571266174316,
+            -0.017666146159172058,
+            0.007586897816509008,
+            0.02484775334596634,
+            0.01387837901711464,
+            -0.013628385961055756,
+            -0.0006983695784583688,
+            -0.016984347254037857,
+            -0.010787560604512691,
+            -0.01535560842603445,
+            -0.02054484933614731,
+            -0.02498411200940609,
+            0.0077952248975634575,
+            -0.004033972043544054,
+            0.014931377954781055,
+            0.008113397285342216,
+            0.0032631612848490477,
+            -0.004371083341538906,
+            -0.0024109138175845146,
+            -0.013196581043303013,
+            0.0012925754999741912,
+            -0.004787737503647804,
+            -0.0017291157273575664,
+            0.03554440662264824,
+            -0.008098246529698372,
+            -0.005871038883924484,
+            0.011060279794037342,
+            0.011128460057079792,
+            -0.013764745555818081,
+            0.0006642796797677875,
+            -0.011249667964875698,
+            0.021847840398550034,
+            -0.021620575338602066,
+            0.019257009029388428,
+            0.0014507147716358304,
+            -0.010840589180588722,
+            0.028999146074056625,
+            0.01892368495464325,
+            0.00743159931153059,
+            0.006579351611435413,
+            0.01672677882015705,
+            0.0044316877610981464,
+            0.016196493059396744,
+            -0.0026552248746156693,
+            -0.004632438998669386,
+            0.009295180439949036,
+            0.015681356191635132,
+            -0.000702630786690861,
+            -0.006942977197468281,
+            0.016438908874988556,
+            0.004056698642671108,
+            0.0075414446182549,
+            -0.01356020662933588,
+            -0.008961857296526432,
+            0.001507531269453466,
+            0.006257391534745693,
+            -0.005939218681305647,
+            -0.0026003022212535143,
+            0.005530139897018671,
+            0.003943065647035837,
+            -0.013302638195455074,
+            0.01689344085752964,
+            -0.016166189685463905,
+            0.008742166683077812,
+            -0.0037972365971654654,
+            0.0121360057964921,
+            -0.006799042224884033,
+            0.0013153020991012454,
+            -0.01818128302693367,
+            -0.003869204083457589,
+            0.0038597346283495426,
+            0.012173883616924286,
+            -0.01262841559946537,
+            0.025484098121523857,
+            0.008166425861418247,
+            0.006602078210562468,
+            0.0059467945247888565,
+            -0.012855681590735912,
+            0.027211319655179977,
+            0.005969521123915911,
+            0.009886072017252445,
+            -0.008893677033483982,
+            0.011136035434901714,
+            0.035362593829631805,
+            0.03060515783727169,
+            0.014135947450995445,
+            0.004469565115869045,
+            -0.018999440595507622,
+            0.015120766125619411,
+            0.0018427487229928374,
+            0.0023806116078048944,
+            0.00863610953092575,
+            -0.011348150670528412,
+            0.01305264513939619,
+            -0.009484568610787392,
+            0.015241974964737892,
+            -0.021272100508213043,
+            -0.011749654076993465,
+            -0.014408666640520096,
+            -0.021075136959552765,
+            -0.01430260855704546,
+            0.034180812537670135,
+            -0.002346521709114313,
+            -0.010393633507192135,
+            0.0045112306252121925,
+            0.008477022871375084,
+            -0.006079366430640221,
+            0.0053142374381423,
+            -0.008204303681850433,
+            -0.013749594800174236,
+            -0.01889338344335556,
+            0.006628592498600483,
+            -0.01550711877644062,
+            0.007651289459317923,
+            -0.003304826794192195,
+            0.0037688282318413258,
+            0.007014944683760405,
+            0.018105527386069298,
+            0.00530666159465909,
+            -0.00135412672534585,
+            0.0014033677289262414,
+            0.0016865032957866788,
+            -0.00931033119559288,
+            0.0108330138027668,
+            0.015408637002110481,
+            -0.0011921997647732496,
+            -0.011279970407485962,
+            -0.009136094711720943,
+            -0.042877525091171265,
+            0.0006264019757509232,
+            -0.013128400780260563,
+            0.0064354161731898785,
+            -0.011196639388799667,
+            0.0022234192583709955,
+            0.015863168984651566,
+            -0.0054051438346505165,
+            -0.0010918239131569862,
+            0.02193874679505825,
+            0.013014768250286579,
+            0.021075136959552765,
+            0.02065090648829937,
+            0.014188976027071476,
+            -0.015302578918635845,
+            -0.03469594568014145,
+            0.002221525413915515,
+            -0.010030006989836693,
+            0.013772320933640003,
+            0.004238511435687542,
+            0.016060132533311844,
+            -0.023893235251307487,
+            -0.011878437362611294,
+            -0.006662682630121708,
+            0.014431392773985863,
+            0.01491622719913721,
+            -0.0032271775417029858,
+            0.02690829709172249,
+            0.009280029684305191,
+            0.0020510759204626083,
+            -0.02281750924885273,
+            0.003312402404844761,
+            0.005321812815964222,
+            -0.008370965719223022,
+            -0.016681326553225517,
+            -0.0004552422615233809,
+            -0.0012168202083557844,
+            -0.0007817004225216806,
+            -0.021650876849889755,
+            0.018832778558135033,
+            -0.004018820822238922,
+            0.0037574649322777987,
+            0.013840501196682453,
+            0.01516621932387352,
+            0.017984317615628242,
+            0.008370965719223022,
+            0.001497115008533001,
+            -0.006605865899473429,
+            -0.0007973249885253608,
+            -0.0005828426801599562,
+            -0.007389933802187443,
+            -0.005598319694399834,
+            0.006382387597113848,
+            -0.01134057529270649,
+            0.008810346014797688,
+            0.000896753859706223,
+            0.006113456096500158,
+            0.010605747811496258,
+            -0.004836978856474161,
+            -0.011628445237874985,
+            0.014628356322646141,
+            -0.0010548931313678622,
+            -0.0055755930952727795,
+            -0.011060279794037342,
+            0.003624893259257078,
+            0.013575357384979725,
+            -0.003096499713137746,
+            -0.013499601744115353,
+            0.00910579226911068,
+            0.002009410411119461,
+            -0.027438584715127945,
+            0.015499543398618698,
+            0.010295150801539421,
+            -0.013370818458497524,
+            0.004980913829058409,
+            -0.003371112747117877,
+            -0.03027183562517166,
+            0.004791525658220053,
+            0.002660906407982111,
+            -0.010643625631928444,
+            0.028938541188836098,
+            -0.003821857040748,
+            0.029256712645292282,
+            -0.004980913829058409,
+            -0.006905099377036095,
+            -0.011355726048350334,
+            0.009416389279067516,
+            -0.018681267276406288,
+            -0.0037953427527099848,
+            -0.008727015927433968,
+            -0.00915882084518671,
+            -0.0009838725673034787,
+            -0.005185453221201897,
+            0.006984642706811428,
+            0.005121061112731695,
+            -0.012704171240329742,
+            0.006757376715540886,
+            -0.010257272981107235,
+            -0.00020856392802670598,
+            0.0006718551740050316,
+            -0.008408842608332634,
+            0.013885954394936562,
+            0.005996035411953926,
+            -0.008340663276612759,
+            -0.0005032995832152665,
+            0.027817361056804657,
+            0.05051366239786148,
+            -0.018863080069422722,
+            0.0009270560112781823,
+            0.01452229917049408,
+            -0.0028824908658862114,
+            -0.0021590273827314377,
+            -0.011976920068264008,
+            0.005821797996759415,
+            -0.019423669204115868,
+            0.013317788951098919,
+            0.006473293993622065,
+            -0.008833073079586029,
+            0.003202557098120451,
+            -0.016711628064513206,
+            0.002621134975925088,
+            -0.0018058179412037134,
+            -0.018681267276406288,
+            -0.015529844909906387,
+            -0.012060250155627728,
+            -0.024181105196475983,
+            0.0004235197266098112,
+            -0.0008546149474568665,
+            -0.013870803639292717,
+            0.006579351611435413,
+            0.008431569673120975,
+            0.032483890652656555,
+            0.0011912527261301875,
+            0.010128489695489407,
+            -2.1513333194889128e-05,
+            -0.0010416359873488545,
+            -0.005988460034132004,
+            -0.01523439958691597,
+            0.00015387803432531655,
+            -0.002342734020203352,
+            0.022590242326259613,
+            0.005549078807234764,
+            0.017560089007019997,
+            0.0017887729918584228,
+            -0.016514664515852928,
+            -0.00596573343500495,
+            -0.007408872712403536,
+            -0.018060073256492615,
+            0.0062649669125676155,
+            -0.005473323632031679,
+            0.009605777449905872,
+            -0.015863168984651566,
+            -0.02731737680733204,
+            0.00792400911450386,
+            0.012234487570822239,
+            0.028514310717582703,
+            -0.0035623949952423573,
+            0.001716805505566299,
+            -0.005533927585929632,
+            0.016408607363700867,
+            0.032180868089199066,
+            -0.0008550884085707366,
+            0.006950552575290203,
+            -0.019999410957098007,
+            0.0017404790269210935,
+            0.013704141601920128,
+            -0.008401267230510712,
+            -0.01821158453822136,
+            0.003939277958124876,
+            -0.020635755732655525,
+            0.004583198111504316,
+            -0.020393338054418564,
+            0.030408194288611412,
+            -0.0023522034753113985,
+            0.008105821907520294,
+            0.004310478921979666,
+            0.01760554127395153,
+            -0.0031646795105189085,
+            0.010545143857598305,
+            0.013916256837546825,
+            0.014908651821315289,
+            0.0009980766335502267,
+            -0.011189064010977745,
+            -0.03790797293186188,
+            0.001859793672338128,
+            0.00817400123924017,
+            0.018635815009474754,
+            -0.01980244740843773,
+            0.011681473813951015,
+            -0.01296931505203247,
+            -0.020363036543130875,
+            -0.01003758329898119,
+            0.011575416661798954,
+            -0.006139970384538174,
+            0.03378688171505928,
+            -0.005874827038496733,
+            -0.008204303681850433,
+            0.005136212334036827,
+            0.010454237461090088,
+            0.0066778333857655525,
+            0.014704111963510513,
+            0.0326354019343853,
+            0.010196669027209282,
+            0.014173824340105057,
+            -0.004859705455601215,
+            0.02399929240345955,
+            -0.003931702114641666,
+            0.003954428713768721,
+            0.00509454682469368,
+            -0.022832660004496574,
+            -0.0040983641520142555,
+            -0.003931702114641666,
+            -0.016711628064513206,
+            -0.007458113599568605,
+            0.0032366469968110323,
+            0.008802770636975765,
+            -0.014045040123164654,
+            -0.00649980828166008,
+            -0.01350717805325985,
+            0.01218903437256813,
+            0.002730980049818754,
+            0.018302490934729576,
+            -0.029802151024341583,
+            0.009825468063354492,
+            -0.017484333366155624,
+            -0.017044952139258385,
+            0.006511171814054251,
+            -0.0038332203403115273,
+            0.006855858489871025,
+            -0.018272189423441887,
+            -0.01318900566548109,
+            0.012817803770303726,
+            0.0007897494360804558,
+            0.013060221448540688,
+            -0.009825468063354492,
+            0.0009706153068691492,
+            0.006064215209335089,
+            -0.009833043441176414,
+            -0.010961798019707203,
+            -0.007977037690579891,
+            -0.009681533090770245,
+            -0.008848223835229874,
+            0.005621046293526888,
+            0.01656011864542961,
+            -0.013742019422352314,
+            -0.003685497445985675,
+            0.016135888174176216,
+            0.006416477728635073,
+            -0.008166425861418247,
+            0.004075637552887201,
+            -0.003793448908254504,
+            -9.948806837201118e-05,
+            0.0013740125577896833,
+            0.012439027428627014,
+            -0.0017632056260481477,
+            0.024665940552949905,
+            -0.02654467150568962,
+            0.007598260883241892,
+            -0.0054051438346505165,
+            0.005446809343993664,
+            0.018226735293865204,
+            -0.02366596832871437,
+            -0.013082947582006454,
+            0.006276329979300499,
+            -0.01966608688235283,
+            -0.008060368709266186,
+            0.0054051438346505165,
+            -0.00016405766655225307,
+            -0.019923655316233635,
+            -0.012923861853778362,
+            -0.034786853939294815,
+            -0.02309022843837738,
+            0.024665940552949905,
+            0.008401267230510712,
+            -0.0073672072030603886,
+            0.018060073256492615,
+            -0.018438849598169327,
+            0.0172722190618515,
+            -0.002225313102826476,
+            -0.006613441277295351,
+            0.003935490269213915,
+            0.007674016058444977,
+            0.025362888351082802,
+            0.002653330797329545,
+            0.008264907635748386,
+            -0.0042082094587385654,
+            -0.014931377954781055,
+            -0.006321783177554607,
+            0.01980244740843773,
+            0.02471139281988144,
+            0.018605511635541916,
+            -0.018575210124254227,
+            0.010742107406258583,
+            -0.011749654076993465,
+            0.01818128302693367,
+            -0.006537686102092266,
+            0.00039037675014697015,
+            0.01859036087989807,
+            0.007969462312757969,
+            0.025241680443286896,
+            0.019923655316233635,
+            0.006852070800960064,
+            0.014128371141850948,
+            0.011908739805221558,
+            0.001204509986564517,
+            -0.012832955457270145,
+            -0.01154511421918869,
+            -0.02840825356543064,
+            0.042847223579883575,
+            0.0025434857234358788,
+            0.008317936211824417,
+            -0.027635548263788223,
+            -0.00021389048197306693,
+            -0.0034942151978611946,
+            -0.018832778558135033,
+            0.005939218681305647,
+            0.03260510042309761,
+            0.03284751623868942,
+            0.021438762545585632,
+            -2.888172457460314e-05,
+            0.0012234487803652883,
+            0.0329081192612648,
+            -0.013673839159309864,
+            -0.00937851145863533,
+            -0.01034060399979353,
+            -0.018393397331237793,
+            -0.006852070800960064,
+            -0.0032423287630081177,
+            -0.01031030248850584,
+            -0.0061172437854111195,
+            -0.009272454306483269,
+            0.0238629337400198,
+            0.0001374249259242788,
+            -0.0022594030015170574,
+            0.006605865899473429,
+            -0.020878173410892487,
+            0.011537538841366768,
+            -0.02813553437590599,
+            -0.013431422412395477,
+            -0.008651260286569595,
+            -0.006950552575290203,
+            -0.018802475184202194,
+            0.008643684908747673,
+            -0.006942977197468281,
+            0.021438762545585632,
+            0.002914686920121312,
+            -0.023590214550495148,
+            0.007530081085860729,
+            0.015317730605602264,
+            -0.0003825644962489605,
+            0.014037464745342731,
+            -0.003890036838129163,
+            0.009060339070856571,
+            0.0045112306252121925,
+            0.011681473813951015,
+            0.016954045742750168,
+            0.010651201009750366,
+            0.009030036628246307,
+            0.003437398700043559,
+            -0.001167579204775393,
+            0.01878732442855835,
+            -0.0007163614500313997,
+            -0.03405960276722908,
+            2.9236827685963362e-05,
+            -0.014628356322646141,
+            0.029499130323529243,
+            -0.009795165620744228,
+            0.009514871053397655,
+            0.008105821907520294,
+            0.007098275702446699,
+            -0.0067119235172867775,
+            0.007916433736681938,
+            -0.008742166683077812,
+            0.0052763596177101135,
+            0.015832867473363876,
+            -0.0042877523228526115,
+            -0.0014980619307607412,
+            0.007840678095817566,
+            0.00407184986397624,
+            -0.005454384721815586,
+            -0.019711541011929512,
+            -0.012287517078220844,
+            0.00726114958524704,
+            -0.02065090648829937,
+            -0.004367295652627945,
+            0.01161329448223114,
+            -0.005454384721815586,
+            -0.009620928205549717,
+            0.01393140759319067,
+            0.02136300690472126,
+            -0.002382505452260375,
+            0.022893264889717102,
+            -0.01349202636629343,
+            -0.009833043441176414,
+            -0.007984613068401814,
+            -0.004374871030449867,
+            -0.041120000183582306,
+            0.025984082370996475,
+            0.01068150345236063,
+            -0.006871009711176157,
+            0.01895398646593094,
+            0.0022499337792396545,
+            -0.002954458352178335,
+            -0.0011041341349482536,
+            0.00010499217023607343,
+            -0.03281721472740173,
+            0.03660498186945915,
+            -0.007007369305938482,
+            0.0005449649761430919,
+            0.010272424668073654,
+            0.042029064148664474,
+            -0.004590773954987526,
+            -0.005859675817191601,
+            0.01225721463561058,
+            0.005336964037269354,
+            0.031938452273607254,
+            -0.010416359640657902,
+            -0.014340486377477646,
+            -0.01815097965300083,
+            -0.004833190701901913,
+            -0.01939336769282818,
+            -0.0038313264958560467,
+            0.014037464745342731,
+            -0.019650936126708984,
+            -0.011242092587053776,
+            -0.001287840772420168,
+            -0.012537509202957153,
+            -0.0010747789638116956,
+            0.008598231710493565,
+            0.017832808196544647,
+            -0.011348150670528412,
+            -0.004450626205652952,
+            -2.2904154320713133e-05,
+            -0.005003640428185463,
+            0.0185146052390337,
+            0.0020056227222085,
+            -0.01702980138361454,
+            0.01321930717676878,
+            -0.00526499655097723,
+            0.006749800872057676,
+            0.0074429623782634735,
+            0.007939159870147705,
+            0.05145302787423134,
+            0.010317877866327763,
+            -0.001339922659099102,
+            -0.01607528328895569,
+            0.023196285590529442,
+            -0.01749948412179947,
+            -0.014181400649249554,
+            -0.02119634486734867,
+            -0.0029923359397798777,
+            -0.007060397882014513,
+            -0.028014326468110085,
+            -9.137987944995984e-05,
+            0.0016221113037317991,
+            -0.0024980325251817703,
+            -0.010045158676803112,
+            0.0005208179936744273,
+            -0.011529963463544846,
+            -0.006954340264201164,
+            -0.008545203134417534,
+            -0.016044981777668,
+            0.02240842953324318,
+            -0.021817538887262344,
+            -0.008855799213051796,
+            0.011514811776578426,
+            0.03118089959025383,
+            -0.005011215806007385,
+            -0.014173824340105057,
+            -0.019817598164081573,
+            0.0025643182452768087,
+            -0.01763584464788437,
+            0.012181458994746208,
+            -0.01225721463561058,
+            0.015045011416077614,
+            -0.03142331540584564,
+            -0.006079366430640221,
+            -0.008992158807814121,
+            0.012840530835092068,
+            0.019817598164081573,
+            -0.0025093958247452974,
+            0.00606800289824605,
+            -0.010295150801539421,
+            0.030211230739951134,
+            -0.00012286569108255208,
+            -0.018605511635541916,
+            0.00873459130525589,
+            0.007977037690579891,
+            0.012537509202957153,
+            -0.004435475450009108,
+            0.00257757562212646,
+            0.011567841283977032,
+            0.012742049060761929,
+            -0.01634800247848034,
+            -0.003174148965626955,
+            -0.017969166859984398,
+            0.005996035411953926,
+            0.006999793462455273,
+            0.021711481735110283,
+            -0.0013550736475735903,
+            0.018272189423441887,
+            -0.01563590206205845,
+            0.00019412307301536202,
+            -0.011749654076993465,
+            -0.0082270298153162,
+            -0.0028237802907824516,
+            -0.003958216868340969,
+            0.006685409229248762,
+            -0.028211290016770363,
+            -0.01274962443858385,
+            0.005287723150104284,
+            -0.004992276895791292,
+            -0.03405960276722908,
+            0.0053710537031292915,
+            -0.02725677192211151,
+            0.013128400780260563,
+            -0.002397656673565507,
+            -0.017847958952188492,
+            -0.00030846628942526877,
+            0.026332557201385498,
+            -0.003677921835333109,
+            -0.012537509202957153,
+            0.015544996596872807,
+            -0.006757376715540886,
+            -0.02119634486734867,
+            -0.005420294590294361,
+            0.014658658765256405,
+            -0.0010719380807131529,
+            -0.012393574230372906,
+            -0.029635490849614143,
+            0.012992041185498238,
+            0.0026874206960201263,
+            -0.007386146113276482,
+            0.010893618687987328,
+            0.025953780859708786,
+            0.008151275105774403,
+            -0.005924067925661802,
+            0.0015757111832499504,
+            0.0049581872299313545,
+            0.00300938100554049,
+            0.020287280902266502,
+            0.02728707529604435,
+            -0.008014915511012077,
+            -0.0011145505122840405,
+            -0.014954105019569397,
+            0.015802564099431038,
+            0.0013020449550822377,
+            0.010507266037166119,
+            0.021893294528126717,
+            0.007045246660709381,
+            -0.005738467443734407,
+            0.003905187826603651,
+            -0.023135682567954063,
+            0.01420412678271532,
+            -0.012272365391254425,
+            0.000622140767518431,
+            0.008749742060899734,
+            -0.007283876184374094,
+            0.01437078882008791,
+            -0.0015245762187987566,
+            0.0339989997446537,
+            0.025211378931999207,
+            0.0043218424543738365,
+            0.0007362472242675722,
+            -0.028559764847159386,
+            0.003547244006767869,
+            0.008651260286569595,
+            0.014499573037028313,
+            -0.0076664406806230545,
+            0.006261179223656654,
+            -0.004355932120233774,
+            0.00959820207208395,
+            -0.0065490491688251495,
+            -0.015893470495939255,
+            0.006113456096500158,
+            0.0012689019786193967,
+            0.006795254070311785,
+            0.0023408401757478714,
+            -0.008863375522196293,
+            -0.024362917989492416,
+            -0.025984082370996475,
+            0.021984200924634933,
+            -0.03918066248297691,
+            0.0073293293826282024,
+            0.011234517209231853,
+            0.03324144333600998,
+            -0.008158850483596325,
+            -0.004980913829058409,
+            0.006276329979300499,
+            -0.016711628064513206,
+            -0.004147605039179325,
+            0.008204303681850433,
+            0.012385998852550983,
+            -0.002420383272692561,
+            0.00974971242249012,
+            0.01318900566548109,
+            -0.012378423474729061,
+            0.03660498186945915,
+            0.016514664515852928,
+            0.01467380952090025,
+            -0.004007457755506039,
+            0.01312082540243864,
+            0.0066172294318675995,
+            0.019044892862439156,
+            0.025165924802422523,
+            -0.017454031854867935,
+            -0.002365460619330406,
+            -0.02292356640100479,
+            0.024165954440832138,
+            0.00726114958524704,
+            0.004556683823466301,
+            -0.008310360834002495,
+            0.014870774000883102,
+            0.0017954016802832484,
+            0.0028199926018714905,
+            -0.006696772295981646,
+            -0.010173942893743515,
+            0.019241856411099434,
+            0.0034317169338464737,
+            0.014870774000883102,
+            4.8501173296244815e-05,
+            -0.005503625608980656,
+            0.006052851676940918,
+            0.014832896180450916,
+            -0.023590214550495148,
+            -0.00203592493198812,
+            -0.019484274089336395,
+            -0.03202936053276062,
+            0.009022461250424385,
+            0.0016135887708514929,
+            -0.007052822504192591,
+            0.03230207785964012,
+            -0.010083036497235298,
+            -0.019787294790148735,
+            -0.008067944087088108,
+            -0.013113250024616718,
+            -0.010469388216733932,
+            0.0008271536789834499,
+            0.003908975515514612,
+            -0.00012428610352799296,
+            -0.009317907504737377,
+            -0.005749830510467291,
+            0.01990850456058979,
+            -0.0030358952935785055,
+            -0.00753386877477169,
+            0.00868156272917986
+          ],
+          "embedding": [
+            0.0040707582550429606,
+            -0.010230625151989194,
+            -0.021520242101903186,
+            -0.010115756270640508,
+            0.026738734586159035,
+            0.01796370233223267,
+            -0.012245199029994153,
+            0.06506924494147726,
+            -0.0014087587222033442,
+            0.009300068719271429,
+            0.003787396973047584,
+            0.002818772639754174,
+            0.022129077062176306,
+            -0.021526709808445683,
+            -0.0034388135367225824,
+            0.02763761747879405,
+            0.013210013805395276,
+            0.025255872672296602,
+            0.00029011480909986703,
+            -0.05438974762290585,
+            0.016671640187490865,
+            -0.016426252259893328,
+            -0.056591180651012554,
+            0.009335648425523075,
+            0.006141624459418827,
+            0.02568943956694641,
+            0.019377132054042116,
+            0.017946464337078746,
+            -0.057184776985927686,
+            0.03208527139096687,
+            0.030132392063755547,
+            -0.003846032371791839,
+            -0.008376524427531415,
+            -0.0010513963058713435,
+            -0.02734868388724371,
+            -0.014879400370545479,
+            0.04383920959748825,
+            0.02563613659516289,
+            -0.020117467074438997,
+            0.0398016699155716,
+            0.029896497884197775,
+            -0.0004195844530067313,
+            -0.023302720788253625,
+            -0.001168337916577517,
+            0.009149021369661628,
+            -0.02508145721900992,
+            -0.004307594138480351,
+            0.00550940415564303,
+            0.014016998918718476,
+            -0.028607379616884317,
+            -0.0018317656587526954,
+            -0.01973464940415756,
+            0.03762738392056255,
+            0.015358106058583114,
+            -0.03931907682104936,
+            0.011519637298832444,
+            0.004482080558226775,
+            0.0747867063383242,
+            0.05550679018263077,
+            0.023243129601931176,
+            0.02671631686665238,
+            -0.009686146566573536,
+            -0.04088582189725495,
+            -0.0065344623594924085,
+            0.019617295999325712,
+            -0.004722087402575777,
+            -0.002871844114437583,
+            0.003935087265300519,
+            0.03229890413393201,
+            -0.02813577047904649,
+            -0.05124172132597234,
+            0.03325235407841968,
+            -0.008687655557217192,
+            -0.011027174838246948,
+            -0.003653553063547375,
+            -0.02350737945182988,
+            0.016223087939183468,
+            0.03789252356462714,
+            0.0034350623378603083,
+            0.00195158657888739,
+            0.011789712862232108,
+            -0.0004435439660769551,
+            -0.0037330456918125894,
+            0.0031552713243866858,
+            0.04465432992298031,
+            -0.0032465421571915712,
+            -0.0075366743861922745,
+            -0.059958752731410256,
+            0.016784616861008125,
+            -0.028734506065772967,
+            -0.03035575736273943,
+            0.02680390892611741,
+            0.02338368719038086,
+            -0.0030658375437784277,
+            -0.03754896050794851,
+            0.07005659523248504,
+            0.0020711575553570695,
+            0.013034425083076923,
+            -0.02316832262132332,
+            0.021332142030732894,
+            -0.0318756233020585,
+            0.02270782509507388,
+            0.01570810309454529,
+            0.004788124787612736,
+            0.026727250200142456,
+            -0.022380142778058276,
+            -0.008593910379436904,
+            0.014439754382387034,
+            0.015007697617383174,
+            0.07662531963612977,
+            0.011195998137723795,
+            -0.010809283538470098,
+            -0.042227822120219305,
+            0.003824320933275175,
+            -0.03797629427607094,
+            -0.02347390684453348,
+            -0.0526876440280276,
+            -0.017799861680287913,
+            0.01701482796266917,
+            0.002730395144917155,
+            0.03789451759899825,
+            0.038338545350313946,
+            0.028463789278406586,
+            0.015494583705011921,
+            -0.05700237266350479,
+            0.047895170359075166,
+            -0.0064406894534929406,
+            -0.0007954721295065988,
+            0.012543029839729358,
+            -0.004703334868741216,
+            -0.0060569679284068,
+            0.012324901254711192,
+            -0.019883935809784068,
+            -0.04253565942154215,
+            0.02164409276211115,
+            -0.0075367555415808635,
+            0.03617159147722157,
+            0.07567745988686005,
+            -0.04225977020036783,
+            -0.022548837589622003,
+            0.029811602145850144,
+            -0.02175792196819533,
+            0.02069861458115363,
+            -0.00941331829664397,
+            -0.035479639885943264,
+            0.020352552580119105,
+            0.003678588588550461,
+            0.06776105274336729,
+            -0.02346047721661687,
+            0.02721075076349443,
+            -0.029386781707644253,
+            0.023530831921117993,
+            -0.02077660533100372,
+            0.020022475323372906,
+            0.024195118676513715,
+            -0.028378518720494666,
+            -0.012900969023913307,
+            0.019264089970719624,
+            -0.0374983624629265,
+            -0.019726171557384355,
+            0.00979636085642416,
+            -0.014479528435832063,
+            -0.014533384583777662,
+            0.005904623758767224,
+            0.010344475500263744,
+            0.05659646358026115,
+            0.04163866113482093,
+            0.007152494819169405,
+            -0.07584500130974763,
+            0.05471858597878099,
+            0.03580211013427615,
+            0.044161429444475306,
+            0.02258180566046248,
+            0.00980779221334715,
+            0.0011193505603397395,
+            0.018086566768892507,
+            0.015605066702191871,
+            -0.00954727399396313,
+            -0.027950079146919375,
+            0.03028514025485407,
+            -0.006351669761257431,
+            0.005095804801686436,
+            0.009620040085743162,
+            0.019748001227987134,
+            0.033709535059510916,
+            0.008203559956940196,
+            0.007389752602218563,
+            0.04197771878351742,
+            -0.02067363758471765,
+            0.007543739188471913,
+            0.011207941261799052,
+            0.013964827251172813,
+            0.03839562075248879,
+            0.0009775244543292637,
+            0.006469507611492216,
+            -0.022808235395746654,
+            -0.024795004398633292,
+            0.006249024864814684,
+            0.011292877109066925,
+            0.01596839619826819,
+            -0.0014105396434174883,
+            -0.04152075438277094,
+            0.008267788411079337,
+            -0.028834191212891387,
+            0.030940526405027906,
+            -0.0018271056434014425,
+            0.006185475047820603,
+            -0.007920574674727123,
+            0.012907560753043616,
+            -0.015165704767187252,
+            -0.021962244496450727,
+            0.0037071053072833147,
+            -0.013508376064336042,
+            -0.015178823540513864,
+            -0.02518557053061311,
+            0.011693256916760944,
+            -0.01716015674592331,
+            0.02048897513447166,
+            -0.003412078966361745,
+            -0.001807298511378529,
+            -0.0012575655363779904,
+            0.03116103961636643,
+            0.011668533361641842,
+            -0.011637010548030312,
+            -0.040260195687925236,
+            -0.011129376691522074,
+            -0.0034493180842871908,
+            -0.02098244789711689,
+            0.026658743116016938,
+            -0.01923736325953796,
+            -0.009723937181300192,
+            -0.013134679989565051,
+            -0.011974548914818263,
+            -0.004028712301052159,
+            0.019628329175867303,
+            -0.028192034966932684,
+            -0.012638520793012082,
+            -0.011568652932013111,
+            -0.0025919910519445737,
+            0.012497908463405111,
+            0.015113186712897145,
+            0.014880607940212641,
+            -0.03059227859452334,
+            -0.021899716217105084,
+            -0.0008513506358489523,
+            0.011985931118464679,
+            -0.01321584595585435,
+            -0.011735288366213981,
+            0.010710683541435907,
+            0.01961722172525993,
+            -0.023571946726192585,
+            -0.03364281880125354,
+            0.0014460076743756465,
+            0.016563167828467187,
+            -0.02891589059260944,
+            0.04595166331401113,
+            0.034271553243501465,
+            -0.027557971577291153,
+            -0.007138821520412525,
+            -0.039508407807651844,
+            0.06814329636830853,
+            0.00022136123564675344,
+            -0.039718016705753735,
+            0.0503457594963753,
+            -0.003580581891576583,
+            0.013820796398521455,
+            0.02054050419088757,
+            0.035682661890720495,
+            -0.0013220739026852943,
+            -0.0052759394697328675,
+            0.04733018719162478,
+            -0.0187845855131196,
+            -0.03982699468608443,
+            -0.023905460465070062,
+            0.002631233133039351,
+            -0.03090995554989513,
+            -0.012617612051488022,
+            -0.013605787969473513,
+            0.039672783793764745,
+            0.008669582242097126,
+            0.009974277556736362,
+            0.005535636914843583,
+            0.014180668493093584,
+            0.012042659947850441,
+            0.011870115226306776,
+            0.00451259454348581,
+            -0.014023048592193627,
+            -0.007769033996392623,
+            -0.05715987527668358,
+            0.003374564518384728,
+            -0.028801158523506985,
+            -0.00756331369170698,
+            -0.02163054894055572,
+            0.005367181573233089,
+            0.02308705505078254,
+            -0.005438909398653553,
+            -0.00873620909426313,
+            -0.0036530652237719077,
+            0.013920407174296543,
+            -0.01539613379172909,
+            -0.03908211213269472,
+            -0.016810726548844992,
+            0.007460173854386443,
+            -0.007069189426742819,
+            -0.041197222498539215,
+            -0.003843233633335066,
+            0.004373367480223065,
+            0.016194121660459908,
+            -0.029165150066068787,
+            -0.00735139702658342,
+            0.017257895082337448,
+            0.014153313280536095,
+            -0.002376178042741926,
+            -0.023223700836427403,
+            -0.0319877537205969,
+            0.02088060774052704,
+            0.0012829290529515294,
+            0.03251550040316258,
+            -0.04128453069131847,
+            -0.023376114412063578,
+            0.04083022735076889,
+            0.005690491480259003,
+            -0.05810403800689653,
+            0.016582434816685452,
+            0.010402038964417569,
+            0.016795644054313566,
+            -0.00908590043098552,
+            0.05003656634510042,
+            -0.037216197954986735,
+            -0.013027960261454162,
+            -0.007090943677204207,
+            -0.05175371787508879,
+            -0.0020065660448063322,
+            -0.014908489552053589,
+            0.010627472758306387,
+            -0.03275473161158777,
+            -0.017334608434133866,
+            0.025321047439048318,
+            -0.0033515195870701753,
+            0.011526068468647207,
+            -0.026783879816504975,
+            -0.0012830935419413407,
+            -0.017506908439998053,
+            -0.02114853058440477,
+            0.004878884893669383,
+            -0.020936900837115968,
+            -0.0025587027781733642,
+            0.01557850246191957,
+            0.023652335794299834,
+            0.011489428313303988,
+            0.0019089137782552768,
+            0.02054907941022243,
+            0.014361232983780595,
+            -0.0056413109308506085,
+            -0.0010300050480036684,
+            -0.03413262505887213,
+            -0.018821160485010483,
+            0.03361989907065425,
+            0.013125889701529858,
+            -0.003742167496379789,
+            0.008813100942466033,
+            -0.0383265293638935,
+            -0.035316957263713485,
+            0.03020499323039515,
+            -0.05001623971243151,
+            -0.00855479667673597,
+            -0.005939674940109814,
+            -0.026451831943555135,
+            -0.008032873049527808,
+            0.0009431571685654594,
+            -0.014387605128858954,
+            0.016382559541755014,
+            -0.024909510232038502,
+            0.003715736505444385,
+            -0.006016712902409688,
+            -0.009110197692342108,
+            0.05507263160354303,
+            -0.022103031032598353,
+            -0.0475415595313825,
+            0.021958434235402655,
+            -0.016670809659931243,
+            0.0323519363801999,
+            0.03867494951405658,
+            0.011124508964363016,
+            -0.008002445714431074,
+            0.034041838454818577,
+            0.018400123235267234,
+            -0.01161754789471199,
+            -0.025901844576206774,
+            0.021800298132870245,
+            0.00943488425697727,
+            -0.05906691123008531,
+            -0.00782549274387971,
+            -0.019357446956797447,
+            0.008682376344965976,
+            0.02863077694455766,
+            0.0024925977373894447,
+            -0.002682170968419412,
+            -0.013590235283325244,
+            -0.021511149507762284,
+            -0.03255453916518719,
+            0.009039307015853905,
+            0.008305286476500875,
+            0.009312625027668458,
+            -0.0014144165766983945,
+            -0.007853869334376857,
+            0.009067065760942163,
+            0.022485201611182105,
+            0.03966189080765722,
+            -0.041627289519629344,
+            0.002707086456239785,
+            -0.022678178050044222,
+            -0.03653753142622053,
+            0.0036574646778835588,
+            0.029996160786576367,
+            0.02901163765559891,
+            -0.01971777193183787,
+            -0.0036589934841109884,
+            0.009181234938554578,
+            -0.025979483002227535,
+            -0.00773645103995767,
+            0.004280770859737866,
+            0.04116668301017644,
+            0.006201085381597264,
+            0.01402728258112097,
+            0.001060407317110527,
+            -0.00920801784000087,
+            0.02109880889040528,
+            0.025211022083982727,
+            -0.029866678264942617,
+            0.014439702646820226,
+            0.009379206296908577,
+            0.04156435812161122,
+            -0.037309655469525846,
+            -0.011819731348299495,
+            -0.02766498020409985,
+            -0.010454143230794476,
+            0.027911532296566614,
+            -0.04127890742215716,
+            -0.006948536183086887,
+            0.010207784113909938,
+            -0.0016584043356136647,
+            0.016437495428919636,
+            -0.014607861750686952,
+            -0.03222311919805017,
+            -0.000610723203118213,
+            -0.02867445198803213,
+            -0.024772816792038133,
+            -0.016510309753766612,
+            -0.018581201775667577,
+            -0.07450654873867177,
+            0.00026832248829154255,
+            -0.06724696990816072,
+            -0.02447002541264633,
+            0.04885569249258513,
+            -0.010195604207134053,
+            0.03332905935704749,
+            -0.02815272112271314,
+            -0.005296340304911926,
+            -0.05908835042526019,
+            0.0038476172058970774,
+            0.025753576863447866,
+            0.02386003857017594,
+            0.005206447371391696,
+            0.024799057871126032,
+            0.012060738268265606,
+            -0.04146503002980051,
+            -0.024949907556862643,
+            -0.014503475223770958,
+            -0.0016340466065744286,
+            -0.0034009167013795906,
+            -0.031298432995411005,
+            0.036241272149880994,
+            0.008902644391348814,
+            0.03550223922667804,
+            0.04463301409716232,
+            -0.010385984413759852,
+            -0.03305004479734889,
+            -0.006952739338820577,
+            -0.0175981588092108,
+            0.042668533202572856,
+            -0.017112184620086308,
+            0.005149173014526686,
+            -0.03219589107997635,
+            0.024181808150698054,
+            0.0014339790422945406,
+            0.01254721190611376,
+            0.011099458649404119,
+            0.0027626836767364253,
+            -0.026883640446090158,
+            0.0024160770766463607,
+            0.023874514847956222,
+            -0.022558433617669373,
+            0.00917667715473093,
+            -0.006754091166465801,
+            -0.033686935718776144,
+            -0.01078389638118685,
+            -0.013595327486167114,
+            0.024712781935748277,
+            0.013168464807451798,
+            0.005875864604862434,
+            -0.02417814420558463,
+            -0.0023978800459714144,
+            0.028650463366555304,
+            0.020176980768301288,
+            -0.02118184201861616,
+            0.019082340279671355,
+            0.019734925740100277,
+            -0.04209645071102001,
+            -0.04217310001316092,
+            0.006553913719098956,
+            -0.003735719227468833,
+            0.0061793909220459,
+            -0.028285889794908054,
+            0.008181788076484304,
+            0.004432488437404309,
+            -0.01743905248466261,
+            0.041221269621866216,
+            -0.014892354419172107,
+            0.005087102208335201,
+            -0.022198383716186407,
+            -0.011936388003605221,
+            -0.00027487042024290816,
+            0.044599215801158845,
+            0.008068975171827777,
+            0.003496950921841073,
+            0.02238779276447619,
+            0.0054839808262270315,
+            0.04816723616939378,
+            -0.02685221260466647,
+            -0.0190356946337995,
+            0.016464136459560113,
+            -0.0176092780782677,
+            -0.012513274095432353,
+            -0.010695421445334002,
+            -0.004784348908457072,
+            -0.022203041334932165,
+            0.010788155380415583,
+            -0.018353897272137543,
+            -0.02277431189907027,
+            -0.003945465850796918,
+            0.016024639792926797,
+            -0.006248368166794934,
+            -0.0038769803892794875,
+            0.0016292556540434277,
+            0.02412864830338344,
+            0.02892955453819876,
+            -0.02408022470555968,
+            0.019190282372105023,
+            0.018941584034503536,
+            -0.006709516380602965,
+            -0.020956377933747532,
+            -0.01964885206488053,
+            0.01434623424964373,
+            -0.01352968586212152,
+            -0.013872556892707697,
+            0.004052798542613835,
+            0.009201147374299997,
+            -0.004086067985859617,
+            -0.04185829685685201,
+            -0.014668698123513235,
+            0.003179403563664022,
+            -0.0006415014901450319,
+            0.019360184344240947,
+            -0.0150817538567364,
+            -0.029771999278536083,
+            0.014221776097046751,
+            0.022944884879589678,
+            -0.04493425220030206,
+            0.02117577719533236,
+            -0.0030648751497261983,
+            -0.00016251533643103604,
+            -3.817296018601414e-05,
+            0.008727767382077138,
+            0.026914779443266063,
+            0.03456565094655899,
+            0.01974806772696219,
+            0.00450800007152441,
+            -0.008952599700759673,
+            0.025184190461155245,
+            0.009846044083064704,
+            -0.025168735183134738,
+            0.010983353565382029,
+            0.016740721743326522,
+            0.0070838930012654296,
+            0.009173879407277282,
+            -0.006078288608766152,
+            -0.018269347893097643,
+            0.013181494188066667,
+            -0.01435693747022231,
+            0.01678907949532146,
+            0.05154463552784974,
+            -0.00039632596424461964,
+            -0.02461439550772063,
+            0.008418147037164186,
+            -0.006544319842335593,
+            -0.010065129286193278,
+            0.02214628550460128,
+            0.0029515574751370417,
+            -0.030862152033847717,
+            0.007509556025223583,
+            0.00675485304374519,
+            0.01894936231559203,
+            -0.02355525295639599,
+            -0.01387885858544803,
+            -0.00431398020087544,
+            0.0021208056468763523,
+            0.01580766504149989,
+            -0.004739477121073512,
+            -0.013204823447960706,
+            0.02994098229517376,
+            0.025496728234662373,
+            -0.00026124946738213036,
+            0.015060064980768175,
+            0.022262345898533903,
+            0.009968408664790368,
+            0.007588431141088797,
+            -0.0083063413609779,
+            -0.031336272171937095,
+            0.0018957858383561667,
+            0.05800811417728381,
+            -0.014937085296865802,
+            0.04313875432235027,
+            0.02018476425923236,
+            0.015657644644987107,
+            0.017880263053253607,
+            -0.01662268082870562,
+            -0.008740053682875323,
+            -0.0017630394204179638,
+            -0.008294417990024647,
+            -0.027613464270562948,
+            -0.005953209556351086,
+            -0.02379961210881074,
+            -0.013878676554701543,
+            0.015418315245979704,
+            -0.014764728065382074,
+            0.0027420640870858457,
+            0.010395538988699663,
+            -0.006590633833835581,
+            -0.00841537209927409,
+            -0.003943772700797381,
+            0.03376338017637296,
+            0.011980354721458463,
+            0.027761929822019422,
+            -0.00011128539412582973,
+            0.009342430952107655,
+            -0.018419526564396063,
+            -0.006839752805318664,
+            0.03271190887864607,
+            0.006011520427944665,
+            0.004652835471939905,
+            0.01485920920301398,
+            0.014305626381398871,
+            0.038015677219487023,
+            -0.005102005401565164,
+            -0.018204460895989052,
+            0.02065495133597456,
+            -0.0049699013356374775,
+            0.019149036693562712,
+            0.008825981577388056,
+            -0.002660055848523568,
+            -0.004292379561953203,
+            0.03090270710860331,
+            -0.020243440437217503,
+            0.018735727616321387,
+            -0.02238236089688497,
+            -0.0276584413929402,
+            0.023028408906176946,
+            0.008320436733620496,
+            0.04047641118026783,
+            0.011463851066078041,
+            -0.02247143742166039,
+            -0.034588725413285185,
+            0.009827713930789184,
+            0.00337071450636392,
+            -0.024446280199879062,
+            0.017542777160283324,
+            0.00780836483490212,
+            0.008453886667283516,
+            0.016624116261219637,
+            -0.004175682824655864,
+            -0.0042784362933758835,
+            -0.001546656717896957,
+            -0.008819739495525477,
+            0.01406790007187259,
+            -0.015487658289541583,
+            -0.021457182907140035,
+            0.024687169583245905,
+            -0.03915911962859764,
+            0.01771290139419629,
+            -0.006510046907173026,
+            -0.00997450880725535,
+            -0.02059560682587115,
+            1.5939836949198004e-05,
+            -0.007159493752862143,
+            -0.0004782363814453761,
+            -0.017503181957042913,
+            0.016643971600145645,
+            0.024056442566869802,
+            -0.013209211443909658,
+            -0.00445521990927108,
+            0.0020313626821877526,
+            -0.029235692455820056,
+            0.006662209097255912,
+            0.016993027053110635,
+            0.00529940612845993,
+            0.0026104743851393716,
+            0.0001707934714983891,
+            -0.031289446611163674,
+            -0.005782208620284939,
+            0.03053516889069683,
+            0.05590704747438521,
+            0.021628254147289437,
+            0.02403314981042378,
+            0.02908606663381421,
+            0.02717206823892633,
+            -0.003198844932040613,
+            0.0016217508103442348,
+            -0.01620660606431983,
+            0.027098323143010743,
+            -0.011664168290676943,
+            0.04322175021589083,
+            0.008652323184522541,
+            -0.049493020736445056,
+            0.004537958157696794,
+            0.00943239454147692,
+            0.034918603888916544,
+            0.016678716235690318,
+            0.0005887414054803805,
+            -0.005894338664998101,
+            0.03279165663336323,
+            -0.02735726709032382,
+            -0.011962070182991011,
+            -0.00036014407560325284,
+            -0.019079003730607717,
+            -0.015938295169860884,
+            0.010357611943935459,
+            0.04278924678845588,
+            0.0009957483598272451,
+            -0.0071422790468850215,
+            0.01571201177743999,
+            -0.009445307674383,
+            0.02286338492075306,
+            0.007336440384050498,
+            0.0042809497342459904,
+            0.025581467690095977,
+            -0.0025007871391868105,
+            0.02300539823997411,
+            -0.000628898771317254,
+            0.004582863909134464,
+            -0.015960578501883316,
+            -0.00930649396114688,
+            0.027838455754803444,
+            -0.011112268986721,
+            -0.029253746504396393,
+            -0.008734785561831333,
+            0.00472579547270089,
+            -0.0003805825712986138,
+            0.0021854358149324227,
+            -0.01183885533661493,
+            -0.010840951201591629,
+            0.019965691788904057,
+            -0.004922943301424112,
+            0.003800048754664761,
+            0.012664049429916252,
+            -0.0030124219674414897,
+            -0.01910844996750822,
+            0.01889983984736845,
+            -0.003858283778519141,
+            -0.024168821338992813,
+            0.020083955711766945,
+            0.01850497889545421,
+            0.045156567155119454,
+            -0.01586824055785609,
+            0.004612757419198795,
+            0.017497774574602505,
+            -0.002172088240279357,
+            -0.019617327081566827,
+            -0.014944795772093997,
+            -0.018212462639041492,
+            -0.015975283672064702,
+            0.012837956594691596,
+            -0.01243789877725244,
+            0.019500571353639314,
+            -0.0380735226388175,
+            0.012590900534826689,
+            0.004832168601136412,
+            0.007505385730121269,
+            0.016011983211212992,
+            0.025994560026860358,
+            0.002157436199771672,
+            -0.033465476168065784,
+            -0.01014599891204166,
+            0.007845373376916211,
+            0.010093432683262838,
+            -0.002849185178237036,
+            -0.018147004183434554,
+            0.003633790611809425,
+            -0.0266147420637404,
+            -0.018644422659309035,
+            0.022926737299761104,
+            -0.04798772015599113,
+            0.017908415301694906,
+            0.024859125158586466,
+            -0.04343699525707025,
+            -0.04886699607556352,
+            -0.011459188432595229,
+            -0.036774381316312464,
+            0.0014128005215813578,
+            -0.0005256067584181517,
+            0.011194992852264102,
+            -0.009342140079829378,
+            -0.018598671981710183,
+            0.01584789414301952,
+            -0.0036412641242200418,
+            -0.004225840239935549,
+            -0.00616696736018045,
+            -0.0025598631463178384,
+            -0.010188803157718543,
+            -0.03244694662411977,
+            -0.03290891136780909,
+            -0.00033742479296272127,
+            -0.009743927051414455,
+            -0.02334288359945272,
+            -0.03447062782872289,
+            -0.008728968107967028,
+            -0.037753659749578956,
+            -0.00515561961442376,
+            0.04070890150714597,
+            0.027517147608335957,
+            0.009449414978133334,
+            0.008086900089929912,
+            -0.013989034604490105,
+            0.05691047488419298,
+            -0.020425883381389086,
+            0.039466620642512155,
+            0.005301493569940605,
+            -0.0037222353899918703,
+            0.013075425214878277,
+            0.027710826422351564,
+            -0.006253665106995653,
+            -0.0017412928073238833,
+            -0.01928096630475438,
+            -0.012920476859996416,
+            0.009444408011467955,
+            0.015799884767184885,
+            -0.0025083061619197524,
+            0.001274457251739847,
+            0.00930533161700852,
+            -0.0006652478540264343,
+            0.017258038491335247,
+            0.03996268817486439,
+            0.003833844612473671,
+            -0.00826404141706295,
+            -0.018822002570963614,
+            -0.003188844900317347,
+            -0.011807006674248829,
+            0.010482058734260632,
+            -0.015410635475145028,
+            -0.011540803927522013,
+            -0.01249428777269578,
+            -0.017081856786754605,
+            -0.002183034976477535,
+            0.003114834030705039,
+            0.0027475411722881747,
+            0.02261627635192778,
+            -0.015872027027080824,
+            -0.022487581027490516,
+            -0.0035823117885461506,
+            -0.02398778449664045,
+            8.13716819067491e-05,
+            -0.01050060782407259,
+            -0.015616181283381218,
+            0.013676480044963827,
+            -0.021513173575512936,
+            -0.01091399089076612,
+            -0.008699176177861256,
+            -0.0006986953850759474,
+            0.005853723710671249,
+            -0.00993355748621621,
+            -0.00010241843582159751,
+            0.01150872694850241,
+            0.005559552266827916,
+            -0.012336449703762907,
+            0.0039431718621504,
+            -0.012740550272925562,
+            -0.004432059174951158,
+            -0.03592309122711106,
+            -0.019665686718735042,
+            -0.01846795637714379,
+            0.005642094448288269,
+            -0.005108964028543505,
+            -0.020586678054151854,
+            0.00668364061048941,
+            0.0034775127050453774,
+            0.004713169526661792,
+            -0.010723667875209928,
+            0.013944735241748791,
+            0.010882420168586074,
+            -0.003518245916227596,
+            0.013827334494230662,
+            -0.001330399936559009,
+            -0.020286697504699764,
+            -0.011139922402071095,
+            0.03538258948179278,
+            -0.024961826464903997,
+            -0.0018019328351763667,
+            -0.0026423695690168066,
+            0.00597003516870726,
+            -0.03071835946501977,
+            -0.013634212232744325,
+            0.02164400349828422,
+            -0.005846552894831441,
+            -0.026367014426434226,
+            0.03399116480234033,
+            0.03473258231629753,
+            0.010041605592929843,
+            -0.013423384340589794,
+            0.04730054254696893,
+            -0.012968978954938308,
+            -0.018274800841400665,
+            0.026826644936918647,
+            0.0278389611094493,
+            0.02525164095926168,
+            -0.029782550447890466,
+            -0.003010713086487275,
+            -0.02393629084357885,
+            0.017993366287762866,
+            -0.0263476663180971,
+            0.01864011499179424,
+            -0.02332263152537849,
+            0.016014908881756034,
+            0.01868464825773604,
+            -0.003104041044182249,
+            0.026313491086684425,
+            -0.0052945248919367815,
+            0.015253731232325074,
+            0.0012371308759955573,
+            -0.02119765517013431,
+            0.007795466385020642,
+            -0.006469500272298062,
+            0.025762169721880456,
+            0.00833992768202466,
+            0.01801901639978976,
+            0.022019940825351116,
+            -0.0409522298213741,
+            0.02003698451081499,
+            -0.001367916864679543,
+            0.01782984895971748,
+            0.0038516134262842373,
+            -0.007885432294782304,
+            0.014221728513400065,
+            0.010991352188016265,
+            -0.012851099261924757,
+            0.003840539873231741,
+            0.0008923482291826014,
+            -0.028928104542716937,
+            0.007066004768771198,
+            -0.026914574065060926,
+            0.025866907400807087,
+            -0.005526774784152086,
+            -0.00998122210956923,
+            -0.020744532673097317,
+            -0.0071425207262770865,
+            0.003754468896273156,
+            -0.012747874997429746,
+            -0.011385805601999295,
+            -0.024700457669046855,
+            -0.02401291382882952,
+            0.029591582802488962,
+            -0.014054332096344003,
+            -0.04015978298687865,
+            0.001234440863385445,
+            -0.023301255402129767,
+            0.01193125007773239,
+            -0.005793124129267758,
+            -0.024440100649464773,
+            -0.042563399082257154,
+            -0.017548888269389457,
+            -0.0009394887961485922,
+            0.022739763644266633,
+            -0.00114156949394707,
+            0.041791395064755515,
+            0.0008816696205437911,
+            0.022207132857533028,
+            -0.015547580543606755,
+            0.005319731264840819,
+            -0.02071229859314312,
+            -0.041495579742912285,
+            0.03660432378985971,
+            -0.014637031447181937,
+            0.025854844497435178,
+            0.03324325296704186,
+            -0.013009347371118843,
+            0.001354504885302625,
+            0.02652570709474851,
+            -0.014618100923440427,
+            -0.010782232747863238,
+            -0.0034936699515378212,
+            0.003461670978569965,
+            -0.006468116285021298,
+            -0.027793221545159466,
+            -0.007752458229193391,
+            0.04193706715422074,
+            0.017894069367616637,
+            0.021928994684423043,
+            0.022606401934291136,
+            -0.017972404868085358,
+            -0.026402048654300563,
+            0.003656646576244765,
+            -0.013129722990153284,
+            -0.018090360146368677,
+            -0.01624598856366761,
+            0.012397150789705746,
+            0.011741002980073786,
+            -0.004052422390675261,
+            -0.01836750423615859,
+            -0.020929335029248472,
+            -0.016327412417122416,
+            -0.0042615345226698015,
+            -0.008458117095535,
+            -0.005002582649405199,
+            -0.012158173174049797,
+            -0.020409616620022143,
+            0.03706368540614453,
+            0.0285565642978937,
+            -0.0041732933884846566,
+            -0.00607154864704767,
+            -0.011142150176334957,
+            0.008834431806990914,
+            0.006021232046087652,
+            -0.005028870471297621,
+            -2.3396043292139426e-05,
+            0.005112033372713154,
+            0.022822312435625314,
+            0.007271235299348001,
+            0.0035791789459533856,
+            -0.06340305811365254,
+            -0.005392000517722054,
+            0.022560201096078862,
+            0.012581401789917604,
+            -0.0032248219865223496,
+            0.025180718735801837,
+            0.0006451724964970927,
+            -0.009714514747463593,
+            -0.018389422219956937,
+            0.024728862150150004,
+            0.008760329201960964,
+            0.020988700992999768,
+            -0.014974763313686516,
+            0.003845297486923296,
+            0.0006445983847138896,
+            0.00750394530065325,
+            -0.0019792992847026793,
+            -0.008578254004439607,
+            0.025776677078006516,
+            0.016593438298233645,
+            -0.005695578040595484,
+            0.01604018433477423,
+            -0.0017497260488899657,
+            -0.021460543443428924,
+            -0.010265213095609554,
+            0.00042017209251926397,
+            -0.010570864288876387,
+            -0.018015790888295944,
+            0.017569414447505878,
+            -0.002601036282535863,
+            0.0072399377790164435,
+            0.030281707151249355,
+            0.0423183067569767,
+            0.009566493398534713,
+            -0.005928603293907002,
+            -0.0037936911251953654,
+            0.015477873304924939,
+            0.01513127995244932,
+            0.010741256034652754,
+            0.03015858543221727,
+            -0.029247492875021843,
+            -0.006690306960497428,
+            0.0046653727410352576,
+            0.004610405633429778,
+            -0.011692179099382171,
+            0.030055897497741527,
+            0.04569730592994177,
+            0.006736813244286893,
+            -0.004799978513117371,
+            0.002121067028052434,
+            0.011299889107780231,
+            0.007768137218140627,
+            -0.018927080714961783,
+            -0.0012222962555201746,
+            -0.006009911026251445,
+            -0.014001766533604134,
+            0.026941585318261085,
+            -0.003960837039351648,
+            -0.011848232223811548,
+            0.02903357274500332,
+            0.1105267435019781,
+            -0.006011029338781496,
+            -0.0053646998281195206,
+            -0.03242321704621517,
+            -0.015194102811332289,
+            -0.008860590919147765,
+            0.0056988205877963076,
+            0.01635957585050017,
+            0.020327661913559888,
+            -0.0008943408274661007,
+            0.016820860249955754,
+            0.02822175614376368,
+            -0.013854827839129885,
+            0.007906244895327178,
+            -0.030891381690403466,
+            0.015702117924362928,
+            -0.005383974926093977,
+            -0.012032955400313954,
+            -0.0026169511828034054,
+            -0.008618307035330928,
+            -0.0037356680273096105,
+            0.0009666430093455632,
+            0.011296307527684261,
+            0.004427516814276546,
+            -0.014484198313369681,
+            0.002829529156309784,
+            -0.008595300985863424,
+            -0.005122996529848575,
+            -0.02393571073523279,
+            -0.024119245543788293,
+            0.005223147047205755,
+            0.0019389305548629142,
+            6.344308292376579e-05,
+            0.006848965054539454,
+            -0.003810712095343454,
+            0.030149081317333418,
+            -0.01700322008272963,
+            0.0012288912810468446,
+            0.01193226302726573,
+            0.012332749933413344,
+            0.01688602002694633,
+            0.0028600984823761366,
+            -1.9882692552530084e-06,
+            0.021172977047420227,
+            -0.010514036509402561,
+            0.01460334896316005,
+            0.013091417303145243,
+            -0.04103572381458182,
+            0.0089746401463863,
+            -0.0027841289484449873,
+            -0.024252447218794344,
+            0.004446472209246425,
+            -0.018987867343500102,
+            -0.00881974688349601,
+            0.0005365625696166774,
+            0.012054719659248512,
+            0.01960300421041832,
+            -0.008553943699846696,
+            0.02111201386480852,
+            -0.0003501657740808554,
+            0.004177088753111459,
+            -0.012879944670340026,
+            0.0065222126975051685,
+            -0.007563354540402343,
+            0.0008168128607271199,
+            -0.018213241937597355,
+            0.012526561612175514,
+            -0.0008285725179894544,
+            -0.015737003159436623,
+            -0.011337688275309921,
+            0.004737346137976519,
+            -0.0027993545140114705,
+            0.0030843163114700313,
+            0.010153740340626154,
+            -0.008815302484351509,
+            0.004919275910399053,
+            -0.0017541042538165297,
+            0.002562816687714991,
+            0.014285249186421434,
+            -0.029816964593980453,
+            0.005357227738099325,
+            0.009154842495811638,
+            -0.00048782303639333943,
+            -0.013074976152495576,
+            -0.004054505559072866,
+            0.0028322102449485015,
+            -0.016877328079971877,
+            -0.012128199639779175,
+            0.023421330164419888,
+            -0.012565145848820333,
+            0.008609070188894315,
+            -0.010936598722317274,
+            -0.018401801734378692,
+            -0.01066119413133089,
+            -0.005254229780560808,
+            0.01234969508270852,
+            -0.0038427767204741877,
+            -0.006947594635952202,
+            -0.01756894543880097,
+            -0.007890897373987378,
+            -0.047246698568222466,
+            0.009617876521097624,
+            0.027157555263862723,
+            -0.011355758309667893,
+            -0.04050311381480088,
+            -0.01017774338301804,
+            0.02437257208967103,
+            -0.004056329295095018,
+            -0.008345450168215274,
+            -0.001599171409135998,
+            0.018358386463243014,
+            0.0027207445181251406,
+            -0.007292967749349572,
+            -0.011267861428599221,
+            0.01657507035043879,
+            0.015105061280816214,
+            0.021419854579631276,
+            -0.02088952706825817,
+            0.006295984081511286,
+            0.020973020776936996,
+            -0.03073246543555915,
+            0.0023720435876790876,
+            0.03363966216709148,
+            0.010412588613998134,
+            0.013023311559810034,
+            -0.0010217662933572685,
+            -0.0075567381078977215,
+            -0.008248818794295818,
+            0.011848200323549865,
+            0.005066463524567732,
+            -0.023174489297821385,
+            0.021823825355858556,
+            0.020368868756448753,
+            0.0026042856858509524,
+            0.003457067317891926,
+            0.0005539745115410357,
+            0.01062670844108589,
+            0.009948889803890108,
+            0.05100741556350918,
+            -0.006493714124814164,
+            -0.014021751055930108,
+            0.015063988102064609,
+            0.02488059237697814,
+            -0.0023499941119596228,
+            0.01087282522970853,
+            -0.02612218935931224,
+            -0.021423880914461062,
+            0.023389294741183817,
+            0.008967763132667892,
+            0.009637806238448432,
+            -0.010815706635624337,
+            -0.011390999462441815,
+            0.031750675158698435,
+            -0.01307829462211914,
+            0.00759749911213318,
+            -0.02169895957209634,
+            0.01387879409130081,
+            -0.008606362268863441,
+            0.032229658009209815,
+            -0.0009066257436946125,
+            0.011062436424260079,
+            0.028010486866468023,
+            -0.007835805879241447,
+            -0.019657486763625547,
+            0.003967480939987674,
+            -0.015577952617624695,
+            0.02188429826317828,
+            -0.020502758155233298,
+            -0.0002362500702658591,
+            0.0040935406015014265,
+            -0.010056462958039452,
+            -0.0057038083589231615,
+            0.011475170331848666,
+            0.0015345076181482267,
+            0.006359262977587827,
+            -0.020293014401106,
+            0.01369706440475074,
+            -0.003989482215141903,
+            0.007887824197823247,
+            -0.015925992045349828,
+            0.014370223626942766,
+            -0.02309139937510304,
+            -0.021808736033141787,
+            0.008142621179624918,
+            0.0108996933426044,
+            -0.007589050511458233,
+            -0.025971539510030145,
+            0.02250992080022228,
+            -0.024980833805349545,
+            0.024979008145354243,
+            -0.006622339940663538,
+            0.002566378488088469,
+            0.004913874740363343,
+            -0.008102103508732738,
+            -0.0028098905427027783,
+            0.0057865587191235565,
+            -0.0073509382400023995,
+            -0.014123756810537343,
+            -0.019453380788786282,
+            -0.04083629135586275,
+            -0.03177927047231389,
+            0.022927581786088044,
+            0.011557232599801421,
+            0.0017073832658476193,
+            0.02339552660545296,
+            -0.02247471449131386,
+            -0.017890866287500053,
+            -0.009775068417716798,
+            0.025184899522229628,
+            -0.04408541669037361,
+            0.012104255392615549,
+            -0.014669458760022245,
+            -0.0245258242481677,
+            0.006274252098230249,
+            -0.004161960147481513,
+            0.009299171724026455,
+            -0.010779421730862708,
+            -0.04713627671791054,
+            -0.005192684784377356,
+            0.02330975631183273,
+            -0.0036802876364168935,
+            -0.006197548344287773,
+            0.03048284650641829,
+            -0.0019503797069087845,
+            -0.005038254331399784,
+            -0.005524875459245342,
+            0.027783596017224226,
+            -0.024950083604372984,
+            0.010988398531346805,
+            0.008304937912856357,
+            -0.008485065895193593,
+            0.006572665697403556,
+            0.0011518846966102158,
+            0.011927348472282199,
+            0.015632588644277183,
+            -0.015559413716826264,
+            0.02262990861928957,
+            0.011446850286896198,
+            0.006506075777765303,
+            0.008015225053785824,
+            0.00010054430757843335,
+            0.0022975742216194894,
+            -0.024948925449668525,
+            0.026047924967851216,
+            0.01888211557921096,
+            0.0001632181017876392,
+            0.012553094476616919,
+            -0.015042621225242777,
+            -0.02782009986290425,
+            0.02606646297992026,
+            -0.016847653820092335,
+            -0.00045272379465939274,
+            0.020053091486344633,
+            -0.016210036358455773,
+            -0.007930550661303182,
+            -0.01123514849773852,
+            -0.0162675150307978,
+            0.01803076559921804,
+            -0.012160217543572069,
+            -0.02216594536057653,
+            0.002594675636753233,
+            0.016443123828746582,
+            0.02134480656850119,
+            0.026694575953313482,
+            -0.011470930697351533,
+            0.017768927048191052,
+            0.0008653408391607816,
+            0.001865765599795092,
+            0.025745954171649543,
+            0.01702677011968492,
+            -0.008474889109168202,
+            0.025347176686722313,
+            -0.0016686223698600755,
+            -0.002421921060670558,
+            -0.012265504084322075,
+            -0.0165214436458303,
+            -0.007545471138114208,
+            -0.015333235556146196,
+            -0.0051830106590765685,
+            0.000762746782664525,
+            -0.0018118022696726346,
+            0.10876576838038368,
+            0.0034948044540582626,
+            -0.011608460689239234,
+            -0.005870263500417206,
+            0.007510774308250375,
+            0.0023525201199670765,
+            -0.006846154764389575,
+            0.016082721103476375,
+            -0.018616626204086575,
+            -0.023593744079394,
+            0.004743481974389998,
+            -0.0009262286319469741,
+            0.00870122878944883,
+            0.02029523251993269,
+            -0.008608837818169835,
+            -0.0200567711498164,
+            0.007089477722191928,
+            0.005535149644041221,
+            0.0023640746516071774,
+            0.005256431824732927,
+            -0.0006118671321824933,
+            0.008209920240361308,
+            0.02448422122961942,
+            -0.015622892878860465,
+            -0.029770836607612903,
+            0.010535371601298532,
+            -0.00015994156470228762,
+            -0.007193425789067776,
+            -0.011046316463797128,
+            0.014728841492009627,
+            -0.0011488451990980268,
+            -0.02751404133172324,
+            0.017663132712536528,
+            -0.019055335178900415,
+            0.014643246200933506,
+            0.0025348345163347464,
+            -0.00016387173410118788,
+            0.022565953069005714,
+            0.013101158251834834,
+            -0.04272085076882881,
+            -0.0034442020309914823,
+            0.01041205537746844,
+            0.011059422927946552,
+            0.0116446016862942,
+            0.0012992535381387887,
+            -0.018670801241345514,
+            -0.011788984490283379,
+            -0.009109849103208542,
+            -0.021268488414296117,
+            -0.012523033785938503,
+            -0.011393853408628214,
+            0.010743646386439633,
+            0.0012964030814325153,
+            0.023809566099779113,
+            -0.003030535542277875,
+            -0.006526351867141901,
+            -0.000430699447948508,
+            0.00880561288237257,
+            0.009694646009259077,
+            -0.02079011138439714,
+            0.008294338719512953,
+            -0.01823922789220214,
+            0.011819769685519291,
+            0.006416044253693268,
+            -0.022799568832306067,
+            -0.009911619276012527,
+            0.007152786695893951,
+            0.008393677195358034,
+            -0.013569953300860946,
+            -0.006503988731872938,
+            0.02299498754558268,
+            0.018870619152710703,
+            0.012233964644738905,
+            0.024994616203196145,
+            0.03838969722235787,
+            0.005744255912479976,
+            0.027682123067526515,
+            -0.016791471092091604,
+            0.008459745383680297,
+            -0.022493868174622695,
+            0.012902262638928338,
+            -0.00460399519266689,
+            -0.0050431343566427886,
+            -0.018188192498411616,
+            0.033765105245412236,
+            -0.009159751849801773,
+            -0.0027150644587327763,
+            0.034778224828120714,
+            -0.0066502021793471075,
+            0.004525230911930458,
+            0.005096748531775903,
+            -0.016067049494073893,
+            0.033274672789154185,
+            0.01686831040872425,
+            0.016353966592947262,
+            -0.021631941883929817,
+            0.011748317746828848,
+            -0.005675701689448899,
+            0.0001273442591116645,
+            -0.020738819037390263,
+            -0.0037528323797385733,
+            0.0027725541343763104,
+            0.0059860509282538546,
+            -0.016556134518370437,
+            0.02033623476757896,
+            0.00019647023567641008,
+            0.0052895744489259875,
+            0.03253462247073863,
+            0.01043311189804924,
+            0.019182938878855655,
+            0.010051051068017807,
+            0.019523739618278902,
+            0.0281939391326836,
+            0.025685787482623124,
+            0.0029932688462254147,
+            -0.018668945726139946,
+            -0.0009326901784812825,
+            -0.007880352863625262,
+            0.010492063055849727,
+            0.022849902214818293,
+            -0.022619228461640748,
+            -0.02772644874135741,
+            0.018566981184526675,
+            -0.0022608185345048414,
+            0.020031103834122595,
+            0.007544464625538984,
+            -0.020935181884939796,
+            0.003407355515103286,
+            -0.023178230166337178,
+            -0.0005418155871620593,
+            -0.031039187720144967,
+            -0.028591174077243604,
+            -0.0007659241904525392,
+            0.011945344534674635,
+            -0.01688153017778311,
+            0.03357788163306365,
+            0.017667778671101954,
+            -0.014289109343082812,
+            0.01604684528817524,
+            -0.011054587325539086,
+            -0.0036959091138078683,
+            -0.005819853870004682,
+            -0.00203755010992417,
+            -0.02157944474979592,
+            0.005138614763345542,
+            -0.0033509138150859593,
+            -0.0024871060745482883,
+            -0.016936575573598154,
+            -0.021901830366872595,
+            0.016375173843894617,
+            0.016742770709641697,
+            -0.006606252542838758,
+            0.039771905404651496,
+            -0.004238071340064403,
+            0.00018364315349545657,
+            0.011690521707403059,
+            0.0010018392078563795,
+            -0.013651921068215563,
+            -0.010829994512513501,
+            0.012362647379788833,
+            0.011321403447882543,
+            -0.018942043512252663,
+            0.005763773608775982,
+            -0.009131565038297889,
+            0.002089216671850269,
+            0.005438887787329238,
+            -0.008183835275629478,
+            0.006690280329947951,
+            -0.01673229763262922,
+            0.015833397876364723,
+            0.00295474085300386,
+            0.003186554859191664,
+            -0.011797859525926772,
+            -0.020991927615680524,
+            0.01576413532776261,
+            0.0032403514518527287,
+            0.018819824905201635,
+            -0.026755218746060662,
+            -0.00012946467986739473,
+            -0.009552681500884924,
+            -0.0028941357661144826,
+            -0.012533800273548772,
+            0.013975640628890588,
+            0.016864739319953755,
+            0.0063941926578529,
+            0.01781016436903586,
+            -0.0026268224264609724,
+            0.015368284773773678,
+            -0.018122416427266218,
+            0.008686009180697955,
+            -0.006801074032721299,
+            -0.018537521789181048,
+            0.0013237338264105415,
+            0.027863741654435434,
+            -0.004670012662973029,
+            0.011066853115455966,
+            0.011704884649534624,
+            0.018954056680143452,
+            -0.0012626215734593707,
+            -0.0030850559460101866,
+            0.015610404665171367,
+            -0.005838226405658572,
+            -0.006861883833171456,
+            0.020872966233916948,
+            0.032951653763679145,
+            -0.006716215648910268,
+            -0.0021292599951407863,
+            0.010886596327481677,
+            -0.004901684487088035,
+            -0.008644584974417106,
+            -0.009587037829603014,
+            -0.03158137900717491,
+            0.024369089446706894,
+            0.012092412439477474,
+            -0.009601042951586109,
+            0.022690862146548786,
+            0.020752792552278052,
+            0.011412743968296708,
+            -0.010471860708195726,
+            0.02291536038589634,
+            -0.01268035142112569,
+            0.024168869865226136,
+            0.032482831775271157,
+            0.004400973556358811,
+            0.0022312906286559167,
+            0.0021175489661843985,
+            0.01691348460546639,
+            0.006050357346231155,
+            -0.007068133214675996,
+            0.011656003476409986,
+            -0.018487923180009552,
+            0.01598530891381001,
+            -0.004432366596808033,
+            0.02808371007118948,
+            -0.01337145216499331,
+            -0.010473700033368562,
+            -0.00537325376270318,
+            0.00010726431372328729,
+            0.006445701087575359,
+            -0.003324002570256215,
+            0.005255443653813165,
+            0.0007072577303252132,
+            -0.017825649146944237,
+            0.003050128130393781,
+            0.004825967170196209,
+            0.009018799597337101,
+            0.004103653497834134,
+            0.007249957401824204,
+            -0.012360355929483,
+            0.01441885664789707,
+            0.004401820363035808,
+            -0.0026190938717927086,
+            -0.0007163225760398784,
+            0.011124538295057815,
+            -0.013232822471919533,
+            -0.004895624967029031,
+            0.005281228035321008,
+            0.0059275753353475414,
+            -0.007507924998741734,
+            -0.004484140989405051,
+            0.011908076549392969,
+            -0.0037917291321648483,
+            0.006351543504726163,
+            -0.004409464890606014,
+            -0.0037760571637986827,
+            0.005303661313434895,
+            -0.00974511544836468,
+            -0.004204167898342997,
+            -0.0007093125445559502,
+            -0.006869250680233214,
+            0.00786202375969793,
+            0.005971033227117194,
+            0.010606085272785863,
+            0.006969437776542303,
+            0.004293129058563288,
+            0.004114452739123398,
+            0.004071017374828283,
+            -0.002445681623750424,
+            -0.00592480769363354,
+            -0.004617501328052899,
+            -0.012067458701076778,
+            -0.008335239153829813,
+            0.009025028998949398,
+            0.008653779421856954,
+            -0.019363018792788574,
+            0.02048011578221327,
+            -0.001188320960771659,
+            0.00772165044232104,
+            -0.01602086976290928,
+            -0.000907553574811279,
+            0.005464880629734941,
+            0.008267702703609528,
+            0.016136762227127184,
+            -0.03654209526492064,
+            0.005179791231187316,
+            -0.0037321883493524766,
+            0.016026013709955444,
+            -0.008215703411373182,
+            -0.006979816286354862,
+            0.02564060526656963,
+            -0.01450763388985563,
+            -0.0009847758228920171,
+            -0.016142304341196283,
+            -0.01407778890808498,
+            0.010815725252959506,
+            0.02050940280233426,
+            0.020684808835848383,
+            -0.0004965802897137407,
+            -0.0005781445948759235,
+            0.00036179841583004193,
+            0.022775804864342608,
+            0.015545681427566235,
+            0.018697154644607316,
+            -0.004543500782944068,
+            0.007028436664826706,
+            -0.023057215501463973,
+            0.009912581556496658,
+            0.0011795417129232596,
+            -0.008215027850908477,
+            -0.008554137076267468,
+            -0.008703378564520125,
+            -0.010448538943182258,
+            -0.011773811327190638,
+            0.004106616839960258,
+            0.013895013678116032,
+            0.004190549543250594,
+            0.0031571190600981074,
+            -0.0225385788570314,
+            -0.006243193914542516,
+            -0.011229641377928767,
+            -0.01674214043453308,
+            -0.007367389908398082,
+            -0.02386334969150212,
+            -0.028021339684039734,
+            0.0002794207370355175,
+            -0.02411861258809571,
+            -0.015958336406924612,
+            0.005268815864773021,
+            -0.01546610225139129,
+            0.00898600137695074,
+            -0.008396481690803911,
+            -0.011873486990732417,
+            0.0074743033679134105,
+            0.015336444788692,
+            -0.017676977079590463,
+            -0.010165317702640217,
+            -0.01690524990506033,
+            -0.013460451917691695,
+            0.010918550283871637,
+            0.00595960977280955,
+            0.009250157926438213,
+            -0.012447979337850305,
+            0.0062932435120863365,
+            -0.012673905639312696,
+            0.015073013552721701,
+            -0.03769166067890086,
+            -0.029992249463405965,
+            -0.00014990754540999537,
+            0.007288036630677436,
+            -0.0020735742042505426,
+            -0.018695120921334806,
+            0.0003860116505754868,
+            -0.003608995261360232,
+            -0.014900191335144973,
+            -0.013866216738266176,
+            0.025251475305524053,
+            -0.006788483107616133,
+            0.01223227424986397,
+            0.0034937277040015626,
+            0.03438333560343413,
+            0.008803852645562129,
+            0.014666206816302367,
+            -0.005709421955160623,
+            0.003742433643393657,
+            -0.0033706258426092744,
+            0.006906464031352315,
+            0.0076578888497945185,
+            0.009276592191710524,
+            0.008025321004305927,
+            0.004576470294889512,
+            0.002887346543823031,
+            -0.001925468214331082,
+            0.02125241545487307,
+            0.025671028960818562,
+            -0.00037467906648160044,
+            0.008981063192139038,
+            0.0009387900084090577,
+            0.0002735789978680719,
+            -0.008374290340790827,
+            0.006547013284767913,
+            -0.008274637814401893,
+            0.004041066530254948,
+            0.012897549078838394,
+            -0.009150863940242148,
+            -0.010211568689323908,
+            0.016575088843292566,
+            -0.012380099999574783,
+            -0.03420455578618604,
+            0.026550654680719987,
+            0.005324512466403649,
+            -0.0035670106106754245,
+            -0.010657928468872608,
+            0.0037176027853158167,
+            -0.01103891334887315,
+            0.008096915671982816,
+            0.022587659771132487,
+            -0.0013527917944322358,
+            -0.0018085171207746743,
+            0.0038503794877980586,
+            -0.011870484733648887,
+            -0.007881489342351292,
+            0.006462772034381968,
+            0.0315696514747912,
+            0.01326417653779577,
+            -0.020400369539140532,
+            -0.015223808971473513,
+            0.020406862349006738,
+            -0.023142176712111575,
+            -0.006254959806032159,
+            0.009039506093607052,
+            0.0009087520492922078,
+            -0.022766046838415165,
+            0.0009662037093312435,
+            0.004701966191595815,
+            0.005946458676746661,
+            -0.006358986495019584,
+            0.03237532341275415,
+            0.001075992257265679,
+            0.015892020271402903,
+            -0.017189453434945788,
+            0.0017514632185789299,
+            -0.014189278671121677,
+            -0.014265447388465805,
+            -0.029809460132405256,
+            0.0005557491671071693,
+            -0.0030324198417091704,
+            0.011038984948790744,
+            0.024808067315720335,
+            0.01145103533194245,
+            0.014432246996007246,
+            0.010552852947578788,
+            0.01513689173494211,
+            -0.013653722180565311,
+            -0.029009757850432732,
+            -0.005171522291984226,
+            -0.018001048743500243,
+            -0.02297482628212251,
+            0.003962613844063587,
+            0.013645125285548414,
+            0.006278662690346082,
+            0.023786381386575035,
+            0.0021738516691483163,
+            0.0009580591750940011,
+            -2.690805459518147e-05,
+            0.008894091070322921,
+            -0.004952203441302471,
+            -0.006553986197076102,
+            0.003570189040872711,
+            -0.000652714137223501,
+            0.021228360881461853,
+            -0.005758946428212041,
+            7.897153237591187e-05,
+            0.014242175492137011,
+            -0.005622256348934008,
+            -0.0035009192529791146,
+            -0.017339926143652377,
+            0.006248842389703505,
+            -0.01622011586976866,
+            0.025783086900871728,
+            0.018450783573486916,
+            -0.009109295293546267,
+            -0.013469521573468915,
+            -0.011259555201313116,
+            0.02773097452736661,
+            0.004631282554715908,
+            0.010787055220617197,
+            0.018228322680582464,
+            0.01461254259249465,
+            0.01784830193964943,
+            0.004664725683522323,
+            -0.0034339781146649384,
+            -0.0007692550982545399,
+            0.021176646860433974,
+            -0.014700178590249292,
+            -0.002701208084390198,
+            -0.008842788249773623,
+            0.0021799005046816444,
+            -0.008431299000066363,
+            0.006086447848362669,
+            -0.010793016137040172,
+            -0.015251381840165954,
+            0.0189503457652375,
+            -0.02660767353735015,
+            0.016423753600066336,
+            -0.00593016170680532,
+            -0.00482891500342298,
+            -0.027459214027897378,
+            0.010728039528784636,
+            0.0027866316191608297,
+            0.0022519974195000687,
+            0.010770472550461752,
+            -0.021427634123401408,
+            -0.01867717541189363,
+            0.008532444554010673,
+            -0.011457117350255678,
+            -0.007152301670244725,
+            -0.00348619742388509,
+            0.0154715607296319,
+            -0.00017898955187849443,
+            -0.009925365923862246,
+            0.02085093731060963,
+            -0.007175309371250024,
+            0.01223979370120031,
+            0.005256127464980655,
+            -0.008756077893734508,
+            -0.0028438320920811362,
+            0.021022708898898203,
+            0.027557382862284808,
+            -0.008342206702778956,
+            0.014526066580760486,
+            -0.011883797998224506,
+            -0.01088335352319397,
+            -0.03479444009382274,
+            0.030104838505937447,
+            -0.013305732257226255,
+            0.026773747582411633,
+            -0.0006968265979630097,
+            -0.017037729111584094,
+            -0.016206207453228583,
+            0.01625225876515654,
+            -0.0024290584023212254,
+            -0.009366287391027166,
+            0.01465885301870365,
+            -0.02428628495473643,
+            0.0058336571592899735,
+            -0.0006627935593670669,
+            -0.011531762838191602,
+            0.013275508677482622,
+            0.0019223252930710324,
+            0.010738622785461513,
+            -0.02030405805119001,
+            0.0009661595529028194,
+            0.009263716373456188,
+            -0.016791855691574455,
+            0.006057830565306032,
+            -0.03730952786847912,
+            0.0029341721588188943,
+            -0.014833895684348656,
+            0.005725615593914644,
+            0.001498755253963699,
+            -0.01820708352937343,
+            -0.03652692232407375,
+            0.00043665701245599497,
+            0.01913883585802903,
+            -0.007975988799790905,
+            0.020538595326175857,
+            0.008243569241768276,
+            0.008421404577931517,
+            0.002314738067443065,
+            0.0016702403734608475,
+            0.008279538988821465,
+            0.03679456861213399,
+            -0.01775927634148114,
+            -0.004836606118958108,
+            -0.0012151329859941935,
+            0.010644195708319258,
+            -0.02028470571116305,
+            -0.009287515726906219,
+            -0.0036506912311261536,
+            -0.013134467570201307,
+            0.030615862647585222,
+            -0.020265555441523482,
+            0.025634449009756734,
+            -0.009300770899023007,
+            0.009274072905190077,
+            -0.02097685009125915,
+            -0.009797008690754956,
+            -0.010661540100131183,
+            0.0019532291898233895,
+            0.013465005282849381,
+            -0.009151430285942593,
+            0.0071467934925125145,
+            -0.006655608635025676,
+            0.01020327561163374,
+            -0.001472597695707201,
+            -0.02443508800166349,
+            -0.0024998811551575563,
+            -0.01107321258077183,
+            0.02456164421154801,
+            -0.0019619696941303388,
+            0.0005043179699158694,
+            0.010951580163871948,
+            -0.00853593956432799,
+            0.023927336608555987,
+            -0.008855702280766362,
+            0.0010093679746665581,
+            -0.004863269000020324,
+            -0.004005691415117686,
+            0.001698212044518236,
+            0.01822024022650613,
+            -0.005707404715849356,
+            0.03722587302228139,
+            -0.01708925706528118,
+            -0.0065500772743641925,
+            -0.00015538686795361824,
+            0.010852925330001283,
+            0.010035359503147549,
+            0.008754719437760327,
+            0.004067820167375348,
+            -0.014861417626433965,
+            0.009346669946626484,
+            -0.020136029447262147,
+            0.007501669351583119,
+            0.023178490169347605,
+            -0.006840359541885465,
+            0.0006676450965357345,
+            -0.01283320739092204,
+            -0.015610538827742276,
+            0.006308543164022682,
+            0.002329420005586303,
+            -0.002295739637504597,
+            -0.02181189272979859,
+            0.006298804075737111,
+            -0.014398470126723037,
+            0.004007419942728446,
+            -0.0007497656993578918,
+            0.013863867106289847,
+            0.016451166273826513,
+            -0.006740209707090658,
+            0.039292260347522444,
+            0.01871723939316731,
+            0.009109005310505462,
+            0.005674782595856756,
+            -0.040588444759736254,
+            -0.008993850998154852,
+            -0.0010197185016393919,
+            0.004866150984083949,
+            -0.01141354216871663,
+            -0.018944881712855702,
+            -0.002646110151132781,
+            0.000308585036997592,
+            -0.010681954707313127,
+            -0.009091585604191968,
+            0.01751599002710486,
+            -0.006034424764302979,
+            0.00872057856926618,
+            0.004146961781109887,
+            0.023758065635807138,
+            -0.009727802672285544,
+            0.01176257119851645,
+            -0.0071822083765114165,
+            -0.005456197751568526,
+            0.0006726765703478756,
+            0.011011628907288813,
+            -0.008031792262886802,
+            0.009763340867550022,
+            -0.004279891138185881,
+            0.008355720233269893,
+            -0.008552596384067825,
+            -0.023952129458400563,
+            -0.014972827087541732,
+            0.014347535404251018,
+            0.011651260496709961,
+            0.008137738462704715,
+            -0.013624435569539285,
+            0.0031728721674035352,
+            0.00675830277744976,
+            -0.009184784787581942,
+            0.009161771311910335,
+            0.017733082848622708,
+            -0.001411542568472906,
+            -0.0017830167499636113,
+            -0.0016130003349297027,
+            -0.012173528776899308,
+            0.0011196437753239162,
+            0.02284887358189995,
+            -0.003054479643937998,
+            0.014614283913976716,
+            0.010689486748036806,
+            0.009643292262102645,
+            -0.00810930837603355,
+            0.001915955511174678,
+            0.02341863048264366,
+            0.015022291783105098,
+            -0.004625715884342598,
+            -0.011422973894623295,
+            0.004259757717597968,
+            -0.010332744704689279,
+            0.007373328473086509,
+            -0.02117940113144938,
+            -0.004672698999328004,
+            0.008896311380763048,
+            -0.007028862917603513,
+            -0.01732432285099198,
+            -0.028026173555055017,
+            -0.016765805892242715,
+            0.019819329380181,
+            0.0027213760678977965,
+            -0.013291251876048186,
+            -0.0032919084721662363,
+            0.0031214883337300926,
+            0.002553005168048115,
+            -0.0071104806975682885,
+            0.005310583052470149,
+            0.007359183453835734,
+            -0.008674133980486269,
+            0.017665958608281126,
+            -0.000521269431423141,
+            -0.0008020251454616109,
+            -0.0005442374261144659,
+            0.0006127687598711624,
+            -0.016387752806126442,
+            0.01666161912459431,
+            -0.010949228376419892,
+            0.015198110119816624,
+            -0.008966726991404666,
+            0.008600499511690635,
+            -0.007617043351740469,
+            0.009458641139070436,
+            0.011597957400445055,
+            0.012755037944648426,
+            -0.004403680820063604,
+            0.008519241473418196,
+            0.004807816191718704,
+            -0.008147125240985166,
+            0.007017601015398136,
+            0.01296082929962557,
+            0.00582353932160707,
+            0.0013634237970372698,
+            -0.0008246894322793264,
+            0.0022035670958705576,
+            -0.002480976219285683,
+            0.0056293125704658135,
+            0.012714850523393719,
+            -0.002497639449055006,
+            -0.007135683201188999,
+            0.014988249771676699,
+            0.014955895630183352,
+            -0.022994948923833993,
+            0.007121634434082677,
+            0.005835971825639792,
+            -0.0030507120821557008,
+            0.002145266953387315,
+            -0.027334525318320587,
+            -0.020359474070022213,
+            0.01901995970350808,
+            -0.015260454474605018,
+            0.02850460517419317,
+            -0.011624138792780823,
+            -0.02284242513728605,
+            -0.01277489645734181,
+            -0.004209529633534193,
+            -0.022684500849916897,
+            -0.004894414098943742,
+            0.01919165657366256,
+            0.013401560972810082,
+            -0.004313549020265677,
+            -0.019089369171102138,
+            0.03863248408555474,
+            0.010497951840980463,
+            0.03508157919687289,
+            -0.008597970944278045,
+            0.0023354831464448313,
+            -0.01563401808449416,
+            0.015551176593436512,
+            0.001075882802133782,
+            -0.011575692834796734,
+            0.01505716108583101,
+            0.031836226915668404,
+            0.0034740889237207494,
+            -0.00617206855371193,
+            0.0018420207148235309,
+            0.00418433325800744,
+            -0.0008167766650900381,
+            0.011160989178055499,
+            -0.018497472318034093,
+            -0.011951262348399922,
+            0.0013093376768780122,
+            0.00216347903176653,
+            0.0059233460419514045,
+            -0.001778091863744448,
+            -0.013317639704511549,
+            0.011433308892643204,
+            0.011596810919469926,
+            -0.01518143105525228,
+            -0.007497373860338335,
+            0.0050957665991362405,
+            -0.018974405317819656,
+            0.01872472902494418,
+            0.011859522398789263,
+            0.000792413325332956,
+            -0.0006503947129906383,
+            0.00808635755624928,
+            0.003749813548995223,
+            0.0033657799335872368,
+            -0.007862215303617144,
+            -0.0028014669075116876,
+            0.0009892310856576799,
+            3.834379772674841e-06,
+            0.015402211364447726,
+            -0.015264893397161102,
+            0.02488855366696587,
+            -0.0025643639525691083,
+            0.00953993868921819,
+            -0.017460590918773188,
+            -0.0017911712931080155,
+            -0.00402374787529868,
+            0.01838051670627292,
+            0.006753072393532114,
+            -0.010130676216186492,
+            0.004921510943896829,
+            0.007975201081296845,
+            -0.012589786869771252,
+            0.00018319054705558588,
+            -0.0022606180909706688,
+            -0.014791528072476771,
+            -0.0031171076370692706,
+            0.017873825561816544,
+            0.007593688317551089,
+            0.0037375070591255375,
+            -0.015863143964928172,
+            0.007192324815779268,
+            0.007522990667369163,
+            0.024504875417877527,
+            -0.026309816223960274,
+            0.024779042752840836,
+            -0.022104087334088543,
+            0.02165781034613893,
+            0.0319051741666687,
+            0.007538146369110045,
+            -0.01690181186708103,
+            -0.006595947249169521,
+            0.006373991305923002,
+            -0.023410814842488123,
+            -0.005494017994983188,
+            0.016903736810723716,
+            -0.01792400152372258,
+            0.000330865061689199,
+            -0.014660413003063089,
+            0.0040825880154611895,
+            0.008248037865838212,
+            0.011540333451799686,
+            -0.0033578830043738085,
+            0.018327525495676525,
+            0.020171745352159914,
+            -0.01138759854371235,
+            -0.0004221149053107702,
+            -0.022321409200941777,
+            0.005046865873981027,
+            0.022509643273449936,
+            0.001111272240506322,
+            -0.009540467231190513,
+            0.018376261423745193,
+            0.008480532791922674,
+            -0.020801891594934313,
+            0.008436374227753589,
+            -0.036175982655998785,
+            -0.010454118523313122,
+            -0.009328507118928412,
+            -0.001218633166683384,
+            -0.004142773602181115,
+            -0.0002494371580772723,
+            -0.011365548842513265,
+            -7.699548642315061e-05,
+            -0.006889339420877534,
+            -0.0037173070609413474,
+            0.014953978004522174,
+            0.011678957032329096,
+            -0.003501047405385249,
+            -0.0073984932581048915,
+            0.008493128567102267,
+            0.006163937693564917,
+            -0.017034696006318337,
+            -0.013268418502042576,
+            0.0037122128689059085,
+            -0.020067656401141467,
+            0.005148329575092478,
+            0.00459729178556505,
+            0.014031669598917171,
+            -0.01849235806598133,
+            -0.009298127665365717,
+            0.019716512208448042,
+            0.0094466405158553,
+            -0.0008988916054855498,
+            0.004264260983102956,
+            -0.00032312548009733906,
+            0.002279987803645912,
+            -0.0008902345374346296,
+            -6.145528806973222e-06,
+            0.003602229465148649,
+            0.012503366799381277,
+            0.007759393855482671,
+            0.03694402922447051,
+            -0.016692148708641812,
+            0.005873765863350649,
+            -0.014187609702966002,
+            -0.0037592517335595654,
+            -0.00749939010396784,
+            -0.0004308157309481927,
+            -0.0022282234382577843,
+            0.0021999996319642854,
+            0.0052176773733353925,
+            0.009353171029508977,
+            0.002155405440765206,
+            -0.00823565584692709,
+            0.009691447250255694,
+            -0.015228173095533029,
+            0.003853858061067531,
+            -0.008207910291425473,
+            0.01017972550354623,
+            -0.009695280827515048,
+            0.0028705587799331246,
+            0.03849358164875786,
+            0.01130021300937487,
+            -0.004934175853966098,
+            -0.014300147534361459,
+            -0.0007796051618993811,
+            -0.01737877501558196,
+            -0.008579119859817895,
+            -0.005434792488602325,
+            -0.02617338653902575,
+            -0.00838916510137031,
+            0.015344260464413648,
+            0.011202567142310492,
+            0.014124853863170132,
+            -0.023894182185855824,
+            0.008897943489322673,
+            -0.03001670823127986,
+            -0.011951507895140876,
+            -0.001530967140156163,
+            -0.015467812535428609,
+            0.0010110999991167703,
+            0.023623196504289838,
+            -0.007762735863648054,
+            0.012654652822750325,
+            0.012867958170258448,
+            0.018777522196571934,
+            0.006390986057157016,
+            0.00038045736974183937,
+            -0.009444911783760569,
+            0.009534310165417939,
+            0.008806031651319704,
+            0.0080954543713255,
+            0.013161833638467553,
+            -0.0022840529693446553,
+            -0.018060162439023572,
+            0.005469288590000856,
+            0.014354475778235131,
+            -0.0033504276914600585,
+            -0.001615191819057537,
+            -0.011088493180661976,
+            0.016039040565630432,
+            -0.023559046333872156,
+            -0.009482398794557168,
+            0.009918576960319548,
+            -0.0027308309155029842,
+            0.009212063547988036,
+            0.0160181200081126,
+            -0.01881373001297218,
+            -0.0024168618343039703,
+            0.016244889977796682,
+            0.0024104546308670025,
+            -0.00022606425578303413,
+            0.016792244238218245,
+            -0.0007616857360728623,
+            -0.004279120886828465,
+            0.00997764266310123,
+            -0.0075631917138318655,
+            7.3878137053128405e-06,
+            -0.002416364915008295,
+            -0.0028253498168087964,
+            0.02054128727706496,
+            -0.021281801678871736,
+            -0.005379259148621041,
+            0.02257306103533458,
+            0.015687105622685975,
+            -0.007843745111084413,
+            6.258498042855072e-05,
+            -0.0219506172079994,
+            0.0019925886885244454,
+            -0.007768133999661569,
+            0.016524850592507592,
+            -0.007957949830060354,
+            0.0030085044673901665,
+            -0.01361024620290394,
+            0.011563781350927133,
+            -0.01277407169098418,
+            0.00873716215529134,
+            -0.002986943948333218,
+            -0.017567646684567492,
+            -0.0053109390854215125,
+            0.004561236136619266,
+            0.0014045175130801308,
+            0.044958041347040185,
+            -0.005539136232703934,
+            0.003729517522976541,
+            0.018086335927383804,
+            -0.014199453349727933,
+            0.02079719360289355,
+            0.016663293302592568,
+            0.00991445855219406,
+            -0.00033967370741635794,
+            -0.0018726748849858836,
+            0.023373354806639515,
+            0.013258226441642464,
+            0.00861986998150065,
+            -0.0014340218811281975,
+            -0.008516541227326329,
+            0.018239230461345617,
+            0.005458629943667832,
+            -0.012421896028975013,
+            0.012732721349758832,
+            0.013559877132508506,
+            -0.013720939472355347,
+            -0.011545210834395917,
+            0.01616358223872008,
+            -0.019904594319155373,
+            -0.018260016571763644,
+            -0.0036124537591368864,
+            -0.01614510591132948,
+            -0.03864699384205461,
+            0.01619401261888991,
+            0.018351607973242735,
+            0.006666313000123845,
+            0.013647410298896132,
+            0.002597680061427196,
+            0.004545530267244157,
+            -0.018531082430861854,
+            -0.0077618625909995275,
+            -0.008085144524135461,
+            0.003744735923004248,
+            0.011059457364156674,
+            -0.011335432904114519,
+            -0.009514379703581616,
+            -0.00034500702695418285,
+            -0.007433132060989644,
+            0.01971420270994951,
+            0.0020753610290475642,
+            0.0034506706065161024,
+            -0.00628437298309269,
+            0.017966485854173167,
+            0.026789678964152908,
+            -0.007532446825077476,
+            0.021886182869092073,
+            -0.002864230253056434,
+            -0.0015870360120997492,
+            -0.014393519359298496,
+            -0.0014439412842964876,
+            -0.011527182213472351,
+            0.0005453493437517964,
+            -0.0006791481533636673,
+            0.014093767997869645,
+            -0.03288824642147967,
+            -0.003957301296796813,
+            0.014444592778851293,
+            0.007538052312876599,
+            -0.005057755472988024,
+            0.011357919882949903,
+            -0.002964451028327082,
+            0.016977317604413924,
+            0.0075201055190542146,
+            0.023391646079994784,
+            -0.016588823248397513,
+            -0.004504858318770565,
+            0.006660877064954993,
+            0.0034631255931793638,
+            0.0014484203909961972,
+            0.01912983041187014,
+            0.023439941258581887,
+            -0.014715886385460541,
+            -0.0013596123366001458,
+            -0.0029020372473879614,
+            -0.021414221343490594,
+            0.014976806123253727,
+            0.0074729481304182865,
+            0.010967351226114886,
+            0.009806663105020302,
+            0.015465076881663691,
+            -0.02109587527354609,
+            0.0011671121739822202,
+            0.020567458365051382,
+            -0.008618333549105003,
+            -0.018815706906171112,
+            -0.009804829158883123,
+            -0.00966828714719191,
+            -0.006078893678150302,
+            -0.00803783313451146,
+            0.005523023167028105,
+            0.0022647506733877573,
+            0.01652997604669998,
+            -0.008204200434441975,
+            -0.008404609147866811,
+            0.008489247125986078,
+            0.007546785673285454,
+            -0.0033093451283570316,
+            0.010614050963656496,
+            0.0030633179859393587,
+            -0.006940540318011839,
+            0.00894884343338777,
+            -0.002787816848430143,
+            0.014347598920245487,
+            -0.02441769157218451,
+            0.004717742147185283,
+            -0.012096876825337124,
+            0.006405151160675699,
+            0.0041568790527989905,
+            0.0016702066682613428,
+            -0.0041159211459004,
+            0.0035217362856754353,
+            0.0008893479210743541,
+            -0.07433312151926533,
+            -0.012629681414004524,
+            -0.0035797012285864486,
+            0.033657838713146884,
+            -0.019695806809053914,
+            0.004477927056336216,
+            0.011966065644382396,
+            0.0096717896523898,
+            0.003070713232538593,
+            0.006136726101157844,
+            0.006112385569942848,
+            -0.0245258242481677,
+            -0.00015284418977092615,
+            0.004821040694679944,
+            0.0020347004045943454,
+            -0.007532408630122127,
+            0.00750565079228405,
+            0.015327710245879296,
+            0.020989926895388523,
+            -0.010970305087289488,
+            0.017227663358885748,
+            -0.003352718496981928,
+            -0.006068892126250978,
+            0.0014285750322333686,
+            -0.010153799722483554,
+            -0.017307415843726394,
+            -0.006495008086641125,
+            0.0068983699213297075,
+            -0.001982568363667046,
+            0.0044032687939897855,
+            0.003333596495330213,
+            -0.0032171586993948696,
+            0.0072365127663633925,
+            0.013920326463272646,
+            0.006415916101287133,
+            -0.013675562796300214,
+            0.003003782758807665,
+            -0.02500375035312058,
+            0.011945988879567646,
+            -0.012708975181025296,
+            -0.0009488161508767279,
+            -0.0047093877027523014,
+            -0.0035750724825159503,
+            -0.002213435677949806,
+            0.02043567538125188,
+            -0.014304009033474186,
+            -0.01802899775650995,
+            0.008187227768196673,
+            -0.011217013195839762,
+            0.008300740149622635,
+            -0.03382831122769974,
+            0.012386335491333264,
+            -0.0276029934382802,
+            -0.0002243752790856292,
+            -0.004521387639042107,
+            -0.0010631721066129095,
+            -0.009491184157278669,
+            -0.03274559774619223,
+            0.015845705808609598,
+            -0.007204751554079551,
+            -0.020338337341085228,
+            0.0007143424225187802,
+            -0.021267094083674304,
+            0.014150458196361147,
+            -0.01851165553224022,
+            0.0033450353123933058,
+            -0.018775439467431685,
+            0.002782289821140401,
+            -0.0018711350216089556,
+            0.010502195116579441,
+            -0.00557814442809019,
+            -0.0007130962242755068,
+            -0.013072988887382269,
+            -0.002121841212381393,
+            -0.006653444103331213,
+            0.009394907690516121,
+            0.0028678074167728367,
+            -0.009601577734902932,
+            0.011153571864462314,
+            0.013046116892519414,
+            0.019812860521973938,
+            -0.00539047787915588,
+            -0.00555139958411929,
+            -0.011553689090179406,
+            0.009852063270798534,
+            -0.016326212706634428,
+            -0.00827402913960903,
+            0.008982303148984887,
+            0.015216947130173639,
+            0.01026889105233137,
+            0.003352028462252994,
+            0.003150801750574548,
+            0.002531620731961765,
+            0.026744691368233237,
+            -0.0015120532859309638,
+            0.0008324701960729416,
+            -0.014508988496009931,
+            0.022127525293249926,
+            0.02772430884375744,
+            -0.0012206102581726523,
+            -0.00018859639594030037,
+            0.017437360169132967,
+            -0.004697052522250805,
+            -0.0038111184371524645,
+            0.024015246959315593,
+            -0.038396108183338655,
+            -0.008416825232533771,
+            -0.006788816882617399,
+            -0.006043428198944214,
+            -0.009587366247924467,
+            0.007294190273465812,
+            0.00041966080769004275,
+            0.0024709692753547823,
+            -0.0005250904505954556,
+            0.03452858797518386,
+            0.003403515654095309,
+            -0.0003392060101496309,
+            -0.0032980864446307116,
+            0.0019125121221204029,
+            0.0032987517853214346,
+            -0.009116761755779728,
+            -0.038030085122772624,
+            0.006006334000396066,
+            0.007932785488877407,
+            0.00975009749364395,
+            -0.008120121513849434,
+            -0.008864461019882246,
+            0.004909348738798099,
+            -0.02273129283865203,
+            -0.01372636307167243,
+            0.02625229144362647,
+            -0.008523028986127237,
+            0.029005877324974557,
+            -0.022435059061182174,
+            -0.013135836801818153,
+            -0.004126433322528265,
+            0.010283975013795537,
+            -0.019732856462444506,
+            0.004343860338939202,
+            -0.004022387065239562,
+            0.023733721026005954,
+            0.0077578144626961245,
+            -0.008371631446146718,
+            0.012386735614153616,
+            0.0064789380436815714,
+            -0.0026638642827013837,
+            0.00136333052963848,
+            -0.007209517067748103,
+            0.016916505654836485,
+            -0.011819784026452944,
+            -0.005833909403097991,
+            0.008977527508585844,
+            -0.0024556434204852946,
+            0.02509293948918158,
+            -0.0237762245071987,
+            0.005288171058938098,
+            -0.00019047961064753207,
+            0.01822542471604037,
+            0.008127347078975925,
+            0.008361588667386882,
+            -0.011835407590060994,
+            0.010004482616697591,
+            -0.03378107067882622,
+            -0.015433906677849963,
+            -0.007702910250795191,
+            -0.0013877702264578865,
+            -0.002059152251387601,
+            -0.015745218548968338,
+            -0.026393221735795527,
+            0.006294583740809379,
+            0.016267525575472445,
+            0.023291528020980265,
+            -0.004683197930892445,
+            0.012912830736650997,
+            -0.002831583991769044,
+            -0.008904271394710672,
+            -0.011441882849296418,
+            -0.012202629605208118,
+            -0.012470773094383122,
+            -0.014560694782340029,
+            -0.017045701603711442,
+            0.012139665577738337,
+            -0.020079061996873537,
+            0.013911518328924777,
+            0.0347210775522539,
+            -0.017176114991595492,
+            -0.0018941772168839354,
+            0.019702722555622765,
+            -0.004221469612316137,
+            0.01212088353057276,
+            0.0063707489618353725,
+            0.015298014532351866,
+            0.0038643174573634725,
+            -0.012181479717826009,
+            -0.007374005740809251,
+            0.019411457664902376,
+            -0.004902591844710158,
+            0.014591446725717338,
+            0.015069599884557399,
+            -0.008675672164385891,
+            0.002544198867387345,
+            0.01951865904496276,
+            -0.01762101777798757,
+            0.01665654707715257,
+            -0.00458402229480694,
+            -0.009936206685538294,
+            -0.013345740581684638,
+            -0.005336416724426228,
+            -0.019469457091169776,
+            0.0061608454109892425,
+            0.018931676738452464,
+            -0.00533013594306834,
+            0.009310871143727683,
+            -0.0037670793158045,
+            -0.005045151384034014,
+            0.0002041909886640482,
+            -0.04152751596227068,
+            -0.008340911336217609,
+            -0.0006651584370005978,
+            -0.02505748815737022,
+            0.025954102004571416,
+            0.007205602377144382,
+            0.022602344303400163,
+            -0.017168610245491406,
+            0.004459735690564842,
+            -0.005882024209102714,
+            0.013217517324383834,
+            0.02255689669623969,
+            0.0017984057469785134,
+            0.005699128497215882,
+            -0.02353849721739205,
+            -0.0034711576718386185,
+            0.015924913374553738,
+            -0.003604478141616307,
+            -0.0034993903155063817,
+            0.012592308877056896,
+            0.012864938348003884,
+            0.01567418642605425,
+            0.008801753550506258,
+            0.015606443428689292,
+            0.010589933994371628,
+            0.008928721556432368,
+            -0.005805129999077126,
+            -0.001025103064751441,
+            -0.015907755521365925,
+            0.0045874330999415985,
+            0.031701049450012876,
+            0.012594483199773746,
+            0.016783858873835704,
+            -0.01778536268185936,
+            0.00789523134933153,
+            -0.0012706411671644011,
+            0.008325108613530198,
+            -0.001677173669891956,
+            0.029540136288412976,
+            0.011739999605699826,
+            0.015670966746894652,
+            0.008732334090105053,
+            -0.016237731209426752,
+            0.045334690225148,
+            -0.016911928942220152,
+            -0.0021139311364009113,
+            -0.01058970056572469,
+            -0.021705338531028618,
+            0.007324792557651635,
+            -0.006556582104686972,
+            0.0018243188317438144,
+            0.010780292738415671,
+            -0.006471850255151019,
+            0.0006997395548596146,
+            0.003907433192252573,
+            -0.023129313269763224,
+            -0.002550875642832235,
+            -0.02237917712764273,
+            -0.003107775639788617,
+            -0.02621608419628401,
+            -0.00012482032027919036,
+            -0.016207377503696517,
+            -0.0038712739995960753,
+            -0.023315508569288476,
+            -0.014963211090562272,
+            -0.01633110043829169,
+            -0.004241081909202262,
+            -0.008295402942465353,
+            0.004976396647108597,
+            -0.007402107755970892,
+            -0.020321297548927468,
+            0.0020986169430507757,
+            0.016803273183070004,
+            0.01434729320047051,
+            0.018612970083178984,
+            0.00603868809307069,
+            0.014278001198851581,
+            -0.0026350854954985117,
+            0.017171016209107656,
+            -0.006968968261909459,
+            0.006881847166565678,
+            0.011454001418121107,
+            0.02140935294963631,
+            0.011121833270235448,
+            -0.012801756637470525,
+            0.0010870041670931706,
+            -0.009655388034828094,
+            0.002094601818569396,
+            -0.00828486034340067,
+            -0.009190975158350585,
+            0.0017762581946855365,
+            -0.007261996868906865,
+            0.004674405202069334,
+            0.0030526253833799426,
+            -0.0014082232541965316,
+            0.010419302200840906,
+            -0.0024200053377751805,
+            -0.012208685875688886,
+            -0.0033037330470612695,
+            -0.017975940046938498,
+            -0.006958042903103732,
+            -0.0014496522238488239,
+            -0.011711502944602638,
+            -0.01016130418405875,
+            0.003321436189059608,
+            -0.00538282506552848,
+            -0.009353581612407435,
+            0.013884147453136417,
+            0.0001309738696853529,
+            0.0005878330075918333,
+            0.023495280602974504,
+            0.006035028203642483,
+            0.008087594627921345,
+            0.013144288945385904,
+            -0.01023995000994454,
+            -0.022574958230635715,
+            -0.0243435727055662,
+            0.007867022215096418,
+            -0.01573478573423379,
+            0.02324699908478915,
+            -0.007929809922043611,
+            -0.00561630876431978,
+            0.0020982723335057625,
+            -0.013699317926847062,
+            0.008578431496569186,
+            -0.011271381061801356,
+            0.0014450645788554738,
+            -0.019581451799757368,
+            0.018059311800267413,
+            -0.010917911869373693,
+            -0.004302687883018328,
+            0.009466103226841532,
+            0.034049086327052606,
+            -0.0005689605174080737,
+            -0.007410970644432269,
+            -0.006594150877740108,
+            -0.00022810271313786974,
+            0.018766765773409123,
+            -0.0029041815756733333,
+            0.005758781908059541,
+            -0.01563519107805783,
+            0.006571628287171105,
+            -0.006132372252825422,
+            0.0012423261344297225,
+            0.01513074831647933,
+            -0.026306320707079913,
+            -0.015491710256868008,
+            -0.007857352164317122,
+            -0.024893430640551818,
+            0.002253574511878983,
+            0.017579477884206534,
+            0.020505606775521504,
+            -0.020324505029926956,
+            -0.02513332780748072,
+            0.005150966733930881,
+            -0.026580994694344065,
+            0.0037390577082486594,
+            0.1241031878517593,
+            -0.002138404898309707,
+            0.002720723037908521,
+            -0.011425036886011179,
+            0.0022399461009316216,
+            0.005389407121650262,
+            -0.019552543955456567,
+            0.05950746473186985,
+            -0.00638023587296763,
+            -0.004023798021748334,
+            0.002170545188776296,
+            0.001785855390676113,
+            -0.005831534307818096,
+            0.008643501093778434,
+            -0.017602787281866816,
+            -0.02184792441891879,
+            -0.0052215705748838905,
+            -0.021252169339074325,
+            -0.008509621422071373,
+            -0.013000137524907565,
+            0.002893880203681282,
+            -0.015821609145923173,
+            0.010397692426614709,
+            -0.01638916243796692,
+            -0.010283706937701179,
+            -0.01103621454761223,
+            -0.00373552043529251,
+            0.01563553822905027,
+            -0.03188533919653949,
+            -0.004873550301425776,
+            -0.005183590054152726,
+            0.018218971950313273,
+            0.006004328765748325,
+            0.004357734370247252,
+            -0.014230318971458851,
+            0.0064792070801456235,
+            0.0215589173714926,
+            0.0055272493764401305,
+            0.014643335833970546,
+            0.010618070641357003,
+            -0.0022177303813575846,
+            -0.0024402761885758007,
+            0.017541799738501007,
+            -0.005445774414228888,
+            -0.00023502936119626386,
+            -0.00537569046685066,
+            -0.005974703247446444,
+            -0.010712244203425036,
+            0.01843613166635215,
+            -0.009938564643456672,
+            -0.017388592763277214,
+            -0.0015797412875366156,
+            0.02028434083164951,
+            -0.012761534382003973,
+            0.0031906812114950982,
+            0.000788436210275669,
+            0.020589196433948156,
+            0.0007726855314648301,
+            -0.005481260996289263,
+            0.007165014826294827,
+            -0.02956334397512344,
+            0.004095715232997753,
+            0.0003568512047626187,
+            0.027290768799658912,
+            -0.0017631129386615335,
+            0.01552542586826711,
+            -0.020871286597113327,
+            0.004388125824324094,
+            -0.016926008631435403,
+            0.005099899758322398,
+            0.012560923968841436,
+            0.002781049890810361,
+            0.010889021753593997,
+            -0.008500358897706648,
+            -0.021353460542554444,
+            0.012663934009161902,
+            -0.018657094130860155,
+            -0.029156017587920958,
+            0.0018889388761887053,
+            -0.028158076315864772,
+            -0.003048854609750186,
+            0.006812072971079798,
+            -0.035217412591558014,
+            -0.02251675180883659,
+            0.007396524512996277,
+            -0.00995571600201846,
+            -0.011912753666073882,
+            0.010675240017836214,
+            -0.02645155743041798,
+            -0.018137648527851668,
+            0.0010666944556449141,
+            0.012588800077165337,
+            -0.004561825939541096,
+            0.007555558090958502,
+            -0.028558603070876605,
+            0.014816816621445352,
+            -0.01313764261744744,
+            0.016036065063831813,
+            -0.0037797584923144134,
+            0.022217516257304868,
+            0.011131597210722573,
+            0.008795376259031849,
+            -0.004950080558710245,
+            0.021409590863550987,
+            0.005574351961953293,
+            0.051358937509801736,
+            0.031972009383746405,
+            0.0019821003238211276,
+            -0.003251257665378234,
+            0.003313045083311649,
+            0.008294776423730887,
+            -0.00940948257229233,
+            0.005114631360856872,
+            0.028053227377164806,
+            0.016738013668710083,
+            -0.018755924318109216,
+            0.006606711923035118,
+            -0.02702692774986498,
+            0.017950884766781056,
+            -0.022054653845163,
+            0.0003755944703013671,
+            0.007766253634286795,
+            0.0016206993755471613,
+            0.010682347575070117,
+            0.0013842930079185415,
+            0.01705234255261425,
+            0.023975191051210525,
+            -0.002976402980885077,
+            0.017466027107477605,
+            0.012734909275073793,
+            0.018143161773162015,
+            -0.0038399373197265827,
+            0.006929440921265761,
+            -0.012003095293274786,
+            -0.0008883470597690258,
+            0.0020490527756307267,
+            -0.004231091383919565,
+            0.012128848448049833,
+            0.008340302519160299,
+            0.012989849936937843,
+            0.002974488532907779,
+            0.003198854381461613,
+            -0.005309753760578859,
+            -0.0017024982049936384,
+            -0.01763395035509857,
+            -0.01819680226734474,
+            0.028417724274372468,
+            -0.020217879775485576,
+            -0.050118184195774004,
+            0.0025509548991664425,
+            0.014185761006969567,
+            -0.0012021669304388173,
+            -0.0051260576846881515,
+            -0.0005774778872927327,
+            0.005718864494530703,
+            -0.016918749210038116,
+            -0.0015472638835434902,
+            0.008125845907627804,
+            0.020347547062899797,
+            -0.002036218493206914,
+            -0.01079739518866248,
+            0.0038112958713600906,
+            0.02169339303105242,
+            0.01667594322285835,
+            0.012338193092567552,
+            -0.009798409233733167,
+            0.0003198215558702285,
+            -1.6308740825163337e-05,
+            0.0076870827874737325,
+            0.004356644356656038,
+            0.003500023005601613,
+            -0.019968549869439228,
+            -0.008634740687755112,
+            0.012781565963545847,
+            -0.0025637758121512927,
+            0.010243567793632335,
+            -0.00042296044205132426,
+            -0.003069251553118065,
+            -0.009344199449486244,
+            -0.004694953215957636,
+            -0.009583993744231791,
+            0.013783878036521561,
+            -0.007705982042252044,
+            -0.01549567538665377,
+            0.00020666298495823114,
+            0.0005515775451348472,
+            0.001667367278489879,
+            -0.004518431526236256,
+            0.03302324454129141,
+            0.009531990824506103,
+            -0.0004937348108340977,
+            -0.004047613286672229,
+            -0.0327383112522095,
+            0.012456150771224677,
+            -0.009677533214778067,
+            0.0006455811519981003,
+            0.020012976827426605,
+            -0.00020277620481825933,
+            0.002168432791818704,
+            -0.015657551629104767,
+            0.0032924357559238502,
+            0.006161295735841883,
+            0.013814165416788555,
+            0.0005164426482972658,
+            0.01076512534105537,
+            -0.006498977875968572,
+            -0.010325144017898257,
+            0.004804158067334203,
+            0.0104064855510473,
+            -0.0012188100892986933,
+            0.005171197790296134
+          ],
+          "title": "Mixture-Of-Experts With Expert Choice Routing",
+          "keyphrases": [
+            "Mixture-Of-Experts",
+            "Expert Choice Routing",
+            "sparsely-activated models",
+            "training efficiency",
+            "downstream performance"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "d08eab82-ce8c-4126-8aee-d1681de9b6e5",
+        "properties": {
+          "page_content": "5 Conclusion\n\nWe propose a new routing method for sparsely activated mixture-of-experts (MoE) models. This method addresses load imbalance and under-utilization of experts in conventional MoE methods, and enables selecting different numbers of experts for each token. Our model demonstrates more than 2x training efficiency improvements when compared to the state-of-the-art GShard and Switch Transformer models, and also achieves strong gains when finetuning on 11 datasets in the GLUE and SuperGLUE benchmark.\n\n",
+          "title": "5 Conclusion",
+          "embedding": [
+            -0.014961956068873405,
+            -0.008189860731363297,
+            -0.023845616728067398,
+            0.002967504784464836,
+            -0.00690029701218009,
+            0.024916483089327812,
+            -0.0020832864101976156,
+            0.07137094438076019,
+            -0.014486853033304214,
+            0.04606231302022934,
+            -0.016666291281580925,
+            0.003244647989049554,
+            0.0073112985119223595,
+            -0.015520012937486172,
+            -0.04539867490530014,
+            0.030662959441542625,
+            0.012692022137343884,
+            0.0035896627232432365,
+            0.013717640191316605,
+            -0.03864166513085365,
+            -0.022352438420057297,
+            -0.008853496052324772,
+            -0.018159471452236176,
+            0.0027733161114156246,
+            0.019019179046154022,
+            -0.0032974369823932648,
+            0.000584922730922699,
+            0.008634798228740692,
+            -0.03864166513085365,
+            0.05327180400490761,
+            0.011213925667107105,
+            0.009728288277983665,
+            -0.0011019736994057894,
+            -0.001177386729978025,
+            -0.03257845342159271,
+            -0.007563932333141565,
+            0.07293953746557236,
+            0.025715861469507217,
+            -0.05375444516539574,
+            0.029727837070822716,
+            0.03912430629134178,
+            0.042442481964826584,
+            -0.03233712911605835,
+            -0.008084282279014587,
+            -0.014434064738452435,
+            -0.0042646098881959915,
+            -0.0013913713628426194,
+            -0.005045135505497456,
+            0.02479582279920578,
+            -0.01170411054044962,
+            0.0019154923502355814,
+            -0.010957521386444569,
+            0.014728175476193428,
+            0.0002771430881693959,
+            -0.04500652849674225,
+            0.03791769966483116,
+            0.018596867099404335,
+            0.017103686928749084,
+            0.03366440162062645,
+            0.007273592054843903,
+            -0.00380270485766232,
+            -0.01829521358013153,
+            -0.018657196313142776,
+            0.0008644224726594985,
+            0.003012752626091242,
+            0.00490185059607029,
+            -0.0039478749968111515,
+            -0.011915267445147038,
+            0.037223897874355316,
+            -0.010565373115241528,
+            -0.06045112758874893,
+            0.03665075823664665,
+            -0.04114537686109543,
+            0.0005467448499985039,
+            -0.0135818962007761,
+            -0.0067796362563967705,
+            0.0271637924015522,
+            0.014562266878783703,
+            -0.00047156744403764606,
+            0.001911721657961607,
+            0.012073635123670101,
+            0.00981124211102724,
+            0.0072698211297392845,
+            -0.007443271577358246,
+            0.05547386407852173,
+            -0.044704876840114594,
+            0.006198955699801445,
+            -0.05541353300213814,
+            0.03659042716026306,
+            -0.0024546959903091192,
+            -0.03007473796606064,
+            -0.021115664392709732,
+            -0.021779298782348633,
+            0.008619715459644794,
+            -0.0393354631960392,
+            0.033362749963998795,
+            0.006523231975734234,
+            -0.013197289779782295,
+            -0.008989240042865276,
+            0.021145828068256378,
+            -0.004758565686643124,
+            0.015218360349535942,
+            -0.015444600023329258,
+            0.008808248676359653,
+            0.05411642789840698,
+            -0.033151593059301376,
+            -0.019366079941391945,
+            0.044644545763731,
+            0.01785781793296337,
+            0.05526270717382431,
+            0.03119085170328617,
+            0.0022322272416204214,
+            0.01181722991168499,
+            0.01852145418524742,
+            -0.003199400147423148,
+            -0.028581559658050537,
+            -0.04585115611553192,
+            -0.05535320192575455,
+            0.023498717695474625,
+            -0.028551394119858742,
+            0.022548511624336243,
+            0.020633019506931305,
+            -0.009117442183196545,
+            -0.023845616728067398,
+            -0.03906397894024849,
+            0.01708860509097576,
+            0.0047095469199121,
+            -0.005278916098177433,
+            -0.012586443684995174,
+            -0.016877448186278343,
+            0.00349162588827312,
+            -0.01137983426451683,
+            -0.02776709757745266,
+            -0.03945612534880638,
+            0.016651209443807602,
+            -0.006040588021278381,
+            0.022186528891324997,
+            0.027917923405766487,
+            -0.020135292783379555,
+            -0.019350998103618622,
+            0.013732722960412502,
+            -0.04171851649880409,
+            0.030285894870758057,
+            0.01812930591404438,
+            -0.005037594120949507,
+            0.010459794662892818,
+            0.0033125197514891624,
+            0.053965602070093155,
+            -0.026213588193058968,
+            0.020844176411628723,
+            -0.04105488210916519,
+            0.04775156453251839,
+            -0.005836972501128912,
+            0.03984827175736427,
+            -0.024117104709148407,
+            -0.013838300481438637,
+            -0.01322745531797409,
+            -0.017556166276335716,
+            -0.006719305645674467,
+            -0.022020621225237846,
+            -0.0005962346913293004,
+            -0.0063196164555847645,
+            -0.0033030931372195482,
+            0.03646976873278618,
+            0.014072081074118614,
+            0.021628472954034805,
+            0.041507359594106674,
+            0.0015572801930829883,
+            -0.07173292338848114,
+            0.030828868970274925,
+            0.007684593088924885,
+            0.030919363722205162,
+            0.006839966867119074,
+            0.015897078439593315,
+            0.006066982634365559,
+            0.021387150511145592,
+            0.002356658922508359,
+            0.004947098437696695,
+            0.005754018202424049,
+            0.04135653376579285,
+            -0.02336297370493412,
+            0.001117998966947198,
+            0.015218360349535942,
+            0.012488407082855701,
+            0.03372473269701004,
+            0.023061320185661316,
+            0.016832200810313225,
+            0.015957407653331757,
+            0.004430518485605717,
+            0.0057163117453455925,
+            0.027118545025587082,
+            0.014901624992489815,
+            -0.0032898958306759596,
+            -0.020753679797053337,
+            0.0024188747629523277,
+            -0.024599747732281685,
+            -0.002049350645393133,
+            -0.01785781793296337,
+            0.034961506724357605,
+            0.0012782517587766051,
+            -0.026183422654867172,
+            -0.014984579756855965,
+            0.005836972501128912,
+            -0.03613794967532158,
+            -0.0010991457384079695,
+            0.0015563375782221556,
+            0.016952861100435257,
+            0.005870908498764038,
+            0.013981585390865803,
+            -0.03405654802918434,
+            0.012141506187617779,
+            -0.023679709061980247,
+            -0.006142395548522472,
+            0.004988575354218483,
+            -0.0007941940566524863,
+            0.013423528522253036,
+            -0.01253365445882082,
+            0.02022578939795494,
+            -0.005361870396882296,
+            -0.028837963938713074,
+            -0.007752465084195137,
+            0.02462991327047348,
+            0.004011976066976786,
+            -0.006097147706896067,
+            -0.010210931301116943,
+            -0.01613840088248253,
+            -0.030723290517926216,
+            -0.017948314547538757,
+            0.03155283257365227,
+            0.014449146576225758,
+            -0.015489847399294376,
+            0.020527441054582596,
+            0.007394252810627222,
+            0.0028939771000295877,
+            -0.014924249611794949,
+            -0.05559452623128891,
+            -0.009969609789550304,
+            -0.012797600589692593,
+            -0.000906370987650007,
+            0.0007253796211443841,
+            -0.04506685957312584,
+            0.02573094516992569,
+            -0.03674125298857689,
+            -0.0009040143340826035,
+            0.015791499987244606,
+            0.006938003934919834,
+            -0.006127313245087862,
+            0.011666404083371162,
+            -0.026590652763843536,
+            0.01303138118237257,
+            -0.030059656128287315,
+            -0.003970498684793711,
+            6.586861854884773e-05,
+            0.015173112042248249,
+            -0.023227229714393616,
+            0.034629687666893005,
+            0.021839629858732224,
+            -0.06274368613958359,
+            -0.008318062871694565,
+            -0.01835554465651512,
+            0.05456890910863876,
+            -0.01449439488351345,
+            -0.008144613355398178,
+            0.052306514233350754,
+            -0.009441718459129333,
+            0.002309525618329644,
+            0.031371843069791794,
+            0.027118545025587082,
+            0.01377042941749096,
+            -0.02897370606660843,
+            0.002942995633929968,
+            -0.07517176121473312,
+            -0.012488407082855701,
+            0.013415987603366375,
+            0.034086715430021286,
+            -0.03063279390335083,
+            0.0013838300947099924,
+            0.015474764630198479,
+            0.01895884983241558,
+            0.03170366212725639,
+            0.002475434448570013,
+            0.02963734231889248,
+            0.010452253744006157,
+            0.01011289469897747,
+            0.004140178207308054,
+            -0.004935786128044128,
+            -0.028340237215161324,
+            -0.0015544522320851684,
+            -0.03167349472641945,
+            0.0005425028502941132,
+            -0.008099365048110485,
+            -0.014094705693423748,
+            0.004355105571448803,
+            0.005403347313404083,
+            0.04651479050517082,
+            -0.006949315778911114,
+            -0.012563819997012615,
+            -0.01306154578924179,
+            -0.0002839773951563984,
+            -0.04144703224301338,
+            -0.0399991013109684,
+            0.014946873299777508,
+            0.005105465650558472,
+            -0.023709872737526894,
+            -0.0516730435192585,
+            -0.012880554422736168,
+            -0.022895412519574165,
+            0.035564810037612915,
+            -0.022729502990841866,
+            -0.012805141508579254,
+            0.0416581891477108,
+            -0.003783851396292448,
+            0.03731439262628555,
+            -0.057494934648275375,
+            -0.008604632690548897,
+            0.006545855663716793,
+            -0.005260062403976917,
+            0.014841294847428799,
+            -0.026424745097756386,
+            -0.024056773632764816,
+            -0.010361758060753345,
+            -0.015180653892457485,
+            -0.04760073870420456,
+            0.0017759781330823898,
+            -0.02028612047433853,
+            0.017118770629167557,
+            0.046152807772159576,
+            0.03052721731364727,
+            -0.02469024434685707,
+            0.014072081074118614,
+            -0.008385934866964817,
+            -0.054689567536115646,
+            -0.025595201179385185,
+            0.010097811929881573,
+            -0.020738597959280014,
+            -0.041235875338315964,
+            -0.02043694630265236,
+            0.0005128089687786996,
+            -0.022669173777103424,
+            0.016168564558029175,
+            -0.024539416655898094,
+            -0.03360407054424286,
+            0.000932294235099107,
+            0.0206631850451231,
+            -0.0028920918703079224,
+            -0.017224349081516266,
+            -0.02153797633945942,
+            -0.02562536671757698,
+            0.013793053105473518,
+            -0.004573803395032883,
+            0.0009091990068554878,
+            0.019592318683862686,
+            0.03254828602075577,
+            0.025534870103001595,
+            0.0035406441893428564,
+            -0.03333258256316185,
+            -0.021191077306866646,
+            0.026032596826553345,
+            0.0018824990838766098,
+            -0.0013084170641377568,
+            0.03245779126882553,
+            -0.03653009608387947,
+            -0.04325694590806961,
+            0.005554173607379198,
+            -0.04910900071263313,
+            0.0020418092608451843,
+            -0.021462563425302505,
+            -0.022503264248371124,
+            -0.013898631557822227,
+            0.002277475083246827,
+            0.003847952699288726,
+            0.02909436821937561,
+            0.011658863164484501,
+            -0.010482418350875378,
+            -0.023438386619091034,
+            0.008521678857505322,
+            0.0483548678457737,
+            -0.010060105472803116,
+            -0.02710346318781376,
+            0.03278961032629013,
+            -0.016455134376883507,
+            0.01168148685246706,
+            -0.011425082571804523,
+            -0.022895412519574165,
+            -0.01592724397778511,
+            0.017827652394771576,
+            0.01895884983241558,
+            0.006172561086714268,
+            -0.041235875338315964,
+            0.025474539026618004,
+            -0.0063497815281152725,
+            -0.03840034082531929,
+            0.014056999236345291,
+            0.004128866363316774,
+            0.01239791139960289,
+            0.0005486301961354911,
+            0.013174666091799736,
+            -0.054297421127557755,
+            -0.005844513885676861,
+            0.006036817561835051,
+            -0.03191481530666351,
+            -0.0058595966547727585,
+            -0.008378393948078156,
+            0.02280491590499878,
+            -0.014946873299777508,
+            0.015391810797154903,
+            -0.012405452318489552,
+            0.008144613355398178,
+            0.031975146383047104,
+            -0.03441853076219559,
+            -0.015067534521222115,
+            -0.036228444427251816,
+            -0.0041929674334824085,
+            -0.022458016872406006,
+            0.020361533388495445,
+            0.04334744065999985,
+            -0.017661744728684425,
+            0.00917777232825756,
+            0.0028185639530420303,
+            -0.004241986200213432,
+            -0.022005537524819374,
+            0.016817117109894753,
+            0.0569821260869503,
+            0.031100355088710785,
+            -0.008039034903049469,
+            -0.0006443105521611869,
+            0.02197537198662758,
+            -0.013121876865625381,
+            -0.008340687490999699,
+            0.006051899865269661,
+            0.017752239480614662,
+            0.010701116174459457,
+            0.02286524698138237,
+            -0.040089596062898636,
+            -0.010150601156055927,
+            0.009109900332987309,
+            -0.02469024434685707,
+            0.027812344953417778,
+            -0.03073837235569954,
+            0.018973931670188904,
+            0.017812570556998253,
+            -0.01956215314567089,
+            0.013627144508063793,
+            -0.010022399015724659,
+            -0.023830534890294075,
+            -0.02512763999402523,
+            -0.02496173046529293,
+            -0.008634798228740692,
+            0.017978480085730553,
+            -0.0139966681599617,
+            -0.05287965387105942,
+            0.04301562160253525,
+            -0.06298501044511795,
+            -0.029773086309432983,
+            0.03191481530666351,
+            -0.02027103677392006,
+            -0.014313403517007828,
+            0.008672504685819149,
+            0.01570100337266922,
+            -0.033362749963998795,
+            0.020452028140425682,
+            0.011643780395388603,
+            0.033151593059301376,
+            -0.028023501858115196,
+            -0.024931564927101135,
+            0.0034765431191772223,
+            -0.02739003300666809,
+            -0.006010422948747873,
+            -0.0002648884546943009,
+            0.014667844399809837,
+            0.00637617614120245,
+            -0.033754896372556686,
+            -0.009743370115756989,
+            0.003800819395110011,
+            0.0027506921906024218,
+            0.03348340839147568,
+            -0.004132636822760105,
+            -0.04205033555626869,
+            0.014170118607580662,
+            0.004321169573813677,
+            -0.03284993767738342,
+            0.006406341679394245,
+            0.003346455516293645,
+            -0.009185313247144222,
+            -0.016334474086761475,
+            0.005478760693222284,
+            0.03221647068858147,
+            0.030210481956601143,
+            0.017812570556998253,
+            0.007118995301425457,
+            0.009871572256088257,
+            -0.01338582206517458,
+            -0.01503736898303032,
+            0.010655868798494339,
+            -0.007782630156725645,
+            -0.03191481530666351,
+            -0.013204830698668957,
+            -0.014780964702367783,
+            0.025866687297821045,
+            -0.012563819997012615,
+            -0.01253365445882082,
+            0.005980257410556078,
+            0.045157354325056076,
+            0.014615056104958057,
+            -0.005267603788524866,
+            -0.01336319837719202,
+            0.007601638790220022,
+            0.012707104906439781,
+            -0.034358199685811996,
+            -0.022277025505900383,
+            0.022065868601202965,
+            0.0023283790796995163,
+            0.017375174909830093,
+            -0.024599747732281685,
+            -0.022216694429516792,
+            0.03239746019244194,
+            -0.03248795494437218,
+            0.030919363722205162,
+            0.01580658182501793,
+            0.006202726159244776,
+            -0.006444048136472702,
+            0.004739712458103895,
+            -0.014569807797670364,
+            0.028400568291544914,
+            -0.029350772500038147,
+            0.027193957939743996,
+            0.020421862602233887,
+            -0.006285680457949638,
+            0.060300301760435104,
+            -0.029682589694857597,
+            -0.016334474086761475,
+            0.009607627056539059,
+            -0.0043777297250926495,
+            0.015391810797154903,
+            -0.019094591960310936,
+            -0.007149160373955965,
+            -0.0060745240189135075,
+            0.005018740892410278,
+            -0.029667507857084274,
+            -0.015897078439593315,
+            0.02164355479180813,
+            -0.005987798795104027,
+            -0.0013272702926769853,
+            0.009660416282713413,
+            -0.02286524698138237,
+            0.017495835199952126,
+            -0.006542085204273462,
+            -0.04533834755420685,
+            -0.00045813448377884924,
+            0.02153797633945942,
+            -0.024388590827584267,
+            -0.015912160277366638,
+            -0.008582009002566338,
+            0.0038837739266455173,
+            -0.00595009233802557,
+            -0.02280491590499878,
+            -0.019245419651269913,
+            -0.005655981134623289,
+            -0.008385934866964817,
+            -0.027480527758598328,
+            -0.011779523454606533,
+            0.034086715430021286,
+            0.01294842641800642,
+            -0.004875455982983112,
+            -0.010301426984369755,
+            -0.029667507857084274,
+            0.016002656891942024,
+            0.04373958706855774,
+            0.0034218686632812023,
+            0.006941774394363165,
+            -0.0065948739647865295,
+            -0.0026451139710843563,
+            -0.014841294847428799,
+            0.009864031337201595,
+            0.0017646661726757884,
+            -0.0028223346453160048,
+            0.01790306530892849,
+            -0.011342127807438374,
+            -0.03381522744894028,
+            -0.026967719197273254,
+            0.04304578900337219,
+            -0.023785287514328957,
+            0.004483307711780071,
+            0.04651479050517082,
+            0.036620594561100006,
+            -0.005343017168343067,
+            -0.014305861666798592,
+            -0.03107019141316414,
+            0.0020663184113800526,
+            -0.02010512910783291,
+            0.009509589523077011,
+            0.03294043615460396,
+            -0.00039686134550720453,
+            -0.034297872334718704,
+            -0.013529106974601746,
+            -0.02099500223994255,
+            -0.010346675291657448,
+            0.015716087073087692,
+            0.017013192176818848,
+            -0.034358199685811996,
+            0.005795495584607124,
+            0.018310297280550003,
+            0.028943542391061783,
+            -0.011349668726325035,
+            -0.016983026638627052,
+            0.024765657261013985,
+            0.017224349081516266,
+            0.0009294662741012871,
+            -0.0029222569428384304,
+            -0.00034312953357584774,
+            0.0068098013289272785,
+            -0.004068535752594471,
+            0.033030930906534195,
+            0.0035274471156299114,
+            0.011206384748220444,
+            -0.02116091176867485,
+            0.00479250168427825,
+            0.020738597959280014,
+            -0.006870131939649582,
+            -0.001355550135485828,
+            0.07384449243545532,
+            0.00214173155836761,
+            0.01355173159390688,
+            0.017209265381097794,
+            0.03444869816303253,
+            0.01773715764284134,
+            -0.017993561923503876,
+            -0.03170366212725639,
+            -0.014313403517007828,
+            -0.007718529086560011,
+            -0.020527441054582596,
+            -0.0049697221256792545,
+            -0.012910719960927963,
+            -0.0433172769844532,
+            -0.0223222728818655,
+            -0.0033502262085676193,
+            -0.002466007834300399,
+            0.025263383984565735,
+            -0.0026319166645407677,
+            0.0009822554420679808,
+            -0.027948088943958282,
+            0.03565530478954315,
+            0.02622867003083229,
+            0.023996442556381226,
+            -0.007880667224526405,
+            0.011100806295871735,
+            -0.0043362523429095745,
+            -0.008491513319313526,
+            0.02754085883498192,
+            0.0074885194189846516,
+            -0.008137071505188942,
+            0.03478051349520683,
+            0.012111341580748558,
+            0.05272882804274559,
+            -0.013114335015416145,
+            -0.024343343451619148,
+            -0.017179099842905998,
+            -0.04018009081482887,
+            0.011372293345630169,
+            -0.007710987702012062,
+            0.0001368040539091453,
+            0.014139953069388866,
+            0.017390256747603416,
+            -0.008167237043380737,
+            0.0271637924015522,
+            0.0038498379290103912,
+            -0.015504930168390274,
+            0.025187969207763672,
+            -0.02892845869064331,
+            0.014253072440624237,
+            -0.0035576121881604195,
+            0.011839854530990124,
+            -0.012903179042041302,
+            0.011538201943039894,
+            0.02358921244740486,
+            -0.025761108845472336,
+            0.028551394119858742,
+            0.017254512757062912,
+            -0.0027412655763328075,
+            0.00480758398771286,
+            0.0026168341282755136,
+            -0.0012169786496087909,
+            -0.022080950438976288,
+            -0.034086715430021286,
+            0.02533879689872265,
+            -0.007473436649888754,
+            -0.023332808166742325,
+            0.02425284869968891,
+            -0.017118770629167557,
+            -0.008928908966481686,
+            -0.0017175329849123955,
+            -0.009358763694763184,
+            -0.007579015102237463,
+            0.04292512685060501,
+            -0.008664963766932487,
+            -0.004291004501283169,
+            -0.01640988700091839,
+            0.02689230628311634,
+            0.04226149246096611,
+            -0.008385934866964817,
+            -0.009072193875908852,
+            0.021899959072470665,
+            -0.012450700625777245,
+            0.0015129749663174152,
+            0.041628021746873856,
+            0.006153707858175039,
+            0.0015365416184067726,
+            0.005712541285902262,
+            -0.03393588960170746,
+            -0.029999325051903725,
+            0.005806807428598404,
+            0.016455134376883507,
+            0.016047904267907143,
+            0.04126603901386261,
+            0.03646976873278618,
+            -0.014969496987760067,
+            0.0025037142913788557,
+            0.0007546021952293813,
+            -0.035232994705438614,
+            0.028747467324137688,
+            -0.005486302077770233,
+            0.021447481587529182,
+            0.02485615201294422,
+            -0.026183422654867172,
+            -0.036620594561100006,
+            -0.00034996384056285024,
+            0.030210481956601143,
+            0.0190040972083807,
+            0.014056999236345291,
+            0.014072081074118614,
+            0.023227229714393616,
+            -0.020316284149885178,
+            -0.024946648627519608,
+            0.02012021094560623,
+            0.02506730891764164,
+            -0.009456800296902657,
+            0.01074636448174715,
+            0.017933230847120285,
+            0.009818783961236477,
+            -0.0012282906100153923,
+            0.03134167939424515,
+            0.009019404649734497,
+            0.016289226710796356,
+            0.008287898264825344,
+            -0.011515578255057335,
+            0.02500697784125805,
+            0.010271262377500534,
+            0.03393588960170746,
+            -0.012412993237376213,
+            0.003929021768271923,
+            0.003689585020765662,
+            -0.004087388981133699,
+            0.037163566797971725,
+            0.005946321878582239,
+            -0.04307595267891884,
+            0.0199995506554842,
+            0.011960514821112156,
+            -0.0001776920835254714,
+            -0.006489295978099108,
+            -0.0026469992008060217,
+            0.018446041271090508,
+            0.02043694630265236,
+            0.018038809299468994,
+            0.009532214142382145,
+            0.005659752059727907,
+            0.005618274677544832,
+            0.006138625089079142,
+            0.03167349472641945,
+            0.0020455799531191587,
+            -0.01978839375078678,
+            -0.00230009900406003,
+            0.005505155306309462,
+            0.021115664392709732,
+            -0.019637567922472954,
+            0.01653054729104042,
+            0.019215254113078117,
+            -0.0011934119975194335,
+            -0.01746566966176033,
+            -0.03671108931303024,
+            -0.007303757127374411,
+            -0.013853383250534534,
+            0.013898631557822227,
+            0.008951532654464245,
+            0.018687361851334572,
+            -0.0221563633531332,
+            0.016078069806098938,
+            0.019592318683862686,
+            -0.01950182393193245,
+            0.006496837362647057,
+            0.008061658591032028,
+            -0.014524560421705246,
+            -0.005580568220466375,
+            -0.016108235344290733,
+            0.002053121104836464,
+            0.06382963061332703,
+            -0.014464229345321655,
+            0.009954527020454407,
+            0.024931564927101135,
+            -0.009879114106297493,
+            -0.030119985342025757,
+            0.0009671727893874049,
+            -0.010429629124701023,
+            -0.002450925298035145,
+            0.024373508989810944,
+            -0.03577596694231033,
+            -0.04319661483168602,
+            -0.005290227942168713,
+            -0.03631894290447235,
+            -0.006560938432812691,
+            0.019034262746572495,
+            0.027948088943958282,
+            -0.014162576757371426,
+            -0.02760118804872036,
+            0.02292557805776596,
+            -0.0020418092608451843,
+            0.009765994735062122,
+            -0.019471658393740654,
+            -0.026545405387878418,
+            -0.03030097670853138,
+            -0.03212597221136093,
+            -0.00961516797542572,
+            -0.025097474455833435,
+            0.008408558554947376,
+            0.0012725958367809653,
+            -0.05061726272106171,
+            -0.030602630227804184,
+            0.010670951567590237,
+            -0.002435842528939247,
+            0.04005942866206169,
+            0.039757777005434036,
+            0.003970498684793711,
+            0.028008420020341873,
+            -0.03613794967532158,
+            0.050828419625759125,
+            -0.006530772894620895,
+            0.03592679277062416,
+            -0.007579015102237463,
+            0.0019890200346708298,
+            -0.014358650892972946,
+            0.04379991814494133,
+            0.000892702373676002,
+            0.002547076903283596,
+            -0.038762323558330536,
+            0.0021455022506415844,
+            -0.023996442556381226,
+            -0.00025451916735619307,
+            -0.0026752790436148643,
+            0.008996780961751938,
+            -0.0009907394414767623,
+            -0.014856377616524696,
+            0.002102139638736844,
+            0.03345324471592903,
+            0.02616834081709385,
+            -0.021779298782348633,
+            -0.027088379487395287,
+            -0.030617712065577507,
+            0.0004138292861171067,
+            -0.008725293911993504,
+            -0.002880779793485999,
+            -0.004860373213887215,
+            -0.024313177913427353,
+            0.014396357350051403,
+            -0.016771869733929634,
+            -0.013415987603366375,
+            -0.002205832628533244,
+            0.02545945718884468,
+            -0.01746566966176033,
+            -0.02259376086294651,
+            -0.005814348813146353,
+            -0.029335688799619675,
+            -0.01675678789615631,
+            0.00022930291015654802,
+            -0.02485615201294422,
+            0.017510918900370598,
+            0.01609315164387226,
+            -0.012103799730539322,
+            0.0002860983950085938,
+            0.0022001767065376043,
+            0.016123317182064056,
+            -0.021794380620121956,
+            -0.004166572820395231,
+            0.0025357650592923164,
+            -0.00017439275688957423,
+            -0.023151816800236702,
+            0.021387150511145592,
+            -0.019411327317357063,
+            0.01631939224898815,
+            -0.04844536632299423,
+            -0.011213925667107105,
+            -0.025037143379449844,
+            0.010587996803224087,
+            0.001273538451641798,
+            -0.02990882843732834,
+            -0.014072081074118614,
+            0.00785050168633461,
+            -0.0031447256915271282,
+            -0.02182454615831375,
+            0.008506596088409424,
+            0.03541398420929909,
+            -0.0014385045506060123,
+            0.009683039970695972,
+            -0.029773086309432983,
+            -0.011590991169214249,
+            -0.010158142074942589,
+            0.03079870343208313,
+            0.0037574570160359144,
+            -0.005131860263645649,
+            -0.00574270635843277,
+            0.0005769100971519947,
+            -0.03472018241882324,
+            -0.021899959072470665,
+            -0.003259730525314808,
+            -0.002062547951936722,
+            -0.005392035469412804,
+            0.034509025514125824,
+            0.03405654802918434,
+            0.03686191514134407,
+            -0.04379991814494133,
+            0.019215254113078117,
+            -0.008325604721903801,
+            -0.011108347214758396,
+            0.01325007900595665,
+            0.018777858465909958,
+            0.01840079203248024,
+            -0.016485299915075302,
+            0.001463956548832357,
+            0.003412442049011588,
+            0.013536648824810982,
+            -0.005603192374110222,
+            0.022985907271504402,
+            -0.00719063775613904,
+            0.013408446684479713,
+            0.014260614290833473,
+            -8.476607035845518e-06,
+            0.022955743595957756,
+            -0.006843737326562405,
+            0.009901737794280052,
+            -0.007043581921607256,
+            -0.014094705693423748,
+            0.0017872900934889913,
+            -0.03882265463471413,
+            -0.013016298413276672,
+            0.015384268946945667,
+            0.018777858465909958,
+            -0.012495948001742363,
+            -0.04874701797962189,
+            0.022020621225237846,
+            0.03671108931303024,
+            -0.019275585189461708,
+            0.0273297019302845,
+            -0.0279933363199234,
+            0.016334474086761475,
+            0.011613614857196808,
+            0.013272702693939209,
+            0.004773647990077734,
+            0.020738597959280014,
+            -0.02512763999402523,
+            0.033966053277254105,
+            -0.0019550842698663473,
+            0.025761108845472336,
+            0.016847282648086548,
+            -0.015391810797154903,
+            -0.04573049396276474,
+            -0.016108235344290733,
+            0.006753241643309593,
+            -0.012028386816382408,
+            -0.002897747792303562,
+            -0.043498266488313675,
+            -0.03420737385749817,
+            0.030210481956601143,
+            0.01418520137667656,
+            -0.0493503212928772,
+            0.00014446319255512208,
+            -0.014034374617040157,
+            -0.0030730832368135452,
+            -0.0052261268720030785,
+            -0.028943542391061783,
+            -0.02336297370493412,
+            0.00801641121506691,
+            0.00752245495095849,
+            0.03981810808181763,
+            -0.023046238347887993,
+            0.026741480454802513,
+            -0.01159853208810091,
+            -0.01719418354332447,
+            -0.02005987986922264,
+            -0.00045224284986034036,
+            -0.021326819434762,
+            -0.01481867115944624,
+            0.005094153806567192,
+            0.020738597959280014,
+            0.009660416282713413,
+            0.029396019876003265,
+            -0.012722187675535679,
+            -0.014856377616524696,
+            -0.0005095096421428025,
+            -0.03465985506772995,
+            -0.028506144881248474,
+            0.007032270077615976,
+            0.0006749471067450941,
+            -0.017269596457481384,
+            -0.02070843242108822,
+            -0.03704290837049484,
+            0.016213813796639442,
+            0.011990680359303951,
+            0.01692269556224346,
+            0.03245779126882553,
+            0.0198336411267519,
+            -0.01386092510074377,
+            0.01669645681977272,
+            -0.007888208143413067,
+            0.0009813127107918262,
+            0.01013551838696003,
+            0.018581783398985863,
+            -0.013114335015416145,
+            -0.0027733161114156246,
+            -0.014765881933271885,
+            -0.029003871604800224,
+            -0.01708860509097576,
+            -0.01806897483766079,
+            -0.0008795050671324134,
+            -0.006149936933070421,
+            -0.03891315311193466,
+            -0.013415987603366375,
+            0.024991896003484726,
+            0.02396627888083458,
+            -0.01534656248986721,
+            -0.007511143106967211,
+            -0.00023295573191717267,
+            -0.0208290945738554,
+            -0.004894309211522341,
+            -0.02342330478131771,
+            -0.003672617254778743,
+            -0.024147270247340202,
+            0.013234996236860752,
+            -0.0045813447795808315,
+            0.010052564553916454,
+            -0.053694114089012146,
+            -0.013174666091799736,
+            0.007326381281018257,
+            -0.010105352848768234,
+            -0.0016996223712339997,
+            0.01746566966176033,
+            -0.003291781060397625,
+            -0.009162689559161663,
+            -0.023845616728067398,
+            0.014155035838484764,
+            0.005772871430963278,
+            0.021025167778134346,
+            -0.002831761259585619,
+            -0.00406476529315114,
+            0.03348340839147568,
+            0.025384044274687767,
+            -5.541094287764281e-05,
+            -0.012269708327949047,
+            0.015897078439593315,
+            0.014788505621254444,
+            0.0014507592422887683,
+            0.0044493721798062325,
+            0.009102359414100647,
+            0.0048264372162520885,
+            -0.010889649391174316,
+            -0.00480758398771286,
+            0.024871235713362694,
+            0.006527002435177565,
+            0.005935009568929672,
+            -0.01463013794273138,
+            0.02897370606660843,
+            0.03752554953098297,
+            0.0196677315980196,
+            0.028068749234080315,
+            -0.015685921534895897,
+            -0.011032934300601482,
+            -0.007760006468743086,
+            -0.013762887567281723,
+            0.031432174146175385,
+            0.008861036971211433,
+            -0.009871572256088257,
+            -0.0104145472869277,
+            0.010037481784820557,
+            -0.009939444251358509,
+            0.002166240941733122,
+            0.02269933745265007,
+            0.009592544287443161,
+            0.005580568220466375,
+            -0.00028892638511024415,
+            0.007548849564045668,
+            0.015912160277366638,
+            0.023770203813910484,
+            -0.02975800260901451,
+            -0.027420196682214737,
+            -0.0009120269678533077,
+            0.0003007096820510924,
+            0.012993674725294113,
+            0.01767682656645775,
+            -0.0011594761162996292,
+            0.006658975500613451,
+            0.04334744065999985,
+            -0.015746252611279488,
+            -0.0034369511995464563,
+            -0.014698009938001633,
+            -0.013272702693939209,
+            -0.0022491952404379845,
+            -0.021945208311080933,
+            0.008687587454915047,
+            0.012382828630506992,
+            0.017118770629167557,
+            0.014245531521737576,
+            -0.008023952133953571,
+            0.012322497554123402,
+            0.018446041271090508,
+            0.011515578255057335,
+            -0.004219362046569586,
+            -0.014780964702367783,
+            -0.002564044902101159,
+            0.015248525887727737,
+            0.009034487418830395,
+            0.022835081443190575,
+            0.021326819434762,
+            0.006949315778911114,
+            0.01322745531797409,
+            -0.025052227079868317,
+            0.025746027007699013,
+            0.0144114401191473,
+            0.02419251762330532,
+            -0.03770654276013374,
+            -0.021145828068256378,
+            0.014501935802400112,
+            -0.007141618989408016,
+            -0.0004892423748970032,
+            0.015384268946945667,
+            -0.008091824129223824,
+            0.05125073343515396,
+            -0.0353536531329155,
+            0.00845380686223507,
+            0.02358921244740486,
+            -0.015881994739174843,
+            0.010271262377500534,
+            0.01818963512778282,
+            0.010595538653433323,
+            -0.010678492486476898,
+            -0.02093467302620411,
+            0.00037730109761469066,
+            0.02551978826522827,
+            -0.03818918392062187,
+            -0.018611948937177658,
+            -0.0065345438197255135,
+            -0.0004892423748970032,
+            0.022729502990841866,
+            -0.007552620489150286,
+            0.014645220711827278,
+            -0.008928908966481686,
+            0.011432623490691185,
+            0.001585560035891831,
+            -0.03134167939424515,
+            -0.007537537720054388,
+            0.014637679792940617,
+            0.01231495663523674,
+            -0.009667957201600075,
+            -0.006674057804048061,
+            -0.0010331592056900263,
+            -0.002075745025649667,
+            -0.014268155209720135,
+            0.02722412347793579,
+            0.0018617605092003942,
+            0.01592724397778511,
+            -0.00208705710247159,
+            -0.014207825064659119,
+            0.018551617860794067,
+            -0.01962248422205448,
+            -0.015256066806614399,
+            -0.004521014168858528,
+            0.001125540235079825,
+            -0.0011905840365216136,
+            -0.031100355088710785,
+            0.015361645258963108,
+            -0.013257619924843311,
+            0.034629687666893005,
+            0.006180102005600929,
+            -0.0012207493418827653,
+            -0.002481090370565653,
+            -0.011606073938310146,
+            -0.0056145042181015015,
+            -0.02821957692503929,
+            -0.017872901633381844,
+            0.01686236634850502,
+            -0.013536648824810982,
+            -0.009909279644489288,
+            -0.026515239849686623,
+            -0.021734051406383514,
+            -0.011040475219488144,
+            0.008793165907263756,
+            0.0014347339747473598,
+            0.022850165143609047,
+            0.00864988099783659,
+            -0.017375174909830093,
+            -0.0033671942073851824,
+            -0.041959840804338455,
+            -0.002809137338772416,
+            0.01033159252256155,
+            -0.017541082575917244,
+            -0.022246859967708588,
+            0.013204830698668957,
+            0.00048052275087684393,
+            -0.018596867099404335,
+            0.019411327317357063,
+            0.0019183203112334013,
+            0.004434289410710335,
+            0.0017147050239145756,
+            -0.0023151817731559277,
+            -0.01653054729104042,
+            0.00573139451444149,
+            0.009554837830364704,
+            0.01536918617784977,
+            -0.047238755971193314,
+            -0.007499831262975931,
+            0.003244647989049554,
+            0.006372405681759119,
+            0.006010422948747873,
+            0.01118376012891531,
+            0.004777418915182352,
+            -0.017767323181033134,
+            0.004471995867788792,
+            0.01906442828476429,
+            -0.022563595324754715,
+            0.01977331005036831,
+            -0.0019051230046898127,
+            0.0027883988805115223,
+            0.002799710724502802,
+            -0.007514914032071829,
+            -0.005490072537213564,
+            0.02583652175962925,
+            -0.010452253744006157,
+            0.009622709825634956,
+            0.021507810801267624,
+            0.02309148572385311,
+            -0.011470329947769642,
+            -0.01862703263759613,
+            0.013710098341107368,
+            0.0316433310508728,
+            0.0006923863547854125,
+            -0.014992120675742626,
+            -0.054086264222860336,
+            -0.011206384748220444,
+            0.03218630328774452,
+            0.015897078439593315,
+            0.017390256747603416,
+            -0.006941774394363165,
+            -0.012518571689724922,
+            0.019139841198921204,
+            0.023559046909213066,
+            0.017043357715010643,
+            -0.026545405387878418,
+            0.013928796164691448,
+            -0.020844176411628723,
+            0.03432803601026535,
+            0.0030429179314523935,
+            -0.008008869364857674,
+            0.004143949132412672,
+            -0.013431070372462273,
+            -0.02313673496246338,
+            0.014200284145772457,
+            -0.003846067236736417,
+            0.0028826650232076645,
+            0.006545855663716793,
+            -0.004864144138991833,
+            0.01939624547958374,
+            -0.0008766771061345935,
+            -0.005897303111851215,
+            0.004747253842651844,
+            0.02787267602980137,
+            0.0013659194810315967,
+            0.003993122838437557,
+            0.0011085723526775837,
+            0.010090271010994911,
+            -0.0019946759566664696,
+            -0.012586443684995174,
+            0.022563595324754715,
+            -0.012865472584962845,
+            -0.0056145042181015015,
+            0.01923033595085144,
+            -0.007903290912508965,
+            -0.00023342706845141947,
+            -0.010452253744006157,
+            -0.008838413283228874,
+            -0.03152266889810562,
+            0.026183422654867172,
+            -0.0036066307220607996,
+            0.014396357350051403,
+            0.021010085940361023,
+            -0.0012094373814761639,
+            -0.005410888697952032,
+            0.022337354719638824,
+            0.0002851557219401002,
+            -0.007458353880792856,
+            0.01386092510074377,
+            -0.02562536671757698,
+            -0.02612309157848358,
+            0.03858133405447006,
+            -0.012269708327949047,
+            0.0005542861763387918,
+            -0.0022491952404379845,
+            -0.00906465295702219,
+            -0.007511143106967211,
+            0.020542524755001068,
+            0.01708860509097576,
+            -0.02793300710618496,
+            -0.010595538653433323,
+            -0.010127977468073368,
+            -0.02909436821937561,
+            0.011590991169214249,
+            -0.0018532765097916126,
+            0.02153797633945942,
+            -0.005799266044050455,
+            -0.03155283257365227,
+            -0.004762336146086454,
+            0.03592679277062416,
+            -0.003981810994446278,
+            0.0010048793628811836,
+            0.015323938801884651,
+            0.0028675824869424105,
+            0.015610508620738983,
+            -0.029456350952386856,
+            0.025474539026618004,
+            -0.01640988700091839,
+            0.020255954936146736,
+            0.013483859598636627,
+            -0.014464229345321655,
+            0.027736932039260864,
+            0.013234996236860752,
+            0.005493842996656895,
+            0.006413882598280907,
+            0.0030052114743739367,
+            -0.010859483852982521,
+            -0.007077517919242382,
+            0.008340687490999699,
+            -0.002134190173819661,
+            0.0003591548011172563,
+            0.02594210021197796,
+            0.0003921480383723974,
+            0.013762887567281723,
+            0.001332926214672625,
+            -0.0001895932073239237,
+            0.0017618382116779685,
+            -0.002952422248199582,
+            -0.01944149285554886,
+            0.011304421350359917,
+            -0.015912160277366638,
+            -0.020964836701750755,
+            0.012443158775568008,
+            -0.017269596457481384,
+            -0.01785781793296337,
+            -0.028113998472690582,
+            0.01283530704677105,
+            -0.008861036971211433,
+            -0.007337693125009537,
+            0.009667957201600075,
+            0.011764440685510635,
+            0.015595425851643085,
+            0.02660573646426201,
+            0.019863806664943695,
+            -0.012744811363518238,
+            0.013762887567281723,
+            0.010489960201084614,
+            0.008830872364342213,
+            -0.00939647015184164,
+            0.0018146273214370012,
+            -0.010308968834578991,
+            0.020240871235728264,
+            0.010912273079156876,
+            0.004038370680063963,
+            0.019200170412659645,
+            -0.007669510785490274,
+            -0.01264677383005619,
+            -0.03505200147628784,
+            -0.004882997367531061,
+            -0.001891925698146224,
+            0.007262280210852623,
+            0.04998379200696945,
+            -0.014245531521737576,
+            -0.03300076723098755,
+            -0.016847282648086548,
+            -0.011802148073911667,
+            0.009645333513617516,
+            0.0012443158775568008,
+            0.010723740793764591,
+            -0.013491400517523289,
+            -0.035564810037612915,
+            0.0030825098510831594,
+            0.015776416286826134,
+            0.003531217575073242,
+            -0.0020041028037667274,
+            0.022337354719638824,
+            -0.028113998472690582,
+            0.0180086437612772,
+            -0.007514914032071829,
+            -0.011470329947769642,
+            -0.02088942378759384,
+            -0.0033671942073851824,
+            -0.0076581984758377075,
+            0.023845616728067398,
+            -0.005497613921761513,
+            -0.01978839375078678,
+            -0.003457689890637994,
+            0.011357210576534271,
+            -0.0009516188292764127,
+            -0.005659752059727907,
+            0.004928245209157467,
+            0.006406341679394245,
+            -0.0029769313987344503,
+            0.033905722200870514,
+            -0.0017128196777775884,
+            -0.0024792051408439875,
+            -0.0063196164555847645,
+            -0.0027733161114156246,
+            0.00947942491620779,
+            0.015022286213934422,
+            -0.047087930142879486,
+            -0.019878888502717018,
+            -0.007254738826304674,
+            -0.004079848062247038,
+            0.0051544844172894955,
+            0.012722187675535679,
+            -0.02104024961590767,
+            -0.00375934224575758,
+            -0.011191301979124546,
+            -0.008536760695278645,
+            -0.029516680166125298,
+            -0.010889649391174316,
+            0.03478051349520683,
+            -0.009426635690033436,
+            0.039094142615795135,
+            0.021130746230483055,
+            -0.0020436944905668497,
+            0.015052451752126217,
+            0.00261117797344923,
+            0.002690361812710762,
+            -0.00906465295702219,
+            -0.013936338014900684,
+            -0.0019626254215836525,
+            -0.005486302077770233,
+            0.0025489621330052614,
+            -0.03960695117712021,
+            0.0004977263743057847,
+            -0.0033276022877544165,
+            0.016998108476400375,
+            -0.035685472190380096,
+            -0.008891202509403229,
+            0.026982801035046577,
+            -0.012035928666591644,
+            0.005026282276958227,
+            -0.003996893297880888,
+            0.01653054729104042,
+            0.042231328785419464,
+            0.011455247178673744,
+            -0.003372850129380822,
+            -0.017782405018806458,
+            -0.0002785570686683059,
+            0.004988575354218483,
+            -0.03846067190170288,
+            0.004732171073555946,
+            -0.039697445929050446,
+            0.030180316418409348,
+            -0.00679094810038805,
+            -0.008333145640790462,
+            0.03541398420929909,
+            -0.018038809299468994,
+            0.01473571639508009,
+            0.009879114106297493,
+            -0.022186528891324997,
+            0.02693755365908146,
+            0.002552732825279236,
+            0.018551617860794067,
+            -0.04147719591856003,
+            0.00793345645070076,
+            -0.008295439183712006,
+            0.00950204860419035,
+            -0.01642496883869171,
+            -0.0017646661726757884,
+            0.016575796529650688,
+            -0.010127977468073368,
+            0.014449146576225758,
+            0.020135292783379555,
+            -0.001117998966947198,
+            0.0006494951667264104,
+            0.021899959072470665,
+            -0.006527002435177565,
+            0.0060330466367304325,
+            0.015489847399294376,
+            0.013868466019630432,
+            0.01346877682954073,
+            0.037827201187610626,
+            -0.002782742725685239,
+            0.0012320613022893667,
+            0.003993122838437557,
+            0.001916434965096414,
+            0.010241096839308739,
+            -0.0067796362563967705,
+            -0.001194354728795588,
+            -0.016605960205197334,
+            0.00782033707946539,
+            0.017013192176818848,
+            -0.006379947066307068,
+            -0.001372518134303391,
+            -0.030376389622688293,
+            -0.010037481784820557,
+            -0.04871685057878494,
+            0.006112230475991964,
+            -0.030919363722205162,
+            -0.010150601156055927,
+            -0.008416100405156612,
+            0.018385710194706917,
+            -0.003538758959621191,
+            0.012412993237376213,
+            0.001619496033526957,
+            -0.019984466955065727,
+            -0.005135631188750267,
+            0.008310521952807903,
+            -0.015059992671012878,
+            0.02325739525258541,
+            0.0071680136024951935,
+            -0.006440277211368084,
+            0.00266585242934525,
+            -0.013348115608096123,
+            -0.006142395548522472,
+            -0.015090158209204674,
+            -0.007017187308520079,
+            0.002581012900918722,
+            0.022654090076684952,
+            0.004143949132412672,
+            -0.0013828874798491597,
+            0.00010999705409631133,
+            -0.007752465084195137,
+            0.015791499987244606,
+            0.0027092150412499905,
+            -0.0003994536818936467,
+            0.004984804894775152,
+            0.015135405585169792,
+            0.008355769328773022,
+            -0.0059312391094863415,
+            0.008446265012025833,
+            0.006447818595916033,
+            -0.009592544287443161,
+            0.025821439921855927,
+            -0.008159696124494076,
+            0.009042028337717056,
+            -0.019712980836629868,
+            0.007639345247298479,
+            0.011093264445662498,
+            -0.014456688426434994,
+            -0.008023952133953571,
+            -0.02155306003987789,
+            0.02076876349747181,
+            0.007473436649888754,
+            0.0074885194189846516,
+            -0.024720408022403717,
+            0.014139953069388866,
+            -0.0055730268359184265,
+            -0.0013706327881664038,
+            -0.024780739098787308,
+            0.016213813796639442,
+            0.024886317551136017,
+            0.00906465295702219,
+            0.02027103677392006,
+            -0.003993122838437557,
+            0.008717752061784267,
+            -0.016017738729715347,
+            0.002801596187055111,
+            -0.012330039404332638,
+            -0.020783845335245132,
+            0.0033992447424679995,
+            0.033151593059301376,
+            -0.0016279799165204167,
+            -0.005870908498764038,
+            0.001966396113857627,
+            0.006131083704531193,
+            -0.004841519985347986,
+            -0.007707217242568731,
+            0.012247084639966488,
+            0.01344615314155817,
+            0.004649216774851084,
+            0.006006652023643255,
+            0.06690648943185806,
+            -0.004015746526420116,
+            0.004879226442426443,
+            -0.0011707880767062306,
+            -0.0031767762266099453,
+            -0.0034011302050203085,
+            -0.001456415164284408,
+            -0.02469024434685707,
+            0.019984466955065727,
+            0.0036556492559611797,
+            -0.009057111106812954,
+            0.009675499051809311,
+            -0.021085498854517937,
+            0.0006980423349887133,
+            0.007051123306155205,
+            0.011176219210028648,
+            0.001268825144506991,
+            0.016078069806098938,
+            0.03161316365003586,
+            0.015376728028059006,
+            -0.016666291281580925,
+            0.005180879030376673,
+            0.01116113644093275,
+            0.006406341679394245,
+            -0.00740933557972312,
+            0.02408693917095661,
+            0.0013781740562990308,
+            0.023815451189875603,
+            0.0019852493423968554,
+            0.020859258249402046,
+            0.010995227843523026,
+            -0.005554173607379198,
+            0.0007465895032510161,
+            -0.008582009002566338,
+            0.0064629013650119305,
+            0.012631691992282867,
+            0.0006551511469297111,
+            0.00011624219769146293,
+            0.0008257732843048871,
+            0.011372293345630169,
+            0.00801641121506691,
+            0.0018089713994413614,
+            -0.005953862797468901,
+            -0.007213261444121599,
+            -0.00876300036907196,
+            0.010957521386444569,
+            0.0025565035175532103,
+            -0.0005443881964311004,
+            -0.01063324511051178,
+            -0.002959963632747531,
+            -0.00010993813339155167,
+            0.013913714326918125,
+            0.016605960205197334,
+            0.009358763694763184,
+            -0.012654315680265427,
+            -0.004166572820395231,
+            0.0208290945738554,
+            -0.004852831829339266,
+            0.024916483089327812,
+            0.010670951567590237,
+            0.004422977566719055,
+            0.014418981969356537,
+            -0.016651209443807602,
+            -0.016349555924534798,
+            0.0030768539290875196,
+            -0.0021323049440979958,
+            0.011508036404848099,
+            0.010640786029398441,
+            -0.002104025101289153,
+            0.020406780764460564,
+            -0.007360316812992096,
+            -0.0025828981306403875,
+            0.015731168910861015,
+            0.016832200810313225,
+            -0.004215591587126255,
+            0.005275145173072815,
+            -0.015881994739174843,
+            -0.016334474086761475,
+            -0.005655981134623289,
+            0.00012619908375199884,
+            0.005810578353703022,
+            0.012873013503849506,
+            -5.217053694650531e-05,
+            0.014607514254748821,
+            -0.005870908498764038,
+            -0.025384044274687767,
+            -0.0007234942750073969,
+            0.003993122838437557,
+            0.01603282243013382,
+            -0.007088829763233662,
+            0.002594209974631667,
+            0.0016977370250970125,
+            0.021749133244156837,
+            -0.02099500223994255,
+            0.0038762325420975685,
+            0.0003674031177069992,
+            -0.01325007900595665,
+            0.0005245922366157174,
+            -0.007978703826665878,
+            -0.012458241544663906,
+            0.015331479720771313,
+            0.005561714991927147,
+            -0.013521566055715084,
+            0.018310297280550003,
+            -0.01052766665816307,
+            -0.0012150933034718037,
+            0.02199045568704605,
+            0.010648327879607677,
+            0.004362646955996752,
+            -0.015278690494596958,
+            -0.0056748343631625175,
+            -0.014871460385620594,
+            0.022382603958249092,
+            0.01708860509097576,
+            -0.004260839428752661,
+            0.003921480383723974,
+            0.013679933734238148,
+            0.0046944646164774895,
+            -0.018883436918258667,
+            -0.010648327879607677,
+            -0.0004682681174017489,
+            -0.010210931301116943,
+            0.004728400148451328,
+            0.0072396560572087765,
+            0.02010512910783291,
+            -0.016500383615493774,
+            -0.005995340179651976,
+            -0.006119771860539913,
+            -0.02930552512407303,
+            -0.029275359585881233,
+            -0.004411665257066488,
+            0.0024867465253919363,
+            -0.0024792051408439875,
+            -0.0026545405853539705,
+            -0.0019550842698663473,
+            0.002839302644133568,
+            0.0005000830278731883,
+            -0.008137071505188942,
+            0.013740263879299164,
+            -0.000598591344896704,
+            0.005554173607379198,
+            0.022126199677586555,
+            -0.002910945098847151,
+            -0.013544189743697643,
+            0.03194498270750046,
+            0.011455247178673744,
+            -0.009743370115756989,
+            -0.027344783768057823,
+            0.01211888249963522,
+            -0.011063099838793278,
+            0.017510918900370598,
+            -0.043558597564697266,
+            -0.004962180741131306,
+            -0.0038592645432800055,
+            -0.007356546353548765,
+            -0.011606073938310146,
+            -0.011002768762409687,
+            -0.016349555924534798,
+            -0.006862590555101633,
+            -0.010949979536235332,
+            -0.022231776267290115,
+            0.007963621988892555,
+            -0.005312851630151272,
+            0.02656048908829689,
+            -0.022216694429516792,
+            0.024448921903967857,
+            0.010158142074942589,
+            -0.003913938999176025,
+            0.002268048468977213,
+            -0.008589549921452999,
+            0.00490562105551362,
+            0.020074963569641113,
+            0.010014857165515423,
+            0.00011942369019379839,
+            0.021010085940361023,
+            0.008883661590516567,
+            -0.0038290992379188538,
+            -0.0038988564629107714,
+            -0.005433512851595879,
+            0.0246751606464386,
+            0.004422977566719055,
+            0.004660528618842363,
+            -0.008076741360127926,
+            0.011055557988584042,
+            0.027043132111430168,
+            0.007111453916877508,
+            -0.009336140006780624,
+            -0.005999110639095306,
+            0.0027959400322288275,
+            -0.011658863164484501,
+            -0.008416100405156612,
+            0.011093264445662498,
+            -0.01098768599331379,
+            -0.023543965071439743,
+            0.01002993993461132,
+            0.02291049435734749,
+            0.002745036268606782,
+            -0.015881994739174843,
+            -0.005640898831188679,
+            -0.0024716637562960386,
+            0.006685370113700628,
+            0.017827652394771576,
+            -0.010980145074427128,
+            -0.0144114401191473,
+            0.010542749427258968,
+            0.004057223908603191,
+            -0.00531662255525589,
+            0.02155306003987789,
+            0.030119985342025757,
+            0.011651321314275265,
+            -0.04367925971746445,
+            -0.005663522519171238,
+            0.005867138039320707,
+            -0.013823218643665314,
+            -0.047962721437215805,
+            0.004784960299730301,
+            -0.008416100405156612,
+            -0.013649768196046352,
+            0.014456688426434994,
+            -0.009562378749251366,
+            -0.014999662525951862,
+            0.02104024961590767,
+            0.026213588193058968,
+            0.0052977693267166615,
+            0.012910719960927963,
+            -0.01418520137667656,
+            -0.019909054040908813,
+            -0.020044798031449318,
+            0.00815215427428484,
+            -0.017706992104649544,
+            -0.009607627056539059,
+            0.0023151817731559277,
+            -0.013333032839000225,
+            0.02297082543373108,
+            0.012412993237376213,
+            0.03152266889810562,
+            0.009683039970695972,
+            -0.004702006001025438,
+            -0.02529354766011238,
+            -0.01719418354332447,
+            -0.002509370446205139,
+            -0.0010963176609948277,
+            0.0049697221256792545,
+            0.005456136539578438,
+            0.017510918900370598,
+            0.012963509187102318,
+            0.009517131373286247,
+            -0.013008756563067436,
+            0.015301315113902092,
+            -2.8662863769568503e-05,
+            -0.003806475317105651,
+            -0.014290779829025269,
+            -0.029561929404735565,
+            -0.0033615382853895426,
+            0.006383717525750399,
+            -0.0040911599062383175,
+            -0.004630363546311855,
+            0.0063196164555847645,
+            0.030014406889677048,
+            -0.009192855097353458,
+            0.002863811794668436,
+            -0.037435054779052734,
+            -0.003587777493521571,
+            -0.0034539191983640194,
+            0.026847057044506073,
+            0.01156836748123169,
+            -0.008755459450185299,
+            -0.006323386915028095,
+            -0.01344615314155817,
+            0.012888096272945404,
+            0.01366485096514225,
+            -0.009683039970695972,
+            0.03312142565846443,
+            0.0020116439554840326,
+            0.01396650355309248,
+            -0.0012084946502000093,
+            -0.01002993993461132,
+            0.016078069806098938,
+            0.028883211314678192,
+            -0.011281797662377357,
+            0.012767435051500797,
+            0.01190018467605114,
+            0.005512696225196123,
+            0.0017071636393666267,
+            -0.027254289016127586,
+            0.0022341127041727304,
+            -0.01021847315132618,
+            0.0214927289634943,
+            0.000284920068224892,
+            0.0015572801930829883,
+            -0.02093467302620411,
+            0.018114222213625908,
+            -0.02506730891764164,
+            -0.005060217808932066,
+            0.012443158775568008,
+            -0.0038102460093796253,
+            0.004717088304460049,
+            -0.02186979353427887,
+            -0.010535207577049732,
+            0.015007203444838524,
+            -0.018808024004101753,
+            -0.004396582953631878,
+            -0.012707104906439781,
+            -0.01553509570658207,
+            0.021356984972953796,
+            0.009750911965966225,
+            -0.018415875732898712,
+            -0.0005943493451923132,
+            0.029396019876003265,
+            -0.027239205315709114,
+            0.010753905400633812,
+            0.0180086437612772,
+            0.016711538657546043,
+            0.01344615314155817,
+            0.001777863479219377,
+            0.00884595513343811,
+            -0.0014799818163737655,
+            -0.012473324313759804,
+            -0.020014632493257523,
+            0.0196677315980196,
+            -0.01473571639508009,
+            0.019697897136211395,
+            0.010097811929881573,
+            -0.00854430254548788,
+            -0.005407118238508701,
+            -0.004815125372260809,
+            -0.003229565219953656,
+            -0.010120435617864132,
+            -0.003642451949417591,
+            0.015776416286826134,
+            0.022955743595957756,
+            -0.004068535752594471,
+            -0.018159471452236176,
+            0.017510918900370598,
+            -0.0015959293814375997,
+            0.0015139176975935698,
+            -0.0011679601157084107,
+            -0.006195184774696827,
+            0.015059992671012878,
+            0.010120435617864132,
+            -0.0002375512121943757,
+            -0.06654450297355652,
+            -0.009539755061268806,
+            -0.004287234041839838,
+            -0.004257068503648043,
+            0.002848729258403182,
+            -0.017812570556998253,
+            -0.03951645642518997,
+            0.011191301979124546,
+            0.016892530024051666,
+            -0.01471309270709753,
+            0.01592724397778511,
+            0.005678605288267136,
+            0.0017250742530450225,
+            -0.01740534044802189,
+            -0.002826105337589979,
+            0.021507810801267624,
+            0.03251812234520912,
+            -0.025052227079868317,
+            -0.01211888249963522,
+            0.012247084639966488,
+            -0.004155260976403952,
+            -0.007775088772177696,
+            -0.021326819434762,
+            -0.00029057604842819273,
+            0.01812930591404438,
+            0.02556503564119339,
+            -0.020844176411628723,
+            -0.024011526256799698,
+            4.7928548156050965e-05,
+            0.011085723526775837,
+            -0.025761108845472336,
+            -0.009924361482262611,
+            -0.015625590458512306,
+            -0.0034086713567376137,
+            0.011206384748220444,
+            -0.011870019137859344,
+            0.007624262943863869,
+            -0.012164130806922913,
+            0.005437283311039209,
+            -0.008121988736093044,
+            -0.02485615201294422,
+            0.011470329947769642,
+            -0.006055670790374279,
+            0.035444147884845734,
+            -0.006662745960056782,
+            -0.0030900510028004646,
+            0.02726937085390091,
+            -0.0008281299378722906,
+            0.005916156340390444,
+            0.014049457386136055,
+            0.03002949059009552,
+            -0.014871460385620594,
+            0.011176219210028648,
+            -0.000854524492751807,
+            -0.004882997367531061,
+            0.0005217642756178975,
+            0.03812885656952858,
+            0.004751024302095175,
+            0.0003579764743335545,
+            -0.0009945100173354149,
+            0.0015940440353006124,
+            -0.0011538201943039894,
+            0.015188194811344147,
+            0.025323713198304176,
+            -0.0012480865698307753,
+            -0.011093264445662498,
+            -0.01576133444905281,
+            -0.013868466019630432,
+            -8.383813838008791e-05,
+            0.0030881657730787992,
+            0.011085723526775837,
+            0.014056999236345291,
+            -0.006527002435177565,
+            0.01675678789615631,
+            -0.01146278902888298,
+            0.00815215427428484,
+            -0.024222683161497116,
+            0.013189747929573059,
+            -0.011666404083371162,
+            0.0006047186907380819,
+            0.0031447256915271282,
+            -0.0015261722728610039,
+            0.01325007900595665,
+            -0.00677586579695344,
+            0.004860373213887215,
+            0.012254626490175724,
+            -0.0007579015218652785,
+            0.0024414986837655306,
+            -0.021115664392709732,
+            -0.013906172476708889,
+            -0.003118331078439951,
+            0.021945208311080933,
+            -0.0030730832368135452,
+            -0.014637679792940617,
+            -4.306793925934471e-05,
+            0.004230673890560865,
+            -0.005192190874367952,
+            0.012171671725809574,
+            0.0008700784528627992,
+            -0.00031437829602509737,
+            0.020859258249402046,
+            0.0039063976146280766,
+            -0.010346675291657448,
+            0.01556526031345129,
+            -0.003041032701730728,
+            -0.0069530862383544445,
+            -0.010120435617864132,
+            0.006508149206638336,
+            0.006229120772331953,
+            -0.005177108105272055,
+            0.026364414021372795,
+            -0.03288010507822037,
+            0.0008507538586854935,
+            0.009705663658678532,
+            -0.0014102247077971697,
+            -0.0029486515559256077,
+            0.004102471750229597,
+            0.04105488210916519,
+            -0.009637792594730854,
+            -0.023830534890294075,
+            0.0002169304498238489,
+            0.01769191026687622,
+            -0.011492953635752201,
+            0.002967504784464836,
+            0.01823488436639309,
+            0.004890538286417723,
+            0.0034859697334468365,
+            0.010007316246628761,
+            -0.03704290837049484,
+            -0.004471995867788792,
+            -0.007940997369587421,
+            -0.011206384748220444,
+            0.007186866831034422,
+            0.02088942378759384,
+            0.01690761372447014,
+            0.014026833698153496,
+            0.005784183740615845,
+            0.030949529260396957,
+            0.0037970487028360367,
+            -0.003135299077257514,
+            0.0016317506087943912,
+            0.00720194960013032,
+            -0.013008756563067436,
+            0.018099140375852585,
+            -0.008137071505188942,
+            -0.015520012937486172,
+            0.004260839428752661,
+            -0.009569920599460602,
+            0.004815125372260809,
+            -0.011032934300601482,
+            0.0051243188790977,
+            -0.006904067937284708,
+            0.0005476875230669975,
+            -0.007439500652253628,
+            -0.00021657695469912142,
+            0.013076628558337688,
+            -3.543825732776895e-05,
+            0.011221466585993767,
+            0.012239543721079826,
+            -0.009057111106812954,
+            -0.024644995108246803,
+            0.012443158775568008,
+            -0.008031493052840233,
+            0.005772871430963278,
+            -0.010927355848252773,
+            0.0009073136607185006,
+            -0.0010576684726402164,
+            0.03791769966483116,
+            0.025414209812879562,
+            -0.0003365308803040534,
+            0.00593878049403429,
+            -0.028551394119858742,
+            0.007360316812992096,
+            -0.024991896003484726,
+            0.009826324880123138,
+            -0.013958961702883244,
+            -0.008287898264825344,
+            0.012171671725809574,
+            0.028460897505283356,
+            -0.0046529872342944145,
+            0.004226903431117535,
+            -0.007266050670295954,
+            -0.020150376483798027,
+            -0.007895749993622303,
+            -0.006530772894620895,
+            -0.011500495485961437,
+            -0.02055760659277439,
+            -0.010286344215273857,
+            0.01785781793296337,
+            -0.009026946499943733,
+            -0.0013678048271685839,
+            0.015286232344806194,
+            -0.0017995446687564254,
+            0.005490072537213564,
+            -0.017058439552783966,
+            0.0038837739266455173,
+            0.0062178089283406734,
+            0.01690761372447014,
+            -0.01553509570658207,
+            0.012209378182888031,
+            0.0033332582097500563,
+            -0.013091711327433586,
+            0.008106906898319721,
+            -0.021070415154099464,
+            -0.015082616358995438,
+            -0.0003834283852484077,
+            0.007695905398577452,
+            -0.010535207577049732,
+            -0.030919363722205162,
+            0.009554837830364704,
+            -0.0043777297250926495,
+            -0.007511143106967211,
+            -0.01613840088248253,
+            0.009901737794280052,
+            0.0022322272416204214,
+            0.03387555852532387,
+            -0.017284678295254707,
+            -0.006741929799318314,
+            0.019712980836629868,
+            0.009109900332987309,
+            -0.01642496883869171,
+            -0.009366304613649845,
+            0.004249527119100094,
+            -0.0012886211043223739,
+            0.006821113638579845,
+            -0.01463013794273138,
+            -0.009939444251358509,
+            0.0031145603861659765,
+            0.005637127906084061,
+            0.019471658393740654,
+            0.005324163939803839,
+            -0.0135818962007761,
+            -0.0032653864473104477,
+            0.009351222775876522,
+            0.006911609321832657,
+            0.009388929232954979,
+            -0.016349555924534798,
+            0.002688476350158453,
+            -0.012254626490175724,
+            0.002351003000512719,
+            0.011628697626292706,
+            -0.006361093837767839,
+            -0.006342240143567324,
+            0.022895412519574165,
+            0.017933230847120285,
+            -0.0018061433220282197,
+            -0.013234996236860752,
+            0.0004694464441854507,
+            -0.0002575828111730516,
+            0.01281268335878849,
+            -0.006557167507708073,
+            -0.005595650989562273,
+            0.0013838300947099924,
+            0.01576133444905281,
+            0.003944104071706533,
+            -0.005090383347123861,
+            -0.013943878933787346,
+            -0.015384268946945667,
+            0.0007640288094989955,
+            -0.027480527758598328,
+            0.0033502262085676193,
+            0.0014385045506060123,
+            0.020452028140425682,
+            0.017812570556998253,
+            0.019019179046154022,
+            -0.014358650892972946,
+            0.001570477499626577,
+            0.005007428582757711,
+            -0.00812953058630228,
+            0.002094598487019539,
+            -0.009879114106297493,
+            -0.0190040972083807,
+            -0.006504378281533718,
+            0.0058595966547727585,
+            0.018370626494288445,
+            0.004656758159399033,
+            -0.004309857729822397,
+            0.029411103576421738,
+            0.019366079941391945,
+            -0.02309148572385311,
+            0.011749358847737312,
+            -0.024554500356316566,
+            0.010301426984369755,
+            0.01962248422205448,
+            0.004592656623572111,
+            -0.016078069806098938,
+            -0.024554500356316566,
+            0.0050111995078623295,
+            -0.010301426984369755,
+            -0.008506596088409424,
+            0.008302980102598667,
+            -0.023785287514328957,
+            0.018084056675434113,
+            -0.011975597590208054,
+            0.0018268820131197572,
+            -0.010844401083886623,
+            -0.004241986200213432,
+            -0.010542749427258968,
+            -0.0031748907640576363,
+            0.007115224376320839,
+            0.01690761372447014,
+            0.005388265009969473,
+            -0.03577596694231033,
+            -0.011010310612618923,
+            0.015580343082547188,
+            -0.00520727364346385,
+            -0.0094190938398242,
+            -0.0020436944905668497,
+            0.019652649760246277,
+            -0.02164355479180813,
+            -0.006764553487300873,
+            -0.0376763753592968,
+            -0.02022578939795494,
+            -0.02755594067275524,
+            -0.028340237215161324,
+            -0.01314450055360794,
+            -0.004615280777215958,
+            0.015550177544355392,
+            0.0017986020538955927,
+            -0.008574468083679676,
+            -0.0008361425716429949,
+            -0.0108293192461133,
+            -0.0003174419398419559,
+            -0.010263720527291298,
+            -0.006417653523385525,
+            0.0033596528228372335,
+            0.007375399582087994,
+            0.002143617020919919,
+            -0.012262167409062386,
+            -0.008039034903049469,
+            -0.011809688992798328,
+            -0.010595538653433323,
+            0.00854430254548788,
+            -0.003346455516293645,
+            -0.019682815298438072,
+            0.016108235344290733,
+            0.008197402581572533,
+            0.012005763128399849,
+            0.013189747929573059,
+            0.0009275809279642999,
+            0.032095808535814285,
+            6.698803190374747e-05,
+            -0.0016854824498295784,
+            -6.112583832873497e-06,
+            -0.009026946499943733,
+            0.015376728028059006,
+            0.0077298409305512905,
+            0.008325604721903801,
+            -0.004079848062247038,
+            0.006293221842497587,
+            -0.007443271577358246,
+            0.0058181192725896835,
+            0.00018040223221760243,
+            -0.007971162907779217,
+            0.0003205056127626449,
+            0.007695905398577452,
+            0.002927913097664714,
+            -0.01418520137667656,
+            0.006817342713475227,
+            0.012699563056230545,
+            0.006934233009815216,
+            -0.01576133444905281,
+            -0.01380059402436018,
+            -0.012156588956713676,
+            -0.008499054238200188,
+            -0.028551394119858742,
+            0.022955743595957756,
+            0.043830085545778275,
+            0.000945962849073112,
+            -0.0190040972083807,
+            -0.0044380598701536655,
+            -0.015444600023329258,
+            -0.015007203444838524,
+            -0.0056748343631625175,
+            -0.005912385880947113,
+            -0.02375512197613716,
+            -0.008114447817206383,
+            0.015278690494596958,
+            0.014577348716557026,
+            0.000577381404582411,
+            -0.01570100337266922,
+            -0.005953862797468901,
+            -0.015987573191523552,
+            -0.013302868232131004,
+            -0.00908727664500475,
+            -0.021960290148854256,
+            -0.012518571689724922,
+            0.012043469585478306,
+            -0.01613840088248253,
+            0.015836747363209724,
+            0.020421862602233887,
+            0.00380836077965796,
+            -0.014109788462519646,
+            -0.015437058173120022,
+            -0.011289338581264019,
+            0.01198313944041729,
+            0.010912273079156876,
+            -0.010723740793764591,
+            0.03107019141316414,
+            -0.016440052539110184,
+            -0.009524672292172909,
+            -0.0033596528228372335,
+            0.007217032369226217,
+            -0.014260614290833473,
+            -0.004147719591856003,
+            -0.016621043905615807,
+            0.01220183726400137,
+            -0.014275697059929371,
+            0.006459130439907312,
+            0.007250967901200056,
+            -0.016334474086761475,
+            0.021326819434762,
+            0.025263383984565735,
+            0.0018579898169264197,
+            0.007635574787855148,
+            0.009962067939341068,
+            -0.013928796164691448,
+            0.010784070938825607,
+            -0.004984804894775152,
+            -0.0012575131841003895,
+            0.028129080310463905,
+            0.008536760695278645,
+            -0.0027506921906024218,
+            -0.015188194811344147,
+            0.013627144508063793,
+            0.020798929035663605,
+            -0.0023604296147823334,
+            -0.006870131939649582,
+            -0.013657309114933014,
+            0.013695016503334045,
+            0.0066363513469696045,
+            -0.00605944124981761,
+            -0.0075978683307766914,
+            -0.008989240042865276,
+            -0.0019230336183682084,
+            -0.021779298782348633,
+            0.012149048037827015,
+            -0.023830534890294075,
+            0.00014540585107170045,
+            -0.009494507685303688,
+            0.009667957201600075,
+            -0.0060028815641999245,
+            0.009939444251358509,
+            -0.010806694626808167,
+            0.013619602657854557,
+            0.007367858197540045,
+            0.0004548351571429521,
+            -0.011274255812168121,
+            0.017767323181033134,
+            0.004260839428752661,
+            0.00856692623347044,
+            0.00245281052775681,
+            -0.019170006737113,
+            0.030557380989193916,
+            -0.006625039502978325,
+            0.009434176608920097,
+            -0.010799153707921505,
+            0.012639232911169529,
+            0.021417316049337387,
+            0.028732385486364365,
+            0.011741816997528076,
+            0.0034765431191772223,
+            -0.015225901268422604,
+            0.024147270247340202,
+            0.0015846174210309982,
+            -0.010075188241899014,
+            0.02016545832157135,
+            0.004159031435847282,
+            0.005195961333811283,
+            -0.007032270077615976,
+            0.0090043218806386,
+            -0.02423776499927044,
+            -0.02551978826522827,
+            -0.007511143106967211,
+            -0.007582785561680794,
+            -0.009328598156571388,
+            0.01495441421866417,
+            -0.0017297875601798296,
+            -0.0019946759566664696,
+            0.0037178650964051485,
+            0.0032031708396971226,
+            0.007496060337871313,
+            0.002641343278810382,
+            -0.013574355281889439,
+            -0.012164130806922913,
+            -0.016711538657546043,
+            0.019215254113078117,
+            -0.008106906898319721,
+            0.001139680272899568,
+            -0.0070247286930680275,
+            0.021417316049337387,
+            0.010188307613134384,
+            0.007160472217947245,
+            0.012707104906439781,
+            -0.006877673324197531,
+            0.01729976199567318,
+            0.004649216774851084,
+            -0.012435617856681347,
+            0.015987573191523552,
+            0.026258835569024086,
+            -0.0012443158775568008,
+            -0.00364999333396554,
+            0.00190418038982898,
+            -0.04192967340350151,
+            0.00040935163269750774,
+            -0.014041916467249393,
+            0.018702445551753044,
+            -0.007888208143413067,
+            0.004988575354218483,
+            0.01459997333586216,
+            -0.005350558087229729,
+            -0.0010887763928622007,
+            0.014011750929057598,
+            0.007326381281018257,
+            0.015670839697122574,
+            0.00720194960013032,
+            0.015550177544355392,
+            -0.013657309114933014,
+            -0.03069312497973442,
+            -0.008174777962267399,
+            -0.003989351913332939,
+            0.015437058173120022,
+            0.0037103237118571997,
+            0.016228895634412766,
+            -0.030949529260396957,
+            -0.005116777960211039,
+            -0.006296992301940918,
+            0.002047465182840824,
+            0.014705551788210869,
+            -0.00354630034416914,
+            0.018038809299468994,
+            0.015489847399294376,
+            -0.008755459450185299,
+            -0.02159830741584301,
+            -0.020844176411628723,
+            0.013936338014900684,
+            0.010060105472803116,
+            -0.020361533388495445,
+            0.0020512358751147985,
+            -0.008634798228740692,
+            -0.006244203541427851,
+            -0.02963734231889248,
+            0.024283012375235558,
+            -0.003429410047829151,
+            0.01305400487035513,
+            0.011636238545179367,
+            0.010482418350875378,
+            0.013521566055715084,
+            0.02942618541419506,
+            -0.004690693691372871,
+            0.009683039970695972,
+            0.005984028335660696,
+            -0.009275808930397034,
+            -0.007164243143051863,
+            -0.009864031337201595,
+            0.015881994739174843,
+            -0.0019701668061316013,
+            0.0027412655763328075,
+            -0.003182432148605585,
+            0.017013192176818848,
+            -0.004626592621207237,
+            -0.024056773632764816,
+            -0.004204279277473688,
+            -0.001689253025688231,
+            -0.010429629124701023,
+            0.011802148073911667,
+            -0.008506596088409424,
+            -0.000715952948667109,
+            0.009735829196870327,
+            -0.004607739392668009,
+            -0.008989240042865276,
+            0.006892756093293428,
+            0.0002959963458124548,
+            -0.04117554426193237,
+            0.0005467448499985039,
+            0.004027058836072683,
+            -0.017752239480614662,
+            0.004668070003390312,
+            -0.0076280334033071995,
+            -0.010346675291657448,
+            0.0005962346913293004,
+            0.005927468184381723,
+            0.003774424782022834,
+            0.0221563633531332,
+            -0.0018711871234700084,
+            0.01438127551227808,
+            -0.006862590555101633,
+            -0.0024942876771092415,
+            -0.009750911965966225,
+            0.0135818962007761,
+            -0.020949754863977432,
+            -0.014720633625984192,
+            -0.01580658182501793,
+            -0.005007428582757711,
+            0.004577574320137501,
+            0.00458511570468545,
+            0.009539755061268806,
+            0.0013272702926769853,
+            -0.006444048136472702,
+            0.010701116174459457,
+            -0.006413882598280907,
+            -0.00032121260301209986,
+            -0.013016298413276672,
+            0.008446265012025833,
+            0.02357413060963154,
+            0.005075300578027964,
+            -0.0064930664375424385,
+            -0.013016298413276672,
+            0.01586691290140152,
+            0.03441853076219559,
+            -0.023016072809696198,
+            0.008664963766932487,
+            0.019712980836629868,
+            -0.0031371843069791794,
+            -0.002475434448570013,
+            -0.011583449319005013,
+            0.003039147239178419,
+            -0.019471658393740654,
+            0.009796159341931343,
+            0.006873902399092913,
+            -0.008227567188441753,
+            0.004882997367531061,
+            -0.015776416286826134,
+            0.0017165903700515628,
+            -0.002958078170195222,
+            -0.008710211142897606,
+            -0.005557944066822529,
+            -0.012692022137343884,
+            -0.006078294478356838,
+            0.0035180202685296535,
+            0.008393475785851479,
+            -0.004498390480875969,
+            0.01856670156121254,
+            -0.004038370680063963,
+            0.02253342978656292,
+            0.0022416538558900356,
+            -0.001372518134303391,
+            -0.009124983102083206,
+            0.0070247286930680275,
+            0.00026041080127470195,
+            -0.004769877530634403,
+            0.0017948313616216183,
+            -0.009072193875908852,
+            0.011334586888551712,
+            -0.0034312952775508165,
+            0.01752600073814392,
+            0.007009646389633417,
+            -0.008717752061784267,
+            -0.006308304611593485,
+            0.0027714308816939592,
+            -0.005712541285902262,
+            0.012322497554123402,
+            -0.004592656623572111,
+            0.01894376613199711,
+            -0.013423528522253036,
+            -0.018702445551753044,
+            0.004717088304460049,
+            0.008476430550217628,
+            0.013287785463035107,
+            -0.009532214142382145,
+            0.01002993993461132,
+            -0.0037819661665707827,
+            0.01823488436639309,
+            0.01806897483766079,
+            -0.007243426516652107,
+            0.015007203444838524,
+            -0.01962248422205448,
+            -0.0013781740562990308,
+            0.012737269513309002,
+            -0.002754462882876396,
+            -0.020361533388495445,
+            -0.011651321314275265,
+            -0.018928684294223785,
+            0.010489960201084614,
+            -0.015489847399294376,
+            0.023709872737526894,
+            0.011274255812168121,
+            0.010572914034128189,
+            -0.001968281576409936,
+            0.006198955699801445,
+            -0.0019202056573703885,
+            0.014539642259478569,
+            -0.0016835971036925912,
+            -0.004864144138991833,
+            -0.010595538653433323,
+            -0.002262392546981573,
+            -0.04057224094867706,
+            8.725529187358916e-05,
+            0.014698009938001633,
+            0.03354373946785927,
+            -0.02170388586819172,
+            0.014336027204990387,
+            -0.011651321314275265,
+            -0.0190040972083807,
+            -0.012043469585478306,
+            0.015158030204474926,
+            -0.015263608656823635,
+            0.019170006737113,
+            -0.009743370115756989,
+            -0.0047661070711910725,
+            0.001808028668165207,
+            0.01201330404728651,
+            0.003014638088643551,
+            0.0039478749968111515,
+            0.02402660809457302,
+            0.015203277580440044,
+            0.00648552505299449,
+            -0.003024064702913165,
+            0.011515578255057335,
+            0.00458511570468545,
+            -0.014863918535411358,
+            -0.005463677924126387,
+            -0.0011594761162996292,
+            -0.007639345247298479,
+            -0.010618162341415882,
+            -0.008529219776391983,
+            -0.0037800809368491173,
+            0.0008403845713473856,
+            0.018747692927718163,
+            -0.006010422948747873,
+            -0.0012942770263180137,
+            -0.012299873866140842,
+            0.012164130806922913,
+            -0.0011886986903846264,
+            0.018928684294223785,
+            -0.027359867468476295,
+            0.026741480454802513,
+            -0.03444869816303253,
+            -0.01388354878872633,
+            -0.006632580887526274,
+            -0.00015966364298947155,
+            0.011809688992798328,
+            0.0028845504857599735,
+            -0.015836747363209724,
+            -0.0008521678391844034,
+            -0.006930462550371885,
+            0.002935454249382019,
+            -0.003197514684870839,
+            0.009524672292172909,
+            0.005855826195329428,
+            -0.005659752059727907,
+            -0.021945208311080933,
+            -0.008891202509403229,
+            -0.012563819997012615,
+            -0.008355769328773022,
+            -0.013438611291348934,
+            0.013627144508063793,
+            -0.003331372980028391,
+            0.0006655204924754798,
+            0.023000990971922874,
+            -0.014283237978816032,
+            -0.01613840088248253,
+            0.015082616358995438,
+            -0.003555726958438754,
+            -0.0056446692906320095,
+            -0.01305400487035513,
+            0.007775088772177696,
+            0.0010850057005882263,
+            0.018249966204166412,
+            -0.008876119740307331,
+            0.022337354719638824,
+            -0.0045097023248672485,
+            0.012322497554123402,
+            0.018099140375852585,
+            -0.023453468456864357,
+            -0.030180316418409348,
+            0.004219362046569586,
+            -0.02793300710618496,
+            -0.0010472992435097694,
+            0.0019739374984055758,
+            -0.024946648627519608,
+            -0.014946873299777508,
+            -0.008853496052324772,
+            -0.0255047045648098,
+            -0.01630430854856968,
+            0.012918260879814625,
+            -0.0006914437399245799,
+            0.014253072440624237,
+            0.011621156707406044,
+            -0.016213813796639442,
+            0.01553509570658207,
+            -0.014418981969356537,
+            -0.02490140125155449,
+            0.0008055060170590878,
+            -0.008506596088409424,
+            0.012910719960927963,
+            0.00406476529315114,
+            0.001550681539811194,
+            0.009622709825634956,
+            -0.005320393014699221,
+            -0.0018759004306048155,
+            0.019049344584345818,
+            0.02186979353427887,
+            0.004275921732187271,
+            -0.00418542604893446,
+            0.00616879016160965,
+            -0.016651209443807602,
+            0.029275359585881233,
+            -0.01220183726400137,
+            -0.015097699128091335,
+            0.025097474455833435,
+            0.006285680457949638,
+            0.019215254113078117,
+            0.004453142639249563,
+            0.005128089804202318,
+            0.001582732074894011,
+            0.011357210576534271,
+            -0.0015261722728610039,
+            -0.01104801706969738,
+            -0.014388816431164742,
+            -0.018325379118323326,
+            0.030889200046658516,
+            0.008174777962267399,
+            0.017616497352719307,
+            -0.028672054409980774,
+            0.010799153707921505,
+            -0.008506596088409424,
+            -0.012888096272945404,
+            0.009818783961236477,
+            0.028445815667510033,
+            0.05140155926346779,
+            0.022578677162528038,
+            0.005735164973884821,
+            -0.000906370987650007,
+            0.034297872334718704,
+            -0.01303138118237257,
+            -0.009622709825634956,
+            -0.006455359980463982,
+            -0.01686236634850502,
+            0.004011976066976786,
+            0.012986132875084877,
+            -0.003627369413152337,
+            -0.008393475785851479,
+            -0.010867025703191757,
+            0.03541398420929909,
+            0.005780412815511227,
+            -0.010143060237169266,
+            0.013868466019630432,
+            -0.01956215314567089,
+            0.012216920033097267,
+            -0.017510918900370598,
+            -0.019049344584345818,
+            -0.013415987603366375,
+            0.0009813127107918262,
+            -0.02313673496246338,
+            0.006757012102752924,
+            -0.00854430254548788,
+            0.018913600593805313,
+            -0.010060105472803116,
+            -0.006017964333295822,
+            0.0034331807401031256,
+            -0.005946321878582239,
+            -0.014373733662068844,
+            0.00480758398771286,
+            -0.008023952133953571,
+            0.020783845335245132,
+            0.0005665408098138869,
+            0.008800706826150417,
+            0.0061348541639745235,
+            0.006425194907933474,
+            -0.0028298760298639536,
+            0.0012377172242850065,
+            0.012186754494905472,
+            0.015912160277366638,
+            0.009404012002050877,
+            -0.030346224084496498,
+            0.004253298044204712,
+            -0.013717640191316605,
+            0.014977038837969303,
+            0.00812953058630228,
+            0.01962248422205448,
+            0.016183648258447647,
+            0.013355657458305359,
+            -9.674084867583588e-05,
+            0.004547408781945705,
+            -0.0012961623724550009,
+            -0.0017043356783688068,
+            0.008499054238200188,
+            -0.00024226453388109803,
+            -0.0069530862383544445,
+            -0.0012047240743413568,
+            0.008031493052840233,
+            -0.008921368047595024,
+            -0.019094591960310936,
+            -0.00184950593393296,
+            -0.002364200074225664,
+            -0.01873260922729969,
+            0.0033671942073851824,
+            0.020180542021989822,
+            -0.015987573191523552,
+            -0.0016562598757445812,
+            0.01846112310886383,
+            0.02153797633945942,
+            -0.008204943500459194,
+            0.009230561554431915,
+            -0.004988575354218483,
+            -0.010693575255572796,
+            -0.02766151912510395,
+            -0.000485943048261106,
+            -0.02814416214823723,
+            0.035564810037612915,
+            -0.00886857882142067,
+            -0.018687361851334572,
+            0.012488407082855701,
+            -0.006560938432812691,
+            -0.01713385246694088,
+            -0.0068512787111103535,
+            0.005075300578027964,
+            -0.030165232717990875,
+            0.033966053277254105,
+            -0.016666291281580925,
+            -0.0009907394414767623,
+            0.020482193678617477,
+            0.05173337459564209,
+            -0.0031692348420619965,
+            -0.01746566966176033,
+            0.017254512757062912,
+            0.014426522888243198,
+            0.03188465163111687,
+            -0.01576133444905281,
+            -0.017390256747603416,
+            -0.010814236477017403,
+            -0.012873013503849506,
+            -0.011990680359303951,
+            -0.020814010873436928,
+            0.03604745492339134,
+            -0.012503488920629025,
+            -0.016726622357964516,
+            -0.0040496825240552425,
+            -0.008167237043380737,
+            -0.011289338581264019,
+            0.022654090076684952,
+            0.027450362220406532,
+            -0.0108293192461133,
+            -0.012963509187102318,
+            -0.0012235773028805852,
+            -0.023227229714393616,
+            0.02088942378759384,
+            -0.0005038536619395018,
+            -0.014698009938001633,
+            0.010867025703191757,
+            -0.0036914704833179712,
+            -0.0016949090640991926,
+            0.002579127438366413,
+            0.004751024302095175,
+            0.0569821260869503,
+            -0.015075075440108776,
+            0.002515026368200779,
+            -0.026515239849686623,
+            0.016213813796639442,
+            -0.014863918535411358,
+            -0.012149048037827015,
+            -0.013302868232131004,
+            -0.0031296429224312305,
+            -0.0158216655254364,
+            -0.02737494930624962,
+            -0.001026560552418232,
+            -0.017616497352719307,
+            0.005162025801837444,
+            -0.014464229345321655,
+            0.008069200441241264,
+            -0.024825986474752426,
+            -0.013506483286619186,
+            -0.011063099838793278,
+            -0.024373508989810944,
+            0.015082616358995438,
+            -0.030617712065577507,
+            -0.007790171541273594,
+            0.014449146576225758,
+            0.015218360349535942,
+            -0.007058664690703154,
+            -0.012179212644696236,
+            -0.0075563909485936165,
+            -0.00038366406806744635,
+            -0.05520237609744072,
+            0.002515026368200779,
+            -0.012699563056230545,
+            0.006836195942014456,
+            -0.028310071676969528,
+            0.005146943032741547,
+            0.014351109974086285,
+            0.023000990971922874,
+            0.024267930537462234,
+            0.008385934866964817,
+            0.0024490398354828358,
+            -0.01989397220313549,
+            0.02689230628311634,
+            0.005252521485090256,
+            -0.021839629858732224,
+            0.01313695963472128,
+            -0.003996893297880888,
+            -0.00015777831140439957,
+            -0.01303138118237257,
+            -0.005248750559985638,
+            0.012284791097044945,
+            0.007390482351183891,
+            -0.005791724659502506,
+            -0.007971162907779217,
+            -0.013159583322703838,
+            -0.002450925298035145,
+            0.00605944124981761,
+            0.010467336513102055,
+            0.005803036969155073,
+            0.01044471189379692,
+            -0.016847282648086548,
+            0.0029297983273863792,
+            0.005263833329081535,
+            -0.00219075009226799,
+            0.003046688623726368,
+            -0.007744923699647188,
+            0.01923033595085144,
+            -0.013461235910654068,
+            -0.00782033707946539,
+            0.00209271302446723,
+            -0.02259376086294651,
+            -0.016832200810313225,
+            0.009313516318798065,
+            -0.027902841567993164,
+            0.016500383615493774,
+            -0.002175667556002736,
+            -0.002022956032305956,
+            -0.007903290912508965,
+            0.012465782463550568,
+            0.0031635789200663567,
+            -0.01495441421866417,
+            -0.0007597868097946048,
+            0.0009223962551914155,
+            -0.028883211314678192,
+            0.011621156707406044,
+            0.015459681861102581,
+            -0.006029276177287102,
+            -0.00030047399923205376,
+            -0.029018955305218697,
+            0.009494507685303688,
+            0.0001621381234144792,
+            -0.001130253542214632,
+            0.008793165907263756,
+            0.020255954936146736,
+            0.008227567188441753,
+            -0.012759894132614136,
+            0.012827765196561813,
+            0.013393363915383816,
+            0.005806807428598404,
+            0.00752245495095849,
+            0.024644995108246803,
+            -0.0047962721437215805,
+            0.011621156707406044,
+            -0.011651321314275265,
+            0.00961516797542572,
+            -0.0104145472869277,
+            0.0004543638206087053,
+            0.016605960205197334,
+            0.012307415716350079,
+            -0.005459907464683056,
+            0.005033823195844889,
+            -0.015474764630198479,
+            0.014909166842699051,
+            -0.020633019506931305,
+            0.0008964730077423155,
+            0.012737269513309002,
+            -0.0036066307220607996,
+            0.005599421449005604,
+            0.0135818962007761,
+            0.03830984607338905,
+            0.015851831063628197,
+            0.012480865232646465,
+            0.006093377247452736,
+            -0.02616834081709385,
+            0.003171120071783662,
+            0.0023114110808819532,
+            0.006119771860539913,
+            -0.0024471546057611704,
+            0.004539867863059044,
+            -0.010459794662892818,
+            0.00020526499429252,
+            0.010210931301116943,
+            -0.017118770629167557,
+            0.012865472584962845,
+            -0.010919814929366112,
+            0.007345234509557486,
+            -0.004445601254701614,
+            -0.010919814929366112,
+            -0.021221240982413292,
+            -0.005999110639095306,
+            0.018265048041939735,
+            -0.031824320554733276,
+            0.010716198943555355,
+            0.007039811462163925,
+            0.0288228802382946,
+            -0.02892845869064331,
+            -0.011824771761894226,
+            0.0033841622062027454,
+            -0.006108460016548634,
+            -0.009215478785336018,
+            0.00837085209786892,
+            0.01767682656645775,
+            -0.009652874432504177,
+            0.01534656248986721,
+            0.010821777395904064,
+            -0.006236662156879902,
+            0.03924496844410896,
+            0.007250967901200056,
+            0.016108235344290733,
+            -0.007895749993622303,
+            0.005546632222831249,
+            0.005490072537213564,
+            0.01576133444905281,
+            0.008061658591032028,
+            -0.02021070569753647,
+            0.0027016736567020416,
+            -0.0011566481553018093,
+            0.024539416655898094,
+            0.0052562919445335865,
+            0.005595650989562273,
+            0.0020663184113800526,
+            0.00731506897136569,
+            0.015987573191523552,
+            0.015271149575710297,
+            -0.007213261444121599,
+            -0.007130307145416737,
+            0.007624262943863869,
+            0.003165464149788022,
+            -0.003484084503725171,
+            0.001214150688610971,
+            -0.012518571689724922,
+            -0.004053453449159861,
+            0.017013192176818848,
+            -0.0067381588742136955,
+            -0.001416823361068964,
+            -0.012737269513309002,
+            -0.02452433481812477,
+            0.013453694060444832,
+            0.0018674165476113558,
+            -0.011651321314275265,
+            0.025549951940774918,
+            -0.007695905398577452,
+            -0.011123429983854294,
+            -0.01625906117260456,
+            -0.016334474086761475,
+            0.0139966681599617,
+            0.018280131742358208,
+            0.0010058219777420163,
+            0.0046831523068249226,
+            -0.003197514684870839,
+            -0.0006815457600168884,
+            0.022895412519574165,
+            -0.010776530019938946,
+            0.009728288277983665,
+            -0.002498058369383216
+          ],
+          "keyphrases": [
+            "routing method",
+            "sparsely activated mixture-of-experts",
+            "load imbalance",
+            "training efficiency improvements",
+            "GLUE and SuperGLUE benchmark"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "52b6136e-418a-4195-828b-4a56a673b905",
+      "type": "child",
+      "source": {
+        "id": "b570d71e-4f83-486e-b70f-b0c6217db455",
+        "properties": {
+          "page_content": "Mixture-Of-Experts With Expert Choice Routing\n\nYanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon Google, Mountain View, CA, USA {yanqiz, taole, hanxiaol, dunan, huangyp, vzhao, adai, zhifengc, qvl, jlaudon}@google.com\n\nAbstract\n\nSparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choice method. Instead of letting tokens select the top-k experts, we have experts selecting the top-k tokens. As a result, each token can be routed to a variable number of experts and each expert can have a fixed bucket size. We systematically study pre-training speedups using the same computational resources of the Switch Transformer top-1 and GShard top-2 gating of prior work and find that our method improves training convergence time by more than 2×. For the same computational cost, our method demonstrates higher performance in fine-tuning 11 selected tasks in the GLUE and SuperGLUE benchmarks. For a smaller activation cost, our method outperforms the T5 dense model in 7 out of the 11 tasks.\n\n1 Introduction\n\nScaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27]. The final model quality has been found to have a power-law relationship with the amount of data, model size, and compute time [16, 20]. However, training efficiency, which is defined as the total amount of computation used to achieve superior model quality than the state of the art system [21], should receive greater attention as we increase our efforts towards green AI [29].\n\nSparsely gated mixture-of-experts [31] (MoE) provides an effective way to scale model capacity given a fixed computational cost, and has recently played an important role in increasing the training efficiency of large-scale language models [10, 21]. MoE operate by adopting a number of experts, each as a sub-network, and by activating only one or a few experts for each input token. A gating network must be chosen and optimized in order to route each token to the most suited expert(s). For example, recent work has implemented sparse routing via k-means clustering [12], linear assignment to maximize token-expert affinities [22], or hashing [8, 28]. Many of the prior work use a routing strategy concerning the token choice, where each token selects the best one or two experts.\n\nWe argue that the independent token choice of prior work often leads to an imbalanced load of experts, which causes training inefficiency and sub-optimal training of the model. In order to mitigate this\n\nissue, previous sparsely gated networks introduce additional auxiliary losses as regularization to prevent too many tokens being routed to a single expert, but the effectiveness is still limited. Recent approaches [8, 22, 28] explore alternative strategies for routing, but they focus on pre-training only and do not demonstrate performance gain on downstream tasks. Moreover, none of the previous methods consider allocating a variable number of experts to each token based on importance, which can be beneficial.\n\nWe propose a very simple yet effective routing method we are calling expert choice. Unlike conventional MoE where tokens select one or two top-scoring experts, our method lets each expert pick the top-k tokens. Our method guarantees perfect load balancing, allows a variable number of experts for each token, and achieves substantial gains in training efficiency and downstream performance as demonstrated in our experiments. Our major contributions include:\n\nWe identify common pitfalls in conventional MoE such as load imbalance as described in Section 3.1. We then propose a heterogeneous, expert choice method to provide a fluid allocation of model parameters based on a learnt token-to-expert importance. This method intrinsically guarantees load balance without imposing an auxiliary loss.\n\nWe show our method provides over 2× faster training convergence in a 8B/64E (8 billion activated parameters, 64 experts) model, compared to the top-1 and top-2 gating counterparts in Switch Transformer [10] and GShard [21].\n\nWe show our method demonstrates strong scaling when increasing the number of experts from 16 to 128, evaluated in training perplexity.\n\nWe show our method demonstrates strong performance on downstream tasks selected from GLUE and SuperGLUE at all the evaluated scales. More specifically, our 8B/64E model outperforms a T5 11B dense model in 7 out of 11 tasks evaluated.\n\n2 Related Work\n\nScaling: Various approaches have been proposed to scale up neural network capacity to improve performance. Recent works have successfully scaled models to billions of parameters via various forms of model parallelism [2, 21, 26, 27, 33]. Model parallelism [30] splits weights and tensors across multiple cores while pipeline parallelism [18, 24] splits different layers across devices with micro-batches pipelined to the different layers. To enable continued scaling of neural networks, improving model training and serving efficiency has become a critical research area.\n\nConditional Computation: Computation decisions can be made dynamically based on the input [23, 25]. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation, by activating certain parameters and computation on demand, on a per-example or per-token basis [3]. Conditional convolution layers [1] with task-specific gating has been used to combat catastrophic forgetting when a sequence of learning problems are optimized. The gating decisions may be binary or sparse and continuous, stochastic or deterministic.\n\nMixture of Experts: Sparsely-gated MoE [31] is the first model to demonstrate massive improvements in model capacity, training time, or model quality with gating. Switch Transformer [10] simplifies the gating by selecting only the top expert per token using a softmax over the hidden state and demonstrates better scaling than previous work. All the prior work requires an auxiliary loss to explicitly encourage balancing. This loss term has to be carefully weighted to not overwhelm the primary loss. However, auxiliary loss does not guarantee balancing and a hard capacity factor has to be imposed. As a result, many tokens can still be unprocessed by the MoE layer. Hard MoE [12] with a single decoding layer can be efficiently trained to good effect on large scale hashtag prediction tasks.\n\nBase Layers [22] formulate a linear assignment that maximizes token-expert affinities while ensuring each expert receives an equal number of tokens. Hash layers [8, 28] devise hashing techniques on input tokens. However, the evaluations are limited to pre-training perplexity. THOR [? ] randomly activates experts during training and inference and is trained with a consistency regularization loss.\n\nTHOR has demonstrated strong performance on translation tasks. Different from these prior works, our method is a learnt method that enables heterogeneous MoE and effectively improves downstream fine-tuning performance.\n\n3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n5 Conclusion\n\nWe propose a new routing method for sparsely activated mixture-of-experts (MoE) models. This method addresses load imbalance and under-utilization of experts in conventional MoE methods, and enables selecting different numbers of experts for each token. Our model demonstrates more than 2x training efficiency improvements when compared to the state-of-the-art GShard and Switch Transformer models, and also achieves strong gains when finetuning on 11 datasets in the GLUE and SuperGLUE benchmark.\n\n6 Limitations\n\nThe expert choice method might not immediately apply to auto-regressive text generation as our current implementation takes in the past and future tokens to perform the top-k selection. One possible solution is to collect a large batch of input sequences, dispatch tokens of the same sequence into separate groups, and perform expert choice routing for each group. Another scenario where the expert choice method does not immediately apply is when the batch size becomes very small during serving or inference. A global top-k can be selected instead and we can cap the number of times each expert or token gets selected. We leave these possible improvements for future work.\n\nAnother long-standing issue with MoE has been the large memory footprint. Even though computational cost can be reduced using sparsely gated networks, the total number of parameters increases linearly or sub-linearly with the number of experts. Increasing the number of experts requires reservation of a large number of hardware devices. Therefore, dynamic (used) power is saved while static (reserved) power is not. Power saving techniques such as the ability to put hardware devices into low power states while not in use [17] can help with reducing the reserved power requirements.\n\nReferences\n\n[1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pages 3930–3939. Computer Vision Foundation / IEEE, 2020.\n\n[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. In Advances in Neural Information Processing Systems.\n\n[3] Kyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning, 2014.\n\n[4] Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems, 2021.\n\n[5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.\n\nTransformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.\n\n[6] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 933–941. JMLR.org, 2017.\n\n[7] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixtureof-experts, 2021.\n\n[8] Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan.\n\nTricks for training sparse translation models, 2021.\n\n[9] Richard L Dykstra. An iterative procedure for obtaining i-projections onto the intersection of convex sets. The annals of Probability, pages 975–984, 1985.\n\n[10] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.\n\n[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. NAS-FPN: learning scalable feature pyramid architecture for object detection. In CVPR, pages 7036–7045. Computer Vision Foundation / IEEE, 2019.\n\n[12] Sam Gross, Marc'Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision, 2017.\n\n[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.\n\n[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 630–645, Cham, 2016. Springer International Publishing.\n\n[15] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2016.\n\n[16] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.\n\n[17] Ping Huang, Zuocheng Xing, Tianran Wang, Qiang Wei, Hongyan Wang, and Guitao Fu. A brief survey on power gating design. In 2010 10th IEEE International Conference on Solid-State and Integrated Circuit Technology, pages 788–790, 2010.\n\n[18] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 103–112, 2019.\n\n[19] Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David A. Patterson. A domain-specific supercomputer for training deep neural networks. Commun. ACM, 63(7):67–78, 2020.\n\n[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.\n\n[21] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.\n\n[22] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6265–6274. PMLR, 18–24 Jul 2021.\n\n[24] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. New York, NY, USA, 2019. Association for Computing Machinery.\n\n[25] Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, Cédric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. In ICLR. OpenReview.net, 2021.\n\n[26] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.\n\n[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.\n\n[28] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models, 2021.\n\n[30] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 10435–10444, Red Hook, NY, USA, 2018. Curran Associates Inc.\n\n[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E.\n\nHinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In ICLR (Poster). OpenReview.net, 2017.\n\n[32] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 10–15 Jul 2018. [23] Min Lin, Jie Fu, and Yoshua Bengio. Conditional computation for continual learning, 2019.\n\n[29] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.\n\n[33] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.\n\n[34] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc.\n\n[35] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.\n\nGLUE: A multi-task benchmark and analysis platform for natural language understanding.\n\nIn Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, November 2018. Association for Computational Linguistics.\n\n[36] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake A. Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen. GSPMD: general and scalable parallelization for ML computation graphs. CoRR, abs/2105.04663, 2021.\n\n7 Checklist\n\n(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Yes (b) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes (c) Did you discuss any potential negative societal impacts of your work? N/A. Not any. (d) Did you describe the limitations of your work? Yes (a) Did you include the code, data, and instructions needed to reproduce the main experimental results? Yes. We include details in the experiment setup to help reproduce the main results. (b) Did you specify all the training details? Yes (c) Did you report error bars? Yes (d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? Yes (a) If your work uses existing assets, did you cite the creators? Yes (b) Did you mention the license of the assets? No. The used dataset is not released yet. (c) Did you include any new assets either in the supplemental material or as a URL? No. The dataset is not released yet.\n\n(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? No. Not using persons' data. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes. The dataset does not contain any personally identifiable information or offensive content.\n\nA Comparison On Fine-Tuning With A Dense Model\n\nOur 8B MoE model achieves stronger pre-training perplexity than its dense counterpart. However, a better perplexity does not always directly translate to downstream performance as demonstrated in Section 4.4. To this end, we compare fine-tuning performance of the 8B dense model and MoE model in Table 1. As shown in the table, our MoE model using expert choice routing consistently outperforms the dense model across the 11 tasks in GLUE and SuperGLUE.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 1: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nB Capacity Factor\n\nWe evaluate the downstream task fine-tuning performance by varying the capacity factors. Note that a capacity factor of n indicates on average how many experts each token can be received. EC-CF2 is our baseline expert choice, which matches GShard top-2 gating computational footprint. EC-CF1, however, matches Switch Transformer top-1 gating computational footprint. EC-CF0.5 further verifies that an aggressively lowered capacity factor can provide strong enough performance, that almost matches the top-2 gating baseline.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Top-2 78.1 87.0 88.3 85.0 82.6 90.1 90.7 81.6 94.7 68.2 67.2 83.0±0.3 EC-CAP2 78.2 88.0 88.5 85.7 83.0 90.8 91.1 80.0 95.4 70.4 64.1 83.2±0.4 EC-CAP3 78.5 91.7 89.3 86.3 83.5 90.9 91.1 81.8 94.9 70.0 65.6 84.0±0.4 EC-CF2 79.1 89.6 89.3 86.8 84.3 91.3 91.2 81.1 95.2 68.1 68.0 84.0±0.2 EC-CF1 77.4 90.6 88.0 85.5 83.6 90.3 91.2 79.8 95.3 66.5 64.9 83.0±0.2 EC-CF0.5 77.4 89.6 86.3 85.2 82.7 91.7 91.0 79.6 94.9 67.3 63.5 83.0 ±0.05 Hash Layers 76.1 85.2 86.7 83.4 82.5 90.0 90.3 75.7 94.0 67.4 63.3 81.3±1.0\n\nTable 2: Comparison between different routing methods in fine-tuning of 100M/64E models. We perform 3 independent fine-tuning runs for each method and report the average results. This gives more accurate difference between the variants of expert choice method, since they achieve close fine-tuning results. We do not report averaged results in other experiments.\n\nC Capped Expert Choice\n\nAs described in Section 4.5, the maximum number of experts each token is assigned can be capped by an entropy-regularized linear programming. Figure 1 compares the validation perplexity when training the 100M/64E models using the base expert choice method (EC-BASE), expert choice capped by two experts per token (EC-CAP2), expert choice capped by three experts per token (EC-CAP3), and GShard top-2 gating.\n\nAs shown in the figure, restricting the number of experts to 2 degrades the perplexity compared to the base expert choice method. This suggests that a more flexible allocation of experts (e.g. more than 2 experts for a token) can enhance model expressiveness. On the other hand, our EC-CAP2 and EC-CAP3 methods still outperform the top-2 gating method by a clear margin. We believe this confirms the effectiveness of a load balanced training, provided by our method. Finally, EC-CAP3 obtains comparable perplexity to EC-BASE. As indicated by Figure 3, only a little fraction of tokens use more than 3 experts therefore we see little or no difference between EC-BASE and EC-CAP3 variants. We present the fine-tuning results of these methods in Table 2.\n\nD Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [? ]. We use mod x to map a token ID to an expert ID. This in some way ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 2. Hashing based routing performs much worse than expert choice in terms of average scores and variance.\n\nE Fine-Tuning Details\n\nWe did a hyperparameter search for both baseline models and expert choice method. For fine-tuning of the 8B dense model, we use a constant learning rate of 0.0001 and a dropout rate of 0.1. We freeze the attention layer and feed-forward layer while leaving the embedding and layer normalization trainable. This setting has been found optimal for the 8B dense model. For MoE 8B/64E models including GShard top-2 gating and expert choice, we found continuing the learning rate from the pre-trained model while using a square root learning rate decay works better. In addition, we do not apply parameter freezing for fine-tuning MoE models. For models with 100M expert size, we use a constant learning rate of 0.0001 and no dropout is used.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/2202.09368v2.md"
+          },
+          "headlines": {
+            "1 Introduction": [],
+            "2 Related Work": [],
+            "3 Method": [
+              "3.1 Pitfalls Of Token-Choice Routing",
+              "3.2 Heterogeneous Moe Via Expert Choice",
+              "3.3 Expert Choice With Additional Constraint",
+              "3.4 Model Architecture"
+            ],
+            "4 Experiments": [
+              "4.1 Setup",
+              "4.2 Training Efficiency",
+              "4.3 Scaling The Number Of Experts",
+              "4.4 Fine-Tuning On Glue And Superglue",
+              "4.5 Heterogeneity Matters",
+              "4.6 Comparison With Hash Layer",
+              "4.7 Ablation"
+            ],
+            "5 Conclusion": [],
+            "6 Limitations": [],
+            "A Comparison On Fine-Tuning With A Dense Model": [],
+            "B Capacity Factor": [],
+            "C Capped Expert Choice": [],
+            "D Comparison With Hash Layer": [],
+            "E Fine-Tuning Details": []
+          },
+          "summary": "The paper introduces a new routing method for sparsely-activated Mixture-of-Experts (MoE) models called 'expert choice,' which addresses load imbalance and under-utilization issues in conventional MoE methods. Unlike traditional token-choice routing, expert choice allows experts to select the top-k tokens, ensuring perfect load balancing and enabling variable expert allocation per token. This method improves training efficiency, achieving over 2x faster convergence compared to existing models like Switch Transformer and GShard. It also demonstrates superior performance on downstream tasks in the GLUE and SuperGLUE benchmarks. The study highlights the benefits of flexible expert allocation and shows that expert choice outperforms hashing-based routing methods. The paper also discusses potential limitations and future improvements for the expert choice method.",
+          "summary_embedding": [
+            -0.009726986289024353,
+            -0.0337262786924839,
+            -0.023302342742681503,
+            -0.019878201186656952,
+            -0.025953780859708786,
+            0.04560471698641777,
+            -0.006874797400087118,
+            0.039665497839450836,
+            -0.006973279174417257,
+            0.034544438123703,
+            -0.025787118822336197,
+            -0.00363057479262352,
+            -0.01052999310195446,
+            0.0054657477885484695,
+            -0.024499278515577316,
+            0.036786794662475586,
+            0.010007280856370926,
+            0.003272630739957094,
+            0.006102093029767275,
+            -0.04048365354537964,
+            -0.018666116520762444,
+            -0.013522328808903694,
+            -0.03230207785964012,
+            0.008969432674348354,
+            0.02484775334596634,
+            0.0016713523073121905,
+            0.0026230288203805685,
+            0.003624893259257078,
+            -0.05311964824795723,
+            0.04339266195893288,
+            0.018681267276406288,
+            0.023711422458291054,
+            -0.016332851722836494,
+            -0.012832955457270145,
+            -0.039120059460401535,
+            -0.0197418425232172,
+            0.030786970630288124,
+            0.039968520402908325,
+            -0.019287310540676117,
+            0.03357476741075516,
+            0.018060073256492615,
+            0.02454473078250885,
+            -0.021272100508213043,
+            -0.029817303642630577,
+            -0.0061892117373645306,
+            -0.007840678095817566,
+            -0.008355814032256603,
+            0.01242387667298317,
+            0.016984347254037857,
+            -0.013007191941142082,
+            0.0212569497525692,
+            -0.01371171697974205,
+            0.03384748846292496,
+            0.008916404098272324,
+            -0.042332086712121964,
+            0.03984731063246727,
+            -0.008007340133190155,
+            0.012105703353881836,
+            0.03281721472740173,
+            0.010333028621971607,
+            -0.005287723150104284,
+            -0.021544819697737694,
+            -0.03378688171505928,
+            0.0010529992869123816,
+            0.00014014738553669304,
+            0.005950582213699818,
+            -0.003069985192269087,
+            -0.009742137044668198,
+            0.03227177634835243,
+            -0.003939277958124876,
+            -0.0740584209561348,
+            0.050089433789253235,
+            -0.026590125635266304,
+            -0.007628562860190868,
+            -0.028938541188836098,
+            -0.01736312545835972,
+            0.037362534552812576,
+            0.005662711802870035,
+            0.0098709212616086,
+            0.0106284748762846,
+            -0.0007892759749665856,
+            0.028923390433192253,
+            0.00910579226911068,
+            -0.024923507124185562,
+            0.04478655755519867,
+            -0.04790768027305603,
+            0.01801462098956108,
+            -0.04336235672235489,
+            0.022756904363632202,
+            1.6393929399782792e-05,
+            -0.020726662129163742,
+            -0.0013304532039910555,
+            -0.016666175797581673,
+            0.0218023881316185,
+            -0.01730252057313919,
+            0.026802239939570427,
+            0.013825350441038609,
+            -0.015802564099431038,
+            -0.029817303642630577,
+            0.025378039106726646,
+            0.003367325058206916,
+            0.010499690659344196,
+            -0.02075696364045143,
+            0.014332910999655724,
+            0.04393810033798218,
+            -0.02593863010406494,
+            -0.02271145209670067,
+            0.027681002393364906,
+            0.014408666640520096,
+            0.0633314698934555,
+            0.036544378846883774,
+            0.006624804809689522,
+            0.022362977266311646,
+            0.020575150847434998,
+            -0.028968842700123787,
+            -0.01963578537106514,
+            -0.046362269669771194,
+            -0.04757435619831085,
+            0.019878201186656952,
+            -0.04566532000899315,
+            0.0050869714468717575,
+            0.038544319570064545,
+            -0.013704141601920128,
+            -0.014446543529629707,
+            -0.04021093621850014,
+            0.017847958952188492,
+            0.0003908502112608403,
+            0.011976920068264008,
+            -0.0022082682698965073,
+            0.00033285003155469894,
+            -0.011825408786535263,
+            0.008878526277840137,
+            -0.03778676688671112,
+            -0.025196226313710213,
+            0.025181075558066368,
+            0.022862963378429413,
+            0.024120500311255455,
+            0.028802180662751198,
+            -0.03077181987464428,
+            -0.009984553791582584,
+            0.021378157660365105,
+            -0.04242299124598503,
+            -0.006556625012308359,
+            0.011332998983561993,
+            0.0026703758630901575,
+            0.003872992005199194,
+            0.032786913216114044,
+            0.052998438477516174,
+            -0.03172633796930313,
+            0.02065090648829937,
+            -0.010431510396301746,
+            0.04957429692149162,
+            0.00024691506405360997,
+            0.028968842700123787,
+            -0.019514575600624084,
+            -0.0134389977902174,
+            -0.029529431834816933,
+            -0.009795165620744228,
+            -0.01356020662933588,
+            -0.008484598249197006,
+            -0.002998017705976963,
+            -0.009575475007295609,
+            0.0040453351102769375,
+            0.0331808403134346,
+            0.00273855566047132,
+            0.030680913478136063,
+            0.014325335621833801,
+            0.0051475754007697105,
+            -0.08745196461677551,
+            0.026059838011860847,
+            -0.0033616432920098305,
+            0.005859675817191601,
+            0.0062460280023515224,
+            0.03013547509908676,
+            -0.0312718041241169,
+            0.020938776433467865,
+            0.026620427146553993,
+            0.0011145505122840405,
+            0.016969196498394012,
+            0.036847397685050964,
+            -0.04181694984436035,
+            0.009643655270338058,
+            0.012590537779033184,
+            -0.0033881578128784895,
+            0.04584713280200958,
+            0.011628445237874985,
+            0.019514575600624084,
+            0.018529755994677544,
+            0.003569970605894923,
+            0.00297907879576087,
+            0.006533898413181305,
+            0.03724132478237152,
+            0.003462019143626094,
+            -0.009833043441176414,
+            0.014976831153035164,
+            -0.010757259093225002,
+            -0.011219366453588009,
+            -0.037998881191015244,
+            0.014454119838774204,
+            0.008742166683077812,
+            -0.017454031854867935,
+            -0.02959003672003746,
+            0.0026949963066726923,
+            -0.031908150762319565,
+            0.026559824123978615,
+            -0.002445003716275096,
+            0.005738467443734407,
+            -0.010454237461090088,
+            -0.006757376715540886,
+            -0.035423196852207184,
+            0.028453707695007324,
+            -0.02095392718911171,
+            0.0053142374381423,
+            0.0007258309051394463,
+            -0.029438525438308716,
+            0.0060073984786868095,
+            0.017863109707832336,
+            -0.0025965142995119095,
+            0.0025302283465862274,
+            -0.02349930815398693,
+            0.024468975141644478,
+            0.025635607540607452,
+            0.004518806003034115,
+            -0.014037464745342731,
+            -0.013113250024616718,
+            -0.003605954349040985,
+            -0.009742137044668198,
+            -0.016196493059396744,
+            0.024468975141644478,
+            -0.014810169115662575,
+            0.00308892410248518,
+            0.021847840398550034,
+            0.014264730736613274,
+            -0.007045246660709381,
+            -0.004007457755506039,
+            -0.04066546633839607,
+            -0.01218903437256813,
+            -0.04760465770959854,
+            -0.011628445237874985,
+            -0.0034601252991706133,
+            -0.028696123510599136,
+            0.0061513339169323444,
+            -0.04481685906648636,
+            -0.01235569640994072,
+            0.02309022843837738,
+            0.018135828897356987,
+            0.0019488062243908644,
+            -0.009173971600830555,
+            -0.022605394944548607,
+            8.486965816700831e-05,
+            -0.01763584464788437,
+            -0.017044952139258385,
+            0.015393485315144062,
+            0.031059689819812775,
+            -0.03101423755288124,
+            0.04024123772978783,
+            0.003893824527040124,
+            -0.08066429197788239,
+            0.010393633507192135,
+            -0.026362858712673187,
+            0.07011914253234863,
+            -0.0025453795678913593,
+            0.014075342565774918,
+            0.06781618297100067,
+            -0.012431452050805092,
+            -0.02007516659796238,
+            0.03142331540584564,
+            -0.008249756880104542,
+            -1.1311515663692262e-05,
+            -0.02884763479232788,
+            -0.02739313244819641,
+            -0.053271155804395676,
+            -0.009704259224236012,
+            -0.010787560604512691,
+            0.03360506892204285,
+            -0.024317465722560883,
+            -0.0007656024536117911,
+            0.012620840221643448,
+            0.02983245439827442,
+            0.016635872423648834,
+            -0.0015681355725973845,
+            0.008272483013570309,
+            0.02881733328104019,
+            0.0274234339594841,
+            -0.0036154238041490316,
+            0.00814369972795248,
+            -0.034211114048957825,
+            -0.0068066176027059555,
+            -0.0493621826171875,
+            0.011302697472274303,
+            -0.02596893161535263,
+            -0.02596893161535263,
+            0.007643714081496,
+            -0.011476934887468815,
+            0.03748374432325363,
+            -0.024696242064237595,
+            -0.007503566797822714,
+            0.005037730094045401,
+            0.012855681590735912,
+            -0.017757052555680275,
+            -0.03612014651298523,
+            -0.0028502948116511106,
+            0.0026381798088550568,
+            0.007560383062809706,
+            -0.053240854293107986,
+            0.004499867558479309,
+            -0.018469152972102165,
+            0.0279688723385334,
+            -0.02139330841600895,
+            -0.013688990846276283,
+            0.042029064148664474,
+            0.011363301426172256,
+            0.027499189600348473,
+            -0.026468917727470398,
+            0.0017849853029474616,
+            0.012825379148125648,
+            -0.0020056227222085,
+            0.032514192163944244,
+            -0.030574856325984,
+            -0.03257479891180992,
+            0.001548249856568873,
+            -0.007840678095817566,
+            -0.0625436082482338,
+            -0.011386028490960598,
+            -0.00981789268553257,
+            0.017317671328783035,
+            0.0446956530213356,
+            0.04557441547513008,
+            -0.01068150345236063,
+            0.002174178371205926,
+            -0.02213571034371853,
+            -0.04542290419340134,
+            -0.02643861435353756,
+            0.0016486257081851363,
+            -0.019999410957098007,
+            -0.02789311669766903,
+            -0.0018313854234293103,
+            0.011211791075766087,
+            -0.013651113025844097,
+            0.01134057529270649,
+            -0.018560059368610382,
+            -0.01862066239118576,
+            -0.012332970276474953,
+            0.018575210124254227,
+            0.009901223704218864,
+            -0.0018796793883666396,
+            -0.015037435106933117,
+            -0.00980274099856615,
+            0.018969137221574783,
+            -0.011279970407485962,
+            0.0009388927719555795,
+            0.01188601367175579,
+            0.024347767233848572,
+            -0.007083124481141567,
+            0.010961798019707203,
+            -0.034756552428007126,
+            -0.01966608688235283,
+            0.007984613068401814,
+            0.002719616750255227,
+            0.020711511373519897,
+            0.008742166683077812,
+            -0.04275631532073021,
+            -0.044301725924015045,
+            0.016044981777668,
+            -0.0562104657292366,
+            0.028120383620262146,
+            -0.03327174484729767,
+            -0.012514783069491386,
+            -0.02671133354306221,
+            0.0013418165035545826,
+            -0.01828734017908573,
+            0.022438732907176018,
+            0.03778676688671112,
+            -0.0011486404109746218,
+            0.005503625608980656,
+            0.014658658765256405,
+            0.04384719207882881,
+            -0.05021063983440399,
+            -0.041180603206157684,
+            0.03754434734582901,
+            -0.01601468026638031,
+            0.0185146052390337,
+            0.00021744151308666915,
+            0.010348179377615452,
+            -0.014431392773985863,
+            0.022772056981921196,
+            0.013757170177996159,
+            -0.008764892816543579,
+            -0.030574856325984,
+            0.04945308715105057,
+            -0.023650817573070526,
+            -0.049513693898916245,
+            0.0018323323456570506,
+            0.018878230825066566,
+            0.018635815009474754,
+            -0.014870774000883102,
+            -0.02907489985227585,
+            -0.024317465722560883,
+            -0.006060427520424128,
+            -0.00947699323296547,
+            -0.03402930125594139,
+            0.011363301426172256,
+            0.0048445542342960835,
+            0.016666175797581673,
+            -0.00645056739449501,
+            -0.0073141781613230705,
+            -0.009370936080813408,
+            0.022559940814971924,
+            0.012635990977287292,
+            -0.03175663948059082,
+            -0.017484333366155624,
+            -0.02528713271021843,
+            0.004359719809144735,
+            0.010136065073311329,
+            0.022166013717651367,
+            0.024287162348628044,
+            -0.0323626808822155,
+            0.04218057543039322,
+            0.01550711877644062,
+            -0.008727015927433968,
+            -0.0019317612750455737,
+            0.02478714846074581,
+            0.04324115067720413,
+            0.0218023881316185,
+            0.026514369994401932,
+            0.002107892418280244,
+            8.705945947440341e-05,
+            -0.015620751306414604,
+            -0.0011391709558665752,
+            0.01862066239118576,
+            0.04911976307630539,
+            -0.012151156552135944,
+            0.01716616004705429,
+            -0.03578682243824005,
+            0.00396200455725193,
+            -0.02580226957798004,
+            -0.030786970630288124,
+            0.03745344281196594,
+            -0.015060162171721458,
+            0.0191964041441679,
+            0.025044716894626617,
+            -0.022469034418463707,
+            -0.011075431481003761,
+            -0.019423669204115868,
+            -0.026181045919656754,
+            -0.01702980138361454,
+            -0.03281721472740173,
+            -0.018135828897356987,
+            -0.002653330797329545,
+            -0.035635314881801605,
+            -0.04987731948494911,
+            -0.004329417832195759,
+            -0.046392571181058884,
+            -0.010507266037166119,
+            0.017378276214003563,
+            -0.011166337877511978,
+            -0.025711363181471825,
+            -0.005336964037269354,
+            0.003973367623984814,
+            -0.022196315228939056,
+            -0.0066588944755494595,
+            0.008439145050942898,
+            0.035938333719968796,
+            -0.02298417128622532,
+            -0.013060221448540688,
+            -0.0024317465722560883,
+            -0.02887793630361557,
+            -0.007878555916249752,
+            0.008999735116958618,
+            0.007723257411271334,
+            -0.0012357590021565557,
+            -0.05196816474199295,
+            -0.006916462909430265,
+            -0.012992041185498238,
+            0.029196109622716904,
+            0.026468917727470398,
+            0.0012168202083557844,
+            -0.02783251367509365,
+            0.008749742060899734,
+            -0.0072535742074251175,
+            0.02339324913918972,
+            -0.007992188446223736,
+            -0.022908415645360947,
+            -0.02034788578748703,
+            0.030786970630288124,
+            -0.0159086212515831,
+            0.009113367646932602,
+            0.009976978413760662,
+            0.0121360057964921,
+            -0.03899884968996048,
+            -0.001445980160497129,
+            0.0005648507503792644,
+            -0.010128489695489407,
+            0.00423093605786562,
+            -0.006738437805324793,
+            -0.007079336792230606,
+            -0.022090258076786995,
+            -0.0218023881316185,
+            0.03818069398403168,
+            -0.009113367646932602,
+            -0.0035661826841533184,
+            -0.01088604237884283,
+            0.03436262533068657,
+            -0.00595815759152174,
+            -0.01993880607187748,
+            -0.019347915425896645,
+            0.014431392773985863,
+            0.05105910077691078,
+            -0.028529461473226547,
+            -0.03984731063246727,
+            0.0011136035900563002,
+            0.030150625854730606,
+            0.004140029661357403,
+            -0.016908591613173485,
+            0.0035851215943694115,
+            0.056392278522253036,
+            -0.011742077767848969,
+            0.028362801298499107,
+            0.027771908789873123,
+            0.02295386977493763,
+            -0.02021152526140213,
+            -0.01678738370537758,
+            -0.020999381318688393,
+            0.011098157614469528,
+            -0.03027183562517166,
+            0.027059808373451233,
+            0.018969137221574783,
+            0.01538590993732214,
+            0.05054396390914917,
+            -0.03178694099187851,
+            -0.02515077404677868,
+            0.008499749936163425,
+            0.0021722845267504454,
+            0.011007251217961311,
+            -0.007727045100182295,
+            -0.01801462098956108,
+            -0.001950700068846345,
+            -0.014022313989698887,
+            -0.019817598164081573,
+            -0.020938776433467865,
+            0.00954517349600792,
+            0.015696506947278976,
+            0.002649543108418584,
+            0.010863316245377064,
+            0.0016590419691056013,
+            0.01686313934624195,
+            0.01321930717676878,
+            -0.0444229319691658,
+            -7.936555630294606e-05,
+            0.03378688171505928,
+            -0.0293476190418005,
+            -0.012264790013432503,
+            -0.016060132533311844,
+            -0.004439263138920069,
+            -0.01706010289490223,
+            -0.02762039750814438,
+            -0.006874797400087118,
+            -0.01889338344335556,
+            -0.00814369972795248,
+            -0.028196139261126518,
+            -0.009992130100727081,
+            0.01949942484498024,
+            0.022332675755023956,
+            0.001196934375911951,
+            -0.006783891003578901,
+            -0.005901341326534748,
+            0.02092362567782402,
+            0.032514192163944244,
+            0.01952972821891308,
+            0.0058331615291535854,
+            -0.015158643946051598,
+            -0.004295328166335821,
+            -0.012370847165584564,
+            0.016060132533311844,
+            0.012537509202957153,
+            6.344509893096983e-05,
+            0.03493836522102356,
+            0.004170331638306379,
+            -0.022726602852344513,
+            -0.001230077352374792,
+            0.024923507124185562,
+            -0.021590271964669228,
+            -0.0108330138027668,
+            0.040029123425483704,
+            0.0085679292678833,
+            -0.0024506852496415377,
+            -0.019287310540676117,
+            -0.019075196236371994,
+            0.02630225569009781,
+            -0.02133270539343357,
+            0.002083271974697709,
+            0.0482713058590889,
+            0.00035344602656550705,
+            -0.029802151024341583,
+            0.017954016104340553,
+            -0.013242034241557121,
+            -0.018560059368610382,
+            0.01848430372774601,
+            0.0012859469279646873,
+            -0.039968520402908325,
+            0.019014591351151466,
+            0.018408548086881638,
+            0.022635696455836296,
+            -0.02918095886707306,
+            0.0002127068000845611,
+            0.03233237937092781,
+            0.004037759732455015,
+            0.001595596899278462,
+            -0.008530051447451115,
+            -0.0006065162015147507,
+            0.010431510396301746,
+            0.001746160676702857,
+            0.02854461409151554,
+            -0.0030434709042310715,
+            0.027771908789873123,
+            -0.006727074272930622,
+            -0.008863375522196293,
+            0.019181253388524055,
+            -0.02109028771519661,
+            0.013166278600692749,
+            0.0685434341430664,
+            0.0021950111258774996,
+            0.025847723707556725,
+            0.007181606721132994,
+            0.024347767233848572,
+            0.02018122375011444,
+            -0.027271922677755356,
+            -0.02021152526140213,
+            -0.005397567991167307,
+            -0.0014033677289262414,
+            -0.03787767142057419,
+            0.011128460057079792,
+            -0.021302402019500732,
+            -0.013249609619379044,
+            -0.0022347825579345226,
+            -0.002316219499334693,
+            -0.006405114196240902,
+            0.034211114048957825,
+            -0.0014630251098424196,
+            -0.018560059368610382,
+            -0.035423196852207184,
+            0.037726160138845444,
+            0.04048365354537964,
+            0.0389382466673851,
+            -0.005984671879559755,
+            0.03548380360007286,
+            0.0030756669584661722,
+            0.0025302283465862274,
+            0.041453324258327484,
+            0.004791525658220053,
+            -0.011567841283977032,
+            0.02295386977493763,
+            0.012416300363838673,
+            0.03351416438817978,
+            -0.02045394293963909,
+            -0.02630225569009781,
+            -0.003299145260825753,
+            -0.026423463597893715,
+            0.012492056004703045,
+            -0.019726691767573357,
+            0.0018768386216834188,
+            0.007117214612662792,
+            0.015196521766483784,
+            -0.016044981777668,
+            0.014719262719154358,
+            -0.006102093029767275,
+            -0.021014532074332237,
+            0.011719351634383202,
+            -0.007780073676258326,
+            0.011961768381297588,
+            -0.02262054570019245,
+            0.001138224033638835,
+            -0.04063516482710838,
+            0.008280059322714806,
+            0.014022313989698887,
+            -0.01598437689244747,
+            0.03205966204404831,
+            0.020969079807400703,
+            -0.01801462098956108,
+            -0.0009682480012997985,
+            0.0034714885987341404,
+            -0.010393633507192135,
+            -0.011067855171859264,
+            -0.019923655316233635,
+            0.02786281518638134,
+            -0.015802564099431038,
+            -0.025908326730132103,
+            0.02454473078250885,
+            -0.01743887923657894,
+            0.015151068568229675,
+            -0.004299115855246782,
+            0.00888610165566206,
+            -0.019120648503303528,
+            0.02284781076014042,
+            -0.023741723969578743,
+            -0.0009128518868237734,
+            -0.020287280902266502,
+            0.0028389315120875835,
+            0.014135947450995445,
+            -0.018272189423441887,
+            -0.02139330841600895,
+            0.019150950014591217,
+            -0.005420294590294361,
+            -0.003931702114641666,
+            0.010014856234192848,
+            0.0030983935575932264,
+            -0.0029715034179389477,
+            -0.004549108445644379,
+            -0.03281721472740173,
+            -0.02915065549314022,
+            0.0033086147159337997,
+            0.011386028490960598,
+            0.015257125720381737,
+            0.027574945241212845,
+            0.03715042024850845,
+            0.006052851676940918,
+            0.0036949669010937214,
+            0.0009327376610599458,
+            -0.020726662129163742,
+            0.042635105550289154,
+            -0.020469093695282936,
+            0.040847279131412506,
+            0.032544493675231934,
+            -0.0197418425232172,
+            -0.023908386006951332,
+            0.0029052174650132656,
+            0.03115059621632099,
+            0.03151422366499901,
+            0.025211378931999207,
+            0.012310243211686611,
+            0.03690800443291664,
+            -0.004359719809144735,
+            -0.032211173325777054,
+            0.015802564099431038,
+            0.022256920114159584,
+            -0.008105821907520294,
+            0.009673957712948322,
+            0.027438584715127945,
+            0.013279912061989307,
+            -0.007757347077131271,
+            0.030150625854730606,
+            0.0012234487803652883,
+            0.02396899089217186,
+            0.02498411200940609,
+            -0.02190844528377056,
+            0.01743887923657894,
+            -0.0001154677156591788,
+            0.006727074272930622,
+            -0.009840618818998337,
+            0.004348356742411852,
+            0.011810258030891418,
+            0.018272189423441887,
+            0.018908534198999405,
+            0.003246116451919079,
+            -0.07060398161411285,
+            0.019726691767573357,
+            0.004507442936301231,
+            -0.02160542458295822,
+            0.00390140013769269,
+            -0.0024866692256182432,
+            0.012704171240329742,
+            0.02105998620390892,
+            0.012461753562092781,
+            -0.01790856383740902,
+            0.0013626492582261562,
+            0.011961768381297588,
+            -0.00595815759152174,
+            0.04730163514614105,
+            0.0019791084341704845,
+            -0.01437078882008791,
+            -0.016105586662888527,
+            0.013726867735385895,
+            0.02525683119893074,
+            -0.02355991117656231,
+            0.006325571332126856,
+            0.005621046293526888,
+            -0.004147605039179325,
+            -0.010742107406258583,
+            -0.030377892777323723,
+            -0.005280147306621075,
+            -0.03436262533068657,
+            0.023150833323597908,
+            0.015491968020796776,
+            0.014764715917408466,
+            -0.025044716894626617,
+            -0.004280176945030689,
+            0.0046400148421525955,
+            -0.021287251263856888,
+            0.003261267440393567,
+            0.01161329448223114,
+            0.0005104016163386405,
+            -0.020984230563044548,
+            0.00959820207208395,
+            0.013469300232827663,
+            0.06848283112049103,
+            0.0007442962378263474,
+            0.029771849513053894,
+            0.0422411784529686,
+            -0.0012243957025930285,
+            -0.006371024530380964,
+            0.0036286809481680393,
+            -0.019953956827521324,
+            0.003473382443189621,
+            0.025878025218844414,
+            -0.03224147483706474,
+            -0.055058982223272324,
+            -0.007545232307165861,
+            -0.04624105989933014,
+            0.001489539397880435,
+            0.014711687341332436,
+            0.023044776171445847,
+            -0.0037782976869493723,
+            -0.034817155450582504,
+            0.016908591613173485,
+            -0.003071879269555211,
+            0.007014944683760405,
+            -0.02593863010406494,
+            -0.0294839795678854,
+            -0.026014383882284164,
+            -0.02339324913918972,
+            -0.012908710166811943,
+            -0.022726602852344513,
+            0.012590537779033184,
+            -0.0015103721525520086,
+            -0.06587684899568558,
+            -0.009780014865100384,
+            0.016166189685463905,
+            0.001200722181238234,
+            0.011939042247831821,
+            0.011083006858825684,
+            -0.007083124481141567,
+            -0.0018275976181030273,
+            -0.022559940814971924,
+            0.03124150261282921,
+            0.005177877843379974,
+            0.03584742918610573,
+            -0.006503595970571041,
+            -0.009128518402576447,
+            0.01073453202843666,
+            0.04527139291167259,
+            0.0017916138749569654,
+            0.010226971469819546,
+            -0.020560000091791153,
+            0.0015065843472257257,
+            -0.02284781076014042,
+            -0.0031135445460677147,
+            -0.008863375522196293,
+            -0.0021211495622992516,
+            0.015893470495939255,
+            0.0019753205124288797,
+            -0.010204244405031204,
+            0.025438643991947174,
+            0.02251448854804039,
+            -0.009605777449905872,
+            -0.015969226136803627,
+            -0.02437806874513626,
+            -0.014545026235282421,
+            -0.0002627763315103948,
+            -0.010007280856370926,
+            -0.022453883662819862,
+            -0.020878173410892487,
+            -0.002566212322562933,
+            -0.02559015527367592,
+            0.022529639303684235,
+            0.01220418605953455,
+            -4.882668508798815e-05,
+            -0.006723286584019661,
+            -0.038544319570064545,
+            -0.003727162955328822,
+            -0.03118089959025383,
+            -0.005374841392040253,
+            0.03287781774997711,
+            -0.03745344281196594,
+            -0.0002231231628684327,
+            0.014491996727883816,
+            -0.01993880607187748,
+            0.0028938541654497385,
+            0.008961857296526432,
+            0.02769615314900875,
+            -0.027408283203840256,
+            0.0001795638381736353,
+            0.006954340264201164,
+            0.016908591613173485,
+            -0.009878496639430523,
+            0.027196168899536133,
+            -0.04275631532073021,
+            0.011037553660571575,
+            -0.037120118737220764,
+            -0.020423639565706253,
+            -0.01523439958691597,
+            0.008363389410078526,
+            -0.005018791649490595,
+            -0.02637801133096218,
+            -0.010454237461090088,
+            0.006026337388902903,
+            -0.00693918950855732,
+            -0.01902974210679531,
+            -0.010620899498462677,
+            0.027438584715127945,
+            0.00931033119559288,
+            0.007977037690579891,
+            -0.02955973520874977,
+            -0.01631770096719265,
+            0.006280118133872747,
+            0.052089374512434006,
+            -0.01584801822900772,
+            0.014938953332602978,
+            -0.006639956030994654,
+            0.02342355251312256,
+            -0.028484009206295013,
+            -0.01872672140598297,
+            0.0315445251762867,
+            0.013946558348834515,
+            0.007064185570925474,
+            0.028953691944479942,
+            0.027196168899536133,
+            0.027105262503027916,
+            -0.04654408246278763,
+            0.009886072017252445,
+            -0.002613559365272522,
+            -0.008757317438721657,
+            0.03402930125594139,
+            0.01916610263288021,
+            0.022150862962007523,
+            0.0009744031121954322,
+            -0.01730252057313919,
+            0.0139844361692667,
+            0.014885924756526947,
+            0.007681591901928186,
+            -0.014242004603147507,
+            -0.02660527639091015,
+            0.003367325058206916,
+            -0.007992188446223736,
+            0.00047276070108637214,
+            0.020726662129163742,
+            -0.009052763693034649,
+            0.030999086797237396,
+            -0.014605630189180374,
+            -0.02593863010406494,
+            0.008916404098272324,
+            -0.02268115058541298,
+            -0.0041741193272173405,
+            0.0004476667381823063,
+            0.010083036497235298,
+            -0.023847781121730804,
+            -0.04669559374451637,
+            0.019408518448472023,
+            0.028393102809786797,
+            0.005579380784183741,
+            0.01328748743981123,
+            -0.035332292318344116,
+            0.008992158807814121,
+            0.012310243211686611,
+            0.011348150670528412,
+            0.0019847899675369263,
+            0.035362593829631805,
+            -0.00028834378463216126,
+            0.012885984033346176,
+            -0.008620957843959332,
+            0.028362801298499107,
+            0.00667025800794363,
+            -0.0323626808822155,
+            -0.0378473699092865,
+            -0.01513591781258583,
+            0.015257125720381737,
+            0.006692984607070684,
+            -0.011840560473501682,
+            -0.02278720773756504,
+            -0.038817036896944046,
+            0.015787413343787193,
+            -0.012916285544633865,
+            -0.038271598517894745,
+            -0.010158791206777096,
+            -0.009742137044668198,
+            0.016635872423648834,
+            -0.006916462909430265,
+            -0.016802534461021423,
+            -0.016090434044599533,
+            0.005821797996759415,
+            7.161247049225494e-05,
+            0.0244083721190691,
+            -0.001251857029274106,
+            -0.006855858489871025,
+            -0.016393456608057022,
+            -0.01188601367175579,
+            -0.0035074723418802023,
+            0.03602924197912216,
+            -0.014219277538359165,
+            -0.02383263036608696,
+            0.014984406530857086,
+            0.02657497487962246,
+            0.011636020615696907,
+            0.03227177634835243,
+            0.007181606721132994,
+            -0.014022313989698887,
+            0.003016956616193056,
+            -0.033029329031705856,
+            -0.0046400148421525955,
+            0.005196816753596067,
+            -0.006427840795367956,
+            -0.015651054680347443,
+            -0.01815097965300083,
+            -0.015832867473363876,
+            0.030756669119000435,
+            -0.02487805485725403,
+            0.0218023881316185,
+            0.030347591266036034,
+            0.003174148965626955,
+            -0.02915065549314022,
+            0.013598084449768066,
+            -0.0072005451656877995,
+            0.01862066239118576,
+            0.0019014590652659535,
+            0.028014326468110085,
+            -0.007461901288479567,
+            -0.004818039946258068,
+            -0.02501441352069378,
+            -0.03696860745549202,
+            -0.023484155535697937,
+            -0.008817922323942184,
+            0.0027537066489458084,
+            0.020984230563044548,
+            -0.0375746488571167,
+            -0.017757052555680275,
+            0.009855770505964756,
+            0.015757111832499504,
+            -0.03027183562517166,
+            -0.0025908327661454678,
+            -0.010158791206777096,
+            -0.0004055278259329498,
+            0.005140000022947788,
+            -0.02227207086980343,
+            -0.008977008052170277,
+            -0.015726808458566666,
+            -0.0010435298318043351,
+            0.006166485138237476,
+            -0.0007409819518215954,
+            -0.05805889517068863,
+            -0.013969285413622856,
+            0.018635815009474754,
+            0.0026400736533105373,
+            0.008908828720450401,
+            0.015938924625515938,
+            -0.0111739132553339,
+            0.0031533162109553814,
+            -0.026756787672638893,
+            0.023893235251307487,
+            -0.009492144919931889,
+            0.009355784393846989,
+            -0.013742019422352314,
+            0.012514783069491386,
+            0.014014738611876965,
+            0.004795313347131014,
+            0.0010681503918021917,
+            -0.0054051438346505165,
+            0.01601468026638031,
+            0.028377952054142952,
+            -0.023756874725222588,
+            0.011454207822680473,
+            -0.0002514130319468677,
+            -0.0032158142421394587,
+            -0.029196109622716904,
+            0.00509454682469368,
+            0.003931702114641666,
+            -0.005109698045998812,
+            0.016651025041937828,
+            -0.036544378846883774,
+            0.01662072166800499,
+            0.02177208475768566,
+            0.032544493675231934,
+            0.014832896180450916,
+            -0.017287369817495346,
+            -0.011658747680485249,
+            -0.012385998852550983,
+            -0.002624922664836049,
+            0.022181164473295212,
+            0.012120855040848255,
+            -0.01815097965300083,
+            -0.024287162348628044,
+            0.011249667964875698,
+            0.0005283934879116714,
+            -0.01203752402216196,
+            0.010795135982334614,
+            -0.013166278600692749,
+            0.01349202636629343,
+            0.012711746618151665,
+            0.00866641104221344,
+            0.00888610165566206,
+            0.022862963378429413,
+            -0.011878437362611294,
+            -0.017711598426103592,
+            0.00656041270121932,
+            -0.008916404098272324,
+            -0.0034089903347194195,
+            -0.012598113156855106,
+            0.010795135982334614,
+            0.014878349378705025,
+            0.06490717828273773,
+            -0.013143551535904408,
+            0.0009526234352961183,
+            -0.020438792183995247,
+            -0.031938452273607254,
+            -0.0017944546416401863,
+            -0.020908474922180176,
+            0.0111739132553339,
+            0.008370965719223022,
+            0.01598437689244747,
+            0.012022373266518116,
+            -0.0021912232041358948,
+            0.00795431062579155,
+            0.019317612051963806,
+            -0.010423935018479824,
+            7.255941454786807e-05,
+            -0.006503595970571041,
+            0.0013683309080079198,
+            0.0035851215943694115,
+            -0.011719351634383202,
+            -0.0026949963066726923,
+            0.035695917904376984,
+            0.01068150345236063,
+            0.011651171371340752,
+            -0.03445352986454964,
+            0.0020435003098100424,
+            0.020150920376181602,
+            0.007602048572152853,
+            -0.008302785456180573,
+            -0.01563590206205845,
+            0.01706010289490223,
+            -0.015317730605602264,
+            -0.0022499337792396545,
+            0.002888172399252653,
+            -0.010151215828955173,
+            0.04581683129072189,
+            -0.019787294790148735,
+            -0.0011742077767848969,
+            0.031847547739744186,
+            0.0030207443051040173,
+            -0.0040983641520142555,
+            0.005408931523561478,
+            -0.003473382443189621,
+            -0.0069013116881251335,
+            -0.02139330841600895,
+            0.0031381649896502495,
+            0.00667025800794363,
+            -0.019287310540676117,
+            -0.0072005451656877995,
+            -0.007719469256699085,
+            0.006897523999214172,
+            0.023105379194021225,
+            -0.009492144919931889,
+            -0.00016145357221830636,
+            -0.008575504645705223,
+            0.009848194196820259,
+            -0.0011912527261301875,
+            -0.044301725924015045,
+            -0.006147546228021383,
+            0.021181194111704826,
+            0.000973456131760031,
+            0.0005757405888289213,
+            -0.007450537756085396,
+            0.005007428117096424,
+            -0.017423728480935097,
+            -0.008454295806586742,
+            0.02369627170264721,
+            0.01736312545835972,
+            -0.009113367646932602,
+            0.010863316245377064,
+            -0.0035680767614394426,
+            0.00433320552110672,
+            -0.020014561712741852,
+            -0.01095422264188528,
+            0.003462019143626094,
+            0.015272277407348156,
+            -0.0009819786064326763,
+            -0.029029447585344315,
+            -0.0012272365856915712,
+            -0.00024194362049456686,
+            0.02918095886707306,
+            0.01486319862306118,
+            -0.008961857296526432,
+            -0.021211495622992516,
+            -0.013242034241557121,
+            -0.001237652963027358,
+            -0.015726808458566666,
+            -0.009620928205549717,
+            0.011855711229145527,
+            0.0009137988090515137,
+            0.008992158807814121,
+            -0.018135828897356987,
+            -0.027347678318619728,
+            -0.016166189685463905,
+            0.0018465365283191204,
+            -0.012870832346379757,
+            0.003340810537338257,
+            0.015514694154262543,
+            -0.03430201858282089,
+            -0.015060162171721458,
+            -0.03805948421359062,
+            0.008280059322714806,
+            -0.0029658216517418623,
+            -0.02024182677268982,
+            -0.023605365306138992,
+            0.009484568610787392,
+            0.019181253388524055,
+            -0.0015548784285783768,
+            0.019847899675369263,
+            -0.000889651826582849,
+            0.020060013979673386,
+            -0.0073141781613230705,
+            0.0031703610438853502,
+            -0.013007191941142082,
+            0.0030245319940149784,
+            0.008333087898790836,
+            0.02890823967754841,
+            -0.023075077682733536,
+            -0.006503595970571041,
+            -0.004685468040406704,
+            0.004746072459965944,
+            0.012658718042075634,
+            0.014613205567002296,
+            0.020741812884807587,
+            -0.008272483013570309,
+            -0.0027612822595983744,
+            0.012484480626881123,
+            -0.003799130441620946,
+            0.007295239716768265,
+            0.002390081062912941,
+            0.0019677451346069574,
+            0.004162756260484457,
+            0.008477022871375084,
+            0.0053710537031292915,
+            0.027650700882077217,
+            -0.0028275682125240564,
+            0.0031703610438853502,
+            0.024620486423373222,
+            0.02322658896446228,
+            -0.018878230825066566,
+            -0.028074929490685463,
+            0.002024561632424593,
+            0.050695475190877914,
+            0.007245998363941908,
+            -0.006810405291616917,
+            -0.03927157074213028,
+            -0.0031760428100824356,
+            0.028635520488023758,
+            0.023650817573070526,
+            0.01584801822900772,
+            -0.010567869991064072,
+            -0.012098127976059914,
+            0.02772645466029644,
+            0.013295062817633152,
+            0.02616589516401291,
+            -0.025302285328507423,
+            0.010704229585826397,
+            -0.03869583085179329,
+            0.0449683703482151,
+            0.004439263138920069,
+            -0.014476845972239971,
+            0.0011884119594469666,
+            0.010386057198047638,
+            -0.004537744913250208,
+            0.006431628484278917,
+            -5.8059369621332735e-05,
+            -0.0008086882880888879,
+            0.011923890560865402,
+            0.012007221579551697,
+            0.004265025723725557,
+            -0.02399929240345955,
+            0.007359631359577179,
+            0.004272601567208767,
+            0.03005971945822239,
+            0.005431658122688532,
+            -0.0013863227795809507,
+            0.012385998852550983,
+            -0.005810434930026531,
+            0.008855799213051796,
+            -0.010719381272792816,
+            0.020469093695282936,
+            -0.02918095886707306,
+            1.0986003871948924e-05,
+            0.02051454596221447,
+            -0.0015785519499331713,
+            0.0038199631962925196,
+            -0.023741723969578743,
+            0.01186328660696745,
+            -0.052968136966228485,
+            0.021166043356060982,
+            -0.018741872161626816,
+            0.021741783246397972,
+            0.019590331241488457,
+            -0.009734561666846275,
+            -0.012052674777805805,
+            0.03545350208878517,
+            0.0026760573964565992,
+            -0.014264730736613274,
+            -0.010643625631928444,
+            -0.017999470233917236,
+            -0.02830219641327858,
+            0.041968461126089096,
+            -0.0006117243901826441,
+            0.0028275682125240564,
+            0.0024222771171480417,
+            -0.010666352696716785,
+            -0.008923979476094246,
+            0.001954487757757306,
+            0.020499395206570625,
+            -0.03369597718119621,
+            -0.011128460057079792,
+            -0.008689138107001781,
+            -0.02566591091454029,
+            -0.00020300064352340996,
+            -0.011961768381297588,
+            0.026211349293589592,
+            0.01240872498601675,
+            -0.03696860745549202,
+            0.003780191531404853,
+            0.023650817573070526,
+            0.001603172393515706,
+            0.018120678141713142,
+            0.002130619017407298,
+            0.0010795136913657188,
+            0.009742137044668198,
+            -0.010476963594555855,
+            0.026559824123978615,
+            -0.009204274043440819,
+            0.008658835664391518,
+            -0.001950700068846345,
+            -0.017287369817495346,
+            0.017969166859984398,
+            0.004083212930709124,
+            0.024120500311255455,
+            0.011954193003475666,
+            0.011605718173086643,
+            0.01112088467925787,
+            -0.019878201186656952,
+            0.020090317353606224,
+            -0.0018891488434746861,
+            0.009075489826500416,
+            0.024726543575525284,
+            -0.005863463506102562,
+            0.020832719281315804,
+            0.015257125720381737,
+            -0.00020229043730068952,
+            0.011295121163129807,
+            -0.020120618864893913,
+            -0.011779955588281155,
+            0.0085679292678833,
+            -0.009605777449905872,
+            -0.010325453244149685,
+            0.009901223704218864,
+            -0.010105762630701065,
+            -0.015052586793899536,
+            -0.0071247899904847145,
+            0.016635872423648834,
+            0.0028976418543606997,
+            -0.02610529027879238,
+            -1.7326075976598077e-05,
+            0.008901252411305904,
+            0.006166485138237476,
+            0.018741872161626816,
+            0.011317848227918148,
+            -0.006692984607070684,
+            0.010363331064581871,
+            0.011476934887468815,
+            0.0053104497492313385,
+            -0.005079395603388548,
+            -0.013431422412395477,
+            -0.0024241709616035223,
+            0.018090376630425453,
+            0.009810317307710648,
+            -0.01922670565545559,
+            0.004564259201288223,
+            0.0008877579239197075,
+            -0.01420412678271532,
+            -0.019347915425896645,
+            -0.0038483713287860155,
+            -0.004170331638306379,
+            0.0027915844693779945,
+            0.0509984977543354,
+            -0.0006216673064045608,
+            0.007893706671893597,
+            -0.025741664692759514,
+            -0.006299056578427553,
+            -0.00027556007262319326,
+            -0.0036097420379519463,
+            0.01272689737379551,
+            -0.01824188604950905,
+            -0.03724132478237152,
+            0.012711746618151665,
+            -0.0012584856012836099,
+            0.014923802576959133,
+            -0.00297907879576087,
+            0.014226853847503662,
+            -0.017347972840070724,
+            0.01916610263288021,
+            -0.0011117096291854978,
+            -0.016029831022024155,
+            0.0029374135192483664,
+            -0.011143610812723637,
+            -0.021317552775144577,
+            0.005916492082178593,
+            -0.006848283112049103,
+            -0.03896854817867279,
+            0.0029885482508689165,
+            -0.010423935018479824,
+            0.0011798894265666604,
+            -0.003912763670086861,
+            0.020166072994470596,
+            -0.0061513339169323444,
+            0.0008697660523466766,
+            0.02177208475768566,
+            0.0017026013229042292,
+            0.01601468026638031,
+            -0.014499573037028313,
+            0.010999675840139389,
+            0.012598113156855106,
+            0.02210540883243084,
+            -0.032180868089199066,
+            -0.01393140759319067,
+            0.014211702160537243,
+            0.0026116655208170414,
+            -0.005382417235523462,
+            0.020635755732655525,
+            -0.01294658798724413,
+            0.019650936126708984,
+            -0.0005577487172558904,
+            -0.011704200878739357,
+            -0.03524138405919075,
+            -0.016529815271496773,
+            -3.343888238305226e-05,
+            -0.008439145050942898,
+            0.025317436084151268,
+            0.01760554127395153,
+            -0.0108330138027668,
+            0.026468917727470398,
+            0.0037574649322777987,
+            0.014598054811358452,
+            -0.014257155358791351,
+            -0.005999823100864887,
+            -0.0003773563075810671,
+            -0.0119314668700099,
+            0.015264702029526234,
+            -0.03512017801403999,
+            0.0008342557121068239,
+            -0.00839369185268879,
+            0.005242269951850176,
+            -0.0254992488771677,
+            0.018257036805152893,
+            0.025817420333623886,
+            -0.0031419529113918543,
+            -0.005037730094045401,
+            -0.022802358493208885,
+            0.028968842700123787,
+            0.03049910068511963,
+            0.0024828813038766384,
+            -0.0013825349742546678,
+            -0.00352072948589921,
+            0.0033218718599528074,
+            0.0211205892264843,
+            -0.037998881191015244,
+            -0.009098216891288757,
+            -0.028696123510599136,
+            0.042332086712121964,
+            -0.008098246529698372,
+            -0.013431422412395477,
+            0.01862066239118576,
+            -0.0024715180043131113,
+            0.02355991117656231,
+            -0.004689255729317665,
+            -0.004855917301028967,
+            0.03148392215371132,
+            0.011514811776578426,
+            0.02772645466029644,
+            -0.04015033319592476,
+            0.014704111963510513,
+            -0.006867222022265196,
+            0.004867280833423138,
+            -0.013484450988471508,
+            0.005302873905748129,
+            0.02580226957798004,
+            -0.018060073256492615,
+            0.0068066176027059555,
+            0.030665762722492218,
+            0.00748084019869566,
+            0.002448791405186057,
+            0.0165904201567173,
+            0.003890036838129163,
+            0.010855740867555141,
+            0.000841831264551729,
+            0.0052574207074940205,
+            0.022423582151532173,
+            0.014454119838774204,
+            -0.016029831022024155,
+            -0.015120766125619411,
+            -0.00044790346873924136,
+            -0.004450626205652952,
+            0.005261208396404982,
+            0.013242034241557121,
+            0.0061172437854111195,
+            -0.015681356191635132,
+            0.01052999310195446,
+            0.014158673584461212,
+            0.013128400780260563,
+            -0.0039847311563789845,
+            -0.03412020578980446,
+            -0.02443867363035679,
+            -0.021438762545585632,
+            0.003104075090959668,
+            -0.0008281006012111902,
+            -0.0021211495622992516,
+            0.0017954016802832484,
+            0.017347972840070724,
+            -0.0008352026343345642,
+            0.00379155483096838,
+            0.01686313934624195,
+            -0.01415109820663929,
+            -0.022090258076786995,
+            -0.0035964848939329386,
+            -0.027529491111636162,
+            0.0053710537031292915,
+            0.001844642567448318,
+            -0.00965123064815998,
+            0.008530051447451115,
+            -0.0046059247106313705,
+            -0.007223271764814854,
+            -0.012196609750390053,
+            -0.02007516659796238,
+            0.005780132487416267,
+            0.016878290101885796,
+            0.005730891600251198,
+            -0.009272454306483269,
+            -0.003939277958124876,
+            0.011575416661798954,
+            0.017484333366155624,
+            0.017317671328783035,
+            -0.009295180439949036,
+            0.009628504514694214,
+            0.0004022135108243674,
+            0.004670316819101572,
+            -0.01834794320166111,
+            0.0045869858004152775,
+            -0.0027953721582889557,
+            -0.008658835664391518,
+            0.01105270441621542,
+            -0.00031201730598695576,
+            0.01628739945590496,
+            -0.013090522959828377,
+            0.005624833982437849,
+            0.010333028621971607,
+            -0.020590301603078842,
+            -0.0067308624275028706,
+            -0.01692374423146248,
+            0.026484068483114243,
+            -0.00504530593752861,
+            0.00045405857963487506,
+            -0.02978700026869774,
+            0.0033616432920098305,
+            0.000848933297675103,
+            -0.000845619011670351,
+            -0.012764775194227695,
+            0.010749682784080505,
+            0.03242328763008118,
+            0.0005179771687835455,
+            0.0216660276055336,
+            -0.0036154238041490316,
+            0.0007480839849449694,
+            -0.00785582885146141,
+            0.004977126140147448,
+            -0.004916521720588207,
+            -0.009901223704218864,
+            -0.010863316245377064,
+            0.021726632490754128,
+            -0.0033029329497367144,
+            0.0003356908564455807,
+            0.011332998983561993,
+            -0.012067826464772224,
+            0.0014620780711993575,
+            -0.0047687990590929985,
+            0.00971941091120243,
+            0.0061702728271484375,
+            0.009113367646932602,
+            -0.01176480483263731,
+            0.05451354384422302,
+            -0.0106284748762846,
+            0.005537715740501881,
+            -0.00439380994066596,
+            -0.022226616740226746,
+            -0.017317671328783035,
+            0.012711746618151665,
+            -0.017544938251376152,
+            0.03375658020377159,
+            0.024150803685188293,
+            0.0006467612111009657,
+            0.010393633507192135,
+            -0.006958128418773413,
+            0.023923536762595177,
+            -0.002602196065708995,
+            0.0010795136913657188,
+            -0.004855917301028967,
+            0.015802564099431038,
+            0.04511988162994385,
+            0.017393426969647408,
+            -0.001152428099885583,
+            0.00829521007835865,
+            0.020575150847434998,
+            0.00814369972795248,
+            -0.0020075165666639805,
+            0.014219277538359165,
+            -0.01024212222546339,
+            0.00900731049478054,
+            -0.004458202049136162,
+            0.027438584715127945,
+            0.00814369972795248,
+            -0.006310420110821724,
+            -0.0036836036015301943,
+            -0.0008209985098801553,
+            0.014173824340105057,
+            0.006950552575290203,
+            0.01963578537106514,
+            -0.00461350055411458,
+            -0.0012575386790558696,
+            0.013870803639292717,
+            0.014491996727883816,
+            -0.00552635220810771,
+            0.001724380999803543,
+            -0.01435563713312149,
+            -0.010204244405031204,
+            0.013575357384979725,
+            0.006086941808462143,
+            0.006310420110821724,
+            -0.014514723792672157,
+            -0.00980274099856615,
+            -0.004670316819101572,
+            0.0015435151290148497,
+            0.015052586793899536,
+            0.011060279794037342,
+            -0.004814252257347107,
+            0.006052851676940918,
+            0.013825350441038609,
+            -0.014014738611876965,
+            0.006147546228021383,
+            0.007208121009171009,
+            0.008855799213051796,
+            0.004787737503647804,
+            -0.010817863047122955,
+            -0.00428396463394165,
+            -0.0063937511295080185,
+            0.0012670081341639161,
+            0.016832837834954262,
+            0.013181429356336594,
+            0.0049581872299313545,
+            0.027529491111636162,
+            -0.00839369185268879,
+            -0.012302667833864689,
+            0.02339324913918972,
+            0.011257244274020195,
+            0.0031703610438853502,
+            0.014651083387434483,
+            -0.012635990977287292,
+            -0.012446602806448936,
+            0.006655106786638498,
+            0.004848341923207045,
+            0.006049063988029957,
+            0.009151245467364788,
+            0.00450365524739027,
+            0.007333117071539164,
+            -0.007098275702446699,
+            -0.007427811156958342,
+            0.0014810169814154506,
+            0.01642375811934471,
+            0.020393338054418564,
+            -0.024968961253762245,
+            -0.0018806264270097017,
+            0.0009052763343788683,
+            0.010878467001020908,
+            -0.003316190093755722,
+            0.012310243211686611,
+            5.65205882594455e-05,
+            -0.0074240234680473804,
+            0.010158791206777096,
+            -0.02119634486734867,
+            -0.012681444175541401,
+            0.0012717428617179394,
+            0.013037494383752346,
+            -0.019590331241488457,
+            0.018226735293865204,
+            -0.008893677033483982,
+            -0.013348091393709183,
+            0.013893529772758484,
+            0.020014561712741852,
+            0.004090788308531046,
+            -0.007738408166915178,
+            -0.021862991154193878,
+            -0.01127239502966404,
+            0.016060132533311844,
+            0.020272130146622658,
+            0.000947888707742095,
+            -0.0006472347304224968,
+            0.010173942893743515,
+            -0.013113250024616718,
+            -0.023605365306138992,
+            -0.005230906419456005,
+            0.010386057198047638,
+            0.01746918261051178,
+            0.018832778558135033,
+            -0.006685409229248762,
+            0.02443867363035679,
+            -0.0059127043932676315,
+            -0.015499543398618698,
+            2.86597842205083e-05,
+            -0.02802947722375393,
+            -0.017423728480935097,
+            0.00013778003631159663,
+            0.00246773031540215,
+            0.008598231710493565,
+            0.0008555619278922677,
+            0.01943882182240486,
+            0.000904329412151128,
+            0.017590390518307686,
+            -0.000659071491099894,
+            -0.002009410411119461,
+            -0.0013995799235999584,
+            -0.0015340456739068031,
+            0.02254479005932808,
+            -0.012166308239102364,
+            -0.01176480483263731,
+            0.022302372381091118,
+            0.006852070800960064,
+            0.014719262719154358,
+            -0.027635548263788223,
+            0.013757170177996159,
+            -0.015499543398618698,
+            0.017423728480935097,
+            -0.03408990427851677,
+            0.006420265417546034,
+            0.01489350013434887,
+            -0.010348179377615452,
+            -0.020499395206570625,
+            -0.029650641605257988,
+            -0.024150803685188293,
+            -0.0038710979279130697,
+            -0.010855740867555141,
+            -0.016847988590598106,
+            0.006121031939983368,
+            -0.005802859086543322,
+            0.018499454483389854,
+            -0.007079336792230606,
+            0.033877789974212646,
+            0.006969491485506296,
+            -0.004056698642671108,
+            0.003835114184767008,
+            0.022469034418463707,
+            -0.006102093029767275,
+            0.02109028771519661,
+            0.004408961161971092,
+            -0.001467759720981121,
+            0.013681414537131786,
+            -0.006552837323397398,
+            -0.002361672930419445,
+            0.005821797996759415,
+            -0.010484539903700352,
+            0.022014502435922623,
+            0.005924067925661802,
+            -0.007708106189966202,
+            -0.0006775368237867951,
+            -0.011454207822680473,
+            0.014726839028298855,
+            0.006530110724270344,
+            0.0022480397019535303,
+            -0.011295121163129807,
+            0.011249667964875698,
+            -0.004859705455601215,
+            0.0040642740204930305,
+            0.009484568610787392,
+            -0.01774190180003643,
+            -0.028862785547971725,
+            0.011416330002248287,
+            0.019120648503303528,
+            0.006382387597113848,
+            -0.0055755930952727795,
+            -0.0025681061670184135,
+            0.00839369185268879,
+            0.010651201009750366,
+            0.022150862962007523,
+            -0.012984465807676315,
+            -8.889415767043829e-05,
+            0.013204156421124935,
+            -0.007170243188738823,
+            -0.01056029461324215,
+            0.00487485621124506,
+            0.023438703268766403,
+            0.0068823727779090405,
+            -0.04327145218849182,
+            -0.018469152972102165,
+            0.01909034699201584,
+            0.007136153522878885,
+            -0.04154422879219055,
+            0.0026381798088550568,
+            -0.003689285134896636,
+            -0.023787178099155426,
+            0.016105586662888527,
+            -0.0011817833874374628,
+            -0.01828734017908573,
+            0.028756728395819664,
+            0.02830219641327858,
+            0.01834794320166111,
+            2.6987840101355687e-05,
+            -0.013332940638065338,
+            -0.0318172425031662,
+            -0.023590214550495148,
+            0.008280059322714806,
+            -0.02660527639091015,
+            -0.0072725131176412106,
+            0.011242092587053776,
+            -0.017454031854867935,
+            0.02972639724612236,
+            -0.008507325313985348,
+            0.026938600465655327,
+            0.011446632444858551,
+            0.001691238023340702,
+            -0.013446573168039322,
+            -0.018802475184202194,
+            -0.01350717805325985,
+            -0.008196728304028511,
+            0.014090493321418762,
+            0.017923714593052864,
+            0.011007251217961311,
+            0.011332998983561993,
+            0.02251448854804039,
+            -0.012158732861280441,
+            0.007969462312757969,
+            -2.888172457460314e-05,
+            -0.008961857296526432,
+            0.0026343921199440956,
+            -0.018393397331237793,
+            0.00868156272917986,
+            0.0062308767810463905,
+            0.005458172410726547,
+            -0.007208121009171009,
+            0.010742107406258583,
+            0.02712041325867176,
+            -0.018847929313778877,
+            0.011401179246604443,
+            -0.031362712383270264,
+            -0.014787442982196808,
+            -0.01056029461324215,
+            0.026862844824790955,
+            0.007757347077131271,
+            -0.015726808458566666,
+            -0.007514929864555597,
+            -0.01545409020036459,
+            0.012885984033346176,
+            0.0032442226074635983,
+            0.005764981731772423,
+            0.015817714855074883,
+            0.021893294528126717,
+            0.01746918261051178,
+            -0.00795431062579155,
+            0.006852070800960064,
+            0.004674104508012533,
+            0.028090080246329308,
+            -0.01777220331132412,
+            0.020802417770028114,
+            0.009954252280294895,
+            -0.003935490269213915,
+            0.005685438401997089,
+            -0.014257155358791351,
+            0.008530051447451115,
+            -0.007325541693717241,
+            0.030514251440763474,
+            0.0033067206386476755,
+            0.014075342565774918,
+            -0.02062060497701168,
+            0.006575563922524452,
+            -0.03172633796930313,
+            0.009022461250424385,
+            0.013704141601920128,
+            0.012976890429854393,
+            -0.007677804213017225,
+            -0.024181105196475983,
+            -0.0119314668700099,
+            0.0057990713976323605,
+            -0.01279507763683796,
+            -0.01046181283891201,
+            -0.0034828518982976675,
+            0.012446602806448936,
+            0.018847929313778877,
+            -0.01848430372774601,
+            -0.019923655316233635,
+            -0.009583051316440105,
+            0.02898399345576763,
+            -0.015787413343787193,
+            -0.001471547526307404,
+            0.02413565292954445,
+            0.021241798996925354,
+            0.015022284351289272,
+            0.002155239460989833,
+            0.020090317353606224,
+            -0.0007111532613635063,
+            -0.0237265732139349,
+            -0.02813553437590599,
+            0.007302815094590187,
+            -0.006912675220519304,
+            0.01943882182240486,
+            -0.0035548193845897913,
+            -0.005658924113959074,
+            -0.0034033088013529778,
+            -0.007503566797822714,
+            -0.0165904201567173,
+            -0.01787826046347618,
+            0.0021647089160978794,
+            -0.008454295806586742,
+            0.005863463506102562,
+            -0.0002795845502987504,
+            -0.013105674646794796,
+            0.013401119969785213,
+            -0.014075342565774918,
+            0.005772557109594345,
+            -0.002653330797329545,
+            0.001962063368409872,
+            0.0035926972050219774,
+            0.009113367646932602,
+            -0.007336904760450125,
+            -0.06381630152463913,
+            -0.013105674646794796,
+            -0.003704436356201768,
+            0.006071790587157011,
+            -0.00019578021601773798,
+            -0.016635872423648834,
+            -0.013196581043303013,
+            0.01706010289490223,
+            0.008863375522196293,
+            -0.013870803639292717,
+            0.016954045742750168,
+            0.009961827658116817,
+            0.003622999181970954,
+            -0.024044746533036232,
+            -0.0008148433989845216,
+            0.008757317438721657,
+            0.032756611704826355,
+            -0.008643684908747673,
+            -0.0003839849086944014,
+            0.0036684523802250624,
+            -0.0033332351595163345,
+            -0.010052734054625034,
+            -0.014582903124392033,
+            -0.011848135851323605,
+            0.011908739805221558,
+            0.01790856383740902,
+            -0.014264730736613274,
+            -0.02007516659796238,
+            -0.0007106798002496362,
+            0.012931437231600285,
+            -0.026590125635266304,
+            0.0007244104635901749,
+            -0.007787649054080248,
+            -0.01052999310195446,
+            0.018302490934729576,
+            -0.019014591351151466,
+            0.005249845329672098,
+            0.00048294031876139343,
+            0.008507325313985348,
+            -0.023544760420918465,
+            -0.024893205612897873,
+            0.0035131541080772877,
+            -0.023847781121730804,
+            0.03612014651298523,
+            -0.021590271964669228,
+            -0.012082977220416069,
+            0.034483831375837326,
+            -0.0037423139438033104,
+            0.01702980138361454,
+            0.0023995505180209875,
+            0.02739313244819641,
+            -0.011439057067036629,
+            -0.00617784820497036,
+            0.012499631382524967,
+            0.009575475007295609,
+            0.0032593735959380865,
+            0.03775646165013313,
+            -0.01257538702338934,
+            -0.013901105150580406,
+            -0.005556654185056686,
+            -0.012954163365066051,
+            0.0006775368237867951,
+            0.014431392773985863,
+            0.028393102809786797,
+            0.004257450345903635,
+            -0.00628769351169467,
+            -0.006143758539110422,
+            -0.0014649189542979002,
+            -0.004242299124598503,
+            -0.00947699323296547,
+            -0.00715130427852273,
+            0.017287369817495346,
+            -0.025241680443286896,
+            0.016711628064513206,
+            -0.006420265417546034,
+            0.021135739982128143,
+            -0.011757229454815388,
+            0.007007369305938482,
+            -0.008961857296526432,
+            0.003492321353405714,
+            0.0023635667748749256,
+            -0.005109698045998812,
+            0.009287605062127113,
+            -0.014772292226552963,
+            0.008454295806586742,
+            0.013742019422352314,
+            0.0011183382011950016,
+            -0.010590597055852413,
+            -0.019272159785032272,
+            0.0022972808219492435,
+            -0.000710206339135766,
+            0.01181783340871334,
+            -0.0047498601488769054,
+            -0.000981031684204936,
+            0.0002921315317507833,
+            -0.014476845972239971,
+            -0.009575475007295609,
+            0.0006050958181731403,
+            0.005302873905748129,
+            -0.008961857296526432,
+            0.010378481820225716,
+            -0.005693014245480299,
+            -0.00034160923678427935,
+            0.02627195231616497,
+            0.006526322569698095,
+            -0.007632351014763117,
+            -0.014098069630563259,
+            0.01563590206205845,
+            0.015370759181678295,
+            -0.007454325910657644,
+            0.01933276280760765,
+            -0.016196493059396744,
+            -0.0008366230758838356,
+            0.013916256837546825,
+            -0.009219424799084663,
+            0.0017698341980576515,
+            -0.008196728304028511,
+            0.03554440662264824,
+            -0.0008787619881331921,
+            -0.0035415622405707836,
+            -0.0006401326390914619,
+            0.012976890429854393,
+            -0.0026779514737427235,
+            0.001182730309665203,
+            0.01132542360574007,
+            0.005321812815964222,
+            0.0185146052390337,
+            0.00328020635060966,
+            -0.03378688171505928,
+            -0.007780073676258326,
+            -0.0065490491688251495,
+            -0.008090670220553875,
+            0.009022461250424385,
+            0.025196226313710213,
+            0.01990850456058979,
+            0.006408901885151863,
+            0.001881573349237442,
+            0.018423698842525482,
+            0.015757111832499504,
+            -0.01489350013434887,
+            0.016363153234124184,
+            0.02802947722375393,
+            -0.010030006989836693,
+            0.022741753607988358,
+            -0.005253633018583059,
+            -0.008893677033483982,
+            0.008946705609560013,
+            -0.0015378334792330861,
+            -0.017393426969647408,
+            -0.015681356191635132,
+            0.005022579338401556,
+            -0.008999735116958618,
+            -0.0022897052112966776,
+            -0.018605511635541916,
+            0.004670316819101572,
+            0.017105557024478912,
+            -0.004965762607753277,
+            -3.5243989259470254e-05,
+            0.019484274089336395,
+            -0.006965703796595335,
+            -0.016908591613173485,
+            0.006325571332126856,
+            -0.015007133595645428,
+            0.007946735247969627,
+            -0.013666263781487942,
+            0.010492115281522274,
+            -0.006098304875195026,
+            0.018817627802491188,
+            0.005446809343993664,
+            -0.008090670220553875,
+            -0.0008958069374784827,
+            -0.018984289839863777,
+            0.02065090648829937,
+            -0.033089932054281235,
+            0.00839369185268879,
+            -0.026226500049233437,
+            -0.0009781909175217152,
+            0.0008849170990288258,
+            0.014605630189180374,
+            -0.001218714052811265,
+            0.019696388393640518,
+            -0.002751812804490328,
+            -0.020090317353606224,
+            -0.007086912170052528,
+            -0.0010634156642481685,
+            -0.01618134044110775,
+            -0.030923331156373024,
+            -0.004272601567208767,
+            0.0036267871037125587,
+            -0.004711982328444719,
+            0.0021135739516466856,
+            0.017984317615628242,
+            -0.005613470915704966,
+            0.009173971600830555,
+            -0.02690829709172249,
+            -0.007416448090225458,
+            -0.00355103169567883,
+            0.0010776197304949164,
+            -0.011242092587053776,
+            0.0010596278589218855,
+            0.011226941831409931,
+            -0.019044892862439156,
+            0.013461724855005741,
+            -0.021378157660365105,
+            -0.004371083341538906,
+            -0.008507325313985348,
+            -0.00135412672534585,
+            -0.013446573168039322,
+            -0.028120383620262146,
+            0.025029566138982773,
+            -0.00021495578403118998,
+            -0.003528305096551776,
+            -0.011999646201729774,
+            0.023181134834885597,
+            0.002914686920121312,
+            0.027817361056804657,
+            -0.03157482668757439,
+            0.0013427634257823229,
+            0.011552689597010612,
+            0.0025984083767980337,
+            -0.017287369817495346,
+            0.004746072459965944,
+            -0.005234694108366966,
+            0.02210540883243084,
+            -0.0011694730492308736,
+            -0.0005340751959010959,
+            -0.008234606124460697,
+            0.030999086797237396,
+            0.009136094711720943,
+            0.0012944694608449936,
+            -0.013226882554590702,
+            -0.008060368709266186,
+            -0.002759388415142894,
+            -0.0011069749016314745,
+            0.012855681590735912,
+            0.01290113478899002,
+            -0.012105703353881836,
+            0.0013607552973553538,
+            -0.006878585088998079,
+            0.0011694730492308736,
+            -0.002935519441962242,
+            -0.0004966709529981017,
+            -0.01149208564311266,
+            0.003257479751482606,
+            0.016272246837615967,
+            -0.00780280027538538,
+            -0.006480869371443987,
+            0.0007750718505121768,
+            0.009181547909975052,
+            0.012461753562092781,
+            -0.012742049060761929,
+            -0.018226735293865204,
+            -0.011908739805221558,
+            0.016029831022024155,
+            0.010522416792809963,
+            -0.008204303681850433,
+            -0.013234458863735199,
+            -0.02583257108926773,
+            0.01169662456959486,
+            -0.03293842449784279,
+            -0.012242063879966736,
+            0.005821797996759415,
+            0.028559764847159386,
+            0.019999410957098007,
+            0.012226912193000317,
+            0.001332347048446536,
+            0.002950670663267374,
+            0.00023342114582192153,
+            -0.01916610263288021,
+            0.014408666640520096,
+            -0.011143610812723637,
+            -0.018166130408644676,
+            0.0075906855054199696,
+            0.011939042247831821,
+            0.03148392215371132,
+            0.006942977197468281,
+            -0.0010160685051232576,
+            0.01686313934624195,
+            -0.001218714052811265,
+            -0.013582932762801647,
+            0.01420412678271532,
+            -0.024256860837340355,
+            0.016438908874988556,
+            0.02860521711409092,
+            -0.004780162125825882,
+            0.0008531945641152561,
+            -0.020938776433467865,
+            0.00482561532407999,
+            -0.011529963463544846,
+            -0.002177966060116887,
+            -0.003287781961262226,
+            -0.03460504114627838,
+            0.007946735247969627,
+            -0.006825556512922049,
+            0.01046181283891201,
+            -0.00814369972795248,
+            0.006655106786638498,
+            -0.012431452050805092,
+            0.004253662656992674,
+            0.01513591781258583,
+            0.016105586662888527,
+            0.007060397882014513,
+            -0.025211378931999207,
+            -0.009196698665618896,
+            0.008105821907520294,
+            -0.01349202636629343,
+            -0.02295386977493763,
+            -0.006852070800960064,
+            0.009840618818998337,
+            -0.01061332318931818,
+            -0.00488243205472827,
+            -0.028180986642837524,
+            -0.013681414537131786,
+            -0.0014327229000627995,
+            -0.017817657440900803,
+            -0.005556654185056686,
+            -0.003100287402048707,
+            0.01571165770292282,
+            -0.0049733384512364864,
+            -0.006185423582792282,
+            0.004742284305393696,
+            -0.023878084495663643,
+            0.0037423139438033104,
+            -0.008280059322714806,
+            0.0008266801596619189,
+            -0.0032517979852855206,
+            -0.0014762822538614273,
+            -0.007969462312757969,
+            -0.014332910999655724,
+            0.006893736310303211,
+            -0.016469212248921394,
+            0.002954458352178335,
+            -0.015173795633018017,
+            -0.016499513760209084,
+            -0.00682934420183301,
+            0.00775355938822031,
+            0.021575121209025383,
+            0.006541473791003227,
+            0.021484214812517166,
+            0.005469535943120718,
+            0.028680972754955292,
+            0.0003953482082579285,
+            0.003041577059775591,
+            -8.958956641436089e-06,
+            0.003232859307900071,
+            0.019347915425896645,
+            0.006571775767952204,
+            0.018984289839863777,
+            -0.008742166683077812,
+            0.010514841414988041,
+            -0.010355755686759949,
+            0.010014856234192848,
+            0.006333146709948778,
+            -0.006473293993622065,
+            -0.003799130441620946,
+            -0.010348179377615452,
+            -0.0031419529113918543,
+            -0.016544966027140617,
+            0.0027366618160158396,
+            0.00693161366507411,
+            0.0071588801220059395,
+            -0.011582992039620876,
+            -0.005230906419456005,
+            -0.014704111963510513,
+            -0.0064354161731898785,
+            -0.01831764169037342,
+            0.014385939575731754,
+            0.029226411134004593,
+            -0.016332851722836494,
+            -0.01977214403450489,
+            -0.007772498298436403,
+            0.0007296186522580683,
+            -0.006636167876422405,
+            -0.01240872498601675,
+            -0.009151245467364788,
+            -0.035059571266174316,
+            -0.017666146159172058,
+            0.007586897816509008,
+            0.02484775334596634,
+            0.01387837901711464,
+            -0.013628385961055756,
+            -0.0006983695784583688,
+            -0.016984347254037857,
+            -0.010787560604512691,
+            -0.01535560842603445,
+            -0.02054484933614731,
+            -0.02498411200940609,
+            0.0077952248975634575,
+            -0.004033972043544054,
+            0.014931377954781055,
+            0.008113397285342216,
+            0.0032631612848490477,
+            -0.004371083341538906,
+            -0.0024109138175845146,
+            -0.013196581043303013,
+            0.0012925754999741912,
+            -0.004787737503647804,
+            -0.0017291157273575664,
+            0.03554440662264824,
+            -0.008098246529698372,
+            -0.005871038883924484,
+            0.011060279794037342,
+            0.011128460057079792,
+            -0.013764745555818081,
+            0.0006642796797677875,
+            -0.011249667964875698,
+            0.021847840398550034,
+            -0.021620575338602066,
+            0.019257009029388428,
+            0.0014507147716358304,
+            -0.010840589180588722,
+            0.028999146074056625,
+            0.01892368495464325,
+            0.00743159931153059,
+            0.006579351611435413,
+            0.01672677882015705,
+            0.0044316877610981464,
+            0.016196493059396744,
+            -0.0026552248746156693,
+            -0.004632438998669386,
+            0.009295180439949036,
+            0.015681356191635132,
+            -0.000702630786690861,
+            -0.006942977197468281,
+            0.016438908874988556,
+            0.004056698642671108,
+            0.0075414446182549,
+            -0.01356020662933588,
+            -0.008961857296526432,
+            0.001507531269453466,
+            0.006257391534745693,
+            -0.005939218681305647,
+            -0.0026003022212535143,
+            0.005530139897018671,
+            0.003943065647035837,
+            -0.013302638195455074,
+            0.01689344085752964,
+            -0.016166189685463905,
+            0.008742166683077812,
+            -0.0037972365971654654,
+            0.0121360057964921,
+            -0.006799042224884033,
+            0.0013153020991012454,
+            -0.01818128302693367,
+            -0.003869204083457589,
+            0.0038597346283495426,
+            0.012173883616924286,
+            -0.01262841559946537,
+            0.025484098121523857,
+            0.008166425861418247,
+            0.006602078210562468,
+            0.0059467945247888565,
+            -0.012855681590735912,
+            0.027211319655179977,
+            0.005969521123915911,
+            0.009886072017252445,
+            -0.008893677033483982,
+            0.011136035434901714,
+            0.035362593829631805,
+            0.03060515783727169,
+            0.014135947450995445,
+            0.004469565115869045,
+            -0.018999440595507622,
+            0.015120766125619411,
+            0.0018427487229928374,
+            0.0023806116078048944,
+            0.00863610953092575,
+            -0.011348150670528412,
+            0.01305264513939619,
+            -0.009484568610787392,
+            0.015241974964737892,
+            -0.021272100508213043,
+            -0.011749654076993465,
+            -0.014408666640520096,
+            -0.021075136959552765,
+            -0.01430260855704546,
+            0.034180812537670135,
+            -0.002346521709114313,
+            -0.010393633507192135,
+            0.0045112306252121925,
+            0.008477022871375084,
+            -0.006079366430640221,
+            0.0053142374381423,
+            -0.008204303681850433,
+            -0.013749594800174236,
+            -0.01889338344335556,
+            0.006628592498600483,
+            -0.01550711877644062,
+            0.007651289459317923,
+            -0.003304826794192195,
+            0.0037688282318413258,
+            0.007014944683760405,
+            0.018105527386069298,
+            0.00530666159465909,
+            -0.00135412672534585,
+            0.0014033677289262414,
+            0.0016865032957866788,
+            -0.00931033119559288,
+            0.0108330138027668,
+            0.015408637002110481,
+            -0.0011921997647732496,
+            -0.011279970407485962,
+            -0.009136094711720943,
+            -0.042877525091171265,
+            0.0006264019757509232,
+            -0.013128400780260563,
+            0.0064354161731898785,
+            -0.011196639388799667,
+            0.0022234192583709955,
+            0.015863168984651566,
+            -0.0054051438346505165,
+            -0.0010918239131569862,
+            0.02193874679505825,
+            0.013014768250286579,
+            0.021075136959552765,
+            0.02065090648829937,
+            0.014188976027071476,
+            -0.015302578918635845,
+            -0.03469594568014145,
+            0.002221525413915515,
+            -0.010030006989836693,
+            0.013772320933640003,
+            0.004238511435687542,
+            0.016060132533311844,
+            -0.023893235251307487,
+            -0.011878437362611294,
+            -0.006662682630121708,
+            0.014431392773985863,
+            0.01491622719913721,
+            -0.0032271775417029858,
+            0.02690829709172249,
+            0.009280029684305191,
+            0.0020510759204626083,
+            -0.02281750924885273,
+            0.003312402404844761,
+            0.005321812815964222,
+            -0.008370965719223022,
+            -0.016681326553225517,
+            -0.0004552422615233809,
+            -0.0012168202083557844,
+            -0.0007817004225216806,
+            -0.021650876849889755,
+            0.018832778558135033,
+            -0.004018820822238922,
+            0.0037574649322777987,
+            0.013840501196682453,
+            0.01516621932387352,
+            0.017984317615628242,
+            0.008370965719223022,
+            0.001497115008533001,
+            -0.006605865899473429,
+            -0.0007973249885253608,
+            -0.0005828426801599562,
+            -0.007389933802187443,
+            -0.005598319694399834,
+            0.006382387597113848,
+            -0.01134057529270649,
+            0.008810346014797688,
+            0.000896753859706223,
+            0.006113456096500158,
+            0.010605747811496258,
+            -0.004836978856474161,
+            -0.011628445237874985,
+            0.014628356322646141,
+            -0.0010548931313678622,
+            -0.0055755930952727795,
+            -0.011060279794037342,
+            0.003624893259257078,
+            0.013575357384979725,
+            -0.003096499713137746,
+            -0.013499601744115353,
+            0.00910579226911068,
+            0.002009410411119461,
+            -0.027438584715127945,
+            0.015499543398618698,
+            0.010295150801539421,
+            -0.013370818458497524,
+            0.004980913829058409,
+            -0.003371112747117877,
+            -0.03027183562517166,
+            0.004791525658220053,
+            0.002660906407982111,
+            -0.010643625631928444,
+            0.028938541188836098,
+            -0.003821857040748,
+            0.029256712645292282,
+            -0.004980913829058409,
+            -0.006905099377036095,
+            -0.011355726048350334,
+            0.009416389279067516,
+            -0.018681267276406288,
+            -0.0037953427527099848,
+            -0.008727015927433968,
+            -0.00915882084518671,
+            -0.0009838725673034787,
+            -0.005185453221201897,
+            0.006984642706811428,
+            0.005121061112731695,
+            -0.012704171240329742,
+            0.006757376715540886,
+            -0.010257272981107235,
+            -0.00020856392802670598,
+            0.0006718551740050316,
+            -0.008408842608332634,
+            0.013885954394936562,
+            0.005996035411953926,
+            -0.008340663276612759,
+            -0.0005032995832152665,
+            0.027817361056804657,
+            0.05051366239786148,
+            -0.018863080069422722,
+            0.0009270560112781823,
+            0.01452229917049408,
+            -0.0028824908658862114,
+            -0.0021590273827314377,
+            -0.011976920068264008,
+            0.005821797996759415,
+            -0.019423669204115868,
+            0.013317788951098919,
+            0.006473293993622065,
+            -0.008833073079586029,
+            0.003202557098120451,
+            -0.016711628064513206,
+            0.002621134975925088,
+            -0.0018058179412037134,
+            -0.018681267276406288,
+            -0.015529844909906387,
+            -0.012060250155627728,
+            -0.024181105196475983,
+            0.0004235197266098112,
+            -0.0008546149474568665,
+            -0.013870803639292717,
+            0.006579351611435413,
+            0.008431569673120975,
+            0.032483890652656555,
+            0.0011912527261301875,
+            0.010128489695489407,
+            -2.1513333194889128e-05,
+            -0.0010416359873488545,
+            -0.005988460034132004,
+            -0.01523439958691597,
+            0.00015387803432531655,
+            -0.002342734020203352,
+            0.022590242326259613,
+            0.005549078807234764,
+            0.017560089007019997,
+            0.0017887729918584228,
+            -0.016514664515852928,
+            -0.00596573343500495,
+            -0.007408872712403536,
+            -0.018060073256492615,
+            0.0062649669125676155,
+            -0.005473323632031679,
+            0.009605777449905872,
+            -0.015863168984651566,
+            -0.02731737680733204,
+            0.00792400911450386,
+            0.012234487570822239,
+            0.028514310717582703,
+            -0.0035623949952423573,
+            0.001716805505566299,
+            -0.005533927585929632,
+            0.016408607363700867,
+            0.032180868089199066,
+            -0.0008550884085707366,
+            0.006950552575290203,
+            -0.019999410957098007,
+            0.0017404790269210935,
+            0.013704141601920128,
+            -0.008401267230510712,
+            -0.01821158453822136,
+            0.003939277958124876,
+            -0.020635755732655525,
+            0.004583198111504316,
+            -0.020393338054418564,
+            0.030408194288611412,
+            -0.0023522034753113985,
+            0.008105821907520294,
+            0.004310478921979666,
+            0.01760554127395153,
+            -0.0031646795105189085,
+            0.010545143857598305,
+            0.013916256837546825,
+            0.014908651821315289,
+            0.0009980766335502267,
+            -0.011189064010977745,
+            -0.03790797293186188,
+            0.001859793672338128,
+            0.00817400123924017,
+            0.018635815009474754,
+            -0.01980244740843773,
+            0.011681473813951015,
+            -0.01296931505203247,
+            -0.020363036543130875,
+            -0.01003758329898119,
+            0.011575416661798954,
+            -0.006139970384538174,
+            0.03378688171505928,
+            -0.005874827038496733,
+            -0.008204303681850433,
+            0.005136212334036827,
+            0.010454237461090088,
+            0.0066778333857655525,
+            0.014704111963510513,
+            0.0326354019343853,
+            0.010196669027209282,
+            0.014173824340105057,
+            -0.004859705455601215,
+            0.02399929240345955,
+            -0.003931702114641666,
+            0.003954428713768721,
+            0.00509454682469368,
+            -0.022832660004496574,
+            -0.0040983641520142555,
+            -0.003931702114641666,
+            -0.016711628064513206,
+            -0.007458113599568605,
+            0.0032366469968110323,
+            0.008802770636975765,
+            -0.014045040123164654,
+            -0.00649980828166008,
+            -0.01350717805325985,
+            0.01218903437256813,
+            0.002730980049818754,
+            0.018302490934729576,
+            -0.029802151024341583,
+            0.009825468063354492,
+            -0.017484333366155624,
+            -0.017044952139258385,
+            0.006511171814054251,
+            -0.0038332203403115273,
+            0.006855858489871025,
+            -0.018272189423441887,
+            -0.01318900566548109,
+            0.012817803770303726,
+            0.0007897494360804558,
+            0.013060221448540688,
+            -0.009825468063354492,
+            0.0009706153068691492,
+            0.006064215209335089,
+            -0.009833043441176414,
+            -0.010961798019707203,
+            -0.007977037690579891,
+            -0.009681533090770245,
+            -0.008848223835229874,
+            0.005621046293526888,
+            0.01656011864542961,
+            -0.013742019422352314,
+            -0.003685497445985675,
+            0.016135888174176216,
+            0.006416477728635073,
+            -0.008166425861418247,
+            0.004075637552887201,
+            -0.003793448908254504,
+            -9.948806837201118e-05,
+            0.0013740125577896833,
+            0.012439027428627014,
+            -0.0017632056260481477,
+            0.024665940552949905,
+            -0.02654467150568962,
+            0.007598260883241892,
+            -0.0054051438346505165,
+            0.005446809343993664,
+            0.018226735293865204,
+            -0.02366596832871437,
+            -0.013082947582006454,
+            0.006276329979300499,
+            -0.01966608688235283,
+            -0.008060368709266186,
+            0.0054051438346505165,
+            -0.00016405766655225307,
+            -0.019923655316233635,
+            -0.012923861853778362,
+            -0.034786853939294815,
+            -0.02309022843837738,
+            0.024665940552949905,
+            0.008401267230510712,
+            -0.0073672072030603886,
+            0.018060073256492615,
+            -0.018438849598169327,
+            0.0172722190618515,
+            -0.002225313102826476,
+            -0.006613441277295351,
+            0.003935490269213915,
+            0.007674016058444977,
+            0.025362888351082802,
+            0.002653330797329545,
+            0.008264907635748386,
+            -0.0042082094587385654,
+            -0.014931377954781055,
+            -0.006321783177554607,
+            0.01980244740843773,
+            0.02471139281988144,
+            0.018605511635541916,
+            -0.018575210124254227,
+            0.010742107406258583,
+            -0.011749654076993465,
+            0.01818128302693367,
+            -0.006537686102092266,
+            0.00039037675014697015,
+            0.01859036087989807,
+            0.007969462312757969,
+            0.025241680443286896,
+            0.019923655316233635,
+            0.006852070800960064,
+            0.014128371141850948,
+            0.011908739805221558,
+            0.001204509986564517,
+            -0.012832955457270145,
+            -0.01154511421918869,
+            -0.02840825356543064,
+            0.042847223579883575,
+            0.0025434857234358788,
+            0.008317936211824417,
+            -0.027635548263788223,
+            -0.00021389048197306693,
+            -0.0034942151978611946,
+            -0.018832778558135033,
+            0.005939218681305647,
+            0.03260510042309761,
+            0.03284751623868942,
+            0.021438762545585632,
+            -2.888172457460314e-05,
+            0.0012234487803652883,
+            0.0329081192612648,
+            -0.013673839159309864,
+            -0.00937851145863533,
+            -0.01034060399979353,
+            -0.018393397331237793,
+            -0.006852070800960064,
+            -0.0032423287630081177,
+            -0.01031030248850584,
+            -0.0061172437854111195,
+            -0.009272454306483269,
+            0.0238629337400198,
+            0.0001374249259242788,
+            -0.0022594030015170574,
+            0.006605865899473429,
+            -0.020878173410892487,
+            0.011537538841366768,
+            -0.02813553437590599,
+            -0.013431422412395477,
+            -0.008651260286569595,
+            -0.006950552575290203,
+            -0.018802475184202194,
+            0.008643684908747673,
+            -0.006942977197468281,
+            0.021438762545585632,
+            0.002914686920121312,
+            -0.023590214550495148,
+            0.007530081085860729,
+            0.015317730605602264,
+            -0.0003825644962489605,
+            0.014037464745342731,
+            -0.003890036838129163,
+            0.009060339070856571,
+            0.0045112306252121925,
+            0.011681473813951015,
+            0.016954045742750168,
+            0.010651201009750366,
+            0.009030036628246307,
+            0.003437398700043559,
+            -0.001167579204775393,
+            0.01878732442855835,
+            -0.0007163614500313997,
+            -0.03405960276722908,
+            2.9236827685963362e-05,
+            -0.014628356322646141,
+            0.029499130323529243,
+            -0.009795165620744228,
+            0.009514871053397655,
+            0.008105821907520294,
+            0.007098275702446699,
+            -0.0067119235172867775,
+            0.007916433736681938,
+            -0.008742166683077812,
+            0.0052763596177101135,
+            0.015832867473363876,
+            -0.0042877523228526115,
+            -0.0014980619307607412,
+            0.007840678095817566,
+            0.00407184986397624,
+            -0.005454384721815586,
+            -0.019711541011929512,
+            -0.012287517078220844,
+            0.00726114958524704,
+            -0.02065090648829937,
+            -0.004367295652627945,
+            0.01161329448223114,
+            -0.005454384721815586,
+            -0.009620928205549717,
+            0.01393140759319067,
+            0.02136300690472126,
+            -0.002382505452260375,
+            0.022893264889717102,
+            -0.01349202636629343,
+            -0.009833043441176414,
+            -0.007984613068401814,
+            -0.004374871030449867,
+            -0.041120000183582306,
+            0.025984082370996475,
+            0.01068150345236063,
+            -0.006871009711176157,
+            0.01895398646593094,
+            0.0022499337792396545,
+            -0.002954458352178335,
+            -0.0011041341349482536,
+            0.00010499217023607343,
+            -0.03281721472740173,
+            0.03660498186945915,
+            -0.007007369305938482,
+            0.0005449649761430919,
+            0.010272424668073654,
+            0.042029064148664474,
+            -0.004590773954987526,
+            -0.005859675817191601,
+            0.01225721463561058,
+            0.005336964037269354,
+            0.031938452273607254,
+            -0.010416359640657902,
+            -0.014340486377477646,
+            -0.01815097965300083,
+            -0.004833190701901913,
+            -0.01939336769282818,
+            -0.0038313264958560467,
+            0.014037464745342731,
+            -0.019650936126708984,
+            -0.011242092587053776,
+            -0.001287840772420168,
+            -0.012537509202957153,
+            -0.0010747789638116956,
+            0.008598231710493565,
+            0.017832808196544647,
+            -0.011348150670528412,
+            -0.004450626205652952,
+            -2.2904154320713133e-05,
+            -0.005003640428185463,
+            0.0185146052390337,
+            0.0020056227222085,
+            -0.01702980138361454,
+            0.01321930717676878,
+            -0.00526499655097723,
+            0.006749800872057676,
+            0.0074429623782634735,
+            0.007939159870147705,
+            0.05145302787423134,
+            0.010317877866327763,
+            -0.001339922659099102,
+            -0.01607528328895569,
+            0.023196285590529442,
+            -0.01749948412179947,
+            -0.014181400649249554,
+            -0.02119634486734867,
+            -0.0029923359397798777,
+            -0.007060397882014513,
+            -0.028014326468110085,
+            -9.137987944995984e-05,
+            0.0016221113037317991,
+            -0.0024980325251817703,
+            -0.010045158676803112,
+            0.0005208179936744273,
+            -0.011529963463544846,
+            -0.006954340264201164,
+            -0.008545203134417534,
+            -0.016044981777668,
+            0.02240842953324318,
+            -0.021817538887262344,
+            -0.008855799213051796,
+            0.011514811776578426,
+            0.03118089959025383,
+            -0.005011215806007385,
+            -0.014173824340105057,
+            -0.019817598164081573,
+            0.0025643182452768087,
+            -0.01763584464788437,
+            0.012181458994746208,
+            -0.01225721463561058,
+            0.015045011416077614,
+            -0.03142331540584564,
+            -0.006079366430640221,
+            -0.008992158807814121,
+            0.012840530835092068,
+            0.019817598164081573,
+            -0.0025093958247452974,
+            0.00606800289824605,
+            -0.010295150801539421,
+            0.030211230739951134,
+            -0.00012286569108255208,
+            -0.018605511635541916,
+            0.00873459130525589,
+            0.007977037690579891,
+            0.012537509202957153,
+            -0.004435475450009108,
+            0.00257757562212646,
+            0.011567841283977032,
+            0.012742049060761929,
+            -0.01634800247848034,
+            -0.003174148965626955,
+            -0.017969166859984398,
+            0.005996035411953926,
+            0.006999793462455273,
+            0.021711481735110283,
+            -0.0013550736475735903,
+            0.018272189423441887,
+            -0.01563590206205845,
+            0.00019412307301536202,
+            -0.011749654076993465,
+            -0.0082270298153162,
+            -0.0028237802907824516,
+            -0.003958216868340969,
+            0.006685409229248762,
+            -0.028211290016770363,
+            -0.01274962443858385,
+            0.005287723150104284,
+            -0.004992276895791292,
+            -0.03405960276722908,
+            0.0053710537031292915,
+            -0.02725677192211151,
+            0.013128400780260563,
+            -0.002397656673565507,
+            -0.017847958952188492,
+            -0.00030846628942526877,
+            0.026332557201385498,
+            -0.003677921835333109,
+            -0.012537509202957153,
+            0.015544996596872807,
+            -0.006757376715540886,
+            -0.02119634486734867,
+            -0.005420294590294361,
+            0.014658658765256405,
+            -0.0010719380807131529,
+            -0.012393574230372906,
+            -0.029635490849614143,
+            0.012992041185498238,
+            0.0026874206960201263,
+            -0.007386146113276482,
+            0.010893618687987328,
+            0.025953780859708786,
+            0.008151275105774403,
+            -0.005924067925661802,
+            0.0015757111832499504,
+            0.0049581872299313545,
+            0.00300938100554049,
+            0.020287280902266502,
+            0.02728707529604435,
+            -0.008014915511012077,
+            -0.0011145505122840405,
+            -0.014954105019569397,
+            0.015802564099431038,
+            0.0013020449550822377,
+            0.010507266037166119,
+            0.021893294528126717,
+            0.007045246660709381,
+            -0.005738467443734407,
+            0.003905187826603651,
+            -0.023135682567954063,
+            0.01420412678271532,
+            -0.012272365391254425,
+            0.000622140767518431,
+            0.008749742060899734,
+            -0.007283876184374094,
+            0.01437078882008791,
+            -0.0015245762187987566,
+            0.0339989997446537,
+            0.025211378931999207,
+            0.0043218424543738365,
+            0.0007362472242675722,
+            -0.028559764847159386,
+            0.003547244006767869,
+            0.008651260286569595,
+            0.014499573037028313,
+            -0.0076664406806230545,
+            0.006261179223656654,
+            -0.004355932120233774,
+            0.00959820207208395,
+            -0.0065490491688251495,
+            -0.015893470495939255,
+            0.006113456096500158,
+            0.0012689019786193967,
+            0.006795254070311785,
+            0.0023408401757478714,
+            -0.008863375522196293,
+            -0.024362917989492416,
+            -0.025984082370996475,
+            0.021984200924634933,
+            -0.03918066248297691,
+            0.0073293293826282024,
+            0.011234517209231853,
+            0.03324144333600998,
+            -0.008158850483596325,
+            -0.004980913829058409,
+            0.006276329979300499,
+            -0.016711628064513206,
+            -0.004147605039179325,
+            0.008204303681850433,
+            0.012385998852550983,
+            -0.002420383272692561,
+            0.00974971242249012,
+            0.01318900566548109,
+            -0.012378423474729061,
+            0.03660498186945915,
+            0.016514664515852928,
+            0.01467380952090025,
+            -0.004007457755506039,
+            0.01312082540243864,
+            0.0066172294318675995,
+            0.019044892862439156,
+            0.025165924802422523,
+            -0.017454031854867935,
+            -0.002365460619330406,
+            -0.02292356640100479,
+            0.024165954440832138,
+            0.00726114958524704,
+            0.004556683823466301,
+            -0.008310360834002495,
+            0.014870774000883102,
+            0.0017954016802832484,
+            0.0028199926018714905,
+            -0.006696772295981646,
+            -0.010173942893743515,
+            0.019241856411099434,
+            0.0034317169338464737,
+            0.014870774000883102,
+            4.8501173296244815e-05,
+            -0.005503625608980656,
+            0.006052851676940918,
+            0.014832896180450916,
+            -0.023590214550495148,
+            -0.00203592493198812,
+            -0.019484274089336395,
+            -0.03202936053276062,
+            0.009022461250424385,
+            0.0016135887708514929,
+            -0.007052822504192591,
+            0.03230207785964012,
+            -0.010083036497235298,
+            -0.019787294790148735,
+            -0.008067944087088108,
+            -0.013113250024616718,
+            -0.010469388216733932,
+            0.0008271536789834499,
+            0.003908975515514612,
+            -0.00012428610352799296,
+            -0.009317907504737377,
+            -0.005749830510467291,
+            0.01990850456058979,
+            -0.0030358952935785055,
+            -0.00753386877477169,
+            0.00868156272917986
+          ],
+          "embedding": [
+            0.0040707582550429606,
+            -0.010230625151989194,
+            -0.021520242101903186,
+            -0.010115756270640508,
+            0.026738734586159035,
+            0.01796370233223267,
+            -0.012245199029994153,
+            0.06506924494147726,
+            -0.0014087587222033442,
+            0.009300068719271429,
+            0.003787396973047584,
+            0.002818772639754174,
+            0.022129077062176306,
+            -0.021526709808445683,
+            -0.0034388135367225824,
+            0.02763761747879405,
+            0.013210013805395276,
+            0.025255872672296602,
+            0.00029011480909986703,
+            -0.05438974762290585,
+            0.016671640187490865,
+            -0.016426252259893328,
+            -0.056591180651012554,
+            0.009335648425523075,
+            0.006141624459418827,
+            0.02568943956694641,
+            0.019377132054042116,
+            0.017946464337078746,
+            -0.057184776985927686,
+            0.03208527139096687,
+            0.030132392063755547,
+            -0.003846032371791839,
+            -0.008376524427531415,
+            -0.0010513963058713435,
+            -0.02734868388724371,
+            -0.014879400370545479,
+            0.04383920959748825,
+            0.02563613659516289,
+            -0.020117467074438997,
+            0.0398016699155716,
+            0.029896497884197775,
+            -0.0004195844530067313,
+            -0.023302720788253625,
+            -0.001168337916577517,
+            0.009149021369661628,
+            -0.02508145721900992,
+            -0.004307594138480351,
+            0.00550940415564303,
+            0.014016998918718476,
+            -0.028607379616884317,
+            -0.0018317656587526954,
+            -0.01973464940415756,
+            0.03762738392056255,
+            0.015358106058583114,
+            -0.03931907682104936,
+            0.011519637298832444,
+            0.004482080558226775,
+            0.0747867063383242,
+            0.05550679018263077,
+            0.023243129601931176,
+            0.02671631686665238,
+            -0.009686146566573536,
+            -0.04088582189725495,
+            -0.0065344623594924085,
+            0.019617295999325712,
+            -0.004722087402575777,
+            -0.002871844114437583,
+            0.003935087265300519,
+            0.03229890413393201,
+            -0.02813577047904649,
+            -0.05124172132597234,
+            0.03325235407841968,
+            -0.008687655557217192,
+            -0.011027174838246948,
+            -0.003653553063547375,
+            -0.02350737945182988,
+            0.016223087939183468,
+            0.03789252356462714,
+            0.0034350623378603083,
+            0.00195158657888739,
+            0.011789712862232108,
+            -0.0004435439660769551,
+            -0.0037330456918125894,
+            0.0031552713243866858,
+            0.04465432992298031,
+            -0.0032465421571915712,
+            -0.0075366743861922745,
+            -0.059958752731410256,
+            0.016784616861008125,
+            -0.028734506065772967,
+            -0.03035575736273943,
+            0.02680390892611741,
+            0.02338368719038086,
+            -0.0030658375437784277,
+            -0.03754896050794851,
+            0.07005659523248504,
+            0.0020711575553570695,
+            0.013034425083076923,
+            -0.02316832262132332,
+            0.021332142030732894,
+            -0.0318756233020585,
+            0.02270782509507388,
+            0.01570810309454529,
+            0.004788124787612736,
+            0.026727250200142456,
+            -0.022380142778058276,
+            -0.008593910379436904,
+            0.014439754382387034,
+            0.015007697617383174,
+            0.07662531963612977,
+            0.011195998137723795,
+            -0.010809283538470098,
+            -0.042227822120219305,
+            0.003824320933275175,
+            -0.03797629427607094,
+            -0.02347390684453348,
+            -0.0526876440280276,
+            -0.017799861680287913,
+            0.01701482796266917,
+            0.002730395144917155,
+            0.03789451759899825,
+            0.038338545350313946,
+            0.028463789278406586,
+            0.015494583705011921,
+            -0.05700237266350479,
+            0.047895170359075166,
+            -0.0064406894534929406,
+            -0.0007954721295065988,
+            0.012543029839729358,
+            -0.004703334868741216,
+            -0.0060569679284068,
+            0.012324901254711192,
+            -0.019883935809784068,
+            -0.04253565942154215,
+            0.02164409276211115,
+            -0.0075367555415808635,
+            0.03617159147722157,
+            0.07567745988686005,
+            -0.04225977020036783,
+            -0.022548837589622003,
+            0.029811602145850144,
+            -0.02175792196819533,
+            0.02069861458115363,
+            -0.00941331829664397,
+            -0.035479639885943264,
+            0.020352552580119105,
+            0.003678588588550461,
+            0.06776105274336729,
+            -0.02346047721661687,
+            0.02721075076349443,
+            -0.029386781707644253,
+            0.023530831921117993,
+            -0.02077660533100372,
+            0.020022475323372906,
+            0.024195118676513715,
+            -0.028378518720494666,
+            -0.012900969023913307,
+            0.019264089970719624,
+            -0.0374983624629265,
+            -0.019726171557384355,
+            0.00979636085642416,
+            -0.014479528435832063,
+            -0.014533384583777662,
+            0.005904623758767224,
+            0.010344475500263744,
+            0.05659646358026115,
+            0.04163866113482093,
+            0.007152494819169405,
+            -0.07584500130974763,
+            0.05471858597878099,
+            0.03580211013427615,
+            0.044161429444475306,
+            0.02258180566046248,
+            0.00980779221334715,
+            0.0011193505603397395,
+            0.018086566768892507,
+            0.015605066702191871,
+            -0.00954727399396313,
+            -0.027950079146919375,
+            0.03028514025485407,
+            -0.006351669761257431,
+            0.005095804801686436,
+            0.009620040085743162,
+            0.019748001227987134,
+            0.033709535059510916,
+            0.008203559956940196,
+            0.007389752602218563,
+            0.04197771878351742,
+            -0.02067363758471765,
+            0.007543739188471913,
+            0.011207941261799052,
+            0.013964827251172813,
+            0.03839562075248879,
+            0.0009775244543292637,
+            0.006469507611492216,
+            -0.022808235395746654,
+            -0.024795004398633292,
+            0.006249024864814684,
+            0.011292877109066925,
+            0.01596839619826819,
+            -0.0014105396434174883,
+            -0.04152075438277094,
+            0.008267788411079337,
+            -0.028834191212891387,
+            0.030940526405027906,
+            -0.0018271056434014425,
+            0.006185475047820603,
+            -0.007920574674727123,
+            0.012907560753043616,
+            -0.015165704767187252,
+            -0.021962244496450727,
+            0.0037071053072833147,
+            -0.013508376064336042,
+            -0.015178823540513864,
+            -0.02518557053061311,
+            0.011693256916760944,
+            -0.01716015674592331,
+            0.02048897513447166,
+            -0.003412078966361745,
+            -0.001807298511378529,
+            -0.0012575655363779904,
+            0.03116103961636643,
+            0.011668533361641842,
+            -0.011637010548030312,
+            -0.040260195687925236,
+            -0.011129376691522074,
+            -0.0034493180842871908,
+            -0.02098244789711689,
+            0.026658743116016938,
+            -0.01923736325953796,
+            -0.009723937181300192,
+            -0.013134679989565051,
+            -0.011974548914818263,
+            -0.004028712301052159,
+            0.019628329175867303,
+            -0.028192034966932684,
+            -0.012638520793012082,
+            -0.011568652932013111,
+            -0.0025919910519445737,
+            0.012497908463405111,
+            0.015113186712897145,
+            0.014880607940212641,
+            -0.03059227859452334,
+            -0.021899716217105084,
+            -0.0008513506358489523,
+            0.011985931118464679,
+            -0.01321584595585435,
+            -0.011735288366213981,
+            0.010710683541435907,
+            0.01961722172525993,
+            -0.023571946726192585,
+            -0.03364281880125354,
+            0.0014460076743756465,
+            0.016563167828467187,
+            -0.02891589059260944,
+            0.04595166331401113,
+            0.034271553243501465,
+            -0.027557971577291153,
+            -0.007138821520412525,
+            -0.039508407807651844,
+            0.06814329636830853,
+            0.00022136123564675344,
+            -0.039718016705753735,
+            0.0503457594963753,
+            -0.003580581891576583,
+            0.013820796398521455,
+            0.02054050419088757,
+            0.035682661890720495,
+            -0.0013220739026852943,
+            -0.0052759394697328675,
+            0.04733018719162478,
+            -0.0187845855131196,
+            -0.03982699468608443,
+            -0.023905460465070062,
+            0.002631233133039351,
+            -0.03090995554989513,
+            -0.012617612051488022,
+            -0.013605787969473513,
+            0.039672783793764745,
+            0.008669582242097126,
+            0.009974277556736362,
+            0.005535636914843583,
+            0.014180668493093584,
+            0.012042659947850441,
+            0.011870115226306776,
+            0.00451259454348581,
+            -0.014023048592193627,
+            -0.007769033996392623,
+            -0.05715987527668358,
+            0.003374564518384728,
+            -0.028801158523506985,
+            -0.00756331369170698,
+            -0.02163054894055572,
+            0.005367181573233089,
+            0.02308705505078254,
+            -0.005438909398653553,
+            -0.00873620909426313,
+            -0.0036530652237719077,
+            0.013920407174296543,
+            -0.01539613379172909,
+            -0.03908211213269472,
+            -0.016810726548844992,
+            0.007460173854386443,
+            -0.007069189426742819,
+            -0.041197222498539215,
+            -0.003843233633335066,
+            0.004373367480223065,
+            0.016194121660459908,
+            -0.029165150066068787,
+            -0.00735139702658342,
+            0.017257895082337448,
+            0.014153313280536095,
+            -0.002376178042741926,
+            -0.023223700836427403,
+            -0.0319877537205969,
+            0.02088060774052704,
+            0.0012829290529515294,
+            0.03251550040316258,
+            -0.04128453069131847,
+            -0.023376114412063578,
+            0.04083022735076889,
+            0.005690491480259003,
+            -0.05810403800689653,
+            0.016582434816685452,
+            0.010402038964417569,
+            0.016795644054313566,
+            -0.00908590043098552,
+            0.05003656634510042,
+            -0.037216197954986735,
+            -0.013027960261454162,
+            -0.007090943677204207,
+            -0.05175371787508879,
+            -0.0020065660448063322,
+            -0.014908489552053589,
+            0.010627472758306387,
+            -0.03275473161158777,
+            -0.017334608434133866,
+            0.025321047439048318,
+            -0.0033515195870701753,
+            0.011526068468647207,
+            -0.026783879816504975,
+            -0.0012830935419413407,
+            -0.017506908439998053,
+            -0.02114853058440477,
+            0.004878884893669383,
+            -0.020936900837115968,
+            -0.0025587027781733642,
+            0.01557850246191957,
+            0.023652335794299834,
+            0.011489428313303988,
+            0.0019089137782552768,
+            0.02054907941022243,
+            0.014361232983780595,
+            -0.0056413109308506085,
+            -0.0010300050480036684,
+            -0.03413262505887213,
+            -0.018821160485010483,
+            0.03361989907065425,
+            0.013125889701529858,
+            -0.003742167496379789,
+            0.008813100942466033,
+            -0.0383265293638935,
+            -0.035316957263713485,
+            0.03020499323039515,
+            -0.05001623971243151,
+            -0.00855479667673597,
+            -0.005939674940109814,
+            -0.026451831943555135,
+            -0.008032873049527808,
+            0.0009431571685654594,
+            -0.014387605128858954,
+            0.016382559541755014,
+            -0.024909510232038502,
+            0.003715736505444385,
+            -0.006016712902409688,
+            -0.009110197692342108,
+            0.05507263160354303,
+            -0.022103031032598353,
+            -0.0475415595313825,
+            0.021958434235402655,
+            -0.016670809659931243,
+            0.0323519363801999,
+            0.03867494951405658,
+            0.011124508964363016,
+            -0.008002445714431074,
+            0.034041838454818577,
+            0.018400123235267234,
+            -0.01161754789471199,
+            -0.025901844576206774,
+            0.021800298132870245,
+            0.00943488425697727,
+            -0.05906691123008531,
+            -0.00782549274387971,
+            -0.019357446956797447,
+            0.008682376344965976,
+            0.02863077694455766,
+            0.0024925977373894447,
+            -0.002682170968419412,
+            -0.013590235283325244,
+            -0.021511149507762284,
+            -0.03255453916518719,
+            0.009039307015853905,
+            0.008305286476500875,
+            0.009312625027668458,
+            -0.0014144165766983945,
+            -0.007853869334376857,
+            0.009067065760942163,
+            0.022485201611182105,
+            0.03966189080765722,
+            -0.041627289519629344,
+            0.002707086456239785,
+            -0.022678178050044222,
+            -0.03653753142622053,
+            0.0036574646778835588,
+            0.029996160786576367,
+            0.02901163765559891,
+            -0.01971777193183787,
+            -0.0036589934841109884,
+            0.009181234938554578,
+            -0.025979483002227535,
+            -0.00773645103995767,
+            0.004280770859737866,
+            0.04116668301017644,
+            0.006201085381597264,
+            0.01402728258112097,
+            0.001060407317110527,
+            -0.00920801784000087,
+            0.02109880889040528,
+            0.025211022083982727,
+            -0.029866678264942617,
+            0.014439702646820226,
+            0.009379206296908577,
+            0.04156435812161122,
+            -0.037309655469525846,
+            -0.011819731348299495,
+            -0.02766498020409985,
+            -0.010454143230794476,
+            0.027911532296566614,
+            -0.04127890742215716,
+            -0.006948536183086887,
+            0.010207784113909938,
+            -0.0016584043356136647,
+            0.016437495428919636,
+            -0.014607861750686952,
+            -0.03222311919805017,
+            -0.000610723203118213,
+            -0.02867445198803213,
+            -0.024772816792038133,
+            -0.016510309753766612,
+            -0.018581201775667577,
+            -0.07450654873867177,
+            0.00026832248829154255,
+            -0.06724696990816072,
+            -0.02447002541264633,
+            0.04885569249258513,
+            -0.010195604207134053,
+            0.03332905935704749,
+            -0.02815272112271314,
+            -0.005296340304911926,
+            -0.05908835042526019,
+            0.0038476172058970774,
+            0.025753576863447866,
+            0.02386003857017594,
+            0.005206447371391696,
+            0.024799057871126032,
+            0.012060738268265606,
+            -0.04146503002980051,
+            -0.024949907556862643,
+            -0.014503475223770958,
+            -0.0016340466065744286,
+            -0.0034009167013795906,
+            -0.031298432995411005,
+            0.036241272149880994,
+            0.008902644391348814,
+            0.03550223922667804,
+            0.04463301409716232,
+            -0.010385984413759852,
+            -0.03305004479734889,
+            -0.006952739338820577,
+            -0.0175981588092108,
+            0.042668533202572856,
+            -0.017112184620086308,
+            0.005149173014526686,
+            -0.03219589107997635,
+            0.024181808150698054,
+            0.0014339790422945406,
+            0.01254721190611376,
+            0.011099458649404119,
+            0.0027626836767364253,
+            -0.026883640446090158,
+            0.0024160770766463607,
+            0.023874514847956222,
+            -0.022558433617669373,
+            0.00917667715473093,
+            -0.006754091166465801,
+            -0.033686935718776144,
+            -0.01078389638118685,
+            -0.013595327486167114,
+            0.024712781935748277,
+            0.013168464807451798,
+            0.005875864604862434,
+            -0.02417814420558463,
+            -0.0023978800459714144,
+            0.028650463366555304,
+            0.020176980768301288,
+            -0.02118184201861616,
+            0.019082340279671355,
+            0.019734925740100277,
+            -0.04209645071102001,
+            -0.04217310001316092,
+            0.006553913719098956,
+            -0.003735719227468833,
+            0.0061793909220459,
+            -0.028285889794908054,
+            0.008181788076484304,
+            0.004432488437404309,
+            -0.01743905248466261,
+            0.041221269621866216,
+            -0.014892354419172107,
+            0.005087102208335201,
+            -0.022198383716186407,
+            -0.011936388003605221,
+            -0.00027487042024290816,
+            0.044599215801158845,
+            0.008068975171827777,
+            0.003496950921841073,
+            0.02238779276447619,
+            0.0054839808262270315,
+            0.04816723616939378,
+            -0.02685221260466647,
+            -0.0190356946337995,
+            0.016464136459560113,
+            -0.0176092780782677,
+            -0.012513274095432353,
+            -0.010695421445334002,
+            -0.004784348908457072,
+            -0.022203041334932165,
+            0.010788155380415583,
+            -0.018353897272137543,
+            -0.02277431189907027,
+            -0.003945465850796918,
+            0.016024639792926797,
+            -0.006248368166794934,
+            -0.0038769803892794875,
+            0.0016292556540434277,
+            0.02412864830338344,
+            0.02892955453819876,
+            -0.02408022470555968,
+            0.019190282372105023,
+            0.018941584034503536,
+            -0.006709516380602965,
+            -0.020956377933747532,
+            -0.01964885206488053,
+            0.01434623424964373,
+            -0.01352968586212152,
+            -0.013872556892707697,
+            0.004052798542613835,
+            0.009201147374299997,
+            -0.004086067985859617,
+            -0.04185829685685201,
+            -0.014668698123513235,
+            0.003179403563664022,
+            -0.0006415014901450319,
+            0.019360184344240947,
+            -0.0150817538567364,
+            -0.029771999278536083,
+            0.014221776097046751,
+            0.022944884879589678,
+            -0.04493425220030206,
+            0.02117577719533236,
+            -0.0030648751497261983,
+            -0.00016251533643103604,
+            -3.817296018601414e-05,
+            0.008727767382077138,
+            0.026914779443266063,
+            0.03456565094655899,
+            0.01974806772696219,
+            0.00450800007152441,
+            -0.008952599700759673,
+            0.025184190461155245,
+            0.009846044083064704,
+            -0.025168735183134738,
+            0.010983353565382029,
+            0.016740721743326522,
+            0.0070838930012654296,
+            0.009173879407277282,
+            -0.006078288608766152,
+            -0.018269347893097643,
+            0.013181494188066667,
+            -0.01435693747022231,
+            0.01678907949532146,
+            0.05154463552784974,
+            -0.00039632596424461964,
+            -0.02461439550772063,
+            0.008418147037164186,
+            -0.006544319842335593,
+            -0.010065129286193278,
+            0.02214628550460128,
+            0.0029515574751370417,
+            -0.030862152033847717,
+            0.007509556025223583,
+            0.00675485304374519,
+            0.01894936231559203,
+            -0.02355525295639599,
+            -0.01387885858544803,
+            -0.00431398020087544,
+            0.0021208056468763523,
+            0.01580766504149989,
+            -0.004739477121073512,
+            -0.013204823447960706,
+            0.02994098229517376,
+            0.025496728234662373,
+            -0.00026124946738213036,
+            0.015060064980768175,
+            0.022262345898533903,
+            0.009968408664790368,
+            0.007588431141088797,
+            -0.0083063413609779,
+            -0.031336272171937095,
+            0.0018957858383561667,
+            0.05800811417728381,
+            -0.014937085296865802,
+            0.04313875432235027,
+            0.02018476425923236,
+            0.015657644644987107,
+            0.017880263053253607,
+            -0.01662268082870562,
+            -0.008740053682875323,
+            -0.0017630394204179638,
+            -0.008294417990024647,
+            -0.027613464270562948,
+            -0.005953209556351086,
+            -0.02379961210881074,
+            -0.013878676554701543,
+            0.015418315245979704,
+            -0.014764728065382074,
+            0.0027420640870858457,
+            0.010395538988699663,
+            -0.006590633833835581,
+            -0.00841537209927409,
+            -0.003943772700797381,
+            0.03376338017637296,
+            0.011980354721458463,
+            0.027761929822019422,
+            -0.00011128539412582973,
+            0.009342430952107655,
+            -0.018419526564396063,
+            -0.006839752805318664,
+            0.03271190887864607,
+            0.006011520427944665,
+            0.004652835471939905,
+            0.01485920920301398,
+            0.014305626381398871,
+            0.038015677219487023,
+            -0.005102005401565164,
+            -0.018204460895989052,
+            0.02065495133597456,
+            -0.0049699013356374775,
+            0.019149036693562712,
+            0.008825981577388056,
+            -0.002660055848523568,
+            -0.004292379561953203,
+            0.03090270710860331,
+            -0.020243440437217503,
+            0.018735727616321387,
+            -0.02238236089688497,
+            -0.0276584413929402,
+            0.023028408906176946,
+            0.008320436733620496,
+            0.04047641118026783,
+            0.011463851066078041,
+            -0.02247143742166039,
+            -0.034588725413285185,
+            0.009827713930789184,
+            0.00337071450636392,
+            -0.024446280199879062,
+            0.017542777160283324,
+            0.00780836483490212,
+            0.008453886667283516,
+            0.016624116261219637,
+            -0.004175682824655864,
+            -0.0042784362933758835,
+            -0.001546656717896957,
+            -0.008819739495525477,
+            0.01406790007187259,
+            -0.015487658289541583,
+            -0.021457182907140035,
+            0.024687169583245905,
+            -0.03915911962859764,
+            0.01771290139419629,
+            -0.006510046907173026,
+            -0.00997450880725535,
+            -0.02059560682587115,
+            1.5939836949198004e-05,
+            -0.007159493752862143,
+            -0.0004782363814453761,
+            -0.017503181957042913,
+            0.016643971600145645,
+            0.024056442566869802,
+            -0.013209211443909658,
+            -0.00445521990927108,
+            0.0020313626821877526,
+            -0.029235692455820056,
+            0.006662209097255912,
+            0.016993027053110635,
+            0.00529940612845993,
+            0.0026104743851393716,
+            0.0001707934714983891,
+            -0.031289446611163674,
+            -0.005782208620284939,
+            0.03053516889069683,
+            0.05590704747438521,
+            0.021628254147289437,
+            0.02403314981042378,
+            0.02908606663381421,
+            0.02717206823892633,
+            -0.003198844932040613,
+            0.0016217508103442348,
+            -0.01620660606431983,
+            0.027098323143010743,
+            -0.011664168290676943,
+            0.04322175021589083,
+            0.008652323184522541,
+            -0.049493020736445056,
+            0.004537958157696794,
+            0.00943239454147692,
+            0.034918603888916544,
+            0.016678716235690318,
+            0.0005887414054803805,
+            -0.005894338664998101,
+            0.03279165663336323,
+            -0.02735726709032382,
+            -0.011962070182991011,
+            -0.00036014407560325284,
+            -0.019079003730607717,
+            -0.015938295169860884,
+            0.010357611943935459,
+            0.04278924678845588,
+            0.0009957483598272451,
+            -0.0071422790468850215,
+            0.01571201177743999,
+            -0.009445307674383,
+            0.02286338492075306,
+            0.007336440384050498,
+            0.0042809497342459904,
+            0.025581467690095977,
+            -0.0025007871391868105,
+            0.02300539823997411,
+            -0.000628898771317254,
+            0.004582863909134464,
+            -0.015960578501883316,
+            -0.00930649396114688,
+            0.027838455754803444,
+            -0.011112268986721,
+            -0.029253746504396393,
+            -0.008734785561831333,
+            0.00472579547270089,
+            -0.0003805825712986138,
+            0.0021854358149324227,
+            -0.01183885533661493,
+            -0.010840951201591629,
+            0.019965691788904057,
+            -0.004922943301424112,
+            0.003800048754664761,
+            0.012664049429916252,
+            -0.0030124219674414897,
+            -0.01910844996750822,
+            0.01889983984736845,
+            -0.003858283778519141,
+            -0.024168821338992813,
+            0.020083955711766945,
+            0.01850497889545421,
+            0.045156567155119454,
+            -0.01586824055785609,
+            0.004612757419198795,
+            0.017497774574602505,
+            -0.002172088240279357,
+            -0.019617327081566827,
+            -0.014944795772093997,
+            -0.018212462639041492,
+            -0.015975283672064702,
+            0.012837956594691596,
+            -0.01243789877725244,
+            0.019500571353639314,
+            -0.0380735226388175,
+            0.012590900534826689,
+            0.004832168601136412,
+            0.007505385730121269,
+            0.016011983211212992,
+            0.025994560026860358,
+            0.002157436199771672,
+            -0.033465476168065784,
+            -0.01014599891204166,
+            0.007845373376916211,
+            0.010093432683262838,
+            -0.002849185178237036,
+            -0.018147004183434554,
+            0.003633790611809425,
+            -0.0266147420637404,
+            -0.018644422659309035,
+            0.022926737299761104,
+            -0.04798772015599113,
+            0.017908415301694906,
+            0.024859125158586466,
+            -0.04343699525707025,
+            -0.04886699607556352,
+            -0.011459188432595229,
+            -0.036774381316312464,
+            0.0014128005215813578,
+            -0.0005256067584181517,
+            0.011194992852264102,
+            -0.009342140079829378,
+            -0.018598671981710183,
+            0.01584789414301952,
+            -0.0036412641242200418,
+            -0.004225840239935549,
+            -0.00616696736018045,
+            -0.0025598631463178384,
+            -0.010188803157718543,
+            -0.03244694662411977,
+            -0.03290891136780909,
+            -0.00033742479296272127,
+            -0.009743927051414455,
+            -0.02334288359945272,
+            -0.03447062782872289,
+            -0.008728968107967028,
+            -0.037753659749578956,
+            -0.00515561961442376,
+            0.04070890150714597,
+            0.027517147608335957,
+            0.009449414978133334,
+            0.008086900089929912,
+            -0.013989034604490105,
+            0.05691047488419298,
+            -0.020425883381389086,
+            0.039466620642512155,
+            0.005301493569940605,
+            -0.0037222353899918703,
+            0.013075425214878277,
+            0.027710826422351564,
+            -0.006253665106995653,
+            -0.0017412928073238833,
+            -0.01928096630475438,
+            -0.012920476859996416,
+            0.009444408011467955,
+            0.015799884767184885,
+            -0.0025083061619197524,
+            0.001274457251739847,
+            0.00930533161700852,
+            -0.0006652478540264343,
+            0.017258038491335247,
+            0.03996268817486439,
+            0.003833844612473671,
+            -0.00826404141706295,
+            -0.018822002570963614,
+            -0.003188844900317347,
+            -0.011807006674248829,
+            0.010482058734260632,
+            -0.015410635475145028,
+            -0.011540803927522013,
+            -0.01249428777269578,
+            -0.017081856786754605,
+            -0.002183034976477535,
+            0.003114834030705039,
+            0.0027475411722881747,
+            0.02261627635192778,
+            -0.015872027027080824,
+            -0.022487581027490516,
+            -0.0035823117885461506,
+            -0.02398778449664045,
+            8.13716819067491e-05,
+            -0.01050060782407259,
+            -0.015616181283381218,
+            0.013676480044963827,
+            -0.021513173575512936,
+            -0.01091399089076612,
+            -0.008699176177861256,
+            -0.0006986953850759474,
+            0.005853723710671249,
+            -0.00993355748621621,
+            -0.00010241843582159751,
+            0.01150872694850241,
+            0.005559552266827916,
+            -0.012336449703762907,
+            0.0039431718621504,
+            -0.012740550272925562,
+            -0.004432059174951158,
+            -0.03592309122711106,
+            -0.019665686718735042,
+            -0.01846795637714379,
+            0.005642094448288269,
+            -0.005108964028543505,
+            -0.020586678054151854,
+            0.00668364061048941,
+            0.0034775127050453774,
+            0.004713169526661792,
+            -0.010723667875209928,
+            0.013944735241748791,
+            0.010882420168586074,
+            -0.003518245916227596,
+            0.013827334494230662,
+            -0.001330399936559009,
+            -0.020286697504699764,
+            -0.011139922402071095,
+            0.03538258948179278,
+            -0.024961826464903997,
+            -0.0018019328351763667,
+            -0.0026423695690168066,
+            0.00597003516870726,
+            -0.03071835946501977,
+            -0.013634212232744325,
+            0.02164400349828422,
+            -0.005846552894831441,
+            -0.026367014426434226,
+            0.03399116480234033,
+            0.03473258231629753,
+            0.010041605592929843,
+            -0.013423384340589794,
+            0.04730054254696893,
+            -0.012968978954938308,
+            -0.018274800841400665,
+            0.026826644936918647,
+            0.0278389611094493,
+            0.02525164095926168,
+            -0.029782550447890466,
+            -0.003010713086487275,
+            -0.02393629084357885,
+            0.017993366287762866,
+            -0.0263476663180971,
+            0.01864011499179424,
+            -0.02332263152537849,
+            0.016014908881756034,
+            0.01868464825773604,
+            -0.003104041044182249,
+            0.026313491086684425,
+            -0.0052945248919367815,
+            0.015253731232325074,
+            0.0012371308759955573,
+            -0.02119765517013431,
+            0.007795466385020642,
+            -0.006469500272298062,
+            0.025762169721880456,
+            0.00833992768202466,
+            0.01801901639978976,
+            0.022019940825351116,
+            -0.0409522298213741,
+            0.02003698451081499,
+            -0.001367916864679543,
+            0.01782984895971748,
+            0.0038516134262842373,
+            -0.007885432294782304,
+            0.014221728513400065,
+            0.010991352188016265,
+            -0.012851099261924757,
+            0.003840539873231741,
+            0.0008923482291826014,
+            -0.028928104542716937,
+            0.007066004768771198,
+            -0.026914574065060926,
+            0.025866907400807087,
+            -0.005526774784152086,
+            -0.00998122210956923,
+            -0.020744532673097317,
+            -0.0071425207262770865,
+            0.003754468896273156,
+            -0.012747874997429746,
+            -0.011385805601999295,
+            -0.024700457669046855,
+            -0.02401291382882952,
+            0.029591582802488962,
+            -0.014054332096344003,
+            -0.04015978298687865,
+            0.001234440863385445,
+            -0.023301255402129767,
+            0.01193125007773239,
+            -0.005793124129267758,
+            -0.024440100649464773,
+            -0.042563399082257154,
+            -0.017548888269389457,
+            -0.0009394887961485922,
+            0.022739763644266633,
+            -0.00114156949394707,
+            0.041791395064755515,
+            0.0008816696205437911,
+            0.022207132857533028,
+            -0.015547580543606755,
+            0.005319731264840819,
+            -0.02071229859314312,
+            -0.041495579742912285,
+            0.03660432378985971,
+            -0.014637031447181937,
+            0.025854844497435178,
+            0.03324325296704186,
+            -0.013009347371118843,
+            0.001354504885302625,
+            0.02652570709474851,
+            -0.014618100923440427,
+            -0.010782232747863238,
+            -0.0034936699515378212,
+            0.003461670978569965,
+            -0.006468116285021298,
+            -0.027793221545159466,
+            -0.007752458229193391,
+            0.04193706715422074,
+            0.017894069367616637,
+            0.021928994684423043,
+            0.022606401934291136,
+            -0.017972404868085358,
+            -0.026402048654300563,
+            0.003656646576244765,
+            -0.013129722990153284,
+            -0.018090360146368677,
+            -0.01624598856366761,
+            0.012397150789705746,
+            0.011741002980073786,
+            -0.004052422390675261,
+            -0.01836750423615859,
+            -0.020929335029248472,
+            -0.016327412417122416,
+            -0.0042615345226698015,
+            -0.008458117095535,
+            -0.005002582649405199,
+            -0.012158173174049797,
+            -0.020409616620022143,
+            0.03706368540614453,
+            0.0285565642978937,
+            -0.0041732933884846566,
+            -0.00607154864704767,
+            -0.011142150176334957,
+            0.008834431806990914,
+            0.006021232046087652,
+            -0.005028870471297621,
+            -2.3396043292139426e-05,
+            0.005112033372713154,
+            0.022822312435625314,
+            0.007271235299348001,
+            0.0035791789459533856,
+            -0.06340305811365254,
+            -0.005392000517722054,
+            0.022560201096078862,
+            0.012581401789917604,
+            -0.0032248219865223496,
+            0.025180718735801837,
+            0.0006451724964970927,
+            -0.009714514747463593,
+            -0.018389422219956937,
+            0.024728862150150004,
+            0.008760329201960964,
+            0.020988700992999768,
+            -0.014974763313686516,
+            0.003845297486923296,
+            0.0006445983847138896,
+            0.00750394530065325,
+            -0.0019792992847026793,
+            -0.008578254004439607,
+            0.025776677078006516,
+            0.016593438298233645,
+            -0.005695578040595484,
+            0.01604018433477423,
+            -0.0017497260488899657,
+            -0.021460543443428924,
+            -0.010265213095609554,
+            0.00042017209251926397,
+            -0.010570864288876387,
+            -0.018015790888295944,
+            0.017569414447505878,
+            -0.002601036282535863,
+            0.0072399377790164435,
+            0.030281707151249355,
+            0.0423183067569767,
+            0.009566493398534713,
+            -0.005928603293907002,
+            -0.0037936911251953654,
+            0.015477873304924939,
+            0.01513127995244932,
+            0.010741256034652754,
+            0.03015858543221727,
+            -0.029247492875021843,
+            -0.006690306960497428,
+            0.0046653727410352576,
+            0.004610405633429778,
+            -0.011692179099382171,
+            0.030055897497741527,
+            0.04569730592994177,
+            0.006736813244286893,
+            -0.004799978513117371,
+            0.002121067028052434,
+            0.011299889107780231,
+            0.007768137218140627,
+            -0.018927080714961783,
+            -0.0012222962555201746,
+            -0.006009911026251445,
+            -0.014001766533604134,
+            0.026941585318261085,
+            -0.003960837039351648,
+            -0.011848232223811548,
+            0.02903357274500332,
+            0.1105267435019781,
+            -0.006011029338781496,
+            -0.0053646998281195206,
+            -0.03242321704621517,
+            -0.015194102811332289,
+            -0.008860590919147765,
+            0.0056988205877963076,
+            0.01635957585050017,
+            0.020327661913559888,
+            -0.0008943408274661007,
+            0.016820860249955754,
+            0.02822175614376368,
+            -0.013854827839129885,
+            0.007906244895327178,
+            -0.030891381690403466,
+            0.015702117924362928,
+            -0.005383974926093977,
+            -0.012032955400313954,
+            -0.0026169511828034054,
+            -0.008618307035330928,
+            -0.0037356680273096105,
+            0.0009666430093455632,
+            0.011296307527684261,
+            0.004427516814276546,
+            -0.014484198313369681,
+            0.002829529156309784,
+            -0.008595300985863424,
+            -0.005122996529848575,
+            -0.02393571073523279,
+            -0.024119245543788293,
+            0.005223147047205755,
+            0.0019389305548629142,
+            6.344308292376579e-05,
+            0.006848965054539454,
+            -0.003810712095343454,
+            0.030149081317333418,
+            -0.01700322008272963,
+            0.0012288912810468446,
+            0.01193226302726573,
+            0.012332749933413344,
+            0.01688602002694633,
+            0.0028600984823761366,
+            -1.9882692552530084e-06,
+            0.021172977047420227,
+            -0.010514036509402561,
+            0.01460334896316005,
+            0.013091417303145243,
+            -0.04103572381458182,
+            0.0089746401463863,
+            -0.0027841289484449873,
+            -0.024252447218794344,
+            0.004446472209246425,
+            -0.018987867343500102,
+            -0.00881974688349601,
+            0.0005365625696166774,
+            0.012054719659248512,
+            0.01960300421041832,
+            -0.008553943699846696,
+            0.02111201386480852,
+            -0.0003501657740808554,
+            0.004177088753111459,
+            -0.012879944670340026,
+            0.0065222126975051685,
+            -0.007563354540402343,
+            0.0008168128607271199,
+            -0.018213241937597355,
+            0.012526561612175514,
+            -0.0008285725179894544,
+            -0.015737003159436623,
+            -0.011337688275309921,
+            0.004737346137976519,
+            -0.0027993545140114705,
+            0.0030843163114700313,
+            0.010153740340626154,
+            -0.008815302484351509,
+            0.004919275910399053,
+            -0.0017541042538165297,
+            0.002562816687714991,
+            0.014285249186421434,
+            -0.029816964593980453,
+            0.005357227738099325,
+            0.009154842495811638,
+            -0.00048782303639333943,
+            -0.013074976152495576,
+            -0.004054505559072866,
+            0.0028322102449485015,
+            -0.016877328079971877,
+            -0.012128199639779175,
+            0.023421330164419888,
+            -0.012565145848820333,
+            0.008609070188894315,
+            -0.010936598722317274,
+            -0.018401801734378692,
+            -0.01066119413133089,
+            -0.005254229780560808,
+            0.01234969508270852,
+            -0.0038427767204741877,
+            -0.006947594635952202,
+            -0.01756894543880097,
+            -0.007890897373987378,
+            -0.047246698568222466,
+            0.009617876521097624,
+            0.027157555263862723,
+            -0.011355758309667893,
+            -0.04050311381480088,
+            -0.01017774338301804,
+            0.02437257208967103,
+            -0.004056329295095018,
+            -0.008345450168215274,
+            -0.001599171409135998,
+            0.018358386463243014,
+            0.0027207445181251406,
+            -0.007292967749349572,
+            -0.011267861428599221,
+            0.01657507035043879,
+            0.015105061280816214,
+            0.021419854579631276,
+            -0.02088952706825817,
+            0.006295984081511286,
+            0.020973020776936996,
+            -0.03073246543555915,
+            0.0023720435876790876,
+            0.03363966216709148,
+            0.010412588613998134,
+            0.013023311559810034,
+            -0.0010217662933572685,
+            -0.0075567381078977215,
+            -0.008248818794295818,
+            0.011848200323549865,
+            0.005066463524567732,
+            -0.023174489297821385,
+            0.021823825355858556,
+            0.020368868756448753,
+            0.0026042856858509524,
+            0.003457067317891926,
+            0.0005539745115410357,
+            0.01062670844108589,
+            0.009948889803890108,
+            0.05100741556350918,
+            -0.006493714124814164,
+            -0.014021751055930108,
+            0.015063988102064609,
+            0.02488059237697814,
+            -0.0023499941119596228,
+            0.01087282522970853,
+            -0.02612218935931224,
+            -0.021423880914461062,
+            0.023389294741183817,
+            0.008967763132667892,
+            0.009637806238448432,
+            -0.010815706635624337,
+            -0.011390999462441815,
+            0.031750675158698435,
+            -0.01307829462211914,
+            0.00759749911213318,
+            -0.02169895957209634,
+            0.01387879409130081,
+            -0.008606362268863441,
+            0.032229658009209815,
+            -0.0009066257436946125,
+            0.011062436424260079,
+            0.028010486866468023,
+            -0.007835805879241447,
+            -0.019657486763625547,
+            0.003967480939987674,
+            -0.015577952617624695,
+            0.02188429826317828,
+            -0.020502758155233298,
+            -0.0002362500702658591,
+            0.0040935406015014265,
+            -0.010056462958039452,
+            -0.0057038083589231615,
+            0.011475170331848666,
+            0.0015345076181482267,
+            0.006359262977587827,
+            -0.020293014401106,
+            0.01369706440475074,
+            -0.003989482215141903,
+            0.007887824197823247,
+            -0.015925992045349828,
+            0.014370223626942766,
+            -0.02309139937510304,
+            -0.021808736033141787,
+            0.008142621179624918,
+            0.0108996933426044,
+            -0.007589050511458233,
+            -0.025971539510030145,
+            0.02250992080022228,
+            -0.024980833805349545,
+            0.024979008145354243,
+            -0.006622339940663538,
+            0.002566378488088469,
+            0.004913874740363343,
+            -0.008102103508732738,
+            -0.0028098905427027783,
+            0.0057865587191235565,
+            -0.0073509382400023995,
+            -0.014123756810537343,
+            -0.019453380788786282,
+            -0.04083629135586275,
+            -0.03177927047231389,
+            0.022927581786088044,
+            0.011557232599801421,
+            0.0017073832658476193,
+            0.02339552660545296,
+            -0.02247471449131386,
+            -0.017890866287500053,
+            -0.009775068417716798,
+            0.025184899522229628,
+            -0.04408541669037361,
+            0.012104255392615549,
+            -0.014669458760022245,
+            -0.0245258242481677,
+            0.006274252098230249,
+            -0.004161960147481513,
+            0.009299171724026455,
+            -0.010779421730862708,
+            -0.04713627671791054,
+            -0.005192684784377356,
+            0.02330975631183273,
+            -0.0036802876364168935,
+            -0.006197548344287773,
+            0.03048284650641829,
+            -0.0019503797069087845,
+            -0.005038254331399784,
+            -0.005524875459245342,
+            0.027783596017224226,
+            -0.024950083604372984,
+            0.010988398531346805,
+            0.008304937912856357,
+            -0.008485065895193593,
+            0.006572665697403556,
+            0.0011518846966102158,
+            0.011927348472282199,
+            0.015632588644277183,
+            -0.015559413716826264,
+            0.02262990861928957,
+            0.011446850286896198,
+            0.006506075777765303,
+            0.008015225053785824,
+            0.00010054430757843335,
+            0.0022975742216194894,
+            -0.024948925449668525,
+            0.026047924967851216,
+            0.01888211557921096,
+            0.0001632181017876392,
+            0.012553094476616919,
+            -0.015042621225242777,
+            -0.02782009986290425,
+            0.02606646297992026,
+            -0.016847653820092335,
+            -0.00045272379465939274,
+            0.020053091486344633,
+            -0.016210036358455773,
+            -0.007930550661303182,
+            -0.01123514849773852,
+            -0.0162675150307978,
+            0.01803076559921804,
+            -0.012160217543572069,
+            -0.02216594536057653,
+            0.002594675636753233,
+            0.016443123828746582,
+            0.02134480656850119,
+            0.026694575953313482,
+            -0.011470930697351533,
+            0.017768927048191052,
+            0.0008653408391607816,
+            0.001865765599795092,
+            0.025745954171649543,
+            0.01702677011968492,
+            -0.008474889109168202,
+            0.025347176686722313,
+            -0.0016686223698600755,
+            -0.002421921060670558,
+            -0.012265504084322075,
+            -0.0165214436458303,
+            -0.007545471138114208,
+            -0.015333235556146196,
+            -0.0051830106590765685,
+            0.000762746782664525,
+            -0.0018118022696726346,
+            0.10876576838038368,
+            0.0034948044540582626,
+            -0.011608460689239234,
+            -0.005870263500417206,
+            0.007510774308250375,
+            0.0023525201199670765,
+            -0.006846154764389575,
+            0.016082721103476375,
+            -0.018616626204086575,
+            -0.023593744079394,
+            0.004743481974389998,
+            -0.0009262286319469741,
+            0.00870122878944883,
+            0.02029523251993269,
+            -0.008608837818169835,
+            -0.0200567711498164,
+            0.007089477722191928,
+            0.005535149644041221,
+            0.0023640746516071774,
+            0.005256431824732927,
+            -0.0006118671321824933,
+            0.008209920240361308,
+            0.02448422122961942,
+            -0.015622892878860465,
+            -0.029770836607612903,
+            0.010535371601298532,
+            -0.00015994156470228762,
+            -0.007193425789067776,
+            -0.011046316463797128,
+            0.014728841492009627,
+            -0.0011488451990980268,
+            -0.02751404133172324,
+            0.017663132712536528,
+            -0.019055335178900415,
+            0.014643246200933506,
+            0.0025348345163347464,
+            -0.00016387173410118788,
+            0.022565953069005714,
+            0.013101158251834834,
+            -0.04272085076882881,
+            -0.0034442020309914823,
+            0.01041205537746844,
+            0.011059422927946552,
+            0.0116446016862942,
+            0.0012992535381387887,
+            -0.018670801241345514,
+            -0.011788984490283379,
+            -0.009109849103208542,
+            -0.021268488414296117,
+            -0.012523033785938503,
+            -0.011393853408628214,
+            0.010743646386439633,
+            0.0012964030814325153,
+            0.023809566099779113,
+            -0.003030535542277875,
+            -0.006526351867141901,
+            -0.000430699447948508,
+            0.00880561288237257,
+            0.009694646009259077,
+            -0.02079011138439714,
+            0.008294338719512953,
+            -0.01823922789220214,
+            0.011819769685519291,
+            0.006416044253693268,
+            -0.022799568832306067,
+            -0.009911619276012527,
+            0.007152786695893951,
+            0.008393677195358034,
+            -0.013569953300860946,
+            -0.006503988731872938,
+            0.02299498754558268,
+            0.018870619152710703,
+            0.012233964644738905,
+            0.024994616203196145,
+            0.03838969722235787,
+            0.005744255912479976,
+            0.027682123067526515,
+            -0.016791471092091604,
+            0.008459745383680297,
+            -0.022493868174622695,
+            0.012902262638928338,
+            -0.00460399519266689,
+            -0.0050431343566427886,
+            -0.018188192498411616,
+            0.033765105245412236,
+            -0.009159751849801773,
+            -0.0027150644587327763,
+            0.034778224828120714,
+            -0.0066502021793471075,
+            0.004525230911930458,
+            0.005096748531775903,
+            -0.016067049494073893,
+            0.033274672789154185,
+            0.01686831040872425,
+            0.016353966592947262,
+            -0.021631941883929817,
+            0.011748317746828848,
+            -0.005675701689448899,
+            0.0001273442591116645,
+            -0.020738819037390263,
+            -0.0037528323797385733,
+            0.0027725541343763104,
+            0.0059860509282538546,
+            -0.016556134518370437,
+            0.02033623476757896,
+            0.00019647023567641008,
+            0.0052895744489259875,
+            0.03253462247073863,
+            0.01043311189804924,
+            0.019182938878855655,
+            0.010051051068017807,
+            0.019523739618278902,
+            0.0281939391326836,
+            0.025685787482623124,
+            0.0029932688462254147,
+            -0.018668945726139946,
+            -0.0009326901784812825,
+            -0.007880352863625262,
+            0.010492063055849727,
+            0.022849902214818293,
+            -0.022619228461640748,
+            -0.02772644874135741,
+            0.018566981184526675,
+            -0.0022608185345048414,
+            0.020031103834122595,
+            0.007544464625538984,
+            -0.020935181884939796,
+            0.003407355515103286,
+            -0.023178230166337178,
+            -0.0005418155871620593,
+            -0.031039187720144967,
+            -0.028591174077243604,
+            -0.0007659241904525392,
+            0.011945344534674635,
+            -0.01688153017778311,
+            0.03357788163306365,
+            0.017667778671101954,
+            -0.014289109343082812,
+            0.01604684528817524,
+            -0.011054587325539086,
+            -0.0036959091138078683,
+            -0.005819853870004682,
+            -0.00203755010992417,
+            -0.02157944474979592,
+            0.005138614763345542,
+            -0.0033509138150859593,
+            -0.0024871060745482883,
+            -0.016936575573598154,
+            -0.021901830366872595,
+            0.016375173843894617,
+            0.016742770709641697,
+            -0.006606252542838758,
+            0.039771905404651496,
+            -0.004238071340064403,
+            0.00018364315349545657,
+            0.011690521707403059,
+            0.0010018392078563795,
+            -0.013651921068215563,
+            -0.010829994512513501,
+            0.012362647379788833,
+            0.011321403447882543,
+            -0.018942043512252663,
+            0.005763773608775982,
+            -0.009131565038297889,
+            0.002089216671850269,
+            0.005438887787329238,
+            -0.008183835275629478,
+            0.006690280329947951,
+            -0.01673229763262922,
+            0.015833397876364723,
+            0.00295474085300386,
+            0.003186554859191664,
+            -0.011797859525926772,
+            -0.020991927615680524,
+            0.01576413532776261,
+            0.0032403514518527287,
+            0.018819824905201635,
+            -0.026755218746060662,
+            -0.00012946467986739473,
+            -0.009552681500884924,
+            -0.0028941357661144826,
+            -0.012533800273548772,
+            0.013975640628890588,
+            0.016864739319953755,
+            0.0063941926578529,
+            0.01781016436903586,
+            -0.0026268224264609724,
+            0.015368284773773678,
+            -0.018122416427266218,
+            0.008686009180697955,
+            -0.006801074032721299,
+            -0.018537521789181048,
+            0.0013237338264105415,
+            0.027863741654435434,
+            -0.004670012662973029,
+            0.011066853115455966,
+            0.011704884649534624,
+            0.018954056680143452,
+            -0.0012626215734593707,
+            -0.0030850559460101866,
+            0.015610404665171367,
+            -0.005838226405658572,
+            -0.006861883833171456,
+            0.020872966233916948,
+            0.032951653763679145,
+            -0.006716215648910268,
+            -0.0021292599951407863,
+            0.010886596327481677,
+            -0.004901684487088035,
+            -0.008644584974417106,
+            -0.009587037829603014,
+            -0.03158137900717491,
+            0.024369089446706894,
+            0.012092412439477474,
+            -0.009601042951586109,
+            0.022690862146548786,
+            0.020752792552278052,
+            0.011412743968296708,
+            -0.010471860708195726,
+            0.02291536038589634,
+            -0.01268035142112569,
+            0.024168869865226136,
+            0.032482831775271157,
+            0.004400973556358811,
+            0.0022312906286559167,
+            0.0021175489661843985,
+            0.01691348460546639,
+            0.006050357346231155,
+            -0.007068133214675996,
+            0.011656003476409986,
+            -0.018487923180009552,
+            0.01598530891381001,
+            -0.004432366596808033,
+            0.02808371007118948,
+            -0.01337145216499331,
+            -0.010473700033368562,
+            -0.00537325376270318,
+            0.00010726431372328729,
+            0.006445701087575359,
+            -0.003324002570256215,
+            0.005255443653813165,
+            0.0007072577303252132,
+            -0.017825649146944237,
+            0.003050128130393781,
+            0.004825967170196209,
+            0.009018799597337101,
+            0.004103653497834134,
+            0.007249957401824204,
+            -0.012360355929483,
+            0.01441885664789707,
+            0.004401820363035808,
+            -0.0026190938717927086,
+            -0.0007163225760398784,
+            0.011124538295057815,
+            -0.013232822471919533,
+            -0.004895624967029031,
+            0.005281228035321008,
+            0.0059275753353475414,
+            -0.007507924998741734,
+            -0.004484140989405051,
+            0.011908076549392969,
+            -0.0037917291321648483,
+            0.006351543504726163,
+            -0.004409464890606014,
+            -0.0037760571637986827,
+            0.005303661313434895,
+            -0.00974511544836468,
+            -0.004204167898342997,
+            -0.0007093125445559502,
+            -0.006869250680233214,
+            0.00786202375969793,
+            0.005971033227117194,
+            0.010606085272785863,
+            0.006969437776542303,
+            0.004293129058563288,
+            0.004114452739123398,
+            0.004071017374828283,
+            -0.002445681623750424,
+            -0.00592480769363354,
+            -0.004617501328052899,
+            -0.012067458701076778,
+            -0.008335239153829813,
+            0.009025028998949398,
+            0.008653779421856954,
+            -0.019363018792788574,
+            0.02048011578221327,
+            -0.001188320960771659,
+            0.00772165044232104,
+            -0.01602086976290928,
+            -0.000907553574811279,
+            0.005464880629734941,
+            0.008267702703609528,
+            0.016136762227127184,
+            -0.03654209526492064,
+            0.005179791231187316,
+            -0.0037321883493524766,
+            0.016026013709955444,
+            -0.008215703411373182,
+            -0.006979816286354862,
+            0.02564060526656963,
+            -0.01450763388985563,
+            -0.0009847758228920171,
+            -0.016142304341196283,
+            -0.01407778890808498,
+            0.010815725252959506,
+            0.02050940280233426,
+            0.020684808835848383,
+            -0.0004965802897137407,
+            -0.0005781445948759235,
+            0.00036179841583004193,
+            0.022775804864342608,
+            0.015545681427566235,
+            0.018697154644607316,
+            -0.004543500782944068,
+            0.007028436664826706,
+            -0.023057215501463973,
+            0.009912581556496658,
+            0.0011795417129232596,
+            -0.008215027850908477,
+            -0.008554137076267468,
+            -0.008703378564520125,
+            -0.010448538943182258,
+            -0.011773811327190638,
+            0.004106616839960258,
+            0.013895013678116032,
+            0.004190549543250594,
+            0.0031571190600981074,
+            -0.0225385788570314,
+            -0.006243193914542516,
+            -0.011229641377928767,
+            -0.01674214043453308,
+            -0.007367389908398082,
+            -0.02386334969150212,
+            -0.028021339684039734,
+            0.0002794207370355175,
+            -0.02411861258809571,
+            -0.015958336406924612,
+            0.005268815864773021,
+            -0.01546610225139129,
+            0.00898600137695074,
+            -0.008396481690803911,
+            -0.011873486990732417,
+            0.0074743033679134105,
+            0.015336444788692,
+            -0.017676977079590463,
+            -0.010165317702640217,
+            -0.01690524990506033,
+            -0.013460451917691695,
+            0.010918550283871637,
+            0.00595960977280955,
+            0.009250157926438213,
+            -0.012447979337850305,
+            0.0062932435120863365,
+            -0.012673905639312696,
+            0.015073013552721701,
+            -0.03769166067890086,
+            -0.029992249463405965,
+            -0.00014990754540999537,
+            0.007288036630677436,
+            -0.0020735742042505426,
+            -0.018695120921334806,
+            0.0003860116505754868,
+            -0.003608995261360232,
+            -0.014900191335144973,
+            -0.013866216738266176,
+            0.025251475305524053,
+            -0.006788483107616133,
+            0.01223227424986397,
+            0.0034937277040015626,
+            0.03438333560343413,
+            0.008803852645562129,
+            0.014666206816302367,
+            -0.005709421955160623,
+            0.003742433643393657,
+            -0.0033706258426092744,
+            0.006906464031352315,
+            0.0076578888497945185,
+            0.009276592191710524,
+            0.008025321004305927,
+            0.004576470294889512,
+            0.002887346543823031,
+            -0.001925468214331082,
+            0.02125241545487307,
+            0.025671028960818562,
+            -0.00037467906648160044,
+            0.008981063192139038,
+            0.0009387900084090577,
+            0.0002735789978680719,
+            -0.008374290340790827,
+            0.006547013284767913,
+            -0.008274637814401893,
+            0.004041066530254948,
+            0.012897549078838394,
+            -0.009150863940242148,
+            -0.010211568689323908,
+            0.016575088843292566,
+            -0.012380099999574783,
+            -0.03420455578618604,
+            0.026550654680719987,
+            0.005324512466403649,
+            -0.0035670106106754245,
+            -0.010657928468872608,
+            0.0037176027853158167,
+            -0.01103891334887315,
+            0.008096915671982816,
+            0.022587659771132487,
+            -0.0013527917944322358,
+            -0.0018085171207746743,
+            0.0038503794877980586,
+            -0.011870484733648887,
+            -0.007881489342351292,
+            0.006462772034381968,
+            0.0315696514747912,
+            0.01326417653779577,
+            -0.020400369539140532,
+            -0.015223808971473513,
+            0.020406862349006738,
+            -0.023142176712111575,
+            -0.006254959806032159,
+            0.009039506093607052,
+            0.0009087520492922078,
+            -0.022766046838415165,
+            0.0009662037093312435,
+            0.004701966191595815,
+            0.005946458676746661,
+            -0.006358986495019584,
+            0.03237532341275415,
+            0.001075992257265679,
+            0.015892020271402903,
+            -0.017189453434945788,
+            0.0017514632185789299,
+            -0.014189278671121677,
+            -0.014265447388465805,
+            -0.029809460132405256,
+            0.0005557491671071693,
+            -0.0030324198417091704,
+            0.011038984948790744,
+            0.024808067315720335,
+            0.01145103533194245,
+            0.014432246996007246,
+            0.010552852947578788,
+            0.01513689173494211,
+            -0.013653722180565311,
+            -0.029009757850432732,
+            -0.005171522291984226,
+            -0.018001048743500243,
+            -0.02297482628212251,
+            0.003962613844063587,
+            0.013645125285548414,
+            0.006278662690346082,
+            0.023786381386575035,
+            0.0021738516691483163,
+            0.0009580591750940011,
+            -2.690805459518147e-05,
+            0.008894091070322921,
+            -0.004952203441302471,
+            -0.006553986197076102,
+            0.003570189040872711,
+            -0.000652714137223501,
+            0.021228360881461853,
+            -0.005758946428212041,
+            7.897153237591187e-05,
+            0.014242175492137011,
+            -0.005622256348934008,
+            -0.0035009192529791146,
+            -0.017339926143652377,
+            0.006248842389703505,
+            -0.01622011586976866,
+            0.025783086900871728,
+            0.018450783573486916,
+            -0.009109295293546267,
+            -0.013469521573468915,
+            -0.011259555201313116,
+            0.02773097452736661,
+            0.004631282554715908,
+            0.010787055220617197,
+            0.018228322680582464,
+            0.01461254259249465,
+            0.01784830193964943,
+            0.004664725683522323,
+            -0.0034339781146649384,
+            -0.0007692550982545399,
+            0.021176646860433974,
+            -0.014700178590249292,
+            -0.002701208084390198,
+            -0.008842788249773623,
+            0.0021799005046816444,
+            -0.008431299000066363,
+            0.006086447848362669,
+            -0.010793016137040172,
+            -0.015251381840165954,
+            0.0189503457652375,
+            -0.02660767353735015,
+            0.016423753600066336,
+            -0.00593016170680532,
+            -0.00482891500342298,
+            -0.027459214027897378,
+            0.010728039528784636,
+            0.0027866316191608297,
+            0.0022519974195000687,
+            0.010770472550461752,
+            -0.021427634123401408,
+            -0.01867717541189363,
+            0.008532444554010673,
+            -0.011457117350255678,
+            -0.007152301670244725,
+            -0.00348619742388509,
+            0.0154715607296319,
+            -0.00017898955187849443,
+            -0.009925365923862246,
+            0.02085093731060963,
+            -0.007175309371250024,
+            0.01223979370120031,
+            0.005256127464980655,
+            -0.008756077893734508,
+            -0.0028438320920811362,
+            0.021022708898898203,
+            0.027557382862284808,
+            -0.008342206702778956,
+            0.014526066580760486,
+            -0.011883797998224506,
+            -0.01088335352319397,
+            -0.03479444009382274,
+            0.030104838505937447,
+            -0.013305732257226255,
+            0.026773747582411633,
+            -0.0006968265979630097,
+            -0.017037729111584094,
+            -0.016206207453228583,
+            0.01625225876515654,
+            -0.0024290584023212254,
+            -0.009366287391027166,
+            0.01465885301870365,
+            -0.02428628495473643,
+            0.0058336571592899735,
+            -0.0006627935593670669,
+            -0.011531762838191602,
+            0.013275508677482622,
+            0.0019223252930710324,
+            0.010738622785461513,
+            -0.02030405805119001,
+            0.0009661595529028194,
+            0.009263716373456188,
+            -0.016791855691574455,
+            0.006057830565306032,
+            -0.03730952786847912,
+            0.0029341721588188943,
+            -0.014833895684348656,
+            0.005725615593914644,
+            0.001498755253963699,
+            -0.01820708352937343,
+            -0.03652692232407375,
+            0.00043665701245599497,
+            0.01913883585802903,
+            -0.007975988799790905,
+            0.020538595326175857,
+            0.008243569241768276,
+            0.008421404577931517,
+            0.002314738067443065,
+            0.0016702403734608475,
+            0.008279538988821465,
+            0.03679456861213399,
+            -0.01775927634148114,
+            -0.004836606118958108,
+            -0.0012151329859941935,
+            0.010644195708319258,
+            -0.02028470571116305,
+            -0.009287515726906219,
+            -0.0036506912311261536,
+            -0.013134467570201307,
+            0.030615862647585222,
+            -0.020265555441523482,
+            0.025634449009756734,
+            -0.009300770899023007,
+            0.009274072905190077,
+            -0.02097685009125915,
+            -0.009797008690754956,
+            -0.010661540100131183,
+            0.0019532291898233895,
+            0.013465005282849381,
+            -0.009151430285942593,
+            0.0071467934925125145,
+            -0.006655608635025676,
+            0.01020327561163374,
+            -0.001472597695707201,
+            -0.02443508800166349,
+            -0.0024998811551575563,
+            -0.01107321258077183,
+            0.02456164421154801,
+            -0.0019619696941303388,
+            0.0005043179699158694,
+            0.010951580163871948,
+            -0.00853593956432799,
+            0.023927336608555987,
+            -0.008855702280766362,
+            0.0010093679746665581,
+            -0.004863269000020324,
+            -0.004005691415117686,
+            0.001698212044518236,
+            0.01822024022650613,
+            -0.005707404715849356,
+            0.03722587302228139,
+            -0.01708925706528118,
+            -0.0065500772743641925,
+            -0.00015538686795361824,
+            0.010852925330001283,
+            0.010035359503147549,
+            0.008754719437760327,
+            0.004067820167375348,
+            -0.014861417626433965,
+            0.009346669946626484,
+            -0.020136029447262147,
+            0.007501669351583119,
+            0.023178490169347605,
+            -0.006840359541885465,
+            0.0006676450965357345,
+            -0.01283320739092204,
+            -0.015610538827742276,
+            0.006308543164022682,
+            0.002329420005586303,
+            -0.002295739637504597,
+            -0.02181189272979859,
+            0.006298804075737111,
+            -0.014398470126723037,
+            0.004007419942728446,
+            -0.0007497656993578918,
+            0.013863867106289847,
+            0.016451166273826513,
+            -0.006740209707090658,
+            0.039292260347522444,
+            0.01871723939316731,
+            0.009109005310505462,
+            0.005674782595856756,
+            -0.040588444759736254,
+            -0.008993850998154852,
+            -0.0010197185016393919,
+            0.004866150984083949,
+            -0.01141354216871663,
+            -0.018944881712855702,
+            -0.002646110151132781,
+            0.000308585036997592,
+            -0.010681954707313127,
+            -0.009091585604191968,
+            0.01751599002710486,
+            -0.006034424764302979,
+            0.00872057856926618,
+            0.004146961781109887,
+            0.023758065635807138,
+            -0.009727802672285544,
+            0.01176257119851645,
+            -0.0071822083765114165,
+            -0.005456197751568526,
+            0.0006726765703478756,
+            0.011011628907288813,
+            -0.008031792262886802,
+            0.009763340867550022,
+            -0.004279891138185881,
+            0.008355720233269893,
+            -0.008552596384067825,
+            -0.023952129458400563,
+            -0.014972827087541732,
+            0.014347535404251018,
+            0.011651260496709961,
+            0.008137738462704715,
+            -0.013624435569539285,
+            0.0031728721674035352,
+            0.00675830277744976,
+            -0.009184784787581942,
+            0.009161771311910335,
+            0.017733082848622708,
+            -0.001411542568472906,
+            -0.0017830167499636113,
+            -0.0016130003349297027,
+            -0.012173528776899308,
+            0.0011196437753239162,
+            0.02284887358189995,
+            -0.003054479643937998,
+            0.014614283913976716,
+            0.010689486748036806,
+            0.009643292262102645,
+            -0.00810930837603355,
+            0.001915955511174678,
+            0.02341863048264366,
+            0.015022291783105098,
+            -0.004625715884342598,
+            -0.011422973894623295,
+            0.004259757717597968,
+            -0.010332744704689279,
+            0.007373328473086509,
+            -0.02117940113144938,
+            -0.004672698999328004,
+            0.008896311380763048,
+            -0.007028862917603513,
+            -0.01732432285099198,
+            -0.028026173555055017,
+            -0.016765805892242715,
+            0.019819329380181,
+            0.0027213760678977965,
+            -0.013291251876048186,
+            -0.0032919084721662363,
+            0.0031214883337300926,
+            0.002553005168048115,
+            -0.0071104806975682885,
+            0.005310583052470149,
+            0.007359183453835734,
+            -0.008674133980486269,
+            0.017665958608281126,
+            -0.000521269431423141,
+            -0.0008020251454616109,
+            -0.0005442374261144659,
+            0.0006127687598711624,
+            -0.016387752806126442,
+            0.01666161912459431,
+            -0.010949228376419892,
+            0.015198110119816624,
+            -0.008966726991404666,
+            0.008600499511690635,
+            -0.007617043351740469,
+            0.009458641139070436,
+            0.011597957400445055,
+            0.012755037944648426,
+            -0.004403680820063604,
+            0.008519241473418196,
+            0.004807816191718704,
+            -0.008147125240985166,
+            0.007017601015398136,
+            0.01296082929962557,
+            0.00582353932160707,
+            0.0013634237970372698,
+            -0.0008246894322793264,
+            0.0022035670958705576,
+            -0.002480976219285683,
+            0.0056293125704658135,
+            0.012714850523393719,
+            -0.002497639449055006,
+            -0.007135683201188999,
+            0.014988249771676699,
+            0.014955895630183352,
+            -0.022994948923833993,
+            0.007121634434082677,
+            0.005835971825639792,
+            -0.0030507120821557008,
+            0.002145266953387315,
+            -0.027334525318320587,
+            -0.020359474070022213,
+            0.01901995970350808,
+            -0.015260454474605018,
+            0.02850460517419317,
+            -0.011624138792780823,
+            -0.02284242513728605,
+            -0.01277489645734181,
+            -0.004209529633534193,
+            -0.022684500849916897,
+            -0.004894414098943742,
+            0.01919165657366256,
+            0.013401560972810082,
+            -0.004313549020265677,
+            -0.019089369171102138,
+            0.03863248408555474,
+            0.010497951840980463,
+            0.03508157919687289,
+            -0.008597970944278045,
+            0.0023354831464448313,
+            -0.01563401808449416,
+            0.015551176593436512,
+            0.001075882802133782,
+            -0.011575692834796734,
+            0.01505716108583101,
+            0.031836226915668404,
+            0.0034740889237207494,
+            -0.00617206855371193,
+            0.0018420207148235309,
+            0.00418433325800744,
+            -0.0008167766650900381,
+            0.011160989178055499,
+            -0.018497472318034093,
+            -0.011951262348399922,
+            0.0013093376768780122,
+            0.00216347903176653,
+            0.0059233460419514045,
+            -0.001778091863744448,
+            -0.013317639704511549,
+            0.011433308892643204,
+            0.011596810919469926,
+            -0.01518143105525228,
+            -0.007497373860338335,
+            0.0050957665991362405,
+            -0.018974405317819656,
+            0.01872472902494418,
+            0.011859522398789263,
+            0.000792413325332956,
+            -0.0006503947129906383,
+            0.00808635755624928,
+            0.003749813548995223,
+            0.0033657799335872368,
+            -0.007862215303617144,
+            -0.0028014669075116876,
+            0.0009892310856576799,
+            3.834379772674841e-06,
+            0.015402211364447726,
+            -0.015264893397161102,
+            0.02488855366696587,
+            -0.0025643639525691083,
+            0.00953993868921819,
+            -0.017460590918773188,
+            -0.0017911712931080155,
+            -0.00402374787529868,
+            0.01838051670627292,
+            0.006753072393532114,
+            -0.010130676216186492,
+            0.004921510943896829,
+            0.007975201081296845,
+            -0.012589786869771252,
+            0.00018319054705558588,
+            -0.0022606180909706688,
+            -0.014791528072476771,
+            -0.0031171076370692706,
+            0.017873825561816544,
+            0.007593688317551089,
+            0.0037375070591255375,
+            -0.015863143964928172,
+            0.007192324815779268,
+            0.007522990667369163,
+            0.024504875417877527,
+            -0.026309816223960274,
+            0.024779042752840836,
+            -0.022104087334088543,
+            0.02165781034613893,
+            0.0319051741666687,
+            0.007538146369110045,
+            -0.01690181186708103,
+            -0.006595947249169521,
+            0.006373991305923002,
+            -0.023410814842488123,
+            -0.005494017994983188,
+            0.016903736810723716,
+            -0.01792400152372258,
+            0.000330865061689199,
+            -0.014660413003063089,
+            0.0040825880154611895,
+            0.008248037865838212,
+            0.011540333451799686,
+            -0.0033578830043738085,
+            0.018327525495676525,
+            0.020171745352159914,
+            -0.01138759854371235,
+            -0.0004221149053107702,
+            -0.022321409200941777,
+            0.005046865873981027,
+            0.022509643273449936,
+            0.001111272240506322,
+            -0.009540467231190513,
+            0.018376261423745193,
+            0.008480532791922674,
+            -0.020801891594934313,
+            0.008436374227753589,
+            -0.036175982655998785,
+            -0.010454118523313122,
+            -0.009328507118928412,
+            -0.001218633166683384,
+            -0.004142773602181115,
+            -0.0002494371580772723,
+            -0.011365548842513265,
+            -7.699548642315061e-05,
+            -0.006889339420877534,
+            -0.0037173070609413474,
+            0.014953978004522174,
+            0.011678957032329096,
+            -0.003501047405385249,
+            -0.0073984932581048915,
+            0.008493128567102267,
+            0.006163937693564917,
+            -0.017034696006318337,
+            -0.013268418502042576,
+            0.0037122128689059085,
+            -0.020067656401141467,
+            0.005148329575092478,
+            0.00459729178556505,
+            0.014031669598917171,
+            -0.01849235806598133,
+            -0.009298127665365717,
+            0.019716512208448042,
+            0.0094466405158553,
+            -0.0008988916054855498,
+            0.004264260983102956,
+            -0.00032312548009733906,
+            0.002279987803645912,
+            -0.0008902345374346296,
+            -6.145528806973222e-06,
+            0.003602229465148649,
+            0.012503366799381277,
+            0.007759393855482671,
+            0.03694402922447051,
+            -0.016692148708641812,
+            0.005873765863350649,
+            -0.014187609702966002,
+            -0.0037592517335595654,
+            -0.00749939010396784,
+            -0.0004308157309481927,
+            -0.0022282234382577843,
+            0.0021999996319642854,
+            0.0052176773733353925,
+            0.009353171029508977,
+            0.002155405440765206,
+            -0.00823565584692709,
+            0.009691447250255694,
+            -0.015228173095533029,
+            0.003853858061067531,
+            -0.008207910291425473,
+            0.01017972550354623,
+            -0.009695280827515048,
+            0.0028705587799331246,
+            0.03849358164875786,
+            0.01130021300937487,
+            -0.004934175853966098,
+            -0.014300147534361459,
+            -0.0007796051618993811,
+            -0.01737877501558196,
+            -0.008579119859817895,
+            -0.005434792488602325,
+            -0.02617338653902575,
+            -0.00838916510137031,
+            0.015344260464413648,
+            0.011202567142310492,
+            0.014124853863170132,
+            -0.023894182185855824,
+            0.008897943489322673,
+            -0.03001670823127986,
+            -0.011951507895140876,
+            -0.001530967140156163,
+            -0.015467812535428609,
+            0.0010110999991167703,
+            0.023623196504289838,
+            -0.007762735863648054,
+            0.012654652822750325,
+            0.012867958170258448,
+            0.018777522196571934,
+            0.006390986057157016,
+            0.00038045736974183937,
+            -0.009444911783760569,
+            0.009534310165417939,
+            0.008806031651319704,
+            0.0080954543713255,
+            0.013161833638467553,
+            -0.0022840529693446553,
+            -0.018060162439023572,
+            0.005469288590000856,
+            0.014354475778235131,
+            -0.0033504276914600585,
+            -0.001615191819057537,
+            -0.011088493180661976,
+            0.016039040565630432,
+            -0.023559046333872156,
+            -0.009482398794557168,
+            0.009918576960319548,
+            -0.0027308309155029842,
+            0.009212063547988036,
+            0.0160181200081126,
+            -0.01881373001297218,
+            -0.0024168618343039703,
+            0.016244889977796682,
+            0.0024104546308670025,
+            -0.00022606425578303413,
+            0.016792244238218245,
+            -0.0007616857360728623,
+            -0.004279120886828465,
+            0.00997764266310123,
+            -0.0075631917138318655,
+            7.3878137053128405e-06,
+            -0.002416364915008295,
+            -0.0028253498168087964,
+            0.02054128727706496,
+            -0.021281801678871736,
+            -0.005379259148621041,
+            0.02257306103533458,
+            0.015687105622685975,
+            -0.007843745111084413,
+            6.258498042855072e-05,
+            -0.0219506172079994,
+            0.0019925886885244454,
+            -0.007768133999661569,
+            0.016524850592507592,
+            -0.007957949830060354,
+            0.0030085044673901665,
+            -0.01361024620290394,
+            0.011563781350927133,
+            -0.01277407169098418,
+            0.00873716215529134,
+            -0.002986943948333218,
+            -0.017567646684567492,
+            -0.0053109390854215125,
+            0.004561236136619266,
+            0.0014045175130801308,
+            0.044958041347040185,
+            -0.005539136232703934,
+            0.003729517522976541,
+            0.018086335927383804,
+            -0.014199453349727933,
+            0.02079719360289355,
+            0.016663293302592568,
+            0.00991445855219406,
+            -0.00033967370741635794,
+            -0.0018726748849858836,
+            0.023373354806639515,
+            0.013258226441642464,
+            0.00861986998150065,
+            -0.0014340218811281975,
+            -0.008516541227326329,
+            0.018239230461345617,
+            0.005458629943667832,
+            -0.012421896028975013,
+            0.012732721349758832,
+            0.013559877132508506,
+            -0.013720939472355347,
+            -0.011545210834395917,
+            0.01616358223872008,
+            -0.019904594319155373,
+            -0.018260016571763644,
+            -0.0036124537591368864,
+            -0.01614510591132948,
+            -0.03864699384205461,
+            0.01619401261888991,
+            0.018351607973242735,
+            0.006666313000123845,
+            0.013647410298896132,
+            0.002597680061427196,
+            0.004545530267244157,
+            -0.018531082430861854,
+            -0.0077618625909995275,
+            -0.008085144524135461,
+            0.003744735923004248,
+            0.011059457364156674,
+            -0.011335432904114519,
+            -0.009514379703581616,
+            -0.00034500702695418285,
+            -0.007433132060989644,
+            0.01971420270994951,
+            0.0020753610290475642,
+            0.0034506706065161024,
+            -0.00628437298309269,
+            0.017966485854173167,
+            0.026789678964152908,
+            -0.007532446825077476,
+            0.021886182869092073,
+            -0.002864230253056434,
+            -0.0015870360120997492,
+            -0.014393519359298496,
+            -0.0014439412842964876,
+            -0.011527182213472351,
+            0.0005453493437517964,
+            -0.0006791481533636673,
+            0.014093767997869645,
+            -0.03288824642147967,
+            -0.003957301296796813,
+            0.014444592778851293,
+            0.007538052312876599,
+            -0.005057755472988024,
+            0.011357919882949903,
+            -0.002964451028327082,
+            0.016977317604413924,
+            0.0075201055190542146,
+            0.023391646079994784,
+            -0.016588823248397513,
+            -0.004504858318770565,
+            0.006660877064954993,
+            0.0034631255931793638,
+            0.0014484203909961972,
+            0.01912983041187014,
+            0.023439941258581887,
+            -0.014715886385460541,
+            -0.0013596123366001458,
+            -0.0029020372473879614,
+            -0.021414221343490594,
+            0.014976806123253727,
+            0.0074729481304182865,
+            0.010967351226114886,
+            0.009806663105020302,
+            0.015465076881663691,
+            -0.02109587527354609,
+            0.0011671121739822202,
+            0.020567458365051382,
+            -0.008618333549105003,
+            -0.018815706906171112,
+            -0.009804829158883123,
+            -0.00966828714719191,
+            -0.006078893678150302,
+            -0.00803783313451146,
+            0.005523023167028105,
+            0.0022647506733877573,
+            0.01652997604669998,
+            -0.008204200434441975,
+            -0.008404609147866811,
+            0.008489247125986078,
+            0.007546785673285454,
+            -0.0033093451283570316,
+            0.010614050963656496,
+            0.0030633179859393587,
+            -0.006940540318011839,
+            0.00894884343338777,
+            -0.002787816848430143,
+            0.014347598920245487,
+            -0.02441769157218451,
+            0.004717742147185283,
+            -0.012096876825337124,
+            0.006405151160675699,
+            0.0041568790527989905,
+            0.0016702066682613428,
+            -0.0041159211459004,
+            0.0035217362856754353,
+            0.0008893479210743541,
+            -0.07433312151926533,
+            -0.012629681414004524,
+            -0.0035797012285864486,
+            0.033657838713146884,
+            -0.019695806809053914,
+            0.004477927056336216,
+            0.011966065644382396,
+            0.0096717896523898,
+            0.003070713232538593,
+            0.006136726101157844,
+            0.006112385569942848,
+            -0.0245258242481677,
+            -0.00015284418977092615,
+            0.004821040694679944,
+            0.0020347004045943454,
+            -0.007532408630122127,
+            0.00750565079228405,
+            0.015327710245879296,
+            0.020989926895388523,
+            -0.010970305087289488,
+            0.017227663358885748,
+            -0.003352718496981928,
+            -0.006068892126250978,
+            0.0014285750322333686,
+            -0.010153799722483554,
+            -0.017307415843726394,
+            -0.006495008086641125,
+            0.0068983699213297075,
+            -0.001982568363667046,
+            0.0044032687939897855,
+            0.003333596495330213,
+            -0.0032171586993948696,
+            0.0072365127663633925,
+            0.013920326463272646,
+            0.006415916101287133,
+            -0.013675562796300214,
+            0.003003782758807665,
+            -0.02500375035312058,
+            0.011945988879567646,
+            -0.012708975181025296,
+            -0.0009488161508767279,
+            -0.0047093877027523014,
+            -0.0035750724825159503,
+            -0.002213435677949806,
+            0.02043567538125188,
+            -0.014304009033474186,
+            -0.01802899775650995,
+            0.008187227768196673,
+            -0.011217013195839762,
+            0.008300740149622635,
+            -0.03382831122769974,
+            0.012386335491333264,
+            -0.0276029934382802,
+            -0.0002243752790856292,
+            -0.004521387639042107,
+            -0.0010631721066129095,
+            -0.009491184157278669,
+            -0.03274559774619223,
+            0.015845705808609598,
+            -0.007204751554079551,
+            -0.020338337341085228,
+            0.0007143424225187802,
+            -0.021267094083674304,
+            0.014150458196361147,
+            -0.01851165553224022,
+            0.0033450353123933058,
+            -0.018775439467431685,
+            0.002782289821140401,
+            -0.0018711350216089556,
+            0.010502195116579441,
+            -0.00557814442809019,
+            -0.0007130962242755068,
+            -0.013072988887382269,
+            -0.002121841212381393,
+            -0.006653444103331213,
+            0.009394907690516121,
+            0.0028678074167728367,
+            -0.009601577734902932,
+            0.011153571864462314,
+            0.013046116892519414,
+            0.019812860521973938,
+            -0.00539047787915588,
+            -0.00555139958411929,
+            -0.011553689090179406,
+            0.009852063270798534,
+            -0.016326212706634428,
+            -0.00827402913960903,
+            0.008982303148984887,
+            0.015216947130173639,
+            0.01026889105233137,
+            0.003352028462252994,
+            0.003150801750574548,
+            0.002531620731961765,
+            0.026744691368233237,
+            -0.0015120532859309638,
+            0.0008324701960729416,
+            -0.014508988496009931,
+            0.022127525293249926,
+            0.02772430884375744,
+            -0.0012206102581726523,
+            -0.00018859639594030037,
+            0.017437360169132967,
+            -0.004697052522250805,
+            -0.0038111184371524645,
+            0.024015246959315593,
+            -0.038396108183338655,
+            -0.008416825232533771,
+            -0.006788816882617399,
+            -0.006043428198944214,
+            -0.009587366247924467,
+            0.007294190273465812,
+            0.00041966080769004275,
+            0.0024709692753547823,
+            -0.0005250904505954556,
+            0.03452858797518386,
+            0.003403515654095309,
+            -0.0003392060101496309,
+            -0.0032980864446307116,
+            0.0019125121221204029,
+            0.0032987517853214346,
+            -0.009116761755779728,
+            -0.038030085122772624,
+            0.006006334000396066,
+            0.007932785488877407,
+            0.00975009749364395,
+            -0.008120121513849434,
+            -0.008864461019882246,
+            0.004909348738798099,
+            -0.02273129283865203,
+            -0.01372636307167243,
+            0.02625229144362647,
+            -0.008523028986127237,
+            0.029005877324974557,
+            -0.022435059061182174,
+            -0.013135836801818153,
+            -0.004126433322528265,
+            0.010283975013795537,
+            -0.019732856462444506,
+            0.004343860338939202,
+            -0.004022387065239562,
+            0.023733721026005954,
+            0.0077578144626961245,
+            -0.008371631446146718,
+            0.012386735614153616,
+            0.0064789380436815714,
+            -0.0026638642827013837,
+            0.00136333052963848,
+            -0.007209517067748103,
+            0.016916505654836485,
+            -0.011819784026452944,
+            -0.005833909403097991,
+            0.008977527508585844,
+            -0.0024556434204852946,
+            0.02509293948918158,
+            -0.0237762245071987,
+            0.005288171058938098,
+            -0.00019047961064753207,
+            0.01822542471604037,
+            0.008127347078975925,
+            0.008361588667386882,
+            -0.011835407590060994,
+            0.010004482616697591,
+            -0.03378107067882622,
+            -0.015433906677849963,
+            -0.007702910250795191,
+            -0.0013877702264578865,
+            -0.002059152251387601,
+            -0.015745218548968338,
+            -0.026393221735795527,
+            0.006294583740809379,
+            0.016267525575472445,
+            0.023291528020980265,
+            -0.004683197930892445,
+            0.012912830736650997,
+            -0.002831583991769044,
+            -0.008904271394710672,
+            -0.011441882849296418,
+            -0.012202629605208118,
+            -0.012470773094383122,
+            -0.014560694782340029,
+            -0.017045701603711442,
+            0.012139665577738337,
+            -0.020079061996873537,
+            0.013911518328924777,
+            0.0347210775522539,
+            -0.017176114991595492,
+            -0.0018941772168839354,
+            0.019702722555622765,
+            -0.004221469612316137,
+            0.01212088353057276,
+            0.0063707489618353725,
+            0.015298014532351866,
+            0.0038643174573634725,
+            -0.012181479717826009,
+            -0.007374005740809251,
+            0.019411457664902376,
+            -0.004902591844710158,
+            0.014591446725717338,
+            0.015069599884557399,
+            -0.008675672164385891,
+            0.002544198867387345,
+            0.01951865904496276,
+            -0.01762101777798757,
+            0.01665654707715257,
+            -0.00458402229480694,
+            -0.009936206685538294,
+            -0.013345740581684638,
+            -0.005336416724426228,
+            -0.019469457091169776,
+            0.0061608454109892425,
+            0.018931676738452464,
+            -0.00533013594306834,
+            0.009310871143727683,
+            -0.0037670793158045,
+            -0.005045151384034014,
+            0.0002041909886640482,
+            -0.04152751596227068,
+            -0.008340911336217609,
+            -0.0006651584370005978,
+            -0.02505748815737022,
+            0.025954102004571416,
+            0.007205602377144382,
+            0.022602344303400163,
+            -0.017168610245491406,
+            0.004459735690564842,
+            -0.005882024209102714,
+            0.013217517324383834,
+            0.02255689669623969,
+            0.0017984057469785134,
+            0.005699128497215882,
+            -0.02353849721739205,
+            -0.0034711576718386185,
+            0.015924913374553738,
+            -0.003604478141616307,
+            -0.0034993903155063817,
+            0.012592308877056896,
+            0.012864938348003884,
+            0.01567418642605425,
+            0.008801753550506258,
+            0.015606443428689292,
+            0.010589933994371628,
+            0.008928721556432368,
+            -0.005805129999077126,
+            -0.001025103064751441,
+            -0.015907755521365925,
+            0.0045874330999415985,
+            0.031701049450012876,
+            0.012594483199773746,
+            0.016783858873835704,
+            -0.01778536268185936,
+            0.00789523134933153,
+            -0.0012706411671644011,
+            0.008325108613530198,
+            -0.001677173669891956,
+            0.029540136288412976,
+            0.011739999605699826,
+            0.015670966746894652,
+            0.008732334090105053,
+            -0.016237731209426752,
+            0.045334690225148,
+            -0.016911928942220152,
+            -0.0021139311364009113,
+            -0.01058970056572469,
+            -0.021705338531028618,
+            0.007324792557651635,
+            -0.006556582104686972,
+            0.0018243188317438144,
+            0.010780292738415671,
+            -0.006471850255151019,
+            0.0006997395548596146,
+            0.003907433192252573,
+            -0.023129313269763224,
+            -0.002550875642832235,
+            -0.02237917712764273,
+            -0.003107775639788617,
+            -0.02621608419628401,
+            -0.00012482032027919036,
+            -0.016207377503696517,
+            -0.0038712739995960753,
+            -0.023315508569288476,
+            -0.014963211090562272,
+            -0.01633110043829169,
+            -0.004241081909202262,
+            -0.008295402942465353,
+            0.004976396647108597,
+            -0.007402107755970892,
+            -0.020321297548927468,
+            0.0020986169430507757,
+            0.016803273183070004,
+            0.01434729320047051,
+            0.018612970083178984,
+            0.00603868809307069,
+            0.014278001198851581,
+            -0.0026350854954985117,
+            0.017171016209107656,
+            -0.006968968261909459,
+            0.006881847166565678,
+            0.011454001418121107,
+            0.02140935294963631,
+            0.011121833270235448,
+            -0.012801756637470525,
+            0.0010870041670931706,
+            -0.009655388034828094,
+            0.002094601818569396,
+            -0.00828486034340067,
+            -0.009190975158350585,
+            0.0017762581946855365,
+            -0.007261996868906865,
+            0.004674405202069334,
+            0.0030526253833799426,
+            -0.0014082232541965316,
+            0.010419302200840906,
+            -0.0024200053377751805,
+            -0.012208685875688886,
+            -0.0033037330470612695,
+            -0.017975940046938498,
+            -0.006958042903103732,
+            -0.0014496522238488239,
+            -0.011711502944602638,
+            -0.01016130418405875,
+            0.003321436189059608,
+            -0.00538282506552848,
+            -0.009353581612407435,
+            0.013884147453136417,
+            0.0001309738696853529,
+            0.0005878330075918333,
+            0.023495280602974504,
+            0.006035028203642483,
+            0.008087594627921345,
+            0.013144288945385904,
+            -0.01023995000994454,
+            -0.022574958230635715,
+            -0.0243435727055662,
+            0.007867022215096418,
+            -0.01573478573423379,
+            0.02324699908478915,
+            -0.007929809922043611,
+            -0.00561630876431978,
+            0.0020982723335057625,
+            -0.013699317926847062,
+            0.008578431496569186,
+            -0.011271381061801356,
+            0.0014450645788554738,
+            -0.019581451799757368,
+            0.018059311800267413,
+            -0.010917911869373693,
+            -0.004302687883018328,
+            0.009466103226841532,
+            0.034049086327052606,
+            -0.0005689605174080737,
+            -0.007410970644432269,
+            -0.006594150877740108,
+            -0.00022810271313786974,
+            0.018766765773409123,
+            -0.0029041815756733333,
+            0.005758781908059541,
+            -0.01563519107805783,
+            0.006571628287171105,
+            -0.006132372252825422,
+            0.0012423261344297225,
+            0.01513074831647933,
+            -0.026306320707079913,
+            -0.015491710256868008,
+            -0.007857352164317122,
+            -0.024893430640551818,
+            0.002253574511878983,
+            0.017579477884206534,
+            0.020505606775521504,
+            -0.020324505029926956,
+            -0.02513332780748072,
+            0.005150966733930881,
+            -0.026580994694344065,
+            0.0037390577082486594,
+            0.1241031878517593,
+            -0.002138404898309707,
+            0.002720723037908521,
+            -0.011425036886011179,
+            0.0022399461009316216,
+            0.005389407121650262,
+            -0.019552543955456567,
+            0.05950746473186985,
+            -0.00638023587296763,
+            -0.004023798021748334,
+            0.002170545188776296,
+            0.001785855390676113,
+            -0.005831534307818096,
+            0.008643501093778434,
+            -0.017602787281866816,
+            -0.02184792441891879,
+            -0.0052215705748838905,
+            -0.021252169339074325,
+            -0.008509621422071373,
+            -0.013000137524907565,
+            0.002893880203681282,
+            -0.015821609145923173,
+            0.010397692426614709,
+            -0.01638916243796692,
+            -0.010283706937701179,
+            -0.01103621454761223,
+            -0.00373552043529251,
+            0.01563553822905027,
+            -0.03188533919653949,
+            -0.004873550301425776,
+            -0.005183590054152726,
+            0.018218971950313273,
+            0.006004328765748325,
+            0.004357734370247252,
+            -0.014230318971458851,
+            0.0064792070801456235,
+            0.0215589173714926,
+            0.0055272493764401305,
+            0.014643335833970546,
+            0.010618070641357003,
+            -0.0022177303813575846,
+            -0.0024402761885758007,
+            0.017541799738501007,
+            -0.005445774414228888,
+            -0.00023502936119626386,
+            -0.00537569046685066,
+            -0.005974703247446444,
+            -0.010712244203425036,
+            0.01843613166635215,
+            -0.009938564643456672,
+            -0.017388592763277214,
+            -0.0015797412875366156,
+            0.02028434083164951,
+            -0.012761534382003973,
+            0.0031906812114950982,
+            0.000788436210275669,
+            0.020589196433948156,
+            0.0007726855314648301,
+            -0.005481260996289263,
+            0.007165014826294827,
+            -0.02956334397512344,
+            0.004095715232997753,
+            0.0003568512047626187,
+            0.027290768799658912,
+            -0.0017631129386615335,
+            0.01552542586826711,
+            -0.020871286597113327,
+            0.004388125824324094,
+            -0.016926008631435403,
+            0.005099899758322398,
+            0.012560923968841436,
+            0.002781049890810361,
+            0.010889021753593997,
+            -0.008500358897706648,
+            -0.021353460542554444,
+            0.012663934009161902,
+            -0.018657094130860155,
+            -0.029156017587920958,
+            0.0018889388761887053,
+            -0.028158076315864772,
+            -0.003048854609750186,
+            0.006812072971079798,
+            -0.035217412591558014,
+            -0.02251675180883659,
+            0.007396524512996277,
+            -0.00995571600201846,
+            -0.011912753666073882,
+            0.010675240017836214,
+            -0.02645155743041798,
+            -0.018137648527851668,
+            0.0010666944556449141,
+            0.012588800077165337,
+            -0.004561825939541096,
+            0.007555558090958502,
+            -0.028558603070876605,
+            0.014816816621445352,
+            -0.01313764261744744,
+            0.016036065063831813,
+            -0.0037797584923144134,
+            0.022217516257304868,
+            0.011131597210722573,
+            0.008795376259031849,
+            -0.004950080558710245,
+            0.021409590863550987,
+            0.005574351961953293,
+            0.051358937509801736,
+            0.031972009383746405,
+            0.0019821003238211276,
+            -0.003251257665378234,
+            0.003313045083311649,
+            0.008294776423730887,
+            -0.00940948257229233,
+            0.005114631360856872,
+            0.028053227377164806,
+            0.016738013668710083,
+            -0.018755924318109216,
+            0.006606711923035118,
+            -0.02702692774986498,
+            0.017950884766781056,
+            -0.022054653845163,
+            0.0003755944703013671,
+            0.007766253634286795,
+            0.0016206993755471613,
+            0.010682347575070117,
+            0.0013842930079185415,
+            0.01705234255261425,
+            0.023975191051210525,
+            -0.002976402980885077,
+            0.017466027107477605,
+            0.012734909275073793,
+            0.018143161773162015,
+            -0.0038399373197265827,
+            0.006929440921265761,
+            -0.012003095293274786,
+            -0.0008883470597690258,
+            0.0020490527756307267,
+            -0.004231091383919565,
+            0.012128848448049833,
+            0.008340302519160299,
+            0.012989849936937843,
+            0.002974488532907779,
+            0.003198854381461613,
+            -0.005309753760578859,
+            -0.0017024982049936384,
+            -0.01763395035509857,
+            -0.01819680226734474,
+            0.028417724274372468,
+            -0.020217879775485576,
+            -0.050118184195774004,
+            0.0025509548991664425,
+            0.014185761006969567,
+            -0.0012021669304388173,
+            -0.0051260576846881515,
+            -0.0005774778872927327,
+            0.005718864494530703,
+            -0.016918749210038116,
+            -0.0015472638835434902,
+            0.008125845907627804,
+            0.020347547062899797,
+            -0.002036218493206914,
+            -0.01079739518866248,
+            0.0038112958713600906,
+            0.02169339303105242,
+            0.01667594322285835,
+            0.012338193092567552,
+            -0.009798409233733167,
+            0.0003198215558702285,
+            -1.6308740825163337e-05,
+            0.0076870827874737325,
+            0.004356644356656038,
+            0.003500023005601613,
+            -0.019968549869439228,
+            -0.008634740687755112,
+            0.012781565963545847,
+            -0.0025637758121512927,
+            0.010243567793632335,
+            -0.00042296044205132426,
+            -0.003069251553118065,
+            -0.009344199449486244,
+            -0.004694953215957636,
+            -0.009583993744231791,
+            0.013783878036521561,
+            -0.007705982042252044,
+            -0.01549567538665377,
+            0.00020666298495823114,
+            0.0005515775451348472,
+            0.001667367278489879,
+            -0.004518431526236256,
+            0.03302324454129141,
+            0.009531990824506103,
+            -0.0004937348108340977,
+            -0.004047613286672229,
+            -0.0327383112522095,
+            0.012456150771224677,
+            -0.009677533214778067,
+            0.0006455811519981003,
+            0.020012976827426605,
+            -0.00020277620481825933,
+            0.002168432791818704,
+            -0.015657551629104767,
+            0.0032924357559238502,
+            0.006161295735841883,
+            0.013814165416788555,
+            0.0005164426482972658,
+            0.01076512534105537,
+            -0.006498977875968572,
+            -0.010325144017898257,
+            0.004804158067334203,
+            0.0104064855510473,
+            -0.0012188100892986933,
+            0.005171197790296134
+          ],
+          "title": "Mixture-Of-Experts With Expert Choice Routing",
+          "keyphrases": [
+            "Mixture-Of-Experts",
+            "Expert Choice Routing",
+            "sparsely-activated models",
+            "training efficiency",
+            "downstream performance"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "bd90d79e-2183-4ea8-ad44-6b8ece583bae",
+        "properties": {
+          "page_content": "6 Limitations\n\nThe expert choice method might not immediately apply to auto-regressive text generation as our current implementation takes in the past and future tokens to perform the top-k selection. One possible solution is to collect a large batch of input sequences, dispatch tokens of the same sequence into separate groups, and perform expert choice routing for each group. Another scenario where the expert choice method does not immediately apply is when the batch size becomes very small during serving or inference. A global top-k can be selected instead and we can cap the number of times each expert or token gets selected. We leave these possible improvements for future work.\n\nAnother long-standing issue with MoE has been the large memory footprint. Even though computational cost can be reduced using sparsely gated networks, the total number of parameters increases linearly or sub-linearly with the number of experts. Increasing the number of experts requires reservation of a large number of hardware devices. Therefore, dynamic (used) power is saved while static (reserved) power is not. Power saving techniques such as the ability to put hardware devices into low power states while not in use [17] can help with reducing the reserved power requirements.\n\nReferences\n\n[1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pages 3930–3939. Computer Vision Foundation / IEEE, 2020.\n\n[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. In Advances in Neural Information Processing Systems.\n\n[3] Kyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning, 2014.\n\n[4] Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems, 2021.\n\n[5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.\n\nTransformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.\n\n[6] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 933–941. JMLR.org, 2017.\n\n[7] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixtureof-experts, 2021.\n\n[8] Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan.\n\nTricks for training sparse translation models, 2021.\n\n[9] Richard L Dykstra. An iterative procedure for obtaining i-projections onto the intersection of convex sets. The annals of Probability, pages 975–984, 1985.\n\n[10] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.\n\n[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. NAS-FPN: learning scalable feature pyramid architecture for object detection. In CVPR, pages 7036–7045. Computer Vision Foundation / IEEE, 2019.\n\n[12] Sam Gross, Marc'Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision, 2017.\n\n[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.\n\n[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 630–645, Cham, 2016. Springer International Publishing.\n\n[15] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2016.\n\n[16] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.\n\n[17] Ping Huang, Zuocheng Xing, Tianran Wang, Qiang Wei, Hongyan Wang, and Guitao Fu. A brief survey on power gating design. In 2010 10th IEEE International Conference on Solid-State and Integrated Circuit Technology, pages 788–790, 2010.\n\n[18] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 103–112, 2019.\n\n[19] Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David A. Patterson. A domain-specific supercomputer for training deep neural networks. Commun. ACM, 63(7):67–78, 2020.\n\n[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.\n\n[21] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.\n\n[22] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6265–6274. PMLR, 18–24 Jul 2021.\n\n[24] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. New York, NY, USA, 2019. Association for Computing Machinery.\n\n[25] Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, Cédric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. In ICLR. OpenReview.net, 2021.\n\n[26] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.\n\n[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.\n\n[28] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models, 2021.\n\n[30] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 10435–10444, Red Hook, NY, USA, 2018. Curran Associates Inc.\n\n[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E.\n\nHinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In ICLR (Poster). OpenReview.net, 2017.\n\n[32] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 10–15 Jul 2018. [23] Min Lin, Jie Fu, and Yoshua Bengio. Conditional computation for continual learning, 2019.\n\n[29] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.\n\n[33] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.\n\n[34] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc.\n\n[35] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.\n\nGLUE: A multi-task benchmark and analysis platform for natural language understanding.\n\nIn Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, November 2018. Association for Computational Linguistics.\n\n[36] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake A. Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen. GSPMD: general and scalable parallelization for ML computation graphs. CoRR, abs/2105.04663, 2021.\n\n7 Checklist\n\n(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Yes (b) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes (c) Did you discuss any potential negative societal impacts of your work? N/A. Not any. (d) Did you describe the limitations of your work? Yes (a) Did you include the code, data, and instructions needed to reproduce the main experimental results? Yes. We include details in the experiment setup to help reproduce the main results. (b) Did you specify all the training details? Yes (c) Did you report error bars? Yes (d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? Yes (a) If your work uses existing assets, did you cite the creators? Yes (b) Did you mention the license of the assets? No. The used dataset is not released yet. (c) Did you include any new assets either in the supplemental material or as a URL? No. The dataset is not released yet.\n\n(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? No. Not using persons' data. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes. The dataset does not contain any personally identifiable information or offensive content.\n\n",
+          "title": "6 Limitations",
+          "embedding": [
+            -0.005417884327471256,
+            -0.002057158388197422,
+            -0.02763829380273819,
+            -0.01599692367017269,
+            0.03654416278004646,
+            -0.02751435525715351,
+            -0.03312699869275093,
+            0.03036494180560112,
+            0.005386899691075087,
+            0.039306219667196274,
+            0.025407399982213974,
+            -0.014332606457173824,
+            0.02131742797791958,
+            -0.005431163590401411,
+            -0.0021988023072481155,
+            0.018927184864878654,
+            -0.01384570449590683,
+            0.01158825308084488,
+            0.00927768461406231,
+            -0.02854127436876297,
+            -0.0014927954180166125,
+            -0.016377592459321022,
+            -0.013624385930597782,
+            0.010021315887570381,
+            0.019777048379182816,
+            0.013535858131945133,
+            0.02400866523385048,
+            0.020803967490792274,
+            -0.02866521291434765,
+            0.00207154406234622,
+            0.029214082285761833,
+            -0.0010053407168015838,
+            -0.03507460653781891,
+            -0.017953382804989815,
+            -0.0425640344619751,
+            -0.03293224051594734,
+            0.08179943263530731,
+            0.002959032543003559,
+            -0.00252082129009068,
+            0.022733869031071663,
+            0.005395752843469381,
+            0.00014842192467767745,
+            -0.006825472228229046,
+            -0.010499364696443081,
+            -0.03427785634994507,
+            0.003585364669561386,
+            -0.016802523285150528,
+            -0.0011071473127231002,
+            -0.025478221476078033,
+            -0.004550314974039793,
+            -0.017519596964120865,
+            -0.032755184918642044,
+            0.031161688268184662,
+            -0.0005953476065769792,
+            -0.03493295982480049,
+            0.01935211755335331,
+            -0.005988887045532465,
+            -0.025230344384908676,
+            0.07755011320114136,
+            0.0031958436593413353,
+            0.01846684142947197,
+            -0.0008542906143702567,
+            -0.007923213765025139,
+            0.007737305480986834,
+            0.025159522891044617,
+            -0.019104240462183952,
+            -0.02299945056438446,
+            0.021813182160258293,
+            -0.0003170392010360956,
+            -0.005537396762520075,
+            -0.03576511889696121,
+            0.019900986924767494,
+            -0.022840101271867752,
+            -0.016377592459321022,
+            0.003184777684509754,
+            -0.012606319040060043,
+            0.019989514723420143,
+            0.033870629966259,
+            0.016784818843007088,
+            0.008303881622850895,
+            0.008781930431723595,
+            0.04323684424161911,
+            0.00798075646162033,
+            0.0331801176071167,
+            0.03234795853495598,
+            0.006143810227513313,
+            -0.0033684722147881985,
+            -0.07485887408256531,
+            0.0303118247538805,
+            -0.0013168469304218888,
+            -0.031002338975667953,
+            -0.0057985526509583,
+            -0.0010662033455446362,
+            -0.036296285688877106,
+            -0.020910201594233513,
+            0.041926637291908264,
+            -0.0014042678521946073,
+            -0.0038487340789288282,
+            -0.024398185312747955,
+            -0.006055282894521952,
+            -0.02207876369357109,
+            6.712184404022992e-05,
+            0.009959346614778042,
+            -0.01853766292333603,
+            -0.016342181712388992,
+            -0.016891051083803177,
+            0.00800731498748064,
+            -0.007122039329260588,
+            0.039164576679468155,
+            0.08378244936466217,
+            -0.007617793511599302,
+            -0.013208306394517422,
+            -0.016165126115083694,
+            -0.03117939457297325,
+            -0.02871832810342312,
+            -0.03197614103555679,
+            -0.03842094540596008,
+            -0.03534018620848656,
+            0.011172173544764519,
+            -0.018148142844438553,
+            0.002722221426665783,
+            0.014332606457173824,
+            0.01792682334780693,
+            -0.014217520132660866,
+            -0.03310929238796234,
+            0.049292124807834625,
+            -0.018944889307022095,
+            -0.03254271671175957,
+            1.4307914852906833e-06,
+            0.008264044299721718,
+            -0.02122890017926693,
+            0.030081652104854584,
+            -0.04224533587694168,
+            -0.05980919674038887,
+            0.024309657514095306,
+            -0.011163320392370224,
+            0.013305687345564365,
+            0.05771994590759277,
+            -0.04791109636425972,
+            -0.008897015824913979,
+            -0.000875315919984132,
+            -0.010844621807336807,
+            0.012951577082276344,
+            0.01338536199182272,
+            -0.020927906036376953,
+            0.0021456859540194273,
+            -0.019918693229556084,
+            0.080949567258358,
+            -0.021512188017368317,
+            0.02303486131131649,
+            -0.03383522108197212,
+            0.02114037238061428,
+            -0.029957713559269905,
+            0.01624480076134205,
+            0.002892636926844716,
+            -0.011287258937954903,
+            -0.03027641400694847,
+            0.004176286049187183,
+            -0.02312338910996914,
+            0.009507856331765652,
+            0.005413458216935396,
+            -0.005324930418282747,
+            0.02753205969929695,
+            0.01435916405171156,
+            0.015386084094643593,
+            0.047379929572343826,
+            0.032755184918642044,
+            0.02939113788306713,
+            -0.05977378413081169,
+            0.05453295633196831,
+            0.016944168135523796,
+            0.03944786638021469,
+            -0.008379130624234676,
+            -0.000762996613048017,
+            -0.0042448947206139565,
+            0.013677502982318401,
+            0.030099358409643173,
+            0.03587135300040245,
+            -0.005559528712183237,
+            0.027018601074814796,
+            -0.023725377395749092,
+            0.018254375085234642,
+            0.02505328878760338,
+            0.005364768207073212,
+            0.01628021150827408,
+            0.012057448737323284,
+            0.03241878002882004,
+            0.017625829204916954,
+            -0.009906229563057423,
+            0.022769279778003693,
+            0.011260701343417168,
+            0.014058170840144157,
+            0.04904424771666527,
+            -0.01383685227483511,
+            0.01018066518008709,
+            -0.013624385930597782,
+            -0.002472131047397852,
+            0.01252664439380169,
+            -0.020573796704411507,
+            0.01570478267967701,
+            -0.018732424825429916,
+            -0.04143088310956955,
+            0.03289682790637016,
+            -0.015943806618452072,
+            0.008649138733744621,
+            0.03750025853514671,
+            0.0009483511094003916,
+            0.03781895712018013,
+            0.012075154110789299,
+            -0.03050658479332924,
+            -0.0330030620098114,
+            -0.0070423646830022335,
+            -0.022645341232419014,
+            -0.01435031183063984,
+            -0.0025562322698533535,
+            0.006546610500663519,
+            -0.004501624498516321,
+            0.013128631748259068,
+            -0.0165015310049057,
+            -0.01853766292333603,
+            -0.021848592907190323,
+            0.010021315887570381,
+            0.038350123912096024,
+            -0.00276205874979496,
+            -0.027992403134703636,
+            -0.011189878918230534,
+            -0.016758261248469353,
+            -0.009773438796401024,
+            0.0026248411741107702,
+            0.006887441501021385,
+            0.028983911499381065,
+            -0.01018066518008709,
+            -0.022379757836461067,
+            0.014961151406168938,
+            0.03390604257583618,
+            -0.005634776782244444,
+            -0.011703338474035263,
+            0.010410836897790432,
+            -0.024380480870604515,
+            -0.003375111846253276,
+            -0.005360341630876064,
+            0.01664317399263382,
+            -0.04606972262263298,
+            -0.009711469523608685,
+            -0.007940919138491154,
+            0.01660776324570179,
+            0.017068106681108475,
+            0.029302610084414482,
+            0.028045520186424255,
+            0.03785436972975731,
+            -0.0027664850931614637,
+            -0.0036738922353833914,
+            -0.0034083097707480192,
+            0.02021968737244606,
+            0.004603431094437838,
+            0.03339258208870888,
+            0.02314109541475773,
+            0.008029446005821228,
+            -0.02868291735649109,
+            -0.02500017359852791,
+            0.06420016288757324,
+            -0.012287620455026627,
+            -0.042811911553144455,
+            0.04486574977636337,
+            -0.02595626935362816,
+            0.03323323279619217,
+            0.03689827397465706,
+            0.011384639889001846,
+            0.0029568192549049854,
+            0.007724026683717966,
+            0.036437928676605225,
+            -0.011482019908726215,
+            -0.037146151065826416,
+            -0.0064270980656147,
+            0.017971087247133255,
+            -0.02781534753739834,
+            0.037181559950113297,
+            -0.0314449779689312,
+            0.03689827397465706,
+            0.020768556743860245,
+            0.009764585644006729,
+            -0.008724387735128403,
+            0.007768290117383003,
+            0.009463592432439327,
+            0.007197287864983082,
+            -0.041997458785772324,
+            -0.013704060576856136,
+            -0.06529790163040161,
+            -0.05779076740145683,
+            0.011030529625713825,
+            -0.020981023088097572,
+            0.002284010173752904,
+            -0.047415342181921005,
+            0.01294272392988205,
+            0.014385722577571869,
+            -0.008857178501784801,
+            0.0027930433861911297,
+            0.03305617719888687,
+            -0.01848454773426056,
+            -0.007538118865340948,
+            -0.05435590073466301,
+            -0.0238847266882658,
+            0.013190601021051407,
+            0.04320143163204193,
+            -0.04688417539000511,
+            -0.0128541961312294,
+            -0.01763468235731125,
+            0.011685633100569248,
+            -0.015439200215041637,
+            -0.03206466883420944,
+            0.01921047270298004,
+            0.0033817514777183533,
+            -0.013199454173445702,
+            -0.060198716819286346,
+            -0.010995118878781796,
+            0.0442991741001606,
+            0.009950493462383747,
+            0.004362193867564201,
+            -0.009835408069193363,
+            -0.033870629966259,
+            0.0515938401222229,
+            -0.015819868072867393,
+            -0.05612644925713539,
+            0.012022037990391254,
+            -0.0076045142486691475,
+            -0.023619143292307854,
+            -0.008795209228992462,
+            0.0637398213148117,
+            0.0008692296105436981,
+            -0.000270700576948002,
+            -0.01936982199549675,
+            -0.004446295090019703,
+            -0.013226011767983437,
+            -0.009127187542617321,
+            -0.0059800343587994576,
+            -0.052939459681510925,
+            -0.022769279778003693,
+            0.009082923643290997,
+            -0.010641008615493774,
+            -0.0031537930481135845,
+            -0.02754976600408554,
+            -0.018980301916599274,
+            -0.004264813382178545,
+            0.004647694993764162,
+            0.02662907913327217,
+            -0.042032867670059204,
+            -0.030081652104854584,
+            0.033870629966259,
+            0.01763468235731125,
+            -0.01628021150827408,
+            0.0017827230039983988,
+            0.018927184864878654,
+            -0.0012139336904510856,
+            0.03760649263858795,
+            0.015403789468109608,
+            -0.01288075465708971,
+            -0.03125021606683731,
+            -0.021813182160258293,
+            0.0017307130619883537,
+            0.01610315591096878,
+            0.024610651656985283,
+            -0.04578643664717674,
+            -0.02399095892906189,
+            0.009915082715451717,
+            -0.06324406713247299,
+            -0.028771445155143738,
+            -0.03849176689982414,
+            0.008064857684075832,
+            0.015952659770846367,
+            -0.013491595163941383,
+            -0.017776327207684517,
+            0.023371266201138496,
+            -0.009684910997748375,
+            0.002646972890943289,
+            -0.007082202006131411,
+            0.024132603779435158,
+            0.03321552649140358,
+            -0.014642452821135521,
+            -0.038350123912096024,
+            0.030028536915779114,
+            -0.00265582581050694,
+            0.011676780879497528,
+            0.046636298298835754,
+            0.014669010415673256,
+            -0.03689827397465706,
+            0.01922817900776863,
+            0.016492677852511406,
+            0.013562416657805443,
+            -0.01197777409106493,
+            0.007060070056468248,
+            -0.023548321798443794,
+            -0.010065579786896706,
+            -0.0017683373298496008,
+            -0.018289785832166672,
+            -0.014961151406168938,
+            0.03194073215126991,
+            0.006196926813572645,
+            0.002070437418296933,
+            0.0043444884940981865,
+            -0.015200176276266575,
+            -0.005333783570677042,
+            -0.008334866724908352,
+            0.016979578882455826,
+            0.01921047270298004,
+            0.01236729510128498,
+            0.013633239082992077,
+            0.01662546955049038,
+            0.027921581640839577,
+            0.03976656496524811,
+            -0.035446420311927795,
+            0.010340014472603798,
+            -0.010286898352205753,
+            -0.03333946689963341,
+            -0.0008493109489791095,
+            0.039979029446840286,
+            0.04132464900612831,
+            -0.02029050886631012,
+            0.008928000926971436,
+            0.008622581139206886,
+            -0.003306502941995859,
+            -0.004944262094795704,
+            0.02774452604353428,
+            0.044653281569480896,
+            0.036402519792318344,
+            -0.009463592432439327,
+            0.008786357007920742,
+            -0.014916887506842613,
+            0.008821767754852772,
+            0.02505328878760338,
+            -0.03436638414859772,
+            -0.0034503601491451263,
+            0.004926556721329689,
+            0.03806683421134949,
+            -0.02131742797791958,
+            -0.003974885679781437,
+            -0.04578643664717674,
+            -0.014846066012978554,
+            -0.015642814338207245,
+            0.011172173544764519,
+            0.007029085420072079,
+            -0.013110926374793053,
+            -0.007471723016351461,
+            0.00531607773154974,
+            -0.01565166562795639,
+            -0.03806683421134949,
+            -0.03208237513899803,
+            -0.04228074476122856,
+            -0.03500378131866455,
+            -0.021653832867741585,
+            -0.021441366523504257,
+            -0.05644514784216881,
+            0.03774813562631607,
+            -0.08640286326408386,
+            -0.021777771413326263,
+            0.02765599824488163,
+            -0.013898821547627449,
+            0.01338536199182272,
+            -0.03898752108216286,
+            0.0128541961312294,
+            -0.05998625233769417,
+            -0.013730619102716446,
+            0.05403720214962959,
+            0.014483102597296238,
+            0.03289682790637016,
+            0.009596383199095726,
+            0.02124660648405552,
+            -0.05499329790472984,
+            -0.027089422568678856,
+            -0.002164497971534729,
+            -0.010933149605989456,
+            0.008511921390891075,
+            -0.002127980347722769,
+            0.004437441937625408,
+            0.012995840981602669,
+            0.0012238930212333798,
+            0.010711830109357834,
+            0.009622941724956036,
+            -0.05092103034257889,
+            -0.00127700949087739,
+            -0.004291371908038855,
+            0.0258854478597641,
+            0.0031537930481135845,
+            -0.0010833556298166513,
+            -0.013721765950322151,
+            0.010021315887570381,
+            0.014784096740186214,
+            0.017351394519209862,
+            0.015518874861299992,
+            0.01065871398895979,
+            -0.04631759971380234,
+            -0.019794754683971405,
+            0.039022933691740036,
+            -0.005015084054321051,
+            0.007015806157141924,
+            -0.01853766292333603,
+            -0.005121317226439714,
+            -0.01759927161037922,
+            -0.003237894270569086,
+            0.01759927161037922,
+            -0.0014042678521946073,
+            -0.014908035285770893,
+            -0.021441366523504257,
+            -0.041041359305381775,
+            0.023707671090960503,
+            0.03392374888062477,
+            -0.0004260387213435024,
+            -0.023318149149417877,
+            0.007307947147637606,
+            -0.04231615737080574,
+            -0.010579039342701435,
+            0.004926556721329689,
+            0.020556090399622917,
+            -0.01921047270298004,
+            -0.007055643480271101,
+            0.013509300537407398,
+            0.009968198835849762,
+            -0.0295327827334404,
+            0.013217159546911716,
+            0.02475229650735855,
+            0.023743081837892532,
+            -0.026328084990382195,
+            0.002602709224447608,
+            -0.003523395396769047,
+            0.048265207558870316,
+            -0.012508939020335674,
+            -0.012535497546195984,
+            -0.0035676592960953712,
+            -0.0030453468207269907,
+            0.017811737954616547,
+            -0.03303847089409828,
+            -0.019175061956048012,
+            0.011605958454310894,
+            -0.012030890211462975,
+            -0.008560611866414547,
+            -0.005125743802636862,
+            -0.012889607809484005,
+            0.010162959806621075,
+            0.01756386086344719,
+            -0.02756747044622898,
+            -0.020857084542512894,
+            -0.0347382016479969,
+            0.04975247010588646,
+            -0.01288075465708971,
+            -0.0023415531031787395,
+            0.03426015377044678,
+            0.009490150958299637,
+            0.008109121583402157,
+            -0.018201259896159172,
+            0.03337487578392029,
+            0.02473459020256996,
+            0.014624747447669506,
+            -0.024327363818883896,
+            0.00028245814610272646,
+            0.025035584345459938,
+            0.0023282738402485847,
+            -0.017900265753269196,
+            -0.018643897026777267,
+            -0.003943901043385267,
+            0.007693042047321796,
+            -0.018998006358742714,
+            -0.010286898352205753,
+            0.004421949852257967,
+            -0.014093581587076187,
+            -0.00017318196478299797,
+            -0.014208667911589146,
+            -0.032719772309064865,
+            0.002005148446187377,
+            0.01525329239666462,
+            0.004298011306673288,
+            0.026345791295170784,
+            -0.009401623159646988,
+            -0.0028904236387461424,
+            0.009985905140638351,
+            -0.003054199507459998,
+            0.036367107182741165,
+            0.02951507642865181,
+            0.007427459117025137,
+            -0.009472444653511047,
+            -0.026328084990382195,
+            -0.014368017204105854,
+            0.005081479903310537,
+            0.008941279724240303,
+            0.001089995144866407,
+            0.02776223234832287,
+            0.026381202042102814,
+            0.014598188921809196,
+            -0.009738028049468994,
+            -0.003412736114114523,
+            0.000571002543438226,
+            -0.009286537766456604,
+            0.02671760693192482,
+            0.05623268336057663,
+            0.012269915081560612,
+            -0.032755184918642044,
+            0.009879671968519688,
+            -0.02678842842578888,
+            0.006055282894521952,
+            0.014270637184381485,
+            0.002030600095167756,
+            -0.056799259036779404,
+            0.02659366838634014,
+            0.014970004558563232,
+            0.014996563084423542,
+            -0.020857084542512894,
+            -0.05410802364349365,
+            0.000554680242203176,
+            0.006431524641811848,
+            0.0010390918469056487,
+            -0.007940919138491154,
+            -0.015456905588507652,
+            0.01718319207429886,
+            -0.0005671294638887048,
+            -0.007405327633023262,
+            0.007498281542211771,
+            0.02484082244336605,
+            -0.012588613666594028,
+            0.007825832813978195,
+            -0.01291616540402174,
+            -0.0009682697709649801,
+            0.005302798934280872,
+            0.030099358409643173,
+            0.0037115165032446384,
+            0.03493295982480049,
+            -0.003399456851184368,
+            0.016359886154532433,
+            0.028913090005517006,
+            -0.010587891563773155,
+            -0.006081840954720974,
+            0.0019055550219491124,
+            0.0035920043010264635,
+            -0.021441366523504257,
+            0.003401670139282942,
+            -0.025991681963205338,
+            -0.03406539186835289,
+            0.010189518332481384,
+            0.0030475601088255644,
+            0.009755733422935009,
+            0.015881838276982307,
+            -0.004410883877426386,
+            -0.0012902886373922229,
+            0.0005508071626536548,
+            0.015005415305495262,
+            0.0038730790838599205,
+            0.12500086426734924,
+            -0.01390767376869917,
+            0.024150308221578598,
+            0.005298372358083725,
+            0.018148142844438553,
+            0.03757108002901077,
+            0.01201318483799696,
+            0.017811737954616547,
+            0.007688615471124649,
+            0.003939474932849407,
+            0.04405129700899124,
+            -0.02103414013981819,
+            -0.01570478267967701,
+            0.016289064660668373,
+            -0.008786357007920742,
+            0.015332967042922974,
+            0.0014540646225214005,
+            -0.015766752883791924,
+            0.011667927727103233,
+            0.02029050886631012,
+            -0.010791505686938763,
+            0.02305256761610508,
+            -0.019086534157395363,
+            -0.031055456027388573,
+            0.01528870314359665,
+            -0.004957541357725859,
+            0.04858390614390373,
+            0.0051611545495688915,
+            -0.004966394044458866,
+            -0.02199023775756359,
+            -0.003468065755441785,
+            -0.02843504026532173,
+            -0.018891774117946625,
+            0.008560611866414547,
+            0.027089422568678856,
+            0.02758517675101757,
+            0.0017041548853740096,
+            -0.010056726634502411,
+            -0.007485002279281616,
+            0.025442810729146004,
+            0.0014927954180166125,
+            -0.00400365749374032,
+            0.006590874399989843,
+            -0.014474250376224518,
+            0.011659074574708939,
+            -0.014376870356500149,
+            0.003954967018216848,
+            0.0004039068298880011,
+            -0.01290731318295002,
+            0.0009544373606331646,
+            0.03569429740309715,
+            0.005404605530202389,
+            -0.01018066518008709,
+            -0.03151579946279526,
+            0.01284534391015768,
+            0.01419096253812313,
+            0.00324232061393559,
+            -0.0023039288353174925,
+            -0.0009594170260243118,
+            -0.011605958454310894,
+            0.029107850044965744,
+            -0.002613775199279189,
+            -0.004528183024376631,
+            0.0027200081385672092,
+            0.0006728091975674033,
+            -0.01940523274242878,
+            0.015828721225261688,
+            0.011260701343417168,
+            0.025389693677425385,
+            0.022362053394317627,
+            0.024097193032503128,
+            0.019830165430903435,
+            0.005900359712541103,
+            -0.005214271135628223,
+            -0.013252570293843746,
+            -0.02030821330845356,
+            0.043838828802108765,
+            0.00038813785067759454,
+            0.012863049283623695,
+            -0.000697707524523139,
+            -0.0684494823217392,
+            -0.033711280673742294,
+            0.019015712663531303,
+            0.02220270223915577,
+            0.024415891617536545,
+            0.0017849361756816506,
+            0.0028616522904485464,
+            0.022468285635113716,
+            -0.04118300601840019,
+            -0.007600088138133287,
+            0.005006231367588043,
+            -0.009074071422219276,
+            -0.004997378680855036,
+            -0.0025141816586256027,
+            0.023442087695002556,
+            -0.01660776324570179,
+            -0.012243356555700302,
+            0.034614261239767075,
+            -0.014943446032702923,
+            0.02662907913327217,
+            -0.00017829996068030596,
+            0.0008636966813355684,
+            0.012199092656373978,
+            0.013270275667309761,
+            0.03250730782747269,
+            -0.020963318645954132,
+            -0.0080781364813447,
+            0.003970459569245577,
+            -0.009622941724956036,
+            0.029037028551101685,
+            -0.009968198835849762,
+            -0.04592807963490486,
+            0.01666088029742241,
+            0.016837935894727707,
+            0.03693368285894394,
+            -0.005453295540064573,
+            0.010189518332481384,
+            -0.010782652534544468,
+            0.041749581694602966,
+            -0.005262961145490408,
+            -0.007569103501737118,
+            0.03410080075263977,
+            -0.009950493462383747,
+            0.0029014896135777235,
+            0.022344347089529037,
+            -0.011198732070624828,
+            -0.031692855060100555,
+            0.019564582034945488,
+            0.016041187569499016,
+            0.059419672936201096,
+            -0.018732424825429916,
+            0.016598910093307495,
+            0.019723933190107346,
+            -0.0020217474084347486,
+            -0.01651923544704914,
+            -0.03689827397465706,
+            0.008308308199048042,
+            -0.007077775429934263,
+            0.007489428389817476,
+            -0.015908395871520042,
+            0.03707532584667206,
+            -0.019759343937039375,
+            -0.013996201567351818,
+            0.008742093108594418,
+            -0.0036318416241556406,
+            -0.01338536199182272,
+            0.02664678543806076,
+            -0.0048822928220033646,
+            -0.03927081078290939,
+            -0.012818785384297371,
+            0.007223845925182104,
+            0.02939113788306713,
+            -0.02112266793847084,
+            0.01057018619030714,
+            0.022344347089529037,
+            -0.015518874861299992,
+            -0.02478770725429058,
+            0.01201318483799696,
+            -0.016784818843007088,
+            0.009481297805905342,
+            0.02480541169643402,
+            -0.03050658479332924,
+            -0.006852030288428068,
+            -0.028399629518389702,
+            -0.01374832447618246,
+            -0.012190240435302258,
+            0.019015712663531303,
+            0.0030187885276973248,
+            0.0060729882679879665,
+            -0.025177227333188057,
+            0.027408121153712273,
+            0.013996201567351818,
+            -0.013854557648301125,
+            -0.01834290288388729,
+            -0.010251487605273724,
+            -0.02496476098895073,
+            -0.0044662137515842915,
+            -0.02036133036017418,
+            -0.016528088599443436,
+            -7.096034096321091e-05,
+            -0.004227189347147942,
+            -0.018927184864878654,
+            0.005891506560146809,
+            -0.00015533814439550042,
+            0.004705238156020641,
+            0.06069447100162506,
+            0.007830259390175343,
+            0.02593856491148472,
+            -0.003412736114114523,
+            -0.020007221028208733,
+            0.03149809315800667,
+            -0.02018427476286888,
+            0.042882733047008514,
+            0.0008465444552712142,
+            -0.03032952919602394,
+            0.01010984368622303,
+            0.037004504352808,
+            -0.009056366048753262,
+            0.006121678277850151,
+            -0.01832519844174385,
+            0.0012006545439362526,
+            -0.013403067365288734,
+            -0.014031612314283848,
+            -0.0007010273402556777,
+            0.006639564409852028,
+            0.01563396118581295,
+            0.01521788164973259,
+            0.017811737954616547,
+            0.011358081363141537,
+            0.0008033872582018375,
+            0.02027280256152153,
+            -0.022609930485486984,
+            0.008888163603842258,
+            -0.009826554916799068,
+            0.021848592907190323,
+            0.006701533682644367,
+            0.0039372616447508335,
+            -0.011207584291696548,
+            0.004322356544435024,
+            -0.038314711302518845,
+            0.0028218147344887257,
+            0.01747533306479454,
+            0.022556813433766365,
+            -0.02595626935362816,
+            -0.04578643664717674,
+            -0.004820323549211025,
+            -0.05775535851716995,
+            -0.028293397277593613,
+            -0.010995118878781796,
+            -0.00369159784168005,
+            0.02209646999835968,
+            -0.002565084956586361,
+            0.024380480870604515,
+            -0.018148142844438553,
+            -0.014854918234050274,
+            0.01195121556520462,
+            -0.022379757836461067,
+            -0.008272897452116013,
+            0.016891051083803177,
+            0.017271719872951508,
+            -0.017971087247133255,
+            0.028045520186424255,
+            -0.024681473150849342,
+            -0.0035809383261948824,
+            -0.04398047551512718,
+            -0.01834290288388729,
+            -0.015934955328702927,
+            -0.0035411010030657053,
+            0.008604875765740871,
+            -0.011933510191738605,
+            -0.016722848638892174,
+            0.0011663500918075442,
+            0.009516708552837372,
+            -0.013633239082992077,
+            0.03346340358257294,
+            -0.01834290288388729,
+            -0.017360247671604156,
+            0.002666891785338521,
+            -0.012349589727818966,
+            -0.021777771413326263,
+            -0.02206105925142765,
+            0.01293387170881033,
+            -0.020626913756132126,
+            0.004922130145132542,
+            -0.017953382804989815,
+            0.013562416657805443,
+            -0.02671760693192482,
+            -0.01844913698732853,
+            -0.0029656721744686365,
+            -0.012925018556416035,
+            -0.05708254873752594,
+            0.019971810281276703,
+            0.05283322557806969,
+            -0.005479853600263596,
+            -0.014890329912304878,
+            0.033551931381225586,
+            -0.008600449189543724,
+            -0.024380480870604515,
+            0.02485852874815464,
+            -0.0019199406960979104,
+            0.023654554039239883,
+            -0.02505328878760338,
+            -0.0035676592960953712,
+            0.00015353992057498544,
+            0.019564582034945488,
+            -0.020927906036376953,
+            0.007405327633023262,
+            -0.013004693202674389,
+            0.022362053394317627,
+            -0.00020485822460614145,
+            0.006391687318682671,
+            0.046636298298835754,
+            -0.014031612314283848,
+            -0.00970261637121439,
+            0.04157252609729767,
+            -0.006059709005057812,
+            0.0032069096341729164,
+            -0.024433596059679985,
+            0.020502975210547447,
+            0.03619005158543587,
+            0.008914722129702568,
+            0.01560740265995264,
+            -0.03523395583033562,
+            0.028736034408211708,
+            0.004696385003626347,
+            0.028895383700728416,
+            0.029196377843618393,
+            -0.01944064348936081,
+            0.026115620508790016,
+            0.007799274753779173,
+            0.0128541961312294,
+            -0.0017174340318888426,
+            0.02018427476286888,
+            -0.018661601468920708,
+            -0.01673170179128647,
+            -0.01566937193274498,
+            0.0442991741001606,
+            -0.002087036380544305,
+            -0.010295751504600048,
+            -0.01721860282123089,
+            -0.02478770725429058,
+            0.0048778667114675045,
+            0.0029435402248054743,
+            -0.01470442209392786,
+            -0.01673170179128647,
+            -0.021972531452775,
+            0.015439200215041637,
+            0.010233782231807709,
+            -0.013527005910873413,
+            0.00711761275306344,
+            -0.002790830098092556,
+            -0.0038575867656618357,
+            -0.002963458886370063,
+            -0.01246467512100935,
+            -0.03760649263858795,
+            -0.022751573473215103,
+            -0.01515591237694025,
+            0.03333946689963341,
+            -0.010136401280760765,
+            0.020591503009200096,
+            -0.006081840954720974,
+            0.012553202919661999,
+            -0.0389166995882988,
+            0.00026378437178209424,
+            -0.015067384578287601,
+            -0.0304888803511858,
+            0.025194933637976646,
+            -0.010605597868561745,
+            0.008879310451447964,
+            0.032578129321336746,
+            -0.027124833315610886,
+            -0.024610651656985283,
+            0.01338536199182272,
+            -0.004353341180831194,
+            0.0017550581833347678,
+            -0.017625829204916954,
+            0.016944168135523796,
+            -0.01942293904721737,
+            -0.028045520186424255,
+            0.013057810254395008,
+            0.03192302584648132,
+            0.012606319040060043,
+            0.012199092656373978,
+            0.03693368285894394,
+            -0.006276601459830999,
+            -0.037181559950113297,
+            -0.006471361964941025,
+            -0.009499003179371357,
+            -0.03572971001267433,
+            -0.004537035711109638,
+            0.006435951218008995,
+            -0.003970459569245577,
+            -0.01669629104435444,
+            -0.02379619888961315,
+            -0.007940919138491154,
+            -0.012349589727818966,
+            -0.05240829288959503,
+            0.01516476459801197,
+            -0.019777048379182816,
+            -0.004286945331841707,
+            -0.03418932855129242,
+            0.05077938735485077,
+            0.02843504026532173,
+            0.007467296905815601,
+            0.006736944429576397,
+            -0.01293387170881033,
+            -0.0021423660218715668,
+            0.00507262721657753,
+            0.0032179756090044975,
+            0.01922817900776863,
+            -0.020910201594233513,
+            0.014987709932029247,
+            -0.003056412795558572,
+            0.005457721650600433,
+            -0.056693024933338165,
+            -0.020927906036376953,
+            0.010056726634502411,
+            0.005581660196185112,
+            -0.0003308716113679111,
+            0.0037911911495029926,
+            0.011783013120293617,
+            -0.01716548763215542,
+            -0.021689243614673615,
+            0.01610315591096878,
+            0.030063947662711143,
+            0.01522673387080431,
+            -0.006365128792822361,
+            0.019104240462183952,
+            0.019599994644522667,
+            -0.006865309551358223,
+            0.005993313621729612,
+            -0.01426178403198719,
+            0.009569825604557991,
+            0.010853474959731102,
+            0.0036583999171853065,
+            0.022822394967079163,
+            -0.00203502643853426,
+            -0.0013876688899472356,
+            -0.00814010575413704,
+            0.003932835068553686,
+            -0.005546249449253082,
+            -0.03025870770215988,
+            -0.014757538214325905,
+            -0.013633239082992077,
+            0.011242995969951153,
+            0.03426015377044678,
+            0.02126431092619896,
+            0.014722127467393875,
+            -0.012747963890433311,
+            -0.006807766854763031,
+            -0.001094974810257554,
+            0.006334144156426191,
+            0.003992591518908739,
+            0.01387226302176714,
+            -0.014474250376224518,
+            -0.008538479916751385,
+            0.002060478087514639,
+            0.003950540907680988,
+            -0.01197777409106493,
+            0.0385625883936882,
+            0.04635301232337952,
+            -0.02106955088675022,
+            -0.017953382804989815,
+            0.006577595137059689,
+            0.016811376437544823,
+            -0.005652482621371746,
+            -0.0037115165032446384,
+            -0.005873801186680794,
+            -0.00027249878621660173,
+            -0.01787370629608631,
+            0.015200176276266575,
+            0.01111020427197218,
+            -0.02767370454967022,
+            0.010835768654942513,
+            0.09653041511774063,
+            -0.02661137469112873,
+            0.009206862188875675,
+            -0.026345791295170784,
+            -0.011995479464530945,
+            -0.004554741084575653,
+            0.0048424554988741875,
+            0.02864750660955906,
+            0.02294633351266384,
+            0.0032179756090044975,
+            0.025389693677425385,
+            -0.0028328807093203068,
+            -0.010791505686938763,
+            0.013199454173445702,
+            -0.0006645097164437175,
+            0.013287981040775776,
+            -0.02395554818212986,
+            0.010286898352205753,
+            -0.011411197483539581,
+            -0.011358081363141537,
+            0.011437756009399891,
+            0.013757177628576756,
+            -0.0006883015157654881,
+            0.021671537309885025,
+            -0.004014723002910614,
+            0.02200794219970703,
+            0.0008731026900932193,
+            -4.9450923143012915e-06,
+            -0.04964623600244522,
+            -0.015085089951753616,
+            0.0024190146941691637,
+            -0.007932065986096859,
+            -0.02115807868540287,
+            0.001672063604928553,
+            -0.00224527926184237,
+            0.01566937193274498,
+            -0.01057018619030714,
+            0.01714778132736683,
+            0.0050682006403803825,
+            -0.010552480816841125,
+            -0.0037292218767106533,
+            0.009321948513388634,
+            -0.011783013120293617,
+            0.012473528273403645,
+            -0.009472444653511047,
+            0.004242681432515383,
+            0.023707671090960503,
+            -0.025336576625704765,
+            0.01191580481827259,
+            -0.009684910997748375,
+            0.000783468596637249,
+            0.016846787184476852,
+            -0.018714718520641327,
+            0.008317161351442337,
+            -0.0012515578418970108,
+            0.006254469510167837,
+            0.001487262430600822,
+            -0.016864493489265442,
+            0.012570908293128014,
+            -0.010853474959731102,
+            0.005417884327471256,
+            0.007245977874845266,
+            0.01778518036007881,
+            -0.014854918234050274,
+            0.010711830109357834,
+            -0.006064135581254959,
+            -0.002739926800131798,
+            -0.0035698723513633013,
+            -0.019086534157395363,
+            0.0028572259470820427,
+            0.00687416223809123,
+            -0.006099546328186989,
+            -0.0249293502420187,
+            0.002538526663556695,
+            -0.004267026670277119,
+            0.019635405391454697,
+            0.00019074915326200426,
+            -0.015279850922524929,
+            -0.007613366935402155,
+            -0.020981023088097572,
+            0.012792226858437061,
+            0.0021589649841189384,
+            0.014164404012262821,
+            0.011986627243459225,
+            -0.007781569380313158,
+            0.008188796229660511,
+            -0.024398185312747955,
+            0.006360702682286501,
+            0.02593856491148472,
+            -0.002722221426665783,
+            -0.0007989609148353338,
+            -0.004023576155304909,
+            -0.010340014472603798,
+            -0.00014330392878036946,
+            -0.007684188894927502,
+            -0.014421133324503899,
+            0.0031936303712427616,
+            0.00054444425040856,
+            -0.011251848191022873,
+            0.00483360281214118,
+            -0.012411559000611305,
+            0.015713635832071304,
+            0.01249123364686966,
+            -0.014075876213610172,
+            -0.020680028945207596,
+            0.00714859738945961,
+            -0.010003610514104366,
+            -0.0013223799178376794,
+            0.0026226278860121965,
+            0.0008360318024642766,
+            0.008742093108594418,
+            -0.026044797152280807,
+            0.003638481255620718,
+            -0.018838657066226006,
+            0.015501169487833977,
+            -0.011304965242743492,
+            0.014067023992538452,
+            -0.022698456421494484,
+            0.01689990423619747,
+            0.020520679652690887,
+            -0.019883282482624054,
+            0.0038664396852254868,
+            0.013580122031271458,
+            -0.021459072828292847,
+            0.0119423633441329,
+            0.015297556295990944,
+            0.00866684503853321,
+            -0.011384639889001846,
+            0.006307586096227169,
+            0.015315261669456959,
+            0.001169669907540083,
+            0.018608486279845238,
+            0.017121223732829094,
+            0.0027000894770026207,
+            0.01332339271903038,
+            -0.02661137469112873,
+            0.020963318645954132,
+            0.012597466818988323,
+            0.03948327526450157,
+            -0.00597560778260231,
+            -0.004873440135270357,
+            0.006157089490443468,
+            0.02204335294663906,
+            -0.0017041548853740096,
+            0.012243356555700302,
+            -0.009817702695727348,
+            -0.01810387894511223,
+            0.03142727166414261,
+            0.00804272573441267,
+            0.025124112144112587,
+            -0.010401983745396137,
+            -0.004286945331841707,
+            0.025088699534535408,
+            0.019139651209115982,
+            0.0030608391389250755,
+            -0.01010984368622303,
+            0.005670187994837761,
+            -0.0015215668827295303,
+            0.023388972505927086,
+            -0.013889968395233154,
+            0.011844982393085957,
+            0.029975419864058495,
+            0.024203425273299217,
+            -0.028116341680288315,
+            0.005807405803352594,
+            -0.012986987829208374,
+            -0.0049354094080626965,
+            -0.008649138733744621,
+            0.009375064633786678,
+            0.0030165754724293947,
+            -0.020733145996928215,
+            -0.004510477185249329,
+            -0.002430080436170101,
+            -0.0031095293816179037,
+            0.039235398173332214,
+            -0.017917970195412636,
+            0.03620775789022446,
+            0.01841372437775135,
+            0.016705144196748734,
+            -0.022468285635113716,
+            0.030719051137566566,
+            -0.016253653913736343,
+            -0.010800357908010483,
+            -0.0007917680195532739,
+            0.008339292369782925,
+            0.0066351378336548805,
+            -0.017289426177740097,
+            0.008423393592238426,
+            -0.018626190721988678,
+            0.02025509811937809,
+            -0.004899998661130667,
+            -0.00035604662843979895,
+            -0.02034362591803074,
+            0.0038332417607307434,
+            0.00898111704736948,
+            0.010410836897790432,
+            -0.007307947147637606,
+            -0.008162237703800201,
+            -0.009025380946695805,
+            -0.01733369007706642,
+            -0.017156634479761124,
+            0.011393492110073566,
+            -0.01201318483799696,
+            -0.01152628380805254,
+            0.02207876369357109,
+            -0.029762953519821167,
+            -0.022415168583393097,
+            0.018077321350574493,
+            -0.0025031156837940216,
+            -0.028948500752449036,
+            0.004541461821645498,
+            -0.019104240462183952,
+            0.004041281528770924,
+            -0.008950132876634598,
+            0.0008830620790831745,
+            0.00894570630043745,
+            -0.008733239956200123,
+            -0.03955409675836563,
+            -0.003014362184330821,
+            0.0037535671144723892,
+            -0.009136040695011616,
+            0.00711318664252758,
+            0.008011740632355213,
+            0.0008902549161575735,
+            0.013447331264615059,
+            -0.009968198835849762,
+            0.008202075026929379,
+            -0.03774813562631607,
+            0.02218499779701233,
+            0.009684910997748375,
+            -0.036437928676605225,
+            0.010295751504600048,
+            -0.00811354722827673,
+            0.010463953018188477,
+            0.02753205969929695,
+            0.00441752327606082,
+            0.020697735249996185,
+            -0.002020640764385462,
+            0.0011962282005697489,
+            0.011862688697874546,
+            -0.0033618328161537647,
+            0.014642452821135521,
+            -0.01619168370962143,
+            0.029338020831346512,
+            -0.0011873753974214196,
+            0.00014551711501553655,
+            0.032737478613853455,
+            -0.003027641447260976,
+            -0.02577921561896801,
+            0.009879671968519688,
+            -0.013766029849648476,
+            -0.030966928228735924,
+            0.030665934085845947,
+            -0.03119709901511669,
+            0.0013046744279563427,
+            0.00852077454328537,
+            -0.006546610500663519,
+            -0.0005032236222177744,
+            -0.02501787804067135,
+            -0.030117064714431763,
+            0.010039021261036396,
+            0.014571630395948887,
+            0.011304965242743492,
+            0.018183553591370583,
+            -0.027018601074814796,
+            0.013004693202674389,
+            -0.008073709905147552,
+            0.006303159520030022,
+            0.009587530978024006,
+            0.012119418010115623,
+            -0.019175061956048012,
+            0.025991681963205338,
+            -0.008432246744632721,
+            0.02397325448691845,
+            0.0037292218767106533,
+            -0.0030741184018552303,
+            -0.00026378437178209424,
+            -0.03240107372403145,
+            0.006241190247237682,
+            -0.006188074126839638,
+            0.003757993457838893,
+            0.0797455906867981,
+            0.0011641369201242924,
+            -0.0020881430245935917,
+            0.022733869031071663,
+            -0.009375064633786678,
+            -0.011030529625713825,
+            -0.015297556295990944,
+            -0.006347423419356346,
+            -0.013527005910873413,
+            -0.024628357961773872,
+            -0.006263322196900845,
+            -0.004260387271642685,
+            -0.005957902409136295,
+            0.01841372437775135,
+            0.0006362915737554431,
+            -0.028222573921084404,
+            0.0002011234755627811,
+            -0.010826916433870792,
+            -0.010844621807336807,
+            0.017953382804989815,
+            0.0037380747962743044,
+            0.007608940824866295,
+            -0.00400365749374032,
+            -0.01758156716823578,
+            -0.029338020831346512,
+            -0.026947777718305588,
+            -0.0002370877773500979,
+            -0.006980395410209894,
+            -0.0016432921402156353,
+            0.001767230685800314,
+            -0.023530615493655205,
+            -0.030028536915779114,
+            0.02115807868540287,
+            -0.015412641689181328,
+            0.005550675559788942,
+            0.011384639889001846,
+            -0.00044540411909110844,
+            1.79648632183671e-05,
+            0.020609207451343536,
+            -0.04808815196156502,
+            -0.015040826052427292,
+            -0.010809211060404778,
+            -0.008653565309941769,
+            -0.016775965690612793,
+            0.015377230942249298,
+            -0.010198370553553104,
+            -0.003839881392195821,
+            -0.014067023992538452,
+            -0.002136833034455776,
+            -0.0013688568724319339,
+            -0.030170179903507233,
+            -0.012668288312852383,
+            -0.015430347062647343,
+            0.031108571216464043,
+            0.008861605077981949,
+            -0.000989848398603499,
+            -0.0019409660017117858,
+            -0.0161562729626894,
+            0.017032695934176445,
+            -0.03788977861404419,
+            0.008923574350774288,
+            -0.008100268431007862,
+            0.028222573921084404,
+            0.02760288119316101,
+            -0.017935676500201225,
+            0.0038819320034235716,
+            -0.003523395396769047,
+            0.003890784690156579,
+            -0.0087730772793293,
+            -0.021848592907190323,
+            0.010782652534544468,
+            0.0102072237059474,
+            0.002379177138209343,
+            0.030612817034125328,
+            0.02590315416455269,
+            -0.001249344670213759,
+            0.014093581587076187,
+            -0.009313095360994339,
+            0.006958263460546732,
+            -0.02108725719153881,
+            0.020042631775140762,
+            -0.008688976988196373,
+            0.0028572259470820427,
+            -0.008476510643959045,
+            0.013978496193885803,
+            -0.014642452821135521,
+            -0.010933149605989456,
+            0.04061642661690712,
+            -0.018697014078497887,
+            -0.011729896999895573,
+            -0.013987348414957523,
+            0.00393504835665226,
+            0.00924227386713028,
+            0.008750945329666138,
+            0.030010830610990524,
+            -0.002717795083299279,
+            0.012314178980886936,
+            0.001447424991056323,
+            0.003954967018216848,
+            -0.017794031649827957,
+            -0.010100990533828735,
+            0.005147875752300024,
+            0.00666612247005105,
+            0.004992952570319176,
+            -0.000783468596637249,
+            -0.011304965242743492,
+            0.0025119686033576727,
+            0.00021924394241068512,
+            -0.018254375085234642,
+            0.003118382068350911,
+            -0.03284371271729469,
+            0.014952299185097218,
+            0.0020637980196624994,
+            0.01794452965259552,
+            0.009746880270540714,
+            -0.026434319093823433,
+            0.018201259896159172,
+            -0.001513820723630488,
+            0.00762664619833231,
+            0.027248771861195564,
+            -0.031126277521252632,
+            -0.012340736575424671,
+            0.014908035285770893,
+            0.022432874888181686,
+            0.013491595163941383,
+            0.008246338926255703,
+            -0.027018601074814796,
+            0.026345791295170784,
+            -0.015554285608232021,
+            0.00805157795548439,
+            -0.02204335294663906,
+            -0.007706320844590664,
+            -0.032631244510412216,
+            -0.00017110710905399173,
+            -0.0017550581833347678,
+            0.013465036638081074,
+            -0.0023836034815758467,
+            -0.02126431092619896,
+            0.0034105228260159492,
+            0.007878949865698814,
+            -0.014368017204105854,
+            -0.008883737027645111,
+            0.0017616976983845234,
+            -0.00552411749958992,
+            0.0007784889312461019,
+            0.008879310451447964,
+            -0.0035477406345307827,
+            -0.03034723550081253,
+            -0.010756094008684158,
+            0.0027023027651011944,
+            0.008897015824913979,
+            -0.0033640458714216948,
+            0.014270637184381485,
+            -0.011827277019619942,
+            0.0018646109383553267,
+            -0.003215762320905924,
+            0.0109862657263875,
+            0.0019863364286720753,
+            -0.008852752856910229,
+            0.0013998415088281035,
+            -0.00798075646162033,
+            -0.0276205874979496,
+            0.015023120678961277,
+            -0.00504606869071722,
+            -0.006493493914604187,
+            -0.0002827347780112177,
+            0.0015337393851950765,
+            -0.005152301862835884,
+            -0.013925379142165184,
+            0.011774160899221897,
+            0.0018491186201572418,
+            0.0014507448067888618,
+            -0.0036318416241556406,
+            -0.004138661548495293,
+            -0.002044985769316554,
+            0.01751074381172657,
+            0.02774452604353428,
+            -0.018626190721988678,
+            0.010490511544048786,
+            -0.01767009310424328,
+            0.00048026180593296885,
+            -0.001060117152519524,
+            0.01763468235731125,
+            0.011375786736607552,
+            0.02114037238061428,
+            0.01516476459801197,
+            -0.008719961158931255,
+            0.010384278371930122,
+            -0.008166664279997349,
+            0.015872985124588013,
+            -0.009348507039248943,
+            -0.0007602301193401217,
+            -0.0009821022395044565,
+            0.02209646999835968,
+            -0.018980301916599274,
+            -0.0017284998903051019,
+            0.009499003179371357,
+            0.022450579330325127,
+            -0.016908757388591766,
+            -0.011242995969951153,
+            0.031108571216464043,
+            0.003414949169382453,
+            0.006975968834012747,
+            0.017236309126019478,
+            0.03229483962059021,
+            0.006307586096227169,
+            0.012252209708094597,
+            0.007843539118766785,
+            0.015872985124588013,
+            0.011340375989675522,
+            -0.010800357908010483,
+            -0.029143260791897774,
+            0.018767835572361946,
+            -0.0001229010958923027,
+            0.00265803886577487,
+            -0.00462113693356514,
+            0.008958985097706318,
+            0.012677141465246677,
+            -0.011685633100569248,
+            0.009206862188875675,
+            -0.013040103949606419,
+            0.017678946256637573,
+            0.01843143068253994,
+            0.0008891483303159475,
+            -0.02121119573712349,
+            0.007431885693222284,
+            0.02103414013981819,
+            0.0035123294219374657,
+            -0.004178499337285757,
+            -0.003775698831304908,
+            -0.01669629104435444,
+            -0.0017550581833347678,
+            0.0016034548170864582,
+            0.03218860924243927,
+            -0.00483360281214118,
+            -0.008383556269109249,
+            0.004625563044101,
+            0.00016543580568395555,
+            0.007883376441895962,
+            0.015412641689181328,
+            0.004811470862478018,
+            0.005701172631233931,
+            -0.012376148253679276,
+            0.011667927727103233,
+            0.024451302364468575,
+            0.011154468171298504,
+            0.0030608391389250755,
+            -0.0005566168110817671,
+            -0.000995934708043933,
+            0.016528088599443436,
+            -0.015014268457889557,
+            -0.006582021247595549,
+            -0.007538118865340948,
+            0.019033417105674744,
+            -0.0024499993305653334,
+            -0.006679401732981205,
+            0.014270637184381485,
+            -0.0009505642810836434,
+            0.004501624498516321,
+            -0.010375426150858402,
+            0.01335880346596241,
+            0.008852752856910229,
+            0.014137845486402512,
+            -0.010242634452879429,
+            -0.018024204298853874,
+            0.005386899691075087,
+            -0.0026602521538734436,
+            -0.021724654361605644,
+            0.007737305480986834,
+            -0.010800357908010483,
+            0.008233060128986835,
+            -0.0007657631067559123,
+            0.00255401898175478,
+            0.0014551711501553655,
+            0.0009533307747915387,
+            -0.0058472431264817715,
+            0.007276962511241436,
+            -0.013465036638081074,
+            -0.0003488537622615695,
+            -0.007347784470766783,
+            -0.028753740713000298,
+            0.0026735311839729548,
+            -5.22865702805575e-05,
+            0.005807405803352594,
+            0.009383917786180973,
+            0.00576314190402627,
+            0.003853160422295332,
+            0.014120140112936497,
+            -0.027868464589118958,
+            -0.0041231694631278515,
+            -0.00735221104696393,
+            0.02032591961324215,
+            0.012544349767267704,
+            -0.01053477544337511,
+            -0.007847964763641357,
+            0.0048778667114675045,
+            0.011783013120293617,
+            -0.0097291748970747,
+            0.008365850895643234,
+            0.02777993679046631,
+            -0.009844260290265083,
+            -0.02131742797791958,
+            0.0007956410991027951,
+            0.0042448947206139565,
+            0.01851995848119259,
+            0.025141816586256027,
+            0.01249123364686966,
+            -0.0028793576639145613,
+            -0.009897377341985703,
+            -0.013305687345564365,
+            0.029160967096686363,
+            -0.006887441501021385,
+            -0.01245582289993763,
+            -0.011871540918946266,
+            0.010676419362425804,
+            -0.026930073276162148,
+            0.0032046963460743427,
+            -0.005621497984975576,
+            -0.0028793576639145613,
+            -0.004366619978100061,
+            -0.007631072774529457,
+            -0.006891867611557245,
+            -0.018785540014505386,
+            -0.005431163590401411,
+            0.012473528273403645,
+            0.006834324914962053,
+            0.006236764136701822,
+            -0.028311101719737053,
+            -0.009587530978024006,
+            -0.015270997770130634,
+            -0.017103517428040504,
+            -0.00213904632255435,
+            -0.031055456027388573,
+            -0.03293224051594734,
+            -0.017678946256637573,
+            -0.014784096740186214,
+            -6.719100929331034e-05,
+            -0.004089971538633108,
+            -0.01467786356806755,
+            0.0022884365171194077,
+            0.009684910997748375,
+            0.01335880346596241,
+            0.0109862657263875,
+            0.0015591910341754556,
+            -0.022326640784740448,
+            0.012553202919661999,
+            -0.017555007711052895,
+            -0.023672260344028473,
+            0.004603431094437838,
+            0.011313817463815212,
+            -0.0026204148307442665,
+            -0.0018491186201572418,
+            0.004492771811783314,
+            0.006050856318324804,
+            0.005223123822361231,
+            -0.044830337166786194,
+            -0.009525561705231667,
+            0.0070423646830022335,
+            0.0040302155539393425,
+            -0.0049708206206560135,
+            -0.013403067365288734,
+            0.012066301889717579,
+            -0.01659005880355835,
+            -0.027886170893907547,
+            -0.009906229563057423,
+            -0.016510384157299995,
+            0.002213188214227557,
+            -0.009321948513388634,
+            -0.0011386852711439133,
+            0.03220631182193756,
+            0.017448775470256805,
+            0.005417884327471256,
+            0.0003460872976575047,
+            0.003917342983186245,
+            0.008715534582734108,
+            -0.010481659322977066,
+            0.0014363591326400638,
+            0.008299455046653748,
+            0.01290731318295002,
+            0.005878227762877941,
+            0.0020394527819007635,
+            -0.013341098092496395,
+            0.025584453716874123,
+            0.01467786356806755,
+            0.01010984368622303,
+            0.0006910680094733834,
+            0.008109121583402157,
+            0.006732518319040537,
+            0.023760788142681122,
+            0.0040589869022369385,
+            0.00846765749156475,
+            -0.007936492562294006,
+            0.0068166195414960384,
+            -0.013562416657805443,
+            -0.021441366523504257,
+            0.015359525568783283,
+            -0.0060729882679879665,
+            -0.03436638414859772,
+            0.02032591961324215,
+            -0.007077775429934263,
+            0.002996656810864806,
+            -0.008866031654179096,
+            -0.011375786736607552,
+            -0.00276205874979496,
+            0.017032695934176445,
+            0.0061659421771764755,
+            -0.006086267530918121,
+            0.0008327120449393988,
+            0.0073212264105677605,
+            -0.0027155817952007055,
+            -0.018838657066226006,
+            -0.0022076552268117666,
+            -0.004240468610078096,
+            0.004306863993406296,
+            -0.053683090955019,
+            -0.016811376437544823,
+            0.012482380494475365,
+            -0.009003248997032642,
+            -0.01747533306479454,
+            0.009401623159646988,
+            -0.008848326280713081,
+            -0.018767835572361946,
+            0.004687532316893339,
+            0.008812914602458477,
+            0.002602709224447608,
+            0.0020660110749304295,
+            0.007502707652747631,
+            0.021405955776572227,
+            0.01153513602912426,
+            -0.016722848638892174,
+            0.0021600716281682253,
+            -0.012677141465246677,
+            -0.000435168098192662,
+            -0.019830165430903435,
+            -0.0169353149831295,
+            -0.00480261817574501,
+            0.010437395423650742,
+            0.04479492828249931,
+            0.013491595163941383,
+            0.010685272514820099,
+            -0.0007209460018202662,
+            0.0005815151962451637,
+            0.001700835069641471,
+            -0.04139547049999237,
+            -0.0019719505216926336,
+            -0.005165581125766039,
+            -0.005692319944500923,
+            -0.010835768654942513,
+            0.012951577082276344,
+            0.028169458732008934,
+            0.02868291735649109,
+            -0.002775337779894471,
+            0.0073831956833601,
+            -2.913454591180198e-05,
+            0.00390185066498816,
+            0.008056004531681538,
+            -0.013075515627861023,
+            0.015085089951753616,
+            -0.0016012416454032063,
+            0.009791144169867039,
+            0.009782291017472744,
+            -0.0008526307065039873,
+            0.014491955749690533,
+            -0.015846427530050278,
+            0.010764947161078453,
+            -0.00964064709842205,
+            0.004149727523326874,
+            0.008998822420835495,
+            0.03764190524816513,
+            0.002179990289732814,
+            -0.01659005880355835,
+            -0.01566937193274498,
+            -0.030063947662711143,
+            0.029692132025957108,
+            -0.013270275667309761,
+            0.016997285187244415,
+            0.017245162278413773,
+            -0.0015647240215912461,
+            0.004930983297526836,
+            0.010463953018188477,
+            0.0017318197060376406,
+            0.011375786736607552,
+            0.014509661123156548,
+            0.00969376415014267,
+            -0.023300444707274437,
+            -0.02013115957379341,
+            0.001953138504177332,
+            -0.015262145549058914,
+            -0.014341458678245544,
+            -0.007679762784391642,
+            -0.011189878918230534,
+            0.0072326986119151115,
+            -0.014854918234050274,
+            0.021724654361605644,
+            -0.016829082742333412,
+            0.0049752467311918736,
+            -0.02944425493478775,
+            -0.006621859036386013,
+            0.0034459338057786226,
+            -0.012066301889717579,
+            -0.00028384136385284364,
+            -0.008308308199048042,
+            -0.002790830098092556,
+            0.017740916460752487,
+            -0.004415310453623533,
+            -0.003992591518908739,
+            0.011136762797832489,
+            0.006701533682644367,
+            0.010100990533828735,
+            -0.015642814338207245,
+            0.005550675559788942,
+            -0.0037646330893039703,
+            0.014005054719746113,
+            -0.018679307773709297,
+            -0.004740648902952671,
+            0.0027664850931614637,
+            0.024327363818883896,
+            0.012889607809484005,
+            0.008941279724240303,
+            0.005895933136343956,
+            -0.007852391339838505,
+            -0.009136040695011616,
+            -0.03739402815699577,
+            0.030860694125294685,
+            -0.016147419810295105,
+            0.022822394967079163,
+            -0.003609709907323122,
+            0.0025119686033576727,
+            -0.0006523372139781713,
+            0.00687416223809123,
+            0.0042448947206139565,
+            -0.021494483575224876,
+            0.013102073222398758,
+            -0.0231942106038332,
+            0.008184369653463364,
+            0.003359619528055191,
+            -0.012588613666594028,
+            0.03693368285894394,
+            -0.002405735431239009,
+            0.014784096740186214,
+            -0.02855897881090641,
+            0.0024079487193375826,
+            0.009764585644006729,
+            -0.005590513348579407,
+            -0.005453295540064573,
+            -0.02942654862999916,
+            -0.0010999544756487012,
+            0.0009915082482621074,
+            0.00035881312214769423,
+            0.006944984197616577,
+            -0.01922817900776863,
+            -0.03128562867641449,
+            0.008184369653463364,
+            0.008688976988196373,
+            0.002562871901318431,
+            0.003731435164809227,
+            -0.0001748418580973521,
+            -0.0018535450799390674,
+            0.014987709932029247,
+            0.01242926437407732,
+            0.019706226885318756,
+            0.03284371271729469,
+            -0.02862980216741562,
+            -0.015713635832071304,
+            -0.002653612522408366,
+            -0.016368739306926727,
+            -0.02569068782031536,
+            -0.02222040854394436,
+            0.0044706398621201515,
+            0.0073212264105677605,
+            0.013562416657805443,
+            -0.030896106734871864,
+            0.013287981040775776,
+            -0.005621497984975576,
+            0.004984099417924881,
+            -0.03032952919602394,
+            -0.026345791295170784,
+            -0.006254469510167837,
+            -0.011765307746827602,
+            0.03208237513899803,
+            -0.010021315887570381,
+            0.004275879357010126,
+            -0.01522673387080431,
+            -0.013456183485686779,
+            -0.00711761275306344,
+            0.0019420725293457508,
+            -0.01335880346596241,
+            -0.012296473607420921,
+            0.038243889808654785,
+            -0.001566937193274498,
+            -0.014598188921809196,
+            0.019723933190107346,
+            0.005546249449253082,
+            0.005501985549926758,
+            0.0040943981148302555,
+            0.013252570293843746,
+            -0.008441099897027016,
+            -0.006404966581612825,
+            -0.0004611730691976845,
+            0.007299094460904598,
+            0.002870504977181554,
+            0.024699179455637932,
+            -0.0042515345849096775,
+            2.7509237042977475e-05,
+            -0.007728452794253826,
+            0.0005483173299580812,
+            0.01844913698732853,
+            0.005422310903668404,
+            -0.007051217369735241,
+            -0.0025717245880514383,
+            0.005298372358083725,
+            -0.03227713704109192,
+            -0.017749767750501633,
+            0.01713007688522339,
+            0.010065579786896706,
+            0.009950493462383747,
+            -0.00028065990773029625,
+            -0.03149809315800667,
+            -0.007901081815361977,
+            0.002615988254547119,
+            0.0026624652091413736,
+            -0.0015160338953137398,
+            0.018820952624082565,
+            -0.027301888912916183,
+            0.002101422054693103,
+            -0.0032511733006685972,
+            0.02108725719153881,
+            0.008445525541901588,
+            0.0005848349537700415,
+            0.024504419416189194,
+            0.0036561868619173765,
+            0.013704060576856136,
+            -0.0018258801428601146,
+            -0.010322309099137783,
+            -0.01747533306479454,
+            -0.010003610514104366,
+            -3.72092254110612e-05,
+            0.0010119802318513393,
+            0.001832519774325192,
+            -0.00021232773724477738,
+            0.0015082877362146974,
+            -0.0038885714020580053,
+            -0.0011630303924903274,
+            0.015970366075634956,
+            0.0014817294431850314,
+            -0.004446295090019703,
+            0.014111286960542202,
+            -0.002376964082941413,
+            -0.007631072774529457,
+            0.006838751491159201,
+            0.0003211889124941081,
+            0.003240107325837016,
+            0.009224568493664265,
+            0.011844982393085957,
+            -0.0016045613447204232,
+            0.020715441554784775,
+            -0.010472806170582771,
+            0.008480937220156193,
+            -0.008888163603842258,
+            -0.0004888379480689764,
+            -0.017519596964120865,
+            0.0034060964826494455,
+            -0.002910342300310731,
+            -0.01423522550612688,
+            -0.020485268905758858,
+            0.006931704934686422,
+            0.006577595137059689,
+            0.00115639076102525,
+            -0.003638481255620718,
+            0.014872624538838863,
+            -0.0038332417607307434,
+            0.0045591676607728004,
+            -0.004886719398200512,
+            -0.006958263460546732,
+            -0.016439560800790787,
+            0.01518247090280056,
+            0.0015492317033931613,
+            0.005307225044816732,
+            0.002216507913544774,
+            0.0034038834273815155,
+            -0.00872881431132555,
+            0.001222786377184093,
+            0.017254015430808067,
+            0.005462148226797581,
+            -0.009994757361710072,
+            0.004408670589327812,
+            -0.009224568493664265,
+            -0.004098824225366116,
+            0.0075115603394806385,
+            -0.009339653886854649,
+            -0.01057018619030714,
+            0.02678842842578888,
+            -0.005355915054678917,
+            -0.0005236956640146673,
+            -0.012659436091780663,
+            -0.004220549948513508,
+            0.0115971053019166,
+            -0.000843777961563319,
+            -0.009251126088202,
+            -0.004705238156020641,
+            0.008958985097706318,
+            -0.01152628380805254,
+            -0.003518969053402543,
+            0.014580483548343182,
+            -0.009853113442659378,
+            -0.009560972452163696,
+            0.02021968737244606,
+            0.0026890235021710396,
+            -0.011650222353637218,
+            -0.006834324914962053,
+            0.0004075032484252006,
+            -0.005581660196185112,
+            0.03217090293765068,
+            0.006772355642169714,
+            0.00970261637121439,
+            0.023601438850164413,
+            0.001833626301959157,
+            0.009543267078697681,
+            -0.0042161233723163605,
+            0.012615172192454338,
+            0.0036716791801154613,
+            0.00873766653239727,
+            0.002321634441614151,
+            0.021724654361605644,
+            -0.0025009026285260916,
+            0.0042161233723163605,
+            0.007772716693580151,
+            -0.011482019908726215,
+            -0.004132022149860859,
+            -0.004036854952573776,
+            -0.005453295540064573,
+            -0.00089689448941499,
+            0.0005737690371461213,
+            0.010880032554268837,
+            0.009950493462383747,
+            -0.0015060745645314455,
+            0.015864131972193718,
+            0.010428542271256447,
+            -0.01422637328505516,
+            -0.0011630303924903274,
+            0.0050682006403803825,
+            0.00449055852368474,
+            0.0009666099213063717,
+            -0.030595112591981888,
+            -0.01291616540402174,
+            0.003156006336212158,
+            -0.015563138760626316,
+            0.015864131972193718,
+            -0.019847871735692024,
+            -0.02379619888961315,
+            -0.019670816138386726,
+            0.0038841450586915016,
+            -0.015005415305495262,
+            -0.01374832447618246,
+            0.021795475855469704,
+            0.007839112542569637,
+            0.0015768965240567923,
+            -0.02122890017926693,
+            0.023442087695002556,
+            -0.009808849543333054,
+            0.023229623213410378,
+            -0.01200433261692524,
+            -0.002098102355375886,
+            -0.011765307746827602,
+            0.01561625488102436,
+            -0.010897737927734852,
+            0.0025163949467241764,
+            0.009516708552837372,
+            0.01659005880355835,
+            0.0030652654822915792,
+            0.0007928746636025608,
+            0.0017882559914141893,
+            0.00480261817574501,
+            0.014438839629292488,
+            0.006834324914962053,
+            0.009410476312041283,
+            -0.01249123364686966,
+            -0.010242634452879429,
+            0.004078905563801527,
+            0.005931344348937273,
+            -0.00015644473023712635,
+            -0.02207876369357109,
+            0.002936900593340397,
+            0.009897377341985703,
+            -0.0005375280743464828,
+            -0.01839601993560791,
+            -0.006210205610841513,
+            -0.009534413926303387,
+            0.027921581640839577,
+            0.0097291748970747,
+            -0.01855536922812462,
+            -0.008950132876634598,
+            0.021565305069088936,
+            0.012411559000611305,
+            0.015235587023198605,
+            -0.016165126115083694,
+            0.013668649829924107,
+            0.008264044299721718,
+            0.0013511513825505972,
+            0.0029546061996370554,
+            -0.01053477544337511,
+            0.028824562206864357,
+            0.0048822928220033646,
+            0.0009881885489448905,
+            -0.013491595163941383,
+            -0.007538118865340948,
+            -0.008264044299721718,
+            0.011455461382865906,
+            0.005957902409136295,
+            0.002804109361022711,
+            0.001394308521412313,
+            -0.007985183037817478,
+            -0.015368377789855003,
+            -0.0019985088147222996,
+            -0.003357406472787261,
+            0.0211934894323349,
+            -0.0032179756090044975,
+            0.003782338462769985,
+            -0.006152662914246321,
+            0.0018004284938797355,
+            -0.009968198835849762,
+            -0.0057188780046999454,
+            -0.0020803967490792274,
+            0.00804272573441267,
+            -0.018218964338302612,
+            0.015979217365384102,
+            -0.020857084542512894,
+            0.018006497994065285,
+            0.028275690972805023,
+            0.00324232061393559,
+            -0.006785634905099869,
+            -0.002070437418296933,
+            0.008472084067761898,
+            -0.034649673849344254,
+            0.003306502941995859,
+            0.017696652561426163,
+            -0.006909573450684547,
+            -0.013615533709526062,
+            -0.018856363371014595,
+            -0.0012172535061836243,
+            -0.007927639409899712,
+            0.005563954822719097,
+            0.005856095813214779,
+            -0.0010518176713958383,
+            0.018661601468920708,
+            -0.002057158388197422,
+            -0.01467786356806755,
+            -0.008131253533065319,
+            -0.0221495870500803,
+            0.01480180211365223,
+            -0.012252209708094597,
+            -0.013465036638081074,
+            0.012818785384297371,
+            0.009746880270540714,
+            -0.019936397671699524,
+            0.002547379583120346,
+            -0.02747894451022148,
+            -0.008480937220156193,
+            -0.005369194317609072,
+            -0.01014525443315506,
+            -0.011685633100569248,
+            -0.004138661548495293,
+            -0.011765307746827602,
+            0.003943901043385267,
+            0.0002846713177859783,
+            -0.011986627243459225,
+            -0.004607857670634985,
+            0.007153023965656757,
+            -0.012287620455026627,
+            -0.009295389987528324,
+            0.009746880270540714,
+            0.004368833266198635,
+            -0.019121944904327393,
+            -0.031657442450523376,
+            0.008657991886138916,
+            -0.03337487578392029,
+            0.004306863993406296,
+            0.015492316335439682,
+            0.004939835984259844,
+            -0.007276962511241436,
+            0.003184777684509754,
+            0.021972531452775,
+            0.014854918234050274,
+            -0.006719239056110382,
+            0.017103517428040504,
+            0.0036296285688877106,
+            0.006617432460188866,
+            -0.012606319040060043,
+            -1.0452126844029408e-05,
+            0.01843143068253994,
+            0.012225651182234287,
+            -0.0028749313205480576,
+            0.01335880346596241,
+            -0.006883014924824238,
+            0.008609301410615444,
+            -0.00020900795061606914,
+            -0.007670910097658634,
+            -0.018661601468920708,
+            -0.006342997308820486,
+            -0.015474610961973667,
+            -0.006095120217651129,
+            0.008427820168435574,
+            -0.005993313621729612,
+            -0.006312012672424316,
+            -0.010605597868561745,
+            0.010286898352205753,
+            -0.017484186217188835,
+            0.00045508681796491146,
+            -0.02572609856724739,
+            -0.00300329620949924,
+            -0.013659796677529812,
+            0.009392770007252693,
+            0.01156169455498457,
+            0.005687893368303776,
+            -0.015749046579003334,
+            -0.019599994644522667,
+            0.0007768290233798325,
+            -0.0029789512045681477,
+            -0.021529894322156906,
+            -0.007684188894927502,
+            -0.018502252176404,
+            -0.012376148253679276,
+            -0.009437033906579018,
+            0.014075876213610172,
+            0.003014362184330821,
+            -0.03256042301654816,
+            0.006205779500305653,
+            -0.027018601074814796,
+            -0.016846787184476852,
+            -0.01201318483799696,
+            -0.0021047419868409634,
+            0.0046565476804971695,
+            0.020113453269004822,
+            0.008516347967088223,
+            0.026575962081551552,
+            -3.7174642784520984e-05,
+            0.005205418448895216,
+            0.0012891821097582579,
+            0.011933510191738605,
+            0.009817702695727348,
+            -0.012323031201958656,
+            0.01620939001441002,
+            -0.008356998674571514,
+            0.0020128944888710976,
+            -0.022716162726283073,
+            -0.0012936084531247616,
+            -0.014438839629292488,
+            -0.006281028036028147,
+            -0.000598667364101857,
+            -0.01834290288388729,
+            -0.006294306833297014,
+            0.023743081837892532,
+            -0.019051123410463333,
+            0.008998822420835495,
+            0.008361424319446087,
+            -0.0003070798411499709,
+            0.01467786356806755,
+            0.029816070571541786,
+            -0.010915443301200867,
+            0.001500541577115655,
+            0.01944064348936081,
+            0.004315716680139303,
+            0.008503069169819355,
+            -0.0008183263125829399,
+            0.0036207756493240595,
+            0.012084007263183594,
+            -0.01155284233391285,
+            0.00648906733840704,
+            -0.005807405803352594,
+            0.008423393592238426,
+            -0.011269553564488888,
+            0.0041585806757211685,
+            -0.01949376054108143,
+            -0.01601462997496128,
+            0.019918693229556084,
+            0.012898460030555725,
+            -0.0003311482723802328,
+            -0.0051036118529737,
+            -0.03233025223016739,
+            -0.0008686763467267156,
+            -0.009817702695727348,
+            0.007431885693222284,
+            -0.013571269810199738,
+            0.011304965242743492,
+            -0.00642267195507884,
+            0.014146698638796806,
+            -0.013677502982318401,
+            0.018856363371014595,
+            -0.011269553564488888,
+            -0.01062330324202776,
+            0.017121223732829094,
+            0.009755733422935009,
+            -0.0037557801697403193,
+            0.029231788590550423,
+            -0.001606774516403675,
+            0.0029501798562705517,
+            -0.002301715547218919,
+            -0.030878400430083275,
+            0.0023747507948428392,
+            0.0028882105834782124,
+            0.00918030459433794,
+            0.004149727523326874,
+            -0.005820684600621462,
+            0.030701344832777977,
+            0.012154828757047653,
+            0.016997285187244415,
+            -0.011437756009399891,
+            -0.0008044939022511244,
+            0.020998729392886162,
+            0.013615533709526062,
+            0.0015481250593438745,
+            0.007400901056826115,
+            -0.0008199862204492092,
+            0.0007674229564145207,
+            -0.021777771413326263,
+            0.010286898352205753,
+            -0.018732424825429916,
+            -0.010428542271256447,
+            -0.002987803891301155,
+            -0.003027641447260976,
+            -0.003027641447260976,
+            0.0010352187091484666,
+            0.00528509309515357,
+            0.0015591910341754556,
+            0.023654554039239883,
+            -0.0009311988833360374,
+            -0.0037491407711058855,
+            -0.028134047985076904,
+            -0.017864855006337166,
+            0.003386177821084857,
+            0.011322670616209507,
+            0.029090143740177155,
+            0.0024920497089624405,
+            -0.009198009967803955,
+            -0.008330440148711205,
+            -0.012615172192454338,
+            0.01782059110701084,
+            0.010773799382150173,
+            -0.0025186080019921064,
+            0.0017229669028893113,
+            -0.011136762797832489,
+            0.014660158194601536,
+            -0.014607041142880917,
+            0.011127909645438194,
+            0.014120140112936497,
+            0.0019420725293457508,
+            -0.01249123364686966,
+            -0.0005477640661410987,
+            0.003862013341858983,
+            0.012145976535975933,
+            0.002469917992129922,
+            0.009171451441943645,
+            -0.02592085860669613,
+            -0.00666169635951519,
+            0.005679040681570768,
+            0.006648417096585035,
+            0.02296403981745243,
+            0.0152444401755929,
+            0.011225289665162563,
+            0.01108364574611187,
+            0.012349589727818966,
+            0.008600449189543724,
+            0.015005415305495262,
+            -0.013881116174161434,
+            0.0025938565377146006,
+            -0.0019044483778998256,
+            0.016492677852511406,
+            0.007812554016709328,
+            0.010729536414146423,
+            -0.02029050886631012,
+            0.0032069096341729164,
+            0.013429625891149044,
+            -0.008817341178655624,
+            0.008551758714020252,
+            -0.008410114794969559,
+            0.01378373522311449,
+            -0.005448868963867426,
+            0.0009323054691776633,
+            -0.0012393853394314647,
+            0.0030873974319547415,
+            0.032525014132261276,
+            -0.004218336660414934,
+            -0.004811470862478018,
+            0.01570478267967701,
+            -0.0045591676607728004,
+            0.0028240280225872993,
+            -0.01525329239666462,
+            0.00879078358411789,
+            0.02480541169643402,
+            0.00883504655212164,
+            -0.016253653913736343,
+            -0.014483102597296238,
+            0.004961967933923006,
+            -0.009321948513388634,
+            0.0011198731372132897,
+            0.007772716693580151,
+            -0.004424163140356541,
+            0.009321948513388634,
+            -0.0013799227308481932,
+            -0.0008244125638157129,
+            0.0049708206206560135,
+            -0.039164576679468155,
+            0.00252303434535861,
+            -0.015350672416388988,
+            0.006103972904384136,
+            0.004452934488654137,
+            -0.0024787706788629293,
+            -0.006462509278208017,
+            -0.006555463187396526,
+            -0.001566937193274498,
+            -0.05403720214962959,
+            0.0032201886642724276,
+            -0.0014573843218386173,
+            0.02393784187734127,
+            -0.01062330324202776,
+            0.00969376415014267,
+            0.015014268457889557,
+            0.013093221001327038,
+            -0.003078544745221734,
+            -0.0012659436324611306,
+            0.013411919586360455,
+            -0.0211934894323349,
+            -0.018626190721988678,
+            0.014837212860584259,
+            -0.007608940824866295,
+            0.00011750645353458822,
+            -0.004050134215503931,
+            0.004798191599547863,
+            -0.004634415730834007,
+            0.0033397008664906025,
+            0.013270275667309761,
+            -0.012730257585644722,
+            -0.01924588344991207,
+            -0.007515986915677786,
+            -0.006338570732623339,
+            -0.019511466845870018,
+            -0.016528088599443436,
+            0.00621463218703866,
+            -0.01714778132736683,
+            0.003678318578749895,
+            -0.0005380813381634653,
+            -0.00555510213598609,
+            0.013633239082992077,
+            0.001355577725917101,
+            0.0020538384560495615,
+            0.0007336718845181167,
+            0.009233420714735985,
+            -0.017431069165468216,
+            0.017652388662099838,
+            0.011216437444090843,
+            -0.014483102597296238,
+            0.00177940318826586,
+            -0.004669826943427324,
+            0.0026646784972399473,
+            0.021494483575224876,
+            -0.0065421839244663715,
+            0.0021224473603069782,
+            0.018767835572361946,
+            -0.0013876688899472356,
+            0.011198732070624828,
+            -0.003757993457838893,
+            0.005298372358083725,
+            -0.012101712636649609,
+            0.0012504513142630458,
+            0.00648906733840704,
+            -0.007812554016709328,
+            0.009065218269824982,
+            -0.04504280537366867,
+            0.012216798029839993,
+            -0.009144892916083336,
+            -0.024451302364468575,
+            -0.006059709005057812,
+            -0.0063164387829601765,
+            -0.0006456976407207549,
+            -0.01653694175183773,
+            0.013137484900653362,
+            -0.011216437444090843,
+            0.006745797581970692,
+            -0.012075154110789299,
+            0.0019885494839400053,
+            0.007839112542569637,
+            0.0003709856537170708,
+            0.002027280395850539,
+            -0.006068561691790819,
+            0.003687171498313546,
+            0.004926556721329689,
+            -0.002126873703673482,
+            -0.008989970199763775,
+            0.013544711284339428,
+            -0.0034171624574810266,
+            0.015952659770846367,
+            0.010756094008684158,
+            -0.008454378694295883,
+            -0.024681473150849342,
+            0.0157224889844656,
+            -0.004678679630160332,
+            -0.008941279724240303,
+            -0.0021556452848017216,
+            0.010065579786896706,
+            -3.074256528634578e-05,
+            -0.008392409421503544,
+            0.0027532060630619526,
+            0.009879671968519688,
+            0.007259257137775421,
+            -0.008967838250100613,
+            -0.004687532316893339,
+            -0.015456905588507652,
+            0.009525561705231667,
+            0.002750992774963379,
+            0.007462870329618454,
+            0.013898821547627449,
+            0.004132022149860859,
+            0.013376508839428425,
+            0.0004337848804425448,
+            -0.00759123545140028,
+            -0.026983190327882767,
+            -0.015766752883791924,
+            -0.010950854979455471,
+            0.0014153338270261884,
+            -0.0011785227106884122,
+            0.02018427476286888,
+            -0.0051611545495688915,
+            0.0046919588930904865,
+            0.0074540176428854465,
+            0.042032867670059204,
+            0.014182109385728836,
+            0.007887802086770535,
+            0.0007380982278846204,
+            -0.013819146901369095,
+            0.00038481809315271676,
+            -0.010463953018188477,
+            -0.03153350576758385,
+            0.009897377341985703,
+            -0.004129808861762285,
+            0.02315879985690117,
+            -0.014819507487118244,
+            0.014589335769414902,
+            -0.004510477185249329,
+            -0.01808617264032364,
+            -0.016944168135523796,
+            0.020626913756132126,
+            0.022840101271867752,
+            0.016377592459321022,
+            4.270761564839631e-05,
+            0.019511466845870018,
+            -0.007055643480271101,
+            0.0161562729626894,
+            -0.020113453269004822,
+            -0.004576873034238815,
+            0.006555463187396526,
+            0.02121119573712349,
+            0.0013467249227687716,
+            0.012765669263899326,
+            0.00711318664252758,
+            -0.0012825424782931805,
+            -0.001180735882371664,
+            -0.012703699991106987,
+            0.0019011285621672869,
+            -0.005174433812499046,
+            -0.0020471990574151278,
+            0.013252570293843746,
+            -0.01009213738143444,
+            -0.01837831363081932,
+            0.01149087306112051,
+            -0.010003610514104366,
+            0.0032622392755001783,
+            0.011234142817556858,
+            0.022344347089529037,
+            0.005515264812856913,
+            0.010880032554268837,
+            -0.006551036611199379,
+            8.638350118417293e-05,
+            -0.022397464141249657,
+            -0.01004787441343069,
+            -0.004138661548495293,
+            -0.01114561501890421,
+            0.003333061235025525,
+            -0.007586808875203133,
+            -0.025495927780866623,
+            0.00041497277561575174,
+            0.007458443753421307,
+            0.030099358409643173,
+            -0.013597827404737473,
+            0.02112266793847084,
+            0.0011148934718221426,
+            -0.0029501798562705517,
+            -0.010649860836565495,
+            -0.01425293181091547,
+            0.00576756801456213,
+            -0.01195121556520462,
+            -0.0014429986476898193,
+            6.57731870887801e-05,
+            -0.01830749213695526,
+            -3.8903697713976726e-05,
+            0.014005054719746113,
+            -0.037323202937841415,
+            -0.003957180306315422,
+            0.004992952570319176,
+            -0.0009284323896281421,
+            0.01387226302176714,
+            0.005643629468977451,
+            0.0036296285688877106,
+            0.0027554191183298826,
+            -0.003492410760372877,
+            0.00022380864538718015,
+            0.01929900050163269,
+            -0.015784457325935364,
+            0.02843504026532173,
+            0.00831273477524519,
+            -0.004891145508736372,
+            -0.0008930214098654687,
+            0.004503837786614895,
+            -0.028753740713000298,
+            0.010897737927734852,
+            -0.015952659770846367,
+            -0.018643897026777267,
+            0.011313817463815212,
+            0.015138207003474236,
+            -0.0025562322698533535,
+            0.002536313608288765,
+            0.015899542719125748,
+            0.011384639889001846,
+            0.005231976509094238,
+            -0.0016753834206610918,
+            -0.014146698638796806,
+            0.00203502643853426,
+            -0.01936982199549675,
+            0.009074071422219276,
+            -0.020024925470352173,
+            -0.0036318416241556406,
+            0.00849421601742506,
+            0.0038022571243345737,
+            0.015811016783118248,
+            0.009268831461668015,
+            0.018112732097506523,
+            -0.015359525568783283,
+            0.00891914777457714,
+            0.03973115235567093,
+            0.01147316675633192,
+            0.013677502982318401,
+            -0.005856095813214779,
+            -0.0005881547695025802,
+            0.025460515171289444,
+            -0.005262961145490408,
+            -0.008414541371166706,
+            0.020502975210547447,
+            0.018909478560090065,
+            0.014819507487118244,
+            -0.001221679849550128,
+            0.009065218269824982,
+            0.029214082285761833,
+            0.00552411749958992,
+            -0.005192139185965061,
+            -0.019511466845870018,
+            -0.006373981945216656,
+            -0.0020903560798615217,
+            0.013164042495191097,
+            0.007166303228586912,
+            0.016846787184476852,
+            -0.028966205194592476,
+            0.0034702790435403585,
+            0.003501263679936528,
+            -0.0001507734414190054,
+            0.013376508839428425,
+            0.02021968737244606,
+            0.003344127209857106,
+            -0.003842094447463751,
+            0.006493493914604187,
+            0.005174433812499046,
+            0.03125021606683731,
+            -0.025283461436629295,
+            -0.0037270088214427233,
+            0.0034038834273815155,
+            -0.015279850922524929,
+            0.013606680557131767,
+            -0.004523756448179483,
+            -0.002558445557951927,
+            -0.007852391339838505,
+            -0.010490511544048786,
+            0.016713997349143028,
+            0.01419096253812313,
+            -0.015456905588507652,
+            -0.025177227333188057,
+            -0.02117578312754631,
+            0.007122039329260588,
+            -0.02108725719153881,
+            -0.0072858151979744434,
+            -0.01009213738143444,
+            -0.006387260742485523,
+            -0.009578677825629711,
+            -0.006475788541138172,
+            -0.009746880270540714,
+            0.005563954822719097,
+            0.0007120932568795979,
+            0.003638481255620718,
+            0.0007353317341767251,
+            -0.01377488300204277,
+            -0.011136762797832489,
+            0.021671537309885025,
+            0.004860160872340202,
+            0.01296928245574236,
+            0.0026071355678141117,
+            0.005231976509094238,
+            0.006250043399631977,
+            -0.04298896715044975,
+            -0.006050856318324804,
+            -0.006679401732981205,
+            0.009357359260320663,
+            0.005563954822719097,
+            0.022716162726283073,
+            -0.012110565789043903,
+            0.015766752883791924,
+            -0.010437395423650742,
+            0.010039021261036396,
+            -0.010242634452879429,
+            0.0026646784972399473,
+            0.008215353824198246,
+            -0.007467296905815601,
+            0.0016565712867304683,
+            -0.004607857670634985,
+            0.007679762784391642,
+            -0.0018391592893749475,
+            0.007608940824866295,
+            -0.00960523635149002,
+            -0.015483464114367962,
+            -0.0023526190780103207,
+            -0.011605958454310894,
+            -0.017864855006337166,
+            5.754288940806873e-05,
+            -0.008587169460952282,
+            -0.0015315262135118246,
+            -0.019688520580530167,
+            -0.01924588344991207,
+            0.011960068717598915,
+            -0.0020615847315639257,
+            0.009091776795685291,
+            0.013916526921093464,
+            0.007763864006847143,
+            0.015306408517062664,
+            0.013376508839428425,
+            -0.004452934488654137,
+            -0.013571269810199738,
+            -0.014208667911589146,
+            0.0023526190780103207,
+            -0.014943446032702923,
+            0.01935211755335331,
+            0.008210928179323673,
+            -0.0017307130619883537,
+            0.0011884820414707065,
+            -0.020998729392886162,
+            -0.0001062330265995115,
+            -0.0022209342569112778,
+            0.0015569778624922037,
+            -0.018590779975056648,
+            0.02200794219970703,
+            -0.018289785832166672,
+            0.002113594673573971,
+            0.019688520580530167,
+            0.029780659824609756,
+            0.0028461599722504616,
+            -0.012925018556416035,
+            0.0007309053908102214,
+            0.008427820168435574,
+            0.03898752108216286,
+            -0.007985183037817478,
+            0.001860184594988823,
+            -0.011853835545480251,
+            -0.010313456878066063,
+            0.00435555400326848,
+            0.006365128792822361,
+            0.006812192965298891,
+            -0.026522846892476082,
+            -0.013597827404737473,
+            -0.010375426150858402,
+            -0.013695208355784416,
+            -0.0058118319138884544,
+            0.04072266072034836,
+            0.001315740286372602,
+            -0.0010070005664601922,
+            -0.00669268099591136,
+            0.00023556621454190463,
+            -0.017280573025345802,
+            0.03047117404639721,
+            0.09419328719377518,
+            -0.008817341178655624,
+            -0.009552120231091976,
+            -0.025035584345459938,
+            0.009198009967803955,
+            -0.001912194536998868,
+            0.011906951665878296,
+            0.06069447100162506,
+            -0.020662324503064156,
+            0.000730352068785578,
+            -0.0007264789892360568,
+            0.009897377341985703,
+            -0.00963179487735033,
+            -0.0026248411741107702,
+            -0.00735663715749979,
+            -0.008888163603842258,
+            0.0017605911707505584,
+            -0.03140956535935402,
+            0.0005306118400767446,
+            -0.006272174883633852,
+            0.005506412126123905,
+            -0.014244078658521175,
+            0.004581299610435963,
+            -0.010818063281476498,
+            0.004220549948513508,
+            -0.013677502982318401,
+            -0.005448868963867426,
+            0.01297813467681408,
+            -0.019688520580530167,
+            -0.0007790421950630844,
+            0.004647694993764162,
+            -0.012641730718314648,
+            0.008392409421503544,
+            0.0026248411741107702,
+            -0.011278406716883183,
+            -0.0015846426831558347,
+            -0.015085089951753616,
+            0.028789151459932327,
+            0.0018834230722859502,
+            0.007259257137775421,
+            -0.009950493462383747,
+            0.00101751321926713,
+            0.01293387170881033,
+            -0.0021711376029998064,
+            0.017121223732829094,
+            0.013571269810199738,
+            -0.001765017514117062,
+            -0.008808488957583904,
+            0.01714778132736683,
+            -0.00046089643728919327,
+            -0.013314539566636086,
+            0.009136040695011616,
+            0.011216437444090843,
+            0.0027266477700322866,
+            0.0037535671144723892,
+            -0.0015724701806902885,
+            0.023548321798443794,
+            -0.017254015430808067,
+            -0.011482019908726215,
+            -0.004072266165167093,
+            -0.009330800734460354,
+            0.021441366523504257,
+            0.006307586096227169,
+            0.03397686406970024,
+            -0.009773438796401024,
+            -0.0012305325362831354,
+            -0.018590779975056648,
+            -0.007259257137775421,
+            -0.010189518332481384,
+            -0.005993313621729612,
+            0.013553564436733723,
+            0.0019299000268802047,
+            0.012694846838712692,
+            -0.019971810281276703,
+            -0.017156634479761124,
+            0.00960523635149002,
+            -0.02294633351266384,
+            -0.0011375787435099483,
+            0.004922130145132542,
+            -0.027213361114263535,
+            -0.004435229115188122,
+            0.008166664279997349,
+            -0.02121119573712349,
+            -0.025230344384908676,
+            0.002651399467140436,
+            -0.0007049004198051989,
+            -0.02025509811937809,
+            0.009711469523608685,
+            0.006015445105731487,
+            -0.008565038442611694,
+            -0.002060478087514639,
+            -0.002961245598271489,
+            -0.020024925470352173,
+            0.0076399254612624645,
+            -0.015058532357215881,
+            -0.008038299158215523,
+            -0.006174794863909483,
+            0.004988525994122028,
+            -0.0012703699758276343,
+            0.02114037238061428,
+            0.006338570732623339,
+            -0.004032428842037916,
+            -0.011623663827776909,
+            0.0018280934309586883,
+            0.008472084067761898,
+            0.017138928174972534,
+            0.02285780757665634,
+            -0.0019442857010290027,
+            -0.006750223692506552,
+            0.001236065523698926,
+            0.015403789468109608,
+            0.005147875752300024,
+            -0.010791505686938763,
+            0.025354282930493355,
+            0.006785634905099869,
+            -0.008033872582018375,
+            0.015483464114367962,
+            -0.012668288312852383,
+            0.001964204479008913,
+            -0.009977051988244057,
+            -0.008219780400395393,
+            0.0042780926451087,
+            -0.0007159663364291191,
+            0.024238836020231247,
+            0.005581660196185112,
+            0.014571630395948887,
+            0.010419690050184727,
+            0.012774521484971046,
+            0.022432874888181686,
+            -0.01201318483799696,
+            0.0019874428398907185,
+            -0.0067280917428433895,
+            0.0018535450799390674,
+            -0.0005510838236659765,
+            -0.007684188894927502,
+            -0.006741371005773544,
+            0.0050283633172512054,
+            0.009445887058973312,
+            -0.0017561647109687328,
+            0.010348867624998093,
+            0.019192766398191452,
+            0.014828360639512539,
+            0.005391326267272234,
+            -0.0064093926921486855,
+            -0.019954103976488113,
+            0.003248960245400667,
+            0.021016433835029602,
+            -0.0034724920988082886,
+            0.010410836897790432,
+            0.0014175469987094402,
+            0.01855536922812462,
+            0.00669268099591136,
+            0.0050637745298445225,
+            -0.00666169635951519,
+            0.008742093108594418,
+            -0.01646612025797367,
+            0.004581299610435963,
+            0.0034437207505106926,
+            0.02206105925142765,
+            -0.007493854966014624,
+            -0.0067236656323075294,
+            0.0002651675895322114,
+            0.031728263944387436,
+            0.01384570449590683,
+            0.010782652534544468,
+            -0.019564582034945488,
+            0.009392770007252693,
+            -0.006705959793180227,
+            0.0015547646908089519,
+            0.0007773823454044759,
+            0.0115971053019166,
+            -0.007436312269419432,
+            -0.005621497984975576,
+            0.011756455525755882,
+            0.009233420714735985,
+            0.00918030459433794,
+            -0.0030453468207269907,
+            -0.017555007711052895,
+            0.006170368287712336,
+            -0.0047849128022789955,
+            -0.012137123383581638,
+            0.026522846892476082,
+            0.0157224889844656,
+            0.032879121601581573,
+            -0.003069691825658083,
+            0.0019619911909103394,
+            0.011402345262467861,
+            0.0018015351379290223,
+            0.014908035285770893,
+            -0.01011869590729475,
+            -0.0109862657263875,
+            0.03392374888062477,
+            -0.01246467512100935,
+            0.008135679177939892,
+            -0.024150308221578598,
+            -0.026469729840755463,
+            -0.0028129620477557182,
+            0.001937646185979247,
+            0.0049708206206560135,
+            0.0027952566742897034,
+            -0.004528183024376631,
+            -0.007338931784033775,
+            0.01296928245574236,
+            -0.001329019432887435,
+            0.0037668461445719004,
+            0.008976691402494907,
+            -0.0044418685138225555,
+            0.0031139557249844074,
+            -0.002733287401497364,
+            -0.000756910361815244,
+            0.010959707200527191
+          ],
+          "keyphrases": [
+            "expert choice method",
+            "auto-regressive text generation",
+            "large memory footprint",
+            "sparsely gated networks",
+            "power saving techniques"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "3e9ef306-e62b-441c-b521-fd7367274358",
+      "type": "child",
+      "source": {
+        "id": "b570d71e-4f83-486e-b70f-b0c6217db455",
+        "properties": {
+          "page_content": "Mixture-Of-Experts With Expert Choice Routing\n\nYanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon Google, Mountain View, CA, USA {yanqiz, taole, hanxiaol, dunan, huangyp, vzhao, adai, zhifengc, qvl, jlaudon}@google.com\n\nAbstract\n\nSparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choice method. Instead of letting tokens select the top-k experts, we have experts selecting the top-k tokens. As a result, each token can be routed to a variable number of experts and each expert can have a fixed bucket size. We systematically study pre-training speedups using the same computational resources of the Switch Transformer top-1 and GShard top-2 gating of prior work and find that our method improves training convergence time by more than 2×. For the same computational cost, our method demonstrates higher performance in fine-tuning 11 selected tasks in the GLUE and SuperGLUE benchmarks. For a smaller activation cost, our method outperforms the T5 dense model in 7 out of the 11 tasks.\n\n1 Introduction\n\nScaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27]. The final model quality has been found to have a power-law relationship with the amount of data, model size, and compute time [16, 20]. However, training efficiency, which is defined as the total amount of computation used to achieve superior model quality than the state of the art system [21], should receive greater attention as we increase our efforts towards green AI [29].\n\nSparsely gated mixture-of-experts [31] (MoE) provides an effective way to scale model capacity given a fixed computational cost, and has recently played an important role in increasing the training efficiency of large-scale language models [10, 21]. MoE operate by adopting a number of experts, each as a sub-network, and by activating only one or a few experts for each input token. A gating network must be chosen and optimized in order to route each token to the most suited expert(s). For example, recent work has implemented sparse routing via k-means clustering [12], linear assignment to maximize token-expert affinities [22], or hashing [8, 28]. Many of the prior work use a routing strategy concerning the token choice, where each token selects the best one or two experts.\n\nWe argue that the independent token choice of prior work often leads to an imbalanced load of experts, which causes training inefficiency and sub-optimal training of the model. In order to mitigate this\n\nissue, previous sparsely gated networks introduce additional auxiliary losses as regularization to prevent too many tokens being routed to a single expert, but the effectiveness is still limited. Recent approaches [8, 22, 28] explore alternative strategies for routing, but they focus on pre-training only and do not demonstrate performance gain on downstream tasks. Moreover, none of the previous methods consider allocating a variable number of experts to each token based on importance, which can be beneficial.\n\nWe propose a very simple yet effective routing method we are calling expert choice. Unlike conventional MoE where tokens select one or two top-scoring experts, our method lets each expert pick the top-k tokens. Our method guarantees perfect load balancing, allows a variable number of experts for each token, and achieves substantial gains in training efficiency and downstream performance as demonstrated in our experiments. Our major contributions include:\n\nWe identify common pitfalls in conventional MoE such as load imbalance as described in Section 3.1. We then propose a heterogeneous, expert choice method to provide a fluid allocation of model parameters based on a learnt token-to-expert importance. This method intrinsically guarantees load balance without imposing an auxiliary loss.\n\nWe show our method provides over 2× faster training convergence in a 8B/64E (8 billion activated parameters, 64 experts) model, compared to the top-1 and top-2 gating counterparts in Switch Transformer [10] and GShard [21].\n\nWe show our method demonstrates strong scaling when increasing the number of experts from 16 to 128, evaluated in training perplexity.\n\nWe show our method demonstrates strong performance on downstream tasks selected from GLUE and SuperGLUE at all the evaluated scales. More specifically, our 8B/64E model outperforms a T5 11B dense model in 7 out of 11 tasks evaluated.\n\n2 Related Work\n\nScaling: Various approaches have been proposed to scale up neural network capacity to improve performance. Recent works have successfully scaled models to billions of parameters via various forms of model parallelism [2, 21, 26, 27, 33]. Model parallelism [30] splits weights and tensors across multiple cores while pipeline parallelism [18, 24] splits different layers across devices with micro-batches pipelined to the different layers. To enable continued scaling of neural networks, improving model training and serving efficiency has become a critical research area.\n\nConditional Computation: Computation decisions can be made dynamically based on the input [23, 25]. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation, by activating certain parameters and computation on demand, on a per-example or per-token basis [3]. Conditional convolution layers [1] with task-specific gating has been used to combat catastrophic forgetting when a sequence of learning problems are optimized. The gating decisions may be binary or sparse and continuous, stochastic or deterministic.\n\nMixture of Experts: Sparsely-gated MoE [31] is the first model to demonstrate massive improvements in model capacity, training time, or model quality with gating. Switch Transformer [10] simplifies the gating by selecting only the top expert per token using a softmax over the hidden state and demonstrates better scaling than previous work. All the prior work requires an auxiliary loss to explicitly encourage balancing. This loss term has to be carefully weighted to not overwhelm the primary loss. However, auxiliary loss does not guarantee balancing and a hard capacity factor has to be imposed. As a result, many tokens can still be unprocessed by the MoE layer. Hard MoE [12] with a single decoding layer can be efficiently trained to good effect on large scale hashtag prediction tasks.\n\nBase Layers [22] formulate a linear assignment that maximizes token-expert affinities while ensuring each expert receives an equal number of tokens. Hash layers [8, 28] devise hashing techniques on input tokens. However, the evaluations are limited to pre-training perplexity. THOR [? ] randomly activates experts during training and inference and is trained with a consistency regularization loss.\n\nTHOR has demonstrated strong performance on translation tasks. Different from these prior works, our method is a learnt method that enables heterogeneous MoE and effectively improves downstream fine-tuning performance.\n\n3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n5 Conclusion\n\nWe propose a new routing method for sparsely activated mixture-of-experts (MoE) models. This method addresses load imbalance and under-utilization of experts in conventional MoE methods, and enables selecting different numbers of experts for each token. Our model demonstrates more than 2x training efficiency improvements when compared to the state-of-the-art GShard and Switch Transformer models, and also achieves strong gains when finetuning on 11 datasets in the GLUE and SuperGLUE benchmark.\n\n6 Limitations\n\nThe expert choice method might not immediately apply to auto-regressive text generation as our current implementation takes in the past and future tokens to perform the top-k selection. One possible solution is to collect a large batch of input sequences, dispatch tokens of the same sequence into separate groups, and perform expert choice routing for each group. Another scenario where the expert choice method does not immediately apply is when the batch size becomes very small during serving or inference. A global top-k can be selected instead and we can cap the number of times each expert or token gets selected. We leave these possible improvements for future work.\n\nAnother long-standing issue with MoE has been the large memory footprint. Even though computational cost can be reduced using sparsely gated networks, the total number of parameters increases linearly or sub-linearly with the number of experts. Increasing the number of experts requires reservation of a large number of hardware devices. Therefore, dynamic (used) power is saved while static (reserved) power is not. Power saving techniques such as the ability to put hardware devices into low power states while not in use [17] can help with reducing the reserved power requirements.\n\nReferences\n\n[1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pages 3930–3939. Computer Vision Foundation / IEEE, 2020.\n\n[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. In Advances in Neural Information Processing Systems.\n\n[3] Kyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning, 2014.\n\n[4] Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems, 2021.\n\n[5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.\n\nTransformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.\n\n[6] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 933–941. JMLR.org, 2017.\n\n[7] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixtureof-experts, 2021.\n\n[8] Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan.\n\nTricks for training sparse translation models, 2021.\n\n[9] Richard L Dykstra. An iterative procedure for obtaining i-projections onto the intersection of convex sets. The annals of Probability, pages 975–984, 1985.\n\n[10] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.\n\n[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. NAS-FPN: learning scalable feature pyramid architecture for object detection. In CVPR, pages 7036–7045. Computer Vision Foundation / IEEE, 2019.\n\n[12] Sam Gross, Marc'Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision, 2017.\n\n[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.\n\n[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 630–645, Cham, 2016. Springer International Publishing.\n\n[15] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2016.\n\n[16] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.\n\n[17] Ping Huang, Zuocheng Xing, Tianran Wang, Qiang Wei, Hongyan Wang, and Guitao Fu. A brief survey on power gating design. In 2010 10th IEEE International Conference on Solid-State and Integrated Circuit Technology, pages 788–790, 2010.\n\n[18] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 103–112, 2019.\n\n[19] Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David A. Patterson. A domain-specific supercomputer for training deep neural networks. Commun. ACM, 63(7):67–78, 2020.\n\n[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.\n\n[21] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.\n\n[22] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6265–6274. PMLR, 18–24 Jul 2021.\n\n[24] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. New York, NY, USA, 2019. Association for Computing Machinery.\n\n[25] Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, Cédric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. In ICLR. OpenReview.net, 2021.\n\n[26] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.\n\n[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.\n\n[28] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models, 2021.\n\n[30] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 10435–10444, Red Hook, NY, USA, 2018. Curran Associates Inc.\n\n[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E.\n\nHinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In ICLR (Poster). OpenReview.net, 2017.\n\n[32] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 10–15 Jul 2018. [23] Min Lin, Jie Fu, and Yoshua Bengio. Conditional computation for continual learning, 2019.\n\n[29] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.\n\n[33] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.\n\n[34] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc.\n\n[35] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.\n\nGLUE: A multi-task benchmark and analysis platform for natural language understanding.\n\nIn Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, November 2018. Association for Computational Linguistics.\n\n[36] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake A. Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen. GSPMD: general and scalable parallelization for ML computation graphs. CoRR, abs/2105.04663, 2021.\n\n7 Checklist\n\n(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Yes (b) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes (c) Did you discuss any potential negative societal impacts of your work? N/A. Not any. (d) Did you describe the limitations of your work? Yes (a) Did you include the code, data, and instructions needed to reproduce the main experimental results? Yes. We include details in the experiment setup to help reproduce the main results. (b) Did you specify all the training details? Yes (c) Did you report error bars? Yes (d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? Yes (a) If your work uses existing assets, did you cite the creators? Yes (b) Did you mention the license of the assets? No. The used dataset is not released yet. (c) Did you include any new assets either in the supplemental material or as a URL? No. The dataset is not released yet.\n\n(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? No. Not using persons' data. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes. The dataset does not contain any personally identifiable information or offensive content.\n\nA Comparison On Fine-Tuning With A Dense Model\n\nOur 8B MoE model achieves stronger pre-training perplexity than its dense counterpart. However, a better perplexity does not always directly translate to downstream performance as demonstrated in Section 4.4. To this end, we compare fine-tuning performance of the 8B dense model and MoE model in Table 1. As shown in the table, our MoE model using expert choice routing consistently outperforms the dense model across the 11 tasks in GLUE and SuperGLUE.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 1: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nB Capacity Factor\n\nWe evaluate the downstream task fine-tuning performance by varying the capacity factors. Note that a capacity factor of n indicates on average how many experts each token can be received. EC-CF2 is our baseline expert choice, which matches GShard top-2 gating computational footprint. EC-CF1, however, matches Switch Transformer top-1 gating computational footprint. EC-CF0.5 further verifies that an aggressively lowered capacity factor can provide strong enough performance, that almost matches the top-2 gating baseline.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Top-2 78.1 87.0 88.3 85.0 82.6 90.1 90.7 81.6 94.7 68.2 67.2 83.0±0.3 EC-CAP2 78.2 88.0 88.5 85.7 83.0 90.8 91.1 80.0 95.4 70.4 64.1 83.2±0.4 EC-CAP3 78.5 91.7 89.3 86.3 83.5 90.9 91.1 81.8 94.9 70.0 65.6 84.0±0.4 EC-CF2 79.1 89.6 89.3 86.8 84.3 91.3 91.2 81.1 95.2 68.1 68.0 84.0±0.2 EC-CF1 77.4 90.6 88.0 85.5 83.6 90.3 91.2 79.8 95.3 66.5 64.9 83.0±0.2 EC-CF0.5 77.4 89.6 86.3 85.2 82.7 91.7 91.0 79.6 94.9 67.3 63.5 83.0 ±0.05 Hash Layers 76.1 85.2 86.7 83.4 82.5 90.0 90.3 75.7 94.0 67.4 63.3 81.3±1.0\n\nTable 2: Comparison between different routing methods in fine-tuning of 100M/64E models. We perform 3 independent fine-tuning runs for each method and report the average results. This gives more accurate difference between the variants of expert choice method, since they achieve close fine-tuning results. We do not report averaged results in other experiments.\n\nC Capped Expert Choice\n\nAs described in Section 4.5, the maximum number of experts each token is assigned can be capped by an entropy-regularized linear programming. Figure 1 compares the validation perplexity when training the 100M/64E models using the base expert choice method (EC-BASE), expert choice capped by two experts per token (EC-CAP2), expert choice capped by three experts per token (EC-CAP3), and GShard top-2 gating.\n\nAs shown in the figure, restricting the number of experts to 2 degrades the perplexity compared to the base expert choice method. This suggests that a more flexible allocation of experts (e.g. more than 2 experts for a token) can enhance model expressiveness. On the other hand, our EC-CAP2 and EC-CAP3 methods still outperform the top-2 gating method by a clear margin. We believe this confirms the effectiveness of a load balanced training, provided by our method. Finally, EC-CAP3 obtains comparable perplexity to EC-BASE. As indicated by Figure 3, only a little fraction of tokens use more than 3 experts therefore we see little or no difference between EC-BASE and EC-CAP3 variants. We present the fine-tuning results of these methods in Table 2.\n\nD Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [? ]. We use mod x to map a token ID to an expert ID. This in some way ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 2. Hashing based routing performs much worse than expert choice in terms of average scores and variance.\n\nE Fine-Tuning Details\n\nWe did a hyperparameter search for both baseline models and expert choice method. For fine-tuning of the 8B dense model, we use a constant learning rate of 0.0001 and a dropout rate of 0.1. We freeze the attention layer and feed-forward layer while leaving the embedding and layer normalization trainable. This setting has been found optimal for the 8B dense model. For MoE 8B/64E models including GShard top-2 gating and expert choice, we found continuing the learning rate from the pre-trained model while using a square root learning rate decay works better. In addition, we do not apply parameter freezing for fine-tuning MoE models. For models with 100M expert size, we use a constant learning rate of 0.0001 and no dropout is used.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/2202.09368v2.md"
+          },
+          "headlines": {
+            "1 Introduction": [],
+            "2 Related Work": [],
+            "3 Method": [
+              "3.1 Pitfalls Of Token-Choice Routing",
+              "3.2 Heterogeneous Moe Via Expert Choice",
+              "3.3 Expert Choice With Additional Constraint",
+              "3.4 Model Architecture"
+            ],
+            "4 Experiments": [
+              "4.1 Setup",
+              "4.2 Training Efficiency",
+              "4.3 Scaling The Number Of Experts",
+              "4.4 Fine-Tuning On Glue And Superglue",
+              "4.5 Heterogeneity Matters",
+              "4.6 Comparison With Hash Layer",
+              "4.7 Ablation"
+            ],
+            "5 Conclusion": [],
+            "6 Limitations": [],
+            "A Comparison On Fine-Tuning With A Dense Model": [],
+            "B Capacity Factor": [],
+            "C Capped Expert Choice": [],
+            "D Comparison With Hash Layer": [],
+            "E Fine-Tuning Details": []
+          },
+          "summary": "The paper introduces a new routing method for sparsely-activated Mixture-of-Experts (MoE) models called 'expert choice,' which addresses load imbalance and under-utilization issues in conventional MoE methods. Unlike traditional token-choice routing, expert choice allows experts to select the top-k tokens, ensuring perfect load balancing and enabling variable expert allocation per token. This method improves training efficiency, achieving over 2x faster convergence compared to existing models like Switch Transformer and GShard. It also demonstrates superior performance on downstream tasks in the GLUE and SuperGLUE benchmarks. The study highlights the benefits of flexible expert allocation and shows that expert choice outperforms hashing-based routing methods. The paper also discusses potential limitations and future improvements for the expert choice method.",
+          "summary_embedding": [
+            -0.009726986289024353,
+            -0.0337262786924839,
+            -0.023302342742681503,
+            -0.019878201186656952,
+            -0.025953780859708786,
+            0.04560471698641777,
+            -0.006874797400087118,
+            0.039665497839450836,
+            -0.006973279174417257,
+            0.034544438123703,
+            -0.025787118822336197,
+            -0.00363057479262352,
+            -0.01052999310195446,
+            0.0054657477885484695,
+            -0.024499278515577316,
+            0.036786794662475586,
+            0.010007280856370926,
+            0.003272630739957094,
+            0.006102093029767275,
+            -0.04048365354537964,
+            -0.018666116520762444,
+            -0.013522328808903694,
+            -0.03230207785964012,
+            0.008969432674348354,
+            0.02484775334596634,
+            0.0016713523073121905,
+            0.0026230288203805685,
+            0.003624893259257078,
+            -0.05311964824795723,
+            0.04339266195893288,
+            0.018681267276406288,
+            0.023711422458291054,
+            -0.016332851722836494,
+            -0.012832955457270145,
+            -0.039120059460401535,
+            -0.0197418425232172,
+            0.030786970630288124,
+            0.039968520402908325,
+            -0.019287310540676117,
+            0.03357476741075516,
+            0.018060073256492615,
+            0.02454473078250885,
+            -0.021272100508213043,
+            -0.029817303642630577,
+            -0.0061892117373645306,
+            -0.007840678095817566,
+            -0.008355814032256603,
+            0.01242387667298317,
+            0.016984347254037857,
+            -0.013007191941142082,
+            0.0212569497525692,
+            -0.01371171697974205,
+            0.03384748846292496,
+            0.008916404098272324,
+            -0.042332086712121964,
+            0.03984731063246727,
+            -0.008007340133190155,
+            0.012105703353881836,
+            0.03281721472740173,
+            0.010333028621971607,
+            -0.005287723150104284,
+            -0.021544819697737694,
+            -0.03378688171505928,
+            0.0010529992869123816,
+            0.00014014738553669304,
+            0.005950582213699818,
+            -0.003069985192269087,
+            -0.009742137044668198,
+            0.03227177634835243,
+            -0.003939277958124876,
+            -0.0740584209561348,
+            0.050089433789253235,
+            -0.026590125635266304,
+            -0.007628562860190868,
+            -0.028938541188836098,
+            -0.01736312545835972,
+            0.037362534552812576,
+            0.005662711802870035,
+            0.0098709212616086,
+            0.0106284748762846,
+            -0.0007892759749665856,
+            0.028923390433192253,
+            0.00910579226911068,
+            -0.024923507124185562,
+            0.04478655755519867,
+            -0.04790768027305603,
+            0.01801462098956108,
+            -0.04336235672235489,
+            0.022756904363632202,
+            1.6393929399782792e-05,
+            -0.020726662129163742,
+            -0.0013304532039910555,
+            -0.016666175797581673,
+            0.0218023881316185,
+            -0.01730252057313919,
+            0.026802239939570427,
+            0.013825350441038609,
+            -0.015802564099431038,
+            -0.029817303642630577,
+            0.025378039106726646,
+            0.003367325058206916,
+            0.010499690659344196,
+            -0.02075696364045143,
+            0.014332910999655724,
+            0.04393810033798218,
+            -0.02593863010406494,
+            -0.02271145209670067,
+            0.027681002393364906,
+            0.014408666640520096,
+            0.0633314698934555,
+            0.036544378846883774,
+            0.006624804809689522,
+            0.022362977266311646,
+            0.020575150847434998,
+            -0.028968842700123787,
+            -0.01963578537106514,
+            -0.046362269669771194,
+            -0.04757435619831085,
+            0.019878201186656952,
+            -0.04566532000899315,
+            0.0050869714468717575,
+            0.038544319570064545,
+            -0.013704141601920128,
+            -0.014446543529629707,
+            -0.04021093621850014,
+            0.017847958952188492,
+            0.0003908502112608403,
+            0.011976920068264008,
+            -0.0022082682698965073,
+            0.00033285003155469894,
+            -0.011825408786535263,
+            0.008878526277840137,
+            -0.03778676688671112,
+            -0.025196226313710213,
+            0.025181075558066368,
+            0.022862963378429413,
+            0.024120500311255455,
+            0.028802180662751198,
+            -0.03077181987464428,
+            -0.009984553791582584,
+            0.021378157660365105,
+            -0.04242299124598503,
+            -0.006556625012308359,
+            0.011332998983561993,
+            0.0026703758630901575,
+            0.003872992005199194,
+            0.032786913216114044,
+            0.052998438477516174,
+            -0.03172633796930313,
+            0.02065090648829937,
+            -0.010431510396301746,
+            0.04957429692149162,
+            0.00024691506405360997,
+            0.028968842700123787,
+            -0.019514575600624084,
+            -0.0134389977902174,
+            -0.029529431834816933,
+            -0.009795165620744228,
+            -0.01356020662933588,
+            -0.008484598249197006,
+            -0.002998017705976963,
+            -0.009575475007295609,
+            0.0040453351102769375,
+            0.0331808403134346,
+            0.00273855566047132,
+            0.030680913478136063,
+            0.014325335621833801,
+            0.0051475754007697105,
+            -0.08745196461677551,
+            0.026059838011860847,
+            -0.0033616432920098305,
+            0.005859675817191601,
+            0.0062460280023515224,
+            0.03013547509908676,
+            -0.0312718041241169,
+            0.020938776433467865,
+            0.026620427146553993,
+            0.0011145505122840405,
+            0.016969196498394012,
+            0.036847397685050964,
+            -0.04181694984436035,
+            0.009643655270338058,
+            0.012590537779033184,
+            -0.0033881578128784895,
+            0.04584713280200958,
+            0.011628445237874985,
+            0.019514575600624084,
+            0.018529755994677544,
+            0.003569970605894923,
+            0.00297907879576087,
+            0.006533898413181305,
+            0.03724132478237152,
+            0.003462019143626094,
+            -0.009833043441176414,
+            0.014976831153035164,
+            -0.010757259093225002,
+            -0.011219366453588009,
+            -0.037998881191015244,
+            0.014454119838774204,
+            0.008742166683077812,
+            -0.017454031854867935,
+            -0.02959003672003746,
+            0.0026949963066726923,
+            -0.031908150762319565,
+            0.026559824123978615,
+            -0.002445003716275096,
+            0.005738467443734407,
+            -0.010454237461090088,
+            -0.006757376715540886,
+            -0.035423196852207184,
+            0.028453707695007324,
+            -0.02095392718911171,
+            0.0053142374381423,
+            0.0007258309051394463,
+            -0.029438525438308716,
+            0.0060073984786868095,
+            0.017863109707832336,
+            -0.0025965142995119095,
+            0.0025302283465862274,
+            -0.02349930815398693,
+            0.024468975141644478,
+            0.025635607540607452,
+            0.004518806003034115,
+            -0.014037464745342731,
+            -0.013113250024616718,
+            -0.003605954349040985,
+            -0.009742137044668198,
+            -0.016196493059396744,
+            0.024468975141644478,
+            -0.014810169115662575,
+            0.00308892410248518,
+            0.021847840398550034,
+            0.014264730736613274,
+            -0.007045246660709381,
+            -0.004007457755506039,
+            -0.04066546633839607,
+            -0.01218903437256813,
+            -0.04760465770959854,
+            -0.011628445237874985,
+            -0.0034601252991706133,
+            -0.028696123510599136,
+            0.0061513339169323444,
+            -0.04481685906648636,
+            -0.01235569640994072,
+            0.02309022843837738,
+            0.018135828897356987,
+            0.0019488062243908644,
+            -0.009173971600830555,
+            -0.022605394944548607,
+            8.486965816700831e-05,
+            -0.01763584464788437,
+            -0.017044952139258385,
+            0.015393485315144062,
+            0.031059689819812775,
+            -0.03101423755288124,
+            0.04024123772978783,
+            0.003893824527040124,
+            -0.08066429197788239,
+            0.010393633507192135,
+            -0.026362858712673187,
+            0.07011914253234863,
+            -0.0025453795678913593,
+            0.014075342565774918,
+            0.06781618297100067,
+            -0.012431452050805092,
+            -0.02007516659796238,
+            0.03142331540584564,
+            -0.008249756880104542,
+            -1.1311515663692262e-05,
+            -0.02884763479232788,
+            -0.02739313244819641,
+            -0.053271155804395676,
+            -0.009704259224236012,
+            -0.010787560604512691,
+            0.03360506892204285,
+            -0.024317465722560883,
+            -0.0007656024536117911,
+            0.012620840221643448,
+            0.02983245439827442,
+            0.016635872423648834,
+            -0.0015681355725973845,
+            0.008272483013570309,
+            0.02881733328104019,
+            0.0274234339594841,
+            -0.0036154238041490316,
+            0.00814369972795248,
+            -0.034211114048957825,
+            -0.0068066176027059555,
+            -0.0493621826171875,
+            0.011302697472274303,
+            -0.02596893161535263,
+            -0.02596893161535263,
+            0.007643714081496,
+            -0.011476934887468815,
+            0.03748374432325363,
+            -0.024696242064237595,
+            -0.007503566797822714,
+            0.005037730094045401,
+            0.012855681590735912,
+            -0.017757052555680275,
+            -0.03612014651298523,
+            -0.0028502948116511106,
+            0.0026381798088550568,
+            0.007560383062809706,
+            -0.053240854293107986,
+            0.004499867558479309,
+            -0.018469152972102165,
+            0.0279688723385334,
+            -0.02139330841600895,
+            -0.013688990846276283,
+            0.042029064148664474,
+            0.011363301426172256,
+            0.027499189600348473,
+            -0.026468917727470398,
+            0.0017849853029474616,
+            0.012825379148125648,
+            -0.0020056227222085,
+            0.032514192163944244,
+            -0.030574856325984,
+            -0.03257479891180992,
+            0.001548249856568873,
+            -0.007840678095817566,
+            -0.0625436082482338,
+            -0.011386028490960598,
+            -0.00981789268553257,
+            0.017317671328783035,
+            0.0446956530213356,
+            0.04557441547513008,
+            -0.01068150345236063,
+            0.002174178371205926,
+            -0.02213571034371853,
+            -0.04542290419340134,
+            -0.02643861435353756,
+            0.0016486257081851363,
+            -0.019999410957098007,
+            -0.02789311669766903,
+            -0.0018313854234293103,
+            0.011211791075766087,
+            -0.013651113025844097,
+            0.01134057529270649,
+            -0.018560059368610382,
+            -0.01862066239118576,
+            -0.012332970276474953,
+            0.018575210124254227,
+            0.009901223704218864,
+            -0.0018796793883666396,
+            -0.015037435106933117,
+            -0.00980274099856615,
+            0.018969137221574783,
+            -0.011279970407485962,
+            0.0009388927719555795,
+            0.01188601367175579,
+            0.024347767233848572,
+            -0.007083124481141567,
+            0.010961798019707203,
+            -0.034756552428007126,
+            -0.01966608688235283,
+            0.007984613068401814,
+            0.002719616750255227,
+            0.020711511373519897,
+            0.008742166683077812,
+            -0.04275631532073021,
+            -0.044301725924015045,
+            0.016044981777668,
+            -0.0562104657292366,
+            0.028120383620262146,
+            -0.03327174484729767,
+            -0.012514783069491386,
+            -0.02671133354306221,
+            0.0013418165035545826,
+            -0.01828734017908573,
+            0.022438732907176018,
+            0.03778676688671112,
+            -0.0011486404109746218,
+            0.005503625608980656,
+            0.014658658765256405,
+            0.04384719207882881,
+            -0.05021063983440399,
+            -0.041180603206157684,
+            0.03754434734582901,
+            -0.01601468026638031,
+            0.0185146052390337,
+            0.00021744151308666915,
+            0.010348179377615452,
+            -0.014431392773985863,
+            0.022772056981921196,
+            0.013757170177996159,
+            -0.008764892816543579,
+            -0.030574856325984,
+            0.04945308715105057,
+            -0.023650817573070526,
+            -0.049513693898916245,
+            0.0018323323456570506,
+            0.018878230825066566,
+            0.018635815009474754,
+            -0.014870774000883102,
+            -0.02907489985227585,
+            -0.024317465722560883,
+            -0.006060427520424128,
+            -0.00947699323296547,
+            -0.03402930125594139,
+            0.011363301426172256,
+            0.0048445542342960835,
+            0.016666175797581673,
+            -0.00645056739449501,
+            -0.0073141781613230705,
+            -0.009370936080813408,
+            0.022559940814971924,
+            0.012635990977287292,
+            -0.03175663948059082,
+            -0.017484333366155624,
+            -0.02528713271021843,
+            0.004359719809144735,
+            0.010136065073311329,
+            0.022166013717651367,
+            0.024287162348628044,
+            -0.0323626808822155,
+            0.04218057543039322,
+            0.01550711877644062,
+            -0.008727015927433968,
+            -0.0019317612750455737,
+            0.02478714846074581,
+            0.04324115067720413,
+            0.0218023881316185,
+            0.026514369994401932,
+            0.002107892418280244,
+            8.705945947440341e-05,
+            -0.015620751306414604,
+            -0.0011391709558665752,
+            0.01862066239118576,
+            0.04911976307630539,
+            -0.012151156552135944,
+            0.01716616004705429,
+            -0.03578682243824005,
+            0.00396200455725193,
+            -0.02580226957798004,
+            -0.030786970630288124,
+            0.03745344281196594,
+            -0.015060162171721458,
+            0.0191964041441679,
+            0.025044716894626617,
+            -0.022469034418463707,
+            -0.011075431481003761,
+            -0.019423669204115868,
+            -0.026181045919656754,
+            -0.01702980138361454,
+            -0.03281721472740173,
+            -0.018135828897356987,
+            -0.002653330797329545,
+            -0.035635314881801605,
+            -0.04987731948494911,
+            -0.004329417832195759,
+            -0.046392571181058884,
+            -0.010507266037166119,
+            0.017378276214003563,
+            -0.011166337877511978,
+            -0.025711363181471825,
+            -0.005336964037269354,
+            0.003973367623984814,
+            -0.022196315228939056,
+            -0.0066588944755494595,
+            0.008439145050942898,
+            0.035938333719968796,
+            -0.02298417128622532,
+            -0.013060221448540688,
+            -0.0024317465722560883,
+            -0.02887793630361557,
+            -0.007878555916249752,
+            0.008999735116958618,
+            0.007723257411271334,
+            -0.0012357590021565557,
+            -0.05196816474199295,
+            -0.006916462909430265,
+            -0.012992041185498238,
+            0.029196109622716904,
+            0.026468917727470398,
+            0.0012168202083557844,
+            -0.02783251367509365,
+            0.008749742060899734,
+            -0.0072535742074251175,
+            0.02339324913918972,
+            -0.007992188446223736,
+            -0.022908415645360947,
+            -0.02034788578748703,
+            0.030786970630288124,
+            -0.0159086212515831,
+            0.009113367646932602,
+            0.009976978413760662,
+            0.0121360057964921,
+            -0.03899884968996048,
+            -0.001445980160497129,
+            0.0005648507503792644,
+            -0.010128489695489407,
+            0.00423093605786562,
+            -0.006738437805324793,
+            -0.007079336792230606,
+            -0.022090258076786995,
+            -0.0218023881316185,
+            0.03818069398403168,
+            -0.009113367646932602,
+            -0.0035661826841533184,
+            -0.01088604237884283,
+            0.03436262533068657,
+            -0.00595815759152174,
+            -0.01993880607187748,
+            -0.019347915425896645,
+            0.014431392773985863,
+            0.05105910077691078,
+            -0.028529461473226547,
+            -0.03984731063246727,
+            0.0011136035900563002,
+            0.030150625854730606,
+            0.004140029661357403,
+            -0.016908591613173485,
+            0.0035851215943694115,
+            0.056392278522253036,
+            -0.011742077767848969,
+            0.028362801298499107,
+            0.027771908789873123,
+            0.02295386977493763,
+            -0.02021152526140213,
+            -0.01678738370537758,
+            -0.020999381318688393,
+            0.011098157614469528,
+            -0.03027183562517166,
+            0.027059808373451233,
+            0.018969137221574783,
+            0.01538590993732214,
+            0.05054396390914917,
+            -0.03178694099187851,
+            -0.02515077404677868,
+            0.008499749936163425,
+            0.0021722845267504454,
+            0.011007251217961311,
+            -0.007727045100182295,
+            -0.01801462098956108,
+            -0.001950700068846345,
+            -0.014022313989698887,
+            -0.019817598164081573,
+            -0.020938776433467865,
+            0.00954517349600792,
+            0.015696506947278976,
+            0.002649543108418584,
+            0.010863316245377064,
+            0.0016590419691056013,
+            0.01686313934624195,
+            0.01321930717676878,
+            -0.0444229319691658,
+            -7.936555630294606e-05,
+            0.03378688171505928,
+            -0.0293476190418005,
+            -0.012264790013432503,
+            -0.016060132533311844,
+            -0.004439263138920069,
+            -0.01706010289490223,
+            -0.02762039750814438,
+            -0.006874797400087118,
+            -0.01889338344335556,
+            -0.00814369972795248,
+            -0.028196139261126518,
+            -0.009992130100727081,
+            0.01949942484498024,
+            0.022332675755023956,
+            0.001196934375911951,
+            -0.006783891003578901,
+            -0.005901341326534748,
+            0.02092362567782402,
+            0.032514192163944244,
+            0.01952972821891308,
+            0.0058331615291535854,
+            -0.015158643946051598,
+            -0.004295328166335821,
+            -0.012370847165584564,
+            0.016060132533311844,
+            0.012537509202957153,
+            6.344509893096983e-05,
+            0.03493836522102356,
+            0.004170331638306379,
+            -0.022726602852344513,
+            -0.001230077352374792,
+            0.024923507124185562,
+            -0.021590271964669228,
+            -0.0108330138027668,
+            0.040029123425483704,
+            0.0085679292678833,
+            -0.0024506852496415377,
+            -0.019287310540676117,
+            -0.019075196236371994,
+            0.02630225569009781,
+            -0.02133270539343357,
+            0.002083271974697709,
+            0.0482713058590889,
+            0.00035344602656550705,
+            -0.029802151024341583,
+            0.017954016104340553,
+            -0.013242034241557121,
+            -0.018560059368610382,
+            0.01848430372774601,
+            0.0012859469279646873,
+            -0.039968520402908325,
+            0.019014591351151466,
+            0.018408548086881638,
+            0.022635696455836296,
+            -0.02918095886707306,
+            0.0002127068000845611,
+            0.03233237937092781,
+            0.004037759732455015,
+            0.001595596899278462,
+            -0.008530051447451115,
+            -0.0006065162015147507,
+            0.010431510396301746,
+            0.001746160676702857,
+            0.02854461409151554,
+            -0.0030434709042310715,
+            0.027771908789873123,
+            -0.006727074272930622,
+            -0.008863375522196293,
+            0.019181253388524055,
+            -0.02109028771519661,
+            0.013166278600692749,
+            0.0685434341430664,
+            0.0021950111258774996,
+            0.025847723707556725,
+            0.007181606721132994,
+            0.024347767233848572,
+            0.02018122375011444,
+            -0.027271922677755356,
+            -0.02021152526140213,
+            -0.005397567991167307,
+            -0.0014033677289262414,
+            -0.03787767142057419,
+            0.011128460057079792,
+            -0.021302402019500732,
+            -0.013249609619379044,
+            -0.0022347825579345226,
+            -0.002316219499334693,
+            -0.006405114196240902,
+            0.034211114048957825,
+            -0.0014630251098424196,
+            -0.018560059368610382,
+            -0.035423196852207184,
+            0.037726160138845444,
+            0.04048365354537964,
+            0.0389382466673851,
+            -0.005984671879559755,
+            0.03548380360007286,
+            0.0030756669584661722,
+            0.0025302283465862274,
+            0.041453324258327484,
+            0.004791525658220053,
+            -0.011567841283977032,
+            0.02295386977493763,
+            0.012416300363838673,
+            0.03351416438817978,
+            -0.02045394293963909,
+            -0.02630225569009781,
+            -0.003299145260825753,
+            -0.026423463597893715,
+            0.012492056004703045,
+            -0.019726691767573357,
+            0.0018768386216834188,
+            0.007117214612662792,
+            0.015196521766483784,
+            -0.016044981777668,
+            0.014719262719154358,
+            -0.006102093029767275,
+            -0.021014532074332237,
+            0.011719351634383202,
+            -0.007780073676258326,
+            0.011961768381297588,
+            -0.02262054570019245,
+            0.001138224033638835,
+            -0.04063516482710838,
+            0.008280059322714806,
+            0.014022313989698887,
+            -0.01598437689244747,
+            0.03205966204404831,
+            0.020969079807400703,
+            -0.01801462098956108,
+            -0.0009682480012997985,
+            0.0034714885987341404,
+            -0.010393633507192135,
+            -0.011067855171859264,
+            -0.019923655316233635,
+            0.02786281518638134,
+            -0.015802564099431038,
+            -0.025908326730132103,
+            0.02454473078250885,
+            -0.01743887923657894,
+            0.015151068568229675,
+            -0.004299115855246782,
+            0.00888610165566206,
+            -0.019120648503303528,
+            0.02284781076014042,
+            -0.023741723969578743,
+            -0.0009128518868237734,
+            -0.020287280902266502,
+            0.0028389315120875835,
+            0.014135947450995445,
+            -0.018272189423441887,
+            -0.02139330841600895,
+            0.019150950014591217,
+            -0.005420294590294361,
+            -0.003931702114641666,
+            0.010014856234192848,
+            0.0030983935575932264,
+            -0.0029715034179389477,
+            -0.004549108445644379,
+            -0.03281721472740173,
+            -0.02915065549314022,
+            0.0033086147159337997,
+            0.011386028490960598,
+            0.015257125720381737,
+            0.027574945241212845,
+            0.03715042024850845,
+            0.006052851676940918,
+            0.0036949669010937214,
+            0.0009327376610599458,
+            -0.020726662129163742,
+            0.042635105550289154,
+            -0.020469093695282936,
+            0.040847279131412506,
+            0.032544493675231934,
+            -0.0197418425232172,
+            -0.023908386006951332,
+            0.0029052174650132656,
+            0.03115059621632099,
+            0.03151422366499901,
+            0.025211378931999207,
+            0.012310243211686611,
+            0.03690800443291664,
+            -0.004359719809144735,
+            -0.032211173325777054,
+            0.015802564099431038,
+            0.022256920114159584,
+            -0.008105821907520294,
+            0.009673957712948322,
+            0.027438584715127945,
+            0.013279912061989307,
+            -0.007757347077131271,
+            0.030150625854730606,
+            0.0012234487803652883,
+            0.02396899089217186,
+            0.02498411200940609,
+            -0.02190844528377056,
+            0.01743887923657894,
+            -0.0001154677156591788,
+            0.006727074272930622,
+            -0.009840618818998337,
+            0.004348356742411852,
+            0.011810258030891418,
+            0.018272189423441887,
+            0.018908534198999405,
+            0.003246116451919079,
+            -0.07060398161411285,
+            0.019726691767573357,
+            0.004507442936301231,
+            -0.02160542458295822,
+            0.00390140013769269,
+            -0.0024866692256182432,
+            0.012704171240329742,
+            0.02105998620390892,
+            0.012461753562092781,
+            -0.01790856383740902,
+            0.0013626492582261562,
+            0.011961768381297588,
+            -0.00595815759152174,
+            0.04730163514614105,
+            0.0019791084341704845,
+            -0.01437078882008791,
+            -0.016105586662888527,
+            0.013726867735385895,
+            0.02525683119893074,
+            -0.02355991117656231,
+            0.006325571332126856,
+            0.005621046293526888,
+            -0.004147605039179325,
+            -0.010742107406258583,
+            -0.030377892777323723,
+            -0.005280147306621075,
+            -0.03436262533068657,
+            0.023150833323597908,
+            0.015491968020796776,
+            0.014764715917408466,
+            -0.025044716894626617,
+            -0.004280176945030689,
+            0.0046400148421525955,
+            -0.021287251263856888,
+            0.003261267440393567,
+            0.01161329448223114,
+            0.0005104016163386405,
+            -0.020984230563044548,
+            0.00959820207208395,
+            0.013469300232827663,
+            0.06848283112049103,
+            0.0007442962378263474,
+            0.029771849513053894,
+            0.0422411784529686,
+            -0.0012243957025930285,
+            -0.006371024530380964,
+            0.0036286809481680393,
+            -0.019953956827521324,
+            0.003473382443189621,
+            0.025878025218844414,
+            -0.03224147483706474,
+            -0.055058982223272324,
+            -0.007545232307165861,
+            -0.04624105989933014,
+            0.001489539397880435,
+            0.014711687341332436,
+            0.023044776171445847,
+            -0.0037782976869493723,
+            -0.034817155450582504,
+            0.016908591613173485,
+            -0.003071879269555211,
+            0.007014944683760405,
+            -0.02593863010406494,
+            -0.0294839795678854,
+            -0.026014383882284164,
+            -0.02339324913918972,
+            -0.012908710166811943,
+            -0.022726602852344513,
+            0.012590537779033184,
+            -0.0015103721525520086,
+            -0.06587684899568558,
+            -0.009780014865100384,
+            0.016166189685463905,
+            0.001200722181238234,
+            0.011939042247831821,
+            0.011083006858825684,
+            -0.007083124481141567,
+            -0.0018275976181030273,
+            -0.022559940814971924,
+            0.03124150261282921,
+            0.005177877843379974,
+            0.03584742918610573,
+            -0.006503595970571041,
+            -0.009128518402576447,
+            0.01073453202843666,
+            0.04527139291167259,
+            0.0017916138749569654,
+            0.010226971469819546,
+            -0.020560000091791153,
+            0.0015065843472257257,
+            -0.02284781076014042,
+            -0.0031135445460677147,
+            -0.008863375522196293,
+            -0.0021211495622992516,
+            0.015893470495939255,
+            0.0019753205124288797,
+            -0.010204244405031204,
+            0.025438643991947174,
+            0.02251448854804039,
+            -0.009605777449905872,
+            -0.015969226136803627,
+            -0.02437806874513626,
+            -0.014545026235282421,
+            -0.0002627763315103948,
+            -0.010007280856370926,
+            -0.022453883662819862,
+            -0.020878173410892487,
+            -0.002566212322562933,
+            -0.02559015527367592,
+            0.022529639303684235,
+            0.01220418605953455,
+            -4.882668508798815e-05,
+            -0.006723286584019661,
+            -0.038544319570064545,
+            -0.003727162955328822,
+            -0.03118089959025383,
+            -0.005374841392040253,
+            0.03287781774997711,
+            -0.03745344281196594,
+            -0.0002231231628684327,
+            0.014491996727883816,
+            -0.01993880607187748,
+            0.0028938541654497385,
+            0.008961857296526432,
+            0.02769615314900875,
+            -0.027408283203840256,
+            0.0001795638381736353,
+            0.006954340264201164,
+            0.016908591613173485,
+            -0.009878496639430523,
+            0.027196168899536133,
+            -0.04275631532073021,
+            0.011037553660571575,
+            -0.037120118737220764,
+            -0.020423639565706253,
+            -0.01523439958691597,
+            0.008363389410078526,
+            -0.005018791649490595,
+            -0.02637801133096218,
+            -0.010454237461090088,
+            0.006026337388902903,
+            -0.00693918950855732,
+            -0.01902974210679531,
+            -0.010620899498462677,
+            0.027438584715127945,
+            0.00931033119559288,
+            0.007977037690579891,
+            -0.02955973520874977,
+            -0.01631770096719265,
+            0.006280118133872747,
+            0.052089374512434006,
+            -0.01584801822900772,
+            0.014938953332602978,
+            -0.006639956030994654,
+            0.02342355251312256,
+            -0.028484009206295013,
+            -0.01872672140598297,
+            0.0315445251762867,
+            0.013946558348834515,
+            0.007064185570925474,
+            0.028953691944479942,
+            0.027196168899536133,
+            0.027105262503027916,
+            -0.04654408246278763,
+            0.009886072017252445,
+            -0.002613559365272522,
+            -0.008757317438721657,
+            0.03402930125594139,
+            0.01916610263288021,
+            0.022150862962007523,
+            0.0009744031121954322,
+            -0.01730252057313919,
+            0.0139844361692667,
+            0.014885924756526947,
+            0.007681591901928186,
+            -0.014242004603147507,
+            -0.02660527639091015,
+            0.003367325058206916,
+            -0.007992188446223736,
+            0.00047276070108637214,
+            0.020726662129163742,
+            -0.009052763693034649,
+            0.030999086797237396,
+            -0.014605630189180374,
+            -0.02593863010406494,
+            0.008916404098272324,
+            -0.02268115058541298,
+            -0.0041741193272173405,
+            0.0004476667381823063,
+            0.010083036497235298,
+            -0.023847781121730804,
+            -0.04669559374451637,
+            0.019408518448472023,
+            0.028393102809786797,
+            0.005579380784183741,
+            0.01328748743981123,
+            -0.035332292318344116,
+            0.008992158807814121,
+            0.012310243211686611,
+            0.011348150670528412,
+            0.0019847899675369263,
+            0.035362593829631805,
+            -0.00028834378463216126,
+            0.012885984033346176,
+            -0.008620957843959332,
+            0.028362801298499107,
+            0.00667025800794363,
+            -0.0323626808822155,
+            -0.0378473699092865,
+            -0.01513591781258583,
+            0.015257125720381737,
+            0.006692984607070684,
+            -0.011840560473501682,
+            -0.02278720773756504,
+            -0.038817036896944046,
+            0.015787413343787193,
+            -0.012916285544633865,
+            -0.038271598517894745,
+            -0.010158791206777096,
+            -0.009742137044668198,
+            0.016635872423648834,
+            -0.006916462909430265,
+            -0.016802534461021423,
+            -0.016090434044599533,
+            0.005821797996759415,
+            7.161247049225494e-05,
+            0.0244083721190691,
+            -0.001251857029274106,
+            -0.006855858489871025,
+            -0.016393456608057022,
+            -0.01188601367175579,
+            -0.0035074723418802023,
+            0.03602924197912216,
+            -0.014219277538359165,
+            -0.02383263036608696,
+            0.014984406530857086,
+            0.02657497487962246,
+            0.011636020615696907,
+            0.03227177634835243,
+            0.007181606721132994,
+            -0.014022313989698887,
+            0.003016956616193056,
+            -0.033029329031705856,
+            -0.0046400148421525955,
+            0.005196816753596067,
+            -0.006427840795367956,
+            -0.015651054680347443,
+            -0.01815097965300083,
+            -0.015832867473363876,
+            0.030756669119000435,
+            -0.02487805485725403,
+            0.0218023881316185,
+            0.030347591266036034,
+            0.003174148965626955,
+            -0.02915065549314022,
+            0.013598084449768066,
+            -0.0072005451656877995,
+            0.01862066239118576,
+            0.0019014590652659535,
+            0.028014326468110085,
+            -0.007461901288479567,
+            -0.004818039946258068,
+            -0.02501441352069378,
+            -0.03696860745549202,
+            -0.023484155535697937,
+            -0.008817922323942184,
+            0.0027537066489458084,
+            0.020984230563044548,
+            -0.0375746488571167,
+            -0.017757052555680275,
+            0.009855770505964756,
+            0.015757111832499504,
+            -0.03027183562517166,
+            -0.0025908327661454678,
+            -0.010158791206777096,
+            -0.0004055278259329498,
+            0.005140000022947788,
+            -0.02227207086980343,
+            -0.008977008052170277,
+            -0.015726808458566666,
+            -0.0010435298318043351,
+            0.006166485138237476,
+            -0.0007409819518215954,
+            -0.05805889517068863,
+            -0.013969285413622856,
+            0.018635815009474754,
+            0.0026400736533105373,
+            0.008908828720450401,
+            0.015938924625515938,
+            -0.0111739132553339,
+            0.0031533162109553814,
+            -0.026756787672638893,
+            0.023893235251307487,
+            -0.009492144919931889,
+            0.009355784393846989,
+            -0.013742019422352314,
+            0.012514783069491386,
+            0.014014738611876965,
+            0.004795313347131014,
+            0.0010681503918021917,
+            -0.0054051438346505165,
+            0.01601468026638031,
+            0.028377952054142952,
+            -0.023756874725222588,
+            0.011454207822680473,
+            -0.0002514130319468677,
+            -0.0032158142421394587,
+            -0.029196109622716904,
+            0.00509454682469368,
+            0.003931702114641666,
+            -0.005109698045998812,
+            0.016651025041937828,
+            -0.036544378846883774,
+            0.01662072166800499,
+            0.02177208475768566,
+            0.032544493675231934,
+            0.014832896180450916,
+            -0.017287369817495346,
+            -0.011658747680485249,
+            -0.012385998852550983,
+            -0.002624922664836049,
+            0.022181164473295212,
+            0.012120855040848255,
+            -0.01815097965300083,
+            -0.024287162348628044,
+            0.011249667964875698,
+            0.0005283934879116714,
+            -0.01203752402216196,
+            0.010795135982334614,
+            -0.013166278600692749,
+            0.01349202636629343,
+            0.012711746618151665,
+            0.00866641104221344,
+            0.00888610165566206,
+            0.022862963378429413,
+            -0.011878437362611294,
+            -0.017711598426103592,
+            0.00656041270121932,
+            -0.008916404098272324,
+            -0.0034089903347194195,
+            -0.012598113156855106,
+            0.010795135982334614,
+            0.014878349378705025,
+            0.06490717828273773,
+            -0.013143551535904408,
+            0.0009526234352961183,
+            -0.020438792183995247,
+            -0.031938452273607254,
+            -0.0017944546416401863,
+            -0.020908474922180176,
+            0.0111739132553339,
+            0.008370965719223022,
+            0.01598437689244747,
+            0.012022373266518116,
+            -0.0021912232041358948,
+            0.00795431062579155,
+            0.019317612051963806,
+            -0.010423935018479824,
+            7.255941454786807e-05,
+            -0.006503595970571041,
+            0.0013683309080079198,
+            0.0035851215943694115,
+            -0.011719351634383202,
+            -0.0026949963066726923,
+            0.035695917904376984,
+            0.01068150345236063,
+            0.011651171371340752,
+            -0.03445352986454964,
+            0.0020435003098100424,
+            0.020150920376181602,
+            0.007602048572152853,
+            -0.008302785456180573,
+            -0.01563590206205845,
+            0.01706010289490223,
+            -0.015317730605602264,
+            -0.0022499337792396545,
+            0.002888172399252653,
+            -0.010151215828955173,
+            0.04581683129072189,
+            -0.019787294790148735,
+            -0.0011742077767848969,
+            0.031847547739744186,
+            0.0030207443051040173,
+            -0.0040983641520142555,
+            0.005408931523561478,
+            -0.003473382443189621,
+            -0.0069013116881251335,
+            -0.02139330841600895,
+            0.0031381649896502495,
+            0.00667025800794363,
+            -0.019287310540676117,
+            -0.0072005451656877995,
+            -0.007719469256699085,
+            0.006897523999214172,
+            0.023105379194021225,
+            -0.009492144919931889,
+            -0.00016145357221830636,
+            -0.008575504645705223,
+            0.009848194196820259,
+            -0.0011912527261301875,
+            -0.044301725924015045,
+            -0.006147546228021383,
+            0.021181194111704826,
+            0.000973456131760031,
+            0.0005757405888289213,
+            -0.007450537756085396,
+            0.005007428117096424,
+            -0.017423728480935097,
+            -0.008454295806586742,
+            0.02369627170264721,
+            0.01736312545835972,
+            -0.009113367646932602,
+            0.010863316245377064,
+            -0.0035680767614394426,
+            0.00433320552110672,
+            -0.020014561712741852,
+            -0.01095422264188528,
+            0.003462019143626094,
+            0.015272277407348156,
+            -0.0009819786064326763,
+            -0.029029447585344315,
+            -0.0012272365856915712,
+            -0.00024194362049456686,
+            0.02918095886707306,
+            0.01486319862306118,
+            -0.008961857296526432,
+            -0.021211495622992516,
+            -0.013242034241557121,
+            -0.001237652963027358,
+            -0.015726808458566666,
+            -0.009620928205549717,
+            0.011855711229145527,
+            0.0009137988090515137,
+            0.008992158807814121,
+            -0.018135828897356987,
+            -0.027347678318619728,
+            -0.016166189685463905,
+            0.0018465365283191204,
+            -0.012870832346379757,
+            0.003340810537338257,
+            0.015514694154262543,
+            -0.03430201858282089,
+            -0.015060162171721458,
+            -0.03805948421359062,
+            0.008280059322714806,
+            -0.0029658216517418623,
+            -0.02024182677268982,
+            -0.023605365306138992,
+            0.009484568610787392,
+            0.019181253388524055,
+            -0.0015548784285783768,
+            0.019847899675369263,
+            -0.000889651826582849,
+            0.020060013979673386,
+            -0.0073141781613230705,
+            0.0031703610438853502,
+            -0.013007191941142082,
+            0.0030245319940149784,
+            0.008333087898790836,
+            0.02890823967754841,
+            -0.023075077682733536,
+            -0.006503595970571041,
+            -0.004685468040406704,
+            0.004746072459965944,
+            0.012658718042075634,
+            0.014613205567002296,
+            0.020741812884807587,
+            -0.008272483013570309,
+            -0.0027612822595983744,
+            0.012484480626881123,
+            -0.003799130441620946,
+            0.007295239716768265,
+            0.002390081062912941,
+            0.0019677451346069574,
+            0.004162756260484457,
+            0.008477022871375084,
+            0.0053710537031292915,
+            0.027650700882077217,
+            -0.0028275682125240564,
+            0.0031703610438853502,
+            0.024620486423373222,
+            0.02322658896446228,
+            -0.018878230825066566,
+            -0.028074929490685463,
+            0.002024561632424593,
+            0.050695475190877914,
+            0.007245998363941908,
+            -0.006810405291616917,
+            -0.03927157074213028,
+            -0.0031760428100824356,
+            0.028635520488023758,
+            0.023650817573070526,
+            0.01584801822900772,
+            -0.010567869991064072,
+            -0.012098127976059914,
+            0.02772645466029644,
+            0.013295062817633152,
+            0.02616589516401291,
+            -0.025302285328507423,
+            0.010704229585826397,
+            -0.03869583085179329,
+            0.0449683703482151,
+            0.004439263138920069,
+            -0.014476845972239971,
+            0.0011884119594469666,
+            0.010386057198047638,
+            -0.004537744913250208,
+            0.006431628484278917,
+            -5.8059369621332735e-05,
+            -0.0008086882880888879,
+            0.011923890560865402,
+            0.012007221579551697,
+            0.004265025723725557,
+            -0.02399929240345955,
+            0.007359631359577179,
+            0.004272601567208767,
+            0.03005971945822239,
+            0.005431658122688532,
+            -0.0013863227795809507,
+            0.012385998852550983,
+            -0.005810434930026531,
+            0.008855799213051796,
+            -0.010719381272792816,
+            0.020469093695282936,
+            -0.02918095886707306,
+            1.0986003871948924e-05,
+            0.02051454596221447,
+            -0.0015785519499331713,
+            0.0038199631962925196,
+            -0.023741723969578743,
+            0.01186328660696745,
+            -0.052968136966228485,
+            0.021166043356060982,
+            -0.018741872161626816,
+            0.021741783246397972,
+            0.019590331241488457,
+            -0.009734561666846275,
+            -0.012052674777805805,
+            0.03545350208878517,
+            0.0026760573964565992,
+            -0.014264730736613274,
+            -0.010643625631928444,
+            -0.017999470233917236,
+            -0.02830219641327858,
+            0.041968461126089096,
+            -0.0006117243901826441,
+            0.0028275682125240564,
+            0.0024222771171480417,
+            -0.010666352696716785,
+            -0.008923979476094246,
+            0.001954487757757306,
+            0.020499395206570625,
+            -0.03369597718119621,
+            -0.011128460057079792,
+            -0.008689138107001781,
+            -0.02566591091454029,
+            -0.00020300064352340996,
+            -0.011961768381297588,
+            0.026211349293589592,
+            0.01240872498601675,
+            -0.03696860745549202,
+            0.003780191531404853,
+            0.023650817573070526,
+            0.001603172393515706,
+            0.018120678141713142,
+            0.002130619017407298,
+            0.0010795136913657188,
+            0.009742137044668198,
+            -0.010476963594555855,
+            0.026559824123978615,
+            -0.009204274043440819,
+            0.008658835664391518,
+            -0.001950700068846345,
+            -0.017287369817495346,
+            0.017969166859984398,
+            0.004083212930709124,
+            0.024120500311255455,
+            0.011954193003475666,
+            0.011605718173086643,
+            0.01112088467925787,
+            -0.019878201186656952,
+            0.020090317353606224,
+            -0.0018891488434746861,
+            0.009075489826500416,
+            0.024726543575525284,
+            -0.005863463506102562,
+            0.020832719281315804,
+            0.015257125720381737,
+            -0.00020229043730068952,
+            0.011295121163129807,
+            -0.020120618864893913,
+            -0.011779955588281155,
+            0.0085679292678833,
+            -0.009605777449905872,
+            -0.010325453244149685,
+            0.009901223704218864,
+            -0.010105762630701065,
+            -0.015052586793899536,
+            -0.0071247899904847145,
+            0.016635872423648834,
+            0.0028976418543606997,
+            -0.02610529027879238,
+            -1.7326075976598077e-05,
+            0.008901252411305904,
+            0.006166485138237476,
+            0.018741872161626816,
+            0.011317848227918148,
+            -0.006692984607070684,
+            0.010363331064581871,
+            0.011476934887468815,
+            0.0053104497492313385,
+            -0.005079395603388548,
+            -0.013431422412395477,
+            -0.0024241709616035223,
+            0.018090376630425453,
+            0.009810317307710648,
+            -0.01922670565545559,
+            0.004564259201288223,
+            0.0008877579239197075,
+            -0.01420412678271532,
+            -0.019347915425896645,
+            -0.0038483713287860155,
+            -0.004170331638306379,
+            0.0027915844693779945,
+            0.0509984977543354,
+            -0.0006216673064045608,
+            0.007893706671893597,
+            -0.025741664692759514,
+            -0.006299056578427553,
+            -0.00027556007262319326,
+            -0.0036097420379519463,
+            0.01272689737379551,
+            -0.01824188604950905,
+            -0.03724132478237152,
+            0.012711746618151665,
+            -0.0012584856012836099,
+            0.014923802576959133,
+            -0.00297907879576087,
+            0.014226853847503662,
+            -0.017347972840070724,
+            0.01916610263288021,
+            -0.0011117096291854978,
+            -0.016029831022024155,
+            0.0029374135192483664,
+            -0.011143610812723637,
+            -0.021317552775144577,
+            0.005916492082178593,
+            -0.006848283112049103,
+            -0.03896854817867279,
+            0.0029885482508689165,
+            -0.010423935018479824,
+            0.0011798894265666604,
+            -0.003912763670086861,
+            0.020166072994470596,
+            -0.0061513339169323444,
+            0.0008697660523466766,
+            0.02177208475768566,
+            0.0017026013229042292,
+            0.01601468026638031,
+            -0.014499573037028313,
+            0.010999675840139389,
+            0.012598113156855106,
+            0.02210540883243084,
+            -0.032180868089199066,
+            -0.01393140759319067,
+            0.014211702160537243,
+            0.0026116655208170414,
+            -0.005382417235523462,
+            0.020635755732655525,
+            -0.01294658798724413,
+            0.019650936126708984,
+            -0.0005577487172558904,
+            -0.011704200878739357,
+            -0.03524138405919075,
+            -0.016529815271496773,
+            -3.343888238305226e-05,
+            -0.008439145050942898,
+            0.025317436084151268,
+            0.01760554127395153,
+            -0.0108330138027668,
+            0.026468917727470398,
+            0.0037574649322777987,
+            0.014598054811358452,
+            -0.014257155358791351,
+            -0.005999823100864887,
+            -0.0003773563075810671,
+            -0.0119314668700099,
+            0.015264702029526234,
+            -0.03512017801403999,
+            0.0008342557121068239,
+            -0.00839369185268879,
+            0.005242269951850176,
+            -0.0254992488771677,
+            0.018257036805152893,
+            0.025817420333623886,
+            -0.0031419529113918543,
+            -0.005037730094045401,
+            -0.022802358493208885,
+            0.028968842700123787,
+            0.03049910068511963,
+            0.0024828813038766384,
+            -0.0013825349742546678,
+            -0.00352072948589921,
+            0.0033218718599528074,
+            0.0211205892264843,
+            -0.037998881191015244,
+            -0.009098216891288757,
+            -0.028696123510599136,
+            0.042332086712121964,
+            -0.008098246529698372,
+            -0.013431422412395477,
+            0.01862066239118576,
+            -0.0024715180043131113,
+            0.02355991117656231,
+            -0.004689255729317665,
+            -0.004855917301028967,
+            0.03148392215371132,
+            0.011514811776578426,
+            0.02772645466029644,
+            -0.04015033319592476,
+            0.014704111963510513,
+            -0.006867222022265196,
+            0.004867280833423138,
+            -0.013484450988471508,
+            0.005302873905748129,
+            0.02580226957798004,
+            -0.018060073256492615,
+            0.0068066176027059555,
+            0.030665762722492218,
+            0.00748084019869566,
+            0.002448791405186057,
+            0.0165904201567173,
+            0.003890036838129163,
+            0.010855740867555141,
+            0.000841831264551729,
+            0.0052574207074940205,
+            0.022423582151532173,
+            0.014454119838774204,
+            -0.016029831022024155,
+            -0.015120766125619411,
+            -0.00044790346873924136,
+            -0.004450626205652952,
+            0.005261208396404982,
+            0.013242034241557121,
+            0.0061172437854111195,
+            -0.015681356191635132,
+            0.01052999310195446,
+            0.014158673584461212,
+            0.013128400780260563,
+            -0.0039847311563789845,
+            -0.03412020578980446,
+            -0.02443867363035679,
+            -0.021438762545585632,
+            0.003104075090959668,
+            -0.0008281006012111902,
+            -0.0021211495622992516,
+            0.0017954016802832484,
+            0.017347972840070724,
+            -0.0008352026343345642,
+            0.00379155483096838,
+            0.01686313934624195,
+            -0.01415109820663929,
+            -0.022090258076786995,
+            -0.0035964848939329386,
+            -0.027529491111636162,
+            0.0053710537031292915,
+            0.001844642567448318,
+            -0.00965123064815998,
+            0.008530051447451115,
+            -0.0046059247106313705,
+            -0.007223271764814854,
+            -0.012196609750390053,
+            -0.02007516659796238,
+            0.005780132487416267,
+            0.016878290101885796,
+            0.005730891600251198,
+            -0.009272454306483269,
+            -0.003939277958124876,
+            0.011575416661798954,
+            0.017484333366155624,
+            0.017317671328783035,
+            -0.009295180439949036,
+            0.009628504514694214,
+            0.0004022135108243674,
+            0.004670316819101572,
+            -0.01834794320166111,
+            0.0045869858004152775,
+            -0.0027953721582889557,
+            -0.008658835664391518,
+            0.01105270441621542,
+            -0.00031201730598695576,
+            0.01628739945590496,
+            -0.013090522959828377,
+            0.005624833982437849,
+            0.010333028621971607,
+            -0.020590301603078842,
+            -0.0067308624275028706,
+            -0.01692374423146248,
+            0.026484068483114243,
+            -0.00504530593752861,
+            0.00045405857963487506,
+            -0.02978700026869774,
+            0.0033616432920098305,
+            0.000848933297675103,
+            -0.000845619011670351,
+            -0.012764775194227695,
+            0.010749682784080505,
+            0.03242328763008118,
+            0.0005179771687835455,
+            0.0216660276055336,
+            -0.0036154238041490316,
+            0.0007480839849449694,
+            -0.00785582885146141,
+            0.004977126140147448,
+            -0.004916521720588207,
+            -0.009901223704218864,
+            -0.010863316245377064,
+            0.021726632490754128,
+            -0.0033029329497367144,
+            0.0003356908564455807,
+            0.011332998983561993,
+            -0.012067826464772224,
+            0.0014620780711993575,
+            -0.0047687990590929985,
+            0.00971941091120243,
+            0.0061702728271484375,
+            0.009113367646932602,
+            -0.01176480483263731,
+            0.05451354384422302,
+            -0.0106284748762846,
+            0.005537715740501881,
+            -0.00439380994066596,
+            -0.022226616740226746,
+            -0.017317671328783035,
+            0.012711746618151665,
+            -0.017544938251376152,
+            0.03375658020377159,
+            0.024150803685188293,
+            0.0006467612111009657,
+            0.010393633507192135,
+            -0.006958128418773413,
+            0.023923536762595177,
+            -0.002602196065708995,
+            0.0010795136913657188,
+            -0.004855917301028967,
+            0.015802564099431038,
+            0.04511988162994385,
+            0.017393426969647408,
+            -0.001152428099885583,
+            0.00829521007835865,
+            0.020575150847434998,
+            0.00814369972795248,
+            -0.0020075165666639805,
+            0.014219277538359165,
+            -0.01024212222546339,
+            0.00900731049478054,
+            -0.004458202049136162,
+            0.027438584715127945,
+            0.00814369972795248,
+            -0.006310420110821724,
+            -0.0036836036015301943,
+            -0.0008209985098801553,
+            0.014173824340105057,
+            0.006950552575290203,
+            0.01963578537106514,
+            -0.00461350055411458,
+            -0.0012575386790558696,
+            0.013870803639292717,
+            0.014491996727883816,
+            -0.00552635220810771,
+            0.001724380999803543,
+            -0.01435563713312149,
+            -0.010204244405031204,
+            0.013575357384979725,
+            0.006086941808462143,
+            0.006310420110821724,
+            -0.014514723792672157,
+            -0.00980274099856615,
+            -0.004670316819101572,
+            0.0015435151290148497,
+            0.015052586793899536,
+            0.011060279794037342,
+            -0.004814252257347107,
+            0.006052851676940918,
+            0.013825350441038609,
+            -0.014014738611876965,
+            0.006147546228021383,
+            0.007208121009171009,
+            0.008855799213051796,
+            0.004787737503647804,
+            -0.010817863047122955,
+            -0.00428396463394165,
+            -0.0063937511295080185,
+            0.0012670081341639161,
+            0.016832837834954262,
+            0.013181429356336594,
+            0.0049581872299313545,
+            0.027529491111636162,
+            -0.00839369185268879,
+            -0.012302667833864689,
+            0.02339324913918972,
+            0.011257244274020195,
+            0.0031703610438853502,
+            0.014651083387434483,
+            -0.012635990977287292,
+            -0.012446602806448936,
+            0.006655106786638498,
+            0.004848341923207045,
+            0.006049063988029957,
+            0.009151245467364788,
+            0.00450365524739027,
+            0.007333117071539164,
+            -0.007098275702446699,
+            -0.007427811156958342,
+            0.0014810169814154506,
+            0.01642375811934471,
+            0.020393338054418564,
+            -0.024968961253762245,
+            -0.0018806264270097017,
+            0.0009052763343788683,
+            0.010878467001020908,
+            -0.003316190093755722,
+            0.012310243211686611,
+            5.65205882594455e-05,
+            -0.0074240234680473804,
+            0.010158791206777096,
+            -0.02119634486734867,
+            -0.012681444175541401,
+            0.0012717428617179394,
+            0.013037494383752346,
+            -0.019590331241488457,
+            0.018226735293865204,
+            -0.008893677033483982,
+            -0.013348091393709183,
+            0.013893529772758484,
+            0.020014561712741852,
+            0.004090788308531046,
+            -0.007738408166915178,
+            -0.021862991154193878,
+            -0.01127239502966404,
+            0.016060132533311844,
+            0.020272130146622658,
+            0.000947888707742095,
+            -0.0006472347304224968,
+            0.010173942893743515,
+            -0.013113250024616718,
+            -0.023605365306138992,
+            -0.005230906419456005,
+            0.010386057198047638,
+            0.01746918261051178,
+            0.018832778558135033,
+            -0.006685409229248762,
+            0.02443867363035679,
+            -0.0059127043932676315,
+            -0.015499543398618698,
+            2.86597842205083e-05,
+            -0.02802947722375393,
+            -0.017423728480935097,
+            0.00013778003631159663,
+            0.00246773031540215,
+            0.008598231710493565,
+            0.0008555619278922677,
+            0.01943882182240486,
+            0.000904329412151128,
+            0.017590390518307686,
+            -0.000659071491099894,
+            -0.002009410411119461,
+            -0.0013995799235999584,
+            -0.0015340456739068031,
+            0.02254479005932808,
+            -0.012166308239102364,
+            -0.01176480483263731,
+            0.022302372381091118,
+            0.006852070800960064,
+            0.014719262719154358,
+            -0.027635548263788223,
+            0.013757170177996159,
+            -0.015499543398618698,
+            0.017423728480935097,
+            -0.03408990427851677,
+            0.006420265417546034,
+            0.01489350013434887,
+            -0.010348179377615452,
+            -0.020499395206570625,
+            -0.029650641605257988,
+            -0.024150803685188293,
+            -0.0038710979279130697,
+            -0.010855740867555141,
+            -0.016847988590598106,
+            0.006121031939983368,
+            -0.005802859086543322,
+            0.018499454483389854,
+            -0.007079336792230606,
+            0.033877789974212646,
+            0.006969491485506296,
+            -0.004056698642671108,
+            0.003835114184767008,
+            0.022469034418463707,
+            -0.006102093029767275,
+            0.02109028771519661,
+            0.004408961161971092,
+            -0.001467759720981121,
+            0.013681414537131786,
+            -0.006552837323397398,
+            -0.002361672930419445,
+            0.005821797996759415,
+            -0.010484539903700352,
+            0.022014502435922623,
+            0.005924067925661802,
+            -0.007708106189966202,
+            -0.0006775368237867951,
+            -0.011454207822680473,
+            0.014726839028298855,
+            0.006530110724270344,
+            0.0022480397019535303,
+            -0.011295121163129807,
+            0.011249667964875698,
+            -0.004859705455601215,
+            0.0040642740204930305,
+            0.009484568610787392,
+            -0.01774190180003643,
+            -0.028862785547971725,
+            0.011416330002248287,
+            0.019120648503303528,
+            0.006382387597113848,
+            -0.0055755930952727795,
+            -0.0025681061670184135,
+            0.00839369185268879,
+            0.010651201009750366,
+            0.022150862962007523,
+            -0.012984465807676315,
+            -8.889415767043829e-05,
+            0.013204156421124935,
+            -0.007170243188738823,
+            -0.01056029461324215,
+            0.00487485621124506,
+            0.023438703268766403,
+            0.0068823727779090405,
+            -0.04327145218849182,
+            -0.018469152972102165,
+            0.01909034699201584,
+            0.007136153522878885,
+            -0.04154422879219055,
+            0.0026381798088550568,
+            -0.003689285134896636,
+            -0.023787178099155426,
+            0.016105586662888527,
+            -0.0011817833874374628,
+            -0.01828734017908573,
+            0.028756728395819664,
+            0.02830219641327858,
+            0.01834794320166111,
+            2.6987840101355687e-05,
+            -0.013332940638065338,
+            -0.0318172425031662,
+            -0.023590214550495148,
+            0.008280059322714806,
+            -0.02660527639091015,
+            -0.0072725131176412106,
+            0.011242092587053776,
+            -0.017454031854867935,
+            0.02972639724612236,
+            -0.008507325313985348,
+            0.026938600465655327,
+            0.011446632444858551,
+            0.001691238023340702,
+            -0.013446573168039322,
+            -0.018802475184202194,
+            -0.01350717805325985,
+            -0.008196728304028511,
+            0.014090493321418762,
+            0.017923714593052864,
+            0.011007251217961311,
+            0.011332998983561993,
+            0.02251448854804039,
+            -0.012158732861280441,
+            0.007969462312757969,
+            -2.888172457460314e-05,
+            -0.008961857296526432,
+            0.0026343921199440956,
+            -0.018393397331237793,
+            0.00868156272917986,
+            0.0062308767810463905,
+            0.005458172410726547,
+            -0.007208121009171009,
+            0.010742107406258583,
+            0.02712041325867176,
+            -0.018847929313778877,
+            0.011401179246604443,
+            -0.031362712383270264,
+            -0.014787442982196808,
+            -0.01056029461324215,
+            0.026862844824790955,
+            0.007757347077131271,
+            -0.015726808458566666,
+            -0.007514929864555597,
+            -0.01545409020036459,
+            0.012885984033346176,
+            0.0032442226074635983,
+            0.005764981731772423,
+            0.015817714855074883,
+            0.021893294528126717,
+            0.01746918261051178,
+            -0.00795431062579155,
+            0.006852070800960064,
+            0.004674104508012533,
+            0.028090080246329308,
+            -0.01777220331132412,
+            0.020802417770028114,
+            0.009954252280294895,
+            -0.003935490269213915,
+            0.005685438401997089,
+            -0.014257155358791351,
+            0.008530051447451115,
+            -0.007325541693717241,
+            0.030514251440763474,
+            0.0033067206386476755,
+            0.014075342565774918,
+            -0.02062060497701168,
+            0.006575563922524452,
+            -0.03172633796930313,
+            0.009022461250424385,
+            0.013704141601920128,
+            0.012976890429854393,
+            -0.007677804213017225,
+            -0.024181105196475983,
+            -0.0119314668700099,
+            0.0057990713976323605,
+            -0.01279507763683796,
+            -0.01046181283891201,
+            -0.0034828518982976675,
+            0.012446602806448936,
+            0.018847929313778877,
+            -0.01848430372774601,
+            -0.019923655316233635,
+            -0.009583051316440105,
+            0.02898399345576763,
+            -0.015787413343787193,
+            -0.001471547526307404,
+            0.02413565292954445,
+            0.021241798996925354,
+            0.015022284351289272,
+            0.002155239460989833,
+            0.020090317353606224,
+            -0.0007111532613635063,
+            -0.0237265732139349,
+            -0.02813553437590599,
+            0.007302815094590187,
+            -0.006912675220519304,
+            0.01943882182240486,
+            -0.0035548193845897913,
+            -0.005658924113959074,
+            -0.0034033088013529778,
+            -0.007503566797822714,
+            -0.0165904201567173,
+            -0.01787826046347618,
+            0.0021647089160978794,
+            -0.008454295806586742,
+            0.005863463506102562,
+            -0.0002795845502987504,
+            -0.013105674646794796,
+            0.013401119969785213,
+            -0.014075342565774918,
+            0.005772557109594345,
+            -0.002653330797329545,
+            0.001962063368409872,
+            0.0035926972050219774,
+            0.009113367646932602,
+            -0.007336904760450125,
+            -0.06381630152463913,
+            -0.013105674646794796,
+            -0.003704436356201768,
+            0.006071790587157011,
+            -0.00019578021601773798,
+            -0.016635872423648834,
+            -0.013196581043303013,
+            0.01706010289490223,
+            0.008863375522196293,
+            -0.013870803639292717,
+            0.016954045742750168,
+            0.009961827658116817,
+            0.003622999181970954,
+            -0.024044746533036232,
+            -0.0008148433989845216,
+            0.008757317438721657,
+            0.032756611704826355,
+            -0.008643684908747673,
+            -0.0003839849086944014,
+            0.0036684523802250624,
+            -0.0033332351595163345,
+            -0.010052734054625034,
+            -0.014582903124392033,
+            -0.011848135851323605,
+            0.011908739805221558,
+            0.01790856383740902,
+            -0.014264730736613274,
+            -0.02007516659796238,
+            -0.0007106798002496362,
+            0.012931437231600285,
+            -0.026590125635266304,
+            0.0007244104635901749,
+            -0.007787649054080248,
+            -0.01052999310195446,
+            0.018302490934729576,
+            -0.019014591351151466,
+            0.005249845329672098,
+            0.00048294031876139343,
+            0.008507325313985348,
+            -0.023544760420918465,
+            -0.024893205612897873,
+            0.0035131541080772877,
+            -0.023847781121730804,
+            0.03612014651298523,
+            -0.021590271964669228,
+            -0.012082977220416069,
+            0.034483831375837326,
+            -0.0037423139438033104,
+            0.01702980138361454,
+            0.0023995505180209875,
+            0.02739313244819641,
+            -0.011439057067036629,
+            -0.00617784820497036,
+            0.012499631382524967,
+            0.009575475007295609,
+            0.0032593735959380865,
+            0.03775646165013313,
+            -0.01257538702338934,
+            -0.013901105150580406,
+            -0.005556654185056686,
+            -0.012954163365066051,
+            0.0006775368237867951,
+            0.014431392773985863,
+            0.028393102809786797,
+            0.004257450345903635,
+            -0.00628769351169467,
+            -0.006143758539110422,
+            -0.0014649189542979002,
+            -0.004242299124598503,
+            -0.00947699323296547,
+            -0.00715130427852273,
+            0.017287369817495346,
+            -0.025241680443286896,
+            0.016711628064513206,
+            -0.006420265417546034,
+            0.021135739982128143,
+            -0.011757229454815388,
+            0.007007369305938482,
+            -0.008961857296526432,
+            0.003492321353405714,
+            0.0023635667748749256,
+            -0.005109698045998812,
+            0.009287605062127113,
+            -0.014772292226552963,
+            0.008454295806586742,
+            0.013742019422352314,
+            0.0011183382011950016,
+            -0.010590597055852413,
+            -0.019272159785032272,
+            0.0022972808219492435,
+            -0.000710206339135766,
+            0.01181783340871334,
+            -0.0047498601488769054,
+            -0.000981031684204936,
+            0.0002921315317507833,
+            -0.014476845972239971,
+            -0.009575475007295609,
+            0.0006050958181731403,
+            0.005302873905748129,
+            -0.008961857296526432,
+            0.010378481820225716,
+            -0.005693014245480299,
+            -0.00034160923678427935,
+            0.02627195231616497,
+            0.006526322569698095,
+            -0.007632351014763117,
+            -0.014098069630563259,
+            0.01563590206205845,
+            0.015370759181678295,
+            -0.007454325910657644,
+            0.01933276280760765,
+            -0.016196493059396744,
+            -0.0008366230758838356,
+            0.013916256837546825,
+            -0.009219424799084663,
+            0.0017698341980576515,
+            -0.008196728304028511,
+            0.03554440662264824,
+            -0.0008787619881331921,
+            -0.0035415622405707836,
+            -0.0006401326390914619,
+            0.012976890429854393,
+            -0.0026779514737427235,
+            0.001182730309665203,
+            0.01132542360574007,
+            0.005321812815964222,
+            0.0185146052390337,
+            0.00328020635060966,
+            -0.03378688171505928,
+            -0.007780073676258326,
+            -0.0065490491688251495,
+            -0.008090670220553875,
+            0.009022461250424385,
+            0.025196226313710213,
+            0.01990850456058979,
+            0.006408901885151863,
+            0.001881573349237442,
+            0.018423698842525482,
+            0.015757111832499504,
+            -0.01489350013434887,
+            0.016363153234124184,
+            0.02802947722375393,
+            -0.010030006989836693,
+            0.022741753607988358,
+            -0.005253633018583059,
+            -0.008893677033483982,
+            0.008946705609560013,
+            -0.0015378334792330861,
+            -0.017393426969647408,
+            -0.015681356191635132,
+            0.005022579338401556,
+            -0.008999735116958618,
+            -0.0022897052112966776,
+            -0.018605511635541916,
+            0.004670316819101572,
+            0.017105557024478912,
+            -0.004965762607753277,
+            -3.5243989259470254e-05,
+            0.019484274089336395,
+            -0.006965703796595335,
+            -0.016908591613173485,
+            0.006325571332126856,
+            -0.015007133595645428,
+            0.007946735247969627,
+            -0.013666263781487942,
+            0.010492115281522274,
+            -0.006098304875195026,
+            0.018817627802491188,
+            0.005446809343993664,
+            -0.008090670220553875,
+            -0.0008958069374784827,
+            -0.018984289839863777,
+            0.02065090648829937,
+            -0.033089932054281235,
+            0.00839369185268879,
+            -0.026226500049233437,
+            -0.0009781909175217152,
+            0.0008849170990288258,
+            0.014605630189180374,
+            -0.001218714052811265,
+            0.019696388393640518,
+            -0.002751812804490328,
+            -0.020090317353606224,
+            -0.007086912170052528,
+            -0.0010634156642481685,
+            -0.01618134044110775,
+            -0.030923331156373024,
+            -0.004272601567208767,
+            0.0036267871037125587,
+            -0.004711982328444719,
+            0.0021135739516466856,
+            0.017984317615628242,
+            -0.005613470915704966,
+            0.009173971600830555,
+            -0.02690829709172249,
+            -0.007416448090225458,
+            -0.00355103169567883,
+            0.0010776197304949164,
+            -0.011242092587053776,
+            0.0010596278589218855,
+            0.011226941831409931,
+            -0.019044892862439156,
+            0.013461724855005741,
+            -0.021378157660365105,
+            -0.004371083341538906,
+            -0.008507325313985348,
+            -0.00135412672534585,
+            -0.013446573168039322,
+            -0.028120383620262146,
+            0.025029566138982773,
+            -0.00021495578403118998,
+            -0.003528305096551776,
+            -0.011999646201729774,
+            0.023181134834885597,
+            0.002914686920121312,
+            0.027817361056804657,
+            -0.03157482668757439,
+            0.0013427634257823229,
+            0.011552689597010612,
+            0.0025984083767980337,
+            -0.017287369817495346,
+            0.004746072459965944,
+            -0.005234694108366966,
+            0.02210540883243084,
+            -0.0011694730492308736,
+            -0.0005340751959010959,
+            -0.008234606124460697,
+            0.030999086797237396,
+            0.009136094711720943,
+            0.0012944694608449936,
+            -0.013226882554590702,
+            -0.008060368709266186,
+            -0.002759388415142894,
+            -0.0011069749016314745,
+            0.012855681590735912,
+            0.01290113478899002,
+            -0.012105703353881836,
+            0.0013607552973553538,
+            -0.006878585088998079,
+            0.0011694730492308736,
+            -0.002935519441962242,
+            -0.0004966709529981017,
+            -0.01149208564311266,
+            0.003257479751482606,
+            0.016272246837615967,
+            -0.00780280027538538,
+            -0.006480869371443987,
+            0.0007750718505121768,
+            0.009181547909975052,
+            0.012461753562092781,
+            -0.012742049060761929,
+            -0.018226735293865204,
+            -0.011908739805221558,
+            0.016029831022024155,
+            0.010522416792809963,
+            -0.008204303681850433,
+            -0.013234458863735199,
+            -0.02583257108926773,
+            0.01169662456959486,
+            -0.03293842449784279,
+            -0.012242063879966736,
+            0.005821797996759415,
+            0.028559764847159386,
+            0.019999410957098007,
+            0.012226912193000317,
+            0.001332347048446536,
+            0.002950670663267374,
+            0.00023342114582192153,
+            -0.01916610263288021,
+            0.014408666640520096,
+            -0.011143610812723637,
+            -0.018166130408644676,
+            0.0075906855054199696,
+            0.011939042247831821,
+            0.03148392215371132,
+            0.006942977197468281,
+            -0.0010160685051232576,
+            0.01686313934624195,
+            -0.001218714052811265,
+            -0.013582932762801647,
+            0.01420412678271532,
+            -0.024256860837340355,
+            0.016438908874988556,
+            0.02860521711409092,
+            -0.004780162125825882,
+            0.0008531945641152561,
+            -0.020938776433467865,
+            0.00482561532407999,
+            -0.011529963463544846,
+            -0.002177966060116887,
+            -0.003287781961262226,
+            -0.03460504114627838,
+            0.007946735247969627,
+            -0.006825556512922049,
+            0.01046181283891201,
+            -0.00814369972795248,
+            0.006655106786638498,
+            -0.012431452050805092,
+            0.004253662656992674,
+            0.01513591781258583,
+            0.016105586662888527,
+            0.007060397882014513,
+            -0.025211378931999207,
+            -0.009196698665618896,
+            0.008105821907520294,
+            -0.01349202636629343,
+            -0.02295386977493763,
+            -0.006852070800960064,
+            0.009840618818998337,
+            -0.01061332318931818,
+            -0.00488243205472827,
+            -0.028180986642837524,
+            -0.013681414537131786,
+            -0.0014327229000627995,
+            -0.017817657440900803,
+            -0.005556654185056686,
+            -0.003100287402048707,
+            0.01571165770292282,
+            -0.0049733384512364864,
+            -0.006185423582792282,
+            0.004742284305393696,
+            -0.023878084495663643,
+            0.0037423139438033104,
+            -0.008280059322714806,
+            0.0008266801596619189,
+            -0.0032517979852855206,
+            -0.0014762822538614273,
+            -0.007969462312757969,
+            -0.014332910999655724,
+            0.006893736310303211,
+            -0.016469212248921394,
+            0.002954458352178335,
+            -0.015173795633018017,
+            -0.016499513760209084,
+            -0.00682934420183301,
+            0.00775355938822031,
+            0.021575121209025383,
+            0.006541473791003227,
+            0.021484214812517166,
+            0.005469535943120718,
+            0.028680972754955292,
+            0.0003953482082579285,
+            0.003041577059775591,
+            -8.958956641436089e-06,
+            0.003232859307900071,
+            0.019347915425896645,
+            0.006571775767952204,
+            0.018984289839863777,
+            -0.008742166683077812,
+            0.010514841414988041,
+            -0.010355755686759949,
+            0.010014856234192848,
+            0.006333146709948778,
+            -0.006473293993622065,
+            -0.003799130441620946,
+            -0.010348179377615452,
+            -0.0031419529113918543,
+            -0.016544966027140617,
+            0.0027366618160158396,
+            0.00693161366507411,
+            0.0071588801220059395,
+            -0.011582992039620876,
+            -0.005230906419456005,
+            -0.014704111963510513,
+            -0.0064354161731898785,
+            -0.01831764169037342,
+            0.014385939575731754,
+            0.029226411134004593,
+            -0.016332851722836494,
+            -0.01977214403450489,
+            -0.007772498298436403,
+            0.0007296186522580683,
+            -0.006636167876422405,
+            -0.01240872498601675,
+            -0.009151245467364788,
+            -0.035059571266174316,
+            -0.017666146159172058,
+            0.007586897816509008,
+            0.02484775334596634,
+            0.01387837901711464,
+            -0.013628385961055756,
+            -0.0006983695784583688,
+            -0.016984347254037857,
+            -0.010787560604512691,
+            -0.01535560842603445,
+            -0.02054484933614731,
+            -0.02498411200940609,
+            0.0077952248975634575,
+            -0.004033972043544054,
+            0.014931377954781055,
+            0.008113397285342216,
+            0.0032631612848490477,
+            -0.004371083341538906,
+            -0.0024109138175845146,
+            -0.013196581043303013,
+            0.0012925754999741912,
+            -0.004787737503647804,
+            -0.0017291157273575664,
+            0.03554440662264824,
+            -0.008098246529698372,
+            -0.005871038883924484,
+            0.011060279794037342,
+            0.011128460057079792,
+            -0.013764745555818081,
+            0.0006642796797677875,
+            -0.011249667964875698,
+            0.021847840398550034,
+            -0.021620575338602066,
+            0.019257009029388428,
+            0.0014507147716358304,
+            -0.010840589180588722,
+            0.028999146074056625,
+            0.01892368495464325,
+            0.00743159931153059,
+            0.006579351611435413,
+            0.01672677882015705,
+            0.0044316877610981464,
+            0.016196493059396744,
+            -0.0026552248746156693,
+            -0.004632438998669386,
+            0.009295180439949036,
+            0.015681356191635132,
+            -0.000702630786690861,
+            -0.006942977197468281,
+            0.016438908874988556,
+            0.004056698642671108,
+            0.0075414446182549,
+            -0.01356020662933588,
+            -0.008961857296526432,
+            0.001507531269453466,
+            0.006257391534745693,
+            -0.005939218681305647,
+            -0.0026003022212535143,
+            0.005530139897018671,
+            0.003943065647035837,
+            -0.013302638195455074,
+            0.01689344085752964,
+            -0.016166189685463905,
+            0.008742166683077812,
+            -0.0037972365971654654,
+            0.0121360057964921,
+            -0.006799042224884033,
+            0.0013153020991012454,
+            -0.01818128302693367,
+            -0.003869204083457589,
+            0.0038597346283495426,
+            0.012173883616924286,
+            -0.01262841559946537,
+            0.025484098121523857,
+            0.008166425861418247,
+            0.006602078210562468,
+            0.0059467945247888565,
+            -0.012855681590735912,
+            0.027211319655179977,
+            0.005969521123915911,
+            0.009886072017252445,
+            -0.008893677033483982,
+            0.011136035434901714,
+            0.035362593829631805,
+            0.03060515783727169,
+            0.014135947450995445,
+            0.004469565115869045,
+            -0.018999440595507622,
+            0.015120766125619411,
+            0.0018427487229928374,
+            0.0023806116078048944,
+            0.00863610953092575,
+            -0.011348150670528412,
+            0.01305264513939619,
+            -0.009484568610787392,
+            0.015241974964737892,
+            -0.021272100508213043,
+            -0.011749654076993465,
+            -0.014408666640520096,
+            -0.021075136959552765,
+            -0.01430260855704546,
+            0.034180812537670135,
+            -0.002346521709114313,
+            -0.010393633507192135,
+            0.0045112306252121925,
+            0.008477022871375084,
+            -0.006079366430640221,
+            0.0053142374381423,
+            -0.008204303681850433,
+            -0.013749594800174236,
+            -0.01889338344335556,
+            0.006628592498600483,
+            -0.01550711877644062,
+            0.007651289459317923,
+            -0.003304826794192195,
+            0.0037688282318413258,
+            0.007014944683760405,
+            0.018105527386069298,
+            0.00530666159465909,
+            -0.00135412672534585,
+            0.0014033677289262414,
+            0.0016865032957866788,
+            -0.00931033119559288,
+            0.0108330138027668,
+            0.015408637002110481,
+            -0.0011921997647732496,
+            -0.011279970407485962,
+            -0.009136094711720943,
+            -0.042877525091171265,
+            0.0006264019757509232,
+            -0.013128400780260563,
+            0.0064354161731898785,
+            -0.011196639388799667,
+            0.0022234192583709955,
+            0.015863168984651566,
+            -0.0054051438346505165,
+            -0.0010918239131569862,
+            0.02193874679505825,
+            0.013014768250286579,
+            0.021075136959552765,
+            0.02065090648829937,
+            0.014188976027071476,
+            -0.015302578918635845,
+            -0.03469594568014145,
+            0.002221525413915515,
+            -0.010030006989836693,
+            0.013772320933640003,
+            0.004238511435687542,
+            0.016060132533311844,
+            -0.023893235251307487,
+            -0.011878437362611294,
+            -0.006662682630121708,
+            0.014431392773985863,
+            0.01491622719913721,
+            -0.0032271775417029858,
+            0.02690829709172249,
+            0.009280029684305191,
+            0.0020510759204626083,
+            -0.02281750924885273,
+            0.003312402404844761,
+            0.005321812815964222,
+            -0.008370965719223022,
+            -0.016681326553225517,
+            -0.0004552422615233809,
+            -0.0012168202083557844,
+            -0.0007817004225216806,
+            -0.021650876849889755,
+            0.018832778558135033,
+            -0.004018820822238922,
+            0.0037574649322777987,
+            0.013840501196682453,
+            0.01516621932387352,
+            0.017984317615628242,
+            0.008370965719223022,
+            0.001497115008533001,
+            -0.006605865899473429,
+            -0.0007973249885253608,
+            -0.0005828426801599562,
+            -0.007389933802187443,
+            -0.005598319694399834,
+            0.006382387597113848,
+            -0.01134057529270649,
+            0.008810346014797688,
+            0.000896753859706223,
+            0.006113456096500158,
+            0.010605747811496258,
+            -0.004836978856474161,
+            -0.011628445237874985,
+            0.014628356322646141,
+            -0.0010548931313678622,
+            -0.0055755930952727795,
+            -0.011060279794037342,
+            0.003624893259257078,
+            0.013575357384979725,
+            -0.003096499713137746,
+            -0.013499601744115353,
+            0.00910579226911068,
+            0.002009410411119461,
+            -0.027438584715127945,
+            0.015499543398618698,
+            0.010295150801539421,
+            -0.013370818458497524,
+            0.004980913829058409,
+            -0.003371112747117877,
+            -0.03027183562517166,
+            0.004791525658220053,
+            0.002660906407982111,
+            -0.010643625631928444,
+            0.028938541188836098,
+            -0.003821857040748,
+            0.029256712645292282,
+            -0.004980913829058409,
+            -0.006905099377036095,
+            -0.011355726048350334,
+            0.009416389279067516,
+            -0.018681267276406288,
+            -0.0037953427527099848,
+            -0.008727015927433968,
+            -0.00915882084518671,
+            -0.0009838725673034787,
+            -0.005185453221201897,
+            0.006984642706811428,
+            0.005121061112731695,
+            -0.012704171240329742,
+            0.006757376715540886,
+            -0.010257272981107235,
+            -0.00020856392802670598,
+            0.0006718551740050316,
+            -0.008408842608332634,
+            0.013885954394936562,
+            0.005996035411953926,
+            -0.008340663276612759,
+            -0.0005032995832152665,
+            0.027817361056804657,
+            0.05051366239786148,
+            -0.018863080069422722,
+            0.0009270560112781823,
+            0.01452229917049408,
+            -0.0028824908658862114,
+            -0.0021590273827314377,
+            -0.011976920068264008,
+            0.005821797996759415,
+            -0.019423669204115868,
+            0.013317788951098919,
+            0.006473293993622065,
+            -0.008833073079586029,
+            0.003202557098120451,
+            -0.016711628064513206,
+            0.002621134975925088,
+            -0.0018058179412037134,
+            -0.018681267276406288,
+            -0.015529844909906387,
+            -0.012060250155627728,
+            -0.024181105196475983,
+            0.0004235197266098112,
+            -0.0008546149474568665,
+            -0.013870803639292717,
+            0.006579351611435413,
+            0.008431569673120975,
+            0.032483890652656555,
+            0.0011912527261301875,
+            0.010128489695489407,
+            -2.1513333194889128e-05,
+            -0.0010416359873488545,
+            -0.005988460034132004,
+            -0.01523439958691597,
+            0.00015387803432531655,
+            -0.002342734020203352,
+            0.022590242326259613,
+            0.005549078807234764,
+            0.017560089007019997,
+            0.0017887729918584228,
+            -0.016514664515852928,
+            -0.00596573343500495,
+            -0.007408872712403536,
+            -0.018060073256492615,
+            0.0062649669125676155,
+            -0.005473323632031679,
+            0.009605777449905872,
+            -0.015863168984651566,
+            -0.02731737680733204,
+            0.00792400911450386,
+            0.012234487570822239,
+            0.028514310717582703,
+            -0.0035623949952423573,
+            0.001716805505566299,
+            -0.005533927585929632,
+            0.016408607363700867,
+            0.032180868089199066,
+            -0.0008550884085707366,
+            0.006950552575290203,
+            -0.019999410957098007,
+            0.0017404790269210935,
+            0.013704141601920128,
+            -0.008401267230510712,
+            -0.01821158453822136,
+            0.003939277958124876,
+            -0.020635755732655525,
+            0.004583198111504316,
+            -0.020393338054418564,
+            0.030408194288611412,
+            -0.0023522034753113985,
+            0.008105821907520294,
+            0.004310478921979666,
+            0.01760554127395153,
+            -0.0031646795105189085,
+            0.010545143857598305,
+            0.013916256837546825,
+            0.014908651821315289,
+            0.0009980766335502267,
+            -0.011189064010977745,
+            -0.03790797293186188,
+            0.001859793672338128,
+            0.00817400123924017,
+            0.018635815009474754,
+            -0.01980244740843773,
+            0.011681473813951015,
+            -0.01296931505203247,
+            -0.020363036543130875,
+            -0.01003758329898119,
+            0.011575416661798954,
+            -0.006139970384538174,
+            0.03378688171505928,
+            -0.005874827038496733,
+            -0.008204303681850433,
+            0.005136212334036827,
+            0.010454237461090088,
+            0.0066778333857655525,
+            0.014704111963510513,
+            0.0326354019343853,
+            0.010196669027209282,
+            0.014173824340105057,
+            -0.004859705455601215,
+            0.02399929240345955,
+            -0.003931702114641666,
+            0.003954428713768721,
+            0.00509454682469368,
+            -0.022832660004496574,
+            -0.0040983641520142555,
+            -0.003931702114641666,
+            -0.016711628064513206,
+            -0.007458113599568605,
+            0.0032366469968110323,
+            0.008802770636975765,
+            -0.014045040123164654,
+            -0.00649980828166008,
+            -0.01350717805325985,
+            0.01218903437256813,
+            0.002730980049818754,
+            0.018302490934729576,
+            -0.029802151024341583,
+            0.009825468063354492,
+            -0.017484333366155624,
+            -0.017044952139258385,
+            0.006511171814054251,
+            -0.0038332203403115273,
+            0.006855858489871025,
+            -0.018272189423441887,
+            -0.01318900566548109,
+            0.012817803770303726,
+            0.0007897494360804558,
+            0.013060221448540688,
+            -0.009825468063354492,
+            0.0009706153068691492,
+            0.006064215209335089,
+            -0.009833043441176414,
+            -0.010961798019707203,
+            -0.007977037690579891,
+            -0.009681533090770245,
+            -0.008848223835229874,
+            0.005621046293526888,
+            0.01656011864542961,
+            -0.013742019422352314,
+            -0.003685497445985675,
+            0.016135888174176216,
+            0.006416477728635073,
+            -0.008166425861418247,
+            0.004075637552887201,
+            -0.003793448908254504,
+            -9.948806837201118e-05,
+            0.0013740125577896833,
+            0.012439027428627014,
+            -0.0017632056260481477,
+            0.024665940552949905,
+            -0.02654467150568962,
+            0.007598260883241892,
+            -0.0054051438346505165,
+            0.005446809343993664,
+            0.018226735293865204,
+            -0.02366596832871437,
+            -0.013082947582006454,
+            0.006276329979300499,
+            -0.01966608688235283,
+            -0.008060368709266186,
+            0.0054051438346505165,
+            -0.00016405766655225307,
+            -0.019923655316233635,
+            -0.012923861853778362,
+            -0.034786853939294815,
+            -0.02309022843837738,
+            0.024665940552949905,
+            0.008401267230510712,
+            -0.0073672072030603886,
+            0.018060073256492615,
+            -0.018438849598169327,
+            0.0172722190618515,
+            -0.002225313102826476,
+            -0.006613441277295351,
+            0.003935490269213915,
+            0.007674016058444977,
+            0.025362888351082802,
+            0.002653330797329545,
+            0.008264907635748386,
+            -0.0042082094587385654,
+            -0.014931377954781055,
+            -0.006321783177554607,
+            0.01980244740843773,
+            0.02471139281988144,
+            0.018605511635541916,
+            -0.018575210124254227,
+            0.010742107406258583,
+            -0.011749654076993465,
+            0.01818128302693367,
+            -0.006537686102092266,
+            0.00039037675014697015,
+            0.01859036087989807,
+            0.007969462312757969,
+            0.025241680443286896,
+            0.019923655316233635,
+            0.006852070800960064,
+            0.014128371141850948,
+            0.011908739805221558,
+            0.001204509986564517,
+            -0.012832955457270145,
+            -0.01154511421918869,
+            -0.02840825356543064,
+            0.042847223579883575,
+            0.0025434857234358788,
+            0.008317936211824417,
+            -0.027635548263788223,
+            -0.00021389048197306693,
+            -0.0034942151978611946,
+            -0.018832778558135033,
+            0.005939218681305647,
+            0.03260510042309761,
+            0.03284751623868942,
+            0.021438762545585632,
+            -2.888172457460314e-05,
+            0.0012234487803652883,
+            0.0329081192612648,
+            -0.013673839159309864,
+            -0.00937851145863533,
+            -0.01034060399979353,
+            -0.018393397331237793,
+            -0.006852070800960064,
+            -0.0032423287630081177,
+            -0.01031030248850584,
+            -0.0061172437854111195,
+            -0.009272454306483269,
+            0.0238629337400198,
+            0.0001374249259242788,
+            -0.0022594030015170574,
+            0.006605865899473429,
+            -0.020878173410892487,
+            0.011537538841366768,
+            -0.02813553437590599,
+            -0.013431422412395477,
+            -0.008651260286569595,
+            -0.006950552575290203,
+            -0.018802475184202194,
+            0.008643684908747673,
+            -0.006942977197468281,
+            0.021438762545585632,
+            0.002914686920121312,
+            -0.023590214550495148,
+            0.007530081085860729,
+            0.015317730605602264,
+            -0.0003825644962489605,
+            0.014037464745342731,
+            -0.003890036838129163,
+            0.009060339070856571,
+            0.0045112306252121925,
+            0.011681473813951015,
+            0.016954045742750168,
+            0.010651201009750366,
+            0.009030036628246307,
+            0.003437398700043559,
+            -0.001167579204775393,
+            0.01878732442855835,
+            -0.0007163614500313997,
+            -0.03405960276722908,
+            2.9236827685963362e-05,
+            -0.014628356322646141,
+            0.029499130323529243,
+            -0.009795165620744228,
+            0.009514871053397655,
+            0.008105821907520294,
+            0.007098275702446699,
+            -0.0067119235172867775,
+            0.007916433736681938,
+            -0.008742166683077812,
+            0.0052763596177101135,
+            0.015832867473363876,
+            -0.0042877523228526115,
+            -0.0014980619307607412,
+            0.007840678095817566,
+            0.00407184986397624,
+            -0.005454384721815586,
+            -0.019711541011929512,
+            -0.012287517078220844,
+            0.00726114958524704,
+            -0.02065090648829937,
+            -0.004367295652627945,
+            0.01161329448223114,
+            -0.005454384721815586,
+            -0.009620928205549717,
+            0.01393140759319067,
+            0.02136300690472126,
+            -0.002382505452260375,
+            0.022893264889717102,
+            -0.01349202636629343,
+            -0.009833043441176414,
+            -0.007984613068401814,
+            -0.004374871030449867,
+            -0.041120000183582306,
+            0.025984082370996475,
+            0.01068150345236063,
+            -0.006871009711176157,
+            0.01895398646593094,
+            0.0022499337792396545,
+            -0.002954458352178335,
+            -0.0011041341349482536,
+            0.00010499217023607343,
+            -0.03281721472740173,
+            0.03660498186945915,
+            -0.007007369305938482,
+            0.0005449649761430919,
+            0.010272424668073654,
+            0.042029064148664474,
+            -0.004590773954987526,
+            -0.005859675817191601,
+            0.01225721463561058,
+            0.005336964037269354,
+            0.031938452273607254,
+            -0.010416359640657902,
+            -0.014340486377477646,
+            -0.01815097965300083,
+            -0.004833190701901913,
+            -0.01939336769282818,
+            -0.0038313264958560467,
+            0.014037464745342731,
+            -0.019650936126708984,
+            -0.011242092587053776,
+            -0.001287840772420168,
+            -0.012537509202957153,
+            -0.0010747789638116956,
+            0.008598231710493565,
+            0.017832808196544647,
+            -0.011348150670528412,
+            -0.004450626205652952,
+            -2.2904154320713133e-05,
+            -0.005003640428185463,
+            0.0185146052390337,
+            0.0020056227222085,
+            -0.01702980138361454,
+            0.01321930717676878,
+            -0.00526499655097723,
+            0.006749800872057676,
+            0.0074429623782634735,
+            0.007939159870147705,
+            0.05145302787423134,
+            0.010317877866327763,
+            -0.001339922659099102,
+            -0.01607528328895569,
+            0.023196285590529442,
+            -0.01749948412179947,
+            -0.014181400649249554,
+            -0.02119634486734867,
+            -0.0029923359397798777,
+            -0.007060397882014513,
+            -0.028014326468110085,
+            -9.137987944995984e-05,
+            0.0016221113037317991,
+            -0.0024980325251817703,
+            -0.010045158676803112,
+            0.0005208179936744273,
+            -0.011529963463544846,
+            -0.006954340264201164,
+            -0.008545203134417534,
+            -0.016044981777668,
+            0.02240842953324318,
+            -0.021817538887262344,
+            -0.008855799213051796,
+            0.011514811776578426,
+            0.03118089959025383,
+            -0.005011215806007385,
+            -0.014173824340105057,
+            -0.019817598164081573,
+            0.0025643182452768087,
+            -0.01763584464788437,
+            0.012181458994746208,
+            -0.01225721463561058,
+            0.015045011416077614,
+            -0.03142331540584564,
+            -0.006079366430640221,
+            -0.008992158807814121,
+            0.012840530835092068,
+            0.019817598164081573,
+            -0.0025093958247452974,
+            0.00606800289824605,
+            -0.010295150801539421,
+            0.030211230739951134,
+            -0.00012286569108255208,
+            -0.018605511635541916,
+            0.00873459130525589,
+            0.007977037690579891,
+            0.012537509202957153,
+            -0.004435475450009108,
+            0.00257757562212646,
+            0.011567841283977032,
+            0.012742049060761929,
+            -0.01634800247848034,
+            -0.003174148965626955,
+            -0.017969166859984398,
+            0.005996035411953926,
+            0.006999793462455273,
+            0.021711481735110283,
+            -0.0013550736475735903,
+            0.018272189423441887,
+            -0.01563590206205845,
+            0.00019412307301536202,
+            -0.011749654076993465,
+            -0.0082270298153162,
+            -0.0028237802907824516,
+            -0.003958216868340969,
+            0.006685409229248762,
+            -0.028211290016770363,
+            -0.01274962443858385,
+            0.005287723150104284,
+            -0.004992276895791292,
+            -0.03405960276722908,
+            0.0053710537031292915,
+            -0.02725677192211151,
+            0.013128400780260563,
+            -0.002397656673565507,
+            -0.017847958952188492,
+            -0.00030846628942526877,
+            0.026332557201385498,
+            -0.003677921835333109,
+            -0.012537509202957153,
+            0.015544996596872807,
+            -0.006757376715540886,
+            -0.02119634486734867,
+            -0.005420294590294361,
+            0.014658658765256405,
+            -0.0010719380807131529,
+            -0.012393574230372906,
+            -0.029635490849614143,
+            0.012992041185498238,
+            0.0026874206960201263,
+            -0.007386146113276482,
+            0.010893618687987328,
+            0.025953780859708786,
+            0.008151275105774403,
+            -0.005924067925661802,
+            0.0015757111832499504,
+            0.0049581872299313545,
+            0.00300938100554049,
+            0.020287280902266502,
+            0.02728707529604435,
+            -0.008014915511012077,
+            -0.0011145505122840405,
+            -0.014954105019569397,
+            0.015802564099431038,
+            0.0013020449550822377,
+            0.010507266037166119,
+            0.021893294528126717,
+            0.007045246660709381,
+            -0.005738467443734407,
+            0.003905187826603651,
+            -0.023135682567954063,
+            0.01420412678271532,
+            -0.012272365391254425,
+            0.000622140767518431,
+            0.008749742060899734,
+            -0.007283876184374094,
+            0.01437078882008791,
+            -0.0015245762187987566,
+            0.0339989997446537,
+            0.025211378931999207,
+            0.0043218424543738365,
+            0.0007362472242675722,
+            -0.028559764847159386,
+            0.003547244006767869,
+            0.008651260286569595,
+            0.014499573037028313,
+            -0.0076664406806230545,
+            0.006261179223656654,
+            -0.004355932120233774,
+            0.00959820207208395,
+            -0.0065490491688251495,
+            -0.015893470495939255,
+            0.006113456096500158,
+            0.0012689019786193967,
+            0.006795254070311785,
+            0.0023408401757478714,
+            -0.008863375522196293,
+            -0.024362917989492416,
+            -0.025984082370996475,
+            0.021984200924634933,
+            -0.03918066248297691,
+            0.0073293293826282024,
+            0.011234517209231853,
+            0.03324144333600998,
+            -0.008158850483596325,
+            -0.004980913829058409,
+            0.006276329979300499,
+            -0.016711628064513206,
+            -0.004147605039179325,
+            0.008204303681850433,
+            0.012385998852550983,
+            -0.002420383272692561,
+            0.00974971242249012,
+            0.01318900566548109,
+            -0.012378423474729061,
+            0.03660498186945915,
+            0.016514664515852928,
+            0.01467380952090025,
+            -0.004007457755506039,
+            0.01312082540243864,
+            0.0066172294318675995,
+            0.019044892862439156,
+            0.025165924802422523,
+            -0.017454031854867935,
+            -0.002365460619330406,
+            -0.02292356640100479,
+            0.024165954440832138,
+            0.00726114958524704,
+            0.004556683823466301,
+            -0.008310360834002495,
+            0.014870774000883102,
+            0.0017954016802832484,
+            0.0028199926018714905,
+            -0.006696772295981646,
+            -0.010173942893743515,
+            0.019241856411099434,
+            0.0034317169338464737,
+            0.014870774000883102,
+            4.8501173296244815e-05,
+            -0.005503625608980656,
+            0.006052851676940918,
+            0.014832896180450916,
+            -0.023590214550495148,
+            -0.00203592493198812,
+            -0.019484274089336395,
+            -0.03202936053276062,
+            0.009022461250424385,
+            0.0016135887708514929,
+            -0.007052822504192591,
+            0.03230207785964012,
+            -0.010083036497235298,
+            -0.019787294790148735,
+            -0.008067944087088108,
+            -0.013113250024616718,
+            -0.010469388216733932,
+            0.0008271536789834499,
+            0.003908975515514612,
+            -0.00012428610352799296,
+            -0.009317907504737377,
+            -0.005749830510467291,
+            0.01990850456058979,
+            -0.0030358952935785055,
+            -0.00753386877477169,
+            0.00868156272917986
+          ],
+          "embedding": [
+            0.0040707582550429606,
+            -0.010230625151989194,
+            -0.021520242101903186,
+            -0.010115756270640508,
+            0.026738734586159035,
+            0.01796370233223267,
+            -0.012245199029994153,
+            0.06506924494147726,
+            -0.0014087587222033442,
+            0.009300068719271429,
+            0.003787396973047584,
+            0.002818772639754174,
+            0.022129077062176306,
+            -0.021526709808445683,
+            -0.0034388135367225824,
+            0.02763761747879405,
+            0.013210013805395276,
+            0.025255872672296602,
+            0.00029011480909986703,
+            -0.05438974762290585,
+            0.016671640187490865,
+            -0.016426252259893328,
+            -0.056591180651012554,
+            0.009335648425523075,
+            0.006141624459418827,
+            0.02568943956694641,
+            0.019377132054042116,
+            0.017946464337078746,
+            -0.057184776985927686,
+            0.03208527139096687,
+            0.030132392063755547,
+            -0.003846032371791839,
+            -0.008376524427531415,
+            -0.0010513963058713435,
+            -0.02734868388724371,
+            -0.014879400370545479,
+            0.04383920959748825,
+            0.02563613659516289,
+            -0.020117467074438997,
+            0.0398016699155716,
+            0.029896497884197775,
+            -0.0004195844530067313,
+            -0.023302720788253625,
+            -0.001168337916577517,
+            0.009149021369661628,
+            -0.02508145721900992,
+            -0.004307594138480351,
+            0.00550940415564303,
+            0.014016998918718476,
+            -0.028607379616884317,
+            -0.0018317656587526954,
+            -0.01973464940415756,
+            0.03762738392056255,
+            0.015358106058583114,
+            -0.03931907682104936,
+            0.011519637298832444,
+            0.004482080558226775,
+            0.0747867063383242,
+            0.05550679018263077,
+            0.023243129601931176,
+            0.02671631686665238,
+            -0.009686146566573536,
+            -0.04088582189725495,
+            -0.0065344623594924085,
+            0.019617295999325712,
+            -0.004722087402575777,
+            -0.002871844114437583,
+            0.003935087265300519,
+            0.03229890413393201,
+            -0.02813577047904649,
+            -0.05124172132597234,
+            0.03325235407841968,
+            -0.008687655557217192,
+            -0.011027174838246948,
+            -0.003653553063547375,
+            -0.02350737945182988,
+            0.016223087939183468,
+            0.03789252356462714,
+            0.0034350623378603083,
+            0.00195158657888739,
+            0.011789712862232108,
+            -0.0004435439660769551,
+            -0.0037330456918125894,
+            0.0031552713243866858,
+            0.04465432992298031,
+            -0.0032465421571915712,
+            -0.0075366743861922745,
+            -0.059958752731410256,
+            0.016784616861008125,
+            -0.028734506065772967,
+            -0.03035575736273943,
+            0.02680390892611741,
+            0.02338368719038086,
+            -0.0030658375437784277,
+            -0.03754896050794851,
+            0.07005659523248504,
+            0.0020711575553570695,
+            0.013034425083076923,
+            -0.02316832262132332,
+            0.021332142030732894,
+            -0.0318756233020585,
+            0.02270782509507388,
+            0.01570810309454529,
+            0.004788124787612736,
+            0.026727250200142456,
+            -0.022380142778058276,
+            -0.008593910379436904,
+            0.014439754382387034,
+            0.015007697617383174,
+            0.07662531963612977,
+            0.011195998137723795,
+            -0.010809283538470098,
+            -0.042227822120219305,
+            0.003824320933275175,
+            -0.03797629427607094,
+            -0.02347390684453348,
+            -0.0526876440280276,
+            -0.017799861680287913,
+            0.01701482796266917,
+            0.002730395144917155,
+            0.03789451759899825,
+            0.038338545350313946,
+            0.028463789278406586,
+            0.015494583705011921,
+            -0.05700237266350479,
+            0.047895170359075166,
+            -0.0064406894534929406,
+            -0.0007954721295065988,
+            0.012543029839729358,
+            -0.004703334868741216,
+            -0.0060569679284068,
+            0.012324901254711192,
+            -0.019883935809784068,
+            -0.04253565942154215,
+            0.02164409276211115,
+            -0.0075367555415808635,
+            0.03617159147722157,
+            0.07567745988686005,
+            -0.04225977020036783,
+            -0.022548837589622003,
+            0.029811602145850144,
+            -0.02175792196819533,
+            0.02069861458115363,
+            -0.00941331829664397,
+            -0.035479639885943264,
+            0.020352552580119105,
+            0.003678588588550461,
+            0.06776105274336729,
+            -0.02346047721661687,
+            0.02721075076349443,
+            -0.029386781707644253,
+            0.023530831921117993,
+            -0.02077660533100372,
+            0.020022475323372906,
+            0.024195118676513715,
+            -0.028378518720494666,
+            -0.012900969023913307,
+            0.019264089970719624,
+            -0.0374983624629265,
+            -0.019726171557384355,
+            0.00979636085642416,
+            -0.014479528435832063,
+            -0.014533384583777662,
+            0.005904623758767224,
+            0.010344475500263744,
+            0.05659646358026115,
+            0.04163866113482093,
+            0.007152494819169405,
+            -0.07584500130974763,
+            0.05471858597878099,
+            0.03580211013427615,
+            0.044161429444475306,
+            0.02258180566046248,
+            0.00980779221334715,
+            0.0011193505603397395,
+            0.018086566768892507,
+            0.015605066702191871,
+            -0.00954727399396313,
+            -0.027950079146919375,
+            0.03028514025485407,
+            -0.006351669761257431,
+            0.005095804801686436,
+            0.009620040085743162,
+            0.019748001227987134,
+            0.033709535059510916,
+            0.008203559956940196,
+            0.007389752602218563,
+            0.04197771878351742,
+            -0.02067363758471765,
+            0.007543739188471913,
+            0.011207941261799052,
+            0.013964827251172813,
+            0.03839562075248879,
+            0.0009775244543292637,
+            0.006469507611492216,
+            -0.022808235395746654,
+            -0.024795004398633292,
+            0.006249024864814684,
+            0.011292877109066925,
+            0.01596839619826819,
+            -0.0014105396434174883,
+            -0.04152075438277094,
+            0.008267788411079337,
+            -0.028834191212891387,
+            0.030940526405027906,
+            -0.0018271056434014425,
+            0.006185475047820603,
+            -0.007920574674727123,
+            0.012907560753043616,
+            -0.015165704767187252,
+            -0.021962244496450727,
+            0.0037071053072833147,
+            -0.013508376064336042,
+            -0.015178823540513864,
+            -0.02518557053061311,
+            0.011693256916760944,
+            -0.01716015674592331,
+            0.02048897513447166,
+            -0.003412078966361745,
+            -0.001807298511378529,
+            -0.0012575655363779904,
+            0.03116103961636643,
+            0.011668533361641842,
+            -0.011637010548030312,
+            -0.040260195687925236,
+            -0.011129376691522074,
+            -0.0034493180842871908,
+            -0.02098244789711689,
+            0.026658743116016938,
+            -0.01923736325953796,
+            -0.009723937181300192,
+            -0.013134679989565051,
+            -0.011974548914818263,
+            -0.004028712301052159,
+            0.019628329175867303,
+            -0.028192034966932684,
+            -0.012638520793012082,
+            -0.011568652932013111,
+            -0.0025919910519445737,
+            0.012497908463405111,
+            0.015113186712897145,
+            0.014880607940212641,
+            -0.03059227859452334,
+            -0.021899716217105084,
+            -0.0008513506358489523,
+            0.011985931118464679,
+            -0.01321584595585435,
+            -0.011735288366213981,
+            0.010710683541435907,
+            0.01961722172525993,
+            -0.023571946726192585,
+            -0.03364281880125354,
+            0.0014460076743756465,
+            0.016563167828467187,
+            -0.02891589059260944,
+            0.04595166331401113,
+            0.034271553243501465,
+            -0.027557971577291153,
+            -0.007138821520412525,
+            -0.039508407807651844,
+            0.06814329636830853,
+            0.00022136123564675344,
+            -0.039718016705753735,
+            0.0503457594963753,
+            -0.003580581891576583,
+            0.013820796398521455,
+            0.02054050419088757,
+            0.035682661890720495,
+            -0.0013220739026852943,
+            -0.0052759394697328675,
+            0.04733018719162478,
+            -0.0187845855131196,
+            -0.03982699468608443,
+            -0.023905460465070062,
+            0.002631233133039351,
+            -0.03090995554989513,
+            -0.012617612051488022,
+            -0.013605787969473513,
+            0.039672783793764745,
+            0.008669582242097126,
+            0.009974277556736362,
+            0.005535636914843583,
+            0.014180668493093584,
+            0.012042659947850441,
+            0.011870115226306776,
+            0.00451259454348581,
+            -0.014023048592193627,
+            -0.007769033996392623,
+            -0.05715987527668358,
+            0.003374564518384728,
+            -0.028801158523506985,
+            -0.00756331369170698,
+            -0.02163054894055572,
+            0.005367181573233089,
+            0.02308705505078254,
+            -0.005438909398653553,
+            -0.00873620909426313,
+            -0.0036530652237719077,
+            0.013920407174296543,
+            -0.01539613379172909,
+            -0.03908211213269472,
+            -0.016810726548844992,
+            0.007460173854386443,
+            -0.007069189426742819,
+            -0.041197222498539215,
+            -0.003843233633335066,
+            0.004373367480223065,
+            0.016194121660459908,
+            -0.029165150066068787,
+            -0.00735139702658342,
+            0.017257895082337448,
+            0.014153313280536095,
+            -0.002376178042741926,
+            -0.023223700836427403,
+            -0.0319877537205969,
+            0.02088060774052704,
+            0.0012829290529515294,
+            0.03251550040316258,
+            -0.04128453069131847,
+            -0.023376114412063578,
+            0.04083022735076889,
+            0.005690491480259003,
+            -0.05810403800689653,
+            0.016582434816685452,
+            0.010402038964417569,
+            0.016795644054313566,
+            -0.00908590043098552,
+            0.05003656634510042,
+            -0.037216197954986735,
+            -0.013027960261454162,
+            -0.007090943677204207,
+            -0.05175371787508879,
+            -0.0020065660448063322,
+            -0.014908489552053589,
+            0.010627472758306387,
+            -0.03275473161158777,
+            -0.017334608434133866,
+            0.025321047439048318,
+            -0.0033515195870701753,
+            0.011526068468647207,
+            -0.026783879816504975,
+            -0.0012830935419413407,
+            -0.017506908439998053,
+            -0.02114853058440477,
+            0.004878884893669383,
+            -0.020936900837115968,
+            -0.0025587027781733642,
+            0.01557850246191957,
+            0.023652335794299834,
+            0.011489428313303988,
+            0.0019089137782552768,
+            0.02054907941022243,
+            0.014361232983780595,
+            -0.0056413109308506085,
+            -0.0010300050480036684,
+            -0.03413262505887213,
+            -0.018821160485010483,
+            0.03361989907065425,
+            0.013125889701529858,
+            -0.003742167496379789,
+            0.008813100942466033,
+            -0.0383265293638935,
+            -0.035316957263713485,
+            0.03020499323039515,
+            -0.05001623971243151,
+            -0.00855479667673597,
+            -0.005939674940109814,
+            -0.026451831943555135,
+            -0.008032873049527808,
+            0.0009431571685654594,
+            -0.014387605128858954,
+            0.016382559541755014,
+            -0.024909510232038502,
+            0.003715736505444385,
+            -0.006016712902409688,
+            -0.009110197692342108,
+            0.05507263160354303,
+            -0.022103031032598353,
+            -0.0475415595313825,
+            0.021958434235402655,
+            -0.016670809659931243,
+            0.0323519363801999,
+            0.03867494951405658,
+            0.011124508964363016,
+            -0.008002445714431074,
+            0.034041838454818577,
+            0.018400123235267234,
+            -0.01161754789471199,
+            -0.025901844576206774,
+            0.021800298132870245,
+            0.00943488425697727,
+            -0.05906691123008531,
+            -0.00782549274387971,
+            -0.019357446956797447,
+            0.008682376344965976,
+            0.02863077694455766,
+            0.0024925977373894447,
+            -0.002682170968419412,
+            -0.013590235283325244,
+            -0.021511149507762284,
+            -0.03255453916518719,
+            0.009039307015853905,
+            0.008305286476500875,
+            0.009312625027668458,
+            -0.0014144165766983945,
+            -0.007853869334376857,
+            0.009067065760942163,
+            0.022485201611182105,
+            0.03966189080765722,
+            -0.041627289519629344,
+            0.002707086456239785,
+            -0.022678178050044222,
+            -0.03653753142622053,
+            0.0036574646778835588,
+            0.029996160786576367,
+            0.02901163765559891,
+            -0.01971777193183787,
+            -0.0036589934841109884,
+            0.009181234938554578,
+            -0.025979483002227535,
+            -0.00773645103995767,
+            0.004280770859737866,
+            0.04116668301017644,
+            0.006201085381597264,
+            0.01402728258112097,
+            0.001060407317110527,
+            -0.00920801784000087,
+            0.02109880889040528,
+            0.025211022083982727,
+            -0.029866678264942617,
+            0.014439702646820226,
+            0.009379206296908577,
+            0.04156435812161122,
+            -0.037309655469525846,
+            -0.011819731348299495,
+            -0.02766498020409985,
+            -0.010454143230794476,
+            0.027911532296566614,
+            -0.04127890742215716,
+            -0.006948536183086887,
+            0.010207784113909938,
+            -0.0016584043356136647,
+            0.016437495428919636,
+            -0.014607861750686952,
+            -0.03222311919805017,
+            -0.000610723203118213,
+            -0.02867445198803213,
+            -0.024772816792038133,
+            -0.016510309753766612,
+            -0.018581201775667577,
+            -0.07450654873867177,
+            0.00026832248829154255,
+            -0.06724696990816072,
+            -0.02447002541264633,
+            0.04885569249258513,
+            -0.010195604207134053,
+            0.03332905935704749,
+            -0.02815272112271314,
+            -0.005296340304911926,
+            -0.05908835042526019,
+            0.0038476172058970774,
+            0.025753576863447866,
+            0.02386003857017594,
+            0.005206447371391696,
+            0.024799057871126032,
+            0.012060738268265606,
+            -0.04146503002980051,
+            -0.024949907556862643,
+            -0.014503475223770958,
+            -0.0016340466065744286,
+            -0.0034009167013795906,
+            -0.031298432995411005,
+            0.036241272149880994,
+            0.008902644391348814,
+            0.03550223922667804,
+            0.04463301409716232,
+            -0.010385984413759852,
+            -0.03305004479734889,
+            -0.006952739338820577,
+            -0.0175981588092108,
+            0.042668533202572856,
+            -0.017112184620086308,
+            0.005149173014526686,
+            -0.03219589107997635,
+            0.024181808150698054,
+            0.0014339790422945406,
+            0.01254721190611376,
+            0.011099458649404119,
+            0.0027626836767364253,
+            -0.026883640446090158,
+            0.0024160770766463607,
+            0.023874514847956222,
+            -0.022558433617669373,
+            0.00917667715473093,
+            -0.006754091166465801,
+            -0.033686935718776144,
+            -0.01078389638118685,
+            -0.013595327486167114,
+            0.024712781935748277,
+            0.013168464807451798,
+            0.005875864604862434,
+            -0.02417814420558463,
+            -0.0023978800459714144,
+            0.028650463366555304,
+            0.020176980768301288,
+            -0.02118184201861616,
+            0.019082340279671355,
+            0.019734925740100277,
+            -0.04209645071102001,
+            -0.04217310001316092,
+            0.006553913719098956,
+            -0.003735719227468833,
+            0.0061793909220459,
+            -0.028285889794908054,
+            0.008181788076484304,
+            0.004432488437404309,
+            -0.01743905248466261,
+            0.041221269621866216,
+            -0.014892354419172107,
+            0.005087102208335201,
+            -0.022198383716186407,
+            -0.011936388003605221,
+            -0.00027487042024290816,
+            0.044599215801158845,
+            0.008068975171827777,
+            0.003496950921841073,
+            0.02238779276447619,
+            0.0054839808262270315,
+            0.04816723616939378,
+            -0.02685221260466647,
+            -0.0190356946337995,
+            0.016464136459560113,
+            -0.0176092780782677,
+            -0.012513274095432353,
+            -0.010695421445334002,
+            -0.004784348908457072,
+            -0.022203041334932165,
+            0.010788155380415583,
+            -0.018353897272137543,
+            -0.02277431189907027,
+            -0.003945465850796918,
+            0.016024639792926797,
+            -0.006248368166794934,
+            -0.0038769803892794875,
+            0.0016292556540434277,
+            0.02412864830338344,
+            0.02892955453819876,
+            -0.02408022470555968,
+            0.019190282372105023,
+            0.018941584034503536,
+            -0.006709516380602965,
+            -0.020956377933747532,
+            -0.01964885206488053,
+            0.01434623424964373,
+            -0.01352968586212152,
+            -0.013872556892707697,
+            0.004052798542613835,
+            0.009201147374299997,
+            -0.004086067985859617,
+            -0.04185829685685201,
+            -0.014668698123513235,
+            0.003179403563664022,
+            -0.0006415014901450319,
+            0.019360184344240947,
+            -0.0150817538567364,
+            -0.029771999278536083,
+            0.014221776097046751,
+            0.022944884879589678,
+            -0.04493425220030206,
+            0.02117577719533236,
+            -0.0030648751497261983,
+            -0.00016251533643103604,
+            -3.817296018601414e-05,
+            0.008727767382077138,
+            0.026914779443266063,
+            0.03456565094655899,
+            0.01974806772696219,
+            0.00450800007152441,
+            -0.008952599700759673,
+            0.025184190461155245,
+            0.009846044083064704,
+            -0.025168735183134738,
+            0.010983353565382029,
+            0.016740721743326522,
+            0.0070838930012654296,
+            0.009173879407277282,
+            -0.006078288608766152,
+            -0.018269347893097643,
+            0.013181494188066667,
+            -0.01435693747022231,
+            0.01678907949532146,
+            0.05154463552784974,
+            -0.00039632596424461964,
+            -0.02461439550772063,
+            0.008418147037164186,
+            -0.006544319842335593,
+            -0.010065129286193278,
+            0.02214628550460128,
+            0.0029515574751370417,
+            -0.030862152033847717,
+            0.007509556025223583,
+            0.00675485304374519,
+            0.01894936231559203,
+            -0.02355525295639599,
+            -0.01387885858544803,
+            -0.00431398020087544,
+            0.0021208056468763523,
+            0.01580766504149989,
+            -0.004739477121073512,
+            -0.013204823447960706,
+            0.02994098229517376,
+            0.025496728234662373,
+            -0.00026124946738213036,
+            0.015060064980768175,
+            0.022262345898533903,
+            0.009968408664790368,
+            0.007588431141088797,
+            -0.0083063413609779,
+            -0.031336272171937095,
+            0.0018957858383561667,
+            0.05800811417728381,
+            -0.014937085296865802,
+            0.04313875432235027,
+            0.02018476425923236,
+            0.015657644644987107,
+            0.017880263053253607,
+            -0.01662268082870562,
+            -0.008740053682875323,
+            -0.0017630394204179638,
+            -0.008294417990024647,
+            -0.027613464270562948,
+            -0.005953209556351086,
+            -0.02379961210881074,
+            -0.013878676554701543,
+            0.015418315245979704,
+            -0.014764728065382074,
+            0.0027420640870858457,
+            0.010395538988699663,
+            -0.006590633833835581,
+            -0.00841537209927409,
+            -0.003943772700797381,
+            0.03376338017637296,
+            0.011980354721458463,
+            0.027761929822019422,
+            -0.00011128539412582973,
+            0.009342430952107655,
+            -0.018419526564396063,
+            -0.006839752805318664,
+            0.03271190887864607,
+            0.006011520427944665,
+            0.004652835471939905,
+            0.01485920920301398,
+            0.014305626381398871,
+            0.038015677219487023,
+            -0.005102005401565164,
+            -0.018204460895989052,
+            0.02065495133597456,
+            -0.0049699013356374775,
+            0.019149036693562712,
+            0.008825981577388056,
+            -0.002660055848523568,
+            -0.004292379561953203,
+            0.03090270710860331,
+            -0.020243440437217503,
+            0.018735727616321387,
+            -0.02238236089688497,
+            -0.0276584413929402,
+            0.023028408906176946,
+            0.008320436733620496,
+            0.04047641118026783,
+            0.011463851066078041,
+            -0.02247143742166039,
+            -0.034588725413285185,
+            0.009827713930789184,
+            0.00337071450636392,
+            -0.024446280199879062,
+            0.017542777160283324,
+            0.00780836483490212,
+            0.008453886667283516,
+            0.016624116261219637,
+            -0.004175682824655864,
+            -0.0042784362933758835,
+            -0.001546656717896957,
+            -0.008819739495525477,
+            0.01406790007187259,
+            -0.015487658289541583,
+            -0.021457182907140035,
+            0.024687169583245905,
+            -0.03915911962859764,
+            0.01771290139419629,
+            -0.006510046907173026,
+            -0.00997450880725535,
+            -0.02059560682587115,
+            1.5939836949198004e-05,
+            -0.007159493752862143,
+            -0.0004782363814453761,
+            -0.017503181957042913,
+            0.016643971600145645,
+            0.024056442566869802,
+            -0.013209211443909658,
+            -0.00445521990927108,
+            0.0020313626821877526,
+            -0.029235692455820056,
+            0.006662209097255912,
+            0.016993027053110635,
+            0.00529940612845993,
+            0.0026104743851393716,
+            0.0001707934714983891,
+            -0.031289446611163674,
+            -0.005782208620284939,
+            0.03053516889069683,
+            0.05590704747438521,
+            0.021628254147289437,
+            0.02403314981042378,
+            0.02908606663381421,
+            0.02717206823892633,
+            -0.003198844932040613,
+            0.0016217508103442348,
+            -0.01620660606431983,
+            0.027098323143010743,
+            -0.011664168290676943,
+            0.04322175021589083,
+            0.008652323184522541,
+            -0.049493020736445056,
+            0.004537958157696794,
+            0.00943239454147692,
+            0.034918603888916544,
+            0.016678716235690318,
+            0.0005887414054803805,
+            -0.005894338664998101,
+            0.03279165663336323,
+            -0.02735726709032382,
+            -0.011962070182991011,
+            -0.00036014407560325284,
+            -0.019079003730607717,
+            -0.015938295169860884,
+            0.010357611943935459,
+            0.04278924678845588,
+            0.0009957483598272451,
+            -0.0071422790468850215,
+            0.01571201177743999,
+            -0.009445307674383,
+            0.02286338492075306,
+            0.007336440384050498,
+            0.0042809497342459904,
+            0.025581467690095977,
+            -0.0025007871391868105,
+            0.02300539823997411,
+            -0.000628898771317254,
+            0.004582863909134464,
+            -0.015960578501883316,
+            -0.00930649396114688,
+            0.027838455754803444,
+            -0.011112268986721,
+            -0.029253746504396393,
+            -0.008734785561831333,
+            0.00472579547270089,
+            -0.0003805825712986138,
+            0.0021854358149324227,
+            -0.01183885533661493,
+            -0.010840951201591629,
+            0.019965691788904057,
+            -0.004922943301424112,
+            0.003800048754664761,
+            0.012664049429916252,
+            -0.0030124219674414897,
+            -0.01910844996750822,
+            0.01889983984736845,
+            -0.003858283778519141,
+            -0.024168821338992813,
+            0.020083955711766945,
+            0.01850497889545421,
+            0.045156567155119454,
+            -0.01586824055785609,
+            0.004612757419198795,
+            0.017497774574602505,
+            -0.002172088240279357,
+            -0.019617327081566827,
+            -0.014944795772093997,
+            -0.018212462639041492,
+            -0.015975283672064702,
+            0.012837956594691596,
+            -0.01243789877725244,
+            0.019500571353639314,
+            -0.0380735226388175,
+            0.012590900534826689,
+            0.004832168601136412,
+            0.007505385730121269,
+            0.016011983211212992,
+            0.025994560026860358,
+            0.002157436199771672,
+            -0.033465476168065784,
+            -0.01014599891204166,
+            0.007845373376916211,
+            0.010093432683262838,
+            -0.002849185178237036,
+            -0.018147004183434554,
+            0.003633790611809425,
+            -0.0266147420637404,
+            -0.018644422659309035,
+            0.022926737299761104,
+            -0.04798772015599113,
+            0.017908415301694906,
+            0.024859125158586466,
+            -0.04343699525707025,
+            -0.04886699607556352,
+            -0.011459188432595229,
+            -0.036774381316312464,
+            0.0014128005215813578,
+            -0.0005256067584181517,
+            0.011194992852264102,
+            -0.009342140079829378,
+            -0.018598671981710183,
+            0.01584789414301952,
+            -0.0036412641242200418,
+            -0.004225840239935549,
+            -0.00616696736018045,
+            -0.0025598631463178384,
+            -0.010188803157718543,
+            -0.03244694662411977,
+            -0.03290891136780909,
+            -0.00033742479296272127,
+            -0.009743927051414455,
+            -0.02334288359945272,
+            -0.03447062782872289,
+            -0.008728968107967028,
+            -0.037753659749578956,
+            -0.00515561961442376,
+            0.04070890150714597,
+            0.027517147608335957,
+            0.009449414978133334,
+            0.008086900089929912,
+            -0.013989034604490105,
+            0.05691047488419298,
+            -0.020425883381389086,
+            0.039466620642512155,
+            0.005301493569940605,
+            -0.0037222353899918703,
+            0.013075425214878277,
+            0.027710826422351564,
+            -0.006253665106995653,
+            -0.0017412928073238833,
+            -0.01928096630475438,
+            -0.012920476859996416,
+            0.009444408011467955,
+            0.015799884767184885,
+            -0.0025083061619197524,
+            0.001274457251739847,
+            0.00930533161700852,
+            -0.0006652478540264343,
+            0.017258038491335247,
+            0.03996268817486439,
+            0.003833844612473671,
+            -0.00826404141706295,
+            -0.018822002570963614,
+            -0.003188844900317347,
+            -0.011807006674248829,
+            0.010482058734260632,
+            -0.015410635475145028,
+            -0.011540803927522013,
+            -0.01249428777269578,
+            -0.017081856786754605,
+            -0.002183034976477535,
+            0.003114834030705039,
+            0.0027475411722881747,
+            0.02261627635192778,
+            -0.015872027027080824,
+            -0.022487581027490516,
+            -0.0035823117885461506,
+            -0.02398778449664045,
+            8.13716819067491e-05,
+            -0.01050060782407259,
+            -0.015616181283381218,
+            0.013676480044963827,
+            -0.021513173575512936,
+            -0.01091399089076612,
+            -0.008699176177861256,
+            -0.0006986953850759474,
+            0.005853723710671249,
+            -0.00993355748621621,
+            -0.00010241843582159751,
+            0.01150872694850241,
+            0.005559552266827916,
+            -0.012336449703762907,
+            0.0039431718621504,
+            -0.012740550272925562,
+            -0.004432059174951158,
+            -0.03592309122711106,
+            -0.019665686718735042,
+            -0.01846795637714379,
+            0.005642094448288269,
+            -0.005108964028543505,
+            -0.020586678054151854,
+            0.00668364061048941,
+            0.0034775127050453774,
+            0.004713169526661792,
+            -0.010723667875209928,
+            0.013944735241748791,
+            0.010882420168586074,
+            -0.003518245916227596,
+            0.013827334494230662,
+            -0.001330399936559009,
+            -0.020286697504699764,
+            -0.011139922402071095,
+            0.03538258948179278,
+            -0.024961826464903997,
+            -0.0018019328351763667,
+            -0.0026423695690168066,
+            0.00597003516870726,
+            -0.03071835946501977,
+            -0.013634212232744325,
+            0.02164400349828422,
+            -0.005846552894831441,
+            -0.026367014426434226,
+            0.03399116480234033,
+            0.03473258231629753,
+            0.010041605592929843,
+            -0.013423384340589794,
+            0.04730054254696893,
+            -0.012968978954938308,
+            -0.018274800841400665,
+            0.026826644936918647,
+            0.0278389611094493,
+            0.02525164095926168,
+            -0.029782550447890466,
+            -0.003010713086487275,
+            -0.02393629084357885,
+            0.017993366287762866,
+            -0.0263476663180971,
+            0.01864011499179424,
+            -0.02332263152537849,
+            0.016014908881756034,
+            0.01868464825773604,
+            -0.003104041044182249,
+            0.026313491086684425,
+            -0.0052945248919367815,
+            0.015253731232325074,
+            0.0012371308759955573,
+            -0.02119765517013431,
+            0.007795466385020642,
+            -0.006469500272298062,
+            0.025762169721880456,
+            0.00833992768202466,
+            0.01801901639978976,
+            0.022019940825351116,
+            -0.0409522298213741,
+            0.02003698451081499,
+            -0.001367916864679543,
+            0.01782984895971748,
+            0.0038516134262842373,
+            -0.007885432294782304,
+            0.014221728513400065,
+            0.010991352188016265,
+            -0.012851099261924757,
+            0.003840539873231741,
+            0.0008923482291826014,
+            -0.028928104542716937,
+            0.007066004768771198,
+            -0.026914574065060926,
+            0.025866907400807087,
+            -0.005526774784152086,
+            -0.00998122210956923,
+            -0.020744532673097317,
+            -0.0071425207262770865,
+            0.003754468896273156,
+            -0.012747874997429746,
+            -0.011385805601999295,
+            -0.024700457669046855,
+            -0.02401291382882952,
+            0.029591582802488962,
+            -0.014054332096344003,
+            -0.04015978298687865,
+            0.001234440863385445,
+            -0.023301255402129767,
+            0.01193125007773239,
+            -0.005793124129267758,
+            -0.024440100649464773,
+            -0.042563399082257154,
+            -0.017548888269389457,
+            -0.0009394887961485922,
+            0.022739763644266633,
+            -0.00114156949394707,
+            0.041791395064755515,
+            0.0008816696205437911,
+            0.022207132857533028,
+            -0.015547580543606755,
+            0.005319731264840819,
+            -0.02071229859314312,
+            -0.041495579742912285,
+            0.03660432378985971,
+            -0.014637031447181937,
+            0.025854844497435178,
+            0.03324325296704186,
+            -0.013009347371118843,
+            0.001354504885302625,
+            0.02652570709474851,
+            -0.014618100923440427,
+            -0.010782232747863238,
+            -0.0034936699515378212,
+            0.003461670978569965,
+            -0.006468116285021298,
+            -0.027793221545159466,
+            -0.007752458229193391,
+            0.04193706715422074,
+            0.017894069367616637,
+            0.021928994684423043,
+            0.022606401934291136,
+            -0.017972404868085358,
+            -0.026402048654300563,
+            0.003656646576244765,
+            -0.013129722990153284,
+            -0.018090360146368677,
+            -0.01624598856366761,
+            0.012397150789705746,
+            0.011741002980073786,
+            -0.004052422390675261,
+            -0.01836750423615859,
+            -0.020929335029248472,
+            -0.016327412417122416,
+            -0.0042615345226698015,
+            -0.008458117095535,
+            -0.005002582649405199,
+            -0.012158173174049797,
+            -0.020409616620022143,
+            0.03706368540614453,
+            0.0285565642978937,
+            -0.0041732933884846566,
+            -0.00607154864704767,
+            -0.011142150176334957,
+            0.008834431806990914,
+            0.006021232046087652,
+            -0.005028870471297621,
+            -2.3396043292139426e-05,
+            0.005112033372713154,
+            0.022822312435625314,
+            0.007271235299348001,
+            0.0035791789459533856,
+            -0.06340305811365254,
+            -0.005392000517722054,
+            0.022560201096078862,
+            0.012581401789917604,
+            -0.0032248219865223496,
+            0.025180718735801837,
+            0.0006451724964970927,
+            -0.009714514747463593,
+            -0.018389422219956937,
+            0.024728862150150004,
+            0.008760329201960964,
+            0.020988700992999768,
+            -0.014974763313686516,
+            0.003845297486923296,
+            0.0006445983847138896,
+            0.00750394530065325,
+            -0.0019792992847026793,
+            -0.008578254004439607,
+            0.025776677078006516,
+            0.016593438298233645,
+            -0.005695578040595484,
+            0.01604018433477423,
+            -0.0017497260488899657,
+            -0.021460543443428924,
+            -0.010265213095609554,
+            0.00042017209251926397,
+            -0.010570864288876387,
+            -0.018015790888295944,
+            0.017569414447505878,
+            -0.002601036282535863,
+            0.0072399377790164435,
+            0.030281707151249355,
+            0.0423183067569767,
+            0.009566493398534713,
+            -0.005928603293907002,
+            -0.0037936911251953654,
+            0.015477873304924939,
+            0.01513127995244932,
+            0.010741256034652754,
+            0.03015858543221727,
+            -0.029247492875021843,
+            -0.006690306960497428,
+            0.0046653727410352576,
+            0.004610405633429778,
+            -0.011692179099382171,
+            0.030055897497741527,
+            0.04569730592994177,
+            0.006736813244286893,
+            -0.004799978513117371,
+            0.002121067028052434,
+            0.011299889107780231,
+            0.007768137218140627,
+            -0.018927080714961783,
+            -0.0012222962555201746,
+            -0.006009911026251445,
+            -0.014001766533604134,
+            0.026941585318261085,
+            -0.003960837039351648,
+            -0.011848232223811548,
+            0.02903357274500332,
+            0.1105267435019781,
+            -0.006011029338781496,
+            -0.0053646998281195206,
+            -0.03242321704621517,
+            -0.015194102811332289,
+            -0.008860590919147765,
+            0.0056988205877963076,
+            0.01635957585050017,
+            0.020327661913559888,
+            -0.0008943408274661007,
+            0.016820860249955754,
+            0.02822175614376368,
+            -0.013854827839129885,
+            0.007906244895327178,
+            -0.030891381690403466,
+            0.015702117924362928,
+            -0.005383974926093977,
+            -0.012032955400313954,
+            -0.0026169511828034054,
+            -0.008618307035330928,
+            -0.0037356680273096105,
+            0.0009666430093455632,
+            0.011296307527684261,
+            0.004427516814276546,
+            -0.014484198313369681,
+            0.002829529156309784,
+            -0.008595300985863424,
+            -0.005122996529848575,
+            -0.02393571073523279,
+            -0.024119245543788293,
+            0.005223147047205755,
+            0.0019389305548629142,
+            6.344308292376579e-05,
+            0.006848965054539454,
+            -0.003810712095343454,
+            0.030149081317333418,
+            -0.01700322008272963,
+            0.0012288912810468446,
+            0.01193226302726573,
+            0.012332749933413344,
+            0.01688602002694633,
+            0.0028600984823761366,
+            -1.9882692552530084e-06,
+            0.021172977047420227,
+            -0.010514036509402561,
+            0.01460334896316005,
+            0.013091417303145243,
+            -0.04103572381458182,
+            0.0089746401463863,
+            -0.0027841289484449873,
+            -0.024252447218794344,
+            0.004446472209246425,
+            -0.018987867343500102,
+            -0.00881974688349601,
+            0.0005365625696166774,
+            0.012054719659248512,
+            0.01960300421041832,
+            -0.008553943699846696,
+            0.02111201386480852,
+            -0.0003501657740808554,
+            0.004177088753111459,
+            -0.012879944670340026,
+            0.0065222126975051685,
+            -0.007563354540402343,
+            0.0008168128607271199,
+            -0.018213241937597355,
+            0.012526561612175514,
+            -0.0008285725179894544,
+            -0.015737003159436623,
+            -0.011337688275309921,
+            0.004737346137976519,
+            -0.0027993545140114705,
+            0.0030843163114700313,
+            0.010153740340626154,
+            -0.008815302484351509,
+            0.004919275910399053,
+            -0.0017541042538165297,
+            0.002562816687714991,
+            0.014285249186421434,
+            -0.029816964593980453,
+            0.005357227738099325,
+            0.009154842495811638,
+            -0.00048782303639333943,
+            -0.013074976152495576,
+            -0.004054505559072866,
+            0.0028322102449485015,
+            -0.016877328079971877,
+            -0.012128199639779175,
+            0.023421330164419888,
+            -0.012565145848820333,
+            0.008609070188894315,
+            -0.010936598722317274,
+            -0.018401801734378692,
+            -0.01066119413133089,
+            -0.005254229780560808,
+            0.01234969508270852,
+            -0.0038427767204741877,
+            -0.006947594635952202,
+            -0.01756894543880097,
+            -0.007890897373987378,
+            -0.047246698568222466,
+            0.009617876521097624,
+            0.027157555263862723,
+            -0.011355758309667893,
+            -0.04050311381480088,
+            -0.01017774338301804,
+            0.02437257208967103,
+            -0.004056329295095018,
+            -0.008345450168215274,
+            -0.001599171409135998,
+            0.018358386463243014,
+            0.0027207445181251406,
+            -0.007292967749349572,
+            -0.011267861428599221,
+            0.01657507035043879,
+            0.015105061280816214,
+            0.021419854579631276,
+            -0.02088952706825817,
+            0.006295984081511286,
+            0.020973020776936996,
+            -0.03073246543555915,
+            0.0023720435876790876,
+            0.03363966216709148,
+            0.010412588613998134,
+            0.013023311559810034,
+            -0.0010217662933572685,
+            -0.0075567381078977215,
+            -0.008248818794295818,
+            0.011848200323549865,
+            0.005066463524567732,
+            -0.023174489297821385,
+            0.021823825355858556,
+            0.020368868756448753,
+            0.0026042856858509524,
+            0.003457067317891926,
+            0.0005539745115410357,
+            0.01062670844108589,
+            0.009948889803890108,
+            0.05100741556350918,
+            -0.006493714124814164,
+            -0.014021751055930108,
+            0.015063988102064609,
+            0.02488059237697814,
+            -0.0023499941119596228,
+            0.01087282522970853,
+            -0.02612218935931224,
+            -0.021423880914461062,
+            0.023389294741183817,
+            0.008967763132667892,
+            0.009637806238448432,
+            -0.010815706635624337,
+            -0.011390999462441815,
+            0.031750675158698435,
+            -0.01307829462211914,
+            0.00759749911213318,
+            -0.02169895957209634,
+            0.01387879409130081,
+            -0.008606362268863441,
+            0.032229658009209815,
+            -0.0009066257436946125,
+            0.011062436424260079,
+            0.028010486866468023,
+            -0.007835805879241447,
+            -0.019657486763625547,
+            0.003967480939987674,
+            -0.015577952617624695,
+            0.02188429826317828,
+            -0.020502758155233298,
+            -0.0002362500702658591,
+            0.0040935406015014265,
+            -0.010056462958039452,
+            -0.0057038083589231615,
+            0.011475170331848666,
+            0.0015345076181482267,
+            0.006359262977587827,
+            -0.020293014401106,
+            0.01369706440475074,
+            -0.003989482215141903,
+            0.007887824197823247,
+            -0.015925992045349828,
+            0.014370223626942766,
+            -0.02309139937510304,
+            -0.021808736033141787,
+            0.008142621179624918,
+            0.0108996933426044,
+            -0.007589050511458233,
+            -0.025971539510030145,
+            0.02250992080022228,
+            -0.024980833805349545,
+            0.024979008145354243,
+            -0.006622339940663538,
+            0.002566378488088469,
+            0.004913874740363343,
+            -0.008102103508732738,
+            -0.0028098905427027783,
+            0.0057865587191235565,
+            -0.0073509382400023995,
+            -0.014123756810537343,
+            -0.019453380788786282,
+            -0.04083629135586275,
+            -0.03177927047231389,
+            0.022927581786088044,
+            0.011557232599801421,
+            0.0017073832658476193,
+            0.02339552660545296,
+            -0.02247471449131386,
+            -0.017890866287500053,
+            -0.009775068417716798,
+            0.025184899522229628,
+            -0.04408541669037361,
+            0.012104255392615549,
+            -0.014669458760022245,
+            -0.0245258242481677,
+            0.006274252098230249,
+            -0.004161960147481513,
+            0.009299171724026455,
+            -0.010779421730862708,
+            -0.04713627671791054,
+            -0.005192684784377356,
+            0.02330975631183273,
+            -0.0036802876364168935,
+            -0.006197548344287773,
+            0.03048284650641829,
+            -0.0019503797069087845,
+            -0.005038254331399784,
+            -0.005524875459245342,
+            0.027783596017224226,
+            -0.024950083604372984,
+            0.010988398531346805,
+            0.008304937912856357,
+            -0.008485065895193593,
+            0.006572665697403556,
+            0.0011518846966102158,
+            0.011927348472282199,
+            0.015632588644277183,
+            -0.015559413716826264,
+            0.02262990861928957,
+            0.011446850286896198,
+            0.006506075777765303,
+            0.008015225053785824,
+            0.00010054430757843335,
+            0.0022975742216194894,
+            -0.024948925449668525,
+            0.026047924967851216,
+            0.01888211557921096,
+            0.0001632181017876392,
+            0.012553094476616919,
+            -0.015042621225242777,
+            -0.02782009986290425,
+            0.02606646297992026,
+            -0.016847653820092335,
+            -0.00045272379465939274,
+            0.020053091486344633,
+            -0.016210036358455773,
+            -0.007930550661303182,
+            -0.01123514849773852,
+            -0.0162675150307978,
+            0.01803076559921804,
+            -0.012160217543572069,
+            -0.02216594536057653,
+            0.002594675636753233,
+            0.016443123828746582,
+            0.02134480656850119,
+            0.026694575953313482,
+            -0.011470930697351533,
+            0.017768927048191052,
+            0.0008653408391607816,
+            0.001865765599795092,
+            0.025745954171649543,
+            0.01702677011968492,
+            -0.008474889109168202,
+            0.025347176686722313,
+            -0.0016686223698600755,
+            -0.002421921060670558,
+            -0.012265504084322075,
+            -0.0165214436458303,
+            -0.007545471138114208,
+            -0.015333235556146196,
+            -0.0051830106590765685,
+            0.000762746782664525,
+            -0.0018118022696726346,
+            0.10876576838038368,
+            0.0034948044540582626,
+            -0.011608460689239234,
+            -0.005870263500417206,
+            0.007510774308250375,
+            0.0023525201199670765,
+            -0.006846154764389575,
+            0.016082721103476375,
+            -0.018616626204086575,
+            -0.023593744079394,
+            0.004743481974389998,
+            -0.0009262286319469741,
+            0.00870122878944883,
+            0.02029523251993269,
+            -0.008608837818169835,
+            -0.0200567711498164,
+            0.007089477722191928,
+            0.005535149644041221,
+            0.0023640746516071774,
+            0.005256431824732927,
+            -0.0006118671321824933,
+            0.008209920240361308,
+            0.02448422122961942,
+            -0.015622892878860465,
+            -0.029770836607612903,
+            0.010535371601298532,
+            -0.00015994156470228762,
+            -0.007193425789067776,
+            -0.011046316463797128,
+            0.014728841492009627,
+            -0.0011488451990980268,
+            -0.02751404133172324,
+            0.017663132712536528,
+            -0.019055335178900415,
+            0.014643246200933506,
+            0.0025348345163347464,
+            -0.00016387173410118788,
+            0.022565953069005714,
+            0.013101158251834834,
+            -0.04272085076882881,
+            -0.0034442020309914823,
+            0.01041205537746844,
+            0.011059422927946552,
+            0.0116446016862942,
+            0.0012992535381387887,
+            -0.018670801241345514,
+            -0.011788984490283379,
+            -0.009109849103208542,
+            -0.021268488414296117,
+            -0.012523033785938503,
+            -0.011393853408628214,
+            0.010743646386439633,
+            0.0012964030814325153,
+            0.023809566099779113,
+            -0.003030535542277875,
+            -0.006526351867141901,
+            -0.000430699447948508,
+            0.00880561288237257,
+            0.009694646009259077,
+            -0.02079011138439714,
+            0.008294338719512953,
+            -0.01823922789220214,
+            0.011819769685519291,
+            0.006416044253693268,
+            -0.022799568832306067,
+            -0.009911619276012527,
+            0.007152786695893951,
+            0.008393677195358034,
+            -0.013569953300860946,
+            -0.006503988731872938,
+            0.02299498754558268,
+            0.018870619152710703,
+            0.012233964644738905,
+            0.024994616203196145,
+            0.03838969722235787,
+            0.005744255912479976,
+            0.027682123067526515,
+            -0.016791471092091604,
+            0.008459745383680297,
+            -0.022493868174622695,
+            0.012902262638928338,
+            -0.00460399519266689,
+            -0.0050431343566427886,
+            -0.018188192498411616,
+            0.033765105245412236,
+            -0.009159751849801773,
+            -0.0027150644587327763,
+            0.034778224828120714,
+            -0.0066502021793471075,
+            0.004525230911930458,
+            0.005096748531775903,
+            -0.016067049494073893,
+            0.033274672789154185,
+            0.01686831040872425,
+            0.016353966592947262,
+            -0.021631941883929817,
+            0.011748317746828848,
+            -0.005675701689448899,
+            0.0001273442591116645,
+            -0.020738819037390263,
+            -0.0037528323797385733,
+            0.0027725541343763104,
+            0.0059860509282538546,
+            -0.016556134518370437,
+            0.02033623476757896,
+            0.00019647023567641008,
+            0.0052895744489259875,
+            0.03253462247073863,
+            0.01043311189804924,
+            0.019182938878855655,
+            0.010051051068017807,
+            0.019523739618278902,
+            0.0281939391326836,
+            0.025685787482623124,
+            0.0029932688462254147,
+            -0.018668945726139946,
+            -0.0009326901784812825,
+            -0.007880352863625262,
+            0.010492063055849727,
+            0.022849902214818293,
+            -0.022619228461640748,
+            -0.02772644874135741,
+            0.018566981184526675,
+            -0.0022608185345048414,
+            0.020031103834122595,
+            0.007544464625538984,
+            -0.020935181884939796,
+            0.003407355515103286,
+            -0.023178230166337178,
+            -0.0005418155871620593,
+            -0.031039187720144967,
+            -0.028591174077243604,
+            -0.0007659241904525392,
+            0.011945344534674635,
+            -0.01688153017778311,
+            0.03357788163306365,
+            0.017667778671101954,
+            -0.014289109343082812,
+            0.01604684528817524,
+            -0.011054587325539086,
+            -0.0036959091138078683,
+            -0.005819853870004682,
+            -0.00203755010992417,
+            -0.02157944474979592,
+            0.005138614763345542,
+            -0.0033509138150859593,
+            -0.0024871060745482883,
+            -0.016936575573598154,
+            -0.021901830366872595,
+            0.016375173843894617,
+            0.016742770709641697,
+            -0.006606252542838758,
+            0.039771905404651496,
+            -0.004238071340064403,
+            0.00018364315349545657,
+            0.011690521707403059,
+            0.0010018392078563795,
+            -0.013651921068215563,
+            -0.010829994512513501,
+            0.012362647379788833,
+            0.011321403447882543,
+            -0.018942043512252663,
+            0.005763773608775982,
+            -0.009131565038297889,
+            0.002089216671850269,
+            0.005438887787329238,
+            -0.008183835275629478,
+            0.006690280329947951,
+            -0.01673229763262922,
+            0.015833397876364723,
+            0.00295474085300386,
+            0.003186554859191664,
+            -0.011797859525926772,
+            -0.020991927615680524,
+            0.01576413532776261,
+            0.0032403514518527287,
+            0.018819824905201635,
+            -0.026755218746060662,
+            -0.00012946467986739473,
+            -0.009552681500884924,
+            -0.0028941357661144826,
+            -0.012533800273548772,
+            0.013975640628890588,
+            0.016864739319953755,
+            0.0063941926578529,
+            0.01781016436903586,
+            -0.0026268224264609724,
+            0.015368284773773678,
+            -0.018122416427266218,
+            0.008686009180697955,
+            -0.006801074032721299,
+            -0.018537521789181048,
+            0.0013237338264105415,
+            0.027863741654435434,
+            -0.004670012662973029,
+            0.011066853115455966,
+            0.011704884649534624,
+            0.018954056680143452,
+            -0.0012626215734593707,
+            -0.0030850559460101866,
+            0.015610404665171367,
+            -0.005838226405658572,
+            -0.006861883833171456,
+            0.020872966233916948,
+            0.032951653763679145,
+            -0.006716215648910268,
+            -0.0021292599951407863,
+            0.010886596327481677,
+            -0.004901684487088035,
+            -0.008644584974417106,
+            -0.009587037829603014,
+            -0.03158137900717491,
+            0.024369089446706894,
+            0.012092412439477474,
+            -0.009601042951586109,
+            0.022690862146548786,
+            0.020752792552278052,
+            0.011412743968296708,
+            -0.010471860708195726,
+            0.02291536038589634,
+            -0.01268035142112569,
+            0.024168869865226136,
+            0.032482831775271157,
+            0.004400973556358811,
+            0.0022312906286559167,
+            0.0021175489661843985,
+            0.01691348460546639,
+            0.006050357346231155,
+            -0.007068133214675996,
+            0.011656003476409986,
+            -0.018487923180009552,
+            0.01598530891381001,
+            -0.004432366596808033,
+            0.02808371007118948,
+            -0.01337145216499331,
+            -0.010473700033368562,
+            -0.00537325376270318,
+            0.00010726431372328729,
+            0.006445701087575359,
+            -0.003324002570256215,
+            0.005255443653813165,
+            0.0007072577303252132,
+            -0.017825649146944237,
+            0.003050128130393781,
+            0.004825967170196209,
+            0.009018799597337101,
+            0.004103653497834134,
+            0.007249957401824204,
+            -0.012360355929483,
+            0.01441885664789707,
+            0.004401820363035808,
+            -0.0026190938717927086,
+            -0.0007163225760398784,
+            0.011124538295057815,
+            -0.013232822471919533,
+            -0.004895624967029031,
+            0.005281228035321008,
+            0.0059275753353475414,
+            -0.007507924998741734,
+            -0.004484140989405051,
+            0.011908076549392969,
+            -0.0037917291321648483,
+            0.006351543504726163,
+            -0.004409464890606014,
+            -0.0037760571637986827,
+            0.005303661313434895,
+            -0.00974511544836468,
+            -0.004204167898342997,
+            -0.0007093125445559502,
+            -0.006869250680233214,
+            0.00786202375969793,
+            0.005971033227117194,
+            0.010606085272785863,
+            0.006969437776542303,
+            0.004293129058563288,
+            0.004114452739123398,
+            0.004071017374828283,
+            -0.002445681623750424,
+            -0.00592480769363354,
+            -0.004617501328052899,
+            -0.012067458701076778,
+            -0.008335239153829813,
+            0.009025028998949398,
+            0.008653779421856954,
+            -0.019363018792788574,
+            0.02048011578221327,
+            -0.001188320960771659,
+            0.00772165044232104,
+            -0.01602086976290928,
+            -0.000907553574811279,
+            0.005464880629734941,
+            0.008267702703609528,
+            0.016136762227127184,
+            -0.03654209526492064,
+            0.005179791231187316,
+            -0.0037321883493524766,
+            0.016026013709955444,
+            -0.008215703411373182,
+            -0.006979816286354862,
+            0.02564060526656963,
+            -0.01450763388985563,
+            -0.0009847758228920171,
+            -0.016142304341196283,
+            -0.01407778890808498,
+            0.010815725252959506,
+            0.02050940280233426,
+            0.020684808835848383,
+            -0.0004965802897137407,
+            -0.0005781445948759235,
+            0.00036179841583004193,
+            0.022775804864342608,
+            0.015545681427566235,
+            0.018697154644607316,
+            -0.004543500782944068,
+            0.007028436664826706,
+            -0.023057215501463973,
+            0.009912581556496658,
+            0.0011795417129232596,
+            -0.008215027850908477,
+            -0.008554137076267468,
+            -0.008703378564520125,
+            -0.010448538943182258,
+            -0.011773811327190638,
+            0.004106616839960258,
+            0.013895013678116032,
+            0.004190549543250594,
+            0.0031571190600981074,
+            -0.0225385788570314,
+            -0.006243193914542516,
+            -0.011229641377928767,
+            -0.01674214043453308,
+            -0.007367389908398082,
+            -0.02386334969150212,
+            -0.028021339684039734,
+            0.0002794207370355175,
+            -0.02411861258809571,
+            -0.015958336406924612,
+            0.005268815864773021,
+            -0.01546610225139129,
+            0.00898600137695074,
+            -0.008396481690803911,
+            -0.011873486990732417,
+            0.0074743033679134105,
+            0.015336444788692,
+            -0.017676977079590463,
+            -0.010165317702640217,
+            -0.01690524990506033,
+            -0.013460451917691695,
+            0.010918550283871637,
+            0.00595960977280955,
+            0.009250157926438213,
+            -0.012447979337850305,
+            0.0062932435120863365,
+            -0.012673905639312696,
+            0.015073013552721701,
+            -0.03769166067890086,
+            -0.029992249463405965,
+            -0.00014990754540999537,
+            0.007288036630677436,
+            -0.0020735742042505426,
+            -0.018695120921334806,
+            0.0003860116505754868,
+            -0.003608995261360232,
+            -0.014900191335144973,
+            -0.013866216738266176,
+            0.025251475305524053,
+            -0.006788483107616133,
+            0.01223227424986397,
+            0.0034937277040015626,
+            0.03438333560343413,
+            0.008803852645562129,
+            0.014666206816302367,
+            -0.005709421955160623,
+            0.003742433643393657,
+            -0.0033706258426092744,
+            0.006906464031352315,
+            0.0076578888497945185,
+            0.009276592191710524,
+            0.008025321004305927,
+            0.004576470294889512,
+            0.002887346543823031,
+            -0.001925468214331082,
+            0.02125241545487307,
+            0.025671028960818562,
+            -0.00037467906648160044,
+            0.008981063192139038,
+            0.0009387900084090577,
+            0.0002735789978680719,
+            -0.008374290340790827,
+            0.006547013284767913,
+            -0.008274637814401893,
+            0.004041066530254948,
+            0.012897549078838394,
+            -0.009150863940242148,
+            -0.010211568689323908,
+            0.016575088843292566,
+            -0.012380099999574783,
+            -0.03420455578618604,
+            0.026550654680719987,
+            0.005324512466403649,
+            -0.0035670106106754245,
+            -0.010657928468872608,
+            0.0037176027853158167,
+            -0.01103891334887315,
+            0.008096915671982816,
+            0.022587659771132487,
+            -0.0013527917944322358,
+            -0.0018085171207746743,
+            0.0038503794877980586,
+            -0.011870484733648887,
+            -0.007881489342351292,
+            0.006462772034381968,
+            0.0315696514747912,
+            0.01326417653779577,
+            -0.020400369539140532,
+            -0.015223808971473513,
+            0.020406862349006738,
+            -0.023142176712111575,
+            -0.006254959806032159,
+            0.009039506093607052,
+            0.0009087520492922078,
+            -0.022766046838415165,
+            0.0009662037093312435,
+            0.004701966191595815,
+            0.005946458676746661,
+            -0.006358986495019584,
+            0.03237532341275415,
+            0.001075992257265679,
+            0.015892020271402903,
+            -0.017189453434945788,
+            0.0017514632185789299,
+            -0.014189278671121677,
+            -0.014265447388465805,
+            -0.029809460132405256,
+            0.0005557491671071693,
+            -0.0030324198417091704,
+            0.011038984948790744,
+            0.024808067315720335,
+            0.01145103533194245,
+            0.014432246996007246,
+            0.010552852947578788,
+            0.01513689173494211,
+            -0.013653722180565311,
+            -0.029009757850432732,
+            -0.005171522291984226,
+            -0.018001048743500243,
+            -0.02297482628212251,
+            0.003962613844063587,
+            0.013645125285548414,
+            0.006278662690346082,
+            0.023786381386575035,
+            0.0021738516691483163,
+            0.0009580591750940011,
+            -2.690805459518147e-05,
+            0.008894091070322921,
+            -0.004952203441302471,
+            -0.006553986197076102,
+            0.003570189040872711,
+            -0.000652714137223501,
+            0.021228360881461853,
+            -0.005758946428212041,
+            7.897153237591187e-05,
+            0.014242175492137011,
+            -0.005622256348934008,
+            -0.0035009192529791146,
+            -0.017339926143652377,
+            0.006248842389703505,
+            -0.01622011586976866,
+            0.025783086900871728,
+            0.018450783573486916,
+            -0.009109295293546267,
+            -0.013469521573468915,
+            -0.011259555201313116,
+            0.02773097452736661,
+            0.004631282554715908,
+            0.010787055220617197,
+            0.018228322680582464,
+            0.01461254259249465,
+            0.01784830193964943,
+            0.004664725683522323,
+            -0.0034339781146649384,
+            -0.0007692550982545399,
+            0.021176646860433974,
+            -0.014700178590249292,
+            -0.002701208084390198,
+            -0.008842788249773623,
+            0.0021799005046816444,
+            -0.008431299000066363,
+            0.006086447848362669,
+            -0.010793016137040172,
+            -0.015251381840165954,
+            0.0189503457652375,
+            -0.02660767353735015,
+            0.016423753600066336,
+            -0.00593016170680532,
+            -0.00482891500342298,
+            -0.027459214027897378,
+            0.010728039528784636,
+            0.0027866316191608297,
+            0.0022519974195000687,
+            0.010770472550461752,
+            -0.021427634123401408,
+            -0.01867717541189363,
+            0.008532444554010673,
+            -0.011457117350255678,
+            -0.007152301670244725,
+            -0.00348619742388509,
+            0.0154715607296319,
+            -0.00017898955187849443,
+            -0.009925365923862246,
+            0.02085093731060963,
+            -0.007175309371250024,
+            0.01223979370120031,
+            0.005256127464980655,
+            -0.008756077893734508,
+            -0.0028438320920811362,
+            0.021022708898898203,
+            0.027557382862284808,
+            -0.008342206702778956,
+            0.014526066580760486,
+            -0.011883797998224506,
+            -0.01088335352319397,
+            -0.03479444009382274,
+            0.030104838505937447,
+            -0.013305732257226255,
+            0.026773747582411633,
+            -0.0006968265979630097,
+            -0.017037729111584094,
+            -0.016206207453228583,
+            0.01625225876515654,
+            -0.0024290584023212254,
+            -0.009366287391027166,
+            0.01465885301870365,
+            -0.02428628495473643,
+            0.0058336571592899735,
+            -0.0006627935593670669,
+            -0.011531762838191602,
+            0.013275508677482622,
+            0.0019223252930710324,
+            0.010738622785461513,
+            -0.02030405805119001,
+            0.0009661595529028194,
+            0.009263716373456188,
+            -0.016791855691574455,
+            0.006057830565306032,
+            -0.03730952786847912,
+            0.0029341721588188943,
+            -0.014833895684348656,
+            0.005725615593914644,
+            0.001498755253963699,
+            -0.01820708352937343,
+            -0.03652692232407375,
+            0.00043665701245599497,
+            0.01913883585802903,
+            -0.007975988799790905,
+            0.020538595326175857,
+            0.008243569241768276,
+            0.008421404577931517,
+            0.002314738067443065,
+            0.0016702403734608475,
+            0.008279538988821465,
+            0.03679456861213399,
+            -0.01775927634148114,
+            -0.004836606118958108,
+            -0.0012151329859941935,
+            0.010644195708319258,
+            -0.02028470571116305,
+            -0.009287515726906219,
+            -0.0036506912311261536,
+            -0.013134467570201307,
+            0.030615862647585222,
+            -0.020265555441523482,
+            0.025634449009756734,
+            -0.009300770899023007,
+            0.009274072905190077,
+            -0.02097685009125915,
+            -0.009797008690754956,
+            -0.010661540100131183,
+            0.0019532291898233895,
+            0.013465005282849381,
+            -0.009151430285942593,
+            0.0071467934925125145,
+            -0.006655608635025676,
+            0.01020327561163374,
+            -0.001472597695707201,
+            -0.02443508800166349,
+            -0.0024998811551575563,
+            -0.01107321258077183,
+            0.02456164421154801,
+            -0.0019619696941303388,
+            0.0005043179699158694,
+            0.010951580163871948,
+            -0.00853593956432799,
+            0.023927336608555987,
+            -0.008855702280766362,
+            0.0010093679746665581,
+            -0.004863269000020324,
+            -0.004005691415117686,
+            0.001698212044518236,
+            0.01822024022650613,
+            -0.005707404715849356,
+            0.03722587302228139,
+            -0.01708925706528118,
+            -0.0065500772743641925,
+            -0.00015538686795361824,
+            0.010852925330001283,
+            0.010035359503147549,
+            0.008754719437760327,
+            0.004067820167375348,
+            -0.014861417626433965,
+            0.009346669946626484,
+            -0.020136029447262147,
+            0.007501669351583119,
+            0.023178490169347605,
+            -0.006840359541885465,
+            0.0006676450965357345,
+            -0.01283320739092204,
+            -0.015610538827742276,
+            0.006308543164022682,
+            0.002329420005586303,
+            -0.002295739637504597,
+            -0.02181189272979859,
+            0.006298804075737111,
+            -0.014398470126723037,
+            0.004007419942728446,
+            -0.0007497656993578918,
+            0.013863867106289847,
+            0.016451166273826513,
+            -0.006740209707090658,
+            0.039292260347522444,
+            0.01871723939316731,
+            0.009109005310505462,
+            0.005674782595856756,
+            -0.040588444759736254,
+            -0.008993850998154852,
+            -0.0010197185016393919,
+            0.004866150984083949,
+            -0.01141354216871663,
+            -0.018944881712855702,
+            -0.002646110151132781,
+            0.000308585036997592,
+            -0.010681954707313127,
+            -0.009091585604191968,
+            0.01751599002710486,
+            -0.006034424764302979,
+            0.00872057856926618,
+            0.004146961781109887,
+            0.023758065635807138,
+            -0.009727802672285544,
+            0.01176257119851645,
+            -0.0071822083765114165,
+            -0.005456197751568526,
+            0.0006726765703478756,
+            0.011011628907288813,
+            -0.008031792262886802,
+            0.009763340867550022,
+            -0.004279891138185881,
+            0.008355720233269893,
+            -0.008552596384067825,
+            -0.023952129458400563,
+            -0.014972827087541732,
+            0.014347535404251018,
+            0.011651260496709961,
+            0.008137738462704715,
+            -0.013624435569539285,
+            0.0031728721674035352,
+            0.00675830277744976,
+            -0.009184784787581942,
+            0.009161771311910335,
+            0.017733082848622708,
+            -0.001411542568472906,
+            -0.0017830167499636113,
+            -0.0016130003349297027,
+            -0.012173528776899308,
+            0.0011196437753239162,
+            0.02284887358189995,
+            -0.003054479643937998,
+            0.014614283913976716,
+            0.010689486748036806,
+            0.009643292262102645,
+            -0.00810930837603355,
+            0.001915955511174678,
+            0.02341863048264366,
+            0.015022291783105098,
+            -0.004625715884342598,
+            -0.011422973894623295,
+            0.004259757717597968,
+            -0.010332744704689279,
+            0.007373328473086509,
+            -0.02117940113144938,
+            -0.004672698999328004,
+            0.008896311380763048,
+            -0.007028862917603513,
+            -0.01732432285099198,
+            -0.028026173555055017,
+            -0.016765805892242715,
+            0.019819329380181,
+            0.0027213760678977965,
+            -0.013291251876048186,
+            -0.0032919084721662363,
+            0.0031214883337300926,
+            0.002553005168048115,
+            -0.0071104806975682885,
+            0.005310583052470149,
+            0.007359183453835734,
+            -0.008674133980486269,
+            0.017665958608281126,
+            -0.000521269431423141,
+            -0.0008020251454616109,
+            -0.0005442374261144659,
+            0.0006127687598711624,
+            -0.016387752806126442,
+            0.01666161912459431,
+            -0.010949228376419892,
+            0.015198110119816624,
+            -0.008966726991404666,
+            0.008600499511690635,
+            -0.007617043351740469,
+            0.009458641139070436,
+            0.011597957400445055,
+            0.012755037944648426,
+            -0.004403680820063604,
+            0.008519241473418196,
+            0.004807816191718704,
+            -0.008147125240985166,
+            0.007017601015398136,
+            0.01296082929962557,
+            0.00582353932160707,
+            0.0013634237970372698,
+            -0.0008246894322793264,
+            0.0022035670958705576,
+            -0.002480976219285683,
+            0.0056293125704658135,
+            0.012714850523393719,
+            -0.002497639449055006,
+            -0.007135683201188999,
+            0.014988249771676699,
+            0.014955895630183352,
+            -0.022994948923833993,
+            0.007121634434082677,
+            0.005835971825639792,
+            -0.0030507120821557008,
+            0.002145266953387315,
+            -0.027334525318320587,
+            -0.020359474070022213,
+            0.01901995970350808,
+            -0.015260454474605018,
+            0.02850460517419317,
+            -0.011624138792780823,
+            -0.02284242513728605,
+            -0.01277489645734181,
+            -0.004209529633534193,
+            -0.022684500849916897,
+            -0.004894414098943742,
+            0.01919165657366256,
+            0.013401560972810082,
+            -0.004313549020265677,
+            -0.019089369171102138,
+            0.03863248408555474,
+            0.010497951840980463,
+            0.03508157919687289,
+            -0.008597970944278045,
+            0.0023354831464448313,
+            -0.01563401808449416,
+            0.015551176593436512,
+            0.001075882802133782,
+            -0.011575692834796734,
+            0.01505716108583101,
+            0.031836226915668404,
+            0.0034740889237207494,
+            -0.00617206855371193,
+            0.0018420207148235309,
+            0.00418433325800744,
+            -0.0008167766650900381,
+            0.011160989178055499,
+            -0.018497472318034093,
+            -0.011951262348399922,
+            0.0013093376768780122,
+            0.00216347903176653,
+            0.0059233460419514045,
+            -0.001778091863744448,
+            -0.013317639704511549,
+            0.011433308892643204,
+            0.011596810919469926,
+            -0.01518143105525228,
+            -0.007497373860338335,
+            0.0050957665991362405,
+            -0.018974405317819656,
+            0.01872472902494418,
+            0.011859522398789263,
+            0.000792413325332956,
+            -0.0006503947129906383,
+            0.00808635755624928,
+            0.003749813548995223,
+            0.0033657799335872368,
+            -0.007862215303617144,
+            -0.0028014669075116876,
+            0.0009892310856576799,
+            3.834379772674841e-06,
+            0.015402211364447726,
+            -0.015264893397161102,
+            0.02488855366696587,
+            -0.0025643639525691083,
+            0.00953993868921819,
+            -0.017460590918773188,
+            -0.0017911712931080155,
+            -0.00402374787529868,
+            0.01838051670627292,
+            0.006753072393532114,
+            -0.010130676216186492,
+            0.004921510943896829,
+            0.007975201081296845,
+            -0.012589786869771252,
+            0.00018319054705558588,
+            -0.0022606180909706688,
+            -0.014791528072476771,
+            -0.0031171076370692706,
+            0.017873825561816544,
+            0.007593688317551089,
+            0.0037375070591255375,
+            -0.015863143964928172,
+            0.007192324815779268,
+            0.007522990667369163,
+            0.024504875417877527,
+            -0.026309816223960274,
+            0.024779042752840836,
+            -0.022104087334088543,
+            0.02165781034613893,
+            0.0319051741666687,
+            0.007538146369110045,
+            -0.01690181186708103,
+            -0.006595947249169521,
+            0.006373991305923002,
+            -0.023410814842488123,
+            -0.005494017994983188,
+            0.016903736810723716,
+            -0.01792400152372258,
+            0.000330865061689199,
+            -0.014660413003063089,
+            0.0040825880154611895,
+            0.008248037865838212,
+            0.011540333451799686,
+            -0.0033578830043738085,
+            0.018327525495676525,
+            0.020171745352159914,
+            -0.01138759854371235,
+            -0.0004221149053107702,
+            -0.022321409200941777,
+            0.005046865873981027,
+            0.022509643273449936,
+            0.001111272240506322,
+            -0.009540467231190513,
+            0.018376261423745193,
+            0.008480532791922674,
+            -0.020801891594934313,
+            0.008436374227753589,
+            -0.036175982655998785,
+            -0.010454118523313122,
+            -0.009328507118928412,
+            -0.001218633166683384,
+            -0.004142773602181115,
+            -0.0002494371580772723,
+            -0.011365548842513265,
+            -7.699548642315061e-05,
+            -0.006889339420877534,
+            -0.0037173070609413474,
+            0.014953978004522174,
+            0.011678957032329096,
+            -0.003501047405385249,
+            -0.0073984932581048915,
+            0.008493128567102267,
+            0.006163937693564917,
+            -0.017034696006318337,
+            -0.013268418502042576,
+            0.0037122128689059085,
+            -0.020067656401141467,
+            0.005148329575092478,
+            0.00459729178556505,
+            0.014031669598917171,
+            -0.01849235806598133,
+            -0.009298127665365717,
+            0.019716512208448042,
+            0.0094466405158553,
+            -0.0008988916054855498,
+            0.004264260983102956,
+            -0.00032312548009733906,
+            0.002279987803645912,
+            -0.0008902345374346296,
+            -6.145528806973222e-06,
+            0.003602229465148649,
+            0.012503366799381277,
+            0.007759393855482671,
+            0.03694402922447051,
+            -0.016692148708641812,
+            0.005873765863350649,
+            -0.014187609702966002,
+            -0.0037592517335595654,
+            -0.00749939010396784,
+            -0.0004308157309481927,
+            -0.0022282234382577843,
+            0.0021999996319642854,
+            0.0052176773733353925,
+            0.009353171029508977,
+            0.002155405440765206,
+            -0.00823565584692709,
+            0.009691447250255694,
+            -0.015228173095533029,
+            0.003853858061067531,
+            -0.008207910291425473,
+            0.01017972550354623,
+            -0.009695280827515048,
+            0.0028705587799331246,
+            0.03849358164875786,
+            0.01130021300937487,
+            -0.004934175853966098,
+            -0.014300147534361459,
+            -0.0007796051618993811,
+            -0.01737877501558196,
+            -0.008579119859817895,
+            -0.005434792488602325,
+            -0.02617338653902575,
+            -0.00838916510137031,
+            0.015344260464413648,
+            0.011202567142310492,
+            0.014124853863170132,
+            -0.023894182185855824,
+            0.008897943489322673,
+            -0.03001670823127986,
+            -0.011951507895140876,
+            -0.001530967140156163,
+            -0.015467812535428609,
+            0.0010110999991167703,
+            0.023623196504289838,
+            -0.007762735863648054,
+            0.012654652822750325,
+            0.012867958170258448,
+            0.018777522196571934,
+            0.006390986057157016,
+            0.00038045736974183937,
+            -0.009444911783760569,
+            0.009534310165417939,
+            0.008806031651319704,
+            0.0080954543713255,
+            0.013161833638467553,
+            -0.0022840529693446553,
+            -0.018060162439023572,
+            0.005469288590000856,
+            0.014354475778235131,
+            -0.0033504276914600585,
+            -0.001615191819057537,
+            -0.011088493180661976,
+            0.016039040565630432,
+            -0.023559046333872156,
+            -0.009482398794557168,
+            0.009918576960319548,
+            -0.0027308309155029842,
+            0.009212063547988036,
+            0.0160181200081126,
+            -0.01881373001297218,
+            -0.0024168618343039703,
+            0.016244889977796682,
+            0.0024104546308670025,
+            -0.00022606425578303413,
+            0.016792244238218245,
+            -0.0007616857360728623,
+            -0.004279120886828465,
+            0.00997764266310123,
+            -0.0075631917138318655,
+            7.3878137053128405e-06,
+            -0.002416364915008295,
+            -0.0028253498168087964,
+            0.02054128727706496,
+            -0.021281801678871736,
+            -0.005379259148621041,
+            0.02257306103533458,
+            0.015687105622685975,
+            -0.007843745111084413,
+            6.258498042855072e-05,
+            -0.0219506172079994,
+            0.0019925886885244454,
+            -0.007768133999661569,
+            0.016524850592507592,
+            -0.007957949830060354,
+            0.0030085044673901665,
+            -0.01361024620290394,
+            0.011563781350927133,
+            -0.01277407169098418,
+            0.00873716215529134,
+            -0.002986943948333218,
+            -0.017567646684567492,
+            -0.0053109390854215125,
+            0.004561236136619266,
+            0.0014045175130801308,
+            0.044958041347040185,
+            -0.005539136232703934,
+            0.003729517522976541,
+            0.018086335927383804,
+            -0.014199453349727933,
+            0.02079719360289355,
+            0.016663293302592568,
+            0.00991445855219406,
+            -0.00033967370741635794,
+            -0.0018726748849858836,
+            0.023373354806639515,
+            0.013258226441642464,
+            0.00861986998150065,
+            -0.0014340218811281975,
+            -0.008516541227326329,
+            0.018239230461345617,
+            0.005458629943667832,
+            -0.012421896028975013,
+            0.012732721349758832,
+            0.013559877132508506,
+            -0.013720939472355347,
+            -0.011545210834395917,
+            0.01616358223872008,
+            -0.019904594319155373,
+            -0.018260016571763644,
+            -0.0036124537591368864,
+            -0.01614510591132948,
+            -0.03864699384205461,
+            0.01619401261888991,
+            0.018351607973242735,
+            0.006666313000123845,
+            0.013647410298896132,
+            0.002597680061427196,
+            0.004545530267244157,
+            -0.018531082430861854,
+            -0.0077618625909995275,
+            -0.008085144524135461,
+            0.003744735923004248,
+            0.011059457364156674,
+            -0.011335432904114519,
+            -0.009514379703581616,
+            -0.00034500702695418285,
+            -0.007433132060989644,
+            0.01971420270994951,
+            0.0020753610290475642,
+            0.0034506706065161024,
+            -0.00628437298309269,
+            0.017966485854173167,
+            0.026789678964152908,
+            -0.007532446825077476,
+            0.021886182869092073,
+            -0.002864230253056434,
+            -0.0015870360120997492,
+            -0.014393519359298496,
+            -0.0014439412842964876,
+            -0.011527182213472351,
+            0.0005453493437517964,
+            -0.0006791481533636673,
+            0.014093767997869645,
+            -0.03288824642147967,
+            -0.003957301296796813,
+            0.014444592778851293,
+            0.007538052312876599,
+            -0.005057755472988024,
+            0.011357919882949903,
+            -0.002964451028327082,
+            0.016977317604413924,
+            0.0075201055190542146,
+            0.023391646079994784,
+            -0.016588823248397513,
+            -0.004504858318770565,
+            0.006660877064954993,
+            0.0034631255931793638,
+            0.0014484203909961972,
+            0.01912983041187014,
+            0.023439941258581887,
+            -0.014715886385460541,
+            -0.0013596123366001458,
+            -0.0029020372473879614,
+            -0.021414221343490594,
+            0.014976806123253727,
+            0.0074729481304182865,
+            0.010967351226114886,
+            0.009806663105020302,
+            0.015465076881663691,
+            -0.02109587527354609,
+            0.0011671121739822202,
+            0.020567458365051382,
+            -0.008618333549105003,
+            -0.018815706906171112,
+            -0.009804829158883123,
+            -0.00966828714719191,
+            -0.006078893678150302,
+            -0.00803783313451146,
+            0.005523023167028105,
+            0.0022647506733877573,
+            0.01652997604669998,
+            -0.008204200434441975,
+            -0.008404609147866811,
+            0.008489247125986078,
+            0.007546785673285454,
+            -0.0033093451283570316,
+            0.010614050963656496,
+            0.0030633179859393587,
+            -0.006940540318011839,
+            0.00894884343338777,
+            -0.002787816848430143,
+            0.014347598920245487,
+            -0.02441769157218451,
+            0.004717742147185283,
+            -0.012096876825337124,
+            0.006405151160675699,
+            0.0041568790527989905,
+            0.0016702066682613428,
+            -0.0041159211459004,
+            0.0035217362856754353,
+            0.0008893479210743541,
+            -0.07433312151926533,
+            -0.012629681414004524,
+            -0.0035797012285864486,
+            0.033657838713146884,
+            -0.019695806809053914,
+            0.004477927056336216,
+            0.011966065644382396,
+            0.0096717896523898,
+            0.003070713232538593,
+            0.006136726101157844,
+            0.006112385569942848,
+            -0.0245258242481677,
+            -0.00015284418977092615,
+            0.004821040694679944,
+            0.0020347004045943454,
+            -0.007532408630122127,
+            0.00750565079228405,
+            0.015327710245879296,
+            0.020989926895388523,
+            -0.010970305087289488,
+            0.017227663358885748,
+            -0.003352718496981928,
+            -0.006068892126250978,
+            0.0014285750322333686,
+            -0.010153799722483554,
+            -0.017307415843726394,
+            -0.006495008086641125,
+            0.0068983699213297075,
+            -0.001982568363667046,
+            0.0044032687939897855,
+            0.003333596495330213,
+            -0.0032171586993948696,
+            0.0072365127663633925,
+            0.013920326463272646,
+            0.006415916101287133,
+            -0.013675562796300214,
+            0.003003782758807665,
+            -0.02500375035312058,
+            0.011945988879567646,
+            -0.012708975181025296,
+            -0.0009488161508767279,
+            -0.0047093877027523014,
+            -0.0035750724825159503,
+            -0.002213435677949806,
+            0.02043567538125188,
+            -0.014304009033474186,
+            -0.01802899775650995,
+            0.008187227768196673,
+            -0.011217013195839762,
+            0.008300740149622635,
+            -0.03382831122769974,
+            0.012386335491333264,
+            -0.0276029934382802,
+            -0.0002243752790856292,
+            -0.004521387639042107,
+            -0.0010631721066129095,
+            -0.009491184157278669,
+            -0.03274559774619223,
+            0.015845705808609598,
+            -0.007204751554079551,
+            -0.020338337341085228,
+            0.0007143424225187802,
+            -0.021267094083674304,
+            0.014150458196361147,
+            -0.01851165553224022,
+            0.0033450353123933058,
+            -0.018775439467431685,
+            0.002782289821140401,
+            -0.0018711350216089556,
+            0.010502195116579441,
+            -0.00557814442809019,
+            -0.0007130962242755068,
+            -0.013072988887382269,
+            -0.002121841212381393,
+            -0.006653444103331213,
+            0.009394907690516121,
+            0.0028678074167728367,
+            -0.009601577734902932,
+            0.011153571864462314,
+            0.013046116892519414,
+            0.019812860521973938,
+            -0.00539047787915588,
+            -0.00555139958411929,
+            -0.011553689090179406,
+            0.009852063270798534,
+            -0.016326212706634428,
+            -0.00827402913960903,
+            0.008982303148984887,
+            0.015216947130173639,
+            0.01026889105233137,
+            0.003352028462252994,
+            0.003150801750574548,
+            0.002531620731961765,
+            0.026744691368233237,
+            -0.0015120532859309638,
+            0.0008324701960729416,
+            -0.014508988496009931,
+            0.022127525293249926,
+            0.02772430884375744,
+            -0.0012206102581726523,
+            -0.00018859639594030037,
+            0.017437360169132967,
+            -0.004697052522250805,
+            -0.0038111184371524645,
+            0.024015246959315593,
+            -0.038396108183338655,
+            -0.008416825232533771,
+            -0.006788816882617399,
+            -0.006043428198944214,
+            -0.009587366247924467,
+            0.007294190273465812,
+            0.00041966080769004275,
+            0.0024709692753547823,
+            -0.0005250904505954556,
+            0.03452858797518386,
+            0.003403515654095309,
+            -0.0003392060101496309,
+            -0.0032980864446307116,
+            0.0019125121221204029,
+            0.0032987517853214346,
+            -0.009116761755779728,
+            -0.038030085122772624,
+            0.006006334000396066,
+            0.007932785488877407,
+            0.00975009749364395,
+            -0.008120121513849434,
+            -0.008864461019882246,
+            0.004909348738798099,
+            -0.02273129283865203,
+            -0.01372636307167243,
+            0.02625229144362647,
+            -0.008523028986127237,
+            0.029005877324974557,
+            -0.022435059061182174,
+            -0.013135836801818153,
+            -0.004126433322528265,
+            0.010283975013795537,
+            -0.019732856462444506,
+            0.004343860338939202,
+            -0.004022387065239562,
+            0.023733721026005954,
+            0.0077578144626961245,
+            -0.008371631446146718,
+            0.012386735614153616,
+            0.0064789380436815714,
+            -0.0026638642827013837,
+            0.00136333052963848,
+            -0.007209517067748103,
+            0.016916505654836485,
+            -0.011819784026452944,
+            -0.005833909403097991,
+            0.008977527508585844,
+            -0.0024556434204852946,
+            0.02509293948918158,
+            -0.0237762245071987,
+            0.005288171058938098,
+            -0.00019047961064753207,
+            0.01822542471604037,
+            0.008127347078975925,
+            0.008361588667386882,
+            -0.011835407590060994,
+            0.010004482616697591,
+            -0.03378107067882622,
+            -0.015433906677849963,
+            -0.007702910250795191,
+            -0.0013877702264578865,
+            -0.002059152251387601,
+            -0.015745218548968338,
+            -0.026393221735795527,
+            0.006294583740809379,
+            0.016267525575472445,
+            0.023291528020980265,
+            -0.004683197930892445,
+            0.012912830736650997,
+            -0.002831583991769044,
+            -0.008904271394710672,
+            -0.011441882849296418,
+            -0.012202629605208118,
+            -0.012470773094383122,
+            -0.014560694782340029,
+            -0.017045701603711442,
+            0.012139665577738337,
+            -0.020079061996873537,
+            0.013911518328924777,
+            0.0347210775522539,
+            -0.017176114991595492,
+            -0.0018941772168839354,
+            0.019702722555622765,
+            -0.004221469612316137,
+            0.01212088353057276,
+            0.0063707489618353725,
+            0.015298014532351866,
+            0.0038643174573634725,
+            -0.012181479717826009,
+            -0.007374005740809251,
+            0.019411457664902376,
+            -0.004902591844710158,
+            0.014591446725717338,
+            0.015069599884557399,
+            -0.008675672164385891,
+            0.002544198867387345,
+            0.01951865904496276,
+            -0.01762101777798757,
+            0.01665654707715257,
+            -0.00458402229480694,
+            -0.009936206685538294,
+            -0.013345740581684638,
+            -0.005336416724426228,
+            -0.019469457091169776,
+            0.0061608454109892425,
+            0.018931676738452464,
+            -0.00533013594306834,
+            0.009310871143727683,
+            -0.0037670793158045,
+            -0.005045151384034014,
+            0.0002041909886640482,
+            -0.04152751596227068,
+            -0.008340911336217609,
+            -0.0006651584370005978,
+            -0.02505748815737022,
+            0.025954102004571416,
+            0.007205602377144382,
+            0.022602344303400163,
+            -0.017168610245491406,
+            0.004459735690564842,
+            -0.005882024209102714,
+            0.013217517324383834,
+            0.02255689669623969,
+            0.0017984057469785134,
+            0.005699128497215882,
+            -0.02353849721739205,
+            -0.0034711576718386185,
+            0.015924913374553738,
+            -0.003604478141616307,
+            -0.0034993903155063817,
+            0.012592308877056896,
+            0.012864938348003884,
+            0.01567418642605425,
+            0.008801753550506258,
+            0.015606443428689292,
+            0.010589933994371628,
+            0.008928721556432368,
+            -0.005805129999077126,
+            -0.001025103064751441,
+            -0.015907755521365925,
+            0.0045874330999415985,
+            0.031701049450012876,
+            0.012594483199773746,
+            0.016783858873835704,
+            -0.01778536268185936,
+            0.00789523134933153,
+            -0.0012706411671644011,
+            0.008325108613530198,
+            -0.001677173669891956,
+            0.029540136288412976,
+            0.011739999605699826,
+            0.015670966746894652,
+            0.008732334090105053,
+            -0.016237731209426752,
+            0.045334690225148,
+            -0.016911928942220152,
+            -0.0021139311364009113,
+            -0.01058970056572469,
+            -0.021705338531028618,
+            0.007324792557651635,
+            -0.006556582104686972,
+            0.0018243188317438144,
+            0.010780292738415671,
+            -0.006471850255151019,
+            0.0006997395548596146,
+            0.003907433192252573,
+            -0.023129313269763224,
+            -0.002550875642832235,
+            -0.02237917712764273,
+            -0.003107775639788617,
+            -0.02621608419628401,
+            -0.00012482032027919036,
+            -0.016207377503696517,
+            -0.0038712739995960753,
+            -0.023315508569288476,
+            -0.014963211090562272,
+            -0.01633110043829169,
+            -0.004241081909202262,
+            -0.008295402942465353,
+            0.004976396647108597,
+            -0.007402107755970892,
+            -0.020321297548927468,
+            0.0020986169430507757,
+            0.016803273183070004,
+            0.01434729320047051,
+            0.018612970083178984,
+            0.00603868809307069,
+            0.014278001198851581,
+            -0.0026350854954985117,
+            0.017171016209107656,
+            -0.006968968261909459,
+            0.006881847166565678,
+            0.011454001418121107,
+            0.02140935294963631,
+            0.011121833270235448,
+            -0.012801756637470525,
+            0.0010870041670931706,
+            -0.009655388034828094,
+            0.002094601818569396,
+            -0.00828486034340067,
+            -0.009190975158350585,
+            0.0017762581946855365,
+            -0.007261996868906865,
+            0.004674405202069334,
+            0.0030526253833799426,
+            -0.0014082232541965316,
+            0.010419302200840906,
+            -0.0024200053377751805,
+            -0.012208685875688886,
+            -0.0033037330470612695,
+            -0.017975940046938498,
+            -0.006958042903103732,
+            -0.0014496522238488239,
+            -0.011711502944602638,
+            -0.01016130418405875,
+            0.003321436189059608,
+            -0.00538282506552848,
+            -0.009353581612407435,
+            0.013884147453136417,
+            0.0001309738696853529,
+            0.0005878330075918333,
+            0.023495280602974504,
+            0.006035028203642483,
+            0.008087594627921345,
+            0.013144288945385904,
+            -0.01023995000994454,
+            -0.022574958230635715,
+            -0.0243435727055662,
+            0.007867022215096418,
+            -0.01573478573423379,
+            0.02324699908478915,
+            -0.007929809922043611,
+            -0.00561630876431978,
+            0.0020982723335057625,
+            -0.013699317926847062,
+            0.008578431496569186,
+            -0.011271381061801356,
+            0.0014450645788554738,
+            -0.019581451799757368,
+            0.018059311800267413,
+            -0.010917911869373693,
+            -0.004302687883018328,
+            0.009466103226841532,
+            0.034049086327052606,
+            -0.0005689605174080737,
+            -0.007410970644432269,
+            -0.006594150877740108,
+            -0.00022810271313786974,
+            0.018766765773409123,
+            -0.0029041815756733333,
+            0.005758781908059541,
+            -0.01563519107805783,
+            0.006571628287171105,
+            -0.006132372252825422,
+            0.0012423261344297225,
+            0.01513074831647933,
+            -0.026306320707079913,
+            -0.015491710256868008,
+            -0.007857352164317122,
+            -0.024893430640551818,
+            0.002253574511878983,
+            0.017579477884206534,
+            0.020505606775521504,
+            -0.020324505029926956,
+            -0.02513332780748072,
+            0.005150966733930881,
+            -0.026580994694344065,
+            0.0037390577082486594,
+            0.1241031878517593,
+            -0.002138404898309707,
+            0.002720723037908521,
+            -0.011425036886011179,
+            0.0022399461009316216,
+            0.005389407121650262,
+            -0.019552543955456567,
+            0.05950746473186985,
+            -0.00638023587296763,
+            -0.004023798021748334,
+            0.002170545188776296,
+            0.001785855390676113,
+            -0.005831534307818096,
+            0.008643501093778434,
+            -0.017602787281866816,
+            -0.02184792441891879,
+            -0.0052215705748838905,
+            -0.021252169339074325,
+            -0.008509621422071373,
+            -0.013000137524907565,
+            0.002893880203681282,
+            -0.015821609145923173,
+            0.010397692426614709,
+            -0.01638916243796692,
+            -0.010283706937701179,
+            -0.01103621454761223,
+            -0.00373552043529251,
+            0.01563553822905027,
+            -0.03188533919653949,
+            -0.004873550301425776,
+            -0.005183590054152726,
+            0.018218971950313273,
+            0.006004328765748325,
+            0.004357734370247252,
+            -0.014230318971458851,
+            0.0064792070801456235,
+            0.0215589173714926,
+            0.0055272493764401305,
+            0.014643335833970546,
+            0.010618070641357003,
+            -0.0022177303813575846,
+            -0.0024402761885758007,
+            0.017541799738501007,
+            -0.005445774414228888,
+            -0.00023502936119626386,
+            -0.00537569046685066,
+            -0.005974703247446444,
+            -0.010712244203425036,
+            0.01843613166635215,
+            -0.009938564643456672,
+            -0.017388592763277214,
+            -0.0015797412875366156,
+            0.02028434083164951,
+            -0.012761534382003973,
+            0.0031906812114950982,
+            0.000788436210275669,
+            0.020589196433948156,
+            0.0007726855314648301,
+            -0.005481260996289263,
+            0.007165014826294827,
+            -0.02956334397512344,
+            0.004095715232997753,
+            0.0003568512047626187,
+            0.027290768799658912,
+            -0.0017631129386615335,
+            0.01552542586826711,
+            -0.020871286597113327,
+            0.004388125824324094,
+            -0.016926008631435403,
+            0.005099899758322398,
+            0.012560923968841436,
+            0.002781049890810361,
+            0.010889021753593997,
+            -0.008500358897706648,
+            -0.021353460542554444,
+            0.012663934009161902,
+            -0.018657094130860155,
+            -0.029156017587920958,
+            0.0018889388761887053,
+            -0.028158076315864772,
+            -0.003048854609750186,
+            0.006812072971079798,
+            -0.035217412591558014,
+            -0.02251675180883659,
+            0.007396524512996277,
+            -0.00995571600201846,
+            -0.011912753666073882,
+            0.010675240017836214,
+            -0.02645155743041798,
+            -0.018137648527851668,
+            0.0010666944556449141,
+            0.012588800077165337,
+            -0.004561825939541096,
+            0.007555558090958502,
+            -0.028558603070876605,
+            0.014816816621445352,
+            -0.01313764261744744,
+            0.016036065063831813,
+            -0.0037797584923144134,
+            0.022217516257304868,
+            0.011131597210722573,
+            0.008795376259031849,
+            -0.004950080558710245,
+            0.021409590863550987,
+            0.005574351961953293,
+            0.051358937509801736,
+            0.031972009383746405,
+            0.0019821003238211276,
+            -0.003251257665378234,
+            0.003313045083311649,
+            0.008294776423730887,
+            -0.00940948257229233,
+            0.005114631360856872,
+            0.028053227377164806,
+            0.016738013668710083,
+            -0.018755924318109216,
+            0.006606711923035118,
+            -0.02702692774986498,
+            0.017950884766781056,
+            -0.022054653845163,
+            0.0003755944703013671,
+            0.007766253634286795,
+            0.0016206993755471613,
+            0.010682347575070117,
+            0.0013842930079185415,
+            0.01705234255261425,
+            0.023975191051210525,
+            -0.002976402980885077,
+            0.017466027107477605,
+            0.012734909275073793,
+            0.018143161773162015,
+            -0.0038399373197265827,
+            0.006929440921265761,
+            -0.012003095293274786,
+            -0.0008883470597690258,
+            0.0020490527756307267,
+            -0.004231091383919565,
+            0.012128848448049833,
+            0.008340302519160299,
+            0.012989849936937843,
+            0.002974488532907779,
+            0.003198854381461613,
+            -0.005309753760578859,
+            -0.0017024982049936384,
+            -0.01763395035509857,
+            -0.01819680226734474,
+            0.028417724274372468,
+            -0.020217879775485576,
+            -0.050118184195774004,
+            0.0025509548991664425,
+            0.014185761006969567,
+            -0.0012021669304388173,
+            -0.0051260576846881515,
+            -0.0005774778872927327,
+            0.005718864494530703,
+            -0.016918749210038116,
+            -0.0015472638835434902,
+            0.008125845907627804,
+            0.020347547062899797,
+            -0.002036218493206914,
+            -0.01079739518866248,
+            0.0038112958713600906,
+            0.02169339303105242,
+            0.01667594322285835,
+            0.012338193092567552,
+            -0.009798409233733167,
+            0.0003198215558702285,
+            -1.6308740825163337e-05,
+            0.0076870827874737325,
+            0.004356644356656038,
+            0.003500023005601613,
+            -0.019968549869439228,
+            -0.008634740687755112,
+            0.012781565963545847,
+            -0.0025637758121512927,
+            0.010243567793632335,
+            -0.00042296044205132426,
+            -0.003069251553118065,
+            -0.009344199449486244,
+            -0.004694953215957636,
+            -0.009583993744231791,
+            0.013783878036521561,
+            -0.007705982042252044,
+            -0.01549567538665377,
+            0.00020666298495823114,
+            0.0005515775451348472,
+            0.001667367278489879,
+            -0.004518431526236256,
+            0.03302324454129141,
+            0.009531990824506103,
+            -0.0004937348108340977,
+            -0.004047613286672229,
+            -0.0327383112522095,
+            0.012456150771224677,
+            -0.009677533214778067,
+            0.0006455811519981003,
+            0.020012976827426605,
+            -0.00020277620481825933,
+            0.002168432791818704,
+            -0.015657551629104767,
+            0.0032924357559238502,
+            0.006161295735841883,
+            0.013814165416788555,
+            0.0005164426482972658,
+            0.01076512534105537,
+            -0.006498977875968572,
+            -0.010325144017898257,
+            0.004804158067334203,
+            0.0104064855510473,
+            -0.0012188100892986933,
+            0.005171197790296134
+          ],
+          "title": "Mixture-Of-Experts With Expert Choice Routing",
+          "keyphrases": [
+            "Mixture-Of-Experts",
+            "Expert Choice Routing",
+            "sparsely-activated models",
+            "training efficiency",
+            "downstream performance"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "dea84165-aef9-4273-9988-e3d233346670",
+        "properties": {
+          "page_content": "A Comparison On Fine-Tuning With A Dense Model\n\nOur 8B MoE model achieves stronger pre-training perplexity than its dense counterpart. However, a better perplexity does not always directly translate to downstream performance as demonstrated in Section 4.4. To this end, we compare fine-tuning performance of the 8B dense model and MoE model in Table 1. As shown in the table, our MoE model using expert choice routing consistently outperforms the dense model across the 11 tasks in GLUE and SuperGLUE.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 1: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\n",
+          "title": "A Comparison On Fine-Tuning With A Dense Model",
+          "embedding": [
+            0.016482073813676834,
+            -0.002563994377851486,
+            -0.0267450250685215,
+            -0.023007972165942192,
+            0.025922315195202827,
+            0.018462155014276505,
+            -0.01591036096215248,
+            0.03547411039471626,
+            -0.012068727053701878,
+            -0.011176296509802341,
+            0.003149651922285557,
+            -0.008659363724291325,
+            -0.021000003442168236,
+            0.016844624653458595,
+            -0.02533665858209133,
+            0.032490044832229614,
+            0.024039844051003456,
+            0.01954980380833149,
+            0.010472113266587257,
+            -0.021557772532105446,
+            -0.0034128492698073387,
+            -0.011873507872223854,
+            -0.011252989992499352,
+            -0.022938251495361328,
+            0.01864342950284481,
+            -0.007376495283097029,
+            0.017639445140957832,
+            -0.010311754420399666,
+            -0.029952196404337883,
+            0.032490044832229614,
+            0.012884464114904404,
+            0.014557771384716034,
+            0.02639641799032688,
+            -0.006728088948875666,
+            -0.05795220285654068,
+            -0.010625500231981277,
+            0.0629163458943367,
+            0.01878287084400654,
+            -0.027177294716238976,
+            0.06062949076294899,
+            0.024179287254810333,
+            0.018978090956807137,
+            -0.004779383074492216,
+            -0.004329681862145662,
+            0.01492032129317522,
+            -0.030314745381474495,
+            -0.018476098775863647,
+            -0.0148645443841815,
+            0.03494422882795334,
+            -0.03650598227977753,
+            -0.010472113266587257,
+            -0.01666335016489029,
+            0.018657373264431953,
+            0.011469125747680664,
+            -0.0498366616666317,
+            0.0530996099114418,
+            0.006975598633289337,
+            0.01773705519735813,
+            0.04239044710993767,
+            0.005661355331540108,
+            -0.007543826010078192,
+            -0.0029038849752396345,
+            -0.026758968830108643,
+            -0.02917131967842579,
+            0.011315738782286644,
+            0.015966137871146202,
+            -0.01385358814150095,
+            -0.02132072113454342,
+            0.03890438750386238,
+            0.01121812965720892,
+            -0.05678088590502739,
+            0.05452192202210426,
+            -0.025239048525691032,
+            0.018099604174494743,
+            -0.009293826296925545,
+            -0.03438645973801613,
+            0.014320719987154007,
+            0.01945219375193119,
+            0.022770920768380165,
+            0.0009926545899361372,
+            -0.012201197445392609,
+            -0.010939245112240314,
+            -0.0014963897410780191,
+            0.009684264659881592,
+            0.03611554577946663,
+            -0.026758968830108643,
+            -0.017848609015345573,
+            -0.05254184082150459,
+            -0.02526693604886532,
+            -0.005563745740801096,
+            -0.006602590903639793,
+            -0.010318727232515812,
+            -0.011308766901493073,
+            0.020693231374025345,
+            0.007257969584316015,
+            0.042920324951410294,
+            -0.004256474785506725,
+            0.01883864775300026,
+            -0.02076295204460621,
+            -0.008903387933969498,
+            0.00044708672794513404,
+            0.03296414762735367,
+            0.018252991139888763,
+            0.004782869480550289,
+            0.04950200021266937,
+            -0.015394425019621849,
+            0.020288847386837006,
+            0.007181276101619005,
+            0.0038242037408053875,
+            0.06832670420408249,
+            -0.005229084752500057,
+            -0.021892433986067772,
+            -0.03126295283436775,
+            0.008854582905769348,
+            -0.014794823713600636,
+            -0.00372310820966959,
+            -0.02597809210419655,
+            -0.011713149026036263,
+            0.04013147950172424,
+            -0.027637455612421036,
+            0.0076762959361076355,
+            0.027972117066383362,
+            0.019033867865800858,
+            -0.03313148021697998,
+            -0.03466534614562988,
+            0.045011959969997406,
+            0.02066534198820591,
+            0.0014536854578182101,
+            0.008087650872766972,
+            -0.014237054623663425,
+            -0.001226220279932022,
+            0.012473110109567642,
+            -0.014188249595463276,
+            -0.02958964742720127,
+            0.030454188585281372,
+            0.0055742040276527405,
+            0.0256852637976408,
+            0.03028685785830021,
+            -0.03555777668952942,
+            -0.03354980796575546,
+            0.019745023921132088,
+            -0.05968128517270088,
+            0.016440242528915405,
+            0.011992033571004868,
+            -0.004497012589126825,
+            0.010904384776949883,
+            0.0532948300242424,
+            0.08717929571866989,
+            -0.013016934506595135,
+            -0.004096116404980421,
+            -0.02710757404565811,
+            0.027065740898251534,
+            0.004817729815840721,
+            0.036394428461790085,
+            -0.007495020981878042,
+            -0.010953188873827457,
+            -0.02603386901319027,
+            0.0032385464292019606,
+            -0.013477093540132046,
+            -0.02132072113454342,
+            0.011280878446996212,
+            0.008241036906838417,
+            -0.004657371435314417,
+            -0.008310758508741856,
+            0.0006462276796810329,
+            0.03182072192430496,
+            0.0277071762830019,
+            -0.007313746493309736,
+            -0.07591236382722855,
+            0.031681280583143234,
+            0.011057770811021328,
+            0.030454188585281372,
+            0.0072161369025707245,
+            0.020037852227687836,
+            0.019061755388975143,
+            0.02533665858209133,
+            0.006578188389539719,
+            -0.014976098202168941,
+            -0.030482076108455658,
+            0.024374505504965782,
+            -0.01294721383601427,
+            -0.027986060827970505,
+            0.041916340589523315,
+            -0.01662151701748371,
+            0.013023906387388706,
+            -0.0030241538770496845,
+            0.010855579748749733,
+            0.025294825434684753,
+            -0.001602714415639639,
+            0.017848609015345573,
+            -0.00970518123358488,
+            0.045820724219083786,
+            0.03028685785830021,
+            -0.01878287084400654,
+            -0.007780877873301506,
+            -0.039992038160562515,
+            -0.024025900289416313,
+            0.01833665743470192,
+            0.01068824902176857,
+            0.017346616834402084,
+            -0.005978586617857218,
+            -0.03592032566666603,
+            -0.006846614647656679,
+            -0.021460162475705147,
+            0.02320319041609764,
+            -0.0015730829909443855,
+            0.0327131524682045,
+            -0.0027417833916842937,
+            -0.003228088142350316,
+            -0.016175301745533943,
+            0.024388451129198074,
+            -0.023998012766242027,
+            -0.020177293568849564,
+            -0.021111557260155678,
+            -0.01812749356031418,
+            0.022715143859386444,
+            0.004165837541222572,
+            -0.002523904899135232,
+            -0.022450203076004982,
+            -0.0008218378061428666,
+            0.008729085326194763,
+            0.028557773679494858,
+            -0.00485259061679244,
+            -0.022464146837592125,
+            -0.05692032724618912,
+            -0.03388446941971779,
+            0.021111557260155678,
+            -0.02998008392751217,
+            0.022603590041399002,
+            -0.012668329291045666,
+            -0.022464146837592125,
+            0.009698208421468735,
+            0.003107819240540266,
+            0.02350996434688568,
+            -0.006414343602955341,
+            -0.02526693604886532,
+            0.018211157992482185,
+            -0.020274903625249863,
+            0.013260958716273308,
+            0.029254985973238945,
+            -0.017262950539588928,
+            0.014111556112766266,
+            -0.03525100275874138,
+            -0.01833665743470192,
+            0.03937849402427673,
+            -0.010486057959496975,
+            -0.006825698539614677,
+            -0.0072509972378611565,
+            0.006532869767397642,
+            0.0019382473547011614,
+            0.005790339782834053,
+            -0.037760961800813675,
+            0.0075089652091264725,
+            0.009795818477869034,
+            0.023523908108472824,
+            0.01970319077372551,
+            0.05895618721842766,
+            -0.04685259610414505,
+            -0.01579880714416504,
+            -0.03965737670660019,
+            0.05374104529619217,
+            0.010297810658812523,
+            -0.03965737670660019,
+            0.015868527814745903,
+            -0.020651398226618767,
+            -0.01768127828836441,
+            0.003187998663634062,
+            0.009942232631146908,
+            -0.014850599691271782,
+            0.033159367740154266,
+            0.01281474344432354,
+            -0.021599605679512024,
+            -0.04947411268949509,
+            0.0050024907104671,
+            0.004908367525786161,
+            -0.019466139376163483,
+            0.010841635055840015,
+            0.004577192012220621,
+            0.0018894425593316555,
+            0.0021927293855696917,
+            0.023677295073866844,
+            0.03257371112704277,
+            0.023886458948254585,
+            0.02978486567735672,
+            -0.006905877497047186,
+            0.0050756982527673244,
+            -0.02279880829155445,
+            -0.01053486205637455,
+            -0.03053785301744938,
+            -0.0037265941500663757,
+            -0.009342631325125694,
+            -0.020944226533174515,
+            -0.024695223197340965,
+            -0.005731076467782259,
+            -0.006975598633289337,
+            0.007578686345368624,
+            -0.010827691294252872,
+            0.005399901419878006,
+            0.04952988773584366,
+            -0.03561355173587799,
+            -0.043478094041347504,
+            -0.007045319769531488,
+            0.02031673677265644,
+            0.002699950709939003,
+            -0.05951395258307457,
+            0.008854582905769348,
+            0.018350601196289062,
+            0.0024506975896656513,
+            -0.02405378967523575,
+            -0.02391434647142887,
+            0.026117533445358276,
+            -0.008729085326194763,
+            -0.013023906387388706,
+            -0.053490050137043,
+            0.0010527890408411622,
+            0.005856574513018131,
+            0.0057973116636276245,
+            0.03184860944747925,
+            -0.03873705863952637,
+            -0.004577192012220621,
+            0.01949402689933777,
+            0.013937253504991531,
+            -0.07686056941747665,
+            0.014627492986619473,
+            -0.004148406907916069,
+            0.027874507009983063,
+            0.045458175241947174,
+            0.039183273911476135,
+            -0.01941036246716976,
+            -6.231325824046507e-05,
+            0.0032559765968471766,
+            -0.0519561842083931,
+            -0.029645422473549843,
+            0.007076694630086422,
+            0.006240040995180607,
+            -0.034776899963617325,
+            -0.0442589707672596,
+            -0.0073067741468548775,
+            -0.02589442767202854,
+            0.013637452386319637,
+            -0.011685260571539402,
+            0.009907372295856476,
+            -0.034776899963617325,
+            0.010095619596540928,
+            0.018824703991413116,
+            -0.009670319966971874,
+            -0.026354586705565453,
+            -0.03572510555386543,
+            0.01180378720164299,
+            -0.017095619812607765,
+            0.0007935136090964079,
+            0.021348608657717705,
+            0.006473606918007135,
+            0.005654383450746536,
+            -0.007515937555581331,
+            -0.03564143925905228,
+            -0.006836156360805035,
+            -0.005201196298003197,
+            0.020400401204824448,
+            0.0342470183968544,
+            -0.011176296509802341,
+            -0.009021913632750511,
+            -0.011036854237318039,
+            0.04741036519408226,
+            -0.05148208141326904,
+            -0.009265937842428684,
+            0.01864342950284481,
+            -0.006484064739197493,
+            0.008143427781760693,
+            0.005086156073957682,
+            -0.026675302535295486,
+            0.02461155876517296,
+            0.01756972447037697,
+            0.0110647426918149,
+            0.028529886156320572,
+            0.00459810858592391,
+            0.030565742403268814,
+            -0.002150896703824401,
+            -0.03064940683543682,
+            0.014669325202703476,
+            -0.020400401204824448,
+            0.02735857106745243,
+            -0.015784863382577896,
+            -0.004960658028721809,
+            -0.008401395753026009,
+            0.032490044832229614,
+            0.01812749356031418,
+            0.005263945087790489,
+            -0.04149801656603813,
+            0.016705181449651718,
+            0.009161355905234814,
+            -0.025852594524621964,
+            0.009516933932900429,
+            0.009565738961100578,
+            0.0181832704693079,
+            0.01777888834476471,
+            -0.023217136040329933,
+            -0.0037056778091937304,
+            0.00459810858592391,
+            -0.013365539722144604,
+            -0.022701198235154152,
+            0.024597613140940666,
+            0.003393675899133086,
+            -0.01849004253745079,
+            0.003433765610679984,
+            0.016119524836540222,
+            0.017918329685926437,
+            0.0071220132522284985,
+            0.008234065026044846,
+            -0.04052191972732544,
+            -0.0022188748698681593,
+            -0.01960558071732521,
+            -0.005361554678529501,
+            0.017792832106351852,
+            0.0275119561702013,
+            0.05123108625411987,
+            0.00025666088913567364,
+            0.03321514651179314,
+            -0.0030642435885965824,
+            0.0009246764821000397,
+            -0.036087654531002045,
+            0.011287850327789783,
+            0.03140239417552948,
+            0.01526892650872469,
+            0.028027893975377083,
+            0.030733073130249977,
+            -0.029338650405406952,
+            0.010262950323522091,
+            0.02005179598927498,
+            0.008373507298529148,
+            0.021460162475705147,
+            -0.005542829632759094,
+            0.034721121191978455,
+            -0.009698208421468735,
+            0.007257969584316015,
+            -0.011985061690211296,
+            -0.017039842903614044,
+            0.022729087620973587,
+            -0.03985259681940079,
+            -0.0009752242476679385,
+            0.022659366950392723,
+            -0.021794823929667473,
+            0.001286354847252369,
+            -0.01939641684293747,
+            -0.0075298817828297615,
+            0.009586654603481293,
+            -0.03586454689502716,
+            0.01934063993394375,
+            -0.021446218714118004,
+            -0.014878488145768642,
+            -0.039183273911476135,
+            -0.0016332174418494105,
+            -0.06866136938333511,
+            8.072399214142933e-05,
+            0.031625501811504364,
+            -0.0016846366925165057,
+            0.023691238835453987,
+            0.007006973493844271,
+            0.014132472686469555,
+            -0.055386465042829514,
+            -0.0040473113767802715,
+            -0.0073625510558485985,
+            0.014432273805141449,
+            -0.009028886444866657,
+            -0.03299203887581825,
+            -0.011922312900424004,
+            -0.0023374008014798164,
+            0.004071713890880346,
+            0.0032472615130245686,
+            0.007599602919071913,
+            -0.029756976291537285,
+            -0.013330679386854172,
+            -0.03854183852672577,
+            -0.010262950323522091,
+            -0.004490040708333254,
+            0.038932278752326965,
+            0.0009220619103871286,
+            -0.03976893052458763,
+            0.008750001899898052,
+            0.021948210895061493,
+            0.00022212715703062713,
+            -0.019745023921132088,
+            0.010423308238387108,
+            -0.042278893291950226,
+            0.02487649768590927,
+            -0.012396416626870632,
+            0.019187254831194878,
+            -0.0077529894188046455,
+            0.014223109930753708,
+            -0.0075647421181201935,
+            -0.009691236540675163,
+            -0.009175300598144531,
+            0.012326695024967194,
+            -0.027930283918976784,
+            -0.009454185143113136,
+            -0.03115139901638031,
+            0.009231077507138252,
+            -0.005284861661493778,
+            0.008896416053175926,
+            0.02795817144215107,
+            -0.004960658028721809,
+            -0.005005977116525173,
+            0.003339641960337758,
+            0.038011956959962845,
+            0.004730578511953354,
+            0.006041335873305798,
+            0.027037853375077248,
+            0.04423108324408531,
+            -0.00539641547948122,
+            -0.00426693307235837,
+            0.02486255392432213,
+            -0.010214145295321941,
+            0.016035858541727066,
+            -0.0133237075060606,
+            -0.006267929449677467,
+            0.027637455612421036,
+            -0.026159366592764854,
+            0.04280877113342285,
+            -0.008519921451807022,
+            0.017541835084557533,
+            -0.025964148342609406,
+            -0.01733267307281494,
+            -0.003034612163901329,
+            0.03686853125691414,
+            -0.000990040018223226,
+            0.006226096767932177,
+            0.017500003799796104,
+            0.0028411359526216984,
+            0.03742630034685135,
+            -0.02861355058848858,
+            -0.04473307356238365,
+            0.012215141206979752,
+            0.02547609992325306,
+            -0.008896416053175926,
+            -0.04094024747610092,
+            -0.020860562101006508,
+            -0.004908367525786161,
+            0.007766933646053076,
+            -0.03276893123984337,
+            -0.0037126499228179455,
+            0.014557771384716034,
+            0.0040089646354317665,
+            0.010848607867956161,
+            0.00960059929639101,
+            0.014153389260172844,
+            0.02851594053208828,
+            0.006755977403372526,
+            -0.03865339234471321,
+            0.0037126499228179455,
+            0.018880480900406837,
+            -0.014237054623663425,
+            -0.011497014202177525,
+            -0.02653586119413376,
+            0.0009717381908558309,
+            -0.009077690541744232,
+            -0.029756976291537285,
+            -0.013149404898285866,
+            0.006156375631690025,
+            -0.013205181807279587,
+            -0.012005978263914585,
+            -0.01853187568485737,
+            0.018559763208031654,
+            -0.002811504527926445,
+            0.01681673526763916,
+            -0.010681277140974998,
+            -0.013491038233041763,
+            0.014027890749275684,
+            0.021794823929667473,
+            -0.04225100204348564,
+            0.0009682521340437233,
+            -0.015812750905752182,
+            0.02734462544322014,
+            -0.005061754025518894,
+            0.01172709371894598,
+            0.012772910296916962,
+            -0.009398408234119415,
+            0.01180378720164299,
+            -0.01950797066092491,
+            -0.024346617981791496,
+            0.01127390656620264,
+            0.030203191563487053,
+            -0.01676095835864544,
+            -0.010276894085109234,
+            0.011741037480533123,
+            0.025615543127059937,
+            -0.023774905130267143,
+            -0.03349402919411659,
+            -0.030816737562417984,
+            0.03151394799351692,
+            -0.007843626663088799,
+            0.014188249595463276,
+            0.027623509988188744,
+            -0.0002780130016617477,
+            -0.03622709959745407,
+            -0.011176296509802341,
+            -0.03182072192430496,
+            -0.01945219375193119,
+            0.011343627236783504,
+            0.009426296688616276,
+            -0.03207171708345413,
+            -0.003395418869331479,
+            0.01839243434369564,
+            0.029756976291537285,
+            0.013421316631138325,
+            0.008296813815832138,
+            0.004503984935581684,
+            0.038876499980688095,
+            0.027023909613490105,
+            -0.003834662027657032,
+            0.0009926545899361372,
+            0.006288845557719469,
+            0.007836654782295227,
+            -0.017764942720532417,
+            0.0006161604542285204,
+            -0.010276894085109234,
+            -0.0008305529481731355,
+            -0.019480083137750626,
+            -0.006264443509280682,
+            0.00691633578389883,
+            -0.007202192675322294,
+            0.05968128517270088,
+            -0.0037928293459117413,
+            0.03449801355600357,
+            -0.018768927082419395,
+            0.026019925251603127,
+            0.04264144226908684,
+            -0.01169920526444912,
+            -0.02903187833726406,
+            -0.010395419783890247,
+            -0.026075702160596848,
+            -0.04492829367518425,
+            0.009928288869559765,
+            -0.015994027256965637,
+            -0.022784864529967308,
+            -0.011427292600274086,
+            -0.014069723896682262,
+            0.002382719423621893,
+            0.012535858899354935,
+            -0.012250002473592758,
+            0.001576569047756493,
+            -0.00861055962741375,
+            0.03301992639899254,
+            -0.0013456178130581975,
+            0.011866535991430283,
+            0.006470120511949062,
+            0.003243775339797139,
+            -0.012605579569935799,
+            -0.0037196220364421606,
+            0.014188249595463276,
+            -0.00018835598893929273,
+            0.014195221476256847,
+            -0.001346489298157394,
+            0.019466139376163483,
+            0.03513944894075394,
+            -0.04046614095568657,
+            -0.027023909613490105,
+            -0.006250499282032251,
+            -0.04955777898430824,
+            0.01650996319949627,
+            0.00397759024053812,
+            0.011587651446461678,
+            0.00866633653640747,
+            0.008875499479472637,
+            -0.03140239417552948,
+            0.024374505504965782,
+            -0.011071715503931046,
+            -0.015213149599730968,
+            0.022296816110610962,
+            -0.0026250004302710295,
+            0.012487053871154785,
+            0.02102789282798767,
+            0.0008022287511266768,
+            -0.041107576340436935,
+            -0.0033117535058408976,
+            0.01874103955924511,
+            -0.028195224702358246,
+            -0.00014674117846880108,
+            0.009035858325660229,
+            0.00034076202427968383,
+            0.0032559765968471766,
+            -0.013163348659873009,
+            0.0034965146332979202,
+            -0.012222114019095898,
+            -0.043478094041347504,
+            0.026061756536364555,
+            -0.002492530271410942,
+            -0.03165339305996895,
+            0.0363665409386158,
+            -0.019173309206962585,
+            -0.009238049387931824,
+            0.008791834115982056,
+            -0.03988048434257507,
+            -0.032796818763017654,
+            0.01677490398287773,
+            -0.017346616834402084,
+            0.009886455722153187,
+            -0.04077291488647461,
+            0.006686256267130375,
+            0.013644424267113209,
+            -0.0008780504576861858,
+            -0.011399404145777225,
+            0.04163745790719986,
+            -0.022980082780122757,
+            -0.004368028603494167,
+            0.0128426318988204,
+            -0.027121517807245255,
+            0.004043825436383486,
+            -0.0054277898743748665,
+            -0.015394425019621849,
+            0.009865539148449898,
+            -0.021111557260155678,
+            0.018964147195219994,
+            0.02330080047249794,
+            0.0016175301279872656,
+            0.022213151678442955,
+            -0.011615539900958538,
+            -0.0015417084796354175,
+            0.019005978479981422,
+            -0.005473108496516943,
+            0.047382477670907974,
+            0.0022816238924860954,
+            0.030760960653424263,
+            0.013337651267647743,
+            -0.024806777015328407,
+            0.0023426299449056387,
+            0.023886458948254585,
+            0.004622510634362698,
+            0.0055219135247170925,
+            0.008122511208057404,
+            0.007209164556115866,
+            -0.02101394720375538,
+            0.0024071219377219677,
+            -0.03600399196147919,
+            0.006030877586454153,
+            -0.0008009214652702212,
+            -0.018225103616714478,
+            -0.008331675082445145,
+            0.02872510440647602,
+            0.025545820593833923,
+            -0.01849004253745079,
+            0.02268725447356701,
+            -0.012731078080832958,
+            -0.002875996520742774,
+            0.011427292600274086,
+            0.002492530271410942,
+            0.023482074961066246,
+            -0.005256973206996918,
+            0.003545319428667426,
+            0.010674304328858852,
+            0.006410857662558556,
+            0.014501994475722313,
+            0.0014685012865811586,
+            0.00866633653640747,
+            0.00848506111651659,
+            -0.038876499980688095,
+            0.011824702844023705,
+            0.03580877184867859,
+            -0.004193725995719433,
+            -0.013281874358654022,
+            -0.01945219375193119,
+            -0.008220121264457703,
+            0.010255977511405945,
+            0.017667334526777267,
+            0.02091633901000023,
+            0.007718128617852926,
+            0.0057798814959824085,
+            -0.029115542769432068,
+            0.021111557260155678,
+            0.010276894085109234,
+            -0.01157370675355196,
+            -0.002255478408187628,
+            0.0025134466122835875,
+            0.04660160094499588,
+            -0.05845419317483902,
+            0.004859562497586012,
+            0.016551796346902847,
+            -0.0180856604129076,
+            -0.002550050150603056,
+            -0.02422112040221691,
+            -0.001696837949566543,
+            -0.004978088662028313,
+            0.02614542283117771,
+            -0.013107571750879288,
+            0.0021840143017470837,
+            -0.03555777668952942,
+            0.009203189052641392,
+            -0.001525149680674076,
+            -0.012417333200573921,
+            0.00012604272342287004,
+            0.002182271331548691,
+            -0.021334664896130562,
+            -0.0246255025267601,
+            0.0032507474534213543,
+            0.014069723896682262,
+            0.03215538337826729,
+            -0.006567730102688074,
+            0.0011983318254351616,
+            0.03714741766452789,
+            -0.0256852637976408,
+            -0.014683269895613194,
+            0.026061756536364555,
+            -0.005229084752500057,
+            0.03505578264594078,
+            0.02542032301425934,
+            -0.03120717592537403,
+            -0.05243028700351715,
+            -0.007264941465109587,
+            -0.011566734872758389,
+            0.01010956335812807,
+            0.009454185143113136,
+            0.032545823603868484,
+            -0.0001861772034317255,
+            -0.0030328689608722925,
+            -0.009691236540675163,
+            -0.01823904737830162,
+            0.034776899963617325,
+            0.00019924991647712886,
+            0.01743028126657009,
+            0.004514443222433329,
+            -0.018601596355438232,
+            -0.025699207559227943,
+            0.013037851080298424,
+            0.003547062398865819,
+            -0.005121016874909401,
+            -0.05212351307272911,
+            0.006982570979744196,
+            0.005054781679064035,
+            0.006173805799335241,
+            0.0377051867544651,
+            0.015994027256965637,
+            -0.0001385707437293604,
+            -0.0017822462832555175,
+            -0.0192430317401886,
+            0.029254985973238945,
+            -0.029729088768363,
+            0.04052191972732544,
+            0.0009011455695144832,
+            0.012326695024967194,
+            0.008791834115982056,
+            0.03221116214990616,
+            -0.017262950539588928,
+            -0.003604582278057933,
+            -0.02339841052889824,
+            -0.018420321866869926,
+            0.009154384024441242,
+            -0.020749008283019066,
+            0.0057066744193434715,
+            -0.007739045191556215,
+            0.0037649408914148808,
+            0.0038381479680538177,
+            0.008366535417735577,
+            0.05220717936754227,
+            0.026270920410752296,
+            0.0006104956264607608,
+            -0.024025900289416313,
+            -0.010813746601343155,
+            -0.016928289085626602,
+            0.005159363616257906,
+            -0.020498011261224747,
+            -0.007739045191556215,
+            -0.032099608331918716,
+            -0.014613548293709755,
+            -0.014529882930219173,
+            -0.025754984468221664,
+            0.00850597769021988,
+            0.014641436748206615,
+            -0.006424801889806986,
+            -0.011176296509802341,
+            -0.019591636955738068,
+            -0.033661361783742905,
+            -0.02081872895359993,
+            -0.009035858325660229,
+            -0.01430677529424429,
+            0.029254985973238945,
+            -0.0028324208687990904,
+            -0.006334164645522833,
+            -0.006623507011681795,
+            -0.030063750222325325,
+            0.018615540117025375,
+            -0.018266934901475906,
+            -0.00789940357208252,
+            -0.012012950144708157,
+            0.02741434797644615,
+            0.0028637952636927366,
+            0.046573713421821594,
+            0.002319970401003957,
+            -0.014209166169166565,
+            -0.05190040543675423,
+            -0.03491634130477905,
+            -0.02947809360921383,
+            0.016886457800865173,
+            0.0032071718014776707,
+            0.0010702193249017,
+            0.002781872870400548,
+            0.006410857662558556,
+            0.002318227430805564,
+            -0.0056508975103497505,
+            -0.029896419495344162,
+            0.0032298313453793526,
+            0.010388447903096676,
+            0.005009463056921959,
+            -0.035083670169115067,
+            -0.021083669736981392,
+            -0.018364544957876205,
+            0.038876499980688095,
+            -0.03742630034685135,
+            0.03438645973801613,
+            -0.011510957963764668,
+            -0.01814143732190132,
+            0.004249502904713154,
+            -0.027037853375077248,
+            0.006121514830738306,
+            0.006292331963777542,
+            0.007578686345368624,
+            0.03600399196147919,
+            0.01837848871946335,
+            0.043840646743774414,
+            -0.0286693274974823,
+            0.03820717707276344,
+            -0.01326793059706688,
+            -0.0006126743974164128,
+            -0.0005874005146324635,
+            0.0038799806497991085,
+            0.02922709658741951,
+            -0.002659860998392105,
+            -0.0006985185318626463,
+            -0.0050931284204125404,
+            -0.004315737634897232,
+            -0.005309263709932566,
+            0.003034612163901329,
+            -0.025657374411821365,
+            0.012068727053701878,
+            0.024904387071728706,
+            -0.015338648110628128,
+            0.03628287464380264,
+            -0.004674801602959633,
+            0.03756574168801308,
+            -0.013511954806745052,
+            -0.027525901794433594,
+            0.027135463431477547,
+            -0.01570119708776474,
+            -0.013525898568332195,
+            0.005086156073957682,
+            0.0018092632526531816,
+            -0.0015643677907064557,
+            -0.02097211591899395,
+            0.024235064163804054,
+            0.02487649768590927,
+            0.003939243499189615,
+            0.0038102595135569572,
+            -0.02953387051820755,
+            -0.009984065778553486,
+            0.0181832704693079,
+            -0.004904881119728088,
+            0.020302793011069298,
+            0.02320319041609764,
+            -0.014118527993559837,
+            0.02193426713347435,
+            -0.01050697360187769,
+            0.016384465619921684,
+            0.022352593019604683,
+            -0.014069723896682262,
+            -0.016802791506052017,
+            -0.020274903625249863,
+            -0.004019422922283411,
+            0.004388945177197456,
+            -0.02284064143896103,
+            -0.03159761428833008,
+            0.0028463650960475206,
+            0.02958964742720127,
+            0.007108069024980068,
+            -0.03658964857459068,
+            -0.005738048814237118,
+            -0.017346616834402084,
+            0.030984068289399147,
+            -0.026521917432546616,
+            -0.0007778263534419239,
+            -0.010576695203781128,
+            -0.001526892650872469,
+            0.012005978263914585,
+            0.04088446870446205,
+            -0.0016088150441646576,
+            0.03095618076622486,
+            0.0171235091984272,
+            0.005145419389009476,
+            -0.0077320728451013565,
+            0.03525100275874138,
+            -0.016035858541727066,
+            -0.022561756893992424,
+            0.031123511493206024,
+            -0.014557771384716034,
+            -0.007843626663088799,
+            0.018043827265501022,
+            0.015547811053693295,
+            -0.015812750905752182,
+            0.026884466409683228,
+            -0.0008057148079387844,
+            0.011476097628474236,
+            0.011789842508733273,
+            -0.0009682521340437233,
+            -0.013177293352782726,
+            -0.0286693274974823,
+            -0.023984067142009735,
+            0.01802988350391388,
+            -0.008861555717885494,
+            0.03246215730905533,
+            0.018754983320832253,
+            0.015492034144699574,
+            -0.03923904895782471,
+            0.0365617610514164,
+            -0.013253985904157162,
+            -0.004510956816375256,
+            0.0032472615130245686,
+            -0.020749008283019066,
+            -0.00024380606191698462,
+            -0.009593627415597439,
+            -0.03377291560173035,
+            -0.03321514651179314,
+            -0.02339841052889824,
+            0.0003054656845051795,
+            0.01162251178175211,
+            -0.00419023958966136,
+            -0.031039845198392868,
+            -0.023189246654510498,
+            0.025057774037122726,
+            0.02826494537293911,
+            -0.008959164842963219,
+            -0.026312753558158875,
+            -0.006731574889272451,
+            -0.019689247012138367,
+            0.020776895806193352,
+            -0.0008392680902034044,
+            0.004573706071823835,
+            0.021794823929667473,
+            0.01076494250446558,
+            -0.006581674329936504,
+            -0.007885459810495377,
+            -0.036952197551727295,
+            0.008540838025510311,
+            0.007285858038812876,
+            -0.0034633970353752375,
+            -0.006787351798266172,
+            0.013686257414519787,
+            0.0005233442061580718,
+            -0.009203189052641392,
+            -0.018252991139888763,
+            0.0011312252609059215,
+            -0.0009516933932900429,
+            0.01317032054066658,
+            -0.00043771794298663735,
+            0.014174304902553558,
+            0.016551796346902847,
+            -0.006009961012750864,
+            -0.005710160359740257,
+            0.008980081416666508,
+            0.04478885233402252,
+            0.02330080047249794,
+            -0.02613147906959057,
+            0.011838647536933422,
+            0.015882473438978195,
+            0.0029282872565090656,
+            -0.015826696529984474,
+            0.006940738297998905,
+            0.023886458948254585,
+            -0.031235065311193466,
+            -0.0027138949371874332,
+            -0.007397411856800318,
+            0.01701195538043976,
+            0.032350603491067886,
+            0.027428291738033295,
+            0.00019129733846057206,
+            -0.005748507101088762,
+            -0.00554631557315588,
+            0.019033867865800858,
+            0.012563747353851795,
+            0.018601596355438232,
+            -0.0071603599935770035,
+            -0.012354583479464054,
+            -0.026103589683771133,
+            0.008450200781226158,
+            -0.0026947215665131807,
+            -0.011531874537467957,
+            0.032545823603868484,
+            0.012556775473058224,
+            0.015826696529984474,
+            -0.012828687205910683,
+            -0.016788847744464874,
+            0.026661358773708344,
+            0.022199207916855812,
+            -0.025308769196271896,
+            -0.021739047020673752,
+            0.010325699113309383,
+            0.0017683020560070872,
+            -0.0029840641655027866,
+            -0.013553787022829056,
+            -0.022617533802986145,
+            0.028585663065314293,
+            0.09086057543754578,
+            -0.015045818872749805,
+            -0.01337251253426075,
+            -0.01733267307281494,
+            -0.023984067142009735,
+            -0.019926298409700394,
+            0.0005407744902186096,
+            0.006145917344838381,
+            0.03313148021697998,
+            0.009886455722153187,
+            -0.003914841450750828,
+            -0.005783367436379194,
+            -0.006135459057986736,
+            0.012054783292114735,
+            -0.031988054513931274,
+            0.02051195502281189,
+            0.003050299361348152,
+            0.016440242528915405,
+            0.01002589799463749,
+            -0.010702192783355713,
+            -0.00901494175195694,
+            0.013435261324048042,
+            0.03438645973801613,
+            0.004789841361343861,
+            -0.021752992644906044,
+            0.001578312017954886,
+            0.004873506724834442,
+            0.010493029840290546,
+            -0.003998506814241409,
+            -0.01844821125268936,
+            0.005487052723765373,
+            0.011594623327255249,
+            -0.003092132043093443,
+            0.0028986558318138123,
+            -0.00843625608831644,
+            0.05429881438612938,
+            -0.009182272478938103,
+            0.01317032054066658,
+            0.0041205184534192085,
+            -0.019368529319763184,
+            -0.00744621641933918,
+            0.022017931565642357,
+            -0.0040089646354317665,
+            -0.00643526017665863,
+            -0.012605579569935799,
+            0.034274905920028687,
+            0.031235065311193466,
+            -0.03486056253314018,
+            0.017541835084557533,
+            -0.009579682722687721,
+            -0.014055779203772545,
+            0.007467132527381182,
+            -0.00782271008938551,
+            0.014216138049960136,
+            0.004636454861611128,
+            0.004761952906847,
+            0.010918328538537025,
+            -0.01878287084400654,
+            -0.011441237293183804,
+            0.003656873246654868,
+            0.0007006973028182983,
+            0.017960162833333015,
+            0.016384465619921684,
+            0.01050697360187769,
+            -0.009795818477869034,
+            0.018364544957876205,
+            0.023370521143078804,
+            0.0019609066657721996,
+            0.003187998663634062,
+            0.0096215158700943,
+            0.011887452565133572,
+            -0.006961654406040907,
+            -0.008847611024975777,
+            -0.015338648110628128,
+            0.0076762959361076355,
+            0.01885259337723255,
+            -0.015505978837609291,
+            0.003221116028726101,
+            0.013637452386319637,
+            -0.04261355474591255,
+            0.0191454216837883,
+            0.010018926113843918,
+            0.013407372869551182,
+            0.01278685498982668,
+            -0.022561756893992424,
+            0.004625997040420771,
+            -0.013470121659338474,
+            -0.018169326707720757,
+            0.002300797263160348,
+            -0.01430677529424429,
+            0.012480081990361214,
+            -0.0020515441428869963,
+            -0.006334164645522833,
+            -0.018057772889733315,
+            0.005640439223498106,
+            -0.005507969297468662,
+            -0.0011809015413746238,
+            -0.000824888062197715,
+            -0.011239045299589634,
+            -0.023412354290485382,
+            -0.03717530518770218,
+            0.016844624653458595,
+            0.01109263114631176,
+            -0.006511953193694353,
+            -0.008896416053175926,
+            -0.013463149778544903,
+            0.018824703991413116,
+            -0.004448208026587963,
+            0.00529183354228735,
+            -0.0115458182990551,
+            0.0037684268318116665,
+            -0.005141932982951403,
+            0.006609562784433365,
+            -0.006428287830203772,
+            0.01013047993183136,
+            0.020233070477843285,
+            0.008129483088850975,
+            -0.0475219190120697,
+            -0.005365040618926287,
+            0.009363546967506409,
+            -0.025768928229808807,
+            0.0037509966641664505,
+            0.027930283918976784,
+            0.008338646963238716,
+            0.002250249497592449,
+            0.01218028087168932,
+            0.0115458182990551,
+            -0.020902393385767937,
+            0.014906376600265503,
+            0.007501993328332901,
+            0.00018890068167820573,
+            0.0064213159494102,
+            0.0019347612978890538,
+            -0.0004309637297410518,
+            0.021460162475705147,
+            -0.007913348264992237,
+            0.0061807781457901,
+            -0.020023908466100693,
+            0.026828689500689507,
+            -0.019382473081350327,
+            -0.0014571715146303177,
+            0.025964148342609406,
+            0.023468131199479103,
+            0.00473406445235014,
+            0.015129484236240387,
+            -0.02618725597858429,
+            0.006086654495447874,
+            0.03246215730905533,
+            0.018169326707720757,
+            0.029645422473549843,
+            -0.014948209747672081,
+            -0.022770920768380165,
+            0.014090639539062977,
+            -0.015826696529984474,
+            -0.006337650585919619,
+            -0.014076695777475834,
+            0.008366535417735577,
+            0.0008431898895651102,
+            0.03820717707276344,
+            0.008450200781226158,
+            -0.026061756536364555,
+            0.029673311859369278,
+            -0.005385957192629576,
+            -0.029115542769432068,
+            0.020442234352231026,
+            -0.009440240450203419,
+            0.009238049387931824,
+            0.004082172177731991,
+            -0.016886457800865173,
+            0.025141438469290733,
+            -0.010276894085109234,
+            -0.004225100390613079,
+            0.01889442466199398,
+            0.027874507009983063,
+            0.0035244030877947807,
+            -0.028348609805107117,
+            0.010374504141509533,
+            -0.004953686147928238,
+            -0.0002974041854031384,
+            -0.0070836665108799934,
+            0.012152392417192459,
+            -0.016537850722670555,
+            0.0013874504948034883,
+            0.006825698539614677,
+            -0.024653390049934387,
+            -0.008603586815297604,
+            -0.001669820980168879,
+            -0.014153389260172844,
+            -0.03896016627550125,
+            0.002949203597381711,
+            -0.006623507011681795,
+            0.003548805369064212,
+            -0.005208168178796768,
+            0.009537850506603718,
+            -0.017290839925408363,
+            0.015101595781743526,
+            0.008324702270328999,
+            -0.0029770920518785715,
+            -0.0069372523576021194,
+            -0.00797609705477953,
+            -0.029143432155251503,
+            0.037454187870025635,
+            0.00015545632049907,
+            -0.0036254986189305782,
+            0.005588148254901171,
+            -0.014195221476256847,
+            0.00343725155107677,
+            0.02724701724946499,
+            0.02406773343682289,
+            -0.043784867972135544,
+            0.001980080036446452,
+            -0.008840639144182205,
+            -0.021766936406493187,
+            0.013491038233041763,
+            -0.003914841450750828,
+            0.0171235091984272,
+            0.0053510963916778564,
+            -0.029729088768363,
+            -0.00030350478482432663,
+            0.010269922204315662,
+            -0.012500998564064503,
+            0.0023984068538993597,
+            0.024751000106334686,
+            -0.00036146046477369964,
+            0.0031043333001434803,
+            -0.009440240450203419,
+            0.02254781313240528,
+            -0.01662151701748371,
+            0.015938250347971916,
+            0.006215638481080532,
+            0.006375997327268124,
+            0.014376496896147728,
+            -0.02005179598927498,
+            0.019982075318694115,
+            0.0038242037408053875,
+            0.004437749739736319,
+            0.0057066744193434715,
+            -0.001433640718460083,
+            0.021906377747654915,
+            -0.0003137450839858502,
+            0.006630479358136654,
+            0.016830680891871452,
+            -0.021083669736981392,
+            0.011413348838686943,
+            -0.006930280011147261,
+            -0.00016743964806664735,
+            -0.0015791835030540824,
+            -0.021083669736981392,
+            -0.027428291738033295,
+            0.035334669053554535,
+            0.001134711317718029,
+            0.00016090329154394567,
+            0.008087650872766972,
+            -0.013393428176641464,
+            -0.030203191563487053,
+            -0.02168327011168003,
+            -0.002867281436920166,
+            0.01281474344432354,
+            0.01762550137937069,
+            -0.017709165811538696,
+            0.007766933646053076,
+            0.011315738782286644,
+            0.02445817179977894,
+            0.03407968580722809,
+            -0.0011983318254351616,
+            0.020484067499637604,
+            0.026466140523552895,
+            0.009321714751422405,
+            0.014976098202168941,
+            -0.008415340445935726,
+            -0.00901494175195694,
+            0.012473110109567642,
+            0.015408368781208992,
+            0.011789842508733273,
+            0.0019678787793964148,
+            -0.024653390049934387,
+            -0.009795818477869034,
+            -0.010262950323522091,
+            0.006463148631155491,
+            -0.005745020695030689,
+            -0.001980080036446452,
+            0.02299402840435505,
+            -0.003981076180934906,
+            -0.018001995980739594,
+            -0.019270919263362885,
+            -0.016956178471446037,
+            -0.01233366783708334,
+            -0.009405380114912987,
+            0.015603587962687016,
+            -0.00584960263222456,
+            -0.015896417200565338,
+            -0.024095620959997177,
+            0.011957173235714436,
+            0.02158566191792488,
+            0.02694024331867695,
+            0.0003128735697828233,
+            -0.004622510634362698,
+            0.01677490398287773,
+            0.0057415347546339035,
+            0.005867032799869776,
+            -0.008931276388466358,
+            -0.0009673806489445269,
+            0.011657372117042542,
+            0.025252992287278175,
+            -0.010897411964833736,
+            -0.019996019080281258,
+            -0.032043829560279846,
+            0.0056125507690012455,
+            0.0004745394107885659,
+            -0.006905877497047186,
+            0.008575698360800743,
+            0.0024890443310141563,
+            0.004608566407114267,
+            0.013574703596532345,
+            -0.006926794070750475,
+            0.009328686632215977,
+            -0.029673311859369278,
+            0.007746017072349787,
+            0.011531874537467957,
+            0.02953387051820755,
+            -0.05231873318552971,
+            -0.019173309206962585,
+            0.0007743402966298163,
+            0.046880487352609634,
+            0.01660757325589657,
+            0.013644424267113209,
+            -0.017555780708789825,
+            -0.011350599117577076,
+            -0.013790839351713657,
+            -0.011057770811021328,
+            -0.0211673341691494,
+            -0.010269922204315662,
+            0.020804785192012787,
+            -0.006337650585919619,
+            0.0421394482254982,
+            0.029812753200531006,
+            -0.004646913148462772,
+            -0.0010850350372493267,
+            -0.01448805071413517,
+            0.014669325202703476,
+            -0.009412351995706558,
+            0.03870917111635208,
+            0.004476096481084824,
+            0.012326695024967194,
+            -0.009021913632750511,
+            -0.00043771794298663735,
+            -0.0496414415538311,
+            -0.015743030235171318,
+            0.017095619812607765,
+            -0.03207171708345413,
+            0.00023116912052500993,
+            0.030370522290468216,
+            0.020679285749793053,
+            -0.00015338648518081754,
+            -0.01228486280888319,
+            0.02608964592218399,
+            0.025503989309072495,
+            0.007962152361869812,
+            0.002156125847250223,
+            -0.01752789132297039,
+            -0.007481076754629612,
+            -0.0056683276779949665,
+            -0.012250002473592758,
+            0.00782271008938551,
+            -0.027470124885439873,
+            0.021976100280880928,
+            -0.012807771563529968,
+            -0.019870521500706673,
+            0.029505981132388115,
+            -0.009049802087247372,
+            0.02187849022448063,
+            0.009565738961100578,
+            -0.0139023931697011,
+            0.03717530518770218,
+            0.011106575839221478,
+            0.011176296509802341,
+            -0.008903387933969498,
+            0.014627492986619473,
+            -0.012033866718411446,
+            0.028585663065314293,
+            -0.017667334526777267,
+            0.015882473438978195,
+            0.006773407571017742,
+            -0.00792729202657938,
+            0.00477589713409543,
+            0.031179288402199745,
+            0.012821715325117111,
+            -0.004908367525786161,
+            0.028139447793364525,
+            0.007941236719489098,
+            0.022073708474636078,
+            0.004768925253301859,
+            0.008220121264457703,
+            0.027930283918976784,
+            0.02786056324839592,
+            -0.010172312147915363,
+            -0.03184860944747925,
+            -0.007257969584316015,
+            -0.005009463056921959,
+            0.018768927082419395,
+            -0.018225103616714478,
+            -0.004472610540688038,
+            -0.0027365542482584715,
+            0.0128426318988204,
+            -0.008854582905769348,
+            0.010960161685943604,
+            0.0010649901814758778,
+            -0.032294824719429016,
+            -0.02406773343682289,
+            -0.03678486868739128,
+            0.0014231825480237603,
+            -0.021920323371887207,
+            -0.015687253326177597,
+            -6.411075446521863e-05,
+            -0.027037853375077248,
+            0.0013708916958421469,
+            0.027428291738033295,
+            -0.008359563536942005,
+            -0.012368528172373772,
+            0.005915837828069925,
+            0.014571716077625751,
+            -0.0057241045869886875,
+            -0.004005478695034981,
+            0.0029021420050412416,
+            -0.0005425175186246634,
+            4.28131315857172e-05,
+            0.010353587567806244,
+            -0.010221117176115513,
+            -0.036087654531002045,
+            -0.02668924815952778,
+            0.013435261324048042,
+            0.0019086158135905862,
+            0.00792729202657938,
+            0.013658368960022926,
+            0.003020667936652899,
+            0.0001337774156127125,
+            0.011768925935029984,
+            0.00047628243919461966,
+            0.003034612163901329,
+            -0.0013830928364768624,
+            0.019033867865800858,
+            0.008617531508207321,
+            -0.015519922599196434,
+            0.018978090956807137,
+            0.002586653921753168,
+            -0.005487052723765373,
+            -0.010332670994102955,
+            -0.004078685771673918,
+            -0.010995022021234035,
+            -0.022045820951461792,
+            0.005577689968049526,
+            0.0076414356008172035,
+            -0.012319723144173622,
+            0.00911952368915081,
+            -0.020637454465031624,
+            0.016830680891871452,
+            0.026814745739102364,
+            0.009335658513009548,
+            -0.030203191563487053,
+            -0.00563695328310132,
+            -0.012912352569401264,
+            -0.01286354847252369,
+            -0.019996019080281258,
+            0.004333168268203735,
+            0.019577693194150925,
+            0.008833667263388634,
+            0.0057798814959824085,
+            0.01716534234583378,
+            0.004036853089928627,
+            -0.010967133566737175,
+            0.0015512950485572219,
+            0.0005643054028041661,
+            -0.0058705187402665615,
+            -0.0182808805257082,
+            0.008526894263923168,
+            -0.001230577938258648,
+            -0.009293826296925545,
+            0.012187252752482891,
+            0.015254982747137547,
+            0.00655727181583643,
+            0.0024332674220204353,
+            0.0171235091984272,
+            -0.00011231638200115412,
+            -0.004368028603494167,
+            0.016272911801934242,
+            0.07898009568452835,
+            0.00919621717184782,
+            0.013707173988223076,
+            0.0021038351114839315,
+            -0.02045617811381817,
+            -0.014934265054762363,
+            -0.0037998014595359564,
+            -0.04255777597427368,
+            0.013484066352248192,
+            0.02005179598927498,
+            -0.004497012589126825,
+            -0.013163348659873009,
+            -0.012152392417192459,
+            -0.00866633653640747,
+            -0.001097236294299364,
+            0.03399602323770523,
+            -0.008185259997844696,
+            0.014808767475187778,
+            0.038318730890750885,
+            0.01878287084400654,
+            -0.006288845557719469,
+            0.00964940432459116,
+            0.025197215378284454,
+            -0.010437252931296825,
+            0.00477589713409543,
+            -0.004971116315573454,
+            0.005654383450746536,
+            0.011008965782821178,
+            0.010771914385259151,
+            0.01814143732190132,
+            0.010304782539606094,
+            -0.0021770421881228685,
+            -0.01950797066092491,
+            0.005511455237865448,
+            -0.011455181054770947,
+            0.013302790932357311,
+            -0.0030572714749723673,
+            -0.0012018178822472692,
+            -0.02629880979657173,
+            0.012905380688607693,
+            0.009461157023906708,
+            -0.013539843261241913,
+            -0.00949601735919714,
+            0.002987550338730216,
+            -0.007278885692358017,
+            0.014320719987154007,
+            -0.010409364476799965,
+            0.0105836670845747,
+            0.009056774899363518,
+            -0.005103586707264185,
+            -0.015380480326712132,
+            -0.001689865835942328,
+            -0.004890936892479658,
+            0.008094622753560543,
+            -0.010988050140440464,
+            -0.011992033571004868,
+            0.015394425019621849,
+            0.012263946235179901,
+            0.012556775473058224,
+            0.017109565436840057,
+            -0.012626496143639088,
+            0.008847611024975777,
+            -0.0028637952636927366,
+            -0.016454186290502548,
+            -0.002661603968590498,
+            -0.018517931923270226,
+            0.011315738782286644,
+            0.004908367525786161,
+            0.005476594436913729,
+            -0.010569723322987556,
+            0.00014347300748340786,
+            -0.006637451238930225,
+            0.008331675082445145,
+            -0.0035331181716173887,
+            -0.0016166586428880692,
+            -0.012096615508198738,
+            -0.010576695203781128,
+            -0.0035052297171205282,
+            0.02314741350710392,
+            0.011789842508733273,
+            0.0009577939636074007,
+            0.009530877694487572,
+            0.0072719138115644455,
+            -0.007031375542283058,
+            -0.013414344750344753,
+            0.002541335066780448,
+            -0.001469372771680355,
+            0.005996016785502434,
+            0.021348608657717705,
+            -0.010520918294787407,
+            -0.0071394434198737144,
+            0.0020811755675822496,
+            0.016747014597058296,
+            0.0058530885726213455,
+            0.0009325200808234513,
+            0.023468131199479103,
+            0.0003483877517282963,
+            0.0030572714749723673,
+            -0.01864342950284481,
+            -0.015394425019621849,
+            0.010346615687012672,
+            0.026675302535295486,
+            0.00863844808191061,
+            0.0010275151580572128,
+            -0.017193229869008064,
+            0.0026162853464484215,
+            0.027372514829039574,
+            0.026019925251603127,
+            0.007543826010078192,
+            -0.0027417833916842937,
+            0.0009037601412273943,
+            -0.010151396505534649,
+            -0.0001501182996435091,
+            0.028390442952513695,
+            -0.015185261145234108,
+            -0.0006593004218302667,
+            -0.005256973206996918,
+            -0.013281874358654022,
+            -0.017151396721601486,
+            0.00957271084189415,
+            0.027233071625232697,
+            0.006222610827535391,
+            0.010409364476799965,
+            -0.01995418593287468,
+            0.01334462407976389,
+            -0.009007969871163368,
+            -0.008282870054244995,
+            -0.009265937842428684,
+            -0.021599605679512024,
+            -0.014376496896147728,
+            -0.011036854237318039,
+            -0.0032228592317551374,
+            -0.014766935259103775,
+            -0.0002703872451093048,
+            0.002027141861617565,
+            0.005727590527385473,
+            -0.003156624035909772,
+            -0.0036080684512853622,
+            -0.0061807781457901,
+            0.010548806749284267,
+            -0.00429482152685523,
+            0.0037684268318116665,
+            -0.023942235857248306,
+            -0.026507971808314323,
+            0.01839243434369564,
+            -0.008422312326729298,
+            -0.00397759024053812,
+            -0.015003986656665802,
+            -0.004782869480550289,
+            -0.023691238835453987,
+            0.01802988350391388,
+            -0.023523908108472824,
+            -0.010590638965368271,
+            0.004381972830742598,
+            0.013511954806745052,
+            -0.016747014597058296,
+            -0.01532470341771841,
+            -0.0022711658384650946,
+            -0.01228486280888319,
+            0.004256474785506725,
+            -0.03380080312490463,
+            0.007578686345368624,
+            -0.000818351749330759,
+            0.012975101359188557,
+            -0.020735062658786774,
+            0.02491833083331585,
+            0.0033117535058408976,
+            -0.013009962625801563,
+            -0.008373507298529148,
+            -0.007223108783364296,
+            -0.0026563750579953194,
+            0.005912351422011852,
+            0.001685508294031024,
+            -0.014766935259103775,
+            0.02081872895359993,
+            0.0016959664644673467,
+            0.009586654603481293,
+            -0.01112051960080862,
+            0.00803187396377325,
+            0.011162352748215199,
+            0.005082670133560896,
+            -0.003649901133030653,
+            0.007648407481610775,
+            -0.0031513948924839497,
+            0.019368529319763184,
+            0.020637454465031624,
+            -0.006341136526316404,
+            0.008038845844566822,
+            0.004015936981886625,
+            -0.017193229869008064,
+            -0.007034861948341131,
+            0.00405428372323513,
+            0.0029126000590622425,
+            -0.018727093935012817,
+            0.025099605321884155,
+            0.027009963989257812,
+            0.001078934408724308,
+            -0.02522510476410389,
+            0.010925300419330597,
+            0.011001993902027607,
+            0.014529882930219173,
+            0.02147410809993744,
+            -0.002708665793761611,
+            -0.023175302892923355,
+            0.009607571177184582,
+            -0.008415340445935726,
+            -0.01920119859278202,
+            0.031430285423994064,
+            0.022003987804055214,
+            0.011127491481602192,
+            -0.05491235852241516,
+            0.003093875013291836,
+            0.01579880714416504,
+            -0.013609563931822777,
+            -0.02953387051820755,
+            -0.011420320719480515,
+            -0.009935260750353336,
+            0.0029091141186654568,
+            0.013909365050494671,
+            -0.015213149599730968,
+            -0.01752789132297039,
+            0.014934265054762363,
+            0.030928291380405426,
+            0.0275119561702013,
+            0.025057774037122726,
+            -0.023426298052072525,
+            0.001805777195841074,
+            -0.025113550946116447,
+            0.00640737172216177,
+            -0.016077691689133644,
+            -0.005867032799869776,
+            0.0123615562915802,
+            -0.007613547146320343,
+            0.020735062658786774,
+            0.002405378967523575,
+            0.02183665707707405,
+            0.017695222049951553,
+            0.03692431002855301,
+            -0.016593627631664276,
+            -0.030231080949306488,
+            -0.011329683475196362,
+            -0.014048807322978973,
+            0.023370521143078804,
+            0.01789044216275215,
+            0.00967729277908802,
+            0.028446219861507416,
+            0.02928287349641323,
+            -0.0020550303161144257,
+            0.00960059929639101,
+            -2.5587109121261165e-05,
+            8.219467417802662e-05,
+            -0.018113549798727036,
+            -0.02314741350710392,
+            -0.0021630979608744383,
+            0.00967729277908802,
+            0.013923308812081814,
+            -0.005605578422546387,
+            -0.0027278391644358635,
+            0.021097613498568535,
+            -0.0032629487104713917,
+            -0.0027905881870537996,
+            -0.03555777668952942,
+            0.0018842134159058332,
+            -0.008924304507672787,
+            0.021808769553899765,
+            -0.00020491474424488842,
+            -0.008094622753560543,
+            -0.027525901794433594,
+            -0.004918825346976519,
+            0.034721121191978455,
+            0.005950698163360357,
+            0.00012168515240773559,
+            0.011469125747680664,
+            0.024193231016397476,
+            0.010513946413993835,
+            0.016328688710927963,
+            -0.001948705525137484,
+            0.001576569047756493,
+            0.022784864529967308,
+            -0.006644423585385084,
+            0.02187849022448063,
+            0.003827689914032817,
+            -0.013044822961091995,
+            -0.0017866039415821433,
+            -0.0019609066657721996,
+            -0.008659363724291325,
+            0.009900400415062904,
+            0.015533867292106152,
+            -0.011810759082436562,
+            0.018406378105282784,
+            -0.016021914780139923,
+            -0.0012401645071804523,
+            -0.023970123380422592,
+            -0.0005242157494649291,
+            0.013651397079229355,
+            -0.00317928334698081,
+            0.004985060542821884,
+            -0.019521916285157204,
+            0.0057624513283371925,
+            0.014111556112766266,
+            -0.012675301171839237,
+            0.004761952906847,
+            -0.02158566191792488,
+            0.006138945464044809,
+            0.011531874537467957,
+            -0.001174801029264927,
+            -0.009412351995706558,
+            -0.005385957192629576,
+            0.01165040023624897,
+            -0.015882473438978195,
+            0.004971116315573454,
+            0.01797410659492016,
+            0.012054783292114735,
+            0.010018926113843918,
+            0.006860558874905109,
+            0.00911952368915081,
+            0.008443228900432587,
+            0.012194225564599037,
+            -0.036952197551727295,
+            0.018071716651320457,
+            -0.01430677529424429,
+            0.009440240450203419,
+            -0.0017552294302731752,
+            -0.007829682901501656,
+            -0.008171316236257553,
+            -0.009175300598144531,
+            0.023133469745516777,
+            -0.004807271528989077,
+            0.0019243031274527311,
+            -0.00861055962741375,
+            0.030231080949306488,
+            0.016175301745533943,
+            -0.006484064739197493,
+            0.025768928229808807,
+            -0.004922311753034592,
+            -0.0020567732863128185,
+            -0.012431276962161064,
+            -0.004458666313439608,
+            0.01342828944325447,
+            -0.011643428355455399,
+            0.01600797101855278,
+            -0.04264144226908684,
+            0.0059088654816150665,
+            -0.011022910475730896,
+            -0.005121016874909401,
+            -0.016189245507121086,
+            -0.0171235091984272,
+            -0.030258968472480774,
+            0.02502988465130329,
+            0.00914741214364767,
+            -0.027079686522483826,
+            -0.0005503611755557358,
+            -0.006679283920675516,
+            0.0005150648066774011,
+            -0.01109263114631176,
+            -0.015812750905752182,
+            0.016551796346902847,
+            0.03787251561880112,
+            -0.012005978263914585,
+            -0.01367231272161007,
+            0.003060757415369153,
+            0.008450200781226158,
+            -0.017444226890802383,
+            -0.006375997327268124,
+            0.0019765938632190228,
+            -0.028097614645957947,
+            0.020274903625249863,
+            -0.005751993041485548,
+            0.0031374506652355194,
+            -0.017346616834402084,
+            0.01600797101855278,
+            -0.015394425019621849,
+            -0.0060587660409510136,
+            -0.02253386750817299,
+            -0.023523908108472824,
+            0.016342632472515106,
+            -0.033466141670942307,
+            -0.011315738782286644,
+            -0.01853187568485737,
+            0.004444722086191177,
+            -0.00848506111651659,
+            -0.001322086900472641,
+            0.009000997990369797,
+            0.00419023958966136,
+            0.032852593809366226,
+            -0.00848506111651659,
+            0.004137949086725712,
+            0.03307570144534111,
+            -0.008394423872232437,
+            0.024709166958928108,
+            0.0022641937248408794,
+            0.023328689858317375,
+            0.009454185143113136,
+            0.0013752492377534509,
+            -0.007369523402303457,
+            0.010925300419330597,
+            -0.0014310261467471719,
+            0.03731474652886391,
+            -0.011901396326720715,
+            -0.018922314047813416,
+            0.00861055962741375,
+            -0.004890936892479658,
+            0.006403885781764984,
+            0.02846016362309456,
+            0.019382473081350327,
+            -0.005738048814237118,
+            0.008275898173451424,
+            -0.01157370675355196,
+            -0.0025918828323483467,
+            0.008059762418270111,
+            0.011713149026036263,
+            0.017039842903614044,
+            0.0002762699732556939,
+            -0.019982075318694115,
+            0.02112550102174282,
+            0.0009569224785082042,
+            -0.006428287830203772,
+            -0.015213149599730968,
+            0.0078087663277983665,
+            -0.006972112692892551,
+            0.00922410562634468,
+            -0.00840836763381958,
+            0.016830680891871452,
+            0.0027034366503357887,
+            -0.02526693604886532,
+            0.0031130483839660883,
+            0.00911952368915081,
+            -0.002987550338730216,
+            -0.004950200207531452,
+            -0.016844624653458595,
+            -0.010904384776949883,
+            0.0058530885726213455,
+            0.015812750905752182,
+            -0.0054068733006715775,
+            -0.02628486417233944,
+            -0.005581176374107599,
+            0.00018236432515550405,
+            0.0008924304274842143,
+            0.014167333021759987,
+            0.0018388947937637568,
+            -0.02320319041609764,
+            0.017960162833333015,
+            -0.0017395421164110303,
+            0.004650399088859558,
+            -0.009614543057978153,
+            0.026410363614559174,
+            -0.004660857375711203,
+            -0.011587651446461678,
+            0.010799802839756012,
+            0.034721121191978455,
+            -0.0019452194683253765,
+            0.01868526265025139,
+            -0.02370518259704113,
+            -0.002764442702755332,
+            -0.00901494175195694,
+            0.004193725995719433,
+            -0.01532470341771841,
+            -0.0007220494444482028,
+            0.016133468598127365,
+            0.0023809764534235,
+            -0.010269922204315662,
+            0.002013197634369135,
+            0.024332674220204353,
+            -0.0009063746547326446,
+            0.003261205740272999,
+            0.015687253326177597,
+            0.005731076467782259,
+            0.018629485741257668,
+            0.011476097628474236,
+            -0.03792829439043999,
+            0.002319970401003957,
+            0.010723109357059002,
+            0.0031339647248387337,
+            0.002788844984024763,
+            0.014320719987154007,
+            -0.0053894431330263615,
+            -0.0061807781457901,
+            -0.0030468131881207228,
+            0.013023906387388706,
+            0.009893427602946758,
+            0.008101594634354115,
+            0.0035906380508095026,
+            0.0022746517788618803,
+            -0.018517931923270226,
+            0.013121516443789005,
+            0.0018458669073879719,
+            0.0069023915566504,
+            0.014739046804606915,
+            -0.015980081632733345,
+            -0.005469622556120157,
+            -0.009928288869559765,
+            0.004197211936116219,
+            0.0017953190254047513,
+            -0.004050797317177057,
+            0.004800299648195505,
+            -0.001281997188925743,
+            0.0014101098058745265,
+            -0.014599604532122612,
+            0.006389941554516554,
+            0.01864342950284481,
+            0.0008105080923996866,
+            -0.004981574602425098,
+            0.03661753609776497,
+            -0.01783466525375843,
+            0.010988050140440464,
+            -0.012061755172908306,
+            0.014153389260172844,
+            -0.02597809210419655,
+            0.041358571499586105,
+            0.016858568415045738,
+            0.00477589713409543,
+            0.006686256267130375,
+            -0.01079283095896244,
+            -0.007522909436374903,
+            -0.01218028087168932,
+            0.013769922778010368,
+            0.013846616260707378,
+            -0.015966137871146202,
+            -0.0008793577435426414,
+            -0.0018127493094652891,
+            -0.0011164095485582948,
+            0.02157171629369259,
+            0.006484064739197493,
+            -0.014697213657200336,
+            0.008840639144182205,
+            -0.0018894425593316555,
+            -0.008791834115982056,
+            -0.01869920641183853,
+            -0.008213148452341557,
+            0.014850599691271782,
+            -0.0072161369025707245,
+            -0.0022955681197345257,
+            0.008708168752491474,
+            0.004155379254370928,
+            0.000675423420034349,
+            -0.018420321866869926,
+            0.019577693194150925,
+            -0.011176296509802341,
+            0.006602590903639793,
+            -0.001901643699966371,
+            -0.02684263326227665,
+            0.015617532655596733,
+            -0.019828688353300095,
+            0.015227094292640686,
+            -0.00016624131239950657,
+            -0.027539845556020737,
+            0.0003377117100171745,
+            0.004664343316107988,
+            -0.023342633619904518,
+            -0.020442234352231026,
+            0.005110558588057756,
+            -0.0008105080923996866,
+            -0.004169323481619358,
+            -0.03352191671729088,
+            0.03681275621056557,
+            0.015826696529984474,
+            0.03879283368587494,
+            -0.019312752410769463,
+            -0.006187750026583672,
+            0.0034790842328220606,
+            -0.00667579798027873,
+            0.003147908952087164,
+            -0.026577694341540337,
+            0.016788847744464874,
+            0.022505979984998703,
+            0.013358567841351032,
+            -0.011866535991430283,
+            -0.009635459631681442,
+            -0.004490040708333254,
+            -0.008059762418270111,
+            0.014641436748206615,
+            -0.006672312039881945,
+            -0.013797811232507229,
+            -0.003052042331546545,
+            0.0004989418084733188,
+            0.003393675899133086,
+            0.016733070835471153,
+            -0.004967630375176668,
+            0.006229582708328962,
+            -0.0043436260893940926,
+            0.00048674060963094234,
+            -0.028223112225532532,
+            0.028892435133457184,
+            -0.021195223554968834,
+            0.03692431002855301,
+            -0.005713646300137043,
+            0.003475598292425275,
+            -0.005180279724299908,
+            0.007801793981343508,
+            0.018503988161683083,
+            -0.002971863141283393,
+            -0.010144423693418503,
+            -0.01772311143577099,
+            7.070157880662009e-05,
+            0.015589644201099873,
+            -0.0016062004724517465,
+            -0.0036882476415485144,
+            0.0189502015709877,
+            1.2528014849522151e-05,
+            -0.008708168752491474,
+            -0.009719124995172024,
+            -0.002668576082214713,
+            0.004842132329940796,
+            0.011587651446461678,
+            0.0275119561702013,
+            0.008017929270863533,
+            0.0029178292024880648,
+            0.006184264086186886,
+            -0.001894671586342156,
+            -0.02678685635328293,
+            -0.010067731142044067,
+            0.01266135647892952,
+            -0.029422316700220108,
+            0.0053720129653811455,
+            0.010988050140440464,
+            0.019061755388975143,
+            0.011134464293718338,
+            -0.021055780351161957,
+            -0.0006266186246648431,
+            0.01681673526763916,
+            -0.006145917344838381,
+            0.029756976291537285,
+            -0.01777888834476471,
+            0.011183268390595913,
+            0.026424307376146317,
+            0.018768927082419395,
+            -0.008847611024975777,
+            -0.01797410659492016,
+            0.008303786627948284,
+            0.0072509972378611565,
+            -0.001050174469128251,
+            0.008924304507672787,
+            -0.04043825343251228,
+            0.004451693966984749,
+            -0.01858765259385109,
+            0.0148645443841815,
+            0.004448208026587963,
+            0.011287850327789783,
+            0.006484064739197493,
+            -0.003172311233356595,
+            0.001901643699966371,
+            0.005528885405510664,
+            0.0006017804844304919,
+            -0.0276095662266016,
+            -0.00949601735919714,
+            -0.0085617545992136,
+            0.0012558518210425973,
+            -0.010262950323522091,
+            -0.0010606326395645738,
+            0.01570119708776474,
+            -0.007285858038812876,
+            0.002699950709939003,
+            -0.041051801294088364,
+            -0.013644424267113209,
+            -0.010632472112774849,
+            -0.012089643627405167,
+            -0.011357571929693222,
+            0.019215142354369164,
+            -0.00269820773974061,
+            -0.01010956335812807,
+            0.008331675082445145,
+            0.005469622556120157,
+            -0.030565742403268814,
+            -0.008603586815297604,
+            0.00922410562634468,
+            -0.006247012875974178,
+            0.0040647415444254875,
+            0.005103586707264185,
+            -0.012026894837617874,
+            -0.024890443310141563,
+            0.009956177324056625,
+            -0.012884464114904404,
+            0.016440242528915405,
+            0.019842632114887238,
+            0.0007159488159231842,
+            -0.005535857751965523,
+            0.0073276907205581665,
+            0.01023506186902523,
+            -0.005880977027118206,
+            0.01677490398287773,
+            -0.010820719413459301,
+            0.01635657623410225,
+            0.014223109930753708,
+            0.01844821125268936,
+            -6.938069418538362e-06,
+            -0.014557771384716034,
+            -0.00029609689954668283,
+            0.006749005056917667,
+            0.00505129573866725,
+            -0.00577290914952755,
+            -0.004197211936116219,
+            -0.012354583479464054,
+            0.010932273231446743,
+            -0.007264941465109587,
+            -0.006909363903105259,
+            -0.001948705525137484,
+            0.00015905132750049233,
+            -0.0050756982527673244,
+            -0.004577192012220621,
+            -0.002827191725373268,
+            -0.014766935259103775,
+            0.0006484065088443458,
+            -0.026661358773708344,
+            -0.00863844808191061,
+            -0.007244025357067585,
+            0.0016645919531583786,
+            -0.006341136526316404,
+            0.0030747016426175833,
+            0.04913945123553276,
+            -0.01446016225963831,
+            0.003154881065711379,
+            -0.003660359187051654,
+            -0.02380279265344143,
+            -0.018406378105282784,
+            0.00012734999472741038,
+            -0.009356575086712837,
+            -0.026480084285140038,
+            -0.016370519995689392,
+            0.011992033571004868,
+            -0.0036708174739032984,
+            0.0009708667057566345,
+            -0.010848607867956161,
+            -0.01717928610742092,
+            -0.02643825113773346,
+            -0.005657869391143322,
+            -0.0021142931655049324,
+            -0.010667332448065281,
+            -0.015589644201099873,
+            0.006473606918007135,
+            -0.010653388686478138,
+            0.009781873784959316,
+            0.02360757440328598,
+            0.004364542663097382,
+            0.019312752410769463,
+            0.00901494175195694,
+            -0.018266934901475906,
+            -0.0015434514498338103,
+            -0.0037998014595359564,
+            -0.018015939742326736,
+            0.006693228147923946,
+            0.01066036056727171,
+            -0.006755977403372526,
+            0.011008965782821178,
+            -0.007578686345368624,
+            -0.014725102111697197,
+            -0.004514443222433329,
+            -0.019968131557106972,
+            0.029003988951444626,
+            -0.02582470513880253,
+            -0.005005977116525173,
+            -0.003897411050274968,
+            -0.012849603779613972,
+            0.027079686522483826,
+            0.0128426318988204,
+            -0.019312752410769463,
+            0.024346617981791496,
+            0.0040473113767802715,
+            0.008366535417735577,
+            0.01374900620430708,
+            0.011789842508733273,
+            0.0007586530409753323,
+            0.011322710663080215,
+            0.027372514829039574,
+            -0.0071952203288674355,
+            -0.030258968472480774,
+            -0.002189243445172906,
+            0.01853187568485737,
+            0.014906376600265503,
+            -0.006909363903105259,
+            -0.002267679665237665,
+            -0.00014162104343995452,
+            0.017249006778001785,
+            -0.001689865835942328,
+            -0.000326599896652624,
+            -0.0008667207439430058,
+            -0.009007969871163368,
+            -0.007683268282562494,
+            0.006424801889806986,
+            -0.026173310354351997,
+            -0.0035278890281915665,
+            -0.02224103920161724,
+            0.011350599117577076,
+            -0.013023906387388706,
+            -0.002933516399934888,
+            -0.007725100964307785,
+            -0.005856574513018131,
+            0.027135463431477547,
+            0.021097613498568535,
+            -0.006700200494378805,
+            0.011113547720015049,
+            -0.008687252178788185,
+            -0.012187252752482891,
+            0.014711158350110054,
+            -0.0018476098775863647,
+            0.0014519424876198173,
+            -0.0021282373927533627,
+            0.005769423209130764,
+            -0.011232073418796062,
+            0.0013046566164121032,
+            0.022059764713048935,
+            0.024025900289416313,
+            -0.01367231272161007,
+            -0.020958170294761658,
+            -0.014934265054762363,
+            0.021892433986067772,
+            -0.0010449454421177506,
+            -0.011252989992499352,
+            0.0074741048738360405,
+            0.003100847126916051,
+            -0.010214145295321941,
+            -0.015715142711997032,
+            0.016300799325108528,
+            -0.02137649804353714,
+            -0.014446217566728592,
+            -0.014780879020690918,
+            -0.0036777895875275135,
+            -0.031179288402199745,
+            0.019326696172356606,
+            0.016886457800865173,
+            -0.004817729815840721,
+            -0.002276394749060273,
+            0.01631474308669567,
+            0.00587749108672142,
+            -0.00845717266201973,
+            -0.011720121838152409,
+            -0.013414344750344753,
+            -0.0017273409757763147,
+            0.0030293830204755068,
+            -0.015631476417183876,
+            -0.0017848608549684286,
+            0.0009211904252879322,
+            0.00805279053747654,
+            0.006030877586454153,
+            -0.0018493529642000794,
+            0.0016733070369809866,
+            -0.002238048240542412,
+            0.031792834401130676,
+            0.012975101359188557,
+            -0.00810856744647026,
+            0.007501993328332901,
+            -0.005535857751965523,
+            0.012696217745542526,
+            -0.003604582278057933,
+            -0.023328689858317375,
+            -0.02705179713666439,
+            -0.01382569968700409,
+            -0.004036853089928627,
+            -0.0043436260893940926,
+            -0.006009961012750864,
+            -0.014390440657734871,
+            0.006498008966445923,
+            -0.008157371543347836,
+            0.013281874358654022,
+            0.005197709891945124,
+            -0.002307769376784563,
+            0.014739046804606915,
+            0.01991235464811325,
+            0.012110560201108456,
+            -0.01446016225963831,
+            -0.014390440657734871,
+            0.012438248842954636,
+            -0.008729085326194763,
+            -0.002525647869333625,
+            0.020581677556037903,
+            0.013658368960022926,
+            -0.01436255220323801,
+            -0.0221294853836298,
+            0.005033865571022034,
+            -0.017695222049951553,
+            0.011950201354920864,
+            0.003034612163901329,
+            0.006651395466178656,
+            0.01849004253745079,
+            0.014501994475722313,
+            -0.010444224812090397,
+            0.003442480694502592,
+            0.011427292600274086,
+            -0.013386456295847893,
+            -0.030816737562417984,
+            -0.02518327161669731,
+            -0.004420319572091103,
+            -0.016175301745533943,
+            -0.03120717592537403,
+            0.0006919822189956903,
+            -0.007397411856800318,
+            0.017555780708789825,
+            0.009433268569409847,
+            0.017388449981808662,
+            0.021641438826918602,
+            0.0078087663277983665,
+            0.004950200207531452,
+            0.018657373264431953,
+            0.004699204117059708,
+            -0.0037440245505422354,
+            -0.006745519116520882,
+            -0.004751494619995356,
+            0.004723606165498495,
+            -0.01175498217344284,
+            0.0005673556588590145,
+            -0.011448209173977375,
+            0.020930282771587372,
+            0.018322711810469627,
+            0.005783367436379194,
+            0.0010597611544653773,
+            0.009942232631146908,
+            0.0016881227493286133,
+            0.00901494175195694,
+            0.009161355905234814,
+            -0.010709165595471859,
+            -0.0003250747686251998,
+            -0.013030878268182278,
+            -0.02816733531653881,
+            0.0090428302064538,
+            0.004472610540688038,
+            -0.0005625623743981123,
+            0.008645419962704182,
+            -0.00746016064658761,
+            -0.02380279265344143,
+            -0.0030136958230286837,
+            0.0062051801942288876,
+            -0.013016934506595135,
+            -0.006717630662024021,
+            -0.0014414843171834946,
+            0.007174304220825434,
+            0.023482074961066246,
+            -0.00970518123358488,
+            0.021139446645975113,
+            0.019745023921132088,
+            -0.0003472983662504703,
+            -0.008526894263923168,
+            -0.019745023921132088,
+            -0.033968131989240646,
+            -0.015575699508190155,
+            0.005358068738132715,
+            -0.0305936299264431,
+            -0.005434761755168438,
+            -0.007055778056383133,
+            -0.00016014071297831833,
+            -0.009656376205384731,
+            0.01023506186902523,
+            -0.0001039280541590415,
+            -0.00861055962741375,
+            0.013233070261776447,
+            -0.0034006480127573013,
+            0.005378984846174717,
+            -0.006923308130353689,
+            0.005392929073423147,
+            -0.01436255220323801,
+            -0.0011870021698996425,
+            0.001350846840068698,
+            0.014599604532122612,
+            -0.02056773193180561,
+            -0.011001993902027607,
+            0.0040298812091350555,
+            -0.0010562750976532698,
+            -0.006323706358671188,
+            -0.0201075728982687,
+            0.012668329291045666,
+            -0.03619920834898949,
+            0.008805778808891773,
+            -0.0035122018307447433,
+            0.002835906809195876,
+            -0.006278387736529112,
+            -0.021237054839730263,
+            -0.004430777858942747,
+            0.006337650585919619,
+            -0.019745023921132088,
+            0.0024158370215445757,
+            -0.01975896768271923,
+            0.010939245112240314,
+            -0.007725100964307785,
+            0.016454186290502548,
+            -0.0123615562915802,
+            -5.822803359478712e-05,
+            0.0030747016426175833,
+            0.01451593916863203,
+            -0.002084661740809679,
+            -0.0022397912107408047,
+            0.0024332674220204353,
+            -0.0017630730289965868,
+            0.003043327247723937,
+            -0.010925300419330597,
+            0.006982570979744196,
+            -0.006449204403907061,
+            0.01121812965720892,
+            0.005086156073957682,
+            0.019786855205893517,
+            0.006877989042550325,
+            0.023007972165942192,
+            -0.020846616476774216,
+            0.006072710268199444,
+            -0.015045818872749805,
+            0.018155381083488464,
+            0.013762950897216797,
+            0.02528088167309761,
+            0.011162352748215199,
+            -0.02582470513880253,
+            0.009830678813159466,
+            0.028501996770501137,
+            0.022017931565642357,
+            -0.008519921451807022,
+            -0.01929880864918232,
+            0.0026703192852437496,
+            0.03990837186574936,
+            0.01340040098875761,
+            0.00011972424545092508,
+            -0.011225101538002491,
+            0.016370519995689392,
+            0.00531972199678421,
+            -0.008631475269794464,
+            0.01492032129317522,
+            -0.02588048204779625,
+            0.0006937252474017441,
+            0.005738048814237118,
+            0.011190241202712059,
+            -0.022868528962135315,
+            0.0160637479275465,
+            0.014697213657200336,
+            0.026828689500689507,
+            -0.01980080083012581,
+            0.0015434514498338103,
+            -0.0033065243624150753,
+            0.008164344355463982,
+            -0.0070488061755895615,
+            0.002985807368531823,
+            0.01342828944325447,
+            -0.010883468203246593,
+            -0.04280877113342285,
+            -0.0006793452193960547,
+            0.009265937842428684,
+            0.016691237688064575,
+            -0.0032629487104713917,
+            0.005072211846709251,
+            -0.023970123380422592,
+            -0.020372513681650162,
+            -0.01053486205637455,
+            0.024806777015328407,
+            -0.0014911605976521969,
+            0.028041837736964226,
+            -0.0002359624340897426,
+            -0.009747013449668884,
+            -0.007101096678525209,
+            0.014083667658269405,
+            -0.005044323392212391,
+            0.007948208600282669,
+            0.03182072192430496,
+            0.014808767475187778,
+            0.005194223951548338,
+            -0.010939245112240314,
+            0.03611554577946663,
+            -0.00010321995068807155,
+            -0.005124502815306187,
+            -0.00845717266201973,
+            0.0007756475242786109,
+            0.0192430317401886,
+            -0.008478089235723019,
+            -0.006539841648191214,
+            0.011057770811021328,
+            0.0022310761269181967,
+            -0.0070104594342410564,
+            -0.023621518164873123,
+            0.015227094292640686,
+            -0.010297810658812523,
+            0.014557771384716034,
+            -0.0004311816010158509,
+            0.011615539900958538,
+            -0.031988054513931274,
+            -0.006592132616788149,
+            -0.0010510459542274475,
+            -0.006191235966980457,
+            0.015408368781208992,
+            -0.014780879020690918,
+            -0.009642431512475014,
+            -0.005884462967514992,
+            -0.011022910475730896,
+            0.021139446645975113,
+            0.00843625608831644,
+            -0.007543826010078192,
+            -0.002124751452356577,
+            -0.004336654208600521,
+            0.0011442980030551553,
+            -0.006550299935042858,
+            -0.010576695203781128,
+            -0.02132072113454342,
+            -0.014320719987154007,
+            -0.019842632114887238,
+            -0.01564542017877102,
+            0.018559763208031654,
+            0.009788846597075462,
+            -0.02218526229262352,
+            0.014501994475722313,
+            -0.002930030459538102,
+            0.0009935260750353336,
+            -0.0015112054534256458,
+            0.008903387933969498,
+            0.011134464293718338,
+            -0.024597613140940666,
+            0.0023426299449056387,
+            -0.014404385350644588,
+            -0.00024097364803310484,
+            -0.02847410924732685,
+            0.015868527814745903,
+            -0.017235063016414642,
+            -0.0007159488159231842,
+            0.020205182954669,
+            -0.013532870449125767,
+            -0.00944721233099699,
+            0.00328560802154243,
+            -0.027874507009983063,
+            0.006877989042550325,
+            0.008324702270328999,
+            -0.005661355331540108,
+            -0.027832673862576485,
+            -0.015213149599730968,
+            -0.037816740572452545,
+            -0.010995022021234035,
+            0.02476494386792183,
+            -0.0189502015709877,
+            -0.00014935572107788175,
+            0.009837650693953037,
+            -0.006417830009013414,
+            0.02502988465130329,
+            -0.01119721308350563,
+            0.005675299558788538,
+            -0.013072711415588856,
+            -0.01621713489294052,
+            0.0030102096498012543,
+            0.0037998014595359564,
+            -0.002534362953156233,
+            -0.012521914206445217,
+            7.544043910456821e-05,
+            0.002996265422552824,
+            0.0004237737157382071,
+            0.013693229295313358,
+            -0.0004531873273663223,
+            -0.019424306228756905,
+            0.014153389260172844,
+            -0.028808770701289177,
+            0.0016062004724517465,
+            -0.012926297262310982,
+            -0.009210160933434963,
+            -0.0072509972378611565,
+            0.007188248448073864,
+            0.005166335497051477,
+            0.0037579687777906656,
+            0.005821714177727699,
+            0.001241907593794167,
+            0.013735062442719936,
+            -0.004517929162830114,
+            -0.011406376026570797,
+            0.007662351708859205,
+            -0.016426296904683113,
+            0.02406773343682289,
+            0.01920119859278202,
+            0.0023600601125508547,
+            -0.02416534349322319,
+            0.011078687384724617,
+            0.003097361186519265,
+            -0.0071394434198737144,
+            0.0028237055521458387,
+            0.02324502356350422,
+            -0.004001992754638195,
+            0.010911356657743454,
+            0.001513820025138557,
+            -0.003372759558260441,
+            0.030677296221256256,
+            -0.009342631325125694,
+            -0.003859064308926463,
+            -0.01263346802443266,
+            -0.00851294957101345,
+            0.0051872520707547665,
+            -0.00021875003585591912,
+            -0.004361056722700596,
+            -0.0013212154153734446,
+            -0.006511953193694353,
+            0.02518327161669731,
+            0.00497460225597024,
+            0.0031984567176550627,
+            0.0076414356008172035,
+            -0.01585458405315876,
+            0.009684264659881592,
+            -0.02522510476410389,
+            -0.005497511010617018,
+            -0.022659366950392723,
+            -0.026159366592764854,
+            -0.030760960653424263,
+            -0.02026095986366272,
+            -0.014404385350644588,
+            0.006288845557719469,
+            0.0015190490521490574,
+            -0.007467132527381182,
+            0.0008680280297994614,
+            -0.007237053010612726,
+            -0.005166335497051477,
+            0.01492032129317522,
+            0.01061155553907156,
+            0.014390440657734871,
+            -0.0016184017295017838,
+            0.022324705496430397,
+            -0.004364542663097382,
+            -0.02253386750817299,
+            -0.008066734299063683,
+            -0.007704184390604496,
+            0.015715142711997032,
+            0.011029882356524467,
+            0.015924304723739624,
+            -0.0020236556883901358,
+            -0.004625997040420771,
+            -0.006689742207527161,
+            0.023217136040329933,
+            0.0013238298706710339,
+            -0.012598607689142227,
+            0.003942729905247688,
+            0.02897610142827034,
+            -0.0026668331120163202,
+            0.005832172464579344,
+            0.013686257414519787,
+            0.0101025914773345,
+            -0.010472113266587257,
+            -0.015533867292106152,
+            -0.008387451991438866,
+            -0.005420817527920008,
+            0.0022188748698681593,
+            -0.005692730192095041,
+            -0.006135459057986736,
+            -0.016928289085626602,
+            0.006100598722696304,
+            -0.01342828944325447,
+            -0.005009463056921959,
+            0.010430281050503254,
+            -0.010088646784424782,
+            0.0028254487551748753,
+            0.012933269143104553,
+            0.013302790932357311,
+            -0.000308733870042488,
+            0.02507171779870987,
+            0.0027836160734295845,
+            0.004549303557723761,
+            -0.012152392417192459,
+            -0.020707175135612488,
+            -0.007017431315034628,
+            0.018517931923270226,
+            0.013086655177175999,
+            0.002142181620001793,
+            -0.006588646676391363,
+            -0.02726096101105213,
+            0.0007133343024179339,
+            -0.009865539148449898,
+            0.008429284207522869,
+            -0.034972116351127625,
+            0.023119525983929634,
+            -0.028139447793364525,
+            0.008366535417735577,
+            0.01727689616382122,
+            0.05053387209773064,
+            0.00041789101669564843,
+            0.0023844626266509295,
+            6.50367364869453e-05,
+            0.015575699508190155,
+            0.02152988500893116,
+            -0.0009952690452337265,
+            -0.0021787851583212614,
+            -0.038011956959962845,
+            -0.005309263709932566,
+            -0.000978710362687707,
+            -0.016021914780139923,
+            0.014271914958953857,
+            -0.0015190490521490574,
+            -0.024751000106334686,
+            -0.0039531877264380455,
+            -0.012494025751948357,
+            -0.005054781679064035,
+            0.017193229869008064,
+            0.014752990566194057,
+            -0.015687253326177597,
+            -0.0010815489804372191,
+            -0.019173309206962585,
+            -0.00797609705477953,
+            0.02208765409886837,
+            0.0013586904387921095,
+            0.008010957390069962,
+            0.004406375344842672,
+            0.007746017072349787,
+            0.0013752492377534509,
+            0.01063944399356842,
+            -0.0017534863436594605,
+            0.05111953243613243,
+            -0.016858568415045738,
+            -0.005671813618391752,
+            -0.006860558874905109,
+            0.0004283491871319711,
+            -0.003201942890882492,
+            0.02147410809993744,
+            0.0023478588555008173,
+            0.0015085908817127347,
+            -0.004789841361343861,
+            -0.019368529319763184,
+            -0.0007015688461251557,
+            0.0009351345943287015,
+            0.0014475849457085133,
+            -0.007990040816366673,
+            0.006208666600286961,
+            -0.028390442952513695,
+            -0.015213149599730968,
+            0.008066734299063683,
+            -0.007773905526846647,
+            0.0004348855291027576,
+            -0.005490538664162159,
+            -0.0015164344804361463,
+            0.0007434015278704464,
+            0.022115541622042656,
+            -0.015059763565659523,
+            -0.0059646423906087875,
+            -0.012682273052632809,
+            0.0005586405168287456,
+            -0.029645422473549843,
+            0.003036355134099722,
+            -0.0017473857151344419,
+            0.022073708474636078,
+            -0.02624303288757801,
+            -0.024932274594902992,
+            0.006362053100019693,
+            0.00866633653640747,
+            -0.0028515940066426992,
+            -0.008687252178788185,
+            0.014599604532122612,
+            -0.02203187718987465,
+            0.019996019080281258,
+            0.0018022911390289664,
+            -0.03374502435326576,
+            -0.009258965961635113,
+            0.014501994475722313,
+            -0.01691434532403946,
+            0.00152950722258538,
+            -0.004800299648195505,
+            0.02051195502281189,
+            0.01777888834476471,
+            -0.010011954233050346,
+            -0.017290839925408363,
+            0.0005067854654043913,
+            0.019047811627388,
+            -0.005828686058521271,
+            0.02805578149855137,
+            -0.023886458948254585,
+            0.012298806570470333,
+            -0.024792833253741264,
+            0.01010956335812807,
+            -0.02060956507921219,
+            0.003900896990671754,
+            0.018810760229825974,
+            -0.009921316057443619,
+            0.022450203076004982,
+            -0.016984065994620323,
+            -0.022519923746585846,
+            0.0076414356008172035,
+            -0.010137451812624931,
+            -0.025294825434684753,
+            0.013560758903622627,
+            -0.023593628779053688,
+            -0.013058766722679138,
+            0.0042878491804003716,
+            -0.014613548293709755,
+            -0.011992033571004868,
+            0.015394425019621849,
+            0.004117032513022423,
+            0.004472610540688038,
+            0.0017343130894005299,
+            -0.021041836589574814,
+            -0.013205181807279587,
+            -0.00663396529853344,
+            0.003052042331546545,
+            -0.0020881476812064648,
+            -0.007599602919071913,
+            -0.0359761007130146,
+            0.020079685375094414,
+            -0.01565936580300331,
+            -0.01382569968700409,
+            -0.0037265941500663757,
+            0.019131477922201157,
+            0.009356575086712837,
+            0.005790339782834053,
+            0.011580679565668106,
+            -0.013358567841351032,
+            -0.007244025357067585,
+            0.006034363526850939,
+            0.018420321866869926,
+            -0.009698208421468735,
+            -0.004273904953151941,
+            -0.010346615687012672,
+            0.006138945464044809,
+            -0.002053287113085389,
+            -0.0039531877264380455,
+            0.00914741214364767,
+            0.011908368207514286,
+            -0.004486554767936468,
+            0.006752490997314453,
+            -0.008945221081376076,
+            0.012187252752482891,
+            -0.017541835084557533,
+            0.0074531883001327515,
+            0.006783865857869387,
+            0.00292480131611228,
+            0.007024403661489487,
+            -0.007202192675322294,
+            0.021097613498568535,
+            0.03586454689502716,
+            0.000885022571310401,
+            0.014962153509259224,
+            0.00420418381690979,
+            0.0039357575587928295,
+            -0.00036712532164528966,
+            0.021655382588505745,
+            -0.005880977027118206,
+            -0.009733069688081741,
+            -0.00405428372323513,
+            -0.014069723896682262,
+            0.01793227344751358,
+            -0.00866633653640747,
+            -0.0015608817338943481,
+            -0.03525100275874138,
+            0.008896416053175926,
+            -0.003384960815310478,
+            -0.005535857751965523,
+            -0.012194225564599037,
+            -0.009942232631146908,
+            0.02310558222234249,
+            -0.0035732078831642866,
+            0.026521917432546616,
+            -0.00013704558659810573,
+            0.027525901794433594,
+            -0.011455181054770947,
+            -0.00563695328310132,
+            -0.015729086473584175,
+            -0.004312251694500446,
+            0.015087652020156384,
+            0.00485259061679244,
+            0.02922709658741951,
+            0.012821715325117111,
+            -0.0006344622815959156,
+            0.010123508051037788,
+            0.004538845270872116,
+            0.01989840902388096,
+            0.015826696529984474,
+            0.013811754994094372,
+            -0.0033535861875861883,
+            -0.0060378494672477245,
+            0.006710658315569162,
+            0.02081872895359993,
+            0.0016515192110091448,
+            -0.00746016064658761,
+            0.002637201687321067,
+            -0.0071220132522284985,
+            0.016844624653458595,
+            0.011406376026570797,
+            0.026047812774777412,
+            -0.014006974175572395,
+            0.005281375255435705,
+            0.006198208313435316,
+            0.0019853091798722744,
+            0.003897411050274968,
+            -0.0078087663277983665,
+            0.02218526229262352,
+            0.00810856744647026,
+            0.003426793497055769,
+            0.0020637453999370337,
+            -0.004639940802007914,
+            -0.000789156008977443,
+            -0.0053894431330263615,
+            -0.012724106200039387,
+            0.005487052723765373,
+            0.004259960725903511,
+            -0.022352593019604683,
+            -0.0026162853464484215,
+            -0.0138047831133008,
+            0.007704184390604496,
+            0.018322711810469627,
+            0.017235063016414642,
+            0.003268177853897214,
+            -0.0026825203094631433,
+            -0.003932271618396044,
+            0.00030306901317089796,
+            0.008024902082979679,
+            0.0025465642102062702,
+            0.016537850722670555,
+            -0.024583669379353523,
+            -0.01878287084400654,
+            0.010060758329927921,
+            -0.006281873676925898,
+            -0.01311454363167286,
+            0.005860060919076204
+          ],
+          "keyphrases": [
+            "fine-tuning",
+            "dense model",
+            "MoE model",
+            "expert choice routing",
+            "downstream performance"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "614524d3-456b-46fb-97ce-c9bdf9dc93e4",
+      "type": "child",
+      "source": {
+        "id": "b570d71e-4f83-486e-b70f-b0c6217db455",
+        "properties": {
+          "page_content": "Mixture-Of-Experts With Expert Choice Routing\n\nYanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon Google, Mountain View, CA, USA {yanqiz, taole, hanxiaol, dunan, huangyp, vzhao, adai, zhifengc, qvl, jlaudon}@google.com\n\nAbstract\n\nSparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choice method. Instead of letting tokens select the top-k experts, we have experts selecting the top-k tokens. As a result, each token can be routed to a variable number of experts and each expert can have a fixed bucket size. We systematically study pre-training speedups using the same computational resources of the Switch Transformer top-1 and GShard top-2 gating of prior work and find that our method improves training convergence time by more than 2×. For the same computational cost, our method demonstrates higher performance in fine-tuning 11 selected tasks in the GLUE and SuperGLUE benchmarks. For a smaller activation cost, our method outperforms the T5 dense model in 7 out of the 11 tasks.\n\n1 Introduction\n\nScaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27]. The final model quality has been found to have a power-law relationship with the amount of data, model size, and compute time [16, 20]. However, training efficiency, which is defined as the total amount of computation used to achieve superior model quality than the state of the art system [21], should receive greater attention as we increase our efforts towards green AI [29].\n\nSparsely gated mixture-of-experts [31] (MoE) provides an effective way to scale model capacity given a fixed computational cost, and has recently played an important role in increasing the training efficiency of large-scale language models [10, 21]. MoE operate by adopting a number of experts, each as a sub-network, and by activating only one or a few experts for each input token. A gating network must be chosen and optimized in order to route each token to the most suited expert(s). For example, recent work has implemented sparse routing via k-means clustering [12], linear assignment to maximize token-expert affinities [22], or hashing [8, 28]. Many of the prior work use a routing strategy concerning the token choice, where each token selects the best one or two experts.\n\nWe argue that the independent token choice of prior work often leads to an imbalanced load of experts, which causes training inefficiency and sub-optimal training of the model. In order to mitigate this\n\nissue, previous sparsely gated networks introduce additional auxiliary losses as regularization to prevent too many tokens being routed to a single expert, but the effectiveness is still limited. Recent approaches [8, 22, 28] explore alternative strategies for routing, but they focus on pre-training only and do not demonstrate performance gain on downstream tasks. Moreover, none of the previous methods consider allocating a variable number of experts to each token based on importance, which can be beneficial.\n\nWe propose a very simple yet effective routing method we are calling expert choice. Unlike conventional MoE where tokens select one or two top-scoring experts, our method lets each expert pick the top-k tokens. Our method guarantees perfect load balancing, allows a variable number of experts for each token, and achieves substantial gains in training efficiency and downstream performance as demonstrated in our experiments. Our major contributions include:\n\nWe identify common pitfalls in conventional MoE such as load imbalance as described in Section 3.1. We then propose a heterogeneous, expert choice method to provide a fluid allocation of model parameters based on a learnt token-to-expert importance. This method intrinsically guarantees load balance without imposing an auxiliary loss.\n\nWe show our method provides over 2× faster training convergence in a 8B/64E (8 billion activated parameters, 64 experts) model, compared to the top-1 and top-2 gating counterparts in Switch Transformer [10] and GShard [21].\n\nWe show our method demonstrates strong scaling when increasing the number of experts from 16 to 128, evaluated in training perplexity.\n\nWe show our method demonstrates strong performance on downstream tasks selected from GLUE and SuperGLUE at all the evaluated scales. More specifically, our 8B/64E model outperforms a T5 11B dense model in 7 out of 11 tasks evaluated.\n\n2 Related Work\n\nScaling: Various approaches have been proposed to scale up neural network capacity to improve performance. Recent works have successfully scaled models to billions of parameters via various forms of model parallelism [2, 21, 26, 27, 33]. Model parallelism [30] splits weights and tensors across multiple cores while pipeline parallelism [18, 24] splits different layers across devices with micro-batches pipelined to the different layers. To enable continued scaling of neural networks, improving model training and serving efficiency has become a critical research area.\n\nConditional Computation: Computation decisions can be made dynamically based on the input [23, 25]. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation, by activating certain parameters and computation on demand, on a per-example or per-token basis [3]. Conditional convolution layers [1] with task-specific gating has been used to combat catastrophic forgetting when a sequence of learning problems are optimized. The gating decisions may be binary or sparse and continuous, stochastic or deterministic.\n\nMixture of Experts: Sparsely-gated MoE [31] is the first model to demonstrate massive improvements in model capacity, training time, or model quality with gating. Switch Transformer [10] simplifies the gating by selecting only the top expert per token using a softmax over the hidden state and demonstrates better scaling than previous work. All the prior work requires an auxiliary loss to explicitly encourage balancing. This loss term has to be carefully weighted to not overwhelm the primary loss. However, auxiliary loss does not guarantee balancing and a hard capacity factor has to be imposed. As a result, many tokens can still be unprocessed by the MoE layer. Hard MoE [12] with a single decoding layer can be efficiently trained to good effect on large scale hashtag prediction tasks.\n\nBase Layers [22] formulate a linear assignment that maximizes token-expert affinities while ensuring each expert receives an equal number of tokens. Hash layers [8, 28] devise hashing techniques on input tokens. However, the evaluations are limited to pre-training perplexity. THOR [? ] randomly activates experts during training and inference and is trained with a consistency regularization loss.\n\nTHOR has demonstrated strong performance on translation tasks. Different from these prior works, our method is a learnt method that enables heterogeneous MoE and effectively improves downstream fine-tuning performance.\n\n3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n5 Conclusion\n\nWe propose a new routing method for sparsely activated mixture-of-experts (MoE) models. This method addresses load imbalance and under-utilization of experts in conventional MoE methods, and enables selecting different numbers of experts for each token. Our model demonstrates more than 2x training efficiency improvements when compared to the state-of-the-art GShard and Switch Transformer models, and also achieves strong gains when finetuning on 11 datasets in the GLUE and SuperGLUE benchmark.\n\n6 Limitations\n\nThe expert choice method might not immediately apply to auto-regressive text generation as our current implementation takes in the past and future tokens to perform the top-k selection. One possible solution is to collect a large batch of input sequences, dispatch tokens of the same sequence into separate groups, and perform expert choice routing for each group. Another scenario where the expert choice method does not immediately apply is when the batch size becomes very small during serving or inference. A global top-k can be selected instead and we can cap the number of times each expert or token gets selected. We leave these possible improvements for future work.\n\nAnother long-standing issue with MoE has been the large memory footprint. Even though computational cost can be reduced using sparsely gated networks, the total number of parameters increases linearly or sub-linearly with the number of experts. Increasing the number of experts requires reservation of a large number of hardware devices. Therefore, dynamic (used) power is saved while static (reserved) power is not. Power saving techniques such as the ability to put hardware devices into low power states while not in use [17] can help with reducing the reserved power requirements.\n\nReferences\n\n[1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pages 3930–3939. Computer Vision Foundation / IEEE, 2020.\n\n[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. In Advances in Neural Information Processing Systems.\n\n[3] Kyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning, 2014.\n\n[4] Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems, 2021.\n\n[5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.\n\nTransformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.\n\n[6] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 933–941. JMLR.org, 2017.\n\n[7] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixtureof-experts, 2021.\n\n[8] Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan.\n\nTricks for training sparse translation models, 2021.\n\n[9] Richard L Dykstra. An iterative procedure for obtaining i-projections onto the intersection of convex sets. The annals of Probability, pages 975–984, 1985.\n\n[10] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.\n\n[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. NAS-FPN: learning scalable feature pyramid architecture for object detection. In CVPR, pages 7036–7045. Computer Vision Foundation / IEEE, 2019.\n\n[12] Sam Gross, Marc'Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision, 2017.\n\n[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.\n\n[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 630–645, Cham, 2016. Springer International Publishing.\n\n[15] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2016.\n\n[16] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.\n\n[17] Ping Huang, Zuocheng Xing, Tianran Wang, Qiang Wei, Hongyan Wang, and Guitao Fu. A brief survey on power gating design. In 2010 10th IEEE International Conference on Solid-State and Integrated Circuit Technology, pages 788–790, 2010.\n\n[18] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 103–112, 2019.\n\n[19] Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David A. Patterson. A domain-specific supercomputer for training deep neural networks. Commun. ACM, 63(7):67–78, 2020.\n\n[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.\n\n[21] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.\n\n[22] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6265–6274. PMLR, 18–24 Jul 2021.\n\n[24] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. New York, NY, USA, 2019. Association for Computing Machinery.\n\n[25] Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, Cédric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. In ICLR. OpenReview.net, 2021.\n\n[26] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.\n\n[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.\n\n[28] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models, 2021.\n\n[30] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 10435–10444, Red Hook, NY, USA, 2018. Curran Associates Inc.\n\n[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E.\n\nHinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In ICLR (Poster). OpenReview.net, 2017.\n\n[32] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 10–15 Jul 2018. [23] Min Lin, Jie Fu, and Yoshua Bengio. Conditional computation for continual learning, 2019.\n\n[29] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.\n\n[33] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.\n\n[34] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc.\n\n[35] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.\n\nGLUE: A multi-task benchmark and analysis platform for natural language understanding.\n\nIn Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, November 2018. Association for Computational Linguistics.\n\n[36] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake A. Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen. GSPMD: general and scalable parallelization for ML computation graphs. CoRR, abs/2105.04663, 2021.\n\n7 Checklist\n\n(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Yes (b) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes (c) Did you discuss any potential negative societal impacts of your work? N/A. Not any. (d) Did you describe the limitations of your work? Yes (a) Did you include the code, data, and instructions needed to reproduce the main experimental results? Yes. We include details in the experiment setup to help reproduce the main results. (b) Did you specify all the training details? Yes (c) Did you report error bars? Yes (d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? Yes (a) If your work uses existing assets, did you cite the creators? Yes (b) Did you mention the license of the assets? No. The used dataset is not released yet. (c) Did you include any new assets either in the supplemental material or as a URL? No. The dataset is not released yet.\n\n(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? No. Not using persons' data. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes. The dataset does not contain any personally identifiable information or offensive content.\n\nA Comparison On Fine-Tuning With A Dense Model\n\nOur 8B MoE model achieves stronger pre-training perplexity than its dense counterpart. However, a better perplexity does not always directly translate to downstream performance as demonstrated in Section 4.4. To this end, we compare fine-tuning performance of the 8B dense model and MoE model in Table 1. As shown in the table, our MoE model using expert choice routing consistently outperforms the dense model across the 11 tasks in GLUE and SuperGLUE.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 1: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nB Capacity Factor\n\nWe evaluate the downstream task fine-tuning performance by varying the capacity factors. Note that a capacity factor of n indicates on average how many experts each token can be received. EC-CF2 is our baseline expert choice, which matches GShard top-2 gating computational footprint. EC-CF1, however, matches Switch Transformer top-1 gating computational footprint. EC-CF0.5 further verifies that an aggressively lowered capacity factor can provide strong enough performance, that almost matches the top-2 gating baseline.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Top-2 78.1 87.0 88.3 85.0 82.6 90.1 90.7 81.6 94.7 68.2 67.2 83.0±0.3 EC-CAP2 78.2 88.0 88.5 85.7 83.0 90.8 91.1 80.0 95.4 70.4 64.1 83.2±0.4 EC-CAP3 78.5 91.7 89.3 86.3 83.5 90.9 91.1 81.8 94.9 70.0 65.6 84.0±0.4 EC-CF2 79.1 89.6 89.3 86.8 84.3 91.3 91.2 81.1 95.2 68.1 68.0 84.0±0.2 EC-CF1 77.4 90.6 88.0 85.5 83.6 90.3 91.2 79.8 95.3 66.5 64.9 83.0±0.2 EC-CF0.5 77.4 89.6 86.3 85.2 82.7 91.7 91.0 79.6 94.9 67.3 63.5 83.0 ±0.05 Hash Layers 76.1 85.2 86.7 83.4 82.5 90.0 90.3 75.7 94.0 67.4 63.3 81.3±1.0\n\nTable 2: Comparison between different routing methods in fine-tuning of 100M/64E models. We perform 3 independent fine-tuning runs for each method and report the average results. This gives more accurate difference between the variants of expert choice method, since they achieve close fine-tuning results. We do not report averaged results in other experiments.\n\nC Capped Expert Choice\n\nAs described in Section 4.5, the maximum number of experts each token is assigned can be capped by an entropy-regularized linear programming. Figure 1 compares the validation perplexity when training the 100M/64E models using the base expert choice method (EC-BASE), expert choice capped by two experts per token (EC-CAP2), expert choice capped by three experts per token (EC-CAP3), and GShard top-2 gating.\n\nAs shown in the figure, restricting the number of experts to 2 degrades the perplexity compared to the base expert choice method. This suggests that a more flexible allocation of experts (e.g. more than 2 experts for a token) can enhance model expressiveness. On the other hand, our EC-CAP2 and EC-CAP3 methods still outperform the top-2 gating method by a clear margin. We believe this confirms the effectiveness of a load balanced training, provided by our method. Finally, EC-CAP3 obtains comparable perplexity to EC-BASE. As indicated by Figure 3, only a little fraction of tokens use more than 3 experts therefore we see little or no difference between EC-BASE and EC-CAP3 variants. We present the fine-tuning results of these methods in Table 2.\n\nD Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [? ]. We use mod x to map a token ID to an expert ID. This in some way ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 2. Hashing based routing performs much worse than expert choice in terms of average scores and variance.\n\nE Fine-Tuning Details\n\nWe did a hyperparameter search for both baseline models and expert choice method. For fine-tuning of the 8B dense model, we use a constant learning rate of 0.0001 and a dropout rate of 0.1. We freeze the attention layer and feed-forward layer while leaving the embedding and layer normalization trainable. This setting has been found optimal for the 8B dense model. For MoE 8B/64E models including GShard top-2 gating and expert choice, we found continuing the learning rate from the pre-trained model while using a square root learning rate decay works better. In addition, we do not apply parameter freezing for fine-tuning MoE models. For models with 100M expert size, we use a constant learning rate of 0.0001 and no dropout is used.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/2202.09368v2.md"
+          },
+          "headlines": {
+            "1 Introduction": [],
+            "2 Related Work": [],
+            "3 Method": [
+              "3.1 Pitfalls Of Token-Choice Routing",
+              "3.2 Heterogeneous Moe Via Expert Choice",
+              "3.3 Expert Choice With Additional Constraint",
+              "3.4 Model Architecture"
+            ],
+            "4 Experiments": [
+              "4.1 Setup",
+              "4.2 Training Efficiency",
+              "4.3 Scaling The Number Of Experts",
+              "4.4 Fine-Tuning On Glue And Superglue",
+              "4.5 Heterogeneity Matters",
+              "4.6 Comparison With Hash Layer",
+              "4.7 Ablation"
+            ],
+            "5 Conclusion": [],
+            "6 Limitations": [],
+            "A Comparison On Fine-Tuning With A Dense Model": [],
+            "B Capacity Factor": [],
+            "C Capped Expert Choice": [],
+            "D Comparison With Hash Layer": [],
+            "E Fine-Tuning Details": []
+          },
+          "summary": "The paper introduces a new routing method for sparsely-activated Mixture-of-Experts (MoE) models called 'expert choice,' which addresses load imbalance and under-utilization issues in conventional MoE methods. Unlike traditional token-choice routing, expert choice allows experts to select the top-k tokens, ensuring perfect load balancing and enabling variable expert allocation per token. This method improves training efficiency, achieving over 2x faster convergence compared to existing models like Switch Transformer and GShard. It also demonstrates superior performance on downstream tasks in the GLUE and SuperGLUE benchmarks. The study highlights the benefits of flexible expert allocation and shows that expert choice outperforms hashing-based routing methods. The paper also discusses potential limitations and future improvements for the expert choice method.",
+          "summary_embedding": [
+            -0.009726986289024353,
+            -0.0337262786924839,
+            -0.023302342742681503,
+            -0.019878201186656952,
+            -0.025953780859708786,
+            0.04560471698641777,
+            -0.006874797400087118,
+            0.039665497839450836,
+            -0.006973279174417257,
+            0.034544438123703,
+            -0.025787118822336197,
+            -0.00363057479262352,
+            -0.01052999310195446,
+            0.0054657477885484695,
+            -0.024499278515577316,
+            0.036786794662475586,
+            0.010007280856370926,
+            0.003272630739957094,
+            0.006102093029767275,
+            -0.04048365354537964,
+            -0.018666116520762444,
+            -0.013522328808903694,
+            -0.03230207785964012,
+            0.008969432674348354,
+            0.02484775334596634,
+            0.0016713523073121905,
+            0.0026230288203805685,
+            0.003624893259257078,
+            -0.05311964824795723,
+            0.04339266195893288,
+            0.018681267276406288,
+            0.023711422458291054,
+            -0.016332851722836494,
+            -0.012832955457270145,
+            -0.039120059460401535,
+            -0.0197418425232172,
+            0.030786970630288124,
+            0.039968520402908325,
+            -0.019287310540676117,
+            0.03357476741075516,
+            0.018060073256492615,
+            0.02454473078250885,
+            -0.021272100508213043,
+            -0.029817303642630577,
+            -0.0061892117373645306,
+            -0.007840678095817566,
+            -0.008355814032256603,
+            0.01242387667298317,
+            0.016984347254037857,
+            -0.013007191941142082,
+            0.0212569497525692,
+            -0.01371171697974205,
+            0.03384748846292496,
+            0.008916404098272324,
+            -0.042332086712121964,
+            0.03984731063246727,
+            -0.008007340133190155,
+            0.012105703353881836,
+            0.03281721472740173,
+            0.010333028621971607,
+            -0.005287723150104284,
+            -0.021544819697737694,
+            -0.03378688171505928,
+            0.0010529992869123816,
+            0.00014014738553669304,
+            0.005950582213699818,
+            -0.003069985192269087,
+            -0.009742137044668198,
+            0.03227177634835243,
+            -0.003939277958124876,
+            -0.0740584209561348,
+            0.050089433789253235,
+            -0.026590125635266304,
+            -0.007628562860190868,
+            -0.028938541188836098,
+            -0.01736312545835972,
+            0.037362534552812576,
+            0.005662711802870035,
+            0.0098709212616086,
+            0.0106284748762846,
+            -0.0007892759749665856,
+            0.028923390433192253,
+            0.00910579226911068,
+            -0.024923507124185562,
+            0.04478655755519867,
+            -0.04790768027305603,
+            0.01801462098956108,
+            -0.04336235672235489,
+            0.022756904363632202,
+            1.6393929399782792e-05,
+            -0.020726662129163742,
+            -0.0013304532039910555,
+            -0.016666175797581673,
+            0.0218023881316185,
+            -0.01730252057313919,
+            0.026802239939570427,
+            0.013825350441038609,
+            -0.015802564099431038,
+            -0.029817303642630577,
+            0.025378039106726646,
+            0.003367325058206916,
+            0.010499690659344196,
+            -0.02075696364045143,
+            0.014332910999655724,
+            0.04393810033798218,
+            -0.02593863010406494,
+            -0.02271145209670067,
+            0.027681002393364906,
+            0.014408666640520096,
+            0.0633314698934555,
+            0.036544378846883774,
+            0.006624804809689522,
+            0.022362977266311646,
+            0.020575150847434998,
+            -0.028968842700123787,
+            -0.01963578537106514,
+            -0.046362269669771194,
+            -0.04757435619831085,
+            0.019878201186656952,
+            -0.04566532000899315,
+            0.0050869714468717575,
+            0.038544319570064545,
+            -0.013704141601920128,
+            -0.014446543529629707,
+            -0.04021093621850014,
+            0.017847958952188492,
+            0.0003908502112608403,
+            0.011976920068264008,
+            -0.0022082682698965073,
+            0.00033285003155469894,
+            -0.011825408786535263,
+            0.008878526277840137,
+            -0.03778676688671112,
+            -0.025196226313710213,
+            0.025181075558066368,
+            0.022862963378429413,
+            0.024120500311255455,
+            0.028802180662751198,
+            -0.03077181987464428,
+            -0.009984553791582584,
+            0.021378157660365105,
+            -0.04242299124598503,
+            -0.006556625012308359,
+            0.011332998983561993,
+            0.0026703758630901575,
+            0.003872992005199194,
+            0.032786913216114044,
+            0.052998438477516174,
+            -0.03172633796930313,
+            0.02065090648829937,
+            -0.010431510396301746,
+            0.04957429692149162,
+            0.00024691506405360997,
+            0.028968842700123787,
+            -0.019514575600624084,
+            -0.0134389977902174,
+            -0.029529431834816933,
+            -0.009795165620744228,
+            -0.01356020662933588,
+            -0.008484598249197006,
+            -0.002998017705976963,
+            -0.009575475007295609,
+            0.0040453351102769375,
+            0.0331808403134346,
+            0.00273855566047132,
+            0.030680913478136063,
+            0.014325335621833801,
+            0.0051475754007697105,
+            -0.08745196461677551,
+            0.026059838011860847,
+            -0.0033616432920098305,
+            0.005859675817191601,
+            0.0062460280023515224,
+            0.03013547509908676,
+            -0.0312718041241169,
+            0.020938776433467865,
+            0.026620427146553993,
+            0.0011145505122840405,
+            0.016969196498394012,
+            0.036847397685050964,
+            -0.04181694984436035,
+            0.009643655270338058,
+            0.012590537779033184,
+            -0.0033881578128784895,
+            0.04584713280200958,
+            0.011628445237874985,
+            0.019514575600624084,
+            0.018529755994677544,
+            0.003569970605894923,
+            0.00297907879576087,
+            0.006533898413181305,
+            0.03724132478237152,
+            0.003462019143626094,
+            -0.009833043441176414,
+            0.014976831153035164,
+            -0.010757259093225002,
+            -0.011219366453588009,
+            -0.037998881191015244,
+            0.014454119838774204,
+            0.008742166683077812,
+            -0.017454031854867935,
+            -0.02959003672003746,
+            0.0026949963066726923,
+            -0.031908150762319565,
+            0.026559824123978615,
+            -0.002445003716275096,
+            0.005738467443734407,
+            -0.010454237461090088,
+            -0.006757376715540886,
+            -0.035423196852207184,
+            0.028453707695007324,
+            -0.02095392718911171,
+            0.0053142374381423,
+            0.0007258309051394463,
+            -0.029438525438308716,
+            0.0060073984786868095,
+            0.017863109707832336,
+            -0.0025965142995119095,
+            0.0025302283465862274,
+            -0.02349930815398693,
+            0.024468975141644478,
+            0.025635607540607452,
+            0.004518806003034115,
+            -0.014037464745342731,
+            -0.013113250024616718,
+            -0.003605954349040985,
+            -0.009742137044668198,
+            -0.016196493059396744,
+            0.024468975141644478,
+            -0.014810169115662575,
+            0.00308892410248518,
+            0.021847840398550034,
+            0.014264730736613274,
+            -0.007045246660709381,
+            -0.004007457755506039,
+            -0.04066546633839607,
+            -0.01218903437256813,
+            -0.04760465770959854,
+            -0.011628445237874985,
+            -0.0034601252991706133,
+            -0.028696123510599136,
+            0.0061513339169323444,
+            -0.04481685906648636,
+            -0.01235569640994072,
+            0.02309022843837738,
+            0.018135828897356987,
+            0.0019488062243908644,
+            -0.009173971600830555,
+            -0.022605394944548607,
+            8.486965816700831e-05,
+            -0.01763584464788437,
+            -0.017044952139258385,
+            0.015393485315144062,
+            0.031059689819812775,
+            -0.03101423755288124,
+            0.04024123772978783,
+            0.003893824527040124,
+            -0.08066429197788239,
+            0.010393633507192135,
+            -0.026362858712673187,
+            0.07011914253234863,
+            -0.0025453795678913593,
+            0.014075342565774918,
+            0.06781618297100067,
+            -0.012431452050805092,
+            -0.02007516659796238,
+            0.03142331540584564,
+            -0.008249756880104542,
+            -1.1311515663692262e-05,
+            -0.02884763479232788,
+            -0.02739313244819641,
+            -0.053271155804395676,
+            -0.009704259224236012,
+            -0.010787560604512691,
+            0.03360506892204285,
+            -0.024317465722560883,
+            -0.0007656024536117911,
+            0.012620840221643448,
+            0.02983245439827442,
+            0.016635872423648834,
+            -0.0015681355725973845,
+            0.008272483013570309,
+            0.02881733328104019,
+            0.0274234339594841,
+            -0.0036154238041490316,
+            0.00814369972795248,
+            -0.034211114048957825,
+            -0.0068066176027059555,
+            -0.0493621826171875,
+            0.011302697472274303,
+            -0.02596893161535263,
+            -0.02596893161535263,
+            0.007643714081496,
+            -0.011476934887468815,
+            0.03748374432325363,
+            -0.024696242064237595,
+            -0.007503566797822714,
+            0.005037730094045401,
+            0.012855681590735912,
+            -0.017757052555680275,
+            -0.03612014651298523,
+            -0.0028502948116511106,
+            0.0026381798088550568,
+            0.007560383062809706,
+            -0.053240854293107986,
+            0.004499867558479309,
+            -0.018469152972102165,
+            0.0279688723385334,
+            -0.02139330841600895,
+            -0.013688990846276283,
+            0.042029064148664474,
+            0.011363301426172256,
+            0.027499189600348473,
+            -0.026468917727470398,
+            0.0017849853029474616,
+            0.012825379148125648,
+            -0.0020056227222085,
+            0.032514192163944244,
+            -0.030574856325984,
+            -0.03257479891180992,
+            0.001548249856568873,
+            -0.007840678095817566,
+            -0.0625436082482338,
+            -0.011386028490960598,
+            -0.00981789268553257,
+            0.017317671328783035,
+            0.0446956530213356,
+            0.04557441547513008,
+            -0.01068150345236063,
+            0.002174178371205926,
+            -0.02213571034371853,
+            -0.04542290419340134,
+            -0.02643861435353756,
+            0.0016486257081851363,
+            -0.019999410957098007,
+            -0.02789311669766903,
+            -0.0018313854234293103,
+            0.011211791075766087,
+            -0.013651113025844097,
+            0.01134057529270649,
+            -0.018560059368610382,
+            -0.01862066239118576,
+            -0.012332970276474953,
+            0.018575210124254227,
+            0.009901223704218864,
+            -0.0018796793883666396,
+            -0.015037435106933117,
+            -0.00980274099856615,
+            0.018969137221574783,
+            -0.011279970407485962,
+            0.0009388927719555795,
+            0.01188601367175579,
+            0.024347767233848572,
+            -0.007083124481141567,
+            0.010961798019707203,
+            -0.034756552428007126,
+            -0.01966608688235283,
+            0.007984613068401814,
+            0.002719616750255227,
+            0.020711511373519897,
+            0.008742166683077812,
+            -0.04275631532073021,
+            -0.044301725924015045,
+            0.016044981777668,
+            -0.0562104657292366,
+            0.028120383620262146,
+            -0.03327174484729767,
+            -0.012514783069491386,
+            -0.02671133354306221,
+            0.0013418165035545826,
+            -0.01828734017908573,
+            0.022438732907176018,
+            0.03778676688671112,
+            -0.0011486404109746218,
+            0.005503625608980656,
+            0.014658658765256405,
+            0.04384719207882881,
+            -0.05021063983440399,
+            -0.041180603206157684,
+            0.03754434734582901,
+            -0.01601468026638031,
+            0.0185146052390337,
+            0.00021744151308666915,
+            0.010348179377615452,
+            -0.014431392773985863,
+            0.022772056981921196,
+            0.013757170177996159,
+            -0.008764892816543579,
+            -0.030574856325984,
+            0.04945308715105057,
+            -0.023650817573070526,
+            -0.049513693898916245,
+            0.0018323323456570506,
+            0.018878230825066566,
+            0.018635815009474754,
+            -0.014870774000883102,
+            -0.02907489985227585,
+            -0.024317465722560883,
+            -0.006060427520424128,
+            -0.00947699323296547,
+            -0.03402930125594139,
+            0.011363301426172256,
+            0.0048445542342960835,
+            0.016666175797581673,
+            -0.00645056739449501,
+            -0.0073141781613230705,
+            -0.009370936080813408,
+            0.022559940814971924,
+            0.012635990977287292,
+            -0.03175663948059082,
+            -0.017484333366155624,
+            -0.02528713271021843,
+            0.004359719809144735,
+            0.010136065073311329,
+            0.022166013717651367,
+            0.024287162348628044,
+            -0.0323626808822155,
+            0.04218057543039322,
+            0.01550711877644062,
+            -0.008727015927433968,
+            -0.0019317612750455737,
+            0.02478714846074581,
+            0.04324115067720413,
+            0.0218023881316185,
+            0.026514369994401932,
+            0.002107892418280244,
+            8.705945947440341e-05,
+            -0.015620751306414604,
+            -0.0011391709558665752,
+            0.01862066239118576,
+            0.04911976307630539,
+            -0.012151156552135944,
+            0.01716616004705429,
+            -0.03578682243824005,
+            0.00396200455725193,
+            -0.02580226957798004,
+            -0.030786970630288124,
+            0.03745344281196594,
+            -0.015060162171721458,
+            0.0191964041441679,
+            0.025044716894626617,
+            -0.022469034418463707,
+            -0.011075431481003761,
+            -0.019423669204115868,
+            -0.026181045919656754,
+            -0.01702980138361454,
+            -0.03281721472740173,
+            -0.018135828897356987,
+            -0.002653330797329545,
+            -0.035635314881801605,
+            -0.04987731948494911,
+            -0.004329417832195759,
+            -0.046392571181058884,
+            -0.010507266037166119,
+            0.017378276214003563,
+            -0.011166337877511978,
+            -0.025711363181471825,
+            -0.005336964037269354,
+            0.003973367623984814,
+            -0.022196315228939056,
+            -0.0066588944755494595,
+            0.008439145050942898,
+            0.035938333719968796,
+            -0.02298417128622532,
+            -0.013060221448540688,
+            -0.0024317465722560883,
+            -0.02887793630361557,
+            -0.007878555916249752,
+            0.008999735116958618,
+            0.007723257411271334,
+            -0.0012357590021565557,
+            -0.05196816474199295,
+            -0.006916462909430265,
+            -0.012992041185498238,
+            0.029196109622716904,
+            0.026468917727470398,
+            0.0012168202083557844,
+            -0.02783251367509365,
+            0.008749742060899734,
+            -0.0072535742074251175,
+            0.02339324913918972,
+            -0.007992188446223736,
+            -0.022908415645360947,
+            -0.02034788578748703,
+            0.030786970630288124,
+            -0.0159086212515831,
+            0.009113367646932602,
+            0.009976978413760662,
+            0.0121360057964921,
+            -0.03899884968996048,
+            -0.001445980160497129,
+            0.0005648507503792644,
+            -0.010128489695489407,
+            0.00423093605786562,
+            -0.006738437805324793,
+            -0.007079336792230606,
+            -0.022090258076786995,
+            -0.0218023881316185,
+            0.03818069398403168,
+            -0.009113367646932602,
+            -0.0035661826841533184,
+            -0.01088604237884283,
+            0.03436262533068657,
+            -0.00595815759152174,
+            -0.01993880607187748,
+            -0.019347915425896645,
+            0.014431392773985863,
+            0.05105910077691078,
+            -0.028529461473226547,
+            -0.03984731063246727,
+            0.0011136035900563002,
+            0.030150625854730606,
+            0.004140029661357403,
+            -0.016908591613173485,
+            0.0035851215943694115,
+            0.056392278522253036,
+            -0.011742077767848969,
+            0.028362801298499107,
+            0.027771908789873123,
+            0.02295386977493763,
+            -0.02021152526140213,
+            -0.01678738370537758,
+            -0.020999381318688393,
+            0.011098157614469528,
+            -0.03027183562517166,
+            0.027059808373451233,
+            0.018969137221574783,
+            0.01538590993732214,
+            0.05054396390914917,
+            -0.03178694099187851,
+            -0.02515077404677868,
+            0.008499749936163425,
+            0.0021722845267504454,
+            0.011007251217961311,
+            -0.007727045100182295,
+            -0.01801462098956108,
+            -0.001950700068846345,
+            -0.014022313989698887,
+            -0.019817598164081573,
+            -0.020938776433467865,
+            0.00954517349600792,
+            0.015696506947278976,
+            0.002649543108418584,
+            0.010863316245377064,
+            0.0016590419691056013,
+            0.01686313934624195,
+            0.01321930717676878,
+            -0.0444229319691658,
+            -7.936555630294606e-05,
+            0.03378688171505928,
+            -0.0293476190418005,
+            -0.012264790013432503,
+            -0.016060132533311844,
+            -0.004439263138920069,
+            -0.01706010289490223,
+            -0.02762039750814438,
+            -0.006874797400087118,
+            -0.01889338344335556,
+            -0.00814369972795248,
+            -0.028196139261126518,
+            -0.009992130100727081,
+            0.01949942484498024,
+            0.022332675755023956,
+            0.001196934375911951,
+            -0.006783891003578901,
+            -0.005901341326534748,
+            0.02092362567782402,
+            0.032514192163944244,
+            0.01952972821891308,
+            0.0058331615291535854,
+            -0.015158643946051598,
+            -0.004295328166335821,
+            -0.012370847165584564,
+            0.016060132533311844,
+            0.012537509202957153,
+            6.344509893096983e-05,
+            0.03493836522102356,
+            0.004170331638306379,
+            -0.022726602852344513,
+            -0.001230077352374792,
+            0.024923507124185562,
+            -0.021590271964669228,
+            -0.0108330138027668,
+            0.040029123425483704,
+            0.0085679292678833,
+            -0.0024506852496415377,
+            -0.019287310540676117,
+            -0.019075196236371994,
+            0.02630225569009781,
+            -0.02133270539343357,
+            0.002083271974697709,
+            0.0482713058590889,
+            0.00035344602656550705,
+            -0.029802151024341583,
+            0.017954016104340553,
+            -0.013242034241557121,
+            -0.018560059368610382,
+            0.01848430372774601,
+            0.0012859469279646873,
+            -0.039968520402908325,
+            0.019014591351151466,
+            0.018408548086881638,
+            0.022635696455836296,
+            -0.02918095886707306,
+            0.0002127068000845611,
+            0.03233237937092781,
+            0.004037759732455015,
+            0.001595596899278462,
+            -0.008530051447451115,
+            -0.0006065162015147507,
+            0.010431510396301746,
+            0.001746160676702857,
+            0.02854461409151554,
+            -0.0030434709042310715,
+            0.027771908789873123,
+            -0.006727074272930622,
+            -0.008863375522196293,
+            0.019181253388524055,
+            -0.02109028771519661,
+            0.013166278600692749,
+            0.0685434341430664,
+            0.0021950111258774996,
+            0.025847723707556725,
+            0.007181606721132994,
+            0.024347767233848572,
+            0.02018122375011444,
+            -0.027271922677755356,
+            -0.02021152526140213,
+            -0.005397567991167307,
+            -0.0014033677289262414,
+            -0.03787767142057419,
+            0.011128460057079792,
+            -0.021302402019500732,
+            -0.013249609619379044,
+            -0.0022347825579345226,
+            -0.002316219499334693,
+            -0.006405114196240902,
+            0.034211114048957825,
+            -0.0014630251098424196,
+            -0.018560059368610382,
+            -0.035423196852207184,
+            0.037726160138845444,
+            0.04048365354537964,
+            0.0389382466673851,
+            -0.005984671879559755,
+            0.03548380360007286,
+            0.0030756669584661722,
+            0.0025302283465862274,
+            0.041453324258327484,
+            0.004791525658220053,
+            -0.011567841283977032,
+            0.02295386977493763,
+            0.012416300363838673,
+            0.03351416438817978,
+            -0.02045394293963909,
+            -0.02630225569009781,
+            -0.003299145260825753,
+            -0.026423463597893715,
+            0.012492056004703045,
+            -0.019726691767573357,
+            0.0018768386216834188,
+            0.007117214612662792,
+            0.015196521766483784,
+            -0.016044981777668,
+            0.014719262719154358,
+            -0.006102093029767275,
+            -0.021014532074332237,
+            0.011719351634383202,
+            -0.007780073676258326,
+            0.011961768381297588,
+            -0.02262054570019245,
+            0.001138224033638835,
+            -0.04063516482710838,
+            0.008280059322714806,
+            0.014022313989698887,
+            -0.01598437689244747,
+            0.03205966204404831,
+            0.020969079807400703,
+            -0.01801462098956108,
+            -0.0009682480012997985,
+            0.0034714885987341404,
+            -0.010393633507192135,
+            -0.011067855171859264,
+            -0.019923655316233635,
+            0.02786281518638134,
+            -0.015802564099431038,
+            -0.025908326730132103,
+            0.02454473078250885,
+            -0.01743887923657894,
+            0.015151068568229675,
+            -0.004299115855246782,
+            0.00888610165566206,
+            -0.019120648503303528,
+            0.02284781076014042,
+            -0.023741723969578743,
+            -0.0009128518868237734,
+            -0.020287280902266502,
+            0.0028389315120875835,
+            0.014135947450995445,
+            -0.018272189423441887,
+            -0.02139330841600895,
+            0.019150950014591217,
+            -0.005420294590294361,
+            -0.003931702114641666,
+            0.010014856234192848,
+            0.0030983935575932264,
+            -0.0029715034179389477,
+            -0.004549108445644379,
+            -0.03281721472740173,
+            -0.02915065549314022,
+            0.0033086147159337997,
+            0.011386028490960598,
+            0.015257125720381737,
+            0.027574945241212845,
+            0.03715042024850845,
+            0.006052851676940918,
+            0.0036949669010937214,
+            0.0009327376610599458,
+            -0.020726662129163742,
+            0.042635105550289154,
+            -0.020469093695282936,
+            0.040847279131412506,
+            0.032544493675231934,
+            -0.0197418425232172,
+            -0.023908386006951332,
+            0.0029052174650132656,
+            0.03115059621632099,
+            0.03151422366499901,
+            0.025211378931999207,
+            0.012310243211686611,
+            0.03690800443291664,
+            -0.004359719809144735,
+            -0.032211173325777054,
+            0.015802564099431038,
+            0.022256920114159584,
+            -0.008105821907520294,
+            0.009673957712948322,
+            0.027438584715127945,
+            0.013279912061989307,
+            -0.007757347077131271,
+            0.030150625854730606,
+            0.0012234487803652883,
+            0.02396899089217186,
+            0.02498411200940609,
+            -0.02190844528377056,
+            0.01743887923657894,
+            -0.0001154677156591788,
+            0.006727074272930622,
+            -0.009840618818998337,
+            0.004348356742411852,
+            0.011810258030891418,
+            0.018272189423441887,
+            0.018908534198999405,
+            0.003246116451919079,
+            -0.07060398161411285,
+            0.019726691767573357,
+            0.004507442936301231,
+            -0.02160542458295822,
+            0.00390140013769269,
+            -0.0024866692256182432,
+            0.012704171240329742,
+            0.02105998620390892,
+            0.012461753562092781,
+            -0.01790856383740902,
+            0.0013626492582261562,
+            0.011961768381297588,
+            -0.00595815759152174,
+            0.04730163514614105,
+            0.0019791084341704845,
+            -0.01437078882008791,
+            -0.016105586662888527,
+            0.013726867735385895,
+            0.02525683119893074,
+            -0.02355991117656231,
+            0.006325571332126856,
+            0.005621046293526888,
+            -0.004147605039179325,
+            -0.010742107406258583,
+            -0.030377892777323723,
+            -0.005280147306621075,
+            -0.03436262533068657,
+            0.023150833323597908,
+            0.015491968020796776,
+            0.014764715917408466,
+            -0.025044716894626617,
+            -0.004280176945030689,
+            0.0046400148421525955,
+            -0.021287251263856888,
+            0.003261267440393567,
+            0.01161329448223114,
+            0.0005104016163386405,
+            -0.020984230563044548,
+            0.00959820207208395,
+            0.013469300232827663,
+            0.06848283112049103,
+            0.0007442962378263474,
+            0.029771849513053894,
+            0.0422411784529686,
+            -0.0012243957025930285,
+            -0.006371024530380964,
+            0.0036286809481680393,
+            -0.019953956827521324,
+            0.003473382443189621,
+            0.025878025218844414,
+            -0.03224147483706474,
+            -0.055058982223272324,
+            -0.007545232307165861,
+            -0.04624105989933014,
+            0.001489539397880435,
+            0.014711687341332436,
+            0.023044776171445847,
+            -0.0037782976869493723,
+            -0.034817155450582504,
+            0.016908591613173485,
+            -0.003071879269555211,
+            0.007014944683760405,
+            -0.02593863010406494,
+            -0.0294839795678854,
+            -0.026014383882284164,
+            -0.02339324913918972,
+            -0.012908710166811943,
+            -0.022726602852344513,
+            0.012590537779033184,
+            -0.0015103721525520086,
+            -0.06587684899568558,
+            -0.009780014865100384,
+            0.016166189685463905,
+            0.001200722181238234,
+            0.011939042247831821,
+            0.011083006858825684,
+            -0.007083124481141567,
+            -0.0018275976181030273,
+            -0.022559940814971924,
+            0.03124150261282921,
+            0.005177877843379974,
+            0.03584742918610573,
+            -0.006503595970571041,
+            -0.009128518402576447,
+            0.01073453202843666,
+            0.04527139291167259,
+            0.0017916138749569654,
+            0.010226971469819546,
+            -0.020560000091791153,
+            0.0015065843472257257,
+            -0.02284781076014042,
+            -0.0031135445460677147,
+            -0.008863375522196293,
+            -0.0021211495622992516,
+            0.015893470495939255,
+            0.0019753205124288797,
+            -0.010204244405031204,
+            0.025438643991947174,
+            0.02251448854804039,
+            -0.009605777449905872,
+            -0.015969226136803627,
+            -0.02437806874513626,
+            -0.014545026235282421,
+            -0.0002627763315103948,
+            -0.010007280856370926,
+            -0.022453883662819862,
+            -0.020878173410892487,
+            -0.002566212322562933,
+            -0.02559015527367592,
+            0.022529639303684235,
+            0.01220418605953455,
+            -4.882668508798815e-05,
+            -0.006723286584019661,
+            -0.038544319570064545,
+            -0.003727162955328822,
+            -0.03118089959025383,
+            -0.005374841392040253,
+            0.03287781774997711,
+            -0.03745344281196594,
+            -0.0002231231628684327,
+            0.014491996727883816,
+            -0.01993880607187748,
+            0.0028938541654497385,
+            0.008961857296526432,
+            0.02769615314900875,
+            -0.027408283203840256,
+            0.0001795638381736353,
+            0.006954340264201164,
+            0.016908591613173485,
+            -0.009878496639430523,
+            0.027196168899536133,
+            -0.04275631532073021,
+            0.011037553660571575,
+            -0.037120118737220764,
+            -0.020423639565706253,
+            -0.01523439958691597,
+            0.008363389410078526,
+            -0.005018791649490595,
+            -0.02637801133096218,
+            -0.010454237461090088,
+            0.006026337388902903,
+            -0.00693918950855732,
+            -0.01902974210679531,
+            -0.010620899498462677,
+            0.027438584715127945,
+            0.00931033119559288,
+            0.007977037690579891,
+            -0.02955973520874977,
+            -0.01631770096719265,
+            0.006280118133872747,
+            0.052089374512434006,
+            -0.01584801822900772,
+            0.014938953332602978,
+            -0.006639956030994654,
+            0.02342355251312256,
+            -0.028484009206295013,
+            -0.01872672140598297,
+            0.0315445251762867,
+            0.013946558348834515,
+            0.007064185570925474,
+            0.028953691944479942,
+            0.027196168899536133,
+            0.027105262503027916,
+            -0.04654408246278763,
+            0.009886072017252445,
+            -0.002613559365272522,
+            -0.008757317438721657,
+            0.03402930125594139,
+            0.01916610263288021,
+            0.022150862962007523,
+            0.0009744031121954322,
+            -0.01730252057313919,
+            0.0139844361692667,
+            0.014885924756526947,
+            0.007681591901928186,
+            -0.014242004603147507,
+            -0.02660527639091015,
+            0.003367325058206916,
+            -0.007992188446223736,
+            0.00047276070108637214,
+            0.020726662129163742,
+            -0.009052763693034649,
+            0.030999086797237396,
+            -0.014605630189180374,
+            -0.02593863010406494,
+            0.008916404098272324,
+            -0.02268115058541298,
+            -0.0041741193272173405,
+            0.0004476667381823063,
+            0.010083036497235298,
+            -0.023847781121730804,
+            -0.04669559374451637,
+            0.019408518448472023,
+            0.028393102809786797,
+            0.005579380784183741,
+            0.01328748743981123,
+            -0.035332292318344116,
+            0.008992158807814121,
+            0.012310243211686611,
+            0.011348150670528412,
+            0.0019847899675369263,
+            0.035362593829631805,
+            -0.00028834378463216126,
+            0.012885984033346176,
+            -0.008620957843959332,
+            0.028362801298499107,
+            0.00667025800794363,
+            -0.0323626808822155,
+            -0.0378473699092865,
+            -0.01513591781258583,
+            0.015257125720381737,
+            0.006692984607070684,
+            -0.011840560473501682,
+            -0.02278720773756504,
+            -0.038817036896944046,
+            0.015787413343787193,
+            -0.012916285544633865,
+            -0.038271598517894745,
+            -0.010158791206777096,
+            -0.009742137044668198,
+            0.016635872423648834,
+            -0.006916462909430265,
+            -0.016802534461021423,
+            -0.016090434044599533,
+            0.005821797996759415,
+            7.161247049225494e-05,
+            0.0244083721190691,
+            -0.001251857029274106,
+            -0.006855858489871025,
+            -0.016393456608057022,
+            -0.01188601367175579,
+            -0.0035074723418802023,
+            0.03602924197912216,
+            -0.014219277538359165,
+            -0.02383263036608696,
+            0.014984406530857086,
+            0.02657497487962246,
+            0.011636020615696907,
+            0.03227177634835243,
+            0.007181606721132994,
+            -0.014022313989698887,
+            0.003016956616193056,
+            -0.033029329031705856,
+            -0.0046400148421525955,
+            0.005196816753596067,
+            -0.006427840795367956,
+            -0.015651054680347443,
+            -0.01815097965300083,
+            -0.015832867473363876,
+            0.030756669119000435,
+            -0.02487805485725403,
+            0.0218023881316185,
+            0.030347591266036034,
+            0.003174148965626955,
+            -0.02915065549314022,
+            0.013598084449768066,
+            -0.0072005451656877995,
+            0.01862066239118576,
+            0.0019014590652659535,
+            0.028014326468110085,
+            -0.007461901288479567,
+            -0.004818039946258068,
+            -0.02501441352069378,
+            -0.03696860745549202,
+            -0.023484155535697937,
+            -0.008817922323942184,
+            0.0027537066489458084,
+            0.020984230563044548,
+            -0.0375746488571167,
+            -0.017757052555680275,
+            0.009855770505964756,
+            0.015757111832499504,
+            -0.03027183562517166,
+            -0.0025908327661454678,
+            -0.010158791206777096,
+            -0.0004055278259329498,
+            0.005140000022947788,
+            -0.02227207086980343,
+            -0.008977008052170277,
+            -0.015726808458566666,
+            -0.0010435298318043351,
+            0.006166485138237476,
+            -0.0007409819518215954,
+            -0.05805889517068863,
+            -0.013969285413622856,
+            0.018635815009474754,
+            0.0026400736533105373,
+            0.008908828720450401,
+            0.015938924625515938,
+            -0.0111739132553339,
+            0.0031533162109553814,
+            -0.026756787672638893,
+            0.023893235251307487,
+            -0.009492144919931889,
+            0.009355784393846989,
+            -0.013742019422352314,
+            0.012514783069491386,
+            0.014014738611876965,
+            0.004795313347131014,
+            0.0010681503918021917,
+            -0.0054051438346505165,
+            0.01601468026638031,
+            0.028377952054142952,
+            -0.023756874725222588,
+            0.011454207822680473,
+            -0.0002514130319468677,
+            -0.0032158142421394587,
+            -0.029196109622716904,
+            0.00509454682469368,
+            0.003931702114641666,
+            -0.005109698045998812,
+            0.016651025041937828,
+            -0.036544378846883774,
+            0.01662072166800499,
+            0.02177208475768566,
+            0.032544493675231934,
+            0.014832896180450916,
+            -0.017287369817495346,
+            -0.011658747680485249,
+            -0.012385998852550983,
+            -0.002624922664836049,
+            0.022181164473295212,
+            0.012120855040848255,
+            -0.01815097965300083,
+            -0.024287162348628044,
+            0.011249667964875698,
+            0.0005283934879116714,
+            -0.01203752402216196,
+            0.010795135982334614,
+            -0.013166278600692749,
+            0.01349202636629343,
+            0.012711746618151665,
+            0.00866641104221344,
+            0.00888610165566206,
+            0.022862963378429413,
+            -0.011878437362611294,
+            -0.017711598426103592,
+            0.00656041270121932,
+            -0.008916404098272324,
+            -0.0034089903347194195,
+            -0.012598113156855106,
+            0.010795135982334614,
+            0.014878349378705025,
+            0.06490717828273773,
+            -0.013143551535904408,
+            0.0009526234352961183,
+            -0.020438792183995247,
+            -0.031938452273607254,
+            -0.0017944546416401863,
+            -0.020908474922180176,
+            0.0111739132553339,
+            0.008370965719223022,
+            0.01598437689244747,
+            0.012022373266518116,
+            -0.0021912232041358948,
+            0.00795431062579155,
+            0.019317612051963806,
+            -0.010423935018479824,
+            7.255941454786807e-05,
+            -0.006503595970571041,
+            0.0013683309080079198,
+            0.0035851215943694115,
+            -0.011719351634383202,
+            -0.0026949963066726923,
+            0.035695917904376984,
+            0.01068150345236063,
+            0.011651171371340752,
+            -0.03445352986454964,
+            0.0020435003098100424,
+            0.020150920376181602,
+            0.007602048572152853,
+            -0.008302785456180573,
+            -0.01563590206205845,
+            0.01706010289490223,
+            -0.015317730605602264,
+            -0.0022499337792396545,
+            0.002888172399252653,
+            -0.010151215828955173,
+            0.04581683129072189,
+            -0.019787294790148735,
+            -0.0011742077767848969,
+            0.031847547739744186,
+            0.0030207443051040173,
+            -0.0040983641520142555,
+            0.005408931523561478,
+            -0.003473382443189621,
+            -0.0069013116881251335,
+            -0.02139330841600895,
+            0.0031381649896502495,
+            0.00667025800794363,
+            -0.019287310540676117,
+            -0.0072005451656877995,
+            -0.007719469256699085,
+            0.006897523999214172,
+            0.023105379194021225,
+            -0.009492144919931889,
+            -0.00016145357221830636,
+            -0.008575504645705223,
+            0.009848194196820259,
+            -0.0011912527261301875,
+            -0.044301725924015045,
+            -0.006147546228021383,
+            0.021181194111704826,
+            0.000973456131760031,
+            0.0005757405888289213,
+            -0.007450537756085396,
+            0.005007428117096424,
+            -0.017423728480935097,
+            -0.008454295806586742,
+            0.02369627170264721,
+            0.01736312545835972,
+            -0.009113367646932602,
+            0.010863316245377064,
+            -0.0035680767614394426,
+            0.00433320552110672,
+            -0.020014561712741852,
+            -0.01095422264188528,
+            0.003462019143626094,
+            0.015272277407348156,
+            -0.0009819786064326763,
+            -0.029029447585344315,
+            -0.0012272365856915712,
+            -0.00024194362049456686,
+            0.02918095886707306,
+            0.01486319862306118,
+            -0.008961857296526432,
+            -0.021211495622992516,
+            -0.013242034241557121,
+            -0.001237652963027358,
+            -0.015726808458566666,
+            -0.009620928205549717,
+            0.011855711229145527,
+            0.0009137988090515137,
+            0.008992158807814121,
+            -0.018135828897356987,
+            -0.027347678318619728,
+            -0.016166189685463905,
+            0.0018465365283191204,
+            -0.012870832346379757,
+            0.003340810537338257,
+            0.015514694154262543,
+            -0.03430201858282089,
+            -0.015060162171721458,
+            -0.03805948421359062,
+            0.008280059322714806,
+            -0.0029658216517418623,
+            -0.02024182677268982,
+            -0.023605365306138992,
+            0.009484568610787392,
+            0.019181253388524055,
+            -0.0015548784285783768,
+            0.019847899675369263,
+            -0.000889651826582849,
+            0.020060013979673386,
+            -0.0073141781613230705,
+            0.0031703610438853502,
+            -0.013007191941142082,
+            0.0030245319940149784,
+            0.008333087898790836,
+            0.02890823967754841,
+            -0.023075077682733536,
+            -0.006503595970571041,
+            -0.004685468040406704,
+            0.004746072459965944,
+            0.012658718042075634,
+            0.014613205567002296,
+            0.020741812884807587,
+            -0.008272483013570309,
+            -0.0027612822595983744,
+            0.012484480626881123,
+            -0.003799130441620946,
+            0.007295239716768265,
+            0.002390081062912941,
+            0.0019677451346069574,
+            0.004162756260484457,
+            0.008477022871375084,
+            0.0053710537031292915,
+            0.027650700882077217,
+            -0.0028275682125240564,
+            0.0031703610438853502,
+            0.024620486423373222,
+            0.02322658896446228,
+            -0.018878230825066566,
+            -0.028074929490685463,
+            0.002024561632424593,
+            0.050695475190877914,
+            0.007245998363941908,
+            -0.006810405291616917,
+            -0.03927157074213028,
+            -0.0031760428100824356,
+            0.028635520488023758,
+            0.023650817573070526,
+            0.01584801822900772,
+            -0.010567869991064072,
+            -0.012098127976059914,
+            0.02772645466029644,
+            0.013295062817633152,
+            0.02616589516401291,
+            -0.025302285328507423,
+            0.010704229585826397,
+            -0.03869583085179329,
+            0.0449683703482151,
+            0.004439263138920069,
+            -0.014476845972239971,
+            0.0011884119594469666,
+            0.010386057198047638,
+            -0.004537744913250208,
+            0.006431628484278917,
+            -5.8059369621332735e-05,
+            -0.0008086882880888879,
+            0.011923890560865402,
+            0.012007221579551697,
+            0.004265025723725557,
+            -0.02399929240345955,
+            0.007359631359577179,
+            0.004272601567208767,
+            0.03005971945822239,
+            0.005431658122688532,
+            -0.0013863227795809507,
+            0.012385998852550983,
+            -0.005810434930026531,
+            0.008855799213051796,
+            -0.010719381272792816,
+            0.020469093695282936,
+            -0.02918095886707306,
+            1.0986003871948924e-05,
+            0.02051454596221447,
+            -0.0015785519499331713,
+            0.0038199631962925196,
+            -0.023741723969578743,
+            0.01186328660696745,
+            -0.052968136966228485,
+            0.021166043356060982,
+            -0.018741872161626816,
+            0.021741783246397972,
+            0.019590331241488457,
+            -0.009734561666846275,
+            -0.012052674777805805,
+            0.03545350208878517,
+            0.0026760573964565992,
+            -0.014264730736613274,
+            -0.010643625631928444,
+            -0.017999470233917236,
+            -0.02830219641327858,
+            0.041968461126089096,
+            -0.0006117243901826441,
+            0.0028275682125240564,
+            0.0024222771171480417,
+            -0.010666352696716785,
+            -0.008923979476094246,
+            0.001954487757757306,
+            0.020499395206570625,
+            -0.03369597718119621,
+            -0.011128460057079792,
+            -0.008689138107001781,
+            -0.02566591091454029,
+            -0.00020300064352340996,
+            -0.011961768381297588,
+            0.026211349293589592,
+            0.01240872498601675,
+            -0.03696860745549202,
+            0.003780191531404853,
+            0.023650817573070526,
+            0.001603172393515706,
+            0.018120678141713142,
+            0.002130619017407298,
+            0.0010795136913657188,
+            0.009742137044668198,
+            -0.010476963594555855,
+            0.026559824123978615,
+            -0.009204274043440819,
+            0.008658835664391518,
+            -0.001950700068846345,
+            -0.017287369817495346,
+            0.017969166859984398,
+            0.004083212930709124,
+            0.024120500311255455,
+            0.011954193003475666,
+            0.011605718173086643,
+            0.01112088467925787,
+            -0.019878201186656952,
+            0.020090317353606224,
+            -0.0018891488434746861,
+            0.009075489826500416,
+            0.024726543575525284,
+            -0.005863463506102562,
+            0.020832719281315804,
+            0.015257125720381737,
+            -0.00020229043730068952,
+            0.011295121163129807,
+            -0.020120618864893913,
+            -0.011779955588281155,
+            0.0085679292678833,
+            -0.009605777449905872,
+            -0.010325453244149685,
+            0.009901223704218864,
+            -0.010105762630701065,
+            -0.015052586793899536,
+            -0.0071247899904847145,
+            0.016635872423648834,
+            0.0028976418543606997,
+            -0.02610529027879238,
+            -1.7326075976598077e-05,
+            0.008901252411305904,
+            0.006166485138237476,
+            0.018741872161626816,
+            0.011317848227918148,
+            -0.006692984607070684,
+            0.010363331064581871,
+            0.011476934887468815,
+            0.0053104497492313385,
+            -0.005079395603388548,
+            -0.013431422412395477,
+            -0.0024241709616035223,
+            0.018090376630425453,
+            0.009810317307710648,
+            -0.01922670565545559,
+            0.004564259201288223,
+            0.0008877579239197075,
+            -0.01420412678271532,
+            -0.019347915425896645,
+            -0.0038483713287860155,
+            -0.004170331638306379,
+            0.0027915844693779945,
+            0.0509984977543354,
+            -0.0006216673064045608,
+            0.007893706671893597,
+            -0.025741664692759514,
+            -0.006299056578427553,
+            -0.00027556007262319326,
+            -0.0036097420379519463,
+            0.01272689737379551,
+            -0.01824188604950905,
+            -0.03724132478237152,
+            0.012711746618151665,
+            -0.0012584856012836099,
+            0.014923802576959133,
+            -0.00297907879576087,
+            0.014226853847503662,
+            -0.017347972840070724,
+            0.01916610263288021,
+            -0.0011117096291854978,
+            -0.016029831022024155,
+            0.0029374135192483664,
+            -0.011143610812723637,
+            -0.021317552775144577,
+            0.005916492082178593,
+            -0.006848283112049103,
+            -0.03896854817867279,
+            0.0029885482508689165,
+            -0.010423935018479824,
+            0.0011798894265666604,
+            -0.003912763670086861,
+            0.020166072994470596,
+            -0.0061513339169323444,
+            0.0008697660523466766,
+            0.02177208475768566,
+            0.0017026013229042292,
+            0.01601468026638031,
+            -0.014499573037028313,
+            0.010999675840139389,
+            0.012598113156855106,
+            0.02210540883243084,
+            -0.032180868089199066,
+            -0.01393140759319067,
+            0.014211702160537243,
+            0.0026116655208170414,
+            -0.005382417235523462,
+            0.020635755732655525,
+            -0.01294658798724413,
+            0.019650936126708984,
+            -0.0005577487172558904,
+            -0.011704200878739357,
+            -0.03524138405919075,
+            -0.016529815271496773,
+            -3.343888238305226e-05,
+            -0.008439145050942898,
+            0.025317436084151268,
+            0.01760554127395153,
+            -0.0108330138027668,
+            0.026468917727470398,
+            0.0037574649322777987,
+            0.014598054811358452,
+            -0.014257155358791351,
+            -0.005999823100864887,
+            -0.0003773563075810671,
+            -0.0119314668700099,
+            0.015264702029526234,
+            -0.03512017801403999,
+            0.0008342557121068239,
+            -0.00839369185268879,
+            0.005242269951850176,
+            -0.0254992488771677,
+            0.018257036805152893,
+            0.025817420333623886,
+            -0.0031419529113918543,
+            -0.005037730094045401,
+            -0.022802358493208885,
+            0.028968842700123787,
+            0.03049910068511963,
+            0.0024828813038766384,
+            -0.0013825349742546678,
+            -0.00352072948589921,
+            0.0033218718599528074,
+            0.0211205892264843,
+            -0.037998881191015244,
+            -0.009098216891288757,
+            -0.028696123510599136,
+            0.042332086712121964,
+            -0.008098246529698372,
+            -0.013431422412395477,
+            0.01862066239118576,
+            -0.0024715180043131113,
+            0.02355991117656231,
+            -0.004689255729317665,
+            -0.004855917301028967,
+            0.03148392215371132,
+            0.011514811776578426,
+            0.02772645466029644,
+            -0.04015033319592476,
+            0.014704111963510513,
+            -0.006867222022265196,
+            0.004867280833423138,
+            -0.013484450988471508,
+            0.005302873905748129,
+            0.02580226957798004,
+            -0.018060073256492615,
+            0.0068066176027059555,
+            0.030665762722492218,
+            0.00748084019869566,
+            0.002448791405186057,
+            0.0165904201567173,
+            0.003890036838129163,
+            0.010855740867555141,
+            0.000841831264551729,
+            0.0052574207074940205,
+            0.022423582151532173,
+            0.014454119838774204,
+            -0.016029831022024155,
+            -0.015120766125619411,
+            -0.00044790346873924136,
+            -0.004450626205652952,
+            0.005261208396404982,
+            0.013242034241557121,
+            0.0061172437854111195,
+            -0.015681356191635132,
+            0.01052999310195446,
+            0.014158673584461212,
+            0.013128400780260563,
+            -0.0039847311563789845,
+            -0.03412020578980446,
+            -0.02443867363035679,
+            -0.021438762545585632,
+            0.003104075090959668,
+            -0.0008281006012111902,
+            -0.0021211495622992516,
+            0.0017954016802832484,
+            0.017347972840070724,
+            -0.0008352026343345642,
+            0.00379155483096838,
+            0.01686313934624195,
+            -0.01415109820663929,
+            -0.022090258076786995,
+            -0.0035964848939329386,
+            -0.027529491111636162,
+            0.0053710537031292915,
+            0.001844642567448318,
+            -0.00965123064815998,
+            0.008530051447451115,
+            -0.0046059247106313705,
+            -0.007223271764814854,
+            -0.012196609750390053,
+            -0.02007516659796238,
+            0.005780132487416267,
+            0.016878290101885796,
+            0.005730891600251198,
+            -0.009272454306483269,
+            -0.003939277958124876,
+            0.011575416661798954,
+            0.017484333366155624,
+            0.017317671328783035,
+            -0.009295180439949036,
+            0.009628504514694214,
+            0.0004022135108243674,
+            0.004670316819101572,
+            -0.01834794320166111,
+            0.0045869858004152775,
+            -0.0027953721582889557,
+            -0.008658835664391518,
+            0.01105270441621542,
+            -0.00031201730598695576,
+            0.01628739945590496,
+            -0.013090522959828377,
+            0.005624833982437849,
+            0.010333028621971607,
+            -0.020590301603078842,
+            -0.0067308624275028706,
+            -0.01692374423146248,
+            0.026484068483114243,
+            -0.00504530593752861,
+            0.00045405857963487506,
+            -0.02978700026869774,
+            0.0033616432920098305,
+            0.000848933297675103,
+            -0.000845619011670351,
+            -0.012764775194227695,
+            0.010749682784080505,
+            0.03242328763008118,
+            0.0005179771687835455,
+            0.0216660276055336,
+            -0.0036154238041490316,
+            0.0007480839849449694,
+            -0.00785582885146141,
+            0.004977126140147448,
+            -0.004916521720588207,
+            -0.009901223704218864,
+            -0.010863316245377064,
+            0.021726632490754128,
+            -0.0033029329497367144,
+            0.0003356908564455807,
+            0.011332998983561993,
+            -0.012067826464772224,
+            0.0014620780711993575,
+            -0.0047687990590929985,
+            0.00971941091120243,
+            0.0061702728271484375,
+            0.009113367646932602,
+            -0.01176480483263731,
+            0.05451354384422302,
+            -0.0106284748762846,
+            0.005537715740501881,
+            -0.00439380994066596,
+            -0.022226616740226746,
+            -0.017317671328783035,
+            0.012711746618151665,
+            -0.017544938251376152,
+            0.03375658020377159,
+            0.024150803685188293,
+            0.0006467612111009657,
+            0.010393633507192135,
+            -0.006958128418773413,
+            0.023923536762595177,
+            -0.002602196065708995,
+            0.0010795136913657188,
+            -0.004855917301028967,
+            0.015802564099431038,
+            0.04511988162994385,
+            0.017393426969647408,
+            -0.001152428099885583,
+            0.00829521007835865,
+            0.020575150847434998,
+            0.00814369972795248,
+            -0.0020075165666639805,
+            0.014219277538359165,
+            -0.01024212222546339,
+            0.00900731049478054,
+            -0.004458202049136162,
+            0.027438584715127945,
+            0.00814369972795248,
+            -0.006310420110821724,
+            -0.0036836036015301943,
+            -0.0008209985098801553,
+            0.014173824340105057,
+            0.006950552575290203,
+            0.01963578537106514,
+            -0.00461350055411458,
+            -0.0012575386790558696,
+            0.013870803639292717,
+            0.014491996727883816,
+            -0.00552635220810771,
+            0.001724380999803543,
+            -0.01435563713312149,
+            -0.010204244405031204,
+            0.013575357384979725,
+            0.006086941808462143,
+            0.006310420110821724,
+            -0.014514723792672157,
+            -0.00980274099856615,
+            -0.004670316819101572,
+            0.0015435151290148497,
+            0.015052586793899536,
+            0.011060279794037342,
+            -0.004814252257347107,
+            0.006052851676940918,
+            0.013825350441038609,
+            -0.014014738611876965,
+            0.006147546228021383,
+            0.007208121009171009,
+            0.008855799213051796,
+            0.004787737503647804,
+            -0.010817863047122955,
+            -0.00428396463394165,
+            -0.0063937511295080185,
+            0.0012670081341639161,
+            0.016832837834954262,
+            0.013181429356336594,
+            0.0049581872299313545,
+            0.027529491111636162,
+            -0.00839369185268879,
+            -0.012302667833864689,
+            0.02339324913918972,
+            0.011257244274020195,
+            0.0031703610438853502,
+            0.014651083387434483,
+            -0.012635990977287292,
+            -0.012446602806448936,
+            0.006655106786638498,
+            0.004848341923207045,
+            0.006049063988029957,
+            0.009151245467364788,
+            0.00450365524739027,
+            0.007333117071539164,
+            -0.007098275702446699,
+            -0.007427811156958342,
+            0.0014810169814154506,
+            0.01642375811934471,
+            0.020393338054418564,
+            -0.024968961253762245,
+            -0.0018806264270097017,
+            0.0009052763343788683,
+            0.010878467001020908,
+            -0.003316190093755722,
+            0.012310243211686611,
+            5.65205882594455e-05,
+            -0.0074240234680473804,
+            0.010158791206777096,
+            -0.02119634486734867,
+            -0.012681444175541401,
+            0.0012717428617179394,
+            0.013037494383752346,
+            -0.019590331241488457,
+            0.018226735293865204,
+            -0.008893677033483982,
+            -0.013348091393709183,
+            0.013893529772758484,
+            0.020014561712741852,
+            0.004090788308531046,
+            -0.007738408166915178,
+            -0.021862991154193878,
+            -0.01127239502966404,
+            0.016060132533311844,
+            0.020272130146622658,
+            0.000947888707742095,
+            -0.0006472347304224968,
+            0.010173942893743515,
+            -0.013113250024616718,
+            -0.023605365306138992,
+            -0.005230906419456005,
+            0.010386057198047638,
+            0.01746918261051178,
+            0.018832778558135033,
+            -0.006685409229248762,
+            0.02443867363035679,
+            -0.0059127043932676315,
+            -0.015499543398618698,
+            2.86597842205083e-05,
+            -0.02802947722375393,
+            -0.017423728480935097,
+            0.00013778003631159663,
+            0.00246773031540215,
+            0.008598231710493565,
+            0.0008555619278922677,
+            0.01943882182240486,
+            0.000904329412151128,
+            0.017590390518307686,
+            -0.000659071491099894,
+            -0.002009410411119461,
+            -0.0013995799235999584,
+            -0.0015340456739068031,
+            0.02254479005932808,
+            -0.012166308239102364,
+            -0.01176480483263731,
+            0.022302372381091118,
+            0.006852070800960064,
+            0.014719262719154358,
+            -0.027635548263788223,
+            0.013757170177996159,
+            -0.015499543398618698,
+            0.017423728480935097,
+            -0.03408990427851677,
+            0.006420265417546034,
+            0.01489350013434887,
+            -0.010348179377615452,
+            -0.020499395206570625,
+            -0.029650641605257988,
+            -0.024150803685188293,
+            -0.0038710979279130697,
+            -0.010855740867555141,
+            -0.016847988590598106,
+            0.006121031939983368,
+            -0.005802859086543322,
+            0.018499454483389854,
+            -0.007079336792230606,
+            0.033877789974212646,
+            0.006969491485506296,
+            -0.004056698642671108,
+            0.003835114184767008,
+            0.022469034418463707,
+            -0.006102093029767275,
+            0.02109028771519661,
+            0.004408961161971092,
+            -0.001467759720981121,
+            0.013681414537131786,
+            -0.006552837323397398,
+            -0.002361672930419445,
+            0.005821797996759415,
+            -0.010484539903700352,
+            0.022014502435922623,
+            0.005924067925661802,
+            -0.007708106189966202,
+            -0.0006775368237867951,
+            -0.011454207822680473,
+            0.014726839028298855,
+            0.006530110724270344,
+            0.0022480397019535303,
+            -0.011295121163129807,
+            0.011249667964875698,
+            -0.004859705455601215,
+            0.0040642740204930305,
+            0.009484568610787392,
+            -0.01774190180003643,
+            -0.028862785547971725,
+            0.011416330002248287,
+            0.019120648503303528,
+            0.006382387597113848,
+            -0.0055755930952727795,
+            -0.0025681061670184135,
+            0.00839369185268879,
+            0.010651201009750366,
+            0.022150862962007523,
+            -0.012984465807676315,
+            -8.889415767043829e-05,
+            0.013204156421124935,
+            -0.007170243188738823,
+            -0.01056029461324215,
+            0.00487485621124506,
+            0.023438703268766403,
+            0.0068823727779090405,
+            -0.04327145218849182,
+            -0.018469152972102165,
+            0.01909034699201584,
+            0.007136153522878885,
+            -0.04154422879219055,
+            0.0026381798088550568,
+            -0.003689285134896636,
+            -0.023787178099155426,
+            0.016105586662888527,
+            -0.0011817833874374628,
+            -0.01828734017908573,
+            0.028756728395819664,
+            0.02830219641327858,
+            0.01834794320166111,
+            2.6987840101355687e-05,
+            -0.013332940638065338,
+            -0.0318172425031662,
+            -0.023590214550495148,
+            0.008280059322714806,
+            -0.02660527639091015,
+            -0.0072725131176412106,
+            0.011242092587053776,
+            -0.017454031854867935,
+            0.02972639724612236,
+            -0.008507325313985348,
+            0.026938600465655327,
+            0.011446632444858551,
+            0.001691238023340702,
+            -0.013446573168039322,
+            -0.018802475184202194,
+            -0.01350717805325985,
+            -0.008196728304028511,
+            0.014090493321418762,
+            0.017923714593052864,
+            0.011007251217961311,
+            0.011332998983561993,
+            0.02251448854804039,
+            -0.012158732861280441,
+            0.007969462312757969,
+            -2.888172457460314e-05,
+            -0.008961857296526432,
+            0.0026343921199440956,
+            -0.018393397331237793,
+            0.00868156272917986,
+            0.0062308767810463905,
+            0.005458172410726547,
+            -0.007208121009171009,
+            0.010742107406258583,
+            0.02712041325867176,
+            -0.018847929313778877,
+            0.011401179246604443,
+            -0.031362712383270264,
+            -0.014787442982196808,
+            -0.01056029461324215,
+            0.026862844824790955,
+            0.007757347077131271,
+            -0.015726808458566666,
+            -0.007514929864555597,
+            -0.01545409020036459,
+            0.012885984033346176,
+            0.0032442226074635983,
+            0.005764981731772423,
+            0.015817714855074883,
+            0.021893294528126717,
+            0.01746918261051178,
+            -0.00795431062579155,
+            0.006852070800960064,
+            0.004674104508012533,
+            0.028090080246329308,
+            -0.01777220331132412,
+            0.020802417770028114,
+            0.009954252280294895,
+            -0.003935490269213915,
+            0.005685438401997089,
+            -0.014257155358791351,
+            0.008530051447451115,
+            -0.007325541693717241,
+            0.030514251440763474,
+            0.0033067206386476755,
+            0.014075342565774918,
+            -0.02062060497701168,
+            0.006575563922524452,
+            -0.03172633796930313,
+            0.009022461250424385,
+            0.013704141601920128,
+            0.012976890429854393,
+            -0.007677804213017225,
+            -0.024181105196475983,
+            -0.0119314668700099,
+            0.0057990713976323605,
+            -0.01279507763683796,
+            -0.01046181283891201,
+            -0.0034828518982976675,
+            0.012446602806448936,
+            0.018847929313778877,
+            -0.01848430372774601,
+            -0.019923655316233635,
+            -0.009583051316440105,
+            0.02898399345576763,
+            -0.015787413343787193,
+            -0.001471547526307404,
+            0.02413565292954445,
+            0.021241798996925354,
+            0.015022284351289272,
+            0.002155239460989833,
+            0.020090317353606224,
+            -0.0007111532613635063,
+            -0.0237265732139349,
+            -0.02813553437590599,
+            0.007302815094590187,
+            -0.006912675220519304,
+            0.01943882182240486,
+            -0.0035548193845897913,
+            -0.005658924113959074,
+            -0.0034033088013529778,
+            -0.007503566797822714,
+            -0.0165904201567173,
+            -0.01787826046347618,
+            0.0021647089160978794,
+            -0.008454295806586742,
+            0.005863463506102562,
+            -0.0002795845502987504,
+            -0.013105674646794796,
+            0.013401119969785213,
+            -0.014075342565774918,
+            0.005772557109594345,
+            -0.002653330797329545,
+            0.001962063368409872,
+            0.0035926972050219774,
+            0.009113367646932602,
+            -0.007336904760450125,
+            -0.06381630152463913,
+            -0.013105674646794796,
+            -0.003704436356201768,
+            0.006071790587157011,
+            -0.00019578021601773798,
+            -0.016635872423648834,
+            -0.013196581043303013,
+            0.01706010289490223,
+            0.008863375522196293,
+            -0.013870803639292717,
+            0.016954045742750168,
+            0.009961827658116817,
+            0.003622999181970954,
+            -0.024044746533036232,
+            -0.0008148433989845216,
+            0.008757317438721657,
+            0.032756611704826355,
+            -0.008643684908747673,
+            -0.0003839849086944014,
+            0.0036684523802250624,
+            -0.0033332351595163345,
+            -0.010052734054625034,
+            -0.014582903124392033,
+            -0.011848135851323605,
+            0.011908739805221558,
+            0.01790856383740902,
+            -0.014264730736613274,
+            -0.02007516659796238,
+            -0.0007106798002496362,
+            0.012931437231600285,
+            -0.026590125635266304,
+            0.0007244104635901749,
+            -0.007787649054080248,
+            -0.01052999310195446,
+            0.018302490934729576,
+            -0.019014591351151466,
+            0.005249845329672098,
+            0.00048294031876139343,
+            0.008507325313985348,
+            -0.023544760420918465,
+            -0.024893205612897873,
+            0.0035131541080772877,
+            -0.023847781121730804,
+            0.03612014651298523,
+            -0.021590271964669228,
+            -0.012082977220416069,
+            0.034483831375837326,
+            -0.0037423139438033104,
+            0.01702980138361454,
+            0.0023995505180209875,
+            0.02739313244819641,
+            -0.011439057067036629,
+            -0.00617784820497036,
+            0.012499631382524967,
+            0.009575475007295609,
+            0.0032593735959380865,
+            0.03775646165013313,
+            -0.01257538702338934,
+            -0.013901105150580406,
+            -0.005556654185056686,
+            -0.012954163365066051,
+            0.0006775368237867951,
+            0.014431392773985863,
+            0.028393102809786797,
+            0.004257450345903635,
+            -0.00628769351169467,
+            -0.006143758539110422,
+            -0.0014649189542979002,
+            -0.004242299124598503,
+            -0.00947699323296547,
+            -0.00715130427852273,
+            0.017287369817495346,
+            -0.025241680443286896,
+            0.016711628064513206,
+            -0.006420265417546034,
+            0.021135739982128143,
+            -0.011757229454815388,
+            0.007007369305938482,
+            -0.008961857296526432,
+            0.003492321353405714,
+            0.0023635667748749256,
+            -0.005109698045998812,
+            0.009287605062127113,
+            -0.014772292226552963,
+            0.008454295806586742,
+            0.013742019422352314,
+            0.0011183382011950016,
+            -0.010590597055852413,
+            -0.019272159785032272,
+            0.0022972808219492435,
+            -0.000710206339135766,
+            0.01181783340871334,
+            -0.0047498601488769054,
+            -0.000981031684204936,
+            0.0002921315317507833,
+            -0.014476845972239971,
+            -0.009575475007295609,
+            0.0006050958181731403,
+            0.005302873905748129,
+            -0.008961857296526432,
+            0.010378481820225716,
+            -0.005693014245480299,
+            -0.00034160923678427935,
+            0.02627195231616497,
+            0.006526322569698095,
+            -0.007632351014763117,
+            -0.014098069630563259,
+            0.01563590206205845,
+            0.015370759181678295,
+            -0.007454325910657644,
+            0.01933276280760765,
+            -0.016196493059396744,
+            -0.0008366230758838356,
+            0.013916256837546825,
+            -0.009219424799084663,
+            0.0017698341980576515,
+            -0.008196728304028511,
+            0.03554440662264824,
+            -0.0008787619881331921,
+            -0.0035415622405707836,
+            -0.0006401326390914619,
+            0.012976890429854393,
+            -0.0026779514737427235,
+            0.001182730309665203,
+            0.01132542360574007,
+            0.005321812815964222,
+            0.0185146052390337,
+            0.00328020635060966,
+            -0.03378688171505928,
+            -0.007780073676258326,
+            -0.0065490491688251495,
+            -0.008090670220553875,
+            0.009022461250424385,
+            0.025196226313710213,
+            0.01990850456058979,
+            0.006408901885151863,
+            0.001881573349237442,
+            0.018423698842525482,
+            0.015757111832499504,
+            -0.01489350013434887,
+            0.016363153234124184,
+            0.02802947722375393,
+            -0.010030006989836693,
+            0.022741753607988358,
+            -0.005253633018583059,
+            -0.008893677033483982,
+            0.008946705609560013,
+            -0.0015378334792330861,
+            -0.017393426969647408,
+            -0.015681356191635132,
+            0.005022579338401556,
+            -0.008999735116958618,
+            -0.0022897052112966776,
+            -0.018605511635541916,
+            0.004670316819101572,
+            0.017105557024478912,
+            -0.004965762607753277,
+            -3.5243989259470254e-05,
+            0.019484274089336395,
+            -0.006965703796595335,
+            -0.016908591613173485,
+            0.006325571332126856,
+            -0.015007133595645428,
+            0.007946735247969627,
+            -0.013666263781487942,
+            0.010492115281522274,
+            -0.006098304875195026,
+            0.018817627802491188,
+            0.005446809343993664,
+            -0.008090670220553875,
+            -0.0008958069374784827,
+            -0.018984289839863777,
+            0.02065090648829937,
+            -0.033089932054281235,
+            0.00839369185268879,
+            -0.026226500049233437,
+            -0.0009781909175217152,
+            0.0008849170990288258,
+            0.014605630189180374,
+            -0.001218714052811265,
+            0.019696388393640518,
+            -0.002751812804490328,
+            -0.020090317353606224,
+            -0.007086912170052528,
+            -0.0010634156642481685,
+            -0.01618134044110775,
+            -0.030923331156373024,
+            -0.004272601567208767,
+            0.0036267871037125587,
+            -0.004711982328444719,
+            0.0021135739516466856,
+            0.017984317615628242,
+            -0.005613470915704966,
+            0.009173971600830555,
+            -0.02690829709172249,
+            -0.007416448090225458,
+            -0.00355103169567883,
+            0.0010776197304949164,
+            -0.011242092587053776,
+            0.0010596278589218855,
+            0.011226941831409931,
+            -0.019044892862439156,
+            0.013461724855005741,
+            -0.021378157660365105,
+            -0.004371083341538906,
+            -0.008507325313985348,
+            -0.00135412672534585,
+            -0.013446573168039322,
+            -0.028120383620262146,
+            0.025029566138982773,
+            -0.00021495578403118998,
+            -0.003528305096551776,
+            -0.011999646201729774,
+            0.023181134834885597,
+            0.002914686920121312,
+            0.027817361056804657,
+            -0.03157482668757439,
+            0.0013427634257823229,
+            0.011552689597010612,
+            0.0025984083767980337,
+            -0.017287369817495346,
+            0.004746072459965944,
+            -0.005234694108366966,
+            0.02210540883243084,
+            -0.0011694730492308736,
+            -0.0005340751959010959,
+            -0.008234606124460697,
+            0.030999086797237396,
+            0.009136094711720943,
+            0.0012944694608449936,
+            -0.013226882554590702,
+            -0.008060368709266186,
+            -0.002759388415142894,
+            -0.0011069749016314745,
+            0.012855681590735912,
+            0.01290113478899002,
+            -0.012105703353881836,
+            0.0013607552973553538,
+            -0.006878585088998079,
+            0.0011694730492308736,
+            -0.002935519441962242,
+            -0.0004966709529981017,
+            -0.01149208564311266,
+            0.003257479751482606,
+            0.016272246837615967,
+            -0.00780280027538538,
+            -0.006480869371443987,
+            0.0007750718505121768,
+            0.009181547909975052,
+            0.012461753562092781,
+            -0.012742049060761929,
+            -0.018226735293865204,
+            -0.011908739805221558,
+            0.016029831022024155,
+            0.010522416792809963,
+            -0.008204303681850433,
+            -0.013234458863735199,
+            -0.02583257108926773,
+            0.01169662456959486,
+            -0.03293842449784279,
+            -0.012242063879966736,
+            0.005821797996759415,
+            0.028559764847159386,
+            0.019999410957098007,
+            0.012226912193000317,
+            0.001332347048446536,
+            0.002950670663267374,
+            0.00023342114582192153,
+            -0.01916610263288021,
+            0.014408666640520096,
+            -0.011143610812723637,
+            -0.018166130408644676,
+            0.0075906855054199696,
+            0.011939042247831821,
+            0.03148392215371132,
+            0.006942977197468281,
+            -0.0010160685051232576,
+            0.01686313934624195,
+            -0.001218714052811265,
+            -0.013582932762801647,
+            0.01420412678271532,
+            -0.024256860837340355,
+            0.016438908874988556,
+            0.02860521711409092,
+            -0.004780162125825882,
+            0.0008531945641152561,
+            -0.020938776433467865,
+            0.00482561532407999,
+            -0.011529963463544846,
+            -0.002177966060116887,
+            -0.003287781961262226,
+            -0.03460504114627838,
+            0.007946735247969627,
+            -0.006825556512922049,
+            0.01046181283891201,
+            -0.00814369972795248,
+            0.006655106786638498,
+            -0.012431452050805092,
+            0.004253662656992674,
+            0.01513591781258583,
+            0.016105586662888527,
+            0.007060397882014513,
+            -0.025211378931999207,
+            -0.009196698665618896,
+            0.008105821907520294,
+            -0.01349202636629343,
+            -0.02295386977493763,
+            -0.006852070800960064,
+            0.009840618818998337,
+            -0.01061332318931818,
+            -0.00488243205472827,
+            -0.028180986642837524,
+            -0.013681414537131786,
+            -0.0014327229000627995,
+            -0.017817657440900803,
+            -0.005556654185056686,
+            -0.003100287402048707,
+            0.01571165770292282,
+            -0.0049733384512364864,
+            -0.006185423582792282,
+            0.004742284305393696,
+            -0.023878084495663643,
+            0.0037423139438033104,
+            -0.008280059322714806,
+            0.0008266801596619189,
+            -0.0032517979852855206,
+            -0.0014762822538614273,
+            -0.007969462312757969,
+            -0.014332910999655724,
+            0.006893736310303211,
+            -0.016469212248921394,
+            0.002954458352178335,
+            -0.015173795633018017,
+            -0.016499513760209084,
+            -0.00682934420183301,
+            0.00775355938822031,
+            0.021575121209025383,
+            0.006541473791003227,
+            0.021484214812517166,
+            0.005469535943120718,
+            0.028680972754955292,
+            0.0003953482082579285,
+            0.003041577059775591,
+            -8.958956641436089e-06,
+            0.003232859307900071,
+            0.019347915425896645,
+            0.006571775767952204,
+            0.018984289839863777,
+            -0.008742166683077812,
+            0.010514841414988041,
+            -0.010355755686759949,
+            0.010014856234192848,
+            0.006333146709948778,
+            -0.006473293993622065,
+            -0.003799130441620946,
+            -0.010348179377615452,
+            -0.0031419529113918543,
+            -0.016544966027140617,
+            0.0027366618160158396,
+            0.00693161366507411,
+            0.0071588801220059395,
+            -0.011582992039620876,
+            -0.005230906419456005,
+            -0.014704111963510513,
+            -0.0064354161731898785,
+            -0.01831764169037342,
+            0.014385939575731754,
+            0.029226411134004593,
+            -0.016332851722836494,
+            -0.01977214403450489,
+            -0.007772498298436403,
+            0.0007296186522580683,
+            -0.006636167876422405,
+            -0.01240872498601675,
+            -0.009151245467364788,
+            -0.035059571266174316,
+            -0.017666146159172058,
+            0.007586897816509008,
+            0.02484775334596634,
+            0.01387837901711464,
+            -0.013628385961055756,
+            -0.0006983695784583688,
+            -0.016984347254037857,
+            -0.010787560604512691,
+            -0.01535560842603445,
+            -0.02054484933614731,
+            -0.02498411200940609,
+            0.0077952248975634575,
+            -0.004033972043544054,
+            0.014931377954781055,
+            0.008113397285342216,
+            0.0032631612848490477,
+            -0.004371083341538906,
+            -0.0024109138175845146,
+            -0.013196581043303013,
+            0.0012925754999741912,
+            -0.004787737503647804,
+            -0.0017291157273575664,
+            0.03554440662264824,
+            -0.008098246529698372,
+            -0.005871038883924484,
+            0.011060279794037342,
+            0.011128460057079792,
+            -0.013764745555818081,
+            0.0006642796797677875,
+            -0.011249667964875698,
+            0.021847840398550034,
+            -0.021620575338602066,
+            0.019257009029388428,
+            0.0014507147716358304,
+            -0.010840589180588722,
+            0.028999146074056625,
+            0.01892368495464325,
+            0.00743159931153059,
+            0.006579351611435413,
+            0.01672677882015705,
+            0.0044316877610981464,
+            0.016196493059396744,
+            -0.0026552248746156693,
+            -0.004632438998669386,
+            0.009295180439949036,
+            0.015681356191635132,
+            -0.000702630786690861,
+            -0.006942977197468281,
+            0.016438908874988556,
+            0.004056698642671108,
+            0.0075414446182549,
+            -0.01356020662933588,
+            -0.008961857296526432,
+            0.001507531269453466,
+            0.006257391534745693,
+            -0.005939218681305647,
+            -0.0026003022212535143,
+            0.005530139897018671,
+            0.003943065647035837,
+            -0.013302638195455074,
+            0.01689344085752964,
+            -0.016166189685463905,
+            0.008742166683077812,
+            -0.0037972365971654654,
+            0.0121360057964921,
+            -0.006799042224884033,
+            0.0013153020991012454,
+            -0.01818128302693367,
+            -0.003869204083457589,
+            0.0038597346283495426,
+            0.012173883616924286,
+            -0.01262841559946537,
+            0.025484098121523857,
+            0.008166425861418247,
+            0.006602078210562468,
+            0.0059467945247888565,
+            -0.012855681590735912,
+            0.027211319655179977,
+            0.005969521123915911,
+            0.009886072017252445,
+            -0.008893677033483982,
+            0.011136035434901714,
+            0.035362593829631805,
+            0.03060515783727169,
+            0.014135947450995445,
+            0.004469565115869045,
+            -0.018999440595507622,
+            0.015120766125619411,
+            0.0018427487229928374,
+            0.0023806116078048944,
+            0.00863610953092575,
+            -0.011348150670528412,
+            0.01305264513939619,
+            -0.009484568610787392,
+            0.015241974964737892,
+            -0.021272100508213043,
+            -0.011749654076993465,
+            -0.014408666640520096,
+            -0.021075136959552765,
+            -0.01430260855704546,
+            0.034180812537670135,
+            -0.002346521709114313,
+            -0.010393633507192135,
+            0.0045112306252121925,
+            0.008477022871375084,
+            -0.006079366430640221,
+            0.0053142374381423,
+            -0.008204303681850433,
+            -0.013749594800174236,
+            -0.01889338344335556,
+            0.006628592498600483,
+            -0.01550711877644062,
+            0.007651289459317923,
+            -0.003304826794192195,
+            0.0037688282318413258,
+            0.007014944683760405,
+            0.018105527386069298,
+            0.00530666159465909,
+            -0.00135412672534585,
+            0.0014033677289262414,
+            0.0016865032957866788,
+            -0.00931033119559288,
+            0.0108330138027668,
+            0.015408637002110481,
+            -0.0011921997647732496,
+            -0.011279970407485962,
+            -0.009136094711720943,
+            -0.042877525091171265,
+            0.0006264019757509232,
+            -0.013128400780260563,
+            0.0064354161731898785,
+            -0.011196639388799667,
+            0.0022234192583709955,
+            0.015863168984651566,
+            -0.0054051438346505165,
+            -0.0010918239131569862,
+            0.02193874679505825,
+            0.013014768250286579,
+            0.021075136959552765,
+            0.02065090648829937,
+            0.014188976027071476,
+            -0.015302578918635845,
+            -0.03469594568014145,
+            0.002221525413915515,
+            -0.010030006989836693,
+            0.013772320933640003,
+            0.004238511435687542,
+            0.016060132533311844,
+            -0.023893235251307487,
+            -0.011878437362611294,
+            -0.006662682630121708,
+            0.014431392773985863,
+            0.01491622719913721,
+            -0.0032271775417029858,
+            0.02690829709172249,
+            0.009280029684305191,
+            0.0020510759204626083,
+            -0.02281750924885273,
+            0.003312402404844761,
+            0.005321812815964222,
+            -0.008370965719223022,
+            -0.016681326553225517,
+            -0.0004552422615233809,
+            -0.0012168202083557844,
+            -0.0007817004225216806,
+            -0.021650876849889755,
+            0.018832778558135033,
+            -0.004018820822238922,
+            0.0037574649322777987,
+            0.013840501196682453,
+            0.01516621932387352,
+            0.017984317615628242,
+            0.008370965719223022,
+            0.001497115008533001,
+            -0.006605865899473429,
+            -0.0007973249885253608,
+            -0.0005828426801599562,
+            -0.007389933802187443,
+            -0.005598319694399834,
+            0.006382387597113848,
+            -0.01134057529270649,
+            0.008810346014797688,
+            0.000896753859706223,
+            0.006113456096500158,
+            0.010605747811496258,
+            -0.004836978856474161,
+            -0.011628445237874985,
+            0.014628356322646141,
+            -0.0010548931313678622,
+            -0.0055755930952727795,
+            -0.011060279794037342,
+            0.003624893259257078,
+            0.013575357384979725,
+            -0.003096499713137746,
+            -0.013499601744115353,
+            0.00910579226911068,
+            0.002009410411119461,
+            -0.027438584715127945,
+            0.015499543398618698,
+            0.010295150801539421,
+            -0.013370818458497524,
+            0.004980913829058409,
+            -0.003371112747117877,
+            -0.03027183562517166,
+            0.004791525658220053,
+            0.002660906407982111,
+            -0.010643625631928444,
+            0.028938541188836098,
+            -0.003821857040748,
+            0.029256712645292282,
+            -0.004980913829058409,
+            -0.006905099377036095,
+            -0.011355726048350334,
+            0.009416389279067516,
+            -0.018681267276406288,
+            -0.0037953427527099848,
+            -0.008727015927433968,
+            -0.00915882084518671,
+            -0.0009838725673034787,
+            -0.005185453221201897,
+            0.006984642706811428,
+            0.005121061112731695,
+            -0.012704171240329742,
+            0.006757376715540886,
+            -0.010257272981107235,
+            -0.00020856392802670598,
+            0.0006718551740050316,
+            -0.008408842608332634,
+            0.013885954394936562,
+            0.005996035411953926,
+            -0.008340663276612759,
+            -0.0005032995832152665,
+            0.027817361056804657,
+            0.05051366239786148,
+            -0.018863080069422722,
+            0.0009270560112781823,
+            0.01452229917049408,
+            -0.0028824908658862114,
+            -0.0021590273827314377,
+            -0.011976920068264008,
+            0.005821797996759415,
+            -0.019423669204115868,
+            0.013317788951098919,
+            0.006473293993622065,
+            -0.008833073079586029,
+            0.003202557098120451,
+            -0.016711628064513206,
+            0.002621134975925088,
+            -0.0018058179412037134,
+            -0.018681267276406288,
+            -0.015529844909906387,
+            -0.012060250155627728,
+            -0.024181105196475983,
+            0.0004235197266098112,
+            -0.0008546149474568665,
+            -0.013870803639292717,
+            0.006579351611435413,
+            0.008431569673120975,
+            0.032483890652656555,
+            0.0011912527261301875,
+            0.010128489695489407,
+            -2.1513333194889128e-05,
+            -0.0010416359873488545,
+            -0.005988460034132004,
+            -0.01523439958691597,
+            0.00015387803432531655,
+            -0.002342734020203352,
+            0.022590242326259613,
+            0.005549078807234764,
+            0.017560089007019997,
+            0.0017887729918584228,
+            -0.016514664515852928,
+            -0.00596573343500495,
+            -0.007408872712403536,
+            -0.018060073256492615,
+            0.0062649669125676155,
+            -0.005473323632031679,
+            0.009605777449905872,
+            -0.015863168984651566,
+            -0.02731737680733204,
+            0.00792400911450386,
+            0.012234487570822239,
+            0.028514310717582703,
+            -0.0035623949952423573,
+            0.001716805505566299,
+            -0.005533927585929632,
+            0.016408607363700867,
+            0.032180868089199066,
+            -0.0008550884085707366,
+            0.006950552575290203,
+            -0.019999410957098007,
+            0.0017404790269210935,
+            0.013704141601920128,
+            -0.008401267230510712,
+            -0.01821158453822136,
+            0.003939277958124876,
+            -0.020635755732655525,
+            0.004583198111504316,
+            -0.020393338054418564,
+            0.030408194288611412,
+            -0.0023522034753113985,
+            0.008105821907520294,
+            0.004310478921979666,
+            0.01760554127395153,
+            -0.0031646795105189085,
+            0.010545143857598305,
+            0.013916256837546825,
+            0.014908651821315289,
+            0.0009980766335502267,
+            -0.011189064010977745,
+            -0.03790797293186188,
+            0.001859793672338128,
+            0.00817400123924017,
+            0.018635815009474754,
+            -0.01980244740843773,
+            0.011681473813951015,
+            -0.01296931505203247,
+            -0.020363036543130875,
+            -0.01003758329898119,
+            0.011575416661798954,
+            -0.006139970384538174,
+            0.03378688171505928,
+            -0.005874827038496733,
+            -0.008204303681850433,
+            0.005136212334036827,
+            0.010454237461090088,
+            0.0066778333857655525,
+            0.014704111963510513,
+            0.0326354019343853,
+            0.010196669027209282,
+            0.014173824340105057,
+            -0.004859705455601215,
+            0.02399929240345955,
+            -0.003931702114641666,
+            0.003954428713768721,
+            0.00509454682469368,
+            -0.022832660004496574,
+            -0.0040983641520142555,
+            -0.003931702114641666,
+            -0.016711628064513206,
+            -0.007458113599568605,
+            0.0032366469968110323,
+            0.008802770636975765,
+            -0.014045040123164654,
+            -0.00649980828166008,
+            -0.01350717805325985,
+            0.01218903437256813,
+            0.002730980049818754,
+            0.018302490934729576,
+            -0.029802151024341583,
+            0.009825468063354492,
+            -0.017484333366155624,
+            -0.017044952139258385,
+            0.006511171814054251,
+            -0.0038332203403115273,
+            0.006855858489871025,
+            -0.018272189423441887,
+            -0.01318900566548109,
+            0.012817803770303726,
+            0.0007897494360804558,
+            0.013060221448540688,
+            -0.009825468063354492,
+            0.0009706153068691492,
+            0.006064215209335089,
+            -0.009833043441176414,
+            -0.010961798019707203,
+            -0.007977037690579891,
+            -0.009681533090770245,
+            -0.008848223835229874,
+            0.005621046293526888,
+            0.01656011864542961,
+            -0.013742019422352314,
+            -0.003685497445985675,
+            0.016135888174176216,
+            0.006416477728635073,
+            -0.008166425861418247,
+            0.004075637552887201,
+            -0.003793448908254504,
+            -9.948806837201118e-05,
+            0.0013740125577896833,
+            0.012439027428627014,
+            -0.0017632056260481477,
+            0.024665940552949905,
+            -0.02654467150568962,
+            0.007598260883241892,
+            -0.0054051438346505165,
+            0.005446809343993664,
+            0.018226735293865204,
+            -0.02366596832871437,
+            -0.013082947582006454,
+            0.006276329979300499,
+            -0.01966608688235283,
+            -0.008060368709266186,
+            0.0054051438346505165,
+            -0.00016405766655225307,
+            -0.019923655316233635,
+            -0.012923861853778362,
+            -0.034786853939294815,
+            -0.02309022843837738,
+            0.024665940552949905,
+            0.008401267230510712,
+            -0.0073672072030603886,
+            0.018060073256492615,
+            -0.018438849598169327,
+            0.0172722190618515,
+            -0.002225313102826476,
+            -0.006613441277295351,
+            0.003935490269213915,
+            0.007674016058444977,
+            0.025362888351082802,
+            0.002653330797329545,
+            0.008264907635748386,
+            -0.0042082094587385654,
+            -0.014931377954781055,
+            -0.006321783177554607,
+            0.01980244740843773,
+            0.02471139281988144,
+            0.018605511635541916,
+            -0.018575210124254227,
+            0.010742107406258583,
+            -0.011749654076993465,
+            0.01818128302693367,
+            -0.006537686102092266,
+            0.00039037675014697015,
+            0.01859036087989807,
+            0.007969462312757969,
+            0.025241680443286896,
+            0.019923655316233635,
+            0.006852070800960064,
+            0.014128371141850948,
+            0.011908739805221558,
+            0.001204509986564517,
+            -0.012832955457270145,
+            -0.01154511421918869,
+            -0.02840825356543064,
+            0.042847223579883575,
+            0.0025434857234358788,
+            0.008317936211824417,
+            -0.027635548263788223,
+            -0.00021389048197306693,
+            -0.0034942151978611946,
+            -0.018832778558135033,
+            0.005939218681305647,
+            0.03260510042309761,
+            0.03284751623868942,
+            0.021438762545585632,
+            -2.888172457460314e-05,
+            0.0012234487803652883,
+            0.0329081192612648,
+            -0.013673839159309864,
+            -0.00937851145863533,
+            -0.01034060399979353,
+            -0.018393397331237793,
+            -0.006852070800960064,
+            -0.0032423287630081177,
+            -0.01031030248850584,
+            -0.0061172437854111195,
+            -0.009272454306483269,
+            0.0238629337400198,
+            0.0001374249259242788,
+            -0.0022594030015170574,
+            0.006605865899473429,
+            -0.020878173410892487,
+            0.011537538841366768,
+            -0.02813553437590599,
+            -0.013431422412395477,
+            -0.008651260286569595,
+            -0.006950552575290203,
+            -0.018802475184202194,
+            0.008643684908747673,
+            -0.006942977197468281,
+            0.021438762545585632,
+            0.002914686920121312,
+            -0.023590214550495148,
+            0.007530081085860729,
+            0.015317730605602264,
+            -0.0003825644962489605,
+            0.014037464745342731,
+            -0.003890036838129163,
+            0.009060339070856571,
+            0.0045112306252121925,
+            0.011681473813951015,
+            0.016954045742750168,
+            0.010651201009750366,
+            0.009030036628246307,
+            0.003437398700043559,
+            -0.001167579204775393,
+            0.01878732442855835,
+            -0.0007163614500313997,
+            -0.03405960276722908,
+            2.9236827685963362e-05,
+            -0.014628356322646141,
+            0.029499130323529243,
+            -0.009795165620744228,
+            0.009514871053397655,
+            0.008105821907520294,
+            0.007098275702446699,
+            -0.0067119235172867775,
+            0.007916433736681938,
+            -0.008742166683077812,
+            0.0052763596177101135,
+            0.015832867473363876,
+            -0.0042877523228526115,
+            -0.0014980619307607412,
+            0.007840678095817566,
+            0.00407184986397624,
+            -0.005454384721815586,
+            -0.019711541011929512,
+            -0.012287517078220844,
+            0.00726114958524704,
+            -0.02065090648829937,
+            -0.004367295652627945,
+            0.01161329448223114,
+            -0.005454384721815586,
+            -0.009620928205549717,
+            0.01393140759319067,
+            0.02136300690472126,
+            -0.002382505452260375,
+            0.022893264889717102,
+            -0.01349202636629343,
+            -0.009833043441176414,
+            -0.007984613068401814,
+            -0.004374871030449867,
+            -0.041120000183582306,
+            0.025984082370996475,
+            0.01068150345236063,
+            -0.006871009711176157,
+            0.01895398646593094,
+            0.0022499337792396545,
+            -0.002954458352178335,
+            -0.0011041341349482536,
+            0.00010499217023607343,
+            -0.03281721472740173,
+            0.03660498186945915,
+            -0.007007369305938482,
+            0.0005449649761430919,
+            0.010272424668073654,
+            0.042029064148664474,
+            -0.004590773954987526,
+            -0.005859675817191601,
+            0.01225721463561058,
+            0.005336964037269354,
+            0.031938452273607254,
+            -0.010416359640657902,
+            -0.014340486377477646,
+            -0.01815097965300083,
+            -0.004833190701901913,
+            -0.01939336769282818,
+            -0.0038313264958560467,
+            0.014037464745342731,
+            -0.019650936126708984,
+            -0.011242092587053776,
+            -0.001287840772420168,
+            -0.012537509202957153,
+            -0.0010747789638116956,
+            0.008598231710493565,
+            0.017832808196544647,
+            -0.011348150670528412,
+            -0.004450626205652952,
+            -2.2904154320713133e-05,
+            -0.005003640428185463,
+            0.0185146052390337,
+            0.0020056227222085,
+            -0.01702980138361454,
+            0.01321930717676878,
+            -0.00526499655097723,
+            0.006749800872057676,
+            0.0074429623782634735,
+            0.007939159870147705,
+            0.05145302787423134,
+            0.010317877866327763,
+            -0.001339922659099102,
+            -0.01607528328895569,
+            0.023196285590529442,
+            -0.01749948412179947,
+            -0.014181400649249554,
+            -0.02119634486734867,
+            -0.0029923359397798777,
+            -0.007060397882014513,
+            -0.028014326468110085,
+            -9.137987944995984e-05,
+            0.0016221113037317991,
+            -0.0024980325251817703,
+            -0.010045158676803112,
+            0.0005208179936744273,
+            -0.011529963463544846,
+            -0.006954340264201164,
+            -0.008545203134417534,
+            -0.016044981777668,
+            0.02240842953324318,
+            -0.021817538887262344,
+            -0.008855799213051796,
+            0.011514811776578426,
+            0.03118089959025383,
+            -0.005011215806007385,
+            -0.014173824340105057,
+            -0.019817598164081573,
+            0.0025643182452768087,
+            -0.01763584464788437,
+            0.012181458994746208,
+            -0.01225721463561058,
+            0.015045011416077614,
+            -0.03142331540584564,
+            -0.006079366430640221,
+            -0.008992158807814121,
+            0.012840530835092068,
+            0.019817598164081573,
+            -0.0025093958247452974,
+            0.00606800289824605,
+            -0.010295150801539421,
+            0.030211230739951134,
+            -0.00012286569108255208,
+            -0.018605511635541916,
+            0.00873459130525589,
+            0.007977037690579891,
+            0.012537509202957153,
+            -0.004435475450009108,
+            0.00257757562212646,
+            0.011567841283977032,
+            0.012742049060761929,
+            -0.01634800247848034,
+            -0.003174148965626955,
+            -0.017969166859984398,
+            0.005996035411953926,
+            0.006999793462455273,
+            0.021711481735110283,
+            -0.0013550736475735903,
+            0.018272189423441887,
+            -0.01563590206205845,
+            0.00019412307301536202,
+            -0.011749654076993465,
+            -0.0082270298153162,
+            -0.0028237802907824516,
+            -0.003958216868340969,
+            0.006685409229248762,
+            -0.028211290016770363,
+            -0.01274962443858385,
+            0.005287723150104284,
+            -0.004992276895791292,
+            -0.03405960276722908,
+            0.0053710537031292915,
+            -0.02725677192211151,
+            0.013128400780260563,
+            -0.002397656673565507,
+            -0.017847958952188492,
+            -0.00030846628942526877,
+            0.026332557201385498,
+            -0.003677921835333109,
+            -0.012537509202957153,
+            0.015544996596872807,
+            -0.006757376715540886,
+            -0.02119634486734867,
+            -0.005420294590294361,
+            0.014658658765256405,
+            -0.0010719380807131529,
+            -0.012393574230372906,
+            -0.029635490849614143,
+            0.012992041185498238,
+            0.0026874206960201263,
+            -0.007386146113276482,
+            0.010893618687987328,
+            0.025953780859708786,
+            0.008151275105774403,
+            -0.005924067925661802,
+            0.0015757111832499504,
+            0.0049581872299313545,
+            0.00300938100554049,
+            0.020287280902266502,
+            0.02728707529604435,
+            -0.008014915511012077,
+            -0.0011145505122840405,
+            -0.014954105019569397,
+            0.015802564099431038,
+            0.0013020449550822377,
+            0.010507266037166119,
+            0.021893294528126717,
+            0.007045246660709381,
+            -0.005738467443734407,
+            0.003905187826603651,
+            -0.023135682567954063,
+            0.01420412678271532,
+            -0.012272365391254425,
+            0.000622140767518431,
+            0.008749742060899734,
+            -0.007283876184374094,
+            0.01437078882008791,
+            -0.0015245762187987566,
+            0.0339989997446537,
+            0.025211378931999207,
+            0.0043218424543738365,
+            0.0007362472242675722,
+            -0.028559764847159386,
+            0.003547244006767869,
+            0.008651260286569595,
+            0.014499573037028313,
+            -0.0076664406806230545,
+            0.006261179223656654,
+            -0.004355932120233774,
+            0.00959820207208395,
+            -0.0065490491688251495,
+            -0.015893470495939255,
+            0.006113456096500158,
+            0.0012689019786193967,
+            0.006795254070311785,
+            0.0023408401757478714,
+            -0.008863375522196293,
+            -0.024362917989492416,
+            -0.025984082370996475,
+            0.021984200924634933,
+            -0.03918066248297691,
+            0.0073293293826282024,
+            0.011234517209231853,
+            0.03324144333600998,
+            -0.008158850483596325,
+            -0.004980913829058409,
+            0.006276329979300499,
+            -0.016711628064513206,
+            -0.004147605039179325,
+            0.008204303681850433,
+            0.012385998852550983,
+            -0.002420383272692561,
+            0.00974971242249012,
+            0.01318900566548109,
+            -0.012378423474729061,
+            0.03660498186945915,
+            0.016514664515852928,
+            0.01467380952090025,
+            -0.004007457755506039,
+            0.01312082540243864,
+            0.0066172294318675995,
+            0.019044892862439156,
+            0.025165924802422523,
+            -0.017454031854867935,
+            -0.002365460619330406,
+            -0.02292356640100479,
+            0.024165954440832138,
+            0.00726114958524704,
+            0.004556683823466301,
+            -0.008310360834002495,
+            0.014870774000883102,
+            0.0017954016802832484,
+            0.0028199926018714905,
+            -0.006696772295981646,
+            -0.010173942893743515,
+            0.019241856411099434,
+            0.0034317169338464737,
+            0.014870774000883102,
+            4.8501173296244815e-05,
+            -0.005503625608980656,
+            0.006052851676940918,
+            0.014832896180450916,
+            -0.023590214550495148,
+            -0.00203592493198812,
+            -0.019484274089336395,
+            -0.03202936053276062,
+            0.009022461250424385,
+            0.0016135887708514929,
+            -0.007052822504192591,
+            0.03230207785964012,
+            -0.010083036497235298,
+            -0.019787294790148735,
+            -0.008067944087088108,
+            -0.013113250024616718,
+            -0.010469388216733932,
+            0.0008271536789834499,
+            0.003908975515514612,
+            -0.00012428610352799296,
+            -0.009317907504737377,
+            -0.005749830510467291,
+            0.01990850456058979,
+            -0.0030358952935785055,
+            -0.00753386877477169,
+            0.00868156272917986
+          ],
+          "embedding": [
+            0.0040707582550429606,
+            -0.010230625151989194,
+            -0.021520242101903186,
+            -0.010115756270640508,
+            0.026738734586159035,
+            0.01796370233223267,
+            -0.012245199029994153,
+            0.06506924494147726,
+            -0.0014087587222033442,
+            0.009300068719271429,
+            0.003787396973047584,
+            0.002818772639754174,
+            0.022129077062176306,
+            -0.021526709808445683,
+            -0.0034388135367225824,
+            0.02763761747879405,
+            0.013210013805395276,
+            0.025255872672296602,
+            0.00029011480909986703,
+            -0.05438974762290585,
+            0.016671640187490865,
+            -0.016426252259893328,
+            -0.056591180651012554,
+            0.009335648425523075,
+            0.006141624459418827,
+            0.02568943956694641,
+            0.019377132054042116,
+            0.017946464337078746,
+            -0.057184776985927686,
+            0.03208527139096687,
+            0.030132392063755547,
+            -0.003846032371791839,
+            -0.008376524427531415,
+            -0.0010513963058713435,
+            -0.02734868388724371,
+            -0.014879400370545479,
+            0.04383920959748825,
+            0.02563613659516289,
+            -0.020117467074438997,
+            0.0398016699155716,
+            0.029896497884197775,
+            -0.0004195844530067313,
+            -0.023302720788253625,
+            -0.001168337916577517,
+            0.009149021369661628,
+            -0.02508145721900992,
+            -0.004307594138480351,
+            0.00550940415564303,
+            0.014016998918718476,
+            -0.028607379616884317,
+            -0.0018317656587526954,
+            -0.01973464940415756,
+            0.03762738392056255,
+            0.015358106058583114,
+            -0.03931907682104936,
+            0.011519637298832444,
+            0.004482080558226775,
+            0.0747867063383242,
+            0.05550679018263077,
+            0.023243129601931176,
+            0.02671631686665238,
+            -0.009686146566573536,
+            -0.04088582189725495,
+            -0.0065344623594924085,
+            0.019617295999325712,
+            -0.004722087402575777,
+            -0.002871844114437583,
+            0.003935087265300519,
+            0.03229890413393201,
+            -0.02813577047904649,
+            -0.05124172132597234,
+            0.03325235407841968,
+            -0.008687655557217192,
+            -0.011027174838246948,
+            -0.003653553063547375,
+            -0.02350737945182988,
+            0.016223087939183468,
+            0.03789252356462714,
+            0.0034350623378603083,
+            0.00195158657888739,
+            0.011789712862232108,
+            -0.0004435439660769551,
+            -0.0037330456918125894,
+            0.0031552713243866858,
+            0.04465432992298031,
+            -0.0032465421571915712,
+            -0.0075366743861922745,
+            -0.059958752731410256,
+            0.016784616861008125,
+            -0.028734506065772967,
+            -0.03035575736273943,
+            0.02680390892611741,
+            0.02338368719038086,
+            -0.0030658375437784277,
+            -0.03754896050794851,
+            0.07005659523248504,
+            0.0020711575553570695,
+            0.013034425083076923,
+            -0.02316832262132332,
+            0.021332142030732894,
+            -0.0318756233020585,
+            0.02270782509507388,
+            0.01570810309454529,
+            0.004788124787612736,
+            0.026727250200142456,
+            -0.022380142778058276,
+            -0.008593910379436904,
+            0.014439754382387034,
+            0.015007697617383174,
+            0.07662531963612977,
+            0.011195998137723795,
+            -0.010809283538470098,
+            -0.042227822120219305,
+            0.003824320933275175,
+            -0.03797629427607094,
+            -0.02347390684453348,
+            -0.0526876440280276,
+            -0.017799861680287913,
+            0.01701482796266917,
+            0.002730395144917155,
+            0.03789451759899825,
+            0.038338545350313946,
+            0.028463789278406586,
+            0.015494583705011921,
+            -0.05700237266350479,
+            0.047895170359075166,
+            -0.0064406894534929406,
+            -0.0007954721295065988,
+            0.012543029839729358,
+            -0.004703334868741216,
+            -0.0060569679284068,
+            0.012324901254711192,
+            -0.019883935809784068,
+            -0.04253565942154215,
+            0.02164409276211115,
+            -0.0075367555415808635,
+            0.03617159147722157,
+            0.07567745988686005,
+            -0.04225977020036783,
+            -0.022548837589622003,
+            0.029811602145850144,
+            -0.02175792196819533,
+            0.02069861458115363,
+            -0.00941331829664397,
+            -0.035479639885943264,
+            0.020352552580119105,
+            0.003678588588550461,
+            0.06776105274336729,
+            -0.02346047721661687,
+            0.02721075076349443,
+            -0.029386781707644253,
+            0.023530831921117993,
+            -0.02077660533100372,
+            0.020022475323372906,
+            0.024195118676513715,
+            -0.028378518720494666,
+            -0.012900969023913307,
+            0.019264089970719624,
+            -0.0374983624629265,
+            -0.019726171557384355,
+            0.00979636085642416,
+            -0.014479528435832063,
+            -0.014533384583777662,
+            0.005904623758767224,
+            0.010344475500263744,
+            0.05659646358026115,
+            0.04163866113482093,
+            0.007152494819169405,
+            -0.07584500130974763,
+            0.05471858597878099,
+            0.03580211013427615,
+            0.044161429444475306,
+            0.02258180566046248,
+            0.00980779221334715,
+            0.0011193505603397395,
+            0.018086566768892507,
+            0.015605066702191871,
+            -0.00954727399396313,
+            -0.027950079146919375,
+            0.03028514025485407,
+            -0.006351669761257431,
+            0.005095804801686436,
+            0.009620040085743162,
+            0.019748001227987134,
+            0.033709535059510916,
+            0.008203559956940196,
+            0.007389752602218563,
+            0.04197771878351742,
+            -0.02067363758471765,
+            0.007543739188471913,
+            0.011207941261799052,
+            0.013964827251172813,
+            0.03839562075248879,
+            0.0009775244543292637,
+            0.006469507611492216,
+            -0.022808235395746654,
+            -0.024795004398633292,
+            0.006249024864814684,
+            0.011292877109066925,
+            0.01596839619826819,
+            -0.0014105396434174883,
+            -0.04152075438277094,
+            0.008267788411079337,
+            -0.028834191212891387,
+            0.030940526405027906,
+            -0.0018271056434014425,
+            0.006185475047820603,
+            -0.007920574674727123,
+            0.012907560753043616,
+            -0.015165704767187252,
+            -0.021962244496450727,
+            0.0037071053072833147,
+            -0.013508376064336042,
+            -0.015178823540513864,
+            -0.02518557053061311,
+            0.011693256916760944,
+            -0.01716015674592331,
+            0.02048897513447166,
+            -0.003412078966361745,
+            -0.001807298511378529,
+            -0.0012575655363779904,
+            0.03116103961636643,
+            0.011668533361641842,
+            -0.011637010548030312,
+            -0.040260195687925236,
+            -0.011129376691522074,
+            -0.0034493180842871908,
+            -0.02098244789711689,
+            0.026658743116016938,
+            -0.01923736325953796,
+            -0.009723937181300192,
+            -0.013134679989565051,
+            -0.011974548914818263,
+            -0.004028712301052159,
+            0.019628329175867303,
+            -0.028192034966932684,
+            -0.012638520793012082,
+            -0.011568652932013111,
+            -0.0025919910519445737,
+            0.012497908463405111,
+            0.015113186712897145,
+            0.014880607940212641,
+            -0.03059227859452334,
+            -0.021899716217105084,
+            -0.0008513506358489523,
+            0.011985931118464679,
+            -0.01321584595585435,
+            -0.011735288366213981,
+            0.010710683541435907,
+            0.01961722172525993,
+            -0.023571946726192585,
+            -0.03364281880125354,
+            0.0014460076743756465,
+            0.016563167828467187,
+            -0.02891589059260944,
+            0.04595166331401113,
+            0.034271553243501465,
+            -0.027557971577291153,
+            -0.007138821520412525,
+            -0.039508407807651844,
+            0.06814329636830853,
+            0.00022136123564675344,
+            -0.039718016705753735,
+            0.0503457594963753,
+            -0.003580581891576583,
+            0.013820796398521455,
+            0.02054050419088757,
+            0.035682661890720495,
+            -0.0013220739026852943,
+            -0.0052759394697328675,
+            0.04733018719162478,
+            -0.0187845855131196,
+            -0.03982699468608443,
+            -0.023905460465070062,
+            0.002631233133039351,
+            -0.03090995554989513,
+            -0.012617612051488022,
+            -0.013605787969473513,
+            0.039672783793764745,
+            0.008669582242097126,
+            0.009974277556736362,
+            0.005535636914843583,
+            0.014180668493093584,
+            0.012042659947850441,
+            0.011870115226306776,
+            0.00451259454348581,
+            -0.014023048592193627,
+            -0.007769033996392623,
+            -0.05715987527668358,
+            0.003374564518384728,
+            -0.028801158523506985,
+            -0.00756331369170698,
+            -0.02163054894055572,
+            0.005367181573233089,
+            0.02308705505078254,
+            -0.005438909398653553,
+            -0.00873620909426313,
+            -0.0036530652237719077,
+            0.013920407174296543,
+            -0.01539613379172909,
+            -0.03908211213269472,
+            -0.016810726548844992,
+            0.007460173854386443,
+            -0.007069189426742819,
+            -0.041197222498539215,
+            -0.003843233633335066,
+            0.004373367480223065,
+            0.016194121660459908,
+            -0.029165150066068787,
+            -0.00735139702658342,
+            0.017257895082337448,
+            0.014153313280536095,
+            -0.002376178042741926,
+            -0.023223700836427403,
+            -0.0319877537205969,
+            0.02088060774052704,
+            0.0012829290529515294,
+            0.03251550040316258,
+            -0.04128453069131847,
+            -0.023376114412063578,
+            0.04083022735076889,
+            0.005690491480259003,
+            -0.05810403800689653,
+            0.016582434816685452,
+            0.010402038964417569,
+            0.016795644054313566,
+            -0.00908590043098552,
+            0.05003656634510042,
+            -0.037216197954986735,
+            -0.013027960261454162,
+            -0.007090943677204207,
+            -0.05175371787508879,
+            -0.0020065660448063322,
+            -0.014908489552053589,
+            0.010627472758306387,
+            -0.03275473161158777,
+            -0.017334608434133866,
+            0.025321047439048318,
+            -0.0033515195870701753,
+            0.011526068468647207,
+            -0.026783879816504975,
+            -0.0012830935419413407,
+            -0.017506908439998053,
+            -0.02114853058440477,
+            0.004878884893669383,
+            -0.020936900837115968,
+            -0.0025587027781733642,
+            0.01557850246191957,
+            0.023652335794299834,
+            0.011489428313303988,
+            0.0019089137782552768,
+            0.02054907941022243,
+            0.014361232983780595,
+            -0.0056413109308506085,
+            -0.0010300050480036684,
+            -0.03413262505887213,
+            -0.018821160485010483,
+            0.03361989907065425,
+            0.013125889701529858,
+            -0.003742167496379789,
+            0.008813100942466033,
+            -0.0383265293638935,
+            -0.035316957263713485,
+            0.03020499323039515,
+            -0.05001623971243151,
+            -0.00855479667673597,
+            -0.005939674940109814,
+            -0.026451831943555135,
+            -0.008032873049527808,
+            0.0009431571685654594,
+            -0.014387605128858954,
+            0.016382559541755014,
+            -0.024909510232038502,
+            0.003715736505444385,
+            -0.006016712902409688,
+            -0.009110197692342108,
+            0.05507263160354303,
+            -0.022103031032598353,
+            -0.0475415595313825,
+            0.021958434235402655,
+            -0.016670809659931243,
+            0.0323519363801999,
+            0.03867494951405658,
+            0.011124508964363016,
+            -0.008002445714431074,
+            0.034041838454818577,
+            0.018400123235267234,
+            -0.01161754789471199,
+            -0.025901844576206774,
+            0.021800298132870245,
+            0.00943488425697727,
+            -0.05906691123008531,
+            -0.00782549274387971,
+            -0.019357446956797447,
+            0.008682376344965976,
+            0.02863077694455766,
+            0.0024925977373894447,
+            -0.002682170968419412,
+            -0.013590235283325244,
+            -0.021511149507762284,
+            -0.03255453916518719,
+            0.009039307015853905,
+            0.008305286476500875,
+            0.009312625027668458,
+            -0.0014144165766983945,
+            -0.007853869334376857,
+            0.009067065760942163,
+            0.022485201611182105,
+            0.03966189080765722,
+            -0.041627289519629344,
+            0.002707086456239785,
+            -0.022678178050044222,
+            -0.03653753142622053,
+            0.0036574646778835588,
+            0.029996160786576367,
+            0.02901163765559891,
+            -0.01971777193183787,
+            -0.0036589934841109884,
+            0.009181234938554578,
+            -0.025979483002227535,
+            -0.00773645103995767,
+            0.004280770859737866,
+            0.04116668301017644,
+            0.006201085381597264,
+            0.01402728258112097,
+            0.001060407317110527,
+            -0.00920801784000087,
+            0.02109880889040528,
+            0.025211022083982727,
+            -0.029866678264942617,
+            0.014439702646820226,
+            0.009379206296908577,
+            0.04156435812161122,
+            -0.037309655469525846,
+            -0.011819731348299495,
+            -0.02766498020409985,
+            -0.010454143230794476,
+            0.027911532296566614,
+            -0.04127890742215716,
+            -0.006948536183086887,
+            0.010207784113909938,
+            -0.0016584043356136647,
+            0.016437495428919636,
+            -0.014607861750686952,
+            -0.03222311919805017,
+            -0.000610723203118213,
+            -0.02867445198803213,
+            -0.024772816792038133,
+            -0.016510309753766612,
+            -0.018581201775667577,
+            -0.07450654873867177,
+            0.00026832248829154255,
+            -0.06724696990816072,
+            -0.02447002541264633,
+            0.04885569249258513,
+            -0.010195604207134053,
+            0.03332905935704749,
+            -0.02815272112271314,
+            -0.005296340304911926,
+            -0.05908835042526019,
+            0.0038476172058970774,
+            0.025753576863447866,
+            0.02386003857017594,
+            0.005206447371391696,
+            0.024799057871126032,
+            0.012060738268265606,
+            -0.04146503002980051,
+            -0.024949907556862643,
+            -0.014503475223770958,
+            -0.0016340466065744286,
+            -0.0034009167013795906,
+            -0.031298432995411005,
+            0.036241272149880994,
+            0.008902644391348814,
+            0.03550223922667804,
+            0.04463301409716232,
+            -0.010385984413759852,
+            -0.03305004479734889,
+            -0.006952739338820577,
+            -0.0175981588092108,
+            0.042668533202572856,
+            -0.017112184620086308,
+            0.005149173014526686,
+            -0.03219589107997635,
+            0.024181808150698054,
+            0.0014339790422945406,
+            0.01254721190611376,
+            0.011099458649404119,
+            0.0027626836767364253,
+            -0.026883640446090158,
+            0.0024160770766463607,
+            0.023874514847956222,
+            -0.022558433617669373,
+            0.00917667715473093,
+            -0.006754091166465801,
+            -0.033686935718776144,
+            -0.01078389638118685,
+            -0.013595327486167114,
+            0.024712781935748277,
+            0.013168464807451798,
+            0.005875864604862434,
+            -0.02417814420558463,
+            -0.0023978800459714144,
+            0.028650463366555304,
+            0.020176980768301288,
+            -0.02118184201861616,
+            0.019082340279671355,
+            0.019734925740100277,
+            -0.04209645071102001,
+            -0.04217310001316092,
+            0.006553913719098956,
+            -0.003735719227468833,
+            0.0061793909220459,
+            -0.028285889794908054,
+            0.008181788076484304,
+            0.004432488437404309,
+            -0.01743905248466261,
+            0.041221269621866216,
+            -0.014892354419172107,
+            0.005087102208335201,
+            -0.022198383716186407,
+            -0.011936388003605221,
+            -0.00027487042024290816,
+            0.044599215801158845,
+            0.008068975171827777,
+            0.003496950921841073,
+            0.02238779276447619,
+            0.0054839808262270315,
+            0.04816723616939378,
+            -0.02685221260466647,
+            -0.0190356946337995,
+            0.016464136459560113,
+            -0.0176092780782677,
+            -0.012513274095432353,
+            -0.010695421445334002,
+            -0.004784348908457072,
+            -0.022203041334932165,
+            0.010788155380415583,
+            -0.018353897272137543,
+            -0.02277431189907027,
+            -0.003945465850796918,
+            0.016024639792926797,
+            -0.006248368166794934,
+            -0.0038769803892794875,
+            0.0016292556540434277,
+            0.02412864830338344,
+            0.02892955453819876,
+            -0.02408022470555968,
+            0.019190282372105023,
+            0.018941584034503536,
+            -0.006709516380602965,
+            -0.020956377933747532,
+            -0.01964885206488053,
+            0.01434623424964373,
+            -0.01352968586212152,
+            -0.013872556892707697,
+            0.004052798542613835,
+            0.009201147374299997,
+            -0.004086067985859617,
+            -0.04185829685685201,
+            -0.014668698123513235,
+            0.003179403563664022,
+            -0.0006415014901450319,
+            0.019360184344240947,
+            -0.0150817538567364,
+            -0.029771999278536083,
+            0.014221776097046751,
+            0.022944884879589678,
+            -0.04493425220030206,
+            0.02117577719533236,
+            -0.0030648751497261983,
+            -0.00016251533643103604,
+            -3.817296018601414e-05,
+            0.008727767382077138,
+            0.026914779443266063,
+            0.03456565094655899,
+            0.01974806772696219,
+            0.00450800007152441,
+            -0.008952599700759673,
+            0.025184190461155245,
+            0.009846044083064704,
+            -0.025168735183134738,
+            0.010983353565382029,
+            0.016740721743326522,
+            0.0070838930012654296,
+            0.009173879407277282,
+            -0.006078288608766152,
+            -0.018269347893097643,
+            0.013181494188066667,
+            -0.01435693747022231,
+            0.01678907949532146,
+            0.05154463552784974,
+            -0.00039632596424461964,
+            -0.02461439550772063,
+            0.008418147037164186,
+            -0.006544319842335593,
+            -0.010065129286193278,
+            0.02214628550460128,
+            0.0029515574751370417,
+            -0.030862152033847717,
+            0.007509556025223583,
+            0.00675485304374519,
+            0.01894936231559203,
+            -0.02355525295639599,
+            -0.01387885858544803,
+            -0.00431398020087544,
+            0.0021208056468763523,
+            0.01580766504149989,
+            -0.004739477121073512,
+            -0.013204823447960706,
+            0.02994098229517376,
+            0.025496728234662373,
+            -0.00026124946738213036,
+            0.015060064980768175,
+            0.022262345898533903,
+            0.009968408664790368,
+            0.007588431141088797,
+            -0.0083063413609779,
+            -0.031336272171937095,
+            0.0018957858383561667,
+            0.05800811417728381,
+            -0.014937085296865802,
+            0.04313875432235027,
+            0.02018476425923236,
+            0.015657644644987107,
+            0.017880263053253607,
+            -0.01662268082870562,
+            -0.008740053682875323,
+            -0.0017630394204179638,
+            -0.008294417990024647,
+            -0.027613464270562948,
+            -0.005953209556351086,
+            -0.02379961210881074,
+            -0.013878676554701543,
+            0.015418315245979704,
+            -0.014764728065382074,
+            0.0027420640870858457,
+            0.010395538988699663,
+            -0.006590633833835581,
+            -0.00841537209927409,
+            -0.003943772700797381,
+            0.03376338017637296,
+            0.011980354721458463,
+            0.027761929822019422,
+            -0.00011128539412582973,
+            0.009342430952107655,
+            -0.018419526564396063,
+            -0.006839752805318664,
+            0.03271190887864607,
+            0.006011520427944665,
+            0.004652835471939905,
+            0.01485920920301398,
+            0.014305626381398871,
+            0.038015677219487023,
+            -0.005102005401565164,
+            -0.018204460895989052,
+            0.02065495133597456,
+            -0.0049699013356374775,
+            0.019149036693562712,
+            0.008825981577388056,
+            -0.002660055848523568,
+            -0.004292379561953203,
+            0.03090270710860331,
+            -0.020243440437217503,
+            0.018735727616321387,
+            -0.02238236089688497,
+            -0.0276584413929402,
+            0.023028408906176946,
+            0.008320436733620496,
+            0.04047641118026783,
+            0.011463851066078041,
+            -0.02247143742166039,
+            -0.034588725413285185,
+            0.009827713930789184,
+            0.00337071450636392,
+            -0.024446280199879062,
+            0.017542777160283324,
+            0.00780836483490212,
+            0.008453886667283516,
+            0.016624116261219637,
+            -0.004175682824655864,
+            -0.0042784362933758835,
+            -0.001546656717896957,
+            -0.008819739495525477,
+            0.01406790007187259,
+            -0.015487658289541583,
+            -0.021457182907140035,
+            0.024687169583245905,
+            -0.03915911962859764,
+            0.01771290139419629,
+            -0.006510046907173026,
+            -0.00997450880725535,
+            -0.02059560682587115,
+            1.5939836949198004e-05,
+            -0.007159493752862143,
+            -0.0004782363814453761,
+            -0.017503181957042913,
+            0.016643971600145645,
+            0.024056442566869802,
+            -0.013209211443909658,
+            -0.00445521990927108,
+            0.0020313626821877526,
+            -0.029235692455820056,
+            0.006662209097255912,
+            0.016993027053110635,
+            0.00529940612845993,
+            0.0026104743851393716,
+            0.0001707934714983891,
+            -0.031289446611163674,
+            -0.005782208620284939,
+            0.03053516889069683,
+            0.05590704747438521,
+            0.021628254147289437,
+            0.02403314981042378,
+            0.02908606663381421,
+            0.02717206823892633,
+            -0.003198844932040613,
+            0.0016217508103442348,
+            -0.01620660606431983,
+            0.027098323143010743,
+            -0.011664168290676943,
+            0.04322175021589083,
+            0.008652323184522541,
+            -0.049493020736445056,
+            0.004537958157696794,
+            0.00943239454147692,
+            0.034918603888916544,
+            0.016678716235690318,
+            0.0005887414054803805,
+            -0.005894338664998101,
+            0.03279165663336323,
+            -0.02735726709032382,
+            -0.011962070182991011,
+            -0.00036014407560325284,
+            -0.019079003730607717,
+            -0.015938295169860884,
+            0.010357611943935459,
+            0.04278924678845588,
+            0.0009957483598272451,
+            -0.0071422790468850215,
+            0.01571201177743999,
+            -0.009445307674383,
+            0.02286338492075306,
+            0.007336440384050498,
+            0.0042809497342459904,
+            0.025581467690095977,
+            -0.0025007871391868105,
+            0.02300539823997411,
+            -0.000628898771317254,
+            0.004582863909134464,
+            -0.015960578501883316,
+            -0.00930649396114688,
+            0.027838455754803444,
+            -0.011112268986721,
+            -0.029253746504396393,
+            -0.008734785561831333,
+            0.00472579547270089,
+            -0.0003805825712986138,
+            0.0021854358149324227,
+            -0.01183885533661493,
+            -0.010840951201591629,
+            0.019965691788904057,
+            -0.004922943301424112,
+            0.003800048754664761,
+            0.012664049429916252,
+            -0.0030124219674414897,
+            -0.01910844996750822,
+            0.01889983984736845,
+            -0.003858283778519141,
+            -0.024168821338992813,
+            0.020083955711766945,
+            0.01850497889545421,
+            0.045156567155119454,
+            -0.01586824055785609,
+            0.004612757419198795,
+            0.017497774574602505,
+            -0.002172088240279357,
+            -0.019617327081566827,
+            -0.014944795772093997,
+            -0.018212462639041492,
+            -0.015975283672064702,
+            0.012837956594691596,
+            -0.01243789877725244,
+            0.019500571353639314,
+            -0.0380735226388175,
+            0.012590900534826689,
+            0.004832168601136412,
+            0.007505385730121269,
+            0.016011983211212992,
+            0.025994560026860358,
+            0.002157436199771672,
+            -0.033465476168065784,
+            -0.01014599891204166,
+            0.007845373376916211,
+            0.010093432683262838,
+            -0.002849185178237036,
+            -0.018147004183434554,
+            0.003633790611809425,
+            -0.0266147420637404,
+            -0.018644422659309035,
+            0.022926737299761104,
+            -0.04798772015599113,
+            0.017908415301694906,
+            0.024859125158586466,
+            -0.04343699525707025,
+            -0.04886699607556352,
+            -0.011459188432595229,
+            -0.036774381316312464,
+            0.0014128005215813578,
+            -0.0005256067584181517,
+            0.011194992852264102,
+            -0.009342140079829378,
+            -0.018598671981710183,
+            0.01584789414301952,
+            -0.0036412641242200418,
+            -0.004225840239935549,
+            -0.00616696736018045,
+            -0.0025598631463178384,
+            -0.010188803157718543,
+            -0.03244694662411977,
+            -0.03290891136780909,
+            -0.00033742479296272127,
+            -0.009743927051414455,
+            -0.02334288359945272,
+            -0.03447062782872289,
+            -0.008728968107967028,
+            -0.037753659749578956,
+            -0.00515561961442376,
+            0.04070890150714597,
+            0.027517147608335957,
+            0.009449414978133334,
+            0.008086900089929912,
+            -0.013989034604490105,
+            0.05691047488419298,
+            -0.020425883381389086,
+            0.039466620642512155,
+            0.005301493569940605,
+            -0.0037222353899918703,
+            0.013075425214878277,
+            0.027710826422351564,
+            -0.006253665106995653,
+            -0.0017412928073238833,
+            -0.01928096630475438,
+            -0.012920476859996416,
+            0.009444408011467955,
+            0.015799884767184885,
+            -0.0025083061619197524,
+            0.001274457251739847,
+            0.00930533161700852,
+            -0.0006652478540264343,
+            0.017258038491335247,
+            0.03996268817486439,
+            0.003833844612473671,
+            -0.00826404141706295,
+            -0.018822002570963614,
+            -0.003188844900317347,
+            -0.011807006674248829,
+            0.010482058734260632,
+            -0.015410635475145028,
+            -0.011540803927522013,
+            -0.01249428777269578,
+            -0.017081856786754605,
+            -0.002183034976477535,
+            0.003114834030705039,
+            0.0027475411722881747,
+            0.02261627635192778,
+            -0.015872027027080824,
+            -0.022487581027490516,
+            -0.0035823117885461506,
+            -0.02398778449664045,
+            8.13716819067491e-05,
+            -0.01050060782407259,
+            -0.015616181283381218,
+            0.013676480044963827,
+            -0.021513173575512936,
+            -0.01091399089076612,
+            -0.008699176177861256,
+            -0.0006986953850759474,
+            0.005853723710671249,
+            -0.00993355748621621,
+            -0.00010241843582159751,
+            0.01150872694850241,
+            0.005559552266827916,
+            -0.012336449703762907,
+            0.0039431718621504,
+            -0.012740550272925562,
+            -0.004432059174951158,
+            -0.03592309122711106,
+            -0.019665686718735042,
+            -0.01846795637714379,
+            0.005642094448288269,
+            -0.005108964028543505,
+            -0.020586678054151854,
+            0.00668364061048941,
+            0.0034775127050453774,
+            0.004713169526661792,
+            -0.010723667875209928,
+            0.013944735241748791,
+            0.010882420168586074,
+            -0.003518245916227596,
+            0.013827334494230662,
+            -0.001330399936559009,
+            -0.020286697504699764,
+            -0.011139922402071095,
+            0.03538258948179278,
+            -0.024961826464903997,
+            -0.0018019328351763667,
+            -0.0026423695690168066,
+            0.00597003516870726,
+            -0.03071835946501977,
+            -0.013634212232744325,
+            0.02164400349828422,
+            -0.005846552894831441,
+            -0.026367014426434226,
+            0.03399116480234033,
+            0.03473258231629753,
+            0.010041605592929843,
+            -0.013423384340589794,
+            0.04730054254696893,
+            -0.012968978954938308,
+            -0.018274800841400665,
+            0.026826644936918647,
+            0.0278389611094493,
+            0.02525164095926168,
+            -0.029782550447890466,
+            -0.003010713086487275,
+            -0.02393629084357885,
+            0.017993366287762866,
+            -0.0263476663180971,
+            0.01864011499179424,
+            -0.02332263152537849,
+            0.016014908881756034,
+            0.01868464825773604,
+            -0.003104041044182249,
+            0.026313491086684425,
+            -0.0052945248919367815,
+            0.015253731232325074,
+            0.0012371308759955573,
+            -0.02119765517013431,
+            0.007795466385020642,
+            -0.006469500272298062,
+            0.025762169721880456,
+            0.00833992768202466,
+            0.01801901639978976,
+            0.022019940825351116,
+            -0.0409522298213741,
+            0.02003698451081499,
+            -0.001367916864679543,
+            0.01782984895971748,
+            0.0038516134262842373,
+            -0.007885432294782304,
+            0.014221728513400065,
+            0.010991352188016265,
+            -0.012851099261924757,
+            0.003840539873231741,
+            0.0008923482291826014,
+            -0.028928104542716937,
+            0.007066004768771198,
+            -0.026914574065060926,
+            0.025866907400807087,
+            -0.005526774784152086,
+            -0.00998122210956923,
+            -0.020744532673097317,
+            -0.0071425207262770865,
+            0.003754468896273156,
+            -0.012747874997429746,
+            -0.011385805601999295,
+            -0.024700457669046855,
+            -0.02401291382882952,
+            0.029591582802488962,
+            -0.014054332096344003,
+            -0.04015978298687865,
+            0.001234440863385445,
+            -0.023301255402129767,
+            0.01193125007773239,
+            -0.005793124129267758,
+            -0.024440100649464773,
+            -0.042563399082257154,
+            -0.017548888269389457,
+            -0.0009394887961485922,
+            0.022739763644266633,
+            -0.00114156949394707,
+            0.041791395064755515,
+            0.0008816696205437911,
+            0.022207132857533028,
+            -0.015547580543606755,
+            0.005319731264840819,
+            -0.02071229859314312,
+            -0.041495579742912285,
+            0.03660432378985971,
+            -0.014637031447181937,
+            0.025854844497435178,
+            0.03324325296704186,
+            -0.013009347371118843,
+            0.001354504885302625,
+            0.02652570709474851,
+            -0.014618100923440427,
+            -0.010782232747863238,
+            -0.0034936699515378212,
+            0.003461670978569965,
+            -0.006468116285021298,
+            -0.027793221545159466,
+            -0.007752458229193391,
+            0.04193706715422074,
+            0.017894069367616637,
+            0.021928994684423043,
+            0.022606401934291136,
+            -0.017972404868085358,
+            -0.026402048654300563,
+            0.003656646576244765,
+            -0.013129722990153284,
+            -0.018090360146368677,
+            -0.01624598856366761,
+            0.012397150789705746,
+            0.011741002980073786,
+            -0.004052422390675261,
+            -0.01836750423615859,
+            -0.020929335029248472,
+            -0.016327412417122416,
+            -0.0042615345226698015,
+            -0.008458117095535,
+            -0.005002582649405199,
+            -0.012158173174049797,
+            -0.020409616620022143,
+            0.03706368540614453,
+            0.0285565642978937,
+            -0.0041732933884846566,
+            -0.00607154864704767,
+            -0.011142150176334957,
+            0.008834431806990914,
+            0.006021232046087652,
+            -0.005028870471297621,
+            -2.3396043292139426e-05,
+            0.005112033372713154,
+            0.022822312435625314,
+            0.007271235299348001,
+            0.0035791789459533856,
+            -0.06340305811365254,
+            -0.005392000517722054,
+            0.022560201096078862,
+            0.012581401789917604,
+            -0.0032248219865223496,
+            0.025180718735801837,
+            0.0006451724964970927,
+            -0.009714514747463593,
+            -0.018389422219956937,
+            0.024728862150150004,
+            0.008760329201960964,
+            0.020988700992999768,
+            -0.014974763313686516,
+            0.003845297486923296,
+            0.0006445983847138896,
+            0.00750394530065325,
+            -0.0019792992847026793,
+            -0.008578254004439607,
+            0.025776677078006516,
+            0.016593438298233645,
+            -0.005695578040595484,
+            0.01604018433477423,
+            -0.0017497260488899657,
+            -0.021460543443428924,
+            -0.010265213095609554,
+            0.00042017209251926397,
+            -0.010570864288876387,
+            -0.018015790888295944,
+            0.017569414447505878,
+            -0.002601036282535863,
+            0.0072399377790164435,
+            0.030281707151249355,
+            0.0423183067569767,
+            0.009566493398534713,
+            -0.005928603293907002,
+            -0.0037936911251953654,
+            0.015477873304924939,
+            0.01513127995244932,
+            0.010741256034652754,
+            0.03015858543221727,
+            -0.029247492875021843,
+            -0.006690306960497428,
+            0.0046653727410352576,
+            0.004610405633429778,
+            -0.011692179099382171,
+            0.030055897497741527,
+            0.04569730592994177,
+            0.006736813244286893,
+            -0.004799978513117371,
+            0.002121067028052434,
+            0.011299889107780231,
+            0.007768137218140627,
+            -0.018927080714961783,
+            -0.0012222962555201746,
+            -0.006009911026251445,
+            -0.014001766533604134,
+            0.026941585318261085,
+            -0.003960837039351648,
+            -0.011848232223811548,
+            0.02903357274500332,
+            0.1105267435019781,
+            -0.006011029338781496,
+            -0.0053646998281195206,
+            -0.03242321704621517,
+            -0.015194102811332289,
+            -0.008860590919147765,
+            0.0056988205877963076,
+            0.01635957585050017,
+            0.020327661913559888,
+            -0.0008943408274661007,
+            0.016820860249955754,
+            0.02822175614376368,
+            -0.013854827839129885,
+            0.007906244895327178,
+            -0.030891381690403466,
+            0.015702117924362928,
+            -0.005383974926093977,
+            -0.012032955400313954,
+            -0.0026169511828034054,
+            -0.008618307035330928,
+            -0.0037356680273096105,
+            0.0009666430093455632,
+            0.011296307527684261,
+            0.004427516814276546,
+            -0.014484198313369681,
+            0.002829529156309784,
+            -0.008595300985863424,
+            -0.005122996529848575,
+            -0.02393571073523279,
+            -0.024119245543788293,
+            0.005223147047205755,
+            0.0019389305548629142,
+            6.344308292376579e-05,
+            0.006848965054539454,
+            -0.003810712095343454,
+            0.030149081317333418,
+            -0.01700322008272963,
+            0.0012288912810468446,
+            0.01193226302726573,
+            0.012332749933413344,
+            0.01688602002694633,
+            0.0028600984823761366,
+            -1.9882692552530084e-06,
+            0.021172977047420227,
+            -0.010514036509402561,
+            0.01460334896316005,
+            0.013091417303145243,
+            -0.04103572381458182,
+            0.0089746401463863,
+            -0.0027841289484449873,
+            -0.024252447218794344,
+            0.004446472209246425,
+            -0.018987867343500102,
+            -0.00881974688349601,
+            0.0005365625696166774,
+            0.012054719659248512,
+            0.01960300421041832,
+            -0.008553943699846696,
+            0.02111201386480852,
+            -0.0003501657740808554,
+            0.004177088753111459,
+            -0.012879944670340026,
+            0.0065222126975051685,
+            -0.007563354540402343,
+            0.0008168128607271199,
+            -0.018213241937597355,
+            0.012526561612175514,
+            -0.0008285725179894544,
+            -0.015737003159436623,
+            -0.011337688275309921,
+            0.004737346137976519,
+            -0.0027993545140114705,
+            0.0030843163114700313,
+            0.010153740340626154,
+            -0.008815302484351509,
+            0.004919275910399053,
+            -0.0017541042538165297,
+            0.002562816687714991,
+            0.014285249186421434,
+            -0.029816964593980453,
+            0.005357227738099325,
+            0.009154842495811638,
+            -0.00048782303639333943,
+            -0.013074976152495576,
+            -0.004054505559072866,
+            0.0028322102449485015,
+            -0.016877328079971877,
+            -0.012128199639779175,
+            0.023421330164419888,
+            -0.012565145848820333,
+            0.008609070188894315,
+            -0.010936598722317274,
+            -0.018401801734378692,
+            -0.01066119413133089,
+            -0.005254229780560808,
+            0.01234969508270852,
+            -0.0038427767204741877,
+            -0.006947594635952202,
+            -0.01756894543880097,
+            -0.007890897373987378,
+            -0.047246698568222466,
+            0.009617876521097624,
+            0.027157555263862723,
+            -0.011355758309667893,
+            -0.04050311381480088,
+            -0.01017774338301804,
+            0.02437257208967103,
+            -0.004056329295095018,
+            -0.008345450168215274,
+            -0.001599171409135998,
+            0.018358386463243014,
+            0.0027207445181251406,
+            -0.007292967749349572,
+            -0.011267861428599221,
+            0.01657507035043879,
+            0.015105061280816214,
+            0.021419854579631276,
+            -0.02088952706825817,
+            0.006295984081511286,
+            0.020973020776936996,
+            -0.03073246543555915,
+            0.0023720435876790876,
+            0.03363966216709148,
+            0.010412588613998134,
+            0.013023311559810034,
+            -0.0010217662933572685,
+            -0.0075567381078977215,
+            -0.008248818794295818,
+            0.011848200323549865,
+            0.005066463524567732,
+            -0.023174489297821385,
+            0.021823825355858556,
+            0.020368868756448753,
+            0.0026042856858509524,
+            0.003457067317891926,
+            0.0005539745115410357,
+            0.01062670844108589,
+            0.009948889803890108,
+            0.05100741556350918,
+            -0.006493714124814164,
+            -0.014021751055930108,
+            0.015063988102064609,
+            0.02488059237697814,
+            -0.0023499941119596228,
+            0.01087282522970853,
+            -0.02612218935931224,
+            -0.021423880914461062,
+            0.023389294741183817,
+            0.008967763132667892,
+            0.009637806238448432,
+            -0.010815706635624337,
+            -0.011390999462441815,
+            0.031750675158698435,
+            -0.01307829462211914,
+            0.00759749911213318,
+            -0.02169895957209634,
+            0.01387879409130081,
+            -0.008606362268863441,
+            0.032229658009209815,
+            -0.0009066257436946125,
+            0.011062436424260079,
+            0.028010486866468023,
+            -0.007835805879241447,
+            -0.019657486763625547,
+            0.003967480939987674,
+            -0.015577952617624695,
+            0.02188429826317828,
+            -0.020502758155233298,
+            -0.0002362500702658591,
+            0.0040935406015014265,
+            -0.010056462958039452,
+            -0.0057038083589231615,
+            0.011475170331848666,
+            0.0015345076181482267,
+            0.006359262977587827,
+            -0.020293014401106,
+            0.01369706440475074,
+            -0.003989482215141903,
+            0.007887824197823247,
+            -0.015925992045349828,
+            0.014370223626942766,
+            -0.02309139937510304,
+            -0.021808736033141787,
+            0.008142621179624918,
+            0.0108996933426044,
+            -0.007589050511458233,
+            -0.025971539510030145,
+            0.02250992080022228,
+            -0.024980833805349545,
+            0.024979008145354243,
+            -0.006622339940663538,
+            0.002566378488088469,
+            0.004913874740363343,
+            -0.008102103508732738,
+            -0.0028098905427027783,
+            0.0057865587191235565,
+            -0.0073509382400023995,
+            -0.014123756810537343,
+            -0.019453380788786282,
+            -0.04083629135586275,
+            -0.03177927047231389,
+            0.022927581786088044,
+            0.011557232599801421,
+            0.0017073832658476193,
+            0.02339552660545296,
+            -0.02247471449131386,
+            -0.017890866287500053,
+            -0.009775068417716798,
+            0.025184899522229628,
+            -0.04408541669037361,
+            0.012104255392615549,
+            -0.014669458760022245,
+            -0.0245258242481677,
+            0.006274252098230249,
+            -0.004161960147481513,
+            0.009299171724026455,
+            -0.010779421730862708,
+            -0.04713627671791054,
+            -0.005192684784377356,
+            0.02330975631183273,
+            -0.0036802876364168935,
+            -0.006197548344287773,
+            0.03048284650641829,
+            -0.0019503797069087845,
+            -0.005038254331399784,
+            -0.005524875459245342,
+            0.027783596017224226,
+            -0.024950083604372984,
+            0.010988398531346805,
+            0.008304937912856357,
+            -0.008485065895193593,
+            0.006572665697403556,
+            0.0011518846966102158,
+            0.011927348472282199,
+            0.015632588644277183,
+            -0.015559413716826264,
+            0.02262990861928957,
+            0.011446850286896198,
+            0.006506075777765303,
+            0.008015225053785824,
+            0.00010054430757843335,
+            0.0022975742216194894,
+            -0.024948925449668525,
+            0.026047924967851216,
+            0.01888211557921096,
+            0.0001632181017876392,
+            0.012553094476616919,
+            -0.015042621225242777,
+            -0.02782009986290425,
+            0.02606646297992026,
+            -0.016847653820092335,
+            -0.00045272379465939274,
+            0.020053091486344633,
+            -0.016210036358455773,
+            -0.007930550661303182,
+            -0.01123514849773852,
+            -0.0162675150307978,
+            0.01803076559921804,
+            -0.012160217543572069,
+            -0.02216594536057653,
+            0.002594675636753233,
+            0.016443123828746582,
+            0.02134480656850119,
+            0.026694575953313482,
+            -0.011470930697351533,
+            0.017768927048191052,
+            0.0008653408391607816,
+            0.001865765599795092,
+            0.025745954171649543,
+            0.01702677011968492,
+            -0.008474889109168202,
+            0.025347176686722313,
+            -0.0016686223698600755,
+            -0.002421921060670558,
+            -0.012265504084322075,
+            -0.0165214436458303,
+            -0.007545471138114208,
+            -0.015333235556146196,
+            -0.0051830106590765685,
+            0.000762746782664525,
+            -0.0018118022696726346,
+            0.10876576838038368,
+            0.0034948044540582626,
+            -0.011608460689239234,
+            -0.005870263500417206,
+            0.007510774308250375,
+            0.0023525201199670765,
+            -0.006846154764389575,
+            0.016082721103476375,
+            -0.018616626204086575,
+            -0.023593744079394,
+            0.004743481974389998,
+            -0.0009262286319469741,
+            0.00870122878944883,
+            0.02029523251993269,
+            -0.008608837818169835,
+            -0.0200567711498164,
+            0.007089477722191928,
+            0.005535149644041221,
+            0.0023640746516071774,
+            0.005256431824732927,
+            -0.0006118671321824933,
+            0.008209920240361308,
+            0.02448422122961942,
+            -0.015622892878860465,
+            -0.029770836607612903,
+            0.010535371601298532,
+            -0.00015994156470228762,
+            -0.007193425789067776,
+            -0.011046316463797128,
+            0.014728841492009627,
+            -0.0011488451990980268,
+            -0.02751404133172324,
+            0.017663132712536528,
+            -0.019055335178900415,
+            0.014643246200933506,
+            0.0025348345163347464,
+            -0.00016387173410118788,
+            0.022565953069005714,
+            0.013101158251834834,
+            -0.04272085076882881,
+            -0.0034442020309914823,
+            0.01041205537746844,
+            0.011059422927946552,
+            0.0116446016862942,
+            0.0012992535381387887,
+            -0.018670801241345514,
+            -0.011788984490283379,
+            -0.009109849103208542,
+            -0.021268488414296117,
+            -0.012523033785938503,
+            -0.011393853408628214,
+            0.010743646386439633,
+            0.0012964030814325153,
+            0.023809566099779113,
+            -0.003030535542277875,
+            -0.006526351867141901,
+            -0.000430699447948508,
+            0.00880561288237257,
+            0.009694646009259077,
+            -0.02079011138439714,
+            0.008294338719512953,
+            -0.01823922789220214,
+            0.011819769685519291,
+            0.006416044253693268,
+            -0.022799568832306067,
+            -0.009911619276012527,
+            0.007152786695893951,
+            0.008393677195358034,
+            -0.013569953300860946,
+            -0.006503988731872938,
+            0.02299498754558268,
+            0.018870619152710703,
+            0.012233964644738905,
+            0.024994616203196145,
+            0.03838969722235787,
+            0.005744255912479976,
+            0.027682123067526515,
+            -0.016791471092091604,
+            0.008459745383680297,
+            -0.022493868174622695,
+            0.012902262638928338,
+            -0.00460399519266689,
+            -0.0050431343566427886,
+            -0.018188192498411616,
+            0.033765105245412236,
+            -0.009159751849801773,
+            -0.0027150644587327763,
+            0.034778224828120714,
+            -0.0066502021793471075,
+            0.004525230911930458,
+            0.005096748531775903,
+            -0.016067049494073893,
+            0.033274672789154185,
+            0.01686831040872425,
+            0.016353966592947262,
+            -0.021631941883929817,
+            0.011748317746828848,
+            -0.005675701689448899,
+            0.0001273442591116645,
+            -0.020738819037390263,
+            -0.0037528323797385733,
+            0.0027725541343763104,
+            0.0059860509282538546,
+            -0.016556134518370437,
+            0.02033623476757896,
+            0.00019647023567641008,
+            0.0052895744489259875,
+            0.03253462247073863,
+            0.01043311189804924,
+            0.019182938878855655,
+            0.010051051068017807,
+            0.019523739618278902,
+            0.0281939391326836,
+            0.025685787482623124,
+            0.0029932688462254147,
+            -0.018668945726139946,
+            -0.0009326901784812825,
+            -0.007880352863625262,
+            0.010492063055849727,
+            0.022849902214818293,
+            -0.022619228461640748,
+            -0.02772644874135741,
+            0.018566981184526675,
+            -0.0022608185345048414,
+            0.020031103834122595,
+            0.007544464625538984,
+            -0.020935181884939796,
+            0.003407355515103286,
+            -0.023178230166337178,
+            -0.0005418155871620593,
+            -0.031039187720144967,
+            -0.028591174077243604,
+            -0.0007659241904525392,
+            0.011945344534674635,
+            -0.01688153017778311,
+            0.03357788163306365,
+            0.017667778671101954,
+            -0.014289109343082812,
+            0.01604684528817524,
+            -0.011054587325539086,
+            -0.0036959091138078683,
+            -0.005819853870004682,
+            -0.00203755010992417,
+            -0.02157944474979592,
+            0.005138614763345542,
+            -0.0033509138150859593,
+            -0.0024871060745482883,
+            -0.016936575573598154,
+            -0.021901830366872595,
+            0.016375173843894617,
+            0.016742770709641697,
+            -0.006606252542838758,
+            0.039771905404651496,
+            -0.004238071340064403,
+            0.00018364315349545657,
+            0.011690521707403059,
+            0.0010018392078563795,
+            -0.013651921068215563,
+            -0.010829994512513501,
+            0.012362647379788833,
+            0.011321403447882543,
+            -0.018942043512252663,
+            0.005763773608775982,
+            -0.009131565038297889,
+            0.002089216671850269,
+            0.005438887787329238,
+            -0.008183835275629478,
+            0.006690280329947951,
+            -0.01673229763262922,
+            0.015833397876364723,
+            0.00295474085300386,
+            0.003186554859191664,
+            -0.011797859525926772,
+            -0.020991927615680524,
+            0.01576413532776261,
+            0.0032403514518527287,
+            0.018819824905201635,
+            -0.026755218746060662,
+            -0.00012946467986739473,
+            -0.009552681500884924,
+            -0.0028941357661144826,
+            -0.012533800273548772,
+            0.013975640628890588,
+            0.016864739319953755,
+            0.0063941926578529,
+            0.01781016436903586,
+            -0.0026268224264609724,
+            0.015368284773773678,
+            -0.018122416427266218,
+            0.008686009180697955,
+            -0.006801074032721299,
+            -0.018537521789181048,
+            0.0013237338264105415,
+            0.027863741654435434,
+            -0.004670012662973029,
+            0.011066853115455966,
+            0.011704884649534624,
+            0.018954056680143452,
+            -0.0012626215734593707,
+            -0.0030850559460101866,
+            0.015610404665171367,
+            -0.005838226405658572,
+            -0.006861883833171456,
+            0.020872966233916948,
+            0.032951653763679145,
+            -0.006716215648910268,
+            -0.0021292599951407863,
+            0.010886596327481677,
+            -0.004901684487088035,
+            -0.008644584974417106,
+            -0.009587037829603014,
+            -0.03158137900717491,
+            0.024369089446706894,
+            0.012092412439477474,
+            -0.009601042951586109,
+            0.022690862146548786,
+            0.020752792552278052,
+            0.011412743968296708,
+            -0.010471860708195726,
+            0.02291536038589634,
+            -0.01268035142112569,
+            0.024168869865226136,
+            0.032482831775271157,
+            0.004400973556358811,
+            0.0022312906286559167,
+            0.0021175489661843985,
+            0.01691348460546639,
+            0.006050357346231155,
+            -0.007068133214675996,
+            0.011656003476409986,
+            -0.018487923180009552,
+            0.01598530891381001,
+            -0.004432366596808033,
+            0.02808371007118948,
+            -0.01337145216499331,
+            -0.010473700033368562,
+            -0.00537325376270318,
+            0.00010726431372328729,
+            0.006445701087575359,
+            -0.003324002570256215,
+            0.005255443653813165,
+            0.0007072577303252132,
+            -0.017825649146944237,
+            0.003050128130393781,
+            0.004825967170196209,
+            0.009018799597337101,
+            0.004103653497834134,
+            0.007249957401824204,
+            -0.012360355929483,
+            0.01441885664789707,
+            0.004401820363035808,
+            -0.0026190938717927086,
+            -0.0007163225760398784,
+            0.011124538295057815,
+            -0.013232822471919533,
+            -0.004895624967029031,
+            0.005281228035321008,
+            0.0059275753353475414,
+            -0.007507924998741734,
+            -0.004484140989405051,
+            0.011908076549392969,
+            -0.0037917291321648483,
+            0.006351543504726163,
+            -0.004409464890606014,
+            -0.0037760571637986827,
+            0.005303661313434895,
+            -0.00974511544836468,
+            -0.004204167898342997,
+            -0.0007093125445559502,
+            -0.006869250680233214,
+            0.00786202375969793,
+            0.005971033227117194,
+            0.010606085272785863,
+            0.006969437776542303,
+            0.004293129058563288,
+            0.004114452739123398,
+            0.004071017374828283,
+            -0.002445681623750424,
+            -0.00592480769363354,
+            -0.004617501328052899,
+            -0.012067458701076778,
+            -0.008335239153829813,
+            0.009025028998949398,
+            0.008653779421856954,
+            -0.019363018792788574,
+            0.02048011578221327,
+            -0.001188320960771659,
+            0.00772165044232104,
+            -0.01602086976290928,
+            -0.000907553574811279,
+            0.005464880629734941,
+            0.008267702703609528,
+            0.016136762227127184,
+            -0.03654209526492064,
+            0.005179791231187316,
+            -0.0037321883493524766,
+            0.016026013709955444,
+            -0.008215703411373182,
+            -0.006979816286354862,
+            0.02564060526656963,
+            -0.01450763388985563,
+            -0.0009847758228920171,
+            -0.016142304341196283,
+            -0.01407778890808498,
+            0.010815725252959506,
+            0.02050940280233426,
+            0.020684808835848383,
+            -0.0004965802897137407,
+            -0.0005781445948759235,
+            0.00036179841583004193,
+            0.022775804864342608,
+            0.015545681427566235,
+            0.018697154644607316,
+            -0.004543500782944068,
+            0.007028436664826706,
+            -0.023057215501463973,
+            0.009912581556496658,
+            0.0011795417129232596,
+            -0.008215027850908477,
+            -0.008554137076267468,
+            -0.008703378564520125,
+            -0.010448538943182258,
+            -0.011773811327190638,
+            0.004106616839960258,
+            0.013895013678116032,
+            0.004190549543250594,
+            0.0031571190600981074,
+            -0.0225385788570314,
+            -0.006243193914542516,
+            -0.011229641377928767,
+            -0.01674214043453308,
+            -0.007367389908398082,
+            -0.02386334969150212,
+            -0.028021339684039734,
+            0.0002794207370355175,
+            -0.02411861258809571,
+            -0.015958336406924612,
+            0.005268815864773021,
+            -0.01546610225139129,
+            0.00898600137695074,
+            -0.008396481690803911,
+            -0.011873486990732417,
+            0.0074743033679134105,
+            0.015336444788692,
+            -0.017676977079590463,
+            -0.010165317702640217,
+            -0.01690524990506033,
+            -0.013460451917691695,
+            0.010918550283871637,
+            0.00595960977280955,
+            0.009250157926438213,
+            -0.012447979337850305,
+            0.0062932435120863365,
+            -0.012673905639312696,
+            0.015073013552721701,
+            -0.03769166067890086,
+            -0.029992249463405965,
+            -0.00014990754540999537,
+            0.007288036630677436,
+            -0.0020735742042505426,
+            -0.018695120921334806,
+            0.0003860116505754868,
+            -0.003608995261360232,
+            -0.014900191335144973,
+            -0.013866216738266176,
+            0.025251475305524053,
+            -0.006788483107616133,
+            0.01223227424986397,
+            0.0034937277040015626,
+            0.03438333560343413,
+            0.008803852645562129,
+            0.014666206816302367,
+            -0.005709421955160623,
+            0.003742433643393657,
+            -0.0033706258426092744,
+            0.006906464031352315,
+            0.0076578888497945185,
+            0.009276592191710524,
+            0.008025321004305927,
+            0.004576470294889512,
+            0.002887346543823031,
+            -0.001925468214331082,
+            0.02125241545487307,
+            0.025671028960818562,
+            -0.00037467906648160044,
+            0.008981063192139038,
+            0.0009387900084090577,
+            0.0002735789978680719,
+            -0.008374290340790827,
+            0.006547013284767913,
+            -0.008274637814401893,
+            0.004041066530254948,
+            0.012897549078838394,
+            -0.009150863940242148,
+            -0.010211568689323908,
+            0.016575088843292566,
+            -0.012380099999574783,
+            -0.03420455578618604,
+            0.026550654680719987,
+            0.005324512466403649,
+            -0.0035670106106754245,
+            -0.010657928468872608,
+            0.0037176027853158167,
+            -0.01103891334887315,
+            0.008096915671982816,
+            0.022587659771132487,
+            -0.0013527917944322358,
+            -0.0018085171207746743,
+            0.0038503794877980586,
+            -0.011870484733648887,
+            -0.007881489342351292,
+            0.006462772034381968,
+            0.0315696514747912,
+            0.01326417653779577,
+            -0.020400369539140532,
+            -0.015223808971473513,
+            0.020406862349006738,
+            -0.023142176712111575,
+            -0.006254959806032159,
+            0.009039506093607052,
+            0.0009087520492922078,
+            -0.022766046838415165,
+            0.0009662037093312435,
+            0.004701966191595815,
+            0.005946458676746661,
+            -0.006358986495019584,
+            0.03237532341275415,
+            0.001075992257265679,
+            0.015892020271402903,
+            -0.017189453434945788,
+            0.0017514632185789299,
+            -0.014189278671121677,
+            -0.014265447388465805,
+            -0.029809460132405256,
+            0.0005557491671071693,
+            -0.0030324198417091704,
+            0.011038984948790744,
+            0.024808067315720335,
+            0.01145103533194245,
+            0.014432246996007246,
+            0.010552852947578788,
+            0.01513689173494211,
+            -0.013653722180565311,
+            -0.029009757850432732,
+            -0.005171522291984226,
+            -0.018001048743500243,
+            -0.02297482628212251,
+            0.003962613844063587,
+            0.013645125285548414,
+            0.006278662690346082,
+            0.023786381386575035,
+            0.0021738516691483163,
+            0.0009580591750940011,
+            -2.690805459518147e-05,
+            0.008894091070322921,
+            -0.004952203441302471,
+            -0.006553986197076102,
+            0.003570189040872711,
+            -0.000652714137223501,
+            0.021228360881461853,
+            -0.005758946428212041,
+            7.897153237591187e-05,
+            0.014242175492137011,
+            -0.005622256348934008,
+            -0.0035009192529791146,
+            -0.017339926143652377,
+            0.006248842389703505,
+            -0.01622011586976866,
+            0.025783086900871728,
+            0.018450783573486916,
+            -0.009109295293546267,
+            -0.013469521573468915,
+            -0.011259555201313116,
+            0.02773097452736661,
+            0.004631282554715908,
+            0.010787055220617197,
+            0.018228322680582464,
+            0.01461254259249465,
+            0.01784830193964943,
+            0.004664725683522323,
+            -0.0034339781146649384,
+            -0.0007692550982545399,
+            0.021176646860433974,
+            -0.014700178590249292,
+            -0.002701208084390198,
+            -0.008842788249773623,
+            0.0021799005046816444,
+            -0.008431299000066363,
+            0.006086447848362669,
+            -0.010793016137040172,
+            -0.015251381840165954,
+            0.0189503457652375,
+            -0.02660767353735015,
+            0.016423753600066336,
+            -0.00593016170680532,
+            -0.00482891500342298,
+            -0.027459214027897378,
+            0.010728039528784636,
+            0.0027866316191608297,
+            0.0022519974195000687,
+            0.010770472550461752,
+            -0.021427634123401408,
+            -0.01867717541189363,
+            0.008532444554010673,
+            -0.011457117350255678,
+            -0.007152301670244725,
+            -0.00348619742388509,
+            0.0154715607296319,
+            -0.00017898955187849443,
+            -0.009925365923862246,
+            0.02085093731060963,
+            -0.007175309371250024,
+            0.01223979370120031,
+            0.005256127464980655,
+            -0.008756077893734508,
+            -0.0028438320920811362,
+            0.021022708898898203,
+            0.027557382862284808,
+            -0.008342206702778956,
+            0.014526066580760486,
+            -0.011883797998224506,
+            -0.01088335352319397,
+            -0.03479444009382274,
+            0.030104838505937447,
+            -0.013305732257226255,
+            0.026773747582411633,
+            -0.0006968265979630097,
+            -0.017037729111584094,
+            -0.016206207453228583,
+            0.01625225876515654,
+            -0.0024290584023212254,
+            -0.009366287391027166,
+            0.01465885301870365,
+            -0.02428628495473643,
+            0.0058336571592899735,
+            -0.0006627935593670669,
+            -0.011531762838191602,
+            0.013275508677482622,
+            0.0019223252930710324,
+            0.010738622785461513,
+            -0.02030405805119001,
+            0.0009661595529028194,
+            0.009263716373456188,
+            -0.016791855691574455,
+            0.006057830565306032,
+            -0.03730952786847912,
+            0.0029341721588188943,
+            -0.014833895684348656,
+            0.005725615593914644,
+            0.001498755253963699,
+            -0.01820708352937343,
+            -0.03652692232407375,
+            0.00043665701245599497,
+            0.01913883585802903,
+            -0.007975988799790905,
+            0.020538595326175857,
+            0.008243569241768276,
+            0.008421404577931517,
+            0.002314738067443065,
+            0.0016702403734608475,
+            0.008279538988821465,
+            0.03679456861213399,
+            -0.01775927634148114,
+            -0.004836606118958108,
+            -0.0012151329859941935,
+            0.010644195708319258,
+            -0.02028470571116305,
+            -0.009287515726906219,
+            -0.0036506912311261536,
+            -0.013134467570201307,
+            0.030615862647585222,
+            -0.020265555441523482,
+            0.025634449009756734,
+            -0.009300770899023007,
+            0.009274072905190077,
+            -0.02097685009125915,
+            -0.009797008690754956,
+            -0.010661540100131183,
+            0.0019532291898233895,
+            0.013465005282849381,
+            -0.009151430285942593,
+            0.0071467934925125145,
+            -0.006655608635025676,
+            0.01020327561163374,
+            -0.001472597695707201,
+            -0.02443508800166349,
+            -0.0024998811551575563,
+            -0.01107321258077183,
+            0.02456164421154801,
+            -0.0019619696941303388,
+            0.0005043179699158694,
+            0.010951580163871948,
+            -0.00853593956432799,
+            0.023927336608555987,
+            -0.008855702280766362,
+            0.0010093679746665581,
+            -0.004863269000020324,
+            -0.004005691415117686,
+            0.001698212044518236,
+            0.01822024022650613,
+            -0.005707404715849356,
+            0.03722587302228139,
+            -0.01708925706528118,
+            -0.0065500772743641925,
+            -0.00015538686795361824,
+            0.010852925330001283,
+            0.010035359503147549,
+            0.008754719437760327,
+            0.004067820167375348,
+            -0.014861417626433965,
+            0.009346669946626484,
+            -0.020136029447262147,
+            0.007501669351583119,
+            0.023178490169347605,
+            -0.006840359541885465,
+            0.0006676450965357345,
+            -0.01283320739092204,
+            -0.015610538827742276,
+            0.006308543164022682,
+            0.002329420005586303,
+            -0.002295739637504597,
+            -0.02181189272979859,
+            0.006298804075737111,
+            -0.014398470126723037,
+            0.004007419942728446,
+            -0.0007497656993578918,
+            0.013863867106289847,
+            0.016451166273826513,
+            -0.006740209707090658,
+            0.039292260347522444,
+            0.01871723939316731,
+            0.009109005310505462,
+            0.005674782595856756,
+            -0.040588444759736254,
+            -0.008993850998154852,
+            -0.0010197185016393919,
+            0.004866150984083949,
+            -0.01141354216871663,
+            -0.018944881712855702,
+            -0.002646110151132781,
+            0.000308585036997592,
+            -0.010681954707313127,
+            -0.009091585604191968,
+            0.01751599002710486,
+            -0.006034424764302979,
+            0.00872057856926618,
+            0.004146961781109887,
+            0.023758065635807138,
+            -0.009727802672285544,
+            0.01176257119851645,
+            -0.0071822083765114165,
+            -0.005456197751568526,
+            0.0006726765703478756,
+            0.011011628907288813,
+            -0.008031792262886802,
+            0.009763340867550022,
+            -0.004279891138185881,
+            0.008355720233269893,
+            -0.008552596384067825,
+            -0.023952129458400563,
+            -0.014972827087541732,
+            0.014347535404251018,
+            0.011651260496709961,
+            0.008137738462704715,
+            -0.013624435569539285,
+            0.0031728721674035352,
+            0.00675830277744976,
+            -0.009184784787581942,
+            0.009161771311910335,
+            0.017733082848622708,
+            -0.001411542568472906,
+            -0.0017830167499636113,
+            -0.0016130003349297027,
+            -0.012173528776899308,
+            0.0011196437753239162,
+            0.02284887358189995,
+            -0.003054479643937998,
+            0.014614283913976716,
+            0.010689486748036806,
+            0.009643292262102645,
+            -0.00810930837603355,
+            0.001915955511174678,
+            0.02341863048264366,
+            0.015022291783105098,
+            -0.004625715884342598,
+            -0.011422973894623295,
+            0.004259757717597968,
+            -0.010332744704689279,
+            0.007373328473086509,
+            -0.02117940113144938,
+            -0.004672698999328004,
+            0.008896311380763048,
+            -0.007028862917603513,
+            -0.01732432285099198,
+            -0.028026173555055017,
+            -0.016765805892242715,
+            0.019819329380181,
+            0.0027213760678977965,
+            -0.013291251876048186,
+            -0.0032919084721662363,
+            0.0031214883337300926,
+            0.002553005168048115,
+            -0.0071104806975682885,
+            0.005310583052470149,
+            0.007359183453835734,
+            -0.008674133980486269,
+            0.017665958608281126,
+            -0.000521269431423141,
+            -0.0008020251454616109,
+            -0.0005442374261144659,
+            0.0006127687598711624,
+            -0.016387752806126442,
+            0.01666161912459431,
+            -0.010949228376419892,
+            0.015198110119816624,
+            -0.008966726991404666,
+            0.008600499511690635,
+            -0.007617043351740469,
+            0.009458641139070436,
+            0.011597957400445055,
+            0.012755037944648426,
+            -0.004403680820063604,
+            0.008519241473418196,
+            0.004807816191718704,
+            -0.008147125240985166,
+            0.007017601015398136,
+            0.01296082929962557,
+            0.00582353932160707,
+            0.0013634237970372698,
+            -0.0008246894322793264,
+            0.0022035670958705576,
+            -0.002480976219285683,
+            0.0056293125704658135,
+            0.012714850523393719,
+            -0.002497639449055006,
+            -0.007135683201188999,
+            0.014988249771676699,
+            0.014955895630183352,
+            -0.022994948923833993,
+            0.007121634434082677,
+            0.005835971825639792,
+            -0.0030507120821557008,
+            0.002145266953387315,
+            -0.027334525318320587,
+            -0.020359474070022213,
+            0.01901995970350808,
+            -0.015260454474605018,
+            0.02850460517419317,
+            -0.011624138792780823,
+            -0.02284242513728605,
+            -0.01277489645734181,
+            -0.004209529633534193,
+            -0.022684500849916897,
+            -0.004894414098943742,
+            0.01919165657366256,
+            0.013401560972810082,
+            -0.004313549020265677,
+            -0.019089369171102138,
+            0.03863248408555474,
+            0.010497951840980463,
+            0.03508157919687289,
+            -0.008597970944278045,
+            0.0023354831464448313,
+            -0.01563401808449416,
+            0.015551176593436512,
+            0.001075882802133782,
+            -0.011575692834796734,
+            0.01505716108583101,
+            0.031836226915668404,
+            0.0034740889237207494,
+            -0.00617206855371193,
+            0.0018420207148235309,
+            0.00418433325800744,
+            -0.0008167766650900381,
+            0.011160989178055499,
+            -0.018497472318034093,
+            -0.011951262348399922,
+            0.0013093376768780122,
+            0.00216347903176653,
+            0.0059233460419514045,
+            -0.001778091863744448,
+            -0.013317639704511549,
+            0.011433308892643204,
+            0.011596810919469926,
+            -0.01518143105525228,
+            -0.007497373860338335,
+            0.0050957665991362405,
+            -0.018974405317819656,
+            0.01872472902494418,
+            0.011859522398789263,
+            0.000792413325332956,
+            -0.0006503947129906383,
+            0.00808635755624928,
+            0.003749813548995223,
+            0.0033657799335872368,
+            -0.007862215303617144,
+            -0.0028014669075116876,
+            0.0009892310856576799,
+            3.834379772674841e-06,
+            0.015402211364447726,
+            -0.015264893397161102,
+            0.02488855366696587,
+            -0.0025643639525691083,
+            0.00953993868921819,
+            -0.017460590918773188,
+            -0.0017911712931080155,
+            -0.00402374787529868,
+            0.01838051670627292,
+            0.006753072393532114,
+            -0.010130676216186492,
+            0.004921510943896829,
+            0.007975201081296845,
+            -0.012589786869771252,
+            0.00018319054705558588,
+            -0.0022606180909706688,
+            -0.014791528072476771,
+            -0.0031171076370692706,
+            0.017873825561816544,
+            0.007593688317551089,
+            0.0037375070591255375,
+            -0.015863143964928172,
+            0.007192324815779268,
+            0.007522990667369163,
+            0.024504875417877527,
+            -0.026309816223960274,
+            0.024779042752840836,
+            -0.022104087334088543,
+            0.02165781034613893,
+            0.0319051741666687,
+            0.007538146369110045,
+            -0.01690181186708103,
+            -0.006595947249169521,
+            0.006373991305923002,
+            -0.023410814842488123,
+            -0.005494017994983188,
+            0.016903736810723716,
+            -0.01792400152372258,
+            0.000330865061689199,
+            -0.014660413003063089,
+            0.0040825880154611895,
+            0.008248037865838212,
+            0.011540333451799686,
+            -0.0033578830043738085,
+            0.018327525495676525,
+            0.020171745352159914,
+            -0.01138759854371235,
+            -0.0004221149053107702,
+            -0.022321409200941777,
+            0.005046865873981027,
+            0.022509643273449936,
+            0.001111272240506322,
+            -0.009540467231190513,
+            0.018376261423745193,
+            0.008480532791922674,
+            -0.020801891594934313,
+            0.008436374227753589,
+            -0.036175982655998785,
+            -0.010454118523313122,
+            -0.009328507118928412,
+            -0.001218633166683384,
+            -0.004142773602181115,
+            -0.0002494371580772723,
+            -0.011365548842513265,
+            -7.699548642315061e-05,
+            -0.006889339420877534,
+            -0.0037173070609413474,
+            0.014953978004522174,
+            0.011678957032329096,
+            -0.003501047405385249,
+            -0.0073984932581048915,
+            0.008493128567102267,
+            0.006163937693564917,
+            -0.017034696006318337,
+            -0.013268418502042576,
+            0.0037122128689059085,
+            -0.020067656401141467,
+            0.005148329575092478,
+            0.00459729178556505,
+            0.014031669598917171,
+            -0.01849235806598133,
+            -0.009298127665365717,
+            0.019716512208448042,
+            0.0094466405158553,
+            -0.0008988916054855498,
+            0.004264260983102956,
+            -0.00032312548009733906,
+            0.002279987803645912,
+            -0.0008902345374346296,
+            -6.145528806973222e-06,
+            0.003602229465148649,
+            0.012503366799381277,
+            0.007759393855482671,
+            0.03694402922447051,
+            -0.016692148708641812,
+            0.005873765863350649,
+            -0.014187609702966002,
+            -0.0037592517335595654,
+            -0.00749939010396784,
+            -0.0004308157309481927,
+            -0.0022282234382577843,
+            0.0021999996319642854,
+            0.0052176773733353925,
+            0.009353171029508977,
+            0.002155405440765206,
+            -0.00823565584692709,
+            0.009691447250255694,
+            -0.015228173095533029,
+            0.003853858061067531,
+            -0.008207910291425473,
+            0.01017972550354623,
+            -0.009695280827515048,
+            0.0028705587799331246,
+            0.03849358164875786,
+            0.01130021300937487,
+            -0.004934175853966098,
+            -0.014300147534361459,
+            -0.0007796051618993811,
+            -0.01737877501558196,
+            -0.008579119859817895,
+            -0.005434792488602325,
+            -0.02617338653902575,
+            -0.00838916510137031,
+            0.015344260464413648,
+            0.011202567142310492,
+            0.014124853863170132,
+            -0.023894182185855824,
+            0.008897943489322673,
+            -0.03001670823127986,
+            -0.011951507895140876,
+            -0.001530967140156163,
+            -0.015467812535428609,
+            0.0010110999991167703,
+            0.023623196504289838,
+            -0.007762735863648054,
+            0.012654652822750325,
+            0.012867958170258448,
+            0.018777522196571934,
+            0.006390986057157016,
+            0.00038045736974183937,
+            -0.009444911783760569,
+            0.009534310165417939,
+            0.008806031651319704,
+            0.0080954543713255,
+            0.013161833638467553,
+            -0.0022840529693446553,
+            -0.018060162439023572,
+            0.005469288590000856,
+            0.014354475778235131,
+            -0.0033504276914600585,
+            -0.001615191819057537,
+            -0.011088493180661976,
+            0.016039040565630432,
+            -0.023559046333872156,
+            -0.009482398794557168,
+            0.009918576960319548,
+            -0.0027308309155029842,
+            0.009212063547988036,
+            0.0160181200081126,
+            -0.01881373001297218,
+            -0.0024168618343039703,
+            0.016244889977796682,
+            0.0024104546308670025,
+            -0.00022606425578303413,
+            0.016792244238218245,
+            -0.0007616857360728623,
+            -0.004279120886828465,
+            0.00997764266310123,
+            -0.0075631917138318655,
+            7.3878137053128405e-06,
+            -0.002416364915008295,
+            -0.0028253498168087964,
+            0.02054128727706496,
+            -0.021281801678871736,
+            -0.005379259148621041,
+            0.02257306103533458,
+            0.015687105622685975,
+            -0.007843745111084413,
+            6.258498042855072e-05,
+            -0.0219506172079994,
+            0.0019925886885244454,
+            -0.007768133999661569,
+            0.016524850592507592,
+            -0.007957949830060354,
+            0.0030085044673901665,
+            -0.01361024620290394,
+            0.011563781350927133,
+            -0.01277407169098418,
+            0.00873716215529134,
+            -0.002986943948333218,
+            -0.017567646684567492,
+            -0.0053109390854215125,
+            0.004561236136619266,
+            0.0014045175130801308,
+            0.044958041347040185,
+            -0.005539136232703934,
+            0.003729517522976541,
+            0.018086335927383804,
+            -0.014199453349727933,
+            0.02079719360289355,
+            0.016663293302592568,
+            0.00991445855219406,
+            -0.00033967370741635794,
+            -0.0018726748849858836,
+            0.023373354806639515,
+            0.013258226441642464,
+            0.00861986998150065,
+            -0.0014340218811281975,
+            -0.008516541227326329,
+            0.018239230461345617,
+            0.005458629943667832,
+            -0.012421896028975013,
+            0.012732721349758832,
+            0.013559877132508506,
+            -0.013720939472355347,
+            -0.011545210834395917,
+            0.01616358223872008,
+            -0.019904594319155373,
+            -0.018260016571763644,
+            -0.0036124537591368864,
+            -0.01614510591132948,
+            -0.03864699384205461,
+            0.01619401261888991,
+            0.018351607973242735,
+            0.006666313000123845,
+            0.013647410298896132,
+            0.002597680061427196,
+            0.004545530267244157,
+            -0.018531082430861854,
+            -0.0077618625909995275,
+            -0.008085144524135461,
+            0.003744735923004248,
+            0.011059457364156674,
+            -0.011335432904114519,
+            -0.009514379703581616,
+            -0.00034500702695418285,
+            -0.007433132060989644,
+            0.01971420270994951,
+            0.0020753610290475642,
+            0.0034506706065161024,
+            -0.00628437298309269,
+            0.017966485854173167,
+            0.026789678964152908,
+            -0.007532446825077476,
+            0.021886182869092073,
+            -0.002864230253056434,
+            -0.0015870360120997492,
+            -0.014393519359298496,
+            -0.0014439412842964876,
+            -0.011527182213472351,
+            0.0005453493437517964,
+            -0.0006791481533636673,
+            0.014093767997869645,
+            -0.03288824642147967,
+            -0.003957301296796813,
+            0.014444592778851293,
+            0.007538052312876599,
+            -0.005057755472988024,
+            0.011357919882949903,
+            -0.002964451028327082,
+            0.016977317604413924,
+            0.0075201055190542146,
+            0.023391646079994784,
+            -0.016588823248397513,
+            -0.004504858318770565,
+            0.006660877064954993,
+            0.0034631255931793638,
+            0.0014484203909961972,
+            0.01912983041187014,
+            0.023439941258581887,
+            -0.014715886385460541,
+            -0.0013596123366001458,
+            -0.0029020372473879614,
+            -0.021414221343490594,
+            0.014976806123253727,
+            0.0074729481304182865,
+            0.010967351226114886,
+            0.009806663105020302,
+            0.015465076881663691,
+            -0.02109587527354609,
+            0.0011671121739822202,
+            0.020567458365051382,
+            -0.008618333549105003,
+            -0.018815706906171112,
+            -0.009804829158883123,
+            -0.00966828714719191,
+            -0.006078893678150302,
+            -0.00803783313451146,
+            0.005523023167028105,
+            0.0022647506733877573,
+            0.01652997604669998,
+            -0.008204200434441975,
+            -0.008404609147866811,
+            0.008489247125986078,
+            0.007546785673285454,
+            -0.0033093451283570316,
+            0.010614050963656496,
+            0.0030633179859393587,
+            -0.006940540318011839,
+            0.00894884343338777,
+            -0.002787816848430143,
+            0.014347598920245487,
+            -0.02441769157218451,
+            0.004717742147185283,
+            -0.012096876825337124,
+            0.006405151160675699,
+            0.0041568790527989905,
+            0.0016702066682613428,
+            -0.0041159211459004,
+            0.0035217362856754353,
+            0.0008893479210743541,
+            -0.07433312151926533,
+            -0.012629681414004524,
+            -0.0035797012285864486,
+            0.033657838713146884,
+            -0.019695806809053914,
+            0.004477927056336216,
+            0.011966065644382396,
+            0.0096717896523898,
+            0.003070713232538593,
+            0.006136726101157844,
+            0.006112385569942848,
+            -0.0245258242481677,
+            -0.00015284418977092615,
+            0.004821040694679944,
+            0.0020347004045943454,
+            -0.007532408630122127,
+            0.00750565079228405,
+            0.015327710245879296,
+            0.020989926895388523,
+            -0.010970305087289488,
+            0.017227663358885748,
+            -0.003352718496981928,
+            -0.006068892126250978,
+            0.0014285750322333686,
+            -0.010153799722483554,
+            -0.017307415843726394,
+            -0.006495008086641125,
+            0.0068983699213297075,
+            -0.001982568363667046,
+            0.0044032687939897855,
+            0.003333596495330213,
+            -0.0032171586993948696,
+            0.0072365127663633925,
+            0.013920326463272646,
+            0.006415916101287133,
+            -0.013675562796300214,
+            0.003003782758807665,
+            -0.02500375035312058,
+            0.011945988879567646,
+            -0.012708975181025296,
+            -0.0009488161508767279,
+            -0.0047093877027523014,
+            -0.0035750724825159503,
+            -0.002213435677949806,
+            0.02043567538125188,
+            -0.014304009033474186,
+            -0.01802899775650995,
+            0.008187227768196673,
+            -0.011217013195839762,
+            0.008300740149622635,
+            -0.03382831122769974,
+            0.012386335491333264,
+            -0.0276029934382802,
+            -0.0002243752790856292,
+            -0.004521387639042107,
+            -0.0010631721066129095,
+            -0.009491184157278669,
+            -0.03274559774619223,
+            0.015845705808609598,
+            -0.007204751554079551,
+            -0.020338337341085228,
+            0.0007143424225187802,
+            -0.021267094083674304,
+            0.014150458196361147,
+            -0.01851165553224022,
+            0.0033450353123933058,
+            -0.018775439467431685,
+            0.002782289821140401,
+            -0.0018711350216089556,
+            0.010502195116579441,
+            -0.00557814442809019,
+            -0.0007130962242755068,
+            -0.013072988887382269,
+            -0.002121841212381393,
+            -0.006653444103331213,
+            0.009394907690516121,
+            0.0028678074167728367,
+            -0.009601577734902932,
+            0.011153571864462314,
+            0.013046116892519414,
+            0.019812860521973938,
+            -0.00539047787915588,
+            -0.00555139958411929,
+            -0.011553689090179406,
+            0.009852063270798534,
+            -0.016326212706634428,
+            -0.00827402913960903,
+            0.008982303148984887,
+            0.015216947130173639,
+            0.01026889105233137,
+            0.003352028462252994,
+            0.003150801750574548,
+            0.002531620731961765,
+            0.026744691368233237,
+            -0.0015120532859309638,
+            0.0008324701960729416,
+            -0.014508988496009931,
+            0.022127525293249926,
+            0.02772430884375744,
+            -0.0012206102581726523,
+            -0.00018859639594030037,
+            0.017437360169132967,
+            -0.004697052522250805,
+            -0.0038111184371524645,
+            0.024015246959315593,
+            -0.038396108183338655,
+            -0.008416825232533771,
+            -0.006788816882617399,
+            -0.006043428198944214,
+            -0.009587366247924467,
+            0.007294190273465812,
+            0.00041966080769004275,
+            0.0024709692753547823,
+            -0.0005250904505954556,
+            0.03452858797518386,
+            0.003403515654095309,
+            -0.0003392060101496309,
+            -0.0032980864446307116,
+            0.0019125121221204029,
+            0.0032987517853214346,
+            -0.009116761755779728,
+            -0.038030085122772624,
+            0.006006334000396066,
+            0.007932785488877407,
+            0.00975009749364395,
+            -0.008120121513849434,
+            -0.008864461019882246,
+            0.004909348738798099,
+            -0.02273129283865203,
+            -0.01372636307167243,
+            0.02625229144362647,
+            -0.008523028986127237,
+            0.029005877324974557,
+            -0.022435059061182174,
+            -0.013135836801818153,
+            -0.004126433322528265,
+            0.010283975013795537,
+            -0.019732856462444506,
+            0.004343860338939202,
+            -0.004022387065239562,
+            0.023733721026005954,
+            0.0077578144626961245,
+            -0.008371631446146718,
+            0.012386735614153616,
+            0.0064789380436815714,
+            -0.0026638642827013837,
+            0.00136333052963848,
+            -0.007209517067748103,
+            0.016916505654836485,
+            -0.011819784026452944,
+            -0.005833909403097991,
+            0.008977527508585844,
+            -0.0024556434204852946,
+            0.02509293948918158,
+            -0.0237762245071987,
+            0.005288171058938098,
+            -0.00019047961064753207,
+            0.01822542471604037,
+            0.008127347078975925,
+            0.008361588667386882,
+            -0.011835407590060994,
+            0.010004482616697591,
+            -0.03378107067882622,
+            -0.015433906677849963,
+            -0.007702910250795191,
+            -0.0013877702264578865,
+            -0.002059152251387601,
+            -0.015745218548968338,
+            -0.026393221735795527,
+            0.006294583740809379,
+            0.016267525575472445,
+            0.023291528020980265,
+            -0.004683197930892445,
+            0.012912830736650997,
+            -0.002831583991769044,
+            -0.008904271394710672,
+            -0.011441882849296418,
+            -0.012202629605208118,
+            -0.012470773094383122,
+            -0.014560694782340029,
+            -0.017045701603711442,
+            0.012139665577738337,
+            -0.020079061996873537,
+            0.013911518328924777,
+            0.0347210775522539,
+            -0.017176114991595492,
+            -0.0018941772168839354,
+            0.019702722555622765,
+            -0.004221469612316137,
+            0.01212088353057276,
+            0.0063707489618353725,
+            0.015298014532351866,
+            0.0038643174573634725,
+            -0.012181479717826009,
+            -0.007374005740809251,
+            0.019411457664902376,
+            -0.004902591844710158,
+            0.014591446725717338,
+            0.015069599884557399,
+            -0.008675672164385891,
+            0.002544198867387345,
+            0.01951865904496276,
+            -0.01762101777798757,
+            0.01665654707715257,
+            -0.00458402229480694,
+            -0.009936206685538294,
+            -0.013345740581684638,
+            -0.005336416724426228,
+            -0.019469457091169776,
+            0.0061608454109892425,
+            0.018931676738452464,
+            -0.00533013594306834,
+            0.009310871143727683,
+            -0.0037670793158045,
+            -0.005045151384034014,
+            0.0002041909886640482,
+            -0.04152751596227068,
+            -0.008340911336217609,
+            -0.0006651584370005978,
+            -0.02505748815737022,
+            0.025954102004571416,
+            0.007205602377144382,
+            0.022602344303400163,
+            -0.017168610245491406,
+            0.004459735690564842,
+            -0.005882024209102714,
+            0.013217517324383834,
+            0.02255689669623969,
+            0.0017984057469785134,
+            0.005699128497215882,
+            -0.02353849721739205,
+            -0.0034711576718386185,
+            0.015924913374553738,
+            -0.003604478141616307,
+            -0.0034993903155063817,
+            0.012592308877056896,
+            0.012864938348003884,
+            0.01567418642605425,
+            0.008801753550506258,
+            0.015606443428689292,
+            0.010589933994371628,
+            0.008928721556432368,
+            -0.005805129999077126,
+            -0.001025103064751441,
+            -0.015907755521365925,
+            0.0045874330999415985,
+            0.031701049450012876,
+            0.012594483199773746,
+            0.016783858873835704,
+            -0.01778536268185936,
+            0.00789523134933153,
+            -0.0012706411671644011,
+            0.008325108613530198,
+            -0.001677173669891956,
+            0.029540136288412976,
+            0.011739999605699826,
+            0.015670966746894652,
+            0.008732334090105053,
+            -0.016237731209426752,
+            0.045334690225148,
+            -0.016911928942220152,
+            -0.0021139311364009113,
+            -0.01058970056572469,
+            -0.021705338531028618,
+            0.007324792557651635,
+            -0.006556582104686972,
+            0.0018243188317438144,
+            0.010780292738415671,
+            -0.006471850255151019,
+            0.0006997395548596146,
+            0.003907433192252573,
+            -0.023129313269763224,
+            -0.002550875642832235,
+            -0.02237917712764273,
+            -0.003107775639788617,
+            -0.02621608419628401,
+            -0.00012482032027919036,
+            -0.016207377503696517,
+            -0.0038712739995960753,
+            -0.023315508569288476,
+            -0.014963211090562272,
+            -0.01633110043829169,
+            -0.004241081909202262,
+            -0.008295402942465353,
+            0.004976396647108597,
+            -0.007402107755970892,
+            -0.020321297548927468,
+            0.0020986169430507757,
+            0.016803273183070004,
+            0.01434729320047051,
+            0.018612970083178984,
+            0.00603868809307069,
+            0.014278001198851581,
+            -0.0026350854954985117,
+            0.017171016209107656,
+            -0.006968968261909459,
+            0.006881847166565678,
+            0.011454001418121107,
+            0.02140935294963631,
+            0.011121833270235448,
+            -0.012801756637470525,
+            0.0010870041670931706,
+            -0.009655388034828094,
+            0.002094601818569396,
+            -0.00828486034340067,
+            -0.009190975158350585,
+            0.0017762581946855365,
+            -0.007261996868906865,
+            0.004674405202069334,
+            0.0030526253833799426,
+            -0.0014082232541965316,
+            0.010419302200840906,
+            -0.0024200053377751805,
+            -0.012208685875688886,
+            -0.0033037330470612695,
+            -0.017975940046938498,
+            -0.006958042903103732,
+            -0.0014496522238488239,
+            -0.011711502944602638,
+            -0.01016130418405875,
+            0.003321436189059608,
+            -0.00538282506552848,
+            -0.009353581612407435,
+            0.013884147453136417,
+            0.0001309738696853529,
+            0.0005878330075918333,
+            0.023495280602974504,
+            0.006035028203642483,
+            0.008087594627921345,
+            0.013144288945385904,
+            -0.01023995000994454,
+            -0.022574958230635715,
+            -0.0243435727055662,
+            0.007867022215096418,
+            -0.01573478573423379,
+            0.02324699908478915,
+            -0.007929809922043611,
+            -0.00561630876431978,
+            0.0020982723335057625,
+            -0.013699317926847062,
+            0.008578431496569186,
+            -0.011271381061801356,
+            0.0014450645788554738,
+            -0.019581451799757368,
+            0.018059311800267413,
+            -0.010917911869373693,
+            -0.004302687883018328,
+            0.009466103226841532,
+            0.034049086327052606,
+            -0.0005689605174080737,
+            -0.007410970644432269,
+            -0.006594150877740108,
+            -0.00022810271313786974,
+            0.018766765773409123,
+            -0.0029041815756733333,
+            0.005758781908059541,
+            -0.01563519107805783,
+            0.006571628287171105,
+            -0.006132372252825422,
+            0.0012423261344297225,
+            0.01513074831647933,
+            -0.026306320707079913,
+            -0.015491710256868008,
+            -0.007857352164317122,
+            -0.024893430640551818,
+            0.002253574511878983,
+            0.017579477884206534,
+            0.020505606775521504,
+            -0.020324505029926956,
+            -0.02513332780748072,
+            0.005150966733930881,
+            -0.026580994694344065,
+            0.0037390577082486594,
+            0.1241031878517593,
+            -0.002138404898309707,
+            0.002720723037908521,
+            -0.011425036886011179,
+            0.0022399461009316216,
+            0.005389407121650262,
+            -0.019552543955456567,
+            0.05950746473186985,
+            -0.00638023587296763,
+            -0.004023798021748334,
+            0.002170545188776296,
+            0.001785855390676113,
+            -0.005831534307818096,
+            0.008643501093778434,
+            -0.017602787281866816,
+            -0.02184792441891879,
+            -0.0052215705748838905,
+            -0.021252169339074325,
+            -0.008509621422071373,
+            -0.013000137524907565,
+            0.002893880203681282,
+            -0.015821609145923173,
+            0.010397692426614709,
+            -0.01638916243796692,
+            -0.010283706937701179,
+            -0.01103621454761223,
+            -0.00373552043529251,
+            0.01563553822905027,
+            -0.03188533919653949,
+            -0.004873550301425776,
+            -0.005183590054152726,
+            0.018218971950313273,
+            0.006004328765748325,
+            0.004357734370247252,
+            -0.014230318971458851,
+            0.0064792070801456235,
+            0.0215589173714926,
+            0.0055272493764401305,
+            0.014643335833970546,
+            0.010618070641357003,
+            -0.0022177303813575846,
+            -0.0024402761885758007,
+            0.017541799738501007,
+            -0.005445774414228888,
+            -0.00023502936119626386,
+            -0.00537569046685066,
+            -0.005974703247446444,
+            -0.010712244203425036,
+            0.01843613166635215,
+            -0.009938564643456672,
+            -0.017388592763277214,
+            -0.0015797412875366156,
+            0.02028434083164951,
+            -0.012761534382003973,
+            0.0031906812114950982,
+            0.000788436210275669,
+            0.020589196433948156,
+            0.0007726855314648301,
+            -0.005481260996289263,
+            0.007165014826294827,
+            -0.02956334397512344,
+            0.004095715232997753,
+            0.0003568512047626187,
+            0.027290768799658912,
+            -0.0017631129386615335,
+            0.01552542586826711,
+            -0.020871286597113327,
+            0.004388125824324094,
+            -0.016926008631435403,
+            0.005099899758322398,
+            0.012560923968841436,
+            0.002781049890810361,
+            0.010889021753593997,
+            -0.008500358897706648,
+            -0.021353460542554444,
+            0.012663934009161902,
+            -0.018657094130860155,
+            -0.029156017587920958,
+            0.0018889388761887053,
+            -0.028158076315864772,
+            -0.003048854609750186,
+            0.006812072971079798,
+            -0.035217412591558014,
+            -0.02251675180883659,
+            0.007396524512996277,
+            -0.00995571600201846,
+            -0.011912753666073882,
+            0.010675240017836214,
+            -0.02645155743041798,
+            -0.018137648527851668,
+            0.0010666944556449141,
+            0.012588800077165337,
+            -0.004561825939541096,
+            0.007555558090958502,
+            -0.028558603070876605,
+            0.014816816621445352,
+            -0.01313764261744744,
+            0.016036065063831813,
+            -0.0037797584923144134,
+            0.022217516257304868,
+            0.011131597210722573,
+            0.008795376259031849,
+            -0.004950080558710245,
+            0.021409590863550987,
+            0.005574351961953293,
+            0.051358937509801736,
+            0.031972009383746405,
+            0.0019821003238211276,
+            -0.003251257665378234,
+            0.003313045083311649,
+            0.008294776423730887,
+            -0.00940948257229233,
+            0.005114631360856872,
+            0.028053227377164806,
+            0.016738013668710083,
+            -0.018755924318109216,
+            0.006606711923035118,
+            -0.02702692774986498,
+            0.017950884766781056,
+            -0.022054653845163,
+            0.0003755944703013671,
+            0.007766253634286795,
+            0.0016206993755471613,
+            0.010682347575070117,
+            0.0013842930079185415,
+            0.01705234255261425,
+            0.023975191051210525,
+            -0.002976402980885077,
+            0.017466027107477605,
+            0.012734909275073793,
+            0.018143161773162015,
+            -0.0038399373197265827,
+            0.006929440921265761,
+            -0.012003095293274786,
+            -0.0008883470597690258,
+            0.0020490527756307267,
+            -0.004231091383919565,
+            0.012128848448049833,
+            0.008340302519160299,
+            0.012989849936937843,
+            0.002974488532907779,
+            0.003198854381461613,
+            -0.005309753760578859,
+            -0.0017024982049936384,
+            -0.01763395035509857,
+            -0.01819680226734474,
+            0.028417724274372468,
+            -0.020217879775485576,
+            -0.050118184195774004,
+            0.0025509548991664425,
+            0.014185761006969567,
+            -0.0012021669304388173,
+            -0.0051260576846881515,
+            -0.0005774778872927327,
+            0.005718864494530703,
+            -0.016918749210038116,
+            -0.0015472638835434902,
+            0.008125845907627804,
+            0.020347547062899797,
+            -0.002036218493206914,
+            -0.01079739518866248,
+            0.0038112958713600906,
+            0.02169339303105242,
+            0.01667594322285835,
+            0.012338193092567552,
+            -0.009798409233733167,
+            0.0003198215558702285,
+            -1.6308740825163337e-05,
+            0.0076870827874737325,
+            0.004356644356656038,
+            0.003500023005601613,
+            -0.019968549869439228,
+            -0.008634740687755112,
+            0.012781565963545847,
+            -0.0025637758121512927,
+            0.010243567793632335,
+            -0.00042296044205132426,
+            -0.003069251553118065,
+            -0.009344199449486244,
+            -0.004694953215957636,
+            -0.009583993744231791,
+            0.013783878036521561,
+            -0.007705982042252044,
+            -0.01549567538665377,
+            0.00020666298495823114,
+            0.0005515775451348472,
+            0.001667367278489879,
+            -0.004518431526236256,
+            0.03302324454129141,
+            0.009531990824506103,
+            -0.0004937348108340977,
+            -0.004047613286672229,
+            -0.0327383112522095,
+            0.012456150771224677,
+            -0.009677533214778067,
+            0.0006455811519981003,
+            0.020012976827426605,
+            -0.00020277620481825933,
+            0.002168432791818704,
+            -0.015657551629104767,
+            0.0032924357559238502,
+            0.006161295735841883,
+            0.013814165416788555,
+            0.0005164426482972658,
+            0.01076512534105537,
+            -0.006498977875968572,
+            -0.010325144017898257,
+            0.004804158067334203,
+            0.0104064855510473,
+            -0.0012188100892986933,
+            0.005171197790296134
+          ],
+          "title": "Mixture-Of-Experts With Expert Choice Routing",
+          "keyphrases": [
+            "Mixture-Of-Experts",
+            "Expert Choice Routing",
+            "sparsely-activated models",
+            "training efficiency",
+            "downstream performance"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "46fb6ac4-1a91-4f74-8c57-236109a5fc32",
+        "properties": {
+          "page_content": "B Capacity Factor\n\nWe evaluate the downstream task fine-tuning performance by varying the capacity factors. Note that a capacity factor of n indicates on average how many experts each token can be received. EC-CF2 is our baseline expert choice, which matches GShard top-2 gating computational footprint. EC-CF1, however, matches Switch Transformer top-1 gating computational footprint. EC-CF0.5 further verifies that an aggressively lowered capacity factor can provide strong enough performance, that almost matches the top-2 gating baseline.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Top-2 78.1 87.0 88.3 85.0 82.6 90.1 90.7 81.6 94.7 68.2 67.2 83.0±0.3 EC-CAP2 78.2 88.0 88.5 85.7 83.0 90.8 91.1 80.0 95.4 70.4 64.1 83.2±0.4 EC-CAP3 78.5 91.7 89.3 86.3 83.5 90.9 91.1 81.8 94.9 70.0 65.6 84.0±0.4 EC-CF2 79.1 89.6 89.3 86.8 84.3 91.3 91.2 81.1 95.2 68.1 68.0 84.0±0.2 EC-CF1 77.4 90.6 88.0 85.5 83.6 90.3 91.2 79.8 95.3 66.5 64.9 83.0±0.2 EC-CF0.5 77.4 89.6 86.3 85.2 82.7 91.7 91.0 79.6 94.9 67.3 63.5 83.0 ±0.05 Hash Layers 76.1 85.2 86.7 83.4 82.5 90.0 90.3 75.7 94.0 67.4 63.3 81.3±1.0\n\nTable 2: Comparison between different routing methods in fine-tuning of 100M/64E models. We perform 3 independent fine-tuning runs for each method and report the average results. This gives more accurate difference between the variants of expert choice method, since they achieve close fine-tuning results. We do not report averaged results in other experiments.\n\n",
+          "embedding": [
+            0.00878380797803402,
+            -0.02230931632220745,
+            -0.015165671706199646,
+            -0.002112389076501131,
+            -0.01730332337319851,
+            -0.0011611338704824448,
+            -0.006440163590013981,
+            0.05074397847056389,
+            -0.042286649346351624,
+            -0.006537329405546188,
+            -0.01575644128024578,
+            -0.005678381770849228,
+            -0.013284537009894848,
+            0.007656681817024946,
+            -0.008605021983385086,
+            0.038244541734457016,
+            0.0028061545453965664,
+            0.026398062705993652,
+            0.030253609642386436,
+            -0.007660568691790104,
+            -0.0003531984693836421,
+            -0.035757094621658325,
+            -0.05282721668481827,
+            -0.022464781999588013,
+            -0.01148890983313322,
+            0.020676927641034126,
+            0.011403403244912624,
+            -0.013937492854893208,
+            -0.02294672653079033,
+            0.030082598328590393,
+            0.0195420291274786,
+            0.025667374953627586,
+            -0.0036165190394967794,
+            -0.03728065639734268,
+            -0.0513969361782074,
+            -0.028014905750751495,
+            0.059232402592897415,
+            0.028496848419308662,
+            -0.028963245451450348,
+            0.035601627081632614,
+            0.0162772499024868,
+            0.02532535046339035,
+            -0.04981118440628052,
+            0.0055617825128138065,
+            0.01784745417535305,
+            -0.031683892011642456,
+            -0.03183935955166817,
+            -0.015344456769526005,
+            0.025092151015996933,
+            -0.016354983672499657,
+            -0.020910125225782394,
+            -0.04604891687631607,
+            0.023195471614599228,
+            0.038742031902074814,
+            -0.030642274767160416,
+            0.03675207123160362,
+            0.003350284183397889,
+            0.05873491242527962,
+            0.0325544998049736,
+            0.01063384860754013,
+            0.0017149801133200526,
+            -0.012211824767291546,
+            -0.048194341361522675,
+            -0.008340730331838131,
+            -0.0009648585109971464,
+            0.015235630795359612,
+            -0.019884053617715836,
+            0.0055034831166267395,
+            0.03799579665064812,
+            -0.008364050649106503,
+            -0.06597960740327835,
+            0.04483628645539284,
+            -0.03423352912068367,
+            1.5910934962448664e-05,
+            -0.0011455873027443886,
+            -0.03270996734499931,
+            0.04250429943203926,
+            0.027486322447657585,
+            0.033642761409282684,
+            0.02807709202170372,
+            -0.008022026158869267,
+            0.02874559350311756,
+            -0.025138791650533676,
+            0.006685021799057722,
+            0.014769233763217926,
+            -0.02674008719623089,
+            -0.01284923404455185,
+            -0.05550122633576393,
+            -0.0366898849606514,
+            0.00012388666800688952,
+            -0.02518543042242527,
+            -0.010587208904325962,
+            -0.009304617531597614,
+            0.007656681817024946,
+            -0.02033490315079689,
+            0.05926349386572838,
+            -0.037840332835912704,
+            0.014349476434290409,
+            -0.008830447681248188,
+            -0.00403044605627656,
+            -0.034948673099279404,
+            0.026600167155265808,
+            0.012499435804784298,
+            0.007835467346012592,
+            0.06616616994142532,
+            -0.018857980147004128,
+            -0.014870286919176579,
+            -0.0012155468575656414,
+            -0.01641716994345188,
+            0.07673782855272293,
+            0.0064595965668559074,
+            -0.007330203894525766,
+            -0.03892859071493149,
+            0.005040972959250212,
+            -0.0508994460105896,
+            -0.027735067531466484,
+            -0.03410915657877922,
+            -0.02756405435502529,
+            0.060289569199085236,
+            -0.028512395918369293,
+            0.0017295550787821412,
+            0.05969879776239395,
+            0.024050531908869743,
+            -0.004873847123235464,
+            -0.03622348979115486,
+            0.053200334310531616,
+            0.02229377068579197,
+            -0.0043763574212789536,
+            0.021438708528876305,
+            -0.021936198696494102,
+            -0.0052197580225765705,
+            0.012592715211212635,
+            -0.013851987197995186,
+            -0.039705920964479446,
+            0.03501085937023163,
+            -0.022371502593159676,
+            0.04325053468346596,
+            0.029336363077163696,
+            -0.009926480241119862,
+            -0.018593687564134598,
+            0.028108185157179832,
+            -0.033518388867378235,
+            0.039426080882549286,
+            0.021998384967446327,
+            -0.02530980296432972,
+            0.03911514952778816,
+            0.010493929497897625,
+            0.04974899813532829,
+            0.007866560481488705,
+            0.013976359739899635,
+            -0.011426723562180996,
+            0.016759194433689117,
+            -0.03389150649309158,
+            0.03146624192595482,
+            -0.013408909551799297,
+            -0.026724539697170258,
+            -0.014450529590249062,
+            0.017520975321531296,
+            0.0022406482603400946,
+            -0.03145069628953934,
+            0.0037214583717286587,
+            0.03283433988690376,
+            -0.01626170426607132,
+            -0.017365509644150734,
+            0.002685668645426631,
+            0.04011012986302376,
+            0.02782834693789482,
+            0.012157411314547062,
+            -0.06523337215185165,
+            0.0254497230052948,
+            0.04878511279821396,
+            -0.018718060106039047,
+            0.038493286818265915,
+            0.012312876991927624,
+            0.007306884042918682,
+            0.016603728756308556,
+            0.002194008557125926,
+            0.016743646934628487,
+            -0.010727128013968468,
+            0.022091664373874664,
+            -0.03799579665064812,
+            -0.009654414840042591,
+            0.011838707141578197,
+            -0.0014555469388142228,
+            0.049904465675354004,
+            -0.008154171518981457,
+            0.02073911391198635,
+            0.04567579925060272,
+            -0.000116234841698315,
+            0.005892146844416857,
+            0.025620734319090843,
+            0.038648754358291626,
+            0.013090205378830433,
+            0.005550122819840908,
+            0.007124212104827166,
+            -0.006261378061026335,
+            -0.004663968458771706,
+            0.018547048792243004,
+            0.0328032448887825,
+            0.01475368719547987,
+            -0.029709480702877045,
+            0.014100732281804085,
+            -0.01612178422510624,
+            -0.0195420291274786,
+            -0.0035193529911339283,
+            0.008037572726607323,
+            0.041882436722517014,
+            -0.01888907328248024,
+            -0.005736681632697582,
+            -0.014932473190128803,
+            -0.005464616697281599,
+            -0.018920166417956352,
+            -0.01707012578845024,
+            -0.02913425676524639,
+            -0.01416291855275631,
+            0.030471261590719223,
+            0.007427370175719261,
+            0.0003400810528546572,
+            -0.004457976669073105,
+            -0.0032783811911940575,
+            0.019231097772717476,
+            0.03058008849620819,
+            0.017256684601306915,
+            -0.01717895083129406,
+            -0.017147857695817947,
+            0.00652955612167716,
+            -0.007248584646731615,
+            0.0002594332618173212,
+            0.010571662336587906,
+            -0.013859760016202927,
+            -0.013486642390489578,
+            -0.0009648585109971464,
+            -0.033238548785448074,
+            0.01120907161384821,
+            0.008962593041360378,
+            -0.020676927641034126,
+            0.010385103523731232,
+            -0.011566642671823502,
+            0.0012272067833691835,
+            0.01416291855275631,
+            0.0003378948022145778,
+            0.008177491836249828,
+            -0.0328032448887825,
+            -0.01850040815770626,
+            0.009872066788375378,
+            -0.0005674495478160679,
+            -0.04645312950015068,
+            0.0049982196651399136,
+            0.008612795732915401,
+            -0.010711581446230412,
+            -0.028543489053845406,
+            -0.01823611743748188,
+            0.04424551501870155,
+            0.012965832836925983,
+            -0.008379597216844559,
+            0.03224356845021248,
+            0.007948179729282856,
+            -0.02806154452264309,
+            -0.030486809089779854,
+            -0.006941540166735649,
+            0.04029668867588043,
+            -0.014147371985018253,
+            -0.026398062705993652,
+            0.04023450240492821,
+            -0.018018465489149094,
+            -0.03126413747668266,
+            0.005355790723115206,
+            0.002260081470012665,
+            -0.019060084596276283,
+            0.00021704458049498498,
+            0.02125214971601963,
+            -0.003983806353062391,
+            -0.03448227420449257,
+            -0.0317305326461792,
+            0.0005280972691252828,
+            -0.02715984359383583,
+            -0.029180897399783134,
+            0.009234657511115074,
+            -0.0008647775393910706,
+            0.028403569012880325,
+            0.02832583710551262,
+            0.018935712054371834,
+            0.003678705310449004,
+            0.025931665673851967,
+            -0.023646321147680283,
+            0.014085185714066029,
+            -0.033145271241664886,
+            -0.01508016511797905,
+            -0.042286649346351624,
+            -0.03162170574069023,
+            -0.015896359458565712,
+            -0.024952232837677002,
+            -0.016075145453214645,
+            -0.007034819573163986,
+            0.012359516695141792,
+            -0.02230931632220745,
+            0.0047689080238342285,
+            -0.011621055193245411,
+            0.03410915657877922,
+            -0.0050759525038301945,
+            -0.05444405972957611,
+            -0.004838867578655481,
+            0.012219597585499287,
+            -0.037467215210199356,
+            -0.04384130612015724,
+            -0.0035912557505071163,
+            -0.023195471614599228,
+            -0.0015235630562528968,
+            -0.010167451575398445,
+            -0.0197907742112875,
+            0.028388023376464844,
+            -0.018407128751277924,
+            -0.009211338125169277,
+            -0.03382932022213936,
+            -0.0013010528637096286,
+            0.004438543692231178,
+            -0.014707047492265701,
+            0.005783321335911751,
+            -0.06358543783426285,
+            -0.01641716994345188,
+            0.02467239461839199,
+            -0.006708341650664806,
+            -0.08401361852884293,
+            0.01442720927298069,
+            -0.03373603895306587,
+            0.012250690720975399,
+            0.011838707141578197,
+            0.04051434248685837,
+            -0.02058364823460579,
+            0.007944293320178986,
+            -0.02860567346215248,
+            -0.017381057143211365,
+            -0.035477254539728165,
+            -0.024066077545285225,
+            -0.03805798292160034,
+            -0.013346723280847073,
+            -0.04738591983914375,
+            -0.017738627269864082,
+            -0.016370529308915138,
+            -0.0037136850878596306,
+            -0.02045927569270134,
+            0.035228509455919266,
+            -0.03927061706781387,
+            0.006078705657273531,
+            0.023817332461476326,
+            -0.012989153154194355,
+            -0.010144132189452648,
+            -0.0031384623143821955,
+            -0.009910933673381805,
+            -0.010493929497897625,
+            0.0010017815511673689,
+            0.0013107694685459137,
+            0.030238064005970955,
+            0.027750613167881966,
+            0.006280811037868261,
+            -0.02384842559695244,
+            0.007827693596482277,
+            0.004306397866457701,
+            0.019355470314621925,
+            -0.03905296325683594,
+            -0.012227371335029602,
+            -0.046515315771102905,
+            -0.014069639146327972,
+            0.05155239999294281,
+            -0.03609911724925041,
+            -0.004306397866457701,
+            0.014784780330955982,
+            -0.017365509644150734,
+            -0.002349474234506488,
+            0.005064292810857296,
+            -0.02795271947979927,
+            0.029693933203816414,
+            0.036658793687820435,
+            0.018578141927719116,
+            0.006615062244236469,
+            -0.002516599837690592,
+            0.023630773648619652,
+            -0.03134186938405037,
+            -0.042566485702991486,
+            0.03581928089261055,
+            -0.02715984359383583,
+            0.01494801975786686,
+            -0.005592875648289919,
+            -0.0026545755099505186,
+            -0.011916439980268478,
+            0.06234171614050865,
+            0.025387536734342575,
+            0.012382837012410164,
+            -0.05024648830294609,
+            0.01851595565676689,
+            -0.0023883406538516283,
+            -0.045333776623010635,
+            -0.024750126525759697,
+            -0.022107211872935295,
+            -0.0038477741181850433,
+            -0.006957086734473705,
+            -0.013774254359304905,
+            0.004971013404428959,
+            0.018034012988209724,
+            -0.012872553430497646,
+            -0.02493668533861637,
+            0.02649134211242199,
+            -0.023133285343647003,
+            -0.012631582096219063,
+            0.01120907161384821,
+            -0.004011013079434633,
+            0.00945230945944786,
+            -0.014567128382623196,
+            0.005977652966976166,
+            0.0023961139377206564,
+            -0.0065451026894152164,
+            -0.021407615393400192,
+            -0.010525022633373737,
+            -0.016759194433689117,
+            0.012188504450023174,
+            0.02845020964741707,
+            -0.003931337036192417,
+            0.03622348979115486,
+            0.002846964169293642,
+            0.0024816200602799654,
+            -0.029585108160972595,
+            0.010043079033493996,
+            0.0501532107591629,
+            0.0012388667091727257,
+            0.05012211576104164,
+            -0.003983806353062391,
+            -0.011038059368729591,
+            0.03501085937023163,
+            -0.009530042298138142,
+            0.012942513450980186,
+            0.011170204728841782,
+            0.007493442855775356,
+            0.00652955612167716,
+            -0.010999192483723164,
+            -0.0027089884970337152,
+            -0.0038808106910437346,
+            -0.0025010532699525356,
+            0.035943653434515,
+            -0.022122757509350777,
+            -0.013035791926085949,
+            0.03603693097829819,
+            -0.005118705797940493,
+            -0.016075145453214645,
+            -0.04909604415297508,
+            -0.021967291831970215,
+            -0.0097710145637393,
+            -0.0015206481330096722,
+            0.019573122262954712,
+            -0.02543417550623417,
+            0.020568102598190308,
+            -0.05755337327718735,
+            0.022340409457683563,
+            -0.06283920258283615,
+            0.013533282093703747,
+            0.028667859733104706,
+            0.002365020802244544,
+            0.02294672653079033,
+            -0.018733607605099678,
+            0.02911871112883091,
+            -0.045333776623010635,
+            -0.0071747382171452045,
+            0.016168424859642982,
+            0.03603693097829819,
+            -0.008022026158869267,
+            -0.0027361949905753136,
+            0.011170204728841782,
+            -0.008744941093027592,
+            -0.0325544998049736,
+            -0.006650041788816452,
+            0.02310219220817089,
+            -0.021065590903162956,
+            0.012048585340380669,
+            -0.02244923636317253,
+            -0.0104317432269454,
+            -0.005775548052042723,
+            0.027579601854085922,
+            0.03976810723543167,
+            -0.048007782548666,
+            -0.005472389981150627,
+            7.518220081692562e-05,
+            0.022620247676968575,
+            -0.0025690693873912096,
+            0.01475368719547987,
+            -0.012312876991927624,
+            0.037063002586364746,
+            -0.019215550273656845,
+            0.010610528290271759,
+            -0.009421216323971748,
+            -0.012818140909075737,
+            -0.018049558624625206,
+            -0.011154658161103725,
+            -0.021936198696494102,
+            0.01679028756916523,
+            -0.013828666880726814,
+            0.002605992369353771,
+            -0.032088104635477066,
+            0.03401587903499603,
+            -0.01941765658557415,
+            0.03264778107404709,
+            -0.014279517345130444,
+            0.005437409970909357,
+            -0.024874499067664146,
+            0.01572534814476967,
+            0.005666722077876329,
+            0.01612178422510624,
+            -0.021454256027936935,
+            0.031699441373348236,
+            0.030782192945480347,
+            -0.049904465675354004,
+            -0.02241814322769642,
+            0.0027284217067062855,
+            -0.012631582096219063,
+            0.010268504731357098,
+            -0.01121684443205595,
+            -0.02021053060889244,
+            0.0354461632668972,
+            -0.008348504081368446,
+            0.0052197580225765705,
+            -0.011100245639681816,
+            0.00485052727162838,
+            -0.03028470277786255,
+            -0.006988179869949818,
+            -0.006863807328045368,
+            0.036503326147794724,
+            -0.014380569569766521,
+            0.005235304590314627,
+            -0.0035407296381890774,
+            0.004601782187819481,
+            0.05105490982532501,
+            0.003196761943399906,
+            -0.00014137655671220273,
+            0.00534413056448102,
+            0.004959353245794773,
+            -0.0008458301308564842,
+            0.00011082998389611021,
+            -0.015826400369405746,
+            -0.007893767207860947,
+            0.00015473687381017953,
+            -0.032336849719285965,
+            -0.011263484135270119,
+            0.012219597585499287,
+            0.007610042113810778,
+            0.0029421867802739143,
+            -0.018857980147004128,
+            0.008698301389813423,
+            0.03504195064306259,
+            0.019479842856526375,
+            -0.019510935992002487,
+            0.0027770046144723892,
+            0.0032997578382492065,
+            -0.01207190565764904,
+            -0.028667859733104706,
+            -0.0040343329310417175,
+            0.0026526320725679398,
+            0.0031423489563167095,
+            -0.010734900832176208,
+            0.032212477177381516,
+            -0.027439681813120842,
+            3.349190956214443e-05,
+            -0.027719520032405853,
+            -0.004209231585264206,
+            0.01599741168320179,
+            -0.0036767618730664253,
+            0.03401587903499603,
+            -0.000832226884085685,
+            0.011605508625507355,
+            0.009755467996001244,
+            0.013043565675616264,
+            -0.04076308757066727,
+            0.00925020407885313,
+            0.011240164749324322,
+            -0.01680583320558071,
+            -0.028263650834560394,
+            0.006720001343637705,
+            0.007847127504646778,
+            0.01808065176010132,
+            -0.010159678757190704,
+            -0.018453769385814667,
+            -0.025543002411723137,
+            0.018034012988209724,
+            0.02179628051817417,
+            -0.0419757179915905,
+            0.007703321520239115,
+            0.015453282743692398,
+            0.03420243784785271,
+            -0.0009502836037427187,
+            -0.0036961950827389956,
+            -0.034326810389757156,
+            0.007015386130660772,
+            0.002021053107455373,
+            -0.013167938217520714,
+            0.050712887197732925,
+            -0.010641621425747871,
+            -0.019635308533906937,
+            0.016728101298213005,
+            -0.010727128013968468,
+            -0.018811339512467384,
+            0.02241814322769642,
+            0.014217331074178219,
+            -0.026429155841469765,
+            -0.0005212956457398832,
+            0.016588181257247925,
+            0.029211990535259247,
+            -0.017007939517498016,
+            0.008542835712432861,
+            0.006024292670190334,
+            -0.0019957898184657097,
+            0.015538789331912994,
+            -0.015328910201787949,
+            0.013680974952876568,
+            0.05121037736535072,
+            0.015103485435247421,
+            0.005631742067635059,
+            0.004601782187819481,
+            -0.006156438495963812,
+            0.012079678475856781,
+            -0.004271417856216431,
+            -0.004286964423954487,
+            -0.015119032002985477,
+            0.00011161946167703718,
+            0.05074397847056389,
+            -0.0008594333776272833,
+            0.028419116511940956,
+            -0.0018976522842422128,
+            0.014396116137504578,
+            0.021423162892460823,
+            -0.006914333440363407,
+            -0.02989603951573372,
+            -0.008379597216844559,
+            -0.008511742576956749,
+            -0.0237551461905241,
+            -0.024687940254807472,
+            -0.028698952868580818,
+            -0.026864459738135338,
+            -0.0002560324501246214,
+            -0.0013982189120724797,
+            0.009980892762541771,
+            0.004372470546513796,
+            0.0037972477730363607,
+            0.0008453443297185004,
+            0.004049879498779774,
+            0.023288751021027565,
+            0.013556602410972118,
+            0.020272716879844666,
+            -0.0042558712884783745,
+            0.007326317485421896,
+            -0.0076800016686320305,
+            0.00330947432667017,
+            -0.0035562762059271336,
+            -0.0010795143898576498,
+            -0.0020579760894179344,
+            0.006024292670190334,
+            0.03659660741686821,
+            0.02977166697382927,
+            0.002528259763494134,
+            0.001742186606861651,
+            0.012507209554314613,
+            -0.010120811872184277,
+            0.011815386824309826,
+            -0.0023786239326000214,
+            -0.011076925322413445,
+            0.011325670406222343,
+            0.02073911391198635,
+            -0.014652634970843792,
+            0.015290044248104095,
+            -0.014279517345130444,
+            -0.05519029498100281,
+            0.006082592532038689,
+            -0.017272230237722397,
+            0.022760167717933655,
+            -0.02555854804813862,
+            -0.003968259785324335,
+            -0.017225591465830803,
+            0.00731077091768384,
+            0.014675954356789589,
+            -0.024174904450774193,
+            -0.004325830843299627,
+            0.04082527384161949,
+            0.0013739274581894279,
+            -0.02518543042242527,
+            -0.005355790723115206,
+            -0.003744778223335743,
+            -0.027097657322883606,
+            -0.012538302689790726,
+            0.033114176243543625,
+            -0.009708828292787075,
+            -0.029600653797388077,
+            0.051490213721990585,
+            -0.01462931465357542,
+            0.0022134417667984962,
+            -0.014932473190128803,
+            -0.002841134322807193,
+            -0.03724956139922142,
+            -0.006308017764240503,
+            -0.02860567346215248,
+            0.035601627081632614,
+            -0.024719033390283585,
+            0.026942191645503044,
+            0.023304296657443047,
+            -0.003422186942771077,
+            -0.02139206975698471,
+            0.026833366602659225,
+            -0.030098143965005875,
+            0.0044618635438382626,
+            0.0270199254155159,
+            -0.0005859110387973487,
+            -0.0018228343687951565,
+            0.002858624095097184,
+            -0.021671907976269722,
+            -0.005351903848350048,
+            -0.0017965995939448476,
+            0.00853506289422512,
+            0.027843892574310303,
+            0.009273524396121502,
+            -0.007042592857033014,
+            -0.005923239979892969,
+            0.0012971663381904364,
+            0.012685994617640972,
+            -0.019635308533906937,
+            0.028030451387166977,
+            -0.011605508625507355,
+            0.011123565025627613,
+            -0.008014252409338951,
+            -0.03031579591333866,
+            -0.02345976233482361,
+            0.014963566325604916,
+            0.03696972504258156,
+            0.02308664470911026,
+            0.014279517345130444,
+            0.009195791557431221,
+            0.02174963988363743,
+            -0.00718639837577939,
+            -0.038617659360170364,
+            -0.0028994339518249035,
+            -0.008115305565297604,
+            -0.010361784137785435,
+            0.01812729239463806,
+            0.026289235800504684,
+            0.0024058304261416197,
+            -0.011970852501690388,
+            0.015383323654532433,
+            0.007726641371846199,
+            -0.01901344582438469,
+            0.01943320222198963,
+            -0.010136358439922333,
+            0.021485349163413048,
+            -0.0025651827454566956,
+            0.015709800645709038,
+            0.016852473840117455,
+            0.00045692318235524,
+            0.007194171659648418,
+            -0.023055551573634148,
+            0.029009884223341942,
+            -0.011395630426704884,
+            -0.05736681446433067,
+            0.0036942518781870604,
+            0.0003740891406778246,
+            -0.022091664373874664,
+            -0.013377816416323185,
+            0.015103485435247421,
+            -0.028932152315974236,
+            0.03724956139922142,
+            0.0032025917898863554,
+            -0.005064292810857296,
+            0.023630773648619652,
+            -0.006918220315128565,
+            -0.03488648682832718,
+            0.007959839887917042,
+            0.029072070494294167,
+            -0.01769198849797249,
+            -0.00037797578261233866,
+            0.043623652309179306,
+            0.005278057884424925,
+            -0.03370494768023491,
+            0.009032553061842918,
+            0.03152842819690704,
+            -0.009623321704566479,
+            -0.0007418625173158944,
+            0.012437249533832073,
+            -0.03302089869976044,
+            -0.01784745417535305,
+            0.019308829680085182,
+            0.005009879823774099,
+            0.018951259553432465,
+            -0.028512395918369293,
+            0.02241814322769642,
+            0.008620568551123142,
+            -0.01239838358014822,
+            -0.007524535991251469,
+            0.026569074019789696,
+            0.004496843088418245,
+            0.006377977319061756,
+            -0.015095711685717106,
+            0.013572148978710175,
+            0.049406975507736206,
+            -0.005655061919242144,
+            -0.018671421334147453,
+            0.05599871650338173,
+            -0.02047482319176197,
+            -0.013719840906560421,
+            0.024299276992678642,
+            -0.01822056993842125,
+            -0.007582835853099823,
+            0.03948826715350151,
+            -0.016292797401547432,
+            -0.03386041149497032,
+            0.009669961407780647,
+            -0.031295228749513626,
+            -0.00787822064012289,
+            -0.002689555287361145,
+            0.025014419108629227,
+            -0.013953039422631264,
+            -0.006148665212094784,
+            0.019122270867228508,
+            -0.027595147490501404,
+            -0.00879158079624176,
+            -0.017241137102246284,
+            0.013028019107878208,
+            -0.011325670406222343,
+            -0.02717539109289646,
+            -0.00024036443210206926,
+            -0.0009653443703427911,
+            0.007633361965417862,
+            -0.0025457495357841253,
+            -0.022635795176029205,
+            0.005946559831500053,
+            0.0035873691085726023,
+            -0.011955305933952332,
+            0.06072487309575081,
+            -0.00031287455931305885,
+            0.029196443036198616,
+            -0.02162526734173298,
+            -0.023428669199347496,
+            0.013284537009894848,
+            -0.011123565025627613,
+            0.022387050092220306,
+            0.0038846973329782486,
+            0.019713040441274643,
+            0.0015167614910751581,
+            0.03687644377350807,
+            0.010081945918500423,
+            -0.008962593041360378,
+            -0.032212477177381516,
+            0.01075822114944458,
+            -0.04051434248685837,
+            -0.011395630426704884,
+            0.008830447681248188,
+            -0.02454802207648754,
+            -0.013020245358347893,
+            -0.003565992694348097,
+            0.0206147413700819,
+            0.06498462706804276,
+            0.02373960055410862,
+            -0.009716601110994816,
+            -0.0272686704993248,
+            -0.004566802643239498,
+            -0.011706560850143433,
+            -0.006463483441621065,
+            -0.023133285343647003,
+            0.01192421279847622,
+            -0.045209404081106186,
+            -0.02557409554719925,
+            -0.021671907976269722,
+            -0.0018811339978128672,
+            -0.029320815578103065,
+            -0.016230611130595207,
+            -0.004403563681989908,
+            -0.012787047773599625,
+            0.007711094804108143,
+            -0.01812729239463806,
+            -0.04116729646921158,
+            -0.0016148991417139769,
+            -0.015678707510232925,
+            0.01651044934988022,
+            -0.02756405435502529,
+            -0.021407615393400192,
+            0.012631582096219063,
+            -0.027051018550992012,
+            0.02112777717411518,
+            -0.01108469907194376,
+            0.012763727456331253,
+            -0.010626074858009815,
+            0.031699441373348236,
+            0.0010979759972542524,
+            0.012499435804784298,
+            -0.006276924628764391,
+            0.005829961039125919,
+            -0.032461222261190414,
+            -0.02386397309601307,
+            -0.033767133951187134,
+            0.017878547310829163,
+            -0.014022999443113804,
+            -0.00747400987893343,
+            -0.022651340812444687,
+            0.0011533605866134167,
+            0.009988666512072086,
+            0.010921459645032883,
+            0.0028780573047697544,
+            0.0016090691788122058,
+            0.01081263367086649,
+            -0.008115305565297604,
+            -0.033642761409282684,
+            -0.021718546748161316,
+            -0.023786239326000214,
+            0.017412150278687477,
+            -0.041789159178733826,
+            -0.0016158708604052663,
+            -0.011473363265395164,
+            0.0005684212082996964,
+            -0.02703547105193138,
+            -0.00731465732678771,
+            -0.001748016569763422,
+            0.0021668020635843277,
+            -0.0013583808904513717,
+            0.015919679775834084,
+            0.019852960482239723,
+            0.01207190565764904,
+            -0.02479676716029644,
+            0.05217426270246506,
+            -0.030098143965005875,
+            -0.011496682651340961,
+            0.019324377179145813,
+            0.010369556955993176,
+            0.010703807696700096,
+            -0.021314335986971855,
+            -0.023397576063871384,
+            -0.016619274392724037,
+            0.015119032002985477,
+            0.004419110249727964,
+            -0.008200811222195625,
+            -0.013354497030377388,
+            0.006778301205486059,
+            0.04225555434823036,
+            -0.024843405932188034,
+            -0.00012303645780775696,
+            -0.019775226712226868,
+            0.01417069137096405,
+            -0.00422866502776742,
+            -0.04004794359207153,
+            0.0027964378241449594,
+            -0.02453247457742691,
+            0.002339757513254881,
+            0.01232065074145794,
+            -0.015297817066311836,
+            0.0104317432269454,
+            -0.023630773648619652,
+            0.046484220772981644,
+            0.02059919573366642,
+            -0.012670448049902916,
+            0.03058008849620819,
+            -0.014784780330955982,
+            -0.01075822114944458,
+            0.024454742670059204,
+            0.0051964386366307735,
+            0.031404055655002594,
+            0.0011339273769408464,
+            -0.02414381131529808,
+            0.01916891150176525,
+            -0.012584942393004894,
+            0.03227466344833374,
+            -0.006762754637748003,
+            -0.017925186082720757,
+            -0.006241944618523121,
+            -0.004442430101335049,
+            0.013206804171204567,
+            -0.014605995267629623,
+            -0.009397896938025951,
+            -0.03135741502046585,
+            -0.007120325695723295,
+            0.026833366602659225,
+            0.005351903848350048,
+            -0.0482565276324749,
+            0.0029091504402458668,
+            0.001017328118905425,
+            -0.009825427085161209,
+            0.02229377068579197,
+            -0.006335224024951458,
+            -0.009094739332795143,
+            -0.0033017010428011417,
+            0.035881467163562775,
+            0.05002883821725845,
+            0.012110771611332893,
+            0.004796114284545183,
+            0.012717087753117085,
+            0.036658793687820435,
+            -0.04306397587060928,
+            0.0127248615026474,
+            -0.01494801975786686,
+            -0.03187045082449913,
+            0.02008615806698799,
+            0.0035348995588719845,
+            -0.010284051299095154,
+            0.0337982252240181,
+            -0.015297817066311836,
+            -0.02153198793530464,
+            6.425102765206248e-05,
+            -0.013401136733591557,
+            -0.013556602410972118,
+            -0.007835467346012592,
+            0.0023261543828994036,
+            -0.007302997633814812,
+            -0.007893767207860947,
+            0.0006388665642589331,
+            0.027377495542168617,
+            -0.015017978847026825,
+            0.027921626344323158,
+            0.01528227049857378,
+            0.013432229869067669,
+            -0.021345429122447968,
+            0.012017492204904556,
+            -0.011776520870625973,
+            -2.3729762688162737e-05,
+            0.016230611130595207,
+            -0.0017237251158803701,
+            -0.005969879683107138,
+            0.025822840631008148,
+            -0.0008924698340706527,
+            -0.01952648162841797,
+            -0.006867693737149239,
+            0.008309637196362019,
+            0.0027167617809027433,
+            -0.0014555469388142228,
+            -0.009405669756233692,
+            0.00044502035598270595,
+            0.019852960482239723,
+            0.03765377402305603,
+            -0.034046970307826996,
+            0.004263644572347403,
+            0.003103482536971569,
+            0.0013544942485168576,
+            0.011675468645989895,
+            -0.00826299749314785,
+            -0.006311904173344374,
+            0.019775226712226868,
+            -0.006673361640423536,
+            -0.0016508505214005709,
+            0.006342997308820486,
+            -0.058579444885253906,
+            -0.0149557925760746,
+            0.043219443410634995,
+            -0.01955757476389408,
+            0.0007306884508579969,
+            0.03183935955166817,
+            -0.005037086084485054,
+            0.013066885061562061,
+            -0.029616201296448708,
+            -0.0018306076526641846,
+            -0.0016421056352555752,
+            0.02453247457742691,
+            -0.013066885061562061,
+            -0.002024939749389887,
+            0.017940733581781387,
+            -0.0015799193643033504,
+            -0.01384421344846487,
+            0.024858953431248665,
+            0.01280259434133768,
+            -0.0065528759732842445,
+            2.377530836383812e-05,
+            0.029569560661911964,
+            0.004279191140085459,
+            0.017132312059402466,
+            -0.023273203521966934,
+            0.0104317432269454,
+            -0.0028158710338175297,
+            -0.011271257884800434,
+            0.024330370128154755,
+            -0.005223644897341728,
+            0.011862026527523994,
+            0.025931665673851967,
+            0.031823813915252686,
+            -0.0011795953614637256,
+            -0.004792227875441313,
+            0.0010289880447089672,
+            0.00016068830154836178,
+            0.023646321147680283,
+            0.020381543785333633,
+            -0.011115792207419872,
+            -0.013883080333471298,
+            -0.019992878660559654,
+            0.005655061919242144,
+            -0.0072524710558354855,
+            0.0032744945492595434,
+            0.034948673099279404,
+            0.027346402406692505,
+            0.000937652017455548,
+            0.028045998886227608,
+            -0.016354983672499657,
+            0.017210043966770172,
+            0.011838707141578197,
+            -0.019339922815561295,
+            -0.014357250183820724,
+            0.00619919179007411,
+            -0.016852473840117455,
+            0.011154658161103725,
+            -0.00277117476798594,
+            -0.013276764191687107,
+            -0.006261378061026335,
+            0.08699855953454971,
+            -0.01742769591510296,
+            0.021687453612685204,
+            -0.056869324296712875,
+            -0.027641788125038147,
+            -0.011574415490031242,
+            -0.02359968051314354,
+            0.019837412983179092,
+            0.04312616214156151,
+            0.014030772261321545,
+            0.01943320222198963,
+            0.011310123838484287,
+            -0.006479030009359121,
+            0.03132632374763489,
+            0.0038438874762505293,
+            0.012958060018718243,
+            0.001303967903368175,
+            -0.010501702316105366,
+            -0.013921946287155151,
+            -0.012281783856451511,
+            -0.0015459113055840135,
+            0.012763727456331253,
+            0.005697815213352442,
+            0.010113039053976536,
+            -0.012141864746809006,
+            -0.013657654635608196,
+            -0.006743321195244789,
+            0.021438708528876305,
+            0.00853506289422512,
+            -0.014225104823708534,
+            0.027253123000264168,
+            0.012584942393004894,
+            -0.00026744944625534117,
+            0.010594981722533703,
+            -0.0025088265538215637,
+            0.053697824478149414,
+            -0.02740858867764473,
+            0.007885993458330631,
+            0.031932637095451355,
+            -0.009281297214329243,
+            -0.0027886645402759314,
+            -0.010952552780508995,
+            0.00701927300542593,
+            0.023304296657443047,
+            -0.010672714561223984,
+            0.006747208070009947,
+            0.028045998886227608,
+            -0.005079839378595352,
+            -0.009079192765057087,
+            -0.02139206975698471,
+            -0.008721621707081795,
+            0.01993069238960743,
+            -0.0048194341361522675,
+            0.007427370175719261,
+            -0.004842753987759352,
+            0.015655387192964554,
+            0.00845733005553484,
+            0.0008463159902021289,
+            0.0013272877549752593,
+            -0.0023747372906655073,
+            0.004321944434195757,
+            0.0003070445964112878,
+            0.01666591502726078,
+            0.03625458478927612,
+            0.011823160573840141,
+            0.00236113416031003,
+            0.011310123838484287,
+            -0.0028119843918830156,
+            -0.01548437587916851,
+            0.011457816697657108,
+            -0.00508761266246438,
+            0.002357247518375516,
+            0.009483402594923973,
+            -0.013735387474298477,
+            -0.0060670459643006325,
+            0.016603728756308556,
+            -0.008822673931717873,
+            -0.0034377335105091333,
+            0.008239678107202053,
+            -0.02321101725101471,
+            0.008014252409338951,
+            -0.0010435630101710558,
+            -0.01482364721596241,
+            0.00891595333814621,
+            -0.026973284780979156,
+            -0.0013496358878910542,
+            -0.0012806480517610908,
+            -0.015694255009293556,
+            0.014015225693583488,
+            0.0013127127895131707,
+            0.0011659921146929264,
+            0.0010503645753487945,
+            -0.02073911391198635,
+            0.00040761142736300826,
+            -0.01186980027705431,
+            -0.004100405611097813,
+            -0.0013826723443344235,
+            -0.000706882739905268,
+            -0.05671386048197746,
+            -0.010548342019319534,
+            -0.03389150649309158,
+            0.003078219247981906,
+            0.029693933203816414,
+            0.0003570851113181561,
+            -0.01113911159336567,
+            -0.011722107417881489,
+            0.025154337286949158,
+            -0.004388017114251852,
+            -0.004481296520680189,
+            -0.00277117476798594,
+            -0.008900406770408154,
+            -0.004908827133476734,
+            -0.0009944941848516464,
+            -0.01851595565676689,
+            0.03270996734499931,
+            0.011030285619199276,
+            0.0045978957787156105,
+            -0.03289652615785599,
+            -0.0015585428336635232,
+            -0.0015196764143183827,
+            -0.02464130148291588,
+            -0.016370529308915138,
+            0.023506401106715202,
+            0.011193525046110153,
+            0.0010474496521055698,
+            0.0028255877550691366,
+            0.02345976233482361,
+            -0.018313849344849586,
+            0.021703001111745834,
+            0.01055611576884985,
+            -0.0010940893553197384,
+            3.4888671507360414e-05,
+            0.00423255143687129,
+            -0.014808100648224354,
+            0.014862513169646263,
+            -0.00866720825433731,
+            0.029802760109305382,
+            -0.01756761595606804,
+            0.008970366790890694,
+            -0.009965346194803715,
+            0.019060084596276283,
+            0.03911514952778816,
+            0.030595634132623672,
+            0.026724539697170258,
+            0.012670448049902916,
+            -0.03647223487496376,
+            -0.01009749248623848,
+            -0.002440810203552246,
+            0.005538462661206722,
+            0.009537816047668457,
+            0.004842753987759352,
+            -0.015212311409413815,
+            0.01851595565676689,
+            -0.0005421863752417266,
+            0.006284697912633419,
+            -0.021594174206256866,
+            0.008239678107202053,
+            -0.002110445871949196,
+            0.025418629869818687,
+            0.025745106860995293,
+            0.011963079683482647,
+            -0.0018160328036174178,
+            -0.013035791926085949,
+            -0.007532309275120497,
+            0.02047482319176197,
+            0.013611014932394028,
+            0.022604702040553093,
+            -0.023319844156503677,
+            0.01344000268727541,
+            0.006047612521797419,
+            0.0022659115493297577,
+            0.01940210908651352,
+            0.02227822318673134,
+            0.021454256027936935,
+            0.0013272877549752593,
+            -0.008869313634932041,
+            0.01941765658557415,
+            -0.027346402406692505,
+            0.0049399202689528465,
+            0.0017655064584687352,
+            0.013696521520614624,
+            -0.0364411398768425,
+            0.0016217008233070374,
+            -0.0026273690164089203,
+            -0.0013807290233671665,
+            -0.0113412169739604,
+            -0.029569560661911964,
+            -0.018173931166529655,
+            -0.041509322822093964,
+            0.02204502560198307,
+            -0.004321944434195757,
+            0.031932637095451355,
+            0.007104779127985239,
+            -0.0095222694799304,
+            0.0037972477730363607,
+            0.012219597585499287,
+            0.013603242114186287,
+            0.004710608161985874,
+            0.007862674072384834,
+            -0.023413121700286865,
+            0.00718639837577939,
+            0.029289722442626953,
+            -0.00826299749314785,
+            0.01744324341416359,
+            0.005965993274003267,
+            -0.019852960482239723,
+            0.0041975718922913074,
+            0.011589962057769299,
+            0.007990933023393154,
+            -0.028559034690260887,
+            0.02439255639910698,
+            -0.01515012513846159,
+            -0.009988666512072086,
+            0.009825427085161209,
+            -0.012841460295021534,
+            0.02153198793530464,
+            -0.019852960482239723,
+            -0.01666591502726078,
+            0.00788210704922676,
+            0.003323077689856291,
+            -0.013976359739899635,
+            0.0032064784318208694,
+            0.023910611867904663,
+            -0.017769720405340195,
+            0.0004063968372065574,
+            -0.0045123896561563015,
+            0.02139206975698471,
+            -0.024890046566724777,
+            0.02151644229888916,
+            -0.010439516045153141,
+            -0.019635308533906937,
+            0.016106238588690758,
+            -0.00715141836553812,
+            0.013618788681924343,
+            0.011108018457889557,
+            0.0034727132879197598,
+            0.001421538763679564,
+            0.01245279610157013,
+            0.01572534814476967,
+            0.009615548886358738,
+            -0.011123565025627613,
+            -0.0012524699559435248,
+            -0.005752228200435638,
+            0.01416291855275631,
+            -0.02638251520693302,
+            -0.0002140081487596035,
+            0.021967291831970215,
+            -0.01836048997938633,
+            -0.015562108717858791,
+            0.006832714192569256,
+            0.00468728831037879,
+            0.009382350370287895,
+            0.003812794340774417,
+            -0.03501085937023163,
+            -0.00603595282882452,
+            -0.013167938217520714,
+            -0.003431903664022684,
+            0.03277215361595154,
+            -0.0025748994667083025,
+            -0.014862513169646263,
+            -0.024952232837677002,
+            -1.4673586520075332e-05,
+            0.010120811872184277,
+            0.01968194730579853,
+            -0.002547692973166704,
+            -0.016106238588690758,
+            0.008581702597439289,
+            0.013028019107878208,
+            0.008294090628623962,
+            0.009879840537905693,
+            -0.009483402594923973,
+            0.00682105403393507,
+            0.02689555287361145,
+            0.03647223487496376,
+            -0.009965346194803715,
+            -0.015468829311430454,
+            0.0007501216023229063,
+            -0.018795793876051903,
+            -0.003995466511696577,
+            -0.013028019107878208,
+            -0.005740568041801453,
+            0.0532936155796051,
+            -0.027377495542168617,
+            -0.004862187430262566,
+            -0.02033490315079689,
+            0.00032696363632567227,
+            0.0016090691788122058,
+            -0.001061052898876369,
+            0.007905426435172558,
+            -0.008698301389813423,
+            -0.03243012726306915,
+            0.007501216139644384,
+            0.009957573376595974,
+            0.000347368506481871,
+            0.03249231353402138,
+            0.0113412169739604,
+            -0.013867533765733242,
+            0.01679028756916523,
+            -0.0021881787106394768,
+            -0.0014001622330397367,
+            -0.01941765658557415,
+            -0.013377816416323185,
+            0.0063740904442965984,
+            0.021827373653650284,
+            -0.0017616198165342212,
+            -0.02059919573366642,
+            -0.00793651957064867,
+            0.0052197580225765705,
+            0.00891595333814621,
+            -0.008550609461963177,
+            0.02345976233482361,
+            0.004380243830382824,
+            -0.018578141927719116,
+            0.020910125225782394,
+            -0.018733607605099678,
+            0.0018704457907006145,
+            -0.010167451575398445,
+            -0.0062963576056063175,
+            0.03790251910686493,
+            0.013579921796917915,
+            -0.020770207047462463,
+            -0.013276764191687107,
+            -0.02715984359383583,
+            0.001835466013289988,
+            0.017241137102246284,
+            0.015686480328440666,
+            0.0036825919523835182,
+            -0.002462186850607395,
+            -0.009017006494104862,
+            -0.01311352476477623,
+            -0.025014419108629227,
+            -0.02516988478600979,
+            0.015818627551198006,
+            -0.0009609718690626323,
+            0.04959353432059288,
+            0.00310153909958899,
+            0.004415223840624094,
+            0.005348017439246178,
+            0.004158705472946167,
+            0.007831580936908722,
+            -0.019635308533906937,
+            -0.007306884042918682,
+            -0.0013107694685459137,
+            -0.021811826154589653,
+            0.004772794432938099,
+            0.012880327180027962,
+            -0.012025265954434872,
+            -0.0007020244374871254,
+            -0.001712065190076828,
+            0.01128680445253849,
+            -0.020521461963653564,
+            0.028014905750751495,
+            0.01874915324151516,
+            0.008294090628623962,
+            -0.0003578138421289623,
+            0.02729976363480091,
+            0.00299854320473969,
+            0.012429476715624332,
+            0.0007656681700609624,
+            0.020288264378905296,
+            0.00422866502776742,
+            0.008970366790890694,
+            -0.015935225412249565,
+            0.008542835712432861,
+            -0.015422189608216286,
+            0.035477254539728165,
+            0.0025496361777186394,
+            -0.0388353131711483,
+            0.02271352708339691,
+            -0.006922106724232435,
+            0.0014477736549451947,
+            -0.010618302039802074,
+            -0.026957739144563675,
+            0.01809619925916195,
+            0.01284923404455185,
+            0.010011985898017883,
+            -0.008441783487796783,
+            0.008193038403987885,
+            -0.02174963988363743,
+            0.005892146844416857,
+            -0.01598186604678631,
+            0.012965832836925983,
+            0.0016595955239608884,
+            0.01396858599036932,
+            -0.003507693065330386,
+            0.023646321147680283,
+            0.0071164388209581375,
+            -0.00330947432667017,
+            0.022184943780303,
+            0.0027109317015856504,
+            0.03491757810115814,
+            0.012934739701449871,
+            0.008022026158869267,
+            0.0211122315376997,
+            0.009537816047668457,
+            0.015290044248104095,
+            -0.013758707791566849,
+            -0.004096519201993942,
+            -0.00017647776985540986,
+            0.015017978847026825,
+            -0.006824940908700228,
+            -0.00743902986869216,
+            -0.02571401372551918,
+            0.009327936917543411,
+            -0.0008166803163476288,
+            0.020288264378905296,
+            -0.0077655077911913395,
+            -0.008706075139343739,
+            -0.015002432279288769,
+            -0.044369887560606,
+            -0.011247937567532063,
+            -0.01599741168320179,
+            0.0025418628938496113,
+            0.0024738467764109373,
+            -0.001715951832011342,
+            -0.020039519295096397,
+            0.02162526734173298,
+            -0.0005528746405616403,
+            -0.023972798138856888,
+            -0.0034377335105091333,
+            0.012903646565973759,
+            -0.01626170426607132,
+            0.012546075507998466,
+            -0.004792227875441313,
+            -0.016572635620832443,
+            0.013976359739899635,
+            -0.0011669638333842158,
+            -0.0036981385201215744,
+            -0.027361949905753136,
+            -0.011589962057769299,
+            -0.01455158181488514,
+            0.003192875301465392,
+            0.0019005672074854374,
+            0.010315144434571266,
+            0.011372310109436512,
+            -0.0179718267172575,
+            0.0017256684368476272,
+            0.015398870222270489,
+            0.008224131539463997,
+            0.004302510991692543,
+            0.002841134322807193,
+            0.012079678475856781,
+            -0.009273524396121502,
+            0.0004821863549295813,
+            -0.011403403244912624,
+            -0.01993069238960743,
+            0.005208098329603672,
+            0.010408422909677029,
+            -0.008643888868391514,
+            -0.03538397699594498,
+            -0.007699435111135244,
+            0.013626561500132084,
+            -0.009794333949685097,
+            -0.0009284212719649076,
+            -0.017163405194878578,
+            -0.0005212956457398832,
+            -0.005122592207044363,
+            0.007959839887917042,
+            -0.03019142337143421,
+            -0.0036223488859832287,
+            -0.007777167949825525,
+            -0.0017781381029635668,
+            -0.008177491836249828,
+            -0.0017470449674874544,
+            0.044121142476797104,
+            0.013696521520614624,
+            0.006941540166735649,
+            -0.006405183579772711,
+            -0.004928260110318661,
+            -0.006273037753999233,
+            -0.0003541701298672706,
+            -0.006720001343637705,
+            0.0004105264088138938,
+            -0.014225104823708534,
+            0.016090691089630127,
+            -0.0058455076068639755,
+            0.01430283673107624,
+            0.011862026527523994,
+            -0.007135872263461351,
+            0.026195956394076347,
+            0.0030374096240848303,
+            0.03370494768023491,
+            -0.012421702966094017,
+            0.0014322270872071385,
+            0.017381057143211365,
+            0.0844489261507988,
+            0.013471095822751522,
+            0.008325183764100075,
+            0.0027964378241449594,
+            -0.017940733581781387,
+            -0.009265750646591187,
+            0.003678705310449004,
+            -0.028279196470975876,
+            0.023024458438158035,
+            0.0020987859461456537,
+            0.00997311994433403,
+            -0.007528422866016626,
+            -0.0010843727504834533,
+            0.006024292670190334,
+            0.009763240814208984,
+            0.011815386824309826,
+            0.002611822448670864,
+            0.01285700686275959,
+            0.040483247488737106,
+            0.021547535434365273,
+            0.0014011339517310262,
+            -0.016479356214404106,
+            -0.0010639678221195936,
+            -0.015173444524407387,
+            0.006739434786140919,
+            0.007827693596482277,
+            -0.008744941093027592,
+            0.023055551573634148,
+            0.0021609722170978785,
+            0.010804860852658749,
+            0.013121298514306545,
+            -0.0049127135425806046,
+            0.0021026725880801678,
+            -0.003991579636931419,
+            0.0007374900160357356,
+            0.007497329730540514,
+            -0.006537329405546188,
+            0.008216357789933681,
+            -0.006082592532038689,
+            -0.013859760016202927,
+            0.006999839562922716,
+            -0.010120811872184277,
+            0.003097652457654476,
+            0.0058105275966227055,
+            -0.012514982372522354,
+            0.012048585340380669,
+            0.012491662986576557,
+            0.006261378061026335,
+            0.0006869637290947139,
+            -0.002522429684177041,
+            -0.005208098329603672,
+            0.006074819248169661,
+            0.011636601760983467,
+            -0.0005669636884704232,
+            -0.031792718917131424,
+            -0.0073651839047670364,
+            0.0075361961498856544,
+            -0.004582349210977554,
+            0.005433523561805487,
+            0.01876470074057579,
+            -0.0022892311681061983,
+            -0.013991906307637691,
+            0.00011848181020468473,
+            -0.015049071982502937,
+            -0.00731077091768384,
+            -0.0023144944570958614,
+            0.015834173187613487,
+            -0.010470609180629253,
+            0.016976846382021904,
+            0.012172957882285118,
+            0.00640129717066884,
+            -0.007159191649407148,
+            0.014388343319296837,
+            -0.023972798138856888,
+            0.0019977332558482885,
+            0.0008827532292343676,
+            -0.01161328237503767,
+            -0.0020502028055489063,
+            0.011768747121095657,
+            0.020023971796035767,
+            0.004609555471688509,
+            0.004329717718064785,
+            -0.0011154658859595656,
+            0.013657654635608196,
+            -0.011714334599673748,
+            -0.007816034369170666,
+            0.006673361640423536,
+            0.0006918220315128565,
+            -0.005713361781090498,
+            -0.024719033390283585,
+            -0.0027808912564069033,
+            0.009055872447788715,
+            0.0190445389598608,
+            -0.0047689080238342285,
+            -0.01943320222198963,
+            0.017474336549639702,
+            0.014201784506440163,
+            -0.00899368617683649,
+            -0.010408422909677029,
+            -0.015810852870345116,
+            0.005721135064959526,
+            0.024485835805535316,
+            -0.0014856683555990458,
+            -0.020505916327238083,
+            -0.011325670406222343,
+            -0.014442755840718746,
+            0.018422676250338554,
+            0.007299110759049654,
+            0.011045832186937332,
+            0.0003648583951871842,
+            -0.003826397703960538,
+            0.00024145754287019372,
+            0.014481622725725174,
+            0.020521461963653564,
+            -0.0021318222861737013,
+            -0.0002613765827845782,
+            0.0023008910939097404,
+            0.00033497982076369226,
+            -0.011372310109436512,
+            0.0018072878010571003,
+            0.0024660734925419092,
+            0.017412150278687477,
+            0.0034299602266401052,
+            0.0016693121287971735,
+            0.0049127135425806046,
+            -0.022184943780303,
+            -0.006168098654597998,
+            -0.01665036752820015,
+            -0.008962593041360378,
+            -0.015220084227621555,
+            -0.015919679775834084,
+            -0.005449070129543543,
+            -0.007800487335771322,
+            0.008091985248029232,
+            0.01823611743748188,
+            -0.01108469907194376,
+            -0.0011990285711362958,
+            0.019510935992002487,
+            -0.0031287455931305885,
+            0.004139272030442953,
+            -0.0028936038725078106,
+            -0.0014487452572211623,
+            -0.01876470074057579,
+            -0.003908017184585333,
+            0.015220084227621555,
+            0.010501702316105366,
+            0.012561622075736523,
+            -0.0066383820958435535,
+            0.01940210908651352,
+            -0.025931665673851967,
+            0.025263164192438126,
+            -0.020770207047462463,
+            -0.0014147371985018253,
+            0.021018952131271362,
+            0.0029655066318809986,
+            -0.004609555471688509,
+            -0.015103485435247421,
+            0.01232065074145794,
+            -0.02097231149673462,
+            -0.005550122819840908,
+            -0.034171342849731445,
+            0.03358057513833046,
+            -0.010696034878492355,
+            0.0009998382302001119,
+            -0.004753361456096172,
+            0.0417269729077816,
+            0.012895873747766018,
+            0.005635628942400217,
+            -0.00011544537119334564,
+            -0.014668181538581848,
+            -0.0025379762519150972,
+            0.01240615639835596,
+            -0.0051886653527617455,
+            0.0040071262046694756,
+            0.01260048896074295,
+            0.0066306088119745255,
+            0.0013049395056441426,
+            0.010626074858009815,
+            0.00833295751363039,
+            0.007955953478813171,
+            -0.01449716929346323,
+            -0.010649395175278187,
+            -0.009234657511115074,
+            0.018065106123685837,
+            0.005449070129543543,
+            0.00866720825433731,
+            -0.003737004939466715,
+            0.008402916602790356,
+            0.015686480328440666,
+            -0.007831580936908722,
+            -0.015771986916661263,
+            -0.010571662336587906,
+            -0.016588181257247925,
+            -0.0330519899725914,
+            -0.003834170987829566,
+            0.011955305933952332,
+            -0.006731661502271891,
+            -0.000800162146333605,
+            0.010696034878492355,
+            -0.003268664702773094,
+            0.014232877641916275,
+            0.028792232275009155,
+            -0.0058105275966227055,
+            -0.0016071258578449488,
+            0.0009478544816374779,
+            0.002372794086113572,
+            0.01599741168320179,
+            0.008193038403987885,
+            0.03504195064306259,
+            -0.015375549905002117,
+            -0.0427219532430172,
+            -0.003435790305957198,
+            0.012965832836925983,
+            -0.01494801975786686,
+            -0.021143324673175812,
+            0.005452956538647413,
+            -0.005934900138527155,
+            -0.0052158716134727,
+            0.028698952868580818,
+            0.0021434822119772434,
+            -0.01469927467405796,
+            0.008737168274819851,
+            0.0453648678958416,
+            -0.0031423489563167095,
+            0.013416683301329613,
+            -0.005585102364420891,
+            0.016992392018437386,
+            -0.032865431159734726,
+            -0.013292310759425163,
+            -0.02347530797123909,
+            0.00932016409933567,
+            0.01232842355966568,
+            -0.0008254252607002854,
+            0.032212477177381516,
+            0.016292797401547432,
+            0.012180731631815434,
+            -0.004201458301395178,
+            0.016292797401547432,
+            -0.032057009637355804,
+            -0.04063871502876282,
+            -0.007097005844116211,
+            -0.012864780612289906,
+            -0.009965346194803715,
+            0.001216518459841609,
+            0.007295224349945784,
+            0.0013253444340080023,
+            0.02768842689692974,
+            0.0032978144008666277,
+            -0.010874819941818714,
+            -2.778340785880573e-05,
+            -0.0068171676248312,
+            -0.002674008719623089,
+            -0.019339922815561295,
+            0.010571662336587906,
+            0.008729394525289536,
+            0.016712553799152374,
+            -0.011574415490031242,
+            -0.012079678475856781,
+            0.0012534415582194924,
+            -0.021189963445067406,
+            -0.019308829680085182,
+            -0.027455229312181473,
+            0.007831580936908722,
+            -0.015709800645709038,
+            0.012872553430497646,
+            0.011247937567532063,
+            0.013743161223828793,
+            0.0018344942945986986,
+            -0.013129071332514286,
+            0.027843892574310303,
+            -0.015841946005821228,
+            0.007217491511255503,
+            0.01344000268727541,
+            0.00871384795755148,
+            0.00820858497172594,
+            0.0021609722170978785,
+            -0.027641788125038147,
+            -0.011691015213727951,
+            0.015663161873817444,
+            -0.010423969477415085,
+            0.005794981028884649,
+            -0.009335710667073727,
+            -0.0017927129520103335,
+            -0.007816034369170666,
+            -0.008224131539463997,
+            -0.003958543296903372,
+            0.011574415490031242,
+            0.01462931465357542,
+            0.0012952230172231793,
+            0.026833366602659225,
+            -0.023008912801742554,
+            0.004003239795565605,
+            -0.005184778477996588,
+            0.0067744143307209015,
+            -0.001097004278562963,
+            0.006685021799057722,
+            0.018142838031053543,
+            -0.007163078524172306,
+            0.011504456400871277,
+            0.00587271386757493,
+            -0.015639841556549072,
+            -0.012172957882285118,
+            -0.02137652225792408,
+            0.002742024837061763,
+            -0.016603728756308556,
+            -0.0005878543597646058,
+            -0.001586721045896411,
+            -0.003968259785324335,
+            0.021936198696494102,
+            -0.0040926323272287846,
+            -0.023444214835762978,
+            0.024066077545285225,
+            0.011605508625507355,
+            0.023553041741251945,
+            -0.0030354661867022514,
+            0.030533447861671448,
+            -0.011706560850143433,
+            0.002701215213164687,
+            -0.027626240625977516,
+            0.03187045082449913,
+            -0.00419368501752615,
+            0.009763240814208984,
+            -0.011434496380388737,
+            -0.01148890983313322,
+            -0.019868506118655205,
+            -0.0043685841374099255,
+            0.008744941093027592,
+            0.012577168643474579,
+            0.0051031592302024364,
+            -0.015803080052137375,
+            0.023521948605775833,
+            0.004858300555497408,
+            -0.01239838358014822,
+            0.0009920649463310838,
+            0.009351257234811783,
+            -0.009032553061842918,
+            -0.006444049999117851,
+            -0.0058183008804917336,
+            0.01848486252129078,
+            -0.006852147169411182,
+            -0.01089036650955677,
+            -0.03267887234687805,
+            -0.01575644128024578,
+            -0.0033580574672669172,
+            0.0021706887055188417,
+            0.01927773654460907,
+            -0.007769394665956497,
+            -0.01795627921819687,
+            0.0028178144712001085,
+            0.010004213079810143,
+            -0.016044052317738533,
+            0.00011028342851204798,
+            -0.0018976522842422128,
+            0.01572534814476967,
+            -0.0047689080238342285,
+            0.018205024302005768,
+            0.0208634864538908,
+            0.036907538771629333,
+            0.0022659115493297577,
+            -0.0029557901434600353,
+            0.010734900832176208,
+            0.010610528290271759,
+            -0.008472876623272896,
+            -0.007384616881608963,
+            -0.007722754962742329,
+            -0.02215385064482689,
+            0.034451182931661606,
+            -0.009219110943377018,
+            -0.006063159089535475,
+            -0.00067287462297827,
+            0.022635795176029205,
+            -0.0034902032930403948,
+            -0.025263164192438126,
+            -0.024765674024820328,
+            -0.011146885342895985,
+            0.04645312950015068,
+            -0.021842919290065765,
+            -0.01153554953634739,
+            -0.026335876435041428,
+            0.01745878905057907,
+            -0.02017943747341633,
+            -0.0035193529911339283,
+            0.0011076926020905375,
+            0.005569555796682835,
+            -0.0033036444801837206,
+            -0.005324697587639093,
+            -0.004213118460029364,
+            0.0054685031063854694,
+            0.01318348478525877,
+            0.016572635620832443,
+            -0.014380569569766521,
+            0.021423162892460823,
+            -0.00932016409933567,
+            0.008325183764100075,
+            -0.016479356214404106,
+            0.007990933023393154,
+            -0.023257657885551453,
+            0.024485835805535316,
+            -0.007812147494405508,
+            -0.024205997586250305,
+            0.012297330424189568,
+            0.012499435804784298,
+            0.012693768367171288,
+            0.00015303646796382964,
+            0.012841460295021534,
+            -0.023677414283156395,
+            -0.0005470446776598692,
+            -0.006929880008101463,
+            -0.00426753144711256,
+            -0.01311352476477623,
+            -0.011317897588014603,
+            0.013657654635608196,
+            -0.0010814577108249068,
+            -0.012118545360863209,
+            0.019386563450098038,
+            -0.0066306088119745255,
+            -0.012996925972402096,
+            -0.02478121966123581,
+            0.006541215814650059,
+            -0.007827693596482277,
+            0.005324697587639093,
+            -0.0007005669758655131,
+            0.01599741168320179,
+            -0.005456843413412571,
+            -0.001588664366863668,
+            0.008814901113510132,
+            0.013704294338822365,
+            0.004566802643239498,
+            -0.012717087753117085,
+            -0.023661866784095764,
+            0.0007758706342428923,
+            0.011504456400871277,
+            0.014730367809534073,
+            -0.006580082233995199,
+            -0.017474336549639702,
+            -0.00017793526058085263,
+            -0.0010843727504834533,
+            -0.00970105454325676,
+            -0.0041975718922913074,
+            -0.00041465595131739974,
+            -0.011131338775157928,
+            -0.003937166649848223,
+            -0.01076599396765232,
+            0.017349964007735252,
+            0.0032025917898863554,
+            0.015546562150120735,
+            -0.0040693124756217,
+            -0.019728587940335274,
+            0.012056359089910984,
+            0.041011832654476166,
+            -0.019713040441274643,
+            0.021967291831970215,
+            -0.008099758997559547,
+            -0.0015993525739759207,
+            -0.014862513169646263,
+            0.015468829311430454,
+            -0.006665588356554508,
+            0.018733607605099678,
+            0.020925672724843025,
+            0.008643888868391514,
+            -0.0021551421377807856,
+            0.016836926341056824,
+            0.021687453612685204,
+            -0.0008210528176277876,
+            0.0062380582094192505,
+            0.019184457138180733,
+            0.008286317810416222,
+            0.005363564006984234,
+            -0.0020016198977828026,
+            -0.030067050829529762,
+            0.006805507466197014,
+            -0.0005766802933067083,
+            -0.004803887568414211,
+            0.021189963445067406,
+            0.02504551224410534,
+            0.010991419665515423,
+            0.0033036444801837206,
+            0.006148665212094784,
+            0.012095225043594837,
+            0.0026409721467643976,
+            0.0031540088821202517,
+            -0.0042558712884783745,
+            -0.0018685024697333574,
+            -0.009856520220637321,
+            0.004003239795565605,
+            0.008177491836249828,
+            -0.004691175185143948,
+            0.017489882186055183,
+            -0.01148890983313322,
+            -0.014504942111670971,
+            -0.010260730981826782,
+            -0.014186237938702106,
+            0.02176518738269806,
+            -0.017878547310829163,
+            -0.000856032595038414,
+            -0.0036301221698522568,
+            -6.595143349841237e-05,
+            0.00037141708889976144,
+            -0.00853506289422512,
+            0.02188955992460251,
+            0.008410690352320671,
+            -0.022651340812444687,
+            0.01154332235455513,
+            -0.004240324720740318,
+            0.005798867903649807,
+            -0.006995953153818846,
+            0.01030737068504095,
+            -0.01598186604678631,
+            0.010571662336587906,
+            -0.008845994248986244,
+            0.02296227216720581,
+            -0.013090205378830433,
+            -0.005425750277936459,
+            0.0041120657697319984,
+            -0.002913037082180381,
+            0.01693020574748516,
+            -0.0037991912104189396,
+            -0.010369556955993176,
+            0.003678705310449004,
+            0.011963079683482647,
+            -0.006350770592689514,
+            0.009506722912192345,
+            0.011185751296579838,
+            0.003565992694348097,
+            -0.006109798792749643,
+            -0.0015595145523548126,
+            0.00619919179007411,
+            -0.02927417680621147,
+            -0.00925020407885313,
+            0.0213609766215086,
+            -0.017862999811768532,
+            -0.004916600417345762,
+            0.017520975321531296,
+            0.015445509925484657,
+            -0.01916891150176525,
+            -0.00558121595531702,
+            0.008744941093027592,
+            -0.0020657493732869625,
+            0.023179924115538597,
+            -0.009351257234811783,
+            -0.01719449833035469,
+            0.01277927402406931,
+            -0.013867533765733242,
+            0.011022512800991535,
+            -0.013261217623949051,
+            0.000477813882753253,
+            -0.015639841556549072,
+            0.007952066138386726,
+            -0.014100732281804085,
+            -0.022216036915779114,
+            0.011675468645989895,
+            -0.008690528571605682,
+            -0.001947206910699606,
+            -0.023972798138856888,
+            0.0299115851521492,
+            0.032057009637355804,
+            0.026133770123124123,
+            -0.02005506493151188,
+            0.0026293122209608555,
+            -0.007695548236370087,
+            -0.018438221886754036,
+            -0.0034688266459852457,
+            -0.01166769489645958,
+            0.01654154248535633,
+            0.01940210908651352,
+            -0.002988826483488083,
+            -0.018142838031053543,
+            -0.016090691089630127,
+            0.004671741742640734,
+            -0.02202947810292244,
+            -0.01113911159336567,
+            -0.014613768085837364,
+            -0.0004333604301791638,
+            -0.02112777717411518,
+            0.007613928988575935,
+            0.002753684762865305,
+            0.008892633952200413,
+            0.0028294743970036507,
+            0.003171498654410243,
+            -0.019884053617715836,
+            -0.007170851808041334,
+            0.011076925322413445,
+            0.007190284784883261,
+            -0.012740408070385456,
+            0.02795271947979927,
+            -0.009265750646591187,
+            -0.0015750611200928688,
+            -0.0020754660945385695,
+            -0.018857980147004128,
+            0.013525509275496006,
+            0.010214091278612614,
+            -0.016090691089630127,
+            -0.0071669649332761765,
+            -0.010548342019319534,
+            0.0025612961035221815,
+            0.010797087103128433,
+            -0.001009554835036397,
+            -0.006129232235252857,
+            -0.010369556955993176,
+            0.008418463170528412,
+            -0.011154658161103725,
+            -0.00800647959113121,
+            -0.012942513450980186,
+            0.002843077527359128,
+            0.017520975321531296,
+            0.004154818598181009,
+            0.0020793527364730835,
+            -0.019479842856526375,
+            0.005351903848350048,
+            -0.01795627921819687,
+            0.0003585426020435989,
+            0.0009386236779391766,
+            -0.020365996286273003,
+            0.007959839887917042,
+            0.006428503431379795,
+            0.004543482791632414,
+            0.003585425904020667,
+            -0.005690041929483414,
+            0.008643888868391514,
+            -0.02109668403863907,
+            -0.010952552780508995,
+            0.025154337286949158,
+            -0.0025826727505773306,
+            0.015360003337264061,
+            0.012157411314547062,
+            -0.0024991098325699568,
+            -0.009592228569090366,
+            -0.026817819103598595,
+            -0.0035232396330684423,
+            -0.007835467346012592,
+            0.011481136083602905,
+            0.0019977332558482885,
+            -0.028932152315974236,
+            -0.0036262355279177427,
+            -0.026600167155265808,
+            -0.0027808912564069033,
+            -0.019060084596276283,
+            0.00800647959113121,
+            0.018205024302005768,
+            0.00252437312155962,
+            0.006615062244236469,
+            0.00041538471123203635,
+            0.0011018626391887665,
+            -0.0027439682744443417,
+            0.013541055843234062,
+            0.018034012988209724,
+            0.011706560850143433,
+            -0.019231097772717476,
+            0.011045832186937332,
+            -0.003169555449858308,
+            -0.020801300182938576,
+            0.006548989098519087,
+            -0.02558964118361473,
+            -0.007089232560247183,
+            -3.501012906781398e-05,
+            0.0018626725068315864,
+            -0.011760974302887917,
+            -1.905364842968993e-05,
+            -0.0007433199789375067,
+            0.001582834403961897,
+            0.005114818923175335,
+            0.013595468364655972,
+            -0.012219597585499287,
+            0.019308829680085182,
+            0.006583969108760357,
+            0.008185264654457569,
+            0.0020910126622766256,
+            0.0048544141463935375,
+            -0.013074658811092377,
+            -0.03389150649309158,
+            -0.017940733581781387,
+            -0.013028019107878208,
+            0.008340730331838131,
+            0.0029616199899464846,
+            0.005410203710198402,
+            -0.02454802207648754,
+            -0.016588181257247925,
+            0.012250690720975399,
+            0.01680583320558071,
+            0.00793651957064867,
+            0.007998705841600895,
+            -0.014139598235487938,
+            -0.00846510287374258,
+            0.0038147377781569958,
+            -8.122471626847982e-06,
+            0.01403854601085186,
+            0.00763724884018302,
+            -0.006937653291970491,
+            0.008278544060885906,
+            -0.0023961139377206564,
+            -0.02321101725101471,
+            -0.0019578952342271805,
+            0.010004213079810143,
+            -0.007784941233694553,
+            -0.006420730147510767,
+            -0.0009779758984223008,
+            0.008519516326487064,
+            0.011582189239561558,
+            -0.00443077040836215,
+            -0.0013904456282034516,
+            -0.0042830780148506165,
+            0.018982352688908577,
+            -0.015181218273937702,
+            -0.006273037753999233,
+            0.010937006212770939,
+            -0.004831094294786453,
+            0.013579921796917915,
+            0.006113685667514801,
+            0.04076308757066727,
+            -0.012413930147886276,
+            -0.013984132558107376,
+            -0.014738140627741814,
+            -0.008908180519938469,
+            -0.031046485528349876,
+            -0.01812729239463806,
+            -0.022262677550315857,
+            -0.005923239979892969,
+            0.0035446162801235914,
+            0.011691015213727951,
+            0.013774254359304905,
+            -0.007135872263461351,
+            -0.0002958705008495599,
+            0.002988826483488083,
+            0.0008429151494055986,
+            -0.01088259369134903,
+            -0.008954820223152637,
+            -0.0009687451529316604,
+            -0.014147371985018253,
+            0.002203725278377533,
+            -0.018920166417956352,
+            0.009716601110994816,
+            0.018982352688908577,
+            0.013719840906560421,
+            -0.007967612706124783,
+            0.01265490148216486,
+            -0.0136343352496624,
+            0.0013807290233671665,
+            -0.007952066138386726,
+            0.011675468645989895,
+            0.03488648682832718,
+            0.0038108511362224817,
+            -0.02257360890507698,
+            0.0009765184368006885,
+            -0.0012524699559435248,
+            -0.018780246376991272,
+            0.0056706084869802,
+            -0.021687453612685204,
+            0.01943320222198963,
+            -0.015461056493222713,
+            -0.006770527921617031,
+            6.097167351981625e-05,
+            0.0009036439005285501,
+            0.02479676716029644,
+            0.010742674581706524,
+            -0.015119032002985477,
+            0.019961785525083542,
+            0.011263484135270119,
+            -0.006168098654597998,
+            0.007990933023393154,
+            0.02229377068579197,
+            -0.01029959786683321,
+            0.010719354264438152,
+            0.01943320222198963,
+            -0.02031935751438141,
+            -0.009460083208978176,
+            -0.00301797641441226,
+            0.019324377179145813,
+            0.00845733005553484,
+            -0.009436762891709805,
+            0.0036301221698522568,
+            0.00021121461759321392,
+            0.004504616372287273,
+            -0.0014244538033381104,
+            -0.011240164749324322,
+            -0.006012632977217436,
+            -0.004038219340145588,
+            -0.0001887449761852622,
+            0.010517248883843422,
+            -0.015072392299771309,
+            -0.005942673422396183,
+            -0.0018490692600607872,
+            0.015367777086794376,
+            -0.022884540259838104,
+            -0.0017295550787821412,
+            0.005068179219961166,
+            -0.006397410295903683,
+            0.007139758672565222,
+            0.020023971796035767,
+            0.01783190667629242,
+            0.033518388867378235,
+            -0.014528262428939342,
+            0.0033483407460153103,
+            0.02232486382126808,
+            -0.0036106889601796865,
+            0.02031935751438141,
+            0.014403889887034893,
+            -0.004178138449788094,
+            -0.007470123004168272,
+            0.013572148978710175,
+            0.029989318922162056,
+            0.011955305933952332,
+            0.001268016523681581,
+            -0.02336648292839527,
+            -0.0017888263100758195,
+            0.021594174206256866,
+            0.009086965583264828,
+            0.0005387855344451964,
+            0.010346237570047379,
+            0.014815873466432095,
+            -0.013152391649782658,
+            -0.011201297864317894,
+            0.033238548785448074,
+            -0.017132312059402466,
+            -0.02451692894101143,
+            -0.00046615395694971085,
+            -0.005472389981150627,
+            -0.06116017699241638,
+            -0.002351417439058423,
+            0.014668181538581848,
+            -0.0032589479815214872,
+            0.018951259553432465,
+            0.007555629126727581,
+            0.00866720825433731,
+            0.0049982196651399136,
+            -0.01371206808835268,
+            0.003525183070451021,
+            -0.017163405194878578,
+            0.005775548052042723,
+            -0.006245831493288279,
+            -0.003826397703960538,
+            -0.0037739279214292765,
+            0.007497329730540514,
+            0.023413121700286865,
+            -0.0022348184138536453,
+            0.013533282093703747,
+            0.00669279508292675,
+            0.0023280975874513388,
+            0.01808065176010132,
+            0.006595628801733255,
+            0.00406542606651783,
+            0.0008555467356927693,
+            -0.003579595824703574,
+            -0.010315144434571266,
+            -0.018686966970562935,
+            -0.017754174768924713,
+            -0.014442755840718746,
+            0.006012632977217436,
+            0.002687611849978566,
+            -0.01860923506319523,
+            -0.006953199859708548,
+            0.013976359739899635,
+            -0.03348729386925697,
+            0.002287287963554263,
+            0.016852473840117455,
+            0.014248424209654331,
+            0.009755467996001244,
+            0.02165636047720909,
+            0.023273203521966934,
+            -0.0081386249512434,
+            -0.021874012425541878,
+            -0.006949313450604677,
+            0.001381700742058456,
+            -0.012468342669308186,
+            0.017894092947244644,
+            0.023661866784095764,
+            -0.013999679125845432,
+            -0.010804860852658749,
+            -0.004800001159310341,
+            -0.006078705657273531,
+            0.014248424209654331,
+            0.01640162244439125,
+            0.01968194730579853,
+            0.006144778802990913,
+            0.010113039053976536,
+            -0.017552068457007408,
+            0.012491662986576557,
+            0.0133855901658535,
+            -0.009149151854217052,
+            -0.019977333024144173,
+            -0.019184457138180733,
+            -0.021936198696494102,
+            -0.01665036752820015,
+            -0.024750126525759697,
+            0.028263650834560394,
+            0.002454413566738367,
+            0.013129071332514286,
+            0.00977878738194704,
+            -0.014528262428939342,
+            -0.0022348184138536453,
+            0.011263484135270119,
+            0.01680583320558071,
+            0.003188988659530878,
+            0.0005562754231505096,
+            -0.0002934413496404886,
+            0.014124051667749882,
+            -0.015134578570723534,
+            0.007656681817024946,
+            -0.004706721752882004,
+            -0.009910933673381805,
+            0.0034902032930403948,
+            0.005445183254778385,
+            -0.0077655077911913395,
+            -0.012763727456331253,
+            -0.007753848098218441,
+            0.0036689885891973972,
+            -0.007909313775599003,
+            -0.003435790305957198,
+            0.00037165998946875334,
+            -0.0005562754231505096,
+            0.018920166417956352,
+            -0.015064618550240993,
+            -0.0013243727153167129,
+            0.010338463820517063,
+            -0.0064090704545378685,
+            -0.00682105403393507,
+            0.004011013079434633,
+            0.00788210704922676,
+            -0.0063740904442965984,
+            0.007781054358929396,
+            0.016075145453214645,
+            -0.009685507975518703,
+            0.012227371335029602,
+            -0.002194008557125926,
+            0.00587271386757493,
+            0.023008912801742554,
+            -0.001738299964927137,
+            0.002992713125422597,
+            0.011838707141578197,
+            -0.021407615393400192,
+            0.0007097977213561535,
+            -0.015935225412249565,
+            -0.012748180888593197,
+            -0.015546562150120735,
+            -0.010493929497897625,
+            -0.02022607810795307,
+            -0.006704454775899649,
+            0.00937457662075758,
+            -0.004178138449788094,
+            0.010159678757190704,
+            -0.0011941703269258142,
+            -5.492916170624085e-05,
+            -0.020537009462714195,
+            0.0059815398417413235,
+            -0.00788210704922676,
+            0.0016731987707316875,
+            0.0018306076526641846,
+            0.027517415583133698,
+            0.005627855658531189,
+            -0.0029791099950671196,
+            0.005522916093468666,
+            0.013253443874418736,
+            -0.01812729239463806,
+            -0.010610528290271759,
+            0.012087452225387096,
+            0.009631095454096794,
+            -0.009382350370287895,
+            -0.009677735157310963,
+            -0.010820407420396805,
+            -0.031543973833322525,
+            0.006902673747390509,
+            -0.009561135433614254,
+            0.001341862604022026,
+            -0.0002895547077059746,
+            -0.04278413951396942,
+            0.006739434786140919,
+            -0.0019063971703872085,
+            -0.013346723280847073,
+            0.002019109670072794,
+            -0.019619761034846306,
+            0.00871384795755148,
+            -0.01599741168320179,
+            0.01331563014537096,
+            -0.012040812522172928,
+            -0.0023242109455168247,
+            0.009980892762541771,
+            0.023288751021027565,
+            0.0058455076068639755,
+            0.005740568041801453,
+            0.007120325695723295,
+            -8.714576688362285e-05,
+            0.005266397725790739,
+            -0.0022814578842371702,
+            0.0022387050557881594,
+            -0.0017791097052395344,
+            0.015103485435247421,
+            0.002036599675193429,
+            0.01654154248535633,
+            -0.007784941233694553,
+            -0.002930526854470372,
+            -0.008123078383505344,
+            0.01174542773514986,
+            -0.023288751021027565,
+            0.0014720651088282466,
+            0.0007588665466755629,
+            0.0006340082618407905,
+            -0.004123725462704897,
+            0.004535709507763386,
+            -0.007182511501014233,
+            0.031046485528349876,
+            0.016557088121771812,
+            -0.009102512151002884,
+            0.003991579636931419,
+            5.286438317853026e-05,
+            -0.0001824291975935921,
+            0.021734094247221947,
+            -0.01993069238960743,
+            -0.0056744953617453575,
+            0.0027187049854546785,
+            -0.01901344582438469,
+            0.0021881787106394768,
+            0.028512395918369293,
+            -0.014901380054652691,
+            0.001269959844648838,
+            0.00017319842299912125,
+            0.003350284183397889,
+            -0.018811339512467384,
+            0.001708178548142314,
+            0.014613768085837364,
+            0.007575062569230795,
+            -0.003012146335095167,
+            0.015429963357746601,
+            -0.012382837012410164,
+            0.009483402594923973,
+            0.0058688269928097725,
+            -0.0010124698746949434,
+            0.004318057559430599,
+            -0.000593198521528393,
+            -0.02977166697382927,
+            -0.005095385946333408,
+            0.004038219340145588,
+            -0.0019501218339428306,
+            -0.010999192483723164,
+            0.003651498816907406,
+            -0.009685507975518703,
+            -0.01212631817907095,
+            -0.007034819573163986,
+            0.03721847012639046,
+            0.015515469014644623,
+            0.0045123896561563015,
+            0.012437249533832073,
+            -0.0019404053455218673,
+            0.013035791926085949,
+            0.01848486252129078,
+            -0.016588181257247925,
+            -0.004115952178835869,
+            0.015367777086794376,
+            0.018671421334147453,
+            0.009825427085161209,
+            -0.004049879498779774,
+            0.025682920590043068,
+            0.0024952231906354427,
+            -0.020676927641034126,
+            0.006918220315128565,
+            0.0007384616765193641,
+            0.02596275880932808,
+            -0.00698429299518466,
+            -0.016463808715343475,
+            -0.01901344582438469,
+            0.00949894916266203,
+            -0.005600648932158947,
+            -0.005884373560547829,
+            -0.009677735157310963,
+            -0.004617328755557537,
+            0.026398062705993652,
+            0.008527289144694805,
+            0.0006412956863641739,
+            -0.015911906957626343,
+            0.008558382280170918,
+            -0.01862478069961071,
+            -0.013401136733591557,
+            -0.004003239795565605,
+            0.015771986916661263,
+            -0.02571401372551918,
+            -0.001339919283054769,
+            -0.008449556306004524,
+            -0.00013530367868952453,
+            0.012180731631815434,
+            0.021951746195554733,
+            0.0015624294755980372,
+            0.001924858777783811,
+            -0.006051499396562576,
+            -0.024081625044345856,
+            -0.015251177363097668,
+            -0.013727614656090736,
+            -0.007761621382087469,
+            -0.030673367902636528,
+            -0.010035306215286255,
+            -0.0006534414133056998,
+            0.00013821866014041007,
+            -0.0012388667091727257,
+            0.037591587752103806,
+            -0.006583969108760357,
+            -0.00244469684548676,
+            -0.006342997308820486,
+            0.009079192765057087,
+            -0.010664941743016243,
+            0.006261378061026335,
+            -0.006533442530781031,
+            -0.0002924696891568601,
+            0.004045992624014616,
+            0.002048259600996971,
+            0.02807709202170372,
+            0.002182348631322384,
+            -0.006591742392629385,
+            0.008954820223152637,
+            -0.01284923404455185,
+            -0.021967291831970215,
+            0.01693020574748516,
+            -0.024003891274333,
+            -0.013813120312988758,
+            -0.026600167155265808,
+            -0.016728101298213005,
+            -0.017256684601306915,
+            -0.01680583320558071,
+            -0.039954666048288345,
+            0.0023222677409648895,
+            0.02728421613574028,
+            -0.006471256725490093,
+            0.017132312059402466,
+            -0.0023747372906655073,
+            -0.009436762891709805,
+            -0.0010163565166294575,
+            -0.02007061243057251,
+            0.006300244480371475,
+            -0.0016595955239608884,
+            -0.005763887893408537,
+            0.013743161223828793,
+            -0.006560649257153273,
+            0.017396602779626846,
+            0.002920810366049409,
+            -0.019619761034846306,
+            -0.0033833205234259367,
+            0.01876470074057579,
+            0.023397576063871384,
+            -0.0033638873137533665,
+            -0.000545587157830596,
+            0.003752551507204771,
+            -0.010120811872184277,
+            0.01809619925916195,
+            -0.015585429035127163,
+            0.005002106539905071,
+            0.0012446966720744967,
+            0.014085185714066029,
+            0.005375223699957132,
+            0.0009070447413250804,
+            0.006992066279053688,
+            0.0006242916570045054,
+            0.006611175369471312,
+            -0.007411823607981205,
+            0.00134574924595654,
+            0.001861700788140297,
+            -0.0183449424803257,
+            0.03395369276404381,
+            0.013097978197038174,
+            0.012934739701449871,
+            -0.03684535250067711,
+            0.017660895362496376,
+            0.0016440489562228322,
+            -0.008449556306004524,
+            -0.007660568691790104,
+            0.030129237100481987,
+            0.011978626251220703,
+            0.00365538545884192,
+            0.009017006494104862,
+            -0.010594981722533703,
+            0.012157411314547062,
+            -0.011185751296579838,
+            -0.004586235620081425,
+            -0.005421863403171301,
+            -0.011193525046110153,
+            0.01823611743748188,
+            0.009825427085161209,
+            0.0022425916977226734,
+            -0.0046561951749026775,
+            -0.013393362984061241,
+            0.029989318922162056,
+            0.008022026158869267,
+            -0.004784454591572285,
+            0.008651661686599255,
+            -0.014310610480606556,
+            0.0005718219908885658,
+            0.000107489904621616,
+            0.007567289285361767,
+            -0.016728101298213005,
+            -5.696357402484864e-05,
+            -0.028139278292655945,
+            -0.017552068457007408,
+            -0.02335093729197979,
+            0.0002368421555729583,
+            0.0014438870130106807,
+            0.0177230816334486,
+            -0.0011941703269258142,
+            0.004527936223894358,
+            -7.360325253102928e-05,
+            0.007128098979592323,
+            0.004527936223894358,
+            0.014792554080486298,
+            0.012266237288713455,
+            0.02638251520693302,
+            -0.00570558849722147,
+            -0.028792232275009155,
+            0.0041975718922913074,
+            0.007909313775599003,
+            0.008962593041360378,
+            0.027455229312181473,
+            0.003634008811786771,
+            -0.0068949004635214806,
+            -0.00751287629827857,
+            0.008441783487796783,
+            0.025247616693377495,
+            -0.003908017184585333,
+            -0.018002919852733612,
+            0.018313849344849586,
+            0.020630288869142532,
+            0.0033600006718188524,
+            -0.0040693124756217,
+            0.01703903265297413,
+            0.011442270129919052,
+            -0.019122270867228508,
+            -0.011512229219079018,
+            0.002283401321619749,
+            0.0055889892391860485,
+            0.0028663973789662123,
+            -0.016494901850819588,
+            -0.007645022124052048,
+            -0.03488648682832718,
+            0.007131985388696194,
+            -0.0158652663230896,
+            0.0071436455473303795,
+            0.010369556955993176,
+            -0.03466883301734924,
+            -0.008099758997559547,
+            0.013214577920734882,
+            0.03211919590830803,
+            0.006758867762982845,
+            0.0075439694337546825,
+            -0.0043413774110376835,
+            -0.0014506885781884193,
+            -0.014559355564415455,
+            0.014707047492265701,
+            -0.0328032448887825,
+            0.02229377068579197,
+            0.01441943645477295,
+            0.004780567716807127,
+            -0.0018140894826501608,
+            -0.010276277549564838,
+            0.00558121595531702,
+            -0.0036301221698522568,
+            0.0033774906769394875,
+            -0.03448227420449257,
+            0.0192932840436697,
+            -0.008760487660765648,
+            0.022931179031729698,
+            0.02361522801220417,
+            0.02611822448670864,
+            -0.006335224024951458,
+            -0.0013700408162549138,
+            0.005464616697281599,
+            0.006486803293228149,
+            0.01694575324654579,
+            -0.009436762891709805,
+            -0.0016382189933210611,
+            -0.02073911391198635,
+            0.012701541185379028,
+            -0.023630773648619652,
+            -0.01063384860754013,
+            0.00269149849191308,
+            -0.013603242114186287,
+            -0.025371989235281944,
+            -0.010144132189452648,
+            -0.004100405611097813,
+            -0.0030587860383093357,
+            -0.0016440489562228322,
+            0.008099758997559547,
+            -0.01742769591510296,
+            -0.02465684711933136,
+            -0.014590448699891567,
+            -0.010742674581706524,
+            0.022651340812444687,
+            0.01075822114944458,
+            -0.016572635620832443,
+            -0.0039993529208004475,
+            -0.005682268645614386,
+            0.0071436455473303795,
+            0.007959839887917042,
+            -0.01416291855275631,
+            0.05506592243909836,
+            0.0006393523653969169,
+            0.0027187049854546785,
+            0.0018743324326351285,
+            0.008216357789933681,
+            0.0012913363752886653,
+            0.003585425904020667,
+            -0.002267854753881693,
+            0.004477410111576319,
+            -0.034575555473566055,
+            -0.018173931166529655,
+            0.0033153044059872627,
+            -0.011589962057769299,
+            0.00945230945944786,
+            -0.01915336400270462,
+            0.011589962057769299,
+            -0.01719449833035469,
+            -0.021718546748161316,
+            0.006109798792749643,
+            -0.0027886645402759314,
+            -0.002106559230014682,
+            -0.014652634970843792,
+            -0.017132312059402466,
+            0.007567289285361767,
+            0.02453247457742691,
+            -0.0024738467764109373,
+            0.01547660306096077,
+            -0.02022607810795307,
+            -0.005254738032817841,
+            -0.012810367159545422,
+            0.01089036650955677,
+            -0.010462836362421513,
+            -0.009017006494104862,
+            -0.009017006494104862,
+            -0.02164081484079361,
+            0.017816361039876938,
+            -0.002372794086113572,
+            0.00800647959113121,
+            -0.0206147413700819,
+            0.011224618181586266,
+            -0.010618302039802074,
+            0.016992392018437386,
+            0.000213036488275975,
+            -0.01915336400270462,
+            -0.014116278849542141,
+            0.01680583320558071,
+            -0.023972798138856888,
+            -0.0012835630914196372,
+            0.027346402406692505,
+            0.005934900138527155,
+            0.025807293131947517,
+            -0.006832714192569256,
+            -0.024983325973153114,
+            -0.01311352476477623,
+            0.016494901850819588,
+            -0.00957668200135231,
+            -0.00015947372594382614,
+            -0.00701927300542593,
+            0.007334090769290924,
+            -0.01482364721596241,
+            0.028683407232165337,
+            -0.01822056993842125,
+            0.007998705841600895,
+            0.0015604861546307802,
+            -0.008247450925409794,
+            0.014069639146327972,
+            -0.0048544141463935375,
+            -0.007493442855775356,
+            -0.0077072083950042725,
+            -0.004489069804549217,
+            -0.010664941743016243,
+            -0.0034571667201817036,
+            -0.029973771423101425,
+            0.0005674495478160679,
+            -0.0005266398075036705,
+            -0.012382837012410164,
+            -0.011753200553357601,
+            -0.004469636827707291,
+            -0.004671741742640734,
+            0.0019238870590925217,
+            0.011006966233253479,
+            0.00012704456457868218,
+            -0.01161328237503767,
+            -0.008379597216844559,
+            0.011776520870625973,
+            -0.00718639837577939,
+            0.005690041929483414,
+            -0.02557409554719925,
+            0.01194753311574459,
+            -0.013758707791566849,
+            0.0057600014843046665,
+            -0.009055872447788715,
+            0.019977333024144173,
+            0.03120194934308529,
+            0.0044618635438382626,
+            0.004166478756815195,
+            -0.007442916743457317,
+            0.0029791099950671196,
+            0.002434980357065797,
+            0.016215063631534576,
+            -0.008091985248029232,
+            0.017225591465830803,
+            -0.012748180888593197,
+            0.007769394665956497,
+            -0.012040812522172928,
+            0.01717895083129406,
+            0.006245831493288279,
+            0.0037953045684844255,
+            -0.009584455750882626,
+            -0.0007778139552101493,
+            -0.02897879108786583,
+            0.010672714561223984,
+            -0.01770753413438797,
+            -0.0009969233069568872,
+            0.016215063631534576,
+            0.009405669756233692,
+            -0.013657654635608196,
+            0.012794820591807365,
+            0.03134186938405037,
+            0.024827860295772552,
+            -0.004115952178835869,
+            -0.0012709314469248056,
+            0.01640162244439125,
+            -0.0039993529208004475,
+            0.004982673097401857,
+            -0.009211338125169277,
+            -0.03373603895306587,
+            -0.019184457138180733,
+            -0.018780246376991272,
+            0.0040071262046694756,
+            0.010058625601232052,
+            -0.00020538465469144285,
+            0.0110924718901515,
+            0.010781540535390377,
+            0.019650854170322418,
+            -0.019308829680085182,
+            -0.0034299602266401052,
+            -0.008519516326487064,
+            -0.005095385946333408,
+            0.028154823929071426,
+            0.008542835712432861,
+            0.021438708528876305,
+            -0.011038059368729591,
+            0.006179758347570896,
+            -0.020925672724843025,
+            -0.01371206808835268,
+            -0.01252275612205267,
+            0.021423162892460823,
+            0.0027944946195930243,
+            -0.0013661541743203998,
+            0.02572956122457981,
+            0.005359677132219076,
+            -0.015181218273937702,
+            0.006486803293228149,
+            -0.006191418506205082,
+            0.010408422909677029,
+            0.006261378061026335,
+            0.008053119294345379,
+            -0.025807293131947517,
+            -0.02507660537958145,
+            -0.01585749350488186,
+            0.010369556955993176,
+            0.013020245358347893,
+            0.01101473905146122,
+            -0.014139598235487938,
+            -0.018593687564134598,
+            0.0056395153515040874,
+            0.01331563014537096,
+            -0.008278544060885906,
+            -0.010968099348247051,
+            -0.0036126323975622654,
+            0.00970105454325676,
+            0.020490368828177452,
+            5.5475720728281885e-05,
+            0.007501216139644384,
+            0.01075822114944458,
+            0.005658948794007301,
+            -0.0043413774110376835,
+            0.008123078383505344,
+            -0.010983645915985107,
+            -0.01694575324654579,
+            0.01548437587916851,
+            -0.010385103523731232,
+            0.009156924672424793,
+            0.010268504731357098,
+            -0.010237411595880985,
+            -0.001987044932320714,
+            -0.007046479266136885,
+            0.01280259434133768,
+            0.025760654360055923,
+            -0.004353037569671869,
+            -0.003435790305957198,
+            -0.017210043966770172,
+            0.002506883116438985,
+            0.0023008910939097404,
+            0.018453769385814667,
+            0.0058183008804917336,
+            0.008822673931717873,
+            -0.010719354264438152,
+            -0.0022697981912642717,
+            0.025543002411723137,
+            0.010618302039802074,
+            -0.006704454775899649,
+            -0.00024813771597109735
+          ],
+          "title": "Capacity Factor",
+          "keyphrases": [
+            "capacity factors",
+            "downstream task fine-tuning",
+            "expert choice",
+            "gating computational footprint",
+            "fine-tuning performance"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "9c6edae0-4b6f-4fa2-8839-b989005b11c8",
+      "type": "child",
+      "source": {
+        "id": "b570d71e-4f83-486e-b70f-b0c6217db455",
+        "properties": {
+          "page_content": "Mixture-Of-Experts With Expert Choice Routing\n\nYanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon Google, Mountain View, CA, USA {yanqiz, taole, hanxiaol, dunan, huangyp, vzhao, adai, zhifengc, qvl, jlaudon}@google.com\n\nAbstract\n\nSparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choice method. Instead of letting tokens select the top-k experts, we have experts selecting the top-k tokens. As a result, each token can be routed to a variable number of experts and each expert can have a fixed bucket size. We systematically study pre-training speedups using the same computational resources of the Switch Transformer top-1 and GShard top-2 gating of prior work and find that our method improves training convergence time by more than 2×. For the same computational cost, our method demonstrates higher performance in fine-tuning 11 selected tasks in the GLUE and SuperGLUE benchmarks. For a smaller activation cost, our method outperforms the T5 dense model in 7 out of the 11 tasks.\n\n1 Introduction\n\nScaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27]. The final model quality has been found to have a power-law relationship with the amount of data, model size, and compute time [16, 20]. However, training efficiency, which is defined as the total amount of computation used to achieve superior model quality than the state of the art system [21], should receive greater attention as we increase our efforts towards green AI [29].\n\nSparsely gated mixture-of-experts [31] (MoE) provides an effective way to scale model capacity given a fixed computational cost, and has recently played an important role in increasing the training efficiency of large-scale language models [10, 21]. MoE operate by adopting a number of experts, each as a sub-network, and by activating only one or a few experts for each input token. A gating network must be chosen and optimized in order to route each token to the most suited expert(s). For example, recent work has implemented sparse routing via k-means clustering [12], linear assignment to maximize token-expert affinities [22], or hashing [8, 28]. Many of the prior work use a routing strategy concerning the token choice, where each token selects the best one or two experts.\n\nWe argue that the independent token choice of prior work often leads to an imbalanced load of experts, which causes training inefficiency and sub-optimal training of the model. In order to mitigate this\n\nissue, previous sparsely gated networks introduce additional auxiliary losses as regularization to prevent too many tokens being routed to a single expert, but the effectiveness is still limited. Recent approaches [8, 22, 28] explore alternative strategies for routing, but they focus on pre-training only and do not demonstrate performance gain on downstream tasks. Moreover, none of the previous methods consider allocating a variable number of experts to each token based on importance, which can be beneficial.\n\nWe propose a very simple yet effective routing method we are calling expert choice. Unlike conventional MoE where tokens select one or two top-scoring experts, our method lets each expert pick the top-k tokens. Our method guarantees perfect load balancing, allows a variable number of experts for each token, and achieves substantial gains in training efficiency and downstream performance as demonstrated in our experiments. Our major contributions include:\n\nWe identify common pitfalls in conventional MoE such as load imbalance as described in Section 3.1. We then propose a heterogeneous, expert choice method to provide a fluid allocation of model parameters based on a learnt token-to-expert importance. This method intrinsically guarantees load balance without imposing an auxiliary loss.\n\nWe show our method provides over 2× faster training convergence in a 8B/64E (8 billion activated parameters, 64 experts) model, compared to the top-1 and top-2 gating counterparts in Switch Transformer [10] and GShard [21].\n\nWe show our method demonstrates strong scaling when increasing the number of experts from 16 to 128, evaluated in training perplexity.\n\nWe show our method demonstrates strong performance on downstream tasks selected from GLUE and SuperGLUE at all the evaluated scales. More specifically, our 8B/64E model outperforms a T5 11B dense model in 7 out of 11 tasks evaluated.\n\n2 Related Work\n\nScaling: Various approaches have been proposed to scale up neural network capacity to improve performance. Recent works have successfully scaled models to billions of parameters via various forms of model parallelism [2, 21, 26, 27, 33]. Model parallelism [30] splits weights and tensors across multiple cores while pipeline parallelism [18, 24] splits different layers across devices with micro-batches pipelined to the different layers. To enable continued scaling of neural networks, improving model training and serving efficiency has become a critical research area.\n\nConditional Computation: Computation decisions can be made dynamically based on the input [23, 25]. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation, by activating certain parameters and computation on demand, on a per-example or per-token basis [3]. Conditional convolution layers [1] with task-specific gating has been used to combat catastrophic forgetting when a sequence of learning problems are optimized. The gating decisions may be binary or sparse and continuous, stochastic or deterministic.\n\nMixture of Experts: Sparsely-gated MoE [31] is the first model to demonstrate massive improvements in model capacity, training time, or model quality with gating. Switch Transformer [10] simplifies the gating by selecting only the top expert per token using a softmax over the hidden state and demonstrates better scaling than previous work. All the prior work requires an auxiliary loss to explicitly encourage balancing. This loss term has to be carefully weighted to not overwhelm the primary loss. However, auxiliary loss does not guarantee balancing and a hard capacity factor has to be imposed. As a result, many tokens can still be unprocessed by the MoE layer. Hard MoE [12] with a single decoding layer can be efficiently trained to good effect on large scale hashtag prediction tasks.\n\nBase Layers [22] formulate a linear assignment that maximizes token-expert affinities while ensuring each expert receives an equal number of tokens. Hash layers [8, 28] devise hashing techniques on input tokens. However, the evaluations are limited to pre-training perplexity. THOR [? ] randomly activates experts during training and inference and is trained with a consistency regularization loss.\n\nTHOR has demonstrated strong performance on translation tasks. Different from these prior works, our method is a learnt method that enables heterogeneous MoE and effectively improves downstream fine-tuning performance.\n\n3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n5 Conclusion\n\nWe propose a new routing method for sparsely activated mixture-of-experts (MoE) models. This method addresses load imbalance and under-utilization of experts in conventional MoE methods, and enables selecting different numbers of experts for each token. Our model demonstrates more than 2x training efficiency improvements when compared to the state-of-the-art GShard and Switch Transformer models, and also achieves strong gains when finetuning on 11 datasets in the GLUE and SuperGLUE benchmark.\n\n6 Limitations\n\nThe expert choice method might not immediately apply to auto-regressive text generation as our current implementation takes in the past and future tokens to perform the top-k selection. One possible solution is to collect a large batch of input sequences, dispatch tokens of the same sequence into separate groups, and perform expert choice routing for each group. Another scenario where the expert choice method does not immediately apply is when the batch size becomes very small during serving or inference. A global top-k can be selected instead and we can cap the number of times each expert or token gets selected. We leave these possible improvements for future work.\n\nAnother long-standing issue with MoE has been the large memory footprint. Even though computational cost can be reduced using sparsely gated networks, the total number of parameters increases linearly or sub-linearly with the number of experts. Increasing the number of experts requires reservation of a large number of hardware devices. Therefore, dynamic (used) power is saved while static (reserved) power is not. Power saving techniques such as the ability to put hardware devices into low power states while not in use [17] can help with reducing the reserved power requirements.\n\nReferences\n\n[1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pages 3930–3939. Computer Vision Foundation / IEEE, 2020.\n\n[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. In Advances in Neural Information Processing Systems.\n\n[3] Kyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning, 2014.\n\n[4] Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems, 2021.\n\n[5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.\n\nTransformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.\n\n[6] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 933–941. JMLR.org, 2017.\n\n[7] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixtureof-experts, 2021.\n\n[8] Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan.\n\nTricks for training sparse translation models, 2021.\n\n[9] Richard L Dykstra. An iterative procedure for obtaining i-projections onto the intersection of convex sets. The annals of Probability, pages 975–984, 1985.\n\n[10] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.\n\n[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. NAS-FPN: learning scalable feature pyramid architecture for object detection. In CVPR, pages 7036–7045. Computer Vision Foundation / IEEE, 2019.\n\n[12] Sam Gross, Marc'Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision, 2017.\n\n[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.\n\n[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 630–645, Cham, 2016. Springer International Publishing.\n\n[15] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2016.\n\n[16] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.\n\n[17] Ping Huang, Zuocheng Xing, Tianran Wang, Qiang Wei, Hongyan Wang, and Guitao Fu. A brief survey on power gating design. In 2010 10th IEEE International Conference on Solid-State and Integrated Circuit Technology, pages 788–790, 2010.\n\n[18] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 103–112, 2019.\n\n[19] Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David A. Patterson. A domain-specific supercomputer for training deep neural networks. Commun. ACM, 63(7):67–78, 2020.\n\n[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.\n\n[21] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.\n\n[22] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6265–6274. PMLR, 18–24 Jul 2021.\n\n[24] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. New York, NY, USA, 2019. Association for Computing Machinery.\n\n[25] Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, Cédric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. In ICLR. OpenReview.net, 2021.\n\n[26] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.\n\n[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.\n\n[28] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models, 2021.\n\n[30] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 10435–10444, Red Hook, NY, USA, 2018. Curran Associates Inc.\n\n[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E.\n\nHinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In ICLR (Poster). OpenReview.net, 2017.\n\n[32] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 10–15 Jul 2018. [23] Min Lin, Jie Fu, and Yoshua Bengio. Conditional computation for continual learning, 2019.\n\n[29] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.\n\n[33] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.\n\n[34] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc.\n\n[35] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.\n\nGLUE: A multi-task benchmark and analysis platform for natural language understanding.\n\nIn Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, November 2018. Association for Computational Linguistics.\n\n[36] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake A. Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen. GSPMD: general and scalable parallelization for ML computation graphs. CoRR, abs/2105.04663, 2021.\n\n7 Checklist\n\n(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Yes (b) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes (c) Did you discuss any potential negative societal impacts of your work? N/A. Not any. (d) Did you describe the limitations of your work? Yes (a) Did you include the code, data, and instructions needed to reproduce the main experimental results? Yes. We include details in the experiment setup to help reproduce the main results. (b) Did you specify all the training details? Yes (c) Did you report error bars? Yes (d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? Yes (a) If your work uses existing assets, did you cite the creators? Yes (b) Did you mention the license of the assets? No. The used dataset is not released yet. (c) Did you include any new assets either in the supplemental material or as a URL? No. The dataset is not released yet.\n\n(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? No. Not using persons' data. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes. The dataset does not contain any personally identifiable information or offensive content.\n\nA Comparison On Fine-Tuning With A Dense Model\n\nOur 8B MoE model achieves stronger pre-training perplexity than its dense counterpart. However, a better perplexity does not always directly translate to downstream performance as demonstrated in Section 4.4. To this end, we compare fine-tuning performance of the 8B dense model and MoE model in Table 1. As shown in the table, our MoE model using expert choice routing consistently outperforms the dense model across the 11 tasks in GLUE and SuperGLUE.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 1: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nB Capacity Factor\n\nWe evaluate the downstream task fine-tuning performance by varying the capacity factors. Note that a capacity factor of n indicates on average how many experts each token can be received. EC-CF2 is our baseline expert choice, which matches GShard top-2 gating computational footprint. EC-CF1, however, matches Switch Transformer top-1 gating computational footprint. EC-CF0.5 further verifies that an aggressively lowered capacity factor can provide strong enough performance, that almost matches the top-2 gating baseline.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Top-2 78.1 87.0 88.3 85.0 82.6 90.1 90.7 81.6 94.7 68.2 67.2 83.0±0.3 EC-CAP2 78.2 88.0 88.5 85.7 83.0 90.8 91.1 80.0 95.4 70.4 64.1 83.2±0.4 EC-CAP3 78.5 91.7 89.3 86.3 83.5 90.9 91.1 81.8 94.9 70.0 65.6 84.0±0.4 EC-CF2 79.1 89.6 89.3 86.8 84.3 91.3 91.2 81.1 95.2 68.1 68.0 84.0±0.2 EC-CF1 77.4 90.6 88.0 85.5 83.6 90.3 91.2 79.8 95.3 66.5 64.9 83.0±0.2 EC-CF0.5 77.4 89.6 86.3 85.2 82.7 91.7 91.0 79.6 94.9 67.3 63.5 83.0 ±0.05 Hash Layers 76.1 85.2 86.7 83.4 82.5 90.0 90.3 75.7 94.0 67.4 63.3 81.3±1.0\n\nTable 2: Comparison between different routing methods in fine-tuning of 100M/64E models. We perform 3 independent fine-tuning runs for each method and report the average results. This gives more accurate difference between the variants of expert choice method, since they achieve close fine-tuning results. We do not report averaged results in other experiments.\n\nC Capped Expert Choice\n\nAs described in Section 4.5, the maximum number of experts each token is assigned can be capped by an entropy-regularized linear programming. Figure 1 compares the validation perplexity when training the 100M/64E models using the base expert choice method (EC-BASE), expert choice capped by two experts per token (EC-CAP2), expert choice capped by three experts per token (EC-CAP3), and GShard top-2 gating.\n\nAs shown in the figure, restricting the number of experts to 2 degrades the perplexity compared to the base expert choice method. This suggests that a more flexible allocation of experts (e.g. more than 2 experts for a token) can enhance model expressiveness. On the other hand, our EC-CAP2 and EC-CAP3 methods still outperform the top-2 gating method by a clear margin. We believe this confirms the effectiveness of a load balanced training, provided by our method. Finally, EC-CAP3 obtains comparable perplexity to EC-BASE. As indicated by Figure 3, only a little fraction of tokens use more than 3 experts therefore we see little or no difference between EC-BASE and EC-CAP3 variants. We present the fine-tuning results of these methods in Table 2.\n\nD Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [? ]. We use mod x to map a token ID to an expert ID. This in some way ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 2. Hashing based routing performs much worse than expert choice in terms of average scores and variance.\n\nE Fine-Tuning Details\n\nWe did a hyperparameter search for both baseline models and expert choice method. For fine-tuning of the 8B dense model, we use a constant learning rate of 0.0001 and a dropout rate of 0.1. We freeze the attention layer and feed-forward layer while leaving the embedding and layer normalization trainable. This setting has been found optimal for the 8B dense model. For MoE 8B/64E models including GShard top-2 gating and expert choice, we found continuing the learning rate from the pre-trained model while using a square root learning rate decay works better. In addition, we do not apply parameter freezing for fine-tuning MoE models. For models with 100M expert size, we use a constant learning rate of 0.0001 and no dropout is used.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/2202.09368v2.md"
+          },
+          "headlines": {
+            "1 Introduction": [],
+            "2 Related Work": [],
+            "3 Method": [
+              "3.1 Pitfalls Of Token-Choice Routing",
+              "3.2 Heterogeneous Moe Via Expert Choice",
+              "3.3 Expert Choice With Additional Constraint",
+              "3.4 Model Architecture"
+            ],
+            "4 Experiments": [
+              "4.1 Setup",
+              "4.2 Training Efficiency",
+              "4.3 Scaling The Number Of Experts",
+              "4.4 Fine-Tuning On Glue And Superglue",
+              "4.5 Heterogeneity Matters",
+              "4.6 Comparison With Hash Layer",
+              "4.7 Ablation"
+            ],
+            "5 Conclusion": [],
+            "6 Limitations": [],
+            "A Comparison On Fine-Tuning With A Dense Model": [],
+            "B Capacity Factor": [],
+            "C Capped Expert Choice": [],
+            "D Comparison With Hash Layer": [],
+            "E Fine-Tuning Details": []
+          },
+          "summary": "The paper introduces a new routing method for sparsely-activated Mixture-of-Experts (MoE) models called 'expert choice,' which addresses load imbalance and under-utilization issues in conventional MoE methods. Unlike traditional token-choice routing, expert choice allows experts to select the top-k tokens, ensuring perfect load balancing and enabling variable expert allocation per token. This method improves training efficiency, achieving over 2x faster convergence compared to existing models like Switch Transformer and GShard. It also demonstrates superior performance on downstream tasks in the GLUE and SuperGLUE benchmarks. The study highlights the benefits of flexible expert allocation and shows that expert choice outperforms hashing-based routing methods. The paper also discusses potential limitations and future improvements for the expert choice method.",
+          "summary_embedding": [
+            -0.009726986289024353,
+            -0.0337262786924839,
+            -0.023302342742681503,
+            -0.019878201186656952,
+            -0.025953780859708786,
+            0.04560471698641777,
+            -0.006874797400087118,
+            0.039665497839450836,
+            -0.006973279174417257,
+            0.034544438123703,
+            -0.025787118822336197,
+            -0.00363057479262352,
+            -0.01052999310195446,
+            0.0054657477885484695,
+            -0.024499278515577316,
+            0.036786794662475586,
+            0.010007280856370926,
+            0.003272630739957094,
+            0.006102093029767275,
+            -0.04048365354537964,
+            -0.018666116520762444,
+            -0.013522328808903694,
+            -0.03230207785964012,
+            0.008969432674348354,
+            0.02484775334596634,
+            0.0016713523073121905,
+            0.0026230288203805685,
+            0.003624893259257078,
+            -0.05311964824795723,
+            0.04339266195893288,
+            0.018681267276406288,
+            0.023711422458291054,
+            -0.016332851722836494,
+            -0.012832955457270145,
+            -0.039120059460401535,
+            -0.0197418425232172,
+            0.030786970630288124,
+            0.039968520402908325,
+            -0.019287310540676117,
+            0.03357476741075516,
+            0.018060073256492615,
+            0.02454473078250885,
+            -0.021272100508213043,
+            -0.029817303642630577,
+            -0.0061892117373645306,
+            -0.007840678095817566,
+            -0.008355814032256603,
+            0.01242387667298317,
+            0.016984347254037857,
+            -0.013007191941142082,
+            0.0212569497525692,
+            -0.01371171697974205,
+            0.03384748846292496,
+            0.008916404098272324,
+            -0.042332086712121964,
+            0.03984731063246727,
+            -0.008007340133190155,
+            0.012105703353881836,
+            0.03281721472740173,
+            0.010333028621971607,
+            -0.005287723150104284,
+            -0.021544819697737694,
+            -0.03378688171505928,
+            0.0010529992869123816,
+            0.00014014738553669304,
+            0.005950582213699818,
+            -0.003069985192269087,
+            -0.009742137044668198,
+            0.03227177634835243,
+            -0.003939277958124876,
+            -0.0740584209561348,
+            0.050089433789253235,
+            -0.026590125635266304,
+            -0.007628562860190868,
+            -0.028938541188836098,
+            -0.01736312545835972,
+            0.037362534552812576,
+            0.005662711802870035,
+            0.0098709212616086,
+            0.0106284748762846,
+            -0.0007892759749665856,
+            0.028923390433192253,
+            0.00910579226911068,
+            -0.024923507124185562,
+            0.04478655755519867,
+            -0.04790768027305603,
+            0.01801462098956108,
+            -0.04336235672235489,
+            0.022756904363632202,
+            1.6393929399782792e-05,
+            -0.020726662129163742,
+            -0.0013304532039910555,
+            -0.016666175797581673,
+            0.0218023881316185,
+            -0.01730252057313919,
+            0.026802239939570427,
+            0.013825350441038609,
+            -0.015802564099431038,
+            -0.029817303642630577,
+            0.025378039106726646,
+            0.003367325058206916,
+            0.010499690659344196,
+            -0.02075696364045143,
+            0.014332910999655724,
+            0.04393810033798218,
+            -0.02593863010406494,
+            -0.02271145209670067,
+            0.027681002393364906,
+            0.014408666640520096,
+            0.0633314698934555,
+            0.036544378846883774,
+            0.006624804809689522,
+            0.022362977266311646,
+            0.020575150847434998,
+            -0.028968842700123787,
+            -0.01963578537106514,
+            -0.046362269669771194,
+            -0.04757435619831085,
+            0.019878201186656952,
+            -0.04566532000899315,
+            0.0050869714468717575,
+            0.038544319570064545,
+            -0.013704141601920128,
+            -0.014446543529629707,
+            -0.04021093621850014,
+            0.017847958952188492,
+            0.0003908502112608403,
+            0.011976920068264008,
+            -0.0022082682698965073,
+            0.00033285003155469894,
+            -0.011825408786535263,
+            0.008878526277840137,
+            -0.03778676688671112,
+            -0.025196226313710213,
+            0.025181075558066368,
+            0.022862963378429413,
+            0.024120500311255455,
+            0.028802180662751198,
+            -0.03077181987464428,
+            -0.009984553791582584,
+            0.021378157660365105,
+            -0.04242299124598503,
+            -0.006556625012308359,
+            0.011332998983561993,
+            0.0026703758630901575,
+            0.003872992005199194,
+            0.032786913216114044,
+            0.052998438477516174,
+            -0.03172633796930313,
+            0.02065090648829937,
+            -0.010431510396301746,
+            0.04957429692149162,
+            0.00024691506405360997,
+            0.028968842700123787,
+            -0.019514575600624084,
+            -0.0134389977902174,
+            -0.029529431834816933,
+            -0.009795165620744228,
+            -0.01356020662933588,
+            -0.008484598249197006,
+            -0.002998017705976963,
+            -0.009575475007295609,
+            0.0040453351102769375,
+            0.0331808403134346,
+            0.00273855566047132,
+            0.030680913478136063,
+            0.014325335621833801,
+            0.0051475754007697105,
+            -0.08745196461677551,
+            0.026059838011860847,
+            -0.0033616432920098305,
+            0.005859675817191601,
+            0.0062460280023515224,
+            0.03013547509908676,
+            -0.0312718041241169,
+            0.020938776433467865,
+            0.026620427146553993,
+            0.0011145505122840405,
+            0.016969196498394012,
+            0.036847397685050964,
+            -0.04181694984436035,
+            0.009643655270338058,
+            0.012590537779033184,
+            -0.0033881578128784895,
+            0.04584713280200958,
+            0.011628445237874985,
+            0.019514575600624084,
+            0.018529755994677544,
+            0.003569970605894923,
+            0.00297907879576087,
+            0.006533898413181305,
+            0.03724132478237152,
+            0.003462019143626094,
+            -0.009833043441176414,
+            0.014976831153035164,
+            -0.010757259093225002,
+            -0.011219366453588009,
+            -0.037998881191015244,
+            0.014454119838774204,
+            0.008742166683077812,
+            -0.017454031854867935,
+            -0.02959003672003746,
+            0.0026949963066726923,
+            -0.031908150762319565,
+            0.026559824123978615,
+            -0.002445003716275096,
+            0.005738467443734407,
+            -0.010454237461090088,
+            -0.006757376715540886,
+            -0.035423196852207184,
+            0.028453707695007324,
+            -0.02095392718911171,
+            0.0053142374381423,
+            0.0007258309051394463,
+            -0.029438525438308716,
+            0.0060073984786868095,
+            0.017863109707832336,
+            -0.0025965142995119095,
+            0.0025302283465862274,
+            -0.02349930815398693,
+            0.024468975141644478,
+            0.025635607540607452,
+            0.004518806003034115,
+            -0.014037464745342731,
+            -0.013113250024616718,
+            -0.003605954349040985,
+            -0.009742137044668198,
+            -0.016196493059396744,
+            0.024468975141644478,
+            -0.014810169115662575,
+            0.00308892410248518,
+            0.021847840398550034,
+            0.014264730736613274,
+            -0.007045246660709381,
+            -0.004007457755506039,
+            -0.04066546633839607,
+            -0.01218903437256813,
+            -0.04760465770959854,
+            -0.011628445237874985,
+            -0.0034601252991706133,
+            -0.028696123510599136,
+            0.0061513339169323444,
+            -0.04481685906648636,
+            -0.01235569640994072,
+            0.02309022843837738,
+            0.018135828897356987,
+            0.0019488062243908644,
+            -0.009173971600830555,
+            -0.022605394944548607,
+            8.486965816700831e-05,
+            -0.01763584464788437,
+            -0.017044952139258385,
+            0.015393485315144062,
+            0.031059689819812775,
+            -0.03101423755288124,
+            0.04024123772978783,
+            0.003893824527040124,
+            -0.08066429197788239,
+            0.010393633507192135,
+            -0.026362858712673187,
+            0.07011914253234863,
+            -0.0025453795678913593,
+            0.014075342565774918,
+            0.06781618297100067,
+            -0.012431452050805092,
+            -0.02007516659796238,
+            0.03142331540584564,
+            -0.008249756880104542,
+            -1.1311515663692262e-05,
+            -0.02884763479232788,
+            -0.02739313244819641,
+            -0.053271155804395676,
+            -0.009704259224236012,
+            -0.010787560604512691,
+            0.03360506892204285,
+            -0.024317465722560883,
+            -0.0007656024536117911,
+            0.012620840221643448,
+            0.02983245439827442,
+            0.016635872423648834,
+            -0.0015681355725973845,
+            0.008272483013570309,
+            0.02881733328104019,
+            0.0274234339594841,
+            -0.0036154238041490316,
+            0.00814369972795248,
+            -0.034211114048957825,
+            -0.0068066176027059555,
+            -0.0493621826171875,
+            0.011302697472274303,
+            -0.02596893161535263,
+            -0.02596893161535263,
+            0.007643714081496,
+            -0.011476934887468815,
+            0.03748374432325363,
+            -0.024696242064237595,
+            -0.007503566797822714,
+            0.005037730094045401,
+            0.012855681590735912,
+            -0.017757052555680275,
+            -0.03612014651298523,
+            -0.0028502948116511106,
+            0.0026381798088550568,
+            0.007560383062809706,
+            -0.053240854293107986,
+            0.004499867558479309,
+            -0.018469152972102165,
+            0.0279688723385334,
+            -0.02139330841600895,
+            -0.013688990846276283,
+            0.042029064148664474,
+            0.011363301426172256,
+            0.027499189600348473,
+            -0.026468917727470398,
+            0.0017849853029474616,
+            0.012825379148125648,
+            -0.0020056227222085,
+            0.032514192163944244,
+            -0.030574856325984,
+            -0.03257479891180992,
+            0.001548249856568873,
+            -0.007840678095817566,
+            -0.0625436082482338,
+            -0.011386028490960598,
+            -0.00981789268553257,
+            0.017317671328783035,
+            0.0446956530213356,
+            0.04557441547513008,
+            -0.01068150345236063,
+            0.002174178371205926,
+            -0.02213571034371853,
+            -0.04542290419340134,
+            -0.02643861435353756,
+            0.0016486257081851363,
+            -0.019999410957098007,
+            -0.02789311669766903,
+            -0.0018313854234293103,
+            0.011211791075766087,
+            -0.013651113025844097,
+            0.01134057529270649,
+            -0.018560059368610382,
+            -0.01862066239118576,
+            -0.012332970276474953,
+            0.018575210124254227,
+            0.009901223704218864,
+            -0.0018796793883666396,
+            -0.015037435106933117,
+            -0.00980274099856615,
+            0.018969137221574783,
+            -0.011279970407485962,
+            0.0009388927719555795,
+            0.01188601367175579,
+            0.024347767233848572,
+            -0.007083124481141567,
+            0.010961798019707203,
+            -0.034756552428007126,
+            -0.01966608688235283,
+            0.007984613068401814,
+            0.002719616750255227,
+            0.020711511373519897,
+            0.008742166683077812,
+            -0.04275631532073021,
+            -0.044301725924015045,
+            0.016044981777668,
+            -0.0562104657292366,
+            0.028120383620262146,
+            -0.03327174484729767,
+            -0.012514783069491386,
+            -0.02671133354306221,
+            0.0013418165035545826,
+            -0.01828734017908573,
+            0.022438732907176018,
+            0.03778676688671112,
+            -0.0011486404109746218,
+            0.005503625608980656,
+            0.014658658765256405,
+            0.04384719207882881,
+            -0.05021063983440399,
+            -0.041180603206157684,
+            0.03754434734582901,
+            -0.01601468026638031,
+            0.0185146052390337,
+            0.00021744151308666915,
+            0.010348179377615452,
+            -0.014431392773985863,
+            0.022772056981921196,
+            0.013757170177996159,
+            -0.008764892816543579,
+            -0.030574856325984,
+            0.04945308715105057,
+            -0.023650817573070526,
+            -0.049513693898916245,
+            0.0018323323456570506,
+            0.018878230825066566,
+            0.018635815009474754,
+            -0.014870774000883102,
+            -0.02907489985227585,
+            -0.024317465722560883,
+            -0.006060427520424128,
+            -0.00947699323296547,
+            -0.03402930125594139,
+            0.011363301426172256,
+            0.0048445542342960835,
+            0.016666175797581673,
+            -0.00645056739449501,
+            -0.0073141781613230705,
+            -0.009370936080813408,
+            0.022559940814971924,
+            0.012635990977287292,
+            -0.03175663948059082,
+            -0.017484333366155624,
+            -0.02528713271021843,
+            0.004359719809144735,
+            0.010136065073311329,
+            0.022166013717651367,
+            0.024287162348628044,
+            -0.0323626808822155,
+            0.04218057543039322,
+            0.01550711877644062,
+            -0.008727015927433968,
+            -0.0019317612750455737,
+            0.02478714846074581,
+            0.04324115067720413,
+            0.0218023881316185,
+            0.026514369994401932,
+            0.002107892418280244,
+            8.705945947440341e-05,
+            -0.015620751306414604,
+            -0.0011391709558665752,
+            0.01862066239118576,
+            0.04911976307630539,
+            -0.012151156552135944,
+            0.01716616004705429,
+            -0.03578682243824005,
+            0.00396200455725193,
+            -0.02580226957798004,
+            -0.030786970630288124,
+            0.03745344281196594,
+            -0.015060162171721458,
+            0.0191964041441679,
+            0.025044716894626617,
+            -0.022469034418463707,
+            -0.011075431481003761,
+            -0.019423669204115868,
+            -0.026181045919656754,
+            -0.01702980138361454,
+            -0.03281721472740173,
+            -0.018135828897356987,
+            -0.002653330797329545,
+            -0.035635314881801605,
+            -0.04987731948494911,
+            -0.004329417832195759,
+            -0.046392571181058884,
+            -0.010507266037166119,
+            0.017378276214003563,
+            -0.011166337877511978,
+            -0.025711363181471825,
+            -0.005336964037269354,
+            0.003973367623984814,
+            -0.022196315228939056,
+            -0.0066588944755494595,
+            0.008439145050942898,
+            0.035938333719968796,
+            -0.02298417128622532,
+            -0.013060221448540688,
+            -0.0024317465722560883,
+            -0.02887793630361557,
+            -0.007878555916249752,
+            0.008999735116958618,
+            0.007723257411271334,
+            -0.0012357590021565557,
+            -0.05196816474199295,
+            -0.006916462909430265,
+            -0.012992041185498238,
+            0.029196109622716904,
+            0.026468917727470398,
+            0.0012168202083557844,
+            -0.02783251367509365,
+            0.008749742060899734,
+            -0.0072535742074251175,
+            0.02339324913918972,
+            -0.007992188446223736,
+            -0.022908415645360947,
+            -0.02034788578748703,
+            0.030786970630288124,
+            -0.0159086212515831,
+            0.009113367646932602,
+            0.009976978413760662,
+            0.0121360057964921,
+            -0.03899884968996048,
+            -0.001445980160497129,
+            0.0005648507503792644,
+            -0.010128489695489407,
+            0.00423093605786562,
+            -0.006738437805324793,
+            -0.007079336792230606,
+            -0.022090258076786995,
+            -0.0218023881316185,
+            0.03818069398403168,
+            -0.009113367646932602,
+            -0.0035661826841533184,
+            -0.01088604237884283,
+            0.03436262533068657,
+            -0.00595815759152174,
+            -0.01993880607187748,
+            -0.019347915425896645,
+            0.014431392773985863,
+            0.05105910077691078,
+            -0.028529461473226547,
+            -0.03984731063246727,
+            0.0011136035900563002,
+            0.030150625854730606,
+            0.004140029661357403,
+            -0.016908591613173485,
+            0.0035851215943694115,
+            0.056392278522253036,
+            -0.011742077767848969,
+            0.028362801298499107,
+            0.027771908789873123,
+            0.02295386977493763,
+            -0.02021152526140213,
+            -0.01678738370537758,
+            -0.020999381318688393,
+            0.011098157614469528,
+            -0.03027183562517166,
+            0.027059808373451233,
+            0.018969137221574783,
+            0.01538590993732214,
+            0.05054396390914917,
+            -0.03178694099187851,
+            -0.02515077404677868,
+            0.008499749936163425,
+            0.0021722845267504454,
+            0.011007251217961311,
+            -0.007727045100182295,
+            -0.01801462098956108,
+            -0.001950700068846345,
+            -0.014022313989698887,
+            -0.019817598164081573,
+            -0.020938776433467865,
+            0.00954517349600792,
+            0.015696506947278976,
+            0.002649543108418584,
+            0.010863316245377064,
+            0.0016590419691056013,
+            0.01686313934624195,
+            0.01321930717676878,
+            -0.0444229319691658,
+            -7.936555630294606e-05,
+            0.03378688171505928,
+            -0.0293476190418005,
+            -0.012264790013432503,
+            -0.016060132533311844,
+            -0.004439263138920069,
+            -0.01706010289490223,
+            -0.02762039750814438,
+            -0.006874797400087118,
+            -0.01889338344335556,
+            -0.00814369972795248,
+            -0.028196139261126518,
+            -0.009992130100727081,
+            0.01949942484498024,
+            0.022332675755023956,
+            0.001196934375911951,
+            -0.006783891003578901,
+            -0.005901341326534748,
+            0.02092362567782402,
+            0.032514192163944244,
+            0.01952972821891308,
+            0.0058331615291535854,
+            -0.015158643946051598,
+            -0.004295328166335821,
+            -0.012370847165584564,
+            0.016060132533311844,
+            0.012537509202957153,
+            6.344509893096983e-05,
+            0.03493836522102356,
+            0.004170331638306379,
+            -0.022726602852344513,
+            -0.001230077352374792,
+            0.024923507124185562,
+            -0.021590271964669228,
+            -0.0108330138027668,
+            0.040029123425483704,
+            0.0085679292678833,
+            -0.0024506852496415377,
+            -0.019287310540676117,
+            -0.019075196236371994,
+            0.02630225569009781,
+            -0.02133270539343357,
+            0.002083271974697709,
+            0.0482713058590889,
+            0.00035344602656550705,
+            -0.029802151024341583,
+            0.017954016104340553,
+            -0.013242034241557121,
+            -0.018560059368610382,
+            0.01848430372774601,
+            0.0012859469279646873,
+            -0.039968520402908325,
+            0.019014591351151466,
+            0.018408548086881638,
+            0.022635696455836296,
+            -0.02918095886707306,
+            0.0002127068000845611,
+            0.03233237937092781,
+            0.004037759732455015,
+            0.001595596899278462,
+            -0.008530051447451115,
+            -0.0006065162015147507,
+            0.010431510396301746,
+            0.001746160676702857,
+            0.02854461409151554,
+            -0.0030434709042310715,
+            0.027771908789873123,
+            -0.006727074272930622,
+            -0.008863375522196293,
+            0.019181253388524055,
+            -0.02109028771519661,
+            0.013166278600692749,
+            0.0685434341430664,
+            0.0021950111258774996,
+            0.025847723707556725,
+            0.007181606721132994,
+            0.024347767233848572,
+            0.02018122375011444,
+            -0.027271922677755356,
+            -0.02021152526140213,
+            -0.005397567991167307,
+            -0.0014033677289262414,
+            -0.03787767142057419,
+            0.011128460057079792,
+            -0.021302402019500732,
+            -0.013249609619379044,
+            -0.0022347825579345226,
+            -0.002316219499334693,
+            -0.006405114196240902,
+            0.034211114048957825,
+            -0.0014630251098424196,
+            -0.018560059368610382,
+            -0.035423196852207184,
+            0.037726160138845444,
+            0.04048365354537964,
+            0.0389382466673851,
+            -0.005984671879559755,
+            0.03548380360007286,
+            0.0030756669584661722,
+            0.0025302283465862274,
+            0.041453324258327484,
+            0.004791525658220053,
+            -0.011567841283977032,
+            0.02295386977493763,
+            0.012416300363838673,
+            0.03351416438817978,
+            -0.02045394293963909,
+            -0.02630225569009781,
+            -0.003299145260825753,
+            -0.026423463597893715,
+            0.012492056004703045,
+            -0.019726691767573357,
+            0.0018768386216834188,
+            0.007117214612662792,
+            0.015196521766483784,
+            -0.016044981777668,
+            0.014719262719154358,
+            -0.006102093029767275,
+            -0.021014532074332237,
+            0.011719351634383202,
+            -0.007780073676258326,
+            0.011961768381297588,
+            -0.02262054570019245,
+            0.001138224033638835,
+            -0.04063516482710838,
+            0.008280059322714806,
+            0.014022313989698887,
+            -0.01598437689244747,
+            0.03205966204404831,
+            0.020969079807400703,
+            -0.01801462098956108,
+            -0.0009682480012997985,
+            0.0034714885987341404,
+            -0.010393633507192135,
+            -0.011067855171859264,
+            -0.019923655316233635,
+            0.02786281518638134,
+            -0.015802564099431038,
+            -0.025908326730132103,
+            0.02454473078250885,
+            -0.01743887923657894,
+            0.015151068568229675,
+            -0.004299115855246782,
+            0.00888610165566206,
+            -0.019120648503303528,
+            0.02284781076014042,
+            -0.023741723969578743,
+            -0.0009128518868237734,
+            -0.020287280902266502,
+            0.0028389315120875835,
+            0.014135947450995445,
+            -0.018272189423441887,
+            -0.02139330841600895,
+            0.019150950014591217,
+            -0.005420294590294361,
+            -0.003931702114641666,
+            0.010014856234192848,
+            0.0030983935575932264,
+            -0.0029715034179389477,
+            -0.004549108445644379,
+            -0.03281721472740173,
+            -0.02915065549314022,
+            0.0033086147159337997,
+            0.011386028490960598,
+            0.015257125720381737,
+            0.027574945241212845,
+            0.03715042024850845,
+            0.006052851676940918,
+            0.0036949669010937214,
+            0.0009327376610599458,
+            -0.020726662129163742,
+            0.042635105550289154,
+            -0.020469093695282936,
+            0.040847279131412506,
+            0.032544493675231934,
+            -0.0197418425232172,
+            -0.023908386006951332,
+            0.0029052174650132656,
+            0.03115059621632099,
+            0.03151422366499901,
+            0.025211378931999207,
+            0.012310243211686611,
+            0.03690800443291664,
+            -0.004359719809144735,
+            -0.032211173325777054,
+            0.015802564099431038,
+            0.022256920114159584,
+            -0.008105821907520294,
+            0.009673957712948322,
+            0.027438584715127945,
+            0.013279912061989307,
+            -0.007757347077131271,
+            0.030150625854730606,
+            0.0012234487803652883,
+            0.02396899089217186,
+            0.02498411200940609,
+            -0.02190844528377056,
+            0.01743887923657894,
+            -0.0001154677156591788,
+            0.006727074272930622,
+            -0.009840618818998337,
+            0.004348356742411852,
+            0.011810258030891418,
+            0.018272189423441887,
+            0.018908534198999405,
+            0.003246116451919079,
+            -0.07060398161411285,
+            0.019726691767573357,
+            0.004507442936301231,
+            -0.02160542458295822,
+            0.00390140013769269,
+            -0.0024866692256182432,
+            0.012704171240329742,
+            0.02105998620390892,
+            0.012461753562092781,
+            -0.01790856383740902,
+            0.0013626492582261562,
+            0.011961768381297588,
+            -0.00595815759152174,
+            0.04730163514614105,
+            0.0019791084341704845,
+            -0.01437078882008791,
+            -0.016105586662888527,
+            0.013726867735385895,
+            0.02525683119893074,
+            -0.02355991117656231,
+            0.006325571332126856,
+            0.005621046293526888,
+            -0.004147605039179325,
+            -0.010742107406258583,
+            -0.030377892777323723,
+            -0.005280147306621075,
+            -0.03436262533068657,
+            0.023150833323597908,
+            0.015491968020796776,
+            0.014764715917408466,
+            -0.025044716894626617,
+            -0.004280176945030689,
+            0.0046400148421525955,
+            -0.021287251263856888,
+            0.003261267440393567,
+            0.01161329448223114,
+            0.0005104016163386405,
+            -0.020984230563044548,
+            0.00959820207208395,
+            0.013469300232827663,
+            0.06848283112049103,
+            0.0007442962378263474,
+            0.029771849513053894,
+            0.0422411784529686,
+            -0.0012243957025930285,
+            -0.006371024530380964,
+            0.0036286809481680393,
+            -0.019953956827521324,
+            0.003473382443189621,
+            0.025878025218844414,
+            -0.03224147483706474,
+            -0.055058982223272324,
+            -0.007545232307165861,
+            -0.04624105989933014,
+            0.001489539397880435,
+            0.014711687341332436,
+            0.023044776171445847,
+            -0.0037782976869493723,
+            -0.034817155450582504,
+            0.016908591613173485,
+            -0.003071879269555211,
+            0.007014944683760405,
+            -0.02593863010406494,
+            -0.0294839795678854,
+            -0.026014383882284164,
+            -0.02339324913918972,
+            -0.012908710166811943,
+            -0.022726602852344513,
+            0.012590537779033184,
+            -0.0015103721525520086,
+            -0.06587684899568558,
+            -0.009780014865100384,
+            0.016166189685463905,
+            0.001200722181238234,
+            0.011939042247831821,
+            0.011083006858825684,
+            -0.007083124481141567,
+            -0.0018275976181030273,
+            -0.022559940814971924,
+            0.03124150261282921,
+            0.005177877843379974,
+            0.03584742918610573,
+            -0.006503595970571041,
+            -0.009128518402576447,
+            0.01073453202843666,
+            0.04527139291167259,
+            0.0017916138749569654,
+            0.010226971469819546,
+            -0.020560000091791153,
+            0.0015065843472257257,
+            -0.02284781076014042,
+            -0.0031135445460677147,
+            -0.008863375522196293,
+            -0.0021211495622992516,
+            0.015893470495939255,
+            0.0019753205124288797,
+            -0.010204244405031204,
+            0.025438643991947174,
+            0.02251448854804039,
+            -0.009605777449905872,
+            -0.015969226136803627,
+            -0.02437806874513626,
+            -0.014545026235282421,
+            -0.0002627763315103948,
+            -0.010007280856370926,
+            -0.022453883662819862,
+            -0.020878173410892487,
+            -0.002566212322562933,
+            -0.02559015527367592,
+            0.022529639303684235,
+            0.01220418605953455,
+            -4.882668508798815e-05,
+            -0.006723286584019661,
+            -0.038544319570064545,
+            -0.003727162955328822,
+            -0.03118089959025383,
+            -0.005374841392040253,
+            0.03287781774997711,
+            -0.03745344281196594,
+            -0.0002231231628684327,
+            0.014491996727883816,
+            -0.01993880607187748,
+            0.0028938541654497385,
+            0.008961857296526432,
+            0.02769615314900875,
+            -0.027408283203840256,
+            0.0001795638381736353,
+            0.006954340264201164,
+            0.016908591613173485,
+            -0.009878496639430523,
+            0.027196168899536133,
+            -0.04275631532073021,
+            0.011037553660571575,
+            -0.037120118737220764,
+            -0.020423639565706253,
+            -0.01523439958691597,
+            0.008363389410078526,
+            -0.005018791649490595,
+            -0.02637801133096218,
+            -0.010454237461090088,
+            0.006026337388902903,
+            -0.00693918950855732,
+            -0.01902974210679531,
+            -0.010620899498462677,
+            0.027438584715127945,
+            0.00931033119559288,
+            0.007977037690579891,
+            -0.02955973520874977,
+            -0.01631770096719265,
+            0.006280118133872747,
+            0.052089374512434006,
+            -0.01584801822900772,
+            0.014938953332602978,
+            -0.006639956030994654,
+            0.02342355251312256,
+            -0.028484009206295013,
+            -0.01872672140598297,
+            0.0315445251762867,
+            0.013946558348834515,
+            0.007064185570925474,
+            0.028953691944479942,
+            0.027196168899536133,
+            0.027105262503027916,
+            -0.04654408246278763,
+            0.009886072017252445,
+            -0.002613559365272522,
+            -0.008757317438721657,
+            0.03402930125594139,
+            0.01916610263288021,
+            0.022150862962007523,
+            0.0009744031121954322,
+            -0.01730252057313919,
+            0.0139844361692667,
+            0.014885924756526947,
+            0.007681591901928186,
+            -0.014242004603147507,
+            -0.02660527639091015,
+            0.003367325058206916,
+            -0.007992188446223736,
+            0.00047276070108637214,
+            0.020726662129163742,
+            -0.009052763693034649,
+            0.030999086797237396,
+            -0.014605630189180374,
+            -0.02593863010406494,
+            0.008916404098272324,
+            -0.02268115058541298,
+            -0.0041741193272173405,
+            0.0004476667381823063,
+            0.010083036497235298,
+            -0.023847781121730804,
+            -0.04669559374451637,
+            0.019408518448472023,
+            0.028393102809786797,
+            0.005579380784183741,
+            0.01328748743981123,
+            -0.035332292318344116,
+            0.008992158807814121,
+            0.012310243211686611,
+            0.011348150670528412,
+            0.0019847899675369263,
+            0.035362593829631805,
+            -0.00028834378463216126,
+            0.012885984033346176,
+            -0.008620957843959332,
+            0.028362801298499107,
+            0.00667025800794363,
+            -0.0323626808822155,
+            -0.0378473699092865,
+            -0.01513591781258583,
+            0.015257125720381737,
+            0.006692984607070684,
+            -0.011840560473501682,
+            -0.02278720773756504,
+            -0.038817036896944046,
+            0.015787413343787193,
+            -0.012916285544633865,
+            -0.038271598517894745,
+            -0.010158791206777096,
+            -0.009742137044668198,
+            0.016635872423648834,
+            -0.006916462909430265,
+            -0.016802534461021423,
+            -0.016090434044599533,
+            0.005821797996759415,
+            7.161247049225494e-05,
+            0.0244083721190691,
+            -0.001251857029274106,
+            -0.006855858489871025,
+            -0.016393456608057022,
+            -0.01188601367175579,
+            -0.0035074723418802023,
+            0.03602924197912216,
+            -0.014219277538359165,
+            -0.02383263036608696,
+            0.014984406530857086,
+            0.02657497487962246,
+            0.011636020615696907,
+            0.03227177634835243,
+            0.007181606721132994,
+            -0.014022313989698887,
+            0.003016956616193056,
+            -0.033029329031705856,
+            -0.0046400148421525955,
+            0.005196816753596067,
+            -0.006427840795367956,
+            -0.015651054680347443,
+            -0.01815097965300083,
+            -0.015832867473363876,
+            0.030756669119000435,
+            -0.02487805485725403,
+            0.0218023881316185,
+            0.030347591266036034,
+            0.003174148965626955,
+            -0.02915065549314022,
+            0.013598084449768066,
+            -0.0072005451656877995,
+            0.01862066239118576,
+            0.0019014590652659535,
+            0.028014326468110085,
+            -0.007461901288479567,
+            -0.004818039946258068,
+            -0.02501441352069378,
+            -0.03696860745549202,
+            -0.023484155535697937,
+            -0.008817922323942184,
+            0.0027537066489458084,
+            0.020984230563044548,
+            -0.0375746488571167,
+            -0.017757052555680275,
+            0.009855770505964756,
+            0.015757111832499504,
+            -0.03027183562517166,
+            -0.0025908327661454678,
+            -0.010158791206777096,
+            -0.0004055278259329498,
+            0.005140000022947788,
+            -0.02227207086980343,
+            -0.008977008052170277,
+            -0.015726808458566666,
+            -0.0010435298318043351,
+            0.006166485138237476,
+            -0.0007409819518215954,
+            -0.05805889517068863,
+            -0.013969285413622856,
+            0.018635815009474754,
+            0.0026400736533105373,
+            0.008908828720450401,
+            0.015938924625515938,
+            -0.0111739132553339,
+            0.0031533162109553814,
+            -0.026756787672638893,
+            0.023893235251307487,
+            -0.009492144919931889,
+            0.009355784393846989,
+            -0.013742019422352314,
+            0.012514783069491386,
+            0.014014738611876965,
+            0.004795313347131014,
+            0.0010681503918021917,
+            -0.0054051438346505165,
+            0.01601468026638031,
+            0.028377952054142952,
+            -0.023756874725222588,
+            0.011454207822680473,
+            -0.0002514130319468677,
+            -0.0032158142421394587,
+            -0.029196109622716904,
+            0.00509454682469368,
+            0.003931702114641666,
+            -0.005109698045998812,
+            0.016651025041937828,
+            -0.036544378846883774,
+            0.01662072166800499,
+            0.02177208475768566,
+            0.032544493675231934,
+            0.014832896180450916,
+            -0.017287369817495346,
+            -0.011658747680485249,
+            -0.012385998852550983,
+            -0.002624922664836049,
+            0.022181164473295212,
+            0.012120855040848255,
+            -0.01815097965300083,
+            -0.024287162348628044,
+            0.011249667964875698,
+            0.0005283934879116714,
+            -0.01203752402216196,
+            0.010795135982334614,
+            -0.013166278600692749,
+            0.01349202636629343,
+            0.012711746618151665,
+            0.00866641104221344,
+            0.00888610165566206,
+            0.022862963378429413,
+            -0.011878437362611294,
+            -0.017711598426103592,
+            0.00656041270121932,
+            -0.008916404098272324,
+            -0.0034089903347194195,
+            -0.012598113156855106,
+            0.010795135982334614,
+            0.014878349378705025,
+            0.06490717828273773,
+            -0.013143551535904408,
+            0.0009526234352961183,
+            -0.020438792183995247,
+            -0.031938452273607254,
+            -0.0017944546416401863,
+            -0.020908474922180176,
+            0.0111739132553339,
+            0.008370965719223022,
+            0.01598437689244747,
+            0.012022373266518116,
+            -0.0021912232041358948,
+            0.00795431062579155,
+            0.019317612051963806,
+            -0.010423935018479824,
+            7.255941454786807e-05,
+            -0.006503595970571041,
+            0.0013683309080079198,
+            0.0035851215943694115,
+            -0.011719351634383202,
+            -0.0026949963066726923,
+            0.035695917904376984,
+            0.01068150345236063,
+            0.011651171371340752,
+            -0.03445352986454964,
+            0.0020435003098100424,
+            0.020150920376181602,
+            0.007602048572152853,
+            -0.008302785456180573,
+            -0.01563590206205845,
+            0.01706010289490223,
+            -0.015317730605602264,
+            -0.0022499337792396545,
+            0.002888172399252653,
+            -0.010151215828955173,
+            0.04581683129072189,
+            -0.019787294790148735,
+            -0.0011742077767848969,
+            0.031847547739744186,
+            0.0030207443051040173,
+            -0.0040983641520142555,
+            0.005408931523561478,
+            -0.003473382443189621,
+            -0.0069013116881251335,
+            -0.02139330841600895,
+            0.0031381649896502495,
+            0.00667025800794363,
+            -0.019287310540676117,
+            -0.0072005451656877995,
+            -0.007719469256699085,
+            0.006897523999214172,
+            0.023105379194021225,
+            -0.009492144919931889,
+            -0.00016145357221830636,
+            -0.008575504645705223,
+            0.009848194196820259,
+            -0.0011912527261301875,
+            -0.044301725924015045,
+            -0.006147546228021383,
+            0.021181194111704826,
+            0.000973456131760031,
+            0.0005757405888289213,
+            -0.007450537756085396,
+            0.005007428117096424,
+            -0.017423728480935097,
+            -0.008454295806586742,
+            0.02369627170264721,
+            0.01736312545835972,
+            -0.009113367646932602,
+            0.010863316245377064,
+            -0.0035680767614394426,
+            0.00433320552110672,
+            -0.020014561712741852,
+            -0.01095422264188528,
+            0.003462019143626094,
+            0.015272277407348156,
+            -0.0009819786064326763,
+            -0.029029447585344315,
+            -0.0012272365856915712,
+            -0.00024194362049456686,
+            0.02918095886707306,
+            0.01486319862306118,
+            -0.008961857296526432,
+            -0.021211495622992516,
+            -0.013242034241557121,
+            -0.001237652963027358,
+            -0.015726808458566666,
+            -0.009620928205549717,
+            0.011855711229145527,
+            0.0009137988090515137,
+            0.008992158807814121,
+            -0.018135828897356987,
+            -0.027347678318619728,
+            -0.016166189685463905,
+            0.0018465365283191204,
+            -0.012870832346379757,
+            0.003340810537338257,
+            0.015514694154262543,
+            -0.03430201858282089,
+            -0.015060162171721458,
+            -0.03805948421359062,
+            0.008280059322714806,
+            -0.0029658216517418623,
+            -0.02024182677268982,
+            -0.023605365306138992,
+            0.009484568610787392,
+            0.019181253388524055,
+            -0.0015548784285783768,
+            0.019847899675369263,
+            -0.000889651826582849,
+            0.020060013979673386,
+            -0.0073141781613230705,
+            0.0031703610438853502,
+            -0.013007191941142082,
+            0.0030245319940149784,
+            0.008333087898790836,
+            0.02890823967754841,
+            -0.023075077682733536,
+            -0.006503595970571041,
+            -0.004685468040406704,
+            0.004746072459965944,
+            0.012658718042075634,
+            0.014613205567002296,
+            0.020741812884807587,
+            -0.008272483013570309,
+            -0.0027612822595983744,
+            0.012484480626881123,
+            -0.003799130441620946,
+            0.007295239716768265,
+            0.002390081062912941,
+            0.0019677451346069574,
+            0.004162756260484457,
+            0.008477022871375084,
+            0.0053710537031292915,
+            0.027650700882077217,
+            -0.0028275682125240564,
+            0.0031703610438853502,
+            0.024620486423373222,
+            0.02322658896446228,
+            -0.018878230825066566,
+            -0.028074929490685463,
+            0.002024561632424593,
+            0.050695475190877914,
+            0.007245998363941908,
+            -0.006810405291616917,
+            -0.03927157074213028,
+            -0.0031760428100824356,
+            0.028635520488023758,
+            0.023650817573070526,
+            0.01584801822900772,
+            -0.010567869991064072,
+            -0.012098127976059914,
+            0.02772645466029644,
+            0.013295062817633152,
+            0.02616589516401291,
+            -0.025302285328507423,
+            0.010704229585826397,
+            -0.03869583085179329,
+            0.0449683703482151,
+            0.004439263138920069,
+            -0.014476845972239971,
+            0.0011884119594469666,
+            0.010386057198047638,
+            -0.004537744913250208,
+            0.006431628484278917,
+            -5.8059369621332735e-05,
+            -0.0008086882880888879,
+            0.011923890560865402,
+            0.012007221579551697,
+            0.004265025723725557,
+            -0.02399929240345955,
+            0.007359631359577179,
+            0.004272601567208767,
+            0.03005971945822239,
+            0.005431658122688532,
+            -0.0013863227795809507,
+            0.012385998852550983,
+            -0.005810434930026531,
+            0.008855799213051796,
+            -0.010719381272792816,
+            0.020469093695282936,
+            -0.02918095886707306,
+            1.0986003871948924e-05,
+            0.02051454596221447,
+            -0.0015785519499331713,
+            0.0038199631962925196,
+            -0.023741723969578743,
+            0.01186328660696745,
+            -0.052968136966228485,
+            0.021166043356060982,
+            -0.018741872161626816,
+            0.021741783246397972,
+            0.019590331241488457,
+            -0.009734561666846275,
+            -0.012052674777805805,
+            0.03545350208878517,
+            0.0026760573964565992,
+            -0.014264730736613274,
+            -0.010643625631928444,
+            -0.017999470233917236,
+            -0.02830219641327858,
+            0.041968461126089096,
+            -0.0006117243901826441,
+            0.0028275682125240564,
+            0.0024222771171480417,
+            -0.010666352696716785,
+            -0.008923979476094246,
+            0.001954487757757306,
+            0.020499395206570625,
+            -0.03369597718119621,
+            -0.011128460057079792,
+            -0.008689138107001781,
+            -0.02566591091454029,
+            -0.00020300064352340996,
+            -0.011961768381297588,
+            0.026211349293589592,
+            0.01240872498601675,
+            -0.03696860745549202,
+            0.003780191531404853,
+            0.023650817573070526,
+            0.001603172393515706,
+            0.018120678141713142,
+            0.002130619017407298,
+            0.0010795136913657188,
+            0.009742137044668198,
+            -0.010476963594555855,
+            0.026559824123978615,
+            -0.009204274043440819,
+            0.008658835664391518,
+            -0.001950700068846345,
+            -0.017287369817495346,
+            0.017969166859984398,
+            0.004083212930709124,
+            0.024120500311255455,
+            0.011954193003475666,
+            0.011605718173086643,
+            0.01112088467925787,
+            -0.019878201186656952,
+            0.020090317353606224,
+            -0.0018891488434746861,
+            0.009075489826500416,
+            0.024726543575525284,
+            -0.005863463506102562,
+            0.020832719281315804,
+            0.015257125720381737,
+            -0.00020229043730068952,
+            0.011295121163129807,
+            -0.020120618864893913,
+            -0.011779955588281155,
+            0.0085679292678833,
+            -0.009605777449905872,
+            -0.010325453244149685,
+            0.009901223704218864,
+            -0.010105762630701065,
+            -0.015052586793899536,
+            -0.0071247899904847145,
+            0.016635872423648834,
+            0.0028976418543606997,
+            -0.02610529027879238,
+            -1.7326075976598077e-05,
+            0.008901252411305904,
+            0.006166485138237476,
+            0.018741872161626816,
+            0.011317848227918148,
+            -0.006692984607070684,
+            0.010363331064581871,
+            0.011476934887468815,
+            0.0053104497492313385,
+            -0.005079395603388548,
+            -0.013431422412395477,
+            -0.0024241709616035223,
+            0.018090376630425453,
+            0.009810317307710648,
+            -0.01922670565545559,
+            0.004564259201288223,
+            0.0008877579239197075,
+            -0.01420412678271532,
+            -0.019347915425896645,
+            -0.0038483713287860155,
+            -0.004170331638306379,
+            0.0027915844693779945,
+            0.0509984977543354,
+            -0.0006216673064045608,
+            0.007893706671893597,
+            -0.025741664692759514,
+            -0.006299056578427553,
+            -0.00027556007262319326,
+            -0.0036097420379519463,
+            0.01272689737379551,
+            -0.01824188604950905,
+            -0.03724132478237152,
+            0.012711746618151665,
+            -0.0012584856012836099,
+            0.014923802576959133,
+            -0.00297907879576087,
+            0.014226853847503662,
+            -0.017347972840070724,
+            0.01916610263288021,
+            -0.0011117096291854978,
+            -0.016029831022024155,
+            0.0029374135192483664,
+            -0.011143610812723637,
+            -0.021317552775144577,
+            0.005916492082178593,
+            -0.006848283112049103,
+            -0.03896854817867279,
+            0.0029885482508689165,
+            -0.010423935018479824,
+            0.0011798894265666604,
+            -0.003912763670086861,
+            0.020166072994470596,
+            -0.0061513339169323444,
+            0.0008697660523466766,
+            0.02177208475768566,
+            0.0017026013229042292,
+            0.01601468026638031,
+            -0.014499573037028313,
+            0.010999675840139389,
+            0.012598113156855106,
+            0.02210540883243084,
+            -0.032180868089199066,
+            -0.01393140759319067,
+            0.014211702160537243,
+            0.0026116655208170414,
+            -0.005382417235523462,
+            0.020635755732655525,
+            -0.01294658798724413,
+            0.019650936126708984,
+            -0.0005577487172558904,
+            -0.011704200878739357,
+            -0.03524138405919075,
+            -0.016529815271496773,
+            -3.343888238305226e-05,
+            -0.008439145050942898,
+            0.025317436084151268,
+            0.01760554127395153,
+            -0.0108330138027668,
+            0.026468917727470398,
+            0.0037574649322777987,
+            0.014598054811358452,
+            -0.014257155358791351,
+            -0.005999823100864887,
+            -0.0003773563075810671,
+            -0.0119314668700099,
+            0.015264702029526234,
+            -0.03512017801403999,
+            0.0008342557121068239,
+            -0.00839369185268879,
+            0.005242269951850176,
+            -0.0254992488771677,
+            0.018257036805152893,
+            0.025817420333623886,
+            -0.0031419529113918543,
+            -0.005037730094045401,
+            -0.022802358493208885,
+            0.028968842700123787,
+            0.03049910068511963,
+            0.0024828813038766384,
+            -0.0013825349742546678,
+            -0.00352072948589921,
+            0.0033218718599528074,
+            0.0211205892264843,
+            -0.037998881191015244,
+            -0.009098216891288757,
+            -0.028696123510599136,
+            0.042332086712121964,
+            -0.008098246529698372,
+            -0.013431422412395477,
+            0.01862066239118576,
+            -0.0024715180043131113,
+            0.02355991117656231,
+            -0.004689255729317665,
+            -0.004855917301028967,
+            0.03148392215371132,
+            0.011514811776578426,
+            0.02772645466029644,
+            -0.04015033319592476,
+            0.014704111963510513,
+            -0.006867222022265196,
+            0.004867280833423138,
+            -0.013484450988471508,
+            0.005302873905748129,
+            0.02580226957798004,
+            -0.018060073256492615,
+            0.0068066176027059555,
+            0.030665762722492218,
+            0.00748084019869566,
+            0.002448791405186057,
+            0.0165904201567173,
+            0.003890036838129163,
+            0.010855740867555141,
+            0.000841831264551729,
+            0.0052574207074940205,
+            0.022423582151532173,
+            0.014454119838774204,
+            -0.016029831022024155,
+            -0.015120766125619411,
+            -0.00044790346873924136,
+            -0.004450626205652952,
+            0.005261208396404982,
+            0.013242034241557121,
+            0.0061172437854111195,
+            -0.015681356191635132,
+            0.01052999310195446,
+            0.014158673584461212,
+            0.013128400780260563,
+            -0.0039847311563789845,
+            -0.03412020578980446,
+            -0.02443867363035679,
+            -0.021438762545585632,
+            0.003104075090959668,
+            -0.0008281006012111902,
+            -0.0021211495622992516,
+            0.0017954016802832484,
+            0.017347972840070724,
+            -0.0008352026343345642,
+            0.00379155483096838,
+            0.01686313934624195,
+            -0.01415109820663929,
+            -0.022090258076786995,
+            -0.0035964848939329386,
+            -0.027529491111636162,
+            0.0053710537031292915,
+            0.001844642567448318,
+            -0.00965123064815998,
+            0.008530051447451115,
+            -0.0046059247106313705,
+            -0.007223271764814854,
+            -0.012196609750390053,
+            -0.02007516659796238,
+            0.005780132487416267,
+            0.016878290101885796,
+            0.005730891600251198,
+            -0.009272454306483269,
+            -0.003939277958124876,
+            0.011575416661798954,
+            0.017484333366155624,
+            0.017317671328783035,
+            -0.009295180439949036,
+            0.009628504514694214,
+            0.0004022135108243674,
+            0.004670316819101572,
+            -0.01834794320166111,
+            0.0045869858004152775,
+            -0.0027953721582889557,
+            -0.008658835664391518,
+            0.01105270441621542,
+            -0.00031201730598695576,
+            0.01628739945590496,
+            -0.013090522959828377,
+            0.005624833982437849,
+            0.010333028621971607,
+            -0.020590301603078842,
+            -0.0067308624275028706,
+            -0.01692374423146248,
+            0.026484068483114243,
+            -0.00504530593752861,
+            0.00045405857963487506,
+            -0.02978700026869774,
+            0.0033616432920098305,
+            0.000848933297675103,
+            -0.000845619011670351,
+            -0.012764775194227695,
+            0.010749682784080505,
+            0.03242328763008118,
+            0.0005179771687835455,
+            0.0216660276055336,
+            -0.0036154238041490316,
+            0.0007480839849449694,
+            -0.00785582885146141,
+            0.004977126140147448,
+            -0.004916521720588207,
+            -0.009901223704218864,
+            -0.010863316245377064,
+            0.021726632490754128,
+            -0.0033029329497367144,
+            0.0003356908564455807,
+            0.011332998983561993,
+            -0.012067826464772224,
+            0.0014620780711993575,
+            -0.0047687990590929985,
+            0.00971941091120243,
+            0.0061702728271484375,
+            0.009113367646932602,
+            -0.01176480483263731,
+            0.05451354384422302,
+            -0.0106284748762846,
+            0.005537715740501881,
+            -0.00439380994066596,
+            -0.022226616740226746,
+            -0.017317671328783035,
+            0.012711746618151665,
+            -0.017544938251376152,
+            0.03375658020377159,
+            0.024150803685188293,
+            0.0006467612111009657,
+            0.010393633507192135,
+            -0.006958128418773413,
+            0.023923536762595177,
+            -0.002602196065708995,
+            0.0010795136913657188,
+            -0.004855917301028967,
+            0.015802564099431038,
+            0.04511988162994385,
+            0.017393426969647408,
+            -0.001152428099885583,
+            0.00829521007835865,
+            0.020575150847434998,
+            0.00814369972795248,
+            -0.0020075165666639805,
+            0.014219277538359165,
+            -0.01024212222546339,
+            0.00900731049478054,
+            -0.004458202049136162,
+            0.027438584715127945,
+            0.00814369972795248,
+            -0.006310420110821724,
+            -0.0036836036015301943,
+            -0.0008209985098801553,
+            0.014173824340105057,
+            0.006950552575290203,
+            0.01963578537106514,
+            -0.00461350055411458,
+            -0.0012575386790558696,
+            0.013870803639292717,
+            0.014491996727883816,
+            -0.00552635220810771,
+            0.001724380999803543,
+            -0.01435563713312149,
+            -0.010204244405031204,
+            0.013575357384979725,
+            0.006086941808462143,
+            0.006310420110821724,
+            -0.014514723792672157,
+            -0.00980274099856615,
+            -0.004670316819101572,
+            0.0015435151290148497,
+            0.015052586793899536,
+            0.011060279794037342,
+            -0.004814252257347107,
+            0.006052851676940918,
+            0.013825350441038609,
+            -0.014014738611876965,
+            0.006147546228021383,
+            0.007208121009171009,
+            0.008855799213051796,
+            0.004787737503647804,
+            -0.010817863047122955,
+            -0.00428396463394165,
+            -0.0063937511295080185,
+            0.0012670081341639161,
+            0.016832837834954262,
+            0.013181429356336594,
+            0.0049581872299313545,
+            0.027529491111636162,
+            -0.00839369185268879,
+            -0.012302667833864689,
+            0.02339324913918972,
+            0.011257244274020195,
+            0.0031703610438853502,
+            0.014651083387434483,
+            -0.012635990977287292,
+            -0.012446602806448936,
+            0.006655106786638498,
+            0.004848341923207045,
+            0.006049063988029957,
+            0.009151245467364788,
+            0.00450365524739027,
+            0.007333117071539164,
+            -0.007098275702446699,
+            -0.007427811156958342,
+            0.0014810169814154506,
+            0.01642375811934471,
+            0.020393338054418564,
+            -0.024968961253762245,
+            -0.0018806264270097017,
+            0.0009052763343788683,
+            0.010878467001020908,
+            -0.003316190093755722,
+            0.012310243211686611,
+            5.65205882594455e-05,
+            -0.0074240234680473804,
+            0.010158791206777096,
+            -0.02119634486734867,
+            -0.012681444175541401,
+            0.0012717428617179394,
+            0.013037494383752346,
+            -0.019590331241488457,
+            0.018226735293865204,
+            -0.008893677033483982,
+            -0.013348091393709183,
+            0.013893529772758484,
+            0.020014561712741852,
+            0.004090788308531046,
+            -0.007738408166915178,
+            -0.021862991154193878,
+            -0.01127239502966404,
+            0.016060132533311844,
+            0.020272130146622658,
+            0.000947888707742095,
+            -0.0006472347304224968,
+            0.010173942893743515,
+            -0.013113250024616718,
+            -0.023605365306138992,
+            -0.005230906419456005,
+            0.010386057198047638,
+            0.01746918261051178,
+            0.018832778558135033,
+            -0.006685409229248762,
+            0.02443867363035679,
+            -0.0059127043932676315,
+            -0.015499543398618698,
+            2.86597842205083e-05,
+            -0.02802947722375393,
+            -0.017423728480935097,
+            0.00013778003631159663,
+            0.00246773031540215,
+            0.008598231710493565,
+            0.0008555619278922677,
+            0.01943882182240486,
+            0.000904329412151128,
+            0.017590390518307686,
+            -0.000659071491099894,
+            -0.002009410411119461,
+            -0.0013995799235999584,
+            -0.0015340456739068031,
+            0.02254479005932808,
+            -0.012166308239102364,
+            -0.01176480483263731,
+            0.022302372381091118,
+            0.006852070800960064,
+            0.014719262719154358,
+            -0.027635548263788223,
+            0.013757170177996159,
+            -0.015499543398618698,
+            0.017423728480935097,
+            -0.03408990427851677,
+            0.006420265417546034,
+            0.01489350013434887,
+            -0.010348179377615452,
+            -0.020499395206570625,
+            -0.029650641605257988,
+            -0.024150803685188293,
+            -0.0038710979279130697,
+            -0.010855740867555141,
+            -0.016847988590598106,
+            0.006121031939983368,
+            -0.005802859086543322,
+            0.018499454483389854,
+            -0.007079336792230606,
+            0.033877789974212646,
+            0.006969491485506296,
+            -0.004056698642671108,
+            0.003835114184767008,
+            0.022469034418463707,
+            -0.006102093029767275,
+            0.02109028771519661,
+            0.004408961161971092,
+            -0.001467759720981121,
+            0.013681414537131786,
+            -0.006552837323397398,
+            -0.002361672930419445,
+            0.005821797996759415,
+            -0.010484539903700352,
+            0.022014502435922623,
+            0.005924067925661802,
+            -0.007708106189966202,
+            -0.0006775368237867951,
+            -0.011454207822680473,
+            0.014726839028298855,
+            0.006530110724270344,
+            0.0022480397019535303,
+            -0.011295121163129807,
+            0.011249667964875698,
+            -0.004859705455601215,
+            0.0040642740204930305,
+            0.009484568610787392,
+            -0.01774190180003643,
+            -0.028862785547971725,
+            0.011416330002248287,
+            0.019120648503303528,
+            0.006382387597113848,
+            -0.0055755930952727795,
+            -0.0025681061670184135,
+            0.00839369185268879,
+            0.010651201009750366,
+            0.022150862962007523,
+            -0.012984465807676315,
+            -8.889415767043829e-05,
+            0.013204156421124935,
+            -0.007170243188738823,
+            -0.01056029461324215,
+            0.00487485621124506,
+            0.023438703268766403,
+            0.0068823727779090405,
+            -0.04327145218849182,
+            -0.018469152972102165,
+            0.01909034699201584,
+            0.007136153522878885,
+            -0.04154422879219055,
+            0.0026381798088550568,
+            -0.003689285134896636,
+            -0.023787178099155426,
+            0.016105586662888527,
+            -0.0011817833874374628,
+            -0.01828734017908573,
+            0.028756728395819664,
+            0.02830219641327858,
+            0.01834794320166111,
+            2.6987840101355687e-05,
+            -0.013332940638065338,
+            -0.0318172425031662,
+            -0.023590214550495148,
+            0.008280059322714806,
+            -0.02660527639091015,
+            -0.0072725131176412106,
+            0.011242092587053776,
+            -0.017454031854867935,
+            0.02972639724612236,
+            -0.008507325313985348,
+            0.026938600465655327,
+            0.011446632444858551,
+            0.001691238023340702,
+            -0.013446573168039322,
+            -0.018802475184202194,
+            -0.01350717805325985,
+            -0.008196728304028511,
+            0.014090493321418762,
+            0.017923714593052864,
+            0.011007251217961311,
+            0.011332998983561993,
+            0.02251448854804039,
+            -0.012158732861280441,
+            0.007969462312757969,
+            -2.888172457460314e-05,
+            -0.008961857296526432,
+            0.0026343921199440956,
+            -0.018393397331237793,
+            0.00868156272917986,
+            0.0062308767810463905,
+            0.005458172410726547,
+            -0.007208121009171009,
+            0.010742107406258583,
+            0.02712041325867176,
+            -0.018847929313778877,
+            0.011401179246604443,
+            -0.031362712383270264,
+            -0.014787442982196808,
+            -0.01056029461324215,
+            0.026862844824790955,
+            0.007757347077131271,
+            -0.015726808458566666,
+            -0.007514929864555597,
+            -0.01545409020036459,
+            0.012885984033346176,
+            0.0032442226074635983,
+            0.005764981731772423,
+            0.015817714855074883,
+            0.021893294528126717,
+            0.01746918261051178,
+            -0.00795431062579155,
+            0.006852070800960064,
+            0.004674104508012533,
+            0.028090080246329308,
+            -0.01777220331132412,
+            0.020802417770028114,
+            0.009954252280294895,
+            -0.003935490269213915,
+            0.005685438401997089,
+            -0.014257155358791351,
+            0.008530051447451115,
+            -0.007325541693717241,
+            0.030514251440763474,
+            0.0033067206386476755,
+            0.014075342565774918,
+            -0.02062060497701168,
+            0.006575563922524452,
+            -0.03172633796930313,
+            0.009022461250424385,
+            0.013704141601920128,
+            0.012976890429854393,
+            -0.007677804213017225,
+            -0.024181105196475983,
+            -0.0119314668700099,
+            0.0057990713976323605,
+            -0.01279507763683796,
+            -0.01046181283891201,
+            -0.0034828518982976675,
+            0.012446602806448936,
+            0.018847929313778877,
+            -0.01848430372774601,
+            -0.019923655316233635,
+            -0.009583051316440105,
+            0.02898399345576763,
+            -0.015787413343787193,
+            -0.001471547526307404,
+            0.02413565292954445,
+            0.021241798996925354,
+            0.015022284351289272,
+            0.002155239460989833,
+            0.020090317353606224,
+            -0.0007111532613635063,
+            -0.0237265732139349,
+            -0.02813553437590599,
+            0.007302815094590187,
+            -0.006912675220519304,
+            0.01943882182240486,
+            -0.0035548193845897913,
+            -0.005658924113959074,
+            -0.0034033088013529778,
+            -0.007503566797822714,
+            -0.0165904201567173,
+            -0.01787826046347618,
+            0.0021647089160978794,
+            -0.008454295806586742,
+            0.005863463506102562,
+            -0.0002795845502987504,
+            -0.013105674646794796,
+            0.013401119969785213,
+            -0.014075342565774918,
+            0.005772557109594345,
+            -0.002653330797329545,
+            0.001962063368409872,
+            0.0035926972050219774,
+            0.009113367646932602,
+            -0.007336904760450125,
+            -0.06381630152463913,
+            -0.013105674646794796,
+            -0.003704436356201768,
+            0.006071790587157011,
+            -0.00019578021601773798,
+            -0.016635872423648834,
+            -0.013196581043303013,
+            0.01706010289490223,
+            0.008863375522196293,
+            -0.013870803639292717,
+            0.016954045742750168,
+            0.009961827658116817,
+            0.003622999181970954,
+            -0.024044746533036232,
+            -0.0008148433989845216,
+            0.008757317438721657,
+            0.032756611704826355,
+            -0.008643684908747673,
+            -0.0003839849086944014,
+            0.0036684523802250624,
+            -0.0033332351595163345,
+            -0.010052734054625034,
+            -0.014582903124392033,
+            -0.011848135851323605,
+            0.011908739805221558,
+            0.01790856383740902,
+            -0.014264730736613274,
+            -0.02007516659796238,
+            -0.0007106798002496362,
+            0.012931437231600285,
+            -0.026590125635266304,
+            0.0007244104635901749,
+            -0.007787649054080248,
+            -0.01052999310195446,
+            0.018302490934729576,
+            -0.019014591351151466,
+            0.005249845329672098,
+            0.00048294031876139343,
+            0.008507325313985348,
+            -0.023544760420918465,
+            -0.024893205612897873,
+            0.0035131541080772877,
+            -0.023847781121730804,
+            0.03612014651298523,
+            -0.021590271964669228,
+            -0.012082977220416069,
+            0.034483831375837326,
+            -0.0037423139438033104,
+            0.01702980138361454,
+            0.0023995505180209875,
+            0.02739313244819641,
+            -0.011439057067036629,
+            -0.00617784820497036,
+            0.012499631382524967,
+            0.009575475007295609,
+            0.0032593735959380865,
+            0.03775646165013313,
+            -0.01257538702338934,
+            -0.013901105150580406,
+            -0.005556654185056686,
+            -0.012954163365066051,
+            0.0006775368237867951,
+            0.014431392773985863,
+            0.028393102809786797,
+            0.004257450345903635,
+            -0.00628769351169467,
+            -0.006143758539110422,
+            -0.0014649189542979002,
+            -0.004242299124598503,
+            -0.00947699323296547,
+            -0.00715130427852273,
+            0.017287369817495346,
+            -0.025241680443286896,
+            0.016711628064513206,
+            -0.006420265417546034,
+            0.021135739982128143,
+            -0.011757229454815388,
+            0.007007369305938482,
+            -0.008961857296526432,
+            0.003492321353405714,
+            0.0023635667748749256,
+            -0.005109698045998812,
+            0.009287605062127113,
+            -0.014772292226552963,
+            0.008454295806586742,
+            0.013742019422352314,
+            0.0011183382011950016,
+            -0.010590597055852413,
+            -0.019272159785032272,
+            0.0022972808219492435,
+            -0.000710206339135766,
+            0.01181783340871334,
+            -0.0047498601488769054,
+            -0.000981031684204936,
+            0.0002921315317507833,
+            -0.014476845972239971,
+            -0.009575475007295609,
+            0.0006050958181731403,
+            0.005302873905748129,
+            -0.008961857296526432,
+            0.010378481820225716,
+            -0.005693014245480299,
+            -0.00034160923678427935,
+            0.02627195231616497,
+            0.006526322569698095,
+            -0.007632351014763117,
+            -0.014098069630563259,
+            0.01563590206205845,
+            0.015370759181678295,
+            -0.007454325910657644,
+            0.01933276280760765,
+            -0.016196493059396744,
+            -0.0008366230758838356,
+            0.013916256837546825,
+            -0.009219424799084663,
+            0.0017698341980576515,
+            -0.008196728304028511,
+            0.03554440662264824,
+            -0.0008787619881331921,
+            -0.0035415622405707836,
+            -0.0006401326390914619,
+            0.012976890429854393,
+            -0.0026779514737427235,
+            0.001182730309665203,
+            0.01132542360574007,
+            0.005321812815964222,
+            0.0185146052390337,
+            0.00328020635060966,
+            -0.03378688171505928,
+            -0.007780073676258326,
+            -0.0065490491688251495,
+            -0.008090670220553875,
+            0.009022461250424385,
+            0.025196226313710213,
+            0.01990850456058979,
+            0.006408901885151863,
+            0.001881573349237442,
+            0.018423698842525482,
+            0.015757111832499504,
+            -0.01489350013434887,
+            0.016363153234124184,
+            0.02802947722375393,
+            -0.010030006989836693,
+            0.022741753607988358,
+            -0.005253633018583059,
+            -0.008893677033483982,
+            0.008946705609560013,
+            -0.0015378334792330861,
+            -0.017393426969647408,
+            -0.015681356191635132,
+            0.005022579338401556,
+            -0.008999735116958618,
+            -0.0022897052112966776,
+            -0.018605511635541916,
+            0.004670316819101572,
+            0.017105557024478912,
+            -0.004965762607753277,
+            -3.5243989259470254e-05,
+            0.019484274089336395,
+            -0.006965703796595335,
+            -0.016908591613173485,
+            0.006325571332126856,
+            -0.015007133595645428,
+            0.007946735247969627,
+            -0.013666263781487942,
+            0.010492115281522274,
+            -0.006098304875195026,
+            0.018817627802491188,
+            0.005446809343993664,
+            -0.008090670220553875,
+            -0.0008958069374784827,
+            -0.018984289839863777,
+            0.02065090648829937,
+            -0.033089932054281235,
+            0.00839369185268879,
+            -0.026226500049233437,
+            -0.0009781909175217152,
+            0.0008849170990288258,
+            0.014605630189180374,
+            -0.001218714052811265,
+            0.019696388393640518,
+            -0.002751812804490328,
+            -0.020090317353606224,
+            -0.007086912170052528,
+            -0.0010634156642481685,
+            -0.01618134044110775,
+            -0.030923331156373024,
+            -0.004272601567208767,
+            0.0036267871037125587,
+            -0.004711982328444719,
+            0.0021135739516466856,
+            0.017984317615628242,
+            -0.005613470915704966,
+            0.009173971600830555,
+            -0.02690829709172249,
+            -0.007416448090225458,
+            -0.00355103169567883,
+            0.0010776197304949164,
+            -0.011242092587053776,
+            0.0010596278589218855,
+            0.011226941831409931,
+            -0.019044892862439156,
+            0.013461724855005741,
+            -0.021378157660365105,
+            -0.004371083341538906,
+            -0.008507325313985348,
+            -0.00135412672534585,
+            -0.013446573168039322,
+            -0.028120383620262146,
+            0.025029566138982773,
+            -0.00021495578403118998,
+            -0.003528305096551776,
+            -0.011999646201729774,
+            0.023181134834885597,
+            0.002914686920121312,
+            0.027817361056804657,
+            -0.03157482668757439,
+            0.0013427634257823229,
+            0.011552689597010612,
+            0.0025984083767980337,
+            -0.017287369817495346,
+            0.004746072459965944,
+            -0.005234694108366966,
+            0.02210540883243084,
+            -0.0011694730492308736,
+            -0.0005340751959010959,
+            -0.008234606124460697,
+            0.030999086797237396,
+            0.009136094711720943,
+            0.0012944694608449936,
+            -0.013226882554590702,
+            -0.008060368709266186,
+            -0.002759388415142894,
+            -0.0011069749016314745,
+            0.012855681590735912,
+            0.01290113478899002,
+            -0.012105703353881836,
+            0.0013607552973553538,
+            -0.006878585088998079,
+            0.0011694730492308736,
+            -0.002935519441962242,
+            -0.0004966709529981017,
+            -0.01149208564311266,
+            0.003257479751482606,
+            0.016272246837615967,
+            -0.00780280027538538,
+            -0.006480869371443987,
+            0.0007750718505121768,
+            0.009181547909975052,
+            0.012461753562092781,
+            -0.012742049060761929,
+            -0.018226735293865204,
+            -0.011908739805221558,
+            0.016029831022024155,
+            0.010522416792809963,
+            -0.008204303681850433,
+            -0.013234458863735199,
+            -0.02583257108926773,
+            0.01169662456959486,
+            -0.03293842449784279,
+            -0.012242063879966736,
+            0.005821797996759415,
+            0.028559764847159386,
+            0.019999410957098007,
+            0.012226912193000317,
+            0.001332347048446536,
+            0.002950670663267374,
+            0.00023342114582192153,
+            -0.01916610263288021,
+            0.014408666640520096,
+            -0.011143610812723637,
+            -0.018166130408644676,
+            0.0075906855054199696,
+            0.011939042247831821,
+            0.03148392215371132,
+            0.006942977197468281,
+            -0.0010160685051232576,
+            0.01686313934624195,
+            -0.001218714052811265,
+            -0.013582932762801647,
+            0.01420412678271532,
+            -0.024256860837340355,
+            0.016438908874988556,
+            0.02860521711409092,
+            -0.004780162125825882,
+            0.0008531945641152561,
+            -0.020938776433467865,
+            0.00482561532407999,
+            -0.011529963463544846,
+            -0.002177966060116887,
+            -0.003287781961262226,
+            -0.03460504114627838,
+            0.007946735247969627,
+            -0.006825556512922049,
+            0.01046181283891201,
+            -0.00814369972795248,
+            0.006655106786638498,
+            -0.012431452050805092,
+            0.004253662656992674,
+            0.01513591781258583,
+            0.016105586662888527,
+            0.007060397882014513,
+            -0.025211378931999207,
+            -0.009196698665618896,
+            0.008105821907520294,
+            -0.01349202636629343,
+            -0.02295386977493763,
+            -0.006852070800960064,
+            0.009840618818998337,
+            -0.01061332318931818,
+            -0.00488243205472827,
+            -0.028180986642837524,
+            -0.013681414537131786,
+            -0.0014327229000627995,
+            -0.017817657440900803,
+            -0.005556654185056686,
+            -0.003100287402048707,
+            0.01571165770292282,
+            -0.0049733384512364864,
+            -0.006185423582792282,
+            0.004742284305393696,
+            -0.023878084495663643,
+            0.0037423139438033104,
+            -0.008280059322714806,
+            0.0008266801596619189,
+            -0.0032517979852855206,
+            -0.0014762822538614273,
+            -0.007969462312757969,
+            -0.014332910999655724,
+            0.006893736310303211,
+            -0.016469212248921394,
+            0.002954458352178335,
+            -0.015173795633018017,
+            -0.016499513760209084,
+            -0.00682934420183301,
+            0.00775355938822031,
+            0.021575121209025383,
+            0.006541473791003227,
+            0.021484214812517166,
+            0.005469535943120718,
+            0.028680972754955292,
+            0.0003953482082579285,
+            0.003041577059775591,
+            -8.958956641436089e-06,
+            0.003232859307900071,
+            0.019347915425896645,
+            0.006571775767952204,
+            0.018984289839863777,
+            -0.008742166683077812,
+            0.010514841414988041,
+            -0.010355755686759949,
+            0.010014856234192848,
+            0.006333146709948778,
+            -0.006473293993622065,
+            -0.003799130441620946,
+            -0.010348179377615452,
+            -0.0031419529113918543,
+            -0.016544966027140617,
+            0.0027366618160158396,
+            0.00693161366507411,
+            0.0071588801220059395,
+            -0.011582992039620876,
+            -0.005230906419456005,
+            -0.014704111963510513,
+            -0.0064354161731898785,
+            -0.01831764169037342,
+            0.014385939575731754,
+            0.029226411134004593,
+            -0.016332851722836494,
+            -0.01977214403450489,
+            -0.007772498298436403,
+            0.0007296186522580683,
+            -0.006636167876422405,
+            -0.01240872498601675,
+            -0.009151245467364788,
+            -0.035059571266174316,
+            -0.017666146159172058,
+            0.007586897816509008,
+            0.02484775334596634,
+            0.01387837901711464,
+            -0.013628385961055756,
+            -0.0006983695784583688,
+            -0.016984347254037857,
+            -0.010787560604512691,
+            -0.01535560842603445,
+            -0.02054484933614731,
+            -0.02498411200940609,
+            0.0077952248975634575,
+            -0.004033972043544054,
+            0.014931377954781055,
+            0.008113397285342216,
+            0.0032631612848490477,
+            -0.004371083341538906,
+            -0.0024109138175845146,
+            -0.013196581043303013,
+            0.0012925754999741912,
+            -0.004787737503647804,
+            -0.0017291157273575664,
+            0.03554440662264824,
+            -0.008098246529698372,
+            -0.005871038883924484,
+            0.011060279794037342,
+            0.011128460057079792,
+            -0.013764745555818081,
+            0.0006642796797677875,
+            -0.011249667964875698,
+            0.021847840398550034,
+            -0.021620575338602066,
+            0.019257009029388428,
+            0.0014507147716358304,
+            -0.010840589180588722,
+            0.028999146074056625,
+            0.01892368495464325,
+            0.00743159931153059,
+            0.006579351611435413,
+            0.01672677882015705,
+            0.0044316877610981464,
+            0.016196493059396744,
+            -0.0026552248746156693,
+            -0.004632438998669386,
+            0.009295180439949036,
+            0.015681356191635132,
+            -0.000702630786690861,
+            -0.006942977197468281,
+            0.016438908874988556,
+            0.004056698642671108,
+            0.0075414446182549,
+            -0.01356020662933588,
+            -0.008961857296526432,
+            0.001507531269453466,
+            0.006257391534745693,
+            -0.005939218681305647,
+            -0.0026003022212535143,
+            0.005530139897018671,
+            0.003943065647035837,
+            -0.013302638195455074,
+            0.01689344085752964,
+            -0.016166189685463905,
+            0.008742166683077812,
+            -0.0037972365971654654,
+            0.0121360057964921,
+            -0.006799042224884033,
+            0.0013153020991012454,
+            -0.01818128302693367,
+            -0.003869204083457589,
+            0.0038597346283495426,
+            0.012173883616924286,
+            -0.01262841559946537,
+            0.025484098121523857,
+            0.008166425861418247,
+            0.006602078210562468,
+            0.0059467945247888565,
+            -0.012855681590735912,
+            0.027211319655179977,
+            0.005969521123915911,
+            0.009886072017252445,
+            -0.008893677033483982,
+            0.011136035434901714,
+            0.035362593829631805,
+            0.03060515783727169,
+            0.014135947450995445,
+            0.004469565115869045,
+            -0.018999440595507622,
+            0.015120766125619411,
+            0.0018427487229928374,
+            0.0023806116078048944,
+            0.00863610953092575,
+            -0.011348150670528412,
+            0.01305264513939619,
+            -0.009484568610787392,
+            0.015241974964737892,
+            -0.021272100508213043,
+            -0.011749654076993465,
+            -0.014408666640520096,
+            -0.021075136959552765,
+            -0.01430260855704546,
+            0.034180812537670135,
+            -0.002346521709114313,
+            -0.010393633507192135,
+            0.0045112306252121925,
+            0.008477022871375084,
+            -0.006079366430640221,
+            0.0053142374381423,
+            -0.008204303681850433,
+            -0.013749594800174236,
+            -0.01889338344335556,
+            0.006628592498600483,
+            -0.01550711877644062,
+            0.007651289459317923,
+            -0.003304826794192195,
+            0.0037688282318413258,
+            0.007014944683760405,
+            0.018105527386069298,
+            0.00530666159465909,
+            -0.00135412672534585,
+            0.0014033677289262414,
+            0.0016865032957866788,
+            -0.00931033119559288,
+            0.0108330138027668,
+            0.015408637002110481,
+            -0.0011921997647732496,
+            -0.011279970407485962,
+            -0.009136094711720943,
+            -0.042877525091171265,
+            0.0006264019757509232,
+            -0.013128400780260563,
+            0.0064354161731898785,
+            -0.011196639388799667,
+            0.0022234192583709955,
+            0.015863168984651566,
+            -0.0054051438346505165,
+            -0.0010918239131569862,
+            0.02193874679505825,
+            0.013014768250286579,
+            0.021075136959552765,
+            0.02065090648829937,
+            0.014188976027071476,
+            -0.015302578918635845,
+            -0.03469594568014145,
+            0.002221525413915515,
+            -0.010030006989836693,
+            0.013772320933640003,
+            0.004238511435687542,
+            0.016060132533311844,
+            -0.023893235251307487,
+            -0.011878437362611294,
+            -0.006662682630121708,
+            0.014431392773985863,
+            0.01491622719913721,
+            -0.0032271775417029858,
+            0.02690829709172249,
+            0.009280029684305191,
+            0.0020510759204626083,
+            -0.02281750924885273,
+            0.003312402404844761,
+            0.005321812815964222,
+            -0.008370965719223022,
+            -0.016681326553225517,
+            -0.0004552422615233809,
+            -0.0012168202083557844,
+            -0.0007817004225216806,
+            -0.021650876849889755,
+            0.018832778558135033,
+            -0.004018820822238922,
+            0.0037574649322777987,
+            0.013840501196682453,
+            0.01516621932387352,
+            0.017984317615628242,
+            0.008370965719223022,
+            0.001497115008533001,
+            -0.006605865899473429,
+            -0.0007973249885253608,
+            -0.0005828426801599562,
+            -0.007389933802187443,
+            -0.005598319694399834,
+            0.006382387597113848,
+            -0.01134057529270649,
+            0.008810346014797688,
+            0.000896753859706223,
+            0.006113456096500158,
+            0.010605747811496258,
+            -0.004836978856474161,
+            -0.011628445237874985,
+            0.014628356322646141,
+            -0.0010548931313678622,
+            -0.0055755930952727795,
+            -0.011060279794037342,
+            0.003624893259257078,
+            0.013575357384979725,
+            -0.003096499713137746,
+            -0.013499601744115353,
+            0.00910579226911068,
+            0.002009410411119461,
+            -0.027438584715127945,
+            0.015499543398618698,
+            0.010295150801539421,
+            -0.013370818458497524,
+            0.004980913829058409,
+            -0.003371112747117877,
+            -0.03027183562517166,
+            0.004791525658220053,
+            0.002660906407982111,
+            -0.010643625631928444,
+            0.028938541188836098,
+            -0.003821857040748,
+            0.029256712645292282,
+            -0.004980913829058409,
+            -0.006905099377036095,
+            -0.011355726048350334,
+            0.009416389279067516,
+            -0.018681267276406288,
+            -0.0037953427527099848,
+            -0.008727015927433968,
+            -0.00915882084518671,
+            -0.0009838725673034787,
+            -0.005185453221201897,
+            0.006984642706811428,
+            0.005121061112731695,
+            -0.012704171240329742,
+            0.006757376715540886,
+            -0.010257272981107235,
+            -0.00020856392802670598,
+            0.0006718551740050316,
+            -0.008408842608332634,
+            0.013885954394936562,
+            0.005996035411953926,
+            -0.008340663276612759,
+            -0.0005032995832152665,
+            0.027817361056804657,
+            0.05051366239786148,
+            -0.018863080069422722,
+            0.0009270560112781823,
+            0.01452229917049408,
+            -0.0028824908658862114,
+            -0.0021590273827314377,
+            -0.011976920068264008,
+            0.005821797996759415,
+            -0.019423669204115868,
+            0.013317788951098919,
+            0.006473293993622065,
+            -0.008833073079586029,
+            0.003202557098120451,
+            -0.016711628064513206,
+            0.002621134975925088,
+            -0.0018058179412037134,
+            -0.018681267276406288,
+            -0.015529844909906387,
+            -0.012060250155627728,
+            -0.024181105196475983,
+            0.0004235197266098112,
+            -0.0008546149474568665,
+            -0.013870803639292717,
+            0.006579351611435413,
+            0.008431569673120975,
+            0.032483890652656555,
+            0.0011912527261301875,
+            0.010128489695489407,
+            -2.1513333194889128e-05,
+            -0.0010416359873488545,
+            -0.005988460034132004,
+            -0.01523439958691597,
+            0.00015387803432531655,
+            -0.002342734020203352,
+            0.022590242326259613,
+            0.005549078807234764,
+            0.017560089007019997,
+            0.0017887729918584228,
+            -0.016514664515852928,
+            -0.00596573343500495,
+            -0.007408872712403536,
+            -0.018060073256492615,
+            0.0062649669125676155,
+            -0.005473323632031679,
+            0.009605777449905872,
+            -0.015863168984651566,
+            -0.02731737680733204,
+            0.00792400911450386,
+            0.012234487570822239,
+            0.028514310717582703,
+            -0.0035623949952423573,
+            0.001716805505566299,
+            -0.005533927585929632,
+            0.016408607363700867,
+            0.032180868089199066,
+            -0.0008550884085707366,
+            0.006950552575290203,
+            -0.019999410957098007,
+            0.0017404790269210935,
+            0.013704141601920128,
+            -0.008401267230510712,
+            -0.01821158453822136,
+            0.003939277958124876,
+            -0.020635755732655525,
+            0.004583198111504316,
+            -0.020393338054418564,
+            0.030408194288611412,
+            -0.0023522034753113985,
+            0.008105821907520294,
+            0.004310478921979666,
+            0.01760554127395153,
+            -0.0031646795105189085,
+            0.010545143857598305,
+            0.013916256837546825,
+            0.014908651821315289,
+            0.0009980766335502267,
+            -0.011189064010977745,
+            -0.03790797293186188,
+            0.001859793672338128,
+            0.00817400123924017,
+            0.018635815009474754,
+            -0.01980244740843773,
+            0.011681473813951015,
+            -0.01296931505203247,
+            -0.020363036543130875,
+            -0.01003758329898119,
+            0.011575416661798954,
+            -0.006139970384538174,
+            0.03378688171505928,
+            -0.005874827038496733,
+            -0.008204303681850433,
+            0.005136212334036827,
+            0.010454237461090088,
+            0.0066778333857655525,
+            0.014704111963510513,
+            0.0326354019343853,
+            0.010196669027209282,
+            0.014173824340105057,
+            -0.004859705455601215,
+            0.02399929240345955,
+            -0.003931702114641666,
+            0.003954428713768721,
+            0.00509454682469368,
+            -0.022832660004496574,
+            -0.0040983641520142555,
+            -0.003931702114641666,
+            -0.016711628064513206,
+            -0.007458113599568605,
+            0.0032366469968110323,
+            0.008802770636975765,
+            -0.014045040123164654,
+            -0.00649980828166008,
+            -0.01350717805325985,
+            0.01218903437256813,
+            0.002730980049818754,
+            0.018302490934729576,
+            -0.029802151024341583,
+            0.009825468063354492,
+            -0.017484333366155624,
+            -0.017044952139258385,
+            0.006511171814054251,
+            -0.0038332203403115273,
+            0.006855858489871025,
+            -0.018272189423441887,
+            -0.01318900566548109,
+            0.012817803770303726,
+            0.0007897494360804558,
+            0.013060221448540688,
+            -0.009825468063354492,
+            0.0009706153068691492,
+            0.006064215209335089,
+            -0.009833043441176414,
+            -0.010961798019707203,
+            -0.007977037690579891,
+            -0.009681533090770245,
+            -0.008848223835229874,
+            0.005621046293526888,
+            0.01656011864542961,
+            -0.013742019422352314,
+            -0.003685497445985675,
+            0.016135888174176216,
+            0.006416477728635073,
+            -0.008166425861418247,
+            0.004075637552887201,
+            -0.003793448908254504,
+            -9.948806837201118e-05,
+            0.0013740125577896833,
+            0.012439027428627014,
+            -0.0017632056260481477,
+            0.024665940552949905,
+            -0.02654467150568962,
+            0.007598260883241892,
+            -0.0054051438346505165,
+            0.005446809343993664,
+            0.018226735293865204,
+            -0.02366596832871437,
+            -0.013082947582006454,
+            0.006276329979300499,
+            -0.01966608688235283,
+            -0.008060368709266186,
+            0.0054051438346505165,
+            -0.00016405766655225307,
+            -0.019923655316233635,
+            -0.012923861853778362,
+            -0.034786853939294815,
+            -0.02309022843837738,
+            0.024665940552949905,
+            0.008401267230510712,
+            -0.0073672072030603886,
+            0.018060073256492615,
+            -0.018438849598169327,
+            0.0172722190618515,
+            -0.002225313102826476,
+            -0.006613441277295351,
+            0.003935490269213915,
+            0.007674016058444977,
+            0.025362888351082802,
+            0.002653330797329545,
+            0.008264907635748386,
+            -0.0042082094587385654,
+            -0.014931377954781055,
+            -0.006321783177554607,
+            0.01980244740843773,
+            0.02471139281988144,
+            0.018605511635541916,
+            -0.018575210124254227,
+            0.010742107406258583,
+            -0.011749654076993465,
+            0.01818128302693367,
+            -0.006537686102092266,
+            0.00039037675014697015,
+            0.01859036087989807,
+            0.007969462312757969,
+            0.025241680443286896,
+            0.019923655316233635,
+            0.006852070800960064,
+            0.014128371141850948,
+            0.011908739805221558,
+            0.001204509986564517,
+            -0.012832955457270145,
+            -0.01154511421918869,
+            -0.02840825356543064,
+            0.042847223579883575,
+            0.0025434857234358788,
+            0.008317936211824417,
+            -0.027635548263788223,
+            -0.00021389048197306693,
+            -0.0034942151978611946,
+            -0.018832778558135033,
+            0.005939218681305647,
+            0.03260510042309761,
+            0.03284751623868942,
+            0.021438762545585632,
+            -2.888172457460314e-05,
+            0.0012234487803652883,
+            0.0329081192612648,
+            -0.013673839159309864,
+            -0.00937851145863533,
+            -0.01034060399979353,
+            -0.018393397331237793,
+            -0.006852070800960064,
+            -0.0032423287630081177,
+            -0.01031030248850584,
+            -0.0061172437854111195,
+            -0.009272454306483269,
+            0.0238629337400198,
+            0.0001374249259242788,
+            -0.0022594030015170574,
+            0.006605865899473429,
+            -0.020878173410892487,
+            0.011537538841366768,
+            -0.02813553437590599,
+            -0.013431422412395477,
+            -0.008651260286569595,
+            -0.006950552575290203,
+            -0.018802475184202194,
+            0.008643684908747673,
+            -0.006942977197468281,
+            0.021438762545585632,
+            0.002914686920121312,
+            -0.023590214550495148,
+            0.007530081085860729,
+            0.015317730605602264,
+            -0.0003825644962489605,
+            0.014037464745342731,
+            -0.003890036838129163,
+            0.009060339070856571,
+            0.0045112306252121925,
+            0.011681473813951015,
+            0.016954045742750168,
+            0.010651201009750366,
+            0.009030036628246307,
+            0.003437398700043559,
+            -0.001167579204775393,
+            0.01878732442855835,
+            -0.0007163614500313997,
+            -0.03405960276722908,
+            2.9236827685963362e-05,
+            -0.014628356322646141,
+            0.029499130323529243,
+            -0.009795165620744228,
+            0.009514871053397655,
+            0.008105821907520294,
+            0.007098275702446699,
+            -0.0067119235172867775,
+            0.007916433736681938,
+            -0.008742166683077812,
+            0.0052763596177101135,
+            0.015832867473363876,
+            -0.0042877523228526115,
+            -0.0014980619307607412,
+            0.007840678095817566,
+            0.00407184986397624,
+            -0.005454384721815586,
+            -0.019711541011929512,
+            -0.012287517078220844,
+            0.00726114958524704,
+            -0.02065090648829937,
+            -0.004367295652627945,
+            0.01161329448223114,
+            -0.005454384721815586,
+            -0.009620928205549717,
+            0.01393140759319067,
+            0.02136300690472126,
+            -0.002382505452260375,
+            0.022893264889717102,
+            -0.01349202636629343,
+            -0.009833043441176414,
+            -0.007984613068401814,
+            -0.004374871030449867,
+            -0.041120000183582306,
+            0.025984082370996475,
+            0.01068150345236063,
+            -0.006871009711176157,
+            0.01895398646593094,
+            0.0022499337792396545,
+            -0.002954458352178335,
+            -0.0011041341349482536,
+            0.00010499217023607343,
+            -0.03281721472740173,
+            0.03660498186945915,
+            -0.007007369305938482,
+            0.0005449649761430919,
+            0.010272424668073654,
+            0.042029064148664474,
+            -0.004590773954987526,
+            -0.005859675817191601,
+            0.01225721463561058,
+            0.005336964037269354,
+            0.031938452273607254,
+            -0.010416359640657902,
+            -0.014340486377477646,
+            -0.01815097965300083,
+            -0.004833190701901913,
+            -0.01939336769282818,
+            -0.0038313264958560467,
+            0.014037464745342731,
+            -0.019650936126708984,
+            -0.011242092587053776,
+            -0.001287840772420168,
+            -0.012537509202957153,
+            -0.0010747789638116956,
+            0.008598231710493565,
+            0.017832808196544647,
+            -0.011348150670528412,
+            -0.004450626205652952,
+            -2.2904154320713133e-05,
+            -0.005003640428185463,
+            0.0185146052390337,
+            0.0020056227222085,
+            -0.01702980138361454,
+            0.01321930717676878,
+            -0.00526499655097723,
+            0.006749800872057676,
+            0.0074429623782634735,
+            0.007939159870147705,
+            0.05145302787423134,
+            0.010317877866327763,
+            -0.001339922659099102,
+            -0.01607528328895569,
+            0.023196285590529442,
+            -0.01749948412179947,
+            -0.014181400649249554,
+            -0.02119634486734867,
+            -0.0029923359397798777,
+            -0.007060397882014513,
+            -0.028014326468110085,
+            -9.137987944995984e-05,
+            0.0016221113037317991,
+            -0.0024980325251817703,
+            -0.010045158676803112,
+            0.0005208179936744273,
+            -0.011529963463544846,
+            -0.006954340264201164,
+            -0.008545203134417534,
+            -0.016044981777668,
+            0.02240842953324318,
+            -0.021817538887262344,
+            -0.008855799213051796,
+            0.011514811776578426,
+            0.03118089959025383,
+            -0.005011215806007385,
+            -0.014173824340105057,
+            -0.019817598164081573,
+            0.0025643182452768087,
+            -0.01763584464788437,
+            0.012181458994746208,
+            -0.01225721463561058,
+            0.015045011416077614,
+            -0.03142331540584564,
+            -0.006079366430640221,
+            -0.008992158807814121,
+            0.012840530835092068,
+            0.019817598164081573,
+            -0.0025093958247452974,
+            0.00606800289824605,
+            -0.010295150801539421,
+            0.030211230739951134,
+            -0.00012286569108255208,
+            -0.018605511635541916,
+            0.00873459130525589,
+            0.007977037690579891,
+            0.012537509202957153,
+            -0.004435475450009108,
+            0.00257757562212646,
+            0.011567841283977032,
+            0.012742049060761929,
+            -0.01634800247848034,
+            -0.003174148965626955,
+            -0.017969166859984398,
+            0.005996035411953926,
+            0.006999793462455273,
+            0.021711481735110283,
+            -0.0013550736475735903,
+            0.018272189423441887,
+            -0.01563590206205845,
+            0.00019412307301536202,
+            -0.011749654076993465,
+            -0.0082270298153162,
+            -0.0028237802907824516,
+            -0.003958216868340969,
+            0.006685409229248762,
+            -0.028211290016770363,
+            -0.01274962443858385,
+            0.005287723150104284,
+            -0.004992276895791292,
+            -0.03405960276722908,
+            0.0053710537031292915,
+            -0.02725677192211151,
+            0.013128400780260563,
+            -0.002397656673565507,
+            -0.017847958952188492,
+            -0.00030846628942526877,
+            0.026332557201385498,
+            -0.003677921835333109,
+            -0.012537509202957153,
+            0.015544996596872807,
+            -0.006757376715540886,
+            -0.02119634486734867,
+            -0.005420294590294361,
+            0.014658658765256405,
+            -0.0010719380807131529,
+            -0.012393574230372906,
+            -0.029635490849614143,
+            0.012992041185498238,
+            0.0026874206960201263,
+            -0.007386146113276482,
+            0.010893618687987328,
+            0.025953780859708786,
+            0.008151275105774403,
+            -0.005924067925661802,
+            0.0015757111832499504,
+            0.0049581872299313545,
+            0.00300938100554049,
+            0.020287280902266502,
+            0.02728707529604435,
+            -0.008014915511012077,
+            -0.0011145505122840405,
+            -0.014954105019569397,
+            0.015802564099431038,
+            0.0013020449550822377,
+            0.010507266037166119,
+            0.021893294528126717,
+            0.007045246660709381,
+            -0.005738467443734407,
+            0.003905187826603651,
+            -0.023135682567954063,
+            0.01420412678271532,
+            -0.012272365391254425,
+            0.000622140767518431,
+            0.008749742060899734,
+            -0.007283876184374094,
+            0.01437078882008791,
+            -0.0015245762187987566,
+            0.0339989997446537,
+            0.025211378931999207,
+            0.0043218424543738365,
+            0.0007362472242675722,
+            -0.028559764847159386,
+            0.003547244006767869,
+            0.008651260286569595,
+            0.014499573037028313,
+            -0.0076664406806230545,
+            0.006261179223656654,
+            -0.004355932120233774,
+            0.00959820207208395,
+            -0.0065490491688251495,
+            -0.015893470495939255,
+            0.006113456096500158,
+            0.0012689019786193967,
+            0.006795254070311785,
+            0.0023408401757478714,
+            -0.008863375522196293,
+            -0.024362917989492416,
+            -0.025984082370996475,
+            0.021984200924634933,
+            -0.03918066248297691,
+            0.0073293293826282024,
+            0.011234517209231853,
+            0.03324144333600998,
+            -0.008158850483596325,
+            -0.004980913829058409,
+            0.006276329979300499,
+            -0.016711628064513206,
+            -0.004147605039179325,
+            0.008204303681850433,
+            0.012385998852550983,
+            -0.002420383272692561,
+            0.00974971242249012,
+            0.01318900566548109,
+            -0.012378423474729061,
+            0.03660498186945915,
+            0.016514664515852928,
+            0.01467380952090025,
+            -0.004007457755506039,
+            0.01312082540243864,
+            0.0066172294318675995,
+            0.019044892862439156,
+            0.025165924802422523,
+            -0.017454031854867935,
+            -0.002365460619330406,
+            -0.02292356640100479,
+            0.024165954440832138,
+            0.00726114958524704,
+            0.004556683823466301,
+            -0.008310360834002495,
+            0.014870774000883102,
+            0.0017954016802832484,
+            0.0028199926018714905,
+            -0.006696772295981646,
+            -0.010173942893743515,
+            0.019241856411099434,
+            0.0034317169338464737,
+            0.014870774000883102,
+            4.8501173296244815e-05,
+            -0.005503625608980656,
+            0.006052851676940918,
+            0.014832896180450916,
+            -0.023590214550495148,
+            -0.00203592493198812,
+            -0.019484274089336395,
+            -0.03202936053276062,
+            0.009022461250424385,
+            0.0016135887708514929,
+            -0.007052822504192591,
+            0.03230207785964012,
+            -0.010083036497235298,
+            -0.019787294790148735,
+            -0.008067944087088108,
+            -0.013113250024616718,
+            -0.010469388216733932,
+            0.0008271536789834499,
+            0.003908975515514612,
+            -0.00012428610352799296,
+            -0.009317907504737377,
+            -0.005749830510467291,
+            0.01990850456058979,
+            -0.0030358952935785055,
+            -0.00753386877477169,
+            0.00868156272917986
+          ],
+          "embedding": [
+            0.0040707582550429606,
+            -0.010230625151989194,
+            -0.021520242101903186,
+            -0.010115756270640508,
+            0.026738734586159035,
+            0.01796370233223267,
+            -0.012245199029994153,
+            0.06506924494147726,
+            -0.0014087587222033442,
+            0.009300068719271429,
+            0.003787396973047584,
+            0.002818772639754174,
+            0.022129077062176306,
+            -0.021526709808445683,
+            -0.0034388135367225824,
+            0.02763761747879405,
+            0.013210013805395276,
+            0.025255872672296602,
+            0.00029011480909986703,
+            -0.05438974762290585,
+            0.016671640187490865,
+            -0.016426252259893328,
+            -0.056591180651012554,
+            0.009335648425523075,
+            0.006141624459418827,
+            0.02568943956694641,
+            0.019377132054042116,
+            0.017946464337078746,
+            -0.057184776985927686,
+            0.03208527139096687,
+            0.030132392063755547,
+            -0.003846032371791839,
+            -0.008376524427531415,
+            -0.0010513963058713435,
+            -0.02734868388724371,
+            -0.014879400370545479,
+            0.04383920959748825,
+            0.02563613659516289,
+            -0.020117467074438997,
+            0.0398016699155716,
+            0.029896497884197775,
+            -0.0004195844530067313,
+            -0.023302720788253625,
+            -0.001168337916577517,
+            0.009149021369661628,
+            -0.02508145721900992,
+            -0.004307594138480351,
+            0.00550940415564303,
+            0.014016998918718476,
+            -0.028607379616884317,
+            -0.0018317656587526954,
+            -0.01973464940415756,
+            0.03762738392056255,
+            0.015358106058583114,
+            -0.03931907682104936,
+            0.011519637298832444,
+            0.004482080558226775,
+            0.0747867063383242,
+            0.05550679018263077,
+            0.023243129601931176,
+            0.02671631686665238,
+            -0.009686146566573536,
+            -0.04088582189725495,
+            -0.0065344623594924085,
+            0.019617295999325712,
+            -0.004722087402575777,
+            -0.002871844114437583,
+            0.003935087265300519,
+            0.03229890413393201,
+            -0.02813577047904649,
+            -0.05124172132597234,
+            0.03325235407841968,
+            -0.008687655557217192,
+            -0.011027174838246948,
+            -0.003653553063547375,
+            -0.02350737945182988,
+            0.016223087939183468,
+            0.03789252356462714,
+            0.0034350623378603083,
+            0.00195158657888739,
+            0.011789712862232108,
+            -0.0004435439660769551,
+            -0.0037330456918125894,
+            0.0031552713243866858,
+            0.04465432992298031,
+            -0.0032465421571915712,
+            -0.0075366743861922745,
+            -0.059958752731410256,
+            0.016784616861008125,
+            -0.028734506065772967,
+            -0.03035575736273943,
+            0.02680390892611741,
+            0.02338368719038086,
+            -0.0030658375437784277,
+            -0.03754896050794851,
+            0.07005659523248504,
+            0.0020711575553570695,
+            0.013034425083076923,
+            -0.02316832262132332,
+            0.021332142030732894,
+            -0.0318756233020585,
+            0.02270782509507388,
+            0.01570810309454529,
+            0.004788124787612736,
+            0.026727250200142456,
+            -0.022380142778058276,
+            -0.008593910379436904,
+            0.014439754382387034,
+            0.015007697617383174,
+            0.07662531963612977,
+            0.011195998137723795,
+            -0.010809283538470098,
+            -0.042227822120219305,
+            0.003824320933275175,
+            -0.03797629427607094,
+            -0.02347390684453348,
+            -0.0526876440280276,
+            -0.017799861680287913,
+            0.01701482796266917,
+            0.002730395144917155,
+            0.03789451759899825,
+            0.038338545350313946,
+            0.028463789278406586,
+            0.015494583705011921,
+            -0.05700237266350479,
+            0.047895170359075166,
+            -0.0064406894534929406,
+            -0.0007954721295065988,
+            0.012543029839729358,
+            -0.004703334868741216,
+            -0.0060569679284068,
+            0.012324901254711192,
+            -0.019883935809784068,
+            -0.04253565942154215,
+            0.02164409276211115,
+            -0.0075367555415808635,
+            0.03617159147722157,
+            0.07567745988686005,
+            -0.04225977020036783,
+            -0.022548837589622003,
+            0.029811602145850144,
+            -0.02175792196819533,
+            0.02069861458115363,
+            -0.00941331829664397,
+            -0.035479639885943264,
+            0.020352552580119105,
+            0.003678588588550461,
+            0.06776105274336729,
+            -0.02346047721661687,
+            0.02721075076349443,
+            -0.029386781707644253,
+            0.023530831921117993,
+            -0.02077660533100372,
+            0.020022475323372906,
+            0.024195118676513715,
+            -0.028378518720494666,
+            -0.012900969023913307,
+            0.019264089970719624,
+            -0.0374983624629265,
+            -0.019726171557384355,
+            0.00979636085642416,
+            -0.014479528435832063,
+            -0.014533384583777662,
+            0.005904623758767224,
+            0.010344475500263744,
+            0.05659646358026115,
+            0.04163866113482093,
+            0.007152494819169405,
+            -0.07584500130974763,
+            0.05471858597878099,
+            0.03580211013427615,
+            0.044161429444475306,
+            0.02258180566046248,
+            0.00980779221334715,
+            0.0011193505603397395,
+            0.018086566768892507,
+            0.015605066702191871,
+            -0.00954727399396313,
+            -0.027950079146919375,
+            0.03028514025485407,
+            -0.006351669761257431,
+            0.005095804801686436,
+            0.009620040085743162,
+            0.019748001227987134,
+            0.033709535059510916,
+            0.008203559956940196,
+            0.007389752602218563,
+            0.04197771878351742,
+            -0.02067363758471765,
+            0.007543739188471913,
+            0.011207941261799052,
+            0.013964827251172813,
+            0.03839562075248879,
+            0.0009775244543292637,
+            0.006469507611492216,
+            -0.022808235395746654,
+            -0.024795004398633292,
+            0.006249024864814684,
+            0.011292877109066925,
+            0.01596839619826819,
+            -0.0014105396434174883,
+            -0.04152075438277094,
+            0.008267788411079337,
+            -0.028834191212891387,
+            0.030940526405027906,
+            -0.0018271056434014425,
+            0.006185475047820603,
+            -0.007920574674727123,
+            0.012907560753043616,
+            -0.015165704767187252,
+            -0.021962244496450727,
+            0.0037071053072833147,
+            -0.013508376064336042,
+            -0.015178823540513864,
+            -0.02518557053061311,
+            0.011693256916760944,
+            -0.01716015674592331,
+            0.02048897513447166,
+            -0.003412078966361745,
+            -0.001807298511378529,
+            -0.0012575655363779904,
+            0.03116103961636643,
+            0.011668533361641842,
+            -0.011637010548030312,
+            -0.040260195687925236,
+            -0.011129376691522074,
+            -0.0034493180842871908,
+            -0.02098244789711689,
+            0.026658743116016938,
+            -0.01923736325953796,
+            -0.009723937181300192,
+            -0.013134679989565051,
+            -0.011974548914818263,
+            -0.004028712301052159,
+            0.019628329175867303,
+            -0.028192034966932684,
+            -0.012638520793012082,
+            -0.011568652932013111,
+            -0.0025919910519445737,
+            0.012497908463405111,
+            0.015113186712897145,
+            0.014880607940212641,
+            -0.03059227859452334,
+            -0.021899716217105084,
+            -0.0008513506358489523,
+            0.011985931118464679,
+            -0.01321584595585435,
+            -0.011735288366213981,
+            0.010710683541435907,
+            0.01961722172525993,
+            -0.023571946726192585,
+            -0.03364281880125354,
+            0.0014460076743756465,
+            0.016563167828467187,
+            -0.02891589059260944,
+            0.04595166331401113,
+            0.034271553243501465,
+            -0.027557971577291153,
+            -0.007138821520412525,
+            -0.039508407807651844,
+            0.06814329636830853,
+            0.00022136123564675344,
+            -0.039718016705753735,
+            0.0503457594963753,
+            -0.003580581891576583,
+            0.013820796398521455,
+            0.02054050419088757,
+            0.035682661890720495,
+            -0.0013220739026852943,
+            -0.0052759394697328675,
+            0.04733018719162478,
+            -0.0187845855131196,
+            -0.03982699468608443,
+            -0.023905460465070062,
+            0.002631233133039351,
+            -0.03090995554989513,
+            -0.012617612051488022,
+            -0.013605787969473513,
+            0.039672783793764745,
+            0.008669582242097126,
+            0.009974277556736362,
+            0.005535636914843583,
+            0.014180668493093584,
+            0.012042659947850441,
+            0.011870115226306776,
+            0.00451259454348581,
+            -0.014023048592193627,
+            -0.007769033996392623,
+            -0.05715987527668358,
+            0.003374564518384728,
+            -0.028801158523506985,
+            -0.00756331369170698,
+            -0.02163054894055572,
+            0.005367181573233089,
+            0.02308705505078254,
+            -0.005438909398653553,
+            -0.00873620909426313,
+            -0.0036530652237719077,
+            0.013920407174296543,
+            -0.01539613379172909,
+            -0.03908211213269472,
+            -0.016810726548844992,
+            0.007460173854386443,
+            -0.007069189426742819,
+            -0.041197222498539215,
+            -0.003843233633335066,
+            0.004373367480223065,
+            0.016194121660459908,
+            -0.029165150066068787,
+            -0.00735139702658342,
+            0.017257895082337448,
+            0.014153313280536095,
+            -0.002376178042741926,
+            -0.023223700836427403,
+            -0.0319877537205969,
+            0.02088060774052704,
+            0.0012829290529515294,
+            0.03251550040316258,
+            -0.04128453069131847,
+            -0.023376114412063578,
+            0.04083022735076889,
+            0.005690491480259003,
+            -0.05810403800689653,
+            0.016582434816685452,
+            0.010402038964417569,
+            0.016795644054313566,
+            -0.00908590043098552,
+            0.05003656634510042,
+            -0.037216197954986735,
+            -0.013027960261454162,
+            -0.007090943677204207,
+            -0.05175371787508879,
+            -0.0020065660448063322,
+            -0.014908489552053589,
+            0.010627472758306387,
+            -0.03275473161158777,
+            -0.017334608434133866,
+            0.025321047439048318,
+            -0.0033515195870701753,
+            0.011526068468647207,
+            -0.026783879816504975,
+            -0.0012830935419413407,
+            -0.017506908439998053,
+            -0.02114853058440477,
+            0.004878884893669383,
+            -0.020936900837115968,
+            -0.0025587027781733642,
+            0.01557850246191957,
+            0.023652335794299834,
+            0.011489428313303988,
+            0.0019089137782552768,
+            0.02054907941022243,
+            0.014361232983780595,
+            -0.0056413109308506085,
+            -0.0010300050480036684,
+            -0.03413262505887213,
+            -0.018821160485010483,
+            0.03361989907065425,
+            0.013125889701529858,
+            -0.003742167496379789,
+            0.008813100942466033,
+            -0.0383265293638935,
+            -0.035316957263713485,
+            0.03020499323039515,
+            -0.05001623971243151,
+            -0.00855479667673597,
+            -0.005939674940109814,
+            -0.026451831943555135,
+            -0.008032873049527808,
+            0.0009431571685654594,
+            -0.014387605128858954,
+            0.016382559541755014,
+            -0.024909510232038502,
+            0.003715736505444385,
+            -0.006016712902409688,
+            -0.009110197692342108,
+            0.05507263160354303,
+            -0.022103031032598353,
+            -0.0475415595313825,
+            0.021958434235402655,
+            -0.016670809659931243,
+            0.0323519363801999,
+            0.03867494951405658,
+            0.011124508964363016,
+            -0.008002445714431074,
+            0.034041838454818577,
+            0.018400123235267234,
+            -0.01161754789471199,
+            -0.025901844576206774,
+            0.021800298132870245,
+            0.00943488425697727,
+            -0.05906691123008531,
+            -0.00782549274387971,
+            -0.019357446956797447,
+            0.008682376344965976,
+            0.02863077694455766,
+            0.0024925977373894447,
+            -0.002682170968419412,
+            -0.013590235283325244,
+            -0.021511149507762284,
+            -0.03255453916518719,
+            0.009039307015853905,
+            0.008305286476500875,
+            0.009312625027668458,
+            -0.0014144165766983945,
+            -0.007853869334376857,
+            0.009067065760942163,
+            0.022485201611182105,
+            0.03966189080765722,
+            -0.041627289519629344,
+            0.002707086456239785,
+            -0.022678178050044222,
+            -0.03653753142622053,
+            0.0036574646778835588,
+            0.029996160786576367,
+            0.02901163765559891,
+            -0.01971777193183787,
+            -0.0036589934841109884,
+            0.009181234938554578,
+            -0.025979483002227535,
+            -0.00773645103995767,
+            0.004280770859737866,
+            0.04116668301017644,
+            0.006201085381597264,
+            0.01402728258112097,
+            0.001060407317110527,
+            -0.00920801784000087,
+            0.02109880889040528,
+            0.025211022083982727,
+            -0.029866678264942617,
+            0.014439702646820226,
+            0.009379206296908577,
+            0.04156435812161122,
+            -0.037309655469525846,
+            -0.011819731348299495,
+            -0.02766498020409985,
+            -0.010454143230794476,
+            0.027911532296566614,
+            -0.04127890742215716,
+            -0.006948536183086887,
+            0.010207784113909938,
+            -0.0016584043356136647,
+            0.016437495428919636,
+            -0.014607861750686952,
+            -0.03222311919805017,
+            -0.000610723203118213,
+            -0.02867445198803213,
+            -0.024772816792038133,
+            -0.016510309753766612,
+            -0.018581201775667577,
+            -0.07450654873867177,
+            0.00026832248829154255,
+            -0.06724696990816072,
+            -0.02447002541264633,
+            0.04885569249258513,
+            -0.010195604207134053,
+            0.03332905935704749,
+            -0.02815272112271314,
+            -0.005296340304911926,
+            -0.05908835042526019,
+            0.0038476172058970774,
+            0.025753576863447866,
+            0.02386003857017594,
+            0.005206447371391696,
+            0.024799057871126032,
+            0.012060738268265606,
+            -0.04146503002980051,
+            -0.024949907556862643,
+            -0.014503475223770958,
+            -0.0016340466065744286,
+            -0.0034009167013795906,
+            -0.031298432995411005,
+            0.036241272149880994,
+            0.008902644391348814,
+            0.03550223922667804,
+            0.04463301409716232,
+            -0.010385984413759852,
+            -0.03305004479734889,
+            -0.006952739338820577,
+            -0.0175981588092108,
+            0.042668533202572856,
+            -0.017112184620086308,
+            0.005149173014526686,
+            -0.03219589107997635,
+            0.024181808150698054,
+            0.0014339790422945406,
+            0.01254721190611376,
+            0.011099458649404119,
+            0.0027626836767364253,
+            -0.026883640446090158,
+            0.0024160770766463607,
+            0.023874514847956222,
+            -0.022558433617669373,
+            0.00917667715473093,
+            -0.006754091166465801,
+            -0.033686935718776144,
+            -0.01078389638118685,
+            -0.013595327486167114,
+            0.024712781935748277,
+            0.013168464807451798,
+            0.005875864604862434,
+            -0.02417814420558463,
+            -0.0023978800459714144,
+            0.028650463366555304,
+            0.020176980768301288,
+            -0.02118184201861616,
+            0.019082340279671355,
+            0.019734925740100277,
+            -0.04209645071102001,
+            -0.04217310001316092,
+            0.006553913719098956,
+            -0.003735719227468833,
+            0.0061793909220459,
+            -0.028285889794908054,
+            0.008181788076484304,
+            0.004432488437404309,
+            -0.01743905248466261,
+            0.041221269621866216,
+            -0.014892354419172107,
+            0.005087102208335201,
+            -0.022198383716186407,
+            -0.011936388003605221,
+            -0.00027487042024290816,
+            0.044599215801158845,
+            0.008068975171827777,
+            0.003496950921841073,
+            0.02238779276447619,
+            0.0054839808262270315,
+            0.04816723616939378,
+            -0.02685221260466647,
+            -0.0190356946337995,
+            0.016464136459560113,
+            -0.0176092780782677,
+            -0.012513274095432353,
+            -0.010695421445334002,
+            -0.004784348908457072,
+            -0.022203041334932165,
+            0.010788155380415583,
+            -0.018353897272137543,
+            -0.02277431189907027,
+            -0.003945465850796918,
+            0.016024639792926797,
+            -0.006248368166794934,
+            -0.0038769803892794875,
+            0.0016292556540434277,
+            0.02412864830338344,
+            0.02892955453819876,
+            -0.02408022470555968,
+            0.019190282372105023,
+            0.018941584034503536,
+            -0.006709516380602965,
+            -0.020956377933747532,
+            -0.01964885206488053,
+            0.01434623424964373,
+            -0.01352968586212152,
+            -0.013872556892707697,
+            0.004052798542613835,
+            0.009201147374299997,
+            -0.004086067985859617,
+            -0.04185829685685201,
+            -0.014668698123513235,
+            0.003179403563664022,
+            -0.0006415014901450319,
+            0.019360184344240947,
+            -0.0150817538567364,
+            -0.029771999278536083,
+            0.014221776097046751,
+            0.022944884879589678,
+            -0.04493425220030206,
+            0.02117577719533236,
+            -0.0030648751497261983,
+            -0.00016251533643103604,
+            -3.817296018601414e-05,
+            0.008727767382077138,
+            0.026914779443266063,
+            0.03456565094655899,
+            0.01974806772696219,
+            0.00450800007152441,
+            -0.008952599700759673,
+            0.025184190461155245,
+            0.009846044083064704,
+            -0.025168735183134738,
+            0.010983353565382029,
+            0.016740721743326522,
+            0.0070838930012654296,
+            0.009173879407277282,
+            -0.006078288608766152,
+            -0.018269347893097643,
+            0.013181494188066667,
+            -0.01435693747022231,
+            0.01678907949532146,
+            0.05154463552784974,
+            -0.00039632596424461964,
+            -0.02461439550772063,
+            0.008418147037164186,
+            -0.006544319842335593,
+            -0.010065129286193278,
+            0.02214628550460128,
+            0.0029515574751370417,
+            -0.030862152033847717,
+            0.007509556025223583,
+            0.00675485304374519,
+            0.01894936231559203,
+            -0.02355525295639599,
+            -0.01387885858544803,
+            -0.00431398020087544,
+            0.0021208056468763523,
+            0.01580766504149989,
+            -0.004739477121073512,
+            -0.013204823447960706,
+            0.02994098229517376,
+            0.025496728234662373,
+            -0.00026124946738213036,
+            0.015060064980768175,
+            0.022262345898533903,
+            0.009968408664790368,
+            0.007588431141088797,
+            -0.0083063413609779,
+            -0.031336272171937095,
+            0.0018957858383561667,
+            0.05800811417728381,
+            -0.014937085296865802,
+            0.04313875432235027,
+            0.02018476425923236,
+            0.015657644644987107,
+            0.017880263053253607,
+            -0.01662268082870562,
+            -0.008740053682875323,
+            -0.0017630394204179638,
+            -0.008294417990024647,
+            -0.027613464270562948,
+            -0.005953209556351086,
+            -0.02379961210881074,
+            -0.013878676554701543,
+            0.015418315245979704,
+            -0.014764728065382074,
+            0.0027420640870858457,
+            0.010395538988699663,
+            -0.006590633833835581,
+            -0.00841537209927409,
+            -0.003943772700797381,
+            0.03376338017637296,
+            0.011980354721458463,
+            0.027761929822019422,
+            -0.00011128539412582973,
+            0.009342430952107655,
+            -0.018419526564396063,
+            -0.006839752805318664,
+            0.03271190887864607,
+            0.006011520427944665,
+            0.004652835471939905,
+            0.01485920920301398,
+            0.014305626381398871,
+            0.038015677219487023,
+            -0.005102005401565164,
+            -0.018204460895989052,
+            0.02065495133597456,
+            -0.0049699013356374775,
+            0.019149036693562712,
+            0.008825981577388056,
+            -0.002660055848523568,
+            -0.004292379561953203,
+            0.03090270710860331,
+            -0.020243440437217503,
+            0.018735727616321387,
+            -0.02238236089688497,
+            -0.0276584413929402,
+            0.023028408906176946,
+            0.008320436733620496,
+            0.04047641118026783,
+            0.011463851066078041,
+            -0.02247143742166039,
+            -0.034588725413285185,
+            0.009827713930789184,
+            0.00337071450636392,
+            -0.024446280199879062,
+            0.017542777160283324,
+            0.00780836483490212,
+            0.008453886667283516,
+            0.016624116261219637,
+            -0.004175682824655864,
+            -0.0042784362933758835,
+            -0.001546656717896957,
+            -0.008819739495525477,
+            0.01406790007187259,
+            -0.015487658289541583,
+            -0.021457182907140035,
+            0.024687169583245905,
+            -0.03915911962859764,
+            0.01771290139419629,
+            -0.006510046907173026,
+            -0.00997450880725535,
+            -0.02059560682587115,
+            1.5939836949198004e-05,
+            -0.007159493752862143,
+            -0.0004782363814453761,
+            -0.017503181957042913,
+            0.016643971600145645,
+            0.024056442566869802,
+            -0.013209211443909658,
+            -0.00445521990927108,
+            0.0020313626821877526,
+            -0.029235692455820056,
+            0.006662209097255912,
+            0.016993027053110635,
+            0.00529940612845993,
+            0.0026104743851393716,
+            0.0001707934714983891,
+            -0.031289446611163674,
+            -0.005782208620284939,
+            0.03053516889069683,
+            0.05590704747438521,
+            0.021628254147289437,
+            0.02403314981042378,
+            0.02908606663381421,
+            0.02717206823892633,
+            -0.003198844932040613,
+            0.0016217508103442348,
+            -0.01620660606431983,
+            0.027098323143010743,
+            -0.011664168290676943,
+            0.04322175021589083,
+            0.008652323184522541,
+            -0.049493020736445056,
+            0.004537958157696794,
+            0.00943239454147692,
+            0.034918603888916544,
+            0.016678716235690318,
+            0.0005887414054803805,
+            -0.005894338664998101,
+            0.03279165663336323,
+            -0.02735726709032382,
+            -0.011962070182991011,
+            -0.00036014407560325284,
+            -0.019079003730607717,
+            -0.015938295169860884,
+            0.010357611943935459,
+            0.04278924678845588,
+            0.0009957483598272451,
+            -0.0071422790468850215,
+            0.01571201177743999,
+            -0.009445307674383,
+            0.02286338492075306,
+            0.007336440384050498,
+            0.0042809497342459904,
+            0.025581467690095977,
+            -0.0025007871391868105,
+            0.02300539823997411,
+            -0.000628898771317254,
+            0.004582863909134464,
+            -0.015960578501883316,
+            -0.00930649396114688,
+            0.027838455754803444,
+            -0.011112268986721,
+            -0.029253746504396393,
+            -0.008734785561831333,
+            0.00472579547270089,
+            -0.0003805825712986138,
+            0.0021854358149324227,
+            -0.01183885533661493,
+            -0.010840951201591629,
+            0.019965691788904057,
+            -0.004922943301424112,
+            0.003800048754664761,
+            0.012664049429916252,
+            -0.0030124219674414897,
+            -0.01910844996750822,
+            0.01889983984736845,
+            -0.003858283778519141,
+            -0.024168821338992813,
+            0.020083955711766945,
+            0.01850497889545421,
+            0.045156567155119454,
+            -0.01586824055785609,
+            0.004612757419198795,
+            0.017497774574602505,
+            -0.002172088240279357,
+            -0.019617327081566827,
+            -0.014944795772093997,
+            -0.018212462639041492,
+            -0.015975283672064702,
+            0.012837956594691596,
+            -0.01243789877725244,
+            0.019500571353639314,
+            -0.0380735226388175,
+            0.012590900534826689,
+            0.004832168601136412,
+            0.007505385730121269,
+            0.016011983211212992,
+            0.025994560026860358,
+            0.002157436199771672,
+            -0.033465476168065784,
+            -0.01014599891204166,
+            0.007845373376916211,
+            0.010093432683262838,
+            -0.002849185178237036,
+            -0.018147004183434554,
+            0.003633790611809425,
+            -0.0266147420637404,
+            -0.018644422659309035,
+            0.022926737299761104,
+            -0.04798772015599113,
+            0.017908415301694906,
+            0.024859125158586466,
+            -0.04343699525707025,
+            -0.04886699607556352,
+            -0.011459188432595229,
+            -0.036774381316312464,
+            0.0014128005215813578,
+            -0.0005256067584181517,
+            0.011194992852264102,
+            -0.009342140079829378,
+            -0.018598671981710183,
+            0.01584789414301952,
+            -0.0036412641242200418,
+            -0.004225840239935549,
+            -0.00616696736018045,
+            -0.0025598631463178384,
+            -0.010188803157718543,
+            -0.03244694662411977,
+            -0.03290891136780909,
+            -0.00033742479296272127,
+            -0.009743927051414455,
+            -0.02334288359945272,
+            -0.03447062782872289,
+            -0.008728968107967028,
+            -0.037753659749578956,
+            -0.00515561961442376,
+            0.04070890150714597,
+            0.027517147608335957,
+            0.009449414978133334,
+            0.008086900089929912,
+            -0.013989034604490105,
+            0.05691047488419298,
+            -0.020425883381389086,
+            0.039466620642512155,
+            0.005301493569940605,
+            -0.0037222353899918703,
+            0.013075425214878277,
+            0.027710826422351564,
+            -0.006253665106995653,
+            -0.0017412928073238833,
+            -0.01928096630475438,
+            -0.012920476859996416,
+            0.009444408011467955,
+            0.015799884767184885,
+            -0.0025083061619197524,
+            0.001274457251739847,
+            0.00930533161700852,
+            -0.0006652478540264343,
+            0.017258038491335247,
+            0.03996268817486439,
+            0.003833844612473671,
+            -0.00826404141706295,
+            -0.018822002570963614,
+            -0.003188844900317347,
+            -0.011807006674248829,
+            0.010482058734260632,
+            -0.015410635475145028,
+            -0.011540803927522013,
+            -0.01249428777269578,
+            -0.017081856786754605,
+            -0.002183034976477535,
+            0.003114834030705039,
+            0.0027475411722881747,
+            0.02261627635192778,
+            -0.015872027027080824,
+            -0.022487581027490516,
+            -0.0035823117885461506,
+            -0.02398778449664045,
+            8.13716819067491e-05,
+            -0.01050060782407259,
+            -0.015616181283381218,
+            0.013676480044963827,
+            -0.021513173575512936,
+            -0.01091399089076612,
+            -0.008699176177861256,
+            -0.0006986953850759474,
+            0.005853723710671249,
+            -0.00993355748621621,
+            -0.00010241843582159751,
+            0.01150872694850241,
+            0.005559552266827916,
+            -0.012336449703762907,
+            0.0039431718621504,
+            -0.012740550272925562,
+            -0.004432059174951158,
+            -0.03592309122711106,
+            -0.019665686718735042,
+            -0.01846795637714379,
+            0.005642094448288269,
+            -0.005108964028543505,
+            -0.020586678054151854,
+            0.00668364061048941,
+            0.0034775127050453774,
+            0.004713169526661792,
+            -0.010723667875209928,
+            0.013944735241748791,
+            0.010882420168586074,
+            -0.003518245916227596,
+            0.013827334494230662,
+            -0.001330399936559009,
+            -0.020286697504699764,
+            -0.011139922402071095,
+            0.03538258948179278,
+            -0.024961826464903997,
+            -0.0018019328351763667,
+            -0.0026423695690168066,
+            0.00597003516870726,
+            -0.03071835946501977,
+            -0.013634212232744325,
+            0.02164400349828422,
+            -0.005846552894831441,
+            -0.026367014426434226,
+            0.03399116480234033,
+            0.03473258231629753,
+            0.010041605592929843,
+            -0.013423384340589794,
+            0.04730054254696893,
+            -0.012968978954938308,
+            -0.018274800841400665,
+            0.026826644936918647,
+            0.0278389611094493,
+            0.02525164095926168,
+            -0.029782550447890466,
+            -0.003010713086487275,
+            -0.02393629084357885,
+            0.017993366287762866,
+            -0.0263476663180971,
+            0.01864011499179424,
+            -0.02332263152537849,
+            0.016014908881756034,
+            0.01868464825773604,
+            -0.003104041044182249,
+            0.026313491086684425,
+            -0.0052945248919367815,
+            0.015253731232325074,
+            0.0012371308759955573,
+            -0.02119765517013431,
+            0.007795466385020642,
+            -0.006469500272298062,
+            0.025762169721880456,
+            0.00833992768202466,
+            0.01801901639978976,
+            0.022019940825351116,
+            -0.0409522298213741,
+            0.02003698451081499,
+            -0.001367916864679543,
+            0.01782984895971748,
+            0.0038516134262842373,
+            -0.007885432294782304,
+            0.014221728513400065,
+            0.010991352188016265,
+            -0.012851099261924757,
+            0.003840539873231741,
+            0.0008923482291826014,
+            -0.028928104542716937,
+            0.007066004768771198,
+            -0.026914574065060926,
+            0.025866907400807087,
+            -0.005526774784152086,
+            -0.00998122210956923,
+            -0.020744532673097317,
+            -0.0071425207262770865,
+            0.003754468896273156,
+            -0.012747874997429746,
+            -0.011385805601999295,
+            -0.024700457669046855,
+            -0.02401291382882952,
+            0.029591582802488962,
+            -0.014054332096344003,
+            -0.04015978298687865,
+            0.001234440863385445,
+            -0.023301255402129767,
+            0.01193125007773239,
+            -0.005793124129267758,
+            -0.024440100649464773,
+            -0.042563399082257154,
+            -0.017548888269389457,
+            -0.0009394887961485922,
+            0.022739763644266633,
+            -0.00114156949394707,
+            0.041791395064755515,
+            0.0008816696205437911,
+            0.022207132857533028,
+            -0.015547580543606755,
+            0.005319731264840819,
+            -0.02071229859314312,
+            -0.041495579742912285,
+            0.03660432378985971,
+            -0.014637031447181937,
+            0.025854844497435178,
+            0.03324325296704186,
+            -0.013009347371118843,
+            0.001354504885302625,
+            0.02652570709474851,
+            -0.014618100923440427,
+            -0.010782232747863238,
+            -0.0034936699515378212,
+            0.003461670978569965,
+            -0.006468116285021298,
+            -0.027793221545159466,
+            -0.007752458229193391,
+            0.04193706715422074,
+            0.017894069367616637,
+            0.021928994684423043,
+            0.022606401934291136,
+            -0.017972404868085358,
+            -0.026402048654300563,
+            0.003656646576244765,
+            -0.013129722990153284,
+            -0.018090360146368677,
+            -0.01624598856366761,
+            0.012397150789705746,
+            0.011741002980073786,
+            -0.004052422390675261,
+            -0.01836750423615859,
+            -0.020929335029248472,
+            -0.016327412417122416,
+            -0.0042615345226698015,
+            -0.008458117095535,
+            -0.005002582649405199,
+            -0.012158173174049797,
+            -0.020409616620022143,
+            0.03706368540614453,
+            0.0285565642978937,
+            -0.0041732933884846566,
+            -0.00607154864704767,
+            -0.011142150176334957,
+            0.008834431806990914,
+            0.006021232046087652,
+            -0.005028870471297621,
+            -2.3396043292139426e-05,
+            0.005112033372713154,
+            0.022822312435625314,
+            0.007271235299348001,
+            0.0035791789459533856,
+            -0.06340305811365254,
+            -0.005392000517722054,
+            0.022560201096078862,
+            0.012581401789917604,
+            -0.0032248219865223496,
+            0.025180718735801837,
+            0.0006451724964970927,
+            -0.009714514747463593,
+            -0.018389422219956937,
+            0.024728862150150004,
+            0.008760329201960964,
+            0.020988700992999768,
+            -0.014974763313686516,
+            0.003845297486923296,
+            0.0006445983847138896,
+            0.00750394530065325,
+            -0.0019792992847026793,
+            -0.008578254004439607,
+            0.025776677078006516,
+            0.016593438298233645,
+            -0.005695578040595484,
+            0.01604018433477423,
+            -0.0017497260488899657,
+            -0.021460543443428924,
+            -0.010265213095609554,
+            0.00042017209251926397,
+            -0.010570864288876387,
+            -0.018015790888295944,
+            0.017569414447505878,
+            -0.002601036282535863,
+            0.0072399377790164435,
+            0.030281707151249355,
+            0.0423183067569767,
+            0.009566493398534713,
+            -0.005928603293907002,
+            -0.0037936911251953654,
+            0.015477873304924939,
+            0.01513127995244932,
+            0.010741256034652754,
+            0.03015858543221727,
+            -0.029247492875021843,
+            -0.006690306960497428,
+            0.0046653727410352576,
+            0.004610405633429778,
+            -0.011692179099382171,
+            0.030055897497741527,
+            0.04569730592994177,
+            0.006736813244286893,
+            -0.004799978513117371,
+            0.002121067028052434,
+            0.011299889107780231,
+            0.007768137218140627,
+            -0.018927080714961783,
+            -0.0012222962555201746,
+            -0.006009911026251445,
+            -0.014001766533604134,
+            0.026941585318261085,
+            -0.003960837039351648,
+            -0.011848232223811548,
+            0.02903357274500332,
+            0.1105267435019781,
+            -0.006011029338781496,
+            -0.0053646998281195206,
+            -0.03242321704621517,
+            -0.015194102811332289,
+            -0.008860590919147765,
+            0.0056988205877963076,
+            0.01635957585050017,
+            0.020327661913559888,
+            -0.0008943408274661007,
+            0.016820860249955754,
+            0.02822175614376368,
+            -0.013854827839129885,
+            0.007906244895327178,
+            -0.030891381690403466,
+            0.015702117924362928,
+            -0.005383974926093977,
+            -0.012032955400313954,
+            -0.0026169511828034054,
+            -0.008618307035330928,
+            -0.0037356680273096105,
+            0.0009666430093455632,
+            0.011296307527684261,
+            0.004427516814276546,
+            -0.014484198313369681,
+            0.002829529156309784,
+            -0.008595300985863424,
+            -0.005122996529848575,
+            -0.02393571073523279,
+            -0.024119245543788293,
+            0.005223147047205755,
+            0.0019389305548629142,
+            6.344308292376579e-05,
+            0.006848965054539454,
+            -0.003810712095343454,
+            0.030149081317333418,
+            -0.01700322008272963,
+            0.0012288912810468446,
+            0.01193226302726573,
+            0.012332749933413344,
+            0.01688602002694633,
+            0.0028600984823761366,
+            -1.9882692552530084e-06,
+            0.021172977047420227,
+            -0.010514036509402561,
+            0.01460334896316005,
+            0.013091417303145243,
+            -0.04103572381458182,
+            0.0089746401463863,
+            -0.0027841289484449873,
+            -0.024252447218794344,
+            0.004446472209246425,
+            -0.018987867343500102,
+            -0.00881974688349601,
+            0.0005365625696166774,
+            0.012054719659248512,
+            0.01960300421041832,
+            -0.008553943699846696,
+            0.02111201386480852,
+            -0.0003501657740808554,
+            0.004177088753111459,
+            -0.012879944670340026,
+            0.0065222126975051685,
+            -0.007563354540402343,
+            0.0008168128607271199,
+            -0.018213241937597355,
+            0.012526561612175514,
+            -0.0008285725179894544,
+            -0.015737003159436623,
+            -0.011337688275309921,
+            0.004737346137976519,
+            -0.0027993545140114705,
+            0.0030843163114700313,
+            0.010153740340626154,
+            -0.008815302484351509,
+            0.004919275910399053,
+            -0.0017541042538165297,
+            0.002562816687714991,
+            0.014285249186421434,
+            -0.029816964593980453,
+            0.005357227738099325,
+            0.009154842495811638,
+            -0.00048782303639333943,
+            -0.013074976152495576,
+            -0.004054505559072866,
+            0.0028322102449485015,
+            -0.016877328079971877,
+            -0.012128199639779175,
+            0.023421330164419888,
+            -0.012565145848820333,
+            0.008609070188894315,
+            -0.010936598722317274,
+            -0.018401801734378692,
+            -0.01066119413133089,
+            -0.005254229780560808,
+            0.01234969508270852,
+            -0.0038427767204741877,
+            -0.006947594635952202,
+            -0.01756894543880097,
+            -0.007890897373987378,
+            -0.047246698568222466,
+            0.009617876521097624,
+            0.027157555263862723,
+            -0.011355758309667893,
+            -0.04050311381480088,
+            -0.01017774338301804,
+            0.02437257208967103,
+            -0.004056329295095018,
+            -0.008345450168215274,
+            -0.001599171409135998,
+            0.018358386463243014,
+            0.0027207445181251406,
+            -0.007292967749349572,
+            -0.011267861428599221,
+            0.01657507035043879,
+            0.015105061280816214,
+            0.021419854579631276,
+            -0.02088952706825817,
+            0.006295984081511286,
+            0.020973020776936996,
+            -0.03073246543555915,
+            0.0023720435876790876,
+            0.03363966216709148,
+            0.010412588613998134,
+            0.013023311559810034,
+            -0.0010217662933572685,
+            -0.0075567381078977215,
+            -0.008248818794295818,
+            0.011848200323549865,
+            0.005066463524567732,
+            -0.023174489297821385,
+            0.021823825355858556,
+            0.020368868756448753,
+            0.0026042856858509524,
+            0.003457067317891926,
+            0.0005539745115410357,
+            0.01062670844108589,
+            0.009948889803890108,
+            0.05100741556350918,
+            -0.006493714124814164,
+            -0.014021751055930108,
+            0.015063988102064609,
+            0.02488059237697814,
+            -0.0023499941119596228,
+            0.01087282522970853,
+            -0.02612218935931224,
+            -0.021423880914461062,
+            0.023389294741183817,
+            0.008967763132667892,
+            0.009637806238448432,
+            -0.010815706635624337,
+            -0.011390999462441815,
+            0.031750675158698435,
+            -0.01307829462211914,
+            0.00759749911213318,
+            -0.02169895957209634,
+            0.01387879409130081,
+            -0.008606362268863441,
+            0.032229658009209815,
+            -0.0009066257436946125,
+            0.011062436424260079,
+            0.028010486866468023,
+            -0.007835805879241447,
+            -0.019657486763625547,
+            0.003967480939987674,
+            -0.015577952617624695,
+            0.02188429826317828,
+            -0.020502758155233298,
+            -0.0002362500702658591,
+            0.0040935406015014265,
+            -0.010056462958039452,
+            -0.0057038083589231615,
+            0.011475170331848666,
+            0.0015345076181482267,
+            0.006359262977587827,
+            -0.020293014401106,
+            0.01369706440475074,
+            -0.003989482215141903,
+            0.007887824197823247,
+            -0.015925992045349828,
+            0.014370223626942766,
+            -0.02309139937510304,
+            -0.021808736033141787,
+            0.008142621179624918,
+            0.0108996933426044,
+            -0.007589050511458233,
+            -0.025971539510030145,
+            0.02250992080022228,
+            -0.024980833805349545,
+            0.024979008145354243,
+            -0.006622339940663538,
+            0.002566378488088469,
+            0.004913874740363343,
+            -0.008102103508732738,
+            -0.0028098905427027783,
+            0.0057865587191235565,
+            -0.0073509382400023995,
+            -0.014123756810537343,
+            -0.019453380788786282,
+            -0.04083629135586275,
+            -0.03177927047231389,
+            0.022927581786088044,
+            0.011557232599801421,
+            0.0017073832658476193,
+            0.02339552660545296,
+            -0.02247471449131386,
+            -0.017890866287500053,
+            -0.009775068417716798,
+            0.025184899522229628,
+            -0.04408541669037361,
+            0.012104255392615549,
+            -0.014669458760022245,
+            -0.0245258242481677,
+            0.006274252098230249,
+            -0.004161960147481513,
+            0.009299171724026455,
+            -0.010779421730862708,
+            -0.04713627671791054,
+            -0.005192684784377356,
+            0.02330975631183273,
+            -0.0036802876364168935,
+            -0.006197548344287773,
+            0.03048284650641829,
+            -0.0019503797069087845,
+            -0.005038254331399784,
+            -0.005524875459245342,
+            0.027783596017224226,
+            -0.024950083604372984,
+            0.010988398531346805,
+            0.008304937912856357,
+            -0.008485065895193593,
+            0.006572665697403556,
+            0.0011518846966102158,
+            0.011927348472282199,
+            0.015632588644277183,
+            -0.015559413716826264,
+            0.02262990861928957,
+            0.011446850286896198,
+            0.006506075777765303,
+            0.008015225053785824,
+            0.00010054430757843335,
+            0.0022975742216194894,
+            -0.024948925449668525,
+            0.026047924967851216,
+            0.01888211557921096,
+            0.0001632181017876392,
+            0.012553094476616919,
+            -0.015042621225242777,
+            -0.02782009986290425,
+            0.02606646297992026,
+            -0.016847653820092335,
+            -0.00045272379465939274,
+            0.020053091486344633,
+            -0.016210036358455773,
+            -0.007930550661303182,
+            -0.01123514849773852,
+            -0.0162675150307978,
+            0.01803076559921804,
+            -0.012160217543572069,
+            -0.02216594536057653,
+            0.002594675636753233,
+            0.016443123828746582,
+            0.02134480656850119,
+            0.026694575953313482,
+            -0.011470930697351533,
+            0.017768927048191052,
+            0.0008653408391607816,
+            0.001865765599795092,
+            0.025745954171649543,
+            0.01702677011968492,
+            -0.008474889109168202,
+            0.025347176686722313,
+            -0.0016686223698600755,
+            -0.002421921060670558,
+            -0.012265504084322075,
+            -0.0165214436458303,
+            -0.007545471138114208,
+            -0.015333235556146196,
+            -0.0051830106590765685,
+            0.000762746782664525,
+            -0.0018118022696726346,
+            0.10876576838038368,
+            0.0034948044540582626,
+            -0.011608460689239234,
+            -0.005870263500417206,
+            0.007510774308250375,
+            0.0023525201199670765,
+            -0.006846154764389575,
+            0.016082721103476375,
+            -0.018616626204086575,
+            -0.023593744079394,
+            0.004743481974389998,
+            -0.0009262286319469741,
+            0.00870122878944883,
+            0.02029523251993269,
+            -0.008608837818169835,
+            -0.0200567711498164,
+            0.007089477722191928,
+            0.005535149644041221,
+            0.0023640746516071774,
+            0.005256431824732927,
+            -0.0006118671321824933,
+            0.008209920240361308,
+            0.02448422122961942,
+            -0.015622892878860465,
+            -0.029770836607612903,
+            0.010535371601298532,
+            -0.00015994156470228762,
+            -0.007193425789067776,
+            -0.011046316463797128,
+            0.014728841492009627,
+            -0.0011488451990980268,
+            -0.02751404133172324,
+            0.017663132712536528,
+            -0.019055335178900415,
+            0.014643246200933506,
+            0.0025348345163347464,
+            -0.00016387173410118788,
+            0.022565953069005714,
+            0.013101158251834834,
+            -0.04272085076882881,
+            -0.0034442020309914823,
+            0.01041205537746844,
+            0.011059422927946552,
+            0.0116446016862942,
+            0.0012992535381387887,
+            -0.018670801241345514,
+            -0.011788984490283379,
+            -0.009109849103208542,
+            -0.021268488414296117,
+            -0.012523033785938503,
+            -0.011393853408628214,
+            0.010743646386439633,
+            0.0012964030814325153,
+            0.023809566099779113,
+            -0.003030535542277875,
+            -0.006526351867141901,
+            -0.000430699447948508,
+            0.00880561288237257,
+            0.009694646009259077,
+            -0.02079011138439714,
+            0.008294338719512953,
+            -0.01823922789220214,
+            0.011819769685519291,
+            0.006416044253693268,
+            -0.022799568832306067,
+            -0.009911619276012527,
+            0.007152786695893951,
+            0.008393677195358034,
+            -0.013569953300860946,
+            -0.006503988731872938,
+            0.02299498754558268,
+            0.018870619152710703,
+            0.012233964644738905,
+            0.024994616203196145,
+            0.03838969722235787,
+            0.005744255912479976,
+            0.027682123067526515,
+            -0.016791471092091604,
+            0.008459745383680297,
+            -0.022493868174622695,
+            0.012902262638928338,
+            -0.00460399519266689,
+            -0.0050431343566427886,
+            -0.018188192498411616,
+            0.033765105245412236,
+            -0.009159751849801773,
+            -0.0027150644587327763,
+            0.034778224828120714,
+            -0.0066502021793471075,
+            0.004525230911930458,
+            0.005096748531775903,
+            -0.016067049494073893,
+            0.033274672789154185,
+            0.01686831040872425,
+            0.016353966592947262,
+            -0.021631941883929817,
+            0.011748317746828848,
+            -0.005675701689448899,
+            0.0001273442591116645,
+            -0.020738819037390263,
+            -0.0037528323797385733,
+            0.0027725541343763104,
+            0.0059860509282538546,
+            -0.016556134518370437,
+            0.02033623476757896,
+            0.00019647023567641008,
+            0.0052895744489259875,
+            0.03253462247073863,
+            0.01043311189804924,
+            0.019182938878855655,
+            0.010051051068017807,
+            0.019523739618278902,
+            0.0281939391326836,
+            0.025685787482623124,
+            0.0029932688462254147,
+            -0.018668945726139946,
+            -0.0009326901784812825,
+            -0.007880352863625262,
+            0.010492063055849727,
+            0.022849902214818293,
+            -0.022619228461640748,
+            -0.02772644874135741,
+            0.018566981184526675,
+            -0.0022608185345048414,
+            0.020031103834122595,
+            0.007544464625538984,
+            -0.020935181884939796,
+            0.003407355515103286,
+            -0.023178230166337178,
+            -0.0005418155871620593,
+            -0.031039187720144967,
+            -0.028591174077243604,
+            -0.0007659241904525392,
+            0.011945344534674635,
+            -0.01688153017778311,
+            0.03357788163306365,
+            0.017667778671101954,
+            -0.014289109343082812,
+            0.01604684528817524,
+            -0.011054587325539086,
+            -0.0036959091138078683,
+            -0.005819853870004682,
+            -0.00203755010992417,
+            -0.02157944474979592,
+            0.005138614763345542,
+            -0.0033509138150859593,
+            -0.0024871060745482883,
+            -0.016936575573598154,
+            -0.021901830366872595,
+            0.016375173843894617,
+            0.016742770709641697,
+            -0.006606252542838758,
+            0.039771905404651496,
+            -0.004238071340064403,
+            0.00018364315349545657,
+            0.011690521707403059,
+            0.0010018392078563795,
+            -0.013651921068215563,
+            -0.010829994512513501,
+            0.012362647379788833,
+            0.011321403447882543,
+            -0.018942043512252663,
+            0.005763773608775982,
+            -0.009131565038297889,
+            0.002089216671850269,
+            0.005438887787329238,
+            -0.008183835275629478,
+            0.006690280329947951,
+            -0.01673229763262922,
+            0.015833397876364723,
+            0.00295474085300386,
+            0.003186554859191664,
+            -0.011797859525926772,
+            -0.020991927615680524,
+            0.01576413532776261,
+            0.0032403514518527287,
+            0.018819824905201635,
+            -0.026755218746060662,
+            -0.00012946467986739473,
+            -0.009552681500884924,
+            -0.0028941357661144826,
+            -0.012533800273548772,
+            0.013975640628890588,
+            0.016864739319953755,
+            0.0063941926578529,
+            0.01781016436903586,
+            -0.0026268224264609724,
+            0.015368284773773678,
+            -0.018122416427266218,
+            0.008686009180697955,
+            -0.006801074032721299,
+            -0.018537521789181048,
+            0.0013237338264105415,
+            0.027863741654435434,
+            -0.004670012662973029,
+            0.011066853115455966,
+            0.011704884649534624,
+            0.018954056680143452,
+            -0.0012626215734593707,
+            -0.0030850559460101866,
+            0.015610404665171367,
+            -0.005838226405658572,
+            -0.006861883833171456,
+            0.020872966233916948,
+            0.032951653763679145,
+            -0.006716215648910268,
+            -0.0021292599951407863,
+            0.010886596327481677,
+            -0.004901684487088035,
+            -0.008644584974417106,
+            -0.009587037829603014,
+            -0.03158137900717491,
+            0.024369089446706894,
+            0.012092412439477474,
+            -0.009601042951586109,
+            0.022690862146548786,
+            0.020752792552278052,
+            0.011412743968296708,
+            -0.010471860708195726,
+            0.02291536038589634,
+            -0.01268035142112569,
+            0.024168869865226136,
+            0.032482831775271157,
+            0.004400973556358811,
+            0.0022312906286559167,
+            0.0021175489661843985,
+            0.01691348460546639,
+            0.006050357346231155,
+            -0.007068133214675996,
+            0.011656003476409986,
+            -0.018487923180009552,
+            0.01598530891381001,
+            -0.004432366596808033,
+            0.02808371007118948,
+            -0.01337145216499331,
+            -0.010473700033368562,
+            -0.00537325376270318,
+            0.00010726431372328729,
+            0.006445701087575359,
+            -0.003324002570256215,
+            0.005255443653813165,
+            0.0007072577303252132,
+            -0.017825649146944237,
+            0.003050128130393781,
+            0.004825967170196209,
+            0.009018799597337101,
+            0.004103653497834134,
+            0.007249957401824204,
+            -0.012360355929483,
+            0.01441885664789707,
+            0.004401820363035808,
+            -0.0026190938717927086,
+            -0.0007163225760398784,
+            0.011124538295057815,
+            -0.013232822471919533,
+            -0.004895624967029031,
+            0.005281228035321008,
+            0.0059275753353475414,
+            -0.007507924998741734,
+            -0.004484140989405051,
+            0.011908076549392969,
+            -0.0037917291321648483,
+            0.006351543504726163,
+            -0.004409464890606014,
+            -0.0037760571637986827,
+            0.005303661313434895,
+            -0.00974511544836468,
+            -0.004204167898342997,
+            -0.0007093125445559502,
+            -0.006869250680233214,
+            0.00786202375969793,
+            0.005971033227117194,
+            0.010606085272785863,
+            0.006969437776542303,
+            0.004293129058563288,
+            0.004114452739123398,
+            0.004071017374828283,
+            -0.002445681623750424,
+            -0.00592480769363354,
+            -0.004617501328052899,
+            -0.012067458701076778,
+            -0.008335239153829813,
+            0.009025028998949398,
+            0.008653779421856954,
+            -0.019363018792788574,
+            0.02048011578221327,
+            -0.001188320960771659,
+            0.00772165044232104,
+            -0.01602086976290928,
+            -0.000907553574811279,
+            0.005464880629734941,
+            0.008267702703609528,
+            0.016136762227127184,
+            -0.03654209526492064,
+            0.005179791231187316,
+            -0.0037321883493524766,
+            0.016026013709955444,
+            -0.008215703411373182,
+            -0.006979816286354862,
+            0.02564060526656963,
+            -0.01450763388985563,
+            -0.0009847758228920171,
+            -0.016142304341196283,
+            -0.01407778890808498,
+            0.010815725252959506,
+            0.02050940280233426,
+            0.020684808835848383,
+            -0.0004965802897137407,
+            -0.0005781445948759235,
+            0.00036179841583004193,
+            0.022775804864342608,
+            0.015545681427566235,
+            0.018697154644607316,
+            -0.004543500782944068,
+            0.007028436664826706,
+            -0.023057215501463973,
+            0.009912581556496658,
+            0.0011795417129232596,
+            -0.008215027850908477,
+            -0.008554137076267468,
+            -0.008703378564520125,
+            -0.010448538943182258,
+            -0.011773811327190638,
+            0.004106616839960258,
+            0.013895013678116032,
+            0.004190549543250594,
+            0.0031571190600981074,
+            -0.0225385788570314,
+            -0.006243193914542516,
+            -0.011229641377928767,
+            -0.01674214043453308,
+            -0.007367389908398082,
+            -0.02386334969150212,
+            -0.028021339684039734,
+            0.0002794207370355175,
+            -0.02411861258809571,
+            -0.015958336406924612,
+            0.005268815864773021,
+            -0.01546610225139129,
+            0.00898600137695074,
+            -0.008396481690803911,
+            -0.011873486990732417,
+            0.0074743033679134105,
+            0.015336444788692,
+            -0.017676977079590463,
+            -0.010165317702640217,
+            -0.01690524990506033,
+            -0.013460451917691695,
+            0.010918550283871637,
+            0.00595960977280955,
+            0.009250157926438213,
+            -0.012447979337850305,
+            0.0062932435120863365,
+            -0.012673905639312696,
+            0.015073013552721701,
+            -0.03769166067890086,
+            -0.029992249463405965,
+            -0.00014990754540999537,
+            0.007288036630677436,
+            -0.0020735742042505426,
+            -0.018695120921334806,
+            0.0003860116505754868,
+            -0.003608995261360232,
+            -0.014900191335144973,
+            -0.013866216738266176,
+            0.025251475305524053,
+            -0.006788483107616133,
+            0.01223227424986397,
+            0.0034937277040015626,
+            0.03438333560343413,
+            0.008803852645562129,
+            0.014666206816302367,
+            -0.005709421955160623,
+            0.003742433643393657,
+            -0.0033706258426092744,
+            0.006906464031352315,
+            0.0076578888497945185,
+            0.009276592191710524,
+            0.008025321004305927,
+            0.004576470294889512,
+            0.002887346543823031,
+            -0.001925468214331082,
+            0.02125241545487307,
+            0.025671028960818562,
+            -0.00037467906648160044,
+            0.008981063192139038,
+            0.0009387900084090577,
+            0.0002735789978680719,
+            -0.008374290340790827,
+            0.006547013284767913,
+            -0.008274637814401893,
+            0.004041066530254948,
+            0.012897549078838394,
+            -0.009150863940242148,
+            -0.010211568689323908,
+            0.016575088843292566,
+            -0.012380099999574783,
+            -0.03420455578618604,
+            0.026550654680719987,
+            0.005324512466403649,
+            -0.0035670106106754245,
+            -0.010657928468872608,
+            0.0037176027853158167,
+            -0.01103891334887315,
+            0.008096915671982816,
+            0.022587659771132487,
+            -0.0013527917944322358,
+            -0.0018085171207746743,
+            0.0038503794877980586,
+            -0.011870484733648887,
+            -0.007881489342351292,
+            0.006462772034381968,
+            0.0315696514747912,
+            0.01326417653779577,
+            -0.020400369539140532,
+            -0.015223808971473513,
+            0.020406862349006738,
+            -0.023142176712111575,
+            -0.006254959806032159,
+            0.009039506093607052,
+            0.0009087520492922078,
+            -0.022766046838415165,
+            0.0009662037093312435,
+            0.004701966191595815,
+            0.005946458676746661,
+            -0.006358986495019584,
+            0.03237532341275415,
+            0.001075992257265679,
+            0.015892020271402903,
+            -0.017189453434945788,
+            0.0017514632185789299,
+            -0.014189278671121677,
+            -0.014265447388465805,
+            -0.029809460132405256,
+            0.0005557491671071693,
+            -0.0030324198417091704,
+            0.011038984948790744,
+            0.024808067315720335,
+            0.01145103533194245,
+            0.014432246996007246,
+            0.010552852947578788,
+            0.01513689173494211,
+            -0.013653722180565311,
+            -0.029009757850432732,
+            -0.005171522291984226,
+            -0.018001048743500243,
+            -0.02297482628212251,
+            0.003962613844063587,
+            0.013645125285548414,
+            0.006278662690346082,
+            0.023786381386575035,
+            0.0021738516691483163,
+            0.0009580591750940011,
+            -2.690805459518147e-05,
+            0.008894091070322921,
+            -0.004952203441302471,
+            -0.006553986197076102,
+            0.003570189040872711,
+            -0.000652714137223501,
+            0.021228360881461853,
+            -0.005758946428212041,
+            7.897153237591187e-05,
+            0.014242175492137011,
+            -0.005622256348934008,
+            -0.0035009192529791146,
+            -0.017339926143652377,
+            0.006248842389703505,
+            -0.01622011586976866,
+            0.025783086900871728,
+            0.018450783573486916,
+            -0.009109295293546267,
+            -0.013469521573468915,
+            -0.011259555201313116,
+            0.02773097452736661,
+            0.004631282554715908,
+            0.010787055220617197,
+            0.018228322680582464,
+            0.01461254259249465,
+            0.01784830193964943,
+            0.004664725683522323,
+            -0.0034339781146649384,
+            -0.0007692550982545399,
+            0.021176646860433974,
+            -0.014700178590249292,
+            -0.002701208084390198,
+            -0.008842788249773623,
+            0.0021799005046816444,
+            -0.008431299000066363,
+            0.006086447848362669,
+            -0.010793016137040172,
+            -0.015251381840165954,
+            0.0189503457652375,
+            -0.02660767353735015,
+            0.016423753600066336,
+            -0.00593016170680532,
+            -0.00482891500342298,
+            -0.027459214027897378,
+            0.010728039528784636,
+            0.0027866316191608297,
+            0.0022519974195000687,
+            0.010770472550461752,
+            -0.021427634123401408,
+            -0.01867717541189363,
+            0.008532444554010673,
+            -0.011457117350255678,
+            -0.007152301670244725,
+            -0.00348619742388509,
+            0.0154715607296319,
+            -0.00017898955187849443,
+            -0.009925365923862246,
+            0.02085093731060963,
+            -0.007175309371250024,
+            0.01223979370120031,
+            0.005256127464980655,
+            -0.008756077893734508,
+            -0.0028438320920811362,
+            0.021022708898898203,
+            0.027557382862284808,
+            -0.008342206702778956,
+            0.014526066580760486,
+            -0.011883797998224506,
+            -0.01088335352319397,
+            -0.03479444009382274,
+            0.030104838505937447,
+            -0.013305732257226255,
+            0.026773747582411633,
+            -0.0006968265979630097,
+            -0.017037729111584094,
+            -0.016206207453228583,
+            0.01625225876515654,
+            -0.0024290584023212254,
+            -0.009366287391027166,
+            0.01465885301870365,
+            -0.02428628495473643,
+            0.0058336571592899735,
+            -0.0006627935593670669,
+            -0.011531762838191602,
+            0.013275508677482622,
+            0.0019223252930710324,
+            0.010738622785461513,
+            -0.02030405805119001,
+            0.0009661595529028194,
+            0.009263716373456188,
+            -0.016791855691574455,
+            0.006057830565306032,
+            -0.03730952786847912,
+            0.0029341721588188943,
+            -0.014833895684348656,
+            0.005725615593914644,
+            0.001498755253963699,
+            -0.01820708352937343,
+            -0.03652692232407375,
+            0.00043665701245599497,
+            0.01913883585802903,
+            -0.007975988799790905,
+            0.020538595326175857,
+            0.008243569241768276,
+            0.008421404577931517,
+            0.002314738067443065,
+            0.0016702403734608475,
+            0.008279538988821465,
+            0.03679456861213399,
+            -0.01775927634148114,
+            -0.004836606118958108,
+            -0.0012151329859941935,
+            0.010644195708319258,
+            -0.02028470571116305,
+            -0.009287515726906219,
+            -0.0036506912311261536,
+            -0.013134467570201307,
+            0.030615862647585222,
+            -0.020265555441523482,
+            0.025634449009756734,
+            -0.009300770899023007,
+            0.009274072905190077,
+            -0.02097685009125915,
+            -0.009797008690754956,
+            -0.010661540100131183,
+            0.0019532291898233895,
+            0.013465005282849381,
+            -0.009151430285942593,
+            0.0071467934925125145,
+            -0.006655608635025676,
+            0.01020327561163374,
+            -0.001472597695707201,
+            -0.02443508800166349,
+            -0.0024998811551575563,
+            -0.01107321258077183,
+            0.02456164421154801,
+            -0.0019619696941303388,
+            0.0005043179699158694,
+            0.010951580163871948,
+            -0.00853593956432799,
+            0.023927336608555987,
+            -0.008855702280766362,
+            0.0010093679746665581,
+            -0.004863269000020324,
+            -0.004005691415117686,
+            0.001698212044518236,
+            0.01822024022650613,
+            -0.005707404715849356,
+            0.03722587302228139,
+            -0.01708925706528118,
+            -0.0065500772743641925,
+            -0.00015538686795361824,
+            0.010852925330001283,
+            0.010035359503147549,
+            0.008754719437760327,
+            0.004067820167375348,
+            -0.014861417626433965,
+            0.009346669946626484,
+            -0.020136029447262147,
+            0.007501669351583119,
+            0.023178490169347605,
+            -0.006840359541885465,
+            0.0006676450965357345,
+            -0.01283320739092204,
+            -0.015610538827742276,
+            0.006308543164022682,
+            0.002329420005586303,
+            -0.002295739637504597,
+            -0.02181189272979859,
+            0.006298804075737111,
+            -0.014398470126723037,
+            0.004007419942728446,
+            -0.0007497656993578918,
+            0.013863867106289847,
+            0.016451166273826513,
+            -0.006740209707090658,
+            0.039292260347522444,
+            0.01871723939316731,
+            0.009109005310505462,
+            0.005674782595856756,
+            -0.040588444759736254,
+            -0.008993850998154852,
+            -0.0010197185016393919,
+            0.004866150984083949,
+            -0.01141354216871663,
+            -0.018944881712855702,
+            -0.002646110151132781,
+            0.000308585036997592,
+            -0.010681954707313127,
+            -0.009091585604191968,
+            0.01751599002710486,
+            -0.006034424764302979,
+            0.00872057856926618,
+            0.004146961781109887,
+            0.023758065635807138,
+            -0.009727802672285544,
+            0.01176257119851645,
+            -0.0071822083765114165,
+            -0.005456197751568526,
+            0.0006726765703478756,
+            0.011011628907288813,
+            -0.008031792262886802,
+            0.009763340867550022,
+            -0.004279891138185881,
+            0.008355720233269893,
+            -0.008552596384067825,
+            -0.023952129458400563,
+            -0.014972827087541732,
+            0.014347535404251018,
+            0.011651260496709961,
+            0.008137738462704715,
+            -0.013624435569539285,
+            0.0031728721674035352,
+            0.00675830277744976,
+            -0.009184784787581942,
+            0.009161771311910335,
+            0.017733082848622708,
+            -0.001411542568472906,
+            -0.0017830167499636113,
+            -0.0016130003349297027,
+            -0.012173528776899308,
+            0.0011196437753239162,
+            0.02284887358189995,
+            -0.003054479643937998,
+            0.014614283913976716,
+            0.010689486748036806,
+            0.009643292262102645,
+            -0.00810930837603355,
+            0.001915955511174678,
+            0.02341863048264366,
+            0.015022291783105098,
+            -0.004625715884342598,
+            -0.011422973894623295,
+            0.004259757717597968,
+            -0.010332744704689279,
+            0.007373328473086509,
+            -0.02117940113144938,
+            -0.004672698999328004,
+            0.008896311380763048,
+            -0.007028862917603513,
+            -0.01732432285099198,
+            -0.028026173555055017,
+            -0.016765805892242715,
+            0.019819329380181,
+            0.0027213760678977965,
+            -0.013291251876048186,
+            -0.0032919084721662363,
+            0.0031214883337300926,
+            0.002553005168048115,
+            -0.0071104806975682885,
+            0.005310583052470149,
+            0.007359183453835734,
+            -0.008674133980486269,
+            0.017665958608281126,
+            -0.000521269431423141,
+            -0.0008020251454616109,
+            -0.0005442374261144659,
+            0.0006127687598711624,
+            -0.016387752806126442,
+            0.01666161912459431,
+            -0.010949228376419892,
+            0.015198110119816624,
+            -0.008966726991404666,
+            0.008600499511690635,
+            -0.007617043351740469,
+            0.009458641139070436,
+            0.011597957400445055,
+            0.012755037944648426,
+            -0.004403680820063604,
+            0.008519241473418196,
+            0.004807816191718704,
+            -0.008147125240985166,
+            0.007017601015398136,
+            0.01296082929962557,
+            0.00582353932160707,
+            0.0013634237970372698,
+            -0.0008246894322793264,
+            0.0022035670958705576,
+            -0.002480976219285683,
+            0.0056293125704658135,
+            0.012714850523393719,
+            -0.002497639449055006,
+            -0.007135683201188999,
+            0.014988249771676699,
+            0.014955895630183352,
+            -0.022994948923833993,
+            0.007121634434082677,
+            0.005835971825639792,
+            -0.0030507120821557008,
+            0.002145266953387315,
+            -0.027334525318320587,
+            -0.020359474070022213,
+            0.01901995970350808,
+            -0.015260454474605018,
+            0.02850460517419317,
+            -0.011624138792780823,
+            -0.02284242513728605,
+            -0.01277489645734181,
+            -0.004209529633534193,
+            -0.022684500849916897,
+            -0.004894414098943742,
+            0.01919165657366256,
+            0.013401560972810082,
+            -0.004313549020265677,
+            -0.019089369171102138,
+            0.03863248408555474,
+            0.010497951840980463,
+            0.03508157919687289,
+            -0.008597970944278045,
+            0.0023354831464448313,
+            -0.01563401808449416,
+            0.015551176593436512,
+            0.001075882802133782,
+            -0.011575692834796734,
+            0.01505716108583101,
+            0.031836226915668404,
+            0.0034740889237207494,
+            -0.00617206855371193,
+            0.0018420207148235309,
+            0.00418433325800744,
+            -0.0008167766650900381,
+            0.011160989178055499,
+            -0.018497472318034093,
+            -0.011951262348399922,
+            0.0013093376768780122,
+            0.00216347903176653,
+            0.0059233460419514045,
+            -0.001778091863744448,
+            -0.013317639704511549,
+            0.011433308892643204,
+            0.011596810919469926,
+            -0.01518143105525228,
+            -0.007497373860338335,
+            0.0050957665991362405,
+            -0.018974405317819656,
+            0.01872472902494418,
+            0.011859522398789263,
+            0.000792413325332956,
+            -0.0006503947129906383,
+            0.00808635755624928,
+            0.003749813548995223,
+            0.0033657799335872368,
+            -0.007862215303617144,
+            -0.0028014669075116876,
+            0.0009892310856576799,
+            3.834379772674841e-06,
+            0.015402211364447726,
+            -0.015264893397161102,
+            0.02488855366696587,
+            -0.0025643639525691083,
+            0.00953993868921819,
+            -0.017460590918773188,
+            -0.0017911712931080155,
+            -0.00402374787529868,
+            0.01838051670627292,
+            0.006753072393532114,
+            -0.010130676216186492,
+            0.004921510943896829,
+            0.007975201081296845,
+            -0.012589786869771252,
+            0.00018319054705558588,
+            -0.0022606180909706688,
+            -0.014791528072476771,
+            -0.0031171076370692706,
+            0.017873825561816544,
+            0.007593688317551089,
+            0.0037375070591255375,
+            -0.015863143964928172,
+            0.007192324815779268,
+            0.007522990667369163,
+            0.024504875417877527,
+            -0.026309816223960274,
+            0.024779042752840836,
+            -0.022104087334088543,
+            0.02165781034613893,
+            0.0319051741666687,
+            0.007538146369110045,
+            -0.01690181186708103,
+            -0.006595947249169521,
+            0.006373991305923002,
+            -0.023410814842488123,
+            -0.005494017994983188,
+            0.016903736810723716,
+            -0.01792400152372258,
+            0.000330865061689199,
+            -0.014660413003063089,
+            0.0040825880154611895,
+            0.008248037865838212,
+            0.011540333451799686,
+            -0.0033578830043738085,
+            0.018327525495676525,
+            0.020171745352159914,
+            -0.01138759854371235,
+            -0.0004221149053107702,
+            -0.022321409200941777,
+            0.005046865873981027,
+            0.022509643273449936,
+            0.001111272240506322,
+            -0.009540467231190513,
+            0.018376261423745193,
+            0.008480532791922674,
+            -0.020801891594934313,
+            0.008436374227753589,
+            -0.036175982655998785,
+            -0.010454118523313122,
+            -0.009328507118928412,
+            -0.001218633166683384,
+            -0.004142773602181115,
+            -0.0002494371580772723,
+            -0.011365548842513265,
+            -7.699548642315061e-05,
+            -0.006889339420877534,
+            -0.0037173070609413474,
+            0.014953978004522174,
+            0.011678957032329096,
+            -0.003501047405385249,
+            -0.0073984932581048915,
+            0.008493128567102267,
+            0.006163937693564917,
+            -0.017034696006318337,
+            -0.013268418502042576,
+            0.0037122128689059085,
+            -0.020067656401141467,
+            0.005148329575092478,
+            0.00459729178556505,
+            0.014031669598917171,
+            -0.01849235806598133,
+            -0.009298127665365717,
+            0.019716512208448042,
+            0.0094466405158553,
+            -0.0008988916054855498,
+            0.004264260983102956,
+            -0.00032312548009733906,
+            0.002279987803645912,
+            -0.0008902345374346296,
+            -6.145528806973222e-06,
+            0.003602229465148649,
+            0.012503366799381277,
+            0.007759393855482671,
+            0.03694402922447051,
+            -0.016692148708641812,
+            0.005873765863350649,
+            -0.014187609702966002,
+            -0.0037592517335595654,
+            -0.00749939010396784,
+            -0.0004308157309481927,
+            -0.0022282234382577843,
+            0.0021999996319642854,
+            0.0052176773733353925,
+            0.009353171029508977,
+            0.002155405440765206,
+            -0.00823565584692709,
+            0.009691447250255694,
+            -0.015228173095533029,
+            0.003853858061067531,
+            -0.008207910291425473,
+            0.01017972550354623,
+            -0.009695280827515048,
+            0.0028705587799331246,
+            0.03849358164875786,
+            0.01130021300937487,
+            -0.004934175853966098,
+            -0.014300147534361459,
+            -0.0007796051618993811,
+            -0.01737877501558196,
+            -0.008579119859817895,
+            -0.005434792488602325,
+            -0.02617338653902575,
+            -0.00838916510137031,
+            0.015344260464413648,
+            0.011202567142310492,
+            0.014124853863170132,
+            -0.023894182185855824,
+            0.008897943489322673,
+            -0.03001670823127986,
+            -0.011951507895140876,
+            -0.001530967140156163,
+            -0.015467812535428609,
+            0.0010110999991167703,
+            0.023623196504289838,
+            -0.007762735863648054,
+            0.012654652822750325,
+            0.012867958170258448,
+            0.018777522196571934,
+            0.006390986057157016,
+            0.00038045736974183937,
+            -0.009444911783760569,
+            0.009534310165417939,
+            0.008806031651319704,
+            0.0080954543713255,
+            0.013161833638467553,
+            -0.0022840529693446553,
+            -0.018060162439023572,
+            0.005469288590000856,
+            0.014354475778235131,
+            -0.0033504276914600585,
+            -0.001615191819057537,
+            -0.011088493180661976,
+            0.016039040565630432,
+            -0.023559046333872156,
+            -0.009482398794557168,
+            0.009918576960319548,
+            -0.0027308309155029842,
+            0.009212063547988036,
+            0.0160181200081126,
+            -0.01881373001297218,
+            -0.0024168618343039703,
+            0.016244889977796682,
+            0.0024104546308670025,
+            -0.00022606425578303413,
+            0.016792244238218245,
+            -0.0007616857360728623,
+            -0.004279120886828465,
+            0.00997764266310123,
+            -0.0075631917138318655,
+            7.3878137053128405e-06,
+            -0.002416364915008295,
+            -0.0028253498168087964,
+            0.02054128727706496,
+            -0.021281801678871736,
+            -0.005379259148621041,
+            0.02257306103533458,
+            0.015687105622685975,
+            -0.007843745111084413,
+            6.258498042855072e-05,
+            -0.0219506172079994,
+            0.0019925886885244454,
+            -0.007768133999661569,
+            0.016524850592507592,
+            -0.007957949830060354,
+            0.0030085044673901665,
+            -0.01361024620290394,
+            0.011563781350927133,
+            -0.01277407169098418,
+            0.00873716215529134,
+            -0.002986943948333218,
+            -0.017567646684567492,
+            -0.0053109390854215125,
+            0.004561236136619266,
+            0.0014045175130801308,
+            0.044958041347040185,
+            -0.005539136232703934,
+            0.003729517522976541,
+            0.018086335927383804,
+            -0.014199453349727933,
+            0.02079719360289355,
+            0.016663293302592568,
+            0.00991445855219406,
+            -0.00033967370741635794,
+            -0.0018726748849858836,
+            0.023373354806639515,
+            0.013258226441642464,
+            0.00861986998150065,
+            -0.0014340218811281975,
+            -0.008516541227326329,
+            0.018239230461345617,
+            0.005458629943667832,
+            -0.012421896028975013,
+            0.012732721349758832,
+            0.013559877132508506,
+            -0.013720939472355347,
+            -0.011545210834395917,
+            0.01616358223872008,
+            -0.019904594319155373,
+            -0.018260016571763644,
+            -0.0036124537591368864,
+            -0.01614510591132948,
+            -0.03864699384205461,
+            0.01619401261888991,
+            0.018351607973242735,
+            0.006666313000123845,
+            0.013647410298896132,
+            0.002597680061427196,
+            0.004545530267244157,
+            -0.018531082430861854,
+            -0.0077618625909995275,
+            -0.008085144524135461,
+            0.003744735923004248,
+            0.011059457364156674,
+            -0.011335432904114519,
+            -0.009514379703581616,
+            -0.00034500702695418285,
+            -0.007433132060989644,
+            0.01971420270994951,
+            0.0020753610290475642,
+            0.0034506706065161024,
+            -0.00628437298309269,
+            0.017966485854173167,
+            0.026789678964152908,
+            -0.007532446825077476,
+            0.021886182869092073,
+            -0.002864230253056434,
+            -0.0015870360120997492,
+            -0.014393519359298496,
+            -0.0014439412842964876,
+            -0.011527182213472351,
+            0.0005453493437517964,
+            -0.0006791481533636673,
+            0.014093767997869645,
+            -0.03288824642147967,
+            -0.003957301296796813,
+            0.014444592778851293,
+            0.007538052312876599,
+            -0.005057755472988024,
+            0.011357919882949903,
+            -0.002964451028327082,
+            0.016977317604413924,
+            0.0075201055190542146,
+            0.023391646079994784,
+            -0.016588823248397513,
+            -0.004504858318770565,
+            0.006660877064954993,
+            0.0034631255931793638,
+            0.0014484203909961972,
+            0.01912983041187014,
+            0.023439941258581887,
+            -0.014715886385460541,
+            -0.0013596123366001458,
+            -0.0029020372473879614,
+            -0.021414221343490594,
+            0.014976806123253727,
+            0.0074729481304182865,
+            0.010967351226114886,
+            0.009806663105020302,
+            0.015465076881663691,
+            -0.02109587527354609,
+            0.0011671121739822202,
+            0.020567458365051382,
+            -0.008618333549105003,
+            -0.018815706906171112,
+            -0.009804829158883123,
+            -0.00966828714719191,
+            -0.006078893678150302,
+            -0.00803783313451146,
+            0.005523023167028105,
+            0.0022647506733877573,
+            0.01652997604669998,
+            -0.008204200434441975,
+            -0.008404609147866811,
+            0.008489247125986078,
+            0.007546785673285454,
+            -0.0033093451283570316,
+            0.010614050963656496,
+            0.0030633179859393587,
+            -0.006940540318011839,
+            0.00894884343338777,
+            -0.002787816848430143,
+            0.014347598920245487,
+            -0.02441769157218451,
+            0.004717742147185283,
+            -0.012096876825337124,
+            0.006405151160675699,
+            0.0041568790527989905,
+            0.0016702066682613428,
+            -0.0041159211459004,
+            0.0035217362856754353,
+            0.0008893479210743541,
+            -0.07433312151926533,
+            -0.012629681414004524,
+            -0.0035797012285864486,
+            0.033657838713146884,
+            -0.019695806809053914,
+            0.004477927056336216,
+            0.011966065644382396,
+            0.0096717896523898,
+            0.003070713232538593,
+            0.006136726101157844,
+            0.006112385569942848,
+            -0.0245258242481677,
+            -0.00015284418977092615,
+            0.004821040694679944,
+            0.0020347004045943454,
+            -0.007532408630122127,
+            0.00750565079228405,
+            0.015327710245879296,
+            0.020989926895388523,
+            -0.010970305087289488,
+            0.017227663358885748,
+            -0.003352718496981928,
+            -0.006068892126250978,
+            0.0014285750322333686,
+            -0.010153799722483554,
+            -0.017307415843726394,
+            -0.006495008086641125,
+            0.0068983699213297075,
+            -0.001982568363667046,
+            0.0044032687939897855,
+            0.003333596495330213,
+            -0.0032171586993948696,
+            0.0072365127663633925,
+            0.013920326463272646,
+            0.006415916101287133,
+            -0.013675562796300214,
+            0.003003782758807665,
+            -0.02500375035312058,
+            0.011945988879567646,
+            -0.012708975181025296,
+            -0.0009488161508767279,
+            -0.0047093877027523014,
+            -0.0035750724825159503,
+            -0.002213435677949806,
+            0.02043567538125188,
+            -0.014304009033474186,
+            -0.01802899775650995,
+            0.008187227768196673,
+            -0.011217013195839762,
+            0.008300740149622635,
+            -0.03382831122769974,
+            0.012386335491333264,
+            -0.0276029934382802,
+            -0.0002243752790856292,
+            -0.004521387639042107,
+            -0.0010631721066129095,
+            -0.009491184157278669,
+            -0.03274559774619223,
+            0.015845705808609598,
+            -0.007204751554079551,
+            -0.020338337341085228,
+            0.0007143424225187802,
+            -0.021267094083674304,
+            0.014150458196361147,
+            -0.01851165553224022,
+            0.0033450353123933058,
+            -0.018775439467431685,
+            0.002782289821140401,
+            -0.0018711350216089556,
+            0.010502195116579441,
+            -0.00557814442809019,
+            -0.0007130962242755068,
+            -0.013072988887382269,
+            -0.002121841212381393,
+            -0.006653444103331213,
+            0.009394907690516121,
+            0.0028678074167728367,
+            -0.009601577734902932,
+            0.011153571864462314,
+            0.013046116892519414,
+            0.019812860521973938,
+            -0.00539047787915588,
+            -0.00555139958411929,
+            -0.011553689090179406,
+            0.009852063270798534,
+            -0.016326212706634428,
+            -0.00827402913960903,
+            0.008982303148984887,
+            0.015216947130173639,
+            0.01026889105233137,
+            0.003352028462252994,
+            0.003150801750574548,
+            0.002531620731961765,
+            0.026744691368233237,
+            -0.0015120532859309638,
+            0.0008324701960729416,
+            -0.014508988496009931,
+            0.022127525293249926,
+            0.02772430884375744,
+            -0.0012206102581726523,
+            -0.00018859639594030037,
+            0.017437360169132967,
+            -0.004697052522250805,
+            -0.0038111184371524645,
+            0.024015246959315593,
+            -0.038396108183338655,
+            -0.008416825232533771,
+            -0.006788816882617399,
+            -0.006043428198944214,
+            -0.009587366247924467,
+            0.007294190273465812,
+            0.00041966080769004275,
+            0.0024709692753547823,
+            -0.0005250904505954556,
+            0.03452858797518386,
+            0.003403515654095309,
+            -0.0003392060101496309,
+            -0.0032980864446307116,
+            0.0019125121221204029,
+            0.0032987517853214346,
+            -0.009116761755779728,
+            -0.038030085122772624,
+            0.006006334000396066,
+            0.007932785488877407,
+            0.00975009749364395,
+            -0.008120121513849434,
+            -0.008864461019882246,
+            0.004909348738798099,
+            -0.02273129283865203,
+            -0.01372636307167243,
+            0.02625229144362647,
+            -0.008523028986127237,
+            0.029005877324974557,
+            -0.022435059061182174,
+            -0.013135836801818153,
+            -0.004126433322528265,
+            0.010283975013795537,
+            -0.019732856462444506,
+            0.004343860338939202,
+            -0.004022387065239562,
+            0.023733721026005954,
+            0.0077578144626961245,
+            -0.008371631446146718,
+            0.012386735614153616,
+            0.0064789380436815714,
+            -0.0026638642827013837,
+            0.00136333052963848,
+            -0.007209517067748103,
+            0.016916505654836485,
+            -0.011819784026452944,
+            -0.005833909403097991,
+            0.008977527508585844,
+            -0.0024556434204852946,
+            0.02509293948918158,
+            -0.0237762245071987,
+            0.005288171058938098,
+            -0.00019047961064753207,
+            0.01822542471604037,
+            0.008127347078975925,
+            0.008361588667386882,
+            -0.011835407590060994,
+            0.010004482616697591,
+            -0.03378107067882622,
+            -0.015433906677849963,
+            -0.007702910250795191,
+            -0.0013877702264578865,
+            -0.002059152251387601,
+            -0.015745218548968338,
+            -0.026393221735795527,
+            0.006294583740809379,
+            0.016267525575472445,
+            0.023291528020980265,
+            -0.004683197930892445,
+            0.012912830736650997,
+            -0.002831583991769044,
+            -0.008904271394710672,
+            -0.011441882849296418,
+            -0.012202629605208118,
+            -0.012470773094383122,
+            -0.014560694782340029,
+            -0.017045701603711442,
+            0.012139665577738337,
+            -0.020079061996873537,
+            0.013911518328924777,
+            0.0347210775522539,
+            -0.017176114991595492,
+            -0.0018941772168839354,
+            0.019702722555622765,
+            -0.004221469612316137,
+            0.01212088353057276,
+            0.0063707489618353725,
+            0.015298014532351866,
+            0.0038643174573634725,
+            -0.012181479717826009,
+            -0.007374005740809251,
+            0.019411457664902376,
+            -0.004902591844710158,
+            0.014591446725717338,
+            0.015069599884557399,
+            -0.008675672164385891,
+            0.002544198867387345,
+            0.01951865904496276,
+            -0.01762101777798757,
+            0.01665654707715257,
+            -0.00458402229480694,
+            -0.009936206685538294,
+            -0.013345740581684638,
+            -0.005336416724426228,
+            -0.019469457091169776,
+            0.0061608454109892425,
+            0.018931676738452464,
+            -0.00533013594306834,
+            0.009310871143727683,
+            -0.0037670793158045,
+            -0.005045151384034014,
+            0.0002041909886640482,
+            -0.04152751596227068,
+            -0.008340911336217609,
+            -0.0006651584370005978,
+            -0.02505748815737022,
+            0.025954102004571416,
+            0.007205602377144382,
+            0.022602344303400163,
+            -0.017168610245491406,
+            0.004459735690564842,
+            -0.005882024209102714,
+            0.013217517324383834,
+            0.02255689669623969,
+            0.0017984057469785134,
+            0.005699128497215882,
+            -0.02353849721739205,
+            -0.0034711576718386185,
+            0.015924913374553738,
+            -0.003604478141616307,
+            -0.0034993903155063817,
+            0.012592308877056896,
+            0.012864938348003884,
+            0.01567418642605425,
+            0.008801753550506258,
+            0.015606443428689292,
+            0.010589933994371628,
+            0.008928721556432368,
+            -0.005805129999077126,
+            -0.001025103064751441,
+            -0.015907755521365925,
+            0.0045874330999415985,
+            0.031701049450012876,
+            0.012594483199773746,
+            0.016783858873835704,
+            -0.01778536268185936,
+            0.00789523134933153,
+            -0.0012706411671644011,
+            0.008325108613530198,
+            -0.001677173669891956,
+            0.029540136288412976,
+            0.011739999605699826,
+            0.015670966746894652,
+            0.008732334090105053,
+            -0.016237731209426752,
+            0.045334690225148,
+            -0.016911928942220152,
+            -0.0021139311364009113,
+            -0.01058970056572469,
+            -0.021705338531028618,
+            0.007324792557651635,
+            -0.006556582104686972,
+            0.0018243188317438144,
+            0.010780292738415671,
+            -0.006471850255151019,
+            0.0006997395548596146,
+            0.003907433192252573,
+            -0.023129313269763224,
+            -0.002550875642832235,
+            -0.02237917712764273,
+            -0.003107775639788617,
+            -0.02621608419628401,
+            -0.00012482032027919036,
+            -0.016207377503696517,
+            -0.0038712739995960753,
+            -0.023315508569288476,
+            -0.014963211090562272,
+            -0.01633110043829169,
+            -0.004241081909202262,
+            -0.008295402942465353,
+            0.004976396647108597,
+            -0.007402107755970892,
+            -0.020321297548927468,
+            0.0020986169430507757,
+            0.016803273183070004,
+            0.01434729320047051,
+            0.018612970083178984,
+            0.00603868809307069,
+            0.014278001198851581,
+            -0.0026350854954985117,
+            0.017171016209107656,
+            -0.006968968261909459,
+            0.006881847166565678,
+            0.011454001418121107,
+            0.02140935294963631,
+            0.011121833270235448,
+            -0.012801756637470525,
+            0.0010870041670931706,
+            -0.009655388034828094,
+            0.002094601818569396,
+            -0.00828486034340067,
+            -0.009190975158350585,
+            0.0017762581946855365,
+            -0.007261996868906865,
+            0.004674405202069334,
+            0.0030526253833799426,
+            -0.0014082232541965316,
+            0.010419302200840906,
+            -0.0024200053377751805,
+            -0.012208685875688886,
+            -0.0033037330470612695,
+            -0.017975940046938498,
+            -0.006958042903103732,
+            -0.0014496522238488239,
+            -0.011711502944602638,
+            -0.01016130418405875,
+            0.003321436189059608,
+            -0.00538282506552848,
+            -0.009353581612407435,
+            0.013884147453136417,
+            0.0001309738696853529,
+            0.0005878330075918333,
+            0.023495280602974504,
+            0.006035028203642483,
+            0.008087594627921345,
+            0.013144288945385904,
+            -0.01023995000994454,
+            -0.022574958230635715,
+            -0.0243435727055662,
+            0.007867022215096418,
+            -0.01573478573423379,
+            0.02324699908478915,
+            -0.007929809922043611,
+            -0.00561630876431978,
+            0.0020982723335057625,
+            -0.013699317926847062,
+            0.008578431496569186,
+            -0.011271381061801356,
+            0.0014450645788554738,
+            -0.019581451799757368,
+            0.018059311800267413,
+            -0.010917911869373693,
+            -0.004302687883018328,
+            0.009466103226841532,
+            0.034049086327052606,
+            -0.0005689605174080737,
+            -0.007410970644432269,
+            -0.006594150877740108,
+            -0.00022810271313786974,
+            0.018766765773409123,
+            -0.0029041815756733333,
+            0.005758781908059541,
+            -0.01563519107805783,
+            0.006571628287171105,
+            -0.006132372252825422,
+            0.0012423261344297225,
+            0.01513074831647933,
+            -0.026306320707079913,
+            -0.015491710256868008,
+            -0.007857352164317122,
+            -0.024893430640551818,
+            0.002253574511878983,
+            0.017579477884206534,
+            0.020505606775521504,
+            -0.020324505029926956,
+            -0.02513332780748072,
+            0.005150966733930881,
+            -0.026580994694344065,
+            0.0037390577082486594,
+            0.1241031878517593,
+            -0.002138404898309707,
+            0.002720723037908521,
+            -0.011425036886011179,
+            0.0022399461009316216,
+            0.005389407121650262,
+            -0.019552543955456567,
+            0.05950746473186985,
+            -0.00638023587296763,
+            -0.004023798021748334,
+            0.002170545188776296,
+            0.001785855390676113,
+            -0.005831534307818096,
+            0.008643501093778434,
+            -0.017602787281866816,
+            -0.02184792441891879,
+            -0.0052215705748838905,
+            -0.021252169339074325,
+            -0.008509621422071373,
+            -0.013000137524907565,
+            0.002893880203681282,
+            -0.015821609145923173,
+            0.010397692426614709,
+            -0.01638916243796692,
+            -0.010283706937701179,
+            -0.01103621454761223,
+            -0.00373552043529251,
+            0.01563553822905027,
+            -0.03188533919653949,
+            -0.004873550301425776,
+            -0.005183590054152726,
+            0.018218971950313273,
+            0.006004328765748325,
+            0.004357734370247252,
+            -0.014230318971458851,
+            0.0064792070801456235,
+            0.0215589173714926,
+            0.0055272493764401305,
+            0.014643335833970546,
+            0.010618070641357003,
+            -0.0022177303813575846,
+            -0.0024402761885758007,
+            0.017541799738501007,
+            -0.005445774414228888,
+            -0.00023502936119626386,
+            -0.00537569046685066,
+            -0.005974703247446444,
+            -0.010712244203425036,
+            0.01843613166635215,
+            -0.009938564643456672,
+            -0.017388592763277214,
+            -0.0015797412875366156,
+            0.02028434083164951,
+            -0.012761534382003973,
+            0.0031906812114950982,
+            0.000788436210275669,
+            0.020589196433948156,
+            0.0007726855314648301,
+            -0.005481260996289263,
+            0.007165014826294827,
+            -0.02956334397512344,
+            0.004095715232997753,
+            0.0003568512047626187,
+            0.027290768799658912,
+            -0.0017631129386615335,
+            0.01552542586826711,
+            -0.020871286597113327,
+            0.004388125824324094,
+            -0.016926008631435403,
+            0.005099899758322398,
+            0.012560923968841436,
+            0.002781049890810361,
+            0.010889021753593997,
+            -0.008500358897706648,
+            -0.021353460542554444,
+            0.012663934009161902,
+            -0.018657094130860155,
+            -0.029156017587920958,
+            0.0018889388761887053,
+            -0.028158076315864772,
+            -0.003048854609750186,
+            0.006812072971079798,
+            -0.035217412591558014,
+            -0.02251675180883659,
+            0.007396524512996277,
+            -0.00995571600201846,
+            -0.011912753666073882,
+            0.010675240017836214,
+            -0.02645155743041798,
+            -0.018137648527851668,
+            0.0010666944556449141,
+            0.012588800077165337,
+            -0.004561825939541096,
+            0.007555558090958502,
+            -0.028558603070876605,
+            0.014816816621445352,
+            -0.01313764261744744,
+            0.016036065063831813,
+            -0.0037797584923144134,
+            0.022217516257304868,
+            0.011131597210722573,
+            0.008795376259031849,
+            -0.004950080558710245,
+            0.021409590863550987,
+            0.005574351961953293,
+            0.051358937509801736,
+            0.031972009383746405,
+            0.0019821003238211276,
+            -0.003251257665378234,
+            0.003313045083311649,
+            0.008294776423730887,
+            -0.00940948257229233,
+            0.005114631360856872,
+            0.028053227377164806,
+            0.016738013668710083,
+            -0.018755924318109216,
+            0.006606711923035118,
+            -0.02702692774986498,
+            0.017950884766781056,
+            -0.022054653845163,
+            0.0003755944703013671,
+            0.007766253634286795,
+            0.0016206993755471613,
+            0.010682347575070117,
+            0.0013842930079185415,
+            0.01705234255261425,
+            0.023975191051210525,
+            -0.002976402980885077,
+            0.017466027107477605,
+            0.012734909275073793,
+            0.018143161773162015,
+            -0.0038399373197265827,
+            0.006929440921265761,
+            -0.012003095293274786,
+            -0.0008883470597690258,
+            0.0020490527756307267,
+            -0.004231091383919565,
+            0.012128848448049833,
+            0.008340302519160299,
+            0.012989849936937843,
+            0.002974488532907779,
+            0.003198854381461613,
+            -0.005309753760578859,
+            -0.0017024982049936384,
+            -0.01763395035509857,
+            -0.01819680226734474,
+            0.028417724274372468,
+            -0.020217879775485576,
+            -0.050118184195774004,
+            0.0025509548991664425,
+            0.014185761006969567,
+            -0.0012021669304388173,
+            -0.0051260576846881515,
+            -0.0005774778872927327,
+            0.005718864494530703,
+            -0.016918749210038116,
+            -0.0015472638835434902,
+            0.008125845907627804,
+            0.020347547062899797,
+            -0.002036218493206914,
+            -0.01079739518866248,
+            0.0038112958713600906,
+            0.02169339303105242,
+            0.01667594322285835,
+            0.012338193092567552,
+            -0.009798409233733167,
+            0.0003198215558702285,
+            -1.6308740825163337e-05,
+            0.0076870827874737325,
+            0.004356644356656038,
+            0.003500023005601613,
+            -0.019968549869439228,
+            -0.008634740687755112,
+            0.012781565963545847,
+            -0.0025637758121512927,
+            0.010243567793632335,
+            -0.00042296044205132426,
+            -0.003069251553118065,
+            -0.009344199449486244,
+            -0.004694953215957636,
+            -0.009583993744231791,
+            0.013783878036521561,
+            -0.007705982042252044,
+            -0.01549567538665377,
+            0.00020666298495823114,
+            0.0005515775451348472,
+            0.001667367278489879,
+            -0.004518431526236256,
+            0.03302324454129141,
+            0.009531990824506103,
+            -0.0004937348108340977,
+            -0.004047613286672229,
+            -0.0327383112522095,
+            0.012456150771224677,
+            -0.009677533214778067,
+            0.0006455811519981003,
+            0.020012976827426605,
+            -0.00020277620481825933,
+            0.002168432791818704,
+            -0.015657551629104767,
+            0.0032924357559238502,
+            0.006161295735841883,
+            0.013814165416788555,
+            0.0005164426482972658,
+            0.01076512534105537,
+            -0.006498977875968572,
+            -0.010325144017898257,
+            0.004804158067334203,
+            0.0104064855510473,
+            -0.0012188100892986933,
+            0.005171197790296134
+          ],
+          "title": "Mixture-Of-Experts With Expert Choice Routing",
+          "keyphrases": [
+            "Mixture-Of-Experts",
+            "Expert Choice Routing",
+            "sparsely-activated models",
+            "training efficiency",
+            "downstream performance"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "d3110cfa-448f-46ee-b3ea-c6734cf305de",
+        "properties": {
+          "page_content": "C Capped Expert Choice\n\nAs described in Section 4.5, the maximum number of experts each token is assigned can be capped by an entropy-regularized linear programming. Figure 1 compares the validation perplexity when training the 100M/64E models using the base expert choice method (EC-BASE), expert choice capped by two experts per token (EC-CAP2), expert choice capped by three experts per token (EC-CAP3), and GShard top-2 gating.\n\nAs shown in the figure, restricting the number of experts to 2 degrades the perplexity compared to the base expert choice method. This suggests that a more flexible allocation of experts (e.g. more than 2 experts for a token) can enhance model expressiveness. On the other hand, our EC-CAP2 and EC-CAP3 methods still outperform the top-2 gating method by a clear margin. We believe this confirms the effectiveness of a load balanced training, provided by our method. Finally, EC-CAP3 obtains comparable perplexity to EC-BASE. As indicated by Figure 3, only a little fraction of tokens use more than 3 experts therefore we see little or no difference between EC-BASE and EC-CAP3 variants. We present the fine-tuning results of these methods in Table 2.\n\n",
+          "title": "C Capped Expert Choice",
+          "embedding": [
+            -0.0025292420759797096,
+            -0.00710443127900362,
+            -0.03731034696102142,
+            -0.015119067393243313,
+            3.892158201779239e-05,
+            0.01479687076061964,
+            -0.02490578405559063,
+            0.06395599246025085,
+            -0.031752459704875946,
+            0.013524195179343224,
+            -0.011389643885195255,
+            -0.03146248310804367,
+            -0.014837145805358887,
+            0.002412445843219757,
+            0.01028612069785595,
+            0.020378924906253815,
+            0.0013713486259803176,
+            0.009432300925254822,
+            0.005433037411421537,
+            -0.013242273591458797,
+            0.0022654435597360134,
+            -0.03563492372632027,
+            -0.02867548167705536,
+            0.004981962498277426,
+            0.034249480813741684,
+            0.023761987686157227,
+            0.00903761014342308,
+            -0.013854446820914745,
+            -0.04288434609770775,
+            0.04004901647567749,
+            0.017833571881055832,
+            0.01280730776488781,
+            -0.019879519939422607,
+            -0.036955930292606354,
+            -0.03078586980700493,
+            -0.02911044843494892,
+            0.03830915689468384,
+            0.022859837859869003,
+            -0.0014800898497924209,
+            0.012509276159107685,
+            0.033411771059036255,
+            0.012307903729379177,
+            -0.02360088936984539,
+            -0.013516140170395374,
+            0.01215486042201519,
+            0.0035884627141058445,
+            -0.03898577019572258,
+            -0.011365478858351707,
+            0.010390834882855415,
+            -0.04159555956721306,
+            0.01303284615278244,
+            -0.039533503353595734,
+            0.013500030152499676,
+            0.010060583241283894,
+            -0.040693409740924835,
+            0.009480630047619343,
+            -0.006468093488365412,
+            0.034346140921115875,
+            0.08435102552175522,
+            0.03172023966908455,
+            0.01726972870528698,
+            -0.031301382929086685,
+            -0.0481683649122715,
+            -0.01268648449331522,
+            -0.008747633546590805,
+            -0.016440073028206825,
+            -0.0063593522645533085,
+            0.0023298829328268766,
+            0.02923932671546936,
+            -0.018606843426823616,
+            -0.054096780717372894,
+            0.03360508754849434,
+            -0.028256626799702644,
+            0.00737427081912756,
+            -0.0055820532143116,
+            -0.01622259058058262,
+            0.037632543593645096,
+            0.03502275049686432,
+            0.01321810856461525,
+            0.03734256699681282,
+            -0.004065716173499823,
+            0.03325067088007927,
+            -0.0012092434335500002,
+            0.0340561643242836,
+            0.04088672623038292,
+            -0.02535685896873474,
+            -0.0193801149725914,
+            -0.057254306972026825,
+            -0.028691591694951057,
+            -0.021426063030958176,
+            0.0004978941869921982,
+            0.008199899457395077,
+            -0.0026259010192006826,
+            -0.018510185182094574,
+            -0.03308957442641258,
+            0.04613852873444557,
+            -0.012895911931991577,
+            0.007221227511763573,
+            -0.012646210379898548,
+            -0.01455522421747446,
+            -0.02828884683549404,
+            0.007462874986231327,
+            0.0039448924362659454,
+            -0.02221544459462166,
+            0.03605378046631813,
+            -0.01749526709318161,
+            -0.01784968189895153,
+            0.010181407444179058,
+            0.014579388312995434,
+            0.09305033087730408,
+            0.009891430847346783,
+            -0.0006464066100306809,
+            -0.007772989105433226,
+            -0.010946623980998993,
+            -0.05528890714049339,
+            -0.023649217560887337,
+            -0.04394759237766266,
+            -0.03685927018523216,
+            0.021184414625167847,
+            -0.01915457844734192,
+            0.0004923564265482128,
+            0.0317363478243351,
+            0.007176925428211689,
+            -0.0029420561622828245,
+            -0.04723399877548218,
+            0.026967842131853104,
+            0.024809125810861588,
+            0.051615867763757706,
+            0.00537665281444788,
+            -0.01221929956227541,
+            -0.006584889721125364,
+            0.03650485724210739,
+            -0.018751831725239754,
+            -0.02867548167705536,
+            0.017817461863160133,
+            0.0152721107006073,
+            0.05358126759529114,
+            0.03866357356309891,
+            0.0025131322909146547,
+            -0.007893812842667103,
+            0.0069594429805874825,
+            -0.0071004037745296955,
+            0.004671848379075527,
+            0.009134269319474697,
+            -0.046686261892318726,
+            0.019235126674175262,
+            9.407883771928027e-05,
+            0.05799535661935806,
+            -0.026693973690271378,
+            0.018284646794199944,
+            0.005928414408117533,
+            0.036827053874731064,
+            -0.028385505080223083,
+            0.046815142035484314,
+            -0.018010780215263367,
+            -0.021748259663581848,
+            -0.013781952671706676,
+            -0.0015817830571904778,
+            -0.009005390107631683,
+            -0.03573158383369446,
+            0.01416053343564272,
+            0.03134971112012863,
+            -0.0021587160881608725,
+            0.016101766377687454,
+            0.0020167483016848564,
+            0.04194997623562813,
+            0.013201998546719551,
+            0.01603732816874981,
+            -0.07481401413679123,
+            0.029062118381261826,
+            0.029223216697573662,
+            -0.029690401628613472,
+            0.04455976560711861,
+            0.014982134103775024,
+            0.0014508907916024327,
+            0.02337535098195076,
+            0.003131346544250846,
+            0.0004191070911474526,
+            -0.011011063121259212,
+            0.0411444827914238,
+            -0.01603732816874981,
+            -0.015376824885606766,
+            0.024712465703487396,
+            -0.011736005544662476,
+            0.033895064145326614,
+            -0.0009348730673082173,
+            0.029915938153862953,
+            0.02841772511601448,
+            -0.013137559406459332,
+            0.007458847481757402,
+            0.013959160074591637,
+            0.041885536164045334,
+            -0.0005522647988982499,
+            0.005771343596279621,
+            0.032010216265916824,
+            -0.01308923028409481,
+            0.0281760785728693,
+            -0.00663724634796381,
+            0.018026890233159065,
+            0.019927849993109703,
+            -0.009891430847346783,
+            -0.010334450751543045,
+            -0.007189007941633463,
+            -0.02864326350390911,
+            0.015594307333230972,
+            0.04852278158068657,
+            0.02666175551712513,
+            0.026275118812918663,
+            -0.005364570766687393,
+            -0.021184414625167847,
+            0.023536449298262596,
+            -0.03252572938799858,
+            0.002066084649413824,
+            -0.02712894044816494,
+            -0.04733065515756607,
+            0.024003634229302406,
+            -0.0006433859816752374,
+            -0.009464520029723644,
+            0.0012585797812789679,
+            0.007760906592011452,
+            0.03172023966908455,
+            0.0423043929040432,
+            0.02500244230031967,
+            -0.020733339712023735,
+            -0.008119350299239159,
+            -0.010994953103363514,
+            -0.01520767156034708,
+            0.006814454682171345,
+            0.02335924096405506,
+            -0.01900959014892578,
+            0.006057293154299259,
+            0.009980034083127975,
+            0.0008528137113898993,
+            -0.0016804557526484132,
+            0.016850873827934265,
+            -0.010866074822843075,
+            0.025163540616631508,
+            -0.02242487110197544,
+            0.016528677195310593,
+            0.012791198678314686,
+            -0.020701121538877487,
+            -0.0028131776489317417,
+            -0.03605378046631813,
+            0.008115322329103947,
+            -0.004542969632893801,
+            -0.008079075254499912,
+            -0.02326258271932602,
+            0.0007093356107361615,
+            -0.0043335421942174435,
+            -0.012042091228067875,
+            -0.027547795325517654,
+            -0.02453525923192501,
+            0.0004686951288022101,
+            0.009392025880515575,
+            -0.016045382246375084,
+            0.026758413761854172,
+            0.004265075083822012,
+            -0.05799535661935806,
+            0.012920076958835125,
+            -0.024245280772447586,
+            0.06907891482114792,
+            0.008812072686851025,
+            -0.0105760982260108,
+            0.015489593148231506,
+            -0.023939194157719612,
+            -0.011510467156767845,
+            0.01462771836668253,
+            -0.00922287255525589,
+            -0.003711300203576684,
+            -0.0029380286578089,
+            0.02770889364182949,
+            -0.01515128742903471,
+            -0.010382779873907566,
+            -0.045848552137613297,
+            0.05499893054366112,
+            -0.029529303312301636,
+            0.0008810058934614062,
+            0.007825345732271671,
+            -0.007084294222295284,
+            0.018703503534197807,
+            0.01973453164100647,
+            0.02186102792620659,
+            0.01009280327707529,
+            0.01975064165890217,
+            0.015707075595855713,
+            0.0022775260731577873,
+            -0.0622805692255497,
+            -0.002076153177767992,
+            -0.03217131271958351,
+            -0.03159135952591896,
+            -0.02806330844759941,
+            -0.03067309968173504,
+            -0.010076693259179592,
+            0.018155768513679504,
+            0.01971842162311077,
+            -0.02537296898663044,
+            -0.015803735703229904,
+            0.008852346800267696,
+            0.004800726659595966,
+            -0.0041885534301400185,
+            -0.023230362683534622,
+            0.0024990360252559185,
+            -0.03511941060423851,
+            -0.028691591694951057,
+            -0.06714573502540588,
+            0.004402008838951588,
+            -0.008115322329103947,
+            0.010962733998894691,
+            -0.044914182275533676,
+            0.03302513435482979,
+            0.03373396769165993,
+            -0.010463329032063484,
+            -0.0003755602228920907,
+            -0.005481366999447346,
+            -0.006588917225599289,
+            0.008054911158978939,
+            0.020282264798879623,
+            0.030463673174381256,
+            -0.028127748519182205,
+            0.005465256981551647,
+            0.04649294540286064,
+            0.019815079867839813,
+            -0.04839390516281128,
+            -0.026355668902397156,
+            0.014426345005631447,
+            0.021442173048853874,
+            0.000837207306176424,
+            0.015699021518230438,
+            -0.021893247961997986,
+            -0.010986898094415665,
+            -0.02608180046081543,
+            -0.04185331612825394,
+            -0.056803230196237564,
+            -0.010849964804947376,
+            -0.03550604730844498,
+            -0.02126496471464634,
+            -0.009762551635503769,
+            0.015425154007971287,
+            -0.019782861694693565,
+            -0.017430827021598816,
+            -0.021989906206727028,
+            0.03956572338938713,
+            -0.045171938836574554,
+            0.02758001536130905,
+            0.02456747740507126,
+            -0.0045671346597373486,
+            -0.01797856017947197,
+            -0.010495549067854881,
+            -0.007740769535303116,
+            -0.002712491201236844,
+            0.0010280079441145062,
+            0.002889699302613735,
+            0.034120600670576096,
+            -5.767442053183913e-05,
+            0.014442455023527145,
+            -0.035667143762111664,
+            -0.010833854787051678,
+            0.01868739351630211,
+            0.026242898777127266,
+            0.005098758731037378,
+            -0.004414091352373362,
+            -0.045880772173404694,
+            -0.045848552137613297,
+            0.03133360296487808,
+            -0.038921330124139786,
+            0.01639174297451973,
+            0.005868002772331238,
+            0.009287312626838684,
+            -0.04030677303671837,
+            -0.002058029640465975,
+            -0.02184491790831089,
+            -0.01926734670996666,
+            0.04101560637354851,
+            0.013338932767510414,
+            0.03102751635015011,
+            -0.0027165187057107687,
+            0.020266154780983925,
+            -0.039307963103055954,
+            -0.049843788146972656,
+            0.019927849993109703,
+            -0.05061705783009529,
+            0.02277928777039051,
+            0.030334794893860817,
+            0.01315366942435503,
+            -0.023987524211406708,
+            0.029690401628613472,
+            -0.00795422401279211,
+            0.016182316467165947,
+            0.0012464975006878376,
+            0.028401615098118782,
+            -0.0399523563683033,
+            -0.03308957442641258,
+            -0.009053720161318779,
+            0.020169496536254883,
+            0.0004284205788280815,
+            0.0036609568633139133,
+            -0.021329404786229134,
+            -0.002535283099859953,
+            0.008352942764759064,
+            -0.018848491832613945,
+            -0.041885536164045334,
+            0.04146667942404747,
+            -0.014434400014579296,
+            0.007233310025185347,
+            0.0005955599481239915,
+            -0.0030608661472797394,
+            0.0036347785498946905,
+            -0.007676329929381609,
+            0.009947814978659153,
+            -0.005002099554985762,
+            -0.013121449388563633,
+            -0.009762551635503769,
+            -0.003427364630624652,
+            -0.008131432346999645,
+            0.0062143635004758835,
+            0.024003634229302406,
+            -0.011083557270467281,
+            0.019428445026278496,
+            -0.010294175706803799,
+            -0.010197517462074757,
+            -0.01303284615278244,
+            0.013073120266199112,
+            0.04085450619459152,
+            -0.02103942632675171,
+            0.0340561643242836,
+            -0.0007098389905877411,
+            -0.02042725309729576,
+            0.029674291610717773,
+            0.012106531299650669,
+            0.004760452080518007,
+            0.037986960262060165,
+            -0.002492995001375675,
+            0.00475642504170537,
+            -0.03534494712948799,
+            -0.010656647384166718,
+            -0.01865517348051071,
+            -0.05429009720683098,
+            0.02993204817175865,
+            -0.010721086524426937,
+            0.00354013335891068,
+            0.01216291543096304,
+            -0.0006494271801784635,
+            0.012396507896482944,
+            -0.02479301579296589,
+            -0.03724590688943863,
+            -0.018864601850509644,
+            -0.006721823010593653,
+            -0.016738105565309525,
+            -0.020620571449398994,
+            0.005002099554985762,
+            -0.06514811515808105,
+            0.0015445291064679623,
+            -0.0563521571457386,
+            0.004945715423673391,
+            0.03946906328201294,
+            -0.010841909795999527,
+            -0.001440822146832943,
+            -0.01950899325311184,
+            0.009480630047619343,
+            -0.048103928565979004,
+            -0.045171938836574554,
+            0.016142040491104126,
+            0.033282890915870667,
+            -0.010890239849686623,
+            -0.00915843341499567,
+            0.014192752540111542,
+            -0.037632543593645096,
+            -0.0017086479347199202,
+            0.01374973263591528,
+            0.012106531299650669,
+            -0.01480492576956749,
+            0.01056804321706295,
+            -0.022038236260414124,
+            -0.021409953013062477,
+            -0.008054911158978939,
+            0.029883719980716705,
+            0.01416053343564272,
+            -0.011945432983338833,
+            0.004009332042187452,
+            0.010350560769438744,
+            0.03325067088007927,
+            -0.0076521653681993484,
+            -0.005300131160765886,
+            0.010849964804947376,
+            0.016142040491104126,
+            -0.0022654435597360134,
+            0.012340123765170574,
+            -0.007285666652023792,
+            -0.015223781578242779,
+            -0.013717513531446457,
+            -0.003880453296005726,
+            -0.014974079094827175,
+            0.0018657187465578318,
+            -0.003938851412385702,
+            0.015014354139566422,
+            -0.014982134103775024,
+            -0.020523913204669952,
+            -0.02596903219819069,
+            0.02442248910665512,
+            0.02994815818965435,
+            -0.01227568369358778,
+            -0.034571677446365356,
+            -0.02100720815360546,
+            -3.951940743718296e-05,
+            0.022038236260414124,
+            -0.02455136738717556,
+            0.029851499944925308,
+            0.033282890915870667,
+            -0.05554666370153427,
+            -0.002996426774188876,
+            0.004804754164069891,
+            0.006383516825735569,
+            -0.00431340467184782,
+            -0.03615044057369232,
+            -0.0028333149384707212,
+            0.03934018313884735,
+            -0.0007793126278556883,
+            0.04887719824910164,
+            0.0021526748314499855,
+            -0.017108630388975143,
+            -0.032686829566955566,
+            -0.01668977551162243,
+            -0.03863135352730751,
+            0.04900607839226723,
+            -0.020298374816775322,
+            0.034120600670576096,
+            0.014998244121670723,
+            -0.026581205427646637,
+            0.060798466205596924,
+            -0.0376003235578537,
+            -0.006427818909287453,
+            -0.00015921035083010793,
+            -0.007039992138743401,
+            -0.01772080361843109,
+            -0.01228373870253563,
+            0.0002982834121212363,
+            -0.00780923618003726,
+            0.004357706755399704,
+            -0.004990017041563988,
+            -0.0010285114403814077,
+            0.004426173400133848,
+            0.014941859990358353,
+            -0.004418118391185999,
+            0.028852690011262894,
+            0.013822226785123348,
+            0.008127405308187008,
+            0.020523913204669952,
+            -0.02572738565504551,
+            0.016625335440039635,
+            0.027998870238661766,
+            -0.00560621777549386,
+            -0.010390834882855415,
+            -0.008119350299239159,
+            -0.0011427904246374965,
+            0.006991662550717592,
+            -0.028820471838116646,
+            0.015304330736398697,
+            0.014579388312995434,
+            -0.0030205915682017803,
+            -0.020991098135709763,
+            -0.02360088936984539,
+            0.018864601850509644,
+            -0.023165924474596977,
+            0.027773331850767136,
+            -0.021071646362543106,
+            0.00044855783926323056,
+            0.026710083708167076,
+            0.03573158383369446,
+            -0.011083557270467281,
+            0.015006299130618572,
+            -0.00043949607061222196,
+            -0.005058484151959419,
+            -0.020491693168878555,
+            0.0045751892030239105,
+            -0.0008291523554362357,
+            0.009593399241566658,
+            0.011913212947547436,
+            0.005022237077355385,
+            -0.018606843426823616,
+            0.01515128742903471,
+            0.01609371230006218,
+            -0.030689209699630737,
+            0.00428924011066556,
+            0.010664702393114567,
+            0.014176643453538418,
+            0.005070566199719906,
+            0.003234046744182706,
+            -0.023101484403014183,
+            -0.004462420474737883,
+            -0.016738105565309525,
+            -0.01950899325311184,
+            0.044463109225034714,
+            0.0016653528437018394,
+            -0.034088384360075,
+            0.021538831293582916,
+            -0.007748824078589678,
+            0.005171252880245447,
+            0.0281760785728693,
+            -0.009488685056567192,
+            -0.0505204014480114,
+            -0.013185889460146427,
+            -0.006077430211007595,
+            0.016005108132958412,
+            -0.01926734670996666,
+            -0.009247037582099438,
+            0.030479783192276955,
+            -0.004977934993803501,
+            0.0051269507966935635,
+            -0.008079075254499912,
+            0.024841345846652985,
+            0.026581205427646637,
+            0.00314342905767262,
+            0.007970334030687809,
+            0.018945150077342987,
+            -0.006592944730073214,
+            0.009649783372879028,
+            -0.009174543432891369,
+            -0.0035300645977258682,
+            -0.0002856976061593741,
+            -0.005348460748791695,
+            0.06611470878124237,
+            0.0017881902167573571,
+            0.030705319717526436,
+            -0.0016039340989664197,
+            0.02758001536130905,
+            0.015497648157179356,
+            -0.010986898094415665,
+            -0.02266651950776577,
+            0.007644110359251499,
+            -0.012251519598066807,
+            -0.05416122078895569,
+            -0.026855072006583214,
+            -0.022875946015119553,
+            -0.016085656359791756,
+            -0.029883719980716705,
+            -0.006685575935989618,
+            2.6902143872575834e-05,
+            0.009859210811555386,
+            0.004510750062763691,
+            -0.01786579191684723,
+            -0.032010216265916824,
+            0.00969811249524355,
+            -0.0011850787559524179,
+            0.025340748950839043,
+            -0.005348460748791695,
+            0.010487494058907032,
+            0.007221227511763573,
+            -0.005783426109701395,
+            0.019557323306798935,
+            -0.027451135218143463,
+            0.014007490128278732,
+            -1.3608394056063844e-06,
+            0.011333259753882885,
+            0.035860463976860046,
+            -0.014394125901162624,
+            -0.01726972870528698,
+            -0.005384707823395729,
+            0.008107267320156097,
+            0.023165924474596977,
+            -0.009263147599995136,
+            -0.014877419918775558,
+            0.0021103867329657078,
+            0.005175280384719372,
+            -0.032477401196956635,
+            0.017591925337910652,
+            -0.014200807549059391,
+            -0.05764094367623329,
+            -0.004764479584991932,
+            -0.021877137944102287,
+            0.019219016656279564,
+            -0.025195760652422905,
+            -0.006890976335853338,
+            -0.018123548477888107,
+            0.0015183506766334176,
+            0.014168588444590569,
+            -0.023633107542991638,
+            0.01103522814810276,
+            0.05899416655302048,
+            0.0044221458956599236,
+            0.01762414537370205,
+            0.009464520029723644,
+            -0.005880084820091724,
+            -0.008071020245552063,
+            0.003036701353266835,
+            0.021812697872519493,
+            -0.0002548624179325998,
+            -0.031301382929086685,
+            0.039050206542015076,
+            -0.0004669331246986985,
+            0.004591299220919609,
+            -0.010721086524426937,
+            -0.0007571616442874074,
+            -0.010463329032063484,
+            0.009843100793659687,
+            -0.02385864593088627,
+            0.008892621845006943,
+            -0.019927849993109703,
+            0.03731034696102142,
+            0.0008155597024597228,
+            -0.003642833326011896,
+            -0.025904593989253044,
+            0.0013814171543344855,
+            -0.03199410438537598,
+            0.016182316467165947,
+            0.02911044843494892,
+            -0.012324013747274876,
+            -0.0014539114199578762,
+            0.012267629615962505,
+            -0.02737058699131012,
+            -0.001367321121506393,
+            -0.009843100793659687,
+            0.01402360014617443,
+            0.015111012384295464,
+            0.0030165640637278557,
+            0.034926094114780426,
+            0.01267842948436737,
+            0.009931704960763454,
+            0.013016736134886742,
+            0.0001655032392591238,
+            0.015747351571917534,
+            -0.0038140001706779003,
+            0.005578025709837675,
+            -0.010874129831790924,
+            -0.013604744337499142,
+            -0.021168306469917297,
+            0.01592455804347992,
+            0.021893247961997986,
+            0.011663510464131832,
+            0.03750366345047951,
+            0.014780761674046516,
+            0.026468437165021896,
+            0.0022714848164469004,
+            -0.03582824394106865,
+            0.001979494234547019,
+            0.007426627911627293,
+            -0.025082992389798164,
+            -0.004889330826699734,
+            0.0193801149725914,
+            0.011204380542039871,
+            -0.02501855231821537,
+            0.005851892754435539,
+            0.005396790336817503,
+            -0.002138578798621893,
+            -0.0091020492836833,
+            -0.03579602390527725,
+            0.0021989906672388315,
+            0.019992288202047348,
+            0.025791823863983154,
+            0.003517982317134738,
+            0.015787625685334206,
+            0.007865620777010918,
+            -0.009657838381826878,
+            0.03534494712948799,
+            0.006633218843489885,
+            -0.039275746792554855,
+            -0.015223781578242779,
+            -0.008650974370539188,
+            0.01797856017947197,
+            -0.00029677312704734504,
+            0.0031857171561568975,
+            -0.00969811249524355,
+            0.0047805896028876305,
+            -0.007023882120847702,
+            -0.009303421713411808,
+            0.03395950421690941,
+            0.007011800073087215,
+            0.011317149735987186,
+            -0.004273130092769861,
+            0.018413526937365532,
+            -0.014466620050370693,
+            -0.006540587637573481,
+            0.010366669856011868,
+            0.02314981445670128,
+            -0.025276310741901398,
+            0.015167397446930408,
+            -0.005718986503779888,
+            0.003246129024773836,
+            0.016174260526895523,
+            -0.00431340467184782,
+            -0.01562652736902237,
+            -0.02972262166440487,
+            0.0038683710154145956,
+            -0.014257192611694336,
+            -0.008401271887123585,
+            -0.020475583150982857,
+            -0.014853255823254585,
+            0.030350904911756516,
+            -0.0193801149725914,
+            0.0023802260402590036,
+            -0.0014559251721948385,
+            0.016238700598478317,
+            -0.02031448483467102,
+            -0.013483921065926552,
+            0.025066882371902466,
+            0.05261467769742012,
+            0.0019684189464896917,
+            0.02970651164650917,
+            0.07938919961452484,
+            -0.030689209699630737,
+            -0.013645019382238388,
+            0.019315676763653755,
+            -0.03028646484017372,
+            0.0034535429440438747,
+            0.03895355015993118,
+            -0.04501084238290787,
+            -0.06166839599609375,
+            0.013177834451198578,
+            -0.034603897482156754,
+            -0.000111384317278862,
+            0.010439164005219936,
+            0.008747633546590805,
+            -0.019347894936800003,
+            -0.00610562227666378,
+            0.028804361820220947,
+            -0.007414545398205519,
+            -0.003664984367787838,
+            -0.029319874942302704,
+            -0.011470193043351173,
+            -0.018123548477888107,
+            -0.005388735327869654,
+            -0.0063593522645533085,
+            0.0211360864341259,
+            -0.0006786262383684516,
+            0.003614641260355711,
+            -0.019573433324694633,
+            0.00046743653365410864,
+            0.013725567609071732,
+            0.018590733408927917,
+            0.049972668290138245,
+            -9.590377885615453e-05,
+            0.013983325101435184,
+            -0.01715696044266224,
+            -0.026355668902397156,
+            0.041273362934589386,
+            -0.006955415476113558,
+            0.040822289884090424,
+            -0.02725781872868538,
+            -0.0021748258732259274,
+            -0.006568779703229666,
+            0.025807933881878853,
+            -0.01913846842944622,
+            0.013934995979070663,
+            -0.010729141533374786,
+            -0.0117521146312356,
+            -0.020604461431503296,
+            -0.00766827492043376,
+            0.0020328580867499113,
+            -0.026387887075543404,
+            -0.005968688987195492,
+            -0.004108004737645388,
+            0.027209488674998283,
+            0.047846172004938126,
+            0.005416927393525839,
+            -0.004124114289879799,
+            -0.009077884256839752,
+            -0.015392934903502464,
+            -0.008667084388434887,
+            -0.01726972870528698,
+            -0.015819845721125603,
+            -0.01579567976295948,
+            -0.028014980256557465,
+            -0.019766751676797867,
+            -0.019831189885735512,
+            -0.007325941231101751,
+            0.0029380286578089,
+            -0.0007752851815894246,
+            0.0076521653681993484,
+            -0.04639628529548645,
+            -0.0002455489302519709,
+            -0.03065699152648449,
+            -0.01426524668931961,
+            0.017060300335288048,
+            -0.03859913349151611,
+            -0.00634324224665761,
+            0.0004344617482274771,
+            -0.0020217825658619404,
+            0.0015999067109078169,
+            -0.015119067393243313,
+            0.007797153666615486,
+            -0.02795054018497467,
+            -0.007865620777010918,
+            0.002593681216239929,
+            0.017817461863160133,
+            0.0013310740469023585,
+            0.021538831293582916,
+            -0.03101140633225441,
+            0.011760169640183449,
+            -0.039662379771471024,
+            -0.02619457058608532,
+            -0.042691025882959366,
+            0.027048390358686447,
+            -0.029770949855446815,
+            -0.01168767549097538,
+            0.008804017677903175,
+            -0.0007818297599442303,
+            0.00716887041926384,
+            -0.0152721107006073,
+            -0.0013622867409139872,
+            -0.0027265872340649366,
+            0.007039992138743401,
+            -0.005400817841291428,
+            -0.021748259663581848,
+            -0.017527485266327858,
+            -0.017285838723182678,
+            0.020749449729919434,
+            -0.035054970532655716,
+            0.02044336311519146,
+            -0.012549551203846931,
+            0.02253764122724533,
+            -0.020250046625733376,
+            -0.017898011952638626,
+            -0.006641273852437735,
+            0.040210116654634476,
+            -0.02033059485256672,
+            0.011309094727039337,
+            0.039275746792554855,
+            0.04240104928612709,
+            -0.028611043468117714,
+            0.02571127563714981,
+            -0.022376542910933495,
+            0.015489593148231506,
+            0.055256687104701996,
+            0.01772080361843109,
+            0.02983538992702961,
+            -0.004909468349069357,
+            -0.026919512078166008,
+            -0.008804017677903175,
+            -0.011011063121259212,
+            0.0034736802335828543,
+            0.003671025624498725,
+            -0.02901378832757473,
+            -0.002052995376288891,
+            0.012025982141494751,
+            -0.02397141419351101,
+            0.043850935995578766,
+            0.0013300671707838774,
+            0.01280730776488781,
+            0.005952578969299793,
+            -0.016979752108454704,
+            0.02207045629620552,
+            -0.010704976506531239,
+            0.0005127454060129821,
+            0.009665893390774727,
+            -0.0060170185752213,
+            -0.004683930426836014,
+            -0.03511941060423851,
+            0.01121243555098772,
+            0.024116402491927147,
+            -0.004635601304471493,
+            0.003783794352784753,
+            -0.01221124455332756,
+            -0.002809150144457817,
+            0.005316241178661585,
+            0.020588351413607597,
+            0.0016633390914648771,
+            0.03180078789591789,
+            -0.011647401377558708,
+            0.00777701660990715,
+            -0.011840718798339367,
+            0.023810315877199173,
+            0.008062966167926788,
+            -0.004031483083963394,
+            -0.023536449298262596,
+            -0.010793580673635006,
+            0.017946342006325722,
+            0.01760803535580635,
+            -0.030818087980151176,
+            -0.040596749633550644,
+            -0.021103866398334503,
+            0.016343414783477783,
+            0.0010471383575350046,
+            -0.04143446311354637,
+            -0.012420672923326492,
+            0.015135177411139011,
+            0.01168767549097538,
+            0.02948097325861454,
+            0.006536560133099556,
+            0.007442737463861704,
+            -0.00027009123004972935,
+            0.010551933199167252,
+            0.036472637206315994,
+            9.779164975043386e-05,
+            -0.009021500125527382,
+            -0.014998244121670723,
+            0.022956496104598045,
+            -0.029787059873342514,
+            0.0323324128985405,
+            0.011292984709143639,
+            -0.030721429735422134,
+            0.017704693600535393,
+            0.004957797471433878,
+            -0.018010780215263367,
+            0.03515163064002991,
+            -0.03653707727789879,
+            0.00525582954287529,
+            0.028691591694951057,
+            -0.008010609075427055,
+            -0.01913846842944622,
+            -0.002654093084856868,
+            0.006234501022845507,
+            -0.019702311605215073,
+            -0.0030326738487929106,
+            -0.001486130990087986,
+            0.015779569745063782,
+            -0.015280165709555149,
+            0.014297466725111008,
+            0.026403997093439102,
+            0.014659937471151352,
+            -0.023182032629847527,
+            -0.01714085042476654,
+            -0.020056728273630142,
+            -0.003956974949687719,
+            -0.007442737463861704,
+            0.008441546931862831,
+            -0.019460665062069893,
+            0.021200524643063545,
+            -0.02524409070611,
+            -0.05213138461112976,
+            -0.0007012806599959731,
+            0.005590108223259449,
+            -0.020620571449398994,
+            0.004824891686439514,
+            -0.046202968806028366,
+            0.014007490128278732,
+            0.012074311263859272,
+            0.03885689005255699,
+            -0.007772989105433226,
+            -0.0030105228070169687,
+            0.006488230545073748,
+            0.010777470655739307,
+            0.02853049337863922,
+            -0.032139092683792114,
+            -0.0010682825231924653,
+            0.019798971712589264,
+            -0.0019231099868193269,
+            -0.0015676870243623853,
+            -0.0054733119904994965,
+            -0.04417313262820244,
+            -0.033186230808496475,
+            0.03708481043577194,
+            0.010986898094415665,
+            -0.003620682517066598,
+            0.03065699152648449,
+            -0.023101484403014183,
+            0.009488685056567192,
+            -0.016142040491104126,
+            0.016593115404248238,
+            0.010954678989946842,
+            0.02912655845284462,
+            -0.02948097325861454,
+            -0.0013653073692694306,
+            0.03315401449799538,
+            0.010841909795999527,
+            -0.014120258390903473,
+            0.00024693337036296725,
+            0.018300756812095642,
+            0.020991098135709763,
+            -0.017237508669495583,
+            0.016270920634269714,
+            0.005686766933649778,
+            -0.004236883018165827,
+            -0.019798971712589264,
+            0.0013149641454219818,
+            0.0223282128572464,
+            -0.015135177411139011,
+            0.03006092645227909,
+            -0.009657838381826878,
+            -0.010004199109971523,
+            0.04613852873444557,
+            0.02606569230556488,
+            -0.013411426916718483,
+            4.716527837445028e-05,
+            0.006238528527319431,
+            0.017994670197367668,
+            0.007551478687673807,
+            -0.012348177842795849,
+            0.0006000908324494958,
+            -0.02856271341443062,
+            -0.0029923992697149515,
+            0.007458847481757402,
+            -0.008884566836059093,
+            0.009367860853672028,
+            0.021345512941479683,
+            0.005686766933649778,
+            -0.006101594772189856,
+            0.003542147111147642,
+            -0.010318340733647346,
+            0.019347894936800003,
+            0.03250962123274803,
+            -0.01865517348051071,
+            -0.018413526937365532,
+            0.012460947036743164,
+            -0.026017362251877785,
+            0.03785808011889458,
+            0.009915594942867756,
+            -0.0058196731843054295,
+            0.029738731682300568,
+            0.09479019045829773,
+            -0.012791198678314686,
+            0.0223282128572464,
+            -0.027435027062892914,
+            -0.007861592806875706,
+            -0.009762551635503769,
+            -0.010471384041011333,
+            0.01416053343564272,
+            0.042336612939834595,
+            -0.0006982600898481905,
+            0.028127748519182205,
+            0.010237791575491428,
+            0.0009162460919469595,
+            0.02372976765036583,
+            -0.013604744337499142,
+            0.009738387539982796,
+            -0.004172443877905607,
+            -0.0050745937041938305,
+            -0.009198708459734917,
+            0.004587271716445684,
+            -0.0017318058526143432,
+            0.01678643375635147,
+            -0.00408383971080184,
+            0.0025614616461098194,
+            -0.019428445026278496,
+            -0.019815079867839813,
+            0.006427818909287453,
+            -0.012203189544379711,
+            -0.018284646794199944,
+            -0.017656365409493446,
+            0.00240640458650887,
+            -0.0062143635004758835,
+            0.0012867719633504748,
+            -0.007704521995037794,
+            -0.024486929178237915,
+            0.01844574511051178,
+            -0.0038764257915318012,
+            0.0015425154706463218,
+            0.021635491400957108,
+            -0.017801353707909584,
+            -0.014821035787463188,
+            0.016866983845829964,
+            -0.010898293927311897,
+            0.022747067734599113,
+            -0.025985142216086388,
+            -0.0026661755982786417,
+            0.0012575729051604867,
+            -0.0069594429805874825,
+            -0.015223781578242779,
+            -0.015578197315335274,
+            -0.0008940951083786786,
+            0.026210680603981018,
+            0.003989194519817829,
+            -0.011405753903090954,
+            -0.01157490722835064,
+            0.01880016177892685,
+            0.021796587854623795,
+            -0.007680357433855534,
+            0.0020117140375077724,
+            0.005139033310115337,
+            0.0028051226399838924,
+            -0.009287312626838684,
+            0.02303704433143139,
+            0.0323324128985405,
+            -0.005553861148655415,
+            -0.000351898925146088,
+            0.01526405569165945,
+            -0.006125759799033403,
+            -0.005896194837987423,
+            0.01063248235732317,
+            0.001617023372091353,
+            0.007877702824771404,
+            0.008852346800267696,
+            -0.025695165619254112,
+            -0.013459756039083004,
+            0.02210267446935177,
+            -0.006351297255605459,
+            5.386721750255674e-05,
+            -0.005948551464825869,
+            -0.025775713846087456,
+            0.021087756380438805,
+            0.001424712361767888,
+            0.013798062689602375,
+            -0.008050883188843727,
+            -0.027692783623933792,
+            -0.0038784395437687635,
+            -0.018864601850509644,
+            -0.00020942768605891615,
+            0.01374973263591528,
+            -0.013290602713823318,
+            -0.010406944900751114,
+            0.0018183961510658264,
+            0.0030628798995167017,
+            -0.00862680934369564,
+            -0.013886665925383568,
+            0.005892167333513498,
+            0.0020640708971768618,
+            -0.014104149304330349,
+            -0.040338993072509766,
+            -0.008236146531999111,
+            -0.02207045629620552,
+            0.010737196542322636,
+            0.028578823432326317,
+            -0.012992571108043194,
+            -0.013540305197238922,
+            -0.005054456647485495,
+            0.02712894044816494,
+            0.0071527608670294285,
+            -0.006621136795729399,
+            0.008328777737915516,
+            0.006496285554021597,
+            0.024841345846652985,
+            -0.007434682920575142,
+            -0.0007128596189431846,
+            0.0016160164959728718,
+            0.005642464850097895,
+            -0.008973171003162861,
+            -0.016343414783477783,
+            0.0005160177242942154,
+            -0.012082366272807121,
+            -0.02630733884871006,
+            0.008812072686851025,
+            0.021909357979893684,
+            -0.006766125094145536,
+            0.001452904543839395,
+            0.010028364136815071,
+            0.016866983845829964,
+            -0.017994670197367668,
+            0.004301322158426046,
+            0.00243862415663898,
+            -0.007495094556361437,
+            -0.01996006816625595,
+            0.010423054918646812,
+            0.001549563487060368,
+            0.01913846842944622,
+            0.004414091352373362,
+            0.0016411880496889353,
+            -0.008352942764759064,
+            0.009440355934202671,
+            -0.008264338597655296,
+            -0.0015143232885748148,
+            0.017318058758974075,
+            0.0399201363325119,
+            0.017463047057390213,
+            0.0328962542116642,
+            -0.03708481043577194,
+            -0.026983952149748802,
+            0.011051337234675884,
+            0.014724376611411572,
+            0.007197062950581312,
+            -0.003234046744182706,
+            -0.015417098999023438,
+            0.030930858105421066,
+            0.008618754334747791,
+            0.005271939095109701,
+            -0.009287312626838684,
+            0.007861592806875706,
+            -0.007096376270055771,
+            0.05271133780479431,
+            0.00677820760756731,
+            -0.014643827453255653,
+            0.016182316467165947,
+            -0.016142040491104126,
+            -0.013701403513550758,
+            -0.005654547363519669,
+            -0.010439164005219936,
+            0.013314767740666866,
+            -0.018977370113134384,
+            0.008377106860280037,
+            0.014031654223799706,
+            0.010294175706803799,
+            0.0036508883349597454,
+            0.012299848720431328,
+            0.014015545137226582,
+            0.012541496194899082,
+            0.01657700724899769,
+            0.0003000454162247479,
+            -0.029674291610717773,
+            -0.004651710856705904,
+            -0.002055009128525853,
+            0.0091020492836833,
+            -0.0211199764162302,
+            -0.008707358501851559,
+            0.0003111209371127188,
+            -0.0018264510435983539,
+            0.002591667464002967,
+            -0.028917130082845688,
+            -0.011437973007559776,
+            -0.038341376930475235,
+            0.027306146919727325,
+            0.007950196973979473,
+            0.021893247961997986,
+            0.002742697251960635,
+            -0.028449945151805878,
+            0.010302230715751648,
+            0.02242487110197544,
+            0.0193962249904871,
+            -0.012775088660418987,
+            -0.013411426916718483,
+            -0.030350904911756516,
+            0.0030266328249126673,
+            0.010535823181271553,
+            0.0009686030098237097,
+            0.009561179205775261,
+            0.005984798539429903,
+            -0.0205722413957119,
+            0.009786716662347317,
+            -0.027869991958141327,
+            0.01725361868739128,
+            -0.04945715144276619,
+            0.0055739982053637505,
+            -0.01068081147968769,
+            -0.02092665806412697,
+            -0.005392762832343578,
+            -0.007648137863725424,
+            0.028208298608660698,
+            -0.005690794438123703,
+            -0.022457091137766838,
+            0.016609225422143936,
+            0.016029272228479385,
+            -0.016295084729790688,
+            0.009488685056567192,
+            0.01062442734837532,
+            0.003735464997589588,
+            -0.001605947851203382,
+            0.011123832315206528,
+            0.03215520456433296,
+            0.0035602706484496593,
+            0.01833297684788704,
+            0.010664702393114567,
+            0.009247037582099438,
+            0.006504340562969446,
+            0.0014307535020634532,
+            0.030834197998046875,
+            -0.005235692020505667,
+            0.0060532656498253345,
+            0.007164843380451202,
+            0.001605947851203382,
+            0.015223781578242779,
+            0.001339128939434886,
+            -0.0020117140375077724,
+            0.009730332531034946,
+            -0.010793580673635006,
+            0.010414999909698963,
+            -0.020024508237838745,
+            -0.00027915299870073795,
+            0.01589233987033367,
+            -0.011083557270467281,
+            -0.024116402491927147,
+            -0.00019784874166361988,
+            -0.0164642371237278,
+            0.009182598441839218,
+            0.0017267714720219374,
+            -0.011325204744935036,
+            -0.0051672253757715225,
+            -0.0016885106451809406,
+            -0.005880084820091724,
+            0.022875946015119553,
+            -0.01215486042201519,
+            -0.019702311605215073,
+            -0.018735721707344055,
+            -0.008852346800267696,
+            0.011977652087807655,
+            0.03162357956171036,
+            -0.015473484061658382,
+            0.00608145771548152,
+            -0.006762097589671612,
+            0.00014549182378686965,
+            0.027773331850767136,
+            -0.0032642525620758533,
+            -0.0074467649683356285,
+            0.013540305197238922,
+            0.01762414537370205,
+            0.005259856581687927,
+            -0.007132623344659805,
+            -0.011800444684922695,
+            -0.012823417782783508,
+            -0.003695190418511629,
+            0.012815362773835659,
+            -0.006037155631929636,
+            -0.017994670197367668,
+            0.04298100247979164,
+            -0.023053154349327087,
+            0.0038663572631776333,
+            -0.00921481754630804,
+            -0.004893358331173658,
+            -0.0025393106043338776,
+            0.010431109927594662,
+            -0.006480175536125898,
+            -0.04336763918399811,
+            -0.043399859219789505,
+            0.007664247881621122,
+            -0.006645301356911659,
+            -0.0004505715623963624,
+            0.047395095229148865,
+            0.010278066620230675,
+            -0.02946486324071884,
+            0.01726972870528698,
+            -0.025195760652422905,
+            -0.006274775601923466,
+            -0.0050745937041938305,
+            -0.022714847698807716,
+            0.009343696758151054,
+            0.0008794955792836845,
+            -0.011292984709143639,
+            -0.012960351072251797,
+            -0.022231554612517357,
+            -0.009053720161318779,
+            -0.001134735532104969,
+            0.012581770308315754,
+            0.004325487185269594,
+            -0.0036267235409468412,
+            -0.026758413761854172,
+            0.017285838723182678,
+            -0.01168767549097538,
+            0.0012213258305564523,
+            -0.000624255568254739,
+            0.00898122601211071,
+            0.019637873396277428,
+            0.016238700598478317,
+            -0.02360088936984539,
+            -0.017124740406870842,
+            -0.026516767218708992,
+            0.017060300335288048,
+            0.00505042914301157,
+            0.026371778920292854,
+            -0.012436782009899616,
+            0.003979125991463661,
+            -0.020846109837293625,
+            -0.013693348504602909,
+            -0.008642919361591339,
+            -0.014031654223799706,
+            0.005827728193253279,
+            -0.003018577815964818,
+            0.03421726077795029,
+            -0.0012364288559183478,
+            -0.007434682920575142,
+            0.004824891686439514,
+            0.008618754334747791,
+            0.025904593989253044,
+            -0.021796587854623795,
+            -0.017189180478453636,
+            -0.011292984709143639,
+            0.010978843085467815,
+            0.005650519859045744,
+            -0.016883093863725662,
+            -0.02548573724925518,
+            -0.016423963010311127,
+            -0.002593681216239929,
+            -0.024148622527718544,
+            0.006230473518371582,
+            0.02970651164650917,
+            0.014442455023527145,
+            0.002138578798621893,
+            -0.01596483401954174,
+            0.03007703647017479,
+            0.010938568972051144,
+            0.011800444684922695,
+            -0.008022691123187542,
+            0.017930231988430023,
+            -0.020266154780983925,
+            0.005296103656291962,
+            -0.024019744247198105,
+            -0.017301948741078377,
+            -0.013250328600406647,
+            0.03511941060423851,
+            -0.010551933199167252,
+            -0.018735721707344055,
+            0.004748370032757521,
+            -0.005187362432479858,
+            0.01678643375635147,
+            -0.01514323242008686,
+            -0.021103866398334503,
+            0.025195760652422905,
+            0.007535369135439396,
+            0.012074311263859272,
+            -0.03136582300066948,
+            0.0022211417090147734,
+            -0.019122358411550522,
+            0.0024205006193369627,
+            -0.012017927132546902,
+            0.0015173438005149364,
+            0.004639628808945417,
+            0.00766827492043376,
+            0.004305349662899971,
+            0.024841345846652985,
+            0.004365761764347553,
+            0.021168306469917297,
+            -0.016512567177414894,
+            -0.0013491975842043757,
+            0.03724590688943863,
+            0.008026719093322754,
+            -0.001129701267927885,
+            0.007712577003985643,
+            0.018864601850509644,
+            0.006721823010593653,
+            -0.009553124196827412,
+            -0.00437784381210804,
+            -0.0011729963589459658,
+            0.023278692737221718,
+            0.01315366942435503,
+            0.0064600384794175625,
+            -0.0015102957841008902,
+            0.008747633546590805,
+            0.006681548431515694,
+            0.001100502209737897,
+            -0.003042742609977722,
+            -0.013894720934331417,
+            -0.020395034924149513,
+            -0.025082992389798164,
+            0.025421299040317535,
+            0.0031353740487247705,
+            -0.006335187237709761,
+            0.009657838381826878,
+            0.006315050181001425,
+            -0.012066256254911423,
+            0.02878825180232525,
+            -0.005718986503779888,
+            -0.024873564019799232,
+            -0.005598163232207298,
+            -0.003679080633446574,
+            -0.011913212947547436,
+            0.030624771490693092,
+            -0.00027487383340485394,
+            0.0059606339782476425,
+            0.01267842948436737,
+            -0.005489421542733908,
+            -0.021232744678854942,
+            -0.02102331817150116,
+            -0.025936812162399292,
+            -0.007619945798069239,
+            0.015014354139566422,
+            -7.803194603184238e-05,
+            0.028852690011262894,
+            0.0065003130584955215,
+            -0.008497931063175201,
+            -0.0058760573156178,
+            -0.023182032629847527,
+            -0.004478530492633581,
+            0.008489876054227352,
+            -0.017898011952638626,
+            0.002058029640465975,
+            -0.015344604849815369,
+            0.011349369771778584,
+            0.0006912120152264833,
+            -0.01327449269592762,
+            -0.002196976915001869,
+            -0.0005774364108219743,
+            -0.01473243162035942,
+            -0.011003008112311363,
+            -0.005553861148655415,
+            0.01256566122174263,
+            -0.01975064165890217,
+            -0.006447955965995789,
+            -0.011719895526766777,
+            0.019879519939422607,
+            -0.010414999909698963,
+            0.009448410011827946,
+            -0.03357286751270294,
+            0.00329043110832572,
+            -0.0003179172636009753,
+            -0.01622259058058262,
+            0.004114045761525631,
+            -0.0016613253392279148,
+            0.02360088936984539,
+            0.01420886255800724,
+            0.014812980778515339,
+            0.004994044546037912,
+            0.017173070460557938,
+            -0.0015143232885748148,
+            -0.002843383466824889,
+            -0.004998072050511837,
+            -0.00487724831327796,
+            -0.010672756470739841,
+            0.006588917225599289,
+            -0.003246129024773836,
+            0.019637873396277428,
+            0.004945715423673391,
+            -0.012928131967782974,
+            0.0066050267778337,
+            0.01656089723110199,
+            0.013258383609354496,
+            -0.00032295158598572016,
+            -0.00684264674782753,
+            0.01562652736902237,
+            0.06869228184223175,
+            -0.0011226532515138388,
+            0.01809133030474186,
+            0.016246754676103592,
+            -0.03191355615854263,
+            -0.022730957716703415,
+            0.01156685221940279,
+            -0.01575540564954281,
+            0.03418504074215889,
+            0.01109161227941513,
+            0.011204380542039871,
+            -0.014966024085879326,
+            0.00651642307639122,
+            0.01913846842944622,
+            0.0170119721442461,
+            -0.005863975267857313,
+            -0.006041183136403561,
+            -0.019170688465237617,
+            0.023101484403014183,
+            0.010769415646791458,
+            -0.006443928461521864,
+            0.004212718456983566,
+            0.008203926496207714,
+            0.003908645361661911,
+            -0.00013315775140654296,
+            0.01633535884320736,
+            -0.018477965146303177,
+            0.001423705485649407,
+            -0.003340774215757847,
+            0.014925749972462654,
+            -0.006298940163105726,
+            -0.01962176337838173,
+            0.0076400828547775745,
+            -0.0019301580032333732,
+            0.011204380542039871,
+            0.013645019382238388,
+            -0.0025232008192688227,
+            0.00663724634796381,
+            -0.003495831275358796,
+            0.0004875738231930882,
+            0.00816767942160368,
+            -0.01668977551162243,
+            0.003558256896212697,
+            0.0003438440035097301,
+            -0.022022126242518425,
+            0.010914403945207596,
+            0.014837145805358887,
+            0.012082366272807121,
+            0.0018032931257039309,
+            -0.008127405308187008,
+            -0.022505421191453934,
+            -0.008949005976319313,
+            0.003348829224705696,
+            0.0002846907591447234,
+            -0.025340748950839043,
+            -0.001157893449999392,
+            -0.012960351072251797,
+            -0.01726972870528698,
+            0.0056666298769414425,
+            0.012726759538054466,
+            0.004663793370127678,
+            -0.004430200904607773,
+            -0.0016019203467294574,
+            -0.0048329466953873634,
+            -0.01509490329772234,
+            0.0019784874748438597,
+            0.012718704529106617,
+            -0.003324664430692792,
+            0.014224972575902939,
+            0.022151004523038864,
+            0.004055647645145655,
+            0.0039026043377816677,
+            0.0030447563622146845,
+            4.613481451087864e-06,
+            0.01581178978085518,
+            0.01925123669207096,
+            -0.0019513021688908339,
+            0.0013199985260143876,
+            -0.00045661276089958847,
+            0.006464065983891487,
+            0.006472120992839336,
+            0.001662332215346396,
+            -0.011462138034403324,
+            0.003439446911215782,
+            -0.01902570016682148,
+            -0.013201998546719551,
+            0.005622327793389559,
+            0.0047805896028876305,
+            0.02147439308464527,
+            -0.01891293004155159,
+            0.0022553750313818455,
+            0.006544615142047405,
+            0.0072051179595291615,
+            -0.0009097015135921538,
+            -0.008409326896071434,
+            0.018268538638949394,
+            0.006149924360215664,
+            0.0034172958694398403,
+            -0.004168416373431683,
+            -0.011333259753882885,
+            0.007950196973979473,
+            0.00033704767702147365,
+            0.0019331786315888166,
+            -0.006383516825735569,
+            -0.013943050988018513,
+            -0.022618189454078674,
+            0.023649217560887337,
+            0.01726972870528698,
+            0.019798971712589264,
+            -0.01797856017947197,
+            0.006258665584027767,
+            -0.011019118130207062,
+            0.010414999909698963,
+            0.01167962048202753,
+            -0.0017408676212653518,
+            -0.004293267615139484,
+            -0.001821416663005948,
+            -0.021055536344647408,
+            -0.01749526709318161,
+            0.005626355297863483,
+            0.02055613324046135,
+            0.0001618533715372905,
+            -0.009826990775763988,
+            -0.029674291610717773,
+            -0.004389926325529814,
+            -0.005678711924701929,
+            -0.0199439600110054,
+            -0.01150241307914257,
+            -0.015392934903502464,
+            -0.015642637386918068,
+            -0.003228005487471819,
+            0.0022271827328950167,
+            -0.005578025709837675,
+            0.004023428075015545,
+            0.018574625253677368,
+            0.0013532249722629786,
+            0.020411144942045212,
+            0.010068638250231743,
+            0.017189180478453636,
+            0.013596689328551292,
+            -0.013411426916718483,
+            0.006282830610871315,
+            -0.009593399241566658,
+            -0.012050146237015724,
+            0.018026890233159065,
+            -0.014386070892214775,
+            -2.73583791567944e-05,
+            -0.023182032629847527,
+            0.001140776788815856,
+            -0.02395530417561531,
+            0.01892904005944729,
+            -0.034475017338991165,
+            -0.011961543001234531,
+            0.005743151530623436,
+            -0.01714085042476654,
+            -0.01949288509786129,
+            -0.028369395062327385,
+            -0.0008281455375254154,
+            -0.018059110268950462,
+            -0.00408383971080184,
+            -0.016770323738455772,
+            0.0012011885410174727,
+            -0.0028836580459028482,
+            0.010785525664687157,
+            -0.008497931063175201,
+            0.051390331238508224,
+            0.0029923992697149515,
+            0.005537751130759716,
+            -0.01263010036200285,
+            0.001191119896247983,
+            -0.012066256254911423,
+            0.013427536003291607,
+            -0.0016885106451809406,
+            -0.0038200414273887873,
+            0.008409326896071434,
+            0.010704976506531239,
+            -0.0032561977859586477,
+            0.004293267615139484,
+            0.02395530417561531,
+            0.02772500365972519,
+            -0.001674414612352848,
+            -0.0120340371504426,
+            0.004990017041563988,
+            0.009061774238944054,
+            0.014861310832202435,
+            0.008264338597655296,
+            -0.016625335440039635,
+            -0.01267842948436737,
+            0.013572524301707745,
+            0.005328323692083359,
+            -0.011832663789391518,
+            0.01868739351630211,
+            -0.013065065257251263,
+            -0.022714847698807716,
+            0.005392762832343578,
+            -0.00798241700977087,
+            0.012815362773835659,
+            -0.00915843341499567,
+            0.006625164300203323,
+            0.009770606644451618,
+            0.007088321726769209,
+            0.025276310741901398,
+            -0.004192580934613943,
+            -0.004216745961457491,
+            -0.015038518235087395,
+            -0.0017388538690283895,
+            -0.0009162460919469595,
+            0.027563905343413353,
+            0.02875603176653385,
+            0.008779852651059628,
+            -0.059960756450891495,
+            -0.003125305287539959,
+            0.016512567177414894,
+            -0.010527768172323704,
+            -0.0182202085852623,
+            0.015320440754294395,
+            -0.0021828808821737766,
+            0.002150661079213023,
+            0.01689920201897621,
+            0.009907539933919907,
+            -0.005340405739843845,
+            0.014845200814306736,
+            0.03531273081898689,
+            0.009432300925254822,
+            0.008203926496207714,
+            -0.007656192872673273,
+            -0.01865517348051071,
+            -0.029303764924407005,
+            -0.004961824975907803,
+            -0.004389926325529814,
+            0.001964391442015767,
+            0.004196608439087868,
+            -0.00904566515237093,
+            0.037890300154685974,
+            0.018735721707344055,
+            0.01904180832207203,
+            0.006238528527319431,
+            0.008393216878175735,
+            -0.02279539778828621,
+            -0.03805140033364296,
+            0.003969057463109493,
+            -0.024019744247198105,
+            0.0018687392584979534,
+            0.010729141533374786,
+            0.01962176337838173,
+            0.005022237077355385,
+            0.011889047920703888,
+            -0.008360997773706913,
+            0.004212718456983566,
+            -2.6917876311927103e-05,
+            0.005783426109701395,
+            -0.01599705219268799,
+            -0.01950899325311184,
+            -0.009013445116579533,
+            0.0043738167732954025,
+            0.010278066620230675,
+            -0.005408872850239277,
+            -0.005300131160765886,
+            0.005936469417065382,
+            -0.02443859912455082,
+            0.0029541384428739548,
+            -0.011993762105703354,
+            0.0037092864513397217,
+            -0.009464520029723644,
+            0.016947532072663307,
+            0.020233936607837677,
+            -0.00956923421472311,
+            -0.0017650322988629341,
+            0.0009832025971263647,
+            0.02300482615828514,
+            -0.009899485856294632,
+            0.01656089723110199,
+            0.005771343596279621,
+            0.029046008363366127,
+            0.01962176337838173,
+            0.015747351571917534,
+            -0.010382779873907566,
+            0.012936186976730824,
+            0.022618189454078674,
+            -0.007495094556361437,
+            0.013572524301707745,
+            -0.00857042521238327,
+            0.012710649520158768,
+            -0.006133814807981253,
+            -0.010704976506531239,
+            0.0034354194067418575,
+            0.01327449269592762,
+            0.011397698894143105,
+            0.003228005487471819,
+            0.02429361082613468,
+            -0.0053202686831355095,
+            -0.010374724864959717,
+            -0.019090138375759125,
+            0.0036468608304858208,
+            -0.0017569774063304067,
+            0.0043738167732954025,
+            0.01420886255800724,
+            -0.02337535098195076,
+            -0.000712356180883944,
+            0.01533654984086752,
+            -0.011599071323871613,
+            -0.001532446825876832,
+            -0.022618189454078674,
+            -0.01599705219268799,
+            0.009851155802607536,
+            -0.0015102957841008902,
+            -0.005968688987195492,
+            -0.024615807458758354,
+            0.01846185512840748,
+            -0.0029138640966266394,
+            0.002593681216239929,
+            0.036214880645275116,
+            0.008892621845006943,
+            0.039050206542015076,
+            -0.0068063996732234955,
+            0.012001817114651203,
+            -0.011292984709143639,
+            -0.011019118130207062,
+            -0.026001252233982086,
+            0.013604744337499142,
+            -0.01015724241733551,
+            0.019315676763653755,
+            0.009311476722359657,
+            0.006472120992839336,
+            -0.0026118047535419464,
+            0.0050745937041938305,
+            0.0008633857360109687,
+            -0.015779569745063782,
+            0.026001252233982086,
+            -0.02195768617093563,
+            0.028240516781806946,
+            0.005638437811285257,
+            -0.0120340371504426,
+            0.012823417782783508,
+            0.011139941401779652,
+            0.004776562098413706,
+            0.01051165908575058,
+            0.0030447563622146845,
+            -0.0054612294770777225,
+            0.005437064915895462,
+            -0.00634324224665761,
+            -0.04262658953666687,
+            -0.025807933881878853,
+            -0.006109649781137705,
+            -0.003950933925807476,
+            0.0075998082756996155,
+            -0.013419480994343758,
+            -0.021699929609894753,
+            0.0030064955353736877,
+            0.023745877668261528,
+            -0.021699929609894753,
+            0.011429918929934502,
+            0.0018697461346164346,
+            0.011655456386506557,
+            -0.01784968189895153,
+            0.013363096863031387,
+            -0.002575557678937912,
+            0.02996426820755005,
+            0.000515262596309185,
+            -0.0031635661143809557,
+            0.012517331168055534,
+            0.000442516669863835,
+            -0.021571051329374313,
+            -0.00028745963936671615,
+            -0.012452892027795315,
+            -0.02020171657204628,
+            0.02182880789041519,
+            -0.008115322329103947,
+            -0.006556697189807892,
+            -0.004216745961457491,
+            0.024116402491927147,
+            -0.0006927223294042051,
+            0.0011951474007219076,
+            -0.005863975267857313,
+            -0.011736005544662476,
+            0.0182040985673666,
+            -0.02688729204237461,
+            -0.0030306600965559483,
+            -0.007285666652023792,
+            0.0007954224711284041,
+            -0.005739124026149511,
+            -0.005275966599583626,
+            0.017189180478453636,
+            -0.017930231988430023,
+            0.005586080718785524,
+            -0.014426345005631447,
+            -0.003695190418511629,
+            0.0008608686039224267,
+            0.011019118130207062,
+            0.03157525137066841,
+            0.004091894719749689,
+            0.030850308015942574,
+            0.014329686760902405,
+            -0.008594590239226818,
+            0.0012535455171018839,
+            0.01878405176103115,
+            -0.010535823181271553,
+            0.02160327136516571,
+            -0.01308923028409481,
+            -0.008650974370539188,
+            0.007241365034133196,
+            -0.007632027845829725,
+            0.00458324421197176,
+            0.011260765604674816,
+            0.015409043990075588,
+            0.0014871378662064672,
+            0.0009711202001199126,
+            -0.016029272228479385,
+            0.010737196542322636,
+            -0.0046799033880233765,
+            -0.008916785940527916,
+            0.00578745361417532,
+            0.007269557099789381,
+            -0.014007490128278732,
+            0.0009691064478829503,
+            -0.007877702824771404,
+            0.0050141820684075356,
+            -0.01889682002365589,
+            -0.008940950967371464,
+            -0.024470819160342216,
+            -0.00391670037060976,
+            -0.013612799346446991,
+            0.005650519859045744,
+            0.001747915637679398,
+            -0.005952578969299793,
+            0.016174260526895523,
+            0.015844009816646576,
+            0.005924386903643608,
+            0.006045210640877485,
+            -0.02666175551712513,
+            -0.0038764257915318012,
+            0.0007294728420674801,
+            -0.0013038887409493327,
+            0.0016653528437018394,
+            -0.011244655586779118,
+            -0.011421863920986652,
+            -0.018703503534197807,
+            -0.010189462453126907,
+            -0.0027084636967629194,
+            0.014514949172735214,
+            -0.0037233824841678143,
+            -0.0014196779811754823,
+            -0.0058760573156178,
+            0.006971525494009256,
+            0.012509276159107685,
+            0.022618189454078674,
+            -0.014498839154839516,
+            -0.0132100535556674,
+            0.015417098999023438,
+            0.03197799623012543,
+            -0.007998526096343994,
+            0.008066993206739426,
+            -0.010141132399439812,
+            -0.00607340270653367,
+            -0.005578025709837675,
+            -0.01809133030474186,
+            0.010616372339427471,
+            0.00018186478700954467,
+            0.02548573724925518,
+            0.00578745361417532,
+            -0.004013359546661377,
+            0.007813263684511185,
+            0.010350560769438744,
+            0.012082366272807121,
+            0.00754745164886117,
+            0.030512001365423203,
+            0.0005930428160354495,
+            0.02360088936984539,
+            -0.0012686484260484576,
+            -0.010938568972051144,
+            -0.0021869083866477013,
+            0.0017297921003773808,
+            0.00484502874314785,
+            0.010076693259179592,
+            0.0281760785728693,
+            0.0035220098216086626,
+            -0.004772534593939781,
+            0.01633535884320736,
+            0.006391571834683418,
+            -0.0005880084936507046,
+            -0.012718704529106617,
+            -0.01227568369358778,
+            -0.006335187237709761,
+            -0.0007934087188914418,
+            0.002002652268856764,
+            -0.016440073028206825,
+            -0.03550604730844498,
+            0.010020309127867222,
+            0.006089512724429369,
+            -0.016995862126350403,
+            -0.01773691363632679,
+            -0.006178116425871849,
+            0.019122358411550522,
+            0.00040048008668236434,
+            -0.009061774238944054,
+            -0.0024305693805217743,
+            0.007575643714517355,
+            0.022263772785663605,
+            -0.0014569320483133197,
+            0.031075846403837204,
+            -0.015860119834542274,
+            -0.01479687076061964,
+            0.012960351072251797,
+            0.0024245281238108873,
+            0.008288503624498844,
+            -0.025195760652422905,
+            0.010125023312866688,
+            -0.006931250914931297,
+            0.022747067734599113,
+            0.007559533696621656,
+            0.010390834882855415,
+            -0.012251519598066807,
+            -0.006242556031793356,
+            0.016367578878998756,
+            -0.010906348936259747,
+            -0.004708095453679562,
+            -0.003236060496419668,
+            -0.026242898777127266,
+            0.0012092434335500002,
+            0.011526577174663544,
+            -0.0015515772392973304,
+            0.015803735703229904,
+            -0.006886948831379414,
+            -0.002825259929522872,
+            -0.0003078486188314855,
+            -0.00019105241517536342,
+            -0.008272393606603146,
+            -0.041982196271419525,
+            -0.002799081616103649,
+            0.02078166976571083,
+            -0.005102786235511303,
+            -0.02210267446935177,
+            0.01725361868739128,
+            0.003308554645627737,
+            0.0036851216573268175,
+            -0.01328254770487547,
+            -0.021571051329374313,
+            -0.015827899798750877,
+            0.01308923028409481,
+            -0.018252428621053696,
+            -0.023536449298262596,
+            -0.0016804557526484132,
+            -0.017946342006325722,
+            0.002428555628284812,
+            -0.02197379618883133,
+            -0.02055613324046135,
+            -0.021780479699373245,
+            -0.005497476551681757,
+            0.004192580934613943,
+            -0.025662945583462715,
+            0.026371778920292854,
+            -0.008441546931862831,
+            -0.006149924360215664,
+            -0.014861310832202435,
+            0.008892621845006943,
+            0.0017126754391938448,
+            0.021764369681477547,
+            -0.02679063379764557,
+            0.009585344232618809,
+            -0.0023681437596678734,
+            -0.015417098999023438,
+            -0.005944523960351944,
+            -0.0015868174377828836,
+            0.009673947468400002,
+            0.03499053418636322,
+            0.006524477619677782,
+            -0.021877137944102287,
+            -0.023343130946159363,
+            0.012847582809627056,
+            0.003991208504885435,
+            -0.010801635682582855,
+            -0.010737196542322636,
+            -0.01715696044266224,
+            -0.015425154007971287,
+            0.014579388312995434,
+            0.01367723848670721,
+            0.018107440322637558,
+            -0.0032380742486566305,
+            0.008199899457395077,
+            -0.01880016177892685,
+            -0.005747179035097361,
+            0.0033790350425988436,
+            0.006262693088501692,
+            -0.018397416919469833,
+            0.035892684012651443,
+            0.005332351196557283,
+            -0.004454365465790033,
+            -0.00581161817535758,
+            0.0052276370115578175,
+            0.0028333149384707212,
+            -0.008844291791319847,
+            -0.01316172443330288,
+            0.00737427081912756,
+            -0.009085939265787601,
+            0.007003745064139366,
+            0.009134269319474697,
+            -0.005541778635233641,
+            -0.019895629957318306,
+            -0.003364939009770751,
+            0.011727950535714626,
+            -0.017237508669495583,
+            -0.01110772229731083,
+            -0.020523913204669952,
+            -0.003024619072675705,
+            0.003979125991463661,
+            -0.00014410738367587328,
+            0.0018687392584979534,
+            -0.01639174297451973,
+            -0.004176471382379532,
+            -0.02911044843494892,
+            0.0008155597024597228,
+            -0.02182880789041519,
+            -0.018848491832613945,
+            0.011921267956495285,
+            0.00975449662655592,
+            0.00604923814535141,
+            0.003705258946865797,
+            0.00047549145529046655,
+            0.014120258390903473,
+            0.002132537541911006,
+            -0.016528677195310593,
+            0.022408761084079742,
+            -0.0063593522645533085,
+            0.015594307333230972,
+            0.029400425031781197,
+            0.01380611676722765,
+            -0.00687486631795764,
+            -0.017801353707909584,
+            -0.01003641914576292,
+            -0.02210267446935177,
+            0.0047685070894658566,
+            -0.01772080361843109,
+            -0.024277500808238983,
+            0.014635773375630379,
+            -0.023069264367222786,
+            0.0011488316813483834,
+            -0.01693142205476761,
+            0.006737933028489351,
+            -0.00637546181678772,
+            0.012597880326211452,
+            0.017833571881055832,
+            0.0016703871078789234,
+            0.0027789443265646696,
+            -0.011880993843078613,
+            -0.0036106137558817863,
+            0.006689603440463543,
+            -0.005175280384719372,
+            -0.004905440844595432,
+            0.001707641058601439,
+            -0.010922458954155445,
+            -0.019766751676797867,
+            -0.012775088660418987,
+            -0.025646835565567017,
+            -0.0010632482590153813,
+            0.0049215503968298435,
+            0.006246583536267281,
+            -0.004273130092769861,
+            -0.00355221563950181,
+            0.0033448017202317715,
+            -0.008594590239226818,
+            0.0031917584128677845,
+            -0.0010144152911379933,
+            -0.005900222342461348,
+            0.006170061882585287,
+            0.0034515291918069124,
+            0.0030387151055037975,
+            0.01427330169826746,
+            0.00243862415663898,
+            -0.005171252880245447,
+            -0.038244716823101044,
+            0.0062546380795538425,
+            -0.018848491832613945,
+            0.001827457919716835,
+            -0.011446028016507626,
+            7.104683027137071e-05,
+            -0.006717795506119728,
+            -0.006967497989535332,
+            0.016770323738455772,
+            0.003882467048242688,
+            0.010116968303918839,
+            0.015940668061375618,
+            -0.010600262321531773,
+            -0.008949005976319313,
+            0.012501221150159836,
+            -9.49441164266318e-06,
+            -0.00481280917301774,
+            0.0016542773228138685,
+            -0.01566680148243904,
+            0.008916785940527916,
+            -0.00951285008341074,
+            0.009593399241566658,
+            -0.012944241985678673,
+            0.007386353332549334,
+            -0.008731523528695107,
+            -0.011405753903090954,
+            0.00024882121942937374,
+            0.008763742633163929,
+            0.01581178978085518,
+            0.003097113221883774,
+            0.006025073118507862,
+            0.0035602706484496593,
+            0.02147439308464527,
+            -0.01255760621279478,
+            0.0009464520262554288,
+            -0.005465256981551647,
+            -0.023455901071429253,
+            -0.015844009816646576,
+            0.025324638932943344,
+            0.03937240317463875,
+            -0.011711840517818928,
+            -0.005429009906947613,
+            0.0011880993843078613,
+            0.009085939265787601,
+            0.0023983498103916645,
+            -0.004490613006055355,
+            -0.01309728529304266,
+            -0.02608180046081543,
+            -0.02018560655415058,
+            0.004555052146315575,
+            0.01760803535580635,
+            -0.010721086524426937,
+            -0.004013359546661377,
+            -0.005940496921539307,
+            -0.00431340467184782,
+            -0.0028534522280097008,
+            -0.008642919361591339,
+            -0.00551761407405138,
+            -0.005723014008253813,
+            -0.004514777567237616,
+            -0.022151004523038864,
+            0.009424245916306973,
+            0.011663510464131832,
+            0.008369052782654762,
+            0.0028172051534056664,
+            0.020829999819397926,
+            -0.02453525923192501,
+            -0.0067057134583592415,
+            -0.000449816434411332,
+            -0.009786716662347317,
+            0.02500244230031967,
+            0.00798241700977087,
+            -0.016947532072663307,
+            0.0012626072857528925,
+            0.006363379769027233,
+            0.00024693337036296725,
+            0.018945150077342987,
+            -0.010439164005219936,
+            0.021554941311478615,
+            -0.011486303061246872,
+            -0.004454365465790033,
+            -0.0022734985686838627,
+            0.006882921326905489,
+            0.03228408098220825,
+            0.012324013747274876,
+            -0.00950479507446289,
+            0.01068886648863554,
+            0.017688583582639694,
+            0.011437973007559776,
+            -0.009947814978659153,
+            0.021989906206727028,
+            -0.004196608439087868,
+            0.015296275727450848,
+            0.016528677195310593,
+            0.008344887755811214,
+            -0.011454083025455475,
+            -0.008465711027383804,
+            0.022167114540934563,
+            0.003783794352784753,
+            -0.02664564549922943,
+            -0.010012254118919373,
+            0.0029319876339286566,
+            0.023552559316158295,
+            0.007712577003985643,
+            0.0028756032697856426,
+            0.001929151127114892,
+            0.024631917476654053,
+            -0.005662602372467518,
+            0.027515575289726257,
+            -0.016754213720560074,
+            0.004542969632893801,
+            -0.001554597751237452,
+            0.014200807549059391,
+            -0.01831686682999134,
+            0.001941233524121344,
+            -0.0036448470782488585,
+            0.004631573799997568,
+            0.008892621845006943,
+            0.007120541296899319,
+            0.019283456727862358,
+            -0.004933632910251617,
+            -0.013258383609354496,
+            0.008457656018435955,
+            0.008868456818163395,
+            -0.01736638695001602,
+            0.027547795325517654,
+            0.020378924906253815,
+            0.00798241700977087,
+            -0.016866983845829964,
+            0.005130978301167488,
+            0.04043565317988396,
+            0.018606843426823616,
+            0.013016736134886742,
+            -0.010358615778386593,
+            -0.022634299471974373,
+            0.007587726227939129,
+            0.009359806776046753,
+            0.0024144595954567194,
+            0.004369789268821478,
+            0.004164388868957758,
+            -0.014321631751954556,
+            0.0001999883388634771,
+            0.03592490404844284,
+            -0.010785525664687157,
+            -0.01599705219268799,
+            -0.004301322158426046,
+            -0.011727950535714626,
+            -0.018477965146303177,
+            0.010414999909698963,
+            0.004865166265517473,
+            -0.020362814888358116,
+            0.00950479507446289,
+            0.001946267788298428,
+            -0.0164481271058321,
+            0.0041281417943537235,
+            0.0032199507113546133,
+            -0.003926768898963928,
+            -0.004841001238673925,
+            0.001975466962903738,
+            -0.009118159301578999,
+            0.005702876951545477,
+            -0.0011528590694069862,
+            -0.003743519773706794,
+            0.010052529163658619,
+            0.01654478721320629,
+            0.0024547341745346785,
+            0.010431109927594662,
+            0.0019734532106667757,
+            0.013298657722771168,
+            -0.018300756812095642,
+            0.00657280720770359,
+            -0.005972716491669416,
+            0.001719723455607891,
+            -0.0024104320909827948,
+            -0.005429009906947613,
+            -0.03685927018523216,
+            0.0029984405264258385,
+            -0.0011961542768403888,
+            0.014450510032474995,
+            -0.01736638695001602,
+            0.00032269986695609987,
+            0.01309728529304266,
+            -0.014361905865371227,
+            -0.00022830638044979423,
+            0.01846185512840748,
+            0.001276703318580985,
+            0.02279539778828621,
+            0.012315958738327026,
+            0.013371151871979237,
+            0.002072125906124711,
+            -0.02326258271932602,
+            0.02442248910665512,
+            -0.009915594942867756,
+            0.004458392970263958,
+            0.0211360864341259,
+            0.03312179446220398,
+            -0.021168306469917297,
+            -0.008546260185539722,
+            -0.015344604849815369,
+            -0.002984344493597746,
+            0.021667709574103355,
+            0.0040757847018539906,
+            0.028192188590765,
+            0.013226163573563099,
+            0.01648840308189392,
+            -0.025936812162399292,
+            0.0021244827657938004,
+            0.014378015883266926,
+            -0.007696467451751232,
+            -0.017930231988430023,
+            -0.01736638695001602,
+            -0.013234218582510948,
+            0.010648592375218868,
+            -0.02149050123989582,
+            0.026484547182917595,
+            -0.0009686030098237097,
+            0.01678643375635147,
+            0.009013445116579533,
+            -0.010471384041011333,
+            0.004929605405777693,
+            0.00627880310639739,
+            0.0037636570632457733,
+            -0.01726972870528698,
+            0.00510681327432394,
+            -0.007712577003985643,
+            0.0061539518646895885,
+            -0.004043565131723881,
+            0.029335984960198402,
+            -0.008010609075427055,
+            0.008489876054227352,
+            -0.011969597078859806,
+            -0.011309094727039337,
+            0.011317149735987186,
+            0.000714873312972486,
+            -0.012404562905430794,
+            0.0065123955719172955,
+            -0.002897754078730941,
+            0.01455522421747446,
+            -0.015932613983750343,
+            0.00405162014067173,
+            0.030334794893860817,
+            -0.01566680148243904,
+            -0.009327586740255356,
+            0.0030326738487929106,
+            0.0057673160918056965,
+            0.0034656254574656487,
+            0.0019684189464896917,
+            0.012847582809627056,
+            -0.00378983560949564,
+            -0.010946623980998993,
+            0.012348177842795849,
+            -0.011486303061246872,
+            0.008530150167644024,
+            0.0016834763810038567,
+            -0.008804017677903175,
+            0.029046008363366127,
+            -0.010133077390491962,
+            0.01257371623069048,
+            0.021087756380438805,
+            -0.023069264367222786,
+            -0.0077528515830636024,
+            -0.0105599882081151,
+            -0.019074028357863426,
+            -0.011518522165715694,
+            -0.005022237077355385,
+            0.00107231002766639,
+            0.0010400903411209583,
+            0.0060129910707473755,
+            3.240842852392234e-05,
+            0.004784617107361555,
+            0.0011589003261178732,
+            -0.0033689665142446756,
+            -0.010914403945207596,
+            0.03006092645227909,
+            -0.018977370113134384,
+            0.024728575721383095,
+            -0.00021584643400274217,
+            0.016947532072663307,
+            0.0017348263645544648,
+            -0.008034773170948029,
+            0.014136368408799171,
+            0.018300756812095642,
+            -0.019589543342590332,
+            -0.008123377338051796,
+            0.004619491286575794,
+            -0.0026601343415677547,
+            -0.009931704960763454,
+            -0.011760169640183449,
+            0.0038280964363366365,
+            -0.028949350118637085,
+            -0.0044221458956599236,
+            -0.011325204744935036,
+            -0.0035441608633846045,
+            -0.00777701660990715,
+            -0.01833297684788704,
+            -0.009118159301578999,
+            -0.01216291543096304,
+            -0.012009872123599052,
+            -0.008280448615550995,
+            -0.018945150077342987,
+            0.006186171434819698,
+            -0.02395530417561531,
+            0.014990189112722874,
+            -0.014200807549059391,
+            -0.005147087853401899,
+            0.015650691464543343,
+            0.03254184126853943,
+            0.006391571834683418,
+            0.0010622413828969002,
+            0.0016713939839974046,
+            0.004269102588295937,
+            0.0031696073710918427,
+            -0.0152721107006073,
+            -0.016641445457935333,
+            -0.0032380742486566305,
+            -0.001123660011216998,
+            0.02020171657204628,
+            0.01846185512840748,
+            -0.0036972041707485914,
+            -0.005384707823395729,
+            0.00869930349290371,
+            -0.002859493251889944,
+            -0.008042828179895878,
+            0.005324296187609434,
+            0.003431391902267933,
+            0.029642071574926376,
+            -0.011808499693870544,
+            -0.006077430211007595,
+            0.016085656359791756,
+            0.008457656018435955,
+            0.006798344664275646,
+            -0.010841909795999527,
+            0.00798241700977087,
+            0.006484203040599823,
+            0.017527485266327858,
+            0.0111641064286232,
+            0.0035723529290407896,
+            -0.010173352435231209,
+            -0.010817745700478554,
+            0.0006177109898999333,
+            0.006979580037295818,
+            -0.004446310922503471,
+            -0.022022126242518425,
+            0.0016673665959388018,
+            -0.0008487862069159746,
+            -0.021748259663581848,
+            -0.017060300335288048,
+            0.032815705984830856,
+            0.012879802845418453,
+            0.009867265820503235,
+            -0.0027708893176168203,
+            0.022505421191453934,
+            0.008449601009488106,
+            0.0018314853077754378,
+            0.008812072686851025,
+            0.0012213258305564523,
+            0.006395599339157343,
+            0.003228005487471819,
+            -0.03595712035894394,
+            0.004800726659595966,
+            -0.0024023770820349455,
+            0.006762097589671612,
+            -0.0031514838337898254,
+            0.004269102588295937,
+            -0.0044745029881596565,
+            -0.010390834882855415,
+            -0.02092665806412697,
+            0.022054346278309822,
+            0.015634581446647644,
+            0.021587161347270012,
+            -0.005437064915895462,
+            -0.0020902494434267282,
+            0.007913949899375439,
+            0.025195760652422905,
+            0.0005698849563486874,
+            0.01633535884320736,
+            0.02372976765036583,
+            0.014893529936671257,
+            0.020169496536254883,
+            -0.0031837034039199352,
+            0.04001679643988609,
+            -0.00114581105299294,
+            -0.0026359695475548506,
+            -0.0025513931177556515,
+            -0.010495549067854881,
+            0.01315366942435503,
+            -7.123561954358593e-05,
+            -0.005038346629589796,
+            0.006029100622981787,
+            0.0003780773840844631,
+            0.011639346368610859,
+            -0.01068081147968769,
+            -0.010978843085467815,
+            0.0021808671299368143,
+            0.019412335008382797,
+            0.01227568369358778,
+            0.024116402491927147,
+            -0.021345512941479683,
+            0.005904249846935272,
+            -0.027209488674998283,
+            -0.012968406081199646,
+            0.0032481427770107985,
+            0.004204663448035717,
+            -0.01456327922642231,
+            -0.021877137944102287,
+            -0.009802826680243015,
+            0.011429918929934502,
+            0.007031937129795551,
+            0.010382779873907566,
+            0.0071004037745296955,
+            -0.0003302513505332172,
+            0.006717795506119728,
+            -0.01997617818415165,
+            -0.020475583150982857,
+            0.0019543226808309555,
+            0.0013421495677903295,
+            -0.022022126242518425,
+            -0.006991662550717592,
+            0.010302230715751648,
+            -6.689351721433923e-05,
+            -0.016472293063998222,
+            0.016254810616374016,
+            -0.01598094217479229,
+            -0.01109161227941513,
+            -0.0014670006930828094,
+            0.007688412442803383,
+            0.0024587614461779594,
+            0.015127122402191162,
+            -0.009553124196827412,
+            0.009609508328139782,
+            0.011760169640183449,
+            -0.013234218582510948,
+            0.024454709142446518,
+            -0.0023399516940116882,
+            -0.015827899798750877,
+            0.016303138807415962,
+            -0.010060583241283894,
+            -0.02550184726715088,
+            0.025775713846087456,
+            -0.013983325101435184,
+            -0.010294175706803799,
+            -0.024631917476654053,
+            -0.019992288202047348,
+            -0.007845482788980007,
+            -0.01162323635071516,
+            -0.034926094114780426,
+            -0.018236318603157997,
+            0.029545413330197334,
+            0.018590733408927917,
+            0.0211360864341259,
+            0.00325821153819561,
+            0.003961002454161644,
+            0.013814171776175499,
+            -0.01566680148243904,
+            0.0010541864903643727,
+            0.0017771146958693862,
+            9.546327783027664e-05,
+            0.016963642090559006,
+            -0.001924116862937808,
+            -0.0028051226399838924,
+            -0.017914121970534325,
+            -0.020636681467294693,
+            0.0041885534301400185,
+            0.002146633807569742,
+            0.019235126674175262,
+            0.014659937471151352,
+            -0.02490578405559063,
+            0.005888139829039574,
+            -0.013403371907770634,
+            0.03428170084953308,
+            -0.01416053343564272,
+            0.022376542910933495,
+            0.00863486435264349,
+            0.02195768617093563,
+            0.0026681891176849604,
+            -0.00037128105759620667,
+            0.008779852651059628,
+            0.010334450751543045,
+            0.0034636117052286863,
+            0.002946083666756749,
+            -0.001038076588883996,
+            8.589807112002745e-05,
+            -0.004897385835647583,
+            0.03075364977121353,
+            0.012710649520158768,
+            -0.0016975724138319492,
+            -0.018171878531575203,
+            -0.009585344232618809,
+            -0.0062143635004758835,
+            -0.01725361868739128,
+            -0.010141132399439812,
+            0.029915938153862953,
+            0.003501872532069683,
+            0.0007470929995179176,
+            3.530316462274641e-05,
+            0.0032159232068806887,
+            0.028385505080223083,
+            -0.02337535098195076,
+            -0.009738387539982796,
+            -0.0036972041707485914,
+            -0.004168416373431683,
+            0.014974079094827175,
+            0.0021124002523720264,
+            -0.006709740497171879,
+            -0.0015425154706463218,
+            0.012549551203846931,
+            0.013846391811966896,
+            0.002241278998553753,
+            -0.006504340562969446,
+            -0.009295366704463959,
+            -0.022698739543557167,
+            0.0036911629140377045,
+            0.0006811433704569936,
+            0.0085140410810709,
+            0.010841909795999527,
+            -0.005698849447071552,
+            -0.02843383513391018,
+            -0.009432300925254822,
+            0.007390380837023258,
+            0.004071757197380066,
+            -0.0037314374931156635,
+            0.005984798539429903,
+            -0.006681548431515694,
+            0.006826536729931831,
+            0.019911739975214005,
+            0.024277500808238983,
+            0.020958878099918365,
+            0.007869647815823555,
+            0.020974988117814064,
+            0.028949350118637085,
+            -0.007015827111899853,
+            0.0006962463376112282,
+            0.00956923421472311,
+            0.014224972575902939,
+            0.0014670006930828094,
+            0.017334168776869774,
+            0.002881644293665886,
+            -0.0058075906708836555,
+            -0.01110772229731083,
+            -0.0008281455375254154,
+            0.0317041277885437,
+            -0.016327304765582085,
+            -0.00031187606509774923,
+            0.006141869351267815,
+            0.019412335008382797,
+            0.009714222513139248,
+            0.004675875883549452,
+            0.005316241178661585,
+            0.013483921065926552,
+            -0.010463329032063484,
+            -0.012710649520158768,
+            -0.0021345512941479683,
+            0.00498599000275135,
+            0.011671565473079681,
+            -0.01626286469399929,
+            0.0032300192397087812,
+            -0.021313294768333435,
+            0.009891430847346783,
+            -0.019090138375759125,
+            -0.011591016314923763,
+            0.0024668164551258087,
+            0.006544615142047405,
+            0.006298940163105726,
+            0.009271202608942986,
+            0.029545413330197334,
+            -0.0005583059974014759,
+            0.034475017338991165,
+            -0.005078621208667755,
+            -0.017785243690013885,
+            -0.016053438186645508,
+            0.00038210483035072684,
+            -0.020266154780983925,
+            -0.003217936959117651,
+            0.016399798914790154,
+            -0.007184980437159538,
+            -0.009746442548930645,
+            0.00451880507171154,
+            0.010253901593387127,
+            -0.0063593522645533085,
+            -0.005525668617337942,
+            -0.037278126925230026,
+            0.021458283066749573,
+            -0.0013814171543344855,
+            0.010326395742595196,
+            0.019122358411550522,
+            0.025807933881878853,
+            -0.01428135670721531,
+            -0.0026641618460416794,
+            0.0181879885494709,
+            0.003195785917341709,
+            0.023633107542991638,
+            -0.0047805896028876305,
+            0.008433491922914982,
+            -0.018300756812095642,
+            0.007732714526355267,
+            -0.022167114540934563,
+            -0.010616372339427471,
+            0.004651710856705904,
+            -0.027676673606038094,
+            0.0012646210379898548,
+            -0.0002251599362352863,
+            -0.013491976074874401,
+            -0.004764479584991932,
+            -0.0021667710971087217,
+            0.0012515317648649216,
+            -0.007555506192147732,
+            -0.014708267524838448,
+            -0.01639174297451973,
+            -0.027290038764476776,
+            0.00852209609001875,
+            0.0020701121538877487,
+            -0.003348829224705696,
+            0.0015102957841008902,
+            0.005247774533927441,
+            0.007229282520711422,
+            0.012815362773835659,
+            -0.007201090455055237,
+            0.060991786420345306,
+            -0.014047764241695404,
+            0.0020388993434607983,
+            -0.00508264871314168,
+            0.018606843426823616,
+            0.00868319347500801,
+            0.0027366559952497482,
+            -0.011832663789391518,
+            0.0030105228070169687,
+            -0.015473484061658382,
+            -0.013943050988018513,
+            -0.005932441912591457,
+            -5.896446600672789e-05,
+            0.002843383466824889,
+            -0.019702311605215073,
+            -0.0011528590694069862,
+            -0.01416053343564272,
+            2.218246845586691e-05,
+            0.002350020222365856,
+            -0.007986444048583508,
+            0.017785243690013885,
+            -0.018622953444719315,
+            -0.0085140410810709,
+            0.015167397446930408,
+            0.005179307889193296,
+            0.007881729863584042,
+            -0.013306712731719017,
+            -0.012533441185951233,
+            0.000388649437809363,
+            -0.001509288907982409,
+            -0.0013310740469023585,
+            -0.0091020492836833,
+            -0.008151569403707981,
+            -0.012654264457523823,
+            -0.005884112324565649,
+            -0.0018234304152429104,
+            0.0009761545225046575,
+            0.011236600577831268,
+            -0.008066993206739426,
+            0.005702876951545477,
+            -0.020298374816775322,
+            0.021635491400957108,
+            -0.003149470081552863,
+            -0.0211199764162302,
+            -0.008763742633163929,
+            0.01586817391216755,
+            0.0007712577353231609,
+            0.0006418757257051766,
+            0.03157525137066841,
+            0.004059675149619579,
+            -0.0011045295977964997,
+            -0.005453174468129873,
+            -0.0013743691379204392,
+            -0.007720632012933493,
+            0.020974988117814064,
+            -0.00757967121899128,
+            0.010398889891803265,
+            0.003677066881209612,
+            0.0211521964520216,
+            -0.025759605690836906,
+            0.006766125094145536,
+            -0.011003008112311363,
+            -0.0026077774818986654,
+            -0.00857042521238327,
+            -0.012863692827522755,
+            0.008868456818163395,
+            -0.02656509540975094,
+            -0.031043626368045807,
+            0.006262693088501692,
+            -0.012106531299650669,
+            -0.03785808011889458,
+            0.008216009475290775,
+            -0.018365196883678436,
+            -0.004393953830003738,
+            0.004365761764347553,
+            -0.014966024085879326,
+            -0.01726972870528698,
+            -0.012058201245963573,
+            -0.01609371230006218,
+            -0.0010884198127314448,
+            0.015634581446647644,
+            -0.02429361082613468,
+            -0.013757787644863129,
+            -0.006649328861385584,
+            0.007595780771225691,
+            -0.006371434312313795,
+            -0.0039247553795576096,
+            -0.02244098111987114,
+            0.007160815875977278,
+            -0.0059082768857479095,
+            0.01022168155759573,
+            -0.009673947468400002,
+            0.008264338597655296,
+            0.0252602007240057,
+            0.0036911629140377045,
+            -0.002927960129454732,
+            0.005002099554985762,
+            0.011792389675974846,
+            -0.0006484203040599823,
+            0.0293682049959898,
+            -0.007031937129795551,
+            0.031526919454336166,
+            0.0020439336076378822,
+            0.003568325424566865,
+            -0.015602362342178822,
+            -0.004655738361179829,
+            0.012622045353055,
+            0.008034773170948029,
+            -0.013717513531446457,
+            -0.003211895702406764,
+            -0.027789441868662834,
+            0.012871747836470604,
+            -0.023536449298262596,
+            -0.0039750984869897366,
+            0.005541778635233641,
+            -0.018252428621053696,
+            0.009400080889463425,
+            0.0028917130548506975,
+            0.03238074108958244,
+            0.017801353707909584,
+            -0.0016633390914648771,
+            0.002116427756845951,
+            0.007877702824771404,
+            -0.0020127207972109318,
+            0.019541213288903236,
+            0.004748370032757521,
+            -0.017285838723182678,
+            -0.015650691464543343,
+            -0.019573433324694633,
+            0.0007576650241389871,
+            0.007901867851614952,
+            -0.01051165908575058,
+            0.012412617914378643,
+            -0.0013975270558148623,
+            0.022134894505143166,
+            -0.00551761407405138,
+            -0.008876511827111244,
+            -0.01315366942435503,
+            -0.01268648449331522,
+            0.01602121815085411,
+            -0.001299861236475408,
+            0.016423963010311127,
+            0.007785071153193712,
+            0.010302230715751648,
+            -0.011115777306258678,
+            -0.00014310052210930735,
+            -0.01602121815085411,
+            -0.005195417441427708,
+            -0.007535369135439396,
+            -0.0008845299016684294,
+            0.03937240317463875,
+            -0.006230473518371582,
+            -0.005863975267857313,
+            0.0012736828066408634,
+            -0.0117521146312356,
+            0.024358050897717476,
+            0.013798062689602375,
+            0.022457091137766838,
+            -0.010414999909698963,
+            0.00035970209864899516,
+            -0.008175734430551529,
+            0.01842963509261608,
+            0.006665438879281282,
+            -0.0037314374931156635,
+            -0.011373533867299557,
+            -0.003958988469094038,
+            0.006492258049547672,
+            0.017543595284223557,
+            0.00019470229744911194,
+            -0.015972888097167015,
+            -0.004498667549341917,
+            0.004156333860009909,
+            0.004341596737504005,
+            0.009408135898411274,
+            -0.004095922224223614,
+            0.019444555044174194,
+            0.008155597373843193,
+            -0.016383688896894455,
+            -0.00754745164886117,
+            -0.015441264025866985,
+            0.0019059933256357908,
+            0.013338932767510414,
+            -0.015392934903502464,
+            0.009134269319474697,
+            0.005352488253265619,
+            -0.028611043468117714,
+            0.013886665925383568,
+            -0.012613990344107151,
+            -0.004760452080518007,
+            0.028095528483390808,
+            0.006383516825735569,
+            -0.02300482615828514,
+            -0.005424982402473688,
+            0.002648051828145981,
+            -0.014498839154839516,
+            0.02219933457672596,
+            -0.006439901422709227,
+            0.003471666481345892,
+            -0.007760906592011452,
+            -0.004869193769991398,
+            0.027531685307621956,
+            -0.00200970028527081,
+            -0.0009978021262213588,
+            0.007325941231101751
+          ],
+          "keyphrases": [
+            "Capped Expert Choice",
+            "entropy-regularized linear programming",
+            "validation perplexity",
+            "load balanced training",
+            "fine-tuning results"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "7af06e8c-2560-4267-bd74-c94a92efdcd1",
+      "type": "child",
+      "source": {
+        "id": "b570d71e-4f83-486e-b70f-b0c6217db455",
+        "properties": {
+          "page_content": "Mixture-Of-Experts With Expert Choice Routing\n\nYanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon Google, Mountain View, CA, USA {yanqiz, taole, hanxiaol, dunan, huangyp, vzhao, adai, zhifengc, qvl, jlaudon}@google.com\n\nAbstract\n\nSparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choice method. Instead of letting tokens select the top-k experts, we have experts selecting the top-k tokens. As a result, each token can be routed to a variable number of experts and each expert can have a fixed bucket size. We systematically study pre-training speedups using the same computational resources of the Switch Transformer top-1 and GShard top-2 gating of prior work and find that our method improves training convergence time by more than 2×. For the same computational cost, our method demonstrates higher performance in fine-tuning 11 selected tasks in the GLUE and SuperGLUE benchmarks. For a smaller activation cost, our method outperforms the T5 dense model in 7 out of the 11 tasks.\n\n1 Introduction\n\nScaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27]. The final model quality has been found to have a power-law relationship with the amount of data, model size, and compute time [16, 20]. However, training efficiency, which is defined as the total amount of computation used to achieve superior model quality than the state of the art system [21], should receive greater attention as we increase our efforts towards green AI [29].\n\nSparsely gated mixture-of-experts [31] (MoE) provides an effective way to scale model capacity given a fixed computational cost, and has recently played an important role in increasing the training efficiency of large-scale language models [10, 21]. MoE operate by adopting a number of experts, each as a sub-network, and by activating only one or a few experts for each input token. A gating network must be chosen and optimized in order to route each token to the most suited expert(s). For example, recent work has implemented sparse routing via k-means clustering [12], linear assignment to maximize token-expert affinities [22], or hashing [8, 28]. Many of the prior work use a routing strategy concerning the token choice, where each token selects the best one or two experts.\n\nWe argue that the independent token choice of prior work often leads to an imbalanced load of experts, which causes training inefficiency and sub-optimal training of the model. In order to mitigate this\n\nissue, previous sparsely gated networks introduce additional auxiliary losses as regularization to prevent too many tokens being routed to a single expert, but the effectiveness is still limited. Recent approaches [8, 22, 28] explore alternative strategies for routing, but they focus on pre-training only and do not demonstrate performance gain on downstream tasks. Moreover, none of the previous methods consider allocating a variable number of experts to each token based on importance, which can be beneficial.\n\nWe propose a very simple yet effective routing method we are calling expert choice. Unlike conventional MoE where tokens select one or two top-scoring experts, our method lets each expert pick the top-k tokens. Our method guarantees perfect load balancing, allows a variable number of experts for each token, and achieves substantial gains in training efficiency and downstream performance as demonstrated in our experiments. Our major contributions include:\n\nWe identify common pitfalls in conventional MoE such as load imbalance as described in Section 3.1. We then propose a heterogeneous, expert choice method to provide a fluid allocation of model parameters based on a learnt token-to-expert importance. This method intrinsically guarantees load balance without imposing an auxiliary loss.\n\nWe show our method provides over 2× faster training convergence in a 8B/64E (8 billion activated parameters, 64 experts) model, compared to the top-1 and top-2 gating counterparts in Switch Transformer [10] and GShard [21].\n\nWe show our method demonstrates strong scaling when increasing the number of experts from 16 to 128, evaluated in training perplexity.\n\nWe show our method demonstrates strong performance on downstream tasks selected from GLUE and SuperGLUE at all the evaluated scales. More specifically, our 8B/64E model outperforms a T5 11B dense model in 7 out of 11 tasks evaluated.\n\n2 Related Work\n\nScaling: Various approaches have been proposed to scale up neural network capacity to improve performance. Recent works have successfully scaled models to billions of parameters via various forms of model parallelism [2, 21, 26, 27, 33]. Model parallelism [30] splits weights and tensors across multiple cores while pipeline parallelism [18, 24] splits different layers across devices with micro-batches pipelined to the different layers. To enable continued scaling of neural networks, improving model training and serving efficiency has become a critical research area.\n\nConditional Computation: Computation decisions can be made dynamically based on the input [23, 25]. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation, by activating certain parameters and computation on demand, on a per-example or per-token basis [3]. Conditional convolution layers [1] with task-specific gating has been used to combat catastrophic forgetting when a sequence of learning problems are optimized. The gating decisions may be binary or sparse and continuous, stochastic or deterministic.\n\nMixture of Experts: Sparsely-gated MoE [31] is the first model to demonstrate massive improvements in model capacity, training time, or model quality with gating. Switch Transformer [10] simplifies the gating by selecting only the top expert per token using a softmax over the hidden state and demonstrates better scaling than previous work. All the prior work requires an auxiliary loss to explicitly encourage balancing. This loss term has to be carefully weighted to not overwhelm the primary loss. However, auxiliary loss does not guarantee balancing and a hard capacity factor has to be imposed. As a result, many tokens can still be unprocessed by the MoE layer. Hard MoE [12] with a single decoding layer can be efficiently trained to good effect on large scale hashtag prediction tasks.\n\nBase Layers [22] formulate a linear assignment that maximizes token-expert affinities while ensuring each expert receives an equal number of tokens. Hash layers [8, 28] devise hashing techniques on input tokens. However, the evaluations are limited to pre-training perplexity. THOR [? ] randomly activates experts during training and inference and is trained with a consistency regularization loss.\n\nTHOR has demonstrated strong performance on translation tasks. Different from these prior works, our method is a learnt method that enables heterogeneous MoE and effectively improves downstream fine-tuning performance.\n\n3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n5 Conclusion\n\nWe propose a new routing method for sparsely activated mixture-of-experts (MoE) models. This method addresses load imbalance and under-utilization of experts in conventional MoE methods, and enables selecting different numbers of experts for each token. Our model demonstrates more than 2x training efficiency improvements when compared to the state-of-the-art GShard and Switch Transformer models, and also achieves strong gains when finetuning on 11 datasets in the GLUE and SuperGLUE benchmark.\n\n6 Limitations\n\nThe expert choice method might not immediately apply to auto-regressive text generation as our current implementation takes in the past and future tokens to perform the top-k selection. One possible solution is to collect a large batch of input sequences, dispatch tokens of the same sequence into separate groups, and perform expert choice routing for each group. Another scenario where the expert choice method does not immediately apply is when the batch size becomes very small during serving or inference. A global top-k can be selected instead and we can cap the number of times each expert or token gets selected. We leave these possible improvements for future work.\n\nAnother long-standing issue with MoE has been the large memory footprint. Even though computational cost can be reduced using sparsely gated networks, the total number of parameters increases linearly or sub-linearly with the number of experts. Increasing the number of experts requires reservation of a large number of hardware devices. Therefore, dynamic (used) power is saved while static (reserved) power is not. Power saving techniques such as the ability to put hardware devices into low power states while not in use [17] can help with reducing the reserved power requirements.\n\nReferences\n\n[1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pages 3930–3939. Computer Vision Foundation / IEEE, 2020.\n\n[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. In Advances in Neural Information Processing Systems.\n\n[3] Kyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning, 2014.\n\n[4] Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems, 2021.\n\n[5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.\n\nTransformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.\n\n[6] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 933–941. JMLR.org, 2017.\n\n[7] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixtureof-experts, 2021.\n\n[8] Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan.\n\nTricks for training sparse translation models, 2021.\n\n[9] Richard L Dykstra. An iterative procedure for obtaining i-projections onto the intersection of convex sets. The annals of Probability, pages 975–984, 1985.\n\n[10] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.\n\n[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. NAS-FPN: learning scalable feature pyramid architecture for object detection. In CVPR, pages 7036–7045. Computer Vision Foundation / IEEE, 2019.\n\n[12] Sam Gross, Marc'Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision, 2017.\n\n[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.\n\n[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 630–645, Cham, 2016. Springer International Publishing.\n\n[15] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2016.\n\n[16] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.\n\n[17] Ping Huang, Zuocheng Xing, Tianran Wang, Qiang Wei, Hongyan Wang, and Guitao Fu. A brief survey on power gating design. In 2010 10th IEEE International Conference on Solid-State and Integrated Circuit Technology, pages 788–790, 2010.\n\n[18] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 103–112, 2019.\n\n[19] Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David A. Patterson. A domain-specific supercomputer for training deep neural networks. Commun. ACM, 63(7):67–78, 2020.\n\n[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.\n\n[21] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.\n\n[22] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6265–6274. PMLR, 18–24 Jul 2021.\n\n[24] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. New York, NY, USA, 2019. Association for Computing Machinery.\n\n[25] Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, Cédric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. In ICLR. OpenReview.net, 2021.\n\n[26] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.\n\n[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.\n\n[28] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models, 2021.\n\n[30] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 10435–10444, Red Hook, NY, USA, 2018. Curran Associates Inc.\n\n[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E.\n\nHinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In ICLR (Poster). OpenReview.net, 2017.\n\n[32] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 10–15 Jul 2018. [23] Min Lin, Jie Fu, and Yoshua Bengio. Conditional computation for continual learning, 2019.\n\n[29] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.\n\n[33] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.\n\n[34] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc.\n\n[35] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.\n\nGLUE: A multi-task benchmark and analysis platform for natural language understanding.\n\nIn Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, November 2018. Association for Computational Linguistics.\n\n[36] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake A. Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen. GSPMD: general and scalable parallelization for ML computation graphs. CoRR, abs/2105.04663, 2021.\n\n7 Checklist\n\n(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Yes (b) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes (c) Did you discuss any potential negative societal impacts of your work? N/A. Not any. (d) Did you describe the limitations of your work? Yes (a) Did you include the code, data, and instructions needed to reproduce the main experimental results? Yes. We include details in the experiment setup to help reproduce the main results. (b) Did you specify all the training details? Yes (c) Did you report error bars? Yes (d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? Yes (a) If your work uses existing assets, did you cite the creators? Yes (b) Did you mention the license of the assets? No. The used dataset is not released yet. (c) Did you include any new assets either in the supplemental material or as a URL? No. The dataset is not released yet.\n\n(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? No. Not using persons' data. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes. The dataset does not contain any personally identifiable information or offensive content.\n\nA Comparison On Fine-Tuning With A Dense Model\n\nOur 8B MoE model achieves stronger pre-training perplexity than its dense counterpart. However, a better perplexity does not always directly translate to downstream performance as demonstrated in Section 4.4. To this end, we compare fine-tuning performance of the 8B dense model and MoE model in Table 1. As shown in the table, our MoE model using expert choice routing consistently outperforms the dense model across the 11 tasks in GLUE and SuperGLUE.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 1: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nB Capacity Factor\n\nWe evaluate the downstream task fine-tuning performance by varying the capacity factors. Note that a capacity factor of n indicates on average how many experts each token can be received. EC-CF2 is our baseline expert choice, which matches GShard top-2 gating computational footprint. EC-CF1, however, matches Switch Transformer top-1 gating computational footprint. EC-CF0.5 further verifies that an aggressively lowered capacity factor can provide strong enough performance, that almost matches the top-2 gating baseline.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Top-2 78.1 87.0 88.3 85.0 82.6 90.1 90.7 81.6 94.7 68.2 67.2 83.0±0.3 EC-CAP2 78.2 88.0 88.5 85.7 83.0 90.8 91.1 80.0 95.4 70.4 64.1 83.2±0.4 EC-CAP3 78.5 91.7 89.3 86.3 83.5 90.9 91.1 81.8 94.9 70.0 65.6 84.0±0.4 EC-CF2 79.1 89.6 89.3 86.8 84.3 91.3 91.2 81.1 95.2 68.1 68.0 84.0±0.2 EC-CF1 77.4 90.6 88.0 85.5 83.6 90.3 91.2 79.8 95.3 66.5 64.9 83.0±0.2 EC-CF0.5 77.4 89.6 86.3 85.2 82.7 91.7 91.0 79.6 94.9 67.3 63.5 83.0 ±0.05 Hash Layers 76.1 85.2 86.7 83.4 82.5 90.0 90.3 75.7 94.0 67.4 63.3 81.3±1.0\n\nTable 2: Comparison between different routing methods in fine-tuning of 100M/64E models. We perform 3 independent fine-tuning runs for each method and report the average results. This gives more accurate difference between the variants of expert choice method, since they achieve close fine-tuning results. We do not report averaged results in other experiments.\n\nC Capped Expert Choice\n\nAs described in Section 4.5, the maximum number of experts each token is assigned can be capped by an entropy-regularized linear programming. Figure 1 compares the validation perplexity when training the 100M/64E models using the base expert choice method (EC-BASE), expert choice capped by two experts per token (EC-CAP2), expert choice capped by three experts per token (EC-CAP3), and GShard top-2 gating.\n\nAs shown in the figure, restricting the number of experts to 2 degrades the perplexity compared to the base expert choice method. This suggests that a more flexible allocation of experts (e.g. more than 2 experts for a token) can enhance model expressiveness. On the other hand, our EC-CAP2 and EC-CAP3 methods still outperform the top-2 gating method by a clear margin. We believe this confirms the effectiveness of a load balanced training, provided by our method. Finally, EC-CAP3 obtains comparable perplexity to EC-BASE. As indicated by Figure 3, only a little fraction of tokens use more than 3 experts therefore we see little or no difference between EC-BASE and EC-CAP3 variants. We present the fine-tuning results of these methods in Table 2.\n\nD Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [? ]. We use mod x to map a token ID to an expert ID. This in some way ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 2. Hashing based routing performs much worse than expert choice in terms of average scores and variance.\n\nE Fine-Tuning Details\n\nWe did a hyperparameter search for both baseline models and expert choice method. For fine-tuning of the 8B dense model, we use a constant learning rate of 0.0001 and a dropout rate of 0.1. We freeze the attention layer and feed-forward layer while leaving the embedding and layer normalization trainable. This setting has been found optimal for the 8B dense model. For MoE 8B/64E models including GShard top-2 gating and expert choice, we found continuing the learning rate from the pre-trained model while using a square root learning rate decay works better. In addition, we do not apply parameter freezing for fine-tuning MoE models. For models with 100M expert size, we use a constant learning rate of 0.0001 and no dropout is used.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/2202.09368v2.md"
+          },
+          "headlines": {
+            "1 Introduction": [],
+            "2 Related Work": [],
+            "3 Method": [
+              "3.1 Pitfalls Of Token-Choice Routing",
+              "3.2 Heterogeneous Moe Via Expert Choice",
+              "3.3 Expert Choice With Additional Constraint",
+              "3.4 Model Architecture"
+            ],
+            "4 Experiments": [
+              "4.1 Setup",
+              "4.2 Training Efficiency",
+              "4.3 Scaling The Number Of Experts",
+              "4.4 Fine-Tuning On Glue And Superglue",
+              "4.5 Heterogeneity Matters",
+              "4.6 Comparison With Hash Layer",
+              "4.7 Ablation"
+            ],
+            "5 Conclusion": [],
+            "6 Limitations": [],
+            "A Comparison On Fine-Tuning With A Dense Model": [],
+            "B Capacity Factor": [],
+            "C Capped Expert Choice": [],
+            "D Comparison With Hash Layer": [],
+            "E Fine-Tuning Details": []
+          },
+          "summary": "The paper introduces a new routing method for sparsely-activated Mixture-of-Experts (MoE) models called 'expert choice,' which addresses load imbalance and under-utilization issues in conventional MoE methods. Unlike traditional token-choice routing, expert choice allows experts to select the top-k tokens, ensuring perfect load balancing and enabling variable expert allocation per token. This method improves training efficiency, achieving over 2x faster convergence compared to existing models like Switch Transformer and GShard. It also demonstrates superior performance on downstream tasks in the GLUE and SuperGLUE benchmarks. The study highlights the benefits of flexible expert allocation and shows that expert choice outperforms hashing-based routing methods. The paper also discusses potential limitations and future improvements for the expert choice method.",
+          "summary_embedding": [
+            -0.009726986289024353,
+            -0.0337262786924839,
+            -0.023302342742681503,
+            -0.019878201186656952,
+            -0.025953780859708786,
+            0.04560471698641777,
+            -0.006874797400087118,
+            0.039665497839450836,
+            -0.006973279174417257,
+            0.034544438123703,
+            -0.025787118822336197,
+            -0.00363057479262352,
+            -0.01052999310195446,
+            0.0054657477885484695,
+            -0.024499278515577316,
+            0.036786794662475586,
+            0.010007280856370926,
+            0.003272630739957094,
+            0.006102093029767275,
+            -0.04048365354537964,
+            -0.018666116520762444,
+            -0.013522328808903694,
+            -0.03230207785964012,
+            0.008969432674348354,
+            0.02484775334596634,
+            0.0016713523073121905,
+            0.0026230288203805685,
+            0.003624893259257078,
+            -0.05311964824795723,
+            0.04339266195893288,
+            0.018681267276406288,
+            0.023711422458291054,
+            -0.016332851722836494,
+            -0.012832955457270145,
+            -0.039120059460401535,
+            -0.0197418425232172,
+            0.030786970630288124,
+            0.039968520402908325,
+            -0.019287310540676117,
+            0.03357476741075516,
+            0.018060073256492615,
+            0.02454473078250885,
+            -0.021272100508213043,
+            -0.029817303642630577,
+            -0.0061892117373645306,
+            -0.007840678095817566,
+            -0.008355814032256603,
+            0.01242387667298317,
+            0.016984347254037857,
+            -0.013007191941142082,
+            0.0212569497525692,
+            -0.01371171697974205,
+            0.03384748846292496,
+            0.008916404098272324,
+            -0.042332086712121964,
+            0.03984731063246727,
+            -0.008007340133190155,
+            0.012105703353881836,
+            0.03281721472740173,
+            0.010333028621971607,
+            -0.005287723150104284,
+            -0.021544819697737694,
+            -0.03378688171505928,
+            0.0010529992869123816,
+            0.00014014738553669304,
+            0.005950582213699818,
+            -0.003069985192269087,
+            -0.009742137044668198,
+            0.03227177634835243,
+            -0.003939277958124876,
+            -0.0740584209561348,
+            0.050089433789253235,
+            -0.026590125635266304,
+            -0.007628562860190868,
+            -0.028938541188836098,
+            -0.01736312545835972,
+            0.037362534552812576,
+            0.005662711802870035,
+            0.0098709212616086,
+            0.0106284748762846,
+            -0.0007892759749665856,
+            0.028923390433192253,
+            0.00910579226911068,
+            -0.024923507124185562,
+            0.04478655755519867,
+            -0.04790768027305603,
+            0.01801462098956108,
+            -0.04336235672235489,
+            0.022756904363632202,
+            1.6393929399782792e-05,
+            -0.020726662129163742,
+            -0.0013304532039910555,
+            -0.016666175797581673,
+            0.0218023881316185,
+            -0.01730252057313919,
+            0.026802239939570427,
+            0.013825350441038609,
+            -0.015802564099431038,
+            -0.029817303642630577,
+            0.025378039106726646,
+            0.003367325058206916,
+            0.010499690659344196,
+            -0.02075696364045143,
+            0.014332910999655724,
+            0.04393810033798218,
+            -0.02593863010406494,
+            -0.02271145209670067,
+            0.027681002393364906,
+            0.014408666640520096,
+            0.0633314698934555,
+            0.036544378846883774,
+            0.006624804809689522,
+            0.022362977266311646,
+            0.020575150847434998,
+            -0.028968842700123787,
+            -0.01963578537106514,
+            -0.046362269669771194,
+            -0.04757435619831085,
+            0.019878201186656952,
+            -0.04566532000899315,
+            0.0050869714468717575,
+            0.038544319570064545,
+            -0.013704141601920128,
+            -0.014446543529629707,
+            -0.04021093621850014,
+            0.017847958952188492,
+            0.0003908502112608403,
+            0.011976920068264008,
+            -0.0022082682698965073,
+            0.00033285003155469894,
+            -0.011825408786535263,
+            0.008878526277840137,
+            -0.03778676688671112,
+            -0.025196226313710213,
+            0.025181075558066368,
+            0.022862963378429413,
+            0.024120500311255455,
+            0.028802180662751198,
+            -0.03077181987464428,
+            -0.009984553791582584,
+            0.021378157660365105,
+            -0.04242299124598503,
+            -0.006556625012308359,
+            0.011332998983561993,
+            0.0026703758630901575,
+            0.003872992005199194,
+            0.032786913216114044,
+            0.052998438477516174,
+            -0.03172633796930313,
+            0.02065090648829937,
+            -0.010431510396301746,
+            0.04957429692149162,
+            0.00024691506405360997,
+            0.028968842700123787,
+            -0.019514575600624084,
+            -0.0134389977902174,
+            -0.029529431834816933,
+            -0.009795165620744228,
+            -0.01356020662933588,
+            -0.008484598249197006,
+            -0.002998017705976963,
+            -0.009575475007295609,
+            0.0040453351102769375,
+            0.0331808403134346,
+            0.00273855566047132,
+            0.030680913478136063,
+            0.014325335621833801,
+            0.0051475754007697105,
+            -0.08745196461677551,
+            0.026059838011860847,
+            -0.0033616432920098305,
+            0.005859675817191601,
+            0.0062460280023515224,
+            0.03013547509908676,
+            -0.0312718041241169,
+            0.020938776433467865,
+            0.026620427146553993,
+            0.0011145505122840405,
+            0.016969196498394012,
+            0.036847397685050964,
+            -0.04181694984436035,
+            0.009643655270338058,
+            0.012590537779033184,
+            -0.0033881578128784895,
+            0.04584713280200958,
+            0.011628445237874985,
+            0.019514575600624084,
+            0.018529755994677544,
+            0.003569970605894923,
+            0.00297907879576087,
+            0.006533898413181305,
+            0.03724132478237152,
+            0.003462019143626094,
+            -0.009833043441176414,
+            0.014976831153035164,
+            -0.010757259093225002,
+            -0.011219366453588009,
+            -0.037998881191015244,
+            0.014454119838774204,
+            0.008742166683077812,
+            -0.017454031854867935,
+            -0.02959003672003746,
+            0.0026949963066726923,
+            -0.031908150762319565,
+            0.026559824123978615,
+            -0.002445003716275096,
+            0.005738467443734407,
+            -0.010454237461090088,
+            -0.006757376715540886,
+            -0.035423196852207184,
+            0.028453707695007324,
+            -0.02095392718911171,
+            0.0053142374381423,
+            0.0007258309051394463,
+            -0.029438525438308716,
+            0.0060073984786868095,
+            0.017863109707832336,
+            -0.0025965142995119095,
+            0.0025302283465862274,
+            -0.02349930815398693,
+            0.024468975141644478,
+            0.025635607540607452,
+            0.004518806003034115,
+            -0.014037464745342731,
+            -0.013113250024616718,
+            -0.003605954349040985,
+            -0.009742137044668198,
+            -0.016196493059396744,
+            0.024468975141644478,
+            -0.014810169115662575,
+            0.00308892410248518,
+            0.021847840398550034,
+            0.014264730736613274,
+            -0.007045246660709381,
+            -0.004007457755506039,
+            -0.04066546633839607,
+            -0.01218903437256813,
+            -0.04760465770959854,
+            -0.011628445237874985,
+            -0.0034601252991706133,
+            -0.028696123510599136,
+            0.0061513339169323444,
+            -0.04481685906648636,
+            -0.01235569640994072,
+            0.02309022843837738,
+            0.018135828897356987,
+            0.0019488062243908644,
+            -0.009173971600830555,
+            -0.022605394944548607,
+            8.486965816700831e-05,
+            -0.01763584464788437,
+            -0.017044952139258385,
+            0.015393485315144062,
+            0.031059689819812775,
+            -0.03101423755288124,
+            0.04024123772978783,
+            0.003893824527040124,
+            -0.08066429197788239,
+            0.010393633507192135,
+            -0.026362858712673187,
+            0.07011914253234863,
+            -0.0025453795678913593,
+            0.014075342565774918,
+            0.06781618297100067,
+            -0.012431452050805092,
+            -0.02007516659796238,
+            0.03142331540584564,
+            -0.008249756880104542,
+            -1.1311515663692262e-05,
+            -0.02884763479232788,
+            -0.02739313244819641,
+            -0.053271155804395676,
+            -0.009704259224236012,
+            -0.010787560604512691,
+            0.03360506892204285,
+            -0.024317465722560883,
+            -0.0007656024536117911,
+            0.012620840221643448,
+            0.02983245439827442,
+            0.016635872423648834,
+            -0.0015681355725973845,
+            0.008272483013570309,
+            0.02881733328104019,
+            0.0274234339594841,
+            -0.0036154238041490316,
+            0.00814369972795248,
+            -0.034211114048957825,
+            -0.0068066176027059555,
+            -0.0493621826171875,
+            0.011302697472274303,
+            -0.02596893161535263,
+            -0.02596893161535263,
+            0.007643714081496,
+            -0.011476934887468815,
+            0.03748374432325363,
+            -0.024696242064237595,
+            -0.007503566797822714,
+            0.005037730094045401,
+            0.012855681590735912,
+            -0.017757052555680275,
+            -0.03612014651298523,
+            -0.0028502948116511106,
+            0.0026381798088550568,
+            0.007560383062809706,
+            -0.053240854293107986,
+            0.004499867558479309,
+            -0.018469152972102165,
+            0.0279688723385334,
+            -0.02139330841600895,
+            -0.013688990846276283,
+            0.042029064148664474,
+            0.011363301426172256,
+            0.027499189600348473,
+            -0.026468917727470398,
+            0.0017849853029474616,
+            0.012825379148125648,
+            -0.0020056227222085,
+            0.032514192163944244,
+            -0.030574856325984,
+            -0.03257479891180992,
+            0.001548249856568873,
+            -0.007840678095817566,
+            -0.0625436082482338,
+            -0.011386028490960598,
+            -0.00981789268553257,
+            0.017317671328783035,
+            0.0446956530213356,
+            0.04557441547513008,
+            -0.01068150345236063,
+            0.002174178371205926,
+            -0.02213571034371853,
+            -0.04542290419340134,
+            -0.02643861435353756,
+            0.0016486257081851363,
+            -0.019999410957098007,
+            -0.02789311669766903,
+            -0.0018313854234293103,
+            0.011211791075766087,
+            -0.013651113025844097,
+            0.01134057529270649,
+            -0.018560059368610382,
+            -0.01862066239118576,
+            -0.012332970276474953,
+            0.018575210124254227,
+            0.009901223704218864,
+            -0.0018796793883666396,
+            -0.015037435106933117,
+            -0.00980274099856615,
+            0.018969137221574783,
+            -0.011279970407485962,
+            0.0009388927719555795,
+            0.01188601367175579,
+            0.024347767233848572,
+            -0.007083124481141567,
+            0.010961798019707203,
+            -0.034756552428007126,
+            -0.01966608688235283,
+            0.007984613068401814,
+            0.002719616750255227,
+            0.020711511373519897,
+            0.008742166683077812,
+            -0.04275631532073021,
+            -0.044301725924015045,
+            0.016044981777668,
+            -0.0562104657292366,
+            0.028120383620262146,
+            -0.03327174484729767,
+            -0.012514783069491386,
+            -0.02671133354306221,
+            0.0013418165035545826,
+            -0.01828734017908573,
+            0.022438732907176018,
+            0.03778676688671112,
+            -0.0011486404109746218,
+            0.005503625608980656,
+            0.014658658765256405,
+            0.04384719207882881,
+            -0.05021063983440399,
+            -0.041180603206157684,
+            0.03754434734582901,
+            -0.01601468026638031,
+            0.0185146052390337,
+            0.00021744151308666915,
+            0.010348179377615452,
+            -0.014431392773985863,
+            0.022772056981921196,
+            0.013757170177996159,
+            -0.008764892816543579,
+            -0.030574856325984,
+            0.04945308715105057,
+            -0.023650817573070526,
+            -0.049513693898916245,
+            0.0018323323456570506,
+            0.018878230825066566,
+            0.018635815009474754,
+            -0.014870774000883102,
+            -0.02907489985227585,
+            -0.024317465722560883,
+            -0.006060427520424128,
+            -0.00947699323296547,
+            -0.03402930125594139,
+            0.011363301426172256,
+            0.0048445542342960835,
+            0.016666175797581673,
+            -0.00645056739449501,
+            -0.0073141781613230705,
+            -0.009370936080813408,
+            0.022559940814971924,
+            0.012635990977287292,
+            -0.03175663948059082,
+            -0.017484333366155624,
+            -0.02528713271021843,
+            0.004359719809144735,
+            0.010136065073311329,
+            0.022166013717651367,
+            0.024287162348628044,
+            -0.0323626808822155,
+            0.04218057543039322,
+            0.01550711877644062,
+            -0.008727015927433968,
+            -0.0019317612750455737,
+            0.02478714846074581,
+            0.04324115067720413,
+            0.0218023881316185,
+            0.026514369994401932,
+            0.002107892418280244,
+            8.705945947440341e-05,
+            -0.015620751306414604,
+            -0.0011391709558665752,
+            0.01862066239118576,
+            0.04911976307630539,
+            -0.012151156552135944,
+            0.01716616004705429,
+            -0.03578682243824005,
+            0.00396200455725193,
+            -0.02580226957798004,
+            -0.030786970630288124,
+            0.03745344281196594,
+            -0.015060162171721458,
+            0.0191964041441679,
+            0.025044716894626617,
+            -0.022469034418463707,
+            -0.011075431481003761,
+            -0.019423669204115868,
+            -0.026181045919656754,
+            -0.01702980138361454,
+            -0.03281721472740173,
+            -0.018135828897356987,
+            -0.002653330797329545,
+            -0.035635314881801605,
+            -0.04987731948494911,
+            -0.004329417832195759,
+            -0.046392571181058884,
+            -0.010507266037166119,
+            0.017378276214003563,
+            -0.011166337877511978,
+            -0.025711363181471825,
+            -0.005336964037269354,
+            0.003973367623984814,
+            -0.022196315228939056,
+            -0.0066588944755494595,
+            0.008439145050942898,
+            0.035938333719968796,
+            -0.02298417128622532,
+            -0.013060221448540688,
+            -0.0024317465722560883,
+            -0.02887793630361557,
+            -0.007878555916249752,
+            0.008999735116958618,
+            0.007723257411271334,
+            -0.0012357590021565557,
+            -0.05196816474199295,
+            -0.006916462909430265,
+            -0.012992041185498238,
+            0.029196109622716904,
+            0.026468917727470398,
+            0.0012168202083557844,
+            -0.02783251367509365,
+            0.008749742060899734,
+            -0.0072535742074251175,
+            0.02339324913918972,
+            -0.007992188446223736,
+            -0.022908415645360947,
+            -0.02034788578748703,
+            0.030786970630288124,
+            -0.0159086212515831,
+            0.009113367646932602,
+            0.009976978413760662,
+            0.0121360057964921,
+            -0.03899884968996048,
+            -0.001445980160497129,
+            0.0005648507503792644,
+            -0.010128489695489407,
+            0.00423093605786562,
+            -0.006738437805324793,
+            -0.007079336792230606,
+            -0.022090258076786995,
+            -0.0218023881316185,
+            0.03818069398403168,
+            -0.009113367646932602,
+            -0.0035661826841533184,
+            -0.01088604237884283,
+            0.03436262533068657,
+            -0.00595815759152174,
+            -0.01993880607187748,
+            -0.019347915425896645,
+            0.014431392773985863,
+            0.05105910077691078,
+            -0.028529461473226547,
+            -0.03984731063246727,
+            0.0011136035900563002,
+            0.030150625854730606,
+            0.004140029661357403,
+            -0.016908591613173485,
+            0.0035851215943694115,
+            0.056392278522253036,
+            -0.011742077767848969,
+            0.028362801298499107,
+            0.027771908789873123,
+            0.02295386977493763,
+            -0.02021152526140213,
+            -0.01678738370537758,
+            -0.020999381318688393,
+            0.011098157614469528,
+            -0.03027183562517166,
+            0.027059808373451233,
+            0.018969137221574783,
+            0.01538590993732214,
+            0.05054396390914917,
+            -0.03178694099187851,
+            -0.02515077404677868,
+            0.008499749936163425,
+            0.0021722845267504454,
+            0.011007251217961311,
+            -0.007727045100182295,
+            -0.01801462098956108,
+            -0.001950700068846345,
+            -0.014022313989698887,
+            -0.019817598164081573,
+            -0.020938776433467865,
+            0.00954517349600792,
+            0.015696506947278976,
+            0.002649543108418584,
+            0.010863316245377064,
+            0.0016590419691056013,
+            0.01686313934624195,
+            0.01321930717676878,
+            -0.0444229319691658,
+            -7.936555630294606e-05,
+            0.03378688171505928,
+            -0.0293476190418005,
+            -0.012264790013432503,
+            -0.016060132533311844,
+            -0.004439263138920069,
+            -0.01706010289490223,
+            -0.02762039750814438,
+            -0.006874797400087118,
+            -0.01889338344335556,
+            -0.00814369972795248,
+            -0.028196139261126518,
+            -0.009992130100727081,
+            0.01949942484498024,
+            0.022332675755023956,
+            0.001196934375911951,
+            -0.006783891003578901,
+            -0.005901341326534748,
+            0.02092362567782402,
+            0.032514192163944244,
+            0.01952972821891308,
+            0.0058331615291535854,
+            -0.015158643946051598,
+            -0.004295328166335821,
+            -0.012370847165584564,
+            0.016060132533311844,
+            0.012537509202957153,
+            6.344509893096983e-05,
+            0.03493836522102356,
+            0.004170331638306379,
+            -0.022726602852344513,
+            -0.001230077352374792,
+            0.024923507124185562,
+            -0.021590271964669228,
+            -0.0108330138027668,
+            0.040029123425483704,
+            0.0085679292678833,
+            -0.0024506852496415377,
+            -0.019287310540676117,
+            -0.019075196236371994,
+            0.02630225569009781,
+            -0.02133270539343357,
+            0.002083271974697709,
+            0.0482713058590889,
+            0.00035344602656550705,
+            -0.029802151024341583,
+            0.017954016104340553,
+            -0.013242034241557121,
+            -0.018560059368610382,
+            0.01848430372774601,
+            0.0012859469279646873,
+            -0.039968520402908325,
+            0.019014591351151466,
+            0.018408548086881638,
+            0.022635696455836296,
+            -0.02918095886707306,
+            0.0002127068000845611,
+            0.03233237937092781,
+            0.004037759732455015,
+            0.001595596899278462,
+            -0.008530051447451115,
+            -0.0006065162015147507,
+            0.010431510396301746,
+            0.001746160676702857,
+            0.02854461409151554,
+            -0.0030434709042310715,
+            0.027771908789873123,
+            -0.006727074272930622,
+            -0.008863375522196293,
+            0.019181253388524055,
+            -0.02109028771519661,
+            0.013166278600692749,
+            0.0685434341430664,
+            0.0021950111258774996,
+            0.025847723707556725,
+            0.007181606721132994,
+            0.024347767233848572,
+            0.02018122375011444,
+            -0.027271922677755356,
+            -0.02021152526140213,
+            -0.005397567991167307,
+            -0.0014033677289262414,
+            -0.03787767142057419,
+            0.011128460057079792,
+            -0.021302402019500732,
+            -0.013249609619379044,
+            -0.0022347825579345226,
+            -0.002316219499334693,
+            -0.006405114196240902,
+            0.034211114048957825,
+            -0.0014630251098424196,
+            -0.018560059368610382,
+            -0.035423196852207184,
+            0.037726160138845444,
+            0.04048365354537964,
+            0.0389382466673851,
+            -0.005984671879559755,
+            0.03548380360007286,
+            0.0030756669584661722,
+            0.0025302283465862274,
+            0.041453324258327484,
+            0.004791525658220053,
+            -0.011567841283977032,
+            0.02295386977493763,
+            0.012416300363838673,
+            0.03351416438817978,
+            -0.02045394293963909,
+            -0.02630225569009781,
+            -0.003299145260825753,
+            -0.026423463597893715,
+            0.012492056004703045,
+            -0.019726691767573357,
+            0.0018768386216834188,
+            0.007117214612662792,
+            0.015196521766483784,
+            -0.016044981777668,
+            0.014719262719154358,
+            -0.006102093029767275,
+            -0.021014532074332237,
+            0.011719351634383202,
+            -0.007780073676258326,
+            0.011961768381297588,
+            -0.02262054570019245,
+            0.001138224033638835,
+            -0.04063516482710838,
+            0.008280059322714806,
+            0.014022313989698887,
+            -0.01598437689244747,
+            0.03205966204404831,
+            0.020969079807400703,
+            -0.01801462098956108,
+            -0.0009682480012997985,
+            0.0034714885987341404,
+            -0.010393633507192135,
+            -0.011067855171859264,
+            -0.019923655316233635,
+            0.02786281518638134,
+            -0.015802564099431038,
+            -0.025908326730132103,
+            0.02454473078250885,
+            -0.01743887923657894,
+            0.015151068568229675,
+            -0.004299115855246782,
+            0.00888610165566206,
+            -0.019120648503303528,
+            0.02284781076014042,
+            -0.023741723969578743,
+            -0.0009128518868237734,
+            -0.020287280902266502,
+            0.0028389315120875835,
+            0.014135947450995445,
+            -0.018272189423441887,
+            -0.02139330841600895,
+            0.019150950014591217,
+            -0.005420294590294361,
+            -0.003931702114641666,
+            0.010014856234192848,
+            0.0030983935575932264,
+            -0.0029715034179389477,
+            -0.004549108445644379,
+            -0.03281721472740173,
+            -0.02915065549314022,
+            0.0033086147159337997,
+            0.011386028490960598,
+            0.015257125720381737,
+            0.027574945241212845,
+            0.03715042024850845,
+            0.006052851676940918,
+            0.0036949669010937214,
+            0.0009327376610599458,
+            -0.020726662129163742,
+            0.042635105550289154,
+            -0.020469093695282936,
+            0.040847279131412506,
+            0.032544493675231934,
+            -0.0197418425232172,
+            -0.023908386006951332,
+            0.0029052174650132656,
+            0.03115059621632099,
+            0.03151422366499901,
+            0.025211378931999207,
+            0.012310243211686611,
+            0.03690800443291664,
+            -0.004359719809144735,
+            -0.032211173325777054,
+            0.015802564099431038,
+            0.022256920114159584,
+            -0.008105821907520294,
+            0.009673957712948322,
+            0.027438584715127945,
+            0.013279912061989307,
+            -0.007757347077131271,
+            0.030150625854730606,
+            0.0012234487803652883,
+            0.02396899089217186,
+            0.02498411200940609,
+            -0.02190844528377056,
+            0.01743887923657894,
+            -0.0001154677156591788,
+            0.006727074272930622,
+            -0.009840618818998337,
+            0.004348356742411852,
+            0.011810258030891418,
+            0.018272189423441887,
+            0.018908534198999405,
+            0.003246116451919079,
+            -0.07060398161411285,
+            0.019726691767573357,
+            0.004507442936301231,
+            -0.02160542458295822,
+            0.00390140013769269,
+            -0.0024866692256182432,
+            0.012704171240329742,
+            0.02105998620390892,
+            0.012461753562092781,
+            -0.01790856383740902,
+            0.0013626492582261562,
+            0.011961768381297588,
+            -0.00595815759152174,
+            0.04730163514614105,
+            0.0019791084341704845,
+            -0.01437078882008791,
+            -0.016105586662888527,
+            0.013726867735385895,
+            0.02525683119893074,
+            -0.02355991117656231,
+            0.006325571332126856,
+            0.005621046293526888,
+            -0.004147605039179325,
+            -0.010742107406258583,
+            -0.030377892777323723,
+            -0.005280147306621075,
+            -0.03436262533068657,
+            0.023150833323597908,
+            0.015491968020796776,
+            0.014764715917408466,
+            -0.025044716894626617,
+            -0.004280176945030689,
+            0.0046400148421525955,
+            -0.021287251263856888,
+            0.003261267440393567,
+            0.01161329448223114,
+            0.0005104016163386405,
+            -0.020984230563044548,
+            0.00959820207208395,
+            0.013469300232827663,
+            0.06848283112049103,
+            0.0007442962378263474,
+            0.029771849513053894,
+            0.0422411784529686,
+            -0.0012243957025930285,
+            -0.006371024530380964,
+            0.0036286809481680393,
+            -0.019953956827521324,
+            0.003473382443189621,
+            0.025878025218844414,
+            -0.03224147483706474,
+            -0.055058982223272324,
+            -0.007545232307165861,
+            -0.04624105989933014,
+            0.001489539397880435,
+            0.014711687341332436,
+            0.023044776171445847,
+            -0.0037782976869493723,
+            -0.034817155450582504,
+            0.016908591613173485,
+            -0.003071879269555211,
+            0.007014944683760405,
+            -0.02593863010406494,
+            -0.0294839795678854,
+            -0.026014383882284164,
+            -0.02339324913918972,
+            -0.012908710166811943,
+            -0.022726602852344513,
+            0.012590537779033184,
+            -0.0015103721525520086,
+            -0.06587684899568558,
+            -0.009780014865100384,
+            0.016166189685463905,
+            0.001200722181238234,
+            0.011939042247831821,
+            0.011083006858825684,
+            -0.007083124481141567,
+            -0.0018275976181030273,
+            -0.022559940814971924,
+            0.03124150261282921,
+            0.005177877843379974,
+            0.03584742918610573,
+            -0.006503595970571041,
+            -0.009128518402576447,
+            0.01073453202843666,
+            0.04527139291167259,
+            0.0017916138749569654,
+            0.010226971469819546,
+            -0.020560000091791153,
+            0.0015065843472257257,
+            -0.02284781076014042,
+            -0.0031135445460677147,
+            -0.008863375522196293,
+            -0.0021211495622992516,
+            0.015893470495939255,
+            0.0019753205124288797,
+            -0.010204244405031204,
+            0.025438643991947174,
+            0.02251448854804039,
+            -0.009605777449905872,
+            -0.015969226136803627,
+            -0.02437806874513626,
+            -0.014545026235282421,
+            -0.0002627763315103948,
+            -0.010007280856370926,
+            -0.022453883662819862,
+            -0.020878173410892487,
+            -0.002566212322562933,
+            -0.02559015527367592,
+            0.022529639303684235,
+            0.01220418605953455,
+            -4.882668508798815e-05,
+            -0.006723286584019661,
+            -0.038544319570064545,
+            -0.003727162955328822,
+            -0.03118089959025383,
+            -0.005374841392040253,
+            0.03287781774997711,
+            -0.03745344281196594,
+            -0.0002231231628684327,
+            0.014491996727883816,
+            -0.01993880607187748,
+            0.0028938541654497385,
+            0.008961857296526432,
+            0.02769615314900875,
+            -0.027408283203840256,
+            0.0001795638381736353,
+            0.006954340264201164,
+            0.016908591613173485,
+            -0.009878496639430523,
+            0.027196168899536133,
+            -0.04275631532073021,
+            0.011037553660571575,
+            -0.037120118737220764,
+            -0.020423639565706253,
+            -0.01523439958691597,
+            0.008363389410078526,
+            -0.005018791649490595,
+            -0.02637801133096218,
+            -0.010454237461090088,
+            0.006026337388902903,
+            -0.00693918950855732,
+            -0.01902974210679531,
+            -0.010620899498462677,
+            0.027438584715127945,
+            0.00931033119559288,
+            0.007977037690579891,
+            -0.02955973520874977,
+            -0.01631770096719265,
+            0.006280118133872747,
+            0.052089374512434006,
+            -0.01584801822900772,
+            0.014938953332602978,
+            -0.006639956030994654,
+            0.02342355251312256,
+            -0.028484009206295013,
+            -0.01872672140598297,
+            0.0315445251762867,
+            0.013946558348834515,
+            0.007064185570925474,
+            0.028953691944479942,
+            0.027196168899536133,
+            0.027105262503027916,
+            -0.04654408246278763,
+            0.009886072017252445,
+            -0.002613559365272522,
+            -0.008757317438721657,
+            0.03402930125594139,
+            0.01916610263288021,
+            0.022150862962007523,
+            0.0009744031121954322,
+            -0.01730252057313919,
+            0.0139844361692667,
+            0.014885924756526947,
+            0.007681591901928186,
+            -0.014242004603147507,
+            -0.02660527639091015,
+            0.003367325058206916,
+            -0.007992188446223736,
+            0.00047276070108637214,
+            0.020726662129163742,
+            -0.009052763693034649,
+            0.030999086797237396,
+            -0.014605630189180374,
+            -0.02593863010406494,
+            0.008916404098272324,
+            -0.02268115058541298,
+            -0.0041741193272173405,
+            0.0004476667381823063,
+            0.010083036497235298,
+            -0.023847781121730804,
+            -0.04669559374451637,
+            0.019408518448472023,
+            0.028393102809786797,
+            0.005579380784183741,
+            0.01328748743981123,
+            -0.035332292318344116,
+            0.008992158807814121,
+            0.012310243211686611,
+            0.011348150670528412,
+            0.0019847899675369263,
+            0.035362593829631805,
+            -0.00028834378463216126,
+            0.012885984033346176,
+            -0.008620957843959332,
+            0.028362801298499107,
+            0.00667025800794363,
+            -0.0323626808822155,
+            -0.0378473699092865,
+            -0.01513591781258583,
+            0.015257125720381737,
+            0.006692984607070684,
+            -0.011840560473501682,
+            -0.02278720773756504,
+            -0.038817036896944046,
+            0.015787413343787193,
+            -0.012916285544633865,
+            -0.038271598517894745,
+            -0.010158791206777096,
+            -0.009742137044668198,
+            0.016635872423648834,
+            -0.006916462909430265,
+            -0.016802534461021423,
+            -0.016090434044599533,
+            0.005821797996759415,
+            7.161247049225494e-05,
+            0.0244083721190691,
+            -0.001251857029274106,
+            -0.006855858489871025,
+            -0.016393456608057022,
+            -0.01188601367175579,
+            -0.0035074723418802023,
+            0.03602924197912216,
+            -0.014219277538359165,
+            -0.02383263036608696,
+            0.014984406530857086,
+            0.02657497487962246,
+            0.011636020615696907,
+            0.03227177634835243,
+            0.007181606721132994,
+            -0.014022313989698887,
+            0.003016956616193056,
+            -0.033029329031705856,
+            -0.0046400148421525955,
+            0.005196816753596067,
+            -0.006427840795367956,
+            -0.015651054680347443,
+            -0.01815097965300083,
+            -0.015832867473363876,
+            0.030756669119000435,
+            -0.02487805485725403,
+            0.0218023881316185,
+            0.030347591266036034,
+            0.003174148965626955,
+            -0.02915065549314022,
+            0.013598084449768066,
+            -0.0072005451656877995,
+            0.01862066239118576,
+            0.0019014590652659535,
+            0.028014326468110085,
+            -0.007461901288479567,
+            -0.004818039946258068,
+            -0.02501441352069378,
+            -0.03696860745549202,
+            -0.023484155535697937,
+            -0.008817922323942184,
+            0.0027537066489458084,
+            0.020984230563044548,
+            -0.0375746488571167,
+            -0.017757052555680275,
+            0.009855770505964756,
+            0.015757111832499504,
+            -0.03027183562517166,
+            -0.0025908327661454678,
+            -0.010158791206777096,
+            -0.0004055278259329498,
+            0.005140000022947788,
+            -0.02227207086980343,
+            -0.008977008052170277,
+            -0.015726808458566666,
+            -0.0010435298318043351,
+            0.006166485138237476,
+            -0.0007409819518215954,
+            -0.05805889517068863,
+            -0.013969285413622856,
+            0.018635815009474754,
+            0.0026400736533105373,
+            0.008908828720450401,
+            0.015938924625515938,
+            -0.0111739132553339,
+            0.0031533162109553814,
+            -0.026756787672638893,
+            0.023893235251307487,
+            -0.009492144919931889,
+            0.009355784393846989,
+            -0.013742019422352314,
+            0.012514783069491386,
+            0.014014738611876965,
+            0.004795313347131014,
+            0.0010681503918021917,
+            -0.0054051438346505165,
+            0.01601468026638031,
+            0.028377952054142952,
+            -0.023756874725222588,
+            0.011454207822680473,
+            -0.0002514130319468677,
+            -0.0032158142421394587,
+            -0.029196109622716904,
+            0.00509454682469368,
+            0.003931702114641666,
+            -0.005109698045998812,
+            0.016651025041937828,
+            -0.036544378846883774,
+            0.01662072166800499,
+            0.02177208475768566,
+            0.032544493675231934,
+            0.014832896180450916,
+            -0.017287369817495346,
+            -0.011658747680485249,
+            -0.012385998852550983,
+            -0.002624922664836049,
+            0.022181164473295212,
+            0.012120855040848255,
+            -0.01815097965300083,
+            -0.024287162348628044,
+            0.011249667964875698,
+            0.0005283934879116714,
+            -0.01203752402216196,
+            0.010795135982334614,
+            -0.013166278600692749,
+            0.01349202636629343,
+            0.012711746618151665,
+            0.00866641104221344,
+            0.00888610165566206,
+            0.022862963378429413,
+            -0.011878437362611294,
+            -0.017711598426103592,
+            0.00656041270121932,
+            -0.008916404098272324,
+            -0.0034089903347194195,
+            -0.012598113156855106,
+            0.010795135982334614,
+            0.014878349378705025,
+            0.06490717828273773,
+            -0.013143551535904408,
+            0.0009526234352961183,
+            -0.020438792183995247,
+            -0.031938452273607254,
+            -0.0017944546416401863,
+            -0.020908474922180176,
+            0.0111739132553339,
+            0.008370965719223022,
+            0.01598437689244747,
+            0.012022373266518116,
+            -0.0021912232041358948,
+            0.00795431062579155,
+            0.019317612051963806,
+            -0.010423935018479824,
+            7.255941454786807e-05,
+            -0.006503595970571041,
+            0.0013683309080079198,
+            0.0035851215943694115,
+            -0.011719351634383202,
+            -0.0026949963066726923,
+            0.035695917904376984,
+            0.01068150345236063,
+            0.011651171371340752,
+            -0.03445352986454964,
+            0.0020435003098100424,
+            0.020150920376181602,
+            0.007602048572152853,
+            -0.008302785456180573,
+            -0.01563590206205845,
+            0.01706010289490223,
+            -0.015317730605602264,
+            -0.0022499337792396545,
+            0.002888172399252653,
+            -0.010151215828955173,
+            0.04581683129072189,
+            -0.019787294790148735,
+            -0.0011742077767848969,
+            0.031847547739744186,
+            0.0030207443051040173,
+            -0.0040983641520142555,
+            0.005408931523561478,
+            -0.003473382443189621,
+            -0.0069013116881251335,
+            -0.02139330841600895,
+            0.0031381649896502495,
+            0.00667025800794363,
+            -0.019287310540676117,
+            -0.0072005451656877995,
+            -0.007719469256699085,
+            0.006897523999214172,
+            0.023105379194021225,
+            -0.009492144919931889,
+            -0.00016145357221830636,
+            -0.008575504645705223,
+            0.009848194196820259,
+            -0.0011912527261301875,
+            -0.044301725924015045,
+            -0.006147546228021383,
+            0.021181194111704826,
+            0.000973456131760031,
+            0.0005757405888289213,
+            -0.007450537756085396,
+            0.005007428117096424,
+            -0.017423728480935097,
+            -0.008454295806586742,
+            0.02369627170264721,
+            0.01736312545835972,
+            -0.009113367646932602,
+            0.010863316245377064,
+            -0.0035680767614394426,
+            0.00433320552110672,
+            -0.020014561712741852,
+            -0.01095422264188528,
+            0.003462019143626094,
+            0.015272277407348156,
+            -0.0009819786064326763,
+            -0.029029447585344315,
+            -0.0012272365856915712,
+            -0.00024194362049456686,
+            0.02918095886707306,
+            0.01486319862306118,
+            -0.008961857296526432,
+            -0.021211495622992516,
+            -0.013242034241557121,
+            -0.001237652963027358,
+            -0.015726808458566666,
+            -0.009620928205549717,
+            0.011855711229145527,
+            0.0009137988090515137,
+            0.008992158807814121,
+            -0.018135828897356987,
+            -0.027347678318619728,
+            -0.016166189685463905,
+            0.0018465365283191204,
+            -0.012870832346379757,
+            0.003340810537338257,
+            0.015514694154262543,
+            -0.03430201858282089,
+            -0.015060162171721458,
+            -0.03805948421359062,
+            0.008280059322714806,
+            -0.0029658216517418623,
+            -0.02024182677268982,
+            -0.023605365306138992,
+            0.009484568610787392,
+            0.019181253388524055,
+            -0.0015548784285783768,
+            0.019847899675369263,
+            -0.000889651826582849,
+            0.020060013979673386,
+            -0.0073141781613230705,
+            0.0031703610438853502,
+            -0.013007191941142082,
+            0.0030245319940149784,
+            0.008333087898790836,
+            0.02890823967754841,
+            -0.023075077682733536,
+            -0.006503595970571041,
+            -0.004685468040406704,
+            0.004746072459965944,
+            0.012658718042075634,
+            0.014613205567002296,
+            0.020741812884807587,
+            -0.008272483013570309,
+            -0.0027612822595983744,
+            0.012484480626881123,
+            -0.003799130441620946,
+            0.007295239716768265,
+            0.002390081062912941,
+            0.0019677451346069574,
+            0.004162756260484457,
+            0.008477022871375084,
+            0.0053710537031292915,
+            0.027650700882077217,
+            -0.0028275682125240564,
+            0.0031703610438853502,
+            0.024620486423373222,
+            0.02322658896446228,
+            -0.018878230825066566,
+            -0.028074929490685463,
+            0.002024561632424593,
+            0.050695475190877914,
+            0.007245998363941908,
+            -0.006810405291616917,
+            -0.03927157074213028,
+            -0.0031760428100824356,
+            0.028635520488023758,
+            0.023650817573070526,
+            0.01584801822900772,
+            -0.010567869991064072,
+            -0.012098127976059914,
+            0.02772645466029644,
+            0.013295062817633152,
+            0.02616589516401291,
+            -0.025302285328507423,
+            0.010704229585826397,
+            -0.03869583085179329,
+            0.0449683703482151,
+            0.004439263138920069,
+            -0.014476845972239971,
+            0.0011884119594469666,
+            0.010386057198047638,
+            -0.004537744913250208,
+            0.006431628484278917,
+            -5.8059369621332735e-05,
+            -0.0008086882880888879,
+            0.011923890560865402,
+            0.012007221579551697,
+            0.004265025723725557,
+            -0.02399929240345955,
+            0.007359631359577179,
+            0.004272601567208767,
+            0.03005971945822239,
+            0.005431658122688532,
+            -0.0013863227795809507,
+            0.012385998852550983,
+            -0.005810434930026531,
+            0.008855799213051796,
+            -0.010719381272792816,
+            0.020469093695282936,
+            -0.02918095886707306,
+            1.0986003871948924e-05,
+            0.02051454596221447,
+            -0.0015785519499331713,
+            0.0038199631962925196,
+            -0.023741723969578743,
+            0.01186328660696745,
+            -0.052968136966228485,
+            0.021166043356060982,
+            -0.018741872161626816,
+            0.021741783246397972,
+            0.019590331241488457,
+            -0.009734561666846275,
+            -0.012052674777805805,
+            0.03545350208878517,
+            0.0026760573964565992,
+            -0.014264730736613274,
+            -0.010643625631928444,
+            -0.017999470233917236,
+            -0.02830219641327858,
+            0.041968461126089096,
+            -0.0006117243901826441,
+            0.0028275682125240564,
+            0.0024222771171480417,
+            -0.010666352696716785,
+            -0.008923979476094246,
+            0.001954487757757306,
+            0.020499395206570625,
+            -0.03369597718119621,
+            -0.011128460057079792,
+            -0.008689138107001781,
+            -0.02566591091454029,
+            -0.00020300064352340996,
+            -0.011961768381297588,
+            0.026211349293589592,
+            0.01240872498601675,
+            -0.03696860745549202,
+            0.003780191531404853,
+            0.023650817573070526,
+            0.001603172393515706,
+            0.018120678141713142,
+            0.002130619017407298,
+            0.0010795136913657188,
+            0.009742137044668198,
+            -0.010476963594555855,
+            0.026559824123978615,
+            -0.009204274043440819,
+            0.008658835664391518,
+            -0.001950700068846345,
+            -0.017287369817495346,
+            0.017969166859984398,
+            0.004083212930709124,
+            0.024120500311255455,
+            0.011954193003475666,
+            0.011605718173086643,
+            0.01112088467925787,
+            -0.019878201186656952,
+            0.020090317353606224,
+            -0.0018891488434746861,
+            0.009075489826500416,
+            0.024726543575525284,
+            -0.005863463506102562,
+            0.020832719281315804,
+            0.015257125720381737,
+            -0.00020229043730068952,
+            0.011295121163129807,
+            -0.020120618864893913,
+            -0.011779955588281155,
+            0.0085679292678833,
+            -0.009605777449905872,
+            -0.010325453244149685,
+            0.009901223704218864,
+            -0.010105762630701065,
+            -0.015052586793899536,
+            -0.0071247899904847145,
+            0.016635872423648834,
+            0.0028976418543606997,
+            -0.02610529027879238,
+            -1.7326075976598077e-05,
+            0.008901252411305904,
+            0.006166485138237476,
+            0.018741872161626816,
+            0.011317848227918148,
+            -0.006692984607070684,
+            0.010363331064581871,
+            0.011476934887468815,
+            0.0053104497492313385,
+            -0.005079395603388548,
+            -0.013431422412395477,
+            -0.0024241709616035223,
+            0.018090376630425453,
+            0.009810317307710648,
+            -0.01922670565545559,
+            0.004564259201288223,
+            0.0008877579239197075,
+            -0.01420412678271532,
+            -0.019347915425896645,
+            -0.0038483713287860155,
+            -0.004170331638306379,
+            0.0027915844693779945,
+            0.0509984977543354,
+            -0.0006216673064045608,
+            0.007893706671893597,
+            -0.025741664692759514,
+            -0.006299056578427553,
+            -0.00027556007262319326,
+            -0.0036097420379519463,
+            0.01272689737379551,
+            -0.01824188604950905,
+            -0.03724132478237152,
+            0.012711746618151665,
+            -0.0012584856012836099,
+            0.014923802576959133,
+            -0.00297907879576087,
+            0.014226853847503662,
+            -0.017347972840070724,
+            0.01916610263288021,
+            -0.0011117096291854978,
+            -0.016029831022024155,
+            0.0029374135192483664,
+            -0.011143610812723637,
+            -0.021317552775144577,
+            0.005916492082178593,
+            -0.006848283112049103,
+            -0.03896854817867279,
+            0.0029885482508689165,
+            -0.010423935018479824,
+            0.0011798894265666604,
+            -0.003912763670086861,
+            0.020166072994470596,
+            -0.0061513339169323444,
+            0.0008697660523466766,
+            0.02177208475768566,
+            0.0017026013229042292,
+            0.01601468026638031,
+            -0.014499573037028313,
+            0.010999675840139389,
+            0.012598113156855106,
+            0.02210540883243084,
+            -0.032180868089199066,
+            -0.01393140759319067,
+            0.014211702160537243,
+            0.0026116655208170414,
+            -0.005382417235523462,
+            0.020635755732655525,
+            -0.01294658798724413,
+            0.019650936126708984,
+            -0.0005577487172558904,
+            -0.011704200878739357,
+            -0.03524138405919075,
+            -0.016529815271496773,
+            -3.343888238305226e-05,
+            -0.008439145050942898,
+            0.025317436084151268,
+            0.01760554127395153,
+            -0.0108330138027668,
+            0.026468917727470398,
+            0.0037574649322777987,
+            0.014598054811358452,
+            -0.014257155358791351,
+            -0.005999823100864887,
+            -0.0003773563075810671,
+            -0.0119314668700099,
+            0.015264702029526234,
+            -0.03512017801403999,
+            0.0008342557121068239,
+            -0.00839369185268879,
+            0.005242269951850176,
+            -0.0254992488771677,
+            0.018257036805152893,
+            0.025817420333623886,
+            -0.0031419529113918543,
+            -0.005037730094045401,
+            -0.022802358493208885,
+            0.028968842700123787,
+            0.03049910068511963,
+            0.0024828813038766384,
+            -0.0013825349742546678,
+            -0.00352072948589921,
+            0.0033218718599528074,
+            0.0211205892264843,
+            -0.037998881191015244,
+            -0.009098216891288757,
+            -0.028696123510599136,
+            0.042332086712121964,
+            -0.008098246529698372,
+            -0.013431422412395477,
+            0.01862066239118576,
+            -0.0024715180043131113,
+            0.02355991117656231,
+            -0.004689255729317665,
+            -0.004855917301028967,
+            0.03148392215371132,
+            0.011514811776578426,
+            0.02772645466029644,
+            -0.04015033319592476,
+            0.014704111963510513,
+            -0.006867222022265196,
+            0.004867280833423138,
+            -0.013484450988471508,
+            0.005302873905748129,
+            0.02580226957798004,
+            -0.018060073256492615,
+            0.0068066176027059555,
+            0.030665762722492218,
+            0.00748084019869566,
+            0.002448791405186057,
+            0.0165904201567173,
+            0.003890036838129163,
+            0.010855740867555141,
+            0.000841831264551729,
+            0.0052574207074940205,
+            0.022423582151532173,
+            0.014454119838774204,
+            -0.016029831022024155,
+            -0.015120766125619411,
+            -0.00044790346873924136,
+            -0.004450626205652952,
+            0.005261208396404982,
+            0.013242034241557121,
+            0.0061172437854111195,
+            -0.015681356191635132,
+            0.01052999310195446,
+            0.014158673584461212,
+            0.013128400780260563,
+            -0.0039847311563789845,
+            -0.03412020578980446,
+            -0.02443867363035679,
+            -0.021438762545585632,
+            0.003104075090959668,
+            -0.0008281006012111902,
+            -0.0021211495622992516,
+            0.0017954016802832484,
+            0.017347972840070724,
+            -0.0008352026343345642,
+            0.00379155483096838,
+            0.01686313934624195,
+            -0.01415109820663929,
+            -0.022090258076786995,
+            -0.0035964848939329386,
+            -0.027529491111636162,
+            0.0053710537031292915,
+            0.001844642567448318,
+            -0.00965123064815998,
+            0.008530051447451115,
+            -0.0046059247106313705,
+            -0.007223271764814854,
+            -0.012196609750390053,
+            -0.02007516659796238,
+            0.005780132487416267,
+            0.016878290101885796,
+            0.005730891600251198,
+            -0.009272454306483269,
+            -0.003939277958124876,
+            0.011575416661798954,
+            0.017484333366155624,
+            0.017317671328783035,
+            -0.009295180439949036,
+            0.009628504514694214,
+            0.0004022135108243674,
+            0.004670316819101572,
+            -0.01834794320166111,
+            0.0045869858004152775,
+            -0.0027953721582889557,
+            -0.008658835664391518,
+            0.01105270441621542,
+            -0.00031201730598695576,
+            0.01628739945590496,
+            -0.013090522959828377,
+            0.005624833982437849,
+            0.010333028621971607,
+            -0.020590301603078842,
+            -0.0067308624275028706,
+            -0.01692374423146248,
+            0.026484068483114243,
+            -0.00504530593752861,
+            0.00045405857963487506,
+            -0.02978700026869774,
+            0.0033616432920098305,
+            0.000848933297675103,
+            -0.000845619011670351,
+            -0.012764775194227695,
+            0.010749682784080505,
+            0.03242328763008118,
+            0.0005179771687835455,
+            0.0216660276055336,
+            -0.0036154238041490316,
+            0.0007480839849449694,
+            -0.00785582885146141,
+            0.004977126140147448,
+            -0.004916521720588207,
+            -0.009901223704218864,
+            -0.010863316245377064,
+            0.021726632490754128,
+            -0.0033029329497367144,
+            0.0003356908564455807,
+            0.011332998983561993,
+            -0.012067826464772224,
+            0.0014620780711993575,
+            -0.0047687990590929985,
+            0.00971941091120243,
+            0.0061702728271484375,
+            0.009113367646932602,
+            -0.01176480483263731,
+            0.05451354384422302,
+            -0.0106284748762846,
+            0.005537715740501881,
+            -0.00439380994066596,
+            -0.022226616740226746,
+            -0.017317671328783035,
+            0.012711746618151665,
+            -0.017544938251376152,
+            0.03375658020377159,
+            0.024150803685188293,
+            0.0006467612111009657,
+            0.010393633507192135,
+            -0.006958128418773413,
+            0.023923536762595177,
+            -0.002602196065708995,
+            0.0010795136913657188,
+            -0.004855917301028967,
+            0.015802564099431038,
+            0.04511988162994385,
+            0.017393426969647408,
+            -0.001152428099885583,
+            0.00829521007835865,
+            0.020575150847434998,
+            0.00814369972795248,
+            -0.0020075165666639805,
+            0.014219277538359165,
+            -0.01024212222546339,
+            0.00900731049478054,
+            -0.004458202049136162,
+            0.027438584715127945,
+            0.00814369972795248,
+            -0.006310420110821724,
+            -0.0036836036015301943,
+            -0.0008209985098801553,
+            0.014173824340105057,
+            0.006950552575290203,
+            0.01963578537106514,
+            -0.00461350055411458,
+            -0.0012575386790558696,
+            0.013870803639292717,
+            0.014491996727883816,
+            -0.00552635220810771,
+            0.001724380999803543,
+            -0.01435563713312149,
+            -0.010204244405031204,
+            0.013575357384979725,
+            0.006086941808462143,
+            0.006310420110821724,
+            -0.014514723792672157,
+            -0.00980274099856615,
+            -0.004670316819101572,
+            0.0015435151290148497,
+            0.015052586793899536,
+            0.011060279794037342,
+            -0.004814252257347107,
+            0.006052851676940918,
+            0.013825350441038609,
+            -0.014014738611876965,
+            0.006147546228021383,
+            0.007208121009171009,
+            0.008855799213051796,
+            0.004787737503647804,
+            -0.010817863047122955,
+            -0.00428396463394165,
+            -0.0063937511295080185,
+            0.0012670081341639161,
+            0.016832837834954262,
+            0.013181429356336594,
+            0.0049581872299313545,
+            0.027529491111636162,
+            -0.00839369185268879,
+            -0.012302667833864689,
+            0.02339324913918972,
+            0.011257244274020195,
+            0.0031703610438853502,
+            0.014651083387434483,
+            -0.012635990977287292,
+            -0.012446602806448936,
+            0.006655106786638498,
+            0.004848341923207045,
+            0.006049063988029957,
+            0.009151245467364788,
+            0.00450365524739027,
+            0.007333117071539164,
+            -0.007098275702446699,
+            -0.007427811156958342,
+            0.0014810169814154506,
+            0.01642375811934471,
+            0.020393338054418564,
+            -0.024968961253762245,
+            -0.0018806264270097017,
+            0.0009052763343788683,
+            0.010878467001020908,
+            -0.003316190093755722,
+            0.012310243211686611,
+            5.65205882594455e-05,
+            -0.0074240234680473804,
+            0.010158791206777096,
+            -0.02119634486734867,
+            -0.012681444175541401,
+            0.0012717428617179394,
+            0.013037494383752346,
+            -0.019590331241488457,
+            0.018226735293865204,
+            -0.008893677033483982,
+            -0.013348091393709183,
+            0.013893529772758484,
+            0.020014561712741852,
+            0.004090788308531046,
+            -0.007738408166915178,
+            -0.021862991154193878,
+            -0.01127239502966404,
+            0.016060132533311844,
+            0.020272130146622658,
+            0.000947888707742095,
+            -0.0006472347304224968,
+            0.010173942893743515,
+            -0.013113250024616718,
+            -0.023605365306138992,
+            -0.005230906419456005,
+            0.010386057198047638,
+            0.01746918261051178,
+            0.018832778558135033,
+            -0.006685409229248762,
+            0.02443867363035679,
+            -0.0059127043932676315,
+            -0.015499543398618698,
+            2.86597842205083e-05,
+            -0.02802947722375393,
+            -0.017423728480935097,
+            0.00013778003631159663,
+            0.00246773031540215,
+            0.008598231710493565,
+            0.0008555619278922677,
+            0.01943882182240486,
+            0.000904329412151128,
+            0.017590390518307686,
+            -0.000659071491099894,
+            -0.002009410411119461,
+            -0.0013995799235999584,
+            -0.0015340456739068031,
+            0.02254479005932808,
+            -0.012166308239102364,
+            -0.01176480483263731,
+            0.022302372381091118,
+            0.006852070800960064,
+            0.014719262719154358,
+            -0.027635548263788223,
+            0.013757170177996159,
+            -0.015499543398618698,
+            0.017423728480935097,
+            -0.03408990427851677,
+            0.006420265417546034,
+            0.01489350013434887,
+            -0.010348179377615452,
+            -0.020499395206570625,
+            -0.029650641605257988,
+            -0.024150803685188293,
+            -0.0038710979279130697,
+            -0.010855740867555141,
+            -0.016847988590598106,
+            0.006121031939983368,
+            -0.005802859086543322,
+            0.018499454483389854,
+            -0.007079336792230606,
+            0.033877789974212646,
+            0.006969491485506296,
+            -0.004056698642671108,
+            0.003835114184767008,
+            0.022469034418463707,
+            -0.006102093029767275,
+            0.02109028771519661,
+            0.004408961161971092,
+            -0.001467759720981121,
+            0.013681414537131786,
+            -0.006552837323397398,
+            -0.002361672930419445,
+            0.005821797996759415,
+            -0.010484539903700352,
+            0.022014502435922623,
+            0.005924067925661802,
+            -0.007708106189966202,
+            -0.0006775368237867951,
+            -0.011454207822680473,
+            0.014726839028298855,
+            0.006530110724270344,
+            0.0022480397019535303,
+            -0.011295121163129807,
+            0.011249667964875698,
+            -0.004859705455601215,
+            0.0040642740204930305,
+            0.009484568610787392,
+            -0.01774190180003643,
+            -0.028862785547971725,
+            0.011416330002248287,
+            0.019120648503303528,
+            0.006382387597113848,
+            -0.0055755930952727795,
+            -0.0025681061670184135,
+            0.00839369185268879,
+            0.010651201009750366,
+            0.022150862962007523,
+            -0.012984465807676315,
+            -8.889415767043829e-05,
+            0.013204156421124935,
+            -0.007170243188738823,
+            -0.01056029461324215,
+            0.00487485621124506,
+            0.023438703268766403,
+            0.0068823727779090405,
+            -0.04327145218849182,
+            -0.018469152972102165,
+            0.01909034699201584,
+            0.007136153522878885,
+            -0.04154422879219055,
+            0.0026381798088550568,
+            -0.003689285134896636,
+            -0.023787178099155426,
+            0.016105586662888527,
+            -0.0011817833874374628,
+            -0.01828734017908573,
+            0.028756728395819664,
+            0.02830219641327858,
+            0.01834794320166111,
+            2.6987840101355687e-05,
+            -0.013332940638065338,
+            -0.0318172425031662,
+            -0.023590214550495148,
+            0.008280059322714806,
+            -0.02660527639091015,
+            -0.0072725131176412106,
+            0.011242092587053776,
+            -0.017454031854867935,
+            0.02972639724612236,
+            -0.008507325313985348,
+            0.026938600465655327,
+            0.011446632444858551,
+            0.001691238023340702,
+            -0.013446573168039322,
+            -0.018802475184202194,
+            -0.01350717805325985,
+            -0.008196728304028511,
+            0.014090493321418762,
+            0.017923714593052864,
+            0.011007251217961311,
+            0.011332998983561993,
+            0.02251448854804039,
+            -0.012158732861280441,
+            0.007969462312757969,
+            -2.888172457460314e-05,
+            -0.008961857296526432,
+            0.0026343921199440956,
+            -0.018393397331237793,
+            0.00868156272917986,
+            0.0062308767810463905,
+            0.005458172410726547,
+            -0.007208121009171009,
+            0.010742107406258583,
+            0.02712041325867176,
+            -0.018847929313778877,
+            0.011401179246604443,
+            -0.031362712383270264,
+            -0.014787442982196808,
+            -0.01056029461324215,
+            0.026862844824790955,
+            0.007757347077131271,
+            -0.015726808458566666,
+            -0.007514929864555597,
+            -0.01545409020036459,
+            0.012885984033346176,
+            0.0032442226074635983,
+            0.005764981731772423,
+            0.015817714855074883,
+            0.021893294528126717,
+            0.01746918261051178,
+            -0.00795431062579155,
+            0.006852070800960064,
+            0.004674104508012533,
+            0.028090080246329308,
+            -0.01777220331132412,
+            0.020802417770028114,
+            0.009954252280294895,
+            -0.003935490269213915,
+            0.005685438401997089,
+            -0.014257155358791351,
+            0.008530051447451115,
+            -0.007325541693717241,
+            0.030514251440763474,
+            0.0033067206386476755,
+            0.014075342565774918,
+            -0.02062060497701168,
+            0.006575563922524452,
+            -0.03172633796930313,
+            0.009022461250424385,
+            0.013704141601920128,
+            0.012976890429854393,
+            -0.007677804213017225,
+            -0.024181105196475983,
+            -0.0119314668700099,
+            0.0057990713976323605,
+            -0.01279507763683796,
+            -0.01046181283891201,
+            -0.0034828518982976675,
+            0.012446602806448936,
+            0.018847929313778877,
+            -0.01848430372774601,
+            -0.019923655316233635,
+            -0.009583051316440105,
+            0.02898399345576763,
+            -0.015787413343787193,
+            -0.001471547526307404,
+            0.02413565292954445,
+            0.021241798996925354,
+            0.015022284351289272,
+            0.002155239460989833,
+            0.020090317353606224,
+            -0.0007111532613635063,
+            -0.0237265732139349,
+            -0.02813553437590599,
+            0.007302815094590187,
+            -0.006912675220519304,
+            0.01943882182240486,
+            -0.0035548193845897913,
+            -0.005658924113959074,
+            -0.0034033088013529778,
+            -0.007503566797822714,
+            -0.0165904201567173,
+            -0.01787826046347618,
+            0.0021647089160978794,
+            -0.008454295806586742,
+            0.005863463506102562,
+            -0.0002795845502987504,
+            -0.013105674646794796,
+            0.013401119969785213,
+            -0.014075342565774918,
+            0.005772557109594345,
+            -0.002653330797329545,
+            0.001962063368409872,
+            0.0035926972050219774,
+            0.009113367646932602,
+            -0.007336904760450125,
+            -0.06381630152463913,
+            -0.013105674646794796,
+            -0.003704436356201768,
+            0.006071790587157011,
+            -0.00019578021601773798,
+            -0.016635872423648834,
+            -0.013196581043303013,
+            0.01706010289490223,
+            0.008863375522196293,
+            -0.013870803639292717,
+            0.016954045742750168,
+            0.009961827658116817,
+            0.003622999181970954,
+            -0.024044746533036232,
+            -0.0008148433989845216,
+            0.008757317438721657,
+            0.032756611704826355,
+            -0.008643684908747673,
+            -0.0003839849086944014,
+            0.0036684523802250624,
+            -0.0033332351595163345,
+            -0.010052734054625034,
+            -0.014582903124392033,
+            -0.011848135851323605,
+            0.011908739805221558,
+            0.01790856383740902,
+            -0.014264730736613274,
+            -0.02007516659796238,
+            -0.0007106798002496362,
+            0.012931437231600285,
+            -0.026590125635266304,
+            0.0007244104635901749,
+            -0.007787649054080248,
+            -0.01052999310195446,
+            0.018302490934729576,
+            -0.019014591351151466,
+            0.005249845329672098,
+            0.00048294031876139343,
+            0.008507325313985348,
+            -0.023544760420918465,
+            -0.024893205612897873,
+            0.0035131541080772877,
+            -0.023847781121730804,
+            0.03612014651298523,
+            -0.021590271964669228,
+            -0.012082977220416069,
+            0.034483831375837326,
+            -0.0037423139438033104,
+            0.01702980138361454,
+            0.0023995505180209875,
+            0.02739313244819641,
+            -0.011439057067036629,
+            -0.00617784820497036,
+            0.012499631382524967,
+            0.009575475007295609,
+            0.0032593735959380865,
+            0.03775646165013313,
+            -0.01257538702338934,
+            -0.013901105150580406,
+            -0.005556654185056686,
+            -0.012954163365066051,
+            0.0006775368237867951,
+            0.014431392773985863,
+            0.028393102809786797,
+            0.004257450345903635,
+            -0.00628769351169467,
+            -0.006143758539110422,
+            -0.0014649189542979002,
+            -0.004242299124598503,
+            -0.00947699323296547,
+            -0.00715130427852273,
+            0.017287369817495346,
+            -0.025241680443286896,
+            0.016711628064513206,
+            -0.006420265417546034,
+            0.021135739982128143,
+            -0.011757229454815388,
+            0.007007369305938482,
+            -0.008961857296526432,
+            0.003492321353405714,
+            0.0023635667748749256,
+            -0.005109698045998812,
+            0.009287605062127113,
+            -0.014772292226552963,
+            0.008454295806586742,
+            0.013742019422352314,
+            0.0011183382011950016,
+            -0.010590597055852413,
+            -0.019272159785032272,
+            0.0022972808219492435,
+            -0.000710206339135766,
+            0.01181783340871334,
+            -0.0047498601488769054,
+            -0.000981031684204936,
+            0.0002921315317507833,
+            -0.014476845972239971,
+            -0.009575475007295609,
+            0.0006050958181731403,
+            0.005302873905748129,
+            -0.008961857296526432,
+            0.010378481820225716,
+            -0.005693014245480299,
+            -0.00034160923678427935,
+            0.02627195231616497,
+            0.006526322569698095,
+            -0.007632351014763117,
+            -0.014098069630563259,
+            0.01563590206205845,
+            0.015370759181678295,
+            -0.007454325910657644,
+            0.01933276280760765,
+            -0.016196493059396744,
+            -0.0008366230758838356,
+            0.013916256837546825,
+            -0.009219424799084663,
+            0.0017698341980576515,
+            -0.008196728304028511,
+            0.03554440662264824,
+            -0.0008787619881331921,
+            -0.0035415622405707836,
+            -0.0006401326390914619,
+            0.012976890429854393,
+            -0.0026779514737427235,
+            0.001182730309665203,
+            0.01132542360574007,
+            0.005321812815964222,
+            0.0185146052390337,
+            0.00328020635060966,
+            -0.03378688171505928,
+            -0.007780073676258326,
+            -0.0065490491688251495,
+            -0.008090670220553875,
+            0.009022461250424385,
+            0.025196226313710213,
+            0.01990850456058979,
+            0.006408901885151863,
+            0.001881573349237442,
+            0.018423698842525482,
+            0.015757111832499504,
+            -0.01489350013434887,
+            0.016363153234124184,
+            0.02802947722375393,
+            -0.010030006989836693,
+            0.022741753607988358,
+            -0.005253633018583059,
+            -0.008893677033483982,
+            0.008946705609560013,
+            -0.0015378334792330861,
+            -0.017393426969647408,
+            -0.015681356191635132,
+            0.005022579338401556,
+            -0.008999735116958618,
+            -0.0022897052112966776,
+            -0.018605511635541916,
+            0.004670316819101572,
+            0.017105557024478912,
+            -0.004965762607753277,
+            -3.5243989259470254e-05,
+            0.019484274089336395,
+            -0.006965703796595335,
+            -0.016908591613173485,
+            0.006325571332126856,
+            -0.015007133595645428,
+            0.007946735247969627,
+            -0.013666263781487942,
+            0.010492115281522274,
+            -0.006098304875195026,
+            0.018817627802491188,
+            0.005446809343993664,
+            -0.008090670220553875,
+            -0.0008958069374784827,
+            -0.018984289839863777,
+            0.02065090648829937,
+            -0.033089932054281235,
+            0.00839369185268879,
+            -0.026226500049233437,
+            -0.0009781909175217152,
+            0.0008849170990288258,
+            0.014605630189180374,
+            -0.001218714052811265,
+            0.019696388393640518,
+            -0.002751812804490328,
+            -0.020090317353606224,
+            -0.007086912170052528,
+            -0.0010634156642481685,
+            -0.01618134044110775,
+            -0.030923331156373024,
+            -0.004272601567208767,
+            0.0036267871037125587,
+            -0.004711982328444719,
+            0.0021135739516466856,
+            0.017984317615628242,
+            -0.005613470915704966,
+            0.009173971600830555,
+            -0.02690829709172249,
+            -0.007416448090225458,
+            -0.00355103169567883,
+            0.0010776197304949164,
+            -0.011242092587053776,
+            0.0010596278589218855,
+            0.011226941831409931,
+            -0.019044892862439156,
+            0.013461724855005741,
+            -0.021378157660365105,
+            -0.004371083341538906,
+            -0.008507325313985348,
+            -0.00135412672534585,
+            -0.013446573168039322,
+            -0.028120383620262146,
+            0.025029566138982773,
+            -0.00021495578403118998,
+            -0.003528305096551776,
+            -0.011999646201729774,
+            0.023181134834885597,
+            0.002914686920121312,
+            0.027817361056804657,
+            -0.03157482668757439,
+            0.0013427634257823229,
+            0.011552689597010612,
+            0.0025984083767980337,
+            -0.017287369817495346,
+            0.004746072459965944,
+            -0.005234694108366966,
+            0.02210540883243084,
+            -0.0011694730492308736,
+            -0.0005340751959010959,
+            -0.008234606124460697,
+            0.030999086797237396,
+            0.009136094711720943,
+            0.0012944694608449936,
+            -0.013226882554590702,
+            -0.008060368709266186,
+            -0.002759388415142894,
+            -0.0011069749016314745,
+            0.012855681590735912,
+            0.01290113478899002,
+            -0.012105703353881836,
+            0.0013607552973553538,
+            -0.006878585088998079,
+            0.0011694730492308736,
+            -0.002935519441962242,
+            -0.0004966709529981017,
+            -0.01149208564311266,
+            0.003257479751482606,
+            0.016272246837615967,
+            -0.00780280027538538,
+            -0.006480869371443987,
+            0.0007750718505121768,
+            0.009181547909975052,
+            0.012461753562092781,
+            -0.012742049060761929,
+            -0.018226735293865204,
+            -0.011908739805221558,
+            0.016029831022024155,
+            0.010522416792809963,
+            -0.008204303681850433,
+            -0.013234458863735199,
+            -0.02583257108926773,
+            0.01169662456959486,
+            -0.03293842449784279,
+            -0.012242063879966736,
+            0.005821797996759415,
+            0.028559764847159386,
+            0.019999410957098007,
+            0.012226912193000317,
+            0.001332347048446536,
+            0.002950670663267374,
+            0.00023342114582192153,
+            -0.01916610263288021,
+            0.014408666640520096,
+            -0.011143610812723637,
+            -0.018166130408644676,
+            0.0075906855054199696,
+            0.011939042247831821,
+            0.03148392215371132,
+            0.006942977197468281,
+            -0.0010160685051232576,
+            0.01686313934624195,
+            -0.001218714052811265,
+            -0.013582932762801647,
+            0.01420412678271532,
+            -0.024256860837340355,
+            0.016438908874988556,
+            0.02860521711409092,
+            -0.004780162125825882,
+            0.0008531945641152561,
+            -0.020938776433467865,
+            0.00482561532407999,
+            -0.011529963463544846,
+            -0.002177966060116887,
+            -0.003287781961262226,
+            -0.03460504114627838,
+            0.007946735247969627,
+            -0.006825556512922049,
+            0.01046181283891201,
+            -0.00814369972795248,
+            0.006655106786638498,
+            -0.012431452050805092,
+            0.004253662656992674,
+            0.01513591781258583,
+            0.016105586662888527,
+            0.007060397882014513,
+            -0.025211378931999207,
+            -0.009196698665618896,
+            0.008105821907520294,
+            -0.01349202636629343,
+            -0.02295386977493763,
+            -0.006852070800960064,
+            0.009840618818998337,
+            -0.01061332318931818,
+            -0.00488243205472827,
+            -0.028180986642837524,
+            -0.013681414537131786,
+            -0.0014327229000627995,
+            -0.017817657440900803,
+            -0.005556654185056686,
+            -0.003100287402048707,
+            0.01571165770292282,
+            -0.0049733384512364864,
+            -0.006185423582792282,
+            0.004742284305393696,
+            -0.023878084495663643,
+            0.0037423139438033104,
+            -0.008280059322714806,
+            0.0008266801596619189,
+            -0.0032517979852855206,
+            -0.0014762822538614273,
+            -0.007969462312757969,
+            -0.014332910999655724,
+            0.006893736310303211,
+            -0.016469212248921394,
+            0.002954458352178335,
+            -0.015173795633018017,
+            -0.016499513760209084,
+            -0.00682934420183301,
+            0.00775355938822031,
+            0.021575121209025383,
+            0.006541473791003227,
+            0.021484214812517166,
+            0.005469535943120718,
+            0.028680972754955292,
+            0.0003953482082579285,
+            0.003041577059775591,
+            -8.958956641436089e-06,
+            0.003232859307900071,
+            0.019347915425896645,
+            0.006571775767952204,
+            0.018984289839863777,
+            -0.008742166683077812,
+            0.010514841414988041,
+            -0.010355755686759949,
+            0.010014856234192848,
+            0.006333146709948778,
+            -0.006473293993622065,
+            -0.003799130441620946,
+            -0.010348179377615452,
+            -0.0031419529113918543,
+            -0.016544966027140617,
+            0.0027366618160158396,
+            0.00693161366507411,
+            0.0071588801220059395,
+            -0.011582992039620876,
+            -0.005230906419456005,
+            -0.014704111963510513,
+            -0.0064354161731898785,
+            -0.01831764169037342,
+            0.014385939575731754,
+            0.029226411134004593,
+            -0.016332851722836494,
+            -0.01977214403450489,
+            -0.007772498298436403,
+            0.0007296186522580683,
+            -0.006636167876422405,
+            -0.01240872498601675,
+            -0.009151245467364788,
+            -0.035059571266174316,
+            -0.017666146159172058,
+            0.007586897816509008,
+            0.02484775334596634,
+            0.01387837901711464,
+            -0.013628385961055756,
+            -0.0006983695784583688,
+            -0.016984347254037857,
+            -0.010787560604512691,
+            -0.01535560842603445,
+            -0.02054484933614731,
+            -0.02498411200940609,
+            0.0077952248975634575,
+            -0.004033972043544054,
+            0.014931377954781055,
+            0.008113397285342216,
+            0.0032631612848490477,
+            -0.004371083341538906,
+            -0.0024109138175845146,
+            -0.013196581043303013,
+            0.0012925754999741912,
+            -0.004787737503647804,
+            -0.0017291157273575664,
+            0.03554440662264824,
+            -0.008098246529698372,
+            -0.005871038883924484,
+            0.011060279794037342,
+            0.011128460057079792,
+            -0.013764745555818081,
+            0.0006642796797677875,
+            -0.011249667964875698,
+            0.021847840398550034,
+            -0.021620575338602066,
+            0.019257009029388428,
+            0.0014507147716358304,
+            -0.010840589180588722,
+            0.028999146074056625,
+            0.01892368495464325,
+            0.00743159931153059,
+            0.006579351611435413,
+            0.01672677882015705,
+            0.0044316877610981464,
+            0.016196493059396744,
+            -0.0026552248746156693,
+            -0.004632438998669386,
+            0.009295180439949036,
+            0.015681356191635132,
+            -0.000702630786690861,
+            -0.006942977197468281,
+            0.016438908874988556,
+            0.004056698642671108,
+            0.0075414446182549,
+            -0.01356020662933588,
+            -0.008961857296526432,
+            0.001507531269453466,
+            0.006257391534745693,
+            -0.005939218681305647,
+            -0.0026003022212535143,
+            0.005530139897018671,
+            0.003943065647035837,
+            -0.013302638195455074,
+            0.01689344085752964,
+            -0.016166189685463905,
+            0.008742166683077812,
+            -0.0037972365971654654,
+            0.0121360057964921,
+            -0.006799042224884033,
+            0.0013153020991012454,
+            -0.01818128302693367,
+            -0.003869204083457589,
+            0.0038597346283495426,
+            0.012173883616924286,
+            -0.01262841559946537,
+            0.025484098121523857,
+            0.008166425861418247,
+            0.006602078210562468,
+            0.0059467945247888565,
+            -0.012855681590735912,
+            0.027211319655179977,
+            0.005969521123915911,
+            0.009886072017252445,
+            -0.008893677033483982,
+            0.011136035434901714,
+            0.035362593829631805,
+            0.03060515783727169,
+            0.014135947450995445,
+            0.004469565115869045,
+            -0.018999440595507622,
+            0.015120766125619411,
+            0.0018427487229928374,
+            0.0023806116078048944,
+            0.00863610953092575,
+            -0.011348150670528412,
+            0.01305264513939619,
+            -0.009484568610787392,
+            0.015241974964737892,
+            -0.021272100508213043,
+            -0.011749654076993465,
+            -0.014408666640520096,
+            -0.021075136959552765,
+            -0.01430260855704546,
+            0.034180812537670135,
+            -0.002346521709114313,
+            -0.010393633507192135,
+            0.0045112306252121925,
+            0.008477022871375084,
+            -0.006079366430640221,
+            0.0053142374381423,
+            -0.008204303681850433,
+            -0.013749594800174236,
+            -0.01889338344335556,
+            0.006628592498600483,
+            -0.01550711877644062,
+            0.007651289459317923,
+            -0.003304826794192195,
+            0.0037688282318413258,
+            0.007014944683760405,
+            0.018105527386069298,
+            0.00530666159465909,
+            -0.00135412672534585,
+            0.0014033677289262414,
+            0.0016865032957866788,
+            -0.00931033119559288,
+            0.0108330138027668,
+            0.015408637002110481,
+            -0.0011921997647732496,
+            -0.011279970407485962,
+            -0.009136094711720943,
+            -0.042877525091171265,
+            0.0006264019757509232,
+            -0.013128400780260563,
+            0.0064354161731898785,
+            -0.011196639388799667,
+            0.0022234192583709955,
+            0.015863168984651566,
+            -0.0054051438346505165,
+            -0.0010918239131569862,
+            0.02193874679505825,
+            0.013014768250286579,
+            0.021075136959552765,
+            0.02065090648829937,
+            0.014188976027071476,
+            -0.015302578918635845,
+            -0.03469594568014145,
+            0.002221525413915515,
+            -0.010030006989836693,
+            0.013772320933640003,
+            0.004238511435687542,
+            0.016060132533311844,
+            -0.023893235251307487,
+            -0.011878437362611294,
+            -0.006662682630121708,
+            0.014431392773985863,
+            0.01491622719913721,
+            -0.0032271775417029858,
+            0.02690829709172249,
+            0.009280029684305191,
+            0.0020510759204626083,
+            -0.02281750924885273,
+            0.003312402404844761,
+            0.005321812815964222,
+            -0.008370965719223022,
+            -0.016681326553225517,
+            -0.0004552422615233809,
+            -0.0012168202083557844,
+            -0.0007817004225216806,
+            -0.021650876849889755,
+            0.018832778558135033,
+            -0.004018820822238922,
+            0.0037574649322777987,
+            0.013840501196682453,
+            0.01516621932387352,
+            0.017984317615628242,
+            0.008370965719223022,
+            0.001497115008533001,
+            -0.006605865899473429,
+            -0.0007973249885253608,
+            -0.0005828426801599562,
+            -0.007389933802187443,
+            -0.005598319694399834,
+            0.006382387597113848,
+            -0.01134057529270649,
+            0.008810346014797688,
+            0.000896753859706223,
+            0.006113456096500158,
+            0.010605747811496258,
+            -0.004836978856474161,
+            -0.011628445237874985,
+            0.014628356322646141,
+            -0.0010548931313678622,
+            -0.0055755930952727795,
+            -0.011060279794037342,
+            0.003624893259257078,
+            0.013575357384979725,
+            -0.003096499713137746,
+            -0.013499601744115353,
+            0.00910579226911068,
+            0.002009410411119461,
+            -0.027438584715127945,
+            0.015499543398618698,
+            0.010295150801539421,
+            -0.013370818458497524,
+            0.004980913829058409,
+            -0.003371112747117877,
+            -0.03027183562517166,
+            0.004791525658220053,
+            0.002660906407982111,
+            -0.010643625631928444,
+            0.028938541188836098,
+            -0.003821857040748,
+            0.029256712645292282,
+            -0.004980913829058409,
+            -0.006905099377036095,
+            -0.011355726048350334,
+            0.009416389279067516,
+            -0.018681267276406288,
+            -0.0037953427527099848,
+            -0.008727015927433968,
+            -0.00915882084518671,
+            -0.0009838725673034787,
+            -0.005185453221201897,
+            0.006984642706811428,
+            0.005121061112731695,
+            -0.012704171240329742,
+            0.006757376715540886,
+            -0.010257272981107235,
+            -0.00020856392802670598,
+            0.0006718551740050316,
+            -0.008408842608332634,
+            0.013885954394936562,
+            0.005996035411953926,
+            -0.008340663276612759,
+            -0.0005032995832152665,
+            0.027817361056804657,
+            0.05051366239786148,
+            -0.018863080069422722,
+            0.0009270560112781823,
+            0.01452229917049408,
+            -0.0028824908658862114,
+            -0.0021590273827314377,
+            -0.011976920068264008,
+            0.005821797996759415,
+            -0.019423669204115868,
+            0.013317788951098919,
+            0.006473293993622065,
+            -0.008833073079586029,
+            0.003202557098120451,
+            -0.016711628064513206,
+            0.002621134975925088,
+            -0.0018058179412037134,
+            -0.018681267276406288,
+            -0.015529844909906387,
+            -0.012060250155627728,
+            -0.024181105196475983,
+            0.0004235197266098112,
+            -0.0008546149474568665,
+            -0.013870803639292717,
+            0.006579351611435413,
+            0.008431569673120975,
+            0.032483890652656555,
+            0.0011912527261301875,
+            0.010128489695489407,
+            -2.1513333194889128e-05,
+            -0.0010416359873488545,
+            -0.005988460034132004,
+            -0.01523439958691597,
+            0.00015387803432531655,
+            -0.002342734020203352,
+            0.022590242326259613,
+            0.005549078807234764,
+            0.017560089007019997,
+            0.0017887729918584228,
+            -0.016514664515852928,
+            -0.00596573343500495,
+            -0.007408872712403536,
+            -0.018060073256492615,
+            0.0062649669125676155,
+            -0.005473323632031679,
+            0.009605777449905872,
+            -0.015863168984651566,
+            -0.02731737680733204,
+            0.00792400911450386,
+            0.012234487570822239,
+            0.028514310717582703,
+            -0.0035623949952423573,
+            0.001716805505566299,
+            -0.005533927585929632,
+            0.016408607363700867,
+            0.032180868089199066,
+            -0.0008550884085707366,
+            0.006950552575290203,
+            -0.019999410957098007,
+            0.0017404790269210935,
+            0.013704141601920128,
+            -0.008401267230510712,
+            -0.01821158453822136,
+            0.003939277958124876,
+            -0.020635755732655525,
+            0.004583198111504316,
+            -0.020393338054418564,
+            0.030408194288611412,
+            -0.0023522034753113985,
+            0.008105821907520294,
+            0.004310478921979666,
+            0.01760554127395153,
+            -0.0031646795105189085,
+            0.010545143857598305,
+            0.013916256837546825,
+            0.014908651821315289,
+            0.0009980766335502267,
+            -0.011189064010977745,
+            -0.03790797293186188,
+            0.001859793672338128,
+            0.00817400123924017,
+            0.018635815009474754,
+            -0.01980244740843773,
+            0.011681473813951015,
+            -0.01296931505203247,
+            -0.020363036543130875,
+            -0.01003758329898119,
+            0.011575416661798954,
+            -0.006139970384538174,
+            0.03378688171505928,
+            -0.005874827038496733,
+            -0.008204303681850433,
+            0.005136212334036827,
+            0.010454237461090088,
+            0.0066778333857655525,
+            0.014704111963510513,
+            0.0326354019343853,
+            0.010196669027209282,
+            0.014173824340105057,
+            -0.004859705455601215,
+            0.02399929240345955,
+            -0.003931702114641666,
+            0.003954428713768721,
+            0.00509454682469368,
+            -0.022832660004496574,
+            -0.0040983641520142555,
+            -0.003931702114641666,
+            -0.016711628064513206,
+            -0.007458113599568605,
+            0.0032366469968110323,
+            0.008802770636975765,
+            -0.014045040123164654,
+            -0.00649980828166008,
+            -0.01350717805325985,
+            0.01218903437256813,
+            0.002730980049818754,
+            0.018302490934729576,
+            -0.029802151024341583,
+            0.009825468063354492,
+            -0.017484333366155624,
+            -0.017044952139258385,
+            0.006511171814054251,
+            -0.0038332203403115273,
+            0.006855858489871025,
+            -0.018272189423441887,
+            -0.01318900566548109,
+            0.012817803770303726,
+            0.0007897494360804558,
+            0.013060221448540688,
+            -0.009825468063354492,
+            0.0009706153068691492,
+            0.006064215209335089,
+            -0.009833043441176414,
+            -0.010961798019707203,
+            -0.007977037690579891,
+            -0.009681533090770245,
+            -0.008848223835229874,
+            0.005621046293526888,
+            0.01656011864542961,
+            -0.013742019422352314,
+            -0.003685497445985675,
+            0.016135888174176216,
+            0.006416477728635073,
+            -0.008166425861418247,
+            0.004075637552887201,
+            -0.003793448908254504,
+            -9.948806837201118e-05,
+            0.0013740125577896833,
+            0.012439027428627014,
+            -0.0017632056260481477,
+            0.024665940552949905,
+            -0.02654467150568962,
+            0.007598260883241892,
+            -0.0054051438346505165,
+            0.005446809343993664,
+            0.018226735293865204,
+            -0.02366596832871437,
+            -0.013082947582006454,
+            0.006276329979300499,
+            -0.01966608688235283,
+            -0.008060368709266186,
+            0.0054051438346505165,
+            -0.00016405766655225307,
+            -0.019923655316233635,
+            -0.012923861853778362,
+            -0.034786853939294815,
+            -0.02309022843837738,
+            0.024665940552949905,
+            0.008401267230510712,
+            -0.0073672072030603886,
+            0.018060073256492615,
+            -0.018438849598169327,
+            0.0172722190618515,
+            -0.002225313102826476,
+            -0.006613441277295351,
+            0.003935490269213915,
+            0.007674016058444977,
+            0.025362888351082802,
+            0.002653330797329545,
+            0.008264907635748386,
+            -0.0042082094587385654,
+            -0.014931377954781055,
+            -0.006321783177554607,
+            0.01980244740843773,
+            0.02471139281988144,
+            0.018605511635541916,
+            -0.018575210124254227,
+            0.010742107406258583,
+            -0.011749654076993465,
+            0.01818128302693367,
+            -0.006537686102092266,
+            0.00039037675014697015,
+            0.01859036087989807,
+            0.007969462312757969,
+            0.025241680443286896,
+            0.019923655316233635,
+            0.006852070800960064,
+            0.014128371141850948,
+            0.011908739805221558,
+            0.001204509986564517,
+            -0.012832955457270145,
+            -0.01154511421918869,
+            -0.02840825356543064,
+            0.042847223579883575,
+            0.0025434857234358788,
+            0.008317936211824417,
+            -0.027635548263788223,
+            -0.00021389048197306693,
+            -0.0034942151978611946,
+            -0.018832778558135033,
+            0.005939218681305647,
+            0.03260510042309761,
+            0.03284751623868942,
+            0.021438762545585632,
+            -2.888172457460314e-05,
+            0.0012234487803652883,
+            0.0329081192612648,
+            -0.013673839159309864,
+            -0.00937851145863533,
+            -0.01034060399979353,
+            -0.018393397331237793,
+            -0.006852070800960064,
+            -0.0032423287630081177,
+            -0.01031030248850584,
+            -0.0061172437854111195,
+            -0.009272454306483269,
+            0.0238629337400198,
+            0.0001374249259242788,
+            -0.0022594030015170574,
+            0.006605865899473429,
+            -0.020878173410892487,
+            0.011537538841366768,
+            -0.02813553437590599,
+            -0.013431422412395477,
+            -0.008651260286569595,
+            -0.006950552575290203,
+            -0.018802475184202194,
+            0.008643684908747673,
+            -0.006942977197468281,
+            0.021438762545585632,
+            0.002914686920121312,
+            -0.023590214550495148,
+            0.007530081085860729,
+            0.015317730605602264,
+            -0.0003825644962489605,
+            0.014037464745342731,
+            -0.003890036838129163,
+            0.009060339070856571,
+            0.0045112306252121925,
+            0.011681473813951015,
+            0.016954045742750168,
+            0.010651201009750366,
+            0.009030036628246307,
+            0.003437398700043559,
+            -0.001167579204775393,
+            0.01878732442855835,
+            -0.0007163614500313997,
+            -0.03405960276722908,
+            2.9236827685963362e-05,
+            -0.014628356322646141,
+            0.029499130323529243,
+            -0.009795165620744228,
+            0.009514871053397655,
+            0.008105821907520294,
+            0.007098275702446699,
+            -0.0067119235172867775,
+            0.007916433736681938,
+            -0.008742166683077812,
+            0.0052763596177101135,
+            0.015832867473363876,
+            -0.0042877523228526115,
+            -0.0014980619307607412,
+            0.007840678095817566,
+            0.00407184986397624,
+            -0.005454384721815586,
+            -0.019711541011929512,
+            -0.012287517078220844,
+            0.00726114958524704,
+            -0.02065090648829937,
+            -0.004367295652627945,
+            0.01161329448223114,
+            -0.005454384721815586,
+            -0.009620928205549717,
+            0.01393140759319067,
+            0.02136300690472126,
+            -0.002382505452260375,
+            0.022893264889717102,
+            -0.01349202636629343,
+            -0.009833043441176414,
+            -0.007984613068401814,
+            -0.004374871030449867,
+            -0.041120000183582306,
+            0.025984082370996475,
+            0.01068150345236063,
+            -0.006871009711176157,
+            0.01895398646593094,
+            0.0022499337792396545,
+            -0.002954458352178335,
+            -0.0011041341349482536,
+            0.00010499217023607343,
+            -0.03281721472740173,
+            0.03660498186945915,
+            -0.007007369305938482,
+            0.0005449649761430919,
+            0.010272424668073654,
+            0.042029064148664474,
+            -0.004590773954987526,
+            -0.005859675817191601,
+            0.01225721463561058,
+            0.005336964037269354,
+            0.031938452273607254,
+            -0.010416359640657902,
+            -0.014340486377477646,
+            -0.01815097965300083,
+            -0.004833190701901913,
+            -0.01939336769282818,
+            -0.0038313264958560467,
+            0.014037464745342731,
+            -0.019650936126708984,
+            -0.011242092587053776,
+            -0.001287840772420168,
+            -0.012537509202957153,
+            -0.0010747789638116956,
+            0.008598231710493565,
+            0.017832808196544647,
+            -0.011348150670528412,
+            -0.004450626205652952,
+            -2.2904154320713133e-05,
+            -0.005003640428185463,
+            0.0185146052390337,
+            0.0020056227222085,
+            -0.01702980138361454,
+            0.01321930717676878,
+            -0.00526499655097723,
+            0.006749800872057676,
+            0.0074429623782634735,
+            0.007939159870147705,
+            0.05145302787423134,
+            0.010317877866327763,
+            -0.001339922659099102,
+            -0.01607528328895569,
+            0.023196285590529442,
+            -0.01749948412179947,
+            -0.014181400649249554,
+            -0.02119634486734867,
+            -0.0029923359397798777,
+            -0.007060397882014513,
+            -0.028014326468110085,
+            -9.137987944995984e-05,
+            0.0016221113037317991,
+            -0.0024980325251817703,
+            -0.010045158676803112,
+            0.0005208179936744273,
+            -0.011529963463544846,
+            -0.006954340264201164,
+            -0.008545203134417534,
+            -0.016044981777668,
+            0.02240842953324318,
+            -0.021817538887262344,
+            -0.008855799213051796,
+            0.011514811776578426,
+            0.03118089959025383,
+            -0.005011215806007385,
+            -0.014173824340105057,
+            -0.019817598164081573,
+            0.0025643182452768087,
+            -0.01763584464788437,
+            0.012181458994746208,
+            -0.01225721463561058,
+            0.015045011416077614,
+            -0.03142331540584564,
+            -0.006079366430640221,
+            -0.008992158807814121,
+            0.012840530835092068,
+            0.019817598164081573,
+            -0.0025093958247452974,
+            0.00606800289824605,
+            -0.010295150801539421,
+            0.030211230739951134,
+            -0.00012286569108255208,
+            -0.018605511635541916,
+            0.00873459130525589,
+            0.007977037690579891,
+            0.012537509202957153,
+            -0.004435475450009108,
+            0.00257757562212646,
+            0.011567841283977032,
+            0.012742049060761929,
+            -0.01634800247848034,
+            -0.003174148965626955,
+            -0.017969166859984398,
+            0.005996035411953926,
+            0.006999793462455273,
+            0.021711481735110283,
+            -0.0013550736475735903,
+            0.018272189423441887,
+            -0.01563590206205845,
+            0.00019412307301536202,
+            -0.011749654076993465,
+            -0.0082270298153162,
+            -0.0028237802907824516,
+            -0.003958216868340969,
+            0.006685409229248762,
+            -0.028211290016770363,
+            -0.01274962443858385,
+            0.005287723150104284,
+            -0.004992276895791292,
+            -0.03405960276722908,
+            0.0053710537031292915,
+            -0.02725677192211151,
+            0.013128400780260563,
+            -0.002397656673565507,
+            -0.017847958952188492,
+            -0.00030846628942526877,
+            0.026332557201385498,
+            -0.003677921835333109,
+            -0.012537509202957153,
+            0.015544996596872807,
+            -0.006757376715540886,
+            -0.02119634486734867,
+            -0.005420294590294361,
+            0.014658658765256405,
+            -0.0010719380807131529,
+            -0.012393574230372906,
+            -0.029635490849614143,
+            0.012992041185498238,
+            0.0026874206960201263,
+            -0.007386146113276482,
+            0.010893618687987328,
+            0.025953780859708786,
+            0.008151275105774403,
+            -0.005924067925661802,
+            0.0015757111832499504,
+            0.0049581872299313545,
+            0.00300938100554049,
+            0.020287280902266502,
+            0.02728707529604435,
+            -0.008014915511012077,
+            -0.0011145505122840405,
+            -0.014954105019569397,
+            0.015802564099431038,
+            0.0013020449550822377,
+            0.010507266037166119,
+            0.021893294528126717,
+            0.007045246660709381,
+            -0.005738467443734407,
+            0.003905187826603651,
+            -0.023135682567954063,
+            0.01420412678271532,
+            -0.012272365391254425,
+            0.000622140767518431,
+            0.008749742060899734,
+            -0.007283876184374094,
+            0.01437078882008791,
+            -0.0015245762187987566,
+            0.0339989997446537,
+            0.025211378931999207,
+            0.0043218424543738365,
+            0.0007362472242675722,
+            -0.028559764847159386,
+            0.003547244006767869,
+            0.008651260286569595,
+            0.014499573037028313,
+            -0.0076664406806230545,
+            0.006261179223656654,
+            -0.004355932120233774,
+            0.00959820207208395,
+            -0.0065490491688251495,
+            -0.015893470495939255,
+            0.006113456096500158,
+            0.0012689019786193967,
+            0.006795254070311785,
+            0.0023408401757478714,
+            -0.008863375522196293,
+            -0.024362917989492416,
+            -0.025984082370996475,
+            0.021984200924634933,
+            -0.03918066248297691,
+            0.0073293293826282024,
+            0.011234517209231853,
+            0.03324144333600998,
+            -0.008158850483596325,
+            -0.004980913829058409,
+            0.006276329979300499,
+            -0.016711628064513206,
+            -0.004147605039179325,
+            0.008204303681850433,
+            0.012385998852550983,
+            -0.002420383272692561,
+            0.00974971242249012,
+            0.01318900566548109,
+            -0.012378423474729061,
+            0.03660498186945915,
+            0.016514664515852928,
+            0.01467380952090025,
+            -0.004007457755506039,
+            0.01312082540243864,
+            0.0066172294318675995,
+            0.019044892862439156,
+            0.025165924802422523,
+            -0.017454031854867935,
+            -0.002365460619330406,
+            -0.02292356640100479,
+            0.024165954440832138,
+            0.00726114958524704,
+            0.004556683823466301,
+            -0.008310360834002495,
+            0.014870774000883102,
+            0.0017954016802832484,
+            0.0028199926018714905,
+            -0.006696772295981646,
+            -0.010173942893743515,
+            0.019241856411099434,
+            0.0034317169338464737,
+            0.014870774000883102,
+            4.8501173296244815e-05,
+            -0.005503625608980656,
+            0.006052851676940918,
+            0.014832896180450916,
+            -0.023590214550495148,
+            -0.00203592493198812,
+            -0.019484274089336395,
+            -0.03202936053276062,
+            0.009022461250424385,
+            0.0016135887708514929,
+            -0.007052822504192591,
+            0.03230207785964012,
+            -0.010083036497235298,
+            -0.019787294790148735,
+            -0.008067944087088108,
+            -0.013113250024616718,
+            -0.010469388216733932,
+            0.0008271536789834499,
+            0.003908975515514612,
+            -0.00012428610352799296,
+            -0.009317907504737377,
+            -0.005749830510467291,
+            0.01990850456058979,
+            -0.0030358952935785055,
+            -0.00753386877477169,
+            0.00868156272917986
+          ],
+          "embedding": [
+            0.0040707582550429606,
+            -0.010230625151989194,
+            -0.021520242101903186,
+            -0.010115756270640508,
+            0.026738734586159035,
+            0.01796370233223267,
+            -0.012245199029994153,
+            0.06506924494147726,
+            -0.0014087587222033442,
+            0.009300068719271429,
+            0.003787396973047584,
+            0.002818772639754174,
+            0.022129077062176306,
+            -0.021526709808445683,
+            -0.0034388135367225824,
+            0.02763761747879405,
+            0.013210013805395276,
+            0.025255872672296602,
+            0.00029011480909986703,
+            -0.05438974762290585,
+            0.016671640187490865,
+            -0.016426252259893328,
+            -0.056591180651012554,
+            0.009335648425523075,
+            0.006141624459418827,
+            0.02568943956694641,
+            0.019377132054042116,
+            0.017946464337078746,
+            -0.057184776985927686,
+            0.03208527139096687,
+            0.030132392063755547,
+            -0.003846032371791839,
+            -0.008376524427531415,
+            -0.0010513963058713435,
+            -0.02734868388724371,
+            -0.014879400370545479,
+            0.04383920959748825,
+            0.02563613659516289,
+            -0.020117467074438997,
+            0.0398016699155716,
+            0.029896497884197775,
+            -0.0004195844530067313,
+            -0.023302720788253625,
+            -0.001168337916577517,
+            0.009149021369661628,
+            -0.02508145721900992,
+            -0.004307594138480351,
+            0.00550940415564303,
+            0.014016998918718476,
+            -0.028607379616884317,
+            -0.0018317656587526954,
+            -0.01973464940415756,
+            0.03762738392056255,
+            0.015358106058583114,
+            -0.03931907682104936,
+            0.011519637298832444,
+            0.004482080558226775,
+            0.0747867063383242,
+            0.05550679018263077,
+            0.023243129601931176,
+            0.02671631686665238,
+            -0.009686146566573536,
+            -0.04088582189725495,
+            -0.0065344623594924085,
+            0.019617295999325712,
+            -0.004722087402575777,
+            -0.002871844114437583,
+            0.003935087265300519,
+            0.03229890413393201,
+            -0.02813577047904649,
+            -0.05124172132597234,
+            0.03325235407841968,
+            -0.008687655557217192,
+            -0.011027174838246948,
+            -0.003653553063547375,
+            -0.02350737945182988,
+            0.016223087939183468,
+            0.03789252356462714,
+            0.0034350623378603083,
+            0.00195158657888739,
+            0.011789712862232108,
+            -0.0004435439660769551,
+            -0.0037330456918125894,
+            0.0031552713243866858,
+            0.04465432992298031,
+            -0.0032465421571915712,
+            -0.0075366743861922745,
+            -0.059958752731410256,
+            0.016784616861008125,
+            -0.028734506065772967,
+            -0.03035575736273943,
+            0.02680390892611741,
+            0.02338368719038086,
+            -0.0030658375437784277,
+            -0.03754896050794851,
+            0.07005659523248504,
+            0.0020711575553570695,
+            0.013034425083076923,
+            -0.02316832262132332,
+            0.021332142030732894,
+            -0.0318756233020585,
+            0.02270782509507388,
+            0.01570810309454529,
+            0.004788124787612736,
+            0.026727250200142456,
+            -0.022380142778058276,
+            -0.008593910379436904,
+            0.014439754382387034,
+            0.015007697617383174,
+            0.07662531963612977,
+            0.011195998137723795,
+            -0.010809283538470098,
+            -0.042227822120219305,
+            0.003824320933275175,
+            -0.03797629427607094,
+            -0.02347390684453348,
+            -0.0526876440280276,
+            -0.017799861680287913,
+            0.01701482796266917,
+            0.002730395144917155,
+            0.03789451759899825,
+            0.038338545350313946,
+            0.028463789278406586,
+            0.015494583705011921,
+            -0.05700237266350479,
+            0.047895170359075166,
+            -0.0064406894534929406,
+            -0.0007954721295065988,
+            0.012543029839729358,
+            -0.004703334868741216,
+            -0.0060569679284068,
+            0.012324901254711192,
+            -0.019883935809784068,
+            -0.04253565942154215,
+            0.02164409276211115,
+            -0.0075367555415808635,
+            0.03617159147722157,
+            0.07567745988686005,
+            -0.04225977020036783,
+            -0.022548837589622003,
+            0.029811602145850144,
+            -0.02175792196819533,
+            0.02069861458115363,
+            -0.00941331829664397,
+            -0.035479639885943264,
+            0.020352552580119105,
+            0.003678588588550461,
+            0.06776105274336729,
+            -0.02346047721661687,
+            0.02721075076349443,
+            -0.029386781707644253,
+            0.023530831921117993,
+            -0.02077660533100372,
+            0.020022475323372906,
+            0.024195118676513715,
+            -0.028378518720494666,
+            -0.012900969023913307,
+            0.019264089970719624,
+            -0.0374983624629265,
+            -0.019726171557384355,
+            0.00979636085642416,
+            -0.014479528435832063,
+            -0.014533384583777662,
+            0.005904623758767224,
+            0.010344475500263744,
+            0.05659646358026115,
+            0.04163866113482093,
+            0.007152494819169405,
+            -0.07584500130974763,
+            0.05471858597878099,
+            0.03580211013427615,
+            0.044161429444475306,
+            0.02258180566046248,
+            0.00980779221334715,
+            0.0011193505603397395,
+            0.018086566768892507,
+            0.015605066702191871,
+            -0.00954727399396313,
+            -0.027950079146919375,
+            0.03028514025485407,
+            -0.006351669761257431,
+            0.005095804801686436,
+            0.009620040085743162,
+            0.019748001227987134,
+            0.033709535059510916,
+            0.008203559956940196,
+            0.007389752602218563,
+            0.04197771878351742,
+            -0.02067363758471765,
+            0.007543739188471913,
+            0.011207941261799052,
+            0.013964827251172813,
+            0.03839562075248879,
+            0.0009775244543292637,
+            0.006469507611492216,
+            -0.022808235395746654,
+            -0.024795004398633292,
+            0.006249024864814684,
+            0.011292877109066925,
+            0.01596839619826819,
+            -0.0014105396434174883,
+            -0.04152075438277094,
+            0.008267788411079337,
+            -0.028834191212891387,
+            0.030940526405027906,
+            -0.0018271056434014425,
+            0.006185475047820603,
+            -0.007920574674727123,
+            0.012907560753043616,
+            -0.015165704767187252,
+            -0.021962244496450727,
+            0.0037071053072833147,
+            -0.013508376064336042,
+            -0.015178823540513864,
+            -0.02518557053061311,
+            0.011693256916760944,
+            -0.01716015674592331,
+            0.02048897513447166,
+            -0.003412078966361745,
+            -0.001807298511378529,
+            -0.0012575655363779904,
+            0.03116103961636643,
+            0.011668533361641842,
+            -0.011637010548030312,
+            -0.040260195687925236,
+            -0.011129376691522074,
+            -0.0034493180842871908,
+            -0.02098244789711689,
+            0.026658743116016938,
+            -0.01923736325953796,
+            -0.009723937181300192,
+            -0.013134679989565051,
+            -0.011974548914818263,
+            -0.004028712301052159,
+            0.019628329175867303,
+            -0.028192034966932684,
+            -0.012638520793012082,
+            -0.011568652932013111,
+            -0.0025919910519445737,
+            0.012497908463405111,
+            0.015113186712897145,
+            0.014880607940212641,
+            -0.03059227859452334,
+            -0.021899716217105084,
+            -0.0008513506358489523,
+            0.011985931118464679,
+            -0.01321584595585435,
+            -0.011735288366213981,
+            0.010710683541435907,
+            0.01961722172525993,
+            -0.023571946726192585,
+            -0.03364281880125354,
+            0.0014460076743756465,
+            0.016563167828467187,
+            -0.02891589059260944,
+            0.04595166331401113,
+            0.034271553243501465,
+            -0.027557971577291153,
+            -0.007138821520412525,
+            -0.039508407807651844,
+            0.06814329636830853,
+            0.00022136123564675344,
+            -0.039718016705753735,
+            0.0503457594963753,
+            -0.003580581891576583,
+            0.013820796398521455,
+            0.02054050419088757,
+            0.035682661890720495,
+            -0.0013220739026852943,
+            -0.0052759394697328675,
+            0.04733018719162478,
+            -0.0187845855131196,
+            -0.03982699468608443,
+            -0.023905460465070062,
+            0.002631233133039351,
+            -0.03090995554989513,
+            -0.012617612051488022,
+            -0.013605787969473513,
+            0.039672783793764745,
+            0.008669582242097126,
+            0.009974277556736362,
+            0.005535636914843583,
+            0.014180668493093584,
+            0.012042659947850441,
+            0.011870115226306776,
+            0.00451259454348581,
+            -0.014023048592193627,
+            -0.007769033996392623,
+            -0.05715987527668358,
+            0.003374564518384728,
+            -0.028801158523506985,
+            -0.00756331369170698,
+            -0.02163054894055572,
+            0.005367181573233089,
+            0.02308705505078254,
+            -0.005438909398653553,
+            -0.00873620909426313,
+            -0.0036530652237719077,
+            0.013920407174296543,
+            -0.01539613379172909,
+            -0.03908211213269472,
+            -0.016810726548844992,
+            0.007460173854386443,
+            -0.007069189426742819,
+            -0.041197222498539215,
+            -0.003843233633335066,
+            0.004373367480223065,
+            0.016194121660459908,
+            -0.029165150066068787,
+            -0.00735139702658342,
+            0.017257895082337448,
+            0.014153313280536095,
+            -0.002376178042741926,
+            -0.023223700836427403,
+            -0.0319877537205969,
+            0.02088060774052704,
+            0.0012829290529515294,
+            0.03251550040316258,
+            -0.04128453069131847,
+            -0.023376114412063578,
+            0.04083022735076889,
+            0.005690491480259003,
+            -0.05810403800689653,
+            0.016582434816685452,
+            0.010402038964417569,
+            0.016795644054313566,
+            -0.00908590043098552,
+            0.05003656634510042,
+            -0.037216197954986735,
+            -0.013027960261454162,
+            -0.007090943677204207,
+            -0.05175371787508879,
+            -0.0020065660448063322,
+            -0.014908489552053589,
+            0.010627472758306387,
+            -0.03275473161158777,
+            -0.017334608434133866,
+            0.025321047439048318,
+            -0.0033515195870701753,
+            0.011526068468647207,
+            -0.026783879816504975,
+            -0.0012830935419413407,
+            -0.017506908439998053,
+            -0.02114853058440477,
+            0.004878884893669383,
+            -0.020936900837115968,
+            -0.0025587027781733642,
+            0.01557850246191957,
+            0.023652335794299834,
+            0.011489428313303988,
+            0.0019089137782552768,
+            0.02054907941022243,
+            0.014361232983780595,
+            -0.0056413109308506085,
+            -0.0010300050480036684,
+            -0.03413262505887213,
+            -0.018821160485010483,
+            0.03361989907065425,
+            0.013125889701529858,
+            -0.003742167496379789,
+            0.008813100942466033,
+            -0.0383265293638935,
+            -0.035316957263713485,
+            0.03020499323039515,
+            -0.05001623971243151,
+            -0.00855479667673597,
+            -0.005939674940109814,
+            -0.026451831943555135,
+            -0.008032873049527808,
+            0.0009431571685654594,
+            -0.014387605128858954,
+            0.016382559541755014,
+            -0.024909510232038502,
+            0.003715736505444385,
+            -0.006016712902409688,
+            -0.009110197692342108,
+            0.05507263160354303,
+            -0.022103031032598353,
+            -0.0475415595313825,
+            0.021958434235402655,
+            -0.016670809659931243,
+            0.0323519363801999,
+            0.03867494951405658,
+            0.011124508964363016,
+            -0.008002445714431074,
+            0.034041838454818577,
+            0.018400123235267234,
+            -0.01161754789471199,
+            -0.025901844576206774,
+            0.021800298132870245,
+            0.00943488425697727,
+            -0.05906691123008531,
+            -0.00782549274387971,
+            -0.019357446956797447,
+            0.008682376344965976,
+            0.02863077694455766,
+            0.0024925977373894447,
+            -0.002682170968419412,
+            -0.013590235283325244,
+            -0.021511149507762284,
+            -0.03255453916518719,
+            0.009039307015853905,
+            0.008305286476500875,
+            0.009312625027668458,
+            -0.0014144165766983945,
+            -0.007853869334376857,
+            0.009067065760942163,
+            0.022485201611182105,
+            0.03966189080765722,
+            -0.041627289519629344,
+            0.002707086456239785,
+            -0.022678178050044222,
+            -0.03653753142622053,
+            0.0036574646778835588,
+            0.029996160786576367,
+            0.02901163765559891,
+            -0.01971777193183787,
+            -0.0036589934841109884,
+            0.009181234938554578,
+            -0.025979483002227535,
+            -0.00773645103995767,
+            0.004280770859737866,
+            0.04116668301017644,
+            0.006201085381597264,
+            0.01402728258112097,
+            0.001060407317110527,
+            -0.00920801784000087,
+            0.02109880889040528,
+            0.025211022083982727,
+            -0.029866678264942617,
+            0.014439702646820226,
+            0.009379206296908577,
+            0.04156435812161122,
+            -0.037309655469525846,
+            -0.011819731348299495,
+            -0.02766498020409985,
+            -0.010454143230794476,
+            0.027911532296566614,
+            -0.04127890742215716,
+            -0.006948536183086887,
+            0.010207784113909938,
+            -0.0016584043356136647,
+            0.016437495428919636,
+            -0.014607861750686952,
+            -0.03222311919805017,
+            -0.000610723203118213,
+            -0.02867445198803213,
+            -0.024772816792038133,
+            -0.016510309753766612,
+            -0.018581201775667577,
+            -0.07450654873867177,
+            0.00026832248829154255,
+            -0.06724696990816072,
+            -0.02447002541264633,
+            0.04885569249258513,
+            -0.010195604207134053,
+            0.03332905935704749,
+            -0.02815272112271314,
+            -0.005296340304911926,
+            -0.05908835042526019,
+            0.0038476172058970774,
+            0.025753576863447866,
+            0.02386003857017594,
+            0.005206447371391696,
+            0.024799057871126032,
+            0.012060738268265606,
+            -0.04146503002980051,
+            -0.024949907556862643,
+            -0.014503475223770958,
+            -0.0016340466065744286,
+            -0.0034009167013795906,
+            -0.031298432995411005,
+            0.036241272149880994,
+            0.008902644391348814,
+            0.03550223922667804,
+            0.04463301409716232,
+            -0.010385984413759852,
+            -0.03305004479734889,
+            -0.006952739338820577,
+            -0.0175981588092108,
+            0.042668533202572856,
+            -0.017112184620086308,
+            0.005149173014526686,
+            -0.03219589107997635,
+            0.024181808150698054,
+            0.0014339790422945406,
+            0.01254721190611376,
+            0.011099458649404119,
+            0.0027626836767364253,
+            -0.026883640446090158,
+            0.0024160770766463607,
+            0.023874514847956222,
+            -0.022558433617669373,
+            0.00917667715473093,
+            -0.006754091166465801,
+            -0.033686935718776144,
+            -0.01078389638118685,
+            -0.013595327486167114,
+            0.024712781935748277,
+            0.013168464807451798,
+            0.005875864604862434,
+            -0.02417814420558463,
+            -0.0023978800459714144,
+            0.028650463366555304,
+            0.020176980768301288,
+            -0.02118184201861616,
+            0.019082340279671355,
+            0.019734925740100277,
+            -0.04209645071102001,
+            -0.04217310001316092,
+            0.006553913719098956,
+            -0.003735719227468833,
+            0.0061793909220459,
+            -0.028285889794908054,
+            0.008181788076484304,
+            0.004432488437404309,
+            -0.01743905248466261,
+            0.041221269621866216,
+            -0.014892354419172107,
+            0.005087102208335201,
+            -0.022198383716186407,
+            -0.011936388003605221,
+            -0.00027487042024290816,
+            0.044599215801158845,
+            0.008068975171827777,
+            0.003496950921841073,
+            0.02238779276447619,
+            0.0054839808262270315,
+            0.04816723616939378,
+            -0.02685221260466647,
+            -0.0190356946337995,
+            0.016464136459560113,
+            -0.0176092780782677,
+            -0.012513274095432353,
+            -0.010695421445334002,
+            -0.004784348908457072,
+            -0.022203041334932165,
+            0.010788155380415583,
+            -0.018353897272137543,
+            -0.02277431189907027,
+            -0.003945465850796918,
+            0.016024639792926797,
+            -0.006248368166794934,
+            -0.0038769803892794875,
+            0.0016292556540434277,
+            0.02412864830338344,
+            0.02892955453819876,
+            -0.02408022470555968,
+            0.019190282372105023,
+            0.018941584034503536,
+            -0.006709516380602965,
+            -0.020956377933747532,
+            -0.01964885206488053,
+            0.01434623424964373,
+            -0.01352968586212152,
+            -0.013872556892707697,
+            0.004052798542613835,
+            0.009201147374299997,
+            -0.004086067985859617,
+            -0.04185829685685201,
+            -0.014668698123513235,
+            0.003179403563664022,
+            -0.0006415014901450319,
+            0.019360184344240947,
+            -0.0150817538567364,
+            -0.029771999278536083,
+            0.014221776097046751,
+            0.022944884879589678,
+            -0.04493425220030206,
+            0.02117577719533236,
+            -0.0030648751497261983,
+            -0.00016251533643103604,
+            -3.817296018601414e-05,
+            0.008727767382077138,
+            0.026914779443266063,
+            0.03456565094655899,
+            0.01974806772696219,
+            0.00450800007152441,
+            -0.008952599700759673,
+            0.025184190461155245,
+            0.009846044083064704,
+            -0.025168735183134738,
+            0.010983353565382029,
+            0.016740721743326522,
+            0.0070838930012654296,
+            0.009173879407277282,
+            -0.006078288608766152,
+            -0.018269347893097643,
+            0.013181494188066667,
+            -0.01435693747022231,
+            0.01678907949532146,
+            0.05154463552784974,
+            -0.00039632596424461964,
+            -0.02461439550772063,
+            0.008418147037164186,
+            -0.006544319842335593,
+            -0.010065129286193278,
+            0.02214628550460128,
+            0.0029515574751370417,
+            -0.030862152033847717,
+            0.007509556025223583,
+            0.00675485304374519,
+            0.01894936231559203,
+            -0.02355525295639599,
+            -0.01387885858544803,
+            -0.00431398020087544,
+            0.0021208056468763523,
+            0.01580766504149989,
+            -0.004739477121073512,
+            -0.013204823447960706,
+            0.02994098229517376,
+            0.025496728234662373,
+            -0.00026124946738213036,
+            0.015060064980768175,
+            0.022262345898533903,
+            0.009968408664790368,
+            0.007588431141088797,
+            -0.0083063413609779,
+            -0.031336272171937095,
+            0.0018957858383561667,
+            0.05800811417728381,
+            -0.014937085296865802,
+            0.04313875432235027,
+            0.02018476425923236,
+            0.015657644644987107,
+            0.017880263053253607,
+            -0.01662268082870562,
+            -0.008740053682875323,
+            -0.0017630394204179638,
+            -0.008294417990024647,
+            -0.027613464270562948,
+            -0.005953209556351086,
+            -0.02379961210881074,
+            -0.013878676554701543,
+            0.015418315245979704,
+            -0.014764728065382074,
+            0.0027420640870858457,
+            0.010395538988699663,
+            -0.006590633833835581,
+            -0.00841537209927409,
+            -0.003943772700797381,
+            0.03376338017637296,
+            0.011980354721458463,
+            0.027761929822019422,
+            -0.00011128539412582973,
+            0.009342430952107655,
+            -0.018419526564396063,
+            -0.006839752805318664,
+            0.03271190887864607,
+            0.006011520427944665,
+            0.004652835471939905,
+            0.01485920920301398,
+            0.014305626381398871,
+            0.038015677219487023,
+            -0.005102005401565164,
+            -0.018204460895989052,
+            0.02065495133597456,
+            -0.0049699013356374775,
+            0.019149036693562712,
+            0.008825981577388056,
+            -0.002660055848523568,
+            -0.004292379561953203,
+            0.03090270710860331,
+            -0.020243440437217503,
+            0.018735727616321387,
+            -0.02238236089688497,
+            -0.0276584413929402,
+            0.023028408906176946,
+            0.008320436733620496,
+            0.04047641118026783,
+            0.011463851066078041,
+            -0.02247143742166039,
+            -0.034588725413285185,
+            0.009827713930789184,
+            0.00337071450636392,
+            -0.024446280199879062,
+            0.017542777160283324,
+            0.00780836483490212,
+            0.008453886667283516,
+            0.016624116261219637,
+            -0.004175682824655864,
+            -0.0042784362933758835,
+            -0.001546656717896957,
+            -0.008819739495525477,
+            0.01406790007187259,
+            -0.015487658289541583,
+            -0.021457182907140035,
+            0.024687169583245905,
+            -0.03915911962859764,
+            0.01771290139419629,
+            -0.006510046907173026,
+            -0.00997450880725535,
+            -0.02059560682587115,
+            1.5939836949198004e-05,
+            -0.007159493752862143,
+            -0.0004782363814453761,
+            -0.017503181957042913,
+            0.016643971600145645,
+            0.024056442566869802,
+            -0.013209211443909658,
+            -0.00445521990927108,
+            0.0020313626821877526,
+            -0.029235692455820056,
+            0.006662209097255912,
+            0.016993027053110635,
+            0.00529940612845993,
+            0.0026104743851393716,
+            0.0001707934714983891,
+            -0.031289446611163674,
+            -0.005782208620284939,
+            0.03053516889069683,
+            0.05590704747438521,
+            0.021628254147289437,
+            0.02403314981042378,
+            0.02908606663381421,
+            0.02717206823892633,
+            -0.003198844932040613,
+            0.0016217508103442348,
+            -0.01620660606431983,
+            0.027098323143010743,
+            -0.011664168290676943,
+            0.04322175021589083,
+            0.008652323184522541,
+            -0.049493020736445056,
+            0.004537958157696794,
+            0.00943239454147692,
+            0.034918603888916544,
+            0.016678716235690318,
+            0.0005887414054803805,
+            -0.005894338664998101,
+            0.03279165663336323,
+            -0.02735726709032382,
+            -0.011962070182991011,
+            -0.00036014407560325284,
+            -0.019079003730607717,
+            -0.015938295169860884,
+            0.010357611943935459,
+            0.04278924678845588,
+            0.0009957483598272451,
+            -0.0071422790468850215,
+            0.01571201177743999,
+            -0.009445307674383,
+            0.02286338492075306,
+            0.007336440384050498,
+            0.0042809497342459904,
+            0.025581467690095977,
+            -0.0025007871391868105,
+            0.02300539823997411,
+            -0.000628898771317254,
+            0.004582863909134464,
+            -0.015960578501883316,
+            -0.00930649396114688,
+            0.027838455754803444,
+            -0.011112268986721,
+            -0.029253746504396393,
+            -0.008734785561831333,
+            0.00472579547270089,
+            -0.0003805825712986138,
+            0.0021854358149324227,
+            -0.01183885533661493,
+            -0.010840951201591629,
+            0.019965691788904057,
+            -0.004922943301424112,
+            0.003800048754664761,
+            0.012664049429916252,
+            -0.0030124219674414897,
+            -0.01910844996750822,
+            0.01889983984736845,
+            -0.003858283778519141,
+            -0.024168821338992813,
+            0.020083955711766945,
+            0.01850497889545421,
+            0.045156567155119454,
+            -0.01586824055785609,
+            0.004612757419198795,
+            0.017497774574602505,
+            -0.002172088240279357,
+            -0.019617327081566827,
+            -0.014944795772093997,
+            -0.018212462639041492,
+            -0.015975283672064702,
+            0.012837956594691596,
+            -0.01243789877725244,
+            0.019500571353639314,
+            -0.0380735226388175,
+            0.012590900534826689,
+            0.004832168601136412,
+            0.007505385730121269,
+            0.016011983211212992,
+            0.025994560026860358,
+            0.002157436199771672,
+            -0.033465476168065784,
+            -0.01014599891204166,
+            0.007845373376916211,
+            0.010093432683262838,
+            -0.002849185178237036,
+            -0.018147004183434554,
+            0.003633790611809425,
+            -0.0266147420637404,
+            -0.018644422659309035,
+            0.022926737299761104,
+            -0.04798772015599113,
+            0.017908415301694906,
+            0.024859125158586466,
+            -0.04343699525707025,
+            -0.04886699607556352,
+            -0.011459188432595229,
+            -0.036774381316312464,
+            0.0014128005215813578,
+            -0.0005256067584181517,
+            0.011194992852264102,
+            -0.009342140079829378,
+            -0.018598671981710183,
+            0.01584789414301952,
+            -0.0036412641242200418,
+            -0.004225840239935549,
+            -0.00616696736018045,
+            -0.0025598631463178384,
+            -0.010188803157718543,
+            -0.03244694662411977,
+            -0.03290891136780909,
+            -0.00033742479296272127,
+            -0.009743927051414455,
+            -0.02334288359945272,
+            -0.03447062782872289,
+            -0.008728968107967028,
+            -0.037753659749578956,
+            -0.00515561961442376,
+            0.04070890150714597,
+            0.027517147608335957,
+            0.009449414978133334,
+            0.008086900089929912,
+            -0.013989034604490105,
+            0.05691047488419298,
+            -0.020425883381389086,
+            0.039466620642512155,
+            0.005301493569940605,
+            -0.0037222353899918703,
+            0.013075425214878277,
+            0.027710826422351564,
+            -0.006253665106995653,
+            -0.0017412928073238833,
+            -0.01928096630475438,
+            -0.012920476859996416,
+            0.009444408011467955,
+            0.015799884767184885,
+            -0.0025083061619197524,
+            0.001274457251739847,
+            0.00930533161700852,
+            -0.0006652478540264343,
+            0.017258038491335247,
+            0.03996268817486439,
+            0.003833844612473671,
+            -0.00826404141706295,
+            -0.018822002570963614,
+            -0.003188844900317347,
+            -0.011807006674248829,
+            0.010482058734260632,
+            -0.015410635475145028,
+            -0.011540803927522013,
+            -0.01249428777269578,
+            -0.017081856786754605,
+            -0.002183034976477535,
+            0.003114834030705039,
+            0.0027475411722881747,
+            0.02261627635192778,
+            -0.015872027027080824,
+            -0.022487581027490516,
+            -0.0035823117885461506,
+            -0.02398778449664045,
+            8.13716819067491e-05,
+            -0.01050060782407259,
+            -0.015616181283381218,
+            0.013676480044963827,
+            -0.021513173575512936,
+            -0.01091399089076612,
+            -0.008699176177861256,
+            -0.0006986953850759474,
+            0.005853723710671249,
+            -0.00993355748621621,
+            -0.00010241843582159751,
+            0.01150872694850241,
+            0.005559552266827916,
+            -0.012336449703762907,
+            0.0039431718621504,
+            -0.012740550272925562,
+            -0.004432059174951158,
+            -0.03592309122711106,
+            -0.019665686718735042,
+            -0.01846795637714379,
+            0.005642094448288269,
+            -0.005108964028543505,
+            -0.020586678054151854,
+            0.00668364061048941,
+            0.0034775127050453774,
+            0.004713169526661792,
+            -0.010723667875209928,
+            0.013944735241748791,
+            0.010882420168586074,
+            -0.003518245916227596,
+            0.013827334494230662,
+            -0.001330399936559009,
+            -0.020286697504699764,
+            -0.011139922402071095,
+            0.03538258948179278,
+            -0.024961826464903997,
+            -0.0018019328351763667,
+            -0.0026423695690168066,
+            0.00597003516870726,
+            -0.03071835946501977,
+            -0.013634212232744325,
+            0.02164400349828422,
+            -0.005846552894831441,
+            -0.026367014426434226,
+            0.03399116480234033,
+            0.03473258231629753,
+            0.010041605592929843,
+            -0.013423384340589794,
+            0.04730054254696893,
+            -0.012968978954938308,
+            -0.018274800841400665,
+            0.026826644936918647,
+            0.0278389611094493,
+            0.02525164095926168,
+            -0.029782550447890466,
+            -0.003010713086487275,
+            -0.02393629084357885,
+            0.017993366287762866,
+            -0.0263476663180971,
+            0.01864011499179424,
+            -0.02332263152537849,
+            0.016014908881756034,
+            0.01868464825773604,
+            -0.003104041044182249,
+            0.026313491086684425,
+            -0.0052945248919367815,
+            0.015253731232325074,
+            0.0012371308759955573,
+            -0.02119765517013431,
+            0.007795466385020642,
+            -0.006469500272298062,
+            0.025762169721880456,
+            0.00833992768202466,
+            0.01801901639978976,
+            0.022019940825351116,
+            -0.0409522298213741,
+            0.02003698451081499,
+            -0.001367916864679543,
+            0.01782984895971748,
+            0.0038516134262842373,
+            -0.007885432294782304,
+            0.014221728513400065,
+            0.010991352188016265,
+            -0.012851099261924757,
+            0.003840539873231741,
+            0.0008923482291826014,
+            -0.028928104542716937,
+            0.007066004768771198,
+            -0.026914574065060926,
+            0.025866907400807087,
+            -0.005526774784152086,
+            -0.00998122210956923,
+            -0.020744532673097317,
+            -0.0071425207262770865,
+            0.003754468896273156,
+            -0.012747874997429746,
+            -0.011385805601999295,
+            -0.024700457669046855,
+            -0.02401291382882952,
+            0.029591582802488962,
+            -0.014054332096344003,
+            -0.04015978298687865,
+            0.001234440863385445,
+            -0.023301255402129767,
+            0.01193125007773239,
+            -0.005793124129267758,
+            -0.024440100649464773,
+            -0.042563399082257154,
+            -0.017548888269389457,
+            -0.0009394887961485922,
+            0.022739763644266633,
+            -0.00114156949394707,
+            0.041791395064755515,
+            0.0008816696205437911,
+            0.022207132857533028,
+            -0.015547580543606755,
+            0.005319731264840819,
+            -0.02071229859314312,
+            -0.041495579742912285,
+            0.03660432378985971,
+            -0.014637031447181937,
+            0.025854844497435178,
+            0.03324325296704186,
+            -0.013009347371118843,
+            0.001354504885302625,
+            0.02652570709474851,
+            -0.014618100923440427,
+            -0.010782232747863238,
+            -0.0034936699515378212,
+            0.003461670978569965,
+            -0.006468116285021298,
+            -0.027793221545159466,
+            -0.007752458229193391,
+            0.04193706715422074,
+            0.017894069367616637,
+            0.021928994684423043,
+            0.022606401934291136,
+            -0.017972404868085358,
+            -0.026402048654300563,
+            0.003656646576244765,
+            -0.013129722990153284,
+            -0.018090360146368677,
+            -0.01624598856366761,
+            0.012397150789705746,
+            0.011741002980073786,
+            -0.004052422390675261,
+            -0.01836750423615859,
+            -0.020929335029248472,
+            -0.016327412417122416,
+            -0.0042615345226698015,
+            -0.008458117095535,
+            -0.005002582649405199,
+            -0.012158173174049797,
+            -0.020409616620022143,
+            0.03706368540614453,
+            0.0285565642978937,
+            -0.0041732933884846566,
+            -0.00607154864704767,
+            -0.011142150176334957,
+            0.008834431806990914,
+            0.006021232046087652,
+            -0.005028870471297621,
+            -2.3396043292139426e-05,
+            0.005112033372713154,
+            0.022822312435625314,
+            0.007271235299348001,
+            0.0035791789459533856,
+            -0.06340305811365254,
+            -0.005392000517722054,
+            0.022560201096078862,
+            0.012581401789917604,
+            -0.0032248219865223496,
+            0.025180718735801837,
+            0.0006451724964970927,
+            -0.009714514747463593,
+            -0.018389422219956937,
+            0.024728862150150004,
+            0.008760329201960964,
+            0.020988700992999768,
+            -0.014974763313686516,
+            0.003845297486923296,
+            0.0006445983847138896,
+            0.00750394530065325,
+            -0.0019792992847026793,
+            -0.008578254004439607,
+            0.025776677078006516,
+            0.016593438298233645,
+            -0.005695578040595484,
+            0.01604018433477423,
+            -0.0017497260488899657,
+            -0.021460543443428924,
+            -0.010265213095609554,
+            0.00042017209251926397,
+            -0.010570864288876387,
+            -0.018015790888295944,
+            0.017569414447505878,
+            -0.002601036282535863,
+            0.0072399377790164435,
+            0.030281707151249355,
+            0.0423183067569767,
+            0.009566493398534713,
+            -0.005928603293907002,
+            -0.0037936911251953654,
+            0.015477873304924939,
+            0.01513127995244932,
+            0.010741256034652754,
+            0.03015858543221727,
+            -0.029247492875021843,
+            -0.006690306960497428,
+            0.0046653727410352576,
+            0.004610405633429778,
+            -0.011692179099382171,
+            0.030055897497741527,
+            0.04569730592994177,
+            0.006736813244286893,
+            -0.004799978513117371,
+            0.002121067028052434,
+            0.011299889107780231,
+            0.007768137218140627,
+            -0.018927080714961783,
+            -0.0012222962555201746,
+            -0.006009911026251445,
+            -0.014001766533604134,
+            0.026941585318261085,
+            -0.003960837039351648,
+            -0.011848232223811548,
+            0.02903357274500332,
+            0.1105267435019781,
+            -0.006011029338781496,
+            -0.0053646998281195206,
+            -0.03242321704621517,
+            -0.015194102811332289,
+            -0.008860590919147765,
+            0.0056988205877963076,
+            0.01635957585050017,
+            0.020327661913559888,
+            -0.0008943408274661007,
+            0.016820860249955754,
+            0.02822175614376368,
+            -0.013854827839129885,
+            0.007906244895327178,
+            -0.030891381690403466,
+            0.015702117924362928,
+            -0.005383974926093977,
+            -0.012032955400313954,
+            -0.0026169511828034054,
+            -0.008618307035330928,
+            -0.0037356680273096105,
+            0.0009666430093455632,
+            0.011296307527684261,
+            0.004427516814276546,
+            -0.014484198313369681,
+            0.002829529156309784,
+            -0.008595300985863424,
+            -0.005122996529848575,
+            -0.02393571073523279,
+            -0.024119245543788293,
+            0.005223147047205755,
+            0.0019389305548629142,
+            6.344308292376579e-05,
+            0.006848965054539454,
+            -0.003810712095343454,
+            0.030149081317333418,
+            -0.01700322008272963,
+            0.0012288912810468446,
+            0.01193226302726573,
+            0.012332749933413344,
+            0.01688602002694633,
+            0.0028600984823761366,
+            -1.9882692552530084e-06,
+            0.021172977047420227,
+            -0.010514036509402561,
+            0.01460334896316005,
+            0.013091417303145243,
+            -0.04103572381458182,
+            0.0089746401463863,
+            -0.0027841289484449873,
+            -0.024252447218794344,
+            0.004446472209246425,
+            -0.018987867343500102,
+            -0.00881974688349601,
+            0.0005365625696166774,
+            0.012054719659248512,
+            0.01960300421041832,
+            -0.008553943699846696,
+            0.02111201386480852,
+            -0.0003501657740808554,
+            0.004177088753111459,
+            -0.012879944670340026,
+            0.0065222126975051685,
+            -0.007563354540402343,
+            0.0008168128607271199,
+            -0.018213241937597355,
+            0.012526561612175514,
+            -0.0008285725179894544,
+            -0.015737003159436623,
+            -0.011337688275309921,
+            0.004737346137976519,
+            -0.0027993545140114705,
+            0.0030843163114700313,
+            0.010153740340626154,
+            -0.008815302484351509,
+            0.004919275910399053,
+            -0.0017541042538165297,
+            0.002562816687714991,
+            0.014285249186421434,
+            -0.029816964593980453,
+            0.005357227738099325,
+            0.009154842495811638,
+            -0.00048782303639333943,
+            -0.013074976152495576,
+            -0.004054505559072866,
+            0.0028322102449485015,
+            -0.016877328079971877,
+            -0.012128199639779175,
+            0.023421330164419888,
+            -0.012565145848820333,
+            0.008609070188894315,
+            -0.010936598722317274,
+            -0.018401801734378692,
+            -0.01066119413133089,
+            -0.005254229780560808,
+            0.01234969508270852,
+            -0.0038427767204741877,
+            -0.006947594635952202,
+            -0.01756894543880097,
+            -0.007890897373987378,
+            -0.047246698568222466,
+            0.009617876521097624,
+            0.027157555263862723,
+            -0.011355758309667893,
+            -0.04050311381480088,
+            -0.01017774338301804,
+            0.02437257208967103,
+            -0.004056329295095018,
+            -0.008345450168215274,
+            -0.001599171409135998,
+            0.018358386463243014,
+            0.0027207445181251406,
+            -0.007292967749349572,
+            -0.011267861428599221,
+            0.01657507035043879,
+            0.015105061280816214,
+            0.021419854579631276,
+            -0.02088952706825817,
+            0.006295984081511286,
+            0.020973020776936996,
+            -0.03073246543555915,
+            0.0023720435876790876,
+            0.03363966216709148,
+            0.010412588613998134,
+            0.013023311559810034,
+            -0.0010217662933572685,
+            -0.0075567381078977215,
+            -0.008248818794295818,
+            0.011848200323549865,
+            0.005066463524567732,
+            -0.023174489297821385,
+            0.021823825355858556,
+            0.020368868756448753,
+            0.0026042856858509524,
+            0.003457067317891926,
+            0.0005539745115410357,
+            0.01062670844108589,
+            0.009948889803890108,
+            0.05100741556350918,
+            -0.006493714124814164,
+            -0.014021751055930108,
+            0.015063988102064609,
+            0.02488059237697814,
+            -0.0023499941119596228,
+            0.01087282522970853,
+            -0.02612218935931224,
+            -0.021423880914461062,
+            0.023389294741183817,
+            0.008967763132667892,
+            0.009637806238448432,
+            -0.010815706635624337,
+            -0.011390999462441815,
+            0.031750675158698435,
+            -0.01307829462211914,
+            0.00759749911213318,
+            -0.02169895957209634,
+            0.01387879409130081,
+            -0.008606362268863441,
+            0.032229658009209815,
+            -0.0009066257436946125,
+            0.011062436424260079,
+            0.028010486866468023,
+            -0.007835805879241447,
+            -0.019657486763625547,
+            0.003967480939987674,
+            -0.015577952617624695,
+            0.02188429826317828,
+            -0.020502758155233298,
+            -0.0002362500702658591,
+            0.0040935406015014265,
+            -0.010056462958039452,
+            -0.0057038083589231615,
+            0.011475170331848666,
+            0.0015345076181482267,
+            0.006359262977587827,
+            -0.020293014401106,
+            0.01369706440475074,
+            -0.003989482215141903,
+            0.007887824197823247,
+            -0.015925992045349828,
+            0.014370223626942766,
+            -0.02309139937510304,
+            -0.021808736033141787,
+            0.008142621179624918,
+            0.0108996933426044,
+            -0.007589050511458233,
+            -0.025971539510030145,
+            0.02250992080022228,
+            -0.024980833805349545,
+            0.024979008145354243,
+            -0.006622339940663538,
+            0.002566378488088469,
+            0.004913874740363343,
+            -0.008102103508732738,
+            -0.0028098905427027783,
+            0.0057865587191235565,
+            -0.0073509382400023995,
+            -0.014123756810537343,
+            -0.019453380788786282,
+            -0.04083629135586275,
+            -0.03177927047231389,
+            0.022927581786088044,
+            0.011557232599801421,
+            0.0017073832658476193,
+            0.02339552660545296,
+            -0.02247471449131386,
+            -0.017890866287500053,
+            -0.009775068417716798,
+            0.025184899522229628,
+            -0.04408541669037361,
+            0.012104255392615549,
+            -0.014669458760022245,
+            -0.0245258242481677,
+            0.006274252098230249,
+            -0.004161960147481513,
+            0.009299171724026455,
+            -0.010779421730862708,
+            -0.04713627671791054,
+            -0.005192684784377356,
+            0.02330975631183273,
+            -0.0036802876364168935,
+            -0.006197548344287773,
+            0.03048284650641829,
+            -0.0019503797069087845,
+            -0.005038254331399784,
+            -0.005524875459245342,
+            0.027783596017224226,
+            -0.024950083604372984,
+            0.010988398531346805,
+            0.008304937912856357,
+            -0.008485065895193593,
+            0.006572665697403556,
+            0.0011518846966102158,
+            0.011927348472282199,
+            0.015632588644277183,
+            -0.015559413716826264,
+            0.02262990861928957,
+            0.011446850286896198,
+            0.006506075777765303,
+            0.008015225053785824,
+            0.00010054430757843335,
+            0.0022975742216194894,
+            -0.024948925449668525,
+            0.026047924967851216,
+            0.01888211557921096,
+            0.0001632181017876392,
+            0.012553094476616919,
+            -0.015042621225242777,
+            -0.02782009986290425,
+            0.02606646297992026,
+            -0.016847653820092335,
+            -0.00045272379465939274,
+            0.020053091486344633,
+            -0.016210036358455773,
+            -0.007930550661303182,
+            -0.01123514849773852,
+            -0.0162675150307978,
+            0.01803076559921804,
+            -0.012160217543572069,
+            -0.02216594536057653,
+            0.002594675636753233,
+            0.016443123828746582,
+            0.02134480656850119,
+            0.026694575953313482,
+            -0.011470930697351533,
+            0.017768927048191052,
+            0.0008653408391607816,
+            0.001865765599795092,
+            0.025745954171649543,
+            0.01702677011968492,
+            -0.008474889109168202,
+            0.025347176686722313,
+            -0.0016686223698600755,
+            -0.002421921060670558,
+            -0.012265504084322075,
+            -0.0165214436458303,
+            -0.007545471138114208,
+            -0.015333235556146196,
+            -0.0051830106590765685,
+            0.000762746782664525,
+            -0.0018118022696726346,
+            0.10876576838038368,
+            0.0034948044540582626,
+            -0.011608460689239234,
+            -0.005870263500417206,
+            0.007510774308250375,
+            0.0023525201199670765,
+            -0.006846154764389575,
+            0.016082721103476375,
+            -0.018616626204086575,
+            -0.023593744079394,
+            0.004743481974389998,
+            -0.0009262286319469741,
+            0.00870122878944883,
+            0.02029523251993269,
+            -0.008608837818169835,
+            -0.0200567711498164,
+            0.007089477722191928,
+            0.005535149644041221,
+            0.0023640746516071774,
+            0.005256431824732927,
+            -0.0006118671321824933,
+            0.008209920240361308,
+            0.02448422122961942,
+            -0.015622892878860465,
+            -0.029770836607612903,
+            0.010535371601298532,
+            -0.00015994156470228762,
+            -0.007193425789067776,
+            -0.011046316463797128,
+            0.014728841492009627,
+            -0.0011488451990980268,
+            -0.02751404133172324,
+            0.017663132712536528,
+            -0.019055335178900415,
+            0.014643246200933506,
+            0.0025348345163347464,
+            -0.00016387173410118788,
+            0.022565953069005714,
+            0.013101158251834834,
+            -0.04272085076882881,
+            -0.0034442020309914823,
+            0.01041205537746844,
+            0.011059422927946552,
+            0.0116446016862942,
+            0.0012992535381387887,
+            -0.018670801241345514,
+            -0.011788984490283379,
+            -0.009109849103208542,
+            -0.021268488414296117,
+            -0.012523033785938503,
+            -0.011393853408628214,
+            0.010743646386439633,
+            0.0012964030814325153,
+            0.023809566099779113,
+            -0.003030535542277875,
+            -0.006526351867141901,
+            -0.000430699447948508,
+            0.00880561288237257,
+            0.009694646009259077,
+            -0.02079011138439714,
+            0.008294338719512953,
+            -0.01823922789220214,
+            0.011819769685519291,
+            0.006416044253693268,
+            -0.022799568832306067,
+            -0.009911619276012527,
+            0.007152786695893951,
+            0.008393677195358034,
+            -0.013569953300860946,
+            -0.006503988731872938,
+            0.02299498754558268,
+            0.018870619152710703,
+            0.012233964644738905,
+            0.024994616203196145,
+            0.03838969722235787,
+            0.005744255912479976,
+            0.027682123067526515,
+            -0.016791471092091604,
+            0.008459745383680297,
+            -0.022493868174622695,
+            0.012902262638928338,
+            -0.00460399519266689,
+            -0.0050431343566427886,
+            -0.018188192498411616,
+            0.033765105245412236,
+            -0.009159751849801773,
+            -0.0027150644587327763,
+            0.034778224828120714,
+            -0.0066502021793471075,
+            0.004525230911930458,
+            0.005096748531775903,
+            -0.016067049494073893,
+            0.033274672789154185,
+            0.01686831040872425,
+            0.016353966592947262,
+            -0.021631941883929817,
+            0.011748317746828848,
+            -0.005675701689448899,
+            0.0001273442591116645,
+            -0.020738819037390263,
+            -0.0037528323797385733,
+            0.0027725541343763104,
+            0.0059860509282538546,
+            -0.016556134518370437,
+            0.02033623476757896,
+            0.00019647023567641008,
+            0.0052895744489259875,
+            0.03253462247073863,
+            0.01043311189804924,
+            0.019182938878855655,
+            0.010051051068017807,
+            0.019523739618278902,
+            0.0281939391326836,
+            0.025685787482623124,
+            0.0029932688462254147,
+            -0.018668945726139946,
+            -0.0009326901784812825,
+            -0.007880352863625262,
+            0.010492063055849727,
+            0.022849902214818293,
+            -0.022619228461640748,
+            -0.02772644874135741,
+            0.018566981184526675,
+            -0.0022608185345048414,
+            0.020031103834122595,
+            0.007544464625538984,
+            -0.020935181884939796,
+            0.003407355515103286,
+            -0.023178230166337178,
+            -0.0005418155871620593,
+            -0.031039187720144967,
+            -0.028591174077243604,
+            -0.0007659241904525392,
+            0.011945344534674635,
+            -0.01688153017778311,
+            0.03357788163306365,
+            0.017667778671101954,
+            -0.014289109343082812,
+            0.01604684528817524,
+            -0.011054587325539086,
+            -0.0036959091138078683,
+            -0.005819853870004682,
+            -0.00203755010992417,
+            -0.02157944474979592,
+            0.005138614763345542,
+            -0.0033509138150859593,
+            -0.0024871060745482883,
+            -0.016936575573598154,
+            -0.021901830366872595,
+            0.016375173843894617,
+            0.016742770709641697,
+            -0.006606252542838758,
+            0.039771905404651496,
+            -0.004238071340064403,
+            0.00018364315349545657,
+            0.011690521707403059,
+            0.0010018392078563795,
+            -0.013651921068215563,
+            -0.010829994512513501,
+            0.012362647379788833,
+            0.011321403447882543,
+            -0.018942043512252663,
+            0.005763773608775982,
+            -0.009131565038297889,
+            0.002089216671850269,
+            0.005438887787329238,
+            -0.008183835275629478,
+            0.006690280329947951,
+            -0.01673229763262922,
+            0.015833397876364723,
+            0.00295474085300386,
+            0.003186554859191664,
+            -0.011797859525926772,
+            -0.020991927615680524,
+            0.01576413532776261,
+            0.0032403514518527287,
+            0.018819824905201635,
+            -0.026755218746060662,
+            -0.00012946467986739473,
+            -0.009552681500884924,
+            -0.0028941357661144826,
+            -0.012533800273548772,
+            0.013975640628890588,
+            0.016864739319953755,
+            0.0063941926578529,
+            0.01781016436903586,
+            -0.0026268224264609724,
+            0.015368284773773678,
+            -0.018122416427266218,
+            0.008686009180697955,
+            -0.006801074032721299,
+            -0.018537521789181048,
+            0.0013237338264105415,
+            0.027863741654435434,
+            -0.004670012662973029,
+            0.011066853115455966,
+            0.011704884649534624,
+            0.018954056680143452,
+            -0.0012626215734593707,
+            -0.0030850559460101866,
+            0.015610404665171367,
+            -0.005838226405658572,
+            -0.006861883833171456,
+            0.020872966233916948,
+            0.032951653763679145,
+            -0.006716215648910268,
+            -0.0021292599951407863,
+            0.010886596327481677,
+            -0.004901684487088035,
+            -0.008644584974417106,
+            -0.009587037829603014,
+            -0.03158137900717491,
+            0.024369089446706894,
+            0.012092412439477474,
+            -0.009601042951586109,
+            0.022690862146548786,
+            0.020752792552278052,
+            0.011412743968296708,
+            -0.010471860708195726,
+            0.02291536038589634,
+            -0.01268035142112569,
+            0.024168869865226136,
+            0.032482831775271157,
+            0.004400973556358811,
+            0.0022312906286559167,
+            0.0021175489661843985,
+            0.01691348460546639,
+            0.006050357346231155,
+            -0.007068133214675996,
+            0.011656003476409986,
+            -0.018487923180009552,
+            0.01598530891381001,
+            -0.004432366596808033,
+            0.02808371007118948,
+            -0.01337145216499331,
+            -0.010473700033368562,
+            -0.00537325376270318,
+            0.00010726431372328729,
+            0.006445701087575359,
+            -0.003324002570256215,
+            0.005255443653813165,
+            0.0007072577303252132,
+            -0.017825649146944237,
+            0.003050128130393781,
+            0.004825967170196209,
+            0.009018799597337101,
+            0.004103653497834134,
+            0.007249957401824204,
+            -0.012360355929483,
+            0.01441885664789707,
+            0.004401820363035808,
+            -0.0026190938717927086,
+            -0.0007163225760398784,
+            0.011124538295057815,
+            -0.013232822471919533,
+            -0.004895624967029031,
+            0.005281228035321008,
+            0.0059275753353475414,
+            -0.007507924998741734,
+            -0.004484140989405051,
+            0.011908076549392969,
+            -0.0037917291321648483,
+            0.006351543504726163,
+            -0.004409464890606014,
+            -0.0037760571637986827,
+            0.005303661313434895,
+            -0.00974511544836468,
+            -0.004204167898342997,
+            -0.0007093125445559502,
+            -0.006869250680233214,
+            0.00786202375969793,
+            0.005971033227117194,
+            0.010606085272785863,
+            0.006969437776542303,
+            0.004293129058563288,
+            0.004114452739123398,
+            0.004071017374828283,
+            -0.002445681623750424,
+            -0.00592480769363354,
+            -0.004617501328052899,
+            -0.012067458701076778,
+            -0.008335239153829813,
+            0.009025028998949398,
+            0.008653779421856954,
+            -0.019363018792788574,
+            0.02048011578221327,
+            -0.001188320960771659,
+            0.00772165044232104,
+            -0.01602086976290928,
+            -0.000907553574811279,
+            0.005464880629734941,
+            0.008267702703609528,
+            0.016136762227127184,
+            -0.03654209526492064,
+            0.005179791231187316,
+            -0.0037321883493524766,
+            0.016026013709955444,
+            -0.008215703411373182,
+            -0.006979816286354862,
+            0.02564060526656963,
+            -0.01450763388985563,
+            -0.0009847758228920171,
+            -0.016142304341196283,
+            -0.01407778890808498,
+            0.010815725252959506,
+            0.02050940280233426,
+            0.020684808835848383,
+            -0.0004965802897137407,
+            -0.0005781445948759235,
+            0.00036179841583004193,
+            0.022775804864342608,
+            0.015545681427566235,
+            0.018697154644607316,
+            -0.004543500782944068,
+            0.007028436664826706,
+            -0.023057215501463973,
+            0.009912581556496658,
+            0.0011795417129232596,
+            -0.008215027850908477,
+            -0.008554137076267468,
+            -0.008703378564520125,
+            -0.010448538943182258,
+            -0.011773811327190638,
+            0.004106616839960258,
+            0.013895013678116032,
+            0.004190549543250594,
+            0.0031571190600981074,
+            -0.0225385788570314,
+            -0.006243193914542516,
+            -0.011229641377928767,
+            -0.01674214043453308,
+            -0.007367389908398082,
+            -0.02386334969150212,
+            -0.028021339684039734,
+            0.0002794207370355175,
+            -0.02411861258809571,
+            -0.015958336406924612,
+            0.005268815864773021,
+            -0.01546610225139129,
+            0.00898600137695074,
+            -0.008396481690803911,
+            -0.011873486990732417,
+            0.0074743033679134105,
+            0.015336444788692,
+            -0.017676977079590463,
+            -0.010165317702640217,
+            -0.01690524990506033,
+            -0.013460451917691695,
+            0.010918550283871637,
+            0.00595960977280955,
+            0.009250157926438213,
+            -0.012447979337850305,
+            0.0062932435120863365,
+            -0.012673905639312696,
+            0.015073013552721701,
+            -0.03769166067890086,
+            -0.029992249463405965,
+            -0.00014990754540999537,
+            0.007288036630677436,
+            -0.0020735742042505426,
+            -0.018695120921334806,
+            0.0003860116505754868,
+            -0.003608995261360232,
+            -0.014900191335144973,
+            -0.013866216738266176,
+            0.025251475305524053,
+            -0.006788483107616133,
+            0.01223227424986397,
+            0.0034937277040015626,
+            0.03438333560343413,
+            0.008803852645562129,
+            0.014666206816302367,
+            -0.005709421955160623,
+            0.003742433643393657,
+            -0.0033706258426092744,
+            0.006906464031352315,
+            0.0076578888497945185,
+            0.009276592191710524,
+            0.008025321004305927,
+            0.004576470294889512,
+            0.002887346543823031,
+            -0.001925468214331082,
+            0.02125241545487307,
+            0.025671028960818562,
+            -0.00037467906648160044,
+            0.008981063192139038,
+            0.0009387900084090577,
+            0.0002735789978680719,
+            -0.008374290340790827,
+            0.006547013284767913,
+            -0.008274637814401893,
+            0.004041066530254948,
+            0.012897549078838394,
+            -0.009150863940242148,
+            -0.010211568689323908,
+            0.016575088843292566,
+            -0.012380099999574783,
+            -0.03420455578618604,
+            0.026550654680719987,
+            0.005324512466403649,
+            -0.0035670106106754245,
+            -0.010657928468872608,
+            0.0037176027853158167,
+            -0.01103891334887315,
+            0.008096915671982816,
+            0.022587659771132487,
+            -0.0013527917944322358,
+            -0.0018085171207746743,
+            0.0038503794877980586,
+            -0.011870484733648887,
+            -0.007881489342351292,
+            0.006462772034381968,
+            0.0315696514747912,
+            0.01326417653779577,
+            -0.020400369539140532,
+            -0.015223808971473513,
+            0.020406862349006738,
+            -0.023142176712111575,
+            -0.006254959806032159,
+            0.009039506093607052,
+            0.0009087520492922078,
+            -0.022766046838415165,
+            0.0009662037093312435,
+            0.004701966191595815,
+            0.005946458676746661,
+            -0.006358986495019584,
+            0.03237532341275415,
+            0.001075992257265679,
+            0.015892020271402903,
+            -0.017189453434945788,
+            0.0017514632185789299,
+            -0.014189278671121677,
+            -0.014265447388465805,
+            -0.029809460132405256,
+            0.0005557491671071693,
+            -0.0030324198417091704,
+            0.011038984948790744,
+            0.024808067315720335,
+            0.01145103533194245,
+            0.014432246996007246,
+            0.010552852947578788,
+            0.01513689173494211,
+            -0.013653722180565311,
+            -0.029009757850432732,
+            -0.005171522291984226,
+            -0.018001048743500243,
+            -0.02297482628212251,
+            0.003962613844063587,
+            0.013645125285548414,
+            0.006278662690346082,
+            0.023786381386575035,
+            0.0021738516691483163,
+            0.0009580591750940011,
+            -2.690805459518147e-05,
+            0.008894091070322921,
+            -0.004952203441302471,
+            -0.006553986197076102,
+            0.003570189040872711,
+            -0.000652714137223501,
+            0.021228360881461853,
+            -0.005758946428212041,
+            7.897153237591187e-05,
+            0.014242175492137011,
+            -0.005622256348934008,
+            -0.0035009192529791146,
+            -0.017339926143652377,
+            0.006248842389703505,
+            -0.01622011586976866,
+            0.025783086900871728,
+            0.018450783573486916,
+            -0.009109295293546267,
+            -0.013469521573468915,
+            -0.011259555201313116,
+            0.02773097452736661,
+            0.004631282554715908,
+            0.010787055220617197,
+            0.018228322680582464,
+            0.01461254259249465,
+            0.01784830193964943,
+            0.004664725683522323,
+            -0.0034339781146649384,
+            -0.0007692550982545399,
+            0.021176646860433974,
+            -0.014700178590249292,
+            -0.002701208084390198,
+            -0.008842788249773623,
+            0.0021799005046816444,
+            -0.008431299000066363,
+            0.006086447848362669,
+            -0.010793016137040172,
+            -0.015251381840165954,
+            0.0189503457652375,
+            -0.02660767353735015,
+            0.016423753600066336,
+            -0.00593016170680532,
+            -0.00482891500342298,
+            -0.027459214027897378,
+            0.010728039528784636,
+            0.0027866316191608297,
+            0.0022519974195000687,
+            0.010770472550461752,
+            -0.021427634123401408,
+            -0.01867717541189363,
+            0.008532444554010673,
+            -0.011457117350255678,
+            -0.007152301670244725,
+            -0.00348619742388509,
+            0.0154715607296319,
+            -0.00017898955187849443,
+            -0.009925365923862246,
+            0.02085093731060963,
+            -0.007175309371250024,
+            0.01223979370120031,
+            0.005256127464980655,
+            -0.008756077893734508,
+            -0.0028438320920811362,
+            0.021022708898898203,
+            0.027557382862284808,
+            -0.008342206702778956,
+            0.014526066580760486,
+            -0.011883797998224506,
+            -0.01088335352319397,
+            -0.03479444009382274,
+            0.030104838505937447,
+            -0.013305732257226255,
+            0.026773747582411633,
+            -0.0006968265979630097,
+            -0.017037729111584094,
+            -0.016206207453228583,
+            0.01625225876515654,
+            -0.0024290584023212254,
+            -0.009366287391027166,
+            0.01465885301870365,
+            -0.02428628495473643,
+            0.0058336571592899735,
+            -0.0006627935593670669,
+            -0.011531762838191602,
+            0.013275508677482622,
+            0.0019223252930710324,
+            0.010738622785461513,
+            -0.02030405805119001,
+            0.0009661595529028194,
+            0.009263716373456188,
+            -0.016791855691574455,
+            0.006057830565306032,
+            -0.03730952786847912,
+            0.0029341721588188943,
+            -0.014833895684348656,
+            0.005725615593914644,
+            0.001498755253963699,
+            -0.01820708352937343,
+            -0.03652692232407375,
+            0.00043665701245599497,
+            0.01913883585802903,
+            -0.007975988799790905,
+            0.020538595326175857,
+            0.008243569241768276,
+            0.008421404577931517,
+            0.002314738067443065,
+            0.0016702403734608475,
+            0.008279538988821465,
+            0.03679456861213399,
+            -0.01775927634148114,
+            -0.004836606118958108,
+            -0.0012151329859941935,
+            0.010644195708319258,
+            -0.02028470571116305,
+            -0.009287515726906219,
+            -0.0036506912311261536,
+            -0.013134467570201307,
+            0.030615862647585222,
+            -0.020265555441523482,
+            0.025634449009756734,
+            -0.009300770899023007,
+            0.009274072905190077,
+            -0.02097685009125915,
+            -0.009797008690754956,
+            -0.010661540100131183,
+            0.0019532291898233895,
+            0.013465005282849381,
+            -0.009151430285942593,
+            0.0071467934925125145,
+            -0.006655608635025676,
+            0.01020327561163374,
+            -0.001472597695707201,
+            -0.02443508800166349,
+            -0.0024998811551575563,
+            -0.01107321258077183,
+            0.02456164421154801,
+            -0.0019619696941303388,
+            0.0005043179699158694,
+            0.010951580163871948,
+            -0.00853593956432799,
+            0.023927336608555987,
+            -0.008855702280766362,
+            0.0010093679746665581,
+            -0.004863269000020324,
+            -0.004005691415117686,
+            0.001698212044518236,
+            0.01822024022650613,
+            -0.005707404715849356,
+            0.03722587302228139,
+            -0.01708925706528118,
+            -0.0065500772743641925,
+            -0.00015538686795361824,
+            0.010852925330001283,
+            0.010035359503147549,
+            0.008754719437760327,
+            0.004067820167375348,
+            -0.014861417626433965,
+            0.009346669946626484,
+            -0.020136029447262147,
+            0.007501669351583119,
+            0.023178490169347605,
+            -0.006840359541885465,
+            0.0006676450965357345,
+            -0.01283320739092204,
+            -0.015610538827742276,
+            0.006308543164022682,
+            0.002329420005586303,
+            -0.002295739637504597,
+            -0.02181189272979859,
+            0.006298804075737111,
+            -0.014398470126723037,
+            0.004007419942728446,
+            -0.0007497656993578918,
+            0.013863867106289847,
+            0.016451166273826513,
+            -0.006740209707090658,
+            0.039292260347522444,
+            0.01871723939316731,
+            0.009109005310505462,
+            0.005674782595856756,
+            -0.040588444759736254,
+            -0.008993850998154852,
+            -0.0010197185016393919,
+            0.004866150984083949,
+            -0.01141354216871663,
+            -0.018944881712855702,
+            -0.002646110151132781,
+            0.000308585036997592,
+            -0.010681954707313127,
+            -0.009091585604191968,
+            0.01751599002710486,
+            -0.006034424764302979,
+            0.00872057856926618,
+            0.004146961781109887,
+            0.023758065635807138,
+            -0.009727802672285544,
+            0.01176257119851645,
+            -0.0071822083765114165,
+            -0.005456197751568526,
+            0.0006726765703478756,
+            0.011011628907288813,
+            -0.008031792262886802,
+            0.009763340867550022,
+            -0.004279891138185881,
+            0.008355720233269893,
+            -0.008552596384067825,
+            -0.023952129458400563,
+            -0.014972827087541732,
+            0.014347535404251018,
+            0.011651260496709961,
+            0.008137738462704715,
+            -0.013624435569539285,
+            0.0031728721674035352,
+            0.00675830277744976,
+            -0.009184784787581942,
+            0.009161771311910335,
+            0.017733082848622708,
+            -0.001411542568472906,
+            -0.0017830167499636113,
+            -0.0016130003349297027,
+            -0.012173528776899308,
+            0.0011196437753239162,
+            0.02284887358189995,
+            -0.003054479643937998,
+            0.014614283913976716,
+            0.010689486748036806,
+            0.009643292262102645,
+            -0.00810930837603355,
+            0.001915955511174678,
+            0.02341863048264366,
+            0.015022291783105098,
+            -0.004625715884342598,
+            -0.011422973894623295,
+            0.004259757717597968,
+            -0.010332744704689279,
+            0.007373328473086509,
+            -0.02117940113144938,
+            -0.004672698999328004,
+            0.008896311380763048,
+            -0.007028862917603513,
+            -0.01732432285099198,
+            -0.028026173555055017,
+            -0.016765805892242715,
+            0.019819329380181,
+            0.0027213760678977965,
+            -0.013291251876048186,
+            -0.0032919084721662363,
+            0.0031214883337300926,
+            0.002553005168048115,
+            -0.0071104806975682885,
+            0.005310583052470149,
+            0.007359183453835734,
+            -0.008674133980486269,
+            0.017665958608281126,
+            -0.000521269431423141,
+            -0.0008020251454616109,
+            -0.0005442374261144659,
+            0.0006127687598711624,
+            -0.016387752806126442,
+            0.01666161912459431,
+            -0.010949228376419892,
+            0.015198110119816624,
+            -0.008966726991404666,
+            0.008600499511690635,
+            -0.007617043351740469,
+            0.009458641139070436,
+            0.011597957400445055,
+            0.012755037944648426,
+            -0.004403680820063604,
+            0.008519241473418196,
+            0.004807816191718704,
+            -0.008147125240985166,
+            0.007017601015398136,
+            0.01296082929962557,
+            0.00582353932160707,
+            0.0013634237970372698,
+            -0.0008246894322793264,
+            0.0022035670958705576,
+            -0.002480976219285683,
+            0.0056293125704658135,
+            0.012714850523393719,
+            -0.002497639449055006,
+            -0.007135683201188999,
+            0.014988249771676699,
+            0.014955895630183352,
+            -0.022994948923833993,
+            0.007121634434082677,
+            0.005835971825639792,
+            -0.0030507120821557008,
+            0.002145266953387315,
+            -0.027334525318320587,
+            -0.020359474070022213,
+            0.01901995970350808,
+            -0.015260454474605018,
+            0.02850460517419317,
+            -0.011624138792780823,
+            -0.02284242513728605,
+            -0.01277489645734181,
+            -0.004209529633534193,
+            -0.022684500849916897,
+            -0.004894414098943742,
+            0.01919165657366256,
+            0.013401560972810082,
+            -0.004313549020265677,
+            -0.019089369171102138,
+            0.03863248408555474,
+            0.010497951840980463,
+            0.03508157919687289,
+            -0.008597970944278045,
+            0.0023354831464448313,
+            -0.01563401808449416,
+            0.015551176593436512,
+            0.001075882802133782,
+            -0.011575692834796734,
+            0.01505716108583101,
+            0.031836226915668404,
+            0.0034740889237207494,
+            -0.00617206855371193,
+            0.0018420207148235309,
+            0.00418433325800744,
+            -0.0008167766650900381,
+            0.011160989178055499,
+            -0.018497472318034093,
+            -0.011951262348399922,
+            0.0013093376768780122,
+            0.00216347903176653,
+            0.0059233460419514045,
+            -0.001778091863744448,
+            -0.013317639704511549,
+            0.011433308892643204,
+            0.011596810919469926,
+            -0.01518143105525228,
+            -0.007497373860338335,
+            0.0050957665991362405,
+            -0.018974405317819656,
+            0.01872472902494418,
+            0.011859522398789263,
+            0.000792413325332956,
+            -0.0006503947129906383,
+            0.00808635755624928,
+            0.003749813548995223,
+            0.0033657799335872368,
+            -0.007862215303617144,
+            -0.0028014669075116876,
+            0.0009892310856576799,
+            3.834379772674841e-06,
+            0.015402211364447726,
+            -0.015264893397161102,
+            0.02488855366696587,
+            -0.0025643639525691083,
+            0.00953993868921819,
+            -0.017460590918773188,
+            -0.0017911712931080155,
+            -0.00402374787529868,
+            0.01838051670627292,
+            0.006753072393532114,
+            -0.010130676216186492,
+            0.004921510943896829,
+            0.007975201081296845,
+            -0.012589786869771252,
+            0.00018319054705558588,
+            -0.0022606180909706688,
+            -0.014791528072476771,
+            -0.0031171076370692706,
+            0.017873825561816544,
+            0.007593688317551089,
+            0.0037375070591255375,
+            -0.015863143964928172,
+            0.007192324815779268,
+            0.007522990667369163,
+            0.024504875417877527,
+            -0.026309816223960274,
+            0.024779042752840836,
+            -0.022104087334088543,
+            0.02165781034613893,
+            0.0319051741666687,
+            0.007538146369110045,
+            -0.01690181186708103,
+            -0.006595947249169521,
+            0.006373991305923002,
+            -0.023410814842488123,
+            -0.005494017994983188,
+            0.016903736810723716,
+            -0.01792400152372258,
+            0.000330865061689199,
+            -0.014660413003063089,
+            0.0040825880154611895,
+            0.008248037865838212,
+            0.011540333451799686,
+            -0.0033578830043738085,
+            0.018327525495676525,
+            0.020171745352159914,
+            -0.01138759854371235,
+            -0.0004221149053107702,
+            -0.022321409200941777,
+            0.005046865873981027,
+            0.022509643273449936,
+            0.001111272240506322,
+            -0.009540467231190513,
+            0.018376261423745193,
+            0.008480532791922674,
+            -0.020801891594934313,
+            0.008436374227753589,
+            -0.036175982655998785,
+            -0.010454118523313122,
+            -0.009328507118928412,
+            -0.001218633166683384,
+            -0.004142773602181115,
+            -0.0002494371580772723,
+            -0.011365548842513265,
+            -7.699548642315061e-05,
+            -0.006889339420877534,
+            -0.0037173070609413474,
+            0.014953978004522174,
+            0.011678957032329096,
+            -0.003501047405385249,
+            -0.0073984932581048915,
+            0.008493128567102267,
+            0.006163937693564917,
+            -0.017034696006318337,
+            -0.013268418502042576,
+            0.0037122128689059085,
+            -0.020067656401141467,
+            0.005148329575092478,
+            0.00459729178556505,
+            0.014031669598917171,
+            -0.01849235806598133,
+            -0.009298127665365717,
+            0.019716512208448042,
+            0.0094466405158553,
+            -0.0008988916054855498,
+            0.004264260983102956,
+            -0.00032312548009733906,
+            0.002279987803645912,
+            -0.0008902345374346296,
+            -6.145528806973222e-06,
+            0.003602229465148649,
+            0.012503366799381277,
+            0.007759393855482671,
+            0.03694402922447051,
+            -0.016692148708641812,
+            0.005873765863350649,
+            -0.014187609702966002,
+            -0.0037592517335595654,
+            -0.00749939010396784,
+            -0.0004308157309481927,
+            -0.0022282234382577843,
+            0.0021999996319642854,
+            0.0052176773733353925,
+            0.009353171029508977,
+            0.002155405440765206,
+            -0.00823565584692709,
+            0.009691447250255694,
+            -0.015228173095533029,
+            0.003853858061067531,
+            -0.008207910291425473,
+            0.01017972550354623,
+            -0.009695280827515048,
+            0.0028705587799331246,
+            0.03849358164875786,
+            0.01130021300937487,
+            -0.004934175853966098,
+            -0.014300147534361459,
+            -0.0007796051618993811,
+            -0.01737877501558196,
+            -0.008579119859817895,
+            -0.005434792488602325,
+            -0.02617338653902575,
+            -0.00838916510137031,
+            0.015344260464413648,
+            0.011202567142310492,
+            0.014124853863170132,
+            -0.023894182185855824,
+            0.008897943489322673,
+            -0.03001670823127986,
+            -0.011951507895140876,
+            -0.001530967140156163,
+            -0.015467812535428609,
+            0.0010110999991167703,
+            0.023623196504289838,
+            -0.007762735863648054,
+            0.012654652822750325,
+            0.012867958170258448,
+            0.018777522196571934,
+            0.006390986057157016,
+            0.00038045736974183937,
+            -0.009444911783760569,
+            0.009534310165417939,
+            0.008806031651319704,
+            0.0080954543713255,
+            0.013161833638467553,
+            -0.0022840529693446553,
+            -0.018060162439023572,
+            0.005469288590000856,
+            0.014354475778235131,
+            -0.0033504276914600585,
+            -0.001615191819057537,
+            -0.011088493180661976,
+            0.016039040565630432,
+            -0.023559046333872156,
+            -0.009482398794557168,
+            0.009918576960319548,
+            -0.0027308309155029842,
+            0.009212063547988036,
+            0.0160181200081126,
+            -0.01881373001297218,
+            -0.0024168618343039703,
+            0.016244889977796682,
+            0.0024104546308670025,
+            -0.00022606425578303413,
+            0.016792244238218245,
+            -0.0007616857360728623,
+            -0.004279120886828465,
+            0.00997764266310123,
+            -0.0075631917138318655,
+            7.3878137053128405e-06,
+            -0.002416364915008295,
+            -0.0028253498168087964,
+            0.02054128727706496,
+            -0.021281801678871736,
+            -0.005379259148621041,
+            0.02257306103533458,
+            0.015687105622685975,
+            -0.007843745111084413,
+            6.258498042855072e-05,
+            -0.0219506172079994,
+            0.0019925886885244454,
+            -0.007768133999661569,
+            0.016524850592507592,
+            -0.007957949830060354,
+            0.0030085044673901665,
+            -0.01361024620290394,
+            0.011563781350927133,
+            -0.01277407169098418,
+            0.00873716215529134,
+            -0.002986943948333218,
+            -0.017567646684567492,
+            -0.0053109390854215125,
+            0.004561236136619266,
+            0.0014045175130801308,
+            0.044958041347040185,
+            -0.005539136232703934,
+            0.003729517522976541,
+            0.018086335927383804,
+            -0.014199453349727933,
+            0.02079719360289355,
+            0.016663293302592568,
+            0.00991445855219406,
+            -0.00033967370741635794,
+            -0.0018726748849858836,
+            0.023373354806639515,
+            0.013258226441642464,
+            0.00861986998150065,
+            -0.0014340218811281975,
+            -0.008516541227326329,
+            0.018239230461345617,
+            0.005458629943667832,
+            -0.012421896028975013,
+            0.012732721349758832,
+            0.013559877132508506,
+            -0.013720939472355347,
+            -0.011545210834395917,
+            0.01616358223872008,
+            -0.019904594319155373,
+            -0.018260016571763644,
+            -0.0036124537591368864,
+            -0.01614510591132948,
+            -0.03864699384205461,
+            0.01619401261888991,
+            0.018351607973242735,
+            0.006666313000123845,
+            0.013647410298896132,
+            0.002597680061427196,
+            0.004545530267244157,
+            -0.018531082430861854,
+            -0.0077618625909995275,
+            -0.008085144524135461,
+            0.003744735923004248,
+            0.011059457364156674,
+            -0.011335432904114519,
+            -0.009514379703581616,
+            -0.00034500702695418285,
+            -0.007433132060989644,
+            0.01971420270994951,
+            0.0020753610290475642,
+            0.0034506706065161024,
+            -0.00628437298309269,
+            0.017966485854173167,
+            0.026789678964152908,
+            -0.007532446825077476,
+            0.021886182869092073,
+            -0.002864230253056434,
+            -0.0015870360120997492,
+            -0.014393519359298496,
+            -0.0014439412842964876,
+            -0.011527182213472351,
+            0.0005453493437517964,
+            -0.0006791481533636673,
+            0.014093767997869645,
+            -0.03288824642147967,
+            -0.003957301296796813,
+            0.014444592778851293,
+            0.007538052312876599,
+            -0.005057755472988024,
+            0.011357919882949903,
+            -0.002964451028327082,
+            0.016977317604413924,
+            0.0075201055190542146,
+            0.023391646079994784,
+            -0.016588823248397513,
+            -0.004504858318770565,
+            0.006660877064954993,
+            0.0034631255931793638,
+            0.0014484203909961972,
+            0.01912983041187014,
+            0.023439941258581887,
+            -0.014715886385460541,
+            -0.0013596123366001458,
+            -0.0029020372473879614,
+            -0.021414221343490594,
+            0.014976806123253727,
+            0.0074729481304182865,
+            0.010967351226114886,
+            0.009806663105020302,
+            0.015465076881663691,
+            -0.02109587527354609,
+            0.0011671121739822202,
+            0.020567458365051382,
+            -0.008618333549105003,
+            -0.018815706906171112,
+            -0.009804829158883123,
+            -0.00966828714719191,
+            -0.006078893678150302,
+            -0.00803783313451146,
+            0.005523023167028105,
+            0.0022647506733877573,
+            0.01652997604669998,
+            -0.008204200434441975,
+            -0.008404609147866811,
+            0.008489247125986078,
+            0.007546785673285454,
+            -0.0033093451283570316,
+            0.010614050963656496,
+            0.0030633179859393587,
+            -0.006940540318011839,
+            0.00894884343338777,
+            -0.002787816848430143,
+            0.014347598920245487,
+            -0.02441769157218451,
+            0.004717742147185283,
+            -0.012096876825337124,
+            0.006405151160675699,
+            0.0041568790527989905,
+            0.0016702066682613428,
+            -0.0041159211459004,
+            0.0035217362856754353,
+            0.0008893479210743541,
+            -0.07433312151926533,
+            -0.012629681414004524,
+            -0.0035797012285864486,
+            0.033657838713146884,
+            -0.019695806809053914,
+            0.004477927056336216,
+            0.011966065644382396,
+            0.0096717896523898,
+            0.003070713232538593,
+            0.006136726101157844,
+            0.006112385569942848,
+            -0.0245258242481677,
+            -0.00015284418977092615,
+            0.004821040694679944,
+            0.0020347004045943454,
+            -0.007532408630122127,
+            0.00750565079228405,
+            0.015327710245879296,
+            0.020989926895388523,
+            -0.010970305087289488,
+            0.017227663358885748,
+            -0.003352718496981928,
+            -0.006068892126250978,
+            0.0014285750322333686,
+            -0.010153799722483554,
+            -0.017307415843726394,
+            -0.006495008086641125,
+            0.0068983699213297075,
+            -0.001982568363667046,
+            0.0044032687939897855,
+            0.003333596495330213,
+            -0.0032171586993948696,
+            0.0072365127663633925,
+            0.013920326463272646,
+            0.006415916101287133,
+            -0.013675562796300214,
+            0.003003782758807665,
+            -0.02500375035312058,
+            0.011945988879567646,
+            -0.012708975181025296,
+            -0.0009488161508767279,
+            -0.0047093877027523014,
+            -0.0035750724825159503,
+            -0.002213435677949806,
+            0.02043567538125188,
+            -0.014304009033474186,
+            -0.01802899775650995,
+            0.008187227768196673,
+            -0.011217013195839762,
+            0.008300740149622635,
+            -0.03382831122769974,
+            0.012386335491333264,
+            -0.0276029934382802,
+            -0.0002243752790856292,
+            -0.004521387639042107,
+            -0.0010631721066129095,
+            -0.009491184157278669,
+            -0.03274559774619223,
+            0.015845705808609598,
+            -0.007204751554079551,
+            -0.020338337341085228,
+            0.0007143424225187802,
+            -0.021267094083674304,
+            0.014150458196361147,
+            -0.01851165553224022,
+            0.0033450353123933058,
+            -0.018775439467431685,
+            0.002782289821140401,
+            -0.0018711350216089556,
+            0.010502195116579441,
+            -0.00557814442809019,
+            -0.0007130962242755068,
+            -0.013072988887382269,
+            -0.002121841212381393,
+            -0.006653444103331213,
+            0.009394907690516121,
+            0.0028678074167728367,
+            -0.009601577734902932,
+            0.011153571864462314,
+            0.013046116892519414,
+            0.019812860521973938,
+            -0.00539047787915588,
+            -0.00555139958411929,
+            -0.011553689090179406,
+            0.009852063270798534,
+            -0.016326212706634428,
+            -0.00827402913960903,
+            0.008982303148984887,
+            0.015216947130173639,
+            0.01026889105233137,
+            0.003352028462252994,
+            0.003150801750574548,
+            0.002531620731961765,
+            0.026744691368233237,
+            -0.0015120532859309638,
+            0.0008324701960729416,
+            -0.014508988496009931,
+            0.022127525293249926,
+            0.02772430884375744,
+            -0.0012206102581726523,
+            -0.00018859639594030037,
+            0.017437360169132967,
+            -0.004697052522250805,
+            -0.0038111184371524645,
+            0.024015246959315593,
+            -0.038396108183338655,
+            -0.008416825232533771,
+            -0.006788816882617399,
+            -0.006043428198944214,
+            -0.009587366247924467,
+            0.007294190273465812,
+            0.00041966080769004275,
+            0.0024709692753547823,
+            -0.0005250904505954556,
+            0.03452858797518386,
+            0.003403515654095309,
+            -0.0003392060101496309,
+            -0.0032980864446307116,
+            0.0019125121221204029,
+            0.0032987517853214346,
+            -0.009116761755779728,
+            -0.038030085122772624,
+            0.006006334000396066,
+            0.007932785488877407,
+            0.00975009749364395,
+            -0.008120121513849434,
+            -0.008864461019882246,
+            0.004909348738798099,
+            -0.02273129283865203,
+            -0.01372636307167243,
+            0.02625229144362647,
+            -0.008523028986127237,
+            0.029005877324974557,
+            -0.022435059061182174,
+            -0.013135836801818153,
+            -0.004126433322528265,
+            0.010283975013795537,
+            -0.019732856462444506,
+            0.004343860338939202,
+            -0.004022387065239562,
+            0.023733721026005954,
+            0.0077578144626961245,
+            -0.008371631446146718,
+            0.012386735614153616,
+            0.0064789380436815714,
+            -0.0026638642827013837,
+            0.00136333052963848,
+            -0.007209517067748103,
+            0.016916505654836485,
+            -0.011819784026452944,
+            -0.005833909403097991,
+            0.008977527508585844,
+            -0.0024556434204852946,
+            0.02509293948918158,
+            -0.0237762245071987,
+            0.005288171058938098,
+            -0.00019047961064753207,
+            0.01822542471604037,
+            0.008127347078975925,
+            0.008361588667386882,
+            -0.011835407590060994,
+            0.010004482616697591,
+            -0.03378107067882622,
+            -0.015433906677849963,
+            -0.007702910250795191,
+            -0.0013877702264578865,
+            -0.002059152251387601,
+            -0.015745218548968338,
+            -0.026393221735795527,
+            0.006294583740809379,
+            0.016267525575472445,
+            0.023291528020980265,
+            -0.004683197930892445,
+            0.012912830736650997,
+            -0.002831583991769044,
+            -0.008904271394710672,
+            -0.011441882849296418,
+            -0.012202629605208118,
+            -0.012470773094383122,
+            -0.014560694782340029,
+            -0.017045701603711442,
+            0.012139665577738337,
+            -0.020079061996873537,
+            0.013911518328924777,
+            0.0347210775522539,
+            -0.017176114991595492,
+            -0.0018941772168839354,
+            0.019702722555622765,
+            -0.004221469612316137,
+            0.01212088353057276,
+            0.0063707489618353725,
+            0.015298014532351866,
+            0.0038643174573634725,
+            -0.012181479717826009,
+            -0.007374005740809251,
+            0.019411457664902376,
+            -0.004902591844710158,
+            0.014591446725717338,
+            0.015069599884557399,
+            -0.008675672164385891,
+            0.002544198867387345,
+            0.01951865904496276,
+            -0.01762101777798757,
+            0.01665654707715257,
+            -0.00458402229480694,
+            -0.009936206685538294,
+            -0.013345740581684638,
+            -0.005336416724426228,
+            -0.019469457091169776,
+            0.0061608454109892425,
+            0.018931676738452464,
+            -0.00533013594306834,
+            0.009310871143727683,
+            -0.0037670793158045,
+            -0.005045151384034014,
+            0.0002041909886640482,
+            -0.04152751596227068,
+            -0.008340911336217609,
+            -0.0006651584370005978,
+            -0.02505748815737022,
+            0.025954102004571416,
+            0.007205602377144382,
+            0.022602344303400163,
+            -0.017168610245491406,
+            0.004459735690564842,
+            -0.005882024209102714,
+            0.013217517324383834,
+            0.02255689669623969,
+            0.0017984057469785134,
+            0.005699128497215882,
+            -0.02353849721739205,
+            -0.0034711576718386185,
+            0.015924913374553738,
+            -0.003604478141616307,
+            -0.0034993903155063817,
+            0.012592308877056896,
+            0.012864938348003884,
+            0.01567418642605425,
+            0.008801753550506258,
+            0.015606443428689292,
+            0.010589933994371628,
+            0.008928721556432368,
+            -0.005805129999077126,
+            -0.001025103064751441,
+            -0.015907755521365925,
+            0.0045874330999415985,
+            0.031701049450012876,
+            0.012594483199773746,
+            0.016783858873835704,
+            -0.01778536268185936,
+            0.00789523134933153,
+            -0.0012706411671644011,
+            0.008325108613530198,
+            -0.001677173669891956,
+            0.029540136288412976,
+            0.011739999605699826,
+            0.015670966746894652,
+            0.008732334090105053,
+            -0.016237731209426752,
+            0.045334690225148,
+            -0.016911928942220152,
+            -0.0021139311364009113,
+            -0.01058970056572469,
+            -0.021705338531028618,
+            0.007324792557651635,
+            -0.006556582104686972,
+            0.0018243188317438144,
+            0.010780292738415671,
+            -0.006471850255151019,
+            0.0006997395548596146,
+            0.003907433192252573,
+            -0.023129313269763224,
+            -0.002550875642832235,
+            -0.02237917712764273,
+            -0.003107775639788617,
+            -0.02621608419628401,
+            -0.00012482032027919036,
+            -0.016207377503696517,
+            -0.0038712739995960753,
+            -0.023315508569288476,
+            -0.014963211090562272,
+            -0.01633110043829169,
+            -0.004241081909202262,
+            -0.008295402942465353,
+            0.004976396647108597,
+            -0.007402107755970892,
+            -0.020321297548927468,
+            0.0020986169430507757,
+            0.016803273183070004,
+            0.01434729320047051,
+            0.018612970083178984,
+            0.00603868809307069,
+            0.014278001198851581,
+            -0.0026350854954985117,
+            0.017171016209107656,
+            -0.006968968261909459,
+            0.006881847166565678,
+            0.011454001418121107,
+            0.02140935294963631,
+            0.011121833270235448,
+            -0.012801756637470525,
+            0.0010870041670931706,
+            -0.009655388034828094,
+            0.002094601818569396,
+            -0.00828486034340067,
+            -0.009190975158350585,
+            0.0017762581946855365,
+            -0.007261996868906865,
+            0.004674405202069334,
+            0.0030526253833799426,
+            -0.0014082232541965316,
+            0.010419302200840906,
+            -0.0024200053377751805,
+            -0.012208685875688886,
+            -0.0033037330470612695,
+            -0.017975940046938498,
+            -0.006958042903103732,
+            -0.0014496522238488239,
+            -0.011711502944602638,
+            -0.01016130418405875,
+            0.003321436189059608,
+            -0.00538282506552848,
+            -0.009353581612407435,
+            0.013884147453136417,
+            0.0001309738696853529,
+            0.0005878330075918333,
+            0.023495280602974504,
+            0.006035028203642483,
+            0.008087594627921345,
+            0.013144288945385904,
+            -0.01023995000994454,
+            -0.022574958230635715,
+            -0.0243435727055662,
+            0.007867022215096418,
+            -0.01573478573423379,
+            0.02324699908478915,
+            -0.007929809922043611,
+            -0.00561630876431978,
+            0.0020982723335057625,
+            -0.013699317926847062,
+            0.008578431496569186,
+            -0.011271381061801356,
+            0.0014450645788554738,
+            -0.019581451799757368,
+            0.018059311800267413,
+            -0.010917911869373693,
+            -0.004302687883018328,
+            0.009466103226841532,
+            0.034049086327052606,
+            -0.0005689605174080737,
+            -0.007410970644432269,
+            -0.006594150877740108,
+            -0.00022810271313786974,
+            0.018766765773409123,
+            -0.0029041815756733333,
+            0.005758781908059541,
+            -0.01563519107805783,
+            0.006571628287171105,
+            -0.006132372252825422,
+            0.0012423261344297225,
+            0.01513074831647933,
+            -0.026306320707079913,
+            -0.015491710256868008,
+            -0.007857352164317122,
+            -0.024893430640551818,
+            0.002253574511878983,
+            0.017579477884206534,
+            0.020505606775521504,
+            -0.020324505029926956,
+            -0.02513332780748072,
+            0.005150966733930881,
+            -0.026580994694344065,
+            0.0037390577082486594,
+            0.1241031878517593,
+            -0.002138404898309707,
+            0.002720723037908521,
+            -0.011425036886011179,
+            0.0022399461009316216,
+            0.005389407121650262,
+            -0.019552543955456567,
+            0.05950746473186985,
+            -0.00638023587296763,
+            -0.004023798021748334,
+            0.002170545188776296,
+            0.001785855390676113,
+            -0.005831534307818096,
+            0.008643501093778434,
+            -0.017602787281866816,
+            -0.02184792441891879,
+            -0.0052215705748838905,
+            -0.021252169339074325,
+            -0.008509621422071373,
+            -0.013000137524907565,
+            0.002893880203681282,
+            -0.015821609145923173,
+            0.010397692426614709,
+            -0.01638916243796692,
+            -0.010283706937701179,
+            -0.01103621454761223,
+            -0.00373552043529251,
+            0.01563553822905027,
+            -0.03188533919653949,
+            -0.004873550301425776,
+            -0.005183590054152726,
+            0.018218971950313273,
+            0.006004328765748325,
+            0.004357734370247252,
+            -0.014230318971458851,
+            0.0064792070801456235,
+            0.0215589173714926,
+            0.0055272493764401305,
+            0.014643335833970546,
+            0.010618070641357003,
+            -0.0022177303813575846,
+            -0.0024402761885758007,
+            0.017541799738501007,
+            -0.005445774414228888,
+            -0.00023502936119626386,
+            -0.00537569046685066,
+            -0.005974703247446444,
+            -0.010712244203425036,
+            0.01843613166635215,
+            -0.009938564643456672,
+            -0.017388592763277214,
+            -0.0015797412875366156,
+            0.02028434083164951,
+            -0.012761534382003973,
+            0.0031906812114950982,
+            0.000788436210275669,
+            0.020589196433948156,
+            0.0007726855314648301,
+            -0.005481260996289263,
+            0.007165014826294827,
+            -0.02956334397512344,
+            0.004095715232997753,
+            0.0003568512047626187,
+            0.027290768799658912,
+            -0.0017631129386615335,
+            0.01552542586826711,
+            -0.020871286597113327,
+            0.004388125824324094,
+            -0.016926008631435403,
+            0.005099899758322398,
+            0.012560923968841436,
+            0.002781049890810361,
+            0.010889021753593997,
+            -0.008500358897706648,
+            -0.021353460542554444,
+            0.012663934009161902,
+            -0.018657094130860155,
+            -0.029156017587920958,
+            0.0018889388761887053,
+            -0.028158076315864772,
+            -0.003048854609750186,
+            0.006812072971079798,
+            -0.035217412591558014,
+            -0.02251675180883659,
+            0.007396524512996277,
+            -0.00995571600201846,
+            -0.011912753666073882,
+            0.010675240017836214,
+            -0.02645155743041798,
+            -0.018137648527851668,
+            0.0010666944556449141,
+            0.012588800077165337,
+            -0.004561825939541096,
+            0.007555558090958502,
+            -0.028558603070876605,
+            0.014816816621445352,
+            -0.01313764261744744,
+            0.016036065063831813,
+            -0.0037797584923144134,
+            0.022217516257304868,
+            0.011131597210722573,
+            0.008795376259031849,
+            -0.004950080558710245,
+            0.021409590863550987,
+            0.005574351961953293,
+            0.051358937509801736,
+            0.031972009383746405,
+            0.0019821003238211276,
+            -0.003251257665378234,
+            0.003313045083311649,
+            0.008294776423730887,
+            -0.00940948257229233,
+            0.005114631360856872,
+            0.028053227377164806,
+            0.016738013668710083,
+            -0.018755924318109216,
+            0.006606711923035118,
+            -0.02702692774986498,
+            0.017950884766781056,
+            -0.022054653845163,
+            0.0003755944703013671,
+            0.007766253634286795,
+            0.0016206993755471613,
+            0.010682347575070117,
+            0.0013842930079185415,
+            0.01705234255261425,
+            0.023975191051210525,
+            -0.002976402980885077,
+            0.017466027107477605,
+            0.012734909275073793,
+            0.018143161773162015,
+            -0.0038399373197265827,
+            0.006929440921265761,
+            -0.012003095293274786,
+            -0.0008883470597690258,
+            0.0020490527756307267,
+            -0.004231091383919565,
+            0.012128848448049833,
+            0.008340302519160299,
+            0.012989849936937843,
+            0.002974488532907779,
+            0.003198854381461613,
+            -0.005309753760578859,
+            -0.0017024982049936384,
+            -0.01763395035509857,
+            -0.01819680226734474,
+            0.028417724274372468,
+            -0.020217879775485576,
+            -0.050118184195774004,
+            0.0025509548991664425,
+            0.014185761006969567,
+            -0.0012021669304388173,
+            -0.0051260576846881515,
+            -0.0005774778872927327,
+            0.005718864494530703,
+            -0.016918749210038116,
+            -0.0015472638835434902,
+            0.008125845907627804,
+            0.020347547062899797,
+            -0.002036218493206914,
+            -0.01079739518866248,
+            0.0038112958713600906,
+            0.02169339303105242,
+            0.01667594322285835,
+            0.012338193092567552,
+            -0.009798409233733167,
+            0.0003198215558702285,
+            -1.6308740825163337e-05,
+            0.0076870827874737325,
+            0.004356644356656038,
+            0.003500023005601613,
+            -0.019968549869439228,
+            -0.008634740687755112,
+            0.012781565963545847,
+            -0.0025637758121512927,
+            0.010243567793632335,
+            -0.00042296044205132426,
+            -0.003069251553118065,
+            -0.009344199449486244,
+            -0.004694953215957636,
+            -0.009583993744231791,
+            0.013783878036521561,
+            -0.007705982042252044,
+            -0.01549567538665377,
+            0.00020666298495823114,
+            0.0005515775451348472,
+            0.001667367278489879,
+            -0.004518431526236256,
+            0.03302324454129141,
+            0.009531990824506103,
+            -0.0004937348108340977,
+            -0.004047613286672229,
+            -0.0327383112522095,
+            0.012456150771224677,
+            -0.009677533214778067,
+            0.0006455811519981003,
+            0.020012976827426605,
+            -0.00020277620481825933,
+            0.002168432791818704,
+            -0.015657551629104767,
+            0.0032924357559238502,
+            0.006161295735841883,
+            0.013814165416788555,
+            0.0005164426482972658,
+            0.01076512534105537,
+            -0.006498977875968572,
+            -0.010325144017898257,
+            0.004804158067334203,
+            0.0104064855510473,
+            -0.0012188100892986933,
+            0.005171197790296134
+          ],
+          "title": "Mixture-Of-Experts With Expert Choice Routing",
+          "keyphrases": [
+            "Mixture-Of-Experts",
+            "Expert Choice Routing",
+            "sparsely-activated models",
+            "training efficiency",
+            "downstream performance"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "e96a5505-edee-45dd-be65-af56dee281e4",
+        "properties": {
+          "page_content": "D Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [? ]. We use mod x to map a token ID to an expert ID. This in some way ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 2. Hashing based routing performs much worse than expert choice in terms of average scores and variance.\n\n",
+          "title": "D Comparison With Hash Layer",
+          "embedding": [
+            -0.015004903078079224,
+            -0.027099307626485825,
+            -0.030763821676373482,
+            -0.012313068844377995,
+            0.005327117629349232,
+            -0.009055723436176777,
+            -0.007592933718115091,
+            0.055405039340257645,
+            -0.039389755576848984,
+            -0.0020019104704260826,
+            -0.016120845451951027,
+            -0.011890820227563381,
+            -0.0219267625361681,
+            0.00947797205299139,
+            -0.006043432280421257,
+            0.06357856839895248,
+            0.02605876699090004,
+            0.014650516211986542,
+            -0.0008935080841183662,
+            -0.05407797545194626,
+            0.01372307725250721,
+            0.008814438246190548,
+            -0.03872622177004814,
+            -0.0012384700821712613,
+            0.021429114043712616,
+            0.0030575317796319723,
+            -0.0006979353493079543,
+            -0.023012544959783554,
+            -0.03145751729607582,
+            0.046960070729255676,
+            0.024007845669984818,
+            0.01857893541455269,
+            -0.006804987788200378,
+            -0.05121271684765816,
+            -0.024294370785355568,
+            -0.013783399015665054,
+            0.046990230679512024,
+            0.038092851638793945,
+            -0.036946747452020645,
+            0.04632669687271118,
+            0.04659814387559891,
+            0.062311820685863495,
+            -0.03190992400050163,
+            -0.010179205797612667,
+            0.000586718088015914,
+            -0.0037851566448807716,
+            -0.014424311928451061,
+            -0.0021338630467653275,
+            0.03184960409998894,
+            -0.0449996292591095,
+            -0.011257447302341461,
+            -0.07594441622495651,
+            0.025395233184099197,
+            0.012343229725956917,
+            -0.005538241937756538,
+            0.050549183040857315,
+            0.0019906002562493086,
+            0.027808083221316338,
+            0.023600677028298378,
+            -0.00661648390814662,
+            -0.004599492996931076,
+            -0.014469552785158157,
+            -3.43724932463374e-05,
+            0.015306509099900723,
+            0.010194285772740841,
+            0.0011376205366104841,
+            -0.013157566078007221,
+            -0.008015181869268417,
+            0.04210421070456505,
+            -0.015427151694893837,
+            -0.039118312299251556,
+            0.031216232106089592,
+            -0.0038567879237234592,
+            0.0019679798278957605,
+            -0.028697820380330086,
+            -0.00024034235684666783,
+            0.021821200847625732,
+            0.03218137100338936,
+            -0.00124601018615067,
+            0.020252849906682968,
+            0.012456332333385944,
+            0.0006861538859084249,
+            -0.002107472624629736,
+            0.0172820296138525,
+            0.018880542367696762,
+            -0.03504662960767746,
+            -0.02144419401884079,
+            -0.06122603639960289,
+            -0.015442232601344585,
+            -0.018232088536024094,
+            -0.01572875864803791,
+            -0.015909722074866295,
+            -0.007057582959532738,
+            -0.0011168851051479578,
+            -0.05905447527766228,
+            0.0761253759264946,
+            0.005873778834939003,
+            -0.0025391464587301016,
+            -0.023660998791456223,
+            -0.00031456572469323874,
+            -0.013579814694821835,
+            0.0022752410732209682,
+            -0.013421471230685711,
+            0.0437932051718235,
+            0.06086410954594612,
+            -0.029512157663702965,
+            0.010978462174534798,
+            0.022107727825641632,
+            -0.0007973711472004652,
+            0.1003141850233078,
+            0.006884159054607153,
+            0.005941640119999647,
+            -0.037549957633018494,
+            0.002186644123867154,
+            -0.01580416038632393,
+            -0.04240581765770912,
+            -0.03700706735253334,
+            -0.05407797545194626,
+            0.019619476050138474,
+            -0.046748943626880646,
+            0.00444114999845624,
+            0.04273758456110954,
+            -0.015140626579523087,
+            -0.0038096620701253414,
+            -0.01768919825553894,
+            0.013383770361542702,
+            -0.022861741483211517,
+            0.011189586482942104,
+            -0.0014024684205651283,
+            0.000858163577504456,
+            -0.004241335671395063,
+            -0.01000578235834837,
+            -0.029059747233986855,
+            -0.007672104984521866,
+            0.003811547067016363,
+            0.04047553986310959,
+            0.047653764486312866,
+            -0.0007059467607177794,
+            0.008761657401919365,
+            0.0011036897776648402,
+            0.012320608831942081,
+            -0.02206248603761196,
+            -0.018443213775753975,
+            0.010133964940905571,
+            0.0034119191113859415,
+            0.018322570249438286,
+            0.005353508051484823,
+            0.04521075263619423,
+            -0.03761028125882149,
+            0.0037681912072002888,
+            0.02813984826207161,
+            0.017870161682367325,
+            -0.006635334342718124,
+            0.0455123595893383,
+            -0.002667329041287303,
+            -0.015118005685508251,
+            -0.004588183015584946,
+            -0.01981552131474018,
+            0.0016041673952713609,
+            -0.0403548963367939,
+            0.014771158806979656,
+            0.00011899303353857249,
+            -6.232407031347975e-05,
+            0.02773268148303032,
+            -0.003076382214203477,
+            0.06930907815694809,
+            0.02446025423705578,
+            -0.0023204819299280643,
+            -0.07292835414409637,
+            0.027431074529886246,
+            0.0247920211404562,
+            0.01482393965125084,
+            0.049644362181425095,
+            0.06182925030589104,
+            -0.016799459233880043,
+            0.02823033183813095,
+            0.012810719199478626,
+            -0.007140524219721556,
+            -0.029994726181030273,
+            0.02352527529001236,
+            -0.04255662113428116,
+            -0.027114387601614,
+            0.046628303825855255,
+            -0.010450650937855244,
+            0.041229553520679474,
+            -0.007977481000125408,
+            0.04726167395710945,
+            0.03507678955793381,
+            -0.007902080193161964,
+            0.022198209539055824,
+            0.005432679783552885,
+            0.03423229232430458,
+            0.00433935783803463,
+            -0.007438360247761011,
+            -0.007193305529654026,
+            -0.03655466064810753,
+            -0.005575942806899548,
+            -0.024203889071941376,
+            0.033357635140419006,
+            0.006936940364539623,
+            -0.0062696365639567375,
+            -0.049855489283800125,
+            0.009251766838133335,
+            -0.014537413604557514,
+            0.01063915528357029,
+            -0.0010669315233826637,
+            0.051242876797914505,
+            -0.008957700803875923,
+            0.00243546930141747,
+            -0.021263230592012405,
+            0.002537261229008436,
+            -0.012177346274256706,
+            -0.006503381300717592,
+            0.014303669333457947,
+            -0.03100510686635971,
+            0.023600677028298378,
+            -0.006578783039003611,
+            -0.0004493459709919989,
+            -0.010986002162098885,
+            0.0029481996316462755,
+            0.040415216237306595,
+            0.015374370850622654,
+            0.017387591302394867,
+            -0.012426171451807022,
+            -0.004840777721256018,
+            0.01446201279759407,
+            -0.00019792901002801955,
+            0.009349789470434189,
+            0.007743736729025841,
+            -0.018533695489168167,
+            0.01826224848628044,
+            -0.020328251644968987,
+            0.014673137106001377,
+            0.0023921134416013956,
+            -0.006541082169860601,
+            -0.042948707938194275,
+            0.0064166695810854435,
+            -0.01943851262331009,
+            0.008188605308532715,
+            -0.002505215583369136,
+            -0.035559359937906265,
+            0.014469552785158157,
+            -0.03426245227456093,
+            0.009176366031169891,
+            -0.0007106593693606555,
+            0.011423330754041672,
+            -0.017719358205795288,
+            0.005127303767949343,
+            -0.0004592424083966762,
+            -0.01776459999382496,
+            0.01362505555152893,
+            -0.038002368062734604,
+            0.0039095692336559296,
+            0.016724057495594025,
+            -0.015743838623166084,
+            0.04246613755822182,
+            0.017824919894337654,
+            -0.03468470275402069,
+            0.00354764168150723,
+            -0.04116923362016678,
+            0.05814965441823006,
+            0.004049061797559261,
+            -0.005813457537442446,
+            0.020735418424010277,
+            -0.01219242624938488,
+            -0.024354692548513412,
+            0.020976703613996506,
+            0.0028464074712246656,
+            0.00209427741356194,
+            -0.027958884835243225,
+            -0.016347050666809082,
+            -0.043401118367910385,
+            -0.04892050847411156,
+            -0.004369518253952265,
+            0.04611557349562645,
+            -0.03272426128387451,
+            -0.022288691252470016,
+            -0.01286350004374981,
+            0.02528967149555683,
+            0.013361150398850441,
+            -0.007585393264889717,
+            0.030326493084430695,
+            0.02560635842382908,
+            -0.00020405538089107722,
+            -0.008761657401919365,
+            0.019906003028154373,
+            -0.04602508991956711,
+            0.002431699074804783,
+            -0.0633372813463211,
+            -0.010307388380169868,
+            -0.004475080408155918,
+            -0.03827381506562233,
+            -0.002079196972772479,
+            -0.0005141441361047328,
+            0.0239324439316988,
+            -0.02617940865457058,
+            -0.019091665744781494,
+            -0.0003159795014653355,
+            0.021006865426898003,
+            -0.019906003028154373,
+            -0.03010028786957264,
+            0.004007590934634209,
+            -0.03317667171359062,
+            -0.006620253901928663,
+            -0.052962031215429306,
+            -0.005176314618438482,
+            -0.009734337218105793,
+            0.01912182755768299,
+            -0.01895594410598278,
+            -0.000523569353390485,
+            0.00784175843000412,
+            0.02225852943956852,
+            0.035891126841306686,
+            -0.03896750882267952,
+            -0.007574083283543587,
+            -0.006273407023400068,
+            -0.017176467925310135,
+            0.02835097350180149,
+            -0.01207178458571434,
+            -0.008980321697890759,
+            -0.003890718799084425,
+            0.011363009922206402,
+            -0.046960070729255676,
+            0.0005061327246949077,
+            -0.013949282467365265,
+            -0.00020111000048927963,
+            0.01545731257647276,
+            0.04370272532105446,
+            -0.038424618542194366,
+            -0.0023751480039209127,
+            -0.04578380659222603,
+            -0.02859225869178772,
+            -0.038907185196876526,
+            -0.030763821676373482,
+            -0.023178428411483765,
+            -0.014575114473700523,
+            0.017538394778966904,
+            0.004173474386334419,
+            -0.021187828853726387,
+            0.009153745137155056,
+            -0.020901301875710487,
+            0.06828361749649048,
+            -0.007562772836536169,
+            0.015336669981479645,
+            0.001535363495349884,
+            -0.024580897763371468,
+            -0.009915300644934177,
+            -0.00974187720566988,
+            0.02385704219341278,
+            -0.007442130707204342,
+            0.0009670245344750583,
+            -0.0019246239680796862,
+            0.02560635842382908,
+            -0.0013713652733713388,
+            -0.005428909789770842,
+            -0.03721819445490837,
+            -0.0062168557196855545,
+            -0.010118884965777397,
+            0.030733661726117134,
+            -0.036132410168647766,
+            -7.999630179256201e-05,
+            -0.011468571610748768,
+            -0.0461457334458828,
+            0.00017471947649028152,
+            -0.04219469428062439,
+            0.003749340772628784,
+            -0.0007229121401906013,
+            -0.021534675732254982,
+            -0.022077566012740135,
+            0.01723678782582283,
+            -0.026858022436499596,
+            -0.022771259769797325,
+            0.006857768632471561,
+            0.0017804185627028346,
+            0.010020862333476543,
+            4.367986912257038e-05,
+            0.030055047944188118,
+            -0.043521761894226074,
+            -0.023133188486099243,
+            0.02761203795671463,
+            -0.022786341607570648,
+            -0.002165908692404628,
+            -0.04219469428062439,
+            -0.007347878534346819,
+            -0.040053289383649826,
+            0.020614776760339737,
+            0.02438485249876976,
+            -0.006861538626253605,
+            -0.0181416068226099,
+            0.026933424174785614,
+            -0.0037248353473842144,
+            -0.05555584281682968,
+            0.01296906266361475,
+            0.0052894167602062225,
+            0.022891903296113014,
+            -0.008279087021946907,
+            -0.024475334212183952,
+            -0.040626343339681625,
+            0.004689974710345268,
+            -0.01695026271045208,
+            -0.04919195547699928,
+            0.0204639732837677,
+            -0.032211530953645706,
+            -0.016573255881667137,
+            -0.007834218442440033,
+            0.0031461287289857864,
+            -0.002904843771830201,
+            0.019921083003282547,
+            0.001642810762859881,
+            -0.009847438894212246,
+            -0.03501646965742111,
+            -0.021474353969097137,
+            -0.03365924209356308,
+            -0.021745799109339714,
+            0.006529772188514471,
+            0.005892629269510508,
+            -0.014311209321022034,
+            0.04267726466059685,
+            -0.015487473458051682,
+            0.034775182604789734,
+            -0.0016041673952713609,
+            -0.016467692330479622,
+            0.05468118563294411,
+            0.012923821806907654,
+            -0.00233367714099586,
+            0.009937920607626438,
+            -0.00265036360360682,
+            0.0019472443964332342,
+            0.012637295760214329,
+            -0.004316737409681082,
+            0.031638480722904205,
+            -0.0005683390190824866,
+            0.009953001514077187,
+            -0.02708422765135765,
+            -0.013104785233736038,
+            -0.011574134230613708,
+            -0.040173932909965515,
+            0.02875814214348793,
+            -0.008037802763283253,
+            -0.002033956116065383,
+            0.018126526847481728,
+            0.012667456641793251,
+            -0.010948301292955875,
+            -0.03374972194433212,
+            -0.015291429124772549,
+            -0.00853545218706131,
+            -0.007879459299147129,
+            -0.012275367975234985,
+            -0.04457738250494003,
+            0.008550533093512058,
+            -0.0535954050719738,
+            -0.01996632292866707,
+            -0.05443990230560303,
+            -0.008867219090461731,
+            0.03038681484758854,
+            -4.72732208436355e-05,
+            0.012916280888020992,
+            0.01054113358259201,
+            0.0054062893614172935,
+            -0.03646417707204819,
+            -0.013677836395800114,
+            0.024897582828998566,
+            0.02136879228055477,
+            -0.01472591795027256,
+            -0.038092851638793945,
+            0.017432833090424538,
+            -0.01666373759508133,
+            -0.047925207763910294,
+            -0.011815418489277363,
+            0.011430870741605759,
+            -0.012554354034364223,
+            -0.060803789645433426,
+            -0.01764395646750927,
+            -0.018563855439424515,
+            -0.008610853925347328,
+            0.028893863782286644,
+            0.020403653383255005,
+            -0.004795536864548922,
+            -0.03679594397544861,
+            0.0018567625666037202,
+            0.015653356909751892,
+            0.006435520015656948,
+            0.032000407576560974,
+            -0.008550533093512058,
+            0.03803252801299095,
+            0.011355469934642315,
+            0.02589288353919983,
+            0.014213187620043755,
+            0.0016013398999348283,
+            0.0071970755234360695,
+            -0.01568351686000824,
+            -0.01362505555152893,
+            -0.011174505576491356,
+            -0.028320813551545143,
+            -0.009349789470434189,
+            -0.018292410299181938,
+            0.02124815061688423,
+            -0.0316988006234169,
+            0.04141051694750786,
+            0.01854877546429634,
+            -0.0007818195736035705,
+            0.010412951000034809,
+            0.001439226558431983,
+            0.00450524128973484,
+            0.007555232848972082,
+            -0.0016211328329518437,
+            0.025018226355314255,
+            0.023721320554614067,
+            -0.034051328897476196,
+            -0.013806018978357315,
+            0.014235807582736015,
+            -0.010103804059326649,
+            0.016889940947294235,
+            -0.005165004637092352,
+            -0.005662654526531696,
+            0.03160832077264786,
+            -0.024852342903614044,
+            0.02103702537715435,
+            0.003636238630861044,
+            -0.014243348501622677,
+            -0.02495790459215641,
+            0.0059944214299321175,
+            -0.023540355265140533,
+            0.04406465217471123,
+            -0.03531807288527489,
+            0.02756679803133011,
+            0.004497700836509466,
+            -0.015351750887930393,
+            0.03059793822467327,
+            -0.011166965588927269,
+            -0.017342351377010345,
+            -0.003223415231332183,
+            0.030160609632730484,
+            -0.010827658697962761,
+            -0.023600677028298378,
+            0.018699578940868378,
+            -0.007121674250811338,
+            0.027219951152801514,
+            -0.01413024589419365,
+            -0.0025466864462941885,
+            0.03332747519016266,
+            0.006303567439317703,
+            -0.0016456382581964135,
+            0.009266847744584084,
+            0.012086864560842514,
+            0.031186070293188095,
+            0.02969312109053135,
+            -0.025274591520428658,
+            -0.013361150398850441,
+            0.04032473638653755,
+            0.010978462174534798,
+            -0.002395883435383439,
+            0.012184886261820793,
+            -0.0032498056534677744,
+            -0.013594894669950008,
+            0.002346872352063656,
+            0.0007521302322857082,
+            -0.012109484523534775,
+            0.006420440040528774,
+            -0.028154930099844933,
+            -0.0017398901982232928,
+            0.01883530057966709,
+            0.013519493862986565,
+            -0.005915249697864056,
+            -0.013557193800807,
+            0.01525372825562954,
+            -0.01984568126499653,
+            0.024158649146556854,
+            -0.013821099884808064,
+            0.006582553032785654,
+            0.01425842847675085,
+            -0.010239526629447937,
+            -0.03281474485993385,
+            0.00847513135522604,
+            -3.4195771149825305e-05,
+            -0.009493052028119564,
+            0.003455274971202016,
+            -0.009794658049941063,
+            -0.008950160816311836,
+            -0.0032611158676445484,
+            0.011513812467455864,
+            -0.02940659411251545,
+            -0.0002133627567673102,
+            0.027762841433286667,
+            0.024988064542412758,
+            -0.013542113825678825,
+            -0.005741826258599758,
+            -0.010759797878563404,
+            0.009681555442512035,
+            -0.004282806534320116,
+            -0.009583533741533756,
+            0.027913644909858704,
+            0.02185136266052723,
+            -0.03212105110287666,
+            0.006420440040528774,
+            -0.005485461093485355,
+            0.0019547846168279648,
+            0.004999121185392141,
+            0.015238648280501366,
+            -0.018729738891124725,
+            0.00609621312469244,
+            -0.006454370450228453,
+            0.013994523324072361,
+            -0.014371531084179878,
+            0.023706238716840744,
+            0.017915401607751846,
+            0.009349789470434189,
+            -0.01482393965125084,
+            -0.006503381300717592,
+            0.010390330106019974,
+            -0.001478812424466014,
+            -0.010556213557720184,
+            0.035800643265247345,
+            0.00704627251252532,
+            -0.0011885165004059672,
+            0.01562319602817297,
+            0.028969265520572662,
+            0.02316334843635559,
+            -0.003754995996132493,
+            0.011853119358420372,
+            0.06774073094129562,
+            -0.018247168511152267,
+            0.005658884532749653,
+            0.0005353508167900145,
+            0.020041724666953087,
+            0.006130144000053406,
+            -0.032000407576560974,
+            -0.034051328897476196,
+            0.0001823774364311248,
+            -0.026073846966028214,
+            -0.04735215753316879,
+            -0.014047304168343544,
+            -0.01809636689722538,
+            -0.03172896057367325,
+            -0.0042639560997486115,
+            -0.00963631458580494,
+            0.005787067115306854,
+            0.023600677028298378,
+            0.004113153088837862,
+            -0.027385834604501724,
+            -0.016678817570209503,
+            0.02699374593794346,
+            0.025274591520428658,
+            0.03736899420619011,
+            -0.005681504961103201,
+            0.011174505576491356,
+            0.022529976442456245,
+            0.002050921553745866,
+            0.00927438773214817,
+            -0.004026441369205713,
+            -0.008052882738411427,
+            0.027174709364771843,
+            0.02703898772597313,
+            0.029270872473716736,
+            -0.0011461031390354037,
+            -0.034775182604789734,
+            -0.015298969112336636,
+            -0.018081285059452057,
+            -0.0006102810730226338,
+            0.008693795651197433,
+            -0.009033102542161942,
+            0.018352732062339783,
+            -0.0004740870790556073,
+            0.018518613651394844,
+            0.025591278448700905,
+            0.0022865512873977423,
+            -0.022394252941012383,
+            0.010495892725884914,
+            -0.0009608981781639159,
+            0.0071669151075184345,
+            -0.029843922704458237,
+            -0.04125971347093582,
+            -0.056430500000715256,
+            0.007068892940878868,
+            0.03254329785704613,
+            -0.042375657707452774,
+            0.01609068550169468,
+            0.04741247743368149,
+            -0.009018022567033768,
+            -0.023676078766584396,
+            -0.015864480286836624,
+            0.005410059355199337,
+            -0.022741099819540977,
+            -0.018307490274310112,
+            0.037067390978336334,
+            -0.015502553433179855,
+            -0.017870161682367325,
+            0.02079574018716812,
+            -0.02295222505927086,
+            0.02041873335838318,
+            -0.02818509005010128,
+            -0.002237540204077959,
+            -0.03311634808778763,
+            0.04014377295970917,
+            -0.012260288000106812,
+            0.00731017766520381,
+            -0.013097245246171951,
+            0.044456738978624344,
+            -0.008083043619990349,
+            0.017101066187024117,
+            0.0012997337616980076,
+            -0.001241297577507794,
+            -0.00830924790352583,
+            0.004893559031188488,
+            0.02838113345205784,
+            -0.017598716542124748,
+            0.0069030094891786575,
+            0.011264988221228123,
+            -0.03136703372001648,
+            -0.0187448188662529,
+            0.0037870416417717934,
+            0.021263230592012405,
+            0.015314050018787384,
+            0.04394400864839554,
+            0.028562096878886223,
+            -0.009930380620062351,
+            -0.008346948772668839,
+            0.009756957180798054,
+            -0.031276553869247437,
+            0.03709755092859268,
+            -0.013579814694821835,
+            0.031759120523929596,
+            0.028622418642044067,
+            -0.011264988221228123,
+            -0.001501432852819562,
+            -0.0027540407609194517,
+            0.02058461681008339,
+            0.03803252801299095,
+            0.007528842426836491,
+            0.005142384208738804,
+            0.02560635842382908,
+            0.010910600423812866,
+            -0.02626989036798477,
+            0.01826224848628044,
+            -0.004509011283516884,
+            -0.004780456889420748,
+            0.025259511545300484,
+            0.014265968464314938,
+            0.01409254502505064,
+            0.0017427176935598254,
+            0.028848623856902122,
+            -0.019875841215252876,
+            0.010118884965777397,
+            0.002904843771830201,
+            -0.012697616592049599,
+            0.015314050018787384,
+            0.0068276082165539265,
+            0.021263230592012405,
+            -0.004618343431502581,
+            0.02985900454223156,
+            0.01768919825553894,
+            0.01649785414338112,
+            0.0386357419192791,
+            0.006024581845849752,
+            -0.049041152000427246,
+            0.012049163691699505,
+            0.017176467925310135,
+            -0.0031423585023730993,
+            -0.021655317395925522,
+            -0.004041521809995174,
+            -0.002746500540524721,
+            0.007924700155854225,
+            0.0039057990070432425,
+            -0.01568351686000824,
+            -0.017824919894337654,
+            -0.004324277397245169,
+            0.010707016102969646,
+            0.021308470517396927,
+            0.010164125822484493,
+            0.005930330138653517,
+            0.006375198718160391,
+            0.040505699813365936,
+            0.02708422765135765,
+            -0.02495790459215641,
+            0.033387795090675354,
+            0.011543973349034786,
+            -0.015110465697944164,
+            -0.015223568305373192,
+            -0.022514894604682922,
+            0.0028690281324088573,
+            -0.02838113345205784,
+            -0.004659814294427633,
+            0.01883530057966709,
+            0.02209264598786831,
+            -0.013542113825678825,
+            -0.002821902045980096,
+            0.024701539427042007,
+            0.0009255537297576666,
+            -0.011363009922206402,
+            -0.021459273993968964,
+            0.011016163043677807,
+            0.0067785969004035,
+            -4.6360157284652814e-05,
+            0.008821978233754635,
+            0.04922211542725563,
+            -0.0055834827944636345,
+            0.027958884835243225,
+            0.049734845757484436,
+            0.0027917413972318172,
+            -0.02871290035545826,
+            0.02225852943956852,
+            0.007359188981354237,
+            -0.004169704392552376,
+            0.0221831277012825,
+            -0.005225325468927622,
+            -0.04819665476679802,
+            0.022861741483211517,
+            -0.028109688311815262,
+            -0.006005731411278248,
+            0.009289467707276344,
+            0.011385629884898663,
+            -0.002250735415145755,
+            -0.026692138984799385,
+            0.036253053694963455,
+            -0.006620253901928663,
+            0.015223568305373192,
+            -0.023374471813440323,
+            -0.027008825913071632,
+            -0.014974743127822876,
+            -0.007687185425311327,
+            -0.009749417193233967,
+            0.0031348182819783688,
+            -0.00901048257946968,
+            0.016241488978266716,
+            -0.03619273379445076,
+            0.003183829365298152,
+            0.012418631464242935,
+            0.00450524128973484,
+            0.04762360453605652,
+            0.00959861371666193,
+            0.00757785327732563,
+            -0.005915249697864056,
+            -0.012162266299128532,
+            0.023540355265140533,
+            0.010933221317827702,
+            0.03516726940870285,
+            -0.004399679135531187,
+            0.007223465945571661,
+            0.030447136610746384,
+            0.024927744641900063,
+            0.02201724611222744,
+            -0.016844701021909714,
+            -0.004181014373898506,
+            -0.00476914644241333,
+            -0.014590195380151272,
+            0.011822959408164024,
+            -0.005805917549878359,
+            0.008987861685454845,
+            0.012380930595099926,
+            0.007434590253978968,
+            0.011581674218177795,
+            0.06249278411269188,
+            0.015879560261964798,
+            -0.00888229999691248,
+            -0.026481015607714653,
+            -0.0032498056534677744,
+            0.007657025009393692,
+            0.00959861371666193,
+            0.0019736348185688257,
+            -0.009794658049941063,
+            0.014265968464314938,
+            0.00824892707169056,
+            -0.03402116894721985,
+            0.002678639255464077,
+            -0.004015131387859583,
+            0.006130144000053406,
+            0.01492950227111578,
+            -0.0403548963367939,
+            -0.00259381253272295,
+            -0.020720338448882103,
+            -0.004799306858330965,
+            0.022620458155870438,
+            -0.010412951000034809,
+            0.005979340989142656,
+            0.0016503508668392897,
+            -0.028803382068872452,
+            0.003958580084145069,
+            -0.0054364497773349285,
+            0.02777792140841484,
+            -0.028079528361558914,
+            -0.008821978233754635,
+            -0.013677836395800114,
+            -0.007600473705679178,
+            -0.017417753115296364,
+            0.029180390760302544,
+            -0.019589316099882126,
+            -0.002714454894885421,
+            -0.031155910342931747,
+            -0.014884261414408684,
+            -0.01476361881941557,
+            0.026375453919172287,
+            -0.01308970432728529,
+            -0.04481866583228111,
+            0.008482671342790127,
+            -0.012441251426935196,
+            0.00222057499922812,
+            -0.026194490492343903,
+            0.0030914624221622944,
+            0.028652578592300415,
+            0.02058461681008339,
+            0.006507151760160923,
+            -0.028411295264959335,
+            -0.010412951000034809,
+            0.01817176677286625,
+            0.04620605334639549,
+            -0.020765580236911774,
+            0.0128257991746068,
+            -0.012252748012542725,
+            0.01069947611540556,
+            -0.044004328548908234,
+            -0.0042337956838309765,
+            0.001988715259358287,
+            0.021821200847625732,
+            -0.015412071719765663,
+            0.002610777737572789,
+            0.05253978073596954,
+            0.017794759944081306,
+            -0.05163496360182762,
+            0.01866941712796688,
+            -0.012019002810120583,
+            0.034533899277448654,
+            0.05534471943974495,
+            -0.004535401705652475,
+            0.006936940364539623,
+            0.020765580236911774,
+            -0.022032326087355614,
+            0.010450650937855244,
+            0.014876720495522022,
+            -0.00576821668073535,
+            -0.02259029634296894,
+            -0.013579814694821835,
+            0.021640237420797348,
+            0.012592054903507233,
+            -0.012305528856813908,
+            0.03872622177004814,
+            -0.0007318660500459373,
+            0.029587557539343834,
+            -0.01678437925875187,
+            -0.03166864067316055,
+            -0.002508985809981823,
+            -0.018081285059452057,
+            0.025274591520428658,
+            0.010390330106019974,
+            0.0015985122881829739,
+            0.006699425633996725,
+            -0.035921286791563034,
+            0.01857893541455269,
+            0.02503330633044243,
+            0.0003843121521640569,
+            0.001393043203279376,
+            -0.02541031315922737,
+            0.02168547920882702,
+            -0.004761606454849243,
+            0.018186848610639572,
+            0.022605376318097115,
+            -0.0027955116238445044,
+            -0.019167067483067513,
+            0.005836077965795994,
+            0.0012083094334229827,
+            0.04346143826842308,
+            -0.001625845325179398,
+            0.0026239731814712286,
+            -0.01826224848628044,
+            -0.0032724260818213224,
+            -0.019001184031367302,
+            0.02572700008749962,
+            -0.005545781925320625,
+            -0.017432833090424538,
+            -0.018352732062339783,
+            0.03136703372001648,
+            -0.0007634404464624822,
+            -0.032332174479961395,
+            -0.003114083083346486,
+            -0.012433711439371109,
+            -0.0018379121320322156,
+            0.018639257177710533,
+            -0.03157815709710121,
+            -0.017251869663596153,
+            -0.0017493154155090451,
+            0.022077566012740135,
+            0.026948504149913788,
+            0.01883530057966709,
+            -0.0024712849408388138,
+            -0.0001750729134073481,
+            0.011076483875513077,
+            -0.0004952937597408891,
+            0.011506272479891777,
+            0.01299168262630701,
+            -0.007370498962700367,
+            0.03287506476044655,
+            -0.006367658730596304,
+            -0.0036173881962895393,
+            0.023887204006314278,
+            -0.01063915528357029,
+            -0.008731496520340443,
+            0.018654337152838707,
+            -0.018352732062339783,
+            -0.015502553433179855,
+            -0.007804058026522398,
+            0.005662654526531696,
+            -0.004886019043624401,
+            -0.0015080304583534598,
+            3.1014766136649996e-05,
+            0.03724835440516472,
+            -0.027265191078186035,
+            0.03178928419947624,
+            0.043220154941082,
+            -0.009847438894212246,
+            -0.02863749861717224,
+            -0.0016013398999348283,
+            -0.003496745601296425,
+            0.02455073595046997,
+            0.006631563883274794,
+            -0.0006592920981347561,
+            -0.04195340722799301,
+            0.015954962000250816,
+            -0.016226409003138542,
+            -0.011679695919156075,
+            -0.006024581845849752,
+            -0.004791766870766878,
+            0.009410110302269459,
+            0.028154930099844933,
+            -0.02801920659840107,
+            0.01190590113401413,
+            0.012418631464242935,
+            0.024083247408270836,
+            -0.005342198070138693,
+            -0.01409254502505064,
+            -0.01103124301880598,
+            -0.011287608183920383,
+            0.012531733140349388,
+            -0.0164224524050951,
+            -0.005319577641785145,
+            0.00519893504679203,
+            -0.005327117629349232,
+            0.01100862305611372,
+            -0.016920102760195732,
+            -0.06375952810049057,
+            -0.011114184744656086,
+            0.03667530044913292,
+            -0.014801319688558578,
+            -0.018880542367696762,
+            0.03712771087884903,
+            0.009545832872390747,
+            0.021640237420797348,
+            -0.023947523906826973,
+            0.0155628751963377,
+            -0.006767286919057369,
+            0.0029953254852443933,
+            -0.019785359501838684,
+            -0.010955841280519962,
+            0.00870133563876152,
+            -0.004429839551448822,
+            -0.004143313970416784,
+            -0.024249130859971046,
+            0.02617940865457058,
+            0.008264007046818733,
+            -0.02271093986928463,
+            0.007592933718115091,
+            0.008656094782054424,
+            0.005376128945499659,
+            -0.01604544371366501,
+            0.013572274707257748,
+            0.011822959408164024,
+            -0.01345917209982872,
+            0.02275617979466915,
+            -0.025063466280698776,
+            -0.004486390855163336,
+            0.013775859028100967,
+            0.030447136610746384,
+            0.014454471878707409,
+            -0.021670397371053696,
+            -0.02148943394422531,
+            0.011242367327213287,
+            0.0015909721842035651,
+            0.0034119191113859415,
+            0.000781348324380815,
+            -0.00709528336301446,
+            -0.04451705887913704,
+            0.01984568126499653,
+            -0.0020660017617046833,
+            -0.015035063959658146,
+            0.00412446353584528,
+            0.015909722074866295,
+            0.013609975576400757,
+            0.007487371563911438,
+            -0.006371428724378347,
+            0.01096338126808405,
+            0.033629078418016434,
+            -0.019001184031367302,
+            -0.022680778056383133,
+            -0.0063638887368142605,
+            -0.0010094379540532827,
+            0.01838289201259613,
+            -0.010360169224441051,
+            0.004395908676087856,
+            0.01809636689722538,
+            0.07220450043678284,
+            -0.014710837043821812,
+            0.00022785397595725954,
+            -0.03206072747707367,
+            -0.02340463362634182,
+            0.00108766695484519,
+            -0.0036569740623235703,
+            0.022198209539055824,
+            0.021293390542268753,
+            0.018066205084323883,
+            0.010691936127841473,
+            -0.007234776392579079,
+            0.002895418554544449,
+            0.025395233184099197,
+            -0.013353610411286354,
+            0.030809063464403152,
+            0.007909620180726051,
+            0.016316890716552734,
+            0.001337434514425695,
+            -0.015080304816365242,
+            0.012509113177657127,
+            0.033267151564359665,
+            -0.010857819579541683,
+            0.002940659411251545,
+            -0.004331817384809256,
+            0.0099454615265131,
+            -0.00028275573276914656,
+            -0.008158445358276367,
+            -0.009206525981426239,
+            -0.023057786747813225,
+            0.014115165919065475,
+            -0.014205647632479668,
+            -0.004735215567052364,
+            0.005263026338070631,
+            -0.016105765476822853,
+            0.062191177159547806,
+            -0.008769197389483452,
+            0.008324328809976578,
+            0.0077361962758004665,
+            -0.008497752249240875,
+            -0.005425139795988798,
+            0.008173525333404541,
+            0.005968030542135239,
+            0.011619375087320805,
+            -0.005149924196302891,
+            0.0037813864182680845,
+            -0.0024335843045264482,
+            -0.011189586482942104,
+            -0.00015857882681302726,
+            -0.023133188486099243,
+            0.016392290592193604,
+            0.006484531331807375,
+            -0.023615757003426552,
+            -0.0007064180099405348,
+            -0.0049425698816776276,
+            -0.00741196982562542,
+            0.002096162410452962,
+            -0.017251869663596153,
+            -0.0034119191113859415,
+            0.014612815342843533,
+            -0.004799306858330965,
+            -0.003898258786648512,
+            0.008075503632426262,
+            0.013127405196428299,
+            -0.008135824464261532,
+            -0.009183906018733978,
+            0.02156483568251133,
+            0.003148013725876808,
+            0.0034816653933376074,
+            0.006156534422188997,
+            -0.009726796299219131,
+            0.024988064542412758,
+            -0.01915198750793934,
+            -0.01435645017772913,
+            0.0050669824704527855,
+            0.010020862333476543,
+            0.019257549196481705,
+            -0.020328251644968987,
+            -0.003347827820107341,
+            -0.010940761305391788,
+            -0.008791817352175713,
+            0.022891903296113014,
+            -0.002870913129299879,
+            -0.023555437102913857,
+            -0.016482774168252945,
+            0.0025184110272675753,
+            -0.01833765022456646,
+            -0.005379898939281702,
+            -0.008339408785104752,
+            -0.005866238847374916,
+            -0.02103702537715435,
+            -0.018563855439424515,
+            0.0033384026028215885,
+            -0.029964566230773926,
+            -0.0036456636153161526,
+            -0.011046322993934155,
+            -0.0005975571111775935,
+            0.0076834154315292835,
+            -0.04578380659222603,
+            0.0005622126045636833,
+            -0.035227593034505844,
+            0.002356297569349408,
+            0.008656094782054424,
+            0.005304497200995684,
+            -0.013783399015665054,
+            0.020071886479854584,
+            0.017372511327266693,
+            -0.01981552131474018,
+            0.015713678672909737,
+            0.005334658082574606,
+            0.00018390902550891042,
+            0.0037983518559485674,
+            0.007672104984521866,
+            -0.007385579403489828,
+            0.0072272359393537045,
+            0.010186745785176754,
+            0.0028407524805516005,
+            -0.017025664448738098,
+            -0.012652375735342503,
+            -0.019167067483067513,
+            -0.008573153056204319,
+            0.006853998638689518,
+            0.021142587065696716,
+            -0.00959861371666193,
+            0.004901099018752575,
+            0.00671827606856823,
+            0.008625934831798077,
+            -0.016271648928523064,
+            0.01972503960132599,
+            -0.009900220669806004,
+            -0.0018906932091340423,
+            -0.018865462392568588,
+            -0.000623005093075335,
+            -0.00036051354254595935,
+            0.01372307725250721,
+            -0.009221606887876987,
+            -0.00444114999845624,
+            -0.0037531109992414713,
+            0.006337498314678669,
+            -0.010940761305391788,
+            -0.0013581699458882213,
+            0.00020865016267634928,
+            0.0267977025359869,
+            0.014363990165293217,
+            0.0008280029869638383,
+            -0.03966120257973671,
+            -0.027989046648144722,
+            0.025274591520428658,
+            0.00513484375551343,
+            0.012953981757164001,
+            -0.02409832738339901,
+            -0.018081285059452057,
+            0.0328449048101902,
+            0.0012808834435418248,
+            -0.000869473849888891,
+            -0.034986305981874466,
+            0.016603415831923485,
+            -0.015577955171465874,
+            0.02875814214348793,
+            0.025259511545300484,
+            -0.014801319688558578,
+            0.027702519670128822,
+            0.005391208920627832,
+            -0.022967305034399033,
+            0.008324328809976578,
+            0.006963330786675215,
+            0.019875841215252876,
+            -0.01515570655465126,
+            0.01601528376340866,
+            0.003886948572471738,
+            -0.017342351377010345,
+            -0.007502451539039612,
+            0.014205647632479668,
+            0.02756679803133011,
+            0.005802147556096315,
+            0.005485461093485355,
+            0.001331779407337308,
+            -0.036494337022304535,
+            0.015638276934623718,
+            -0.0019415892893448472,
+            0.01345917209982872,
+            -0.01637721061706543,
+            0.004181014373898506,
+            0.007924700155854225,
+            -0.015170786529779434,
+            -0.013270668685436249,
+            -0.029346274212002754,
+            0.005519391503185034,
+            -0.009116044268012047,
+            0.0020245308987796307,
+            -0.0033723332453519106,
+            0.0071669151075184345,
+            0.020569536834955215,
+            -0.014333830215036869,
+            -0.0035646071191877127,
+            0.005112223327159882,
+            0.008407269604504108,
+            -0.001880325493402779,
+            0.002735190326347947,
+            0.002011335687711835,
+            -0.00020111000048927963,
+            0.02835097350180149,
+            -0.016000203788280487,
+            -0.013134945183992386,
+            0.004101843107491732,
+            -0.00937240943312645,
+            0.002397768432274461,
+            -0.00646568089723587,
+            0.009628774598240852,
+            -0.0328449048101902,
+            -0.0018595900619402528,
+            -0.014409231022000313,
+            -0.021881522610783577,
+            -0.0013072739820927382,
+            0.008927540853619576,
+            0.009485512040555477,
+            -0.02099178545176983,
+            0.008422350510954857,
+            0.020433813333511353,
+            0.007702265866100788,
+            0.00867871567606926,
+            0.02299746498465538,
+            0.027793001383543015,
+            0.010028403252363205,
+            -0.008814438246190548,
+            -0.016000203788280487,
+            0.019830601289868355,
+            -0.015909722074866295,
+            0.013308369554579258,
+            0.00810566358268261,
+            0.004361978266388178,
+            0.03002488799393177,
+            0.018518613651394844,
+            0.00933470856398344,
+            0.011792798526585102,
+            0.004591953009366989,
+            0.01425842847675085,
+            -0.0042111752554774284,
+            0.01482393965125084,
+            0.00023786825477145612,
+            -0.00845251139253378,
+            0.00014538357208948582,
+            -0.0051876250654459,
+            0.026601657271385193,
+            -0.011755097657442093,
+            -0.00017742921772878617,
+            -0.011355469934642315,
+            -0.027340592816472054,
+            -0.010571293532848358,
+            0.005477920640259981,
+            -0.024249130859971046,
+            0.004637193866074085,
+            0.004486390855163336,
+            -0.01296906266361475,
+            -0.016482774168252945,
+            -0.015306509099900723,
+            0.010925681330263615,
+            -0.0006616484024561942,
+            0.008919999934732914,
+            0.0057003553956747055,
+            -0.011023703031241894,
+            0.003400608664378524,
+            0.020162368193268776,
+            0.021640237420797348,
+            0.0011074598878622055,
+            0.02646593563258648,
+            0.0013864454813301563,
+            -0.010616534389555454,
+            -0.0071669151075184345,
+            0.0017643957398831844,
+            -0.007962401024997234,
+            -0.005168774630874395,
+            -0.009093424305319786,
+            -0.0035551819019019604,
+            0.020252849906682968,
+            -0.004799306858330965,
+            -0.015758918598294258,
+            0.0006687172572128475,
+            -0.01649785414338112,
+            -0.0038831785786896944,
+            -0.008972781710326672,
+            0.04044537618756294,
+            -0.021459273993968964,
+            -0.007547692861407995,
+            -0.02981376275420189,
+            0.0024769401643425226,
+            -0.0015928571810945868,
+            0.00629979744553566,
+            0.004927489440888166,
+            -0.03402116894721985,
+            -0.05658130347728729,
+            0.027355672791600227,
+            0.002610777737572789,
+            0.01386634074151516,
+            0.022982385009527206,
+            -0.001093322061933577,
+            -0.033960845321416855,
+            0.010887980461120605,
+            -0.008497752249240875,
+            -0.007969941012561321,
+            0.005247946362942457,
+            -0.002974590053781867,
+            0.0007238546386361122,
+            0.02136879228055477,
+            -0.011958681978285313,
+            -0.030824143439531326,
+            -0.013187726959586143,
+            -0.0007653255015611649,
+            0.01425842847675085,
+            -0.004267726093530655,
+            0.03758012130856514,
+            0.01604544371366501,
+            -0.02013220638036728,
+            0.034986305981874466,
+            0.007102823816239834,
+            0.016075605526566505,
+            0.007751276716589928,
+            0.012305528856813908,
+            0.02642069384455681,
+            0.02237917296588421,
+            -0.04065650328993797,
+            0.008301707915961742,
+            -0.01403222419321537,
+            0.0008421407546848059,
+            0.010028403252363205,
+            0.029587557539343834,
+            0.000784647127147764,
+            0.0027615807484835386,
+            0.0011649535736069083,
+            -0.008716416545212269,
+            -0.038213491439819336,
+            0.0007181995315477252,
+            0.021911682561039925,
+            0.02111242711544037,
+            0.03480534255504608,
+            0.012652375735342503,
+            0.008188605308532715,
+            0.029512157663702965,
+            -0.005545781925320625,
+            -0.003053761785849929,
+            -0.014778698794543743,
+            -0.021655317395925522,
+            -0.004049061797559261,
+            -0.008098123595118523,
+            0.012614674866199493,
+            -0.0045731025747954845,
+            0.002378917997702956,
+            0.02177596092224121,
+            0.018774978816509247,
+            -0.0017568555194884539,
+            -0.011822959408164024,
+            0.02304270677268505,
+            -0.0025297212414443493,
+            -0.016151007264852524,
+            -0.009364869445562363,
+            0.015434691682457924,
+            0.03782140463590622,
+            0.0018954058177769184,
+            0.002769120968878269,
+            0.015080304816365242,
+            -0.008218766190111637,
+            0.024113407358527184,
+            -0.02103702537715435,
+            -0.03365924209356308,
+            -0.02340463362634182,
+            0.03854525834321976,
+            0.012901200912892818,
+            -0.008369569666683674,
+            0.011513812467455864,
+            0.004878478590399027,
+            0.03549903631210327,
+            0.0010339433792978525,
+            -0.04113907366991043,
+            0.006318647880107164,
+            0.007321488112211227,
+            -0.004026441369205713,
+            -0.026827862486243248,
+            0.0010782418539747596,
+            -0.02964787930250168,
+            0.017055824398994446,
+            -0.0014712722040712833,
+            0.015374370850622654,
+            0.02115766704082489,
+            0.009379949420690536,
+            0.00980219803750515,
+            0.006665494758635759,
+            -0.0008567498298361897,
+            0.009922840632498264,
+            0.008565613068640232,
+            -0.006737126037478447,
+            0.007087743375450373,
+            0.015713678672909737,
+            0.00824892707169056,
+            0.02364591881632805,
+            0.028215250000357628,
+            0.018729738891124725,
+            -0.014801319688558578,
+            -0.018729738891124725,
+            0.005511851515620947,
+            0.03236233443021774,
+            0.012757938355207443,
+            -0.001835084636695683,
+            -0.03209088742733002,
+            0.011574134230613708,
+            0.030567778274416924,
+            0.021519595757126808,
+            -0.004705055151134729,
+            -0.011566594243049622,
+            -0.024535655975341797,
+            -0.02720486931502819,
+            0.007381809409707785,
+            0.0069030094891786575,
+            -0.008218766190111637,
+            -0.019287709146738052,
+            -0.01149873249232769,
+            -0.019574236124753952,
+            0.03190992400050163,
+            0.021293390542268753,
+            -0.00450524128973484,
+            -0.009591073729097843,
+            0.0011281953193247318,
+            -0.004331817384809256,
+            0.013934201560914516,
+            -0.0003169220290146768,
+            -0.01229798886924982,
+            0.01160429511219263,
+            -0.0139869824051857,
+            -0.009379949420690536,
+            -0.01661849580705166,
+            -0.028577178716659546,
+            -0.004968960303813219,
+            0.016362130641937256,
+            0.008784277364611626,
+            0.011672155931591988,
+            -0.017749518156051636,
+            -0.002957624848932028,
+            0.014446931891143322,
+            0.0067333560436964035,
+            -0.010782417841255665,
+            0.0011187701020389795,
+            -0.0009953001281246543,
+            0.009470431134104729,
+            -0.028170010074973106,
+            0.012124565429985523,
+            -0.02782316319644451,
+            -0.0009792773053050041,
+            0.013572274707257748,
+            -0.0014703297056257725,
+            0.0007945435936562717,
+            -0.015608116053044796,
+            -0.010556213557720184,
+            -0.003159323940053582,
+            -0.008671175688505173,
+            -0.005210245493799448,
+            -0.02390228398144245,
+            0.01609068550169468,
+            -0.005990650970488787,
+            0.011106644757091999,
+            -0.038092851638793945,
+            0.001858647563494742,
+            0.010797498747706413,
+            -0.012237667106091976,
+            -0.024927744641900063,
+            0.003643778618425131,
+            0.040988270193338394,
+            0.0017691082321107388,
+            0.006876619067043066,
+            0.0011640110751613975,
+            0.0008100951090455055,
+            -0.021172748878598213,
+            -0.00022054946748539805,
+            -0.009455351158976555,
+            -0.008731496520340443,
+            -0.0077135758474469185,
+            0.011182046495378017,
+            0.006567472591996193,
+            -0.007393119391053915,
+            0.02299746498465538,
+            -0.0036494338419288397,
+            -0.002514640800654888,
+            0.002282781060785055,
+            0.023178428411483765,
+            -0.00608490314334631,
+            0.02503330633044243,
+            0.006869079079478979,
+            0.06514691561460495,
+            -0.0035947677679359913,
+            -0.005949180107563734,
+            0.0032592308707535267,
+            -0.015819240361452103,
+            0.0011008622823283076,
+            0.011875740252435207,
+            -0.022198209539055824,
+            0.015027523972094059,
+            0.0014175486285239458,
+            -0.010179205797612667,
+            -0.012199967168271542,
+            -0.002771005965769291,
+            0.012486492283642292,
+            0.022047406062483788,
+            0.021338632330298424,
+            0.001983060035854578,
+            0.01609068550169468,
+            0.03172896057367325,
+            0.013225427828729153,
+            -0.011378089897334576,
+            0.0013402621261775494,
+            -0.00019215607608202845,
+            0.017598716542124748,
+            -0.007819138467311859,
+            0.019694877788424492,
+            -0.00296705006621778,
+            0.011883280239999294,
+            0.010118884965777397,
+            0.025561116635799408,
+            0.01895594410598278,
+            0.0010179205564782023,
+            0.00916882511228323,
+            0.008301707915961742,
+            0.017734438180923462,
+            -0.001773820840753615,
+            0.011197126470506191,
+            -0.007280017249286175,
+            0.0012592055136337876,
+            0.019544076174497604,
+            0.02041873335838318,
+            -0.024520576000213623,
+            0.003958580084145069,
+            -0.008218766190111637,
+            -0.0024053086526691914,
+            -0.0005197992431931198,
+            0.0064166695810854435,
+            0.010103804059326649,
+            -0.0061942352913320065,
+            -0.02847161516547203,
+            -0.006122604012489319,
+            -0.004912409465759993,
+            0.005368588492274284,
+            -0.02263553813099861,
+            -0.013519493862986565,
+            0.00188126799184829,
+            0.01106894388794899,
+            -0.007114133797585964,
+            -0.0011592984665185213,
+            0.002567421877756715,
+            0.021534675732254982,
+            0.011528893373906612,
+            -0.0068426881916821,
+            -0.008799358271062374,
+            -0.001767223235219717,
+            0.006963330786675215,
+            0.022605376318097115,
+            0.002514640800654888,
+            0.012712697498500347,
+            0.014597735367715359,
+            0.006518461741507053,
+            -0.009658935479819775,
+            0.01446201279759407,
+            0.0005636264104396105,
+            0.001902946038171649,
+            0.0005537299439311028,
+            -0.0008228191290982068,
+            0.004784226883202791,
+            0.00927438773214817,
+            0.01972503960132599,
+            0.0071970755234360695,
+            0.00015186337986961007,
+            0.0028765681199729443,
+            0.014100085012614727,
+            -0.016105765476822853,
+            -0.0037926966324448586,
+            -0.0012771133333444595,
+            0.012056703679263592,
+            0.007042502518743277,
+            -0.0021489434875547886,
+            0.004561792127788067,
+            0.0123960105702281,
+            0.006503381300717592,
+            -0.008791817352175713,
+            -0.016407372429966927,
+            0.005643804091960192,
+            -0.007449670694768429,
+            -0.009206525981426239,
+            -0.007717345841228962,
+            0.009553372859954834,
+            0.005681504961103201,
+            0.010307388380169868,
+            -0.019694877788424492,
+            0.007528842426836491,
+            -0.0021998395677655935,
+            -0.009930380620062351,
+            -0.003935959655791521,
+            0.010737176984548569,
+            0.019860761240124702,
+            -0.0017945562722161412,
+            0.011709856800734997,
+            -0.01606052555143833,
+            0.0024222740903496742,
+            0.009093424305319786,
+            -0.0027766611892729998,
+            0.008859679102897644,
+            0.015668436884880066,
+            -0.008625934831798077,
+            -0.03483550623059273,
+            -0.007721116300672293,
+            0.015298969112336636,
+            -0.003477895399555564,
+            0.011219747364521027,
+            -0.01658833585679531,
+            0.013270668685436249,
+            -0.017251869663596153,
+            -0.018639257177710533,
+            -0.012999222613871098,
+            -0.01776459999382496,
+            -0.010631615296006203,
+            -0.028079528361558914,
+            -0.01190590113401413,
+            0.019106745719909668,
+            0.0027615807484835386,
+            0.004486390855163336,
+            -0.00036286984686739743,
+            7.49302635085769e-05,
+            0.0012884235475212336,
+            0.02067509852349758,
+            -0.010428030975162983,
+            0.0032139900140464306,
+            0.015879560261964798,
+            -0.025229349732398987,
+            -0.0054138293489813805,
+            0.011815418489277363,
+            -0.017251869663596153,
+            0.008618393912911415,
+            -0.024234050884842873,
+            0.002989670494571328,
+            -0.01562319602817297,
+            0.018201928585767746,
+            -0.020147288218140602,
+            -0.013730617240071297,
+            0.010020862333476543,
+            -0.015065224841237068,
+            -0.013142486102879047,
+            -0.024927744641900063,
+            -5.4754465963924304e-05,
+            -0.006032121833413839,
+            -0.0070274220779538155,
+            -0.020388571545481682,
+            -0.0058775488287210464,
+            -0.007721116300672293,
+            0.0042564161121845245,
+            -0.012418631464242935,
+            0.03447357565164566,
+            0.012705156579613686,
+            0.01695026271045208,
+            -0.00990776065737009,
+            -0.027144549414515495,
+            0.004328047391027212,
+            0.03190992400050163,
+            0.013632595539093018,
+            0.007404429838061333,
+            0.007811598014086485,
+            -0.0012884235475212336,
+            -0.004659814294427633,
+            0.02197200432419777,
+            0.00021937131532467902,
+            0.013949282467365265,
+            0.004331817384809256,
+            -0.015200947411358356,
+            -0.0028859933372586966,
+            0.021006865426898003,
+            0.021610077470541,
+            -0.0015174556756392121,
+            0.0006545794894918799,
+            -0.0049425698816776276,
+            0.01366275642067194,
+            -0.008935080841183662,
+            -0.009493052028119564,
+            0.007242316380143166,
+            -0.01752331480383873,
+            -0.01817176677286625,
+            -0.0024618597235530615,
+            0.017719358205795288,
+            0.004245105665177107,
+            -0.005394978914409876,
+            0.003647548845037818,
+            0.014756078831851482,
+            0.019227389246225357,
+            0.006326187867671251,
+            -0.02194184437394142,
+            -0.0033384026028215885,
+            0.018473373726010323,
+            -0.013006762601435184,
+            -0.010013322345912457,
+            0.01143841166049242,
+            0.026737380772829056,
+            -0.0024618597235530615,
+            -0.055525682866573334,
+            -0.0018360271351411939,
+            0.012554354034364223,
+            -0.009327168576419353,
+            -0.040505699813365936,
+            -0.004607032984495163,
+            2.5757279217941687e-05,
+            -0.023057786747813225,
+            0.029104989022016525,
+            -0.00974187720566988,
+            -0.008648554794490337,
+            0.015321590006351471,
+            0.01981552131474018,
+            -0.010345089249312878,
+            -0.0054590702056884766,
+            0.009877599775791168,
+            -0.015344209969043732,
+            -0.02263553813099861,
+            0.005168774630874395,
+            -0.023103026673197746,
+            -0.02438485249876976,
+            -0.009817278943955898,
+            0.005285646766424179,
+            0.01924246922135353,
+            0.013806018978357315,
+            0.01809636689722538,
+            0.019348030909895897,
+            0.0058775488287210464,
+            -0.011830499395728111,
+            -0.015713678672909737,
+            0.008754117414355278,
+            -0.014688217081129551,
+            -0.0007342223543673754,
+            -0.001240355079062283,
+            0.008527912199497223,
+            0.006213085725903511,
+            0.013165106065571308,
+            -0.015834320336580276,
+            -0.020840981975197792,
+            -2.8039941753377207e-05,
+            -0.004177244380116463,
+            0.007272477261722088,
+            -0.015879560261964798,
+            -0.013549653813242912,
+            0.006522231735289097,
+            0.0022054945584386587,
+            -0.010081184096634388,
+            -0.0009868174092844129,
+            0.02964787930250168,
+            -0.015608116053044796,
+            -0.005820997525006533,
+            -0.025500794872641563,
+            0.0007535439799539745,
+            -0.011981301940977573,
+            0.0239324439316988,
+            0.009183906018733978,
+            -0.023510195314884186,
+            0.009515671990811825,
+            -0.027355672791600227,
+            0.017779679968953133,
+            -0.006887929514050484,
+            0.0034062638878822327,
+            0.01891070231795311,
+            0.008218766190111637,
+            0.02177596092224121,
+            0.011551513336598873,
+            -0.0007243258878588676,
+            -0.0010066103423014283,
+            0.03390052542090416,
+            -0.009930380620062351,
+            0.016347050666809082,
+            -0.0027201101183891296,
+            -0.008754117414355278,
+            0.010216906666755676,
+            -0.004079222679138184,
+            -0.007404429838061333,
+            0.011325309053063393,
+            0.01785508170723915,
+            0.002763465978205204,
+            0.007457210682332516,
+            -0.021987084299325943,
+            -0.005538241937756538,
+            -0.010058563202619553,
+            0.009078343398869038,
+            -0.000991530017927289,
+            0.008060422725975513,
+            0.0034703551791608334,
+            -0.020388571545481682,
+            -0.01833765022456646,
+            0.0028181320521980524,
+            -0.015608116053044796,
+            -0.006009501405060291,
+            -0.0075891632586717606,
+            -0.0065561626106500626,
+            0.015298969112336636,
+            -0.017327269539237022,
+            -0.004248876124620438,
+            0.0068502286449074745,
+            0.016724057495594025,
+            -0.006126374006271362,
+            -0.016151007264852524,
+            0.019830601289868355,
+            0.013798478990793228,
+            0.0322718508541584,
+            0.002786086406558752,
+            0.01589464209973812,
+            -0.01711614616215229,
+            -0.014914421364665031,
+            -0.024897582828998566,
+            0.018699578940868378,
+            -0.006458140444010496,
+            0.014507253654301167,
+            -0.01654309406876564,
+            -0.0030462215654551983,
+            -0.018654337152838707,
+            0.0002771005965769291,
+            -0.0007808770169503987,
+            -0.020162368193268776,
+            0.011958681978285313,
+            -0.0016795690171420574,
+            0.023419713601469994,
+            -0.01621132716536522,
+            -0.014514793641865253,
+            0.002884108340367675,
+            -0.021459273993968964,
+            0.008339408785104752,
+            -0.0031875993590801954,
+            -0.006597633473575115,
+            0.0005037764203734696,
+            0.00910850428044796,
+            -0.006642874330282211,
+            -0.03803252801299095,
+            -0.007076432928442955,
+            -0.004000050947070122,
+            0.006963330786675215,
+            -0.006243246141821146,
+            -0.01756855472922325,
+            -0.018081285059452057,
+            0.013489332981407642,
+            0.02646593563258648,
+            -0.005741826258599758,
+            0.007246086373925209,
+            0.01026968751102686,
+            0.01964963786303997,
+            -0.01654309406876564,
+            0.019544076174497604,
+            0.0015721217496320605,
+            0.00870133563876152,
+            0.003347827820107341,
+            0.008482671342790127,
+            0.013730617240071297,
+            0.0023826882243156433,
+            -0.00746475113555789,
+            -0.00976449716836214,
+            -0.006759746931493282,
+            0.006133913993835449,
+            0.016648657619953156,
+            -0.002486365381628275,
+            -0.01478623878210783,
+            0.00413577351719141,
+            -0.003489205613732338,
+            -0.02168547920882702,
+            -0.002601352520287037,
+            -0.012855960056185722,
+            -0.014529873616993427,
+            0.015909722074866295,
+            -0.013406391255557537,
+            -0.017749518156051636,
+            -0.00034331256756559014,
+            0.007117903791368008,
+            -0.006303567439317703,
+            -0.03103526681661606,
+            0.005839847959578037,
+            -0.007758816704154015,
+            0.030416974797844887,
+            -0.0065109217539429665,
+            0.008188605308532715,
+            0.005655114538967609,
+            0.011121724732220173,
+            0.0267977025359869,
+            -0.0002858189109247178,
+            0.015819240361452103,
+            0.008098123595118523,
+            -0.006936940364539623,
+            -0.003790811635553837,
+            0.026435773819684982,
+            -0.01505014393478632,
+            0.02124815061688423,
+            0.003585342550650239,
+            -0.012117025442421436,
+            0.017794759944081306,
+            -0.0037418007850646973,
+            0.014688217081129551,
+            0.0007851183763705194,
+            0.0030160609167069197,
+            0.00608490314334631,
+            0.0017625106265768409,
+            -0.022560136392712593,
+            -0.010850279591977596,
+            0.0024392392951995134,
+            -0.011770177632570267,
+            0.023660998791456223,
+            0.015434691682457924,
+            -0.007506221998482943,
+            0.016331970691680908,
+            -0.027838243171572685,
+            0.009206525981426239,
+            -0.013127405196428299,
+            0.011076483875513077,
+            0.0031875993590801954,
+            -0.004007590934634209,
+            -0.007596703711897135,
+            -0.013376230373978615,
+            -0.006835148204118013,
+            -0.00609621312469244,
+            0.013406391255557537,
+            0.03317667171359062,
+            0.008301707915961742,
+            -0.0060698227025568485,
+            -0.010691936127841473,
+            -0.003943499643355608,
+            0.004497700836509466,
+            0.011657075956463814,
+            0.0052064755000174046,
+            -0.00963631458580494,
+            -0.005745596252381802,
+            -0.007012341637164354,
+            -0.013760778121650219,
+            -0.007170685101300478,
+            0.009153745137155056,
+            0.004181014373898506,
+            0.006397819612175226,
+            0.0053836689330637455,
+            -0.0025485714431852102,
+            -0.0010848394595086575,
+            0.010714557021856308,
+            -0.004644733853638172,
+            -0.004614573437720537,
+            0.006820067763328552,
+            0.026164328679442406,
+            -0.0013110439758747816,
+            0.01456003449857235,
+            -0.0032875065226107836,
+            -0.002005680464208126,
+            -0.001410951022990048,
+            -0.00970417633652687,
+            -0.017553474754095078,
+            0.002520296024158597,
+            0.048015691339969635,
+            -0.01472591795027256,
+            -0.014379071071743965,
+            -0.004859628155827522,
+            0.03026617132127285,
+            -0.0014118935214355588,
+            -0.006024581845849752,
+            0.01764395646750927,
+            -0.007939780130982399,
+            0.014167946763336658,
+            0.008995401673018932,
+            -0.026043687015771866,
+            0.0014345140662044287,
+            -0.00726116681471467,
+            -0.0006875676335766912,
+            0.011649535968899727,
+            0.025108708068728447,
+            0.012184886261820793,
+            0.00275215576402843,
+            0.002882223343476653,
+            0.020479053258895874,
+            0.015200947411358356,
+            -0.00540251936763525,
+            0.002458089729771018,
+            0.0017087870510295033,
+            0.005274336785078049,
+            -0.009447811171412468,
+            -0.011219747364521027,
+            -0.017387591302394867,
+            0.00014950709010008723,
+            -0.002073541982099414,
+            0.0005749366246163845,
+            -0.022439494729042053,
+            -0.00841481052339077,
+            0.013466712087392807,
+            -0.01262221485376358,
+            -0.00963631458580494,
+            0.002147058490663767,
+            -0.0011687236838042736,
+            -0.005240405909717083,
+            0.0055307019501924515,
+            0.02405308559536934,
+            -0.007442130707204342,
+            -0.014967203140258789,
+            0.003630583407357335,
+            0.01100862305611372,
+            0.013994523324072361,
+            -0.0005881318938918412,
+            0.008399729616940022,
+            -0.005251716356724501,
+            0.011589214205741882,
+            0.010737176984548569,
+            -0.005304497200995684,
+            -0.010593914426863194,
+            -0.007178225088864565,
+            0.00746475113555789,
+            -0.015427151694893837,
+            -0.004316737409681082,
+            -2.169266554119531e-05,
+            -0.005319577641785145,
+            0.014997363090515137,
+            0.021293390542268753,
+            -0.00672958604991436,
+            0.020192528143525124,
+            -0.006284717004746199,
+            0.00540251936763525,
+            -0.0034929756075143814,
+            -0.0011517582461237907,
+            -0.008980321697890759,
+            -0.03540855646133423,
+            -0.011951141990721226,
+            0.027355672791600227,
+            -0.01039787009358406,
+            -0.008256467059254646,
+            0.023314151912927628,
+            0.003951040096580982,
+            0.009417650289833546,
+            -0.008633474819362164,
+            0.009515671990811825,
+            -0.009342248551547527,
+            0.007581623271107674,
+            0.004316737409681082,
+            0.0009962426265701652,
+            -0.004361978266388178,
+            -0.024731699377298355,
+            -0.005809687543660402,
+            -0.006220625713467598,
+            -0.02450549602508545,
+            -0.006872849073261023,
+            0.004120693542063236,
+            -0.008927540853619576,
+            -0.022695859894156456,
+            0.0007804057677276433,
+            -0.005949180107563734,
+            0.0007690955535508692,
+            -0.020433813333511353,
+            0.01039787009358406,
+            0.020569536834955215,
+            0.0346243791282177,
+            -0.03230201452970505,
+            0.00651469174772501,
+            -0.0012752283364534378,
+            0.010586374439299107,
+            -0.026888184249401093,
+            -0.0009274387848563492,
+            0.008060422725975513,
+            0.007442130707204342,
+            0.001478812424466014,
+            -0.0026579038240015507,
+            -0.003954810090363026,
+            0.025877803564071655,
+            -0.002356297569349408,
+            0.0025693068746477365,
+            -0.008837059140205383,
+            -0.013594894669950008,
+            -0.02405308559536934,
+            0.012486492283642292,
+            0.012350769713521004,
+            0.01299168262630701,
+            -0.010940761305391788,
+            -0.0008407270070165396,
+            -0.006032121833413839,
+            -0.003853017929941416,
+            0.007879459299147129,
+            0.006612713914364576,
+            -0.01202654279768467,
+            0.023057786747813225,
+            -0.011815418489277363,
+            -0.0030462215654551983,
+            0.0029972107149660587,
+            -0.0039057990070432425,
+            0.001874670386314392,
+            0.011747557669878006,
+            -0.011393170803785324,
+            -0.018684497103095055,
+            -0.009357329457998276,
+            0.013248047791421413,
+            0.010827658697962761,
+            -0.00870133563876152,
+            -0.034292612224817276,
+            -0.02156483568251133,
+            0.03172896057367325,
+            -0.01345917209982872,
+            0.010654235258698463,
+            -0.00619800528511405,
+            0.011453491635620594,
+            0.012343229725956917,
+            0.0015655241440981627,
+            -0.0034194590989500284,
+            -0.01063915528357029,
+            -0.011385629884898663,
+            -0.00843743048608303,
+            -0.0003246978158131242,
+            -0.0029764752835035324,
+            -0.014612815342843533,
+            -0.005926559679210186,
+            0.017085986211895943,
+            -0.0015589265385642648,
+            0.007668334990739822,
+            0.0027201101183891296,
+            0.013134945183992386,
+            0.006940710358321667,
+            7.58727837819606e-05,
+            0.01802096515893936,
+            -0.014363990165293217,
+            0.012750397436320782,
+            0.041651804000139236,
+            -0.0020999324042350054,
+            -0.006669264752417803,
+            -0.019227389246225357,
+            -0.007083973381668329,
+            -0.029104989022016525,
+            0.005357278510928154,
+            -0.016362130641937256,
+            -0.026375453919172287,
+            0.011641995050013065,
+            -0.016271648928523064,
+            -0.012117025442421436,
+            -0.023555437102913857,
+            0.008165985345840454,
+            -0.0038944887928664684,
+            0.011151885613799095,
+            0.0022752410732209682,
+            0.01216980628669262,
+            -0.009146205149590969,
+            -0.020494135096669197,
+            0.005353508051484823,
+            0.013798478990793228,
+            -0.002884108340367675,
+            -0.02515394799411297,
+            -0.012908740900456905,
+            0.005296957213431597,
+            -0.017719358205795288,
+            -0.010111344046890736,
+            -0.03966120257973671,
+            -0.017659036442637444,
+            0.0207505002617836,
+            0.010593914426863194,
+            -0.007909620180726051,
+            0.005545781925320625,
+            -0.0012167921522632241,
+            0.004591953009366989,
+            0.0030236011371016502,
+            0.0038096620701253414,
+            0.013180186972022057,
+            -0.002090507186949253,
+            0.004354438278824091,
+            0.00976449716836214,
+            0.013617515563964844,
+            0.006590093486011028,
+            -0.01349687296897173,
+            -0.019785359501838684,
+            0.011657075956463814,
+            -0.005063212476670742,
+            0.005342198070138693,
+            -0.0019161412492394447,
+            -0.00466735428199172,
+            -0.012674996629357338,
+            0.010111344046890736,
+            0.01308970432728529,
+            0.016030363738536835,
+            0.017779679968953133,
+            0.014115165919065475,
+            0.005919019691646099,
+            -0.015427151694893837,
+            0.001035828492604196,
+            -7.470199761883123e-06,
+            0.004018901381641626,
+            -0.007083973381668329,
+            0.006224395707249641,
+            0.003477895399555564,
+            0.010201826691627502,
+            0.011355469934642315,
+            -0.0016701437998563051,
+            0.012750397436320782,
+            -0.007332798093557358,
+            -0.003246035659685731,
+            0.0022563906386494637,
+            0.003076382214203477,
+            -0.009319628588855267,
+            0.009568453766405582,
+            0.0017172697698697448,
+            -0.004780456889420748,
+            0.010578833520412445,
+            -0.02450549602508545,
+            -0.008181065320968628,
+            -0.01276547834277153,
+            0.0011866315035149455,
+            -0.023480035364627838,
+            0.01262221485376358,
+            0.03377988189458847,
+            0.0009773923084139824,
+            -0.00225262064486742,
+            -0.00038784657954238355,
+            0.010744716972112656,
+            -0.01666373759508133,
+            -0.009493052028119564,
+            -0.0007412912673316896,
+            -0.01862417720258236,
+            -0.024580897763371468,
+            0.01833765022456646,
+            0.017839999869465828,
+            -0.013308369554579258,
+            -0.016241488978266716,
+            -0.002853947691619396,
+            -0.01833765022456646,
+            -0.010020862333476543,
+            -0.0019180262461304665,
+            -0.012984142638742924,
+            -0.019574236124753952,
+            -0.002957624848932028,
+            -0.0291200689971447,
+            0.0021150128450244665,
+            0.010096264071762562,
+            0.011385629884898663,
+            -0.010744716972112656,
+            0.0062394761480391026,
+            -0.01190590113401413,
+            -0.0033685630187392235,
+            0.006593863479793072,
+            0.01256189402192831,
+            0.03390052542090416,
+            -0.006789907347410917,
+            -0.010737176984548569,
+            0.01525372825562954,
+            0.007634404581040144,
+            0.002652248600497842,
+            0.018729738891124725,
+            0.0013921007048338652,
+            0.010616534389555454,
+            0.0018614751752465963,
+            0.015412071719765663,
+            0.01349687296897173,
+            -0.008935080841183662,
+            0.017734438180923462,
+            0.0116646159440279,
+            -0.0014986053574830294,
+            -0.019393272697925568,
+            0.014477092772722244,
+            0.010330009274184704,
+            0.013014303520321846,
+            0.0009905875194817781,
+            0.014914421364665031,
+            -0.012463872320950031,
+            -0.01768919825553894,
+            -0.01431874930858612,
+            0.00016482302453368902,
+            -0.006929399911314249,
+            0.010111344046890736,
+            -0.0006734298658557236,
+            -0.02405308559536934,
+            -0.0012714582262560725,
+            0.0035928827710449696,
+            -0.002141403267160058,
+            0.0018944633193314075,
+            -0.005342198070138693,
+            0.011973761953413486,
+            0.013263127766549587,
+            0.007871919311583042,
+            0.012750397436320782,
+            -0.008927540853619576,
+            0.009447811171412468,
+            -0.004312967415899038,
+            0.012840880081057549,
+            -0.0025108708068728447,
+            -0.0020999324042350054,
+            -3.84960112569388e-05,
+            0.0003195139579474926,
+            0.007110363803803921,
+            0.00963631458580494,
+            0.008874759078025818,
+            0.026088926941156387,
+            -0.012780558317899704,
+            0.012584514915943146,
+            0.009530752897262573,
+            -0.013700457289814949,
+            0.03133687376976013,
+            0.017025664448738098,
+            0.011920981109142303,
+            -0.017101066187024117,
+            0.010518512688577175,
+            0.023781640455126762,
+            0.024188809096813202,
+            -0.00870133563876152,
+            -0.005964260548353195,
+            -0.0003664042742457241,
+            0.022967305034399033,
+            0.015374370850622654,
+            -0.017508234828710556,
+            0.01519340742379427,
+            0.011174505576491356,
+            0.009183906018733978,
+            -0.015057684853672981,
+            0.040777143090963364,
+            -0.008686255663633347,
+            -0.017387591302394867,
+            -0.011687235906720161,
+            -0.014160406775772572,
+            -0.016392290592193604,
+            -0.005176314618438482,
+            -0.006537312176078558,
+            -0.009953001514077187,
+            0.0018482798477634788,
+            0.0016239603282883763,
+            -0.031216232106089592,
+            -0.003966120071709156,
+            -0.006314877886325121,
+            -0.013655216433107853,
+            -0.015065224841237068,
+            0.008482671342790127,
+            -0.010812578722834587,
+            0.0030575317796319723,
+            0.004482620861381292,
+            -0.020690178498625755,
+            0.01744791306555271,
+            0.023148268461227417,
+            0.0033968386705964804,
+            0.011249907314777374,
+            -0.00466735428199172,
+            0.0058775488287210464,
+            -0.0025448014494031668,
+            0.025395233184099197,
+            -0.007479831110686064,
+            0.010345089249312878,
+            -0.002599467523396015,
+            -0.01519340742379427,
+            -0.04545203968882561,
+            -0.0024147338699549437,
+            0.006865308620035648,
+            0.0181416068226099,
+            -0.01356473471969366,
+            0.007649484556168318,
+            0.013519493862986565,
+            -0.013338529504835606,
+            0.0050368220545351505,
+            0.023314151912927628,
+            -0.006152764428406954,
+            0.013044463470578194,
+            0.00953829288482666,
+            0.02304270677268505,
+            -0.011626915074884892,
+            -0.02414356917142868,
+            0.006013271398842335,
+            0.013707997277379036,
+            -0.005255486350506544,
+            0.017146306112408638,
+            0.012207507155835629,
+            -0.0139869824051857,
+            -0.004478850401937962,
+            -0.014514793641865253,
+            -0.010345089249312878,
+            0.012848420068621635,
+            -0.0047502960078418255,
+            0.018036045134067535,
+            -0.002073541982099414,
+            -0.004825697746127844,
+            -0.016724057495594025,
+            0.011792798526585102,
+            0.007849298417568207,
+            -0.007212155964225531,
+            -0.004037751816213131,
+            -0.02124815061688423,
+            -0.021519595757126808,
+            0.003251690650358796,
+            -0.024897582828998566,
+            0.02622465044260025,
+            -0.0005909594474360347,
+            0.03218137100338936,
+            0.000863818742800504,
+            -0.002763465978205204,
+            0.019378192722797394,
+            0.01707090437412262,
+            -0.013383770361542702,
+            0.0038794083520770073,
+            0.009862519800662994,
+            0.0032931615132838488,
+            -0.0008280029869638383,
+            -0.00904818344861269,
+            -0.0006696598138660192,
+            -0.0007083030650392175,
+            0.009236686863005161,
+            0.0010527938138693571,
+            0.0009330938919447362,
+            -0.008181065320968628,
+            -0.024882502853870392,
+            -0.006597633473575115,
+            0.010767337866127491,
+            0.005734285805374384,
+            -0.006337498314678669,
+            -0.012878580018877983,
+            0.007415739819407463,
+            -0.006529772188514471,
+            -0.01084273960441351,
+            -0.00873903650790453,
+            0.009779578074812889,
+            0.0020660017617046833,
+            -0.01026968751102686,
+            0.021006865426898003,
+            0.0021432882640510798,
+            -0.006910549942404032,
+            0.014379071071743965,
+            0.001902946038171649,
+            -0.017387591302394867,
+            -0.00996808148920536,
+            0.009847438894212246,
+            -0.01000578235834837,
+            0.002179104136303067,
+            0.010058563202619553,
+            -0.0009604269289411604,
+            0.0136853763833642,
+            0.003257345873862505,
+            -0.011958681978285313,
+            -0.012690076604485512,
+            -0.0282604917883873,
+            -0.00847513135522604,
+            -0.028245411813259125,
+            0.012509113177657127,
+            0.0029764752835035324,
+            0.0017945562722161412,
+            0.013504412956535816,
+            0.0018209468107670546,
+            -0.0207505002617836,
+            0.017779679968953133,
+            -0.01768919825553894,
+            0.011257447302341461,
+            0.004716365598142147,
+            0.007962401024997234,
+            -0.006797447334975004,
+            0.011302688159048557,
+            0.009787118062376976,
+            -0.02185136266052723,
+            0.018156686797738075,
+            0.02173071913421154,
+            -0.02070525847375393,
+            0.010201826691627502,
+            0.009900220669806004,
+            0.007864379324018955,
+            -0.0006555219879373908,
+            -0.013971902430057526,
+            -0.008505292236804962,
+            -0.017960643395781517,
+            0.003438309533521533,
+            0.003342172596603632,
+            -0.006360118743032217,
+            -0.005455300211906433,
+            -0.015027523972094059,
+            0.011038783006370068,
+            0.00530072720721364,
+            -0.021640237420797348,
+            -0.017266949638724327,
+            -0.0038605581503361464,
+            -0.01924246922135353,
+            0.004803077317774296,
+            0.027551718056201935,
+            -0.008346948772668839,
+            -0.015268809162080288,
+            0.024611057713627815,
+            0.02630005218088627,
+            0.009440271183848381,
+            -0.010986002162098885,
+            0.0015410187188535929,
+            -0.004855858162045479,
+            -0.011340389028191566,
+            0.005312037654221058,
+            -0.004723905585706234,
+            -0.010035943239927292,
+            0.008542993105947971,
+            -0.003732375567778945,
+            0.00858069397509098,
+            0.022349011152982712,
+            -0.006642874330282211,
+            0.004459999967366457,
+            0.009123584255576134,
+            -0.0012422400759533048,
+            0.01339885126799345,
+            -0.02577224187552929,
+            0.0108050387352705,
+            -0.025455554947257042,
+            -0.002605122746899724,
+            0.01086535956710577,
+            0.01919722743332386,
+            0.025470634922385216,
+            -0.00963631458580494,
+            0.0008741864003241062,
+            -0.0009552430710755289,
+            0.027958884835243225,
+            0.021142587065696716,
+            -0.00422625569626689,
+            -0.01678437925875187,
+            -0.014017143286764622,
+            -0.004248876124620438,
+            -0.002265815855935216,
+            0.016648657619953156,
+            -0.024203889071941376,
+            0.0044336095452308655,
+            -0.018473373726010323,
+            -0.002045266330242157,
+            -0.0013402621261775494,
+            0.014115165919065475,
+            0.015261268243193626,
+            0.002590042306110263,
+            -0.0019566696137189865,
+            0.019875841215252876,
+            0.005647574085742235,
+            -0.008196145296096802,
+            0.0042564161121845245,
+            0.005805917549878359,
+            -0.001035828492604196,
+            0.007698495872318745,
+            -0.05235881730914116,
+            0.008203686214983463,
+            0.01780983991920948,
+            0.010292308405041695,
+            0.005168774630874395,
+            0.001149873249232769,
+            -0.005640034098178148,
+            -0.015366830863058567,
+            -0.010887980461120605,
+            0.017749518156051636,
+            -0.015472392551600933,
+            0.0064128995873034,
+            0.00392087921500206,
+            -0.007826678454875946,
+            0.021293390542268753,
+            0.008075503632426262,
+            0.015035063959658146,
+            0.009651395492255688,
+            0.03047729656100273,
+            0.011423330754041672,
+            0.010827658697962761,
+            -0.005624953657388687,
+            0.00304810656234622,
+            0.0010999197838827968,
+            0.005621183663606644,
+            0.0196043960750103,
+            -0.019634557887911797,
+            -0.004821927286684513,
+            0.006066052708774805,
+            -0.0279739648103714,
+            -0.005851158406585455,
+            0.012320608831942081,
+            -0.010081184096634388,
+            -0.006722046062350273,
+            0.007121674250811338,
+            -0.0014543068828061223,
+            0.0004229554324410856,
+            0.01572875864803791,
+            0.0022073795553296804,
+            -0.013285748660564423,
+            0.01568351686000824,
+            -0.002940659411251545,
+            -0.03136703372001648,
+            -0.011740017682313919,
+            -0.011845579370856285,
+            -0.007155604660511017,
+            -0.005191395059227943,
+            -0.03091462515294552,
+            0.0008732439018785954,
+            -0.011280068196356297,
+            0.016799459233880043,
+            -0.0010509087005630136,
+            0.006853998638689518,
+            0.006752206478267908,
+            -0.010797498747706413,
+            -0.011657075956463814,
+            -0.01797572337090969,
+            -0.0011395055335015059,
+            -0.01372307725250721,
+            -0.0075891632586717606,
+            0.01752331480383873,
+            -0.021217988803982735,
+            -0.007321488112211227,
+            0.025546036660671234,
+            -0.011310229077935219,
+            -0.0027653509750962257,
+            0.006691885180771351,
+            0.0026484786067157984,
+            0.0027069146744906902,
+            0.002339332364499569,
+            0.009033102542161942,
+            -0.0008756002062000334,
+            0.002906728768721223,
+            -0.02005680650472641,
+            0.03287506476044655,
+            0.011355469934642315,
+            -0.0011395055335015059,
+            0.010691936127841473,
+            -0.02687310427427292,
+            -0.025425394997000694,
+            0.019257549196481705,
+            -0.0311257503926754,
+            -0.006111293565481901,
+            -0.01250157319009304,
+            -0.0009213123703375459,
+            -0.03218137100338936,
+            -0.025108708068728447,
+            -0.029195470735430717,
+            -0.040746983140707016,
+            0.008482671342790127,
+            -0.00225262064486742,
+            0.005519391503185034,
+            0.007773897144943476,
+            -0.025636518374085426,
+            0.013021843507885933,
+            -0.008625934831798077,
+            -0.0013242393033578992,
+            0.018986104056239128,
+            -0.007434590253978968,
+            0.004486390855163336,
+            0.017025664448738098,
+            -0.0005692815175279975,
+            -0.006620253901928663,
+            -0.014145325869321823,
+            -0.006808757781982422,
+            0.026767540723085403,
+            0.025455554947257042,
+            0.022484734654426575,
+            -0.003114083083346486,
+            0.00249202037230134,
+            -0.003898258786648512,
+            0.019619476050138474,
+            -0.018066205084323883,
+            -0.008331868797540665,
+            0.003496745601296425,
+            0.007204615511000156,
+            0.008897379972040653,
+            0.004241335671395063,
+            0.03655466064810753,
+            0.020644936710596085,
+            0.015140626579523087,
+            -0.011740017682313919,
+            -0.013308369554579258,
+            -0.01103124301880598,
+            -0.013738158158957958,
+            0.02421896904706955,
+            0.024490416049957275,
+            -0.006925629917532206,
+            -0.02304270677268505,
+            -0.011461031623184681,
+            -0.00904818344861269,
+            -0.02053937502205372,
+            0.004901099018752575,
+            0.02254505641758442,
+            0.009990702383220196,
+            0.014326289296150208,
+            -0.010608994401991367,
+            0.011468571610748768,
+            0.03299570828676224,
+            -0.012124565429985523,
+            0.006118833553045988,
+            -0.0026465936098247766,
+            -0.016362130641937256,
+            0.012742857448756695,
+            -0.015208487398922443,
+            -0.019016264006495476,
+            -0.00466735428199172,
+            -0.021881522610783577,
+            0.018126526847481728,
+            0.00035768598900176585,
+            0.003087692428380251,
+            -0.012818259187042713,
+            -0.02443009428679943,
+            0.0060170418582856655,
+            0.00046089180978015065,
+            0.003898258786648512,
+            -0.00619800528511405,
+            0.00571920583024621,
+            -0.01711614616215229,
+            0.003393068676814437,
+            -0.017719358205795288,
+            0.0219267625361681,
+            -0.010043483227491379,
+            -0.011936061084270477,
+            0.004886019043624401,
+            0.008098123595118523,
+            0.009666475467383862,
+            0.0017879586666822433,
+            -0.0028690281324088573,
+            0.007562772836536169,
+            -0.017327269539237022,
+            0.017085986211895943,
+            0.01006610319018364,
+            0.011400710791349411,
+            0.003739915555343032,
+            0.002853947691619396,
+            -0.012546814046800137,
+            0.02801920659840107,
+            -0.0008986919419839978,
+            -0.008135824464261532,
+            0.00841481052339077,
+            -0.002000025473535061,
+            0.011958681978285313,
+            0.013263127766549587,
+            -0.00798502191901207,
+            0.0006074535194784403,
+            0.003754995996132493,
+            0.008369569666683674,
+            0.015321590006351471,
+            -0.0040339813567698,
+            0.020267929881811142,
+            0.0063337283208966255,
+            -0.00022243449348025024,
+            0.012275367975234985,
+            0.0037022149190306664,
+            0.015019983984529972,
+            -0.023962603881955147,
+            -0.009206525981426239,
+            -0.023721320554614067,
+            -0.0016739139100536704,
+            -0.00741196982562542,
+            0.008754117414355278,
+            -0.0029727050568908453,
+            -0.011521353386342525,
+            0.005455300211906433,
+            0.025907963514328003,
+            0.023299071937799454,
+            -0.004267726093530655,
+            0.016648657619953156,
+            -0.006744666490703821,
+            -0.023389553651213646,
+            -0.010707016102969646,
+            -0.006624023895710707,
+            -0.03145751729607582,
+            0.022514894604682922,
+            -0.007004801649600267,
+            0.005195165053009987,
+            0.02242441289126873,
+            -0.007238546386361122,
+            -0.0033384026028215885,
+            0.01202654279768467,
+            -0.004316737409681082,
+            -0.026330212131142616,
+            0.00830924790352583,
+            -0.013285748660564423,
+            0.014575114473700523,
+            0.011996382847428322,
+            0.031638480722904205,
+            -0.013225427828729153,
+            -0.002260160632431507,
+            0.009410110302269459,
+            0.012388470582664013,
+            0.020267929881811142,
+            -0.022695859894156456,
+            -0.003210220020264387,
+            -0.001869957777671516,
+            -0.0014656170969828963,
+            -0.0029255792032927275,
+            -0.004490160848945379,
+            0.008784277364611626,
+            -0.01972503960132599,
+            -0.002989670494571328,
+            -0.005082062911242247,
+            0.003183829365298152,
+            0.003858672920614481,
+            0.0026805242523550987,
+            0.002616432961076498,
+            -0.012516653165221214,
+            -0.008037802763283253,
+            -0.006526002194732428,
+            -0.015819240361452103,
+            0.012388470582664013,
+            0.000850152166094631,
+            -0.01597004383802414,
+            0.00847513135522604,
+            0.005673964973539114,
+            0.0030914624221622944,
+            0.003460929961875081,
+            0.0034703551791608334,
+            0.05558600276708603,
+            -0.008814438246190548,
+            -0.007140524219721556,
+            -0.019348030909895897,
+            0.011166965588927269,
+            -0.013904041610658169,
+            -0.0017417751951143146,
+            0.0007804057677276433,
+            0.02194184437394142,
+            -0.028969265520572662,
+            -0.018307490274310112,
+            0.010782417841255665,
+            -0.011423330754041672,
+            -0.01127252820879221,
+            -0.0005843618419021368,
+            0.0030613020062446594,
+            -0.03172896057367325,
+            0.003935959655791521,
+            0.005572172813117504,
+            -0.004154623951762915,
+            0.01462035533040762,
+            -0.004637193866074085,
+            -0.005481690634042025,
+            0.02230377122759819,
+            0.015057684853672981,
+            -0.006235706154257059,
+            0.002520296024158597,
+            -0.00810566358268261,
+            0.014273508451879025,
+            -0.024611057713627815,
+            -0.013557193800807,
+            0.002192299347370863,
+            0.0026258581783622503,
+            -0.017221707850694656,
+            -0.007404429838061333,
+            0.0009679670911282301,
+            -0.003574032336473465,
+            0.017598716542124748,
+            -0.004659814294427633,
+            0.015223568305373192,
+            -0.0119662219658494,
+            0.03305602818727493,
+            -0.004128233529627323,
+            -0.023253830149769783,
+            0.006390279158949852,
+            0.0029990957118570805,
+            0.011363009922206402,
+            0.0004262542352080345,
+            0.01833765022456646,
+            0.015268809162080288,
+            0.0019161412492394447,
+            0.006872849073261023,
+            0.005293187219649553,
+            -0.01744791306555271,
+            0.011920981109142303,
+            -0.008158445358276367,
+            0.008791817352175713,
+            -0.0108050387352705,
+            0.0030839224345982075,
+            -0.008316787891089916,
+            0.009093424305319786,
+            -0.004595723003149033,
+            0.0025221810210496187,
+            -0.0001197588280774653,
+            -0.001998140476644039,
+            -0.0016692013014107943,
+            -0.017659036442637444,
+            -0.010186745785176754,
+            0.02058461681008339,
+            -0.02115766704082489,
+            -0.007132984232157469,
+            0.00661648390814662,
+            -0.025048386305570602,
+            -0.010887980461120605,
+            0.017327269539237022,
+            -0.015140626579523087,
+            -0.007276247255504131,
+            0.011709856800734997,
+            0.0024844801519066095,
+            -0.01323296781629324,
+            0.014205647632479668,
+            0.002084852196276188,
+            -0.036946747452020645,
+            0.011385629884898663,
+            -0.003988740500062704,
+            0.00010674028453649953,
+            -0.01054113358259201,
+            -0.047653764486312866,
+            0.006861538626253605,
+            0.0006300740060396492,
+            -0.013994523324072361,
+            0.004784226883202791,
+            0.009116044268012047,
+            0.0155628751963377,
+            -0.0032875065226107836,
+            0.009425190277397633,
+            0.014605275355279446,
+            0.0007756931590847671,
+            -3.722950350493193e-05,
+            0.019875841215252876,
+            0.0010103804524987936,
+            0.022273611277341843,
+            -0.011672155931591988,
+            -0.005168774630874395,
+            -0.023434793576598167,
+            0.010051023215055466,
+            -0.0020584615413099527,
+            -0.0005358220660127699,
+            -0.019167067483067513,
+            -0.0032139900140464306,
+            -0.015577955171465874,
+            0.02859225869178772,
+            -0.003751225769519806,
+            -0.0033685630187392235,
+            0.013082164339721203,
+            -0.003721065353602171,
+            -0.007238546386361122,
+            0.014431851916015148,
+            0.030703501775860786,
+            0.00422625569626689,
+            -0.003709755139425397,
+            0.007325258105993271,
+            -0.006469450891017914,
+            -0.012886120937764645,
+            0.0010895520681515336,
+            -0.0008284742361865938,
+            -0.01086535956710577,
+            -0.0039057990070432425,
+            0.0011451606405898929,
+            -0.011755097657442093,
+            -0.01515570655465126,
+            -0.014348910190165043,
+            0.016241488978266716,
+            -0.003223415231332183,
+            -0.00413577351719141,
+            -0.021187828853726387,
+            0.0029104987625032663,
+            -0.015065224841237068,
+            -0.004874708596616983,
+            0.014627896249294281,
+            -0.011898360215127468,
+            0.03377988189458847,
+            -0.006631563883274794,
+            0.0239324439316988,
+            -0.004297886975109577,
+            -0.0011366780381649733,
+            0.0004455758898984641,
+            -0.004482620861381292,
+            -0.014884261414408684,
+            -0.005968030542135239,
+            0.008565613068640232,
+            0.0028916485607624054,
+            0.007102823816239834,
+            0.01826224848628044,
+            -0.0015118005685508251,
+            0.045844126492738724,
+            -0.006737126037478447,
+            0.02572700008749962,
+            -0.008158445358276367,
+            -0.002590042306110263,
+            0.00386432814411819,
+            0.012531733140349388,
+            0.005157464183866978,
+            0.002361952792853117,
+            -0.022906983271241188,
+            -0.02177596092224121,
+            0.020569536834955215,
+            -0.0062470161356031895,
+            -0.0011479882523417473,
+            -0.027069147676229477,
+            0.00037135250749997795,
+            0.009078343398869038,
+            0.0009217836195603013,
+            -0.001966094831004739,
+            0.00671827606856823,
+            0.015306509099900723,
+            0.003590997541323304,
+            -0.014680677093565464,
+            -0.004935029894113541,
+            -0.026692138984799385,
+            -0.004297886975109577,
+            0.010744716972112656,
+            0.006314877886325121,
+            0.012222587130963802,
+            -0.013481792993843555,
+            -0.031547997146844864,
+            0.007796517573297024,
+            -0.009251766838133335,
+            -0.007023652084171772,
+            0.03139719367027283,
+            -0.02103702537715435,
+            -0.010247067548334599,
+            -0.02099178545176983,
+            -0.018654337152838707,
+            -0.008075503632426262,
+            0.020690178498625755,
+            -0.010624075308442116,
+            0.0046975151635706425,
+            -0.02500314638018608,
+            -0.00152405328117311,
+            0.017055824398994446,
+            0.02053937502205372,
+            0.006205545272678137,
+            0.014326289296150208
+          ],
+          "keyphrases": [
+            "Hash Layers",
+            "token ID",
+            "expert ID",
+            "load balance",
+            "Hashing based routing"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "215cb1c2-314b-443d-92c5-a97fb190f7c4",
+      "type": "child",
+      "source": {
+        "id": "b570d71e-4f83-486e-b70f-b0c6217db455",
+        "properties": {
+          "page_content": "Mixture-Of-Experts With Expert Choice Routing\n\nYanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon Google, Mountain View, CA, USA {yanqiz, taole, hanxiaol, dunan, huangyp, vzhao, adai, zhifengc, qvl, jlaudon}@google.com\n\nAbstract\n\nSparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choice method. Instead of letting tokens select the top-k experts, we have experts selecting the top-k tokens. As a result, each token can be routed to a variable number of experts and each expert can have a fixed bucket size. We systematically study pre-training speedups using the same computational resources of the Switch Transformer top-1 and GShard top-2 gating of prior work and find that our method improves training convergence time by more than 2×. For the same computational cost, our method demonstrates higher performance in fine-tuning 11 selected tasks in the GLUE and SuperGLUE benchmarks. For a smaller activation cost, our method outperforms the T5 dense model in 7 out of the 11 tasks.\n\n1 Introduction\n\nScaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27]. The final model quality has been found to have a power-law relationship with the amount of data, model size, and compute time [16, 20]. However, training efficiency, which is defined as the total amount of computation used to achieve superior model quality than the state of the art system [21], should receive greater attention as we increase our efforts towards green AI [29].\n\nSparsely gated mixture-of-experts [31] (MoE) provides an effective way to scale model capacity given a fixed computational cost, and has recently played an important role in increasing the training efficiency of large-scale language models [10, 21]. MoE operate by adopting a number of experts, each as a sub-network, and by activating only one or a few experts for each input token. A gating network must be chosen and optimized in order to route each token to the most suited expert(s). For example, recent work has implemented sparse routing via k-means clustering [12], linear assignment to maximize token-expert affinities [22], or hashing [8, 28]. Many of the prior work use a routing strategy concerning the token choice, where each token selects the best one or two experts.\n\nWe argue that the independent token choice of prior work often leads to an imbalanced load of experts, which causes training inefficiency and sub-optimal training of the model. In order to mitigate this\n\nissue, previous sparsely gated networks introduce additional auxiliary losses as regularization to prevent too many tokens being routed to a single expert, but the effectiveness is still limited. Recent approaches [8, 22, 28] explore alternative strategies for routing, but they focus on pre-training only and do not demonstrate performance gain on downstream tasks. Moreover, none of the previous methods consider allocating a variable number of experts to each token based on importance, which can be beneficial.\n\nWe propose a very simple yet effective routing method we are calling expert choice. Unlike conventional MoE where tokens select one or two top-scoring experts, our method lets each expert pick the top-k tokens. Our method guarantees perfect load balancing, allows a variable number of experts for each token, and achieves substantial gains in training efficiency and downstream performance as demonstrated in our experiments. Our major contributions include:\n\nWe identify common pitfalls in conventional MoE such as load imbalance as described in Section 3.1. We then propose a heterogeneous, expert choice method to provide a fluid allocation of model parameters based on a learnt token-to-expert importance. This method intrinsically guarantees load balance without imposing an auxiliary loss.\n\nWe show our method provides over 2× faster training convergence in a 8B/64E (8 billion activated parameters, 64 experts) model, compared to the top-1 and top-2 gating counterparts in Switch Transformer [10] and GShard [21].\n\nWe show our method demonstrates strong scaling when increasing the number of experts from 16 to 128, evaluated in training perplexity.\n\nWe show our method demonstrates strong performance on downstream tasks selected from GLUE and SuperGLUE at all the evaluated scales. More specifically, our 8B/64E model outperforms a T5 11B dense model in 7 out of 11 tasks evaluated.\n\n2 Related Work\n\nScaling: Various approaches have been proposed to scale up neural network capacity to improve performance. Recent works have successfully scaled models to billions of parameters via various forms of model parallelism [2, 21, 26, 27, 33]. Model parallelism [30] splits weights and tensors across multiple cores while pipeline parallelism [18, 24] splits different layers across devices with micro-batches pipelined to the different layers. To enable continued scaling of neural networks, improving model training and serving efficiency has become a critical research area.\n\nConditional Computation: Computation decisions can be made dynamically based on the input [23, 25]. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation, by activating certain parameters and computation on demand, on a per-example or per-token basis [3]. Conditional convolution layers [1] with task-specific gating has been used to combat catastrophic forgetting when a sequence of learning problems are optimized. The gating decisions may be binary or sparse and continuous, stochastic or deterministic.\n\nMixture of Experts: Sparsely-gated MoE [31] is the first model to demonstrate massive improvements in model capacity, training time, or model quality with gating. Switch Transformer [10] simplifies the gating by selecting only the top expert per token using a softmax over the hidden state and demonstrates better scaling than previous work. All the prior work requires an auxiliary loss to explicitly encourage balancing. This loss term has to be carefully weighted to not overwhelm the primary loss. However, auxiliary loss does not guarantee balancing and a hard capacity factor has to be imposed. As a result, many tokens can still be unprocessed by the MoE layer. Hard MoE [12] with a single decoding layer can be efficiently trained to good effect on large scale hashtag prediction tasks.\n\nBase Layers [22] formulate a linear assignment that maximizes token-expert affinities while ensuring each expert receives an equal number of tokens. Hash layers [8, 28] devise hashing techniques on input tokens. However, the evaluations are limited to pre-training perplexity. THOR [? ] randomly activates experts during training and inference and is trained with a consistency regularization loss.\n\nTHOR has demonstrated strong performance on translation tasks. Different from these prior works, our method is a learnt method that enables heterogeneous MoE and effectively improves downstream fine-tuning performance.\n\n3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n5 Conclusion\n\nWe propose a new routing method for sparsely activated mixture-of-experts (MoE) models. This method addresses load imbalance and under-utilization of experts in conventional MoE methods, and enables selecting different numbers of experts for each token. Our model demonstrates more than 2x training efficiency improvements when compared to the state-of-the-art GShard and Switch Transformer models, and also achieves strong gains when finetuning on 11 datasets in the GLUE and SuperGLUE benchmark.\n\n6 Limitations\n\nThe expert choice method might not immediately apply to auto-regressive text generation as our current implementation takes in the past and future tokens to perform the top-k selection. One possible solution is to collect a large batch of input sequences, dispatch tokens of the same sequence into separate groups, and perform expert choice routing for each group. Another scenario where the expert choice method does not immediately apply is when the batch size becomes very small during serving or inference. A global top-k can be selected instead and we can cap the number of times each expert or token gets selected. We leave these possible improvements for future work.\n\nAnother long-standing issue with MoE has been the large memory footprint. Even though computational cost can be reduced using sparsely gated networks, the total number of parameters increases linearly or sub-linearly with the number of experts. Increasing the number of experts requires reservation of a large number of hardware devices. Therefore, dynamic (used) power is saved while static (reserved) power is not. Power saving techniques such as the ability to put hardware devices into low power states while not in use [17] can help with reducing the reserved power requirements.\n\nReferences\n\n[1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pages 3930–3939. Computer Vision Foundation / IEEE, 2020.\n\n[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. In Advances in Neural Information Processing Systems.\n\n[3] Kyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning, 2014.\n\n[4] Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems, 2021.\n\n[5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.\n\nTransformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.\n\n[6] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 933–941. JMLR.org, 2017.\n\n[7] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixtureof-experts, 2021.\n\n[8] Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan.\n\nTricks for training sparse translation models, 2021.\n\n[9] Richard L Dykstra. An iterative procedure for obtaining i-projections onto the intersection of convex sets. The annals of Probability, pages 975–984, 1985.\n\n[10] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.\n\n[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. NAS-FPN: learning scalable feature pyramid architecture for object detection. In CVPR, pages 7036–7045. Computer Vision Foundation / IEEE, 2019.\n\n[12] Sam Gross, Marc'Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision, 2017.\n\n[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.\n\n[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 630–645, Cham, 2016. Springer International Publishing.\n\n[15] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2016.\n\n[16] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.\n\n[17] Ping Huang, Zuocheng Xing, Tianran Wang, Qiang Wei, Hongyan Wang, and Guitao Fu. A brief survey on power gating design. In 2010 10th IEEE International Conference on Solid-State and Integrated Circuit Technology, pages 788–790, 2010.\n\n[18] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 103–112, 2019.\n\n[19] Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David A. Patterson. A domain-specific supercomputer for training deep neural networks. Commun. ACM, 63(7):67–78, 2020.\n\n[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.\n\n[21] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.\n\n[22] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6265–6274. PMLR, 18–24 Jul 2021.\n\n[24] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. New York, NY, USA, 2019. Association for Computing Machinery.\n\n[25] Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, Cédric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. In ICLR. OpenReview.net, 2021.\n\n[26] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.\n\n[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.\n\n[28] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models, 2021.\n\n[30] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 10435–10444, Red Hook, NY, USA, 2018. Curran Associates Inc.\n\n[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E.\n\nHinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In ICLR (Poster). OpenReview.net, 2017.\n\n[32] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 10–15 Jul 2018. [23] Min Lin, Jie Fu, and Yoshua Bengio. Conditional computation for continual learning, 2019.\n\n[29] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.\n\n[33] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.\n\n[34] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc.\n\n[35] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.\n\nGLUE: A multi-task benchmark and analysis platform for natural language understanding.\n\nIn Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, November 2018. Association for Computational Linguistics.\n\n[36] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake A. Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen. GSPMD: general and scalable parallelization for ML computation graphs. CoRR, abs/2105.04663, 2021.\n\n7 Checklist\n\n(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Yes (b) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes (c) Did you discuss any potential negative societal impacts of your work? N/A. Not any. (d) Did you describe the limitations of your work? Yes (a) Did you include the code, data, and instructions needed to reproduce the main experimental results? Yes. We include details in the experiment setup to help reproduce the main results. (b) Did you specify all the training details? Yes (c) Did you report error bars? Yes (d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? Yes (a) If your work uses existing assets, did you cite the creators? Yes (b) Did you mention the license of the assets? No. The used dataset is not released yet. (c) Did you include any new assets either in the supplemental material or as a URL? No. The dataset is not released yet.\n\n(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? No. Not using persons' data. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes. The dataset does not contain any personally identifiable information or offensive content.\n\nA Comparison On Fine-Tuning With A Dense Model\n\nOur 8B MoE model achieves stronger pre-training perplexity than its dense counterpart. However, a better perplexity does not always directly translate to downstream performance as demonstrated in Section 4.4. To this end, we compare fine-tuning performance of the 8B dense model and MoE model in Table 1. As shown in the table, our MoE model using expert choice routing consistently outperforms the dense model across the 11 tasks in GLUE and SuperGLUE.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 1: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nB Capacity Factor\n\nWe evaluate the downstream task fine-tuning performance by varying the capacity factors. Note that a capacity factor of n indicates on average how many experts each token can be received. EC-CF2 is our baseline expert choice, which matches GShard top-2 gating computational footprint. EC-CF1, however, matches Switch Transformer top-1 gating computational footprint. EC-CF0.5 further verifies that an aggressively lowered capacity factor can provide strong enough performance, that almost matches the top-2 gating baseline.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Top-2 78.1 87.0 88.3 85.0 82.6 90.1 90.7 81.6 94.7 68.2 67.2 83.0±0.3 EC-CAP2 78.2 88.0 88.5 85.7 83.0 90.8 91.1 80.0 95.4 70.4 64.1 83.2±0.4 EC-CAP3 78.5 91.7 89.3 86.3 83.5 90.9 91.1 81.8 94.9 70.0 65.6 84.0±0.4 EC-CF2 79.1 89.6 89.3 86.8 84.3 91.3 91.2 81.1 95.2 68.1 68.0 84.0±0.2 EC-CF1 77.4 90.6 88.0 85.5 83.6 90.3 91.2 79.8 95.3 66.5 64.9 83.0±0.2 EC-CF0.5 77.4 89.6 86.3 85.2 82.7 91.7 91.0 79.6 94.9 67.3 63.5 83.0 ±0.05 Hash Layers 76.1 85.2 86.7 83.4 82.5 90.0 90.3 75.7 94.0 67.4 63.3 81.3±1.0\n\nTable 2: Comparison between different routing methods in fine-tuning of 100M/64E models. We perform 3 independent fine-tuning runs for each method and report the average results. This gives more accurate difference between the variants of expert choice method, since they achieve close fine-tuning results. We do not report averaged results in other experiments.\n\nC Capped Expert Choice\n\nAs described in Section 4.5, the maximum number of experts each token is assigned can be capped by an entropy-regularized linear programming. Figure 1 compares the validation perplexity when training the 100M/64E models using the base expert choice method (EC-BASE), expert choice capped by two experts per token (EC-CAP2), expert choice capped by three experts per token (EC-CAP3), and GShard top-2 gating.\n\nAs shown in the figure, restricting the number of experts to 2 degrades the perplexity compared to the base expert choice method. This suggests that a more flexible allocation of experts (e.g. more than 2 experts for a token) can enhance model expressiveness. On the other hand, our EC-CAP2 and EC-CAP3 methods still outperform the top-2 gating method by a clear margin. We believe this confirms the effectiveness of a load balanced training, provided by our method. Finally, EC-CAP3 obtains comparable perplexity to EC-BASE. As indicated by Figure 3, only a little fraction of tokens use more than 3 experts therefore we see little or no difference between EC-BASE and EC-CAP3 variants. We present the fine-tuning results of these methods in Table 2.\n\nD Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [? ]. We use mod x to map a token ID to an expert ID. This in some way ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 2. Hashing based routing performs much worse than expert choice in terms of average scores and variance.\n\nE Fine-Tuning Details\n\nWe did a hyperparameter search for both baseline models and expert choice method. For fine-tuning of the 8B dense model, we use a constant learning rate of 0.0001 and a dropout rate of 0.1. We freeze the attention layer and feed-forward layer while leaving the embedding and layer normalization trainable. This setting has been found optimal for the 8B dense model. For MoE 8B/64E models including GShard top-2 gating and expert choice, we found continuing the learning rate from the pre-trained model while using a square root learning rate decay works better. In addition, we do not apply parameter freezing for fine-tuning MoE models. For models with 100M expert size, we use a constant learning rate of 0.0001 and no dropout is used.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/2202.09368v2.md"
+          },
+          "headlines": {
+            "1 Introduction": [],
+            "2 Related Work": [],
+            "3 Method": [
+              "3.1 Pitfalls Of Token-Choice Routing",
+              "3.2 Heterogeneous Moe Via Expert Choice",
+              "3.3 Expert Choice With Additional Constraint",
+              "3.4 Model Architecture"
+            ],
+            "4 Experiments": [
+              "4.1 Setup",
+              "4.2 Training Efficiency",
+              "4.3 Scaling The Number Of Experts",
+              "4.4 Fine-Tuning On Glue And Superglue",
+              "4.5 Heterogeneity Matters",
+              "4.6 Comparison With Hash Layer",
+              "4.7 Ablation"
+            ],
+            "5 Conclusion": [],
+            "6 Limitations": [],
+            "A Comparison On Fine-Tuning With A Dense Model": [],
+            "B Capacity Factor": [],
+            "C Capped Expert Choice": [],
+            "D Comparison With Hash Layer": [],
+            "E Fine-Tuning Details": []
+          },
+          "summary": "The paper introduces a new routing method for sparsely-activated Mixture-of-Experts (MoE) models called 'expert choice,' which addresses load imbalance and under-utilization issues in conventional MoE methods. Unlike traditional token-choice routing, expert choice allows experts to select the top-k tokens, ensuring perfect load balancing and enabling variable expert allocation per token. This method improves training efficiency, achieving over 2x faster convergence compared to existing models like Switch Transformer and GShard. It also demonstrates superior performance on downstream tasks in the GLUE and SuperGLUE benchmarks. The study highlights the benefits of flexible expert allocation and shows that expert choice outperforms hashing-based routing methods. The paper also discusses potential limitations and future improvements for the expert choice method.",
+          "summary_embedding": [
+            -0.009726986289024353,
+            -0.0337262786924839,
+            -0.023302342742681503,
+            -0.019878201186656952,
+            -0.025953780859708786,
+            0.04560471698641777,
+            -0.006874797400087118,
+            0.039665497839450836,
+            -0.006973279174417257,
+            0.034544438123703,
+            -0.025787118822336197,
+            -0.00363057479262352,
+            -0.01052999310195446,
+            0.0054657477885484695,
+            -0.024499278515577316,
+            0.036786794662475586,
+            0.010007280856370926,
+            0.003272630739957094,
+            0.006102093029767275,
+            -0.04048365354537964,
+            -0.018666116520762444,
+            -0.013522328808903694,
+            -0.03230207785964012,
+            0.008969432674348354,
+            0.02484775334596634,
+            0.0016713523073121905,
+            0.0026230288203805685,
+            0.003624893259257078,
+            -0.05311964824795723,
+            0.04339266195893288,
+            0.018681267276406288,
+            0.023711422458291054,
+            -0.016332851722836494,
+            -0.012832955457270145,
+            -0.039120059460401535,
+            -0.0197418425232172,
+            0.030786970630288124,
+            0.039968520402908325,
+            -0.019287310540676117,
+            0.03357476741075516,
+            0.018060073256492615,
+            0.02454473078250885,
+            -0.021272100508213043,
+            -0.029817303642630577,
+            -0.0061892117373645306,
+            -0.007840678095817566,
+            -0.008355814032256603,
+            0.01242387667298317,
+            0.016984347254037857,
+            -0.013007191941142082,
+            0.0212569497525692,
+            -0.01371171697974205,
+            0.03384748846292496,
+            0.008916404098272324,
+            -0.042332086712121964,
+            0.03984731063246727,
+            -0.008007340133190155,
+            0.012105703353881836,
+            0.03281721472740173,
+            0.010333028621971607,
+            -0.005287723150104284,
+            -0.021544819697737694,
+            -0.03378688171505928,
+            0.0010529992869123816,
+            0.00014014738553669304,
+            0.005950582213699818,
+            -0.003069985192269087,
+            -0.009742137044668198,
+            0.03227177634835243,
+            -0.003939277958124876,
+            -0.0740584209561348,
+            0.050089433789253235,
+            -0.026590125635266304,
+            -0.007628562860190868,
+            -0.028938541188836098,
+            -0.01736312545835972,
+            0.037362534552812576,
+            0.005662711802870035,
+            0.0098709212616086,
+            0.0106284748762846,
+            -0.0007892759749665856,
+            0.028923390433192253,
+            0.00910579226911068,
+            -0.024923507124185562,
+            0.04478655755519867,
+            -0.04790768027305603,
+            0.01801462098956108,
+            -0.04336235672235489,
+            0.022756904363632202,
+            1.6393929399782792e-05,
+            -0.020726662129163742,
+            -0.0013304532039910555,
+            -0.016666175797581673,
+            0.0218023881316185,
+            -0.01730252057313919,
+            0.026802239939570427,
+            0.013825350441038609,
+            -0.015802564099431038,
+            -0.029817303642630577,
+            0.025378039106726646,
+            0.003367325058206916,
+            0.010499690659344196,
+            -0.02075696364045143,
+            0.014332910999655724,
+            0.04393810033798218,
+            -0.02593863010406494,
+            -0.02271145209670067,
+            0.027681002393364906,
+            0.014408666640520096,
+            0.0633314698934555,
+            0.036544378846883774,
+            0.006624804809689522,
+            0.022362977266311646,
+            0.020575150847434998,
+            -0.028968842700123787,
+            -0.01963578537106514,
+            -0.046362269669771194,
+            -0.04757435619831085,
+            0.019878201186656952,
+            -0.04566532000899315,
+            0.0050869714468717575,
+            0.038544319570064545,
+            -0.013704141601920128,
+            -0.014446543529629707,
+            -0.04021093621850014,
+            0.017847958952188492,
+            0.0003908502112608403,
+            0.011976920068264008,
+            -0.0022082682698965073,
+            0.00033285003155469894,
+            -0.011825408786535263,
+            0.008878526277840137,
+            -0.03778676688671112,
+            -0.025196226313710213,
+            0.025181075558066368,
+            0.022862963378429413,
+            0.024120500311255455,
+            0.028802180662751198,
+            -0.03077181987464428,
+            -0.009984553791582584,
+            0.021378157660365105,
+            -0.04242299124598503,
+            -0.006556625012308359,
+            0.011332998983561993,
+            0.0026703758630901575,
+            0.003872992005199194,
+            0.032786913216114044,
+            0.052998438477516174,
+            -0.03172633796930313,
+            0.02065090648829937,
+            -0.010431510396301746,
+            0.04957429692149162,
+            0.00024691506405360997,
+            0.028968842700123787,
+            -0.019514575600624084,
+            -0.0134389977902174,
+            -0.029529431834816933,
+            -0.009795165620744228,
+            -0.01356020662933588,
+            -0.008484598249197006,
+            -0.002998017705976963,
+            -0.009575475007295609,
+            0.0040453351102769375,
+            0.0331808403134346,
+            0.00273855566047132,
+            0.030680913478136063,
+            0.014325335621833801,
+            0.0051475754007697105,
+            -0.08745196461677551,
+            0.026059838011860847,
+            -0.0033616432920098305,
+            0.005859675817191601,
+            0.0062460280023515224,
+            0.03013547509908676,
+            -0.0312718041241169,
+            0.020938776433467865,
+            0.026620427146553993,
+            0.0011145505122840405,
+            0.016969196498394012,
+            0.036847397685050964,
+            -0.04181694984436035,
+            0.009643655270338058,
+            0.012590537779033184,
+            -0.0033881578128784895,
+            0.04584713280200958,
+            0.011628445237874985,
+            0.019514575600624084,
+            0.018529755994677544,
+            0.003569970605894923,
+            0.00297907879576087,
+            0.006533898413181305,
+            0.03724132478237152,
+            0.003462019143626094,
+            -0.009833043441176414,
+            0.014976831153035164,
+            -0.010757259093225002,
+            -0.011219366453588009,
+            -0.037998881191015244,
+            0.014454119838774204,
+            0.008742166683077812,
+            -0.017454031854867935,
+            -0.02959003672003746,
+            0.0026949963066726923,
+            -0.031908150762319565,
+            0.026559824123978615,
+            -0.002445003716275096,
+            0.005738467443734407,
+            -0.010454237461090088,
+            -0.006757376715540886,
+            -0.035423196852207184,
+            0.028453707695007324,
+            -0.02095392718911171,
+            0.0053142374381423,
+            0.0007258309051394463,
+            -0.029438525438308716,
+            0.0060073984786868095,
+            0.017863109707832336,
+            -0.0025965142995119095,
+            0.0025302283465862274,
+            -0.02349930815398693,
+            0.024468975141644478,
+            0.025635607540607452,
+            0.004518806003034115,
+            -0.014037464745342731,
+            -0.013113250024616718,
+            -0.003605954349040985,
+            -0.009742137044668198,
+            -0.016196493059396744,
+            0.024468975141644478,
+            -0.014810169115662575,
+            0.00308892410248518,
+            0.021847840398550034,
+            0.014264730736613274,
+            -0.007045246660709381,
+            -0.004007457755506039,
+            -0.04066546633839607,
+            -0.01218903437256813,
+            -0.04760465770959854,
+            -0.011628445237874985,
+            -0.0034601252991706133,
+            -0.028696123510599136,
+            0.0061513339169323444,
+            -0.04481685906648636,
+            -0.01235569640994072,
+            0.02309022843837738,
+            0.018135828897356987,
+            0.0019488062243908644,
+            -0.009173971600830555,
+            -0.022605394944548607,
+            8.486965816700831e-05,
+            -0.01763584464788437,
+            -0.017044952139258385,
+            0.015393485315144062,
+            0.031059689819812775,
+            -0.03101423755288124,
+            0.04024123772978783,
+            0.003893824527040124,
+            -0.08066429197788239,
+            0.010393633507192135,
+            -0.026362858712673187,
+            0.07011914253234863,
+            -0.0025453795678913593,
+            0.014075342565774918,
+            0.06781618297100067,
+            -0.012431452050805092,
+            -0.02007516659796238,
+            0.03142331540584564,
+            -0.008249756880104542,
+            -1.1311515663692262e-05,
+            -0.02884763479232788,
+            -0.02739313244819641,
+            -0.053271155804395676,
+            -0.009704259224236012,
+            -0.010787560604512691,
+            0.03360506892204285,
+            -0.024317465722560883,
+            -0.0007656024536117911,
+            0.012620840221643448,
+            0.02983245439827442,
+            0.016635872423648834,
+            -0.0015681355725973845,
+            0.008272483013570309,
+            0.02881733328104019,
+            0.0274234339594841,
+            -0.0036154238041490316,
+            0.00814369972795248,
+            -0.034211114048957825,
+            -0.0068066176027059555,
+            -0.0493621826171875,
+            0.011302697472274303,
+            -0.02596893161535263,
+            -0.02596893161535263,
+            0.007643714081496,
+            -0.011476934887468815,
+            0.03748374432325363,
+            -0.024696242064237595,
+            -0.007503566797822714,
+            0.005037730094045401,
+            0.012855681590735912,
+            -0.017757052555680275,
+            -0.03612014651298523,
+            -0.0028502948116511106,
+            0.0026381798088550568,
+            0.007560383062809706,
+            -0.053240854293107986,
+            0.004499867558479309,
+            -0.018469152972102165,
+            0.0279688723385334,
+            -0.02139330841600895,
+            -0.013688990846276283,
+            0.042029064148664474,
+            0.011363301426172256,
+            0.027499189600348473,
+            -0.026468917727470398,
+            0.0017849853029474616,
+            0.012825379148125648,
+            -0.0020056227222085,
+            0.032514192163944244,
+            -0.030574856325984,
+            -0.03257479891180992,
+            0.001548249856568873,
+            -0.007840678095817566,
+            -0.0625436082482338,
+            -0.011386028490960598,
+            -0.00981789268553257,
+            0.017317671328783035,
+            0.0446956530213356,
+            0.04557441547513008,
+            -0.01068150345236063,
+            0.002174178371205926,
+            -0.02213571034371853,
+            -0.04542290419340134,
+            -0.02643861435353756,
+            0.0016486257081851363,
+            -0.019999410957098007,
+            -0.02789311669766903,
+            -0.0018313854234293103,
+            0.011211791075766087,
+            -0.013651113025844097,
+            0.01134057529270649,
+            -0.018560059368610382,
+            -0.01862066239118576,
+            -0.012332970276474953,
+            0.018575210124254227,
+            0.009901223704218864,
+            -0.0018796793883666396,
+            -0.015037435106933117,
+            -0.00980274099856615,
+            0.018969137221574783,
+            -0.011279970407485962,
+            0.0009388927719555795,
+            0.01188601367175579,
+            0.024347767233848572,
+            -0.007083124481141567,
+            0.010961798019707203,
+            -0.034756552428007126,
+            -0.01966608688235283,
+            0.007984613068401814,
+            0.002719616750255227,
+            0.020711511373519897,
+            0.008742166683077812,
+            -0.04275631532073021,
+            -0.044301725924015045,
+            0.016044981777668,
+            -0.0562104657292366,
+            0.028120383620262146,
+            -0.03327174484729767,
+            -0.012514783069491386,
+            -0.02671133354306221,
+            0.0013418165035545826,
+            -0.01828734017908573,
+            0.022438732907176018,
+            0.03778676688671112,
+            -0.0011486404109746218,
+            0.005503625608980656,
+            0.014658658765256405,
+            0.04384719207882881,
+            -0.05021063983440399,
+            -0.041180603206157684,
+            0.03754434734582901,
+            -0.01601468026638031,
+            0.0185146052390337,
+            0.00021744151308666915,
+            0.010348179377615452,
+            -0.014431392773985863,
+            0.022772056981921196,
+            0.013757170177996159,
+            -0.008764892816543579,
+            -0.030574856325984,
+            0.04945308715105057,
+            -0.023650817573070526,
+            -0.049513693898916245,
+            0.0018323323456570506,
+            0.018878230825066566,
+            0.018635815009474754,
+            -0.014870774000883102,
+            -0.02907489985227585,
+            -0.024317465722560883,
+            -0.006060427520424128,
+            -0.00947699323296547,
+            -0.03402930125594139,
+            0.011363301426172256,
+            0.0048445542342960835,
+            0.016666175797581673,
+            -0.00645056739449501,
+            -0.0073141781613230705,
+            -0.009370936080813408,
+            0.022559940814971924,
+            0.012635990977287292,
+            -0.03175663948059082,
+            -0.017484333366155624,
+            -0.02528713271021843,
+            0.004359719809144735,
+            0.010136065073311329,
+            0.022166013717651367,
+            0.024287162348628044,
+            -0.0323626808822155,
+            0.04218057543039322,
+            0.01550711877644062,
+            -0.008727015927433968,
+            -0.0019317612750455737,
+            0.02478714846074581,
+            0.04324115067720413,
+            0.0218023881316185,
+            0.026514369994401932,
+            0.002107892418280244,
+            8.705945947440341e-05,
+            -0.015620751306414604,
+            -0.0011391709558665752,
+            0.01862066239118576,
+            0.04911976307630539,
+            -0.012151156552135944,
+            0.01716616004705429,
+            -0.03578682243824005,
+            0.00396200455725193,
+            -0.02580226957798004,
+            -0.030786970630288124,
+            0.03745344281196594,
+            -0.015060162171721458,
+            0.0191964041441679,
+            0.025044716894626617,
+            -0.022469034418463707,
+            -0.011075431481003761,
+            -0.019423669204115868,
+            -0.026181045919656754,
+            -0.01702980138361454,
+            -0.03281721472740173,
+            -0.018135828897356987,
+            -0.002653330797329545,
+            -0.035635314881801605,
+            -0.04987731948494911,
+            -0.004329417832195759,
+            -0.046392571181058884,
+            -0.010507266037166119,
+            0.017378276214003563,
+            -0.011166337877511978,
+            -0.025711363181471825,
+            -0.005336964037269354,
+            0.003973367623984814,
+            -0.022196315228939056,
+            -0.0066588944755494595,
+            0.008439145050942898,
+            0.035938333719968796,
+            -0.02298417128622532,
+            -0.013060221448540688,
+            -0.0024317465722560883,
+            -0.02887793630361557,
+            -0.007878555916249752,
+            0.008999735116958618,
+            0.007723257411271334,
+            -0.0012357590021565557,
+            -0.05196816474199295,
+            -0.006916462909430265,
+            -0.012992041185498238,
+            0.029196109622716904,
+            0.026468917727470398,
+            0.0012168202083557844,
+            -0.02783251367509365,
+            0.008749742060899734,
+            -0.0072535742074251175,
+            0.02339324913918972,
+            -0.007992188446223736,
+            -0.022908415645360947,
+            -0.02034788578748703,
+            0.030786970630288124,
+            -0.0159086212515831,
+            0.009113367646932602,
+            0.009976978413760662,
+            0.0121360057964921,
+            -0.03899884968996048,
+            -0.001445980160497129,
+            0.0005648507503792644,
+            -0.010128489695489407,
+            0.00423093605786562,
+            -0.006738437805324793,
+            -0.007079336792230606,
+            -0.022090258076786995,
+            -0.0218023881316185,
+            0.03818069398403168,
+            -0.009113367646932602,
+            -0.0035661826841533184,
+            -0.01088604237884283,
+            0.03436262533068657,
+            -0.00595815759152174,
+            -0.01993880607187748,
+            -0.019347915425896645,
+            0.014431392773985863,
+            0.05105910077691078,
+            -0.028529461473226547,
+            -0.03984731063246727,
+            0.0011136035900563002,
+            0.030150625854730606,
+            0.004140029661357403,
+            -0.016908591613173485,
+            0.0035851215943694115,
+            0.056392278522253036,
+            -0.011742077767848969,
+            0.028362801298499107,
+            0.027771908789873123,
+            0.02295386977493763,
+            -0.02021152526140213,
+            -0.01678738370537758,
+            -0.020999381318688393,
+            0.011098157614469528,
+            -0.03027183562517166,
+            0.027059808373451233,
+            0.018969137221574783,
+            0.01538590993732214,
+            0.05054396390914917,
+            -0.03178694099187851,
+            -0.02515077404677868,
+            0.008499749936163425,
+            0.0021722845267504454,
+            0.011007251217961311,
+            -0.007727045100182295,
+            -0.01801462098956108,
+            -0.001950700068846345,
+            -0.014022313989698887,
+            -0.019817598164081573,
+            -0.020938776433467865,
+            0.00954517349600792,
+            0.015696506947278976,
+            0.002649543108418584,
+            0.010863316245377064,
+            0.0016590419691056013,
+            0.01686313934624195,
+            0.01321930717676878,
+            -0.0444229319691658,
+            -7.936555630294606e-05,
+            0.03378688171505928,
+            -0.0293476190418005,
+            -0.012264790013432503,
+            -0.016060132533311844,
+            -0.004439263138920069,
+            -0.01706010289490223,
+            -0.02762039750814438,
+            -0.006874797400087118,
+            -0.01889338344335556,
+            -0.00814369972795248,
+            -0.028196139261126518,
+            -0.009992130100727081,
+            0.01949942484498024,
+            0.022332675755023956,
+            0.001196934375911951,
+            -0.006783891003578901,
+            -0.005901341326534748,
+            0.02092362567782402,
+            0.032514192163944244,
+            0.01952972821891308,
+            0.0058331615291535854,
+            -0.015158643946051598,
+            -0.004295328166335821,
+            -0.012370847165584564,
+            0.016060132533311844,
+            0.012537509202957153,
+            6.344509893096983e-05,
+            0.03493836522102356,
+            0.004170331638306379,
+            -0.022726602852344513,
+            -0.001230077352374792,
+            0.024923507124185562,
+            -0.021590271964669228,
+            -0.0108330138027668,
+            0.040029123425483704,
+            0.0085679292678833,
+            -0.0024506852496415377,
+            -0.019287310540676117,
+            -0.019075196236371994,
+            0.02630225569009781,
+            -0.02133270539343357,
+            0.002083271974697709,
+            0.0482713058590889,
+            0.00035344602656550705,
+            -0.029802151024341583,
+            0.017954016104340553,
+            -0.013242034241557121,
+            -0.018560059368610382,
+            0.01848430372774601,
+            0.0012859469279646873,
+            -0.039968520402908325,
+            0.019014591351151466,
+            0.018408548086881638,
+            0.022635696455836296,
+            -0.02918095886707306,
+            0.0002127068000845611,
+            0.03233237937092781,
+            0.004037759732455015,
+            0.001595596899278462,
+            -0.008530051447451115,
+            -0.0006065162015147507,
+            0.010431510396301746,
+            0.001746160676702857,
+            0.02854461409151554,
+            -0.0030434709042310715,
+            0.027771908789873123,
+            -0.006727074272930622,
+            -0.008863375522196293,
+            0.019181253388524055,
+            -0.02109028771519661,
+            0.013166278600692749,
+            0.0685434341430664,
+            0.0021950111258774996,
+            0.025847723707556725,
+            0.007181606721132994,
+            0.024347767233848572,
+            0.02018122375011444,
+            -0.027271922677755356,
+            -0.02021152526140213,
+            -0.005397567991167307,
+            -0.0014033677289262414,
+            -0.03787767142057419,
+            0.011128460057079792,
+            -0.021302402019500732,
+            -0.013249609619379044,
+            -0.0022347825579345226,
+            -0.002316219499334693,
+            -0.006405114196240902,
+            0.034211114048957825,
+            -0.0014630251098424196,
+            -0.018560059368610382,
+            -0.035423196852207184,
+            0.037726160138845444,
+            0.04048365354537964,
+            0.0389382466673851,
+            -0.005984671879559755,
+            0.03548380360007286,
+            0.0030756669584661722,
+            0.0025302283465862274,
+            0.041453324258327484,
+            0.004791525658220053,
+            -0.011567841283977032,
+            0.02295386977493763,
+            0.012416300363838673,
+            0.03351416438817978,
+            -0.02045394293963909,
+            -0.02630225569009781,
+            -0.003299145260825753,
+            -0.026423463597893715,
+            0.012492056004703045,
+            -0.019726691767573357,
+            0.0018768386216834188,
+            0.007117214612662792,
+            0.015196521766483784,
+            -0.016044981777668,
+            0.014719262719154358,
+            -0.006102093029767275,
+            -0.021014532074332237,
+            0.011719351634383202,
+            -0.007780073676258326,
+            0.011961768381297588,
+            -0.02262054570019245,
+            0.001138224033638835,
+            -0.04063516482710838,
+            0.008280059322714806,
+            0.014022313989698887,
+            -0.01598437689244747,
+            0.03205966204404831,
+            0.020969079807400703,
+            -0.01801462098956108,
+            -0.0009682480012997985,
+            0.0034714885987341404,
+            -0.010393633507192135,
+            -0.011067855171859264,
+            -0.019923655316233635,
+            0.02786281518638134,
+            -0.015802564099431038,
+            -0.025908326730132103,
+            0.02454473078250885,
+            -0.01743887923657894,
+            0.015151068568229675,
+            -0.004299115855246782,
+            0.00888610165566206,
+            -0.019120648503303528,
+            0.02284781076014042,
+            -0.023741723969578743,
+            -0.0009128518868237734,
+            -0.020287280902266502,
+            0.0028389315120875835,
+            0.014135947450995445,
+            -0.018272189423441887,
+            -0.02139330841600895,
+            0.019150950014591217,
+            -0.005420294590294361,
+            -0.003931702114641666,
+            0.010014856234192848,
+            0.0030983935575932264,
+            -0.0029715034179389477,
+            -0.004549108445644379,
+            -0.03281721472740173,
+            -0.02915065549314022,
+            0.0033086147159337997,
+            0.011386028490960598,
+            0.015257125720381737,
+            0.027574945241212845,
+            0.03715042024850845,
+            0.006052851676940918,
+            0.0036949669010937214,
+            0.0009327376610599458,
+            -0.020726662129163742,
+            0.042635105550289154,
+            -0.020469093695282936,
+            0.040847279131412506,
+            0.032544493675231934,
+            -0.0197418425232172,
+            -0.023908386006951332,
+            0.0029052174650132656,
+            0.03115059621632099,
+            0.03151422366499901,
+            0.025211378931999207,
+            0.012310243211686611,
+            0.03690800443291664,
+            -0.004359719809144735,
+            -0.032211173325777054,
+            0.015802564099431038,
+            0.022256920114159584,
+            -0.008105821907520294,
+            0.009673957712948322,
+            0.027438584715127945,
+            0.013279912061989307,
+            -0.007757347077131271,
+            0.030150625854730606,
+            0.0012234487803652883,
+            0.02396899089217186,
+            0.02498411200940609,
+            -0.02190844528377056,
+            0.01743887923657894,
+            -0.0001154677156591788,
+            0.006727074272930622,
+            -0.009840618818998337,
+            0.004348356742411852,
+            0.011810258030891418,
+            0.018272189423441887,
+            0.018908534198999405,
+            0.003246116451919079,
+            -0.07060398161411285,
+            0.019726691767573357,
+            0.004507442936301231,
+            -0.02160542458295822,
+            0.00390140013769269,
+            -0.0024866692256182432,
+            0.012704171240329742,
+            0.02105998620390892,
+            0.012461753562092781,
+            -0.01790856383740902,
+            0.0013626492582261562,
+            0.011961768381297588,
+            -0.00595815759152174,
+            0.04730163514614105,
+            0.0019791084341704845,
+            -0.01437078882008791,
+            -0.016105586662888527,
+            0.013726867735385895,
+            0.02525683119893074,
+            -0.02355991117656231,
+            0.006325571332126856,
+            0.005621046293526888,
+            -0.004147605039179325,
+            -0.010742107406258583,
+            -0.030377892777323723,
+            -0.005280147306621075,
+            -0.03436262533068657,
+            0.023150833323597908,
+            0.015491968020796776,
+            0.014764715917408466,
+            -0.025044716894626617,
+            -0.004280176945030689,
+            0.0046400148421525955,
+            -0.021287251263856888,
+            0.003261267440393567,
+            0.01161329448223114,
+            0.0005104016163386405,
+            -0.020984230563044548,
+            0.00959820207208395,
+            0.013469300232827663,
+            0.06848283112049103,
+            0.0007442962378263474,
+            0.029771849513053894,
+            0.0422411784529686,
+            -0.0012243957025930285,
+            -0.006371024530380964,
+            0.0036286809481680393,
+            -0.019953956827521324,
+            0.003473382443189621,
+            0.025878025218844414,
+            -0.03224147483706474,
+            -0.055058982223272324,
+            -0.007545232307165861,
+            -0.04624105989933014,
+            0.001489539397880435,
+            0.014711687341332436,
+            0.023044776171445847,
+            -0.0037782976869493723,
+            -0.034817155450582504,
+            0.016908591613173485,
+            -0.003071879269555211,
+            0.007014944683760405,
+            -0.02593863010406494,
+            -0.0294839795678854,
+            -0.026014383882284164,
+            -0.02339324913918972,
+            -0.012908710166811943,
+            -0.022726602852344513,
+            0.012590537779033184,
+            -0.0015103721525520086,
+            -0.06587684899568558,
+            -0.009780014865100384,
+            0.016166189685463905,
+            0.001200722181238234,
+            0.011939042247831821,
+            0.011083006858825684,
+            -0.007083124481141567,
+            -0.0018275976181030273,
+            -0.022559940814971924,
+            0.03124150261282921,
+            0.005177877843379974,
+            0.03584742918610573,
+            -0.006503595970571041,
+            -0.009128518402576447,
+            0.01073453202843666,
+            0.04527139291167259,
+            0.0017916138749569654,
+            0.010226971469819546,
+            -0.020560000091791153,
+            0.0015065843472257257,
+            -0.02284781076014042,
+            -0.0031135445460677147,
+            -0.008863375522196293,
+            -0.0021211495622992516,
+            0.015893470495939255,
+            0.0019753205124288797,
+            -0.010204244405031204,
+            0.025438643991947174,
+            0.02251448854804039,
+            -0.009605777449905872,
+            -0.015969226136803627,
+            -0.02437806874513626,
+            -0.014545026235282421,
+            -0.0002627763315103948,
+            -0.010007280856370926,
+            -0.022453883662819862,
+            -0.020878173410892487,
+            -0.002566212322562933,
+            -0.02559015527367592,
+            0.022529639303684235,
+            0.01220418605953455,
+            -4.882668508798815e-05,
+            -0.006723286584019661,
+            -0.038544319570064545,
+            -0.003727162955328822,
+            -0.03118089959025383,
+            -0.005374841392040253,
+            0.03287781774997711,
+            -0.03745344281196594,
+            -0.0002231231628684327,
+            0.014491996727883816,
+            -0.01993880607187748,
+            0.0028938541654497385,
+            0.008961857296526432,
+            0.02769615314900875,
+            -0.027408283203840256,
+            0.0001795638381736353,
+            0.006954340264201164,
+            0.016908591613173485,
+            -0.009878496639430523,
+            0.027196168899536133,
+            -0.04275631532073021,
+            0.011037553660571575,
+            -0.037120118737220764,
+            -0.020423639565706253,
+            -0.01523439958691597,
+            0.008363389410078526,
+            -0.005018791649490595,
+            -0.02637801133096218,
+            -0.010454237461090088,
+            0.006026337388902903,
+            -0.00693918950855732,
+            -0.01902974210679531,
+            -0.010620899498462677,
+            0.027438584715127945,
+            0.00931033119559288,
+            0.007977037690579891,
+            -0.02955973520874977,
+            -0.01631770096719265,
+            0.006280118133872747,
+            0.052089374512434006,
+            -0.01584801822900772,
+            0.014938953332602978,
+            -0.006639956030994654,
+            0.02342355251312256,
+            -0.028484009206295013,
+            -0.01872672140598297,
+            0.0315445251762867,
+            0.013946558348834515,
+            0.007064185570925474,
+            0.028953691944479942,
+            0.027196168899536133,
+            0.027105262503027916,
+            -0.04654408246278763,
+            0.009886072017252445,
+            -0.002613559365272522,
+            -0.008757317438721657,
+            0.03402930125594139,
+            0.01916610263288021,
+            0.022150862962007523,
+            0.0009744031121954322,
+            -0.01730252057313919,
+            0.0139844361692667,
+            0.014885924756526947,
+            0.007681591901928186,
+            -0.014242004603147507,
+            -0.02660527639091015,
+            0.003367325058206916,
+            -0.007992188446223736,
+            0.00047276070108637214,
+            0.020726662129163742,
+            -0.009052763693034649,
+            0.030999086797237396,
+            -0.014605630189180374,
+            -0.02593863010406494,
+            0.008916404098272324,
+            -0.02268115058541298,
+            -0.0041741193272173405,
+            0.0004476667381823063,
+            0.010083036497235298,
+            -0.023847781121730804,
+            -0.04669559374451637,
+            0.019408518448472023,
+            0.028393102809786797,
+            0.005579380784183741,
+            0.01328748743981123,
+            -0.035332292318344116,
+            0.008992158807814121,
+            0.012310243211686611,
+            0.011348150670528412,
+            0.0019847899675369263,
+            0.035362593829631805,
+            -0.00028834378463216126,
+            0.012885984033346176,
+            -0.008620957843959332,
+            0.028362801298499107,
+            0.00667025800794363,
+            -0.0323626808822155,
+            -0.0378473699092865,
+            -0.01513591781258583,
+            0.015257125720381737,
+            0.006692984607070684,
+            -0.011840560473501682,
+            -0.02278720773756504,
+            -0.038817036896944046,
+            0.015787413343787193,
+            -0.012916285544633865,
+            -0.038271598517894745,
+            -0.010158791206777096,
+            -0.009742137044668198,
+            0.016635872423648834,
+            -0.006916462909430265,
+            -0.016802534461021423,
+            -0.016090434044599533,
+            0.005821797996759415,
+            7.161247049225494e-05,
+            0.0244083721190691,
+            -0.001251857029274106,
+            -0.006855858489871025,
+            -0.016393456608057022,
+            -0.01188601367175579,
+            -0.0035074723418802023,
+            0.03602924197912216,
+            -0.014219277538359165,
+            -0.02383263036608696,
+            0.014984406530857086,
+            0.02657497487962246,
+            0.011636020615696907,
+            0.03227177634835243,
+            0.007181606721132994,
+            -0.014022313989698887,
+            0.003016956616193056,
+            -0.033029329031705856,
+            -0.0046400148421525955,
+            0.005196816753596067,
+            -0.006427840795367956,
+            -0.015651054680347443,
+            -0.01815097965300083,
+            -0.015832867473363876,
+            0.030756669119000435,
+            -0.02487805485725403,
+            0.0218023881316185,
+            0.030347591266036034,
+            0.003174148965626955,
+            -0.02915065549314022,
+            0.013598084449768066,
+            -0.0072005451656877995,
+            0.01862066239118576,
+            0.0019014590652659535,
+            0.028014326468110085,
+            -0.007461901288479567,
+            -0.004818039946258068,
+            -0.02501441352069378,
+            -0.03696860745549202,
+            -0.023484155535697937,
+            -0.008817922323942184,
+            0.0027537066489458084,
+            0.020984230563044548,
+            -0.0375746488571167,
+            -0.017757052555680275,
+            0.009855770505964756,
+            0.015757111832499504,
+            -0.03027183562517166,
+            -0.0025908327661454678,
+            -0.010158791206777096,
+            -0.0004055278259329498,
+            0.005140000022947788,
+            -0.02227207086980343,
+            -0.008977008052170277,
+            -0.015726808458566666,
+            -0.0010435298318043351,
+            0.006166485138237476,
+            -0.0007409819518215954,
+            -0.05805889517068863,
+            -0.013969285413622856,
+            0.018635815009474754,
+            0.0026400736533105373,
+            0.008908828720450401,
+            0.015938924625515938,
+            -0.0111739132553339,
+            0.0031533162109553814,
+            -0.026756787672638893,
+            0.023893235251307487,
+            -0.009492144919931889,
+            0.009355784393846989,
+            -0.013742019422352314,
+            0.012514783069491386,
+            0.014014738611876965,
+            0.004795313347131014,
+            0.0010681503918021917,
+            -0.0054051438346505165,
+            0.01601468026638031,
+            0.028377952054142952,
+            -0.023756874725222588,
+            0.011454207822680473,
+            -0.0002514130319468677,
+            -0.0032158142421394587,
+            -0.029196109622716904,
+            0.00509454682469368,
+            0.003931702114641666,
+            -0.005109698045998812,
+            0.016651025041937828,
+            -0.036544378846883774,
+            0.01662072166800499,
+            0.02177208475768566,
+            0.032544493675231934,
+            0.014832896180450916,
+            -0.017287369817495346,
+            -0.011658747680485249,
+            -0.012385998852550983,
+            -0.002624922664836049,
+            0.022181164473295212,
+            0.012120855040848255,
+            -0.01815097965300083,
+            -0.024287162348628044,
+            0.011249667964875698,
+            0.0005283934879116714,
+            -0.01203752402216196,
+            0.010795135982334614,
+            -0.013166278600692749,
+            0.01349202636629343,
+            0.012711746618151665,
+            0.00866641104221344,
+            0.00888610165566206,
+            0.022862963378429413,
+            -0.011878437362611294,
+            -0.017711598426103592,
+            0.00656041270121932,
+            -0.008916404098272324,
+            -0.0034089903347194195,
+            -0.012598113156855106,
+            0.010795135982334614,
+            0.014878349378705025,
+            0.06490717828273773,
+            -0.013143551535904408,
+            0.0009526234352961183,
+            -0.020438792183995247,
+            -0.031938452273607254,
+            -0.0017944546416401863,
+            -0.020908474922180176,
+            0.0111739132553339,
+            0.008370965719223022,
+            0.01598437689244747,
+            0.012022373266518116,
+            -0.0021912232041358948,
+            0.00795431062579155,
+            0.019317612051963806,
+            -0.010423935018479824,
+            7.255941454786807e-05,
+            -0.006503595970571041,
+            0.0013683309080079198,
+            0.0035851215943694115,
+            -0.011719351634383202,
+            -0.0026949963066726923,
+            0.035695917904376984,
+            0.01068150345236063,
+            0.011651171371340752,
+            -0.03445352986454964,
+            0.0020435003098100424,
+            0.020150920376181602,
+            0.007602048572152853,
+            -0.008302785456180573,
+            -0.01563590206205845,
+            0.01706010289490223,
+            -0.015317730605602264,
+            -0.0022499337792396545,
+            0.002888172399252653,
+            -0.010151215828955173,
+            0.04581683129072189,
+            -0.019787294790148735,
+            -0.0011742077767848969,
+            0.031847547739744186,
+            0.0030207443051040173,
+            -0.0040983641520142555,
+            0.005408931523561478,
+            -0.003473382443189621,
+            -0.0069013116881251335,
+            -0.02139330841600895,
+            0.0031381649896502495,
+            0.00667025800794363,
+            -0.019287310540676117,
+            -0.0072005451656877995,
+            -0.007719469256699085,
+            0.006897523999214172,
+            0.023105379194021225,
+            -0.009492144919931889,
+            -0.00016145357221830636,
+            -0.008575504645705223,
+            0.009848194196820259,
+            -0.0011912527261301875,
+            -0.044301725924015045,
+            -0.006147546228021383,
+            0.021181194111704826,
+            0.000973456131760031,
+            0.0005757405888289213,
+            -0.007450537756085396,
+            0.005007428117096424,
+            -0.017423728480935097,
+            -0.008454295806586742,
+            0.02369627170264721,
+            0.01736312545835972,
+            -0.009113367646932602,
+            0.010863316245377064,
+            -0.0035680767614394426,
+            0.00433320552110672,
+            -0.020014561712741852,
+            -0.01095422264188528,
+            0.003462019143626094,
+            0.015272277407348156,
+            -0.0009819786064326763,
+            -0.029029447585344315,
+            -0.0012272365856915712,
+            -0.00024194362049456686,
+            0.02918095886707306,
+            0.01486319862306118,
+            -0.008961857296526432,
+            -0.021211495622992516,
+            -0.013242034241557121,
+            -0.001237652963027358,
+            -0.015726808458566666,
+            -0.009620928205549717,
+            0.011855711229145527,
+            0.0009137988090515137,
+            0.008992158807814121,
+            -0.018135828897356987,
+            -0.027347678318619728,
+            -0.016166189685463905,
+            0.0018465365283191204,
+            -0.012870832346379757,
+            0.003340810537338257,
+            0.015514694154262543,
+            -0.03430201858282089,
+            -0.015060162171721458,
+            -0.03805948421359062,
+            0.008280059322714806,
+            -0.0029658216517418623,
+            -0.02024182677268982,
+            -0.023605365306138992,
+            0.009484568610787392,
+            0.019181253388524055,
+            -0.0015548784285783768,
+            0.019847899675369263,
+            -0.000889651826582849,
+            0.020060013979673386,
+            -0.0073141781613230705,
+            0.0031703610438853502,
+            -0.013007191941142082,
+            0.0030245319940149784,
+            0.008333087898790836,
+            0.02890823967754841,
+            -0.023075077682733536,
+            -0.006503595970571041,
+            -0.004685468040406704,
+            0.004746072459965944,
+            0.012658718042075634,
+            0.014613205567002296,
+            0.020741812884807587,
+            -0.008272483013570309,
+            -0.0027612822595983744,
+            0.012484480626881123,
+            -0.003799130441620946,
+            0.007295239716768265,
+            0.002390081062912941,
+            0.0019677451346069574,
+            0.004162756260484457,
+            0.008477022871375084,
+            0.0053710537031292915,
+            0.027650700882077217,
+            -0.0028275682125240564,
+            0.0031703610438853502,
+            0.024620486423373222,
+            0.02322658896446228,
+            -0.018878230825066566,
+            -0.028074929490685463,
+            0.002024561632424593,
+            0.050695475190877914,
+            0.007245998363941908,
+            -0.006810405291616917,
+            -0.03927157074213028,
+            -0.0031760428100824356,
+            0.028635520488023758,
+            0.023650817573070526,
+            0.01584801822900772,
+            -0.010567869991064072,
+            -0.012098127976059914,
+            0.02772645466029644,
+            0.013295062817633152,
+            0.02616589516401291,
+            -0.025302285328507423,
+            0.010704229585826397,
+            -0.03869583085179329,
+            0.0449683703482151,
+            0.004439263138920069,
+            -0.014476845972239971,
+            0.0011884119594469666,
+            0.010386057198047638,
+            -0.004537744913250208,
+            0.006431628484278917,
+            -5.8059369621332735e-05,
+            -0.0008086882880888879,
+            0.011923890560865402,
+            0.012007221579551697,
+            0.004265025723725557,
+            -0.02399929240345955,
+            0.007359631359577179,
+            0.004272601567208767,
+            0.03005971945822239,
+            0.005431658122688532,
+            -0.0013863227795809507,
+            0.012385998852550983,
+            -0.005810434930026531,
+            0.008855799213051796,
+            -0.010719381272792816,
+            0.020469093695282936,
+            -0.02918095886707306,
+            1.0986003871948924e-05,
+            0.02051454596221447,
+            -0.0015785519499331713,
+            0.0038199631962925196,
+            -0.023741723969578743,
+            0.01186328660696745,
+            -0.052968136966228485,
+            0.021166043356060982,
+            -0.018741872161626816,
+            0.021741783246397972,
+            0.019590331241488457,
+            -0.009734561666846275,
+            -0.012052674777805805,
+            0.03545350208878517,
+            0.0026760573964565992,
+            -0.014264730736613274,
+            -0.010643625631928444,
+            -0.017999470233917236,
+            -0.02830219641327858,
+            0.041968461126089096,
+            -0.0006117243901826441,
+            0.0028275682125240564,
+            0.0024222771171480417,
+            -0.010666352696716785,
+            -0.008923979476094246,
+            0.001954487757757306,
+            0.020499395206570625,
+            -0.03369597718119621,
+            -0.011128460057079792,
+            -0.008689138107001781,
+            -0.02566591091454029,
+            -0.00020300064352340996,
+            -0.011961768381297588,
+            0.026211349293589592,
+            0.01240872498601675,
+            -0.03696860745549202,
+            0.003780191531404853,
+            0.023650817573070526,
+            0.001603172393515706,
+            0.018120678141713142,
+            0.002130619017407298,
+            0.0010795136913657188,
+            0.009742137044668198,
+            -0.010476963594555855,
+            0.026559824123978615,
+            -0.009204274043440819,
+            0.008658835664391518,
+            -0.001950700068846345,
+            -0.017287369817495346,
+            0.017969166859984398,
+            0.004083212930709124,
+            0.024120500311255455,
+            0.011954193003475666,
+            0.011605718173086643,
+            0.01112088467925787,
+            -0.019878201186656952,
+            0.020090317353606224,
+            -0.0018891488434746861,
+            0.009075489826500416,
+            0.024726543575525284,
+            -0.005863463506102562,
+            0.020832719281315804,
+            0.015257125720381737,
+            -0.00020229043730068952,
+            0.011295121163129807,
+            -0.020120618864893913,
+            -0.011779955588281155,
+            0.0085679292678833,
+            -0.009605777449905872,
+            -0.010325453244149685,
+            0.009901223704218864,
+            -0.010105762630701065,
+            -0.015052586793899536,
+            -0.0071247899904847145,
+            0.016635872423648834,
+            0.0028976418543606997,
+            -0.02610529027879238,
+            -1.7326075976598077e-05,
+            0.008901252411305904,
+            0.006166485138237476,
+            0.018741872161626816,
+            0.011317848227918148,
+            -0.006692984607070684,
+            0.010363331064581871,
+            0.011476934887468815,
+            0.0053104497492313385,
+            -0.005079395603388548,
+            -0.013431422412395477,
+            -0.0024241709616035223,
+            0.018090376630425453,
+            0.009810317307710648,
+            -0.01922670565545559,
+            0.004564259201288223,
+            0.0008877579239197075,
+            -0.01420412678271532,
+            -0.019347915425896645,
+            -0.0038483713287860155,
+            -0.004170331638306379,
+            0.0027915844693779945,
+            0.0509984977543354,
+            -0.0006216673064045608,
+            0.007893706671893597,
+            -0.025741664692759514,
+            -0.006299056578427553,
+            -0.00027556007262319326,
+            -0.0036097420379519463,
+            0.01272689737379551,
+            -0.01824188604950905,
+            -0.03724132478237152,
+            0.012711746618151665,
+            -0.0012584856012836099,
+            0.014923802576959133,
+            -0.00297907879576087,
+            0.014226853847503662,
+            -0.017347972840070724,
+            0.01916610263288021,
+            -0.0011117096291854978,
+            -0.016029831022024155,
+            0.0029374135192483664,
+            -0.011143610812723637,
+            -0.021317552775144577,
+            0.005916492082178593,
+            -0.006848283112049103,
+            -0.03896854817867279,
+            0.0029885482508689165,
+            -0.010423935018479824,
+            0.0011798894265666604,
+            -0.003912763670086861,
+            0.020166072994470596,
+            -0.0061513339169323444,
+            0.0008697660523466766,
+            0.02177208475768566,
+            0.0017026013229042292,
+            0.01601468026638031,
+            -0.014499573037028313,
+            0.010999675840139389,
+            0.012598113156855106,
+            0.02210540883243084,
+            -0.032180868089199066,
+            -0.01393140759319067,
+            0.014211702160537243,
+            0.0026116655208170414,
+            -0.005382417235523462,
+            0.020635755732655525,
+            -0.01294658798724413,
+            0.019650936126708984,
+            -0.0005577487172558904,
+            -0.011704200878739357,
+            -0.03524138405919075,
+            -0.016529815271496773,
+            -3.343888238305226e-05,
+            -0.008439145050942898,
+            0.025317436084151268,
+            0.01760554127395153,
+            -0.0108330138027668,
+            0.026468917727470398,
+            0.0037574649322777987,
+            0.014598054811358452,
+            -0.014257155358791351,
+            -0.005999823100864887,
+            -0.0003773563075810671,
+            -0.0119314668700099,
+            0.015264702029526234,
+            -0.03512017801403999,
+            0.0008342557121068239,
+            -0.00839369185268879,
+            0.005242269951850176,
+            -0.0254992488771677,
+            0.018257036805152893,
+            0.025817420333623886,
+            -0.0031419529113918543,
+            -0.005037730094045401,
+            -0.022802358493208885,
+            0.028968842700123787,
+            0.03049910068511963,
+            0.0024828813038766384,
+            -0.0013825349742546678,
+            -0.00352072948589921,
+            0.0033218718599528074,
+            0.0211205892264843,
+            -0.037998881191015244,
+            -0.009098216891288757,
+            -0.028696123510599136,
+            0.042332086712121964,
+            -0.008098246529698372,
+            -0.013431422412395477,
+            0.01862066239118576,
+            -0.0024715180043131113,
+            0.02355991117656231,
+            -0.004689255729317665,
+            -0.004855917301028967,
+            0.03148392215371132,
+            0.011514811776578426,
+            0.02772645466029644,
+            -0.04015033319592476,
+            0.014704111963510513,
+            -0.006867222022265196,
+            0.004867280833423138,
+            -0.013484450988471508,
+            0.005302873905748129,
+            0.02580226957798004,
+            -0.018060073256492615,
+            0.0068066176027059555,
+            0.030665762722492218,
+            0.00748084019869566,
+            0.002448791405186057,
+            0.0165904201567173,
+            0.003890036838129163,
+            0.010855740867555141,
+            0.000841831264551729,
+            0.0052574207074940205,
+            0.022423582151532173,
+            0.014454119838774204,
+            -0.016029831022024155,
+            -0.015120766125619411,
+            -0.00044790346873924136,
+            -0.004450626205652952,
+            0.005261208396404982,
+            0.013242034241557121,
+            0.0061172437854111195,
+            -0.015681356191635132,
+            0.01052999310195446,
+            0.014158673584461212,
+            0.013128400780260563,
+            -0.0039847311563789845,
+            -0.03412020578980446,
+            -0.02443867363035679,
+            -0.021438762545585632,
+            0.003104075090959668,
+            -0.0008281006012111902,
+            -0.0021211495622992516,
+            0.0017954016802832484,
+            0.017347972840070724,
+            -0.0008352026343345642,
+            0.00379155483096838,
+            0.01686313934624195,
+            -0.01415109820663929,
+            -0.022090258076786995,
+            -0.0035964848939329386,
+            -0.027529491111636162,
+            0.0053710537031292915,
+            0.001844642567448318,
+            -0.00965123064815998,
+            0.008530051447451115,
+            -0.0046059247106313705,
+            -0.007223271764814854,
+            -0.012196609750390053,
+            -0.02007516659796238,
+            0.005780132487416267,
+            0.016878290101885796,
+            0.005730891600251198,
+            -0.009272454306483269,
+            -0.003939277958124876,
+            0.011575416661798954,
+            0.017484333366155624,
+            0.017317671328783035,
+            -0.009295180439949036,
+            0.009628504514694214,
+            0.0004022135108243674,
+            0.004670316819101572,
+            -0.01834794320166111,
+            0.0045869858004152775,
+            -0.0027953721582889557,
+            -0.008658835664391518,
+            0.01105270441621542,
+            -0.00031201730598695576,
+            0.01628739945590496,
+            -0.013090522959828377,
+            0.005624833982437849,
+            0.010333028621971607,
+            -0.020590301603078842,
+            -0.0067308624275028706,
+            -0.01692374423146248,
+            0.026484068483114243,
+            -0.00504530593752861,
+            0.00045405857963487506,
+            -0.02978700026869774,
+            0.0033616432920098305,
+            0.000848933297675103,
+            -0.000845619011670351,
+            -0.012764775194227695,
+            0.010749682784080505,
+            0.03242328763008118,
+            0.0005179771687835455,
+            0.0216660276055336,
+            -0.0036154238041490316,
+            0.0007480839849449694,
+            -0.00785582885146141,
+            0.004977126140147448,
+            -0.004916521720588207,
+            -0.009901223704218864,
+            -0.010863316245377064,
+            0.021726632490754128,
+            -0.0033029329497367144,
+            0.0003356908564455807,
+            0.011332998983561993,
+            -0.012067826464772224,
+            0.0014620780711993575,
+            -0.0047687990590929985,
+            0.00971941091120243,
+            0.0061702728271484375,
+            0.009113367646932602,
+            -0.01176480483263731,
+            0.05451354384422302,
+            -0.0106284748762846,
+            0.005537715740501881,
+            -0.00439380994066596,
+            -0.022226616740226746,
+            -0.017317671328783035,
+            0.012711746618151665,
+            -0.017544938251376152,
+            0.03375658020377159,
+            0.024150803685188293,
+            0.0006467612111009657,
+            0.010393633507192135,
+            -0.006958128418773413,
+            0.023923536762595177,
+            -0.002602196065708995,
+            0.0010795136913657188,
+            -0.004855917301028967,
+            0.015802564099431038,
+            0.04511988162994385,
+            0.017393426969647408,
+            -0.001152428099885583,
+            0.00829521007835865,
+            0.020575150847434998,
+            0.00814369972795248,
+            -0.0020075165666639805,
+            0.014219277538359165,
+            -0.01024212222546339,
+            0.00900731049478054,
+            -0.004458202049136162,
+            0.027438584715127945,
+            0.00814369972795248,
+            -0.006310420110821724,
+            -0.0036836036015301943,
+            -0.0008209985098801553,
+            0.014173824340105057,
+            0.006950552575290203,
+            0.01963578537106514,
+            -0.00461350055411458,
+            -0.0012575386790558696,
+            0.013870803639292717,
+            0.014491996727883816,
+            -0.00552635220810771,
+            0.001724380999803543,
+            -0.01435563713312149,
+            -0.010204244405031204,
+            0.013575357384979725,
+            0.006086941808462143,
+            0.006310420110821724,
+            -0.014514723792672157,
+            -0.00980274099856615,
+            -0.004670316819101572,
+            0.0015435151290148497,
+            0.015052586793899536,
+            0.011060279794037342,
+            -0.004814252257347107,
+            0.006052851676940918,
+            0.013825350441038609,
+            -0.014014738611876965,
+            0.006147546228021383,
+            0.007208121009171009,
+            0.008855799213051796,
+            0.004787737503647804,
+            -0.010817863047122955,
+            -0.00428396463394165,
+            -0.0063937511295080185,
+            0.0012670081341639161,
+            0.016832837834954262,
+            0.013181429356336594,
+            0.0049581872299313545,
+            0.027529491111636162,
+            -0.00839369185268879,
+            -0.012302667833864689,
+            0.02339324913918972,
+            0.011257244274020195,
+            0.0031703610438853502,
+            0.014651083387434483,
+            -0.012635990977287292,
+            -0.012446602806448936,
+            0.006655106786638498,
+            0.004848341923207045,
+            0.006049063988029957,
+            0.009151245467364788,
+            0.00450365524739027,
+            0.007333117071539164,
+            -0.007098275702446699,
+            -0.007427811156958342,
+            0.0014810169814154506,
+            0.01642375811934471,
+            0.020393338054418564,
+            -0.024968961253762245,
+            -0.0018806264270097017,
+            0.0009052763343788683,
+            0.010878467001020908,
+            -0.003316190093755722,
+            0.012310243211686611,
+            5.65205882594455e-05,
+            -0.0074240234680473804,
+            0.010158791206777096,
+            -0.02119634486734867,
+            -0.012681444175541401,
+            0.0012717428617179394,
+            0.013037494383752346,
+            -0.019590331241488457,
+            0.018226735293865204,
+            -0.008893677033483982,
+            -0.013348091393709183,
+            0.013893529772758484,
+            0.020014561712741852,
+            0.004090788308531046,
+            -0.007738408166915178,
+            -0.021862991154193878,
+            -0.01127239502966404,
+            0.016060132533311844,
+            0.020272130146622658,
+            0.000947888707742095,
+            -0.0006472347304224968,
+            0.010173942893743515,
+            -0.013113250024616718,
+            -0.023605365306138992,
+            -0.005230906419456005,
+            0.010386057198047638,
+            0.01746918261051178,
+            0.018832778558135033,
+            -0.006685409229248762,
+            0.02443867363035679,
+            -0.0059127043932676315,
+            -0.015499543398618698,
+            2.86597842205083e-05,
+            -0.02802947722375393,
+            -0.017423728480935097,
+            0.00013778003631159663,
+            0.00246773031540215,
+            0.008598231710493565,
+            0.0008555619278922677,
+            0.01943882182240486,
+            0.000904329412151128,
+            0.017590390518307686,
+            -0.000659071491099894,
+            -0.002009410411119461,
+            -0.0013995799235999584,
+            -0.0015340456739068031,
+            0.02254479005932808,
+            -0.012166308239102364,
+            -0.01176480483263731,
+            0.022302372381091118,
+            0.006852070800960064,
+            0.014719262719154358,
+            -0.027635548263788223,
+            0.013757170177996159,
+            -0.015499543398618698,
+            0.017423728480935097,
+            -0.03408990427851677,
+            0.006420265417546034,
+            0.01489350013434887,
+            -0.010348179377615452,
+            -0.020499395206570625,
+            -0.029650641605257988,
+            -0.024150803685188293,
+            -0.0038710979279130697,
+            -0.010855740867555141,
+            -0.016847988590598106,
+            0.006121031939983368,
+            -0.005802859086543322,
+            0.018499454483389854,
+            -0.007079336792230606,
+            0.033877789974212646,
+            0.006969491485506296,
+            -0.004056698642671108,
+            0.003835114184767008,
+            0.022469034418463707,
+            -0.006102093029767275,
+            0.02109028771519661,
+            0.004408961161971092,
+            -0.001467759720981121,
+            0.013681414537131786,
+            -0.006552837323397398,
+            -0.002361672930419445,
+            0.005821797996759415,
+            -0.010484539903700352,
+            0.022014502435922623,
+            0.005924067925661802,
+            -0.007708106189966202,
+            -0.0006775368237867951,
+            -0.011454207822680473,
+            0.014726839028298855,
+            0.006530110724270344,
+            0.0022480397019535303,
+            -0.011295121163129807,
+            0.011249667964875698,
+            -0.004859705455601215,
+            0.0040642740204930305,
+            0.009484568610787392,
+            -0.01774190180003643,
+            -0.028862785547971725,
+            0.011416330002248287,
+            0.019120648503303528,
+            0.006382387597113848,
+            -0.0055755930952727795,
+            -0.0025681061670184135,
+            0.00839369185268879,
+            0.010651201009750366,
+            0.022150862962007523,
+            -0.012984465807676315,
+            -8.889415767043829e-05,
+            0.013204156421124935,
+            -0.007170243188738823,
+            -0.01056029461324215,
+            0.00487485621124506,
+            0.023438703268766403,
+            0.0068823727779090405,
+            -0.04327145218849182,
+            -0.018469152972102165,
+            0.01909034699201584,
+            0.007136153522878885,
+            -0.04154422879219055,
+            0.0026381798088550568,
+            -0.003689285134896636,
+            -0.023787178099155426,
+            0.016105586662888527,
+            -0.0011817833874374628,
+            -0.01828734017908573,
+            0.028756728395819664,
+            0.02830219641327858,
+            0.01834794320166111,
+            2.6987840101355687e-05,
+            -0.013332940638065338,
+            -0.0318172425031662,
+            -0.023590214550495148,
+            0.008280059322714806,
+            -0.02660527639091015,
+            -0.0072725131176412106,
+            0.011242092587053776,
+            -0.017454031854867935,
+            0.02972639724612236,
+            -0.008507325313985348,
+            0.026938600465655327,
+            0.011446632444858551,
+            0.001691238023340702,
+            -0.013446573168039322,
+            -0.018802475184202194,
+            -0.01350717805325985,
+            -0.008196728304028511,
+            0.014090493321418762,
+            0.017923714593052864,
+            0.011007251217961311,
+            0.011332998983561993,
+            0.02251448854804039,
+            -0.012158732861280441,
+            0.007969462312757969,
+            -2.888172457460314e-05,
+            -0.008961857296526432,
+            0.0026343921199440956,
+            -0.018393397331237793,
+            0.00868156272917986,
+            0.0062308767810463905,
+            0.005458172410726547,
+            -0.007208121009171009,
+            0.010742107406258583,
+            0.02712041325867176,
+            -0.018847929313778877,
+            0.011401179246604443,
+            -0.031362712383270264,
+            -0.014787442982196808,
+            -0.01056029461324215,
+            0.026862844824790955,
+            0.007757347077131271,
+            -0.015726808458566666,
+            -0.007514929864555597,
+            -0.01545409020036459,
+            0.012885984033346176,
+            0.0032442226074635983,
+            0.005764981731772423,
+            0.015817714855074883,
+            0.021893294528126717,
+            0.01746918261051178,
+            -0.00795431062579155,
+            0.006852070800960064,
+            0.004674104508012533,
+            0.028090080246329308,
+            -0.01777220331132412,
+            0.020802417770028114,
+            0.009954252280294895,
+            -0.003935490269213915,
+            0.005685438401997089,
+            -0.014257155358791351,
+            0.008530051447451115,
+            -0.007325541693717241,
+            0.030514251440763474,
+            0.0033067206386476755,
+            0.014075342565774918,
+            -0.02062060497701168,
+            0.006575563922524452,
+            -0.03172633796930313,
+            0.009022461250424385,
+            0.013704141601920128,
+            0.012976890429854393,
+            -0.007677804213017225,
+            -0.024181105196475983,
+            -0.0119314668700099,
+            0.0057990713976323605,
+            -0.01279507763683796,
+            -0.01046181283891201,
+            -0.0034828518982976675,
+            0.012446602806448936,
+            0.018847929313778877,
+            -0.01848430372774601,
+            -0.019923655316233635,
+            -0.009583051316440105,
+            0.02898399345576763,
+            -0.015787413343787193,
+            -0.001471547526307404,
+            0.02413565292954445,
+            0.021241798996925354,
+            0.015022284351289272,
+            0.002155239460989833,
+            0.020090317353606224,
+            -0.0007111532613635063,
+            -0.0237265732139349,
+            -0.02813553437590599,
+            0.007302815094590187,
+            -0.006912675220519304,
+            0.01943882182240486,
+            -0.0035548193845897913,
+            -0.005658924113959074,
+            -0.0034033088013529778,
+            -0.007503566797822714,
+            -0.0165904201567173,
+            -0.01787826046347618,
+            0.0021647089160978794,
+            -0.008454295806586742,
+            0.005863463506102562,
+            -0.0002795845502987504,
+            -0.013105674646794796,
+            0.013401119969785213,
+            -0.014075342565774918,
+            0.005772557109594345,
+            -0.002653330797329545,
+            0.001962063368409872,
+            0.0035926972050219774,
+            0.009113367646932602,
+            -0.007336904760450125,
+            -0.06381630152463913,
+            -0.013105674646794796,
+            -0.003704436356201768,
+            0.006071790587157011,
+            -0.00019578021601773798,
+            -0.016635872423648834,
+            -0.013196581043303013,
+            0.01706010289490223,
+            0.008863375522196293,
+            -0.013870803639292717,
+            0.016954045742750168,
+            0.009961827658116817,
+            0.003622999181970954,
+            -0.024044746533036232,
+            -0.0008148433989845216,
+            0.008757317438721657,
+            0.032756611704826355,
+            -0.008643684908747673,
+            -0.0003839849086944014,
+            0.0036684523802250624,
+            -0.0033332351595163345,
+            -0.010052734054625034,
+            -0.014582903124392033,
+            -0.011848135851323605,
+            0.011908739805221558,
+            0.01790856383740902,
+            -0.014264730736613274,
+            -0.02007516659796238,
+            -0.0007106798002496362,
+            0.012931437231600285,
+            -0.026590125635266304,
+            0.0007244104635901749,
+            -0.007787649054080248,
+            -0.01052999310195446,
+            0.018302490934729576,
+            -0.019014591351151466,
+            0.005249845329672098,
+            0.00048294031876139343,
+            0.008507325313985348,
+            -0.023544760420918465,
+            -0.024893205612897873,
+            0.0035131541080772877,
+            -0.023847781121730804,
+            0.03612014651298523,
+            -0.021590271964669228,
+            -0.012082977220416069,
+            0.034483831375837326,
+            -0.0037423139438033104,
+            0.01702980138361454,
+            0.0023995505180209875,
+            0.02739313244819641,
+            -0.011439057067036629,
+            -0.00617784820497036,
+            0.012499631382524967,
+            0.009575475007295609,
+            0.0032593735959380865,
+            0.03775646165013313,
+            -0.01257538702338934,
+            -0.013901105150580406,
+            -0.005556654185056686,
+            -0.012954163365066051,
+            0.0006775368237867951,
+            0.014431392773985863,
+            0.028393102809786797,
+            0.004257450345903635,
+            -0.00628769351169467,
+            -0.006143758539110422,
+            -0.0014649189542979002,
+            -0.004242299124598503,
+            -0.00947699323296547,
+            -0.00715130427852273,
+            0.017287369817495346,
+            -0.025241680443286896,
+            0.016711628064513206,
+            -0.006420265417546034,
+            0.021135739982128143,
+            -0.011757229454815388,
+            0.007007369305938482,
+            -0.008961857296526432,
+            0.003492321353405714,
+            0.0023635667748749256,
+            -0.005109698045998812,
+            0.009287605062127113,
+            -0.014772292226552963,
+            0.008454295806586742,
+            0.013742019422352314,
+            0.0011183382011950016,
+            -0.010590597055852413,
+            -0.019272159785032272,
+            0.0022972808219492435,
+            -0.000710206339135766,
+            0.01181783340871334,
+            -0.0047498601488769054,
+            -0.000981031684204936,
+            0.0002921315317507833,
+            -0.014476845972239971,
+            -0.009575475007295609,
+            0.0006050958181731403,
+            0.005302873905748129,
+            -0.008961857296526432,
+            0.010378481820225716,
+            -0.005693014245480299,
+            -0.00034160923678427935,
+            0.02627195231616497,
+            0.006526322569698095,
+            -0.007632351014763117,
+            -0.014098069630563259,
+            0.01563590206205845,
+            0.015370759181678295,
+            -0.007454325910657644,
+            0.01933276280760765,
+            -0.016196493059396744,
+            -0.0008366230758838356,
+            0.013916256837546825,
+            -0.009219424799084663,
+            0.0017698341980576515,
+            -0.008196728304028511,
+            0.03554440662264824,
+            -0.0008787619881331921,
+            -0.0035415622405707836,
+            -0.0006401326390914619,
+            0.012976890429854393,
+            -0.0026779514737427235,
+            0.001182730309665203,
+            0.01132542360574007,
+            0.005321812815964222,
+            0.0185146052390337,
+            0.00328020635060966,
+            -0.03378688171505928,
+            -0.007780073676258326,
+            -0.0065490491688251495,
+            -0.008090670220553875,
+            0.009022461250424385,
+            0.025196226313710213,
+            0.01990850456058979,
+            0.006408901885151863,
+            0.001881573349237442,
+            0.018423698842525482,
+            0.015757111832499504,
+            -0.01489350013434887,
+            0.016363153234124184,
+            0.02802947722375393,
+            -0.010030006989836693,
+            0.022741753607988358,
+            -0.005253633018583059,
+            -0.008893677033483982,
+            0.008946705609560013,
+            -0.0015378334792330861,
+            -0.017393426969647408,
+            -0.015681356191635132,
+            0.005022579338401556,
+            -0.008999735116958618,
+            -0.0022897052112966776,
+            -0.018605511635541916,
+            0.004670316819101572,
+            0.017105557024478912,
+            -0.004965762607753277,
+            -3.5243989259470254e-05,
+            0.019484274089336395,
+            -0.006965703796595335,
+            -0.016908591613173485,
+            0.006325571332126856,
+            -0.015007133595645428,
+            0.007946735247969627,
+            -0.013666263781487942,
+            0.010492115281522274,
+            -0.006098304875195026,
+            0.018817627802491188,
+            0.005446809343993664,
+            -0.008090670220553875,
+            -0.0008958069374784827,
+            -0.018984289839863777,
+            0.02065090648829937,
+            -0.033089932054281235,
+            0.00839369185268879,
+            -0.026226500049233437,
+            -0.0009781909175217152,
+            0.0008849170990288258,
+            0.014605630189180374,
+            -0.001218714052811265,
+            0.019696388393640518,
+            -0.002751812804490328,
+            -0.020090317353606224,
+            -0.007086912170052528,
+            -0.0010634156642481685,
+            -0.01618134044110775,
+            -0.030923331156373024,
+            -0.004272601567208767,
+            0.0036267871037125587,
+            -0.004711982328444719,
+            0.0021135739516466856,
+            0.017984317615628242,
+            -0.005613470915704966,
+            0.009173971600830555,
+            -0.02690829709172249,
+            -0.007416448090225458,
+            -0.00355103169567883,
+            0.0010776197304949164,
+            -0.011242092587053776,
+            0.0010596278589218855,
+            0.011226941831409931,
+            -0.019044892862439156,
+            0.013461724855005741,
+            -0.021378157660365105,
+            -0.004371083341538906,
+            -0.008507325313985348,
+            -0.00135412672534585,
+            -0.013446573168039322,
+            -0.028120383620262146,
+            0.025029566138982773,
+            -0.00021495578403118998,
+            -0.003528305096551776,
+            -0.011999646201729774,
+            0.023181134834885597,
+            0.002914686920121312,
+            0.027817361056804657,
+            -0.03157482668757439,
+            0.0013427634257823229,
+            0.011552689597010612,
+            0.0025984083767980337,
+            -0.017287369817495346,
+            0.004746072459965944,
+            -0.005234694108366966,
+            0.02210540883243084,
+            -0.0011694730492308736,
+            -0.0005340751959010959,
+            -0.008234606124460697,
+            0.030999086797237396,
+            0.009136094711720943,
+            0.0012944694608449936,
+            -0.013226882554590702,
+            -0.008060368709266186,
+            -0.002759388415142894,
+            -0.0011069749016314745,
+            0.012855681590735912,
+            0.01290113478899002,
+            -0.012105703353881836,
+            0.0013607552973553538,
+            -0.006878585088998079,
+            0.0011694730492308736,
+            -0.002935519441962242,
+            -0.0004966709529981017,
+            -0.01149208564311266,
+            0.003257479751482606,
+            0.016272246837615967,
+            -0.00780280027538538,
+            -0.006480869371443987,
+            0.0007750718505121768,
+            0.009181547909975052,
+            0.012461753562092781,
+            -0.012742049060761929,
+            -0.018226735293865204,
+            -0.011908739805221558,
+            0.016029831022024155,
+            0.010522416792809963,
+            -0.008204303681850433,
+            -0.013234458863735199,
+            -0.02583257108926773,
+            0.01169662456959486,
+            -0.03293842449784279,
+            -0.012242063879966736,
+            0.005821797996759415,
+            0.028559764847159386,
+            0.019999410957098007,
+            0.012226912193000317,
+            0.001332347048446536,
+            0.002950670663267374,
+            0.00023342114582192153,
+            -0.01916610263288021,
+            0.014408666640520096,
+            -0.011143610812723637,
+            -0.018166130408644676,
+            0.0075906855054199696,
+            0.011939042247831821,
+            0.03148392215371132,
+            0.006942977197468281,
+            -0.0010160685051232576,
+            0.01686313934624195,
+            -0.001218714052811265,
+            -0.013582932762801647,
+            0.01420412678271532,
+            -0.024256860837340355,
+            0.016438908874988556,
+            0.02860521711409092,
+            -0.004780162125825882,
+            0.0008531945641152561,
+            -0.020938776433467865,
+            0.00482561532407999,
+            -0.011529963463544846,
+            -0.002177966060116887,
+            -0.003287781961262226,
+            -0.03460504114627838,
+            0.007946735247969627,
+            -0.006825556512922049,
+            0.01046181283891201,
+            -0.00814369972795248,
+            0.006655106786638498,
+            -0.012431452050805092,
+            0.004253662656992674,
+            0.01513591781258583,
+            0.016105586662888527,
+            0.007060397882014513,
+            -0.025211378931999207,
+            -0.009196698665618896,
+            0.008105821907520294,
+            -0.01349202636629343,
+            -0.02295386977493763,
+            -0.006852070800960064,
+            0.009840618818998337,
+            -0.01061332318931818,
+            -0.00488243205472827,
+            -0.028180986642837524,
+            -0.013681414537131786,
+            -0.0014327229000627995,
+            -0.017817657440900803,
+            -0.005556654185056686,
+            -0.003100287402048707,
+            0.01571165770292282,
+            -0.0049733384512364864,
+            -0.006185423582792282,
+            0.004742284305393696,
+            -0.023878084495663643,
+            0.0037423139438033104,
+            -0.008280059322714806,
+            0.0008266801596619189,
+            -0.0032517979852855206,
+            -0.0014762822538614273,
+            -0.007969462312757969,
+            -0.014332910999655724,
+            0.006893736310303211,
+            -0.016469212248921394,
+            0.002954458352178335,
+            -0.015173795633018017,
+            -0.016499513760209084,
+            -0.00682934420183301,
+            0.00775355938822031,
+            0.021575121209025383,
+            0.006541473791003227,
+            0.021484214812517166,
+            0.005469535943120718,
+            0.028680972754955292,
+            0.0003953482082579285,
+            0.003041577059775591,
+            -8.958956641436089e-06,
+            0.003232859307900071,
+            0.019347915425896645,
+            0.006571775767952204,
+            0.018984289839863777,
+            -0.008742166683077812,
+            0.010514841414988041,
+            -0.010355755686759949,
+            0.010014856234192848,
+            0.006333146709948778,
+            -0.006473293993622065,
+            -0.003799130441620946,
+            -0.010348179377615452,
+            -0.0031419529113918543,
+            -0.016544966027140617,
+            0.0027366618160158396,
+            0.00693161366507411,
+            0.0071588801220059395,
+            -0.011582992039620876,
+            -0.005230906419456005,
+            -0.014704111963510513,
+            -0.0064354161731898785,
+            -0.01831764169037342,
+            0.014385939575731754,
+            0.029226411134004593,
+            -0.016332851722836494,
+            -0.01977214403450489,
+            -0.007772498298436403,
+            0.0007296186522580683,
+            -0.006636167876422405,
+            -0.01240872498601675,
+            -0.009151245467364788,
+            -0.035059571266174316,
+            -0.017666146159172058,
+            0.007586897816509008,
+            0.02484775334596634,
+            0.01387837901711464,
+            -0.013628385961055756,
+            -0.0006983695784583688,
+            -0.016984347254037857,
+            -0.010787560604512691,
+            -0.01535560842603445,
+            -0.02054484933614731,
+            -0.02498411200940609,
+            0.0077952248975634575,
+            -0.004033972043544054,
+            0.014931377954781055,
+            0.008113397285342216,
+            0.0032631612848490477,
+            -0.004371083341538906,
+            -0.0024109138175845146,
+            -0.013196581043303013,
+            0.0012925754999741912,
+            -0.004787737503647804,
+            -0.0017291157273575664,
+            0.03554440662264824,
+            -0.008098246529698372,
+            -0.005871038883924484,
+            0.011060279794037342,
+            0.011128460057079792,
+            -0.013764745555818081,
+            0.0006642796797677875,
+            -0.011249667964875698,
+            0.021847840398550034,
+            -0.021620575338602066,
+            0.019257009029388428,
+            0.0014507147716358304,
+            -0.010840589180588722,
+            0.028999146074056625,
+            0.01892368495464325,
+            0.00743159931153059,
+            0.006579351611435413,
+            0.01672677882015705,
+            0.0044316877610981464,
+            0.016196493059396744,
+            -0.0026552248746156693,
+            -0.004632438998669386,
+            0.009295180439949036,
+            0.015681356191635132,
+            -0.000702630786690861,
+            -0.006942977197468281,
+            0.016438908874988556,
+            0.004056698642671108,
+            0.0075414446182549,
+            -0.01356020662933588,
+            -0.008961857296526432,
+            0.001507531269453466,
+            0.006257391534745693,
+            -0.005939218681305647,
+            -0.0026003022212535143,
+            0.005530139897018671,
+            0.003943065647035837,
+            -0.013302638195455074,
+            0.01689344085752964,
+            -0.016166189685463905,
+            0.008742166683077812,
+            -0.0037972365971654654,
+            0.0121360057964921,
+            -0.006799042224884033,
+            0.0013153020991012454,
+            -0.01818128302693367,
+            -0.003869204083457589,
+            0.0038597346283495426,
+            0.012173883616924286,
+            -0.01262841559946537,
+            0.025484098121523857,
+            0.008166425861418247,
+            0.006602078210562468,
+            0.0059467945247888565,
+            -0.012855681590735912,
+            0.027211319655179977,
+            0.005969521123915911,
+            0.009886072017252445,
+            -0.008893677033483982,
+            0.011136035434901714,
+            0.035362593829631805,
+            0.03060515783727169,
+            0.014135947450995445,
+            0.004469565115869045,
+            -0.018999440595507622,
+            0.015120766125619411,
+            0.0018427487229928374,
+            0.0023806116078048944,
+            0.00863610953092575,
+            -0.011348150670528412,
+            0.01305264513939619,
+            -0.009484568610787392,
+            0.015241974964737892,
+            -0.021272100508213043,
+            -0.011749654076993465,
+            -0.014408666640520096,
+            -0.021075136959552765,
+            -0.01430260855704546,
+            0.034180812537670135,
+            -0.002346521709114313,
+            -0.010393633507192135,
+            0.0045112306252121925,
+            0.008477022871375084,
+            -0.006079366430640221,
+            0.0053142374381423,
+            -0.008204303681850433,
+            -0.013749594800174236,
+            -0.01889338344335556,
+            0.006628592498600483,
+            -0.01550711877644062,
+            0.007651289459317923,
+            -0.003304826794192195,
+            0.0037688282318413258,
+            0.007014944683760405,
+            0.018105527386069298,
+            0.00530666159465909,
+            -0.00135412672534585,
+            0.0014033677289262414,
+            0.0016865032957866788,
+            -0.00931033119559288,
+            0.0108330138027668,
+            0.015408637002110481,
+            -0.0011921997647732496,
+            -0.011279970407485962,
+            -0.009136094711720943,
+            -0.042877525091171265,
+            0.0006264019757509232,
+            -0.013128400780260563,
+            0.0064354161731898785,
+            -0.011196639388799667,
+            0.0022234192583709955,
+            0.015863168984651566,
+            -0.0054051438346505165,
+            -0.0010918239131569862,
+            0.02193874679505825,
+            0.013014768250286579,
+            0.021075136959552765,
+            0.02065090648829937,
+            0.014188976027071476,
+            -0.015302578918635845,
+            -0.03469594568014145,
+            0.002221525413915515,
+            -0.010030006989836693,
+            0.013772320933640003,
+            0.004238511435687542,
+            0.016060132533311844,
+            -0.023893235251307487,
+            -0.011878437362611294,
+            -0.006662682630121708,
+            0.014431392773985863,
+            0.01491622719913721,
+            -0.0032271775417029858,
+            0.02690829709172249,
+            0.009280029684305191,
+            0.0020510759204626083,
+            -0.02281750924885273,
+            0.003312402404844761,
+            0.005321812815964222,
+            -0.008370965719223022,
+            -0.016681326553225517,
+            -0.0004552422615233809,
+            -0.0012168202083557844,
+            -0.0007817004225216806,
+            -0.021650876849889755,
+            0.018832778558135033,
+            -0.004018820822238922,
+            0.0037574649322777987,
+            0.013840501196682453,
+            0.01516621932387352,
+            0.017984317615628242,
+            0.008370965719223022,
+            0.001497115008533001,
+            -0.006605865899473429,
+            -0.0007973249885253608,
+            -0.0005828426801599562,
+            -0.007389933802187443,
+            -0.005598319694399834,
+            0.006382387597113848,
+            -0.01134057529270649,
+            0.008810346014797688,
+            0.000896753859706223,
+            0.006113456096500158,
+            0.010605747811496258,
+            -0.004836978856474161,
+            -0.011628445237874985,
+            0.014628356322646141,
+            -0.0010548931313678622,
+            -0.0055755930952727795,
+            -0.011060279794037342,
+            0.003624893259257078,
+            0.013575357384979725,
+            -0.003096499713137746,
+            -0.013499601744115353,
+            0.00910579226911068,
+            0.002009410411119461,
+            -0.027438584715127945,
+            0.015499543398618698,
+            0.010295150801539421,
+            -0.013370818458497524,
+            0.004980913829058409,
+            -0.003371112747117877,
+            -0.03027183562517166,
+            0.004791525658220053,
+            0.002660906407982111,
+            -0.010643625631928444,
+            0.028938541188836098,
+            -0.003821857040748,
+            0.029256712645292282,
+            -0.004980913829058409,
+            -0.006905099377036095,
+            -0.011355726048350334,
+            0.009416389279067516,
+            -0.018681267276406288,
+            -0.0037953427527099848,
+            -0.008727015927433968,
+            -0.00915882084518671,
+            -0.0009838725673034787,
+            -0.005185453221201897,
+            0.006984642706811428,
+            0.005121061112731695,
+            -0.012704171240329742,
+            0.006757376715540886,
+            -0.010257272981107235,
+            -0.00020856392802670598,
+            0.0006718551740050316,
+            -0.008408842608332634,
+            0.013885954394936562,
+            0.005996035411953926,
+            -0.008340663276612759,
+            -0.0005032995832152665,
+            0.027817361056804657,
+            0.05051366239786148,
+            -0.018863080069422722,
+            0.0009270560112781823,
+            0.01452229917049408,
+            -0.0028824908658862114,
+            -0.0021590273827314377,
+            -0.011976920068264008,
+            0.005821797996759415,
+            -0.019423669204115868,
+            0.013317788951098919,
+            0.006473293993622065,
+            -0.008833073079586029,
+            0.003202557098120451,
+            -0.016711628064513206,
+            0.002621134975925088,
+            -0.0018058179412037134,
+            -0.018681267276406288,
+            -0.015529844909906387,
+            -0.012060250155627728,
+            -0.024181105196475983,
+            0.0004235197266098112,
+            -0.0008546149474568665,
+            -0.013870803639292717,
+            0.006579351611435413,
+            0.008431569673120975,
+            0.032483890652656555,
+            0.0011912527261301875,
+            0.010128489695489407,
+            -2.1513333194889128e-05,
+            -0.0010416359873488545,
+            -0.005988460034132004,
+            -0.01523439958691597,
+            0.00015387803432531655,
+            -0.002342734020203352,
+            0.022590242326259613,
+            0.005549078807234764,
+            0.017560089007019997,
+            0.0017887729918584228,
+            -0.016514664515852928,
+            -0.00596573343500495,
+            -0.007408872712403536,
+            -0.018060073256492615,
+            0.0062649669125676155,
+            -0.005473323632031679,
+            0.009605777449905872,
+            -0.015863168984651566,
+            -0.02731737680733204,
+            0.00792400911450386,
+            0.012234487570822239,
+            0.028514310717582703,
+            -0.0035623949952423573,
+            0.001716805505566299,
+            -0.005533927585929632,
+            0.016408607363700867,
+            0.032180868089199066,
+            -0.0008550884085707366,
+            0.006950552575290203,
+            -0.019999410957098007,
+            0.0017404790269210935,
+            0.013704141601920128,
+            -0.008401267230510712,
+            -0.01821158453822136,
+            0.003939277958124876,
+            -0.020635755732655525,
+            0.004583198111504316,
+            -0.020393338054418564,
+            0.030408194288611412,
+            -0.0023522034753113985,
+            0.008105821907520294,
+            0.004310478921979666,
+            0.01760554127395153,
+            -0.0031646795105189085,
+            0.010545143857598305,
+            0.013916256837546825,
+            0.014908651821315289,
+            0.0009980766335502267,
+            -0.011189064010977745,
+            -0.03790797293186188,
+            0.001859793672338128,
+            0.00817400123924017,
+            0.018635815009474754,
+            -0.01980244740843773,
+            0.011681473813951015,
+            -0.01296931505203247,
+            -0.020363036543130875,
+            -0.01003758329898119,
+            0.011575416661798954,
+            -0.006139970384538174,
+            0.03378688171505928,
+            -0.005874827038496733,
+            -0.008204303681850433,
+            0.005136212334036827,
+            0.010454237461090088,
+            0.0066778333857655525,
+            0.014704111963510513,
+            0.0326354019343853,
+            0.010196669027209282,
+            0.014173824340105057,
+            -0.004859705455601215,
+            0.02399929240345955,
+            -0.003931702114641666,
+            0.003954428713768721,
+            0.00509454682469368,
+            -0.022832660004496574,
+            -0.0040983641520142555,
+            -0.003931702114641666,
+            -0.016711628064513206,
+            -0.007458113599568605,
+            0.0032366469968110323,
+            0.008802770636975765,
+            -0.014045040123164654,
+            -0.00649980828166008,
+            -0.01350717805325985,
+            0.01218903437256813,
+            0.002730980049818754,
+            0.018302490934729576,
+            -0.029802151024341583,
+            0.009825468063354492,
+            -0.017484333366155624,
+            -0.017044952139258385,
+            0.006511171814054251,
+            -0.0038332203403115273,
+            0.006855858489871025,
+            -0.018272189423441887,
+            -0.01318900566548109,
+            0.012817803770303726,
+            0.0007897494360804558,
+            0.013060221448540688,
+            -0.009825468063354492,
+            0.0009706153068691492,
+            0.006064215209335089,
+            -0.009833043441176414,
+            -0.010961798019707203,
+            -0.007977037690579891,
+            -0.009681533090770245,
+            -0.008848223835229874,
+            0.005621046293526888,
+            0.01656011864542961,
+            -0.013742019422352314,
+            -0.003685497445985675,
+            0.016135888174176216,
+            0.006416477728635073,
+            -0.008166425861418247,
+            0.004075637552887201,
+            -0.003793448908254504,
+            -9.948806837201118e-05,
+            0.0013740125577896833,
+            0.012439027428627014,
+            -0.0017632056260481477,
+            0.024665940552949905,
+            -0.02654467150568962,
+            0.007598260883241892,
+            -0.0054051438346505165,
+            0.005446809343993664,
+            0.018226735293865204,
+            -0.02366596832871437,
+            -0.013082947582006454,
+            0.006276329979300499,
+            -0.01966608688235283,
+            -0.008060368709266186,
+            0.0054051438346505165,
+            -0.00016405766655225307,
+            -0.019923655316233635,
+            -0.012923861853778362,
+            -0.034786853939294815,
+            -0.02309022843837738,
+            0.024665940552949905,
+            0.008401267230510712,
+            -0.0073672072030603886,
+            0.018060073256492615,
+            -0.018438849598169327,
+            0.0172722190618515,
+            -0.002225313102826476,
+            -0.006613441277295351,
+            0.003935490269213915,
+            0.007674016058444977,
+            0.025362888351082802,
+            0.002653330797329545,
+            0.008264907635748386,
+            -0.0042082094587385654,
+            -0.014931377954781055,
+            -0.006321783177554607,
+            0.01980244740843773,
+            0.02471139281988144,
+            0.018605511635541916,
+            -0.018575210124254227,
+            0.010742107406258583,
+            -0.011749654076993465,
+            0.01818128302693367,
+            -0.006537686102092266,
+            0.00039037675014697015,
+            0.01859036087989807,
+            0.007969462312757969,
+            0.025241680443286896,
+            0.019923655316233635,
+            0.006852070800960064,
+            0.014128371141850948,
+            0.011908739805221558,
+            0.001204509986564517,
+            -0.012832955457270145,
+            -0.01154511421918869,
+            -0.02840825356543064,
+            0.042847223579883575,
+            0.0025434857234358788,
+            0.008317936211824417,
+            -0.027635548263788223,
+            -0.00021389048197306693,
+            -0.0034942151978611946,
+            -0.018832778558135033,
+            0.005939218681305647,
+            0.03260510042309761,
+            0.03284751623868942,
+            0.021438762545585632,
+            -2.888172457460314e-05,
+            0.0012234487803652883,
+            0.0329081192612648,
+            -0.013673839159309864,
+            -0.00937851145863533,
+            -0.01034060399979353,
+            -0.018393397331237793,
+            -0.006852070800960064,
+            -0.0032423287630081177,
+            -0.01031030248850584,
+            -0.0061172437854111195,
+            -0.009272454306483269,
+            0.0238629337400198,
+            0.0001374249259242788,
+            -0.0022594030015170574,
+            0.006605865899473429,
+            -0.020878173410892487,
+            0.011537538841366768,
+            -0.02813553437590599,
+            -0.013431422412395477,
+            -0.008651260286569595,
+            -0.006950552575290203,
+            -0.018802475184202194,
+            0.008643684908747673,
+            -0.006942977197468281,
+            0.021438762545585632,
+            0.002914686920121312,
+            -0.023590214550495148,
+            0.007530081085860729,
+            0.015317730605602264,
+            -0.0003825644962489605,
+            0.014037464745342731,
+            -0.003890036838129163,
+            0.009060339070856571,
+            0.0045112306252121925,
+            0.011681473813951015,
+            0.016954045742750168,
+            0.010651201009750366,
+            0.009030036628246307,
+            0.003437398700043559,
+            -0.001167579204775393,
+            0.01878732442855835,
+            -0.0007163614500313997,
+            -0.03405960276722908,
+            2.9236827685963362e-05,
+            -0.014628356322646141,
+            0.029499130323529243,
+            -0.009795165620744228,
+            0.009514871053397655,
+            0.008105821907520294,
+            0.007098275702446699,
+            -0.0067119235172867775,
+            0.007916433736681938,
+            -0.008742166683077812,
+            0.0052763596177101135,
+            0.015832867473363876,
+            -0.0042877523228526115,
+            -0.0014980619307607412,
+            0.007840678095817566,
+            0.00407184986397624,
+            -0.005454384721815586,
+            -0.019711541011929512,
+            -0.012287517078220844,
+            0.00726114958524704,
+            -0.02065090648829937,
+            -0.004367295652627945,
+            0.01161329448223114,
+            -0.005454384721815586,
+            -0.009620928205549717,
+            0.01393140759319067,
+            0.02136300690472126,
+            -0.002382505452260375,
+            0.022893264889717102,
+            -0.01349202636629343,
+            -0.009833043441176414,
+            -0.007984613068401814,
+            -0.004374871030449867,
+            -0.041120000183582306,
+            0.025984082370996475,
+            0.01068150345236063,
+            -0.006871009711176157,
+            0.01895398646593094,
+            0.0022499337792396545,
+            -0.002954458352178335,
+            -0.0011041341349482536,
+            0.00010499217023607343,
+            -0.03281721472740173,
+            0.03660498186945915,
+            -0.007007369305938482,
+            0.0005449649761430919,
+            0.010272424668073654,
+            0.042029064148664474,
+            -0.004590773954987526,
+            -0.005859675817191601,
+            0.01225721463561058,
+            0.005336964037269354,
+            0.031938452273607254,
+            -0.010416359640657902,
+            -0.014340486377477646,
+            -0.01815097965300083,
+            -0.004833190701901913,
+            -0.01939336769282818,
+            -0.0038313264958560467,
+            0.014037464745342731,
+            -0.019650936126708984,
+            -0.011242092587053776,
+            -0.001287840772420168,
+            -0.012537509202957153,
+            -0.0010747789638116956,
+            0.008598231710493565,
+            0.017832808196544647,
+            -0.011348150670528412,
+            -0.004450626205652952,
+            -2.2904154320713133e-05,
+            -0.005003640428185463,
+            0.0185146052390337,
+            0.0020056227222085,
+            -0.01702980138361454,
+            0.01321930717676878,
+            -0.00526499655097723,
+            0.006749800872057676,
+            0.0074429623782634735,
+            0.007939159870147705,
+            0.05145302787423134,
+            0.010317877866327763,
+            -0.001339922659099102,
+            -0.01607528328895569,
+            0.023196285590529442,
+            -0.01749948412179947,
+            -0.014181400649249554,
+            -0.02119634486734867,
+            -0.0029923359397798777,
+            -0.007060397882014513,
+            -0.028014326468110085,
+            -9.137987944995984e-05,
+            0.0016221113037317991,
+            -0.0024980325251817703,
+            -0.010045158676803112,
+            0.0005208179936744273,
+            -0.011529963463544846,
+            -0.006954340264201164,
+            -0.008545203134417534,
+            -0.016044981777668,
+            0.02240842953324318,
+            -0.021817538887262344,
+            -0.008855799213051796,
+            0.011514811776578426,
+            0.03118089959025383,
+            -0.005011215806007385,
+            -0.014173824340105057,
+            -0.019817598164081573,
+            0.0025643182452768087,
+            -0.01763584464788437,
+            0.012181458994746208,
+            -0.01225721463561058,
+            0.015045011416077614,
+            -0.03142331540584564,
+            -0.006079366430640221,
+            -0.008992158807814121,
+            0.012840530835092068,
+            0.019817598164081573,
+            -0.0025093958247452974,
+            0.00606800289824605,
+            -0.010295150801539421,
+            0.030211230739951134,
+            -0.00012286569108255208,
+            -0.018605511635541916,
+            0.00873459130525589,
+            0.007977037690579891,
+            0.012537509202957153,
+            -0.004435475450009108,
+            0.00257757562212646,
+            0.011567841283977032,
+            0.012742049060761929,
+            -0.01634800247848034,
+            -0.003174148965626955,
+            -0.017969166859984398,
+            0.005996035411953926,
+            0.006999793462455273,
+            0.021711481735110283,
+            -0.0013550736475735903,
+            0.018272189423441887,
+            -0.01563590206205845,
+            0.00019412307301536202,
+            -0.011749654076993465,
+            -0.0082270298153162,
+            -0.0028237802907824516,
+            -0.003958216868340969,
+            0.006685409229248762,
+            -0.028211290016770363,
+            -0.01274962443858385,
+            0.005287723150104284,
+            -0.004992276895791292,
+            -0.03405960276722908,
+            0.0053710537031292915,
+            -0.02725677192211151,
+            0.013128400780260563,
+            -0.002397656673565507,
+            -0.017847958952188492,
+            -0.00030846628942526877,
+            0.026332557201385498,
+            -0.003677921835333109,
+            -0.012537509202957153,
+            0.015544996596872807,
+            -0.006757376715540886,
+            -0.02119634486734867,
+            -0.005420294590294361,
+            0.014658658765256405,
+            -0.0010719380807131529,
+            -0.012393574230372906,
+            -0.029635490849614143,
+            0.012992041185498238,
+            0.0026874206960201263,
+            -0.007386146113276482,
+            0.010893618687987328,
+            0.025953780859708786,
+            0.008151275105774403,
+            -0.005924067925661802,
+            0.0015757111832499504,
+            0.0049581872299313545,
+            0.00300938100554049,
+            0.020287280902266502,
+            0.02728707529604435,
+            -0.008014915511012077,
+            -0.0011145505122840405,
+            -0.014954105019569397,
+            0.015802564099431038,
+            0.0013020449550822377,
+            0.010507266037166119,
+            0.021893294528126717,
+            0.007045246660709381,
+            -0.005738467443734407,
+            0.003905187826603651,
+            -0.023135682567954063,
+            0.01420412678271532,
+            -0.012272365391254425,
+            0.000622140767518431,
+            0.008749742060899734,
+            -0.007283876184374094,
+            0.01437078882008791,
+            -0.0015245762187987566,
+            0.0339989997446537,
+            0.025211378931999207,
+            0.0043218424543738365,
+            0.0007362472242675722,
+            -0.028559764847159386,
+            0.003547244006767869,
+            0.008651260286569595,
+            0.014499573037028313,
+            -0.0076664406806230545,
+            0.006261179223656654,
+            -0.004355932120233774,
+            0.00959820207208395,
+            -0.0065490491688251495,
+            -0.015893470495939255,
+            0.006113456096500158,
+            0.0012689019786193967,
+            0.006795254070311785,
+            0.0023408401757478714,
+            -0.008863375522196293,
+            -0.024362917989492416,
+            -0.025984082370996475,
+            0.021984200924634933,
+            -0.03918066248297691,
+            0.0073293293826282024,
+            0.011234517209231853,
+            0.03324144333600998,
+            -0.008158850483596325,
+            -0.004980913829058409,
+            0.006276329979300499,
+            -0.016711628064513206,
+            -0.004147605039179325,
+            0.008204303681850433,
+            0.012385998852550983,
+            -0.002420383272692561,
+            0.00974971242249012,
+            0.01318900566548109,
+            -0.012378423474729061,
+            0.03660498186945915,
+            0.016514664515852928,
+            0.01467380952090025,
+            -0.004007457755506039,
+            0.01312082540243864,
+            0.0066172294318675995,
+            0.019044892862439156,
+            0.025165924802422523,
+            -0.017454031854867935,
+            -0.002365460619330406,
+            -0.02292356640100479,
+            0.024165954440832138,
+            0.00726114958524704,
+            0.004556683823466301,
+            -0.008310360834002495,
+            0.014870774000883102,
+            0.0017954016802832484,
+            0.0028199926018714905,
+            -0.006696772295981646,
+            -0.010173942893743515,
+            0.019241856411099434,
+            0.0034317169338464737,
+            0.014870774000883102,
+            4.8501173296244815e-05,
+            -0.005503625608980656,
+            0.006052851676940918,
+            0.014832896180450916,
+            -0.023590214550495148,
+            -0.00203592493198812,
+            -0.019484274089336395,
+            -0.03202936053276062,
+            0.009022461250424385,
+            0.0016135887708514929,
+            -0.007052822504192591,
+            0.03230207785964012,
+            -0.010083036497235298,
+            -0.019787294790148735,
+            -0.008067944087088108,
+            -0.013113250024616718,
+            -0.010469388216733932,
+            0.0008271536789834499,
+            0.003908975515514612,
+            -0.00012428610352799296,
+            -0.009317907504737377,
+            -0.005749830510467291,
+            0.01990850456058979,
+            -0.0030358952935785055,
+            -0.00753386877477169,
+            0.00868156272917986
+          ],
+          "embedding": [
+            0.0040707582550429606,
+            -0.010230625151989194,
+            -0.021520242101903186,
+            -0.010115756270640508,
+            0.026738734586159035,
+            0.01796370233223267,
+            -0.012245199029994153,
+            0.06506924494147726,
+            -0.0014087587222033442,
+            0.009300068719271429,
+            0.003787396973047584,
+            0.002818772639754174,
+            0.022129077062176306,
+            -0.021526709808445683,
+            -0.0034388135367225824,
+            0.02763761747879405,
+            0.013210013805395276,
+            0.025255872672296602,
+            0.00029011480909986703,
+            -0.05438974762290585,
+            0.016671640187490865,
+            -0.016426252259893328,
+            -0.056591180651012554,
+            0.009335648425523075,
+            0.006141624459418827,
+            0.02568943956694641,
+            0.019377132054042116,
+            0.017946464337078746,
+            -0.057184776985927686,
+            0.03208527139096687,
+            0.030132392063755547,
+            -0.003846032371791839,
+            -0.008376524427531415,
+            -0.0010513963058713435,
+            -0.02734868388724371,
+            -0.014879400370545479,
+            0.04383920959748825,
+            0.02563613659516289,
+            -0.020117467074438997,
+            0.0398016699155716,
+            0.029896497884197775,
+            -0.0004195844530067313,
+            -0.023302720788253625,
+            -0.001168337916577517,
+            0.009149021369661628,
+            -0.02508145721900992,
+            -0.004307594138480351,
+            0.00550940415564303,
+            0.014016998918718476,
+            -0.028607379616884317,
+            -0.0018317656587526954,
+            -0.01973464940415756,
+            0.03762738392056255,
+            0.015358106058583114,
+            -0.03931907682104936,
+            0.011519637298832444,
+            0.004482080558226775,
+            0.0747867063383242,
+            0.05550679018263077,
+            0.023243129601931176,
+            0.02671631686665238,
+            -0.009686146566573536,
+            -0.04088582189725495,
+            -0.0065344623594924085,
+            0.019617295999325712,
+            -0.004722087402575777,
+            -0.002871844114437583,
+            0.003935087265300519,
+            0.03229890413393201,
+            -0.02813577047904649,
+            -0.05124172132597234,
+            0.03325235407841968,
+            -0.008687655557217192,
+            -0.011027174838246948,
+            -0.003653553063547375,
+            -0.02350737945182988,
+            0.016223087939183468,
+            0.03789252356462714,
+            0.0034350623378603083,
+            0.00195158657888739,
+            0.011789712862232108,
+            -0.0004435439660769551,
+            -0.0037330456918125894,
+            0.0031552713243866858,
+            0.04465432992298031,
+            -0.0032465421571915712,
+            -0.0075366743861922745,
+            -0.059958752731410256,
+            0.016784616861008125,
+            -0.028734506065772967,
+            -0.03035575736273943,
+            0.02680390892611741,
+            0.02338368719038086,
+            -0.0030658375437784277,
+            -0.03754896050794851,
+            0.07005659523248504,
+            0.0020711575553570695,
+            0.013034425083076923,
+            -0.02316832262132332,
+            0.021332142030732894,
+            -0.0318756233020585,
+            0.02270782509507388,
+            0.01570810309454529,
+            0.004788124787612736,
+            0.026727250200142456,
+            -0.022380142778058276,
+            -0.008593910379436904,
+            0.014439754382387034,
+            0.015007697617383174,
+            0.07662531963612977,
+            0.011195998137723795,
+            -0.010809283538470098,
+            -0.042227822120219305,
+            0.003824320933275175,
+            -0.03797629427607094,
+            -0.02347390684453348,
+            -0.0526876440280276,
+            -0.017799861680287913,
+            0.01701482796266917,
+            0.002730395144917155,
+            0.03789451759899825,
+            0.038338545350313946,
+            0.028463789278406586,
+            0.015494583705011921,
+            -0.05700237266350479,
+            0.047895170359075166,
+            -0.0064406894534929406,
+            -0.0007954721295065988,
+            0.012543029839729358,
+            -0.004703334868741216,
+            -0.0060569679284068,
+            0.012324901254711192,
+            -0.019883935809784068,
+            -0.04253565942154215,
+            0.02164409276211115,
+            -0.0075367555415808635,
+            0.03617159147722157,
+            0.07567745988686005,
+            -0.04225977020036783,
+            -0.022548837589622003,
+            0.029811602145850144,
+            -0.02175792196819533,
+            0.02069861458115363,
+            -0.00941331829664397,
+            -0.035479639885943264,
+            0.020352552580119105,
+            0.003678588588550461,
+            0.06776105274336729,
+            -0.02346047721661687,
+            0.02721075076349443,
+            -0.029386781707644253,
+            0.023530831921117993,
+            -0.02077660533100372,
+            0.020022475323372906,
+            0.024195118676513715,
+            -0.028378518720494666,
+            -0.012900969023913307,
+            0.019264089970719624,
+            -0.0374983624629265,
+            -0.019726171557384355,
+            0.00979636085642416,
+            -0.014479528435832063,
+            -0.014533384583777662,
+            0.005904623758767224,
+            0.010344475500263744,
+            0.05659646358026115,
+            0.04163866113482093,
+            0.007152494819169405,
+            -0.07584500130974763,
+            0.05471858597878099,
+            0.03580211013427615,
+            0.044161429444475306,
+            0.02258180566046248,
+            0.00980779221334715,
+            0.0011193505603397395,
+            0.018086566768892507,
+            0.015605066702191871,
+            -0.00954727399396313,
+            -0.027950079146919375,
+            0.03028514025485407,
+            -0.006351669761257431,
+            0.005095804801686436,
+            0.009620040085743162,
+            0.019748001227987134,
+            0.033709535059510916,
+            0.008203559956940196,
+            0.007389752602218563,
+            0.04197771878351742,
+            -0.02067363758471765,
+            0.007543739188471913,
+            0.011207941261799052,
+            0.013964827251172813,
+            0.03839562075248879,
+            0.0009775244543292637,
+            0.006469507611492216,
+            -0.022808235395746654,
+            -0.024795004398633292,
+            0.006249024864814684,
+            0.011292877109066925,
+            0.01596839619826819,
+            -0.0014105396434174883,
+            -0.04152075438277094,
+            0.008267788411079337,
+            -0.028834191212891387,
+            0.030940526405027906,
+            -0.0018271056434014425,
+            0.006185475047820603,
+            -0.007920574674727123,
+            0.012907560753043616,
+            -0.015165704767187252,
+            -0.021962244496450727,
+            0.0037071053072833147,
+            -0.013508376064336042,
+            -0.015178823540513864,
+            -0.02518557053061311,
+            0.011693256916760944,
+            -0.01716015674592331,
+            0.02048897513447166,
+            -0.003412078966361745,
+            -0.001807298511378529,
+            -0.0012575655363779904,
+            0.03116103961636643,
+            0.011668533361641842,
+            -0.011637010548030312,
+            -0.040260195687925236,
+            -0.011129376691522074,
+            -0.0034493180842871908,
+            -0.02098244789711689,
+            0.026658743116016938,
+            -0.01923736325953796,
+            -0.009723937181300192,
+            -0.013134679989565051,
+            -0.011974548914818263,
+            -0.004028712301052159,
+            0.019628329175867303,
+            -0.028192034966932684,
+            -0.012638520793012082,
+            -0.011568652932013111,
+            -0.0025919910519445737,
+            0.012497908463405111,
+            0.015113186712897145,
+            0.014880607940212641,
+            -0.03059227859452334,
+            -0.021899716217105084,
+            -0.0008513506358489523,
+            0.011985931118464679,
+            -0.01321584595585435,
+            -0.011735288366213981,
+            0.010710683541435907,
+            0.01961722172525993,
+            -0.023571946726192585,
+            -0.03364281880125354,
+            0.0014460076743756465,
+            0.016563167828467187,
+            -0.02891589059260944,
+            0.04595166331401113,
+            0.034271553243501465,
+            -0.027557971577291153,
+            -0.007138821520412525,
+            -0.039508407807651844,
+            0.06814329636830853,
+            0.00022136123564675344,
+            -0.039718016705753735,
+            0.0503457594963753,
+            -0.003580581891576583,
+            0.013820796398521455,
+            0.02054050419088757,
+            0.035682661890720495,
+            -0.0013220739026852943,
+            -0.0052759394697328675,
+            0.04733018719162478,
+            -0.0187845855131196,
+            -0.03982699468608443,
+            -0.023905460465070062,
+            0.002631233133039351,
+            -0.03090995554989513,
+            -0.012617612051488022,
+            -0.013605787969473513,
+            0.039672783793764745,
+            0.008669582242097126,
+            0.009974277556736362,
+            0.005535636914843583,
+            0.014180668493093584,
+            0.012042659947850441,
+            0.011870115226306776,
+            0.00451259454348581,
+            -0.014023048592193627,
+            -0.007769033996392623,
+            -0.05715987527668358,
+            0.003374564518384728,
+            -0.028801158523506985,
+            -0.00756331369170698,
+            -0.02163054894055572,
+            0.005367181573233089,
+            0.02308705505078254,
+            -0.005438909398653553,
+            -0.00873620909426313,
+            -0.0036530652237719077,
+            0.013920407174296543,
+            -0.01539613379172909,
+            -0.03908211213269472,
+            -0.016810726548844992,
+            0.007460173854386443,
+            -0.007069189426742819,
+            -0.041197222498539215,
+            -0.003843233633335066,
+            0.004373367480223065,
+            0.016194121660459908,
+            -0.029165150066068787,
+            -0.00735139702658342,
+            0.017257895082337448,
+            0.014153313280536095,
+            -0.002376178042741926,
+            -0.023223700836427403,
+            -0.0319877537205969,
+            0.02088060774052704,
+            0.0012829290529515294,
+            0.03251550040316258,
+            -0.04128453069131847,
+            -0.023376114412063578,
+            0.04083022735076889,
+            0.005690491480259003,
+            -0.05810403800689653,
+            0.016582434816685452,
+            0.010402038964417569,
+            0.016795644054313566,
+            -0.00908590043098552,
+            0.05003656634510042,
+            -0.037216197954986735,
+            -0.013027960261454162,
+            -0.007090943677204207,
+            -0.05175371787508879,
+            -0.0020065660448063322,
+            -0.014908489552053589,
+            0.010627472758306387,
+            -0.03275473161158777,
+            -0.017334608434133866,
+            0.025321047439048318,
+            -0.0033515195870701753,
+            0.011526068468647207,
+            -0.026783879816504975,
+            -0.0012830935419413407,
+            -0.017506908439998053,
+            -0.02114853058440477,
+            0.004878884893669383,
+            -0.020936900837115968,
+            -0.0025587027781733642,
+            0.01557850246191957,
+            0.023652335794299834,
+            0.011489428313303988,
+            0.0019089137782552768,
+            0.02054907941022243,
+            0.014361232983780595,
+            -0.0056413109308506085,
+            -0.0010300050480036684,
+            -0.03413262505887213,
+            -0.018821160485010483,
+            0.03361989907065425,
+            0.013125889701529858,
+            -0.003742167496379789,
+            0.008813100942466033,
+            -0.0383265293638935,
+            -0.035316957263713485,
+            0.03020499323039515,
+            -0.05001623971243151,
+            -0.00855479667673597,
+            -0.005939674940109814,
+            -0.026451831943555135,
+            -0.008032873049527808,
+            0.0009431571685654594,
+            -0.014387605128858954,
+            0.016382559541755014,
+            -0.024909510232038502,
+            0.003715736505444385,
+            -0.006016712902409688,
+            -0.009110197692342108,
+            0.05507263160354303,
+            -0.022103031032598353,
+            -0.0475415595313825,
+            0.021958434235402655,
+            -0.016670809659931243,
+            0.0323519363801999,
+            0.03867494951405658,
+            0.011124508964363016,
+            -0.008002445714431074,
+            0.034041838454818577,
+            0.018400123235267234,
+            -0.01161754789471199,
+            -0.025901844576206774,
+            0.021800298132870245,
+            0.00943488425697727,
+            -0.05906691123008531,
+            -0.00782549274387971,
+            -0.019357446956797447,
+            0.008682376344965976,
+            0.02863077694455766,
+            0.0024925977373894447,
+            -0.002682170968419412,
+            -0.013590235283325244,
+            -0.021511149507762284,
+            -0.03255453916518719,
+            0.009039307015853905,
+            0.008305286476500875,
+            0.009312625027668458,
+            -0.0014144165766983945,
+            -0.007853869334376857,
+            0.009067065760942163,
+            0.022485201611182105,
+            0.03966189080765722,
+            -0.041627289519629344,
+            0.002707086456239785,
+            -0.022678178050044222,
+            -0.03653753142622053,
+            0.0036574646778835588,
+            0.029996160786576367,
+            0.02901163765559891,
+            -0.01971777193183787,
+            -0.0036589934841109884,
+            0.009181234938554578,
+            -0.025979483002227535,
+            -0.00773645103995767,
+            0.004280770859737866,
+            0.04116668301017644,
+            0.006201085381597264,
+            0.01402728258112097,
+            0.001060407317110527,
+            -0.00920801784000087,
+            0.02109880889040528,
+            0.025211022083982727,
+            -0.029866678264942617,
+            0.014439702646820226,
+            0.009379206296908577,
+            0.04156435812161122,
+            -0.037309655469525846,
+            -0.011819731348299495,
+            -0.02766498020409985,
+            -0.010454143230794476,
+            0.027911532296566614,
+            -0.04127890742215716,
+            -0.006948536183086887,
+            0.010207784113909938,
+            -0.0016584043356136647,
+            0.016437495428919636,
+            -0.014607861750686952,
+            -0.03222311919805017,
+            -0.000610723203118213,
+            -0.02867445198803213,
+            -0.024772816792038133,
+            -0.016510309753766612,
+            -0.018581201775667577,
+            -0.07450654873867177,
+            0.00026832248829154255,
+            -0.06724696990816072,
+            -0.02447002541264633,
+            0.04885569249258513,
+            -0.010195604207134053,
+            0.03332905935704749,
+            -0.02815272112271314,
+            -0.005296340304911926,
+            -0.05908835042526019,
+            0.0038476172058970774,
+            0.025753576863447866,
+            0.02386003857017594,
+            0.005206447371391696,
+            0.024799057871126032,
+            0.012060738268265606,
+            -0.04146503002980051,
+            -0.024949907556862643,
+            -0.014503475223770958,
+            -0.0016340466065744286,
+            -0.0034009167013795906,
+            -0.031298432995411005,
+            0.036241272149880994,
+            0.008902644391348814,
+            0.03550223922667804,
+            0.04463301409716232,
+            -0.010385984413759852,
+            -0.03305004479734889,
+            -0.006952739338820577,
+            -0.0175981588092108,
+            0.042668533202572856,
+            -0.017112184620086308,
+            0.005149173014526686,
+            -0.03219589107997635,
+            0.024181808150698054,
+            0.0014339790422945406,
+            0.01254721190611376,
+            0.011099458649404119,
+            0.0027626836767364253,
+            -0.026883640446090158,
+            0.0024160770766463607,
+            0.023874514847956222,
+            -0.022558433617669373,
+            0.00917667715473093,
+            -0.006754091166465801,
+            -0.033686935718776144,
+            -0.01078389638118685,
+            -0.013595327486167114,
+            0.024712781935748277,
+            0.013168464807451798,
+            0.005875864604862434,
+            -0.02417814420558463,
+            -0.0023978800459714144,
+            0.028650463366555304,
+            0.020176980768301288,
+            -0.02118184201861616,
+            0.019082340279671355,
+            0.019734925740100277,
+            -0.04209645071102001,
+            -0.04217310001316092,
+            0.006553913719098956,
+            -0.003735719227468833,
+            0.0061793909220459,
+            -0.028285889794908054,
+            0.008181788076484304,
+            0.004432488437404309,
+            -0.01743905248466261,
+            0.041221269621866216,
+            -0.014892354419172107,
+            0.005087102208335201,
+            -0.022198383716186407,
+            -0.011936388003605221,
+            -0.00027487042024290816,
+            0.044599215801158845,
+            0.008068975171827777,
+            0.003496950921841073,
+            0.02238779276447619,
+            0.0054839808262270315,
+            0.04816723616939378,
+            -0.02685221260466647,
+            -0.0190356946337995,
+            0.016464136459560113,
+            -0.0176092780782677,
+            -0.012513274095432353,
+            -0.010695421445334002,
+            -0.004784348908457072,
+            -0.022203041334932165,
+            0.010788155380415583,
+            -0.018353897272137543,
+            -0.02277431189907027,
+            -0.003945465850796918,
+            0.016024639792926797,
+            -0.006248368166794934,
+            -0.0038769803892794875,
+            0.0016292556540434277,
+            0.02412864830338344,
+            0.02892955453819876,
+            -0.02408022470555968,
+            0.019190282372105023,
+            0.018941584034503536,
+            -0.006709516380602965,
+            -0.020956377933747532,
+            -0.01964885206488053,
+            0.01434623424964373,
+            -0.01352968586212152,
+            -0.013872556892707697,
+            0.004052798542613835,
+            0.009201147374299997,
+            -0.004086067985859617,
+            -0.04185829685685201,
+            -0.014668698123513235,
+            0.003179403563664022,
+            -0.0006415014901450319,
+            0.019360184344240947,
+            -0.0150817538567364,
+            -0.029771999278536083,
+            0.014221776097046751,
+            0.022944884879589678,
+            -0.04493425220030206,
+            0.02117577719533236,
+            -0.0030648751497261983,
+            -0.00016251533643103604,
+            -3.817296018601414e-05,
+            0.008727767382077138,
+            0.026914779443266063,
+            0.03456565094655899,
+            0.01974806772696219,
+            0.00450800007152441,
+            -0.008952599700759673,
+            0.025184190461155245,
+            0.009846044083064704,
+            -0.025168735183134738,
+            0.010983353565382029,
+            0.016740721743326522,
+            0.0070838930012654296,
+            0.009173879407277282,
+            -0.006078288608766152,
+            -0.018269347893097643,
+            0.013181494188066667,
+            -0.01435693747022231,
+            0.01678907949532146,
+            0.05154463552784974,
+            -0.00039632596424461964,
+            -0.02461439550772063,
+            0.008418147037164186,
+            -0.006544319842335593,
+            -0.010065129286193278,
+            0.02214628550460128,
+            0.0029515574751370417,
+            -0.030862152033847717,
+            0.007509556025223583,
+            0.00675485304374519,
+            0.01894936231559203,
+            -0.02355525295639599,
+            -0.01387885858544803,
+            -0.00431398020087544,
+            0.0021208056468763523,
+            0.01580766504149989,
+            -0.004739477121073512,
+            -0.013204823447960706,
+            0.02994098229517376,
+            0.025496728234662373,
+            -0.00026124946738213036,
+            0.015060064980768175,
+            0.022262345898533903,
+            0.009968408664790368,
+            0.007588431141088797,
+            -0.0083063413609779,
+            -0.031336272171937095,
+            0.0018957858383561667,
+            0.05800811417728381,
+            -0.014937085296865802,
+            0.04313875432235027,
+            0.02018476425923236,
+            0.015657644644987107,
+            0.017880263053253607,
+            -0.01662268082870562,
+            -0.008740053682875323,
+            -0.0017630394204179638,
+            -0.008294417990024647,
+            -0.027613464270562948,
+            -0.005953209556351086,
+            -0.02379961210881074,
+            -0.013878676554701543,
+            0.015418315245979704,
+            -0.014764728065382074,
+            0.0027420640870858457,
+            0.010395538988699663,
+            -0.006590633833835581,
+            -0.00841537209927409,
+            -0.003943772700797381,
+            0.03376338017637296,
+            0.011980354721458463,
+            0.027761929822019422,
+            -0.00011128539412582973,
+            0.009342430952107655,
+            -0.018419526564396063,
+            -0.006839752805318664,
+            0.03271190887864607,
+            0.006011520427944665,
+            0.004652835471939905,
+            0.01485920920301398,
+            0.014305626381398871,
+            0.038015677219487023,
+            -0.005102005401565164,
+            -0.018204460895989052,
+            0.02065495133597456,
+            -0.0049699013356374775,
+            0.019149036693562712,
+            0.008825981577388056,
+            -0.002660055848523568,
+            -0.004292379561953203,
+            0.03090270710860331,
+            -0.020243440437217503,
+            0.018735727616321387,
+            -0.02238236089688497,
+            -0.0276584413929402,
+            0.023028408906176946,
+            0.008320436733620496,
+            0.04047641118026783,
+            0.011463851066078041,
+            -0.02247143742166039,
+            -0.034588725413285185,
+            0.009827713930789184,
+            0.00337071450636392,
+            -0.024446280199879062,
+            0.017542777160283324,
+            0.00780836483490212,
+            0.008453886667283516,
+            0.016624116261219637,
+            -0.004175682824655864,
+            -0.0042784362933758835,
+            -0.001546656717896957,
+            -0.008819739495525477,
+            0.01406790007187259,
+            -0.015487658289541583,
+            -0.021457182907140035,
+            0.024687169583245905,
+            -0.03915911962859764,
+            0.01771290139419629,
+            -0.006510046907173026,
+            -0.00997450880725535,
+            -0.02059560682587115,
+            1.5939836949198004e-05,
+            -0.007159493752862143,
+            -0.0004782363814453761,
+            -0.017503181957042913,
+            0.016643971600145645,
+            0.024056442566869802,
+            -0.013209211443909658,
+            -0.00445521990927108,
+            0.0020313626821877526,
+            -0.029235692455820056,
+            0.006662209097255912,
+            0.016993027053110635,
+            0.00529940612845993,
+            0.0026104743851393716,
+            0.0001707934714983891,
+            -0.031289446611163674,
+            -0.005782208620284939,
+            0.03053516889069683,
+            0.05590704747438521,
+            0.021628254147289437,
+            0.02403314981042378,
+            0.02908606663381421,
+            0.02717206823892633,
+            -0.003198844932040613,
+            0.0016217508103442348,
+            -0.01620660606431983,
+            0.027098323143010743,
+            -0.011664168290676943,
+            0.04322175021589083,
+            0.008652323184522541,
+            -0.049493020736445056,
+            0.004537958157696794,
+            0.00943239454147692,
+            0.034918603888916544,
+            0.016678716235690318,
+            0.0005887414054803805,
+            -0.005894338664998101,
+            0.03279165663336323,
+            -0.02735726709032382,
+            -0.011962070182991011,
+            -0.00036014407560325284,
+            -0.019079003730607717,
+            -0.015938295169860884,
+            0.010357611943935459,
+            0.04278924678845588,
+            0.0009957483598272451,
+            -0.0071422790468850215,
+            0.01571201177743999,
+            -0.009445307674383,
+            0.02286338492075306,
+            0.007336440384050498,
+            0.0042809497342459904,
+            0.025581467690095977,
+            -0.0025007871391868105,
+            0.02300539823997411,
+            -0.000628898771317254,
+            0.004582863909134464,
+            -0.015960578501883316,
+            -0.00930649396114688,
+            0.027838455754803444,
+            -0.011112268986721,
+            -0.029253746504396393,
+            -0.008734785561831333,
+            0.00472579547270089,
+            -0.0003805825712986138,
+            0.0021854358149324227,
+            -0.01183885533661493,
+            -0.010840951201591629,
+            0.019965691788904057,
+            -0.004922943301424112,
+            0.003800048754664761,
+            0.012664049429916252,
+            -0.0030124219674414897,
+            -0.01910844996750822,
+            0.01889983984736845,
+            -0.003858283778519141,
+            -0.024168821338992813,
+            0.020083955711766945,
+            0.01850497889545421,
+            0.045156567155119454,
+            -0.01586824055785609,
+            0.004612757419198795,
+            0.017497774574602505,
+            -0.002172088240279357,
+            -0.019617327081566827,
+            -0.014944795772093997,
+            -0.018212462639041492,
+            -0.015975283672064702,
+            0.012837956594691596,
+            -0.01243789877725244,
+            0.019500571353639314,
+            -0.0380735226388175,
+            0.012590900534826689,
+            0.004832168601136412,
+            0.007505385730121269,
+            0.016011983211212992,
+            0.025994560026860358,
+            0.002157436199771672,
+            -0.033465476168065784,
+            -0.01014599891204166,
+            0.007845373376916211,
+            0.010093432683262838,
+            -0.002849185178237036,
+            -0.018147004183434554,
+            0.003633790611809425,
+            -0.0266147420637404,
+            -0.018644422659309035,
+            0.022926737299761104,
+            -0.04798772015599113,
+            0.017908415301694906,
+            0.024859125158586466,
+            -0.04343699525707025,
+            -0.04886699607556352,
+            -0.011459188432595229,
+            -0.036774381316312464,
+            0.0014128005215813578,
+            -0.0005256067584181517,
+            0.011194992852264102,
+            -0.009342140079829378,
+            -0.018598671981710183,
+            0.01584789414301952,
+            -0.0036412641242200418,
+            -0.004225840239935549,
+            -0.00616696736018045,
+            -0.0025598631463178384,
+            -0.010188803157718543,
+            -0.03244694662411977,
+            -0.03290891136780909,
+            -0.00033742479296272127,
+            -0.009743927051414455,
+            -0.02334288359945272,
+            -0.03447062782872289,
+            -0.008728968107967028,
+            -0.037753659749578956,
+            -0.00515561961442376,
+            0.04070890150714597,
+            0.027517147608335957,
+            0.009449414978133334,
+            0.008086900089929912,
+            -0.013989034604490105,
+            0.05691047488419298,
+            -0.020425883381389086,
+            0.039466620642512155,
+            0.005301493569940605,
+            -0.0037222353899918703,
+            0.013075425214878277,
+            0.027710826422351564,
+            -0.006253665106995653,
+            -0.0017412928073238833,
+            -0.01928096630475438,
+            -0.012920476859996416,
+            0.009444408011467955,
+            0.015799884767184885,
+            -0.0025083061619197524,
+            0.001274457251739847,
+            0.00930533161700852,
+            -0.0006652478540264343,
+            0.017258038491335247,
+            0.03996268817486439,
+            0.003833844612473671,
+            -0.00826404141706295,
+            -0.018822002570963614,
+            -0.003188844900317347,
+            -0.011807006674248829,
+            0.010482058734260632,
+            -0.015410635475145028,
+            -0.011540803927522013,
+            -0.01249428777269578,
+            -0.017081856786754605,
+            -0.002183034976477535,
+            0.003114834030705039,
+            0.0027475411722881747,
+            0.02261627635192778,
+            -0.015872027027080824,
+            -0.022487581027490516,
+            -0.0035823117885461506,
+            -0.02398778449664045,
+            8.13716819067491e-05,
+            -0.01050060782407259,
+            -0.015616181283381218,
+            0.013676480044963827,
+            -0.021513173575512936,
+            -0.01091399089076612,
+            -0.008699176177861256,
+            -0.0006986953850759474,
+            0.005853723710671249,
+            -0.00993355748621621,
+            -0.00010241843582159751,
+            0.01150872694850241,
+            0.005559552266827916,
+            -0.012336449703762907,
+            0.0039431718621504,
+            -0.012740550272925562,
+            -0.004432059174951158,
+            -0.03592309122711106,
+            -0.019665686718735042,
+            -0.01846795637714379,
+            0.005642094448288269,
+            -0.005108964028543505,
+            -0.020586678054151854,
+            0.00668364061048941,
+            0.0034775127050453774,
+            0.004713169526661792,
+            -0.010723667875209928,
+            0.013944735241748791,
+            0.010882420168586074,
+            -0.003518245916227596,
+            0.013827334494230662,
+            -0.001330399936559009,
+            -0.020286697504699764,
+            -0.011139922402071095,
+            0.03538258948179278,
+            -0.024961826464903997,
+            -0.0018019328351763667,
+            -0.0026423695690168066,
+            0.00597003516870726,
+            -0.03071835946501977,
+            -0.013634212232744325,
+            0.02164400349828422,
+            -0.005846552894831441,
+            -0.026367014426434226,
+            0.03399116480234033,
+            0.03473258231629753,
+            0.010041605592929843,
+            -0.013423384340589794,
+            0.04730054254696893,
+            -0.012968978954938308,
+            -0.018274800841400665,
+            0.026826644936918647,
+            0.0278389611094493,
+            0.02525164095926168,
+            -0.029782550447890466,
+            -0.003010713086487275,
+            -0.02393629084357885,
+            0.017993366287762866,
+            -0.0263476663180971,
+            0.01864011499179424,
+            -0.02332263152537849,
+            0.016014908881756034,
+            0.01868464825773604,
+            -0.003104041044182249,
+            0.026313491086684425,
+            -0.0052945248919367815,
+            0.015253731232325074,
+            0.0012371308759955573,
+            -0.02119765517013431,
+            0.007795466385020642,
+            -0.006469500272298062,
+            0.025762169721880456,
+            0.00833992768202466,
+            0.01801901639978976,
+            0.022019940825351116,
+            -0.0409522298213741,
+            0.02003698451081499,
+            -0.001367916864679543,
+            0.01782984895971748,
+            0.0038516134262842373,
+            -0.007885432294782304,
+            0.014221728513400065,
+            0.010991352188016265,
+            -0.012851099261924757,
+            0.003840539873231741,
+            0.0008923482291826014,
+            -0.028928104542716937,
+            0.007066004768771198,
+            -0.026914574065060926,
+            0.025866907400807087,
+            -0.005526774784152086,
+            -0.00998122210956923,
+            -0.020744532673097317,
+            -0.0071425207262770865,
+            0.003754468896273156,
+            -0.012747874997429746,
+            -0.011385805601999295,
+            -0.024700457669046855,
+            -0.02401291382882952,
+            0.029591582802488962,
+            -0.014054332096344003,
+            -0.04015978298687865,
+            0.001234440863385445,
+            -0.023301255402129767,
+            0.01193125007773239,
+            -0.005793124129267758,
+            -0.024440100649464773,
+            -0.042563399082257154,
+            -0.017548888269389457,
+            -0.0009394887961485922,
+            0.022739763644266633,
+            -0.00114156949394707,
+            0.041791395064755515,
+            0.0008816696205437911,
+            0.022207132857533028,
+            -0.015547580543606755,
+            0.005319731264840819,
+            -0.02071229859314312,
+            -0.041495579742912285,
+            0.03660432378985971,
+            -0.014637031447181937,
+            0.025854844497435178,
+            0.03324325296704186,
+            -0.013009347371118843,
+            0.001354504885302625,
+            0.02652570709474851,
+            -0.014618100923440427,
+            -0.010782232747863238,
+            -0.0034936699515378212,
+            0.003461670978569965,
+            -0.006468116285021298,
+            -0.027793221545159466,
+            -0.007752458229193391,
+            0.04193706715422074,
+            0.017894069367616637,
+            0.021928994684423043,
+            0.022606401934291136,
+            -0.017972404868085358,
+            -0.026402048654300563,
+            0.003656646576244765,
+            -0.013129722990153284,
+            -0.018090360146368677,
+            -0.01624598856366761,
+            0.012397150789705746,
+            0.011741002980073786,
+            -0.004052422390675261,
+            -0.01836750423615859,
+            -0.020929335029248472,
+            -0.016327412417122416,
+            -0.0042615345226698015,
+            -0.008458117095535,
+            -0.005002582649405199,
+            -0.012158173174049797,
+            -0.020409616620022143,
+            0.03706368540614453,
+            0.0285565642978937,
+            -0.0041732933884846566,
+            -0.00607154864704767,
+            -0.011142150176334957,
+            0.008834431806990914,
+            0.006021232046087652,
+            -0.005028870471297621,
+            -2.3396043292139426e-05,
+            0.005112033372713154,
+            0.022822312435625314,
+            0.007271235299348001,
+            0.0035791789459533856,
+            -0.06340305811365254,
+            -0.005392000517722054,
+            0.022560201096078862,
+            0.012581401789917604,
+            -0.0032248219865223496,
+            0.025180718735801837,
+            0.0006451724964970927,
+            -0.009714514747463593,
+            -0.018389422219956937,
+            0.024728862150150004,
+            0.008760329201960964,
+            0.020988700992999768,
+            -0.014974763313686516,
+            0.003845297486923296,
+            0.0006445983847138896,
+            0.00750394530065325,
+            -0.0019792992847026793,
+            -0.008578254004439607,
+            0.025776677078006516,
+            0.016593438298233645,
+            -0.005695578040595484,
+            0.01604018433477423,
+            -0.0017497260488899657,
+            -0.021460543443428924,
+            -0.010265213095609554,
+            0.00042017209251926397,
+            -0.010570864288876387,
+            -0.018015790888295944,
+            0.017569414447505878,
+            -0.002601036282535863,
+            0.0072399377790164435,
+            0.030281707151249355,
+            0.0423183067569767,
+            0.009566493398534713,
+            -0.005928603293907002,
+            -0.0037936911251953654,
+            0.015477873304924939,
+            0.01513127995244932,
+            0.010741256034652754,
+            0.03015858543221727,
+            -0.029247492875021843,
+            -0.006690306960497428,
+            0.0046653727410352576,
+            0.004610405633429778,
+            -0.011692179099382171,
+            0.030055897497741527,
+            0.04569730592994177,
+            0.006736813244286893,
+            -0.004799978513117371,
+            0.002121067028052434,
+            0.011299889107780231,
+            0.007768137218140627,
+            -0.018927080714961783,
+            -0.0012222962555201746,
+            -0.006009911026251445,
+            -0.014001766533604134,
+            0.026941585318261085,
+            -0.003960837039351648,
+            -0.011848232223811548,
+            0.02903357274500332,
+            0.1105267435019781,
+            -0.006011029338781496,
+            -0.0053646998281195206,
+            -0.03242321704621517,
+            -0.015194102811332289,
+            -0.008860590919147765,
+            0.0056988205877963076,
+            0.01635957585050017,
+            0.020327661913559888,
+            -0.0008943408274661007,
+            0.016820860249955754,
+            0.02822175614376368,
+            -0.013854827839129885,
+            0.007906244895327178,
+            -0.030891381690403466,
+            0.015702117924362928,
+            -0.005383974926093977,
+            -0.012032955400313954,
+            -0.0026169511828034054,
+            -0.008618307035330928,
+            -0.0037356680273096105,
+            0.0009666430093455632,
+            0.011296307527684261,
+            0.004427516814276546,
+            -0.014484198313369681,
+            0.002829529156309784,
+            -0.008595300985863424,
+            -0.005122996529848575,
+            -0.02393571073523279,
+            -0.024119245543788293,
+            0.005223147047205755,
+            0.0019389305548629142,
+            6.344308292376579e-05,
+            0.006848965054539454,
+            -0.003810712095343454,
+            0.030149081317333418,
+            -0.01700322008272963,
+            0.0012288912810468446,
+            0.01193226302726573,
+            0.012332749933413344,
+            0.01688602002694633,
+            0.0028600984823761366,
+            -1.9882692552530084e-06,
+            0.021172977047420227,
+            -0.010514036509402561,
+            0.01460334896316005,
+            0.013091417303145243,
+            -0.04103572381458182,
+            0.0089746401463863,
+            -0.0027841289484449873,
+            -0.024252447218794344,
+            0.004446472209246425,
+            -0.018987867343500102,
+            -0.00881974688349601,
+            0.0005365625696166774,
+            0.012054719659248512,
+            0.01960300421041832,
+            -0.008553943699846696,
+            0.02111201386480852,
+            -0.0003501657740808554,
+            0.004177088753111459,
+            -0.012879944670340026,
+            0.0065222126975051685,
+            -0.007563354540402343,
+            0.0008168128607271199,
+            -0.018213241937597355,
+            0.012526561612175514,
+            -0.0008285725179894544,
+            -0.015737003159436623,
+            -0.011337688275309921,
+            0.004737346137976519,
+            -0.0027993545140114705,
+            0.0030843163114700313,
+            0.010153740340626154,
+            -0.008815302484351509,
+            0.004919275910399053,
+            -0.0017541042538165297,
+            0.002562816687714991,
+            0.014285249186421434,
+            -0.029816964593980453,
+            0.005357227738099325,
+            0.009154842495811638,
+            -0.00048782303639333943,
+            -0.013074976152495576,
+            -0.004054505559072866,
+            0.0028322102449485015,
+            -0.016877328079971877,
+            -0.012128199639779175,
+            0.023421330164419888,
+            -0.012565145848820333,
+            0.008609070188894315,
+            -0.010936598722317274,
+            -0.018401801734378692,
+            -0.01066119413133089,
+            -0.005254229780560808,
+            0.01234969508270852,
+            -0.0038427767204741877,
+            -0.006947594635952202,
+            -0.01756894543880097,
+            -0.007890897373987378,
+            -0.047246698568222466,
+            0.009617876521097624,
+            0.027157555263862723,
+            -0.011355758309667893,
+            -0.04050311381480088,
+            -0.01017774338301804,
+            0.02437257208967103,
+            -0.004056329295095018,
+            -0.008345450168215274,
+            -0.001599171409135998,
+            0.018358386463243014,
+            0.0027207445181251406,
+            -0.007292967749349572,
+            -0.011267861428599221,
+            0.01657507035043879,
+            0.015105061280816214,
+            0.021419854579631276,
+            -0.02088952706825817,
+            0.006295984081511286,
+            0.020973020776936996,
+            -0.03073246543555915,
+            0.0023720435876790876,
+            0.03363966216709148,
+            0.010412588613998134,
+            0.013023311559810034,
+            -0.0010217662933572685,
+            -0.0075567381078977215,
+            -0.008248818794295818,
+            0.011848200323549865,
+            0.005066463524567732,
+            -0.023174489297821385,
+            0.021823825355858556,
+            0.020368868756448753,
+            0.0026042856858509524,
+            0.003457067317891926,
+            0.0005539745115410357,
+            0.01062670844108589,
+            0.009948889803890108,
+            0.05100741556350918,
+            -0.006493714124814164,
+            -0.014021751055930108,
+            0.015063988102064609,
+            0.02488059237697814,
+            -0.0023499941119596228,
+            0.01087282522970853,
+            -0.02612218935931224,
+            -0.021423880914461062,
+            0.023389294741183817,
+            0.008967763132667892,
+            0.009637806238448432,
+            -0.010815706635624337,
+            -0.011390999462441815,
+            0.031750675158698435,
+            -0.01307829462211914,
+            0.00759749911213318,
+            -0.02169895957209634,
+            0.01387879409130081,
+            -0.008606362268863441,
+            0.032229658009209815,
+            -0.0009066257436946125,
+            0.011062436424260079,
+            0.028010486866468023,
+            -0.007835805879241447,
+            -0.019657486763625547,
+            0.003967480939987674,
+            -0.015577952617624695,
+            0.02188429826317828,
+            -0.020502758155233298,
+            -0.0002362500702658591,
+            0.0040935406015014265,
+            -0.010056462958039452,
+            -0.0057038083589231615,
+            0.011475170331848666,
+            0.0015345076181482267,
+            0.006359262977587827,
+            -0.020293014401106,
+            0.01369706440475074,
+            -0.003989482215141903,
+            0.007887824197823247,
+            -0.015925992045349828,
+            0.014370223626942766,
+            -0.02309139937510304,
+            -0.021808736033141787,
+            0.008142621179624918,
+            0.0108996933426044,
+            -0.007589050511458233,
+            -0.025971539510030145,
+            0.02250992080022228,
+            -0.024980833805349545,
+            0.024979008145354243,
+            -0.006622339940663538,
+            0.002566378488088469,
+            0.004913874740363343,
+            -0.008102103508732738,
+            -0.0028098905427027783,
+            0.0057865587191235565,
+            -0.0073509382400023995,
+            -0.014123756810537343,
+            -0.019453380788786282,
+            -0.04083629135586275,
+            -0.03177927047231389,
+            0.022927581786088044,
+            0.011557232599801421,
+            0.0017073832658476193,
+            0.02339552660545296,
+            -0.02247471449131386,
+            -0.017890866287500053,
+            -0.009775068417716798,
+            0.025184899522229628,
+            -0.04408541669037361,
+            0.012104255392615549,
+            -0.014669458760022245,
+            -0.0245258242481677,
+            0.006274252098230249,
+            -0.004161960147481513,
+            0.009299171724026455,
+            -0.010779421730862708,
+            -0.04713627671791054,
+            -0.005192684784377356,
+            0.02330975631183273,
+            -0.0036802876364168935,
+            -0.006197548344287773,
+            0.03048284650641829,
+            -0.0019503797069087845,
+            -0.005038254331399784,
+            -0.005524875459245342,
+            0.027783596017224226,
+            -0.024950083604372984,
+            0.010988398531346805,
+            0.008304937912856357,
+            -0.008485065895193593,
+            0.006572665697403556,
+            0.0011518846966102158,
+            0.011927348472282199,
+            0.015632588644277183,
+            -0.015559413716826264,
+            0.02262990861928957,
+            0.011446850286896198,
+            0.006506075777765303,
+            0.008015225053785824,
+            0.00010054430757843335,
+            0.0022975742216194894,
+            -0.024948925449668525,
+            0.026047924967851216,
+            0.01888211557921096,
+            0.0001632181017876392,
+            0.012553094476616919,
+            -0.015042621225242777,
+            -0.02782009986290425,
+            0.02606646297992026,
+            -0.016847653820092335,
+            -0.00045272379465939274,
+            0.020053091486344633,
+            -0.016210036358455773,
+            -0.007930550661303182,
+            -0.01123514849773852,
+            -0.0162675150307978,
+            0.01803076559921804,
+            -0.012160217543572069,
+            -0.02216594536057653,
+            0.002594675636753233,
+            0.016443123828746582,
+            0.02134480656850119,
+            0.026694575953313482,
+            -0.011470930697351533,
+            0.017768927048191052,
+            0.0008653408391607816,
+            0.001865765599795092,
+            0.025745954171649543,
+            0.01702677011968492,
+            -0.008474889109168202,
+            0.025347176686722313,
+            -0.0016686223698600755,
+            -0.002421921060670558,
+            -0.012265504084322075,
+            -0.0165214436458303,
+            -0.007545471138114208,
+            -0.015333235556146196,
+            -0.0051830106590765685,
+            0.000762746782664525,
+            -0.0018118022696726346,
+            0.10876576838038368,
+            0.0034948044540582626,
+            -0.011608460689239234,
+            -0.005870263500417206,
+            0.007510774308250375,
+            0.0023525201199670765,
+            -0.006846154764389575,
+            0.016082721103476375,
+            -0.018616626204086575,
+            -0.023593744079394,
+            0.004743481974389998,
+            -0.0009262286319469741,
+            0.00870122878944883,
+            0.02029523251993269,
+            -0.008608837818169835,
+            -0.0200567711498164,
+            0.007089477722191928,
+            0.005535149644041221,
+            0.0023640746516071774,
+            0.005256431824732927,
+            -0.0006118671321824933,
+            0.008209920240361308,
+            0.02448422122961942,
+            -0.015622892878860465,
+            -0.029770836607612903,
+            0.010535371601298532,
+            -0.00015994156470228762,
+            -0.007193425789067776,
+            -0.011046316463797128,
+            0.014728841492009627,
+            -0.0011488451990980268,
+            -0.02751404133172324,
+            0.017663132712536528,
+            -0.019055335178900415,
+            0.014643246200933506,
+            0.0025348345163347464,
+            -0.00016387173410118788,
+            0.022565953069005714,
+            0.013101158251834834,
+            -0.04272085076882881,
+            -0.0034442020309914823,
+            0.01041205537746844,
+            0.011059422927946552,
+            0.0116446016862942,
+            0.0012992535381387887,
+            -0.018670801241345514,
+            -0.011788984490283379,
+            -0.009109849103208542,
+            -0.021268488414296117,
+            -0.012523033785938503,
+            -0.011393853408628214,
+            0.010743646386439633,
+            0.0012964030814325153,
+            0.023809566099779113,
+            -0.003030535542277875,
+            -0.006526351867141901,
+            -0.000430699447948508,
+            0.00880561288237257,
+            0.009694646009259077,
+            -0.02079011138439714,
+            0.008294338719512953,
+            -0.01823922789220214,
+            0.011819769685519291,
+            0.006416044253693268,
+            -0.022799568832306067,
+            -0.009911619276012527,
+            0.007152786695893951,
+            0.008393677195358034,
+            -0.013569953300860946,
+            -0.006503988731872938,
+            0.02299498754558268,
+            0.018870619152710703,
+            0.012233964644738905,
+            0.024994616203196145,
+            0.03838969722235787,
+            0.005744255912479976,
+            0.027682123067526515,
+            -0.016791471092091604,
+            0.008459745383680297,
+            -0.022493868174622695,
+            0.012902262638928338,
+            -0.00460399519266689,
+            -0.0050431343566427886,
+            -0.018188192498411616,
+            0.033765105245412236,
+            -0.009159751849801773,
+            -0.0027150644587327763,
+            0.034778224828120714,
+            -0.0066502021793471075,
+            0.004525230911930458,
+            0.005096748531775903,
+            -0.016067049494073893,
+            0.033274672789154185,
+            0.01686831040872425,
+            0.016353966592947262,
+            -0.021631941883929817,
+            0.011748317746828848,
+            -0.005675701689448899,
+            0.0001273442591116645,
+            -0.020738819037390263,
+            -0.0037528323797385733,
+            0.0027725541343763104,
+            0.0059860509282538546,
+            -0.016556134518370437,
+            0.02033623476757896,
+            0.00019647023567641008,
+            0.0052895744489259875,
+            0.03253462247073863,
+            0.01043311189804924,
+            0.019182938878855655,
+            0.010051051068017807,
+            0.019523739618278902,
+            0.0281939391326836,
+            0.025685787482623124,
+            0.0029932688462254147,
+            -0.018668945726139946,
+            -0.0009326901784812825,
+            -0.007880352863625262,
+            0.010492063055849727,
+            0.022849902214818293,
+            -0.022619228461640748,
+            -0.02772644874135741,
+            0.018566981184526675,
+            -0.0022608185345048414,
+            0.020031103834122595,
+            0.007544464625538984,
+            -0.020935181884939796,
+            0.003407355515103286,
+            -0.023178230166337178,
+            -0.0005418155871620593,
+            -0.031039187720144967,
+            -0.028591174077243604,
+            -0.0007659241904525392,
+            0.011945344534674635,
+            -0.01688153017778311,
+            0.03357788163306365,
+            0.017667778671101954,
+            -0.014289109343082812,
+            0.01604684528817524,
+            -0.011054587325539086,
+            -0.0036959091138078683,
+            -0.005819853870004682,
+            -0.00203755010992417,
+            -0.02157944474979592,
+            0.005138614763345542,
+            -0.0033509138150859593,
+            -0.0024871060745482883,
+            -0.016936575573598154,
+            -0.021901830366872595,
+            0.016375173843894617,
+            0.016742770709641697,
+            -0.006606252542838758,
+            0.039771905404651496,
+            -0.004238071340064403,
+            0.00018364315349545657,
+            0.011690521707403059,
+            0.0010018392078563795,
+            -0.013651921068215563,
+            -0.010829994512513501,
+            0.012362647379788833,
+            0.011321403447882543,
+            -0.018942043512252663,
+            0.005763773608775982,
+            -0.009131565038297889,
+            0.002089216671850269,
+            0.005438887787329238,
+            -0.008183835275629478,
+            0.006690280329947951,
+            -0.01673229763262922,
+            0.015833397876364723,
+            0.00295474085300386,
+            0.003186554859191664,
+            -0.011797859525926772,
+            -0.020991927615680524,
+            0.01576413532776261,
+            0.0032403514518527287,
+            0.018819824905201635,
+            -0.026755218746060662,
+            -0.00012946467986739473,
+            -0.009552681500884924,
+            -0.0028941357661144826,
+            -0.012533800273548772,
+            0.013975640628890588,
+            0.016864739319953755,
+            0.0063941926578529,
+            0.01781016436903586,
+            -0.0026268224264609724,
+            0.015368284773773678,
+            -0.018122416427266218,
+            0.008686009180697955,
+            -0.006801074032721299,
+            -0.018537521789181048,
+            0.0013237338264105415,
+            0.027863741654435434,
+            -0.004670012662973029,
+            0.011066853115455966,
+            0.011704884649534624,
+            0.018954056680143452,
+            -0.0012626215734593707,
+            -0.0030850559460101866,
+            0.015610404665171367,
+            -0.005838226405658572,
+            -0.006861883833171456,
+            0.020872966233916948,
+            0.032951653763679145,
+            -0.006716215648910268,
+            -0.0021292599951407863,
+            0.010886596327481677,
+            -0.004901684487088035,
+            -0.008644584974417106,
+            -0.009587037829603014,
+            -0.03158137900717491,
+            0.024369089446706894,
+            0.012092412439477474,
+            -0.009601042951586109,
+            0.022690862146548786,
+            0.020752792552278052,
+            0.011412743968296708,
+            -0.010471860708195726,
+            0.02291536038589634,
+            -0.01268035142112569,
+            0.024168869865226136,
+            0.032482831775271157,
+            0.004400973556358811,
+            0.0022312906286559167,
+            0.0021175489661843985,
+            0.01691348460546639,
+            0.006050357346231155,
+            -0.007068133214675996,
+            0.011656003476409986,
+            -0.018487923180009552,
+            0.01598530891381001,
+            -0.004432366596808033,
+            0.02808371007118948,
+            -0.01337145216499331,
+            -0.010473700033368562,
+            -0.00537325376270318,
+            0.00010726431372328729,
+            0.006445701087575359,
+            -0.003324002570256215,
+            0.005255443653813165,
+            0.0007072577303252132,
+            -0.017825649146944237,
+            0.003050128130393781,
+            0.004825967170196209,
+            0.009018799597337101,
+            0.004103653497834134,
+            0.007249957401824204,
+            -0.012360355929483,
+            0.01441885664789707,
+            0.004401820363035808,
+            -0.0026190938717927086,
+            -0.0007163225760398784,
+            0.011124538295057815,
+            -0.013232822471919533,
+            -0.004895624967029031,
+            0.005281228035321008,
+            0.0059275753353475414,
+            -0.007507924998741734,
+            -0.004484140989405051,
+            0.011908076549392969,
+            -0.0037917291321648483,
+            0.006351543504726163,
+            -0.004409464890606014,
+            -0.0037760571637986827,
+            0.005303661313434895,
+            -0.00974511544836468,
+            -0.004204167898342997,
+            -0.0007093125445559502,
+            -0.006869250680233214,
+            0.00786202375969793,
+            0.005971033227117194,
+            0.010606085272785863,
+            0.006969437776542303,
+            0.004293129058563288,
+            0.004114452739123398,
+            0.004071017374828283,
+            -0.002445681623750424,
+            -0.00592480769363354,
+            -0.004617501328052899,
+            -0.012067458701076778,
+            -0.008335239153829813,
+            0.009025028998949398,
+            0.008653779421856954,
+            -0.019363018792788574,
+            0.02048011578221327,
+            -0.001188320960771659,
+            0.00772165044232104,
+            -0.01602086976290928,
+            -0.000907553574811279,
+            0.005464880629734941,
+            0.008267702703609528,
+            0.016136762227127184,
+            -0.03654209526492064,
+            0.005179791231187316,
+            -0.0037321883493524766,
+            0.016026013709955444,
+            -0.008215703411373182,
+            -0.006979816286354862,
+            0.02564060526656963,
+            -0.01450763388985563,
+            -0.0009847758228920171,
+            -0.016142304341196283,
+            -0.01407778890808498,
+            0.010815725252959506,
+            0.02050940280233426,
+            0.020684808835848383,
+            -0.0004965802897137407,
+            -0.0005781445948759235,
+            0.00036179841583004193,
+            0.022775804864342608,
+            0.015545681427566235,
+            0.018697154644607316,
+            -0.004543500782944068,
+            0.007028436664826706,
+            -0.023057215501463973,
+            0.009912581556496658,
+            0.0011795417129232596,
+            -0.008215027850908477,
+            -0.008554137076267468,
+            -0.008703378564520125,
+            -0.010448538943182258,
+            -0.011773811327190638,
+            0.004106616839960258,
+            0.013895013678116032,
+            0.004190549543250594,
+            0.0031571190600981074,
+            -0.0225385788570314,
+            -0.006243193914542516,
+            -0.011229641377928767,
+            -0.01674214043453308,
+            -0.007367389908398082,
+            -0.02386334969150212,
+            -0.028021339684039734,
+            0.0002794207370355175,
+            -0.02411861258809571,
+            -0.015958336406924612,
+            0.005268815864773021,
+            -0.01546610225139129,
+            0.00898600137695074,
+            -0.008396481690803911,
+            -0.011873486990732417,
+            0.0074743033679134105,
+            0.015336444788692,
+            -0.017676977079590463,
+            -0.010165317702640217,
+            -0.01690524990506033,
+            -0.013460451917691695,
+            0.010918550283871637,
+            0.00595960977280955,
+            0.009250157926438213,
+            -0.012447979337850305,
+            0.0062932435120863365,
+            -0.012673905639312696,
+            0.015073013552721701,
+            -0.03769166067890086,
+            -0.029992249463405965,
+            -0.00014990754540999537,
+            0.007288036630677436,
+            -0.0020735742042505426,
+            -0.018695120921334806,
+            0.0003860116505754868,
+            -0.003608995261360232,
+            -0.014900191335144973,
+            -0.013866216738266176,
+            0.025251475305524053,
+            -0.006788483107616133,
+            0.01223227424986397,
+            0.0034937277040015626,
+            0.03438333560343413,
+            0.008803852645562129,
+            0.014666206816302367,
+            -0.005709421955160623,
+            0.003742433643393657,
+            -0.0033706258426092744,
+            0.006906464031352315,
+            0.0076578888497945185,
+            0.009276592191710524,
+            0.008025321004305927,
+            0.004576470294889512,
+            0.002887346543823031,
+            -0.001925468214331082,
+            0.02125241545487307,
+            0.025671028960818562,
+            -0.00037467906648160044,
+            0.008981063192139038,
+            0.0009387900084090577,
+            0.0002735789978680719,
+            -0.008374290340790827,
+            0.006547013284767913,
+            -0.008274637814401893,
+            0.004041066530254948,
+            0.012897549078838394,
+            -0.009150863940242148,
+            -0.010211568689323908,
+            0.016575088843292566,
+            -0.012380099999574783,
+            -0.03420455578618604,
+            0.026550654680719987,
+            0.005324512466403649,
+            -0.0035670106106754245,
+            -0.010657928468872608,
+            0.0037176027853158167,
+            -0.01103891334887315,
+            0.008096915671982816,
+            0.022587659771132487,
+            -0.0013527917944322358,
+            -0.0018085171207746743,
+            0.0038503794877980586,
+            -0.011870484733648887,
+            -0.007881489342351292,
+            0.006462772034381968,
+            0.0315696514747912,
+            0.01326417653779577,
+            -0.020400369539140532,
+            -0.015223808971473513,
+            0.020406862349006738,
+            -0.023142176712111575,
+            -0.006254959806032159,
+            0.009039506093607052,
+            0.0009087520492922078,
+            -0.022766046838415165,
+            0.0009662037093312435,
+            0.004701966191595815,
+            0.005946458676746661,
+            -0.006358986495019584,
+            0.03237532341275415,
+            0.001075992257265679,
+            0.015892020271402903,
+            -0.017189453434945788,
+            0.0017514632185789299,
+            -0.014189278671121677,
+            -0.014265447388465805,
+            -0.029809460132405256,
+            0.0005557491671071693,
+            -0.0030324198417091704,
+            0.011038984948790744,
+            0.024808067315720335,
+            0.01145103533194245,
+            0.014432246996007246,
+            0.010552852947578788,
+            0.01513689173494211,
+            -0.013653722180565311,
+            -0.029009757850432732,
+            -0.005171522291984226,
+            -0.018001048743500243,
+            -0.02297482628212251,
+            0.003962613844063587,
+            0.013645125285548414,
+            0.006278662690346082,
+            0.023786381386575035,
+            0.0021738516691483163,
+            0.0009580591750940011,
+            -2.690805459518147e-05,
+            0.008894091070322921,
+            -0.004952203441302471,
+            -0.006553986197076102,
+            0.003570189040872711,
+            -0.000652714137223501,
+            0.021228360881461853,
+            -0.005758946428212041,
+            7.897153237591187e-05,
+            0.014242175492137011,
+            -0.005622256348934008,
+            -0.0035009192529791146,
+            -0.017339926143652377,
+            0.006248842389703505,
+            -0.01622011586976866,
+            0.025783086900871728,
+            0.018450783573486916,
+            -0.009109295293546267,
+            -0.013469521573468915,
+            -0.011259555201313116,
+            0.02773097452736661,
+            0.004631282554715908,
+            0.010787055220617197,
+            0.018228322680582464,
+            0.01461254259249465,
+            0.01784830193964943,
+            0.004664725683522323,
+            -0.0034339781146649384,
+            -0.0007692550982545399,
+            0.021176646860433974,
+            -0.014700178590249292,
+            -0.002701208084390198,
+            -0.008842788249773623,
+            0.0021799005046816444,
+            -0.008431299000066363,
+            0.006086447848362669,
+            -0.010793016137040172,
+            -0.015251381840165954,
+            0.0189503457652375,
+            -0.02660767353735015,
+            0.016423753600066336,
+            -0.00593016170680532,
+            -0.00482891500342298,
+            -0.027459214027897378,
+            0.010728039528784636,
+            0.0027866316191608297,
+            0.0022519974195000687,
+            0.010770472550461752,
+            -0.021427634123401408,
+            -0.01867717541189363,
+            0.008532444554010673,
+            -0.011457117350255678,
+            -0.007152301670244725,
+            -0.00348619742388509,
+            0.0154715607296319,
+            -0.00017898955187849443,
+            -0.009925365923862246,
+            0.02085093731060963,
+            -0.007175309371250024,
+            0.01223979370120031,
+            0.005256127464980655,
+            -0.008756077893734508,
+            -0.0028438320920811362,
+            0.021022708898898203,
+            0.027557382862284808,
+            -0.008342206702778956,
+            0.014526066580760486,
+            -0.011883797998224506,
+            -0.01088335352319397,
+            -0.03479444009382274,
+            0.030104838505937447,
+            -0.013305732257226255,
+            0.026773747582411633,
+            -0.0006968265979630097,
+            -0.017037729111584094,
+            -0.016206207453228583,
+            0.01625225876515654,
+            -0.0024290584023212254,
+            -0.009366287391027166,
+            0.01465885301870365,
+            -0.02428628495473643,
+            0.0058336571592899735,
+            -0.0006627935593670669,
+            -0.011531762838191602,
+            0.013275508677482622,
+            0.0019223252930710324,
+            0.010738622785461513,
+            -0.02030405805119001,
+            0.0009661595529028194,
+            0.009263716373456188,
+            -0.016791855691574455,
+            0.006057830565306032,
+            -0.03730952786847912,
+            0.0029341721588188943,
+            -0.014833895684348656,
+            0.005725615593914644,
+            0.001498755253963699,
+            -0.01820708352937343,
+            -0.03652692232407375,
+            0.00043665701245599497,
+            0.01913883585802903,
+            -0.007975988799790905,
+            0.020538595326175857,
+            0.008243569241768276,
+            0.008421404577931517,
+            0.002314738067443065,
+            0.0016702403734608475,
+            0.008279538988821465,
+            0.03679456861213399,
+            -0.01775927634148114,
+            -0.004836606118958108,
+            -0.0012151329859941935,
+            0.010644195708319258,
+            -0.02028470571116305,
+            -0.009287515726906219,
+            -0.0036506912311261536,
+            -0.013134467570201307,
+            0.030615862647585222,
+            -0.020265555441523482,
+            0.025634449009756734,
+            -0.009300770899023007,
+            0.009274072905190077,
+            -0.02097685009125915,
+            -0.009797008690754956,
+            -0.010661540100131183,
+            0.0019532291898233895,
+            0.013465005282849381,
+            -0.009151430285942593,
+            0.0071467934925125145,
+            -0.006655608635025676,
+            0.01020327561163374,
+            -0.001472597695707201,
+            -0.02443508800166349,
+            -0.0024998811551575563,
+            -0.01107321258077183,
+            0.02456164421154801,
+            -0.0019619696941303388,
+            0.0005043179699158694,
+            0.010951580163871948,
+            -0.00853593956432799,
+            0.023927336608555987,
+            -0.008855702280766362,
+            0.0010093679746665581,
+            -0.004863269000020324,
+            -0.004005691415117686,
+            0.001698212044518236,
+            0.01822024022650613,
+            -0.005707404715849356,
+            0.03722587302228139,
+            -0.01708925706528118,
+            -0.0065500772743641925,
+            -0.00015538686795361824,
+            0.010852925330001283,
+            0.010035359503147549,
+            0.008754719437760327,
+            0.004067820167375348,
+            -0.014861417626433965,
+            0.009346669946626484,
+            -0.020136029447262147,
+            0.007501669351583119,
+            0.023178490169347605,
+            -0.006840359541885465,
+            0.0006676450965357345,
+            -0.01283320739092204,
+            -0.015610538827742276,
+            0.006308543164022682,
+            0.002329420005586303,
+            -0.002295739637504597,
+            -0.02181189272979859,
+            0.006298804075737111,
+            -0.014398470126723037,
+            0.004007419942728446,
+            -0.0007497656993578918,
+            0.013863867106289847,
+            0.016451166273826513,
+            -0.006740209707090658,
+            0.039292260347522444,
+            0.01871723939316731,
+            0.009109005310505462,
+            0.005674782595856756,
+            -0.040588444759736254,
+            -0.008993850998154852,
+            -0.0010197185016393919,
+            0.004866150984083949,
+            -0.01141354216871663,
+            -0.018944881712855702,
+            -0.002646110151132781,
+            0.000308585036997592,
+            -0.010681954707313127,
+            -0.009091585604191968,
+            0.01751599002710486,
+            -0.006034424764302979,
+            0.00872057856926618,
+            0.004146961781109887,
+            0.023758065635807138,
+            -0.009727802672285544,
+            0.01176257119851645,
+            -0.0071822083765114165,
+            -0.005456197751568526,
+            0.0006726765703478756,
+            0.011011628907288813,
+            -0.008031792262886802,
+            0.009763340867550022,
+            -0.004279891138185881,
+            0.008355720233269893,
+            -0.008552596384067825,
+            -0.023952129458400563,
+            -0.014972827087541732,
+            0.014347535404251018,
+            0.011651260496709961,
+            0.008137738462704715,
+            -0.013624435569539285,
+            0.0031728721674035352,
+            0.00675830277744976,
+            -0.009184784787581942,
+            0.009161771311910335,
+            0.017733082848622708,
+            -0.001411542568472906,
+            -0.0017830167499636113,
+            -0.0016130003349297027,
+            -0.012173528776899308,
+            0.0011196437753239162,
+            0.02284887358189995,
+            -0.003054479643937998,
+            0.014614283913976716,
+            0.010689486748036806,
+            0.009643292262102645,
+            -0.00810930837603355,
+            0.001915955511174678,
+            0.02341863048264366,
+            0.015022291783105098,
+            -0.004625715884342598,
+            -0.011422973894623295,
+            0.004259757717597968,
+            -0.010332744704689279,
+            0.007373328473086509,
+            -0.02117940113144938,
+            -0.004672698999328004,
+            0.008896311380763048,
+            -0.007028862917603513,
+            -0.01732432285099198,
+            -0.028026173555055017,
+            -0.016765805892242715,
+            0.019819329380181,
+            0.0027213760678977965,
+            -0.013291251876048186,
+            -0.0032919084721662363,
+            0.0031214883337300926,
+            0.002553005168048115,
+            -0.0071104806975682885,
+            0.005310583052470149,
+            0.007359183453835734,
+            -0.008674133980486269,
+            0.017665958608281126,
+            -0.000521269431423141,
+            -0.0008020251454616109,
+            -0.0005442374261144659,
+            0.0006127687598711624,
+            -0.016387752806126442,
+            0.01666161912459431,
+            -0.010949228376419892,
+            0.015198110119816624,
+            -0.008966726991404666,
+            0.008600499511690635,
+            -0.007617043351740469,
+            0.009458641139070436,
+            0.011597957400445055,
+            0.012755037944648426,
+            -0.004403680820063604,
+            0.008519241473418196,
+            0.004807816191718704,
+            -0.008147125240985166,
+            0.007017601015398136,
+            0.01296082929962557,
+            0.00582353932160707,
+            0.0013634237970372698,
+            -0.0008246894322793264,
+            0.0022035670958705576,
+            -0.002480976219285683,
+            0.0056293125704658135,
+            0.012714850523393719,
+            -0.002497639449055006,
+            -0.007135683201188999,
+            0.014988249771676699,
+            0.014955895630183352,
+            -0.022994948923833993,
+            0.007121634434082677,
+            0.005835971825639792,
+            -0.0030507120821557008,
+            0.002145266953387315,
+            -0.027334525318320587,
+            -0.020359474070022213,
+            0.01901995970350808,
+            -0.015260454474605018,
+            0.02850460517419317,
+            -0.011624138792780823,
+            -0.02284242513728605,
+            -0.01277489645734181,
+            -0.004209529633534193,
+            -0.022684500849916897,
+            -0.004894414098943742,
+            0.01919165657366256,
+            0.013401560972810082,
+            -0.004313549020265677,
+            -0.019089369171102138,
+            0.03863248408555474,
+            0.010497951840980463,
+            0.03508157919687289,
+            -0.008597970944278045,
+            0.0023354831464448313,
+            -0.01563401808449416,
+            0.015551176593436512,
+            0.001075882802133782,
+            -0.011575692834796734,
+            0.01505716108583101,
+            0.031836226915668404,
+            0.0034740889237207494,
+            -0.00617206855371193,
+            0.0018420207148235309,
+            0.00418433325800744,
+            -0.0008167766650900381,
+            0.011160989178055499,
+            -0.018497472318034093,
+            -0.011951262348399922,
+            0.0013093376768780122,
+            0.00216347903176653,
+            0.0059233460419514045,
+            -0.001778091863744448,
+            -0.013317639704511549,
+            0.011433308892643204,
+            0.011596810919469926,
+            -0.01518143105525228,
+            -0.007497373860338335,
+            0.0050957665991362405,
+            -0.018974405317819656,
+            0.01872472902494418,
+            0.011859522398789263,
+            0.000792413325332956,
+            -0.0006503947129906383,
+            0.00808635755624928,
+            0.003749813548995223,
+            0.0033657799335872368,
+            -0.007862215303617144,
+            -0.0028014669075116876,
+            0.0009892310856576799,
+            3.834379772674841e-06,
+            0.015402211364447726,
+            -0.015264893397161102,
+            0.02488855366696587,
+            -0.0025643639525691083,
+            0.00953993868921819,
+            -0.017460590918773188,
+            -0.0017911712931080155,
+            -0.00402374787529868,
+            0.01838051670627292,
+            0.006753072393532114,
+            -0.010130676216186492,
+            0.004921510943896829,
+            0.007975201081296845,
+            -0.012589786869771252,
+            0.00018319054705558588,
+            -0.0022606180909706688,
+            -0.014791528072476771,
+            -0.0031171076370692706,
+            0.017873825561816544,
+            0.007593688317551089,
+            0.0037375070591255375,
+            -0.015863143964928172,
+            0.007192324815779268,
+            0.007522990667369163,
+            0.024504875417877527,
+            -0.026309816223960274,
+            0.024779042752840836,
+            -0.022104087334088543,
+            0.02165781034613893,
+            0.0319051741666687,
+            0.007538146369110045,
+            -0.01690181186708103,
+            -0.006595947249169521,
+            0.006373991305923002,
+            -0.023410814842488123,
+            -0.005494017994983188,
+            0.016903736810723716,
+            -0.01792400152372258,
+            0.000330865061689199,
+            -0.014660413003063089,
+            0.0040825880154611895,
+            0.008248037865838212,
+            0.011540333451799686,
+            -0.0033578830043738085,
+            0.018327525495676525,
+            0.020171745352159914,
+            -0.01138759854371235,
+            -0.0004221149053107702,
+            -0.022321409200941777,
+            0.005046865873981027,
+            0.022509643273449936,
+            0.001111272240506322,
+            -0.009540467231190513,
+            0.018376261423745193,
+            0.008480532791922674,
+            -0.020801891594934313,
+            0.008436374227753589,
+            -0.036175982655998785,
+            -0.010454118523313122,
+            -0.009328507118928412,
+            -0.001218633166683384,
+            -0.004142773602181115,
+            -0.0002494371580772723,
+            -0.011365548842513265,
+            -7.699548642315061e-05,
+            -0.006889339420877534,
+            -0.0037173070609413474,
+            0.014953978004522174,
+            0.011678957032329096,
+            -0.003501047405385249,
+            -0.0073984932581048915,
+            0.008493128567102267,
+            0.006163937693564917,
+            -0.017034696006318337,
+            -0.013268418502042576,
+            0.0037122128689059085,
+            -0.020067656401141467,
+            0.005148329575092478,
+            0.00459729178556505,
+            0.014031669598917171,
+            -0.01849235806598133,
+            -0.009298127665365717,
+            0.019716512208448042,
+            0.0094466405158553,
+            -0.0008988916054855498,
+            0.004264260983102956,
+            -0.00032312548009733906,
+            0.002279987803645912,
+            -0.0008902345374346296,
+            -6.145528806973222e-06,
+            0.003602229465148649,
+            0.012503366799381277,
+            0.007759393855482671,
+            0.03694402922447051,
+            -0.016692148708641812,
+            0.005873765863350649,
+            -0.014187609702966002,
+            -0.0037592517335595654,
+            -0.00749939010396784,
+            -0.0004308157309481927,
+            -0.0022282234382577843,
+            0.0021999996319642854,
+            0.0052176773733353925,
+            0.009353171029508977,
+            0.002155405440765206,
+            -0.00823565584692709,
+            0.009691447250255694,
+            -0.015228173095533029,
+            0.003853858061067531,
+            -0.008207910291425473,
+            0.01017972550354623,
+            -0.009695280827515048,
+            0.0028705587799331246,
+            0.03849358164875786,
+            0.01130021300937487,
+            -0.004934175853966098,
+            -0.014300147534361459,
+            -0.0007796051618993811,
+            -0.01737877501558196,
+            -0.008579119859817895,
+            -0.005434792488602325,
+            -0.02617338653902575,
+            -0.00838916510137031,
+            0.015344260464413648,
+            0.011202567142310492,
+            0.014124853863170132,
+            -0.023894182185855824,
+            0.008897943489322673,
+            -0.03001670823127986,
+            -0.011951507895140876,
+            -0.001530967140156163,
+            -0.015467812535428609,
+            0.0010110999991167703,
+            0.023623196504289838,
+            -0.007762735863648054,
+            0.012654652822750325,
+            0.012867958170258448,
+            0.018777522196571934,
+            0.006390986057157016,
+            0.00038045736974183937,
+            -0.009444911783760569,
+            0.009534310165417939,
+            0.008806031651319704,
+            0.0080954543713255,
+            0.013161833638467553,
+            -0.0022840529693446553,
+            -0.018060162439023572,
+            0.005469288590000856,
+            0.014354475778235131,
+            -0.0033504276914600585,
+            -0.001615191819057537,
+            -0.011088493180661976,
+            0.016039040565630432,
+            -0.023559046333872156,
+            -0.009482398794557168,
+            0.009918576960319548,
+            -0.0027308309155029842,
+            0.009212063547988036,
+            0.0160181200081126,
+            -0.01881373001297218,
+            -0.0024168618343039703,
+            0.016244889977796682,
+            0.0024104546308670025,
+            -0.00022606425578303413,
+            0.016792244238218245,
+            -0.0007616857360728623,
+            -0.004279120886828465,
+            0.00997764266310123,
+            -0.0075631917138318655,
+            7.3878137053128405e-06,
+            -0.002416364915008295,
+            -0.0028253498168087964,
+            0.02054128727706496,
+            -0.021281801678871736,
+            -0.005379259148621041,
+            0.02257306103533458,
+            0.015687105622685975,
+            -0.007843745111084413,
+            6.258498042855072e-05,
+            -0.0219506172079994,
+            0.0019925886885244454,
+            -0.007768133999661569,
+            0.016524850592507592,
+            -0.007957949830060354,
+            0.0030085044673901665,
+            -0.01361024620290394,
+            0.011563781350927133,
+            -0.01277407169098418,
+            0.00873716215529134,
+            -0.002986943948333218,
+            -0.017567646684567492,
+            -0.0053109390854215125,
+            0.004561236136619266,
+            0.0014045175130801308,
+            0.044958041347040185,
+            -0.005539136232703934,
+            0.003729517522976541,
+            0.018086335927383804,
+            -0.014199453349727933,
+            0.02079719360289355,
+            0.016663293302592568,
+            0.00991445855219406,
+            -0.00033967370741635794,
+            -0.0018726748849858836,
+            0.023373354806639515,
+            0.013258226441642464,
+            0.00861986998150065,
+            -0.0014340218811281975,
+            -0.008516541227326329,
+            0.018239230461345617,
+            0.005458629943667832,
+            -0.012421896028975013,
+            0.012732721349758832,
+            0.013559877132508506,
+            -0.013720939472355347,
+            -0.011545210834395917,
+            0.01616358223872008,
+            -0.019904594319155373,
+            -0.018260016571763644,
+            -0.0036124537591368864,
+            -0.01614510591132948,
+            -0.03864699384205461,
+            0.01619401261888991,
+            0.018351607973242735,
+            0.006666313000123845,
+            0.013647410298896132,
+            0.002597680061427196,
+            0.004545530267244157,
+            -0.018531082430861854,
+            -0.0077618625909995275,
+            -0.008085144524135461,
+            0.003744735923004248,
+            0.011059457364156674,
+            -0.011335432904114519,
+            -0.009514379703581616,
+            -0.00034500702695418285,
+            -0.007433132060989644,
+            0.01971420270994951,
+            0.0020753610290475642,
+            0.0034506706065161024,
+            -0.00628437298309269,
+            0.017966485854173167,
+            0.026789678964152908,
+            -0.007532446825077476,
+            0.021886182869092073,
+            -0.002864230253056434,
+            -0.0015870360120997492,
+            -0.014393519359298496,
+            -0.0014439412842964876,
+            -0.011527182213472351,
+            0.0005453493437517964,
+            -0.0006791481533636673,
+            0.014093767997869645,
+            -0.03288824642147967,
+            -0.003957301296796813,
+            0.014444592778851293,
+            0.007538052312876599,
+            -0.005057755472988024,
+            0.011357919882949903,
+            -0.002964451028327082,
+            0.016977317604413924,
+            0.0075201055190542146,
+            0.023391646079994784,
+            -0.016588823248397513,
+            -0.004504858318770565,
+            0.006660877064954993,
+            0.0034631255931793638,
+            0.0014484203909961972,
+            0.01912983041187014,
+            0.023439941258581887,
+            -0.014715886385460541,
+            -0.0013596123366001458,
+            -0.0029020372473879614,
+            -0.021414221343490594,
+            0.014976806123253727,
+            0.0074729481304182865,
+            0.010967351226114886,
+            0.009806663105020302,
+            0.015465076881663691,
+            -0.02109587527354609,
+            0.0011671121739822202,
+            0.020567458365051382,
+            -0.008618333549105003,
+            -0.018815706906171112,
+            -0.009804829158883123,
+            -0.00966828714719191,
+            -0.006078893678150302,
+            -0.00803783313451146,
+            0.005523023167028105,
+            0.0022647506733877573,
+            0.01652997604669998,
+            -0.008204200434441975,
+            -0.008404609147866811,
+            0.008489247125986078,
+            0.007546785673285454,
+            -0.0033093451283570316,
+            0.010614050963656496,
+            0.0030633179859393587,
+            -0.006940540318011839,
+            0.00894884343338777,
+            -0.002787816848430143,
+            0.014347598920245487,
+            -0.02441769157218451,
+            0.004717742147185283,
+            -0.012096876825337124,
+            0.006405151160675699,
+            0.0041568790527989905,
+            0.0016702066682613428,
+            -0.0041159211459004,
+            0.0035217362856754353,
+            0.0008893479210743541,
+            -0.07433312151926533,
+            -0.012629681414004524,
+            -0.0035797012285864486,
+            0.033657838713146884,
+            -0.019695806809053914,
+            0.004477927056336216,
+            0.011966065644382396,
+            0.0096717896523898,
+            0.003070713232538593,
+            0.006136726101157844,
+            0.006112385569942848,
+            -0.0245258242481677,
+            -0.00015284418977092615,
+            0.004821040694679944,
+            0.0020347004045943454,
+            -0.007532408630122127,
+            0.00750565079228405,
+            0.015327710245879296,
+            0.020989926895388523,
+            -0.010970305087289488,
+            0.017227663358885748,
+            -0.003352718496981928,
+            -0.006068892126250978,
+            0.0014285750322333686,
+            -0.010153799722483554,
+            -0.017307415843726394,
+            -0.006495008086641125,
+            0.0068983699213297075,
+            -0.001982568363667046,
+            0.0044032687939897855,
+            0.003333596495330213,
+            -0.0032171586993948696,
+            0.0072365127663633925,
+            0.013920326463272646,
+            0.006415916101287133,
+            -0.013675562796300214,
+            0.003003782758807665,
+            -0.02500375035312058,
+            0.011945988879567646,
+            -0.012708975181025296,
+            -0.0009488161508767279,
+            -0.0047093877027523014,
+            -0.0035750724825159503,
+            -0.002213435677949806,
+            0.02043567538125188,
+            -0.014304009033474186,
+            -0.01802899775650995,
+            0.008187227768196673,
+            -0.011217013195839762,
+            0.008300740149622635,
+            -0.03382831122769974,
+            0.012386335491333264,
+            -0.0276029934382802,
+            -0.0002243752790856292,
+            -0.004521387639042107,
+            -0.0010631721066129095,
+            -0.009491184157278669,
+            -0.03274559774619223,
+            0.015845705808609598,
+            -0.007204751554079551,
+            -0.020338337341085228,
+            0.0007143424225187802,
+            -0.021267094083674304,
+            0.014150458196361147,
+            -0.01851165553224022,
+            0.0033450353123933058,
+            -0.018775439467431685,
+            0.002782289821140401,
+            -0.0018711350216089556,
+            0.010502195116579441,
+            -0.00557814442809019,
+            -0.0007130962242755068,
+            -0.013072988887382269,
+            -0.002121841212381393,
+            -0.006653444103331213,
+            0.009394907690516121,
+            0.0028678074167728367,
+            -0.009601577734902932,
+            0.011153571864462314,
+            0.013046116892519414,
+            0.019812860521973938,
+            -0.00539047787915588,
+            -0.00555139958411929,
+            -0.011553689090179406,
+            0.009852063270798534,
+            -0.016326212706634428,
+            -0.00827402913960903,
+            0.008982303148984887,
+            0.015216947130173639,
+            0.01026889105233137,
+            0.003352028462252994,
+            0.003150801750574548,
+            0.002531620731961765,
+            0.026744691368233237,
+            -0.0015120532859309638,
+            0.0008324701960729416,
+            -0.014508988496009931,
+            0.022127525293249926,
+            0.02772430884375744,
+            -0.0012206102581726523,
+            -0.00018859639594030037,
+            0.017437360169132967,
+            -0.004697052522250805,
+            -0.0038111184371524645,
+            0.024015246959315593,
+            -0.038396108183338655,
+            -0.008416825232533771,
+            -0.006788816882617399,
+            -0.006043428198944214,
+            -0.009587366247924467,
+            0.007294190273465812,
+            0.00041966080769004275,
+            0.0024709692753547823,
+            -0.0005250904505954556,
+            0.03452858797518386,
+            0.003403515654095309,
+            -0.0003392060101496309,
+            -0.0032980864446307116,
+            0.0019125121221204029,
+            0.0032987517853214346,
+            -0.009116761755779728,
+            -0.038030085122772624,
+            0.006006334000396066,
+            0.007932785488877407,
+            0.00975009749364395,
+            -0.008120121513849434,
+            -0.008864461019882246,
+            0.004909348738798099,
+            -0.02273129283865203,
+            -0.01372636307167243,
+            0.02625229144362647,
+            -0.008523028986127237,
+            0.029005877324974557,
+            -0.022435059061182174,
+            -0.013135836801818153,
+            -0.004126433322528265,
+            0.010283975013795537,
+            -0.019732856462444506,
+            0.004343860338939202,
+            -0.004022387065239562,
+            0.023733721026005954,
+            0.0077578144626961245,
+            -0.008371631446146718,
+            0.012386735614153616,
+            0.0064789380436815714,
+            -0.0026638642827013837,
+            0.00136333052963848,
+            -0.007209517067748103,
+            0.016916505654836485,
+            -0.011819784026452944,
+            -0.005833909403097991,
+            0.008977527508585844,
+            -0.0024556434204852946,
+            0.02509293948918158,
+            -0.0237762245071987,
+            0.005288171058938098,
+            -0.00019047961064753207,
+            0.01822542471604037,
+            0.008127347078975925,
+            0.008361588667386882,
+            -0.011835407590060994,
+            0.010004482616697591,
+            -0.03378107067882622,
+            -0.015433906677849963,
+            -0.007702910250795191,
+            -0.0013877702264578865,
+            -0.002059152251387601,
+            -0.015745218548968338,
+            -0.026393221735795527,
+            0.006294583740809379,
+            0.016267525575472445,
+            0.023291528020980265,
+            -0.004683197930892445,
+            0.012912830736650997,
+            -0.002831583991769044,
+            -0.008904271394710672,
+            -0.011441882849296418,
+            -0.012202629605208118,
+            -0.012470773094383122,
+            -0.014560694782340029,
+            -0.017045701603711442,
+            0.012139665577738337,
+            -0.020079061996873537,
+            0.013911518328924777,
+            0.0347210775522539,
+            -0.017176114991595492,
+            -0.0018941772168839354,
+            0.019702722555622765,
+            -0.004221469612316137,
+            0.01212088353057276,
+            0.0063707489618353725,
+            0.015298014532351866,
+            0.0038643174573634725,
+            -0.012181479717826009,
+            -0.007374005740809251,
+            0.019411457664902376,
+            -0.004902591844710158,
+            0.014591446725717338,
+            0.015069599884557399,
+            -0.008675672164385891,
+            0.002544198867387345,
+            0.01951865904496276,
+            -0.01762101777798757,
+            0.01665654707715257,
+            -0.00458402229480694,
+            -0.009936206685538294,
+            -0.013345740581684638,
+            -0.005336416724426228,
+            -0.019469457091169776,
+            0.0061608454109892425,
+            0.018931676738452464,
+            -0.00533013594306834,
+            0.009310871143727683,
+            -0.0037670793158045,
+            -0.005045151384034014,
+            0.0002041909886640482,
+            -0.04152751596227068,
+            -0.008340911336217609,
+            -0.0006651584370005978,
+            -0.02505748815737022,
+            0.025954102004571416,
+            0.007205602377144382,
+            0.022602344303400163,
+            -0.017168610245491406,
+            0.004459735690564842,
+            -0.005882024209102714,
+            0.013217517324383834,
+            0.02255689669623969,
+            0.0017984057469785134,
+            0.005699128497215882,
+            -0.02353849721739205,
+            -0.0034711576718386185,
+            0.015924913374553738,
+            -0.003604478141616307,
+            -0.0034993903155063817,
+            0.012592308877056896,
+            0.012864938348003884,
+            0.01567418642605425,
+            0.008801753550506258,
+            0.015606443428689292,
+            0.010589933994371628,
+            0.008928721556432368,
+            -0.005805129999077126,
+            -0.001025103064751441,
+            -0.015907755521365925,
+            0.0045874330999415985,
+            0.031701049450012876,
+            0.012594483199773746,
+            0.016783858873835704,
+            -0.01778536268185936,
+            0.00789523134933153,
+            -0.0012706411671644011,
+            0.008325108613530198,
+            -0.001677173669891956,
+            0.029540136288412976,
+            0.011739999605699826,
+            0.015670966746894652,
+            0.008732334090105053,
+            -0.016237731209426752,
+            0.045334690225148,
+            -0.016911928942220152,
+            -0.0021139311364009113,
+            -0.01058970056572469,
+            -0.021705338531028618,
+            0.007324792557651635,
+            -0.006556582104686972,
+            0.0018243188317438144,
+            0.010780292738415671,
+            -0.006471850255151019,
+            0.0006997395548596146,
+            0.003907433192252573,
+            -0.023129313269763224,
+            -0.002550875642832235,
+            -0.02237917712764273,
+            -0.003107775639788617,
+            -0.02621608419628401,
+            -0.00012482032027919036,
+            -0.016207377503696517,
+            -0.0038712739995960753,
+            -0.023315508569288476,
+            -0.014963211090562272,
+            -0.01633110043829169,
+            -0.004241081909202262,
+            -0.008295402942465353,
+            0.004976396647108597,
+            -0.007402107755970892,
+            -0.020321297548927468,
+            0.0020986169430507757,
+            0.016803273183070004,
+            0.01434729320047051,
+            0.018612970083178984,
+            0.00603868809307069,
+            0.014278001198851581,
+            -0.0026350854954985117,
+            0.017171016209107656,
+            -0.006968968261909459,
+            0.006881847166565678,
+            0.011454001418121107,
+            0.02140935294963631,
+            0.011121833270235448,
+            -0.012801756637470525,
+            0.0010870041670931706,
+            -0.009655388034828094,
+            0.002094601818569396,
+            -0.00828486034340067,
+            -0.009190975158350585,
+            0.0017762581946855365,
+            -0.007261996868906865,
+            0.004674405202069334,
+            0.0030526253833799426,
+            -0.0014082232541965316,
+            0.010419302200840906,
+            -0.0024200053377751805,
+            -0.012208685875688886,
+            -0.0033037330470612695,
+            -0.017975940046938498,
+            -0.006958042903103732,
+            -0.0014496522238488239,
+            -0.011711502944602638,
+            -0.01016130418405875,
+            0.003321436189059608,
+            -0.00538282506552848,
+            -0.009353581612407435,
+            0.013884147453136417,
+            0.0001309738696853529,
+            0.0005878330075918333,
+            0.023495280602974504,
+            0.006035028203642483,
+            0.008087594627921345,
+            0.013144288945385904,
+            -0.01023995000994454,
+            -0.022574958230635715,
+            -0.0243435727055662,
+            0.007867022215096418,
+            -0.01573478573423379,
+            0.02324699908478915,
+            -0.007929809922043611,
+            -0.00561630876431978,
+            0.0020982723335057625,
+            -0.013699317926847062,
+            0.008578431496569186,
+            -0.011271381061801356,
+            0.0014450645788554738,
+            -0.019581451799757368,
+            0.018059311800267413,
+            -0.010917911869373693,
+            -0.004302687883018328,
+            0.009466103226841532,
+            0.034049086327052606,
+            -0.0005689605174080737,
+            -0.007410970644432269,
+            -0.006594150877740108,
+            -0.00022810271313786974,
+            0.018766765773409123,
+            -0.0029041815756733333,
+            0.005758781908059541,
+            -0.01563519107805783,
+            0.006571628287171105,
+            -0.006132372252825422,
+            0.0012423261344297225,
+            0.01513074831647933,
+            -0.026306320707079913,
+            -0.015491710256868008,
+            -0.007857352164317122,
+            -0.024893430640551818,
+            0.002253574511878983,
+            0.017579477884206534,
+            0.020505606775521504,
+            -0.020324505029926956,
+            -0.02513332780748072,
+            0.005150966733930881,
+            -0.026580994694344065,
+            0.0037390577082486594,
+            0.1241031878517593,
+            -0.002138404898309707,
+            0.002720723037908521,
+            -0.011425036886011179,
+            0.0022399461009316216,
+            0.005389407121650262,
+            -0.019552543955456567,
+            0.05950746473186985,
+            -0.00638023587296763,
+            -0.004023798021748334,
+            0.002170545188776296,
+            0.001785855390676113,
+            -0.005831534307818096,
+            0.008643501093778434,
+            -0.017602787281866816,
+            -0.02184792441891879,
+            -0.0052215705748838905,
+            -0.021252169339074325,
+            -0.008509621422071373,
+            -0.013000137524907565,
+            0.002893880203681282,
+            -0.015821609145923173,
+            0.010397692426614709,
+            -0.01638916243796692,
+            -0.010283706937701179,
+            -0.01103621454761223,
+            -0.00373552043529251,
+            0.01563553822905027,
+            -0.03188533919653949,
+            -0.004873550301425776,
+            -0.005183590054152726,
+            0.018218971950313273,
+            0.006004328765748325,
+            0.004357734370247252,
+            -0.014230318971458851,
+            0.0064792070801456235,
+            0.0215589173714926,
+            0.0055272493764401305,
+            0.014643335833970546,
+            0.010618070641357003,
+            -0.0022177303813575846,
+            -0.0024402761885758007,
+            0.017541799738501007,
+            -0.005445774414228888,
+            -0.00023502936119626386,
+            -0.00537569046685066,
+            -0.005974703247446444,
+            -0.010712244203425036,
+            0.01843613166635215,
+            -0.009938564643456672,
+            -0.017388592763277214,
+            -0.0015797412875366156,
+            0.02028434083164951,
+            -0.012761534382003973,
+            0.0031906812114950982,
+            0.000788436210275669,
+            0.020589196433948156,
+            0.0007726855314648301,
+            -0.005481260996289263,
+            0.007165014826294827,
+            -0.02956334397512344,
+            0.004095715232997753,
+            0.0003568512047626187,
+            0.027290768799658912,
+            -0.0017631129386615335,
+            0.01552542586826711,
+            -0.020871286597113327,
+            0.004388125824324094,
+            -0.016926008631435403,
+            0.005099899758322398,
+            0.012560923968841436,
+            0.002781049890810361,
+            0.010889021753593997,
+            -0.008500358897706648,
+            -0.021353460542554444,
+            0.012663934009161902,
+            -0.018657094130860155,
+            -0.029156017587920958,
+            0.0018889388761887053,
+            -0.028158076315864772,
+            -0.003048854609750186,
+            0.006812072971079798,
+            -0.035217412591558014,
+            -0.02251675180883659,
+            0.007396524512996277,
+            -0.00995571600201846,
+            -0.011912753666073882,
+            0.010675240017836214,
+            -0.02645155743041798,
+            -0.018137648527851668,
+            0.0010666944556449141,
+            0.012588800077165337,
+            -0.004561825939541096,
+            0.007555558090958502,
+            -0.028558603070876605,
+            0.014816816621445352,
+            -0.01313764261744744,
+            0.016036065063831813,
+            -0.0037797584923144134,
+            0.022217516257304868,
+            0.011131597210722573,
+            0.008795376259031849,
+            -0.004950080558710245,
+            0.021409590863550987,
+            0.005574351961953293,
+            0.051358937509801736,
+            0.031972009383746405,
+            0.0019821003238211276,
+            -0.003251257665378234,
+            0.003313045083311649,
+            0.008294776423730887,
+            -0.00940948257229233,
+            0.005114631360856872,
+            0.028053227377164806,
+            0.016738013668710083,
+            -0.018755924318109216,
+            0.006606711923035118,
+            -0.02702692774986498,
+            0.017950884766781056,
+            -0.022054653845163,
+            0.0003755944703013671,
+            0.007766253634286795,
+            0.0016206993755471613,
+            0.010682347575070117,
+            0.0013842930079185415,
+            0.01705234255261425,
+            0.023975191051210525,
+            -0.002976402980885077,
+            0.017466027107477605,
+            0.012734909275073793,
+            0.018143161773162015,
+            -0.0038399373197265827,
+            0.006929440921265761,
+            -0.012003095293274786,
+            -0.0008883470597690258,
+            0.0020490527756307267,
+            -0.004231091383919565,
+            0.012128848448049833,
+            0.008340302519160299,
+            0.012989849936937843,
+            0.002974488532907779,
+            0.003198854381461613,
+            -0.005309753760578859,
+            -0.0017024982049936384,
+            -0.01763395035509857,
+            -0.01819680226734474,
+            0.028417724274372468,
+            -0.020217879775485576,
+            -0.050118184195774004,
+            0.0025509548991664425,
+            0.014185761006969567,
+            -0.0012021669304388173,
+            -0.0051260576846881515,
+            -0.0005774778872927327,
+            0.005718864494530703,
+            -0.016918749210038116,
+            -0.0015472638835434902,
+            0.008125845907627804,
+            0.020347547062899797,
+            -0.002036218493206914,
+            -0.01079739518866248,
+            0.0038112958713600906,
+            0.02169339303105242,
+            0.01667594322285835,
+            0.012338193092567552,
+            -0.009798409233733167,
+            0.0003198215558702285,
+            -1.6308740825163337e-05,
+            0.0076870827874737325,
+            0.004356644356656038,
+            0.003500023005601613,
+            -0.019968549869439228,
+            -0.008634740687755112,
+            0.012781565963545847,
+            -0.0025637758121512927,
+            0.010243567793632335,
+            -0.00042296044205132426,
+            -0.003069251553118065,
+            -0.009344199449486244,
+            -0.004694953215957636,
+            -0.009583993744231791,
+            0.013783878036521561,
+            -0.007705982042252044,
+            -0.01549567538665377,
+            0.00020666298495823114,
+            0.0005515775451348472,
+            0.001667367278489879,
+            -0.004518431526236256,
+            0.03302324454129141,
+            0.009531990824506103,
+            -0.0004937348108340977,
+            -0.004047613286672229,
+            -0.0327383112522095,
+            0.012456150771224677,
+            -0.009677533214778067,
+            0.0006455811519981003,
+            0.020012976827426605,
+            -0.00020277620481825933,
+            0.002168432791818704,
+            -0.015657551629104767,
+            0.0032924357559238502,
+            0.006161295735841883,
+            0.013814165416788555,
+            0.0005164426482972658,
+            0.01076512534105537,
+            -0.006498977875968572,
+            -0.010325144017898257,
+            0.004804158067334203,
+            0.0104064855510473,
+            -0.0012188100892986933,
+            0.005171197790296134
+          ],
+          "title": "Mixture-Of-Experts With Expert Choice Routing",
+          "keyphrases": [
+            "Mixture-Of-Experts",
+            "Expert Choice Routing",
+            "sparsely-activated models",
+            "training efficiency",
+            "downstream performance"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "2d39419a-1397-4001-8e1e-a802772c4f79",
+        "properties": {
+          "page_content": "E Fine-Tuning Details\n\nWe did a hyperparameter search for both baseline models and expert choice method. For fine-tuning of the 8B dense model, we use a constant learning rate of 0.0001 and a dropout rate of 0.1. We freeze the attention layer and feed-forward layer while leaving the embedding and layer normalization trainable. This setting has been found optimal for the 8B dense model. For MoE 8B/64E models including GShard top-2 gating and expert choice, we found continuing the learning rate from the pre-trained model while using a square root learning rate decay works better. In addition, we do not apply parameter freezing for fine-tuning MoE models. For models with 100M expert size, we use a constant learning rate of 0.0001 and no dropout is used.",
+          "title": "E Fine-Tuning Details",
+          "embedding": [
+            0.022390764206647873,
+            -0.0034248128067702055,
+            -0.016889112070202827,
+            -0.013610072433948517,
+            0.024709414690732956,
+            0.026149997487664223,
+            -0.02159501425921917,
+            0.05106521025300026,
+            -0.016985150054097176,
+            0.006918224971741438,
+            0.008664073422551155,
+            -0.005028318148106337,
+            0.013939348049461842,
+            0.006582088768482208,
+            0.030732421204447746,
+            0.0230218768119812,
+            0.008314217440783978,
+            0.024915212765336037,
+            0.007045133039355278,
+            -0.03493069112300873,
+            0.022267285734415054,
+            -0.0422845184803009,
+            0.0016892541898414493,
+            -0.017520224675536156,
+            0.0009578156750649214,
+            -0.003855272429063916,
+            0.014789978042244911,
+            0.014254904352128506,
+            -0.01691655069589615,
+            -0.0006456895498558879,
+            0.039293594658374786,
+            0.015421089716255665,
+            0.0020151000935584307,
+            0.005889237392693758,
+            -0.05144936591386795,
+            -0.019770275801420212,
+            0.042311958968639374,
+            0.03193976730108261,
+            -0.0038278328720480204,
+            0.021361777558922768,
+            0.03226904198527336,
+            0.015064374543726444,
+            -0.023460911586880684,
+            -0.014995775185525417,
+            -0.019139165058732033,
+            -0.019098004326224327,
+            -0.017163507640361786,
+            0.007635085843503475,
+            0.004493244457989931,
+            -0.06272706389427185,
+            -0.01129142101854086,
+            -0.024627096951007843,
+            0.009597022086381912,
+            0.02502497099339962,
+            -0.044781528413295746,
+            -0.0015443384181708097,
+            -0.00871209241449833,
+            0.06366001814603806,
+            0.05512627959251404,
+            0.0056354207918047905,
+            0.022527962923049927,
+            0.0003787959576584399,
+            -0.040638137608766556,
+            -0.015023214742541313,
+            0.025903040543198586,
+            0.011387459933757782,
+            0.027220144867897034,
+            -0.023309992626309395,
+            -0.024174341931939125,
+            0.003937591332942247,
+            -0.04969322681427002,
+            0.03051290474832058,
+            -0.018123896792531013,
+            0.014186304993927479,
+            -0.005940686911344528,
+            -0.04017166420817375,
+            0.020140711218118668,
+            0.023749027401208878,
+            0.039952147752046585,
+            -0.011373739689588547,
+            0.015187853015959263,
+            -0.026561593636870384,
+            -0.00991943757981062,
+            0.040308862924575806,
+            0.052601832896471024,
+            -0.006551219150424004,
+            -0.022774919867515564,
+            -0.036549631506204605,
+            -0.026410674676299095,
+            0.015352491289377213,
+            -0.02174593135714531,
+            -0.005419333465397358,
+            -0.019619358703494072,
+            -0.006074455101042986,
+            -0.029278120025992393,
+            0.047168780118227005,
+            -0.032735515385866165,
+            0.011908813379704952,
+            -0.02487405389547348,
+            -0.0023117915261536837,
+            -0.016710754483938217,
+            0.005851508118212223,
+            -0.0023615260142832994,
+            -0.002745681209489703,
+            0.02949763648211956,
+            -0.01580524444580078,
+            -0.00518266623839736,
+            -0.005628560669720173,
+            0.0011584681924432516,
+            0.05894039571285248,
+            -0.024627096951007843,
+            -0.04722365736961365,
+            -0.016271719709038734,
+            0.03597339615225792,
+            -0.043354667723178864,
+            0.01352775376290083,
+            -0.008101560175418854,
+            -0.013013259507715702,
+            0.012965240515768528,
+            -0.04494616761803627,
+            0.03976007178425789,
+            0.01974283717572689,
+            0.034162379801273346,
+            0.012690844014286995,
+            -0.04423273354768753,
+            0.03284527361392975,
+            0.021938009187579155,
+            0.014817417599260807,
+            0.004798511043190956,
+            -0.021732212975621223,
+            0.016930270940065384,
+            0.008163299411535263,
+            -0.022706320509314537,
+            -0.0387173630297184,
+            0.0006821328424848616,
+            -0.0073538292199373245,
+            0.04705902189016342,
+            0.05186096206307411,
+            -0.026945747435092926,
+            -0.03528740629553795,
+            0.014693939127027988,
+            -0.01673819310963154,
+            0.026712510734796524,
+            0.004249717574566603,
+            -0.03471117094159126,
+            0.021512694656848907,
+            0.027000626549124718,
+            0.06689789891242981,
+            0.008897310122847557,
+            0.03155561164021492,
+            -0.03427213802933693,
+            0.005693729966878891,
+            -0.03163792937994003,
+            0.00961760152131319,
+            -0.005920107010751963,
+            -0.007587066385895014,
+            -0.02540912665426731,
+            0.0006538357120007277,
+            -0.015174132771790028,
+            0.0062905424274504185,
+            -7.942495722090825e-05,
+            0.007401848677545786,
+            0.0049631488509476185,
+            0.03336662799119949,
+            0.011627556756138802,
+            0.06398928910493851,
+            0.007834023796021938,
+            0.01986631564795971,
+            -0.07842255383729935,
+            0.028317730873823166,
+            0.040994856506586075,
+            0.017739741131663322,
+            0.00991943757981062,
+            0.007984941825270653,
+            0.0012545069912448525,
+            0.025971639901399612,
+            0.016326598823070526,
+            0.023762747645378113,
+            -0.031418412923812866,
+            0.02243192307651043,
+            -0.01140803936868906,
+            0.009898858144879341,
+            0.021732212975621223,
+            0.00917170662432909,
+            0.0123341279104352,
+            -0.01002233661711216,
+            -0.007415568456053734,
+            0.030677542090415955,
+            -0.0015400510746985674,
+            0.00259133311919868,
+            0.0060950350016355515,
+            0.020209310576319695,
+            -0.005601121112704277,
+            -0.0056731500662863255,
+            0.007628226187080145,
+            -0.0274396613240242,
+            0.004613293334841728,
+            -0.014515581540763378,
+            0.008259338326752186,
+            -0.013678671792149544,
+            -0.010289872996509075,
+            -0.03328431025147438,
+            -0.007820303551852703,
+            -0.04431505501270294,
+            0.006654118187725544,
+            0.0023580959532409906,
+            0.01473509892821312,
+            -0.01882360875606537,
+            0.04211987927556038,
+            -0.006077885162085295,
+            -0.001488601672463119,
+            -0.018988246098160744,
+            -0.02525820955634117,
+            -0.01018697489053011,
+            -0.02695946767926216,
+            0.02335115149617195,
+            0.01634031906723976,
+            0.023159075528383255,
+            0.015325050801038742,
+            -0.012313548475503922,
+            0.02746710181236267,
+            0.029717154800891876,
+            0.0010469945846125484,
+            -0.006835905835032463,
+            -0.038168568164110184,
+            -0.03122633509337902,
+            0.008444556035101414,
+            -0.017726020887494087,
+            0.013431714847683907,
+            -0.00541247334331274,
+            -0.01422746479511261,
+            -0.004575563594698906,
+            -0.023762747645378113,
+            0.019043125212192535,
+            0.02012699283659458,
+            0.006005856208503246,
+            0.002215752610936761,
+            -0.032433681190013885,
+            0.038909438997507095,
+            -0.01458417996764183,
+            -0.00680160615593195,
+            0.011696156114339828,
+            -0.018590370193123817,
+            -0.04447969049215317,
+            0.014968335628509521,
+            -0.008135859854519367,
+            -0.01580524444580078,
+            6.827759352745488e-05,
+            0.007491027936339378,
+            -0.006427740678191185,
+            -0.030046429485082626,
+            -0.03558924049139023,
+            -0.005481072701513767,
+            0.00847199559211731,
+            -0.009775379672646523,
+            0.04601631313562393,
+            0.03797649219632149,
+            -0.012704563327133656,
+            0.002215752610936761,
+            -0.02090902253985405,
+            0.040967416018247604,
+            9.405587479704991e-05,
+            -0.00932262558490038,
+            -0.0003839409037027508,
+            -0.008094700053334236,
+            -0.005031748209148645,
+            0.024407578632235527,
+            -0.00319843553006649,
+            0.016024762764573097,
+            0.007676245644688606,
+            0.014597900211811066,
+            -0.006715857423841953,
+            -0.013863889500498772,
+            0.0021608732640743256,
+            0.02388622611761093,
+            -0.018384573981165886,
+            0.006842765491455793,
+            -0.027906136587262154,
+            0.0010975864715874195,
+            0.011744175106287003,
+            0.006232233252376318,
+            0.03454653546214104,
+            0.0224593635648489,
+            0.029634835198521614,
+            -0.03369590640068054,
+            -0.011627556756138802,
+            -0.03471117094159126,
+            -0.03907407820224762,
+            -0.015105534344911575,
+            0.004887689836323261,
+            -0.0009843978332355618,
+            -0.025189610198140144,
+            -0.009164847433567047,
+            -0.0026496422942727804,
+            -0.007662525400519371,
+            0.026835989207029343,
+            -0.015283891931176186,
+            0.007532187271863222,
+            0.00599556602537632,
+            -0.028207972645759583,
+            -0.03177512809634209,
+            -0.0002012956392718479,
+            0.003044087439775467,
+            -0.01789066009223461,
+            -0.0331745520234108,
+            -0.004390345886349678,
+            0.01697143167257309,
+            0.011174802668392658,
+            -0.041653405874967575,
+            0.00736754946410656,
+            0.021402936428785324,
+            -0.005913247354328632,
+            0.011325720697641373,
+            -0.05452260747551918,
+            0.007024553604424,
+            -0.005014598369598389,
+            -0.0038758523296564817,
+            0.016697034239768982,
+            -0.03117145597934723,
+            0.006071025040000677,
+            0.03292759507894516,
+            0.025422846898436546,
+            -0.03965030983090401,
+            0.00899334903806448,
+            -0.007168611511588097,
+            0.0097959591075778,
+            -0.007655665744096041,
+            0.055702514946460724,
+            -0.03641243278980255,
+            -0.014803697355091572,
+            -0.030595222488045692,
+            -0.0227886401116848,
+            -0.03490325063467026,
+            0.0001791081012925133,
+            -0.05101033300161362,
+            0.003769523464143276,
+            -0.0037043543998152018,
+            0.017465345561504364,
+            0.017588824033737183,
+            -0.008952190168201923,
+            -0.018947087228298187,
+            0.004757351242005825,
+            -0.04008934646844864,
+            -0.028976282104849815,
+            0.0049494290724396706,
+            0.026534153148531914,
+            -0.031363531947135925,
+            -0.013459154404699802,
+            -0.0084102563560009,
+            -0.04329978674650192,
+            0.0007151461904868484,
+            -0.0018041577422991395,
+            0.021814530715346336,
+            0.0021831681951880455,
+            -0.0057554692029953,
+            -0.03226904198527336,
+            -0.01306127943098545,
+            0.00024459886481054127,
+            -0.00391358183696866,
+            0.03366846591234207,
+            0.006746727041900158,
+            -0.02487405389547348,
+            -0.028235411271452904,
+            -0.006650688126683235,
+            -0.0272064246237278,
+            -0.031418412923812866,
+            0.027343623340129852,
+            0.015846405178308487,
+            -0.016326598823070526,
+            0.0035671561490744352,
+            -0.025121010839939117,
+            0.017698582261800766,
+            0.01223122887313366,
+            0.014776257798075676,
+            0.030677542090415955,
+            -0.009988036938011646,
+            -0.0024575646966695786,
+            -0.020991342142224312,
+            -0.05183352157473564,
+            0.024105742573738098,
+            -0.012368427589535713,
+            0.007470448035746813,
+            0.034189820289611816,
+            -0.008499435149133205,
+            0.03196720778942108,
+            0.028317730873823166,
+            0.0037043543998152018,
+            0.016203120350837708,
+            -0.0460437536239624,
+            0.03520508483052254,
+            0.018329694867134094,
+            -0.06069653108716011,
+            0.016326598823070526,
+            0.019070565700531006,
+            0.013219057582318783,
+            0.049857866019010544,
+            -0.028262851759791374,
+            -0.006492909975349903,
+            -0.004112519323825836,
+            0.004455515183508396,
+            -0.027576860040426254,
+            0.018453173339366913,
+            0.007415568456053734,
+            -0.03978750854730606,
+            0.007772284094244242,
+            -0.013925628736615181,
+            -0.014001087285578251,
+            -0.004304597154259682,
+            0.0400344654917717,
+            -0.03635755181312561,
+            -0.02261028066277504,
+            -0.013088718988001347,
+            -0.0036494750529527664,
+            -0.002870874712243676,
+            0.02812565304338932,
+            0.03709842264652252,
+            -0.018755009397864342,
+            0.010468230582773685,
+            -0.007443008478730917,
+            -0.014254904352128506,
+            0.00044632324716076255,
+            0.01163441687822342,
+            -0.016930270940065384,
+            0.003308194223791361,
+            0.007093152496963739,
+            -0.0071343122981488705,
+            -0.02121085859835148,
+            0.022267285734415054,
+            0.024256661534309387,
+            -0.013500313274562359,
+            0.00056208431487903,
+            -0.0007906052633188665,
+            0.030787300318479538,
+            -0.03811369091272354,
+            -0.03940335288643837,
+            -0.01440582238137722,
+            -0.00802610069513321,
+            0.034491654485464096,
+            -0.03794905170798302,
+            0.0024078304413706064,
+            -0.006842765491455793,
+            -0.005597691051661968,
+            -0.005782908760011196,
+            0.0013642656849697232,
+            -0.011956832371652126,
+            -0.01691655069589615,
+            -0.02782381698489189,
+            -0.016504956409335136,
+            -0.02723386511206627,
+            -0.007161751855164766,
+            -0.053287822753190994,
+            0.01197741273790598,
+            -0.010111515410244465,
+            -0.020826702937483788,
+            0.04072045907378197,
+            -0.00449667451903224,
+            0.006071025040000677,
+            -0.01583268493413925,
+            0.007189191412180662,
+            -0.050598736852407455,
+            -0.01819249615073204,
+            -0.0011936252703890204,
+            0.04316258803009987,
+            0.0065649389289319515,
+            -0.022171247750520706,
+            -0.0007284372695721686,
+            -0.022692600265145302,
+            -0.008863010443747044,
+            0.006355711724609137,
+            0.013520893640816212,
+            -0.027700338512659073,
+            -0.022692600265145302,
+            0.009562722407281399,
+            0.015407370403409004,
+            0.024613376706838608,
+            0.04428761452436447,
+            -0.011538377963006496,
+            -0.061190444976091385,
+            0.01717722788453102,
+            -0.015036934986710548,
+            0.028029615059494972,
+            -0.005323294550180435,
+            -0.0008073262870311737,
+            -0.01771230250597,
+            -0.014762538485229015,
+            -0.004976868629455566,
+            0.019948633387684822,
+            -0.006108754780143499,
+            0.0035568661987781525,
+            0.031390972435474396,
+            -0.016422636806964874,
+            0.0037146443501114845,
+            -0.0012545069912448525,
+            0.0039787511341273785,
+            -0.01640891842544079,
+            -0.05007738247513771,
+            0.025793282315135002,
+            -0.040939975529909134,
+            0.010536829940974712,
+            -0.0021385785657912493,
+            -0.02108738012611866,
+            -0.01575036533176899,
+            0.01370611134916544,
+            -0.015668047592043877,
+            0.013500313274562359,
+            0.030046429485082626,
+            0.02266516163945198,
+            0.045055925846099854,
+            -0.036933783441782,
+            -0.007285230327397585,
+            0.03844296559691429,
+            -0.014309783466160297,
+            -0.003944451455026865,
+            -0.013129877857863903,
+            0.0026324924547225237,
+            0.027576860040426254,
+            -0.01006349641829729,
+            0.04911699518561363,
+            -0.006359141785651445,
+            0.009631321765482426,
+            0.010660308413207531,
+            -0.00233408622443676,
+            -0.012642824091017246,
+            0.04192780330777168,
+            -0.029003722593188286,
+            0.012937800958752632,
+            0.012073451653122902,
+            -0.011099343188107014,
+            0.020442547276616096,
+            -0.0004986301064491272,
+            0.002894884441047907,
+            0.015517128631472588,
+            0.008616053499281406,
+            -0.04483640938997269,
+            -0.006372861564159393,
+            -0.01175103522837162,
+            -0.020497428253293037,
+            0.022500522434711456,
+            -0.013692391104996204,
+            -0.019619358703494072,
+            -0.03147329390048981,
+            -0.019770275801420212,
+            -0.0037180741783231497,
+            -0.003611745545640588,
+            0.00996059738099575,
+            -0.0018041577422991395,
+            0.026616472750902176,
+            -0.03605571761727333,
+            0.011435478925704956,
+            0.030458025634288788,
+            -0.005467352457344532,
+            -0.024297820404171944,
+            -0.03391542285680771,
+            0.025930481031537056,
+            -0.02073066495358944,
+            -0.019646797329187393,
+            0.004397206008434296,
+            0.027906136587262154,
+            0.012807462364435196,
+            0.002601622836664319,
+            -0.005957836750894785,
+            0.00736754946410656,
+            -0.01684795320034027,
+            -0.007840882986783981,
+            -0.024407578632235527,
+            -0.04244915768504143,
+            0.015489689074456692,
+            -0.004716191906481981,
+            -0.011428618803620338,
+            0.016065921634435654,
+            -0.02248680219054222,
+            -0.01577780582010746,
+            0.008458275347948074,
+            0.01261538453400135,
+            0.00167038943618536,
+            0.002545028692111373,
+            -0.006139624398201704,
+            0.0024181201588362455,
+            -0.0041605387814342976,
+            0.008375956676900387,
+            0.04516568407416344,
+            -0.016299158334732056,
+            -0.006609528791159391,
+            0.035863637924194336,
+            0.027576860040426254,
+            -0.008417116478085518,
+            -0.010660308413207531,
+            -0.013726690784096718,
+            0.0022929266560822725,
+            -0.033778224140405655,
+            0.023749027401208878,
+            0.051559124141931534,
+            0.007326389662921429,
+            -0.024531057104468346,
+            -0.030650101602077484,
+            -0.015709206461906433,
+            -0.005398753564804792,
+            0.049171872437000275,
+            0.02446245774626732,
+            -0.043683942407369614,
+            -0.02601279877126217,
+            0.03161048889160156,
+            0.04469921067357063,
+            0.0077517046593129635,
+            -0.011997992172837257,
+            0.0050557577051222324,
+            0.030293386429548264,
+            0.01628543995320797,
+            -0.03528740629553795,
+            0.011353160254657269,
+            0.028592128306627274,
+            -0.0013273936929181218,
+            -0.012265528552234173,
+            -0.004163968842476606,
+            -0.006300832144916058,
+            0.015366210602223873,
+            -0.008773831650614738,
+            -0.009302045218646526,
+            -0.016697034239768982,
+            -0.026712510734796524,
+            0.07606274634599686,
+            -0.00541247334331274,
+            0.022774919867515564,
+            0.008890450932085514,
+            0.00915798731148243,
+            0.04044606164097786,
+            0.003930731676518917,
+            -0.009494123049080372,
+            0.011867653578519821,
+            -0.006976534146815538,
+            -0.02621859684586525,
+            -0.011819634586572647,
+            -0.01605220139026642,
+            -0.0019053415162488818,
+            0.016450077295303345,
+            -0.006221943534910679,
+            0.03018362820148468,
+            0.03737282007932663,
+            0.013150458224117756,
+            -0.01854921132326126,
+            -0.01580524444580078,
+            0.00403363024815917,
+            0.01849433220922947,
+            0.03986983001232147,
+            -0.011373739689588547,
+            -0.0018436022801324725,
+            0.0026925168931484222,
+            -0.006414020899683237,
+            0.003155561164021492,
+            -0.011435478925704956,
+            -0.013712971471250057,
+            -0.008142719976603985,
+            0.022006608545780182,
+            0.0311988964676857,
+            -0.0031864307820796967,
+            -0.027508260682225227,
+            -0.004013050347566605,
+            0.022377043962478638,
+            0.01575036533176899,
+            0.0004411783011164516,
+            0.0066198185086250305,
+            0.014103986322879791,
+            0.006335131824016571,
+            -0.0015443384181708097,
+            0.02487405389547348,
+            -0.02645183354616165,
+            0.0016129376599565148,
+            0.021718492731451988,
+            -0.02431154064834118,
+            0.02436641976237297,
+            -0.011654996313154697,
+            -0.030759861692786217,
+            -0.027371061965823174,
+            -0.007141171954572201,
+            0.02764545939862728,
+            -0.02057974599301815,
+            -9.443102317163721e-05,
+            0.0246682558208704,
+            0.01406282652169466,
+            0.0007717405096627772,
+            0.0036494750529527664,
+            0.008506295271217823,
+            0.008835570886731148,
+            -0.02988179214298725,
+            -0.009535282850265503,
+            -0.00014116420061327517,
+            -0.013171037659049034,
+            0.04244915768504143,
+            -0.05485188588500023,
+            -0.013905048370361328,
+            -0.011847074143588543,
+            -0.02347462996840477,
+            -0.01819249615073204,
+            0.004479524679481983,
+            0.0017295562429353595,
+            -0.006379721220582724,
+            -0.024023424834012985,
+            0.0349581278860569,
+            0.00917170662432909,
+            -0.020113272592425346,
+            -0.019962353631854057,
+            0.03048546425998211,
+            -0.01867268979549408,
+            0.03254343941807747,
+            0.0033322039525955915,
+            -0.012622244656085968,
+            -0.005096917040646076,
+            -0.01267712377011776,
+            -0.02159501425921917,
+            -0.03572643920779228,
+            -0.020634625107049942,
+            0.04552239924669266,
+            0.015393650159239769,
+            0.003906721714884043,
+            -0.01572292670607567,
+            0.023282554000616074,
+            -0.008211318403482437,
+            0.009940017946064472,
+            0.011997992172837257,
+            0.05221767723560333,
+            0.0049631488509476185,
+            0.03630267456173897,
+            0.006163633894175291,
+            -0.06206851452589035,
+            0.0012793742353096604,
+            0.020236751064658165,
+            0.020140711218118668,
+            0.03397030010819435,
+            0.016450077295303345,
+            0.004887689836323261,
+            -0.021526414901018143,
+            -0.021197138354182243,
+            -0.04703158140182495,
+            -0.00927460566163063,
+            0.005168946459889412,
+            -0.011661856435239315,
+            -0.02812565304338932,
+            0.04722365736961365,
+            0.024736855179071426,
+            -0.003387083299458027,
+            0.018535491079092026,
+            -0.01009779516607523,
+            0.0016318024136126041,
+            0.01610708236694336,
+            -0.018425732851028442,
+            0.036220353096723557,
+            -0.005337014328688383,
+            0.03081474080681801,
+            -0.025546325370669365,
+            0.02034650929272175,
+            0.02182825095951557,
+            -0.012313548475503922,
+            0.034519094973802567,
+            0.005607980769127607,
+            -0.0623977929353714,
+            -0.0028760195709764957,
+            0.027906136587262154,
+            0.013452294282615185,
+            0.0015837829560041428,
+            0.0022826369386166334,
+            -0.01634031906723976,
+            0.019166603684425354,
+            -0.002447274746373296,
+            0.0115109384059906,
+            0.04524800181388855,
+            -0.0025621785316616297,
+            0.003800393082201481,
+            0.026794830337166786,
+            0.0058549377135932446,
+            -0.014296064153313637,
+            0.0036529048811644316,
+            -0.012039151974022388,
+            0.04524800181388855,
+            -0.03778441622853279,
+            -0.00043946332880295813,
+            0.0522451177239418,
+            -0.005186096299439669,
+            0.004208558239042759,
+            -0.015915004536509514,
+            -0.01867268979549408,
+            -0.005957836750894785,
+            0.02001723274588585,
+            0.004736771807074547,
+            0.04058326035737991,
+            -0.040336303412914276,
+            0.010145815089344978,
+            -0.020895302295684814,
+            0.0041742585599422455,
+            0.010776927694678307,
+            0.015764085575938225,
+            0.013390555046498775,
+            -0.038936879485845566,
+            -0.006983393803238869,
+            0.015283891931176186,
+            0.06398928910493851,
+            -0.006307692267000675,
+            -0.012491906061768532,
+            0.04601631313562393,
+            -0.005313004367053509,
+            -0.015078093856573105,
+            0.016422636806964874,
+            0.004225708078593016,
+            0.02848236821591854,
+            0.02001723274588585,
+            -0.02812565304338932,
+            -0.04137900844216347,
+            0.009137407876551151,
+            0.01473509892821312,
+            0.005309574771672487,
+            -0.001153323333710432,
+            -0.006846195552498102,
+            -0.013966787606477737,
+            -0.013054419308900833,
+            -0.0038209729827940464,
+            -0.007587066385895014,
+            0.023309992626309395,
+            -0.02352951094508171,
+            0.007635085843503475,
+            0.02949763648211956,
+            -0.013493454083800316,
+            -0.020360229536890984,
+            -0.005282134748995304,
+            -0.005127786658704281,
+            -0.002359810983762145,
+            -0.02481917478144169,
+            -0.010488810949027538,
+            -0.018988246098160744,
+            -0.019701676443219185,
+            0.04036374390125275,
+            0.016806792467832565,
+            0.005889237392693758,
+            0.009343205019831657,
+            -0.007292089983820915,
+            0.04500104486942291,
+            -0.028015894815325737,
+            0.05216279625892639,
+            -0.014872296713292599,
+            -0.003529426408931613,
+            -0.005518801975995302,
+            0.02654787339270115,
+            -0.014940896071493626,
+            0.019976073876023293,
+            -0.03215928375720978,
+            0.00703141326084733,
+            0.009144267067313194,
+            -0.005402183625847101,
+            -0.00858861394226551,
+            -0.01124340109527111,
+            0.001927636214531958,
+            -0.0026925168931484222,
+            0.02576584182679653,
+            0.033723343163728714,
+            0.010255573317408562,
+            0.004397206008434296,
+            -0.032790396362543106,
+            -0.005638850387185812,
+            -0.0007215773803181946,
+            -0.0030749570578336716,
+            -0.033723343163728714,
+            -0.018864767625927925,
+            -0.037510018795728683,
+            -0.013925628736615181,
+            0.009809679351747036,
+            -0.0035637260880321264,
+            0.021965449675917625,
+            0.019125444814562798,
+            -0.0017475634813308716,
+            -0.01605220139026642,
+            -0.013870748691260815,
+            0.006942234467715025,
+            0.013308236375451088,
+            -0.01565432734787464,
+            0.0015494833933189511,
+            0.0007811728864908218,
+            -0.010584849864244461,
+            0.021005060523748398,
+            -0.01989375427365303,
+            -0.025601204484701157,
+            -0.008273057639598846,
+            -0.03303735330700874,
+            0.00472648162394762,
+            0.0199211947619915,
+            -0.0038072532042860985,
+            0.01971539668738842,
+            0.009013929404318333,
+            0.002479859394952655,
+            -0.006678127683699131,
+            -0.03386054188013077,
+            -0.031418412923812866,
+            -0.034436777234077454,
+            -0.013857029378414154,
+            0.013994227163493633,
+            0.0008519157418049872,
+            0.02009955234825611,
+            0.008156439289450645,
+            0.00359802576713264,
+            0.018439453095197678,
+            -0.0007511607254855335,
+            -0.013356255367398262,
+            -0.013500313274562359,
+            -0.001734701101668179,
+            -0.01732814684510231,
+            0.0061979335732758045,
+            -0.03084217943251133,
+            0.0460437536239624,
+            -0.02230844460427761,
+            0.009343205019831657,
+            0.014940896071493626,
+            -0.009933157823979855,
+            -0.0026856570038944483,
+            -0.027247583493590355,
+            0.02248680219054222,
+            0.04596143215894699,
+            -0.013342536054551601,
+            0.033394068479537964,
+            0.03056778386235237,
+            0.0224593635648489,
+            -0.025450287386775017,
+            0.02759058028459549,
+            0.013664951547980309,
+            -0.008986488915979862,
+            0.030320826917886734,
+            -0.020442547276616096,
+            0.025875601917505264,
+            -0.007086292840540409,
+            -0.022884678095579147,
+            0.004294306971132755,
+            0.025367967784404755,
+            0.010866106487810612,
+            0.014186304993927479,
+            -0.01918032392859459,
+            0.004061069805175066,
+            0.0029669133946299553,
+            -0.019619358703494072,
+            0.04725109785795212,
+            0.002308361465111375,
+            0.021128538995981216,
+            0.0113943200558424,
+            -0.012773162685334682,
+            0.022034049034118652,
+            -0.016203120350837708,
+            -0.01933124102652073,
+            -0.0028365750331431627,
+            0.010927845723927021,
+            0.015379930846393108,
+            -0.048540763556957245,
+            0.03355870768427849,
+            0.028098214417696,
+            0.009645041078329086,
+            0.0058309282176196575,
+            0.008725812658667564,
+            -0.002208892721682787,
+            0.006352281663566828,
+            -0.00680160615593195,
+            -0.0009157987078651786,
+            0.044754087924957275,
+            -0.01175103522837162,
+            -0.0019019115716218948,
+            -0.0049871583469212055,
+            0.030101308599114418,
+            0.0017158363480120897,
+            -0.008252478204667568,
+            -0.008808131329715252,
+            -0.01458417996764183,
+            0.00013408990344032645,
+            0.00420512817800045,
+            -0.004887689836323261,
+            -0.016244279220700264,
+            -0.027096666395664215,
+            0.01319847721606493,
+            0.004808800760656595,
+            -0.03421725705265999,
+            -0.01060542929917574,
+            -0.02075810357928276,
+            0.010502530261874199,
+            -0.006544359494000673,
+            -0.0030458024702966213,
+            0.00018693269521463662,
+            -0.007923202589154243,
+            -0.014076546765863895,
+            0.03314711153507233,
+            0.003947881516069174,
+            0.04434249550104141,
+            -0.00342652783729136,
+            0.030979378148913383,
+            -0.02621859684586525,
+            0.03473861142992973,
+            -0.004194838460534811,
+            -0.015791526064276695,
+            0.02505241148173809,
+            -0.03498556837439537,
+            0.031089136376976967,
+            0.02225356549024582,
+            0.004983728751540184,
+            -0.0004171686014160514,
+            0.02949763648211956,
+            -0.04563215747475624,
+            -5.0806247600121424e-05,
+            -0.00553252175450325,
+            -0.0035843057557940483,
+            0.012148910202085972,
+            -0.01048195082694292,
+            -0.012965240515768528,
+            0.0027748357970267534,
+            0.015338771045207977,
+            0.02261028066277504,
+            0.03056778386235237,
+            0.00863663386553526,
+            -0.013116158545017242,
+            0.013520893640816212,
+            -0.0012802317505702376,
+            0.0026925168931484222,
+            -0.015969883650541306,
+            0.020181871950626373,
+            0.00204253988340497,
+            -0.03290015459060669,
+            -0.005786338821053505,
+            -0.06234291195869446,
+            -0.009075668640434742,
+            0.007154891733080149,
+            0.009151127189397812,
+            -0.004925419110804796,
+            -0.029717154800891876,
+            -0.038964319974184036,
+            0.017616262659430504,
+            0.02675366960465908,
+            -0.006623248569667339,
+            -0.008773831650614738,
+            -0.0224593635648489,
+            -0.016024762764573097,
+            0.03320199251174927,
+            -0.0007057138136588037,
+            0.012786882929503918,
+            0.004338896367698908,
+            0.03578132018446922,
+            -0.0038689924404025078,
+            -0.010715187527239323,
+            -0.04810172691941261,
+            -0.02949763648211956,
+            0.016244279220700264,
+            0.015475969761610031,
+            -0.009624461643397808,
+            0.019290082156658173,
+            -0.005782908760011196,
+            -0.02550516650080681,
+            -0.017245827242732048,
+            0.03594595566391945,
+            -0.012080310843884945,
+            0.028893964365124702,
+            0.01544852927327156,
+            -0.021512694656848907,
+            0.002509013982489705,
+            -0.0018195925513282418,
+            -0.009398084133863449,
+            0.005700589623302221,
+            0.02484661340713501,
+            0.0013719830894842744,
+            0.005611410830169916,
+            0.011922532692551613,
+            0.008252478204667568,
+            0.002426695078611374,
+            -0.022294726222753525,
+            -0.003380223410204053,
+            0.015215292572975159,
+            -0.03701610490679741,
+            0.008760112337768078,
+            -0.005282134748995304,
+            0.004908269736915827,
+            0.008382816798985004,
+            0.04346442595124245,
+            -0.005048897583037615,
+            0.005216965917497873,
+            -0.011483498848974705,
+            0.01699887029826641,
+            0.011606977321207523,
+            0.0033596435096114874,
+            -0.00541247334331274,
+            -0.010955285280942917,
+            -0.026794830337166786,
+            -0.008691512979567051,
+            0.01388446893543005,
+            0.012505626305937767,
+            0.01457046065479517,
+            0.007209771312773228,
+            0.004942568950355053,
+            -0.016834232956171036,
+            -0.006149914115667343,
+            0.010550550185143948,
+            0.01915288344025612,
+            -0.022061487659811974,
+            -0.02293955720961094,
+            0.01966051757335663,
+            -0.003011502791196108,
+            0.022322164848446846,
+            -0.021883130073547363,
+            -0.020675785839557648,
+            0.026287196204066277,
+            0.0712333619594574,
+            -0.0003322771517559886,
+            -0.010756347328424454,
+            -0.02228100597858429,
+            -0.011888233944773674,
+            -0.01664215512573719,
+            -0.017163507640361786,
+            0.027851257473230362,
+            0.03591851890087128,
+            -0.008238758891820908,
+            0.018466891720891,
+            -0.0020288201048970222,
+            0.0038038231432437897,
+            0.004218847956508398,
+            -0.030430585145950317,
+            0.01326021645218134,
+            -0.007552767172455788,
+            0.006366001442074776,
+            0.012999540194869041,
+            0.003989040851593018,
+            -0.008897310122847557,
+            -0.0018710419535636902,
+            0.009885137900710106,
+            0.012930940836668015,
+            -0.005299284588545561,
+            0.02126573771238327,
+            -0.0062253731302917,
+            0.009981176815927029,
+            -0.011318860575556755,
+            -0.018151337280869484,
+            0.00951470248401165,
+            0.01926264353096485,
+            -0.014474421739578247,
+            0.011936252936720848,
+            -0.007600786164402962,
+            0.03440933674573898,
+            -0.011421759612858295,
+            0.0029754883144050837,
+            -0.0011610407382249832,
+            -0.019454719498753548,
+            -0.002123143756762147,
+            0.0010718618286773562,
+            -0.002905174158513546,
+            0.02052486687898636,
+            0.004064499866217375,
+            0.03770209476351738,
+            0.021402936428785324,
+            -0.04862308129668236,
+            0.03633011132478714,
+            -0.009686200879514217,
+            -0.01388446893543005,
+            0.00857489462941885,
+            -0.009754800237715244,
+            -0.0019207763252779841,
+            -0.016271719709038734,
+            -0.014158865436911583,
+            0.013870748691260815,
+            -0.007086292840540409,
+            0.003745513968169689,
+            -0.004808800760656595,
+            0.015009495429694653,
+            0.0020545446313917637,
+            0.00772426463663578,
+            0.017259547486901283,
+            -0.006348851602524519,
+            -0.01661471463739872,
+            0.03520508483052254,
+            0.021869409829378128,
+            0.0004092368471901864,
+            0.0024215502198785543,
+            -0.004359476268291473,
+            0.02815309353172779,
+            0.009432383812963963,
+            -0.010042916052043438,
+            -0.015942443162202835,
+            -0.0019224913557991385,
+            -0.003222445258870721,
+            0.03657706826925278,
+            0.03180256858468056,
+            -0.02001723274588585,
+            0.0018933366518467665,
+            -0.0011258836602792144,
+            -0.009864558465778828,
+            0.009164847433567047,
+            -0.025903040543198586,
+            0.01816505566239357,
+            -0.006636968348175287,
+            0.007621366064995527,
+            0.02878420613706112,
+            -0.0044486550614237785,
+            0.012656544335186481,
+            0.013973647728562355,
+            0.007587066385895014,
+            0.003308194223791361,
+            0.006242522969841957,
+            0.018096456304192543,
+            -0.006667837966233492,
+            0.005230685696005821,
+            -0.029031163081526756,
+            -0.02075810357928276,
+            -0.034134939312934875,
+            0.01122968178242445,
+            0.011600117199122906,
+            -0.01343857403844595,
+            -0.014028526842594147,
+            0.009315765462815762,
+            0.026849709451198578,
+            0.005649140570312738,
+            -0.0046990420669317245,
+            -0.020620906725525856,
+            0.005553101655095816,
+            -0.005587401334196329,
+            -0.02711038663983345,
+            -0.01364437211304903,
+            0.02469569630920887,
+            0.0045275441370904446,
+            0.019070565700531006,
+            -0.008279917761683464,
+            0.0073538292199373245,
+            0.0004371052491478622,
+            -0.014611619524657726,
+            0.008279917761683464,
+            0.02845492959022522,
+            0.022870957851409912,
+            -0.008622913621366024,
+            0.0019224913557991385,
+            -0.0018333124462515116,
+            -0.007271510548889637,
+            0.021691052243113518,
+            -0.0008163299644365907,
+            -0.009974316693842411,
+            0.025079850107431412,
+            0.011668716557323933,
+            -0.016532396897673607,
+            0.014501861296594143,
+            -0.02594420127570629,
+            0.006297402549535036,
+            0.016175679862499237,
+            0.009603881277143955,
+            -0.005957836750894785,
+            -0.0030921068973839283,
+            0.016902832314372063,
+            0.03128121420741081,
+            0.006379721220582724,
+            0.0027782658580690622,
+            -0.021005060523748398,
+            -0.014639060012996197,
+            -1.834223439800553e-05,
+            0.00466131279245019,
+            0.042641233652830124,
+            -0.009775379672646523,
+            -0.018069017678499222,
+            0.013747270219027996,
+            -0.010433931834995747,
+            -0.0021728782448917627,
+            -0.014323503710329533,
+            0.01864525116980076,
+            0.005433053243905306,
+            0.015860125422477722,
+            0.007758564315736294,
+            -0.005210105795413256,
+            0.02314535528421402,
+            0.0026890868321061134,
+            -0.028674446046352386,
+            0.01834341324865818,
+            0.01613452099263668,
+            0.017012590542435646,
+            -0.006204793695360422,
+            -0.00229978677816689,
+            0.013685530982911587,
+            -0.007978081703186035,
+            0.016326598823070526,
+            0.030732421204447746,
+            0.03133609518408775,
+            -0.0006752729532308877,
+            -0.024119462817907333,
+            0.014611619524657726,
+            -0.003472832264378667,
+            -0.012046011164784431,
+            -0.0255600456148386,
+            0.018604090437293053,
+            -0.0039650313556194305,
+            -0.01631287857890129,
+            -0.009953737258911133,
+            -0.025614924728870392,
+            0.00036314676981419325,
+            -0.010598569177091122,
+            0.003944451455026865,
+            -0.011661856435239315,
+            0.03536972403526306,
+            0.012073451653122902,
+            0.019015686586499214,
+            0.0018607521196827292,
+            -0.02728874422609806,
+            -0.018178775906562805,
+            0.03226904198527336,
+            -0.012882920913398266,
+            -0.00656150933355093,
+            -0.0058309282176196575,
+            -0.0037146443501114845,
+            -0.03569899871945381,
+            0.0154348099604249,
+            0.027343623340129852,
+            0.013795290142297745,
+            0.0010615718783810735,
+            -0.024448739364743233,
+            -0.01557200774550438,
+            0.01249876618385315,
+            0.01422746479511261,
+            -0.026026519015431404,
+            0.027700338512659073,
+            -0.006441460456699133,
+            -0.016203120350837708,
+            0.011620696634054184,
+            -0.007813443429768085,
+            -0.010536829940974712,
+            0.00691136484965682,
+            -0.02767289988696575,
+            0.002752541098743677,
+            0.024435019120573997,
+            -0.019043125212192535,
+            0.00564571050927043,
+            0.012011712417006493,
+            0.035891078412532806,
+            0.013911908492445946,
+            0.01867268979549408,
+            -0.00025253064814023674,
+            -0.02914092130959034,
+            0.01291722059249878,
+            -0.00158464047126472,
+            -0.022377043962478638,
+            0.026794830337166786,
+            -0.015736646950244904,
+            0.017136069014668465,
+            -0.005803488660603762,
+            -0.011552097275853157,
+            0.020236751064658165,
+            0.013225916773080826,
+            0.01237528771162033,
+            -0.0002709666732698679,
+            0.004743631463497877,
+            0.007539046928286552,
+            -0.011346300132572651,
+            0.020181871950626373,
+            -0.013911908492445946,
+            -0.00023923955450300127,
+            -0.004709331784397364,
+            -0.008547455072402954,
+            -0.031418412923812866,
+            0.03427213802933693,
+            0.0014825991820544004,
+            -0.01492717582732439,
+            0.005913247354328632,
+            -0.025422846898436546,
+            -0.025971639901399612,
+            -0.006880495231598616,
+            -0.008725812658667564,
+            -0.0039787511341273785,
+            -0.0054227630607783794,
+            -0.007820303551852703,
+            -0.016889112070202827,
+            0.007792863994836807,
+            0.019399840384721756,
+            0.02764545939862728,
+            -0.005378173664212227,
+            0.0374002605676651,
+            0.01346601452678442,
+            0.02228100597858429,
+            -0.004397206008434296,
+            0.01006349641829729,
+            0.01565432734787464,
+            0.011819634586572647,
+            -0.015530848875641823,
+            0.03572643920779228,
+            -0.0014242900069803,
+            -0.03130865469574928,
+            0.002548458520323038,
+            -0.016957711428403854,
+            -0.012828041799366474,
+            -0.009603881277143955,
+            -0.004218847956508398,
+            0.025464005768299103,
+            0.0020785543601959944,
+            0.023995984345674515,
+            -0.011703016236424446,
+            -0.002551888581365347,
+            0.0007665955345146358,
+            -0.01557200774550438,
+            -0.01082494668662548,
+            -0.026739951223134995,
+            -0.009946877136826515,
+            -0.027096666395664215,
+            0.005443342961370945,
+            0.013514033518731594,
+            0.009020788595080376,
+            -0.00950098317116499,
+            -0.003594595706090331,
+            0.0013291086070239544,
+            -0.025464005768299103,
+            -0.02129317820072174,
+            0.01473509892821312,
+            -0.0007095725159160793,
+            0.004534404259175062,
+            0.010269293561577797,
+            0.015064374543726444,
+            -0.013054419308900833,
+            -0.013253356330096722,
+            -0.01337683480232954,
+            -0.007244070526212454,
+            -0.01974283717572689,
+            -0.0016472372226417065,
+            0.015297611244022846,
+            0.008266198448836803,
+            0.00876697152853012,
+            -0.014200025238096714,
+            0.01562688685953617,
+            0.0028468649834394455,
+            0.0038895721081644297,
+            0.014021667651832104,
+            0.034107498824596405,
+            -0.036522191017866135,
+            -0.0035071317106485367,
+            -0.029964111745357513,
+            0.02182825095951557,
+            0.012162630446255207,
+            0.020332789048552513,
+            -0.012299828231334686,
+            -0.008739531971514225,
+            -0.0362752340734005,
+            -0.0041605387814342976,
+            -0.018027858808636665,
+            0.00719605153426528,
+            0.013088718988001347,
+            -0.009192286990582943,
+            0.04672974348068237,
+            0.0196879580616951,
+            -0.025395406410098076,
+            -0.014446982182562351,
+            0.01989375427365303,
+            0.022857237607240677,
+            -0.006684987805783749,
+            0.02421550080180168,
+            0.0057280296459794044,
+            0.015036934986710548,
+            -0.012485045939683914,
+            -0.0071823312900960445,
+            -0.03081474080681801,
+            -0.013610072433948517,
+            0.007456728257238865,
+            0.005559961311519146,
+            0.011600117199122906,
+            0.003968460951000452,
+            0.009933157823979855,
+            -0.007676245644688606,
+            -0.02711038663983345,
+            0.027247583493590355,
+            0.018178775906562805,
+            0.022226126864552498,
+            -0.020716944709420204,
+            0.0006019576103426516,
+            -0.006770736537873745,
+            0.013932487927377224,
+            0.0048465305007994175,
+            0.0007541619706898928,
+            -0.02182825095951557,
+            0.035452041774988174,
+            -0.02979947254061699,
+            -0.016861671581864357,
+            -0.0007215773803181946,
+            -0.01789066009223461,
+            0.01664215512573719,
+            -0.013616932556033134,
+            -0.029415316879749298,
+            0.016573555767536163,
+            0.015544568188488483,
+            0.015311331488192081,
+            0.0035500063095241785,
+            0.015846405178308487,
+            -0.012416446581482887,
+            0.001987660536542535,
+            -0.022774919867515564,
+            -0.004122809041291475,
+            0.009507843293249607,
+            -0.025820722803473473,
+            -0.024078303948044777,
+            0.004119379445910454,
+            0.010502530261874199,
+            -0.017972977831959724,
+            0.039595432579517365,
+            0.011531517840921879,
+            -0.008828711695969105,
+            0.005649140570312738,
+            0.0005252122646197677,
+            -0.006383151281625032,
+            0.04700414091348648,
+            0.027014346793293953,
+            -0.022212406620383263,
+            0.013445434160530567,
+            -0.004071359988301992,
+            0.012828041799366474,
+            -0.004894549492746592,
+            0.017108628526329994,
+            -0.02012699283659458,
+            0.009850838221609592,
+            -0.011353160254657269,
+            0.006084744818508625,
+            0.007936921902000904,
+            -0.023872505873441696,
+            -0.02660275250673294,
+            -0.036467310041189194,
+            0.008965909481048584,
+            -0.023749027401208878,
+            -0.01312301866710186,
+            0.028619566932320595,
+            -0.011147363111376762,
+            -0.03161048889160156,
+            0.027700338512659073,
+            0.00910996738821268,
+            -0.014014807529747486,
+            0.024352699518203735,
+            0.019962353631854057,
+            0.0069628143683075905,
+            0.0067124273627996445,
+            -0.01048195082694292,
+            -0.013534612953662872,
+            0.0013625507708638906,
+            0.004215417895466089,
+            0.00524097541347146,
+            -0.029387878254055977,
+            -0.009905718266963959,
+            0.0016369473887607455,
+            0.01897452585399151,
+            0.00673300726339221,
+            0.01124340109527111,
+            -0.006029865704476833,
+            0.002486719284206629,
+            -6.742010737070814e-05,
+            0.010296733118593693,
+            -0.004287447314709425,
+            -0.0019259213004261255,
+            0.0127800228074193,
+            0.028647007420659065,
+            -0.024627096951007843,
+            0.008629773743450642,
+            -0.016655875369906425,
+            -0.015393650159239769,
+            0.003227590350434184,
+            0.007038273382931948,
+            0.0018093027174472809,
+            -0.007456728257238865,
+            -0.00771740498021245,
+            0.006897645071148872,
+            -0.008451416157186031,
+            0.0019173463806509972,
+            -0.02012699283659458,
+            0.00431831693276763,
+            0.027782658115029335,
+            0.007292089983820915,
+            -0.034491654485464096,
+            -0.0019430710235610604,
+            -0.015009495429694653,
+            -0.011895093135535717,
+            -0.009041368961334229,
+            0.018617810681462288,
+            0.01631287857890129,
+            -0.013562052510678768,
+            0.0067913164384663105,
+            0.002584473229944706,
+            0.027933575212955475,
+            -0.011483498848974705,
+            -0.007278370205312967,
+            0.01923520304262638,
+            -0.00697310408577323,
+            0.002812565304338932,
+            0.027810096740722656,
+            -0.003769523464143276,
+            -0.017040029168128967,
+            0.0008077550446614623,
+            0.00961760152131319,
+            0.0084102563560009,
+            -0.01798669807612896,
+            0.010344752110540867,
+            0.0020116702653467655,
+            -0.006071025040000677,
+            0.03421725705265999,
+            0.059544067829847336,
+            -0.0027542561292648315,
+            0.0035002718213945627,
+            0.04563215747475624,
+            -0.02034650929272175,
+            -0.027028067037463188,
+            -0.01006349641829729,
+            -0.014460701495409012,
+            0.012011712417006493,
+            0.011723595671355724,
+            -0.005168946459889412,
+            0.01336311548948288,
+            -4.842135604121722e-05,
+            0.030704982578754425,
+            -0.01819249615073204,
+            0.03888200223445892,
+            0.010852386243641376,
+            0.011312000453472137,
+            0.015160413458943367,
+            0.020030952990055084,
+            0.008540594950318336,
+            -0.0007533044554293156,
+            0.006180783733725548,
+            -0.014337223023176193,
+            0.006321412045508623,
+            -0.012800602242350578,
+            -0.012526205740869045,
+            0.014076546765863895,
+            0.0013908478431403637,
+            0.02797473594546318,
+            -0.007463587913662195,
+            -0.005103777162730694,
+            -0.014090266078710556,
+            -0.00472648162394762,
+            0.0034351027570664883,
+            0.012251809239387512,
+            -0.00010434574505779892,
+            -0.007648805622011423,
+            -0.025971639901399612,
+            0.00926088634878397,
+            -3.971247861045413e-05,
+            -0.014117705635726452,
+            0.0012853766093030572,
+            -0.0011653281981125474,
+            -0.014625339768826962,
+            0.026355795562267303,
+            -0.004942568950355053,
+            0.0037386538460850716,
+            0.011517798528075218,
+            0.0009732504840940237,
+            -0.0034848370123654604,
+            0.0023358012549579144,
+            0.011346300132572651,
+            -0.00023559521650895476,
+            -0.015640607103705406,
+            -0.004654452670365572,
+            -0.009226586669683456,
+            0.005529091693460941,
+            -0.006506629753857851,
+            0.009302045218646526,
+            0.003512276802212,
+            0.006307692267000675,
+            -0.01509181410074234,
+            -0.016573555767536163,
+            -0.012128330767154694,
+            -0.006784456316381693,
+            0.011620696634054184,
+            0.0010504245292395353,
+            -0.002776550827547908,
+            -0.004356046207249165,
+            -0.0005886664730496705,
+            -0.024256661534309387,
+            -0.007573346607387066,
+            0.003195005701854825,
+            -0.004441795405000448,
+            0.00019036265439353883,
+            0.003772953525185585,
+            -0.020264189690351486,
+            0.033064793795347214,
+            0.014090266078710556,
+            -0.008965909481048584,
+            0.006787886377424002,
+            -0.003872422268614173,
+            -0.00663010822609067,
+            -0.028619566932320595,
+            -0.0017681432655081153,
+            -0.01870013028383255,
+            0.019976073876023293,
+            0.025793282315135002,
+            -0.0227886401116848,
+            0.0012124900240451097,
+            -0.009013929404318333,
+            0.011456059291958809,
+            -0.0024747145362198353,
+            0.0008373384480364621,
+            0.028921402990818024,
+            -0.01679307222366333,
+            -0.019303802400827408,
+            -0.02350207045674324,
+            0.003423097776249051,
+            0.004486384801566601,
+            0.024887774139642715,
+            -0.0014148575719445944,
+            0.00871209241449833,
+            -0.011360020376741886,
+            -0.0012416447279974818,
+            0.016806792467832565,
+            0.01867268979549408,
+            0.021608734503388405,
+            -0.009254026226699352,
+            -0.0022003178019076586,
+            -0.011037603951990604,
+            -0.0013711255742236972,
+            0.026739951223134995,
+            0.003776383586227894,
+            -0.02540912665426731,
+            -0.008808131329715252,
+            -0.007051993161439896,
+            -0.005374743603169918,
+            0.004510394297540188,
+            -0.0026324924547225237,
+            0.002447274746373296,
+            0.001353118335828185,
+            -0.012210649438202381,
+            0.020442547276616096,
+            0.0015186137752607465,
+            0.0048396703787148,
+            -0.010728907771408558,
+            -0.013774710707366467,
+            -0.026657631620764732,
+            -0.01006349641829729,
+            -0.028647007420659065,
+            -0.003505416912958026,
+            0.0070657129399478436,
+            -0.005841217935085297,
+            0.0009269460570067167,
+            0.013959928415715694,
+            -0.0020408248528838158,
+            -0.005848078057169914,
+            0.0199211947619915,
+            -0.013095579110085964,
+            -0.010310453362762928,
+            -0.008197599090635777,
+            -0.017726020887494087,
+            0.005971556529402733,
+            -0.010296733118593693,
+            0.0003800822014454752,
+            -0.005144936498254538,
+            -0.0018264524405822158,
+            -0.04769013077020645,
+            -0.006715857423841953,
+            -0.014995775185525417,
+            -0.023364871740341187,
+            -0.006691847462207079,
+            0.018357133492827415,
+            0.007038273382931948,
+            -0.018206216394901276,
+            -0.008732672780752182,
+            -0.004747061524540186,
+            8.54139852890512e-06,
+            -0.03391542285680771,
+            0.023543229326605797,
+            0.0018213075818493962,
+            0.011922532692551613,
+            -0.012368427589535713,
+            0.012738863006234169,
+            0.007100012619048357,
+            0.020716944709420204,
+            -0.007326389662921429,
+            -0.01730070635676384,
+            -0.01920776255428791,
+            -0.0009861128637567163,
+            -0.010255573317408562,
+            -0.024325260892510414,
+            -0.01054369006305933,
+            0.0012836616951972246,
+            0.004342326428741217,
+            0.006846195552498102,
+            0.010818086564540863,
+            0.02558748424053192,
+            -0.012491906061768532,
+            0.002020245185121894,
+            0.019797716289758682,
+            0.002472999505698681,
+            0.02210264839231968,
+            0.01890592649579048,
+            0.007278370205312967,
+            -0.001317103742621839,
+            0.002605052897706628,
+            -0.0033390638418495655,
+            -0.01583268493413925,
+            0.007292089983820915,
+            -0.00865035317838192,
+            -0.03320199251174927,
+            0.025107290595769882,
+            0.027069225907325745,
+            0.010756347328424454,
+            -0.006897645071148872,
+            0.004397206008434296,
+            0.0056354207918047905,
+            0.008190738968551159,
+            0.02952507697045803,
+            -0.017383025959134102,
+            0.00253473874181509,
+            0.00788890291005373,
+            -0.011017024517059326,
+            -0.008321077562868595,
+            0.018933366984128952,
+            0.033064793795347214,
+            -0.005837787874042988,
+            -0.039924707263708115,
+            -0.00871209241449833,
+            0.016271719709038734,
+            -0.01406282652169466,
+            -0.015105534344911575,
+            -0.007552767172455788,
+            0.004613293334841728,
+            -0.020113272592425346,
+            0.0019619357772171497,
+            -0.003622035263106227,
+            -0.0028434349223971367,
+            0.008808131329715252,
+            0.028537249192595482,
+            0.003776383586227894,
+            0.02652043290436268,
+            -0.011805914342403412,
+            0.008019241504371166,
+            -0.026561593636870384,
+            0.011517798528075218,
+            -0.02073066495358944,
+            -0.021375495940446854,
+            0.006091604940593243,
+            0.0019173463806509972,
+            0.02807077392935753,
+            0.010516250506043434,
+            0.01923520304262638,
+            0.0023975404910743237,
+            0.00319843553006649,
+            -0.03358614444732666,
+            -0.022061487659811974,
+            -0.0015906429616734385,
+            -0.0195919182151556,
+            0.006890784949064255,
+            -0.006757016759365797,
+            0.014446982182562351,
+            0.016889112070202827,
+            0.015215292572975159,
+            -0.008760112337768078,
+            0.019276361912488937,
+            -2.2576088667847216e-05,
+            0.01712234877049923,
+            -0.02416062168776989,
+            0.00414681900292635,
+            0.02147153578698635,
+            -0.0009320910321548581,
+            0.017547663301229477,
+            0.0003359214751981199,
+            -0.0007018551114015281,
+            0.0069868238642811775,
+            -0.0046201529912650585,
+            -0.012903501279652119,
+            -0.006444890517741442,
+            0.023008156567811966,
+            -0.03163792937994003,
+            0.0110650435090065,
+            0.009020788595080376,
+            -0.006695277523249388,
+            -0.014954615384340286,
+            -0.007271510548889637,
+            0.03737282007932663,
+            0.004040490370243788,
+            -0.00011104487930424511,
+            0.014433261938393116,
+            0.019221482798457146,
+            0.007984941825270653,
+            0.012382147833704948,
+            -0.017081189900636673,
+            0.020854143425822258,
+            0.04041862115263939,
+            -0.007319530006498098,
+            -0.0038689924404025078,
+            0.007669385522603989,
+            0.008821851573884487,
+            -0.01037905178964138,
+            0.008444556035101414,
+            -0.004006190691143274,
+            0.002958338474854827,
+            0.009226586669683456,
+            -0.022212406620383263,
+            0.026438115164637566,
+            -0.022870957851409912,
+            -0.0019756555557250977,
+            -0.004510394297540188,
+            1.4858684153296053e-05,
+            0.004795080982148647,
+            -3.577338793547824e-05,
+            0.02022303082048893,
+            -0.003222445258870721,
+            -0.02197916992008686,
+            -0.006592378951609135,
+            -0.00663010822609067,
+            0.00737440912052989,
+            -0.02019559033215046,
+            0.0004523256793618202,
+            0.024352699518203735,
+            0.005223825573921204,
+            0.0037420839071273804,
+            -0.014001087285578251,
+            -0.00024116890563163906,
+            -0.0035534361377358437,
+            0.021348057314753532,
+            0.01676563359797001,
+            0.024942653253674507,
+            0.02639695443212986,
+            0.003330488922074437,
+            0.012416446581482887,
+            -0.007223491091281176,
+            0.003690634621307254,
+            -0.0243389792740345,
+            0.025985360145568848,
+            -0.011538377963006496,
+            -0.010468230582773685,
+            -0.003851842600852251,
+            -0.010056636296212673,
+            -0.010310453362762928,
+            0.004647592548280954,
+            0.01394620817154646,
+            0.010276153683662415,
+            0.026177437976002693,
+            -0.023762747645378113,
+            0.013342536054551601,
+            0.012135190889239311,
+            -0.018658969551324844,
+            0.025491446256637573,
+            -0.012423306703567505,
+            -0.006366001442074776,
+            -0.0016335173277184367,
+            -0.005961266346275806,
+            0.010811226442456245,
+            -0.021787092089653015,
+            0.009061948396265507,
+            -0.026136277243494987,
+            -0.006064165383577347,
+            -0.003941021393984556,
+            0.0014946040464565158,
+            -0.0014380097854882479,
+            -0.0005633705877698958,
+            -0.028592128306627274,
+            0.0006589806289412081,
+            0.008197599090635777,
+            5.399503788794391e-05,
+            0.014995775185525417,
+            -0.011085623875260353,
+            -0.008423976600170135,
+            -0.0052135358564555645,
+            -0.006201363634318113,
+            -0.008327937684953213,
+            0.03229648247361183,
+            0.003476262092590332,
+            -0.01122968178242445,
+            0.003707784228026867,
+            0.0034248128067702055,
+            -0.030979378148913383,
+            0.006472330074757338,
+            0.008259338326752186,
+            -0.017904380336403847,
+            0.020387668162584305,
+            0.002598193008452654,
+            0.015997322276234627,
+            -0.01813761703670025,
+            0.007957502268254757,
+            -0.017561383545398712,
+            -0.012052871286869049,
+            8.306928975798655e-07,
+            -0.019290082156658173,
+            0.012464466504752636,
+            -0.02352951094508171,
+            0.022130087018013,
+            -0.02070322446525097,
+            -0.003759233746677637,
+            -0.02344719134271145,
+            0.02055230736732483,
+            -0.0018779018428176641,
+            0.007504747714847326,
+            0.013500313274562359,
+            0.0017972978530451655,
+            0.004441795405000448,
+            0.016600996255874634,
+            0.0035431464202702045,
+            0.014515581540763378,
+            0.01798669807612896,
+            0.02233588509261608,
+            0.006811895873397589,
+            0.0033184841740876436,
+            -0.020319068804383278,
+            0.004229137673974037,
+            -0.007346969563513994,
+            0.05205303803086281,
+            0.005710879806429148,
+            -0.007984941825270653,
+            0.00466817244887352,
+            0.0035362865310162306,
+            0.013445434160530567,
+            0.0038141130935400724,
+            0.02129317820072174,
+            -0.016985150054097176,
+            0.014337223023176193,
+            -0.020826702937483788,
+            0.00656150933355093,
+            0.017643703147768974,
+            0.03193976730108261,
+            0.009391224011778831,
+            -0.012505626305937767,
+            -0.020140711218118668,
+            0.011353160254657269,
+            -0.008273057639598846,
+            0.00014716661826241761,
+            -0.018562931567430496,
+            0.007244070526212454,
+            -0.007744844537228346,
+            0.011593257077038288,
+            0.002666792133823037,
+            0.02317279390990734,
+            0.012279248796403408,
+            -0.013212197460234165,
+            0.00680160615593195,
+            -0.009507843293249607,
+            -0.010008616372942924,
+            0.006297402549535036,
+            -0.03462885320186615,
+            -0.02225356549024582,
+            0.009809679351747036,
+            0.030430585145950317,
+            -0.0020459697116166353,
+            -0.03144585341215134,
+            -0.0038758523296564817,
+            -0.015242732129991055,
+            -0.025971639901399612,
+            0.017780901864171028,
+            0.02952507697045803,
+            -0.011154222302138805,
+            0.0038278328720480204,
+            -0.00014041388931218535,
+            0.009603881277143955,
+            0.008135859854519367,
+            0.021348057314753532,
+            -0.008177019655704498,
+            -0.009624461643397808,
+            0.007244070526212454,
+            0.024242941290140152,
+            -0.00978223979473114,
+            0.022006608545780182,
+            -0.014103986322879791,
+            0.009597022086381912,
+            -0.018302254378795624,
+            -0.00991943757981062,
+            -0.01559944823384285,
+            -6.082815525587648e-05,
+            -0.0032567449379712343,
+            0.003855272429063916,
+            -0.00950098317116499,
+            -0.0010512820445001125,
+            0.01026243343949318,
+            0.003989040851593018,
+            -0.004352616146206856,
+            0.00851315539330244,
+            0.005234115291386843,
+            0.01666959375143051,
+            0.003155561164021492,
+            -0.0362752340734005,
+            0.005402183625847101,
+            0.004050780087709427,
+            -0.0024284101091325283,
+            0.01572292670607567,
+            0.02705550752580166,
+            0.00799180194735527,
+            0.014282343909144402,
+            0.015311331488192081,
+            0.018000418320298195,
+            0.014296064153313637,
+            -0.010420211590826511,
+            0.0020939891692250967,
+            0.016203120350837708,
+            -0.0017338435864076018,
+            -0.010660308413207531,
+            -0.004808800760656595,
+            -0.024229221045970917,
+            0.030320826917886734,
+            0.013678671792149544,
+            -0.008506295271217823,
+            -0.027727779000997543,
+            -0.00933634489774704,
+            0.020620906725525856,
+            0.002779980655759573,
+            -0.011462918482720852,
+            -0.006952524185180664,
+            -0.007216630969196558,
+            0.018947087228298187,
+            0.0019070565467700362,
+            0.0015417659888044,
+            -0.005203245673328638,
+            -0.007957502268254757,
+            0.0195919182151556,
+            -0.018425732851028442,
+            -0.006050445605069399,
+            -0.0074224285781383514,
+            0.00836909655481577,
+            0.00043989208643324673,
+            0.03904663771390915,
+            0.009425523690879345,
+            0.030650101602077484,
+            0.0009218011400662363,
+            -0.018466891720891,
+            0.007477307692170143,
+            0.008115280419588089,
+            0.00893160980194807,
+            0.0305403433740139,
+            -0.017904380336403847,
+            0.003666624892503023,
+            0.013534612953662872,
+            0.0007691680220887065,
+            0.0005582256126217544,
+            0.0005740891792811453,
+            0.004640732891857624,
+            -0.004657882731407881,
+            -0.015105534344911575,
+            -0.0008634918485768139,
+            -0.01780834048986435,
+            0.016011042520403862,
+            0.012491906061768532,
+            0.006760446820408106,
+            -0.01129142101854086,
+            0.003366503631696105,
+            0.0092334458604455,
+            -0.015544568188488483,
+            -0.00865035317838192,
+            0.007854603230953217,
+            -0.01216948963701725,
+            0.017437905073165894,
+            -0.00021179989562369883,
+            -0.018000418320298195,
+            -0.002262057038024068,
+            0.007635085843503475,
+            0.005974986590445042,
+            0.008087839931249619,
+            -0.02281607873737812,
+            -0.016600996255874634,
+            -0.0032207302283495665,
+            -0.02385878562927246,
+            -0.02764545939862728,
+            0.0025827581994235516,
+            0.009535282850265503,
+            -0.007518467493355274,
+            -0.00851315539330244,
+            0.01920776255428791,
+            0.011888233944773674,
+            0.023913664743304253,
+            -0.024640817195177078,
+            0.019468439742922783,
+            -0.00032477412605658174,
+            -0.016189400106668472,
+            0.004585853312164545,
+            -0.025285648182034492,
+            0.02436641976237297,
+            0.0011018739314749837,
+            0.00720291119068861,
+            -0.0031212614849209785,
+            -0.006482620257884264,
+            0.002944618696346879,
+            -0.007120592053979635,
+            0.009974316693842411,
+            0.0017921528778970242,
+            -0.020634625107049942,
+            -0.0010975864715874195,
+            -0.007285230327397585,
+            -0.009315765462815762,
+            0.011819634586572647,
+            -0.027631739154458046,
+            -0.0007005688967183232,
+            -0.00535416416823864,
+            -0.0031932906713336706,
+            -0.003143556183204055,
+            0.017136069014668465,
+            -0.022555401548743248,
+            0.035863637924194336,
+            0.008225038647651672,
+            0.012629104778170586,
+            -0.00316928094252944,
+            0.01607964187860489,
+            0.008938469924032688,
+            -0.0029497635550796986,
+            -0.0013676956295967102,
+            -0.007470448035746813,
+            -0.0061670639552176,
+            0.011545238085091114,
+            0.0025467437226325274,
+            -0.012457606382668018,
+            0.009912577457726002,
+            -0.0038038231432437897,
+            -0.02078554406762123,
+            -0.008773831650614738,
+            0.011799054220318794,
+            0.010975864715874195,
+            -0.002409545239061117,
+            0.005546241533011198,
+            0.0067364368587732315,
+            0.010111515410244465,
+            -0.004966578911989927,
+            -0.001727841212414205,
+            0.004486384801566601,
+            -0.017396746203303337,
+            -0.006407161243259907,
+            -0.014968335628509521,
+            -0.009631321765482426,
+            0.020799264311790466,
+            0.01849433220922947,
+            0.007497887592762709,
+            5.8094909036299214e-05,
+            -0.0019001965411007404,
+            -0.009116827510297298,
+            -0.0002758972404990345,
+            0.011826494708657265,
+            -0.02848236821591854,
+            0.019825154915452003,
+            0.01789066009223461,
+            0.022870957851409912,
+            -0.028180532157421112,
+            -0.007429288700222969,
+            0.0007117162458598614,
+            -0.007003973703831434,
+            -0.017314426600933075,
+            -0.0033973732497543097,
+            -0.008849291130900383,
+            0.014501861296594143,
+            -0.02004467323422432,
+            0.01509181410074234,
+            0.0022894968278706074,
+            0.0016755344113335013,
+            -0.0011944827856495976,
+            0.010838666930794716,
+            0.007175471633672714,
+            -0.0037180741783231497,
+            -0.002824570285156369,
+            -0.018274815753102303,
+            -0.007971221581101418,
+            0.02182825095951557,
+            0.002627347595989704,
+            -0.023433471098542213,
+            0.002495294203981757,
+            0.004558413755148649,
+            -0.025875601917505264,
+            0.006925084628164768,
+            -0.037564896047115326,
+            -0.009432383812963963,
+            -0.015407370403409004,
+            0.0019790856167674065,
+            -0.02009955234825611,
+            0.006221943534910679,
+            0.001680679270066321,
+            0.010132094845175743,
+            0.003201865591108799,
+            -0.007971221581101418,
+            -0.00945296324789524,
+            0.002495294203981757,
+            -0.004530974198132753,
+            0.0019756555557250977,
+            0.01044079102575779,
+            0.005916676949709654,
+            -0.02842748910188675,
+            -0.020538587123155594,
+            0.005388463847339153,
+            -0.014639060012996197,
+            -0.0007507320260629058,
+            -0.005446773022413254,
+            0.00019089858687948436,
+            -0.00044417951721698046,
+            -0.004589283373206854,
+            0.007587066385895014,
+            0.014625339768826962,
+            -0.00019818724831566215,
+            0.009754800237715244,
+            0.005326724145561457,
+            -0.0005595118855126202,
+            0.020497428253293037,
+            -9.680251423560549e-06,
+            0.0218968503177166,
+            0.024270381778478622,
+            -0.013342536054551601,
+            0.02314535528421402,
+            -0.009933157823979855,
+            0.013699251227080822,
+            -0.008698373101651669,
+            0.017040029168128967,
+            -0.0031006818171590567,
+            0.012121470645070076,
+            -0.006451750639826059,
+            -0.011757895350456238,
+            -0.006077885162085295,
+            0.009686200879514217,
+            -0.0006636968464590609,
+            -0.010193834081292152,
+            0.019962353631854057,
+            -0.014789978042244911,
+            0.011558957397937775,
+            -0.0038827122189104557,
+            0.001445727190002799,
+            0.0020442549139261246,
+            -0.0004028056573588401,
+            0.03125377371907234,
+            0.00919914711266756,
+            -0.00639001140370965,
+            -0.017822060734033585,
+            0.006846195552498102,
+            -0.029387878254055977,
+            -0.011737314984202385,
+            -0.004115949384868145,
+            -0.01474881824105978,
+            -0.0022963567171245813,
+            -0.0019979504868388176,
+            0.031363531947135925,
+            -0.005192955955862999,
+            -0.020675785839557648,
+            -0.005196386016905308,
+            -0.01664215512573719,
+            -0.00736754946410656,
+            -0.00799180194735527,
+            -0.0062528131529688835,
+            0.003152131102979183,
+            -8.617768617114052e-05,
+            -0.023378591984510422,
+            0.009116827510297298,
+            -0.0047127618454396725,
+            0.017437905073165894,
+            -0.001723553752526641,
+            0.0224593635648489,
+            -0.020154431462287903,
+            0.011888233944773674,
+            -0.009610741399228573,
+            0.002551888581365347,
+            0.018439453095197678,
+            0.0027748357970267534,
+            -0.0068153259344398975,
+            0.0040953694842755795,
+            0.015695486217737198,
+            -0.015517128631472588,
+            -0.0025467437226325274,
+            -0.011716735549271107,
+            0.02121085859835148,
+            3.5398235922912136e-05,
+            -0.007779144216328859,
+            -0.015050654299557209,
+            -0.02359810844063759,
+            0.016065921634435654,
+            -0.006321412045508623,
+            0.014200025238096714,
+            0.01181277446448803,
+            0.02362554892897606,
+            0.0022243275307118893,
+            -8.730314584681764e-05,
+            0.009103108197450638,
+            -0.007758564315736294,
+            0.008863010443747044,
+            0.021183418110013008,
+            0.0035362865310162306,
+            -0.005745179019868374,
+            0.006293972488492727,
+            -0.0014320074114948511,
+            0.02261028066277504,
+            -0.018562931567430496,
+            0.001973940758034587,
+            0.014762538485229015,
+            0.0020511148031800985,
+            -0.010639728978276253,
+            0.015942443162202835,
+            -0.0013462584465742111,
+            -0.0037180741783231497,
+            -0.01750650443136692,
+            0.0031349812634289265,
+            -0.00891789048910141,
+            0.009144267067313194,
+            -0.01352775376290083,
+            0.010015476495027542,
+            -0.02258284203708172,
+            0.007401848677545786,
+            0.00016388767107855529,
+            -0.002584473229944706,
+            0.02621859684586525,
+            -0.006465470418334007,
+            0.0009226586553268135,
+            0.030403144657611847,
+            -0.0035843057557940483,
+            -0.01966051757335663,
+            0.00133854104205966,
+            -0.0032790396362543106,
+            -0.004242857918143272,
+            0.00691136484965682,
+            0.0015572007978335023,
+            -0.016573555767536163,
+            -0.005491362418979406,
+            0.014213744550943375,
+            0.02565608359873295,
+            0.0015229012351483107,
+            -0.022377043962478638,
+            0.002049399772658944,
+            0.01572292670607567,
+            -0.0029617685358971357,
+            -0.0002458851085975766,
+            0.005820638500154018,
+            0.009789098985493183,
+            0.0006461183074861765,
+            -0.006595808546990156,
+            0.010866106487810612,
+            -0.02797473594546318,
+            -0.008554314263164997,
+            -0.01640891842544079,
+            -0.008760112337768078,
+            -0.04157108813524246,
+            0.024435019120573997,
+            0.00459957355633378,
+            0.003469402203336358,
+            0.008965909481048584,
+            0.005529091693460941,
+            -0.004009620752185583,
+            -0.018933366984128952,
+            -0.002056259661912918,
+            -0.006376291625201702,
+            0.01130514033138752,
+            -0.008828711695969105,
+            -0.007957502268254757,
+            0.005559961311519146,
+            0.0029754883144050837,
+            -0.005275275092571974,
+            0.024092022329568863,
+            -0.001313673797994852,
+            0.0031075417064130306,
+            0.0038106830324977636,
+            -0.004157108720391989,
+            -0.0013128162827342749,
+            0.015640607103705406,
+            0.014090266078710556,
+            -4.6465207560686395e-05,
+            0.009672480635344982,
+            0.008259338326752186,
+            0.001888191793113947,
+            -0.006403731182217598,
+            -0.009391224011778831,
+            0.007024553604424,
+            0.018206216394901276,
+            -0.0035465762484818697,
+            -0.004294306971132755,
+            0.019605638459324837,
+            -0.014529300853610039,
+            -0.0008008951554074883,
+            0.012642824091017246,
+            -0.015201572328805923,
+            0.017904380336403847,
+            0.008972769603133202,
+            0.011538377963006496,
+            -0.010509390383958817,
+            -0.04598887264728546,
+            0.004139958880841732,
+            -0.00737440912052989,
+            0.009727359749376774,
+            0.0072372108697891235,
+            0.02248680219054222,
+            -0.013486593961715698,
+            -0.03534228354692459,
+            0.004805370699614286,
+            0.00216258829459548,
+            0.020442547276616096,
+            0.008122139610350132,
+            0.018809888511896133,
+            0.008629773743450642,
+            0.01559944823384285,
+            -0.006674697622656822,
+            0.009638180956244469,
+            0.022857237607240677,
+            -0.011997992172837257,
+            -0.014213744550943375,
+            -0.004242857918143272,
+            0.0022792068775743246,
+            -0.005628560669720173,
+            -0.009768519550561905,
+            0.001305956393480301,
+            -0.013486593961715698,
+            0.0048465305007994175,
+            0.01422746479511261,
+            0.015078093856573105,
+            0.010502530261874199,
+            0.006492909975349903,
+            0.0033390638418495655,
+            -0.003954741172492504,
+            0.0006765591679140925,
+            0.006506629753857851,
+            0.018535491079092026,
+            -0.013877608813345432,
+            0.026163717731833458,
+            -0.0008446270949207246,
+            0.013074998743832111,
+            0.005621700547635555,
+            0.003694064449518919,
+            0.005162086337804794,
+            0.008698373101651669,
+            0.0032258753199130297,
+            0.013555193319916725,
+            -0.004122809041291475,
+            0.0014294348657131195,
+            0.0016095077153295279,
+            -0.008780691772699356,
+            0.007882042787969112,
+            -0.020113272592425346,
+            0.011757895350456238,
+            0.009528422728180885,
+            0.00034042331390082836,
+            0.010248714126646519,
+            0.018755009397864342,
+            0.015352491289377213,
+            -0.031390972435474396,
+            -0.0072372108697891235,
+            -0.007504747714847326,
+            -0.002130003646016121,
+            0.01676563359797001,
+            0.002109423978254199,
+            0.014076546765863895,
+            0.0424765981733799,
+            0.006129334680736065,
+            0.02335115149617195,
+            0.0017852929886430502,
+            -0.019317522644996643,
+            -0.00250558415427804,
+            0.0039821807295084,
+            -0.02147153578698635,
+            0.0016832518158480525,
+            -0.002937758807092905,
+            -0.01598360389471054,
+            -0.0028897393494844437,
+            -0.0028039903845638037,
+            0.0029960679821670055,
+            -0.00598870636895299,
+            -0.002406115410849452,
+            0.0039238715544342995,
+            0.0035431464202702045,
+            0.022088928148150444,
+            -0.019098004326224327,
+            0.0013333960669115186,
+            -0.006125904619693756,
+            0.006136194337159395,
+            -0.015517128631472588,
+            -0.010015476495027542,
+            -0.013857029378414154,
+            0.02944275736808777,
+            -0.021718492731451988,
+            -0.009549002163112164,
+            0.013006399385631084,
+            -0.012423306703567505,
+            -0.006582088768482208,
+            0.0013899903278797865,
+            -0.002208892721682787,
+            -0.026479274034500122,
+            -0.0006332559278234839,
+            -0.005144936498254538,
+            -0.015325050801038742,
+            -0.01343857403844595,
+            -0.04494616761803627,
+            0.021691052243113518,
+            0.004105659201741219,
+            -0.007854603230953217,
+            -0.0023495210334658623,
+            -0.020058393478393555,
+            0.014501861296594143,
+            0.0007142886752262712,
+            0.01920776255428791,
+            -0.011490358039736748,
+            -0.014350943267345428,
+            -0.004047350026667118,
+            0.009946877136826515,
+            -0.020291630178689957,
+            0.02034650929272175,
+            0.002613627817481756,
+            -0.024860333651304245,
+            -0.012608524411916733,
+            0.0040953694842755795,
+            -0.0020065251737833023,
+            -0.008389676921069622,
+            0.0038758523296564817,
+            -0.0040953694842755795,
+            0.00276283104903996,
+            -0.0058446479961276054,
+            0.014268624596297741,
+            -0.02293955720961094,
+            0.021608734503388405,
+            0.0013436859007924795,
+            0.0006739866803400218,
+            0.014529300853610039,
+            0.014872296713292599,
+            0.016203120350837708,
+            -0.014680218882858753,
+            -0.007593926507979631,
+            0.02009955234825611,
+            0.035808760672807693,
+            -0.009686200879514217,
+            0.007223491091281176,
+            0.011785334907472134,
+            0.02591676078736782,
+            0.011504078283905983,
+            -0.002399255521595478,
+            -0.015791526064276695,
+            0.014803697355091572,
+            0.004664742387831211,
+            0.004551554098725319,
+            -0.008540594950318336,
+            -0.031006818637251854,
+            0.0025930481497198343,
+            -0.008238758891820908,
+            -0.023419750854372978,
+            -0.0031281213741749525,
+            0.011023884639143944,
+            0.021787092089653015,
+            0.008780691772699356,
+            -0.004867109935730696,
+            0.01440582238137722,
+            -0.01849433220922947,
+            0.00805354118347168,
+            0.010392772033810616,
+            -0.011010164394974709,
+            0.014042247086763382,
+            -0.02675366960465908,
+            -0.042696114629507065,
+            0.0023718157317489386,
+            0.006684987805783749,
+            0.01257422473281622,
+            0.005717739462852478,
+            0.0070794327184557915,
+            -0.006441460456699133,
+            -0.008375956676900387,
+            -0.006136194337159395,
+            0.0023718157317489386,
+            -0.009082527831196785,
+            0.018384573981165886,
+            0.005436482839286327,
+            -0.008074120618402958,
+            0.0014217174611985683,
+            0.017945539206266403,
+            -0.007648805622011423,
+            0.009247166104614735,
+            0.008856151252985,
+            0.015393650159239769,
+            0.017726020887494087,
+            -0.008595474064350128,
+            0.038580164313316345,
+            0.010934704914689064,
+            -0.014543021097779274,
+            0.00045275440788827837,
+            -0.0044486550614237785,
+            0.030622662976384163,
+            -0.0003732222830876708,
+            -0.01583268493413925,
+            0.023927384987473488,
+            0.002188313053920865,
+            -0.00013923484948463738,
+            -0.014721378684043884,
+            0.0032498850487172604,
+            -0.006475760135799646,
+            0.01643635705113411,
+            -0.009192286990582943,
+            0.001149893389083445,
+            -0.020291630178689957,
+            -0.005611410830169916,
+            0.0004626155423466116,
+            -0.008897310122847557,
+            0.008677792735397816,
+            -0.0017750031547620893,
+            0.0038998620584607124,
+            -0.013459154404699802,
+            -0.018892208114266396,
+            0.0027079517021775246,
+            0.008938469924032688,
+            0.02012699283659458,
+            0.015105534344911575,
+            0.0004733341629616916,
+            -0.011881373822689056,
+            -0.00945296324789524,
+            -0.0035431464202702045,
+            0.005179236177355051,
+            0.0023049316368997097,
+            -0.01941356062889099,
+            -0.012004852294921875,
+            0.010132094845175743,
+            -0.0005805203691124916,
+            -0.003659765003249049,
+            0.00720291119068861,
+            -0.007209771312773228,
+            -0.00598870636895299,
+            0.009926297701895237,
+            -0.0018161626067012548,
+            0.0035105617716908455,
+            -0.006969674024730921,
+            -0.014268624596297741,
+            -0.00933634489774704,
+            0.003944451455026865,
+            -0.01664215512573719,
+            0.02797473594546318,
+            -0.00991943757981062,
+            0.0017518509412184358,
+            0.008876730687916279,
+            0.013171037659049034,
+            0.0017295562429353595,
+            0.02263772115111351,
+            -0.02446245774626732,
+            -0.006966243963688612,
+            0.0029926381539553404,
+            -0.022047769278287888,
+            -0.01780834048986435,
+            -0.017918098717927933,
+            -0.04151620715856552,
+            0.012656544335186481,
+            0.026561593636870384,
+            -0.01646379753947258,
+            0.02979947254061699,
+            -0.0002896170481108129,
+            0.005837787874042988,
+            0.011222821660339832,
+            -0.008314217440783978,
+            -0.003522566519677639,
+            -0.017753461375832558,
+            -0.014014807529747486,
+            0.02004467323422432,
+            -0.005186096299439669,
+            0.005148366559296846,
+            -0.01124340109527111,
+            -0.009747940115630627,
+            -0.0050660474225878716,
+            -0.0002876876969821751,
+            0.01628543995320797,
+            0.015421089716255665,
+            -0.005734889302402735,
+            -0.012313548475503922,
+            -0.020154431462287903,
+            0.025189610198140144,
+            -0.0020854142494499683,
+            -0.009837118908762932,
+            0.009432383812963963,
+            0.02317279390990734,
+            -0.013925628736615181,
+            -0.00645861029624939,
+            0.003143556183204055,
+            0.005676580127328634,
+            -0.0029531936161220074,
+            0.018933366984128952,
+            -0.011483498848974705,
+            0.0018813317874446511,
+            -0.010420211590826511,
+            0.016710754483938217,
+            0.02042882889509201,
+            0.021046221256256104,
+            -0.019756557419896126,
+            0.014501861296594143,
+            -0.01157267764210701,
+            -0.0038244028110057116,
+            0.014076546765863895,
+            0.014789978042244911,
+            0.002035679994150996,
+            0.007346969563513994,
+            0.025779562070965767,
+            -0.007024553604424,
+            0.02090902253985405,
+            -0.011113063432276249,
+            -0.015338771045207977,
+            -0.014433261938393116,
+            -0.02573840320110321,
+            0.003189860610291362,
+            -0.006451750639826059,
+            -0.018151337280869484,
+            -0.007298950105905533,
+            -0.006969674024730921,
+            0.004476095084100962,
+            -0.004801940638571978,
+            -0.02352951094508171,
+            0.01223122887313366,
+            -0.0202504713088274,
+            0.02276119962334633,
+            -0.023282554000616074,
+            -0.0097959591075778,
+            -0.018069017678499222,
+            -0.011133642867207527,
+            -0.008327937684953213,
+            -0.008952190168201923,
+            0.0079643614590168,
+            -0.006678127683699131,
+            0.005656000226736069,
+            -0.005247835535556078,
+            0.0027045216411352158,
+            -0.005467352457344532,
+            0.009363784454762936,
+            0.019701676443219185,
+            0.02162245288491249,
+            0.00472648162394762,
+            0.016065921634435654,
+            0.015489689074456692,
+            0.006623248569667339,
+            -0.016546115279197693,
+            0.006753586698323488,
+            -0.0075802067294716835,
+            0.008115280419588089,
+            0.020085832104086876,
+            -0.010735767893493176,
+            -0.00633856188505888,
+            -0.009686200879514217,
+            -0.0017870080191642046,
+            0.02398226410150528,
+            -0.004692182410508394,
+            -0.002123143756762147,
+            -0.001323963631875813,
+            0.016450077295303345,
+            0.0029549086466431618,
+            0.020058393478393555,
+            0.0019893755670636892,
+            0.01509181410074234,
+            -0.007868323475122452,
+            -0.019221482798457146,
+            -0.014872296713292599,
+            -0.02070322446525097,
+            0.010084075853228569,
+            -0.00909624807536602,
+            -0.01804157719016075,
+            -0.014995775185525417,
+            0.001756138401106,
+            -0.005930396728217602,
+            7.590121185785392e-06,
+            0.005515371914952993,
+            0.007477307692170143,
+            0.00945296324789524,
+            0.014172585681080818,
+            0.001274229260161519,
+            -0.0008952189818955958,
+            0.023556949570775032,
+            -0.015338771045207977,
+            0.00530271464958787,
+            -0.010310453362762928,
+            -0.01129142101854086,
+            -0.008266198448836803,
+            0.0006778453825972974,
+            0.006849625613540411,
+            -0.028537249192595482,
+            0.0061739240773022175,
+            -0.02807077392935753,
+            0.004194838460534811,
+            0.0071343122981488705,
+            -0.001848747255280614,
+            -0.007792863994836807,
+            0.0005758041515946388,
+            -0.009267745539546013,
+            0.011517798528075218,
+            0.021252017468214035,
+            0.021814530715346336,
+            -0.002745681209489703,
+            -0.005234115291386843,
+            0.013788430020213127,
+            0.020140711218118668,
+            0.016559835523366928,
+            0.003587735816836357,
+            0.005642280448228121,
+            -0.025491446256637573,
+            -0.0003918726579286158,
+            -0.012361567467451096,
+            -0.018123896792531013,
+            0.01918032392859459,
+            8.628487557871267e-05,
+            0.00598870636895299,
+            -0.008218178525567055,
+            -0.006726147141307592,
+            -0.004853390157222748,
+            0.01424118410795927,
+            0.010872965678572655,
+            -0.02842748910188675,
+            -0.004805370699614286,
+            -0.022171247750520706,
+            -0.004393775947391987,
+            0.023008156567811966,
+            0.0032207302283495665,
+            -0.003690634621307254,
+            -0.0011661855969578028,
+            -0.009864558465778828,
+            0.004386915825307369,
+            0.006071025040000677,
+            0.0006191073916852474,
+            0.04810172691941261,
+            0.008465135470032692,
+            -0.006894215010106564,
+            -0.00863663386553526,
+            0.0007318672141991556,
+            -0.008533734828233719,
+            0.004736771807074547,
+            0.01002233661711216,
+            0.0018967665964737535,
+            -0.013191617093980312,
+            -0.016381477937102318,
+            -0.009706780314445496,
+            -0.009555862285196781,
+            -0.004215417895466089,
+            -0.023515790700912476,
+            -0.00026753669953905046,
+            -0.009830258786678314,
+            0.002063119551166892,
+            -0.005117496941238642,
+            -0.011888233944773674,
+            0.01944100111722946,
+            -0.02619115822017193,
+            0.007271510548889637,
+            0.01768486201763153,
+            0.006726147141307592,
+            0.012560505419969559,
+            -0.005978416185826063,
+            -0.013520893640816212,
+            0.003927301615476608,
+            -0.014021667651832104,
+            -0.010557410307228565,
+            0.002400970319285989,
+            0.0027816956862807274,
+            -0.01966051757335663,
+            -0.01780834048986435,
+            0.002287781797349453,
+            0.011984271928668022,
+            0.00244898977689445,
+            0.005395323503762484,
+            0.012992680072784424,
+            -0.0004488957056310028,
+            0.02037394978106022,
+            0.0005959551781415939,
+            -0.02947019785642624,
+            0.006016145925968885,
+            0.010845526121556759,
+            0.005539381876587868,
+            -0.0004690467321779579,
+            0.010914125479757786,
+            0.008259338326752186,
+            0.0028674446512013674,
+            -0.02398226410150528,
+            0.0020974192302674055,
+            -0.015338771045207977,
+            0.010090935975313187,
+            -0.026355795562267303,
+            0.014995775185525417,
+            -0.02604023925960064,
+            0.01130514033138752,
+            -0.02850980870425701,
+            0.01225866936147213,
+            0.002647927263751626,
+            0.001377985579892993,
+            0.0071823312900960445,
+            -0.011545238085091114,
+            -0.00035114193451590836,
+            -0.015530848875641823,
+            0.0009037938434630632,
+            -0.0009681055671535432,
+            -0.0261225588619709,
+            -0.029223239049315453,
+            -0.0013376835267990828,
+            -0.007264650426805019,
+            0.004393775947391987,
+            0.009864558465778828,
+            -0.012018571607768536,
+            -0.02174593135714531,
+            0.007038273382931948,
+            0.002641067374497652,
+            -0.0014877441572025418,
+            0.026328355073928833,
+            -0.025875601917505264,
+            -0.02096390165388584,
+            0.003472832264378667,
+            0.01938612200319767,
+            0.005100347101688385,
+            0.006105324719101191,
+            -0.0193586815148592,
+            0.01831597462296486,
+            -0.0070657129399478436,
+            -0.02192429080605507,
+            -0.008595474064350128,
+            0.02096390165388584,
+            0.014378382824361324,
+            0.012382147833704948,
+            0.010475090704858303,
+            -0.0009543857304379344,
+            0.002027105074375868,
+            -0.0014980339910835028,
+            0.013109298422932625,
+            0.008924749679863453,
+            -0.004887689836323261,
+            -0.015174132771790028,
+            -0.0014011377934366465,
+            -0.008794412016868591,
+            0.0028640148229897022,
+            0.013630651868879795,
+            0.011346300132572651,
+            -0.017163507640361786,
+            -0.01583268493413925,
+            -0.03158305212855339,
+            0.002167733386158943,
+            -0.01983887515962124,
+            -0.008986488915979862,
+            -0.00034042331390082836,
+            0.006746727041900158,
+            0.008664073422551155,
+            0.008005521260201931,
+            0.015297611244022846,
+            0.01651867665350437,
+            0.019729116931557655,
+            0.027659179642796516,
+            -0.00056208431487903,
+            -0.004603003151714802,
+            0.004050780087709427,
+            0.0025107290130108595,
+            -0.028537249192595482,
+            -0.00927460566163063,
+            -0.012368427589535713,
+            -0.0006302547408267856,
+            0.016093362122774124,
+            0.001071004313416779,
+            0.007895763032138348,
+            -0.015119253657758236,
+            0.013829589821398258,
+            -0.010914125479757786,
+            -0.010584849864244461,
+            -0.02848236821591854,
+            0.0002185526245739311,
+            0.021608734503388405,
+            -0.025450287386775017,
+            0.013280796818435192,
+            -0.018027858808636665,
+            0.02004467323422432,
+            -0.008060400374233723,
+            -0.004736771807074547,
+            0.0014577320544049144,
+            0.005656000226736069,
+            -0.00611561443656683,
+            0.01020069420337677,
+            0.01136687956750393,
+            0.012704563327133656,
+            -0.013095579110085964,
+            0.009507843293249607,
+            -0.0021643033251166344,
+            0.02192429080605507,
+            0.024558497592806816,
+            0.015283891931176186,
+            -0.012340988032519817,
+            -0.017588824033737183,
+            0.00875325221568346,
+            0.008732672780752182,
+            -0.005803488660603762,
+            0.012649684213101864,
+            -0.01631287857890129,
+            0.01003605592995882,
+            0.01457046065479517,
+            0.005076337605714798,
+            0.020840423181653023,
+            -0.005381603725254536,
+            -0.011071903631091118,
+            0.01562688685953617,
+            0.012059731408953667,
+            -0.004263437353074551,
+            -0.011901953257620335,
+            0.009226586669683456,
+            0.011462918482720852,
+            0.008677792735397816,
+            0.016175679862499237,
+            0.009075668640434742,
+            0.0077517046593129635,
+            0.012251809239387512,
+            -0.028345171362161636,
+            0.004887689836323261,
+            -0.0031641360837966204,
+            -0.017396746203303337,
+            0.008396536111831665,
+            -0.01003605592995882,
+            -0.006496340036392212,
+            0.024407578632235527,
+            0.017314426600933075,
+            0.0020391098223626614,
+            -0.010132094845175743,
+            -0.0007327247294597328,
+            -0.00010531042062211782,
+            0.0023838207125663757,
+            0.0028897393494844437,
+            0.011860793456435204,
+            -0.01651867665350437,
+            -0.019495880231261253,
+            0.029031163081526756,
+            -0.007511607371270657,
+            0.006132764276117086,
+            0.001967080868780613
+          ],
+          "keyphrases": [
+            "hyperparameter search",
+            "8B dense model",
+            "constant learning rate",
+            "MoE 8B/64E models",
+            "square root learning rate decay"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "51851d30-50b5-4e5d-a90e-74e63004a7a5",
+      "type": "next",
+      "source": {
+        "id": "8bd19de5-c2d4-458d-833b-5962c425f915",
+        "properties": {
+          "page_content": "1 Introduction\n\nScaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27]. The final model quality has been found to have a power-law relationship with the amount of data, model size, and compute time [16, 20]. However, training efficiency, which is defined as the total amount of computation used to achieve superior model quality than the state of the art system [21], should receive greater attention as we increase our efforts towards green AI [29].\n\nSparsely gated mixture-of-experts [31] (MoE) provides an effective way to scale model capacity given a fixed computational cost, and has recently played an important role in increasing the training efficiency of large-scale language models [10, 21]. MoE operate by adopting a number of experts, each as a sub-network, and by activating only one or a few experts for each input token. A gating network must be chosen and optimized in order to route each token to the most suited expert(s). For example, recent work has implemented sparse routing via k-means clustering [12], linear assignment to maximize token-expert affinities [22], or hashing [8, 28]. Many of the prior work use a routing strategy concerning the token choice, where each token selects the best one or two experts.\n\nWe argue that the independent token choice of prior work often leads to an imbalanced load of experts, which causes training inefficiency and sub-optimal training of the model. In order to mitigate this\n\nissue, previous sparsely gated networks introduce additional auxiliary losses as regularization to prevent too many tokens being routed to a single expert, but the effectiveness is still limited. Recent approaches [8, 22, 28] explore alternative strategies for routing, but they focus on pre-training only and do not demonstrate performance gain on downstream tasks. Moreover, none of the previous methods consider allocating a variable number of experts to each token based on importance, which can be beneficial.\n\nWe propose a very simple yet effective routing method we are calling expert choice. Unlike conventional MoE where tokens select one or two top-scoring experts, our method lets each expert pick the top-k tokens. Our method guarantees perfect load balancing, allows a variable number of experts for each token, and achieves substantial gains in training efficiency and downstream performance as demonstrated in our experiments. Our major contributions include:\n\nWe identify common pitfalls in conventional MoE such as load imbalance as described in Section 3.1. We then propose a heterogeneous, expert choice method to provide a fluid allocation of model parameters based on a learnt token-to-expert importance. This method intrinsically guarantees load balance without imposing an auxiliary loss.\n\nWe show our method provides over 2× faster training convergence in a 8B/64E (8 billion activated parameters, 64 experts) model, compared to the top-1 and top-2 gating counterparts in Switch Transformer [10] and GShard [21].\n\nWe show our method demonstrates strong scaling when increasing the number of experts from 16 to 128, evaluated in training perplexity.\n\nWe show our method demonstrates strong performance on downstream tasks selected from GLUE and SuperGLUE at all the evaluated scales. More specifically, our 8B/64E model outperforms a T5 11B dense model in 7 out of 11 tasks evaluated.\n\n",
+          "title": "Introduction",
+          "embedding": [
+            0.0024422307033091784,
+            -0.014919951558113098,
+            -0.01768755167722702,
+            -0.02871091105043888,
+            0.020494353026151657,
+            0.03838575258851051,
+            -0.01718577742576599,
+            0.0458810068666935,
+            -0.00809895433485508,
+            0.02212512120604515,
+            -0.01569613441824913,
+            -0.011674097739160061,
+            0.024147899821400642,
+            -0.016072465106844902,
+            -0.003586903912946582,
+            0.029040200635790825,
+            0.00016084715025499463,
+            0.0012985375942662358,
+            0.021921275183558464,
+            -0.021889913827180862,
+            -0.0052411919459700584,
+            -0.014426017180085182,
+            -0.01895766891539097,
+            -0.002695078030228615,
+            0.025621861219406128,
+            0.010239336639642715,
+            0.029526295140385628,
+            -0.00020776601741090417,
+            -0.05397212132811546,
+            0.04054965451359749,
+            0.015868619084358215,
+            0.025010323151946068,
+            -0.020776601508259773,
+            -0.014253532513976097,
+            -0.047417692840099335,
+            -0.029228366911411285,
+            0.05165141448378563,
+            0.018314769491553307,
+            -0.027127185836434364,
+            0.030765051022171974,
+            0.0260295532643795,
+            0.018973348662257195,
+            -0.037350840866565704,
+            -0.009784603491425514,
+            0.0013112779706716537,
+            0.00785982795059681,
+            -0.029761500656604767,
+            0.00613497756421566,
+            0.022031037136912346,
+            -0.01818932592868805,
+            0.02342659793794155,
+            -0.01548444852232933,
+            0.015492288395762444,
+            0.006064415443688631,
+            -0.06635968387126923,
+            0.04058101400732994,
+            0.0031008098740130663,
+            0.008036232553422451,
+            0.05513247475028038,
+            0.019961217418313026,
+            0.018205007538199425,
+            -0.0099727688357234,
+            -0.04525379091501236,
+            0.008130315691232681,
+            0.006036974955350161,
+            -0.019506484270095825,
+            -0.008647770620882511,
+            0.015061075799167156,
+            0.023410918191075325,
+            0.01010605227202177,
+            -0.055038392543792725,
+            0.029275408014655113,
+            -0.019929856061935425,
+            0.005562641192227602,
+            0.006781796459108591,
+            -0.01776595413684845,
+            0.01925559714436531,
+            0.01078031212091446,
+            0.005041265860199928,
+            0.0031556913163512945,
+            -0.01123504526913166,
+            0.038072142750024796,
+            0.004821739625185728,
+            0.014802348800003529,
+            0.04224314168095589,
+            -0.03951474279165268,
+            0.0034300992265343666,
+            -0.06153010204434395,
+            0.01611950621008873,
+            -0.012019068002700806,
+            -0.030733689665794373,
+            -0.003298775525763631,
+            -0.01593918167054653,
+            0.017828676849603653,
+            -0.003900512820109725,
+            0.041521843522787094,
+            0.011423210613429546,
+            -0.006993482355028391,
+            -0.02982422336935997,
+            -0.0009119163732975721,
+            -0.005151029210537672,
+            0.017985479906201363,
+            0.0025931550189852715,
+            -0.023175710812211037,
+            0.019396722316741943,
+            -0.03333664685487747,
+            -0.033023037016391754,
+            0.035281021147966385,
+            0.017907077446579933,
+            0.05739046260714531,
+            0.024100856855511665,
+            -0.011509452946484089,
+            0.011870103888213634,
+            0.025621861219406128,
+            -0.02924404665827751,
+            -0.017279859632253647,
+            -0.03810350224375725,
+            -0.06316086649894714,
+            0.019020389765501022,
+            -0.013657675124704838,
+            0.01680944673717022,
+            0.006762195844203234,
+            -0.012716848403215408,
+            -0.008624250069260597,
+            -0.06369400024414062,
+            0.045347873121500015,
+            -0.014026165939867496,
+            -0.011352648958563805,
+            -0.0018924347823485732,
+            -0.02450854890048504,
+            -0.003341896692290902,
+            0.028397301211953163,
+            -0.03418339043855667,
+            -0.05814312398433685,
+            0.040330126881599426,
+            0.013148060999810696,
+            0.03534374386072159,
+            0.04365438222885132,
+            -0.015782376751303673,
+            -0.022234883159399033,
+            0.026217719539999962,
+            -0.035124216228723526,
+            0.007946070283651352,
+            -0.0005517559475265443,
+            -0.02668813243508339,
+            0.01322646252810955,
+            0.010011970065534115,
+            0.06447802484035492,
+            -0.02052571438252926,
+            0.021889913827180862,
+            -0.02918132394552231,
+            0.057108212262392044,
+            -0.016260631382465363,
+            0.02201535739004612,
+            -0.023018905892968178,
+            -0.0031008098740130663,
+            -0.031219784170389175,
+            0.0018375532235950232,
+            -0.019584886729717255,
+            -0.011485932394862175,
+            -0.02443014644086361,
+            -0.01301477663218975,
+            -0.007569739129394293,
+            0.01749938726425171,
+            0.010686228983104229,
+            0.024931922554969788,
+            0.030827771872282028,
+            0.002036498859524727,
+            -0.07018571346998215,
+            0.026468606665730476,
+            0.018205007538199425,
+            0.01676240563392639,
+            0.01731122098863125,
+            0.019004710018634796,
+            0.006217299960553646,
+            0.026656771078705788,
+            0.011148802936077118,
+            0.011195844039320946,
+            -0.008059753105044365,
+            0.03985971212387085,
+            -0.03716267645359039,
+            -0.007585419807583094,
+            0.01726417988538742,
+            0.0046766954474151134,
+            0.026249079033732414,
+            0.018079563975334167,
+            0.01726417988538742,
+            0.012748208828270435,
+            0.007448215503245592,
+            0.01121936459094286,
+            0.00966699980199337,
+            0.03672362491488457,
+            0.01779731549322605,
+            -0.01638607494533062,
+            0.016433116048574448,
+            -0.02817777544260025,
+            -0.01050590444356203,
+            -0.004990304354578257,
+            0.02722126804292202,
+            0.00673867529258132,
+            -0.02339523658156395,
+            -0.0031027698423713446,
+            0.009196585975587368,
+            -0.022532811388373375,
+            0.008702651597559452,
+            0.006174178794026375,
+            -6.774446228519082e-05,
+            0.009525875560939312,
+            0.007271810434758663,
+            -0.030043749138712883,
+            0.023630443960428238,
+            -0.01404968649148941,
+            -0.008726172149181366,
+            -0.01792275905609131,
+            -0.007032683584839106,
+            -0.000910936389118433,
+            0.004002436064183712,
+            0.006323142908513546,
+            0.006664192769676447,
+            -0.022501451894640923,
+            -0.004837419837713242,
+            0.011454571969807148,
+            0.02488487958908081,
+            0.002355988137423992,
+            -0.0009266168344765902,
+            -0.020353229716420174,
+            -0.006664192769676447,
+            -0.01840885356068611,
+            0.04258811101317406,
+            -0.005386236123740673,
+            -0.03223901242017746,
+            0.009902206249535084,
+            0.014849389903247356,
+            0.010200135409832,
+            0.010286377742886543,
+            -0.042556751519441605,
+            0.00047947888378985226,
+            0.0019267357420176268,
+            0.005268632434308529,
+            -0.01114096213132143,
+            -0.019114473834633827,
+            0.019882814958691597,
+            -0.051431890577077866,
+            -0.014347615651786327,
+            0.029306767508387566,
+            0.02737807296216488,
+            -0.020070981234312057,
+            0.01941240206360817,
+            -0.007851987145841122,
+            -0.004453248810023069,
+            -0.0246967151761055,
+            -0.024336064234375954,
+            0.01234835758805275,
+            0.014378976076841354,
+            -0.0033595373388379812,
+            0.03434019535779953,
+            0.016213588416576385,
+            -0.034089308232069016,
+            -0.017091695219278336,
+            -0.0149669935926795,
+            0.050052009522914886,
+            -0.013006936758756638,
+            -0.015194360166788101,
+            0.06372536718845367,
+            -0.0255748201161623,
+            0.012622765265405178,
+            0.03515557944774628,
+            0.014684745110571384,
+            -0.002234464744105935,
+            -0.024947602301836014,
+            -0.0025970751885324717,
+            -0.05425437167286873,
+            -0.0051588695496320724,
+            -0.013720396906137466,
+            0.04224314168095589,
+            -0.04142775759100914,
+            -0.004076918121427298,
+            -0.005880170036107302,
+            0.01577453687787056,
+            0.03024759516119957,
+            0.006785716395825148,
+            0.00034153988235630095,
+            0.01049022376537323,
+            0.02308162860572338,
+            -0.0034457796718925238,
+            -0.014465218409895897,
+            -0.005151029210537672,
+            -0.018471574410796165,
+            -0.04838988184928894,
+            -0.011046879924833775,
+            -0.01742098480463028,
+            -0.004813899286091328,
+            -0.031454991549253464,
+            -0.00391815323382616,
+            0.03531238064169884,
+            -0.010647027753293514,
+            -0.015421726740896702,
+            0.008522327058017254,
+            0.00809895433485508,
+            -0.022877782583236694,
+            -0.04719816520810127,
+            0.004974624142050743,
+            0.00666027283295989,
+            -0.011321287602186203,
+            -0.0572650171816349,
+            0.006750435568392277,
+            -0.02461831271648407,
+            0.01837749220430851,
+            -0.021999677643179893,
+            -0.0099727688357234,
+            0.025872748345136642,
+            0.0016327272169291973,
+            0.0045120506547391415,
+            -0.02573162503540516,
+            0.0011064520804211497,
+            0.022595534101128578,
+            0.006699474062770605,
+            0.021105891093611717,
+            -0.027958249673247337,
+            -0.03418339043855667,
+            0.020839322358369827,
+            -0.0055077592842280865,
+            -0.06350583583116531,
+            -0.0026676373090595007,
+            -0.02251713164150715,
+            0.0027009581681340933,
+            0.019161514937877655,
+            0.04694727808237076,
+            -0.030184874311089516,
+            0.008663450367748737,
+            -0.023646123707294464,
+            -0.04650822654366493,
+            -0.030467122793197632,
+            -0.0027303590904921293,
+            -0.007957830093801022,
+            -0.026547009125351906,
+            -0.012991256080567837,
+            0.008467445150017738,
+            -0.03901296854019165,
+            0.009274988435208797,
+            -0.014245692640542984,
+            -0.026233399286866188,
+            -0.028648190200328827,
+            0.010427501983940601,
+            0.00468061538413167,
+            -0.002107060980051756,
+            -0.03603368252515793,
+            0.0060722557827830315,
+            0.004939342848956585,
+            -0.005162789486348629,
+            0.0010476503521203995,
+            0.002673517446964979,
+            0.004296444356441498,
+            0.005880170036107302,
+            -0.0027538796421140432,
+            -0.04183544963598251,
+            -0.026750855147838593,
+            0.016072465106844902,
+            0.02090204507112503,
+            -0.001751310657709837,
+            0.029197005555033684,
+            -0.050898753106594086,
+            -0.03230173513293266,
+            0.025951150804758072,
+            -0.06002477928996086,
+            0.006272181402891874,
+            -0.011995547451078892,
+            -0.022376008331775665,
+            -0.00047531374730169773,
+            -0.014073207043111324,
+            -0.02063547819852829,
+            0.026249079033732414,
+            0.02626476064324379,
+            0.017624830827116966,
+            0.017938438802957535,
+            0.013336226344108582,
+            0.02342659793794155,
+            -0.02220352366566658,
+            -0.06391353160142899,
+            0.035625990480184555,
+            -0.01566477306187153,
+            0.007781425025314093,
+            0.017860036343336105,
+            -0.012536522932350636,
+            -0.004010275937616825,
+            0.0303573589771986,
+            0.011713298968970776,
+            0.004022036213427782,
+            -0.018346130847930908,
+            0.05221591144800186,
+            -0.005986013449728489,
+            -0.04886029288172722,
+            -0.0021325417328625917,
+            -0.012120991013944149,
+            0.023489320650696754,
+            0.023348195478320122,
+            -0.00301064713858068,
+            -0.028742272406816483,
+            -0.022830741479992867,
+            -0.007197328377515078,
+            -0.015194360166788101,
+            0.02798961102962494,
+            -0.00789902824908495,
+            0.01466122455894947,
+            -0.0194908045232296,
+            -0.001328918500803411,
+            -0.004104358609765768,
+            0.01139185018837452,
+            0.04287036135792732,
+            -0.040643736720085144,
+            -0.009235787205398083,
+            -0.03361889347434044,
+            -0.03261534497141838,
+            0.00725613022223115,
+            0.017562108114361763,
+            0.03844847157597542,
+            -0.02182719111442566,
+            0.01078031212091446,
+            0.01853429712355137,
+            -0.010349099524319172,
+            -0.012418919242918491,
+            0.03393250331282616,
+            0.04873485118150711,
+            0.03452835977077484,
+            0.011956346221268177,
+            -0.00046306339208967984,
+            0.008992739953100681,
+            0.011415370739996433,
+            0.01152513362467289,
+            0.012967735528945923,
+            0.029338128864765167,
+            -0.008067593909800053,
+            0.006969961803406477,
+            -0.018126605078577995,
+            -0.020118022337555885,
+            -0.01611950621008873,
+            -0.011713298968970776,
+            0.030529843643307686,
+            -0.011768180876970291,
+            -0.005782167427241802,
+            0.0294792540371418,
+            -0.008498805575072765,
+            0.00907114241272211,
+            -0.009392592124640942,
+            -0.023755887523293495,
+            -0.03085913322865963,
+            -0.04202361777424812,
+            -0.005292152985930443,
+            -0.007753984536975622,
+            -0.035092856734991074,
+            -0.0529058501124382,
+            0.010380460880696774,
+            -0.07514073699712753,
+            -0.002481431933119893,
+            0.029965346679091454,
+            0.0010692109353840351,
+            0.0008305740193463862,
+            -0.0010888115502893925,
+            0.008208717219531536,
+            -0.03763309121131897,
+            0.0031478512100875378,
+            0.02243872918188572,
+            0.03089049458503723,
+            -0.0294792540371418,
+            -0.010991998016834259,
+            0.007103245705366135,
+            -0.03879344463348389,
+            -0.0038828724063932896,
+            0.0039887153543531895,
+            0.005190230440348387,
+            0.013312705792486668,
+            -0.03227037563920021,
+            -0.0069033196195960045,
+            0.004982464015483856,
+            0.01630767248570919,
+            0.03835438936948776,
+            -0.0023677486460655928,
+            -0.04512834548950195,
+            0.019004710018634796,
+            -0.0007732423837296665,
+            0.01531196292489767,
+            -0.00893785897642374,
+            -0.020839322358369827,
+            -0.0015778456581756473,
+            -0.00539799639955163,
+            0.0006272181635722518,
+            0.023332515731453896,
+            -0.008530166931450367,
+            0.01787571795284748,
+            -0.01707601360976696,
+            -0.015210039913654327,
+            -0.002385389059782028,
+            -0.006679873447865248,
+            0.018863586708903313,
+            0.0031125701498240232,
+            -0.021121570840477943,
+            -0.026092275977134705,
+            -0.017217138782143593,
+            0.027503516525030136,
+            0.0035927840508520603,
+            -0.040298767387866974,
+            -0.012795249931514263,
+            0.024289023131132126,
+            0.019726011902093887,
+            0.006613231264054775,
+            -0.019616248086094856,
+            0.008208717219531536,
+            0.03024759516119957,
+            -0.044657934457063675,
+            -0.02522985078394413,
+            0.0026931180618703365,
+            0.0033066156320273876,
+            -0.004888381343334913,
+            -0.011148802936077118,
+            0.0006757295341230929,
+            0.04365438222885132,
+            -0.010576466098427773,
+            0.021717429161071777,
+            0.019631927832961082,
+            0.01684080809354782,
+            -0.01898903027176857,
+            -0.013649835251271725,
+            -0.011454571969807148,
+            0.025010323151946068,
+            -0.014669064432382584,
+            0.027205588296055794,
+            0.022219203412532806,
+            0.002944005187600851,
+            0.06447802484035492,
+            -0.026672452688217163,
+            -0.025559140369296074,
+            0.011705459095537663,
+            -0.013579272665083408,
+            0.0009697380592115223,
+            -0.019475122913718224,
+            -0.007812785916030407,
+            -0.004092598333954811,
+            0.022109439596533775,
+            -0.028914757072925568,
+            -0.028507065027952194,
+            0.0017552308272570372,
+            0.010129572823643684,
+            -0.009925727732479572,
+            0.01257572416216135,
+            0.0027950408402830362,
+            0.004539491608738899,
+            0.0212470144033432,
+            -0.04149048030376434,
+            0.011681938543915749,
+            0.035469185560941696,
+            -0.02477511763572693,
+            -0.034402914345264435,
+            -0.031219784170389175,
+            0.011862263083457947,
+            0.0029890865553170443,
+            -0.04434432461857796,
+            -0.015037555247545242,
+            -0.029996708035469055,
+            -0.0015660853823646903,
+            -0.03572007268667221,
+            -0.00958859734237194,
+            0.010082531720399857,
+            -0.0037143074441701174,
+            0.01183090265840292,
+            -0.022987544536590576,
+            -0.005445037502795458,
+            0.028852034360170364,
+            0.04478337615728378,
+            0.012168032117187977,
+            0.020337548106908798,
+            -0.001616066787391901,
+            -0.024978963658213615,
+            -0.015492288395762444,
+            0.012599244713783264,
+            -0.0007776524871587753,
+            0.0031968525145202875,
+            0.0294478926807642,
+            -0.004907981958240271,
+            -0.03763309121131897,
+            0.00632706331089139,
+            0.04095734655857086,
+            -0.020572755485773087,
+            0.007146366871893406,
+            0.04192953556776047,
+            0.025935471057891846,
+            -0.019239917397499084,
+            -0.020008258521556854,
+            -0.031298186630010605,
+            0.004951103124767542,
+            -0.02817777544260025,
+            -0.005201990716159344,
+            0.03688042610883713,
+            -0.006185939069837332,
+            -0.026390204206109047,
+            0.012262115254998207,
+            -0.02297186478972435,
+            -0.009337710216641426,
+            0.01278741005808115,
+            0.01100767869502306,
+            -0.03625321015715599,
+            0.013328385539352894,
+            0.019537845626473427,
+            0.023332515731453896,
+            -0.008882977068424225,
+            -0.018973348662257195,
+            0.021074529737234116,
+            0.006166338454931974,
+            0.0032301736064255238,
+            -0.004202361684292555,
+            0.0075501385144889355,
+            0.019773053005337715,
+            -0.0036045443266630173,
+            0.011305606923997402,
+            -0.0034300992265343666,
+            0.03317984193563461,
+            -0.014473059214651585,
+            -0.002887163544073701,
+            0.01188578363507986,
+            -0.014622023329138756,
+            -0.009314189665019512,
+            0.059523001313209534,
+            0.007213008590042591,
+            0.040486931800842285,
+            -0.003974995110183954,
+            0.018847905099391937,
+            0.024633992463350296,
+            -0.01627631112933159,
+            -0.027205588296055794,
+            0.0006580890621989965,
+            -0.013869361020624638,
+            -0.03085913322865963,
+            -0.01699761115014553,
+            -0.03534374386072159,
+            -0.018471574410796165,
+            -0.00014565671153832227,
+            -0.01404968649148941,
+            0.0007394314161501825,
+            0.028428662568330765,
+            0.010239336639642715,
+            -0.012168032117187977,
+            -0.033901140093803406,
+            0.03885616362094879,
+            0.030874812975525856,
+            0.03261534497141838,
+            -0.007236529607325792,
+            0.019271278753876686,
+            -0.010074691846966743,
+            -0.0058135283179581165,
+            0.020807962864637375,
+            0.015437406487762928,
+            -0.00238342909142375,
+            0.014825869351625443,
+            0.01577453687787056,
+            0.0459437295794487,
+            -0.025621861219406128,
+            -0.03024759516119957,
+            -0.010176614858210087,
+            -0.0260295532643795,
+            0.027817124500870705,
+            0.012983416207134724,
+            -0.015445247292518616,
+            0.015727495774626732,
+            0.017044654116034508,
+            -0.029620377346873283,
+            0.02010234259068966,
+            -0.01374391745775938,
+            -0.03195676580071449,
+            0.030325997620821,
+            -0.016825126484036446,
+            0.013242143206298351,
+            -0.007013082969933748,
+            -0.004033796954900026,
+            -0.020086660981178284,
+            0.009298508986830711,
+            0.01131344772875309,
+            -0.028554106131196022,
+            0.038072142750024796,
+            0.04368574544787407,
+            0.003083169227465987,
+            0.009604278020560741,
+            0.014574982225894928,
+            -0.0007943129749037325,
+            -0.020353229716420174,
+            -0.012724688276648521,
+            0.006679873447865248,
+            -0.001151533331722021,
+            -0.028836354613304138,
+            0.0307807307690382,
+            -0.016103826463222504,
+            0.007495257072150707,
+            -0.01154081430286169,
+            -0.006087936460971832,
+            0.0063192229717969894,
+            -0.0007805926143191755,
+            0.006558349821716547,
+            0.0007997031207196414,
+            -0.03170587867498398,
+            0.014888591133058071,
+            0.031110020354390144,
+            -0.005758646875619888,
+            -0.01079599279910326,
+            0.01737394370138645,
+            -0.02090204507112503,
+            0.009180906228721142,
+            0.028005290776491165,
+            0.015186519362032413,
+            0.004206281621009111,
+            -0.0051980703137815,
+            -0.030310317873954773,
+            -0.020243465900421143,
+            -0.001436721533536911,
+            0.024304702877998352,
+            0.014841550029814243,
+            0.026437245309352875,
+            0.026735173538327217,
+            -0.00780886597931385,
+            0.0013436188455671072,
+            -0.0022677856031805277,
+            -0.03396386280655861,
+            0.04440704733133316,
+            -1.7808952179620974e-05,
+            0.01779731549322605,
+            0.021152932196855545,
+            -0.027095824480056763,
+            -0.027268309146165848,
+            -0.001135852886363864,
+            0.032333094626665115,
+            0.03484196960926056,
+            0.01699761115014553,
+            0.004590453114360571,
+            0.04202361777424812,
+            -0.02464967407286167,
+            -0.03741356357932091,
+            0.017891397699713707,
+            0.030043749138712883,
+            -0.009188746102154255,
+            0.0016738884150981903,
+            0.027017422020435333,
+            -0.011838742531836033,
+            -0.005574401468038559,
+            0.024367425590753555,
+            0.019083112478256226,
+            0.008914338424801826,
+            0.004104358609765768,
+            -0.009659159928560257,
+            0.02369316667318344,
+            0.002107060980051756,
+            0.024336064234375954,
+            -0.01960056647658348,
+            0.007318852003663778,
+            0.00945531390607357,
+            -0.026891978457570076,
+            0.033023037016391754,
+            0.009941407479345798,
+            -0.0502401739358902,
+            0.025668902322649956,
+            0.0032517341896891594,
+            -0.004304284695535898,
+            0.003394818166270852,
+            0.0059076109901070595,
+            0.007789265364408493,
+            0.023363877087831497,
+            0.00446108914911747,
+            -0.004096518736332655,
+            0.030106471851468086,
+            0.023912692442536354,
+            -0.0003197342448402196,
+            0.020666837692260742,
+            -0.0019110552966594696,
+            -0.02074524015188217,
+            -0.002781320596113801,
+            -0.0063780248165130615,
+            0.04105142876505852,
+            -0.0390443317592144,
+            0.014825869351625443,
+            0.02374020777642727,
+            0.003420298919081688,
+            -0.002857762621715665,
+            -0.02320707216858864,
+            -0.008443924598395824,
+            -0.011180163361132145,
+            0.024257661774754524,
+            -0.010145253501832485,
+            0.023301154375076294,
+            -0.02833458036184311,
+            -0.0038926727138459682,
+            0.01353223156183958,
+            -0.030498482286930084,
+            0.011031199246644974,
+            0.02722126804292202,
+            0.0010897915344685316,
+            -0.02010234259068966,
+            -0.005852729547768831,
+            0.036849066615104675,
+            0.055226556956768036,
+            0.0054959990084171295,
+            0.01630767248570919,
+            0.02867954969406128,
+            -0.02140381932258606,
+            -0.0106235072016716,
+            0.019537845626473427,
+            -0.018126605078577995,
+            0.014582822099328041,
+            0.04302716627717018,
+            -0.036347292363643646,
+            -0.03782125562429428,
+            -0.008130315691232681,
+            -0.032333094626665115,
+            0.008898657746613026,
+            -0.012003387324512005,
+            0.012889333069324493,
+            -0.009204426780343056,
+            -0.02841298282146454,
+            0.02798961102962494,
+            0.015170839615166187,
+            0.009384751319885254,
+            -0.014622023329138756,
+            -0.02182719111442566,
+            -0.02411653846502304,
+            -0.031078658998012543,
+            -0.008898657746613026,
+            -0.01765619032084942,
+            0.01975737139582634,
+            -0.0006556389853358269,
+            -0.04224314168095589,
+            -0.007848067209124565,
+            0.0194908045232296,
+            -0.012120991013944149,
+            0.040675096213817596,
+            0.02416357956826687,
+            0.016558559611439705,
+            -0.009267148561775684,
+            -0.03760172799229622,
+            0.05425437167286873,
+            -2.794612191792112e-05,
+            0.03027895651757717,
+            -0.013179421424865723,
+            -0.007655981462448835,
+            0.0026107956655323505,
+            0.027958249673247337,
+            0.005201990716159344,
+            0.009470994584262371,
+            -0.031909722834825516,
+            0.011078240349888802,
+            -0.024304702877998352,
+            -0.011932825669646263,
+            -0.006899399682879448,
+            -0.0034673402551561594,
+            0.02450854890048504,
+            0.00416316045448184,
+            0.006217299960553646,
+            0.02875795215368271,
+            0.012599244713783264,
+            -0.0017483705887570977,
+            -0.023834289982914925,
+            -0.013493030332028866,
+            -0.0015964661724865437,
+            0.003849551547318697,
+            -0.01676240563392639,
+            -0.013053977862000465,
+            -0.03835438936948776,
+            -0.010748950764536858,
+            0.0034810607321560383,
+            0.021152932196855545,
+            0.02151358313858509,
+            0.005688084755092859,
+            -0.010819513350725174,
+            -0.03085913322865963,
+            -0.007405094336718321,
+            -0.04735497012734413,
+            -0.016001902520656586,
+            0.018628379330039024,
+            -0.021043168380856514,
+            0.010356939397752285,
+            0.00260295532643795,
+            0.01590782031416893,
+            -0.0012289555743336678,
+            -0.01925559714436531,
+            0.003136090701445937,
+            -0.009086823090910912,
+            0.011548654176294804,
+            0.0010986118577420712,
+            0.03932657837867737,
+            -0.004084758460521698,
+            0.027660321444272995,
+            -0.029604697600007057,
+            0.008585048839449883,
+            -0.044124796986579895,
+            -0.0097610829398036,
+            -0.03866799920797348,
+            0.009855165146291256,
+            -0.008161676116287708,
+            -0.0338384211063385,
+            0.004523810930550098,
+            0.008357682265341282,
+            -0.002804841147735715,
+            -0.015892140567302704,
+            -0.006468187086284161,
+            0.02294050343334675,
+            -0.0047668581828475,
+            0.008953538723289967,
+            -0.03938930109143257,
+            -0.016166547313332558,
+            -0.007510937284678221,
+            0.03675498440861702,
+            -0.0036241449415683746,
+            0.017436664551496506,
+            -0.015727495774626732,
+            0.016213588416576385,
+            -0.04443840682506561,
+            -0.025825707241892815,
+            0.034057945013046265,
+            0.017389623448252678,
+            -0.021874234080314636,
+            0.026625411584973335,
+            0.019710330292582512,
+            0.04042420908808708,
+            -0.0580490417778492,
+            0.023802928626537323,
+            -0.0018473535310477018,
+            -0.009008420631289482,
+            0.019098792225122452,
+            0.010756791569292545,
+            0.03289759159088135,
+            -0.028961798176169395,
+            -0.014441697858273983,
+            0.006017374340444803,
+            0.020666837692260742,
+            8.042357512749732e-05,
+            0.006691633723676205,
+            -0.02358340285718441,
+            0.014496579766273499,
+            0.017562108114361763,
+            0.005190230440348387,
+            0.02703310176730156,
+            -0.026390204206109047,
+            0.028428662568330765,
+            0.014324095100164413,
+            -0.027770083397626877,
+            0.02132541686296463,
+            -0.027127185836434364,
+            -0.01390072237700224,
+            -0.001531784306280315,
+            0.02780144475400448,
+            0.011360488831996918,
+            -0.060244303196668625,
+            0.02063547819852829,
+            0.03173723816871643,
+            -0.006189859006553888,
+            0.03134522587060928,
+            -0.024053815752267838,
+            0.024665353819727898,
+            0.021372457966208458,
+            0.022799380123615265,
+            0.018440213054418564,
+            0.046037811785936356,
+            -0.00785982795059681,
+            0.01184658333659172,
+            -0.020118022337555885,
+            0.022579854354262352,
+            0.011595695279538631,
+            -0.009541556239128113,
+            -0.037570368498563766,
+            -0.021152932196855545,
+            0.007648141589015722,
+            0.013469509780406952,
+            -0.00831064023077488,
+            -0.03317984193563461,
+            -0.0458810068666935,
+            0.010090372525155544,
+            0.00569592509418726,
+            -0.057484544813632965,
+            -0.013555752113461494,
+            -0.009110343642532825,
+            0.005229431204497814,
+            -0.008459605276584625,
+            0.004061237443238497,
+            -0.028397301211953163,
+            0.015335483476519585,
+            -0.014457378536462784,
+            0.03716267645359039,
+            -0.011485932394862175,
+            0.022140800952911377,
+            -0.015837257727980614,
+            -0.004719816613942385,
+            -0.025292571634054184,
+            0.02488487958908081,
+            -0.021466542035341263,
+            -0.04199225455522537,
+            0.010835194028913975,
+            0.004480689764022827,
+            0.01622927002608776,
+            0.0537525974214077,
+            -0.005884090438485146,
+            -0.010317739099264145,
+            0.020321868360042572,
+            -0.019506484270095825,
+            -0.014002645388245583,
+            0.007679502479732037,
+            -0.005844889208674431,
+            -0.022031037136912346,
+            -0.01665264181792736,
+            -0.012262115254998207,
+            0.0247280765324831,
+            -0.00041602205601520836,
+            0.022611213847994804,
+            0.026390204206109047,
+            0.0195221658796072,
+            -0.02814641408622265,
+            0.009855165146291256,
+            -0.005127508658915758,
+            -0.011031199246644974,
+            0.004245482850819826,
+            0.021356778219342232,
+            -0.004755097441375256,
+            0.009000580757856369,
+            -0.01646447740495205,
+            -0.021121570840477943,
+            -0.03731948137283325,
+            -0.01249732170253992,
+            -0.003145891008898616,
+            0.0031478512100875378,
+            -0.035625990480184555,
+            -0.018126605078577995,
+            0.026060914620757103,
+            0.03070232830941677,
+            -0.028303219005465508,
+            0.0019051751587539911,
+            -0.002246225019916892,
+            -0.01132912840694189,
+            -0.003904432989656925,
+            -0.011791701428592205,
+            0.007040523923933506,
+            -0.015225720591843128,
+            8.691626862855628e-05,
+            -0.002042379230260849,
+            0.0023736287839710712,
+            -0.05792359635233879,
+            -0.017781633883714676,
+            0.02913428284227848,
+            0.002930284710600972,
+            0.00550383934751153,
+            -0.008279279805719852,
+            -0.01322646252810955,
+            -0.007009163033217192,
+            -0.02679789625108242,
+            0.024367425590753555,
+            0.015437406487762928,
+            0.019647609442472458,
+            -0.003559462958946824,
+            0.0017885518027469516,
+            0.018659740686416626,
+            0.0014171210350468755,
+            0.003351696999743581,
+            -0.007279650773853064,
+            0.006507388316094875,
+            0.02182719111442566,
+            -0.012795249931514263,
+            0.0158294178545475,
+            0.005778247490525246,
+            0.002685277722775936,
+            -0.02679789625108242,
+            -0.003682946553453803,
+            0.016793766990303993,
+            0.002657837001606822,
+            0.005296073388308287,
+            -0.0255434587597847,
+            0.038072142750024796,
+            0.04039284959435463,
+            0.025872748345136642,
+            0.020337548106908798,
+            -0.03220765292644501,
+            -0.004206281621009111,
+            -0.00758149940520525,
+            0.01121936459094286,
+            0.021811511367559433,
+            0.020055299624800682,
+            -0.010427501983940601,
+            -0.015743175521492958,
+            0.011062560603022575,
+            -0.02013370208442211,
+            -0.0019473163411021233,
+            0.018894946202635765,
+            0.011266405694186687,
+            -0.006201619748026133,
+            0.009988449513912201,
+            0.0004351325915195048,
+            0.005268632434308529,
+            0.02140381932258606,
+            -0.030529843643307686,
+            -0.025167128071188927,
+            0.0212470144033432,
+            -0.017123056575655937,
+            0.01154081430286169,
+            0.004876621067523956,
+            0.0032517341896891594,
+            0.020776601508259773,
+            0.07445079833269119,
+            -0.020070981234312057,
+            0.0043591661378741264,
+            -0.022768018767237663,
+            -0.02576298639178276,
+            -0.012614925391972065,
+            -0.022109439596533775,
+            0.005378395784646273,
+            0.013571432791650295,
+            -0.00024022944853641093,
+            0.020651157945394516,
+            0.01056862622499466,
+            -0.00644074659794569,
+            0.007216928992420435,
+            -0.01173681952059269,
+            -0.0007139506633393466,
+            -0.006585790775716305,
+            -0.0027087985072284937,
+            0.002058059675619006,
+            0.0015229640994220972,
+            0.002697037998586893,
+            0.02836594171822071,
+            0.012371878139674664,
+            0.013116699643433094,
+            -0.03327392414212227,
+            0.009815963916480541,
+            0.02151358313858509,
+            0.018095243722200394,
+            -0.03317984193563461,
+            -0.021388139575719833,
+            0.004543411545455456,
+            0.005017745308578014,
+            0.012928534299135208,
+            0.009196585975587368,
+            -0.006977802142500877,
+            0.02504168450832367,
+            -0.0355319082736969,
+            0.01814228482544422,
+            0.016527198255062103,
+            -0.011705459095537663,
+            -0.004931502975523472,
+            0.01986713521182537,
+            -0.0020090581383556128,
+            0.006860198453068733,
+            -0.005946812219917774,
+            0.0025794347748160362,
+            0.00653874920681119,
+            -0.021356778219342232,
+            -0.003559462958946824,
+            -0.012983416207134724,
+            -0.001520024030469358,
+            0.021419500932097435,
+            -0.017170097678899765,
+            0.008726172149181366,
+            -0.008655610494315624,
+            0.028836354613304138,
+            0.013414627872407436,
+            -0.029933987185359,
+            0.0007183607667684555,
+            0.023065946996212006,
+            0.01173681952059269,
+            -0.002052179304882884,
+            0.00922794733196497,
+            0.008851616643369198,
+            -0.016825126484036446,
+            6.958201265661046e-05,
+            0.012520842254161835,
+            0.018079563975334167,
+            0.007957830093801022,
+            0.006879799067974091,
+            0.006417225580662489,
+            0.019616248086094856,
+            -0.018785184249281883,
+            -0.02010234259068966,
+            0.00456693209707737,
+            0.010521584190428257,
+            -0.0065426696091890335,
+            -0.020996127277612686,
+            0.007060124538838863,
+            -0.016401754692196846,
+            0.037350840866565704,
+            0.009690520353615284,
+            -0.010803832672536373,
+            -0.0035790635738521814,
+            -0.031141381710767746,
+            -0.0027479995042085648,
+            -0.021983996033668518,
+            -0.010153094306588173,
+            0.019004710018634796,
+            -0.003353656968101859,
+            -0.00965131912380457,
+            -0.01906743273139,
+            -0.011015518568456173,
+            -0.008734012953937054,
+            0.0017326901433989406,
+            -0.008984900079667568,
+            0.01944376341998577,
+            0.01144673116505146,
+            -0.02844434417784214,
+            -0.031799960881471634,
+            -0.03361889347434044,
+            0.011054719798266888,
+            0.0025559139903634787,
+            -0.02913428284227848,
+            -0.011721138842403889,
+            -0.004555171821266413,
+            -0.004080838058143854,
+            -0.008000951260328293,
+            0.008145995438098907,
+            -0.003831910900771618,
+            0.006017374340444803,
+            -0.013830160722136497,
+            -0.005554800853133202,
+            -0.008130315691232681,
+            0.00788726843893528,
+            0.0149513129144907,
+            -0.005488158669322729,
+            -0.03093753568828106,
+            -0.017781633883714676,
+            0.00965131912380457,
+            -0.012826611287891865,
+            0.008145995438098907,
+            0.01684080809354782,
+            -0.004923662636429071,
+            0.00594289181753993,
+            0.01848725602030754,
+            0.018095243722200394,
+            -0.0030753291212022305,
+            0.014363295398652554,
+            -0.0011162522714585066,
+            -0.006385864689946175,
+            0.012607084587216377,
+            0.008953538723289967,
+            0.01487291045486927,
+            0.02507304586470127,
+            -0.010294217616319656,
+            -0.006087936460971832,
+            -0.0016954491147771478,
+            0.03274078667163849,
+            -0.023599082604050636,
+            -0.019804412499070168,
+            0.015688294544816017,
+            0.05425437167286873,
+            -0.0010662708664312959,
+            -0.0020913805346935987,
+            -0.022344646975398064,
+            -0.01102335937321186,
+            0.023489320650696754,
+            0.016793766990303993,
+            0.008036232553422451,
+            -0.010843033902347088,
+            -0.014386816881597042,
+            0.010545105673372746,
+            0.014418177306652069,
+            0.01922423578798771,
+            -0.024053815752267838,
+            0.021278375759720802,
+            -0.026750855147838593,
+            0.036127764731645584,
+            0.0075501385144889355,
+            -0.005974252708256245,
+            0.015539329499006271,
+            0.0036378654185682535,
+            -0.006526988931000233,
+            0.014324095100164413,
+            -0.01070190966129303,
+            0.011893624439835548,
+            0.006868038792163134,
+            -0.0015004234155640006,
+            0.021043168380856514,
+            -0.01108608115464449,
+            -0.0010045290691778064,
+            -0.0027048783376812935,
+            0.02090204507112503,
+            0.0031184502877295017,
+            -0.0038671919610351324,
+            0.01464554388076067,
+            0.0028773632366210222,
+            -0.004351325798779726,
+            -0.010074691846966743,
+            0.007350212894380093,
+            -0.03173723816871643,
+            -0.008138155564665794,
+            0.013924242928624153,
+            -0.005735125858336687,
+            0.002120781457051635,
+            -0.011015518568456173,
+            -0.002642156556248665,
+            -0.03609640523791313,
+            0.01787571795284748,
+            -0.000642898608930409,
+            0.017091695219278336,
+            0.0028656029608100653,
+            -0.00936123076826334,
+            -0.004809979349374771,
+            0.02297186478972435,
+            -0.009565076790750027,
+            -0.006930760573595762,
+            -0.018769502639770508,
+            -0.021497901529073715,
+            -0.025135766714811325,
+            0.021121570840477943,
+            -0.009580757468938828,
+            0.0018189327092841268,
+            -0.007706942968070507,
+            -0.0029126442968845367,
+            -0.00791470892727375,
+            0.012222914025187492,
+            0.018706781789660454,
+            -0.030576884746551514,
+            0.006617151666432619,
+            -0.006170258391648531,
+            -0.022109439596533775,
+            0.006456426810473204,
+            -0.009518035687506199,
+            0.01684080809354782,
+            -0.0009344570571556687,
+            -0.04123959317803383,
+            0.006303542293608189,
+            0.026484286412596703,
+            -0.017969800159335136,
+            -0.0015141437761485577,
+            0.010043330490589142,
+            0.008718332275748253,
+            -0.0026499966625124216,
+            -0.018894946202635765,
+            0.01811092346906662,
+            -0.017436664551496506,
+            0.018675420433282852,
+            0.026844937354326248,
+            -0.008757533505558968,
+            0.028993159532546997,
+            0.0001089056531782262,
+            0.01864405907690525,
+            0.0029832064174115658,
+            0.005641043186187744,
+            0.010082531720399857,
+            0.0038985528517514467,
+            0.020070981234312057,
+            0.00295772566460073,
+            0.007949990220367908,
+            0.038072142750024796,
+            -0.010788151994347572,
+            0.02182719111442566,
+            0.011305606923997402,
+            -0.00019184054690413177,
+            0.01615086756646633,
+            -0.007628540974110365,
+            -0.032489899545907974,
+            0.01566477306187153,
+            -0.027770083397626877,
+            -0.004668855108320713,
+            0.00922794733196497,
+            -0.0147396270185709,
+            -0.00934555009007454,
+            -0.014418177306652069,
+            0.009235787205398083,
+            0.012779570184648037,
+            -0.013908562250435352,
+            -0.0005326454411260784,
+            0.018973348662257195,
+            0.019381040707230568,
+            0.014527940191328526,
+            0.005982093047350645,
+            -0.005182390101253986,
+            0.013830160722136497,
+            0.0030204474460333586,
+            0.023646123707294464,
+            0.0016895688604563475,
+            0.005633202847093344,
+            -0.010592146776616573,
+            0.0145122604444623,
+            0.01508459635078907,
+            -0.0017875717021524906,
+            0.01092927623540163,
+            -0.004845260176807642,
+            -0.005817448254674673,
+            -0.01960056647658348,
+            -0.007636380847543478,
+            -0.009110343642532825,
+            0.006170258391648531,
+            0.04719816520810127,
+            0.0004782538453582674,
+            -0.002846002345904708,
+            -0.01765619032084942,
+            0.004559091757982969,
+            0.012685487046837807,
+            -0.0097454022616148,
+            -0.0034987013787031174,
+            -0.019365360960364342,
+            -0.022720977663993835,
+            0.0034144187811762094,
+            -0.0050843870267271996,
+            0.027001742273569107,
+            -0.003822110593318939,
+            0.02767600119113922,
+            -0.029102923348546028,
+            0.01956920698285103,
+            -0.016135187819600105,
+            -0.0097454022616148,
+            -0.0022560253273695707,
+            -0.012160192243754864,
+            -0.014331934973597527,
+            0.02300322614610195,
+            0.00936123076826334,
+            -0.032678067684173584,
+            -0.0016190068563446403,
+            0.01010605227202177,
+            -0.009306349791586399,
+            0.009008420631289482,
+            0.006860198453068733,
+            0.010803832672536373,
+            -0.0015798057429492474,
+            0.01190930511802435,
+            0.0013592992909252644,
+            -0.001451421994715929,
+            -9.536901052342728e-05,
+            -0.02294050343334675,
+            0.018346130847930908,
+            0.030482802540063858,
+            -0.05265496298670769,
+            -0.01914583332836628,
+            0.020118022337555885,
+            0.00569592509418726,
+            0.0005865469574928284,
+            0.013061817735433578,
+            -0.024367425590753555,
+            0.01027853786945343,
+            -0.018675420433282852,
+            -0.014316254295408726,
+            -0.022140800952911377,
+            -0.005460718180984259,
+            0.0151786794885993,
+            -0.027581918984651566,
+            0.025794345885515213,
+            0.017170097678899765,
+            -0.0015023835003376007,
+            0.006452506873756647,
+            0.0007472716388292611,
+            0.008851616643369198,
+            -0.03255262225866318,
+            -0.006617151666432619,
+            -0.0059664128348231316,
+            0.006009534001350403,
+            0.003655505832284689,
+            -0.02897747792303562,
+            0.014527940191328526,
+            -0.02074524015188217,
+            0.008263599127531052,
+            -0.029949666932225227,
+            0.011321287602186203,
+            0.03622184693813324,
+            -2.7471420253277756e-05,
+            0.009157384745776653,
+            -0.0031282505951821804,
+            0.04218042269349098,
+            0.026625411584973335,
+            0.0015709854196757078,
+            -0.008024472743272781,
+            -0.019694650545716286,
+            -0.002522593131288886,
+            0.00673867529258132,
+            -0.02918132394552231,
+            0.0012113150442019105,
+            -0.035092856734991074,
+            0.02209375984966755,
+            -0.006099696736782789,
+            -0.025512099266052246,
+            0.033556170761585236,
+            -0.020384589210152626,
+            0.002808761317282915,
+            -0.005347034893929958,
+            0.0027068383060395718,
+            0.030561204999685287,
+            0.0024990723468363285,
+            0.03005943074822426,
+            -0.03826030716300011,
+            -0.0013504790840670466,
+            0.0028244417626410723,
+            0.0038162304554134607,
+            -0.010435341857373714,
+            -0.015711814165115356,
+            0.016903528943657875,
+            -0.010200135409832,
+            0.004292524419724941,
+            0.02522985078394413,
+            -0.003655505832284689,
+            -0.013132380321621895,
+            0.021278375759720802,
+            -0.002808761317282915,
+            0.005417597014456987,
+            -0.005460718180984259,
+            0.000567926443181932,
+            0.023222751915454865,
+            0.02568458393216133,
+            -0.01136832870543003,
+            -0.011336968280375004,
+            -0.002781320596113801,
+            -0.00031654915073886514,
+            0.014104568399488926,
+            0.015264921821653843,
+            -0.008200877346098423,
+            -0.020541394129395485,
+            0.006817077286541462,
+            0.009808124043047428,
+            0.018549976870417595,
+            0.0012946174247190356,
+            -0.0255748201161623,
+            -0.008459605276584625,
+            -0.03027895651757717,
+            0.010153094306588173,
+            -0.016558559611439705,
+            -0.005601841956377029,
+            0.0013112779706716537,
+            -0.001341658877208829,
+            -0.009549396112561226,
+            0.014394656755030155,
+            0.015194360166788101,
+            -0.03901296854019165,
+            -0.015813738107681274,
+            -0.0007222809363156557,
+            -0.018502935767173767,
+            0.0024539909791201353,
+            -0.0011662337929010391,
+            0.001961036818102002,
+            0.0021658625919371843,
+            -0.003165491623803973,
+            0.004739417228847742,
+            -0.020070981234312057,
+            -0.0024324303958564997,
+            0.005629282910376787,
+            0.02113725244998932,
+            0.004535571206361055,
+            -0.0015014035161584616,
+            0.0022442650515586138,
+            0.0038907125126570463,
+            0.0017464105039834976,
+            0.0063897850923240185,
+            -0.00673867529258132,
+            0.015108116902410984,
+            -0.00020262086763978004,
+            0.015327643603086472,
+            -0.01906743273139,
+            0.007557978853583336,
+            -0.002369708614423871,
+            -0.005123588256537914,
+            0.015782376751303673,
+            -0.00037020572926849127,
+            -0.0020482593681663275,
+            -0.010294217616319656,
+            0.007605019956827164,
+            0.015586371533572674,
+            -0.022501451894640923,
+            -0.009063302539288998,
+            -0.014614183455705643,
+            0.02711150422692299,
+            0.0153746847063303,
+            0.008122474886476994,
+            -0.03223901242017746,
+            -0.0056488835252821445,
+            -0.012120991013944149,
+            -0.0059193712659180164,
+            -0.015296283178031445,
+            0.018863586708903313,
+            0.019349679350852966,
+            -0.0051784696988761425,
+            0.01619790866971016,
+            -0.0015592251438647509,
+            0.020855003967881203,
+            -0.00924362801015377,
+            0.006476027425378561,
+            -0.0007590319728478789,
+            -0.006315303035080433,
+            0.010168774053454399,
+            0.03521829843521118,
+            -0.017389623448252678,
+            0.008240078575909138,
+            -0.000719830859452486,
+            -0.002530433237552643,
+            -0.00306356861256063,
+            -0.004022036213427782,
+            0.006358424201607704,
+            0.014316254295408726,
+            0.009369071573019028,
+            0.0029342048801481724,
+            0.07056204229593277,
+            -0.003506541484966874,
+            0.020243465900421143,
+            0.006323142908513546,
+            -0.014574982225894928,
+            -0.005891930777579546,
+            0.0022109439596533775,
+            -0.028397301211953163,
+            0.031266823410987854,
+            0.020086660981178284,
+            0.0048413402400910854,
+            0.003782909596338868,
+            -0.014692584984004498,
+            0.023034587502479553,
+            0.004453248810023069,
+            0.0030616086442023516,
+            -0.0049119023606181145,
+            0.005864489823579788,
+            0.047762662172317505,
+            0.012403239496052265,
+            -0.007420775014907122,
+            -0.0022658256348222494,
+            0.014292733743786812,
+            0.0054254368878901005,
+            -0.011493772268295288,
+            -0.0003150791162624955,
+            -0.000643878651317209,
+            0.015037555247545242,
+            -0.001587645965628326,
+            0.021952634677290916,
+            -0.01114096213132143,
+            -0.01826772838830948,
+            0.0018375532235950232,
+            0.004394447430968285,
+            0.01571965590119362,
+            0.009486674331128597,
+            0.008318481035530567,
+            -0.005884090438485146,
+            0.004700215999037027,
+            0.006397624965757132,
+            0.005664564203470945,
+            0.00790294911712408,
+            -0.005852729547768831,
+            -0.008655610494315624,
+            -0.01826772838830948,
+            0.02637452259659767,
+            -0.002616675803437829,
+            0.003259574295952916,
+            -0.016856487840414047,
+            0.002438310533761978,
+            -0.0034712604247033596,
+            -0.007769664749503136,
+            0.010937117040157318,
+            0.017750274389982224,
+            -0.009392592124640942,
+            -0.008185196667909622,
+            0.0147239463403821,
+            -0.017028972506523132,
+            0.004457169212400913,
+            0.012622765265405178,
+            -0.004778618458658457,
+            0.006311382632702589,
+            0.003381097922101617,
+            -0.008898657746613026,
+            0.0012073949910700321,
+            0.0013671396300196648,
+            0.015539329499006271,
+            0.015249241143465042,
+            0.007075804751366377,
+            0.022031037136912346,
+            0.0016395874554291368,
+            0.0009217166807502508,
+            0.01654287800192833,
+            0.015609892085194588,
+            0.0064721074886620045,
+            0.022078078240156174,
+            -0.015445247292518616,
+            -0.016668323427438736,
+            0.003367377445101738,
+            0.006895479746162891,
+            0.0025441537145525217,
+            0.0030243676155805588,
+            0.0036751064471900463,
+            0.010545105673372746,
+            -0.014394656755030155,
+            -0.006946441251784563,
+            -0.009322029538452625,
+            0.018973348662257195,
+            0.022689616307616234,
+            -0.020588435232639313,
+            0.0003922563628293574,
+            0.004958943463861942,
+            0.017091695219278336,
+            -0.001889494713395834,
+            0.006558349821716547,
+            0.01760914921760559,
+            -0.006574030499905348,
+            -0.008389042690396309,
+            -0.02316003106534481,
+            -0.012426760047674179,
+            -0.0006556389853358269,
+            0.019898496568202972,
+            -0.011407529935240746,
+            0.01286581251770258,
+            -0.015633411705493927,
+            -0.0042768437415361404,
+            0.029918305575847626,
+            0.011877943761646748,
+            0.005739046260714531,
+            -0.01330486498773098,
+            -0.011674097739160061,
+            -0.013673355802893639,
+            0.014324095100164413,
+            0.010317739099264145,
+            0.003122370457276702,
+            -0.010121732950210571,
+            -0.0017013292526826262,
+            -0.005515599623322487,
+            -0.008961379528045654,
+            -0.00685235857963562,
+            0.011462411843240261,
+            -0.0027617199812084436,
+            0.003949514124542475,
+            -0.011987706646323204,
+            0.0009903187165036798,
+            -0.010748950764536858,
+            -0.006495628040283918,
+            -0.0015415846137329936,
+            -0.03481060639023781,
+            -0.0299026258289814,
+            -0.001531784306280315,
+            -0.005107908044010401,
+            0.005652803461998701,
+            0.004123959224671125,
+            0.011611375957727432,
+            0.0029185244347900152,
+            0.00937691144645214,
+            0.009251467883586884,
+            0.011062560603022575,
+            3.1728417525300756e-05,
+            -0.004202361684292555,
+            0.011995547451078892,
+            -0.013602794148027897,
+            -0.016244949772953987,
+            0.024916240945458412,
+            0.004751177504658699,
+            0.013461669906973839,
+            -0.028397301211953163,
+            0.011783861555159092,
+            -0.002575514605268836,
+            0.010654868558049202,
+            -0.031141381710767746,
+            -0.0039769550785422325,
+            0.0020443391986191273,
+            -0.0033242562785744667,
+            -0.0101844547316432,
+            -0.028052331879734993,
+            -0.0075383782386779785,
+            -0.017812995240092278,
+            -0.01630767248570919,
+            -0.024759436026215553,
+            0.0050138249062001705,
+            -0.012034748680889606,
+            0.021341098472476006,
+            -0.010286377742886543,
+            0.03725675866007805,
+            0.006040894892066717,
+            -0.0015719655202701688,
+            0.01361063402146101,
+            0.0030478881672024727,
+            -0.0020090581383556128,
+            0.016636962071061134,
+            -0.006942520849406719,
+            -0.0013749798526987433,
+            0.017969800159335136,
+            0.01102335937321186,
+            0.003243893850594759,
+            -0.003602584358304739,
+            0.0029949666932225227,
+            0.026562688872218132,
+            0.01814228482544422,
+            0.00426900340244174,
+            0.005840969271957874,
+            0.002998886862769723,
+            0.016417434439063072,
+            0.009808124043047428,
+            -0.014778828248381615,
+            -0.008145995438098907,
+            0.011611375957727432,
+            -0.01173681952059269,
+            -0.006832757964730263,
+            0.004845260176807642,
+            -0.0097454022616148,
+            -0.023097308352589607,
+            0.012748208828270435,
+            0.006256501190364361,
+            -0.0005919371615163982,
+            -0.019239917397499084,
+            -0.012920694425702095,
+            0.0017620910657569766,
+            0.01225427445024252,
+            0.016323352232575417,
+            -0.01914583332836628,
+            0.002738199196755886,
+            0.008153836242854595,
+            -0.006417225580662489,
+            -0.011580015532672405,
+            0.016950570046901703,
+            0.022313285619020462,
+            0.014888591133058071,
+            -0.04788810759782791,
+            -0.008945698849856853,
+            0.017546428367495537,
+            -0.012183712795376778,
+            -0.043967992067337036,
+            -0.0015455047832801938,
+            -0.0018395131919533014,
+            -0.025982512161135674,
+            0.0052490318194031715,
+            0.006182019133120775,
+            -0.014700425788760185,
+            0.016527198255062103,
+            0.03200380504131317,
+            0.00019159555085934699,
+            0.009102503769099712,
+            -0.011109601706266403,
+            -0.014426017180085182,
+            -0.009047621861100197,
+            0.008726172149181366,
+            -0.01175250019878149,
+            -0.009847325272858143,
+            0.00248927203938365,
+            -0.015555010177195072,
+            0.01848725602030754,
+            0.007569739129394293,
+            0.03150203078985214,
+            0.01132912840694189,
+            -0.00248927203938365,
+            -0.01760914921760559,
+            -0.020964767783880234,
+            -0.009533716365695,
+            0.003741748398169875,
+            0.007997031323611736,
+            0.007005242630839348,
+            0.010160934180021286,
+            0.013328385539352894,
+            0.022673936560750008,
+            -0.006252580787986517,
+            0.014825869351625443,
+            -3.0059307391638868e-05,
+            -0.0014680825406685472,
+            0.0025676742661744356,
+            -0.020149383693933487,
+            0.005143188871443272,
+            0.0018228527624160051,
+            -0.004904062021523714,
+            -0.007013082969933748,
+            -0.003083169227465987,
+            0.032066527754068375,
+            -0.016480157151818275,
+            0.015970543026924133,
+            -0.031078658998012543,
+            -0.009337710216641426,
+            0.001246596104465425,
+            0.02814641408622265,
+            0.012701167725026608,
+            -0.02841298282146454,
+            -0.011297767050564289,
+            -0.026515647768974304,
+            0.02278370037674904,
+            -0.014300573617219925,
+            0.004723736550658941,
+            0.015845099464058876,
+            0.01444953866302967,
+            0.01194850541651249,
+            -0.01165841706097126,
+            -0.00878889486193657,
+            0.011776020750403404,
+            0.021999677643179893,
+            0.007679502479732037,
+            0.0022266244050115347,
+            -0.0009261267841793597,
+            -0.001644487609155476,
+            -0.004304284695535898,
+            -0.017985479906201363,
+            0.006711234338581562,
+            -0.00819303747266531,
+            0.027127185836434364,
+            -0.007597180083394051,
+            0.016746724024415016,
+            -0.0062957024201750755,
+            -0.00582920853048563,
+            -0.02932244911789894,
+            -0.0203689094632864,
+            0.007605019956827164,
+            -0.006703393999487162,
+            0.012959894724190235,
+            -0.019365360960364342,
+            -0.0164958368986845,
+            0.010733271017670631,
+            -0.02010234259068966,
+            -0.002083540428429842,
+            -0.004719816613942385,
+            0.0022442650515586138,
+            0.00676611578091979,
+            0.00736197317019105,
+            -0.012826611287891865,
+            -0.006883719470351934,
+            0.016072465106844902,
+            -0.007663821801543236,
+            0.007640301249921322,
+            0.029871264472603798,
+            0.01768755167722702,
+            0.026844937354326248,
+            0.007048364263027906,
+            0.013751758262515068,
+            -0.010364780202507973,
+            -0.017546428367495537,
+            -0.023301154375076294,
+            0.01278741005808115,
+            -0.0025519938208162785,
+            0.03368161618709564,
+            0.00478645833209157,
+            -0.007820626720786095,
+            -0.0031027698423713446,
+            0.005417597014456987,
+            0.0008408643188886344,
+            -0.01414376962929964,
+            0.0029518455266952515,
+            -0.00551167968660593,
+            0.012826611287891865,
+            0.016033263877034187,
+            -0.01883222535252571,
+            0.02522985078394413,
+            -0.016856487840414047,
+            0.012779570184648037,
+            -0.003602584358304739,
+            0.01630767248570919,
+            0.009753242135047913,
+            -0.0003626105026341975,
+            -0.0015415846137329936,
+            -0.05394076183438301,
+            -0.010458862408995628,
+            0.0038456313777714968,
+            -0.004970703739672899,
+            0.000352075177943334,
+            -0.012638445943593979,
+            -0.038573917001485825,
+            0.0034810607321560383,
+            0.013391107320785522,
+            -0.011556494981050491,
+            0.013422468677163124,
+            -0.002016898477450013,
+            0.006617151666432619,
+            -0.007922549732029438,
+            -0.006260421127080917,
+            0.016480157151818275,
+            0.02573162503540516,
+            -0.021121570840477943,
+            -0.022344646975398064,
+            0.005778247490525246,
+            -0.005543040577322245,
+            -0.014684745110571384,
+            -0.005021665245294571,
+            -0.01598622277379036,
+            0.0025069124531000853,
+            0.023567723110318184,
+            -0.01426137238740921,
+            -0.01776595413684845,
+            0.010145253501832485,
+            0.007757904473692179,
+            -0.024853520095348358,
+            -0.002355988137423992,
+            -0.009815963916480541,
+            -0.010803832672536373,
+            0.018393171951174736,
+            -0.01362631469964981,
+            0.008906497620046139,
+            -0.013124539516866207,
+            0.010396140627563,
+            -0.02825617790222168,
+            -0.01226995512843132,
+            -0.0019502564100548625,
+            -0.008882977068424225,
+            0.028083693236112595,
+            -0.010772472247481346,
+            -0.010200135409832,
+            0.020070981234312057,
+            -0.008663450367748737,
+            0.012567883357405663,
+            0.007855907082557678,
+            0.028648190200328827,
+            -0.0025853149127215147,
+            0.009141704998910427,
+            0.0028930436819791794,
+            0.01390072237700224,
+            0.0011936745140701532,
+            0.035437826067209244,
+            -0.00957291666418314,
+            -0.00030307378619909286,
+            -0.008835935965180397,
+            -0.0017140696290880442,
+            0.0010427501983940601,
+            0.010991998016834259,
+            0.014151609502732754,
+            0.012693327851593494,
+            -0.006244740914553404,
+            -0.0077853454276919365,
+            -0.015500128269195557,
+            -0.004527730867266655,
+            -0.004888381343334913,
+            0.007428614888340235,
+            0.012442439794540405,
+            -0.020039619877934456,
+            0.005358795169740915,
+            -0.012144511565566063,
+            0.0011946546146646142,
+            -0.01587645895779133,
+            -0.0022560253273695707,
+            -0.013634154573082924,
+            -0.004080838058143854,
+            -0.004331725183874369,
+            0.003790749702602625,
+            0.008749693632125854,
+            -0.016401754692196846,
+            0.0199455376714468,
+            0.008153836242854595,
+            -0.0004074467869941145,
+            -0.0028107212856411934,
+            -0.024931922554969788,
+            -0.014010485261678696,
+            0.00014859679504297674,
+            0.016668323427438736,
+            0.0050138249062001705,
+            -0.0051784696988761425,
+            0.002026698552072048,
+            -0.012826611287891865,
+            0.005280392710119486,
+            -0.0009040761506184936,
+            0.0022285846062004566,
+            -0.0030067269690334797,
+            0.0057312059216201305,
+            0.006762195844203234,
+            0.003700587200000882,
+            0.012959894724190235,
+            0.01487291045486927,
+            -0.011760340072214603,
+            -0.00966699980199337,
+            0.012285635806620121,
+            0.015217880718410015,
+            -0.008914338424801826,
+            0.01941240206360817,
+            -0.02878931351006031,
+            0.002820521593093872,
+            0.005233351606875658,
+            -0.0042533231899142265,
+            -0.0021854632068425417,
+            7.748349162284285e-05,
+            0.028475703671574593,
+            -0.0010221695993095636,
+            -0.010388300754129887,
+            -0.0009854185627773404,
+            0.01707601360976696,
+            -0.0006600490887649357,
+            0.006123217288404703,
+            0.023724526166915894,
+            0.00023790188424754888,
+            0.009768922813236713,
+            0.009776762686669827,
+            -0.019506484270095825,
+            -0.003145891008898616,
+            0.004633574280887842,
+            -0.012442439794540405,
+            0.018393171951174736,
+            0.014841550029814243,
+            0.008389042690396309,
+            0.0038005500100553036,
+            0.007510937284678221,
+            0.020086660981178284,
+            0.006374104414135218,
+            -0.002336387522518635,
+            -0.0002511322672944516,
+            0.0066445921547710896,
+            -0.011721138842403889,
+            0.017405303195118904,
+            -0.004363086074590683,
+            -0.015437406487762928,
+            0.006679873447865248,
+            0.011674097739160061,
+            -0.006773956120014191,
+            -0.004033796954900026,
+            0.0022540653590112925,
+            0.005711605306714773,
+            0.005464638117700815,
+            -0.003888752544298768,
+            -0.006574030499905348,
+            0.0012505161575973034,
+            0.017812995240092278,
+            0.010639187879860401,
+            0.016433116048574448,
+            -0.014990514144301414,
+            -0.011697618290781975,
+            0.0023050266318023205,
+            -0.018016841262578964,
+            0.002996926661580801,
+            -0.011517293751239777,
+            0.006299622356891632,
+            -0.007420775014907122,
+            0.02568458393216133,
+            0.019616248086094856,
+            0.007573659066110849,
+            0.011932825669646263,
+            -0.015539329499006271,
+            0.016715364530682564,
+            -0.00913386419415474,
+            0.012120991013944149,
+            -0.018847905099391937,
+            -0.0017346502281725407,
+            0.005296073388308287,
+            0.027628960087895393,
+            0.010302058421075344,
+            0.006064415443688631,
+            -0.001778751495294273,
+            -0.005182390101253986,
+            -0.008318481035530567,
+            -0.009619958698749542,
+            -0.014755306765437126,
+            -0.026233399286866188,
+            -0.00022810160589870065,
+            0.016339033842086792,
+            -0.00195907661691308,
+            0.0022521051578223705,
+            0.014590661972761154,
+            0.013187261298298836,
+            -0.0024128297809511423,
+            -0.013908562250435352,
+            -0.007252209819853306,
+            0.010835194028913975,
+            0.005119668319821358,
+            -0.008357682265341282,
+            -0.009141704998910427,
+            0.008138155564665794,
+            -0.013273504562675953,
+            0.016433116048574448,
+            -0.016182228922843933,
+            -0.009949248284101486,
+            -0.013649835251271725,
+            -0.0005929171456955373,
+            -0.008263599127531052,
+            -0.031454991549253464,
+            0.012152351438999176,
+            0.005707685369998217,
+            -0.0006365284207277,
+            -0.025715945288538933,
+            0.02369316667318344,
+            -0.008569368161261082,
+            0.03656681999564171,
+            -0.01593918167054653,
+            0.015633411705493927,
+            0.0010623506968840957,
+            0.011493772268295288,
+            -0.003392858197912574,
+            0.0029479253571480513,
+            0.007953910157084465,
+            0.002550033852458,
+            0.003055728506296873,
+            -0.00539799639955163,
+            -0.0033066156320273876,
+            0.008898657746613026,
+            -0.006448586471378803,
+            0.008490965701639652,
+            -0.014849389903247356,
+            -0.02320707216858864,
+            -0.01144673116505146,
+            0.009502355009317398,
+            0.014104568399488926,
+            0.00861640926450491,
+            -0.014473059214651585,
+            0.00882809516042471,
+            -0.02435174584388733,
+            -0.0023461878299713135,
+            -0.0031282505951821804,
+            0.004782538395375013,
+            -0.017201457172632217,
+            0.016605600714683533,
+            0.02193695493042469,
+            -3.518914309097454e-05,
+            -0.0007879427867010236,
+            0.01382231991738081,
+            0.010160934180021286,
+            0.009423952549695969,
+            -0.010011970065534115,
+            0.0017826715484261513,
+            -8.403743413509801e-05,
+            0.02044731192290783,
+            -0.002397149335592985,
+            -0.009831644594669342,
+            -0.0035379023756831884,
+            -0.01399480551481247,
+            0.009698361158370972,
+            -0.029275408014655113,
+            -0.0052490318194031715,
+            -0.006523068994283676,
+            0.017170097678899765,
+            0.00684451824054122,
+            0.021639026701450348,
+            0.0065622697584331036,
+            0.005523439962416887,
+            -8.844755939207971e-05,
+            -0.008240078575909138,
+            0.012097470462322235,
+            -0.00809895433485508,
+            -0.007201248314231634,
+            0.006013453938066959,
+            0.004492450039833784,
+            0.016856487840414047,
+            0.008067593909800053,
+            -0.009180906228721142,
+            0.030106471851468086,
+            0.014904271811246872,
+            -0.013939923606812954,
+            0.01814228482544422,
+            -0.01826772838830948,
+            0.020776601508259773,
+            0.01853429712355137,
+            0.010239336639642715,
+            -0.015304123051464558,
+            -0.015680454671382904,
+            0.01784435659646988,
+            -0.010466703213751316,
+            -0.004657094832509756,
+            0.0018052122322842479,
+            -0.017938438802957535,
+            8.092890311672818e-06,
+            -0.011972026899456978,
+            0.015429566614329815,
+            -0.0021991836838424206,
+            -0.00012042098387610167,
+            -0.020776601508259773,
+            -0.00043831768562085927,
+            0.016903528943657875,
+            0.01615086756646633,
+            -0.0033340565860271454,
+            -0.04478337615728378,
+            -0.006178098730742931,
+            0.009855165146291256,
+            -0.00706404447555542,
+            -0.025982512161135674,
+            0.01803252287209034,
+            0.0034104986116290092,
+            -0.019428081810474396,
+            -0.01184658333659172,
+            -0.018502935767173767,
+            -0.019396722316741943,
+            -0.008820255286991596,
+            -0.004563012160360813,
+            -0.015272761695086956,
+            -0.003884832374751568,
+            0.008224397897720337,
+            -0.007213008590042591,
+            -0.006064415443688631,
+            0.004061237443238497,
+            -0.007354132831096649,
+            -0.0003518301818985492,
+            -0.010866554453969002,
+            -0.005629282910376787,
+            0.011133122257888317,
+            0.002452031010761857,
+            -0.01508459635078907,
+            -0.015366844832897186,
+            -0.011564334854483604,
+            -0.010537264868617058,
+            0.004139639902859926,
+            -0.009212266653776169,
+            -0.005080467090010643,
+            -0.020149383693933487,
+            0.0006615191232413054,
+            0.013469509780406952,
+            0.0104118213057518,
+            0.0006154578295536339,
+            0.004931502975523472,
+            0.022313285619020462,
+            -0.006189859006553888,
+            -0.0006242780946195126,
+            -9.363864592160098e-06,
+            -0.002955765463411808,
+            0.02687629871070385,
+            0.004312125034630299,
+            0.01286581251770258,
+            -0.005562641192227602,
+            0.009494515135884285,
+            -0.01975737139582634,
+            -0.0023755887523293495,
+            -0.010850873775780201,
+            -0.019773053005337715,
+            -0.011979866772890091,
+            -5.2125258662272245e-05,
+            0.004743337165564299,
+            -0.013069658540189266,
+            0.0021541023161262274,
+            -0.00467277504503727,
+            0.006734754890203476,
+            -0.01508459635078907,
+            -0.005782167427241802,
+            -0.007134606596082449,
+            -0.010662708431482315,
+            -0.02416357956826687,
+            0.018283409997820854,
+            0.02889907732605934,
+            0.001601366326212883,
+            -0.007554058451205492,
+            -0.01391640305519104,
+            -0.008177356794476509,
+            -0.004041636828333139,
+            -0.018048202618956566,
+            -0.008702651597559452,
+            -0.027926888316869736,
+            -0.013641994446516037,
+            -0.0003746158326976001,
+            0.010772472247481346,
+            0.0036065042950212955,
+            -0.018126605078577995,
+            0.005107908044010401,
+            -0.025527779012918472,
+            -0.025668902322649956,
+            -0.019428081810474396,
+            -0.017295541241765022,
+            0.0025676742661744356,
+            0.003888752544298768,
+            0.0021149013191461563,
+            0.017358262091875076,
+            0.020227786153554916,
+            0.004813899286091328,
+            -0.002424590289592743,
+            0.008592888712882996,
+            -0.014974833466112614,
+            0.0046766954474151134,
+            0.0023305073846131563,
+            -0.013422468677163124,
+            0.02358340285718441,
+            -0.011917144991457462,
+            -0.00353594240732491,
+            0.0008570348145440221,
+            0.0034575401805341244,
+            -0.028193457052111626,
+            -0.002152142347767949,
+            -0.0164958368986845,
+            0.02270529791712761,
+            -0.029651738703250885,
+            -0.0021443020086735487,
+            -0.00416316045448184,
+            -0.007895108312368393,
+            0.026578368619084358,
+            0.021356778219342232,
+            -0.0004645334556698799,
+            0.008984900079667568,
+            0.010709750466048717,
+            -0.0013063778169453144,
+            0.01869110018014908,
+            -0.017483705654740334,
+            -0.014151609502732754,
+            0.022234883159399033,
+            0.008757533505558968,
+            0.0036163046024739742,
+            -0.011242885142564774,
+            0.017562108114361763,
+            0.017468025907874107,
+            0.0059193712659180164,
+            -0.011485932394862175,
+            -0.016244949772953987,
+            0.0007933329907245934,
+            0.019773053005337715,
+            -0.012403239496052265,
+            0.0027440793346613646,
+            -0.010631348006427288,
+            0.008091114461421967,
+            -0.014708265662193298,
+            0.017750274389982224,
+            -0.027503516525030136,
+            -0.005060866475105286,
+            -0.0016092065488919616,
+            0.01795412041246891,
+            -0.008012712001800537,
+            0.015711814165115356,
+            -0.00696604186668992,
+            -0.009470994584262371,
+            0.014888591133058071,
+            -0.0005556760588660836,
+            -0.004739417228847742,
+            0.022909143939614296,
+            0.00653874920681119,
+            -0.0009614078444428742,
+            0.004884461406618357,
+            -0.018424533307552338,
+            0.025010323151946068,
+            -0.010074691846966743,
+            0.003000846831128001,
+            0.0012524762423709035,
+            0.0013710596831515431,
+            0.026844937354326248,
+            0.02634316310286522,
+            0.0057429661974310875,
+            0.008585048839449883,
+            -0.01351655088365078,
+            0.02519848942756653,
+            0.00467277504503727,
+            0.0012720768572762609,
+            -0.0009893386159092188,
+            -0.0024167499504983425,
+            -0.0017454305198043585,
+            -0.009180906228721142,
+            0.015743175521492958,
+            -0.01213667169213295,
+            -0.025935471057891846,
+            -0.009322029538452625,
+            -0.004492450039833784,
+            -0.017107374966144562,
+            0.020055299624800682,
+            0.0004699236014857888,
+            -0.003992635756731033,
+            0.011681938543915749,
+            0.007428614888340235,
+            0.0032360537443310022,
+            -0.008655610494315624,
+            0.0038671919610351324,
+            -0.007130686193704605,
+            -0.013046137988567352,
+            0.012450280599296093,
+            -0.019208556041121483,
+            -0.008984900079667568,
+            -0.008553687483072281,
+            0.009063302539288998,
+            0.010443182662129402,
+            0.00928282830864191,
+            0.015656933188438416,
+            0.004441488534212112,
+            0.00819303747266531,
+            0.012489481829106808,
+            -0.024931922554969788,
+            0.01707601360976696,
+            0.021764470264315605,
+            -0.0005096147651784122,
+            0.001671928446739912,
+            0.00045350813888944685,
+            -0.03446563705801964,
+            0.00027293790481053293,
+            0.0014034006744623184,
+            0.0077853454276919365,
+            -0.013406787998974323,
+            0.0022677856031805277,
+            0.022673936560750008,
+            -0.011109601706266403,
+            0.020996127277612686,
+            0.005970332771539688,
+            0.002479471731930971,
+            0.03343072906136513,
+            0.012230753898620605,
+            0.016339033842086792,
+            -0.0068915593437850475,
+            -0.026625411584973335,
+            0.0054371971637010574,
+            -0.018518615514039993,
+            0.009714040905237198,
+            0.0014386816183105111,
+            0.013148060999810696,
+            -0.01792275905609131,
+            -0.006025214213877916,
+            0.0027715202886611223,
+            -0.011039039120078087,
+            0.01654287800192833,
+            0.0014043806586414576,
+            0.019631927832961082,
+            0.001000609016045928,
+            -0.0028264017309993505,
+            -0.012967735528945923,
+            -0.01404968649148941,
+            0.0029773262795060873,
+            -0.009659159928560257,
+            -0.01699761115014553,
+            0.004966783802956343,
+            -0.008342001587152481,
+            0.007091485429555178,
+            -0.01587645895779133,
+            0.02044731192290783,
+            0.0005806668195873499,
+            0.00853800680488348,
+            0.015523649752140045,
+            0.0010319699067622423,
+            0.017750274389982224,
+            0.010341259650886059,
+            -0.0075501385144889355,
+            0.0062957024201750755,
+            0.0002594625111669302,
+            0.0029636058025062084,
+            0.0017895317869260907,
+            -0.01219155266880989,
+            0.0058135283179581165,
+            -0.004159240517765284,
+            0.00015165939112193882,
+            -0.006479947362095118,
+            -0.004092598333954811,
+            0.0034046184737235308,
+            -0.013406787998974323,
+            -0.00539015606045723,
+            0.002242304850369692,
+            0.001080971327610314,
+            0.02148222178220749,
+            -0.0017905118875205517,
+            0.00878889486193657,
+            0.02228192426264286,
+            -0.015570690855383873,
+            0.006703393999487162,
+            0.010059011168777943,
+            -0.004892301745712757,
+            -0.011156642809510231,
+            0.004661014769226313,
+            0.016166547313332558,
+            -0.01587645895779133,
+            -0.01131344772875309,
+            0.011501613073050976,
+            -0.021921275183558464,
+            0.004390527028590441,
+            -0.003963234834372997,
+            -0.007338452618569136,
+            0.02143518067896366,
+            -0.0012436560355126858,
+            0.032333094626665115,
+            -0.004770778119564056,
+            -0.015460927970707417,
+            -0.01930263824760914,
+            0.0012554163113236427,
+            -0.025512099266052246,
+            -0.005637123249471188,
+            -0.009188746102154255,
+            -0.006848438177257776,
+            0.00958859734237194,
+            0.0006865098839625716,
+            0.004919742234051228,
+            0.005950732156634331,
+            -0.004390527028590441,
+            0.004012236371636391,
+            -0.0031400108709931374,
+            0.007883348502218723,
+            -0.02143518067896366,
+            0.018628379330039024,
+            0.014120249077677727,
+            0.0005228451336733997,
+            -0.019741691648960114,
+            -0.008945698849856853,
+            0.020086660981178284,
+            0.04537923261523247,
+            -0.023222751915454865,
+            -0.0033085758332163095,
+            0.018330451101064682,
+            0.0038691519293934107,
+            0.0014475019415840507,
+            -0.00924362801015377,
+            -0.005570481065660715,
+            -0.04202361777424812,
+            0.0016415475402027369,
+            0.001603326410986483,
+            -0.009306349791586399,
+            0.007048364263027906,
+            -0.017812995240092278,
+            -0.007275730837136507,
+            -0.012638445943593979,
+            -0.020274827256798744,
+            -0.004610053263604641,
+            -0.010341259650886059,
+            -0.006503468379378319,
+            -0.011415370739996433,
+            0.013046137988567352,
+            -0.005774327088147402,
+            0.010082531720399857,
+            -0.004104358609765768,
+            0.01768755167722702,
+            -0.0013240183470770717,
+            0.012544362805783749,
+            -0.0005821368540637195,
+            0.009768922813236713,
+            0.00665243249386549,
+            -0.006789636332541704,
+            -0.008255759254097939,
+            -0.007667741738259792,
+            0.018659740686416626,
+            0.009768922813236713,
+            0.010498063638806343,
+            0.0014337814645841718,
+            -0.012552203610539436,
+            -0.011674097739160061,
+            5.987360782455653e-05,
+            -0.010333418846130371,
+            0.0006835698150098324,
+            0.006742595229297876,
+            0.026970380917191505,
+            -0.0033340565860271454,
+            -0.025527779012918472,
+            0.004649254493415356,
+            0.009847325272858143,
+            0.020917724817991257,
+            -0.013148060999810696,
+            0.007354132831096649,
+            -0.010498063638806343,
+            0.013108859769999981,
+            0.026311801746487617,
+            -0.009212266653776169,
+            0.018283409997820854,
+            -0.006205539684742689,
+            0.013947763480246067,
+            0.006836677901446819,
+            -0.00416316045448184,
+            -0.02963605709373951,
+            -0.008302800357341766,
+            0.0016131267184391618,
+            0.0036319850478321314,
+            -0.009369071573019028,
+            0.030874812975525856,
+            0.010482383891940117,
+            0.016088144853711128,
+            -0.009580757468938828,
+            0.010403981432318687,
+            0.00539015606045723,
+            0.019694650545716286,
+            0.012238594703376293,
+            0.00021413620561361313,
+            0.0019806374330073595,
+            -0.008498805575072765,
+            -0.045504678040742874,
+            0.00957291666418314,
+            0.006668113172054291,
+            0.02538665570318699,
+            -0.04196089506149292,
+            0.01569613441824913,
+            -0.010200135409832,
+            -0.0059899333864450455,
+            -0.003700587200000882,
+            0.007667741738259792,
+            -0.003696667030453682,
+            0.0290245208889246,
+            -0.005790007766336203,
+            0.0019355560652911663,
+            0.016684003174304962,
+            0.014073207043111324,
+            -0.007683422416448593,
+            0.01567261293530464,
+            0.023018905892968178,
+            0.01898903027176857,
+            0.013979124836623669,
+            -0.00748349679633975,
+            0.025621861219406128,
+            0.0006811197381466627,
+            -0.003218413097783923,
+            -0.008240078575909138,
+            -0.016166547313332558,
+            0.0024128297809511423,
+            -0.013116699643433094,
+            0.003659426001831889,
+            -0.00561752263456583,
+            -0.004633574280887842,
+            0.010905755683779716,
+            -0.010991998016834259,
+            0.0017454305198043585,
+            0.0018081523012369871,
+            0.007040523923933506,
+            0.0003800060076173395,
+            0.02703310176730156,
+            -0.032521262764930725,
+            0.014081047847867012,
+            -0.03258398175239563,
+            -0.01657423935830593,
+            0.006942520849406719,
+            -0.0044179679825901985,
+            0.00687195872887969,
+            -0.0195221658796072,
+            -0.017358262091875076,
+            0.00591545132920146,
+            0.01829908974468708,
+            0.01502971537411213,
+            0.00716988742351532,
+            0.006091856397688389,
+            0.004884461406618357,
+            -0.010059011168777943,
+            -0.006299622356891632,
+            -0.009557236917316914,
+            -0.00884377583861351,
+            -0.005049106199294329,
+            -0.02010234259068966,
+            0.015453087165951729,
+            -0.0070562041364610195,
+            -0.007922549732029438,
+            0.016746724024415016,
+            -0.01026285719126463,
+            -0.018424533307552338,
+            0.005539120174944401,
+            -0.0038926727138459682,
+            -0.005025585647672415,
+            -0.016213588416576385,
+            0.00922794733196497,
+            -0.0032752547413110733,
+            0.0039240336045622826,
+            -0.01309317909181118,
+            0.026672452688217163,
+            -0.006785716395825148,
+            0.001506303553469479,
+            0.011924984864890575,
+            -0.009408272802829742,
+            -0.024853520095348358,
+            -0.0021991836838424206,
+            -0.014896431006491184,
+            0.0027323190588504076,
+            -0.0028773632366210222,
+            -0.01673104427754879,
+            -0.008443924598395824,
+            0.0035967042203992605,
+            -0.02698606066405773,
+            -0.011384009383618832,
+            0.0099570881575346,
+            0.011783861555159092,
+            0.009431793354451656,
+            0.011932825669646263,
+            -0.0021011808421462774,
+            0.013148060999810696,
+            -0.029008839279413223,
+            -0.014496579766273499,
+            -0.01362631469964981,
+            -0.009549396112561226,
+            0.01412808895111084,
+            0.0017954119248315692,
+            0.007416854612529278,
+            -0.004480689764022827,
+            -0.005252952221781015,
+            -0.0011201724410057068,
+            0.011776020750403404,
+            0.028804993256926537,
+            0.0003814760420937091,
+            -0.003884832374751568,
+            -0.001314218039624393,
+            -0.011423210613429546,
+            0.028381621465086937,
+            -0.0041121989488601685,
+            -0.0072090886533260345,
+            0.03449700027704239,
+            0.01933399960398674,
+            0.027362391352653503,
+            -0.0001465142413508147,
+            -0.006025214213877916,
+            -0.0005091247148811817,
+            0.004931502975523472,
+            0.004974624142050743,
+            -0.01226995512843132,
+            -0.00842040404677391,
+            -0.004751177504658699,
+            0.03465380147099495,
+            0.0022952265571802855,
+            0.011133122257888317,
+            -0.0307493694126606,
+            0.015225720591843128,
+            -0.003577103605493903,
+            -0.007546218577772379,
+            0.003269374603405595,
+            0.036692261695861816,
+            0.02576298639178276,
+            0.0069738817401230335,
+            0.009447473101317883,
+            8.630374941276386e-05,
+            0.0398910753428936,
+            -0.02438310533761978,
+            -0.020196424797177315,
+            -0.02273665741086006,
+            -0.021717429161071777,
+            0.0016601680545136333,
+            0.007271810434758663,
+            0.003628065111115575,
+            -0.0009770882315933704,
+            0.0015259041683748364,
+            0.028585467487573624,
+            0.0015788257587701082,
+            -0.010341259650886059,
+            -0.0014308413956314325,
+            -0.034089308232069016,
+            0.013618473894894123,
+            -0.017389623448252678,
+            -0.010200135409832,
+            -0.016511518508195877,
+            -0.013006936758756638,
+            -0.025982512161135674,
+            -0.010607827454805374,
+            -0.0009599378099665046,
+            0.015970543026924133,
+            0.010004129260778427,
+            -0.0019120353972539306,
+            0.005613602697849274,
+            -0.023834289982914925,
+            -0.006703393999487162,
+            0.020917724817991257,
+            0.01567261293530464,
+            0.015790216624736786,
+            0.0046531748957931995,
+            0.01622927002608776,
+            -0.0003020937438122928,
+            -0.015923500061035156,
+            -0.0020913805346935987,
+            0.002767600119113922,
+            0.006413305643945932,
+            0.017091695219278336,
+            0.012222914025187492,
+            -0.018769502639770508,
+            0.0071267662569880486,
+            -0.012795249931514263,
+            0.03139226883649826,
+            -0.006197699345648289,
+            0.01205826923251152,
+            0.009682680480182171,
+            0.0032830950804054737,
+            -0.012489481829106808,
+            -0.006158498115837574,
+            -0.00479429867118597,
+            0.01569613441824913,
+            -0.00831064023077488,
+            -0.011062560603022575,
+            -0.005676324479281902,
+            0.009729721583425999,
+            -0.005535200238227844,
+            -0.0010349099757149816,
+            -0.015609892085194588,
+            -0.014347615651786327,
+            0.0030753291212022305,
+            -0.012826611287891865,
+            -0.0010711710201576352,
+            0.019036071375012398,
+            -0.005033425521105528,
+            -0.0004390527028590441,
+            0.014684745110571384,
+            0.016589920967817307,
+            0.004649254493415356,
+            0.018926307559013367,
+            -0.009549396112561226,
+            0.0022932663559913635,
+            -0.021623345091938972,
+            -0.0024716316256672144,
+            -0.032364457845687866,
+            0.019647609442472458,
+            0.0017836516490206122,
+            -0.020070981234312057,
+            0.010944956913590431,
+            -0.0016239070100709796,
+            -0.009165225550532341,
+            -0.0099727688357234,
+            0.002452031010761857,
+            -0.024179259315133095,
+            0.024147899821400642,
+            -0.016048945486545563,
+            0.008906497620046139,
+            0.01161921676248312,
+            0.041333675384521484,
+            0.0038573916535824537,
+            -0.00747565645724535,
+            0.021152932196855545,
+            -0.0002717128663789481,
+            0.02982422336935997,
+            -0.001121152425184846,
+            -0.012081789784133434,
+            -0.022689616307616234,
+            -0.005237271543592215,
+            -0.015202200040221214,
+            -0.014112408272922039,
+            0.0147239463403821,
+            -0.0195221658796072,
+            -0.015217880718410015,
+            -0.010004129260778427,
+            -0.019741691648960114,
+            -0.011509452946484089,
+            0.025982512161135674,
+            0.015758855268359184,
+            -0.011705459095537663,
+            -0.006425065919756889,
+            -0.0055077592842280865,
+            -0.02711150422692299,
+            0.021952634677290916,
+            0.008012712001800537,
+            -0.01983577385544777,
+            -0.0032772149424999952,
+            -0.009768922813236713,
+            0.004378766752779484,
+            0.01588430069386959,
+            -0.0001214010117109865,
+            0.05789223685860634,
+            -0.008804574608802795,
+            -0.0015945062041282654,
+            -0.020008258521556854,
+            0.009298508986830711,
+            -0.01575101539492607,
+            -0.006374104414135218,
+            -0.026672452688217163,
+            -0.008255759254097939,
+            -0.005119668319821358,
+            -0.021105891093611717,
+            -0.0022481849882751703,
+            -0.008287119679152966,
+            0.0057547264732420444,
+            -0.01280309073626995,
+            0.0008050933247432113,
+            -0.01665264181792736,
+            -0.00624866085126996,
+            -0.012089629657566547,
+            -0.012371878139674664,
+            0.01925559714436531,
+            -0.028303219005465508,
+            -0.0033869780600070953,
+            0.008153836242854595,
+            0.01353223156183958,
+            0.006268261466175318,
+            -0.021043168380856514,
+            -0.012614925391972065,
+            -0.011917144991457462,
+            -0.018894946202635765,
+            0.010733271017670631,
+            -0.010741110891103745,
+            0.013140220195055008,
+            -0.03186268359422684,
+            -0.012607084587216377,
+            -0.0019522164948284626,
+            0.019976899027824402,
+            0.01753074675798416,
+            -0.0004733536916319281,
+            -0.0059193712659180164,
+            -0.014002645388245583,
+            0.02278370037674904,
+            -0.013877201825380325,
+            -0.032678067684173584,
+            -0.0013494990998879075,
+            0.002218784298747778,
+            -0.00104079011362046,
+            -0.007816705852746964,
+            0.0027538796421140432,
+            0.004214121960103512,
+            0.004092598333954811,
+            -0.007546218577772379,
+            -0.0016464476939290762,
+            -0.01508459635078907,
+            0.0047668581828475,
+            -3.659793583210558e-05,
+            0.029730141162872314,
+            -0.0035339822061359882,
+            0.008992739953100681,
+            -0.02182719111442566,
+            0.0007546218694187701,
+            -0.01317158155143261,
+            -0.005782167427241802,
+            0.000176037588971667,
+            0.003330136416479945,
+            0.01956920698285103,
+            -0.025794345885515213,
+            -0.013038297183811665,
+            0.01205826923251152,
+            -0.019929856061935425,
+            -0.03572007268667221,
+            0.002736239228397608,
+            -0.029463572427630424,
+            -0.0013053978327661753,
+            -0.003777029225602746,
+            -0.015703974291682243,
+            -0.026217719539999962,
+            0.0060722557827830315,
+            -0.005366635508835316,
+            -0.005786087363958359,
+            0.003516341792419553,
+            -0.015233561396598816,
+            -0.015531489625573158,
+            0.005151029210537672,
+            0.007346292492002249,
+            -0.006503468379378319,
+            0.008561528287827969,
+            -0.0238186102360487,
+            0.011274246498942375,
+            -0.008145995438098907,
+            0.0037358682602643967,
+            0.005135348532348871,
+            0.026123635470867157,
+            0.008561528287827969,
+            -0.004000475630164146,
+            0.009306349791586399,
+            0.005472478456795216,
+            -0.010333418846130371,
+            0.011070400476455688,
+            0.02400677464902401,
+            0.0005184350302442908,
+            0.0032340935431420803,
+            -0.002973406109958887,
+            0.005746886134147644,
+            -0.001685648807324469,
+            -0.007612860295921564,
+            0.020933406427502632,
+            0.01593918167054653,
+            -0.019239917397499084,
+            0.0062369005754590034,
+            -0.020039619877934456,
+            0.010976317338645458,
+            -0.028742272406816483,
+            0.0005703765200451016,
+            0.005840969271957874,
+            -0.0050020646303892136,
+            0.011979866772890091,
+            0.006860198453068733,
+            0.03151771053671837,
+            0.026280440390110016,
+            -0.0018875346286222339,
+            0.0048178196884691715,
+            -0.02002394013106823,
+            0.01280309073626995,
+            0.009118183515965939,
+            0.010066851042211056,
+            -0.010694069787859917,
+            0.00791470892727375,
+            -0.00878889486193657,
+            0.014606342650949955,
+            -0.00260295532643795,
+            -0.014214331284165382,
+            0.006350583862513304,
+            -0.006287862081080675,
+            0.014441697858273983,
+            0.014081047847867012,
+            -0.013148060999810696,
+            -0.026923339813947678,
+            -0.006162418518215418,
+            0.018095243722200394,
+            -0.023912692442536354,
+            0.02443014644086361,
+            0.006464267149567604,
+            0.02265825681388378,
+            -0.009988449513912201,
+            -0.0030655288137495518,
+            -0.001984557369723916,
+            0.002657837001606822,
+            -0.00685235857963562,
+            0.00014345165982376784,
+            0.02243872918188572,
+            -0.009510194882750511,
+            0.005601841956377029,
+            0.012175872921943665,
+            -0.0016052864957600832,
+            0.03622184693813324,
+            0.018769502639770508,
+            0.024978963658213615,
+            -0.0017366103129461408,
+            0.010584305971860886,
+            0.0008266539080068469,
+            0.010537264868617058,
+            0.013845840469002724,
+            -0.010984158143401146,
+            0.0008913357742130756,
+            -0.014347615651786327,
+            0.017028972506523132,
+            0.024665353819727898,
+            0.009463153779506683,
+            0.0002709778491407633,
+            0.019506484270095825,
+            0.023191390559077263,
+            0.0031419708393514156,
+            -0.00707188481464982,
+            -0.007753984536975622,
+            0.017091695219278336,
+            0.01420649141073227,
+            0.007240449544042349,
+            0.0054254368878901005,
+            -0.007189488038420677,
+            -0.006036974955350161,
+            0.019192876294255257,
+            -0.019804412499070168,
+            -0.0018150125397369266,
+            0.0015288442373275757,
+            -0.04042420908808708,
+            0.010286377742886543,
+            -0.00031115900492295623,
+            -0.014669064432382584,
+            0.011180163361132145,
+            0.004072997719049454,
+            -0.010976317338645458,
+            -0.010176614858210087,
+            -0.016448795795440674,
+            -0.006879799067974091,
+            0.007593259681016207,
+            0.0024324303958564997,
+            0.008506646379828453,
+            -0.01050590444356203,
+            -0.005884090438485146,
+            0.019961217418313026,
+            0.003396778367459774,
+            0.0062369005754590034,
+            -0.0001754250843077898
+          ],
+          "keyphrases": [
+            "scaling up model capacity",
+            "training efficiency",
+            "sparsely gated mixture-of-experts",
+            "expert choice method",
+            "downstream performance"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "3639a96c-e9a4-45ed-a3ac-1e18b2cfe8d4",
+        "properties": {
+          "page_content": "2 Related Work\n\nScaling: Various approaches have been proposed to scale up neural network capacity to improve performance. Recent works have successfully scaled models to billions of parameters via various forms of model parallelism [2, 21, 26, 27, 33]. Model parallelism [30] splits weights and tensors across multiple cores while pipeline parallelism [18, 24] splits different layers across devices with micro-batches pipelined to the different layers. To enable continued scaling of neural networks, improving model training and serving efficiency has become a critical research area.\n\nConditional Computation: Computation decisions can be made dynamically based on the input [23, 25]. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation, by activating certain parameters and computation on demand, on a per-example or per-token basis [3]. Conditional convolution layers [1] with task-specific gating has been used to combat catastrophic forgetting when a sequence of learning problems are optimized. The gating decisions may be binary or sparse and continuous, stochastic or deterministic.\n\nMixture of Experts: Sparsely-gated MoE [31] is the first model to demonstrate massive improvements in model capacity, training time, or model quality with gating. Switch Transformer [10] simplifies the gating by selecting only the top expert per token using a softmax over the hidden state and demonstrates better scaling than previous work. All the prior work requires an auxiliary loss to explicitly encourage balancing. This loss term has to be carefully weighted to not overwhelm the primary loss. However, auxiliary loss does not guarantee balancing and a hard capacity factor has to be imposed. As a result, many tokens can still be unprocessed by the MoE layer. Hard MoE [12] with a single decoding layer can be efficiently trained to good effect on large scale hashtag prediction tasks.\n\nBase Layers [22] formulate a linear assignment that maximizes token-expert affinities while ensuring each expert receives an equal number of tokens. Hash layers [8, 28] devise hashing techniques on input tokens. However, the evaluations are limited to pre-training perplexity. THOR [? ] randomly activates experts during training and inference and is trained with a consistency regularization loss.\n\nTHOR has demonstrated strong performance on translation tasks. Different from these prior works, our method is a learnt method that enables heterogeneous MoE and effectively improves downstream fine-tuning performance.\n\n",
+          "embedding": [
+            -0.013259558007121086,
+            -0.007197607308626175,
+            -0.015822459012269974,
+            0.013052377849817276,
+            0.029680538922548294,
+            0.02572108618915081,
+            -0.01866159960627556,
+            0.040392544120550156,
+            -0.039103418588638306,
+            0.017602676525712013,
+            0.020380431786179543,
+            -0.012024147436022758,
+            0.0011442891554906964,
+            0.006088807247579098,
+            0.009645408019423485,
+            -0.014533335343003273,
+            -0.004703766666352749,
+            0.0049493140541017056,
+            0.02260570414364338,
+            -0.03012559376657009,
+            -0.008847379125654697,
+            -0.007220627274364233,
+            -0.03612615540623665,
+            -0.01026694942265749,
+            0.03443801775574684,
+            0.0267186239361763,
+            0.03146075829863548,
+            -0.011226118542253971,
+            -0.04496586322784424,
+            0.025629006326198578,
+            0.015055122785270214,
+            0.009300106205046177,
+            -0.016912074759602547,
+            -0.01592988520860672,
+            -0.0355122871696949,
+            -0.04048462212085724,
+            0.05073622614145279,
+            0.013781346380710602,
+            -0.01992003060877323,
+            0.02774685248732567,
+            0.021807676181197166,
+            0.0150397764518857,
+            -0.03502119332551956,
+            -0.02219134382903576,
+            0.00023379754566121846,
+            0.0007141016540117562,
+            -0.032136011868715286,
+            -0.009069905616343021,
+            -0.008317917585372925,
+            -0.00694822333753109,
+            0.028268640860915184,
+            -0.04450546205043793,
+            0.01573037914931774,
+            0.012584302574396133,
+            -0.038274697959423065,
+            0.039932142943143845,
+            0.013228864409029484,
+            0.005678283050656319,
+            0.053621407598257065,
+            0.03726181387901306,
+            0.011333545669913292,
+            0.006449454929679632,
+            -0.04613221064209938,
+            0.002123601036146283,
+            0.010443436913192272,
+            -0.008847379125654697,
+            -0.00621158117428422,
+            0.0015471403021365404,
+            0.0041819787584245205,
+            0.010289969854056835,
+            -0.0367400236427784,
+            -0.0014166932087391615,
+            -0.011893700808286667,
+            -0.003069342114031315,
+            0.010834777727723122,
+            0.001008087070658803,
+            0.019367549568414688,
+            0.039011336863040924,
+            0.03511327505111694,
+            -0.008678564801812172,
+            -0.01622147299349308,
+            0.03317958861589432,
+            -0.002503432100638747,
+            0.016574447974562645,
+            0.02119380794465542,
+            -0.03713903948664665,
+            -0.0017399332718923688,
+            -0.06629779189825058,
+            -0.018983881920576096,
+            -0.01323653757572174,
+            -0.02991073951125145,
+            0.0056015495210886,
+            -0.01734178327023983,
+            -0.0033398279920220375,
+            -0.020273005589842796,
+            0.04646984115242958,
+            -0.018446745350956917,
+            -0.013482085429131985,
+            -0.032872654497623444,
+            0.004730623681098223,
+            -0.01812446489930153,
+            0.03631031885743141,
+            -0.0027451429050415754,
+            -0.0076541719026863575,
+            0.01571503095328808,
+            0.009092926047742367,
+            -0.05006096884608269,
+            0.034622177481651306,
+            -0.00013188579760026187,
+            0.074830561876297,
+            0.012898909859359264,
+            -0.033210281282663345,
+            0.00029134770738892257,
+            0.04054601117968559,
+            -0.03152214363217354,
+            -0.02452404424548149,
+            -0.04149750620126724,
+            -0.03597268834710121,
+            0.001932726358063519,
+            -0.028406761586666107,
+            0.024416616186499596,
+            0.00759278517216444,
+            -0.0010752289090305567,
+            -0.012430835515260696,
+            -0.05288476496934891,
+            0.050889693200588226,
+            -0.013812039978802204,
+            -0.01692742109298706,
+            -0.005755016580224037,
+            -0.02243689075112343,
+            -0.011356566101312637,
+            0.023541854694485664,
+            -0.032872654497623444,
+            -0.05638381466269493,
+            0.01705019548535347,
+            0.016160085797309875,
+            0.051380787044763565,
+            0.02021161839365959,
+            -0.018738333135843277,
+            -0.010044422000646591,
+            0.03278057277202606,
+            -0.005781873129308224,
+            0.02360324189066887,
+            -0.022022530436515808,
+            -0.028821121901273727,
+            0.013942486606538296,
+            -0.033455830067396164,
+            0.06227695196866989,
+            -0.01651306077837944,
+            0.023787401616573334,
+            -0.013474412262439728,
+            0.04542626440525055,
+            -0.03324097394943237,
+            0.03235086426138878,
+            -0.029388951137661934,
+            0.021500742062926292,
+            0.005755016580224037,
+            -0.00864787120372057,
+            0.0028065296355634928,
+            0.00858648493885994,
+            -0.008279550820589066,
+            -0.020902220159769058,
+            0.005098944529891014,
+            0.030800849199295044,
+            0.0026933476328849792,
+            0.024278495460748672,
+            0.04907878115773201,
+            0.02077944576740265,
+            -0.052915457636117935,
+            0.029726577922701836,
+            0.029695885255932808,
+            0.020426472648978233,
+            0.0444747656583786,
+            0.007554418407380581,
+            0.010604577139019966,
+            0.006587575655430555,
+            0.014993736520409584,
+            0.0070403036661446095,
+            -0.01655910164117813,
+            0.03879648447036743,
+            -0.0241403765976429,
+            -0.0031268922612071037,
+            0.024784937500953674,
+            0.0015365893486887217,
+            0.036617252975702286,
+            0.005482612177729607,
+            0.013512779027223587,
+            0.02136262133717537,
+            0.00933079980313778,
+            0.027808239683508873,
+            0.02021161839365959,
+            0.030171632766723633,
+            0.020150231197476387,
+            -0.024401269853115082,
+            0.03198254480957985,
+            -0.056813523173332214,
+            -0.015945231541991234,
+            -0.009323126636445522,
+            0.028283987194299698,
+            -0.006169377826154232,
+            -0.01812446489930153,
+            -0.017894264310598373,
+            0.008049349300563335,
+            0.002048785798251629,
+            -0.009676100686192513,
+            0.008931785821914673,
+            0.033916231244802475,
+            0.026288915425539017,
+            0.024002255871891975,
+            -0.037937067449092865,
+            -0.0052025350742042065,
+            -0.010903838090598583,
+            -0.013374658301472664,
+            -0.034990500658750534,
+            -0.00218306970782578,
+            0.009461247362196445,
+            0.0012622670037671924,
+            0.009630060754716396,
+            -0.002929303329437971,
+            0.0027355512138456106,
+            0.03015628643333912,
+            -0.0046423799358308315,
+            0.045610424131155014,
+            -0.014425908215343952,
+            -0.0034817850682884455,
+            -0.020503206178545952,
+            -0.006576065439730883,
+            0.003627578727900982,
+            0.01622147299349308,
+            0.011824640445411205,
+            0.0031134639866650105,
+            0.012661036103963852,
+            0.004577156621962786,
+            -0.004465892910957336,
+            0.04435199499130249,
+            -0.016697220504283905,
+            0.01414966769516468,
+            -0.011064978316426277,
+            0.008195143193006516,
+            0.003395459847524762,
+            -0.005912320222705603,
+            0.041466813534498215,
+            -0.047851044684648514,
+            -0.0132749043405056,
+            0.020718058571219444,
+            0.0033820313401520252,
+            -0.03176769241690636,
+            0.012001127935945988,
+            -0.0009505368652753532,
+            -0.01609869860112667,
+            -0.01435684785246849,
+            -0.02364928089082241,
+            -0.005543999373912811,
+            0.00993699487298727,
+            0.00022012938279658556,
+            0.05666005611419678,
+            0.018078425899147987,
+            -0.01528532337397337,
+            -0.01978190988302231,
+            -0.02140866219997406,
+            0.03962520509958267,
+            -0.004730623681098223,
+            -0.035819221287965775,
+            0.013934813439846039,
+            -0.012039494700729847,
+            -0.01883041486144066,
+            0.030938968062400818,
+            0.01470982190221548,
+            -0.003180605825036764,
+            -0.019029920920729637,
+            0.008862725459039211,
+            -0.05260852351784706,
+            -0.005716649815440178,
+            0.0007275299867615104,
+            0.06727997958660126,
+            -0.024984445422887802,
+            -0.010934531688690186,
+            0.0050490680150687695,
+            0.0011596358381211758,
+            0.02364928089082241,
+            -0.00858648493885994,
+            -0.003474111668765545,
+            -0.021961143240332603,
+            0.03003351204097271,
+            -0.008010982535779476,
+            0.005578529555350542,
+            -0.026979517191648483,
+            -0.017955651506781578,
+            -0.04849560558795929,
+            -0.02978796511888504,
+            -0.014510314911603928,
+            0.012607323005795479,
+            -0.05273129791021347,
+            0.017648717388510704,
+            0.026856742799282074,
+            -0.021086379885673523,
+            0.009668427519500256,
+            0.012208308093249798,
+            0.027286451309919357,
+            -0.034407325088977814,
+            -0.05260852351784706,
+            -0.02941964380443096,
+            0.014157340861856937,
+            -0.016942769289016724,
+            -0.05512538552284241,
+            0.0036122319288551807,
+            0.012868217192590237,
+            0.015546217560768127,
+            -0.004849560558795929,
+            -0.0013322862796485424,
+            0.02133192867040634,
+            -0.006986590102314949,
+            0.0062768044881522655,
+            -0.01721900887787342,
+            -0.00873227883130312,
+            0.011793946847319603,
+            0.010282295756042004,
+            -0.000877640035469085,
+            -0.03738458827137947,
+            -0.013466738164424896,
+            0.0236953217536211,
+            0.018600212410092354,
+            -0.07317311316728592,
+            0.011487012729048729,
+            -0.027501305565238,
+            -0.007170750759541988,
+            -0.0042126718908548355,
+            0.03302612155675888,
+            -0.019060615450143814,
+            0.016083352267742157,
+            -0.0020583774894475937,
+            -0.02808448113501072,
+            -0.027639426290988922,
+            -0.006537698674947023,
+            -0.03410039097070694,
+            -0.03207462653517723,
+            -0.028468148782849312,
+            -0.004178142175078392,
+            -0.03143006190657616,
+            -0.029220137745141983,
+            -0.024508696049451828,
+            0.0023173533845692873,
+            -0.00552865257486701,
+            0.017357129603624344,
+            0.01899922825396061,
+            -0.03373207151889801,
+            -0.045364875346422195,
+            0.013190497644245625,
+            -0.006587575655430555,
+            0.01259197574108839,
+            0.0009654040331952274,
+            -0.012062514200806618,
+            0.016282860189676285,
+            -0.003094280604273081,
+            -0.03015628643333912,
+            -0.046193599700927734,
+            -0.013297924771904945,
+            0.014610068872570992,
+            0.008095390163362026,
+            -0.0002774397435132414,
+            0.010006055235862732,
+            -0.04686885327100754,
+            -0.024171069264411926,
+            0.01066596433520317,
+            -0.04008561000227928,
+            -0.024002255871891975,
+            -0.0045694829896092415,
+            -0.025444846600294113,
+            -0.006391904782503843,
+            -0.006844632793217897,
+            -0.0222373828291893,
+            0.03729250654578209,
+            0.018262585625052452,
+            -0.00933079980313778,
+            0.02941964380443096,
+            0.012162268161773682,
+            0.020672019571065903,
+            -0.01617543213069439,
+            -0.04640845209360123,
+            0.030048860237002373,
+            -0.019444283097982407,
+            -0.03098500892519951,
+            8.272836566902697e-05,
+            -0.022252731025218964,
+            0.007792292162775993,
+            0.00012229410640429705,
+            0.015945231541991234,
+            -0.006422598380595446,
+            -0.04573319852352142,
+            0.02932756394147873,
+            0.004872580524533987,
+            -0.04956987500190735,
+            -0.0022137630730867386,
+            -0.04791243001818657,
+            -0.0065607186406850815,
+            0.030340446159243584,
+            -0.004914783872663975,
+            -0.027977053076028824,
+            -0.03459148481488228,
+            -0.0015605686930939555,
+            -0.011433299630880356,
+            0.014625415205955505,
+            -0.013643225654959679,
+            0.013727632351219654,
+            -0.008041676133871078,
+            -0.0188764538615942,
+            -0.010719677433371544,
+            0.015776418149471283,
+            0.03759944066405296,
+            -0.024155722931027412,
+            -0.022682437673211098,
+            -0.03572714328765869,
+            -0.03953312709927559,
+            0.013221191242337227,
+            0.024201761931180954,
+            0.01867694780230522,
+            -0.010972898453474045,
+            0.018078425899147987,
+            0.0068369596265256405,
+            -0.014878636226058006,
+            -0.025153258815407753,
+            0.0029810986015945673,
+            0.05441943556070328,
+            0.00807236973196268,
+            -0.00013368423969950527,
+            0.002990690292790532,
+            0.009798875078558922,
+            0.017188316211104393,
+            0.005555509123951197,
+            -0.014280114322900772,
+            0.04373812675476074,
+            -0.014433581382036209,
+            0.039686594158411026,
+            -0.01065829023718834,
+            -0.02070271223783493,
+            -0.03634101152420044,
+            0.023679975420236588,
+            0.005313798785209656,
+            -0.03443801775574684,
+            -0.012361775152385235,
+            0.029358256608247757,
+            0.005720486398786306,
+            -0.010773390531539917,
+            0.012599649839103222,
+            -0.035358820110559464,
+            -0.029557764530181885,
+            -0.022713132202625275,
+            -0.037108346819877625,
+            -0.00947659369558096,
+            -0.03453009948134422,
+            -0.06365815550088882,
+            -0.004247202072292566,
+            -0.061294760555028915,
+            -0.014878636226058006,
+            0.02820725366473198,
+            -0.0006987549131736159,
+            0.0162982065230608,
+            -0.00065223517594859,
+            -0.007700212299823761,
+            -0.03833608329296112,
+            0.005206371657550335,
+            0.04174305126070976,
+            0.01746455766260624,
+            -0.0380905345082283,
+            -0.017280396074056625,
+            -0.009031538851559162,
+            -0.04263316094875336,
+            -0.029342910274863243,
+            0.009031538851559162,
+            -0.013643225654959679,
+            6.396460958058015e-05,
+            -0.03941035270690918,
+            0.003128810552880168,
+            0.006840796209871769,
+            0.018109118565917015,
+            0.04444407299160957,
+            0.022129956632852554,
+            -0.06040465459227562,
+            0.007987963035702705,
+            -0.003629497019574046,
+            0.027240412309765816,
+            -0.014072933234274387,
+            0.018017038702964783,
+            -0.0006474393303506076,
+            -0.007009610068053007,
+            -0.02662654221057892,
+            0.009054559282958508,
+            -0.0012507569044828415,
+            -0.011878354474902153,
+            -0.0038596978411078453,
+            0.004316262435168028,
+            0.00914663914591074,
+            0.03182907775044441,
+            0.010282295756042004,
+            0.009537980891764164,
+            -0.04082224890589714,
+            -0.012860543094575405,
+            -0.01130285207182169,
+            0.04042323678731918,
+            0.009246393106877804,
+            -0.034990500658750534,
+            0.007316544186323881,
+            0.03253502771258354,
+            0.028406761586666107,
+            0.028115173801779747,
+            -0.011072651483118534,
+            0.012730096466839314,
+            0.020349739119410515,
+            -0.07004238665103912,
+            -0.005758853163570166,
+            -0.0069904266856610775,
+            -0.0037503524217754602,
+            0.0037407607305794954,
+            -0.02289729192852974,
+            -0.00019531087309587747,
+            0.042541082948446274,
+            -0.016896728426218033,
+            -0.00151644682046026,
+            -0.011003591120243073,
+            0.029235484078526497,
+            0.002560982247814536,
+            0.004086061846464872,
+            -0.012607323005795479,
+            0.058470968157052994,
+            -0.017863571643829346,
+            0.013343964703381062,
+            0.006077297497540712,
+            -0.01306772418320179,
+            0.024861671030521393,
+            -0.021715596318244934,
+            -0.007051813416182995,
+            0.005056741181761026,
+            0.0094382269307971,
+            -0.017157621681690216,
+            -0.003107708878815174,
+            -0.0026991027407348156,
+            -0.01741851679980755,
+            0.010811757296323776,
+            -0.0489560067653656,
+            -0.03471425920724869,
+            -0.0022463747300207615,
+            0.01005976926535368,
+            -0.020027456805109978,
+            0.010305316187441349,
+            0.014326154254376888,
+            0.007934249006211758,
+            0.055739253759384155,
+            -0.03143006190657616,
+            -0.0016229146858677268,
+            -0.003309134393930435,
+            -0.023925522342324257,
+            -0.028636962175369263,
+            -0.023004719987511635,
+            -0.0056130592711269855,
+            -0.015822459012269974,
+            -0.04386089742183685,
+            -0.008693911135196686,
+            -0.021439354866743088,
+            0.021270541474223137,
+            -0.04094502329826355,
+            0.016912074759602547,
+            0.008724604733288288,
+            0.0006124296924099326,
+            -0.009008519351482391,
+            -0.01995072327554226,
+            -0.0042625488713383675,
+            0.029757272452116013,
+            0.032166704535484314,
+            0.018861107528209686,
+            0.034867726266384125,
+            -0.005636079702526331,
+            -0.0231888797134161,
+            -0.013581839390099049,
+            0.03572714328765869,
+            -0.00854811817407608,
+            0.020058151334524155,
+            0.004980007652193308,
+            -0.0006061950698494911,
+            -0.0224061980843544,
+            -0.0077270688489079475,
+            0.03646378591656685,
+            -0.02555227279663086,
+            0.014425908215343952,
+            0.05122731998562813,
+            0.032749880105257034,
+            0.0006071542156860232,
+            -0.008640198037028313,
+            -0.01472516916692257,
+            0.005919993855059147,
+            -0.021792329847812653,
+            0.03198254480957985,
+            0.01987398974597454,
+            -0.001803238526917994,
+            -0.010535516776144505,
+            0.01325188484042883,
+            -0.030217673629522324,
+            0.012469202280044556,
+            0.006668145768344402,
+            0.003606477053835988,
+            -0.034867726266384125,
+            -0.02136262133717537,
+            -0.002419025171548128,
+            -0.0007759680738672614,
+            -0.01929081603884697,
+            -0.015553891658782959,
+            -0.02286659926176071,
+            0.002154294401407242,
+            -0.006821612827479839,
+            0.004281732253730297,
+            0.016697220504283905,
+            0.02837606891989708,
+            -0.0006138684111647308,
+            0.03701626509428024,
+            0.00303289364092052,
+            0.023879481479525566,
+            0.004945477470755577,
+            -0.003512478433549404,
+            -0.0070403036661446095,
+            -0.011755580082535744,
+            -0.014487294480204582,
+            0.057273924350738525,
+            0.017019502818584442,
+            0.05957593023777008,
+            -0.011064978316426277,
+            -0.010819431394338608,
+            0.030938968062400818,
+            0.004316262435168028,
+            -0.020733406767249107,
+            0.008670891635119915,
+            -0.016160085797309875,
+            -0.02871369570493698,
+            -0.0027470611967146397,
+            -0.03799845650792122,
+            -0.0188764538615942,
+            -0.022621052339673042,
+            -0.018308626487851143,
+            0.004684583283960819,
+            0.0196591354906559,
+            0.016436327248811722,
+            -0.00016737506666686386,
+            -0.011763254180550575,
+            0.031583528965711594,
+            0.019306162372231483,
+            0.028928549960255623,
+            -0.011809294112026691,
+            0.007331890985369682,
+            -0.01576107181608677,
+            0.0052831051871180534,
+            0.019137348979711533,
+            -0.01253058947622776,
+            0.009453574195504189,
+            -0.0008905887953005731,
+            0.011180078610777855,
+            0.03821330890059471,
+            -0.010036748833954334,
+            -0.03953312709927559,
+            -0.006280641537159681,
+            -0.014778882265090942,
+            0.007278177421540022,
+            0.0067065125331282616,
+            -0.020932912826538086,
+            0.012254348024725914,
+            0.007730905432254076,
+            -0.01479422952979803,
+            0.02128588780760765,
+            -0.007205280475318432,
+            -0.027624079957604408,
+            0.03468356654047966,
+            -0.014134320430457592,
+            0.01787891797721386,
+            -0.017986344173550606,
+            -0.0028755899984389544,
+            -0.025828514248132706,
+            0.022559665143489838,
+            0.01751059666275978,
+            -0.027317145839333534,
+            0.022989371791481972,
+            0.04564111679792404,
+            0.020150231197476387,
+            0.0007615805370733142,
+            0.012415489181876183,
+            0.0057665263302624226,
+            -0.036494478583335876,
+            -0.02057993970811367,
+            -0.006944386754184961,
+            -0.0027950196526944637,
+            -0.016574447974562645,
+            0.030217673629522324,
+            -0.03824400156736374,
+            -0.0031518307514488697,
+            -0.0052025350742042065,
+            -0.024078989401459694,
+            0.0055823661386966705,
+            0.009069905616343021,
+            0.003951778169721365,
+            0.017142275348305702,
+            -0.014172687195241451,
+            0.03020232729613781,
+            0.03566575422883034,
+            0.006322844885289669,
+            -0.017848225310444832,
+            0.011256812140345573,
+            -0.024508696049451828,
+            0.0048457239754498005,
+            0.043001484125852585,
+            0.014072933234274387,
+            0.008494404144585133,
+            -0.020426472648978233,
+            -0.02584386058151722,
+            0.007235974073410034,
+            0.008709258399903774,
+            0.04198860004544258,
+            0.02630426175892353,
+            0.03606477007269859,
+            0.009369166567921638,
+            -0.01199345476925373,
+            -0.013819713145494461,
+            -0.0049493140541017056,
+            -0.023265613242983818,
+            0.021761635318398476,
+            -0.02228342369198799,
+            0.01717296987771988,
+            0.012039494700729847,
+            -0.028759736567735672,
+            0.00017528820899315178,
+            0.01576107181608677,
+            0.03575783595442772,
+            0.04361535236239433,
+            0.023925522342324257,
+            0.0032765227369964123,
+            0.039686594158411026,
+            -0.04493517056107521,
+            -0.03716973215341568,
+            0.00753907160833478,
+            0.017571983858942986,
+            0.009154313243925571,
+            0.0058164033107459545,
+            0.03290334716439247,
+            -0.00947659369558096,
+            -0.003376276232302189,
+            0.012024147436022758,
+            0.02074875310063362,
+            0.002031520940363407,
+            -0.00429324246942997,
+            -0.008095390163362026,
+            0.02289729192852974,
+            0.021086379885673523,
+            0.03937966004014015,
+            -0.020058151334524155,
+            0.01780218444764614,
+            0.0009936995338648558,
+            -0.02552158012986183,
+            0.02559831365942955,
+            0.00021701208606828004,
+            -0.04530349001288414,
+            0.03339444100856781,
+            0.017249703407287598,
+            0.015108836814761162,
+            0.011786273680627346,
+            -0.006131011061370373,
+            0.012553608976304531,
+            0.030524607747793198,
+            -0.0031019540037959814,
+            0.0011423707474023104,
+            0.03115382231771946,
+            -0.003311052918434143,
+            -0.004807357210665941,
+            0.020886873826384544,
+            0.004887927323579788,
+            -0.013167478144168854,
+            0.01867694780230522,
+            -0.012169941328465939,
+            0.044075753539800644,
+            -0.024692857638001442,
+            0.006760226096957922,
+            0.010205562226474285,
+            -0.01883041486144066,
+            -0.007293524220585823,
+            -0.0013188580051064491,
+            -0.022268077358603477,
+            0.004500423092395067,
+            0.018155159428715706,
+            -0.012331082485616207,
+            0.0295731108635664,
+            -0.028606269508600235,
+            0.0012430836213752627,
+            0.015945231541991234,
+            -0.026933476328849792,
+            0.02767011895775795,
+            0.013006336987018585,
+            0.02713298425078392,
+            -0.009576347656548023,
+            -0.014456601813435555,
+            0.04797381907701492,
+            0.0563531219959259,
+            0.0060504404827952385,
+            -0.008578810840845108,
+            0.013742979615926743,
+            0.0003174850717186928,
+            -0.02360324189066887,
+            0.024048294872045517,
+            -0.03572714328765869,
+            0.0162982065230608,
+            0.03107708878815174,
+            -0.014617742039263248,
+            -0.014095953665673733,
+            -0.034622177481651306,
+            -0.0052831051871180534,
+            0.00429324246942997,
+            0.004381485749036074,
+            0.0036966388579458,
+            -0.025659700855612755,
+            -0.015599931590259075,
+            0.022252731025218964,
+            0.005379022099077702,
+            0.0010157604701817036,
+            -0.025076525285840034,
+            0.015530871227383614,
+            -0.01617543213069439,
+            -0.04162028059363365,
+            -0.006330518051981926,
+            -0.012492222711443901,
+            0.025214646011590958,
+            -6.204627425177023e-05,
+            -0.03477564454078674,
+            -0.006718022748827934,
+            -0.005191024858504534,
+            -0.009054559282958508,
+            0.044536154717206955,
+            0.023741360753774643,
+            0.03812123090028763,
+            -0.02170024812221527,
+            -0.03141471743583679,
+            0.02460077777504921,
+            -0.03489841893315315,
+            0.030770154669880867,
+            -0.0032439110800623894,
+            0.007186097092926502,
+            -0.0052025350742042065,
+            0.039348967373371124,
+            -0.012146921828389168,
+            0.008571137674152851,
+            -0.03588061034679413,
+            -0.004980007652193308,
+            -0.01945962943136692,
+            -0.008517424575984478,
+            0.0031153822783380747,
+            0.011870680376887321,
+            -0.009568674489855766,
+            0.005659099668264389,
+            0.04720648378133774,
+            0.031583528965711594,
+            0.0018675028113648295,
+            0.026565156877040863,
+            -0.02862161584198475,
+            -0.017986344173550606,
+            0.01464076153934002,
+            -0.0064072515815496445,
+            -0.00785751547664404,
+            -0.013973180204629898,
+            -0.031798385083675385,
+            -0.012799156829714775,
+            0.011187751777470112,
+            0.013581839390099049,
+            -0.011602113023400307,
+            0.003735005622729659,
+            0.012906583957374096,
+            -0.014172687195241451,
+            -0.008057023398578167,
+            -0.0393182709813118,
+            0.0011222282191738486,
+            0.024370577186346054,
+            -0.027977053076028824,
+            0.013121437281370163,
+            0.002286659786477685,
+            0.007389441132545471,
+            -0.023618588224053383,
+            -0.02087152563035488,
+            0.008218163624405861,
+            -0.012154594995081425,
+            0.005248575005680323,
+            -0.002712531015276909,
+            0.034744951874017715,
+            -0.00800330936908722,
+            0.03351721540093422,
+            -0.012975644320249557,
+            0.01953636296093464,
+            -0.025506233796477318,
+            -0.02389482781291008,
+            -0.04272524267435074,
+            -0.0012977562146261334,
+            0.0029580784030258656,
+            -0.02820725366473198,
+            -0.006495495326817036,
+            -0.001958623994141817,
+            0.017249703407287598,
+            -0.012246674858033657,
+            -0.003274604445323348,
+            0.006622105836868286,
+            -0.01354347262531519,
+            -0.0020104190334677696,
+            -0.03987075388431549,
+            -0.02825329452753067,
+            -0.03406969830393791,
+            0.016835341230034828,
+            -0.022667091339826584,
+            0.017234355211257935,
+            -0.01995072327554226,
+            0.0005678283050656319,
+            -0.047359950840473175,
+            -0.01573037914931774,
+            0.02991073951125145,
+            0.008141430094838142,
+            -0.04204998537898064,
+            0.021470047533512115,
+            0.01755663752555847,
+            0.017479903995990753,
+            -0.02311214618384838,
+            0.018492786213755608,
+            0.002338455058634281,
+            0.004933967255055904,
+            0.024416616186499596,
+            -0.02535276673734188,
+            0.02177698165178299,
+            -0.024907711893320084,
+            -0.021838368847966194,
+            -0.004020838066935539,
+            0.013827386312186718,
+            0.01726504974067211,
+            0.0020104190334677696,
+            -0.01709623634815216,
+            0.016743261367082596,
+            0.018323972821235657,
+            0.003915329463779926,
+            0.022636398673057556,
+            -0.02664189040660858,
+            0.016620486974716187,
+            0.011809294112026691,
+            -0.02323492057621479,
+            0.0036256604362279177,
+            -0.01483259629458189,
+            -0.007174587342888117,
+            0.017525942996144295,
+            0.03146075829863548,
+            0.014617742039263248,
+            -0.03683210536837578,
+            0.012860543094575405,
+            0.011747906915843487,
+            -0.0014147749170660973,
+            0.038274697959423065,
+            0.0027240412309765816,
+            0.0239715613424778,
+            0.03397761657834053,
+            0.0066105956211686134,
+            0.017664063721895218,
+            0.03548159450292587,
+            -0.022912638261914253,
+            -0.009277086704969406,
+            -0.023802747949957848,
+            0.014111299999058247,
+            0.014310807920992374,
+            -0.012753116898238659,
+            -0.012001127935945988,
+            -0.015239283442497253,
+            -0.014118974097073078,
+            0.01879972033202648,
+            -0.0008503036806359887,
+            -0.015745725482702255,
+            -0.029711231589317322,
+            0.024692857638001442,
+            0.0006781327538192272,
+            -0.044658929109573364,
+            0.012323408387601376,
+            -0.014218727126717567,
+            -0.0037868008948862553,
+            -0.0027835096698254347,
+            0.007769272197037935,
+            -0.005605386104434729,
+            0.0033359911758452654,
+            0.004266385454684496,
+            0.05135009437799454,
+            -0.0014166932087391615,
+            0.017311088740825653,
+            -0.008839705027639866,
+            0.01970517635345459,
+            -0.010988244786858559,
+            0.022912638261914253,
+            -0.014556354843080044,
+            -0.051503561437129974,
+            0.02618148922920227,
+            -0.011041957885026932,
+            0.040883637964725494,
+            0.05098177120089531,
+            -0.01841605268418789,
+            -0.010750371031463146,
+            0.03575783595442772,
+            0.004680746700614691,
+            -0.025337418541312218,
+            0.007888209074735641,
+            -0.0012296552304178476,
+            -0.007504541426897049,
+            -0.019613096490502357,
+            -0.012077861465513706,
+            0.019720522686839104,
+            -0.0038712078239768744,
+            0.017372475937008858,
+            0.040638089179992676,
+            0.03161422535777092,
+            -0.021301234140992165,
+            0.009522633627057076,
+            0.01331327110528946,
+            -0.009215699508786201,
+            -0.007796129211783409,
+            0.0254295002669096,
+            -0.017034849151968956,
+            -0.0008320794440805912,
+            -0.012177614495158195,
+            -0.00400165468454361,
+            -0.035941995680332184,
+            0.001303511206060648,
+            -0.016605140641331673,
+            -0.0018761353567242622,
+            -0.007692538667470217,
+            -0.030923621729016304,
+            0.03228947892785072,
+            0.028698349371552467,
+            -0.0070633236318826675,
+            -0.009077579714357853,
+            -0.009468920528888702,
+            0.005037557799369097,
+            0.0036333338357508183,
+            -0.0019969907589256763,
+            0.0034357449039816856,
+            -0.01792495884001255,
+            0.007443154696375132,
+            -0.006775572896003723,
+            -0.01325188484042883,
+            -0.051135238260030746,
+            -0.01867694780230522,
+            0.025045832619071007,
+            0.004830377176403999,
+            -0.01692742109298706,
+            -0.003261175937950611,
+            0.008103063330054283,
+            -0.005862443707883358,
+            -0.02496909908950329,
+            0.03135332837700844,
+            0.01668187417089939,
+            0.01875368133187294,
+            -0.0017456883797422051,
+            -0.0022578847128897905,
+            0.01746455766260624,
+            0.006084970664232969,
+            -0.01547715812921524,
+            -0.009499614126980305,
+            0.021761635318398476,
+            0.004120592027902603,
+            0.020932912826538086,
+            -0.010735023766756058,
+            0.021086379885673523,
+            0.007957269437611103,
+            -0.012123901396989822,
+            -0.0027202044147998095,
+            -0.0012718586949631572,
+            0.004105245228856802,
+            0.016912074759602547,
+            -0.013980853371322155,
+            0.024953750893473625,
+            0.020088844001293182,
+            0.007109363563358784,
+            0.0044889128766953945,
+            -0.01668187417089939,
+            -0.02452404424548149,
+            0.016574447974562645,
+            0.002522615483030677,
+            0.027977053076028824,
+            0.012139247730374336,
+            -0.004013164900243282,
+            -0.02759338542819023,
+            -0.013052377849817276,
+            -0.024217110127210617,
+            0.01005976926535368,
+            0.023664627224206924,
+            0.030632033944129944,
+            -0.015461810864508152,
+            -0.013489758595824242,
+            -0.025276033207774162,
+            0.00469993008300662,
+            0.008908765390515327,
+            -0.027347838506102562,
+            -0.01992003060877323,
+            0.018032385036349297,
+            -0.025291379541158676,
+            0.026488423347473145,
+            -0.0019269713666290045,
+            -0.012852869927883148,
+            0.006675818935036659,
+            0.07071764022111893,
+            -0.013090744614601135,
+            -0.0012200635392218828,
+            -0.028652308508753777,
+            -0.017618024721741676,
+            -0.005931503605097532,
+            -0.008617177605628967,
+            0.0004412179405335337,
+            0.018492786213755608,
+            -0.008624851703643799,
+            0.020902220159769058,
+            0.01668187417089939,
+            0.0013006337685510516,
+            0.0032074626069515944,
+            -0.03526674211025238,
+            0.03032509982585907,
+            -0.004224182106554508,
+            0.0014838350471109152,
+            0.00244204537011683,
+            0.006307498086243868,
+            0.018400706350803375,
+            0.016896728426218033,
+            0.005931503605097532,
+            0.009200353175401688,
+            -0.005912320222705603,
+            -0.003738842438906431,
+            0.016114046797156334,
+            0.019858643412590027,
+            -0.01111869141459465,
+            -0.012944950722157955,
+            0.005728159565478563,
+            0.014901655726134777,
+            -0.004853397142142057,
+            -0.00414744857698679,
+            0.0075736017897725105,
+            0.020472511649131775,
+            -0.01617543213069439,
+            -0.002286659786477685,
+            0.004742133431136608,
+            -0.017372475937008858,
+            0.0007548663415946066,
+            0.01875368133187294,
+            0.007565928157418966,
+            0.015853151679039,
+            -0.015484831295907497,
+            -0.009161986410617828,
+            0.0021255193278193474,
+            -0.022421544417738914,
+            0.003121137386187911,
+            -0.012192961759865284,
+            -0.01058155670762062,
+            -0.00239600520581007,
+            -0.007416298147290945,
+            -0.006487822160124779,
+            -0.006568392273038626,
+            0.004630869720131159,
+            0.009430553764104843,
+            -0.004151285160332918,
+            0.01738782227039337,
+            0.005252411589026451,
+            0.013988526538014412,
+            -0.0053560021333396435,
+            0.01489398255944252,
+            0.02136262133717537,
+            -0.010213236324489117,
+            0.00075822341023013,
+            0.01775614358484745,
+            -0.0006344905705191195,
+            0.006741042714565992,
+            0.004013164900243282,
+            1.707021874608472e-05,
+            0.010842450894415379,
+            -0.004343118984252214,
+            -0.01992003060877323,
+            -0.0055094691924750805,
+            -0.011440972797572613,
+            0.002133192727342248,
+            0.002252129837870598,
+            0.03716973215341568,
+            -0.02513791248202324,
+            0.03235086426138878,
+            0.012914257124066353,
+            -0.014011546969413757,
+            -0.00569362984970212,
+            -0.047482721507549286,
+            0.01338233146816492,
+            -0.014694475568830967,
+            3.977555752499029e-05,
+            0.017970997840166092,
+            -0.00914663914591074,
+            -0.0014799984637647867,
+            0.00502988463267684,
+            -0.005482612177729607,
+            -0.0009457410196773708,
+            -0.01924477517604828,
+            -0.0067563895136117935,
+            0.010382049717009068,
+            0.013842732645571232,
+            -0.03471425920724869,
+            -0.02243689075112343,
+            -0.030263712629675865,
+            0.00392875773832202,
+            0.01883041486144066,
+            -0.010942204855382442,
+            -0.025828514248132706,
+            -0.009537980891764164,
+            0.013405351899564266,
+            -0.01697346195578575,
+            0.006388068199157715,
+            -0.003997818101197481,
+            0.014326154254376888,
+            -0.008832031860947609,
+            0.001874217065051198,
+            -0.01899922825396061,
+            0.01487096305936575,
+            0.0028314681258052588,
+            -0.00420116214081645,
+            -0.030340446159243584,
+            0.007120873779058456,
+            -0.00020574184600263834,
+            -0.01450264174491167,
+            -0.012269695289433002,
+            0.017280396074056625,
+            -0.011134038679301739,
+            -0.004745970480144024,
+            0.016743261367082596,
+            0.006695002317428589,
+            0.0006618269253522158,
+            0.018277931958436966,
+            0.005352165549993515,
+            -0.0005002068355679512,
+            0.005336818750947714,
+            0.013106090947985649,
+            0.0012517160503193736,
+            0.024048294872045517,
+            -0.01617543213069439,
+            0.00997536163777113,
+            -0.014333827421069145,
+            0.033701375126838684,
+            -0.01597592607140541,
+            -0.0016027720412239432,
+            0.012177614495158195,
+            0.040883637964725494,
+            -0.013497431762516499,
+            0.01712692901492119,
+            -0.0291280560195446,
+            -0.015354383736848831,
+            0.03465287387371063,
+            0.01688138209283352,
+            0.010988244786858559,
+            -0.010359029285609722,
+            -0.01332094520330429,
+            0.008801338262856007,
+            0.01470982190221548,
+            0.008824358694255352,
+            -0.0241403765976429,
+            0.02908201701939106,
+            -0.014571702107787132,
+            0.04162028059363365,
+            -0.0001700127759249881,
+            -0.003117300570011139,
+            0.023204226046800613,
+            -0.0009577306336723268,
+            -0.02265174500644207,
+            0.014947696588933468,
+            0.00268183765001595,
+            0.01203182153403759,
+            -0.01066596433520317,
+            0.010527843609452248,
+            0.023787401616573334,
+            0.006537698674947023,
+            -0.005018374416977167,
+            -7.163796544773504e-05,
+            0.016912074759602547,
+            0.0005409715231508017,
+            -0.005033721216022968,
+            0.027731506153941154,
+            -0.007481521461158991,
+            -0.010842450894415379,
+            -0.01875368133187294,
+            -0.0012171859852969646,
+            -0.018247239291667938,
+            -0.00043474353151395917,
+            -0.007700212299823761,
+            0.0008320794440805912,
+            -0.010589230805635452,
+            -0.015154876746237278,
+            -0.003944104537367821,
+            -0.025076525285840034,
+            0.03480634093284607,
+            0.009676100686192513,
+            0.013597185723483562,
+            0.009200353175401688,
+            0.00365059869363904,
+            -0.000640245561953634,
+            0.0214547012001276,
+            -0.0030175470747053623,
+            0.0005649508093483746,
+            -0.005685956217348576,
+            -0.0119857806712389,
+            -0.0122390016913414,
+            0.013980853371322155,
+            -0.008317917585372925,
+            -0.008640198037028313,
+            -0.00844069104641676,
+            -0.01821654476225376,
+            -0.03477564454078674,
+            -0.0013610613532364368,
+            0.008893419057130814,
+            -0.028360720723867416,
+            0.014909329824149609,
+            -0.006441781762987375,
+            -0.01600661873817444,
+            0.0008009064476937056,
+            0.006161704193800688,
+            0.018554173409938812,
+            -0.007972615770995617,
+            -0.03425385802984238,
+            0.031384024769067764,
+            0.01646701991558075,
+            0.003899982897564769,
+            0.0033820313401520252,
+            0.008010982535779476,
+            0.008064696565270424,
+            -0.01395783293992281,
+            -0.0099139753729105,
+            0.028115173801779747,
+            -0.026212181895971298,
+            0.0239715613424778,
+            0.012561283074319363,
+            0.0003963767667300999,
+            0.027240412309765816,
+            -0.019474975764751434,
+            0.02630426175892353,
+            -0.0076042949222028255,
+            -5.1765175157925114e-05,
+            0.015200916677713394,
+            0.02099430002272129,
+            0.020319044589996338,
+            0.02272847853600979,
+            -0.02265174500644207,
+            0.038643017411231995,
+            -0.0023787401150912046,
+            0.03201323747634888,
+            0.006307498086243868,
+            -0.00015094928676262498,
+            0.008486730977892876,
+            0.008172123692929745,
+            -0.017679410055279732,
+            -0.0010176787618547678,
+            -0.03069342114031315,
+            0.010604577139019966,
+            -0.005229391623288393,
+            -0.01867694780230522,
+            -0.020794792100787163,
+            -0.033916231244802475,
+            0.010888490825891495,
+            0.001221981830894947,
+            -0.013037030585110188,
+            -0.004742133431136608,
+            0.0078076389618217945,
+            0.01306772418320179,
+            0.022713132202625275,
+            -0.0037100673653185368,
+            0.0032688493374735117,
+            0.019643789157271385,
+            -0.005298451986163855,
+            0.004331609234213829,
+            -0.01617543213069439,
+            0.02211461029946804,
+            0.005555509123951197,
+            0.030754808336496353,
+            0.02476959116756916,
+            0.008540444076061249,
+            0.004611686337739229,
+            -0.004204998724162579,
+            0.017817530781030655,
+            -0.01600661873817444,
+            -0.010481803677976131,
+            -0.001481916755437851,
+            -0.0038366776425391436,
+            0.06037395820021629,
+            -0.024692857638001442,
+            -0.02506117895245552,
+            -0.005559345707297325,
+            -0.011487012729048729,
+            0.006675818935036659,
+            -0.03247363865375519,
+            -0.028928549960255623,
+            -0.023511160165071487,
+            -0.0175873301923275,
+            -0.0019624605774879456,
+            -0.009998382069170475,
+            0.014049913734197617,
+            -0.00569362984970212,
+            0.011778600513935089,
+            -0.0239715613424778,
+            0.000355372263584286,
+            -0.016651181504130363,
+            -0.005685956217348576,
+            -0.014026893302798271,
+            -0.0070403036661446095,
+            -0.000628735579084605,
+            0.02949637733399868,
+            0.011824640445411205,
+            -0.019229428842663765,
+            -0.0032918695360422134,
+            0.012745442800223827,
+            -0.008279550820589066,
+            -0.007362584583461285,
+            0.012246674858033657,
+            0.01916804164648056,
+            -0.012208308093249798,
+            0.037937067449092865,
+            -0.0037983108777552843,
+            -0.00038582590059377253,
+            -0.005904647056013346,
+            -0.012584302574396133,
+            0.024877017363905907,
+            0.028314681723713875,
+            -0.03984006121754646,
+            -0.0013303679879754782,
+            0.0018694212194532156,
+            0.007600458338856697,
+            -0.005321471951901913,
+            0.0030866072047501802,
+            -0.027117637917399406,
+            -0.0012776136863976717,
+            -0.026872090995311737,
+            -0.01573037914931774,
+            -0.021470047533512115,
+            -0.005923830438405275,
+            0.0032458293717354536,
+            -0.023050758987665176,
+            0.030094899237155914,
+            0.02077944576740265,
+            -0.014487294480204582,
+            -0.0038596978411078453,
+            -0.003500968450680375,
+            0.008126083761453629,
+            -0.03189046308398247,
+            -0.004362302366644144,
+            -0.009745161049067974,
+            -9.597688767826185e-05,
+            -0.00962238758802414,
+            -0.04198860004544258,
+            0.0012239001225680113,
+            -0.003167177317664027,
+            -0.0017955651273950934,
+            -0.008494404144585133,
+            -0.016727915033698082,
+            0.02177698165178299,
+            0.020303698256611824,
+            -0.010880817659199238,
+            -0.008532770909368992,
+            0.045610424131155014,
+            0.008425343781709671,
+            0.0016411389224231243,
+            -0.010888490825891495,
+            0.002762407995760441,
+            0.0028314681258052588,
+            0.008486730977892876,
+            -0.013528125360608101,
+            0.011463993228971958,
+            -0.005298451986163855,
+            0.014118974097073078,
+            -0.0067065125331282616,
+            -0.016114046797156334,
+            0.023204226046800613,
+            -0.02829933539032936,
+            0.000396856339648366,
+            -0.010174869559705257,
+            -0.001934644766151905,
+            0.01395783293992281,
+            0.010259276255965233,
+            0.01726504974067211,
+            -0.03953312709927559,
+            -0.004815030377358198,
+            -0.008386977016925812,
+            -0.0004716715484391898,
+            -0.0059468504041433334,
+            -0.026994863525032997,
+            0.013443718664348125,
+            0.0023557201493531466,
+            -0.010880817659199238,
+            0.018323972821235657,
+            -0.004515769425779581,
+            -0.009960015304386616,
+            0.022375503554940224,
+            -0.0019020329928025603,
+            0.015362057834863663,
+            0.029235484078526497,
+            -0.016789302229881287,
+            0.0183700118213892,
+            0.032504335045814514,
+            -0.009369166567921638,
+            -0.0037139039486646652,
+            -0.0028142030350863934,
+            0.0009936995338648558,
+            0.016328901052474976,
+            0.017525942996144295,
+            0.0007447950774803758,
+            -0.03228947892785072,
+            0.00829489715397358,
+            -0.00218306970782578,
+            0.007918902672827244,
+            0.009514960460364819,
+            -0.014318481087684631,
+            -0.000949098146520555,
+            -0.025997327640652657,
+            0.006952059920877218,
+            -0.0209482591599226,
+            -0.009422880597412586,
+            0.024171069264411926,
+            -0.017786838114261627,
+            -0.019152695313096046,
+            0.003688965691253543,
+            0.02837606891989708,
+            -0.029680538922548294,
+            0.01130285207182169,
+            0.004139775410294533,
+            -0.005812566727399826,
+            0.006361211650073528,
+            0.000759182614274323,
+            -0.01895318739116192,
+            0.004500423092395067,
+            -0.009507287293672562,
+            -0.012169941328465939,
+            -0.020257657393813133,
+            -0.013835059478878975,
+            -0.0006699798395857215,
+            0.014617742039263248,
+            0.015822459012269974,
+            0.028851816430687904,
+            0.0016516897594556212,
+            -0.0069405497051775455,
+            -0.014656108804047108,
+            0.006879162974655628,
+            -0.0064801485277712345,
+            0.012990990653634071,
+            -0.008709258399903774,
+            0.02535276673734188,
+            -0.00811073649674654,
+            -0.0006037971470504999,
+            -0.010497150011360645,
+            -0.013198170810937881,
+            0.008762971498072147,
+            -0.003247747663408518,
+            0.0017994018271565437,
+            -0.011732560582458973,
+            0.004354629199951887,
+            0.01692742109298706,
+            -0.01863090693950653,
+            -0.02177698165178299,
+            -0.023449774831533432,
+            0.021976489573717117,
+            0.03766082972288132,
+            0.0009236801415681839,
+            -0.050214435905218124,
+            -0.0021063359454274178,
+            -0.007677191868424416,
+            -0.012077861465513706,
+            -0.02265174500644207,
+            0.011487012729048729,
+            0.018139811232686043,
+            -0.0008004268747754395,
+            -0.0019893173594027758,
+            0.0031978709157556295,
+            0.015108836814761162,
+            -0.0013207762967795134,
+            0.0053866952657699585,
+            0.00251494231633842,
+            -0.0075007048435509205,
+            0.004155121743679047,
+            0.029941432178020477,
+            -0.027731506153941154,
+            0.00941520743072033,
+            0.002618532394990325,
+            0.011755580082535744,
+            0.004669236950576305,
+            0.009177332744002342,
+            0.01470982190221548,
+            0.0032400742638856173,
+            -0.00709401722997427,
+            0.0196591354906559,
+            0.07206815481185913,
+            -0.0028314681258052588,
+            0.015385077334940434,
+            0.00972981471568346,
+            -0.012822176329791546,
+            -0.004040021449327469,
+            -0.010312989354133606,
+            -0.03848955035209656,
+            0.025398805737495422,
+            0.008034002967178822,
+            -0.012077861465513706,
+            0.006483985111117363,
+            -0.014126647263765335,
+            0.008878071792423725,
+            0.029020629823207855,
+            0.010366703383624554,
+            -0.020794792100787163,
+            0.004281732253730297,
+            0.03453009948134422,
+            0.0214547012001276,
+            0.004542626440525055,
+            -0.01780218444764614,
+            0.006219254340976477,
+            -0.0014195707626640797,
+            -0.0038712078239768744,
+            0.009614714421331882,
+            -0.004465892910957336,
+            0.022375503554940224,
+            -0.004078388214111328,
+            0.014111299999058247,
+            -0.013535798527300358,
+            -0.013911793008446693,
+            0.006115664262324572,
+            -0.001624832977540791,
+            0.0014790392015129328,
+            0.020150231197476387,
+            0.0028314681258052588,
+            -0.006760226096957922,
+            0.0024861672427505255,
+            -0.0005850933375768363,
+            0.013681592419743538,
+            0.004818866960704327,
+            0.007423971313983202,
+            0.006614432204514742,
+            -0.023050758987665176,
+            0.01879972033202648,
+            0.006434108596295118,
+            0.00976818148046732,
+            -0.013037030585110188,
+            -0.005682119634002447,
+            -0.005720486398786306,
+            -0.004757480230182409,
+            0.0018454419914633036,
+            -0.00196437886916101,
+            -0.01573037914931774,
+            -0.004435199312865734,
+            0.009960015304386616,
+            -0.024662164971232414,
+            0.0048764171078801155,
+            0.010842450894415379,
+            0.004991517867892981,
+            0.0009601286146789789,
+            -0.01124146580696106,
+            -0.03256572037935257,
+            5.1435461500659585e-05,
+            -0.005647589452564716,
+            -0.0021255193278193474,
+            0.004853397142142057,
+            0.010919184423983097,
+            0.0222373828291893,
+            0.004396832548081875,
+            -0.0013831222895532846,
+            0.021715596318244934,
+            0.0007064282544888556,
+            -0.0033014612272381783,
+            0.016866035759449005,
+            -0.00735491095110774,
+            -0.013129111379384995,
+            0.00012517160212155432,
+            0.003560436889529228,
+            0.014755862765014172,
+            0.0057358331978321075,
+            0.01617543213069439,
+            0.010351356118917465,
+            -0.02555227279663086,
+            -0.020840832963585854,
+            9.196036990033463e-05,
+            0.026319608092308044,
+            -0.012653362937271595,
+            -0.01912200078368187,
+            -0.00412826519459486,
+            -0.005187188275158405,
+            0.006925203371793032,
+            -0.012231328524649143,
+            0.008977825753390789,
+            0.016451673582196236,
+            -0.011364239268004894,
+            -0.026273569092154503,
+            -0.020472511649131775,
+            -0.005847096908837557,
+            0.009775854647159576,
+            0.009814221411943436,
+            -0.006741042714565992,
+            0.012983317486941814,
+            0.002706776140257716,
+            -0.021761635318398476,
+            0.022835904732346535,
+            0.013489758595824242,
+            0.005689792800694704,
+            -0.005229391623288393,
+            -0.004523443058133125,
+            -0.011479339562356472,
+            0.0068753263913095,
+            0.004070715047419071,
+            -0.007293524220585823,
+            -0.030923621729016304,
+            -0.0024880855344235897,
+            -0.006537698674947023,
+            -0.00125747115816921,
+            -0.009514960460364819,
+            0.00602742051705718,
+            -0.029972126707434654,
+            0.004020838066935539,
+            -0.0060926442965865135,
+            -0.0041819787584245205,
+            -0.006449454929679632,
+            -0.0009423839510418475,
+            -0.009238719940185547,
+            -0.028237948194146156,
+            -0.027271104976534843,
+            0.0011807375121861696,
+            0.01199345476925373,
+            -0.0016392205143347383,
+            -0.002380658406764269,
+            -0.024247802793979645,
+            -0.0004181978583801538,
+            0.008640198037028313,
+            -0.00744699127972126,
+            0.01846209354698658,
+            -0.003890391206368804,
+            -0.006952059920877218,
+            -0.0004949314170517027,
+            -0.0012392469216138124,
+            -0.026749316602945328,
+            0.017080888152122498,
+            0.0031000354792922735,
+            0.014326154254376888,
+            -0.02435522899031639,
+            0.011279832571744919,
+            -0.013366985134780407,
+            0.007780782412737608,
+            -0.01841605268418789,
+            -0.008762971498072147,
+            -0.00786518957465887,
+            -0.012392468750476837,
+            -0.007826822809875011,
+            -0.029143404215574265,
+            0.00844069104641676,
+            -0.008801338262856007,
+            -0.025813167914748192,
+            -0.02908201701939106,
+            0.009353820234537125,
+            -0.015776418149471283,
+            0.015561564825475216,
+            -0.009215699508786201,
+            0.04527279734611511,
+            0.020027456805109978,
+            0.015354383736848831,
+            0.008824358694255352,
+            -0.017863571643829346,
+            -0.0032937878277152777,
+            0.012653362937271595,
+            -0.026273569092154503,
+            0.0003601681091822684,
+            0.012553608976304531,
+            0.01487096305936575,
+            -0.011318199336528778,
+            0.001353388070128858,
+            0.01651306077837944,
+            0.015177897177636623,
+            0.01622147299349308,
+            0.0024036786053329706,
+            -0.009131292812526226,
+            0.010305316187441349,
+            0.023633934557437897,
+            0.02567504718899727,
+            -0.004803520627319813,
+            0.0019662973936647177,
+            0.0011625132756307721,
+            -0.018477439880371094,
+            -0.009077579714357853,
+            -0.013205844908952713,
+            -0.009783527813851833,
+            -0.02659584954380989,
+            0.011525379493832588,
+            -0.0038002291694283485,
+            -0.010013728402554989,
+            -0.01580711267888546,
+            -0.014978389255702496,
+            -0.005133474711328745,
+            0.006599085405468941,
+            0.021546781063079834,
+            -0.010712004266679287,
+            0.010919184423983097,
+            -0.010052095167338848,
+            0.0020545409061014652,
+            -0.00196437886916101,
+            0.0081491032615304,
+            0.027225064113736153,
+            0.010274622589349747,
+            -0.055524397641420364,
+            -0.006499331910163164,
+            0.002762407995760441,
+            -0.010650617070496082,
+            -0.051503561437129974,
+            0.002656899392604828,
+            0.007324217818677425,
+            -0.03185977041721344,
+            0.008655544370412827,
+            -0.0023154348600655794,
+            -0.0079265758395195,
+            0.006595248822122812,
+            0.019935376942157745,
+            -0.016206126660108566,
+            0.013259558007121086,
+            -6.013542588334531e-06,
+            0.004320099018514156,
+            -0.014924676157534122,
+            0.0062230913899838924,
+            -0.015147203579545021,
+            -0.014471948146820068,
+            0.01734178327023983,
+            -0.001127024064771831,
+            -0.002089071087539196,
+            0.012008801102638245,
+            0.02506117895245552,
+            0.02579781971871853,
+            -0.012162268161773682,
+            -0.01592988520860672,
+            -0.01879972033202648,
+            0.0007073874585330486,
+            -0.0055401623249053955,
+            -0.010712004266679287,
+            -0.0019624605774879456,
+            0.008417670615017414,
+            0.01209320779889822,
+            0.01987398974597454,
+            0.008417670615017414,
+            0.006315171252936125,
+            -3.35109798470512e-05,
+            0.009898628108203411,
+            0.0023422916419804096,
+            -0.020303698256611824,
+            -0.0065530454739928246,
+            0.012983317486941814,
+            0.019797256216406822,
+            0.0017936468357220292,
+            0.004297079052776098,
+            0.018185852095484734,
+            -0.00773857906460762,
+            -0.009208026342093945,
+            -0.0336092971265316,
+            -0.0006901223678141832,
+            -0.017203662544488907,
+            0.016236819326877594,
+            0.01945962943136692,
+            -0.02136262133717537,
+            -0.01022858265787363,
+            -0.01866159960627556,
+            0.02394086867570877,
+            -0.010735023766756058,
+            0.005912320222705603,
+            0.0022252730559557676,
+            0.008739951997995377,
+            0.0019068288384005427,
+            -0.008517424575984478,
+            -0.014272441156208515,
+            0.009277086704969406,
+            0.00048318158951587975,
+            0.007661845069378614,
+            -0.0061195008456707,
+            -0.012039494700729847,
+            0.001056045526638627,
+            -0.009860261343419552,
+            -0.028283987194299698,
+            0.011579093523323536,
+            0.0008670891402289271,
+            0.01821654476225376,
+            -0.019597750157117844,
+            0.01675860770046711,
+            0.0007380808820016682,
+            -0.013520452193915844,
+            -0.02788497321307659,
+            -0.015277650207281113,
+            0.003919166047126055,
+            0.003792555769905448,
+            0.012661036103963852,
+            0.0034817850682884455,
+            -0.01658979430794716,
+            0.008739951997995377,
+            -0.005647589452564716,
+            -0.005751179996877909,
+            -0.001697729923762381,
+            -0.01170186698436737,
+            -0.001447386690415442,
+            0.011072651483118534,
+            -0.01037437655031681,
+            -0.005574692506343126,
+            0.01468680240213871,
+            0.006411088164895773,
+            -0.002988772001117468,
+            0.02742457203567028,
+            0.01821654476225376,
+            0.02153143472969532,
+            0.0006488781073130667,
+            0.01066596433520317,
+            -0.0018473602831363678,
+            0.003558518597856164,
+            -0.03505188599228859,
+            0.025122566148638725,
+            -0.01576107181608677,
+            0.023373039439320564,
+            0.008532770909368992,
+            -0.00694822333753109,
+            -0.005881627090275288,
+            0.01331327110528946,
+            0.006088807247579098,
+            -0.004803520627319813,
+            0.00602742051705718,
+            -0.003128810552880168,
+            0.009844915010035038,
+            -0.0013102254597470164,
+            -0.02493840456008911,
+            0.020012110471725464,
+            -0.024416616186499596,
+            0.009990708902478218,
+            -0.002990690292790532,
+            0.009123619645833969,
+            -0.006894509773701429,
+            -5.931863415753469e-05,
+            -0.00457331957295537,
+            -0.040638089179992676,
+            -0.001970133977010846,
+            -0.010773390531539917,
+            0.004450546111911535,
+            0.016819994896650314,
+            -0.00021497384295798838,
+            -0.04220345616340637,
+            -0.005916156806051731,
+            0.01958240196108818,
+            -0.00021737176575697958,
+            0.009883281774818897,
+            -0.008271876722574234,
+            -0.004086061846464872,
+            -0.004320099018514156,
+            -0.007792292162775993,
+            0.016543753445148468,
+            0.013136784546077251,
+            -0.01563829742372036,
+            -0.023618588224053383,
+            0.012507569044828415,
+            -0.01879972033202648,
+            -0.019567055627703667,
+            0.00530996173620224,
+            -0.01267638336867094,
+            0.001553854439407587,
+            0.022268077358603477,
+            -0.004715276882052422,
+            0.011164732277393341,
+            -0.0056130592711269855,
+            0.013282578438520432,
+            -0.01543111726641655,
+            -0.0019471138948574662,
+            -0.0047651538625359535,
+            -0.02348046749830246,
+            0.025567619130015373,
+            -0.018937841057777405,
+            0.0019931539427489042,
+            -0.016528407111763954,
+            0.02228342369198799,
+            -0.020088844001293182,
+            -0.002275149803608656,
+            0.008010982535779476,
+            0.010174869559705257,
+            0.029803311452269554,
+            0.01345906499773264,
+            -0.007408624514937401,
+            0.009169659577310085,
+            0.008824358694255352,
+            0.019827950745821,
+            0.00156632368452847,
+            0.029803311452269554,
+            -0.005271594971418381,
+            0.01041274331510067,
+            -0.001517405966296792,
+            0.018968533724546432,
+            0.0067065125331282616,
+            0.029926085844635963,
+            0.0003174850717186928,
+            0.00015430638450197875,
+            -0.0001380004978273064,
+            0.018600212410092354,
+            -0.004369975998997688,
+            -0.005313798785209656,
+            0.007765435613691807,
+            -0.0059698703698813915,
+            -0.0015797520754858851,
+            -0.020441818982362747,
+            -0.005877790041267872,
+            0.001410938217304647,
+            0.013789019547402859,
+            -0.0005241860635578632,
+            0.005390532314777374,
+            0.0008200898300856352,
+            0.005586202722042799,
+            -0.014617742039263248,
+            0.011402606032788754,
+            -0.015668991953134537,
+            0.01883041486144066,
+            -0.01005976926535368,
+            0.01008278876543045,
+            -0.0009692406747490168,
+            0.00811073649674654,
+            0.011072651483118534,
+            -0.013574165292084217,
+            0.01614473946392536,
+            0.012139247730374336,
+            -0.010589230805635452,
+            0.016313552856445312,
+            -0.007228300906717777,
+            0.005083597730845213,
+            0.0038616161327809095,
+            0.0005870116874575615,
+            0.003153749043121934,
+            -0.018109118565917015,
+            -0.009461247362196445,
+            -0.0001405183138558641,
+            -0.012576629407703876,
+            -0.0030501587316393852,
+            0.012331082485616207,
+            -0.005505632609128952,
+            0.015507850795984268,
+            0.010197889059782028,
+            0.012285041622817516,
+            0.004933967255055904,
+            0.011172405444085598,
+            -0.0014186116168275476,
+            -0.0034299897961318493,
+            0.004684583283960819,
+            0.032964736223220825,
+            0.0008824358810670674,
+            0.025905247777700424,
+            -0.014978389255702496,
+            0.006084970664232969,
+            0.001282409531995654,
+            -0.0016181188402697444,
+            -0.008701585233211517,
+            0.016236819326877594,
+            0.020088844001293182,
+            0.0018934004474431276,
+            -0.011893700808286667,
+            -0.003499049926176667,
+            0.029404297471046448,
+            -0.005501795560121536,
+            0.007454664912074804,
+            0.010696657001972198,
+            -0.006196234375238419,
+            0.016605140641331673,
+            0.004381485749036074,
+            -0.024692857638001442,
+            0.0013457146706059575,
+            0.004047695081681013,
+            -0.019152695313096046,
+            0.020917566493153572,
+            0.02104034088551998,
+            0.0076042949222028255,
+            0.002739387797191739,
+            0.01345906499773264,
+            0.010497150011360645,
+            0.021623514592647552,
+            0.0032400742638856173,
+            0.004580993205308914,
+            0.002643470885232091,
+            -0.012952623888850212,
+            -0.00123445107601583,
+            -0.020963607355952263,
+            -0.011533052660524845,
+            -0.0020583774894475937,
+            0.018983881920576096,
+            0.0026530625764280558,
+            -0.013290251605212688,
+            -0.003959451336413622,
+            0.008855052292346954,
+            0.004381485749036074,
+            -0.006299824919551611,
+            -0.022053223103284836,
+            -0.008202817291021347,
+            0.024094335734844208,
+            0.008095390163362026,
+            0.010550864040851593,
+            -0.0031729324255138636,
+            -0.002160049509257078,
+            -0.0006109909154474735,
+            -0.006614432204514742,
+            -0.0006450414075516164,
+            -0.01597592607140541,
+            -0.0021063359454274178,
+            -0.016666527837514877,
+            0.015346710570156574,
+            0.015791764482855797,
+            0.01763337105512619,
+            0.018139811232686043,
+            -0.02102499268949032,
+            0.017602676525712013,
+            -0.0071438937447965145,
+            0.001838727737776935,
+            -0.0034414997790008783,
+            0.00010826624929904938,
+            0.019275467842817307,
+            0.029388951137661934,
+            0.0056015495210886,
+            0.011149385012686253,
+            -0.0013831222895532846,
+            0.005927667021751404,
+            -0.006457128562033176,
+            -0.013980853371322155,
+            -0.0033935413230210543,
+            -0.01892249472439289,
+            0.004907110705971718,
+            0.017848225310444832,
+            -0.015139530412852764,
+            -0.004922457505017519,
+            0.014349174685776234,
+            0.009867934510111809,
+            0.008340937085449696,
+            -0.008931785821914673,
+            -0.004112918395549059,
+            0.0008714054129086435,
+            0.01557691115885973,
+            -0.007585111539810896,
+            -0.016605140641331673,
+            0.008394651114940643,
+            -0.006146357394754887,
+            0.02123984694480896,
+            -0.004542626440525055,
+            -0.01605265960097313,
+            -0.010282295756042004,
+            0.003186360700055957,
+            -0.021439354866743088,
+            -0.014272441156208515,
+            0.01002140250056982,
+            0.0021312744356691837,
+            0.006936713121831417,
+            -0.017326436936855316,
+            0.013060051016509533,
+            0.004834213759750128,
+            0.03351721540093422,
+            -0.008133756928145885,
+            -0.0005424103001132607,
+            0.0003779127437155694,
+            0.005916156806051731,
+            0.0002481850970070809,
+            -0.01755663752555847,
+            0.026933476328849792,
+            0.006587575655430555,
+            0.013781346380710602,
+            -0.0003863054735120386,
+            -0.012645689770579338,
+            0.006994263269007206,
+            -0.01564597152173519,
+            0.001863666228018701,
+            -0.010205562226474285,
+            -0.020088844001293182,
+            -0.01058155670762062,
+            0.02165420912206173,
+            0.0070403036661446095,
+            -0.00732038076967001,
+            -0.0001907548139570281,
+            -0.0077769458293914795,
+            -0.016359593719244003,
+            -0.009852588176727295,
+            -0.015492504462599754,
+            0.008386977016925812,
+            -0.01663583517074585,
+            0.0029005282558500767,
+            0.008126083761453629,
+            0.009806548245251179,
+            0.005586202722042799,
+            0.021623514592647552,
+            0.002338455058634281,
+            0.016405634582042694,
+            0.0001647373428568244,
+            -0.0019049104303121567,
+            0.005651426035910845,
+            0.014771209098398685,
+            0.0072973608039319515,
+            -0.008555791340768337,
+            -0.0046423799358308315,
+            -0.012691729702055454,
+            0.0005198698490858078,
+            -0.009676100686192513,
+            0.01066596433520317,
+            0.0064302715472877026,
+            -0.0009404656011611223,
+            -0.00678324606269598,
+            0.021715596318244934,
+            -0.0038980646058917046,
+            0.007711722049862146,
+            -0.007957269437611103,
+            -0.004300915636122227,
+            0.01065829023718834,
+            0.0010848206002265215,
+            -0.013765999116003513,
+            0.008210490457713604,
+            0.00919268000870943,
+            -0.002263639820739627,
+            -0.003238155972212553,
+            -0.020672019571065903,
+            0.02874438837170601,
+            0.008885745890438557,
+            -0.010182542726397514,
+            0.014425908215343952,
+            -0.0052831051871180534,
+            0.019306162372231483,
+            0.016282860189676285,
+            0.0185388270765543,
+            -0.018323972821235657,
+            -0.019643789157271385,
+            0.007251320872455835,
+            -0.015561564825475216,
+            -0.0027835096698254347,
+            0.0029715069103986025,
+            -0.0022809049114584923,
+            -0.0002340373321203515,
+            -0.016068005934357643,
+            0.0009150475962087512,
+            -0.0046500531025230885,
+            0.0024938404094427824,
+            0.000545287795830518,
+            0.0077347420156002045,
+            0.0009395064553245902,
+            0.012760790064930916,
+            -0.006196234375238419,
+            -0.01783287711441517,
+            -0.006606759037822485,
+            0.013512779027223587,
+            0.006906019989401102,
+            -0.03600338473916054,
+            0.004980007652193308,
+            -0.012499895878136158,
+            -0.01751059666275978,
+            -0.002232946455478668,
+            -0.02157747559249401,
+            -0.02116311341524124,
+            -0.001316939597018063,
+            0.0009831486968323588,
+            -0.011510033160448074,
+            0.0021869062911719084,
+            0.006648962385952473,
+            -0.009300106205046177,
+            -0.00919268000870943,
+            0.0054595922119915485,
+            -0.0018751762108877301,
+            0.012645689770579338,
+            -0.012392468750476837,
+            -0.010443436913192272,
+            0.035174660384655,
+            0.013635552488267422,
+            -0.00766568211838603,
+            -0.013812039978802204,
+            -0.010873144492506981,
+            -0.014280114322900772,
+            0.010980571620166302,
+            -0.0030827706214040518,
+            -0.00466539990156889,
+            -0.037814296782016754,
+            0.0013543472159653902,
+            0.0011884109117090702,
+            0.01524695660918951,
+            -0.008862725459039211,
+            0.014533335343003273,
+            0.005793383345007896,
+            -0.00058077706489712,
+            -0.012354101985692978,
+            -6.549328645633068e-06,
+            0.01080408412963152,
+            0.010136502794921398,
+            -0.007197607308626175,
+            0.012100880965590477,
+            -0.013006336987018585,
+            0.007527561392635107,
+            -0.014633088372647762,
+            -0.0077769458293914795,
+            -0.01933685503900051,
+            -0.01787891797721386,
+            -0.007516051642596722,
+            0.01634424738585949,
+            0.009890954941511154,
+            -0.01787891797721386,
+            -0.004815030377358198,
+            -0.014525661244988441,
+            0.013451391831040382,
+            -0.011533052660524845,
+            -0.006015910767018795,
+            -0.01306772418320179,
+            0.001044535543769598,
+            -0.018784373998641968,
+            0.01385807991027832,
+            0.01775614358484745,
+            0.01639028638601303,
+            0.0004093255556654185,
+            -0.0053866952657699585,
+            -0.02248293161392212,
+            -0.01870764046907425,
+            -0.004462056327611208,
+            0.002572492463514209,
+            -0.013781346380710602,
+            -0.003744597313925624,
+            0.011732560582458973,
+            0.021915102377533913,
+            -0.0017984426813200116,
+            -0.013129111379384995,
+            0.00854811817407608,
+            -0.016958115622401237,
+            -0.017311088740825653,
+            -0.018277931958436966,
+            0.007358747534453869,
+            0.008172123692929745,
+            0.01605265960097313,
+            0.0024746570270508528,
+            0.006058114115148783,
+            -0.004623196553438902,
+            -0.007730905432254076,
+            -0.024278495460748672,
+            0.005110454745590687,
+            -0.0112030990421772,
+            0.010075115598738194,
+            -0.0021408661268651485,
+            -0.004070715047419071,
+            0.023465121164917946,
+            0.009760508313775063,
+            -0.004995354451239109,
+            -0.00018631866259966046,
+            0.020840832963585854,
+            -0.008686237968504429,
+            -0.00602742051705718,
+            -0.008057023398578167,
+            0.017771491780877113,
+            -0.021961143240332603,
+            0.00016090067219920456,
+            -0.011187751777470112,
+            -0.0222373828291893,
+            0.003272686153650284,
+            0.012200634926557541,
+            0.00606578728184104,
+            -0.003355174558237195,
+            0.003453009994700551,
+            -0.008433017879724503,
+            0.024094335734844208,
+            -0.01435684785246849,
+            0.002348046749830246,
+            0.013620206154882908,
+            -0.0021638860926032066,
+            -0.002465065335854888,
+            0.013727632351219654,
+            -0.005417388863861561,
+            0.009530307725071907,
+            0.0035738651640713215,
+            0.00011857732170028612,
+            -0.009660754352807999,
+            0.016988808289170265,
+            0.02535276673734188,
+            -0.00860950443893671,
+            0.0008152939844876528,
+            -0.003782964078709483,
+            0.00602742051705718,
+            -0.0070403036661446095,
+            0.03201323747634888,
+            -0.008287223987281322,
+            0.0005500836414285004,
+            0.0010100053623318672,
+            0.03161422535777092,
+            -0.0014771209098398685,
+            0.010044422000646591,
+            0.0014541008276864886,
+            -0.0034184798132628202,
+            0.022927986457943916,
+            0.0025667373556643724,
+            0.004531116224825382,
+            0.027102291584014893,
+            -0.013574165292084217,
+            -0.00493780430406332,
+            0.005543999373912811,
+            -0.015200916677713394,
+            0.012392468750476837,
+            -0.02048785798251629,
+            0.0009188842959702015,
+            -0.01522393710911274,
+            0.014280114322900772,
+            0.019981417804956436,
+            0.030386487022042274,
+            -0.014418235048651695,
+            -0.00619239779189229,
+            -0.00728201400488615,
+            0.03143006190657616,
+            0.00044217711547389627,
+            0.006288314703851938,
+            -0.00030045982566662133,
+            0.011471666395664215,
+            -0.015177897177636623,
+            -0.01962844282388687,
+            0.012499895878136158,
+            -0.019551709294319153,
+            -0.03413108363747597,
+            -0.004634706769138575,
+            0.005670609418302774,
+            -0.0030175470747053623,
+            0.0037100673653185368,
+            0.012146921828389168,
+            0.004358465783298016,
+            0.022145302966237068,
+            0.007243647240102291,
+            0.009752834215760231,
+            -0.016482368111610413,
+            3.229703224860714e-06,
+            -0.006848469842225313,
+            -0.025398805737495422,
+            0.016037313267588615,
+            -0.021516088396310806,
+            0.0014138157712295651,
+            -0.008433017879724503,
+            -0.002647307701408863,
+            0.009967688471078873,
+            0.004780500195920467,
+            0.014778882265090942,
+            0.005789546761661768,
+            0.016282860189676285,
+            0.019551709294319153,
+            -0.01573037914931774,
+            0.011586766690015793,
+            0.027025558054447174,
+            0.00113853404764086,
+            -0.017188316211104393,
+            -0.007784618996083736,
+            -0.01975121721625328,
+            -0.01945962943136692,
+            0.011256812140345573,
+            0.02041112445294857,
+            -0.00846371054649353,
+            -0.001863666228018701,
+            0.03904203325510025,
+            -0.013113764114677906,
+            0.023956215009093285,
+            -0.00775776244699955,
+            0.005240901838988066,
+            0.03468356654047966,
+            0.010827104561030865,
+            0.018692294135689735,
+            -0.00613868422806263,
+            -0.034407325088977814,
+            -0.002121682744473219,
+            -0.00680242944508791,
+            0.0065607186406850815,
+            -0.011049631983041763,
+            0.018032385036349297,
+            -0.013673919253051281,
+            -0.019812602549791336,
+            0.02036508545279503,
+            -0.02243689075112343,
+            0.021377967670559883,
+            0.007562091574072838,
+            0.008578810840845108,
+            -0.006269131321460009,
+            -0.011479339562356472,
+            -3.974558421759866e-05,
+            -0.0077270688489079475,
+            0.011318199336528778,
+            0.008034002967178822,
+            -0.011832313612103462,
+            -0.006806266028434038,
+            0.0031729324255138636,
+            -0.004634706769138575,
+            -0.02311214618384838,
+            0.012860543094575405,
+            0.008640198037028313,
+            0.013428371399641037,
+            0.007381767965853214,
+            0.005075924564152956,
+            0.010635270737111568,
+            0.0034817850682884455,
+            -0.013113764114677906,
+            -0.01203182153403759,
+            0.00914663914591074,
+            -0.006127174012362957,
+            -0.0007332850364036858,
+            -0.026687929406762123,
+            0.014702148735523224,
+            -0.022743824869394302,
+            -0.0032573393546044827,
+            -0.0018195443553850055,
+            0.0039786347188055515,
+            0.005486449226737022,
+            -0.01866159960627556,
+            -0.005636079702526331,
+            -0.00028990895953029394,
+            -0.009392186999320984,
+            0.01188602764159441,
+            -0.013620206154882908,
+            0.01646701991558075,
+            0.027056250721216202,
+            -0.01487096305936575,
+            -0.0036199053283780813,
+            0.00554783595725894,
+            0.0077040488831698895,
+            -0.012522916309535503,
+            0.016988808289170265,
+            0.015108836814761162,
+            -0.01205484103411436,
+            0.008847379125654697,
+            0.03083154186606407,
+            -0.006579902023077011,
+            0.0005088393809273839,
+            -0.00012493181566242129,
+            0.0024995955172926188,
+            0.029726577922701836,
+            0.0016066087409853935,
+            0.024293843656778336,
+            0.005873953457921743,
+            -0.0008862725226208568,
+            -0.014610068872570992,
+            -0.014364521019160748,
+            -0.009522633627057076,
+            -0.009990708902478218,
+            -0.005520978942513466,
+            -0.010497150011360645,
+            0.009706794284284115,
+            0.012699402868747711,
+            -0.004676910117268562,
+            0.0038884729146957397,
+            -0.0019164205295965075,
+            0.02131658047437668,
+            -0.005052904598414898,
+            0.013113764114677906,
+            -0.022206690162420273,
+            0.01962844282388687,
+            0.020595286041498184,
+            0.011839987710118294,
+            -0.016313552856445312,
+            -0.005643752869218588,
+            0.016819994896650314,
+            0.025214646011590958,
+            -0.00538285868242383,
+            0.001968215685337782,
+            0.019444283097982407,
+            0.0031384022440761328,
+            0.005252411589026451,
+            -0.0026626542676240206,
+            -0.01987398974597454,
+            -0.027194371446967125,
+            -0.003568110289052129,
+            -0.006737206131219864,
+            0.002689511049538851,
+            0.004097571596503258,
+            -0.011839987710118294,
+            -0.011609786190092564,
+            0.0046500531025230885,
+            -0.005026047583669424,
+            0.0011020856909453869,
+            -0.0099139753729105,
+            -0.007849842309951782,
+            -0.038059841841459274,
+            0.020150231197476387,
+            0.0004304272879380733,
+            0.004396832548081875,
+            0.004239528905600309,
+            0.015108836814761162,
+            -0.007723232265561819,
+            -0.0016229146858677268,
+            0.015085816383361816,
+            0.006069623865187168,
+            -0.0018233810551464558,
+            -0.0003510560200084001,
+            -0.0035719468723982573,
+            -0.007197607308626175,
+            0.0104741295799613,
+            -0.004980007652193308,
+            -0.010274622589349747,
+            0.0084023242816329,
+            -0.012852869927883148,
+            -0.015062796883285046,
+            0.008225836791098118,
+            -0.025153258815407753,
+            0.00501070125028491,
+            -0.000793712679296732,
+            0.010159522294998169,
+            -0.004661563318222761,
+            -0.008809012360870838,
+            0.0010032912250608206,
+            0.013873426243662834,
+            0.007032630033791065,
+            -0.004726787097752094,
+            0.016083352267742157,
+            -0.007400951348245144,
+            0.007124710362404585,
+            0.029220137745141983,
+            -0.007949596270918846,
+            0.008601831272244453,
+            0.018477439880371094,
+            0.0013687347527593374,
+            0.008471384644508362,
+            0.01130285207182169,
+            -0.03643308952450752,
+            -0.016743261367082596,
+            -0.0008809971041046083,
+            0.0051603312604129314,
+            -0.0011174323735758662,
+            0.0188764538615942,
+            0.00919268000870943,
+            0.02725575864315033,
+            -0.006177050992846489,
+            0.012684056535363197,
+            0.0019509505946189165,
+            0.023572547361254692,
+            0.012108555063605309,
+            0.01199345476925373,
+            -0.005747342947870493,
+            -0.0021312744356691837,
+            -0.03588061034679413,
+            0.0035661919973790646,
+            0.022175997495651245,
+            0.019490322098135948,
+            -0.02579781971871853,
+            0.017203662544488907,
+            -0.0057435063645243645,
+            0.0033129712101072073,
+            -0.013428371399641037,
+            0.017694758251309395,
+            0.004392995964735746,
+            0.017571983858942986,
+            -0.008156776428222656,
+            -0.002417106879875064,
+            0.016866035759449005,
+            0.010466456413269043,
+            -0.012914257124066353,
+            0.021470047533512115,
+            0.012806829996407032,
+            0.02119380794465542,
+            0.019198734313249588,
+            -0.018308626487851143,
+            0.00908525288105011,
+            0.00875529833137989,
+            -0.012200634926557541,
+            -0.0081491032615304,
+            0.00457331957295537,
+            0.007232137490063906,
+            -0.016605140641331673,
+            -0.005018374416977167,
+            -0.0005678283050656319,
+            0.00023475672060158104,
+            0.012768463231623173,
+            -0.012668709270656109,
+            0.000462079857243225,
+            0.00412826519459486,
+            0.011080324649810791,
+            -0.0065607186406850815,
+            0.01734178327023983,
+            -0.028729042038321495,
+            0.012223655357956886,
+            -0.022022530436515808,
+            -0.0239715613424778,
+            0.0014195707626640797,
+            -0.004481239710003138,
+            -0.0036333338357508183,
+            -0.0051027811132371426,
+            0.0015154876746237278,
+            0.005647589452564716,
+            0.009983035735785961,
+            0.028391415253281593,
+            0.007715558633208275,
+            0.0013601022073999047,
+            -0.0004165193240623921,
+            -0.02302006632089615,
+            -0.010827104561030865,
+            -0.006054277531802654,
+            -0.0021389478351920843,
+            -0.006445618346333504,
+            -0.0336092971265316,
+            -0.00022852211259305477,
+            -0.0057358331978321075,
+            0.0029216299299150705,
+            0.0008435894851572812,
+            -0.02007349766790867,
+            -0.021500742062926292,
+            0.00926941353827715,
+            -0.008870398625731468,
+            -0.014026893302798271,
+            -0.0168046485632658,
+            0.006472475361078978,
+            0.00037431585951708257,
+            -0.0036563538014888763,
+            0.010151849128305912,
+            0.01846209354698658,
+            -0.004676910117268562,
+            -0.0006843673763796687,
+            0.010090461932122707,
+            0.0026396343018859625,
+            -0.02165420912206173,
+            0.008617177605628967,
+            -0.014111299999058247,
+            0.003951778169721365,
+            -0.01130285207182169,
+            -0.01709623634815216,
+            -0.004964660853147507,
+            -0.003967124503105879,
+            -0.028391415253281593,
+            0.0011673092376440763,
+            0.011832313612103462,
+            -0.007067160215228796,
+            0.013635552488267422,
+            0.020441818982362747,
+            0.010405070148408413,
+            0.00084119156235829,
+            -0.014863288961350918,
+            0.011425626464188099,
+            -0.01457937527447939,
+            -0.012998663820326328,
+            0.01970517635345459,
+            0.012929603457450867,
+            0.0112030990421772,
+            -0.0008503036806359887,
+            -0.014019220136106014,
+            0.011072651483118534,
+            0.015745725482702255,
+            0.025245338678359985,
+            -0.006272967904806137,
+            -0.0007697334513068199,
+            -0.0162982065230608,
+            -0.018446745350956917,
+            0.034161776304244995,
+            -0.011932067573070526,
+            -0.013681592419743538,
+            0.01655910164117813,
+            0.019613096490502357,
+            0.012668709270656109,
+            -0.005616896320134401,
+            -0.004684583283960819,
+            -0.0014684883644804358,
+            0.012346428819000721,
+            0.005820239894092083,
+            -0.025199299678206444,
+            -0.010681310668587685,
+            0.004400669131428003,
+            0.0325964130461216,
+            0.011747906915843487,
+            0.006069623865187168,
+            -0.043001484125852585,
+            0.014172687195241451,
+            0.0008853133767843246,
+            0.0009505368652753532,
+            0.013980853371322155,
+            0.029481031000614166,
+            0.020257657393813133,
+            0.0009706794517114758,
+            0.01962844282388687,
+            -0.015914538875222206,
+            0.02269778586924076,
+            -0.020978953689336777,
+            -0.010919184423983097,
+            -0.0241403765976429,
+            -0.005006864201277494,
+            -0.008479057811200619,
+            0.0028257130179554224,
+            0.009031538851559162,
+            0.009637733921408653,
+            -0.012262022122740746,
+            0.02021161839365959,
+            0.013221191242337227,
+            -0.010044422000646591,
+            -0.001552895293571055,
+            -0.02070271223783493,
+            0.02547553926706314,
+            0.003771454095840454,
+            -0.0008680483442731202,
+            -0.010612250305712223,
+            -0.005018374416977167,
+            -0.021132420748472214,
+            0.0018368094461038709,
+            -0.0028506515081971884,
+            0.012906583957374096,
+            -0.0016085271490737796,
+            0.008348610252141953,
+            0.016942769289016724,
+            -0.015208589844405651,
+            -0.003878881223499775,
+            0.007846006192266941,
+            0.02104034088551998,
+            0.006437945179641247,
+            0.01190904714167118,
+            0.02854488231241703,
+            -0.0013936731265857816,
+            -0.027485959231853485,
+            0.012008801102638245,
+            -0.006403414998203516,
+            -0.003366684541106224,
+            0.01111869141459465,
+            0.02131658047437668,
+            -0.021746288985013962,
+            0.0011557991383597255,
+            -0.007335727568715811,
+            0.020150231197476387,
+            3.1173007300822064e-05,
+            0.013198170810937881,
+            0.009138965979218483,
+            0.003936431370675564,
+            -0.01111869141459465,
+            -0.01421105396002531,
+            -0.008026329800486565,
+            0.01170186698436737,
+            -0.026979517191648483,
+            -0.0024036786053329706,
+            0.001851196982897818,
+            -0.003021383658051491,
+            0.006311334669589996,
+            0.006230764556676149,
+            -0.022099263966083527,
+            -0.010635270737111568,
+            -0.0028564066160470247,
+            -0.010182542726397514,
+            -0.005206371657550335,
+            0.012476875446736813,
+            -0.005409715697169304,
+            0.0005798179190605879,
+            0.00875529833137989,
+            0.009630060754716396,
+            0.004300915636122227,
+            0.025506233796477318,
+            -0.012584302574396133,
+            0.01924477517604828,
+            -0.030647380277514458,
+            0.001827217754907906,
+            -0.008578810840845108,
+            0.024171069264411926,
+            -0.0024497187696397305,
+            -0.019075961783528328,
+            0.012047167867422104,
+            -0.008195143193006516,
+            0.001614282140508294,
+            -0.002810366451740265,
+            -0.008433017879724503,
+            -0.0065300255082547665,
+            0.012545935809612274,
+            -0.018155159428715706,
+            0.004308588802814484,
+            0.025444846600294113,
+            0.0429094024002552,
+            -0.013643225654959679,
+            -0.013121437281370163,
+            0.0052255550399422646,
+            0.009983035735785961,
+            0.024094335734844208,
+            -0.01144864596426487,
+            -0.009307780303061008,
+            -0.01159443985670805,
+            0.0075812749564647675,
+            -0.016896728426218033,
+            -0.023833442479372025,
+            0.011563746258616447,
+            -0.015960579738020897,
+            -0.016850687563419342,
+            -0.018155159428715706,
+            -0.017955651506781578,
+            -0.0027662445791065693,
+            0.013773673214018345,
+            0.005904647056013346,
+            -0.0028046113438904285,
+            -0.0015615278389304876,
+            -0.006111827678978443,
+            -0.021270541474223137,
+            0.016896728426218033,
+            0.005133474711328745,
+            -0.01614473946392536,
+            -0.004373812582343817,
+            -0.012039494700729847,
+            0.002562900772318244,
+            0.0037522707134485245,
+            -0.003075097221881151,
+            0.05506399646401405,
+            0.003518233308568597,
+            0.009507287293672562,
+            -0.02107103355228901,
+            0.004266385454684496,
+            -0.010796410962939262,
+            -0.01074269786477089,
+            -0.007707885466516018,
+            -0.012116228230297565,
+            -0.003617987036705017,
+            -0.011479339562356472,
+            0.0019969907589256763,
+            -0.01543111726641655,
+            0.008133756928145885,
+            -0.013673919253051281,
+            -0.013474412262439728,
+            -0.017571983858942986,
+            -0.0015864662127569318,
+            -0.003451091470196843,
+            0.016114046797156334,
+            0.01325188484042883,
+            -0.016374940052628517,
+            -0.0030904437880963087,
+            0.022268077358603477,
+            0.007895882241427898,
+            0.018968533724546432,
+            -0.0035048050340265036,
+            -0.013129111379384995,
+            -0.020012110471725464,
+            0.0006565514486283064,
+            0.013075397349894047,
+            -0.0075812749564647675,
+            0.002915875054895878,
+            -0.029097363352775574,
+            -0.006748715881258249,
+            0.0014090199256315827,
+            0.01600661873817444,
+            0.01668187417089939,
+            0.008862725459039211,
+            -0.0018732579192146659,
+            -0.020856179296970367,
+            0.0332716703414917,
+            -0.007753925397992134,
+            -0.022743824869394302,
+            0.0019758890848606825,
+            0.007328054402023554,
+            -0.005144984927028418,
+            0.0024036786053329706,
+            0.013351637870073318,
+            0.004903274122625589,
+            -0.008340937085449696,
+            -0.004887927323579788,
+            0.0011836150661110878,
+            -0.021715596318244934,
+            0.01838536001741886,
+            -0.0033263994846493006,
+            0.00854811817407608,
+            -0.004838050343096256,
+            0.004696093499660492,
+            -0.023158187046647072,
+            0.014126647263765335,
+            -0.008164450526237488,
+            -0.0069098565727472305,
+            0.008878071792423725,
+            -0.0023653118405491114,
+            0.02231411822140217,
+            -0.020273005589842796,
+            -0.008908765390515327,
+            0.008855052292346954,
+            -0.022129956632852554,
+            -0.02206856943666935,
+            -0.0004939722712151706,
+            -0.019275467842817307,
+            -0.0014780800556764007,
+            -0.00021233613369986415,
+            -0.032964736223220825,
+            -0.026856742799282074,
+            0.003044403623789549,
+            -0.0073472377844154835,
+            -0.008287223987281322,
+            0.012753116898238659,
+            -7.697334513068199e-05,
+            -0.013850406743586063,
+            -0.003665945492684841,
+            0.01041274331510067,
+            0.00023139962286222726,
+            0.01800169236958027,
+            -0.021930448710918427,
+            0.0020372758153826,
+            0.004431362729519606,
+            -0.004327772185206413,
+            0.006725695915520191,
+            0.01712692901492119,
+            0.0005990013014525175,
+            0.0003234798787161708,
+            0.006084970664232969,
+            0.0034932950511574745,
+            0.010850124061107635,
+            0.015745725482702255,
+            0.015032103285193443,
+            -0.004174305126070976,
+            0.016068005934357643,
+            -0.001623873831704259,
+            2.7096535632153973e-05,
+            -0.02460077777504921,
+            2.6736846848507412e-05,
+            0.012415489181876183,
+            0.004450546111911535,
+            -0.02410968206822872,
+            -0.0031997892074286938,
+            -0.03278057277202606,
+            0.005133474711328745,
+            -0.019382895901799202,
+            0.0013802447356283665,
+            0.017571983858942986,
+            0.00728201400488615,
+            -0.004446709528565407,
+            0.014402887783944607,
+            0.022789865732192993,
+            -0.0010502905352041125,
+            0.0006623064982704818,
+            0.004515769425779581,
+            -0.003224727464839816,
+            0.002537962282076478,
+            0.0022080079652369022,
+            -0.01597592607140541,
+            -0.01414966769516468,
+            0.008156776428222656,
+            0.004611686337739229,
+            0.011310526169836521,
+            0.007385604549199343,
+            -0.015446464531123638,
+            0.012223655357956886,
+            -0.013934813439846039,
+            0.008164450526237488,
+            -0.00473829684779048,
+            -0.013988526538014412,
+            -0.013466738164424896,
+            0.005359838716685772,
+            0.017618024721741676,
+            -0.006012073718011379,
+            0.033425137400627136,
+            -0.00925406627357006,
+            0.01533136423677206,
+            -0.019306162372231483,
+            -0.009491940960288048,
+            -0.010282295756042004,
+            0.01184766087681055,
+            -0.01567666418850422,
+            -0.0009020988363772631,
+            0.0035508451983332634,
+            -0.010573883540928364,
+            -0.010351356118917465,
+            0.005294615402817726,
+            -0.0030309753492474556,
+            0.03956381976604462,
+            -0.0024631470441818237,
+            0.025705739855766296,
+            0.006031257100403309,
+            -0.011517706327140331,
+            -0.010075115598738194,
+            0.002643470885232091,
+            0.00800330936908722,
+            0.0180937722325325,
+            -0.006568392273038626,
+            -0.004354629199951887,
+            0.011041957885026932,
+            0.017955651506781578,
+            0.007251320872455835,
+            0.011563746258616447,
+            -0.004093735013157129,
+            0.0037311690393835306,
+            0.002374903531745076,
+            -0.013666246086359024,
+            -0.0031096271704882383,
+            0.011916721239686012,
+            0.017019502818584442,
+            0.00786518957465887,
+            0.012852869927883148,
+            -0.0003946982033085078,
+            0.004611686337739229,
+            0.013996199704706669,
+            -0.013121437281370163,
+            -0.005240901838988066,
+            0.008241184055805206,
+            -0.01721900887787342,
+            0.009223372675478458,
+            0.0044083427637815475,
+            -0.004638543352484703,
+            0.00893945898860693,
+            -0.005348328500986099,
+            -0.0018483194289729,
+            -0.02099430002272129,
+            -0.014134320430457592,
+            -0.0033494196832180023,
+            0.020932912826538086,
+            -0.006299824919551611,
+            0.0015452218940481544,
+            -0.006790919695049524,
+            -0.02567504718899727,
+            0.01705019548535347,
+            0.004550299607217312,
+            0.0065530454739928246,
+            0.005966033786535263
+          ],
+          "title": "2 Related Work",
+          "keyphrases": [
+            "neural network capacity",
+            "model parallelism",
+            "conditional computation",
+            "mixture of experts",
+            "Switch Transformer"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "8468c5de-3375-4dc2-98b3-5a802dfc53c0",
+      "type": "next",
+      "source": {
+        "id": "3639a96c-e9a4-45ed-a3ac-1e18b2cfe8d4",
+        "properties": {
+          "page_content": "2 Related Work\n\nScaling: Various approaches have been proposed to scale up neural network capacity to improve performance. Recent works have successfully scaled models to billions of parameters via various forms of model parallelism [2, 21, 26, 27, 33]. Model parallelism [30] splits weights and tensors across multiple cores while pipeline parallelism [18, 24] splits different layers across devices with micro-batches pipelined to the different layers. To enable continued scaling of neural networks, improving model training and serving efficiency has become a critical research area.\n\nConditional Computation: Computation decisions can be made dynamically based on the input [23, 25]. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation, by activating certain parameters and computation on demand, on a per-example or per-token basis [3]. Conditional convolution layers [1] with task-specific gating has been used to combat catastrophic forgetting when a sequence of learning problems are optimized. The gating decisions may be binary or sparse and continuous, stochastic or deterministic.\n\nMixture of Experts: Sparsely-gated MoE [31] is the first model to demonstrate massive improvements in model capacity, training time, or model quality with gating. Switch Transformer [10] simplifies the gating by selecting only the top expert per token using a softmax over the hidden state and demonstrates better scaling than previous work. All the prior work requires an auxiliary loss to explicitly encourage balancing. This loss term has to be carefully weighted to not overwhelm the primary loss. However, auxiliary loss does not guarantee balancing and a hard capacity factor has to be imposed. As a result, many tokens can still be unprocessed by the MoE layer. Hard MoE [12] with a single decoding layer can be efficiently trained to good effect on large scale hashtag prediction tasks.\n\nBase Layers [22] formulate a linear assignment that maximizes token-expert affinities while ensuring each expert receives an equal number of tokens. Hash layers [8, 28] devise hashing techniques on input tokens. However, the evaluations are limited to pre-training perplexity. THOR [? ] randomly activates experts during training and inference and is trained with a consistency regularization loss.\n\nTHOR has demonstrated strong performance on translation tasks. Different from these prior works, our method is a learnt method that enables heterogeneous MoE and effectively improves downstream fine-tuning performance.\n\n",
+          "embedding": [
+            -0.013259558007121086,
+            -0.007197607308626175,
+            -0.015822459012269974,
+            0.013052377849817276,
+            0.029680538922548294,
+            0.02572108618915081,
+            -0.01866159960627556,
+            0.040392544120550156,
+            -0.039103418588638306,
+            0.017602676525712013,
+            0.020380431786179543,
+            -0.012024147436022758,
+            0.0011442891554906964,
+            0.006088807247579098,
+            0.009645408019423485,
+            -0.014533335343003273,
+            -0.004703766666352749,
+            0.0049493140541017056,
+            0.02260570414364338,
+            -0.03012559376657009,
+            -0.008847379125654697,
+            -0.007220627274364233,
+            -0.03612615540623665,
+            -0.01026694942265749,
+            0.03443801775574684,
+            0.0267186239361763,
+            0.03146075829863548,
+            -0.011226118542253971,
+            -0.04496586322784424,
+            0.025629006326198578,
+            0.015055122785270214,
+            0.009300106205046177,
+            -0.016912074759602547,
+            -0.01592988520860672,
+            -0.0355122871696949,
+            -0.04048462212085724,
+            0.05073622614145279,
+            0.013781346380710602,
+            -0.01992003060877323,
+            0.02774685248732567,
+            0.021807676181197166,
+            0.0150397764518857,
+            -0.03502119332551956,
+            -0.02219134382903576,
+            0.00023379754566121846,
+            0.0007141016540117562,
+            -0.032136011868715286,
+            -0.009069905616343021,
+            -0.008317917585372925,
+            -0.00694822333753109,
+            0.028268640860915184,
+            -0.04450546205043793,
+            0.01573037914931774,
+            0.012584302574396133,
+            -0.038274697959423065,
+            0.039932142943143845,
+            0.013228864409029484,
+            0.005678283050656319,
+            0.053621407598257065,
+            0.03726181387901306,
+            0.011333545669913292,
+            0.006449454929679632,
+            -0.04613221064209938,
+            0.002123601036146283,
+            0.010443436913192272,
+            -0.008847379125654697,
+            -0.00621158117428422,
+            0.0015471403021365404,
+            0.0041819787584245205,
+            0.010289969854056835,
+            -0.0367400236427784,
+            -0.0014166932087391615,
+            -0.011893700808286667,
+            -0.003069342114031315,
+            0.010834777727723122,
+            0.001008087070658803,
+            0.019367549568414688,
+            0.039011336863040924,
+            0.03511327505111694,
+            -0.008678564801812172,
+            -0.01622147299349308,
+            0.03317958861589432,
+            -0.002503432100638747,
+            0.016574447974562645,
+            0.02119380794465542,
+            -0.03713903948664665,
+            -0.0017399332718923688,
+            -0.06629779189825058,
+            -0.018983881920576096,
+            -0.01323653757572174,
+            -0.02991073951125145,
+            0.0056015495210886,
+            -0.01734178327023983,
+            -0.0033398279920220375,
+            -0.020273005589842796,
+            0.04646984115242958,
+            -0.018446745350956917,
+            -0.013482085429131985,
+            -0.032872654497623444,
+            0.004730623681098223,
+            -0.01812446489930153,
+            0.03631031885743141,
+            -0.0027451429050415754,
+            -0.0076541719026863575,
+            0.01571503095328808,
+            0.009092926047742367,
+            -0.05006096884608269,
+            0.034622177481651306,
+            -0.00013188579760026187,
+            0.074830561876297,
+            0.012898909859359264,
+            -0.033210281282663345,
+            0.00029134770738892257,
+            0.04054601117968559,
+            -0.03152214363217354,
+            -0.02452404424548149,
+            -0.04149750620126724,
+            -0.03597268834710121,
+            0.001932726358063519,
+            -0.028406761586666107,
+            0.024416616186499596,
+            0.00759278517216444,
+            -0.0010752289090305567,
+            -0.012430835515260696,
+            -0.05288476496934891,
+            0.050889693200588226,
+            -0.013812039978802204,
+            -0.01692742109298706,
+            -0.005755016580224037,
+            -0.02243689075112343,
+            -0.011356566101312637,
+            0.023541854694485664,
+            -0.032872654497623444,
+            -0.05638381466269493,
+            0.01705019548535347,
+            0.016160085797309875,
+            0.051380787044763565,
+            0.02021161839365959,
+            -0.018738333135843277,
+            -0.010044422000646591,
+            0.03278057277202606,
+            -0.005781873129308224,
+            0.02360324189066887,
+            -0.022022530436515808,
+            -0.028821121901273727,
+            0.013942486606538296,
+            -0.033455830067396164,
+            0.06227695196866989,
+            -0.01651306077837944,
+            0.023787401616573334,
+            -0.013474412262439728,
+            0.04542626440525055,
+            -0.03324097394943237,
+            0.03235086426138878,
+            -0.029388951137661934,
+            0.021500742062926292,
+            0.005755016580224037,
+            -0.00864787120372057,
+            0.0028065296355634928,
+            0.00858648493885994,
+            -0.008279550820589066,
+            -0.020902220159769058,
+            0.005098944529891014,
+            0.030800849199295044,
+            0.0026933476328849792,
+            0.024278495460748672,
+            0.04907878115773201,
+            0.02077944576740265,
+            -0.052915457636117935,
+            0.029726577922701836,
+            0.029695885255932808,
+            0.020426472648978233,
+            0.0444747656583786,
+            0.007554418407380581,
+            0.010604577139019966,
+            0.006587575655430555,
+            0.014993736520409584,
+            0.0070403036661446095,
+            -0.01655910164117813,
+            0.03879648447036743,
+            -0.0241403765976429,
+            -0.0031268922612071037,
+            0.024784937500953674,
+            0.0015365893486887217,
+            0.036617252975702286,
+            0.005482612177729607,
+            0.013512779027223587,
+            0.02136262133717537,
+            0.00933079980313778,
+            0.027808239683508873,
+            0.02021161839365959,
+            0.030171632766723633,
+            0.020150231197476387,
+            -0.024401269853115082,
+            0.03198254480957985,
+            -0.056813523173332214,
+            -0.015945231541991234,
+            -0.009323126636445522,
+            0.028283987194299698,
+            -0.006169377826154232,
+            -0.01812446489930153,
+            -0.017894264310598373,
+            0.008049349300563335,
+            0.002048785798251629,
+            -0.009676100686192513,
+            0.008931785821914673,
+            0.033916231244802475,
+            0.026288915425539017,
+            0.024002255871891975,
+            -0.037937067449092865,
+            -0.0052025350742042065,
+            -0.010903838090598583,
+            -0.013374658301472664,
+            -0.034990500658750534,
+            -0.00218306970782578,
+            0.009461247362196445,
+            0.0012622670037671924,
+            0.009630060754716396,
+            -0.002929303329437971,
+            0.0027355512138456106,
+            0.03015628643333912,
+            -0.0046423799358308315,
+            0.045610424131155014,
+            -0.014425908215343952,
+            -0.0034817850682884455,
+            -0.020503206178545952,
+            -0.006576065439730883,
+            0.003627578727900982,
+            0.01622147299349308,
+            0.011824640445411205,
+            0.0031134639866650105,
+            0.012661036103963852,
+            0.004577156621962786,
+            -0.004465892910957336,
+            0.04435199499130249,
+            -0.016697220504283905,
+            0.01414966769516468,
+            -0.011064978316426277,
+            0.008195143193006516,
+            0.003395459847524762,
+            -0.005912320222705603,
+            0.041466813534498215,
+            -0.047851044684648514,
+            -0.0132749043405056,
+            0.020718058571219444,
+            0.0033820313401520252,
+            -0.03176769241690636,
+            0.012001127935945988,
+            -0.0009505368652753532,
+            -0.01609869860112667,
+            -0.01435684785246849,
+            -0.02364928089082241,
+            -0.005543999373912811,
+            0.00993699487298727,
+            0.00022012938279658556,
+            0.05666005611419678,
+            0.018078425899147987,
+            -0.01528532337397337,
+            -0.01978190988302231,
+            -0.02140866219997406,
+            0.03962520509958267,
+            -0.004730623681098223,
+            -0.035819221287965775,
+            0.013934813439846039,
+            -0.012039494700729847,
+            -0.01883041486144066,
+            0.030938968062400818,
+            0.01470982190221548,
+            -0.003180605825036764,
+            -0.019029920920729637,
+            0.008862725459039211,
+            -0.05260852351784706,
+            -0.005716649815440178,
+            0.0007275299867615104,
+            0.06727997958660126,
+            -0.024984445422887802,
+            -0.010934531688690186,
+            0.0050490680150687695,
+            0.0011596358381211758,
+            0.02364928089082241,
+            -0.00858648493885994,
+            -0.003474111668765545,
+            -0.021961143240332603,
+            0.03003351204097271,
+            -0.008010982535779476,
+            0.005578529555350542,
+            -0.026979517191648483,
+            -0.017955651506781578,
+            -0.04849560558795929,
+            -0.02978796511888504,
+            -0.014510314911603928,
+            0.012607323005795479,
+            -0.05273129791021347,
+            0.017648717388510704,
+            0.026856742799282074,
+            -0.021086379885673523,
+            0.009668427519500256,
+            0.012208308093249798,
+            0.027286451309919357,
+            -0.034407325088977814,
+            -0.05260852351784706,
+            -0.02941964380443096,
+            0.014157340861856937,
+            -0.016942769289016724,
+            -0.05512538552284241,
+            0.0036122319288551807,
+            0.012868217192590237,
+            0.015546217560768127,
+            -0.004849560558795929,
+            -0.0013322862796485424,
+            0.02133192867040634,
+            -0.006986590102314949,
+            0.0062768044881522655,
+            -0.01721900887787342,
+            -0.00873227883130312,
+            0.011793946847319603,
+            0.010282295756042004,
+            -0.000877640035469085,
+            -0.03738458827137947,
+            -0.013466738164424896,
+            0.0236953217536211,
+            0.018600212410092354,
+            -0.07317311316728592,
+            0.011487012729048729,
+            -0.027501305565238,
+            -0.007170750759541988,
+            -0.0042126718908548355,
+            0.03302612155675888,
+            -0.019060615450143814,
+            0.016083352267742157,
+            -0.0020583774894475937,
+            -0.02808448113501072,
+            -0.027639426290988922,
+            -0.006537698674947023,
+            -0.03410039097070694,
+            -0.03207462653517723,
+            -0.028468148782849312,
+            -0.004178142175078392,
+            -0.03143006190657616,
+            -0.029220137745141983,
+            -0.024508696049451828,
+            0.0023173533845692873,
+            -0.00552865257486701,
+            0.017357129603624344,
+            0.01899922825396061,
+            -0.03373207151889801,
+            -0.045364875346422195,
+            0.013190497644245625,
+            -0.006587575655430555,
+            0.01259197574108839,
+            0.0009654040331952274,
+            -0.012062514200806618,
+            0.016282860189676285,
+            -0.003094280604273081,
+            -0.03015628643333912,
+            -0.046193599700927734,
+            -0.013297924771904945,
+            0.014610068872570992,
+            0.008095390163362026,
+            -0.0002774397435132414,
+            0.010006055235862732,
+            -0.04686885327100754,
+            -0.024171069264411926,
+            0.01066596433520317,
+            -0.04008561000227928,
+            -0.024002255871891975,
+            -0.0045694829896092415,
+            -0.025444846600294113,
+            -0.006391904782503843,
+            -0.006844632793217897,
+            -0.0222373828291893,
+            0.03729250654578209,
+            0.018262585625052452,
+            -0.00933079980313778,
+            0.02941964380443096,
+            0.012162268161773682,
+            0.020672019571065903,
+            -0.01617543213069439,
+            -0.04640845209360123,
+            0.030048860237002373,
+            -0.019444283097982407,
+            -0.03098500892519951,
+            8.272836566902697e-05,
+            -0.022252731025218964,
+            0.007792292162775993,
+            0.00012229410640429705,
+            0.015945231541991234,
+            -0.006422598380595446,
+            -0.04573319852352142,
+            0.02932756394147873,
+            0.004872580524533987,
+            -0.04956987500190735,
+            -0.0022137630730867386,
+            -0.04791243001818657,
+            -0.0065607186406850815,
+            0.030340446159243584,
+            -0.004914783872663975,
+            -0.027977053076028824,
+            -0.03459148481488228,
+            -0.0015605686930939555,
+            -0.011433299630880356,
+            0.014625415205955505,
+            -0.013643225654959679,
+            0.013727632351219654,
+            -0.008041676133871078,
+            -0.0188764538615942,
+            -0.010719677433371544,
+            0.015776418149471283,
+            0.03759944066405296,
+            -0.024155722931027412,
+            -0.022682437673211098,
+            -0.03572714328765869,
+            -0.03953312709927559,
+            0.013221191242337227,
+            0.024201761931180954,
+            0.01867694780230522,
+            -0.010972898453474045,
+            0.018078425899147987,
+            0.0068369596265256405,
+            -0.014878636226058006,
+            -0.025153258815407753,
+            0.0029810986015945673,
+            0.05441943556070328,
+            0.00807236973196268,
+            -0.00013368423969950527,
+            0.002990690292790532,
+            0.009798875078558922,
+            0.017188316211104393,
+            0.005555509123951197,
+            -0.014280114322900772,
+            0.04373812675476074,
+            -0.014433581382036209,
+            0.039686594158411026,
+            -0.01065829023718834,
+            -0.02070271223783493,
+            -0.03634101152420044,
+            0.023679975420236588,
+            0.005313798785209656,
+            -0.03443801775574684,
+            -0.012361775152385235,
+            0.029358256608247757,
+            0.005720486398786306,
+            -0.010773390531539917,
+            0.012599649839103222,
+            -0.035358820110559464,
+            -0.029557764530181885,
+            -0.022713132202625275,
+            -0.037108346819877625,
+            -0.00947659369558096,
+            -0.03453009948134422,
+            -0.06365815550088882,
+            -0.004247202072292566,
+            -0.061294760555028915,
+            -0.014878636226058006,
+            0.02820725366473198,
+            -0.0006987549131736159,
+            0.0162982065230608,
+            -0.00065223517594859,
+            -0.007700212299823761,
+            -0.03833608329296112,
+            0.005206371657550335,
+            0.04174305126070976,
+            0.01746455766260624,
+            -0.0380905345082283,
+            -0.017280396074056625,
+            -0.009031538851559162,
+            -0.04263316094875336,
+            -0.029342910274863243,
+            0.009031538851559162,
+            -0.013643225654959679,
+            6.396460958058015e-05,
+            -0.03941035270690918,
+            0.003128810552880168,
+            0.006840796209871769,
+            0.018109118565917015,
+            0.04444407299160957,
+            0.022129956632852554,
+            -0.06040465459227562,
+            0.007987963035702705,
+            -0.003629497019574046,
+            0.027240412309765816,
+            -0.014072933234274387,
+            0.018017038702964783,
+            -0.0006474393303506076,
+            -0.007009610068053007,
+            -0.02662654221057892,
+            0.009054559282958508,
+            -0.0012507569044828415,
+            -0.011878354474902153,
+            -0.0038596978411078453,
+            0.004316262435168028,
+            0.00914663914591074,
+            0.03182907775044441,
+            0.010282295756042004,
+            0.009537980891764164,
+            -0.04082224890589714,
+            -0.012860543094575405,
+            -0.01130285207182169,
+            0.04042323678731918,
+            0.009246393106877804,
+            -0.034990500658750534,
+            0.007316544186323881,
+            0.03253502771258354,
+            0.028406761586666107,
+            0.028115173801779747,
+            -0.011072651483118534,
+            0.012730096466839314,
+            0.020349739119410515,
+            -0.07004238665103912,
+            -0.005758853163570166,
+            -0.0069904266856610775,
+            -0.0037503524217754602,
+            0.0037407607305794954,
+            -0.02289729192852974,
+            -0.00019531087309587747,
+            0.042541082948446274,
+            -0.016896728426218033,
+            -0.00151644682046026,
+            -0.011003591120243073,
+            0.029235484078526497,
+            0.002560982247814536,
+            0.004086061846464872,
+            -0.012607323005795479,
+            0.058470968157052994,
+            -0.017863571643829346,
+            0.013343964703381062,
+            0.006077297497540712,
+            -0.01306772418320179,
+            0.024861671030521393,
+            -0.021715596318244934,
+            -0.007051813416182995,
+            0.005056741181761026,
+            0.0094382269307971,
+            -0.017157621681690216,
+            -0.003107708878815174,
+            -0.0026991027407348156,
+            -0.01741851679980755,
+            0.010811757296323776,
+            -0.0489560067653656,
+            -0.03471425920724869,
+            -0.0022463747300207615,
+            0.01005976926535368,
+            -0.020027456805109978,
+            0.010305316187441349,
+            0.014326154254376888,
+            0.007934249006211758,
+            0.055739253759384155,
+            -0.03143006190657616,
+            -0.0016229146858677268,
+            -0.003309134393930435,
+            -0.023925522342324257,
+            -0.028636962175369263,
+            -0.023004719987511635,
+            -0.0056130592711269855,
+            -0.015822459012269974,
+            -0.04386089742183685,
+            -0.008693911135196686,
+            -0.021439354866743088,
+            0.021270541474223137,
+            -0.04094502329826355,
+            0.016912074759602547,
+            0.008724604733288288,
+            0.0006124296924099326,
+            -0.009008519351482391,
+            -0.01995072327554226,
+            -0.0042625488713383675,
+            0.029757272452116013,
+            0.032166704535484314,
+            0.018861107528209686,
+            0.034867726266384125,
+            -0.005636079702526331,
+            -0.0231888797134161,
+            -0.013581839390099049,
+            0.03572714328765869,
+            -0.00854811817407608,
+            0.020058151334524155,
+            0.004980007652193308,
+            -0.0006061950698494911,
+            -0.0224061980843544,
+            -0.0077270688489079475,
+            0.03646378591656685,
+            -0.02555227279663086,
+            0.014425908215343952,
+            0.05122731998562813,
+            0.032749880105257034,
+            0.0006071542156860232,
+            -0.008640198037028313,
+            -0.01472516916692257,
+            0.005919993855059147,
+            -0.021792329847812653,
+            0.03198254480957985,
+            0.01987398974597454,
+            -0.001803238526917994,
+            -0.010535516776144505,
+            0.01325188484042883,
+            -0.030217673629522324,
+            0.012469202280044556,
+            0.006668145768344402,
+            0.003606477053835988,
+            -0.034867726266384125,
+            -0.02136262133717537,
+            -0.002419025171548128,
+            -0.0007759680738672614,
+            -0.01929081603884697,
+            -0.015553891658782959,
+            -0.02286659926176071,
+            0.002154294401407242,
+            -0.006821612827479839,
+            0.004281732253730297,
+            0.016697220504283905,
+            0.02837606891989708,
+            -0.0006138684111647308,
+            0.03701626509428024,
+            0.00303289364092052,
+            0.023879481479525566,
+            0.004945477470755577,
+            -0.003512478433549404,
+            -0.0070403036661446095,
+            -0.011755580082535744,
+            -0.014487294480204582,
+            0.057273924350738525,
+            0.017019502818584442,
+            0.05957593023777008,
+            -0.011064978316426277,
+            -0.010819431394338608,
+            0.030938968062400818,
+            0.004316262435168028,
+            -0.020733406767249107,
+            0.008670891635119915,
+            -0.016160085797309875,
+            -0.02871369570493698,
+            -0.0027470611967146397,
+            -0.03799845650792122,
+            -0.0188764538615942,
+            -0.022621052339673042,
+            -0.018308626487851143,
+            0.004684583283960819,
+            0.0196591354906559,
+            0.016436327248811722,
+            -0.00016737506666686386,
+            -0.011763254180550575,
+            0.031583528965711594,
+            0.019306162372231483,
+            0.028928549960255623,
+            -0.011809294112026691,
+            0.007331890985369682,
+            -0.01576107181608677,
+            0.0052831051871180534,
+            0.019137348979711533,
+            -0.01253058947622776,
+            0.009453574195504189,
+            -0.0008905887953005731,
+            0.011180078610777855,
+            0.03821330890059471,
+            -0.010036748833954334,
+            -0.03953312709927559,
+            -0.006280641537159681,
+            -0.014778882265090942,
+            0.007278177421540022,
+            0.0067065125331282616,
+            -0.020932912826538086,
+            0.012254348024725914,
+            0.007730905432254076,
+            -0.01479422952979803,
+            0.02128588780760765,
+            -0.007205280475318432,
+            -0.027624079957604408,
+            0.03468356654047966,
+            -0.014134320430457592,
+            0.01787891797721386,
+            -0.017986344173550606,
+            -0.0028755899984389544,
+            -0.025828514248132706,
+            0.022559665143489838,
+            0.01751059666275978,
+            -0.027317145839333534,
+            0.022989371791481972,
+            0.04564111679792404,
+            0.020150231197476387,
+            0.0007615805370733142,
+            0.012415489181876183,
+            0.0057665263302624226,
+            -0.036494478583335876,
+            -0.02057993970811367,
+            -0.006944386754184961,
+            -0.0027950196526944637,
+            -0.016574447974562645,
+            0.030217673629522324,
+            -0.03824400156736374,
+            -0.0031518307514488697,
+            -0.0052025350742042065,
+            -0.024078989401459694,
+            0.0055823661386966705,
+            0.009069905616343021,
+            0.003951778169721365,
+            0.017142275348305702,
+            -0.014172687195241451,
+            0.03020232729613781,
+            0.03566575422883034,
+            0.006322844885289669,
+            -0.017848225310444832,
+            0.011256812140345573,
+            -0.024508696049451828,
+            0.0048457239754498005,
+            0.043001484125852585,
+            0.014072933234274387,
+            0.008494404144585133,
+            -0.020426472648978233,
+            -0.02584386058151722,
+            0.007235974073410034,
+            0.008709258399903774,
+            0.04198860004544258,
+            0.02630426175892353,
+            0.03606477007269859,
+            0.009369166567921638,
+            -0.01199345476925373,
+            -0.013819713145494461,
+            -0.0049493140541017056,
+            -0.023265613242983818,
+            0.021761635318398476,
+            -0.02228342369198799,
+            0.01717296987771988,
+            0.012039494700729847,
+            -0.028759736567735672,
+            0.00017528820899315178,
+            0.01576107181608677,
+            0.03575783595442772,
+            0.04361535236239433,
+            0.023925522342324257,
+            0.0032765227369964123,
+            0.039686594158411026,
+            -0.04493517056107521,
+            -0.03716973215341568,
+            0.00753907160833478,
+            0.017571983858942986,
+            0.009154313243925571,
+            0.0058164033107459545,
+            0.03290334716439247,
+            -0.00947659369558096,
+            -0.003376276232302189,
+            0.012024147436022758,
+            0.02074875310063362,
+            0.002031520940363407,
+            -0.00429324246942997,
+            -0.008095390163362026,
+            0.02289729192852974,
+            0.021086379885673523,
+            0.03937966004014015,
+            -0.020058151334524155,
+            0.01780218444764614,
+            0.0009936995338648558,
+            -0.02552158012986183,
+            0.02559831365942955,
+            0.00021701208606828004,
+            -0.04530349001288414,
+            0.03339444100856781,
+            0.017249703407287598,
+            0.015108836814761162,
+            0.011786273680627346,
+            -0.006131011061370373,
+            0.012553608976304531,
+            0.030524607747793198,
+            -0.0031019540037959814,
+            0.0011423707474023104,
+            0.03115382231771946,
+            -0.003311052918434143,
+            -0.004807357210665941,
+            0.020886873826384544,
+            0.004887927323579788,
+            -0.013167478144168854,
+            0.01867694780230522,
+            -0.012169941328465939,
+            0.044075753539800644,
+            -0.024692857638001442,
+            0.006760226096957922,
+            0.010205562226474285,
+            -0.01883041486144066,
+            -0.007293524220585823,
+            -0.0013188580051064491,
+            -0.022268077358603477,
+            0.004500423092395067,
+            0.018155159428715706,
+            -0.012331082485616207,
+            0.0295731108635664,
+            -0.028606269508600235,
+            0.0012430836213752627,
+            0.015945231541991234,
+            -0.026933476328849792,
+            0.02767011895775795,
+            0.013006336987018585,
+            0.02713298425078392,
+            -0.009576347656548023,
+            -0.014456601813435555,
+            0.04797381907701492,
+            0.0563531219959259,
+            0.0060504404827952385,
+            -0.008578810840845108,
+            0.013742979615926743,
+            0.0003174850717186928,
+            -0.02360324189066887,
+            0.024048294872045517,
+            -0.03572714328765869,
+            0.0162982065230608,
+            0.03107708878815174,
+            -0.014617742039263248,
+            -0.014095953665673733,
+            -0.034622177481651306,
+            -0.0052831051871180534,
+            0.00429324246942997,
+            0.004381485749036074,
+            0.0036966388579458,
+            -0.025659700855612755,
+            -0.015599931590259075,
+            0.022252731025218964,
+            0.005379022099077702,
+            0.0010157604701817036,
+            -0.025076525285840034,
+            0.015530871227383614,
+            -0.01617543213069439,
+            -0.04162028059363365,
+            -0.006330518051981926,
+            -0.012492222711443901,
+            0.025214646011590958,
+            -6.204627425177023e-05,
+            -0.03477564454078674,
+            -0.006718022748827934,
+            -0.005191024858504534,
+            -0.009054559282958508,
+            0.044536154717206955,
+            0.023741360753774643,
+            0.03812123090028763,
+            -0.02170024812221527,
+            -0.03141471743583679,
+            0.02460077777504921,
+            -0.03489841893315315,
+            0.030770154669880867,
+            -0.0032439110800623894,
+            0.007186097092926502,
+            -0.0052025350742042065,
+            0.039348967373371124,
+            -0.012146921828389168,
+            0.008571137674152851,
+            -0.03588061034679413,
+            -0.004980007652193308,
+            -0.01945962943136692,
+            -0.008517424575984478,
+            0.0031153822783380747,
+            0.011870680376887321,
+            -0.009568674489855766,
+            0.005659099668264389,
+            0.04720648378133774,
+            0.031583528965711594,
+            0.0018675028113648295,
+            0.026565156877040863,
+            -0.02862161584198475,
+            -0.017986344173550606,
+            0.01464076153934002,
+            -0.0064072515815496445,
+            -0.00785751547664404,
+            -0.013973180204629898,
+            -0.031798385083675385,
+            -0.012799156829714775,
+            0.011187751777470112,
+            0.013581839390099049,
+            -0.011602113023400307,
+            0.003735005622729659,
+            0.012906583957374096,
+            -0.014172687195241451,
+            -0.008057023398578167,
+            -0.0393182709813118,
+            0.0011222282191738486,
+            0.024370577186346054,
+            -0.027977053076028824,
+            0.013121437281370163,
+            0.002286659786477685,
+            0.007389441132545471,
+            -0.023618588224053383,
+            -0.02087152563035488,
+            0.008218163624405861,
+            -0.012154594995081425,
+            0.005248575005680323,
+            -0.002712531015276909,
+            0.034744951874017715,
+            -0.00800330936908722,
+            0.03351721540093422,
+            -0.012975644320249557,
+            0.01953636296093464,
+            -0.025506233796477318,
+            -0.02389482781291008,
+            -0.04272524267435074,
+            -0.0012977562146261334,
+            0.0029580784030258656,
+            -0.02820725366473198,
+            -0.006495495326817036,
+            -0.001958623994141817,
+            0.017249703407287598,
+            -0.012246674858033657,
+            -0.003274604445323348,
+            0.006622105836868286,
+            -0.01354347262531519,
+            -0.0020104190334677696,
+            -0.03987075388431549,
+            -0.02825329452753067,
+            -0.03406969830393791,
+            0.016835341230034828,
+            -0.022667091339826584,
+            0.017234355211257935,
+            -0.01995072327554226,
+            0.0005678283050656319,
+            -0.047359950840473175,
+            -0.01573037914931774,
+            0.02991073951125145,
+            0.008141430094838142,
+            -0.04204998537898064,
+            0.021470047533512115,
+            0.01755663752555847,
+            0.017479903995990753,
+            -0.02311214618384838,
+            0.018492786213755608,
+            0.002338455058634281,
+            0.004933967255055904,
+            0.024416616186499596,
+            -0.02535276673734188,
+            0.02177698165178299,
+            -0.024907711893320084,
+            -0.021838368847966194,
+            -0.004020838066935539,
+            0.013827386312186718,
+            0.01726504974067211,
+            0.0020104190334677696,
+            -0.01709623634815216,
+            0.016743261367082596,
+            0.018323972821235657,
+            0.003915329463779926,
+            0.022636398673057556,
+            -0.02664189040660858,
+            0.016620486974716187,
+            0.011809294112026691,
+            -0.02323492057621479,
+            0.0036256604362279177,
+            -0.01483259629458189,
+            -0.007174587342888117,
+            0.017525942996144295,
+            0.03146075829863548,
+            0.014617742039263248,
+            -0.03683210536837578,
+            0.012860543094575405,
+            0.011747906915843487,
+            -0.0014147749170660973,
+            0.038274697959423065,
+            0.0027240412309765816,
+            0.0239715613424778,
+            0.03397761657834053,
+            0.0066105956211686134,
+            0.017664063721895218,
+            0.03548159450292587,
+            -0.022912638261914253,
+            -0.009277086704969406,
+            -0.023802747949957848,
+            0.014111299999058247,
+            0.014310807920992374,
+            -0.012753116898238659,
+            -0.012001127935945988,
+            -0.015239283442497253,
+            -0.014118974097073078,
+            0.01879972033202648,
+            -0.0008503036806359887,
+            -0.015745725482702255,
+            -0.029711231589317322,
+            0.024692857638001442,
+            0.0006781327538192272,
+            -0.044658929109573364,
+            0.012323408387601376,
+            -0.014218727126717567,
+            -0.0037868008948862553,
+            -0.0027835096698254347,
+            0.007769272197037935,
+            -0.005605386104434729,
+            0.0033359911758452654,
+            0.004266385454684496,
+            0.05135009437799454,
+            -0.0014166932087391615,
+            0.017311088740825653,
+            -0.008839705027639866,
+            0.01970517635345459,
+            -0.010988244786858559,
+            0.022912638261914253,
+            -0.014556354843080044,
+            -0.051503561437129974,
+            0.02618148922920227,
+            -0.011041957885026932,
+            0.040883637964725494,
+            0.05098177120089531,
+            -0.01841605268418789,
+            -0.010750371031463146,
+            0.03575783595442772,
+            0.004680746700614691,
+            -0.025337418541312218,
+            0.007888209074735641,
+            -0.0012296552304178476,
+            -0.007504541426897049,
+            -0.019613096490502357,
+            -0.012077861465513706,
+            0.019720522686839104,
+            -0.0038712078239768744,
+            0.017372475937008858,
+            0.040638089179992676,
+            0.03161422535777092,
+            -0.021301234140992165,
+            0.009522633627057076,
+            0.01331327110528946,
+            -0.009215699508786201,
+            -0.007796129211783409,
+            0.0254295002669096,
+            -0.017034849151968956,
+            -0.0008320794440805912,
+            -0.012177614495158195,
+            -0.00400165468454361,
+            -0.035941995680332184,
+            0.001303511206060648,
+            -0.016605140641331673,
+            -0.0018761353567242622,
+            -0.007692538667470217,
+            -0.030923621729016304,
+            0.03228947892785072,
+            0.028698349371552467,
+            -0.0070633236318826675,
+            -0.009077579714357853,
+            -0.009468920528888702,
+            0.005037557799369097,
+            0.0036333338357508183,
+            -0.0019969907589256763,
+            0.0034357449039816856,
+            -0.01792495884001255,
+            0.007443154696375132,
+            -0.006775572896003723,
+            -0.01325188484042883,
+            -0.051135238260030746,
+            -0.01867694780230522,
+            0.025045832619071007,
+            0.004830377176403999,
+            -0.01692742109298706,
+            -0.003261175937950611,
+            0.008103063330054283,
+            -0.005862443707883358,
+            -0.02496909908950329,
+            0.03135332837700844,
+            0.01668187417089939,
+            0.01875368133187294,
+            -0.0017456883797422051,
+            -0.0022578847128897905,
+            0.01746455766260624,
+            0.006084970664232969,
+            -0.01547715812921524,
+            -0.009499614126980305,
+            0.021761635318398476,
+            0.004120592027902603,
+            0.020932912826538086,
+            -0.010735023766756058,
+            0.021086379885673523,
+            0.007957269437611103,
+            -0.012123901396989822,
+            -0.0027202044147998095,
+            -0.0012718586949631572,
+            0.004105245228856802,
+            0.016912074759602547,
+            -0.013980853371322155,
+            0.024953750893473625,
+            0.020088844001293182,
+            0.007109363563358784,
+            0.0044889128766953945,
+            -0.01668187417089939,
+            -0.02452404424548149,
+            0.016574447974562645,
+            0.002522615483030677,
+            0.027977053076028824,
+            0.012139247730374336,
+            -0.004013164900243282,
+            -0.02759338542819023,
+            -0.013052377849817276,
+            -0.024217110127210617,
+            0.01005976926535368,
+            0.023664627224206924,
+            0.030632033944129944,
+            -0.015461810864508152,
+            -0.013489758595824242,
+            -0.025276033207774162,
+            0.00469993008300662,
+            0.008908765390515327,
+            -0.027347838506102562,
+            -0.01992003060877323,
+            0.018032385036349297,
+            -0.025291379541158676,
+            0.026488423347473145,
+            -0.0019269713666290045,
+            -0.012852869927883148,
+            0.006675818935036659,
+            0.07071764022111893,
+            -0.013090744614601135,
+            -0.0012200635392218828,
+            -0.028652308508753777,
+            -0.017618024721741676,
+            -0.005931503605097532,
+            -0.008617177605628967,
+            0.0004412179405335337,
+            0.018492786213755608,
+            -0.008624851703643799,
+            0.020902220159769058,
+            0.01668187417089939,
+            0.0013006337685510516,
+            0.0032074626069515944,
+            -0.03526674211025238,
+            0.03032509982585907,
+            -0.004224182106554508,
+            0.0014838350471109152,
+            0.00244204537011683,
+            0.006307498086243868,
+            0.018400706350803375,
+            0.016896728426218033,
+            0.005931503605097532,
+            0.009200353175401688,
+            -0.005912320222705603,
+            -0.003738842438906431,
+            0.016114046797156334,
+            0.019858643412590027,
+            -0.01111869141459465,
+            -0.012944950722157955,
+            0.005728159565478563,
+            0.014901655726134777,
+            -0.004853397142142057,
+            -0.00414744857698679,
+            0.0075736017897725105,
+            0.020472511649131775,
+            -0.01617543213069439,
+            -0.002286659786477685,
+            0.004742133431136608,
+            -0.017372475937008858,
+            0.0007548663415946066,
+            0.01875368133187294,
+            0.007565928157418966,
+            0.015853151679039,
+            -0.015484831295907497,
+            -0.009161986410617828,
+            0.0021255193278193474,
+            -0.022421544417738914,
+            0.003121137386187911,
+            -0.012192961759865284,
+            -0.01058155670762062,
+            -0.00239600520581007,
+            -0.007416298147290945,
+            -0.006487822160124779,
+            -0.006568392273038626,
+            0.004630869720131159,
+            0.009430553764104843,
+            -0.004151285160332918,
+            0.01738782227039337,
+            0.005252411589026451,
+            0.013988526538014412,
+            -0.0053560021333396435,
+            0.01489398255944252,
+            0.02136262133717537,
+            -0.010213236324489117,
+            0.00075822341023013,
+            0.01775614358484745,
+            -0.0006344905705191195,
+            0.006741042714565992,
+            0.004013164900243282,
+            1.707021874608472e-05,
+            0.010842450894415379,
+            -0.004343118984252214,
+            -0.01992003060877323,
+            -0.0055094691924750805,
+            -0.011440972797572613,
+            0.002133192727342248,
+            0.002252129837870598,
+            0.03716973215341568,
+            -0.02513791248202324,
+            0.03235086426138878,
+            0.012914257124066353,
+            -0.014011546969413757,
+            -0.00569362984970212,
+            -0.047482721507549286,
+            0.01338233146816492,
+            -0.014694475568830967,
+            3.977555752499029e-05,
+            0.017970997840166092,
+            -0.00914663914591074,
+            -0.0014799984637647867,
+            0.00502988463267684,
+            -0.005482612177729607,
+            -0.0009457410196773708,
+            -0.01924477517604828,
+            -0.0067563895136117935,
+            0.010382049717009068,
+            0.013842732645571232,
+            -0.03471425920724869,
+            -0.02243689075112343,
+            -0.030263712629675865,
+            0.00392875773832202,
+            0.01883041486144066,
+            -0.010942204855382442,
+            -0.025828514248132706,
+            -0.009537980891764164,
+            0.013405351899564266,
+            -0.01697346195578575,
+            0.006388068199157715,
+            -0.003997818101197481,
+            0.014326154254376888,
+            -0.008832031860947609,
+            0.001874217065051198,
+            -0.01899922825396061,
+            0.01487096305936575,
+            0.0028314681258052588,
+            -0.00420116214081645,
+            -0.030340446159243584,
+            0.007120873779058456,
+            -0.00020574184600263834,
+            -0.01450264174491167,
+            -0.012269695289433002,
+            0.017280396074056625,
+            -0.011134038679301739,
+            -0.004745970480144024,
+            0.016743261367082596,
+            0.006695002317428589,
+            0.0006618269253522158,
+            0.018277931958436966,
+            0.005352165549993515,
+            -0.0005002068355679512,
+            0.005336818750947714,
+            0.013106090947985649,
+            0.0012517160503193736,
+            0.024048294872045517,
+            -0.01617543213069439,
+            0.00997536163777113,
+            -0.014333827421069145,
+            0.033701375126838684,
+            -0.01597592607140541,
+            -0.0016027720412239432,
+            0.012177614495158195,
+            0.040883637964725494,
+            -0.013497431762516499,
+            0.01712692901492119,
+            -0.0291280560195446,
+            -0.015354383736848831,
+            0.03465287387371063,
+            0.01688138209283352,
+            0.010988244786858559,
+            -0.010359029285609722,
+            -0.01332094520330429,
+            0.008801338262856007,
+            0.01470982190221548,
+            0.008824358694255352,
+            -0.0241403765976429,
+            0.02908201701939106,
+            -0.014571702107787132,
+            0.04162028059363365,
+            -0.0001700127759249881,
+            -0.003117300570011139,
+            0.023204226046800613,
+            -0.0009577306336723268,
+            -0.02265174500644207,
+            0.014947696588933468,
+            0.00268183765001595,
+            0.01203182153403759,
+            -0.01066596433520317,
+            0.010527843609452248,
+            0.023787401616573334,
+            0.006537698674947023,
+            -0.005018374416977167,
+            -7.163796544773504e-05,
+            0.016912074759602547,
+            0.0005409715231508017,
+            -0.005033721216022968,
+            0.027731506153941154,
+            -0.007481521461158991,
+            -0.010842450894415379,
+            -0.01875368133187294,
+            -0.0012171859852969646,
+            -0.018247239291667938,
+            -0.00043474353151395917,
+            -0.007700212299823761,
+            0.0008320794440805912,
+            -0.010589230805635452,
+            -0.015154876746237278,
+            -0.003944104537367821,
+            -0.025076525285840034,
+            0.03480634093284607,
+            0.009676100686192513,
+            0.013597185723483562,
+            0.009200353175401688,
+            0.00365059869363904,
+            -0.000640245561953634,
+            0.0214547012001276,
+            -0.0030175470747053623,
+            0.0005649508093483746,
+            -0.005685956217348576,
+            -0.0119857806712389,
+            -0.0122390016913414,
+            0.013980853371322155,
+            -0.008317917585372925,
+            -0.008640198037028313,
+            -0.00844069104641676,
+            -0.01821654476225376,
+            -0.03477564454078674,
+            -0.0013610613532364368,
+            0.008893419057130814,
+            -0.028360720723867416,
+            0.014909329824149609,
+            -0.006441781762987375,
+            -0.01600661873817444,
+            0.0008009064476937056,
+            0.006161704193800688,
+            0.018554173409938812,
+            -0.007972615770995617,
+            -0.03425385802984238,
+            0.031384024769067764,
+            0.01646701991558075,
+            0.003899982897564769,
+            0.0033820313401520252,
+            0.008010982535779476,
+            0.008064696565270424,
+            -0.01395783293992281,
+            -0.0099139753729105,
+            0.028115173801779747,
+            -0.026212181895971298,
+            0.0239715613424778,
+            0.012561283074319363,
+            0.0003963767667300999,
+            0.027240412309765816,
+            -0.019474975764751434,
+            0.02630426175892353,
+            -0.0076042949222028255,
+            -5.1765175157925114e-05,
+            0.015200916677713394,
+            0.02099430002272129,
+            0.020319044589996338,
+            0.02272847853600979,
+            -0.02265174500644207,
+            0.038643017411231995,
+            -0.0023787401150912046,
+            0.03201323747634888,
+            0.006307498086243868,
+            -0.00015094928676262498,
+            0.008486730977892876,
+            0.008172123692929745,
+            -0.017679410055279732,
+            -0.0010176787618547678,
+            -0.03069342114031315,
+            0.010604577139019966,
+            -0.005229391623288393,
+            -0.01867694780230522,
+            -0.020794792100787163,
+            -0.033916231244802475,
+            0.010888490825891495,
+            0.001221981830894947,
+            -0.013037030585110188,
+            -0.004742133431136608,
+            0.0078076389618217945,
+            0.01306772418320179,
+            0.022713132202625275,
+            -0.0037100673653185368,
+            0.0032688493374735117,
+            0.019643789157271385,
+            -0.005298451986163855,
+            0.004331609234213829,
+            -0.01617543213069439,
+            0.02211461029946804,
+            0.005555509123951197,
+            0.030754808336496353,
+            0.02476959116756916,
+            0.008540444076061249,
+            0.004611686337739229,
+            -0.004204998724162579,
+            0.017817530781030655,
+            -0.01600661873817444,
+            -0.010481803677976131,
+            -0.001481916755437851,
+            -0.0038366776425391436,
+            0.06037395820021629,
+            -0.024692857638001442,
+            -0.02506117895245552,
+            -0.005559345707297325,
+            -0.011487012729048729,
+            0.006675818935036659,
+            -0.03247363865375519,
+            -0.028928549960255623,
+            -0.023511160165071487,
+            -0.0175873301923275,
+            -0.0019624605774879456,
+            -0.009998382069170475,
+            0.014049913734197617,
+            -0.00569362984970212,
+            0.011778600513935089,
+            -0.0239715613424778,
+            0.000355372263584286,
+            -0.016651181504130363,
+            -0.005685956217348576,
+            -0.014026893302798271,
+            -0.0070403036661446095,
+            -0.000628735579084605,
+            0.02949637733399868,
+            0.011824640445411205,
+            -0.019229428842663765,
+            -0.0032918695360422134,
+            0.012745442800223827,
+            -0.008279550820589066,
+            -0.007362584583461285,
+            0.012246674858033657,
+            0.01916804164648056,
+            -0.012208308093249798,
+            0.037937067449092865,
+            -0.0037983108777552843,
+            -0.00038582590059377253,
+            -0.005904647056013346,
+            -0.012584302574396133,
+            0.024877017363905907,
+            0.028314681723713875,
+            -0.03984006121754646,
+            -0.0013303679879754782,
+            0.0018694212194532156,
+            0.007600458338856697,
+            -0.005321471951901913,
+            0.0030866072047501802,
+            -0.027117637917399406,
+            -0.0012776136863976717,
+            -0.026872090995311737,
+            -0.01573037914931774,
+            -0.021470047533512115,
+            -0.005923830438405275,
+            0.0032458293717354536,
+            -0.023050758987665176,
+            0.030094899237155914,
+            0.02077944576740265,
+            -0.014487294480204582,
+            -0.0038596978411078453,
+            -0.003500968450680375,
+            0.008126083761453629,
+            -0.03189046308398247,
+            -0.004362302366644144,
+            -0.009745161049067974,
+            -9.597688767826185e-05,
+            -0.00962238758802414,
+            -0.04198860004544258,
+            0.0012239001225680113,
+            -0.003167177317664027,
+            -0.0017955651273950934,
+            -0.008494404144585133,
+            -0.016727915033698082,
+            0.02177698165178299,
+            0.020303698256611824,
+            -0.010880817659199238,
+            -0.008532770909368992,
+            0.045610424131155014,
+            0.008425343781709671,
+            0.0016411389224231243,
+            -0.010888490825891495,
+            0.002762407995760441,
+            0.0028314681258052588,
+            0.008486730977892876,
+            -0.013528125360608101,
+            0.011463993228971958,
+            -0.005298451986163855,
+            0.014118974097073078,
+            -0.0067065125331282616,
+            -0.016114046797156334,
+            0.023204226046800613,
+            -0.02829933539032936,
+            0.000396856339648366,
+            -0.010174869559705257,
+            -0.001934644766151905,
+            0.01395783293992281,
+            0.010259276255965233,
+            0.01726504974067211,
+            -0.03953312709927559,
+            -0.004815030377358198,
+            -0.008386977016925812,
+            -0.0004716715484391898,
+            -0.0059468504041433334,
+            -0.026994863525032997,
+            0.013443718664348125,
+            0.0023557201493531466,
+            -0.010880817659199238,
+            0.018323972821235657,
+            -0.004515769425779581,
+            -0.009960015304386616,
+            0.022375503554940224,
+            -0.0019020329928025603,
+            0.015362057834863663,
+            0.029235484078526497,
+            -0.016789302229881287,
+            0.0183700118213892,
+            0.032504335045814514,
+            -0.009369166567921638,
+            -0.0037139039486646652,
+            -0.0028142030350863934,
+            0.0009936995338648558,
+            0.016328901052474976,
+            0.017525942996144295,
+            0.0007447950774803758,
+            -0.03228947892785072,
+            0.00829489715397358,
+            -0.00218306970782578,
+            0.007918902672827244,
+            0.009514960460364819,
+            -0.014318481087684631,
+            -0.000949098146520555,
+            -0.025997327640652657,
+            0.006952059920877218,
+            -0.0209482591599226,
+            -0.009422880597412586,
+            0.024171069264411926,
+            -0.017786838114261627,
+            -0.019152695313096046,
+            0.003688965691253543,
+            0.02837606891989708,
+            -0.029680538922548294,
+            0.01130285207182169,
+            0.004139775410294533,
+            -0.005812566727399826,
+            0.006361211650073528,
+            0.000759182614274323,
+            -0.01895318739116192,
+            0.004500423092395067,
+            -0.009507287293672562,
+            -0.012169941328465939,
+            -0.020257657393813133,
+            -0.013835059478878975,
+            -0.0006699798395857215,
+            0.014617742039263248,
+            0.015822459012269974,
+            0.028851816430687904,
+            0.0016516897594556212,
+            -0.0069405497051775455,
+            -0.014656108804047108,
+            0.006879162974655628,
+            -0.0064801485277712345,
+            0.012990990653634071,
+            -0.008709258399903774,
+            0.02535276673734188,
+            -0.00811073649674654,
+            -0.0006037971470504999,
+            -0.010497150011360645,
+            -0.013198170810937881,
+            0.008762971498072147,
+            -0.003247747663408518,
+            0.0017994018271565437,
+            -0.011732560582458973,
+            0.004354629199951887,
+            0.01692742109298706,
+            -0.01863090693950653,
+            -0.02177698165178299,
+            -0.023449774831533432,
+            0.021976489573717117,
+            0.03766082972288132,
+            0.0009236801415681839,
+            -0.050214435905218124,
+            -0.0021063359454274178,
+            -0.007677191868424416,
+            -0.012077861465513706,
+            -0.02265174500644207,
+            0.011487012729048729,
+            0.018139811232686043,
+            -0.0008004268747754395,
+            -0.0019893173594027758,
+            0.0031978709157556295,
+            0.015108836814761162,
+            -0.0013207762967795134,
+            0.0053866952657699585,
+            0.00251494231633842,
+            -0.0075007048435509205,
+            0.004155121743679047,
+            0.029941432178020477,
+            -0.027731506153941154,
+            0.00941520743072033,
+            0.002618532394990325,
+            0.011755580082535744,
+            0.004669236950576305,
+            0.009177332744002342,
+            0.01470982190221548,
+            0.0032400742638856173,
+            -0.00709401722997427,
+            0.0196591354906559,
+            0.07206815481185913,
+            -0.0028314681258052588,
+            0.015385077334940434,
+            0.00972981471568346,
+            -0.012822176329791546,
+            -0.004040021449327469,
+            -0.010312989354133606,
+            -0.03848955035209656,
+            0.025398805737495422,
+            0.008034002967178822,
+            -0.012077861465513706,
+            0.006483985111117363,
+            -0.014126647263765335,
+            0.008878071792423725,
+            0.029020629823207855,
+            0.010366703383624554,
+            -0.020794792100787163,
+            0.004281732253730297,
+            0.03453009948134422,
+            0.0214547012001276,
+            0.004542626440525055,
+            -0.01780218444764614,
+            0.006219254340976477,
+            -0.0014195707626640797,
+            -0.0038712078239768744,
+            0.009614714421331882,
+            -0.004465892910957336,
+            0.022375503554940224,
+            -0.004078388214111328,
+            0.014111299999058247,
+            -0.013535798527300358,
+            -0.013911793008446693,
+            0.006115664262324572,
+            -0.001624832977540791,
+            0.0014790392015129328,
+            0.020150231197476387,
+            0.0028314681258052588,
+            -0.006760226096957922,
+            0.0024861672427505255,
+            -0.0005850933375768363,
+            0.013681592419743538,
+            0.004818866960704327,
+            0.007423971313983202,
+            0.006614432204514742,
+            -0.023050758987665176,
+            0.01879972033202648,
+            0.006434108596295118,
+            0.00976818148046732,
+            -0.013037030585110188,
+            -0.005682119634002447,
+            -0.005720486398786306,
+            -0.004757480230182409,
+            0.0018454419914633036,
+            -0.00196437886916101,
+            -0.01573037914931774,
+            -0.004435199312865734,
+            0.009960015304386616,
+            -0.024662164971232414,
+            0.0048764171078801155,
+            0.010842450894415379,
+            0.004991517867892981,
+            0.0009601286146789789,
+            -0.01124146580696106,
+            -0.03256572037935257,
+            5.1435461500659585e-05,
+            -0.005647589452564716,
+            -0.0021255193278193474,
+            0.004853397142142057,
+            0.010919184423983097,
+            0.0222373828291893,
+            0.004396832548081875,
+            -0.0013831222895532846,
+            0.021715596318244934,
+            0.0007064282544888556,
+            -0.0033014612272381783,
+            0.016866035759449005,
+            -0.00735491095110774,
+            -0.013129111379384995,
+            0.00012517160212155432,
+            0.003560436889529228,
+            0.014755862765014172,
+            0.0057358331978321075,
+            0.01617543213069439,
+            0.010351356118917465,
+            -0.02555227279663086,
+            -0.020840832963585854,
+            9.196036990033463e-05,
+            0.026319608092308044,
+            -0.012653362937271595,
+            -0.01912200078368187,
+            -0.00412826519459486,
+            -0.005187188275158405,
+            0.006925203371793032,
+            -0.012231328524649143,
+            0.008977825753390789,
+            0.016451673582196236,
+            -0.011364239268004894,
+            -0.026273569092154503,
+            -0.020472511649131775,
+            -0.005847096908837557,
+            0.009775854647159576,
+            0.009814221411943436,
+            -0.006741042714565992,
+            0.012983317486941814,
+            0.002706776140257716,
+            -0.021761635318398476,
+            0.022835904732346535,
+            0.013489758595824242,
+            0.005689792800694704,
+            -0.005229391623288393,
+            -0.004523443058133125,
+            -0.011479339562356472,
+            0.0068753263913095,
+            0.004070715047419071,
+            -0.007293524220585823,
+            -0.030923621729016304,
+            -0.0024880855344235897,
+            -0.006537698674947023,
+            -0.00125747115816921,
+            -0.009514960460364819,
+            0.00602742051705718,
+            -0.029972126707434654,
+            0.004020838066935539,
+            -0.0060926442965865135,
+            -0.0041819787584245205,
+            -0.006449454929679632,
+            -0.0009423839510418475,
+            -0.009238719940185547,
+            -0.028237948194146156,
+            -0.027271104976534843,
+            0.0011807375121861696,
+            0.01199345476925373,
+            -0.0016392205143347383,
+            -0.002380658406764269,
+            -0.024247802793979645,
+            -0.0004181978583801538,
+            0.008640198037028313,
+            -0.00744699127972126,
+            0.01846209354698658,
+            -0.003890391206368804,
+            -0.006952059920877218,
+            -0.0004949314170517027,
+            -0.0012392469216138124,
+            -0.026749316602945328,
+            0.017080888152122498,
+            0.0031000354792922735,
+            0.014326154254376888,
+            -0.02435522899031639,
+            0.011279832571744919,
+            -0.013366985134780407,
+            0.007780782412737608,
+            -0.01841605268418789,
+            -0.008762971498072147,
+            -0.00786518957465887,
+            -0.012392468750476837,
+            -0.007826822809875011,
+            -0.029143404215574265,
+            0.00844069104641676,
+            -0.008801338262856007,
+            -0.025813167914748192,
+            -0.02908201701939106,
+            0.009353820234537125,
+            -0.015776418149471283,
+            0.015561564825475216,
+            -0.009215699508786201,
+            0.04527279734611511,
+            0.020027456805109978,
+            0.015354383736848831,
+            0.008824358694255352,
+            -0.017863571643829346,
+            -0.0032937878277152777,
+            0.012653362937271595,
+            -0.026273569092154503,
+            0.0003601681091822684,
+            0.012553608976304531,
+            0.01487096305936575,
+            -0.011318199336528778,
+            0.001353388070128858,
+            0.01651306077837944,
+            0.015177897177636623,
+            0.01622147299349308,
+            0.0024036786053329706,
+            -0.009131292812526226,
+            0.010305316187441349,
+            0.023633934557437897,
+            0.02567504718899727,
+            -0.004803520627319813,
+            0.0019662973936647177,
+            0.0011625132756307721,
+            -0.018477439880371094,
+            -0.009077579714357853,
+            -0.013205844908952713,
+            -0.009783527813851833,
+            -0.02659584954380989,
+            0.011525379493832588,
+            -0.0038002291694283485,
+            -0.010013728402554989,
+            -0.01580711267888546,
+            -0.014978389255702496,
+            -0.005133474711328745,
+            0.006599085405468941,
+            0.021546781063079834,
+            -0.010712004266679287,
+            0.010919184423983097,
+            -0.010052095167338848,
+            0.0020545409061014652,
+            -0.00196437886916101,
+            0.0081491032615304,
+            0.027225064113736153,
+            0.010274622589349747,
+            -0.055524397641420364,
+            -0.006499331910163164,
+            0.002762407995760441,
+            -0.010650617070496082,
+            -0.051503561437129974,
+            0.002656899392604828,
+            0.007324217818677425,
+            -0.03185977041721344,
+            0.008655544370412827,
+            -0.0023154348600655794,
+            -0.0079265758395195,
+            0.006595248822122812,
+            0.019935376942157745,
+            -0.016206126660108566,
+            0.013259558007121086,
+            -6.013542588334531e-06,
+            0.004320099018514156,
+            -0.014924676157534122,
+            0.0062230913899838924,
+            -0.015147203579545021,
+            -0.014471948146820068,
+            0.01734178327023983,
+            -0.001127024064771831,
+            -0.002089071087539196,
+            0.012008801102638245,
+            0.02506117895245552,
+            0.02579781971871853,
+            -0.012162268161773682,
+            -0.01592988520860672,
+            -0.01879972033202648,
+            0.0007073874585330486,
+            -0.0055401623249053955,
+            -0.010712004266679287,
+            -0.0019624605774879456,
+            0.008417670615017414,
+            0.01209320779889822,
+            0.01987398974597454,
+            0.008417670615017414,
+            0.006315171252936125,
+            -3.35109798470512e-05,
+            0.009898628108203411,
+            0.0023422916419804096,
+            -0.020303698256611824,
+            -0.0065530454739928246,
+            0.012983317486941814,
+            0.019797256216406822,
+            0.0017936468357220292,
+            0.004297079052776098,
+            0.018185852095484734,
+            -0.00773857906460762,
+            -0.009208026342093945,
+            -0.0336092971265316,
+            -0.0006901223678141832,
+            -0.017203662544488907,
+            0.016236819326877594,
+            0.01945962943136692,
+            -0.02136262133717537,
+            -0.01022858265787363,
+            -0.01866159960627556,
+            0.02394086867570877,
+            -0.010735023766756058,
+            0.005912320222705603,
+            0.0022252730559557676,
+            0.008739951997995377,
+            0.0019068288384005427,
+            -0.008517424575984478,
+            -0.014272441156208515,
+            0.009277086704969406,
+            0.00048318158951587975,
+            0.007661845069378614,
+            -0.0061195008456707,
+            -0.012039494700729847,
+            0.001056045526638627,
+            -0.009860261343419552,
+            -0.028283987194299698,
+            0.011579093523323536,
+            0.0008670891402289271,
+            0.01821654476225376,
+            -0.019597750157117844,
+            0.01675860770046711,
+            0.0007380808820016682,
+            -0.013520452193915844,
+            -0.02788497321307659,
+            -0.015277650207281113,
+            0.003919166047126055,
+            0.003792555769905448,
+            0.012661036103963852,
+            0.0034817850682884455,
+            -0.01658979430794716,
+            0.008739951997995377,
+            -0.005647589452564716,
+            -0.005751179996877909,
+            -0.001697729923762381,
+            -0.01170186698436737,
+            -0.001447386690415442,
+            0.011072651483118534,
+            -0.01037437655031681,
+            -0.005574692506343126,
+            0.01468680240213871,
+            0.006411088164895773,
+            -0.002988772001117468,
+            0.02742457203567028,
+            0.01821654476225376,
+            0.02153143472969532,
+            0.0006488781073130667,
+            0.01066596433520317,
+            -0.0018473602831363678,
+            0.003558518597856164,
+            -0.03505188599228859,
+            0.025122566148638725,
+            -0.01576107181608677,
+            0.023373039439320564,
+            0.008532770909368992,
+            -0.00694822333753109,
+            -0.005881627090275288,
+            0.01331327110528946,
+            0.006088807247579098,
+            -0.004803520627319813,
+            0.00602742051705718,
+            -0.003128810552880168,
+            0.009844915010035038,
+            -0.0013102254597470164,
+            -0.02493840456008911,
+            0.020012110471725464,
+            -0.024416616186499596,
+            0.009990708902478218,
+            -0.002990690292790532,
+            0.009123619645833969,
+            -0.006894509773701429,
+            -5.931863415753469e-05,
+            -0.00457331957295537,
+            -0.040638089179992676,
+            -0.001970133977010846,
+            -0.010773390531539917,
+            0.004450546111911535,
+            0.016819994896650314,
+            -0.00021497384295798838,
+            -0.04220345616340637,
+            -0.005916156806051731,
+            0.01958240196108818,
+            -0.00021737176575697958,
+            0.009883281774818897,
+            -0.008271876722574234,
+            -0.004086061846464872,
+            -0.004320099018514156,
+            -0.007792292162775993,
+            0.016543753445148468,
+            0.013136784546077251,
+            -0.01563829742372036,
+            -0.023618588224053383,
+            0.012507569044828415,
+            -0.01879972033202648,
+            -0.019567055627703667,
+            0.00530996173620224,
+            -0.01267638336867094,
+            0.001553854439407587,
+            0.022268077358603477,
+            -0.004715276882052422,
+            0.011164732277393341,
+            -0.0056130592711269855,
+            0.013282578438520432,
+            -0.01543111726641655,
+            -0.0019471138948574662,
+            -0.0047651538625359535,
+            -0.02348046749830246,
+            0.025567619130015373,
+            -0.018937841057777405,
+            0.0019931539427489042,
+            -0.016528407111763954,
+            0.02228342369198799,
+            -0.020088844001293182,
+            -0.002275149803608656,
+            0.008010982535779476,
+            0.010174869559705257,
+            0.029803311452269554,
+            0.01345906499773264,
+            -0.007408624514937401,
+            0.009169659577310085,
+            0.008824358694255352,
+            0.019827950745821,
+            0.00156632368452847,
+            0.029803311452269554,
+            -0.005271594971418381,
+            0.01041274331510067,
+            -0.001517405966296792,
+            0.018968533724546432,
+            0.0067065125331282616,
+            0.029926085844635963,
+            0.0003174850717186928,
+            0.00015430638450197875,
+            -0.0001380004978273064,
+            0.018600212410092354,
+            -0.004369975998997688,
+            -0.005313798785209656,
+            0.007765435613691807,
+            -0.0059698703698813915,
+            -0.0015797520754858851,
+            -0.020441818982362747,
+            -0.005877790041267872,
+            0.001410938217304647,
+            0.013789019547402859,
+            -0.0005241860635578632,
+            0.005390532314777374,
+            0.0008200898300856352,
+            0.005586202722042799,
+            -0.014617742039263248,
+            0.011402606032788754,
+            -0.015668991953134537,
+            0.01883041486144066,
+            -0.01005976926535368,
+            0.01008278876543045,
+            -0.0009692406747490168,
+            0.00811073649674654,
+            0.011072651483118534,
+            -0.013574165292084217,
+            0.01614473946392536,
+            0.012139247730374336,
+            -0.010589230805635452,
+            0.016313552856445312,
+            -0.007228300906717777,
+            0.005083597730845213,
+            0.0038616161327809095,
+            0.0005870116874575615,
+            0.003153749043121934,
+            -0.018109118565917015,
+            -0.009461247362196445,
+            -0.0001405183138558641,
+            -0.012576629407703876,
+            -0.0030501587316393852,
+            0.012331082485616207,
+            -0.005505632609128952,
+            0.015507850795984268,
+            0.010197889059782028,
+            0.012285041622817516,
+            0.004933967255055904,
+            0.011172405444085598,
+            -0.0014186116168275476,
+            -0.0034299897961318493,
+            0.004684583283960819,
+            0.032964736223220825,
+            0.0008824358810670674,
+            0.025905247777700424,
+            -0.014978389255702496,
+            0.006084970664232969,
+            0.001282409531995654,
+            -0.0016181188402697444,
+            -0.008701585233211517,
+            0.016236819326877594,
+            0.020088844001293182,
+            0.0018934004474431276,
+            -0.011893700808286667,
+            -0.003499049926176667,
+            0.029404297471046448,
+            -0.005501795560121536,
+            0.007454664912074804,
+            0.010696657001972198,
+            -0.006196234375238419,
+            0.016605140641331673,
+            0.004381485749036074,
+            -0.024692857638001442,
+            0.0013457146706059575,
+            0.004047695081681013,
+            -0.019152695313096046,
+            0.020917566493153572,
+            0.02104034088551998,
+            0.0076042949222028255,
+            0.002739387797191739,
+            0.01345906499773264,
+            0.010497150011360645,
+            0.021623514592647552,
+            0.0032400742638856173,
+            0.004580993205308914,
+            0.002643470885232091,
+            -0.012952623888850212,
+            -0.00123445107601583,
+            -0.020963607355952263,
+            -0.011533052660524845,
+            -0.0020583774894475937,
+            0.018983881920576096,
+            0.0026530625764280558,
+            -0.013290251605212688,
+            -0.003959451336413622,
+            0.008855052292346954,
+            0.004381485749036074,
+            -0.006299824919551611,
+            -0.022053223103284836,
+            -0.008202817291021347,
+            0.024094335734844208,
+            0.008095390163362026,
+            0.010550864040851593,
+            -0.0031729324255138636,
+            -0.002160049509257078,
+            -0.0006109909154474735,
+            -0.006614432204514742,
+            -0.0006450414075516164,
+            -0.01597592607140541,
+            -0.0021063359454274178,
+            -0.016666527837514877,
+            0.015346710570156574,
+            0.015791764482855797,
+            0.01763337105512619,
+            0.018139811232686043,
+            -0.02102499268949032,
+            0.017602676525712013,
+            -0.0071438937447965145,
+            0.001838727737776935,
+            -0.0034414997790008783,
+            0.00010826624929904938,
+            0.019275467842817307,
+            0.029388951137661934,
+            0.0056015495210886,
+            0.011149385012686253,
+            -0.0013831222895532846,
+            0.005927667021751404,
+            -0.006457128562033176,
+            -0.013980853371322155,
+            -0.0033935413230210543,
+            -0.01892249472439289,
+            0.004907110705971718,
+            0.017848225310444832,
+            -0.015139530412852764,
+            -0.004922457505017519,
+            0.014349174685776234,
+            0.009867934510111809,
+            0.008340937085449696,
+            -0.008931785821914673,
+            -0.004112918395549059,
+            0.0008714054129086435,
+            0.01557691115885973,
+            -0.007585111539810896,
+            -0.016605140641331673,
+            0.008394651114940643,
+            -0.006146357394754887,
+            0.02123984694480896,
+            -0.004542626440525055,
+            -0.01605265960097313,
+            -0.010282295756042004,
+            0.003186360700055957,
+            -0.021439354866743088,
+            -0.014272441156208515,
+            0.01002140250056982,
+            0.0021312744356691837,
+            0.006936713121831417,
+            -0.017326436936855316,
+            0.013060051016509533,
+            0.004834213759750128,
+            0.03351721540093422,
+            -0.008133756928145885,
+            -0.0005424103001132607,
+            0.0003779127437155694,
+            0.005916156806051731,
+            0.0002481850970070809,
+            -0.01755663752555847,
+            0.026933476328849792,
+            0.006587575655430555,
+            0.013781346380710602,
+            -0.0003863054735120386,
+            -0.012645689770579338,
+            0.006994263269007206,
+            -0.01564597152173519,
+            0.001863666228018701,
+            -0.010205562226474285,
+            -0.020088844001293182,
+            -0.01058155670762062,
+            0.02165420912206173,
+            0.0070403036661446095,
+            -0.00732038076967001,
+            -0.0001907548139570281,
+            -0.0077769458293914795,
+            -0.016359593719244003,
+            -0.009852588176727295,
+            -0.015492504462599754,
+            0.008386977016925812,
+            -0.01663583517074585,
+            0.0029005282558500767,
+            0.008126083761453629,
+            0.009806548245251179,
+            0.005586202722042799,
+            0.021623514592647552,
+            0.002338455058634281,
+            0.016405634582042694,
+            0.0001647373428568244,
+            -0.0019049104303121567,
+            0.005651426035910845,
+            0.014771209098398685,
+            0.0072973608039319515,
+            -0.008555791340768337,
+            -0.0046423799358308315,
+            -0.012691729702055454,
+            0.0005198698490858078,
+            -0.009676100686192513,
+            0.01066596433520317,
+            0.0064302715472877026,
+            -0.0009404656011611223,
+            -0.00678324606269598,
+            0.021715596318244934,
+            -0.0038980646058917046,
+            0.007711722049862146,
+            -0.007957269437611103,
+            -0.004300915636122227,
+            0.01065829023718834,
+            0.0010848206002265215,
+            -0.013765999116003513,
+            0.008210490457713604,
+            0.00919268000870943,
+            -0.002263639820739627,
+            -0.003238155972212553,
+            -0.020672019571065903,
+            0.02874438837170601,
+            0.008885745890438557,
+            -0.010182542726397514,
+            0.014425908215343952,
+            -0.0052831051871180534,
+            0.019306162372231483,
+            0.016282860189676285,
+            0.0185388270765543,
+            -0.018323972821235657,
+            -0.019643789157271385,
+            0.007251320872455835,
+            -0.015561564825475216,
+            -0.0027835096698254347,
+            0.0029715069103986025,
+            -0.0022809049114584923,
+            -0.0002340373321203515,
+            -0.016068005934357643,
+            0.0009150475962087512,
+            -0.0046500531025230885,
+            0.0024938404094427824,
+            0.000545287795830518,
+            0.0077347420156002045,
+            0.0009395064553245902,
+            0.012760790064930916,
+            -0.006196234375238419,
+            -0.01783287711441517,
+            -0.006606759037822485,
+            0.013512779027223587,
+            0.006906019989401102,
+            -0.03600338473916054,
+            0.004980007652193308,
+            -0.012499895878136158,
+            -0.01751059666275978,
+            -0.002232946455478668,
+            -0.02157747559249401,
+            -0.02116311341524124,
+            -0.001316939597018063,
+            0.0009831486968323588,
+            -0.011510033160448074,
+            0.0021869062911719084,
+            0.006648962385952473,
+            -0.009300106205046177,
+            -0.00919268000870943,
+            0.0054595922119915485,
+            -0.0018751762108877301,
+            0.012645689770579338,
+            -0.012392468750476837,
+            -0.010443436913192272,
+            0.035174660384655,
+            0.013635552488267422,
+            -0.00766568211838603,
+            -0.013812039978802204,
+            -0.010873144492506981,
+            -0.014280114322900772,
+            0.010980571620166302,
+            -0.0030827706214040518,
+            -0.00466539990156889,
+            -0.037814296782016754,
+            0.0013543472159653902,
+            0.0011884109117090702,
+            0.01524695660918951,
+            -0.008862725459039211,
+            0.014533335343003273,
+            0.005793383345007896,
+            -0.00058077706489712,
+            -0.012354101985692978,
+            -6.549328645633068e-06,
+            0.01080408412963152,
+            0.010136502794921398,
+            -0.007197607308626175,
+            0.012100880965590477,
+            -0.013006336987018585,
+            0.007527561392635107,
+            -0.014633088372647762,
+            -0.0077769458293914795,
+            -0.01933685503900051,
+            -0.01787891797721386,
+            -0.007516051642596722,
+            0.01634424738585949,
+            0.009890954941511154,
+            -0.01787891797721386,
+            -0.004815030377358198,
+            -0.014525661244988441,
+            0.013451391831040382,
+            -0.011533052660524845,
+            -0.006015910767018795,
+            -0.01306772418320179,
+            0.001044535543769598,
+            -0.018784373998641968,
+            0.01385807991027832,
+            0.01775614358484745,
+            0.01639028638601303,
+            0.0004093255556654185,
+            -0.0053866952657699585,
+            -0.02248293161392212,
+            -0.01870764046907425,
+            -0.004462056327611208,
+            0.002572492463514209,
+            -0.013781346380710602,
+            -0.003744597313925624,
+            0.011732560582458973,
+            0.021915102377533913,
+            -0.0017984426813200116,
+            -0.013129111379384995,
+            0.00854811817407608,
+            -0.016958115622401237,
+            -0.017311088740825653,
+            -0.018277931958436966,
+            0.007358747534453869,
+            0.008172123692929745,
+            0.01605265960097313,
+            0.0024746570270508528,
+            0.006058114115148783,
+            -0.004623196553438902,
+            -0.007730905432254076,
+            -0.024278495460748672,
+            0.005110454745590687,
+            -0.0112030990421772,
+            0.010075115598738194,
+            -0.0021408661268651485,
+            -0.004070715047419071,
+            0.023465121164917946,
+            0.009760508313775063,
+            -0.004995354451239109,
+            -0.00018631866259966046,
+            0.020840832963585854,
+            -0.008686237968504429,
+            -0.00602742051705718,
+            -0.008057023398578167,
+            0.017771491780877113,
+            -0.021961143240332603,
+            0.00016090067219920456,
+            -0.011187751777470112,
+            -0.0222373828291893,
+            0.003272686153650284,
+            0.012200634926557541,
+            0.00606578728184104,
+            -0.003355174558237195,
+            0.003453009994700551,
+            -0.008433017879724503,
+            0.024094335734844208,
+            -0.01435684785246849,
+            0.002348046749830246,
+            0.013620206154882908,
+            -0.0021638860926032066,
+            -0.002465065335854888,
+            0.013727632351219654,
+            -0.005417388863861561,
+            0.009530307725071907,
+            0.0035738651640713215,
+            0.00011857732170028612,
+            -0.009660754352807999,
+            0.016988808289170265,
+            0.02535276673734188,
+            -0.00860950443893671,
+            0.0008152939844876528,
+            -0.003782964078709483,
+            0.00602742051705718,
+            -0.0070403036661446095,
+            0.03201323747634888,
+            -0.008287223987281322,
+            0.0005500836414285004,
+            0.0010100053623318672,
+            0.03161422535777092,
+            -0.0014771209098398685,
+            0.010044422000646591,
+            0.0014541008276864886,
+            -0.0034184798132628202,
+            0.022927986457943916,
+            0.0025667373556643724,
+            0.004531116224825382,
+            0.027102291584014893,
+            -0.013574165292084217,
+            -0.00493780430406332,
+            0.005543999373912811,
+            -0.015200916677713394,
+            0.012392468750476837,
+            -0.02048785798251629,
+            0.0009188842959702015,
+            -0.01522393710911274,
+            0.014280114322900772,
+            0.019981417804956436,
+            0.030386487022042274,
+            -0.014418235048651695,
+            -0.00619239779189229,
+            -0.00728201400488615,
+            0.03143006190657616,
+            0.00044217711547389627,
+            0.006288314703851938,
+            -0.00030045982566662133,
+            0.011471666395664215,
+            -0.015177897177636623,
+            -0.01962844282388687,
+            0.012499895878136158,
+            -0.019551709294319153,
+            -0.03413108363747597,
+            -0.004634706769138575,
+            0.005670609418302774,
+            -0.0030175470747053623,
+            0.0037100673653185368,
+            0.012146921828389168,
+            0.004358465783298016,
+            0.022145302966237068,
+            0.007243647240102291,
+            0.009752834215760231,
+            -0.016482368111610413,
+            3.229703224860714e-06,
+            -0.006848469842225313,
+            -0.025398805737495422,
+            0.016037313267588615,
+            -0.021516088396310806,
+            0.0014138157712295651,
+            -0.008433017879724503,
+            -0.002647307701408863,
+            0.009967688471078873,
+            0.004780500195920467,
+            0.014778882265090942,
+            0.005789546761661768,
+            0.016282860189676285,
+            0.019551709294319153,
+            -0.01573037914931774,
+            0.011586766690015793,
+            0.027025558054447174,
+            0.00113853404764086,
+            -0.017188316211104393,
+            -0.007784618996083736,
+            -0.01975121721625328,
+            -0.01945962943136692,
+            0.011256812140345573,
+            0.02041112445294857,
+            -0.00846371054649353,
+            -0.001863666228018701,
+            0.03904203325510025,
+            -0.013113764114677906,
+            0.023956215009093285,
+            -0.00775776244699955,
+            0.005240901838988066,
+            0.03468356654047966,
+            0.010827104561030865,
+            0.018692294135689735,
+            -0.00613868422806263,
+            -0.034407325088977814,
+            -0.002121682744473219,
+            -0.00680242944508791,
+            0.0065607186406850815,
+            -0.011049631983041763,
+            0.018032385036349297,
+            -0.013673919253051281,
+            -0.019812602549791336,
+            0.02036508545279503,
+            -0.02243689075112343,
+            0.021377967670559883,
+            0.007562091574072838,
+            0.008578810840845108,
+            -0.006269131321460009,
+            -0.011479339562356472,
+            -3.974558421759866e-05,
+            -0.0077270688489079475,
+            0.011318199336528778,
+            0.008034002967178822,
+            -0.011832313612103462,
+            -0.006806266028434038,
+            0.0031729324255138636,
+            -0.004634706769138575,
+            -0.02311214618384838,
+            0.012860543094575405,
+            0.008640198037028313,
+            0.013428371399641037,
+            0.007381767965853214,
+            0.005075924564152956,
+            0.010635270737111568,
+            0.0034817850682884455,
+            -0.013113764114677906,
+            -0.01203182153403759,
+            0.00914663914591074,
+            -0.006127174012362957,
+            -0.0007332850364036858,
+            -0.026687929406762123,
+            0.014702148735523224,
+            -0.022743824869394302,
+            -0.0032573393546044827,
+            -0.0018195443553850055,
+            0.0039786347188055515,
+            0.005486449226737022,
+            -0.01866159960627556,
+            -0.005636079702526331,
+            -0.00028990895953029394,
+            -0.009392186999320984,
+            0.01188602764159441,
+            -0.013620206154882908,
+            0.01646701991558075,
+            0.027056250721216202,
+            -0.01487096305936575,
+            -0.0036199053283780813,
+            0.00554783595725894,
+            0.0077040488831698895,
+            -0.012522916309535503,
+            0.016988808289170265,
+            0.015108836814761162,
+            -0.01205484103411436,
+            0.008847379125654697,
+            0.03083154186606407,
+            -0.006579902023077011,
+            0.0005088393809273839,
+            -0.00012493181566242129,
+            0.0024995955172926188,
+            0.029726577922701836,
+            0.0016066087409853935,
+            0.024293843656778336,
+            0.005873953457921743,
+            -0.0008862725226208568,
+            -0.014610068872570992,
+            -0.014364521019160748,
+            -0.009522633627057076,
+            -0.009990708902478218,
+            -0.005520978942513466,
+            -0.010497150011360645,
+            0.009706794284284115,
+            0.012699402868747711,
+            -0.004676910117268562,
+            0.0038884729146957397,
+            -0.0019164205295965075,
+            0.02131658047437668,
+            -0.005052904598414898,
+            0.013113764114677906,
+            -0.022206690162420273,
+            0.01962844282388687,
+            0.020595286041498184,
+            0.011839987710118294,
+            -0.016313552856445312,
+            -0.005643752869218588,
+            0.016819994896650314,
+            0.025214646011590958,
+            -0.00538285868242383,
+            0.001968215685337782,
+            0.019444283097982407,
+            0.0031384022440761328,
+            0.005252411589026451,
+            -0.0026626542676240206,
+            -0.01987398974597454,
+            -0.027194371446967125,
+            -0.003568110289052129,
+            -0.006737206131219864,
+            0.002689511049538851,
+            0.004097571596503258,
+            -0.011839987710118294,
+            -0.011609786190092564,
+            0.0046500531025230885,
+            -0.005026047583669424,
+            0.0011020856909453869,
+            -0.0099139753729105,
+            -0.007849842309951782,
+            -0.038059841841459274,
+            0.020150231197476387,
+            0.0004304272879380733,
+            0.004396832548081875,
+            0.004239528905600309,
+            0.015108836814761162,
+            -0.007723232265561819,
+            -0.0016229146858677268,
+            0.015085816383361816,
+            0.006069623865187168,
+            -0.0018233810551464558,
+            -0.0003510560200084001,
+            -0.0035719468723982573,
+            -0.007197607308626175,
+            0.0104741295799613,
+            -0.004980007652193308,
+            -0.010274622589349747,
+            0.0084023242816329,
+            -0.012852869927883148,
+            -0.015062796883285046,
+            0.008225836791098118,
+            -0.025153258815407753,
+            0.00501070125028491,
+            -0.000793712679296732,
+            0.010159522294998169,
+            -0.004661563318222761,
+            -0.008809012360870838,
+            0.0010032912250608206,
+            0.013873426243662834,
+            0.007032630033791065,
+            -0.004726787097752094,
+            0.016083352267742157,
+            -0.007400951348245144,
+            0.007124710362404585,
+            0.029220137745141983,
+            -0.007949596270918846,
+            0.008601831272244453,
+            0.018477439880371094,
+            0.0013687347527593374,
+            0.008471384644508362,
+            0.01130285207182169,
+            -0.03643308952450752,
+            -0.016743261367082596,
+            -0.0008809971041046083,
+            0.0051603312604129314,
+            -0.0011174323735758662,
+            0.0188764538615942,
+            0.00919268000870943,
+            0.02725575864315033,
+            -0.006177050992846489,
+            0.012684056535363197,
+            0.0019509505946189165,
+            0.023572547361254692,
+            0.012108555063605309,
+            0.01199345476925373,
+            -0.005747342947870493,
+            -0.0021312744356691837,
+            -0.03588061034679413,
+            0.0035661919973790646,
+            0.022175997495651245,
+            0.019490322098135948,
+            -0.02579781971871853,
+            0.017203662544488907,
+            -0.0057435063645243645,
+            0.0033129712101072073,
+            -0.013428371399641037,
+            0.017694758251309395,
+            0.004392995964735746,
+            0.017571983858942986,
+            -0.008156776428222656,
+            -0.002417106879875064,
+            0.016866035759449005,
+            0.010466456413269043,
+            -0.012914257124066353,
+            0.021470047533512115,
+            0.012806829996407032,
+            0.02119380794465542,
+            0.019198734313249588,
+            -0.018308626487851143,
+            0.00908525288105011,
+            0.00875529833137989,
+            -0.012200634926557541,
+            -0.0081491032615304,
+            0.00457331957295537,
+            0.007232137490063906,
+            -0.016605140641331673,
+            -0.005018374416977167,
+            -0.0005678283050656319,
+            0.00023475672060158104,
+            0.012768463231623173,
+            -0.012668709270656109,
+            0.000462079857243225,
+            0.00412826519459486,
+            0.011080324649810791,
+            -0.0065607186406850815,
+            0.01734178327023983,
+            -0.028729042038321495,
+            0.012223655357956886,
+            -0.022022530436515808,
+            -0.0239715613424778,
+            0.0014195707626640797,
+            -0.004481239710003138,
+            -0.0036333338357508183,
+            -0.0051027811132371426,
+            0.0015154876746237278,
+            0.005647589452564716,
+            0.009983035735785961,
+            0.028391415253281593,
+            0.007715558633208275,
+            0.0013601022073999047,
+            -0.0004165193240623921,
+            -0.02302006632089615,
+            -0.010827104561030865,
+            -0.006054277531802654,
+            -0.0021389478351920843,
+            -0.006445618346333504,
+            -0.0336092971265316,
+            -0.00022852211259305477,
+            -0.0057358331978321075,
+            0.0029216299299150705,
+            0.0008435894851572812,
+            -0.02007349766790867,
+            -0.021500742062926292,
+            0.00926941353827715,
+            -0.008870398625731468,
+            -0.014026893302798271,
+            -0.0168046485632658,
+            0.006472475361078978,
+            0.00037431585951708257,
+            -0.0036563538014888763,
+            0.010151849128305912,
+            0.01846209354698658,
+            -0.004676910117268562,
+            -0.0006843673763796687,
+            0.010090461932122707,
+            0.0026396343018859625,
+            -0.02165420912206173,
+            0.008617177605628967,
+            -0.014111299999058247,
+            0.003951778169721365,
+            -0.01130285207182169,
+            -0.01709623634815216,
+            -0.004964660853147507,
+            -0.003967124503105879,
+            -0.028391415253281593,
+            0.0011673092376440763,
+            0.011832313612103462,
+            -0.007067160215228796,
+            0.013635552488267422,
+            0.020441818982362747,
+            0.010405070148408413,
+            0.00084119156235829,
+            -0.014863288961350918,
+            0.011425626464188099,
+            -0.01457937527447939,
+            -0.012998663820326328,
+            0.01970517635345459,
+            0.012929603457450867,
+            0.0112030990421772,
+            -0.0008503036806359887,
+            -0.014019220136106014,
+            0.011072651483118534,
+            0.015745725482702255,
+            0.025245338678359985,
+            -0.006272967904806137,
+            -0.0007697334513068199,
+            -0.0162982065230608,
+            -0.018446745350956917,
+            0.034161776304244995,
+            -0.011932067573070526,
+            -0.013681592419743538,
+            0.01655910164117813,
+            0.019613096490502357,
+            0.012668709270656109,
+            -0.005616896320134401,
+            -0.004684583283960819,
+            -0.0014684883644804358,
+            0.012346428819000721,
+            0.005820239894092083,
+            -0.025199299678206444,
+            -0.010681310668587685,
+            0.004400669131428003,
+            0.0325964130461216,
+            0.011747906915843487,
+            0.006069623865187168,
+            -0.043001484125852585,
+            0.014172687195241451,
+            0.0008853133767843246,
+            0.0009505368652753532,
+            0.013980853371322155,
+            0.029481031000614166,
+            0.020257657393813133,
+            0.0009706794517114758,
+            0.01962844282388687,
+            -0.015914538875222206,
+            0.02269778586924076,
+            -0.020978953689336777,
+            -0.010919184423983097,
+            -0.0241403765976429,
+            -0.005006864201277494,
+            -0.008479057811200619,
+            0.0028257130179554224,
+            0.009031538851559162,
+            0.009637733921408653,
+            -0.012262022122740746,
+            0.02021161839365959,
+            0.013221191242337227,
+            -0.010044422000646591,
+            -0.001552895293571055,
+            -0.02070271223783493,
+            0.02547553926706314,
+            0.003771454095840454,
+            -0.0008680483442731202,
+            -0.010612250305712223,
+            -0.005018374416977167,
+            -0.021132420748472214,
+            0.0018368094461038709,
+            -0.0028506515081971884,
+            0.012906583957374096,
+            -0.0016085271490737796,
+            0.008348610252141953,
+            0.016942769289016724,
+            -0.015208589844405651,
+            -0.003878881223499775,
+            0.007846006192266941,
+            0.02104034088551998,
+            0.006437945179641247,
+            0.01190904714167118,
+            0.02854488231241703,
+            -0.0013936731265857816,
+            -0.027485959231853485,
+            0.012008801102638245,
+            -0.006403414998203516,
+            -0.003366684541106224,
+            0.01111869141459465,
+            0.02131658047437668,
+            -0.021746288985013962,
+            0.0011557991383597255,
+            -0.007335727568715811,
+            0.020150231197476387,
+            3.1173007300822064e-05,
+            0.013198170810937881,
+            0.009138965979218483,
+            0.003936431370675564,
+            -0.01111869141459465,
+            -0.01421105396002531,
+            -0.008026329800486565,
+            0.01170186698436737,
+            -0.026979517191648483,
+            -0.0024036786053329706,
+            0.001851196982897818,
+            -0.003021383658051491,
+            0.006311334669589996,
+            0.006230764556676149,
+            -0.022099263966083527,
+            -0.010635270737111568,
+            -0.0028564066160470247,
+            -0.010182542726397514,
+            -0.005206371657550335,
+            0.012476875446736813,
+            -0.005409715697169304,
+            0.0005798179190605879,
+            0.00875529833137989,
+            0.009630060754716396,
+            0.004300915636122227,
+            0.025506233796477318,
+            -0.012584302574396133,
+            0.01924477517604828,
+            -0.030647380277514458,
+            0.001827217754907906,
+            -0.008578810840845108,
+            0.024171069264411926,
+            -0.0024497187696397305,
+            -0.019075961783528328,
+            0.012047167867422104,
+            -0.008195143193006516,
+            0.001614282140508294,
+            -0.002810366451740265,
+            -0.008433017879724503,
+            -0.0065300255082547665,
+            0.012545935809612274,
+            -0.018155159428715706,
+            0.004308588802814484,
+            0.025444846600294113,
+            0.0429094024002552,
+            -0.013643225654959679,
+            -0.013121437281370163,
+            0.0052255550399422646,
+            0.009983035735785961,
+            0.024094335734844208,
+            -0.01144864596426487,
+            -0.009307780303061008,
+            -0.01159443985670805,
+            0.0075812749564647675,
+            -0.016896728426218033,
+            -0.023833442479372025,
+            0.011563746258616447,
+            -0.015960579738020897,
+            -0.016850687563419342,
+            -0.018155159428715706,
+            -0.017955651506781578,
+            -0.0027662445791065693,
+            0.013773673214018345,
+            0.005904647056013346,
+            -0.0028046113438904285,
+            -0.0015615278389304876,
+            -0.006111827678978443,
+            -0.021270541474223137,
+            0.016896728426218033,
+            0.005133474711328745,
+            -0.01614473946392536,
+            -0.004373812582343817,
+            -0.012039494700729847,
+            0.002562900772318244,
+            0.0037522707134485245,
+            -0.003075097221881151,
+            0.05506399646401405,
+            0.003518233308568597,
+            0.009507287293672562,
+            -0.02107103355228901,
+            0.004266385454684496,
+            -0.010796410962939262,
+            -0.01074269786477089,
+            -0.007707885466516018,
+            -0.012116228230297565,
+            -0.003617987036705017,
+            -0.011479339562356472,
+            0.0019969907589256763,
+            -0.01543111726641655,
+            0.008133756928145885,
+            -0.013673919253051281,
+            -0.013474412262439728,
+            -0.017571983858942986,
+            -0.0015864662127569318,
+            -0.003451091470196843,
+            0.016114046797156334,
+            0.01325188484042883,
+            -0.016374940052628517,
+            -0.0030904437880963087,
+            0.022268077358603477,
+            0.007895882241427898,
+            0.018968533724546432,
+            -0.0035048050340265036,
+            -0.013129111379384995,
+            -0.020012110471725464,
+            0.0006565514486283064,
+            0.013075397349894047,
+            -0.0075812749564647675,
+            0.002915875054895878,
+            -0.029097363352775574,
+            -0.006748715881258249,
+            0.0014090199256315827,
+            0.01600661873817444,
+            0.01668187417089939,
+            0.008862725459039211,
+            -0.0018732579192146659,
+            -0.020856179296970367,
+            0.0332716703414917,
+            -0.007753925397992134,
+            -0.022743824869394302,
+            0.0019758890848606825,
+            0.007328054402023554,
+            -0.005144984927028418,
+            0.0024036786053329706,
+            0.013351637870073318,
+            0.004903274122625589,
+            -0.008340937085449696,
+            -0.004887927323579788,
+            0.0011836150661110878,
+            -0.021715596318244934,
+            0.01838536001741886,
+            -0.0033263994846493006,
+            0.00854811817407608,
+            -0.004838050343096256,
+            0.004696093499660492,
+            -0.023158187046647072,
+            0.014126647263765335,
+            -0.008164450526237488,
+            -0.0069098565727472305,
+            0.008878071792423725,
+            -0.0023653118405491114,
+            0.02231411822140217,
+            -0.020273005589842796,
+            -0.008908765390515327,
+            0.008855052292346954,
+            -0.022129956632852554,
+            -0.02206856943666935,
+            -0.0004939722712151706,
+            -0.019275467842817307,
+            -0.0014780800556764007,
+            -0.00021233613369986415,
+            -0.032964736223220825,
+            -0.026856742799282074,
+            0.003044403623789549,
+            -0.0073472377844154835,
+            -0.008287223987281322,
+            0.012753116898238659,
+            -7.697334513068199e-05,
+            -0.013850406743586063,
+            -0.003665945492684841,
+            0.01041274331510067,
+            0.00023139962286222726,
+            0.01800169236958027,
+            -0.021930448710918427,
+            0.0020372758153826,
+            0.004431362729519606,
+            -0.004327772185206413,
+            0.006725695915520191,
+            0.01712692901492119,
+            0.0005990013014525175,
+            0.0003234798787161708,
+            0.006084970664232969,
+            0.0034932950511574745,
+            0.010850124061107635,
+            0.015745725482702255,
+            0.015032103285193443,
+            -0.004174305126070976,
+            0.016068005934357643,
+            -0.001623873831704259,
+            2.7096535632153973e-05,
+            -0.02460077777504921,
+            2.6736846848507412e-05,
+            0.012415489181876183,
+            0.004450546111911535,
+            -0.02410968206822872,
+            -0.0031997892074286938,
+            -0.03278057277202606,
+            0.005133474711328745,
+            -0.019382895901799202,
+            0.0013802447356283665,
+            0.017571983858942986,
+            0.00728201400488615,
+            -0.004446709528565407,
+            0.014402887783944607,
+            0.022789865732192993,
+            -0.0010502905352041125,
+            0.0006623064982704818,
+            0.004515769425779581,
+            -0.003224727464839816,
+            0.002537962282076478,
+            0.0022080079652369022,
+            -0.01597592607140541,
+            -0.01414966769516468,
+            0.008156776428222656,
+            0.004611686337739229,
+            0.011310526169836521,
+            0.007385604549199343,
+            -0.015446464531123638,
+            0.012223655357956886,
+            -0.013934813439846039,
+            0.008164450526237488,
+            -0.00473829684779048,
+            -0.013988526538014412,
+            -0.013466738164424896,
+            0.005359838716685772,
+            0.017618024721741676,
+            -0.006012073718011379,
+            0.033425137400627136,
+            -0.00925406627357006,
+            0.01533136423677206,
+            -0.019306162372231483,
+            -0.009491940960288048,
+            -0.010282295756042004,
+            0.01184766087681055,
+            -0.01567666418850422,
+            -0.0009020988363772631,
+            0.0035508451983332634,
+            -0.010573883540928364,
+            -0.010351356118917465,
+            0.005294615402817726,
+            -0.0030309753492474556,
+            0.03956381976604462,
+            -0.0024631470441818237,
+            0.025705739855766296,
+            0.006031257100403309,
+            -0.011517706327140331,
+            -0.010075115598738194,
+            0.002643470885232091,
+            0.00800330936908722,
+            0.0180937722325325,
+            -0.006568392273038626,
+            -0.004354629199951887,
+            0.011041957885026932,
+            0.017955651506781578,
+            0.007251320872455835,
+            0.011563746258616447,
+            -0.004093735013157129,
+            0.0037311690393835306,
+            0.002374903531745076,
+            -0.013666246086359024,
+            -0.0031096271704882383,
+            0.011916721239686012,
+            0.017019502818584442,
+            0.00786518957465887,
+            0.012852869927883148,
+            -0.0003946982033085078,
+            0.004611686337739229,
+            0.013996199704706669,
+            -0.013121437281370163,
+            -0.005240901838988066,
+            0.008241184055805206,
+            -0.01721900887787342,
+            0.009223372675478458,
+            0.0044083427637815475,
+            -0.004638543352484703,
+            0.00893945898860693,
+            -0.005348328500986099,
+            -0.0018483194289729,
+            -0.02099430002272129,
+            -0.014134320430457592,
+            -0.0033494196832180023,
+            0.020932912826538086,
+            -0.006299824919551611,
+            0.0015452218940481544,
+            -0.006790919695049524,
+            -0.02567504718899727,
+            0.01705019548535347,
+            0.004550299607217312,
+            0.0065530454739928246,
+            0.005966033786535263
+          ],
+          "title": "2 Related Work",
+          "keyphrases": [
+            "neural network capacity",
+            "model parallelism",
+            "conditional computation",
+            "mixture of experts",
+            "Switch Transformer"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "86d0cb2e-1f6d-4e38-a077-7b7de2fe9e4a",
+        "properties": {
+          "page_content": "3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n",
+          "embedding": [
+            0.010104401037096977,
+            -0.020574966445565224,
+            -0.02167346142232418,
+            -0.031542472541332245,
+            0.03588413819670677,
+            0.03464615345001221,
+            -0.009755672886967659,
+            0.046869076788425446,
+            0.004803731571882963,
+            0.0364421010017395,
+            0.016547156497836113,
+            -0.007223033811897039,
+            0.009546436369419098,
+            -0.0017371026333421469,
+            0.005762734450399876,
+            0.04048734903335571,
+            0.011603932827711105,
+            0.01784616895020008,
+            0.011150586418807507,
+            -0.02915368229150772,
+            0.005383492447435856,
+            -0.01708768494427204,
+            -0.04317255690693855,
+            -0.006194285582751036,
+            0.009546436369419098,
+            0.005871712230145931,
+            0.027340294793248177,
+            0.0036812627222388983,
+            -0.055901139974594116,
+            0.023800702765583992,
+            0.021708333864808083,
+            0.010984940454363823,
+            -0.01007824670523405,
+            -0.010601338930428028,
+            -0.026573093608021736,
+            -0.021969880908727646,
+            0.04634598642587662,
+            0.007685098797082901,
+            -0.006399163510650396,
+            0.042475100606679916,
+            0.03703493997454643,
+            0.012728581205010414,
+            -0.012772172689437866,
+            -0.014175803400576115,
+            -0.003430614247918129,
+            -0.0012979229213669896,
+            -0.004712190479040146,
+            0.0065735275857150555,
+            0.009311044588685036,
+            -0.026172054931521416,
+            -0.001842810888774693,
+            -0.021900134161114693,
+            0.02798544242978096,
+            0.020609840750694275,
+            -0.049379922449588776,
+            0.028072625398635864,
+            -0.004890913609415293,
+            0.06461934745311737,
+            0.047461915761232376,
+            0.014602995477616787,
+            0.03794163465499878,
+            -0.034035876393318176,
+            -0.029275737702846527,
+            -0.003177786245942116,
+            0.03234454616904259,
+            0.007654585409909487,
+            -0.01033979281783104,
+            0.0010679803090170026,
+            0.011717269197106361,
+            -0.005706066265702248,
+            -0.044532597064971924,
+            0.010496720671653748,
+            -0.014925569295883179,
+            0.002881367225199938,
+            0.004034350160509348,
+            -0.020017001777887344,
+            0.03030448593199253,
+            0.0268520750105381,
+            0.014899414964020252,
+            0.010740830563008785,
+            0.010653648525476456,
+            0.03204812481999397,
+            -0.0026961052790284157,
+            0.009555154480040073,
+            0.0364421010017395,
+            -0.029415227472782135,
+            -0.01360040158033371,
+            -0.062178246676921844,
+            0.010540311224758625,
+            0.0028072623535990715,
+            -0.02815980650484562,
+            0.0018340926617383957,
+            -0.006569168530404568,
+            0.0036986989434808493,
+            -0.033094312995672226,
+            0.0727447122335434,
+            0.010653648525476456,
+            0.00023280335881281644,
+            -0.024044813588261604,
+            0.006394804455339909,
+            -0.015762517228722572,
+            0.021969880908727646,
+            0.0006560450419783592,
+            0.004008195362985134,
+            0.0229811929166317,
+            -0.02859571762382984,
+            -0.004690395202487707,
+            0.03815086930990219,
+            0.02148166112601757,
+            0.08676359057426453,
+            0.01007824670523405,
+            -0.017593340948224068,
+            -0.011211613193154335,
+            0.018482597544789314,
+            -0.04278895631432533,
+            -0.017689241096377373,
+            -0.03965040296316147,
+            -0.03215274587273598,
+            0.0114818774163723,
+            -0.025038689374923706,
+            0.01996469311416149,
+            0.02566639892756939,
+            -0.0011344566009938717,
+            -0.025422289967536926,
+            -0.07155904173851013,
+            0.04292844980955124,
+            -0.023696085438132286,
+            0.01780257746577263,
+            0.001439593848772347,
+            -0.0022144243121147156,
+            -0.00422179140150547,
+            0.0037183149252086878,
+            -0.020261110737919807,
+            -0.041882261633872986,
+            0.01856978051364422,
+            0.010479284450411797,
+            0.046834204345941544,
+            0.0633290484547615,
+            -0.021132932975888252,
+            -0.0072448295541107655,
+            0.017375385388731956,
+            -0.022894009947776794,
+            0.003243172774091363,
+            -0.0038142153061926365,
+            -0.029450101777911186,
+            0.013975284993648529,
+            0.028508534654974937,
+            0.08578714728355408,
+            -0.01956365443766117,
+            0.012022406794130802,
+            -0.033286113291978836,
+            0.037662651389837265,
+            -0.009799264371395111,
+            0.023417102172970772,
+            -0.025404853746294975,
+            -0.021708333864808083,
+            -0.025352545082569122,
+            0.010705957189202309,
+            -0.014995315112173557,
+            0.0059632533229887486,
+            -0.005889148451387882,
+            -0.0062945447862148285,
+            -0.018691834062337875,
+            0.010470565408468246,
+            0.007685098797082901,
+            0.047985006123781204,
+            0.029816266149282455,
+            0.005858634598553181,
+            -0.07769665867090225,
+            0.038325235247612,
+            0.017514877021312714,
+            0.036546722054481506,
+            0.01543994341045618,
+            0.033443041145801544,
+            -0.026538221165537834,
+            0.017680522054433823,
+            0.0032475318294018507,
+            0.01094134896993637,
+            -0.03197837993502617,
+            0.044916197657585144,
+            -0.00843050517141819,
+            0.002720080316066742,
+            0.01972058229148388,
+            0.013312701135873795,
+            0.04062684252858162,
+            0.027915697544813156,
+            0.034210242331027985,
+            0.03815086930990219,
+            -0.01332141924649477,
+            0.010453129187226295,
+            0.029380355030298233,
+            0.03515180945396423,
+            0.02796800620853901,
+            0.0027636713348329067,
+            0.0041890982538461685,
+            -0.026590529829263687,
+            -0.006721737328916788,
+            0.009964910335838795,
+            0.007039951626211405,
+            0.016390228644013405,
+            -0.012780890800058842,
+            -0.028124934062361717,
+            0.0220221895724535,
+            -0.028648026287555695,
+            0.030740395188331604,
+            0.008626664988696575,
+            -0.01083673071116209,
+            0.013208082877099514,
+            0.009895164519548416,
+            -0.029833702370524406,
+            0.004243587143719196,
+            -0.006250953767448664,
+            -0.004468080587685108,
+            -0.025021253153681755,
+            -0.01933698169887066,
+            0.00797280017286539,
+            0.0029532923363149166,
+            0.005021686665713787,
+            0.009380790404975414,
+            -0.022545281797647476,
+            0.004284998401999474,
+            0.02127242274582386,
+            0.022876573726534843,
+            -0.018325669690966606,
+            -0.018901070579886436,
+            -0.011438286863267422,
+            -0.01914518140256405,
+            -0.011865478940308094,
+            0.044393107295036316,
+            -0.012711144983768463,
+            -0.009694645181298256,
+            0.020574966445565224,
+            0.006774046458303928,
+            -0.005043482407927513,
+            0.017872322350740433,
+            -0.029484974220395088,
+            -0.010932630859315395,
+            -0.0035592077765613794,
+            0.011072122491896152,
+            0.010243892669677734,
+            -0.00997362844645977,
+            0.01550968922674656,
+            -0.050879452377557755,
+            -0.010932630859315395,
+            0.007881258614361286,
+            0.015082497149705887,
+            -0.0110285310074687,
+            0.02339966595172882,
+            0.012161897495388985,
+            0.016224581748247147,
+            -0.003081885864958167,
+            -0.014829669147729874,
+            0.008221268653869629,
+            0.017384104430675507,
+            0.007881258614361286,
+            0.05712168663740158,
+            0.02338222973048687,
+            -0.03947603702545166,
+            0.006538654677569866,
+            -0.029973194003105164,
+            0.05579652264714241,
+            0.007994594983756542,
+            -0.029833702370524406,
+            0.053215932101011276,
+            -0.023434538394212723,
+            -0.0024193022400140762,
+            0.060853078961372375,
+            -0.0007007258245721459,
+            -0.013548092916607857,
+            0.0013011923292651772,
+            0.02435866929590702,
+            -0.056075502187013626,
+            -0.04899631813168526,
+            -0.02148166112601757,
+            0.02857828140258789,
+            -0.02453303337097168,
+            0.0077156126499176025,
+            -0.015405070967972279,
+            0.04735729843378067,
+            0.036721084266901016,
+            0.018116433173418045,
+            -0.008731283247470856,
+            0.02008674666285515,
+            0.04749678820371628,
+            -0.007977158762514591,
+            -0.004742704331874847,
+            -0.02833417057991028,
+            0.005152460187673569,
+            -0.033216364681720734,
+            -0.009485408663749695,
+            -0.027462350204586983,
+            -0.011133149266242981,
+            -0.02125498652458191,
+            -0.0030317562632262707,
+            0.022336045280098915,
+            0.0041389684192836285,
+            5.1253518904559314e-05,
+            -0.0020389705896377563,
+            0.040522221475839615,
+            -0.024951506406068802,
+            -0.04470696300268173,
+            -0.00825178250670433,
+            0.01704409345984459,
+            0.0038556267973035574,
+            -0.05475033447146416,
+            -0.006250953767448664,
+            -0.0029707287903875113,
+            0.01304243691265583,
+            -0.023539157584309578,
+            -0.00863538309931755,
+            0.016573309898376465,
+            0.002870469354093075,
+            0.009807982482016087,
+            -0.02800287865102291,
+            -0.014629150740802288,
+            0.007406116463243961,
+            0.01841285265982151,
+            0.02817724272608757,
+            -0.0555872842669487,
+            -0.03225736320018768,
+            0.02067958563566208,
+            -0.009040780365467072,
+            -0.06409624963998795,
+            0.007833308540284634,
+            -0.001987751107662916,
+            0.01799437776207924,
+            0.016948193311691284,
+            0.04160328209400177,
+            -0.019581090658903122,
+            -0.022283734753727913,
+            -0.016599465161561966,
+            -0.0364072285592556,
+            -0.028700334951281548,
+            -0.009712082333862782,
+            -0.008299732580780983,
+            -0.028857262805104256,
+            -0.0017839629435911775,
+            0.01342603750526905,
+            -0.03759290650486946,
+            0.006176849361509085,
+            -0.021603714674711227,
+            0.0028486738447099924,
+            -0.03703493997454643,
+            0.007842026650905609,
+            -0.0030339357908815145,
+            -0.01613740064203739,
+            -0.02125498652458191,
+            -0.01561430748552084,
+            0.014725050888955593,
+            -0.0002036518562817946,
+            0.0017142173601314425,
+            0.0229637548327446,
+            0.007484580390155315,
+            0.009328480809926987,
+            -0.008535124361515045,
+            -0.029798829928040504,
+            -0.013382446952164173,
+            0.006586604751646519,
+            0.018534906208515167,
+            0.019999565556645393,
+            0.005501188337802887,
+            -0.04958915710449219,
+            -0.03584926575422287,
+            0.021621152758598328,
+            -0.05890020355582237,
+            0.006394804455339909,
+            -0.0017425514524802566,
+            -0.008622306399047375,
+            0.0033150978852063417,
+            -0.005832480266690254,
+            -0.026294110342860222,
+            0.015570716932415962,
+            0.0086135882884264,
+            0.012275234796106815,
+            0.007488939445465803,
+            0.015980472788214684,
+            0.02852597087621689,
+            -0.02451559714972973,
+            -0.05788889154791832,
+            0.03427998721599579,
+            -0.019075434654951096,
+            0.014210676774382591,
+            0.0440792515873909,
+            0.005475033540278673,
+            -0.006551732309162617,
+            0.025003815069794655,
+            0.013399883173406124,
+            -0.010034655220806599,
+            -0.03407074883580208,
+            0.04376539587974548,
+            -0.003430614247918129,
+            -0.07121030986309052,
+            0.005479392595589161,
+            0.0014810053398832679,
+            0.013103464618325233,
+            0.015283015556633472,
+            -0.003430614247918129,
+            -0.023643774911761284,
+            -0.01956365443766117,
+            -0.014088621363043785,
+            -0.018168741837143898,
+            0.029484974220395088,
+            0.003539591794833541,
+            0.005004250444471836,
+            -0.0006091846735216677,
+            -0.0014341449132189155,
+            0.004677317570894957,
+            0.014027593657374382,
+            0.04080120474100113,
+            -0.0364769771695137,
+            0.0025958458427339792,
+            -0.04275408387184143,
+            -0.03874370828270912,
+            -0.0046380856074392796,
+            0.018360542133450508,
+            0.026485910639166832,
+            -0.025910509750247,
+            0.02320786565542221,
+            0.0100608104839921,
+            -0.015675334259867668,
+            -0.005008609499782324,
+            0.028090061619877815,
+            0.025369981303811073,
+            0.01285063661634922,
+            -0.004533467348664999,
+            0.014690177515149117,
+            -0.012623962946236134,
+            0.0026045641861855984,
+            0.01574508100748062,
+            -0.00010305192699888721,
+            0.03490769863128662,
+            0.001937621389515698,
+            0.018918508663773537,
+            -0.04693882167339325,
+            -0.02699156664311886,
+            -0.027689022943377495,
+            -0.0172620490193367,
+            0.03874370828270912,
+            -0.006503781769424677,
+            -0.012728581205010414,
+            0.020261110737919807,
+            0.004282819107174873,
+            0.0017044093692675233,
+            -0.04066171497106552,
+            -0.028630590066313744,
+            -0.022283734753727913,
+            -0.044916197657585144,
+            -0.005618884228169918,
+            -0.02183038927614689,
+            -0.019057998433709145,
+            -0.05342517048120499,
+            0.0010527233825996518,
+            -0.07811512798070908,
+            -0.005370415281504393,
+            0.04048734903335571,
+            0.004686036147177219,
+            0.009424380958080292,
+            -0.028682898730039597,
+            0.01025261078029871,
+            -0.036337483674287796,
+            -0.02568383514881134,
+            0.017785141244530678,
+            0.04387001320719719,
+            -0.011813169345259666,
+            0.0019997386261820793,
+            0.019057998433709145,
+            -0.05056559666991234,
+            -0.005370415281504393,
+            -0.0031799657735973597,
+            0.007641507778316736,
+            0.012920381501317024,
+            -0.016974348574876785,
+            0.05457597225904465,
+            0.011054685339331627,
+            0.02223142609000206,
+            0.02606743760406971,
+            -0.00020024630066473037,
+            -0.038046251982450485,
+            0.017584621906280518,
+            0.008535124361515045,
+            0.02796800620853901,
+            0.0015681873774155974,
+            0.0073014977388083935,
+            -0.019807765260338783,
+            0.008979752659797668,
+            -0.009485408663749695,
+            0.01646869257092476,
+            0.018866198137402534,
+            0.02645103819668293,
+            -0.008644101209938526,
+            -0.002175192581489682,
+            0.0033434322103857994,
+            -0.0034785643219947815,
+            0.028700334951281548,
+            0.0076371487230062485,
+            -0.038848329335451126,
+            0.006813278421759605,
+            -0.002189359627664089,
+            0.023120682686567307,
+            0.030374230816960335,
+            -0.04108018800616264,
+            -0.02338222973048687,
+            -0.002796364715322852,
+            0.016974348574876785,
+            0.011019812896847725,
+            -0.020662149414420128,
+            -0.0031668883748352528,
+            0.044602345675230026,
+            -0.010479284450411797,
+            -0.032484035938978195,
+            0.01293781865388155,
+            -0.0018504393519833684,
+            0.0059806895442306995,
+            0.0024476363323628902,
+            0.012772172689437866,
+            0.013469628989696503,
+            -0.019598528742790222,
+            0.04087095335125923,
+            -0.004965018481016159,
+            0.013208082877099514,
+            -0.012031124904751778,
+            -0.010052092373371124,
+            -0.015195833519101143,
+            0.03926680237054825,
+            -0.0047906544059515,
+            0.02013905718922615,
+            0.024864325299859047,
+            0.006338136270642281,
+            0.05216974765062332,
+            -0.04125455394387245,
+            -0.03724417835474014,
+            0.005971971433609724,
+            0.0016215863870456815,
+            0.005871712230145931,
+            -0.011272640898823738,
+            -0.030932195484638214,
+            -0.012074715457856655,
+            0.004768858663737774,
+            -0.015535843558609486,
+            -0.015684053301811218,
+            0.00019043832435272634,
+            0.022144244983792305,
+            0.0011257383739575744,
+            0.005475033540278673,
+            0.00287700816988945,
+            0.014585559256374836,
+            0.03581439331173897,
+            -0.02589307352900505,
+            0.010758266784250736,
+            0.03147272393107414,
+            -0.009136680513620377,
+            -0.026764893904328346,
+            -0.027305422350764275,
+            0.015073779039084911,
+            -0.018918508663773537,
+            -0.028874699026346207,
+            3.807404209510423e-05,
+            -0.0034872826654464006,
+            0.008391273207962513,
+            -0.03584926575422287,
+            -0.0076763806864619255,
+            0.0021283321548253298,
+            -0.00196377607062459,
+            0.0007078094058670104,
+            -0.022318609058856964,
+            -0.013591683469712734,
+            0.01741897687315941,
+            0.024219177663326263,
+            0.005727861542254686,
+            0.03028704971075058,
+            -0.010243892669677734,
+            -0.00016768925706855953,
+            0.006525577511638403,
+            0.01589328981935978,
+            0.023050937801599503,
+            0.023312482982873917,
+            0.026573093608021736,
+            0.0009344827267341316,
+            -0.012327543459832668,
+            0.004252305254340172,
+            0.032484035938978195,
+            -0.016058936715126038,
+            -0.0030993223190307617,
+            0.014812232926487923,
+            0.03274558484554291,
+            0.010627493262290955,
+            -0.012188052758574486,
+            -0.02528279833495617,
+            0.015431225299835205,
+            -0.022283734753727913,
+            0.012510626576840878,
+            0.04777577146887779,
+            0.004393976181745529,
+            -0.029240863397717476,
+            0.02430635876953602,
+            -0.009546436369419098,
+            -0.024044813588261604,
+            0.027165930718183517,
+            0.003757546888664365,
+            -0.03271070867776871,
+            -6.98818766977638e-05,
+            -0.00018321855168323964,
+            0.014794795773923397,
+            -0.026015127077698708,
+            -0.015579435043036938,
+            0.02322530187666416,
+            0.009293608367443085,
+            0.0032627887558192015,
+            -0.021603714674711227,
+            0.001829733606427908,
+            0.015344043262302876,
+            0.0018177460879087448,
+            0.009424380958080292,
+            0.009145398624241352,
+            0.01505634281784296,
+            0.006939692422747612,
+            -0.013190646655857563,
+            -0.0032344546634703875,
+            -0.02720080316066742,
+            -0.006098385434597731,
+            0.0652819275856018,
+            -0.007292779628187418,
+            0.03686057776212692,
+            0.0015213270671665668,
+            0.012868072837591171,
+            0.025788454338908195,
+            -0.025160742923617363,
+            -0.02437610551714897,
+            -0.005047841463238001,
+            0.006961487699300051,
+            -0.04292844980955124,
+            -0.013269110582768917,
+            -0.023818138986825943,
+            -0.030217302963137627,
+            0.01933698169887066,
+            0.0005187332862988114,
+            0.0019267236348241568,
+            0.024829450994729996,
+            0.0051742554642260075,
+            -0.015038905665278435,
+            -0.017009221017360687,
+            0.0383601076900959,
+            0.016625620424747467,
+            0.028229551389813423,
+            -0.015989189967513084,
+            0.01625945419073105,
+            -0.008099214173853397,
+            0.003171247662976384,
+            0.03236198052763939,
+            -0.0026982848066836596,
+            -0.0030186788644641638,
+            0.009197707287967205,
+            0.01881388947367668,
+            0.040731459856033325,
+            -0.01745384931564331,
+            -0.030827578157186508,
+            -0.008914366364479065,
+            -0.014236831106245518,
+            0.021969880908727646,
+            -0.0009491947130300105,
+            -0.00041084547410719097,
+            -0.003934090491384268,
+            0.04352128505706787,
+            -0.029572155326604843,
+            0.014428631402552128,
+            -0.024445850402116776,
+            -0.039022691547870636,
+            0.017706677317619324,
+            -0.006102744489908218,
+            0.032693274319171906,
+            0.006883023772388697,
+            -0.007833308540284634,
+            -0.032867636531591415,
+            0.0016749853966757655,
+            0.0086179468780756,
+            -0.03194350749254227,
+            0.020418038591742516,
+            0.02472483366727829,
+            -0.00651250034570694,
+            0.0033063797745853662,
+            -0.016782546415925026,
+            0.008142804726958275,
+            -0.013844511471688747,
+            -0.022946318611502647,
+            0.022737082093954086,
+            -0.004298075567930937,
+            -0.028055187314748764,
+            0.006686864420771599,
+            -0.022458098828792572,
+            0.004707831423729658,
+            -0.0016771649243310094,
+            -0.00017014124023262411,
+            -0.00797280017286539,
+            0.010461847297847271,
+            -0.007571762427687645,
+            -0.01073211245238781,
+            -0.022353481501340866,
+            0.009991064667701721,
+            0.025631526485085487,
+            0.0017588981427252293,
+            -0.004672958515584469,
+            0.006848151329904795,
+            -0.03658159449696541,
+            0.00574093870818615,
+            0.02413199469447136,
+            -0.012205488979816437,
+            -0.008713847026228905,
+            -0.018255924805998802,
+            -0.029380355030298233,
+            -0.002366993110626936,
+            0.01130751334130764,
+            0.03584926575422287,
+            0.025788454338908195,
+            0.030409103259444237,
+            0.03391382470726967,
+            0.010810576379299164,
+            -0.00398204056546092,
+            0.009127962402999401,
+            -0.029031626880168915,
+            0.04006887599825859,
+            0.0040823002345860004,
+            0.02608487382531166,
+            0.016172273084521294,
+            -0.0459623858332634,
+            -0.00968592707067728,
+            0.014681459404528141,
+            0.027898261323571205,
+            0.044148996472358704,
+            0.025038689374923706,
+            -0.0010232995264232159,
+            0.027654150500893593,
+            -0.022196553647518158,
+            -0.02878751792013645,
+            0.01956365443766117,
+            -0.005235282704234123,
+            -0.00918898917734623,
+            -0.0013131798477843404,
+            0.021533969789743423,
+            -0.011385977268218994,
+            0.016058936715126038,
+            0.03415793180465698,
+            -0.008461019024252892,
+            0.012135743163526058,
+            0.0005590550135821104,
+            -0.008286654949188232,
+            0.026764893904328346,
+            -0.007214315701276064,
+            0.03485538810491562,
+            -0.01650356501340866,
+            -0.0018656961619853973,
+            0.000780279457103461,
+            -0.010766984894871712,
+            0.028020314872264862,
+            0.007270984351634979,
+            -0.06172490119934082,
+            0.00873564276844263,
+            0.011063403449952602,
+            0.013835793361067772,
+            -0.001278307056054473,
+            -0.0033957413397729397,
+            -0.025788454338908195,
+            0.019040562212467194,
+            0.008129727095365524,
+            0.02472483366727829,
+            0.010679802857339382,
+            0.019493909552693367,
+            -0.016643056645989418,
+            0.016599465161561966,
+            -0.0026590528432279825,
+            -0.02854340709745884,
+            0.007266624830663204,
+            0.013181927613914013,
+            0.0402083694934845,
+            -0.026538221165537834,
+            -0.002582768676802516,
+            0.010889039374887943,
+            -0.006970205809921026,
+            -0.011194176971912384,
+            -0.030443977564573288,
+            -0.0010756086558103561,
+            -0.028421353548765182,
+            0.02416686899960041,
+            -0.005139382556080818,
+            0.019406726583838463,
+            -0.027357731014490128,
+            0.0063337767496705055,
+            0.00043264098349027336,
+            -0.027654150500893593,
+            0.006390445400029421,
+            0.01782001368701458,
+            0.0031189383007586002,
+            -0.031176306307315826,
+            -0.006006844341754913,
+            0.023242738097906113,
+            0.03619799390435219,
+            0.006713018752634525,
+            -0.006848151329904795,
+            0.021533969789743423,
+            -0.015579435043036938,
+            -0.006150694563984871,
+            0.036546722054481506,
+            -0.025910509750247,
+            0.012231643311679363,
+            0.03759290650486946,
+            -0.03543078899383545,
+            -0.03196094557642937,
+            -0.0004021272761747241,
+            -0.04142891615629196,
+            0.001680434332229197,
+            -0.002975087845697999,
+            0.00835204217582941,
+            -0.0009715351043269038,
+            -0.027113622054457664,
+            0.024812014773488045,
+            -0.0003577189054340124,
+            0.013373728841543198,
+            -0.006848151329904795,
+            0.0021065366454422474,
+            -0.025143306702375412,
+            -0.01862208917737007,
+            -0.010793139226734638,
+            0.00015651904686819762,
+            0.0023059656377881765,
+            -0.008112290874123573,
+            -0.03944116458296776,
+            -0.020976005122065544,
+            0.007257906720042229,
+            0.00045770584256388247,
+            0.03584926575422287,
+            0.018482597544789314,
+            0.012188052758574486,
+            -0.02165602520108223,
+            -0.01898825354874134,
+            0.04756653308868408,
+            -0.026416165754199028,
+            0.0402432419359684,
+            -0.012135743163526058,
+            -0.017488721758127213,
+            0.002946753753349185,
+            0.02625923790037632,
+            -0.020923694595694542,
+            0.016817420721054077,
+            -0.01561430748552084,
+            0.005719143431633711,
+            -0.01409733947366476,
+            -0.004755781497806311,
+            0.009642336517572403,
+            -0.0017796038882806897,
+            0.018255924805998802,
+            0.0009072383400052786,
+            -0.0031123997177928686,
+            0.03183889016509056,
+            0.013652711175382137,
+            0.007763562723994255,
+            -0.016773829236626625,
+            -0.012222925201058388,
+            -0.010766984894871712,
+            0.007087901700288057,
+            -0.015073779039084911,
+            -0.007066106423735619,
+            -0.029659338295459747,
+            0.001694601378403604,
+            0.0007154378108680248,
+            -0.015256861224770546,
+            0.01235369872301817,
+            0.02706131339073181,
+            -0.013295264914631844,
+            -0.03157734498381615,
+            -0.006229158490896225,
+            -0.03944116458296776,
+            -0.021045750007033348,
+            0.019511345773935318,
+            -0.0005190056981518865,
+            0.009127962402999401,
+            0.004250125493854284,
+            0.00043509298120625317,
+            -0.022946318611502647,
+            -0.017976941540837288,
+            0.00714892940595746,
+            -0.009572590701282024,
+            0.00107724335975945,
+            0.00131862866692245,
+            0.01822105050086975,
+            -0.010156710632145405,
+            0.02800287865102291,
+            -0.017497440800070763,
+            0.015265579335391521,
+            -0.029851138591766357,
+            -0.02760184183716774,
+            -0.03199581801891327,
+            0.01304243691265583,
+            -0.014611713588237762,
+            -0.008195114322006702,
+            -0.010113119147717953,
+            0.0004999346565455198,
+            0.007567403372377157,
+            -0.009441817179322243,
+            0.00842614658176899,
+            0.011603932827711105,
+            0.013303983025252819,
+            0.007406116463243961,
+            -0.03244916349649429,
+            -0.02029598504304886,
+            -0.02397506684064865,
+            0.043974634259939194,
+            -0.030513722449541092,
+            0.01703537441790104,
+            -0.017392821609973907,
+            0.005823761690407991,
+            -0.0363723561167717,
+            -0.019075434654951096,
+            0.008953598327934742,
+            0.020749330520629883,
+            -0.025910509750247,
+            0.02741004154086113,
+            0.02934548258781433,
+            0.03013012185692787,
+            -0.036546722054481506,
+            0.025352545082569122,
+            -0.010505438782274723,
+            -0.02512587048113346,
+            0.027357731014490128,
+            0.03731392323970795,
+            0.03679082915186882,
+            -0.013460910879075527,
+            -0.012589089572429657,
+            -0.014690177515149117,
+            0.012318825349211693,
+            -0.0021555765997618437,
+            0.005801966413855553,
+            -0.037069812417030334,
+            0.01862208917737007,
+            0.009895164519548416,
+            0.005121946334838867,
+            0.03773239627480507,
+            -0.01898825354874134,
+            0.02054009400308132,
+            0.007471502758562565,
+            -0.01839541457593441,
+            0.006974565330892801,
+            0.000942111189942807,
+            -0.0004040343628730625,
+            0.003602798795327544,
+            0.017584621906280518,
+            0.005553497467190027,
+            -0.054541099816560745,
+            0.006582245696336031,
+            0.027898261323571205,
+            0.020418038591742516,
+            0.005235282704234123,
+            -0.026398729532957077,
+            0.014629150740802288,
+            0.015823544934391975,
+            0.002135960618034005,
+            0.007035592570900917,
+            0.016538437455892563,
+            -0.011080840602517128,
+            0.00042283302173018456,
+            0.0030339357908815145,
+            0.03720930591225624,
+            0.01475120522081852,
+            -0.013190646655857563,
+            -0.025056125596165657,
+            -0.016364073380827904,
+            0.0066519915126264095,
+            -0.005915303248912096,
+            -0.0004182014672551304,
+            -0.037488289177417755,
+            -0.03598875552415848,
+            0.013521937653422356,
+            -0.01818617805838585,
+            -0.03815086930990219,
+            -0.0042741005308926105,
+            -0.03222249075770378,
+            0.005200410261750221,
+            -0.014890696853399277,
+            -0.019319545477628708,
+            -0.03926680237054825,
+            -0.0038970382884144783,
+            0.00012893410166725516,
+            0.04045247659087181,
+            0.00037842465098947287,
+            0.03476820886135101,
+            -0.007253547664731741,
+            -0.0013676686212420464,
+            -0.004143327474594116,
+            0.022876573726534843,
+            -0.03147272393107414,
+            -0.04529980197548866,
+            0.026573093608021736,
+            0.005832480266690254,
+            0.016024062409996986,
+            0.035744644701480865,
+            -0.013870666734874249,
+            -0.010200301185250282,
+            0.024864325299859047,
+            -0.016747673973441124,
+            -0.010984940454363823,
+            -0.0022231426555663347,
+            0.0025980256032198668,
+            -0.012231643311679363,
+            -0.013521937653422356,
+            -0.011656241491436958,
+            0.05157690867781639,
+            0.005575293209403753,
+            0.027845950797200203,
+            0.04038273170590401,
+            0.0010271136416122317,
+            -0.04059197008609772,
+            0.0032889433205127716,
+            -0.0091018071398139,
+            -0.009642336517572403,
+            -0.004625008441507816,
+            0.033024564385414124,
+            0.001587803359143436,
+            0.017872322350740433,
+            -0.0042174323461949825,
+            -0.016416382044553757,
+            -0.020418038591742516,
+            -0.012318825349211693,
+            -0.00761535344645381,
+            -0.004145507235080004,
+            -0.03373945876955986,
+            -0.01256293524056673,
+            0.031716834753751755,
+            0.025335106998682022,
+            -0.022388353943824768,
+            -0.004925786517560482,
+            -0.01782001368701458,
+            -0.007672021631151438,
+            0.010479284450411797,
+            -0.002959831152111292,
+            -0.00416948227211833,
+            0.009250016883015633,
+            0.018133869394659996,
+            0.01007824670523405,
+            0.00766766257584095,
+            -0.05715656280517578,
+            -0.0019398009171709418,
+            0.011351104825735092,
+            0.010400820523500443,
+            -0.003968963399529457,
+            0.014541968703269958,
+            -0.012580371461808681,
+            -0.01265011727809906,
+            -0.01975545659661293,
+            0.0326235294342041,
+            0.006355572491884232,
+            0.030426539480686188,
+            -0.0005541510181501508,
+            0.009075652807950974,
+            0.02149909734725952,
+            0.004101916216313839,
+            0.012807045131921768,
+            0.00199102028273046,
+            0.019807765260338783,
+            0.00986029114574194,
+            -0.03142041712999344,
+            0.012414725497364998,
+            0.0010429153917357326,
+            -0.0024519956205040216,
+            -0.017872322350740433,
+            0.006922255735844374,
+            0.0023757112212479115,
+            -0.013949130661785603,
+            0.0041215321980416775,
+            -0.022370917722582817,
+            0.005954534746706486,
+            0.031089123338460922,
+            0.033233802765607834,
+            0.009206426329910755,
+            -0.015169679187238216,
+            -0.010810576379299164,
+            0.002057496691122651,
+            0.02008674666285515,
+            0.025404853746294975,
+            0.025003815069794655,
+            -0.0268172025680542,
+            -0.014393758960068226,
+            -0.003729212796315551,
+            -0.001253242138773203,
+            -0.010932630859315395,
+            0.03046141378581524,
+            0.02833417057991028,
+            -0.0008914365898817778,
+            0.0028835467528551817,
+            -0.0051916916854679585,
+            0.01523070689290762,
+            0.01898825354874134,
+            -0.017340512946248055,
+            0.0017665266059339046,
+            0.022283734753727913,
+            -0.020191365852952003,
+            0.022946318611502647,
+            -0.014672741293907166,
+            -0.011429568752646446,
+            0.02013905718922615,
+            0.09366840869188309,
+            -0.005248360335826874,
+            0.01120289508253336,
+            -0.028700334951281548,
+            -0.019214926287531853,
+            -0.008474096655845642,
+            -0.0035330532118678093,
+            0.016765110194683075,
+            0.026503346860408783,
+            0.009511562995612621,
+            0.021359605714678764,
+            0.015466097742319107,
+            -0.01666049286723137,
+            0.014411195181310177,
+            -0.009694645181298256,
+            0.016198428347706795,
+            -0.01362655684351921,
+            -0.004267562180757523,
+            0.010069528594613075,
+            -0.001332795829512179,
+            0.009816700592637062,
+            0.018674397841095924,
+            0.016799984499812126,
+            0.013190646655857563,
+            -0.024864325299859047,
+            0.00949412677437067,
+            0.002284170128405094,
+            -0.007763562723994255,
+            -0.02013905718922615,
+            -0.02322530187666416,
+            0.004999891389161348,
+            0.004428848624229431,
+            0.005296310409903526,
+            0.010331074707210064,
+            -0.0032889433205127716,
+            0.03811599686741829,
+            -0.03319893032312393,
+            0.01496044173836708,
+            0.006917896680533886,
+            -0.013103464618325233,
+            0.011935224756598473,
+            0.016198428347706795,
+            0.004049607086926699,
+            0.012249079532921314,
+            -0.011569059453904629,
+            0.014245549216866493,
+            0.02052265778183937,
+            -0.029781393706798553,
+            -0.00266123260371387,
+            -0.016372792422771454,
+            -0.008474096655845642,
+            0.0026677711866796017,
+            -0.016172273084521294,
+            -0.006006844341754913,
+            -0.00546195637434721,
+            0.01169983297586441,
+            0.0024825092405080795,
+            -0.0325886569917202,
+            0.005060918629169464,
+            0.00261110276915133,
+            0.005928380414843559,
+            0.018168741837143898,
+            0.01632048189640045,
+            0.01765436865389347,
+            -0.011525468900799751,
+            -0.01975545659661293,
+            0.006285826675593853,
+            0.0045029534958302975,
+            -0.01073211245238781,
+            0.0023473771288990974,
+            0.0024912275839596987,
+            0.01588457264006138,
+            -0.011185458861291409,
+            -0.005823761690407991,
+            -0.004141148179769516,
+            0.011664960533380508,
+            0.0031516316812485456,
+            -0.014541968703269958,
+            0.003940629307180643,
+            -0.021150369197130203,
+            0.020644713193178177,
+            0.011577778495848179,
+            -0.0019932000432163477,
+            -0.02088882215321064,
+            -0.02873520739376545,
+            0.01332141924649477,
+            -0.020836513489484787,
+            0.0031952227000147104,
+            0.015771236270666122,
+            -0.011673678644001484,
+            0.013260391540825367,
+            -0.018255924805998802,
+            -0.018866198137402534,
+            -0.0114818774163723,
+            -0.00473834527656436,
+            -0.005226564593613148,
+            -0.007576121482998133,
+            0.007502016611397266,
+            -0.01823848858475685,
+            -0.0306706503033638,
+            -0.033634841442108154,
+            0.019023125991225243,
+            0.01670408435165882,
+            -0.009467972442507744,
+            -0.015823544934391975,
+            -0.004642445128411055,
+            0.02319042943418026,
+            -0.0034415118861943483,
+            0.0007110786973498762,
+            -0.0026830281130969524,
+            0.013391165062785149,
+            -0.007610994391143322,
+            0.00016441992192994803,
+            -0.02284170128405094,
+            0.00044217653339728713,
+            0.0011725988006219268,
+            0.013391165062785149,
+            -0.027357731014490128,
+            -0.003380484413355589,
+            0.0003002332232426852,
+            -0.018098996952176094,
+            -0.004476799163967371,
+            0.04254484549164772,
+            -0.013417319394648075,
+            0.008979752659797668,
+            0.021551406010985374,
+            0.010104401037096977,
+            -0.00977310910820961,
+            0.010095682926476002,
+            0.007850744761526585,
+            -0.0028857262805104256,
+            0.026538221165537834,
+            0.022911446169018745,
+            0.01914518140256405,
+            0.01670408435165882,
+            -0.022161681205034256,
+            0.012528062798082829,
+            0.021969880908727646,
+            0.03752316161990166,
+            -0.017331793904304504,
+            -0.021952442824840546,
+            0.019860073924064636,
+            0.04676445946097374,
+            0.0021032672375440598,
+            -0.0006200824282132089,
+            -0.028648026287555695,
+            -0.017201021313667297,
+            0.024689961224794388,
+            0.020557530224323273,
+            0.016224581748247147,
+            -0.010121837258338928,
+            -0.031507596373558044,
+            0.014280421659350395,
+            -0.00309278373606503,
+            0.006865587551146746,
+            -0.01996469311416149,
+            0.01688716560602188,
+            -0.015004033222794533,
+            0.04257972165942192,
+            -0.012658835388720036,
+            0.004829886369407177,
+            0.017148712649941444,
+            0.011150586418807507,
+            -0.022318609058856964,
+            0.0009263094398193061,
+            -0.01666049286723137,
+            0.027933133766055107,
+            -0.018883634358644485,
+            0.003618055721744895,
+            0.01914518140256405,
+            -0.016608182340860367,
+            -0.014533249661326408,
+            0.004191277548670769,
+            0.019459037110209465,
+            0.006107103545218706,
+            -0.025247925892472267,
+            0.018098996952176094,
+            0.0019190951716154814,
+            0.009171552956104279,
+            -0.009930036962032318,
+            0.023870449513196945,
+            -0.027636714279651642,
+            -0.015387633815407753,
+            0.025003815069794655,
+            0.0041607641614973545,
+            0.013565529137849808,
+            -0.026311546564102173,
+            0.009380790404975414,
+            -0.029049063101410866,
+            0.02336479350924492,
+            0.0032584297005087137,
+            0.014280421659350395,
+            0.0015147883677855134,
+            0.002866110298782587,
+            -0.0015115190763026476,
+            0.014446067623794079,
+            -0.000550881668459624,
+            -0.022527845576405525,
+            -0.01569277234375477,
+            -0.03696519508957863,
+            -0.022737082093954086,
+            0.023818138986825943,
+            -0.002715721260756254,
+            0.003528693923726678,
+            0.02430635876953602,
+            -0.027462350204586983,
+            -0.01435888558626175,
+            0.009572590701282024,
+            0.02549203485250473,
+            -0.043625906109809875,
+            4.093470488442108e-05,
+            0.0013829254312440753,
+            -0.019232362508773804,
+            0.010627493262290955,
+            -0.005440160632133484,
+            0.015500971116125584,
+            0.0018776836805045605,
+            -0.037662651389837265,
+            0.011586496606469154,
+            0.03762777894735336,
+            -0.017924632877111435,
+            0.0028225192800164223,
+            0.017706677317619324,
+            0.012493189424276352,
+            -0.0012390750925987959,
+            -0.0017033196054399014,
+            0.01877901703119278,
+            -0.025945382192730904,
+            0.026102310046553612,
+            0.017026657238602638,
+            -0.014881978742778301,
+            0.007842026650905609,
+            -0.008753078989684582,
+            0.020731894299387932,
+            -0.0013208083109930158,
+            -0.013155773282051086,
+            0.017375385388731956,
+            0.002195898210629821,
+            0.01349578332155943,
+            -0.001912556472234428,
+            0.004304614383727312,
+            0.03107168711721897,
+            -0.01609380915760994,
+            0.02280682697892189,
+            0.01994725689291954,
+            3.6405323044164106e-05,
+            0.015405070967972279,
+            -0.019877510145306587,
+            -0.03312918543815613,
+            0.02221398986876011,
+            -0.013591683469712734,
+            -0.007310215849429369,
+            0.023347357288002968,
+            -0.010453129187226295,
+            -0.006250953767448664,
+            -0.004282819107174873,
+            0.004077941179275513,
+            0.012406007386744022,
+            -0.0230160653591156,
+            -0.015631744638085365,
+            0.022318609058856964,
+            0.017131276428699493,
+            0.022179117426276207,
+            0.023800702765583992,
+            -0.015378915704786777,
+            0.03182145208120346,
+            0.009223862551152706,
+            0.0065735275857150555,
+            0.0076371487230062485,
+            0.008395632728934288,
+            -0.005871712230145931,
+            0.01933698169887066,
+            0.014114775694906712,
+            0.007336370646953583,
+            -0.003565746359527111,
+            -0.02029598504304886,
+            -0.011447004973888397,
+            -0.027532095089554787,
+            -0.00045825072447769344,
+            -0.013914257287979126,
+            0.00704866973683238,
+            0.07797563821077347,
+            -0.006682505365461111,
+            0.0018144766800105572,
+            -0.007868180982768536,
+            0.000739412906114012,
+            0.0016771649243310094,
+            -0.005209128372371197,
+            0.016538437455892563,
+            -0.020662149414420128,
+            -0.020749330520629883,
+            -0.006163771729916334,
+            -0.010967503301799297,
+            0.013460910879075527,
+            0.01741897687315941,
+            0.0030034221708774567,
+            -0.02277195453643799,
+            0.011141867376863956,
+            -0.016355354338884354,
+            -0.011106994934380054,
+            0.010348510928452015,
+            -0.007231752388179302,
+            -0.011656241491436958,
+            0.03295481950044632,
+            -0.009206426329910755,
+            -0.03147272393107414,
+            0.015640461817383766,
+            -0.0024541751481592655,
+            -0.005976330488920212,
+            0.00470347236841917,
+            0.00853076484054327,
+            -0.004986814223229885,
+            0.0005544234300032258,
+            0.013504501432180405,
+            -0.002068394562229514,
+            0.005414006300270557,
+            0.005318105686455965,
+            -0.013565529137849808,
+            0.017296921461820602,
+            0.022527845576405525,
+            -0.033477913588285446,
+            -0.004755781497806311,
+            -0.013940411619842052,
+            0.010479284450411797,
+            0.018709270283579826,
+            0.015535843558609486,
+            -0.022580154240131378,
+            0.0025871277321130037,
+            -0.009154116734862328,
+            -0.012702426873147488,
+            -0.02741004154086113,
+            -0.018255924805998802,
+            0.00723611144348979,
+            -0.00851768720895052,
+            0.018656961619853973,
+            0.007907412946224213,
+            -0.005745298229157925,
+            0.005012968555092812,
+            -0.005178614519536495,
+            0.015518407337367535,
+            -0.0326235294342041,
+            0.010461847297847271,
+            -0.009337198920547962,
+            0.009581308811903,
+            0.01055774837732315,
+            -0.022388353943824768,
+            -0.014759923331439495,
+            -0.023417102172970772,
+            0.016555873677134514,
+            -0.010086964815855026,
+            -0.0020934592466801405,
+            0.02008674666285515,
+            0.0009088730439543724,
+            0.015413789078593254,
+            -0.001491903094574809,
+            0.023242738097906113,
+            0.014140930958092213,
+            0.016590746119618416,
+            -0.01215317938476801,
+            -0.001640112604945898,
+            -0.011124431155622005,
+            0.012719863094389439,
+            -0.02303350158035755,
+            -0.006813278421759605,
+            -0.026015127077698708,
+            0.03598875552415848,
+            -0.007602275814861059,
+            -0.01588457264006138,
+            0.02857828140258789,
+            -0.012414725497364998,
+            -0.0037052377592772245,
+            0.0058847893960773945,
+            -0.015431225299835205,
+            0.023451974615454674,
+            0.0004955755430273712,
+            0.011499314568936825,
+            -0.02549203485250473,
+            0.00863102450966835,
+            -0.007087901700288057,
+            0.006076589692384005,
+            -0.02128986082971096,
+            -0.002423661295324564,
+            0.012135743163526058,
+            -0.010426974855363369,
+            0.00411717314273119,
+            0.021185241639614105,
+            0.005331183318048716,
+            -0.01744513027369976,
+            0.03951091319322586,
+            -0.004025631584227085,
+            0.010653648525476456,
+            0.005475033540278673,
+            -0.0009028792264871299,
+            0.016799984499812126,
+            0.01742769405245781,
+            -0.004925786517560482,
+            -0.016547156497836113,
+            0.0032235567923635244,
+            -0.015021469444036484,
+            0.01561430748552084,
+            0.016233300790190697,
+            -0.02125498652458191,
+            -0.032658401876688004,
+            0.01779385842382908,
+            0.002061855746433139,
+            0.028072625398635864,
+            0.006983283441513777,
+            -0.025997690856456757,
+            -0.005296310409903526,
+            -0.03815086930990219,
+            0.012885509058833122,
+            -0.01801181398332119,
+            -0.02437610551714897,
+            0.002785466844215989,
+            -0.00986029114574194,
+            -0.01856978051364422,
+            0.02512587048113346,
+            0.013652711175382137,
+            -0.02552690915763378,
+            0.010889039374887943,
+            0.0013011923292651772,
+            -0.017549749463796616,
+            0.010052092373371124,
+            0.004350385162979364,
+            -0.012493189424276352,
+            0.007410475518554449,
+            0.005130664445459843,
+            -0.006978924386203289,
+            -0.02890957146883011,
+            -0.01670408435165882,
+            0.01972058229148388,
+            0.010993658564984798,
+            -0.0058455574326217175,
+            0.011290077120065689,
+            -0.00997362844645977,
+            0.0033456117380410433,
+            0.0035221553407609463,
+            0.00901462510228157,
+            -0.013051155023276806,
+            -0.007371243555098772,
+            -0.007162006571888924,
+            0.01973801851272583,
+            -0.016024062409996986,
+            0.007990236394107342,
+            -0.013443474657833576,
+            -0.007981518283486366,
+            0.011429568752646446,
+            -0.0013774766121059656,
+            0.014986597001552582,
+            -0.018744144588708878,
+            0.0073407297022640705,
+            0.01189163327217102,
+            -0.008883852511644363,
+            -0.010671084746718407,
+            -0.02437610551714897,
+            0.019110308960080147,
+            0.01438504084944725,
+            0.004694754257798195,
+            -0.03679082915186882,
+            0.003951527178287506,
+            -0.010453129187226295,
+            -0.0009775288635864854,
+            -0.008796670474112034,
+            0.019249800592660904,
+            0.018831325694918633,
+            0.007985876873135567,
+            0.013277828693389893,
+            -0.0023081451654434204,
+            0.013548092916607857,
+            -0.012414725497364998,
+            0.017593340948224068,
+            -0.004036529455333948,
+            -0.017575904726982117,
+            -0.005714784376323223,
+            0.019406726583838463,
+            -0.001491903094574809,
+            0.007903054356575012,
+            0.004372180439531803,
+            0.010226456448435783,
+            -0.012275234796106815,
+            -0.007493298500776291,
+            0.014045030809938908,
+            0.008378196507692337,
+            -0.0004288267809897661,
+            0.009250016883015633,
+            0.06758353859186172,
+            -0.00870512891560793,
+            0.023853013291954994,
+            0.006311981473118067,
+            -0.009345917031168938,
+            -0.008112290874123573,
+            0.0019932000432163477,
+            -0.02951984666287899,
+            0.023329921066761017,
+            0.011970097199082375,
+            -0.009180271066725254,
+            0.00036126066697761416,
+            -0.006726096384227276,
+            0.018325669690966606,
+            -0.009694645181298256,
+            0.016948193311691284,
+            -0.010052092373371124,
+            0.02435866929590702,
+            0.019162617623806,
+            0.005640679504722357,
+            0.005536061245948076,
+            0.0028203397523611784,
+            0.019459037110209465,
+            0.012868072837591171,
+            -0.002165384590625763,
+            -0.003123297356069088,
+            -0.009877728298306465,
+            0.019389290362596512,
+            -0.001910376944579184,
+            0.030792705714702606,
+            -0.0022144243121147156,
+            -0.005182973574846983,
+            -0.009842854924499989,
+            0.0041607641614973545,
+            0.011569059453904629,
+            0.009520281106233597,
+            0.012876790948212147,
+            -0.0013567708665505052,
+            -0.010679802857339382,
+            0.011909069493412971,
+            0.00863102450966835,
+            0.005326824262738228,
+            -0.0032518908847123384,
+            0.0045247492380440235,
+            -0.008404350839555264,
+            0.014001439325511456,
+            -0.0024781501851975918,
+            0.0015354941133409739,
+            -0.008774874731898308,
+            0.004426669329404831,
+            -0.01312961895018816,
+            0.0005348074482753873,
+            0.009607463143765926,
+            0.012868072837591171,
+            0.0015213270671665668,
+            -0.002273272257298231,
+            0.015405070967972279,
+            0.0025980256032198668,
+            0.0073014977388083935,
+            0.0009486498311161995,
+            -0.009537717327475548,
+            0.019057998433709145,
+            -0.01033979281783104,
+            -0.0019212746992707253,
+            0.010705957189202309,
+            0.006067871581763029,
+            0.008260500617325306,
+            0.01496916078031063,
+            0.009354635141789913,
+            0.003968963399529457,
+            0.0013545913388952613,
+            -0.00565375667065382,
+            -0.003535232739523053,
+            0.0036267738323658705,
+            0.0016771649243310094,
+            0.01646869257092476,
+            -0.019459037110209465,
+            -0.007039951626211405,
+            0.010679802857339382,
+            0.011438286863267422,
+            0.0017839629435911775,
+            0.015684053301811218,
+            0.014428631402552128,
+            0.01994725689291954,
+            -0.022562718018889427,
+            0.003831651760265231,
+            -0.002920599188655615,
+            0.019057998433709145,
+            0.008770515210926533,
+            -0.024114558473229408,
+            0.0003645300166681409,
+            -0.01063621137291193,
+            0.012676272541284561,
+            -0.0067086596973240376,
+            0.0023473771288990974,
+            0.01665177382528782,
+            -0.010121837258338928,
+            -0.009930036962032318,
+            -0.019424164667725563,
+            -0.016983065754175186,
+            0.010976221412420273,
+            0.014437349513173103,
+            0.0014188881032168865,
+            0.019232362508773804,
+            -0.0012412546202540398,
+            -0.008474096655845642,
+            0.030827578157186508,
+            0.016765110194683075,
+            -0.0004157494695391506,
+            -0.008962316438555717,
+            -0.0006544103380292654,
+            -0.015082497149705887,
+            0.01074954867362976,
+            0.004426669329404831,
+            0.005108869168907404,
+            0.003387023229151964,
+            9.174550359603018e-05,
+            -0.016712801530957222,
+            -0.014620431698858738,
+            -0.004668599460273981,
+            0.011045967228710651,
+            -0.003751008305698633,
+            0.009197707287967205,
+            -0.01609380915760994,
+            0.003121117828413844,
+            -0.008173318579792976,
+            -0.010714675299823284,
+            -0.008356400765478611,
+            -0.039406292140483856,
+            -0.02067958563566208,
+            0.0003173972072545439,
+            -0.00473834527656436,
+            -0.002358274767175317,
+            0.009223862551152706,
+            -0.003672544378787279,
+            0.013670147396624088,
+            -0.0015551100950688124,
+            0.0047906544059515,
+            0.008657178841531277,
+            0.01561430748552084,
+            -0.013504501432180405,
+            -0.0006631285650655627,
+            -0.014071185141801834,
+            -0.026782330125570297,
+            0.01707896590232849,
+            0.003199581755325198,
+            0.014446067623794079,
+            -0.015187115408480167,
+            0.009528999216854572,
+            -0.012589089572429657,
+            0.014463504776358604,
+            -0.0421612448990345,
+            -0.009520281106233597,
+            0.007031233515590429,
+            -0.0075979167595505714,
+            -0.005496829282492399,
+            -0.025369981303811073,
+            -0.0012205488746985793,
+            -0.011054685339331627,
+            -0.01666049286723137,
+            -0.019459037110209465,
+            0.00574093870818615,
+            -0.009311044588685036,
+            0.008591792546212673,
+            -0.006120180711150169,
+            0.02761927805840969,
+            0.00532246520742774,
+            -0.001346962875686586,
+            -0.006128899287432432,
+            0.009206426329910755,
+            -0.0007503106608055532,
+            0.012885509058833122,
+            0.008648460730910301,
+            0.005823761690407991,
+            0.023434538394212723,
+            0.0009339378448203206,
+            0.004690395202487707,
+            -0.006141976453363895,
+            0.014541968703269958,
+            0.024689961224794388,
+            0.011342386715114117,
+            0.011455723084509373,
+            0.005967612378299236,
+            0.0005950175691395998,
+            0.011429568752646446,
+            0.017392821609973907,
+            -0.003990759141743183,
+            -0.009171552956104279,
+            0.004869118332862854,
+            -0.01761949434876442,
+            -0.0033347138669341803,
+            0.012885509058833122,
+            -0.006128899287432432,
+            -0.02706131339073181,
+            0.01285063661634922,
+            0.004317691549658775,
+            -0.006673786789178848,
+            -0.016948193311691284,
+            -0.00661711860448122,
+            0.004646804183721542,
+            0.004729627165943384,
+            0.024236613884568214,
+            -0.022144244983792305,
+            -0.008644101209938526,
+            0.011917788535356522,
+            -0.014446067623794079,
+            -0.01063621137291193,
+            0.009790545329451561,
+            0.029484974220395088,
+            0.003496000776067376,
+            -0.05307644233107567,
+            -0.012467035092413425,
+            0.014236831106245518,
+            -0.01503018755465746,
+            -0.025980254635214806,
+            -0.0014428631402552128,
+            0.0038578063249588013,
+            -0.014010157436132431,
+            0.006625836715102196,
+            0.0011606112821027637,
+            -0.011708551086485386,
+            0.017139993607997894,
+            0.028874699026346207,
+            0.01312961895018816,
+            0.0114818774163723,
+            -0.025213053449988365,
+            -0.0008952508214861155,
+            -0.01914518140256405,
+            0.011752142570912838,
+            -0.030566031113266945,
+            -0.013984003104269505,
+            -0.0023342997301369905,
+            0.004141148179769516,
+            0.027444913983345032,
+            0.014376322738826275,
+            0.025962818413972855,
+            0.006778405513614416,
+            0.0037989583797752857,
+            -0.02125498652458191,
+            -0.018447725102305412,
+            0.0033456117380410433,
+            -0.0008304091752506793,
+            -0.002329940674826503,
+            -0.008216909132897854,
+            0.012275234796106815,
+            0.012039843015372753,
+            0.026172054931521416,
+            0.005087073426693678,
+            0.005553497467190027,
+            -2.538836997700855e-05,
+            -0.00494322320446372,
+            0.007593557704240084,
+            -0.019232362508773804,
+            -0.0029925242997705936,
+            -0.004559622146189213,
+            0.015405070967972279,
+            0.0045029534958302975,
+            0.0022275017108768225,
+            0.017357949167490005,
+            -0.0030099607538431883,
+            0.003729212796315551,
+            -0.019982129335403442,
+            -0.014672741293907166,
+            -0.0032518908847123384,
+            0.03525642678141594,
+            0.003971143160015345,
+            -0.02761927805840969,
+            -0.016224581748247147,
+            -0.0153353251516819,
+            0.030792705714702606,
+            -0.009581308811903,
+            0.004315512254834175,
+            0.02683463878929615,
+            0.01593688130378723,
+            0.012388571165502071,
+            0.006399163510650396,
+            -0.01822105050086975,
+            0.005021686665713787,
+            0.02931061014533043,
+            -0.0030317562632262707,
+            -0.010086964815855026,
+            -0.0066912234760820866,
+            0.007144570350646973,
+            -0.007798435632139444,
+            0.007632789667695761,
+            -0.0035068984143435955,
+            -0.013417319394648075,
+            0.029554719105362892,
+            -0.00968592707067728,
+            0.022370917722582817,
+            -0.010130555368959904,
+            0.00268738716840744,
+            -0.038848329335451126,
+            -0.002004097681492567,
+            0.0011453543556854129,
+            0.0019692247733473778,
+            0.004555262625217438,
+            -0.01881388947367668,
+            -0.008753078989684582,
+            0.011839324608445168,
+            -0.01276345457881689,
+            -0.007885618135333061,
+            -0.006137617398053408,
+            0.002983806189149618,
+            0.01977289281785488,
+            -0.00947669055312872,
+            0.0009857022669166327,
+            -0.0210980586707592,
+            0.01256293524056673,
+            -0.00282687833532691,
+            0.0017567186150699854,
+            0.010453129187226295,
+            0.007009437773376703,
+            0.03502975404262543,
+            0.01094134896993637,
+            0.005531701724976301,
+            -0.0032780456822365522,
+            -0.014053748920559883,
+            -0.03218761831521988,
+            0.013295264914631844,
+            -0.010453129187226295,
+            0.023800702765583992,
+            0.0036703648511320353,
+            -0.008103572763502598,
+            -0.0051960512064397335,
+            0.0037139558698982,
+            0.009590026922523975,
+            -0.0041106343269348145,
+            0.013478347100317478,
+            -0.010243892669677734,
+            0.021167805418372154,
+            0.0031298361718654633,
+            -0.005121946334838867,
+            0.012571653351187706,
+            -0.017689241096377373,
+            0.01055774837732315,
+            -0.01744513027369976,
+            0.0028225192800164223,
+            0.01360040158033371,
+            -0.010296201333403587,
+            0.0056276023387908936,
+            -0.052588220685720444,
+            -0.001197663601487875,
+            -0.016521001234650612,
+            -0.00675660977140069,
+            -0.008456660434603691,
+            -0.021394478157162666,
+            -0.03319893032312393,
+            -0.0036660057958215475,
+            0.015370197594165802,
+            0.006551732309162617,
+            0.023260174319148064,
+            0.0009895163821056485,
+            0.012257798574864864,
+            0.012807045131921768,
+            -0.0047906544059515,
+            0.012519344687461853,
+            0.033861514180898666,
+            -0.027741333469748497,
+            -0.019685709848999977,
+            0.005257078446447849,
+            -0.00028552126605063677,
+            -0.015788672491908073,
+            -0.013103464618325233,
+            -0.0017665266059339046,
+            0.0024955866392701864,
+            0.030443977564573288,
+            -0.020191365852952003,
+            -0.005579652264714241,
+            -0.007053028792142868,
+            0.004433208145201206,
+            -0.01918005384504795,
+            -0.0031450928654521704,
+            -0.011708551086485386,
+            -0.012780890800058842,
+            0.016041500493884087,
+            -0.011089558713138103,
+            0.011159304529428482,
+            -0.019511345773935318,
+            0.008626664988696575,
+            -0.015378915704786777,
+            -0.010627493262290955,
+            -0.0031690681353211403,
+            -0.010461847297847271,
+            0.030164994299411774,
+            -0.0041890982538461685,
+            0.0048386044800281525,
+            0.01670408435165882,
+            -0.00853076484054327,
+            0.011603932827711105,
+            -0.0006124540232121944,
+            0.007898694835603237,
+            -0.002704823622480035,
+            -0.0086135882884264,
+            0.0015278656501322985,
+            0.009537717327475548,
+            -0.010950067080557346,
+            0.03846472501754761,
+            -0.013731175102293491,
+            -0.008879492990672588,
+            -0.003859985852614045,
+            -0.004511671606451273,
+            0.002425840822979808,
+            0.0021762822289019823,
+            0.010984940454363823,
+            -0.0012870251666754484,
+            0.006835073698312044,
+            -0.024219177663326263,
+            -0.011717269197106361,
+            0.0013164491392672062,
+            0.006468909326940775,
+            0.003321636701002717,
+            0.008016390725970268,
+            -0.025579217821359634,
+            -0.0020782025530934334,
+            -0.012859354726970196,
+            0.005475033540278673,
+            -0.011499314568936825,
+            0.007332011591643095,
+            -0.01215317938476801,
+            -0.0013295264216139913,
+            0.009337198920547962,
+            0.012946536764502525,
+            0.013861947692930698,
+            -0.006608400493860245,
+            0.026573093608021736,
+            0.014899414964020252,
+            0.008709488436579704,
+            0.005405288189649582,
+            -0.033216364681720734,
+            -0.007802794687449932,
+            0.010365947149693966,
+            0.007519452832639217,
+            0.009232580661773682,
+            -0.017689241096377373,
+            -0.004176021087914705,
+            0.0029402151703834534,
+            -0.007039951626211405,
+            4.651980270864442e-05,
+            0.011080840602517128,
+            -0.010627493262290955,
+            0.005505547393113375,
+            -0.00031521765049546957,
+            0.0027549532242119312,
+            -0.006460190750658512,
+            0.012188052758574486,
+            -0.013286546804010868,
+            -0.008168959058821201,
+            0.0211154967546463,
+            0.02760184183716774,
+            -0.005531701724976301,
+            0.023696085438132286,
+            -0.021917570382356644,
+            0.008487174287438393,
+            -0.005043482407927513,
+            -0.014149649068713188,
+            -0.015300451777875423,
+            0.006883023772388697,
+            0.014777359552681446,
+            0.004760140553116798,
+            -0.013155773282051086,
+            -0.003027397207915783,
+            0.016983065754175186,
+            -0.0010952246375381947,
+            0.006216081324964762,
+            0.01593688130378723,
+            -0.00026971951592713594,
+            0.00017667989595793188,
+            -0.006002485286444426,
+            -0.01876158080995083,
+            -0.006290185730904341,
+            0.017924632877111435,
+            -0.0073799616657197475,
+            0.014262985438108444,
+            0.02320786565542221,
+            0.0005429807933978736,
+            0.011351104825735092,
+            -0.0038469084538519382,
+            0.01351321954280138,
+            0.01505634281784296,
+            -0.005235282704234123,
+            -0.0030600903555750847,
+            -0.0036965194158256054,
+            0.00011858122888952494,
+            0.012022406794130802,
+            -0.01741897687315941,
+            -0.011333668604493141,
+            0.0114818774163723,
+            -0.002121793571859598,
+            -0.008404350839555264,
+            -0.023486848920583725,
+            0.0016313943779096007,
+            0.0077156126499176025,
+            0.011447004973888397,
+            -0.016425101086497307,
+            -0.0005239096935838461,
+            0.000875089957844466,
+            0.010801857337355614,
+            -0.007497657556086779,
+            0.01159521471709013,
+            -0.007907412946224213,
+            -0.00863102450966835,
+            0.006381727289408445,
+            0.0015507509233430028,
+            0.007414834573864937,
+            -0.009999782778322697,
+            0.0015987011138349771,
+            -0.008025108836591244,
+            0.026590529829263687,
+            0.008317168802022934,
+            -0.0013011923292651772,
+            -0.005640679504722357,
+            0.004995532333850861,
+            0.008421787060797215,
+            -0.0100608104839921,
+            0.012815763242542744,
+            0.006107103545218706,
+            -0.002933676354587078,
+            0.010592620819807053,
+            0.015283015556633472,
+            0.0025631526950746775,
+            0.009328480809926987,
+            0.003443691413849592,
+            0.0059022256173193455,
+            -0.007022515404969454,
+            0.0017861425876617432,
+            -0.01550968922674656,
+            -0.020801641047000885,
+            -0.0035831828135997057,
+            0.016991784796118736,
+            -0.010583902709186077,
+            -0.001182406791485846,
+            0.017924632877111435,
+            0.01371373888105154,
+            -0.005034764297306538,
+            0.00551426550373435,
+            0.002217693720012903,
+            -0.0015529305674135685,
+            0.003323816228657961,
+            -0.014280421659350395,
+            -0.012493189424276352,
+            0.008726924657821655,
+            -0.02087138593196869,
+            0.01952878199517727,
+            -0.02165602520108223,
+            -0.035361044108867645,
+            -0.008831542916595936,
+            -0.009197707287967205,
+            -0.01707896590232849,
+            -0.013574247248470783,
+            0.023661212995648384,
+            0.016207145527005196,
+            -0.009354635141789913,
+            -0.034384604543447495,
+            0.03155990689992905,
+            0.00918898917734623,
+            0.027933133766055107,
+            -0.015544561669230461,
+            0.004343846347182989,
+            -0.0015213270671665668,
+            0.014707613736391068,
+            -0.006163771729916334,
+            -0.006686864420771599,
+            0.0091018071398139,
+            0.014620431698858738,
+            0.010897758416831493,
+            0.003245352301746607,
+            -0.003160349791869521,
+            0.007279702462255955,
+            0.002530459314584732,
+            0.0022285915911197662,
+            -0.008583074435591698,
+            -0.02282426506280899,
+            -0.009781827218830585,
+            0.017942069098353386,
+            0.022266298532485962,
+            0.012746017426252365,
+            -0.017131276428699493,
+            0.0020520477555692196,
+            -0.00484296353533864,
+            -0.014053748920559883,
+            -0.009991064667701721,
+            0.009642336517572403,
+            -0.033617403358221054,
+            0.02317299321293831,
+            0.004760140553116798,
+            -0.00627274950966239,
+            -0.0015006213216111064,
+            0.003836010815575719,
+            0.005679911468178034,
+            -0.004313332494348288,
+            0.0002961465797852725,
+            -0.002532638842239976,
+            0.010897758416831493,
+            0.016599465161561966,
+            0.005100150592625141,
+            -0.010400820523500443,
+            0.012388571165502071,
+            -0.0031407338101416826,
+            -0.002761491807177663,
+            -0.013914257287979126,
+            -0.006011203397065401,
+            -0.009502844884991646,
+            0.010261328890919685,
+            0.01483838725835085,
+            0.0054924702271819115,
+            0.006791482679545879,
+            0.004335128236562014,
+            -0.00013084120291750878,
+            -0.0002569146454334259,
+            0.008469737134873867,
+            -0.0005070181796327233,
+            -0.0045639812014997005,
+            0.0010821473551914096,
+            -0.0014679280575364828,
+            0.02320786565542221,
+            -0.00532246520742774,
+            -0.010950067080557346,
+            0.007196879480034113,
+            0.02033085748553276,
+            -0.022370917722582817,
+            0.022091934457421303,
+            -0.022179117426276207,
+            0.023853013291954994,
+            0.03482051566243172,
+            0.013975284993648529,
+            -0.011717269197106361,
+            -0.007096619810909033,
+            0.012815763242542744,
+            -0.012510626576840878,
+            -0.0009600924677215517,
+            0.0018079380970448256,
+            -0.022580154240131378,
+            0.015178397297859192,
+            -0.023347357288002968,
+            0.014280421659350395,
+            0.005121946334838867,
+            -0.00852640625089407,
+            -0.0073232934810221195,
+            -0.0009764390997588634,
+            0.018133869394659996,
+            -0.003990759141743183,
+            -0.0006255313055589795,
+            -0.03028704971075058,
+            -0.006930973846465349,
+            0.02163858897984028,
+            -0.0045247492380440235,
+            -0.013949130661785603,
+            0.007253547664731741,
+            -8.973622243502177e-06,
+            -0.016154836863279343,
+            -0.005876071285456419,
+            -0.03166452422738075,
+            -0.008142804726958275,
+            -0.02264990098774433,
+            -0.0075979167595505714,
+            -0.011577778495848179,
+            -0.012292671017348766,
+            -0.0038512677419930696,
+            -0.016276890411973,
+            0.001949608908034861,
+            -0.001063076313585043,
+            0.0033020207192748785,
+            0.0027091826777905226,
+            -0.009563872590661049,
+            -0.003968963399529457,
+            0.021324733272194862,
+            0.010705957189202309,
+            -0.016041500493884087,
+            -0.026206929236650467,
+            0.011638805270195007,
+            -0.012205488979816437,
+            -0.007593557704240084,
+            0.001547481631860137,
+            -0.005915303248912096,
+            -0.019267236813902855,
+            -0.01860465295612812,
+            0.005967612378299236,
+            0.0003718859807122499,
+            0.0051175872795283794,
+            0.013582965359091759,
+            0.014341449365019798,
+            -0.0019180054077878594,
+            -0.006734814494848251,
+            -1.330718441749923e-05,
+            -0.002195898210629821,
+            0.014027593657374382,
+            0.004348205402493477,
+            0.024445850402116776,
+            -0.008744360879063606,
+            0.0028944446239620447,
+            -0.017924632877111435,
+            0.00449859444051981,
+            -0.007393038831651211,
+            -0.005679911468178034,
+            -0.004064863547682762,
+            -0.011089558713138103,
+            0.005993766710162163,
+            -0.0011126610916107893,
+            -0.00247161160223186,
+            -0.007911772467195988,
+            -0.0020912797190248966,
+            -0.01707024872303009,
+            -0.0029794471338391304,
+            -0.020557530224323273,
+            -0.0013905538944527507,
+            -0.014916851185262203,
+            0.017000501975417137,
+            0.026172054931521416,
+            -0.00513502350077033,
+            -0.0006091846735216677,
+            -0.011229049414396286,
+            -4.08665910072159e-05,
+            -0.011909069493412971,
+            -0.00637736776843667,
+            -0.016416382044553757,
+            -0.0325189083814621,
+            -0.016904601827263832,
+            0.009049498476088047,
+            0.024288922548294067,
+            0.0019398009171709418,
+            -0.024986378848552704,
+            0.0029511128086596727,
+            -0.026607966050505638,
+            -0.014228112995624542,
+            -0.01496916078031063,
+            -0.006529936566948891,
+            -0.009136680513620377,
+            0.017392821609973907,
+            -0.0026350778061896563,
+            0.01571020856499672,
+            0.01747128553688526,
+            0.010906476527452469,
+            0.007497657556086779,
+            0.010880321264266968,
+            -0.009528999216854572,
+            -0.00016932391736190766,
+            0.009869009256362915,
+            -0.006311981473118067,
+            0.011839324608445168,
+            -0.00815152283757925,
+            -0.0011933045461773872,
+            0.01879645325243473,
+            0.008016390725970268,
+            -0.006255313288420439,
+            -0.0026699507143348455,
+            -0.02742747776210308,
+            0.017506157979369164,
+            -0.018552342429757118,
+            -0.00258494820445776,
+            -0.002944574225693941,
+            -0.016738956794142723,
+            0.02319042943418026,
+            0.017323076725006104,
+            -0.005021686665713787,
+            -0.0027549532242119312,
+            0.017889760434627533,
+            -0.006451472640037537,
+            0.00398204056546092,
+            0.0025762300938367844,
+            -0.0033281752839684486,
+            0.009790545329451561,
+            0.0027636713348329067,
+            -0.004577058367431164,
+            -0.0025587936397641897,
+            0.012196770869195461,
+            0.003901397343724966,
+            0.013530656695365906,
+            -0.027131058275699615,
+            -0.010566466487944126,
+            -0.0010303830495104194,
+            0.01935441792011261,
+            -0.005405288189649582,
+            -0.0007906323298811913,
+            -0.013312701135873795,
+            -0.0008854428306221962,
+            -0.005719143431633711,
+            0.023818138986825943,
+            -0.01992982067167759,
+            0.006120180711150169,
+            -0.008395632728934288,
+            0.013861947692930698,
+            -0.0032191977370530367,
+            0.010801857337355614,
+            -0.0007955363253131509,
+            -1.3375295566220302e-05,
+            0.009964910335838795,
+            0.012981409206986427,
+            -0.0022024367935955524,
+            0.024027377367019653,
+            -0.0004737800336442888,
+            -0.006403522565960884,
+            0.0027745692059397697,
+            -0.01895338110625744,
+            0.0229637548327446,
+            0.003402280155569315,
+            0.004315512254834175,
+            -0.0058629936538636684,
+            -0.008199472911655903,
+            0.039789892733097076,
+            0.021516533568501472,
+            0.010392102412879467,
+            -0.000592293159570545,
+            -0.01494300551712513,
+            0.028229551389813423,
+            0.007519452832639217,
+            0.00930232647806406,
+            -0.004012554418295622,
+            0.0027113622054457664,
+            -0.015178397297859192,
+            -0.010976221412420273,
+            0.019807765260338783,
+            -0.019999565556645393,
+            -0.008731283247470856,
+            -0.0026328982785344124,
+            -0.019703146070241928,
+            -0.029293173924088478,
+            0.015474816784262657,
+            0.009764390997588634,
+            0.0028443147893995047,
+            0.008495892398059368,
+            0.0037335718516260386,
+            -0.0015028008492663503,
+            -0.011647523380815983,
+            -0.003986400086432695,
+            -0.008574356324970722,
+            -0.01044441107660532,
+            0.006848151329904795,
+            -0.012004970572888851,
+            0.006717377807945013,
+            -0.008256141096353531,
+            -0.00024533577379770577,
+            0.01285063661634922,
+            0.005091432482004166,
+            0.0062771085649728775,
+            0.003443691413849592,
+            0.005078355316072702,
+            0.01591944508254528,
+            -0.013757329434156418,
+            0.016346637159585953,
+            0.017113838344812393,
+            0.0062945447862148285,
+            -0.0034131777938455343,
+            0.0023103246930986643,
+            -0.029188554733991623,
+            -0.00022381270537152886,
+            -0.0016869729151949286,
+            0.012632681056857109,
+            -0.02163858897984028,
+            0.004733986221253872,
+            0.021568842232227325,
+            -0.0016172273317351937,
+            0.009816700592637062,
+            0.014864541590213776,
+            -0.0001531135058030486,
+            0.028037751093506813,
+            0.011342386715114117,
+            0.02416686899960041,
+            -0.009459254331886768,
+            -0.02852597087621689,
+            0.006368649657815695,
+            -0.0038904997054487467,
+            0.014376322738826275,
+            0.0091018071398139,
+            0.0201216209679842,
+            -0.01369630265980959,
+            -0.006695582531392574,
+            0.007789717521518469,
+            0.0018918508430942893,
+            0.014306576922535896,
+            0.003217018209397793,
+            0.019302109256386757,
+            0.009119244292378426,
+            -0.0008625575574114919,
+            -0.007737408392131329,
+            8.588795753894374e-05,
+            0.006250953767448664,
+            -0.003500359831377864,
+            -0.03047885000705719,
+            0.0016390228411182761,
+            -0.004834245424717665,
+            0.004978095646947622,
+            -0.014245549216866493,
+            0.010409538634121418,
+            -0.0010015039006248116,
+            0.022196553647518158,
+            -0.0041498662903904915,
+            0.004064863547682762,
+            0.01708768494427204,
+            0.00938950851559639,
+            0.004991173278540373,
+            0.015300451777875423,
+            0.005658115725964308,
+            0.004511671606451273,
+            -0.008038186468183994,
+            -0.019493909552693367,
+            0.019162617623806,
+            -0.027462350204586983,
+            0.010653648525476456,
+            -0.00873564276844263,
+            0.004317691549658775,
+            0.003925372380763292,
+            -0.006887382827699184,
+            -0.002567511750385165,
+            0.003607157850638032,
+            0.001491903094574809,
+            -0.059144310653209686,
+            0.0037706242874264717,
+            0.0007388679659925401,
+            0.020452912896871567,
+            -0.00881846621632576,
+            0.0007590288296341896,
+            0.0026394370943307877,
+            0.00390357687138021,
+            -0.013347573578357697,
+            -0.0023604542948305607,
+            0.01571892574429512,
+            -0.01858721673488617,
+            -0.0059632533229887486,
+            0.004038709215819836,
+            -0.005557856522500515,
+            0.0035940806847065687,
+            -0.008779234252870083,
+            -0.002622000640258193,
+            0.02838647924363613,
+            -0.001655369414947927,
+            0.03140297904610634,
+            0.002506484277546406,
+            -0.0004756871494464576,
+            -0.01666049286723137,
+            0.007410475518554449,
+            -0.016738956794142723,
+            -0.00425884360447526,
+            0.008927443064749241,
+            -0.018726706504821777,
+            -0.00123580580111593,
+            0.001693511614575982,
+            -0.0013382446486502886,
+            -0.0077722808346152306,
+            -0.001763257198035717,
+            0.0010429153917357326,
+            -0.02048778533935547,
+            0.015997909009456635,
+            -0.023853013291954994,
+            0.01625945419073105,
+            -0.0034371528308838606,
+            -0.002946753753349185,
+            -0.0013033718569204211,
+            -0.0038098562508821487,
+            0.005666834302246571,
+            0.02531767077744007,
+            -0.015448661521077156,
+            -0.00522220553830266,
+            0.008365118876099586,
+            -0.003567925887182355,
+            0.002730978187173605,
+            -0.008761797100305557,
+            0.006656350567936897,
+            -0.0325886569917202,
+            -0.008112290874123573,
+            0.004413592163473368,
+            0.0031908636447042227,
+            -0.001236895564943552,
+            -0.02758440561592579,
+            0.003960245288908482,
+            -0.00854384247213602,
+            -0.024201741442084312,
+            -0.00027380615938454866,
+            -0.020592402666807175,
+            0.006717377807945013,
+            -0.00881846621632576,
+            0.01761949434876442,
+            -0.014568123035132885,
+            0.011098276823759079,
+            -0.014367603696882725,
+            0.016006626188755035,
+            -0.007406116463243961,
+            -0.009398226626217365,
+            -0.0069266147911548615,
+            -0.006338136270642281,
+            0.01044441107660532,
+            -0.014916851185262203,
+            -0.009511562995612621,
+            -0.007641507778316736,
+            0.005161178298294544,
+            0.011045967228710651,
+            0.013574247248470783,
+            0.0011333668371662498,
+            -0.00880538858473301,
+            -0.016547156497836113,
+            -0.005392210558056831,
+            -0.014881978742778301,
+            -0.0009301236714236438,
+            0.0042174323461949825,
+            0.020853949710726738,
+            0.008160240948200226,
+            -0.003753187833353877,
+            0.011586496606469154,
+            0.020801641047000885,
+            0.014132212847471237,
+            -0.014777359552681446,
+            0.0021697436459362507,
+            -0.010095682926476002,
+            0.013417319394648075,
+            0.022144244983792305,
+            0.002098908182233572,
+            0.002796364715322852,
+            0.006085308268666267,
+            0.007951004430651665,
+            -0.004396155476570129,
+            0.008291014470160007,
+            -0.025457162410020828,
+            -0.012667553499341011,
+            -0.006569168530404568,
+            0.005387851502746344,
+            -0.015378915704786777,
+            0.029973194003105164,
+            -0.003609337378293276,
+            0.004472440108656883,
+            -0.002182820811867714,
+            0.0326409637928009,
+            0.007196879480034113,
+            0.00473834527656436,
+            0.002874828642234206,
+            9.467427298659459e-05,
+            0.009136680513620377,
+            -0.01761077716946602,
+            -0.03888320177793503,
+            0.003254070645198226,
+            -0.000592293159570545,
+            0.01766308583319187,
+            -0.01514352485537529,
+            0.006878664717078209,
+            -0.0036899808328598738,
+            -0.01282448135316372,
+            -0.009633618406951427,
+            0.03138554468750954,
+            -0.005466315429657698,
+            0.03387894853949547,
+            -0.007558684796094894,
+            -0.010104401037096977,
+            0.021725770086050034,
+            0.021900134161114693,
+            -0.014210676774382591,
+            0.005531701724976301,
+            0.015762517228722572,
+            0.020278548821806908,
+            0.005209128372371197,
+            -0.00806434080004692,
+            0.02627667412161827,
+            0.012946536764502525,
+            -0.012807045131921768,
+            -0.0026350778061896563,
+            -0.009572590701282024,
+            0.006726096384227276,
+            -0.012301389127969742,
+            0.005387851502746344,
+            0.006434036418795586,
+            0.004986814223229885,
+            0.012963972985744476,
+            -0.01879645325243473,
+            -0.003831651760265231,
+            0.0073014977388083935,
+            0.013966566883027554,
+            0.00651250034570694,
+            0.017558468505740166,
+            -0.023643774911761284,
+            0.0028966241516172886,
+            -0.02493407018482685,
+            -0.008103572763502598,
+            0.0029968833550810814,
+            0.001748000388033688,
+            0.008875134401023388,
+            -0.01592816226184368,
+            -0.029484974220395088,
+            0.011516750790178776,
+            0.01708768494427204,
+            0.02568383514881134,
+            0.0011126610916107893,
+            0.008914366364479065,
+            0.00560144754126668,
+            -0.013739893212914467,
+            -0.00805998221039772,
+            -0.00997362844645977,
+            -0.01073211245238781,
+            -0.011159304529428482,
+            -0.002761491807177663,
+            0.009119244292378426,
+            -0.010592620819807053,
+            0.0006821996648795903,
+            0.016346637159585953,
+            -0.011717269197106361,
+            -0.0051960512064397335,
+            0.015535843558609486,
+            -0.005008609499782324,
+            0.0007977159111760557,
+            0.0016891524428501725,
+            0.013086027465760708,
+            0.0034480507019907236,
+            -0.0063730087131261826,
+            -0.01591944508254528,
+            0.01215317938476801,
+            -0.00417166156694293,
+            0.0016074193408712745,
+            0.00708354264497757,
+            -0.007933568209409714,
+            -0.0047819362953305244,
+            0.013373728841543198,
+            -0.00854384247213602,
+            0.0016259454423561692,
+            -0.007606634870171547,
+            -0.017549749463796616,
+            -0.0076938169077038765,
+            0.006394804455339909,
+            -0.026730021461844444,
+            -0.013478347100317478,
+            0.016233300790190697,
+            -0.003570105414837599,
+            0.001546391868032515,
+            -0.0005138292908668518,
+            -0.011124431155622005,
+            0.003160349791869521,
+            -0.019598528742790222,
+            -0.01064493041485548,
+            -0.004864759277552366,
+            -0.02662540227174759,
+            0.017549749463796616,
+            0.015091215260326862,
+            0.0033172776456922293,
+            -0.00699636060744524,
+            -0.005592729430645704,
+            0.002567511750385165,
+            0.010348510928452015,
+            0.01763693243265152,
+            0.009311044588685036,
+            -0.005592729430645704,
+            -0.019685709848999977,
+            -0.003443691413849592,
+            0.021324733272194862,
+            -6.375188240781426e-05,
+            -0.016590746119618416,
+            0.020609840750694275,
+            0.007109697442501783,
+            0.014306576922535896,
+            0.0055883703753352165,
+            0.0023451976012438536,
+            0.005383492447435856,
+            0.014114775694906712,
+            -0.0001486181718064472,
+            -0.01783744990825653,
+            -0.013984003104269505,
+            0.0004890369018539786,
+            0.03140297904610634,
+            0.00708354264497757,
+            0.017139993607997894,
+            -0.03623286634683609,
+            0.0006642183288931847,
+            -0.005335542373359203,
+            -0.0038490882143378258,
+            -0.0045988536439836025,
+            0.027671586722135544,
+            0.010766984894871712,
+            -0.0001184450084110722,
+            0.015605589374899864,
+            -0.010043373331427574,
+            0.0422309935092926,
+            -0.015378915704786777,
+            -0.00737560261040926,
+            -0.010052092373371124,
+            -0.01685229316353798,
+            -0.003940629307180643,
+            -0.0011769578559324145,
+            -0.01689588464796543,
+            -0.006647632457315922,
+            -0.006320699583739042,
+            0.021411914378404617,
+            0.0046380856074392796,
+            -0.01665177382528782,
+            0.0017065888969227672,
+            -0.023835577070713043,
+            -0.0010946798138320446,
+            -0.022475536912679672,
+            -0.006403522565960884,
+            -0.012327543459832668,
+            -0.01349578332155943,
+            -0.020993441343307495,
+            -0.02031342126429081,
+            -0.004908350296318531,
+            0.016547156497836113,
+            0.00532246520742774,
+            -0.0069091785699129105,
+            -0.0021043571177870035,
+            -0.014402477070689201,
+            -0.010278765112161636,
+            0.02144678682088852,
+            0.015605589374899864,
+            0.012432162649929523,
+            0.01332141924649477,
+            0.01685229316353798,
+            0.0037597264163196087,
+            -0.034593842923641205,
+            0.006128899287432432,
+            0.0041607641614973545,
+            -0.005435801576822996,
+            0.01782873272895813,
+            0.000511104823090136,
+            -0.010793139226734638,
+            0.013399883173406124,
+            -0.007593557704240084,
+            0.018098996952176094,
+            -0.003212659154087305,
+            -0.004184739198535681,
+            -0.005536061245948076,
+            -0.008740002289414406,
+            -0.007371243555098772,
+            0.00455090356990695,
+            0.0021043571177870035,
+            0.013643993064761162,
+            0.0005775811732746661,
+            -0.004625008441507816,
+            -0.011333668604493141,
+            0.0014602995943278074,
+            -0.0032257363200187683,
+            -0.001628125086426735,
+            -0.023835577070713043,
+            -0.012170616537332535,
+            0.0017120378324761987,
+            -0.0086179468780756,
+            -0.01235369872301817,
+            0.02856084331870079,
+            -0.0017011400777846575,
+            -0.0008009852026589215,
+            0.010688520967960358,
+            0.015675334259867668,
+            0.001266319421119988,
+            0.01837797835469246,
+            -0.012170616537332535,
+            -0.011185458861291409,
+            -0.01649484597146511,
+            -0.01282448135316372,
+            -0.01122033130377531,
+            0.02090625837445259,
+            0.0033979208674281836,
+            -0.00708354264497757,
+            0.010418256744742393,
+            -0.011106994934380054,
+            -0.005056559573858976,
+            -0.0091018071398139,
+            0.011708551086485386,
+            -0.02969421073794365,
+            0.02069702185690403,
+            -0.012711144983768463,
+            0.006024280562996864,
+            0.018639525398612022,
+            0.039231929928064346,
+            -0.006608400493860245,
+            -0.006869946606457233,
+            0.010034655220806599,
+            -0.003068808699026704,
+            0.029205990955233574,
+            -0.0059414575807750225,
+            0.004237048327922821,
+            -0.021551406010985374,
+            0.0018449904164299369,
+            -0.01074954867362976,
+            0.0012815763475373387,
+            0.01991238258779049,
+            -0.0191974900662899,
+            -0.013722456991672516,
+            -0.00675660977140069,
+            -0.016058936715126038,
+            -0.00823870487511158,
+            0.022283734753727913,
+            0.011377259157598019,
+            -0.006111462600529194,
+            -0.008273578248918056,
+            -0.005736579652875662,
+            -0.024201741442084312,
+            0.014716332778334618,
+            0.08174190670251846,
+            -0.009982346557080746,
+            -0.0001858067698776722,
+            0.0003367407189216465,
+            0.0017806936521083117,
+            0.007907412946224213,
+            -0.00031821453012526035,
+            0.05956278741359711,
+            -0.003509078174829483,
+            0.003899217816069722,
+            -0.012998845428228378,
+            0.01083673071116209,
+            -0.01438504084944725,
+            -0.005684270523488522,
+            -0.009712082333862782,
+            -0.01197881530970335,
+            -0.0008167869527824223,
+            -0.02203962579369545,
+            -0.001063076313585043,
+            -0.011996251530945301,
+            -0.005021686665713787,
+            -0.021324733272194862,
+            0.017514877021312714,
+            -0.02087138593196869,
+            -0.01648612879216671,
+            -0.004594494588673115,
+            -0.010705957189202309,
+            0.012118306942284107,
+            -0.016555873677134514,
+            0.002380070276558399,
+            0.007270984351634979,
+            0.014228112995624542,
+            0.007475861813873053,
+            -0.008199472911655903,
+            -0.008408710360527039,
+            -0.0076197125017642975,
+            0.004986814223229885,
+            0.0069876424968242645,
+            -0.0011088468600064516,
+            0.014637868851423264,
+            -0.02088882215321064,
+            -0.016058936715126038,
+            0.007589198648929596,
+            0.0017817834159359336,
+            0.009502844884991646,
+            -0.007820230908691883,
+            0.007449707482010126,
+            0.00020882829267065972,
+            0.022754518315196037,
+            -0.010156710632145405,
+            -0.0268520750105381,
+            -0.007933568209409714,
+            0.008644101209938526,
+            -0.013138337060809135,
+            -0.006102744489908218,
+            0.005331183318048716,
+            0.009807982482016087,
+            0.008081777021288872,
+            -0.009877728298306465,
+            0.0007110786973498762,
+            -0.018918508663773537,
+            0.013373728841543198,
+            0.003238813718780875,
+            0.022196553647518158,
+            -0.012667553499341011,
+            0.010880321264266968,
+            -0.02683463878929615,
+            -0.002574050333350897,
+            -0.014419913291931152,
+            0.0008592882077209651,
+            0.008434864692389965,
+            0.004742704331874847,
+            0.01684357412159443,
+            -0.017192302271723747,
+            -0.017096402123570442,
+            0.009311044588685036,
+            -0.01782873272895813,
+            -0.02376583032310009,
+            -0.0018471699440851808,
+            -0.02624180167913437,
+            -0.0038163948338478804,
+            0.007532530464231968,
+            -0.019389290362596512,
+            -0.01369630265980959,
+            0.0065953233279287815,
+            -0.0033412526827305555,
+            -0.014611713588237762,
+            0.017183585092425346,
+            -0.011970097199082375,
+            -0.03544822707772255,
+            0.003727033268660307,
+            0.008382555097341537,
+            -0.004729627165943384,
+            0.006747891660779715,
+            -0.024410977959632874,
+            0.016015345230698586,
+            -0.002397506730630994,
+            0.004293716512620449,
+            0.008029468357563019,
+            0.02606743760406971,
+            0.0010309278732165694,
+            0.0034545892849564552,
+            -0.005544779356569052,
+            0.014411195181310177,
+            0.005801966413855553,
+            0.023504285141825676,
+            0.026416165754199028,
+            0.0015529305674135685,
+            -0.010958785191178322,
+            -0.01081929448992014,
+            0.01592816226184368,
+            -0.011586496606469154,
+            -0.0048386044800281525,
+            0.022737082093954086,
+            0.013827075250446796,
+            -0.02203962579369545,
+            0.00017259323794860393,
+            -0.02149909734725952,
+            0.0069876424968242645,
+            -0.017148712649941444,
+            0.009467972442507744,
+            0.008731283247470856,
+            -0.005073996260762215,
+            0.02069702185690403,
+            0.010095682926476002,
+            0.030984506011009216,
+            0.01991238258779049,
+            0.002920599188655615,
+            0.011412132531404495,
+            -0.006451472640037537,
+            0.010470565408468246,
+            0.004537826403975487,
+            0.0022275017108768225,
+            -0.006608400493860245,
+            0.006930973846465349,
+            0.0023451976012438536,
+            -0.001170419156551361,
+            0.008583074435591698,
+            -0.007471502758562565,
+            0.009921318851411343,
+            0.0005939278053119779,
+            0.0055273426696658134,
+            -0.004287178162485361,
+            -0.010575184598565102,
+            -0.030339358374476433,
+            -0.003132015699520707,
+            0.020958568900823593,
+            -0.023539157584309578,
+            -0.008003313094377518,
+            0.005932739470154047,
+            0.016983065754175186,
+            -0.008282296359539032,
+            -0.012798327021300793,
+            -0.01708768494427204,
+            0.003105861134827137,
+            -0.0004893093137070537,
+            -0.010313638485968113,
+            0.01747128553688526,
+            0.0009546435903757811,
+            -0.004951941315084696,
+            0.008953598327934742,
+            -0.00680456031113863,
+            0.028212115168571472,
+            0.01505634281784296,
+            0.011150586418807507,
+            -0.012100870721042156,
+            0.014201958663761616,
+            -0.012711144983768463,
+            0.007680739741772413,
+            0.009799264371395111,
+            -0.006416599731892347,
+            -0.021220114082098007,
+            -0.017558468505740166,
+            0.023451974615454674,
+            0.006848151329904795,
+            0.012222925201058388,
+            -0.011987533420324326,
+            -0.0012815763475373387,
+            0.007475861813873053,
+            0.004834245424717665,
+            -0.01611996442079544,
+            -0.00048631246318109334,
+            0.0061986446380615234,
+            0.038604218512773514,
+            0.005087073426693678,
+            0.01072339341044426,
+            -0.0035984397400170565,
+            -0.0002066487359115854,
+            0.019162617623806,
+            -0.01952878199517727,
+            -0.006503781769424677,
+            -0.005832480266690254,
+            -0.02739260531961918,
+            0.016573309898376465,
+            -0.01685229316353798,
+            -0.016451256349682808,
+            0.010976221412420273,
+            -0.00041356991278007627,
+            -0.004422310274094343,
+            -0.01284191757440567,
+            -0.008766156621277332,
+            0.00794228632003069,
+            0.010810576379299164,
+            -0.009136680513620377,
+            0.00532246520742774,
+            0.00357446470297873,
+            -0.0035984397400170565,
+            0.02127242274582386,
+            0.0006467819330282509,
+            -0.002837776206433773,
+            0.007358166389167309
+          ],
+          "title": "Method",
+          "keyphrases": [
+            "mixture-of-experts (MoE) models",
+            "token-choice routing",
+            "load imbalance",
+            "expert choice method",
+            "Transformer architecture"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "50cfd5d2-e1de-4b02-aa71-bf2b53475f7b",
+      "type": "next",
+      "source": {
+        "id": "86d0cb2e-1f6d-4e38-a077-7b7de2fe9e4a",
+        "properties": {
+          "page_content": "3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n",
+          "embedding": [
+            0.010104401037096977,
+            -0.020574966445565224,
+            -0.02167346142232418,
+            -0.031542472541332245,
+            0.03588413819670677,
+            0.03464615345001221,
+            -0.009755672886967659,
+            0.046869076788425446,
+            0.004803731571882963,
+            0.0364421010017395,
+            0.016547156497836113,
+            -0.007223033811897039,
+            0.009546436369419098,
+            -0.0017371026333421469,
+            0.005762734450399876,
+            0.04048734903335571,
+            0.011603932827711105,
+            0.01784616895020008,
+            0.011150586418807507,
+            -0.02915368229150772,
+            0.005383492447435856,
+            -0.01708768494427204,
+            -0.04317255690693855,
+            -0.006194285582751036,
+            0.009546436369419098,
+            0.005871712230145931,
+            0.027340294793248177,
+            0.0036812627222388983,
+            -0.055901139974594116,
+            0.023800702765583992,
+            0.021708333864808083,
+            0.010984940454363823,
+            -0.01007824670523405,
+            -0.010601338930428028,
+            -0.026573093608021736,
+            -0.021969880908727646,
+            0.04634598642587662,
+            0.007685098797082901,
+            -0.006399163510650396,
+            0.042475100606679916,
+            0.03703493997454643,
+            0.012728581205010414,
+            -0.012772172689437866,
+            -0.014175803400576115,
+            -0.003430614247918129,
+            -0.0012979229213669896,
+            -0.004712190479040146,
+            0.0065735275857150555,
+            0.009311044588685036,
+            -0.026172054931521416,
+            -0.001842810888774693,
+            -0.021900134161114693,
+            0.02798544242978096,
+            0.020609840750694275,
+            -0.049379922449588776,
+            0.028072625398635864,
+            -0.004890913609415293,
+            0.06461934745311737,
+            0.047461915761232376,
+            0.014602995477616787,
+            0.03794163465499878,
+            -0.034035876393318176,
+            -0.029275737702846527,
+            -0.003177786245942116,
+            0.03234454616904259,
+            0.007654585409909487,
+            -0.01033979281783104,
+            0.0010679803090170026,
+            0.011717269197106361,
+            -0.005706066265702248,
+            -0.044532597064971924,
+            0.010496720671653748,
+            -0.014925569295883179,
+            0.002881367225199938,
+            0.004034350160509348,
+            -0.020017001777887344,
+            0.03030448593199253,
+            0.0268520750105381,
+            0.014899414964020252,
+            0.010740830563008785,
+            0.010653648525476456,
+            0.03204812481999397,
+            -0.0026961052790284157,
+            0.009555154480040073,
+            0.0364421010017395,
+            -0.029415227472782135,
+            -0.01360040158033371,
+            -0.062178246676921844,
+            0.010540311224758625,
+            0.0028072623535990715,
+            -0.02815980650484562,
+            0.0018340926617383957,
+            -0.006569168530404568,
+            0.0036986989434808493,
+            -0.033094312995672226,
+            0.0727447122335434,
+            0.010653648525476456,
+            0.00023280335881281644,
+            -0.024044813588261604,
+            0.006394804455339909,
+            -0.015762517228722572,
+            0.021969880908727646,
+            0.0006560450419783592,
+            0.004008195362985134,
+            0.0229811929166317,
+            -0.02859571762382984,
+            -0.004690395202487707,
+            0.03815086930990219,
+            0.02148166112601757,
+            0.08676359057426453,
+            0.01007824670523405,
+            -0.017593340948224068,
+            -0.011211613193154335,
+            0.018482597544789314,
+            -0.04278895631432533,
+            -0.017689241096377373,
+            -0.03965040296316147,
+            -0.03215274587273598,
+            0.0114818774163723,
+            -0.025038689374923706,
+            0.01996469311416149,
+            0.02566639892756939,
+            -0.0011344566009938717,
+            -0.025422289967536926,
+            -0.07155904173851013,
+            0.04292844980955124,
+            -0.023696085438132286,
+            0.01780257746577263,
+            0.001439593848772347,
+            -0.0022144243121147156,
+            -0.00422179140150547,
+            0.0037183149252086878,
+            -0.020261110737919807,
+            -0.041882261633872986,
+            0.01856978051364422,
+            0.010479284450411797,
+            0.046834204345941544,
+            0.0633290484547615,
+            -0.021132932975888252,
+            -0.0072448295541107655,
+            0.017375385388731956,
+            -0.022894009947776794,
+            0.003243172774091363,
+            -0.0038142153061926365,
+            -0.029450101777911186,
+            0.013975284993648529,
+            0.028508534654974937,
+            0.08578714728355408,
+            -0.01956365443766117,
+            0.012022406794130802,
+            -0.033286113291978836,
+            0.037662651389837265,
+            -0.009799264371395111,
+            0.023417102172970772,
+            -0.025404853746294975,
+            -0.021708333864808083,
+            -0.025352545082569122,
+            0.010705957189202309,
+            -0.014995315112173557,
+            0.0059632533229887486,
+            -0.005889148451387882,
+            -0.0062945447862148285,
+            -0.018691834062337875,
+            0.010470565408468246,
+            0.007685098797082901,
+            0.047985006123781204,
+            0.029816266149282455,
+            0.005858634598553181,
+            -0.07769665867090225,
+            0.038325235247612,
+            0.017514877021312714,
+            0.036546722054481506,
+            0.01543994341045618,
+            0.033443041145801544,
+            -0.026538221165537834,
+            0.017680522054433823,
+            0.0032475318294018507,
+            0.01094134896993637,
+            -0.03197837993502617,
+            0.044916197657585144,
+            -0.00843050517141819,
+            0.002720080316066742,
+            0.01972058229148388,
+            0.013312701135873795,
+            0.04062684252858162,
+            0.027915697544813156,
+            0.034210242331027985,
+            0.03815086930990219,
+            -0.01332141924649477,
+            0.010453129187226295,
+            0.029380355030298233,
+            0.03515180945396423,
+            0.02796800620853901,
+            0.0027636713348329067,
+            0.0041890982538461685,
+            -0.026590529829263687,
+            -0.006721737328916788,
+            0.009964910335838795,
+            0.007039951626211405,
+            0.016390228644013405,
+            -0.012780890800058842,
+            -0.028124934062361717,
+            0.0220221895724535,
+            -0.028648026287555695,
+            0.030740395188331604,
+            0.008626664988696575,
+            -0.01083673071116209,
+            0.013208082877099514,
+            0.009895164519548416,
+            -0.029833702370524406,
+            0.004243587143719196,
+            -0.006250953767448664,
+            -0.004468080587685108,
+            -0.025021253153681755,
+            -0.01933698169887066,
+            0.00797280017286539,
+            0.0029532923363149166,
+            0.005021686665713787,
+            0.009380790404975414,
+            -0.022545281797647476,
+            0.004284998401999474,
+            0.02127242274582386,
+            0.022876573726534843,
+            -0.018325669690966606,
+            -0.018901070579886436,
+            -0.011438286863267422,
+            -0.01914518140256405,
+            -0.011865478940308094,
+            0.044393107295036316,
+            -0.012711144983768463,
+            -0.009694645181298256,
+            0.020574966445565224,
+            0.006774046458303928,
+            -0.005043482407927513,
+            0.017872322350740433,
+            -0.029484974220395088,
+            -0.010932630859315395,
+            -0.0035592077765613794,
+            0.011072122491896152,
+            0.010243892669677734,
+            -0.00997362844645977,
+            0.01550968922674656,
+            -0.050879452377557755,
+            -0.010932630859315395,
+            0.007881258614361286,
+            0.015082497149705887,
+            -0.0110285310074687,
+            0.02339966595172882,
+            0.012161897495388985,
+            0.016224581748247147,
+            -0.003081885864958167,
+            -0.014829669147729874,
+            0.008221268653869629,
+            0.017384104430675507,
+            0.007881258614361286,
+            0.05712168663740158,
+            0.02338222973048687,
+            -0.03947603702545166,
+            0.006538654677569866,
+            -0.029973194003105164,
+            0.05579652264714241,
+            0.007994594983756542,
+            -0.029833702370524406,
+            0.053215932101011276,
+            -0.023434538394212723,
+            -0.0024193022400140762,
+            0.060853078961372375,
+            -0.0007007258245721459,
+            -0.013548092916607857,
+            0.0013011923292651772,
+            0.02435866929590702,
+            -0.056075502187013626,
+            -0.04899631813168526,
+            -0.02148166112601757,
+            0.02857828140258789,
+            -0.02453303337097168,
+            0.0077156126499176025,
+            -0.015405070967972279,
+            0.04735729843378067,
+            0.036721084266901016,
+            0.018116433173418045,
+            -0.008731283247470856,
+            0.02008674666285515,
+            0.04749678820371628,
+            -0.007977158762514591,
+            -0.004742704331874847,
+            -0.02833417057991028,
+            0.005152460187673569,
+            -0.033216364681720734,
+            -0.009485408663749695,
+            -0.027462350204586983,
+            -0.011133149266242981,
+            -0.02125498652458191,
+            -0.0030317562632262707,
+            0.022336045280098915,
+            0.0041389684192836285,
+            5.1253518904559314e-05,
+            -0.0020389705896377563,
+            0.040522221475839615,
+            -0.024951506406068802,
+            -0.04470696300268173,
+            -0.00825178250670433,
+            0.01704409345984459,
+            0.0038556267973035574,
+            -0.05475033447146416,
+            -0.006250953767448664,
+            -0.0029707287903875113,
+            0.01304243691265583,
+            -0.023539157584309578,
+            -0.00863538309931755,
+            0.016573309898376465,
+            0.002870469354093075,
+            0.009807982482016087,
+            -0.02800287865102291,
+            -0.014629150740802288,
+            0.007406116463243961,
+            0.01841285265982151,
+            0.02817724272608757,
+            -0.0555872842669487,
+            -0.03225736320018768,
+            0.02067958563566208,
+            -0.009040780365467072,
+            -0.06409624963998795,
+            0.007833308540284634,
+            -0.001987751107662916,
+            0.01799437776207924,
+            0.016948193311691284,
+            0.04160328209400177,
+            -0.019581090658903122,
+            -0.022283734753727913,
+            -0.016599465161561966,
+            -0.0364072285592556,
+            -0.028700334951281548,
+            -0.009712082333862782,
+            -0.008299732580780983,
+            -0.028857262805104256,
+            -0.0017839629435911775,
+            0.01342603750526905,
+            -0.03759290650486946,
+            0.006176849361509085,
+            -0.021603714674711227,
+            0.0028486738447099924,
+            -0.03703493997454643,
+            0.007842026650905609,
+            -0.0030339357908815145,
+            -0.01613740064203739,
+            -0.02125498652458191,
+            -0.01561430748552084,
+            0.014725050888955593,
+            -0.0002036518562817946,
+            0.0017142173601314425,
+            0.0229637548327446,
+            0.007484580390155315,
+            0.009328480809926987,
+            -0.008535124361515045,
+            -0.029798829928040504,
+            -0.013382446952164173,
+            0.006586604751646519,
+            0.018534906208515167,
+            0.019999565556645393,
+            0.005501188337802887,
+            -0.04958915710449219,
+            -0.03584926575422287,
+            0.021621152758598328,
+            -0.05890020355582237,
+            0.006394804455339909,
+            -0.0017425514524802566,
+            -0.008622306399047375,
+            0.0033150978852063417,
+            -0.005832480266690254,
+            -0.026294110342860222,
+            0.015570716932415962,
+            0.0086135882884264,
+            0.012275234796106815,
+            0.007488939445465803,
+            0.015980472788214684,
+            0.02852597087621689,
+            -0.02451559714972973,
+            -0.05788889154791832,
+            0.03427998721599579,
+            -0.019075434654951096,
+            0.014210676774382591,
+            0.0440792515873909,
+            0.005475033540278673,
+            -0.006551732309162617,
+            0.025003815069794655,
+            0.013399883173406124,
+            -0.010034655220806599,
+            -0.03407074883580208,
+            0.04376539587974548,
+            -0.003430614247918129,
+            -0.07121030986309052,
+            0.005479392595589161,
+            0.0014810053398832679,
+            0.013103464618325233,
+            0.015283015556633472,
+            -0.003430614247918129,
+            -0.023643774911761284,
+            -0.01956365443766117,
+            -0.014088621363043785,
+            -0.018168741837143898,
+            0.029484974220395088,
+            0.003539591794833541,
+            0.005004250444471836,
+            -0.0006091846735216677,
+            -0.0014341449132189155,
+            0.004677317570894957,
+            0.014027593657374382,
+            0.04080120474100113,
+            -0.0364769771695137,
+            0.0025958458427339792,
+            -0.04275408387184143,
+            -0.03874370828270912,
+            -0.0046380856074392796,
+            0.018360542133450508,
+            0.026485910639166832,
+            -0.025910509750247,
+            0.02320786565542221,
+            0.0100608104839921,
+            -0.015675334259867668,
+            -0.005008609499782324,
+            0.028090061619877815,
+            0.025369981303811073,
+            0.01285063661634922,
+            -0.004533467348664999,
+            0.014690177515149117,
+            -0.012623962946236134,
+            0.0026045641861855984,
+            0.01574508100748062,
+            -0.00010305192699888721,
+            0.03490769863128662,
+            0.001937621389515698,
+            0.018918508663773537,
+            -0.04693882167339325,
+            -0.02699156664311886,
+            -0.027689022943377495,
+            -0.0172620490193367,
+            0.03874370828270912,
+            -0.006503781769424677,
+            -0.012728581205010414,
+            0.020261110737919807,
+            0.004282819107174873,
+            0.0017044093692675233,
+            -0.04066171497106552,
+            -0.028630590066313744,
+            -0.022283734753727913,
+            -0.044916197657585144,
+            -0.005618884228169918,
+            -0.02183038927614689,
+            -0.019057998433709145,
+            -0.05342517048120499,
+            0.0010527233825996518,
+            -0.07811512798070908,
+            -0.005370415281504393,
+            0.04048734903335571,
+            0.004686036147177219,
+            0.009424380958080292,
+            -0.028682898730039597,
+            0.01025261078029871,
+            -0.036337483674287796,
+            -0.02568383514881134,
+            0.017785141244530678,
+            0.04387001320719719,
+            -0.011813169345259666,
+            0.0019997386261820793,
+            0.019057998433709145,
+            -0.05056559666991234,
+            -0.005370415281504393,
+            -0.0031799657735973597,
+            0.007641507778316736,
+            0.012920381501317024,
+            -0.016974348574876785,
+            0.05457597225904465,
+            0.011054685339331627,
+            0.02223142609000206,
+            0.02606743760406971,
+            -0.00020024630066473037,
+            -0.038046251982450485,
+            0.017584621906280518,
+            0.008535124361515045,
+            0.02796800620853901,
+            0.0015681873774155974,
+            0.0073014977388083935,
+            -0.019807765260338783,
+            0.008979752659797668,
+            -0.009485408663749695,
+            0.01646869257092476,
+            0.018866198137402534,
+            0.02645103819668293,
+            -0.008644101209938526,
+            -0.002175192581489682,
+            0.0033434322103857994,
+            -0.0034785643219947815,
+            0.028700334951281548,
+            0.0076371487230062485,
+            -0.038848329335451126,
+            0.006813278421759605,
+            -0.002189359627664089,
+            0.023120682686567307,
+            0.030374230816960335,
+            -0.04108018800616264,
+            -0.02338222973048687,
+            -0.002796364715322852,
+            0.016974348574876785,
+            0.011019812896847725,
+            -0.020662149414420128,
+            -0.0031668883748352528,
+            0.044602345675230026,
+            -0.010479284450411797,
+            -0.032484035938978195,
+            0.01293781865388155,
+            -0.0018504393519833684,
+            0.0059806895442306995,
+            0.0024476363323628902,
+            0.012772172689437866,
+            0.013469628989696503,
+            -0.019598528742790222,
+            0.04087095335125923,
+            -0.004965018481016159,
+            0.013208082877099514,
+            -0.012031124904751778,
+            -0.010052092373371124,
+            -0.015195833519101143,
+            0.03926680237054825,
+            -0.0047906544059515,
+            0.02013905718922615,
+            0.024864325299859047,
+            0.006338136270642281,
+            0.05216974765062332,
+            -0.04125455394387245,
+            -0.03724417835474014,
+            0.005971971433609724,
+            0.0016215863870456815,
+            0.005871712230145931,
+            -0.011272640898823738,
+            -0.030932195484638214,
+            -0.012074715457856655,
+            0.004768858663737774,
+            -0.015535843558609486,
+            -0.015684053301811218,
+            0.00019043832435272634,
+            0.022144244983792305,
+            0.0011257383739575744,
+            0.005475033540278673,
+            0.00287700816988945,
+            0.014585559256374836,
+            0.03581439331173897,
+            -0.02589307352900505,
+            0.010758266784250736,
+            0.03147272393107414,
+            -0.009136680513620377,
+            -0.026764893904328346,
+            -0.027305422350764275,
+            0.015073779039084911,
+            -0.018918508663773537,
+            -0.028874699026346207,
+            3.807404209510423e-05,
+            -0.0034872826654464006,
+            0.008391273207962513,
+            -0.03584926575422287,
+            -0.0076763806864619255,
+            0.0021283321548253298,
+            -0.00196377607062459,
+            0.0007078094058670104,
+            -0.022318609058856964,
+            -0.013591683469712734,
+            0.01741897687315941,
+            0.024219177663326263,
+            0.005727861542254686,
+            0.03028704971075058,
+            -0.010243892669677734,
+            -0.00016768925706855953,
+            0.006525577511638403,
+            0.01589328981935978,
+            0.023050937801599503,
+            0.023312482982873917,
+            0.026573093608021736,
+            0.0009344827267341316,
+            -0.012327543459832668,
+            0.004252305254340172,
+            0.032484035938978195,
+            -0.016058936715126038,
+            -0.0030993223190307617,
+            0.014812232926487923,
+            0.03274558484554291,
+            0.010627493262290955,
+            -0.012188052758574486,
+            -0.02528279833495617,
+            0.015431225299835205,
+            -0.022283734753727913,
+            0.012510626576840878,
+            0.04777577146887779,
+            0.004393976181745529,
+            -0.029240863397717476,
+            0.02430635876953602,
+            -0.009546436369419098,
+            -0.024044813588261604,
+            0.027165930718183517,
+            0.003757546888664365,
+            -0.03271070867776871,
+            -6.98818766977638e-05,
+            -0.00018321855168323964,
+            0.014794795773923397,
+            -0.026015127077698708,
+            -0.015579435043036938,
+            0.02322530187666416,
+            0.009293608367443085,
+            0.0032627887558192015,
+            -0.021603714674711227,
+            0.001829733606427908,
+            0.015344043262302876,
+            0.0018177460879087448,
+            0.009424380958080292,
+            0.009145398624241352,
+            0.01505634281784296,
+            0.006939692422747612,
+            -0.013190646655857563,
+            -0.0032344546634703875,
+            -0.02720080316066742,
+            -0.006098385434597731,
+            0.0652819275856018,
+            -0.007292779628187418,
+            0.03686057776212692,
+            0.0015213270671665668,
+            0.012868072837591171,
+            0.025788454338908195,
+            -0.025160742923617363,
+            -0.02437610551714897,
+            -0.005047841463238001,
+            0.006961487699300051,
+            -0.04292844980955124,
+            -0.013269110582768917,
+            -0.023818138986825943,
+            -0.030217302963137627,
+            0.01933698169887066,
+            0.0005187332862988114,
+            0.0019267236348241568,
+            0.024829450994729996,
+            0.0051742554642260075,
+            -0.015038905665278435,
+            -0.017009221017360687,
+            0.0383601076900959,
+            0.016625620424747467,
+            0.028229551389813423,
+            -0.015989189967513084,
+            0.01625945419073105,
+            -0.008099214173853397,
+            0.003171247662976384,
+            0.03236198052763939,
+            -0.0026982848066836596,
+            -0.0030186788644641638,
+            0.009197707287967205,
+            0.01881388947367668,
+            0.040731459856033325,
+            -0.01745384931564331,
+            -0.030827578157186508,
+            -0.008914366364479065,
+            -0.014236831106245518,
+            0.021969880908727646,
+            -0.0009491947130300105,
+            -0.00041084547410719097,
+            -0.003934090491384268,
+            0.04352128505706787,
+            -0.029572155326604843,
+            0.014428631402552128,
+            -0.024445850402116776,
+            -0.039022691547870636,
+            0.017706677317619324,
+            -0.006102744489908218,
+            0.032693274319171906,
+            0.006883023772388697,
+            -0.007833308540284634,
+            -0.032867636531591415,
+            0.0016749853966757655,
+            0.0086179468780756,
+            -0.03194350749254227,
+            0.020418038591742516,
+            0.02472483366727829,
+            -0.00651250034570694,
+            0.0033063797745853662,
+            -0.016782546415925026,
+            0.008142804726958275,
+            -0.013844511471688747,
+            -0.022946318611502647,
+            0.022737082093954086,
+            -0.004298075567930937,
+            -0.028055187314748764,
+            0.006686864420771599,
+            -0.022458098828792572,
+            0.004707831423729658,
+            -0.0016771649243310094,
+            -0.00017014124023262411,
+            -0.00797280017286539,
+            0.010461847297847271,
+            -0.007571762427687645,
+            -0.01073211245238781,
+            -0.022353481501340866,
+            0.009991064667701721,
+            0.025631526485085487,
+            0.0017588981427252293,
+            -0.004672958515584469,
+            0.006848151329904795,
+            -0.03658159449696541,
+            0.00574093870818615,
+            0.02413199469447136,
+            -0.012205488979816437,
+            -0.008713847026228905,
+            -0.018255924805998802,
+            -0.029380355030298233,
+            -0.002366993110626936,
+            0.01130751334130764,
+            0.03584926575422287,
+            0.025788454338908195,
+            0.030409103259444237,
+            0.03391382470726967,
+            0.010810576379299164,
+            -0.00398204056546092,
+            0.009127962402999401,
+            -0.029031626880168915,
+            0.04006887599825859,
+            0.0040823002345860004,
+            0.02608487382531166,
+            0.016172273084521294,
+            -0.0459623858332634,
+            -0.00968592707067728,
+            0.014681459404528141,
+            0.027898261323571205,
+            0.044148996472358704,
+            0.025038689374923706,
+            -0.0010232995264232159,
+            0.027654150500893593,
+            -0.022196553647518158,
+            -0.02878751792013645,
+            0.01956365443766117,
+            -0.005235282704234123,
+            -0.00918898917734623,
+            -0.0013131798477843404,
+            0.021533969789743423,
+            -0.011385977268218994,
+            0.016058936715126038,
+            0.03415793180465698,
+            -0.008461019024252892,
+            0.012135743163526058,
+            0.0005590550135821104,
+            -0.008286654949188232,
+            0.026764893904328346,
+            -0.007214315701276064,
+            0.03485538810491562,
+            -0.01650356501340866,
+            -0.0018656961619853973,
+            0.000780279457103461,
+            -0.010766984894871712,
+            0.028020314872264862,
+            0.007270984351634979,
+            -0.06172490119934082,
+            0.00873564276844263,
+            0.011063403449952602,
+            0.013835793361067772,
+            -0.001278307056054473,
+            -0.0033957413397729397,
+            -0.025788454338908195,
+            0.019040562212467194,
+            0.008129727095365524,
+            0.02472483366727829,
+            0.010679802857339382,
+            0.019493909552693367,
+            -0.016643056645989418,
+            0.016599465161561966,
+            -0.0026590528432279825,
+            -0.02854340709745884,
+            0.007266624830663204,
+            0.013181927613914013,
+            0.0402083694934845,
+            -0.026538221165537834,
+            -0.002582768676802516,
+            0.010889039374887943,
+            -0.006970205809921026,
+            -0.011194176971912384,
+            -0.030443977564573288,
+            -0.0010756086558103561,
+            -0.028421353548765182,
+            0.02416686899960041,
+            -0.005139382556080818,
+            0.019406726583838463,
+            -0.027357731014490128,
+            0.0063337767496705055,
+            0.00043264098349027336,
+            -0.027654150500893593,
+            0.006390445400029421,
+            0.01782001368701458,
+            0.0031189383007586002,
+            -0.031176306307315826,
+            -0.006006844341754913,
+            0.023242738097906113,
+            0.03619799390435219,
+            0.006713018752634525,
+            -0.006848151329904795,
+            0.021533969789743423,
+            -0.015579435043036938,
+            -0.006150694563984871,
+            0.036546722054481506,
+            -0.025910509750247,
+            0.012231643311679363,
+            0.03759290650486946,
+            -0.03543078899383545,
+            -0.03196094557642937,
+            -0.0004021272761747241,
+            -0.04142891615629196,
+            0.001680434332229197,
+            -0.002975087845697999,
+            0.00835204217582941,
+            -0.0009715351043269038,
+            -0.027113622054457664,
+            0.024812014773488045,
+            -0.0003577189054340124,
+            0.013373728841543198,
+            -0.006848151329904795,
+            0.0021065366454422474,
+            -0.025143306702375412,
+            -0.01862208917737007,
+            -0.010793139226734638,
+            0.00015651904686819762,
+            0.0023059656377881765,
+            -0.008112290874123573,
+            -0.03944116458296776,
+            -0.020976005122065544,
+            0.007257906720042229,
+            0.00045770584256388247,
+            0.03584926575422287,
+            0.018482597544789314,
+            0.012188052758574486,
+            -0.02165602520108223,
+            -0.01898825354874134,
+            0.04756653308868408,
+            -0.026416165754199028,
+            0.0402432419359684,
+            -0.012135743163526058,
+            -0.017488721758127213,
+            0.002946753753349185,
+            0.02625923790037632,
+            -0.020923694595694542,
+            0.016817420721054077,
+            -0.01561430748552084,
+            0.005719143431633711,
+            -0.01409733947366476,
+            -0.004755781497806311,
+            0.009642336517572403,
+            -0.0017796038882806897,
+            0.018255924805998802,
+            0.0009072383400052786,
+            -0.0031123997177928686,
+            0.03183889016509056,
+            0.013652711175382137,
+            0.007763562723994255,
+            -0.016773829236626625,
+            -0.012222925201058388,
+            -0.010766984894871712,
+            0.007087901700288057,
+            -0.015073779039084911,
+            -0.007066106423735619,
+            -0.029659338295459747,
+            0.001694601378403604,
+            0.0007154378108680248,
+            -0.015256861224770546,
+            0.01235369872301817,
+            0.02706131339073181,
+            -0.013295264914631844,
+            -0.03157734498381615,
+            -0.006229158490896225,
+            -0.03944116458296776,
+            -0.021045750007033348,
+            0.019511345773935318,
+            -0.0005190056981518865,
+            0.009127962402999401,
+            0.004250125493854284,
+            0.00043509298120625317,
+            -0.022946318611502647,
+            -0.017976941540837288,
+            0.00714892940595746,
+            -0.009572590701282024,
+            0.00107724335975945,
+            0.00131862866692245,
+            0.01822105050086975,
+            -0.010156710632145405,
+            0.02800287865102291,
+            -0.017497440800070763,
+            0.015265579335391521,
+            -0.029851138591766357,
+            -0.02760184183716774,
+            -0.03199581801891327,
+            0.01304243691265583,
+            -0.014611713588237762,
+            -0.008195114322006702,
+            -0.010113119147717953,
+            0.0004999346565455198,
+            0.007567403372377157,
+            -0.009441817179322243,
+            0.00842614658176899,
+            0.011603932827711105,
+            0.013303983025252819,
+            0.007406116463243961,
+            -0.03244916349649429,
+            -0.02029598504304886,
+            -0.02397506684064865,
+            0.043974634259939194,
+            -0.030513722449541092,
+            0.01703537441790104,
+            -0.017392821609973907,
+            0.005823761690407991,
+            -0.0363723561167717,
+            -0.019075434654951096,
+            0.008953598327934742,
+            0.020749330520629883,
+            -0.025910509750247,
+            0.02741004154086113,
+            0.02934548258781433,
+            0.03013012185692787,
+            -0.036546722054481506,
+            0.025352545082569122,
+            -0.010505438782274723,
+            -0.02512587048113346,
+            0.027357731014490128,
+            0.03731392323970795,
+            0.03679082915186882,
+            -0.013460910879075527,
+            -0.012589089572429657,
+            -0.014690177515149117,
+            0.012318825349211693,
+            -0.0021555765997618437,
+            0.005801966413855553,
+            -0.037069812417030334,
+            0.01862208917737007,
+            0.009895164519548416,
+            0.005121946334838867,
+            0.03773239627480507,
+            -0.01898825354874134,
+            0.02054009400308132,
+            0.007471502758562565,
+            -0.01839541457593441,
+            0.006974565330892801,
+            0.000942111189942807,
+            -0.0004040343628730625,
+            0.003602798795327544,
+            0.017584621906280518,
+            0.005553497467190027,
+            -0.054541099816560745,
+            0.006582245696336031,
+            0.027898261323571205,
+            0.020418038591742516,
+            0.005235282704234123,
+            -0.026398729532957077,
+            0.014629150740802288,
+            0.015823544934391975,
+            0.002135960618034005,
+            0.007035592570900917,
+            0.016538437455892563,
+            -0.011080840602517128,
+            0.00042283302173018456,
+            0.0030339357908815145,
+            0.03720930591225624,
+            0.01475120522081852,
+            -0.013190646655857563,
+            -0.025056125596165657,
+            -0.016364073380827904,
+            0.0066519915126264095,
+            -0.005915303248912096,
+            -0.0004182014672551304,
+            -0.037488289177417755,
+            -0.03598875552415848,
+            0.013521937653422356,
+            -0.01818617805838585,
+            -0.03815086930990219,
+            -0.0042741005308926105,
+            -0.03222249075770378,
+            0.005200410261750221,
+            -0.014890696853399277,
+            -0.019319545477628708,
+            -0.03926680237054825,
+            -0.0038970382884144783,
+            0.00012893410166725516,
+            0.04045247659087181,
+            0.00037842465098947287,
+            0.03476820886135101,
+            -0.007253547664731741,
+            -0.0013676686212420464,
+            -0.004143327474594116,
+            0.022876573726534843,
+            -0.03147272393107414,
+            -0.04529980197548866,
+            0.026573093608021736,
+            0.005832480266690254,
+            0.016024062409996986,
+            0.035744644701480865,
+            -0.013870666734874249,
+            -0.010200301185250282,
+            0.024864325299859047,
+            -0.016747673973441124,
+            -0.010984940454363823,
+            -0.0022231426555663347,
+            0.0025980256032198668,
+            -0.012231643311679363,
+            -0.013521937653422356,
+            -0.011656241491436958,
+            0.05157690867781639,
+            0.005575293209403753,
+            0.027845950797200203,
+            0.04038273170590401,
+            0.0010271136416122317,
+            -0.04059197008609772,
+            0.0032889433205127716,
+            -0.0091018071398139,
+            -0.009642336517572403,
+            -0.004625008441507816,
+            0.033024564385414124,
+            0.001587803359143436,
+            0.017872322350740433,
+            -0.0042174323461949825,
+            -0.016416382044553757,
+            -0.020418038591742516,
+            -0.012318825349211693,
+            -0.00761535344645381,
+            -0.004145507235080004,
+            -0.03373945876955986,
+            -0.01256293524056673,
+            0.031716834753751755,
+            0.025335106998682022,
+            -0.022388353943824768,
+            -0.004925786517560482,
+            -0.01782001368701458,
+            -0.007672021631151438,
+            0.010479284450411797,
+            -0.002959831152111292,
+            -0.00416948227211833,
+            0.009250016883015633,
+            0.018133869394659996,
+            0.01007824670523405,
+            0.00766766257584095,
+            -0.05715656280517578,
+            -0.0019398009171709418,
+            0.011351104825735092,
+            0.010400820523500443,
+            -0.003968963399529457,
+            0.014541968703269958,
+            -0.012580371461808681,
+            -0.01265011727809906,
+            -0.01975545659661293,
+            0.0326235294342041,
+            0.006355572491884232,
+            0.030426539480686188,
+            -0.0005541510181501508,
+            0.009075652807950974,
+            0.02149909734725952,
+            0.004101916216313839,
+            0.012807045131921768,
+            0.00199102028273046,
+            0.019807765260338783,
+            0.00986029114574194,
+            -0.03142041712999344,
+            0.012414725497364998,
+            0.0010429153917357326,
+            -0.0024519956205040216,
+            -0.017872322350740433,
+            0.006922255735844374,
+            0.0023757112212479115,
+            -0.013949130661785603,
+            0.0041215321980416775,
+            -0.022370917722582817,
+            0.005954534746706486,
+            0.031089123338460922,
+            0.033233802765607834,
+            0.009206426329910755,
+            -0.015169679187238216,
+            -0.010810576379299164,
+            0.002057496691122651,
+            0.02008674666285515,
+            0.025404853746294975,
+            0.025003815069794655,
+            -0.0268172025680542,
+            -0.014393758960068226,
+            -0.003729212796315551,
+            -0.001253242138773203,
+            -0.010932630859315395,
+            0.03046141378581524,
+            0.02833417057991028,
+            -0.0008914365898817778,
+            0.0028835467528551817,
+            -0.0051916916854679585,
+            0.01523070689290762,
+            0.01898825354874134,
+            -0.017340512946248055,
+            0.0017665266059339046,
+            0.022283734753727913,
+            -0.020191365852952003,
+            0.022946318611502647,
+            -0.014672741293907166,
+            -0.011429568752646446,
+            0.02013905718922615,
+            0.09366840869188309,
+            -0.005248360335826874,
+            0.01120289508253336,
+            -0.028700334951281548,
+            -0.019214926287531853,
+            -0.008474096655845642,
+            -0.0035330532118678093,
+            0.016765110194683075,
+            0.026503346860408783,
+            0.009511562995612621,
+            0.021359605714678764,
+            0.015466097742319107,
+            -0.01666049286723137,
+            0.014411195181310177,
+            -0.009694645181298256,
+            0.016198428347706795,
+            -0.01362655684351921,
+            -0.004267562180757523,
+            0.010069528594613075,
+            -0.001332795829512179,
+            0.009816700592637062,
+            0.018674397841095924,
+            0.016799984499812126,
+            0.013190646655857563,
+            -0.024864325299859047,
+            0.00949412677437067,
+            0.002284170128405094,
+            -0.007763562723994255,
+            -0.02013905718922615,
+            -0.02322530187666416,
+            0.004999891389161348,
+            0.004428848624229431,
+            0.005296310409903526,
+            0.010331074707210064,
+            -0.0032889433205127716,
+            0.03811599686741829,
+            -0.03319893032312393,
+            0.01496044173836708,
+            0.006917896680533886,
+            -0.013103464618325233,
+            0.011935224756598473,
+            0.016198428347706795,
+            0.004049607086926699,
+            0.012249079532921314,
+            -0.011569059453904629,
+            0.014245549216866493,
+            0.02052265778183937,
+            -0.029781393706798553,
+            -0.00266123260371387,
+            -0.016372792422771454,
+            -0.008474096655845642,
+            0.0026677711866796017,
+            -0.016172273084521294,
+            -0.006006844341754913,
+            -0.00546195637434721,
+            0.01169983297586441,
+            0.0024825092405080795,
+            -0.0325886569917202,
+            0.005060918629169464,
+            0.00261110276915133,
+            0.005928380414843559,
+            0.018168741837143898,
+            0.01632048189640045,
+            0.01765436865389347,
+            -0.011525468900799751,
+            -0.01975545659661293,
+            0.006285826675593853,
+            0.0045029534958302975,
+            -0.01073211245238781,
+            0.0023473771288990974,
+            0.0024912275839596987,
+            0.01588457264006138,
+            -0.011185458861291409,
+            -0.005823761690407991,
+            -0.004141148179769516,
+            0.011664960533380508,
+            0.0031516316812485456,
+            -0.014541968703269958,
+            0.003940629307180643,
+            -0.021150369197130203,
+            0.020644713193178177,
+            0.011577778495848179,
+            -0.0019932000432163477,
+            -0.02088882215321064,
+            -0.02873520739376545,
+            0.01332141924649477,
+            -0.020836513489484787,
+            0.0031952227000147104,
+            0.015771236270666122,
+            -0.011673678644001484,
+            0.013260391540825367,
+            -0.018255924805998802,
+            -0.018866198137402534,
+            -0.0114818774163723,
+            -0.00473834527656436,
+            -0.005226564593613148,
+            -0.007576121482998133,
+            0.007502016611397266,
+            -0.01823848858475685,
+            -0.0306706503033638,
+            -0.033634841442108154,
+            0.019023125991225243,
+            0.01670408435165882,
+            -0.009467972442507744,
+            -0.015823544934391975,
+            -0.004642445128411055,
+            0.02319042943418026,
+            -0.0034415118861943483,
+            0.0007110786973498762,
+            -0.0026830281130969524,
+            0.013391165062785149,
+            -0.007610994391143322,
+            0.00016441992192994803,
+            -0.02284170128405094,
+            0.00044217653339728713,
+            0.0011725988006219268,
+            0.013391165062785149,
+            -0.027357731014490128,
+            -0.003380484413355589,
+            0.0003002332232426852,
+            -0.018098996952176094,
+            -0.004476799163967371,
+            0.04254484549164772,
+            -0.013417319394648075,
+            0.008979752659797668,
+            0.021551406010985374,
+            0.010104401037096977,
+            -0.00977310910820961,
+            0.010095682926476002,
+            0.007850744761526585,
+            -0.0028857262805104256,
+            0.026538221165537834,
+            0.022911446169018745,
+            0.01914518140256405,
+            0.01670408435165882,
+            -0.022161681205034256,
+            0.012528062798082829,
+            0.021969880908727646,
+            0.03752316161990166,
+            -0.017331793904304504,
+            -0.021952442824840546,
+            0.019860073924064636,
+            0.04676445946097374,
+            0.0021032672375440598,
+            -0.0006200824282132089,
+            -0.028648026287555695,
+            -0.017201021313667297,
+            0.024689961224794388,
+            0.020557530224323273,
+            0.016224581748247147,
+            -0.010121837258338928,
+            -0.031507596373558044,
+            0.014280421659350395,
+            -0.00309278373606503,
+            0.006865587551146746,
+            -0.01996469311416149,
+            0.01688716560602188,
+            -0.015004033222794533,
+            0.04257972165942192,
+            -0.012658835388720036,
+            0.004829886369407177,
+            0.017148712649941444,
+            0.011150586418807507,
+            -0.022318609058856964,
+            0.0009263094398193061,
+            -0.01666049286723137,
+            0.027933133766055107,
+            -0.018883634358644485,
+            0.003618055721744895,
+            0.01914518140256405,
+            -0.016608182340860367,
+            -0.014533249661326408,
+            0.004191277548670769,
+            0.019459037110209465,
+            0.006107103545218706,
+            -0.025247925892472267,
+            0.018098996952176094,
+            0.0019190951716154814,
+            0.009171552956104279,
+            -0.009930036962032318,
+            0.023870449513196945,
+            -0.027636714279651642,
+            -0.015387633815407753,
+            0.025003815069794655,
+            0.0041607641614973545,
+            0.013565529137849808,
+            -0.026311546564102173,
+            0.009380790404975414,
+            -0.029049063101410866,
+            0.02336479350924492,
+            0.0032584297005087137,
+            0.014280421659350395,
+            0.0015147883677855134,
+            0.002866110298782587,
+            -0.0015115190763026476,
+            0.014446067623794079,
+            -0.000550881668459624,
+            -0.022527845576405525,
+            -0.01569277234375477,
+            -0.03696519508957863,
+            -0.022737082093954086,
+            0.023818138986825943,
+            -0.002715721260756254,
+            0.003528693923726678,
+            0.02430635876953602,
+            -0.027462350204586983,
+            -0.01435888558626175,
+            0.009572590701282024,
+            0.02549203485250473,
+            -0.043625906109809875,
+            4.093470488442108e-05,
+            0.0013829254312440753,
+            -0.019232362508773804,
+            0.010627493262290955,
+            -0.005440160632133484,
+            0.015500971116125584,
+            0.0018776836805045605,
+            -0.037662651389837265,
+            0.011586496606469154,
+            0.03762777894735336,
+            -0.017924632877111435,
+            0.0028225192800164223,
+            0.017706677317619324,
+            0.012493189424276352,
+            -0.0012390750925987959,
+            -0.0017033196054399014,
+            0.01877901703119278,
+            -0.025945382192730904,
+            0.026102310046553612,
+            0.017026657238602638,
+            -0.014881978742778301,
+            0.007842026650905609,
+            -0.008753078989684582,
+            0.020731894299387932,
+            -0.0013208083109930158,
+            -0.013155773282051086,
+            0.017375385388731956,
+            0.002195898210629821,
+            0.01349578332155943,
+            -0.001912556472234428,
+            0.004304614383727312,
+            0.03107168711721897,
+            -0.01609380915760994,
+            0.02280682697892189,
+            0.01994725689291954,
+            3.6405323044164106e-05,
+            0.015405070967972279,
+            -0.019877510145306587,
+            -0.03312918543815613,
+            0.02221398986876011,
+            -0.013591683469712734,
+            -0.007310215849429369,
+            0.023347357288002968,
+            -0.010453129187226295,
+            -0.006250953767448664,
+            -0.004282819107174873,
+            0.004077941179275513,
+            0.012406007386744022,
+            -0.0230160653591156,
+            -0.015631744638085365,
+            0.022318609058856964,
+            0.017131276428699493,
+            0.022179117426276207,
+            0.023800702765583992,
+            -0.015378915704786777,
+            0.03182145208120346,
+            0.009223862551152706,
+            0.0065735275857150555,
+            0.0076371487230062485,
+            0.008395632728934288,
+            -0.005871712230145931,
+            0.01933698169887066,
+            0.014114775694906712,
+            0.007336370646953583,
+            -0.003565746359527111,
+            -0.02029598504304886,
+            -0.011447004973888397,
+            -0.027532095089554787,
+            -0.00045825072447769344,
+            -0.013914257287979126,
+            0.00704866973683238,
+            0.07797563821077347,
+            -0.006682505365461111,
+            0.0018144766800105572,
+            -0.007868180982768536,
+            0.000739412906114012,
+            0.0016771649243310094,
+            -0.005209128372371197,
+            0.016538437455892563,
+            -0.020662149414420128,
+            -0.020749330520629883,
+            -0.006163771729916334,
+            -0.010967503301799297,
+            0.013460910879075527,
+            0.01741897687315941,
+            0.0030034221708774567,
+            -0.02277195453643799,
+            0.011141867376863956,
+            -0.016355354338884354,
+            -0.011106994934380054,
+            0.010348510928452015,
+            -0.007231752388179302,
+            -0.011656241491436958,
+            0.03295481950044632,
+            -0.009206426329910755,
+            -0.03147272393107414,
+            0.015640461817383766,
+            -0.0024541751481592655,
+            -0.005976330488920212,
+            0.00470347236841917,
+            0.00853076484054327,
+            -0.004986814223229885,
+            0.0005544234300032258,
+            0.013504501432180405,
+            -0.002068394562229514,
+            0.005414006300270557,
+            0.005318105686455965,
+            -0.013565529137849808,
+            0.017296921461820602,
+            0.022527845576405525,
+            -0.033477913588285446,
+            -0.004755781497806311,
+            -0.013940411619842052,
+            0.010479284450411797,
+            0.018709270283579826,
+            0.015535843558609486,
+            -0.022580154240131378,
+            0.0025871277321130037,
+            -0.009154116734862328,
+            -0.012702426873147488,
+            -0.02741004154086113,
+            -0.018255924805998802,
+            0.00723611144348979,
+            -0.00851768720895052,
+            0.018656961619853973,
+            0.007907412946224213,
+            -0.005745298229157925,
+            0.005012968555092812,
+            -0.005178614519536495,
+            0.015518407337367535,
+            -0.0326235294342041,
+            0.010461847297847271,
+            -0.009337198920547962,
+            0.009581308811903,
+            0.01055774837732315,
+            -0.022388353943824768,
+            -0.014759923331439495,
+            -0.023417102172970772,
+            0.016555873677134514,
+            -0.010086964815855026,
+            -0.0020934592466801405,
+            0.02008674666285515,
+            0.0009088730439543724,
+            0.015413789078593254,
+            -0.001491903094574809,
+            0.023242738097906113,
+            0.014140930958092213,
+            0.016590746119618416,
+            -0.01215317938476801,
+            -0.001640112604945898,
+            -0.011124431155622005,
+            0.012719863094389439,
+            -0.02303350158035755,
+            -0.006813278421759605,
+            -0.026015127077698708,
+            0.03598875552415848,
+            -0.007602275814861059,
+            -0.01588457264006138,
+            0.02857828140258789,
+            -0.012414725497364998,
+            -0.0037052377592772245,
+            0.0058847893960773945,
+            -0.015431225299835205,
+            0.023451974615454674,
+            0.0004955755430273712,
+            0.011499314568936825,
+            -0.02549203485250473,
+            0.00863102450966835,
+            -0.007087901700288057,
+            0.006076589692384005,
+            -0.02128986082971096,
+            -0.002423661295324564,
+            0.012135743163526058,
+            -0.010426974855363369,
+            0.00411717314273119,
+            0.021185241639614105,
+            0.005331183318048716,
+            -0.01744513027369976,
+            0.03951091319322586,
+            -0.004025631584227085,
+            0.010653648525476456,
+            0.005475033540278673,
+            -0.0009028792264871299,
+            0.016799984499812126,
+            0.01742769405245781,
+            -0.004925786517560482,
+            -0.016547156497836113,
+            0.0032235567923635244,
+            -0.015021469444036484,
+            0.01561430748552084,
+            0.016233300790190697,
+            -0.02125498652458191,
+            -0.032658401876688004,
+            0.01779385842382908,
+            0.002061855746433139,
+            0.028072625398635864,
+            0.006983283441513777,
+            -0.025997690856456757,
+            -0.005296310409903526,
+            -0.03815086930990219,
+            0.012885509058833122,
+            -0.01801181398332119,
+            -0.02437610551714897,
+            0.002785466844215989,
+            -0.00986029114574194,
+            -0.01856978051364422,
+            0.02512587048113346,
+            0.013652711175382137,
+            -0.02552690915763378,
+            0.010889039374887943,
+            0.0013011923292651772,
+            -0.017549749463796616,
+            0.010052092373371124,
+            0.004350385162979364,
+            -0.012493189424276352,
+            0.007410475518554449,
+            0.005130664445459843,
+            -0.006978924386203289,
+            -0.02890957146883011,
+            -0.01670408435165882,
+            0.01972058229148388,
+            0.010993658564984798,
+            -0.0058455574326217175,
+            0.011290077120065689,
+            -0.00997362844645977,
+            0.0033456117380410433,
+            0.0035221553407609463,
+            0.00901462510228157,
+            -0.013051155023276806,
+            -0.007371243555098772,
+            -0.007162006571888924,
+            0.01973801851272583,
+            -0.016024062409996986,
+            0.007990236394107342,
+            -0.013443474657833576,
+            -0.007981518283486366,
+            0.011429568752646446,
+            -0.0013774766121059656,
+            0.014986597001552582,
+            -0.018744144588708878,
+            0.0073407297022640705,
+            0.01189163327217102,
+            -0.008883852511644363,
+            -0.010671084746718407,
+            -0.02437610551714897,
+            0.019110308960080147,
+            0.01438504084944725,
+            0.004694754257798195,
+            -0.03679082915186882,
+            0.003951527178287506,
+            -0.010453129187226295,
+            -0.0009775288635864854,
+            -0.008796670474112034,
+            0.019249800592660904,
+            0.018831325694918633,
+            0.007985876873135567,
+            0.013277828693389893,
+            -0.0023081451654434204,
+            0.013548092916607857,
+            -0.012414725497364998,
+            0.017593340948224068,
+            -0.004036529455333948,
+            -0.017575904726982117,
+            -0.005714784376323223,
+            0.019406726583838463,
+            -0.001491903094574809,
+            0.007903054356575012,
+            0.004372180439531803,
+            0.010226456448435783,
+            -0.012275234796106815,
+            -0.007493298500776291,
+            0.014045030809938908,
+            0.008378196507692337,
+            -0.0004288267809897661,
+            0.009250016883015633,
+            0.06758353859186172,
+            -0.00870512891560793,
+            0.023853013291954994,
+            0.006311981473118067,
+            -0.009345917031168938,
+            -0.008112290874123573,
+            0.0019932000432163477,
+            -0.02951984666287899,
+            0.023329921066761017,
+            0.011970097199082375,
+            -0.009180271066725254,
+            0.00036126066697761416,
+            -0.006726096384227276,
+            0.018325669690966606,
+            -0.009694645181298256,
+            0.016948193311691284,
+            -0.010052092373371124,
+            0.02435866929590702,
+            0.019162617623806,
+            0.005640679504722357,
+            0.005536061245948076,
+            0.0028203397523611784,
+            0.019459037110209465,
+            0.012868072837591171,
+            -0.002165384590625763,
+            -0.003123297356069088,
+            -0.009877728298306465,
+            0.019389290362596512,
+            -0.001910376944579184,
+            0.030792705714702606,
+            -0.0022144243121147156,
+            -0.005182973574846983,
+            -0.009842854924499989,
+            0.0041607641614973545,
+            0.011569059453904629,
+            0.009520281106233597,
+            0.012876790948212147,
+            -0.0013567708665505052,
+            -0.010679802857339382,
+            0.011909069493412971,
+            0.00863102450966835,
+            0.005326824262738228,
+            -0.0032518908847123384,
+            0.0045247492380440235,
+            -0.008404350839555264,
+            0.014001439325511456,
+            -0.0024781501851975918,
+            0.0015354941133409739,
+            -0.008774874731898308,
+            0.004426669329404831,
+            -0.01312961895018816,
+            0.0005348074482753873,
+            0.009607463143765926,
+            0.012868072837591171,
+            0.0015213270671665668,
+            -0.002273272257298231,
+            0.015405070967972279,
+            0.0025980256032198668,
+            0.0073014977388083935,
+            0.0009486498311161995,
+            -0.009537717327475548,
+            0.019057998433709145,
+            -0.01033979281783104,
+            -0.0019212746992707253,
+            0.010705957189202309,
+            0.006067871581763029,
+            0.008260500617325306,
+            0.01496916078031063,
+            0.009354635141789913,
+            0.003968963399529457,
+            0.0013545913388952613,
+            -0.00565375667065382,
+            -0.003535232739523053,
+            0.0036267738323658705,
+            0.0016771649243310094,
+            0.01646869257092476,
+            -0.019459037110209465,
+            -0.007039951626211405,
+            0.010679802857339382,
+            0.011438286863267422,
+            0.0017839629435911775,
+            0.015684053301811218,
+            0.014428631402552128,
+            0.01994725689291954,
+            -0.022562718018889427,
+            0.003831651760265231,
+            -0.002920599188655615,
+            0.019057998433709145,
+            0.008770515210926533,
+            -0.024114558473229408,
+            0.0003645300166681409,
+            -0.01063621137291193,
+            0.012676272541284561,
+            -0.0067086596973240376,
+            0.0023473771288990974,
+            0.01665177382528782,
+            -0.010121837258338928,
+            -0.009930036962032318,
+            -0.019424164667725563,
+            -0.016983065754175186,
+            0.010976221412420273,
+            0.014437349513173103,
+            0.0014188881032168865,
+            0.019232362508773804,
+            -0.0012412546202540398,
+            -0.008474096655845642,
+            0.030827578157186508,
+            0.016765110194683075,
+            -0.0004157494695391506,
+            -0.008962316438555717,
+            -0.0006544103380292654,
+            -0.015082497149705887,
+            0.01074954867362976,
+            0.004426669329404831,
+            0.005108869168907404,
+            0.003387023229151964,
+            9.174550359603018e-05,
+            -0.016712801530957222,
+            -0.014620431698858738,
+            -0.004668599460273981,
+            0.011045967228710651,
+            -0.003751008305698633,
+            0.009197707287967205,
+            -0.01609380915760994,
+            0.003121117828413844,
+            -0.008173318579792976,
+            -0.010714675299823284,
+            -0.008356400765478611,
+            -0.039406292140483856,
+            -0.02067958563566208,
+            0.0003173972072545439,
+            -0.00473834527656436,
+            -0.002358274767175317,
+            0.009223862551152706,
+            -0.003672544378787279,
+            0.013670147396624088,
+            -0.0015551100950688124,
+            0.0047906544059515,
+            0.008657178841531277,
+            0.01561430748552084,
+            -0.013504501432180405,
+            -0.0006631285650655627,
+            -0.014071185141801834,
+            -0.026782330125570297,
+            0.01707896590232849,
+            0.003199581755325198,
+            0.014446067623794079,
+            -0.015187115408480167,
+            0.009528999216854572,
+            -0.012589089572429657,
+            0.014463504776358604,
+            -0.0421612448990345,
+            -0.009520281106233597,
+            0.007031233515590429,
+            -0.0075979167595505714,
+            -0.005496829282492399,
+            -0.025369981303811073,
+            -0.0012205488746985793,
+            -0.011054685339331627,
+            -0.01666049286723137,
+            -0.019459037110209465,
+            0.00574093870818615,
+            -0.009311044588685036,
+            0.008591792546212673,
+            -0.006120180711150169,
+            0.02761927805840969,
+            0.00532246520742774,
+            -0.001346962875686586,
+            -0.006128899287432432,
+            0.009206426329910755,
+            -0.0007503106608055532,
+            0.012885509058833122,
+            0.008648460730910301,
+            0.005823761690407991,
+            0.023434538394212723,
+            0.0009339378448203206,
+            0.004690395202487707,
+            -0.006141976453363895,
+            0.014541968703269958,
+            0.024689961224794388,
+            0.011342386715114117,
+            0.011455723084509373,
+            0.005967612378299236,
+            0.0005950175691395998,
+            0.011429568752646446,
+            0.017392821609973907,
+            -0.003990759141743183,
+            -0.009171552956104279,
+            0.004869118332862854,
+            -0.01761949434876442,
+            -0.0033347138669341803,
+            0.012885509058833122,
+            -0.006128899287432432,
+            -0.02706131339073181,
+            0.01285063661634922,
+            0.004317691549658775,
+            -0.006673786789178848,
+            -0.016948193311691284,
+            -0.00661711860448122,
+            0.004646804183721542,
+            0.004729627165943384,
+            0.024236613884568214,
+            -0.022144244983792305,
+            -0.008644101209938526,
+            0.011917788535356522,
+            -0.014446067623794079,
+            -0.01063621137291193,
+            0.009790545329451561,
+            0.029484974220395088,
+            0.003496000776067376,
+            -0.05307644233107567,
+            -0.012467035092413425,
+            0.014236831106245518,
+            -0.01503018755465746,
+            -0.025980254635214806,
+            -0.0014428631402552128,
+            0.0038578063249588013,
+            -0.014010157436132431,
+            0.006625836715102196,
+            0.0011606112821027637,
+            -0.011708551086485386,
+            0.017139993607997894,
+            0.028874699026346207,
+            0.01312961895018816,
+            0.0114818774163723,
+            -0.025213053449988365,
+            -0.0008952508214861155,
+            -0.01914518140256405,
+            0.011752142570912838,
+            -0.030566031113266945,
+            -0.013984003104269505,
+            -0.0023342997301369905,
+            0.004141148179769516,
+            0.027444913983345032,
+            0.014376322738826275,
+            0.025962818413972855,
+            0.006778405513614416,
+            0.0037989583797752857,
+            -0.02125498652458191,
+            -0.018447725102305412,
+            0.0033456117380410433,
+            -0.0008304091752506793,
+            -0.002329940674826503,
+            -0.008216909132897854,
+            0.012275234796106815,
+            0.012039843015372753,
+            0.026172054931521416,
+            0.005087073426693678,
+            0.005553497467190027,
+            -2.538836997700855e-05,
+            -0.00494322320446372,
+            0.007593557704240084,
+            -0.019232362508773804,
+            -0.0029925242997705936,
+            -0.004559622146189213,
+            0.015405070967972279,
+            0.0045029534958302975,
+            0.0022275017108768225,
+            0.017357949167490005,
+            -0.0030099607538431883,
+            0.003729212796315551,
+            -0.019982129335403442,
+            -0.014672741293907166,
+            -0.0032518908847123384,
+            0.03525642678141594,
+            0.003971143160015345,
+            -0.02761927805840969,
+            -0.016224581748247147,
+            -0.0153353251516819,
+            0.030792705714702606,
+            -0.009581308811903,
+            0.004315512254834175,
+            0.02683463878929615,
+            0.01593688130378723,
+            0.012388571165502071,
+            0.006399163510650396,
+            -0.01822105050086975,
+            0.005021686665713787,
+            0.02931061014533043,
+            -0.0030317562632262707,
+            -0.010086964815855026,
+            -0.0066912234760820866,
+            0.007144570350646973,
+            -0.007798435632139444,
+            0.007632789667695761,
+            -0.0035068984143435955,
+            -0.013417319394648075,
+            0.029554719105362892,
+            -0.00968592707067728,
+            0.022370917722582817,
+            -0.010130555368959904,
+            0.00268738716840744,
+            -0.038848329335451126,
+            -0.002004097681492567,
+            0.0011453543556854129,
+            0.0019692247733473778,
+            0.004555262625217438,
+            -0.01881388947367668,
+            -0.008753078989684582,
+            0.011839324608445168,
+            -0.01276345457881689,
+            -0.007885618135333061,
+            -0.006137617398053408,
+            0.002983806189149618,
+            0.01977289281785488,
+            -0.00947669055312872,
+            0.0009857022669166327,
+            -0.0210980586707592,
+            0.01256293524056673,
+            -0.00282687833532691,
+            0.0017567186150699854,
+            0.010453129187226295,
+            0.007009437773376703,
+            0.03502975404262543,
+            0.01094134896993637,
+            0.005531701724976301,
+            -0.0032780456822365522,
+            -0.014053748920559883,
+            -0.03218761831521988,
+            0.013295264914631844,
+            -0.010453129187226295,
+            0.023800702765583992,
+            0.0036703648511320353,
+            -0.008103572763502598,
+            -0.0051960512064397335,
+            0.0037139558698982,
+            0.009590026922523975,
+            -0.0041106343269348145,
+            0.013478347100317478,
+            -0.010243892669677734,
+            0.021167805418372154,
+            0.0031298361718654633,
+            -0.005121946334838867,
+            0.012571653351187706,
+            -0.017689241096377373,
+            0.01055774837732315,
+            -0.01744513027369976,
+            0.0028225192800164223,
+            0.01360040158033371,
+            -0.010296201333403587,
+            0.0056276023387908936,
+            -0.052588220685720444,
+            -0.001197663601487875,
+            -0.016521001234650612,
+            -0.00675660977140069,
+            -0.008456660434603691,
+            -0.021394478157162666,
+            -0.03319893032312393,
+            -0.0036660057958215475,
+            0.015370197594165802,
+            0.006551732309162617,
+            0.023260174319148064,
+            0.0009895163821056485,
+            0.012257798574864864,
+            0.012807045131921768,
+            -0.0047906544059515,
+            0.012519344687461853,
+            0.033861514180898666,
+            -0.027741333469748497,
+            -0.019685709848999977,
+            0.005257078446447849,
+            -0.00028552126605063677,
+            -0.015788672491908073,
+            -0.013103464618325233,
+            -0.0017665266059339046,
+            0.0024955866392701864,
+            0.030443977564573288,
+            -0.020191365852952003,
+            -0.005579652264714241,
+            -0.007053028792142868,
+            0.004433208145201206,
+            -0.01918005384504795,
+            -0.0031450928654521704,
+            -0.011708551086485386,
+            -0.012780890800058842,
+            0.016041500493884087,
+            -0.011089558713138103,
+            0.011159304529428482,
+            -0.019511345773935318,
+            0.008626664988696575,
+            -0.015378915704786777,
+            -0.010627493262290955,
+            -0.0031690681353211403,
+            -0.010461847297847271,
+            0.030164994299411774,
+            -0.0041890982538461685,
+            0.0048386044800281525,
+            0.01670408435165882,
+            -0.00853076484054327,
+            0.011603932827711105,
+            -0.0006124540232121944,
+            0.007898694835603237,
+            -0.002704823622480035,
+            -0.0086135882884264,
+            0.0015278656501322985,
+            0.009537717327475548,
+            -0.010950067080557346,
+            0.03846472501754761,
+            -0.013731175102293491,
+            -0.008879492990672588,
+            -0.003859985852614045,
+            -0.004511671606451273,
+            0.002425840822979808,
+            0.0021762822289019823,
+            0.010984940454363823,
+            -0.0012870251666754484,
+            0.006835073698312044,
+            -0.024219177663326263,
+            -0.011717269197106361,
+            0.0013164491392672062,
+            0.006468909326940775,
+            0.003321636701002717,
+            0.008016390725970268,
+            -0.025579217821359634,
+            -0.0020782025530934334,
+            -0.012859354726970196,
+            0.005475033540278673,
+            -0.011499314568936825,
+            0.007332011591643095,
+            -0.01215317938476801,
+            -0.0013295264216139913,
+            0.009337198920547962,
+            0.012946536764502525,
+            0.013861947692930698,
+            -0.006608400493860245,
+            0.026573093608021736,
+            0.014899414964020252,
+            0.008709488436579704,
+            0.005405288189649582,
+            -0.033216364681720734,
+            -0.007802794687449932,
+            0.010365947149693966,
+            0.007519452832639217,
+            0.009232580661773682,
+            -0.017689241096377373,
+            -0.004176021087914705,
+            0.0029402151703834534,
+            -0.007039951626211405,
+            4.651980270864442e-05,
+            0.011080840602517128,
+            -0.010627493262290955,
+            0.005505547393113375,
+            -0.00031521765049546957,
+            0.0027549532242119312,
+            -0.006460190750658512,
+            0.012188052758574486,
+            -0.013286546804010868,
+            -0.008168959058821201,
+            0.0211154967546463,
+            0.02760184183716774,
+            -0.005531701724976301,
+            0.023696085438132286,
+            -0.021917570382356644,
+            0.008487174287438393,
+            -0.005043482407927513,
+            -0.014149649068713188,
+            -0.015300451777875423,
+            0.006883023772388697,
+            0.014777359552681446,
+            0.004760140553116798,
+            -0.013155773282051086,
+            -0.003027397207915783,
+            0.016983065754175186,
+            -0.0010952246375381947,
+            0.006216081324964762,
+            0.01593688130378723,
+            -0.00026971951592713594,
+            0.00017667989595793188,
+            -0.006002485286444426,
+            -0.01876158080995083,
+            -0.006290185730904341,
+            0.017924632877111435,
+            -0.0073799616657197475,
+            0.014262985438108444,
+            0.02320786565542221,
+            0.0005429807933978736,
+            0.011351104825735092,
+            -0.0038469084538519382,
+            0.01351321954280138,
+            0.01505634281784296,
+            -0.005235282704234123,
+            -0.0030600903555750847,
+            -0.0036965194158256054,
+            0.00011858122888952494,
+            0.012022406794130802,
+            -0.01741897687315941,
+            -0.011333668604493141,
+            0.0114818774163723,
+            -0.002121793571859598,
+            -0.008404350839555264,
+            -0.023486848920583725,
+            0.0016313943779096007,
+            0.0077156126499176025,
+            0.011447004973888397,
+            -0.016425101086497307,
+            -0.0005239096935838461,
+            0.000875089957844466,
+            0.010801857337355614,
+            -0.007497657556086779,
+            0.01159521471709013,
+            -0.007907412946224213,
+            -0.00863102450966835,
+            0.006381727289408445,
+            0.0015507509233430028,
+            0.007414834573864937,
+            -0.009999782778322697,
+            0.0015987011138349771,
+            -0.008025108836591244,
+            0.026590529829263687,
+            0.008317168802022934,
+            -0.0013011923292651772,
+            -0.005640679504722357,
+            0.004995532333850861,
+            0.008421787060797215,
+            -0.0100608104839921,
+            0.012815763242542744,
+            0.006107103545218706,
+            -0.002933676354587078,
+            0.010592620819807053,
+            0.015283015556633472,
+            0.0025631526950746775,
+            0.009328480809926987,
+            0.003443691413849592,
+            0.0059022256173193455,
+            -0.007022515404969454,
+            0.0017861425876617432,
+            -0.01550968922674656,
+            -0.020801641047000885,
+            -0.0035831828135997057,
+            0.016991784796118736,
+            -0.010583902709186077,
+            -0.001182406791485846,
+            0.017924632877111435,
+            0.01371373888105154,
+            -0.005034764297306538,
+            0.00551426550373435,
+            0.002217693720012903,
+            -0.0015529305674135685,
+            0.003323816228657961,
+            -0.014280421659350395,
+            -0.012493189424276352,
+            0.008726924657821655,
+            -0.02087138593196869,
+            0.01952878199517727,
+            -0.02165602520108223,
+            -0.035361044108867645,
+            -0.008831542916595936,
+            -0.009197707287967205,
+            -0.01707896590232849,
+            -0.013574247248470783,
+            0.023661212995648384,
+            0.016207145527005196,
+            -0.009354635141789913,
+            -0.034384604543447495,
+            0.03155990689992905,
+            0.00918898917734623,
+            0.027933133766055107,
+            -0.015544561669230461,
+            0.004343846347182989,
+            -0.0015213270671665668,
+            0.014707613736391068,
+            -0.006163771729916334,
+            -0.006686864420771599,
+            0.0091018071398139,
+            0.014620431698858738,
+            0.010897758416831493,
+            0.003245352301746607,
+            -0.003160349791869521,
+            0.007279702462255955,
+            0.002530459314584732,
+            0.0022285915911197662,
+            -0.008583074435591698,
+            -0.02282426506280899,
+            -0.009781827218830585,
+            0.017942069098353386,
+            0.022266298532485962,
+            0.012746017426252365,
+            -0.017131276428699493,
+            0.0020520477555692196,
+            -0.00484296353533864,
+            -0.014053748920559883,
+            -0.009991064667701721,
+            0.009642336517572403,
+            -0.033617403358221054,
+            0.02317299321293831,
+            0.004760140553116798,
+            -0.00627274950966239,
+            -0.0015006213216111064,
+            0.003836010815575719,
+            0.005679911468178034,
+            -0.004313332494348288,
+            0.0002961465797852725,
+            -0.002532638842239976,
+            0.010897758416831493,
+            0.016599465161561966,
+            0.005100150592625141,
+            -0.010400820523500443,
+            0.012388571165502071,
+            -0.0031407338101416826,
+            -0.002761491807177663,
+            -0.013914257287979126,
+            -0.006011203397065401,
+            -0.009502844884991646,
+            0.010261328890919685,
+            0.01483838725835085,
+            0.0054924702271819115,
+            0.006791482679545879,
+            0.004335128236562014,
+            -0.00013084120291750878,
+            -0.0002569146454334259,
+            0.008469737134873867,
+            -0.0005070181796327233,
+            -0.0045639812014997005,
+            0.0010821473551914096,
+            -0.0014679280575364828,
+            0.02320786565542221,
+            -0.00532246520742774,
+            -0.010950067080557346,
+            0.007196879480034113,
+            0.02033085748553276,
+            -0.022370917722582817,
+            0.022091934457421303,
+            -0.022179117426276207,
+            0.023853013291954994,
+            0.03482051566243172,
+            0.013975284993648529,
+            -0.011717269197106361,
+            -0.007096619810909033,
+            0.012815763242542744,
+            -0.012510626576840878,
+            -0.0009600924677215517,
+            0.0018079380970448256,
+            -0.022580154240131378,
+            0.015178397297859192,
+            -0.023347357288002968,
+            0.014280421659350395,
+            0.005121946334838867,
+            -0.00852640625089407,
+            -0.0073232934810221195,
+            -0.0009764390997588634,
+            0.018133869394659996,
+            -0.003990759141743183,
+            -0.0006255313055589795,
+            -0.03028704971075058,
+            -0.006930973846465349,
+            0.02163858897984028,
+            -0.0045247492380440235,
+            -0.013949130661785603,
+            0.007253547664731741,
+            -8.973622243502177e-06,
+            -0.016154836863279343,
+            -0.005876071285456419,
+            -0.03166452422738075,
+            -0.008142804726958275,
+            -0.02264990098774433,
+            -0.0075979167595505714,
+            -0.011577778495848179,
+            -0.012292671017348766,
+            -0.0038512677419930696,
+            -0.016276890411973,
+            0.001949608908034861,
+            -0.001063076313585043,
+            0.0033020207192748785,
+            0.0027091826777905226,
+            -0.009563872590661049,
+            -0.003968963399529457,
+            0.021324733272194862,
+            0.010705957189202309,
+            -0.016041500493884087,
+            -0.026206929236650467,
+            0.011638805270195007,
+            -0.012205488979816437,
+            -0.007593557704240084,
+            0.001547481631860137,
+            -0.005915303248912096,
+            -0.019267236813902855,
+            -0.01860465295612812,
+            0.005967612378299236,
+            0.0003718859807122499,
+            0.0051175872795283794,
+            0.013582965359091759,
+            0.014341449365019798,
+            -0.0019180054077878594,
+            -0.006734814494848251,
+            -1.330718441749923e-05,
+            -0.002195898210629821,
+            0.014027593657374382,
+            0.004348205402493477,
+            0.024445850402116776,
+            -0.008744360879063606,
+            0.0028944446239620447,
+            -0.017924632877111435,
+            0.00449859444051981,
+            -0.007393038831651211,
+            -0.005679911468178034,
+            -0.004064863547682762,
+            -0.011089558713138103,
+            0.005993766710162163,
+            -0.0011126610916107893,
+            -0.00247161160223186,
+            -0.007911772467195988,
+            -0.0020912797190248966,
+            -0.01707024872303009,
+            -0.0029794471338391304,
+            -0.020557530224323273,
+            -0.0013905538944527507,
+            -0.014916851185262203,
+            0.017000501975417137,
+            0.026172054931521416,
+            -0.00513502350077033,
+            -0.0006091846735216677,
+            -0.011229049414396286,
+            -4.08665910072159e-05,
+            -0.011909069493412971,
+            -0.00637736776843667,
+            -0.016416382044553757,
+            -0.0325189083814621,
+            -0.016904601827263832,
+            0.009049498476088047,
+            0.024288922548294067,
+            0.0019398009171709418,
+            -0.024986378848552704,
+            0.0029511128086596727,
+            -0.026607966050505638,
+            -0.014228112995624542,
+            -0.01496916078031063,
+            -0.006529936566948891,
+            -0.009136680513620377,
+            0.017392821609973907,
+            -0.0026350778061896563,
+            0.01571020856499672,
+            0.01747128553688526,
+            0.010906476527452469,
+            0.007497657556086779,
+            0.010880321264266968,
+            -0.009528999216854572,
+            -0.00016932391736190766,
+            0.009869009256362915,
+            -0.006311981473118067,
+            0.011839324608445168,
+            -0.00815152283757925,
+            -0.0011933045461773872,
+            0.01879645325243473,
+            0.008016390725970268,
+            -0.006255313288420439,
+            -0.0026699507143348455,
+            -0.02742747776210308,
+            0.017506157979369164,
+            -0.018552342429757118,
+            -0.00258494820445776,
+            -0.002944574225693941,
+            -0.016738956794142723,
+            0.02319042943418026,
+            0.017323076725006104,
+            -0.005021686665713787,
+            -0.0027549532242119312,
+            0.017889760434627533,
+            -0.006451472640037537,
+            0.00398204056546092,
+            0.0025762300938367844,
+            -0.0033281752839684486,
+            0.009790545329451561,
+            0.0027636713348329067,
+            -0.004577058367431164,
+            -0.0025587936397641897,
+            0.012196770869195461,
+            0.003901397343724966,
+            0.013530656695365906,
+            -0.027131058275699615,
+            -0.010566466487944126,
+            -0.0010303830495104194,
+            0.01935441792011261,
+            -0.005405288189649582,
+            -0.0007906323298811913,
+            -0.013312701135873795,
+            -0.0008854428306221962,
+            -0.005719143431633711,
+            0.023818138986825943,
+            -0.01992982067167759,
+            0.006120180711150169,
+            -0.008395632728934288,
+            0.013861947692930698,
+            -0.0032191977370530367,
+            0.010801857337355614,
+            -0.0007955363253131509,
+            -1.3375295566220302e-05,
+            0.009964910335838795,
+            0.012981409206986427,
+            -0.0022024367935955524,
+            0.024027377367019653,
+            -0.0004737800336442888,
+            -0.006403522565960884,
+            0.0027745692059397697,
+            -0.01895338110625744,
+            0.0229637548327446,
+            0.003402280155569315,
+            0.004315512254834175,
+            -0.0058629936538636684,
+            -0.008199472911655903,
+            0.039789892733097076,
+            0.021516533568501472,
+            0.010392102412879467,
+            -0.000592293159570545,
+            -0.01494300551712513,
+            0.028229551389813423,
+            0.007519452832639217,
+            0.00930232647806406,
+            -0.004012554418295622,
+            0.0027113622054457664,
+            -0.015178397297859192,
+            -0.010976221412420273,
+            0.019807765260338783,
+            -0.019999565556645393,
+            -0.008731283247470856,
+            -0.0026328982785344124,
+            -0.019703146070241928,
+            -0.029293173924088478,
+            0.015474816784262657,
+            0.009764390997588634,
+            0.0028443147893995047,
+            0.008495892398059368,
+            0.0037335718516260386,
+            -0.0015028008492663503,
+            -0.011647523380815983,
+            -0.003986400086432695,
+            -0.008574356324970722,
+            -0.01044441107660532,
+            0.006848151329904795,
+            -0.012004970572888851,
+            0.006717377807945013,
+            -0.008256141096353531,
+            -0.00024533577379770577,
+            0.01285063661634922,
+            0.005091432482004166,
+            0.0062771085649728775,
+            0.003443691413849592,
+            0.005078355316072702,
+            0.01591944508254528,
+            -0.013757329434156418,
+            0.016346637159585953,
+            0.017113838344812393,
+            0.0062945447862148285,
+            -0.0034131777938455343,
+            0.0023103246930986643,
+            -0.029188554733991623,
+            -0.00022381270537152886,
+            -0.0016869729151949286,
+            0.012632681056857109,
+            -0.02163858897984028,
+            0.004733986221253872,
+            0.021568842232227325,
+            -0.0016172273317351937,
+            0.009816700592637062,
+            0.014864541590213776,
+            -0.0001531135058030486,
+            0.028037751093506813,
+            0.011342386715114117,
+            0.02416686899960041,
+            -0.009459254331886768,
+            -0.02852597087621689,
+            0.006368649657815695,
+            -0.0038904997054487467,
+            0.014376322738826275,
+            0.0091018071398139,
+            0.0201216209679842,
+            -0.01369630265980959,
+            -0.006695582531392574,
+            0.007789717521518469,
+            0.0018918508430942893,
+            0.014306576922535896,
+            0.003217018209397793,
+            0.019302109256386757,
+            0.009119244292378426,
+            -0.0008625575574114919,
+            -0.007737408392131329,
+            8.588795753894374e-05,
+            0.006250953767448664,
+            -0.003500359831377864,
+            -0.03047885000705719,
+            0.0016390228411182761,
+            -0.004834245424717665,
+            0.004978095646947622,
+            -0.014245549216866493,
+            0.010409538634121418,
+            -0.0010015039006248116,
+            0.022196553647518158,
+            -0.0041498662903904915,
+            0.004064863547682762,
+            0.01708768494427204,
+            0.00938950851559639,
+            0.004991173278540373,
+            0.015300451777875423,
+            0.005658115725964308,
+            0.004511671606451273,
+            -0.008038186468183994,
+            -0.019493909552693367,
+            0.019162617623806,
+            -0.027462350204586983,
+            0.010653648525476456,
+            -0.00873564276844263,
+            0.004317691549658775,
+            0.003925372380763292,
+            -0.006887382827699184,
+            -0.002567511750385165,
+            0.003607157850638032,
+            0.001491903094574809,
+            -0.059144310653209686,
+            0.0037706242874264717,
+            0.0007388679659925401,
+            0.020452912896871567,
+            -0.00881846621632576,
+            0.0007590288296341896,
+            0.0026394370943307877,
+            0.00390357687138021,
+            -0.013347573578357697,
+            -0.0023604542948305607,
+            0.01571892574429512,
+            -0.01858721673488617,
+            -0.0059632533229887486,
+            0.004038709215819836,
+            -0.005557856522500515,
+            0.0035940806847065687,
+            -0.008779234252870083,
+            -0.002622000640258193,
+            0.02838647924363613,
+            -0.001655369414947927,
+            0.03140297904610634,
+            0.002506484277546406,
+            -0.0004756871494464576,
+            -0.01666049286723137,
+            0.007410475518554449,
+            -0.016738956794142723,
+            -0.00425884360447526,
+            0.008927443064749241,
+            -0.018726706504821777,
+            -0.00123580580111593,
+            0.001693511614575982,
+            -0.0013382446486502886,
+            -0.0077722808346152306,
+            -0.001763257198035717,
+            0.0010429153917357326,
+            -0.02048778533935547,
+            0.015997909009456635,
+            -0.023853013291954994,
+            0.01625945419073105,
+            -0.0034371528308838606,
+            -0.002946753753349185,
+            -0.0013033718569204211,
+            -0.0038098562508821487,
+            0.005666834302246571,
+            0.02531767077744007,
+            -0.015448661521077156,
+            -0.00522220553830266,
+            0.008365118876099586,
+            -0.003567925887182355,
+            0.002730978187173605,
+            -0.008761797100305557,
+            0.006656350567936897,
+            -0.0325886569917202,
+            -0.008112290874123573,
+            0.004413592163473368,
+            0.0031908636447042227,
+            -0.001236895564943552,
+            -0.02758440561592579,
+            0.003960245288908482,
+            -0.00854384247213602,
+            -0.024201741442084312,
+            -0.00027380615938454866,
+            -0.020592402666807175,
+            0.006717377807945013,
+            -0.00881846621632576,
+            0.01761949434876442,
+            -0.014568123035132885,
+            0.011098276823759079,
+            -0.014367603696882725,
+            0.016006626188755035,
+            -0.007406116463243961,
+            -0.009398226626217365,
+            -0.0069266147911548615,
+            -0.006338136270642281,
+            0.01044441107660532,
+            -0.014916851185262203,
+            -0.009511562995612621,
+            -0.007641507778316736,
+            0.005161178298294544,
+            0.011045967228710651,
+            0.013574247248470783,
+            0.0011333668371662498,
+            -0.00880538858473301,
+            -0.016547156497836113,
+            -0.005392210558056831,
+            -0.014881978742778301,
+            -0.0009301236714236438,
+            0.0042174323461949825,
+            0.020853949710726738,
+            0.008160240948200226,
+            -0.003753187833353877,
+            0.011586496606469154,
+            0.020801641047000885,
+            0.014132212847471237,
+            -0.014777359552681446,
+            0.0021697436459362507,
+            -0.010095682926476002,
+            0.013417319394648075,
+            0.022144244983792305,
+            0.002098908182233572,
+            0.002796364715322852,
+            0.006085308268666267,
+            0.007951004430651665,
+            -0.004396155476570129,
+            0.008291014470160007,
+            -0.025457162410020828,
+            -0.012667553499341011,
+            -0.006569168530404568,
+            0.005387851502746344,
+            -0.015378915704786777,
+            0.029973194003105164,
+            -0.003609337378293276,
+            0.004472440108656883,
+            -0.002182820811867714,
+            0.0326409637928009,
+            0.007196879480034113,
+            0.00473834527656436,
+            0.002874828642234206,
+            9.467427298659459e-05,
+            0.009136680513620377,
+            -0.01761077716946602,
+            -0.03888320177793503,
+            0.003254070645198226,
+            -0.000592293159570545,
+            0.01766308583319187,
+            -0.01514352485537529,
+            0.006878664717078209,
+            -0.0036899808328598738,
+            -0.01282448135316372,
+            -0.009633618406951427,
+            0.03138554468750954,
+            -0.005466315429657698,
+            0.03387894853949547,
+            -0.007558684796094894,
+            -0.010104401037096977,
+            0.021725770086050034,
+            0.021900134161114693,
+            -0.014210676774382591,
+            0.005531701724976301,
+            0.015762517228722572,
+            0.020278548821806908,
+            0.005209128372371197,
+            -0.00806434080004692,
+            0.02627667412161827,
+            0.012946536764502525,
+            -0.012807045131921768,
+            -0.0026350778061896563,
+            -0.009572590701282024,
+            0.006726096384227276,
+            -0.012301389127969742,
+            0.005387851502746344,
+            0.006434036418795586,
+            0.004986814223229885,
+            0.012963972985744476,
+            -0.01879645325243473,
+            -0.003831651760265231,
+            0.0073014977388083935,
+            0.013966566883027554,
+            0.00651250034570694,
+            0.017558468505740166,
+            -0.023643774911761284,
+            0.0028966241516172886,
+            -0.02493407018482685,
+            -0.008103572763502598,
+            0.0029968833550810814,
+            0.001748000388033688,
+            0.008875134401023388,
+            -0.01592816226184368,
+            -0.029484974220395088,
+            0.011516750790178776,
+            0.01708768494427204,
+            0.02568383514881134,
+            0.0011126610916107893,
+            0.008914366364479065,
+            0.00560144754126668,
+            -0.013739893212914467,
+            -0.00805998221039772,
+            -0.00997362844645977,
+            -0.01073211245238781,
+            -0.011159304529428482,
+            -0.002761491807177663,
+            0.009119244292378426,
+            -0.010592620819807053,
+            0.0006821996648795903,
+            0.016346637159585953,
+            -0.011717269197106361,
+            -0.0051960512064397335,
+            0.015535843558609486,
+            -0.005008609499782324,
+            0.0007977159111760557,
+            0.0016891524428501725,
+            0.013086027465760708,
+            0.0034480507019907236,
+            -0.0063730087131261826,
+            -0.01591944508254528,
+            0.01215317938476801,
+            -0.00417166156694293,
+            0.0016074193408712745,
+            0.00708354264497757,
+            -0.007933568209409714,
+            -0.0047819362953305244,
+            0.013373728841543198,
+            -0.00854384247213602,
+            0.0016259454423561692,
+            -0.007606634870171547,
+            -0.017549749463796616,
+            -0.0076938169077038765,
+            0.006394804455339909,
+            -0.026730021461844444,
+            -0.013478347100317478,
+            0.016233300790190697,
+            -0.003570105414837599,
+            0.001546391868032515,
+            -0.0005138292908668518,
+            -0.011124431155622005,
+            0.003160349791869521,
+            -0.019598528742790222,
+            -0.01064493041485548,
+            -0.004864759277552366,
+            -0.02662540227174759,
+            0.017549749463796616,
+            0.015091215260326862,
+            0.0033172776456922293,
+            -0.00699636060744524,
+            -0.005592729430645704,
+            0.002567511750385165,
+            0.010348510928452015,
+            0.01763693243265152,
+            0.009311044588685036,
+            -0.005592729430645704,
+            -0.019685709848999977,
+            -0.003443691413849592,
+            0.021324733272194862,
+            -6.375188240781426e-05,
+            -0.016590746119618416,
+            0.020609840750694275,
+            0.007109697442501783,
+            0.014306576922535896,
+            0.0055883703753352165,
+            0.0023451976012438536,
+            0.005383492447435856,
+            0.014114775694906712,
+            -0.0001486181718064472,
+            -0.01783744990825653,
+            -0.013984003104269505,
+            0.0004890369018539786,
+            0.03140297904610634,
+            0.00708354264497757,
+            0.017139993607997894,
+            -0.03623286634683609,
+            0.0006642183288931847,
+            -0.005335542373359203,
+            -0.0038490882143378258,
+            -0.0045988536439836025,
+            0.027671586722135544,
+            0.010766984894871712,
+            -0.0001184450084110722,
+            0.015605589374899864,
+            -0.010043373331427574,
+            0.0422309935092926,
+            -0.015378915704786777,
+            -0.00737560261040926,
+            -0.010052092373371124,
+            -0.01685229316353798,
+            -0.003940629307180643,
+            -0.0011769578559324145,
+            -0.01689588464796543,
+            -0.006647632457315922,
+            -0.006320699583739042,
+            0.021411914378404617,
+            0.0046380856074392796,
+            -0.01665177382528782,
+            0.0017065888969227672,
+            -0.023835577070713043,
+            -0.0010946798138320446,
+            -0.022475536912679672,
+            -0.006403522565960884,
+            -0.012327543459832668,
+            -0.01349578332155943,
+            -0.020993441343307495,
+            -0.02031342126429081,
+            -0.004908350296318531,
+            0.016547156497836113,
+            0.00532246520742774,
+            -0.0069091785699129105,
+            -0.0021043571177870035,
+            -0.014402477070689201,
+            -0.010278765112161636,
+            0.02144678682088852,
+            0.015605589374899864,
+            0.012432162649929523,
+            0.01332141924649477,
+            0.01685229316353798,
+            0.0037597264163196087,
+            -0.034593842923641205,
+            0.006128899287432432,
+            0.0041607641614973545,
+            -0.005435801576822996,
+            0.01782873272895813,
+            0.000511104823090136,
+            -0.010793139226734638,
+            0.013399883173406124,
+            -0.007593557704240084,
+            0.018098996952176094,
+            -0.003212659154087305,
+            -0.004184739198535681,
+            -0.005536061245948076,
+            -0.008740002289414406,
+            -0.007371243555098772,
+            0.00455090356990695,
+            0.0021043571177870035,
+            0.013643993064761162,
+            0.0005775811732746661,
+            -0.004625008441507816,
+            -0.011333668604493141,
+            0.0014602995943278074,
+            -0.0032257363200187683,
+            -0.001628125086426735,
+            -0.023835577070713043,
+            -0.012170616537332535,
+            0.0017120378324761987,
+            -0.0086179468780756,
+            -0.01235369872301817,
+            0.02856084331870079,
+            -0.0017011400777846575,
+            -0.0008009852026589215,
+            0.010688520967960358,
+            0.015675334259867668,
+            0.001266319421119988,
+            0.01837797835469246,
+            -0.012170616537332535,
+            -0.011185458861291409,
+            -0.01649484597146511,
+            -0.01282448135316372,
+            -0.01122033130377531,
+            0.02090625837445259,
+            0.0033979208674281836,
+            -0.00708354264497757,
+            0.010418256744742393,
+            -0.011106994934380054,
+            -0.005056559573858976,
+            -0.0091018071398139,
+            0.011708551086485386,
+            -0.02969421073794365,
+            0.02069702185690403,
+            -0.012711144983768463,
+            0.006024280562996864,
+            0.018639525398612022,
+            0.039231929928064346,
+            -0.006608400493860245,
+            -0.006869946606457233,
+            0.010034655220806599,
+            -0.003068808699026704,
+            0.029205990955233574,
+            -0.0059414575807750225,
+            0.004237048327922821,
+            -0.021551406010985374,
+            0.0018449904164299369,
+            -0.01074954867362976,
+            0.0012815763475373387,
+            0.01991238258779049,
+            -0.0191974900662899,
+            -0.013722456991672516,
+            -0.00675660977140069,
+            -0.016058936715126038,
+            -0.00823870487511158,
+            0.022283734753727913,
+            0.011377259157598019,
+            -0.006111462600529194,
+            -0.008273578248918056,
+            -0.005736579652875662,
+            -0.024201741442084312,
+            0.014716332778334618,
+            0.08174190670251846,
+            -0.009982346557080746,
+            -0.0001858067698776722,
+            0.0003367407189216465,
+            0.0017806936521083117,
+            0.007907412946224213,
+            -0.00031821453012526035,
+            0.05956278741359711,
+            -0.003509078174829483,
+            0.003899217816069722,
+            -0.012998845428228378,
+            0.01083673071116209,
+            -0.01438504084944725,
+            -0.005684270523488522,
+            -0.009712082333862782,
+            -0.01197881530970335,
+            -0.0008167869527824223,
+            -0.02203962579369545,
+            -0.001063076313585043,
+            -0.011996251530945301,
+            -0.005021686665713787,
+            -0.021324733272194862,
+            0.017514877021312714,
+            -0.02087138593196869,
+            -0.01648612879216671,
+            -0.004594494588673115,
+            -0.010705957189202309,
+            0.012118306942284107,
+            -0.016555873677134514,
+            0.002380070276558399,
+            0.007270984351634979,
+            0.014228112995624542,
+            0.007475861813873053,
+            -0.008199472911655903,
+            -0.008408710360527039,
+            -0.0076197125017642975,
+            0.004986814223229885,
+            0.0069876424968242645,
+            -0.0011088468600064516,
+            0.014637868851423264,
+            -0.02088882215321064,
+            -0.016058936715126038,
+            0.007589198648929596,
+            0.0017817834159359336,
+            0.009502844884991646,
+            -0.007820230908691883,
+            0.007449707482010126,
+            0.00020882829267065972,
+            0.022754518315196037,
+            -0.010156710632145405,
+            -0.0268520750105381,
+            -0.007933568209409714,
+            0.008644101209938526,
+            -0.013138337060809135,
+            -0.006102744489908218,
+            0.005331183318048716,
+            0.009807982482016087,
+            0.008081777021288872,
+            -0.009877728298306465,
+            0.0007110786973498762,
+            -0.018918508663773537,
+            0.013373728841543198,
+            0.003238813718780875,
+            0.022196553647518158,
+            -0.012667553499341011,
+            0.010880321264266968,
+            -0.02683463878929615,
+            -0.002574050333350897,
+            -0.014419913291931152,
+            0.0008592882077209651,
+            0.008434864692389965,
+            0.004742704331874847,
+            0.01684357412159443,
+            -0.017192302271723747,
+            -0.017096402123570442,
+            0.009311044588685036,
+            -0.01782873272895813,
+            -0.02376583032310009,
+            -0.0018471699440851808,
+            -0.02624180167913437,
+            -0.0038163948338478804,
+            0.007532530464231968,
+            -0.019389290362596512,
+            -0.01369630265980959,
+            0.0065953233279287815,
+            -0.0033412526827305555,
+            -0.014611713588237762,
+            0.017183585092425346,
+            -0.011970097199082375,
+            -0.03544822707772255,
+            0.003727033268660307,
+            0.008382555097341537,
+            -0.004729627165943384,
+            0.006747891660779715,
+            -0.024410977959632874,
+            0.016015345230698586,
+            -0.002397506730630994,
+            0.004293716512620449,
+            0.008029468357563019,
+            0.02606743760406971,
+            0.0010309278732165694,
+            0.0034545892849564552,
+            -0.005544779356569052,
+            0.014411195181310177,
+            0.005801966413855553,
+            0.023504285141825676,
+            0.026416165754199028,
+            0.0015529305674135685,
+            -0.010958785191178322,
+            -0.01081929448992014,
+            0.01592816226184368,
+            -0.011586496606469154,
+            -0.0048386044800281525,
+            0.022737082093954086,
+            0.013827075250446796,
+            -0.02203962579369545,
+            0.00017259323794860393,
+            -0.02149909734725952,
+            0.0069876424968242645,
+            -0.017148712649941444,
+            0.009467972442507744,
+            0.008731283247470856,
+            -0.005073996260762215,
+            0.02069702185690403,
+            0.010095682926476002,
+            0.030984506011009216,
+            0.01991238258779049,
+            0.002920599188655615,
+            0.011412132531404495,
+            -0.006451472640037537,
+            0.010470565408468246,
+            0.004537826403975487,
+            0.0022275017108768225,
+            -0.006608400493860245,
+            0.006930973846465349,
+            0.0023451976012438536,
+            -0.001170419156551361,
+            0.008583074435591698,
+            -0.007471502758562565,
+            0.009921318851411343,
+            0.0005939278053119779,
+            0.0055273426696658134,
+            -0.004287178162485361,
+            -0.010575184598565102,
+            -0.030339358374476433,
+            -0.003132015699520707,
+            0.020958568900823593,
+            -0.023539157584309578,
+            -0.008003313094377518,
+            0.005932739470154047,
+            0.016983065754175186,
+            -0.008282296359539032,
+            -0.012798327021300793,
+            -0.01708768494427204,
+            0.003105861134827137,
+            -0.0004893093137070537,
+            -0.010313638485968113,
+            0.01747128553688526,
+            0.0009546435903757811,
+            -0.004951941315084696,
+            0.008953598327934742,
+            -0.00680456031113863,
+            0.028212115168571472,
+            0.01505634281784296,
+            0.011150586418807507,
+            -0.012100870721042156,
+            0.014201958663761616,
+            -0.012711144983768463,
+            0.007680739741772413,
+            0.009799264371395111,
+            -0.006416599731892347,
+            -0.021220114082098007,
+            -0.017558468505740166,
+            0.023451974615454674,
+            0.006848151329904795,
+            0.012222925201058388,
+            -0.011987533420324326,
+            -0.0012815763475373387,
+            0.007475861813873053,
+            0.004834245424717665,
+            -0.01611996442079544,
+            -0.00048631246318109334,
+            0.0061986446380615234,
+            0.038604218512773514,
+            0.005087073426693678,
+            0.01072339341044426,
+            -0.0035984397400170565,
+            -0.0002066487359115854,
+            0.019162617623806,
+            -0.01952878199517727,
+            -0.006503781769424677,
+            -0.005832480266690254,
+            -0.02739260531961918,
+            0.016573309898376465,
+            -0.01685229316353798,
+            -0.016451256349682808,
+            0.010976221412420273,
+            -0.00041356991278007627,
+            -0.004422310274094343,
+            -0.01284191757440567,
+            -0.008766156621277332,
+            0.00794228632003069,
+            0.010810576379299164,
+            -0.009136680513620377,
+            0.00532246520742774,
+            0.00357446470297873,
+            -0.0035984397400170565,
+            0.02127242274582386,
+            0.0006467819330282509,
+            -0.002837776206433773,
+            0.007358166389167309
+          ],
+          "title": "Method",
+          "keyphrases": [
+            "mixture-of-experts (MoE) models",
+            "token-choice routing",
+            "load imbalance",
+            "expert choice method",
+            "Transformer architecture"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "33412f87-7418-4ce2-9116-377a27fd2732",
+        "properties": {
+          "page_content": "4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n",
+          "title": "4 Experiments",
+          "embedding": [
+            0.009938324801623821,
+            -0.009144282899796963,
+            -0.026211904361844063,
+            -0.022352691739797592,
+            0.016043050214648247,
+            0.016247965395450592,
+            -0.02783413790166378,
+            0.0537557490170002,
+            -0.02827811799943447,
+            -0.01755429059267044,
+            0.010237157344818115,
+            0.001297788112424314,
+            0.014762339182198048,
+            -0.01001516729593277,
+            0.007487895432859659,
+            0.015283161774277687,
+            0.019347287714481354,
+            0.019910801202058792,
+            0.02535809576511383,
+            -0.04251963645219803,
+            0.0020213902462273836,
+            -0.032034873962402344,
+            -0.058332156389951706,
+            0.0034600566141307354,
+            0.001006959704682231,
+            0.01755429059267044,
+            0.011645940132439137,
+            -0.0015795447397977114,
+            -0.053584985435009,
+            0.04794985428452492,
+            0.016555335372686386,
+            0.0044910297729074955,
+            -0.007833687588572502,
+            -0.004798400681465864,
+            -0.04204150289297104,
+            -0.028961164876818657,
+            0.05720513314008713,
+            0.005165537819266319,
+            -0.005519868340343237,
+            0.05481446906924248,
+            0.020252324640750885,
+            0.010245694778859615,
+            -0.03961668908596039,
+            -0.009904172271490097,
+            0.010134699754416943,
+            0.001490962109528482,
+            -0.0010427129454910755,
+            -0.007479357533156872,
+            0.02295035682618618,
+            -0.04146091267466545,
+            -0.005549751687794924,
+            -0.03811398521065712,
+            0.01863008923828602,
+            0.013780459761619568,
+            -0.04426140338182449,
+            0.023223575204610825,
+            -0.0052978782914578915,
+            0.03527934476733208,
+            0.057854026556015015,
+            0.02783413790166378,
+            0.024162763729691505,
+            -0.02590453252196312,
+            -0.04340759664773941,
+            -0.0036180110182613134,
+            0.010143238119781017,
+            0.008119713515043259,
+            -0.01344747468829155,
+            -0.010527451522648335,
+            0.046173933893442154,
+            -0.023701708763837814,
+            -0.05327761545777321,
+            0.027680452913045883,
+            -0.013584083877503872,
+            -0.008081291802227497,
+            0.0036372218746691942,
+            -0.03517688810825348,
+            0.012807118706405163,
+            0.03548425808548927,
+            0.01004078146070242,
+            0.015018481761217117,
+            0.001224147155880928,
+            0.008418546058237553,
+            -0.0014920294051989913,
+            0.017912890762090683,
+            0.028517184779047966,
+            -0.01417321152985096,
+            -0.006309640593826771,
+            -0.0686461552977562,
+            0.025767924264073372,
+            -0.007368362508714199,
+            -0.020115714520215988,
+            0.00026281276950612664,
+            -0.007099412847310305,
+            0.0033597343135625124,
+            -0.020781684666872025,
+            0.07472527027130127,
+            -0.013105952180922031,
+            0.014386664144694805,
+            -0.02096952311694622,
+            0.007300057914108038,
+            -0.031932417303323746,
+            0.02236976847052574,
+            0.01146664097905159,
+            0.005916888825595379,
+            0.038318898528814316,
+            -0.02834642305970192,
+            -0.005869929678738117,
+            0.0122009152546525,
+            0.0292856115847826,
+            0.08387809246778488,
+            0.01419028826057911,
+            -0.014352511614561081,
+            -0.025563009083271027,
+            -0.016538260504603386,
+            -0.046105626970529556,
+            -0.0292856115847826,
+            -0.025170257315039635,
+            -0.027099864557385445,
+            0.036338064819574356,
+            -0.0010613900376483798,
+            0.012388752773404121,
+            0.05338007211685181,
+            0.015863751992583275,
+            -0.00874299369752407,
+            -0.039241012185811996,
+            0.0438857264816761,
+            -0.006198645569384098,
+            0.010809208266437054,
+            0.002330895746126771,
+            -0.0146428057923913,
+            -0.006185838486999273,
+            0.011833777651190758,
+            -0.013763383962213993,
+            -0.029456373304128647,
+            0.031949494034051895,
+            -0.01727253384888172,
+            0.03944592550396919,
+            0.06239628419280052,
+            -0.03562086820602417,
+            -0.02001325786113739,
+            0.03189826384186745,
+            -0.035825781524181366,
+            0.018288565799593925,
+            -0.0005250918911769986,
+            -0.030993228778243065,
+            0.026246055960655212,
+            0.01562468521296978,
+            0.06092773377895355,
+            -0.024196917191147804,
+            0.028705023229122162,
+            -0.019176525995135307,
+            0.01628211699426174,
+            -0.020627999678254128,
+            0.03596239164471626,
+            0.0023351646959781647,
+            -0.024265222251415253,
+            -0.015752756968140602,
+            0.014694035053253174,
+            -0.014275669120252132,
+            -0.022830823436379433,
+            0.019740039482712746,
+            0.02100367471575737,
+            -0.010450609028339386,
+            -0.0049051265232264996,
+            0.012013077735900879,
+            0.04989653453230858,
+            0.016205275431275368,
+            -0.014130521565675735,
+            -0.07479357719421387,
+            0.04801815748214722,
+            0.021703798323869705,
+            0.01958635449409485,
+            0.023104043677449226,
+            0.035825781524181366,
+            -0.001056053675711155,
+            0.030566323548555374,
+            0.0019007899099960923,
+            -0.013601160608232021,
+            -0.02044016122817993,
+            0.043646663427352905,
+            -0.0039019023533910513,
+            -0.0019274713704362512,
+            0.0076373121701180935,
+            0.018288565799593925,
+            0.03123229369521141,
+            0.005430218297988176,
+            0.010399380698800087,
+            0.039719145745038986,
+            -0.011586173437535763,
+            0.01414759736508131,
+            0.009460192173719406,
+            0.02091829478740692,
+            0.04299776628613472,
+            0.02245514839887619,
+            -0.008670419454574585,
+            -0.02194286324083805,
+            0.00032551426556892693,
+            0.0219599399715662,
+            0.01811780408024788,
+            0.02199409157037735,
+            -0.012013077735900879,
+            -0.0220111683011055,
+            0.009383349679410458,
+            -0.02199409157037735,
+            0.01682855375111103,
+            -0.00167666538618505,
+            0.01859593763947487,
+            0.013865840621292591,
+            -0.006258412264287472,
+            -0.01677732542157173,
+            0.0002502724528312683,
+            -0.018186109140515327,
+            -0.008380125276744366,
+            -0.010945817455649376,
+            -0.016495568677783012,
+            0.03364003449678421,
+            -0.020747533068060875,
+            0.006599935237318277,
+            -0.01810072734951973,
+            -0.011885006912052631,
+            -0.005835777148604393,
+            0.03596239164471626,
+            0.012209453620016575,
+            -0.01851055584847927,
+            -0.026058217510581017,
+            -0.011108040809631348,
+            -0.018817927688360214,
+            -0.006463326048105955,
+            0.0342206209897995,
+            -0.01750306226313114,
+            0.006659701932221651,
+            -0.004211407620459795,
+            -0.002409872831776738,
+            0.004258367232978344,
+            0.033435117453336716,
+            -0.03082246705889702,
+            -0.01676025055348873,
+            0.023582175374031067,
+            0.008059946820139885,
+            0.029046546667814255,
+            -0.016888320446014404,
+            -0.006275488529354334,
+            -0.024231068789958954,
+            -0.016973702237010002,
+            -0.01643580198287964,
+            -0.005118578672409058,
+            -0.02199409157037735,
+            -0.016025975346565247,
+            0.02790244296193123,
+            0.013686541467905045,
+            -0.00790199264883995,
+            -0.019313136115670204,
+            -0.002333030104637146,
+            0.02542640082538128,
+            -0.013592622242867947,
+            0.04064125940203667,
+            0.050545431673526764,
+            -0.05713682621717453,
+            -0.0122521435841918,
+            -0.03661128506064415,
+            0.07684271782636642,
+            -0.015377080999314785,
+            -0.03032725863158703,
+            0.033964481204748154,
+            -0.029388070106506348,
+            -0.008990597911179066,
+            0.026997406035661697,
+            0.025272715836763382,
+            -0.014719649218022823,
+            0.01123611256480217,
+            0.03756754845380783,
+            -0.030139420181512833,
+            -0.0439881831407547,
+            -0.03777246177196503,
+            0.019859572872519493,
+            -0.025289790704846382,
+            -0.01419028826057911,
+            -0.0110311983153224,
+            0.023530947044491768,
+            0.01801534742116928,
+            0.010664060711860657,
+            0.007547662127763033,
+            0.006843270733952522,
+            0.021157361567020416,
+            0.02144765481352806,
+            0.007073798682540655,
+            -0.03237639740109444,
+            0.0003433908859733492,
+            -0.03944592550396919,
+            -0.009093054570257664,
+            -0.025289790704846382,
+            -0.02095244638621807,
+            -0.023821242153644562,
+            0.00949434470385313,
+            0.03275207430124283,
+            -0.0008143192972056568,
+            0.0067365444265306,
+            0.015044095925986767,
+            0.015488076023757458,
+            -0.03807983547449112,
+            -0.03555256128311157,
+            0.006348061840981245,
+            0.01682855375111103,
+            -0.0055625587701797485,
+            -0.059220120310783386,
+            -0.00042770442087203264,
+            -0.003165493020787835,
+            -0.005016121547669172,
+            -0.028909936547279358,
+            0.010476223193109035,
+            0.013345018029212952,
+            0.010237157344818115,
+            0.0110311983153224,
+            -0.027526767924427986,
+            -0.014651344157755375,
+            -0.0036073385272175074,
+            0.006019345950335264,
+            0.02100367471575737,
+            -0.040709562599658966,
+            -0.022830823436379433,
+            0.03818229213356972,
+            0.0003810117777902633,
+            -0.06598227471113205,
+            -0.005387527868151665,
+            -0.010757979936897755,
+            0.015436847694218159,
+            -0.0005656477296724916,
+            0.034476764500141144,
+            -0.03121521882712841,
+            -0.019296059384942055,
+            -0.020815838128328323,
+            -0.051979828625917435,
+            -0.03531349450349808,
+            -0.015436847694218159,
+            0.007389707490801811,
+            -0.038353051990270615,
+            -0.024265222251415253,
+            0.004111085087060928,
+            -0.0268949493765831,
+            0.007398245856165886,
+            -0.013567008078098297,
+            0.015633223578333855,
+            -0.03852381557226181,
+            -0.010092009790241718,
+            0.004262635949999094,
+            -0.01861301250755787,
+            -0.005507061257958412,
+            0.014369587413966656,
+            0.013925607316195965,
+            0.014719649218022823,
+            0.0017268265364691615,
+            0.015266085974872112,
+            0.0062071834690868855,
+            -0.00769280968233943,
+            0.007833687588572502,
+            -0.04091447591781616,
+            0.0014461372047662735,
+            0.018852079287171364,
+            0.011432488448917866,
+            -0.011082426644861698,
+            0.004151640925556421,
+            -0.010373766534030437,
+            -0.03034433349967003,
+            0.029917430132627487,
+            -0.05419972911477089,
+            0.0024141420144587755,
+            0.005554020404815674,
+            -0.016410188749432564,
+            -0.008384393528103828,
+            -0.009690719656646252,
+            -0.03708941861987114,
+            -0.01248267199844122,
+            0.0039019023533910513,
+            0.00019050588889513165,
+            0.009349197149276733,
+            0.002870929194614291,
+            0.05471201241016388,
+            -0.003165493020787835,
+            -0.04484199360013008,
+            0.023308956995606422,
+            -0.03671374171972275,
+            0.018749622628092766,
+            0.04245132952928543,
+            0.020713381469249725,
+            -0.017212767153978348,
+            0.04791570082306862,
+            0.0195692777633667,
+            0.000224524803343229,
+            -0.016999315470457077,
+            0.025153182446956635,
+            -0.007099412847310305,
+            -0.061303410679101944,
+            -0.004448339343070984,
+            -0.018749622628092766,
+            0.018664240837097168,
+            0.016674868762493134,
+            -0.002407738473266363,
+            -0.010032243095338345,
+            0.009426039643585682,
+            -0.01510386262089014,
+            -0.017443295568227768,
+            0.01705908216536045,
+            -0.011458102613687515,
+            -0.004525181837379932,
+            0.007526317145675421,
+            0.013148642145097256,
+            0.01271320041269064,
+            0.008853988721966743,
+            0.023992003872990608,
+            -0.045627497136592865,
+            0.009127207100391388,
+            -0.01997910626232624,
+            -0.036918655037879944,
+            -0.002990462351590395,
+            0.01318279467523098,
+            0.04183658957481384,
+            -0.02779998630285263,
+            0.0019253368955105543,
+            0.005007583647966385,
+            -0.013515779748558998,
+            -0.021106131374835968,
+            0.00341309723444283,
+            0.04600317031145096,
+            -0.0015763428527861834,
+            0.021618416532874107,
+            -0.011304416693747044,
+            -0.03562086820602417,
+            0.008811297826468945,
+            0.04142676293849945,
+            -0.005028928630053997,
+            0.01663217879831791,
+            0.009434578008949757,
+            0.03548425808548927,
+            -0.04590071365237236,
+            -0.007833687588572502,
+            -0.02344556525349617,
+            -0.01813488081097603,
+            0.04105108603835106,
+            -0.03075416199862957,
+            -0.01365238893777132,
+            0.03524519130587578,
+            -0.005306416191160679,
+            0.02441890724003315,
+            -0.018271489068865776,
+            -0.02588745579123497,
+            0.0014600115828216076,
+            -0.03712356835603714,
+            -0.00708660576492548,
+            -0.01806657575070858,
+            -0.006160224322229624,
+            -0.06557244807481766,
+            0.00010459146869834512,
+            -0.07431544363498688,
+            -0.002902946900576353,
+            0.025136105716228485,
+            -0.006552976090461016,
+            0.04429555684328079,
+            -0.026228979229927063,
+            0.02098659984767437,
+            -0.05553166940808296,
+            -0.01367800310254097,
+            0.00978463888168335,
+            0.03375956416130066,
+            0.007611697539687157,
+            -0.001995776081457734,
+            0.0015507286880165339,
+            -0.026997406035661697,
+            -0.0018303507240489125,
+            -0.007282981649041176,
+            -0.001387437921948731,
+            4.89605481561739e-05,
+            -0.018852079287171364,
+            0.00950288213789463,
+            0.000838332693092525,
+            0.01610281690955162,
+            0.031454283744096756,
+            0.022711291909217834,
+            -0.03070293366909027,
+            -0.008862526156008244,
+            0.005195421166718006,
+            0.02587038092315197,
+            -0.02978082001209259,
+            0.003897633170709014,
+            -0.03464752435684204,
+            0.014864795841276646,
+            5.636466448777355e-05,
+            0.016657792031764984,
+            0.008008718490600586,
+            0.017016392201185226,
+            -0.040197279304265976,
+            0.0030160765163600445,
+            -0.0016254368238151073,
+            -0.027202321216464043,
+            0.020867066457867622,
+            0.002305281348526478,
+            -0.022147778421640396,
+            0.0054472945630550385,
+            -0.004207138437777758,
+            0.024213993921875954,
+            0.04473953694105148,
+            -0.0010592554463073611,
+            -0.02930268831551075,
+            -0.012704662047326565,
+            0.042212262749671936,
+            0.02633143588900566,
+            -0.028483033180236816,
+            0.017118848860263824,
+            0.02103782817721367,
+            -0.02341141365468502,
+            -0.021089056506752968,
+            0.013925607316195965,
+            -0.016179660335183144,
+            -0.0007694943924434483,
+            -0.005468640010803938,
+            0.010399380698800087,
+            0.01240582950413227,
+            -0.029883278533816338,
+            0.05676115304231644,
+            -0.016820017248392105,
+            -0.006834732368588448,
+            -0.03034433349967003,
+            -0.006168762221932411,
+            -0.014531810767948627,
+            0.05553166940808296,
+            0.0028175662737339735,
+            0.02728770114481449,
+            0.01658095046877861,
+            0.002264725510030985,
+            0.05628301948308945,
+            -0.03678204491734505,
+            -0.03451091796159744,
+            0.018681317567825317,
+            -0.0055625587701797485,
+            -0.00671093026176095,
+            -0.022728366777300835,
+            -0.014207364059984684,
+            -0.013353556394577026,
+            0.016965163871645927,
+            -0.0268778745085001,
+            -0.012260681949555874,
+            0.019740039482712746,
+            0.017690900713205338,
+            -2.0469710761972237e-06,
+            0.02293328195810318,
+            0.012644895352423191,
+            0.014284206554293633,
+            0.017878737300634384,
+            -0.020867066457867622,
+            0.019142374396324158,
+            0.024111535400152206,
+            0.009946862235665321,
+            -0.01763967238366604,
+            -0.017878737300634384,
+            0.007347017526626587,
+            -0.0038613462820649147,
+            -0.007125027012079954,
+            0.01437812577933073,
+            0.007043915335088968,
+            -0.004047049675136805,
+            -0.027151092886924744,
+            -0.018356870859861374,
+            0.02093537151813507,
+            -0.0023757205344736576,
+            0.01539415679872036,
+            -0.00928943045437336,
+            -0.010612832382321358,
+            0.03763585537672043,
+            0.01861301250755787,
+            -0.046139780431985855,
+            0.026280207559466362,
+            0.01051037572324276,
+            -0.006228528916835785,
+            -0.013729231432080269,
+            0.005626594182103872,
+            0.009101592935621738,
+            0.010220080614089966,
+            0.016572412103414536,
+            -0.005097233224660158,
+            -0.03328143432736397,
+            0.01564176194369793,
+            0.026211904361844063,
+            -0.020798761397600174,
+            0.012021616101264954,
+            0.00017663151083979756,
+            0.022762520238757133,
+            -0.005028928630053997,
+            -0.012311910279095173,
+            -0.01562468521296978,
+            0.009741948917508125,
+            -0.012235067784786224,
+            0.01123611256480217,
+            0.037396788597106934,
+            0.015914980322122574,
+            -0.03565501794219017,
+            0.015735680237412453,
+            -0.01686270721256733,
+            -0.015684451907873154,
+            0.03261546418070793,
+            0.008264861069619656,
+            -0.02636558935046196,
+            -0.007142103277146816,
+            -0.006527361460030079,
+            0.02001325786113739,
+            -0.02102075144648552,
+            -0.008508196100592613,
+            0.004418455995619297,
+            0.0035646480973809958,
+            0.009938324801623821,
+            0.006745082791894674,
+            0.005016121547669172,
+            0.019159449264407158,
+            0.013806073926389217,
+            -0.0005186883499845862,
+            0.012730276212096214,
+            0.011577636003494263,
+            0.002902946900576353,
+            0.0075391242280602455,
+            -0.010997045785188675,
+            -0.02295035682618618,
+            0.004734364803880453,
+            0.05945918336510658,
+            -0.005852853413671255,
+            0.03876287862658501,
+            0.010407919064164162,
+            0.028961164876818657,
+            0.021686721593141556,
+            -0.024299373850226402,
+            -0.01681147888302803,
+            -0.018151957541704178,
+            -0.021737949922680855,
+            -0.04453462362289429,
+            -0.01510386262089014,
+            -0.01629919372498989,
+            -0.013404784724116325,
+            0.005694898776710033,
+            -0.00013474156730808318,
+            -0.0013116624904796481,
+            0.011355645954608917,
+            -0.002003246918320656,
+            -0.012491210363805294,
+            -0.003355465130880475,
+            0.05324346199631691,
+            -0.0011056812945753336,
+            0.017912890762090683,
+            -0.004576410632580519,
+            0.015539304353296757,
+            -0.007300057914108038,
+            -0.016111355274915695,
+            0.029131926596164703,
+            0.0035561099648475647,
+            -0.0007540191290900111,
+            0.012106996960937977,
+            0.004469684325158596,
+            0.04149506613612175,
+            -0.005887005943804979,
+            -0.026058217510581017,
+            0.011910621076822281,
+            -0.017187153920531273,
+            0.020371858030557632,
+            0.00769707839936018,
+            -0.007607428822666407,
+            0.001932807732373476,
+            0.012909576296806335,
+            -0.027116939425468445,
+            0.019705887883901596,
+            -0.020645076408982277,
+            -0.045149363577365875,
+            0.009280892089009285,
+            -0.01175693515688181,
+            0.023035738617181778,
+            0.001117421081289649,
+            -0.010800670832395554,
+            -0.03507443144917488,
+            0.0021387888118624687,
+            0.0039040367119014263,
+            -0.03372541442513466,
+            0.006548706907778978,
+            0.01863008923828602,
+            0.015539304353296757,
+            0.009485806338489056,
+            -0.012465596199035645,
+            0.003758889390155673,
+            0.0031847036443650723,
+            -0.012371676973998547,
+            0.02776583470404148,
+            -0.004742903169244528,
+            -0.031983643770217896,
+            0.035859934985637665,
+            -0.02733892947435379,
+            0.009468729607760906,
+            -0.020730456337332726,
+            -0.012346062809228897,
+            -0.01608574204146862,
+            0.00441418681293726,
+            -0.02148180827498436,
+            -0.002561423694714904,
+            -0.02983205020427704,
+            0.027577996253967285,
+            0.009383349679410458,
+            -0.015249010175466537,
+            -0.004862436093389988,
+            0.009101592935621738,
+            -0.025255639106035233,
+            0.007846495136618614,
+            0.012585128657519817,
+            -0.010646984912455082,
+            -0.01367800310254097,
+            -0.00706099160015583,
+            -0.0014685495989397168,
+            0.005972386337816715,
+            0.0158210601657629,
+            0.04672037065029144,
+            0.004431263078004122,
+            0.03712356835603714,
+            0.02337726205587387,
+            0.013532855547964573,
+            0.0037012575194239616,
+            0.0047898623161017895,
+            -0.0067322757095098495,
+            0.04012897238135338,
+            0.006249873898923397,
+            0.0219599399715662,
+            0.008247784338891506,
+            -0.044671230018138885,
+            0.008503926917910576,
+            0.03264961391687393,
+            0.026621730998158455,
+            0.018715469166636467,
+            0.009588262997567654,
+            -0.0024013349320739508,
+            0.021191513165831566,
+            -0.020252324640750885,
+            -0.01705908216536045,
+            0.0035902622621506453,
+            -0.007530585862696171,
+            -0.0056650154292583466,
+            -0.009716334752738476,
+            0.03387909755110741,
+            -0.003229528432711959,
+            -0.008862526156008244,
+            0.018186109140515327,
+            -0.007432397920638323,
+            0.00475997943431139,
+            0.006245605181902647,
+            0.0011163539020344615,
+            0.02739015780389309,
+            0.008683227002620697,
+            0.027680452913045883,
+            0.005391797050833702,
+            0.007573276292532682,
+            0.0058571225963532925,
+            -0.009451653808355331,
+            0.018920384347438812,
+            -0.0097590247169137,
+            -0.033366814255714417,
+            -0.009443115442991257,
+            0.0016393112018704414,
+            0.014070754870772362,
+            0.0004023569927085191,
+            -0.014062216505408287,
+            -0.011893544346094131,
+            0.007859301753342152,
+            0.012533900327980518,
+            0.02107197977602482,
+            0.018169032409787178,
+            -0.005016121547669172,
+            -0.024726277217268944,
+            0.012593667022883892,
+            0.011056812480092049,
+            -0.01861301250755787,
+            0.008465506136417389,
+            0.012542438693344593,
+            0.057410046458244324,
+            -0.044705383479595184,
+            0.016538260504603386,
+            0.02151595987379551,
+            -0.004896588623523712,
+            -0.015249010175466537,
+            -0.010851899161934853,
+            0.0031633584294468164,
+            -0.017477449029684067,
+            0.014975790865719318,
+            -0.01798119582235813,
+            0.011193421669304371,
+            -0.014745263382792473,
+            0.00743666710332036,
+            0.012866885401308537,
+            -0.009682182222604752,
+            0.003942457959055901,
+            0.020371858030557632,
+            0.0028068935498595238,
+            -0.0077867284417152405,
+            0.005387527868151665,
+            0.01390853151679039,
+            0.020354781299829483,
+            0.006066305562853813,
+            -0.0005933965439908206,
+            0.019876649603247643,
+            -0.028175661340355873,
+            -0.025170257315039635,
+            0.01909114606678486,
+            -0.03551841154694557,
+            0.005472908727824688,
+            0.02790244296193123,
+            -0.04296361654996872,
+            -0.05020390823483467,
+            0.0027407235465943813,
+            -0.039719145745038986,
+            0.0004191663465462625,
+            -0.0023287611547857523,
+            0.01536854263395071,
+            -0.004057722166180611,
+            -0.017212767153978348,
+            0.0146428057923913,
+            -0.012397291138768196,
+            -0.0067322757095098495,
+            0.009835867211222649,
+            0.009016212075948715,
+            -0.024572592228651047,
+            -0.0016222350532189012,
+            -0.030890770256519318,
+            0.009169897064566612,
+            -0.017844585701823235,
+            -0.019142374396324158,
+            -0.025528857484459877,
+            -0.0005656477296724916,
+            -0.010638446547091007,
+            -0.001638244022615254,
+            0.051501695066690445,
+            0.021652569994330406,
+            0.020166942849755287,
+            -0.007726961746811867,
+            -0.023650480434298515,
+            0.05457540228962898,
+            0.0018132745753973722,
+            0.03814813867211342,
+            -0.012380215339362621,
+            -0.0002006448630709201,
+            -0.0008852921309880912,
+            0.022113624960184097,
+            -0.017101772129535675,
+            -0.0011547752656042576,
+            -0.011022660881280899,
+            -0.00586566049605608,
+            -0.0070780678652226925,
+            0.014045140706002712,
+            0.015974747017025948,
+            -0.002610517665743828,
+            0.014156135730445385,
+            0.01221799198538065,
+            0.022694215178489685,
+            0.03130060061812401,
+            0.016230888664722443,
+            -0.0034664601553231478,
+            -0.022199006751179695,
+            -0.012918113730847836,
+            -0.008200825192034245,
+            0.008999135345220566,
+            -0.008525271899998188,
+            -0.0031847036443650723,
+            -0.02631436102092266,
+            -0.020730456337332726,
+            -0.011312955059111118,
+            -0.01292665209621191,
+            -0.0031462821643799543,
+            0.02641681768000126,
+            -0.002469639526680112,
+            -0.03763585537672043,
+            -0.01511240005493164,
+            -0.03524519130587578,
+            -0.015607609413564205,
+            -0.0017876604106277227,
+            -0.01748598739504814,
+            0.016538260504603386,
+            -0.003780234605073929,
+            -0.007833687588572502,
+            -0.0008964983280748129,
+            -0.018169032409787178,
+            0.009690719656646252,
+            -0.011048275046050549,
+            -0.000543768925126642,
+            -0.0017503062263131142,
+            0.013345018029212952,
+            -0.008205094374716282,
+            0.008973521180450916,
+            -0.01412198320031166,
+            0.006868884898722172,
+            -0.048120614141225815,
+            -0.027731681242585182,
+            -0.027099864557385445,
+            0.022557605057954788,
+            -0.015214857645332813,
+            -0.023189423605799675,
+            0.008734455332159996,
+            -0.013686541467905045,
+            0.008939368650317192,
+            -0.012021616101264954,
+            0.008388662710785866,
+            0.019210679456591606,
+            0.009024749509990215,
+            -0.002614786848425865,
+            -0.013669464737176895,
+            -0.02390662208199501,
+            -0.02585330419242382,
+            0.023650480434298515,
+            -0.026485122740268707,
+            0.01071528997272253,
+            -0.012021616101264954,
+            0.008585038594901562,
+            -0.029917430132627487,
+            -0.020730456337332726,
+            0.0044910297729074955,
+            0.011347107589244843,
+            -0.021140284836292267,
+            0.044671230018138885,
+            0.04043634235858917,
+            0.029985735192894936,
+            -0.020252324640750885,
+            0.05139923840761185,
+            -0.022796671837568283,
+            0.007312864996492863,
+            0.036440521478652954,
+            0.030446790158748627,
+            0.026092370972037315,
+            -0.015581995248794556,
+            -0.012294834479689598,
+            -0.015496614389121532,
+            0.013404784724116325,
+            -0.024333525449037552,
+            0.002821835223585367,
+            -0.02042308636009693,
+            0.013643850572407246,
+            0.03451091796159744,
+            -0.004917933605611324,
+            0.029934506863355637,
+            -0.013720693998038769,
+            0.030361410230398178,
+            0.0017652479000389576,
+            -0.026587579399347305,
+            0.015667375177145004,
+            -0.014232978224754333,
+            0.004418455995619297,
+            0.017964119091629982,
+            -0.006087650544941425,
+            0.013575546443462372,
+            -0.061337560415267944,
+            0.028141509741544724,
+            0.01758844405412674,
+            0.01123611256480217,
+            -0.002180411946028471,
+            -0.024726277217268944,
+            -0.0021142419427633286,
+            0.009340658783912659,
+            -0.0016745307948440313,
+            0.011859391815960407,
+            0.01193623524159193,
+            -0.033418044447898865,
+            0.010578680783510208,
+            -0.013635313138365746,
+            0.024299373850226402,
+            0.006258412264287472,
+            0.0007476155878975987,
+            -0.03961668908596039,
+            -0.00560524920001626,
+            0.014830644242465496,
+            0.01095435582101345,
+            -0.005989462602883577,
+            -0.04624223709106445,
+            -0.030105268582701683,
+            0.025563009083271027,
+            0.005110040307044983,
+            -0.05167245492339134,
+            0.014557424932718277,
+            -0.015334390103816986,
+            0.018766699358820915,
+            -0.008708841167390347,
+            -0.025972837582230568,
+            -0.029985735192894936,
+            -0.017187153920531273,
+            0.016726097092032433,
+            0.048564594238996506,
+            0.004207138437777758,
+            0.013345018029212952,
+            -0.0048538981936872005,
+            0.006651164032518864,
+            -0.024760430678725243,
+            0.012030153535306454,
+            -0.027151092886924744,
+            -0.02971251681447029,
+            0.029012393206357956,
+            0.009981014765799046,
+            -0.011330030858516693,
+            0.019330210983753204,
+            -0.017084697261452675,
+            -0.0014141193823888898,
+            0.02240392006933689,
+            -0.0006163425859995186,
+            -0.007295788731426001,
+            -0.010382304899394512,
+            0.013618236407637596,
+            -0.0014354645973071456,
+            -0.01762259565293789,
+            -0.009571187198162079,
+            0.04241717979311943,
+            0.010092009790241718,
+            0.018835002556443214,
+            0.03261546418070793,
+            -0.0018933190731331706,
+            -0.02400907874107361,
+            -0.0027215126901865005,
+            -0.026638807728886604,
+            -0.017033468931913376,
+            -0.014548887498676777,
+            0.016674868762493134,
+            0.023530947044491768,
+            0.015249010175466537,
+            -0.027526767924427986,
+            -0.030019886791706085,
+            -0.01950097270309925,
+            -0.022199006751179695,
+            -0.0034493841230869293,
+            -0.0004213008505757898,
+            -0.026502197608351707,
+            0.007005494087934494,
+            0.0341181643307209,
+            0.03357172757387161,
+            -0.0010870042024180293,
+            -0.009016212075948715,
+            -0.010638446547091007,
+            0.0007721625734120607,
+            0.015231933444738388,
+            -0.015018481761217117,
+            0.0012326851719990373,
+            0.005028928630053997,
+            0.005383259151130915,
+            0.0029285612981766462,
+            -0.007095144130289555,
+            -0.05747834965586662,
+            0.003961668815463781,
+            0.02544347569346428,
+            0.005084426142275333,
+            -0.0008377990452572703,
+            0.02629728429019451,
+            0.0008276600856333971,
+            0.008123982697725296,
+            -0.03142013028264046,
+            0.011108040809631348,
+            0.01803242415189743,
+            0.015163629315793514,
+            -0.018151957541704178,
+            0.0122521435841918,
+            0.0064206356182694435,
+            0.004811207763850689,
+            -0.00030950538348406553,
+            0.010143238119781017,
+            0.028636718168854713,
+            0.019620506092905998,
+            -0.014736725017428398,
+            0.023821242153644562,
+            0.0006435577524825931,
+            -0.012072844430804253,
+            -0.024333525449037552,
+            -0.001040044822730124,
+            0.0029883277602493763,
+            -0.011176345869898796,
+            0.014702572487294674,
+            -0.0023949311580508947,
+            0.0007956422632560134,
+            0.04139260947704315,
+            0.0437491200864315,
+            0.013481627218425274,
+            -0.011295879259705544,
+            0.003304236801341176,
+            0.026502197608351707,
+            0.020132791250944138,
+            0.004862436093389988,
+            0.011458102613687515,
+            -0.03331558406352997,
+            -0.019620506092905998,
+            -0.002981924219056964,
+            0.0014290610561147332,
+            -0.0026788224931806326,
+            0.03876287862658501,
+            0.029524678364396095,
+            0.009682182222604752,
+            -0.005741858389228582,
+            -0.011364183388650417,
+            0.013643850572407246,
+            0.020201096311211586,
+            -0.023599252104759216,
+            -0.0038015798199921846,
+            0.0016745307948440313,
+            -0.017170077189803123,
+            0.02288205362856388,
+            -0.0070311082527041435,
+            -0.00928943045437336,
+            0.02645096927881241,
+            0.09186973422765732,
+            -0.00464471522718668,
+            0.012141148559749126,
+            -0.03948007896542549,
+            -0.021823331713676453,
+            -0.006975610740482807,
+            -0.0036671049892902374,
+            0.015402695164084435,
+            0.02739015780389309,
+            0.009135744534432888,
+            0.007133565377444029,
+            0.017878737300634384,
+            -0.004606293980032206,
+            0.012508286163210869,
+            -0.02826104313135147,
+            0.020166942849755287,
+            -0.005007583647966385,
+            -0.0038335975259542465,
+            -0.011389797553420067,
+            -0.005989462602883577,
+            0.011765473522245884,
+            0.005468640010803938,
+            0.017016392201185226,
+            0.0019893725402653217,
+            -0.0024760430678725243,
+            0.0014333301223814487,
+            0.003031018190085888,
+            -0.010877513326704502,
+            -0.009383349679410458,
+            -0.021396426483988762,
+            0.003827193984761834,
+            0.0009914844995364547,
+            -0.007884915918111801,
+            0.012755890376865864,
+            0.005502792075276375,
+            0.024043232202529907,
+            -0.028175661340355873,
+            -0.0030609015375375748,
+            0.00633525475859642,
+            -0.008956445381045341,
+            0.010390842333436012,
+            0.012380215339362621,
+            -0.008640536107122898,
+            0.022830823436379433,
+            -0.01466841995716095,
+            0.014762339182198048,
+            0.01170570682734251,
+            -0.021601341664791107,
+            -0.004155910108238459,
+            -0.009938324801623821,
+            -0.02192578837275505,
+            0.020252324640750885,
+            -0.018715469166636467,
+            0.005033197812736034,
+            0.00073320756200701,
+            0.006774966139346361,
+            -0.0010496501345187426,
+            -0.01740914396941662,
+            0.010348152369260788,
+            -0.008785683661699295,
+            0.006839001551270485,
+            -0.005058811977505684,
+            0.018254414200782776,
+            0.012166762724518776,
+            0.0028090281412005424,
+            -0.0011622461024671793,
+            0.014600115828216076,
+            -0.00372900627553463,
+            -0.018886230885982513,
+            -0.0036713741719722748,
+            0.01633334532380104,
+            0.01051037572324276,
+            0.0002606782363727689,
+            -0.010305461473762989,
+            -0.0146342683583498,
+            0.014139058999717236,
+            0.000977076473645866,
+            -0.0018122073961421847,
+            -0.0005285604856908321,
+            -0.03760170191526413,
+            0.002606248715892434,
+            0.013532855547964573,
+            0.012704662047326565,
+            -0.005818700883537531,
+            -0.017477449029684067,
+            0.007983104325830936,
+            -0.004190062638372183,
+            -0.0003204447857569903,
+            0.01661510206758976,
+            -0.019774191081523895,
+            0.01002370472997427,
+            -0.014779414981603622,
+            -0.03278622403740883,
+            -0.0029776552692055702,
+            -0.008380125276744366,
+            0.005822970066219568,
+            -0.005592442117631435,
+            -0.0071805245243012905,
+            -0.01414759736508131,
+            -0.005741858389228582,
+            -0.02979789674282074,
+            0.008947907015681267,
+            0.0317445807158947,
+            0.004473953507840633,
+            -0.028978241607546806,
+            -0.012158225290477276,
+            0.025580085813999176,
+            -0.0024931191001087427,
+            -0.006915844511240721,
+            0.003009672975167632,
+            0.0012572321575134993,
+            -0.005784548819065094,
+            -0.009656568057835102,
+            -0.012935190461575985,
+            0.013063261285424232,
+            0.01320840884000063,
+            0.011244650930166245,
+            -0.03357172757387161,
+            0.010664060711860657,
+            0.01813488081097603,
+            -0.02042308636009693,
+            0.011748396791517735,
+            0.028585489839315414,
+            0.0029435029719024897,
+            0.010399380698800087,
+            0.007334209978580475,
+            0.008286206051707268,
+            -0.017153002321720123,
+            0.016683407127857208,
+            0.009186973795294762,
+            -0.012559514492750168,
+            0.004563603550195694,
+            0.007381169591099024,
+            0.008542348630726337,
+            0.01960342936217785,
+            -0.011603250168263912,
+            0.013609698973596096,
+            -0.0068816919811069965,
+            0.03321312740445137,
+            -0.011919158510863781,
+            -0.014881872572004795,
+            0.03229101747274399,
+            0.027560919523239136,
+            -0.004563603550195694,
+            0.017246920615434647,
+            -0.01943266950547695,
+            -0.01267050951719284,
+            0.019296059384942055,
+            0.0005907283630222082,
+            0.019313136115670204,
+            -0.011304416693747044,
+            -0.02595576085150242,
+            0.029012393206357956,
+            -0.008106905966997147,
+            -0.001259366748854518,
+            -0.015829598531126976,
+            0.0043458822183310986,
+            -0.0005789885181002319,
+            0.037328481674194336,
+            0.012653433717787266,
+            0.0011985328746959567,
+            0.026570502668619156,
+            -0.00769280968233943,
+            -0.025067800655961037,
+            0.001990439835935831,
+            -0.015214857645332813,
+            0.013336479663848877,
+            -0.027970748022198677,
+            -0.00560524920001626,
+            0.014506196603178978,
+            0.003026749240234494,
+            -0.002644670195877552,
+            0.01666633039712906,
+            -0.007513510063290596,
+            0.011791087687015533,
+            -0.019244831055402756,
+            0.016717558726668358,
+            -0.011517869308590889,
+            0.004473953507840633,
+            -0.011910621076822281,
+            0.026092370972037315,
+            -0.025563009083271027,
+            -0.00999809056520462,
+            0.013660927303135395,
+            -0.008354511111974716,
+            0.0006376878009177744,
+            -0.02252345345914364,
+            -0.008089830167591572,
+            -0.020371858030557632,
+            0.01679440215229988,
+            -0.005682091694325209,
+            0.011902082711458206,
+            -0.00306303589604795,
+            0.0025528857950121164,
+            -0.008585038594901562,
+            0.010322538204491138,
+            -0.0009199780761264265,
+            -0.017417682334780693,
+            -0.005968117620795965,
+            -0.03025895357131958,
+            -0.030361410230398178,
+            0.03072001039981842,
+            -0.00880275946110487,
+            0.003923247568309307,
+            0.02728770114481449,
+            -0.02540932409465313,
+            -0.005485715810209513,
+            0.001918933354318142,
+            0.022728366777300835,
+            -0.030651705339550972,
+            0.007914799265563488,
+            -0.009391887113451958,
+            -0.016452878713607788,
+            0.007184793706983328,
+            -0.0008874266059137881,
+            0.015880826860666275,
+            -0.0025635582860559225,
+            -0.04924764111638069,
+            -0.0007081269286572933,
+            0.017135925590991974,
+            -0.013806073926389217,
+            0.008081291802227497,
+            0.02151595987379551,
+            -0.009630953893065453,
+            -0.008439891040325165,
+            -0.0011334300506860018,
+            0.025989914312958717,
+            -0.022625910118222237,
+            0.030019886791706085,
+            0.005767472553998232,
+            0.004713019821792841,
+            0.005340568721294403,
+            0.002962713595479727,
+            0.01730668731033802,
+            0.002080089645460248,
+            0.004258367232978344,
+            0.025238562375307083,
+            0.013148642145097256,
+            0.0037418133579194546,
+            -0.0032978332601487637,
+            -0.0030352871399372816,
+            0.016538260504603386,
+            -0.018442250788211823,
+            0.02100367471575737,
+            -0.0004714620590675622,
+            0.00013560871593654156,
+            0.016256503760814667,
+            -0.01484772004187107,
+            -0.027219397947192192,
+            0.03075416199862957,
+            -0.022335615009069443,
+            0.01757136732339859,
+            0.023770013824105263,
+            -0.01339624635875225,
+            -0.004922202788293362,
+            -0.007073798682540655,
+            0.001534719718620181,
+            0.018339794129133224,
+            -0.0134218605235219,
+            -0.023308956995606422,
+            -0.00030790449818596244,
+            0.0220111683011055,
+            0.02293328195810318,
+            0.028175661340355873,
+            -0.021140284836292267,
+            0.01412198320031166,
+            0.007466550450772047,
+            -0.008311820216476917,
+            0.016469955444335938,
+            0.008452698588371277,
+            -0.0025272713974118233,
+            0.021293969824910164,
+            0.0075903525575995445,
+            0.004576410632580519,
+            -0.008896678686141968,
+            -0.013242561370134354,
+            -0.01267050951719284,
+            -0.006749351508915424,
+            -0.006164493504911661,
+            -0.008264861069619656,
+            -0.01709323562681675,
+            0.09548988193273544,
+            -0.011867930181324482,
+            -0.0028410458471626043,
+            -0.009195511229336262,
+            0.0026041141245514154,
+            -0.006949996575713158,
+            0.0013479492627084255,
+            0.023086966946721077,
+            -0.022728366777300835,
+            -0.03671374171972275,
+            0.011167807504534721,
+            -0.004070529248565435,
+            0.006288295611739159,
+            0.03712356835603714,
+            0.00038314631092362106,
+            -0.016589488834142685,
+            -6.773765198886395e-05,
+            -0.003372541395947337,
+            0.01295226626098156,
+            0.0028197006322443485,
+            -0.0036158766597509384,
+            0.0109799699857831,
+            0.029200231656432152,
+            -0.017810434103012085,
+            -0.02144765481352806,
+            -0.012525362893939018,
+            0.0032615463715046644,
+            -0.002578499959781766,
+            0.003707661060616374,
+            0.0051996903494000435,
+            -0.014053679071366787,
+            -0.0134218605235219,
+            0.018817927688360214,
+            -0.016094280406832695,
+            0.0008025794522836804,
+            -0.012627819553017616,
+            -0.008222170174121857,
+            0.01753721572458744,
+            0.01757136732339859,
+            -0.03406693786382675,
+            -0.009921248070895672,
+            -0.03855796530842781,
+            0.018749622628092766,
+            0.01643580198287964,
+            0.014540349133312702,
+            -0.020884141325950623,
+            -0.012644895352423191,
+            -0.004520913120359182,
+            -0.02981497347354889,
+            -0.02243807166814804,
+            -0.026536351069808006,
+            0.009605339728295803,
+            0.00365856708958745,
+            0.027595072984695435,
+            0.005259457044303417,
+            -0.006262681446969509,
+            0.00801725685596466,
+            -0.01025423314422369,
+            0.00755620002746582,
+            -0.02146473154425621,
+            0.007107951212674379,
+            -0.0011409008875489235,
+            0.0012529630912467837,
+            0.005592442117631435,
+            -0.008853988721966743,
+            -0.01735791563987732,
+            -0.0079276068136096,
+            0.01466841995716095,
+            -0.015752756968140602,
+            -0.00821790099143982,
+            0.020798761397600174,
+            0.009383349679410458,
+            0.00931504461914301,
+            0.015616146847605705,
+            0.045661646872758865,
+            0.008273398503661156,
+            0.00783795677125454,
+            -0.011056812480092049,
+            0.009648029692471027,
+            -0.01813488081097603,
+            0.00745801255106926,
+            -0.019330210983753204,
+            -0.006292564328759909,
+            -0.019722962751984596,
+            0.04012897238135338,
+            -0.009366272948682308,
+            -0.01635042205452919,
+            0.033435117453336716,
+            -0.011808163486421108,
+            0.01910822093486786,
+            0.0035475720651447773,
+            -0.022830823436379433,
+            0.03352050110697746,
+            0.003876287955790758,
+            0.012918113730847836,
+            -0.025682542473077774,
+            0.0035859933122992516,
+            -0.00877714529633522,
+            0.006032153032720089,
+            -0.011586173437535763,
+            0.003494208911433816,
+            0.0023885276168584824,
+            -0.004841091111302376,
+            -0.008760069496929646,
+            0.018698394298553467,
+            0.006595666520297527,
+            0.0002183880569646135,
+            0.028056127950549126,
+            -0.000954130373429507,
+            0.018220260739326477,
+            6.110063259257004e-05,
+            0.006010808050632477,
+            0.027680452913045883,
+            0.002821835223585367,
+            0.006809118203818798,
+            -0.01714446395635605,
+            -0.0009199780761264265,
+            -0.009485806338489056,
+            0.009596801362931728,
+            0.02639974094927311,
+            -0.010612832382321358,
+            -0.031078608706593513,
+            0.022574681788682938,
+            -0.007048184517771006,
+            0.015966208651661873,
+            0.002999000484123826,
+            -0.014062216505408287,
+            -0.0023437028285115957,
+            -0.027680452913045883,
+            0.014454968273639679,
+            -0.017349377274513245,
+            -0.024265222251415253,
+            -0.007231753319501877,
+            -0.01045914739370346,
+            -0.004845359828323126,
+            0.038353051990270615,
+            0.012474133633077145,
+            -0.024572592228651047,
+            0.006817656569182873,
+            -0.003974475897848606,
+            -0.005989462602883577,
+            0.0057119750417768955,
+            0.005976655520498753,
+            0.004098278004676104,
+            0.010945817455649376,
+            -0.0065700518898665905,
+            -0.01320840884000063,
+            -0.03324728086590767,
+            -0.02240392006933689,
+            0.014941639266908169,
+            0.00028068936080671847,
+            -0.01315718051046133,
+            0.028483033180236816,
+            -0.01002370472997427,
+            -0.005575365852564573,
+            0.0071292961947619915,
+            0.00013887719251215458,
+            -0.0022796671837568283,
+            -0.017844585701823235,
+            0.002273263642564416,
+            0.006245605181902647,
+            -0.014950176700949669,
+            0.009844405576586723,
+            0.0032124524004757404,
+            -0.014540349133312702,
+            0.005635132547467947,
+            -0.0015027020126581192,
+            -0.003562513506039977,
+            -0.02481165900826454,
+            0.0059595792554318905,
+            0.00708233704790473,
+            -0.0018036692636087537,
+            0.002390662208199501,
+            -0.018169032409787178,
+            0.01801534742116928,
+            0.001955220242962241,
+            0.0071762558072805405,
+            -0.029166078194975853,
+            0.007282981649041176,
+            -0.006740813609212637,
+            0.0015080382581800222,
+            -0.0195692777633667,
+            0.009161358699202538,
+            0.020798761397600174,
+            0.012491210363805294,
+            -0.0005250918911769986,
+            0.010442070662975311,
+            0.008175211027264595,
+            -0.012704662047326565,
+            0.003950996324419975,
+            -0.02050846628844738,
+            -0.007308595813810825,
+            -0.010057857260107994,
+            0.011372721754014492,
+            -0.004219945520162582,
+            0.015983285382390022,
+            0.009511420503258705,
+            0.005600980017334223,
+            -0.009699258022010326,
+            0.00743666710332036,
+            0.01292665209621191,
+            0.005874198395758867,
+            -0.0011099503608420491,
+            0.017827508971095085,
+            0.07363239675760269,
+            0.005336299538612366,
+            0.013618236407637596,
+            6.870485958643258e-05,
+            -0.013071799650788307,
+            0.0015624684747308493,
+            -0.004939279053360224,
+            -0.01748598739504814,
+            0.026126522570848465,
+            0.026587579399347305,
+            -0.006382214371114969,
+            0.0004917400074191391,
+            0.009016212075948715,
+            0.005558289587497711,
+            0.005485715810209513,
+            0.022216081619262695,
+            -0.0077867284417152405,
+            0.01753721572458744,
+            0.024179840460419655,
+            0.005541213322430849,
+            -0.0019093280425295234,
+            -0.0022305732127279043,
+            0.009152821265161037,
+            0.004209273029118776,
+            -0.009050363674759865,
+            0.01150079257786274,
+            -0.01437812577933073,
+            0.01909114606678486,
+            0.011389797553420067,
+            0.02146473154425621,
+            0.007073798682540655,
+            -0.01704200729727745,
+            -0.018356870859861374,
+            0.0041281613521277905,
+            -0.0013479492627084255,
+            0.0059083509258925915,
+            0.0063224476762115955,
+            0.005477177910506725,
+            -0.011611787602305412,
+            -0.0041793896816670895,
+            0.010356690734624863,
+            -0.002471773885190487,
+            -0.0008500725380145013,
+            0.01757136732339859,
+            -0.01268758624792099,
+            0.01168009266257286,
+            -0.001839956152252853,
+            -0.0009984216885641217,
+            0.009682182222604752,
+            0.0014354645973071456,
+            -0.011825240217149258,
+            -0.004166582599282265,
+            0.0021003675647079945,
+            0.01217530108988285,
+            -0.013259637169539928,
+            -0.01048476155847311,
+            0.011825240217149258,
+            -0.006168762221932411,
+            0.009169897064566612,
+            -0.009579724632203579,
+            -0.0025315405800938606,
+            -0.00039648707024753094,
+            -0.008085560984909534,
+            -0.0014578771078959107,
+            0.007820880971848965,
+            0.0008324627415277064,
+            0.010518914088606834,
+            0.011560559272766113,
+            0.01150079257786274,
+            0.003560379147529602,
+            0.004002224653959274,
+            -0.002964847953990102,
+            -0.0064206356182694435,
+            -0.011432488448917866,
+            8.15119783510454e-05,
+            0.005776010919362307,
+            -0.02242099680006504,
+            -0.00024800453684292734,
+            0.008465506136417389,
+            0.005046004895120859,
+            -0.0007710952777415514,
+            0.020849989727139473,
+            0.001264702994376421,
+            0.007820880971848965,
+            -0.025221485644578934,
+            0.0011238247388973832,
+            0.006890229880809784,
+            0.007718423847109079,
+            0.012798581272363663,
+            -0.028636718168854713,
+            -0.0038528083823621273,
+            -0.0038634808734059334,
+            0.0171273872256279,
+            -0.00011606451880652457,
+            -0.0033106403425335884,
+            0.021686721593141556,
+            -0.0007833687704987824,
+            -0.003430173499509692,
+            -0.00755620002746582,
+            -0.005579635035246611,
+            0.008004449307918549,
+            0.012994957156479359,
+            0.024623820558190346,
+            -0.0019968433771282434,
+            -0.0055967108346521854,
+            -0.00809409935027361,
+            0.0244530588388443,
+            0.015249010175466537,
+            0.01863008923828602,
+            -0.0219087116420269,
+            0.010638446547091007,
+            -0.013865840621292591,
+            0.009067440405488014,
+            0.011714245192706585,
+            -0.010911665856838226,
+            0.0004290384822525084,
+            -0.010757979936897755,
+            -0.017135925590991974,
+            -0.022711291909217834,
+            0.00502039073035121,
+            0.02839765138924122,
+            -0.0002676154254004359,
+            0.007193332072347403,
+            -0.016358960419893265,
+            -0.0018442251021042466,
+            -0.011312955059111118,
+            -0.025750847533345222,
+            -0.016222350299358368,
+            -0.021772101521492004,
+            -0.02141350321471691,
+            -0.0038592119235545397,
+            -0.002131317974999547,
+            -0.01907406933605671,
+            0.00013153978215996176,
+            -0.0012817791430279613,
+            0.01902284100651741,
+            -0.0014888276346027851,
+            0.0015624684747308493,
+            0.012337524443864822,
+            0.012943727895617485,
+            -0.012824195437133312,
+            -0.004228483885526657,
+            -0.019261907786130905,
+            -0.024179840460419655,
+            0.0032487392891198397,
+            -0.007726961746811867,
+            0.013874378986656666,
+            -0.008234977722167969,
+            0.004260501824319363,
+            -0.017187153920531273,
+            0.014702572487294674,
+            -0.03579162806272507,
+            -0.024128612130880356,
+            0.0006531630642712116,
+            -0.0040299734100699425,
+            -0.009605339728295803,
+            -0.02585330419242382,
+            0.01198746357113123,
+            -0.019620506092905998,
+            0.002104636514559388,
+            -0.018442250788211823,
+            0.011398335918784142,
+            -0.015283161774277687,
+            0.018681317567825317,
+            -0.00025520852068439126,
+            0.03179580718278885,
+            0.004405648913234472,
+            0.007978835143148899,
+            -0.004525181837379932,
+            0.003327716374769807,
+            -0.005468640010803938,
+            0.006365138106048107,
+            0.006988417822867632,
+            0.008055677637457848,
+            0.007825150154531002,
+            -0.004768517334014177,
+            0.004061991348862648,
+            0.005110040307044983,
+            0.030515095219016075,
+            0.028021976351737976,
+            0.0027706066612154245,
+            0.0014674824196845293,
+            0.0034685947466641665,
+            0.004576410632580519,
+            -0.00038021133514121175,
+            0.008862526156008244,
+            -0.011287340894341469,
+            -0.005391797050833702,
+            0.01417321152985096,
+            -0.012448519468307495,
+            -0.0037695621140301228,
+            0.015377080999314785,
+            -0.015086785890161991,
+            -0.026092370972037315,
+            0.021823331713676453,
+            -0.006552976090461016,
+            -0.002450428670272231,
+            -0.006689585279673338,
+            0.008115444332361221,
+            -0.005946772173047066,
+            0.004194331355392933,
+            0.028107358142733574,
+            -0.005737589206546545,
+            -0.010945817455649376,
+            0.012636357918381691,
+            -0.002330895746126771,
+            -0.007948951795697212,
+            0.02291620522737503,
+            0.035894084721803665,
+            0.00999809056520462,
+            -0.026041142642498016,
+            -0.007381169591099024,
+            0.02144765481352806,
+            -0.00973341055214405,
+            -0.021755026653409004,
+            0.008273398503661156,
+            0.007522047962993383,
+            -0.005421680398285389,
+            0.018937459215521812,
+            0.004239156376570463,
+            -0.010792132467031479,
+            0.009041826240718365,
+            0.022711291909217834,
+            0.021686721593141556,
+            0.008947907015681267,
+            -0.023548023775219917,
+            0.01025423314422369,
+            -0.026228979229927063,
+            -0.013831689022481441,
+            -0.023821242153644562,
+            0.003581724362447858,
+            0.003857077332213521,
+            -0.003257277188822627,
+            0.026536351069808006,
+            0.011876468546688557,
+            0.011645940132439137,
+            0.0030950538348406553,
+            0.023052813485264778,
+            -0.015223395079374313,
+            -0.028107358142733574,
+            -0.0009733410552144051,
+            -0.015479537658393383,
+            -0.0018282162491232157,
+            0.002166537567973137,
+            0.007150641642510891,
+            0.005852853413671255,
+            0.01953512616455555,
+            0.000376209121895954,
+            0.0019584218971431255,
+            -2.4263486920972355e-05,
+            0.010100548155605793,
+            -0.015727141872048378,
+            -0.019705887883901596,
+            -0.008922292850911617,
+            6.470263178925961e-05,
+            0.011637401767075062,
+            -0.005208228249102831,
+            -3.448583447607234e-05,
+            0.007846495136618614,
+            -0.015897903591394424,
+            -0.0097590247169137,
+            -0.014412278309464455,
+            -0.0029306956566870213,
+            -0.005071619059890509,
+            0.03136890381574631,
+            0.014437892474234104,
+            -0.004747172351926565,
+            -0.011543483473360538,
+            -0.005741858389228582,
+            0.03370833769440651,
+            0.0017204229952767491,
+            0.012422905303537846,
+            0.0146342683583498,
+            0.015684451907873154,
+            0.027014482766389847,
+            0.012807118706405163,
+            0.003018211107701063,
+            0.002516598906368017,
+            0.02346264198422432,
+            -0.013951221480965614,
+            0.008730186149477959,
+            -0.011645940132439137,
+            -0.003521957667544484,
+            -0.010937280021607876,
+            0.015078248456120491,
+            -0.013003494590520859,
+            -0.011278802528977394,
+            0.023069890215992928,
+            -0.0004381102044135332,
+            0.013276712968945503,
+            -0.0004242358263581991,
+            -0.003953130915760994,
+            -0.02534101903438568,
+            0.0015464596217498183,
+            0.0006531630642712116,
+            -0.004401379730552435,
+            0.010578680783510208,
+            -0.027475539594888687,
+            0.0013660927070304751,
+            0.015240471810102463,
+            -0.007116489112377167,
+            0.00038981667603366077,
+            -0.010826284997165203,
+            0.005904081743210554,
+            0.00011606451880652457,
+            -0.02385539375245571,
+            0.015488076023757458,
+            -0.006698123179376125,
+            0.01243144366890192,
+            -0.0018548977095633745,
+            -0.013788998126983643,
+            0.008687496185302734,
+            0.019296059384942055,
+            0.027560919523239136,
+            -0.007825150154531002,
+            0.0005528406472876668,
+            -0.0001855698210420087,
+            -0.006027883850038052,
+            -0.02737308293581009,
+            0.02054261974990368,
+            -0.007863570936024189,
+            0.01953512616455555,
+            -0.009682182222604752,
+            -0.018186109140515327,
+            -0.014856258407235146,
+            0.011927696876227856,
+            0.007603159639984369,
+            0.0010304395109415054,
+            0.01489894837141037,
+            -0.024077383801341057,
+            0.020867066457867622,
+            0.0033383890986442566,
+            -0.0064718639478087425,
+            0.008303281851112843,
+            0.004230618476867676,
+            0.004695943556725979,
+            -0.008384393528103828,
+            0.007658657152205706,
+            0.007569007575511932,
+            -0.007253098301589489,
+            0.004326671827584505,
+            -0.04057295247912407,
+            0.010732365772128105,
+            -0.011090965010225773,
+            0.005878467578440905,
+            -0.004742903169244528,
+            -0.022745443508028984,
+            -0.035825781524181366,
+            0.004567872267216444,
+            0.019313136115670204,
+            -0.020235247910022736,
+            0.018408099189400673,
+            0.012124072760343552,
+            0.014574501663446426,
+            0.01859593763947487,
+            0.008853988721966743,
+            0.01461719162762165,
+            0.0438857264816761,
+            -0.024231068789958954,
+            6.483604374807328e-05,
+            0.008734455332159996,
+            0.011381260119378567,
+            -0.013404784724116325,
+            -0.010126162320375443,
+            0.005784548819065094,
+            -0.0015197781613096595,
+            0.04067540913820267,
+            -0.019210679456591606,
+            0.010886050760746002,
+            -0.010962894186377525,
+            0.008371586911380291,
+            -0.009562648832798004,
+            -0.0036158766597509384,
+            -0.012559514492750168,
+            -0.011893544346094131,
+            0.014745263382792473,
+            -0.019671734422445297,
+            -0.005460101645439863,
+            -0.007282981649041176,
+            0.01048476155847311,
+            -0.006096188444644213,
+            -0.020730456337332726,
+            0.018647165969014168,
+            -0.0018079383298754692,
+            0.029968658462166786,
+            -0.012755890376865864,
+            0.00574612757191062,
+            0.022591758519411087,
+            -0.010604294948279858,
+            0.018732545897364616,
+            -0.004362958483397961,
+            0.008853988721966743,
+            0.0055625587701797485,
+            -0.0065188235603272915,
+            -0.0014685495989397168,
+            0.01389145478606224,
+            -0.0027385889552533627,
+            0.029934506863355637,
+            -0.017400605604052544,
+            -0.009818791411817074,
+            0.009707796387374401,
+            0.0073769004084169865,
+            0.0042370217852294445,
+            0.003878422547131777,
+            0.009579724632203579,
+            -0.01071528997272253,
+            0.0009728074073791504,
+            -0.021686721593141556,
+            -0.015453923493623734,
+            -0.0011483717244118452,
+            -0.004683136474341154,
+            -0.0018730411538854241,
+            -0.002467504935339093,
+            -0.02044016122817993,
+            0.004627638962119818,
+            -0.0021579996682703495,
+            -8.451364556094632e-05,
+            -0.024487212300300598,
+            -0.00769280968233943,
+            -0.010057857260107994,
+            -0.005161269102245569,
+            -0.003139878623187542,
+            0.018271489068865776,
+            0.0034856710117310286,
+            -0.004862436093389988,
+            0.0293197650462389,
+            0.005404604133218527,
+            0.01536854263395071,
+            0.00280262460000813,
+            -0.05334591865539551,
+            -0.011014122515916824,
+            -0.004563603550195694,
+            -0.009912709705531597,
+            -0.007261636666953564,
+            -0.008456967771053314,
+            -0.010681137442588806,
+            -0.0019061262719333172,
+            0.0032914294861257076,
+            -0.004170851781964302,
+            0.0122521435841918,
+            -0.01700785383582115,
+            0.025494705885648727,
+            -0.004017166327685118,
+            0.015744218602776527,
+            -0.011722782626748085,
+            0.011654478497803211,
+            -0.006194376386702061,
+            -0.010851899161934853,
+            0.010544528253376484,
+            0.0146342683583498,
+            -0.0030544979963451624,
+            0.020371858030557632,
+            -0.014736725017428398,
+            0.003197510726749897,
+            -0.008303281851112843,
+            -0.02143057994544506,
+            -0.015991821885108948,
+            0.011295879259705544,
+            0.03130060061812401,
+            0.010066395625472069,
+            -0.0037524858489632607,
+            0.004072663839906454,
+            0.021669644862413406,
+            -0.004352285992354155,
+            0.0063181789591908455,
+            0.03025895357131958,
+            -0.0010982104577124119,
+            0.017742129042744637,
+            0.0001935742620844394,
+            -0.005028928630053997,
+            -0.002362913452088833,
+            0.01808365248143673,
+            -0.002766337711364031,
+            0.0057119750417768955,
+            0.02441890724003315,
+            0.010997045785188675,
+            -0.0064718639478087425,
+            0.008230708539485931,
+            0.014309820719063282,
+            0.013874378986656666,
+            -0.015351466834545135,
+            -0.0122009152546525,
+            -0.011534945107996464,
+            -0.015778370201587677,
+            0.007291519548743963,
+            -0.019261907786130905,
+            -0.004049184266477823,
+            0.007829418405890465,
+            -0.015138015151023865,
+            -0.012807118706405163,
+            -0.023548023775219917,
+            -0.015914980322122574,
+            0.01863008923828602,
+            0.005357644986361265,
+            -0.01465988252311945,
+            -0.004369362257421017,
+            0.0013714289525523782,
+            0.006181569304317236,
+            -0.0054985228925943375,
+            0.021840406581759453,
+            -0.004098278004676104,
+            -0.011125117540359497,
+            0.019330210983753204,
+            0.0017897948855534196,
+            0.005097233224660158,
+            -0.0064206356182694435,
+            0.004550796467810869,
+            -0.011364183388650417,
+            0.022096550092101097,
+            -0.003470729338005185,
+            0.012636357918381691,
+            -0.0037033918779343367,
+            0.004631908144801855,
+            -0.001978699816390872,
+            0.00821790099143982,
+            0.004017166327685118,
+            0.010629909113049507,
+            -0.005656477529555559,
+            0.004525181837379932,
+            0.012491210363805294,
+            -0.01641872711479664,
+            0.007265905383974314,
+            0.004559334367513657,
+            -4.365760105429217e-05,
+            0.005780279636383057,
+            -0.007645850069820881,
+            -0.005417411215603352,
+            -0.015488076023757458,
+            0.010100548155605793,
+            0.030873695388436317,
+            -0.0016649254830554128,
+            -0.007240291219204664,
+            0.01704200729727745,
+            0.015735680237412453,
+            -0.003816521493718028,
+            0.0061218030750751495,
+            0.009280892089009285,
+            -0.003148416755720973,
+            0.01370361726731062,
+            -0.024487212300300598,
+            -0.029524678364396095,
+            0.01763967238366604,
+            -0.020867066457867622,
+            0.022779595106840134,
+            -0.019210679456591606,
+            -0.027731681242585182,
+            -0.01847640424966812,
+            -0.0032530082389712334,
+            -0.019842496141791344,
+            -0.01611989364027977,
+            0.02532394416630268,
+            0.00685180863365531,
+            -0.006949996575713158,
+            -0.031112760305404663,
+            0.04050464928150177,
+            0.014489120803773403,
+            0.038318898528814316,
+            -0.011603250168263912,
+            -2.554753518779762e-05,
+            -0.007504971697926521,
+            0.012064306065440178,
+            -0.01146664097905159,
+            -0.014258592389523983,
+            0.007509240880608559,
+            0.028175661340355873,
+            0.015231933444738388,
+            -0.015223395079374313,
+            -0.007944682613015175,
+            0.015991821885108948,
+            -0.007043915335088968,
+            -0.0005480379913933575,
+            -0.008836911991238594,
+            -0.014830644242465496,
+            -0.010066395625472069,
+            -0.0005312286084517837,
+            0.012243606150150299,
+            0.016179660335183144,
+            -0.0007257367251440883,
+            0.01390853151679039,
+            -0.009955400601029396,
+            -0.014420815743505955,
+            -0.004602024797350168,
+            0.005327761638909578,
+            -0.02252345345914364,
+            0.02105490304529667,
+            0.002292474266141653,
+            -0.009537034668028355,
+            0.0030865157023072243,
+            -0.004572141449898481,
+            0.012602205388247967,
+            0.009391887113451958,
+            -0.0038421356584876776,
+            -0.0036137420684099197,
+            0.00572051340714097,
+            -0.001715086749754846,
+            0.015001405030488968,
+            -0.011714245192706585,
+            0.011842316016554832,
+            -0.007432397920638323,
+            -0.00019931078713852912,
+            -0.009648029692471027,
+            -0.005238111596554518,
+            -0.010493299923837185,
+            0.0028794673271477222,
+            0.012943727895617485,
+            -0.014548887498676777,
+            0.012807118706405163,
+            -0.0008132520597428083,
+            -0.00949434470385313,
+            -0.015138015151023865,
+            -0.0055155991576612,
+            -0.011364183388650417,
+            -0.016683407127857208,
+            0.007069529499858618,
+            0.0007855033036321402,
+            0.019347287714481354,
+            0.0030054040253162384,
+            -0.004572141449898481,
+            -0.00017262929759453982,
+            0.018903307616710663,
+            -0.02580207586288452,
+            0.03403278440237045,
+            -0.00010045584349427372,
+            0.022711291909217834,
+            0.04251963645219803,
+            0.00954557303339243,
+            -0.006663971114903688,
+            -0.013754845596849918,
+            0.005801625084131956,
+            -0.013609698973596096,
+            0.009981014765799046,
+            0.014335434883832932,
+            -0.03235932067036629,
+            -0.003944592550396919,
+            -0.013976835645735264,
+            0.011586173437535763,
+            -0.007957490161061287,
+            0.0032850259449332952,
+            -0.010587218217551708,
+            0.011765473522245884,
+            0.01679440215229988,
+            -0.009075978770852089,
+            -0.004311730153858662,
+            -0.027492616325616837,
+            0.00032524747075513005,
+            0.016674868762493134,
+            -0.010911665856838226,
+            -0.01684563048183918,
+            0.010945817455649376,
+            -0.007218946237117052,
+            -0.015453923493623734,
+            -0.006898768246173859,
+            -0.03278622403740883,
+            -0.00020504731219261885,
+            -0.01076651830226183,
+            -0.003927516285330057,
+            -0.0028794673271477222,
+            0.0011024795239791274,
+            -0.0033917520195245743,
+            -0.0067066615447402,
+            -0.0025891726836562157,
+            0.0067664277739822865,
+            0.005011852364987135,
+            0.002080089645460248,
+            0.0005987327895127237,
+            -0.00023506399884354323,
+            0.009255277924239635,
+            0.010168852284550667,
+            -0.007419590838253498,
+            -0.017212767153978348,
+            0.010834822431206703,
+            -0.020730456337332726,
+            0.005938234273344278,
+            0.00783795677125454,
+            0.011039736680686474,
+            -0.0220111683011055,
+            -0.010868974961340427,
+            0.009818791411817074,
+            0.003835732117295265,
+            0.01902284100651741,
+            0.010886050760746002,
+            0.0022455148864537477,
+            -0.0035646480973809958,
+            0.0028495839796960354,
+            -1.0947752343781758e-05,
+            -0.0012508286163210869,
+            0.015308775939047337,
+            0.0006504949415102601,
+            0.02974666841328144,
+            -0.012055767700076103,
+            0.011201960034668446,
+            -0.009340658783912659,
+            0.0013842361513525248,
+            -0.009741948917508125,
+            -0.001973363570868969,
+            -0.004194331355392933,
+            -0.006484671495854855,
+            0.013148642145097256,
+            0.007714154664427042,
+            -0.004239156376570463,
+            -0.01416467409580946,
+            0.01194477267563343,
+            -0.023035738617181778,
+            -0.012269220314919949,
+            -0.00881983619183302,
+            0.012935190461575985,
+            -0.0053960662335157394,
+            0.01170570682734251,
+            0.03457922115921974,
+            -0.0018559650052338839,
+            0.0007860369514673948,
+            -0.013746308162808418,
+            -0.008311820216476917,
+            -0.02728770114481449,
+            -0.01510386262089014,
+            -0.008922292850911617,
+            -0.03454506769776344,
+            -0.011919158510863781,
+            0.008239246904850006,
+            0.008111175149679184,
+            0.0014237246941775084,
+            -0.019176525995135307,
+            -0.008888140320777893,
+            -0.01492456253618002,
+            -0.017460372298955917,
+            -0.00924673955887556,
+            -0.01175693515688181,
+            -0.0037119300104677677,
+            0.020884141325950623,
+            -0.004162313882261515,
+            0.014267130754888058,
+            0.016939548775553703,
+            1.3132299500284716e-05,
+            0.015334390103816986,
+            0.009665105491876602,
+            -0.009127207100391388,
+            -0.004725826904177666,
+            0.006885961163789034,
+            0.0027706066612154245,
+            0.018749622628092766,
+            -0.006702392362058163,
+            -0.014796491712331772,
+            0.004298923071473837,
+            0.005571096669882536,
+            -0.012346062809228897,
+            0.010655523277819157,
+            -0.00476424815133214,
+            0.02054261974990368,
+            -0.018254414200782776,
+            -0.011885006912052631,
+            0.014472044073045254,
+            -0.008862526156008244,
+            0.02394077368080616,
+            0.013951221480965614,
+            -0.015291700139641762,
+            0.007167717441916466,
+            0.013029108755290508,
+            0.007615966722369194,
+            0.0018495614640414715,
+            0.022147778421640396,
+            -0.0011846584966406226,
+            0.0031590894795954227,
+            0.010237157344818115,
+            -0.018835002556443214,
+            -0.02042308636009693,
+            -0.003118533408269286,
+            0.02247222512960434,
+            0.007961759343743324,
+            -0.016222350299358368,
+            0.0037930416874587536,
+            0.012849809601902962,
+            0.012072844430804253,
+            0.0039787450805306435,
+            -0.001204936415888369,
+            -0.02248930186033249,
+            0.0035155541263520718,
+            0.0001910395221784711,
+            0.018664240837097168,
+            -0.011116579174995422,
+            0.001461078878492117,
+            -0.011389797553420067,
+            0.012747352942824364,
+            -0.005221035331487656,
+            0.01193623524159193,
+            0.0011761204805225134,
+            -0.005353375803679228,
+            0.0001780989987310022,
+            0.014941639266908169,
+            0.014070754870772362,
+            0.037738312035799026,
+            -0.005916888825595379,
+            -0.006480402313172817,
+            0.017170077189803123,
+            -0.012388752773404121,
+            0.018715469166636467,
+            0.006497478578239679,
+            0.0170249305665493,
+            -0.011090965010225773,
+            0.0011344973463565111,
+            0.04115354269742966,
+            0.015949131920933723,
+            -6.583659705938771e-05,
+            -0.008760069496929646,
+            -0.00805140845477581,
+            0.0244530588388443,
+            0.009067440405488014,
+            -2.7882166250492446e-05,
+            0.015744218602776527,
+            0.006096188444644213,
+            -0.016324806958436966,
+            -0.012918113730847836,
+            0.03555256128311157,
+            -0.007718423847109079,
+            -0.022045321762561798,
+            -0.0009311842732131481,
+            -0.009383349679410458,
+            -0.02884163148701191,
+            0.013959759846329689,
+            0.013788998126983643,
+            -0.008162403479218483,
+            0.006992687005549669,
+            0.008439891040325165,
+            -0.004117488861083984,
+            -0.004435532260686159,
+            0.0005346972029656172,
+            -0.0007049251580610871,
+            0.006177300587296486,
+            0.005092964507639408,
+            -0.014463506639003754,
+            -0.006787773221731186,
+            -0.005669284611940384,
+            -0.004064125940203667,
+            0.009016212075948715,
+            0.005058811977505684,
+            0.004870974458754063,
+            -0.008482581935822964,
+            0.018339794129133224,
+            0.02298451028764248,
+            -0.0122009152546525,
+            0.012115534394979477,
+            0.005233842879533768,
+            -0.0146428057923913,
+            -0.004614831879734993,
+            -0.01121903583407402,
+            -0.026570502668619156,
+            -0.006113264709711075,
+            0.0033490615896880627,
+            0.015428309328854084,
+            -0.022796671837568283,
+            0.0008500725380145013,
+            0.007790997624397278,
+            2.7365211280994117e-05,
+            -0.008879602886736393,
+            0.007573276292532682,
+            -0.00044157879892736673,
+            0.022301463410258293,
+            0.019808344542980194,
+            0.025631314143538475,
+            -0.008414276875555515,
+            -0.00499904528260231,
+            0.014301283285021782,
+            0.00433947890996933,
+            0.009374811314046383,
+            0.027543844655156136,
+            0.022164853289723396,
+            -0.013857303187251091,
+            0.006429173983633518,
+            0.000640355923678726,
+            -0.014250054024159908,
+            0.0066682398319244385,
+            0.013054722920060158,
+            0.012414366938173771,
+            0.0028410458471626043,
+            -0.0013415457215160131,
+            -0.015957670286297798,
+            0.004824014846235514,
+            0.018408099189400673,
+            -0.017118848860263824,
+            -0.03471583127975464,
+            -0.011842316016554832,
+            -0.012158225290477276,
+            -0.00624133599922061,
+            -0.012875423766672611,
+            0.01440373994410038,
+            0.004418455995619297,
+            0.019705887883901596,
+            0.0002309283590875566,
+            -0.01022861897945404,
+            0.014250054024159908,
+            0.005110040307044983,
+            -0.0016638581873849034,
+            0.0005352308508008718,
+            0.007231753319501877,
+            -0.0030331527814269066,
+            -0.0022476494777947664,
+            -0.00027055040118284523,
+            0.016239427030086517,
+            -0.02532394416630268,
+            0.008947907015681267,
+            -0.017067620530724525,
+            0.0017951311310753226,
+            0.015249010175466537,
+            -0.003775965655222535,
+            -0.016025975346565247,
+            0.0011216901475563645,
+            -0.008619191125035286,
+            -0.07288104295730591,
+            -0.005882736761122942,
+            0.004512374754995108,
+            0.023992003872990608,
+            -0.013336479663848877,
+            -0.0024824466090649366,
+            0.0018474268727004528,
+            0.008657612837851048,
+            -0.010134699754416943,
+            0.003995821345597506,
+            0.010390842333436012,
+            -0.003319178242236376,
+            -0.009221125394105911,
+            0.002508060773834586,
+            -0.008106905966997147,
+            -0.0038805571384727955,
+            -0.00783795677125454,
+            0.003402424743399024,
+            0.025221485644578934,
+            -0.010868974961340427,
+            0.024657974019646645,
+            0.009699258022010326,
+            -0.0075433929450809956,
+            -0.0027428579051047564,
+            -0.007761114276945591,
+            -0.027543844655156136,
+            -0.012397291138768196,
+            -0.003396020969375968,
+            -0.01292665209621191,
+            0.0029264267068356276,
+            0.001692674239166081,
+            -0.007167717441916466,
+            -0.004348016809672117,
+            0.0021547977812588215,
+            0.00030336863710545003,
+            -0.022147778421640396,
+            -0.0009407896432094276,
+            -0.014267130754888058,
+            0.02105490304529667,
+            -0.012371676973998547,
+            -0.004461146425455809,
+            0.0030544979963451624,
+            -0.010109085589647293,
+            0.004580679349601269,
+            0.019313136115670204,
+            -0.007308595813810825,
+            -0.009682182222604752,
+            0.014045140706002712,
+            -0.003690584795549512,
+            0.00185916677583009,
+            -0.01798119582235813,
+            0.010032243095338345,
+            -0.03232516720890999,
+            -0.003583858720958233,
+            -0.009981014765799046,
+            -0.0031847036443650723,
+            -0.0067151994444429874,
+            -0.03162504732608795,
+            0.015138015151023865,
+            -0.007039646618068218,
+            -0.02346264198422432,
+            -0.0011793222511187196,
+            -0.01859593763947487,
+            0.01419028826057911,
+            -0.01487333420664072,
+            0.01393414568156004,
+            -0.020645076408982277,
+            0.012533900327980518,
+            0.012798581272363663,
+            0.021123208105564117,
+            0.006493209395557642,
+            -0.005302147474139929,
+            0.002676687901839614,
+            0.0018730411538854241,
+            0.014352511614561081,
+            -0.008905217051506042,
+            0.00462336977943778,
+            -0.002390662208199501,
+            0.005737589206546545,
+            0.009426039643585682,
+            0.02107197977602482,
+            -0.0005896611255593598,
+            -0.0037823691964149475,
+            -0.017468910664319992,
+            0.00757754547521472,
+            -0.01559907104820013,
+            0.005485715810209513,
+            0.0025251370389014482,
+            0.023787088692188263,
+            0.010886050760746002,
+            0.0035859933122992516,
+            0.004900857340544462,
+            0.004520913120359182,
+            0.01861301250755787,
+            -0.006890229880809784,
+            -0.004747172351926565,
+            -0.018186109140515327,
+            0.019688811153173447,
+            0.007705616764724255,
+            -0.005306416191160679,
+            -0.011902082711458206,
+            0.006010808050632477,
+            0.002134519862011075,
+            0.0017577770631760359,
+            0.02288205362856388,
+            -0.02974666841328144,
+            -0.008964983746409416,
+            -0.007808073423802853,
+            -0.009682182222604752,
+            -0.017844585701823235,
+            0.017084697261452675,
+            -0.008012987673282623,
+            0.00461910106241703,
+            -0.012790042906999588,
+            0.033418044447898865,
+            0.002969117136672139,
+            0.00513992365449667,
+            0.013729231432080269,
+            -0.00024373548512812704,
+            0.017844585701823235,
+            -0.008367317728698254,
+            -0.04036803916096687,
+            -0.0031911071855574846,
+            0.0034515184815973043,
+            0.007645850069820881,
+            -0.010903127491474152,
+            0.004862436093389988,
+            -0.006339523941278458,
+            -0.01854470930993557,
+            -0.014232978224754333,
+            0.037840768694877625,
+            -0.005225304514169693,
+            0.017947042360901833,
+            -0.016145508736371994,
+            -0.0033682722132653,
+            0.007107951212674379,
+            0.022762520238757133,
+            -0.012602205388247967,
+            0.00791053008288145,
+            0.009571187198162079,
+            0.02199409157037735,
+            0.00978463888168335,
+            -0.01565883681178093,
+            0.023735860362648964,
+            0.005741858389228582,
+            -0.0023565099108964205,
+            0.005276532843708992,
+            -0.0030352871399372816,
+            0.016000360250473022,
+            -0.014813567511737347,
+            -0.011876468546688557,
+            0.002303146990016103,
+            -0.010168852284550667,
+            0.0037055264692753553,
+            -0.024589668959379196,
+            0.00037540867924690247,
+            -0.009844405576586723,
+            0.021089056506752968,
+            0.01027130987495184,
+            0.00877714529633522,
+            -0.019637582823634148,
+            -0.0048026698641479015,
+            -0.02645096927881241,
+            -0.014574501663446426,
+            -0.01022861897945404,
+            -0.0011451698374003172,
+            -0.004431263078004122,
+            -0.007765382993966341,
+            -0.03133475035429001,
+            -0.004202869720757008,
+            0.013541393913328648,
+            0.014975790865719318,
+            -0.0013714289525523782,
+            0.012089920230209827,
+            -0.00019544197130016983,
+            -0.008106905966997147,
+            -0.02236976847052574,
+            -0.010407919064164162,
+            -0.02091829478740692,
+            -0.03123229369521141,
+            -0.022199006751179695,
+            0.008670419454574585,
+            -0.012021616101264954,
+            -0.002941368380561471,
+            0.028961164876818657,
+            -0.022130701690912247,
+            0.007513510063290596,
+            0.005400335416197777,
+            0.0028666602447628975,
+            0.014856258407235146,
+            0.0054472945630550385,
+            0.0003634020104072988,
+            0.0037652929313480854,
+            -0.008341703563928604,
+            -0.008081291802227497,
+            0.01799827069044113,
+            -0.003331985557451844,
+            0.005097233224660158,
+            0.010809208266437054,
+            -0.006749351508915424,
+            -0.005438756663352251,
+            0.02392369881272316,
+            -0.015718603506684303,
+            0.002044870052486658,
+            -0.008610652759671211,
+            -0.014079293236136436,
+            -0.00949434470385313,
+            -0.005793086718767881,
+            -0.01813488081097603,
+            -0.015240471810102463,
+            0.01689685881137848,
+            -0.0004028906114399433,
+            0.016025975346565247,
+            -0.005289340391755104,
+            -0.005007583647966385,
+            0.014437892474234104,
+            -0.03558671474456787,
+            -0.003402424743399024,
+            0.006377945188432932,
+            -0.026211904361844063,
+            0.02484581060707569,
+            0.005276532843708992,
+            0.01605158858001232,
+            -0.013780459761619568,
+            0.0026318628806620836,
+            -0.006365138106048107,
+            0.004358689300715923,
+            0.018903307616710663,
+            0.007308595813810825,
+            0.004858167376369238,
+            -0.006689585279673338,
+            -0.008047140203416348,
+            0.016077203676104546,
+            -0.007415322121232748,
+            0.0018719738582149148,
+            0.016785863786935806,
+            0.018425175920128822,
+            0.012004539370536804,
+            0.0059553105384111404,
+            0.013071799650788307,
+            0.01750306226313114,
+            -0.007043915335088968,
+            -0.005835777148604393,
+            -0.013481627218425274,
+            -0.00802579429000616,
+            -0.0013105951948091388,
+            0.03859211876988411,
+            0.018920384347438812,
+            0.007385438773781061,
+            -0.02054261974990368,
+            0.013456013053655624,
+            0.0043928418308496475,
+            6.841302820248529e-06,
+            -0.0027385889552533627,
+            0.02293328195810318,
+            0.0024952536914497614,
+            0.007125027012079954,
+            0.006258412264287472,
+            -0.017212767153978348,
+            0.03309359401464462,
+            -0.020713381469249725,
+            0.0008532743086107075,
+            -0.013319403864443302,
+            -0.012166762724518776,
+            0.015530765987932682,
+            0.002104636514559388,
+            -0.004164448007941246,
+            -0.006296833511441946,
+            -0.0033575997222214937,
+            0.013515779748558998,
+            0.008499657735228539,
+            -0.012875423766672611,
+            0.004516643937677145,
+            -0.02542640082538128,
+            0.004315999336540699,
+            -0.015334390103816986,
+            0.021755026653409004,
+            -0.004559334367513657,
+            -0.004674598574638367,
+            -0.019261907786130905,
+            -0.017067620530724525,
+            -0.00999809056520462,
+            0.002403469290584326,
+            -0.004755710251629353,
+            0.004422725178301334,
+            -0.009468729607760906,
+            -0.008499657735228539,
+            0.0027599341701716185,
+            0.0036628360394388437,
+            0.012465596199035645,
+            0.013669464737176895,
+            0.019893724471330643,
+            0.017759205773472786,
+            -0.01536000519990921,
+            -0.0341864712536335,
+            0.0006056699785403907,
+            0.005084426142275333,
+            -0.0036436254158616066,
+            0.018664240837097168,
+            -0.001938143977895379,
+            -0.0009968207450583577,
+            -0.0007150641758926213,
+            0.0005419012159109116,
+            0.02004740945994854,
+            -0.011014122515916824,
+            -0.016657792031764984,
+            0.00021545309573411942,
+            0.008853988721966743,
+            0.00502039073035121,
+            -0.0025934416335076094,
+            -0.010296924039721489,
+            -0.0009455922991037369,
+            0.0020341973286122084,
+            0.006032153032720089,
+            -0.009101592935621738,
+            -0.0010496501345187426,
+            -0.0013671600027009845,
+            -0.0030544979963451624,
+            -0.006254143081605434,
+            -0.009844405576586723,
+            0.008222170174121857,
+            -0.012602205388247967,
+            -0.01123611256480217,
+            0.016222350299358368,
+            -0.015086785890161991,
+            0.001136631821282208,
+            0.023821242153644562,
+            0.022557605057954788,
+            -0.0008842248353175819,
+            0.006774966139346361,
+            -0.007936145178973675,
+            -0.011517869308590889,
+            -0.02785121463239193,
+            -0.019637582823634148,
+            -0.00806848518550396,
+            0.017947042360901833,
+            -0.002053408185020089,
+            -0.004049184266477823,
+            -0.0018869155319407582,
+            -0.008580769412219524,
+            0.008444160223007202,
+            -0.013814612291753292,
+            0.011620325967669487,
+            -0.034425534307956696,
+            0.013088875450193882,
+            -0.006702392362058163,
+            0.0010485828388482332,
+            0.015009943395853043,
+            0.04351005330681801,
+            -0.007351286243647337,
+            0.0029541754629462957,
+            0.001540056080557406,
+            0.01389145478606224,
+            0.028209814801812172,
+            0.004369362257421017,
+            0.0006136744632385671,
+            -0.01765674725174904,
+            0.004091874696314335,
+            0.002264725510030985,
+            -0.013456013053655624,
+            0.009579724632203579,
+            -0.023035738617181778,
+            -0.023155272006988525,
+            -0.009067440405488014,
+            -0.021123208105564117,
+            -0.002608383307233453,
+            0.002011785050854087,
+            0.012883962132036686,
+            -0.009878558106720448,
+            -0.012098458595573902,
+            -0.0005656477296724916,
+            -0.032991137355566025,
+            0.016709022223949432,
+            0.09665105491876602,
+            0.00875153113156557,
+            -0.003908305894583464,
+            -0.0021515958942472935,
+            0.002499522641301155,
+            0.009562648832798004,
+            -0.004768517334014177,
+            0.055360905826091766,
+            -0.018852079287171364,
+            0.008311820216476917,
+            -0.0008105839369818568,
+            0.008064216002821922,
+            -0.007223214954137802,
+            0.015530765987932682,
+            -0.012209453620016575,
+            -0.005951041355729103,
+            -0.009460192173719406,
+            -0.01123611256480217,
+            -0.0017246920615434647,
+            -0.016640717163681984,
+            -0.004986238200217485,
+            -0.020645076408982277,
+            0.013336479663848877,
+            -0.03234224393963814,
+            -0.011509330943226814,
+            -0.012474133633077145,
+            -0.0063224476762115955,
+            0.006066305562853813,
+            -0.015684451907873154,
+            -0.002595576224848628,
+            -0.002099300269037485,
+            0.015402695164084435,
+            0.0013863706262782216,
+            -0.00232235761359334,
+            -0.008328896015882492,
+            0.0005405671545304358,
+            0.005024659913033247,
+            0.003940323833376169,
+            0.006557244807481766,
+            0.006792041938751936,
+            -0.005848584230989218,
+            -0.008811297826468945,
+            0.010655523277819157,
+            -0.002510195365175605,
+            -0.0027919518761336803,
+            -0.008871064521372318,
+            0.010382304899394512,
+            -0.013481627218425274,
+            0.018356870859861374,
+            -0.013840226456522942,
+            -0.029507601633667946,
+            -0.015547842718660831,
+            0.02100367471575737,
+            -0.01996202953159809,
+            -0.0039381892420351505,
+            0.010553065687417984,
+            0.02829519473016262,
+            0.004113219678401947,
+            -0.001529383473098278,
+            -0.004909395705908537,
+            -0.017204230651259422,
+            0.009511420503258705,
+            0.008196556009352207,
+            0.018442250788211823,
+            -0.012371676973998547,
+            0.016487032175064087,
+            -0.018288565799593925,
+            0.006177300587296486,
+            -0.01663217879831791,
+            0.0006083381595090032,
+            0.00011866596469189972,
+            -0.0041793896816670895,
+            0.01608574204146862,
+            -0.01534292846918106,
+            -0.026263132691383362,
+            0.01900576427578926,
+            -0.013635313138365746,
+            -0.03307652100920677,
+            0.003374675754457712,
+            -0.034937821328639984,
+            -0.0025827689096331596,
+            0.009716334752738476,
+            -0.02149888314306736,
+            -0.018305642530322075,
+            0.007487895432859659,
+            -0.010066395625472069,
+            -0.008871064521372318,
+            0.00502039073035121,
+            -0.01689685881137848,
+            -0.012149686925113201,
+            -0.001152640674263239,
+            0.006753620691597462,
+            -0.00499477656558156,
+            0.0057632033713161945,
+            -0.026075294241309166,
+            0.014010988175868988,
+            -0.0170249305665493,
+            0.01144102681428194,
+            -0.010587218217551708,
+            0.022762520238757133,
+            0.011748396791517735,
+            -0.010356690734624863,
+            -0.0008303282083943486,
+            0.006036422215402126,
+            0.0005264259525574744,
+            0.026126522570848465,
+            0.03237639740109444,
+            -0.0011889275629073381,
+            0.008469774387776852,
+            0.007586083374917507,
+            0.003921112976968288,
+            -0.011014122515916824,
+            0.0015923518221825361,
+            0.014830644242465496,
+            0.01992787793278694,
+            -0.013729231432080269,
+            0.008879602886736393,
+            -0.026672959327697754,
+            0.02103782817721367,
+            -0.022096550092101097,
+            -0.000982412719167769,
+            0.010066395625472069,
+            -0.004777055233716965,
+            0.007090874947607517,
+            0.0066682398319244385,
+            0.021362274885177612,
+            0.03375956416130066,
+            0.00807702261954546,
+            0.008879602886736393,
+            0.007257367484271526,
+            -0.004798400681465864,
+            0.002044870052486658,
+            0.009135744534432888,
+            -0.005869929678738117,
+            -0.0040940092876553535,
+            -0.010083471424877644,
+            -0.006685316096991301,
+            0.016205275431275368,
+            0.0008079157560132444,
+            0.012602205388247967,
+            0.007018301170319319,
+            -0.007428129203617573,
+            -0.005669284611940384,
+            0.007014031987637281,
+            -0.023787088692188263,
+            -0.00950288213789463,
+            0.025221485644578934,
+            -0.013814612291753292,
+            -0.043202683329582214,
+            -0.0024333526380360126,
+            0.008853988721966743,
+            -0.017964119091629982,
+            -0.01245705783367157,
+            -0.004772786516696215,
+            0.010117623955011368,
+            0.0007241358398459852,
+            -0.007569007575511932,
+            0.028175661340355873,
+            0.00502039073035121,
+            -0.005169807001948357,
+            0.003571051638573408,
+            -2.8232359909452498e-05,
+            0.019244831055402756,
+            0.01536000519990921,
+            0.008294744417071342,
+            -0.03266669064760208,
+            -0.007304327096790075,
+            -0.005780279636383057,
+            0.013114489614963531,
+            0.002813297091051936,
+            0.003423769725486636,
+            -0.019740039482712746,
+            -0.020798761397600174,
+            0.019210679456591606,
+            0.004935009870678186,
+            0.019825419411063194,
+            -0.00867895781993866,
+            -0.006548706907778978,
+            -0.004841091111302376,
+            -0.004742903169244528,
+            0.00499477656558156,
+            0.0076415808871388435,
+            0.0038933642208576202,
+            0.03377664089202881,
+            -0.001280711847357452,
+            0.004042780492454767,
+            -0.0036350872833281755,
+            -0.0002441357064526528,
+            0.014787953346967697,
+            -0.00790199264883995,
+            0.008905217051506042,
+            0.018800850957632065,
+            -0.02887578494846821,
+            0.011398335918784142,
+            -0.01679440215229988,
+            -0.00828193686902523,
+            0.016000360250473022,
+            -0.0005320290802046657,
+            -0.004922202788293362,
+            -0.014583039097487926,
+            0.0008127184119075537,
+            0.007752575911581516,
+            0.01996202953159809,
+            -0.004091874696314335,
+            0.011287340894341469,
+            -0.009648029692471027,
+            -0.004305326379835606,
+            0.010083471424877644,
+            0.006087650544941425,
+            -0.0033213128335773945,
+            0.006232798099517822
+          ],
+          "keyphrases": [
+            "MoE models",
+            "expert choice",
+            "training efficiency",
+            "fine-tuning on GLUE and SuperGLUE",
+            "capacity factor"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "abae7e30-c027-4320-907f-db68716624b9",
+      "type": "next",
+      "source": {
+        "id": "33412f87-7418-4ce2-9116-377a27fd2732",
+        "properties": {
+          "page_content": "4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n",
+          "title": "4 Experiments",
+          "embedding": [
+            0.009938324801623821,
+            -0.009144282899796963,
+            -0.026211904361844063,
+            -0.022352691739797592,
+            0.016043050214648247,
+            0.016247965395450592,
+            -0.02783413790166378,
+            0.0537557490170002,
+            -0.02827811799943447,
+            -0.01755429059267044,
+            0.010237157344818115,
+            0.001297788112424314,
+            0.014762339182198048,
+            -0.01001516729593277,
+            0.007487895432859659,
+            0.015283161774277687,
+            0.019347287714481354,
+            0.019910801202058792,
+            0.02535809576511383,
+            -0.04251963645219803,
+            0.0020213902462273836,
+            -0.032034873962402344,
+            -0.058332156389951706,
+            0.0034600566141307354,
+            0.001006959704682231,
+            0.01755429059267044,
+            0.011645940132439137,
+            -0.0015795447397977114,
+            -0.053584985435009,
+            0.04794985428452492,
+            0.016555335372686386,
+            0.0044910297729074955,
+            -0.007833687588572502,
+            -0.004798400681465864,
+            -0.04204150289297104,
+            -0.028961164876818657,
+            0.05720513314008713,
+            0.005165537819266319,
+            -0.005519868340343237,
+            0.05481446906924248,
+            0.020252324640750885,
+            0.010245694778859615,
+            -0.03961668908596039,
+            -0.009904172271490097,
+            0.010134699754416943,
+            0.001490962109528482,
+            -0.0010427129454910755,
+            -0.007479357533156872,
+            0.02295035682618618,
+            -0.04146091267466545,
+            -0.005549751687794924,
+            -0.03811398521065712,
+            0.01863008923828602,
+            0.013780459761619568,
+            -0.04426140338182449,
+            0.023223575204610825,
+            -0.0052978782914578915,
+            0.03527934476733208,
+            0.057854026556015015,
+            0.02783413790166378,
+            0.024162763729691505,
+            -0.02590453252196312,
+            -0.04340759664773941,
+            -0.0036180110182613134,
+            0.010143238119781017,
+            0.008119713515043259,
+            -0.01344747468829155,
+            -0.010527451522648335,
+            0.046173933893442154,
+            -0.023701708763837814,
+            -0.05327761545777321,
+            0.027680452913045883,
+            -0.013584083877503872,
+            -0.008081291802227497,
+            0.0036372218746691942,
+            -0.03517688810825348,
+            0.012807118706405163,
+            0.03548425808548927,
+            0.01004078146070242,
+            0.015018481761217117,
+            0.001224147155880928,
+            0.008418546058237553,
+            -0.0014920294051989913,
+            0.017912890762090683,
+            0.028517184779047966,
+            -0.01417321152985096,
+            -0.006309640593826771,
+            -0.0686461552977562,
+            0.025767924264073372,
+            -0.007368362508714199,
+            -0.020115714520215988,
+            0.00026281276950612664,
+            -0.007099412847310305,
+            0.0033597343135625124,
+            -0.020781684666872025,
+            0.07472527027130127,
+            -0.013105952180922031,
+            0.014386664144694805,
+            -0.02096952311694622,
+            0.007300057914108038,
+            -0.031932417303323746,
+            0.02236976847052574,
+            0.01146664097905159,
+            0.005916888825595379,
+            0.038318898528814316,
+            -0.02834642305970192,
+            -0.005869929678738117,
+            0.0122009152546525,
+            0.0292856115847826,
+            0.08387809246778488,
+            0.01419028826057911,
+            -0.014352511614561081,
+            -0.025563009083271027,
+            -0.016538260504603386,
+            -0.046105626970529556,
+            -0.0292856115847826,
+            -0.025170257315039635,
+            -0.027099864557385445,
+            0.036338064819574356,
+            -0.0010613900376483798,
+            0.012388752773404121,
+            0.05338007211685181,
+            0.015863751992583275,
+            -0.00874299369752407,
+            -0.039241012185811996,
+            0.0438857264816761,
+            -0.006198645569384098,
+            0.010809208266437054,
+            0.002330895746126771,
+            -0.0146428057923913,
+            -0.006185838486999273,
+            0.011833777651190758,
+            -0.013763383962213993,
+            -0.029456373304128647,
+            0.031949494034051895,
+            -0.01727253384888172,
+            0.03944592550396919,
+            0.06239628419280052,
+            -0.03562086820602417,
+            -0.02001325786113739,
+            0.03189826384186745,
+            -0.035825781524181366,
+            0.018288565799593925,
+            -0.0005250918911769986,
+            -0.030993228778243065,
+            0.026246055960655212,
+            0.01562468521296978,
+            0.06092773377895355,
+            -0.024196917191147804,
+            0.028705023229122162,
+            -0.019176525995135307,
+            0.01628211699426174,
+            -0.020627999678254128,
+            0.03596239164471626,
+            0.0023351646959781647,
+            -0.024265222251415253,
+            -0.015752756968140602,
+            0.014694035053253174,
+            -0.014275669120252132,
+            -0.022830823436379433,
+            0.019740039482712746,
+            0.02100367471575737,
+            -0.010450609028339386,
+            -0.0049051265232264996,
+            0.012013077735900879,
+            0.04989653453230858,
+            0.016205275431275368,
+            -0.014130521565675735,
+            -0.07479357719421387,
+            0.04801815748214722,
+            0.021703798323869705,
+            0.01958635449409485,
+            0.023104043677449226,
+            0.035825781524181366,
+            -0.001056053675711155,
+            0.030566323548555374,
+            0.0019007899099960923,
+            -0.013601160608232021,
+            -0.02044016122817993,
+            0.043646663427352905,
+            -0.0039019023533910513,
+            -0.0019274713704362512,
+            0.0076373121701180935,
+            0.018288565799593925,
+            0.03123229369521141,
+            0.005430218297988176,
+            0.010399380698800087,
+            0.039719145745038986,
+            -0.011586173437535763,
+            0.01414759736508131,
+            0.009460192173719406,
+            0.02091829478740692,
+            0.04299776628613472,
+            0.02245514839887619,
+            -0.008670419454574585,
+            -0.02194286324083805,
+            0.00032551426556892693,
+            0.0219599399715662,
+            0.01811780408024788,
+            0.02199409157037735,
+            -0.012013077735900879,
+            -0.0220111683011055,
+            0.009383349679410458,
+            -0.02199409157037735,
+            0.01682855375111103,
+            -0.00167666538618505,
+            0.01859593763947487,
+            0.013865840621292591,
+            -0.006258412264287472,
+            -0.01677732542157173,
+            0.0002502724528312683,
+            -0.018186109140515327,
+            -0.008380125276744366,
+            -0.010945817455649376,
+            -0.016495568677783012,
+            0.03364003449678421,
+            -0.020747533068060875,
+            0.006599935237318277,
+            -0.01810072734951973,
+            -0.011885006912052631,
+            -0.005835777148604393,
+            0.03596239164471626,
+            0.012209453620016575,
+            -0.01851055584847927,
+            -0.026058217510581017,
+            -0.011108040809631348,
+            -0.018817927688360214,
+            -0.006463326048105955,
+            0.0342206209897995,
+            -0.01750306226313114,
+            0.006659701932221651,
+            -0.004211407620459795,
+            -0.002409872831776738,
+            0.004258367232978344,
+            0.033435117453336716,
+            -0.03082246705889702,
+            -0.01676025055348873,
+            0.023582175374031067,
+            0.008059946820139885,
+            0.029046546667814255,
+            -0.016888320446014404,
+            -0.006275488529354334,
+            -0.024231068789958954,
+            -0.016973702237010002,
+            -0.01643580198287964,
+            -0.005118578672409058,
+            -0.02199409157037735,
+            -0.016025975346565247,
+            0.02790244296193123,
+            0.013686541467905045,
+            -0.00790199264883995,
+            -0.019313136115670204,
+            -0.002333030104637146,
+            0.02542640082538128,
+            -0.013592622242867947,
+            0.04064125940203667,
+            0.050545431673526764,
+            -0.05713682621717453,
+            -0.0122521435841918,
+            -0.03661128506064415,
+            0.07684271782636642,
+            -0.015377080999314785,
+            -0.03032725863158703,
+            0.033964481204748154,
+            -0.029388070106506348,
+            -0.008990597911179066,
+            0.026997406035661697,
+            0.025272715836763382,
+            -0.014719649218022823,
+            0.01123611256480217,
+            0.03756754845380783,
+            -0.030139420181512833,
+            -0.0439881831407547,
+            -0.03777246177196503,
+            0.019859572872519493,
+            -0.025289790704846382,
+            -0.01419028826057911,
+            -0.0110311983153224,
+            0.023530947044491768,
+            0.01801534742116928,
+            0.010664060711860657,
+            0.007547662127763033,
+            0.006843270733952522,
+            0.021157361567020416,
+            0.02144765481352806,
+            0.007073798682540655,
+            -0.03237639740109444,
+            0.0003433908859733492,
+            -0.03944592550396919,
+            -0.009093054570257664,
+            -0.025289790704846382,
+            -0.02095244638621807,
+            -0.023821242153644562,
+            0.00949434470385313,
+            0.03275207430124283,
+            -0.0008143192972056568,
+            0.0067365444265306,
+            0.015044095925986767,
+            0.015488076023757458,
+            -0.03807983547449112,
+            -0.03555256128311157,
+            0.006348061840981245,
+            0.01682855375111103,
+            -0.0055625587701797485,
+            -0.059220120310783386,
+            -0.00042770442087203264,
+            -0.003165493020787835,
+            -0.005016121547669172,
+            -0.028909936547279358,
+            0.010476223193109035,
+            0.013345018029212952,
+            0.010237157344818115,
+            0.0110311983153224,
+            -0.027526767924427986,
+            -0.014651344157755375,
+            -0.0036073385272175074,
+            0.006019345950335264,
+            0.02100367471575737,
+            -0.040709562599658966,
+            -0.022830823436379433,
+            0.03818229213356972,
+            0.0003810117777902633,
+            -0.06598227471113205,
+            -0.005387527868151665,
+            -0.010757979936897755,
+            0.015436847694218159,
+            -0.0005656477296724916,
+            0.034476764500141144,
+            -0.03121521882712841,
+            -0.019296059384942055,
+            -0.020815838128328323,
+            -0.051979828625917435,
+            -0.03531349450349808,
+            -0.015436847694218159,
+            0.007389707490801811,
+            -0.038353051990270615,
+            -0.024265222251415253,
+            0.004111085087060928,
+            -0.0268949493765831,
+            0.007398245856165886,
+            -0.013567008078098297,
+            0.015633223578333855,
+            -0.03852381557226181,
+            -0.010092009790241718,
+            0.004262635949999094,
+            -0.01861301250755787,
+            -0.005507061257958412,
+            0.014369587413966656,
+            0.013925607316195965,
+            0.014719649218022823,
+            0.0017268265364691615,
+            0.015266085974872112,
+            0.0062071834690868855,
+            -0.00769280968233943,
+            0.007833687588572502,
+            -0.04091447591781616,
+            0.0014461372047662735,
+            0.018852079287171364,
+            0.011432488448917866,
+            -0.011082426644861698,
+            0.004151640925556421,
+            -0.010373766534030437,
+            -0.03034433349967003,
+            0.029917430132627487,
+            -0.05419972911477089,
+            0.0024141420144587755,
+            0.005554020404815674,
+            -0.016410188749432564,
+            -0.008384393528103828,
+            -0.009690719656646252,
+            -0.03708941861987114,
+            -0.01248267199844122,
+            0.0039019023533910513,
+            0.00019050588889513165,
+            0.009349197149276733,
+            0.002870929194614291,
+            0.05471201241016388,
+            -0.003165493020787835,
+            -0.04484199360013008,
+            0.023308956995606422,
+            -0.03671374171972275,
+            0.018749622628092766,
+            0.04245132952928543,
+            0.020713381469249725,
+            -0.017212767153978348,
+            0.04791570082306862,
+            0.0195692777633667,
+            0.000224524803343229,
+            -0.016999315470457077,
+            0.025153182446956635,
+            -0.007099412847310305,
+            -0.061303410679101944,
+            -0.004448339343070984,
+            -0.018749622628092766,
+            0.018664240837097168,
+            0.016674868762493134,
+            -0.002407738473266363,
+            -0.010032243095338345,
+            0.009426039643585682,
+            -0.01510386262089014,
+            -0.017443295568227768,
+            0.01705908216536045,
+            -0.011458102613687515,
+            -0.004525181837379932,
+            0.007526317145675421,
+            0.013148642145097256,
+            0.01271320041269064,
+            0.008853988721966743,
+            0.023992003872990608,
+            -0.045627497136592865,
+            0.009127207100391388,
+            -0.01997910626232624,
+            -0.036918655037879944,
+            -0.002990462351590395,
+            0.01318279467523098,
+            0.04183658957481384,
+            -0.02779998630285263,
+            0.0019253368955105543,
+            0.005007583647966385,
+            -0.013515779748558998,
+            -0.021106131374835968,
+            0.00341309723444283,
+            0.04600317031145096,
+            -0.0015763428527861834,
+            0.021618416532874107,
+            -0.011304416693747044,
+            -0.03562086820602417,
+            0.008811297826468945,
+            0.04142676293849945,
+            -0.005028928630053997,
+            0.01663217879831791,
+            0.009434578008949757,
+            0.03548425808548927,
+            -0.04590071365237236,
+            -0.007833687588572502,
+            -0.02344556525349617,
+            -0.01813488081097603,
+            0.04105108603835106,
+            -0.03075416199862957,
+            -0.01365238893777132,
+            0.03524519130587578,
+            -0.005306416191160679,
+            0.02441890724003315,
+            -0.018271489068865776,
+            -0.02588745579123497,
+            0.0014600115828216076,
+            -0.03712356835603714,
+            -0.00708660576492548,
+            -0.01806657575070858,
+            -0.006160224322229624,
+            -0.06557244807481766,
+            0.00010459146869834512,
+            -0.07431544363498688,
+            -0.002902946900576353,
+            0.025136105716228485,
+            -0.006552976090461016,
+            0.04429555684328079,
+            -0.026228979229927063,
+            0.02098659984767437,
+            -0.05553166940808296,
+            -0.01367800310254097,
+            0.00978463888168335,
+            0.03375956416130066,
+            0.007611697539687157,
+            -0.001995776081457734,
+            0.0015507286880165339,
+            -0.026997406035661697,
+            -0.0018303507240489125,
+            -0.007282981649041176,
+            -0.001387437921948731,
+            4.89605481561739e-05,
+            -0.018852079287171364,
+            0.00950288213789463,
+            0.000838332693092525,
+            0.01610281690955162,
+            0.031454283744096756,
+            0.022711291909217834,
+            -0.03070293366909027,
+            -0.008862526156008244,
+            0.005195421166718006,
+            0.02587038092315197,
+            -0.02978082001209259,
+            0.003897633170709014,
+            -0.03464752435684204,
+            0.014864795841276646,
+            5.636466448777355e-05,
+            0.016657792031764984,
+            0.008008718490600586,
+            0.017016392201185226,
+            -0.040197279304265976,
+            0.0030160765163600445,
+            -0.0016254368238151073,
+            -0.027202321216464043,
+            0.020867066457867622,
+            0.002305281348526478,
+            -0.022147778421640396,
+            0.0054472945630550385,
+            -0.004207138437777758,
+            0.024213993921875954,
+            0.04473953694105148,
+            -0.0010592554463073611,
+            -0.02930268831551075,
+            -0.012704662047326565,
+            0.042212262749671936,
+            0.02633143588900566,
+            -0.028483033180236816,
+            0.017118848860263824,
+            0.02103782817721367,
+            -0.02341141365468502,
+            -0.021089056506752968,
+            0.013925607316195965,
+            -0.016179660335183144,
+            -0.0007694943924434483,
+            -0.005468640010803938,
+            0.010399380698800087,
+            0.01240582950413227,
+            -0.029883278533816338,
+            0.05676115304231644,
+            -0.016820017248392105,
+            -0.006834732368588448,
+            -0.03034433349967003,
+            -0.006168762221932411,
+            -0.014531810767948627,
+            0.05553166940808296,
+            0.0028175662737339735,
+            0.02728770114481449,
+            0.01658095046877861,
+            0.002264725510030985,
+            0.05628301948308945,
+            -0.03678204491734505,
+            -0.03451091796159744,
+            0.018681317567825317,
+            -0.0055625587701797485,
+            -0.00671093026176095,
+            -0.022728366777300835,
+            -0.014207364059984684,
+            -0.013353556394577026,
+            0.016965163871645927,
+            -0.0268778745085001,
+            -0.012260681949555874,
+            0.019740039482712746,
+            0.017690900713205338,
+            -2.0469710761972237e-06,
+            0.02293328195810318,
+            0.012644895352423191,
+            0.014284206554293633,
+            0.017878737300634384,
+            -0.020867066457867622,
+            0.019142374396324158,
+            0.024111535400152206,
+            0.009946862235665321,
+            -0.01763967238366604,
+            -0.017878737300634384,
+            0.007347017526626587,
+            -0.0038613462820649147,
+            -0.007125027012079954,
+            0.01437812577933073,
+            0.007043915335088968,
+            -0.004047049675136805,
+            -0.027151092886924744,
+            -0.018356870859861374,
+            0.02093537151813507,
+            -0.0023757205344736576,
+            0.01539415679872036,
+            -0.00928943045437336,
+            -0.010612832382321358,
+            0.03763585537672043,
+            0.01861301250755787,
+            -0.046139780431985855,
+            0.026280207559466362,
+            0.01051037572324276,
+            -0.006228528916835785,
+            -0.013729231432080269,
+            0.005626594182103872,
+            0.009101592935621738,
+            0.010220080614089966,
+            0.016572412103414536,
+            -0.005097233224660158,
+            -0.03328143432736397,
+            0.01564176194369793,
+            0.026211904361844063,
+            -0.020798761397600174,
+            0.012021616101264954,
+            0.00017663151083979756,
+            0.022762520238757133,
+            -0.005028928630053997,
+            -0.012311910279095173,
+            -0.01562468521296978,
+            0.009741948917508125,
+            -0.012235067784786224,
+            0.01123611256480217,
+            0.037396788597106934,
+            0.015914980322122574,
+            -0.03565501794219017,
+            0.015735680237412453,
+            -0.01686270721256733,
+            -0.015684451907873154,
+            0.03261546418070793,
+            0.008264861069619656,
+            -0.02636558935046196,
+            -0.007142103277146816,
+            -0.006527361460030079,
+            0.02001325786113739,
+            -0.02102075144648552,
+            -0.008508196100592613,
+            0.004418455995619297,
+            0.0035646480973809958,
+            0.009938324801623821,
+            0.006745082791894674,
+            0.005016121547669172,
+            0.019159449264407158,
+            0.013806073926389217,
+            -0.0005186883499845862,
+            0.012730276212096214,
+            0.011577636003494263,
+            0.002902946900576353,
+            0.0075391242280602455,
+            -0.010997045785188675,
+            -0.02295035682618618,
+            0.004734364803880453,
+            0.05945918336510658,
+            -0.005852853413671255,
+            0.03876287862658501,
+            0.010407919064164162,
+            0.028961164876818657,
+            0.021686721593141556,
+            -0.024299373850226402,
+            -0.01681147888302803,
+            -0.018151957541704178,
+            -0.021737949922680855,
+            -0.04453462362289429,
+            -0.01510386262089014,
+            -0.01629919372498989,
+            -0.013404784724116325,
+            0.005694898776710033,
+            -0.00013474156730808318,
+            -0.0013116624904796481,
+            0.011355645954608917,
+            -0.002003246918320656,
+            -0.012491210363805294,
+            -0.003355465130880475,
+            0.05324346199631691,
+            -0.0011056812945753336,
+            0.017912890762090683,
+            -0.004576410632580519,
+            0.015539304353296757,
+            -0.007300057914108038,
+            -0.016111355274915695,
+            0.029131926596164703,
+            0.0035561099648475647,
+            -0.0007540191290900111,
+            0.012106996960937977,
+            0.004469684325158596,
+            0.04149506613612175,
+            -0.005887005943804979,
+            -0.026058217510581017,
+            0.011910621076822281,
+            -0.017187153920531273,
+            0.020371858030557632,
+            0.00769707839936018,
+            -0.007607428822666407,
+            0.001932807732373476,
+            0.012909576296806335,
+            -0.027116939425468445,
+            0.019705887883901596,
+            -0.020645076408982277,
+            -0.045149363577365875,
+            0.009280892089009285,
+            -0.01175693515688181,
+            0.023035738617181778,
+            0.001117421081289649,
+            -0.010800670832395554,
+            -0.03507443144917488,
+            0.0021387888118624687,
+            0.0039040367119014263,
+            -0.03372541442513466,
+            0.006548706907778978,
+            0.01863008923828602,
+            0.015539304353296757,
+            0.009485806338489056,
+            -0.012465596199035645,
+            0.003758889390155673,
+            0.0031847036443650723,
+            -0.012371676973998547,
+            0.02776583470404148,
+            -0.004742903169244528,
+            -0.031983643770217896,
+            0.035859934985637665,
+            -0.02733892947435379,
+            0.009468729607760906,
+            -0.020730456337332726,
+            -0.012346062809228897,
+            -0.01608574204146862,
+            0.00441418681293726,
+            -0.02148180827498436,
+            -0.002561423694714904,
+            -0.02983205020427704,
+            0.027577996253967285,
+            0.009383349679410458,
+            -0.015249010175466537,
+            -0.004862436093389988,
+            0.009101592935621738,
+            -0.025255639106035233,
+            0.007846495136618614,
+            0.012585128657519817,
+            -0.010646984912455082,
+            -0.01367800310254097,
+            -0.00706099160015583,
+            -0.0014685495989397168,
+            0.005972386337816715,
+            0.0158210601657629,
+            0.04672037065029144,
+            0.004431263078004122,
+            0.03712356835603714,
+            0.02337726205587387,
+            0.013532855547964573,
+            0.0037012575194239616,
+            0.0047898623161017895,
+            -0.0067322757095098495,
+            0.04012897238135338,
+            0.006249873898923397,
+            0.0219599399715662,
+            0.008247784338891506,
+            -0.044671230018138885,
+            0.008503926917910576,
+            0.03264961391687393,
+            0.026621730998158455,
+            0.018715469166636467,
+            0.009588262997567654,
+            -0.0024013349320739508,
+            0.021191513165831566,
+            -0.020252324640750885,
+            -0.01705908216536045,
+            0.0035902622621506453,
+            -0.007530585862696171,
+            -0.0056650154292583466,
+            -0.009716334752738476,
+            0.03387909755110741,
+            -0.003229528432711959,
+            -0.008862526156008244,
+            0.018186109140515327,
+            -0.007432397920638323,
+            0.00475997943431139,
+            0.006245605181902647,
+            0.0011163539020344615,
+            0.02739015780389309,
+            0.008683227002620697,
+            0.027680452913045883,
+            0.005391797050833702,
+            0.007573276292532682,
+            0.0058571225963532925,
+            -0.009451653808355331,
+            0.018920384347438812,
+            -0.0097590247169137,
+            -0.033366814255714417,
+            -0.009443115442991257,
+            0.0016393112018704414,
+            0.014070754870772362,
+            0.0004023569927085191,
+            -0.014062216505408287,
+            -0.011893544346094131,
+            0.007859301753342152,
+            0.012533900327980518,
+            0.02107197977602482,
+            0.018169032409787178,
+            -0.005016121547669172,
+            -0.024726277217268944,
+            0.012593667022883892,
+            0.011056812480092049,
+            -0.01861301250755787,
+            0.008465506136417389,
+            0.012542438693344593,
+            0.057410046458244324,
+            -0.044705383479595184,
+            0.016538260504603386,
+            0.02151595987379551,
+            -0.004896588623523712,
+            -0.015249010175466537,
+            -0.010851899161934853,
+            0.0031633584294468164,
+            -0.017477449029684067,
+            0.014975790865719318,
+            -0.01798119582235813,
+            0.011193421669304371,
+            -0.014745263382792473,
+            0.00743666710332036,
+            0.012866885401308537,
+            -0.009682182222604752,
+            0.003942457959055901,
+            0.020371858030557632,
+            0.0028068935498595238,
+            -0.0077867284417152405,
+            0.005387527868151665,
+            0.01390853151679039,
+            0.020354781299829483,
+            0.006066305562853813,
+            -0.0005933965439908206,
+            0.019876649603247643,
+            -0.028175661340355873,
+            -0.025170257315039635,
+            0.01909114606678486,
+            -0.03551841154694557,
+            0.005472908727824688,
+            0.02790244296193123,
+            -0.04296361654996872,
+            -0.05020390823483467,
+            0.0027407235465943813,
+            -0.039719145745038986,
+            0.0004191663465462625,
+            -0.0023287611547857523,
+            0.01536854263395071,
+            -0.004057722166180611,
+            -0.017212767153978348,
+            0.0146428057923913,
+            -0.012397291138768196,
+            -0.0067322757095098495,
+            0.009835867211222649,
+            0.009016212075948715,
+            -0.024572592228651047,
+            -0.0016222350532189012,
+            -0.030890770256519318,
+            0.009169897064566612,
+            -0.017844585701823235,
+            -0.019142374396324158,
+            -0.025528857484459877,
+            -0.0005656477296724916,
+            -0.010638446547091007,
+            -0.001638244022615254,
+            0.051501695066690445,
+            0.021652569994330406,
+            0.020166942849755287,
+            -0.007726961746811867,
+            -0.023650480434298515,
+            0.05457540228962898,
+            0.0018132745753973722,
+            0.03814813867211342,
+            -0.012380215339362621,
+            -0.0002006448630709201,
+            -0.0008852921309880912,
+            0.022113624960184097,
+            -0.017101772129535675,
+            -0.0011547752656042576,
+            -0.011022660881280899,
+            -0.00586566049605608,
+            -0.0070780678652226925,
+            0.014045140706002712,
+            0.015974747017025948,
+            -0.002610517665743828,
+            0.014156135730445385,
+            0.01221799198538065,
+            0.022694215178489685,
+            0.03130060061812401,
+            0.016230888664722443,
+            -0.0034664601553231478,
+            -0.022199006751179695,
+            -0.012918113730847836,
+            -0.008200825192034245,
+            0.008999135345220566,
+            -0.008525271899998188,
+            -0.0031847036443650723,
+            -0.02631436102092266,
+            -0.020730456337332726,
+            -0.011312955059111118,
+            -0.01292665209621191,
+            -0.0031462821643799543,
+            0.02641681768000126,
+            -0.002469639526680112,
+            -0.03763585537672043,
+            -0.01511240005493164,
+            -0.03524519130587578,
+            -0.015607609413564205,
+            -0.0017876604106277227,
+            -0.01748598739504814,
+            0.016538260504603386,
+            -0.003780234605073929,
+            -0.007833687588572502,
+            -0.0008964983280748129,
+            -0.018169032409787178,
+            0.009690719656646252,
+            -0.011048275046050549,
+            -0.000543768925126642,
+            -0.0017503062263131142,
+            0.013345018029212952,
+            -0.008205094374716282,
+            0.008973521180450916,
+            -0.01412198320031166,
+            0.006868884898722172,
+            -0.048120614141225815,
+            -0.027731681242585182,
+            -0.027099864557385445,
+            0.022557605057954788,
+            -0.015214857645332813,
+            -0.023189423605799675,
+            0.008734455332159996,
+            -0.013686541467905045,
+            0.008939368650317192,
+            -0.012021616101264954,
+            0.008388662710785866,
+            0.019210679456591606,
+            0.009024749509990215,
+            -0.002614786848425865,
+            -0.013669464737176895,
+            -0.02390662208199501,
+            -0.02585330419242382,
+            0.023650480434298515,
+            -0.026485122740268707,
+            0.01071528997272253,
+            -0.012021616101264954,
+            0.008585038594901562,
+            -0.029917430132627487,
+            -0.020730456337332726,
+            0.0044910297729074955,
+            0.011347107589244843,
+            -0.021140284836292267,
+            0.044671230018138885,
+            0.04043634235858917,
+            0.029985735192894936,
+            -0.020252324640750885,
+            0.05139923840761185,
+            -0.022796671837568283,
+            0.007312864996492863,
+            0.036440521478652954,
+            0.030446790158748627,
+            0.026092370972037315,
+            -0.015581995248794556,
+            -0.012294834479689598,
+            -0.015496614389121532,
+            0.013404784724116325,
+            -0.024333525449037552,
+            0.002821835223585367,
+            -0.02042308636009693,
+            0.013643850572407246,
+            0.03451091796159744,
+            -0.004917933605611324,
+            0.029934506863355637,
+            -0.013720693998038769,
+            0.030361410230398178,
+            0.0017652479000389576,
+            -0.026587579399347305,
+            0.015667375177145004,
+            -0.014232978224754333,
+            0.004418455995619297,
+            0.017964119091629982,
+            -0.006087650544941425,
+            0.013575546443462372,
+            -0.061337560415267944,
+            0.028141509741544724,
+            0.01758844405412674,
+            0.01123611256480217,
+            -0.002180411946028471,
+            -0.024726277217268944,
+            -0.0021142419427633286,
+            0.009340658783912659,
+            -0.0016745307948440313,
+            0.011859391815960407,
+            0.01193623524159193,
+            -0.033418044447898865,
+            0.010578680783510208,
+            -0.013635313138365746,
+            0.024299373850226402,
+            0.006258412264287472,
+            0.0007476155878975987,
+            -0.03961668908596039,
+            -0.00560524920001626,
+            0.014830644242465496,
+            0.01095435582101345,
+            -0.005989462602883577,
+            -0.04624223709106445,
+            -0.030105268582701683,
+            0.025563009083271027,
+            0.005110040307044983,
+            -0.05167245492339134,
+            0.014557424932718277,
+            -0.015334390103816986,
+            0.018766699358820915,
+            -0.008708841167390347,
+            -0.025972837582230568,
+            -0.029985735192894936,
+            -0.017187153920531273,
+            0.016726097092032433,
+            0.048564594238996506,
+            0.004207138437777758,
+            0.013345018029212952,
+            -0.0048538981936872005,
+            0.006651164032518864,
+            -0.024760430678725243,
+            0.012030153535306454,
+            -0.027151092886924744,
+            -0.02971251681447029,
+            0.029012393206357956,
+            0.009981014765799046,
+            -0.011330030858516693,
+            0.019330210983753204,
+            -0.017084697261452675,
+            -0.0014141193823888898,
+            0.02240392006933689,
+            -0.0006163425859995186,
+            -0.007295788731426001,
+            -0.010382304899394512,
+            0.013618236407637596,
+            -0.0014354645973071456,
+            -0.01762259565293789,
+            -0.009571187198162079,
+            0.04241717979311943,
+            0.010092009790241718,
+            0.018835002556443214,
+            0.03261546418070793,
+            -0.0018933190731331706,
+            -0.02400907874107361,
+            -0.0027215126901865005,
+            -0.026638807728886604,
+            -0.017033468931913376,
+            -0.014548887498676777,
+            0.016674868762493134,
+            0.023530947044491768,
+            0.015249010175466537,
+            -0.027526767924427986,
+            -0.030019886791706085,
+            -0.01950097270309925,
+            -0.022199006751179695,
+            -0.0034493841230869293,
+            -0.0004213008505757898,
+            -0.026502197608351707,
+            0.007005494087934494,
+            0.0341181643307209,
+            0.03357172757387161,
+            -0.0010870042024180293,
+            -0.009016212075948715,
+            -0.010638446547091007,
+            0.0007721625734120607,
+            0.015231933444738388,
+            -0.015018481761217117,
+            0.0012326851719990373,
+            0.005028928630053997,
+            0.005383259151130915,
+            0.0029285612981766462,
+            -0.007095144130289555,
+            -0.05747834965586662,
+            0.003961668815463781,
+            0.02544347569346428,
+            0.005084426142275333,
+            -0.0008377990452572703,
+            0.02629728429019451,
+            0.0008276600856333971,
+            0.008123982697725296,
+            -0.03142013028264046,
+            0.011108040809631348,
+            0.01803242415189743,
+            0.015163629315793514,
+            -0.018151957541704178,
+            0.0122521435841918,
+            0.0064206356182694435,
+            0.004811207763850689,
+            -0.00030950538348406553,
+            0.010143238119781017,
+            0.028636718168854713,
+            0.019620506092905998,
+            -0.014736725017428398,
+            0.023821242153644562,
+            0.0006435577524825931,
+            -0.012072844430804253,
+            -0.024333525449037552,
+            -0.001040044822730124,
+            0.0029883277602493763,
+            -0.011176345869898796,
+            0.014702572487294674,
+            -0.0023949311580508947,
+            0.0007956422632560134,
+            0.04139260947704315,
+            0.0437491200864315,
+            0.013481627218425274,
+            -0.011295879259705544,
+            0.003304236801341176,
+            0.026502197608351707,
+            0.020132791250944138,
+            0.004862436093389988,
+            0.011458102613687515,
+            -0.03331558406352997,
+            -0.019620506092905998,
+            -0.002981924219056964,
+            0.0014290610561147332,
+            -0.0026788224931806326,
+            0.03876287862658501,
+            0.029524678364396095,
+            0.009682182222604752,
+            -0.005741858389228582,
+            -0.011364183388650417,
+            0.013643850572407246,
+            0.020201096311211586,
+            -0.023599252104759216,
+            -0.0038015798199921846,
+            0.0016745307948440313,
+            -0.017170077189803123,
+            0.02288205362856388,
+            -0.0070311082527041435,
+            -0.00928943045437336,
+            0.02645096927881241,
+            0.09186973422765732,
+            -0.00464471522718668,
+            0.012141148559749126,
+            -0.03948007896542549,
+            -0.021823331713676453,
+            -0.006975610740482807,
+            -0.0036671049892902374,
+            0.015402695164084435,
+            0.02739015780389309,
+            0.009135744534432888,
+            0.007133565377444029,
+            0.017878737300634384,
+            -0.004606293980032206,
+            0.012508286163210869,
+            -0.02826104313135147,
+            0.020166942849755287,
+            -0.005007583647966385,
+            -0.0038335975259542465,
+            -0.011389797553420067,
+            -0.005989462602883577,
+            0.011765473522245884,
+            0.005468640010803938,
+            0.017016392201185226,
+            0.0019893725402653217,
+            -0.0024760430678725243,
+            0.0014333301223814487,
+            0.003031018190085888,
+            -0.010877513326704502,
+            -0.009383349679410458,
+            -0.021396426483988762,
+            0.003827193984761834,
+            0.0009914844995364547,
+            -0.007884915918111801,
+            0.012755890376865864,
+            0.005502792075276375,
+            0.024043232202529907,
+            -0.028175661340355873,
+            -0.0030609015375375748,
+            0.00633525475859642,
+            -0.008956445381045341,
+            0.010390842333436012,
+            0.012380215339362621,
+            -0.008640536107122898,
+            0.022830823436379433,
+            -0.01466841995716095,
+            0.014762339182198048,
+            0.01170570682734251,
+            -0.021601341664791107,
+            -0.004155910108238459,
+            -0.009938324801623821,
+            -0.02192578837275505,
+            0.020252324640750885,
+            -0.018715469166636467,
+            0.005033197812736034,
+            0.00073320756200701,
+            0.006774966139346361,
+            -0.0010496501345187426,
+            -0.01740914396941662,
+            0.010348152369260788,
+            -0.008785683661699295,
+            0.006839001551270485,
+            -0.005058811977505684,
+            0.018254414200782776,
+            0.012166762724518776,
+            0.0028090281412005424,
+            -0.0011622461024671793,
+            0.014600115828216076,
+            -0.00372900627553463,
+            -0.018886230885982513,
+            -0.0036713741719722748,
+            0.01633334532380104,
+            0.01051037572324276,
+            0.0002606782363727689,
+            -0.010305461473762989,
+            -0.0146342683583498,
+            0.014139058999717236,
+            0.000977076473645866,
+            -0.0018122073961421847,
+            -0.0005285604856908321,
+            -0.03760170191526413,
+            0.002606248715892434,
+            0.013532855547964573,
+            0.012704662047326565,
+            -0.005818700883537531,
+            -0.017477449029684067,
+            0.007983104325830936,
+            -0.004190062638372183,
+            -0.0003204447857569903,
+            0.01661510206758976,
+            -0.019774191081523895,
+            0.01002370472997427,
+            -0.014779414981603622,
+            -0.03278622403740883,
+            -0.0029776552692055702,
+            -0.008380125276744366,
+            0.005822970066219568,
+            -0.005592442117631435,
+            -0.0071805245243012905,
+            -0.01414759736508131,
+            -0.005741858389228582,
+            -0.02979789674282074,
+            0.008947907015681267,
+            0.0317445807158947,
+            0.004473953507840633,
+            -0.028978241607546806,
+            -0.012158225290477276,
+            0.025580085813999176,
+            -0.0024931191001087427,
+            -0.006915844511240721,
+            0.003009672975167632,
+            0.0012572321575134993,
+            -0.005784548819065094,
+            -0.009656568057835102,
+            -0.012935190461575985,
+            0.013063261285424232,
+            0.01320840884000063,
+            0.011244650930166245,
+            -0.03357172757387161,
+            0.010664060711860657,
+            0.01813488081097603,
+            -0.02042308636009693,
+            0.011748396791517735,
+            0.028585489839315414,
+            0.0029435029719024897,
+            0.010399380698800087,
+            0.007334209978580475,
+            0.008286206051707268,
+            -0.017153002321720123,
+            0.016683407127857208,
+            0.009186973795294762,
+            -0.012559514492750168,
+            0.004563603550195694,
+            0.007381169591099024,
+            0.008542348630726337,
+            0.01960342936217785,
+            -0.011603250168263912,
+            0.013609698973596096,
+            -0.0068816919811069965,
+            0.03321312740445137,
+            -0.011919158510863781,
+            -0.014881872572004795,
+            0.03229101747274399,
+            0.027560919523239136,
+            -0.004563603550195694,
+            0.017246920615434647,
+            -0.01943266950547695,
+            -0.01267050951719284,
+            0.019296059384942055,
+            0.0005907283630222082,
+            0.019313136115670204,
+            -0.011304416693747044,
+            -0.02595576085150242,
+            0.029012393206357956,
+            -0.008106905966997147,
+            -0.001259366748854518,
+            -0.015829598531126976,
+            0.0043458822183310986,
+            -0.0005789885181002319,
+            0.037328481674194336,
+            0.012653433717787266,
+            0.0011985328746959567,
+            0.026570502668619156,
+            -0.00769280968233943,
+            -0.025067800655961037,
+            0.001990439835935831,
+            -0.015214857645332813,
+            0.013336479663848877,
+            -0.027970748022198677,
+            -0.00560524920001626,
+            0.014506196603178978,
+            0.003026749240234494,
+            -0.002644670195877552,
+            0.01666633039712906,
+            -0.007513510063290596,
+            0.011791087687015533,
+            -0.019244831055402756,
+            0.016717558726668358,
+            -0.011517869308590889,
+            0.004473953507840633,
+            -0.011910621076822281,
+            0.026092370972037315,
+            -0.025563009083271027,
+            -0.00999809056520462,
+            0.013660927303135395,
+            -0.008354511111974716,
+            0.0006376878009177744,
+            -0.02252345345914364,
+            -0.008089830167591572,
+            -0.020371858030557632,
+            0.01679440215229988,
+            -0.005682091694325209,
+            0.011902082711458206,
+            -0.00306303589604795,
+            0.0025528857950121164,
+            -0.008585038594901562,
+            0.010322538204491138,
+            -0.0009199780761264265,
+            -0.017417682334780693,
+            -0.005968117620795965,
+            -0.03025895357131958,
+            -0.030361410230398178,
+            0.03072001039981842,
+            -0.00880275946110487,
+            0.003923247568309307,
+            0.02728770114481449,
+            -0.02540932409465313,
+            -0.005485715810209513,
+            0.001918933354318142,
+            0.022728366777300835,
+            -0.030651705339550972,
+            0.007914799265563488,
+            -0.009391887113451958,
+            -0.016452878713607788,
+            0.007184793706983328,
+            -0.0008874266059137881,
+            0.015880826860666275,
+            -0.0025635582860559225,
+            -0.04924764111638069,
+            -0.0007081269286572933,
+            0.017135925590991974,
+            -0.013806073926389217,
+            0.008081291802227497,
+            0.02151595987379551,
+            -0.009630953893065453,
+            -0.008439891040325165,
+            -0.0011334300506860018,
+            0.025989914312958717,
+            -0.022625910118222237,
+            0.030019886791706085,
+            0.005767472553998232,
+            0.004713019821792841,
+            0.005340568721294403,
+            0.002962713595479727,
+            0.01730668731033802,
+            0.002080089645460248,
+            0.004258367232978344,
+            0.025238562375307083,
+            0.013148642145097256,
+            0.0037418133579194546,
+            -0.0032978332601487637,
+            -0.0030352871399372816,
+            0.016538260504603386,
+            -0.018442250788211823,
+            0.02100367471575737,
+            -0.0004714620590675622,
+            0.00013560871593654156,
+            0.016256503760814667,
+            -0.01484772004187107,
+            -0.027219397947192192,
+            0.03075416199862957,
+            -0.022335615009069443,
+            0.01757136732339859,
+            0.023770013824105263,
+            -0.01339624635875225,
+            -0.004922202788293362,
+            -0.007073798682540655,
+            0.001534719718620181,
+            0.018339794129133224,
+            -0.0134218605235219,
+            -0.023308956995606422,
+            -0.00030790449818596244,
+            0.0220111683011055,
+            0.02293328195810318,
+            0.028175661340355873,
+            -0.021140284836292267,
+            0.01412198320031166,
+            0.007466550450772047,
+            -0.008311820216476917,
+            0.016469955444335938,
+            0.008452698588371277,
+            -0.0025272713974118233,
+            0.021293969824910164,
+            0.0075903525575995445,
+            0.004576410632580519,
+            -0.008896678686141968,
+            -0.013242561370134354,
+            -0.01267050951719284,
+            -0.006749351508915424,
+            -0.006164493504911661,
+            -0.008264861069619656,
+            -0.01709323562681675,
+            0.09548988193273544,
+            -0.011867930181324482,
+            -0.0028410458471626043,
+            -0.009195511229336262,
+            0.0026041141245514154,
+            -0.006949996575713158,
+            0.0013479492627084255,
+            0.023086966946721077,
+            -0.022728366777300835,
+            -0.03671374171972275,
+            0.011167807504534721,
+            -0.004070529248565435,
+            0.006288295611739159,
+            0.03712356835603714,
+            0.00038314631092362106,
+            -0.016589488834142685,
+            -6.773765198886395e-05,
+            -0.003372541395947337,
+            0.01295226626098156,
+            0.0028197006322443485,
+            -0.0036158766597509384,
+            0.0109799699857831,
+            0.029200231656432152,
+            -0.017810434103012085,
+            -0.02144765481352806,
+            -0.012525362893939018,
+            0.0032615463715046644,
+            -0.002578499959781766,
+            0.003707661060616374,
+            0.0051996903494000435,
+            -0.014053679071366787,
+            -0.0134218605235219,
+            0.018817927688360214,
+            -0.016094280406832695,
+            0.0008025794522836804,
+            -0.012627819553017616,
+            -0.008222170174121857,
+            0.01753721572458744,
+            0.01757136732339859,
+            -0.03406693786382675,
+            -0.009921248070895672,
+            -0.03855796530842781,
+            0.018749622628092766,
+            0.01643580198287964,
+            0.014540349133312702,
+            -0.020884141325950623,
+            -0.012644895352423191,
+            -0.004520913120359182,
+            -0.02981497347354889,
+            -0.02243807166814804,
+            -0.026536351069808006,
+            0.009605339728295803,
+            0.00365856708958745,
+            0.027595072984695435,
+            0.005259457044303417,
+            -0.006262681446969509,
+            0.00801725685596466,
+            -0.01025423314422369,
+            0.00755620002746582,
+            -0.02146473154425621,
+            0.007107951212674379,
+            -0.0011409008875489235,
+            0.0012529630912467837,
+            0.005592442117631435,
+            -0.008853988721966743,
+            -0.01735791563987732,
+            -0.0079276068136096,
+            0.01466841995716095,
+            -0.015752756968140602,
+            -0.00821790099143982,
+            0.020798761397600174,
+            0.009383349679410458,
+            0.00931504461914301,
+            0.015616146847605705,
+            0.045661646872758865,
+            0.008273398503661156,
+            0.00783795677125454,
+            -0.011056812480092049,
+            0.009648029692471027,
+            -0.01813488081097603,
+            0.00745801255106926,
+            -0.019330210983753204,
+            -0.006292564328759909,
+            -0.019722962751984596,
+            0.04012897238135338,
+            -0.009366272948682308,
+            -0.01635042205452919,
+            0.033435117453336716,
+            -0.011808163486421108,
+            0.01910822093486786,
+            0.0035475720651447773,
+            -0.022830823436379433,
+            0.03352050110697746,
+            0.003876287955790758,
+            0.012918113730847836,
+            -0.025682542473077774,
+            0.0035859933122992516,
+            -0.00877714529633522,
+            0.006032153032720089,
+            -0.011586173437535763,
+            0.003494208911433816,
+            0.0023885276168584824,
+            -0.004841091111302376,
+            -0.008760069496929646,
+            0.018698394298553467,
+            0.006595666520297527,
+            0.0002183880569646135,
+            0.028056127950549126,
+            -0.000954130373429507,
+            0.018220260739326477,
+            6.110063259257004e-05,
+            0.006010808050632477,
+            0.027680452913045883,
+            0.002821835223585367,
+            0.006809118203818798,
+            -0.01714446395635605,
+            -0.0009199780761264265,
+            -0.009485806338489056,
+            0.009596801362931728,
+            0.02639974094927311,
+            -0.010612832382321358,
+            -0.031078608706593513,
+            0.022574681788682938,
+            -0.007048184517771006,
+            0.015966208651661873,
+            0.002999000484123826,
+            -0.014062216505408287,
+            -0.0023437028285115957,
+            -0.027680452913045883,
+            0.014454968273639679,
+            -0.017349377274513245,
+            -0.024265222251415253,
+            -0.007231753319501877,
+            -0.01045914739370346,
+            -0.004845359828323126,
+            0.038353051990270615,
+            0.012474133633077145,
+            -0.024572592228651047,
+            0.006817656569182873,
+            -0.003974475897848606,
+            -0.005989462602883577,
+            0.0057119750417768955,
+            0.005976655520498753,
+            0.004098278004676104,
+            0.010945817455649376,
+            -0.0065700518898665905,
+            -0.01320840884000063,
+            -0.03324728086590767,
+            -0.02240392006933689,
+            0.014941639266908169,
+            0.00028068936080671847,
+            -0.01315718051046133,
+            0.028483033180236816,
+            -0.01002370472997427,
+            -0.005575365852564573,
+            0.0071292961947619915,
+            0.00013887719251215458,
+            -0.0022796671837568283,
+            -0.017844585701823235,
+            0.002273263642564416,
+            0.006245605181902647,
+            -0.014950176700949669,
+            0.009844405576586723,
+            0.0032124524004757404,
+            -0.014540349133312702,
+            0.005635132547467947,
+            -0.0015027020126581192,
+            -0.003562513506039977,
+            -0.02481165900826454,
+            0.0059595792554318905,
+            0.00708233704790473,
+            -0.0018036692636087537,
+            0.002390662208199501,
+            -0.018169032409787178,
+            0.01801534742116928,
+            0.001955220242962241,
+            0.0071762558072805405,
+            -0.029166078194975853,
+            0.007282981649041176,
+            -0.006740813609212637,
+            0.0015080382581800222,
+            -0.0195692777633667,
+            0.009161358699202538,
+            0.020798761397600174,
+            0.012491210363805294,
+            -0.0005250918911769986,
+            0.010442070662975311,
+            0.008175211027264595,
+            -0.012704662047326565,
+            0.003950996324419975,
+            -0.02050846628844738,
+            -0.007308595813810825,
+            -0.010057857260107994,
+            0.011372721754014492,
+            -0.004219945520162582,
+            0.015983285382390022,
+            0.009511420503258705,
+            0.005600980017334223,
+            -0.009699258022010326,
+            0.00743666710332036,
+            0.01292665209621191,
+            0.005874198395758867,
+            -0.0011099503608420491,
+            0.017827508971095085,
+            0.07363239675760269,
+            0.005336299538612366,
+            0.013618236407637596,
+            6.870485958643258e-05,
+            -0.013071799650788307,
+            0.0015624684747308493,
+            -0.004939279053360224,
+            -0.01748598739504814,
+            0.026126522570848465,
+            0.026587579399347305,
+            -0.006382214371114969,
+            0.0004917400074191391,
+            0.009016212075948715,
+            0.005558289587497711,
+            0.005485715810209513,
+            0.022216081619262695,
+            -0.0077867284417152405,
+            0.01753721572458744,
+            0.024179840460419655,
+            0.005541213322430849,
+            -0.0019093280425295234,
+            -0.0022305732127279043,
+            0.009152821265161037,
+            0.004209273029118776,
+            -0.009050363674759865,
+            0.01150079257786274,
+            -0.01437812577933073,
+            0.01909114606678486,
+            0.011389797553420067,
+            0.02146473154425621,
+            0.007073798682540655,
+            -0.01704200729727745,
+            -0.018356870859861374,
+            0.0041281613521277905,
+            -0.0013479492627084255,
+            0.0059083509258925915,
+            0.0063224476762115955,
+            0.005477177910506725,
+            -0.011611787602305412,
+            -0.0041793896816670895,
+            0.010356690734624863,
+            -0.002471773885190487,
+            -0.0008500725380145013,
+            0.01757136732339859,
+            -0.01268758624792099,
+            0.01168009266257286,
+            -0.001839956152252853,
+            -0.0009984216885641217,
+            0.009682182222604752,
+            0.0014354645973071456,
+            -0.011825240217149258,
+            -0.004166582599282265,
+            0.0021003675647079945,
+            0.01217530108988285,
+            -0.013259637169539928,
+            -0.01048476155847311,
+            0.011825240217149258,
+            -0.006168762221932411,
+            0.009169897064566612,
+            -0.009579724632203579,
+            -0.0025315405800938606,
+            -0.00039648707024753094,
+            -0.008085560984909534,
+            -0.0014578771078959107,
+            0.007820880971848965,
+            0.0008324627415277064,
+            0.010518914088606834,
+            0.011560559272766113,
+            0.01150079257786274,
+            0.003560379147529602,
+            0.004002224653959274,
+            -0.002964847953990102,
+            -0.0064206356182694435,
+            -0.011432488448917866,
+            8.15119783510454e-05,
+            0.005776010919362307,
+            -0.02242099680006504,
+            -0.00024800453684292734,
+            0.008465506136417389,
+            0.005046004895120859,
+            -0.0007710952777415514,
+            0.020849989727139473,
+            0.001264702994376421,
+            0.007820880971848965,
+            -0.025221485644578934,
+            0.0011238247388973832,
+            0.006890229880809784,
+            0.007718423847109079,
+            0.012798581272363663,
+            -0.028636718168854713,
+            -0.0038528083823621273,
+            -0.0038634808734059334,
+            0.0171273872256279,
+            -0.00011606451880652457,
+            -0.0033106403425335884,
+            0.021686721593141556,
+            -0.0007833687704987824,
+            -0.003430173499509692,
+            -0.00755620002746582,
+            -0.005579635035246611,
+            0.008004449307918549,
+            0.012994957156479359,
+            0.024623820558190346,
+            -0.0019968433771282434,
+            -0.0055967108346521854,
+            -0.00809409935027361,
+            0.0244530588388443,
+            0.015249010175466537,
+            0.01863008923828602,
+            -0.0219087116420269,
+            0.010638446547091007,
+            -0.013865840621292591,
+            0.009067440405488014,
+            0.011714245192706585,
+            -0.010911665856838226,
+            0.0004290384822525084,
+            -0.010757979936897755,
+            -0.017135925590991974,
+            -0.022711291909217834,
+            0.00502039073035121,
+            0.02839765138924122,
+            -0.0002676154254004359,
+            0.007193332072347403,
+            -0.016358960419893265,
+            -0.0018442251021042466,
+            -0.011312955059111118,
+            -0.025750847533345222,
+            -0.016222350299358368,
+            -0.021772101521492004,
+            -0.02141350321471691,
+            -0.0038592119235545397,
+            -0.002131317974999547,
+            -0.01907406933605671,
+            0.00013153978215996176,
+            -0.0012817791430279613,
+            0.01902284100651741,
+            -0.0014888276346027851,
+            0.0015624684747308493,
+            0.012337524443864822,
+            0.012943727895617485,
+            -0.012824195437133312,
+            -0.004228483885526657,
+            -0.019261907786130905,
+            -0.024179840460419655,
+            0.0032487392891198397,
+            -0.007726961746811867,
+            0.013874378986656666,
+            -0.008234977722167969,
+            0.004260501824319363,
+            -0.017187153920531273,
+            0.014702572487294674,
+            -0.03579162806272507,
+            -0.024128612130880356,
+            0.0006531630642712116,
+            -0.0040299734100699425,
+            -0.009605339728295803,
+            -0.02585330419242382,
+            0.01198746357113123,
+            -0.019620506092905998,
+            0.002104636514559388,
+            -0.018442250788211823,
+            0.011398335918784142,
+            -0.015283161774277687,
+            0.018681317567825317,
+            -0.00025520852068439126,
+            0.03179580718278885,
+            0.004405648913234472,
+            0.007978835143148899,
+            -0.004525181837379932,
+            0.003327716374769807,
+            -0.005468640010803938,
+            0.006365138106048107,
+            0.006988417822867632,
+            0.008055677637457848,
+            0.007825150154531002,
+            -0.004768517334014177,
+            0.004061991348862648,
+            0.005110040307044983,
+            0.030515095219016075,
+            0.028021976351737976,
+            0.0027706066612154245,
+            0.0014674824196845293,
+            0.0034685947466641665,
+            0.004576410632580519,
+            -0.00038021133514121175,
+            0.008862526156008244,
+            -0.011287340894341469,
+            -0.005391797050833702,
+            0.01417321152985096,
+            -0.012448519468307495,
+            -0.0037695621140301228,
+            0.015377080999314785,
+            -0.015086785890161991,
+            -0.026092370972037315,
+            0.021823331713676453,
+            -0.006552976090461016,
+            -0.002450428670272231,
+            -0.006689585279673338,
+            0.008115444332361221,
+            -0.005946772173047066,
+            0.004194331355392933,
+            0.028107358142733574,
+            -0.005737589206546545,
+            -0.010945817455649376,
+            0.012636357918381691,
+            -0.002330895746126771,
+            -0.007948951795697212,
+            0.02291620522737503,
+            0.035894084721803665,
+            0.00999809056520462,
+            -0.026041142642498016,
+            -0.007381169591099024,
+            0.02144765481352806,
+            -0.00973341055214405,
+            -0.021755026653409004,
+            0.008273398503661156,
+            0.007522047962993383,
+            -0.005421680398285389,
+            0.018937459215521812,
+            0.004239156376570463,
+            -0.010792132467031479,
+            0.009041826240718365,
+            0.022711291909217834,
+            0.021686721593141556,
+            0.008947907015681267,
+            -0.023548023775219917,
+            0.01025423314422369,
+            -0.026228979229927063,
+            -0.013831689022481441,
+            -0.023821242153644562,
+            0.003581724362447858,
+            0.003857077332213521,
+            -0.003257277188822627,
+            0.026536351069808006,
+            0.011876468546688557,
+            0.011645940132439137,
+            0.0030950538348406553,
+            0.023052813485264778,
+            -0.015223395079374313,
+            -0.028107358142733574,
+            -0.0009733410552144051,
+            -0.015479537658393383,
+            -0.0018282162491232157,
+            0.002166537567973137,
+            0.007150641642510891,
+            0.005852853413671255,
+            0.01953512616455555,
+            0.000376209121895954,
+            0.0019584218971431255,
+            -2.4263486920972355e-05,
+            0.010100548155605793,
+            -0.015727141872048378,
+            -0.019705887883901596,
+            -0.008922292850911617,
+            6.470263178925961e-05,
+            0.011637401767075062,
+            -0.005208228249102831,
+            -3.448583447607234e-05,
+            0.007846495136618614,
+            -0.015897903591394424,
+            -0.0097590247169137,
+            -0.014412278309464455,
+            -0.0029306956566870213,
+            -0.005071619059890509,
+            0.03136890381574631,
+            0.014437892474234104,
+            -0.004747172351926565,
+            -0.011543483473360538,
+            -0.005741858389228582,
+            0.03370833769440651,
+            0.0017204229952767491,
+            0.012422905303537846,
+            0.0146342683583498,
+            0.015684451907873154,
+            0.027014482766389847,
+            0.012807118706405163,
+            0.003018211107701063,
+            0.002516598906368017,
+            0.02346264198422432,
+            -0.013951221480965614,
+            0.008730186149477959,
+            -0.011645940132439137,
+            -0.003521957667544484,
+            -0.010937280021607876,
+            0.015078248456120491,
+            -0.013003494590520859,
+            -0.011278802528977394,
+            0.023069890215992928,
+            -0.0004381102044135332,
+            0.013276712968945503,
+            -0.0004242358263581991,
+            -0.003953130915760994,
+            -0.02534101903438568,
+            0.0015464596217498183,
+            0.0006531630642712116,
+            -0.004401379730552435,
+            0.010578680783510208,
+            -0.027475539594888687,
+            0.0013660927070304751,
+            0.015240471810102463,
+            -0.007116489112377167,
+            0.00038981667603366077,
+            -0.010826284997165203,
+            0.005904081743210554,
+            0.00011606451880652457,
+            -0.02385539375245571,
+            0.015488076023757458,
+            -0.006698123179376125,
+            0.01243144366890192,
+            -0.0018548977095633745,
+            -0.013788998126983643,
+            0.008687496185302734,
+            0.019296059384942055,
+            0.027560919523239136,
+            -0.007825150154531002,
+            0.0005528406472876668,
+            -0.0001855698210420087,
+            -0.006027883850038052,
+            -0.02737308293581009,
+            0.02054261974990368,
+            -0.007863570936024189,
+            0.01953512616455555,
+            -0.009682182222604752,
+            -0.018186109140515327,
+            -0.014856258407235146,
+            0.011927696876227856,
+            0.007603159639984369,
+            0.0010304395109415054,
+            0.01489894837141037,
+            -0.024077383801341057,
+            0.020867066457867622,
+            0.0033383890986442566,
+            -0.0064718639478087425,
+            0.008303281851112843,
+            0.004230618476867676,
+            0.004695943556725979,
+            -0.008384393528103828,
+            0.007658657152205706,
+            0.007569007575511932,
+            -0.007253098301589489,
+            0.004326671827584505,
+            -0.04057295247912407,
+            0.010732365772128105,
+            -0.011090965010225773,
+            0.005878467578440905,
+            -0.004742903169244528,
+            -0.022745443508028984,
+            -0.035825781524181366,
+            0.004567872267216444,
+            0.019313136115670204,
+            -0.020235247910022736,
+            0.018408099189400673,
+            0.012124072760343552,
+            0.014574501663446426,
+            0.01859593763947487,
+            0.008853988721966743,
+            0.01461719162762165,
+            0.0438857264816761,
+            -0.024231068789958954,
+            6.483604374807328e-05,
+            0.008734455332159996,
+            0.011381260119378567,
+            -0.013404784724116325,
+            -0.010126162320375443,
+            0.005784548819065094,
+            -0.0015197781613096595,
+            0.04067540913820267,
+            -0.019210679456591606,
+            0.010886050760746002,
+            -0.010962894186377525,
+            0.008371586911380291,
+            -0.009562648832798004,
+            -0.0036158766597509384,
+            -0.012559514492750168,
+            -0.011893544346094131,
+            0.014745263382792473,
+            -0.019671734422445297,
+            -0.005460101645439863,
+            -0.007282981649041176,
+            0.01048476155847311,
+            -0.006096188444644213,
+            -0.020730456337332726,
+            0.018647165969014168,
+            -0.0018079383298754692,
+            0.029968658462166786,
+            -0.012755890376865864,
+            0.00574612757191062,
+            0.022591758519411087,
+            -0.010604294948279858,
+            0.018732545897364616,
+            -0.004362958483397961,
+            0.008853988721966743,
+            0.0055625587701797485,
+            -0.0065188235603272915,
+            -0.0014685495989397168,
+            0.01389145478606224,
+            -0.0027385889552533627,
+            0.029934506863355637,
+            -0.017400605604052544,
+            -0.009818791411817074,
+            0.009707796387374401,
+            0.0073769004084169865,
+            0.0042370217852294445,
+            0.003878422547131777,
+            0.009579724632203579,
+            -0.01071528997272253,
+            0.0009728074073791504,
+            -0.021686721593141556,
+            -0.015453923493623734,
+            -0.0011483717244118452,
+            -0.004683136474341154,
+            -0.0018730411538854241,
+            -0.002467504935339093,
+            -0.02044016122817993,
+            0.004627638962119818,
+            -0.0021579996682703495,
+            -8.451364556094632e-05,
+            -0.024487212300300598,
+            -0.00769280968233943,
+            -0.010057857260107994,
+            -0.005161269102245569,
+            -0.003139878623187542,
+            0.018271489068865776,
+            0.0034856710117310286,
+            -0.004862436093389988,
+            0.0293197650462389,
+            0.005404604133218527,
+            0.01536854263395071,
+            0.00280262460000813,
+            -0.05334591865539551,
+            -0.011014122515916824,
+            -0.004563603550195694,
+            -0.009912709705531597,
+            -0.007261636666953564,
+            -0.008456967771053314,
+            -0.010681137442588806,
+            -0.0019061262719333172,
+            0.0032914294861257076,
+            -0.004170851781964302,
+            0.0122521435841918,
+            -0.01700785383582115,
+            0.025494705885648727,
+            -0.004017166327685118,
+            0.015744218602776527,
+            -0.011722782626748085,
+            0.011654478497803211,
+            -0.006194376386702061,
+            -0.010851899161934853,
+            0.010544528253376484,
+            0.0146342683583498,
+            -0.0030544979963451624,
+            0.020371858030557632,
+            -0.014736725017428398,
+            0.003197510726749897,
+            -0.008303281851112843,
+            -0.02143057994544506,
+            -0.015991821885108948,
+            0.011295879259705544,
+            0.03130060061812401,
+            0.010066395625472069,
+            -0.0037524858489632607,
+            0.004072663839906454,
+            0.021669644862413406,
+            -0.004352285992354155,
+            0.0063181789591908455,
+            0.03025895357131958,
+            -0.0010982104577124119,
+            0.017742129042744637,
+            0.0001935742620844394,
+            -0.005028928630053997,
+            -0.002362913452088833,
+            0.01808365248143673,
+            -0.002766337711364031,
+            0.0057119750417768955,
+            0.02441890724003315,
+            0.010997045785188675,
+            -0.0064718639478087425,
+            0.008230708539485931,
+            0.014309820719063282,
+            0.013874378986656666,
+            -0.015351466834545135,
+            -0.0122009152546525,
+            -0.011534945107996464,
+            -0.015778370201587677,
+            0.007291519548743963,
+            -0.019261907786130905,
+            -0.004049184266477823,
+            0.007829418405890465,
+            -0.015138015151023865,
+            -0.012807118706405163,
+            -0.023548023775219917,
+            -0.015914980322122574,
+            0.01863008923828602,
+            0.005357644986361265,
+            -0.01465988252311945,
+            -0.004369362257421017,
+            0.0013714289525523782,
+            0.006181569304317236,
+            -0.0054985228925943375,
+            0.021840406581759453,
+            -0.004098278004676104,
+            -0.011125117540359497,
+            0.019330210983753204,
+            0.0017897948855534196,
+            0.005097233224660158,
+            -0.0064206356182694435,
+            0.004550796467810869,
+            -0.011364183388650417,
+            0.022096550092101097,
+            -0.003470729338005185,
+            0.012636357918381691,
+            -0.0037033918779343367,
+            0.004631908144801855,
+            -0.001978699816390872,
+            0.00821790099143982,
+            0.004017166327685118,
+            0.010629909113049507,
+            -0.005656477529555559,
+            0.004525181837379932,
+            0.012491210363805294,
+            -0.01641872711479664,
+            0.007265905383974314,
+            0.004559334367513657,
+            -4.365760105429217e-05,
+            0.005780279636383057,
+            -0.007645850069820881,
+            -0.005417411215603352,
+            -0.015488076023757458,
+            0.010100548155605793,
+            0.030873695388436317,
+            -0.0016649254830554128,
+            -0.007240291219204664,
+            0.01704200729727745,
+            0.015735680237412453,
+            -0.003816521493718028,
+            0.0061218030750751495,
+            0.009280892089009285,
+            -0.003148416755720973,
+            0.01370361726731062,
+            -0.024487212300300598,
+            -0.029524678364396095,
+            0.01763967238366604,
+            -0.020867066457867622,
+            0.022779595106840134,
+            -0.019210679456591606,
+            -0.027731681242585182,
+            -0.01847640424966812,
+            -0.0032530082389712334,
+            -0.019842496141791344,
+            -0.01611989364027977,
+            0.02532394416630268,
+            0.00685180863365531,
+            -0.006949996575713158,
+            -0.031112760305404663,
+            0.04050464928150177,
+            0.014489120803773403,
+            0.038318898528814316,
+            -0.011603250168263912,
+            -2.554753518779762e-05,
+            -0.007504971697926521,
+            0.012064306065440178,
+            -0.01146664097905159,
+            -0.014258592389523983,
+            0.007509240880608559,
+            0.028175661340355873,
+            0.015231933444738388,
+            -0.015223395079374313,
+            -0.007944682613015175,
+            0.015991821885108948,
+            -0.007043915335088968,
+            -0.0005480379913933575,
+            -0.008836911991238594,
+            -0.014830644242465496,
+            -0.010066395625472069,
+            -0.0005312286084517837,
+            0.012243606150150299,
+            0.016179660335183144,
+            -0.0007257367251440883,
+            0.01390853151679039,
+            -0.009955400601029396,
+            -0.014420815743505955,
+            -0.004602024797350168,
+            0.005327761638909578,
+            -0.02252345345914364,
+            0.02105490304529667,
+            0.002292474266141653,
+            -0.009537034668028355,
+            0.0030865157023072243,
+            -0.004572141449898481,
+            0.012602205388247967,
+            0.009391887113451958,
+            -0.0038421356584876776,
+            -0.0036137420684099197,
+            0.00572051340714097,
+            -0.001715086749754846,
+            0.015001405030488968,
+            -0.011714245192706585,
+            0.011842316016554832,
+            -0.007432397920638323,
+            -0.00019931078713852912,
+            -0.009648029692471027,
+            -0.005238111596554518,
+            -0.010493299923837185,
+            0.0028794673271477222,
+            0.012943727895617485,
+            -0.014548887498676777,
+            0.012807118706405163,
+            -0.0008132520597428083,
+            -0.00949434470385313,
+            -0.015138015151023865,
+            -0.0055155991576612,
+            -0.011364183388650417,
+            -0.016683407127857208,
+            0.007069529499858618,
+            0.0007855033036321402,
+            0.019347287714481354,
+            0.0030054040253162384,
+            -0.004572141449898481,
+            -0.00017262929759453982,
+            0.018903307616710663,
+            -0.02580207586288452,
+            0.03403278440237045,
+            -0.00010045584349427372,
+            0.022711291909217834,
+            0.04251963645219803,
+            0.00954557303339243,
+            -0.006663971114903688,
+            -0.013754845596849918,
+            0.005801625084131956,
+            -0.013609698973596096,
+            0.009981014765799046,
+            0.014335434883832932,
+            -0.03235932067036629,
+            -0.003944592550396919,
+            -0.013976835645735264,
+            0.011586173437535763,
+            -0.007957490161061287,
+            0.0032850259449332952,
+            -0.010587218217551708,
+            0.011765473522245884,
+            0.01679440215229988,
+            -0.009075978770852089,
+            -0.004311730153858662,
+            -0.027492616325616837,
+            0.00032524747075513005,
+            0.016674868762493134,
+            -0.010911665856838226,
+            -0.01684563048183918,
+            0.010945817455649376,
+            -0.007218946237117052,
+            -0.015453923493623734,
+            -0.006898768246173859,
+            -0.03278622403740883,
+            -0.00020504731219261885,
+            -0.01076651830226183,
+            -0.003927516285330057,
+            -0.0028794673271477222,
+            0.0011024795239791274,
+            -0.0033917520195245743,
+            -0.0067066615447402,
+            -0.0025891726836562157,
+            0.0067664277739822865,
+            0.005011852364987135,
+            0.002080089645460248,
+            0.0005987327895127237,
+            -0.00023506399884354323,
+            0.009255277924239635,
+            0.010168852284550667,
+            -0.007419590838253498,
+            -0.017212767153978348,
+            0.010834822431206703,
+            -0.020730456337332726,
+            0.005938234273344278,
+            0.00783795677125454,
+            0.011039736680686474,
+            -0.0220111683011055,
+            -0.010868974961340427,
+            0.009818791411817074,
+            0.003835732117295265,
+            0.01902284100651741,
+            0.010886050760746002,
+            0.0022455148864537477,
+            -0.0035646480973809958,
+            0.0028495839796960354,
+            -1.0947752343781758e-05,
+            -0.0012508286163210869,
+            0.015308775939047337,
+            0.0006504949415102601,
+            0.02974666841328144,
+            -0.012055767700076103,
+            0.011201960034668446,
+            -0.009340658783912659,
+            0.0013842361513525248,
+            -0.009741948917508125,
+            -0.001973363570868969,
+            -0.004194331355392933,
+            -0.006484671495854855,
+            0.013148642145097256,
+            0.007714154664427042,
+            -0.004239156376570463,
+            -0.01416467409580946,
+            0.01194477267563343,
+            -0.023035738617181778,
+            -0.012269220314919949,
+            -0.00881983619183302,
+            0.012935190461575985,
+            -0.0053960662335157394,
+            0.01170570682734251,
+            0.03457922115921974,
+            -0.0018559650052338839,
+            0.0007860369514673948,
+            -0.013746308162808418,
+            -0.008311820216476917,
+            -0.02728770114481449,
+            -0.01510386262089014,
+            -0.008922292850911617,
+            -0.03454506769776344,
+            -0.011919158510863781,
+            0.008239246904850006,
+            0.008111175149679184,
+            0.0014237246941775084,
+            -0.019176525995135307,
+            -0.008888140320777893,
+            -0.01492456253618002,
+            -0.017460372298955917,
+            -0.00924673955887556,
+            -0.01175693515688181,
+            -0.0037119300104677677,
+            0.020884141325950623,
+            -0.004162313882261515,
+            0.014267130754888058,
+            0.016939548775553703,
+            1.3132299500284716e-05,
+            0.015334390103816986,
+            0.009665105491876602,
+            -0.009127207100391388,
+            -0.004725826904177666,
+            0.006885961163789034,
+            0.0027706066612154245,
+            0.018749622628092766,
+            -0.006702392362058163,
+            -0.014796491712331772,
+            0.004298923071473837,
+            0.005571096669882536,
+            -0.012346062809228897,
+            0.010655523277819157,
+            -0.00476424815133214,
+            0.02054261974990368,
+            -0.018254414200782776,
+            -0.011885006912052631,
+            0.014472044073045254,
+            -0.008862526156008244,
+            0.02394077368080616,
+            0.013951221480965614,
+            -0.015291700139641762,
+            0.007167717441916466,
+            0.013029108755290508,
+            0.007615966722369194,
+            0.0018495614640414715,
+            0.022147778421640396,
+            -0.0011846584966406226,
+            0.0031590894795954227,
+            0.010237157344818115,
+            -0.018835002556443214,
+            -0.02042308636009693,
+            -0.003118533408269286,
+            0.02247222512960434,
+            0.007961759343743324,
+            -0.016222350299358368,
+            0.0037930416874587536,
+            0.012849809601902962,
+            0.012072844430804253,
+            0.0039787450805306435,
+            -0.001204936415888369,
+            -0.02248930186033249,
+            0.0035155541263520718,
+            0.0001910395221784711,
+            0.018664240837097168,
+            -0.011116579174995422,
+            0.001461078878492117,
+            -0.011389797553420067,
+            0.012747352942824364,
+            -0.005221035331487656,
+            0.01193623524159193,
+            0.0011761204805225134,
+            -0.005353375803679228,
+            0.0001780989987310022,
+            0.014941639266908169,
+            0.014070754870772362,
+            0.037738312035799026,
+            -0.005916888825595379,
+            -0.006480402313172817,
+            0.017170077189803123,
+            -0.012388752773404121,
+            0.018715469166636467,
+            0.006497478578239679,
+            0.0170249305665493,
+            -0.011090965010225773,
+            0.0011344973463565111,
+            0.04115354269742966,
+            0.015949131920933723,
+            -6.583659705938771e-05,
+            -0.008760069496929646,
+            -0.00805140845477581,
+            0.0244530588388443,
+            0.009067440405488014,
+            -2.7882166250492446e-05,
+            0.015744218602776527,
+            0.006096188444644213,
+            -0.016324806958436966,
+            -0.012918113730847836,
+            0.03555256128311157,
+            -0.007718423847109079,
+            -0.022045321762561798,
+            -0.0009311842732131481,
+            -0.009383349679410458,
+            -0.02884163148701191,
+            0.013959759846329689,
+            0.013788998126983643,
+            -0.008162403479218483,
+            0.006992687005549669,
+            0.008439891040325165,
+            -0.004117488861083984,
+            -0.004435532260686159,
+            0.0005346972029656172,
+            -0.0007049251580610871,
+            0.006177300587296486,
+            0.005092964507639408,
+            -0.014463506639003754,
+            -0.006787773221731186,
+            -0.005669284611940384,
+            -0.004064125940203667,
+            0.009016212075948715,
+            0.005058811977505684,
+            0.004870974458754063,
+            -0.008482581935822964,
+            0.018339794129133224,
+            0.02298451028764248,
+            -0.0122009152546525,
+            0.012115534394979477,
+            0.005233842879533768,
+            -0.0146428057923913,
+            -0.004614831879734993,
+            -0.01121903583407402,
+            -0.026570502668619156,
+            -0.006113264709711075,
+            0.0033490615896880627,
+            0.015428309328854084,
+            -0.022796671837568283,
+            0.0008500725380145013,
+            0.007790997624397278,
+            2.7365211280994117e-05,
+            -0.008879602886736393,
+            0.007573276292532682,
+            -0.00044157879892736673,
+            0.022301463410258293,
+            0.019808344542980194,
+            0.025631314143538475,
+            -0.008414276875555515,
+            -0.00499904528260231,
+            0.014301283285021782,
+            0.00433947890996933,
+            0.009374811314046383,
+            0.027543844655156136,
+            0.022164853289723396,
+            -0.013857303187251091,
+            0.006429173983633518,
+            0.000640355923678726,
+            -0.014250054024159908,
+            0.0066682398319244385,
+            0.013054722920060158,
+            0.012414366938173771,
+            0.0028410458471626043,
+            -0.0013415457215160131,
+            -0.015957670286297798,
+            0.004824014846235514,
+            0.018408099189400673,
+            -0.017118848860263824,
+            -0.03471583127975464,
+            -0.011842316016554832,
+            -0.012158225290477276,
+            -0.00624133599922061,
+            -0.012875423766672611,
+            0.01440373994410038,
+            0.004418455995619297,
+            0.019705887883901596,
+            0.0002309283590875566,
+            -0.01022861897945404,
+            0.014250054024159908,
+            0.005110040307044983,
+            -0.0016638581873849034,
+            0.0005352308508008718,
+            0.007231753319501877,
+            -0.0030331527814269066,
+            -0.0022476494777947664,
+            -0.00027055040118284523,
+            0.016239427030086517,
+            -0.02532394416630268,
+            0.008947907015681267,
+            -0.017067620530724525,
+            0.0017951311310753226,
+            0.015249010175466537,
+            -0.003775965655222535,
+            -0.016025975346565247,
+            0.0011216901475563645,
+            -0.008619191125035286,
+            -0.07288104295730591,
+            -0.005882736761122942,
+            0.004512374754995108,
+            0.023992003872990608,
+            -0.013336479663848877,
+            -0.0024824466090649366,
+            0.0018474268727004528,
+            0.008657612837851048,
+            -0.010134699754416943,
+            0.003995821345597506,
+            0.010390842333436012,
+            -0.003319178242236376,
+            -0.009221125394105911,
+            0.002508060773834586,
+            -0.008106905966997147,
+            -0.0038805571384727955,
+            -0.00783795677125454,
+            0.003402424743399024,
+            0.025221485644578934,
+            -0.010868974961340427,
+            0.024657974019646645,
+            0.009699258022010326,
+            -0.0075433929450809956,
+            -0.0027428579051047564,
+            -0.007761114276945591,
+            -0.027543844655156136,
+            -0.012397291138768196,
+            -0.003396020969375968,
+            -0.01292665209621191,
+            0.0029264267068356276,
+            0.001692674239166081,
+            -0.007167717441916466,
+            -0.004348016809672117,
+            0.0021547977812588215,
+            0.00030336863710545003,
+            -0.022147778421640396,
+            -0.0009407896432094276,
+            -0.014267130754888058,
+            0.02105490304529667,
+            -0.012371676973998547,
+            -0.004461146425455809,
+            0.0030544979963451624,
+            -0.010109085589647293,
+            0.004580679349601269,
+            0.019313136115670204,
+            -0.007308595813810825,
+            -0.009682182222604752,
+            0.014045140706002712,
+            -0.003690584795549512,
+            0.00185916677583009,
+            -0.01798119582235813,
+            0.010032243095338345,
+            -0.03232516720890999,
+            -0.003583858720958233,
+            -0.009981014765799046,
+            -0.0031847036443650723,
+            -0.0067151994444429874,
+            -0.03162504732608795,
+            0.015138015151023865,
+            -0.007039646618068218,
+            -0.02346264198422432,
+            -0.0011793222511187196,
+            -0.01859593763947487,
+            0.01419028826057911,
+            -0.01487333420664072,
+            0.01393414568156004,
+            -0.020645076408982277,
+            0.012533900327980518,
+            0.012798581272363663,
+            0.021123208105564117,
+            0.006493209395557642,
+            -0.005302147474139929,
+            0.002676687901839614,
+            0.0018730411538854241,
+            0.014352511614561081,
+            -0.008905217051506042,
+            0.00462336977943778,
+            -0.002390662208199501,
+            0.005737589206546545,
+            0.009426039643585682,
+            0.02107197977602482,
+            -0.0005896611255593598,
+            -0.0037823691964149475,
+            -0.017468910664319992,
+            0.00757754547521472,
+            -0.01559907104820013,
+            0.005485715810209513,
+            0.0025251370389014482,
+            0.023787088692188263,
+            0.010886050760746002,
+            0.0035859933122992516,
+            0.004900857340544462,
+            0.004520913120359182,
+            0.01861301250755787,
+            -0.006890229880809784,
+            -0.004747172351926565,
+            -0.018186109140515327,
+            0.019688811153173447,
+            0.007705616764724255,
+            -0.005306416191160679,
+            -0.011902082711458206,
+            0.006010808050632477,
+            0.002134519862011075,
+            0.0017577770631760359,
+            0.02288205362856388,
+            -0.02974666841328144,
+            -0.008964983746409416,
+            -0.007808073423802853,
+            -0.009682182222604752,
+            -0.017844585701823235,
+            0.017084697261452675,
+            -0.008012987673282623,
+            0.00461910106241703,
+            -0.012790042906999588,
+            0.033418044447898865,
+            0.002969117136672139,
+            0.00513992365449667,
+            0.013729231432080269,
+            -0.00024373548512812704,
+            0.017844585701823235,
+            -0.008367317728698254,
+            -0.04036803916096687,
+            -0.0031911071855574846,
+            0.0034515184815973043,
+            0.007645850069820881,
+            -0.010903127491474152,
+            0.004862436093389988,
+            -0.006339523941278458,
+            -0.01854470930993557,
+            -0.014232978224754333,
+            0.037840768694877625,
+            -0.005225304514169693,
+            0.017947042360901833,
+            -0.016145508736371994,
+            -0.0033682722132653,
+            0.007107951212674379,
+            0.022762520238757133,
+            -0.012602205388247967,
+            0.00791053008288145,
+            0.009571187198162079,
+            0.02199409157037735,
+            0.00978463888168335,
+            -0.01565883681178093,
+            0.023735860362648964,
+            0.005741858389228582,
+            -0.0023565099108964205,
+            0.005276532843708992,
+            -0.0030352871399372816,
+            0.016000360250473022,
+            -0.014813567511737347,
+            -0.011876468546688557,
+            0.002303146990016103,
+            -0.010168852284550667,
+            0.0037055264692753553,
+            -0.024589668959379196,
+            0.00037540867924690247,
+            -0.009844405576586723,
+            0.021089056506752968,
+            0.01027130987495184,
+            0.00877714529633522,
+            -0.019637582823634148,
+            -0.0048026698641479015,
+            -0.02645096927881241,
+            -0.014574501663446426,
+            -0.01022861897945404,
+            -0.0011451698374003172,
+            -0.004431263078004122,
+            -0.007765382993966341,
+            -0.03133475035429001,
+            -0.004202869720757008,
+            0.013541393913328648,
+            0.014975790865719318,
+            -0.0013714289525523782,
+            0.012089920230209827,
+            -0.00019544197130016983,
+            -0.008106905966997147,
+            -0.02236976847052574,
+            -0.010407919064164162,
+            -0.02091829478740692,
+            -0.03123229369521141,
+            -0.022199006751179695,
+            0.008670419454574585,
+            -0.012021616101264954,
+            -0.002941368380561471,
+            0.028961164876818657,
+            -0.022130701690912247,
+            0.007513510063290596,
+            0.005400335416197777,
+            0.0028666602447628975,
+            0.014856258407235146,
+            0.0054472945630550385,
+            0.0003634020104072988,
+            0.0037652929313480854,
+            -0.008341703563928604,
+            -0.008081291802227497,
+            0.01799827069044113,
+            -0.003331985557451844,
+            0.005097233224660158,
+            0.010809208266437054,
+            -0.006749351508915424,
+            -0.005438756663352251,
+            0.02392369881272316,
+            -0.015718603506684303,
+            0.002044870052486658,
+            -0.008610652759671211,
+            -0.014079293236136436,
+            -0.00949434470385313,
+            -0.005793086718767881,
+            -0.01813488081097603,
+            -0.015240471810102463,
+            0.01689685881137848,
+            -0.0004028906114399433,
+            0.016025975346565247,
+            -0.005289340391755104,
+            -0.005007583647966385,
+            0.014437892474234104,
+            -0.03558671474456787,
+            -0.003402424743399024,
+            0.006377945188432932,
+            -0.026211904361844063,
+            0.02484581060707569,
+            0.005276532843708992,
+            0.01605158858001232,
+            -0.013780459761619568,
+            0.0026318628806620836,
+            -0.006365138106048107,
+            0.004358689300715923,
+            0.018903307616710663,
+            0.007308595813810825,
+            0.004858167376369238,
+            -0.006689585279673338,
+            -0.008047140203416348,
+            0.016077203676104546,
+            -0.007415322121232748,
+            0.0018719738582149148,
+            0.016785863786935806,
+            0.018425175920128822,
+            0.012004539370536804,
+            0.0059553105384111404,
+            0.013071799650788307,
+            0.01750306226313114,
+            -0.007043915335088968,
+            -0.005835777148604393,
+            -0.013481627218425274,
+            -0.00802579429000616,
+            -0.0013105951948091388,
+            0.03859211876988411,
+            0.018920384347438812,
+            0.007385438773781061,
+            -0.02054261974990368,
+            0.013456013053655624,
+            0.0043928418308496475,
+            6.841302820248529e-06,
+            -0.0027385889552533627,
+            0.02293328195810318,
+            0.0024952536914497614,
+            0.007125027012079954,
+            0.006258412264287472,
+            -0.017212767153978348,
+            0.03309359401464462,
+            -0.020713381469249725,
+            0.0008532743086107075,
+            -0.013319403864443302,
+            -0.012166762724518776,
+            0.015530765987932682,
+            0.002104636514559388,
+            -0.004164448007941246,
+            -0.006296833511441946,
+            -0.0033575997222214937,
+            0.013515779748558998,
+            0.008499657735228539,
+            -0.012875423766672611,
+            0.004516643937677145,
+            -0.02542640082538128,
+            0.004315999336540699,
+            -0.015334390103816986,
+            0.021755026653409004,
+            -0.004559334367513657,
+            -0.004674598574638367,
+            -0.019261907786130905,
+            -0.017067620530724525,
+            -0.00999809056520462,
+            0.002403469290584326,
+            -0.004755710251629353,
+            0.004422725178301334,
+            -0.009468729607760906,
+            -0.008499657735228539,
+            0.0027599341701716185,
+            0.0036628360394388437,
+            0.012465596199035645,
+            0.013669464737176895,
+            0.019893724471330643,
+            0.017759205773472786,
+            -0.01536000519990921,
+            -0.0341864712536335,
+            0.0006056699785403907,
+            0.005084426142275333,
+            -0.0036436254158616066,
+            0.018664240837097168,
+            -0.001938143977895379,
+            -0.0009968207450583577,
+            -0.0007150641758926213,
+            0.0005419012159109116,
+            0.02004740945994854,
+            -0.011014122515916824,
+            -0.016657792031764984,
+            0.00021545309573411942,
+            0.008853988721966743,
+            0.00502039073035121,
+            -0.0025934416335076094,
+            -0.010296924039721489,
+            -0.0009455922991037369,
+            0.0020341973286122084,
+            0.006032153032720089,
+            -0.009101592935621738,
+            -0.0010496501345187426,
+            -0.0013671600027009845,
+            -0.0030544979963451624,
+            -0.006254143081605434,
+            -0.009844405576586723,
+            0.008222170174121857,
+            -0.012602205388247967,
+            -0.01123611256480217,
+            0.016222350299358368,
+            -0.015086785890161991,
+            0.001136631821282208,
+            0.023821242153644562,
+            0.022557605057954788,
+            -0.0008842248353175819,
+            0.006774966139346361,
+            -0.007936145178973675,
+            -0.011517869308590889,
+            -0.02785121463239193,
+            -0.019637582823634148,
+            -0.00806848518550396,
+            0.017947042360901833,
+            -0.002053408185020089,
+            -0.004049184266477823,
+            -0.0018869155319407582,
+            -0.008580769412219524,
+            0.008444160223007202,
+            -0.013814612291753292,
+            0.011620325967669487,
+            -0.034425534307956696,
+            0.013088875450193882,
+            -0.006702392362058163,
+            0.0010485828388482332,
+            0.015009943395853043,
+            0.04351005330681801,
+            -0.007351286243647337,
+            0.0029541754629462957,
+            0.001540056080557406,
+            0.01389145478606224,
+            0.028209814801812172,
+            0.004369362257421017,
+            0.0006136744632385671,
+            -0.01765674725174904,
+            0.004091874696314335,
+            0.002264725510030985,
+            -0.013456013053655624,
+            0.009579724632203579,
+            -0.023035738617181778,
+            -0.023155272006988525,
+            -0.009067440405488014,
+            -0.021123208105564117,
+            -0.002608383307233453,
+            0.002011785050854087,
+            0.012883962132036686,
+            -0.009878558106720448,
+            -0.012098458595573902,
+            -0.0005656477296724916,
+            -0.032991137355566025,
+            0.016709022223949432,
+            0.09665105491876602,
+            0.00875153113156557,
+            -0.003908305894583464,
+            -0.0021515958942472935,
+            0.002499522641301155,
+            0.009562648832798004,
+            -0.004768517334014177,
+            0.055360905826091766,
+            -0.018852079287171364,
+            0.008311820216476917,
+            -0.0008105839369818568,
+            0.008064216002821922,
+            -0.007223214954137802,
+            0.015530765987932682,
+            -0.012209453620016575,
+            -0.005951041355729103,
+            -0.009460192173719406,
+            -0.01123611256480217,
+            -0.0017246920615434647,
+            -0.016640717163681984,
+            -0.004986238200217485,
+            -0.020645076408982277,
+            0.013336479663848877,
+            -0.03234224393963814,
+            -0.011509330943226814,
+            -0.012474133633077145,
+            -0.0063224476762115955,
+            0.006066305562853813,
+            -0.015684451907873154,
+            -0.002595576224848628,
+            -0.002099300269037485,
+            0.015402695164084435,
+            0.0013863706262782216,
+            -0.00232235761359334,
+            -0.008328896015882492,
+            0.0005405671545304358,
+            0.005024659913033247,
+            0.003940323833376169,
+            0.006557244807481766,
+            0.006792041938751936,
+            -0.005848584230989218,
+            -0.008811297826468945,
+            0.010655523277819157,
+            -0.002510195365175605,
+            -0.0027919518761336803,
+            -0.008871064521372318,
+            0.010382304899394512,
+            -0.013481627218425274,
+            0.018356870859861374,
+            -0.013840226456522942,
+            -0.029507601633667946,
+            -0.015547842718660831,
+            0.02100367471575737,
+            -0.01996202953159809,
+            -0.0039381892420351505,
+            0.010553065687417984,
+            0.02829519473016262,
+            0.004113219678401947,
+            -0.001529383473098278,
+            -0.004909395705908537,
+            -0.017204230651259422,
+            0.009511420503258705,
+            0.008196556009352207,
+            0.018442250788211823,
+            -0.012371676973998547,
+            0.016487032175064087,
+            -0.018288565799593925,
+            0.006177300587296486,
+            -0.01663217879831791,
+            0.0006083381595090032,
+            0.00011866596469189972,
+            -0.0041793896816670895,
+            0.01608574204146862,
+            -0.01534292846918106,
+            -0.026263132691383362,
+            0.01900576427578926,
+            -0.013635313138365746,
+            -0.03307652100920677,
+            0.003374675754457712,
+            -0.034937821328639984,
+            -0.0025827689096331596,
+            0.009716334752738476,
+            -0.02149888314306736,
+            -0.018305642530322075,
+            0.007487895432859659,
+            -0.010066395625472069,
+            -0.008871064521372318,
+            0.00502039073035121,
+            -0.01689685881137848,
+            -0.012149686925113201,
+            -0.001152640674263239,
+            0.006753620691597462,
+            -0.00499477656558156,
+            0.0057632033713161945,
+            -0.026075294241309166,
+            0.014010988175868988,
+            -0.0170249305665493,
+            0.01144102681428194,
+            -0.010587218217551708,
+            0.022762520238757133,
+            0.011748396791517735,
+            -0.010356690734624863,
+            -0.0008303282083943486,
+            0.006036422215402126,
+            0.0005264259525574744,
+            0.026126522570848465,
+            0.03237639740109444,
+            -0.0011889275629073381,
+            0.008469774387776852,
+            0.007586083374917507,
+            0.003921112976968288,
+            -0.011014122515916824,
+            0.0015923518221825361,
+            0.014830644242465496,
+            0.01992787793278694,
+            -0.013729231432080269,
+            0.008879602886736393,
+            -0.026672959327697754,
+            0.02103782817721367,
+            -0.022096550092101097,
+            -0.000982412719167769,
+            0.010066395625472069,
+            -0.004777055233716965,
+            0.007090874947607517,
+            0.0066682398319244385,
+            0.021362274885177612,
+            0.03375956416130066,
+            0.00807702261954546,
+            0.008879602886736393,
+            0.007257367484271526,
+            -0.004798400681465864,
+            0.002044870052486658,
+            0.009135744534432888,
+            -0.005869929678738117,
+            -0.0040940092876553535,
+            -0.010083471424877644,
+            -0.006685316096991301,
+            0.016205275431275368,
+            0.0008079157560132444,
+            0.012602205388247967,
+            0.007018301170319319,
+            -0.007428129203617573,
+            -0.005669284611940384,
+            0.007014031987637281,
+            -0.023787088692188263,
+            -0.00950288213789463,
+            0.025221485644578934,
+            -0.013814612291753292,
+            -0.043202683329582214,
+            -0.0024333526380360126,
+            0.008853988721966743,
+            -0.017964119091629982,
+            -0.01245705783367157,
+            -0.004772786516696215,
+            0.010117623955011368,
+            0.0007241358398459852,
+            -0.007569007575511932,
+            0.028175661340355873,
+            0.00502039073035121,
+            -0.005169807001948357,
+            0.003571051638573408,
+            -2.8232359909452498e-05,
+            0.019244831055402756,
+            0.01536000519990921,
+            0.008294744417071342,
+            -0.03266669064760208,
+            -0.007304327096790075,
+            -0.005780279636383057,
+            0.013114489614963531,
+            0.002813297091051936,
+            0.003423769725486636,
+            -0.019740039482712746,
+            -0.020798761397600174,
+            0.019210679456591606,
+            0.004935009870678186,
+            0.019825419411063194,
+            -0.00867895781993866,
+            -0.006548706907778978,
+            -0.004841091111302376,
+            -0.004742903169244528,
+            0.00499477656558156,
+            0.0076415808871388435,
+            0.0038933642208576202,
+            0.03377664089202881,
+            -0.001280711847357452,
+            0.004042780492454767,
+            -0.0036350872833281755,
+            -0.0002441357064526528,
+            0.014787953346967697,
+            -0.00790199264883995,
+            0.008905217051506042,
+            0.018800850957632065,
+            -0.02887578494846821,
+            0.011398335918784142,
+            -0.01679440215229988,
+            -0.00828193686902523,
+            0.016000360250473022,
+            -0.0005320290802046657,
+            -0.004922202788293362,
+            -0.014583039097487926,
+            0.0008127184119075537,
+            0.007752575911581516,
+            0.01996202953159809,
+            -0.004091874696314335,
+            0.011287340894341469,
+            -0.009648029692471027,
+            -0.004305326379835606,
+            0.010083471424877644,
+            0.006087650544941425,
+            -0.0033213128335773945,
+            0.006232798099517822
+          ],
+          "keyphrases": [
+            "MoE models",
+            "expert choice",
+            "training efficiency",
+            "fine-tuning on GLUE and SuperGLUE",
+            "capacity factor"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "d08eab82-ce8c-4126-8aee-d1681de9b6e5",
+        "properties": {
+          "page_content": "5 Conclusion\n\nWe propose a new routing method for sparsely activated mixture-of-experts (MoE) models. This method addresses load imbalance and under-utilization of experts in conventional MoE methods, and enables selecting different numbers of experts for each token. Our model demonstrates more than 2x training efficiency improvements when compared to the state-of-the-art GShard and Switch Transformer models, and also achieves strong gains when finetuning on 11 datasets in the GLUE and SuperGLUE benchmark.\n\n",
+          "title": "5 Conclusion",
+          "embedding": [
+            -0.014961956068873405,
+            -0.008189860731363297,
+            -0.023845616728067398,
+            0.002967504784464836,
+            -0.00690029701218009,
+            0.024916483089327812,
+            -0.0020832864101976156,
+            0.07137094438076019,
+            -0.014486853033304214,
+            0.04606231302022934,
+            -0.016666291281580925,
+            0.003244647989049554,
+            0.0073112985119223595,
+            -0.015520012937486172,
+            -0.04539867490530014,
+            0.030662959441542625,
+            0.012692022137343884,
+            0.0035896627232432365,
+            0.013717640191316605,
+            -0.03864166513085365,
+            -0.022352438420057297,
+            -0.008853496052324772,
+            -0.018159471452236176,
+            0.0027733161114156246,
+            0.019019179046154022,
+            -0.0032974369823932648,
+            0.000584922730922699,
+            0.008634798228740692,
+            -0.03864166513085365,
+            0.05327180400490761,
+            0.011213925667107105,
+            0.009728288277983665,
+            -0.0011019736994057894,
+            -0.001177386729978025,
+            -0.03257845342159271,
+            -0.007563932333141565,
+            0.07293953746557236,
+            0.025715861469507217,
+            -0.05375444516539574,
+            0.029727837070822716,
+            0.03912430629134178,
+            0.042442481964826584,
+            -0.03233712911605835,
+            -0.008084282279014587,
+            -0.014434064738452435,
+            -0.0042646098881959915,
+            -0.0013913713628426194,
+            -0.005045135505497456,
+            0.02479582279920578,
+            -0.01170411054044962,
+            0.0019154923502355814,
+            -0.010957521386444569,
+            0.014728175476193428,
+            0.0002771430881693959,
+            -0.04500652849674225,
+            0.03791769966483116,
+            0.018596867099404335,
+            0.017103686928749084,
+            0.03366440162062645,
+            0.007273592054843903,
+            -0.00380270485766232,
+            -0.01829521358013153,
+            -0.018657196313142776,
+            0.0008644224726594985,
+            0.003012752626091242,
+            0.00490185059607029,
+            -0.0039478749968111515,
+            -0.011915267445147038,
+            0.037223897874355316,
+            -0.010565373115241528,
+            -0.06045112758874893,
+            0.03665075823664665,
+            -0.04114537686109543,
+            0.0005467448499985039,
+            -0.0135818962007761,
+            -0.0067796362563967705,
+            0.0271637924015522,
+            0.014562266878783703,
+            -0.00047156744403764606,
+            0.001911721657961607,
+            0.012073635123670101,
+            0.00981124211102724,
+            0.0072698211297392845,
+            -0.007443271577358246,
+            0.05547386407852173,
+            -0.044704876840114594,
+            0.006198955699801445,
+            -0.05541353300213814,
+            0.03659042716026306,
+            -0.0024546959903091192,
+            -0.03007473796606064,
+            -0.021115664392709732,
+            -0.021779298782348633,
+            0.008619715459644794,
+            -0.0393354631960392,
+            0.033362749963998795,
+            0.006523231975734234,
+            -0.013197289779782295,
+            -0.008989240042865276,
+            0.021145828068256378,
+            -0.004758565686643124,
+            0.015218360349535942,
+            -0.015444600023329258,
+            0.008808248676359653,
+            0.05411642789840698,
+            -0.033151593059301376,
+            -0.019366079941391945,
+            0.044644545763731,
+            0.01785781793296337,
+            0.05526270717382431,
+            0.03119085170328617,
+            0.0022322272416204214,
+            0.01181722991168499,
+            0.01852145418524742,
+            -0.003199400147423148,
+            -0.028581559658050537,
+            -0.04585115611553192,
+            -0.05535320192575455,
+            0.023498717695474625,
+            -0.028551394119858742,
+            0.022548511624336243,
+            0.020633019506931305,
+            -0.009117442183196545,
+            -0.023845616728067398,
+            -0.03906397894024849,
+            0.01708860509097576,
+            0.0047095469199121,
+            -0.005278916098177433,
+            -0.012586443684995174,
+            -0.016877448186278343,
+            0.00349162588827312,
+            -0.01137983426451683,
+            -0.02776709757745266,
+            -0.03945612534880638,
+            0.016651209443807602,
+            -0.006040588021278381,
+            0.022186528891324997,
+            0.027917923405766487,
+            -0.020135292783379555,
+            -0.019350998103618622,
+            0.013732722960412502,
+            -0.04171851649880409,
+            0.030285894870758057,
+            0.01812930591404438,
+            -0.005037594120949507,
+            0.010459794662892818,
+            0.0033125197514891624,
+            0.053965602070093155,
+            -0.026213588193058968,
+            0.020844176411628723,
+            -0.04105488210916519,
+            0.04775156453251839,
+            -0.005836972501128912,
+            0.03984827175736427,
+            -0.024117104709148407,
+            -0.013838300481438637,
+            -0.01322745531797409,
+            -0.017556166276335716,
+            -0.006719305645674467,
+            -0.022020621225237846,
+            -0.0005962346913293004,
+            -0.0063196164555847645,
+            -0.0033030931372195482,
+            0.03646976873278618,
+            0.014072081074118614,
+            0.021628472954034805,
+            0.041507359594106674,
+            0.0015572801930829883,
+            -0.07173292338848114,
+            0.030828868970274925,
+            0.007684593088924885,
+            0.030919363722205162,
+            0.006839966867119074,
+            0.015897078439593315,
+            0.006066982634365559,
+            0.021387150511145592,
+            0.002356658922508359,
+            0.004947098437696695,
+            0.005754018202424049,
+            0.04135653376579285,
+            -0.02336297370493412,
+            0.001117998966947198,
+            0.015218360349535942,
+            0.012488407082855701,
+            0.03372473269701004,
+            0.023061320185661316,
+            0.016832200810313225,
+            0.015957407653331757,
+            0.004430518485605717,
+            0.0057163117453455925,
+            0.027118545025587082,
+            0.014901624992489815,
+            -0.0032898958306759596,
+            -0.020753679797053337,
+            0.0024188747629523277,
+            -0.024599747732281685,
+            -0.002049350645393133,
+            -0.01785781793296337,
+            0.034961506724357605,
+            0.0012782517587766051,
+            -0.026183422654867172,
+            -0.014984579756855965,
+            0.005836972501128912,
+            -0.03613794967532158,
+            -0.0010991457384079695,
+            0.0015563375782221556,
+            0.016952861100435257,
+            0.005870908498764038,
+            0.013981585390865803,
+            -0.03405654802918434,
+            0.012141506187617779,
+            -0.023679709061980247,
+            -0.006142395548522472,
+            0.004988575354218483,
+            -0.0007941940566524863,
+            0.013423528522253036,
+            -0.01253365445882082,
+            0.02022578939795494,
+            -0.005361870396882296,
+            -0.028837963938713074,
+            -0.007752465084195137,
+            0.02462991327047348,
+            0.004011976066976786,
+            -0.006097147706896067,
+            -0.010210931301116943,
+            -0.01613840088248253,
+            -0.030723290517926216,
+            -0.017948314547538757,
+            0.03155283257365227,
+            0.014449146576225758,
+            -0.015489847399294376,
+            0.020527441054582596,
+            0.007394252810627222,
+            0.0028939771000295877,
+            -0.014924249611794949,
+            -0.05559452623128891,
+            -0.009969609789550304,
+            -0.012797600589692593,
+            -0.000906370987650007,
+            0.0007253796211443841,
+            -0.04506685957312584,
+            0.02573094516992569,
+            -0.03674125298857689,
+            -0.0009040143340826035,
+            0.015791499987244606,
+            0.006938003934919834,
+            -0.006127313245087862,
+            0.011666404083371162,
+            -0.026590652763843536,
+            0.01303138118237257,
+            -0.030059656128287315,
+            -0.003970498684793711,
+            6.586861854884773e-05,
+            0.015173112042248249,
+            -0.023227229714393616,
+            0.034629687666893005,
+            0.021839629858732224,
+            -0.06274368613958359,
+            -0.008318062871694565,
+            -0.01835554465651512,
+            0.05456890910863876,
+            -0.01449439488351345,
+            -0.008144613355398178,
+            0.052306514233350754,
+            -0.009441718459129333,
+            0.002309525618329644,
+            0.031371843069791794,
+            0.027118545025587082,
+            0.01377042941749096,
+            -0.02897370606660843,
+            0.002942995633929968,
+            -0.07517176121473312,
+            -0.012488407082855701,
+            0.013415987603366375,
+            0.034086715430021286,
+            -0.03063279390335083,
+            0.0013838300947099924,
+            0.015474764630198479,
+            0.01895884983241558,
+            0.03170366212725639,
+            0.002475434448570013,
+            0.02963734231889248,
+            0.010452253744006157,
+            0.01011289469897747,
+            0.004140178207308054,
+            -0.004935786128044128,
+            -0.028340237215161324,
+            -0.0015544522320851684,
+            -0.03167349472641945,
+            0.0005425028502941132,
+            -0.008099365048110485,
+            -0.014094705693423748,
+            0.004355105571448803,
+            0.005403347313404083,
+            0.04651479050517082,
+            -0.006949315778911114,
+            -0.012563819997012615,
+            -0.01306154578924179,
+            -0.0002839773951563984,
+            -0.04144703224301338,
+            -0.0399991013109684,
+            0.014946873299777508,
+            0.005105465650558472,
+            -0.023709872737526894,
+            -0.0516730435192585,
+            -0.012880554422736168,
+            -0.022895412519574165,
+            0.035564810037612915,
+            -0.022729502990841866,
+            -0.012805141508579254,
+            0.0416581891477108,
+            -0.003783851396292448,
+            0.03731439262628555,
+            -0.057494934648275375,
+            -0.008604632690548897,
+            0.006545855663716793,
+            -0.005260062403976917,
+            0.014841294847428799,
+            -0.026424745097756386,
+            -0.024056773632764816,
+            -0.010361758060753345,
+            -0.015180653892457485,
+            -0.04760073870420456,
+            0.0017759781330823898,
+            -0.02028612047433853,
+            0.017118770629167557,
+            0.046152807772159576,
+            0.03052721731364727,
+            -0.02469024434685707,
+            0.014072081074118614,
+            -0.008385934866964817,
+            -0.054689567536115646,
+            -0.025595201179385185,
+            0.010097811929881573,
+            -0.020738597959280014,
+            -0.041235875338315964,
+            -0.02043694630265236,
+            0.0005128089687786996,
+            -0.022669173777103424,
+            0.016168564558029175,
+            -0.024539416655898094,
+            -0.03360407054424286,
+            0.000932294235099107,
+            0.0206631850451231,
+            -0.0028920918703079224,
+            -0.017224349081516266,
+            -0.02153797633945942,
+            -0.02562536671757698,
+            0.013793053105473518,
+            -0.004573803395032883,
+            0.0009091990068554878,
+            0.019592318683862686,
+            0.03254828602075577,
+            0.025534870103001595,
+            0.0035406441893428564,
+            -0.03333258256316185,
+            -0.021191077306866646,
+            0.026032596826553345,
+            0.0018824990838766098,
+            -0.0013084170641377568,
+            0.03245779126882553,
+            -0.03653009608387947,
+            -0.04325694590806961,
+            0.005554173607379198,
+            -0.04910900071263313,
+            0.0020418092608451843,
+            -0.021462563425302505,
+            -0.022503264248371124,
+            -0.013898631557822227,
+            0.002277475083246827,
+            0.003847952699288726,
+            0.02909436821937561,
+            0.011658863164484501,
+            -0.010482418350875378,
+            -0.023438386619091034,
+            0.008521678857505322,
+            0.0483548678457737,
+            -0.010060105472803116,
+            -0.02710346318781376,
+            0.03278961032629013,
+            -0.016455134376883507,
+            0.01168148685246706,
+            -0.011425082571804523,
+            -0.022895412519574165,
+            -0.01592724397778511,
+            0.017827652394771576,
+            0.01895884983241558,
+            0.006172561086714268,
+            -0.041235875338315964,
+            0.025474539026618004,
+            -0.0063497815281152725,
+            -0.03840034082531929,
+            0.014056999236345291,
+            0.004128866363316774,
+            0.01239791139960289,
+            0.0005486301961354911,
+            0.013174666091799736,
+            -0.054297421127557755,
+            -0.005844513885676861,
+            0.006036817561835051,
+            -0.03191481530666351,
+            -0.0058595966547727585,
+            -0.008378393948078156,
+            0.02280491590499878,
+            -0.014946873299777508,
+            0.015391810797154903,
+            -0.012405452318489552,
+            0.008144613355398178,
+            0.031975146383047104,
+            -0.03441853076219559,
+            -0.015067534521222115,
+            -0.036228444427251816,
+            -0.0041929674334824085,
+            -0.022458016872406006,
+            0.020361533388495445,
+            0.04334744065999985,
+            -0.017661744728684425,
+            0.00917777232825756,
+            0.0028185639530420303,
+            -0.004241986200213432,
+            -0.022005537524819374,
+            0.016817117109894753,
+            0.0569821260869503,
+            0.031100355088710785,
+            -0.008039034903049469,
+            -0.0006443105521611869,
+            0.02197537198662758,
+            -0.013121876865625381,
+            -0.008340687490999699,
+            0.006051899865269661,
+            0.017752239480614662,
+            0.010701116174459457,
+            0.02286524698138237,
+            -0.040089596062898636,
+            -0.010150601156055927,
+            0.009109900332987309,
+            -0.02469024434685707,
+            0.027812344953417778,
+            -0.03073837235569954,
+            0.018973931670188904,
+            0.017812570556998253,
+            -0.01956215314567089,
+            0.013627144508063793,
+            -0.010022399015724659,
+            -0.023830534890294075,
+            -0.02512763999402523,
+            -0.02496173046529293,
+            -0.008634798228740692,
+            0.017978480085730553,
+            -0.0139966681599617,
+            -0.05287965387105942,
+            0.04301562160253525,
+            -0.06298501044511795,
+            -0.029773086309432983,
+            0.03191481530666351,
+            -0.02027103677392006,
+            -0.014313403517007828,
+            0.008672504685819149,
+            0.01570100337266922,
+            -0.033362749963998795,
+            0.020452028140425682,
+            0.011643780395388603,
+            0.033151593059301376,
+            -0.028023501858115196,
+            -0.024931564927101135,
+            0.0034765431191772223,
+            -0.02739003300666809,
+            -0.006010422948747873,
+            -0.0002648884546943009,
+            0.014667844399809837,
+            0.00637617614120245,
+            -0.033754896372556686,
+            -0.009743370115756989,
+            0.003800819395110011,
+            0.0027506921906024218,
+            0.03348340839147568,
+            -0.004132636822760105,
+            -0.04205033555626869,
+            0.014170118607580662,
+            0.004321169573813677,
+            -0.03284993767738342,
+            0.006406341679394245,
+            0.003346455516293645,
+            -0.009185313247144222,
+            -0.016334474086761475,
+            0.005478760693222284,
+            0.03221647068858147,
+            0.030210481956601143,
+            0.017812570556998253,
+            0.007118995301425457,
+            0.009871572256088257,
+            -0.01338582206517458,
+            -0.01503736898303032,
+            0.010655868798494339,
+            -0.007782630156725645,
+            -0.03191481530666351,
+            -0.013204830698668957,
+            -0.014780964702367783,
+            0.025866687297821045,
+            -0.012563819997012615,
+            -0.01253365445882082,
+            0.005980257410556078,
+            0.045157354325056076,
+            0.014615056104958057,
+            -0.005267603788524866,
+            -0.01336319837719202,
+            0.007601638790220022,
+            0.012707104906439781,
+            -0.034358199685811996,
+            -0.022277025505900383,
+            0.022065868601202965,
+            0.0023283790796995163,
+            0.017375174909830093,
+            -0.024599747732281685,
+            -0.022216694429516792,
+            0.03239746019244194,
+            -0.03248795494437218,
+            0.030919363722205162,
+            0.01580658182501793,
+            0.006202726159244776,
+            -0.006444048136472702,
+            0.004739712458103895,
+            -0.014569807797670364,
+            0.028400568291544914,
+            -0.029350772500038147,
+            0.027193957939743996,
+            0.020421862602233887,
+            -0.006285680457949638,
+            0.060300301760435104,
+            -0.029682589694857597,
+            -0.016334474086761475,
+            0.009607627056539059,
+            -0.0043777297250926495,
+            0.015391810797154903,
+            -0.019094591960310936,
+            -0.007149160373955965,
+            -0.0060745240189135075,
+            0.005018740892410278,
+            -0.029667507857084274,
+            -0.015897078439593315,
+            0.02164355479180813,
+            -0.005987798795104027,
+            -0.0013272702926769853,
+            0.009660416282713413,
+            -0.02286524698138237,
+            0.017495835199952126,
+            -0.006542085204273462,
+            -0.04533834755420685,
+            -0.00045813448377884924,
+            0.02153797633945942,
+            -0.024388590827584267,
+            -0.015912160277366638,
+            -0.008582009002566338,
+            0.0038837739266455173,
+            -0.00595009233802557,
+            -0.02280491590499878,
+            -0.019245419651269913,
+            -0.005655981134623289,
+            -0.008385934866964817,
+            -0.027480527758598328,
+            -0.011779523454606533,
+            0.034086715430021286,
+            0.01294842641800642,
+            -0.004875455982983112,
+            -0.010301426984369755,
+            -0.029667507857084274,
+            0.016002656891942024,
+            0.04373958706855774,
+            0.0034218686632812023,
+            0.006941774394363165,
+            -0.0065948739647865295,
+            -0.0026451139710843563,
+            -0.014841294847428799,
+            0.009864031337201595,
+            0.0017646661726757884,
+            -0.0028223346453160048,
+            0.01790306530892849,
+            -0.011342127807438374,
+            -0.03381522744894028,
+            -0.026967719197273254,
+            0.04304578900337219,
+            -0.023785287514328957,
+            0.004483307711780071,
+            0.04651479050517082,
+            0.036620594561100006,
+            -0.005343017168343067,
+            -0.014305861666798592,
+            -0.03107019141316414,
+            0.0020663184113800526,
+            -0.02010512910783291,
+            0.009509589523077011,
+            0.03294043615460396,
+            -0.00039686134550720453,
+            -0.034297872334718704,
+            -0.013529106974601746,
+            -0.02099500223994255,
+            -0.010346675291657448,
+            0.015716087073087692,
+            0.017013192176818848,
+            -0.034358199685811996,
+            0.005795495584607124,
+            0.018310297280550003,
+            0.028943542391061783,
+            -0.011349668726325035,
+            -0.016983026638627052,
+            0.024765657261013985,
+            0.017224349081516266,
+            0.0009294662741012871,
+            -0.0029222569428384304,
+            -0.00034312953357584774,
+            0.0068098013289272785,
+            -0.004068535752594471,
+            0.033030930906534195,
+            0.0035274471156299114,
+            0.011206384748220444,
+            -0.02116091176867485,
+            0.00479250168427825,
+            0.020738597959280014,
+            -0.006870131939649582,
+            -0.001355550135485828,
+            0.07384449243545532,
+            0.00214173155836761,
+            0.01355173159390688,
+            0.017209265381097794,
+            0.03444869816303253,
+            0.01773715764284134,
+            -0.017993561923503876,
+            -0.03170366212725639,
+            -0.014313403517007828,
+            -0.007718529086560011,
+            -0.020527441054582596,
+            -0.0049697221256792545,
+            -0.012910719960927963,
+            -0.0433172769844532,
+            -0.0223222728818655,
+            -0.0033502262085676193,
+            -0.002466007834300399,
+            0.025263383984565735,
+            -0.0026319166645407677,
+            0.0009822554420679808,
+            -0.027948088943958282,
+            0.03565530478954315,
+            0.02622867003083229,
+            0.023996442556381226,
+            -0.007880667224526405,
+            0.011100806295871735,
+            -0.0043362523429095745,
+            -0.008491513319313526,
+            0.02754085883498192,
+            0.0074885194189846516,
+            -0.008137071505188942,
+            0.03478051349520683,
+            0.012111341580748558,
+            0.05272882804274559,
+            -0.013114335015416145,
+            -0.024343343451619148,
+            -0.017179099842905998,
+            -0.04018009081482887,
+            0.011372293345630169,
+            -0.007710987702012062,
+            0.0001368040539091453,
+            0.014139953069388866,
+            0.017390256747603416,
+            -0.008167237043380737,
+            0.0271637924015522,
+            0.0038498379290103912,
+            -0.015504930168390274,
+            0.025187969207763672,
+            -0.02892845869064331,
+            0.014253072440624237,
+            -0.0035576121881604195,
+            0.011839854530990124,
+            -0.012903179042041302,
+            0.011538201943039894,
+            0.02358921244740486,
+            -0.025761108845472336,
+            0.028551394119858742,
+            0.017254512757062912,
+            -0.0027412655763328075,
+            0.00480758398771286,
+            0.0026168341282755136,
+            -0.0012169786496087909,
+            -0.022080950438976288,
+            -0.034086715430021286,
+            0.02533879689872265,
+            -0.007473436649888754,
+            -0.023332808166742325,
+            0.02425284869968891,
+            -0.017118770629167557,
+            -0.008928908966481686,
+            -0.0017175329849123955,
+            -0.009358763694763184,
+            -0.007579015102237463,
+            0.04292512685060501,
+            -0.008664963766932487,
+            -0.004291004501283169,
+            -0.01640988700091839,
+            0.02689230628311634,
+            0.04226149246096611,
+            -0.008385934866964817,
+            -0.009072193875908852,
+            0.021899959072470665,
+            -0.012450700625777245,
+            0.0015129749663174152,
+            0.041628021746873856,
+            0.006153707858175039,
+            0.0015365416184067726,
+            0.005712541285902262,
+            -0.03393588960170746,
+            -0.029999325051903725,
+            0.005806807428598404,
+            0.016455134376883507,
+            0.016047904267907143,
+            0.04126603901386261,
+            0.03646976873278618,
+            -0.014969496987760067,
+            0.0025037142913788557,
+            0.0007546021952293813,
+            -0.035232994705438614,
+            0.028747467324137688,
+            -0.005486302077770233,
+            0.021447481587529182,
+            0.02485615201294422,
+            -0.026183422654867172,
+            -0.036620594561100006,
+            -0.00034996384056285024,
+            0.030210481956601143,
+            0.0190040972083807,
+            0.014056999236345291,
+            0.014072081074118614,
+            0.023227229714393616,
+            -0.020316284149885178,
+            -0.024946648627519608,
+            0.02012021094560623,
+            0.02506730891764164,
+            -0.009456800296902657,
+            0.01074636448174715,
+            0.017933230847120285,
+            0.009818783961236477,
+            -0.0012282906100153923,
+            0.03134167939424515,
+            0.009019404649734497,
+            0.016289226710796356,
+            0.008287898264825344,
+            -0.011515578255057335,
+            0.02500697784125805,
+            0.010271262377500534,
+            0.03393588960170746,
+            -0.012412993237376213,
+            0.003929021768271923,
+            0.003689585020765662,
+            -0.004087388981133699,
+            0.037163566797971725,
+            0.005946321878582239,
+            -0.04307595267891884,
+            0.0199995506554842,
+            0.011960514821112156,
+            -0.0001776920835254714,
+            -0.006489295978099108,
+            -0.0026469992008060217,
+            0.018446041271090508,
+            0.02043694630265236,
+            0.018038809299468994,
+            0.009532214142382145,
+            0.005659752059727907,
+            0.005618274677544832,
+            0.006138625089079142,
+            0.03167349472641945,
+            0.0020455799531191587,
+            -0.01978839375078678,
+            -0.00230009900406003,
+            0.005505155306309462,
+            0.021115664392709732,
+            -0.019637567922472954,
+            0.01653054729104042,
+            0.019215254113078117,
+            -0.0011934119975194335,
+            -0.01746566966176033,
+            -0.03671108931303024,
+            -0.007303757127374411,
+            -0.013853383250534534,
+            0.013898631557822227,
+            0.008951532654464245,
+            0.018687361851334572,
+            -0.0221563633531332,
+            0.016078069806098938,
+            0.019592318683862686,
+            -0.01950182393193245,
+            0.006496837362647057,
+            0.008061658591032028,
+            -0.014524560421705246,
+            -0.005580568220466375,
+            -0.016108235344290733,
+            0.002053121104836464,
+            0.06382963061332703,
+            -0.014464229345321655,
+            0.009954527020454407,
+            0.024931564927101135,
+            -0.009879114106297493,
+            -0.030119985342025757,
+            0.0009671727893874049,
+            -0.010429629124701023,
+            -0.002450925298035145,
+            0.024373508989810944,
+            -0.03577596694231033,
+            -0.04319661483168602,
+            -0.005290227942168713,
+            -0.03631894290447235,
+            -0.006560938432812691,
+            0.019034262746572495,
+            0.027948088943958282,
+            -0.014162576757371426,
+            -0.02760118804872036,
+            0.02292557805776596,
+            -0.0020418092608451843,
+            0.009765994735062122,
+            -0.019471658393740654,
+            -0.026545405387878418,
+            -0.03030097670853138,
+            -0.03212597221136093,
+            -0.00961516797542572,
+            -0.025097474455833435,
+            0.008408558554947376,
+            0.0012725958367809653,
+            -0.05061726272106171,
+            -0.030602630227804184,
+            0.010670951567590237,
+            -0.002435842528939247,
+            0.04005942866206169,
+            0.039757777005434036,
+            0.003970498684793711,
+            0.028008420020341873,
+            -0.03613794967532158,
+            0.050828419625759125,
+            -0.006530772894620895,
+            0.03592679277062416,
+            -0.007579015102237463,
+            0.0019890200346708298,
+            -0.014358650892972946,
+            0.04379991814494133,
+            0.000892702373676002,
+            0.002547076903283596,
+            -0.038762323558330536,
+            0.0021455022506415844,
+            -0.023996442556381226,
+            -0.00025451916735619307,
+            -0.0026752790436148643,
+            0.008996780961751938,
+            -0.0009907394414767623,
+            -0.014856377616524696,
+            0.002102139638736844,
+            0.03345324471592903,
+            0.02616834081709385,
+            -0.021779298782348633,
+            -0.027088379487395287,
+            -0.030617712065577507,
+            0.0004138292861171067,
+            -0.008725293911993504,
+            -0.002880779793485999,
+            -0.004860373213887215,
+            -0.024313177913427353,
+            0.014396357350051403,
+            -0.016771869733929634,
+            -0.013415987603366375,
+            -0.002205832628533244,
+            0.02545945718884468,
+            -0.01746566966176033,
+            -0.02259376086294651,
+            -0.005814348813146353,
+            -0.029335688799619675,
+            -0.01675678789615631,
+            0.00022930291015654802,
+            -0.02485615201294422,
+            0.017510918900370598,
+            0.01609315164387226,
+            -0.012103799730539322,
+            0.0002860983950085938,
+            0.0022001767065376043,
+            0.016123317182064056,
+            -0.021794380620121956,
+            -0.004166572820395231,
+            0.0025357650592923164,
+            -0.00017439275688957423,
+            -0.023151816800236702,
+            0.021387150511145592,
+            -0.019411327317357063,
+            0.01631939224898815,
+            -0.04844536632299423,
+            -0.011213925667107105,
+            -0.025037143379449844,
+            0.010587996803224087,
+            0.001273538451641798,
+            -0.02990882843732834,
+            -0.014072081074118614,
+            0.00785050168633461,
+            -0.0031447256915271282,
+            -0.02182454615831375,
+            0.008506596088409424,
+            0.03541398420929909,
+            -0.0014385045506060123,
+            0.009683039970695972,
+            -0.029773086309432983,
+            -0.011590991169214249,
+            -0.010158142074942589,
+            0.03079870343208313,
+            0.0037574570160359144,
+            -0.005131860263645649,
+            -0.00574270635843277,
+            0.0005769100971519947,
+            -0.03472018241882324,
+            -0.021899959072470665,
+            -0.003259730525314808,
+            -0.002062547951936722,
+            -0.005392035469412804,
+            0.034509025514125824,
+            0.03405654802918434,
+            0.03686191514134407,
+            -0.04379991814494133,
+            0.019215254113078117,
+            -0.008325604721903801,
+            -0.011108347214758396,
+            0.01325007900595665,
+            0.018777858465909958,
+            0.01840079203248024,
+            -0.016485299915075302,
+            0.001463956548832357,
+            0.003412442049011588,
+            0.013536648824810982,
+            -0.005603192374110222,
+            0.022985907271504402,
+            -0.00719063775613904,
+            0.013408446684479713,
+            0.014260614290833473,
+            -8.476607035845518e-06,
+            0.022955743595957756,
+            -0.006843737326562405,
+            0.009901737794280052,
+            -0.007043581921607256,
+            -0.014094705693423748,
+            0.0017872900934889913,
+            -0.03882265463471413,
+            -0.013016298413276672,
+            0.015384268946945667,
+            0.018777858465909958,
+            -0.012495948001742363,
+            -0.04874701797962189,
+            0.022020621225237846,
+            0.03671108931303024,
+            -0.019275585189461708,
+            0.0273297019302845,
+            -0.0279933363199234,
+            0.016334474086761475,
+            0.011613614857196808,
+            0.013272702693939209,
+            0.004773647990077734,
+            0.020738597959280014,
+            -0.02512763999402523,
+            0.033966053277254105,
+            -0.0019550842698663473,
+            0.025761108845472336,
+            0.016847282648086548,
+            -0.015391810797154903,
+            -0.04573049396276474,
+            -0.016108235344290733,
+            0.006753241643309593,
+            -0.012028386816382408,
+            -0.002897747792303562,
+            -0.043498266488313675,
+            -0.03420737385749817,
+            0.030210481956601143,
+            0.01418520137667656,
+            -0.0493503212928772,
+            0.00014446319255512208,
+            -0.014034374617040157,
+            -0.0030730832368135452,
+            -0.0052261268720030785,
+            -0.028943542391061783,
+            -0.02336297370493412,
+            0.00801641121506691,
+            0.00752245495095849,
+            0.03981810808181763,
+            -0.023046238347887993,
+            0.026741480454802513,
+            -0.01159853208810091,
+            -0.01719418354332447,
+            -0.02005987986922264,
+            -0.00045224284986034036,
+            -0.021326819434762,
+            -0.01481867115944624,
+            0.005094153806567192,
+            0.020738597959280014,
+            0.009660416282713413,
+            0.029396019876003265,
+            -0.012722187675535679,
+            -0.014856377616524696,
+            -0.0005095096421428025,
+            -0.03465985506772995,
+            -0.028506144881248474,
+            0.007032270077615976,
+            0.0006749471067450941,
+            -0.017269596457481384,
+            -0.02070843242108822,
+            -0.03704290837049484,
+            0.016213813796639442,
+            0.011990680359303951,
+            0.01692269556224346,
+            0.03245779126882553,
+            0.0198336411267519,
+            -0.01386092510074377,
+            0.01669645681977272,
+            -0.007888208143413067,
+            0.0009813127107918262,
+            0.01013551838696003,
+            0.018581783398985863,
+            -0.013114335015416145,
+            -0.0027733161114156246,
+            -0.014765881933271885,
+            -0.029003871604800224,
+            -0.01708860509097576,
+            -0.01806897483766079,
+            -0.0008795050671324134,
+            -0.006149936933070421,
+            -0.03891315311193466,
+            -0.013415987603366375,
+            0.024991896003484726,
+            0.02396627888083458,
+            -0.01534656248986721,
+            -0.007511143106967211,
+            -0.00023295573191717267,
+            -0.0208290945738554,
+            -0.004894309211522341,
+            -0.02342330478131771,
+            -0.003672617254778743,
+            -0.024147270247340202,
+            0.013234996236860752,
+            -0.0045813447795808315,
+            0.010052564553916454,
+            -0.053694114089012146,
+            -0.013174666091799736,
+            0.007326381281018257,
+            -0.010105352848768234,
+            -0.0016996223712339997,
+            0.01746566966176033,
+            -0.003291781060397625,
+            -0.009162689559161663,
+            -0.023845616728067398,
+            0.014155035838484764,
+            0.005772871430963278,
+            0.021025167778134346,
+            -0.002831761259585619,
+            -0.00406476529315114,
+            0.03348340839147568,
+            0.025384044274687767,
+            -5.541094287764281e-05,
+            -0.012269708327949047,
+            0.015897078439593315,
+            0.014788505621254444,
+            0.0014507592422887683,
+            0.0044493721798062325,
+            0.009102359414100647,
+            0.0048264372162520885,
+            -0.010889649391174316,
+            -0.00480758398771286,
+            0.024871235713362694,
+            0.006527002435177565,
+            0.005935009568929672,
+            -0.01463013794273138,
+            0.02897370606660843,
+            0.03752554953098297,
+            0.0196677315980196,
+            0.028068749234080315,
+            -0.015685921534895897,
+            -0.011032934300601482,
+            -0.007760006468743086,
+            -0.013762887567281723,
+            0.031432174146175385,
+            0.008861036971211433,
+            -0.009871572256088257,
+            -0.0104145472869277,
+            0.010037481784820557,
+            -0.009939444251358509,
+            0.002166240941733122,
+            0.02269933745265007,
+            0.009592544287443161,
+            0.005580568220466375,
+            -0.00028892638511024415,
+            0.007548849564045668,
+            0.015912160277366638,
+            0.023770203813910484,
+            -0.02975800260901451,
+            -0.027420196682214737,
+            -0.0009120269678533077,
+            0.0003007096820510924,
+            0.012993674725294113,
+            0.01767682656645775,
+            -0.0011594761162996292,
+            0.006658975500613451,
+            0.04334744065999985,
+            -0.015746252611279488,
+            -0.0034369511995464563,
+            -0.014698009938001633,
+            -0.013272702693939209,
+            -0.0022491952404379845,
+            -0.021945208311080933,
+            0.008687587454915047,
+            0.012382828630506992,
+            0.017118770629167557,
+            0.014245531521737576,
+            -0.008023952133953571,
+            0.012322497554123402,
+            0.018446041271090508,
+            0.011515578255057335,
+            -0.004219362046569586,
+            -0.014780964702367783,
+            -0.002564044902101159,
+            0.015248525887727737,
+            0.009034487418830395,
+            0.022835081443190575,
+            0.021326819434762,
+            0.006949315778911114,
+            0.01322745531797409,
+            -0.025052227079868317,
+            0.025746027007699013,
+            0.0144114401191473,
+            0.02419251762330532,
+            -0.03770654276013374,
+            -0.021145828068256378,
+            0.014501935802400112,
+            -0.007141618989408016,
+            -0.0004892423748970032,
+            0.015384268946945667,
+            -0.008091824129223824,
+            0.05125073343515396,
+            -0.0353536531329155,
+            0.00845380686223507,
+            0.02358921244740486,
+            -0.015881994739174843,
+            0.010271262377500534,
+            0.01818963512778282,
+            0.010595538653433323,
+            -0.010678492486476898,
+            -0.02093467302620411,
+            0.00037730109761469066,
+            0.02551978826522827,
+            -0.03818918392062187,
+            -0.018611948937177658,
+            -0.0065345438197255135,
+            -0.0004892423748970032,
+            0.022729502990841866,
+            -0.007552620489150286,
+            0.014645220711827278,
+            -0.008928908966481686,
+            0.011432623490691185,
+            0.001585560035891831,
+            -0.03134167939424515,
+            -0.007537537720054388,
+            0.014637679792940617,
+            0.01231495663523674,
+            -0.009667957201600075,
+            -0.006674057804048061,
+            -0.0010331592056900263,
+            -0.002075745025649667,
+            -0.014268155209720135,
+            0.02722412347793579,
+            0.0018617605092003942,
+            0.01592724397778511,
+            -0.00208705710247159,
+            -0.014207825064659119,
+            0.018551617860794067,
+            -0.01962248422205448,
+            -0.015256066806614399,
+            -0.004521014168858528,
+            0.001125540235079825,
+            -0.0011905840365216136,
+            -0.031100355088710785,
+            0.015361645258963108,
+            -0.013257619924843311,
+            0.034629687666893005,
+            0.006180102005600929,
+            -0.0012207493418827653,
+            -0.002481090370565653,
+            -0.011606073938310146,
+            -0.0056145042181015015,
+            -0.02821957692503929,
+            -0.017872901633381844,
+            0.01686236634850502,
+            -0.013536648824810982,
+            -0.009909279644489288,
+            -0.026515239849686623,
+            -0.021734051406383514,
+            -0.011040475219488144,
+            0.008793165907263756,
+            0.0014347339747473598,
+            0.022850165143609047,
+            0.00864988099783659,
+            -0.017375174909830093,
+            -0.0033671942073851824,
+            -0.041959840804338455,
+            -0.002809137338772416,
+            0.01033159252256155,
+            -0.017541082575917244,
+            -0.022246859967708588,
+            0.013204830698668957,
+            0.00048052275087684393,
+            -0.018596867099404335,
+            0.019411327317357063,
+            0.0019183203112334013,
+            0.004434289410710335,
+            0.0017147050239145756,
+            -0.0023151817731559277,
+            -0.01653054729104042,
+            0.00573139451444149,
+            0.009554837830364704,
+            0.01536918617784977,
+            -0.047238755971193314,
+            -0.007499831262975931,
+            0.003244647989049554,
+            0.006372405681759119,
+            0.006010422948747873,
+            0.01118376012891531,
+            0.004777418915182352,
+            -0.017767323181033134,
+            0.004471995867788792,
+            0.01906442828476429,
+            -0.022563595324754715,
+            0.01977331005036831,
+            -0.0019051230046898127,
+            0.0027883988805115223,
+            0.002799710724502802,
+            -0.007514914032071829,
+            -0.005490072537213564,
+            0.02583652175962925,
+            -0.010452253744006157,
+            0.009622709825634956,
+            0.021507810801267624,
+            0.02309148572385311,
+            -0.011470329947769642,
+            -0.01862703263759613,
+            0.013710098341107368,
+            0.0316433310508728,
+            0.0006923863547854125,
+            -0.014992120675742626,
+            -0.054086264222860336,
+            -0.011206384748220444,
+            0.03218630328774452,
+            0.015897078439593315,
+            0.017390256747603416,
+            -0.006941774394363165,
+            -0.012518571689724922,
+            0.019139841198921204,
+            0.023559046909213066,
+            0.017043357715010643,
+            -0.026545405387878418,
+            0.013928796164691448,
+            -0.020844176411628723,
+            0.03432803601026535,
+            0.0030429179314523935,
+            -0.008008869364857674,
+            0.004143949132412672,
+            -0.013431070372462273,
+            -0.02313673496246338,
+            0.014200284145772457,
+            -0.003846067236736417,
+            0.0028826650232076645,
+            0.006545855663716793,
+            -0.004864144138991833,
+            0.01939624547958374,
+            -0.0008766771061345935,
+            -0.005897303111851215,
+            0.004747253842651844,
+            0.02787267602980137,
+            0.0013659194810315967,
+            0.003993122838437557,
+            0.0011085723526775837,
+            0.010090271010994911,
+            -0.0019946759566664696,
+            -0.012586443684995174,
+            0.022563595324754715,
+            -0.012865472584962845,
+            -0.0056145042181015015,
+            0.01923033595085144,
+            -0.007903290912508965,
+            -0.00023342706845141947,
+            -0.010452253744006157,
+            -0.008838413283228874,
+            -0.03152266889810562,
+            0.026183422654867172,
+            -0.0036066307220607996,
+            0.014396357350051403,
+            0.021010085940361023,
+            -0.0012094373814761639,
+            -0.005410888697952032,
+            0.022337354719638824,
+            0.0002851557219401002,
+            -0.007458353880792856,
+            0.01386092510074377,
+            -0.02562536671757698,
+            -0.02612309157848358,
+            0.03858133405447006,
+            -0.012269708327949047,
+            0.0005542861763387918,
+            -0.0022491952404379845,
+            -0.00906465295702219,
+            -0.007511143106967211,
+            0.020542524755001068,
+            0.01708860509097576,
+            -0.02793300710618496,
+            -0.010595538653433323,
+            -0.010127977468073368,
+            -0.02909436821937561,
+            0.011590991169214249,
+            -0.0018532765097916126,
+            0.02153797633945942,
+            -0.005799266044050455,
+            -0.03155283257365227,
+            -0.004762336146086454,
+            0.03592679277062416,
+            -0.003981810994446278,
+            0.0010048793628811836,
+            0.015323938801884651,
+            0.0028675824869424105,
+            0.015610508620738983,
+            -0.029456350952386856,
+            0.025474539026618004,
+            -0.01640988700091839,
+            0.020255954936146736,
+            0.013483859598636627,
+            -0.014464229345321655,
+            0.027736932039260864,
+            0.013234996236860752,
+            0.005493842996656895,
+            0.006413882598280907,
+            0.0030052114743739367,
+            -0.010859483852982521,
+            -0.007077517919242382,
+            0.008340687490999699,
+            -0.002134190173819661,
+            0.0003591548011172563,
+            0.02594210021197796,
+            0.0003921480383723974,
+            0.013762887567281723,
+            0.001332926214672625,
+            -0.0001895932073239237,
+            0.0017618382116779685,
+            -0.002952422248199582,
+            -0.01944149285554886,
+            0.011304421350359917,
+            -0.015912160277366638,
+            -0.020964836701750755,
+            0.012443158775568008,
+            -0.017269596457481384,
+            -0.01785781793296337,
+            -0.028113998472690582,
+            0.01283530704677105,
+            -0.008861036971211433,
+            -0.007337693125009537,
+            0.009667957201600075,
+            0.011764440685510635,
+            0.015595425851643085,
+            0.02660573646426201,
+            0.019863806664943695,
+            -0.012744811363518238,
+            0.013762887567281723,
+            0.010489960201084614,
+            0.008830872364342213,
+            -0.00939647015184164,
+            0.0018146273214370012,
+            -0.010308968834578991,
+            0.020240871235728264,
+            0.010912273079156876,
+            0.004038370680063963,
+            0.019200170412659645,
+            -0.007669510785490274,
+            -0.01264677383005619,
+            -0.03505200147628784,
+            -0.004882997367531061,
+            -0.001891925698146224,
+            0.007262280210852623,
+            0.04998379200696945,
+            -0.014245531521737576,
+            -0.03300076723098755,
+            -0.016847282648086548,
+            -0.011802148073911667,
+            0.009645333513617516,
+            0.0012443158775568008,
+            0.010723740793764591,
+            -0.013491400517523289,
+            -0.035564810037612915,
+            0.0030825098510831594,
+            0.015776416286826134,
+            0.003531217575073242,
+            -0.0020041028037667274,
+            0.022337354719638824,
+            -0.028113998472690582,
+            0.0180086437612772,
+            -0.007514914032071829,
+            -0.011470329947769642,
+            -0.02088942378759384,
+            -0.0033671942073851824,
+            -0.0076581984758377075,
+            0.023845616728067398,
+            -0.005497613921761513,
+            -0.01978839375078678,
+            -0.003457689890637994,
+            0.011357210576534271,
+            -0.0009516188292764127,
+            -0.005659752059727907,
+            0.004928245209157467,
+            0.006406341679394245,
+            -0.0029769313987344503,
+            0.033905722200870514,
+            -0.0017128196777775884,
+            -0.0024792051408439875,
+            -0.0063196164555847645,
+            -0.0027733161114156246,
+            0.00947942491620779,
+            0.015022286213934422,
+            -0.047087930142879486,
+            -0.019878888502717018,
+            -0.007254738826304674,
+            -0.004079848062247038,
+            0.0051544844172894955,
+            0.012722187675535679,
+            -0.02104024961590767,
+            -0.00375934224575758,
+            -0.011191301979124546,
+            -0.008536760695278645,
+            -0.029516680166125298,
+            -0.010889649391174316,
+            0.03478051349520683,
+            -0.009426635690033436,
+            0.039094142615795135,
+            0.021130746230483055,
+            -0.0020436944905668497,
+            0.015052451752126217,
+            0.00261117797344923,
+            0.002690361812710762,
+            -0.00906465295702219,
+            -0.013936338014900684,
+            -0.0019626254215836525,
+            -0.005486302077770233,
+            0.0025489621330052614,
+            -0.03960695117712021,
+            0.0004977263743057847,
+            -0.0033276022877544165,
+            0.016998108476400375,
+            -0.035685472190380096,
+            -0.008891202509403229,
+            0.026982801035046577,
+            -0.012035928666591644,
+            0.005026282276958227,
+            -0.003996893297880888,
+            0.01653054729104042,
+            0.042231328785419464,
+            0.011455247178673744,
+            -0.003372850129380822,
+            -0.017782405018806458,
+            -0.0002785570686683059,
+            0.004988575354218483,
+            -0.03846067190170288,
+            0.004732171073555946,
+            -0.039697445929050446,
+            0.030180316418409348,
+            -0.00679094810038805,
+            -0.008333145640790462,
+            0.03541398420929909,
+            -0.018038809299468994,
+            0.01473571639508009,
+            0.009879114106297493,
+            -0.022186528891324997,
+            0.02693755365908146,
+            0.002552732825279236,
+            0.018551617860794067,
+            -0.04147719591856003,
+            0.00793345645070076,
+            -0.008295439183712006,
+            0.00950204860419035,
+            -0.01642496883869171,
+            -0.0017646661726757884,
+            0.016575796529650688,
+            -0.010127977468073368,
+            0.014449146576225758,
+            0.020135292783379555,
+            -0.001117998966947198,
+            0.0006494951667264104,
+            0.021899959072470665,
+            -0.006527002435177565,
+            0.0060330466367304325,
+            0.015489847399294376,
+            0.013868466019630432,
+            0.01346877682954073,
+            0.037827201187610626,
+            -0.002782742725685239,
+            0.0012320613022893667,
+            0.003993122838437557,
+            0.001916434965096414,
+            0.010241096839308739,
+            -0.0067796362563967705,
+            -0.001194354728795588,
+            -0.016605960205197334,
+            0.00782033707946539,
+            0.017013192176818848,
+            -0.006379947066307068,
+            -0.001372518134303391,
+            -0.030376389622688293,
+            -0.010037481784820557,
+            -0.04871685057878494,
+            0.006112230475991964,
+            -0.030919363722205162,
+            -0.010150601156055927,
+            -0.008416100405156612,
+            0.018385710194706917,
+            -0.003538758959621191,
+            0.012412993237376213,
+            0.001619496033526957,
+            -0.019984466955065727,
+            -0.005135631188750267,
+            0.008310521952807903,
+            -0.015059992671012878,
+            0.02325739525258541,
+            0.0071680136024951935,
+            -0.006440277211368084,
+            0.00266585242934525,
+            -0.013348115608096123,
+            -0.006142395548522472,
+            -0.015090158209204674,
+            -0.007017187308520079,
+            0.002581012900918722,
+            0.022654090076684952,
+            0.004143949132412672,
+            -0.0013828874798491597,
+            0.00010999705409631133,
+            -0.007752465084195137,
+            0.015791499987244606,
+            0.0027092150412499905,
+            -0.0003994536818936467,
+            0.004984804894775152,
+            0.015135405585169792,
+            0.008355769328773022,
+            -0.0059312391094863415,
+            0.008446265012025833,
+            0.006447818595916033,
+            -0.009592544287443161,
+            0.025821439921855927,
+            -0.008159696124494076,
+            0.009042028337717056,
+            -0.019712980836629868,
+            0.007639345247298479,
+            0.011093264445662498,
+            -0.014456688426434994,
+            -0.008023952133953571,
+            -0.02155306003987789,
+            0.02076876349747181,
+            0.007473436649888754,
+            0.0074885194189846516,
+            -0.024720408022403717,
+            0.014139953069388866,
+            -0.0055730268359184265,
+            -0.0013706327881664038,
+            -0.024780739098787308,
+            0.016213813796639442,
+            0.024886317551136017,
+            0.00906465295702219,
+            0.02027103677392006,
+            -0.003993122838437557,
+            0.008717752061784267,
+            -0.016017738729715347,
+            0.002801596187055111,
+            -0.012330039404332638,
+            -0.020783845335245132,
+            0.0033992447424679995,
+            0.033151593059301376,
+            -0.0016279799165204167,
+            -0.005870908498764038,
+            0.001966396113857627,
+            0.006131083704531193,
+            -0.004841519985347986,
+            -0.007707217242568731,
+            0.012247084639966488,
+            0.01344615314155817,
+            0.004649216774851084,
+            0.006006652023643255,
+            0.06690648943185806,
+            -0.004015746526420116,
+            0.004879226442426443,
+            -0.0011707880767062306,
+            -0.0031767762266099453,
+            -0.0034011302050203085,
+            -0.001456415164284408,
+            -0.02469024434685707,
+            0.019984466955065727,
+            0.0036556492559611797,
+            -0.009057111106812954,
+            0.009675499051809311,
+            -0.021085498854517937,
+            0.0006980423349887133,
+            0.007051123306155205,
+            0.011176219210028648,
+            0.001268825144506991,
+            0.016078069806098938,
+            0.03161316365003586,
+            0.015376728028059006,
+            -0.016666291281580925,
+            0.005180879030376673,
+            0.01116113644093275,
+            0.006406341679394245,
+            -0.00740933557972312,
+            0.02408693917095661,
+            0.0013781740562990308,
+            0.023815451189875603,
+            0.0019852493423968554,
+            0.020859258249402046,
+            0.010995227843523026,
+            -0.005554173607379198,
+            0.0007465895032510161,
+            -0.008582009002566338,
+            0.0064629013650119305,
+            0.012631691992282867,
+            0.0006551511469297111,
+            0.00011624219769146293,
+            0.0008257732843048871,
+            0.011372293345630169,
+            0.00801641121506691,
+            0.0018089713994413614,
+            -0.005953862797468901,
+            -0.007213261444121599,
+            -0.00876300036907196,
+            0.010957521386444569,
+            0.0025565035175532103,
+            -0.0005443881964311004,
+            -0.01063324511051178,
+            -0.002959963632747531,
+            -0.00010993813339155167,
+            0.013913714326918125,
+            0.016605960205197334,
+            0.009358763694763184,
+            -0.012654315680265427,
+            -0.004166572820395231,
+            0.0208290945738554,
+            -0.004852831829339266,
+            0.024916483089327812,
+            0.010670951567590237,
+            0.004422977566719055,
+            0.014418981969356537,
+            -0.016651209443807602,
+            -0.016349555924534798,
+            0.0030768539290875196,
+            -0.0021323049440979958,
+            0.011508036404848099,
+            0.010640786029398441,
+            -0.002104025101289153,
+            0.020406780764460564,
+            -0.007360316812992096,
+            -0.0025828981306403875,
+            0.015731168910861015,
+            0.016832200810313225,
+            -0.004215591587126255,
+            0.005275145173072815,
+            -0.015881994739174843,
+            -0.016334474086761475,
+            -0.005655981134623289,
+            0.00012619908375199884,
+            0.005810578353703022,
+            0.012873013503849506,
+            -5.217053694650531e-05,
+            0.014607514254748821,
+            -0.005870908498764038,
+            -0.025384044274687767,
+            -0.0007234942750073969,
+            0.003993122838437557,
+            0.01603282243013382,
+            -0.007088829763233662,
+            0.002594209974631667,
+            0.0016977370250970125,
+            0.021749133244156837,
+            -0.02099500223994255,
+            0.0038762325420975685,
+            0.0003674031177069992,
+            -0.01325007900595665,
+            0.0005245922366157174,
+            -0.007978703826665878,
+            -0.012458241544663906,
+            0.015331479720771313,
+            0.005561714991927147,
+            -0.013521566055715084,
+            0.018310297280550003,
+            -0.01052766665816307,
+            -0.0012150933034718037,
+            0.02199045568704605,
+            0.010648327879607677,
+            0.004362646955996752,
+            -0.015278690494596958,
+            -0.0056748343631625175,
+            -0.014871460385620594,
+            0.022382603958249092,
+            0.01708860509097576,
+            -0.004260839428752661,
+            0.003921480383723974,
+            0.013679933734238148,
+            0.0046944646164774895,
+            -0.018883436918258667,
+            -0.010648327879607677,
+            -0.0004682681174017489,
+            -0.010210931301116943,
+            0.004728400148451328,
+            0.0072396560572087765,
+            0.02010512910783291,
+            -0.016500383615493774,
+            -0.005995340179651976,
+            -0.006119771860539913,
+            -0.02930552512407303,
+            -0.029275359585881233,
+            -0.004411665257066488,
+            0.0024867465253919363,
+            -0.0024792051408439875,
+            -0.0026545405853539705,
+            -0.0019550842698663473,
+            0.002839302644133568,
+            0.0005000830278731883,
+            -0.008137071505188942,
+            0.013740263879299164,
+            -0.000598591344896704,
+            0.005554173607379198,
+            0.022126199677586555,
+            -0.002910945098847151,
+            -0.013544189743697643,
+            0.03194498270750046,
+            0.011455247178673744,
+            -0.009743370115756989,
+            -0.027344783768057823,
+            0.01211888249963522,
+            -0.011063099838793278,
+            0.017510918900370598,
+            -0.043558597564697266,
+            -0.004962180741131306,
+            -0.0038592645432800055,
+            -0.007356546353548765,
+            -0.011606073938310146,
+            -0.011002768762409687,
+            -0.016349555924534798,
+            -0.006862590555101633,
+            -0.010949979536235332,
+            -0.022231776267290115,
+            0.007963621988892555,
+            -0.005312851630151272,
+            0.02656048908829689,
+            -0.022216694429516792,
+            0.024448921903967857,
+            0.010158142074942589,
+            -0.003913938999176025,
+            0.002268048468977213,
+            -0.008589549921452999,
+            0.00490562105551362,
+            0.020074963569641113,
+            0.010014857165515423,
+            0.00011942369019379839,
+            0.021010085940361023,
+            0.008883661590516567,
+            -0.0038290992379188538,
+            -0.0038988564629107714,
+            -0.005433512851595879,
+            0.0246751606464386,
+            0.004422977566719055,
+            0.004660528618842363,
+            -0.008076741360127926,
+            0.011055557988584042,
+            0.027043132111430168,
+            0.007111453916877508,
+            -0.009336140006780624,
+            -0.005999110639095306,
+            0.0027959400322288275,
+            -0.011658863164484501,
+            -0.008416100405156612,
+            0.011093264445662498,
+            -0.01098768599331379,
+            -0.023543965071439743,
+            0.01002993993461132,
+            0.02291049435734749,
+            0.002745036268606782,
+            -0.015881994739174843,
+            -0.005640898831188679,
+            -0.0024716637562960386,
+            0.006685370113700628,
+            0.017827652394771576,
+            -0.010980145074427128,
+            -0.0144114401191473,
+            0.010542749427258968,
+            0.004057223908603191,
+            -0.00531662255525589,
+            0.02155306003987789,
+            0.030119985342025757,
+            0.011651321314275265,
+            -0.04367925971746445,
+            -0.005663522519171238,
+            0.005867138039320707,
+            -0.013823218643665314,
+            -0.047962721437215805,
+            0.004784960299730301,
+            -0.008416100405156612,
+            -0.013649768196046352,
+            0.014456688426434994,
+            -0.009562378749251366,
+            -0.014999662525951862,
+            0.02104024961590767,
+            0.026213588193058968,
+            0.0052977693267166615,
+            0.012910719960927963,
+            -0.01418520137667656,
+            -0.019909054040908813,
+            -0.020044798031449318,
+            0.00815215427428484,
+            -0.017706992104649544,
+            -0.009607627056539059,
+            0.0023151817731559277,
+            -0.013333032839000225,
+            0.02297082543373108,
+            0.012412993237376213,
+            0.03152266889810562,
+            0.009683039970695972,
+            -0.004702006001025438,
+            -0.02529354766011238,
+            -0.01719418354332447,
+            -0.002509370446205139,
+            -0.0010963176609948277,
+            0.0049697221256792545,
+            0.005456136539578438,
+            0.017510918900370598,
+            0.012963509187102318,
+            0.009517131373286247,
+            -0.013008756563067436,
+            0.015301315113902092,
+            -2.8662863769568503e-05,
+            -0.003806475317105651,
+            -0.014290779829025269,
+            -0.029561929404735565,
+            -0.0033615382853895426,
+            0.006383717525750399,
+            -0.0040911599062383175,
+            -0.004630363546311855,
+            0.0063196164555847645,
+            0.030014406889677048,
+            -0.009192855097353458,
+            0.002863811794668436,
+            -0.037435054779052734,
+            -0.003587777493521571,
+            -0.0034539191983640194,
+            0.026847057044506073,
+            0.01156836748123169,
+            -0.008755459450185299,
+            -0.006323386915028095,
+            -0.01344615314155817,
+            0.012888096272945404,
+            0.01366485096514225,
+            -0.009683039970695972,
+            0.03312142565846443,
+            0.0020116439554840326,
+            0.01396650355309248,
+            -0.0012084946502000093,
+            -0.01002993993461132,
+            0.016078069806098938,
+            0.028883211314678192,
+            -0.011281797662377357,
+            0.012767435051500797,
+            0.01190018467605114,
+            0.005512696225196123,
+            0.0017071636393666267,
+            -0.027254289016127586,
+            0.0022341127041727304,
+            -0.01021847315132618,
+            0.0214927289634943,
+            0.000284920068224892,
+            0.0015572801930829883,
+            -0.02093467302620411,
+            0.018114222213625908,
+            -0.02506730891764164,
+            -0.005060217808932066,
+            0.012443158775568008,
+            -0.0038102460093796253,
+            0.004717088304460049,
+            -0.02186979353427887,
+            -0.010535207577049732,
+            0.015007203444838524,
+            -0.018808024004101753,
+            -0.004396582953631878,
+            -0.012707104906439781,
+            -0.01553509570658207,
+            0.021356984972953796,
+            0.009750911965966225,
+            -0.018415875732898712,
+            -0.0005943493451923132,
+            0.029396019876003265,
+            -0.027239205315709114,
+            0.010753905400633812,
+            0.0180086437612772,
+            0.016711538657546043,
+            0.01344615314155817,
+            0.001777863479219377,
+            0.00884595513343811,
+            -0.0014799818163737655,
+            -0.012473324313759804,
+            -0.020014632493257523,
+            0.0196677315980196,
+            -0.01473571639508009,
+            0.019697897136211395,
+            0.010097811929881573,
+            -0.00854430254548788,
+            -0.005407118238508701,
+            -0.004815125372260809,
+            -0.003229565219953656,
+            -0.010120435617864132,
+            -0.003642451949417591,
+            0.015776416286826134,
+            0.022955743595957756,
+            -0.004068535752594471,
+            -0.018159471452236176,
+            0.017510918900370598,
+            -0.0015959293814375997,
+            0.0015139176975935698,
+            -0.0011679601157084107,
+            -0.006195184774696827,
+            0.015059992671012878,
+            0.010120435617864132,
+            -0.0002375512121943757,
+            -0.06654450297355652,
+            -0.009539755061268806,
+            -0.004287234041839838,
+            -0.004257068503648043,
+            0.002848729258403182,
+            -0.017812570556998253,
+            -0.03951645642518997,
+            0.011191301979124546,
+            0.016892530024051666,
+            -0.01471309270709753,
+            0.01592724397778511,
+            0.005678605288267136,
+            0.0017250742530450225,
+            -0.01740534044802189,
+            -0.002826105337589979,
+            0.021507810801267624,
+            0.03251812234520912,
+            -0.025052227079868317,
+            -0.01211888249963522,
+            0.012247084639966488,
+            -0.004155260976403952,
+            -0.007775088772177696,
+            -0.021326819434762,
+            -0.00029057604842819273,
+            0.01812930591404438,
+            0.02556503564119339,
+            -0.020844176411628723,
+            -0.024011526256799698,
+            4.7928548156050965e-05,
+            0.011085723526775837,
+            -0.025761108845472336,
+            -0.009924361482262611,
+            -0.015625590458512306,
+            -0.0034086713567376137,
+            0.011206384748220444,
+            -0.011870019137859344,
+            0.007624262943863869,
+            -0.012164130806922913,
+            0.005437283311039209,
+            -0.008121988736093044,
+            -0.02485615201294422,
+            0.011470329947769642,
+            -0.006055670790374279,
+            0.035444147884845734,
+            -0.006662745960056782,
+            -0.0030900510028004646,
+            0.02726937085390091,
+            -0.0008281299378722906,
+            0.005916156340390444,
+            0.014049457386136055,
+            0.03002949059009552,
+            -0.014871460385620594,
+            0.011176219210028648,
+            -0.000854524492751807,
+            -0.004882997367531061,
+            0.0005217642756178975,
+            0.03812885656952858,
+            0.004751024302095175,
+            0.0003579764743335545,
+            -0.0009945100173354149,
+            0.0015940440353006124,
+            -0.0011538201943039894,
+            0.015188194811344147,
+            0.025323713198304176,
+            -0.0012480865698307753,
+            -0.011093264445662498,
+            -0.01576133444905281,
+            -0.013868466019630432,
+            -8.383813838008791e-05,
+            0.0030881657730787992,
+            0.011085723526775837,
+            0.014056999236345291,
+            -0.006527002435177565,
+            0.01675678789615631,
+            -0.01146278902888298,
+            0.00815215427428484,
+            -0.024222683161497116,
+            0.013189747929573059,
+            -0.011666404083371162,
+            0.0006047186907380819,
+            0.0031447256915271282,
+            -0.0015261722728610039,
+            0.01325007900595665,
+            -0.00677586579695344,
+            0.004860373213887215,
+            0.012254626490175724,
+            -0.0007579015218652785,
+            0.0024414986837655306,
+            -0.021115664392709732,
+            -0.013906172476708889,
+            -0.003118331078439951,
+            0.021945208311080933,
+            -0.0030730832368135452,
+            -0.014637679792940617,
+            -4.306793925934471e-05,
+            0.004230673890560865,
+            -0.005192190874367952,
+            0.012171671725809574,
+            0.0008700784528627992,
+            -0.00031437829602509737,
+            0.020859258249402046,
+            0.0039063976146280766,
+            -0.010346675291657448,
+            0.01556526031345129,
+            -0.003041032701730728,
+            -0.0069530862383544445,
+            -0.010120435617864132,
+            0.006508149206638336,
+            0.006229120772331953,
+            -0.005177108105272055,
+            0.026364414021372795,
+            -0.03288010507822037,
+            0.0008507538586854935,
+            0.009705663658678532,
+            -0.0014102247077971697,
+            -0.0029486515559256077,
+            0.004102471750229597,
+            0.04105488210916519,
+            -0.009637792594730854,
+            -0.023830534890294075,
+            0.0002169304498238489,
+            0.01769191026687622,
+            -0.011492953635752201,
+            0.002967504784464836,
+            0.01823488436639309,
+            0.004890538286417723,
+            0.0034859697334468365,
+            0.010007316246628761,
+            -0.03704290837049484,
+            -0.004471995867788792,
+            -0.007940997369587421,
+            -0.011206384748220444,
+            0.007186866831034422,
+            0.02088942378759384,
+            0.01690761372447014,
+            0.014026833698153496,
+            0.005784183740615845,
+            0.030949529260396957,
+            0.0037970487028360367,
+            -0.003135299077257514,
+            0.0016317506087943912,
+            0.00720194960013032,
+            -0.013008756563067436,
+            0.018099140375852585,
+            -0.008137071505188942,
+            -0.015520012937486172,
+            0.004260839428752661,
+            -0.009569920599460602,
+            0.004815125372260809,
+            -0.011032934300601482,
+            0.0051243188790977,
+            -0.006904067937284708,
+            0.0005476875230669975,
+            -0.007439500652253628,
+            -0.00021657695469912142,
+            0.013076628558337688,
+            -3.543825732776895e-05,
+            0.011221466585993767,
+            0.012239543721079826,
+            -0.009057111106812954,
+            -0.024644995108246803,
+            0.012443158775568008,
+            -0.008031493052840233,
+            0.005772871430963278,
+            -0.010927355848252773,
+            0.0009073136607185006,
+            -0.0010576684726402164,
+            0.03791769966483116,
+            0.025414209812879562,
+            -0.0003365308803040534,
+            0.00593878049403429,
+            -0.028551394119858742,
+            0.007360316812992096,
+            -0.024991896003484726,
+            0.009826324880123138,
+            -0.013958961702883244,
+            -0.008287898264825344,
+            0.012171671725809574,
+            0.028460897505283356,
+            -0.0046529872342944145,
+            0.004226903431117535,
+            -0.007266050670295954,
+            -0.020150376483798027,
+            -0.007895749993622303,
+            -0.006530772894620895,
+            -0.011500495485961437,
+            -0.02055760659277439,
+            -0.010286344215273857,
+            0.01785781793296337,
+            -0.009026946499943733,
+            -0.0013678048271685839,
+            0.015286232344806194,
+            -0.0017995446687564254,
+            0.005490072537213564,
+            -0.017058439552783966,
+            0.0038837739266455173,
+            0.0062178089283406734,
+            0.01690761372447014,
+            -0.01553509570658207,
+            0.012209378182888031,
+            0.0033332582097500563,
+            -0.013091711327433586,
+            0.008106906898319721,
+            -0.021070415154099464,
+            -0.015082616358995438,
+            -0.0003834283852484077,
+            0.007695905398577452,
+            -0.010535207577049732,
+            -0.030919363722205162,
+            0.009554837830364704,
+            -0.0043777297250926495,
+            -0.007511143106967211,
+            -0.01613840088248253,
+            0.009901737794280052,
+            0.0022322272416204214,
+            0.03387555852532387,
+            -0.017284678295254707,
+            -0.006741929799318314,
+            0.019712980836629868,
+            0.009109900332987309,
+            -0.01642496883869171,
+            -0.009366304613649845,
+            0.004249527119100094,
+            -0.0012886211043223739,
+            0.006821113638579845,
+            -0.01463013794273138,
+            -0.009939444251358509,
+            0.0031145603861659765,
+            0.005637127906084061,
+            0.019471658393740654,
+            0.005324163939803839,
+            -0.0135818962007761,
+            -0.0032653864473104477,
+            0.009351222775876522,
+            0.006911609321832657,
+            0.009388929232954979,
+            -0.016349555924534798,
+            0.002688476350158453,
+            -0.012254626490175724,
+            0.002351003000512719,
+            0.011628697626292706,
+            -0.006361093837767839,
+            -0.006342240143567324,
+            0.022895412519574165,
+            0.017933230847120285,
+            -0.0018061433220282197,
+            -0.013234996236860752,
+            0.0004694464441854507,
+            -0.0002575828111730516,
+            0.01281268335878849,
+            -0.006557167507708073,
+            -0.005595650989562273,
+            0.0013838300947099924,
+            0.01576133444905281,
+            0.003944104071706533,
+            -0.005090383347123861,
+            -0.013943878933787346,
+            -0.015384268946945667,
+            0.0007640288094989955,
+            -0.027480527758598328,
+            0.0033502262085676193,
+            0.0014385045506060123,
+            0.020452028140425682,
+            0.017812570556998253,
+            0.019019179046154022,
+            -0.014358650892972946,
+            0.001570477499626577,
+            0.005007428582757711,
+            -0.00812953058630228,
+            0.002094598487019539,
+            -0.009879114106297493,
+            -0.0190040972083807,
+            -0.006504378281533718,
+            0.0058595966547727585,
+            0.018370626494288445,
+            0.004656758159399033,
+            -0.004309857729822397,
+            0.029411103576421738,
+            0.019366079941391945,
+            -0.02309148572385311,
+            0.011749358847737312,
+            -0.024554500356316566,
+            0.010301426984369755,
+            0.01962248422205448,
+            0.004592656623572111,
+            -0.016078069806098938,
+            -0.024554500356316566,
+            0.0050111995078623295,
+            -0.010301426984369755,
+            -0.008506596088409424,
+            0.008302980102598667,
+            -0.023785287514328957,
+            0.018084056675434113,
+            -0.011975597590208054,
+            0.0018268820131197572,
+            -0.010844401083886623,
+            -0.004241986200213432,
+            -0.010542749427258968,
+            -0.0031748907640576363,
+            0.007115224376320839,
+            0.01690761372447014,
+            0.005388265009969473,
+            -0.03577596694231033,
+            -0.011010310612618923,
+            0.015580343082547188,
+            -0.00520727364346385,
+            -0.0094190938398242,
+            -0.0020436944905668497,
+            0.019652649760246277,
+            -0.02164355479180813,
+            -0.006764553487300873,
+            -0.0376763753592968,
+            -0.02022578939795494,
+            -0.02755594067275524,
+            -0.028340237215161324,
+            -0.01314450055360794,
+            -0.004615280777215958,
+            0.015550177544355392,
+            0.0017986020538955927,
+            -0.008574468083679676,
+            -0.0008361425716429949,
+            -0.0108293192461133,
+            -0.0003174419398419559,
+            -0.010263720527291298,
+            -0.006417653523385525,
+            0.0033596528228372335,
+            0.007375399582087994,
+            0.002143617020919919,
+            -0.012262167409062386,
+            -0.008039034903049469,
+            -0.011809688992798328,
+            -0.010595538653433323,
+            0.00854430254548788,
+            -0.003346455516293645,
+            -0.019682815298438072,
+            0.016108235344290733,
+            0.008197402581572533,
+            0.012005763128399849,
+            0.013189747929573059,
+            0.0009275809279642999,
+            0.032095808535814285,
+            6.698803190374747e-05,
+            -0.0016854824498295784,
+            -6.112583832873497e-06,
+            -0.009026946499943733,
+            0.015376728028059006,
+            0.0077298409305512905,
+            0.008325604721903801,
+            -0.004079848062247038,
+            0.006293221842497587,
+            -0.007443271577358246,
+            0.0058181192725896835,
+            0.00018040223221760243,
+            -0.007971162907779217,
+            0.0003205056127626449,
+            0.007695905398577452,
+            0.002927913097664714,
+            -0.01418520137667656,
+            0.006817342713475227,
+            0.012699563056230545,
+            0.006934233009815216,
+            -0.01576133444905281,
+            -0.01380059402436018,
+            -0.012156588956713676,
+            -0.008499054238200188,
+            -0.028551394119858742,
+            0.022955743595957756,
+            0.043830085545778275,
+            0.000945962849073112,
+            -0.0190040972083807,
+            -0.0044380598701536655,
+            -0.015444600023329258,
+            -0.015007203444838524,
+            -0.0056748343631625175,
+            -0.005912385880947113,
+            -0.02375512197613716,
+            -0.008114447817206383,
+            0.015278690494596958,
+            0.014577348716557026,
+            0.000577381404582411,
+            -0.01570100337266922,
+            -0.005953862797468901,
+            -0.015987573191523552,
+            -0.013302868232131004,
+            -0.00908727664500475,
+            -0.021960290148854256,
+            -0.012518571689724922,
+            0.012043469585478306,
+            -0.01613840088248253,
+            0.015836747363209724,
+            0.020421862602233887,
+            0.00380836077965796,
+            -0.014109788462519646,
+            -0.015437058173120022,
+            -0.011289338581264019,
+            0.01198313944041729,
+            0.010912273079156876,
+            -0.010723740793764591,
+            0.03107019141316414,
+            -0.016440052539110184,
+            -0.009524672292172909,
+            -0.0033596528228372335,
+            0.007217032369226217,
+            -0.014260614290833473,
+            -0.004147719591856003,
+            -0.016621043905615807,
+            0.01220183726400137,
+            -0.014275697059929371,
+            0.006459130439907312,
+            0.007250967901200056,
+            -0.016334474086761475,
+            0.021326819434762,
+            0.025263383984565735,
+            0.0018579898169264197,
+            0.007635574787855148,
+            0.009962067939341068,
+            -0.013928796164691448,
+            0.010784070938825607,
+            -0.004984804894775152,
+            -0.0012575131841003895,
+            0.028129080310463905,
+            0.008536760695278645,
+            -0.0027506921906024218,
+            -0.015188194811344147,
+            0.013627144508063793,
+            0.020798929035663605,
+            -0.0023604296147823334,
+            -0.006870131939649582,
+            -0.013657309114933014,
+            0.013695016503334045,
+            0.0066363513469696045,
+            -0.00605944124981761,
+            -0.0075978683307766914,
+            -0.008989240042865276,
+            -0.0019230336183682084,
+            -0.021779298782348633,
+            0.012149048037827015,
+            -0.023830534890294075,
+            0.00014540585107170045,
+            -0.009494507685303688,
+            0.009667957201600075,
+            -0.0060028815641999245,
+            0.009939444251358509,
+            -0.010806694626808167,
+            0.013619602657854557,
+            0.007367858197540045,
+            0.0004548351571429521,
+            -0.011274255812168121,
+            0.017767323181033134,
+            0.004260839428752661,
+            0.00856692623347044,
+            0.00245281052775681,
+            -0.019170006737113,
+            0.030557380989193916,
+            -0.006625039502978325,
+            0.009434176608920097,
+            -0.010799153707921505,
+            0.012639232911169529,
+            0.021417316049337387,
+            0.028732385486364365,
+            0.011741816997528076,
+            0.0034765431191772223,
+            -0.015225901268422604,
+            0.024147270247340202,
+            0.0015846174210309982,
+            -0.010075188241899014,
+            0.02016545832157135,
+            0.004159031435847282,
+            0.005195961333811283,
+            -0.007032270077615976,
+            0.0090043218806386,
+            -0.02423776499927044,
+            -0.02551978826522827,
+            -0.007511143106967211,
+            -0.007582785561680794,
+            -0.009328598156571388,
+            0.01495441421866417,
+            -0.0017297875601798296,
+            -0.0019946759566664696,
+            0.0037178650964051485,
+            0.0032031708396971226,
+            0.007496060337871313,
+            0.002641343278810382,
+            -0.013574355281889439,
+            -0.012164130806922913,
+            -0.016711538657546043,
+            0.019215254113078117,
+            -0.008106906898319721,
+            0.001139680272899568,
+            -0.0070247286930680275,
+            0.021417316049337387,
+            0.010188307613134384,
+            0.007160472217947245,
+            0.012707104906439781,
+            -0.006877673324197531,
+            0.01729976199567318,
+            0.004649216774851084,
+            -0.012435617856681347,
+            0.015987573191523552,
+            0.026258835569024086,
+            -0.0012443158775568008,
+            -0.00364999333396554,
+            0.00190418038982898,
+            -0.04192967340350151,
+            0.00040935163269750774,
+            -0.014041916467249393,
+            0.018702445551753044,
+            -0.007888208143413067,
+            0.004988575354218483,
+            0.01459997333586216,
+            -0.005350558087229729,
+            -0.0010887763928622007,
+            0.014011750929057598,
+            0.007326381281018257,
+            0.015670839697122574,
+            0.00720194960013032,
+            0.015550177544355392,
+            -0.013657309114933014,
+            -0.03069312497973442,
+            -0.008174777962267399,
+            -0.003989351913332939,
+            0.015437058173120022,
+            0.0037103237118571997,
+            0.016228895634412766,
+            -0.030949529260396957,
+            -0.005116777960211039,
+            -0.006296992301940918,
+            0.002047465182840824,
+            0.014705551788210869,
+            -0.00354630034416914,
+            0.018038809299468994,
+            0.015489847399294376,
+            -0.008755459450185299,
+            -0.02159830741584301,
+            -0.020844176411628723,
+            0.013936338014900684,
+            0.010060105472803116,
+            -0.020361533388495445,
+            0.0020512358751147985,
+            -0.008634798228740692,
+            -0.006244203541427851,
+            -0.02963734231889248,
+            0.024283012375235558,
+            -0.003429410047829151,
+            0.01305400487035513,
+            0.011636238545179367,
+            0.010482418350875378,
+            0.013521566055715084,
+            0.02942618541419506,
+            -0.004690693691372871,
+            0.009683039970695972,
+            0.005984028335660696,
+            -0.009275808930397034,
+            -0.007164243143051863,
+            -0.009864031337201595,
+            0.015881994739174843,
+            -0.0019701668061316013,
+            0.0027412655763328075,
+            -0.003182432148605585,
+            0.017013192176818848,
+            -0.004626592621207237,
+            -0.024056773632764816,
+            -0.004204279277473688,
+            -0.001689253025688231,
+            -0.010429629124701023,
+            0.011802148073911667,
+            -0.008506596088409424,
+            -0.000715952948667109,
+            0.009735829196870327,
+            -0.004607739392668009,
+            -0.008989240042865276,
+            0.006892756093293428,
+            0.0002959963458124548,
+            -0.04117554426193237,
+            0.0005467448499985039,
+            0.004027058836072683,
+            -0.017752239480614662,
+            0.004668070003390312,
+            -0.0076280334033071995,
+            -0.010346675291657448,
+            0.0005962346913293004,
+            0.005927468184381723,
+            0.003774424782022834,
+            0.0221563633531332,
+            -0.0018711871234700084,
+            0.01438127551227808,
+            -0.006862590555101633,
+            -0.0024942876771092415,
+            -0.009750911965966225,
+            0.0135818962007761,
+            -0.020949754863977432,
+            -0.014720633625984192,
+            -0.01580658182501793,
+            -0.005007428582757711,
+            0.004577574320137501,
+            0.00458511570468545,
+            0.009539755061268806,
+            0.0013272702926769853,
+            -0.006444048136472702,
+            0.010701116174459457,
+            -0.006413882598280907,
+            -0.00032121260301209986,
+            -0.013016298413276672,
+            0.008446265012025833,
+            0.02357413060963154,
+            0.005075300578027964,
+            -0.0064930664375424385,
+            -0.013016298413276672,
+            0.01586691290140152,
+            0.03441853076219559,
+            -0.023016072809696198,
+            0.008664963766932487,
+            0.019712980836629868,
+            -0.0031371843069791794,
+            -0.002475434448570013,
+            -0.011583449319005013,
+            0.003039147239178419,
+            -0.019471658393740654,
+            0.009796159341931343,
+            0.006873902399092913,
+            -0.008227567188441753,
+            0.004882997367531061,
+            -0.015776416286826134,
+            0.0017165903700515628,
+            -0.002958078170195222,
+            -0.008710211142897606,
+            -0.005557944066822529,
+            -0.012692022137343884,
+            -0.006078294478356838,
+            0.0035180202685296535,
+            0.008393475785851479,
+            -0.004498390480875969,
+            0.01856670156121254,
+            -0.004038370680063963,
+            0.02253342978656292,
+            0.0022416538558900356,
+            -0.001372518134303391,
+            -0.009124983102083206,
+            0.0070247286930680275,
+            0.00026041080127470195,
+            -0.004769877530634403,
+            0.0017948313616216183,
+            -0.009072193875908852,
+            0.011334586888551712,
+            -0.0034312952775508165,
+            0.01752600073814392,
+            0.007009646389633417,
+            -0.008717752061784267,
+            -0.006308304611593485,
+            0.0027714308816939592,
+            -0.005712541285902262,
+            0.012322497554123402,
+            -0.004592656623572111,
+            0.01894376613199711,
+            -0.013423528522253036,
+            -0.018702445551753044,
+            0.004717088304460049,
+            0.008476430550217628,
+            0.013287785463035107,
+            -0.009532214142382145,
+            0.01002993993461132,
+            -0.0037819661665707827,
+            0.01823488436639309,
+            0.01806897483766079,
+            -0.007243426516652107,
+            0.015007203444838524,
+            -0.01962248422205448,
+            -0.0013781740562990308,
+            0.012737269513309002,
+            -0.002754462882876396,
+            -0.020361533388495445,
+            -0.011651321314275265,
+            -0.018928684294223785,
+            0.010489960201084614,
+            -0.015489847399294376,
+            0.023709872737526894,
+            0.011274255812168121,
+            0.010572914034128189,
+            -0.001968281576409936,
+            0.006198955699801445,
+            -0.0019202056573703885,
+            0.014539642259478569,
+            -0.0016835971036925912,
+            -0.004864144138991833,
+            -0.010595538653433323,
+            -0.002262392546981573,
+            -0.04057224094867706,
+            8.725529187358916e-05,
+            0.014698009938001633,
+            0.03354373946785927,
+            -0.02170388586819172,
+            0.014336027204990387,
+            -0.011651321314275265,
+            -0.0190040972083807,
+            -0.012043469585478306,
+            0.015158030204474926,
+            -0.015263608656823635,
+            0.019170006737113,
+            -0.009743370115756989,
+            -0.0047661070711910725,
+            0.001808028668165207,
+            0.01201330404728651,
+            0.003014638088643551,
+            0.0039478749968111515,
+            0.02402660809457302,
+            0.015203277580440044,
+            0.00648552505299449,
+            -0.003024064702913165,
+            0.011515578255057335,
+            0.00458511570468545,
+            -0.014863918535411358,
+            -0.005463677924126387,
+            -0.0011594761162996292,
+            -0.007639345247298479,
+            -0.010618162341415882,
+            -0.008529219776391983,
+            -0.0037800809368491173,
+            0.0008403845713473856,
+            0.018747692927718163,
+            -0.006010422948747873,
+            -0.0012942770263180137,
+            -0.012299873866140842,
+            0.012164130806922913,
+            -0.0011886986903846264,
+            0.018928684294223785,
+            -0.027359867468476295,
+            0.026741480454802513,
+            -0.03444869816303253,
+            -0.01388354878872633,
+            -0.006632580887526274,
+            -0.00015966364298947155,
+            0.011809688992798328,
+            0.0028845504857599735,
+            -0.015836747363209724,
+            -0.0008521678391844034,
+            -0.006930462550371885,
+            0.002935454249382019,
+            -0.003197514684870839,
+            0.009524672292172909,
+            0.005855826195329428,
+            -0.005659752059727907,
+            -0.021945208311080933,
+            -0.008891202509403229,
+            -0.012563819997012615,
+            -0.008355769328773022,
+            -0.013438611291348934,
+            0.013627144508063793,
+            -0.003331372980028391,
+            0.0006655204924754798,
+            0.023000990971922874,
+            -0.014283237978816032,
+            -0.01613840088248253,
+            0.015082616358995438,
+            -0.003555726958438754,
+            -0.0056446692906320095,
+            -0.01305400487035513,
+            0.007775088772177696,
+            0.0010850057005882263,
+            0.018249966204166412,
+            -0.008876119740307331,
+            0.022337354719638824,
+            -0.0045097023248672485,
+            0.012322497554123402,
+            0.018099140375852585,
+            -0.023453468456864357,
+            -0.030180316418409348,
+            0.004219362046569586,
+            -0.02793300710618496,
+            -0.0010472992435097694,
+            0.0019739374984055758,
+            -0.024946648627519608,
+            -0.014946873299777508,
+            -0.008853496052324772,
+            -0.0255047045648098,
+            -0.01630430854856968,
+            0.012918260879814625,
+            -0.0006914437399245799,
+            0.014253072440624237,
+            0.011621156707406044,
+            -0.016213813796639442,
+            0.01553509570658207,
+            -0.014418981969356537,
+            -0.02490140125155449,
+            0.0008055060170590878,
+            -0.008506596088409424,
+            0.012910719960927963,
+            0.00406476529315114,
+            0.001550681539811194,
+            0.009622709825634956,
+            -0.005320393014699221,
+            -0.0018759004306048155,
+            0.019049344584345818,
+            0.02186979353427887,
+            0.004275921732187271,
+            -0.00418542604893446,
+            0.00616879016160965,
+            -0.016651209443807602,
+            0.029275359585881233,
+            -0.01220183726400137,
+            -0.015097699128091335,
+            0.025097474455833435,
+            0.006285680457949638,
+            0.019215254113078117,
+            0.004453142639249563,
+            0.005128089804202318,
+            0.001582732074894011,
+            0.011357210576534271,
+            -0.0015261722728610039,
+            -0.01104801706969738,
+            -0.014388816431164742,
+            -0.018325379118323326,
+            0.030889200046658516,
+            0.008174777962267399,
+            0.017616497352719307,
+            -0.028672054409980774,
+            0.010799153707921505,
+            -0.008506596088409424,
+            -0.012888096272945404,
+            0.009818783961236477,
+            0.028445815667510033,
+            0.05140155926346779,
+            0.022578677162528038,
+            0.005735164973884821,
+            -0.000906370987650007,
+            0.034297872334718704,
+            -0.01303138118237257,
+            -0.009622709825634956,
+            -0.006455359980463982,
+            -0.01686236634850502,
+            0.004011976066976786,
+            0.012986132875084877,
+            -0.003627369413152337,
+            -0.008393475785851479,
+            -0.010867025703191757,
+            0.03541398420929909,
+            0.005780412815511227,
+            -0.010143060237169266,
+            0.013868466019630432,
+            -0.01956215314567089,
+            0.012216920033097267,
+            -0.017510918900370598,
+            -0.019049344584345818,
+            -0.013415987603366375,
+            0.0009813127107918262,
+            -0.02313673496246338,
+            0.006757012102752924,
+            -0.00854430254548788,
+            0.018913600593805313,
+            -0.010060105472803116,
+            -0.006017964333295822,
+            0.0034331807401031256,
+            -0.005946321878582239,
+            -0.014373733662068844,
+            0.00480758398771286,
+            -0.008023952133953571,
+            0.020783845335245132,
+            0.0005665408098138869,
+            0.008800706826150417,
+            0.0061348541639745235,
+            0.006425194907933474,
+            -0.0028298760298639536,
+            0.0012377172242850065,
+            0.012186754494905472,
+            0.015912160277366638,
+            0.009404012002050877,
+            -0.030346224084496498,
+            0.004253298044204712,
+            -0.013717640191316605,
+            0.014977038837969303,
+            0.00812953058630228,
+            0.01962248422205448,
+            0.016183648258447647,
+            0.013355657458305359,
+            -9.674084867583588e-05,
+            0.004547408781945705,
+            -0.0012961623724550009,
+            -0.0017043356783688068,
+            0.008499054238200188,
+            -0.00024226453388109803,
+            -0.0069530862383544445,
+            -0.0012047240743413568,
+            0.008031493052840233,
+            -0.008921368047595024,
+            -0.019094591960310936,
+            -0.00184950593393296,
+            -0.002364200074225664,
+            -0.01873260922729969,
+            0.0033671942073851824,
+            0.020180542021989822,
+            -0.015987573191523552,
+            -0.0016562598757445812,
+            0.01846112310886383,
+            0.02153797633945942,
+            -0.008204943500459194,
+            0.009230561554431915,
+            -0.004988575354218483,
+            -0.010693575255572796,
+            -0.02766151912510395,
+            -0.000485943048261106,
+            -0.02814416214823723,
+            0.035564810037612915,
+            -0.00886857882142067,
+            -0.018687361851334572,
+            0.012488407082855701,
+            -0.006560938432812691,
+            -0.01713385246694088,
+            -0.0068512787111103535,
+            0.005075300578027964,
+            -0.030165232717990875,
+            0.033966053277254105,
+            -0.016666291281580925,
+            -0.0009907394414767623,
+            0.020482193678617477,
+            0.05173337459564209,
+            -0.0031692348420619965,
+            -0.01746566966176033,
+            0.017254512757062912,
+            0.014426522888243198,
+            0.03188465163111687,
+            -0.01576133444905281,
+            -0.017390256747603416,
+            -0.010814236477017403,
+            -0.012873013503849506,
+            -0.011990680359303951,
+            -0.020814010873436928,
+            0.03604745492339134,
+            -0.012503488920629025,
+            -0.016726622357964516,
+            -0.0040496825240552425,
+            -0.008167237043380737,
+            -0.011289338581264019,
+            0.022654090076684952,
+            0.027450362220406532,
+            -0.0108293192461133,
+            -0.012963509187102318,
+            -0.0012235773028805852,
+            -0.023227229714393616,
+            0.02088942378759384,
+            -0.0005038536619395018,
+            -0.014698009938001633,
+            0.010867025703191757,
+            -0.0036914704833179712,
+            -0.0016949090640991926,
+            0.002579127438366413,
+            0.004751024302095175,
+            0.0569821260869503,
+            -0.015075075440108776,
+            0.002515026368200779,
+            -0.026515239849686623,
+            0.016213813796639442,
+            -0.014863918535411358,
+            -0.012149048037827015,
+            -0.013302868232131004,
+            -0.0031296429224312305,
+            -0.0158216655254364,
+            -0.02737494930624962,
+            -0.001026560552418232,
+            -0.017616497352719307,
+            0.005162025801837444,
+            -0.014464229345321655,
+            0.008069200441241264,
+            -0.024825986474752426,
+            -0.013506483286619186,
+            -0.011063099838793278,
+            -0.024373508989810944,
+            0.015082616358995438,
+            -0.030617712065577507,
+            -0.007790171541273594,
+            0.014449146576225758,
+            0.015218360349535942,
+            -0.007058664690703154,
+            -0.012179212644696236,
+            -0.0075563909485936165,
+            -0.00038366406806744635,
+            -0.05520237609744072,
+            0.002515026368200779,
+            -0.012699563056230545,
+            0.006836195942014456,
+            -0.028310071676969528,
+            0.005146943032741547,
+            0.014351109974086285,
+            0.023000990971922874,
+            0.024267930537462234,
+            0.008385934866964817,
+            0.0024490398354828358,
+            -0.01989397220313549,
+            0.02689230628311634,
+            0.005252521485090256,
+            -0.021839629858732224,
+            0.01313695963472128,
+            -0.003996893297880888,
+            -0.00015777831140439957,
+            -0.01303138118237257,
+            -0.005248750559985638,
+            0.012284791097044945,
+            0.007390482351183891,
+            -0.005791724659502506,
+            -0.007971162907779217,
+            -0.013159583322703838,
+            -0.002450925298035145,
+            0.00605944124981761,
+            0.010467336513102055,
+            0.005803036969155073,
+            0.01044471189379692,
+            -0.016847282648086548,
+            0.0029297983273863792,
+            0.005263833329081535,
+            -0.00219075009226799,
+            0.003046688623726368,
+            -0.007744923699647188,
+            0.01923033595085144,
+            -0.013461235910654068,
+            -0.00782033707946539,
+            0.00209271302446723,
+            -0.02259376086294651,
+            -0.016832200810313225,
+            0.009313516318798065,
+            -0.027902841567993164,
+            0.016500383615493774,
+            -0.002175667556002736,
+            -0.002022956032305956,
+            -0.007903290912508965,
+            0.012465782463550568,
+            0.0031635789200663567,
+            -0.01495441421866417,
+            -0.0007597868097946048,
+            0.0009223962551914155,
+            -0.028883211314678192,
+            0.011621156707406044,
+            0.015459681861102581,
+            -0.006029276177287102,
+            -0.00030047399923205376,
+            -0.029018955305218697,
+            0.009494507685303688,
+            0.0001621381234144792,
+            -0.001130253542214632,
+            0.008793165907263756,
+            0.020255954936146736,
+            0.008227567188441753,
+            -0.012759894132614136,
+            0.012827765196561813,
+            0.013393363915383816,
+            0.005806807428598404,
+            0.00752245495095849,
+            0.024644995108246803,
+            -0.0047962721437215805,
+            0.011621156707406044,
+            -0.011651321314275265,
+            0.00961516797542572,
+            -0.0104145472869277,
+            0.0004543638206087053,
+            0.016605960205197334,
+            0.012307415716350079,
+            -0.005459907464683056,
+            0.005033823195844889,
+            -0.015474764630198479,
+            0.014909166842699051,
+            -0.020633019506931305,
+            0.0008964730077423155,
+            0.012737269513309002,
+            -0.0036066307220607996,
+            0.005599421449005604,
+            0.0135818962007761,
+            0.03830984607338905,
+            0.015851831063628197,
+            0.012480865232646465,
+            0.006093377247452736,
+            -0.02616834081709385,
+            0.003171120071783662,
+            0.0023114110808819532,
+            0.006119771860539913,
+            -0.0024471546057611704,
+            0.004539867863059044,
+            -0.010459794662892818,
+            0.00020526499429252,
+            0.010210931301116943,
+            -0.017118770629167557,
+            0.012865472584962845,
+            -0.010919814929366112,
+            0.007345234509557486,
+            -0.004445601254701614,
+            -0.010919814929366112,
+            -0.021221240982413292,
+            -0.005999110639095306,
+            0.018265048041939735,
+            -0.031824320554733276,
+            0.010716198943555355,
+            0.007039811462163925,
+            0.0288228802382946,
+            -0.02892845869064331,
+            -0.011824771761894226,
+            0.0033841622062027454,
+            -0.006108460016548634,
+            -0.009215478785336018,
+            0.00837085209786892,
+            0.01767682656645775,
+            -0.009652874432504177,
+            0.01534656248986721,
+            0.010821777395904064,
+            -0.006236662156879902,
+            0.03924496844410896,
+            0.007250967901200056,
+            0.016108235344290733,
+            -0.007895749993622303,
+            0.005546632222831249,
+            0.005490072537213564,
+            0.01576133444905281,
+            0.008061658591032028,
+            -0.02021070569753647,
+            0.0027016736567020416,
+            -0.0011566481553018093,
+            0.024539416655898094,
+            0.0052562919445335865,
+            0.005595650989562273,
+            0.0020663184113800526,
+            0.00731506897136569,
+            0.015987573191523552,
+            0.015271149575710297,
+            -0.007213261444121599,
+            -0.007130307145416737,
+            0.007624262943863869,
+            0.003165464149788022,
+            -0.003484084503725171,
+            0.001214150688610971,
+            -0.012518571689724922,
+            -0.004053453449159861,
+            0.017013192176818848,
+            -0.0067381588742136955,
+            -0.001416823361068964,
+            -0.012737269513309002,
+            -0.02452433481812477,
+            0.013453694060444832,
+            0.0018674165476113558,
+            -0.011651321314275265,
+            0.025549951940774918,
+            -0.007695905398577452,
+            -0.011123429983854294,
+            -0.01625906117260456,
+            -0.016334474086761475,
+            0.0139966681599617,
+            0.018280131742358208,
+            0.0010058219777420163,
+            0.0046831523068249226,
+            -0.003197514684870839,
+            -0.0006815457600168884,
+            0.022895412519574165,
+            -0.010776530019938946,
+            0.009728288277983665,
+            -0.002498058369383216
+          ],
+          "keyphrases": [
+            "routing method",
+            "sparsely activated mixture-of-experts",
+            "load imbalance",
+            "training efficiency improvements",
+            "GLUE and SuperGLUE benchmark"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "67c43254-9b93-4d13-b56b-7532088c30f3",
+      "type": "next",
+      "source": {
+        "id": "d08eab82-ce8c-4126-8aee-d1681de9b6e5",
+        "properties": {
+          "page_content": "5 Conclusion\n\nWe propose a new routing method for sparsely activated mixture-of-experts (MoE) models. This method addresses load imbalance and under-utilization of experts in conventional MoE methods, and enables selecting different numbers of experts for each token. Our model demonstrates more than 2x training efficiency improvements when compared to the state-of-the-art GShard and Switch Transformer models, and also achieves strong gains when finetuning on 11 datasets in the GLUE and SuperGLUE benchmark.\n\n",
+          "title": "5 Conclusion",
+          "embedding": [
+            -0.014961956068873405,
+            -0.008189860731363297,
+            -0.023845616728067398,
+            0.002967504784464836,
+            -0.00690029701218009,
+            0.024916483089327812,
+            -0.0020832864101976156,
+            0.07137094438076019,
+            -0.014486853033304214,
+            0.04606231302022934,
+            -0.016666291281580925,
+            0.003244647989049554,
+            0.0073112985119223595,
+            -0.015520012937486172,
+            -0.04539867490530014,
+            0.030662959441542625,
+            0.012692022137343884,
+            0.0035896627232432365,
+            0.013717640191316605,
+            -0.03864166513085365,
+            -0.022352438420057297,
+            -0.008853496052324772,
+            -0.018159471452236176,
+            0.0027733161114156246,
+            0.019019179046154022,
+            -0.0032974369823932648,
+            0.000584922730922699,
+            0.008634798228740692,
+            -0.03864166513085365,
+            0.05327180400490761,
+            0.011213925667107105,
+            0.009728288277983665,
+            -0.0011019736994057894,
+            -0.001177386729978025,
+            -0.03257845342159271,
+            -0.007563932333141565,
+            0.07293953746557236,
+            0.025715861469507217,
+            -0.05375444516539574,
+            0.029727837070822716,
+            0.03912430629134178,
+            0.042442481964826584,
+            -0.03233712911605835,
+            -0.008084282279014587,
+            -0.014434064738452435,
+            -0.0042646098881959915,
+            -0.0013913713628426194,
+            -0.005045135505497456,
+            0.02479582279920578,
+            -0.01170411054044962,
+            0.0019154923502355814,
+            -0.010957521386444569,
+            0.014728175476193428,
+            0.0002771430881693959,
+            -0.04500652849674225,
+            0.03791769966483116,
+            0.018596867099404335,
+            0.017103686928749084,
+            0.03366440162062645,
+            0.007273592054843903,
+            -0.00380270485766232,
+            -0.01829521358013153,
+            -0.018657196313142776,
+            0.0008644224726594985,
+            0.003012752626091242,
+            0.00490185059607029,
+            -0.0039478749968111515,
+            -0.011915267445147038,
+            0.037223897874355316,
+            -0.010565373115241528,
+            -0.06045112758874893,
+            0.03665075823664665,
+            -0.04114537686109543,
+            0.0005467448499985039,
+            -0.0135818962007761,
+            -0.0067796362563967705,
+            0.0271637924015522,
+            0.014562266878783703,
+            -0.00047156744403764606,
+            0.001911721657961607,
+            0.012073635123670101,
+            0.00981124211102724,
+            0.0072698211297392845,
+            -0.007443271577358246,
+            0.05547386407852173,
+            -0.044704876840114594,
+            0.006198955699801445,
+            -0.05541353300213814,
+            0.03659042716026306,
+            -0.0024546959903091192,
+            -0.03007473796606064,
+            -0.021115664392709732,
+            -0.021779298782348633,
+            0.008619715459644794,
+            -0.0393354631960392,
+            0.033362749963998795,
+            0.006523231975734234,
+            -0.013197289779782295,
+            -0.008989240042865276,
+            0.021145828068256378,
+            -0.004758565686643124,
+            0.015218360349535942,
+            -0.015444600023329258,
+            0.008808248676359653,
+            0.05411642789840698,
+            -0.033151593059301376,
+            -0.019366079941391945,
+            0.044644545763731,
+            0.01785781793296337,
+            0.05526270717382431,
+            0.03119085170328617,
+            0.0022322272416204214,
+            0.01181722991168499,
+            0.01852145418524742,
+            -0.003199400147423148,
+            -0.028581559658050537,
+            -0.04585115611553192,
+            -0.05535320192575455,
+            0.023498717695474625,
+            -0.028551394119858742,
+            0.022548511624336243,
+            0.020633019506931305,
+            -0.009117442183196545,
+            -0.023845616728067398,
+            -0.03906397894024849,
+            0.01708860509097576,
+            0.0047095469199121,
+            -0.005278916098177433,
+            -0.012586443684995174,
+            -0.016877448186278343,
+            0.00349162588827312,
+            -0.01137983426451683,
+            -0.02776709757745266,
+            -0.03945612534880638,
+            0.016651209443807602,
+            -0.006040588021278381,
+            0.022186528891324997,
+            0.027917923405766487,
+            -0.020135292783379555,
+            -0.019350998103618622,
+            0.013732722960412502,
+            -0.04171851649880409,
+            0.030285894870758057,
+            0.01812930591404438,
+            -0.005037594120949507,
+            0.010459794662892818,
+            0.0033125197514891624,
+            0.053965602070093155,
+            -0.026213588193058968,
+            0.020844176411628723,
+            -0.04105488210916519,
+            0.04775156453251839,
+            -0.005836972501128912,
+            0.03984827175736427,
+            -0.024117104709148407,
+            -0.013838300481438637,
+            -0.01322745531797409,
+            -0.017556166276335716,
+            -0.006719305645674467,
+            -0.022020621225237846,
+            -0.0005962346913293004,
+            -0.0063196164555847645,
+            -0.0033030931372195482,
+            0.03646976873278618,
+            0.014072081074118614,
+            0.021628472954034805,
+            0.041507359594106674,
+            0.0015572801930829883,
+            -0.07173292338848114,
+            0.030828868970274925,
+            0.007684593088924885,
+            0.030919363722205162,
+            0.006839966867119074,
+            0.015897078439593315,
+            0.006066982634365559,
+            0.021387150511145592,
+            0.002356658922508359,
+            0.004947098437696695,
+            0.005754018202424049,
+            0.04135653376579285,
+            -0.02336297370493412,
+            0.001117998966947198,
+            0.015218360349535942,
+            0.012488407082855701,
+            0.03372473269701004,
+            0.023061320185661316,
+            0.016832200810313225,
+            0.015957407653331757,
+            0.004430518485605717,
+            0.0057163117453455925,
+            0.027118545025587082,
+            0.014901624992489815,
+            -0.0032898958306759596,
+            -0.020753679797053337,
+            0.0024188747629523277,
+            -0.024599747732281685,
+            -0.002049350645393133,
+            -0.01785781793296337,
+            0.034961506724357605,
+            0.0012782517587766051,
+            -0.026183422654867172,
+            -0.014984579756855965,
+            0.005836972501128912,
+            -0.03613794967532158,
+            -0.0010991457384079695,
+            0.0015563375782221556,
+            0.016952861100435257,
+            0.005870908498764038,
+            0.013981585390865803,
+            -0.03405654802918434,
+            0.012141506187617779,
+            -0.023679709061980247,
+            -0.006142395548522472,
+            0.004988575354218483,
+            -0.0007941940566524863,
+            0.013423528522253036,
+            -0.01253365445882082,
+            0.02022578939795494,
+            -0.005361870396882296,
+            -0.028837963938713074,
+            -0.007752465084195137,
+            0.02462991327047348,
+            0.004011976066976786,
+            -0.006097147706896067,
+            -0.010210931301116943,
+            -0.01613840088248253,
+            -0.030723290517926216,
+            -0.017948314547538757,
+            0.03155283257365227,
+            0.014449146576225758,
+            -0.015489847399294376,
+            0.020527441054582596,
+            0.007394252810627222,
+            0.0028939771000295877,
+            -0.014924249611794949,
+            -0.05559452623128891,
+            -0.009969609789550304,
+            -0.012797600589692593,
+            -0.000906370987650007,
+            0.0007253796211443841,
+            -0.04506685957312584,
+            0.02573094516992569,
+            -0.03674125298857689,
+            -0.0009040143340826035,
+            0.015791499987244606,
+            0.006938003934919834,
+            -0.006127313245087862,
+            0.011666404083371162,
+            -0.026590652763843536,
+            0.01303138118237257,
+            -0.030059656128287315,
+            -0.003970498684793711,
+            6.586861854884773e-05,
+            0.015173112042248249,
+            -0.023227229714393616,
+            0.034629687666893005,
+            0.021839629858732224,
+            -0.06274368613958359,
+            -0.008318062871694565,
+            -0.01835554465651512,
+            0.05456890910863876,
+            -0.01449439488351345,
+            -0.008144613355398178,
+            0.052306514233350754,
+            -0.009441718459129333,
+            0.002309525618329644,
+            0.031371843069791794,
+            0.027118545025587082,
+            0.01377042941749096,
+            -0.02897370606660843,
+            0.002942995633929968,
+            -0.07517176121473312,
+            -0.012488407082855701,
+            0.013415987603366375,
+            0.034086715430021286,
+            -0.03063279390335083,
+            0.0013838300947099924,
+            0.015474764630198479,
+            0.01895884983241558,
+            0.03170366212725639,
+            0.002475434448570013,
+            0.02963734231889248,
+            0.010452253744006157,
+            0.01011289469897747,
+            0.004140178207308054,
+            -0.004935786128044128,
+            -0.028340237215161324,
+            -0.0015544522320851684,
+            -0.03167349472641945,
+            0.0005425028502941132,
+            -0.008099365048110485,
+            -0.014094705693423748,
+            0.004355105571448803,
+            0.005403347313404083,
+            0.04651479050517082,
+            -0.006949315778911114,
+            -0.012563819997012615,
+            -0.01306154578924179,
+            -0.0002839773951563984,
+            -0.04144703224301338,
+            -0.0399991013109684,
+            0.014946873299777508,
+            0.005105465650558472,
+            -0.023709872737526894,
+            -0.0516730435192585,
+            -0.012880554422736168,
+            -0.022895412519574165,
+            0.035564810037612915,
+            -0.022729502990841866,
+            -0.012805141508579254,
+            0.0416581891477108,
+            -0.003783851396292448,
+            0.03731439262628555,
+            -0.057494934648275375,
+            -0.008604632690548897,
+            0.006545855663716793,
+            -0.005260062403976917,
+            0.014841294847428799,
+            -0.026424745097756386,
+            -0.024056773632764816,
+            -0.010361758060753345,
+            -0.015180653892457485,
+            -0.04760073870420456,
+            0.0017759781330823898,
+            -0.02028612047433853,
+            0.017118770629167557,
+            0.046152807772159576,
+            0.03052721731364727,
+            -0.02469024434685707,
+            0.014072081074118614,
+            -0.008385934866964817,
+            -0.054689567536115646,
+            -0.025595201179385185,
+            0.010097811929881573,
+            -0.020738597959280014,
+            -0.041235875338315964,
+            -0.02043694630265236,
+            0.0005128089687786996,
+            -0.022669173777103424,
+            0.016168564558029175,
+            -0.024539416655898094,
+            -0.03360407054424286,
+            0.000932294235099107,
+            0.0206631850451231,
+            -0.0028920918703079224,
+            -0.017224349081516266,
+            -0.02153797633945942,
+            -0.02562536671757698,
+            0.013793053105473518,
+            -0.004573803395032883,
+            0.0009091990068554878,
+            0.019592318683862686,
+            0.03254828602075577,
+            0.025534870103001595,
+            0.0035406441893428564,
+            -0.03333258256316185,
+            -0.021191077306866646,
+            0.026032596826553345,
+            0.0018824990838766098,
+            -0.0013084170641377568,
+            0.03245779126882553,
+            -0.03653009608387947,
+            -0.04325694590806961,
+            0.005554173607379198,
+            -0.04910900071263313,
+            0.0020418092608451843,
+            -0.021462563425302505,
+            -0.022503264248371124,
+            -0.013898631557822227,
+            0.002277475083246827,
+            0.003847952699288726,
+            0.02909436821937561,
+            0.011658863164484501,
+            -0.010482418350875378,
+            -0.023438386619091034,
+            0.008521678857505322,
+            0.0483548678457737,
+            -0.010060105472803116,
+            -0.02710346318781376,
+            0.03278961032629013,
+            -0.016455134376883507,
+            0.01168148685246706,
+            -0.011425082571804523,
+            -0.022895412519574165,
+            -0.01592724397778511,
+            0.017827652394771576,
+            0.01895884983241558,
+            0.006172561086714268,
+            -0.041235875338315964,
+            0.025474539026618004,
+            -0.0063497815281152725,
+            -0.03840034082531929,
+            0.014056999236345291,
+            0.004128866363316774,
+            0.01239791139960289,
+            0.0005486301961354911,
+            0.013174666091799736,
+            -0.054297421127557755,
+            -0.005844513885676861,
+            0.006036817561835051,
+            -0.03191481530666351,
+            -0.0058595966547727585,
+            -0.008378393948078156,
+            0.02280491590499878,
+            -0.014946873299777508,
+            0.015391810797154903,
+            -0.012405452318489552,
+            0.008144613355398178,
+            0.031975146383047104,
+            -0.03441853076219559,
+            -0.015067534521222115,
+            -0.036228444427251816,
+            -0.0041929674334824085,
+            -0.022458016872406006,
+            0.020361533388495445,
+            0.04334744065999985,
+            -0.017661744728684425,
+            0.00917777232825756,
+            0.0028185639530420303,
+            -0.004241986200213432,
+            -0.022005537524819374,
+            0.016817117109894753,
+            0.0569821260869503,
+            0.031100355088710785,
+            -0.008039034903049469,
+            -0.0006443105521611869,
+            0.02197537198662758,
+            -0.013121876865625381,
+            -0.008340687490999699,
+            0.006051899865269661,
+            0.017752239480614662,
+            0.010701116174459457,
+            0.02286524698138237,
+            -0.040089596062898636,
+            -0.010150601156055927,
+            0.009109900332987309,
+            -0.02469024434685707,
+            0.027812344953417778,
+            -0.03073837235569954,
+            0.018973931670188904,
+            0.017812570556998253,
+            -0.01956215314567089,
+            0.013627144508063793,
+            -0.010022399015724659,
+            -0.023830534890294075,
+            -0.02512763999402523,
+            -0.02496173046529293,
+            -0.008634798228740692,
+            0.017978480085730553,
+            -0.0139966681599617,
+            -0.05287965387105942,
+            0.04301562160253525,
+            -0.06298501044511795,
+            -0.029773086309432983,
+            0.03191481530666351,
+            -0.02027103677392006,
+            -0.014313403517007828,
+            0.008672504685819149,
+            0.01570100337266922,
+            -0.033362749963998795,
+            0.020452028140425682,
+            0.011643780395388603,
+            0.033151593059301376,
+            -0.028023501858115196,
+            -0.024931564927101135,
+            0.0034765431191772223,
+            -0.02739003300666809,
+            -0.006010422948747873,
+            -0.0002648884546943009,
+            0.014667844399809837,
+            0.00637617614120245,
+            -0.033754896372556686,
+            -0.009743370115756989,
+            0.003800819395110011,
+            0.0027506921906024218,
+            0.03348340839147568,
+            -0.004132636822760105,
+            -0.04205033555626869,
+            0.014170118607580662,
+            0.004321169573813677,
+            -0.03284993767738342,
+            0.006406341679394245,
+            0.003346455516293645,
+            -0.009185313247144222,
+            -0.016334474086761475,
+            0.005478760693222284,
+            0.03221647068858147,
+            0.030210481956601143,
+            0.017812570556998253,
+            0.007118995301425457,
+            0.009871572256088257,
+            -0.01338582206517458,
+            -0.01503736898303032,
+            0.010655868798494339,
+            -0.007782630156725645,
+            -0.03191481530666351,
+            -0.013204830698668957,
+            -0.014780964702367783,
+            0.025866687297821045,
+            -0.012563819997012615,
+            -0.01253365445882082,
+            0.005980257410556078,
+            0.045157354325056076,
+            0.014615056104958057,
+            -0.005267603788524866,
+            -0.01336319837719202,
+            0.007601638790220022,
+            0.012707104906439781,
+            -0.034358199685811996,
+            -0.022277025505900383,
+            0.022065868601202965,
+            0.0023283790796995163,
+            0.017375174909830093,
+            -0.024599747732281685,
+            -0.022216694429516792,
+            0.03239746019244194,
+            -0.03248795494437218,
+            0.030919363722205162,
+            0.01580658182501793,
+            0.006202726159244776,
+            -0.006444048136472702,
+            0.004739712458103895,
+            -0.014569807797670364,
+            0.028400568291544914,
+            -0.029350772500038147,
+            0.027193957939743996,
+            0.020421862602233887,
+            -0.006285680457949638,
+            0.060300301760435104,
+            -0.029682589694857597,
+            -0.016334474086761475,
+            0.009607627056539059,
+            -0.0043777297250926495,
+            0.015391810797154903,
+            -0.019094591960310936,
+            -0.007149160373955965,
+            -0.0060745240189135075,
+            0.005018740892410278,
+            -0.029667507857084274,
+            -0.015897078439593315,
+            0.02164355479180813,
+            -0.005987798795104027,
+            -0.0013272702926769853,
+            0.009660416282713413,
+            -0.02286524698138237,
+            0.017495835199952126,
+            -0.006542085204273462,
+            -0.04533834755420685,
+            -0.00045813448377884924,
+            0.02153797633945942,
+            -0.024388590827584267,
+            -0.015912160277366638,
+            -0.008582009002566338,
+            0.0038837739266455173,
+            -0.00595009233802557,
+            -0.02280491590499878,
+            -0.019245419651269913,
+            -0.005655981134623289,
+            -0.008385934866964817,
+            -0.027480527758598328,
+            -0.011779523454606533,
+            0.034086715430021286,
+            0.01294842641800642,
+            -0.004875455982983112,
+            -0.010301426984369755,
+            -0.029667507857084274,
+            0.016002656891942024,
+            0.04373958706855774,
+            0.0034218686632812023,
+            0.006941774394363165,
+            -0.0065948739647865295,
+            -0.0026451139710843563,
+            -0.014841294847428799,
+            0.009864031337201595,
+            0.0017646661726757884,
+            -0.0028223346453160048,
+            0.01790306530892849,
+            -0.011342127807438374,
+            -0.03381522744894028,
+            -0.026967719197273254,
+            0.04304578900337219,
+            -0.023785287514328957,
+            0.004483307711780071,
+            0.04651479050517082,
+            0.036620594561100006,
+            -0.005343017168343067,
+            -0.014305861666798592,
+            -0.03107019141316414,
+            0.0020663184113800526,
+            -0.02010512910783291,
+            0.009509589523077011,
+            0.03294043615460396,
+            -0.00039686134550720453,
+            -0.034297872334718704,
+            -0.013529106974601746,
+            -0.02099500223994255,
+            -0.010346675291657448,
+            0.015716087073087692,
+            0.017013192176818848,
+            -0.034358199685811996,
+            0.005795495584607124,
+            0.018310297280550003,
+            0.028943542391061783,
+            -0.011349668726325035,
+            -0.016983026638627052,
+            0.024765657261013985,
+            0.017224349081516266,
+            0.0009294662741012871,
+            -0.0029222569428384304,
+            -0.00034312953357584774,
+            0.0068098013289272785,
+            -0.004068535752594471,
+            0.033030930906534195,
+            0.0035274471156299114,
+            0.011206384748220444,
+            -0.02116091176867485,
+            0.00479250168427825,
+            0.020738597959280014,
+            -0.006870131939649582,
+            -0.001355550135485828,
+            0.07384449243545532,
+            0.00214173155836761,
+            0.01355173159390688,
+            0.017209265381097794,
+            0.03444869816303253,
+            0.01773715764284134,
+            -0.017993561923503876,
+            -0.03170366212725639,
+            -0.014313403517007828,
+            -0.007718529086560011,
+            -0.020527441054582596,
+            -0.0049697221256792545,
+            -0.012910719960927963,
+            -0.0433172769844532,
+            -0.0223222728818655,
+            -0.0033502262085676193,
+            -0.002466007834300399,
+            0.025263383984565735,
+            -0.0026319166645407677,
+            0.0009822554420679808,
+            -0.027948088943958282,
+            0.03565530478954315,
+            0.02622867003083229,
+            0.023996442556381226,
+            -0.007880667224526405,
+            0.011100806295871735,
+            -0.0043362523429095745,
+            -0.008491513319313526,
+            0.02754085883498192,
+            0.0074885194189846516,
+            -0.008137071505188942,
+            0.03478051349520683,
+            0.012111341580748558,
+            0.05272882804274559,
+            -0.013114335015416145,
+            -0.024343343451619148,
+            -0.017179099842905998,
+            -0.04018009081482887,
+            0.011372293345630169,
+            -0.007710987702012062,
+            0.0001368040539091453,
+            0.014139953069388866,
+            0.017390256747603416,
+            -0.008167237043380737,
+            0.0271637924015522,
+            0.0038498379290103912,
+            -0.015504930168390274,
+            0.025187969207763672,
+            -0.02892845869064331,
+            0.014253072440624237,
+            -0.0035576121881604195,
+            0.011839854530990124,
+            -0.012903179042041302,
+            0.011538201943039894,
+            0.02358921244740486,
+            -0.025761108845472336,
+            0.028551394119858742,
+            0.017254512757062912,
+            -0.0027412655763328075,
+            0.00480758398771286,
+            0.0026168341282755136,
+            -0.0012169786496087909,
+            -0.022080950438976288,
+            -0.034086715430021286,
+            0.02533879689872265,
+            -0.007473436649888754,
+            -0.023332808166742325,
+            0.02425284869968891,
+            -0.017118770629167557,
+            -0.008928908966481686,
+            -0.0017175329849123955,
+            -0.009358763694763184,
+            -0.007579015102237463,
+            0.04292512685060501,
+            -0.008664963766932487,
+            -0.004291004501283169,
+            -0.01640988700091839,
+            0.02689230628311634,
+            0.04226149246096611,
+            -0.008385934866964817,
+            -0.009072193875908852,
+            0.021899959072470665,
+            -0.012450700625777245,
+            0.0015129749663174152,
+            0.041628021746873856,
+            0.006153707858175039,
+            0.0015365416184067726,
+            0.005712541285902262,
+            -0.03393588960170746,
+            -0.029999325051903725,
+            0.005806807428598404,
+            0.016455134376883507,
+            0.016047904267907143,
+            0.04126603901386261,
+            0.03646976873278618,
+            -0.014969496987760067,
+            0.0025037142913788557,
+            0.0007546021952293813,
+            -0.035232994705438614,
+            0.028747467324137688,
+            -0.005486302077770233,
+            0.021447481587529182,
+            0.02485615201294422,
+            -0.026183422654867172,
+            -0.036620594561100006,
+            -0.00034996384056285024,
+            0.030210481956601143,
+            0.0190040972083807,
+            0.014056999236345291,
+            0.014072081074118614,
+            0.023227229714393616,
+            -0.020316284149885178,
+            -0.024946648627519608,
+            0.02012021094560623,
+            0.02506730891764164,
+            -0.009456800296902657,
+            0.01074636448174715,
+            0.017933230847120285,
+            0.009818783961236477,
+            -0.0012282906100153923,
+            0.03134167939424515,
+            0.009019404649734497,
+            0.016289226710796356,
+            0.008287898264825344,
+            -0.011515578255057335,
+            0.02500697784125805,
+            0.010271262377500534,
+            0.03393588960170746,
+            -0.012412993237376213,
+            0.003929021768271923,
+            0.003689585020765662,
+            -0.004087388981133699,
+            0.037163566797971725,
+            0.005946321878582239,
+            -0.04307595267891884,
+            0.0199995506554842,
+            0.011960514821112156,
+            -0.0001776920835254714,
+            -0.006489295978099108,
+            -0.0026469992008060217,
+            0.018446041271090508,
+            0.02043694630265236,
+            0.018038809299468994,
+            0.009532214142382145,
+            0.005659752059727907,
+            0.005618274677544832,
+            0.006138625089079142,
+            0.03167349472641945,
+            0.0020455799531191587,
+            -0.01978839375078678,
+            -0.00230009900406003,
+            0.005505155306309462,
+            0.021115664392709732,
+            -0.019637567922472954,
+            0.01653054729104042,
+            0.019215254113078117,
+            -0.0011934119975194335,
+            -0.01746566966176033,
+            -0.03671108931303024,
+            -0.007303757127374411,
+            -0.013853383250534534,
+            0.013898631557822227,
+            0.008951532654464245,
+            0.018687361851334572,
+            -0.0221563633531332,
+            0.016078069806098938,
+            0.019592318683862686,
+            -0.01950182393193245,
+            0.006496837362647057,
+            0.008061658591032028,
+            -0.014524560421705246,
+            -0.005580568220466375,
+            -0.016108235344290733,
+            0.002053121104836464,
+            0.06382963061332703,
+            -0.014464229345321655,
+            0.009954527020454407,
+            0.024931564927101135,
+            -0.009879114106297493,
+            -0.030119985342025757,
+            0.0009671727893874049,
+            -0.010429629124701023,
+            -0.002450925298035145,
+            0.024373508989810944,
+            -0.03577596694231033,
+            -0.04319661483168602,
+            -0.005290227942168713,
+            -0.03631894290447235,
+            -0.006560938432812691,
+            0.019034262746572495,
+            0.027948088943958282,
+            -0.014162576757371426,
+            -0.02760118804872036,
+            0.02292557805776596,
+            -0.0020418092608451843,
+            0.009765994735062122,
+            -0.019471658393740654,
+            -0.026545405387878418,
+            -0.03030097670853138,
+            -0.03212597221136093,
+            -0.00961516797542572,
+            -0.025097474455833435,
+            0.008408558554947376,
+            0.0012725958367809653,
+            -0.05061726272106171,
+            -0.030602630227804184,
+            0.010670951567590237,
+            -0.002435842528939247,
+            0.04005942866206169,
+            0.039757777005434036,
+            0.003970498684793711,
+            0.028008420020341873,
+            -0.03613794967532158,
+            0.050828419625759125,
+            -0.006530772894620895,
+            0.03592679277062416,
+            -0.007579015102237463,
+            0.0019890200346708298,
+            -0.014358650892972946,
+            0.04379991814494133,
+            0.000892702373676002,
+            0.002547076903283596,
+            -0.038762323558330536,
+            0.0021455022506415844,
+            -0.023996442556381226,
+            -0.00025451916735619307,
+            -0.0026752790436148643,
+            0.008996780961751938,
+            -0.0009907394414767623,
+            -0.014856377616524696,
+            0.002102139638736844,
+            0.03345324471592903,
+            0.02616834081709385,
+            -0.021779298782348633,
+            -0.027088379487395287,
+            -0.030617712065577507,
+            0.0004138292861171067,
+            -0.008725293911993504,
+            -0.002880779793485999,
+            -0.004860373213887215,
+            -0.024313177913427353,
+            0.014396357350051403,
+            -0.016771869733929634,
+            -0.013415987603366375,
+            -0.002205832628533244,
+            0.02545945718884468,
+            -0.01746566966176033,
+            -0.02259376086294651,
+            -0.005814348813146353,
+            -0.029335688799619675,
+            -0.01675678789615631,
+            0.00022930291015654802,
+            -0.02485615201294422,
+            0.017510918900370598,
+            0.01609315164387226,
+            -0.012103799730539322,
+            0.0002860983950085938,
+            0.0022001767065376043,
+            0.016123317182064056,
+            -0.021794380620121956,
+            -0.004166572820395231,
+            0.0025357650592923164,
+            -0.00017439275688957423,
+            -0.023151816800236702,
+            0.021387150511145592,
+            -0.019411327317357063,
+            0.01631939224898815,
+            -0.04844536632299423,
+            -0.011213925667107105,
+            -0.025037143379449844,
+            0.010587996803224087,
+            0.001273538451641798,
+            -0.02990882843732834,
+            -0.014072081074118614,
+            0.00785050168633461,
+            -0.0031447256915271282,
+            -0.02182454615831375,
+            0.008506596088409424,
+            0.03541398420929909,
+            -0.0014385045506060123,
+            0.009683039970695972,
+            -0.029773086309432983,
+            -0.011590991169214249,
+            -0.010158142074942589,
+            0.03079870343208313,
+            0.0037574570160359144,
+            -0.005131860263645649,
+            -0.00574270635843277,
+            0.0005769100971519947,
+            -0.03472018241882324,
+            -0.021899959072470665,
+            -0.003259730525314808,
+            -0.002062547951936722,
+            -0.005392035469412804,
+            0.034509025514125824,
+            0.03405654802918434,
+            0.03686191514134407,
+            -0.04379991814494133,
+            0.019215254113078117,
+            -0.008325604721903801,
+            -0.011108347214758396,
+            0.01325007900595665,
+            0.018777858465909958,
+            0.01840079203248024,
+            -0.016485299915075302,
+            0.001463956548832357,
+            0.003412442049011588,
+            0.013536648824810982,
+            -0.005603192374110222,
+            0.022985907271504402,
+            -0.00719063775613904,
+            0.013408446684479713,
+            0.014260614290833473,
+            -8.476607035845518e-06,
+            0.022955743595957756,
+            -0.006843737326562405,
+            0.009901737794280052,
+            -0.007043581921607256,
+            -0.014094705693423748,
+            0.0017872900934889913,
+            -0.03882265463471413,
+            -0.013016298413276672,
+            0.015384268946945667,
+            0.018777858465909958,
+            -0.012495948001742363,
+            -0.04874701797962189,
+            0.022020621225237846,
+            0.03671108931303024,
+            -0.019275585189461708,
+            0.0273297019302845,
+            -0.0279933363199234,
+            0.016334474086761475,
+            0.011613614857196808,
+            0.013272702693939209,
+            0.004773647990077734,
+            0.020738597959280014,
+            -0.02512763999402523,
+            0.033966053277254105,
+            -0.0019550842698663473,
+            0.025761108845472336,
+            0.016847282648086548,
+            -0.015391810797154903,
+            -0.04573049396276474,
+            -0.016108235344290733,
+            0.006753241643309593,
+            -0.012028386816382408,
+            -0.002897747792303562,
+            -0.043498266488313675,
+            -0.03420737385749817,
+            0.030210481956601143,
+            0.01418520137667656,
+            -0.0493503212928772,
+            0.00014446319255512208,
+            -0.014034374617040157,
+            -0.0030730832368135452,
+            -0.0052261268720030785,
+            -0.028943542391061783,
+            -0.02336297370493412,
+            0.00801641121506691,
+            0.00752245495095849,
+            0.03981810808181763,
+            -0.023046238347887993,
+            0.026741480454802513,
+            -0.01159853208810091,
+            -0.01719418354332447,
+            -0.02005987986922264,
+            -0.00045224284986034036,
+            -0.021326819434762,
+            -0.01481867115944624,
+            0.005094153806567192,
+            0.020738597959280014,
+            0.009660416282713413,
+            0.029396019876003265,
+            -0.012722187675535679,
+            -0.014856377616524696,
+            -0.0005095096421428025,
+            -0.03465985506772995,
+            -0.028506144881248474,
+            0.007032270077615976,
+            0.0006749471067450941,
+            -0.017269596457481384,
+            -0.02070843242108822,
+            -0.03704290837049484,
+            0.016213813796639442,
+            0.011990680359303951,
+            0.01692269556224346,
+            0.03245779126882553,
+            0.0198336411267519,
+            -0.01386092510074377,
+            0.01669645681977272,
+            -0.007888208143413067,
+            0.0009813127107918262,
+            0.01013551838696003,
+            0.018581783398985863,
+            -0.013114335015416145,
+            -0.0027733161114156246,
+            -0.014765881933271885,
+            -0.029003871604800224,
+            -0.01708860509097576,
+            -0.01806897483766079,
+            -0.0008795050671324134,
+            -0.006149936933070421,
+            -0.03891315311193466,
+            -0.013415987603366375,
+            0.024991896003484726,
+            0.02396627888083458,
+            -0.01534656248986721,
+            -0.007511143106967211,
+            -0.00023295573191717267,
+            -0.0208290945738554,
+            -0.004894309211522341,
+            -0.02342330478131771,
+            -0.003672617254778743,
+            -0.024147270247340202,
+            0.013234996236860752,
+            -0.0045813447795808315,
+            0.010052564553916454,
+            -0.053694114089012146,
+            -0.013174666091799736,
+            0.007326381281018257,
+            -0.010105352848768234,
+            -0.0016996223712339997,
+            0.01746566966176033,
+            -0.003291781060397625,
+            -0.009162689559161663,
+            -0.023845616728067398,
+            0.014155035838484764,
+            0.005772871430963278,
+            0.021025167778134346,
+            -0.002831761259585619,
+            -0.00406476529315114,
+            0.03348340839147568,
+            0.025384044274687767,
+            -5.541094287764281e-05,
+            -0.012269708327949047,
+            0.015897078439593315,
+            0.014788505621254444,
+            0.0014507592422887683,
+            0.0044493721798062325,
+            0.009102359414100647,
+            0.0048264372162520885,
+            -0.010889649391174316,
+            -0.00480758398771286,
+            0.024871235713362694,
+            0.006527002435177565,
+            0.005935009568929672,
+            -0.01463013794273138,
+            0.02897370606660843,
+            0.03752554953098297,
+            0.0196677315980196,
+            0.028068749234080315,
+            -0.015685921534895897,
+            -0.011032934300601482,
+            -0.007760006468743086,
+            -0.013762887567281723,
+            0.031432174146175385,
+            0.008861036971211433,
+            -0.009871572256088257,
+            -0.0104145472869277,
+            0.010037481784820557,
+            -0.009939444251358509,
+            0.002166240941733122,
+            0.02269933745265007,
+            0.009592544287443161,
+            0.005580568220466375,
+            -0.00028892638511024415,
+            0.007548849564045668,
+            0.015912160277366638,
+            0.023770203813910484,
+            -0.02975800260901451,
+            -0.027420196682214737,
+            -0.0009120269678533077,
+            0.0003007096820510924,
+            0.012993674725294113,
+            0.01767682656645775,
+            -0.0011594761162996292,
+            0.006658975500613451,
+            0.04334744065999985,
+            -0.015746252611279488,
+            -0.0034369511995464563,
+            -0.014698009938001633,
+            -0.013272702693939209,
+            -0.0022491952404379845,
+            -0.021945208311080933,
+            0.008687587454915047,
+            0.012382828630506992,
+            0.017118770629167557,
+            0.014245531521737576,
+            -0.008023952133953571,
+            0.012322497554123402,
+            0.018446041271090508,
+            0.011515578255057335,
+            -0.004219362046569586,
+            -0.014780964702367783,
+            -0.002564044902101159,
+            0.015248525887727737,
+            0.009034487418830395,
+            0.022835081443190575,
+            0.021326819434762,
+            0.006949315778911114,
+            0.01322745531797409,
+            -0.025052227079868317,
+            0.025746027007699013,
+            0.0144114401191473,
+            0.02419251762330532,
+            -0.03770654276013374,
+            -0.021145828068256378,
+            0.014501935802400112,
+            -0.007141618989408016,
+            -0.0004892423748970032,
+            0.015384268946945667,
+            -0.008091824129223824,
+            0.05125073343515396,
+            -0.0353536531329155,
+            0.00845380686223507,
+            0.02358921244740486,
+            -0.015881994739174843,
+            0.010271262377500534,
+            0.01818963512778282,
+            0.010595538653433323,
+            -0.010678492486476898,
+            -0.02093467302620411,
+            0.00037730109761469066,
+            0.02551978826522827,
+            -0.03818918392062187,
+            -0.018611948937177658,
+            -0.0065345438197255135,
+            -0.0004892423748970032,
+            0.022729502990841866,
+            -0.007552620489150286,
+            0.014645220711827278,
+            -0.008928908966481686,
+            0.011432623490691185,
+            0.001585560035891831,
+            -0.03134167939424515,
+            -0.007537537720054388,
+            0.014637679792940617,
+            0.01231495663523674,
+            -0.009667957201600075,
+            -0.006674057804048061,
+            -0.0010331592056900263,
+            -0.002075745025649667,
+            -0.014268155209720135,
+            0.02722412347793579,
+            0.0018617605092003942,
+            0.01592724397778511,
+            -0.00208705710247159,
+            -0.014207825064659119,
+            0.018551617860794067,
+            -0.01962248422205448,
+            -0.015256066806614399,
+            -0.004521014168858528,
+            0.001125540235079825,
+            -0.0011905840365216136,
+            -0.031100355088710785,
+            0.015361645258963108,
+            -0.013257619924843311,
+            0.034629687666893005,
+            0.006180102005600929,
+            -0.0012207493418827653,
+            -0.002481090370565653,
+            -0.011606073938310146,
+            -0.0056145042181015015,
+            -0.02821957692503929,
+            -0.017872901633381844,
+            0.01686236634850502,
+            -0.013536648824810982,
+            -0.009909279644489288,
+            -0.026515239849686623,
+            -0.021734051406383514,
+            -0.011040475219488144,
+            0.008793165907263756,
+            0.0014347339747473598,
+            0.022850165143609047,
+            0.00864988099783659,
+            -0.017375174909830093,
+            -0.0033671942073851824,
+            -0.041959840804338455,
+            -0.002809137338772416,
+            0.01033159252256155,
+            -0.017541082575917244,
+            -0.022246859967708588,
+            0.013204830698668957,
+            0.00048052275087684393,
+            -0.018596867099404335,
+            0.019411327317357063,
+            0.0019183203112334013,
+            0.004434289410710335,
+            0.0017147050239145756,
+            -0.0023151817731559277,
+            -0.01653054729104042,
+            0.00573139451444149,
+            0.009554837830364704,
+            0.01536918617784977,
+            -0.047238755971193314,
+            -0.007499831262975931,
+            0.003244647989049554,
+            0.006372405681759119,
+            0.006010422948747873,
+            0.01118376012891531,
+            0.004777418915182352,
+            -0.017767323181033134,
+            0.004471995867788792,
+            0.01906442828476429,
+            -0.022563595324754715,
+            0.01977331005036831,
+            -0.0019051230046898127,
+            0.0027883988805115223,
+            0.002799710724502802,
+            -0.007514914032071829,
+            -0.005490072537213564,
+            0.02583652175962925,
+            -0.010452253744006157,
+            0.009622709825634956,
+            0.021507810801267624,
+            0.02309148572385311,
+            -0.011470329947769642,
+            -0.01862703263759613,
+            0.013710098341107368,
+            0.0316433310508728,
+            0.0006923863547854125,
+            -0.014992120675742626,
+            -0.054086264222860336,
+            -0.011206384748220444,
+            0.03218630328774452,
+            0.015897078439593315,
+            0.017390256747603416,
+            -0.006941774394363165,
+            -0.012518571689724922,
+            0.019139841198921204,
+            0.023559046909213066,
+            0.017043357715010643,
+            -0.026545405387878418,
+            0.013928796164691448,
+            -0.020844176411628723,
+            0.03432803601026535,
+            0.0030429179314523935,
+            -0.008008869364857674,
+            0.004143949132412672,
+            -0.013431070372462273,
+            -0.02313673496246338,
+            0.014200284145772457,
+            -0.003846067236736417,
+            0.0028826650232076645,
+            0.006545855663716793,
+            -0.004864144138991833,
+            0.01939624547958374,
+            -0.0008766771061345935,
+            -0.005897303111851215,
+            0.004747253842651844,
+            0.02787267602980137,
+            0.0013659194810315967,
+            0.003993122838437557,
+            0.0011085723526775837,
+            0.010090271010994911,
+            -0.0019946759566664696,
+            -0.012586443684995174,
+            0.022563595324754715,
+            -0.012865472584962845,
+            -0.0056145042181015015,
+            0.01923033595085144,
+            -0.007903290912508965,
+            -0.00023342706845141947,
+            -0.010452253744006157,
+            -0.008838413283228874,
+            -0.03152266889810562,
+            0.026183422654867172,
+            -0.0036066307220607996,
+            0.014396357350051403,
+            0.021010085940361023,
+            -0.0012094373814761639,
+            -0.005410888697952032,
+            0.022337354719638824,
+            0.0002851557219401002,
+            -0.007458353880792856,
+            0.01386092510074377,
+            -0.02562536671757698,
+            -0.02612309157848358,
+            0.03858133405447006,
+            -0.012269708327949047,
+            0.0005542861763387918,
+            -0.0022491952404379845,
+            -0.00906465295702219,
+            -0.007511143106967211,
+            0.020542524755001068,
+            0.01708860509097576,
+            -0.02793300710618496,
+            -0.010595538653433323,
+            -0.010127977468073368,
+            -0.02909436821937561,
+            0.011590991169214249,
+            -0.0018532765097916126,
+            0.02153797633945942,
+            -0.005799266044050455,
+            -0.03155283257365227,
+            -0.004762336146086454,
+            0.03592679277062416,
+            -0.003981810994446278,
+            0.0010048793628811836,
+            0.015323938801884651,
+            0.0028675824869424105,
+            0.015610508620738983,
+            -0.029456350952386856,
+            0.025474539026618004,
+            -0.01640988700091839,
+            0.020255954936146736,
+            0.013483859598636627,
+            -0.014464229345321655,
+            0.027736932039260864,
+            0.013234996236860752,
+            0.005493842996656895,
+            0.006413882598280907,
+            0.0030052114743739367,
+            -0.010859483852982521,
+            -0.007077517919242382,
+            0.008340687490999699,
+            -0.002134190173819661,
+            0.0003591548011172563,
+            0.02594210021197796,
+            0.0003921480383723974,
+            0.013762887567281723,
+            0.001332926214672625,
+            -0.0001895932073239237,
+            0.0017618382116779685,
+            -0.002952422248199582,
+            -0.01944149285554886,
+            0.011304421350359917,
+            -0.015912160277366638,
+            -0.020964836701750755,
+            0.012443158775568008,
+            -0.017269596457481384,
+            -0.01785781793296337,
+            -0.028113998472690582,
+            0.01283530704677105,
+            -0.008861036971211433,
+            -0.007337693125009537,
+            0.009667957201600075,
+            0.011764440685510635,
+            0.015595425851643085,
+            0.02660573646426201,
+            0.019863806664943695,
+            -0.012744811363518238,
+            0.013762887567281723,
+            0.010489960201084614,
+            0.008830872364342213,
+            -0.00939647015184164,
+            0.0018146273214370012,
+            -0.010308968834578991,
+            0.020240871235728264,
+            0.010912273079156876,
+            0.004038370680063963,
+            0.019200170412659645,
+            -0.007669510785490274,
+            -0.01264677383005619,
+            -0.03505200147628784,
+            -0.004882997367531061,
+            -0.001891925698146224,
+            0.007262280210852623,
+            0.04998379200696945,
+            -0.014245531521737576,
+            -0.03300076723098755,
+            -0.016847282648086548,
+            -0.011802148073911667,
+            0.009645333513617516,
+            0.0012443158775568008,
+            0.010723740793764591,
+            -0.013491400517523289,
+            -0.035564810037612915,
+            0.0030825098510831594,
+            0.015776416286826134,
+            0.003531217575073242,
+            -0.0020041028037667274,
+            0.022337354719638824,
+            -0.028113998472690582,
+            0.0180086437612772,
+            -0.007514914032071829,
+            -0.011470329947769642,
+            -0.02088942378759384,
+            -0.0033671942073851824,
+            -0.0076581984758377075,
+            0.023845616728067398,
+            -0.005497613921761513,
+            -0.01978839375078678,
+            -0.003457689890637994,
+            0.011357210576534271,
+            -0.0009516188292764127,
+            -0.005659752059727907,
+            0.004928245209157467,
+            0.006406341679394245,
+            -0.0029769313987344503,
+            0.033905722200870514,
+            -0.0017128196777775884,
+            -0.0024792051408439875,
+            -0.0063196164555847645,
+            -0.0027733161114156246,
+            0.00947942491620779,
+            0.015022286213934422,
+            -0.047087930142879486,
+            -0.019878888502717018,
+            -0.007254738826304674,
+            -0.004079848062247038,
+            0.0051544844172894955,
+            0.012722187675535679,
+            -0.02104024961590767,
+            -0.00375934224575758,
+            -0.011191301979124546,
+            -0.008536760695278645,
+            -0.029516680166125298,
+            -0.010889649391174316,
+            0.03478051349520683,
+            -0.009426635690033436,
+            0.039094142615795135,
+            0.021130746230483055,
+            -0.0020436944905668497,
+            0.015052451752126217,
+            0.00261117797344923,
+            0.002690361812710762,
+            -0.00906465295702219,
+            -0.013936338014900684,
+            -0.0019626254215836525,
+            -0.005486302077770233,
+            0.0025489621330052614,
+            -0.03960695117712021,
+            0.0004977263743057847,
+            -0.0033276022877544165,
+            0.016998108476400375,
+            -0.035685472190380096,
+            -0.008891202509403229,
+            0.026982801035046577,
+            -0.012035928666591644,
+            0.005026282276958227,
+            -0.003996893297880888,
+            0.01653054729104042,
+            0.042231328785419464,
+            0.011455247178673744,
+            -0.003372850129380822,
+            -0.017782405018806458,
+            -0.0002785570686683059,
+            0.004988575354218483,
+            -0.03846067190170288,
+            0.004732171073555946,
+            -0.039697445929050446,
+            0.030180316418409348,
+            -0.00679094810038805,
+            -0.008333145640790462,
+            0.03541398420929909,
+            -0.018038809299468994,
+            0.01473571639508009,
+            0.009879114106297493,
+            -0.022186528891324997,
+            0.02693755365908146,
+            0.002552732825279236,
+            0.018551617860794067,
+            -0.04147719591856003,
+            0.00793345645070076,
+            -0.008295439183712006,
+            0.00950204860419035,
+            -0.01642496883869171,
+            -0.0017646661726757884,
+            0.016575796529650688,
+            -0.010127977468073368,
+            0.014449146576225758,
+            0.020135292783379555,
+            -0.001117998966947198,
+            0.0006494951667264104,
+            0.021899959072470665,
+            -0.006527002435177565,
+            0.0060330466367304325,
+            0.015489847399294376,
+            0.013868466019630432,
+            0.01346877682954073,
+            0.037827201187610626,
+            -0.002782742725685239,
+            0.0012320613022893667,
+            0.003993122838437557,
+            0.001916434965096414,
+            0.010241096839308739,
+            -0.0067796362563967705,
+            -0.001194354728795588,
+            -0.016605960205197334,
+            0.00782033707946539,
+            0.017013192176818848,
+            -0.006379947066307068,
+            -0.001372518134303391,
+            -0.030376389622688293,
+            -0.010037481784820557,
+            -0.04871685057878494,
+            0.006112230475991964,
+            -0.030919363722205162,
+            -0.010150601156055927,
+            -0.008416100405156612,
+            0.018385710194706917,
+            -0.003538758959621191,
+            0.012412993237376213,
+            0.001619496033526957,
+            -0.019984466955065727,
+            -0.005135631188750267,
+            0.008310521952807903,
+            -0.015059992671012878,
+            0.02325739525258541,
+            0.0071680136024951935,
+            -0.006440277211368084,
+            0.00266585242934525,
+            -0.013348115608096123,
+            -0.006142395548522472,
+            -0.015090158209204674,
+            -0.007017187308520079,
+            0.002581012900918722,
+            0.022654090076684952,
+            0.004143949132412672,
+            -0.0013828874798491597,
+            0.00010999705409631133,
+            -0.007752465084195137,
+            0.015791499987244606,
+            0.0027092150412499905,
+            -0.0003994536818936467,
+            0.004984804894775152,
+            0.015135405585169792,
+            0.008355769328773022,
+            -0.0059312391094863415,
+            0.008446265012025833,
+            0.006447818595916033,
+            -0.009592544287443161,
+            0.025821439921855927,
+            -0.008159696124494076,
+            0.009042028337717056,
+            -0.019712980836629868,
+            0.007639345247298479,
+            0.011093264445662498,
+            -0.014456688426434994,
+            -0.008023952133953571,
+            -0.02155306003987789,
+            0.02076876349747181,
+            0.007473436649888754,
+            0.0074885194189846516,
+            -0.024720408022403717,
+            0.014139953069388866,
+            -0.0055730268359184265,
+            -0.0013706327881664038,
+            -0.024780739098787308,
+            0.016213813796639442,
+            0.024886317551136017,
+            0.00906465295702219,
+            0.02027103677392006,
+            -0.003993122838437557,
+            0.008717752061784267,
+            -0.016017738729715347,
+            0.002801596187055111,
+            -0.012330039404332638,
+            -0.020783845335245132,
+            0.0033992447424679995,
+            0.033151593059301376,
+            -0.0016279799165204167,
+            -0.005870908498764038,
+            0.001966396113857627,
+            0.006131083704531193,
+            -0.004841519985347986,
+            -0.007707217242568731,
+            0.012247084639966488,
+            0.01344615314155817,
+            0.004649216774851084,
+            0.006006652023643255,
+            0.06690648943185806,
+            -0.004015746526420116,
+            0.004879226442426443,
+            -0.0011707880767062306,
+            -0.0031767762266099453,
+            -0.0034011302050203085,
+            -0.001456415164284408,
+            -0.02469024434685707,
+            0.019984466955065727,
+            0.0036556492559611797,
+            -0.009057111106812954,
+            0.009675499051809311,
+            -0.021085498854517937,
+            0.0006980423349887133,
+            0.007051123306155205,
+            0.011176219210028648,
+            0.001268825144506991,
+            0.016078069806098938,
+            0.03161316365003586,
+            0.015376728028059006,
+            -0.016666291281580925,
+            0.005180879030376673,
+            0.01116113644093275,
+            0.006406341679394245,
+            -0.00740933557972312,
+            0.02408693917095661,
+            0.0013781740562990308,
+            0.023815451189875603,
+            0.0019852493423968554,
+            0.020859258249402046,
+            0.010995227843523026,
+            -0.005554173607379198,
+            0.0007465895032510161,
+            -0.008582009002566338,
+            0.0064629013650119305,
+            0.012631691992282867,
+            0.0006551511469297111,
+            0.00011624219769146293,
+            0.0008257732843048871,
+            0.011372293345630169,
+            0.00801641121506691,
+            0.0018089713994413614,
+            -0.005953862797468901,
+            -0.007213261444121599,
+            -0.00876300036907196,
+            0.010957521386444569,
+            0.0025565035175532103,
+            -0.0005443881964311004,
+            -0.01063324511051178,
+            -0.002959963632747531,
+            -0.00010993813339155167,
+            0.013913714326918125,
+            0.016605960205197334,
+            0.009358763694763184,
+            -0.012654315680265427,
+            -0.004166572820395231,
+            0.0208290945738554,
+            -0.004852831829339266,
+            0.024916483089327812,
+            0.010670951567590237,
+            0.004422977566719055,
+            0.014418981969356537,
+            -0.016651209443807602,
+            -0.016349555924534798,
+            0.0030768539290875196,
+            -0.0021323049440979958,
+            0.011508036404848099,
+            0.010640786029398441,
+            -0.002104025101289153,
+            0.020406780764460564,
+            -0.007360316812992096,
+            -0.0025828981306403875,
+            0.015731168910861015,
+            0.016832200810313225,
+            -0.004215591587126255,
+            0.005275145173072815,
+            -0.015881994739174843,
+            -0.016334474086761475,
+            -0.005655981134623289,
+            0.00012619908375199884,
+            0.005810578353703022,
+            0.012873013503849506,
+            -5.217053694650531e-05,
+            0.014607514254748821,
+            -0.005870908498764038,
+            -0.025384044274687767,
+            -0.0007234942750073969,
+            0.003993122838437557,
+            0.01603282243013382,
+            -0.007088829763233662,
+            0.002594209974631667,
+            0.0016977370250970125,
+            0.021749133244156837,
+            -0.02099500223994255,
+            0.0038762325420975685,
+            0.0003674031177069992,
+            -0.01325007900595665,
+            0.0005245922366157174,
+            -0.007978703826665878,
+            -0.012458241544663906,
+            0.015331479720771313,
+            0.005561714991927147,
+            -0.013521566055715084,
+            0.018310297280550003,
+            -0.01052766665816307,
+            -0.0012150933034718037,
+            0.02199045568704605,
+            0.010648327879607677,
+            0.004362646955996752,
+            -0.015278690494596958,
+            -0.0056748343631625175,
+            -0.014871460385620594,
+            0.022382603958249092,
+            0.01708860509097576,
+            -0.004260839428752661,
+            0.003921480383723974,
+            0.013679933734238148,
+            0.0046944646164774895,
+            -0.018883436918258667,
+            -0.010648327879607677,
+            -0.0004682681174017489,
+            -0.010210931301116943,
+            0.004728400148451328,
+            0.0072396560572087765,
+            0.02010512910783291,
+            -0.016500383615493774,
+            -0.005995340179651976,
+            -0.006119771860539913,
+            -0.02930552512407303,
+            -0.029275359585881233,
+            -0.004411665257066488,
+            0.0024867465253919363,
+            -0.0024792051408439875,
+            -0.0026545405853539705,
+            -0.0019550842698663473,
+            0.002839302644133568,
+            0.0005000830278731883,
+            -0.008137071505188942,
+            0.013740263879299164,
+            -0.000598591344896704,
+            0.005554173607379198,
+            0.022126199677586555,
+            -0.002910945098847151,
+            -0.013544189743697643,
+            0.03194498270750046,
+            0.011455247178673744,
+            -0.009743370115756989,
+            -0.027344783768057823,
+            0.01211888249963522,
+            -0.011063099838793278,
+            0.017510918900370598,
+            -0.043558597564697266,
+            -0.004962180741131306,
+            -0.0038592645432800055,
+            -0.007356546353548765,
+            -0.011606073938310146,
+            -0.011002768762409687,
+            -0.016349555924534798,
+            -0.006862590555101633,
+            -0.010949979536235332,
+            -0.022231776267290115,
+            0.007963621988892555,
+            -0.005312851630151272,
+            0.02656048908829689,
+            -0.022216694429516792,
+            0.024448921903967857,
+            0.010158142074942589,
+            -0.003913938999176025,
+            0.002268048468977213,
+            -0.008589549921452999,
+            0.00490562105551362,
+            0.020074963569641113,
+            0.010014857165515423,
+            0.00011942369019379839,
+            0.021010085940361023,
+            0.008883661590516567,
+            -0.0038290992379188538,
+            -0.0038988564629107714,
+            -0.005433512851595879,
+            0.0246751606464386,
+            0.004422977566719055,
+            0.004660528618842363,
+            -0.008076741360127926,
+            0.011055557988584042,
+            0.027043132111430168,
+            0.007111453916877508,
+            -0.009336140006780624,
+            -0.005999110639095306,
+            0.0027959400322288275,
+            -0.011658863164484501,
+            -0.008416100405156612,
+            0.011093264445662498,
+            -0.01098768599331379,
+            -0.023543965071439743,
+            0.01002993993461132,
+            0.02291049435734749,
+            0.002745036268606782,
+            -0.015881994739174843,
+            -0.005640898831188679,
+            -0.0024716637562960386,
+            0.006685370113700628,
+            0.017827652394771576,
+            -0.010980145074427128,
+            -0.0144114401191473,
+            0.010542749427258968,
+            0.004057223908603191,
+            -0.00531662255525589,
+            0.02155306003987789,
+            0.030119985342025757,
+            0.011651321314275265,
+            -0.04367925971746445,
+            -0.005663522519171238,
+            0.005867138039320707,
+            -0.013823218643665314,
+            -0.047962721437215805,
+            0.004784960299730301,
+            -0.008416100405156612,
+            -0.013649768196046352,
+            0.014456688426434994,
+            -0.009562378749251366,
+            -0.014999662525951862,
+            0.02104024961590767,
+            0.026213588193058968,
+            0.0052977693267166615,
+            0.012910719960927963,
+            -0.01418520137667656,
+            -0.019909054040908813,
+            -0.020044798031449318,
+            0.00815215427428484,
+            -0.017706992104649544,
+            -0.009607627056539059,
+            0.0023151817731559277,
+            -0.013333032839000225,
+            0.02297082543373108,
+            0.012412993237376213,
+            0.03152266889810562,
+            0.009683039970695972,
+            -0.004702006001025438,
+            -0.02529354766011238,
+            -0.01719418354332447,
+            -0.002509370446205139,
+            -0.0010963176609948277,
+            0.0049697221256792545,
+            0.005456136539578438,
+            0.017510918900370598,
+            0.012963509187102318,
+            0.009517131373286247,
+            -0.013008756563067436,
+            0.015301315113902092,
+            -2.8662863769568503e-05,
+            -0.003806475317105651,
+            -0.014290779829025269,
+            -0.029561929404735565,
+            -0.0033615382853895426,
+            0.006383717525750399,
+            -0.0040911599062383175,
+            -0.004630363546311855,
+            0.0063196164555847645,
+            0.030014406889677048,
+            -0.009192855097353458,
+            0.002863811794668436,
+            -0.037435054779052734,
+            -0.003587777493521571,
+            -0.0034539191983640194,
+            0.026847057044506073,
+            0.01156836748123169,
+            -0.008755459450185299,
+            -0.006323386915028095,
+            -0.01344615314155817,
+            0.012888096272945404,
+            0.01366485096514225,
+            -0.009683039970695972,
+            0.03312142565846443,
+            0.0020116439554840326,
+            0.01396650355309248,
+            -0.0012084946502000093,
+            -0.01002993993461132,
+            0.016078069806098938,
+            0.028883211314678192,
+            -0.011281797662377357,
+            0.012767435051500797,
+            0.01190018467605114,
+            0.005512696225196123,
+            0.0017071636393666267,
+            -0.027254289016127586,
+            0.0022341127041727304,
+            -0.01021847315132618,
+            0.0214927289634943,
+            0.000284920068224892,
+            0.0015572801930829883,
+            -0.02093467302620411,
+            0.018114222213625908,
+            -0.02506730891764164,
+            -0.005060217808932066,
+            0.012443158775568008,
+            -0.0038102460093796253,
+            0.004717088304460049,
+            -0.02186979353427887,
+            -0.010535207577049732,
+            0.015007203444838524,
+            -0.018808024004101753,
+            -0.004396582953631878,
+            -0.012707104906439781,
+            -0.01553509570658207,
+            0.021356984972953796,
+            0.009750911965966225,
+            -0.018415875732898712,
+            -0.0005943493451923132,
+            0.029396019876003265,
+            -0.027239205315709114,
+            0.010753905400633812,
+            0.0180086437612772,
+            0.016711538657546043,
+            0.01344615314155817,
+            0.001777863479219377,
+            0.00884595513343811,
+            -0.0014799818163737655,
+            -0.012473324313759804,
+            -0.020014632493257523,
+            0.0196677315980196,
+            -0.01473571639508009,
+            0.019697897136211395,
+            0.010097811929881573,
+            -0.00854430254548788,
+            -0.005407118238508701,
+            -0.004815125372260809,
+            -0.003229565219953656,
+            -0.010120435617864132,
+            -0.003642451949417591,
+            0.015776416286826134,
+            0.022955743595957756,
+            -0.004068535752594471,
+            -0.018159471452236176,
+            0.017510918900370598,
+            -0.0015959293814375997,
+            0.0015139176975935698,
+            -0.0011679601157084107,
+            -0.006195184774696827,
+            0.015059992671012878,
+            0.010120435617864132,
+            -0.0002375512121943757,
+            -0.06654450297355652,
+            -0.009539755061268806,
+            -0.004287234041839838,
+            -0.004257068503648043,
+            0.002848729258403182,
+            -0.017812570556998253,
+            -0.03951645642518997,
+            0.011191301979124546,
+            0.016892530024051666,
+            -0.01471309270709753,
+            0.01592724397778511,
+            0.005678605288267136,
+            0.0017250742530450225,
+            -0.01740534044802189,
+            -0.002826105337589979,
+            0.021507810801267624,
+            0.03251812234520912,
+            -0.025052227079868317,
+            -0.01211888249963522,
+            0.012247084639966488,
+            -0.004155260976403952,
+            -0.007775088772177696,
+            -0.021326819434762,
+            -0.00029057604842819273,
+            0.01812930591404438,
+            0.02556503564119339,
+            -0.020844176411628723,
+            -0.024011526256799698,
+            4.7928548156050965e-05,
+            0.011085723526775837,
+            -0.025761108845472336,
+            -0.009924361482262611,
+            -0.015625590458512306,
+            -0.0034086713567376137,
+            0.011206384748220444,
+            -0.011870019137859344,
+            0.007624262943863869,
+            -0.012164130806922913,
+            0.005437283311039209,
+            -0.008121988736093044,
+            -0.02485615201294422,
+            0.011470329947769642,
+            -0.006055670790374279,
+            0.035444147884845734,
+            -0.006662745960056782,
+            -0.0030900510028004646,
+            0.02726937085390091,
+            -0.0008281299378722906,
+            0.005916156340390444,
+            0.014049457386136055,
+            0.03002949059009552,
+            -0.014871460385620594,
+            0.011176219210028648,
+            -0.000854524492751807,
+            -0.004882997367531061,
+            0.0005217642756178975,
+            0.03812885656952858,
+            0.004751024302095175,
+            0.0003579764743335545,
+            -0.0009945100173354149,
+            0.0015940440353006124,
+            -0.0011538201943039894,
+            0.015188194811344147,
+            0.025323713198304176,
+            -0.0012480865698307753,
+            -0.011093264445662498,
+            -0.01576133444905281,
+            -0.013868466019630432,
+            -8.383813838008791e-05,
+            0.0030881657730787992,
+            0.011085723526775837,
+            0.014056999236345291,
+            -0.006527002435177565,
+            0.01675678789615631,
+            -0.01146278902888298,
+            0.00815215427428484,
+            -0.024222683161497116,
+            0.013189747929573059,
+            -0.011666404083371162,
+            0.0006047186907380819,
+            0.0031447256915271282,
+            -0.0015261722728610039,
+            0.01325007900595665,
+            -0.00677586579695344,
+            0.004860373213887215,
+            0.012254626490175724,
+            -0.0007579015218652785,
+            0.0024414986837655306,
+            -0.021115664392709732,
+            -0.013906172476708889,
+            -0.003118331078439951,
+            0.021945208311080933,
+            -0.0030730832368135452,
+            -0.014637679792940617,
+            -4.306793925934471e-05,
+            0.004230673890560865,
+            -0.005192190874367952,
+            0.012171671725809574,
+            0.0008700784528627992,
+            -0.00031437829602509737,
+            0.020859258249402046,
+            0.0039063976146280766,
+            -0.010346675291657448,
+            0.01556526031345129,
+            -0.003041032701730728,
+            -0.0069530862383544445,
+            -0.010120435617864132,
+            0.006508149206638336,
+            0.006229120772331953,
+            -0.005177108105272055,
+            0.026364414021372795,
+            -0.03288010507822037,
+            0.0008507538586854935,
+            0.009705663658678532,
+            -0.0014102247077971697,
+            -0.0029486515559256077,
+            0.004102471750229597,
+            0.04105488210916519,
+            -0.009637792594730854,
+            -0.023830534890294075,
+            0.0002169304498238489,
+            0.01769191026687622,
+            -0.011492953635752201,
+            0.002967504784464836,
+            0.01823488436639309,
+            0.004890538286417723,
+            0.0034859697334468365,
+            0.010007316246628761,
+            -0.03704290837049484,
+            -0.004471995867788792,
+            -0.007940997369587421,
+            -0.011206384748220444,
+            0.007186866831034422,
+            0.02088942378759384,
+            0.01690761372447014,
+            0.014026833698153496,
+            0.005784183740615845,
+            0.030949529260396957,
+            0.0037970487028360367,
+            -0.003135299077257514,
+            0.0016317506087943912,
+            0.00720194960013032,
+            -0.013008756563067436,
+            0.018099140375852585,
+            -0.008137071505188942,
+            -0.015520012937486172,
+            0.004260839428752661,
+            -0.009569920599460602,
+            0.004815125372260809,
+            -0.011032934300601482,
+            0.0051243188790977,
+            -0.006904067937284708,
+            0.0005476875230669975,
+            -0.007439500652253628,
+            -0.00021657695469912142,
+            0.013076628558337688,
+            -3.543825732776895e-05,
+            0.011221466585993767,
+            0.012239543721079826,
+            -0.009057111106812954,
+            -0.024644995108246803,
+            0.012443158775568008,
+            -0.008031493052840233,
+            0.005772871430963278,
+            -0.010927355848252773,
+            0.0009073136607185006,
+            -0.0010576684726402164,
+            0.03791769966483116,
+            0.025414209812879562,
+            -0.0003365308803040534,
+            0.00593878049403429,
+            -0.028551394119858742,
+            0.007360316812992096,
+            -0.024991896003484726,
+            0.009826324880123138,
+            -0.013958961702883244,
+            -0.008287898264825344,
+            0.012171671725809574,
+            0.028460897505283356,
+            -0.0046529872342944145,
+            0.004226903431117535,
+            -0.007266050670295954,
+            -0.020150376483798027,
+            -0.007895749993622303,
+            -0.006530772894620895,
+            -0.011500495485961437,
+            -0.02055760659277439,
+            -0.010286344215273857,
+            0.01785781793296337,
+            -0.009026946499943733,
+            -0.0013678048271685839,
+            0.015286232344806194,
+            -0.0017995446687564254,
+            0.005490072537213564,
+            -0.017058439552783966,
+            0.0038837739266455173,
+            0.0062178089283406734,
+            0.01690761372447014,
+            -0.01553509570658207,
+            0.012209378182888031,
+            0.0033332582097500563,
+            -0.013091711327433586,
+            0.008106906898319721,
+            -0.021070415154099464,
+            -0.015082616358995438,
+            -0.0003834283852484077,
+            0.007695905398577452,
+            -0.010535207577049732,
+            -0.030919363722205162,
+            0.009554837830364704,
+            -0.0043777297250926495,
+            -0.007511143106967211,
+            -0.01613840088248253,
+            0.009901737794280052,
+            0.0022322272416204214,
+            0.03387555852532387,
+            -0.017284678295254707,
+            -0.006741929799318314,
+            0.019712980836629868,
+            0.009109900332987309,
+            -0.01642496883869171,
+            -0.009366304613649845,
+            0.004249527119100094,
+            -0.0012886211043223739,
+            0.006821113638579845,
+            -0.01463013794273138,
+            -0.009939444251358509,
+            0.0031145603861659765,
+            0.005637127906084061,
+            0.019471658393740654,
+            0.005324163939803839,
+            -0.0135818962007761,
+            -0.0032653864473104477,
+            0.009351222775876522,
+            0.006911609321832657,
+            0.009388929232954979,
+            -0.016349555924534798,
+            0.002688476350158453,
+            -0.012254626490175724,
+            0.002351003000512719,
+            0.011628697626292706,
+            -0.006361093837767839,
+            -0.006342240143567324,
+            0.022895412519574165,
+            0.017933230847120285,
+            -0.0018061433220282197,
+            -0.013234996236860752,
+            0.0004694464441854507,
+            -0.0002575828111730516,
+            0.01281268335878849,
+            -0.006557167507708073,
+            -0.005595650989562273,
+            0.0013838300947099924,
+            0.01576133444905281,
+            0.003944104071706533,
+            -0.005090383347123861,
+            -0.013943878933787346,
+            -0.015384268946945667,
+            0.0007640288094989955,
+            -0.027480527758598328,
+            0.0033502262085676193,
+            0.0014385045506060123,
+            0.020452028140425682,
+            0.017812570556998253,
+            0.019019179046154022,
+            -0.014358650892972946,
+            0.001570477499626577,
+            0.005007428582757711,
+            -0.00812953058630228,
+            0.002094598487019539,
+            -0.009879114106297493,
+            -0.0190040972083807,
+            -0.006504378281533718,
+            0.0058595966547727585,
+            0.018370626494288445,
+            0.004656758159399033,
+            -0.004309857729822397,
+            0.029411103576421738,
+            0.019366079941391945,
+            -0.02309148572385311,
+            0.011749358847737312,
+            -0.024554500356316566,
+            0.010301426984369755,
+            0.01962248422205448,
+            0.004592656623572111,
+            -0.016078069806098938,
+            -0.024554500356316566,
+            0.0050111995078623295,
+            -0.010301426984369755,
+            -0.008506596088409424,
+            0.008302980102598667,
+            -0.023785287514328957,
+            0.018084056675434113,
+            -0.011975597590208054,
+            0.0018268820131197572,
+            -0.010844401083886623,
+            -0.004241986200213432,
+            -0.010542749427258968,
+            -0.0031748907640576363,
+            0.007115224376320839,
+            0.01690761372447014,
+            0.005388265009969473,
+            -0.03577596694231033,
+            -0.011010310612618923,
+            0.015580343082547188,
+            -0.00520727364346385,
+            -0.0094190938398242,
+            -0.0020436944905668497,
+            0.019652649760246277,
+            -0.02164355479180813,
+            -0.006764553487300873,
+            -0.0376763753592968,
+            -0.02022578939795494,
+            -0.02755594067275524,
+            -0.028340237215161324,
+            -0.01314450055360794,
+            -0.004615280777215958,
+            0.015550177544355392,
+            0.0017986020538955927,
+            -0.008574468083679676,
+            -0.0008361425716429949,
+            -0.0108293192461133,
+            -0.0003174419398419559,
+            -0.010263720527291298,
+            -0.006417653523385525,
+            0.0033596528228372335,
+            0.007375399582087994,
+            0.002143617020919919,
+            -0.012262167409062386,
+            -0.008039034903049469,
+            -0.011809688992798328,
+            -0.010595538653433323,
+            0.00854430254548788,
+            -0.003346455516293645,
+            -0.019682815298438072,
+            0.016108235344290733,
+            0.008197402581572533,
+            0.012005763128399849,
+            0.013189747929573059,
+            0.0009275809279642999,
+            0.032095808535814285,
+            6.698803190374747e-05,
+            -0.0016854824498295784,
+            -6.112583832873497e-06,
+            -0.009026946499943733,
+            0.015376728028059006,
+            0.0077298409305512905,
+            0.008325604721903801,
+            -0.004079848062247038,
+            0.006293221842497587,
+            -0.007443271577358246,
+            0.0058181192725896835,
+            0.00018040223221760243,
+            -0.007971162907779217,
+            0.0003205056127626449,
+            0.007695905398577452,
+            0.002927913097664714,
+            -0.01418520137667656,
+            0.006817342713475227,
+            0.012699563056230545,
+            0.006934233009815216,
+            -0.01576133444905281,
+            -0.01380059402436018,
+            -0.012156588956713676,
+            -0.008499054238200188,
+            -0.028551394119858742,
+            0.022955743595957756,
+            0.043830085545778275,
+            0.000945962849073112,
+            -0.0190040972083807,
+            -0.0044380598701536655,
+            -0.015444600023329258,
+            -0.015007203444838524,
+            -0.0056748343631625175,
+            -0.005912385880947113,
+            -0.02375512197613716,
+            -0.008114447817206383,
+            0.015278690494596958,
+            0.014577348716557026,
+            0.000577381404582411,
+            -0.01570100337266922,
+            -0.005953862797468901,
+            -0.015987573191523552,
+            -0.013302868232131004,
+            -0.00908727664500475,
+            -0.021960290148854256,
+            -0.012518571689724922,
+            0.012043469585478306,
+            -0.01613840088248253,
+            0.015836747363209724,
+            0.020421862602233887,
+            0.00380836077965796,
+            -0.014109788462519646,
+            -0.015437058173120022,
+            -0.011289338581264019,
+            0.01198313944041729,
+            0.010912273079156876,
+            -0.010723740793764591,
+            0.03107019141316414,
+            -0.016440052539110184,
+            -0.009524672292172909,
+            -0.0033596528228372335,
+            0.007217032369226217,
+            -0.014260614290833473,
+            -0.004147719591856003,
+            -0.016621043905615807,
+            0.01220183726400137,
+            -0.014275697059929371,
+            0.006459130439907312,
+            0.007250967901200056,
+            -0.016334474086761475,
+            0.021326819434762,
+            0.025263383984565735,
+            0.0018579898169264197,
+            0.007635574787855148,
+            0.009962067939341068,
+            -0.013928796164691448,
+            0.010784070938825607,
+            -0.004984804894775152,
+            -0.0012575131841003895,
+            0.028129080310463905,
+            0.008536760695278645,
+            -0.0027506921906024218,
+            -0.015188194811344147,
+            0.013627144508063793,
+            0.020798929035663605,
+            -0.0023604296147823334,
+            -0.006870131939649582,
+            -0.013657309114933014,
+            0.013695016503334045,
+            0.0066363513469696045,
+            -0.00605944124981761,
+            -0.0075978683307766914,
+            -0.008989240042865276,
+            -0.0019230336183682084,
+            -0.021779298782348633,
+            0.012149048037827015,
+            -0.023830534890294075,
+            0.00014540585107170045,
+            -0.009494507685303688,
+            0.009667957201600075,
+            -0.0060028815641999245,
+            0.009939444251358509,
+            -0.010806694626808167,
+            0.013619602657854557,
+            0.007367858197540045,
+            0.0004548351571429521,
+            -0.011274255812168121,
+            0.017767323181033134,
+            0.004260839428752661,
+            0.00856692623347044,
+            0.00245281052775681,
+            -0.019170006737113,
+            0.030557380989193916,
+            -0.006625039502978325,
+            0.009434176608920097,
+            -0.010799153707921505,
+            0.012639232911169529,
+            0.021417316049337387,
+            0.028732385486364365,
+            0.011741816997528076,
+            0.0034765431191772223,
+            -0.015225901268422604,
+            0.024147270247340202,
+            0.0015846174210309982,
+            -0.010075188241899014,
+            0.02016545832157135,
+            0.004159031435847282,
+            0.005195961333811283,
+            -0.007032270077615976,
+            0.0090043218806386,
+            -0.02423776499927044,
+            -0.02551978826522827,
+            -0.007511143106967211,
+            -0.007582785561680794,
+            -0.009328598156571388,
+            0.01495441421866417,
+            -0.0017297875601798296,
+            -0.0019946759566664696,
+            0.0037178650964051485,
+            0.0032031708396971226,
+            0.007496060337871313,
+            0.002641343278810382,
+            -0.013574355281889439,
+            -0.012164130806922913,
+            -0.016711538657546043,
+            0.019215254113078117,
+            -0.008106906898319721,
+            0.001139680272899568,
+            -0.0070247286930680275,
+            0.021417316049337387,
+            0.010188307613134384,
+            0.007160472217947245,
+            0.012707104906439781,
+            -0.006877673324197531,
+            0.01729976199567318,
+            0.004649216774851084,
+            -0.012435617856681347,
+            0.015987573191523552,
+            0.026258835569024086,
+            -0.0012443158775568008,
+            -0.00364999333396554,
+            0.00190418038982898,
+            -0.04192967340350151,
+            0.00040935163269750774,
+            -0.014041916467249393,
+            0.018702445551753044,
+            -0.007888208143413067,
+            0.004988575354218483,
+            0.01459997333586216,
+            -0.005350558087229729,
+            -0.0010887763928622007,
+            0.014011750929057598,
+            0.007326381281018257,
+            0.015670839697122574,
+            0.00720194960013032,
+            0.015550177544355392,
+            -0.013657309114933014,
+            -0.03069312497973442,
+            -0.008174777962267399,
+            -0.003989351913332939,
+            0.015437058173120022,
+            0.0037103237118571997,
+            0.016228895634412766,
+            -0.030949529260396957,
+            -0.005116777960211039,
+            -0.006296992301940918,
+            0.002047465182840824,
+            0.014705551788210869,
+            -0.00354630034416914,
+            0.018038809299468994,
+            0.015489847399294376,
+            -0.008755459450185299,
+            -0.02159830741584301,
+            -0.020844176411628723,
+            0.013936338014900684,
+            0.010060105472803116,
+            -0.020361533388495445,
+            0.0020512358751147985,
+            -0.008634798228740692,
+            -0.006244203541427851,
+            -0.02963734231889248,
+            0.024283012375235558,
+            -0.003429410047829151,
+            0.01305400487035513,
+            0.011636238545179367,
+            0.010482418350875378,
+            0.013521566055715084,
+            0.02942618541419506,
+            -0.004690693691372871,
+            0.009683039970695972,
+            0.005984028335660696,
+            -0.009275808930397034,
+            -0.007164243143051863,
+            -0.009864031337201595,
+            0.015881994739174843,
+            -0.0019701668061316013,
+            0.0027412655763328075,
+            -0.003182432148605585,
+            0.017013192176818848,
+            -0.004626592621207237,
+            -0.024056773632764816,
+            -0.004204279277473688,
+            -0.001689253025688231,
+            -0.010429629124701023,
+            0.011802148073911667,
+            -0.008506596088409424,
+            -0.000715952948667109,
+            0.009735829196870327,
+            -0.004607739392668009,
+            -0.008989240042865276,
+            0.006892756093293428,
+            0.0002959963458124548,
+            -0.04117554426193237,
+            0.0005467448499985039,
+            0.004027058836072683,
+            -0.017752239480614662,
+            0.004668070003390312,
+            -0.0076280334033071995,
+            -0.010346675291657448,
+            0.0005962346913293004,
+            0.005927468184381723,
+            0.003774424782022834,
+            0.0221563633531332,
+            -0.0018711871234700084,
+            0.01438127551227808,
+            -0.006862590555101633,
+            -0.0024942876771092415,
+            -0.009750911965966225,
+            0.0135818962007761,
+            -0.020949754863977432,
+            -0.014720633625984192,
+            -0.01580658182501793,
+            -0.005007428582757711,
+            0.004577574320137501,
+            0.00458511570468545,
+            0.009539755061268806,
+            0.0013272702926769853,
+            -0.006444048136472702,
+            0.010701116174459457,
+            -0.006413882598280907,
+            -0.00032121260301209986,
+            -0.013016298413276672,
+            0.008446265012025833,
+            0.02357413060963154,
+            0.005075300578027964,
+            -0.0064930664375424385,
+            -0.013016298413276672,
+            0.01586691290140152,
+            0.03441853076219559,
+            -0.023016072809696198,
+            0.008664963766932487,
+            0.019712980836629868,
+            -0.0031371843069791794,
+            -0.002475434448570013,
+            -0.011583449319005013,
+            0.003039147239178419,
+            -0.019471658393740654,
+            0.009796159341931343,
+            0.006873902399092913,
+            -0.008227567188441753,
+            0.004882997367531061,
+            -0.015776416286826134,
+            0.0017165903700515628,
+            -0.002958078170195222,
+            -0.008710211142897606,
+            -0.005557944066822529,
+            -0.012692022137343884,
+            -0.006078294478356838,
+            0.0035180202685296535,
+            0.008393475785851479,
+            -0.004498390480875969,
+            0.01856670156121254,
+            -0.004038370680063963,
+            0.02253342978656292,
+            0.0022416538558900356,
+            -0.001372518134303391,
+            -0.009124983102083206,
+            0.0070247286930680275,
+            0.00026041080127470195,
+            -0.004769877530634403,
+            0.0017948313616216183,
+            -0.009072193875908852,
+            0.011334586888551712,
+            -0.0034312952775508165,
+            0.01752600073814392,
+            0.007009646389633417,
+            -0.008717752061784267,
+            -0.006308304611593485,
+            0.0027714308816939592,
+            -0.005712541285902262,
+            0.012322497554123402,
+            -0.004592656623572111,
+            0.01894376613199711,
+            -0.013423528522253036,
+            -0.018702445551753044,
+            0.004717088304460049,
+            0.008476430550217628,
+            0.013287785463035107,
+            -0.009532214142382145,
+            0.01002993993461132,
+            -0.0037819661665707827,
+            0.01823488436639309,
+            0.01806897483766079,
+            -0.007243426516652107,
+            0.015007203444838524,
+            -0.01962248422205448,
+            -0.0013781740562990308,
+            0.012737269513309002,
+            -0.002754462882876396,
+            -0.020361533388495445,
+            -0.011651321314275265,
+            -0.018928684294223785,
+            0.010489960201084614,
+            -0.015489847399294376,
+            0.023709872737526894,
+            0.011274255812168121,
+            0.010572914034128189,
+            -0.001968281576409936,
+            0.006198955699801445,
+            -0.0019202056573703885,
+            0.014539642259478569,
+            -0.0016835971036925912,
+            -0.004864144138991833,
+            -0.010595538653433323,
+            -0.002262392546981573,
+            -0.04057224094867706,
+            8.725529187358916e-05,
+            0.014698009938001633,
+            0.03354373946785927,
+            -0.02170388586819172,
+            0.014336027204990387,
+            -0.011651321314275265,
+            -0.0190040972083807,
+            -0.012043469585478306,
+            0.015158030204474926,
+            -0.015263608656823635,
+            0.019170006737113,
+            -0.009743370115756989,
+            -0.0047661070711910725,
+            0.001808028668165207,
+            0.01201330404728651,
+            0.003014638088643551,
+            0.0039478749968111515,
+            0.02402660809457302,
+            0.015203277580440044,
+            0.00648552505299449,
+            -0.003024064702913165,
+            0.011515578255057335,
+            0.00458511570468545,
+            -0.014863918535411358,
+            -0.005463677924126387,
+            -0.0011594761162996292,
+            -0.007639345247298479,
+            -0.010618162341415882,
+            -0.008529219776391983,
+            -0.0037800809368491173,
+            0.0008403845713473856,
+            0.018747692927718163,
+            -0.006010422948747873,
+            -0.0012942770263180137,
+            -0.012299873866140842,
+            0.012164130806922913,
+            -0.0011886986903846264,
+            0.018928684294223785,
+            -0.027359867468476295,
+            0.026741480454802513,
+            -0.03444869816303253,
+            -0.01388354878872633,
+            -0.006632580887526274,
+            -0.00015966364298947155,
+            0.011809688992798328,
+            0.0028845504857599735,
+            -0.015836747363209724,
+            -0.0008521678391844034,
+            -0.006930462550371885,
+            0.002935454249382019,
+            -0.003197514684870839,
+            0.009524672292172909,
+            0.005855826195329428,
+            -0.005659752059727907,
+            -0.021945208311080933,
+            -0.008891202509403229,
+            -0.012563819997012615,
+            -0.008355769328773022,
+            -0.013438611291348934,
+            0.013627144508063793,
+            -0.003331372980028391,
+            0.0006655204924754798,
+            0.023000990971922874,
+            -0.014283237978816032,
+            -0.01613840088248253,
+            0.015082616358995438,
+            -0.003555726958438754,
+            -0.0056446692906320095,
+            -0.01305400487035513,
+            0.007775088772177696,
+            0.0010850057005882263,
+            0.018249966204166412,
+            -0.008876119740307331,
+            0.022337354719638824,
+            -0.0045097023248672485,
+            0.012322497554123402,
+            0.018099140375852585,
+            -0.023453468456864357,
+            -0.030180316418409348,
+            0.004219362046569586,
+            -0.02793300710618496,
+            -0.0010472992435097694,
+            0.0019739374984055758,
+            -0.024946648627519608,
+            -0.014946873299777508,
+            -0.008853496052324772,
+            -0.0255047045648098,
+            -0.01630430854856968,
+            0.012918260879814625,
+            -0.0006914437399245799,
+            0.014253072440624237,
+            0.011621156707406044,
+            -0.016213813796639442,
+            0.01553509570658207,
+            -0.014418981969356537,
+            -0.02490140125155449,
+            0.0008055060170590878,
+            -0.008506596088409424,
+            0.012910719960927963,
+            0.00406476529315114,
+            0.001550681539811194,
+            0.009622709825634956,
+            -0.005320393014699221,
+            -0.0018759004306048155,
+            0.019049344584345818,
+            0.02186979353427887,
+            0.004275921732187271,
+            -0.00418542604893446,
+            0.00616879016160965,
+            -0.016651209443807602,
+            0.029275359585881233,
+            -0.01220183726400137,
+            -0.015097699128091335,
+            0.025097474455833435,
+            0.006285680457949638,
+            0.019215254113078117,
+            0.004453142639249563,
+            0.005128089804202318,
+            0.001582732074894011,
+            0.011357210576534271,
+            -0.0015261722728610039,
+            -0.01104801706969738,
+            -0.014388816431164742,
+            -0.018325379118323326,
+            0.030889200046658516,
+            0.008174777962267399,
+            0.017616497352719307,
+            -0.028672054409980774,
+            0.010799153707921505,
+            -0.008506596088409424,
+            -0.012888096272945404,
+            0.009818783961236477,
+            0.028445815667510033,
+            0.05140155926346779,
+            0.022578677162528038,
+            0.005735164973884821,
+            -0.000906370987650007,
+            0.034297872334718704,
+            -0.01303138118237257,
+            -0.009622709825634956,
+            -0.006455359980463982,
+            -0.01686236634850502,
+            0.004011976066976786,
+            0.012986132875084877,
+            -0.003627369413152337,
+            -0.008393475785851479,
+            -0.010867025703191757,
+            0.03541398420929909,
+            0.005780412815511227,
+            -0.010143060237169266,
+            0.013868466019630432,
+            -0.01956215314567089,
+            0.012216920033097267,
+            -0.017510918900370598,
+            -0.019049344584345818,
+            -0.013415987603366375,
+            0.0009813127107918262,
+            -0.02313673496246338,
+            0.006757012102752924,
+            -0.00854430254548788,
+            0.018913600593805313,
+            -0.010060105472803116,
+            -0.006017964333295822,
+            0.0034331807401031256,
+            -0.005946321878582239,
+            -0.014373733662068844,
+            0.00480758398771286,
+            -0.008023952133953571,
+            0.020783845335245132,
+            0.0005665408098138869,
+            0.008800706826150417,
+            0.0061348541639745235,
+            0.006425194907933474,
+            -0.0028298760298639536,
+            0.0012377172242850065,
+            0.012186754494905472,
+            0.015912160277366638,
+            0.009404012002050877,
+            -0.030346224084496498,
+            0.004253298044204712,
+            -0.013717640191316605,
+            0.014977038837969303,
+            0.00812953058630228,
+            0.01962248422205448,
+            0.016183648258447647,
+            0.013355657458305359,
+            -9.674084867583588e-05,
+            0.004547408781945705,
+            -0.0012961623724550009,
+            -0.0017043356783688068,
+            0.008499054238200188,
+            -0.00024226453388109803,
+            -0.0069530862383544445,
+            -0.0012047240743413568,
+            0.008031493052840233,
+            -0.008921368047595024,
+            -0.019094591960310936,
+            -0.00184950593393296,
+            -0.002364200074225664,
+            -0.01873260922729969,
+            0.0033671942073851824,
+            0.020180542021989822,
+            -0.015987573191523552,
+            -0.0016562598757445812,
+            0.01846112310886383,
+            0.02153797633945942,
+            -0.008204943500459194,
+            0.009230561554431915,
+            -0.004988575354218483,
+            -0.010693575255572796,
+            -0.02766151912510395,
+            -0.000485943048261106,
+            -0.02814416214823723,
+            0.035564810037612915,
+            -0.00886857882142067,
+            -0.018687361851334572,
+            0.012488407082855701,
+            -0.006560938432812691,
+            -0.01713385246694088,
+            -0.0068512787111103535,
+            0.005075300578027964,
+            -0.030165232717990875,
+            0.033966053277254105,
+            -0.016666291281580925,
+            -0.0009907394414767623,
+            0.020482193678617477,
+            0.05173337459564209,
+            -0.0031692348420619965,
+            -0.01746566966176033,
+            0.017254512757062912,
+            0.014426522888243198,
+            0.03188465163111687,
+            -0.01576133444905281,
+            -0.017390256747603416,
+            -0.010814236477017403,
+            -0.012873013503849506,
+            -0.011990680359303951,
+            -0.020814010873436928,
+            0.03604745492339134,
+            -0.012503488920629025,
+            -0.016726622357964516,
+            -0.0040496825240552425,
+            -0.008167237043380737,
+            -0.011289338581264019,
+            0.022654090076684952,
+            0.027450362220406532,
+            -0.0108293192461133,
+            -0.012963509187102318,
+            -0.0012235773028805852,
+            -0.023227229714393616,
+            0.02088942378759384,
+            -0.0005038536619395018,
+            -0.014698009938001633,
+            0.010867025703191757,
+            -0.0036914704833179712,
+            -0.0016949090640991926,
+            0.002579127438366413,
+            0.004751024302095175,
+            0.0569821260869503,
+            -0.015075075440108776,
+            0.002515026368200779,
+            -0.026515239849686623,
+            0.016213813796639442,
+            -0.014863918535411358,
+            -0.012149048037827015,
+            -0.013302868232131004,
+            -0.0031296429224312305,
+            -0.0158216655254364,
+            -0.02737494930624962,
+            -0.001026560552418232,
+            -0.017616497352719307,
+            0.005162025801837444,
+            -0.014464229345321655,
+            0.008069200441241264,
+            -0.024825986474752426,
+            -0.013506483286619186,
+            -0.011063099838793278,
+            -0.024373508989810944,
+            0.015082616358995438,
+            -0.030617712065577507,
+            -0.007790171541273594,
+            0.014449146576225758,
+            0.015218360349535942,
+            -0.007058664690703154,
+            -0.012179212644696236,
+            -0.0075563909485936165,
+            -0.00038366406806744635,
+            -0.05520237609744072,
+            0.002515026368200779,
+            -0.012699563056230545,
+            0.006836195942014456,
+            -0.028310071676969528,
+            0.005146943032741547,
+            0.014351109974086285,
+            0.023000990971922874,
+            0.024267930537462234,
+            0.008385934866964817,
+            0.0024490398354828358,
+            -0.01989397220313549,
+            0.02689230628311634,
+            0.005252521485090256,
+            -0.021839629858732224,
+            0.01313695963472128,
+            -0.003996893297880888,
+            -0.00015777831140439957,
+            -0.01303138118237257,
+            -0.005248750559985638,
+            0.012284791097044945,
+            0.007390482351183891,
+            -0.005791724659502506,
+            -0.007971162907779217,
+            -0.013159583322703838,
+            -0.002450925298035145,
+            0.00605944124981761,
+            0.010467336513102055,
+            0.005803036969155073,
+            0.01044471189379692,
+            -0.016847282648086548,
+            0.0029297983273863792,
+            0.005263833329081535,
+            -0.00219075009226799,
+            0.003046688623726368,
+            -0.007744923699647188,
+            0.01923033595085144,
+            -0.013461235910654068,
+            -0.00782033707946539,
+            0.00209271302446723,
+            -0.02259376086294651,
+            -0.016832200810313225,
+            0.009313516318798065,
+            -0.027902841567993164,
+            0.016500383615493774,
+            -0.002175667556002736,
+            -0.002022956032305956,
+            -0.007903290912508965,
+            0.012465782463550568,
+            0.0031635789200663567,
+            -0.01495441421866417,
+            -0.0007597868097946048,
+            0.0009223962551914155,
+            -0.028883211314678192,
+            0.011621156707406044,
+            0.015459681861102581,
+            -0.006029276177287102,
+            -0.00030047399923205376,
+            -0.029018955305218697,
+            0.009494507685303688,
+            0.0001621381234144792,
+            -0.001130253542214632,
+            0.008793165907263756,
+            0.020255954936146736,
+            0.008227567188441753,
+            -0.012759894132614136,
+            0.012827765196561813,
+            0.013393363915383816,
+            0.005806807428598404,
+            0.00752245495095849,
+            0.024644995108246803,
+            -0.0047962721437215805,
+            0.011621156707406044,
+            -0.011651321314275265,
+            0.00961516797542572,
+            -0.0104145472869277,
+            0.0004543638206087053,
+            0.016605960205197334,
+            0.012307415716350079,
+            -0.005459907464683056,
+            0.005033823195844889,
+            -0.015474764630198479,
+            0.014909166842699051,
+            -0.020633019506931305,
+            0.0008964730077423155,
+            0.012737269513309002,
+            -0.0036066307220607996,
+            0.005599421449005604,
+            0.0135818962007761,
+            0.03830984607338905,
+            0.015851831063628197,
+            0.012480865232646465,
+            0.006093377247452736,
+            -0.02616834081709385,
+            0.003171120071783662,
+            0.0023114110808819532,
+            0.006119771860539913,
+            -0.0024471546057611704,
+            0.004539867863059044,
+            -0.010459794662892818,
+            0.00020526499429252,
+            0.010210931301116943,
+            -0.017118770629167557,
+            0.012865472584962845,
+            -0.010919814929366112,
+            0.007345234509557486,
+            -0.004445601254701614,
+            -0.010919814929366112,
+            -0.021221240982413292,
+            -0.005999110639095306,
+            0.018265048041939735,
+            -0.031824320554733276,
+            0.010716198943555355,
+            0.007039811462163925,
+            0.0288228802382946,
+            -0.02892845869064331,
+            -0.011824771761894226,
+            0.0033841622062027454,
+            -0.006108460016548634,
+            -0.009215478785336018,
+            0.00837085209786892,
+            0.01767682656645775,
+            -0.009652874432504177,
+            0.01534656248986721,
+            0.010821777395904064,
+            -0.006236662156879902,
+            0.03924496844410896,
+            0.007250967901200056,
+            0.016108235344290733,
+            -0.007895749993622303,
+            0.005546632222831249,
+            0.005490072537213564,
+            0.01576133444905281,
+            0.008061658591032028,
+            -0.02021070569753647,
+            0.0027016736567020416,
+            -0.0011566481553018093,
+            0.024539416655898094,
+            0.0052562919445335865,
+            0.005595650989562273,
+            0.0020663184113800526,
+            0.00731506897136569,
+            0.015987573191523552,
+            0.015271149575710297,
+            -0.007213261444121599,
+            -0.007130307145416737,
+            0.007624262943863869,
+            0.003165464149788022,
+            -0.003484084503725171,
+            0.001214150688610971,
+            -0.012518571689724922,
+            -0.004053453449159861,
+            0.017013192176818848,
+            -0.0067381588742136955,
+            -0.001416823361068964,
+            -0.012737269513309002,
+            -0.02452433481812477,
+            0.013453694060444832,
+            0.0018674165476113558,
+            -0.011651321314275265,
+            0.025549951940774918,
+            -0.007695905398577452,
+            -0.011123429983854294,
+            -0.01625906117260456,
+            -0.016334474086761475,
+            0.0139966681599617,
+            0.018280131742358208,
+            0.0010058219777420163,
+            0.0046831523068249226,
+            -0.003197514684870839,
+            -0.0006815457600168884,
+            0.022895412519574165,
+            -0.010776530019938946,
+            0.009728288277983665,
+            -0.002498058369383216
+          ],
+          "keyphrases": [
+            "routing method",
+            "sparsely activated mixture-of-experts",
+            "load imbalance",
+            "training efficiency improvements",
+            "GLUE and SuperGLUE benchmark"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "bd90d79e-2183-4ea8-ad44-6b8ece583bae",
+        "properties": {
+          "page_content": "6 Limitations\n\nThe expert choice method might not immediately apply to auto-regressive text generation as our current implementation takes in the past and future tokens to perform the top-k selection. One possible solution is to collect a large batch of input sequences, dispatch tokens of the same sequence into separate groups, and perform expert choice routing for each group. Another scenario where the expert choice method does not immediately apply is when the batch size becomes very small during serving or inference. A global top-k can be selected instead and we can cap the number of times each expert or token gets selected. We leave these possible improvements for future work.\n\nAnother long-standing issue with MoE has been the large memory footprint. Even though computational cost can be reduced using sparsely gated networks, the total number of parameters increases linearly or sub-linearly with the number of experts. Increasing the number of experts requires reservation of a large number of hardware devices. Therefore, dynamic (used) power is saved while static (reserved) power is not. Power saving techniques such as the ability to put hardware devices into low power states while not in use [17] can help with reducing the reserved power requirements.\n\nReferences\n\n[1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pages 3930–3939. Computer Vision Foundation / IEEE, 2020.\n\n[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. In Advances in Neural Information Processing Systems.\n\n[3] Kyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning, 2014.\n\n[4] Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems, 2021.\n\n[5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.\n\nTransformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.\n\n[6] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 933–941. JMLR.org, 2017.\n\n[7] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixtureof-experts, 2021.\n\n[8] Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan.\n\nTricks for training sparse translation models, 2021.\n\n[9] Richard L Dykstra. An iterative procedure for obtaining i-projections onto the intersection of convex sets. The annals of Probability, pages 975–984, 1985.\n\n[10] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.\n\n[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. NAS-FPN: learning scalable feature pyramid architecture for object detection. In CVPR, pages 7036–7045. Computer Vision Foundation / IEEE, 2019.\n\n[12] Sam Gross, Marc'Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision, 2017.\n\n[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.\n\n[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 630–645, Cham, 2016. Springer International Publishing.\n\n[15] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2016.\n\n[16] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.\n\n[17] Ping Huang, Zuocheng Xing, Tianran Wang, Qiang Wei, Hongyan Wang, and Guitao Fu. A brief survey on power gating design. In 2010 10th IEEE International Conference on Solid-State and Integrated Circuit Technology, pages 788–790, 2010.\n\n[18] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 103–112, 2019.\n\n[19] Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David A. Patterson. A domain-specific supercomputer for training deep neural networks. Commun. ACM, 63(7):67–78, 2020.\n\n[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.\n\n[21] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.\n\n[22] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6265–6274. PMLR, 18–24 Jul 2021.\n\n[24] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. New York, NY, USA, 2019. Association for Computing Machinery.\n\n[25] Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, Cédric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. In ICLR. OpenReview.net, 2021.\n\n[26] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.\n\n[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.\n\n[28] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models, 2021.\n\n[30] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 10435–10444, Red Hook, NY, USA, 2018. Curran Associates Inc.\n\n[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E.\n\nHinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In ICLR (Poster). OpenReview.net, 2017.\n\n[32] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 10–15 Jul 2018. [23] Min Lin, Jie Fu, and Yoshua Bengio. Conditional computation for continual learning, 2019.\n\n[29] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.\n\n[33] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.\n\n[34] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc.\n\n[35] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.\n\nGLUE: A multi-task benchmark and analysis platform for natural language understanding.\n\nIn Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, November 2018. Association for Computational Linguistics.\n\n[36] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake A. Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen. GSPMD: general and scalable parallelization for ML computation graphs. CoRR, abs/2105.04663, 2021.\n\n7 Checklist\n\n(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Yes (b) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes (c) Did you discuss any potential negative societal impacts of your work? N/A. Not any. (d) Did you describe the limitations of your work? Yes (a) Did you include the code, data, and instructions needed to reproduce the main experimental results? Yes. We include details in the experiment setup to help reproduce the main results. (b) Did you specify all the training details? Yes (c) Did you report error bars? Yes (d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? Yes (a) If your work uses existing assets, did you cite the creators? Yes (b) Did you mention the license of the assets? No. The used dataset is not released yet. (c) Did you include any new assets either in the supplemental material or as a URL? No. The dataset is not released yet.\n\n(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? No. Not using persons' data. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes. The dataset does not contain any personally identifiable information or offensive content.\n\n",
+          "title": "6 Limitations",
+          "embedding": [
+            -0.005417884327471256,
+            -0.002057158388197422,
+            -0.02763829380273819,
+            -0.01599692367017269,
+            0.03654416278004646,
+            -0.02751435525715351,
+            -0.03312699869275093,
+            0.03036494180560112,
+            0.005386899691075087,
+            0.039306219667196274,
+            0.025407399982213974,
+            -0.014332606457173824,
+            0.02131742797791958,
+            -0.005431163590401411,
+            -0.0021988023072481155,
+            0.018927184864878654,
+            -0.01384570449590683,
+            0.01158825308084488,
+            0.00927768461406231,
+            -0.02854127436876297,
+            -0.0014927954180166125,
+            -0.016377592459321022,
+            -0.013624385930597782,
+            0.010021315887570381,
+            0.019777048379182816,
+            0.013535858131945133,
+            0.02400866523385048,
+            0.020803967490792274,
+            -0.02866521291434765,
+            0.00207154406234622,
+            0.029214082285761833,
+            -0.0010053407168015838,
+            -0.03507460653781891,
+            -0.017953382804989815,
+            -0.0425640344619751,
+            -0.03293224051594734,
+            0.08179943263530731,
+            0.002959032543003559,
+            -0.00252082129009068,
+            0.022733869031071663,
+            0.005395752843469381,
+            0.00014842192467767745,
+            -0.006825472228229046,
+            -0.010499364696443081,
+            -0.03427785634994507,
+            0.003585364669561386,
+            -0.016802523285150528,
+            -0.0011071473127231002,
+            -0.025478221476078033,
+            -0.004550314974039793,
+            -0.017519596964120865,
+            -0.032755184918642044,
+            0.031161688268184662,
+            -0.0005953476065769792,
+            -0.03493295982480049,
+            0.01935211755335331,
+            -0.005988887045532465,
+            -0.025230344384908676,
+            0.07755011320114136,
+            0.0031958436593413353,
+            0.01846684142947197,
+            -0.0008542906143702567,
+            -0.007923213765025139,
+            0.007737305480986834,
+            0.025159522891044617,
+            -0.019104240462183952,
+            -0.02299945056438446,
+            0.021813182160258293,
+            -0.0003170392010360956,
+            -0.005537396762520075,
+            -0.03576511889696121,
+            0.019900986924767494,
+            -0.022840101271867752,
+            -0.016377592459321022,
+            0.003184777684509754,
+            -0.012606319040060043,
+            0.019989514723420143,
+            0.033870629966259,
+            0.016784818843007088,
+            0.008303881622850895,
+            0.008781930431723595,
+            0.04323684424161911,
+            0.00798075646162033,
+            0.0331801176071167,
+            0.03234795853495598,
+            0.006143810227513313,
+            -0.0033684722147881985,
+            -0.07485887408256531,
+            0.0303118247538805,
+            -0.0013168469304218888,
+            -0.031002338975667953,
+            -0.0057985526509583,
+            -0.0010662033455446362,
+            -0.036296285688877106,
+            -0.020910201594233513,
+            0.041926637291908264,
+            -0.0014042678521946073,
+            -0.0038487340789288282,
+            -0.024398185312747955,
+            -0.006055282894521952,
+            -0.02207876369357109,
+            6.712184404022992e-05,
+            0.009959346614778042,
+            -0.01853766292333603,
+            -0.016342181712388992,
+            -0.016891051083803177,
+            0.00800731498748064,
+            -0.007122039329260588,
+            0.039164576679468155,
+            0.08378244936466217,
+            -0.007617793511599302,
+            -0.013208306394517422,
+            -0.016165126115083694,
+            -0.03117939457297325,
+            -0.02871832810342312,
+            -0.03197614103555679,
+            -0.03842094540596008,
+            -0.03534018620848656,
+            0.011172173544764519,
+            -0.018148142844438553,
+            0.002722221426665783,
+            0.014332606457173824,
+            0.01792682334780693,
+            -0.014217520132660866,
+            -0.03310929238796234,
+            0.049292124807834625,
+            -0.018944889307022095,
+            -0.03254271671175957,
+            1.4307914852906833e-06,
+            0.008264044299721718,
+            -0.02122890017926693,
+            0.030081652104854584,
+            -0.04224533587694168,
+            -0.05980919674038887,
+            0.024309657514095306,
+            -0.011163320392370224,
+            0.013305687345564365,
+            0.05771994590759277,
+            -0.04791109636425972,
+            -0.008897015824913979,
+            -0.000875315919984132,
+            -0.010844621807336807,
+            0.012951577082276344,
+            0.01338536199182272,
+            -0.020927906036376953,
+            0.0021456859540194273,
+            -0.019918693229556084,
+            0.080949567258358,
+            -0.021512188017368317,
+            0.02303486131131649,
+            -0.03383522108197212,
+            0.02114037238061428,
+            -0.029957713559269905,
+            0.01624480076134205,
+            0.002892636926844716,
+            -0.011287258937954903,
+            -0.03027641400694847,
+            0.004176286049187183,
+            -0.02312338910996914,
+            0.009507856331765652,
+            0.005413458216935396,
+            -0.005324930418282747,
+            0.02753205969929695,
+            0.01435916405171156,
+            0.015386084094643593,
+            0.047379929572343826,
+            0.032755184918642044,
+            0.02939113788306713,
+            -0.05977378413081169,
+            0.05453295633196831,
+            0.016944168135523796,
+            0.03944786638021469,
+            -0.008379130624234676,
+            -0.000762996613048017,
+            -0.0042448947206139565,
+            0.013677502982318401,
+            0.030099358409643173,
+            0.03587135300040245,
+            -0.005559528712183237,
+            0.027018601074814796,
+            -0.023725377395749092,
+            0.018254375085234642,
+            0.02505328878760338,
+            0.005364768207073212,
+            0.01628021150827408,
+            0.012057448737323284,
+            0.03241878002882004,
+            0.017625829204916954,
+            -0.009906229563057423,
+            0.022769279778003693,
+            0.011260701343417168,
+            0.014058170840144157,
+            0.04904424771666527,
+            -0.01383685227483511,
+            0.01018066518008709,
+            -0.013624385930597782,
+            -0.002472131047397852,
+            0.01252664439380169,
+            -0.020573796704411507,
+            0.01570478267967701,
+            -0.018732424825429916,
+            -0.04143088310956955,
+            0.03289682790637016,
+            -0.015943806618452072,
+            0.008649138733744621,
+            0.03750025853514671,
+            0.0009483511094003916,
+            0.03781895712018013,
+            0.012075154110789299,
+            -0.03050658479332924,
+            -0.0330030620098114,
+            -0.0070423646830022335,
+            -0.022645341232419014,
+            -0.01435031183063984,
+            -0.0025562322698533535,
+            0.006546610500663519,
+            -0.004501624498516321,
+            0.013128631748259068,
+            -0.0165015310049057,
+            -0.01853766292333603,
+            -0.021848592907190323,
+            0.010021315887570381,
+            0.038350123912096024,
+            -0.00276205874979496,
+            -0.027992403134703636,
+            -0.011189878918230534,
+            -0.016758261248469353,
+            -0.009773438796401024,
+            0.0026248411741107702,
+            0.006887441501021385,
+            0.028983911499381065,
+            -0.01018066518008709,
+            -0.022379757836461067,
+            0.014961151406168938,
+            0.03390604257583618,
+            -0.005634776782244444,
+            -0.011703338474035263,
+            0.010410836897790432,
+            -0.024380480870604515,
+            -0.003375111846253276,
+            -0.005360341630876064,
+            0.01664317399263382,
+            -0.04606972262263298,
+            -0.009711469523608685,
+            -0.007940919138491154,
+            0.01660776324570179,
+            0.017068106681108475,
+            0.029302610084414482,
+            0.028045520186424255,
+            0.03785436972975731,
+            -0.0027664850931614637,
+            -0.0036738922353833914,
+            -0.0034083097707480192,
+            0.02021968737244606,
+            0.004603431094437838,
+            0.03339258208870888,
+            0.02314109541475773,
+            0.008029446005821228,
+            -0.02868291735649109,
+            -0.02500017359852791,
+            0.06420016288757324,
+            -0.012287620455026627,
+            -0.042811911553144455,
+            0.04486574977636337,
+            -0.02595626935362816,
+            0.03323323279619217,
+            0.03689827397465706,
+            0.011384639889001846,
+            0.0029568192549049854,
+            0.007724026683717966,
+            0.036437928676605225,
+            -0.011482019908726215,
+            -0.037146151065826416,
+            -0.0064270980656147,
+            0.017971087247133255,
+            -0.02781534753739834,
+            0.037181559950113297,
+            -0.0314449779689312,
+            0.03689827397465706,
+            0.020768556743860245,
+            0.009764585644006729,
+            -0.008724387735128403,
+            0.007768290117383003,
+            0.009463592432439327,
+            0.007197287864983082,
+            -0.041997458785772324,
+            -0.013704060576856136,
+            -0.06529790163040161,
+            -0.05779076740145683,
+            0.011030529625713825,
+            -0.020981023088097572,
+            0.002284010173752904,
+            -0.047415342181921005,
+            0.01294272392988205,
+            0.014385722577571869,
+            -0.008857178501784801,
+            0.0027930433861911297,
+            0.03305617719888687,
+            -0.01848454773426056,
+            -0.007538118865340948,
+            -0.05435590073466301,
+            -0.0238847266882658,
+            0.013190601021051407,
+            0.04320143163204193,
+            -0.04688417539000511,
+            -0.0128541961312294,
+            -0.01763468235731125,
+            0.011685633100569248,
+            -0.015439200215041637,
+            -0.03206466883420944,
+            0.01921047270298004,
+            0.0033817514777183533,
+            -0.013199454173445702,
+            -0.060198716819286346,
+            -0.010995118878781796,
+            0.0442991741001606,
+            0.009950493462383747,
+            0.004362193867564201,
+            -0.009835408069193363,
+            -0.033870629966259,
+            0.0515938401222229,
+            -0.015819868072867393,
+            -0.05612644925713539,
+            0.012022037990391254,
+            -0.0076045142486691475,
+            -0.023619143292307854,
+            -0.008795209228992462,
+            0.0637398213148117,
+            0.0008692296105436981,
+            -0.000270700576948002,
+            -0.01936982199549675,
+            -0.004446295090019703,
+            -0.013226011767983437,
+            -0.009127187542617321,
+            -0.0059800343587994576,
+            -0.052939459681510925,
+            -0.022769279778003693,
+            0.009082923643290997,
+            -0.010641008615493774,
+            -0.0031537930481135845,
+            -0.02754976600408554,
+            -0.018980301916599274,
+            -0.004264813382178545,
+            0.004647694993764162,
+            0.02662907913327217,
+            -0.042032867670059204,
+            -0.030081652104854584,
+            0.033870629966259,
+            0.01763468235731125,
+            -0.01628021150827408,
+            0.0017827230039983988,
+            0.018927184864878654,
+            -0.0012139336904510856,
+            0.03760649263858795,
+            0.015403789468109608,
+            -0.01288075465708971,
+            -0.03125021606683731,
+            -0.021813182160258293,
+            0.0017307130619883537,
+            0.01610315591096878,
+            0.024610651656985283,
+            -0.04578643664717674,
+            -0.02399095892906189,
+            0.009915082715451717,
+            -0.06324406713247299,
+            -0.028771445155143738,
+            -0.03849176689982414,
+            0.008064857684075832,
+            0.015952659770846367,
+            -0.013491595163941383,
+            -0.017776327207684517,
+            0.023371266201138496,
+            -0.009684910997748375,
+            0.002646972890943289,
+            -0.007082202006131411,
+            0.024132603779435158,
+            0.03321552649140358,
+            -0.014642452821135521,
+            -0.038350123912096024,
+            0.030028536915779114,
+            -0.00265582581050694,
+            0.011676780879497528,
+            0.046636298298835754,
+            0.014669010415673256,
+            -0.03689827397465706,
+            0.01922817900776863,
+            0.016492677852511406,
+            0.013562416657805443,
+            -0.01197777409106493,
+            0.007060070056468248,
+            -0.023548321798443794,
+            -0.010065579786896706,
+            -0.0017683373298496008,
+            -0.018289785832166672,
+            -0.014961151406168938,
+            0.03194073215126991,
+            0.006196926813572645,
+            0.002070437418296933,
+            0.0043444884940981865,
+            -0.015200176276266575,
+            -0.005333783570677042,
+            -0.008334866724908352,
+            0.016979578882455826,
+            0.01921047270298004,
+            0.01236729510128498,
+            0.013633239082992077,
+            0.01662546955049038,
+            0.027921581640839577,
+            0.03976656496524811,
+            -0.035446420311927795,
+            0.010340014472603798,
+            -0.010286898352205753,
+            -0.03333946689963341,
+            -0.0008493109489791095,
+            0.039979029446840286,
+            0.04132464900612831,
+            -0.02029050886631012,
+            0.008928000926971436,
+            0.008622581139206886,
+            -0.003306502941995859,
+            -0.004944262094795704,
+            0.02774452604353428,
+            0.044653281569480896,
+            0.036402519792318344,
+            -0.009463592432439327,
+            0.008786357007920742,
+            -0.014916887506842613,
+            0.008821767754852772,
+            0.02505328878760338,
+            -0.03436638414859772,
+            -0.0034503601491451263,
+            0.004926556721329689,
+            0.03806683421134949,
+            -0.02131742797791958,
+            -0.003974885679781437,
+            -0.04578643664717674,
+            -0.014846066012978554,
+            -0.015642814338207245,
+            0.011172173544764519,
+            0.007029085420072079,
+            -0.013110926374793053,
+            -0.007471723016351461,
+            0.00531607773154974,
+            -0.01565166562795639,
+            -0.03806683421134949,
+            -0.03208237513899803,
+            -0.04228074476122856,
+            -0.03500378131866455,
+            -0.021653832867741585,
+            -0.021441366523504257,
+            -0.05644514784216881,
+            0.03774813562631607,
+            -0.08640286326408386,
+            -0.021777771413326263,
+            0.02765599824488163,
+            -0.013898821547627449,
+            0.01338536199182272,
+            -0.03898752108216286,
+            0.0128541961312294,
+            -0.05998625233769417,
+            -0.013730619102716446,
+            0.05403720214962959,
+            0.014483102597296238,
+            0.03289682790637016,
+            0.009596383199095726,
+            0.02124660648405552,
+            -0.05499329790472984,
+            -0.027089422568678856,
+            -0.002164497971534729,
+            -0.010933149605989456,
+            0.008511921390891075,
+            -0.002127980347722769,
+            0.004437441937625408,
+            0.012995840981602669,
+            0.0012238930212333798,
+            0.010711830109357834,
+            0.009622941724956036,
+            -0.05092103034257889,
+            -0.00127700949087739,
+            -0.004291371908038855,
+            0.0258854478597641,
+            0.0031537930481135845,
+            -0.0010833556298166513,
+            -0.013721765950322151,
+            0.010021315887570381,
+            0.014784096740186214,
+            0.017351394519209862,
+            0.015518874861299992,
+            0.01065871398895979,
+            -0.04631759971380234,
+            -0.019794754683971405,
+            0.039022933691740036,
+            -0.005015084054321051,
+            0.007015806157141924,
+            -0.01853766292333603,
+            -0.005121317226439714,
+            -0.01759927161037922,
+            -0.003237894270569086,
+            0.01759927161037922,
+            -0.0014042678521946073,
+            -0.014908035285770893,
+            -0.021441366523504257,
+            -0.041041359305381775,
+            0.023707671090960503,
+            0.03392374888062477,
+            -0.0004260387213435024,
+            -0.023318149149417877,
+            0.007307947147637606,
+            -0.04231615737080574,
+            -0.010579039342701435,
+            0.004926556721329689,
+            0.020556090399622917,
+            -0.01921047270298004,
+            -0.007055643480271101,
+            0.013509300537407398,
+            0.009968198835849762,
+            -0.0295327827334404,
+            0.013217159546911716,
+            0.02475229650735855,
+            0.023743081837892532,
+            -0.026328084990382195,
+            0.002602709224447608,
+            -0.003523395396769047,
+            0.048265207558870316,
+            -0.012508939020335674,
+            -0.012535497546195984,
+            -0.0035676592960953712,
+            -0.0030453468207269907,
+            0.017811737954616547,
+            -0.03303847089409828,
+            -0.019175061956048012,
+            0.011605958454310894,
+            -0.012030890211462975,
+            -0.008560611866414547,
+            -0.005125743802636862,
+            -0.012889607809484005,
+            0.010162959806621075,
+            0.01756386086344719,
+            -0.02756747044622898,
+            -0.020857084542512894,
+            -0.0347382016479969,
+            0.04975247010588646,
+            -0.01288075465708971,
+            -0.0023415531031787395,
+            0.03426015377044678,
+            0.009490150958299637,
+            0.008109121583402157,
+            -0.018201259896159172,
+            0.03337487578392029,
+            0.02473459020256996,
+            0.014624747447669506,
+            -0.024327363818883896,
+            0.00028245814610272646,
+            0.025035584345459938,
+            0.0023282738402485847,
+            -0.017900265753269196,
+            -0.018643897026777267,
+            -0.003943901043385267,
+            0.007693042047321796,
+            -0.018998006358742714,
+            -0.010286898352205753,
+            0.004421949852257967,
+            -0.014093581587076187,
+            -0.00017318196478299797,
+            -0.014208667911589146,
+            -0.032719772309064865,
+            0.002005148446187377,
+            0.01525329239666462,
+            0.004298011306673288,
+            0.026345791295170784,
+            -0.009401623159646988,
+            -0.0028904236387461424,
+            0.009985905140638351,
+            -0.003054199507459998,
+            0.036367107182741165,
+            0.02951507642865181,
+            0.007427459117025137,
+            -0.009472444653511047,
+            -0.026328084990382195,
+            -0.014368017204105854,
+            0.005081479903310537,
+            0.008941279724240303,
+            0.001089995144866407,
+            0.02776223234832287,
+            0.026381202042102814,
+            0.014598188921809196,
+            -0.009738028049468994,
+            -0.003412736114114523,
+            0.000571002543438226,
+            -0.009286537766456604,
+            0.02671760693192482,
+            0.05623268336057663,
+            0.012269915081560612,
+            -0.032755184918642044,
+            0.009879671968519688,
+            -0.02678842842578888,
+            0.006055282894521952,
+            0.014270637184381485,
+            0.002030600095167756,
+            -0.056799259036779404,
+            0.02659366838634014,
+            0.014970004558563232,
+            0.014996563084423542,
+            -0.020857084542512894,
+            -0.05410802364349365,
+            0.000554680242203176,
+            0.006431524641811848,
+            0.0010390918469056487,
+            -0.007940919138491154,
+            -0.015456905588507652,
+            0.01718319207429886,
+            -0.0005671294638887048,
+            -0.007405327633023262,
+            0.007498281542211771,
+            0.02484082244336605,
+            -0.012588613666594028,
+            0.007825832813978195,
+            -0.01291616540402174,
+            -0.0009682697709649801,
+            0.005302798934280872,
+            0.030099358409643173,
+            0.0037115165032446384,
+            0.03493295982480049,
+            -0.003399456851184368,
+            0.016359886154532433,
+            0.028913090005517006,
+            -0.010587891563773155,
+            -0.006081840954720974,
+            0.0019055550219491124,
+            0.0035920043010264635,
+            -0.021441366523504257,
+            0.003401670139282942,
+            -0.025991681963205338,
+            -0.03406539186835289,
+            0.010189518332481384,
+            0.0030475601088255644,
+            0.009755733422935009,
+            0.015881838276982307,
+            -0.004410883877426386,
+            -0.0012902886373922229,
+            0.0005508071626536548,
+            0.015005415305495262,
+            0.0038730790838599205,
+            0.12500086426734924,
+            -0.01390767376869917,
+            0.024150308221578598,
+            0.005298372358083725,
+            0.018148142844438553,
+            0.03757108002901077,
+            0.01201318483799696,
+            0.017811737954616547,
+            0.007688615471124649,
+            0.003939474932849407,
+            0.04405129700899124,
+            -0.02103414013981819,
+            -0.01570478267967701,
+            0.016289064660668373,
+            -0.008786357007920742,
+            0.015332967042922974,
+            0.0014540646225214005,
+            -0.015766752883791924,
+            0.011667927727103233,
+            0.02029050886631012,
+            -0.010791505686938763,
+            0.02305256761610508,
+            -0.019086534157395363,
+            -0.031055456027388573,
+            0.01528870314359665,
+            -0.004957541357725859,
+            0.04858390614390373,
+            0.0051611545495688915,
+            -0.004966394044458866,
+            -0.02199023775756359,
+            -0.003468065755441785,
+            -0.02843504026532173,
+            -0.018891774117946625,
+            0.008560611866414547,
+            0.027089422568678856,
+            0.02758517675101757,
+            0.0017041548853740096,
+            -0.010056726634502411,
+            -0.007485002279281616,
+            0.025442810729146004,
+            0.0014927954180166125,
+            -0.00400365749374032,
+            0.006590874399989843,
+            -0.014474250376224518,
+            0.011659074574708939,
+            -0.014376870356500149,
+            0.003954967018216848,
+            0.0004039068298880011,
+            -0.01290731318295002,
+            0.0009544373606331646,
+            0.03569429740309715,
+            0.005404605530202389,
+            -0.01018066518008709,
+            -0.03151579946279526,
+            0.01284534391015768,
+            0.01419096253812313,
+            0.00324232061393559,
+            -0.0023039288353174925,
+            -0.0009594170260243118,
+            -0.011605958454310894,
+            0.029107850044965744,
+            -0.002613775199279189,
+            -0.004528183024376631,
+            0.0027200081385672092,
+            0.0006728091975674033,
+            -0.01940523274242878,
+            0.015828721225261688,
+            0.011260701343417168,
+            0.025389693677425385,
+            0.022362053394317627,
+            0.024097193032503128,
+            0.019830165430903435,
+            0.005900359712541103,
+            -0.005214271135628223,
+            -0.013252570293843746,
+            -0.02030821330845356,
+            0.043838828802108765,
+            0.00038813785067759454,
+            0.012863049283623695,
+            -0.000697707524523139,
+            -0.0684494823217392,
+            -0.033711280673742294,
+            0.019015712663531303,
+            0.02220270223915577,
+            0.024415891617536545,
+            0.0017849361756816506,
+            0.0028616522904485464,
+            0.022468285635113716,
+            -0.04118300601840019,
+            -0.007600088138133287,
+            0.005006231367588043,
+            -0.009074071422219276,
+            -0.004997378680855036,
+            -0.0025141816586256027,
+            0.023442087695002556,
+            -0.01660776324570179,
+            -0.012243356555700302,
+            0.034614261239767075,
+            -0.014943446032702923,
+            0.02662907913327217,
+            -0.00017829996068030596,
+            0.0008636966813355684,
+            0.012199092656373978,
+            0.013270275667309761,
+            0.03250730782747269,
+            -0.020963318645954132,
+            -0.0080781364813447,
+            0.003970459569245577,
+            -0.009622941724956036,
+            0.029037028551101685,
+            -0.009968198835849762,
+            -0.04592807963490486,
+            0.01666088029742241,
+            0.016837935894727707,
+            0.03693368285894394,
+            -0.005453295540064573,
+            0.010189518332481384,
+            -0.010782652534544468,
+            0.041749581694602966,
+            -0.005262961145490408,
+            -0.007569103501737118,
+            0.03410080075263977,
+            -0.009950493462383747,
+            0.0029014896135777235,
+            0.022344347089529037,
+            -0.011198732070624828,
+            -0.031692855060100555,
+            0.019564582034945488,
+            0.016041187569499016,
+            0.059419672936201096,
+            -0.018732424825429916,
+            0.016598910093307495,
+            0.019723933190107346,
+            -0.0020217474084347486,
+            -0.01651923544704914,
+            -0.03689827397465706,
+            0.008308308199048042,
+            -0.007077775429934263,
+            0.007489428389817476,
+            -0.015908395871520042,
+            0.03707532584667206,
+            -0.019759343937039375,
+            -0.013996201567351818,
+            0.008742093108594418,
+            -0.0036318416241556406,
+            -0.01338536199182272,
+            0.02664678543806076,
+            -0.0048822928220033646,
+            -0.03927081078290939,
+            -0.012818785384297371,
+            0.007223845925182104,
+            0.02939113788306713,
+            -0.02112266793847084,
+            0.01057018619030714,
+            0.022344347089529037,
+            -0.015518874861299992,
+            -0.02478770725429058,
+            0.01201318483799696,
+            -0.016784818843007088,
+            0.009481297805905342,
+            0.02480541169643402,
+            -0.03050658479332924,
+            -0.006852030288428068,
+            -0.028399629518389702,
+            -0.01374832447618246,
+            -0.012190240435302258,
+            0.019015712663531303,
+            0.0030187885276973248,
+            0.0060729882679879665,
+            -0.025177227333188057,
+            0.027408121153712273,
+            0.013996201567351818,
+            -0.013854557648301125,
+            -0.01834290288388729,
+            -0.010251487605273724,
+            -0.02496476098895073,
+            -0.0044662137515842915,
+            -0.02036133036017418,
+            -0.016528088599443436,
+            -7.096034096321091e-05,
+            -0.004227189347147942,
+            -0.018927184864878654,
+            0.005891506560146809,
+            -0.00015533814439550042,
+            0.004705238156020641,
+            0.06069447100162506,
+            0.007830259390175343,
+            0.02593856491148472,
+            -0.003412736114114523,
+            -0.020007221028208733,
+            0.03149809315800667,
+            -0.02018427476286888,
+            0.042882733047008514,
+            0.0008465444552712142,
+            -0.03032952919602394,
+            0.01010984368622303,
+            0.037004504352808,
+            -0.009056366048753262,
+            0.006121678277850151,
+            -0.01832519844174385,
+            0.0012006545439362526,
+            -0.013403067365288734,
+            -0.014031612314283848,
+            -0.0007010273402556777,
+            0.006639564409852028,
+            0.01563396118581295,
+            0.01521788164973259,
+            0.017811737954616547,
+            0.011358081363141537,
+            0.0008033872582018375,
+            0.02027280256152153,
+            -0.022609930485486984,
+            0.008888163603842258,
+            -0.009826554916799068,
+            0.021848592907190323,
+            0.006701533682644367,
+            0.0039372616447508335,
+            -0.011207584291696548,
+            0.004322356544435024,
+            -0.038314711302518845,
+            0.0028218147344887257,
+            0.01747533306479454,
+            0.022556813433766365,
+            -0.02595626935362816,
+            -0.04578643664717674,
+            -0.004820323549211025,
+            -0.05775535851716995,
+            -0.028293397277593613,
+            -0.010995118878781796,
+            -0.00369159784168005,
+            0.02209646999835968,
+            -0.002565084956586361,
+            0.024380480870604515,
+            -0.018148142844438553,
+            -0.014854918234050274,
+            0.01195121556520462,
+            -0.022379757836461067,
+            -0.008272897452116013,
+            0.016891051083803177,
+            0.017271719872951508,
+            -0.017971087247133255,
+            0.028045520186424255,
+            -0.024681473150849342,
+            -0.0035809383261948824,
+            -0.04398047551512718,
+            -0.01834290288388729,
+            -0.015934955328702927,
+            -0.0035411010030657053,
+            0.008604875765740871,
+            -0.011933510191738605,
+            -0.016722848638892174,
+            0.0011663500918075442,
+            0.009516708552837372,
+            -0.013633239082992077,
+            0.03346340358257294,
+            -0.01834290288388729,
+            -0.017360247671604156,
+            0.002666891785338521,
+            -0.012349589727818966,
+            -0.021777771413326263,
+            -0.02206105925142765,
+            0.01293387170881033,
+            -0.020626913756132126,
+            0.004922130145132542,
+            -0.017953382804989815,
+            0.013562416657805443,
+            -0.02671760693192482,
+            -0.01844913698732853,
+            -0.0029656721744686365,
+            -0.012925018556416035,
+            -0.05708254873752594,
+            0.019971810281276703,
+            0.05283322557806969,
+            -0.005479853600263596,
+            -0.014890329912304878,
+            0.033551931381225586,
+            -0.008600449189543724,
+            -0.024380480870604515,
+            0.02485852874815464,
+            -0.0019199406960979104,
+            0.023654554039239883,
+            -0.02505328878760338,
+            -0.0035676592960953712,
+            0.00015353992057498544,
+            0.019564582034945488,
+            -0.020927906036376953,
+            0.007405327633023262,
+            -0.013004693202674389,
+            0.022362053394317627,
+            -0.00020485822460614145,
+            0.006391687318682671,
+            0.046636298298835754,
+            -0.014031612314283848,
+            -0.00970261637121439,
+            0.04157252609729767,
+            -0.006059709005057812,
+            0.0032069096341729164,
+            -0.024433596059679985,
+            0.020502975210547447,
+            0.03619005158543587,
+            0.008914722129702568,
+            0.01560740265995264,
+            -0.03523395583033562,
+            0.028736034408211708,
+            0.004696385003626347,
+            0.028895383700728416,
+            0.029196377843618393,
+            -0.01944064348936081,
+            0.026115620508790016,
+            0.007799274753779173,
+            0.0128541961312294,
+            -0.0017174340318888426,
+            0.02018427476286888,
+            -0.018661601468920708,
+            -0.01673170179128647,
+            -0.01566937193274498,
+            0.0442991741001606,
+            -0.002087036380544305,
+            -0.010295751504600048,
+            -0.01721860282123089,
+            -0.02478770725429058,
+            0.0048778667114675045,
+            0.0029435402248054743,
+            -0.01470442209392786,
+            -0.01673170179128647,
+            -0.021972531452775,
+            0.015439200215041637,
+            0.010233782231807709,
+            -0.013527005910873413,
+            0.00711761275306344,
+            -0.002790830098092556,
+            -0.0038575867656618357,
+            -0.002963458886370063,
+            -0.01246467512100935,
+            -0.03760649263858795,
+            -0.022751573473215103,
+            -0.01515591237694025,
+            0.03333946689963341,
+            -0.010136401280760765,
+            0.020591503009200096,
+            -0.006081840954720974,
+            0.012553202919661999,
+            -0.0389166995882988,
+            0.00026378437178209424,
+            -0.015067384578287601,
+            -0.0304888803511858,
+            0.025194933637976646,
+            -0.010605597868561745,
+            0.008879310451447964,
+            0.032578129321336746,
+            -0.027124833315610886,
+            -0.024610651656985283,
+            0.01338536199182272,
+            -0.004353341180831194,
+            0.0017550581833347678,
+            -0.017625829204916954,
+            0.016944168135523796,
+            -0.01942293904721737,
+            -0.028045520186424255,
+            0.013057810254395008,
+            0.03192302584648132,
+            0.012606319040060043,
+            0.012199092656373978,
+            0.03693368285894394,
+            -0.006276601459830999,
+            -0.037181559950113297,
+            -0.006471361964941025,
+            -0.009499003179371357,
+            -0.03572971001267433,
+            -0.004537035711109638,
+            0.006435951218008995,
+            -0.003970459569245577,
+            -0.01669629104435444,
+            -0.02379619888961315,
+            -0.007940919138491154,
+            -0.012349589727818966,
+            -0.05240829288959503,
+            0.01516476459801197,
+            -0.019777048379182816,
+            -0.004286945331841707,
+            -0.03418932855129242,
+            0.05077938735485077,
+            0.02843504026532173,
+            0.007467296905815601,
+            0.006736944429576397,
+            -0.01293387170881033,
+            -0.0021423660218715668,
+            0.00507262721657753,
+            0.0032179756090044975,
+            0.01922817900776863,
+            -0.020910201594233513,
+            0.014987709932029247,
+            -0.003056412795558572,
+            0.005457721650600433,
+            -0.056693024933338165,
+            -0.020927906036376953,
+            0.010056726634502411,
+            0.005581660196185112,
+            -0.0003308716113679111,
+            0.0037911911495029926,
+            0.011783013120293617,
+            -0.01716548763215542,
+            -0.021689243614673615,
+            0.01610315591096878,
+            0.030063947662711143,
+            0.01522673387080431,
+            -0.006365128792822361,
+            0.019104240462183952,
+            0.019599994644522667,
+            -0.006865309551358223,
+            0.005993313621729612,
+            -0.01426178403198719,
+            0.009569825604557991,
+            0.010853474959731102,
+            0.0036583999171853065,
+            0.022822394967079163,
+            -0.00203502643853426,
+            -0.0013876688899472356,
+            -0.00814010575413704,
+            0.003932835068553686,
+            -0.005546249449253082,
+            -0.03025870770215988,
+            -0.014757538214325905,
+            -0.013633239082992077,
+            0.011242995969951153,
+            0.03426015377044678,
+            0.02126431092619896,
+            0.014722127467393875,
+            -0.012747963890433311,
+            -0.006807766854763031,
+            -0.001094974810257554,
+            0.006334144156426191,
+            0.003992591518908739,
+            0.01387226302176714,
+            -0.014474250376224518,
+            -0.008538479916751385,
+            0.002060478087514639,
+            0.003950540907680988,
+            -0.01197777409106493,
+            0.0385625883936882,
+            0.04635301232337952,
+            -0.02106955088675022,
+            -0.017953382804989815,
+            0.006577595137059689,
+            0.016811376437544823,
+            -0.005652482621371746,
+            -0.0037115165032446384,
+            -0.005873801186680794,
+            -0.00027249878621660173,
+            -0.01787370629608631,
+            0.015200176276266575,
+            0.01111020427197218,
+            -0.02767370454967022,
+            0.010835768654942513,
+            0.09653041511774063,
+            -0.02661137469112873,
+            0.009206862188875675,
+            -0.026345791295170784,
+            -0.011995479464530945,
+            -0.004554741084575653,
+            0.0048424554988741875,
+            0.02864750660955906,
+            0.02294633351266384,
+            0.0032179756090044975,
+            0.025389693677425385,
+            -0.0028328807093203068,
+            -0.010791505686938763,
+            0.013199454173445702,
+            -0.0006645097164437175,
+            0.013287981040775776,
+            -0.02395554818212986,
+            0.010286898352205753,
+            -0.011411197483539581,
+            -0.011358081363141537,
+            0.011437756009399891,
+            0.013757177628576756,
+            -0.0006883015157654881,
+            0.021671537309885025,
+            -0.004014723002910614,
+            0.02200794219970703,
+            0.0008731026900932193,
+            -4.9450923143012915e-06,
+            -0.04964623600244522,
+            -0.015085089951753616,
+            0.0024190146941691637,
+            -0.007932065986096859,
+            -0.02115807868540287,
+            0.001672063604928553,
+            -0.00224527926184237,
+            0.01566937193274498,
+            -0.01057018619030714,
+            0.01714778132736683,
+            0.0050682006403803825,
+            -0.010552480816841125,
+            -0.0037292218767106533,
+            0.009321948513388634,
+            -0.011783013120293617,
+            0.012473528273403645,
+            -0.009472444653511047,
+            0.004242681432515383,
+            0.023707671090960503,
+            -0.025336576625704765,
+            0.01191580481827259,
+            -0.009684910997748375,
+            0.000783468596637249,
+            0.016846787184476852,
+            -0.018714718520641327,
+            0.008317161351442337,
+            -0.0012515578418970108,
+            0.006254469510167837,
+            0.001487262430600822,
+            -0.016864493489265442,
+            0.012570908293128014,
+            -0.010853474959731102,
+            0.005417884327471256,
+            0.007245977874845266,
+            0.01778518036007881,
+            -0.014854918234050274,
+            0.010711830109357834,
+            -0.006064135581254959,
+            -0.002739926800131798,
+            -0.0035698723513633013,
+            -0.019086534157395363,
+            0.0028572259470820427,
+            0.00687416223809123,
+            -0.006099546328186989,
+            -0.0249293502420187,
+            0.002538526663556695,
+            -0.004267026670277119,
+            0.019635405391454697,
+            0.00019074915326200426,
+            -0.015279850922524929,
+            -0.007613366935402155,
+            -0.020981023088097572,
+            0.012792226858437061,
+            0.0021589649841189384,
+            0.014164404012262821,
+            0.011986627243459225,
+            -0.007781569380313158,
+            0.008188796229660511,
+            -0.024398185312747955,
+            0.006360702682286501,
+            0.02593856491148472,
+            -0.002722221426665783,
+            -0.0007989609148353338,
+            -0.004023576155304909,
+            -0.010340014472603798,
+            -0.00014330392878036946,
+            -0.007684188894927502,
+            -0.014421133324503899,
+            0.0031936303712427616,
+            0.00054444425040856,
+            -0.011251848191022873,
+            0.00483360281214118,
+            -0.012411559000611305,
+            0.015713635832071304,
+            0.01249123364686966,
+            -0.014075876213610172,
+            -0.020680028945207596,
+            0.00714859738945961,
+            -0.010003610514104366,
+            -0.0013223799178376794,
+            0.0026226278860121965,
+            0.0008360318024642766,
+            0.008742093108594418,
+            -0.026044797152280807,
+            0.003638481255620718,
+            -0.018838657066226006,
+            0.015501169487833977,
+            -0.011304965242743492,
+            0.014067023992538452,
+            -0.022698456421494484,
+            0.01689990423619747,
+            0.020520679652690887,
+            -0.019883282482624054,
+            0.0038664396852254868,
+            0.013580122031271458,
+            -0.021459072828292847,
+            0.0119423633441329,
+            0.015297556295990944,
+            0.00866684503853321,
+            -0.011384639889001846,
+            0.006307586096227169,
+            0.015315261669456959,
+            0.001169669907540083,
+            0.018608486279845238,
+            0.017121223732829094,
+            0.0027000894770026207,
+            0.01332339271903038,
+            -0.02661137469112873,
+            0.020963318645954132,
+            0.012597466818988323,
+            0.03948327526450157,
+            -0.00597560778260231,
+            -0.004873440135270357,
+            0.006157089490443468,
+            0.02204335294663906,
+            -0.0017041548853740096,
+            0.012243356555700302,
+            -0.009817702695727348,
+            -0.01810387894511223,
+            0.03142727166414261,
+            0.00804272573441267,
+            0.025124112144112587,
+            -0.010401983745396137,
+            -0.004286945331841707,
+            0.025088699534535408,
+            0.019139651209115982,
+            0.0030608391389250755,
+            -0.01010984368622303,
+            0.005670187994837761,
+            -0.0015215668827295303,
+            0.023388972505927086,
+            -0.013889968395233154,
+            0.011844982393085957,
+            0.029975419864058495,
+            0.024203425273299217,
+            -0.028116341680288315,
+            0.005807405803352594,
+            -0.012986987829208374,
+            -0.0049354094080626965,
+            -0.008649138733744621,
+            0.009375064633786678,
+            0.0030165754724293947,
+            -0.020733145996928215,
+            -0.004510477185249329,
+            -0.002430080436170101,
+            -0.0031095293816179037,
+            0.039235398173332214,
+            -0.017917970195412636,
+            0.03620775789022446,
+            0.01841372437775135,
+            0.016705144196748734,
+            -0.022468285635113716,
+            0.030719051137566566,
+            -0.016253653913736343,
+            -0.010800357908010483,
+            -0.0007917680195532739,
+            0.008339292369782925,
+            0.0066351378336548805,
+            -0.017289426177740097,
+            0.008423393592238426,
+            -0.018626190721988678,
+            0.02025509811937809,
+            -0.004899998661130667,
+            -0.00035604662843979895,
+            -0.02034362591803074,
+            0.0038332417607307434,
+            0.00898111704736948,
+            0.010410836897790432,
+            -0.007307947147637606,
+            -0.008162237703800201,
+            -0.009025380946695805,
+            -0.01733369007706642,
+            -0.017156634479761124,
+            0.011393492110073566,
+            -0.01201318483799696,
+            -0.01152628380805254,
+            0.02207876369357109,
+            -0.029762953519821167,
+            -0.022415168583393097,
+            0.018077321350574493,
+            -0.0025031156837940216,
+            -0.028948500752449036,
+            0.004541461821645498,
+            -0.019104240462183952,
+            0.004041281528770924,
+            -0.008950132876634598,
+            0.0008830620790831745,
+            0.00894570630043745,
+            -0.008733239956200123,
+            -0.03955409675836563,
+            -0.003014362184330821,
+            0.0037535671144723892,
+            -0.009136040695011616,
+            0.00711318664252758,
+            0.008011740632355213,
+            0.0008902549161575735,
+            0.013447331264615059,
+            -0.009968198835849762,
+            0.008202075026929379,
+            -0.03774813562631607,
+            0.02218499779701233,
+            0.009684910997748375,
+            -0.036437928676605225,
+            0.010295751504600048,
+            -0.00811354722827673,
+            0.010463953018188477,
+            0.02753205969929695,
+            0.00441752327606082,
+            0.020697735249996185,
+            -0.002020640764385462,
+            0.0011962282005697489,
+            0.011862688697874546,
+            -0.0033618328161537647,
+            0.014642452821135521,
+            -0.01619168370962143,
+            0.029338020831346512,
+            -0.0011873753974214196,
+            0.00014551711501553655,
+            0.032737478613853455,
+            -0.003027641447260976,
+            -0.02577921561896801,
+            0.009879671968519688,
+            -0.013766029849648476,
+            -0.030966928228735924,
+            0.030665934085845947,
+            -0.03119709901511669,
+            0.0013046744279563427,
+            0.00852077454328537,
+            -0.006546610500663519,
+            -0.0005032236222177744,
+            -0.02501787804067135,
+            -0.030117064714431763,
+            0.010039021261036396,
+            0.014571630395948887,
+            0.011304965242743492,
+            0.018183553591370583,
+            -0.027018601074814796,
+            0.013004693202674389,
+            -0.008073709905147552,
+            0.006303159520030022,
+            0.009587530978024006,
+            0.012119418010115623,
+            -0.019175061956048012,
+            0.025991681963205338,
+            -0.008432246744632721,
+            0.02397325448691845,
+            0.0037292218767106533,
+            -0.0030741184018552303,
+            -0.00026378437178209424,
+            -0.03240107372403145,
+            0.006241190247237682,
+            -0.006188074126839638,
+            0.003757993457838893,
+            0.0797455906867981,
+            0.0011641369201242924,
+            -0.0020881430245935917,
+            0.022733869031071663,
+            -0.009375064633786678,
+            -0.011030529625713825,
+            -0.015297556295990944,
+            -0.006347423419356346,
+            -0.013527005910873413,
+            -0.024628357961773872,
+            -0.006263322196900845,
+            -0.004260387271642685,
+            -0.005957902409136295,
+            0.01841372437775135,
+            0.0006362915737554431,
+            -0.028222573921084404,
+            0.0002011234755627811,
+            -0.010826916433870792,
+            -0.010844621807336807,
+            0.017953382804989815,
+            0.0037380747962743044,
+            0.007608940824866295,
+            -0.00400365749374032,
+            -0.01758156716823578,
+            -0.029338020831346512,
+            -0.026947777718305588,
+            -0.0002370877773500979,
+            -0.006980395410209894,
+            -0.0016432921402156353,
+            0.001767230685800314,
+            -0.023530615493655205,
+            -0.030028536915779114,
+            0.02115807868540287,
+            -0.015412641689181328,
+            0.005550675559788942,
+            0.011384639889001846,
+            -0.00044540411909110844,
+            1.79648632183671e-05,
+            0.020609207451343536,
+            -0.04808815196156502,
+            -0.015040826052427292,
+            -0.010809211060404778,
+            -0.008653565309941769,
+            -0.016775965690612793,
+            0.015377230942249298,
+            -0.010198370553553104,
+            -0.003839881392195821,
+            -0.014067023992538452,
+            -0.002136833034455776,
+            -0.0013688568724319339,
+            -0.030170179903507233,
+            -0.012668288312852383,
+            -0.015430347062647343,
+            0.031108571216464043,
+            0.008861605077981949,
+            -0.000989848398603499,
+            -0.0019409660017117858,
+            -0.0161562729626894,
+            0.017032695934176445,
+            -0.03788977861404419,
+            0.008923574350774288,
+            -0.008100268431007862,
+            0.028222573921084404,
+            0.02760288119316101,
+            -0.017935676500201225,
+            0.0038819320034235716,
+            -0.003523395396769047,
+            0.003890784690156579,
+            -0.0087730772793293,
+            -0.021848592907190323,
+            0.010782652534544468,
+            0.0102072237059474,
+            0.002379177138209343,
+            0.030612817034125328,
+            0.02590315416455269,
+            -0.001249344670213759,
+            0.014093581587076187,
+            -0.009313095360994339,
+            0.006958263460546732,
+            -0.02108725719153881,
+            0.020042631775140762,
+            -0.008688976988196373,
+            0.0028572259470820427,
+            -0.008476510643959045,
+            0.013978496193885803,
+            -0.014642452821135521,
+            -0.010933149605989456,
+            0.04061642661690712,
+            -0.018697014078497887,
+            -0.011729896999895573,
+            -0.013987348414957523,
+            0.00393504835665226,
+            0.00924227386713028,
+            0.008750945329666138,
+            0.030010830610990524,
+            -0.002717795083299279,
+            0.012314178980886936,
+            0.001447424991056323,
+            0.003954967018216848,
+            -0.017794031649827957,
+            -0.010100990533828735,
+            0.005147875752300024,
+            0.00666612247005105,
+            0.004992952570319176,
+            -0.000783468596637249,
+            -0.011304965242743492,
+            0.0025119686033576727,
+            0.00021924394241068512,
+            -0.018254375085234642,
+            0.003118382068350911,
+            -0.03284371271729469,
+            0.014952299185097218,
+            0.0020637980196624994,
+            0.01794452965259552,
+            0.009746880270540714,
+            -0.026434319093823433,
+            0.018201259896159172,
+            -0.001513820723630488,
+            0.00762664619833231,
+            0.027248771861195564,
+            -0.031126277521252632,
+            -0.012340736575424671,
+            0.014908035285770893,
+            0.022432874888181686,
+            0.013491595163941383,
+            0.008246338926255703,
+            -0.027018601074814796,
+            0.026345791295170784,
+            -0.015554285608232021,
+            0.00805157795548439,
+            -0.02204335294663906,
+            -0.007706320844590664,
+            -0.032631244510412216,
+            -0.00017110710905399173,
+            -0.0017550581833347678,
+            0.013465036638081074,
+            -0.0023836034815758467,
+            -0.02126431092619896,
+            0.0034105228260159492,
+            0.007878949865698814,
+            -0.014368017204105854,
+            -0.008883737027645111,
+            0.0017616976983845234,
+            -0.00552411749958992,
+            0.0007784889312461019,
+            0.008879310451447964,
+            -0.0035477406345307827,
+            -0.03034723550081253,
+            -0.010756094008684158,
+            0.0027023027651011944,
+            0.008897015824913979,
+            -0.0033640458714216948,
+            0.014270637184381485,
+            -0.011827277019619942,
+            0.0018646109383553267,
+            -0.003215762320905924,
+            0.0109862657263875,
+            0.0019863364286720753,
+            -0.008852752856910229,
+            0.0013998415088281035,
+            -0.00798075646162033,
+            -0.0276205874979496,
+            0.015023120678961277,
+            -0.00504606869071722,
+            -0.006493493914604187,
+            -0.0002827347780112177,
+            0.0015337393851950765,
+            -0.005152301862835884,
+            -0.013925379142165184,
+            0.011774160899221897,
+            0.0018491186201572418,
+            0.0014507448067888618,
+            -0.0036318416241556406,
+            -0.004138661548495293,
+            -0.002044985769316554,
+            0.01751074381172657,
+            0.02774452604353428,
+            -0.018626190721988678,
+            0.010490511544048786,
+            -0.01767009310424328,
+            0.00048026180593296885,
+            -0.001060117152519524,
+            0.01763468235731125,
+            0.011375786736607552,
+            0.02114037238061428,
+            0.01516476459801197,
+            -0.008719961158931255,
+            0.010384278371930122,
+            -0.008166664279997349,
+            0.015872985124588013,
+            -0.009348507039248943,
+            -0.0007602301193401217,
+            -0.0009821022395044565,
+            0.02209646999835968,
+            -0.018980301916599274,
+            -0.0017284998903051019,
+            0.009499003179371357,
+            0.022450579330325127,
+            -0.016908757388591766,
+            -0.011242995969951153,
+            0.031108571216464043,
+            0.003414949169382453,
+            0.006975968834012747,
+            0.017236309126019478,
+            0.03229483962059021,
+            0.006307586096227169,
+            0.012252209708094597,
+            0.007843539118766785,
+            0.015872985124588013,
+            0.011340375989675522,
+            -0.010800357908010483,
+            -0.029143260791897774,
+            0.018767835572361946,
+            -0.0001229010958923027,
+            0.00265803886577487,
+            -0.00462113693356514,
+            0.008958985097706318,
+            0.012677141465246677,
+            -0.011685633100569248,
+            0.009206862188875675,
+            -0.013040103949606419,
+            0.017678946256637573,
+            0.01843143068253994,
+            0.0008891483303159475,
+            -0.02121119573712349,
+            0.007431885693222284,
+            0.02103414013981819,
+            0.0035123294219374657,
+            -0.004178499337285757,
+            -0.003775698831304908,
+            -0.01669629104435444,
+            -0.0017550581833347678,
+            0.0016034548170864582,
+            0.03218860924243927,
+            -0.00483360281214118,
+            -0.008383556269109249,
+            0.004625563044101,
+            0.00016543580568395555,
+            0.007883376441895962,
+            0.015412641689181328,
+            0.004811470862478018,
+            0.005701172631233931,
+            -0.012376148253679276,
+            0.011667927727103233,
+            0.024451302364468575,
+            0.011154468171298504,
+            0.0030608391389250755,
+            -0.0005566168110817671,
+            -0.000995934708043933,
+            0.016528088599443436,
+            -0.015014268457889557,
+            -0.006582021247595549,
+            -0.007538118865340948,
+            0.019033417105674744,
+            -0.0024499993305653334,
+            -0.006679401732981205,
+            0.014270637184381485,
+            -0.0009505642810836434,
+            0.004501624498516321,
+            -0.010375426150858402,
+            0.01335880346596241,
+            0.008852752856910229,
+            0.014137845486402512,
+            -0.010242634452879429,
+            -0.018024204298853874,
+            0.005386899691075087,
+            -0.0026602521538734436,
+            -0.021724654361605644,
+            0.007737305480986834,
+            -0.010800357908010483,
+            0.008233060128986835,
+            -0.0007657631067559123,
+            0.00255401898175478,
+            0.0014551711501553655,
+            0.0009533307747915387,
+            -0.0058472431264817715,
+            0.007276962511241436,
+            -0.013465036638081074,
+            -0.0003488537622615695,
+            -0.007347784470766783,
+            -0.028753740713000298,
+            0.0026735311839729548,
+            -5.22865702805575e-05,
+            0.005807405803352594,
+            0.009383917786180973,
+            0.00576314190402627,
+            0.003853160422295332,
+            0.014120140112936497,
+            -0.027868464589118958,
+            -0.0041231694631278515,
+            -0.00735221104696393,
+            0.02032591961324215,
+            0.012544349767267704,
+            -0.01053477544337511,
+            -0.007847964763641357,
+            0.0048778667114675045,
+            0.011783013120293617,
+            -0.0097291748970747,
+            0.008365850895643234,
+            0.02777993679046631,
+            -0.009844260290265083,
+            -0.02131742797791958,
+            0.0007956410991027951,
+            0.0042448947206139565,
+            0.01851995848119259,
+            0.025141816586256027,
+            0.01249123364686966,
+            -0.0028793576639145613,
+            -0.009897377341985703,
+            -0.013305687345564365,
+            0.029160967096686363,
+            -0.006887441501021385,
+            -0.01245582289993763,
+            -0.011871540918946266,
+            0.010676419362425804,
+            -0.026930073276162148,
+            0.0032046963460743427,
+            -0.005621497984975576,
+            -0.0028793576639145613,
+            -0.004366619978100061,
+            -0.007631072774529457,
+            -0.006891867611557245,
+            -0.018785540014505386,
+            -0.005431163590401411,
+            0.012473528273403645,
+            0.006834324914962053,
+            0.006236764136701822,
+            -0.028311101719737053,
+            -0.009587530978024006,
+            -0.015270997770130634,
+            -0.017103517428040504,
+            -0.00213904632255435,
+            -0.031055456027388573,
+            -0.03293224051594734,
+            -0.017678946256637573,
+            -0.014784096740186214,
+            -6.719100929331034e-05,
+            -0.004089971538633108,
+            -0.01467786356806755,
+            0.0022884365171194077,
+            0.009684910997748375,
+            0.01335880346596241,
+            0.0109862657263875,
+            0.0015591910341754556,
+            -0.022326640784740448,
+            0.012553202919661999,
+            -0.017555007711052895,
+            -0.023672260344028473,
+            0.004603431094437838,
+            0.011313817463815212,
+            -0.0026204148307442665,
+            -0.0018491186201572418,
+            0.004492771811783314,
+            0.006050856318324804,
+            0.005223123822361231,
+            -0.044830337166786194,
+            -0.009525561705231667,
+            0.0070423646830022335,
+            0.0040302155539393425,
+            -0.0049708206206560135,
+            -0.013403067365288734,
+            0.012066301889717579,
+            -0.01659005880355835,
+            -0.027886170893907547,
+            -0.009906229563057423,
+            -0.016510384157299995,
+            0.002213188214227557,
+            -0.009321948513388634,
+            -0.0011386852711439133,
+            0.03220631182193756,
+            0.017448775470256805,
+            0.005417884327471256,
+            0.0003460872976575047,
+            0.003917342983186245,
+            0.008715534582734108,
+            -0.010481659322977066,
+            0.0014363591326400638,
+            0.008299455046653748,
+            0.01290731318295002,
+            0.005878227762877941,
+            0.0020394527819007635,
+            -0.013341098092496395,
+            0.025584453716874123,
+            0.01467786356806755,
+            0.01010984368622303,
+            0.0006910680094733834,
+            0.008109121583402157,
+            0.006732518319040537,
+            0.023760788142681122,
+            0.0040589869022369385,
+            0.00846765749156475,
+            -0.007936492562294006,
+            0.0068166195414960384,
+            -0.013562416657805443,
+            -0.021441366523504257,
+            0.015359525568783283,
+            -0.0060729882679879665,
+            -0.03436638414859772,
+            0.02032591961324215,
+            -0.007077775429934263,
+            0.002996656810864806,
+            -0.008866031654179096,
+            -0.011375786736607552,
+            -0.00276205874979496,
+            0.017032695934176445,
+            0.0061659421771764755,
+            -0.006086267530918121,
+            0.0008327120449393988,
+            0.0073212264105677605,
+            -0.0027155817952007055,
+            -0.018838657066226006,
+            -0.0022076552268117666,
+            -0.004240468610078096,
+            0.004306863993406296,
+            -0.053683090955019,
+            -0.016811376437544823,
+            0.012482380494475365,
+            -0.009003248997032642,
+            -0.01747533306479454,
+            0.009401623159646988,
+            -0.008848326280713081,
+            -0.018767835572361946,
+            0.004687532316893339,
+            0.008812914602458477,
+            0.002602709224447608,
+            0.0020660110749304295,
+            0.007502707652747631,
+            0.021405955776572227,
+            0.01153513602912426,
+            -0.016722848638892174,
+            0.0021600716281682253,
+            -0.012677141465246677,
+            -0.000435168098192662,
+            -0.019830165430903435,
+            -0.0169353149831295,
+            -0.00480261817574501,
+            0.010437395423650742,
+            0.04479492828249931,
+            0.013491595163941383,
+            0.010685272514820099,
+            -0.0007209460018202662,
+            0.0005815151962451637,
+            0.001700835069641471,
+            -0.04139547049999237,
+            -0.0019719505216926336,
+            -0.005165581125766039,
+            -0.005692319944500923,
+            -0.010835768654942513,
+            0.012951577082276344,
+            0.028169458732008934,
+            0.02868291735649109,
+            -0.002775337779894471,
+            0.0073831956833601,
+            -2.913454591180198e-05,
+            0.00390185066498816,
+            0.008056004531681538,
+            -0.013075515627861023,
+            0.015085089951753616,
+            -0.0016012416454032063,
+            0.009791144169867039,
+            0.009782291017472744,
+            -0.0008526307065039873,
+            0.014491955749690533,
+            -0.015846427530050278,
+            0.010764947161078453,
+            -0.00964064709842205,
+            0.004149727523326874,
+            0.008998822420835495,
+            0.03764190524816513,
+            0.002179990289732814,
+            -0.01659005880355835,
+            -0.01566937193274498,
+            -0.030063947662711143,
+            0.029692132025957108,
+            -0.013270275667309761,
+            0.016997285187244415,
+            0.017245162278413773,
+            -0.0015647240215912461,
+            0.004930983297526836,
+            0.010463953018188477,
+            0.0017318197060376406,
+            0.011375786736607552,
+            0.014509661123156548,
+            0.00969376415014267,
+            -0.023300444707274437,
+            -0.02013115957379341,
+            0.001953138504177332,
+            -0.015262145549058914,
+            -0.014341458678245544,
+            -0.007679762784391642,
+            -0.011189878918230534,
+            0.0072326986119151115,
+            -0.014854918234050274,
+            0.021724654361605644,
+            -0.016829082742333412,
+            0.0049752467311918736,
+            -0.02944425493478775,
+            -0.006621859036386013,
+            0.0034459338057786226,
+            -0.012066301889717579,
+            -0.00028384136385284364,
+            -0.008308308199048042,
+            -0.002790830098092556,
+            0.017740916460752487,
+            -0.004415310453623533,
+            -0.003992591518908739,
+            0.011136762797832489,
+            0.006701533682644367,
+            0.010100990533828735,
+            -0.015642814338207245,
+            0.005550675559788942,
+            -0.0037646330893039703,
+            0.014005054719746113,
+            -0.018679307773709297,
+            -0.004740648902952671,
+            0.0027664850931614637,
+            0.024327363818883896,
+            0.012889607809484005,
+            0.008941279724240303,
+            0.005895933136343956,
+            -0.007852391339838505,
+            -0.009136040695011616,
+            -0.03739402815699577,
+            0.030860694125294685,
+            -0.016147419810295105,
+            0.022822394967079163,
+            -0.003609709907323122,
+            0.0025119686033576727,
+            -0.0006523372139781713,
+            0.00687416223809123,
+            0.0042448947206139565,
+            -0.021494483575224876,
+            0.013102073222398758,
+            -0.0231942106038332,
+            0.008184369653463364,
+            0.003359619528055191,
+            -0.012588613666594028,
+            0.03693368285894394,
+            -0.002405735431239009,
+            0.014784096740186214,
+            -0.02855897881090641,
+            0.0024079487193375826,
+            0.009764585644006729,
+            -0.005590513348579407,
+            -0.005453295540064573,
+            -0.02942654862999916,
+            -0.0010999544756487012,
+            0.0009915082482621074,
+            0.00035881312214769423,
+            0.006944984197616577,
+            -0.01922817900776863,
+            -0.03128562867641449,
+            0.008184369653463364,
+            0.008688976988196373,
+            0.002562871901318431,
+            0.003731435164809227,
+            -0.0001748418580973521,
+            -0.0018535450799390674,
+            0.014987709932029247,
+            0.01242926437407732,
+            0.019706226885318756,
+            0.03284371271729469,
+            -0.02862980216741562,
+            -0.015713635832071304,
+            -0.002653612522408366,
+            -0.016368739306926727,
+            -0.02569068782031536,
+            -0.02222040854394436,
+            0.0044706398621201515,
+            0.0073212264105677605,
+            0.013562416657805443,
+            -0.030896106734871864,
+            0.013287981040775776,
+            -0.005621497984975576,
+            0.004984099417924881,
+            -0.03032952919602394,
+            -0.026345791295170784,
+            -0.006254469510167837,
+            -0.011765307746827602,
+            0.03208237513899803,
+            -0.010021315887570381,
+            0.004275879357010126,
+            -0.01522673387080431,
+            -0.013456183485686779,
+            -0.00711761275306344,
+            0.0019420725293457508,
+            -0.01335880346596241,
+            -0.012296473607420921,
+            0.038243889808654785,
+            -0.001566937193274498,
+            -0.014598188921809196,
+            0.019723933190107346,
+            0.005546249449253082,
+            0.005501985549926758,
+            0.0040943981148302555,
+            0.013252570293843746,
+            -0.008441099897027016,
+            -0.006404966581612825,
+            -0.0004611730691976845,
+            0.007299094460904598,
+            0.002870504977181554,
+            0.024699179455637932,
+            -0.0042515345849096775,
+            2.7509237042977475e-05,
+            -0.007728452794253826,
+            0.0005483173299580812,
+            0.01844913698732853,
+            0.005422310903668404,
+            -0.007051217369735241,
+            -0.0025717245880514383,
+            0.005298372358083725,
+            -0.03227713704109192,
+            -0.017749767750501633,
+            0.01713007688522339,
+            0.010065579786896706,
+            0.009950493462383747,
+            -0.00028065990773029625,
+            -0.03149809315800667,
+            -0.007901081815361977,
+            0.002615988254547119,
+            0.0026624652091413736,
+            -0.0015160338953137398,
+            0.018820952624082565,
+            -0.027301888912916183,
+            0.002101422054693103,
+            -0.0032511733006685972,
+            0.02108725719153881,
+            0.008445525541901588,
+            0.0005848349537700415,
+            0.024504419416189194,
+            0.0036561868619173765,
+            0.013704060576856136,
+            -0.0018258801428601146,
+            -0.010322309099137783,
+            -0.01747533306479454,
+            -0.010003610514104366,
+            -3.72092254110612e-05,
+            0.0010119802318513393,
+            0.001832519774325192,
+            -0.00021232773724477738,
+            0.0015082877362146974,
+            -0.0038885714020580053,
+            -0.0011630303924903274,
+            0.015970366075634956,
+            0.0014817294431850314,
+            -0.004446295090019703,
+            0.014111286960542202,
+            -0.002376964082941413,
+            -0.007631072774529457,
+            0.006838751491159201,
+            0.0003211889124941081,
+            0.003240107325837016,
+            0.009224568493664265,
+            0.011844982393085957,
+            -0.0016045613447204232,
+            0.020715441554784775,
+            -0.010472806170582771,
+            0.008480937220156193,
+            -0.008888163603842258,
+            -0.0004888379480689764,
+            -0.017519596964120865,
+            0.0034060964826494455,
+            -0.002910342300310731,
+            -0.01423522550612688,
+            -0.020485268905758858,
+            0.006931704934686422,
+            0.006577595137059689,
+            0.00115639076102525,
+            -0.003638481255620718,
+            0.014872624538838863,
+            -0.0038332417607307434,
+            0.0045591676607728004,
+            -0.004886719398200512,
+            -0.006958263460546732,
+            -0.016439560800790787,
+            0.01518247090280056,
+            0.0015492317033931613,
+            0.005307225044816732,
+            0.002216507913544774,
+            0.0034038834273815155,
+            -0.00872881431132555,
+            0.001222786377184093,
+            0.017254015430808067,
+            0.005462148226797581,
+            -0.009994757361710072,
+            0.004408670589327812,
+            -0.009224568493664265,
+            -0.004098824225366116,
+            0.0075115603394806385,
+            -0.009339653886854649,
+            -0.01057018619030714,
+            0.02678842842578888,
+            -0.005355915054678917,
+            -0.0005236956640146673,
+            -0.012659436091780663,
+            -0.004220549948513508,
+            0.0115971053019166,
+            -0.000843777961563319,
+            -0.009251126088202,
+            -0.004705238156020641,
+            0.008958985097706318,
+            -0.01152628380805254,
+            -0.003518969053402543,
+            0.014580483548343182,
+            -0.009853113442659378,
+            -0.009560972452163696,
+            0.02021968737244606,
+            0.0026890235021710396,
+            -0.011650222353637218,
+            -0.006834324914962053,
+            0.0004075032484252006,
+            -0.005581660196185112,
+            0.03217090293765068,
+            0.006772355642169714,
+            0.00970261637121439,
+            0.023601438850164413,
+            0.001833626301959157,
+            0.009543267078697681,
+            -0.0042161233723163605,
+            0.012615172192454338,
+            0.0036716791801154613,
+            0.00873766653239727,
+            0.002321634441614151,
+            0.021724654361605644,
+            -0.0025009026285260916,
+            0.0042161233723163605,
+            0.007772716693580151,
+            -0.011482019908726215,
+            -0.004132022149860859,
+            -0.004036854952573776,
+            -0.005453295540064573,
+            -0.00089689448941499,
+            0.0005737690371461213,
+            0.010880032554268837,
+            0.009950493462383747,
+            -0.0015060745645314455,
+            0.015864131972193718,
+            0.010428542271256447,
+            -0.01422637328505516,
+            -0.0011630303924903274,
+            0.0050682006403803825,
+            0.00449055852368474,
+            0.0009666099213063717,
+            -0.030595112591981888,
+            -0.01291616540402174,
+            0.003156006336212158,
+            -0.015563138760626316,
+            0.015864131972193718,
+            -0.019847871735692024,
+            -0.02379619888961315,
+            -0.019670816138386726,
+            0.0038841450586915016,
+            -0.015005415305495262,
+            -0.01374832447618246,
+            0.021795475855469704,
+            0.007839112542569637,
+            0.0015768965240567923,
+            -0.02122890017926693,
+            0.023442087695002556,
+            -0.009808849543333054,
+            0.023229623213410378,
+            -0.01200433261692524,
+            -0.002098102355375886,
+            -0.011765307746827602,
+            0.01561625488102436,
+            -0.010897737927734852,
+            0.0025163949467241764,
+            0.009516708552837372,
+            0.01659005880355835,
+            0.0030652654822915792,
+            0.0007928746636025608,
+            0.0017882559914141893,
+            0.00480261817574501,
+            0.014438839629292488,
+            0.006834324914962053,
+            0.009410476312041283,
+            -0.01249123364686966,
+            -0.010242634452879429,
+            0.004078905563801527,
+            0.005931344348937273,
+            -0.00015644473023712635,
+            -0.02207876369357109,
+            0.002936900593340397,
+            0.009897377341985703,
+            -0.0005375280743464828,
+            -0.01839601993560791,
+            -0.006210205610841513,
+            -0.009534413926303387,
+            0.027921581640839577,
+            0.0097291748970747,
+            -0.01855536922812462,
+            -0.008950132876634598,
+            0.021565305069088936,
+            0.012411559000611305,
+            0.015235587023198605,
+            -0.016165126115083694,
+            0.013668649829924107,
+            0.008264044299721718,
+            0.0013511513825505972,
+            0.0029546061996370554,
+            -0.01053477544337511,
+            0.028824562206864357,
+            0.0048822928220033646,
+            0.0009881885489448905,
+            -0.013491595163941383,
+            -0.007538118865340948,
+            -0.008264044299721718,
+            0.011455461382865906,
+            0.005957902409136295,
+            0.002804109361022711,
+            0.001394308521412313,
+            -0.007985183037817478,
+            -0.015368377789855003,
+            -0.0019985088147222996,
+            -0.003357406472787261,
+            0.0211934894323349,
+            -0.0032179756090044975,
+            0.003782338462769985,
+            -0.006152662914246321,
+            0.0018004284938797355,
+            -0.009968198835849762,
+            -0.0057188780046999454,
+            -0.0020803967490792274,
+            0.00804272573441267,
+            -0.018218964338302612,
+            0.015979217365384102,
+            -0.020857084542512894,
+            0.018006497994065285,
+            0.028275690972805023,
+            0.00324232061393559,
+            -0.006785634905099869,
+            -0.002070437418296933,
+            0.008472084067761898,
+            -0.034649673849344254,
+            0.003306502941995859,
+            0.017696652561426163,
+            -0.006909573450684547,
+            -0.013615533709526062,
+            -0.018856363371014595,
+            -0.0012172535061836243,
+            -0.007927639409899712,
+            0.005563954822719097,
+            0.005856095813214779,
+            -0.0010518176713958383,
+            0.018661601468920708,
+            -0.002057158388197422,
+            -0.01467786356806755,
+            -0.008131253533065319,
+            -0.0221495870500803,
+            0.01480180211365223,
+            -0.012252209708094597,
+            -0.013465036638081074,
+            0.012818785384297371,
+            0.009746880270540714,
+            -0.019936397671699524,
+            0.002547379583120346,
+            -0.02747894451022148,
+            -0.008480937220156193,
+            -0.005369194317609072,
+            -0.01014525443315506,
+            -0.011685633100569248,
+            -0.004138661548495293,
+            -0.011765307746827602,
+            0.003943901043385267,
+            0.0002846713177859783,
+            -0.011986627243459225,
+            -0.004607857670634985,
+            0.007153023965656757,
+            -0.012287620455026627,
+            -0.009295389987528324,
+            0.009746880270540714,
+            0.004368833266198635,
+            -0.019121944904327393,
+            -0.031657442450523376,
+            0.008657991886138916,
+            -0.03337487578392029,
+            0.004306863993406296,
+            0.015492316335439682,
+            0.004939835984259844,
+            -0.007276962511241436,
+            0.003184777684509754,
+            0.021972531452775,
+            0.014854918234050274,
+            -0.006719239056110382,
+            0.017103517428040504,
+            0.0036296285688877106,
+            0.006617432460188866,
+            -0.012606319040060043,
+            -1.0452126844029408e-05,
+            0.01843143068253994,
+            0.012225651182234287,
+            -0.0028749313205480576,
+            0.01335880346596241,
+            -0.006883014924824238,
+            0.008609301410615444,
+            -0.00020900795061606914,
+            -0.007670910097658634,
+            -0.018661601468920708,
+            -0.006342997308820486,
+            -0.015474610961973667,
+            -0.006095120217651129,
+            0.008427820168435574,
+            -0.005993313621729612,
+            -0.006312012672424316,
+            -0.010605597868561745,
+            0.010286898352205753,
+            -0.017484186217188835,
+            0.00045508681796491146,
+            -0.02572609856724739,
+            -0.00300329620949924,
+            -0.013659796677529812,
+            0.009392770007252693,
+            0.01156169455498457,
+            0.005687893368303776,
+            -0.015749046579003334,
+            -0.019599994644522667,
+            0.0007768290233798325,
+            -0.0029789512045681477,
+            -0.021529894322156906,
+            -0.007684188894927502,
+            -0.018502252176404,
+            -0.012376148253679276,
+            -0.009437033906579018,
+            0.014075876213610172,
+            0.003014362184330821,
+            -0.03256042301654816,
+            0.006205779500305653,
+            -0.027018601074814796,
+            -0.016846787184476852,
+            -0.01201318483799696,
+            -0.0021047419868409634,
+            0.0046565476804971695,
+            0.020113453269004822,
+            0.008516347967088223,
+            0.026575962081551552,
+            -3.7174642784520984e-05,
+            0.005205418448895216,
+            0.0012891821097582579,
+            0.011933510191738605,
+            0.009817702695727348,
+            -0.012323031201958656,
+            0.01620939001441002,
+            -0.008356998674571514,
+            0.0020128944888710976,
+            -0.022716162726283073,
+            -0.0012936084531247616,
+            -0.014438839629292488,
+            -0.006281028036028147,
+            -0.000598667364101857,
+            -0.01834290288388729,
+            -0.006294306833297014,
+            0.023743081837892532,
+            -0.019051123410463333,
+            0.008998822420835495,
+            0.008361424319446087,
+            -0.0003070798411499709,
+            0.01467786356806755,
+            0.029816070571541786,
+            -0.010915443301200867,
+            0.001500541577115655,
+            0.01944064348936081,
+            0.004315716680139303,
+            0.008503069169819355,
+            -0.0008183263125829399,
+            0.0036207756493240595,
+            0.012084007263183594,
+            -0.01155284233391285,
+            0.00648906733840704,
+            -0.005807405803352594,
+            0.008423393592238426,
+            -0.011269553564488888,
+            0.0041585806757211685,
+            -0.01949376054108143,
+            -0.01601462997496128,
+            0.019918693229556084,
+            0.012898460030555725,
+            -0.0003311482723802328,
+            -0.0051036118529737,
+            -0.03233025223016739,
+            -0.0008686763467267156,
+            -0.009817702695727348,
+            0.007431885693222284,
+            -0.013571269810199738,
+            0.011304965242743492,
+            -0.00642267195507884,
+            0.014146698638796806,
+            -0.013677502982318401,
+            0.018856363371014595,
+            -0.011269553564488888,
+            -0.01062330324202776,
+            0.017121223732829094,
+            0.009755733422935009,
+            -0.0037557801697403193,
+            0.029231788590550423,
+            -0.001606774516403675,
+            0.0029501798562705517,
+            -0.002301715547218919,
+            -0.030878400430083275,
+            0.0023747507948428392,
+            0.0028882105834782124,
+            0.00918030459433794,
+            0.004149727523326874,
+            -0.005820684600621462,
+            0.030701344832777977,
+            0.012154828757047653,
+            0.016997285187244415,
+            -0.011437756009399891,
+            -0.0008044939022511244,
+            0.020998729392886162,
+            0.013615533709526062,
+            0.0015481250593438745,
+            0.007400901056826115,
+            -0.0008199862204492092,
+            0.0007674229564145207,
+            -0.021777771413326263,
+            0.010286898352205753,
+            -0.018732424825429916,
+            -0.010428542271256447,
+            -0.002987803891301155,
+            -0.003027641447260976,
+            -0.003027641447260976,
+            0.0010352187091484666,
+            0.00528509309515357,
+            0.0015591910341754556,
+            0.023654554039239883,
+            -0.0009311988833360374,
+            -0.0037491407711058855,
+            -0.028134047985076904,
+            -0.017864855006337166,
+            0.003386177821084857,
+            0.011322670616209507,
+            0.029090143740177155,
+            0.0024920497089624405,
+            -0.009198009967803955,
+            -0.008330440148711205,
+            -0.012615172192454338,
+            0.01782059110701084,
+            0.010773799382150173,
+            -0.0025186080019921064,
+            0.0017229669028893113,
+            -0.011136762797832489,
+            0.014660158194601536,
+            -0.014607041142880917,
+            0.011127909645438194,
+            0.014120140112936497,
+            0.0019420725293457508,
+            -0.01249123364686966,
+            -0.0005477640661410987,
+            0.003862013341858983,
+            0.012145976535975933,
+            0.002469917992129922,
+            0.009171451441943645,
+            -0.02592085860669613,
+            -0.00666169635951519,
+            0.005679040681570768,
+            0.006648417096585035,
+            0.02296403981745243,
+            0.0152444401755929,
+            0.011225289665162563,
+            0.01108364574611187,
+            0.012349589727818966,
+            0.008600449189543724,
+            0.015005415305495262,
+            -0.013881116174161434,
+            0.0025938565377146006,
+            -0.0019044483778998256,
+            0.016492677852511406,
+            0.007812554016709328,
+            0.010729536414146423,
+            -0.02029050886631012,
+            0.0032069096341729164,
+            0.013429625891149044,
+            -0.008817341178655624,
+            0.008551758714020252,
+            -0.008410114794969559,
+            0.01378373522311449,
+            -0.005448868963867426,
+            0.0009323054691776633,
+            -0.0012393853394314647,
+            0.0030873974319547415,
+            0.032525014132261276,
+            -0.004218336660414934,
+            -0.004811470862478018,
+            0.01570478267967701,
+            -0.0045591676607728004,
+            0.0028240280225872993,
+            -0.01525329239666462,
+            0.00879078358411789,
+            0.02480541169643402,
+            0.00883504655212164,
+            -0.016253653913736343,
+            -0.014483102597296238,
+            0.004961967933923006,
+            -0.009321948513388634,
+            0.0011198731372132897,
+            0.007772716693580151,
+            -0.004424163140356541,
+            0.009321948513388634,
+            -0.0013799227308481932,
+            -0.0008244125638157129,
+            0.0049708206206560135,
+            -0.039164576679468155,
+            0.00252303434535861,
+            -0.015350672416388988,
+            0.006103972904384136,
+            0.004452934488654137,
+            -0.0024787706788629293,
+            -0.006462509278208017,
+            -0.006555463187396526,
+            -0.001566937193274498,
+            -0.05403720214962959,
+            0.0032201886642724276,
+            -0.0014573843218386173,
+            0.02393784187734127,
+            -0.01062330324202776,
+            0.00969376415014267,
+            0.015014268457889557,
+            0.013093221001327038,
+            -0.003078544745221734,
+            -0.0012659436324611306,
+            0.013411919586360455,
+            -0.0211934894323349,
+            -0.018626190721988678,
+            0.014837212860584259,
+            -0.007608940824866295,
+            0.00011750645353458822,
+            -0.004050134215503931,
+            0.004798191599547863,
+            -0.004634415730834007,
+            0.0033397008664906025,
+            0.013270275667309761,
+            -0.012730257585644722,
+            -0.01924588344991207,
+            -0.007515986915677786,
+            -0.006338570732623339,
+            -0.019511466845870018,
+            -0.016528088599443436,
+            0.00621463218703866,
+            -0.01714778132736683,
+            0.003678318578749895,
+            -0.0005380813381634653,
+            -0.00555510213598609,
+            0.013633239082992077,
+            0.001355577725917101,
+            0.0020538384560495615,
+            0.0007336718845181167,
+            0.009233420714735985,
+            -0.017431069165468216,
+            0.017652388662099838,
+            0.011216437444090843,
+            -0.014483102597296238,
+            0.00177940318826586,
+            -0.004669826943427324,
+            0.0026646784972399473,
+            0.021494483575224876,
+            -0.0065421839244663715,
+            0.0021224473603069782,
+            0.018767835572361946,
+            -0.0013876688899472356,
+            0.011198732070624828,
+            -0.003757993457838893,
+            0.005298372358083725,
+            -0.012101712636649609,
+            0.0012504513142630458,
+            0.00648906733840704,
+            -0.007812554016709328,
+            0.009065218269824982,
+            -0.04504280537366867,
+            0.012216798029839993,
+            -0.009144892916083336,
+            -0.024451302364468575,
+            -0.006059709005057812,
+            -0.0063164387829601765,
+            -0.0006456976407207549,
+            -0.01653694175183773,
+            0.013137484900653362,
+            -0.011216437444090843,
+            0.006745797581970692,
+            -0.012075154110789299,
+            0.0019885494839400053,
+            0.007839112542569637,
+            0.0003709856537170708,
+            0.002027280395850539,
+            -0.006068561691790819,
+            0.003687171498313546,
+            0.004926556721329689,
+            -0.002126873703673482,
+            -0.008989970199763775,
+            0.013544711284339428,
+            -0.0034171624574810266,
+            0.015952659770846367,
+            0.010756094008684158,
+            -0.008454378694295883,
+            -0.024681473150849342,
+            0.0157224889844656,
+            -0.004678679630160332,
+            -0.008941279724240303,
+            -0.0021556452848017216,
+            0.010065579786896706,
+            -3.074256528634578e-05,
+            -0.008392409421503544,
+            0.0027532060630619526,
+            0.009879671968519688,
+            0.007259257137775421,
+            -0.008967838250100613,
+            -0.004687532316893339,
+            -0.015456905588507652,
+            0.009525561705231667,
+            0.002750992774963379,
+            0.007462870329618454,
+            0.013898821547627449,
+            0.004132022149860859,
+            0.013376508839428425,
+            0.0004337848804425448,
+            -0.00759123545140028,
+            -0.026983190327882767,
+            -0.015766752883791924,
+            -0.010950854979455471,
+            0.0014153338270261884,
+            -0.0011785227106884122,
+            0.02018427476286888,
+            -0.0051611545495688915,
+            0.0046919588930904865,
+            0.0074540176428854465,
+            0.042032867670059204,
+            0.014182109385728836,
+            0.007887802086770535,
+            0.0007380982278846204,
+            -0.013819146901369095,
+            0.00038481809315271676,
+            -0.010463953018188477,
+            -0.03153350576758385,
+            0.009897377341985703,
+            -0.004129808861762285,
+            0.02315879985690117,
+            -0.014819507487118244,
+            0.014589335769414902,
+            -0.004510477185249329,
+            -0.01808617264032364,
+            -0.016944168135523796,
+            0.020626913756132126,
+            0.022840101271867752,
+            0.016377592459321022,
+            4.270761564839631e-05,
+            0.019511466845870018,
+            -0.007055643480271101,
+            0.0161562729626894,
+            -0.020113453269004822,
+            -0.004576873034238815,
+            0.006555463187396526,
+            0.02121119573712349,
+            0.0013467249227687716,
+            0.012765669263899326,
+            0.00711318664252758,
+            -0.0012825424782931805,
+            -0.001180735882371664,
+            -0.012703699991106987,
+            0.0019011285621672869,
+            -0.005174433812499046,
+            -0.0020471990574151278,
+            0.013252570293843746,
+            -0.01009213738143444,
+            -0.01837831363081932,
+            0.01149087306112051,
+            -0.010003610514104366,
+            0.0032622392755001783,
+            0.011234142817556858,
+            0.022344347089529037,
+            0.005515264812856913,
+            0.010880032554268837,
+            -0.006551036611199379,
+            8.638350118417293e-05,
+            -0.022397464141249657,
+            -0.01004787441343069,
+            -0.004138661548495293,
+            -0.01114561501890421,
+            0.003333061235025525,
+            -0.007586808875203133,
+            -0.025495927780866623,
+            0.00041497277561575174,
+            0.007458443753421307,
+            0.030099358409643173,
+            -0.013597827404737473,
+            0.02112266793847084,
+            0.0011148934718221426,
+            -0.0029501798562705517,
+            -0.010649860836565495,
+            -0.01425293181091547,
+            0.00576756801456213,
+            -0.01195121556520462,
+            -0.0014429986476898193,
+            6.57731870887801e-05,
+            -0.01830749213695526,
+            -3.8903697713976726e-05,
+            0.014005054719746113,
+            -0.037323202937841415,
+            -0.003957180306315422,
+            0.004992952570319176,
+            -0.0009284323896281421,
+            0.01387226302176714,
+            0.005643629468977451,
+            0.0036296285688877106,
+            0.0027554191183298826,
+            -0.003492410760372877,
+            0.00022380864538718015,
+            0.01929900050163269,
+            -0.015784457325935364,
+            0.02843504026532173,
+            0.00831273477524519,
+            -0.004891145508736372,
+            -0.0008930214098654687,
+            0.004503837786614895,
+            -0.028753740713000298,
+            0.010897737927734852,
+            -0.015952659770846367,
+            -0.018643897026777267,
+            0.011313817463815212,
+            0.015138207003474236,
+            -0.0025562322698533535,
+            0.002536313608288765,
+            0.015899542719125748,
+            0.011384639889001846,
+            0.005231976509094238,
+            -0.0016753834206610918,
+            -0.014146698638796806,
+            0.00203502643853426,
+            -0.01936982199549675,
+            0.009074071422219276,
+            -0.020024925470352173,
+            -0.0036318416241556406,
+            0.00849421601742506,
+            0.0038022571243345737,
+            0.015811016783118248,
+            0.009268831461668015,
+            0.018112732097506523,
+            -0.015359525568783283,
+            0.00891914777457714,
+            0.03973115235567093,
+            0.01147316675633192,
+            0.013677502982318401,
+            -0.005856095813214779,
+            -0.0005881547695025802,
+            0.025460515171289444,
+            -0.005262961145490408,
+            -0.008414541371166706,
+            0.020502975210547447,
+            0.018909478560090065,
+            0.014819507487118244,
+            -0.001221679849550128,
+            0.009065218269824982,
+            0.029214082285761833,
+            0.00552411749958992,
+            -0.005192139185965061,
+            -0.019511466845870018,
+            -0.006373981945216656,
+            -0.0020903560798615217,
+            0.013164042495191097,
+            0.007166303228586912,
+            0.016846787184476852,
+            -0.028966205194592476,
+            0.0034702790435403585,
+            0.003501263679936528,
+            -0.0001507734414190054,
+            0.013376508839428425,
+            0.02021968737244606,
+            0.003344127209857106,
+            -0.003842094447463751,
+            0.006493493914604187,
+            0.005174433812499046,
+            0.03125021606683731,
+            -0.025283461436629295,
+            -0.0037270088214427233,
+            0.0034038834273815155,
+            -0.015279850922524929,
+            0.013606680557131767,
+            -0.004523756448179483,
+            -0.002558445557951927,
+            -0.007852391339838505,
+            -0.010490511544048786,
+            0.016713997349143028,
+            0.01419096253812313,
+            -0.015456905588507652,
+            -0.025177227333188057,
+            -0.02117578312754631,
+            0.007122039329260588,
+            -0.02108725719153881,
+            -0.0072858151979744434,
+            -0.01009213738143444,
+            -0.006387260742485523,
+            -0.009578677825629711,
+            -0.006475788541138172,
+            -0.009746880270540714,
+            0.005563954822719097,
+            0.0007120932568795979,
+            0.003638481255620718,
+            0.0007353317341767251,
+            -0.01377488300204277,
+            -0.011136762797832489,
+            0.021671537309885025,
+            0.004860160872340202,
+            0.01296928245574236,
+            0.0026071355678141117,
+            0.005231976509094238,
+            0.006250043399631977,
+            -0.04298896715044975,
+            -0.006050856318324804,
+            -0.006679401732981205,
+            0.009357359260320663,
+            0.005563954822719097,
+            0.022716162726283073,
+            -0.012110565789043903,
+            0.015766752883791924,
+            -0.010437395423650742,
+            0.010039021261036396,
+            -0.010242634452879429,
+            0.0026646784972399473,
+            0.008215353824198246,
+            -0.007467296905815601,
+            0.0016565712867304683,
+            -0.004607857670634985,
+            0.007679762784391642,
+            -0.0018391592893749475,
+            0.007608940824866295,
+            -0.00960523635149002,
+            -0.015483464114367962,
+            -0.0023526190780103207,
+            -0.011605958454310894,
+            -0.017864855006337166,
+            5.754288940806873e-05,
+            -0.008587169460952282,
+            -0.0015315262135118246,
+            -0.019688520580530167,
+            -0.01924588344991207,
+            0.011960068717598915,
+            -0.0020615847315639257,
+            0.009091776795685291,
+            0.013916526921093464,
+            0.007763864006847143,
+            0.015306408517062664,
+            0.013376508839428425,
+            -0.004452934488654137,
+            -0.013571269810199738,
+            -0.014208667911589146,
+            0.0023526190780103207,
+            -0.014943446032702923,
+            0.01935211755335331,
+            0.008210928179323673,
+            -0.0017307130619883537,
+            0.0011884820414707065,
+            -0.020998729392886162,
+            -0.0001062330265995115,
+            -0.0022209342569112778,
+            0.0015569778624922037,
+            -0.018590779975056648,
+            0.02200794219970703,
+            -0.018289785832166672,
+            0.002113594673573971,
+            0.019688520580530167,
+            0.029780659824609756,
+            0.0028461599722504616,
+            -0.012925018556416035,
+            0.0007309053908102214,
+            0.008427820168435574,
+            0.03898752108216286,
+            -0.007985183037817478,
+            0.001860184594988823,
+            -0.011853835545480251,
+            -0.010313456878066063,
+            0.00435555400326848,
+            0.006365128792822361,
+            0.006812192965298891,
+            -0.026522846892476082,
+            -0.013597827404737473,
+            -0.010375426150858402,
+            -0.013695208355784416,
+            -0.0058118319138884544,
+            0.04072266072034836,
+            0.001315740286372602,
+            -0.0010070005664601922,
+            -0.00669268099591136,
+            0.00023556621454190463,
+            -0.017280573025345802,
+            0.03047117404639721,
+            0.09419328719377518,
+            -0.008817341178655624,
+            -0.009552120231091976,
+            -0.025035584345459938,
+            0.009198009967803955,
+            -0.001912194536998868,
+            0.011906951665878296,
+            0.06069447100162506,
+            -0.020662324503064156,
+            0.000730352068785578,
+            -0.0007264789892360568,
+            0.009897377341985703,
+            -0.00963179487735033,
+            -0.0026248411741107702,
+            -0.00735663715749979,
+            -0.008888163603842258,
+            0.0017605911707505584,
+            -0.03140956535935402,
+            0.0005306118400767446,
+            -0.006272174883633852,
+            0.005506412126123905,
+            -0.014244078658521175,
+            0.004581299610435963,
+            -0.010818063281476498,
+            0.004220549948513508,
+            -0.013677502982318401,
+            -0.005448868963867426,
+            0.01297813467681408,
+            -0.019688520580530167,
+            -0.0007790421950630844,
+            0.004647694993764162,
+            -0.012641730718314648,
+            0.008392409421503544,
+            0.0026248411741107702,
+            -0.011278406716883183,
+            -0.0015846426831558347,
+            -0.015085089951753616,
+            0.028789151459932327,
+            0.0018834230722859502,
+            0.007259257137775421,
+            -0.009950493462383747,
+            0.00101751321926713,
+            0.01293387170881033,
+            -0.0021711376029998064,
+            0.017121223732829094,
+            0.013571269810199738,
+            -0.001765017514117062,
+            -0.008808488957583904,
+            0.01714778132736683,
+            -0.00046089643728919327,
+            -0.013314539566636086,
+            0.009136040695011616,
+            0.011216437444090843,
+            0.0027266477700322866,
+            0.0037535671144723892,
+            -0.0015724701806902885,
+            0.023548321798443794,
+            -0.017254015430808067,
+            -0.011482019908726215,
+            -0.004072266165167093,
+            -0.009330800734460354,
+            0.021441366523504257,
+            0.006307586096227169,
+            0.03397686406970024,
+            -0.009773438796401024,
+            -0.0012305325362831354,
+            -0.018590779975056648,
+            -0.007259257137775421,
+            -0.010189518332481384,
+            -0.005993313621729612,
+            0.013553564436733723,
+            0.0019299000268802047,
+            0.012694846838712692,
+            -0.019971810281276703,
+            -0.017156634479761124,
+            0.00960523635149002,
+            -0.02294633351266384,
+            -0.0011375787435099483,
+            0.004922130145132542,
+            -0.027213361114263535,
+            -0.004435229115188122,
+            0.008166664279997349,
+            -0.02121119573712349,
+            -0.025230344384908676,
+            0.002651399467140436,
+            -0.0007049004198051989,
+            -0.02025509811937809,
+            0.009711469523608685,
+            0.006015445105731487,
+            -0.008565038442611694,
+            -0.002060478087514639,
+            -0.002961245598271489,
+            -0.020024925470352173,
+            0.0076399254612624645,
+            -0.015058532357215881,
+            -0.008038299158215523,
+            -0.006174794863909483,
+            0.004988525994122028,
+            -0.0012703699758276343,
+            0.02114037238061428,
+            0.006338570732623339,
+            -0.004032428842037916,
+            -0.011623663827776909,
+            0.0018280934309586883,
+            0.008472084067761898,
+            0.017138928174972534,
+            0.02285780757665634,
+            -0.0019442857010290027,
+            -0.006750223692506552,
+            0.001236065523698926,
+            0.015403789468109608,
+            0.005147875752300024,
+            -0.010791505686938763,
+            0.025354282930493355,
+            0.006785634905099869,
+            -0.008033872582018375,
+            0.015483464114367962,
+            -0.012668288312852383,
+            0.001964204479008913,
+            -0.009977051988244057,
+            -0.008219780400395393,
+            0.0042780926451087,
+            -0.0007159663364291191,
+            0.024238836020231247,
+            0.005581660196185112,
+            0.014571630395948887,
+            0.010419690050184727,
+            0.012774521484971046,
+            0.022432874888181686,
+            -0.01201318483799696,
+            0.0019874428398907185,
+            -0.0067280917428433895,
+            0.0018535450799390674,
+            -0.0005510838236659765,
+            -0.007684188894927502,
+            -0.006741371005773544,
+            0.0050283633172512054,
+            0.009445887058973312,
+            -0.0017561647109687328,
+            0.010348867624998093,
+            0.019192766398191452,
+            0.014828360639512539,
+            0.005391326267272234,
+            -0.0064093926921486855,
+            -0.019954103976488113,
+            0.003248960245400667,
+            0.021016433835029602,
+            -0.0034724920988082886,
+            0.010410836897790432,
+            0.0014175469987094402,
+            0.01855536922812462,
+            0.00669268099591136,
+            0.0050637745298445225,
+            -0.00666169635951519,
+            0.008742093108594418,
+            -0.01646612025797367,
+            0.004581299610435963,
+            0.0034437207505106926,
+            0.02206105925142765,
+            -0.007493854966014624,
+            -0.0067236656323075294,
+            0.0002651675895322114,
+            0.031728263944387436,
+            0.01384570449590683,
+            0.010782652534544468,
+            -0.019564582034945488,
+            0.009392770007252693,
+            -0.006705959793180227,
+            0.0015547646908089519,
+            0.0007773823454044759,
+            0.0115971053019166,
+            -0.007436312269419432,
+            -0.005621497984975576,
+            0.011756455525755882,
+            0.009233420714735985,
+            0.00918030459433794,
+            -0.0030453468207269907,
+            -0.017555007711052895,
+            0.006170368287712336,
+            -0.0047849128022789955,
+            -0.012137123383581638,
+            0.026522846892476082,
+            0.0157224889844656,
+            0.032879121601581573,
+            -0.003069691825658083,
+            0.0019619911909103394,
+            0.011402345262467861,
+            0.0018015351379290223,
+            0.014908035285770893,
+            -0.01011869590729475,
+            -0.0109862657263875,
+            0.03392374888062477,
+            -0.01246467512100935,
+            0.008135679177939892,
+            -0.024150308221578598,
+            -0.026469729840755463,
+            -0.0028129620477557182,
+            0.001937646185979247,
+            0.0049708206206560135,
+            0.0027952566742897034,
+            -0.004528183024376631,
+            -0.007338931784033775,
+            0.01296928245574236,
+            -0.001329019432887435,
+            0.0037668461445719004,
+            0.008976691402494907,
+            -0.0044418685138225555,
+            0.0031139557249844074,
+            -0.002733287401497364,
+            -0.000756910361815244,
+            0.010959707200527191
+          ],
+          "keyphrases": [
+            "expert choice method",
+            "auto-regressive text generation",
+            "large memory footprint",
+            "sparsely gated networks",
+            "power saving techniques"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "964ffde3-7f2f-4be2-a576-f37b83a5cdc7",
+      "type": "next",
+      "source": {
+        "id": "bd90d79e-2183-4ea8-ad44-6b8ece583bae",
+        "properties": {
+          "page_content": "6 Limitations\n\nThe expert choice method might not immediately apply to auto-regressive text generation as our current implementation takes in the past and future tokens to perform the top-k selection. One possible solution is to collect a large batch of input sequences, dispatch tokens of the same sequence into separate groups, and perform expert choice routing for each group. Another scenario where the expert choice method does not immediately apply is when the batch size becomes very small during serving or inference. A global top-k can be selected instead and we can cap the number of times each expert or token gets selected. We leave these possible improvements for future work.\n\nAnother long-standing issue with MoE has been the large memory footprint. Even though computational cost can be reduced using sparsely gated networks, the total number of parameters increases linearly or sub-linearly with the number of experts. Increasing the number of experts requires reservation of a large number of hardware devices. Therefore, dynamic (used) power is saved while static (reserved) power is not. Power saving techniques such as the ability to put hardware devices into low power states while not in use [17] can help with reducing the reserved power requirements.\n\nReferences\n\n[1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pages 3930–3939. Computer Vision Foundation / IEEE, 2020.\n\n[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. In Advances in Neural Information Processing Systems.\n\n[3] Kyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning, 2014.\n\n[4] Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems, 2021.\n\n[5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.\n\nTransformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.\n\n[6] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 933–941. JMLR.org, 2017.\n\n[7] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixtureof-experts, 2021.\n\n[8] Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan.\n\nTricks for training sparse translation models, 2021.\n\n[9] Richard L Dykstra. An iterative procedure for obtaining i-projections onto the intersection of convex sets. The annals of Probability, pages 975–984, 1985.\n\n[10] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.\n\n[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. NAS-FPN: learning scalable feature pyramid architecture for object detection. In CVPR, pages 7036–7045. Computer Vision Foundation / IEEE, 2019.\n\n[12] Sam Gross, Marc'Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision, 2017.\n\n[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.\n\n[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 630–645, Cham, 2016. Springer International Publishing.\n\n[15] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2016.\n\n[16] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.\n\n[17] Ping Huang, Zuocheng Xing, Tianran Wang, Qiang Wei, Hongyan Wang, and Guitao Fu. A brief survey on power gating design. In 2010 10th IEEE International Conference on Solid-State and Integrated Circuit Technology, pages 788–790, 2010.\n\n[18] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 103–112, 2019.\n\n[19] Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David A. Patterson. A domain-specific supercomputer for training deep neural networks. Commun. ACM, 63(7):67–78, 2020.\n\n[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.\n\n[21] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.\n\n[22] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6265–6274. PMLR, 18–24 Jul 2021.\n\n[24] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. New York, NY, USA, 2019. Association for Computing Machinery.\n\n[25] Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, Cédric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. In ICLR. OpenReview.net, 2021.\n\n[26] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.\n\n[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.\n\n[28] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models, 2021.\n\n[30] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 10435–10444, Red Hook, NY, USA, 2018. Curran Associates Inc.\n\n[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E.\n\nHinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In ICLR (Poster). OpenReview.net, 2017.\n\n[32] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 10–15 Jul 2018. [23] Min Lin, Jie Fu, and Yoshua Bengio. Conditional computation for continual learning, 2019.\n\n[29] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.\n\n[33] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.\n\n[34] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc.\n\n[35] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.\n\nGLUE: A multi-task benchmark and analysis platform for natural language understanding.\n\nIn Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, November 2018. Association for Computational Linguistics.\n\n[36] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake A. Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen. GSPMD: general and scalable parallelization for ML computation graphs. CoRR, abs/2105.04663, 2021.\n\n7 Checklist\n\n(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Yes (b) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes (c) Did you discuss any potential negative societal impacts of your work? N/A. Not any. (d) Did you describe the limitations of your work? Yes (a) Did you include the code, data, and instructions needed to reproduce the main experimental results? Yes. We include details in the experiment setup to help reproduce the main results. (b) Did you specify all the training details? Yes (c) Did you report error bars? Yes (d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? Yes (a) If your work uses existing assets, did you cite the creators? Yes (b) Did you mention the license of the assets? No. The used dataset is not released yet. (c) Did you include any new assets either in the supplemental material or as a URL? No. The dataset is not released yet.\n\n(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? No. Not using persons' data. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes. The dataset does not contain any personally identifiable information or offensive content.\n\n",
+          "title": "6 Limitations",
+          "embedding": [
+            -0.005417884327471256,
+            -0.002057158388197422,
+            -0.02763829380273819,
+            -0.01599692367017269,
+            0.03654416278004646,
+            -0.02751435525715351,
+            -0.03312699869275093,
+            0.03036494180560112,
+            0.005386899691075087,
+            0.039306219667196274,
+            0.025407399982213974,
+            -0.014332606457173824,
+            0.02131742797791958,
+            -0.005431163590401411,
+            -0.0021988023072481155,
+            0.018927184864878654,
+            -0.01384570449590683,
+            0.01158825308084488,
+            0.00927768461406231,
+            -0.02854127436876297,
+            -0.0014927954180166125,
+            -0.016377592459321022,
+            -0.013624385930597782,
+            0.010021315887570381,
+            0.019777048379182816,
+            0.013535858131945133,
+            0.02400866523385048,
+            0.020803967490792274,
+            -0.02866521291434765,
+            0.00207154406234622,
+            0.029214082285761833,
+            -0.0010053407168015838,
+            -0.03507460653781891,
+            -0.017953382804989815,
+            -0.0425640344619751,
+            -0.03293224051594734,
+            0.08179943263530731,
+            0.002959032543003559,
+            -0.00252082129009068,
+            0.022733869031071663,
+            0.005395752843469381,
+            0.00014842192467767745,
+            -0.006825472228229046,
+            -0.010499364696443081,
+            -0.03427785634994507,
+            0.003585364669561386,
+            -0.016802523285150528,
+            -0.0011071473127231002,
+            -0.025478221476078033,
+            -0.004550314974039793,
+            -0.017519596964120865,
+            -0.032755184918642044,
+            0.031161688268184662,
+            -0.0005953476065769792,
+            -0.03493295982480049,
+            0.01935211755335331,
+            -0.005988887045532465,
+            -0.025230344384908676,
+            0.07755011320114136,
+            0.0031958436593413353,
+            0.01846684142947197,
+            -0.0008542906143702567,
+            -0.007923213765025139,
+            0.007737305480986834,
+            0.025159522891044617,
+            -0.019104240462183952,
+            -0.02299945056438446,
+            0.021813182160258293,
+            -0.0003170392010360956,
+            -0.005537396762520075,
+            -0.03576511889696121,
+            0.019900986924767494,
+            -0.022840101271867752,
+            -0.016377592459321022,
+            0.003184777684509754,
+            -0.012606319040060043,
+            0.019989514723420143,
+            0.033870629966259,
+            0.016784818843007088,
+            0.008303881622850895,
+            0.008781930431723595,
+            0.04323684424161911,
+            0.00798075646162033,
+            0.0331801176071167,
+            0.03234795853495598,
+            0.006143810227513313,
+            -0.0033684722147881985,
+            -0.07485887408256531,
+            0.0303118247538805,
+            -0.0013168469304218888,
+            -0.031002338975667953,
+            -0.0057985526509583,
+            -0.0010662033455446362,
+            -0.036296285688877106,
+            -0.020910201594233513,
+            0.041926637291908264,
+            -0.0014042678521946073,
+            -0.0038487340789288282,
+            -0.024398185312747955,
+            -0.006055282894521952,
+            -0.02207876369357109,
+            6.712184404022992e-05,
+            0.009959346614778042,
+            -0.01853766292333603,
+            -0.016342181712388992,
+            -0.016891051083803177,
+            0.00800731498748064,
+            -0.007122039329260588,
+            0.039164576679468155,
+            0.08378244936466217,
+            -0.007617793511599302,
+            -0.013208306394517422,
+            -0.016165126115083694,
+            -0.03117939457297325,
+            -0.02871832810342312,
+            -0.03197614103555679,
+            -0.03842094540596008,
+            -0.03534018620848656,
+            0.011172173544764519,
+            -0.018148142844438553,
+            0.002722221426665783,
+            0.014332606457173824,
+            0.01792682334780693,
+            -0.014217520132660866,
+            -0.03310929238796234,
+            0.049292124807834625,
+            -0.018944889307022095,
+            -0.03254271671175957,
+            1.4307914852906833e-06,
+            0.008264044299721718,
+            -0.02122890017926693,
+            0.030081652104854584,
+            -0.04224533587694168,
+            -0.05980919674038887,
+            0.024309657514095306,
+            -0.011163320392370224,
+            0.013305687345564365,
+            0.05771994590759277,
+            -0.04791109636425972,
+            -0.008897015824913979,
+            -0.000875315919984132,
+            -0.010844621807336807,
+            0.012951577082276344,
+            0.01338536199182272,
+            -0.020927906036376953,
+            0.0021456859540194273,
+            -0.019918693229556084,
+            0.080949567258358,
+            -0.021512188017368317,
+            0.02303486131131649,
+            -0.03383522108197212,
+            0.02114037238061428,
+            -0.029957713559269905,
+            0.01624480076134205,
+            0.002892636926844716,
+            -0.011287258937954903,
+            -0.03027641400694847,
+            0.004176286049187183,
+            -0.02312338910996914,
+            0.009507856331765652,
+            0.005413458216935396,
+            -0.005324930418282747,
+            0.02753205969929695,
+            0.01435916405171156,
+            0.015386084094643593,
+            0.047379929572343826,
+            0.032755184918642044,
+            0.02939113788306713,
+            -0.05977378413081169,
+            0.05453295633196831,
+            0.016944168135523796,
+            0.03944786638021469,
+            -0.008379130624234676,
+            -0.000762996613048017,
+            -0.0042448947206139565,
+            0.013677502982318401,
+            0.030099358409643173,
+            0.03587135300040245,
+            -0.005559528712183237,
+            0.027018601074814796,
+            -0.023725377395749092,
+            0.018254375085234642,
+            0.02505328878760338,
+            0.005364768207073212,
+            0.01628021150827408,
+            0.012057448737323284,
+            0.03241878002882004,
+            0.017625829204916954,
+            -0.009906229563057423,
+            0.022769279778003693,
+            0.011260701343417168,
+            0.014058170840144157,
+            0.04904424771666527,
+            -0.01383685227483511,
+            0.01018066518008709,
+            -0.013624385930597782,
+            -0.002472131047397852,
+            0.01252664439380169,
+            -0.020573796704411507,
+            0.01570478267967701,
+            -0.018732424825429916,
+            -0.04143088310956955,
+            0.03289682790637016,
+            -0.015943806618452072,
+            0.008649138733744621,
+            0.03750025853514671,
+            0.0009483511094003916,
+            0.03781895712018013,
+            0.012075154110789299,
+            -0.03050658479332924,
+            -0.0330030620098114,
+            -0.0070423646830022335,
+            -0.022645341232419014,
+            -0.01435031183063984,
+            -0.0025562322698533535,
+            0.006546610500663519,
+            -0.004501624498516321,
+            0.013128631748259068,
+            -0.0165015310049057,
+            -0.01853766292333603,
+            -0.021848592907190323,
+            0.010021315887570381,
+            0.038350123912096024,
+            -0.00276205874979496,
+            -0.027992403134703636,
+            -0.011189878918230534,
+            -0.016758261248469353,
+            -0.009773438796401024,
+            0.0026248411741107702,
+            0.006887441501021385,
+            0.028983911499381065,
+            -0.01018066518008709,
+            -0.022379757836461067,
+            0.014961151406168938,
+            0.03390604257583618,
+            -0.005634776782244444,
+            -0.011703338474035263,
+            0.010410836897790432,
+            -0.024380480870604515,
+            -0.003375111846253276,
+            -0.005360341630876064,
+            0.01664317399263382,
+            -0.04606972262263298,
+            -0.009711469523608685,
+            -0.007940919138491154,
+            0.01660776324570179,
+            0.017068106681108475,
+            0.029302610084414482,
+            0.028045520186424255,
+            0.03785436972975731,
+            -0.0027664850931614637,
+            -0.0036738922353833914,
+            -0.0034083097707480192,
+            0.02021968737244606,
+            0.004603431094437838,
+            0.03339258208870888,
+            0.02314109541475773,
+            0.008029446005821228,
+            -0.02868291735649109,
+            -0.02500017359852791,
+            0.06420016288757324,
+            -0.012287620455026627,
+            -0.042811911553144455,
+            0.04486574977636337,
+            -0.02595626935362816,
+            0.03323323279619217,
+            0.03689827397465706,
+            0.011384639889001846,
+            0.0029568192549049854,
+            0.007724026683717966,
+            0.036437928676605225,
+            -0.011482019908726215,
+            -0.037146151065826416,
+            -0.0064270980656147,
+            0.017971087247133255,
+            -0.02781534753739834,
+            0.037181559950113297,
+            -0.0314449779689312,
+            0.03689827397465706,
+            0.020768556743860245,
+            0.009764585644006729,
+            -0.008724387735128403,
+            0.007768290117383003,
+            0.009463592432439327,
+            0.007197287864983082,
+            -0.041997458785772324,
+            -0.013704060576856136,
+            -0.06529790163040161,
+            -0.05779076740145683,
+            0.011030529625713825,
+            -0.020981023088097572,
+            0.002284010173752904,
+            -0.047415342181921005,
+            0.01294272392988205,
+            0.014385722577571869,
+            -0.008857178501784801,
+            0.0027930433861911297,
+            0.03305617719888687,
+            -0.01848454773426056,
+            -0.007538118865340948,
+            -0.05435590073466301,
+            -0.0238847266882658,
+            0.013190601021051407,
+            0.04320143163204193,
+            -0.04688417539000511,
+            -0.0128541961312294,
+            -0.01763468235731125,
+            0.011685633100569248,
+            -0.015439200215041637,
+            -0.03206466883420944,
+            0.01921047270298004,
+            0.0033817514777183533,
+            -0.013199454173445702,
+            -0.060198716819286346,
+            -0.010995118878781796,
+            0.0442991741001606,
+            0.009950493462383747,
+            0.004362193867564201,
+            -0.009835408069193363,
+            -0.033870629966259,
+            0.0515938401222229,
+            -0.015819868072867393,
+            -0.05612644925713539,
+            0.012022037990391254,
+            -0.0076045142486691475,
+            -0.023619143292307854,
+            -0.008795209228992462,
+            0.0637398213148117,
+            0.0008692296105436981,
+            -0.000270700576948002,
+            -0.01936982199549675,
+            -0.004446295090019703,
+            -0.013226011767983437,
+            -0.009127187542617321,
+            -0.0059800343587994576,
+            -0.052939459681510925,
+            -0.022769279778003693,
+            0.009082923643290997,
+            -0.010641008615493774,
+            -0.0031537930481135845,
+            -0.02754976600408554,
+            -0.018980301916599274,
+            -0.004264813382178545,
+            0.004647694993764162,
+            0.02662907913327217,
+            -0.042032867670059204,
+            -0.030081652104854584,
+            0.033870629966259,
+            0.01763468235731125,
+            -0.01628021150827408,
+            0.0017827230039983988,
+            0.018927184864878654,
+            -0.0012139336904510856,
+            0.03760649263858795,
+            0.015403789468109608,
+            -0.01288075465708971,
+            -0.03125021606683731,
+            -0.021813182160258293,
+            0.0017307130619883537,
+            0.01610315591096878,
+            0.024610651656985283,
+            -0.04578643664717674,
+            -0.02399095892906189,
+            0.009915082715451717,
+            -0.06324406713247299,
+            -0.028771445155143738,
+            -0.03849176689982414,
+            0.008064857684075832,
+            0.015952659770846367,
+            -0.013491595163941383,
+            -0.017776327207684517,
+            0.023371266201138496,
+            -0.009684910997748375,
+            0.002646972890943289,
+            -0.007082202006131411,
+            0.024132603779435158,
+            0.03321552649140358,
+            -0.014642452821135521,
+            -0.038350123912096024,
+            0.030028536915779114,
+            -0.00265582581050694,
+            0.011676780879497528,
+            0.046636298298835754,
+            0.014669010415673256,
+            -0.03689827397465706,
+            0.01922817900776863,
+            0.016492677852511406,
+            0.013562416657805443,
+            -0.01197777409106493,
+            0.007060070056468248,
+            -0.023548321798443794,
+            -0.010065579786896706,
+            -0.0017683373298496008,
+            -0.018289785832166672,
+            -0.014961151406168938,
+            0.03194073215126991,
+            0.006196926813572645,
+            0.002070437418296933,
+            0.0043444884940981865,
+            -0.015200176276266575,
+            -0.005333783570677042,
+            -0.008334866724908352,
+            0.016979578882455826,
+            0.01921047270298004,
+            0.01236729510128498,
+            0.013633239082992077,
+            0.01662546955049038,
+            0.027921581640839577,
+            0.03976656496524811,
+            -0.035446420311927795,
+            0.010340014472603798,
+            -0.010286898352205753,
+            -0.03333946689963341,
+            -0.0008493109489791095,
+            0.039979029446840286,
+            0.04132464900612831,
+            -0.02029050886631012,
+            0.008928000926971436,
+            0.008622581139206886,
+            -0.003306502941995859,
+            -0.004944262094795704,
+            0.02774452604353428,
+            0.044653281569480896,
+            0.036402519792318344,
+            -0.009463592432439327,
+            0.008786357007920742,
+            -0.014916887506842613,
+            0.008821767754852772,
+            0.02505328878760338,
+            -0.03436638414859772,
+            -0.0034503601491451263,
+            0.004926556721329689,
+            0.03806683421134949,
+            -0.02131742797791958,
+            -0.003974885679781437,
+            -0.04578643664717674,
+            -0.014846066012978554,
+            -0.015642814338207245,
+            0.011172173544764519,
+            0.007029085420072079,
+            -0.013110926374793053,
+            -0.007471723016351461,
+            0.00531607773154974,
+            -0.01565166562795639,
+            -0.03806683421134949,
+            -0.03208237513899803,
+            -0.04228074476122856,
+            -0.03500378131866455,
+            -0.021653832867741585,
+            -0.021441366523504257,
+            -0.05644514784216881,
+            0.03774813562631607,
+            -0.08640286326408386,
+            -0.021777771413326263,
+            0.02765599824488163,
+            -0.013898821547627449,
+            0.01338536199182272,
+            -0.03898752108216286,
+            0.0128541961312294,
+            -0.05998625233769417,
+            -0.013730619102716446,
+            0.05403720214962959,
+            0.014483102597296238,
+            0.03289682790637016,
+            0.009596383199095726,
+            0.02124660648405552,
+            -0.05499329790472984,
+            -0.027089422568678856,
+            -0.002164497971534729,
+            -0.010933149605989456,
+            0.008511921390891075,
+            -0.002127980347722769,
+            0.004437441937625408,
+            0.012995840981602669,
+            0.0012238930212333798,
+            0.010711830109357834,
+            0.009622941724956036,
+            -0.05092103034257889,
+            -0.00127700949087739,
+            -0.004291371908038855,
+            0.0258854478597641,
+            0.0031537930481135845,
+            -0.0010833556298166513,
+            -0.013721765950322151,
+            0.010021315887570381,
+            0.014784096740186214,
+            0.017351394519209862,
+            0.015518874861299992,
+            0.01065871398895979,
+            -0.04631759971380234,
+            -0.019794754683971405,
+            0.039022933691740036,
+            -0.005015084054321051,
+            0.007015806157141924,
+            -0.01853766292333603,
+            -0.005121317226439714,
+            -0.01759927161037922,
+            -0.003237894270569086,
+            0.01759927161037922,
+            -0.0014042678521946073,
+            -0.014908035285770893,
+            -0.021441366523504257,
+            -0.041041359305381775,
+            0.023707671090960503,
+            0.03392374888062477,
+            -0.0004260387213435024,
+            -0.023318149149417877,
+            0.007307947147637606,
+            -0.04231615737080574,
+            -0.010579039342701435,
+            0.004926556721329689,
+            0.020556090399622917,
+            -0.01921047270298004,
+            -0.007055643480271101,
+            0.013509300537407398,
+            0.009968198835849762,
+            -0.0295327827334404,
+            0.013217159546911716,
+            0.02475229650735855,
+            0.023743081837892532,
+            -0.026328084990382195,
+            0.002602709224447608,
+            -0.003523395396769047,
+            0.048265207558870316,
+            -0.012508939020335674,
+            -0.012535497546195984,
+            -0.0035676592960953712,
+            -0.0030453468207269907,
+            0.017811737954616547,
+            -0.03303847089409828,
+            -0.019175061956048012,
+            0.011605958454310894,
+            -0.012030890211462975,
+            -0.008560611866414547,
+            -0.005125743802636862,
+            -0.012889607809484005,
+            0.010162959806621075,
+            0.01756386086344719,
+            -0.02756747044622898,
+            -0.020857084542512894,
+            -0.0347382016479969,
+            0.04975247010588646,
+            -0.01288075465708971,
+            -0.0023415531031787395,
+            0.03426015377044678,
+            0.009490150958299637,
+            0.008109121583402157,
+            -0.018201259896159172,
+            0.03337487578392029,
+            0.02473459020256996,
+            0.014624747447669506,
+            -0.024327363818883896,
+            0.00028245814610272646,
+            0.025035584345459938,
+            0.0023282738402485847,
+            -0.017900265753269196,
+            -0.018643897026777267,
+            -0.003943901043385267,
+            0.007693042047321796,
+            -0.018998006358742714,
+            -0.010286898352205753,
+            0.004421949852257967,
+            -0.014093581587076187,
+            -0.00017318196478299797,
+            -0.014208667911589146,
+            -0.032719772309064865,
+            0.002005148446187377,
+            0.01525329239666462,
+            0.004298011306673288,
+            0.026345791295170784,
+            -0.009401623159646988,
+            -0.0028904236387461424,
+            0.009985905140638351,
+            -0.003054199507459998,
+            0.036367107182741165,
+            0.02951507642865181,
+            0.007427459117025137,
+            -0.009472444653511047,
+            -0.026328084990382195,
+            -0.014368017204105854,
+            0.005081479903310537,
+            0.008941279724240303,
+            0.001089995144866407,
+            0.02776223234832287,
+            0.026381202042102814,
+            0.014598188921809196,
+            -0.009738028049468994,
+            -0.003412736114114523,
+            0.000571002543438226,
+            -0.009286537766456604,
+            0.02671760693192482,
+            0.05623268336057663,
+            0.012269915081560612,
+            -0.032755184918642044,
+            0.009879671968519688,
+            -0.02678842842578888,
+            0.006055282894521952,
+            0.014270637184381485,
+            0.002030600095167756,
+            -0.056799259036779404,
+            0.02659366838634014,
+            0.014970004558563232,
+            0.014996563084423542,
+            -0.020857084542512894,
+            -0.05410802364349365,
+            0.000554680242203176,
+            0.006431524641811848,
+            0.0010390918469056487,
+            -0.007940919138491154,
+            -0.015456905588507652,
+            0.01718319207429886,
+            -0.0005671294638887048,
+            -0.007405327633023262,
+            0.007498281542211771,
+            0.02484082244336605,
+            -0.012588613666594028,
+            0.007825832813978195,
+            -0.01291616540402174,
+            -0.0009682697709649801,
+            0.005302798934280872,
+            0.030099358409643173,
+            0.0037115165032446384,
+            0.03493295982480049,
+            -0.003399456851184368,
+            0.016359886154532433,
+            0.028913090005517006,
+            -0.010587891563773155,
+            -0.006081840954720974,
+            0.0019055550219491124,
+            0.0035920043010264635,
+            -0.021441366523504257,
+            0.003401670139282942,
+            -0.025991681963205338,
+            -0.03406539186835289,
+            0.010189518332481384,
+            0.0030475601088255644,
+            0.009755733422935009,
+            0.015881838276982307,
+            -0.004410883877426386,
+            -0.0012902886373922229,
+            0.0005508071626536548,
+            0.015005415305495262,
+            0.0038730790838599205,
+            0.12500086426734924,
+            -0.01390767376869917,
+            0.024150308221578598,
+            0.005298372358083725,
+            0.018148142844438553,
+            0.03757108002901077,
+            0.01201318483799696,
+            0.017811737954616547,
+            0.007688615471124649,
+            0.003939474932849407,
+            0.04405129700899124,
+            -0.02103414013981819,
+            -0.01570478267967701,
+            0.016289064660668373,
+            -0.008786357007920742,
+            0.015332967042922974,
+            0.0014540646225214005,
+            -0.015766752883791924,
+            0.011667927727103233,
+            0.02029050886631012,
+            -0.010791505686938763,
+            0.02305256761610508,
+            -0.019086534157395363,
+            -0.031055456027388573,
+            0.01528870314359665,
+            -0.004957541357725859,
+            0.04858390614390373,
+            0.0051611545495688915,
+            -0.004966394044458866,
+            -0.02199023775756359,
+            -0.003468065755441785,
+            -0.02843504026532173,
+            -0.018891774117946625,
+            0.008560611866414547,
+            0.027089422568678856,
+            0.02758517675101757,
+            0.0017041548853740096,
+            -0.010056726634502411,
+            -0.007485002279281616,
+            0.025442810729146004,
+            0.0014927954180166125,
+            -0.00400365749374032,
+            0.006590874399989843,
+            -0.014474250376224518,
+            0.011659074574708939,
+            -0.014376870356500149,
+            0.003954967018216848,
+            0.0004039068298880011,
+            -0.01290731318295002,
+            0.0009544373606331646,
+            0.03569429740309715,
+            0.005404605530202389,
+            -0.01018066518008709,
+            -0.03151579946279526,
+            0.01284534391015768,
+            0.01419096253812313,
+            0.00324232061393559,
+            -0.0023039288353174925,
+            -0.0009594170260243118,
+            -0.011605958454310894,
+            0.029107850044965744,
+            -0.002613775199279189,
+            -0.004528183024376631,
+            0.0027200081385672092,
+            0.0006728091975674033,
+            -0.01940523274242878,
+            0.015828721225261688,
+            0.011260701343417168,
+            0.025389693677425385,
+            0.022362053394317627,
+            0.024097193032503128,
+            0.019830165430903435,
+            0.005900359712541103,
+            -0.005214271135628223,
+            -0.013252570293843746,
+            -0.02030821330845356,
+            0.043838828802108765,
+            0.00038813785067759454,
+            0.012863049283623695,
+            -0.000697707524523139,
+            -0.0684494823217392,
+            -0.033711280673742294,
+            0.019015712663531303,
+            0.02220270223915577,
+            0.024415891617536545,
+            0.0017849361756816506,
+            0.0028616522904485464,
+            0.022468285635113716,
+            -0.04118300601840019,
+            -0.007600088138133287,
+            0.005006231367588043,
+            -0.009074071422219276,
+            -0.004997378680855036,
+            -0.0025141816586256027,
+            0.023442087695002556,
+            -0.01660776324570179,
+            -0.012243356555700302,
+            0.034614261239767075,
+            -0.014943446032702923,
+            0.02662907913327217,
+            -0.00017829996068030596,
+            0.0008636966813355684,
+            0.012199092656373978,
+            0.013270275667309761,
+            0.03250730782747269,
+            -0.020963318645954132,
+            -0.0080781364813447,
+            0.003970459569245577,
+            -0.009622941724956036,
+            0.029037028551101685,
+            -0.009968198835849762,
+            -0.04592807963490486,
+            0.01666088029742241,
+            0.016837935894727707,
+            0.03693368285894394,
+            -0.005453295540064573,
+            0.010189518332481384,
+            -0.010782652534544468,
+            0.041749581694602966,
+            -0.005262961145490408,
+            -0.007569103501737118,
+            0.03410080075263977,
+            -0.009950493462383747,
+            0.0029014896135777235,
+            0.022344347089529037,
+            -0.011198732070624828,
+            -0.031692855060100555,
+            0.019564582034945488,
+            0.016041187569499016,
+            0.059419672936201096,
+            -0.018732424825429916,
+            0.016598910093307495,
+            0.019723933190107346,
+            -0.0020217474084347486,
+            -0.01651923544704914,
+            -0.03689827397465706,
+            0.008308308199048042,
+            -0.007077775429934263,
+            0.007489428389817476,
+            -0.015908395871520042,
+            0.03707532584667206,
+            -0.019759343937039375,
+            -0.013996201567351818,
+            0.008742093108594418,
+            -0.0036318416241556406,
+            -0.01338536199182272,
+            0.02664678543806076,
+            -0.0048822928220033646,
+            -0.03927081078290939,
+            -0.012818785384297371,
+            0.007223845925182104,
+            0.02939113788306713,
+            -0.02112266793847084,
+            0.01057018619030714,
+            0.022344347089529037,
+            -0.015518874861299992,
+            -0.02478770725429058,
+            0.01201318483799696,
+            -0.016784818843007088,
+            0.009481297805905342,
+            0.02480541169643402,
+            -0.03050658479332924,
+            -0.006852030288428068,
+            -0.028399629518389702,
+            -0.01374832447618246,
+            -0.012190240435302258,
+            0.019015712663531303,
+            0.0030187885276973248,
+            0.0060729882679879665,
+            -0.025177227333188057,
+            0.027408121153712273,
+            0.013996201567351818,
+            -0.013854557648301125,
+            -0.01834290288388729,
+            -0.010251487605273724,
+            -0.02496476098895073,
+            -0.0044662137515842915,
+            -0.02036133036017418,
+            -0.016528088599443436,
+            -7.096034096321091e-05,
+            -0.004227189347147942,
+            -0.018927184864878654,
+            0.005891506560146809,
+            -0.00015533814439550042,
+            0.004705238156020641,
+            0.06069447100162506,
+            0.007830259390175343,
+            0.02593856491148472,
+            -0.003412736114114523,
+            -0.020007221028208733,
+            0.03149809315800667,
+            -0.02018427476286888,
+            0.042882733047008514,
+            0.0008465444552712142,
+            -0.03032952919602394,
+            0.01010984368622303,
+            0.037004504352808,
+            -0.009056366048753262,
+            0.006121678277850151,
+            -0.01832519844174385,
+            0.0012006545439362526,
+            -0.013403067365288734,
+            -0.014031612314283848,
+            -0.0007010273402556777,
+            0.006639564409852028,
+            0.01563396118581295,
+            0.01521788164973259,
+            0.017811737954616547,
+            0.011358081363141537,
+            0.0008033872582018375,
+            0.02027280256152153,
+            -0.022609930485486984,
+            0.008888163603842258,
+            -0.009826554916799068,
+            0.021848592907190323,
+            0.006701533682644367,
+            0.0039372616447508335,
+            -0.011207584291696548,
+            0.004322356544435024,
+            -0.038314711302518845,
+            0.0028218147344887257,
+            0.01747533306479454,
+            0.022556813433766365,
+            -0.02595626935362816,
+            -0.04578643664717674,
+            -0.004820323549211025,
+            -0.05775535851716995,
+            -0.028293397277593613,
+            -0.010995118878781796,
+            -0.00369159784168005,
+            0.02209646999835968,
+            -0.002565084956586361,
+            0.024380480870604515,
+            -0.018148142844438553,
+            -0.014854918234050274,
+            0.01195121556520462,
+            -0.022379757836461067,
+            -0.008272897452116013,
+            0.016891051083803177,
+            0.017271719872951508,
+            -0.017971087247133255,
+            0.028045520186424255,
+            -0.024681473150849342,
+            -0.0035809383261948824,
+            -0.04398047551512718,
+            -0.01834290288388729,
+            -0.015934955328702927,
+            -0.0035411010030657053,
+            0.008604875765740871,
+            -0.011933510191738605,
+            -0.016722848638892174,
+            0.0011663500918075442,
+            0.009516708552837372,
+            -0.013633239082992077,
+            0.03346340358257294,
+            -0.01834290288388729,
+            -0.017360247671604156,
+            0.002666891785338521,
+            -0.012349589727818966,
+            -0.021777771413326263,
+            -0.02206105925142765,
+            0.01293387170881033,
+            -0.020626913756132126,
+            0.004922130145132542,
+            -0.017953382804989815,
+            0.013562416657805443,
+            -0.02671760693192482,
+            -0.01844913698732853,
+            -0.0029656721744686365,
+            -0.012925018556416035,
+            -0.05708254873752594,
+            0.019971810281276703,
+            0.05283322557806969,
+            -0.005479853600263596,
+            -0.014890329912304878,
+            0.033551931381225586,
+            -0.008600449189543724,
+            -0.024380480870604515,
+            0.02485852874815464,
+            -0.0019199406960979104,
+            0.023654554039239883,
+            -0.02505328878760338,
+            -0.0035676592960953712,
+            0.00015353992057498544,
+            0.019564582034945488,
+            -0.020927906036376953,
+            0.007405327633023262,
+            -0.013004693202674389,
+            0.022362053394317627,
+            -0.00020485822460614145,
+            0.006391687318682671,
+            0.046636298298835754,
+            -0.014031612314283848,
+            -0.00970261637121439,
+            0.04157252609729767,
+            -0.006059709005057812,
+            0.0032069096341729164,
+            -0.024433596059679985,
+            0.020502975210547447,
+            0.03619005158543587,
+            0.008914722129702568,
+            0.01560740265995264,
+            -0.03523395583033562,
+            0.028736034408211708,
+            0.004696385003626347,
+            0.028895383700728416,
+            0.029196377843618393,
+            -0.01944064348936081,
+            0.026115620508790016,
+            0.007799274753779173,
+            0.0128541961312294,
+            -0.0017174340318888426,
+            0.02018427476286888,
+            -0.018661601468920708,
+            -0.01673170179128647,
+            -0.01566937193274498,
+            0.0442991741001606,
+            -0.002087036380544305,
+            -0.010295751504600048,
+            -0.01721860282123089,
+            -0.02478770725429058,
+            0.0048778667114675045,
+            0.0029435402248054743,
+            -0.01470442209392786,
+            -0.01673170179128647,
+            -0.021972531452775,
+            0.015439200215041637,
+            0.010233782231807709,
+            -0.013527005910873413,
+            0.00711761275306344,
+            -0.002790830098092556,
+            -0.0038575867656618357,
+            -0.002963458886370063,
+            -0.01246467512100935,
+            -0.03760649263858795,
+            -0.022751573473215103,
+            -0.01515591237694025,
+            0.03333946689963341,
+            -0.010136401280760765,
+            0.020591503009200096,
+            -0.006081840954720974,
+            0.012553202919661999,
+            -0.0389166995882988,
+            0.00026378437178209424,
+            -0.015067384578287601,
+            -0.0304888803511858,
+            0.025194933637976646,
+            -0.010605597868561745,
+            0.008879310451447964,
+            0.032578129321336746,
+            -0.027124833315610886,
+            -0.024610651656985283,
+            0.01338536199182272,
+            -0.004353341180831194,
+            0.0017550581833347678,
+            -0.017625829204916954,
+            0.016944168135523796,
+            -0.01942293904721737,
+            -0.028045520186424255,
+            0.013057810254395008,
+            0.03192302584648132,
+            0.012606319040060043,
+            0.012199092656373978,
+            0.03693368285894394,
+            -0.006276601459830999,
+            -0.037181559950113297,
+            -0.006471361964941025,
+            -0.009499003179371357,
+            -0.03572971001267433,
+            -0.004537035711109638,
+            0.006435951218008995,
+            -0.003970459569245577,
+            -0.01669629104435444,
+            -0.02379619888961315,
+            -0.007940919138491154,
+            -0.012349589727818966,
+            -0.05240829288959503,
+            0.01516476459801197,
+            -0.019777048379182816,
+            -0.004286945331841707,
+            -0.03418932855129242,
+            0.05077938735485077,
+            0.02843504026532173,
+            0.007467296905815601,
+            0.006736944429576397,
+            -0.01293387170881033,
+            -0.0021423660218715668,
+            0.00507262721657753,
+            0.0032179756090044975,
+            0.01922817900776863,
+            -0.020910201594233513,
+            0.014987709932029247,
+            -0.003056412795558572,
+            0.005457721650600433,
+            -0.056693024933338165,
+            -0.020927906036376953,
+            0.010056726634502411,
+            0.005581660196185112,
+            -0.0003308716113679111,
+            0.0037911911495029926,
+            0.011783013120293617,
+            -0.01716548763215542,
+            -0.021689243614673615,
+            0.01610315591096878,
+            0.030063947662711143,
+            0.01522673387080431,
+            -0.006365128792822361,
+            0.019104240462183952,
+            0.019599994644522667,
+            -0.006865309551358223,
+            0.005993313621729612,
+            -0.01426178403198719,
+            0.009569825604557991,
+            0.010853474959731102,
+            0.0036583999171853065,
+            0.022822394967079163,
+            -0.00203502643853426,
+            -0.0013876688899472356,
+            -0.00814010575413704,
+            0.003932835068553686,
+            -0.005546249449253082,
+            -0.03025870770215988,
+            -0.014757538214325905,
+            -0.013633239082992077,
+            0.011242995969951153,
+            0.03426015377044678,
+            0.02126431092619896,
+            0.014722127467393875,
+            -0.012747963890433311,
+            -0.006807766854763031,
+            -0.001094974810257554,
+            0.006334144156426191,
+            0.003992591518908739,
+            0.01387226302176714,
+            -0.014474250376224518,
+            -0.008538479916751385,
+            0.002060478087514639,
+            0.003950540907680988,
+            -0.01197777409106493,
+            0.0385625883936882,
+            0.04635301232337952,
+            -0.02106955088675022,
+            -0.017953382804989815,
+            0.006577595137059689,
+            0.016811376437544823,
+            -0.005652482621371746,
+            -0.0037115165032446384,
+            -0.005873801186680794,
+            -0.00027249878621660173,
+            -0.01787370629608631,
+            0.015200176276266575,
+            0.01111020427197218,
+            -0.02767370454967022,
+            0.010835768654942513,
+            0.09653041511774063,
+            -0.02661137469112873,
+            0.009206862188875675,
+            -0.026345791295170784,
+            -0.011995479464530945,
+            -0.004554741084575653,
+            0.0048424554988741875,
+            0.02864750660955906,
+            0.02294633351266384,
+            0.0032179756090044975,
+            0.025389693677425385,
+            -0.0028328807093203068,
+            -0.010791505686938763,
+            0.013199454173445702,
+            -0.0006645097164437175,
+            0.013287981040775776,
+            -0.02395554818212986,
+            0.010286898352205753,
+            -0.011411197483539581,
+            -0.011358081363141537,
+            0.011437756009399891,
+            0.013757177628576756,
+            -0.0006883015157654881,
+            0.021671537309885025,
+            -0.004014723002910614,
+            0.02200794219970703,
+            0.0008731026900932193,
+            -4.9450923143012915e-06,
+            -0.04964623600244522,
+            -0.015085089951753616,
+            0.0024190146941691637,
+            -0.007932065986096859,
+            -0.02115807868540287,
+            0.001672063604928553,
+            -0.00224527926184237,
+            0.01566937193274498,
+            -0.01057018619030714,
+            0.01714778132736683,
+            0.0050682006403803825,
+            -0.010552480816841125,
+            -0.0037292218767106533,
+            0.009321948513388634,
+            -0.011783013120293617,
+            0.012473528273403645,
+            -0.009472444653511047,
+            0.004242681432515383,
+            0.023707671090960503,
+            -0.025336576625704765,
+            0.01191580481827259,
+            -0.009684910997748375,
+            0.000783468596637249,
+            0.016846787184476852,
+            -0.018714718520641327,
+            0.008317161351442337,
+            -0.0012515578418970108,
+            0.006254469510167837,
+            0.001487262430600822,
+            -0.016864493489265442,
+            0.012570908293128014,
+            -0.010853474959731102,
+            0.005417884327471256,
+            0.007245977874845266,
+            0.01778518036007881,
+            -0.014854918234050274,
+            0.010711830109357834,
+            -0.006064135581254959,
+            -0.002739926800131798,
+            -0.0035698723513633013,
+            -0.019086534157395363,
+            0.0028572259470820427,
+            0.00687416223809123,
+            -0.006099546328186989,
+            -0.0249293502420187,
+            0.002538526663556695,
+            -0.004267026670277119,
+            0.019635405391454697,
+            0.00019074915326200426,
+            -0.015279850922524929,
+            -0.007613366935402155,
+            -0.020981023088097572,
+            0.012792226858437061,
+            0.0021589649841189384,
+            0.014164404012262821,
+            0.011986627243459225,
+            -0.007781569380313158,
+            0.008188796229660511,
+            -0.024398185312747955,
+            0.006360702682286501,
+            0.02593856491148472,
+            -0.002722221426665783,
+            -0.0007989609148353338,
+            -0.004023576155304909,
+            -0.010340014472603798,
+            -0.00014330392878036946,
+            -0.007684188894927502,
+            -0.014421133324503899,
+            0.0031936303712427616,
+            0.00054444425040856,
+            -0.011251848191022873,
+            0.00483360281214118,
+            -0.012411559000611305,
+            0.015713635832071304,
+            0.01249123364686966,
+            -0.014075876213610172,
+            -0.020680028945207596,
+            0.00714859738945961,
+            -0.010003610514104366,
+            -0.0013223799178376794,
+            0.0026226278860121965,
+            0.0008360318024642766,
+            0.008742093108594418,
+            -0.026044797152280807,
+            0.003638481255620718,
+            -0.018838657066226006,
+            0.015501169487833977,
+            -0.011304965242743492,
+            0.014067023992538452,
+            -0.022698456421494484,
+            0.01689990423619747,
+            0.020520679652690887,
+            -0.019883282482624054,
+            0.0038664396852254868,
+            0.013580122031271458,
+            -0.021459072828292847,
+            0.0119423633441329,
+            0.015297556295990944,
+            0.00866684503853321,
+            -0.011384639889001846,
+            0.006307586096227169,
+            0.015315261669456959,
+            0.001169669907540083,
+            0.018608486279845238,
+            0.017121223732829094,
+            0.0027000894770026207,
+            0.01332339271903038,
+            -0.02661137469112873,
+            0.020963318645954132,
+            0.012597466818988323,
+            0.03948327526450157,
+            -0.00597560778260231,
+            -0.004873440135270357,
+            0.006157089490443468,
+            0.02204335294663906,
+            -0.0017041548853740096,
+            0.012243356555700302,
+            -0.009817702695727348,
+            -0.01810387894511223,
+            0.03142727166414261,
+            0.00804272573441267,
+            0.025124112144112587,
+            -0.010401983745396137,
+            -0.004286945331841707,
+            0.025088699534535408,
+            0.019139651209115982,
+            0.0030608391389250755,
+            -0.01010984368622303,
+            0.005670187994837761,
+            -0.0015215668827295303,
+            0.023388972505927086,
+            -0.013889968395233154,
+            0.011844982393085957,
+            0.029975419864058495,
+            0.024203425273299217,
+            -0.028116341680288315,
+            0.005807405803352594,
+            -0.012986987829208374,
+            -0.0049354094080626965,
+            -0.008649138733744621,
+            0.009375064633786678,
+            0.0030165754724293947,
+            -0.020733145996928215,
+            -0.004510477185249329,
+            -0.002430080436170101,
+            -0.0031095293816179037,
+            0.039235398173332214,
+            -0.017917970195412636,
+            0.03620775789022446,
+            0.01841372437775135,
+            0.016705144196748734,
+            -0.022468285635113716,
+            0.030719051137566566,
+            -0.016253653913736343,
+            -0.010800357908010483,
+            -0.0007917680195532739,
+            0.008339292369782925,
+            0.0066351378336548805,
+            -0.017289426177740097,
+            0.008423393592238426,
+            -0.018626190721988678,
+            0.02025509811937809,
+            -0.004899998661130667,
+            -0.00035604662843979895,
+            -0.02034362591803074,
+            0.0038332417607307434,
+            0.00898111704736948,
+            0.010410836897790432,
+            -0.007307947147637606,
+            -0.008162237703800201,
+            -0.009025380946695805,
+            -0.01733369007706642,
+            -0.017156634479761124,
+            0.011393492110073566,
+            -0.01201318483799696,
+            -0.01152628380805254,
+            0.02207876369357109,
+            -0.029762953519821167,
+            -0.022415168583393097,
+            0.018077321350574493,
+            -0.0025031156837940216,
+            -0.028948500752449036,
+            0.004541461821645498,
+            -0.019104240462183952,
+            0.004041281528770924,
+            -0.008950132876634598,
+            0.0008830620790831745,
+            0.00894570630043745,
+            -0.008733239956200123,
+            -0.03955409675836563,
+            -0.003014362184330821,
+            0.0037535671144723892,
+            -0.009136040695011616,
+            0.00711318664252758,
+            0.008011740632355213,
+            0.0008902549161575735,
+            0.013447331264615059,
+            -0.009968198835849762,
+            0.008202075026929379,
+            -0.03774813562631607,
+            0.02218499779701233,
+            0.009684910997748375,
+            -0.036437928676605225,
+            0.010295751504600048,
+            -0.00811354722827673,
+            0.010463953018188477,
+            0.02753205969929695,
+            0.00441752327606082,
+            0.020697735249996185,
+            -0.002020640764385462,
+            0.0011962282005697489,
+            0.011862688697874546,
+            -0.0033618328161537647,
+            0.014642452821135521,
+            -0.01619168370962143,
+            0.029338020831346512,
+            -0.0011873753974214196,
+            0.00014551711501553655,
+            0.032737478613853455,
+            -0.003027641447260976,
+            -0.02577921561896801,
+            0.009879671968519688,
+            -0.013766029849648476,
+            -0.030966928228735924,
+            0.030665934085845947,
+            -0.03119709901511669,
+            0.0013046744279563427,
+            0.00852077454328537,
+            -0.006546610500663519,
+            -0.0005032236222177744,
+            -0.02501787804067135,
+            -0.030117064714431763,
+            0.010039021261036396,
+            0.014571630395948887,
+            0.011304965242743492,
+            0.018183553591370583,
+            -0.027018601074814796,
+            0.013004693202674389,
+            -0.008073709905147552,
+            0.006303159520030022,
+            0.009587530978024006,
+            0.012119418010115623,
+            -0.019175061956048012,
+            0.025991681963205338,
+            -0.008432246744632721,
+            0.02397325448691845,
+            0.0037292218767106533,
+            -0.0030741184018552303,
+            -0.00026378437178209424,
+            -0.03240107372403145,
+            0.006241190247237682,
+            -0.006188074126839638,
+            0.003757993457838893,
+            0.0797455906867981,
+            0.0011641369201242924,
+            -0.0020881430245935917,
+            0.022733869031071663,
+            -0.009375064633786678,
+            -0.011030529625713825,
+            -0.015297556295990944,
+            -0.006347423419356346,
+            -0.013527005910873413,
+            -0.024628357961773872,
+            -0.006263322196900845,
+            -0.004260387271642685,
+            -0.005957902409136295,
+            0.01841372437775135,
+            0.0006362915737554431,
+            -0.028222573921084404,
+            0.0002011234755627811,
+            -0.010826916433870792,
+            -0.010844621807336807,
+            0.017953382804989815,
+            0.0037380747962743044,
+            0.007608940824866295,
+            -0.00400365749374032,
+            -0.01758156716823578,
+            -0.029338020831346512,
+            -0.026947777718305588,
+            -0.0002370877773500979,
+            -0.006980395410209894,
+            -0.0016432921402156353,
+            0.001767230685800314,
+            -0.023530615493655205,
+            -0.030028536915779114,
+            0.02115807868540287,
+            -0.015412641689181328,
+            0.005550675559788942,
+            0.011384639889001846,
+            -0.00044540411909110844,
+            1.79648632183671e-05,
+            0.020609207451343536,
+            -0.04808815196156502,
+            -0.015040826052427292,
+            -0.010809211060404778,
+            -0.008653565309941769,
+            -0.016775965690612793,
+            0.015377230942249298,
+            -0.010198370553553104,
+            -0.003839881392195821,
+            -0.014067023992538452,
+            -0.002136833034455776,
+            -0.0013688568724319339,
+            -0.030170179903507233,
+            -0.012668288312852383,
+            -0.015430347062647343,
+            0.031108571216464043,
+            0.008861605077981949,
+            -0.000989848398603499,
+            -0.0019409660017117858,
+            -0.0161562729626894,
+            0.017032695934176445,
+            -0.03788977861404419,
+            0.008923574350774288,
+            -0.008100268431007862,
+            0.028222573921084404,
+            0.02760288119316101,
+            -0.017935676500201225,
+            0.0038819320034235716,
+            -0.003523395396769047,
+            0.003890784690156579,
+            -0.0087730772793293,
+            -0.021848592907190323,
+            0.010782652534544468,
+            0.0102072237059474,
+            0.002379177138209343,
+            0.030612817034125328,
+            0.02590315416455269,
+            -0.001249344670213759,
+            0.014093581587076187,
+            -0.009313095360994339,
+            0.006958263460546732,
+            -0.02108725719153881,
+            0.020042631775140762,
+            -0.008688976988196373,
+            0.0028572259470820427,
+            -0.008476510643959045,
+            0.013978496193885803,
+            -0.014642452821135521,
+            -0.010933149605989456,
+            0.04061642661690712,
+            -0.018697014078497887,
+            -0.011729896999895573,
+            -0.013987348414957523,
+            0.00393504835665226,
+            0.00924227386713028,
+            0.008750945329666138,
+            0.030010830610990524,
+            -0.002717795083299279,
+            0.012314178980886936,
+            0.001447424991056323,
+            0.003954967018216848,
+            -0.017794031649827957,
+            -0.010100990533828735,
+            0.005147875752300024,
+            0.00666612247005105,
+            0.004992952570319176,
+            -0.000783468596637249,
+            -0.011304965242743492,
+            0.0025119686033576727,
+            0.00021924394241068512,
+            -0.018254375085234642,
+            0.003118382068350911,
+            -0.03284371271729469,
+            0.014952299185097218,
+            0.0020637980196624994,
+            0.01794452965259552,
+            0.009746880270540714,
+            -0.026434319093823433,
+            0.018201259896159172,
+            -0.001513820723630488,
+            0.00762664619833231,
+            0.027248771861195564,
+            -0.031126277521252632,
+            -0.012340736575424671,
+            0.014908035285770893,
+            0.022432874888181686,
+            0.013491595163941383,
+            0.008246338926255703,
+            -0.027018601074814796,
+            0.026345791295170784,
+            -0.015554285608232021,
+            0.00805157795548439,
+            -0.02204335294663906,
+            -0.007706320844590664,
+            -0.032631244510412216,
+            -0.00017110710905399173,
+            -0.0017550581833347678,
+            0.013465036638081074,
+            -0.0023836034815758467,
+            -0.02126431092619896,
+            0.0034105228260159492,
+            0.007878949865698814,
+            -0.014368017204105854,
+            -0.008883737027645111,
+            0.0017616976983845234,
+            -0.00552411749958992,
+            0.0007784889312461019,
+            0.008879310451447964,
+            -0.0035477406345307827,
+            -0.03034723550081253,
+            -0.010756094008684158,
+            0.0027023027651011944,
+            0.008897015824913979,
+            -0.0033640458714216948,
+            0.014270637184381485,
+            -0.011827277019619942,
+            0.0018646109383553267,
+            -0.003215762320905924,
+            0.0109862657263875,
+            0.0019863364286720753,
+            -0.008852752856910229,
+            0.0013998415088281035,
+            -0.00798075646162033,
+            -0.0276205874979496,
+            0.015023120678961277,
+            -0.00504606869071722,
+            -0.006493493914604187,
+            -0.0002827347780112177,
+            0.0015337393851950765,
+            -0.005152301862835884,
+            -0.013925379142165184,
+            0.011774160899221897,
+            0.0018491186201572418,
+            0.0014507448067888618,
+            -0.0036318416241556406,
+            -0.004138661548495293,
+            -0.002044985769316554,
+            0.01751074381172657,
+            0.02774452604353428,
+            -0.018626190721988678,
+            0.010490511544048786,
+            -0.01767009310424328,
+            0.00048026180593296885,
+            -0.001060117152519524,
+            0.01763468235731125,
+            0.011375786736607552,
+            0.02114037238061428,
+            0.01516476459801197,
+            -0.008719961158931255,
+            0.010384278371930122,
+            -0.008166664279997349,
+            0.015872985124588013,
+            -0.009348507039248943,
+            -0.0007602301193401217,
+            -0.0009821022395044565,
+            0.02209646999835968,
+            -0.018980301916599274,
+            -0.0017284998903051019,
+            0.009499003179371357,
+            0.022450579330325127,
+            -0.016908757388591766,
+            -0.011242995969951153,
+            0.031108571216464043,
+            0.003414949169382453,
+            0.006975968834012747,
+            0.017236309126019478,
+            0.03229483962059021,
+            0.006307586096227169,
+            0.012252209708094597,
+            0.007843539118766785,
+            0.015872985124588013,
+            0.011340375989675522,
+            -0.010800357908010483,
+            -0.029143260791897774,
+            0.018767835572361946,
+            -0.0001229010958923027,
+            0.00265803886577487,
+            -0.00462113693356514,
+            0.008958985097706318,
+            0.012677141465246677,
+            -0.011685633100569248,
+            0.009206862188875675,
+            -0.013040103949606419,
+            0.017678946256637573,
+            0.01843143068253994,
+            0.0008891483303159475,
+            -0.02121119573712349,
+            0.007431885693222284,
+            0.02103414013981819,
+            0.0035123294219374657,
+            -0.004178499337285757,
+            -0.003775698831304908,
+            -0.01669629104435444,
+            -0.0017550581833347678,
+            0.0016034548170864582,
+            0.03218860924243927,
+            -0.00483360281214118,
+            -0.008383556269109249,
+            0.004625563044101,
+            0.00016543580568395555,
+            0.007883376441895962,
+            0.015412641689181328,
+            0.004811470862478018,
+            0.005701172631233931,
+            -0.012376148253679276,
+            0.011667927727103233,
+            0.024451302364468575,
+            0.011154468171298504,
+            0.0030608391389250755,
+            -0.0005566168110817671,
+            -0.000995934708043933,
+            0.016528088599443436,
+            -0.015014268457889557,
+            -0.006582021247595549,
+            -0.007538118865340948,
+            0.019033417105674744,
+            -0.0024499993305653334,
+            -0.006679401732981205,
+            0.014270637184381485,
+            -0.0009505642810836434,
+            0.004501624498516321,
+            -0.010375426150858402,
+            0.01335880346596241,
+            0.008852752856910229,
+            0.014137845486402512,
+            -0.010242634452879429,
+            -0.018024204298853874,
+            0.005386899691075087,
+            -0.0026602521538734436,
+            -0.021724654361605644,
+            0.007737305480986834,
+            -0.010800357908010483,
+            0.008233060128986835,
+            -0.0007657631067559123,
+            0.00255401898175478,
+            0.0014551711501553655,
+            0.0009533307747915387,
+            -0.0058472431264817715,
+            0.007276962511241436,
+            -0.013465036638081074,
+            -0.0003488537622615695,
+            -0.007347784470766783,
+            -0.028753740713000298,
+            0.0026735311839729548,
+            -5.22865702805575e-05,
+            0.005807405803352594,
+            0.009383917786180973,
+            0.00576314190402627,
+            0.003853160422295332,
+            0.014120140112936497,
+            -0.027868464589118958,
+            -0.0041231694631278515,
+            -0.00735221104696393,
+            0.02032591961324215,
+            0.012544349767267704,
+            -0.01053477544337511,
+            -0.007847964763641357,
+            0.0048778667114675045,
+            0.011783013120293617,
+            -0.0097291748970747,
+            0.008365850895643234,
+            0.02777993679046631,
+            -0.009844260290265083,
+            -0.02131742797791958,
+            0.0007956410991027951,
+            0.0042448947206139565,
+            0.01851995848119259,
+            0.025141816586256027,
+            0.01249123364686966,
+            -0.0028793576639145613,
+            -0.009897377341985703,
+            -0.013305687345564365,
+            0.029160967096686363,
+            -0.006887441501021385,
+            -0.01245582289993763,
+            -0.011871540918946266,
+            0.010676419362425804,
+            -0.026930073276162148,
+            0.0032046963460743427,
+            -0.005621497984975576,
+            -0.0028793576639145613,
+            -0.004366619978100061,
+            -0.007631072774529457,
+            -0.006891867611557245,
+            -0.018785540014505386,
+            -0.005431163590401411,
+            0.012473528273403645,
+            0.006834324914962053,
+            0.006236764136701822,
+            -0.028311101719737053,
+            -0.009587530978024006,
+            -0.015270997770130634,
+            -0.017103517428040504,
+            -0.00213904632255435,
+            -0.031055456027388573,
+            -0.03293224051594734,
+            -0.017678946256637573,
+            -0.014784096740186214,
+            -6.719100929331034e-05,
+            -0.004089971538633108,
+            -0.01467786356806755,
+            0.0022884365171194077,
+            0.009684910997748375,
+            0.01335880346596241,
+            0.0109862657263875,
+            0.0015591910341754556,
+            -0.022326640784740448,
+            0.012553202919661999,
+            -0.017555007711052895,
+            -0.023672260344028473,
+            0.004603431094437838,
+            0.011313817463815212,
+            -0.0026204148307442665,
+            -0.0018491186201572418,
+            0.004492771811783314,
+            0.006050856318324804,
+            0.005223123822361231,
+            -0.044830337166786194,
+            -0.009525561705231667,
+            0.0070423646830022335,
+            0.0040302155539393425,
+            -0.0049708206206560135,
+            -0.013403067365288734,
+            0.012066301889717579,
+            -0.01659005880355835,
+            -0.027886170893907547,
+            -0.009906229563057423,
+            -0.016510384157299995,
+            0.002213188214227557,
+            -0.009321948513388634,
+            -0.0011386852711439133,
+            0.03220631182193756,
+            0.017448775470256805,
+            0.005417884327471256,
+            0.0003460872976575047,
+            0.003917342983186245,
+            0.008715534582734108,
+            -0.010481659322977066,
+            0.0014363591326400638,
+            0.008299455046653748,
+            0.01290731318295002,
+            0.005878227762877941,
+            0.0020394527819007635,
+            -0.013341098092496395,
+            0.025584453716874123,
+            0.01467786356806755,
+            0.01010984368622303,
+            0.0006910680094733834,
+            0.008109121583402157,
+            0.006732518319040537,
+            0.023760788142681122,
+            0.0040589869022369385,
+            0.00846765749156475,
+            -0.007936492562294006,
+            0.0068166195414960384,
+            -0.013562416657805443,
+            -0.021441366523504257,
+            0.015359525568783283,
+            -0.0060729882679879665,
+            -0.03436638414859772,
+            0.02032591961324215,
+            -0.007077775429934263,
+            0.002996656810864806,
+            -0.008866031654179096,
+            -0.011375786736607552,
+            -0.00276205874979496,
+            0.017032695934176445,
+            0.0061659421771764755,
+            -0.006086267530918121,
+            0.0008327120449393988,
+            0.0073212264105677605,
+            -0.0027155817952007055,
+            -0.018838657066226006,
+            -0.0022076552268117666,
+            -0.004240468610078096,
+            0.004306863993406296,
+            -0.053683090955019,
+            -0.016811376437544823,
+            0.012482380494475365,
+            -0.009003248997032642,
+            -0.01747533306479454,
+            0.009401623159646988,
+            -0.008848326280713081,
+            -0.018767835572361946,
+            0.004687532316893339,
+            0.008812914602458477,
+            0.002602709224447608,
+            0.0020660110749304295,
+            0.007502707652747631,
+            0.021405955776572227,
+            0.01153513602912426,
+            -0.016722848638892174,
+            0.0021600716281682253,
+            -0.012677141465246677,
+            -0.000435168098192662,
+            -0.019830165430903435,
+            -0.0169353149831295,
+            -0.00480261817574501,
+            0.010437395423650742,
+            0.04479492828249931,
+            0.013491595163941383,
+            0.010685272514820099,
+            -0.0007209460018202662,
+            0.0005815151962451637,
+            0.001700835069641471,
+            -0.04139547049999237,
+            -0.0019719505216926336,
+            -0.005165581125766039,
+            -0.005692319944500923,
+            -0.010835768654942513,
+            0.012951577082276344,
+            0.028169458732008934,
+            0.02868291735649109,
+            -0.002775337779894471,
+            0.0073831956833601,
+            -2.913454591180198e-05,
+            0.00390185066498816,
+            0.008056004531681538,
+            -0.013075515627861023,
+            0.015085089951753616,
+            -0.0016012416454032063,
+            0.009791144169867039,
+            0.009782291017472744,
+            -0.0008526307065039873,
+            0.014491955749690533,
+            -0.015846427530050278,
+            0.010764947161078453,
+            -0.00964064709842205,
+            0.004149727523326874,
+            0.008998822420835495,
+            0.03764190524816513,
+            0.002179990289732814,
+            -0.01659005880355835,
+            -0.01566937193274498,
+            -0.030063947662711143,
+            0.029692132025957108,
+            -0.013270275667309761,
+            0.016997285187244415,
+            0.017245162278413773,
+            -0.0015647240215912461,
+            0.004930983297526836,
+            0.010463953018188477,
+            0.0017318197060376406,
+            0.011375786736607552,
+            0.014509661123156548,
+            0.00969376415014267,
+            -0.023300444707274437,
+            -0.02013115957379341,
+            0.001953138504177332,
+            -0.015262145549058914,
+            -0.014341458678245544,
+            -0.007679762784391642,
+            -0.011189878918230534,
+            0.0072326986119151115,
+            -0.014854918234050274,
+            0.021724654361605644,
+            -0.016829082742333412,
+            0.0049752467311918736,
+            -0.02944425493478775,
+            -0.006621859036386013,
+            0.0034459338057786226,
+            -0.012066301889717579,
+            -0.00028384136385284364,
+            -0.008308308199048042,
+            -0.002790830098092556,
+            0.017740916460752487,
+            -0.004415310453623533,
+            -0.003992591518908739,
+            0.011136762797832489,
+            0.006701533682644367,
+            0.010100990533828735,
+            -0.015642814338207245,
+            0.005550675559788942,
+            -0.0037646330893039703,
+            0.014005054719746113,
+            -0.018679307773709297,
+            -0.004740648902952671,
+            0.0027664850931614637,
+            0.024327363818883896,
+            0.012889607809484005,
+            0.008941279724240303,
+            0.005895933136343956,
+            -0.007852391339838505,
+            -0.009136040695011616,
+            -0.03739402815699577,
+            0.030860694125294685,
+            -0.016147419810295105,
+            0.022822394967079163,
+            -0.003609709907323122,
+            0.0025119686033576727,
+            -0.0006523372139781713,
+            0.00687416223809123,
+            0.0042448947206139565,
+            -0.021494483575224876,
+            0.013102073222398758,
+            -0.0231942106038332,
+            0.008184369653463364,
+            0.003359619528055191,
+            -0.012588613666594028,
+            0.03693368285894394,
+            -0.002405735431239009,
+            0.014784096740186214,
+            -0.02855897881090641,
+            0.0024079487193375826,
+            0.009764585644006729,
+            -0.005590513348579407,
+            -0.005453295540064573,
+            -0.02942654862999916,
+            -0.0010999544756487012,
+            0.0009915082482621074,
+            0.00035881312214769423,
+            0.006944984197616577,
+            -0.01922817900776863,
+            -0.03128562867641449,
+            0.008184369653463364,
+            0.008688976988196373,
+            0.002562871901318431,
+            0.003731435164809227,
+            -0.0001748418580973521,
+            -0.0018535450799390674,
+            0.014987709932029247,
+            0.01242926437407732,
+            0.019706226885318756,
+            0.03284371271729469,
+            -0.02862980216741562,
+            -0.015713635832071304,
+            -0.002653612522408366,
+            -0.016368739306926727,
+            -0.02569068782031536,
+            -0.02222040854394436,
+            0.0044706398621201515,
+            0.0073212264105677605,
+            0.013562416657805443,
+            -0.030896106734871864,
+            0.013287981040775776,
+            -0.005621497984975576,
+            0.004984099417924881,
+            -0.03032952919602394,
+            -0.026345791295170784,
+            -0.006254469510167837,
+            -0.011765307746827602,
+            0.03208237513899803,
+            -0.010021315887570381,
+            0.004275879357010126,
+            -0.01522673387080431,
+            -0.013456183485686779,
+            -0.00711761275306344,
+            0.0019420725293457508,
+            -0.01335880346596241,
+            -0.012296473607420921,
+            0.038243889808654785,
+            -0.001566937193274498,
+            -0.014598188921809196,
+            0.019723933190107346,
+            0.005546249449253082,
+            0.005501985549926758,
+            0.0040943981148302555,
+            0.013252570293843746,
+            -0.008441099897027016,
+            -0.006404966581612825,
+            -0.0004611730691976845,
+            0.007299094460904598,
+            0.002870504977181554,
+            0.024699179455637932,
+            -0.0042515345849096775,
+            2.7509237042977475e-05,
+            -0.007728452794253826,
+            0.0005483173299580812,
+            0.01844913698732853,
+            0.005422310903668404,
+            -0.007051217369735241,
+            -0.0025717245880514383,
+            0.005298372358083725,
+            -0.03227713704109192,
+            -0.017749767750501633,
+            0.01713007688522339,
+            0.010065579786896706,
+            0.009950493462383747,
+            -0.00028065990773029625,
+            -0.03149809315800667,
+            -0.007901081815361977,
+            0.002615988254547119,
+            0.0026624652091413736,
+            -0.0015160338953137398,
+            0.018820952624082565,
+            -0.027301888912916183,
+            0.002101422054693103,
+            -0.0032511733006685972,
+            0.02108725719153881,
+            0.008445525541901588,
+            0.0005848349537700415,
+            0.024504419416189194,
+            0.0036561868619173765,
+            0.013704060576856136,
+            -0.0018258801428601146,
+            -0.010322309099137783,
+            -0.01747533306479454,
+            -0.010003610514104366,
+            -3.72092254110612e-05,
+            0.0010119802318513393,
+            0.001832519774325192,
+            -0.00021232773724477738,
+            0.0015082877362146974,
+            -0.0038885714020580053,
+            -0.0011630303924903274,
+            0.015970366075634956,
+            0.0014817294431850314,
+            -0.004446295090019703,
+            0.014111286960542202,
+            -0.002376964082941413,
+            -0.007631072774529457,
+            0.006838751491159201,
+            0.0003211889124941081,
+            0.003240107325837016,
+            0.009224568493664265,
+            0.011844982393085957,
+            -0.0016045613447204232,
+            0.020715441554784775,
+            -0.010472806170582771,
+            0.008480937220156193,
+            -0.008888163603842258,
+            -0.0004888379480689764,
+            -0.017519596964120865,
+            0.0034060964826494455,
+            -0.002910342300310731,
+            -0.01423522550612688,
+            -0.020485268905758858,
+            0.006931704934686422,
+            0.006577595137059689,
+            0.00115639076102525,
+            -0.003638481255620718,
+            0.014872624538838863,
+            -0.0038332417607307434,
+            0.0045591676607728004,
+            -0.004886719398200512,
+            -0.006958263460546732,
+            -0.016439560800790787,
+            0.01518247090280056,
+            0.0015492317033931613,
+            0.005307225044816732,
+            0.002216507913544774,
+            0.0034038834273815155,
+            -0.00872881431132555,
+            0.001222786377184093,
+            0.017254015430808067,
+            0.005462148226797581,
+            -0.009994757361710072,
+            0.004408670589327812,
+            -0.009224568493664265,
+            -0.004098824225366116,
+            0.0075115603394806385,
+            -0.009339653886854649,
+            -0.01057018619030714,
+            0.02678842842578888,
+            -0.005355915054678917,
+            -0.0005236956640146673,
+            -0.012659436091780663,
+            -0.004220549948513508,
+            0.0115971053019166,
+            -0.000843777961563319,
+            -0.009251126088202,
+            -0.004705238156020641,
+            0.008958985097706318,
+            -0.01152628380805254,
+            -0.003518969053402543,
+            0.014580483548343182,
+            -0.009853113442659378,
+            -0.009560972452163696,
+            0.02021968737244606,
+            0.0026890235021710396,
+            -0.011650222353637218,
+            -0.006834324914962053,
+            0.0004075032484252006,
+            -0.005581660196185112,
+            0.03217090293765068,
+            0.006772355642169714,
+            0.00970261637121439,
+            0.023601438850164413,
+            0.001833626301959157,
+            0.009543267078697681,
+            -0.0042161233723163605,
+            0.012615172192454338,
+            0.0036716791801154613,
+            0.00873766653239727,
+            0.002321634441614151,
+            0.021724654361605644,
+            -0.0025009026285260916,
+            0.0042161233723163605,
+            0.007772716693580151,
+            -0.011482019908726215,
+            -0.004132022149860859,
+            -0.004036854952573776,
+            -0.005453295540064573,
+            -0.00089689448941499,
+            0.0005737690371461213,
+            0.010880032554268837,
+            0.009950493462383747,
+            -0.0015060745645314455,
+            0.015864131972193718,
+            0.010428542271256447,
+            -0.01422637328505516,
+            -0.0011630303924903274,
+            0.0050682006403803825,
+            0.00449055852368474,
+            0.0009666099213063717,
+            -0.030595112591981888,
+            -0.01291616540402174,
+            0.003156006336212158,
+            -0.015563138760626316,
+            0.015864131972193718,
+            -0.019847871735692024,
+            -0.02379619888961315,
+            -0.019670816138386726,
+            0.0038841450586915016,
+            -0.015005415305495262,
+            -0.01374832447618246,
+            0.021795475855469704,
+            0.007839112542569637,
+            0.0015768965240567923,
+            -0.02122890017926693,
+            0.023442087695002556,
+            -0.009808849543333054,
+            0.023229623213410378,
+            -0.01200433261692524,
+            -0.002098102355375886,
+            -0.011765307746827602,
+            0.01561625488102436,
+            -0.010897737927734852,
+            0.0025163949467241764,
+            0.009516708552837372,
+            0.01659005880355835,
+            0.0030652654822915792,
+            0.0007928746636025608,
+            0.0017882559914141893,
+            0.00480261817574501,
+            0.014438839629292488,
+            0.006834324914962053,
+            0.009410476312041283,
+            -0.01249123364686966,
+            -0.010242634452879429,
+            0.004078905563801527,
+            0.005931344348937273,
+            -0.00015644473023712635,
+            -0.02207876369357109,
+            0.002936900593340397,
+            0.009897377341985703,
+            -0.0005375280743464828,
+            -0.01839601993560791,
+            -0.006210205610841513,
+            -0.009534413926303387,
+            0.027921581640839577,
+            0.0097291748970747,
+            -0.01855536922812462,
+            -0.008950132876634598,
+            0.021565305069088936,
+            0.012411559000611305,
+            0.015235587023198605,
+            -0.016165126115083694,
+            0.013668649829924107,
+            0.008264044299721718,
+            0.0013511513825505972,
+            0.0029546061996370554,
+            -0.01053477544337511,
+            0.028824562206864357,
+            0.0048822928220033646,
+            0.0009881885489448905,
+            -0.013491595163941383,
+            -0.007538118865340948,
+            -0.008264044299721718,
+            0.011455461382865906,
+            0.005957902409136295,
+            0.002804109361022711,
+            0.001394308521412313,
+            -0.007985183037817478,
+            -0.015368377789855003,
+            -0.0019985088147222996,
+            -0.003357406472787261,
+            0.0211934894323349,
+            -0.0032179756090044975,
+            0.003782338462769985,
+            -0.006152662914246321,
+            0.0018004284938797355,
+            -0.009968198835849762,
+            -0.0057188780046999454,
+            -0.0020803967490792274,
+            0.00804272573441267,
+            -0.018218964338302612,
+            0.015979217365384102,
+            -0.020857084542512894,
+            0.018006497994065285,
+            0.028275690972805023,
+            0.00324232061393559,
+            -0.006785634905099869,
+            -0.002070437418296933,
+            0.008472084067761898,
+            -0.034649673849344254,
+            0.003306502941995859,
+            0.017696652561426163,
+            -0.006909573450684547,
+            -0.013615533709526062,
+            -0.018856363371014595,
+            -0.0012172535061836243,
+            -0.007927639409899712,
+            0.005563954822719097,
+            0.005856095813214779,
+            -0.0010518176713958383,
+            0.018661601468920708,
+            -0.002057158388197422,
+            -0.01467786356806755,
+            -0.008131253533065319,
+            -0.0221495870500803,
+            0.01480180211365223,
+            -0.012252209708094597,
+            -0.013465036638081074,
+            0.012818785384297371,
+            0.009746880270540714,
+            -0.019936397671699524,
+            0.002547379583120346,
+            -0.02747894451022148,
+            -0.008480937220156193,
+            -0.005369194317609072,
+            -0.01014525443315506,
+            -0.011685633100569248,
+            -0.004138661548495293,
+            -0.011765307746827602,
+            0.003943901043385267,
+            0.0002846713177859783,
+            -0.011986627243459225,
+            -0.004607857670634985,
+            0.007153023965656757,
+            -0.012287620455026627,
+            -0.009295389987528324,
+            0.009746880270540714,
+            0.004368833266198635,
+            -0.019121944904327393,
+            -0.031657442450523376,
+            0.008657991886138916,
+            -0.03337487578392029,
+            0.004306863993406296,
+            0.015492316335439682,
+            0.004939835984259844,
+            -0.007276962511241436,
+            0.003184777684509754,
+            0.021972531452775,
+            0.014854918234050274,
+            -0.006719239056110382,
+            0.017103517428040504,
+            0.0036296285688877106,
+            0.006617432460188866,
+            -0.012606319040060043,
+            -1.0452126844029408e-05,
+            0.01843143068253994,
+            0.012225651182234287,
+            -0.0028749313205480576,
+            0.01335880346596241,
+            -0.006883014924824238,
+            0.008609301410615444,
+            -0.00020900795061606914,
+            -0.007670910097658634,
+            -0.018661601468920708,
+            -0.006342997308820486,
+            -0.015474610961973667,
+            -0.006095120217651129,
+            0.008427820168435574,
+            -0.005993313621729612,
+            -0.006312012672424316,
+            -0.010605597868561745,
+            0.010286898352205753,
+            -0.017484186217188835,
+            0.00045508681796491146,
+            -0.02572609856724739,
+            -0.00300329620949924,
+            -0.013659796677529812,
+            0.009392770007252693,
+            0.01156169455498457,
+            0.005687893368303776,
+            -0.015749046579003334,
+            -0.019599994644522667,
+            0.0007768290233798325,
+            -0.0029789512045681477,
+            -0.021529894322156906,
+            -0.007684188894927502,
+            -0.018502252176404,
+            -0.012376148253679276,
+            -0.009437033906579018,
+            0.014075876213610172,
+            0.003014362184330821,
+            -0.03256042301654816,
+            0.006205779500305653,
+            -0.027018601074814796,
+            -0.016846787184476852,
+            -0.01201318483799696,
+            -0.0021047419868409634,
+            0.0046565476804971695,
+            0.020113453269004822,
+            0.008516347967088223,
+            0.026575962081551552,
+            -3.7174642784520984e-05,
+            0.005205418448895216,
+            0.0012891821097582579,
+            0.011933510191738605,
+            0.009817702695727348,
+            -0.012323031201958656,
+            0.01620939001441002,
+            -0.008356998674571514,
+            0.0020128944888710976,
+            -0.022716162726283073,
+            -0.0012936084531247616,
+            -0.014438839629292488,
+            -0.006281028036028147,
+            -0.000598667364101857,
+            -0.01834290288388729,
+            -0.006294306833297014,
+            0.023743081837892532,
+            -0.019051123410463333,
+            0.008998822420835495,
+            0.008361424319446087,
+            -0.0003070798411499709,
+            0.01467786356806755,
+            0.029816070571541786,
+            -0.010915443301200867,
+            0.001500541577115655,
+            0.01944064348936081,
+            0.004315716680139303,
+            0.008503069169819355,
+            -0.0008183263125829399,
+            0.0036207756493240595,
+            0.012084007263183594,
+            -0.01155284233391285,
+            0.00648906733840704,
+            -0.005807405803352594,
+            0.008423393592238426,
+            -0.011269553564488888,
+            0.0041585806757211685,
+            -0.01949376054108143,
+            -0.01601462997496128,
+            0.019918693229556084,
+            0.012898460030555725,
+            -0.0003311482723802328,
+            -0.0051036118529737,
+            -0.03233025223016739,
+            -0.0008686763467267156,
+            -0.009817702695727348,
+            0.007431885693222284,
+            -0.013571269810199738,
+            0.011304965242743492,
+            -0.00642267195507884,
+            0.014146698638796806,
+            -0.013677502982318401,
+            0.018856363371014595,
+            -0.011269553564488888,
+            -0.01062330324202776,
+            0.017121223732829094,
+            0.009755733422935009,
+            -0.0037557801697403193,
+            0.029231788590550423,
+            -0.001606774516403675,
+            0.0029501798562705517,
+            -0.002301715547218919,
+            -0.030878400430083275,
+            0.0023747507948428392,
+            0.0028882105834782124,
+            0.00918030459433794,
+            0.004149727523326874,
+            -0.005820684600621462,
+            0.030701344832777977,
+            0.012154828757047653,
+            0.016997285187244415,
+            -0.011437756009399891,
+            -0.0008044939022511244,
+            0.020998729392886162,
+            0.013615533709526062,
+            0.0015481250593438745,
+            0.007400901056826115,
+            -0.0008199862204492092,
+            0.0007674229564145207,
+            -0.021777771413326263,
+            0.010286898352205753,
+            -0.018732424825429916,
+            -0.010428542271256447,
+            -0.002987803891301155,
+            -0.003027641447260976,
+            -0.003027641447260976,
+            0.0010352187091484666,
+            0.00528509309515357,
+            0.0015591910341754556,
+            0.023654554039239883,
+            -0.0009311988833360374,
+            -0.0037491407711058855,
+            -0.028134047985076904,
+            -0.017864855006337166,
+            0.003386177821084857,
+            0.011322670616209507,
+            0.029090143740177155,
+            0.0024920497089624405,
+            -0.009198009967803955,
+            -0.008330440148711205,
+            -0.012615172192454338,
+            0.01782059110701084,
+            0.010773799382150173,
+            -0.0025186080019921064,
+            0.0017229669028893113,
+            -0.011136762797832489,
+            0.014660158194601536,
+            -0.014607041142880917,
+            0.011127909645438194,
+            0.014120140112936497,
+            0.0019420725293457508,
+            -0.01249123364686966,
+            -0.0005477640661410987,
+            0.003862013341858983,
+            0.012145976535975933,
+            0.002469917992129922,
+            0.009171451441943645,
+            -0.02592085860669613,
+            -0.00666169635951519,
+            0.005679040681570768,
+            0.006648417096585035,
+            0.02296403981745243,
+            0.0152444401755929,
+            0.011225289665162563,
+            0.01108364574611187,
+            0.012349589727818966,
+            0.008600449189543724,
+            0.015005415305495262,
+            -0.013881116174161434,
+            0.0025938565377146006,
+            -0.0019044483778998256,
+            0.016492677852511406,
+            0.007812554016709328,
+            0.010729536414146423,
+            -0.02029050886631012,
+            0.0032069096341729164,
+            0.013429625891149044,
+            -0.008817341178655624,
+            0.008551758714020252,
+            -0.008410114794969559,
+            0.01378373522311449,
+            -0.005448868963867426,
+            0.0009323054691776633,
+            -0.0012393853394314647,
+            0.0030873974319547415,
+            0.032525014132261276,
+            -0.004218336660414934,
+            -0.004811470862478018,
+            0.01570478267967701,
+            -0.0045591676607728004,
+            0.0028240280225872993,
+            -0.01525329239666462,
+            0.00879078358411789,
+            0.02480541169643402,
+            0.00883504655212164,
+            -0.016253653913736343,
+            -0.014483102597296238,
+            0.004961967933923006,
+            -0.009321948513388634,
+            0.0011198731372132897,
+            0.007772716693580151,
+            -0.004424163140356541,
+            0.009321948513388634,
+            -0.0013799227308481932,
+            -0.0008244125638157129,
+            0.0049708206206560135,
+            -0.039164576679468155,
+            0.00252303434535861,
+            -0.015350672416388988,
+            0.006103972904384136,
+            0.004452934488654137,
+            -0.0024787706788629293,
+            -0.006462509278208017,
+            -0.006555463187396526,
+            -0.001566937193274498,
+            -0.05403720214962959,
+            0.0032201886642724276,
+            -0.0014573843218386173,
+            0.02393784187734127,
+            -0.01062330324202776,
+            0.00969376415014267,
+            0.015014268457889557,
+            0.013093221001327038,
+            -0.003078544745221734,
+            -0.0012659436324611306,
+            0.013411919586360455,
+            -0.0211934894323349,
+            -0.018626190721988678,
+            0.014837212860584259,
+            -0.007608940824866295,
+            0.00011750645353458822,
+            -0.004050134215503931,
+            0.004798191599547863,
+            -0.004634415730834007,
+            0.0033397008664906025,
+            0.013270275667309761,
+            -0.012730257585644722,
+            -0.01924588344991207,
+            -0.007515986915677786,
+            -0.006338570732623339,
+            -0.019511466845870018,
+            -0.016528088599443436,
+            0.00621463218703866,
+            -0.01714778132736683,
+            0.003678318578749895,
+            -0.0005380813381634653,
+            -0.00555510213598609,
+            0.013633239082992077,
+            0.001355577725917101,
+            0.0020538384560495615,
+            0.0007336718845181167,
+            0.009233420714735985,
+            -0.017431069165468216,
+            0.017652388662099838,
+            0.011216437444090843,
+            -0.014483102597296238,
+            0.00177940318826586,
+            -0.004669826943427324,
+            0.0026646784972399473,
+            0.021494483575224876,
+            -0.0065421839244663715,
+            0.0021224473603069782,
+            0.018767835572361946,
+            -0.0013876688899472356,
+            0.011198732070624828,
+            -0.003757993457838893,
+            0.005298372358083725,
+            -0.012101712636649609,
+            0.0012504513142630458,
+            0.00648906733840704,
+            -0.007812554016709328,
+            0.009065218269824982,
+            -0.04504280537366867,
+            0.012216798029839993,
+            -0.009144892916083336,
+            -0.024451302364468575,
+            -0.006059709005057812,
+            -0.0063164387829601765,
+            -0.0006456976407207549,
+            -0.01653694175183773,
+            0.013137484900653362,
+            -0.011216437444090843,
+            0.006745797581970692,
+            -0.012075154110789299,
+            0.0019885494839400053,
+            0.007839112542569637,
+            0.0003709856537170708,
+            0.002027280395850539,
+            -0.006068561691790819,
+            0.003687171498313546,
+            0.004926556721329689,
+            -0.002126873703673482,
+            -0.008989970199763775,
+            0.013544711284339428,
+            -0.0034171624574810266,
+            0.015952659770846367,
+            0.010756094008684158,
+            -0.008454378694295883,
+            -0.024681473150849342,
+            0.0157224889844656,
+            -0.004678679630160332,
+            -0.008941279724240303,
+            -0.0021556452848017216,
+            0.010065579786896706,
+            -3.074256528634578e-05,
+            -0.008392409421503544,
+            0.0027532060630619526,
+            0.009879671968519688,
+            0.007259257137775421,
+            -0.008967838250100613,
+            -0.004687532316893339,
+            -0.015456905588507652,
+            0.009525561705231667,
+            0.002750992774963379,
+            0.007462870329618454,
+            0.013898821547627449,
+            0.004132022149860859,
+            0.013376508839428425,
+            0.0004337848804425448,
+            -0.00759123545140028,
+            -0.026983190327882767,
+            -0.015766752883791924,
+            -0.010950854979455471,
+            0.0014153338270261884,
+            -0.0011785227106884122,
+            0.02018427476286888,
+            -0.0051611545495688915,
+            0.0046919588930904865,
+            0.0074540176428854465,
+            0.042032867670059204,
+            0.014182109385728836,
+            0.007887802086770535,
+            0.0007380982278846204,
+            -0.013819146901369095,
+            0.00038481809315271676,
+            -0.010463953018188477,
+            -0.03153350576758385,
+            0.009897377341985703,
+            -0.004129808861762285,
+            0.02315879985690117,
+            -0.014819507487118244,
+            0.014589335769414902,
+            -0.004510477185249329,
+            -0.01808617264032364,
+            -0.016944168135523796,
+            0.020626913756132126,
+            0.022840101271867752,
+            0.016377592459321022,
+            4.270761564839631e-05,
+            0.019511466845870018,
+            -0.007055643480271101,
+            0.0161562729626894,
+            -0.020113453269004822,
+            -0.004576873034238815,
+            0.006555463187396526,
+            0.02121119573712349,
+            0.0013467249227687716,
+            0.012765669263899326,
+            0.00711318664252758,
+            -0.0012825424782931805,
+            -0.001180735882371664,
+            -0.012703699991106987,
+            0.0019011285621672869,
+            -0.005174433812499046,
+            -0.0020471990574151278,
+            0.013252570293843746,
+            -0.01009213738143444,
+            -0.01837831363081932,
+            0.01149087306112051,
+            -0.010003610514104366,
+            0.0032622392755001783,
+            0.011234142817556858,
+            0.022344347089529037,
+            0.005515264812856913,
+            0.010880032554268837,
+            -0.006551036611199379,
+            8.638350118417293e-05,
+            -0.022397464141249657,
+            -0.01004787441343069,
+            -0.004138661548495293,
+            -0.01114561501890421,
+            0.003333061235025525,
+            -0.007586808875203133,
+            -0.025495927780866623,
+            0.00041497277561575174,
+            0.007458443753421307,
+            0.030099358409643173,
+            -0.013597827404737473,
+            0.02112266793847084,
+            0.0011148934718221426,
+            -0.0029501798562705517,
+            -0.010649860836565495,
+            -0.01425293181091547,
+            0.00576756801456213,
+            -0.01195121556520462,
+            -0.0014429986476898193,
+            6.57731870887801e-05,
+            -0.01830749213695526,
+            -3.8903697713976726e-05,
+            0.014005054719746113,
+            -0.037323202937841415,
+            -0.003957180306315422,
+            0.004992952570319176,
+            -0.0009284323896281421,
+            0.01387226302176714,
+            0.005643629468977451,
+            0.0036296285688877106,
+            0.0027554191183298826,
+            -0.003492410760372877,
+            0.00022380864538718015,
+            0.01929900050163269,
+            -0.015784457325935364,
+            0.02843504026532173,
+            0.00831273477524519,
+            -0.004891145508736372,
+            -0.0008930214098654687,
+            0.004503837786614895,
+            -0.028753740713000298,
+            0.010897737927734852,
+            -0.015952659770846367,
+            -0.018643897026777267,
+            0.011313817463815212,
+            0.015138207003474236,
+            -0.0025562322698533535,
+            0.002536313608288765,
+            0.015899542719125748,
+            0.011384639889001846,
+            0.005231976509094238,
+            -0.0016753834206610918,
+            -0.014146698638796806,
+            0.00203502643853426,
+            -0.01936982199549675,
+            0.009074071422219276,
+            -0.020024925470352173,
+            -0.0036318416241556406,
+            0.00849421601742506,
+            0.0038022571243345737,
+            0.015811016783118248,
+            0.009268831461668015,
+            0.018112732097506523,
+            -0.015359525568783283,
+            0.00891914777457714,
+            0.03973115235567093,
+            0.01147316675633192,
+            0.013677502982318401,
+            -0.005856095813214779,
+            -0.0005881547695025802,
+            0.025460515171289444,
+            -0.005262961145490408,
+            -0.008414541371166706,
+            0.020502975210547447,
+            0.018909478560090065,
+            0.014819507487118244,
+            -0.001221679849550128,
+            0.009065218269824982,
+            0.029214082285761833,
+            0.00552411749958992,
+            -0.005192139185965061,
+            -0.019511466845870018,
+            -0.006373981945216656,
+            -0.0020903560798615217,
+            0.013164042495191097,
+            0.007166303228586912,
+            0.016846787184476852,
+            -0.028966205194592476,
+            0.0034702790435403585,
+            0.003501263679936528,
+            -0.0001507734414190054,
+            0.013376508839428425,
+            0.02021968737244606,
+            0.003344127209857106,
+            -0.003842094447463751,
+            0.006493493914604187,
+            0.005174433812499046,
+            0.03125021606683731,
+            -0.025283461436629295,
+            -0.0037270088214427233,
+            0.0034038834273815155,
+            -0.015279850922524929,
+            0.013606680557131767,
+            -0.004523756448179483,
+            -0.002558445557951927,
+            -0.007852391339838505,
+            -0.010490511544048786,
+            0.016713997349143028,
+            0.01419096253812313,
+            -0.015456905588507652,
+            -0.025177227333188057,
+            -0.02117578312754631,
+            0.007122039329260588,
+            -0.02108725719153881,
+            -0.0072858151979744434,
+            -0.01009213738143444,
+            -0.006387260742485523,
+            -0.009578677825629711,
+            -0.006475788541138172,
+            -0.009746880270540714,
+            0.005563954822719097,
+            0.0007120932568795979,
+            0.003638481255620718,
+            0.0007353317341767251,
+            -0.01377488300204277,
+            -0.011136762797832489,
+            0.021671537309885025,
+            0.004860160872340202,
+            0.01296928245574236,
+            0.0026071355678141117,
+            0.005231976509094238,
+            0.006250043399631977,
+            -0.04298896715044975,
+            -0.006050856318324804,
+            -0.006679401732981205,
+            0.009357359260320663,
+            0.005563954822719097,
+            0.022716162726283073,
+            -0.012110565789043903,
+            0.015766752883791924,
+            -0.010437395423650742,
+            0.010039021261036396,
+            -0.010242634452879429,
+            0.0026646784972399473,
+            0.008215353824198246,
+            -0.007467296905815601,
+            0.0016565712867304683,
+            -0.004607857670634985,
+            0.007679762784391642,
+            -0.0018391592893749475,
+            0.007608940824866295,
+            -0.00960523635149002,
+            -0.015483464114367962,
+            -0.0023526190780103207,
+            -0.011605958454310894,
+            -0.017864855006337166,
+            5.754288940806873e-05,
+            -0.008587169460952282,
+            -0.0015315262135118246,
+            -0.019688520580530167,
+            -0.01924588344991207,
+            0.011960068717598915,
+            -0.0020615847315639257,
+            0.009091776795685291,
+            0.013916526921093464,
+            0.007763864006847143,
+            0.015306408517062664,
+            0.013376508839428425,
+            -0.004452934488654137,
+            -0.013571269810199738,
+            -0.014208667911589146,
+            0.0023526190780103207,
+            -0.014943446032702923,
+            0.01935211755335331,
+            0.008210928179323673,
+            -0.0017307130619883537,
+            0.0011884820414707065,
+            -0.020998729392886162,
+            -0.0001062330265995115,
+            -0.0022209342569112778,
+            0.0015569778624922037,
+            -0.018590779975056648,
+            0.02200794219970703,
+            -0.018289785832166672,
+            0.002113594673573971,
+            0.019688520580530167,
+            0.029780659824609756,
+            0.0028461599722504616,
+            -0.012925018556416035,
+            0.0007309053908102214,
+            0.008427820168435574,
+            0.03898752108216286,
+            -0.007985183037817478,
+            0.001860184594988823,
+            -0.011853835545480251,
+            -0.010313456878066063,
+            0.00435555400326848,
+            0.006365128792822361,
+            0.006812192965298891,
+            -0.026522846892476082,
+            -0.013597827404737473,
+            -0.010375426150858402,
+            -0.013695208355784416,
+            -0.0058118319138884544,
+            0.04072266072034836,
+            0.001315740286372602,
+            -0.0010070005664601922,
+            -0.00669268099591136,
+            0.00023556621454190463,
+            -0.017280573025345802,
+            0.03047117404639721,
+            0.09419328719377518,
+            -0.008817341178655624,
+            -0.009552120231091976,
+            -0.025035584345459938,
+            0.009198009967803955,
+            -0.001912194536998868,
+            0.011906951665878296,
+            0.06069447100162506,
+            -0.020662324503064156,
+            0.000730352068785578,
+            -0.0007264789892360568,
+            0.009897377341985703,
+            -0.00963179487735033,
+            -0.0026248411741107702,
+            -0.00735663715749979,
+            -0.008888163603842258,
+            0.0017605911707505584,
+            -0.03140956535935402,
+            0.0005306118400767446,
+            -0.006272174883633852,
+            0.005506412126123905,
+            -0.014244078658521175,
+            0.004581299610435963,
+            -0.010818063281476498,
+            0.004220549948513508,
+            -0.013677502982318401,
+            -0.005448868963867426,
+            0.01297813467681408,
+            -0.019688520580530167,
+            -0.0007790421950630844,
+            0.004647694993764162,
+            -0.012641730718314648,
+            0.008392409421503544,
+            0.0026248411741107702,
+            -0.011278406716883183,
+            -0.0015846426831558347,
+            -0.015085089951753616,
+            0.028789151459932327,
+            0.0018834230722859502,
+            0.007259257137775421,
+            -0.009950493462383747,
+            0.00101751321926713,
+            0.01293387170881033,
+            -0.0021711376029998064,
+            0.017121223732829094,
+            0.013571269810199738,
+            -0.001765017514117062,
+            -0.008808488957583904,
+            0.01714778132736683,
+            -0.00046089643728919327,
+            -0.013314539566636086,
+            0.009136040695011616,
+            0.011216437444090843,
+            0.0027266477700322866,
+            0.0037535671144723892,
+            -0.0015724701806902885,
+            0.023548321798443794,
+            -0.017254015430808067,
+            -0.011482019908726215,
+            -0.004072266165167093,
+            -0.009330800734460354,
+            0.021441366523504257,
+            0.006307586096227169,
+            0.03397686406970024,
+            -0.009773438796401024,
+            -0.0012305325362831354,
+            -0.018590779975056648,
+            -0.007259257137775421,
+            -0.010189518332481384,
+            -0.005993313621729612,
+            0.013553564436733723,
+            0.0019299000268802047,
+            0.012694846838712692,
+            -0.019971810281276703,
+            -0.017156634479761124,
+            0.00960523635149002,
+            -0.02294633351266384,
+            -0.0011375787435099483,
+            0.004922130145132542,
+            -0.027213361114263535,
+            -0.004435229115188122,
+            0.008166664279997349,
+            -0.02121119573712349,
+            -0.025230344384908676,
+            0.002651399467140436,
+            -0.0007049004198051989,
+            -0.02025509811937809,
+            0.009711469523608685,
+            0.006015445105731487,
+            -0.008565038442611694,
+            -0.002060478087514639,
+            -0.002961245598271489,
+            -0.020024925470352173,
+            0.0076399254612624645,
+            -0.015058532357215881,
+            -0.008038299158215523,
+            -0.006174794863909483,
+            0.004988525994122028,
+            -0.0012703699758276343,
+            0.02114037238061428,
+            0.006338570732623339,
+            -0.004032428842037916,
+            -0.011623663827776909,
+            0.0018280934309586883,
+            0.008472084067761898,
+            0.017138928174972534,
+            0.02285780757665634,
+            -0.0019442857010290027,
+            -0.006750223692506552,
+            0.001236065523698926,
+            0.015403789468109608,
+            0.005147875752300024,
+            -0.010791505686938763,
+            0.025354282930493355,
+            0.006785634905099869,
+            -0.008033872582018375,
+            0.015483464114367962,
+            -0.012668288312852383,
+            0.001964204479008913,
+            -0.009977051988244057,
+            -0.008219780400395393,
+            0.0042780926451087,
+            -0.0007159663364291191,
+            0.024238836020231247,
+            0.005581660196185112,
+            0.014571630395948887,
+            0.010419690050184727,
+            0.012774521484971046,
+            0.022432874888181686,
+            -0.01201318483799696,
+            0.0019874428398907185,
+            -0.0067280917428433895,
+            0.0018535450799390674,
+            -0.0005510838236659765,
+            -0.007684188894927502,
+            -0.006741371005773544,
+            0.0050283633172512054,
+            0.009445887058973312,
+            -0.0017561647109687328,
+            0.010348867624998093,
+            0.019192766398191452,
+            0.014828360639512539,
+            0.005391326267272234,
+            -0.0064093926921486855,
+            -0.019954103976488113,
+            0.003248960245400667,
+            0.021016433835029602,
+            -0.0034724920988082886,
+            0.010410836897790432,
+            0.0014175469987094402,
+            0.01855536922812462,
+            0.00669268099591136,
+            0.0050637745298445225,
+            -0.00666169635951519,
+            0.008742093108594418,
+            -0.01646612025797367,
+            0.004581299610435963,
+            0.0034437207505106926,
+            0.02206105925142765,
+            -0.007493854966014624,
+            -0.0067236656323075294,
+            0.0002651675895322114,
+            0.031728263944387436,
+            0.01384570449590683,
+            0.010782652534544468,
+            -0.019564582034945488,
+            0.009392770007252693,
+            -0.006705959793180227,
+            0.0015547646908089519,
+            0.0007773823454044759,
+            0.0115971053019166,
+            -0.007436312269419432,
+            -0.005621497984975576,
+            0.011756455525755882,
+            0.009233420714735985,
+            0.00918030459433794,
+            -0.0030453468207269907,
+            -0.017555007711052895,
+            0.006170368287712336,
+            -0.0047849128022789955,
+            -0.012137123383581638,
+            0.026522846892476082,
+            0.0157224889844656,
+            0.032879121601581573,
+            -0.003069691825658083,
+            0.0019619911909103394,
+            0.011402345262467861,
+            0.0018015351379290223,
+            0.014908035285770893,
+            -0.01011869590729475,
+            -0.0109862657263875,
+            0.03392374888062477,
+            -0.01246467512100935,
+            0.008135679177939892,
+            -0.024150308221578598,
+            -0.026469729840755463,
+            -0.0028129620477557182,
+            0.001937646185979247,
+            0.0049708206206560135,
+            0.0027952566742897034,
+            -0.004528183024376631,
+            -0.007338931784033775,
+            0.01296928245574236,
+            -0.001329019432887435,
+            0.0037668461445719004,
+            0.008976691402494907,
+            -0.0044418685138225555,
+            0.0031139557249844074,
+            -0.002733287401497364,
+            -0.000756910361815244,
+            0.010959707200527191
+          ],
+          "keyphrases": [
+            "expert choice method",
+            "auto-regressive text generation",
+            "large memory footprint",
+            "sparsely gated networks",
+            "power saving techniques"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "dea84165-aef9-4273-9988-e3d233346670",
+        "properties": {
+          "page_content": "A Comparison On Fine-Tuning With A Dense Model\n\nOur 8B MoE model achieves stronger pre-training perplexity than its dense counterpart. However, a better perplexity does not always directly translate to downstream performance as demonstrated in Section 4.4. To this end, we compare fine-tuning performance of the 8B dense model and MoE model in Table 1. As shown in the table, our MoE model using expert choice routing consistently outperforms the dense model across the 11 tasks in GLUE and SuperGLUE.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 1: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\n",
+          "title": "A Comparison On Fine-Tuning With A Dense Model",
+          "embedding": [
+            0.016482073813676834,
+            -0.002563994377851486,
+            -0.0267450250685215,
+            -0.023007972165942192,
+            0.025922315195202827,
+            0.018462155014276505,
+            -0.01591036096215248,
+            0.03547411039471626,
+            -0.012068727053701878,
+            -0.011176296509802341,
+            0.003149651922285557,
+            -0.008659363724291325,
+            -0.021000003442168236,
+            0.016844624653458595,
+            -0.02533665858209133,
+            0.032490044832229614,
+            0.024039844051003456,
+            0.01954980380833149,
+            0.010472113266587257,
+            -0.021557772532105446,
+            -0.0034128492698073387,
+            -0.011873507872223854,
+            -0.011252989992499352,
+            -0.022938251495361328,
+            0.01864342950284481,
+            -0.007376495283097029,
+            0.017639445140957832,
+            -0.010311754420399666,
+            -0.029952196404337883,
+            0.032490044832229614,
+            0.012884464114904404,
+            0.014557771384716034,
+            0.02639641799032688,
+            -0.006728088948875666,
+            -0.05795220285654068,
+            -0.010625500231981277,
+            0.0629163458943367,
+            0.01878287084400654,
+            -0.027177294716238976,
+            0.06062949076294899,
+            0.024179287254810333,
+            0.018978090956807137,
+            -0.004779383074492216,
+            -0.004329681862145662,
+            0.01492032129317522,
+            -0.030314745381474495,
+            -0.018476098775863647,
+            -0.0148645443841815,
+            0.03494422882795334,
+            -0.03650598227977753,
+            -0.010472113266587257,
+            -0.01666335016489029,
+            0.018657373264431953,
+            0.011469125747680664,
+            -0.0498366616666317,
+            0.0530996099114418,
+            0.006975598633289337,
+            0.01773705519735813,
+            0.04239044710993767,
+            0.005661355331540108,
+            -0.007543826010078192,
+            -0.0029038849752396345,
+            -0.026758968830108643,
+            -0.02917131967842579,
+            0.011315738782286644,
+            0.015966137871146202,
+            -0.01385358814150095,
+            -0.02132072113454342,
+            0.03890438750386238,
+            0.01121812965720892,
+            -0.05678088590502739,
+            0.05452192202210426,
+            -0.025239048525691032,
+            0.018099604174494743,
+            -0.009293826296925545,
+            -0.03438645973801613,
+            0.014320719987154007,
+            0.01945219375193119,
+            0.022770920768380165,
+            0.0009926545899361372,
+            -0.012201197445392609,
+            -0.010939245112240314,
+            -0.0014963897410780191,
+            0.009684264659881592,
+            0.03611554577946663,
+            -0.026758968830108643,
+            -0.017848609015345573,
+            -0.05254184082150459,
+            -0.02526693604886532,
+            -0.005563745740801096,
+            -0.006602590903639793,
+            -0.010318727232515812,
+            -0.011308766901493073,
+            0.020693231374025345,
+            0.007257969584316015,
+            0.042920324951410294,
+            -0.004256474785506725,
+            0.01883864775300026,
+            -0.02076295204460621,
+            -0.008903387933969498,
+            0.00044708672794513404,
+            0.03296414762735367,
+            0.018252991139888763,
+            0.004782869480550289,
+            0.04950200021266937,
+            -0.015394425019621849,
+            0.020288847386837006,
+            0.007181276101619005,
+            0.0038242037408053875,
+            0.06832670420408249,
+            -0.005229084752500057,
+            -0.021892433986067772,
+            -0.03126295283436775,
+            0.008854582905769348,
+            -0.014794823713600636,
+            -0.00372310820966959,
+            -0.02597809210419655,
+            -0.011713149026036263,
+            0.04013147950172424,
+            -0.027637455612421036,
+            0.0076762959361076355,
+            0.027972117066383362,
+            0.019033867865800858,
+            -0.03313148021697998,
+            -0.03466534614562988,
+            0.045011959969997406,
+            0.02066534198820591,
+            0.0014536854578182101,
+            0.008087650872766972,
+            -0.014237054623663425,
+            -0.001226220279932022,
+            0.012473110109567642,
+            -0.014188249595463276,
+            -0.02958964742720127,
+            0.030454188585281372,
+            0.0055742040276527405,
+            0.0256852637976408,
+            0.03028685785830021,
+            -0.03555777668952942,
+            -0.03354980796575546,
+            0.019745023921132088,
+            -0.05968128517270088,
+            0.016440242528915405,
+            0.011992033571004868,
+            -0.004497012589126825,
+            0.010904384776949883,
+            0.0532948300242424,
+            0.08717929571866989,
+            -0.013016934506595135,
+            -0.004096116404980421,
+            -0.02710757404565811,
+            0.027065740898251534,
+            0.004817729815840721,
+            0.036394428461790085,
+            -0.007495020981878042,
+            -0.010953188873827457,
+            -0.02603386901319027,
+            0.0032385464292019606,
+            -0.013477093540132046,
+            -0.02132072113454342,
+            0.011280878446996212,
+            0.008241036906838417,
+            -0.004657371435314417,
+            -0.008310758508741856,
+            0.0006462276796810329,
+            0.03182072192430496,
+            0.0277071762830019,
+            -0.007313746493309736,
+            -0.07591236382722855,
+            0.031681280583143234,
+            0.011057770811021328,
+            0.030454188585281372,
+            0.0072161369025707245,
+            0.020037852227687836,
+            0.019061755388975143,
+            0.02533665858209133,
+            0.006578188389539719,
+            -0.014976098202168941,
+            -0.030482076108455658,
+            0.024374505504965782,
+            -0.01294721383601427,
+            -0.027986060827970505,
+            0.041916340589523315,
+            -0.01662151701748371,
+            0.013023906387388706,
+            -0.0030241538770496845,
+            0.010855579748749733,
+            0.025294825434684753,
+            -0.001602714415639639,
+            0.017848609015345573,
+            -0.00970518123358488,
+            0.045820724219083786,
+            0.03028685785830021,
+            -0.01878287084400654,
+            -0.007780877873301506,
+            -0.039992038160562515,
+            -0.024025900289416313,
+            0.01833665743470192,
+            0.01068824902176857,
+            0.017346616834402084,
+            -0.005978586617857218,
+            -0.03592032566666603,
+            -0.006846614647656679,
+            -0.021460162475705147,
+            0.02320319041609764,
+            -0.0015730829909443855,
+            0.0327131524682045,
+            -0.0027417833916842937,
+            -0.003228088142350316,
+            -0.016175301745533943,
+            0.024388451129198074,
+            -0.023998012766242027,
+            -0.020177293568849564,
+            -0.021111557260155678,
+            -0.01812749356031418,
+            0.022715143859386444,
+            0.004165837541222572,
+            -0.002523904899135232,
+            -0.022450203076004982,
+            -0.0008218378061428666,
+            0.008729085326194763,
+            0.028557773679494858,
+            -0.00485259061679244,
+            -0.022464146837592125,
+            -0.05692032724618912,
+            -0.03388446941971779,
+            0.021111557260155678,
+            -0.02998008392751217,
+            0.022603590041399002,
+            -0.012668329291045666,
+            -0.022464146837592125,
+            0.009698208421468735,
+            0.003107819240540266,
+            0.02350996434688568,
+            -0.006414343602955341,
+            -0.02526693604886532,
+            0.018211157992482185,
+            -0.020274903625249863,
+            0.013260958716273308,
+            0.029254985973238945,
+            -0.017262950539588928,
+            0.014111556112766266,
+            -0.03525100275874138,
+            -0.01833665743470192,
+            0.03937849402427673,
+            -0.010486057959496975,
+            -0.006825698539614677,
+            -0.0072509972378611565,
+            0.006532869767397642,
+            0.0019382473547011614,
+            0.005790339782834053,
+            -0.037760961800813675,
+            0.0075089652091264725,
+            0.009795818477869034,
+            0.023523908108472824,
+            0.01970319077372551,
+            0.05895618721842766,
+            -0.04685259610414505,
+            -0.01579880714416504,
+            -0.03965737670660019,
+            0.05374104529619217,
+            0.010297810658812523,
+            -0.03965737670660019,
+            0.015868527814745903,
+            -0.020651398226618767,
+            -0.01768127828836441,
+            0.003187998663634062,
+            0.009942232631146908,
+            -0.014850599691271782,
+            0.033159367740154266,
+            0.01281474344432354,
+            -0.021599605679512024,
+            -0.04947411268949509,
+            0.0050024907104671,
+            0.004908367525786161,
+            -0.019466139376163483,
+            0.010841635055840015,
+            0.004577192012220621,
+            0.0018894425593316555,
+            0.0021927293855696917,
+            0.023677295073866844,
+            0.03257371112704277,
+            0.023886458948254585,
+            0.02978486567735672,
+            -0.006905877497047186,
+            0.0050756982527673244,
+            -0.02279880829155445,
+            -0.01053486205637455,
+            -0.03053785301744938,
+            -0.0037265941500663757,
+            -0.009342631325125694,
+            -0.020944226533174515,
+            -0.024695223197340965,
+            -0.005731076467782259,
+            -0.006975598633289337,
+            0.007578686345368624,
+            -0.010827691294252872,
+            0.005399901419878006,
+            0.04952988773584366,
+            -0.03561355173587799,
+            -0.043478094041347504,
+            -0.007045319769531488,
+            0.02031673677265644,
+            0.002699950709939003,
+            -0.05951395258307457,
+            0.008854582905769348,
+            0.018350601196289062,
+            0.0024506975896656513,
+            -0.02405378967523575,
+            -0.02391434647142887,
+            0.026117533445358276,
+            -0.008729085326194763,
+            -0.013023906387388706,
+            -0.053490050137043,
+            0.0010527890408411622,
+            0.005856574513018131,
+            0.0057973116636276245,
+            0.03184860944747925,
+            -0.03873705863952637,
+            -0.004577192012220621,
+            0.01949402689933777,
+            0.013937253504991531,
+            -0.07686056941747665,
+            0.014627492986619473,
+            -0.004148406907916069,
+            0.027874507009983063,
+            0.045458175241947174,
+            0.039183273911476135,
+            -0.01941036246716976,
+            -6.231325824046507e-05,
+            0.0032559765968471766,
+            -0.0519561842083931,
+            -0.029645422473549843,
+            0.007076694630086422,
+            0.006240040995180607,
+            -0.034776899963617325,
+            -0.0442589707672596,
+            -0.0073067741468548775,
+            -0.02589442767202854,
+            0.013637452386319637,
+            -0.011685260571539402,
+            0.009907372295856476,
+            -0.034776899963617325,
+            0.010095619596540928,
+            0.018824703991413116,
+            -0.009670319966971874,
+            -0.026354586705565453,
+            -0.03572510555386543,
+            0.01180378720164299,
+            -0.017095619812607765,
+            0.0007935136090964079,
+            0.021348608657717705,
+            0.006473606918007135,
+            0.005654383450746536,
+            -0.007515937555581331,
+            -0.03564143925905228,
+            -0.006836156360805035,
+            -0.005201196298003197,
+            0.020400401204824448,
+            0.0342470183968544,
+            -0.011176296509802341,
+            -0.009021913632750511,
+            -0.011036854237318039,
+            0.04741036519408226,
+            -0.05148208141326904,
+            -0.009265937842428684,
+            0.01864342950284481,
+            -0.006484064739197493,
+            0.008143427781760693,
+            0.005086156073957682,
+            -0.026675302535295486,
+            0.02461155876517296,
+            0.01756972447037697,
+            0.0110647426918149,
+            0.028529886156320572,
+            0.00459810858592391,
+            0.030565742403268814,
+            -0.002150896703824401,
+            -0.03064940683543682,
+            0.014669325202703476,
+            -0.020400401204824448,
+            0.02735857106745243,
+            -0.015784863382577896,
+            -0.004960658028721809,
+            -0.008401395753026009,
+            0.032490044832229614,
+            0.01812749356031418,
+            0.005263945087790489,
+            -0.04149801656603813,
+            0.016705181449651718,
+            0.009161355905234814,
+            -0.025852594524621964,
+            0.009516933932900429,
+            0.009565738961100578,
+            0.0181832704693079,
+            0.01777888834476471,
+            -0.023217136040329933,
+            -0.0037056778091937304,
+            0.00459810858592391,
+            -0.013365539722144604,
+            -0.022701198235154152,
+            0.024597613140940666,
+            0.003393675899133086,
+            -0.01849004253745079,
+            0.003433765610679984,
+            0.016119524836540222,
+            0.017918329685926437,
+            0.0071220132522284985,
+            0.008234065026044846,
+            -0.04052191972732544,
+            -0.0022188748698681593,
+            -0.01960558071732521,
+            -0.005361554678529501,
+            0.017792832106351852,
+            0.0275119561702013,
+            0.05123108625411987,
+            0.00025666088913567364,
+            0.03321514651179314,
+            -0.0030642435885965824,
+            0.0009246764821000397,
+            -0.036087654531002045,
+            0.011287850327789783,
+            0.03140239417552948,
+            0.01526892650872469,
+            0.028027893975377083,
+            0.030733073130249977,
+            -0.029338650405406952,
+            0.010262950323522091,
+            0.02005179598927498,
+            0.008373507298529148,
+            0.021460162475705147,
+            -0.005542829632759094,
+            0.034721121191978455,
+            -0.009698208421468735,
+            0.007257969584316015,
+            -0.011985061690211296,
+            -0.017039842903614044,
+            0.022729087620973587,
+            -0.03985259681940079,
+            -0.0009752242476679385,
+            0.022659366950392723,
+            -0.021794823929667473,
+            0.001286354847252369,
+            -0.01939641684293747,
+            -0.0075298817828297615,
+            0.009586654603481293,
+            -0.03586454689502716,
+            0.01934063993394375,
+            -0.021446218714118004,
+            -0.014878488145768642,
+            -0.039183273911476135,
+            -0.0016332174418494105,
+            -0.06866136938333511,
+            8.072399214142933e-05,
+            0.031625501811504364,
+            -0.0016846366925165057,
+            0.023691238835453987,
+            0.007006973493844271,
+            0.014132472686469555,
+            -0.055386465042829514,
+            -0.0040473113767802715,
+            -0.0073625510558485985,
+            0.014432273805141449,
+            -0.009028886444866657,
+            -0.03299203887581825,
+            -0.011922312900424004,
+            -0.0023374008014798164,
+            0.004071713890880346,
+            0.0032472615130245686,
+            0.007599602919071913,
+            -0.029756976291537285,
+            -0.013330679386854172,
+            -0.03854183852672577,
+            -0.010262950323522091,
+            -0.004490040708333254,
+            0.038932278752326965,
+            0.0009220619103871286,
+            -0.03976893052458763,
+            0.008750001899898052,
+            0.021948210895061493,
+            0.00022212715703062713,
+            -0.019745023921132088,
+            0.010423308238387108,
+            -0.042278893291950226,
+            0.02487649768590927,
+            -0.012396416626870632,
+            0.019187254831194878,
+            -0.0077529894188046455,
+            0.014223109930753708,
+            -0.0075647421181201935,
+            -0.009691236540675163,
+            -0.009175300598144531,
+            0.012326695024967194,
+            -0.027930283918976784,
+            -0.009454185143113136,
+            -0.03115139901638031,
+            0.009231077507138252,
+            -0.005284861661493778,
+            0.008896416053175926,
+            0.02795817144215107,
+            -0.004960658028721809,
+            -0.005005977116525173,
+            0.003339641960337758,
+            0.038011956959962845,
+            0.004730578511953354,
+            0.006041335873305798,
+            0.027037853375077248,
+            0.04423108324408531,
+            -0.00539641547948122,
+            -0.00426693307235837,
+            0.02486255392432213,
+            -0.010214145295321941,
+            0.016035858541727066,
+            -0.0133237075060606,
+            -0.006267929449677467,
+            0.027637455612421036,
+            -0.026159366592764854,
+            0.04280877113342285,
+            -0.008519921451807022,
+            0.017541835084557533,
+            -0.025964148342609406,
+            -0.01733267307281494,
+            -0.003034612163901329,
+            0.03686853125691414,
+            -0.000990040018223226,
+            0.006226096767932177,
+            0.017500003799796104,
+            0.0028411359526216984,
+            0.03742630034685135,
+            -0.02861355058848858,
+            -0.04473307356238365,
+            0.012215141206979752,
+            0.02547609992325306,
+            -0.008896416053175926,
+            -0.04094024747610092,
+            -0.020860562101006508,
+            -0.004908367525786161,
+            0.007766933646053076,
+            -0.03276893123984337,
+            -0.0037126499228179455,
+            0.014557771384716034,
+            0.0040089646354317665,
+            0.010848607867956161,
+            0.00960059929639101,
+            0.014153389260172844,
+            0.02851594053208828,
+            0.006755977403372526,
+            -0.03865339234471321,
+            0.0037126499228179455,
+            0.018880480900406837,
+            -0.014237054623663425,
+            -0.011497014202177525,
+            -0.02653586119413376,
+            0.0009717381908558309,
+            -0.009077690541744232,
+            -0.029756976291537285,
+            -0.013149404898285866,
+            0.006156375631690025,
+            -0.013205181807279587,
+            -0.012005978263914585,
+            -0.01853187568485737,
+            0.018559763208031654,
+            -0.002811504527926445,
+            0.01681673526763916,
+            -0.010681277140974998,
+            -0.013491038233041763,
+            0.014027890749275684,
+            0.021794823929667473,
+            -0.04225100204348564,
+            0.0009682521340437233,
+            -0.015812750905752182,
+            0.02734462544322014,
+            -0.005061754025518894,
+            0.01172709371894598,
+            0.012772910296916962,
+            -0.009398408234119415,
+            0.01180378720164299,
+            -0.01950797066092491,
+            -0.024346617981791496,
+            0.01127390656620264,
+            0.030203191563487053,
+            -0.01676095835864544,
+            -0.010276894085109234,
+            0.011741037480533123,
+            0.025615543127059937,
+            -0.023774905130267143,
+            -0.03349402919411659,
+            -0.030816737562417984,
+            0.03151394799351692,
+            -0.007843626663088799,
+            0.014188249595463276,
+            0.027623509988188744,
+            -0.0002780130016617477,
+            -0.03622709959745407,
+            -0.011176296509802341,
+            -0.03182072192430496,
+            -0.01945219375193119,
+            0.011343627236783504,
+            0.009426296688616276,
+            -0.03207171708345413,
+            -0.003395418869331479,
+            0.01839243434369564,
+            0.029756976291537285,
+            0.013421316631138325,
+            0.008296813815832138,
+            0.004503984935581684,
+            0.038876499980688095,
+            0.027023909613490105,
+            -0.003834662027657032,
+            0.0009926545899361372,
+            0.006288845557719469,
+            0.007836654782295227,
+            -0.017764942720532417,
+            0.0006161604542285204,
+            -0.010276894085109234,
+            -0.0008305529481731355,
+            -0.019480083137750626,
+            -0.006264443509280682,
+            0.00691633578389883,
+            -0.007202192675322294,
+            0.05968128517270088,
+            -0.0037928293459117413,
+            0.03449801355600357,
+            -0.018768927082419395,
+            0.026019925251603127,
+            0.04264144226908684,
+            -0.01169920526444912,
+            -0.02903187833726406,
+            -0.010395419783890247,
+            -0.026075702160596848,
+            -0.04492829367518425,
+            0.009928288869559765,
+            -0.015994027256965637,
+            -0.022784864529967308,
+            -0.011427292600274086,
+            -0.014069723896682262,
+            0.002382719423621893,
+            0.012535858899354935,
+            -0.012250002473592758,
+            0.001576569047756493,
+            -0.00861055962741375,
+            0.03301992639899254,
+            -0.0013456178130581975,
+            0.011866535991430283,
+            0.006470120511949062,
+            0.003243775339797139,
+            -0.012605579569935799,
+            -0.0037196220364421606,
+            0.014188249595463276,
+            -0.00018835598893929273,
+            0.014195221476256847,
+            -0.001346489298157394,
+            0.019466139376163483,
+            0.03513944894075394,
+            -0.04046614095568657,
+            -0.027023909613490105,
+            -0.006250499282032251,
+            -0.04955777898430824,
+            0.01650996319949627,
+            0.00397759024053812,
+            0.011587651446461678,
+            0.00866633653640747,
+            0.008875499479472637,
+            -0.03140239417552948,
+            0.024374505504965782,
+            -0.011071715503931046,
+            -0.015213149599730968,
+            0.022296816110610962,
+            -0.0026250004302710295,
+            0.012487053871154785,
+            0.02102789282798767,
+            0.0008022287511266768,
+            -0.041107576340436935,
+            -0.0033117535058408976,
+            0.01874103955924511,
+            -0.028195224702358246,
+            -0.00014674117846880108,
+            0.009035858325660229,
+            0.00034076202427968383,
+            0.0032559765968471766,
+            -0.013163348659873009,
+            0.0034965146332979202,
+            -0.012222114019095898,
+            -0.043478094041347504,
+            0.026061756536364555,
+            -0.002492530271410942,
+            -0.03165339305996895,
+            0.0363665409386158,
+            -0.019173309206962585,
+            -0.009238049387931824,
+            0.008791834115982056,
+            -0.03988048434257507,
+            -0.032796818763017654,
+            0.01677490398287773,
+            -0.017346616834402084,
+            0.009886455722153187,
+            -0.04077291488647461,
+            0.006686256267130375,
+            0.013644424267113209,
+            -0.0008780504576861858,
+            -0.011399404145777225,
+            0.04163745790719986,
+            -0.022980082780122757,
+            -0.004368028603494167,
+            0.0128426318988204,
+            -0.027121517807245255,
+            0.004043825436383486,
+            -0.0054277898743748665,
+            -0.015394425019621849,
+            0.009865539148449898,
+            -0.021111557260155678,
+            0.018964147195219994,
+            0.02330080047249794,
+            0.0016175301279872656,
+            0.022213151678442955,
+            -0.011615539900958538,
+            -0.0015417084796354175,
+            0.019005978479981422,
+            -0.005473108496516943,
+            0.047382477670907974,
+            0.0022816238924860954,
+            0.030760960653424263,
+            0.013337651267647743,
+            -0.024806777015328407,
+            0.0023426299449056387,
+            0.023886458948254585,
+            0.004622510634362698,
+            0.0055219135247170925,
+            0.008122511208057404,
+            0.007209164556115866,
+            -0.02101394720375538,
+            0.0024071219377219677,
+            -0.03600399196147919,
+            0.006030877586454153,
+            -0.0008009214652702212,
+            -0.018225103616714478,
+            -0.008331675082445145,
+            0.02872510440647602,
+            0.025545820593833923,
+            -0.01849004253745079,
+            0.02268725447356701,
+            -0.012731078080832958,
+            -0.002875996520742774,
+            0.011427292600274086,
+            0.002492530271410942,
+            0.023482074961066246,
+            -0.005256973206996918,
+            0.003545319428667426,
+            0.010674304328858852,
+            0.006410857662558556,
+            0.014501994475722313,
+            0.0014685012865811586,
+            0.00866633653640747,
+            0.00848506111651659,
+            -0.038876499980688095,
+            0.011824702844023705,
+            0.03580877184867859,
+            -0.004193725995719433,
+            -0.013281874358654022,
+            -0.01945219375193119,
+            -0.008220121264457703,
+            0.010255977511405945,
+            0.017667334526777267,
+            0.02091633901000023,
+            0.007718128617852926,
+            0.0057798814959824085,
+            -0.029115542769432068,
+            0.021111557260155678,
+            0.010276894085109234,
+            -0.01157370675355196,
+            -0.002255478408187628,
+            0.0025134466122835875,
+            0.04660160094499588,
+            -0.05845419317483902,
+            0.004859562497586012,
+            0.016551796346902847,
+            -0.0180856604129076,
+            -0.002550050150603056,
+            -0.02422112040221691,
+            -0.001696837949566543,
+            -0.004978088662028313,
+            0.02614542283117771,
+            -0.013107571750879288,
+            0.0021840143017470837,
+            -0.03555777668952942,
+            0.009203189052641392,
+            -0.001525149680674076,
+            -0.012417333200573921,
+            0.00012604272342287004,
+            0.002182271331548691,
+            -0.021334664896130562,
+            -0.0246255025267601,
+            0.0032507474534213543,
+            0.014069723896682262,
+            0.03215538337826729,
+            -0.006567730102688074,
+            0.0011983318254351616,
+            0.03714741766452789,
+            -0.0256852637976408,
+            -0.014683269895613194,
+            0.026061756536364555,
+            -0.005229084752500057,
+            0.03505578264594078,
+            0.02542032301425934,
+            -0.03120717592537403,
+            -0.05243028700351715,
+            -0.007264941465109587,
+            -0.011566734872758389,
+            0.01010956335812807,
+            0.009454185143113136,
+            0.032545823603868484,
+            -0.0001861772034317255,
+            -0.0030328689608722925,
+            -0.009691236540675163,
+            -0.01823904737830162,
+            0.034776899963617325,
+            0.00019924991647712886,
+            0.01743028126657009,
+            0.004514443222433329,
+            -0.018601596355438232,
+            -0.025699207559227943,
+            0.013037851080298424,
+            0.003547062398865819,
+            -0.005121016874909401,
+            -0.05212351307272911,
+            0.006982570979744196,
+            0.005054781679064035,
+            0.006173805799335241,
+            0.0377051867544651,
+            0.015994027256965637,
+            -0.0001385707437293604,
+            -0.0017822462832555175,
+            -0.0192430317401886,
+            0.029254985973238945,
+            -0.029729088768363,
+            0.04052191972732544,
+            0.0009011455695144832,
+            0.012326695024967194,
+            0.008791834115982056,
+            0.03221116214990616,
+            -0.017262950539588928,
+            -0.003604582278057933,
+            -0.02339841052889824,
+            -0.018420321866869926,
+            0.009154384024441242,
+            -0.020749008283019066,
+            0.0057066744193434715,
+            -0.007739045191556215,
+            0.0037649408914148808,
+            0.0038381479680538177,
+            0.008366535417735577,
+            0.05220717936754227,
+            0.026270920410752296,
+            0.0006104956264607608,
+            -0.024025900289416313,
+            -0.010813746601343155,
+            -0.016928289085626602,
+            0.005159363616257906,
+            -0.020498011261224747,
+            -0.007739045191556215,
+            -0.032099608331918716,
+            -0.014613548293709755,
+            -0.014529882930219173,
+            -0.025754984468221664,
+            0.00850597769021988,
+            0.014641436748206615,
+            -0.006424801889806986,
+            -0.011176296509802341,
+            -0.019591636955738068,
+            -0.033661361783742905,
+            -0.02081872895359993,
+            -0.009035858325660229,
+            -0.01430677529424429,
+            0.029254985973238945,
+            -0.0028324208687990904,
+            -0.006334164645522833,
+            -0.006623507011681795,
+            -0.030063750222325325,
+            0.018615540117025375,
+            -0.018266934901475906,
+            -0.00789940357208252,
+            -0.012012950144708157,
+            0.02741434797644615,
+            0.0028637952636927366,
+            0.046573713421821594,
+            0.002319970401003957,
+            -0.014209166169166565,
+            -0.05190040543675423,
+            -0.03491634130477905,
+            -0.02947809360921383,
+            0.016886457800865173,
+            0.0032071718014776707,
+            0.0010702193249017,
+            0.002781872870400548,
+            0.006410857662558556,
+            0.002318227430805564,
+            -0.0056508975103497505,
+            -0.029896419495344162,
+            0.0032298313453793526,
+            0.010388447903096676,
+            0.005009463056921959,
+            -0.035083670169115067,
+            -0.021083669736981392,
+            -0.018364544957876205,
+            0.038876499980688095,
+            -0.03742630034685135,
+            0.03438645973801613,
+            -0.011510957963764668,
+            -0.01814143732190132,
+            0.004249502904713154,
+            -0.027037853375077248,
+            0.006121514830738306,
+            0.006292331963777542,
+            0.007578686345368624,
+            0.03600399196147919,
+            0.01837848871946335,
+            0.043840646743774414,
+            -0.0286693274974823,
+            0.03820717707276344,
+            -0.01326793059706688,
+            -0.0006126743974164128,
+            -0.0005874005146324635,
+            0.0038799806497991085,
+            0.02922709658741951,
+            -0.002659860998392105,
+            -0.0006985185318626463,
+            -0.0050931284204125404,
+            -0.004315737634897232,
+            -0.005309263709932566,
+            0.003034612163901329,
+            -0.025657374411821365,
+            0.012068727053701878,
+            0.024904387071728706,
+            -0.015338648110628128,
+            0.03628287464380264,
+            -0.004674801602959633,
+            0.03756574168801308,
+            -0.013511954806745052,
+            -0.027525901794433594,
+            0.027135463431477547,
+            -0.01570119708776474,
+            -0.013525898568332195,
+            0.005086156073957682,
+            0.0018092632526531816,
+            -0.0015643677907064557,
+            -0.02097211591899395,
+            0.024235064163804054,
+            0.02487649768590927,
+            0.003939243499189615,
+            0.0038102595135569572,
+            -0.02953387051820755,
+            -0.009984065778553486,
+            0.0181832704693079,
+            -0.004904881119728088,
+            0.020302793011069298,
+            0.02320319041609764,
+            -0.014118527993559837,
+            0.02193426713347435,
+            -0.01050697360187769,
+            0.016384465619921684,
+            0.022352593019604683,
+            -0.014069723896682262,
+            -0.016802791506052017,
+            -0.020274903625249863,
+            -0.004019422922283411,
+            0.004388945177197456,
+            -0.02284064143896103,
+            -0.03159761428833008,
+            0.0028463650960475206,
+            0.02958964742720127,
+            0.007108069024980068,
+            -0.03658964857459068,
+            -0.005738048814237118,
+            -0.017346616834402084,
+            0.030984068289399147,
+            -0.026521917432546616,
+            -0.0007778263534419239,
+            -0.010576695203781128,
+            -0.001526892650872469,
+            0.012005978263914585,
+            0.04088446870446205,
+            -0.0016088150441646576,
+            0.03095618076622486,
+            0.0171235091984272,
+            0.005145419389009476,
+            -0.0077320728451013565,
+            0.03525100275874138,
+            -0.016035858541727066,
+            -0.022561756893992424,
+            0.031123511493206024,
+            -0.014557771384716034,
+            -0.007843626663088799,
+            0.018043827265501022,
+            0.015547811053693295,
+            -0.015812750905752182,
+            0.026884466409683228,
+            -0.0008057148079387844,
+            0.011476097628474236,
+            0.011789842508733273,
+            -0.0009682521340437233,
+            -0.013177293352782726,
+            -0.0286693274974823,
+            -0.023984067142009735,
+            0.01802988350391388,
+            -0.008861555717885494,
+            0.03246215730905533,
+            0.018754983320832253,
+            0.015492034144699574,
+            -0.03923904895782471,
+            0.0365617610514164,
+            -0.013253985904157162,
+            -0.004510956816375256,
+            0.0032472615130245686,
+            -0.020749008283019066,
+            -0.00024380606191698462,
+            -0.009593627415597439,
+            -0.03377291560173035,
+            -0.03321514651179314,
+            -0.02339841052889824,
+            0.0003054656845051795,
+            0.01162251178175211,
+            -0.00419023958966136,
+            -0.031039845198392868,
+            -0.023189246654510498,
+            0.025057774037122726,
+            0.02826494537293911,
+            -0.008959164842963219,
+            -0.026312753558158875,
+            -0.006731574889272451,
+            -0.019689247012138367,
+            0.020776895806193352,
+            -0.0008392680902034044,
+            0.004573706071823835,
+            0.021794823929667473,
+            0.01076494250446558,
+            -0.006581674329936504,
+            -0.007885459810495377,
+            -0.036952197551727295,
+            0.008540838025510311,
+            0.007285858038812876,
+            -0.0034633970353752375,
+            -0.006787351798266172,
+            0.013686257414519787,
+            0.0005233442061580718,
+            -0.009203189052641392,
+            -0.018252991139888763,
+            0.0011312252609059215,
+            -0.0009516933932900429,
+            0.01317032054066658,
+            -0.00043771794298663735,
+            0.014174304902553558,
+            0.016551796346902847,
+            -0.006009961012750864,
+            -0.005710160359740257,
+            0.008980081416666508,
+            0.04478885233402252,
+            0.02330080047249794,
+            -0.02613147906959057,
+            0.011838647536933422,
+            0.015882473438978195,
+            0.0029282872565090656,
+            -0.015826696529984474,
+            0.006940738297998905,
+            0.023886458948254585,
+            -0.031235065311193466,
+            -0.0027138949371874332,
+            -0.007397411856800318,
+            0.01701195538043976,
+            0.032350603491067886,
+            0.027428291738033295,
+            0.00019129733846057206,
+            -0.005748507101088762,
+            -0.00554631557315588,
+            0.019033867865800858,
+            0.012563747353851795,
+            0.018601596355438232,
+            -0.0071603599935770035,
+            -0.012354583479464054,
+            -0.026103589683771133,
+            0.008450200781226158,
+            -0.0026947215665131807,
+            -0.011531874537467957,
+            0.032545823603868484,
+            0.012556775473058224,
+            0.015826696529984474,
+            -0.012828687205910683,
+            -0.016788847744464874,
+            0.026661358773708344,
+            0.022199207916855812,
+            -0.025308769196271896,
+            -0.021739047020673752,
+            0.010325699113309383,
+            0.0017683020560070872,
+            -0.0029840641655027866,
+            -0.013553787022829056,
+            -0.022617533802986145,
+            0.028585663065314293,
+            0.09086057543754578,
+            -0.015045818872749805,
+            -0.01337251253426075,
+            -0.01733267307281494,
+            -0.023984067142009735,
+            -0.019926298409700394,
+            0.0005407744902186096,
+            0.006145917344838381,
+            0.03313148021697998,
+            0.009886455722153187,
+            -0.003914841450750828,
+            -0.005783367436379194,
+            -0.006135459057986736,
+            0.012054783292114735,
+            -0.031988054513931274,
+            0.02051195502281189,
+            0.003050299361348152,
+            0.016440242528915405,
+            0.01002589799463749,
+            -0.010702192783355713,
+            -0.00901494175195694,
+            0.013435261324048042,
+            0.03438645973801613,
+            0.004789841361343861,
+            -0.021752992644906044,
+            0.001578312017954886,
+            0.004873506724834442,
+            0.010493029840290546,
+            -0.003998506814241409,
+            -0.01844821125268936,
+            0.005487052723765373,
+            0.011594623327255249,
+            -0.003092132043093443,
+            0.0028986558318138123,
+            -0.00843625608831644,
+            0.05429881438612938,
+            -0.009182272478938103,
+            0.01317032054066658,
+            0.0041205184534192085,
+            -0.019368529319763184,
+            -0.00744621641933918,
+            0.022017931565642357,
+            -0.0040089646354317665,
+            -0.00643526017665863,
+            -0.012605579569935799,
+            0.034274905920028687,
+            0.031235065311193466,
+            -0.03486056253314018,
+            0.017541835084557533,
+            -0.009579682722687721,
+            -0.014055779203772545,
+            0.007467132527381182,
+            -0.00782271008938551,
+            0.014216138049960136,
+            0.004636454861611128,
+            0.004761952906847,
+            0.010918328538537025,
+            -0.01878287084400654,
+            -0.011441237293183804,
+            0.003656873246654868,
+            0.0007006973028182983,
+            0.017960162833333015,
+            0.016384465619921684,
+            0.01050697360187769,
+            -0.009795818477869034,
+            0.018364544957876205,
+            0.023370521143078804,
+            0.0019609066657721996,
+            0.003187998663634062,
+            0.0096215158700943,
+            0.011887452565133572,
+            -0.006961654406040907,
+            -0.008847611024975777,
+            -0.015338648110628128,
+            0.0076762959361076355,
+            0.01885259337723255,
+            -0.015505978837609291,
+            0.003221116028726101,
+            0.013637452386319637,
+            -0.04261355474591255,
+            0.0191454216837883,
+            0.010018926113843918,
+            0.013407372869551182,
+            0.01278685498982668,
+            -0.022561756893992424,
+            0.004625997040420771,
+            -0.013470121659338474,
+            -0.018169326707720757,
+            0.002300797263160348,
+            -0.01430677529424429,
+            0.012480081990361214,
+            -0.0020515441428869963,
+            -0.006334164645522833,
+            -0.018057772889733315,
+            0.005640439223498106,
+            -0.005507969297468662,
+            -0.0011809015413746238,
+            -0.000824888062197715,
+            -0.011239045299589634,
+            -0.023412354290485382,
+            -0.03717530518770218,
+            0.016844624653458595,
+            0.01109263114631176,
+            -0.006511953193694353,
+            -0.008896416053175926,
+            -0.013463149778544903,
+            0.018824703991413116,
+            -0.004448208026587963,
+            0.00529183354228735,
+            -0.0115458182990551,
+            0.0037684268318116665,
+            -0.005141932982951403,
+            0.006609562784433365,
+            -0.006428287830203772,
+            0.01013047993183136,
+            0.020233070477843285,
+            0.008129483088850975,
+            -0.0475219190120697,
+            -0.005365040618926287,
+            0.009363546967506409,
+            -0.025768928229808807,
+            0.0037509966641664505,
+            0.027930283918976784,
+            0.008338646963238716,
+            0.002250249497592449,
+            0.01218028087168932,
+            0.0115458182990551,
+            -0.020902393385767937,
+            0.014906376600265503,
+            0.007501993328332901,
+            0.00018890068167820573,
+            0.0064213159494102,
+            0.0019347612978890538,
+            -0.0004309637297410518,
+            0.021460162475705147,
+            -0.007913348264992237,
+            0.0061807781457901,
+            -0.020023908466100693,
+            0.026828689500689507,
+            -0.019382473081350327,
+            -0.0014571715146303177,
+            0.025964148342609406,
+            0.023468131199479103,
+            0.00473406445235014,
+            0.015129484236240387,
+            -0.02618725597858429,
+            0.006086654495447874,
+            0.03246215730905533,
+            0.018169326707720757,
+            0.029645422473549843,
+            -0.014948209747672081,
+            -0.022770920768380165,
+            0.014090639539062977,
+            -0.015826696529984474,
+            -0.006337650585919619,
+            -0.014076695777475834,
+            0.008366535417735577,
+            0.0008431898895651102,
+            0.03820717707276344,
+            0.008450200781226158,
+            -0.026061756536364555,
+            0.029673311859369278,
+            -0.005385957192629576,
+            -0.029115542769432068,
+            0.020442234352231026,
+            -0.009440240450203419,
+            0.009238049387931824,
+            0.004082172177731991,
+            -0.016886457800865173,
+            0.025141438469290733,
+            -0.010276894085109234,
+            -0.004225100390613079,
+            0.01889442466199398,
+            0.027874507009983063,
+            0.0035244030877947807,
+            -0.028348609805107117,
+            0.010374504141509533,
+            -0.004953686147928238,
+            -0.0002974041854031384,
+            -0.0070836665108799934,
+            0.012152392417192459,
+            -0.016537850722670555,
+            0.0013874504948034883,
+            0.006825698539614677,
+            -0.024653390049934387,
+            -0.008603586815297604,
+            -0.001669820980168879,
+            -0.014153389260172844,
+            -0.03896016627550125,
+            0.002949203597381711,
+            -0.006623507011681795,
+            0.003548805369064212,
+            -0.005208168178796768,
+            0.009537850506603718,
+            -0.017290839925408363,
+            0.015101595781743526,
+            0.008324702270328999,
+            -0.0029770920518785715,
+            -0.0069372523576021194,
+            -0.00797609705477953,
+            -0.029143432155251503,
+            0.037454187870025635,
+            0.00015545632049907,
+            -0.0036254986189305782,
+            0.005588148254901171,
+            -0.014195221476256847,
+            0.00343725155107677,
+            0.02724701724946499,
+            0.02406773343682289,
+            -0.043784867972135544,
+            0.001980080036446452,
+            -0.008840639144182205,
+            -0.021766936406493187,
+            0.013491038233041763,
+            -0.003914841450750828,
+            0.0171235091984272,
+            0.0053510963916778564,
+            -0.029729088768363,
+            -0.00030350478482432663,
+            0.010269922204315662,
+            -0.012500998564064503,
+            0.0023984068538993597,
+            0.024751000106334686,
+            -0.00036146046477369964,
+            0.0031043333001434803,
+            -0.009440240450203419,
+            0.02254781313240528,
+            -0.01662151701748371,
+            0.015938250347971916,
+            0.006215638481080532,
+            0.006375997327268124,
+            0.014376496896147728,
+            -0.02005179598927498,
+            0.019982075318694115,
+            0.0038242037408053875,
+            0.004437749739736319,
+            0.0057066744193434715,
+            -0.001433640718460083,
+            0.021906377747654915,
+            -0.0003137450839858502,
+            0.006630479358136654,
+            0.016830680891871452,
+            -0.021083669736981392,
+            0.011413348838686943,
+            -0.006930280011147261,
+            -0.00016743964806664735,
+            -0.0015791835030540824,
+            -0.021083669736981392,
+            -0.027428291738033295,
+            0.035334669053554535,
+            0.001134711317718029,
+            0.00016090329154394567,
+            0.008087650872766972,
+            -0.013393428176641464,
+            -0.030203191563487053,
+            -0.02168327011168003,
+            -0.002867281436920166,
+            0.01281474344432354,
+            0.01762550137937069,
+            -0.017709165811538696,
+            0.007766933646053076,
+            0.011315738782286644,
+            0.02445817179977894,
+            0.03407968580722809,
+            -0.0011983318254351616,
+            0.020484067499637604,
+            0.026466140523552895,
+            0.009321714751422405,
+            0.014976098202168941,
+            -0.008415340445935726,
+            -0.00901494175195694,
+            0.012473110109567642,
+            0.015408368781208992,
+            0.011789842508733273,
+            0.0019678787793964148,
+            -0.024653390049934387,
+            -0.009795818477869034,
+            -0.010262950323522091,
+            0.006463148631155491,
+            -0.005745020695030689,
+            -0.001980080036446452,
+            0.02299402840435505,
+            -0.003981076180934906,
+            -0.018001995980739594,
+            -0.019270919263362885,
+            -0.016956178471446037,
+            -0.01233366783708334,
+            -0.009405380114912987,
+            0.015603587962687016,
+            -0.00584960263222456,
+            -0.015896417200565338,
+            -0.024095620959997177,
+            0.011957173235714436,
+            0.02158566191792488,
+            0.02694024331867695,
+            0.0003128735697828233,
+            -0.004622510634362698,
+            0.01677490398287773,
+            0.0057415347546339035,
+            0.005867032799869776,
+            -0.008931276388466358,
+            -0.0009673806489445269,
+            0.011657372117042542,
+            0.025252992287278175,
+            -0.010897411964833736,
+            -0.019996019080281258,
+            -0.032043829560279846,
+            0.0056125507690012455,
+            0.0004745394107885659,
+            -0.006905877497047186,
+            0.008575698360800743,
+            0.0024890443310141563,
+            0.004608566407114267,
+            0.013574703596532345,
+            -0.006926794070750475,
+            0.009328686632215977,
+            -0.029673311859369278,
+            0.007746017072349787,
+            0.011531874537467957,
+            0.02953387051820755,
+            -0.05231873318552971,
+            -0.019173309206962585,
+            0.0007743402966298163,
+            0.046880487352609634,
+            0.01660757325589657,
+            0.013644424267113209,
+            -0.017555780708789825,
+            -0.011350599117577076,
+            -0.013790839351713657,
+            -0.011057770811021328,
+            -0.0211673341691494,
+            -0.010269922204315662,
+            0.020804785192012787,
+            -0.006337650585919619,
+            0.0421394482254982,
+            0.029812753200531006,
+            -0.004646913148462772,
+            -0.0010850350372493267,
+            -0.01448805071413517,
+            0.014669325202703476,
+            -0.009412351995706558,
+            0.03870917111635208,
+            0.004476096481084824,
+            0.012326695024967194,
+            -0.009021913632750511,
+            -0.00043771794298663735,
+            -0.0496414415538311,
+            -0.015743030235171318,
+            0.017095619812607765,
+            -0.03207171708345413,
+            0.00023116912052500993,
+            0.030370522290468216,
+            0.020679285749793053,
+            -0.00015338648518081754,
+            -0.01228486280888319,
+            0.02608964592218399,
+            0.025503989309072495,
+            0.007962152361869812,
+            0.002156125847250223,
+            -0.01752789132297039,
+            -0.007481076754629612,
+            -0.0056683276779949665,
+            -0.012250002473592758,
+            0.00782271008938551,
+            -0.027470124885439873,
+            0.021976100280880928,
+            -0.012807771563529968,
+            -0.019870521500706673,
+            0.029505981132388115,
+            -0.009049802087247372,
+            0.02187849022448063,
+            0.009565738961100578,
+            -0.0139023931697011,
+            0.03717530518770218,
+            0.011106575839221478,
+            0.011176296509802341,
+            -0.008903387933969498,
+            0.014627492986619473,
+            -0.012033866718411446,
+            0.028585663065314293,
+            -0.017667334526777267,
+            0.015882473438978195,
+            0.006773407571017742,
+            -0.00792729202657938,
+            0.00477589713409543,
+            0.031179288402199745,
+            0.012821715325117111,
+            -0.004908367525786161,
+            0.028139447793364525,
+            0.007941236719489098,
+            0.022073708474636078,
+            0.004768925253301859,
+            0.008220121264457703,
+            0.027930283918976784,
+            0.02786056324839592,
+            -0.010172312147915363,
+            -0.03184860944747925,
+            -0.007257969584316015,
+            -0.005009463056921959,
+            0.018768927082419395,
+            -0.018225103616714478,
+            -0.004472610540688038,
+            -0.0027365542482584715,
+            0.0128426318988204,
+            -0.008854582905769348,
+            0.010960161685943604,
+            0.0010649901814758778,
+            -0.032294824719429016,
+            -0.02406773343682289,
+            -0.03678486868739128,
+            0.0014231825480237603,
+            -0.021920323371887207,
+            -0.015687253326177597,
+            -6.411075446521863e-05,
+            -0.027037853375077248,
+            0.0013708916958421469,
+            0.027428291738033295,
+            -0.008359563536942005,
+            -0.012368528172373772,
+            0.005915837828069925,
+            0.014571716077625751,
+            -0.0057241045869886875,
+            -0.004005478695034981,
+            0.0029021420050412416,
+            -0.0005425175186246634,
+            4.28131315857172e-05,
+            0.010353587567806244,
+            -0.010221117176115513,
+            -0.036087654531002045,
+            -0.02668924815952778,
+            0.013435261324048042,
+            0.0019086158135905862,
+            0.00792729202657938,
+            0.013658368960022926,
+            0.003020667936652899,
+            0.0001337774156127125,
+            0.011768925935029984,
+            0.00047628243919461966,
+            0.003034612163901329,
+            -0.0013830928364768624,
+            0.019033867865800858,
+            0.008617531508207321,
+            -0.015519922599196434,
+            0.018978090956807137,
+            0.002586653921753168,
+            -0.005487052723765373,
+            -0.010332670994102955,
+            -0.004078685771673918,
+            -0.010995022021234035,
+            -0.022045820951461792,
+            0.005577689968049526,
+            0.0076414356008172035,
+            -0.012319723144173622,
+            0.00911952368915081,
+            -0.020637454465031624,
+            0.016830680891871452,
+            0.026814745739102364,
+            0.009335658513009548,
+            -0.030203191563487053,
+            -0.00563695328310132,
+            -0.012912352569401264,
+            -0.01286354847252369,
+            -0.019996019080281258,
+            0.004333168268203735,
+            0.019577693194150925,
+            0.008833667263388634,
+            0.0057798814959824085,
+            0.01716534234583378,
+            0.004036853089928627,
+            -0.010967133566737175,
+            0.0015512950485572219,
+            0.0005643054028041661,
+            -0.0058705187402665615,
+            -0.0182808805257082,
+            0.008526894263923168,
+            -0.001230577938258648,
+            -0.009293826296925545,
+            0.012187252752482891,
+            0.015254982747137547,
+            0.00655727181583643,
+            0.0024332674220204353,
+            0.0171235091984272,
+            -0.00011231638200115412,
+            -0.004368028603494167,
+            0.016272911801934242,
+            0.07898009568452835,
+            0.00919621717184782,
+            0.013707173988223076,
+            0.0021038351114839315,
+            -0.02045617811381817,
+            -0.014934265054762363,
+            -0.0037998014595359564,
+            -0.04255777597427368,
+            0.013484066352248192,
+            0.02005179598927498,
+            -0.004497012589126825,
+            -0.013163348659873009,
+            -0.012152392417192459,
+            -0.00866633653640747,
+            -0.001097236294299364,
+            0.03399602323770523,
+            -0.008185259997844696,
+            0.014808767475187778,
+            0.038318730890750885,
+            0.01878287084400654,
+            -0.006288845557719469,
+            0.00964940432459116,
+            0.025197215378284454,
+            -0.010437252931296825,
+            0.00477589713409543,
+            -0.004971116315573454,
+            0.005654383450746536,
+            0.011008965782821178,
+            0.010771914385259151,
+            0.01814143732190132,
+            0.010304782539606094,
+            -0.0021770421881228685,
+            -0.01950797066092491,
+            0.005511455237865448,
+            -0.011455181054770947,
+            0.013302790932357311,
+            -0.0030572714749723673,
+            -0.0012018178822472692,
+            -0.02629880979657173,
+            0.012905380688607693,
+            0.009461157023906708,
+            -0.013539843261241913,
+            -0.00949601735919714,
+            0.002987550338730216,
+            -0.007278885692358017,
+            0.014320719987154007,
+            -0.010409364476799965,
+            0.0105836670845747,
+            0.009056774899363518,
+            -0.005103586707264185,
+            -0.015380480326712132,
+            -0.001689865835942328,
+            -0.004890936892479658,
+            0.008094622753560543,
+            -0.010988050140440464,
+            -0.011992033571004868,
+            0.015394425019621849,
+            0.012263946235179901,
+            0.012556775473058224,
+            0.017109565436840057,
+            -0.012626496143639088,
+            0.008847611024975777,
+            -0.0028637952636927366,
+            -0.016454186290502548,
+            -0.002661603968590498,
+            -0.018517931923270226,
+            0.011315738782286644,
+            0.004908367525786161,
+            0.005476594436913729,
+            -0.010569723322987556,
+            0.00014347300748340786,
+            -0.006637451238930225,
+            0.008331675082445145,
+            -0.0035331181716173887,
+            -0.0016166586428880692,
+            -0.012096615508198738,
+            -0.010576695203781128,
+            -0.0035052297171205282,
+            0.02314741350710392,
+            0.011789842508733273,
+            0.0009577939636074007,
+            0.009530877694487572,
+            0.0072719138115644455,
+            -0.007031375542283058,
+            -0.013414344750344753,
+            0.002541335066780448,
+            -0.001469372771680355,
+            0.005996016785502434,
+            0.021348608657717705,
+            -0.010520918294787407,
+            -0.0071394434198737144,
+            0.0020811755675822496,
+            0.016747014597058296,
+            0.0058530885726213455,
+            0.0009325200808234513,
+            0.023468131199479103,
+            0.0003483877517282963,
+            0.0030572714749723673,
+            -0.01864342950284481,
+            -0.015394425019621849,
+            0.010346615687012672,
+            0.026675302535295486,
+            0.00863844808191061,
+            0.0010275151580572128,
+            -0.017193229869008064,
+            0.0026162853464484215,
+            0.027372514829039574,
+            0.026019925251603127,
+            0.007543826010078192,
+            -0.0027417833916842937,
+            0.0009037601412273943,
+            -0.010151396505534649,
+            -0.0001501182996435091,
+            0.028390442952513695,
+            -0.015185261145234108,
+            -0.0006593004218302667,
+            -0.005256973206996918,
+            -0.013281874358654022,
+            -0.017151396721601486,
+            0.00957271084189415,
+            0.027233071625232697,
+            0.006222610827535391,
+            0.010409364476799965,
+            -0.01995418593287468,
+            0.01334462407976389,
+            -0.009007969871163368,
+            -0.008282870054244995,
+            -0.009265937842428684,
+            -0.021599605679512024,
+            -0.014376496896147728,
+            -0.011036854237318039,
+            -0.0032228592317551374,
+            -0.014766935259103775,
+            -0.0002703872451093048,
+            0.002027141861617565,
+            0.005727590527385473,
+            -0.003156624035909772,
+            -0.0036080684512853622,
+            -0.0061807781457901,
+            0.010548806749284267,
+            -0.00429482152685523,
+            0.0037684268318116665,
+            -0.023942235857248306,
+            -0.026507971808314323,
+            0.01839243434369564,
+            -0.008422312326729298,
+            -0.00397759024053812,
+            -0.015003986656665802,
+            -0.004782869480550289,
+            -0.023691238835453987,
+            0.01802988350391388,
+            -0.023523908108472824,
+            -0.010590638965368271,
+            0.004381972830742598,
+            0.013511954806745052,
+            -0.016747014597058296,
+            -0.01532470341771841,
+            -0.0022711658384650946,
+            -0.01228486280888319,
+            0.004256474785506725,
+            -0.03380080312490463,
+            0.007578686345368624,
+            -0.000818351749330759,
+            0.012975101359188557,
+            -0.020735062658786774,
+            0.02491833083331585,
+            0.0033117535058408976,
+            -0.013009962625801563,
+            -0.008373507298529148,
+            -0.007223108783364296,
+            -0.0026563750579953194,
+            0.005912351422011852,
+            0.001685508294031024,
+            -0.014766935259103775,
+            0.02081872895359993,
+            0.0016959664644673467,
+            0.009586654603481293,
+            -0.01112051960080862,
+            0.00803187396377325,
+            0.011162352748215199,
+            0.005082670133560896,
+            -0.003649901133030653,
+            0.007648407481610775,
+            -0.0031513948924839497,
+            0.019368529319763184,
+            0.020637454465031624,
+            -0.006341136526316404,
+            0.008038845844566822,
+            0.004015936981886625,
+            -0.017193229869008064,
+            -0.007034861948341131,
+            0.00405428372323513,
+            0.0029126000590622425,
+            -0.018727093935012817,
+            0.025099605321884155,
+            0.027009963989257812,
+            0.001078934408724308,
+            -0.02522510476410389,
+            0.010925300419330597,
+            0.011001993902027607,
+            0.014529882930219173,
+            0.02147410809993744,
+            -0.002708665793761611,
+            -0.023175302892923355,
+            0.009607571177184582,
+            -0.008415340445935726,
+            -0.01920119859278202,
+            0.031430285423994064,
+            0.022003987804055214,
+            0.011127491481602192,
+            -0.05491235852241516,
+            0.003093875013291836,
+            0.01579880714416504,
+            -0.013609563931822777,
+            -0.02953387051820755,
+            -0.011420320719480515,
+            -0.009935260750353336,
+            0.0029091141186654568,
+            0.013909365050494671,
+            -0.015213149599730968,
+            -0.01752789132297039,
+            0.014934265054762363,
+            0.030928291380405426,
+            0.0275119561702013,
+            0.025057774037122726,
+            -0.023426298052072525,
+            0.001805777195841074,
+            -0.025113550946116447,
+            0.00640737172216177,
+            -0.016077691689133644,
+            -0.005867032799869776,
+            0.0123615562915802,
+            -0.007613547146320343,
+            0.020735062658786774,
+            0.002405378967523575,
+            0.02183665707707405,
+            0.017695222049951553,
+            0.03692431002855301,
+            -0.016593627631664276,
+            -0.030231080949306488,
+            -0.011329683475196362,
+            -0.014048807322978973,
+            0.023370521143078804,
+            0.01789044216275215,
+            0.00967729277908802,
+            0.028446219861507416,
+            0.02928287349641323,
+            -0.0020550303161144257,
+            0.00960059929639101,
+            -2.5587109121261165e-05,
+            8.219467417802662e-05,
+            -0.018113549798727036,
+            -0.02314741350710392,
+            -0.0021630979608744383,
+            0.00967729277908802,
+            0.013923308812081814,
+            -0.005605578422546387,
+            -0.0027278391644358635,
+            0.021097613498568535,
+            -0.0032629487104713917,
+            -0.0027905881870537996,
+            -0.03555777668952942,
+            0.0018842134159058332,
+            -0.008924304507672787,
+            0.021808769553899765,
+            -0.00020491474424488842,
+            -0.008094622753560543,
+            -0.027525901794433594,
+            -0.004918825346976519,
+            0.034721121191978455,
+            0.005950698163360357,
+            0.00012168515240773559,
+            0.011469125747680664,
+            0.024193231016397476,
+            0.010513946413993835,
+            0.016328688710927963,
+            -0.001948705525137484,
+            0.001576569047756493,
+            0.022784864529967308,
+            -0.006644423585385084,
+            0.02187849022448063,
+            0.003827689914032817,
+            -0.013044822961091995,
+            -0.0017866039415821433,
+            -0.0019609066657721996,
+            -0.008659363724291325,
+            0.009900400415062904,
+            0.015533867292106152,
+            -0.011810759082436562,
+            0.018406378105282784,
+            -0.016021914780139923,
+            -0.0012401645071804523,
+            -0.023970123380422592,
+            -0.0005242157494649291,
+            0.013651397079229355,
+            -0.00317928334698081,
+            0.004985060542821884,
+            -0.019521916285157204,
+            0.0057624513283371925,
+            0.014111556112766266,
+            -0.012675301171839237,
+            0.004761952906847,
+            -0.02158566191792488,
+            0.006138945464044809,
+            0.011531874537467957,
+            -0.001174801029264927,
+            -0.009412351995706558,
+            -0.005385957192629576,
+            0.01165040023624897,
+            -0.015882473438978195,
+            0.004971116315573454,
+            0.01797410659492016,
+            0.012054783292114735,
+            0.010018926113843918,
+            0.006860558874905109,
+            0.00911952368915081,
+            0.008443228900432587,
+            0.012194225564599037,
+            -0.036952197551727295,
+            0.018071716651320457,
+            -0.01430677529424429,
+            0.009440240450203419,
+            -0.0017552294302731752,
+            -0.007829682901501656,
+            -0.008171316236257553,
+            -0.009175300598144531,
+            0.023133469745516777,
+            -0.004807271528989077,
+            0.0019243031274527311,
+            -0.00861055962741375,
+            0.030231080949306488,
+            0.016175301745533943,
+            -0.006484064739197493,
+            0.025768928229808807,
+            -0.004922311753034592,
+            -0.0020567732863128185,
+            -0.012431276962161064,
+            -0.004458666313439608,
+            0.01342828944325447,
+            -0.011643428355455399,
+            0.01600797101855278,
+            -0.04264144226908684,
+            0.0059088654816150665,
+            -0.011022910475730896,
+            -0.005121016874909401,
+            -0.016189245507121086,
+            -0.0171235091984272,
+            -0.030258968472480774,
+            0.02502988465130329,
+            0.00914741214364767,
+            -0.027079686522483826,
+            -0.0005503611755557358,
+            -0.006679283920675516,
+            0.0005150648066774011,
+            -0.01109263114631176,
+            -0.015812750905752182,
+            0.016551796346902847,
+            0.03787251561880112,
+            -0.012005978263914585,
+            -0.01367231272161007,
+            0.003060757415369153,
+            0.008450200781226158,
+            -0.017444226890802383,
+            -0.006375997327268124,
+            0.0019765938632190228,
+            -0.028097614645957947,
+            0.020274903625249863,
+            -0.005751993041485548,
+            0.0031374506652355194,
+            -0.017346616834402084,
+            0.01600797101855278,
+            -0.015394425019621849,
+            -0.0060587660409510136,
+            -0.02253386750817299,
+            -0.023523908108472824,
+            0.016342632472515106,
+            -0.033466141670942307,
+            -0.011315738782286644,
+            -0.01853187568485737,
+            0.004444722086191177,
+            -0.00848506111651659,
+            -0.001322086900472641,
+            0.009000997990369797,
+            0.00419023958966136,
+            0.032852593809366226,
+            -0.00848506111651659,
+            0.004137949086725712,
+            0.03307570144534111,
+            -0.008394423872232437,
+            0.024709166958928108,
+            0.0022641937248408794,
+            0.023328689858317375,
+            0.009454185143113136,
+            0.0013752492377534509,
+            -0.007369523402303457,
+            0.010925300419330597,
+            -0.0014310261467471719,
+            0.03731474652886391,
+            -0.011901396326720715,
+            -0.018922314047813416,
+            0.00861055962741375,
+            -0.004890936892479658,
+            0.006403885781764984,
+            0.02846016362309456,
+            0.019382473081350327,
+            -0.005738048814237118,
+            0.008275898173451424,
+            -0.01157370675355196,
+            -0.0025918828323483467,
+            0.008059762418270111,
+            0.011713149026036263,
+            0.017039842903614044,
+            0.0002762699732556939,
+            -0.019982075318694115,
+            0.02112550102174282,
+            0.0009569224785082042,
+            -0.006428287830203772,
+            -0.015213149599730968,
+            0.0078087663277983665,
+            -0.006972112692892551,
+            0.00922410562634468,
+            -0.00840836763381958,
+            0.016830680891871452,
+            0.0027034366503357887,
+            -0.02526693604886532,
+            0.0031130483839660883,
+            0.00911952368915081,
+            -0.002987550338730216,
+            -0.004950200207531452,
+            -0.016844624653458595,
+            -0.010904384776949883,
+            0.0058530885726213455,
+            0.015812750905752182,
+            -0.0054068733006715775,
+            -0.02628486417233944,
+            -0.005581176374107599,
+            0.00018236432515550405,
+            0.0008924304274842143,
+            0.014167333021759987,
+            0.0018388947937637568,
+            -0.02320319041609764,
+            0.017960162833333015,
+            -0.0017395421164110303,
+            0.004650399088859558,
+            -0.009614543057978153,
+            0.026410363614559174,
+            -0.004660857375711203,
+            -0.011587651446461678,
+            0.010799802839756012,
+            0.034721121191978455,
+            -0.0019452194683253765,
+            0.01868526265025139,
+            -0.02370518259704113,
+            -0.002764442702755332,
+            -0.00901494175195694,
+            0.004193725995719433,
+            -0.01532470341771841,
+            -0.0007220494444482028,
+            0.016133468598127365,
+            0.0023809764534235,
+            -0.010269922204315662,
+            0.002013197634369135,
+            0.024332674220204353,
+            -0.0009063746547326446,
+            0.003261205740272999,
+            0.015687253326177597,
+            0.005731076467782259,
+            0.018629485741257668,
+            0.011476097628474236,
+            -0.03792829439043999,
+            0.002319970401003957,
+            0.010723109357059002,
+            0.0031339647248387337,
+            0.002788844984024763,
+            0.014320719987154007,
+            -0.0053894431330263615,
+            -0.0061807781457901,
+            -0.0030468131881207228,
+            0.013023906387388706,
+            0.009893427602946758,
+            0.008101594634354115,
+            0.0035906380508095026,
+            0.0022746517788618803,
+            -0.018517931923270226,
+            0.013121516443789005,
+            0.0018458669073879719,
+            0.0069023915566504,
+            0.014739046804606915,
+            -0.015980081632733345,
+            -0.005469622556120157,
+            -0.009928288869559765,
+            0.004197211936116219,
+            0.0017953190254047513,
+            -0.004050797317177057,
+            0.004800299648195505,
+            -0.001281997188925743,
+            0.0014101098058745265,
+            -0.014599604532122612,
+            0.006389941554516554,
+            0.01864342950284481,
+            0.0008105080923996866,
+            -0.004981574602425098,
+            0.03661753609776497,
+            -0.01783466525375843,
+            0.010988050140440464,
+            -0.012061755172908306,
+            0.014153389260172844,
+            -0.02597809210419655,
+            0.041358571499586105,
+            0.016858568415045738,
+            0.00477589713409543,
+            0.006686256267130375,
+            -0.01079283095896244,
+            -0.007522909436374903,
+            -0.01218028087168932,
+            0.013769922778010368,
+            0.013846616260707378,
+            -0.015966137871146202,
+            -0.0008793577435426414,
+            -0.0018127493094652891,
+            -0.0011164095485582948,
+            0.02157171629369259,
+            0.006484064739197493,
+            -0.014697213657200336,
+            0.008840639144182205,
+            -0.0018894425593316555,
+            -0.008791834115982056,
+            -0.01869920641183853,
+            -0.008213148452341557,
+            0.014850599691271782,
+            -0.0072161369025707245,
+            -0.0022955681197345257,
+            0.008708168752491474,
+            0.004155379254370928,
+            0.000675423420034349,
+            -0.018420321866869926,
+            0.019577693194150925,
+            -0.011176296509802341,
+            0.006602590903639793,
+            -0.001901643699966371,
+            -0.02684263326227665,
+            0.015617532655596733,
+            -0.019828688353300095,
+            0.015227094292640686,
+            -0.00016624131239950657,
+            -0.027539845556020737,
+            0.0003377117100171745,
+            0.004664343316107988,
+            -0.023342633619904518,
+            -0.020442234352231026,
+            0.005110558588057756,
+            -0.0008105080923996866,
+            -0.004169323481619358,
+            -0.03352191671729088,
+            0.03681275621056557,
+            0.015826696529984474,
+            0.03879283368587494,
+            -0.019312752410769463,
+            -0.006187750026583672,
+            0.0034790842328220606,
+            -0.00667579798027873,
+            0.003147908952087164,
+            -0.026577694341540337,
+            0.016788847744464874,
+            0.022505979984998703,
+            0.013358567841351032,
+            -0.011866535991430283,
+            -0.009635459631681442,
+            -0.004490040708333254,
+            -0.008059762418270111,
+            0.014641436748206615,
+            -0.006672312039881945,
+            -0.013797811232507229,
+            -0.003052042331546545,
+            0.0004989418084733188,
+            0.003393675899133086,
+            0.016733070835471153,
+            -0.004967630375176668,
+            0.006229582708328962,
+            -0.0043436260893940926,
+            0.00048674060963094234,
+            -0.028223112225532532,
+            0.028892435133457184,
+            -0.021195223554968834,
+            0.03692431002855301,
+            -0.005713646300137043,
+            0.003475598292425275,
+            -0.005180279724299908,
+            0.007801793981343508,
+            0.018503988161683083,
+            -0.002971863141283393,
+            -0.010144423693418503,
+            -0.01772311143577099,
+            7.070157880662009e-05,
+            0.015589644201099873,
+            -0.0016062004724517465,
+            -0.0036882476415485144,
+            0.0189502015709877,
+            1.2528014849522151e-05,
+            -0.008708168752491474,
+            -0.009719124995172024,
+            -0.002668576082214713,
+            0.004842132329940796,
+            0.011587651446461678,
+            0.0275119561702013,
+            0.008017929270863533,
+            0.0029178292024880648,
+            0.006184264086186886,
+            -0.001894671586342156,
+            -0.02678685635328293,
+            -0.010067731142044067,
+            0.01266135647892952,
+            -0.029422316700220108,
+            0.0053720129653811455,
+            0.010988050140440464,
+            0.019061755388975143,
+            0.011134464293718338,
+            -0.021055780351161957,
+            -0.0006266186246648431,
+            0.01681673526763916,
+            -0.006145917344838381,
+            0.029756976291537285,
+            -0.01777888834476471,
+            0.011183268390595913,
+            0.026424307376146317,
+            0.018768927082419395,
+            -0.008847611024975777,
+            -0.01797410659492016,
+            0.008303786627948284,
+            0.0072509972378611565,
+            -0.001050174469128251,
+            0.008924304507672787,
+            -0.04043825343251228,
+            0.004451693966984749,
+            -0.01858765259385109,
+            0.0148645443841815,
+            0.004448208026587963,
+            0.011287850327789783,
+            0.006484064739197493,
+            -0.003172311233356595,
+            0.001901643699966371,
+            0.005528885405510664,
+            0.0006017804844304919,
+            -0.0276095662266016,
+            -0.00949601735919714,
+            -0.0085617545992136,
+            0.0012558518210425973,
+            -0.010262950323522091,
+            -0.0010606326395645738,
+            0.01570119708776474,
+            -0.007285858038812876,
+            0.002699950709939003,
+            -0.041051801294088364,
+            -0.013644424267113209,
+            -0.010632472112774849,
+            -0.012089643627405167,
+            -0.011357571929693222,
+            0.019215142354369164,
+            -0.00269820773974061,
+            -0.01010956335812807,
+            0.008331675082445145,
+            0.005469622556120157,
+            -0.030565742403268814,
+            -0.008603586815297604,
+            0.00922410562634468,
+            -0.006247012875974178,
+            0.0040647415444254875,
+            0.005103586707264185,
+            -0.012026894837617874,
+            -0.024890443310141563,
+            0.009956177324056625,
+            -0.012884464114904404,
+            0.016440242528915405,
+            0.019842632114887238,
+            0.0007159488159231842,
+            -0.005535857751965523,
+            0.0073276907205581665,
+            0.01023506186902523,
+            -0.005880977027118206,
+            0.01677490398287773,
+            -0.010820719413459301,
+            0.01635657623410225,
+            0.014223109930753708,
+            0.01844821125268936,
+            -6.938069418538362e-06,
+            -0.014557771384716034,
+            -0.00029609689954668283,
+            0.006749005056917667,
+            0.00505129573866725,
+            -0.00577290914952755,
+            -0.004197211936116219,
+            -0.012354583479464054,
+            0.010932273231446743,
+            -0.007264941465109587,
+            -0.006909363903105259,
+            -0.001948705525137484,
+            0.00015905132750049233,
+            -0.0050756982527673244,
+            -0.004577192012220621,
+            -0.002827191725373268,
+            -0.014766935259103775,
+            0.0006484065088443458,
+            -0.026661358773708344,
+            -0.00863844808191061,
+            -0.007244025357067585,
+            0.0016645919531583786,
+            -0.006341136526316404,
+            0.0030747016426175833,
+            0.04913945123553276,
+            -0.01446016225963831,
+            0.003154881065711379,
+            -0.003660359187051654,
+            -0.02380279265344143,
+            -0.018406378105282784,
+            0.00012734999472741038,
+            -0.009356575086712837,
+            -0.026480084285140038,
+            -0.016370519995689392,
+            0.011992033571004868,
+            -0.0036708174739032984,
+            0.0009708667057566345,
+            -0.010848607867956161,
+            -0.01717928610742092,
+            -0.02643825113773346,
+            -0.005657869391143322,
+            -0.0021142931655049324,
+            -0.010667332448065281,
+            -0.015589644201099873,
+            0.006473606918007135,
+            -0.010653388686478138,
+            0.009781873784959316,
+            0.02360757440328598,
+            0.004364542663097382,
+            0.019312752410769463,
+            0.00901494175195694,
+            -0.018266934901475906,
+            -0.0015434514498338103,
+            -0.0037998014595359564,
+            -0.018015939742326736,
+            0.006693228147923946,
+            0.01066036056727171,
+            -0.006755977403372526,
+            0.011008965782821178,
+            -0.007578686345368624,
+            -0.014725102111697197,
+            -0.004514443222433329,
+            -0.019968131557106972,
+            0.029003988951444626,
+            -0.02582470513880253,
+            -0.005005977116525173,
+            -0.003897411050274968,
+            -0.012849603779613972,
+            0.027079686522483826,
+            0.0128426318988204,
+            -0.019312752410769463,
+            0.024346617981791496,
+            0.0040473113767802715,
+            0.008366535417735577,
+            0.01374900620430708,
+            0.011789842508733273,
+            0.0007586530409753323,
+            0.011322710663080215,
+            0.027372514829039574,
+            -0.0071952203288674355,
+            -0.030258968472480774,
+            -0.002189243445172906,
+            0.01853187568485737,
+            0.014906376600265503,
+            -0.006909363903105259,
+            -0.002267679665237665,
+            -0.00014162104343995452,
+            0.017249006778001785,
+            -0.001689865835942328,
+            -0.000326599896652624,
+            -0.0008667207439430058,
+            -0.009007969871163368,
+            -0.007683268282562494,
+            0.006424801889806986,
+            -0.026173310354351997,
+            -0.0035278890281915665,
+            -0.02224103920161724,
+            0.011350599117577076,
+            -0.013023906387388706,
+            -0.002933516399934888,
+            -0.007725100964307785,
+            -0.005856574513018131,
+            0.027135463431477547,
+            0.021097613498568535,
+            -0.006700200494378805,
+            0.011113547720015049,
+            -0.008687252178788185,
+            -0.012187252752482891,
+            0.014711158350110054,
+            -0.0018476098775863647,
+            0.0014519424876198173,
+            -0.0021282373927533627,
+            0.005769423209130764,
+            -0.011232073418796062,
+            0.0013046566164121032,
+            0.022059764713048935,
+            0.024025900289416313,
+            -0.01367231272161007,
+            -0.020958170294761658,
+            -0.014934265054762363,
+            0.021892433986067772,
+            -0.0010449454421177506,
+            -0.011252989992499352,
+            0.0074741048738360405,
+            0.003100847126916051,
+            -0.010214145295321941,
+            -0.015715142711997032,
+            0.016300799325108528,
+            -0.02137649804353714,
+            -0.014446217566728592,
+            -0.014780879020690918,
+            -0.0036777895875275135,
+            -0.031179288402199745,
+            0.019326696172356606,
+            0.016886457800865173,
+            -0.004817729815840721,
+            -0.002276394749060273,
+            0.01631474308669567,
+            0.00587749108672142,
+            -0.00845717266201973,
+            -0.011720121838152409,
+            -0.013414344750344753,
+            -0.0017273409757763147,
+            0.0030293830204755068,
+            -0.015631476417183876,
+            -0.0017848608549684286,
+            0.0009211904252879322,
+            0.00805279053747654,
+            0.006030877586454153,
+            -0.0018493529642000794,
+            0.0016733070369809866,
+            -0.002238048240542412,
+            0.031792834401130676,
+            0.012975101359188557,
+            -0.00810856744647026,
+            0.007501993328332901,
+            -0.005535857751965523,
+            0.012696217745542526,
+            -0.003604582278057933,
+            -0.023328689858317375,
+            -0.02705179713666439,
+            -0.01382569968700409,
+            -0.004036853089928627,
+            -0.0043436260893940926,
+            -0.006009961012750864,
+            -0.014390440657734871,
+            0.006498008966445923,
+            -0.008157371543347836,
+            0.013281874358654022,
+            0.005197709891945124,
+            -0.002307769376784563,
+            0.014739046804606915,
+            0.01991235464811325,
+            0.012110560201108456,
+            -0.01446016225963831,
+            -0.014390440657734871,
+            0.012438248842954636,
+            -0.008729085326194763,
+            -0.002525647869333625,
+            0.020581677556037903,
+            0.013658368960022926,
+            -0.01436255220323801,
+            -0.0221294853836298,
+            0.005033865571022034,
+            -0.017695222049951553,
+            0.011950201354920864,
+            0.003034612163901329,
+            0.006651395466178656,
+            0.01849004253745079,
+            0.014501994475722313,
+            -0.010444224812090397,
+            0.003442480694502592,
+            0.011427292600274086,
+            -0.013386456295847893,
+            -0.030816737562417984,
+            -0.02518327161669731,
+            -0.004420319572091103,
+            -0.016175301745533943,
+            -0.03120717592537403,
+            0.0006919822189956903,
+            -0.007397411856800318,
+            0.017555780708789825,
+            0.009433268569409847,
+            0.017388449981808662,
+            0.021641438826918602,
+            0.0078087663277983665,
+            0.004950200207531452,
+            0.018657373264431953,
+            0.004699204117059708,
+            -0.0037440245505422354,
+            -0.006745519116520882,
+            -0.004751494619995356,
+            0.004723606165498495,
+            -0.01175498217344284,
+            0.0005673556588590145,
+            -0.011448209173977375,
+            0.020930282771587372,
+            0.018322711810469627,
+            0.005783367436379194,
+            0.0010597611544653773,
+            0.009942232631146908,
+            0.0016881227493286133,
+            0.00901494175195694,
+            0.009161355905234814,
+            -0.010709165595471859,
+            -0.0003250747686251998,
+            -0.013030878268182278,
+            -0.02816733531653881,
+            0.0090428302064538,
+            0.004472610540688038,
+            -0.0005625623743981123,
+            0.008645419962704182,
+            -0.00746016064658761,
+            -0.02380279265344143,
+            -0.0030136958230286837,
+            0.0062051801942288876,
+            -0.013016934506595135,
+            -0.006717630662024021,
+            -0.0014414843171834946,
+            0.007174304220825434,
+            0.023482074961066246,
+            -0.00970518123358488,
+            0.021139446645975113,
+            0.019745023921132088,
+            -0.0003472983662504703,
+            -0.008526894263923168,
+            -0.019745023921132088,
+            -0.033968131989240646,
+            -0.015575699508190155,
+            0.005358068738132715,
+            -0.0305936299264431,
+            -0.005434761755168438,
+            -0.007055778056383133,
+            -0.00016014071297831833,
+            -0.009656376205384731,
+            0.01023506186902523,
+            -0.0001039280541590415,
+            -0.00861055962741375,
+            0.013233070261776447,
+            -0.0034006480127573013,
+            0.005378984846174717,
+            -0.006923308130353689,
+            0.005392929073423147,
+            -0.01436255220323801,
+            -0.0011870021698996425,
+            0.001350846840068698,
+            0.014599604532122612,
+            -0.02056773193180561,
+            -0.011001993902027607,
+            0.0040298812091350555,
+            -0.0010562750976532698,
+            -0.006323706358671188,
+            -0.0201075728982687,
+            0.012668329291045666,
+            -0.03619920834898949,
+            0.008805778808891773,
+            -0.0035122018307447433,
+            0.002835906809195876,
+            -0.006278387736529112,
+            -0.021237054839730263,
+            -0.004430777858942747,
+            0.006337650585919619,
+            -0.019745023921132088,
+            0.0024158370215445757,
+            -0.01975896768271923,
+            0.010939245112240314,
+            -0.007725100964307785,
+            0.016454186290502548,
+            -0.0123615562915802,
+            -5.822803359478712e-05,
+            0.0030747016426175833,
+            0.01451593916863203,
+            -0.002084661740809679,
+            -0.0022397912107408047,
+            0.0024332674220204353,
+            -0.0017630730289965868,
+            0.003043327247723937,
+            -0.010925300419330597,
+            0.006982570979744196,
+            -0.006449204403907061,
+            0.01121812965720892,
+            0.005086156073957682,
+            0.019786855205893517,
+            0.006877989042550325,
+            0.023007972165942192,
+            -0.020846616476774216,
+            0.006072710268199444,
+            -0.015045818872749805,
+            0.018155381083488464,
+            0.013762950897216797,
+            0.02528088167309761,
+            0.011162352748215199,
+            -0.02582470513880253,
+            0.009830678813159466,
+            0.028501996770501137,
+            0.022017931565642357,
+            -0.008519921451807022,
+            -0.01929880864918232,
+            0.0026703192852437496,
+            0.03990837186574936,
+            0.01340040098875761,
+            0.00011972424545092508,
+            -0.011225101538002491,
+            0.016370519995689392,
+            0.00531972199678421,
+            -0.008631475269794464,
+            0.01492032129317522,
+            -0.02588048204779625,
+            0.0006937252474017441,
+            0.005738048814237118,
+            0.011190241202712059,
+            -0.022868528962135315,
+            0.0160637479275465,
+            0.014697213657200336,
+            0.026828689500689507,
+            -0.01980080083012581,
+            0.0015434514498338103,
+            -0.0033065243624150753,
+            0.008164344355463982,
+            -0.0070488061755895615,
+            0.002985807368531823,
+            0.01342828944325447,
+            -0.010883468203246593,
+            -0.04280877113342285,
+            -0.0006793452193960547,
+            0.009265937842428684,
+            0.016691237688064575,
+            -0.0032629487104713917,
+            0.005072211846709251,
+            -0.023970123380422592,
+            -0.020372513681650162,
+            -0.01053486205637455,
+            0.024806777015328407,
+            -0.0014911605976521969,
+            0.028041837736964226,
+            -0.0002359624340897426,
+            -0.009747013449668884,
+            -0.007101096678525209,
+            0.014083667658269405,
+            -0.005044323392212391,
+            0.007948208600282669,
+            0.03182072192430496,
+            0.014808767475187778,
+            0.005194223951548338,
+            -0.010939245112240314,
+            0.03611554577946663,
+            -0.00010321995068807155,
+            -0.005124502815306187,
+            -0.00845717266201973,
+            0.0007756475242786109,
+            0.0192430317401886,
+            -0.008478089235723019,
+            -0.006539841648191214,
+            0.011057770811021328,
+            0.0022310761269181967,
+            -0.0070104594342410564,
+            -0.023621518164873123,
+            0.015227094292640686,
+            -0.010297810658812523,
+            0.014557771384716034,
+            -0.0004311816010158509,
+            0.011615539900958538,
+            -0.031988054513931274,
+            -0.006592132616788149,
+            -0.0010510459542274475,
+            -0.006191235966980457,
+            0.015408368781208992,
+            -0.014780879020690918,
+            -0.009642431512475014,
+            -0.005884462967514992,
+            -0.011022910475730896,
+            0.021139446645975113,
+            0.00843625608831644,
+            -0.007543826010078192,
+            -0.002124751452356577,
+            -0.004336654208600521,
+            0.0011442980030551553,
+            -0.006550299935042858,
+            -0.010576695203781128,
+            -0.02132072113454342,
+            -0.014320719987154007,
+            -0.019842632114887238,
+            -0.01564542017877102,
+            0.018559763208031654,
+            0.009788846597075462,
+            -0.02218526229262352,
+            0.014501994475722313,
+            -0.002930030459538102,
+            0.0009935260750353336,
+            -0.0015112054534256458,
+            0.008903387933969498,
+            0.011134464293718338,
+            -0.024597613140940666,
+            0.0023426299449056387,
+            -0.014404385350644588,
+            -0.00024097364803310484,
+            -0.02847410924732685,
+            0.015868527814745903,
+            -0.017235063016414642,
+            -0.0007159488159231842,
+            0.020205182954669,
+            -0.013532870449125767,
+            -0.00944721233099699,
+            0.00328560802154243,
+            -0.027874507009983063,
+            0.006877989042550325,
+            0.008324702270328999,
+            -0.005661355331540108,
+            -0.027832673862576485,
+            -0.015213149599730968,
+            -0.037816740572452545,
+            -0.010995022021234035,
+            0.02476494386792183,
+            -0.0189502015709877,
+            -0.00014935572107788175,
+            0.009837650693953037,
+            -0.006417830009013414,
+            0.02502988465130329,
+            -0.01119721308350563,
+            0.005675299558788538,
+            -0.013072711415588856,
+            -0.01621713489294052,
+            0.0030102096498012543,
+            0.0037998014595359564,
+            -0.002534362953156233,
+            -0.012521914206445217,
+            7.544043910456821e-05,
+            0.002996265422552824,
+            0.0004237737157382071,
+            0.013693229295313358,
+            -0.0004531873273663223,
+            -0.019424306228756905,
+            0.014153389260172844,
+            -0.028808770701289177,
+            0.0016062004724517465,
+            -0.012926297262310982,
+            -0.009210160933434963,
+            -0.0072509972378611565,
+            0.007188248448073864,
+            0.005166335497051477,
+            0.0037579687777906656,
+            0.005821714177727699,
+            0.001241907593794167,
+            0.013735062442719936,
+            -0.004517929162830114,
+            -0.011406376026570797,
+            0.007662351708859205,
+            -0.016426296904683113,
+            0.02406773343682289,
+            0.01920119859278202,
+            0.0023600601125508547,
+            -0.02416534349322319,
+            0.011078687384724617,
+            0.003097361186519265,
+            -0.0071394434198737144,
+            0.0028237055521458387,
+            0.02324502356350422,
+            -0.004001992754638195,
+            0.010911356657743454,
+            0.001513820025138557,
+            -0.003372759558260441,
+            0.030677296221256256,
+            -0.009342631325125694,
+            -0.003859064308926463,
+            -0.01263346802443266,
+            -0.00851294957101345,
+            0.0051872520707547665,
+            -0.00021875003585591912,
+            -0.004361056722700596,
+            -0.0013212154153734446,
+            -0.006511953193694353,
+            0.02518327161669731,
+            0.00497460225597024,
+            0.0031984567176550627,
+            0.0076414356008172035,
+            -0.01585458405315876,
+            0.009684264659881592,
+            -0.02522510476410389,
+            -0.005497511010617018,
+            -0.022659366950392723,
+            -0.026159366592764854,
+            -0.030760960653424263,
+            -0.02026095986366272,
+            -0.014404385350644588,
+            0.006288845557719469,
+            0.0015190490521490574,
+            -0.007467132527381182,
+            0.0008680280297994614,
+            -0.007237053010612726,
+            -0.005166335497051477,
+            0.01492032129317522,
+            0.01061155553907156,
+            0.014390440657734871,
+            -0.0016184017295017838,
+            0.022324705496430397,
+            -0.004364542663097382,
+            -0.02253386750817299,
+            -0.008066734299063683,
+            -0.007704184390604496,
+            0.015715142711997032,
+            0.011029882356524467,
+            0.015924304723739624,
+            -0.0020236556883901358,
+            -0.004625997040420771,
+            -0.006689742207527161,
+            0.023217136040329933,
+            0.0013238298706710339,
+            -0.012598607689142227,
+            0.003942729905247688,
+            0.02897610142827034,
+            -0.0026668331120163202,
+            0.005832172464579344,
+            0.013686257414519787,
+            0.0101025914773345,
+            -0.010472113266587257,
+            -0.015533867292106152,
+            -0.008387451991438866,
+            -0.005420817527920008,
+            0.0022188748698681593,
+            -0.005692730192095041,
+            -0.006135459057986736,
+            -0.016928289085626602,
+            0.006100598722696304,
+            -0.01342828944325447,
+            -0.005009463056921959,
+            0.010430281050503254,
+            -0.010088646784424782,
+            0.0028254487551748753,
+            0.012933269143104553,
+            0.013302790932357311,
+            -0.000308733870042488,
+            0.02507171779870987,
+            0.0027836160734295845,
+            0.004549303557723761,
+            -0.012152392417192459,
+            -0.020707175135612488,
+            -0.007017431315034628,
+            0.018517931923270226,
+            0.013086655177175999,
+            0.002142181620001793,
+            -0.006588646676391363,
+            -0.02726096101105213,
+            0.0007133343024179339,
+            -0.009865539148449898,
+            0.008429284207522869,
+            -0.034972116351127625,
+            0.023119525983929634,
+            -0.028139447793364525,
+            0.008366535417735577,
+            0.01727689616382122,
+            0.05053387209773064,
+            0.00041789101669564843,
+            0.0023844626266509295,
+            6.50367364869453e-05,
+            0.015575699508190155,
+            0.02152988500893116,
+            -0.0009952690452337265,
+            -0.0021787851583212614,
+            -0.038011956959962845,
+            -0.005309263709932566,
+            -0.000978710362687707,
+            -0.016021914780139923,
+            0.014271914958953857,
+            -0.0015190490521490574,
+            -0.024751000106334686,
+            -0.0039531877264380455,
+            -0.012494025751948357,
+            -0.005054781679064035,
+            0.017193229869008064,
+            0.014752990566194057,
+            -0.015687253326177597,
+            -0.0010815489804372191,
+            -0.019173309206962585,
+            -0.00797609705477953,
+            0.02208765409886837,
+            0.0013586904387921095,
+            0.008010957390069962,
+            0.004406375344842672,
+            0.007746017072349787,
+            0.0013752492377534509,
+            0.01063944399356842,
+            -0.0017534863436594605,
+            0.05111953243613243,
+            -0.016858568415045738,
+            -0.005671813618391752,
+            -0.006860558874905109,
+            0.0004283491871319711,
+            -0.003201942890882492,
+            0.02147410809993744,
+            0.0023478588555008173,
+            0.0015085908817127347,
+            -0.004789841361343861,
+            -0.019368529319763184,
+            -0.0007015688461251557,
+            0.0009351345943287015,
+            0.0014475849457085133,
+            -0.007990040816366673,
+            0.006208666600286961,
+            -0.028390442952513695,
+            -0.015213149599730968,
+            0.008066734299063683,
+            -0.007773905526846647,
+            0.0004348855291027576,
+            -0.005490538664162159,
+            -0.0015164344804361463,
+            0.0007434015278704464,
+            0.022115541622042656,
+            -0.015059763565659523,
+            -0.0059646423906087875,
+            -0.012682273052632809,
+            0.0005586405168287456,
+            -0.029645422473549843,
+            0.003036355134099722,
+            -0.0017473857151344419,
+            0.022073708474636078,
+            -0.02624303288757801,
+            -0.024932274594902992,
+            0.006362053100019693,
+            0.00866633653640747,
+            -0.0028515940066426992,
+            -0.008687252178788185,
+            0.014599604532122612,
+            -0.02203187718987465,
+            0.019996019080281258,
+            0.0018022911390289664,
+            -0.03374502435326576,
+            -0.009258965961635113,
+            0.014501994475722313,
+            -0.01691434532403946,
+            0.00152950722258538,
+            -0.004800299648195505,
+            0.02051195502281189,
+            0.01777888834476471,
+            -0.010011954233050346,
+            -0.017290839925408363,
+            0.0005067854654043913,
+            0.019047811627388,
+            -0.005828686058521271,
+            0.02805578149855137,
+            -0.023886458948254585,
+            0.012298806570470333,
+            -0.024792833253741264,
+            0.01010956335812807,
+            -0.02060956507921219,
+            0.003900896990671754,
+            0.018810760229825974,
+            -0.009921316057443619,
+            0.022450203076004982,
+            -0.016984065994620323,
+            -0.022519923746585846,
+            0.0076414356008172035,
+            -0.010137451812624931,
+            -0.025294825434684753,
+            0.013560758903622627,
+            -0.023593628779053688,
+            -0.013058766722679138,
+            0.0042878491804003716,
+            -0.014613548293709755,
+            -0.011992033571004868,
+            0.015394425019621849,
+            0.004117032513022423,
+            0.004472610540688038,
+            0.0017343130894005299,
+            -0.021041836589574814,
+            -0.013205181807279587,
+            -0.00663396529853344,
+            0.003052042331546545,
+            -0.0020881476812064648,
+            -0.007599602919071913,
+            -0.0359761007130146,
+            0.020079685375094414,
+            -0.01565936580300331,
+            -0.01382569968700409,
+            -0.0037265941500663757,
+            0.019131477922201157,
+            0.009356575086712837,
+            0.005790339782834053,
+            0.011580679565668106,
+            -0.013358567841351032,
+            -0.007244025357067585,
+            0.006034363526850939,
+            0.018420321866869926,
+            -0.009698208421468735,
+            -0.004273904953151941,
+            -0.010346615687012672,
+            0.006138945464044809,
+            -0.002053287113085389,
+            -0.0039531877264380455,
+            0.00914741214364767,
+            0.011908368207514286,
+            -0.004486554767936468,
+            0.006752490997314453,
+            -0.008945221081376076,
+            0.012187252752482891,
+            -0.017541835084557533,
+            0.0074531883001327515,
+            0.006783865857869387,
+            0.00292480131611228,
+            0.007024403661489487,
+            -0.007202192675322294,
+            0.021097613498568535,
+            0.03586454689502716,
+            0.000885022571310401,
+            0.014962153509259224,
+            0.00420418381690979,
+            0.0039357575587928295,
+            -0.00036712532164528966,
+            0.021655382588505745,
+            -0.005880977027118206,
+            -0.009733069688081741,
+            -0.00405428372323513,
+            -0.014069723896682262,
+            0.01793227344751358,
+            -0.00866633653640747,
+            -0.0015608817338943481,
+            -0.03525100275874138,
+            0.008896416053175926,
+            -0.003384960815310478,
+            -0.005535857751965523,
+            -0.012194225564599037,
+            -0.009942232631146908,
+            0.02310558222234249,
+            -0.0035732078831642866,
+            0.026521917432546616,
+            -0.00013704558659810573,
+            0.027525901794433594,
+            -0.011455181054770947,
+            -0.00563695328310132,
+            -0.015729086473584175,
+            -0.004312251694500446,
+            0.015087652020156384,
+            0.00485259061679244,
+            0.02922709658741951,
+            0.012821715325117111,
+            -0.0006344622815959156,
+            0.010123508051037788,
+            0.004538845270872116,
+            0.01989840902388096,
+            0.015826696529984474,
+            0.013811754994094372,
+            -0.0033535861875861883,
+            -0.0060378494672477245,
+            0.006710658315569162,
+            0.02081872895359993,
+            0.0016515192110091448,
+            -0.00746016064658761,
+            0.002637201687321067,
+            -0.0071220132522284985,
+            0.016844624653458595,
+            0.011406376026570797,
+            0.026047812774777412,
+            -0.014006974175572395,
+            0.005281375255435705,
+            0.006198208313435316,
+            0.0019853091798722744,
+            0.003897411050274968,
+            -0.0078087663277983665,
+            0.02218526229262352,
+            0.00810856744647026,
+            0.003426793497055769,
+            0.0020637453999370337,
+            -0.004639940802007914,
+            -0.000789156008977443,
+            -0.0053894431330263615,
+            -0.012724106200039387,
+            0.005487052723765373,
+            0.004259960725903511,
+            -0.022352593019604683,
+            -0.0026162853464484215,
+            -0.0138047831133008,
+            0.007704184390604496,
+            0.018322711810469627,
+            0.017235063016414642,
+            0.003268177853897214,
+            -0.0026825203094631433,
+            -0.003932271618396044,
+            0.00030306901317089796,
+            0.008024902082979679,
+            0.0025465642102062702,
+            0.016537850722670555,
+            -0.024583669379353523,
+            -0.01878287084400654,
+            0.010060758329927921,
+            -0.006281873676925898,
+            -0.01311454363167286,
+            0.005860060919076204
+          ],
+          "keyphrases": [
+            "fine-tuning",
+            "dense model",
+            "MoE model",
+            "expert choice routing",
+            "downstream performance"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "375257d5-5932-4d64-8b2a-1c5c6b719117",
+      "type": "next",
+      "source": {
+        "id": "dea84165-aef9-4273-9988-e3d233346670",
+        "properties": {
+          "page_content": "A Comparison On Fine-Tuning With A Dense Model\n\nOur 8B MoE model achieves stronger pre-training perplexity than its dense counterpart. However, a better perplexity does not always directly translate to downstream performance as demonstrated in Section 4.4. To this end, we compare fine-tuning performance of the 8B dense model and MoE model in Table 1. As shown in the table, our MoE model using expert choice routing consistently outperforms the dense model across the 11 tasks in GLUE and SuperGLUE.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 1: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\n",
+          "title": "A Comparison On Fine-Tuning With A Dense Model",
+          "embedding": [
+            0.016482073813676834,
+            -0.002563994377851486,
+            -0.0267450250685215,
+            -0.023007972165942192,
+            0.025922315195202827,
+            0.018462155014276505,
+            -0.01591036096215248,
+            0.03547411039471626,
+            -0.012068727053701878,
+            -0.011176296509802341,
+            0.003149651922285557,
+            -0.008659363724291325,
+            -0.021000003442168236,
+            0.016844624653458595,
+            -0.02533665858209133,
+            0.032490044832229614,
+            0.024039844051003456,
+            0.01954980380833149,
+            0.010472113266587257,
+            -0.021557772532105446,
+            -0.0034128492698073387,
+            -0.011873507872223854,
+            -0.011252989992499352,
+            -0.022938251495361328,
+            0.01864342950284481,
+            -0.007376495283097029,
+            0.017639445140957832,
+            -0.010311754420399666,
+            -0.029952196404337883,
+            0.032490044832229614,
+            0.012884464114904404,
+            0.014557771384716034,
+            0.02639641799032688,
+            -0.006728088948875666,
+            -0.05795220285654068,
+            -0.010625500231981277,
+            0.0629163458943367,
+            0.01878287084400654,
+            -0.027177294716238976,
+            0.06062949076294899,
+            0.024179287254810333,
+            0.018978090956807137,
+            -0.004779383074492216,
+            -0.004329681862145662,
+            0.01492032129317522,
+            -0.030314745381474495,
+            -0.018476098775863647,
+            -0.0148645443841815,
+            0.03494422882795334,
+            -0.03650598227977753,
+            -0.010472113266587257,
+            -0.01666335016489029,
+            0.018657373264431953,
+            0.011469125747680664,
+            -0.0498366616666317,
+            0.0530996099114418,
+            0.006975598633289337,
+            0.01773705519735813,
+            0.04239044710993767,
+            0.005661355331540108,
+            -0.007543826010078192,
+            -0.0029038849752396345,
+            -0.026758968830108643,
+            -0.02917131967842579,
+            0.011315738782286644,
+            0.015966137871146202,
+            -0.01385358814150095,
+            -0.02132072113454342,
+            0.03890438750386238,
+            0.01121812965720892,
+            -0.05678088590502739,
+            0.05452192202210426,
+            -0.025239048525691032,
+            0.018099604174494743,
+            -0.009293826296925545,
+            -0.03438645973801613,
+            0.014320719987154007,
+            0.01945219375193119,
+            0.022770920768380165,
+            0.0009926545899361372,
+            -0.012201197445392609,
+            -0.010939245112240314,
+            -0.0014963897410780191,
+            0.009684264659881592,
+            0.03611554577946663,
+            -0.026758968830108643,
+            -0.017848609015345573,
+            -0.05254184082150459,
+            -0.02526693604886532,
+            -0.005563745740801096,
+            -0.006602590903639793,
+            -0.010318727232515812,
+            -0.011308766901493073,
+            0.020693231374025345,
+            0.007257969584316015,
+            0.042920324951410294,
+            -0.004256474785506725,
+            0.01883864775300026,
+            -0.02076295204460621,
+            -0.008903387933969498,
+            0.00044708672794513404,
+            0.03296414762735367,
+            0.018252991139888763,
+            0.004782869480550289,
+            0.04950200021266937,
+            -0.015394425019621849,
+            0.020288847386837006,
+            0.007181276101619005,
+            0.0038242037408053875,
+            0.06832670420408249,
+            -0.005229084752500057,
+            -0.021892433986067772,
+            -0.03126295283436775,
+            0.008854582905769348,
+            -0.014794823713600636,
+            -0.00372310820966959,
+            -0.02597809210419655,
+            -0.011713149026036263,
+            0.04013147950172424,
+            -0.027637455612421036,
+            0.0076762959361076355,
+            0.027972117066383362,
+            0.019033867865800858,
+            -0.03313148021697998,
+            -0.03466534614562988,
+            0.045011959969997406,
+            0.02066534198820591,
+            0.0014536854578182101,
+            0.008087650872766972,
+            -0.014237054623663425,
+            -0.001226220279932022,
+            0.012473110109567642,
+            -0.014188249595463276,
+            -0.02958964742720127,
+            0.030454188585281372,
+            0.0055742040276527405,
+            0.0256852637976408,
+            0.03028685785830021,
+            -0.03555777668952942,
+            -0.03354980796575546,
+            0.019745023921132088,
+            -0.05968128517270088,
+            0.016440242528915405,
+            0.011992033571004868,
+            -0.004497012589126825,
+            0.010904384776949883,
+            0.0532948300242424,
+            0.08717929571866989,
+            -0.013016934506595135,
+            -0.004096116404980421,
+            -0.02710757404565811,
+            0.027065740898251534,
+            0.004817729815840721,
+            0.036394428461790085,
+            -0.007495020981878042,
+            -0.010953188873827457,
+            -0.02603386901319027,
+            0.0032385464292019606,
+            -0.013477093540132046,
+            -0.02132072113454342,
+            0.011280878446996212,
+            0.008241036906838417,
+            -0.004657371435314417,
+            -0.008310758508741856,
+            0.0006462276796810329,
+            0.03182072192430496,
+            0.0277071762830019,
+            -0.007313746493309736,
+            -0.07591236382722855,
+            0.031681280583143234,
+            0.011057770811021328,
+            0.030454188585281372,
+            0.0072161369025707245,
+            0.020037852227687836,
+            0.019061755388975143,
+            0.02533665858209133,
+            0.006578188389539719,
+            -0.014976098202168941,
+            -0.030482076108455658,
+            0.024374505504965782,
+            -0.01294721383601427,
+            -0.027986060827970505,
+            0.041916340589523315,
+            -0.01662151701748371,
+            0.013023906387388706,
+            -0.0030241538770496845,
+            0.010855579748749733,
+            0.025294825434684753,
+            -0.001602714415639639,
+            0.017848609015345573,
+            -0.00970518123358488,
+            0.045820724219083786,
+            0.03028685785830021,
+            -0.01878287084400654,
+            -0.007780877873301506,
+            -0.039992038160562515,
+            -0.024025900289416313,
+            0.01833665743470192,
+            0.01068824902176857,
+            0.017346616834402084,
+            -0.005978586617857218,
+            -0.03592032566666603,
+            -0.006846614647656679,
+            -0.021460162475705147,
+            0.02320319041609764,
+            -0.0015730829909443855,
+            0.0327131524682045,
+            -0.0027417833916842937,
+            -0.003228088142350316,
+            -0.016175301745533943,
+            0.024388451129198074,
+            -0.023998012766242027,
+            -0.020177293568849564,
+            -0.021111557260155678,
+            -0.01812749356031418,
+            0.022715143859386444,
+            0.004165837541222572,
+            -0.002523904899135232,
+            -0.022450203076004982,
+            -0.0008218378061428666,
+            0.008729085326194763,
+            0.028557773679494858,
+            -0.00485259061679244,
+            -0.022464146837592125,
+            -0.05692032724618912,
+            -0.03388446941971779,
+            0.021111557260155678,
+            -0.02998008392751217,
+            0.022603590041399002,
+            -0.012668329291045666,
+            -0.022464146837592125,
+            0.009698208421468735,
+            0.003107819240540266,
+            0.02350996434688568,
+            -0.006414343602955341,
+            -0.02526693604886532,
+            0.018211157992482185,
+            -0.020274903625249863,
+            0.013260958716273308,
+            0.029254985973238945,
+            -0.017262950539588928,
+            0.014111556112766266,
+            -0.03525100275874138,
+            -0.01833665743470192,
+            0.03937849402427673,
+            -0.010486057959496975,
+            -0.006825698539614677,
+            -0.0072509972378611565,
+            0.006532869767397642,
+            0.0019382473547011614,
+            0.005790339782834053,
+            -0.037760961800813675,
+            0.0075089652091264725,
+            0.009795818477869034,
+            0.023523908108472824,
+            0.01970319077372551,
+            0.05895618721842766,
+            -0.04685259610414505,
+            -0.01579880714416504,
+            -0.03965737670660019,
+            0.05374104529619217,
+            0.010297810658812523,
+            -0.03965737670660019,
+            0.015868527814745903,
+            -0.020651398226618767,
+            -0.01768127828836441,
+            0.003187998663634062,
+            0.009942232631146908,
+            -0.014850599691271782,
+            0.033159367740154266,
+            0.01281474344432354,
+            -0.021599605679512024,
+            -0.04947411268949509,
+            0.0050024907104671,
+            0.004908367525786161,
+            -0.019466139376163483,
+            0.010841635055840015,
+            0.004577192012220621,
+            0.0018894425593316555,
+            0.0021927293855696917,
+            0.023677295073866844,
+            0.03257371112704277,
+            0.023886458948254585,
+            0.02978486567735672,
+            -0.006905877497047186,
+            0.0050756982527673244,
+            -0.02279880829155445,
+            -0.01053486205637455,
+            -0.03053785301744938,
+            -0.0037265941500663757,
+            -0.009342631325125694,
+            -0.020944226533174515,
+            -0.024695223197340965,
+            -0.005731076467782259,
+            -0.006975598633289337,
+            0.007578686345368624,
+            -0.010827691294252872,
+            0.005399901419878006,
+            0.04952988773584366,
+            -0.03561355173587799,
+            -0.043478094041347504,
+            -0.007045319769531488,
+            0.02031673677265644,
+            0.002699950709939003,
+            -0.05951395258307457,
+            0.008854582905769348,
+            0.018350601196289062,
+            0.0024506975896656513,
+            -0.02405378967523575,
+            -0.02391434647142887,
+            0.026117533445358276,
+            -0.008729085326194763,
+            -0.013023906387388706,
+            -0.053490050137043,
+            0.0010527890408411622,
+            0.005856574513018131,
+            0.0057973116636276245,
+            0.03184860944747925,
+            -0.03873705863952637,
+            -0.004577192012220621,
+            0.01949402689933777,
+            0.013937253504991531,
+            -0.07686056941747665,
+            0.014627492986619473,
+            -0.004148406907916069,
+            0.027874507009983063,
+            0.045458175241947174,
+            0.039183273911476135,
+            -0.01941036246716976,
+            -6.231325824046507e-05,
+            0.0032559765968471766,
+            -0.0519561842083931,
+            -0.029645422473549843,
+            0.007076694630086422,
+            0.006240040995180607,
+            -0.034776899963617325,
+            -0.0442589707672596,
+            -0.0073067741468548775,
+            -0.02589442767202854,
+            0.013637452386319637,
+            -0.011685260571539402,
+            0.009907372295856476,
+            -0.034776899963617325,
+            0.010095619596540928,
+            0.018824703991413116,
+            -0.009670319966971874,
+            -0.026354586705565453,
+            -0.03572510555386543,
+            0.01180378720164299,
+            -0.017095619812607765,
+            0.0007935136090964079,
+            0.021348608657717705,
+            0.006473606918007135,
+            0.005654383450746536,
+            -0.007515937555581331,
+            -0.03564143925905228,
+            -0.006836156360805035,
+            -0.005201196298003197,
+            0.020400401204824448,
+            0.0342470183968544,
+            -0.011176296509802341,
+            -0.009021913632750511,
+            -0.011036854237318039,
+            0.04741036519408226,
+            -0.05148208141326904,
+            -0.009265937842428684,
+            0.01864342950284481,
+            -0.006484064739197493,
+            0.008143427781760693,
+            0.005086156073957682,
+            -0.026675302535295486,
+            0.02461155876517296,
+            0.01756972447037697,
+            0.0110647426918149,
+            0.028529886156320572,
+            0.00459810858592391,
+            0.030565742403268814,
+            -0.002150896703824401,
+            -0.03064940683543682,
+            0.014669325202703476,
+            -0.020400401204824448,
+            0.02735857106745243,
+            -0.015784863382577896,
+            -0.004960658028721809,
+            -0.008401395753026009,
+            0.032490044832229614,
+            0.01812749356031418,
+            0.005263945087790489,
+            -0.04149801656603813,
+            0.016705181449651718,
+            0.009161355905234814,
+            -0.025852594524621964,
+            0.009516933932900429,
+            0.009565738961100578,
+            0.0181832704693079,
+            0.01777888834476471,
+            -0.023217136040329933,
+            -0.0037056778091937304,
+            0.00459810858592391,
+            -0.013365539722144604,
+            -0.022701198235154152,
+            0.024597613140940666,
+            0.003393675899133086,
+            -0.01849004253745079,
+            0.003433765610679984,
+            0.016119524836540222,
+            0.017918329685926437,
+            0.0071220132522284985,
+            0.008234065026044846,
+            -0.04052191972732544,
+            -0.0022188748698681593,
+            -0.01960558071732521,
+            -0.005361554678529501,
+            0.017792832106351852,
+            0.0275119561702013,
+            0.05123108625411987,
+            0.00025666088913567364,
+            0.03321514651179314,
+            -0.0030642435885965824,
+            0.0009246764821000397,
+            -0.036087654531002045,
+            0.011287850327789783,
+            0.03140239417552948,
+            0.01526892650872469,
+            0.028027893975377083,
+            0.030733073130249977,
+            -0.029338650405406952,
+            0.010262950323522091,
+            0.02005179598927498,
+            0.008373507298529148,
+            0.021460162475705147,
+            -0.005542829632759094,
+            0.034721121191978455,
+            -0.009698208421468735,
+            0.007257969584316015,
+            -0.011985061690211296,
+            -0.017039842903614044,
+            0.022729087620973587,
+            -0.03985259681940079,
+            -0.0009752242476679385,
+            0.022659366950392723,
+            -0.021794823929667473,
+            0.001286354847252369,
+            -0.01939641684293747,
+            -0.0075298817828297615,
+            0.009586654603481293,
+            -0.03586454689502716,
+            0.01934063993394375,
+            -0.021446218714118004,
+            -0.014878488145768642,
+            -0.039183273911476135,
+            -0.0016332174418494105,
+            -0.06866136938333511,
+            8.072399214142933e-05,
+            0.031625501811504364,
+            -0.0016846366925165057,
+            0.023691238835453987,
+            0.007006973493844271,
+            0.014132472686469555,
+            -0.055386465042829514,
+            -0.0040473113767802715,
+            -0.0073625510558485985,
+            0.014432273805141449,
+            -0.009028886444866657,
+            -0.03299203887581825,
+            -0.011922312900424004,
+            -0.0023374008014798164,
+            0.004071713890880346,
+            0.0032472615130245686,
+            0.007599602919071913,
+            -0.029756976291537285,
+            -0.013330679386854172,
+            -0.03854183852672577,
+            -0.010262950323522091,
+            -0.004490040708333254,
+            0.038932278752326965,
+            0.0009220619103871286,
+            -0.03976893052458763,
+            0.008750001899898052,
+            0.021948210895061493,
+            0.00022212715703062713,
+            -0.019745023921132088,
+            0.010423308238387108,
+            -0.042278893291950226,
+            0.02487649768590927,
+            -0.012396416626870632,
+            0.019187254831194878,
+            -0.0077529894188046455,
+            0.014223109930753708,
+            -0.0075647421181201935,
+            -0.009691236540675163,
+            -0.009175300598144531,
+            0.012326695024967194,
+            -0.027930283918976784,
+            -0.009454185143113136,
+            -0.03115139901638031,
+            0.009231077507138252,
+            -0.005284861661493778,
+            0.008896416053175926,
+            0.02795817144215107,
+            -0.004960658028721809,
+            -0.005005977116525173,
+            0.003339641960337758,
+            0.038011956959962845,
+            0.004730578511953354,
+            0.006041335873305798,
+            0.027037853375077248,
+            0.04423108324408531,
+            -0.00539641547948122,
+            -0.00426693307235837,
+            0.02486255392432213,
+            -0.010214145295321941,
+            0.016035858541727066,
+            -0.0133237075060606,
+            -0.006267929449677467,
+            0.027637455612421036,
+            -0.026159366592764854,
+            0.04280877113342285,
+            -0.008519921451807022,
+            0.017541835084557533,
+            -0.025964148342609406,
+            -0.01733267307281494,
+            -0.003034612163901329,
+            0.03686853125691414,
+            -0.000990040018223226,
+            0.006226096767932177,
+            0.017500003799796104,
+            0.0028411359526216984,
+            0.03742630034685135,
+            -0.02861355058848858,
+            -0.04473307356238365,
+            0.012215141206979752,
+            0.02547609992325306,
+            -0.008896416053175926,
+            -0.04094024747610092,
+            -0.020860562101006508,
+            -0.004908367525786161,
+            0.007766933646053076,
+            -0.03276893123984337,
+            -0.0037126499228179455,
+            0.014557771384716034,
+            0.0040089646354317665,
+            0.010848607867956161,
+            0.00960059929639101,
+            0.014153389260172844,
+            0.02851594053208828,
+            0.006755977403372526,
+            -0.03865339234471321,
+            0.0037126499228179455,
+            0.018880480900406837,
+            -0.014237054623663425,
+            -0.011497014202177525,
+            -0.02653586119413376,
+            0.0009717381908558309,
+            -0.009077690541744232,
+            -0.029756976291537285,
+            -0.013149404898285866,
+            0.006156375631690025,
+            -0.013205181807279587,
+            -0.012005978263914585,
+            -0.01853187568485737,
+            0.018559763208031654,
+            -0.002811504527926445,
+            0.01681673526763916,
+            -0.010681277140974998,
+            -0.013491038233041763,
+            0.014027890749275684,
+            0.021794823929667473,
+            -0.04225100204348564,
+            0.0009682521340437233,
+            -0.015812750905752182,
+            0.02734462544322014,
+            -0.005061754025518894,
+            0.01172709371894598,
+            0.012772910296916962,
+            -0.009398408234119415,
+            0.01180378720164299,
+            -0.01950797066092491,
+            -0.024346617981791496,
+            0.01127390656620264,
+            0.030203191563487053,
+            -0.01676095835864544,
+            -0.010276894085109234,
+            0.011741037480533123,
+            0.025615543127059937,
+            -0.023774905130267143,
+            -0.03349402919411659,
+            -0.030816737562417984,
+            0.03151394799351692,
+            -0.007843626663088799,
+            0.014188249595463276,
+            0.027623509988188744,
+            -0.0002780130016617477,
+            -0.03622709959745407,
+            -0.011176296509802341,
+            -0.03182072192430496,
+            -0.01945219375193119,
+            0.011343627236783504,
+            0.009426296688616276,
+            -0.03207171708345413,
+            -0.003395418869331479,
+            0.01839243434369564,
+            0.029756976291537285,
+            0.013421316631138325,
+            0.008296813815832138,
+            0.004503984935581684,
+            0.038876499980688095,
+            0.027023909613490105,
+            -0.003834662027657032,
+            0.0009926545899361372,
+            0.006288845557719469,
+            0.007836654782295227,
+            -0.017764942720532417,
+            0.0006161604542285204,
+            -0.010276894085109234,
+            -0.0008305529481731355,
+            -0.019480083137750626,
+            -0.006264443509280682,
+            0.00691633578389883,
+            -0.007202192675322294,
+            0.05968128517270088,
+            -0.0037928293459117413,
+            0.03449801355600357,
+            -0.018768927082419395,
+            0.026019925251603127,
+            0.04264144226908684,
+            -0.01169920526444912,
+            -0.02903187833726406,
+            -0.010395419783890247,
+            -0.026075702160596848,
+            -0.04492829367518425,
+            0.009928288869559765,
+            -0.015994027256965637,
+            -0.022784864529967308,
+            -0.011427292600274086,
+            -0.014069723896682262,
+            0.002382719423621893,
+            0.012535858899354935,
+            -0.012250002473592758,
+            0.001576569047756493,
+            -0.00861055962741375,
+            0.03301992639899254,
+            -0.0013456178130581975,
+            0.011866535991430283,
+            0.006470120511949062,
+            0.003243775339797139,
+            -0.012605579569935799,
+            -0.0037196220364421606,
+            0.014188249595463276,
+            -0.00018835598893929273,
+            0.014195221476256847,
+            -0.001346489298157394,
+            0.019466139376163483,
+            0.03513944894075394,
+            -0.04046614095568657,
+            -0.027023909613490105,
+            -0.006250499282032251,
+            -0.04955777898430824,
+            0.01650996319949627,
+            0.00397759024053812,
+            0.011587651446461678,
+            0.00866633653640747,
+            0.008875499479472637,
+            -0.03140239417552948,
+            0.024374505504965782,
+            -0.011071715503931046,
+            -0.015213149599730968,
+            0.022296816110610962,
+            -0.0026250004302710295,
+            0.012487053871154785,
+            0.02102789282798767,
+            0.0008022287511266768,
+            -0.041107576340436935,
+            -0.0033117535058408976,
+            0.01874103955924511,
+            -0.028195224702358246,
+            -0.00014674117846880108,
+            0.009035858325660229,
+            0.00034076202427968383,
+            0.0032559765968471766,
+            -0.013163348659873009,
+            0.0034965146332979202,
+            -0.012222114019095898,
+            -0.043478094041347504,
+            0.026061756536364555,
+            -0.002492530271410942,
+            -0.03165339305996895,
+            0.0363665409386158,
+            -0.019173309206962585,
+            -0.009238049387931824,
+            0.008791834115982056,
+            -0.03988048434257507,
+            -0.032796818763017654,
+            0.01677490398287773,
+            -0.017346616834402084,
+            0.009886455722153187,
+            -0.04077291488647461,
+            0.006686256267130375,
+            0.013644424267113209,
+            -0.0008780504576861858,
+            -0.011399404145777225,
+            0.04163745790719986,
+            -0.022980082780122757,
+            -0.004368028603494167,
+            0.0128426318988204,
+            -0.027121517807245255,
+            0.004043825436383486,
+            -0.0054277898743748665,
+            -0.015394425019621849,
+            0.009865539148449898,
+            -0.021111557260155678,
+            0.018964147195219994,
+            0.02330080047249794,
+            0.0016175301279872656,
+            0.022213151678442955,
+            -0.011615539900958538,
+            -0.0015417084796354175,
+            0.019005978479981422,
+            -0.005473108496516943,
+            0.047382477670907974,
+            0.0022816238924860954,
+            0.030760960653424263,
+            0.013337651267647743,
+            -0.024806777015328407,
+            0.0023426299449056387,
+            0.023886458948254585,
+            0.004622510634362698,
+            0.0055219135247170925,
+            0.008122511208057404,
+            0.007209164556115866,
+            -0.02101394720375538,
+            0.0024071219377219677,
+            -0.03600399196147919,
+            0.006030877586454153,
+            -0.0008009214652702212,
+            -0.018225103616714478,
+            -0.008331675082445145,
+            0.02872510440647602,
+            0.025545820593833923,
+            -0.01849004253745079,
+            0.02268725447356701,
+            -0.012731078080832958,
+            -0.002875996520742774,
+            0.011427292600274086,
+            0.002492530271410942,
+            0.023482074961066246,
+            -0.005256973206996918,
+            0.003545319428667426,
+            0.010674304328858852,
+            0.006410857662558556,
+            0.014501994475722313,
+            0.0014685012865811586,
+            0.00866633653640747,
+            0.00848506111651659,
+            -0.038876499980688095,
+            0.011824702844023705,
+            0.03580877184867859,
+            -0.004193725995719433,
+            -0.013281874358654022,
+            -0.01945219375193119,
+            -0.008220121264457703,
+            0.010255977511405945,
+            0.017667334526777267,
+            0.02091633901000023,
+            0.007718128617852926,
+            0.0057798814959824085,
+            -0.029115542769432068,
+            0.021111557260155678,
+            0.010276894085109234,
+            -0.01157370675355196,
+            -0.002255478408187628,
+            0.0025134466122835875,
+            0.04660160094499588,
+            -0.05845419317483902,
+            0.004859562497586012,
+            0.016551796346902847,
+            -0.0180856604129076,
+            -0.002550050150603056,
+            -0.02422112040221691,
+            -0.001696837949566543,
+            -0.004978088662028313,
+            0.02614542283117771,
+            -0.013107571750879288,
+            0.0021840143017470837,
+            -0.03555777668952942,
+            0.009203189052641392,
+            -0.001525149680674076,
+            -0.012417333200573921,
+            0.00012604272342287004,
+            0.002182271331548691,
+            -0.021334664896130562,
+            -0.0246255025267601,
+            0.0032507474534213543,
+            0.014069723896682262,
+            0.03215538337826729,
+            -0.006567730102688074,
+            0.0011983318254351616,
+            0.03714741766452789,
+            -0.0256852637976408,
+            -0.014683269895613194,
+            0.026061756536364555,
+            -0.005229084752500057,
+            0.03505578264594078,
+            0.02542032301425934,
+            -0.03120717592537403,
+            -0.05243028700351715,
+            -0.007264941465109587,
+            -0.011566734872758389,
+            0.01010956335812807,
+            0.009454185143113136,
+            0.032545823603868484,
+            -0.0001861772034317255,
+            -0.0030328689608722925,
+            -0.009691236540675163,
+            -0.01823904737830162,
+            0.034776899963617325,
+            0.00019924991647712886,
+            0.01743028126657009,
+            0.004514443222433329,
+            -0.018601596355438232,
+            -0.025699207559227943,
+            0.013037851080298424,
+            0.003547062398865819,
+            -0.005121016874909401,
+            -0.05212351307272911,
+            0.006982570979744196,
+            0.005054781679064035,
+            0.006173805799335241,
+            0.0377051867544651,
+            0.015994027256965637,
+            -0.0001385707437293604,
+            -0.0017822462832555175,
+            -0.0192430317401886,
+            0.029254985973238945,
+            -0.029729088768363,
+            0.04052191972732544,
+            0.0009011455695144832,
+            0.012326695024967194,
+            0.008791834115982056,
+            0.03221116214990616,
+            -0.017262950539588928,
+            -0.003604582278057933,
+            -0.02339841052889824,
+            -0.018420321866869926,
+            0.009154384024441242,
+            -0.020749008283019066,
+            0.0057066744193434715,
+            -0.007739045191556215,
+            0.0037649408914148808,
+            0.0038381479680538177,
+            0.008366535417735577,
+            0.05220717936754227,
+            0.026270920410752296,
+            0.0006104956264607608,
+            -0.024025900289416313,
+            -0.010813746601343155,
+            -0.016928289085626602,
+            0.005159363616257906,
+            -0.020498011261224747,
+            -0.007739045191556215,
+            -0.032099608331918716,
+            -0.014613548293709755,
+            -0.014529882930219173,
+            -0.025754984468221664,
+            0.00850597769021988,
+            0.014641436748206615,
+            -0.006424801889806986,
+            -0.011176296509802341,
+            -0.019591636955738068,
+            -0.033661361783742905,
+            -0.02081872895359993,
+            -0.009035858325660229,
+            -0.01430677529424429,
+            0.029254985973238945,
+            -0.0028324208687990904,
+            -0.006334164645522833,
+            -0.006623507011681795,
+            -0.030063750222325325,
+            0.018615540117025375,
+            -0.018266934901475906,
+            -0.00789940357208252,
+            -0.012012950144708157,
+            0.02741434797644615,
+            0.0028637952636927366,
+            0.046573713421821594,
+            0.002319970401003957,
+            -0.014209166169166565,
+            -0.05190040543675423,
+            -0.03491634130477905,
+            -0.02947809360921383,
+            0.016886457800865173,
+            0.0032071718014776707,
+            0.0010702193249017,
+            0.002781872870400548,
+            0.006410857662558556,
+            0.002318227430805564,
+            -0.0056508975103497505,
+            -0.029896419495344162,
+            0.0032298313453793526,
+            0.010388447903096676,
+            0.005009463056921959,
+            -0.035083670169115067,
+            -0.021083669736981392,
+            -0.018364544957876205,
+            0.038876499980688095,
+            -0.03742630034685135,
+            0.03438645973801613,
+            -0.011510957963764668,
+            -0.01814143732190132,
+            0.004249502904713154,
+            -0.027037853375077248,
+            0.006121514830738306,
+            0.006292331963777542,
+            0.007578686345368624,
+            0.03600399196147919,
+            0.01837848871946335,
+            0.043840646743774414,
+            -0.0286693274974823,
+            0.03820717707276344,
+            -0.01326793059706688,
+            -0.0006126743974164128,
+            -0.0005874005146324635,
+            0.0038799806497991085,
+            0.02922709658741951,
+            -0.002659860998392105,
+            -0.0006985185318626463,
+            -0.0050931284204125404,
+            -0.004315737634897232,
+            -0.005309263709932566,
+            0.003034612163901329,
+            -0.025657374411821365,
+            0.012068727053701878,
+            0.024904387071728706,
+            -0.015338648110628128,
+            0.03628287464380264,
+            -0.004674801602959633,
+            0.03756574168801308,
+            -0.013511954806745052,
+            -0.027525901794433594,
+            0.027135463431477547,
+            -0.01570119708776474,
+            -0.013525898568332195,
+            0.005086156073957682,
+            0.0018092632526531816,
+            -0.0015643677907064557,
+            -0.02097211591899395,
+            0.024235064163804054,
+            0.02487649768590927,
+            0.003939243499189615,
+            0.0038102595135569572,
+            -0.02953387051820755,
+            -0.009984065778553486,
+            0.0181832704693079,
+            -0.004904881119728088,
+            0.020302793011069298,
+            0.02320319041609764,
+            -0.014118527993559837,
+            0.02193426713347435,
+            -0.01050697360187769,
+            0.016384465619921684,
+            0.022352593019604683,
+            -0.014069723896682262,
+            -0.016802791506052017,
+            -0.020274903625249863,
+            -0.004019422922283411,
+            0.004388945177197456,
+            -0.02284064143896103,
+            -0.03159761428833008,
+            0.0028463650960475206,
+            0.02958964742720127,
+            0.007108069024980068,
+            -0.03658964857459068,
+            -0.005738048814237118,
+            -0.017346616834402084,
+            0.030984068289399147,
+            -0.026521917432546616,
+            -0.0007778263534419239,
+            -0.010576695203781128,
+            -0.001526892650872469,
+            0.012005978263914585,
+            0.04088446870446205,
+            -0.0016088150441646576,
+            0.03095618076622486,
+            0.0171235091984272,
+            0.005145419389009476,
+            -0.0077320728451013565,
+            0.03525100275874138,
+            -0.016035858541727066,
+            -0.022561756893992424,
+            0.031123511493206024,
+            -0.014557771384716034,
+            -0.007843626663088799,
+            0.018043827265501022,
+            0.015547811053693295,
+            -0.015812750905752182,
+            0.026884466409683228,
+            -0.0008057148079387844,
+            0.011476097628474236,
+            0.011789842508733273,
+            -0.0009682521340437233,
+            -0.013177293352782726,
+            -0.0286693274974823,
+            -0.023984067142009735,
+            0.01802988350391388,
+            -0.008861555717885494,
+            0.03246215730905533,
+            0.018754983320832253,
+            0.015492034144699574,
+            -0.03923904895782471,
+            0.0365617610514164,
+            -0.013253985904157162,
+            -0.004510956816375256,
+            0.0032472615130245686,
+            -0.020749008283019066,
+            -0.00024380606191698462,
+            -0.009593627415597439,
+            -0.03377291560173035,
+            -0.03321514651179314,
+            -0.02339841052889824,
+            0.0003054656845051795,
+            0.01162251178175211,
+            -0.00419023958966136,
+            -0.031039845198392868,
+            -0.023189246654510498,
+            0.025057774037122726,
+            0.02826494537293911,
+            -0.008959164842963219,
+            -0.026312753558158875,
+            -0.006731574889272451,
+            -0.019689247012138367,
+            0.020776895806193352,
+            -0.0008392680902034044,
+            0.004573706071823835,
+            0.021794823929667473,
+            0.01076494250446558,
+            -0.006581674329936504,
+            -0.007885459810495377,
+            -0.036952197551727295,
+            0.008540838025510311,
+            0.007285858038812876,
+            -0.0034633970353752375,
+            -0.006787351798266172,
+            0.013686257414519787,
+            0.0005233442061580718,
+            -0.009203189052641392,
+            -0.018252991139888763,
+            0.0011312252609059215,
+            -0.0009516933932900429,
+            0.01317032054066658,
+            -0.00043771794298663735,
+            0.014174304902553558,
+            0.016551796346902847,
+            -0.006009961012750864,
+            -0.005710160359740257,
+            0.008980081416666508,
+            0.04478885233402252,
+            0.02330080047249794,
+            -0.02613147906959057,
+            0.011838647536933422,
+            0.015882473438978195,
+            0.0029282872565090656,
+            -0.015826696529984474,
+            0.006940738297998905,
+            0.023886458948254585,
+            -0.031235065311193466,
+            -0.0027138949371874332,
+            -0.007397411856800318,
+            0.01701195538043976,
+            0.032350603491067886,
+            0.027428291738033295,
+            0.00019129733846057206,
+            -0.005748507101088762,
+            -0.00554631557315588,
+            0.019033867865800858,
+            0.012563747353851795,
+            0.018601596355438232,
+            -0.0071603599935770035,
+            -0.012354583479464054,
+            -0.026103589683771133,
+            0.008450200781226158,
+            -0.0026947215665131807,
+            -0.011531874537467957,
+            0.032545823603868484,
+            0.012556775473058224,
+            0.015826696529984474,
+            -0.012828687205910683,
+            -0.016788847744464874,
+            0.026661358773708344,
+            0.022199207916855812,
+            -0.025308769196271896,
+            -0.021739047020673752,
+            0.010325699113309383,
+            0.0017683020560070872,
+            -0.0029840641655027866,
+            -0.013553787022829056,
+            -0.022617533802986145,
+            0.028585663065314293,
+            0.09086057543754578,
+            -0.015045818872749805,
+            -0.01337251253426075,
+            -0.01733267307281494,
+            -0.023984067142009735,
+            -0.019926298409700394,
+            0.0005407744902186096,
+            0.006145917344838381,
+            0.03313148021697998,
+            0.009886455722153187,
+            -0.003914841450750828,
+            -0.005783367436379194,
+            -0.006135459057986736,
+            0.012054783292114735,
+            -0.031988054513931274,
+            0.02051195502281189,
+            0.003050299361348152,
+            0.016440242528915405,
+            0.01002589799463749,
+            -0.010702192783355713,
+            -0.00901494175195694,
+            0.013435261324048042,
+            0.03438645973801613,
+            0.004789841361343861,
+            -0.021752992644906044,
+            0.001578312017954886,
+            0.004873506724834442,
+            0.010493029840290546,
+            -0.003998506814241409,
+            -0.01844821125268936,
+            0.005487052723765373,
+            0.011594623327255249,
+            -0.003092132043093443,
+            0.0028986558318138123,
+            -0.00843625608831644,
+            0.05429881438612938,
+            -0.009182272478938103,
+            0.01317032054066658,
+            0.0041205184534192085,
+            -0.019368529319763184,
+            -0.00744621641933918,
+            0.022017931565642357,
+            -0.0040089646354317665,
+            -0.00643526017665863,
+            -0.012605579569935799,
+            0.034274905920028687,
+            0.031235065311193466,
+            -0.03486056253314018,
+            0.017541835084557533,
+            -0.009579682722687721,
+            -0.014055779203772545,
+            0.007467132527381182,
+            -0.00782271008938551,
+            0.014216138049960136,
+            0.004636454861611128,
+            0.004761952906847,
+            0.010918328538537025,
+            -0.01878287084400654,
+            -0.011441237293183804,
+            0.003656873246654868,
+            0.0007006973028182983,
+            0.017960162833333015,
+            0.016384465619921684,
+            0.01050697360187769,
+            -0.009795818477869034,
+            0.018364544957876205,
+            0.023370521143078804,
+            0.0019609066657721996,
+            0.003187998663634062,
+            0.0096215158700943,
+            0.011887452565133572,
+            -0.006961654406040907,
+            -0.008847611024975777,
+            -0.015338648110628128,
+            0.0076762959361076355,
+            0.01885259337723255,
+            -0.015505978837609291,
+            0.003221116028726101,
+            0.013637452386319637,
+            -0.04261355474591255,
+            0.0191454216837883,
+            0.010018926113843918,
+            0.013407372869551182,
+            0.01278685498982668,
+            -0.022561756893992424,
+            0.004625997040420771,
+            -0.013470121659338474,
+            -0.018169326707720757,
+            0.002300797263160348,
+            -0.01430677529424429,
+            0.012480081990361214,
+            -0.0020515441428869963,
+            -0.006334164645522833,
+            -0.018057772889733315,
+            0.005640439223498106,
+            -0.005507969297468662,
+            -0.0011809015413746238,
+            -0.000824888062197715,
+            -0.011239045299589634,
+            -0.023412354290485382,
+            -0.03717530518770218,
+            0.016844624653458595,
+            0.01109263114631176,
+            -0.006511953193694353,
+            -0.008896416053175926,
+            -0.013463149778544903,
+            0.018824703991413116,
+            -0.004448208026587963,
+            0.00529183354228735,
+            -0.0115458182990551,
+            0.0037684268318116665,
+            -0.005141932982951403,
+            0.006609562784433365,
+            -0.006428287830203772,
+            0.01013047993183136,
+            0.020233070477843285,
+            0.008129483088850975,
+            -0.0475219190120697,
+            -0.005365040618926287,
+            0.009363546967506409,
+            -0.025768928229808807,
+            0.0037509966641664505,
+            0.027930283918976784,
+            0.008338646963238716,
+            0.002250249497592449,
+            0.01218028087168932,
+            0.0115458182990551,
+            -0.020902393385767937,
+            0.014906376600265503,
+            0.007501993328332901,
+            0.00018890068167820573,
+            0.0064213159494102,
+            0.0019347612978890538,
+            -0.0004309637297410518,
+            0.021460162475705147,
+            -0.007913348264992237,
+            0.0061807781457901,
+            -0.020023908466100693,
+            0.026828689500689507,
+            -0.019382473081350327,
+            -0.0014571715146303177,
+            0.025964148342609406,
+            0.023468131199479103,
+            0.00473406445235014,
+            0.015129484236240387,
+            -0.02618725597858429,
+            0.006086654495447874,
+            0.03246215730905533,
+            0.018169326707720757,
+            0.029645422473549843,
+            -0.014948209747672081,
+            -0.022770920768380165,
+            0.014090639539062977,
+            -0.015826696529984474,
+            -0.006337650585919619,
+            -0.014076695777475834,
+            0.008366535417735577,
+            0.0008431898895651102,
+            0.03820717707276344,
+            0.008450200781226158,
+            -0.026061756536364555,
+            0.029673311859369278,
+            -0.005385957192629576,
+            -0.029115542769432068,
+            0.020442234352231026,
+            -0.009440240450203419,
+            0.009238049387931824,
+            0.004082172177731991,
+            -0.016886457800865173,
+            0.025141438469290733,
+            -0.010276894085109234,
+            -0.004225100390613079,
+            0.01889442466199398,
+            0.027874507009983063,
+            0.0035244030877947807,
+            -0.028348609805107117,
+            0.010374504141509533,
+            -0.004953686147928238,
+            -0.0002974041854031384,
+            -0.0070836665108799934,
+            0.012152392417192459,
+            -0.016537850722670555,
+            0.0013874504948034883,
+            0.006825698539614677,
+            -0.024653390049934387,
+            -0.008603586815297604,
+            -0.001669820980168879,
+            -0.014153389260172844,
+            -0.03896016627550125,
+            0.002949203597381711,
+            -0.006623507011681795,
+            0.003548805369064212,
+            -0.005208168178796768,
+            0.009537850506603718,
+            -0.017290839925408363,
+            0.015101595781743526,
+            0.008324702270328999,
+            -0.0029770920518785715,
+            -0.0069372523576021194,
+            -0.00797609705477953,
+            -0.029143432155251503,
+            0.037454187870025635,
+            0.00015545632049907,
+            -0.0036254986189305782,
+            0.005588148254901171,
+            -0.014195221476256847,
+            0.00343725155107677,
+            0.02724701724946499,
+            0.02406773343682289,
+            -0.043784867972135544,
+            0.001980080036446452,
+            -0.008840639144182205,
+            -0.021766936406493187,
+            0.013491038233041763,
+            -0.003914841450750828,
+            0.0171235091984272,
+            0.0053510963916778564,
+            -0.029729088768363,
+            -0.00030350478482432663,
+            0.010269922204315662,
+            -0.012500998564064503,
+            0.0023984068538993597,
+            0.024751000106334686,
+            -0.00036146046477369964,
+            0.0031043333001434803,
+            -0.009440240450203419,
+            0.02254781313240528,
+            -0.01662151701748371,
+            0.015938250347971916,
+            0.006215638481080532,
+            0.006375997327268124,
+            0.014376496896147728,
+            -0.02005179598927498,
+            0.019982075318694115,
+            0.0038242037408053875,
+            0.004437749739736319,
+            0.0057066744193434715,
+            -0.001433640718460083,
+            0.021906377747654915,
+            -0.0003137450839858502,
+            0.006630479358136654,
+            0.016830680891871452,
+            -0.021083669736981392,
+            0.011413348838686943,
+            -0.006930280011147261,
+            -0.00016743964806664735,
+            -0.0015791835030540824,
+            -0.021083669736981392,
+            -0.027428291738033295,
+            0.035334669053554535,
+            0.001134711317718029,
+            0.00016090329154394567,
+            0.008087650872766972,
+            -0.013393428176641464,
+            -0.030203191563487053,
+            -0.02168327011168003,
+            -0.002867281436920166,
+            0.01281474344432354,
+            0.01762550137937069,
+            -0.017709165811538696,
+            0.007766933646053076,
+            0.011315738782286644,
+            0.02445817179977894,
+            0.03407968580722809,
+            -0.0011983318254351616,
+            0.020484067499637604,
+            0.026466140523552895,
+            0.009321714751422405,
+            0.014976098202168941,
+            -0.008415340445935726,
+            -0.00901494175195694,
+            0.012473110109567642,
+            0.015408368781208992,
+            0.011789842508733273,
+            0.0019678787793964148,
+            -0.024653390049934387,
+            -0.009795818477869034,
+            -0.010262950323522091,
+            0.006463148631155491,
+            -0.005745020695030689,
+            -0.001980080036446452,
+            0.02299402840435505,
+            -0.003981076180934906,
+            -0.018001995980739594,
+            -0.019270919263362885,
+            -0.016956178471446037,
+            -0.01233366783708334,
+            -0.009405380114912987,
+            0.015603587962687016,
+            -0.00584960263222456,
+            -0.015896417200565338,
+            -0.024095620959997177,
+            0.011957173235714436,
+            0.02158566191792488,
+            0.02694024331867695,
+            0.0003128735697828233,
+            -0.004622510634362698,
+            0.01677490398287773,
+            0.0057415347546339035,
+            0.005867032799869776,
+            -0.008931276388466358,
+            -0.0009673806489445269,
+            0.011657372117042542,
+            0.025252992287278175,
+            -0.010897411964833736,
+            -0.019996019080281258,
+            -0.032043829560279846,
+            0.0056125507690012455,
+            0.0004745394107885659,
+            -0.006905877497047186,
+            0.008575698360800743,
+            0.0024890443310141563,
+            0.004608566407114267,
+            0.013574703596532345,
+            -0.006926794070750475,
+            0.009328686632215977,
+            -0.029673311859369278,
+            0.007746017072349787,
+            0.011531874537467957,
+            0.02953387051820755,
+            -0.05231873318552971,
+            -0.019173309206962585,
+            0.0007743402966298163,
+            0.046880487352609634,
+            0.01660757325589657,
+            0.013644424267113209,
+            -0.017555780708789825,
+            -0.011350599117577076,
+            -0.013790839351713657,
+            -0.011057770811021328,
+            -0.0211673341691494,
+            -0.010269922204315662,
+            0.020804785192012787,
+            -0.006337650585919619,
+            0.0421394482254982,
+            0.029812753200531006,
+            -0.004646913148462772,
+            -0.0010850350372493267,
+            -0.01448805071413517,
+            0.014669325202703476,
+            -0.009412351995706558,
+            0.03870917111635208,
+            0.004476096481084824,
+            0.012326695024967194,
+            -0.009021913632750511,
+            -0.00043771794298663735,
+            -0.0496414415538311,
+            -0.015743030235171318,
+            0.017095619812607765,
+            -0.03207171708345413,
+            0.00023116912052500993,
+            0.030370522290468216,
+            0.020679285749793053,
+            -0.00015338648518081754,
+            -0.01228486280888319,
+            0.02608964592218399,
+            0.025503989309072495,
+            0.007962152361869812,
+            0.002156125847250223,
+            -0.01752789132297039,
+            -0.007481076754629612,
+            -0.0056683276779949665,
+            -0.012250002473592758,
+            0.00782271008938551,
+            -0.027470124885439873,
+            0.021976100280880928,
+            -0.012807771563529968,
+            -0.019870521500706673,
+            0.029505981132388115,
+            -0.009049802087247372,
+            0.02187849022448063,
+            0.009565738961100578,
+            -0.0139023931697011,
+            0.03717530518770218,
+            0.011106575839221478,
+            0.011176296509802341,
+            -0.008903387933969498,
+            0.014627492986619473,
+            -0.012033866718411446,
+            0.028585663065314293,
+            -0.017667334526777267,
+            0.015882473438978195,
+            0.006773407571017742,
+            -0.00792729202657938,
+            0.00477589713409543,
+            0.031179288402199745,
+            0.012821715325117111,
+            -0.004908367525786161,
+            0.028139447793364525,
+            0.007941236719489098,
+            0.022073708474636078,
+            0.004768925253301859,
+            0.008220121264457703,
+            0.027930283918976784,
+            0.02786056324839592,
+            -0.010172312147915363,
+            -0.03184860944747925,
+            -0.007257969584316015,
+            -0.005009463056921959,
+            0.018768927082419395,
+            -0.018225103616714478,
+            -0.004472610540688038,
+            -0.0027365542482584715,
+            0.0128426318988204,
+            -0.008854582905769348,
+            0.010960161685943604,
+            0.0010649901814758778,
+            -0.032294824719429016,
+            -0.02406773343682289,
+            -0.03678486868739128,
+            0.0014231825480237603,
+            -0.021920323371887207,
+            -0.015687253326177597,
+            -6.411075446521863e-05,
+            -0.027037853375077248,
+            0.0013708916958421469,
+            0.027428291738033295,
+            -0.008359563536942005,
+            -0.012368528172373772,
+            0.005915837828069925,
+            0.014571716077625751,
+            -0.0057241045869886875,
+            -0.004005478695034981,
+            0.0029021420050412416,
+            -0.0005425175186246634,
+            4.28131315857172e-05,
+            0.010353587567806244,
+            -0.010221117176115513,
+            -0.036087654531002045,
+            -0.02668924815952778,
+            0.013435261324048042,
+            0.0019086158135905862,
+            0.00792729202657938,
+            0.013658368960022926,
+            0.003020667936652899,
+            0.0001337774156127125,
+            0.011768925935029984,
+            0.00047628243919461966,
+            0.003034612163901329,
+            -0.0013830928364768624,
+            0.019033867865800858,
+            0.008617531508207321,
+            -0.015519922599196434,
+            0.018978090956807137,
+            0.002586653921753168,
+            -0.005487052723765373,
+            -0.010332670994102955,
+            -0.004078685771673918,
+            -0.010995022021234035,
+            -0.022045820951461792,
+            0.005577689968049526,
+            0.0076414356008172035,
+            -0.012319723144173622,
+            0.00911952368915081,
+            -0.020637454465031624,
+            0.016830680891871452,
+            0.026814745739102364,
+            0.009335658513009548,
+            -0.030203191563487053,
+            -0.00563695328310132,
+            -0.012912352569401264,
+            -0.01286354847252369,
+            -0.019996019080281258,
+            0.004333168268203735,
+            0.019577693194150925,
+            0.008833667263388634,
+            0.0057798814959824085,
+            0.01716534234583378,
+            0.004036853089928627,
+            -0.010967133566737175,
+            0.0015512950485572219,
+            0.0005643054028041661,
+            -0.0058705187402665615,
+            -0.0182808805257082,
+            0.008526894263923168,
+            -0.001230577938258648,
+            -0.009293826296925545,
+            0.012187252752482891,
+            0.015254982747137547,
+            0.00655727181583643,
+            0.0024332674220204353,
+            0.0171235091984272,
+            -0.00011231638200115412,
+            -0.004368028603494167,
+            0.016272911801934242,
+            0.07898009568452835,
+            0.00919621717184782,
+            0.013707173988223076,
+            0.0021038351114839315,
+            -0.02045617811381817,
+            -0.014934265054762363,
+            -0.0037998014595359564,
+            -0.04255777597427368,
+            0.013484066352248192,
+            0.02005179598927498,
+            -0.004497012589126825,
+            -0.013163348659873009,
+            -0.012152392417192459,
+            -0.00866633653640747,
+            -0.001097236294299364,
+            0.03399602323770523,
+            -0.008185259997844696,
+            0.014808767475187778,
+            0.038318730890750885,
+            0.01878287084400654,
+            -0.006288845557719469,
+            0.00964940432459116,
+            0.025197215378284454,
+            -0.010437252931296825,
+            0.00477589713409543,
+            -0.004971116315573454,
+            0.005654383450746536,
+            0.011008965782821178,
+            0.010771914385259151,
+            0.01814143732190132,
+            0.010304782539606094,
+            -0.0021770421881228685,
+            -0.01950797066092491,
+            0.005511455237865448,
+            -0.011455181054770947,
+            0.013302790932357311,
+            -0.0030572714749723673,
+            -0.0012018178822472692,
+            -0.02629880979657173,
+            0.012905380688607693,
+            0.009461157023906708,
+            -0.013539843261241913,
+            -0.00949601735919714,
+            0.002987550338730216,
+            -0.007278885692358017,
+            0.014320719987154007,
+            -0.010409364476799965,
+            0.0105836670845747,
+            0.009056774899363518,
+            -0.005103586707264185,
+            -0.015380480326712132,
+            -0.001689865835942328,
+            -0.004890936892479658,
+            0.008094622753560543,
+            -0.010988050140440464,
+            -0.011992033571004868,
+            0.015394425019621849,
+            0.012263946235179901,
+            0.012556775473058224,
+            0.017109565436840057,
+            -0.012626496143639088,
+            0.008847611024975777,
+            -0.0028637952636927366,
+            -0.016454186290502548,
+            -0.002661603968590498,
+            -0.018517931923270226,
+            0.011315738782286644,
+            0.004908367525786161,
+            0.005476594436913729,
+            -0.010569723322987556,
+            0.00014347300748340786,
+            -0.006637451238930225,
+            0.008331675082445145,
+            -0.0035331181716173887,
+            -0.0016166586428880692,
+            -0.012096615508198738,
+            -0.010576695203781128,
+            -0.0035052297171205282,
+            0.02314741350710392,
+            0.011789842508733273,
+            0.0009577939636074007,
+            0.009530877694487572,
+            0.0072719138115644455,
+            -0.007031375542283058,
+            -0.013414344750344753,
+            0.002541335066780448,
+            -0.001469372771680355,
+            0.005996016785502434,
+            0.021348608657717705,
+            -0.010520918294787407,
+            -0.0071394434198737144,
+            0.0020811755675822496,
+            0.016747014597058296,
+            0.0058530885726213455,
+            0.0009325200808234513,
+            0.023468131199479103,
+            0.0003483877517282963,
+            0.0030572714749723673,
+            -0.01864342950284481,
+            -0.015394425019621849,
+            0.010346615687012672,
+            0.026675302535295486,
+            0.00863844808191061,
+            0.0010275151580572128,
+            -0.017193229869008064,
+            0.0026162853464484215,
+            0.027372514829039574,
+            0.026019925251603127,
+            0.007543826010078192,
+            -0.0027417833916842937,
+            0.0009037601412273943,
+            -0.010151396505534649,
+            -0.0001501182996435091,
+            0.028390442952513695,
+            -0.015185261145234108,
+            -0.0006593004218302667,
+            -0.005256973206996918,
+            -0.013281874358654022,
+            -0.017151396721601486,
+            0.00957271084189415,
+            0.027233071625232697,
+            0.006222610827535391,
+            0.010409364476799965,
+            -0.01995418593287468,
+            0.01334462407976389,
+            -0.009007969871163368,
+            -0.008282870054244995,
+            -0.009265937842428684,
+            -0.021599605679512024,
+            -0.014376496896147728,
+            -0.011036854237318039,
+            -0.0032228592317551374,
+            -0.014766935259103775,
+            -0.0002703872451093048,
+            0.002027141861617565,
+            0.005727590527385473,
+            -0.003156624035909772,
+            -0.0036080684512853622,
+            -0.0061807781457901,
+            0.010548806749284267,
+            -0.00429482152685523,
+            0.0037684268318116665,
+            -0.023942235857248306,
+            -0.026507971808314323,
+            0.01839243434369564,
+            -0.008422312326729298,
+            -0.00397759024053812,
+            -0.015003986656665802,
+            -0.004782869480550289,
+            -0.023691238835453987,
+            0.01802988350391388,
+            -0.023523908108472824,
+            -0.010590638965368271,
+            0.004381972830742598,
+            0.013511954806745052,
+            -0.016747014597058296,
+            -0.01532470341771841,
+            -0.0022711658384650946,
+            -0.01228486280888319,
+            0.004256474785506725,
+            -0.03380080312490463,
+            0.007578686345368624,
+            -0.000818351749330759,
+            0.012975101359188557,
+            -0.020735062658786774,
+            0.02491833083331585,
+            0.0033117535058408976,
+            -0.013009962625801563,
+            -0.008373507298529148,
+            -0.007223108783364296,
+            -0.0026563750579953194,
+            0.005912351422011852,
+            0.001685508294031024,
+            -0.014766935259103775,
+            0.02081872895359993,
+            0.0016959664644673467,
+            0.009586654603481293,
+            -0.01112051960080862,
+            0.00803187396377325,
+            0.011162352748215199,
+            0.005082670133560896,
+            -0.003649901133030653,
+            0.007648407481610775,
+            -0.0031513948924839497,
+            0.019368529319763184,
+            0.020637454465031624,
+            -0.006341136526316404,
+            0.008038845844566822,
+            0.004015936981886625,
+            -0.017193229869008064,
+            -0.007034861948341131,
+            0.00405428372323513,
+            0.0029126000590622425,
+            -0.018727093935012817,
+            0.025099605321884155,
+            0.027009963989257812,
+            0.001078934408724308,
+            -0.02522510476410389,
+            0.010925300419330597,
+            0.011001993902027607,
+            0.014529882930219173,
+            0.02147410809993744,
+            -0.002708665793761611,
+            -0.023175302892923355,
+            0.009607571177184582,
+            -0.008415340445935726,
+            -0.01920119859278202,
+            0.031430285423994064,
+            0.022003987804055214,
+            0.011127491481602192,
+            -0.05491235852241516,
+            0.003093875013291836,
+            0.01579880714416504,
+            -0.013609563931822777,
+            -0.02953387051820755,
+            -0.011420320719480515,
+            -0.009935260750353336,
+            0.0029091141186654568,
+            0.013909365050494671,
+            -0.015213149599730968,
+            -0.01752789132297039,
+            0.014934265054762363,
+            0.030928291380405426,
+            0.0275119561702013,
+            0.025057774037122726,
+            -0.023426298052072525,
+            0.001805777195841074,
+            -0.025113550946116447,
+            0.00640737172216177,
+            -0.016077691689133644,
+            -0.005867032799869776,
+            0.0123615562915802,
+            -0.007613547146320343,
+            0.020735062658786774,
+            0.002405378967523575,
+            0.02183665707707405,
+            0.017695222049951553,
+            0.03692431002855301,
+            -0.016593627631664276,
+            -0.030231080949306488,
+            -0.011329683475196362,
+            -0.014048807322978973,
+            0.023370521143078804,
+            0.01789044216275215,
+            0.00967729277908802,
+            0.028446219861507416,
+            0.02928287349641323,
+            -0.0020550303161144257,
+            0.00960059929639101,
+            -2.5587109121261165e-05,
+            8.219467417802662e-05,
+            -0.018113549798727036,
+            -0.02314741350710392,
+            -0.0021630979608744383,
+            0.00967729277908802,
+            0.013923308812081814,
+            -0.005605578422546387,
+            -0.0027278391644358635,
+            0.021097613498568535,
+            -0.0032629487104713917,
+            -0.0027905881870537996,
+            -0.03555777668952942,
+            0.0018842134159058332,
+            -0.008924304507672787,
+            0.021808769553899765,
+            -0.00020491474424488842,
+            -0.008094622753560543,
+            -0.027525901794433594,
+            -0.004918825346976519,
+            0.034721121191978455,
+            0.005950698163360357,
+            0.00012168515240773559,
+            0.011469125747680664,
+            0.024193231016397476,
+            0.010513946413993835,
+            0.016328688710927963,
+            -0.001948705525137484,
+            0.001576569047756493,
+            0.022784864529967308,
+            -0.006644423585385084,
+            0.02187849022448063,
+            0.003827689914032817,
+            -0.013044822961091995,
+            -0.0017866039415821433,
+            -0.0019609066657721996,
+            -0.008659363724291325,
+            0.009900400415062904,
+            0.015533867292106152,
+            -0.011810759082436562,
+            0.018406378105282784,
+            -0.016021914780139923,
+            -0.0012401645071804523,
+            -0.023970123380422592,
+            -0.0005242157494649291,
+            0.013651397079229355,
+            -0.00317928334698081,
+            0.004985060542821884,
+            -0.019521916285157204,
+            0.0057624513283371925,
+            0.014111556112766266,
+            -0.012675301171839237,
+            0.004761952906847,
+            -0.02158566191792488,
+            0.006138945464044809,
+            0.011531874537467957,
+            -0.001174801029264927,
+            -0.009412351995706558,
+            -0.005385957192629576,
+            0.01165040023624897,
+            -0.015882473438978195,
+            0.004971116315573454,
+            0.01797410659492016,
+            0.012054783292114735,
+            0.010018926113843918,
+            0.006860558874905109,
+            0.00911952368915081,
+            0.008443228900432587,
+            0.012194225564599037,
+            -0.036952197551727295,
+            0.018071716651320457,
+            -0.01430677529424429,
+            0.009440240450203419,
+            -0.0017552294302731752,
+            -0.007829682901501656,
+            -0.008171316236257553,
+            -0.009175300598144531,
+            0.023133469745516777,
+            -0.004807271528989077,
+            0.0019243031274527311,
+            -0.00861055962741375,
+            0.030231080949306488,
+            0.016175301745533943,
+            -0.006484064739197493,
+            0.025768928229808807,
+            -0.004922311753034592,
+            -0.0020567732863128185,
+            -0.012431276962161064,
+            -0.004458666313439608,
+            0.01342828944325447,
+            -0.011643428355455399,
+            0.01600797101855278,
+            -0.04264144226908684,
+            0.0059088654816150665,
+            -0.011022910475730896,
+            -0.005121016874909401,
+            -0.016189245507121086,
+            -0.0171235091984272,
+            -0.030258968472480774,
+            0.02502988465130329,
+            0.00914741214364767,
+            -0.027079686522483826,
+            -0.0005503611755557358,
+            -0.006679283920675516,
+            0.0005150648066774011,
+            -0.01109263114631176,
+            -0.015812750905752182,
+            0.016551796346902847,
+            0.03787251561880112,
+            -0.012005978263914585,
+            -0.01367231272161007,
+            0.003060757415369153,
+            0.008450200781226158,
+            -0.017444226890802383,
+            -0.006375997327268124,
+            0.0019765938632190228,
+            -0.028097614645957947,
+            0.020274903625249863,
+            -0.005751993041485548,
+            0.0031374506652355194,
+            -0.017346616834402084,
+            0.01600797101855278,
+            -0.015394425019621849,
+            -0.0060587660409510136,
+            -0.02253386750817299,
+            -0.023523908108472824,
+            0.016342632472515106,
+            -0.033466141670942307,
+            -0.011315738782286644,
+            -0.01853187568485737,
+            0.004444722086191177,
+            -0.00848506111651659,
+            -0.001322086900472641,
+            0.009000997990369797,
+            0.00419023958966136,
+            0.032852593809366226,
+            -0.00848506111651659,
+            0.004137949086725712,
+            0.03307570144534111,
+            -0.008394423872232437,
+            0.024709166958928108,
+            0.0022641937248408794,
+            0.023328689858317375,
+            0.009454185143113136,
+            0.0013752492377534509,
+            -0.007369523402303457,
+            0.010925300419330597,
+            -0.0014310261467471719,
+            0.03731474652886391,
+            -0.011901396326720715,
+            -0.018922314047813416,
+            0.00861055962741375,
+            -0.004890936892479658,
+            0.006403885781764984,
+            0.02846016362309456,
+            0.019382473081350327,
+            -0.005738048814237118,
+            0.008275898173451424,
+            -0.01157370675355196,
+            -0.0025918828323483467,
+            0.008059762418270111,
+            0.011713149026036263,
+            0.017039842903614044,
+            0.0002762699732556939,
+            -0.019982075318694115,
+            0.02112550102174282,
+            0.0009569224785082042,
+            -0.006428287830203772,
+            -0.015213149599730968,
+            0.0078087663277983665,
+            -0.006972112692892551,
+            0.00922410562634468,
+            -0.00840836763381958,
+            0.016830680891871452,
+            0.0027034366503357887,
+            -0.02526693604886532,
+            0.0031130483839660883,
+            0.00911952368915081,
+            -0.002987550338730216,
+            -0.004950200207531452,
+            -0.016844624653458595,
+            -0.010904384776949883,
+            0.0058530885726213455,
+            0.015812750905752182,
+            -0.0054068733006715775,
+            -0.02628486417233944,
+            -0.005581176374107599,
+            0.00018236432515550405,
+            0.0008924304274842143,
+            0.014167333021759987,
+            0.0018388947937637568,
+            -0.02320319041609764,
+            0.017960162833333015,
+            -0.0017395421164110303,
+            0.004650399088859558,
+            -0.009614543057978153,
+            0.026410363614559174,
+            -0.004660857375711203,
+            -0.011587651446461678,
+            0.010799802839756012,
+            0.034721121191978455,
+            -0.0019452194683253765,
+            0.01868526265025139,
+            -0.02370518259704113,
+            -0.002764442702755332,
+            -0.00901494175195694,
+            0.004193725995719433,
+            -0.01532470341771841,
+            -0.0007220494444482028,
+            0.016133468598127365,
+            0.0023809764534235,
+            -0.010269922204315662,
+            0.002013197634369135,
+            0.024332674220204353,
+            -0.0009063746547326446,
+            0.003261205740272999,
+            0.015687253326177597,
+            0.005731076467782259,
+            0.018629485741257668,
+            0.011476097628474236,
+            -0.03792829439043999,
+            0.002319970401003957,
+            0.010723109357059002,
+            0.0031339647248387337,
+            0.002788844984024763,
+            0.014320719987154007,
+            -0.0053894431330263615,
+            -0.0061807781457901,
+            -0.0030468131881207228,
+            0.013023906387388706,
+            0.009893427602946758,
+            0.008101594634354115,
+            0.0035906380508095026,
+            0.0022746517788618803,
+            -0.018517931923270226,
+            0.013121516443789005,
+            0.0018458669073879719,
+            0.0069023915566504,
+            0.014739046804606915,
+            -0.015980081632733345,
+            -0.005469622556120157,
+            -0.009928288869559765,
+            0.004197211936116219,
+            0.0017953190254047513,
+            -0.004050797317177057,
+            0.004800299648195505,
+            -0.001281997188925743,
+            0.0014101098058745265,
+            -0.014599604532122612,
+            0.006389941554516554,
+            0.01864342950284481,
+            0.0008105080923996866,
+            -0.004981574602425098,
+            0.03661753609776497,
+            -0.01783466525375843,
+            0.010988050140440464,
+            -0.012061755172908306,
+            0.014153389260172844,
+            -0.02597809210419655,
+            0.041358571499586105,
+            0.016858568415045738,
+            0.00477589713409543,
+            0.006686256267130375,
+            -0.01079283095896244,
+            -0.007522909436374903,
+            -0.01218028087168932,
+            0.013769922778010368,
+            0.013846616260707378,
+            -0.015966137871146202,
+            -0.0008793577435426414,
+            -0.0018127493094652891,
+            -0.0011164095485582948,
+            0.02157171629369259,
+            0.006484064739197493,
+            -0.014697213657200336,
+            0.008840639144182205,
+            -0.0018894425593316555,
+            -0.008791834115982056,
+            -0.01869920641183853,
+            -0.008213148452341557,
+            0.014850599691271782,
+            -0.0072161369025707245,
+            -0.0022955681197345257,
+            0.008708168752491474,
+            0.004155379254370928,
+            0.000675423420034349,
+            -0.018420321866869926,
+            0.019577693194150925,
+            -0.011176296509802341,
+            0.006602590903639793,
+            -0.001901643699966371,
+            -0.02684263326227665,
+            0.015617532655596733,
+            -0.019828688353300095,
+            0.015227094292640686,
+            -0.00016624131239950657,
+            -0.027539845556020737,
+            0.0003377117100171745,
+            0.004664343316107988,
+            -0.023342633619904518,
+            -0.020442234352231026,
+            0.005110558588057756,
+            -0.0008105080923996866,
+            -0.004169323481619358,
+            -0.03352191671729088,
+            0.03681275621056557,
+            0.015826696529984474,
+            0.03879283368587494,
+            -0.019312752410769463,
+            -0.006187750026583672,
+            0.0034790842328220606,
+            -0.00667579798027873,
+            0.003147908952087164,
+            -0.026577694341540337,
+            0.016788847744464874,
+            0.022505979984998703,
+            0.013358567841351032,
+            -0.011866535991430283,
+            -0.009635459631681442,
+            -0.004490040708333254,
+            -0.008059762418270111,
+            0.014641436748206615,
+            -0.006672312039881945,
+            -0.013797811232507229,
+            -0.003052042331546545,
+            0.0004989418084733188,
+            0.003393675899133086,
+            0.016733070835471153,
+            -0.004967630375176668,
+            0.006229582708328962,
+            -0.0043436260893940926,
+            0.00048674060963094234,
+            -0.028223112225532532,
+            0.028892435133457184,
+            -0.021195223554968834,
+            0.03692431002855301,
+            -0.005713646300137043,
+            0.003475598292425275,
+            -0.005180279724299908,
+            0.007801793981343508,
+            0.018503988161683083,
+            -0.002971863141283393,
+            -0.010144423693418503,
+            -0.01772311143577099,
+            7.070157880662009e-05,
+            0.015589644201099873,
+            -0.0016062004724517465,
+            -0.0036882476415485144,
+            0.0189502015709877,
+            1.2528014849522151e-05,
+            -0.008708168752491474,
+            -0.009719124995172024,
+            -0.002668576082214713,
+            0.004842132329940796,
+            0.011587651446461678,
+            0.0275119561702013,
+            0.008017929270863533,
+            0.0029178292024880648,
+            0.006184264086186886,
+            -0.001894671586342156,
+            -0.02678685635328293,
+            -0.010067731142044067,
+            0.01266135647892952,
+            -0.029422316700220108,
+            0.0053720129653811455,
+            0.010988050140440464,
+            0.019061755388975143,
+            0.011134464293718338,
+            -0.021055780351161957,
+            -0.0006266186246648431,
+            0.01681673526763916,
+            -0.006145917344838381,
+            0.029756976291537285,
+            -0.01777888834476471,
+            0.011183268390595913,
+            0.026424307376146317,
+            0.018768927082419395,
+            -0.008847611024975777,
+            -0.01797410659492016,
+            0.008303786627948284,
+            0.0072509972378611565,
+            -0.001050174469128251,
+            0.008924304507672787,
+            -0.04043825343251228,
+            0.004451693966984749,
+            -0.01858765259385109,
+            0.0148645443841815,
+            0.004448208026587963,
+            0.011287850327789783,
+            0.006484064739197493,
+            -0.003172311233356595,
+            0.001901643699966371,
+            0.005528885405510664,
+            0.0006017804844304919,
+            -0.0276095662266016,
+            -0.00949601735919714,
+            -0.0085617545992136,
+            0.0012558518210425973,
+            -0.010262950323522091,
+            -0.0010606326395645738,
+            0.01570119708776474,
+            -0.007285858038812876,
+            0.002699950709939003,
+            -0.041051801294088364,
+            -0.013644424267113209,
+            -0.010632472112774849,
+            -0.012089643627405167,
+            -0.011357571929693222,
+            0.019215142354369164,
+            -0.00269820773974061,
+            -0.01010956335812807,
+            0.008331675082445145,
+            0.005469622556120157,
+            -0.030565742403268814,
+            -0.008603586815297604,
+            0.00922410562634468,
+            -0.006247012875974178,
+            0.0040647415444254875,
+            0.005103586707264185,
+            -0.012026894837617874,
+            -0.024890443310141563,
+            0.009956177324056625,
+            -0.012884464114904404,
+            0.016440242528915405,
+            0.019842632114887238,
+            0.0007159488159231842,
+            -0.005535857751965523,
+            0.0073276907205581665,
+            0.01023506186902523,
+            -0.005880977027118206,
+            0.01677490398287773,
+            -0.010820719413459301,
+            0.01635657623410225,
+            0.014223109930753708,
+            0.01844821125268936,
+            -6.938069418538362e-06,
+            -0.014557771384716034,
+            -0.00029609689954668283,
+            0.006749005056917667,
+            0.00505129573866725,
+            -0.00577290914952755,
+            -0.004197211936116219,
+            -0.012354583479464054,
+            0.010932273231446743,
+            -0.007264941465109587,
+            -0.006909363903105259,
+            -0.001948705525137484,
+            0.00015905132750049233,
+            -0.0050756982527673244,
+            -0.004577192012220621,
+            -0.002827191725373268,
+            -0.014766935259103775,
+            0.0006484065088443458,
+            -0.026661358773708344,
+            -0.00863844808191061,
+            -0.007244025357067585,
+            0.0016645919531583786,
+            -0.006341136526316404,
+            0.0030747016426175833,
+            0.04913945123553276,
+            -0.01446016225963831,
+            0.003154881065711379,
+            -0.003660359187051654,
+            -0.02380279265344143,
+            -0.018406378105282784,
+            0.00012734999472741038,
+            -0.009356575086712837,
+            -0.026480084285140038,
+            -0.016370519995689392,
+            0.011992033571004868,
+            -0.0036708174739032984,
+            0.0009708667057566345,
+            -0.010848607867956161,
+            -0.01717928610742092,
+            -0.02643825113773346,
+            -0.005657869391143322,
+            -0.0021142931655049324,
+            -0.010667332448065281,
+            -0.015589644201099873,
+            0.006473606918007135,
+            -0.010653388686478138,
+            0.009781873784959316,
+            0.02360757440328598,
+            0.004364542663097382,
+            0.019312752410769463,
+            0.00901494175195694,
+            -0.018266934901475906,
+            -0.0015434514498338103,
+            -0.0037998014595359564,
+            -0.018015939742326736,
+            0.006693228147923946,
+            0.01066036056727171,
+            -0.006755977403372526,
+            0.011008965782821178,
+            -0.007578686345368624,
+            -0.014725102111697197,
+            -0.004514443222433329,
+            -0.019968131557106972,
+            0.029003988951444626,
+            -0.02582470513880253,
+            -0.005005977116525173,
+            -0.003897411050274968,
+            -0.012849603779613972,
+            0.027079686522483826,
+            0.0128426318988204,
+            -0.019312752410769463,
+            0.024346617981791496,
+            0.0040473113767802715,
+            0.008366535417735577,
+            0.01374900620430708,
+            0.011789842508733273,
+            0.0007586530409753323,
+            0.011322710663080215,
+            0.027372514829039574,
+            -0.0071952203288674355,
+            -0.030258968472480774,
+            -0.002189243445172906,
+            0.01853187568485737,
+            0.014906376600265503,
+            -0.006909363903105259,
+            -0.002267679665237665,
+            -0.00014162104343995452,
+            0.017249006778001785,
+            -0.001689865835942328,
+            -0.000326599896652624,
+            -0.0008667207439430058,
+            -0.009007969871163368,
+            -0.007683268282562494,
+            0.006424801889806986,
+            -0.026173310354351997,
+            -0.0035278890281915665,
+            -0.02224103920161724,
+            0.011350599117577076,
+            -0.013023906387388706,
+            -0.002933516399934888,
+            -0.007725100964307785,
+            -0.005856574513018131,
+            0.027135463431477547,
+            0.021097613498568535,
+            -0.006700200494378805,
+            0.011113547720015049,
+            -0.008687252178788185,
+            -0.012187252752482891,
+            0.014711158350110054,
+            -0.0018476098775863647,
+            0.0014519424876198173,
+            -0.0021282373927533627,
+            0.005769423209130764,
+            -0.011232073418796062,
+            0.0013046566164121032,
+            0.022059764713048935,
+            0.024025900289416313,
+            -0.01367231272161007,
+            -0.020958170294761658,
+            -0.014934265054762363,
+            0.021892433986067772,
+            -0.0010449454421177506,
+            -0.011252989992499352,
+            0.0074741048738360405,
+            0.003100847126916051,
+            -0.010214145295321941,
+            -0.015715142711997032,
+            0.016300799325108528,
+            -0.02137649804353714,
+            -0.014446217566728592,
+            -0.014780879020690918,
+            -0.0036777895875275135,
+            -0.031179288402199745,
+            0.019326696172356606,
+            0.016886457800865173,
+            -0.004817729815840721,
+            -0.002276394749060273,
+            0.01631474308669567,
+            0.00587749108672142,
+            -0.00845717266201973,
+            -0.011720121838152409,
+            -0.013414344750344753,
+            -0.0017273409757763147,
+            0.0030293830204755068,
+            -0.015631476417183876,
+            -0.0017848608549684286,
+            0.0009211904252879322,
+            0.00805279053747654,
+            0.006030877586454153,
+            -0.0018493529642000794,
+            0.0016733070369809866,
+            -0.002238048240542412,
+            0.031792834401130676,
+            0.012975101359188557,
+            -0.00810856744647026,
+            0.007501993328332901,
+            -0.005535857751965523,
+            0.012696217745542526,
+            -0.003604582278057933,
+            -0.023328689858317375,
+            -0.02705179713666439,
+            -0.01382569968700409,
+            -0.004036853089928627,
+            -0.0043436260893940926,
+            -0.006009961012750864,
+            -0.014390440657734871,
+            0.006498008966445923,
+            -0.008157371543347836,
+            0.013281874358654022,
+            0.005197709891945124,
+            -0.002307769376784563,
+            0.014739046804606915,
+            0.01991235464811325,
+            0.012110560201108456,
+            -0.01446016225963831,
+            -0.014390440657734871,
+            0.012438248842954636,
+            -0.008729085326194763,
+            -0.002525647869333625,
+            0.020581677556037903,
+            0.013658368960022926,
+            -0.01436255220323801,
+            -0.0221294853836298,
+            0.005033865571022034,
+            -0.017695222049951553,
+            0.011950201354920864,
+            0.003034612163901329,
+            0.006651395466178656,
+            0.01849004253745079,
+            0.014501994475722313,
+            -0.010444224812090397,
+            0.003442480694502592,
+            0.011427292600274086,
+            -0.013386456295847893,
+            -0.030816737562417984,
+            -0.02518327161669731,
+            -0.004420319572091103,
+            -0.016175301745533943,
+            -0.03120717592537403,
+            0.0006919822189956903,
+            -0.007397411856800318,
+            0.017555780708789825,
+            0.009433268569409847,
+            0.017388449981808662,
+            0.021641438826918602,
+            0.0078087663277983665,
+            0.004950200207531452,
+            0.018657373264431953,
+            0.004699204117059708,
+            -0.0037440245505422354,
+            -0.006745519116520882,
+            -0.004751494619995356,
+            0.004723606165498495,
+            -0.01175498217344284,
+            0.0005673556588590145,
+            -0.011448209173977375,
+            0.020930282771587372,
+            0.018322711810469627,
+            0.005783367436379194,
+            0.0010597611544653773,
+            0.009942232631146908,
+            0.0016881227493286133,
+            0.00901494175195694,
+            0.009161355905234814,
+            -0.010709165595471859,
+            -0.0003250747686251998,
+            -0.013030878268182278,
+            -0.02816733531653881,
+            0.0090428302064538,
+            0.004472610540688038,
+            -0.0005625623743981123,
+            0.008645419962704182,
+            -0.00746016064658761,
+            -0.02380279265344143,
+            -0.0030136958230286837,
+            0.0062051801942288876,
+            -0.013016934506595135,
+            -0.006717630662024021,
+            -0.0014414843171834946,
+            0.007174304220825434,
+            0.023482074961066246,
+            -0.00970518123358488,
+            0.021139446645975113,
+            0.019745023921132088,
+            -0.0003472983662504703,
+            -0.008526894263923168,
+            -0.019745023921132088,
+            -0.033968131989240646,
+            -0.015575699508190155,
+            0.005358068738132715,
+            -0.0305936299264431,
+            -0.005434761755168438,
+            -0.007055778056383133,
+            -0.00016014071297831833,
+            -0.009656376205384731,
+            0.01023506186902523,
+            -0.0001039280541590415,
+            -0.00861055962741375,
+            0.013233070261776447,
+            -0.0034006480127573013,
+            0.005378984846174717,
+            -0.006923308130353689,
+            0.005392929073423147,
+            -0.01436255220323801,
+            -0.0011870021698996425,
+            0.001350846840068698,
+            0.014599604532122612,
+            -0.02056773193180561,
+            -0.011001993902027607,
+            0.0040298812091350555,
+            -0.0010562750976532698,
+            -0.006323706358671188,
+            -0.0201075728982687,
+            0.012668329291045666,
+            -0.03619920834898949,
+            0.008805778808891773,
+            -0.0035122018307447433,
+            0.002835906809195876,
+            -0.006278387736529112,
+            -0.021237054839730263,
+            -0.004430777858942747,
+            0.006337650585919619,
+            -0.019745023921132088,
+            0.0024158370215445757,
+            -0.01975896768271923,
+            0.010939245112240314,
+            -0.007725100964307785,
+            0.016454186290502548,
+            -0.0123615562915802,
+            -5.822803359478712e-05,
+            0.0030747016426175833,
+            0.01451593916863203,
+            -0.002084661740809679,
+            -0.0022397912107408047,
+            0.0024332674220204353,
+            -0.0017630730289965868,
+            0.003043327247723937,
+            -0.010925300419330597,
+            0.006982570979744196,
+            -0.006449204403907061,
+            0.01121812965720892,
+            0.005086156073957682,
+            0.019786855205893517,
+            0.006877989042550325,
+            0.023007972165942192,
+            -0.020846616476774216,
+            0.006072710268199444,
+            -0.015045818872749805,
+            0.018155381083488464,
+            0.013762950897216797,
+            0.02528088167309761,
+            0.011162352748215199,
+            -0.02582470513880253,
+            0.009830678813159466,
+            0.028501996770501137,
+            0.022017931565642357,
+            -0.008519921451807022,
+            -0.01929880864918232,
+            0.0026703192852437496,
+            0.03990837186574936,
+            0.01340040098875761,
+            0.00011972424545092508,
+            -0.011225101538002491,
+            0.016370519995689392,
+            0.00531972199678421,
+            -0.008631475269794464,
+            0.01492032129317522,
+            -0.02588048204779625,
+            0.0006937252474017441,
+            0.005738048814237118,
+            0.011190241202712059,
+            -0.022868528962135315,
+            0.0160637479275465,
+            0.014697213657200336,
+            0.026828689500689507,
+            -0.01980080083012581,
+            0.0015434514498338103,
+            -0.0033065243624150753,
+            0.008164344355463982,
+            -0.0070488061755895615,
+            0.002985807368531823,
+            0.01342828944325447,
+            -0.010883468203246593,
+            -0.04280877113342285,
+            -0.0006793452193960547,
+            0.009265937842428684,
+            0.016691237688064575,
+            -0.0032629487104713917,
+            0.005072211846709251,
+            -0.023970123380422592,
+            -0.020372513681650162,
+            -0.01053486205637455,
+            0.024806777015328407,
+            -0.0014911605976521969,
+            0.028041837736964226,
+            -0.0002359624340897426,
+            -0.009747013449668884,
+            -0.007101096678525209,
+            0.014083667658269405,
+            -0.005044323392212391,
+            0.007948208600282669,
+            0.03182072192430496,
+            0.014808767475187778,
+            0.005194223951548338,
+            -0.010939245112240314,
+            0.03611554577946663,
+            -0.00010321995068807155,
+            -0.005124502815306187,
+            -0.00845717266201973,
+            0.0007756475242786109,
+            0.0192430317401886,
+            -0.008478089235723019,
+            -0.006539841648191214,
+            0.011057770811021328,
+            0.0022310761269181967,
+            -0.0070104594342410564,
+            -0.023621518164873123,
+            0.015227094292640686,
+            -0.010297810658812523,
+            0.014557771384716034,
+            -0.0004311816010158509,
+            0.011615539900958538,
+            -0.031988054513931274,
+            -0.006592132616788149,
+            -0.0010510459542274475,
+            -0.006191235966980457,
+            0.015408368781208992,
+            -0.014780879020690918,
+            -0.009642431512475014,
+            -0.005884462967514992,
+            -0.011022910475730896,
+            0.021139446645975113,
+            0.00843625608831644,
+            -0.007543826010078192,
+            -0.002124751452356577,
+            -0.004336654208600521,
+            0.0011442980030551553,
+            -0.006550299935042858,
+            -0.010576695203781128,
+            -0.02132072113454342,
+            -0.014320719987154007,
+            -0.019842632114887238,
+            -0.01564542017877102,
+            0.018559763208031654,
+            0.009788846597075462,
+            -0.02218526229262352,
+            0.014501994475722313,
+            -0.002930030459538102,
+            0.0009935260750353336,
+            -0.0015112054534256458,
+            0.008903387933969498,
+            0.011134464293718338,
+            -0.024597613140940666,
+            0.0023426299449056387,
+            -0.014404385350644588,
+            -0.00024097364803310484,
+            -0.02847410924732685,
+            0.015868527814745903,
+            -0.017235063016414642,
+            -0.0007159488159231842,
+            0.020205182954669,
+            -0.013532870449125767,
+            -0.00944721233099699,
+            0.00328560802154243,
+            -0.027874507009983063,
+            0.006877989042550325,
+            0.008324702270328999,
+            -0.005661355331540108,
+            -0.027832673862576485,
+            -0.015213149599730968,
+            -0.037816740572452545,
+            -0.010995022021234035,
+            0.02476494386792183,
+            -0.0189502015709877,
+            -0.00014935572107788175,
+            0.009837650693953037,
+            -0.006417830009013414,
+            0.02502988465130329,
+            -0.01119721308350563,
+            0.005675299558788538,
+            -0.013072711415588856,
+            -0.01621713489294052,
+            0.0030102096498012543,
+            0.0037998014595359564,
+            -0.002534362953156233,
+            -0.012521914206445217,
+            7.544043910456821e-05,
+            0.002996265422552824,
+            0.0004237737157382071,
+            0.013693229295313358,
+            -0.0004531873273663223,
+            -0.019424306228756905,
+            0.014153389260172844,
+            -0.028808770701289177,
+            0.0016062004724517465,
+            -0.012926297262310982,
+            -0.009210160933434963,
+            -0.0072509972378611565,
+            0.007188248448073864,
+            0.005166335497051477,
+            0.0037579687777906656,
+            0.005821714177727699,
+            0.001241907593794167,
+            0.013735062442719936,
+            -0.004517929162830114,
+            -0.011406376026570797,
+            0.007662351708859205,
+            -0.016426296904683113,
+            0.02406773343682289,
+            0.01920119859278202,
+            0.0023600601125508547,
+            -0.02416534349322319,
+            0.011078687384724617,
+            0.003097361186519265,
+            -0.0071394434198737144,
+            0.0028237055521458387,
+            0.02324502356350422,
+            -0.004001992754638195,
+            0.010911356657743454,
+            0.001513820025138557,
+            -0.003372759558260441,
+            0.030677296221256256,
+            -0.009342631325125694,
+            -0.003859064308926463,
+            -0.01263346802443266,
+            -0.00851294957101345,
+            0.0051872520707547665,
+            -0.00021875003585591912,
+            -0.004361056722700596,
+            -0.0013212154153734446,
+            -0.006511953193694353,
+            0.02518327161669731,
+            0.00497460225597024,
+            0.0031984567176550627,
+            0.0076414356008172035,
+            -0.01585458405315876,
+            0.009684264659881592,
+            -0.02522510476410389,
+            -0.005497511010617018,
+            -0.022659366950392723,
+            -0.026159366592764854,
+            -0.030760960653424263,
+            -0.02026095986366272,
+            -0.014404385350644588,
+            0.006288845557719469,
+            0.0015190490521490574,
+            -0.007467132527381182,
+            0.0008680280297994614,
+            -0.007237053010612726,
+            -0.005166335497051477,
+            0.01492032129317522,
+            0.01061155553907156,
+            0.014390440657734871,
+            -0.0016184017295017838,
+            0.022324705496430397,
+            -0.004364542663097382,
+            -0.02253386750817299,
+            -0.008066734299063683,
+            -0.007704184390604496,
+            0.015715142711997032,
+            0.011029882356524467,
+            0.015924304723739624,
+            -0.0020236556883901358,
+            -0.004625997040420771,
+            -0.006689742207527161,
+            0.023217136040329933,
+            0.0013238298706710339,
+            -0.012598607689142227,
+            0.003942729905247688,
+            0.02897610142827034,
+            -0.0026668331120163202,
+            0.005832172464579344,
+            0.013686257414519787,
+            0.0101025914773345,
+            -0.010472113266587257,
+            -0.015533867292106152,
+            -0.008387451991438866,
+            -0.005420817527920008,
+            0.0022188748698681593,
+            -0.005692730192095041,
+            -0.006135459057986736,
+            -0.016928289085626602,
+            0.006100598722696304,
+            -0.01342828944325447,
+            -0.005009463056921959,
+            0.010430281050503254,
+            -0.010088646784424782,
+            0.0028254487551748753,
+            0.012933269143104553,
+            0.013302790932357311,
+            -0.000308733870042488,
+            0.02507171779870987,
+            0.0027836160734295845,
+            0.004549303557723761,
+            -0.012152392417192459,
+            -0.020707175135612488,
+            -0.007017431315034628,
+            0.018517931923270226,
+            0.013086655177175999,
+            0.002142181620001793,
+            -0.006588646676391363,
+            -0.02726096101105213,
+            0.0007133343024179339,
+            -0.009865539148449898,
+            0.008429284207522869,
+            -0.034972116351127625,
+            0.023119525983929634,
+            -0.028139447793364525,
+            0.008366535417735577,
+            0.01727689616382122,
+            0.05053387209773064,
+            0.00041789101669564843,
+            0.0023844626266509295,
+            6.50367364869453e-05,
+            0.015575699508190155,
+            0.02152988500893116,
+            -0.0009952690452337265,
+            -0.0021787851583212614,
+            -0.038011956959962845,
+            -0.005309263709932566,
+            -0.000978710362687707,
+            -0.016021914780139923,
+            0.014271914958953857,
+            -0.0015190490521490574,
+            -0.024751000106334686,
+            -0.0039531877264380455,
+            -0.012494025751948357,
+            -0.005054781679064035,
+            0.017193229869008064,
+            0.014752990566194057,
+            -0.015687253326177597,
+            -0.0010815489804372191,
+            -0.019173309206962585,
+            -0.00797609705477953,
+            0.02208765409886837,
+            0.0013586904387921095,
+            0.008010957390069962,
+            0.004406375344842672,
+            0.007746017072349787,
+            0.0013752492377534509,
+            0.01063944399356842,
+            -0.0017534863436594605,
+            0.05111953243613243,
+            -0.016858568415045738,
+            -0.005671813618391752,
+            -0.006860558874905109,
+            0.0004283491871319711,
+            -0.003201942890882492,
+            0.02147410809993744,
+            0.0023478588555008173,
+            0.0015085908817127347,
+            -0.004789841361343861,
+            -0.019368529319763184,
+            -0.0007015688461251557,
+            0.0009351345943287015,
+            0.0014475849457085133,
+            -0.007990040816366673,
+            0.006208666600286961,
+            -0.028390442952513695,
+            -0.015213149599730968,
+            0.008066734299063683,
+            -0.007773905526846647,
+            0.0004348855291027576,
+            -0.005490538664162159,
+            -0.0015164344804361463,
+            0.0007434015278704464,
+            0.022115541622042656,
+            -0.015059763565659523,
+            -0.0059646423906087875,
+            -0.012682273052632809,
+            0.0005586405168287456,
+            -0.029645422473549843,
+            0.003036355134099722,
+            -0.0017473857151344419,
+            0.022073708474636078,
+            -0.02624303288757801,
+            -0.024932274594902992,
+            0.006362053100019693,
+            0.00866633653640747,
+            -0.0028515940066426992,
+            -0.008687252178788185,
+            0.014599604532122612,
+            -0.02203187718987465,
+            0.019996019080281258,
+            0.0018022911390289664,
+            -0.03374502435326576,
+            -0.009258965961635113,
+            0.014501994475722313,
+            -0.01691434532403946,
+            0.00152950722258538,
+            -0.004800299648195505,
+            0.02051195502281189,
+            0.01777888834476471,
+            -0.010011954233050346,
+            -0.017290839925408363,
+            0.0005067854654043913,
+            0.019047811627388,
+            -0.005828686058521271,
+            0.02805578149855137,
+            -0.023886458948254585,
+            0.012298806570470333,
+            -0.024792833253741264,
+            0.01010956335812807,
+            -0.02060956507921219,
+            0.003900896990671754,
+            0.018810760229825974,
+            -0.009921316057443619,
+            0.022450203076004982,
+            -0.016984065994620323,
+            -0.022519923746585846,
+            0.0076414356008172035,
+            -0.010137451812624931,
+            -0.025294825434684753,
+            0.013560758903622627,
+            -0.023593628779053688,
+            -0.013058766722679138,
+            0.0042878491804003716,
+            -0.014613548293709755,
+            -0.011992033571004868,
+            0.015394425019621849,
+            0.004117032513022423,
+            0.004472610540688038,
+            0.0017343130894005299,
+            -0.021041836589574814,
+            -0.013205181807279587,
+            -0.00663396529853344,
+            0.003052042331546545,
+            -0.0020881476812064648,
+            -0.007599602919071913,
+            -0.0359761007130146,
+            0.020079685375094414,
+            -0.01565936580300331,
+            -0.01382569968700409,
+            -0.0037265941500663757,
+            0.019131477922201157,
+            0.009356575086712837,
+            0.005790339782834053,
+            0.011580679565668106,
+            -0.013358567841351032,
+            -0.007244025357067585,
+            0.006034363526850939,
+            0.018420321866869926,
+            -0.009698208421468735,
+            -0.004273904953151941,
+            -0.010346615687012672,
+            0.006138945464044809,
+            -0.002053287113085389,
+            -0.0039531877264380455,
+            0.00914741214364767,
+            0.011908368207514286,
+            -0.004486554767936468,
+            0.006752490997314453,
+            -0.008945221081376076,
+            0.012187252752482891,
+            -0.017541835084557533,
+            0.0074531883001327515,
+            0.006783865857869387,
+            0.00292480131611228,
+            0.007024403661489487,
+            -0.007202192675322294,
+            0.021097613498568535,
+            0.03586454689502716,
+            0.000885022571310401,
+            0.014962153509259224,
+            0.00420418381690979,
+            0.0039357575587928295,
+            -0.00036712532164528966,
+            0.021655382588505745,
+            -0.005880977027118206,
+            -0.009733069688081741,
+            -0.00405428372323513,
+            -0.014069723896682262,
+            0.01793227344751358,
+            -0.00866633653640747,
+            -0.0015608817338943481,
+            -0.03525100275874138,
+            0.008896416053175926,
+            -0.003384960815310478,
+            -0.005535857751965523,
+            -0.012194225564599037,
+            -0.009942232631146908,
+            0.02310558222234249,
+            -0.0035732078831642866,
+            0.026521917432546616,
+            -0.00013704558659810573,
+            0.027525901794433594,
+            -0.011455181054770947,
+            -0.00563695328310132,
+            -0.015729086473584175,
+            -0.004312251694500446,
+            0.015087652020156384,
+            0.00485259061679244,
+            0.02922709658741951,
+            0.012821715325117111,
+            -0.0006344622815959156,
+            0.010123508051037788,
+            0.004538845270872116,
+            0.01989840902388096,
+            0.015826696529984474,
+            0.013811754994094372,
+            -0.0033535861875861883,
+            -0.0060378494672477245,
+            0.006710658315569162,
+            0.02081872895359993,
+            0.0016515192110091448,
+            -0.00746016064658761,
+            0.002637201687321067,
+            -0.0071220132522284985,
+            0.016844624653458595,
+            0.011406376026570797,
+            0.026047812774777412,
+            -0.014006974175572395,
+            0.005281375255435705,
+            0.006198208313435316,
+            0.0019853091798722744,
+            0.003897411050274968,
+            -0.0078087663277983665,
+            0.02218526229262352,
+            0.00810856744647026,
+            0.003426793497055769,
+            0.0020637453999370337,
+            -0.004639940802007914,
+            -0.000789156008977443,
+            -0.0053894431330263615,
+            -0.012724106200039387,
+            0.005487052723765373,
+            0.004259960725903511,
+            -0.022352593019604683,
+            -0.0026162853464484215,
+            -0.0138047831133008,
+            0.007704184390604496,
+            0.018322711810469627,
+            0.017235063016414642,
+            0.003268177853897214,
+            -0.0026825203094631433,
+            -0.003932271618396044,
+            0.00030306901317089796,
+            0.008024902082979679,
+            0.0025465642102062702,
+            0.016537850722670555,
+            -0.024583669379353523,
+            -0.01878287084400654,
+            0.010060758329927921,
+            -0.006281873676925898,
+            -0.01311454363167286,
+            0.005860060919076204
+          ],
+          "keyphrases": [
+            "fine-tuning",
+            "dense model",
+            "MoE model",
+            "expert choice routing",
+            "downstream performance"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "46fb6ac4-1a91-4f74-8c57-236109a5fc32",
+        "properties": {
+          "page_content": "B Capacity Factor\n\nWe evaluate the downstream task fine-tuning performance by varying the capacity factors. Note that a capacity factor of n indicates on average how many experts each token can be received. EC-CF2 is our baseline expert choice, which matches GShard top-2 gating computational footprint. EC-CF1, however, matches Switch Transformer top-1 gating computational footprint. EC-CF0.5 further verifies that an aggressively lowered capacity factor can provide strong enough performance, that almost matches the top-2 gating baseline.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Top-2 78.1 87.0 88.3 85.0 82.6 90.1 90.7 81.6 94.7 68.2 67.2 83.0±0.3 EC-CAP2 78.2 88.0 88.5 85.7 83.0 90.8 91.1 80.0 95.4 70.4 64.1 83.2±0.4 EC-CAP3 78.5 91.7 89.3 86.3 83.5 90.9 91.1 81.8 94.9 70.0 65.6 84.0±0.4 EC-CF2 79.1 89.6 89.3 86.8 84.3 91.3 91.2 81.1 95.2 68.1 68.0 84.0±0.2 EC-CF1 77.4 90.6 88.0 85.5 83.6 90.3 91.2 79.8 95.3 66.5 64.9 83.0±0.2 EC-CF0.5 77.4 89.6 86.3 85.2 82.7 91.7 91.0 79.6 94.9 67.3 63.5 83.0 ±0.05 Hash Layers 76.1 85.2 86.7 83.4 82.5 90.0 90.3 75.7 94.0 67.4 63.3 81.3±1.0\n\nTable 2: Comparison between different routing methods in fine-tuning of 100M/64E models. We perform 3 independent fine-tuning runs for each method and report the average results. This gives more accurate difference between the variants of expert choice method, since they achieve close fine-tuning results. We do not report averaged results in other experiments.\n\n",
+          "embedding": [
+            0.00878380797803402,
+            -0.02230931632220745,
+            -0.015165671706199646,
+            -0.002112389076501131,
+            -0.01730332337319851,
+            -0.0011611338704824448,
+            -0.006440163590013981,
+            0.05074397847056389,
+            -0.042286649346351624,
+            -0.006537329405546188,
+            -0.01575644128024578,
+            -0.005678381770849228,
+            -0.013284537009894848,
+            0.007656681817024946,
+            -0.008605021983385086,
+            0.038244541734457016,
+            0.0028061545453965664,
+            0.026398062705993652,
+            0.030253609642386436,
+            -0.007660568691790104,
+            -0.0003531984693836421,
+            -0.035757094621658325,
+            -0.05282721668481827,
+            -0.022464781999588013,
+            -0.01148890983313322,
+            0.020676927641034126,
+            0.011403403244912624,
+            -0.013937492854893208,
+            -0.02294672653079033,
+            0.030082598328590393,
+            0.0195420291274786,
+            0.025667374953627586,
+            -0.0036165190394967794,
+            -0.03728065639734268,
+            -0.0513969361782074,
+            -0.028014905750751495,
+            0.059232402592897415,
+            0.028496848419308662,
+            -0.028963245451450348,
+            0.035601627081632614,
+            0.0162772499024868,
+            0.02532535046339035,
+            -0.04981118440628052,
+            0.0055617825128138065,
+            0.01784745417535305,
+            -0.031683892011642456,
+            -0.03183935955166817,
+            -0.015344456769526005,
+            0.025092151015996933,
+            -0.016354983672499657,
+            -0.020910125225782394,
+            -0.04604891687631607,
+            0.023195471614599228,
+            0.038742031902074814,
+            -0.030642274767160416,
+            0.03675207123160362,
+            0.003350284183397889,
+            0.05873491242527962,
+            0.0325544998049736,
+            0.01063384860754013,
+            0.0017149801133200526,
+            -0.012211824767291546,
+            -0.048194341361522675,
+            -0.008340730331838131,
+            -0.0009648585109971464,
+            0.015235630795359612,
+            -0.019884053617715836,
+            0.0055034831166267395,
+            0.03799579665064812,
+            -0.008364050649106503,
+            -0.06597960740327835,
+            0.04483628645539284,
+            -0.03423352912068367,
+            1.5910934962448664e-05,
+            -0.0011455873027443886,
+            -0.03270996734499931,
+            0.04250429943203926,
+            0.027486322447657585,
+            0.033642761409282684,
+            0.02807709202170372,
+            -0.008022026158869267,
+            0.02874559350311756,
+            -0.025138791650533676,
+            0.006685021799057722,
+            0.014769233763217926,
+            -0.02674008719623089,
+            -0.01284923404455185,
+            -0.05550122633576393,
+            -0.0366898849606514,
+            0.00012388666800688952,
+            -0.02518543042242527,
+            -0.010587208904325962,
+            -0.009304617531597614,
+            0.007656681817024946,
+            -0.02033490315079689,
+            0.05926349386572838,
+            -0.037840332835912704,
+            0.014349476434290409,
+            -0.008830447681248188,
+            -0.00403044605627656,
+            -0.034948673099279404,
+            0.026600167155265808,
+            0.012499435804784298,
+            0.007835467346012592,
+            0.06616616994142532,
+            -0.018857980147004128,
+            -0.014870286919176579,
+            -0.0012155468575656414,
+            -0.01641716994345188,
+            0.07673782855272293,
+            0.0064595965668559074,
+            -0.007330203894525766,
+            -0.03892859071493149,
+            0.005040972959250212,
+            -0.0508994460105896,
+            -0.027735067531466484,
+            -0.03410915657877922,
+            -0.02756405435502529,
+            0.060289569199085236,
+            -0.028512395918369293,
+            0.0017295550787821412,
+            0.05969879776239395,
+            0.024050531908869743,
+            -0.004873847123235464,
+            -0.03622348979115486,
+            0.053200334310531616,
+            0.02229377068579197,
+            -0.0043763574212789536,
+            0.021438708528876305,
+            -0.021936198696494102,
+            -0.0052197580225765705,
+            0.012592715211212635,
+            -0.013851987197995186,
+            -0.039705920964479446,
+            0.03501085937023163,
+            -0.022371502593159676,
+            0.04325053468346596,
+            0.029336363077163696,
+            -0.009926480241119862,
+            -0.018593687564134598,
+            0.028108185157179832,
+            -0.033518388867378235,
+            0.039426080882549286,
+            0.021998384967446327,
+            -0.02530980296432972,
+            0.03911514952778816,
+            0.010493929497897625,
+            0.04974899813532829,
+            0.007866560481488705,
+            0.013976359739899635,
+            -0.011426723562180996,
+            0.016759194433689117,
+            -0.03389150649309158,
+            0.03146624192595482,
+            -0.013408909551799297,
+            -0.026724539697170258,
+            -0.014450529590249062,
+            0.017520975321531296,
+            0.0022406482603400946,
+            -0.03145069628953934,
+            0.0037214583717286587,
+            0.03283433988690376,
+            -0.01626170426607132,
+            -0.017365509644150734,
+            0.002685668645426631,
+            0.04011012986302376,
+            0.02782834693789482,
+            0.012157411314547062,
+            -0.06523337215185165,
+            0.0254497230052948,
+            0.04878511279821396,
+            -0.018718060106039047,
+            0.038493286818265915,
+            0.012312876991927624,
+            0.007306884042918682,
+            0.016603728756308556,
+            0.002194008557125926,
+            0.016743646934628487,
+            -0.010727128013968468,
+            0.022091664373874664,
+            -0.03799579665064812,
+            -0.009654414840042591,
+            0.011838707141578197,
+            -0.0014555469388142228,
+            0.049904465675354004,
+            -0.008154171518981457,
+            0.02073911391198635,
+            0.04567579925060272,
+            -0.000116234841698315,
+            0.005892146844416857,
+            0.025620734319090843,
+            0.038648754358291626,
+            0.013090205378830433,
+            0.005550122819840908,
+            0.007124212104827166,
+            -0.006261378061026335,
+            -0.004663968458771706,
+            0.018547048792243004,
+            0.0328032448887825,
+            0.01475368719547987,
+            -0.029709480702877045,
+            0.014100732281804085,
+            -0.01612178422510624,
+            -0.0195420291274786,
+            -0.0035193529911339283,
+            0.008037572726607323,
+            0.041882436722517014,
+            -0.01888907328248024,
+            -0.005736681632697582,
+            -0.014932473190128803,
+            -0.005464616697281599,
+            -0.018920166417956352,
+            -0.01707012578845024,
+            -0.02913425676524639,
+            -0.01416291855275631,
+            0.030471261590719223,
+            0.007427370175719261,
+            0.0003400810528546572,
+            -0.004457976669073105,
+            -0.0032783811911940575,
+            0.019231097772717476,
+            0.03058008849620819,
+            0.017256684601306915,
+            -0.01717895083129406,
+            -0.017147857695817947,
+            0.00652955612167716,
+            -0.007248584646731615,
+            0.0002594332618173212,
+            0.010571662336587906,
+            -0.013859760016202927,
+            -0.013486642390489578,
+            -0.0009648585109971464,
+            -0.033238548785448074,
+            0.01120907161384821,
+            0.008962593041360378,
+            -0.020676927641034126,
+            0.010385103523731232,
+            -0.011566642671823502,
+            0.0012272067833691835,
+            0.01416291855275631,
+            0.0003378948022145778,
+            0.008177491836249828,
+            -0.0328032448887825,
+            -0.01850040815770626,
+            0.009872066788375378,
+            -0.0005674495478160679,
+            -0.04645312950015068,
+            0.0049982196651399136,
+            0.008612795732915401,
+            -0.010711581446230412,
+            -0.028543489053845406,
+            -0.01823611743748188,
+            0.04424551501870155,
+            0.012965832836925983,
+            -0.008379597216844559,
+            0.03224356845021248,
+            0.007948179729282856,
+            -0.02806154452264309,
+            -0.030486809089779854,
+            -0.006941540166735649,
+            0.04029668867588043,
+            -0.014147371985018253,
+            -0.026398062705993652,
+            0.04023450240492821,
+            -0.018018465489149094,
+            -0.03126413747668266,
+            0.005355790723115206,
+            0.002260081470012665,
+            -0.019060084596276283,
+            0.00021704458049498498,
+            0.02125214971601963,
+            -0.003983806353062391,
+            -0.03448227420449257,
+            -0.0317305326461792,
+            0.0005280972691252828,
+            -0.02715984359383583,
+            -0.029180897399783134,
+            0.009234657511115074,
+            -0.0008647775393910706,
+            0.028403569012880325,
+            0.02832583710551262,
+            0.018935712054371834,
+            0.003678705310449004,
+            0.025931665673851967,
+            -0.023646321147680283,
+            0.014085185714066029,
+            -0.033145271241664886,
+            -0.01508016511797905,
+            -0.042286649346351624,
+            -0.03162170574069023,
+            -0.015896359458565712,
+            -0.024952232837677002,
+            -0.016075145453214645,
+            -0.007034819573163986,
+            0.012359516695141792,
+            -0.02230931632220745,
+            0.0047689080238342285,
+            -0.011621055193245411,
+            0.03410915657877922,
+            -0.0050759525038301945,
+            -0.05444405972957611,
+            -0.004838867578655481,
+            0.012219597585499287,
+            -0.037467215210199356,
+            -0.04384130612015724,
+            -0.0035912557505071163,
+            -0.023195471614599228,
+            -0.0015235630562528968,
+            -0.010167451575398445,
+            -0.0197907742112875,
+            0.028388023376464844,
+            -0.018407128751277924,
+            -0.009211338125169277,
+            -0.03382932022213936,
+            -0.0013010528637096286,
+            0.004438543692231178,
+            -0.014707047492265701,
+            0.005783321335911751,
+            -0.06358543783426285,
+            -0.01641716994345188,
+            0.02467239461839199,
+            -0.006708341650664806,
+            -0.08401361852884293,
+            0.01442720927298069,
+            -0.03373603895306587,
+            0.012250690720975399,
+            0.011838707141578197,
+            0.04051434248685837,
+            -0.02058364823460579,
+            0.007944293320178986,
+            -0.02860567346215248,
+            -0.017381057143211365,
+            -0.035477254539728165,
+            -0.024066077545285225,
+            -0.03805798292160034,
+            -0.013346723280847073,
+            -0.04738591983914375,
+            -0.017738627269864082,
+            -0.016370529308915138,
+            -0.0037136850878596306,
+            -0.02045927569270134,
+            0.035228509455919266,
+            -0.03927061706781387,
+            0.006078705657273531,
+            0.023817332461476326,
+            -0.012989153154194355,
+            -0.010144132189452648,
+            -0.0031384623143821955,
+            -0.009910933673381805,
+            -0.010493929497897625,
+            0.0010017815511673689,
+            0.0013107694685459137,
+            0.030238064005970955,
+            0.027750613167881966,
+            0.006280811037868261,
+            -0.02384842559695244,
+            0.007827693596482277,
+            0.004306397866457701,
+            0.019355470314621925,
+            -0.03905296325683594,
+            -0.012227371335029602,
+            -0.046515315771102905,
+            -0.014069639146327972,
+            0.05155239999294281,
+            -0.03609911724925041,
+            -0.004306397866457701,
+            0.014784780330955982,
+            -0.017365509644150734,
+            -0.002349474234506488,
+            0.005064292810857296,
+            -0.02795271947979927,
+            0.029693933203816414,
+            0.036658793687820435,
+            0.018578141927719116,
+            0.006615062244236469,
+            -0.002516599837690592,
+            0.023630773648619652,
+            -0.03134186938405037,
+            -0.042566485702991486,
+            0.03581928089261055,
+            -0.02715984359383583,
+            0.01494801975786686,
+            -0.005592875648289919,
+            -0.0026545755099505186,
+            -0.011916439980268478,
+            0.06234171614050865,
+            0.025387536734342575,
+            0.012382837012410164,
+            -0.05024648830294609,
+            0.01851595565676689,
+            -0.0023883406538516283,
+            -0.045333776623010635,
+            -0.024750126525759697,
+            -0.022107211872935295,
+            -0.0038477741181850433,
+            -0.006957086734473705,
+            -0.013774254359304905,
+            0.004971013404428959,
+            0.018034012988209724,
+            -0.012872553430497646,
+            -0.02493668533861637,
+            0.02649134211242199,
+            -0.023133285343647003,
+            -0.012631582096219063,
+            0.01120907161384821,
+            -0.004011013079434633,
+            0.00945230945944786,
+            -0.014567128382623196,
+            0.005977652966976166,
+            0.0023961139377206564,
+            -0.0065451026894152164,
+            -0.021407615393400192,
+            -0.010525022633373737,
+            -0.016759194433689117,
+            0.012188504450023174,
+            0.02845020964741707,
+            -0.003931337036192417,
+            0.03622348979115486,
+            0.002846964169293642,
+            0.0024816200602799654,
+            -0.029585108160972595,
+            0.010043079033493996,
+            0.0501532107591629,
+            0.0012388667091727257,
+            0.05012211576104164,
+            -0.003983806353062391,
+            -0.011038059368729591,
+            0.03501085937023163,
+            -0.009530042298138142,
+            0.012942513450980186,
+            0.011170204728841782,
+            0.007493442855775356,
+            0.00652955612167716,
+            -0.010999192483723164,
+            -0.0027089884970337152,
+            -0.0038808106910437346,
+            -0.0025010532699525356,
+            0.035943653434515,
+            -0.022122757509350777,
+            -0.013035791926085949,
+            0.03603693097829819,
+            -0.005118705797940493,
+            -0.016075145453214645,
+            -0.04909604415297508,
+            -0.021967291831970215,
+            -0.0097710145637393,
+            -0.0015206481330096722,
+            0.019573122262954712,
+            -0.02543417550623417,
+            0.020568102598190308,
+            -0.05755337327718735,
+            0.022340409457683563,
+            -0.06283920258283615,
+            0.013533282093703747,
+            0.028667859733104706,
+            0.002365020802244544,
+            0.02294672653079033,
+            -0.018733607605099678,
+            0.02911871112883091,
+            -0.045333776623010635,
+            -0.0071747382171452045,
+            0.016168424859642982,
+            0.03603693097829819,
+            -0.008022026158869267,
+            -0.0027361949905753136,
+            0.011170204728841782,
+            -0.008744941093027592,
+            -0.0325544998049736,
+            -0.006650041788816452,
+            0.02310219220817089,
+            -0.021065590903162956,
+            0.012048585340380669,
+            -0.02244923636317253,
+            -0.0104317432269454,
+            -0.005775548052042723,
+            0.027579601854085922,
+            0.03976810723543167,
+            -0.048007782548666,
+            -0.005472389981150627,
+            7.518220081692562e-05,
+            0.022620247676968575,
+            -0.0025690693873912096,
+            0.01475368719547987,
+            -0.012312876991927624,
+            0.037063002586364746,
+            -0.019215550273656845,
+            0.010610528290271759,
+            -0.009421216323971748,
+            -0.012818140909075737,
+            -0.018049558624625206,
+            -0.011154658161103725,
+            -0.021936198696494102,
+            0.01679028756916523,
+            -0.013828666880726814,
+            0.002605992369353771,
+            -0.032088104635477066,
+            0.03401587903499603,
+            -0.01941765658557415,
+            0.03264778107404709,
+            -0.014279517345130444,
+            0.005437409970909357,
+            -0.024874499067664146,
+            0.01572534814476967,
+            0.005666722077876329,
+            0.01612178422510624,
+            -0.021454256027936935,
+            0.031699441373348236,
+            0.030782192945480347,
+            -0.049904465675354004,
+            -0.02241814322769642,
+            0.0027284217067062855,
+            -0.012631582096219063,
+            0.010268504731357098,
+            -0.01121684443205595,
+            -0.02021053060889244,
+            0.0354461632668972,
+            -0.008348504081368446,
+            0.0052197580225765705,
+            -0.011100245639681816,
+            0.00485052727162838,
+            -0.03028470277786255,
+            -0.006988179869949818,
+            -0.006863807328045368,
+            0.036503326147794724,
+            -0.014380569569766521,
+            0.005235304590314627,
+            -0.0035407296381890774,
+            0.004601782187819481,
+            0.05105490982532501,
+            0.003196761943399906,
+            -0.00014137655671220273,
+            0.00534413056448102,
+            0.004959353245794773,
+            -0.0008458301308564842,
+            0.00011082998389611021,
+            -0.015826400369405746,
+            -0.007893767207860947,
+            0.00015473687381017953,
+            -0.032336849719285965,
+            -0.011263484135270119,
+            0.012219597585499287,
+            0.007610042113810778,
+            0.0029421867802739143,
+            -0.018857980147004128,
+            0.008698301389813423,
+            0.03504195064306259,
+            0.019479842856526375,
+            -0.019510935992002487,
+            0.0027770046144723892,
+            0.0032997578382492065,
+            -0.01207190565764904,
+            -0.028667859733104706,
+            -0.0040343329310417175,
+            0.0026526320725679398,
+            0.0031423489563167095,
+            -0.010734900832176208,
+            0.032212477177381516,
+            -0.027439681813120842,
+            3.349190956214443e-05,
+            -0.027719520032405853,
+            -0.004209231585264206,
+            0.01599741168320179,
+            -0.0036767618730664253,
+            0.03401587903499603,
+            -0.000832226884085685,
+            0.011605508625507355,
+            0.009755467996001244,
+            0.013043565675616264,
+            -0.04076308757066727,
+            0.00925020407885313,
+            0.011240164749324322,
+            -0.01680583320558071,
+            -0.028263650834560394,
+            0.006720001343637705,
+            0.007847127504646778,
+            0.01808065176010132,
+            -0.010159678757190704,
+            -0.018453769385814667,
+            -0.025543002411723137,
+            0.018034012988209724,
+            0.02179628051817417,
+            -0.0419757179915905,
+            0.007703321520239115,
+            0.015453282743692398,
+            0.03420243784785271,
+            -0.0009502836037427187,
+            -0.0036961950827389956,
+            -0.034326810389757156,
+            0.007015386130660772,
+            0.002021053107455373,
+            -0.013167938217520714,
+            0.050712887197732925,
+            -0.010641621425747871,
+            -0.019635308533906937,
+            0.016728101298213005,
+            -0.010727128013968468,
+            -0.018811339512467384,
+            0.02241814322769642,
+            0.014217331074178219,
+            -0.026429155841469765,
+            -0.0005212956457398832,
+            0.016588181257247925,
+            0.029211990535259247,
+            -0.017007939517498016,
+            0.008542835712432861,
+            0.006024292670190334,
+            -0.0019957898184657097,
+            0.015538789331912994,
+            -0.015328910201787949,
+            0.013680974952876568,
+            0.05121037736535072,
+            0.015103485435247421,
+            0.005631742067635059,
+            0.004601782187819481,
+            -0.006156438495963812,
+            0.012079678475856781,
+            -0.004271417856216431,
+            -0.004286964423954487,
+            -0.015119032002985477,
+            0.00011161946167703718,
+            0.05074397847056389,
+            -0.0008594333776272833,
+            0.028419116511940956,
+            -0.0018976522842422128,
+            0.014396116137504578,
+            0.021423162892460823,
+            -0.006914333440363407,
+            -0.02989603951573372,
+            -0.008379597216844559,
+            -0.008511742576956749,
+            -0.0237551461905241,
+            -0.024687940254807472,
+            -0.028698952868580818,
+            -0.026864459738135338,
+            -0.0002560324501246214,
+            -0.0013982189120724797,
+            0.009980892762541771,
+            0.004372470546513796,
+            0.0037972477730363607,
+            0.0008453443297185004,
+            0.004049879498779774,
+            0.023288751021027565,
+            0.013556602410972118,
+            0.020272716879844666,
+            -0.0042558712884783745,
+            0.007326317485421896,
+            -0.0076800016686320305,
+            0.00330947432667017,
+            -0.0035562762059271336,
+            -0.0010795143898576498,
+            -0.0020579760894179344,
+            0.006024292670190334,
+            0.03659660741686821,
+            0.02977166697382927,
+            0.002528259763494134,
+            0.001742186606861651,
+            0.012507209554314613,
+            -0.010120811872184277,
+            0.011815386824309826,
+            -0.0023786239326000214,
+            -0.011076925322413445,
+            0.011325670406222343,
+            0.02073911391198635,
+            -0.014652634970843792,
+            0.015290044248104095,
+            -0.014279517345130444,
+            -0.05519029498100281,
+            0.006082592532038689,
+            -0.017272230237722397,
+            0.022760167717933655,
+            -0.02555854804813862,
+            -0.003968259785324335,
+            -0.017225591465830803,
+            0.00731077091768384,
+            0.014675954356789589,
+            -0.024174904450774193,
+            -0.004325830843299627,
+            0.04082527384161949,
+            0.0013739274581894279,
+            -0.02518543042242527,
+            -0.005355790723115206,
+            -0.003744778223335743,
+            -0.027097657322883606,
+            -0.012538302689790726,
+            0.033114176243543625,
+            -0.009708828292787075,
+            -0.029600653797388077,
+            0.051490213721990585,
+            -0.01462931465357542,
+            0.0022134417667984962,
+            -0.014932473190128803,
+            -0.002841134322807193,
+            -0.03724956139922142,
+            -0.006308017764240503,
+            -0.02860567346215248,
+            0.035601627081632614,
+            -0.024719033390283585,
+            0.026942191645503044,
+            0.023304296657443047,
+            -0.003422186942771077,
+            -0.02139206975698471,
+            0.026833366602659225,
+            -0.030098143965005875,
+            0.0044618635438382626,
+            0.0270199254155159,
+            -0.0005859110387973487,
+            -0.0018228343687951565,
+            0.002858624095097184,
+            -0.021671907976269722,
+            -0.005351903848350048,
+            -0.0017965995939448476,
+            0.00853506289422512,
+            0.027843892574310303,
+            0.009273524396121502,
+            -0.007042592857033014,
+            -0.005923239979892969,
+            0.0012971663381904364,
+            0.012685994617640972,
+            -0.019635308533906937,
+            0.028030451387166977,
+            -0.011605508625507355,
+            0.011123565025627613,
+            -0.008014252409338951,
+            -0.03031579591333866,
+            -0.02345976233482361,
+            0.014963566325604916,
+            0.03696972504258156,
+            0.02308664470911026,
+            0.014279517345130444,
+            0.009195791557431221,
+            0.02174963988363743,
+            -0.00718639837577939,
+            -0.038617659360170364,
+            -0.0028994339518249035,
+            -0.008115305565297604,
+            -0.010361784137785435,
+            0.01812729239463806,
+            0.026289235800504684,
+            0.0024058304261416197,
+            -0.011970852501690388,
+            0.015383323654532433,
+            0.007726641371846199,
+            -0.01901344582438469,
+            0.01943320222198963,
+            -0.010136358439922333,
+            0.021485349163413048,
+            -0.0025651827454566956,
+            0.015709800645709038,
+            0.016852473840117455,
+            0.00045692318235524,
+            0.007194171659648418,
+            -0.023055551573634148,
+            0.029009884223341942,
+            -0.011395630426704884,
+            -0.05736681446433067,
+            0.0036942518781870604,
+            0.0003740891406778246,
+            -0.022091664373874664,
+            -0.013377816416323185,
+            0.015103485435247421,
+            -0.028932152315974236,
+            0.03724956139922142,
+            0.0032025917898863554,
+            -0.005064292810857296,
+            0.023630773648619652,
+            -0.006918220315128565,
+            -0.03488648682832718,
+            0.007959839887917042,
+            0.029072070494294167,
+            -0.01769198849797249,
+            -0.00037797578261233866,
+            0.043623652309179306,
+            0.005278057884424925,
+            -0.03370494768023491,
+            0.009032553061842918,
+            0.03152842819690704,
+            -0.009623321704566479,
+            -0.0007418625173158944,
+            0.012437249533832073,
+            -0.03302089869976044,
+            -0.01784745417535305,
+            0.019308829680085182,
+            0.005009879823774099,
+            0.018951259553432465,
+            -0.028512395918369293,
+            0.02241814322769642,
+            0.008620568551123142,
+            -0.01239838358014822,
+            -0.007524535991251469,
+            0.026569074019789696,
+            0.004496843088418245,
+            0.006377977319061756,
+            -0.015095711685717106,
+            0.013572148978710175,
+            0.049406975507736206,
+            -0.005655061919242144,
+            -0.018671421334147453,
+            0.05599871650338173,
+            -0.02047482319176197,
+            -0.013719840906560421,
+            0.024299276992678642,
+            -0.01822056993842125,
+            -0.007582835853099823,
+            0.03948826715350151,
+            -0.016292797401547432,
+            -0.03386041149497032,
+            0.009669961407780647,
+            -0.031295228749513626,
+            -0.00787822064012289,
+            -0.002689555287361145,
+            0.025014419108629227,
+            -0.013953039422631264,
+            -0.006148665212094784,
+            0.019122270867228508,
+            -0.027595147490501404,
+            -0.00879158079624176,
+            -0.017241137102246284,
+            0.013028019107878208,
+            -0.011325670406222343,
+            -0.02717539109289646,
+            -0.00024036443210206926,
+            -0.0009653443703427911,
+            0.007633361965417862,
+            -0.0025457495357841253,
+            -0.022635795176029205,
+            0.005946559831500053,
+            0.0035873691085726023,
+            -0.011955305933952332,
+            0.06072487309575081,
+            -0.00031287455931305885,
+            0.029196443036198616,
+            -0.02162526734173298,
+            -0.023428669199347496,
+            0.013284537009894848,
+            -0.011123565025627613,
+            0.022387050092220306,
+            0.0038846973329782486,
+            0.019713040441274643,
+            0.0015167614910751581,
+            0.03687644377350807,
+            0.010081945918500423,
+            -0.008962593041360378,
+            -0.032212477177381516,
+            0.01075822114944458,
+            -0.04051434248685837,
+            -0.011395630426704884,
+            0.008830447681248188,
+            -0.02454802207648754,
+            -0.013020245358347893,
+            -0.003565992694348097,
+            0.0206147413700819,
+            0.06498462706804276,
+            0.02373960055410862,
+            -0.009716601110994816,
+            -0.0272686704993248,
+            -0.004566802643239498,
+            -0.011706560850143433,
+            -0.006463483441621065,
+            -0.023133285343647003,
+            0.01192421279847622,
+            -0.045209404081106186,
+            -0.02557409554719925,
+            -0.021671907976269722,
+            -0.0018811339978128672,
+            -0.029320815578103065,
+            -0.016230611130595207,
+            -0.004403563681989908,
+            -0.012787047773599625,
+            0.007711094804108143,
+            -0.01812729239463806,
+            -0.04116729646921158,
+            -0.0016148991417139769,
+            -0.015678707510232925,
+            0.01651044934988022,
+            -0.02756405435502529,
+            -0.021407615393400192,
+            0.012631582096219063,
+            -0.027051018550992012,
+            0.02112777717411518,
+            -0.01108469907194376,
+            0.012763727456331253,
+            -0.010626074858009815,
+            0.031699441373348236,
+            0.0010979759972542524,
+            0.012499435804784298,
+            -0.006276924628764391,
+            0.005829961039125919,
+            -0.032461222261190414,
+            -0.02386397309601307,
+            -0.033767133951187134,
+            0.017878547310829163,
+            -0.014022999443113804,
+            -0.00747400987893343,
+            -0.022651340812444687,
+            0.0011533605866134167,
+            0.009988666512072086,
+            0.010921459645032883,
+            0.0028780573047697544,
+            0.0016090691788122058,
+            0.01081263367086649,
+            -0.008115305565297604,
+            -0.033642761409282684,
+            -0.021718546748161316,
+            -0.023786239326000214,
+            0.017412150278687477,
+            -0.041789159178733826,
+            -0.0016158708604052663,
+            -0.011473363265395164,
+            0.0005684212082996964,
+            -0.02703547105193138,
+            -0.00731465732678771,
+            -0.001748016569763422,
+            0.0021668020635843277,
+            -0.0013583808904513717,
+            0.015919679775834084,
+            0.019852960482239723,
+            0.01207190565764904,
+            -0.02479676716029644,
+            0.05217426270246506,
+            -0.030098143965005875,
+            -0.011496682651340961,
+            0.019324377179145813,
+            0.010369556955993176,
+            0.010703807696700096,
+            -0.021314335986971855,
+            -0.023397576063871384,
+            -0.016619274392724037,
+            0.015119032002985477,
+            0.004419110249727964,
+            -0.008200811222195625,
+            -0.013354497030377388,
+            0.006778301205486059,
+            0.04225555434823036,
+            -0.024843405932188034,
+            -0.00012303645780775696,
+            -0.019775226712226868,
+            0.01417069137096405,
+            -0.00422866502776742,
+            -0.04004794359207153,
+            0.0027964378241449594,
+            -0.02453247457742691,
+            0.002339757513254881,
+            0.01232065074145794,
+            -0.015297817066311836,
+            0.0104317432269454,
+            -0.023630773648619652,
+            0.046484220772981644,
+            0.02059919573366642,
+            -0.012670448049902916,
+            0.03058008849620819,
+            -0.014784780330955982,
+            -0.01075822114944458,
+            0.024454742670059204,
+            0.0051964386366307735,
+            0.031404055655002594,
+            0.0011339273769408464,
+            -0.02414381131529808,
+            0.01916891150176525,
+            -0.012584942393004894,
+            0.03227466344833374,
+            -0.006762754637748003,
+            -0.017925186082720757,
+            -0.006241944618523121,
+            -0.004442430101335049,
+            0.013206804171204567,
+            -0.014605995267629623,
+            -0.009397896938025951,
+            -0.03135741502046585,
+            -0.007120325695723295,
+            0.026833366602659225,
+            0.005351903848350048,
+            -0.0482565276324749,
+            0.0029091504402458668,
+            0.001017328118905425,
+            -0.009825427085161209,
+            0.02229377068579197,
+            -0.006335224024951458,
+            -0.009094739332795143,
+            -0.0033017010428011417,
+            0.035881467163562775,
+            0.05002883821725845,
+            0.012110771611332893,
+            0.004796114284545183,
+            0.012717087753117085,
+            0.036658793687820435,
+            -0.04306397587060928,
+            0.0127248615026474,
+            -0.01494801975786686,
+            -0.03187045082449913,
+            0.02008615806698799,
+            0.0035348995588719845,
+            -0.010284051299095154,
+            0.0337982252240181,
+            -0.015297817066311836,
+            -0.02153198793530464,
+            6.425102765206248e-05,
+            -0.013401136733591557,
+            -0.013556602410972118,
+            -0.007835467346012592,
+            0.0023261543828994036,
+            -0.007302997633814812,
+            -0.007893767207860947,
+            0.0006388665642589331,
+            0.027377495542168617,
+            -0.015017978847026825,
+            0.027921626344323158,
+            0.01528227049857378,
+            0.013432229869067669,
+            -0.021345429122447968,
+            0.012017492204904556,
+            -0.011776520870625973,
+            -2.3729762688162737e-05,
+            0.016230611130595207,
+            -0.0017237251158803701,
+            -0.005969879683107138,
+            0.025822840631008148,
+            -0.0008924698340706527,
+            -0.01952648162841797,
+            -0.006867693737149239,
+            0.008309637196362019,
+            0.0027167617809027433,
+            -0.0014555469388142228,
+            -0.009405669756233692,
+            0.00044502035598270595,
+            0.019852960482239723,
+            0.03765377402305603,
+            -0.034046970307826996,
+            0.004263644572347403,
+            0.003103482536971569,
+            0.0013544942485168576,
+            0.011675468645989895,
+            -0.00826299749314785,
+            -0.006311904173344374,
+            0.019775226712226868,
+            -0.006673361640423536,
+            -0.0016508505214005709,
+            0.006342997308820486,
+            -0.058579444885253906,
+            -0.0149557925760746,
+            0.043219443410634995,
+            -0.01955757476389408,
+            0.0007306884508579969,
+            0.03183935955166817,
+            -0.005037086084485054,
+            0.013066885061562061,
+            -0.029616201296448708,
+            -0.0018306076526641846,
+            -0.0016421056352555752,
+            0.02453247457742691,
+            -0.013066885061562061,
+            -0.002024939749389887,
+            0.017940733581781387,
+            -0.0015799193643033504,
+            -0.01384421344846487,
+            0.024858953431248665,
+            0.01280259434133768,
+            -0.0065528759732842445,
+            2.377530836383812e-05,
+            0.029569560661911964,
+            0.004279191140085459,
+            0.017132312059402466,
+            -0.023273203521966934,
+            0.0104317432269454,
+            -0.0028158710338175297,
+            -0.011271257884800434,
+            0.024330370128154755,
+            -0.005223644897341728,
+            0.011862026527523994,
+            0.025931665673851967,
+            0.031823813915252686,
+            -0.0011795953614637256,
+            -0.004792227875441313,
+            0.0010289880447089672,
+            0.00016068830154836178,
+            0.023646321147680283,
+            0.020381543785333633,
+            -0.011115792207419872,
+            -0.013883080333471298,
+            -0.019992878660559654,
+            0.005655061919242144,
+            -0.0072524710558354855,
+            0.0032744945492595434,
+            0.034948673099279404,
+            0.027346402406692505,
+            0.000937652017455548,
+            0.028045998886227608,
+            -0.016354983672499657,
+            0.017210043966770172,
+            0.011838707141578197,
+            -0.019339922815561295,
+            -0.014357250183820724,
+            0.00619919179007411,
+            -0.016852473840117455,
+            0.011154658161103725,
+            -0.00277117476798594,
+            -0.013276764191687107,
+            -0.006261378061026335,
+            0.08699855953454971,
+            -0.01742769591510296,
+            0.021687453612685204,
+            -0.056869324296712875,
+            -0.027641788125038147,
+            -0.011574415490031242,
+            -0.02359968051314354,
+            0.019837412983179092,
+            0.04312616214156151,
+            0.014030772261321545,
+            0.01943320222198963,
+            0.011310123838484287,
+            -0.006479030009359121,
+            0.03132632374763489,
+            0.0038438874762505293,
+            0.012958060018718243,
+            0.001303967903368175,
+            -0.010501702316105366,
+            -0.013921946287155151,
+            -0.012281783856451511,
+            -0.0015459113055840135,
+            0.012763727456331253,
+            0.005697815213352442,
+            0.010113039053976536,
+            -0.012141864746809006,
+            -0.013657654635608196,
+            -0.006743321195244789,
+            0.021438708528876305,
+            0.00853506289422512,
+            -0.014225104823708534,
+            0.027253123000264168,
+            0.012584942393004894,
+            -0.00026744944625534117,
+            0.010594981722533703,
+            -0.0025088265538215637,
+            0.053697824478149414,
+            -0.02740858867764473,
+            0.007885993458330631,
+            0.031932637095451355,
+            -0.009281297214329243,
+            -0.0027886645402759314,
+            -0.010952552780508995,
+            0.00701927300542593,
+            0.023304296657443047,
+            -0.010672714561223984,
+            0.006747208070009947,
+            0.028045998886227608,
+            -0.005079839378595352,
+            -0.009079192765057087,
+            -0.02139206975698471,
+            -0.008721621707081795,
+            0.01993069238960743,
+            -0.0048194341361522675,
+            0.007427370175719261,
+            -0.004842753987759352,
+            0.015655387192964554,
+            0.00845733005553484,
+            0.0008463159902021289,
+            0.0013272877549752593,
+            -0.0023747372906655073,
+            0.004321944434195757,
+            0.0003070445964112878,
+            0.01666591502726078,
+            0.03625458478927612,
+            0.011823160573840141,
+            0.00236113416031003,
+            0.011310123838484287,
+            -0.0028119843918830156,
+            -0.01548437587916851,
+            0.011457816697657108,
+            -0.00508761266246438,
+            0.002357247518375516,
+            0.009483402594923973,
+            -0.013735387474298477,
+            -0.0060670459643006325,
+            0.016603728756308556,
+            -0.008822673931717873,
+            -0.0034377335105091333,
+            0.008239678107202053,
+            -0.02321101725101471,
+            0.008014252409338951,
+            -0.0010435630101710558,
+            -0.01482364721596241,
+            0.00891595333814621,
+            -0.026973284780979156,
+            -0.0013496358878910542,
+            -0.0012806480517610908,
+            -0.015694255009293556,
+            0.014015225693583488,
+            0.0013127127895131707,
+            0.0011659921146929264,
+            0.0010503645753487945,
+            -0.02073911391198635,
+            0.00040761142736300826,
+            -0.01186980027705431,
+            -0.004100405611097813,
+            -0.0013826723443344235,
+            -0.000706882739905268,
+            -0.05671386048197746,
+            -0.010548342019319534,
+            -0.03389150649309158,
+            0.003078219247981906,
+            0.029693933203816414,
+            0.0003570851113181561,
+            -0.01113911159336567,
+            -0.011722107417881489,
+            0.025154337286949158,
+            -0.004388017114251852,
+            -0.004481296520680189,
+            -0.00277117476798594,
+            -0.008900406770408154,
+            -0.004908827133476734,
+            -0.0009944941848516464,
+            -0.01851595565676689,
+            0.03270996734499931,
+            0.011030285619199276,
+            0.0045978957787156105,
+            -0.03289652615785599,
+            -0.0015585428336635232,
+            -0.0015196764143183827,
+            -0.02464130148291588,
+            -0.016370529308915138,
+            0.023506401106715202,
+            0.011193525046110153,
+            0.0010474496521055698,
+            0.0028255877550691366,
+            0.02345976233482361,
+            -0.018313849344849586,
+            0.021703001111745834,
+            0.01055611576884985,
+            -0.0010940893553197384,
+            3.4888671507360414e-05,
+            0.00423255143687129,
+            -0.014808100648224354,
+            0.014862513169646263,
+            -0.00866720825433731,
+            0.029802760109305382,
+            -0.01756761595606804,
+            0.008970366790890694,
+            -0.009965346194803715,
+            0.019060084596276283,
+            0.03911514952778816,
+            0.030595634132623672,
+            0.026724539697170258,
+            0.012670448049902916,
+            -0.03647223487496376,
+            -0.01009749248623848,
+            -0.002440810203552246,
+            0.005538462661206722,
+            0.009537816047668457,
+            0.004842753987759352,
+            -0.015212311409413815,
+            0.01851595565676689,
+            -0.0005421863752417266,
+            0.006284697912633419,
+            -0.021594174206256866,
+            0.008239678107202053,
+            -0.002110445871949196,
+            0.025418629869818687,
+            0.025745106860995293,
+            0.011963079683482647,
+            -0.0018160328036174178,
+            -0.013035791926085949,
+            -0.007532309275120497,
+            0.02047482319176197,
+            0.013611014932394028,
+            0.022604702040553093,
+            -0.023319844156503677,
+            0.01344000268727541,
+            0.006047612521797419,
+            0.0022659115493297577,
+            0.01940210908651352,
+            0.02227822318673134,
+            0.021454256027936935,
+            0.0013272877549752593,
+            -0.008869313634932041,
+            0.01941765658557415,
+            -0.027346402406692505,
+            0.0049399202689528465,
+            0.0017655064584687352,
+            0.013696521520614624,
+            -0.0364411398768425,
+            0.0016217008233070374,
+            -0.0026273690164089203,
+            -0.0013807290233671665,
+            -0.0113412169739604,
+            -0.029569560661911964,
+            -0.018173931166529655,
+            -0.041509322822093964,
+            0.02204502560198307,
+            -0.004321944434195757,
+            0.031932637095451355,
+            0.007104779127985239,
+            -0.0095222694799304,
+            0.0037972477730363607,
+            0.012219597585499287,
+            0.013603242114186287,
+            0.004710608161985874,
+            0.007862674072384834,
+            -0.023413121700286865,
+            0.00718639837577939,
+            0.029289722442626953,
+            -0.00826299749314785,
+            0.01744324341416359,
+            0.005965993274003267,
+            -0.019852960482239723,
+            0.0041975718922913074,
+            0.011589962057769299,
+            0.007990933023393154,
+            -0.028559034690260887,
+            0.02439255639910698,
+            -0.01515012513846159,
+            -0.009988666512072086,
+            0.009825427085161209,
+            -0.012841460295021534,
+            0.02153198793530464,
+            -0.019852960482239723,
+            -0.01666591502726078,
+            0.00788210704922676,
+            0.003323077689856291,
+            -0.013976359739899635,
+            0.0032064784318208694,
+            0.023910611867904663,
+            -0.017769720405340195,
+            0.0004063968372065574,
+            -0.0045123896561563015,
+            0.02139206975698471,
+            -0.024890046566724777,
+            0.02151644229888916,
+            -0.010439516045153141,
+            -0.019635308533906937,
+            0.016106238588690758,
+            -0.00715141836553812,
+            0.013618788681924343,
+            0.011108018457889557,
+            0.0034727132879197598,
+            0.001421538763679564,
+            0.01245279610157013,
+            0.01572534814476967,
+            0.009615548886358738,
+            -0.011123565025627613,
+            -0.0012524699559435248,
+            -0.005752228200435638,
+            0.01416291855275631,
+            -0.02638251520693302,
+            -0.0002140081487596035,
+            0.021967291831970215,
+            -0.01836048997938633,
+            -0.015562108717858791,
+            0.006832714192569256,
+            0.00468728831037879,
+            0.009382350370287895,
+            0.003812794340774417,
+            -0.03501085937023163,
+            -0.00603595282882452,
+            -0.013167938217520714,
+            -0.003431903664022684,
+            0.03277215361595154,
+            -0.0025748994667083025,
+            -0.014862513169646263,
+            -0.024952232837677002,
+            -1.4673586520075332e-05,
+            0.010120811872184277,
+            0.01968194730579853,
+            -0.002547692973166704,
+            -0.016106238588690758,
+            0.008581702597439289,
+            0.013028019107878208,
+            0.008294090628623962,
+            0.009879840537905693,
+            -0.009483402594923973,
+            0.00682105403393507,
+            0.02689555287361145,
+            0.03647223487496376,
+            -0.009965346194803715,
+            -0.015468829311430454,
+            0.0007501216023229063,
+            -0.018795793876051903,
+            -0.003995466511696577,
+            -0.013028019107878208,
+            -0.005740568041801453,
+            0.0532936155796051,
+            -0.027377495542168617,
+            -0.004862187430262566,
+            -0.02033490315079689,
+            0.00032696363632567227,
+            0.0016090691788122058,
+            -0.001061052898876369,
+            0.007905426435172558,
+            -0.008698301389813423,
+            -0.03243012726306915,
+            0.007501216139644384,
+            0.009957573376595974,
+            0.000347368506481871,
+            0.03249231353402138,
+            0.0113412169739604,
+            -0.013867533765733242,
+            0.01679028756916523,
+            -0.0021881787106394768,
+            -0.0014001622330397367,
+            -0.01941765658557415,
+            -0.013377816416323185,
+            0.0063740904442965984,
+            0.021827373653650284,
+            -0.0017616198165342212,
+            -0.02059919573366642,
+            -0.00793651957064867,
+            0.0052197580225765705,
+            0.00891595333814621,
+            -0.008550609461963177,
+            0.02345976233482361,
+            0.004380243830382824,
+            -0.018578141927719116,
+            0.020910125225782394,
+            -0.018733607605099678,
+            0.0018704457907006145,
+            -0.010167451575398445,
+            -0.0062963576056063175,
+            0.03790251910686493,
+            0.013579921796917915,
+            -0.020770207047462463,
+            -0.013276764191687107,
+            -0.02715984359383583,
+            0.001835466013289988,
+            0.017241137102246284,
+            0.015686480328440666,
+            0.0036825919523835182,
+            -0.002462186850607395,
+            -0.009017006494104862,
+            -0.01311352476477623,
+            -0.025014419108629227,
+            -0.02516988478600979,
+            0.015818627551198006,
+            -0.0009609718690626323,
+            0.04959353432059288,
+            0.00310153909958899,
+            0.004415223840624094,
+            0.005348017439246178,
+            0.004158705472946167,
+            0.007831580936908722,
+            -0.019635308533906937,
+            -0.007306884042918682,
+            -0.0013107694685459137,
+            -0.021811826154589653,
+            0.004772794432938099,
+            0.012880327180027962,
+            -0.012025265954434872,
+            -0.0007020244374871254,
+            -0.001712065190076828,
+            0.01128680445253849,
+            -0.020521461963653564,
+            0.028014905750751495,
+            0.01874915324151516,
+            0.008294090628623962,
+            -0.0003578138421289623,
+            0.02729976363480091,
+            0.00299854320473969,
+            0.012429476715624332,
+            0.0007656681700609624,
+            0.020288264378905296,
+            0.00422866502776742,
+            0.008970366790890694,
+            -0.015935225412249565,
+            0.008542835712432861,
+            -0.015422189608216286,
+            0.035477254539728165,
+            0.0025496361777186394,
+            -0.0388353131711483,
+            0.02271352708339691,
+            -0.006922106724232435,
+            0.0014477736549451947,
+            -0.010618302039802074,
+            -0.026957739144563675,
+            0.01809619925916195,
+            0.01284923404455185,
+            0.010011985898017883,
+            -0.008441783487796783,
+            0.008193038403987885,
+            -0.02174963988363743,
+            0.005892146844416857,
+            -0.01598186604678631,
+            0.012965832836925983,
+            0.0016595955239608884,
+            0.01396858599036932,
+            -0.003507693065330386,
+            0.023646321147680283,
+            0.0071164388209581375,
+            -0.00330947432667017,
+            0.022184943780303,
+            0.0027109317015856504,
+            0.03491757810115814,
+            0.012934739701449871,
+            0.008022026158869267,
+            0.0211122315376997,
+            0.009537816047668457,
+            0.015290044248104095,
+            -0.013758707791566849,
+            -0.004096519201993942,
+            -0.00017647776985540986,
+            0.015017978847026825,
+            -0.006824940908700228,
+            -0.00743902986869216,
+            -0.02571401372551918,
+            0.009327936917543411,
+            -0.0008166803163476288,
+            0.020288264378905296,
+            -0.0077655077911913395,
+            -0.008706075139343739,
+            -0.015002432279288769,
+            -0.044369887560606,
+            -0.011247937567532063,
+            -0.01599741168320179,
+            0.0025418628938496113,
+            0.0024738467764109373,
+            -0.001715951832011342,
+            -0.020039519295096397,
+            0.02162526734173298,
+            -0.0005528746405616403,
+            -0.023972798138856888,
+            -0.0034377335105091333,
+            0.012903646565973759,
+            -0.01626170426607132,
+            0.012546075507998466,
+            -0.004792227875441313,
+            -0.016572635620832443,
+            0.013976359739899635,
+            -0.0011669638333842158,
+            -0.0036981385201215744,
+            -0.027361949905753136,
+            -0.011589962057769299,
+            -0.01455158181488514,
+            0.003192875301465392,
+            0.0019005672074854374,
+            0.010315144434571266,
+            0.011372310109436512,
+            -0.0179718267172575,
+            0.0017256684368476272,
+            0.015398870222270489,
+            0.008224131539463997,
+            0.004302510991692543,
+            0.002841134322807193,
+            0.012079678475856781,
+            -0.009273524396121502,
+            0.0004821863549295813,
+            -0.011403403244912624,
+            -0.01993069238960743,
+            0.005208098329603672,
+            0.010408422909677029,
+            -0.008643888868391514,
+            -0.03538397699594498,
+            -0.007699435111135244,
+            0.013626561500132084,
+            -0.009794333949685097,
+            -0.0009284212719649076,
+            -0.017163405194878578,
+            -0.0005212956457398832,
+            -0.005122592207044363,
+            0.007959839887917042,
+            -0.03019142337143421,
+            -0.0036223488859832287,
+            -0.007777167949825525,
+            -0.0017781381029635668,
+            -0.008177491836249828,
+            -0.0017470449674874544,
+            0.044121142476797104,
+            0.013696521520614624,
+            0.006941540166735649,
+            -0.006405183579772711,
+            -0.004928260110318661,
+            -0.006273037753999233,
+            -0.0003541701298672706,
+            -0.006720001343637705,
+            0.0004105264088138938,
+            -0.014225104823708534,
+            0.016090691089630127,
+            -0.0058455076068639755,
+            0.01430283673107624,
+            0.011862026527523994,
+            -0.007135872263461351,
+            0.026195956394076347,
+            0.0030374096240848303,
+            0.03370494768023491,
+            -0.012421702966094017,
+            0.0014322270872071385,
+            0.017381057143211365,
+            0.0844489261507988,
+            0.013471095822751522,
+            0.008325183764100075,
+            0.0027964378241449594,
+            -0.017940733581781387,
+            -0.009265750646591187,
+            0.003678705310449004,
+            -0.028279196470975876,
+            0.023024458438158035,
+            0.0020987859461456537,
+            0.00997311994433403,
+            -0.007528422866016626,
+            -0.0010843727504834533,
+            0.006024292670190334,
+            0.009763240814208984,
+            0.011815386824309826,
+            0.002611822448670864,
+            0.01285700686275959,
+            0.040483247488737106,
+            0.021547535434365273,
+            0.0014011339517310262,
+            -0.016479356214404106,
+            -0.0010639678221195936,
+            -0.015173444524407387,
+            0.006739434786140919,
+            0.007827693596482277,
+            -0.008744941093027592,
+            0.023055551573634148,
+            0.0021609722170978785,
+            0.010804860852658749,
+            0.013121298514306545,
+            -0.0049127135425806046,
+            0.0021026725880801678,
+            -0.003991579636931419,
+            0.0007374900160357356,
+            0.007497329730540514,
+            -0.006537329405546188,
+            0.008216357789933681,
+            -0.006082592532038689,
+            -0.013859760016202927,
+            0.006999839562922716,
+            -0.010120811872184277,
+            0.003097652457654476,
+            0.0058105275966227055,
+            -0.012514982372522354,
+            0.012048585340380669,
+            0.012491662986576557,
+            0.006261378061026335,
+            0.0006869637290947139,
+            -0.002522429684177041,
+            -0.005208098329603672,
+            0.006074819248169661,
+            0.011636601760983467,
+            -0.0005669636884704232,
+            -0.031792718917131424,
+            -0.0073651839047670364,
+            0.0075361961498856544,
+            -0.004582349210977554,
+            0.005433523561805487,
+            0.01876470074057579,
+            -0.0022892311681061983,
+            -0.013991906307637691,
+            0.00011848181020468473,
+            -0.015049071982502937,
+            -0.00731077091768384,
+            -0.0023144944570958614,
+            0.015834173187613487,
+            -0.010470609180629253,
+            0.016976846382021904,
+            0.012172957882285118,
+            0.00640129717066884,
+            -0.007159191649407148,
+            0.014388343319296837,
+            -0.023972798138856888,
+            0.0019977332558482885,
+            0.0008827532292343676,
+            -0.01161328237503767,
+            -0.0020502028055489063,
+            0.011768747121095657,
+            0.020023971796035767,
+            0.004609555471688509,
+            0.004329717718064785,
+            -0.0011154658859595656,
+            0.013657654635608196,
+            -0.011714334599673748,
+            -0.007816034369170666,
+            0.006673361640423536,
+            0.0006918220315128565,
+            -0.005713361781090498,
+            -0.024719033390283585,
+            -0.0027808912564069033,
+            0.009055872447788715,
+            0.0190445389598608,
+            -0.0047689080238342285,
+            -0.01943320222198963,
+            0.017474336549639702,
+            0.014201784506440163,
+            -0.00899368617683649,
+            -0.010408422909677029,
+            -0.015810852870345116,
+            0.005721135064959526,
+            0.024485835805535316,
+            -0.0014856683555990458,
+            -0.020505916327238083,
+            -0.011325670406222343,
+            -0.014442755840718746,
+            0.018422676250338554,
+            0.007299110759049654,
+            0.011045832186937332,
+            0.0003648583951871842,
+            -0.003826397703960538,
+            0.00024145754287019372,
+            0.014481622725725174,
+            0.020521461963653564,
+            -0.0021318222861737013,
+            -0.0002613765827845782,
+            0.0023008910939097404,
+            0.00033497982076369226,
+            -0.011372310109436512,
+            0.0018072878010571003,
+            0.0024660734925419092,
+            0.017412150278687477,
+            0.0034299602266401052,
+            0.0016693121287971735,
+            0.0049127135425806046,
+            -0.022184943780303,
+            -0.006168098654597998,
+            -0.01665036752820015,
+            -0.008962593041360378,
+            -0.015220084227621555,
+            -0.015919679775834084,
+            -0.005449070129543543,
+            -0.007800487335771322,
+            0.008091985248029232,
+            0.01823611743748188,
+            -0.01108469907194376,
+            -0.0011990285711362958,
+            0.019510935992002487,
+            -0.0031287455931305885,
+            0.004139272030442953,
+            -0.0028936038725078106,
+            -0.0014487452572211623,
+            -0.01876470074057579,
+            -0.003908017184585333,
+            0.015220084227621555,
+            0.010501702316105366,
+            0.012561622075736523,
+            -0.0066383820958435535,
+            0.01940210908651352,
+            -0.025931665673851967,
+            0.025263164192438126,
+            -0.020770207047462463,
+            -0.0014147371985018253,
+            0.021018952131271362,
+            0.0029655066318809986,
+            -0.004609555471688509,
+            -0.015103485435247421,
+            0.01232065074145794,
+            -0.02097231149673462,
+            -0.005550122819840908,
+            -0.034171342849731445,
+            0.03358057513833046,
+            -0.010696034878492355,
+            0.0009998382302001119,
+            -0.004753361456096172,
+            0.0417269729077816,
+            0.012895873747766018,
+            0.005635628942400217,
+            -0.00011544537119334564,
+            -0.014668181538581848,
+            -0.0025379762519150972,
+            0.01240615639835596,
+            -0.0051886653527617455,
+            0.0040071262046694756,
+            0.01260048896074295,
+            0.0066306088119745255,
+            0.0013049395056441426,
+            0.010626074858009815,
+            0.00833295751363039,
+            0.007955953478813171,
+            -0.01449716929346323,
+            -0.010649395175278187,
+            -0.009234657511115074,
+            0.018065106123685837,
+            0.005449070129543543,
+            0.00866720825433731,
+            -0.003737004939466715,
+            0.008402916602790356,
+            0.015686480328440666,
+            -0.007831580936908722,
+            -0.015771986916661263,
+            -0.010571662336587906,
+            -0.016588181257247925,
+            -0.0330519899725914,
+            -0.003834170987829566,
+            0.011955305933952332,
+            -0.006731661502271891,
+            -0.000800162146333605,
+            0.010696034878492355,
+            -0.003268664702773094,
+            0.014232877641916275,
+            0.028792232275009155,
+            -0.0058105275966227055,
+            -0.0016071258578449488,
+            0.0009478544816374779,
+            0.002372794086113572,
+            0.01599741168320179,
+            0.008193038403987885,
+            0.03504195064306259,
+            -0.015375549905002117,
+            -0.0427219532430172,
+            -0.003435790305957198,
+            0.012965832836925983,
+            -0.01494801975786686,
+            -0.021143324673175812,
+            0.005452956538647413,
+            -0.005934900138527155,
+            -0.0052158716134727,
+            0.028698952868580818,
+            0.0021434822119772434,
+            -0.01469927467405796,
+            0.008737168274819851,
+            0.0453648678958416,
+            -0.0031423489563167095,
+            0.013416683301329613,
+            -0.005585102364420891,
+            0.016992392018437386,
+            -0.032865431159734726,
+            -0.013292310759425163,
+            -0.02347530797123909,
+            0.00932016409933567,
+            0.01232842355966568,
+            -0.0008254252607002854,
+            0.032212477177381516,
+            0.016292797401547432,
+            0.012180731631815434,
+            -0.004201458301395178,
+            0.016292797401547432,
+            -0.032057009637355804,
+            -0.04063871502876282,
+            -0.007097005844116211,
+            -0.012864780612289906,
+            -0.009965346194803715,
+            0.001216518459841609,
+            0.007295224349945784,
+            0.0013253444340080023,
+            0.02768842689692974,
+            0.0032978144008666277,
+            -0.010874819941818714,
+            -2.778340785880573e-05,
+            -0.0068171676248312,
+            -0.002674008719623089,
+            -0.019339922815561295,
+            0.010571662336587906,
+            0.008729394525289536,
+            0.016712553799152374,
+            -0.011574415490031242,
+            -0.012079678475856781,
+            0.0012534415582194924,
+            -0.021189963445067406,
+            -0.019308829680085182,
+            -0.027455229312181473,
+            0.007831580936908722,
+            -0.015709800645709038,
+            0.012872553430497646,
+            0.011247937567532063,
+            0.013743161223828793,
+            0.0018344942945986986,
+            -0.013129071332514286,
+            0.027843892574310303,
+            -0.015841946005821228,
+            0.007217491511255503,
+            0.01344000268727541,
+            0.00871384795755148,
+            0.00820858497172594,
+            0.0021609722170978785,
+            -0.027641788125038147,
+            -0.011691015213727951,
+            0.015663161873817444,
+            -0.010423969477415085,
+            0.005794981028884649,
+            -0.009335710667073727,
+            -0.0017927129520103335,
+            -0.007816034369170666,
+            -0.008224131539463997,
+            -0.003958543296903372,
+            0.011574415490031242,
+            0.01462931465357542,
+            0.0012952230172231793,
+            0.026833366602659225,
+            -0.023008912801742554,
+            0.004003239795565605,
+            -0.005184778477996588,
+            0.0067744143307209015,
+            -0.001097004278562963,
+            0.006685021799057722,
+            0.018142838031053543,
+            -0.007163078524172306,
+            0.011504456400871277,
+            0.00587271386757493,
+            -0.015639841556549072,
+            -0.012172957882285118,
+            -0.02137652225792408,
+            0.002742024837061763,
+            -0.016603728756308556,
+            -0.0005878543597646058,
+            -0.001586721045896411,
+            -0.003968259785324335,
+            0.021936198696494102,
+            -0.0040926323272287846,
+            -0.023444214835762978,
+            0.024066077545285225,
+            0.011605508625507355,
+            0.023553041741251945,
+            -0.0030354661867022514,
+            0.030533447861671448,
+            -0.011706560850143433,
+            0.002701215213164687,
+            -0.027626240625977516,
+            0.03187045082449913,
+            -0.00419368501752615,
+            0.009763240814208984,
+            -0.011434496380388737,
+            -0.01148890983313322,
+            -0.019868506118655205,
+            -0.0043685841374099255,
+            0.008744941093027592,
+            0.012577168643474579,
+            0.0051031592302024364,
+            -0.015803080052137375,
+            0.023521948605775833,
+            0.004858300555497408,
+            -0.01239838358014822,
+            0.0009920649463310838,
+            0.009351257234811783,
+            -0.009032553061842918,
+            -0.006444049999117851,
+            -0.0058183008804917336,
+            0.01848486252129078,
+            -0.006852147169411182,
+            -0.01089036650955677,
+            -0.03267887234687805,
+            -0.01575644128024578,
+            -0.0033580574672669172,
+            0.0021706887055188417,
+            0.01927773654460907,
+            -0.007769394665956497,
+            -0.01795627921819687,
+            0.0028178144712001085,
+            0.010004213079810143,
+            -0.016044052317738533,
+            0.00011028342851204798,
+            -0.0018976522842422128,
+            0.01572534814476967,
+            -0.0047689080238342285,
+            0.018205024302005768,
+            0.0208634864538908,
+            0.036907538771629333,
+            0.0022659115493297577,
+            -0.0029557901434600353,
+            0.010734900832176208,
+            0.010610528290271759,
+            -0.008472876623272896,
+            -0.007384616881608963,
+            -0.007722754962742329,
+            -0.02215385064482689,
+            0.034451182931661606,
+            -0.009219110943377018,
+            -0.006063159089535475,
+            -0.00067287462297827,
+            0.022635795176029205,
+            -0.0034902032930403948,
+            -0.025263164192438126,
+            -0.024765674024820328,
+            -0.011146885342895985,
+            0.04645312950015068,
+            -0.021842919290065765,
+            -0.01153554953634739,
+            -0.026335876435041428,
+            0.01745878905057907,
+            -0.02017943747341633,
+            -0.0035193529911339283,
+            0.0011076926020905375,
+            0.005569555796682835,
+            -0.0033036444801837206,
+            -0.005324697587639093,
+            -0.004213118460029364,
+            0.0054685031063854694,
+            0.01318348478525877,
+            0.016572635620832443,
+            -0.014380569569766521,
+            0.021423162892460823,
+            -0.00932016409933567,
+            0.008325183764100075,
+            -0.016479356214404106,
+            0.007990933023393154,
+            -0.023257657885551453,
+            0.024485835805535316,
+            -0.007812147494405508,
+            -0.024205997586250305,
+            0.012297330424189568,
+            0.012499435804784298,
+            0.012693768367171288,
+            0.00015303646796382964,
+            0.012841460295021534,
+            -0.023677414283156395,
+            -0.0005470446776598692,
+            -0.006929880008101463,
+            -0.00426753144711256,
+            -0.01311352476477623,
+            -0.011317897588014603,
+            0.013657654635608196,
+            -0.0010814577108249068,
+            -0.012118545360863209,
+            0.019386563450098038,
+            -0.0066306088119745255,
+            -0.012996925972402096,
+            -0.02478121966123581,
+            0.006541215814650059,
+            -0.007827693596482277,
+            0.005324697587639093,
+            -0.0007005669758655131,
+            0.01599741168320179,
+            -0.005456843413412571,
+            -0.001588664366863668,
+            0.008814901113510132,
+            0.013704294338822365,
+            0.004566802643239498,
+            -0.012717087753117085,
+            -0.023661866784095764,
+            0.0007758706342428923,
+            0.011504456400871277,
+            0.014730367809534073,
+            -0.006580082233995199,
+            -0.017474336549639702,
+            -0.00017793526058085263,
+            -0.0010843727504834533,
+            -0.00970105454325676,
+            -0.0041975718922913074,
+            -0.00041465595131739974,
+            -0.011131338775157928,
+            -0.003937166649848223,
+            -0.01076599396765232,
+            0.017349964007735252,
+            0.0032025917898863554,
+            0.015546562150120735,
+            -0.0040693124756217,
+            -0.019728587940335274,
+            0.012056359089910984,
+            0.041011832654476166,
+            -0.019713040441274643,
+            0.021967291831970215,
+            -0.008099758997559547,
+            -0.0015993525739759207,
+            -0.014862513169646263,
+            0.015468829311430454,
+            -0.006665588356554508,
+            0.018733607605099678,
+            0.020925672724843025,
+            0.008643888868391514,
+            -0.0021551421377807856,
+            0.016836926341056824,
+            0.021687453612685204,
+            -0.0008210528176277876,
+            0.0062380582094192505,
+            0.019184457138180733,
+            0.008286317810416222,
+            0.005363564006984234,
+            -0.0020016198977828026,
+            -0.030067050829529762,
+            0.006805507466197014,
+            -0.0005766802933067083,
+            -0.004803887568414211,
+            0.021189963445067406,
+            0.02504551224410534,
+            0.010991419665515423,
+            0.0033036444801837206,
+            0.006148665212094784,
+            0.012095225043594837,
+            0.0026409721467643976,
+            0.0031540088821202517,
+            -0.0042558712884783745,
+            -0.0018685024697333574,
+            -0.009856520220637321,
+            0.004003239795565605,
+            0.008177491836249828,
+            -0.004691175185143948,
+            0.017489882186055183,
+            -0.01148890983313322,
+            -0.014504942111670971,
+            -0.010260730981826782,
+            -0.014186237938702106,
+            0.02176518738269806,
+            -0.017878547310829163,
+            -0.000856032595038414,
+            -0.0036301221698522568,
+            -6.595143349841237e-05,
+            0.00037141708889976144,
+            -0.00853506289422512,
+            0.02188955992460251,
+            0.008410690352320671,
+            -0.022651340812444687,
+            0.01154332235455513,
+            -0.004240324720740318,
+            0.005798867903649807,
+            -0.006995953153818846,
+            0.01030737068504095,
+            -0.01598186604678631,
+            0.010571662336587906,
+            -0.008845994248986244,
+            0.02296227216720581,
+            -0.013090205378830433,
+            -0.005425750277936459,
+            0.0041120657697319984,
+            -0.002913037082180381,
+            0.01693020574748516,
+            -0.0037991912104189396,
+            -0.010369556955993176,
+            0.003678705310449004,
+            0.011963079683482647,
+            -0.006350770592689514,
+            0.009506722912192345,
+            0.011185751296579838,
+            0.003565992694348097,
+            -0.006109798792749643,
+            -0.0015595145523548126,
+            0.00619919179007411,
+            -0.02927417680621147,
+            -0.00925020407885313,
+            0.0213609766215086,
+            -0.017862999811768532,
+            -0.004916600417345762,
+            0.017520975321531296,
+            0.015445509925484657,
+            -0.01916891150176525,
+            -0.00558121595531702,
+            0.008744941093027592,
+            -0.0020657493732869625,
+            0.023179924115538597,
+            -0.009351257234811783,
+            -0.01719449833035469,
+            0.01277927402406931,
+            -0.013867533765733242,
+            0.011022512800991535,
+            -0.013261217623949051,
+            0.000477813882753253,
+            -0.015639841556549072,
+            0.007952066138386726,
+            -0.014100732281804085,
+            -0.022216036915779114,
+            0.011675468645989895,
+            -0.008690528571605682,
+            -0.001947206910699606,
+            -0.023972798138856888,
+            0.0299115851521492,
+            0.032057009637355804,
+            0.026133770123124123,
+            -0.02005506493151188,
+            0.0026293122209608555,
+            -0.007695548236370087,
+            -0.018438221886754036,
+            -0.0034688266459852457,
+            -0.01166769489645958,
+            0.01654154248535633,
+            0.01940210908651352,
+            -0.002988826483488083,
+            -0.018142838031053543,
+            -0.016090691089630127,
+            0.004671741742640734,
+            -0.02202947810292244,
+            -0.01113911159336567,
+            -0.014613768085837364,
+            -0.0004333604301791638,
+            -0.02112777717411518,
+            0.007613928988575935,
+            0.002753684762865305,
+            0.008892633952200413,
+            0.0028294743970036507,
+            0.003171498654410243,
+            -0.019884053617715836,
+            -0.007170851808041334,
+            0.011076925322413445,
+            0.007190284784883261,
+            -0.012740408070385456,
+            0.02795271947979927,
+            -0.009265750646591187,
+            -0.0015750611200928688,
+            -0.0020754660945385695,
+            -0.018857980147004128,
+            0.013525509275496006,
+            0.010214091278612614,
+            -0.016090691089630127,
+            -0.0071669649332761765,
+            -0.010548342019319534,
+            0.0025612961035221815,
+            0.010797087103128433,
+            -0.001009554835036397,
+            -0.006129232235252857,
+            -0.010369556955993176,
+            0.008418463170528412,
+            -0.011154658161103725,
+            -0.00800647959113121,
+            -0.012942513450980186,
+            0.002843077527359128,
+            0.017520975321531296,
+            0.004154818598181009,
+            0.0020793527364730835,
+            -0.019479842856526375,
+            0.005351903848350048,
+            -0.01795627921819687,
+            0.0003585426020435989,
+            0.0009386236779391766,
+            -0.020365996286273003,
+            0.007959839887917042,
+            0.006428503431379795,
+            0.004543482791632414,
+            0.003585425904020667,
+            -0.005690041929483414,
+            0.008643888868391514,
+            -0.02109668403863907,
+            -0.010952552780508995,
+            0.025154337286949158,
+            -0.0025826727505773306,
+            0.015360003337264061,
+            0.012157411314547062,
+            -0.0024991098325699568,
+            -0.009592228569090366,
+            -0.026817819103598595,
+            -0.0035232396330684423,
+            -0.007835467346012592,
+            0.011481136083602905,
+            0.0019977332558482885,
+            -0.028932152315974236,
+            -0.0036262355279177427,
+            -0.026600167155265808,
+            -0.0027808912564069033,
+            -0.019060084596276283,
+            0.00800647959113121,
+            0.018205024302005768,
+            0.00252437312155962,
+            0.006615062244236469,
+            0.00041538471123203635,
+            0.0011018626391887665,
+            -0.0027439682744443417,
+            0.013541055843234062,
+            0.018034012988209724,
+            0.011706560850143433,
+            -0.019231097772717476,
+            0.011045832186937332,
+            -0.003169555449858308,
+            -0.020801300182938576,
+            0.006548989098519087,
+            -0.02558964118361473,
+            -0.007089232560247183,
+            -3.501012906781398e-05,
+            0.0018626725068315864,
+            -0.011760974302887917,
+            -1.905364842968993e-05,
+            -0.0007433199789375067,
+            0.001582834403961897,
+            0.005114818923175335,
+            0.013595468364655972,
+            -0.012219597585499287,
+            0.019308829680085182,
+            0.006583969108760357,
+            0.008185264654457569,
+            0.0020910126622766256,
+            0.0048544141463935375,
+            -0.013074658811092377,
+            -0.03389150649309158,
+            -0.017940733581781387,
+            -0.013028019107878208,
+            0.008340730331838131,
+            0.0029616199899464846,
+            0.005410203710198402,
+            -0.02454802207648754,
+            -0.016588181257247925,
+            0.012250690720975399,
+            0.01680583320558071,
+            0.00793651957064867,
+            0.007998705841600895,
+            -0.014139598235487938,
+            -0.00846510287374258,
+            0.0038147377781569958,
+            -8.122471626847982e-06,
+            0.01403854601085186,
+            0.00763724884018302,
+            -0.006937653291970491,
+            0.008278544060885906,
+            -0.0023961139377206564,
+            -0.02321101725101471,
+            -0.0019578952342271805,
+            0.010004213079810143,
+            -0.007784941233694553,
+            -0.006420730147510767,
+            -0.0009779758984223008,
+            0.008519516326487064,
+            0.011582189239561558,
+            -0.00443077040836215,
+            -0.0013904456282034516,
+            -0.0042830780148506165,
+            0.018982352688908577,
+            -0.015181218273937702,
+            -0.006273037753999233,
+            0.010937006212770939,
+            -0.004831094294786453,
+            0.013579921796917915,
+            0.006113685667514801,
+            0.04076308757066727,
+            -0.012413930147886276,
+            -0.013984132558107376,
+            -0.014738140627741814,
+            -0.008908180519938469,
+            -0.031046485528349876,
+            -0.01812729239463806,
+            -0.022262677550315857,
+            -0.005923239979892969,
+            0.0035446162801235914,
+            0.011691015213727951,
+            0.013774254359304905,
+            -0.007135872263461351,
+            -0.0002958705008495599,
+            0.002988826483488083,
+            0.0008429151494055986,
+            -0.01088259369134903,
+            -0.008954820223152637,
+            -0.0009687451529316604,
+            -0.014147371985018253,
+            0.002203725278377533,
+            -0.018920166417956352,
+            0.009716601110994816,
+            0.018982352688908577,
+            0.013719840906560421,
+            -0.007967612706124783,
+            0.01265490148216486,
+            -0.0136343352496624,
+            0.0013807290233671665,
+            -0.007952066138386726,
+            0.011675468645989895,
+            0.03488648682832718,
+            0.0038108511362224817,
+            -0.02257360890507698,
+            0.0009765184368006885,
+            -0.0012524699559435248,
+            -0.018780246376991272,
+            0.0056706084869802,
+            -0.021687453612685204,
+            0.01943320222198963,
+            -0.015461056493222713,
+            -0.006770527921617031,
+            6.097167351981625e-05,
+            0.0009036439005285501,
+            0.02479676716029644,
+            0.010742674581706524,
+            -0.015119032002985477,
+            0.019961785525083542,
+            0.011263484135270119,
+            -0.006168098654597998,
+            0.007990933023393154,
+            0.02229377068579197,
+            -0.01029959786683321,
+            0.010719354264438152,
+            0.01943320222198963,
+            -0.02031935751438141,
+            -0.009460083208978176,
+            -0.00301797641441226,
+            0.019324377179145813,
+            0.00845733005553484,
+            -0.009436762891709805,
+            0.0036301221698522568,
+            0.00021121461759321392,
+            0.004504616372287273,
+            -0.0014244538033381104,
+            -0.011240164749324322,
+            -0.006012632977217436,
+            -0.004038219340145588,
+            -0.0001887449761852622,
+            0.010517248883843422,
+            -0.015072392299771309,
+            -0.005942673422396183,
+            -0.0018490692600607872,
+            0.015367777086794376,
+            -0.022884540259838104,
+            -0.0017295550787821412,
+            0.005068179219961166,
+            -0.006397410295903683,
+            0.007139758672565222,
+            0.020023971796035767,
+            0.01783190667629242,
+            0.033518388867378235,
+            -0.014528262428939342,
+            0.0033483407460153103,
+            0.02232486382126808,
+            -0.0036106889601796865,
+            0.02031935751438141,
+            0.014403889887034893,
+            -0.004178138449788094,
+            -0.007470123004168272,
+            0.013572148978710175,
+            0.029989318922162056,
+            0.011955305933952332,
+            0.001268016523681581,
+            -0.02336648292839527,
+            -0.0017888263100758195,
+            0.021594174206256866,
+            0.009086965583264828,
+            0.0005387855344451964,
+            0.010346237570047379,
+            0.014815873466432095,
+            -0.013152391649782658,
+            -0.011201297864317894,
+            0.033238548785448074,
+            -0.017132312059402466,
+            -0.02451692894101143,
+            -0.00046615395694971085,
+            -0.005472389981150627,
+            -0.06116017699241638,
+            -0.002351417439058423,
+            0.014668181538581848,
+            -0.0032589479815214872,
+            0.018951259553432465,
+            0.007555629126727581,
+            0.00866720825433731,
+            0.0049982196651399136,
+            -0.01371206808835268,
+            0.003525183070451021,
+            -0.017163405194878578,
+            0.005775548052042723,
+            -0.006245831493288279,
+            -0.003826397703960538,
+            -0.0037739279214292765,
+            0.007497329730540514,
+            0.023413121700286865,
+            -0.0022348184138536453,
+            0.013533282093703747,
+            0.00669279508292675,
+            0.0023280975874513388,
+            0.01808065176010132,
+            0.006595628801733255,
+            0.00406542606651783,
+            0.0008555467356927693,
+            -0.003579595824703574,
+            -0.010315144434571266,
+            -0.018686966970562935,
+            -0.017754174768924713,
+            -0.014442755840718746,
+            0.006012632977217436,
+            0.002687611849978566,
+            -0.01860923506319523,
+            -0.006953199859708548,
+            0.013976359739899635,
+            -0.03348729386925697,
+            0.002287287963554263,
+            0.016852473840117455,
+            0.014248424209654331,
+            0.009755467996001244,
+            0.02165636047720909,
+            0.023273203521966934,
+            -0.0081386249512434,
+            -0.021874012425541878,
+            -0.006949313450604677,
+            0.001381700742058456,
+            -0.012468342669308186,
+            0.017894092947244644,
+            0.023661866784095764,
+            -0.013999679125845432,
+            -0.010804860852658749,
+            -0.004800001159310341,
+            -0.006078705657273531,
+            0.014248424209654331,
+            0.01640162244439125,
+            0.01968194730579853,
+            0.006144778802990913,
+            0.010113039053976536,
+            -0.017552068457007408,
+            0.012491662986576557,
+            0.0133855901658535,
+            -0.009149151854217052,
+            -0.019977333024144173,
+            -0.019184457138180733,
+            -0.021936198696494102,
+            -0.01665036752820015,
+            -0.024750126525759697,
+            0.028263650834560394,
+            0.002454413566738367,
+            0.013129071332514286,
+            0.00977878738194704,
+            -0.014528262428939342,
+            -0.0022348184138536453,
+            0.011263484135270119,
+            0.01680583320558071,
+            0.003188988659530878,
+            0.0005562754231505096,
+            -0.0002934413496404886,
+            0.014124051667749882,
+            -0.015134578570723534,
+            0.007656681817024946,
+            -0.004706721752882004,
+            -0.009910933673381805,
+            0.0034902032930403948,
+            0.005445183254778385,
+            -0.0077655077911913395,
+            -0.012763727456331253,
+            -0.007753848098218441,
+            0.0036689885891973972,
+            -0.007909313775599003,
+            -0.003435790305957198,
+            0.00037165998946875334,
+            -0.0005562754231505096,
+            0.018920166417956352,
+            -0.015064618550240993,
+            -0.0013243727153167129,
+            0.010338463820517063,
+            -0.0064090704545378685,
+            -0.00682105403393507,
+            0.004011013079434633,
+            0.00788210704922676,
+            -0.0063740904442965984,
+            0.007781054358929396,
+            0.016075145453214645,
+            -0.009685507975518703,
+            0.012227371335029602,
+            -0.002194008557125926,
+            0.00587271386757493,
+            0.023008912801742554,
+            -0.001738299964927137,
+            0.002992713125422597,
+            0.011838707141578197,
+            -0.021407615393400192,
+            0.0007097977213561535,
+            -0.015935225412249565,
+            -0.012748180888593197,
+            -0.015546562150120735,
+            -0.010493929497897625,
+            -0.02022607810795307,
+            -0.006704454775899649,
+            0.00937457662075758,
+            -0.004178138449788094,
+            0.010159678757190704,
+            -0.0011941703269258142,
+            -5.492916170624085e-05,
+            -0.020537009462714195,
+            0.0059815398417413235,
+            -0.00788210704922676,
+            0.0016731987707316875,
+            0.0018306076526641846,
+            0.027517415583133698,
+            0.005627855658531189,
+            -0.0029791099950671196,
+            0.005522916093468666,
+            0.013253443874418736,
+            -0.01812729239463806,
+            -0.010610528290271759,
+            0.012087452225387096,
+            0.009631095454096794,
+            -0.009382350370287895,
+            -0.009677735157310963,
+            -0.010820407420396805,
+            -0.031543973833322525,
+            0.006902673747390509,
+            -0.009561135433614254,
+            0.001341862604022026,
+            -0.0002895547077059746,
+            -0.04278413951396942,
+            0.006739434786140919,
+            -0.0019063971703872085,
+            -0.013346723280847073,
+            0.002019109670072794,
+            -0.019619761034846306,
+            0.00871384795755148,
+            -0.01599741168320179,
+            0.01331563014537096,
+            -0.012040812522172928,
+            -0.0023242109455168247,
+            0.009980892762541771,
+            0.023288751021027565,
+            0.0058455076068639755,
+            0.005740568041801453,
+            0.007120325695723295,
+            -8.714576688362285e-05,
+            0.005266397725790739,
+            -0.0022814578842371702,
+            0.0022387050557881594,
+            -0.0017791097052395344,
+            0.015103485435247421,
+            0.002036599675193429,
+            0.01654154248535633,
+            -0.007784941233694553,
+            -0.002930526854470372,
+            -0.008123078383505344,
+            0.01174542773514986,
+            -0.023288751021027565,
+            0.0014720651088282466,
+            0.0007588665466755629,
+            0.0006340082618407905,
+            -0.004123725462704897,
+            0.004535709507763386,
+            -0.007182511501014233,
+            0.031046485528349876,
+            0.016557088121771812,
+            -0.009102512151002884,
+            0.003991579636931419,
+            5.286438317853026e-05,
+            -0.0001824291975935921,
+            0.021734094247221947,
+            -0.01993069238960743,
+            -0.0056744953617453575,
+            0.0027187049854546785,
+            -0.01901344582438469,
+            0.0021881787106394768,
+            0.028512395918369293,
+            -0.014901380054652691,
+            0.001269959844648838,
+            0.00017319842299912125,
+            0.003350284183397889,
+            -0.018811339512467384,
+            0.001708178548142314,
+            0.014613768085837364,
+            0.007575062569230795,
+            -0.003012146335095167,
+            0.015429963357746601,
+            -0.012382837012410164,
+            0.009483402594923973,
+            0.0058688269928097725,
+            -0.0010124698746949434,
+            0.004318057559430599,
+            -0.000593198521528393,
+            -0.02977166697382927,
+            -0.005095385946333408,
+            0.004038219340145588,
+            -0.0019501218339428306,
+            -0.010999192483723164,
+            0.003651498816907406,
+            -0.009685507975518703,
+            -0.01212631817907095,
+            -0.007034819573163986,
+            0.03721847012639046,
+            0.015515469014644623,
+            0.0045123896561563015,
+            0.012437249533832073,
+            -0.0019404053455218673,
+            0.013035791926085949,
+            0.01848486252129078,
+            -0.016588181257247925,
+            -0.004115952178835869,
+            0.015367777086794376,
+            0.018671421334147453,
+            0.009825427085161209,
+            -0.004049879498779774,
+            0.025682920590043068,
+            0.0024952231906354427,
+            -0.020676927641034126,
+            0.006918220315128565,
+            0.0007384616765193641,
+            0.02596275880932808,
+            -0.00698429299518466,
+            -0.016463808715343475,
+            -0.01901344582438469,
+            0.00949894916266203,
+            -0.005600648932158947,
+            -0.005884373560547829,
+            -0.009677735157310963,
+            -0.004617328755557537,
+            0.026398062705993652,
+            0.008527289144694805,
+            0.0006412956863641739,
+            -0.015911906957626343,
+            0.008558382280170918,
+            -0.01862478069961071,
+            -0.013401136733591557,
+            -0.004003239795565605,
+            0.015771986916661263,
+            -0.02571401372551918,
+            -0.001339919283054769,
+            -0.008449556306004524,
+            -0.00013530367868952453,
+            0.012180731631815434,
+            0.021951746195554733,
+            0.0015624294755980372,
+            0.001924858777783811,
+            -0.006051499396562576,
+            -0.024081625044345856,
+            -0.015251177363097668,
+            -0.013727614656090736,
+            -0.007761621382087469,
+            -0.030673367902636528,
+            -0.010035306215286255,
+            -0.0006534414133056998,
+            0.00013821866014041007,
+            -0.0012388667091727257,
+            0.037591587752103806,
+            -0.006583969108760357,
+            -0.00244469684548676,
+            -0.006342997308820486,
+            0.009079192765057087,
+            -0.010664941743016243,
+            0.006261378061026335,
+            -0.006533442530781031,
+            -0.0002924696891568601,
+            0.004045992624014616,
+            0.002048259600996971,
+            0.02807709202170372,
+            0.002182348631322384,
+            -0.006591742392629385,
+            0.008954820223152637,
+            -0.01284923404455185,
+            -0.021967291831970215,
+            0.01693020574748516,
+            -0.024003891274333,
+            -0.013813120312988758,
+            -0.026600167155265808,
+            -0.016728101298213005,
+            -0.017256684601306915,
+            -0.01680583320558071,
+            -0.039954666048288345,
+            0.0023222677409648895,
+            0.02728421613574028,
+            -0.006471256725490093,
+            0.017132312059402466,
+            -0.0023747372906655073,
+            -0.009436762891709805,
+            -0.0010163565166294575,
+            -0.02007061243057251,
+            0.006300244480371475,
+            -0.0016595955239608884,
+            -0.005763887893408537,
+            0.013743161223828793,
+            -0.006560649257153273,
+            0.017396602779626846,
+            0.002920810366049409,
+            -0.019619761034846306,
+            -0.0033833205234259367,
+            0.01876470074057579,
+            0.023397576063871384,
+            -0.0033638873137533665,
+            -0.000545587157830596,
+            0.003752551507204771,
+            -0.010120811872184277,
+            0.01809619925916195,
+            -0.015585429035127163,
+            0.005002106539905071,
+            0.0012446966720744967,
+            0.014085185714066029,
+            0.005375223699957132,
+            0.0009070447413250804,
+            0.006992066279053688,
+            0.0006242916570045054,
+            0.006611175369471312,
+            -0.007411823607981205,
+            0.00134574924595654,
+            0.001861700788140297,
+            -0.0183449424803257,
+            0.03395369276404381,
+            0.013097978197038174,
+            0.012934739701449871,
+            -0.03684535250067711,
+            0.017660895362496376,
+            0.0016440489562228322,
+            -0.008449556306004524,
+            -0.007660568691790104,
+            0.030129237100481987,
+            0.011978626251220703,
+            0.00365538545884192,
+            0.009017006494104862,
+            -0.010594981722533703,
+            0.012157411314547062,
+            -0.011185751296579838,
+            -0.004586235620081425,
+            -0.005421863403171301,
+            -0.011193525046110153,
+            0.01823611743748188,
+            0.009825427085161209,
+            0.0022425916977226734,
+            -0.0046561951749026775,
+            -0.013393362984061241,
+            0.029989318922162056,
+            0.008022026158869267,
+            -0.004784454591572285,
+            0.008651661686599255,
+            -0.014310610480606556,
+            0.0005718219908885658,
+            0.000107489904621616,
+            0.007567289285361767,
+            -0.016728101298213005,
+            -5.696357402484864e-05,
+            -0.028139278292655945,
+            -0.017552068457007408,
+            -0.02335093729197979,
+            0.0002368421555729583,
+            0.0014438870130106807,
+            0.0177230816334486,
+            -0.0011941703269258142,
+            0.004527936223894358,
+            -7.360325253102928e-05,
+            0.007128098979592323,
+            0.004527936223894358,
+            0.014792554080486298,
+            0.012266237288713455,
+            0.02638251520693302,
+            -0.00570558849722147,
+            -0.028792232275009155,
+            0.0041975718922913074,
+            0.007909313775599003,
+            0.008962593041360378,
+            0.027455229312181473,
+            0.003634008811786771,
+            -0.0068949004635214806,
+            -0.00751287629827857,
+            0.008441783487796783,
+            0.025247616693377495,
+            -0.003908017184585333,
+            -0.018002919852733612,
+            0.018313849344849586,
+            0.020630288869142532,
+            0.0033600006718188524,
+            -0.0040693124756217,
+            0.01703903265297413,
+            0.011442270129919052,
+            -0.019122270867228508,
+            -0.011512229219079018,
+            0.002283401321619749,
+            0.0055889892391860485,
+            0.0028663973789662123,
+            -0.016494901850819588,
+            -0.007645022124052048,
+            -0.03488648682832718,
+            0.007131985388696194,
+            -0.0158652663230896,
+            0.0071436455473303795,
+            0.010369556955993176,
+            -0.03466883301734924,
+            -0.008099758997559547,
+            0.013214577920734882,
+            0.03211919590830803,
+            0.006758867762982845,
+            0.0075439694337546825,
+            -0.0043413774110376835,
+            -0.0014506885781884193,
+            -0.014559355564415455,
+            0.014707047492265701,
+            -0.0328032448887825,
+            0.02229377068579197,
+            0.01441943645477295,
+            0.004780567716807127,
+            -0.0018140894826501608,
+            -0.010276277549564838,
+            0.00558121595531702,
+            -0.0036301221698522568,
+            0.0033774906769394875,
+            -0.03448227420449257,
+            0.0192932840436697,
+            -0.008760487660765648,
+            0.022931179031729698,
+            0.02361522801220417,
+            0.02611822448670864,
+            -0.006335224024951458,
+            -0.0013700408162549138,
+            0.005464616697281599,
+            0.006486803293228149,
+            0.01694575324654579,
+            -0.009436762891709805,
+            -0.0016382189933210611,
+            -0.02073911391198635,
+            0.012701541185379028,
+            -0.023630773648619652,
+            -0.01063384860754013,
+            0.00269149849191308,
+            -0.013603242114186287,
+            -0.025371989235281944,
+            -0.010144132189452648,
+            -0.004100405611097813,
+            -0.0030587860383093357,
+            -0.0016440489562228322,
+            0.008099758997559547,
+            -0.01742769591510296,
+            -0.02465684711933136,
+            -0.014590448699891567,
+            -0.010742674581706524,
+            0.022651340812444687,
+            0.01075822114944458,
+            -0.016572635620832443,
+            -0.0039993529208004475,
+            -0.005682268645614386,
+            0.0071436455473303795,
+            0.007959839887917042,
+            -0.01416291855275631,
+            0.05506592243909836,
+            0.0006393523653969169,
+            0.0027187049854546785,
+            0.0018743324326351285,
+            0.008216357789933681,
+            0.0012913363752886653,
+            0.003585425904020667,
+            -0.002267854753881693,
+            0.004477410111576319,
+            -0.034575555473566055,
+            -0.018173931166529655,
+            0.0033153044059872627,
+            -0.011589962057769299,
+            0.00945230945944786,
+            -0.01915336400270462,
+            0.011589962057769299,
+            -0.01719449833035469,
+            -0.021718546748161316,
+            0.006109798792749643,
+            -0.0027886645402759314,
+            -0.002106559230014682,
+            -0.014652634970843792,
+            -0.017132312059402466,
+            0.007567289285361767,
+            0.02453247457742691,
+            -0.0024738467764109373,
+            0.01547660306096077,
+            -0.02022607810795307,
+            -0.005254738032817841,
+            -0.012810367159545422,
+            0.01089036650955677,
+            -0.010462836362421513,
+            -0.009017006494104862,
+            -0.009017006494104862,
+            -0.02164081484079361,
+            0.017816361039876938,
+            -0.002372794086113572,
+            0.00800647959113121,
+            -0.0206147413700819,
+            0.011224618181586266,
+            -0.010618302039802074,
+            0.016992392018437386,
+            0.000213036488275975,
+            -0.01915336400270462,
+            -0.014116278849542141,
+            0.01680583320558071,
+            -0.023972798138856888,
+            -0.0012835630914196372,
+            0.027346402406692505,
+            0.005934900138527155,
+            0.025807293131947517,
+            -0.006832714192569256,
+            -0.024983325973153114,
+            -0.01311352476477623,
+            0.016494901850819588,
+            -0.00957668200135231,
+            -0.00015947372594382614,
+            -0.00701927300542593,
+            0.007334090769290924,
+            -0.01482364721596241,
+            0.028683407232165337,
+            -0.01822056993842125,
+            0.007998705841600895,
+            0.0015604861546307802,
+            -0.008247450925409794,
+            0.014069639146327972,
+            -0.0048544141463935375,
+            -0.007493442855775356,
+            -0.0077072083950042725,
+            -0.004489069804549217,
+            -0.010664941743016243,
+            -0.0034571667201817036,
+            -0.029973771423101425,
+            0.0005674495478160679,
+            -0.0005266398075036705,
+            -0.012382837012410164,
+            -0.011753200553357601,
+            -0.004469636827707291,
+            -0.004671741742640734,
+            0.0019238870590925217,
+            0.011006966233253479,
+            0.00012704456457868218,
+            -0.01161328237503767,
+            -0.008379597216844559,
+            0.011776520870625973,
+            -0.00718639837577939,
+            0.005690041929483414,
+            -0.02557409554719925,
+            0.01194753311574459,
+            -0.013758707791566849,
+            0.0057600014843046665,
+            -0.009055872447788715,
+            0.019977333024144173,
+            0.03120194934308529,
+            0.0044618635438382626,
+            0.004166478756815195,
+            -0.007442916743457317,
+            0.0029791099950671196,
+            0.002434980357065797,
+            0.016215063631534576,
+            -0.008091985248029232,
+            0.017225591465830803,
+            -0.012748180888593197,
+            0.007769394665956497,
+            -0.012040812522172928,
+            0.01717895083129406,
+            0.006245831493288279,
+            0.0037953045684844255,
+            -0.009584455750882626,
+            -0.0007778139552101493,
+            -0.02897879108786583,
+            0.010672714561223984,
+            -0.01770753413438797,
+            -0.0009969233069568872,
+            0.016215063631534576,
+            0.009405669756233692,
+            -0.013657654635608196,
+            0.012794820591807365,
+            0.03134186938405037,
+            0.024827860295772552,
+            -0.004115952178835869,
+            -0.0012709314469248056,
+            0.01640162244439125,
+            -0.0039993529208004475,
+            0.004982673097401857,
+            -0.009211338125169277,
+            -0.03373603895306587,
+            -0.019184457138180733,
+            -0.018780246376991272,
+            0.0040071262046694756,
+            0.010058625601232052,
+            -0.00020538465469144285,
+            0.0110924718901515,
+            0.010781540535390377,
+            0.019650854170322418,
+            -0.019308829680085182,
+            -0.0034299602266401052,
+            -0.008519516326487064,
+            -0.005095385946333408,
+            0.028154823929071426,
+            0.008542835712432861,
+            0.021438708528876305,
+            -0.011038059368729591,
+            0.006179758347570896,
+            -0.020925672724843025,
+            -0.01371206808835268,
+            -0.01252275612205267,
+            0.021423162892460823,
+            0.0027944946195930243,
+            -0.0013661541743203998,
+            0.02572956122457981,
+            0.005359677132219076,
+            -0.015181218273937702,
+            0.006486803293228149,
+            -0.006191418506205082,
+            0.010408422909677029,
+            0.006261378061026335,
+            0.008053119294345379,
+            -0.025807293131947517,
+            -0.02507660537958145,
+            -0.01585749350488186,
+            0.010369556955993176,
+            0.013020245358347893,
+            0.01101473905146122,
+            -0.014139598235487938,
+            -0.018593687564134598,
+            0.0056395153515040874,
+            0.01331563014537096,
+            -0.008278544060885906,
+            -0.010968099348247051,
+            -0.0036126323975622654,
+            0.00970105454325676,
+            0.020490368828177452,
+            5.5475720728281885e-05,
+            0.007501216139644384,
+            0.01075822114944458,
+            0.005658948794007301,
+            -0.0043413774110376835,
+            0.008123078383505344,
+            -0.010983645915985107,
+            -0.01694575324654579,
+            0.01548437587916851,
+            -0.010385103523731232,
+            0.009156924672424793,
+            0.010268504731357098,
+            -0.010237411595880985,
+            -0.001987044932320714,
+            -0.007046479266136885,
+            0.01280259434133768,
+            0.025760654360055923,
+            -0.004353037569671869,
+            -0.003435790305957198,
+            -0.017210043966770172,
+            0.002506883116438985,
+            0.0023008910939097404,
+            0.018453769385814667,
+            0.0058183008804917336,
+            0.008822673931717873,
+            -0.010719354264438152,
+            -0.0022697981912642717,
+            0.025543002411723137,
+            0.010618302039802074,
+            -0.006704454775899649,
+            -0.00024813771597109735
+          ],
+          "title": "Capacity Factor",
+          "keyphrases": [
+            "capacity factors",
+            "downstream task fine-tuning",
+            "expert choice",
+            "gating computational footprint",
+            "fine-tuning performance"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "9377be8f-b643-455d-ba8b-f68f53a0898c",
+      "type": "next",
+      "source": {
+        "id": "46fb6ac4-1a91-4f74-8c57-236109a5fc32",
+        "properties": {
+          "page_content": "B Capacity Factor\n\nWe evaluate the downstream task fine-tuning performance by varying the capacity factors. Note that a capacity factor of n indicates on average how many experts each token can be received. EC-CF2 is our baseline expert choice, which matches GShard top-2 gating computational footprint. EC-CF1, however, matches Switch Transformer top-1 gating computational footprint. EC-CF0.5 further verifies that an aggressively lowered capacity factor can provide strong enough performance, that almost matches the top-2 gating baseline.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Top-2 78.1 87.0 88.3 85.0 82.6 90.1 90.7 81.6 94.7 68.2 67.2 83.0±0.3 EC-CAP2 78.2 88.0 88.5 85.7 83.0 90.8 91.1 80.0 95.4 70.4 64.1 83.2±0.4 EC-CAP3 78.5 91.7 89.3 86.3 83.5 90.9 91.1 81.8 94.9 70.0 65.6 84.0±0.4 EC-CF2 79.1 89.6 89.3 86.8 84.3 91.3 91.2 81.1 95.2 68.1 68.0 84.0±0.2 EC-CF1 77.4 90.6 88.0 85.5 83.6 90.3 91.2 79.8 95.3 66.5 64.9 83.0±0.2 EC-CF0.5 77.4 89.6 86.3 85.2 82.7 91.7 91.0 79.6 94.9 67.3 63.5 83.0 ±0.05 Hash Layers 76.1 85.2 86.7 83.4 82.5 90.0 90.3 75.7 94.0 67.4 63.3 81.3±1.0\n\nTable 2: Comparison between different routing methods in fine-tuning of 100M/64E models. We perform 3 independent fine-tuning runs for each method and report the average results. This gives more accurate difference between the variants of expert choice method, since they achieve close fine-tuning results. We do not report averaged results in other experiments.\n\n",
+          "embedding": [
+            0.00878380797803402,
+            -0.02230931632220745,
+            -0.015165671706199646,
+            -0.002112389076501131,
+            -0.01730332337319851,
+            -0.0011611338704824448,
+            -0.006440163590013981,
+            0.05074397847056389,
+            -0.042286649346351624,
+            -0.006537329405546188,
+            -0.01575644128024578,
+            -0.005678381770849228,
+            -0.013284537009894848,
+            0.007656681817024946,
+            -0.008605021983385086,
+            0.038244541734457016,
+            0.0028061545453965664,
+            0.026398062705993652,
+            0.030253609642386436,
+            -0.007660568691790104,
+            -0.0003531984693836421,
+            -0.035757094621658325,
+            -0.05282721668481827,
+            -0.022464781999588013,
+            -0.01148890983313322,
+            0.020676927641034126,
+            0.011403403244912624,
+            -0.013937492854893208,
+            -0.02294672653079033,
+            0.030082598328590393,
+            0.0195420291274786,
+            0.025667374953627586,
+            -0.0036165190394967794,
+            -0.03728065639734268,
+            -0.0513969361782074,
+            -0.028014905750751495,
+            0.059232402592897415,
+            0.028496848419308662,
+            -0.028963245451450348,
+            0.035601627081632614,
+            0.0162772499024868,
+            0.02532535046339035,
+            -0.04981118440628052,
+            0.0055617825128138065,
+            0.01784745417535305,
+            -0.031683892011642456,
+            -0.03183935955166817,
+            -0.015344456769526005,
+            0.025092151015996933,
+            -0.016354983672499657,
+            -0.020910125225782394,
+            -0.04604891687631607,
+            0.023195471614599228,
+            0.038742031902074814,
+            -0.030642274767160416,
+            0.03675207123160362,
+            0.003350284183397889,
+            0.05873491242527962,
+            0.0325544998049736,
+            0.01063384860754013,
+            0.0017149801133200526,
+            -0.012211824767291546,
+            -0.048194341361522675,
+            -0.008340730331838131,
+            -0.0009648585109971464,
+            0.015235630795359612,
+            -0.019884053617715836,
+            0.0055034831166267395,
+            0.03799579665064812,
+            -0.008364050649106503,
+            -0.06597960740327835,
+            0.04483628645539284,
+            -0.03423352912068367,
+            1.5910934962448664e-05,
+            -0.0011455873027443886,
+            -0.03270996734499931,
+            0.04250429943203926,
+            0.027486322447657585,
+            0.033642761409282684,
+            0.02807709202170372,
+            -0.008022026158869267,
+            0.02874559350311756,
+            -0.025138791650533676,
+            0.006685021799057722,
+            0.014769233763217926,
+            -0.02674008719623089,
+            -0.01284923404455185,
+            -0.05550122633576393,
+            -0.0366898849606514,
+            0.00012388666800688952,
+            -0.02518543042242527,
+            -0.010587208904325962,
+            -0.009304617531597614,
+            0.007656681817024946,
+            -0.02033490315079689,
+            0.05926349386572838,
+            -0.037840332835912704,
+            0.014349476434290409,
+            -0.008830447681248188,
+            -0.00403044605627656,
+            -0.034948673099279404,
+            0.026600167155265808,
+            0.012499435804784298,
+            0.007835467346012592,
+            0.06616616994142532,
+            -0.018857980147004128,
+            -0.014870286919176579,
+            -0.0012155468575656414,
+            -0.01641716994345188,
+            0.07673782855272293,
+            0.0064595965668559074,
+            -0.007330203894525766,
+            -0.03892859071493149,
+            0.005040972959250212,
+            -0.0508994460105896,
+            -0.027735067531466484,
+            -0.03410915657877922,
+            -0.02756405435502529,
+            0.060289569199085236,
+            -0.028512395918369293,
+            0.0017295550787821412,
+            0.05969879776239395,
+            0.024050531908869743,
+            -0.004873847123235464,
+            -0.03622348979115486,
+            0.053200334310531616,
+            0.02229377068579197,
+            -0.0043763574212789536,
+            0.021438708528876305,
+            -0.021936198696494102,
+            -0.0052197580225765705,
+            0.012592715211212635,
+            -0.013851987197995186,
+            -0.039705920964479446,
+            0.03501085937023163,
+            -0.022371502593159676,
+            0.04325053468346596,
+            0.029336363077163696,
+            -0.009926480241119862,
+            -0.018593687564134598,
+            0.028108185157179832,
+            -0.033518388867378235,
+            0.039426080882549286,
+            0.021998384967446327,
+            -0.02530980296432972,
+            0.03911514952778816,
+            0.010493929497897625,
+            0.04974899813532829,
+            0.007866560481488705,
+            0.013976359739899635,
+            -0.011426723562180996,
+            0.016759194433689117,
+            -0.03389150649309158,
+            0.03146624192595482,
+            -0.013408909551799297,
+            -0.026724539697170258,
+            -0.014450529590249062,
+            0.017520975321531296,
+            0.0022406482603400946,
+            -0.03145069628953934,
+            0.0037214583717286587,
+            0.03283433988690376,
+            -0.01626170426607132,
+            -0.017365509644150734,
+            0.002685668645426631,
+            0.04011012986302376,
+            0.02782834693789482,
+            0.012157411314547062,
+            -0.06523337215185165,
+            0.0254497230052948,
+            0.04878511279821396,
+            -0.018718060106039047,
+            0.038493286818265915,
+            0.012312876991927624,
+            0.007306884042918682,
+            0.016603728756308556,
+            0.002194008557125926,
+            0.016743646934628487,
+            -0.010727128013968468,
+            0.022091664373874664,
+            -0.03799579665064812,
+            -0.009654414840042591,
+            0.011838707141578197,
+            -0.0014555469388142228,
+            0.049904465675354004,
+            -0.008154171518981457,
+            0.02073911391198635,
+            0.04567579925060272,
+            -0.000116234841698315,
+            0.005892146844416857,
+            0.025620734319090843,
+            0.038648754358291626,
+            0.013090205378830433,
+            0.005550122819840908,
+            0.007124212104827166,
+            -0.006261378061026335,
+            -0.004663968458771706,
+            0.018547048792243004,
+            0.0328032448887825,
+            0.01475368719547987,
+            -0.029709480702877045,
+            0.014100732281804085,
+            -0.01612178422510624,
+            -0.0195420291274786,
+            -0.0035193529911339283,
+            0.008037572726607323,
+            0.041882436722517014,
+            -0.01888907328248024,
+            -0.005736681632697582,
+            -0.014932473190128803,
+            -0.005464616697281599,
+            -0.018920166417956352,
+            -0.01707012578845024,
+            -0.02913425676524639,
+            -0.01416291855275631,
+            0.030471261590719223,
+            0.007427370175719261,
+            0.0003400810528546572,
+            -0.004457976669073105,
+            -0.0032783811911940575,
+            0.019231097772717476,
+            0.03058008849620819,
+            0.017256684601306915,
+            -0.01717895083129406,
+            -0.017147857695817947,
+            0.00652955612167716,
+            -0.007248584646731615,
+            0.0002594332618173212,
+            0.010571662336587906,
+            -0.013859760016202927,
+            -0.013486642390489578,
+            -0.0009648585109971464,
+            -0.033238548785448074,
+            0.01120907161384821,
+            0.008962593041360378,
+            -0.020676927641034126,
+            0.010385103523731232,
+            -0.011566642671823502,
+            0.0012272067833691835,
+            0.01416291855275631,
+            0.0003378948022145778,
+            0.008177491836249828,
+            -0.0328032448887825,
+            -0.01850040815770626,
+            0.009872066788375378,
+            -0.0005674495478160679,
+            -0.04645312950015068,
+            0.0049982196651399136,
+            0.008612795732915401,
+            -0.010711581446230412,
+            -0.028543489053845406,
+            -0.01823611743748188,
+            0.04424551501870155,
+            0.012965832836925983,
+            -0.008379597216844559,
+            0.03224356845021248,
+            0.007948179729282856,
+            -0.02806154452264309,
+            -0.030486809089779854,
+            -0.006941540166735649,
+            0.04029668867588043,
+            -0.014147371985018253,
+            -0.026398062705993652,
+            0.04023450240492821,
+            -0.018018465489149094,
+            -0.03126413747668266,
+            0.005355790723115206,
+            0.002260081470012665,
+            -0.019060084596276283,
+            0.00021704458049498498,
+            0.02125214971601963,
+            -0.003983806353062391,
+            -0.03448227420449257,
+            -0.0317305326461792,
+            0.0005280972691252828,
+            -0.02715984359383583,
+            -0.029180897399783134,
+            0.009234657511115074,
+            -0.0008647775393910706,
+            0.028403569012880325,
+            0.02832583710551262,
+            0.018935712054371834,
+            0.003678705310449004,
+            0.025931665673851967,
+            -0.023646321147680283,
+            0.014085185714066029,
+            -0.033145271241664886,
+            -0.01508016511797905,
+            -0.042286649346351624,
+            -0.03162170574069023,
+            -0.015896359458565712,
+            -0.024952232837677002,
+            -0.016075145453214645,
+            -0.007034819573163986,
+            0.012359516695141792,
+            -0.02230931632220745,
+            0.0047689080238342285,
+            -0.011621055193245411,
+            0.03410915657877922,
+            -0.0050759525038301945,
+            -0.05444405972957611,
+            -0.004838867578655481,
+            0.012219597585499287,
+            -0.037467215210199356,
+            -0.04384130612015724,
+            -0.0035912557505071163,
+            -0.023195471614599228,
+            -0.0015235630562528968,
+            -0.010167451575398445,
+            -0.0197907742112875,
+            0.028388023376464844,
+            -0.018407128751277924,
+            -0.009211338125169277,
+            -0.03382932022213936,
+            -0.0013010528637096286,
+            0.004438543692231178,
+            -0.014707047492265701,
+            0.005783321335911751,
+            -0.06358543783426285,
+            -0.01641716994345188,
+            0.02467239461839199,
+            -0.006708341650664806,
+            -0.08401361852884293,
+            0.01442720927298069,
+            -0.03373603895306587,
+            0.012250690720975399,
+            0.011838707141578197,
+            0.04051434248685837,
+            -0.02058364823460579,
+            0.007944293320178986,
+            -0.02860567346215248,
+            -0.017381057143211365,
+            -0.035477254539728165,
+            -0.024066077545285225,
+            -0.03805798292160034,
+            -0.013346723280847073,
+            -0.04738591983914375,
+            -0.017738627269864082,
+            -0.016370529308915138,
+            -0.0037136850878596306,
+            -0.02045927569270134,
+            0.035228509455919266,
+            -0.03927061706781387,
+            0.006078705657273531,
+            0.023817332461476326,
+            -0.012989153154194355,
+            -0.010144132189452648,
+            -0.0031384623143821955,
+            -0.009910933673381805,
+            -0.010493929497897625,
+            0.0010017815511673689,
+            0.0013107694685459137,
+            0.030238064005970955,
+            0.027750613167881966,
+            0.006280811037868261,
+            -0.02384842559695244,
+            0.007827693596482277,
+            0.004306397866457701,
+            0.019355470314621925,
+            -0.03905296325683594,
+            -0.012227371335029602,
+            -0.046515315771102905,
+            -0.014069639146327972,
+            0.05155239999294281,
+            -0.03609911724925041,
+            -0.004306397866457701,
+            0.014784780330955982,
+            -0.017365509644150734,
+            -0.002349474234506488,
+            0.005064292810857296,
+            -0.02795271947979927,
+            0.029693933203816414,
+            0.036658793687820435,
+            0.018578141927719116,
+            0.006615062244236469,
+            -0.002516599837690592,
+            0.023630773648619652,
+            -0.03134186938405037,
+            -0.042566485702991486,
+            0.03581928089261055,
+            -0.02715984359383583,
+            0.01494801975786686,
+            -0.005592875648289919,
+            -0.0026545755099505186,
+            -0.011916439980268478,
+            0.06234171614050865,
+            0.025387536734342575,
+            0.012382837012410164,
+            -0.05024648830294609,
+            0.01851595565676689,
+            -0.0023883406538516283,
+            -0.045333776623010635,
+            -0.024750126525759697,
+            -0.022107211872935295,
+            -0.0038477741181850433,
+            -0.006957086734473705,
+            -0.013774254359304905,
+            0.004971013404428959,
+            0.018034012988209724,
+            -0.012872553430497646,
+            -0.02493668533861637,
+            0.02649134211242199,
+            -0.023133285343647003,
+            -0.012631582096219063,
+            0.01120907161384821,
+            -0.004011013079434633,
+            0.00945230945944786,
+            -0.014567128382623196,
+            0.005977652966976166,
+            0.0023961139377206564,
+            -0.0065451026894152164,
+            -0.021407615393400192,
+            -0.010525022633373737,
+            -0.016759194433689117,
+            0.012188504450023174,
+            0.02845020964741707,
+            -0.003931337036192417,
+            0.03622348979115486,
+            0.002846964169293642,
+            0.0024816200602799654,
+            -0.029585108160972595,
+            0.010043079033493996,
+            0.0501532107591629,
+            0.0012388667091727257,
+            0.05012211576104164,
+            -0.003983806353062391,
+            -0.011038059368729591,
+            0.03501085937023163,
+            -0.009530042298138142,
+            0.012942513450980186,
+            0.011170204728841782,
+            0.007493442855775356,
+            0.00652955612167716,
+            -0.010999192483723164,
+            -0.0027089884970337152,
+            -0.0038808106910437346,
+            -0.0025010532699525356,
+            0.035943653434515,
+            -0.022122757509350777,
+            -0.013035791926085949,
+            0.03603693097829819,
+            -0.005118705797940493,
+            -0.016075145453214645,
+            -0.04909604415297508,
+            -0.021967291831970215,
+            -0.0097710145637393,
+            -0.0015206481330096722,
+            0.019573122262954712,
+            -0.02543417550623417,
+            0.020568102598190308,
+            -0.05755337327718735,
+            0.022340409457683563,
+            -0.06283920258283615,
+            0.013533282093703747,
+            0.028667859733104706,
+            0.002365020802244544,
+            0.02294672653079033,
+            -0.018733607605099678,
+            0.02911871112883091,
+            -0.045333776623010635,
+            -0.0071747382171452045,
+            0.016168424859642982,
+            0.03603693097829819,
+            -0.008022026158869267,
+            -0.0027361949905753136,
+            0.011170204728841782,
+            -0.008744941093027592,
+            -0.0325544998049736,
+            -0.006650041788816452,
+            0.02310219220817089,
+            -0.021065590903162956,
+            0.012048585340380669,
+            -0.02244923636317253,
+            -0.0104317432269454,
+            -0.005775548052042723,
+            0.027579601854085922,
+            0.03976810723543167,
+            -0.048007782548666,
+            -0.005472389981150627,
+            7.518220081692562e-05,
+            0.022620247676968575,
+            -0.0025690693873912096,
+            0.01475368719547987,
+            -0.012312876991927624,
+            0.037063002586364746,
+            -0.019215550273656845,
+            0.010610528290271759,
+            -0.009421216323971748,
+            -0.012818140909075737,
+            -0.018049558624625206,
+            -0.011154658161103725,
+            -0.021936198696494102,
+            0.01679028756916523,
+            -0.013828666880726814,
+            0.002605992369353771,
+            -0.032088104635477066,
+            0.03401587903499603,
+            -0.01941765658557415,
+            0.03264778107404709,
+            -0.014279517345130444,
+            0.005437409970909357,
+            -0.024874499067664146,
+            0.01572534814476967,
+            0.005666722077876329,
+            0.01612178422510624,
+            -0.021454256027936935,
+            0.031699441373348236,
+            0.030782192945480347,
+            -0.049904465675354004,
+            -0.02241814322769642,
+            0.0027284217067062855,
+            -0.012631582096219063,
+            0.010268504731357098,
+            -0.01121684443205595,
+            -0.02021053060889244,
+            0.0354461632668972,
+            -0.008348504081368446,
+            0.0052197580225765705,
+            -0.011100245639681816,
+            0.00485052727162838,
+            -0.03028470277786255,
+            -0.006988179869949818,
+            -0.006863807328045368,
+            0.036503326147794724,
+            -0.014380569569766521,
+            0.005235304590314627,
+            -0.0035407296381890774,
+            0.004601782187819481,
+            0.05105490982532501,
+            0.003196761943399906,
+            -0.00014137655671220273,
+            0.00534413056448102,
+            0.004959353245794773,
+            -0.0008458301308564842,
+            0.00011082998389611021,
+            -0.015826400369405746,
+            -0.007893767207860947,
+            0.00015473687381017953,
+            -0.032336849719285965,
+            -0.011263484135270119,
+            0.012219597585499287,
+            0.007610042113810778,
+            0.0029421867802739143,
+            -0.018857980147004128,
+            0.008698301389813423,
+            0.03504195064306259,
+            0.019479842856526375,
+            -0.019510935992002487,
+            0.0027770046144723892,
+            0.0032997578382492065,
+            -0.01207190565764904,
+            -0.028667859733104706,
+            -0.0040343329310417175,
+            0.0026526320725679398,
+            0.0031423489563167095,
+            -0.010734900832176208,
+            0.032212477177381516,
+            -0.027439681813120842,
+            3.349190956214443e-05,
+            -0.027719520032405853,
+            -0.004209231585264206,
+            0.01599741168320179,
+            -0.0036767618730664253,
+            0.03401587903499603,
+            -0.000832226884085685,
+            0.011605508625507355,
+            0.009755467996001244,
+            0.013043565675616264,
+            -0.04076308757066727,
+            0.00925020407885313,
+            0.011240164749324322,
+            -0.01680583320558071,
+            -0.028263650834560394,
+            0.006720001343637705,
+            0.007847127504646778,
+            0.01808065176010132,
+            -0.010159678757190704,
+            -0.018453769385814667,
+            -0.025543002411723137,
+            0.018034012988209724,
+            0.02179628051817417,
+            -0.0419757179915905,
+            0.007703321520239115,
+            0.015453282743692398,
+            0.03420243784785271,
+            -0.0009502836037427187,
+            -0.0036961950827389956,
+            -0.034326810389757156,
+            0.007015386130660772,
+            0.002021053107455373,
+            -0.013167938217520714,
+            0.050712887197732925,
+            -0.010641621425747871,
+            -0.019635308533906937,
+            0.016728101298213005,
+            -0.010727128013968468,
+            -0.018811339512467384,
+            0.02241814322769642,
+            0.014217331074178219,
+            -0.026429155841469765,
+            -0.0005212956457398832,
+            0.016588181257247925,
+            0.029211990535259247,
+            -0.017007939517498016,
+            0.008542835712432861,
+            0.006024292670190334,
+            -0.0019957898184657097,
+            0.015538789331912994,
+            -0.015328910201787949,
+            0.013680974952876568,
+            0.05121037736535072,
+            0.015103485435247421,
+            0.005631742067635059,
+            0.004601782187819481,
+            -0.006156438495963812,
+            0.012079678475856781,
+            -0.004271417856216431,
+            -0.004286964423954487,
+            -0.015119032002985477,
+            0.00011161946167703718,
+            0.05074397847056389,
+            -0.0008594333776272833,
+            0.028419116511940956,
+            -0.0018976522842422128,
+            0.014396116137504578,
+            0.021423162892460823,
+            -0.006914333440363407,
+            -0.02989603951573372,
+            -0.008379597216844559,
+            -0.008511742576956749,
+            -0.0237551461905241,
+            -0.024687940254807472,
+            -0.028698952868580818,
+            -0.026864459738135338,
+            -0.0002560324501246214,
+            -0.0013982189120724797,
+            0.009980892762541771,
+            0.004372470546513796,
+            0.0037972477730363607,
+            0.0008453443297185004,
+            0.004049879498779774,
+            0.023288751021027565,
+            0.013556602410972118,
+            0.020272716879844666,
+            -0.0042558712884783745,
+            0.007326317485421896,
+            -0.0076800016686320305,
+            0.00330947432667017,
+            -0.0035562762059271336,
+            -0.0010795143898576498,
+            -0.0020579760894179344,
+            0.006024292670190334,
+            0.03659660741686821,
+            0.02977166697382927,
+            0.002528259763494134,
+            0.001742186606861651,
+            0.012507209554314613,
+            -0.010120811872184277,
+            0.011815386824309826,
+            -0.0023786239326000214,
+            -0.011076925322413445,
+            0.011325670406222343,
+            0.02073911391198635,
+            -0.014652634970843792,
+            0.015290044248104095,
+            -0.014279517345130444,
+            -0.05519029498100281,
+            0.006082592532038689,
+            -0.017272230237722397,
+            0.022760167717933655,
+            -0.02555854804813862,
+            -0.003968259785324335,
+            -0.017225591465830803,
+            0.00731077091768384,
+            0.014675954356789589,
+            -0.024174904450774193,
+            -0.004325830843299627,
+            0.04082527384161949,
+            0.0013739274581894279,
+            -0.02518543042242527,
+            -0.005355790723115206,
+            -0.003744778223335743,
+            -0.027097657322883606,
+            -0.012538302689790726,
+            0.033114176243543625,
+            -0.009708828292787075,
+            -0.029600653797388077,
+            0.051490213721990585,
+            -0.01462931465357542,
+            0.0022134417667984962,
+            -0.014932473190128803,
+            -0.002841134322807193,
+            -0.03724956139922142,
+            -0.006308017764240503,
+            -0.02860567346215248,
+            0.035601627081632614,
+            -0.024719033390283585,
+            0.026942191645503044,
+            0.023304296657443047,
+            -0.003422186942771077,
+            -0.02139206975698471,
+            0.026833366602659225,
+            -0.030098143965005875,
+            0.0044618635438382626,
+            0.0270199254155159,
+            -0.0005859110387973487,
+            -0.0018228343687951565,
+            0.002858624095097184,
+            -0.021671907976269722,
+            -0.005351903848350048,
+            -0.0017965995939448476,
+            0.00853506289422512,
+            0.027843892574310303,
+            0.009273524396121502,
+            -0.007042592857033014,
+            -0.005923239979892969,
+            0.0012971663381904364,
+            0.012685994617640972,
+            -0.019635308533906937,
+            0.028030451387166977,
+            -0.011605508625507355,
+            0.011123565025627613,
+            -0.008014252409338951,
+            -0.03031579591333866,
+            -0.02345976233482361,
+            0.014963566325604916,
+            0.03696972504258156,
+            0.02308664470911026,
+            0.014279517345130444,
+            0.009195791557431221,
+            0.02174963988363743,
+            -0.00718639837577939,
+            -0.038617659360170364,
+            -0.0028994339518249035,
+            -0.008115305565297604,
+            -0.010361784137785435,
+            0.01812729239463806,
+            0.026289235800504684,
+            0.0024058304261416197,
+            -0.011970852501690388,
+            0.015383323654532433,
+            0.007726641371846199,
+            -0.01901344582438469,
+            0.01943320222198963,
+            -0.010136358439922333,
+            0.021485349163413048,
+            -0.0025651827454566956,
+            0.015709800645709038,
+            0.016852473840117455,
+            0.00045692318235524,
+            0.007194171659648418,
+            -0.023055551573634148,
+            0.029009884223341942,
+            -0.011395630426704884,
+            -0.05736681446433067,
+            0.0036942518781870604,
+            0.0003740891406778246,
+            -0.022091664373874664,
+            -0.013377816416323185,
+            0.015103485435247421,
+            -0.028932152315974236,
+            0.03724956139922142,
+            0.0032025917898863554,
+            -0.005064292810857296,
+            0.023630773648619652,
+            -0.006918220315128565,
+            -0.03488648682832718,
+            0.007959839887917042,
+            0.029072070494294167,
+            -0.01769198849797249,
+            -0.00037797578261233866,
+            0.043623652309179306,
+            0.005278057884424925,
+            -0.03370494768023491,
+            0.009032553061842918,
+            0.03152842819690704,
+            -0.009623321704566479,
+            -0.0007418625173158944,
+            0.012437249533832073,
+            -0.03302089869976044,
+            -0.01784745417535305,
+            0.019308829680085182,
+            0.005009879823774099,
+            0.018951259553432465,
+            -0.028512395918369293,
+            0.02241814322769642,
+            0.008620568551123142,
+            -0.01239838358014822,
+            -0.007524535991251469,
+            0.026569074019789696,
+            0.004496843088418245,
+            0.006377977319061756,
+            -0.015095711685717106,
+            0.013572148978710175,
+            0.049406975507736206,
+            -0.005655061919242144,
+            -0.018671421334147453,
+            0.05599871650338173,
+            -0.02047482319176197,
+            -0.013719840906560421,
+            0.024299276992678642,
+            -0.01822056993842125,
+            -0.007582835853099823,
+            0.03948826715350151,
+            -0.016292797401547432,
+            -0.03386041149497032,
+            0.009669961407780647,
+            -0.031295228749513626,
+            -0.00787822064012289,
+            -0.002689555287361145,
+            0.025014419108629227,
+            -0.013953039422631264,
+            -0.006148665212094784,
+            0.019122270867228508,
+            -0.027595147490501404,
+            -0.00879158079624176,
+            -0.017241137102246284,
+            0.013028019107878208,
+            -0.011325670406222343,
+            -0.02717539109289646,
+            -0.00024036443210206926,
+            -0.0009653443703427911,
+            0.007633361965417862,
+            -0.0025457495357841253,
+            -0.022635795176029205,
+            0.005946559831500053,
+            0.0035873691085726023,
+            -0.011955305933952332,
+            0.06072487309575081,
+            -0.00031287455931305885,
+            0.029196443036198616,
+            -0.02162526734173298,
+            -0.023428669199347496,
+            0.013284537009894848,
+            -0.011123565025627613,
+            0.022387050092220306,
+            0.0038846973329782486,
+            0.019713040441274643,
+            0.0015167614910751581,
+            0.03687644377350807,
+            0.010081945918500423,
+            -0.008962593041360378,
+            -0.032212477177381516,
+            0.01075822114944458,
+            -0.04051434248685837,
+            -0.011395630426704884,
+            0.008830447681248188,
+            -0.02454802207648754,
+            -0.013020245358347893,
+            -0.003565992694348097,
+            0.0206147413700819,
+            0.06498462706804276,
+            0.02373960055410862,
+            -0.009716601110994816,
+            -0.0272686704993248,
+            -0.004566802643239498,
+            -0.011706560850143433,
+            -0.006463483441621065,
+            -0.023133285343647003,
+            0.01192421279847622,
+            -0.045209404081106186,
+            -0.02557409554719925,
+            -0.021671907976269722,
+            -0.0018811339978128672,
+            -0.029320815578103065,
+            -0.016230611130595207,
+            -0.004403563681989908,
+            -0.012787047773599625,
+            0.007711094804108143,
+            -0.01812729239463806,
+            -0.04116729646921158,
+            -0.0016148991417139769,
+            -0.015678707510232925,
+            0.01651044934988022,
+            -0.02756405435502529,
+            -0.021407615393400192,
+            0.012631582096219063,
+            -0.027051018550992012,
+            0.02112777717411518,
+            -0.01108469907194376,
+            0.012763727456331253,
+            -0.010626074858009815,
+            0.031699441373348236,
+            0.0010979759972542524,
+            0.012499435804784298,
+            -0.006276924628764391,
+            0.005829961039125919,
+            -0.032461222261190414,
+            -0.02386397309601307,
+            -0.033767133951187134,
+            0.017878547310829163,
+            -0.014022999443113804,
+            -0.00747400987893343,
+            -0.022651340812444687,
+            0.0011533605866134167,
+            0.009988666512072086,
+            0.010921459645032883,
+            0.0028780573047697544,
+            0.0016090691788122058,
+            0.01081263367086649,
+            -0.008115305565297604,
+            -0.033642761409282684,
+            -0.021718546748161316,
+            -0.023786239326000214,
+            0.017412150278687477,
+            -0.041789159178733826,
+            -0.0016158708604052663,
+            -0.011473363265395164,
+            0.0005684212082996964,
+            -0.02703547105193138,
+            -0.00731465732678771,
+            -0.001748016569763422,
+            0.0021668020635843277,
+            -0.0013583808904513717,
+            0.015919679775834084,
+            0.019852960482239723,
+            0.01207190565764904,
+            -0.02479676716029644,
+            0.05217426270246506,
+            -0.030098143965005875,
+            -0.011496682651340961,
+            0.019324377179145813,
+            0.010369556955993176,
+            0.010703807696700096,
+            -0.021314335986971855,
+            -0.023397576063871384,
+            -0.016619274392724037,
+            0.015119032002985477,
+            0.004419110249727964,
+            -0.008200811222195625,
+            -0.013354497030377388,
+            0.006778301205486059,
+            0.04225555434823036,
+            -0.024843405932188034,
+            -0.00012303645780775696,
+            -0.019775226712226868,
+            0.01417069137096405,
+            -0.00422866502776742,
+            -0.04004794359207153,
+            0.0027964378241449594,
+            -0.02453247457742691,
+            0.002339757513254881,
+            0.01232065074145794,
+            -0.015297817066311836,
+            0.0104317432269454,
+            -0.023630773648619652,
+            0.046484220772981644,
+            0.02059919573366642,
+            -0.012670448049902916,
+            0.03058008849620819,
+            -0.014784780330955982,
+            -0.01075822114944458,
+            0.024454742670059204,
+            0.0051964386366307735,
+            0.031404055655002594,
+            0.0011339273769408464,
+            -0.02414381131529808,
+            0.01916891150176525,
+            -0.012584942393004894,
+            0.03227466344833374,
+            -0.006762754637748003,
+            -0.017925186082720757,
+            -0.006241944618523121,
+            -0.004442430101335049,
+            0.013206804171204567,
+            -0.014605995267629623,
+            -0.009397896938025951,
+            -0.03135741502046585,
+            -0.007120325695723295,
+            0.026833366602659225,
+            0.005351903848350048,
+            -0.0482565276324749,
+            0.0029091504402458668,
+            0.001017328118905425,
+            -0.009825427085161209,
+            0.02229377068579197,
+            -0.006335224024951458,
+            -0.009094739332795143,
+            -0.0033017010428011417,
+            0.035881467163562775,
+            0.05002883821725845,
+            0.012110771611332893,
+            0.004796114284545183,
+            0.012717087753117085,
+            0.036658793687820435,
+            -0.04306397587060928,
+            0.0127248615026474,
+            -0.01494801975786686,
+            -0.03187045082449913,
+            0.02008615806698799,
+            0.0035348995588719845,
+            -0.010284051299095154,
+            0.0337982252240181,
+            -0.015297817066311836,
+            -0.02153198793530464,
+            6.425102765206248e-05,
+            -0.013401136733591557,
+            -0.013556602410972118,
+            -0.007835467346012592,
+            0.0023261543828994036,
+            -0.007302997633814812,
+            -0.007893767207860947,
+            0.0006388665642589331,
+            0.027377495542168617,
+            -0.015017978847026825,
+            0.027921626344323158,
+            0.01528227049857378,
+            0.013432229869067669,
+            -0.021345429122447968,
+            0.012017492204904556,
+            -0.011776520870625973,
+            -2.3729762688162737e-05,
+            0.016230611130595207,
+            -0.0017237251158803701,
+            -0.005969879683107138,
+            0.025822840631008148,
+            -0.0008924698340706527,
+            -0.01952648162841797,
+            -0.006867693737149239,
+            0.008309637196362019,
+            0.0027167617809027433,
+            -0.0014555469388142228,
+            -0.009405669756233692,
+            0.00044502035598270595,
+            0.019852960482239723,
+            0.03765377402305603,
+            -0.034046970307826996,
+            0.004263644572347403,
+            0.003103482536971569,
+            0.0013544942485168576,
+            0.011675468645989895,
+            -0.00826299749314785,
+            -0.006311904173344374,
+            0.019775226712226868,
+            -0.006673361640423536,
+            -0.0016508505214005709,
+            0.006342997308820486,
+            -0.058579444885253906,
+            -0.0149557925760746,
+            0.043219443410634995,
+            -0.01955757476389408,
+            0.0007306884508579969,
+            0.03183935955166817,
+            -0.005037086084485054,
+            0.013066885061562061,
+            -0.029616201296448708,
+            -0.0018306076526641846,
+            -0.0016421056352555752,
+            0.02453247457742691,
+            -0.013066885061562061,
+            -0.002024939749389887,
+            0.017940733581781387,
+            -0.0015799193643033504,
+            -0.01384421344846487,
+            0.024858953431248665,
+            0.01280259434133768,
+            -0.0065528759732842445,
+            2.377530836383812e-05,
+            0.029569560661911964,
+            0.004279191140085459,
+            0.017132312059402466,
+            -0.023273203521966934,
+            0.0104317432269454,
+            -0.0028158710338175297,
+            -0.011271257884800434,
+            0.024330370128154755,
+            -0.005223644897341728,
+            0.011862026527523994,
+            0.025931665673851967,
+            0.031823813915252686,
+            -0.0011795953614637256,
+            -0.004792227875441313,
+            0.0010289880447089672,
+            0.00016068830154836178,
+            0.023646321147680283,
+            0.020381543785333633,
+            -0.011115792207419872,
+            -0.013883080333471298,
+            -0.019992878660559654,
+            0.005655061919242144,
+            -0.0072524710558354855,
+            0.0032744945492595434,
+            0.034948673099279404,
+            0.027346402406692505,
+            0.000937652017455548,
+            0.028045998886227608,
+            -0.016354983672499657,
+            0.017210043966770172,
+            0.011838707141578197,
+            -0.019339922815561295,
+            -0.014357250183820724,
+            0.00619919179007411,
+            -0.016852473840117455,
+            0.011154658161103725,
+            -0.00277117476798594,
+            -0.013276764191687107,
+            -0.006261378061026335,
+            0.08699855953454971,
+            -0.01742769591510296,
+            0.021687453612685204,
+            -0.056869324296712875,
+            -0.027641788125038147,
+            -0.011574415490031242,
+            -0.02359968051314354,
+            0.019837412983179092,
+            0.04312616214156151,
+            0.014030772261321545,
+            0.01943320222198963,
+            0.011310123838484287,
+            -0.006479030009359121,
+            0.03132632374763489,
+            0.0038438874762505293,
+            0.012958060018718243,
+            0.001303967903368175,
+            -0.010501702316105366,
+            -0.013921946287155151,
+            -0.012281783856451511,
+            -0.0015459113055840135,
+            0.012763727456331253,
+            0.005697815213352442,
+            0.010113039053976536,
+            -0.012141864746809006,
+            -0.013657654635608196,
+            -0.006743321195244789,
+            0.021438708528876305,
+            0.00853506289422512,
+            -0.014225104823708534,
+            0.027253123000264168,
+            0.012584942393004894,
+            -0.00026744944625534117,
+            0.010594981722533703,
+            -0.0025088265538215637,
+            0.053697824478149414,
+            -0.02740858867764473,
+            0.007885993458330631,
+            0.031932637095451355,
+            -0.009281297214329243,
+            -0.0027886645402759314,
+            -0.010952552780508995,
+            0.00701927300542593,
+            0.023304296657443047,
+            -0.010672714561223984,
+            0.006747208070009947,
+            0.028045998886227608,
+            -0.005079839378595352,
+            -0.009079192765057087,
+            -0.02139206975698471,
+            -0.008721621707081795,
+            0.01993069238960743,
+            -0.0048194341361522675,
+            0.007427370175719261,
+            -0.004842753987759352,
+            0.015655387192964554,
+            0.00845733005553484,
+            0.0008463159902021289,
+            0.0013272877549752593,
+            -0.0023747372906655073,
+            0.004321944434195757,
+            0.0003070445964112878,
+            0.01666591502726078,
+            0.03625458478927612,
+            0.011823160573840141,
+            0.00236113416031003,
+            0.011310123838484287,
+            -0.0028119843918830156,
+            -0.01548437587916851,
+            0.011457816697657108,
+            -0.00508761266246438,
+            0.002357247518375516,
+            0.009483402594923973,
+            -0.013735387474298477,
+            -0.0060670459643006325,
+            0.016603728756308556,
+            -0.008822673931717873,
+            -0.0034377335105091333,
+            0.008239678107202053,
+            -0.02321101725101471,
+            0.008014252409338951,
+            -0.0010435630101710558,
+            -0.01482364721596241,
+            0.00891595333814621,
+            -0.026973284780979156,
+            -0.0013496358878910542,
+            -0.0012806480517610908,
+            -0.015694255009293556,
+            0.014015225693583488,
+            0.0013127127895131707,
+            0.0011659921146929264,
+            0.0010503645753487945,
+            -0.02073911391198635,
+            0.00040761142736300826,
+            -0.01186980027705431,
+            -0.004100405611097813,
+            -0.0013826723443344235,
+            -0.000706882739905268,
+            -0.05671386048197746,
+            -0.010548342019319534,
+            -0.03389150649309158,
+            0.003078219247981906,
+            0.029693933203816414,
+            0.0003570851113181561,
+            -0.01113911159336567,
+            -0.011722107417881489,
+            0.025154337286949158,
+            -0.004388017114251852,
+            -0.004481296520680189,
+            -0.00277117476798594,
+            -0.008900406770408154,
+            -0.004908827133476734,
+            -0.0009944941848516464,
+            -0.01851595565676689,
+            0.03270996734499931,
+            0.011030285619199276,
+            0.0045978957787156105,
+            -0.03289652615785599,
+            -0.0015585428336635232,
+            -0.0015196764143183827,
+            -0.02464130148291588,
+            -0.016370529308915138,
+            0.023506401106715202,
+            0.011193525046110153,
+            0.0010474496521055698,
+            0.0028255877550691366,
+            0.02345976233482361,
+            -0.018313849344849586,
+            0.021703001111745834,
+            0.01055611576884985,
+            -0.0010940893553197384,
+            3.4888671507360414e-05,
+            0.00423255143687129,
+            -0.014808100648224354,
+            0.014862513169646263,
+            -0.00866720825433731,
+            0.029802760109305382,
+            -0.01756761595606804,
+            0.008970366790890694,
+            -0.009965346194803715,
+            0.019060084596276283,
+            0.03911514952778816,
+            0.030595634132623672,
+            0.026724539697170258,
+            0.012670448049902916,
+            -0.03647223487496376,
+            -0.01009749248623848,
+            -0.002440810203552246,
+            0.005538462661206722,
+            0.009537816047668457,
+            0.004842753987759352,
+            -0.015212311409413815,
+            0.01851595565676689,
+            -0.0005421863752417266,
+            0.006284697912633419,
+            -0.021594174206256866,
+            0.008239678107202053,
+            -0.002110445871949196,
+            0.025418629869818687,
+            0.025745106860995293,
+            0.011963079683482647,
+            -0.0018160328036174178,
+            -0.013035791926085949,
+            -0.007532309275120497,
+            0.02047482319176197,
+            0.013611014932394028,
+            0.022604702040553093,
+            -0.023319844156503677,
+            0.01344000268727541,
+            0.006047612521797419,
+            0.0022659115493297577,
+            0.01940210908651352,
+            0.02227822318673134,
+            0.021454256027936935,
+            0.0013272877549752593,
+            -0.008869313634932041,
+            0.01941765658557415,
+            -0.027346402406692505,
+            0.0049399202689528465,
+            0.0017655064584687352,
+            0.013696521520614624,
+            -0.0364411398768425,
+            0.0016217008233070374,
+            -0.0026273690164089203,
+            -0.0013807290233671665,
+            -0.0113412169739604,
+            -0.029569560661911964,
+            -0.018173931166529655,
+            -0.041509322822093964,
+            0.02204502560198307,
+            -0.004321944434195757,
+            0.031932637095451355,
+            0.007104779127985239,
+            -0.0095222694799304,
+            0.0037972477730363607,
+            0.012219597585499287,
+            0.013603242114186287,
+            0.004710608161985874,
+            0.007862674072384834,
+            -0.023413121700286865,
+            0.00718639837577939,
+            0.029289722442626953,
+            -0.00826299749314785,
+            0.01744324341416359,
+            0.005965993274003267,
+            -0.019852960482239723,
+            0.0041975718922913074,
+            0.011589962057769299,
+            0.007990933023393154,
+            -0.028559034690260887,
+            0.02439255639910698,
+            -0.01515012513846159,
+            -0.009988666512072086,
+            0.009825427085161209,
+            -0.012841460295021534,
+            0.02153198793530464,
+            -0.019852960482239723,
+            -0.01666591502726078,
+            0.00788210704922676,
+            0.003323077689856291,
+            -0.013976359739899635,
+            0.0032064784318208694,
+            0.023910611867904663,
+            -0.017769720405340195,
+            0.0004063968372065574,
+            -0.0045123896561563015,
+            0.02139206975698471,
+            -0.024890046566724777,
+            0.02151644229888916,
+            -0.010439516045153141,
+            -0.019635308533906937,
+            0.016106238588690758,
+            -0.00715141836553812,
+            0.013618788681924343,
+            0.011108018457889557,
+            0.0034727132879197598,
+            0.001421538763679564,
+            0.01245279610157013,
+            0.01572534814476967,
+            0.009615548886358738,
+            -0.011123565025627613,
+            -0.0012524699559435248,
+            -0.005752228200435638,
+            0.01416291855275631,
+            -0.02638251520693302,
+            -0.0002140081487596035,
+            0.021967291831970215,
+            -0.01836048997938633,
+            -0.015562108717858791,
+            0.006832714192569256,
+            0.00468728831037879,
+            0.009382350370287895,
+            0.003812794340774417,
+            -0.03501085937023163,
+            -0.00603595282882452,
+            -0.013167938217520714,
+            -0.003431903664022684,
+            0.03277215361595154,
+            -0.0025748994667083025,
+            -0.014862513169646263,
+            -0.024952232837677002,
+            -1.4673586520075332e-05,
+            0.010120811872184277,
+            0.01968194730579853,
+            -0.002547692973166704,
+            -0.016106238588690758,
+            0.008581702597439289,
+            0.013028019107878208,
+            0.008294090628623962,
+            0.009879840537905693,
+            -0.009483402594923973,
+            0.00682105403393507,
+            0.02689555287361145,
+            0.03647223487496376,
+            -0.009965346194803715,
+            -0.015468829311430454,
+            0.0007501216023229063,
+            -0.018795793876051903,
+            -0.003995466511696577,
+            -0.013028019107878208,
+            -0.005740568041801453,
+            0.0532936155796051,
+            -0.027377495542168617,
+            -0.004862187430262566,
+            -0.02033490315079689,
+            0.00032696363632567227,
+            0.0016090691788122058,
+            -0.001061052898876369,
+            0.007905426435172558,
+            -0.008698301389813423,
+            -0.03243012726306915,
+            0.007501216139644384,
+            0.009957573376595974,
+            0.000347368506481871,
+            0.03249231353402138,
+            0.0113412169739604,
+            -0.013867533765733242,
+            0.01679028756916523,
+            -0.0021881787106394768,
+            -0.0014001622330397367,
+            -0.01941765658557415,
+            -0.013377816416323185,
+            0.0063740904442965984,
+            0.021827373653650284,
+            -0.0017616198165342212,
+            -0.02059919573366642,
+            -0.00793651957064867,
+            0.0052197580225765705,
+            0.00891595333814621,
+            -0.008550609461963177,
+            0.02345976233482361,
+            0.004380243830382824,
+            -0.018578141927719116,
+            0.020910125225782394,
+            -0.018733607605099678,
+            0.0018704457907006145,
+            -0.010167451575398445,
+            -0.0062963576056063175,
+            0.03790251910686493,
+            0.013579921796917915,
+            -0.020770207047462463,
+            -0.013276764191687107,
+            -0.02715984359383583,
+            0.001835466013289988,
+            0.017241137102246284,
+            0.015686480328440666,
+            0.0036825919523835182,
+            -0.002462186850607395,
+            -0.009017006494104862,
+            -0.01311352476477623,
+            -0.025014419108629227,
+            -0.02516988478600979,
+            0.015818627551198006,
+            -0.0009609718690626323,
+            0.04959353432059288,
+            0.00310153909958899,
+            0.004415223840624094,
+            0.005348017439246178,
+            0.004158705472946167,
+            0.007831580936908722,
+            -0.019635308533906937,
+            -0.007306884042918682,
+            -0.0013107694685459137,
+            -0.021811826154589653,
+            0.004772794432938099,
+            0.012880327180027962,
+            -0.012025265954434872,
+            -0.0007020244374871254,
+            -0.001712065190076828,
+            0.01128680445253849,
+            -0.020521461963653564,
+            0.028014905750751495,
+            0.01874915324151516,
+            0.008294090628623962,
+            -0.0003578138421289623,
+            0.02729976363480091,
+            0.00299854320473969,
+            0.012429476715624332,
+            0.0007656681700609624,
+            0.020288264378905296,
+            0.00422866502776742,
+            0.008970366790890694,
+            -0.015935225412249565,
+            0.008542835712432861,
+            -0.015422189608216286,
+            0.035477254539728165,
+            0.0025496361777186394,
+            -0.0388353131711483,
+            0.02271352708339691,
+            -0.006922106724232435,
+            0.0014477736549451947,
+            -0.010618302039802074,
+            -0.026957739144563675,
+            0.01809619925916195,
+            0.01284923404455185,
+            0.010011985898017883,
+            -0.008441783487796783,
+            0.008193038403987885,
+            -0.02174963988363743,
+            0.005892146844416857,
+            -0.01598186604678631,
+            0.012965832836925983,
+            0.0016595955239608884,
+            0.01396858599036932,
+            -0.003507693065330386,
+            0.023646321147680283,
+            0.0071164388209581375,
+            -0.00330947432667017,
+            0.022184943780303,
+            0.0027109317015856504,
+            0.03491757810115814,
+            0.012934739701449871,
+            0.008022026158869267,
+            0.0211122315376997,
+            0.009537816047668457,
+            0.015290044248104095,
+            -0.013758707791566849,
+            -0.004096519201993942,
+            -0.00017647776985540986,
+            0.015017978847026825,
+            -0.006824940908700228,
+            -0.00743902986869216,
+            -0.02571401372551918,
+            0.009327936917543411,
+            -0.0008166803163476288,
+            0.020288264378905296,
+            -0.0077655077911913395,
+            -0.008706075139343739,
+            -0.015002432279288769,
+            -0.044369887560606,
+            -0.011247937567532063,
+            -0.01599741168320179,
+            0.0025418628938496113,
+            0.0024738467764109373,
+            -0.001715951832011342,
+            -0.020039519295096397,
+            0.02162526734173298,
+            -0.0005528746405616403,
+            -0.023972798138856888,
+            -0.0034377335105091333,
+            0.012903646565973759,
+            -0.01626170426607132,
+            0.012546075507998466,
+            -0.004792227875441313,
+            -0.016572635620832443,
+            0.013976359739899635,
+            -0.0011669638333842158,
+            -0.0036981385201215744,
+            -0.027361949905753136,
+            -0.011589962057769299,
+            -0.01455158181488514,
+            0.003192875301465392,
+            0.0019005672074854374,
+            0.010315144434571266,
+            0.011372310109436512,
+            -0.0179718267172575,
+            0.0017256684368476272,
+            0.015398870222270489,
+            0.008224131539463997,
+            0.004302510991692543,
+            0.002841134322807193,
+            0.012079678475856781,
+            -0.009273524396121502,
+            0.0004821863549295813,
+            -0.011403403244912624,
+            -0.01993069238960743,
+            0.005208098329603672,
+            0.010408422909677029,
+            -0.008643888868391514,
+            -0.03538397699594498,
+            -0.007699435111135244,
+            0.013626561500132084,
+            -0.009794333949685097,
+            -0.0009284212719649076,
+            -0.017163405194878578,
+            -0.0005212956457398832,
+            -0.005122592207044363,
+            0.007959839887917042,
+            -0.03019142337143421,
+            -0.0036223488859832287,
+            -0.007777167949825525,
+            -0.0017781381029635668,
+            -0.008177491836249828,
+            -0.0017470449674874544,
+            0.044121142476797104,
+            0.013696521520614624,
+            0.006941540166735649,
+            -0.006405183579772711,
+            -0.004928260110318661,
+            -0.006273037753999233,
+            -0.0003541701298672706,
+            -0.006720001343637705,
+            0.0004105264088138938,
+            -0.014225104823708534,
+            0.016090691089630127,
+            -0.0058455076068639755,
+            0.01430283673107624,
+            0.011862026527523994,
+            -0.007135872263461351,
+            0.026195956394076347,
+            0.0030374096240848303,
+            0.03370494768023491,
+            -0.012421702966094017,
+            0.0014322270872071385,
+            0.017381057143211365,
+            0.0844489261507988,
+            0.013471095822751522,
+            0.008325183764100075,
+            0.0027964378241449594,
+            -0.017940733581781387,
+            -0.009265750646591187,
+            0.003678705310449004,
+            -0.028279196470975876,
+            0.023024458438158035,
+            0.0020987859461456537,
+            0.00997311994433403,
+            -0.007528422866016626,
+            -0.0010843727504834533,
+            0.006024292670190334,
+            0.009763240814208984,
+            0.011815386824309826,
+            0.002611822448670864,
+            0.01285700686275959,
+            0.040483247488737106,
+            0.021547535434365273,
+            0.0014011339517310262,
+            -0.016479356214404106,
+            -0.0010639678221195936,
+            -0.015173444524407387,
+            0.006739434786140919,
+            0.007827693596482277,
+            -0.008744941093027592,
+            0.023055551573634148,
+            0.0021609722170978785,
+            0.010804860852658749,
+            0.013121298514306545,
+            -0.0049127135425806046,
+            0.0021026725880801678,
+            -0.003991579636931419,
+            0.0007374900160357356,
+            0.007497329730540514,
+            -0.006537329405546188,
+            0.008216357789933681,
+            -0.006082592532038689,
+            -0.013859760016202927,
+            0.006999839562922716,
+            -0.010120811872184277,
+            0.003097652457654476,
+            0.0058105275966227055,
+            -0.012514982372522354,
+            0.012048585340380669,
+            0.012491662986576557,
+            0.006261378061026335,
+            0.0006869637290947139,
+            -0.002522429684177041,
+            -0.005208098329603672,
+            0.006074819248169661,
+            0.011636601760983467,
+            -0.0005669636884704232,
+            -0.031792718917131424,
+            -0.0073651839047670364,
+            0.0075361961498856544,
+            -0.004582349210977554,
+            0.005433523561805487,
+            0.01876470074057579,
+            -0.0022892311681061983,
+            -0.013991906307637691,
+            0.00011848181020468473,
+            -0.015049071982502937,
+            -0.00731077091768384,
+            -0.0023144944570958614,
+            0.015834173187613487,
+            -0.010470609180629253,
+            0.016976846382021904,
+            0.012172957882285118,
+            0.00640129717066884,
+            -0.007159191649407148,
+            0.014388343319296837,
+            -0.023972798138856888,
+            0.0019977332558482885,
+            0.0008827532292343676,
+            -0.01161328237503767,
+            -0.0020502028055489063,
+            0.011768747121095657,
+            0.020023971796035767,
+            0.004609555471688509,
+            0.004329717718064785,
+            -0.0011154658859595656,
+            0.013657654635608196,
+            -0.011714334599673748,
+            -0.007816034369170666,
+            0.006673361640423536,
+            0.0006918220315128565,
+            -0.005713361781090498,
+            -0.024719033390283585,
+            -0.0027808912564069033,
+            0.009055872447788715,
+            0.0190445389598608,
+            -0.0047689080238342285,
+            -0.01943320222198963,
+            0.017474336549639702,
+            0.014201784506440163,
+            -0.00899368617683649,
+            -0.010408422909677029,
+            -0.015810852870345116,
+            0.005721135064959526,
+            0.024485835805535316,
+            -0.0014856683555990458,
+            -0.020505916327238083,
+            -0.011325670406222343,
+            -0.014442755840718746,
+            0.018422676250338554,
+            0.007299110759049654,
+            0.011045832186937332,
+            0.0003648583951871842,
+            -0.003826397703960538,
+            0.00024145754287019372,
+            0.014481622725725174,
+            0.020521461963653564,
+            -0.0021318222861737013,
+            -0.0002613765827845782,
+            0.0023008910939097404,
+            0.00033497982076369226,
+            -0.011372310109436512,
+            0.0018072878010571003,
+            0.0024660734925419092,
+            0.017412150278687477,
+            0.0034299602266401052,
+            0.0016693121287971735,
+            0.0049127135425806046,
+            -0.022184943780303,
+            -0.006168098654597998,
+            -0.01665036752820015,
+            -0.008962593041360378,
+            -0.015220084227621555,
+            -0.015919679775834084,
+            -0.005449070129543543,
+            -0.007800487335771322,
+            0.008091985248029232,
+            0.01823611743748188,
+            -0.01108469907194376,
+            -0.0011990285711362958,
+            0.019510935992002487,
+            -0.0031287455931305885,
+            0.004139272030442953,
+            -0.0028936038725078106,
+            -0.0014487452572211623,
+            -0.01876470074057579,
+            -0.003908017184585333,
+            0.015220084227621555,
+            0.010501702316105366,
+            0.012561622075736523,
+            -0.0066383820958435535,
+            0.01940210908651352,
+            -0.025931665673851967,
+            0.025263164192438126,
+            -0.020770207047462463,
+            -0.0014147371985018253,
+            0.021018952131271362,
+            0.0029655066318809986,
+            -0.004609555471688509,
+            -0.015103485435247421,
+            0.01232065074145794,
+            -0.02097231149673462,
+            -0.005550122819840908,
+            -0.034171342849731445,
+            0.03358057513833046,
+            -0.010696034878492355,
+            0.0009998382302001119,
+            -0.004753361456096172,
+            0.0417269729077816,
+            0.012895873747766018,
+            0.005635628942400217,
+            -0.00011544537119334564,
+            -0.014668181538581848,
+            -0.0025379762519150972,
+            0.01240615639835596,
+            -0.0051886653527617455,
+            0.0040071262046694756,
+            0.01260048896074295,
+            0.0066306088119745255,
+            0.0013049395056441426,
+            0.010626074858009815,
+            0.00833295751363039,
+            0.007955953478813171,
+            -0.01449716929346323,
+            -0.010649395175278187,
+            -0.009234657511115074,
+            0.018065106123685837,
+            0.005449070129543543,
+            0.00866720825433731,
+            -0.003737004939466715,
+            0.008402916602790356,
+            0.015686480328440666,
+            -0.007831580936908722,
+            -0.015771986916661263,
+            -0.010571662336587906,
+            -0.016588181257247925,
+            -0.0330519899725914,
+            -0.003834170987829566,
+            0.011955305933952332,
+            -0.006731661502271891,
+            -0.000800162146333605,
+            0.010696034878492355,
+            -0.003268664702773094,
+            0.014232877641916275,
+            0.028792232275009155,
+            -0.0058105275966227055,
+            -0.0016071258578449488,
+            0.0009478544816374779,
+            0.002372794086113572,
+            0.01599741168320179,
+            0.008193038403987885,
+            0.03504195064306259,
+            -0.015375549905002117,
+            -0.0427219532430172,
+            -0.003435790305957198,
+            0.012965832836925983,
+            -0.01494801975786686,
+            -0.021143324673175812,
+            0.005452956538647413,
+            -0.005934900138527155,
+            -0.0052158716134727,
+            0.028698952868580818,
+            0.0021434822119772434,
+            -0.01469927467405796,
+            0.008737168274819851,
+            0.0453648678958416,
+            -0.0031423489563167095,
+            0.013416683301329613,
+            -0.005585102364420891,
+            0.016992392018437386,
+            -0.032865431159734726,
+            -0.013292310759425163,
+            -0.02347530797123909,
+            0.00932016409933567,
+            0.01232842355966568,
+            -0.0008254252607002854,
+            0.032212477177381516,
+            0.016292797401547432,
+            0.012180731631815434,
+            -0.004201458301395178,
+            0.016292797401547432,
+            -0.032057009637355804,
+            -0.04063871502876282,
+            -0.007097005844116211,
+            -0.012864780612289906,
+            -0.009965346194803715,
+            0.001216518459841609,
+            0.007295224349945784,
+            0.0013253444340080023,
+            0.02768842689692974,
+            0.0032978144008666277,
+            -0.010874819941818714,
+            -2.778340785880573e-05,
+            -0.0068171676248312,
+            -0.002674008719623089,
+            -0.019339922815561295,
+            0.010571662336587906,
+            0.008729394525289536,
+            0.016712553799152374,
+            -0.011574415490031242,
+            -0.012079678475856781,
+            0.0012534415582194924,
+            -0.021189963445067406,
+            -0.019308829680085182,
+            -0.027455229312181473,
+            0.007831580936908722,
+            -0.015709800645709038,
+            0.012872553430497646,
+            0.011247937567532063,
+            0.013743161223828793,
+            0.0018344942945986986,
+            -0.013129071332514286,
+            0.027843892574310303,
+            -0.015841946005821228,
+            0.007217491511255503,
+            0.01344000268727541,
+            0.00871384795755148,
+            0.00820858497172594,
+            0.0021609722170978785,
+            -0.027641788125038147,
+            -0.011691015213727951,
+            0.015663161873817444,
+            -0.010423969477415085,
+            0.005794981028884649,
+            -0.009335710667073727,
+            -0.0017927129520103335,
+            -0.007816034369170666,
+            -0.008224131539463997,
+            -0.003958543296903372,
+            0.011574415490031242,
+            0.01462931465357542,
+            0.0012952230172231793,
+            0.026833366602659225,
+            -0.023008912801742554,
+            0.004003239795565605,
+            -0.005184778477996588,
+            0.0067744143307209015,
+            -0.001097004278562963,
+            0.006685021799057722,
+            0.018142838031053543,
+            -0.007163078524172306,
+            0.011504456400871277,
+            0.00587271386757493,
+            -0.015639841556549072,
+            -0.012172957882285118,
+            -0.02137652225792408,
+            0.002742024837061763,
+            -0.016603728756308556,
+            -0.0005878543597646058,
+            -0.001586721045896411,
+            -0.003968259785324335,
+            0.021936198696494102,
+            -0.0040926323272287846,
+            -0.023444214835762978,
+            0.024066077545285225,
+            0.011605508625507355,
+            0.023553041741251945,
+            -0.0030354661867022514,
+            0.030533447861671448,
+            -0.011706560850143433,
+            0.002701215213164687,
+            -0.027626240625977516,
+            0.03187045082449913,
+            -0.00419368501752615,
+            0.009763240814208984,
+            -0.011434496380388737,
+            -0.01148890983313322,
+            -0.019868506118655205,
+            -0.0043685841374099255,
+            0.008744941093027592,
+            0.012577168643474579,
+            0.0051031592302024364,
+            -0.015803080052137375,
+            0.023521948605775833,
+            0.004858300555497408,
+            -0.01239838358014822,
+            0.0009920649463310838,
+            0.009351257234811783,
+            -0.009032553061842918,
+            -0.006444049999117851,
+            -0.0058183008804917336,
+            0.01848486252129078,
+            -0.006852147169411182,
+            -0.01089036650955677,
+            -0.03267887234687805,
+            -0.01575644128024578,
+            -0.0033580574672669172,
+            0.0021706887055188417,
+            0.01927773654460907,
+            -0.007769394665956497,
+            -0.01795627921819687,
+            0.0028178144712001085,
+            0.010004213079810143,
+            -0.016044052317738533,
+            0.00011028342851204798,
+            -0.0018976522842422128,
+            0.01572534814476967,
+            -0.0047689080238342285,
+            0.018205024302005768,
+            0.0208634864538908,
+            0.036907538771629333,
+            0.0022659115493297577,
+            -0.0029557901434600353,
+            0.010734900832176208,
+            0.010610528290271759,
+            -0.008472876623272896,
+            -0.007384616881608963,
+            -0.007722754962742329,
+            -0.02215385064482689,
+            0.034451182931661606,
+            -0.009219110943377018,
+            -0.006063159089535475,
+            -0.00067287462297827,
+            0.022635795176029205,
+            -0.0034902032930403948,
+            -0.025263164192438126,
+            -0.024765674024820328,
+            -0.011146885342895985,
+            0.04645312950015068,
+            -0.021842919290065765,
+            -0.01153554953634739,
+            -0.026335876435041428,
+            0.01745878905057907,
+            -0.02017943747341633,
+            -0.0035193529911339283,
+            0.0011076926020905375,
+            0.005569555796682835,
+            -0.0033036444801837206,
+            -0.005324697587639093,
+            -0.004213118460029364,
+            0.0054685031063854694,
+            0.01318348478525877,
+            0.016572635620832443,
+            -0.014380569569766521,
+            0.021423162892460823,
+            -0.00932016409933567,
+            0.008325183764100075,
+            -0.016479356214404106,
+            0.007990933023393154,
+            -0.023257657885551453,
+            0.024485835805535316,
+            -0.007812147494405508,
+            -0.024205997586250305,
+            0.012297330424189568,
+            0.012499435804784298,
+            0.012693768367171288,
+            0.00015303646796382964,
+            0.012841460295021534,
+            -0.023677414283156395,
+            -0.0005470446776598692,
+            -0.006929880008101463,
+            -0.00426753144711256,
+            -0.01311352476477623,
+            -0.011317897588014603,
+            0.013657654635608196,
+            -0.0010814577108249068,
+            -0.012118545360863209,
+            0.019386563450098038,
+            -0.0066306088119745255,
+            -0.012996925972402096,
+            -0.02478121966123581,
+            0.006541215814650059,
+            -0.007827693596482277,
+            0.005324697587639093,
+            -0.0007005669758655131,
+            0.01599741168320179,
+            -0.005456843413412571,
+            -0.001588664366863668,
+            0.008814901113510132,
+            0.013704294338822365,
+            0.004566802643239498,
+            -0.012717087753117085,
+            -0.023661866784095764,
+            0.0007758706342428923,
+            0.011504456400871277,
+            0.014730367809534073,
+            -0.006580082233995199,
+            -0.017474336549639702,
+            -0.00017793526058085263,
+            -0.0010843727504834533,
+            -0.00970105454325676,
+            -0.0041975718922913074,
+            -0.00041465595131739974,
+            -0.011131338775157928,
+            -0.003937166649848223,
+            -0.01076599396765232,
+            0.017349964007735252,
+            0.0032025917898863554,
+            0.015546562150120735,
+            -0.0040693124756217,
+            -0.019728587940335274,
+            0.012056359089910984,
+            0.041011832654476166,
+            -0.019713040441274643,
+            0.021967291831970215,
+            -0.008099758997559547,
+            -0.0015993525739759207,
+            -0.014862513169646263,
+            0.015468829311430454,
+            -0.006665588356554508,
+            0.018733607605099678,
+            0.020925672724843025,
+            0.008643888868391514,
+            -0.0021551421377807856,
+            0.016836926341056824,
+            0.021687453612685204,
+            -0.0008210528176277876,
+            0.0062380582094192505,
+            0.019184457138180733,
+            0.008286317810416222,
+            0.005363564006984234,
+            -0.0020016198977828026,
+            -0.030067050829529762,
+            0.006805507466197014,
+            -0.0005766802933067083,
+            -0.004803887568414211,
+            0.021189963445067406,
+            0.02504551224410534,
+            0.010991419665515423,
+            0.0033036444801837206,
+            0.006148665212094784,
+            0.012095225043594837,
+            0.0026409721467643976,
+            0.0031540088821202517,
+            -0.0042558712884783745,
+            -0.0018685024697333574,
+            -0.009856520220637321,
+            0.004003239795565605,
+            0.008177491836249828,
+            -0.004691175185143948,
+            0.017489882186055183,
+            -0.01148890983313322,
+            -0.014504942111670971,
+            -0.010260730981826782,
+            -0.014186237938702106,
+            0.02176518738269806,
+            -0.017878547310829163,
+            -0.000856032595038414,
+            -0.0036301221698522568,
+            -6.595143349841237e-05,
+            0.00037141708889976144,
+            -0.00853506289422512,
+            0.02188955992460251,
+            0.008410690352320671,
+            -0.022651340812444687,
+            0.01154332235455513,
+            -0.004240324720740318,
+            0.005798867903649807,
+            -0.006995953153818846,
+            0.01030737068504095,
+            -0.01598186604678631,
+            0.010571662336587906,
+            -0.008845994248986244,
+            0.02296227216720581,
+            -0.013090205378830433,
+            -0.005425750277936459,
+            0.0041120657697319984,
+            -0.002913037082180381,
+            0.01693020574748516,
+            -0.0037991912104189396,
+            -0.010369556955993176,
+            0.003678705310449004,
+            0.011963079683482647,
+            -0.006350770592689514,
+            0.009506722912192345,
+            0.011185751296579838,
+            0.003565992694348097,
+            -0.006109798792749643,
+            -0.0015595145523548126,
+            0.00619919179007411,
+            -0.02927417680621147,
+            -0.00925020407885313,
+            0.0213609766215086,
+            -0.017862999811768532,
+            -0.004916600417345762,
+            0.017520975321531296,
+            0.015445509925484657,
+            -0.01916891150176525,
+            -0.00558121595531702,
+            0.008744941093027592,
+            -0.0020657493732869625,
+            0.023179924115538597,
+            -0.009351257234811783,
+            -0.01719449833035469,
+            0.01277927402406931,
+            -0.013867533765733242,
+            0.011022512800991535,
+            -0.013261217623949051,
+            0.000477813882753253,
+            -0.015639841556549072,
+            0.007952066138386726,
+            -0.014100732281804085,
+            -0.022216036915779114,
+            0.011675468645989895,
+            -0.008690528571605682,
+            -0.001947206910699606,
+            -0.023972798138856888,
+            0.0299115851521492,
+            0.032057009637355804,
+            0.026133770123124123,
+            -0.02005506493151188,
+            0.0026293122209608555,
+            -0.007695548236370087,
+            -0.018438221886754036,
+            -0.0034688266459852457,
+            -0.01166769489645958,
+            0.01654154248535633,
+            0.01940210908651352,
+            -0.002988826483488083,
+            -0.018142838031053543,
+            -0.016090691089630127,
+            0.004671741742640734,
+            -0.02202947810292244,
+            -0.01113911159336567,
+            -0.014613768085837364,
+            -0.0004333604301791638,
+            -0.02112777717411518,
+            0.007613928988575935,
+            0.002753684762865305,
+            0.008892633952200413,
+            0.0028294743970036507,
+            0.003171498654410243,
+            -0.019884053617715836,
+            -0.007170851808041334,
+            0.011076925322413445,
+            0.007190284784883261,
+            -0.012740408070385456,
+            0.02795271947979927,
+            -0.009265750646591187,
+            -0.0015750611200928688,
+            -0.0020754660945385695,
+            -0.018857980147004128,
+            0.013525509275496006,
+            0.010214091278612614,
+            -0.016090691089630127,
+            -0.0071669649332761765,
+            -0.010548342019319534,
+            0.0025612961035221815,
+            0.010797087103128433,
+            -0.001009554835036397,
+            -0.006129232235252857,
+            -0.010369556955993176,
+            0.008418463170528412,
+            -0.011154658161103725,
+            -0.00800647959113121,
+            -0.012942513450980186,
+            0.002843077527359128,
+            0.017520975321531296,
+            0.004154818598181009,
+            0.0020793527364730835,
+            -0.019479842856526375,
+            0.005351903848350048,
+            -0.01795627921819687,
+            0.0003585426020435989,
+            0.0009386236779391766,
+            -0.020365996286273003,
+            0.007959839887917042,
+            0.006428503431379795,
+            0.004543482791632414,
+            0.003585425904020667,
+            -0.005690041929483414,
+            0.008643888868391514,
+            -0.02109668403863907,
+            -0.010952552780508995,
+            0.025154337286949158,
+            -0.0025826727505773306,
+            0.015360003337264061,
+            0.012157411314547062,
+            -0.0024991098325699568,
+            -0.009592228569090366,
+            -0.026817819103598595,
+            -0.0035232396330684423,
+            -0.007835467346012592,
+            0.011481136083602905,
+            0.0019977332558482885,
+            -0.028932152315974236,
+            -0.0036262355279177427,
+            -0.026600167155265808,
+            -0.0027808912564069033,
+            -0.019060084596276283,
+            0.00800647959113121,
+            0.018205024302005768,
+            0.00252437312155962,
+            0.006615062244236469,
+            0.00041538471123203635,
+            0.0011018626391887665,
+            -0.0027439682744443417,
+            0.013541055843234062,
+            0.018034012988209724,
+            0.011706560850143433,
+            -0.019231097772717476,
+            0.011045832186937332,
+            -0.003169555449858308,
+            -0.020801300182938576,
+            0.006548989098519087,
+            -0.02558964118361473,
+            -0.007089232560247183,
+            -3.501012906781398e-05,
+            0.0018626725068315864,
+            -0.011760974302887917,
+            -1.905364842968993e-05,
+            -0.0007433199789375067,
+            0.001582834403961897,
+            0.005114818923175335,
+            0.013595468364655972,
+            -0.012219597585499287,
+            0.019308829680085182,
+            0.006583969108760357,
+            0.008185264654457569,
+            0.0020910126622766256,
+            0.0048544141463935375,
+            -0.013074658811092377,
+            -0.03389150649309158,
+            -0.017940733581781387,
+            -0.013028019107878208,
+            0.008340730331838131,
+            0.0029616199899464846,
+            0.005410203710198402,
+            -0.02454802207648754,
+            -0.016588181257247925,
+            0.012250690720975399,
+            0.01680583320558071,
+            0.00793651957064867,
+            0.007998705841600895,
+            -0.014139598235487938,
+            -0.00846510287374258,
+            0.0038147377781569958,
+            -8.122471626847982e-06,
+            0.01403854601085186,
+            0.00763724884018302,
+            -0.006937653291970491,
+            0.008278544060885906,
+            -0.0023961139377206564,
+            -0.02321101725101471,
+            -0.0019578952342271805,
+            0.010004213079810143,
+            -0.007784941233694553,
+            -0.006420730147510767,
+            -0.0009779758984223008,
+            0.008519516326487064,
+            0.011582189239561558,
+            -0.00443077040836215,
+            -0.0013904456282034516,
+            -0.0042830780148506165,
+            0.018982352688908577,
+            -0.015181218273937702,
+            -0.006273037753999233,
+            0.010937006212770939,
+            -0.004831094294786453,
+            0.013579921796917915,
+            0.006113685667514801,
+            0.04076308757066727,
+            -0.012413930147886276,
+            -0.013984132558107376,
+            -0.014738140627741814,
+            -0.008908180519938469,
+            -0.031046485528349876,
+            -0.01812729239463806,
+            -0.022262677550315857,
+            -0.005923239979892969,
+            0.0035446162801235914,
+            0.011691015213727951,
+            0.013774254359304905,
+            -0.007135872263461351,
+            -0.0002958705008495599,
+            0.002988826483488083,
+            0.0008429151494055986,
+            -0.01088259369134903,
+            -0.008954820223152637,
+            -0.0009687451529316604,
+            -0.014147371985018253,
+            0.002203725278377533,
+            -0.018920166417956352,
+            0.009716601110994816,
+            0.018982352688908577,
+            0.013719840906560421,
+            -0.007967612706124783,
+            0.01265490148216486,
+            -0.0136343352496624,
+            0.0013807290233671665,
+            -0.007952066138386726,
+            0.011675468645989895,
+            0.03488648682832718,
+            0.0038108511362224817,
+            -0.02257360890507698,
+            0.0009765184368006885,
+            -0.0012524699559435248,
+            -0.018780246376991272,
+            0.0056706084869802,
+            -0.021687453612685204,
+            0.01943320222198963,
+            -0.015461056493222713,
+            -0.006770527921617031,
+            6.097167351981625e-05,
+            0.0009036439005285501,
+            0.02479676716029644,
+            0.010742674581706524,
+            -0.015119032002985477,
+            0.019961785525083542,
+            0.011263484135270119,
+            -0.006168098654597998,
+            0.007990933023393154,
+            0.02229377068579197,
+            -0.01029959786683321,
+            0.010719354264438152,
+            0.01943320222198963,
+            -0.02031935751438141,
+            -0.009460083208978176,
+            -0.00301797641441226,
+            0.019324377179145813,
+            0.00845733005553484,
+            -0.009436762891709805,
+            0.0036301221698522568,
+            0.00021121461759321392,
+            0.004504616372287273,
+            -0.0014244538033381104,
+            -0.011240164749324322,
+            -0.006012632977217436,
+            -0.004038219340145588,
+            -0.0001887449761852622,
+            0.010517248883843422,
+            -0.015072392299771309,
+            -0.005942673422396183,
+            -0.0018490692600607872,
+            0.015367777086794376,
+            -0.022884540259838104,
+            -0.0017295550787821412,
+            0.005068179219961166,
+            -0.006397410295903683,
+            0.007139758672565222,
+            0.020023971796035767,
+            0.01783190667629242,
+            0.033518388867378235,
+            -0.014528262428939342,
+            0.0033483407460153103,
+            0.02232486382126808,
+            -0.0036106889601796865,
+            0.02031935751438141,
+            0.014403889887034893,
+            -0.004178138449788094,
+            -0.007470123004168272,
+            0.013572148978710175,
+            0.029989318922162056,
+            0.011955305933952332,
+            0.001268016523681581,
+            -0.02336648292839527,
+            -0.0017888263100758195,
+            0.021594174206256866,
+            0.009086965583264828,
+            0.0005387855344451964,
+            0.010346237570047379,
+            0.014815873466432095,
+            -0.013152391649782658,
+            -0.011201297864317894,
+            0.033238548785448074,
+            -0.017132312059402466,
+            -0.02451692894101143,
+            -0.00046615395694971085,
+            -0.005472389981150627,
+            -0.06116017699241638,
+            -0.002351417439058423,
+            0.014668181538581848,
+            -0.0032589479815214872,
+            0.018951259553432465,
+            0.007555629126727581,
+            0.00866720825433731,
+            0.0049982196651399136,
+            -0.01371206808835268,
+            0.003525183070451021,
+            -0.017163405194878578,
+            0.005775548052042723,
+            -0.006245831493288279,
+            -0.003826397703960538,
+            -0.0037739279214292765,
+            0.007497329730540514,
+            0.023413121700286865,
+            -0.0022348184138536453,
+            0.013533282093703747,
+            0.00669279508292675,
+            0.0023280975874513388,
+            0.01808065176010132,
+            0.006595628801733255,
+            0.00406542606651783,
+            0.0008555467356927693,
+            -0.003579595824703574,
+            -0.010315144434571266,
+            -0.018686966970562935,
+            -0.017754174768924713,
+            -0.014442755840718746,
+            0.006012632977217436,
+            0.002687611849978566,
+            -0.01860923506319523,
+            -0.006953199859708548,
+            0.013976359739899635,
+            -0.03348729386925697,
+            0.002287287963554263,
+            0.016852473840117455,
+            0.014248424209654331,
+            0.009755467996001244,
+            0.02165636047720909,
+            0.023273203521966934,
+            -0.0081386249512434,
+            -0.021874012425541878,
+            -0.006949313450604677,
+            0.001381700742058456,
+            -0.012468342669308186,
+            0.017894092947244644,
+            0.023661866784095764,
+            -0.013999679125845432,
+            -0.010804860852658749,
+            -0.004800001159310341,
+            -0.006078705657273531,
+            0.014248424209654331,
+            0.01640162244439125,
+            0.01968194730579853,
+            0.006144778802990913,
+            0.010113039053976536,
+            -0.017552068457007408,
+            0.012491662986576557,
+            0.0133855901658535,
+            -0.009149151854217052,
+            -0.019977333024144173,
+            -0.019184457138180733,
+            -0.021936198696494102,
+            -0.01665036752820015,
+            -0.024750126525759697,
+            0.028263650834560394,
+            0.002454413566738367,
+            0.013129071332514286,
+            0.00977878738194704,
+            -0.014528262428939342,
+            -0.0022348184138536453,
+            0.011263484135270119,
+            0.01680583320558071,
+            0.003188988659530878,
+            0.0005562754231505096,
+            -0.0002934413496404886,
+            0.014124051667749882,
+            -0.015134578570723534,
+            0.007656681817024946,
+            -0.004706721752882004,
+            -0.009910933673381805,
+            0.0034902032930403948,
+            0.005445183254778385,
+            -0.0077655077911913395,
+            -0.012763727456331253,
+            -0.007753848098218441,
+            0.0036689885891973972,
+            -0.007909313775599003,
+            -0.003435790305957198,
+            0.00037165998946875334,
+            -0.0005562754231505096,
+            0.018920166417956352,
+            -0.015064618550240993,
+            -0.0013243727153167129,
+            0.010338463820517063,
+            -0.0064090704545378685,
+            -0.00682105403393507,
+            0.004011013079434633,
+            0.00788210704922676,
+            -0.0063740904442965984,
+            0.007781054358929396,
+            0.016075145453214645,
+            -0.009685507975518703,
+            0.012227371335029602,
+            -0.002194008557125926,
+            0.00587271386757493,
+            0.023008912801742554,
+            -0.001738299964927137,
+            0.002992713125422597,
+            0.011838707141578197,
+            -0.021407615393400192,
+            0.0007097977213561535,
+            -0.015935225412249565,
+            -0.012748180888593197,
+            -0.015546562150120735,
+            -0.010493929497897625,
+            -0.02022607810795307,
+            -0.006704454775899649,
+            0.00937457662075758,
+            -0.004178138449788094,
+            0.010159678757190704,
+            -0.0011941703269258142,
+            -5.492916170624085e-05,
+            -0.020537009462714195,
+            0.0059815398417413235,
+            -0.00788210704922676,
+            0.0016731987707316875,
+            0.0018306076526641846,
+            0.027517415583133698,
+            0.005627855658531189,
+            -0.0029791099950671196,
+            0.005522916093468666,
+            0.013253443874418736,
+            -0.01812729239463806,
+            -0.010610528290271759,
+            0.012087452225387096,
+            0.009631095454096794,
+            -0.009382350370287895,
+            -0.009677735157310963,
+            -0.010820407420396805,
+            -0.031543973833322525,
+            0.006902673747390509,
+            -0.009561135433614254,
+            0.001341862604022026,
+            -0.0002895547077059746,
+            -0.04278413951396942,
+            0.006739434786140919,
+            -0.0019063971703872085,
+            -0.013346723280847073,
+            0.002019109670072794,
+            -0.019619761034846306,
+            0.00871384795755148,
+            -0.01599741168320179,
+            0.01331563014537096,
+            -0.012040812522172928,
+            -0.0023242109455168247,
+            0.009980892762541771,
+            0.023288751021027565,
+            0.0058455076068639755,
+            0.005740568041801453,
+            0.007120325695723295,
+            -8.714576688362285e-05,
+            0.005266397725790739,
+            -0.0022814578842371702,
+            0.0022387050557881594,
+            -0.0017791097052395344,
+            0.015103485435247421,
+            0.002036599675193429,
+            0.01654154248535633,
+            -0.007784941233694553,
+            -0.002930526854470372,
+            -0.008123078383505344,
+            0.01174542773514986,
+            -0.023288751021027565,
+            0.0014720651088282466,
+            0.0007588665466755629,
+            0.0006340082618407905,
+            -0.004123725462704897,
+            0.004535709507763386,
+            -0.007182511501014233,
+            0.031046485528349876,
+            0.016557088121771812,
+            -0.009102512151002884,
+            0.003991579636931419,
+            5.286438317853026e-05,
+            -0.0001824291975935921,
+            0.021734094247221947,
+            -0.01993069238960743,
+            -0.0056744953617453575,
+            0.0027187049854546785,
+            -0.01901344582438469,
+            0.0021881787106394768,
+            0.028512395918369293,
+            -0.014901380054652691,
+            0.001269959844648838,
+            0.00017319842299912125,
+            0.003350284183397889,
+            -0.018811339512467384,
+            0.001708178548142314,
+            0.014613768085837364,
+            0.007575062569230795,
+            -0.003012146335095167,
+            0.015429963357746601,
+            -0.012382837012410164,
+            0.009483402594923973,
+            0.0058688269928097725,
+            -0.0010124698746949434,
+            0.004318057559430599,
+            -0.000593198521528393,
+            -0.02977166697382927,
+            -0.005095385946333408,
+            0.004038219340145588,
+            -0.0019501218339428306,
+            -0.010999192483723164,
+            0.003651498816907406,
+            -0.009685507975518703,
+            -0.01212631817907095,
+            -0.007034819573163986,
+            0.03721847012639046,
+            0.015515469014644623,
+            0.0045123896561563015,
+            0.012437249533832073,
+            -0.0019404053455218673,
+            0.013035791926085949,
+            0.01848486252129078,
+            -0.016588181257247925,
+            -0.004115952178835869,
+            0.015367777086794376,
+            0.018671421334147453,
+            0.009825427085161209,
+            -0.004049879498779774,
+            0.025682920590043068,
+            0.0024952231906354427,
+            -0.020676927641034126,
+            0.006918220315128565,
+            0.0007384616765193641,
+            0.02596275880932808,
+            -0.00698429299518466,
+            -0.016463808715343475,
+            -0.01901344582438469,
+            0.00949894916266203,
+            -0.005600648932158947,
+            -0.005884373560547829,
+            -0.009677735157310963,
+            -0.004617328755557537,
+            0.026398062705993652,
+            0.008527289144694805,
+            0.0006412956863641739,
+            -0.015911906957626343,
+            0.008558382280170918,
+            -0.01862478069961071,
+            -0.013401136733591557,
+            -0.004003239795565605,
+            0.015771986916661263,
+            -0.02571401372551918,
+            -0.001339919283054769,
+            -0.008449556306004524,
+            -0.00013530367868952453,
+            0.012180731631815434,
+            0.021951746195554733,
+            0.0015624294755980372,
+            0.001924858777783811,
+            -0.006051499396562576,
+            -0.024081625044345856,
+            -0.015251177363097668,
+            -0.013727614656090736,
+            -0.007761621382087469,
+            -0.030673367902636528,
+            -0.010035306215286255,
+            -0.0006534414133056998,
+            0.00013821866014041007,
+            -0.0012388667091727257,
+            0.037591587752103806,
+            -0.006583969108760357,
+            -0.00244469684548676,
+            -0.006342997308820486,
+            0.009079192765057087,
+            -0.010664941743016243,
+            0.006261378061026335,
+            -0.006533442530781031,
+            -0.0002924696891568601,
+            0.004045992624014616,
+            0.002048259600996971,
+            0.02807709202170372,
+            0.002182348631322384,
+            -0.006591742392629385,
+            0.008954820223152637,
+            -0.01284923404455185,
+            -0.021967291831970215,
+            0.01693020574748516,
+            -0.024003891274333,
+            -0.013813120312988758,
+            -0.026600167155265808,
+            -0.016728101298213005,
+            -0.017256684601306915,
+            -0.01680583320558071,
+            -0.039954666048288345,
+            0.0023222677409648895,
+            0.02728421613574028,
+            -0.006471256725490093,
+            0.017132312059402466,
+            -0.0023747372906655073,
+            -0.009436762891709805,
+            -0.0010163565166294575,
+            -0.02007061243057251,
+            0.006300244480371475,
+            -0.0016595955239608884,
+            -0.005763887893408537,
+            0.013743161223828793,
+            -0.006560649257153273,
+            0.017396602779626846,
+            0.002920810366049409,
+            -0.019619761034846306,
+            -0.0033833205234259367,
+            0.01876470074057579,
+            0.023397576063871384,
+            -0.0033638873137533665,
+            -0.000545587157830596,
+            0.003752551507204771,
+            -0.010120811872184277,
+            0.01809619925916195,
+            -0.015585429035127163,
+            0.005002106539905071,
+            0.0012446966720744967,
+            0.014085185714066029,
+            0.005375223699957132,
+            0.0009070447413250804,
+            0.006992066279053688,
+            0.0006242916570045054,
+            0.006611175369471312,
+            -0.007411823607981205,
+            0.00134574924595654,
+            0.001861700788140297,
+            -0.0183449424803257,
+            0.03395369276404381,
+            0.013097978197038174,
+            0.012934739701449871,
+            -0.03684535250067711,
+            0.017660895362496376,
+            0.0016440489562228322,
+            -0.008449556306004524,
+            -0.007660568691790104,
+            0.030129237100481987,
+            0.011978626251220703,
+            0.00365538545884192,
+            0.009017006494104862,
+            -0.010594981722533703,
+            0.012157411314547062,
+            -0.011185751296579838,
+            -0.004586235620081425,
+            -0.005421863403171301,
+            -0.011193525046110153,
+            0.01823611743748188,
+            0.009825427085161209,
+            0.0022425916977226734,
+            -0.0046561951749026775,
+            -0.013393362984061241,
+            0.029989318922162056,
+            0.008022026158869267,
+            -0.004784454591572285,
+            0.008651661686599255,
+            -0.014310610480606556,
+            0.0005718219908885658,
+            0.000107489904621616,
+            0.007567289285361767,
+            -0.016728101298213005,
+            -5.696357402484864e-05,
+            -0.028139278292655945,
+            -0.017552068457007408,
+            -0.02335093729197979,
+            0.0002368421555729583,
+            0.0014438870130106807,
+            0.0177230816334486,
+            -0.0011941703269258142,
+            0.004527936223894358,
+            -7.360325253102928e-05,
+            0.007128098979592323,
+            0.004527936223894358,
+            0.014792554080486298,
+            0.012266237288713455,
+            0.02638251520693302,
+            -0.00570558849722147,
+            -0.028792232275009155,
+            0.0041975718922913074,
+            0.007909313775599003,
+            0.008962593041360378,
+            0.027455229312181473,
+            0.003634008811786771,
+            -0.0068949004635214806,
+            -0.00751287629827857,
+            0.008441783487796783,
+            0.025247616693377495,
+            -0.003908017184585333,
+            -0.018002919852733612,
+            0.018313849344849586,
+            0.020630288869142532,
+            0.0033600006718188524,
+            -0.0040693124756217,
+            0.01703903265297413,
+            0.011442270129919052,
+            -0.019122270867228508,
+            -0.011512229219079018,
+            0.002283401321619749,
+            0.0055889892391860485,
+            0.0028663973789662123,
+            -0.016494901850819588,
+            -0.007645022124052048,
+            -0.03488648682832718,
+            0.007131985388696194,
+            -0.0158652663230896,
+            0.0071436455473303795,
+            0.010369556955993176,
+            -0.03466883301734924,
+            -0.008099758997559547,
+            0.013214577920734882,
+            0.03211919590830803,
+            0.006758867762982845,
+            0.0075439694337546825,
+            -0.0043413774110376835,
+            -0.0014506885781884193,
+            -0.014559355564415455,
+            0.014707047492265701,
+            -0.0328032448887825,
+            0.02229377068579197,
+            0.01441943645477295,
+            0.004780567716807127,
+            -0.0018140894826501608,
+            -0.010276277549564838,
+            0.00558121595531702,
+            -0.0036301221698522568,
+            0.0033774906769394875,
+            -0.03448227420449257,
+            0.0192932840436697,
+            -0.008760487660765648,
+            0.022931179031729698,
+            0.02361522801220417,
+            0.02611822448670864,
+            -0.006335224024951458,
+            -0.0013700408162549138,
+            0.005464616697281599,
+            0.006486803293228149,
+            0.01694575324654579,
+            -0.009436762891709805,
+            -0.0016382189933210611,
+            -0.02073911391198635,
+            0.012701541185379028,
+            -0.023630773648619652,
+            -0.01063384860754013,
+            0.00269149849191308,
+            -0.013603242114186287,
+            -0.025371989235281944,
+            -0.010144132189452648,
+            -0.004100405611097813,
+            -0.0030587860383093357,
+            -0.0016440489562228322,
+            0.008099758997559547,
+            -0.01742769591510296,
+            -0.02465684711933136,
+            -0.014590448699891567,
+            -0.010742674581706524,
+            0.022651340812444687,
+            0.01075822114944458,
+            -0.016572635620832443,
+            -0.0039993529208004475,
+            -0.005682268645614386,
+            0.0071436455473303795,
+            0.007959839887917042,
+            -0.01416291855275631,
+            0.05506592243909836,
+            0.0006393523653969169,
+            0.0027187049854546785,
+            0.0018743324326351285,
+            0.008216357789933681,
+            0.0012913363752886653,
+            0.003585425904020667,
+            -0.002267854753881693,
+            0.004477410111576319,
+            -0.034575555473566055,
+            -0.018173931166529655,
+            0.0033153044059872627,
+            -0.011589962057769299,
+            0.00945230945944786,
+            -0.01915336400270462,
+            0.011589962057769299,
+            -0.01719449833035469,
+            -0.021718546748161316,
+            0.006109798792749643,
+            -0.0027886645402759314,
+            -0.002106559230014682,
+            -0.014652634970843792,
+            -0.017132312059402466,
+            0.007567289285361767,
+            0.02453247457742691,
+            -0.0024738467764109373,
+            0.01547660306096077,
+            -0.02022607810795307,
+            -0.005254738032817841,
+            -0.012810367159545422,
+            0.01089036650955677,
+            -0.010462836362421513,
+            -0.009017006494104862,
+            -0.009017006494104862,
+            -0.02164081484079361,
+            0.017816361039876938,
+            -0.002372794086113572,
+            0.00800647959113121,
+            -0.0206147413700819,
+            0.011224618181586266,
+            -0.010618302039802074,
+            0.016992392018437386,
+            0.000213036488275975,
+            -0.01915336400270462,
+            -0.014116278849542141,
+            0.01680583320558071,
+            -0.023972798138856888,
+            -0.0012835630914196372,
+            0.027346402406692505,
+            0.005934900138527155,
+            0.025807293131947517,
+            -0.006832714192569256,
+            -0.024983325973153114,
+            -0.01311352476477623,
+            0.016494901850819588,
+            -0.00957668200135231,
+            -0.00015947372594382614,
+            -0.00701927300542593,
+            0.007334090769290924,
+            -0.01482364721596241,
+            0.028683407232165337,
+            -0.01822056993842125,
+            0.007998705841600895,
+            0.0015604861546307802,
+            -0.008247450925409794,
+            0.014069639146327972,
+            -0.0048544141463935375,
+            -0.007493442855775356,
+            -0.0077072083950042725,
+            -0.004489069804549217,
+            -0.010664941743016243,
+            -0.0034571667201817036,
+            -0.029973771423101425,
+            0.0005674495478160679,
+            -0.0005266398075036705,
+            -0.012382837012410164,
+            -0.011753200553357601,
+            -0.004469636827707291,
+            -0.004671741742640734,
+            0.0019238870590925217,
+            0.011006966233253479,
+            0.00012704456457868218,
+            -0.01161328237503767,
+            -0.008379597216844559,
+            0.011776520870625973,
+            -0.00718639837577939,
+            0.005690041929483414,
+            -0.02557409554719925,
+            0.01194753311574459,
+            -0.013758707791566849,
+            0.0057600014843046665,
+            -0.009055872447788715,
+            0.019977333024144173,
+            0.03120194934308529,
+            0.0044618635438382626,
+            0.004166478756815195,
+            -0.007442916743457317,
+            0.0029791099950671196,
+            0.002434980357065797,
+            0.016215063631534576,
+            -0.008091985248029232,
+            0.017225591465830803,
+            -0.012748180888593197,
+            0.007769394665956497,
+            -0.012040812522172928,
+            0.01717895083129406,
+            0.006245831493288279,
+            0.0037953045684844255,
+            -0.009584455750882626,
+            -0.0007778139552101493,
+            -0.02897879108786583,
+            0.010672714561223984,
+            -0.01770753413438797,
+            -0.0009969233069568872,
+            0.016215063631534576,
+            0.009405669756233692,
+            -0.013657654635608196,
+            0.012794820591807365,
+            0.03134186938405037,
+            0.024827860295772552,
+            -0.004115952178835869,
+            -0.0012709314469248056,
+            0.01640162244439125,
+            -0.0039993529208004475,
+            0.004982673097401857,
+            -0.009211338125169277,
+            -0.03373603895306587,
+            -0.019184457138180733,
+            -0.018780246376991272,
+            0.0040071262046694756,
+            0.010058625601232052,
+            -0.00020538465469144285,
+            0.0110924718901515,
+            0.010781540535390377,
+            0.019650854170322418,
+            -0.019308829680085182,
+            -0.0034299602266401052,
+            -0.008519516326487064,
+            -0.005095385946333408,
+            0.028154823929071426,
+            0.008542835712432861,
+            0.021438708528876305,
+            -0.011038059368729591,
+            0.006179758347570896,
+            -0.020925672724843025,
+            -0.01371206808835268,
+            -0.01252275612205267,
+            0.021423162892460823,
+            0.0027944946195930243,
+            -0.0013661541743203998,
+            0.02572956122457981,
+            0.005359677132219076,
+            -0.015181218273937702,
+            0.006486803293228149,
+            -0.006191418506205082,
+            0.010408422909677029,
+            0.006261378061026335,
+            0.008053119294345379,
+            -0.025807293131947517,
+            -0.02507660537958145,
+            -0.01585749350488186,
+            0.010369556955993176,
+            0.013020245358347893,
+            0.01101473905146122,
+            -0.014139598235487938,
+            -0.018593687564134598,
+            0.0056395153515040874,
+            0.01331563014537096,
+            -0.008278544060885906,
+            -0.010968099348247051,
+            -0.0036126323975622654,
+            0.00970105454325676,
+            0.020490368828177452,
+            5.5475720728281885e-05,
+            0.007501216139644384,
+            0.01075822114944458,
+            0.005658948794007301,
+            -0.0043413774110376835,
+            0.008123078383505344,
+            -0.010983645915985107,
+            -0.01694575324654579,
+            0.01548437587916851,
+            -0.010385103523731232,
+            0.009156924672424793,
+            0.010268504731357098,
+            -0.010237411595880985,
+            -0.001987044932320714,
+            -0.007046479266136885,
+            0.01280259434133768,
+            0.025760654360055923,
+            -0.004353037569671869,
+            -0.003435790305957198,
+            -0.017210043966770172,
+            0.002506883116438985,
+            0.0023008910939097404,
+            0.018453769385814667,
+            0.0058183008804917336,
+            0.008822673931717873,
+            -0.010719354264438152,
+            -0.0022697981912642717,
+            0.025543002411723137,
+            0.010618302039802074,
+            -0.006704454775899649,
+            -0.00024813771597109735
+          ],
+          "title": "Capacity Factor",
+          "keyphrases": [
+            "capacity factors",
+            "downstream task fine-tuning",
+            "expert choice",
+            "gating computational footprint",
+            "fine-tuning performance"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "d3110cfa-448f-46ee-b3ea-c6734cf305de",
+        "properties": {
+          "page_content": "C Capped Expert Choice\n\nAs described in Section 4.5, the maximum number of experts each token is assigned can be capped by an entropy-regularized linear programming. Figure 1 compares the validation perplexity when training the 100M/64E models using the base expert choice method (EC-BASE), expert choice capped by two experts per token (EC-CAP2), expert choice capped by three experts per token (EC-CAP3), and GShard top-2 gating.\n\nAs shown in the figure, restricting the number of experts to 2 degrades the perplexity compared to the base expert choice method. This suggests that a more flexible allocation of experts (e.g. more than 2 experts for a token) can enhance model expressiveness. On the other hand, our EC-CAP2 and EC-CAP3 methods still outperform the top-2 gating method by a clear margin. We believe this confirms the effectiveness of a load balanced training, provided by our method. Finally, EC-CAP3 obtains comparable perplexity to EC-BASE. As indicated by Figure 3, only a little fraction of tokens use more than 3 experts therefore we see little or no difference between EC-BASE and EC-CAP3 variants. We present the fine-tuning results of these methods in Table 2.\n\n",
+          "title": "C Capped Expert Choice",
+          "embedding": [
+            -0.0025292420759797096,
+            -0.00710443127900362,
+            -0.03731034696102142,
+            -0.015119067393243313,
+            3.892158201779239e-05,
+            0.01479687076061964,
+            -0.02490578405559063,
+            0.06395599246025085,
+            -0.031752459704875946,
+            0.013524195179343224,
+            -0.011389643885195255,
+            -0.03146248310804367,
+            -0.014837145805358887,
+            0.002412445843219757,
+            0.01028612069785595,
+            0.020378924906253815,
+            0.0013713486259803176,
+            0.009432300925254822,
+            0.005433037411421537,
+            -0.013242273591458797,
+            0.0022654435597360134,
+            -0.03563492372632027,
+            -0.02867548167705536,
+            0.004981962498277426,
+            0.034249480813741684,
+            0.023761987686157227,
+            0.00903761014342308,
+            -0.013854446820914745,
+            -0.04288434609770775,
+            0.04004901647567749,
+            0.017833571881055832,
+            0.01280730776488781,
+            -0.019879519939422607,
+            -0.036955930292606354,
+            -0.03078586980700493,
+            -0.02911044843494892,
+            0.03830915689468384,
+            0.022859837859869003,
+            -0.0014800898497924209,
+            0.012509276159107685,
+            0.033411771059036255,
+            0.012307903729379177,
+            -0.02360088936984539,
+            -0.013516140170395374,
+            0.01215486042201519,
+            0.0035884627141058445,
+            -0.03898577019572258,
+            -0.011365478858351707,
+            0.010390834882855415,
+            -0.04159555956721306,
+            0.01303284615278244,
+            -0.039533503353595734,
+            0.013500030152499676,
+            0.010060583241283894,
+            -0.040693409740924835,
+            0.009480630047619343,
+            -0.006468093488365412,
+            0.034346140921115875,
+            0.08435102552175522,
+            0.03172023966908455,
+            0.01726972870528698,
+            -0.031301382929086685,
+            -0.0481683649122715,
+            -0.01268648449331522,
+            -0.008747633546590805,
+            -0.016440073028206825,
+            -0.0063593522645533085,
+            0.0023298829328268766,
+            0.02923932671546936,
+            -0.018606843426823616,
+            -0.054096780717372894,
+            0.03360508754849434,
+            -0.028256626799702644,
+            0.00737427081912756,
+            -0.0055820532143116,
+            -0.01622259058058262,
+            0.037632543593645096,
+            0.03502275049686432,
+            0.01321810856461525,
+            0.03734256699681282,
+            -0.004065716173499823,
+            0.03325067088007927,
+            -0.0012092434335500002,
+            0.0340561643242836,
+            0.04088672623038292,
+            -0.02535685896873474,
+            -0.0193801149725914,
+            -0.057254306972026825,
+            -0.028691591694951057,
+            -0.021426063030958176,
+            0.0004978941869921982,
+            0.008199899457395077,
+            -0.0026259010192006826,
+            -0.018510185182094574,
+            -0.03308957442641258,
+            0.04613852873444557,
+            -0.012895911931991577,
+            0.007221227511763573,
+            -0.012646210379898548,
+            -0.01455522421747446,
+            -0.02828884683549404,
+            0.007462874986231327,
+            0.0039448924362659454,
+            -0.02221544459462166,
+            0.03605378046631813,
+            -0.01749526709318161,
+            -0.01784968189895153,
+            0.010181407444179058,
+            0.014579388312995434,
+            0.09305033087730408,
+            0.009891430847346783,
+            -0.0006464066100306809,
+            -0.007772989105433226,
+            -0.010946623980998993,
+            -0.05528890714049339,
+            -0.023649217560887337,
+            -0.04394759237766266,
+            -0.03685927018523216,
+            0.021184414625167847,
+            -0.01915457844734192,
+            0.0004923564265482128,
+            0.0317363478243351,
+            0.007176925428211689,
+            -0.0029420561622828245,
+            -0.04723399877548218,
+            0.026967842131853104,
+            0.024809125810861588,
+            0.051615867763757706,
+            0.00537665281444788,
+            -0.01221929956227541,
+            -0.006584889721125364,
+            0.03650485724210739,
+            -0.018751831725239754,
+            -0.02867548167705536,
+            0.017817461863160133,
+            0.0152721107006073,
+            0.05358126759529114,
+            0.03866357356309891,
+            0.0025131322909146547,
+            -0.007893812842667103,
+            0.0069594429805874825,
+            -0.0071004037745296955,
+            0.004671848379075527,
+            0.009134269319474697,
+            -0.046686261892318726,
+            0.019235126674175262,
+            9.407883771928027e-05,
+            0.05799535661935806,
+            -0.026693973690271378,
+            0.018284646794199944,
+            0.005928414408117533,
+            0.036827053874731064,
+            -0.028385505080223083,
+            0.046815142035484314,
+            -0.018010780215263367,
+            -0.021748259663581848,
+            -0.013781952671706676,
+            -0.0015817830571904778,
+            -0.009005390107631683,
+            -0.03573158383369446,
+            0.01416053343564272,
+            0.03134971112012863,
+            -0.0021587160881608725,
+            0.016101766377687454,
+            0.0020167483016848564,
+            0.04194997623562813,
+            0.013201998546719551,
+            0.01603732816874981,
+            -0.07481401413679123,
+            0.029062118381261826,
+            0.029223216697573662,
+            -0.029690401628613472,
+            0.04455976560711861,
+            0.014982134103775024,
+            0.0014508907916024327,
+            0.02337535098195076,
+            0.003131346544250846,
+            0.0004191070911474526,
+            -0.011011063121259212,
+            0.0411444827914238,
+            -0.01603732816874981,
+            -0.015376824885606766,
+            0.024712465703487396,
+            -0.011736005544662476,
+            0.033895064145326614,
+            -0.0009348730673082173,
+            0.029915938153862953,
+            0.02841772511601448,
+            -0.013137559406459332,
+            0.007458847481757402,
+            0.013959160074591637,
+            0.041885536164045334,
+            -0.0005522647988982499,
+            0.005771343596279621,
+            0.032010216265916824,
+            -0.01308923028409481,
+            0.0281760785728693,
+            -0.00663724634796381,
+            0.018026890233159065,
+            0.019927849993109703,
+            -0.009891430847346783,
+            -0.010334450751543045,
+            -0.007189007941633463,
+            -0.02864326350390911,
+            0.015594307333230972,
+            0.04852278158068657,
+            0.02666175551712513,
+            0.026275118812918663,
+            -0.005364570766687393,
+            -0.021184414625167847,
+            0.023536449298262596,
+            -0.03252572938799858,
+            0.002066084649413824,
+            -0.02712894044816494,
+            -0.04733065515756607,
+            0.024003634229302406,
+            -0.0006433859816752374,
+            -0.009464520029723644,
+            0.0012585797812789679,
+            0.007760906592011452,
+            0.03172023966908455,
+            0.0423043929040432,
+            0.02500244230031967,
+            -0.020733339712023735,
+            -0.008119350299239159,
+            -0.010994953103363514,
+            -0.01520767156034708,
+            0.006814454682171345,
+            0.02335924096405506,
+            -0.01900959014892578,
+            0.006057293154299259,
+            0.009980034083127975,
+            0.0008528137113898993,
+            -0.0016804557526484132,
+            0.016850873827934265,
+            -0.010866074822843075,
+            0.025163540616631508,
+            -0.02242487110197544,
+            0.016528677195310593,
+            0.012791198678314686,
+            -0.020701121538877487,
+            -0.0028131776489317417,
+            -0.03605378046631813,
+            0.008115322329103947,
+            -0.004542969632893801,
+            -0.008079075254499912,
+            -0.02326258271932602,
+            0.0007093356107361615,
+            -0.0043335421942174435,
+            -0.012042091228067875,
+            -0.027547795325517654,
+            -0.02453525923192501,
+            0.0004686951288022101,
+            0.009392025880515575,
+            -0.016045382246375084,
+            0.026758413761854172,
+            0.004265075083822012,
+            -0.05799535661935806,
+            0.012920076958835125,
+            -0.024245280772447586,
+            0.06907891482114792,
+            0.008812072686851025,
+            -0.0105760982260108,
+            0.015489593148231506,
+            -0.023939194157719612,
+            -0.011510467156767845,
+            0.01462771836668253,
+            -0.00922287255525589,
+            -0.003711300203576684,
+            -0.0029380286578089,
+            0.02770889364182949,
+            -0.01515128742903471,
+            -0.010382779873907566,
+            -0.045848552137613297,
+            0.05499893054366112,
+            -0.029529303312301636,
+            0.0008810058934614062,
+            0.007825345732271671,
+            -0.007084294222295284,
+            0.018703503534197807,
+            0.01973453164100647,
+            0.02186102792620659,
+            0.01009280327707529,
+            0.01975064165890217,
+            0.015707075595855713,
+            0.0022775260731577873,
+            -0.0622805692255497,
+            -0.002076153177767992,
+            -0.03217131271958351,
+            -0.03159135952591896,
+            -0.02806330844759941,
+            -0.03067309968173504,
+            -0.010076693259179592,
+            0.018155768513679504,
+            0.01971842162311077,
+            -0.02537296898663044,
+            -0.015803735703229904,
+            0.008852346800267696,
+            0.004800726659595966,
+            -0.0041885534301400185,
+            -0.023230362683534622,
+            0.0024990360252559185,
+            -0.03511941060423851,
+            -0.028691591694951057,
+            -0.06714573502540588,
+            0.004402008838951588,
+            -0.008115322329103947,
+            0.010962733998894691,
+            -0.044914182275533676,
+            0.03302513435482979,
+            0.03373396769165993,
+            -0.010463329032063484,
+            -0.0003755602228920907,
+            -0.005481366999447346,
+            -0.006588917225599289,
+            0.008054911158978939,
+            0.020282264798879623,
+            0.030463673174381256,
+            -0.028127748519182205,
+            0.005465256981551647,
+            0.04649294540286064,
+            0.019815079867839813,
+            -0.04839390516281128,
+            -0.026355668902397156,
+            0.014426345005631447,
+            0.021442173048853874,
+            0.000837207306176424,
+            0.015699021518230438,
+            -0.021893247961997986,
+            -0.010986898094415665,
+            -0.02608180046081543,
+            -0.04185331612825394,
+            -0.056803230196237564,
+            -0.010849964804947376,
+            -0.03550604730844498,
+            -0.02126496471464634,
+            -0.009762551635503769,
+            0.015425154007971287,
+            -0.019782861694693565,
+            -0.017430827021598816,
+            -0.021989906206727028,
+            0.03956572338938713,
+            -0.045171938836574554,
+            0.02758001536130905,
+            0.02456747740507126,
+            -0.0045671346597373486,
+            -0.01797856017947197,
+            -0.010495549067854881,
+            -0.007740769535303116,
+            -0.002712491201236844,
+            0.0010280079441145062,
+            0.002889699302613735,
+            0.034120600670576096,
+            -5.767442053183913e-05,
+            0.014442455023527145,
+            -0.035667143762111664,
+            -0.010833854787051678,
+            0.01868739351630211,
+            0.026242898777127266,
+            0.005098758731037378,
+            -0.004414091352373362,
+            -0.045880772173404694,
+            -0.045848552137613297,
+            0.03133360296487808,
+            -0.038921330124139786,
+            0.01639174297451973,
+            0.005868002772331238,
+            0.009287312626838684,
+            -0.04030677303671837,
+            -0.002058029640465975,
+            -0.02184491790831089,
+            -0.01926734670996666,
+            0.04101560637354851,
+            0.013338932767510414,
+            0.03102751635015011,
+            -0.0027165187057107687,
+            0.020266154780983925,
+            -0.039307963103055954,
+            -0.049843788146972656,
+            0.019927849993109703,
+            -0.05061705783009529,
+            0.02277928777039051,
+            0.030334794893860817,
+            0.01315366942435503,
+            -0.023987524211406708,
+            0.029690401628613472,
+            -0.00795422401279211,
+            0.016182316467165947,
+            0.0012464975006878376,
+            0.028401615098118782,
+            -0.0399523563683033,
+            -0.03308957442641258,
+            -0.009053720161318779,
+            0.020169496536254883,
+            0.0004284205788280815,
+            0.0036609568633139133,
+            -0.021329404786229134,
+            -0.002535283099859953,
+            0.008352942764759064,
+            -0.018848491832613945,
+            -0.041885536164045334,
+            0.04146667942404747,
+            -0.014434400014579296,
+            0.007233310025185347,
+            0.0005955599481239915,
+            -0.0030608661472797394,
+            0.0036347785498946905,
+            -0.007676329929381609,
+            0.009947814978659153,
+            -0.005002099554985762,
+            -0.013121449388563633,
+            -0.009762551635503769,
+            -0.003427364630624652,
+            -0.008131432346999645,
+            0.0062143635004758835,
+            0.024003634229302406,
+            -0.011083557270467281,
+            0.019428445026278496,
+            -0.010294175706803799,
+            -0.010197517462074757,
+            -0.01303284615278244,
+            0.013073120266199112,
+            0.04085450619459152,
+            -0.02103942632675171,
+            0.0340561643242836,
+            -0.0007098389905877411,
+            -0.02042725309729576,
+            0.029674291610717773,
+            0.012106531299650669,
+            0.004760452080518007,
+            0.037986960262060165,
+            -0.002492995001375675,
+            0.00475642504170537,
+            -0.03534494712948799,
+            -0.010656647384166718,
+            -0.01865517348051071,
+            -0.05429009720683098,
+            0.02993204817175865,
+            -0.010721086524426937,
+            0.00354013335891068,
+            0.01216291543096304,
+            -0.0006494271801784635,
+            0.012396507896482944,
+            -0.02479301579296589,
+            -0.03724590688943863,
+            -0.018864601850509644,
+            -0.006721823010593653,
+            -0.016738105565309525,
+            -0.020620571449398994,
+            0.005002099554985762,
+            -0.06514811515808105,
+            0.0015445291064679623,
+            -0.0563521571457386,
+            0.004945715423673391,
+            0.03946906328201294,
+            -0.010841909795999527,
+            -0.001440822146832943,
+            -0.01950899325311184,
+            0.009480630047619343,
+            -0.048103928565979004,
+            -0.045171938836574554,
+            0.016142040491104126,
+            0.033282890915870667,
+            -0.010890239849686623,
+            -0.00915843341499567,
+            0.014192752540111542,
+            -0.037632543593645096,
+            -0.0017086479347199202,
+            0.01374973263591528,
+            0.012106531299650669,
+            -0.01480492576956749,
+            0.01056804321706295,
+            -0.022038236260414124,
+            -0.021409953013062477,
+            -0.008054911158978939,
+            0.029883719980716705,
+            0.01416053343564272,
+            -0.011945432983338833,
+            0.004009332042187452,
+            0.010350560769438744,
+            0.03325067088007927,
+            -0.0076521653681993484,
+            -0.005300131160765886,
+            0.010849964804947376,
+            0.016142040491104126,
+            -0.0022654435597360134,
+            0.012340123765170574,
+            -0.007285666652023792,
+            -0.015223781578242779,
+            -0.013717513531446457,
+            -0.003880453296005726,
+            -0.014974079094827175,
+            0.0018657187465578318,
+            -0.003938851412385702,
+            0.015014354139566422,
+            -0.014982134103775024,
+            -0.020523913204669952,
+            -0.02596903219819069,
+            0.02442248910665512,
+            0.02994815818965435,
+            -0.01227568369358778,
+            -0.034571677446365356,
+            -0.02100720815360546,
+            -3.951940743718296e-05,
+            0.022038236260414124,
+            -0.02455136738717556,
+            0.029851499944925308,
+            0.033282890915870667,
+            -0.05554666370153427,
+            -0.002996426774188876,
+            0.004804754164069891,
+            0.006383516825735569,
+            -0.00431340467184782,
+            -0.03615044057369232,
+            -0.0028333149384707212,
+            0.03934018313884735,
+            -0.0007793126278556883,
+            0.04887719824910164,
+            0.0021526748314499855,
+            -0.017108630388975143,
+            -0.032686829566955566,
+            -0.01668977551162243,
+            -0.03863135352730751,
+            0.04900607839226723,
+            -0.020298374816775322,
+            0.034120600670576096,
+            0.014998244121670723,
+            -0.026581205427646637,
+            0.060798466205596924,
+            -0.0376003235578537,
+            -0.006427818909287453,
+            -0.00015921035083010793,
+            -0.007039992138743401,
+            -0.01772080361843109,
+            -0.01228373870253563,
+            0.0002982834121212363,
+            -0.00780923618003726,
+            0.004357706755399704,
+            -0.004990017041563988,
+            -0.0010285114403814077,
+            0.004426173400133848,
+            0.014941859990358353,
+            -0.004418118391185999,
+            0.028852690011262894,
+            0.013822226785123348,
+            0.008127405308187008,
+            0.020523913204669952,
+            -0.02572738565504551,
+            0.016625335440039635,
+            0.027998870238661766,
+            -0.00560621777549386,
+            -0.010390834882855415,
+            -0.008119350299239159,
+            -0.0011427904246374965,
+            0.006991662550717592,
+            -0.028820471838116646,
+            0.015304330736398697,
+            0.014579388312995434,
+            -0.0030205915682017803,
+            -0.020991098135709763,
+            -0.02360088936984539,
+            0.018864601850509644,
+            -0.023165924474596977,
+            0.027773331850767136,
+            -0.021071646362543106,
+            0.00044855783926323056,
+            0.026710083708167076,
+            0.03573158383369446,
+            -0.011083557270467281,
+            0.015006299130618572,
+            -0.00043949607061222196,
+            -0.005058484151959419,
+            -0.020491693168878555,
+            0.0045751892030239105,
+            -0.0008291523554362357,
+            0.009593399241566658,
+            0.011913212947547436,
+            0.005022237077355385,
+            -0.018606843426823616,
+            0.01515128742903471,
+            0.01609371230006218,
+            -0.030689209699630737,
+            0.00428924011066556,
+            0.010664702393114567,
+            0.014176643453538418,
+            0.005070566199719906,
+            0.003234046744182706,
+            -0.023101484403014183,
+            -0.004462420474737883,
+            -0.016738105565309525,
+            -0.01950899325311184,
+            0.044463109225034714,
+            0.0016653528437018394,
+            -0.034088384360075,
+            0.021538831293582916,
+            -0.007748824078589678,
+            0.005171252880245447,
+            0.0281760785728693,
+            -0.009488685056567192,
+            -0.0505204014480114,
+            -0.013185889460146427,
+            -0.006077430211007595,
+            0.016005108132958412,
+            -0.01926734670996666,
+            -0.009247037582099438,
+            0.030479783192276955,
+            -0.004977934993803501,
+            0.0051269507966935635,
+            -0.008079075254499912,
+            0.024841345846652985,
+            0.026581205427646637,
+            0.00314342905767262,
+            0.007970334030687809,
+            0.018945150077342987,
+            -0.006592944730073214,
+            0.009649783372879028,
+            -0.009174543432891369,
+            -0.0035300645977258682,
+            -0.0002856976061593741,
+            -0.005348460748791695,
+            0.06611470878124237,
+            0.0017881902167573571,
+            0.030705319717526436,
+            -0.0016039340989664197,
+            0.02758001536130905,
+            0.015497648157179356,
+            -0.010986898094415665,
+            -0.02266651950776577,
+            0.007644110359251499,
+            -0.012251519598066807,
+            -0.05416122078895569,
+            -0.026855072006583214,
+            -0.022875946015119553,
+            -0.016085656359791756,
+            -0.029883719980716705,
+            -0.006685575935989618,
+            2.6902143872575834e-05,
+            0.009859210811555386,
+            0.004510750062763691,
+            -0.01786579191684723,
+            -0.032010216265916824,
+            0.00969811249524355,
+            -0.0011850787559524179,
+            0.025340748950839043,
+            -0.005348460748791695,
+            0.010487494058907032,
+            0.007221227511763573,
+            -0.005783426109701395,
+            0.019557323306798935,
+            -0.027451135218143463,
+            0.014007490128278732,
+            -1.3608394056063844e-06,
+            0.011333259753882885,
+            0.035860463976860046,
+            -0.014394125901162624,
+            -0.01726972870528698,
+            -0.005384707823395729,
+            0.008107267320156097,
+            0.023165924474596977,
+            -0.009263147599995136,
+            -0.014877419918775558,
+            0.0021103867329657078,
+            0.005175280384719372,
+            -0.032477401196956635,
+            0.017591925337910652,
+            -0.014200807549059391,
+            -0.05764094367623329,
+            -0.004764479584991932,
+            -0.021877137944102287,
+            0.019219016656279564,
+            -0.025195760652422905,
+            -0.006890976335853338,
+            -0.018123548477888107,
+            0.0015183506766334176,
+            0.014168588444590569,
+            -0.023633107542991638,
+            0.01103522814810276,
+            0.05899416655302048,
+            0.0044221458956599236,
+            0.01762414537370205,
+            0.009464520029723644,
+            -0.005880084820091724,
+            -0.008071020245552063,
+            0.003036701353266835,
+            0.021812697872519493,
+            -0.0002548624179325998,
+            -0.031301382929086685,
+            0.039050206542015076,
+            -0.0004669331246986985,
+            0.004591299220919609,
+            -0.010721086524426937,
+            -0.0007571616442874074,
+            -0.010463329032063484,
+            0.009843100793659687,
+            -0.02385864593088627,
+            0.008892621845006943,
+            -0.019927849993109703,
+            0.03731034696102142,
+            0.0008155597024597228,
+            -0.003642833326011896,
+            -0.025904593989253044,
+            0.0013814171543344855,
+            -0.03199410438537598,
+            0.016182316467165947,
+            0.02911044843494892,
+            -0.012324013747274876,
+            -0.0014539114199578762,
+            0.012267629615962505,
+            -0.02737058699131012,
+            -0.001367321121506393,
+            -0.009843100793659687,
+            0.01402360014617443,
+            0.015111012384295464,
+            0.0030165640637278557,
+            0.034926094114780426,
+            0.01267842948436737,
+            0.009931704960763454,
+            0.013016736134886742,
+            0.0001655032392591238,
+            0.015747351571917534,
+            -0.0038140001706779003,
+            0.005578025709837675,
+            -0.010874129831790924,
+            -0.013604744337499142,
+            -0.021168306469917297,
+            0.01592455804347992,
+            0.021893247961997986,
+            0.011663510464131832,
+            0.03750366345047951,
+            0.014780761674046516,
+            0.026468437165021896,
+            0.0022714848164469004,
+            -0.03582824394106865,
+            0.001979494234547019,
+            0.007426627911627293,
+            -0.025082992389798164,
+            -0.004889330826699734,
+            0.0193801149725914,
+            0.011204380542039871,
+            -0.02501855231821537,
+            0.005851892754435539,
+            0.005396790336817503,
+            -0.002138578798621893,
+            -0.0091020492836833,
+            -0.03579602390527725,
+            0.0021989906672388315,
+            0.019992288202047348,
+            0.025791823863983154,
+            0.003517982317134738,
+            0.015787625685334206,
+            0.007865620777010918,
+            -0.009657838381826878,
+            0.03534494712948799,
+            0.006633218843489885,
+            -0.039275746792554855,
+            -0.015223781578242779,
+            -0.008650974370539188,
+            0.01797856017947197,
+            -0.00029677312704734504,
+            0.0031857171561568975,
+            -0.00969811249524355,
+            0.0047805896028876305,
+            -0.007023882120847702,
+            -0.009303421713411808,
+            0.03395950421690941,
+            0.007011800073087215,
+            0.011317149735987186,
+            -0.004273130092769861,
+            0.018413526937365532,
+            -0.014466620050370693,
+            -0.006540587637573481,
+            0.010366669856011868,
+            0.02314981445670128,
+            -0.025276310741901398,
+            0.015167397446930408,
+            -0.005718986503779888,
+            0.003246129024773836,
+            0.016174260526895523,
+            -0.00431340467184782,
+            -0.01562652736902237,
+            -0.02972262166440487,
+            0.0038683710154145956,
+            -0.014257192611694336,
+            -0.008401271887123585,
+            -0.020475583150982857,
+            -0.014853255823254585,
+            0.030350904911756516,
+            -0.0193801149725914,
+            0.0023802260402590036,
+            -0.0014559251721948385,
+            0.016238700598478317,
+            -0.02031448483467102,
+            -0.013483921065926552,
+            0.025066882371902466,
+            0.05261467769742012,
+            0.0019684189464896917,
+            0.02970651164650917,
+            0.07938919961452484,
+            -0.030689209699630737,
+            -0.013645019382238388,
+            0.019315676763653755,
+            -0.03028646484017372,
+            0.0034535429440438747,
+            0.03895355015993118,
+            -0.04501084238290787,
+            -0.06166839599609375,
+            0.013177834451198578,
+            -0.034603897482156754,
+            -0.000111384317278862,
+            0.010439164005219936,
+            0.008747633546590805,
+            -0.019347894936800003,
+            -0.00610562227666378,
+            0.028804361820220947,
+            -0.007414545398205519,
+            -0.003664984367787838,
+            -0.029319874942302704,
+            -0.011470193043351173,
+            -0.018123548477888107,
+            -0.005388735327869654,
+            -0.0063593522645533085,
+            0.0211360864341259,
+            -0.0006786262383684516,
+            0.003614641260355711,
+            -0.019573433324694633,
+            0.00046743653365410864,
+            0.013725567609071732,
+            0.018590733408927917,
+            0.049972668290138245,
+            -9.590377885615453e-05,
+            0.013983325101435184,
+            -0.01715696044266224,
+            -0.026355668902397156,
+            0.041273362934589386,
+            -0.006955415476113558,
+            0.040822289884090424,
+            -0.02725781872868538,
+            -0.0021748258732259274,
+            -0.006568779703229666,
+            0.025807933881878853,
+            -0.01913846842944622,
+            0.013934995979070663,
+            -0.010729141533374786,
+            -0.0117521146312356,
+            -0.020604461431503296,
+            -0.00766827492043376,
+            0.0020328580867499113,
+            -0.026387887075543404,
+            -0.005968688987195492,
+            -0.004108004737645388,
+            0.027209488674998283,
+            0.047846172004938126,
+            0.005416927393525839,
+            -0.004124114289879799,
+            -0.009077884256839752,
+            -0.015392934903502464,
+            -0.008667084388434887,
+            -0.01726972870528698,
+            -0.015819845721125603,
+            -0.01579567976295948,
+            -0.028014980256557465,
+            -0.019766751676797867,
+            -0.019831189885735512,
+            -0.007325941231101751,
+            0.0029380286578089,
+            -0.0007752851815894246,
+            0.0076521653681993484,
+            -0.04639628529548645,
+            -0.0002455489302519709,
+            -0.03065699152648449,
+            -0.01426524668931961,
+            0.017060300335288048,
+            -0.03859913349151611,
+            -0.00634324224665761,
+            0.0004344617482274771,
+            -0.0020217825658619404,
+            0.0015999067109078169,
+            -0.015119067393243313,
+            0.007797153666615486,
+            -0.02795054018497467,
+            -0.007865620777010918,
+            0.002593681216239929,
+            0.017817461863160133,
+            0.0013310740469023585,
+            0.021538831293582916,
+            -0.03101140633225441,
+            0.011760169640183449,
+            -0.039662379771471024,
+            -0.02619457058608532,
+            -0.042691025882959366,
+            0.027048390358686447,
+            -0.029770949855446815,
+            -0.01168767549097538,
+            0.008804017677903175,
+            -0.0007818297599442303,
+            0.00716887041926384,
+            -0.0152721107006073,
+            -0.0013622867409139872,
+            -0.0027265872340649366,
+            0.007039992138743401,
+            -0.005400817841291428,
+            -0.021748259663581848,
+            -0.017527485266327858,
+            -0.017285838723182678,
+            0.020749449729919434,
+            -0.035054970532655716,
+            0.02044336311519146,
+            -0.012549551203846931,
+            0.02253764122724533,
+            -0.020250046625733376,
+            -0.017898011952638626,
+            -0.006641273852437735,
+            0.040210116654634476,
+            -0.02033059485256672,
+            0.011309094727039337,
+            0.039275746792554855,
+            0.04240104928612709,
+            -0.028611043468117714,
+            0.02571127563714981,
+            -0.022376542910933495,
+            0.015489593148231506,
+            0.055256687104701996,
+            0.01772080361843109,
+            0.02983538992702961,
+            -0.004909468349069357,
+            -0.026919512078166008,
+            -0.008804017677903175,
+            -0.011011063121259212,
+            0.0034736802335828543,
+            0.003671025624498725,
+            -0.02901378832757473,
+            -0.002052995376288891,
+            0.012025982141494751,
+            -0.02397141419351101,
+            0.043850935995578766,
+            0.0013300671707838774,
+            0.01280730776488781,
+            0.005952578969299793,
+            -0.016979752108454704,
+            0.02207045629620552,
+            -0.010704976506531239,
+            0.0005127454060129821,
+            0.009665893390774727,
+            -0.0060170185752213,
+            -0.004683930426836014,
+            -0.03511941060423851,
+            0.01121243555098772,
+            0.024116402491927147,
+            -0.004635601304471493,
+            0.003783794352784753,
+            -0.01221124455332756,
+            -0.002809150144457817,
+            0.005316241178661585,
+            0.020588351413607597,
+            0.0016633390914648771,
+            0.03180078789591789,
+            -0.011647401377558708,
+            0.00777701660990715,
+            -0.011840718798339367,
+            0.023810315877199173,
+            0.008062966167926788,
+            -0.004031483083963394,
+            -0.023536449298262596,
+            -0.010793580673635006,
+            0.017946342006325722,
+            0.01760803535580635,
+            -0.030818087980151176,
+            -0.040596749633550644,
+            -0.021103866398334503,
+            0.016343414783477783,
+            0.0010471383575350046,
+            -0.04143446311354637,
+            -0.012420672923326492,
+            0.015135177411139011,
+            0.01168767549097538,
+            0.02948097325861454,
+            0.006536560133099556,
+            0.007442737463861704,
+            -0.00027009123004972935,
+            0.010551933199167252,
+            0.036472637206315994,
+            9.779164975043386e-05,
+            -0.009021500125527382,
+            -0.014998244121670723,
+            0.022956496104598045,
+            -0.029787059873342514,
+            0.0323324128985405,
+            0.011292984709143639,
+            -0.030721429735422134,
+            0.017704693600535393,
+            0.004957797471433878,
+            -0.018010780215263367,
+            0.03515163064002991,
+            -0.03653707727789879,
+            0.00525582954287529,
+            0.028691591694951057,
+            -0.008010609075427055,
+            -0.01913846842944622,
+            -0.002654093084856868,
+            0.006234501022845507,
+            -0.019702311605215073,
+            -0.0030326738487929106,
+            -0.001486130990087986,
+            0.015779569745063782,
+            -0.015280165709555149,
+            0.014297466725111008,
+            0.026403997093439102,
+            0.014659937471151352,
+            -0.023182032629847527,
+            -0.01714085042476654,
+            -0.020056728273630142,
+            -0.003956974949687719,
+            -0.007442737463861704,
+            0.008441546931862831,
+            -0.019460665062069893,
+            0.021200524643063545,
+            -0.02524409070611,
+            -0.05213138461112976,
+            -0.0007012806599959731,
+            0.005590108223259449,
+            -0.020620571449398994,
+            0.004824891686439514,
+            -0.046202968806028366,
+            0.014007490128278732,
+            0.012074311263859272,
+            0.03885689005255699,
+            -0.007772989105433226,
+            -0.0030105228070169687,
+            0.006488230545073748,
+            0.010777470655739307,
+            0.02853049337863922,
+            -0.032139092683792114,
+            -0.0010682825231924653,
+            0.019798971712589264,
+            -0.0019231099868193269,
+            -0.0015676870243623853,
+            -0.0054733119904994965,
+            -0.04417313262820244,
+            -0.033186230808496475,
+            0.03708481043577194,
+            0.010986898094415665,
+            -0.003620682517066598,
+            0.03065699152648449,
+            -0.023101484403014183,
+            0.009488685056567192,
+            -0.016142040491104126,
+            0.016593115404248238,
+            0.010954678989946842,
+            0.02912655845284462,
+            -0.02948097325861454,
+            -0.0013653073692694306,
+            0.03315401449799538,
+            0.010841909795999527,
+            -0.014120258390903473,
+            0.00024693337036296725,
+            0.018300756812095642,
+            0.020991098135709763,
+            -0.017237508669495583,
+            0.016270920634269714,
+            0.005686766933649778,
+            -0.004236883018165827,
+            -0.019798971712589264,
+            0.0013149641454219818,
+            0.0223282128572464,
+            -0.015135177411139011,
+            0.03006092645227909,
+            -0.009657838381826878,
+            -0.010004199109971523,
+            0.04613852873444557,
+            0.02606569230556488,
+            -0.013411426916718483,
+            4.716527837445028e-05,
+            0.006238528527319431,
+            0.017994670197367668,
+            0.007551478687673807,
+            -0.012348177842795849,
+            0.0006000908324494958,
+            -0.02856271341443062,
+            -0.0029923992697149515,
+            0.007458847481757402,
+            -0.008884566836059093,
+            0.009367860853672028,
+            0.021345512941479683,
+            0.005686766933649778,
+            -0.006101594772189856,
+            0.003542147111147642,
+            -0.010318340733647346,
+            0.019347894936800003,
+            0.03250962123274803,
+            -0.01865517348051071,
+            -0.018413526937365532,
+            0.012460947036743164,
+            -0.026017362251877785,
+            0.03785808011889458,
+            0.009915594942867756,
+            -0.0058196731843054295,
+            0.029738731682300568,
+            0.09479019045829773,
+            -0.012791198678314686,
+            0.0223282128572464,
+            -0.027435027062892914,
+            -0.007861592806875706,
+            -0.009762551635503769,
+            -0.010471384041011333,
+            0.01416053343564272,
+            0.042336612939834595,
+            -0.0006982600898481905,
+            0.028127748519182205,
+            0.010237791575491428,
+            0.0009162460919469595,
+            0.02372976765036583,
+            -0.013604744337499142,
+            0.009738387539982796,
+            -0.004172443877905607,
+            -0.0050745937041938305,
+            -0.009198708459734917,
+            0.004587271716445684,
+            -0.0017318058526143432,
+            0.01678643375635147,
+            -0.00408383971080184,
+            0.0025614616461098194,
+            -0.019428445026278496,
+            -0.019815079867839813,
+            0.006427818909287453,
+            -0.012203189544379711,
+            -0.018284646794199944,
+            -0.017656365409493446,
+            0.00240640458650887,
+            -0.0062143635004758835,
+            0.0012867719633504748,
+            -0.007704521995037794,
+            -0.024486929178237915,
+            0.01844574511051178,
+            -0.0038764257915318012,
+            0.0015425154706463218,
+            0.021635491400957108,
+            -0.017801353707909584,
+            -0.014821035787463188,
+            0.016866983845829964,
+            -0.010898293927311897,
+            0.022747067734599113,
+            -0.025985142216086388,
+            -0.0026661755982786417,
+            0.0012575729051604867,
+            -0.0069594429805874825,
+            -0.015223781578242779,
+            -0.015578197315335274,
+            -0.0008940951083786786,
+            0.026210680603981018,
+            0.003989194519817829,
+            -0.011405753903090954,
+            -0.01157490722835064,
+            0.01880016177892685,
+            0.021796587854623795,
+            -0.007680357433855534,
+            0.0020117140375077724,
+            0.005139033310115337,
+            0.0028051226399838924,
+            -0.009287312626838684,
+            0.02303704433143139,
+            0.0323324128985405,
+            -0.005553861148655415,
+            -0.000351898925146088,
+            0.01526405569165945,
+            -0.006125759799033403,
+            -0.005896194837987423,
+            0.01063248235732317,
+            0.001617023372091353,
+            0.007877702824771404,
+            0.008852346800267696,
+            -0.025695165619254112,
+            -0.013459756039083004,
+            0.02210267446935177,
+            -0.006351297255605459,
+            5.386721750255674e-05,
+            -0.005948551464825869,
+            -0.025775713846087456,
+            0.021087756380438805,
+            0.001424712361767888,
+            0.013798062689602375,
+            -0.008050883188843727,
+            -0.027692783623933792,
+            -0.0038784395437687635,
+            -0.018864601850509644,
+            -0.00020942768605891615,
+            0.01374973263591528,
+            -0.013290602713823318,
+            -0.010406944900751114,
+            0.0018183961510658264,
+            0.0030628798995167017,
+            -0.00862680934369564,
+            -0.013886665925383568,
+            0.005892167333513498,
+            0.0020640708971768618,
+            -0.014104149304330349,
+            -0.040338993072509766,
+            -0.008236146531999111,
+            -0.02207045629620552,
+            0.010737196542322636,
+            0.028578823432326317,
+            -0.012992571108043194,
+            -0.013540305197238922,
+            -0.005054456647485495,
+            0.02712894044816494,
+            0.0071527608670294285,
+            -0.006621136795729399,
+            0.008328777737915516,
+            0.006496285554021597,
+            0.024841345846652985,
+            -0.007434682920575142,
+            -0.0007128596189431846,
+            0.0016160164959728718,
+            0.005642464850097895,
+            -0.008973171003162861,
+            -0.016343414783477783,
+            0.0005160177242942154,
+            -0.012082366272807121,
+            -0.02630733884871006,
+            0.008812072686851025,
+            0.021909357979893684,
+            -0.006766125094145536,
+            0.001452904543839395,
+            0.010028364136815071,
+            0.016866983845829964,
+            -0.017994670197367668,
+            0.004301322158426046,
+            0.00243862415663898,
+            -0.007495094556361437,
+            -0.01996006816625595,
+            0.010423054918646812,
+            0.001549563487060368,
+            0.01913846842944622,
+            0.004414091352373362,
+            0.0016411880496889353,
+            -0.008352942764759064,
+            0.009440355934202671,
+            -0.008264338597655296,
+            -0.0015143232885748148,
+            0.017318058758974075,
+            0.0399201363325119,
+            0.017463047057390213,
+            0.0328962542116642,
+            -0.03708481043577194,
+            -0.026983952149748802,
+            0.011051337234675884,
+            0.014724376611411572,
+            0.007197062950581312,
+            -0.003234046744182706,
+            -0.015417098999023438,
+            0.030930858105421066,
+            0.008618754334747791,
+            0.005271939095109701,
+            -0.009287312626838684,
+            0.007861592806875706,
+            -0.007096376270055771,
+            0.05271133780479431,
+            0.00677820760756731,
+            -0.014643827453255653,
+            0.016182316467165947,
+            -0.016142040491104126,
+            -0.013701403513550758,
+            -0.005654547363519669,
+            -0.010439164005219936,
+            0.013314767740666866,
+            -0.018977370113134384,
+            0.008377106860280037,
+            0.014031654223799706,
+            0.010294175706803799,
+            0.0036508883349597454,
+            0.012299848720431328,
+            0.014015545137226582,
+            0.012541496194899082,
+            0.01657700724899769,
+            0.0003000454162247479,
+            -0.029674291610717773,
+            -0.004651710856705904,
+            -0.002055009128525853,
+            0.0091020492836833,
+            -0.0211199764162302,
+            -0.008707358501851559,
+            0.0003111209371127188,
+            -0.0018264510435983539,
+            0.002591667464002967,
+            -0.028917130082845688,
+            -0.011437973007559776,
+            -0.038341376930475235,
+            0.027306146919727325,
+            0.007950196973979473,
+            0.021893247961997986,
+            0.002742697251960635,
+            -0.028449945151805878,
+            0.010302230715751648,
+            0.02242487110197544,
+            0.0193962249904871,
+            -0.012775088660418987,
+            -0.013411426916718483,
+            -0.030350904911756516,
+            0.0030266328249126673,
+            0.010535823181271553,
+            0.0009686030098237097,
+            0.009561179205775261,
+            0.005984798539429903,
+            -0.0205722413957119,
+            0.009786716662347317,
+            -0.027869991958141327,
+            0.01725361868739128,
+            -0.04945715144276619,
+            0.0055739982053637505,
+            -0.01068081147968769,
+            -0.02092665806412697,
+            -0.005392762832343578,
+            -0.007648137863725424,
+            0.028208298608660698,
+            -0.005690794438123703,
+            -0.022457091137766838,
+            0.016609225422143936,
+            0.016029272228479385,
+            -0.016295084729790688,
+            0.009488685056567192,
+            0.01062442734837532,
+            0.003735464997589588,
+            -0.001605947851203382,
+            0.011123832315206528,
+            0.03215520456433296,
+            0.0035602706484496593,
+            0.01833297684788704,
+            0.010664702393114567,
+            0.009247037582099438,
+            0.006504340562969446,
+            0.0014307535020634532,
+            0.030834197998046875,
+            -0.005235692020505667,
+            0.0060532656498253345,
+            0.007164843380451202,
+            0.001605947851203382,
+            0.015223781578242779,
+            0.001339128939434886,
+            -0.0020117140375077724,
+            0.009730332531034946,
+            -0.010793580673635006,
+            0.010414999909698963,
+            -0.020024508237838745,
+            -0.00027915299870073795,
+            0.01589233987033367,
+            -0.011083557270467281,
+            -0.024116402491927147,
+            -0.00019784874166361988,
+            -0.0164642371237278,
+            0.009182598441839218,
+            0.0017267714720219374,
+            -0.011325204744935036,
+            -0.0051672253757715225,
+            -0.0016885106451809406,
+            -0.005880084820091724,
+            0.022875946015119553,
+            -0.01215486042201519,
+            -0.019702311605215073,
+            -0.018735721707344055,
+            -0.008852346800267696,
+            0.011977652087807655,
+            0.03162357956171036,
+            -0.015473484061658382,
+            0.00608145771548152,
+            -0.006762097589671612,
+            0.00014549182378686965,
+            0.027773331850767136,
+            -0.0032642525620758533,
+            -0.0074467649683356285,
+            0.013540305197238922,
+            0.01762414537370205,
+            0.005259856581687927,
+            -0.007132623344659805,
+            -0.011800444684922695,
+            -0.012823417782783508,
+            -0.003695190418511629,
+            0.012815362773835659,
+            -0.006037155631929636,
+            -0.017994670197367668,
+            0.04298100247979164,
+            -0.023053154349327087,
+            0.0038663572631776333,
+            -0.00921481754630804,
+            -0.004893358331173658,
+            -0.0025393106043338776,
+            0.010431109927594662,
+            -0.006480175536125898,
+            -0.04336763918399811,
+            -0.043399859219789505,
+            0.007664247881621122,
+            -0.006645301356911659,
+            -0.0004505715623963624,
+            0.047395095229148865,
+            0.010278066620230675,
+            -0.02946486324071884,
+            0.01726972870528698,
+            -0.025195760652422905,
+            -0.006274775601923466,
+            -0.0050745937041938305,
+            -0.022714847698807716,
+            0.009343696758151054,
+            0.0008794955792836845,
+            -0.011292984709143639,
+            -0.012960351072251797,
+            -0.022231554612517357,
+            -0.009053720161318779,
+            -0.001134735532104969,
+            0.012581770308315754,
+            0.004325487185269594,
+            -0.0036267235409468412,
+            -0.026758413761854172,
+            0.017285838723182678,
+            -0.01168767549097538,
+            0.0012213258305564523,
+            -0.000624255568254739,
+            0.00898122601211071,
+            0.019637873396277428,
+            0.016238700598478317,
+            -0.02360088936984539,
+            -0.017124740406870842,
+            -0.026516767218708992,
+            0.017060300335288048,
+            0.00505042914301157,
+            0.026371778920292854,
+            -0.012436782009899616,
+            0.003979125991463661,
+            -0.020846109837293625,
+            -0.013693348504602909,
+            -0.008642919361591339,
+            -0.014031654223799706,
+            0.005827728193253279,
+            -0.003018577815964818,
+            0.03421726077795029,
+            -0.0012364288559183478,
+            -0.007434682920575142,
+            0.004824891686439514,
+            0.008618754334747791,
+            0.025904593989253044,
+            -0.021796587854623795,
+            -0.017189180478453636,
+            -0.011292984709143639,
+            0.010978843085467815,
+            0.005650519859045744,
+            -0.016883093863725662,
+            -0.02548573724925518,
+            -0.016423963010311127,
+            -0.002593681216239929,
+            -0.024148622527718544,
+            0.006230473518371582,
+            0.02970651164650917,
+            0.014442455023527145,
+            0.002138578798621893,
+            -0.01596483401954174,
+            0.03007703647017479,
+            0.010938568972051144,
+            0.011800444684922695,
+            -0.008022691123187542,
+            0.017930231988430023,
+            -0.020266154780983925,
+            0.005296103656291962,
+            -0.024019744247198105,
+            -0.017301948741078377,
+            -0.013250328600406647,
+            0.03511941060423851,
+            -0.010551933199167252,
+            -0.018735721707344055,
+            0.004748370032757521,
+            -0.005187362432479858,
+            0.01678643375635147,
+            -0.01514323242008686,
+            -0.021103866398334503,
+            0.025195760652422905,
+            0.007535369135439396,
+            0.012074311263859272,
+            -0.03136582300066948,
+            0.0022211417090147734,
+            -0.019122358411550522,
+            0.0024205006193369627,
+            -0.012017927132546902,
+            0.0015173438005149364,
+            0.004639628808945417,
+            0.00766827492043376,
+            0.004305349662899971,
+            0.024841345846652985,
+            0.004365761764347553,
+            0.021168306469917297,
+            -0.016512567177414894,
+            -0.0013491975842043757,
+            0.03724590688943863,
+            0.008026719093322754,
+            -0.001129701267927885,
+            0.007712577003985643,
+            0.018864601850509644,
+            0.006721823010593653,
+            -0.009553124196827412,
+            -0.00437784381210804,
+            -0.0011729963589459658,
+            0.023278692737221718,
+            0.01315366942435503,
+            0.0064600384794175625,
+            -0.0015102957841008902,
+            0.008747633546590805,
+            0.006681548431515694,
+            0.001100502209737897,
+            -0.003042742609977722,
+            -0.013894720934331417,
+            -0.020395034924149513,
+            -0.025082992389798164,
+            0.025421299040317535,
+            0.0031353740487247705,
+            -0.006335187237709761,
+            0.009657838381826878,
+            0.006315050181001425,
+            -0.012066256254911423,
+            0.02878825180232525,
+            -0.005718986503779888,
+            -0.024873564019799232,
+            -0.005598163232207298,
+            -0.003679080633446574,
+            -0.011913212947547436,
+            0.030624771490693092,
+            -0.00027487383340485394,
+            0.0059606339782476425,
+            0.01267842948436737,
+            -0.005489421542733908,
+            -0.021232744678854942,
+            -0.02102331817150116,
+            -0.025936812162399292,
+            -0.007619945798069239,
+            0.015014354139566422,
+            -7.803194603184238e-05,
+            0.028852690011262894,
+            0.0065003130584955215,
+            -0.008497931063175201,
+            -0.0058760573156178,
+            -0.023182032629847527,
+            -0.004478530492633581,
+            0.008489876054227352,
+            -0.017898011952638626,
+            0.002058029640465975,
+            -0.015344604849815369,
+            0.011349369771778584,
+            0.0006912120152264833,
+            -0.01327449269592762,
+            -0.002196976915001869,
+            -0.0005774364108219743,
+            -0.01473243162035942,
+            -0.011003008112311363,
+            -0.005553861148655415,
+            0.01256566122174263,
+            -0.01975064165890217,
+            -0.006447955965995789,
+            -0.011719895526766777,
+            0.019879519939422607,
+            -0.010414999909698963,
+            0.009448410011827946,
+            -0.03357286751270294,
+            0.00329043110832572,
+            -0.0003179172636009753,
+            -0.01622259058058262,
+            0.004114045761525631,
+            -0.0016613253392279148,
+            0.02360088936984539,
+            0.01420886255800724,
+            0.014812980778515339,
+            0.004994044546037912,
+            0.017173070460557938,
+            -0.0015143232885748148,
+            -0.002843383466824889,
+            -0.004998072050511837,
+            -0.00487724831327796,
+            -0.010672756470739841,
+            0.006588917225599289,
+            -0.003246129024773836,
+            0.019637873396277428,
+            0.004945715423673391,
+            -0.012928131967782974,
+            0.0066050267778337,
+            0.01656089723110199,
+            0.013258383609354496,
+            -0.00032295158598572016,
+            -0.00684264674782753,
+            0.01562652736902237,
+            0.06869228184223175,
+            -0.0011226532515138388,
+            0.01809133030474186,
+            0.016246754676103592,
+            -0.03191355615854263,
+            -0.022730957716703415,
+            0.01156685221940279,
+            -0.01575540564954281,
+            0.03418504074215889,
+            0.01109161227941513,
+            0.011204380542039871,
+            -0.014966024085879326,
+            0.00651642307639122,
+            0.01913846842944622,
+            0.0170119721442461,
+            -0.005863975267857313,
+            -0.006041183136403561,
+            -0.019170688465237617,
+            0.023101484403014183,
+            0.010769415646791458,
+            -0.006443928461521864,
+            0.004212718456983566,
+            0.008203926496207714,
+            0.003908645361661911,
+            -0.00013315775140654296,
+            0.01633535884320736,
+            -0.018477965146303177,
+            0.001423705485649407,
+            -0.003340774215757847,
+            0.014925749972462654,
+            -0.006298940163105726,
+            -0.01962176337838173,
+            0.0076400828547775745,
+            -0.0019301580032333732,
+            0.011204380542039871,
+            0.013645019382238388,
+            -0.0025232008192688227,
+            0.00663724634796381,
+            -0.003495831275358796,
+            0.0004875738231930882,
+            0.00816767942160368,
+            -0.01668977551162243,
+            0.003558256896212697,
+            0.0003438440035097301,
+            -0.022022126242518425,
+            0.010914403945207596,
+            0.014837145805358887,
+            0.012082366272807121,
+            0.0018032931257039309,
+            -0.008127405308187008,
+            -0.022505421191453934,
+            -0.008949005976319313,
+            0.003348829224705696,
+            0.0002846907591447234,
+            -0.025340748950839043,
+            -0.001157893449999392,
+            -0.012960351072251797,
+            -0.01726972870528698,
+            0.0056666298769414425,
+            0.012726759538054466,
+            0.004663793370127678,
+            -0.004430200904607773,
+            -0.0016019203467294574,
+            -0.0048329466953873634,
+            -0.01509490329772234,
+            0.0019784874748438597,
+            0.012718704529106617,
+            -0.003324664430692792,
+            0.014224972575902939,
+            0.022151004523038864,
+            0.004055647645145655,
+            0.0039026043377816677,
+            0.0030447563622146845,
+            4.613481451087864e-06,
+            0.01581178978085518,
+            0.01925123669207096,
+            -0.0019513021688908339,
+            0.0013199985260143876,
+            -0.00045661276089958847,
+            0.006464065983891487,
+            0.006472120992839336,
+            0.001662332215346396,
+            -0.011462138034403324,
+            0.003439446911215782,
+            -0.01902570016682148,
+            -0.013201998546719551,
+            0.005622327793389559,
+            0.0047805896028876305,
+            0.02147439308464527,
+            -0.01891293004155159,
+            0.0022553750313818455,
+            0.006544615142047405,
+            0.0072051179595291615,
+            -0.0009097015135921538,
+            -0.008409326896071434,
+            0.018268538638949394,
+            0.006149924360215664,
+            0.0034172958694398403,
+            -0.004168416373431683,
+            -0.011333259753882885,
+            0.007950196973979473,
+            0.00033704767702147365,
+            0.0019331786315888166,
+            -0.006383516825735569,
+            -0.013943050988018513,
+            -0.022618189454078674,
+            0.023649217560887337,
+            0.01726972870528698,
+            0.019798971712589264,
+            -0.01797856017947197,
+            0.006258665584027767,
+            -0.011019118130207062,
+            0.010414999909698963,
+            0.01167962048202753,
+            -0.0017408676212653518,
+            -0.004293267615139484,
+            -0.001821416663005948,
+            -0.021055536344647408,
+            -0.01749526709318161,
+            0.005626355297863483,
+            0.02055613324046135,
+            0.0001618533715372905,
+            -0.009826990775763988,
+            -0.029674291610717773,
+            -0.004389926325529814,
+            -0.005678711924701929,
+            -0.0199439600110054,
+            -0.01150241307914257,
+            -0.015392934903502464,
+            -0.015642637386918068,
+            -0.003228005487471819,
+            0.0022271827328950167,
+            -0.005578025709837675,
+            0.004023428075015545,
+            0.018574625253677368,
+            0.0013532249722629786,
+            0.020411144942045212,
+            0.010068638250231743,
+            0.017189180478453636,
+            0.013596689328551292,
+            -0.013411426916718483,
+            0.006282830610871315,
+            -0.009593399241566658,
+            -0.012050146237015724,
+            0.018026890233159065,
+            -0.014386070892214775,
+            -2.73583791567944e-05,
+            -0.023182032629847527,
+            0.001140776788815856,
+            -0.02395530417561531,
+            0.01892904005944729,
+            -0.034475017338991165,
+            -0.011961543001234531,
+            0.005743151530623436,
+            -0.01714085042476654,
+            -0.01949288509786129,
+            -0.028369395062327385,
+            -0.0008281455375254154,
+            -0.018059110268950462,
+            -0.00408383971080184,
+            -0.016770323738455772,
+            0.0012011885410174727,
+            -0.0028836580459028482,
+            0.010785525664687157,
+            -0.008497931063175201,
+            0.051390331238508224,
+            0.0029923992697149515,
+            0.005537751130759716,
+            -0.01263010036200285,
+            0.001191119896247983,
+            -0.012066256254911423,
+            0.013427536003291607,
+            -0.0016885106451809406,
+            -0.0038200414273887873,
+            0.008409326896071434,
+            0.010704976506531239,
+            -0.0032561977859586477,
+            0.004293267615139484,
+            0.02395530417561531,
+            0.02772500365972519,
+            -0.001674414612352848,
+            -0.0120340371504426,
+            0.004990017041563988,
+            0.009061774238944054,
+            0.014861310832202435,
+            0.008264338597655296,
+            -0.016625335440039635,
+            -0.01267842948436737,
+            0.013572524301707745,
+            0.005328323692083359,
+            -0.011832663789391518,
+            0.01868739351630211,
+            -0.013065065257251263,
+            -0.022714847698807716,
+            0.005392762832343578,
+            -0.00798241700977087,
+            0.012815362773835659,
+            -0.00915843341499567,
+            0.006625164300203323,
+            0.009770606644451618,
+            0.007088321726769209,
+            0.025276310741901398,
+            -0.004192580934613943,
+            -0.004216745961457491,
+            -0.015038518235087395,
+            -0.0017388538690283895,
+            -0.0009162460919469595,
+            0.027563905343413353,
+            0.02875603176653385,
+            0.008779852651059628,
+            -0.059960756450891495,
+            -0.003125305287539959,
+            0.016512567177414894,
+            -0.010527768172323704,
+            -0.0182202085852623,
+            0.015320440754294395,
+            -0.0021828808821737766,
+            0.002150661079213023,
+            0.01689920201897621,
+            0.009907539933919907,
+            -0.005340405739843845,
+            0.014845200814306736,
+            0.03531273081898689,
+            0.009432300925254822,
+            0.008203926496207714,
+            -0.007656192872673273,
+            -0.01865517348051071,
+            -0.029303764924407005,
+            -0.004961824975907803,
+            -0.004389926325529814,
+            0.001964391442015767,
+            0.004196608439087868,
+            -0.00904566515237093,
+            0.037890300154685974,
+            0.018735721707344055,
+            0.01904180832207203,
+            0.006238528527319431,
+            0.008393216878175735,
+            -0.02279539778828621,
+            -0.03805140033364296,
+            0.003969057463109493,
+            -0.024019744247198105,
+            0.0018687392584979534,
+            0.010729141533374786,
+            0.01962176337838173,
+            0.005022237077355385,
+            0.011889047920703888,
+            -0.008360997773706913,
+            0.004212718456983566,
+            -2.6917876311927103e-05,
+            0.005783426109701395,
+            -0.01599705219268799,
+            -0.01950899325311184,
+            -0.009013445116579533,
+            0.0043738167732954025,
+            0.010278066620230675,
+            -0.005408872850239277,
+            -0.005300131160765886,
+            0.005936469417065382,
+            -0.02443859912455082,
+            0.0029541384428739548,
+            -0.011993762105703354,
+            0.0037092864513397217,
+            -0.009464520029723644,
+            0.016947532072663307,
+            0.020233936607837677,
+            -0.00956923421472311,
+            -0.0017650322988629341,
+            0.0009832025971263647,
+            0.02300482615828514,
+            -0.009899485856294632,
+            0.01656089723110199,
+            0.005771343596279621,
+            0.029046008363366127,
+            0.01962176337838173,
+            0.015747351571917534,
+            -0.010382779873907566,
+            0.012936186976730824,
+            0.022618189454078674,
+            -0.007495094556361437,
+            0.013572524301707745,
+            -0.00857042521238327,
+            0.012710649520158768,
+            -0.006133814807981253,
+            -0.010704976506531239,
+            0.0034354194067418575,
+            0.01327449269592762,
+            0.011397698894143105,
+            0.003228005487471819,
+            0.02429361082613468,
+            -0.0053202686831355095,
+            -0.010374724864959717,
+            -0.019090138375759125,
+            0.0036468608304858208,
+            -0.0017569774063304067,
+            0.0043738167732954025,
+            0.01420886255800724,
+            -0.02337535098195076,
+            -0.000712356180883944,
+            0.01533654984086752,
+            -0.011599071323871613,
+            -0.001532446825876832,
+            -0.022618189454078674,
+            -0.01599705219268799,
+            0.009851155802607536,
+            -0.0015102957841008902,
+            -0.005968688987195492,
+            -0.024615807458758354,
+            0.01846185512840748,
+            -0.0029138640966266394,
+            0.002593681216239929,
+            0.036214880645275116,
+            0.008892621845006943,
+            0.039050206542015076,
+            -0.0068063996732234955,
+            0.012001817114651203,
+            -0.011292984709143639,
+            -0.011019118130207062,
+            -0.026001252233982086,
+            0.013604744337499142,
+            -0.01015724241733551,
+            0.019315676763653755,
+            0.009311476722359657,
+            0.006472120992839336,
+            -0.0026118047535419464,
+            0.0050745937041938305,
+            0.0008633857360109687,
+            -0.015779569745063782,
+            0.026001252233982086,
+            -0.02195768617093563,
+            0.028240516781806946,
+            0.005638437811285257,
+            -0.0120340371504426,
+            0.012823417782783508,
+            0.011139941401779652,
+            0.004776562098413706,
+            0.01051165908575058,
+            0.0030447563622146845,
+            -0.0054612294770777225,
+            0.005437064915895462,
+            -0.00634324224665761,
+            -0.04262658953666687,
+            -0.025807933881878853,
+            -0.006109649781137705,
+            -0.003950933925807476,
+            0.0075998082756996155,
+            -0.013419480994343758,
+            -0.021699929609894753,
+            0.0030064955353736877,
+            0.023745877668261528,
+            -0.021699929609894753,
+            0.011429918929934502,
+            0.0018697461346164346,
+            0.011655456386506557,
+            -0.01784968189895153,
+            0.013363096863031387,
+            -0.002575557678937912,
+            0.02996426820755005,
+            0.000515262596309185,
+            -0.0031635661143809557,
+            0.012517331168055534,
+            0.000442516669863835,
+            -0.021571051329374313,
+            -0.00028745963936671615,
+            -0.012452892027795315,
+            -0.02020171657204628,
+            0.02182880789041519,
+            -0.008115322329103947,
+            -0.006556697189807892,
+            -0.004216745961457491,
+            0.024116402491927147,
+            -0.0006927223294042051,
+            0.0011951474007219076,
+            -0.005863975267857313,
+            -0.011736005544662476,
+            0.0182040985673666,
+            -0.02688729204237461,
+            -0.0030306600965559483,
+            -0.007285666652023792,
+            0.0007954224711284041,
+            -0.005739124026149511,
+            -0.005275966599583626,
+            0.017189180478453636,
+            -0.017930231988430023,
+            0.005586080718785524,
+            -0.014426345005631447,
+            -0.003695190418511629,
+            0.0008608686039224267,
+            0.011019118130207062,
+            0.03157525137066841,
+            0.004091894719749689,
+            0.030850308015942574,
+            0.014329686760902405,
+            -0.008594590239226818,
+            0.0012535455171018839,
+            0.01878405176103115,
+            -0.010535823181271553,
+            0.02160327136516571,
+            -0.01308923028409481,
+            -0.008650974370539188,
+            0.007241365034133196,
+            -0.007632027845829725,
+            0.00458324421197176,
+            0.011260765604674816,
+            0.015409043990075588,
+            0.0014871378662064672,
+            0.0009711202001199126,
+            -0.016029272228479385,
+            0.010737196542322636,
+            -0.0046799033880233765,
+            -0.008916785940527916,
+            0.00578745361417532,
+            0.007269557099789381,
+            -0.014007490128278732,
+            0.0009691064478829503,
+            -0.007877702824771404,
+            0.0050141820684075356,
+            -0.01889682002365589,
+            -0.008940950967371464,
+            -0.024470819160342216,
+            -0.00391670037060976,
+            -0.013612799346446991,
+            0.005650519859045744,
+            0.001747915637679398,
+            -0.005952578969299793,
+            0.016174260526895523,
+            0.015844009816646576,
+            0.005924386903643608,
+            0.006045210640877485,
+            -0.02666175551712513,
+            -0.0038764257915318012,
+            0.0007294728420674801,
+            -0.0013038887409493327,
+            0.0016653528437018394,
+            -0.011244655586779118,
+            -0.011421863920986652,
+            -0.018703503534197807,
+            -0.010189462453126907,
+            -0.0027084636967629194,
+            0.014514949172735214,
+            -0.0037233824841678143,
+            -0.0014196779811754823,
+            -0.0058760573156178,
+            0.006971525494009256,
+            0.012509276159107685,
+            0.022618189454078674,
+            -0.014498839154839516,
+            -0.0132100535556674,
+            0.015417098999023438,
+            0.03197799623012543,
+            -0.007998526096343994,
+            0.008066993206739426,
+            -0.010141132399439812,
+            -0.00607340270653367,
+            -0.005578025709837675,
+            -0.01809133030474186,
+            0.010616372339427471,
+            0.00018186478700954467,
+            0.02548573724925518,
+            0.00578745361417532,
+            -0.004013359546661377,
+            0.007813263684511185,
+            0.010350560769438744,
+            0.012082366272807121,
+            0.00754745164886117,
+            0.030512001365423203,
+            0.0005930428160354495,
+            0.02360088936984539,
+            -0.0012686484260484576,
+            -0.010938568972051144,
+            -0.0021869083866477013,
+            0.0017297921003773808,
+            0.00484502874314785,
+            0.010076693259179592,
+            0.0281760785728693,
+            0.0035220098216086626,
+            -0.004772534593939781,
+            0.01633535884320736,
+            0.006391571834683418,
+            -0.0005880084936507046,
+            -0.012718704529106617,
+            -0.01227568369358778,
+            -0.006335187237709761,
+            -0.0007934087188914418,
+            0.002002652268856764,
+            -0.016440073028206825,
+            -0.03550604730844498,
+            0.010020309127867222,
+            0.006089512724429369,
+            -0.016995862126350403,
+            -0.01773691363632679,
+            -0.006178116425871849,
+            0.019122358411550522,
+            0.00040048008668236434,
+            -0.009061774238944054,
+            -0.0024305693805217743,
+            0.007575643714517355,
+            0.022263772785663605,
+            -0.0014569320483133197,
+            0.031075846403837204,
+            -0.015860119834542274,
+            -0.01479687076061964,
+            0.012960351072251797,
+            0.0024245281238108873,
+            0.008288503624498844,
+            -0.025195760652422905,
+            0.010125023312866688,
+            -0.006931250914931297,
+            0.022747067734599113,
+            0.007559533696621656,
+            0.010390834882855415,
+            -0.012251519598066807,
+            -0.006242556031793356,
+            0.016367578878998756,
+            -0.010906348936259747,
+            -0.004708095453679562,
+            -0.003236060496419668,
+            -0.026242898777127266,
+            0.0012092434335500002,
+            0.011526577174663544,
+            -0.0015515772392973304,
+            0.015803735703229904,
+            -0.006886948831379414,
+            -0.002825259929522872,
+            -0.0003078486188314855,
+            -0.00019105241517536342,
+            -0.008272393606603146,
+            -0.041982196271419525,
+            -0.002799081616103649,
+            0.02078166976571083,
+            -0.005102786235511303,
+            -0.02210267446935177,
+            0.01725361868739128,
+            0.003308554645627737,
+            0.0036851216573268175,
+            -0.01328254770487547,
+            -0.021571051329374313,
+            -0.015827899798750877,
+            0.01308923028409481,
+            -0.018252428621053696,
+            -0.023536449298262596,
+            -0.0016804557526484132,
+            -0.017946342006325722,
+            0.002428555628284812,
+            -0.02197379618883133,
+            -0.02055613324046135,
+            -0.021780479699373245,
+            -0.005497476551681757,
+            0.004192580934613943,
+            -0.025662945583462715,
+            0.026371778920292854,
+            -0.008441546931862831,
+            -0.006149924360215664,
+            -0.014861310832202435,
+            0.008892621845006943,
+            0.0017126754391938448,
+            0.021764369681477547,
+            -0.02679063379764557,
+            0.009585344232618809,
+            -0.0023681437596678734,
+            -0.015417098999023438,
+            -0.005944523960351944,
+            -0.0015868174377828836,
+            0.009673947468400002,
+            0.03499053418636322,
+            0.006524477619677782,
+            -0.021877137944102287,
+            -0.023343130946159363,
+            0.012847582809627056,
+            0.003991208504885435,
+            -0.010801635682582855,
+            -0.010737196542322636,
+            -0.01715696044266224,
+            -0.015425154007971287,
+            0.014579388312995434,
+            0.01367723848670721,
+            0.018107440322637558,
+            -0.0032380742486566305,
+            0.008199899457395077,
+            -0.01880016177892685,
+            -0.005747179035097361,
+            0.0033790350425988436,
+            0.006262693088501692,
+            -0.018397416919469833,
+            0.035892684012651443,
+            0.005332351196557283,
+            -0.004454365465790033,
+            -0.00581161817535758,
+            0.0052276370115578175,
+            0.0028333149384707212,
+            -0.008844291791319847,
+            -0.01316172443330288,
+            0.00737427081912756,
+            -0.009085939265787601,
+            0.007003745064139366,
+            0.009134269319474697,
+            -0.005541778635233641,
+            -0.019895629957318306,
+            -0.003364939009770751,
+            0.011727950535714626,
+            -0.017237508669495583,
+            -0.01110772229731083,
+            -0.020523913204669952,
+            -0.003024619072675705,
+            0.003979125991463661,
+            -0.00014410738367587328,
+            0.0018687392584979534,
+            -0.01639174297451973,
+            -0.004176471382379532,
+            -0.02911044843494892,
+            0.0008155597024597228,
+            -0.02182880789041519,
+            -0.018848491832613945,
+            0.011921267956495285,
+            0.00975449662655592,
+            0.00604923814535141,
+            0.003705258946865797,
+            0.00047549145529046655,
+            0.014120258390903473,
+            0.002132537541911006,
+            -0.016528677195310593,
+            0.022408761084079742,
+            -0.0063593522645533085,
+            0.015594307333230972,
+            0.029400425031781197,
+            0.01380611676722765,
+            -0.00687486631795764,
+            -0.017801353707909584,
+            -0.01003641914576292,
+            -0.02210267446935177,
+            0.0047685070894658566,
+            -0.01772080361843109,
+            -0.024277500808238983,
+            0.014635773375630379,
+            -0.023069264367222786,
+            0.0011488316813483834,
+            -0.01693142205476761,
+            0.006737933028489351,
+            -0.00637546181678772,
+            0.012597880326211452,
+            0.017833571881055832,
+            0.0016703871078789234,
+            0.0027789443265646696,
+            -0.011880993843078613,
+            -0.0036106137558817863,
+            0.006689603440463543,
+            -0.005175280384719372,
+            -0.004905440844595432,
+            0.001707641058601439,
+            -0.010922458954155445,
+            -0.019766751676797867,
+            -0.012775088660418987,
+            -0.025646835565567017,
+            -0.0010632482590153813,
+            0.0049215503968298435,
+            0.006246583536267281,
+            -0.004273130092769861,
+            -0.00355221563950181,
+            0.0033448017202317715,
+            -0.008594590239226818,
+            0.0031917584128677845,
+            -0.0010144152911379933,
+            -0.005900222342461348,
+            0.006170061882585287,
+            0.0034515291918069124,
+            0.0030387151055037975,
+            0.01427330169826746,
+            0.00243862415663898,
+            -0.005171252880245447,
+            -0.038244716823101044,
+            0.0062546380795538425,
+            -0.018848491832613945,
+            0.001827457919716835,
+            -0.011446028016507626,
+            7.104683027137071e-05,
+            -0.006717795506119728,
+            -0.006967497989535332,
+            0.016770323738455772,
+            0.003882467048242688,
+            0.010116968303918839,
+            0.015940668061375618,
+            -0.010600262321531773,
+            -0.008949005976319313,
+            0.012501221150159836,
+            -9.49441164266318e-06,
+            -0.00481280917301774,
+            0.0016542773228138685,
+            -0.01566680148243904,
+            0.008916785940527916,
+            -0.00951285008341074,
+            0.009593399241566658,
+            -0.012944241985678673,
+            0.007386353332549334,
+            -0.008731523528695107,
+            -0.011405753903090954,
+            0.00024882121942937374,
+            0.008763742633163929,
+            0.01581178978085518,
+            0.003097113221883774,
+            0.006025073118507862,
+            0.0035602706484496593,
+            0.02147439308464527,
+            -0.01255760621279478,
+            0.0009464520262554288,
+            -0.005465256981551647,
+            -0.023455901071429253,
+            -0.015844009816646576,
+            0.025324638932943344,
+            0.03937240317463875,
+            -0.011711840517818928,
+            -0.005429009906947613,
+            0.0011880993843078613,
+            0.009085939265787601,
+            0.0023983498103916645,
+            -0.004490613006055355,
+            -0.01309728529304266,
+            -0.02608180046081543,
+            -0.02018560655415058,
+            0.004555052146315575,
+            0.01760803535580635,
+            -0.010721086524426937,
+            -0.004013359546661377,
+            -0.005940496921539307,
+            -0.00431340467184782,
+            -0.0028534522280097008,
+            -0.008642919361591339,
+            -0.00551761407405138,
+            -0.005723014008253813,
+            -0.004514777567237616,
+            -0.022151004523038864,
+            0.009424245916306973,
+            0.011663510464131832,
+            0.008369052782654762,
+            0.0028172051534056664,
+            0.020829999819397926,
+            -0.02453525923192501,
+            -0.0067057134583592415,
+            -0.000449816434411332,
+            -0.009786716662347317,
+            0.02500244230031967,
+            0.00798241700977087,
+            -0.016947532072663307,
+            0.0012626072857528925,
+            0.006363379769027233,
+            0.00024693337036296725,
+            0.018945150077342987,
+            -0.010439164005219936,
+            0.021554941311478615,
+            -0.011486303061246872,
+            -0.004454365465790033,
+            -0.0022734985686838627,
+            0.006882921326905489,
+            0.03228408098220825,
+            0.012324013747274876,
+            -0.00950479507446289,
+            0.01068886648863554,
+            0.017688583582639694,
+            0.011437973007559776,
+            -0.009947814978659153,
+            0.021989906206727028,
+            -0.004196608439087868,
+            0.015296275727450848,
+            0.016528677195310593,
+            0.008344887755811214,
+            -0.011454083025455475,
+            -0.008465711027383804,
+            0.022167114540934563,
+            0.003783794352784753,
+            -0.02664564549922943,
+            -0.010012254118919373,
+            0.0029319876339286566,
+            0.023552559316158295,
+            0.007712577003985643,
+            0.0028756032697856426,
+            0.001929151127114892,
+            0.024631917476654053,
+            -0.005662602372467518,
+            0.027515575289726257,
+            -0.016754213720560074,
+            0.004542969632893801,
+            -0.001554597751237452,
+            0.014200807549059391,
+            -0.01831686682999134,
+            0.001941233524121344,
+            -0.0036448470782488585,
+            0.004631573799997568,
+            0.008892621845006943,
+            0.007120541296899319,
+            0.019283456727862358,
+            -0.004933632910251617,
+            -0.013258383609354496,
+            0.008457656018435955,
+            0.008868456818163395,
+            -0.01736638695001602,
+            0.027547795325517654,
+            0.020378924906253815,
+            0.00798241700977087,
+            -0.016866983845829964,
+            0.005130978301167488,
+            0.04043565317988396,
+            0.018606843426823616,
+            0.013016736134886742,
+            -0.010358615778386593,
+            -0.022634299471974373,
+            0.007587726227939129,
+            0.009359806776046753,
+            0.0024144595954567194,
+            0.004369789268821478,
+            0.004164388868957758,
+            -0.014321631751954556,
+            0.0001999883388634771,
+            0.03592490404844284,
+            -0.010785525664687157,
+            -0.01599705219268799,
+            -0.004301322158426046,
+            -0.011727950535714626,
+            -0.018477965146303177,
+            0.010414999909698963,
+            0.004865166265517473,
+            -0.020362814888358116,
+            0.00950479507446289,
+            0.001946267788298428,
+            -0.0164481271058321,
+            0.0041281417943537235,
+            0.0032199507113546133,
+            -0.003926768898963928,
+            -0.004841001238673925,
+            0.001975466962903738,
+            -0.009118159301578999,
+            0.005702876951545477,
+            -0.0011528590694069862,
+            -0.003743519773706794,
+            0.010052529163658619,
+            0.01654478721320629,
+            0.0024547341745346785,
+            0.010431109927594662,
+            0.0019734532106667757,
+            0.013298657722771168,
+            -0.018300756812095642,
+            0.00657280720770359,
+            -0.005972716491669416,
+            0.001719723455607891,
+            -0.0024104320909827948,
+            -0.005429009906947613,
+            -0.03685927018523216,
+            0.0029984405264258385,
+            -0.0011961542768403888,
+            0.014450510032474995,
+            -0.01736638695001602,
+            0.00032269986695609987,
+            0.01309728529304266,
+            -0.014361905865371227,
+            -0.00022830638044979423,
+            0.01846185512840748,
+            0.001276703318580985,
+            0.02279539778828621,
+            0.012315958738327026,
+            0.013371151871979237,
+            0.002072125906124711,
+            -0.02326258271932602,
+            0.02442248910665512,
+            -0.009915594942867756,
+            0.004458392970263958,
+            0.0211360864341259,
+            0.03312179446220398,
+            -0.021168306469917297,
+            -0.008546260185539722,
+            -0.015344604849815369,
+            -0.002984344493597746,
+            0.021667709574103355,
+            0.0040757847018539906,
+            0.028192188590765,
+            0.013226163573563099,
+            0.01648840308189392,
+            -0.025936812162399292,
+            0.0021244827657938004,
+            0.014378015883266926,
+            -0.007696467451751232,
+            -0.017930231988430023,
+            -0.01736638695001602,
+            -0.013234218582510948,
+            0.010648592375218868,
+            -0.02149050123989582,
+            0.026484547182917595,
+            -0.0009686030098237097,
+            0.01678643375635147,
+            0.009013445116579533,
+            -0.010471384041011333,
+            0.004929605405777693,
+            0.00627880310639739,
+            0.0037636570632457733,
+            -0.01726972870528698,
+            0.00510681327432394,
+            -0.007712577003985643,
+            0.0061539518646895885,
+            -0.004043565131723881,
+            0.029335984960198402,
+            -0.008010609075427055,
+            0.008489876054227352,
+            -0.011969597078859806,
+            -0.011309094727039337,
+            0.011317149735987186,
+            0.000714873312972486,
+            -0.012404562905430794,
+            0.0065123955719172955,
+            -0.002897754078730941,
+            0.01455522421747446,
+            -0.015932613983750343,
+            0.00405162014067173,
+            0.030334794893860817,
+            -0.01566680148243904,
+            -0.009327586740255356,
+            0.0030326738487929106,
+            0.0057673160918056965,
+            0.0034656254574656487,
+            0.0019684189464896917,
+            0.012847582809627056,
+            -0.00378983560949564,
+            -0.010946623980998993,
+            0.012348177842795849,
+            -0.011486303061246872,
+            0.008530150167644024,
+            0.0016834763810038567,
+            -0.008804017677903175,
+            0.029046008363366127,
+            -0.010133077390491962,
+            0.01257371623069048,
+            0.021087756380438805,
+            -0.023069264367222786,
+            -0.0077528515830636024,
+            -0.0105599882081151,
+            -0.019074028357863426,
+            -0.011518522165715694,
+            -0.005022237077355385,
+            0.00107231002766639,
+            0.0010400903411209583,
+            0.0060129910707473755,
+            3.240842852392234e-05,
+            0.004784617107361555,
+            0.0011589003261178732,
+            -0.0033689665142446756,
+            -0.010914403945207596,
+            0.03006092645227909,
+            -0.018977370113134384,
+            0.024728575721383095,
+            -0.00021584643400274217,
+            0.016947532072663307,
+            0.0017348263645544648,
+            -0.008034773170948029,
+            0.014136368408799171,
+            0.018300756812095642,
+            -0.019589543342590332,
+            -0.008123377338051796,
+            0.004619491286575794,
+            -0.0026601343415677547,
+            -0.009931704960763454,
+            -0.011760169640183449,
+            0.0038280964363366365,
+            -0.028949350118637085,
+            -0.0044221458956599236,
+            -0.011325204744935036,
+            -0.0035441608633846045,
+            -0.00777701660990715,
+            -0.01833297684788704,
+            -0.009118159301578999,
+            -0.01216291543096304,
+            -0.012009872123599052,
+            -0.008280448615550995,
+            -0.018945150077342987,
+            0.006186171434819698,
+            -0.02395530417561531,
+            0.014990189112722874,
+            -0.014200807549059391,
+            -0.005147087853401899,
+            0.015650691464543343,
+            0.03254184126853943,
+            0.006391571834683418,
+            0.0010622413828969002,
+            0.0016713939839974046,
+            0.004269102588295937,
+            0.0031696073710918427,
+            -0.0152721107006073,
+            -0.016641445457935333,
+            -0.0032380742486566305,
+            -0.001123660011216998,
+            0.02020171657204628,
+            0.01846185512840748,
+            -0.0036972041707485914,
+            -0.005384707823395729,
+            0.00869930349290371,
+            -0.002859493251889944,
+            -0.008042828179895878,
+            0.005324296187609434,
+            0.003431391902267933,
+            0.029642071574926376,
+            -0.011808499693870544,
+            -0.006077430211007595,
+            0.016085656359791756,
+            0.008457656018435955,
+            0.006798344664275646,
+            -0.010841909795999527,
+            0.00798241700977087,
+            0.006484203040599823,
+            0.017527485266327858,
+            0.0111641064286232,
+            0.0035723529290407896,
+            -0.010173352435231209,
+            -0.010817745700478554,
+            0.0006177109898999333,
+            0.006979580037295818,
+            -0.004446310922503471,
+            -0.022022126242518425,
+            0.0016673665959388018,
+            -0.0008487862069159746,
+            -0.021748259663581848,
+            -0.017060300335288048,
+            0.032815705984830856,
+            0.012879802845418453,
+            0.009867265820503235,
+            -0.0027708893176168203,
+            0.022505421191453934,
+            0.008449601009488106,
+            0.0018314853077754378,
+            0.008812072686851025,
+            0.0012213258305564523,
+            0.006395599339157343,
+            0.003228005487471819,
+            -0.03595712035894394,
+            0.004800726659595966,
+            -0.0024023770820349455,
+            0.006762097589671612,
+            -0.0031514838337898254,
+            0.004269102588295937,
+            -0.0044745029881596565,
+            -0.010390834882855415,
+            -0.02092665806412697,
+            0.022054346278309822,
+            0.015634581446647644,
+            0.021587161347270012,
+            -0.005437064915895462,
+            -0.0020902494434267282,
+            0.007913949899375439,
+            0.025195760652422905,
+            0.0005698849563486874,
+            0.01633535884320736,
+            0.02372976765036583,
+            0.014893529936671257,
+            0.020169496536254883,
+            -0.0031837034039199352,
+            0.04001679643988609,
+            -0.00114581105299294,
+            -0.0026359695475548506,
+            -0.0025513931177556515,
+            -0.010495549067854881,
+            0.01315366942435503,
+            -7.123561954358593e-05,
+            -0.005038346629589796,
+            0.006029100622981787,
+            0.0003780773840844631,
+            0.011639346368610859,
+            -0.01068081147968769,
+            -0.010978843085467815,
+            0.0021808671299368143,
+            0.019412335008382797,
+            0.01227568369358778,
+            0.024116402491927147,
+            -0.021345512941479683,
+            0.005904249846935272,
+            -0.027209488674998283,
+            -0.012968406081199646,
+            0.0032481427770107985,
+            0.004204663448035717,
+            -0.01456327922642231,
+            -0.021877137944102287,
+            -0.009802826680243015,
+            0.011429918929934502,
+            0.007031937129795551,
+            0.010382779873907566,
+            0.0071004037745296955,
+            -0.0003302513505332172,
+            0.006717795506119728,
+            -0.01997617818415165,
+            -0.020475583150982857,
+            0.0019543226808309555,
+            0.0013421495677903295,
+            -0.022022126242518425,
+            -0.006991662550717592,
+            0.010302230715751648,
+            -6.689351721433923e-05,
+            -0.016472293063998222,
+            0.016254810616374016,
+            -0.01598094217479229,
+            -0.01109161227941513,
+            -0.0014670006930828094,
+            0.007688412442803383,
+            0.0024587614461779594,
+            0.015127122402191162,
+            -0.009553124196827412,
+            0.009609508328139782,
+            0.011760169640183449,
+            -0.013234218582510948,
+            0.024454709142446518,
+            -0.0023399516940116882,
+            -0.015827899798750877,
+            0.016303138807415962,
+            -0.010060583241283894,
+            -0.02550184726715088,
+            0.025775713846087456,
+            -0.013983325101435184,
+            -0.010294175706803799,
+            -0.024631917476654053,
+            -0.019992288202047348,
+            -0.007845482788980007,
+            -0.01162323635071516,
+            -0.034926094114780426,
+            -0.018236318603157997,
+            0.029545413330197334,
+            0.018590733408927917,
+            0.0211360864341259,
+            0.00325821153819561,
+            0.003961002454161644,
+            0.013814171776175499,
+            -0.01566680148243904,
+            0.0010541864903643727,
+            0.0017771146958693862,
+            9.546327783027664e-05,
+            0.016963642090559006,
+            -0.001924116862937808,
+            -0.0028051226399838924,
+            -0.017914121970534325,
+            -0.020636681467294693,
+            0.0041885534301400185,
+            0.002146633807569742,
+            0.019235126674175262,
+            0.014659937471151352,
+            -0.02490578405559063,
+            0.005888139829039574,
+            -0.013403371907770634,
+            0.03428170084953308,
+            -0.01416053343564272,
+            0.022376542910933495,
+            0.00863486435264349,
+            0.02195768617093563,
+            0.0026681891176849604,
+            -0.00037128105759620667,
+            0.008779852651059628,
+            0.010334450751543045,
+            0.0034636117052286863,
+            0.002946083666756749,
+            -0.001038076588883996,
+            8.589807112002745e-05,
+            -0.004897385835647583,
+            0.03075364977121353,
+            0.012710649520158768,
+            -0.0016975724138319492,
+            -0.018171878531575203,
+            -0.009585344232618809,
+            -0.0062143635004758835,
+            -0.01725361868739128,
+            -0.010141132399439812,
+            0.029915938153862953,
+            0.003501872532069683,
+            0.0007470929995179176,
+            3.530316462274641e-05,
+            0.0032159232068806887,
+            0.028385505080223083,
+            -0.02337535098195076,
+            -0.009738387539982796,
+            -0.0036972041707485914,
+            -0.004168416373431683,
+            0.014974079094827175,
+            0.0021124002523720264,
+            -0.006709740497171879,
+            -0.0015425154706463218,
+            0.012549551203846931,
+            0.013846391811966896,
+            0.002241278998553753,
+            -0.006504340562969446,
+            -0.009295366704463959,
+            -0.022698739543557167,
+            0.0036911629140377045,
+            0.0006811433704569936,
+            0.0085140410810709,
+            0.010841909795999527,
+            -0.005698849447071552,
+            -0.02843383513391018,
+            -0.009432300925254822,
+            0.007390380837023258,
+            0.004071757197380066,
+            -0.0037314374931156635,
+            0.005984798539429903,
+            -0.006681548431515694,
+            0.006826536729931831,
+            0.019911739975214005,
+            0.024277500808238983,
+            0.020958878099918365,
+            0.007869647815823555,
+            0.020974988117814064,
+            0.028949350118637085,
+            -0.007015827111899853,
+            0.0006962463376112282,
+            0.00956923421472311,
+            0.014224972575902939,
+            0.0014670006930828094,
+            0.017334168776869774,
+            0.002881644293665886,
+            -0.0058075906708836555,
+            -0.01110772229731083,
+            -0.0008281455375254154,
+            0.0317041277885437,
+            -0.016327304765582085,
+            -0.00031187606509774923,
+            0.006141869351267815,
+            0.019412335008382797,
+            0.009714222513139248,
+            0.004675875883549452,
+            0.005316241178661585,
+            0.013483921065926552,
+            -0.010463329032063484,
+            -0.012710649520158768,
+            -0.0021345512941479683,
+            0.00498599000275135,
+            0.011671565473079681,
+            -0.01626286469399929,
+            0.0032300192397087812,
+            -0.021313294768333435,
+            0.009891430847346783,
+            -0.019090138375759125,
+            -0.011591016314923763,
+            0.0024668164551258087,
+            0.006544615142047405,
+            0.006298940163105726,
+            0.009271202608942986,
+            0.029545413330197334,
+            -0.0005583059974014759,
+            0.034475017338991165,
+            -0.005078621208667755,
+            -0.017785243690013885,
+            -0.016053438186645508,
+            0.00038210483035072684,
+            -0.020266154780983925,
+            -0.003217936959117651,
+            0.016399798914790154,
+            -0.007184980437159538,
+            -0.009746442548930645,
+            0.00451880507171154,
+            0.010253901593387127,
+            -0.0063593522645533085,
+            -0.005525668617337942,
+            -0.037278126925230026,
+            0.021458283066749573,
+            -0.0013814171543344855,
+            0.010326395742595196,
+            0.019122358411550522,
+            0.025807933881878853,
+            -0.01428135670721531,
+            -0.0026641618460416794,
+            0.0181879885494709,
+            0.003195785917341709,
+            0.023633107542991638,
+            -0.0047805896028876305,
+            0.008433491922914982,
+            -0.018300756812095642,
+            0.007732714526355267,
+            -0.022167114540934563,
+            -0.010616372339427471,
+            0.004651710856705904,
+            -0.027676673606038094,
+            0.0012646210379898548,
+            -0.0002251599362352863,
+            -0.013491976074874401,
+            -0.004764479584991932,
+            -0.0021667710971087217,
+            0.0012515317648649216,
+            -0.007555506192147732,
+            -0.014708267524838448,
+            -0.01639174297451973,
+            -0.027290038764476776,
+            0.00852209609001875,
+            0.0020701121538877487,
+            -0.003348829224705696,
+            0.0015102957841008902,
+            0.005247774533927441,
+            0.007229282520711422,
+            0.012815362773835659,
+            -0.007201090455055237,
+            0.060991786420345306,
+            -0.014047764241695404,
+            0.0020388993434607983,
+            -0.00508264871314168,
+            0.018606843426823616,
+            0.00868319347500801,
+            0.0027366559952497482,
+            -0.011832663789391518,
+            0.0030105228070169687,
+            -0.015473484061658382,
+            -0.013943050988018513,
+            -0.005932441912591457,
+            -5.896446600672789e-05,
+            0.002843383466824889,
+            -0.019702311605215073,
+            -0.0011528590694069862,
+            -0.01416053343564272,
+            2.218246845586691e-05,
+            0.002350020222365856,
+            -0.007986444048583508,
+            0.017785243690013885,
+            -0.018622953444719315,
+            -0.0085140410810709,
+            0.015167397446930408,
+            0.005179307889193296,
+            0.007881729863584042,
+            -0.013306712731719017,
+            -0.012533441185951233,
+            0.000388649437809363,
+            -0.001509288907982409,
+            -0.0013310740469023585,
+            -0.0091020492836833,
+            -0.008151569403707981,
+            -0.012654264457523823,
+            -0.005884112324565649,
+            -0.0018234304152429104,
+            0.0009761545225046575,
+            0.011236600577831268,
+            -0.008066993206739426,
+            0.005702876951545477,
+            -0.020298374816775322,
+            0.021635491400957108,
+            -0.003149470081552863,
+            -0.0211199764162302,
+            -0.008763742633163929,
+            0.01586817391216755,
+            0.0007712577353231609,
+            0.0006418757257051766,
+            0.03157525137066841,
+            0.004059675149619579,
+            -0.0011045295977964997,
+            -0.005453174468129873,
+            -0.0013743691379204392,
+            -0.007720632012933493,
+            0.020974988117814064,
+            -0.00757967121899128,
+            0.010398889891803265,
+            0.003677066881209612,
+            0.0211521964520216,
+            -0.025759605690836906,
+            0.006766125094145536,
+            -0.011003008112311363,
+            -0.0026077774818986654,
+            -0.00857042521238327,
+            -0.012863692827522755,
+            0.008868456818163395,
+            -0.02656509540975094,
+            -0.031043626368045807,
+            0.006262693088501692,
+            -0.012106531299650669,
+            -0.03785808011889458,
+            0.008216009475290775,
+            -0.018365196883678436,
+            -0.004393953830003738,
+            0.004365761764347553,
+            -0.014966024085879326,
+            -0.01726972870528698,
+            -0.012058201245963573,
+            -0.01609371230006218,
+            -0.0010884198127314448,
+            0.015634581446647644,
+            -0.02429361082613468,
+            -0.013757787644863129,
+            -0.006649328861385584,
+            0.007595780771225691,
+            -0.006371434312313795,
+            -0.0039247553795576096,
+            -0.02244098111987114,
+            0.007160815875977278,
+            -0.0059082768857479095,
+            0.01022168155759573,
+            -0.009673947468400002,
+            0.008264338597655296,
+            0.0252602007240057,
+            0.0036911629140377045,
+            -0.002927960129454732,
+            0.005002099554985762,
+            0.011792389675974846,
+            -0.0006484203040599823,
+            0.0293682049959898,
+            -0.007031937129795551,
+            0.031526919454336166,
+            0.0020439336076378822,
+            0.003568325424566865,
+            -0.015602362342178822,
+            -0.004655738361179829,
+            0.012622045353055,
+            0.008034773170948029,
+            -0.013717513531446457,
+            -0.003211895702406764,
+            -0.027789441868662834,
+            0.012871747836470604,
+            -0.023536449298262596,
+            -0.0039750984869897366,
+            0.005541778635233641,
+            -0.018252428621053696,
+            0.009400080889463425,
+            0.0028917130548506975,
+            0.03238074108958244,
+            0.017801353707909584,
+            -0.0016633390914648771,
+            0.002116427756845951,
+            0.007877702824771404,
+            -0.0020127207972109318,
+            0.019541213288903236,
+            0.004748370032757521,
+            -0.017285838723182678,
+            -0.015650691464543343,
+            -0.019573433324694633,
+            0.0007576650241389871,
+            0.007901867851614952,
+            -0.01051165908575058,
+            0.012412617914378643,
+            -0.0013975270558148623,
+            0.022134894505143166,
+            -0.00551761407405138,
+            -0.008876511827111244,
+            -0.01315366942435503,
+            -0.01268648449331522,
+            0.01602121815085411,
+            -0.001299861236475408,
+            0.016423963010311127,
+            0.007785071153193712,
+            0.010302230715751648,
+            -0.011115777306258678,
+            -0.00014310052210930735,
+            -0.01602121815085411,
+            -0.005195417441427708,
+            -0.007535369135439396,
+            -0.0008845299016684294,
+            0.03937240317463875,
+            -0.006230473518371582,
+            -0.005863975267857313,
+            0.0012736828066408634,
+            -0.0117521146312356,
+            0.024358050897717476,
+            0.013798062689602375,
+            0.022457091137766838,
+            -0.010414999909698963,
+            0.00035970209864899516,
+            -0.008175734430551529,
+            0.01842963509261608,
+            0.006665438879281282,
+            -0.0037314374931156635,
+            -0.011373533867299557,
+            -0.003958988469094038,
+            0.006492258049547672,
+            0.017543595284223557,
+            0.00019470229744911194,
+            -0.015972888097167015,
+            -0.004498667549341917,
+            0.004156333860009909,
+            0.004341596737504005,
+            0.009408135898411274,
+            -0.004095922224223614,
+            0.019444555044174194,
+            0.008155597373843193,
+            -0.016383688896894455,
+            -0.00754745164886117,
+            -0.015441264025866985,
+            0.0019059933256357908,
+            0.013338932767510414,
+            -0.015392934903502464,
+            0.009134269319474697,
+            0.005352488253265619,
+            -0.028611043468117714,
+            0.013886665925383568,
+            -0.012613990344107151,
+            -0.004760452080518007,
+            0.028095528483390808,
+            0.006383516825735569,
+            -0.02300482615828514,
+            -0.005424982402473688,
+            0.002648051828145981,
+            -0.014498839154839516,
+            0.02219933457672596,
+            -0.006439901422709227,
+            0.003471666481345892,
+            -0.007760906592011452,
+            -0.004869193769991398,
+            0.027531685307621956,
+            -0.00200970028527081,
+            -0.0009978021262213588,
+            0.007325941231101751
+          ],
+          "keyphrases": [
+            "Capped Expert Choice",
+            "entropy-regularized linear programming",
+            "validation perplexity",
+            "load balanced training",
+            "fine-tuning results"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "f3c5b4a3-1fce-4703-9a7b-1aa67482fdb5",
+      "type": "next",
+      "source": {
+        "id": "d3110cfa-448f-46ee-b3ea-c6734cf305de",
+        "properties": {
+          "page_content": "C Capped Expert Choice\n\nAs described in Section 4.5, the maximum number of experts each token is assigned can be capped by an entropy-regularized linear programming. Figure 1 compares the validation perplexity when training the 100M/64E models using the base expert choice method (EC-BASE), expert choice capped by two experts per token (EC-CAP2), expert choice capped by three experts per token (EC-CAP3), and GShard top-2 gating.\n\nAs shown in the figure, restricting the number of experts to 2 degrades the perplexity compared to the base expert choice method. This suggests that a more flexible allocation of experts (e.g. more than 2 experts for a token) can enhance model expressiveness. On the other hand, our EC-CAP2 and EC-CAP3 methods still outperform the top-2 gating method by a clear margin. We believe this confirms the effectiveness of a load balanced training, provided by our method. Finally, EC-CAP3 obtains comparable perplexity to EC-BASE. As indicated by Figure 3, only a little fraction of tokens use more than 3 experts therefore we see little or no difference between EC-BASE and EC-CAP3 variants. We present the fine-tuning results of these methods in Table 2.\n\n",
+          "title": "C Capped Expert Choice",
+          "embedding": [
+            -0.0025292420759797096,
+            -0.00710443127900362,
+            -0.03731034696102142,
+            -0.015119067393243313,
+            3.892158201779239e-05,
+            0.01479687076061964,
+            -0.02490578405559063,
+            0.06395599246025085,
+            -0.031752459704875946,
+            0.013524195179343224,
+            -0.011389643885195255,
+            -0.03146248310804367,
+            -0.014837145805358887,
+            0.002412445843219757,
+            0.01028612069785595,
+            0.020378924906253815,
+            0.0013713486259803176,
+            0.009432300925254822,
+            0.005433037411421537,
+            -0.013242273591458797,
+            0.0022654435597360134,
+            -0.03563492372632027,
+            -0.02867548167705536,
+            0.004981962498277426,
+            0.034249480813741684,
+            0.023761987686157227,
+            0.00903761014342308,
+            -0.013854446820914745,
+            -0.04288434609770775,
+            0.04004901647567749,
+            0.017833571881055832,
+            0.01280730776488781,
+            -0.019879519939422607,
+            -0.036955930292606354,
+            -0.03078586980700493,
+            -0.02911044843494892,
+            0.03830915689468384,
+            0.022859837859869003,
+            -0.0014800898497924209,
+            0.012509276159107685,
+            0.033411771059036255,
+            0.012307903729379177,
+            -0.02360088936984539,
+            -0.013516140170395374,
+            0.01215486042201519,
+            0.0035884627141058445,
+            -0.03898577019572258,
+            -0.011365478858351707,
+            0.010390834882855415,
+            -0.04159555956721306,
+            0.01303284615278244,
+            -0.039533503353595734,
+            0.013500030152499676,
+            0.010060583241283894,
+            -0.040693409740924835,
+            0.009480630047619343,
+            -0.006468093488365412,
+            0.034346140921115875,
+            0.08435102552175522,
+            0.03172023966908455,
+            0.01726972870528698,
+            -0.031301382929086685,
+            -0.0481683649122715,
+            -0.01268648449331522,
+            -0.008747633546590805,
+            -0.016440073028206825,
+            -0.0063593522645533085,
+            0.0023298829328268766,
+            0.02923932671546936,
+            -0.018606843426823616,
+            -0.054096780717372894,
+            0.03360508754849434,
+            -0.028256626799702644,
+            0.00737427081912756,
+            -0.0055820532143116,
+            -0.01622259058058262,
+            0.037632543593645096,
+            0.03502275049686432,
+            0.01321810856461525,
+            0.03734256699681282,
+            -0.004065716173499823,
+            0.03325067088007927,
+            -0.0012092434335500002,
+            0.0340561643242836,
+            0.04088672623038292,
+            -0.02535685896873474,
+            -0.0193801149725914,
+            -0.057254306972026825,
+            -0.028691591694951057,
+            -0.021426063030958176,
+            0.0004978941869921982,
+            0.008199899457395077,
+            -0.0026259010192006826,
+            -0.018510185182094574,
+            -0.03308957442641258,
+            0.04613852873444557,
+            -0.012895911931991577,
+            0.007221227511763573,
+            -0.012646210379898548,
+            -0.01455522421747446,
+            -0.02828884683549404,
+            0.007462874986231327,
+            0.0039448924362659454,
+            -0.02221544459462166,
+            0.03605378046631813,
+            -0.01749526709318161,
+            -0.01784968189895153,
+            0.010181407444179058,
+            0.014579388312995434,
+            0.09305033087730408,
+            0.009891430847346783,
+            -0.0006464066100306809,
+            -0.007772989105433226,
+            -0.010946623980998993,
+            -0.05528890714049339,
+            -0.023649217560887337,
+            -0.04394759237766266,
+            -0.03685927018523216,
+            0.021184414625167847,
+            -0.01915457844734192,
+            0.0004923564265482128,
+            0.0317363478243351,
+            0.007176925428211689,
+            -0.0029420561622828245,
+            -0.04723399877548218,
+            0.026967842131853104,
+            0.024809125810861588,
+            0.051615867763757706,
+            0.00537665281444788,
+            -0.01221929956227541,
+            -0.006584889721125364,
+            0.03650485724210739,
+            -0.018751831725239754,
+            -0.02867548167705536,
+            0.017817461863160133,
+            0.0152721107006073,
+            0.05358126759529114,
+            0.03866357356309891,
+            0.0025131322909146547,
+            -0.007893812842667103,
+            0.0069594429805874825,
+            -0.0071004037745296955,
+            0.004671848379075527,
+            0.009134269319474697,
+            -0.046686261892318726,
+            0.019235126674175262,
+            9.407883771928027e-05,
+            0.05799535661935806,
+            -0.026693973690271378,
+            0.018284646794199944,
+            0.005928414408117533,
+            0.036827053874731064,
+            -0.028385505080223083,
+            0.046815142035484314,
+            -0.018010780215263367,
+            -0.021748259663581848,
+            -0.013781952671706676,
+            -0.0015817830571904778,
+            -0.009005390107631683,
+            -0.03573158383369446,
+            0.01416053343564272,
+            0.03134971112012863,
+            -0.0021587160881608725,
+            0.016101766377687454,
+            0.0020167483016848564,
+            0.04194997623562813,
+            0.013201998546719551,
+            0.01603732816874981,
+            -0.07481401413679123,
+            0.029062118381261826,
+            0.029223216697573662,
+            -0.029690401628613472,
+            0.04455976560711861,
+            0.014982134103775024,
+            0.0014508907916024327,
+            0.02337535098195076,
+            0.003131346544250846,
+            0.0004191070911474526,
+            -0.011011063121259212,
+            0.0411444827914238,
+            -0.01603732816874981,
+            -0.015376824885606766,
+            0.024712465703487396,
+            -0.011736005544662476,
+            0.033895064145326614,
+            -0.0009348730673082173,
+            0.029915938153862953,
+            0.02841772511601448,
+            -0.013137559406459332,
+            0.007458847481757402,
+            0.013959160074591637,
+            0.041885536164045334,
+            -0.0005522647988982499,
+            0.005771343596279621,
+            0.032010216265916824,
+            -0.01308923028409481,
+            0.0281760785728693,
+            -0.00663724634796381,
+            0.018026890233159065,
+            0.019927849993109703,
+            -0.009891430847346783,
+            -0.010334450751543045,
+            -0.007189007941633463,
+            -0.02864326350390911,
+            0.015594307333230972,
+            0.04852278158068657,
+            0.02666175551712513,
+            0.026275118812918663,
+            -0.005364570766687393,
+            -0.021184414625167847,
+            0.023536449298262596,
+            -0.03252572938799858,
+            0.002066084649413824,
+            -0.02712894044816494,
+            -0.04733065515756607,
+            0.024003634229302406,
+            -0.0006433859816752374,
+            -0.009464520029723644,
+            0.0012585797812789679,
+            0.007760906592011452,
+            0.03172023966908455,
+            0.0423043929040432,
+            0.02500244230031967,
+            -0.020733339712023735,
+            -0.008119350299239159,
+            -0.010994953103363514,
+            -0.01520767156034708,
+            0.006814454682171345,
+            0.02335924096405506,
+            -0.01900959014892578,
+            0.006057293154299259,
+            0.009980034083127975,
+            0.0008528137113898993,
+            -0.0016804557526484132,
+            0.016850873827934265,
+            -0.010866074822843075,
+            0.025163540616631508,
+            -0.02242487110197544,
+            0.016528677195310593,
+            0.012791198678314686,
+            -0.020701121538877487,
+            -0.0028131776489317417,
+            -0.03605378046631813,
+            0.008115322329103947,
+            -0.004542969632893801,
+            -0.008079075254499912,
+            -0.02326258271932602,
+            0.0007093356107361615,
+            -0.0043335421942174435,
+            -0.012042091228067875,
+            -0.027547795325517654,
+            -0.02453525923192501,
+            0.0004686951288022101,
+            0.009392025880515575,
+            -0.016045382246375084,
+            0.026758413761854172,
+            0.004265075083822012,
+            -0.05799535661935806,
+            0.012920076958835125,
+            -0.024245280772447586,
+            0.06907891482114792,
+            0.008812072686851025,
+            -0.0105760982260108,
+            0.015489593148231506,
+            -0.023939194157719612,
+            -0.011510467156767845,
+            0.01462771836668253,
+            -0.00922287255525589,
+            -0.003711300203576684,
+            -0.0029380286578089,
+            0.02770889364182949,
+            -0.01515128742903471,
+            -0.010382779873907566,
+            -0.045848552137613297,
+            0.05499893054366112,
+            -0.029529303312301636,
+            0.0008810058934614062,
+            0.007825345732271671,
+            -0.007084294222295284,
+            0.018703503534197807,
+            0.01973453164100647,
+            0.02186102792620659,
+            0.01009280327707529,
+            0.01975064165890217,
+            0.015707075595855713,
+            0.0022775260731577873,
+            -0.0622805692255497,
+            -0.002076153177767992,
+            -0.03217131271958351,
+            -0.03159135952591896,
+            -0.02806330844759941,
+            -0.03067309968173504,
+            -0.010076693259179592,
+            0.018155768513679504,
+            0.01971842162311077,
+            -0.02537296898663044,
+            -0.015803735703229904,
+            0.008852346800267696,
+            0.004800726659595966,
+            -0.0041885534301400185,
+            -0.023230362683534622,
+            0.0024990360252559185,
+            -0.03511941060423851,
+            -0.028691591694951057,
+            -0.06714573502540588,
+            0.004402008838951588,
+            -0.008115322329103947,
+            0.010962733998894691,
+            -0.044914182275533676,
+            0.03302513435482979,
+            0.03373396769165993,
+            -0.010463329032063484,
+            -0.0003755602228920907,
+            -0.005481366999447346,
+            -0.006588917225599289,
+            0.008054911158978939,
+            0.020282264798879623,
+            0.030463673174381256,
+            -0.028127748519182205,
+            0.005465256981551647,
+            0.04649294540286064,
+            0.019815079867839813,
+            -0.04839390516281128,
+            -0.026355668902397156,
+            0.014426345005631447,
+            0.021442173048853874,
+            0.000837207306176424,
+            0.015699021518230438,
+            -0.021893247961997986,
+            -0.010986898094415665,
+            -0.02608180046081543,
+            -0.04185331612825394,
+            -0.056803230196237564,
+            -0.010849964804947376,
+            -0.03550604730844498,
+            -0.02126496471464634,
+            -0.009762551635503769,
+            0.015425154007971287,
+            -0.019782861694693565,
+            -0.017430827021598816,
+            -0.021989906206727028,
+            0.03956572338938713,
+            -0.045171938836574554,
+            0.02758001536130905,
+            0.02456747740507126,
+            -0.0045671346597373486,
+            -0.01797856017947197,
+            -0.010495549067854881,
+            -0.007740769535303116,
+            -0.002712491201236844,
+            0.0010280079441145062,
+            0.002889699302613735,
+            0.034120600670576096,
+            -5.767442053183913e-05,
+            0.014442455023527145,
+            -0.035667143762111664,
+            -0.010833854787051678,
+            0.01868739351630211,
+            0.026242898777127266,
+            0.005098758731037378,
+            -0.004414091352373362,
+            -0.045880772173404694,
+            -0.045848552137613297,
+            0.03133360296487808,
+            -0.038921330124139786,
+            0.01639174297451973,
+            0.005868002772331238,
+            0.009287312626838684,
+            -0.04030677303671837,
+            -0.002058029640465975,
+            -0.02184491790831089,
+            -0.01926734670996666,
+            0.04101560637354851,
+            0.013338932767510414,
+            0.03102751635015011,
+            -0.0027165187057107687,
+            0.020266154780983925,
+            -0.039307963103055954,
+            -0.049843788146972656,
+            0.019927849993109703,
+            -0.05061705783009529,
+            0.02277928777039051,
+            0.030334794893860817,
+            0.01315366942435503,
+            -0.023987524211406708,
+            0.029690401628613472,
+            -0.00795422401279211,
+            0.016182316467165947,
+            0.0012464975006878376,
+            0.028401615098118782,
+            -0.0399523563683033,
+            -0.03308957442641258,
+            -0.009053720161318779,
+            0.020169496536254883,
+            0.0004284205788280815,
+            0.0036609568633139133,
+            -0.021329404786229134,
+            -0.002535283099859953,
+            0.008352942764759064,
+            -0.018848491832613945,
+            -0.041885536164045334,
+            0.04146667942404747,
+            -0.014434400014579296,
+            0.007233310025185347,
+            0.0005955599481239915,
+            -0.0030608661472797394,
+            0.0036347785498946905,
+            -0.007676329929381609,
+            0.009947814978659153,
+            -0.005002099554985762,
+            -0.013121449388563633,
+            -0.009762551635503769,
+            -0.003427364630624652,
+            -0.008131432346999645,
+            0.0062143635004758835,
+            0.024003634229302406,
+            -0.011083557270467281,
+            0.019428445026278496,
+            -0.010294175706803799,
+            -0.010197517462074757,
+            -0.01303284615278244,
+            0.013073120266199112,
+            0.04085450619459152,
+            -0.02103942632675171,
+            0.0340561643242836,
+            -0.0007098389905877411,
+            -0.02042725309729576,
+            0.029674291610717773,
+            0.012106531299650669,
+            0.004760452080518007,
+            0.037986960262060165,
+            -0.002492995001375675,
+            0.00475642504170537,
+            -0.03534494712948799,
+            -0.010656647384166718,
+            -0.01865517348051071,
+            -0.05429009720683098,
+            0.02993204817175865,
+            -0.010721086524426937,
+            0.00354013335891068,
+            0.01216291543096304,
+            -0.0006494271801784635,
+            0.012396507896482944,
+            -0.02479301579296589,
+            -0.03724590688943863,
+            -0.018864601850509644,
+            -0.006721823010593653,
+            -0.016738105565309525,
+            -0.020620571449398994,
+            0.005002099554985762,
+            -0.06514811515808105,
+            0.0015445291064679623,
+            -0.0563521571457386,
+            0.004945715423673391,
+            0.03946906328201294,
+            -0.010841909795999527,
+            -0.001440822146832943,
+            -0.01950899325311184,
+            0.009480630047619343,
+            -0.048103928565979004,
+            -0.045171938836574554,
+            0.016142040491104126,
+            0.033282890915870667,
+            -0.010890239849686623,
+            -0.00915843341499567,
+            0.014192752540111542,
+            -0.037632543593645096,
+            -0.0017086479347199202,
+            0.01374973263591528,
+            0.012106531299650669,
+            -0.01480492576956749,
+            0.01056804321706295,
+            -0.022038236260414124,
+            -0.021409953013062477,
+            -0.008054911158978939,
+            0.029883719980716705,
+            0.01416053343564272,
+            -0.011945432983338833,
+            0.004009332042187452,
+            0.010350560769438744,
+            0.03325067088007927,
+            -0.0076521653681993484,
+            -0.005300131160765886,
+            0.010849964804947376,
+            0.016142040491104126,
+            -0.0022654435597360134,
+            0.012340123765170574,
+            -0.007285666652023792,
+            -0.015223781578242779,
+            -0.013717513531446457,
+            -0.003880453296005726,
+            -0.014974079094827175,
+            0.0018657187465578318,
+            -0.003938851412385702,
+            0.015014354139566422,
+            -0.014982134103775024,
+            -0.020523913204669952,
+            -0.02596903219819069,
+            0.02442248910665512,
+            0.02994815818965435,
+            -0.01227568369358778,
+            -0.034571677446365356,
+            -0.02100720815360546,
+            -3.951940743718296e-05,
+            0.022038236260414124,
+            -0.02455136738717556,
+            0.029851499944925308,
+            0.033282890915870667,
+            -0.05554666370153427,
+            -0.002996426774188876,
+            0.004804754164069891,
+            0.006383516825735569,
+            -0.00431340467184782,
+            -0.03615044057369232,
+            -0.0028333149384707212,
+            0.03934018313884735,
+            -0.0007793126278556883,
+            0.04887719824910164,
+            0.0021526748314499855,
+            -0.017108630388975143,
+            -0.032686829566955566,
+            -0.01668977551162243,
+            -0.03863135352730751,
+            0.04900607839226723,
+            -0.020298374816775322,
+            0.034120600670576096,
+            0.014998244121670723,
+            -0.026581205427646637,
+            0.060798466205596924,
+            -0.0376003235578537,
+            -0.006427818909287453,
+            -0.00015921035083010793,
+            -0.007039992138743401,
+            -0.01772080361843109,
+            -0.01228373870253563,
+            0.0002982834121212363,
+            -0.00780923618003726,
+            0.004357706755399704,
+            -0.004990017041563988,
+            -0.0010285114403814077,
+            0.004426173400133848,
+            0.014941859990358353,
+            -0.004418118391185999,
+            0.028852690011262894,
+            0.013822226785123348,
+            0.008127405308187008,
+            0.020523913204669952,
+            -0.02572738565504551,
+            0.016625335440039635,
+            0.027998870238661766,
+            -0.00560621777549386,
+            -0.010390834882855415,
+            -0.008119350299239159,
+            -0.0011427904246374965,
+            0.006991662550717592,
+            -0.028820471838116646,
+            0.015304330736398697,
+            0.014579388312995434,
+            -0.0030205915682017803,
+            -0.020991098135709763,
+            -0.02360088936984539,
+            0.018864601850509644,
+            -0.023165924474596977,
+            0.027773331850767136,
+            -0.021071646362543106,
+            0.00044855783926323056,
+            0.026710083708167076,
+            0.03573158383369446,
+            -0.011083557270467281,
+            0.015006299130618572,
+            -0.00043949607061222196,
+            -0.005058484151959419,
+            -0.020491693168878555,
+            0.0045751892030239105,
+            -0.0008291523554362357,
+            0.009593399241566658,
+            0.011913212947547436,
+            0.005022237077355385,
+            -0.018606843426823616,
+            0.01515128742903471,
+            0.01609371230006218,
+            -0.030689209699630737,
+            0.00428924011066556,
+            0.010664702393114567,
+            0.014176643453538418,
+            0.005070566199719906,
+            0.003234046744182706,
+            -0.023101484403014183,
+            -0.004462420474737883,
+            -0.016738105565309525,
+            -0.01950899325311184,
+            0.044463109225034714,
+            0.0016653528437018394,
+            -0.034088384360075,
+            0.021538831293582916,
+            -0.007748824078589678,
+            0.005171252880245447,
+            0.0281760785728693,
+            -0.009488685056567192,
+            -0.0505204014480114,
+            -0.013185889460146427,
+            -0.006077430211007595,
+            0.016005108132958412,
+            -0.01926734670996666,
+            -0.009247037582099438,
+            0.030479783192276955,
+            -0.004977934993803501,
+            0.0051269507966935635,
+            -0.008079075254499912,
+            0.024841345846652985,
+            0.026581205427646637,
+            0.00314342905767262,
+            0.007970334030687809,
+            0.018945150077342987,
+            -0.006592944730073214,
+            0.009649783372879028,
+            -0.009174543432891369,
+            -0.0035300645977258682,
+            -0.0002856976061593741,
+            -0.005348460748791695,
+            0.06611470878124237,
+            0.0017881902167573571,
+            0.030705319717526436,
+            -0.0016039340989664197,
+            0.02758001536130905,
+            0.015497648157179356,
+            -0.010986898094415665,
+            -0.02266651950776577,
+            0.007644110359251499,
+            -0.012251519598066807,
+            -0.05416122078895569,
+            -0.026855072006583214,
+            -0.022875946015119553,
+            -0.016085656359791756,
+            -0.029883719980716705,
+            -0.006685575935989618,
+            2.6902143872575834e-05,
+            0.009859210811555386,
+            0.004510750062763691,
+            -0.01786579191684723,
+            -0.032010216265916824,
+            0.00969811249524355,
+            -0.0011850787559524179,
+            0.025340748950839043,
+            -0.005348460748791695,
+            0.010487494058907032,
+            0.007221227511763573,
+            -0.005783426109701395,
+            0.019557323306798935,
+            -0.027451135218143463,
+            0.014007490128278732,
+            -1.3608394056063844e-06,
+            0.011333259753882885,
+            0.035860463976860046,
+            -0.014394125901162624,
+            -0.01726972870528698,
+            -0.005384707823395729,
+            0.008107267320156097,
+            0.023165924474596977,
+            -0.009263147599995136,
+            -0.014877419918775558,
+            0.0021103867329657078,
+            0.005175280384719372,
+            -0.032477401196956635,
+            0.017591925337910652,
+            -0.014200807549059391,
+            -0.05764094367623329,
+            -0.004764479584991932,
+            -0.021877137944102287,
+            0.019219016656279564,
+            -0.025195760652422905,
+            -0.006890976335853338,
+            -0.018123548477888107,
+            0.0015183506766334176,
+            0.014168588444590569,
+            -0.023633107542991638,
+            0.01103522814810276,
+            0.05899416655302048,
+            0.0044221458956599236,
+            0.01762414537370205,
+            0.009464520029723644,
+            -0.005880084820091724,
+            -0.008071020245552063,
+            0.003036701353266835,
+            0.021812697872519493,
+            -0.0002548624179325998,
+            -0.031301382929086685,
+            0.039050206542015076,
+            -0.0004669331246986985,
+            0.004591299220919609,
+            -0.010721086524426937,
+            -0.0007571616442874074,
+            -0.010463329032063484,
+            0.009843100793659687,
+            -0.02385864593088627,
+            0.008892621845006943,
+            -0.019927849993109703,
+            0.03731034696102142,
+            0.0008155597024597228,
+            -0.003642833326011896,
+            -0.025904593989253044,
+            0.0013814171543344855,
+            -0.03199410438537598,
+            0.016182316467165947,
+            0.02911044843494892,
+            -0.012324013747274876,
+            -0.0014539114199578762,
+            0.012267629615962505,
+            -0.02737058699131012,
+            -0.001367321121506393,
+            -0.009843100793659687,
+            0.01402360014617443,
+            0.015111012384295464,
+            0.0030165640637278557,
+            0.034926094114780426,
+            0.01267842948436737,
+            0.009931704960763454,
+            0.013016736134886742,
+            0.0001655032392591238,
+            0.015747351571917534,
+            -0.0038140001706779003,
+            0.005578025709837675,
+            -0.010874129831790924,
+            -0.013604744337499142,
+            -0.021168306469917297,
+            0.01592455804347992,
+            0.021893247961997986,
+            0.011663510464131832,
+            0.03750366345047951,
+            0.014780761674046516,
+            0.026468437165021896,
+            0.0022714848164469004,
+            -0.03582824394106865,
+            0.001979494234547019,
+            0.007426627911627293,
+            -0.025082992389798164,
+            -0.004889330826699734,
+            0.0193801149725914,
+            0.011204380542039871,
+            -0.02501855231821537,
+            0.005851892754435539,
+            0.005396790336817503,
+            -0.002138578798621893,
+            -0.0091020492836833,
+            -0.03579602390527725,
+            0.0021989906672388315,
+            0.019992288202047348,
+            0.025791823863983154,
+            0.003517982317134738,
+            0.015787625685334206,
+            0.007865620777010918,
+            -0.009657838381826878,
+            0.03534494712948799,
+            0.006633218843489885,
+            -0.039275746792554855,
+            -0.015223781578242779,
+            -0.008650974370539188,
+            0.01797856017947197,
+            -0.00029677312704734504,
+            0.0031857171561568975,
+            -0.00969811249524355,
+            0.0047805896028876305,
+            -0.007023882120847702,
+            -0.009303421713411808,
+            0.03395950421690941,
+            0.007011800073087215,
+            0.011317149735987186,
+            -0.004273130092769861,
+            0.018413526937365532,
+            -0.014466620050370693,
+            -0.006540587637573481,
+            0.010366669856011868,
+            0.02314981445670128,
+            -0.025276310741901398,
+            0.015167397446930408,
+            -0.005718986503779888,
+            0.003246129024773836,
+            0.016174260526895523,
+            -0.00431340467184782,
+            -0.01562652736902237,
+            -0.02972262166440487,
+            0.0038683710154145956,
+            -0.014257192611694336,
+            -0.008401271887123585,
+            -0.020475583150982857,
+            -0.014853255823254585,
+            0.030350904911756516,
+            -0.0193801149725914,
+            0.0023802260402590036,
+            -0.0014559251721948385,
+            0.016238700598478317,
+            -0.02031448483467102,
+            -0.013483921065926552,
+            0.025066882371902466,
+            0.05261467769742012,
+            0.0019684189464896917,
+            0.02970651164650917,
+            0.07938919961452484,
+            -0.030689209699630737,
+            -0.013645019382238388,
+            0.019315676763653755,
+            -0.03028646484017372,
+            0.0034535429440438747,
+            0.03895355015993118,
+            -0.04501084238290787,
+            -0.06166839599609375,
+            0.013177834451198578,
+            -0.034603897482156754,
+            -0.000111384317278862,
+            0.010439164005219936,
+            0.008747633546590805,
+            -0.019347894936800003,
+            -0.00610562227666378,
+            0.028804361820220947,
+            -0.007414545398205519,
+            -0.003664984367787838,
+            -0.029319874942302704,
+            -0.011470193043351173,
+            -0.018123548477888107,
+            -0.005388735327869654,
+            -0.0063593522645533085,
+            0.0211360864341259,
+            -0.0006786262383684516,
+            0.003614641260355711,
+            -0.019573433324694633,
+            0.00046743653365410864,
+            0.013725567609071732,
+            0.018590733408927917,
+            0.049972668290138245,
+            -9.590377885615453e-05,
+            0.013983325101435184,
+            -0.01715696044266224,
+            -0.026355668902397156,
+            0.041273362934589386,
+            -0.006955415476113558,
+            0.040822289884090424,
+            -0.02725781872868538,
+            -0.0021748258732259274,
+            -0.006568779703229666,
+            0.025807933881878853,
+            -0.01913846842944622,
+            0.013934995979070663,
+            -0.010729141533374786,
+            -0.0117521146312356,
+            -0.020604461431503296,
+            -0.00766827492043376,
+            0.0020328580867499113,
+            -0.026387887075543404,
+            -0.005968688987195492,
+            -0.004108004737645388,
+            0.027209488674998283,
+            0.047846172004938126,
+            0.005416927393525839,
+            -0.004124114289879799,
+            -0.009077884256839752,
+            -0.015392934903502464,
+            -0.008667084388434887,
+            -0.01726972870528698,
+            -0.015819845721125603,
+            -0.01579567976295948,
+            -0.028014980256557465,
+            -0.019766751676797867,
+            -0.019831189885735512,
+            -0.007325941231101751,
+            0.0029380286578089,
+            -0.0007752851815894246,
+            0.0076521653681993484,
+            -0.04639628529548645,
+            -0.0002455489302519709,
+            -0.03065699152648449,
+            -0.01426524668931961,
+            0.017060300335288048,
+            -0.03859913349151611,
+            -0.00634324224665761,
+            0.0004344617482274771,
+            -0.0020217825658619404,
+            0.0015999067109078169,
+            -0.015119067393243313,
+            0.007797153666615486,
+            -0.02795054018497467,
+            -0.007865620777010918,
+            0.002593681216239929,
+            0.017817461863160133,
+            0.0013310740469023585,
+            0.021538831293582916,
+            -0.03101140633225441,
+            0.011760169640183449,
+            -0.039662379771471024,
+            -0.02619457058608532,
+            -0.042691025882959366,
+            0.027048390358686447,
+            -0.029770949855446815,
+            -0.01168767549097538,
+            0.008804017677903175,
+            -0.0007818297599442303,
+            0.00716887041926384,
+            -0.0152721107006073,
+            -0.0013622867409139872,
+            -0.0027265872340649366,
+            0.007039992138743401,
+            -0.005400817841291428,
+            -0.021748259663581848,
+            -0.017527485266327858,
+            -0.017285838723182678,
+            0.020749449729919434,
+            -0.035054970532655716,
+            0.02044336311519146,
+            -0.012549551203846931,
+            0.02253764122724533,
+            -0.020250046625733376,
+            -0.017898011952638626,
+            -0.006641273852437735,
+            0.040210116654634476,
+            -0.02033059485256672,
+            0.011309094727039337,
+            0.039275746792554855,
+            0.04240104928612709,
+            -0.028611043468117714,
+            0.02571127563714981,
+            -0.022376542910933495,
+            0.015489593148231506,
+            0.055256687104701996,
+            0.01772080361843109,
+            0.02983538992702961,
+            -0.004909468349069357,
+            -0.026919512078166008,
+            -0.008804017677903175,
+            -0.011011063121259212,
+            0.0034736802335828543,
+            0.003671025624498725,
+            -0.02901378832757473,
+            -0.002052995376288891,
+            0.012025982141494751,
+            -0.02397141419351101,
+            0.043850935995578766,
+            0.0013300671707838774,
+            0.01280730776488781,
+            0.005952578969299793,
+            -0.016979752108454704,
+            0.02207045629620552,
+            -0.010704976506531239,
+            0.0005127454060129821,
+            0.009665893390774727,
+            -0.0060170185752213,
+            -0.004683930426836014,
+            -0.03511941060423851,
+            0.01121243555098772,
+            0.024116402491927147,
+            -0.004635601304471493,
+            0.003783794352784753,
+            -0.01221124455332756,
+            -0.002809150144457817,
+            0.005316241178661585,
+            0.020588351413607597,
+            0.0016633390914648771,
+            0.03180078789591789,
+            -0.011647401377558708,
+            0.00777701660990715,
+            -0.011840718798339367,
+            0.023810315877199173,
+            0.008062966167926788,
+            -0.004031483083963394,
+            -0.023536449298262596,
+            -0.010793580673635006,
+            0.017946342006325722,
+            0.01760803535580635,
+            -0.030818087980151176,
+            -0.040596749633550644,
+            -0.021103866398334503,
+            0.016343414783477783,
+            0.0010471383575350046,
+            -0.04143446311354637,
+            -0.012420672923326492,
+            0.015135177411139011,
+            0.01168767549097538,
+            0.02948097325861454,
+            0.006536560133099556,
+            0.007442737463861704,
+            -0.00027009123004972935,
+            0.010551933199167252,
+            0.036472637206315994,
+            9.779164975043386e-05,
+            -0.009021500125527382,
+            -0.014998244121670723,
+            0.022956496104598045,
+            -0.029787059873342514,
+            0.0323324128985405,
+            0.011292984709143639,
+            -0.030721429735422134,
+            0.017704693600535393,
+            0.004957797471433878,
+            -0.018010780215263367,
+            0.03515163064002991,
+            -0.03653707727789879,
+            0.00525582954287529,
+            0.028691591694951057,
+            -0.008010609075427055,
+            -0.01913846842944622,
+            -0.002654093084856868,
+            0.006234501022845507,
+            -0.019702311605215073,
+            -0.0030326738487929106,
+            -0.001486130990087986,
+            0.015779569745063782,
+            -0.015280165709555149,
+            0.014297466725111008,
+            0.026403997093439102,
+            0.014659937471151352,
+            -0.023182032629847527,
+            -0.01714085042476654,
+            -0.020056728273630142,
+            -0.003956974949687719,
+            -0.007442737463861704,
+            0.008441546931862831,
+            -0.019460665062069893,
+            0.021200524643063545,
+            -0.02524409070611,
+            -0.05213138461112976,
+            -0.0007012806599959731,
+            0.005590108223259449,
+            -0.020620571449398994,
+            0.004824891686439514,
+            -0.046202968806028366,
+            0.014007490128278732,
+            0.012074311263859272,
+            0.03885689005255699,
+            -0.007772989105433226,
+            -0.0030105228070169687,
+            0.006488230545073748,
+            0.010777470655739307,
+            0.02853049337863922,
+            -0.032139092683792114,
+            -0.0010682825231924653,
+            0.019798971712589264,
+            -0.0019231099868193269,
+            -0.0015676870243623853,
+            -0.0054733119904994965,
+            -0.04417313262820244,
+            -0.033186230808496475,
+            0.03708481043577194,
+            0.010986898094415665,
+            -0.003620682517066598,
+            0.03065699152648449,
+            -0.023101484403014183,
+            0.009488685056567192,
+            -0.016142040491104126,
+            0.016593115404248238,
+            0.010954678989946842,
+            0.02912655845284462,
+            -0.02948097325861454,
+            -0.0013653073692694306,
+            0.03315401449799538,
+            0.010841909795999527,
+            -0.014120258390903473,
+            0.00024693337036296725,
+            0.018300756812095642,
+            0.020991098135709763,
+            -0.017237508669495583,
+            0.016270920634269714,
+            0.005686766933649778,
+            -0.004236883018165827,
+            -0.019798971712589264,
+            0.0013149641454219818,
+            0.0223282128572464,
+            -0.015135177411139011,
+            0.03006092645227909,
+            -0.009657838381826878,
+            -0.010004199109971523,
+            0.04613852873444557,
+            0.02606569230556488,
+            -0.013411426916718483,
+            4.716527837445028e-05,
+            0.006238528527319431,
+            0.017994670197367668,
+            0.007551478687673807,
+            -0.012348177842795849,
+            0.0006000908324494958,
+            -0.02856271341443062,
+            -0.0029923992697149515,
+            0.007458847481757402,
+            -0.008884566836059093,
+            0.009367860853672028,
+            0.021345512941479683,
+            0.005686766933649778,
+            -0.006101594772189856,
+            0.003542147111147642,
+            -0.010318340733647346,
+            0.019347894936800003,
+            0.03250962123274803,
+            -0.01865517348051071,
+            -0.018413526937365532,
+            0.012460947036743164,
+            -0.026017362251877785,
+            0.03785808011889458,
+            0.009915594942867756,
+            -0.0058196731843054295,
+            0.029738731682300568,
+            0.09479019045829773,
+            -0.012791198678314686,
+            0.0223282128572464,
+            -0.027435027062892914,
+            -0.007861592806875706,
+            -0.009762551635503769,
+            -0.010471384041011333,
+            0.01416053343564272,
+            0.042336612939834595,
+            -0.0006982600898481905,
+            0.028127748519182205,
+            0.010237791575491428,
+            0.0009162460919469595,
+            0.02372976765036583,
+            -0.013604744337499142,
+            0.009738387539982796,
+            -0.004172443877905607,
+            -0.0050745937041938305,
+            -0.009198708459734917,
+            0.004587271716445684,
+            -0.0017318058526143432,
+            0.01678643375635147,
+            -0.00408383971080184,
+            0.0025614616461098194,
+            -0.019428445026278496,
+            -0.019815079867839813,
+            0.006427818909287453,
+            -0.012203189544379711,
+            -0.018284646794199944,
+            -0.017656365409493446,
+            0.00240640458650887,
+            -0.0062143635004758835,
+            0.0012867719633504748,
+            -0.007704521995037794,
+            -0.024486929178237915,
+            0.01844574511051178,
+            -0.0038764257915318012,
+            0.0015425154706463218,
+            0.021635491400957108,
+            -0.017801353707909584,
+            -0.014821035787463188,
+            0.016866983845829964,
+            -0.010898293927311897,
+            0.022747067734599113,
+            -0.025985142216086388,
+            -0.0026661755982786417,
+            0.0012575729051604867,
+            -0.0069594429805874825,
+            -0.015223781578242779,
+            -0.015578197315335274,
+            -0.0008940951083786786,
+            0.026210680603981018,
+            0.003989194519817829,
+            -0.011405753903090954,
+            -0.01157490722835064,
+            0.01880016177892685,
+            0.021796587854623795,
+            -0.007680357433855534,
+            0.0020117140375077724,
+            0.005139033310115337,
+            0.0028051226399838924,
+            -0.009287312626838684,
+            0.02303704433143139,
+            0.0323324128985405,
+            -0.005553861148655415,
+            -0.000351898925146088,
+            0.01526405569165945,
+            -0.006125759799033403,
+            -0.005896194837987423,
+            0.01063248235732317,
+            0.001617023372091353,
+            0.007877702824771404,
+            0.008852346800267696,
+            -0.025695165619254112,
+            -0.013459756039083004,
+            0.02210267446935177,
+            -0.006351297255605459,
+            5.386721750255674e-05,
+            -0.005948551464825869,
+            -0.025775713846087456,
+            0.021087756380438805,
+            0.001424712361767888,
+            0.013798062689602375,
+            -0.008050883188843727,
+            -0.027692783623933792,
+            -0.0038784395437687635,
+            -0.018864601850509644,
+            -0.00020942768605891615,
+            0.01374973263591528,
+            -0.013290602713823318,
+            -0.010406944900751114,
+            0.0018183961510658264,
+            0.0030628798995167017,
+            -0.00862680934369564,
+            -0.013886665925383568,
+            0.005892167333513498,
+            0.0020640708971768618,
+            -0.014104149304330349,
+            -0.040338993072509766,
+            -0.008236146531999111,
+            -0.02207045629620552,
+            0.010737196542322636,
+            0.028578823432326317,
+            -0.012992571108043194,
+            -0.013540305197238922,
+            -0.005054456647485495,
+            0.02712894044816494,
+            0.0071527608670294285,
+            -0.006621136795729399,
+            0.008328777737915516,
+            0.006496285554021597,
+            0.024841345846652985,
+            -0.007434682920575142,
+            -0.0007128596189431846,
+            0.0016160164959728718,
+            0.005642464850097895,
+            -0.008973171003162861,
+            -0.016343414783477783,
+            0.0005160177242942154,
+            -0.012082366272807121,
+            -0.02630733884871006,
+            0.008812072686851025,
+            0.021909357979893684,
+            -0.006766125094145536,
+            0.001452904543839395,
+            0.010028364136815071,
+            0.016866983845829964,
+            -0.017994670197367668,
+            0.004301322158426046,
+            0.00243862415663898,
+            -0.007495094556361437,
+            -0.01996006816625595,
+            0.010423054918646812,
+            0.001549563487060368,
+            0.01913846842944622,
+            0.004414091352373362,
+            0.0016411880496889353,
+            -0.008352942764759064,
+            0.009440355934202671,
+            -0.008264338597655296,
+            -0.0015143232885748148,
+            0.017318058758974075,
+            0.0399201363325119,
+            0.017463047057390213,
+            0.0328962542116642,
+            -0.03708481043577194,
+            -0.026983952149748802,
+            0.011051337234675884,
+            0.014724376611411572,
+            0.007197062950581312,
+            -0.003234046744182706,
+            -0.015417098999023438,
+            0.030930858105421066,
+            0.008618754334747791,
+            0.005271939095109701,
+            -0.009287312626838684,
+            0.007861592806875706,
+            -0.007096376270055771,
+            0.05271133780479431,
+            0.00677820760756731,
+            -0.014643827453255653,
+            0.016182316467165947,
+            -0.016142040491104126,
+            -0.013701403513550758,
+            -0.005654547363519669,
+            -0.010439164005219936,
+            0.013314767740666866,
+            -0.018977370113134384,
+            0.008377106860280037,
+            0.014031654223799706,
+            0.010294175706803799,
+            0.0036508883349597454,
+            0.012299848720431328,
+            0.014015545137226582,
+            0.012541496194899082,
+            0.01657700724899769,
+            0.0003000454162247479,
+            -0.029674291610717773,
+            -0.004651710856705904,
+            -0.002055009128525853,
+            0.0091020492836833,
+            -0.0211199764162302,
+            -0.008707358501851559,
+            0.0003111209371127188,
+            -0.0018264510435983539,
+            0.002591667464002967,
+            -0.028917130082845688,
+            -0.011437973007559776,
+            -0.038341376930475235,
+            0.027306146919727325,
+            0.007950196973979473,
+            0.021893247961997986,
+            0.002742697251960635,
+            -0.028449945151805878,
+            0.010302230715751648,
+            0.02242487110197544,
+            0.0193962249904871,
+            -0.012775088660418987,
+            -0.013411426916718483,
+            -0.030350904911756516,
+            0.0030266328249126673,
+            0.010535823181271553,
+            0.0009686030098237097,
+            0.009561179205775261,
+            0.005984798539429903,
+            -0.0205722413957119,
+            0.009786716662347317,
+            -0.027869991958141327,
+            0.01725361868739128,
+            -0.04945715144276619,
+            0.0055739982053637505,
+            -0.01068081147968769,
+            -0.02092665806412697,
+            -0.005392762832343578,
+            -0.007648137863725424,
+            0.028208298608660698,
+            -0.005690794438123703,
+            -0.022457091137766838,
+            0.016609225422143936,
+            0.016029272228479385,
+            -0.016295084729790688,
+            0.009488685056567192,
+            0.01062442734837532,
+            0.003735464997589588,
+            -0.001605947851203382,
+            0.011123832315206528,
+            0.03215520456433296,
+            0.0035602706484496593,
+            0.01833297684788704,
+            0.010664702393114567,
+            0.009247037582099438,
+            0.006504340562969446,
+            0.0014307535020634532,
+            0.030834197998046875,
+            -0.005235692020505667,
+            0.0060532656498253345,
+            0.007164843380451202,
+            0.001605947851203382,
+            0.015223781578242779,
+            0.001339128939434886,
+            -0.0020117140375077724,
+            0.009730332531034946,
+            -0.010793580673635006,
+            0.010414999909698963,
+            -0.020024508237838745,
+            -0.00027915299870073795,
+            0.01589233987033367,
+            -0.011083557270467281,
+            -0.024116402491927147,
+            -0.00019784874166361988,
+            -0.0164642371237278,
+            0.009182598441839218,
+            0.0017267714720219374,
+            -0.011325204744935036,
+            -0.0051672253757715225,
+            -0.0016885106451809406,
+            -0.005880084820091724,
+            0.022875946015119553,
+            -0.01215486042201519,
+            -0.019702311605215073,
+            -0.018735721707344055,
+            -0.008852346800267696,
+            0.011977652087807655,
+            0.03162357956171036,
+            -0.015473484061658382,
+            0.00608145771548152,
+            -0.006762097589671612,
+            0.00014549182378686965,
+            0.027773331850767136,
+            -0.0032642525620758533,
+            -0.0074467649683356285,
+            0.013540305197238922,
+            0.01762414537370205,
+            0.005259856581687927,
+            -0.007132623344659805,
+            -0.011800444684922695,
+            -0.012823417782783508,
+            -0.003695190418511629,
+            0.012815362773835659,
+            -0.006037155631929636,
+            -0.017994670197367668,
+            0.04298100247979164,
+            -0.023053154349327087,
+            0.0038663572631776333,
+            -0.00921481754630804,
+            -0.004893358331173658,
+            -0.0025393106043338776,
+            0.010431109927594662,
+            -0.006480175536125898,
+            -0.04336763918399811,
+            -0.043399859219789505,
+            0.007664247881621122,
+            -0.006645301356911659,
+            -0.0004505715623963624,
+            0.047395095229148865,
+            0.010278066620230675,
+            -0.02946486324071884,
+            0.01726972870528698,
+            -0.025195760652422905,
+            -0.006274775601923466,
+            -0.0050745937041938305,
+            -0.022714847698807716,
+            0.009343696758151054,
+            0.0008794955792836845,
+            -0.011292984709143639,
+            -0.012960351072251797,
+            -0.022231554612517357,
+            -0.009053720161318779,
+            -0.001134735532104969,
+            0.012581770308315754,
+            0.004325487185269594,
+            -0.0036267235409468412,
+            -0.026758413761854172,
+            0.017285838723182678,
+            -0.01168767549097538,
+            0.0012213258305564523,
+            -0.000624255568254739,
+            0.00898122601211071,
+            0.019637873396277428,
+            0.016238700598478317,
+            -0.02360088936984539,
+            -0.017124740406870842,
+            -0.026516767218708992,
+            0.017060300335288048,
+            0.00505042914301157,
+            0.026371778920292854,
+            -0.012436782009899616,
+            0.003979125991463661,
+            -0.020846109837293625,
+            -0.013693348504602909,
+            -0.008642919361591339,
+            -0.014031654223799706,
+            0.005827728193253279,
+            -0.003018577815964818,
+            0.03421726077795029,
+            -0.0012364288559183478,
+            -0.007434682920575142,
+            0.004824891686439514,
+            0.008618754334747791,
+            0.025904593989253044,
+            -0.021796587854623795,
+            -0.017189180478453636,
+            -0.011292984709143639,
+            0.010978843085467815,
+            0.005650519859045744,
+            -0.016883093863725662,
+            -0.02548573724925518,
+            -0.016423963010311127,
+            -0.002593681216239929,
+            -0.024148622527718544,
+            0.006230473518371582,
+            0.02970651164650917,
+            0.014442455023527145,
+            0.002138578798621893,
+            -0.01596483401954174,
+            0.03007703647017479,
+            0.010938568972051144,
+            0.011800444684922695,
+            -0.008022691123187542,
+            0.017930231988430023,
+            -0.020266154780983925,
+            0.005296103656291962,
+            -0.024019744247198105,
+            -0.017301948741078377,
+            -0.013250328600406647,
+            0.03511941060423851,
+            -0.010551933199167252,
+            -0.018735721707344055,
+            0.004748370032757521,
+            -0.005187362432479858,
+            0.01678643375635147,
+            -0.01514323242008686,
+            -0.021103866398334503,
+            0.025195760652422905,
+            0.007535369135439396,
+            0.012074311263859272,
+            -0.03136582300066948,
+            0.0022211417090147734,
+            -0.019122358411550522,
+            0.0024205006193369627,
+            -0.012017927132546902,
+            0.0015173438005149364,
+            0.004639628808945417,
+            0.00766827492043376,
+            0.004305349662899971,
+            0.024841345846652985,
+            0.004365761764347553,
+            0.021168306469917297,
+            -0.016512567177414894,
+            -0.0013491975842043757,
+            0.03724590688943863,
+            0.008026719093322754,
+            -0.001129701267927885,
+            0.007712577003985643,
+            0.018864601850509644,
+            0.006721823010593653,
+            -0.009553124196827412,
+            -0.00437784381210804,
+            -0.0011729963589459658,
+            0.023278692737221718,
+            0.01315366942435503,
+            0.0064600384794175625,
+            -0.0015102957841008902,
+            0.008747633546590805,
+            0.006681548431515694,
+            0.001100502209737897,
+            -0.003042742609977722,
+            -0.013894720934331417,
+            -0.020395034924149513,
+            -0.025082992389798164,
+            0.025421299040317535,
+            0.0031353740487247705,
+            -0.006335187237709761,
+            0.009657838381826878,
+            0.006315050181001425,
+            -0.012066256254911423,
+            0.02878825180232525,
+            -0.005718986503779888,
+            -0.024873564019799232,
+            -0.005598163232207298,
+            -0.003679080633446574,
+            -0.011913212947547436,
+            0.030624771490693092,
+            -0.00027487383340485394,
+            0.0059606339782476425,
+            0.01267842948436737,
+            -0.005489421542733908,
+            -0.021232744678854942,
+            -0.02102331817150116,
+            -0.025936812162399292,
+            -0.007619945798069239,
+            0.015014354139566422,
+            -7.803194603184238e-05,
+            0.028852690011262894,
+            0.0065003130584955215,
+            -0.008497931063175201,
+            -0.0058760573156178,
+            -0.023182032629847527,
+            -0.004478530492633581,
+            0.008489876054227352,
+            -0.017898011952638626,
+            0.002058029640465975,
+            -0.015344604849815369,
+            0.011349369771778584,
+            0.0006912120152264833,
+            -0.01327449269592762,
+            -0.002196976915001869,
+            -0.0005774364108219743,
+            -0.01473243162035942,
+            -0.011003008112311363,
+            -0.005553861148655415,
+            0.01256566122174263,
+            -0.01975064165890217,
+            -0.006447955965995789,
+            -0.011719895526766777,
+            0.019879519939422607,
+            -0.010414999909698963,
+            0.009448410011827946,
+            -0.03357286751270294,
+            0.00329043110832572,
+            -0.0003179172636009753,
+            -0.01622259058058262,
+            0.004114045761525631,
+            -0.0016613253392279148,
+            0.02360088936984539,
+            0.01420886255800724,
+            0.014812980778515339,
+            0.004994044546037912,
+            0.017173070460557938,
+            -0.0015143232885748148,
+            -0.002843383466824889,
+            -0.004998072050511837,
+            -0.00487724831327796,
+            -0.010672756470739841,
+            0.006588917225599289,
+            -0.003246129024773836,
+            0.019637873396277428,
+            0.004945715423673391,
+            -0.012928131967782974,
+            0.0066050267778337,
+            0.01656089723110199,
+            0.013258383609354496,
+            -0.00032295158598572016,
+            -0.00684264674782753,
+            0.01562652736902237,
+            0.06869228184223175,
+            -0.0011226532515138388,
+            0.01809133030474186,
+            0.016246754676103592,
+            -0.03191355615854263,
+            -0.022730957716703415,
+            0.01156685221940279,
+            -0.01575540564954281,
+            0.03418504074215889,
+            0.01109161227941513,
+            0.011204380542039871,
+            -0.014966024085879326,
+            0.00651642307639122,
+            0.01913846842944622,
+            0.0170119721442461,
+            -0.005863975267857313,
+            -0.006041183136403561,
+            -0.019170688465237617,
+            0.023101484403014183,
+            0.010769415646791458,
+            -0.006443928461521864,
+            0.004212718456983566,
+            0.008203926496207714,
+            0.003908645361661911,
+            -0.00013315775140654296,
+            0.01633535884320736,
+            -0.018477965146303177,
+            0.001423705485649407,
+            -0.003340774215757847,
+            0.014925749972462654,
+            -0.006298940163105726,
+            -0.01962176337838173,
+            0.0076400828547775745,
+            -0.0019301580032333732,
+            0.011204380542039871,
+            0.013645019382238388,
+            -0.0025232008192688227,
+            0.00663724634796381,
+            -0.003495831275358796,
+            0.0004875738231930882,
+            0.00816767942160368,
+            -0.01668977551162243,
+            0.003558256896212697,
+            0.0003438440035097301,
+            -0.022022126242518425,
+            0.010914403945207596,
+            0.014837145805358887,
+            0.012082366272807121,
+            0.0018032931257039309,
+            -0.008127405308187008,
+            -0.022505421191453934,
+            -0.008949005976319313,
+            0.003348829224705696,
+            0.0002846907591447234,
+            -0.025340748950839043,
+            -0.001157893449999392,
+            -0.012960351072251797,
+            -0.01726972870528698,
+            0.0056666298769414425,
+            0.012726759538054466,
+            0.004663793370127678,
+            -0.004430200904607773,
+            -0.0016019203467294574,
+            -0.0048329466953873634,
+            -0.01509490329772234,
+            0.0019784874748438597,
+            0.012718704529106617,
+            -0.003324664430692792,
+            0.014224972575902939,
+            0.022151004523038864,
+            0.004055647645145655,
+            0.0039026043377816677,
+            0.0030447563622146845,
+            4.613481451087864e-06,
+            0.01581178978085518,
+            0.01925123669207096,
+            -0.0019513021688908339,
+            0.0013199985260143876,
+            -0.00045661276089958847,
+            0.006464065983891487,
+            0.006472120992839336,
+            0.001662332215346396,
+            -0.011462138034403324,
+            0.003439446911215782,
+            -0.01902570016682148,
+            -0.013201998546719551,
+            0.005622327793389559,
+            0.0047805896028876305,
+            0.02147439308464527,
+            -0.01891293004155159,
+            0.0022553750313818455,
+            0.006544615142047405,
+            0.0072051179595291615,
+            -0.0009097015135921538,
+            -0.008409326896071434,
+            0.018268538638949394,
+            0.006149924360215664,
+            0.0034172958694398403,
+            -0.004168416373431683,
+            -0.011333259753882885,
+            0.007950196973979473,
+            0.00033704767702147365,
+            0.0019331786315888166,
+            -0.006383516825735569,
+            -0.013943050988018513,
+            -0.022618189454078674,
+            0.023649217560887337,
+            0.01726972870528698,
+            0.019798971712589264,
+            -0.01797856017947197,
+            0.006258665584027767,
+            -0.011019118130207062,
+            0.010414999909698963,
+            0.01167962048202753,
+            -0.0017408676212653518,
+            -0.004293267615139484,
+            -0.001821416663005948,
+            -0.021055536344647408,
+            -0.01749526709318161,
+            0.005626355297863483,
+            0.02055613324046135,
+            0.0001618533715372905,
+            -0.009826990775763988,
+            -0.029674291610717773,
+            -0.004389926325529814,
+            -0.005678711924701929,
+            -0.0199439600110054,
+            -0.01150241307914257,
+            -0.015392934903502464,
+            -0.015642637386918068,
+            -0.003228005487471819,
+            0.0022271827328950167,
+            -0.005578025709837675,
+            0.004023428075015545,
+            0.018574625253677368,
+            0.0013532249722629786,
+            0.020411144942045212,
+            0.010068638250231743,
+            0.017189180478453636,
+            0.013596689328551292,
+            -0.013411426916718483,
+            0.006282830610871315,
+            -0.009593399241566658,
+            -0.012050146237015724,
+            0.018026890233159065,
+            -0.014386070892214775,
+            -2.73583791567944e-05,
+            -0.023182032629847527,
+            0.001140776788815856,
+            -0.02395530417561531,
+            0.01892904005944729,
+            -0.034475017338991165,
+            -0.011961543001234531,
+            0.005743151530623436,
+            -0.01714085042476654,
+            -0.01949288509786129,
+            -0.028369395062327385,
+            -0.0008281455375254154,
+            -0.018059110268950462,
+            -0.00408383971080184,
+            -0.016770323738455772,
+            0.0012011885410174727,
+            -0.0028836580459028482,
+            0.010785525664687157,
+            -0.008497931063175201,
+            0.051390331238508224,
+            0.0029923992697149515,
+            0.005537751130759716,
+            -0.01263010036200285,
+            0.001191119896247983,
+            -0.012066256254911423,
+            0.013427536003291607,
+            -0.0016885106451809406,
+            -0.0038200414273887873,
+            0.008409326896071434,
+            0.010704976506531239,
+            -0.0032561977859586477,
+            0.004293267615139484,
+            0.02395530417561531,
+            0.02772500365972519,
+            -0.001674414612352848,
+            -0.0120340371504426,
+            0.004990017041563988,
+            0.009061774238944054,
+            0.014861310832202435,
+            0.008264338597655296,
+            -0.016625335440039635,
+            -0.01267842948436737,
+            0.013572524301707745,
+            0.005328323692083359,
+            -0.011832663789391518,
+            0.01868739351630211,
+            -0.013065065257251263,
+            -0.022714847698807716,
+            0.005392762832343578,
+            -0.00798241700977087,
+            0.012815362773835659,
+            -0.00915843341499567,
+            0.006625164300203323,
+            0.009770606644451618,
+            0.007088321726769209,
+            0.025276310741901398,
+            -0.004192580934613943,
+            -0.004216745961457491,
+            -0.015038518235087395,
+            -0.0017388538690283895,
+            -0.0009162460919469595,
+            0.027563905343413353,
+            0.02875603176653385,
+            0.008779852651059628,
+            -0.059960756450891495,
+            -0.003125305287539959,
+            0.016512567177414894,
+            -0.010527768172323704,
+            -0.0182202085852623,
+            0.015320440754294395,
+            -0.0021828808821737766,
+            0.002150661079213023,
+            0.01689920201897621,
+            0.009907539933919907,
+            -0.005340405739843845,
+            0.014845200814306736,
+            0.03531273081898689,
+            0.009432300925254822,
+            0.008203926496207714,
+            -0.007656192872673273,
+            -0.01865517348051071,
+            -0.029303764924407005,
+            -0.004961824975907803,
+            -0.004389926325529814,
+            0.001964391442015767,
+            0.004196608439087868,
+            -0.00904566515237093,
+            0.037890300154685974,
+            0.018735721707344055,
+            0.01904180832207203,
+            0.006238528527319431,
+            0.008393216878175735,
+            -0.02279539778828621,
+            -0.03805140033364296,
+            0.003969057463109493,
+            -0.024019744247198105,
+            0.0018687392584979534,
+            0.010729141533374786,
+            0.01962176337838173,
+            0.005022237077355385,
+            0.011889047920703888,
+            -0.008360997773706913,
+            0.004212718456983566,
+            -2.6917876311927103e-05,
+            0.005783426109701395,
+            -0.01599705219268799,
+            -0.01950899325311184,
+            -0.009013445116579533,
+            0.0043738167732954025,
+            0.010278066620230675,
+            -0.005408872850239277,
+            -0.005300131160765886,
+            0.005936469417065382,
+            -0.02443859912455082,
+            0.0029541384428739548,
+            -0.011993762105703354,
+            0.0037092864513397217,
+            -0.009464520029723644,
+            0.016947532072663307,
+            0.020233936607837677,
+            -0.00956923421472311,
+            -0.0017650322988629341,
+            0.0009832025971263647,
+            0.02300482615828514,
+            -0.009899485856294632,
+            0.01656089723110199,
+            0.005771343596279621,
+            0.029046008363366127,
+            0.01962176337838173,
+            0.015747351571917534,
+            -0.010382779873907566,
+            0.012936186976730824,
+            0.022618189454078674,
+            -0.007495094556361437,
+            0.013572524301707745,
+            -0.00857042521238327,
+            0.012710649520158768,
+            -0.006133814807981253,
+            -0.010704976506531239,
+            0.0034354194067418575,
+            0.01327449269592762,
+            0.011397698894143105,
+            0.003228005487471819,
+            0.02429361082613468,
+            -0.0053202686831355095,
+            -0.010374724864959717,
+            -0.019090138375759125,
+            0.0036468608304858208,
+            -0.0017569774063304067,
+            0.0043738167732954025,
+            0.01420886255800724,
+            -0.02337535098195076,
+            -0.000712356180883944,
+            0.01533654984086752,
+            -0.011599071323871613,
+            -0.001532446825876832,
+            -0.022618189454078674,
+            -0.01599705219268799,
+            0.009851155802607536,
+            -0.0015102957841008902,
+            -0.005968688987195492,
+            -0.024615807458758354,
+            0.01846185512840748,
+            -0.0029138640966266394,
+            0.002593681216239929,
+            0.036214880645275116,
+            0.008892621845006943,
+            0.039050206542015076,
+            -0.0068063996732234955,
+            0.012001817114651203,
+            -0.011292984709143639,
+            -0.011019118130207062,
+            -0.026001252233982086,
+            0.013604744337499142,
+            -0.01015724241733551,
+            0.019315676763653755,
+            0.009311476722359657,
+            0.006472120992839336,
+            -0.0026118047535419464,
+            0.0050745937041938305,
+            0.0008633857360109687,
+            -0.015779569745063782,
+            0.026001252233982086,
+            -0.02195768617093563,
+            0.028240516781806946,
+            0.005638437811285257,
+            -0.0120340371504426,
+            0.012823417782783508,
+            0.011139941401779652,
+            0.004776562098413706,
+            0.01051165908575058,
+            0.0030447563622146845,
+            -0.0054612294770777225,
+            0.005437064915895462,
+            -0.00634324224665761,
+            -0.04262658953666687,
+            -0.025807933881878853,
+            -0.006109649781137705,
+            -0.003950933925807476,
+            0.0075998082756996155,
+            -0.013419480994343758,
+            -0.021699929609894753,
+            0.0030064955353736877,
+            0.023745877668261528,
+            -0.021699929609894753,
+            0.011429918929934502,
+            0.0018697461346164346,
+            0.011655456386506557,
+            -0.01784968189895153,
+            0.013363096863031387,
+            -0.002575557678937912,
+            0.02996426820755005,
+            0.000515262596309185,
+            -0.0031635661143809557,
+            0.012517331168055534,
+            0.000442516669863835,
+            -0.021571051329374313,
+            -0.00028745963936671615,
+            -0.012452892027795315,
+            -0.02020171657204628,
+            0.02182880789041519,
+            -0.008115322329103947,
+            -0.006556697189807892,
+            -0.004216745961457491,
+            0.024116402491927147,
+            -0.0006927223294042051,
+            0.0011951474007219076,
+            -0.005863975267857313,
+            -0.011736005544662476,
+            0.0182040985673666,
+            -0.02688729204237461,
+            -0.0030306600965559483,
+            -0.007285666652023792,
+            0.0007954224711284041,
+            -0.005739124026149511,
+            -0.005275966599583626,
+            0.017189180478453636,
+            -0.017930231988430023,
+            0.005586080718785524,
+            -0.014426345005631447,
+            -0.003695190418511629,
+            0.0008608686039224267,
+            0.011019118130207062,
+            0.03157525137066841,
+            0.004091894719749689,
+            0.030850308015942574,
+            0.014329686760902405,
+            -0.008594590239226818,
+            0.0012535455171018839,
+            0.01878405176103115,
+            -0.010535823181271553,
+            0.02160327136516571,
+            -0.01308923028409481,
+            -0.008650974370539188,
+            0.007241365034133196,
+            -0.007632027845829725,
+            0.00458324421197176,
+            0.011260765604674816,
+            0.015409043990075588,
+            0.0014871378662064672,
+            0.0009711202001199126,
+            -0.016029272228479385,
+            0.010737196542322636,
+            -0.0046799033880233765,
+            -0.008916785940527916,
+            0.00578745361417532,
+            0.007269557099789381,
+            -0.014007490128278732,
+            0.0009691064478829503,
+            -0.007877702824771404,
+            0.0050141820684075356,
+            -0.01889682002365589,
+            -0.008940950967371464,
+            -0.024470819160342216,
+            -0.00391670037060976,
+            -0.013612799346446991,
+            0.005650519859045744,
+            0.001747915637679398,
+            -0.005952578969299793,
+            0.016174260526895523,
+            0.015844009816646576,
+            0.005924386903643608,
+            0.006045210640877485,
+            -0.02666175551712513,
+            -0.0038764257915318012,
+            0.0007294728420674801,
+            -0.0013038887409493327,
+            0.0016653528437018394,
+            -0.011244655586779118,
+            -0.011421863920986652,
+            -0.018703503534197807,
+            -0.010189462453126907,
+            -0.0027084636967629194,
+            0.014514949172735214,
+            -0.0037233824841678143,
+            -0.0014196779811754823,
+            -0.0058760573156178,
+            0.006971525494009256,
+            0.012509276159107685,
+            0.022618189454078674,
+            -0.014498839154839516,
+            -0.0132100535556674,
+            0.015417098999023438,
+            0.03197799623012543,
+            -0.007998526096343994,
+            0.008066993206739426,
+            -0.010141132399439812,
+            -0.00607340270653367,
+            -0.005578025709837675,
+            -0.01809133030474186,
+            0.010616372339427471,
+            0.00018186478700954467,
+            0.02548573724925518,
+            0.00578745361417532,
+            -0.004013359546661377,
+            0.007813263684511185,
+            0.010350560769438744,
+            0.012082366272807121,
+            0.00754745164886117,
+            0.030512001365423203,
+            0.0005930428160354495,
+            0.02360088936984539,
+            -0.0012686484260484576,
+            -0.010938568972051144,
+            -0.0021869083866477013,
+            0.0017297921003773808,
+            0.00484502874314785,
+            0.010076693259179592,
+            0.0281760785728693,
+            0.0035220098216086626,
+            -0.004772534593939781,
+            0.01633535884320736,
+            0.006391571834683418,
+            -0.0005880084936507046,
+            -0.012718704529106617,
+            -0.01227568369358778,
+            -0.006335187237709761,
+            -0.0007934087188914418,
+            0.002002652268856764,
+            -0.016440073028206825,
+            -0.03550604730844498,
+            0.010020309127867222,
+            0.006089512724429369,
+            -0.016995862126350403,
+            -0.01773691363632679,
+            -0.006178116425871849,
+            0.019122358411550522,
+            0.00040048008668236434,
+            -0.009061774238944054,
+            -0.0024305693805217743,
+            0.007575643714517355,
+            0.022263772785663605,
+            -0.0014569320483133197,
+            0.031075846403837204,
+            -0.015860119834542274,
+            -0.01479687076061964,
+            0.012960351072251797,
+            0.0024245281238108873,
+            0.008288503624498844,
+            -0.025195760652422905,
+            0.010125023312866688,
+            -0.006931250914931297,
+            0.022747067734599113,
+            0.007559533696621656,
+            0.010390834882855415,
+            -0.012251519598066807,
+            -0.006242556031793356,
+            0.016367578878998756,
+            -0.010906348936259747,
+            -0.004708095453679562,
+            -0.003236060496419668,
+            -0.026242898777127266,
+            0.0012092434335500002,
+            0.011526577174663544,
+            -0.0015515772392973304,
+            0.015803735703229904,
+            -0.006886948831379414,
+            -0.002825259929522872,
+            -0.0003078486188314855,
+            -0.00019105241517536342,
+            -0.008272393606603146,
+            -0.041982196271419525,
+            -0.002799081616103649,
+            0.02078166976571083,
+            -0.005102786235511303,
+            -0.02210267446935177,
+            0.01725361868739128,
+            0.003308554645627737,
+            0.0036851216573268175,
+            -0.01328254770487547,
+            -0.021571051329374313,
+            -0.015827899798750877,
+            0.01308923028409481,
+            -0.018252428621053696,
+            -0.023536449298262596,
+            -0.0016804557526484132,
+            -0.017946342006325722,
+            0.002428555628284812,
+            -0.02197379618883133,
+            -0.02055613324046135,
+            -0.021780479699373245,
+            -0.005497476551681757,
+            0.004192580934613943,
+            -0.025662945583462715,
+            0.026371778920292854,
+            -0.008441546931862831,
+            -0.006149924360215664,
+            -0.014861310832202435,
+            0.008892621845006943,
+            0.0017126754391938448,
+            0.021764369681477547,
+            -0.02679063379764557,
+            0.009585344232618809,
+            -0.0023681437596678734,
+            -0.015417098999023438,
+            -0.005944523960351944,
+            -0.0015868174377828836,
+            0.009673947468400002,
+            0.03499053418636322,
+            0.006524477619677782,
+            -0.021877137944102287,
+            -0.023343130946159363,
+            0.012847582809627056,
+            0.003991208504885435,
+            -0.010801635682582855,
+            -0.010737196542322636,
+            -0.01715696044266224,
+            -0.015425154007971287,
+            0.014579388312995434,
+            0.01367723848670721,
+            0.018107440322637558,
+            -0.0032380742486566305,
+            0.008199899457395077,
+            -0.01880016177892685,
+            -0.005747179035097361,
+            0.0033790350425988436,
+            0.006262693088501692,
+            -0.018397416919469833,
+            0.035892684012651443,
+            0.005332351196557283,
+            -0.004454365465790033,
+            -0.00581161817535758,
+            0.0052276370115578175,
+            0.0028333149384707212,
+            -0.008844291791319847,
+            -0.01316172443330288,
+            0.00737427081912756,
+            -0.009085939265787601,
+            0.007003745064139366,
+            0.009134269319474697,
+            -0.005541778635233641,
+            -0.019895629957318306,
+            -0.003364939009770751,
+            0.011727950535714626,
+            -0.017237508669495583,
+            -0.01110772229731083,
+            -0.020523913204669952,
+            -0.003024619072675705,
+            0.003979125991463661,
+            -0.00014410738367587328,
+            0.0018687392584979534,
+            -0.01639174297451973,
+            -0.004176471382379532,
+            -0.02911044843494892,
+            0.0008155597024597228,
+            -0.02182880789041519,
+            -0.018848491832613945,
+            0.011921267956495285,
+            0.00975449662655592,
+            0.00604923814535141,
+            0.003705258946865797,
+            0.00047549145529046655,
+            0.014120258390903473,
+            0.002132537541911006,
+            -0.016528677195310593,
+            0.022408761084079742,
+            -0.0063593522645533085,
+            0.015594307333230972,
+            0.029400425031781197,
+            0.01380611676722765,
+            -0.00687486631795764,
+            -0.017801353707909584,
+            -0.01003641914576292,
+            -0.02210267446935177,
+            0.0047685070894658566,
+            -0.01772080361843109,
+            -0.024277500808238983,
+            0.014635773375630379,
+            -0.023069264367222786,
+            0.0011488316813483834,
+            -0.01693142205476761,
+            0.006737933028489351,
+            -0.00637546181678772,
+            0.012597880326211452,
+            0.017833571881055832,
+            0.0016703871078789234,
+            0.0027789443265646696,
+            -0.011880993843078613,
+            -0.0036106137558817863,
+            0.006689603440463543,
+            -0.005175280384719372,
+            -0.004905440844595432,
+            0.001707641058601439,
+            -0.010922458954155445,
+            -0.019766751676797867,
+            -0.012775088660418987,
+            -0.025646835565567017,
+            -0.0010632482590153813,
+            0.0049215503968298435,
+            0.006246583536267281,
+            -0.004273130092769861,
+            -0.00355221563950181,
+            0.0033448017202317715,
+            -0.008594590239226818,
+            0.0031917584128677845,
+            -0.0010144152911379933,
+            -0.005900222342461348,
+            0.006170061882585287,
+            0.0034515291918069124,
+            0.0030387151055037975,
+            0.01427330169826746,
+            0.00243862415663898,
+            -0.005171252880245447,
+            -0.038244716823101044,
+            0.0062546380795538425,
+            -0.018848491832613945,
+            0.001827457919716835,
+            -0.011446028016507626,
+            7.104683027137071e-05,
+            -0.006717795506119728,
+            -0.006967497989535332,
+            0.016770323738455772,
+            0.003882467048242688,
+            0.010116968303918839,
+            0.015940668061375618,
+            -0.010600262321531773,
+            -0.008949005976319313,
+            0.012501221150159836,
+            -9.49441164266318e-06,
+            -0.00481280917301774,
+            0.0016542773228138685,
+            -0.01566680148243904,
+            0.008916785940527916,
+            -0.00951285008341074,
+            0.009593399241566658,
+            -0.012944241985678673,
+            0.007386353332549334,
+            -0.008731523528695107,
+            -0.011405753903090954,
+            0.00024882121942937374,
+            0.008763742633163929,
+            0.01581178978085518,
+            0.003097113221883774,
+            0.006025073118507862,
+            0.0035602706484496593,
+            0.02147439308464527,
+            -0.01255760621279478,
+            0.0009464520262554288,
+            -0.005465256981551647,
+            -0.023455901071429253,
+            -0.015844009816646576,
+            0.025324638932943344,
+            0.03937240317463875,
+            -0.011711840517818928,
+            -0.005429009906947613,
+            0.0011880993843078613,
+            0.009085939265787601,
+            0.0023983498103916645,
+            -0.004490613006055355,
+            -0.01309728529304266,
+            -0.02608180046081543,
+            -0.02018560655415058,
+            0.004555052146315575,
+            0.01760803535580635,
+            -0.010721086524426937,
+            -0.004013359546661377,
+            -0.005940496921539307,
+            -0.00431340467184782,
+            -0.0028534522280097008,
+            -0.008642919361591339,
+            -0.00551761407405138,
+            -0.005723014008253813,
+            -0.004514777567237616,
+            -0.022151004523038864,
+            0.009424245916306973,
+            0.011663510464131832,
+            0.008369052782654762,
+            0.0028172051534056664,
+            0.020829999819397926,
+            -0.02453525923192501,
+            -0.0067057134583592415,
+            -0.000449816434411332,
+            -0.009786716662347317,
+            0.02500244230031967,
+            0.00798241700977087,
+            -0.016947532072663307,
+            0.0012626072857528925,
+            0.006363379769027233,
+            0.00024693337036296725,
+            0.018945150077342987,
+            -0.010439164005219936,
+            0.021554941311478615,
+            -0.011486303061246872,
+            -0.004454365465790033,
+            -0.0022734985686838627,
+            0.006882921326905489,
+            0.03228408098220825,
+            0.012324013747274876,
+            -0.00950479507446289,
+            0.01068886648863554,
+            0.017688583582639694,
+            0.011437973007559776,
+            -0.009947814978659153,
+            0.021989906206727028,
+            -0.004196608439087868,
+            0.015296275727450848,
+            0.016528677195310593,
+            0.008344887755811214,
+            -0.011454083025455475,
+            -0.008465711027383804,
+            0.022167114540934563,
+            0.003783794352784753,
+            -0.02664564549922943,
+            -0.010012254118919373,
+            0.0029319876339286566,
+            0.023552559316158295,
+            0.007712577003985643,
+            0.0028756032697856426,
+            0.001929151127114892,
+            0.024631917476654053,
+            -0.005662602372467518,
+            0.027515575289726257,
+            -0.016754213720560074,
+            0.004542969632893801,
+            -0.001554597751237452,
+            0.014200807549059391,
+            -0.01831686682999134,
+            0.001941233524121344,
+            -0.0036448470782488585,
+            0.004631573799997568,
+            0.008892621845006943,
+            0.007120541296899319,
+            0.019283456727862358,
+            -0.004933632910251617,
+            -0.013258383609354496,
+            0.008457656018435955,
+            0.008868456818163395,
+            -0.01736638695001602,
+            0.027547795325517654,
+            0.020378924906253815,
+            0.00798241700977087,
+            -0.016866983845829964,
+            0.005130978301167488,
+            0.04043565317988396,
+            0.018606843426823616,
+            0.013016736134886742,
+            -0.010358615778386593,
+            -0.022634299471974373,
+            0.007587726227939129,
+            0.009359806776046753,
+            0.0024144595954567194,
+            0.004369789268821478,
+            0.004164388868957758,
+            -0.014321631751954556,
+            0.0001999883388634771,
+            0.03592490404844284,
+            -0.010785525664687157,
+            -0.01599705219268799,
+            -0.004301322158426046,
+            -0.011727950535714626,
+            -0.018477965146303177,
+            0.010414999909698963,
+            0.004865166265517473,
+            -0.020362814888358116,
+            0.00950479507446289,
+            0.001946267788298428,
+            -0.0164481271058321,
+            0.0041281417943537235,
+            0.0032199507113546133,
+            -0.003926768898963928,
+            -0.004841001238673925,
+            0.001975466962903738,
+            -0.009118159301578999,
+            0.005702876951545477,
+            -0.0011528590694069862,
+            -0.003743519773706794,
+            0.010052529163658619,
+            0.01654478721320629,
+            0.0024547341745346785,
+            0.010431109927594662,
+            0.0019734532106667757,
+            0.013298657722771168,
+            -0.018300756812095642,
+            0.00657280720770359,
+            -0.005972716491669416,
+            0.001719723455607891,
+            -0.0024104320909827948,
+            -0.005429009906947613,
+            -0.03685927018523216,
+            0.0029984405264258385,
+            -0.0011961542768403888,
+            0.014450510032474995,
+            -0.01736638695001602,
+            0.00032269986695609987,
+            0.01309728529304266,
+            -0.014361905865371227,
+            -0.00022830638044979423,
+            0.01846185512840748,
+            0.001276703318580985,
+            0.02279539778828621,
+            0.012315958738327026,
+            0.013371151871979237,
+            0.002072125906124711,
+            -0.02326258271932602,
+            0.02442248910665512,
+            -0.009915594942867756,
+            0.004458392970263958,
+            0.0211360864341259,
+            0.03312179446220398,
+            -0.021168306469917297,
+            -0.008546260185539722,
+            -0.015344604849815369,
+            -0.002984344493597746,
+            0.021667709574103355,
+            0.0040757847018539906,
+            0.028192188590765,
+            0.013226163573563099,
+            0.01648840308189392,
+            -0.025936812162399292,
+            0.0021244827657938004,
+            0.014378015883266926,
+            -0.007696467451751232,
+            -0.017930231988430023,
+            -0.01736638695001602,
+            -0.013234218582510948,
+            0.010648592375218868,
+            -0.02149050123989582,
+            0.026484547182917595,
+            -0.0009686030098237097,
+            0.01678643375635147,
+            0.009013445116579533,
+            -0.010471384041011333,
+            0.004929605405777693,
+            0.00627880310639739,
+            0.0037636570632457733,
+            -0.01726972870528698,
+            0.00510681327432394,
+            -0.007712577003985643,
+            0.0061539518646895885,
+            -0.004043565131723881,
+            0.029335984960198402,
+            -0.008010609075427055,
+            0.008489876054227352,
+            -0.011969597078859806,
+            -0.011309094727039337,
+            0.011317149735987186,
+            0.000714873312972486,
+            -0.012404562905430794,
+            0.0065123955719172955,
+            -0.002897754078730941,
+            0.01455522421747446,
+            -0.015932613983750343,
+            0.00405162014067173,
+            0.030334794893860817,
+            -0.01566680148243904,
+            -0.009327586740255356,
+            0.0030326738487929106,
+            0.0057673160918056965,
+            0.0034656254574656487,
+            0.0019684189464896917,
+            0.012847582809627056,
+            -0.00378983560949564,
+            -0.010946623980998993,
+            0.012348177842795849,
+            -0.011486303061246872,
+            0.008530150167644024,
+            0.0016834763810038567,
+            -0.008804017677903175,
+            0.029046008363366127,
+            -0.010133077390491962,
+            0.01257371623069048,
+            0.021087756380438805,
+            -0.023069264367222786,
+            -0.0077528515830636024,
+            -0.0105599882081151,
+            -0.019074028357863426,
+            -0.011518522165715694,
+            -0.005022237077355385,
+            0.00107231002766639,
+            0.0010400903411209583,
+            0.0060129910707473755,
+            3.240842852392234e-05,
+            0.004784617107361555,
+            0.0011589003261178732,
+            -0.0033689665142446756,
+            -0.010914403945207596,
+            0.03006092645227909,
+            -0.018977370113134384,
+            0.024728575721383095,
+            -0.00021584643400274217,
+            0.016947532072663307,
+            0.0017348263645544648,
+            -0.008034773170948029,
+            0.014136368408799171,
+            0.018300756812095642,
+            -0.019589543342590332,
+            -0.008123377338051796,
+            0.004619491286575794,
+            -0.0026601343415677547,
+            -0.009931704960763454,
+            -0.011760169640183449,
+            0.0038280964363366365,
+            -0.028949350118637085,
+            -0.0044221458956599236,
+            -0.011325204744935036,
+            -0.0035441608633846045,
+            -0.00777701660990715,
+            -0.01833297684788704,
+            -0.009118159301578999,
+            -0.01216291543096304,
+            -0.012009872123599052,
+            -0.008280448615550995,
+            -0.018945150077342987,
+            0.006186171434819698,
+            -0.02395530417561531,
+            0.014990189112722874,
+            -0.014200807549059391,
+            -0.005147087853401899,
+            0.015650691464543343,
+            0.03254184126853943,
+            0.006391571834683418,
+            0.0010622413828969002,
+            0.0016713939839974046,
+            0.004269102588295937,
+            0.0031696073710918427,
+            -0.0152721107006073,
+            -0.016641445457935333,
+            -0.0032380742486566305,
+            -0.001123660011216998,
+            0.02020171657204628,
+            0.01846185512840748,
+            -0.0036972041707485914,
+            -0.005384707823395729,
+            0.00869930349290371,
+            -0.002859493251889944,
+            -0.008042828179895878,
+            0.005324296187609434,
+            0.003431391902267933,
+            0.029642071574926376,
+            -0.011808499693870544,
+            -0.006077430211007595,
+            0.016085656359791756,
+            0.008457656018435955,
+            0.006798344664275646,
+            -0.010841909795999527,
+            0.00798241700977087,
+            0.006484203040599823,
+            0.017527485266327858,
+            0.0111641064286232,
+            0.0035723529290407896,
+            -0.010173352435231209,
+            -0.010817745700478554,
+            0.0006177109898999333,
+            0.006979580037295818,
+            -0.004446310922503471,
+            -0.022022126242518425,
+            0.0016673665959388018,
+            -0.0008487862069159746,
+            -0.021748259663581848,
+            -0.017060300335288048,
+            0.032815705984830856,
+            0.012879802845418453,
+            0.009867265820503235,
+            -0.0027708893176168203,
+            0.022505421191453934,
+            0.008449601009488106,
+            0.0018314853077754378,
+            0.008812072686851025,
+            0.0012213258305564523,
+            0.006395599339157343,
+            0.003228005487471819,
+            -0.03595712035894394,
+            0.004800726659595966,
+            -0.0024023770820349455,
+            0.006762097589671612,
+            -0.0031514838337898254,
+            0.004269102588295937,
+            -0.0044745029881596565,
+            -0.010390834882855415,
+            -0.02092665806412697,
+            0.022054346278309822,
+            0.015634581446647644,
+            0.021587161347270012,
+            -0.005437064915895462,
+            -0.0020902494434267282,
+            0.007913949899375439,
+            0.025195760652422905,
+            0.0005698849563486874,
+            0.01633535884320736,
+            0.02372976765036583,
+            0.014893529936671257,
+            0.020169496536254883,
+            -0.0031837034039199352,
+            0.04001679643988609,
+            -0.00114581105299294,
+            -0.0026359695475548506,
+            -0.0025513931177556515,
+            -0.010495549067854881,
+            0.01315366942435503,
+            -7.123561954358593e-05,
+            -0.005038346629589796,
+            0.006029100622981787,
+            0.0003780773840844631,
+            0.011639346368610859,
+            -0.01068081147968769,
+            -0.010978843085467815,
+            0.0021808671299368143,
+            0.019412335008382797,
+            0.01227568369358778,
+            0.024116402491927147,
+            -0.021345512941479683,
+            0.005904249846935272,
+            -0.027209488674998283,
+            -0.012968406081199646,
+            0.0032481427770107985,
+            0.004204663448035717,
+            -0.01456327922642231,
+            -0.021877137944102287,
+            -0.009802826680243015,
+            0.011429918929934502,
+            0.007031937129795551,
+            0.010382779873907566,
+            0.0071004037745296955,
+            -0.0003302513505332172,
+            0.006717795506119728,
+            -0.01997617818415165,
+            -0.020475583150982857,
+            0.0019543226808309555,
+            0.0013421495677903295,
+            -0.022022126242518425,
+            -0.006991662550717592,
+            0.010302230715751648,
+            -6.689351721433923e-05,
+            -0.016472293063998222,
+            0.016254810616374016,
+            -0.01598094217479229,
+            -0.01109161227941513,
+            -0.0014670006930828094,
+            0.007688412442803383,
+            0.0024587614461779594,
+            0.015127122402191162,
+            -0.009553124196827412,
+            0.009609508328139782,
+            0.011760169640183449,
+            -0.013234218582510948,
+            0.024454709142446518,
+            -0.0023399516940116882,
+            -0.015827899798750877,
+            0.016303138807415962,
+            -0.010060583241283894,
+            -0.02550184726715088,
+            0.025775713846087456,
+            -0.013983325101435184,
+            -0.010294175706803799,
+            -0.024631917476654053,
+            -0.019992288202047348,
+            -0.007845482788980007,
+            -0.01162323635071516,
+            -0.034926094114780426,
+            -0.018236318603157997,
+            0.029545413330197334,
+            0.018590733408927917,
+            0.0211360864341259,
+            0.00325821153819561,
+            0.003961002454161644,
+            0.013814171776175499,
+            -0.01566680148243904,
+            0.0010541864903643727,
+            0.0017771146958693862,
+            9.546327783027664e-05,
+            0.016963642090559006,
+            -0.001924116862937808,
+            -0.0028051226399838924,
+            -0.017914121970534325,
+            -0.020636681467294693,
+            0.0041885534301400185,
+            0.002146633807569742,
+            0.019235126674175262,
+            0.014659937471151352,
+            -0.02490578405559063,
+            0.005888139829039574,
+            -0.013403371907770634,
+            0.03428170084953308,
+            -0.01416053343564272,
+            0.022376542910933495,
+            0.00863486435264349,
+            0.02195768617093563,
+            0.0026681891176849604,
+            -0.00037128105759620667,
+            0.008779852651059628,
+            0.010334450751543045,
+            0.0034636117052286863,
+            0.002946083666756749,
+            -0.001038076588883996,
+            8.589807112002745e-05,
+            -0.004897385835647583,
+            0.03075364977121353,
+            0.012710649520158768,
+            -0.0016975724138319492,
+            -0.018171878531575203,
+            -0.009585344232618809,
+            -0.0062143635004758835,
+            -0.01725361868739128,
+            -0.010141132399439812,
+            0.029915938153862953,
+            0.003501872532069683,
+            0.0007470929995179176,
+            3.530316462274641e-05,
+            0.0032159232068806887,
+            0.028385505080223083,
+            -0.02337535098195076,
+            -0.009738387539982796,
+            -0.0036972041707485914,
+            -0.004168416373431683,
+            0.014974079094827175,
+            0.0021124002523720264,
+            -0.006709740497171879,
+            -0.0015425154706463218,
+            0.012549551203846931,
+            0.013846391811966896,
+            0.002241278998553753,
+            -0.006504340562969446,
+            -0.009295366704463959,
+            -0.022698739543557167,
+            0.0036911629140377045,
+            0.0006811433704569936,
+            0.0085140410810709,
+            0.010841909795999527,
+            -0.005698849447071552,
+            -0.02843383513391018,
+            -0.009432300925254822,
+            0.007390380837023258,
+            0.004071757197380066,
+            -0.0037314374931156635,
+            0.005984798539429903,
+            -0.006681548431515694,
+            0.006826536729931831,
+            0.019911739975214005,
+            0.024277500808238983,
+            0.020958878099918365,
+            0.007869647815823555,
+            0.020974988117814064,
+            0.028949350118637085,
+            -0.007015827111899853,
+            0.0006962463376112282,
+            0.00956923421472311,
+            0.014224972575902939,
+            0.0014670006930828094,
+            0.017334168776869774,
+            0.002881644293665886,
+            -0.0058075906708836555,
+            -0.01110772229731083,
+            -0.0008281455375254154,
+            0.0317041277885437,
+            -0.016327304765582085,
+            -0.00031187606509774923,
+            0.006141869351267815,
+            0.019412335008382797,
+            0.009714222513139248,
+            0.004675875883549452,
+            0.005316241178661585,
+            0.013483921065926552,
+            -0.010463329032063484,
+            -0.012710649520158768,
+            -0.0021345512941479683,
+            0.00498599000275135,
+            0.011671565473079681,
+            -0.01626286469399929,
+            0.0032300192397087812,
+            -0.021313294768333435,
+            0.009891430847346783,
+            -0.019090138375759125,
+            -0.011591016314923763,
+            0.0024668164551258087,
+            0.006544615142047405,
+            0.006298940163105726,
+            0.009271202608942986,
+            0.029545413330197334,
+            -0.0005583059974014759,
+            0.034475017338991165,
+            -0.005078621208667755,
+            -0.017785243690013885,
+            -0.016053438186645508,
+            0.00038210483035072684,
+            -0.020266154780983925,
+            -0.003217936959117651,
+            0.016399798914790154,
+            -0.007184980437159538,
+            -0.009746442548930645,
+            0.00451880507171154,
+            0.010253901593387127,
+            -0.0063593522645533085,
+            -0.005525668617337942,
+            -0.037278126925230026,
+            0.021458283066749573,
+            -0.0013814171543344855,
+            0.010326395742595196,
+            0.019122358411550522,
+            0.025807933881878853,
+            -0.01428135670721531,
+            -0.0026641618460416794,
+            0.0181879885494709,
+            0.003195785917341709,
+            0.023633107542991638,
+            -0.0047805896028876305,
+            0.008433491922914982,
+            -0.018300756812095642,
+            0.007732714526355267,
+            -0.022167114540934563,
+            -0.010616372339427471,
+            0.004651710856705904,
+            -0.027676673606038094,
+            0.0012646210379898548,
+            -0.0002251599362352863,
+            -0.013491976074874401,
+            -0.004764479584991932,
+            -0.0021667710971087217,
+            0.0012515317648649216,
+            -0.007555506192147732,
+            -0.014708267524838448,
+            -0.01639174297451973,
+            -0.027290038764476776,
+            0.00852209609001875,
+            0.0020701121538877487,
+            -0.003348829224705696,
+            0.0015102957841008902,
+            0.005247774533927441,
+            0.007229282520711422,
+            0.012815362773835659,
+            -0.007201090455055237,
+            0.060991786420345306,
+            -0.014047764241695404,
+            0.0020388993434607983,
+            -0.00508264871314168,
+            0.018606843426823616,
+            0.00868319347500801,
+            0.0027366559952497482,
+            -0.011832663789391518,
+            0.0030105228070169687,
+            -0.015473484061658382,
+            -0.013943050988018513,
+            -0.005932441912591457,
+            -5.896446600672789e-05,
+            0.002843383466824889,
+            -0.019702311605215073,
+            -0.0011528590694069862,
+            -0.01416053343564272,
+            2.218246845586691e-05,
+            0.002350020222365856,
+            -0.007986444048583508,
+            0.017785243690013885,
+            -0.018622953444719315,
+            -0.0085140410810709,
+            0.015167397446930408,
+            0.005179307889193296,
+            0.007881729863584042,
+            -0.013306712731719017,
+            -0.012533441185951233,
+            0.000388649437809363,
+            -0.001509288907982409,
+            -0.0013310740469023585,
+            -0.0091020492836833,
+            -0.008151569403707981,
+            -0.012654264457523823,
+            -0.005884112324565649,
+            -0.0018234304152429104,
+            0.0009761545225046575,
+            0.011236600577831268,
+            -0.008066993206739426,
+            0.005702876951545477,
+            -0.020298374816775322,
+            0.021635491400957108,
+            -0.003149470081552863,
+            -0.0211199764162302,
+            -0.008763742633163929,
+            0.01586817391216755,
+            0.0007712577353231609,
+            0.0006418757257051766,
+            0.03157525137066841,
+            0.004059675149619579,
+            -0.0011045295977964997,
+            -0.005453174468129873,
+            -0.0013743691379204392,
+            -0.007720632012933493,
+            0.020974988117814064,
+            -0.00757967121899128,
+            0.010398889891803265,
+            0.003677066881209612,
+            0.0211521964520216,
+            -0.025759605690836906,
+            0.006766125094145536,
+            -0.011003008112311363,
+            -0.0026077774818986654,
+            -0.00857042521238327,
+            -0.012863692827522755,
+            0.008868456818163395,
+            -0.02656509540975094,
+            -0.031043626368045807,
+            0.006262693088501692,
+            -0.012106531299650669,
+            -0.03785808011889458,
+            0.008216009475290775,
+            -0.018365196883678436,
+            -0.004393953830003738,
+            0.004365761764347553,
+            -0.014966024085879326,
+            -0.01726972870528698,
+            -0.012058201245963573,
+            -0.01609371230006218,
+            -0.0010884198127314448,
+            0.015634581446647644,
+            -0.02429361082613468,
+            -0.013757787644863129,
+            -0.006649328861385584,
+            0.007595780771225691,
+            -0.006371434312313795,
+            -0.0039247553795576096,
+            -0.02244098111987114,
+            0.007160815875977278,
+            -0.0059082768857479095,
+            0.01022168155759573,
+            -0.009673947468400002,
+            0.008264338597655296,
+            0.0252602007240057,
+            0.0036911629140377045,
+            -0.002927960129454732,
+            0.005002099554985762,
+            0.011792389675974846,
+            -0.0006484203040599823,
+            0.0293682049959898,
+            -0.007031937129795551,
+            0.031526919454336166,
+            0.0020439336076378822,
+            0.003568325424566865,
+            -0.015602362342178822,
+            -0.004655738361179829,
+            0.012622045353055,
+            0.008034773170948029,
+            -0.013717513531446457,
+            -0.003211895702406764,
+            -0.027789441868662834,
+            0.012871747836470604,
+            -0.023536449298262596,
+            -0.0039750984869897366,
+            0.005541778635233641,
+            -0.018252428621053696,
+            0.009400080889463425,
+            0.0028917130548506975,
+            0.03238074108958244,
+            0.017801353707909584,
+            -0.0016633390914648771,
+            0.002116427756845951,
+            0.007877702824771404,
+            -0.0020127207972109318,
+            0.019541213288903236,
+            0.004748370032757521,
+            -0.017285838723182678,
+            -0.015650691464543343,
+            -0.019573433324694633,
+            0.0007576650241389871,
+            0.007901867851614952,
+            -0.01051165908575058,
+            0.012412617914378643,
+            -0.0013975270558148623,
+            0.022134894505143166,
+            -0.00551761407405138,
+            -0.008876511827111244,
+            -0.01315366942435503,
+            -0.01268648449331522,
+            0.01602121815085411,
+            -0.001299861236475408,
+            0.016423963010311127,
+            0.007785071153193712,
+            0.010302230715751648,
+            -0.011115777306258678,
+            -0.00014310052210930735,
+            -0.01602121815085411,
+            -0.005195417441427708,
+            -0.007535369135439396,
+            -0.0008845299016684294,
+            0.03937240317463875,
+            -0.006230473518371582,
+            -0.005863975267857313,
+            0.0012736828066408634,
+            -0.0117521146312356,
+            0.024358050897717476,
+            0.013798062689602375,
+            0.022457091137766838,
+            -0.010414999909698963,
+            0.00035970209864899516,
+            -0.008175734430551529,
+            0.01842963509261608,
+            0.006665438879281282,
+            -0.0037314374931156635,
+            -0.011373533867299557,
+            -0.003958988469094038,
+            0.006492258049547672,
+            0.017543595284223557,
+            0.00019470229744911194,
+            -0.015972888097167015,
+            -0.004498667549341917,
+            0.004156333860009909,
+            0.004341596737504005,
+            0.009408135898411274,
+            -0.004095922224223614,
+            0.019444555044174194,
+            0.008155597373843193,
+            -0.016383688896894455,
+            -0.00754745164886117,
+            -0.015441264025866985,
+            0.0019059933256357908,
+            0.013338932767510414,
+            -0.015392934903502464,
+            0.009134269319474697,
+            0.005352488253265619,
+            -0.028611043468117714,
+            0.013886665925383568,
+            -0.012613990344107151,
+            -0.004760452080518007,
+            0.028095528483390808,
+            0.006383516825735569,
+            -0.02300482615828514,
+            -0.005424982402473688,
+            0.002648051828145981,
+            -0.014498839154839516,
+            0.02219933457672596,
+            -0.006439901422709227,
+            0.003471666481345892,
+            -0.007760906592011452,
+            -0.004869193769991398,
+            0.027531685307621956,
+            -0.00200970028527081,
+            -0.0009978021262213588,
+            0.007325941231101751
+          ],
+          "keyphrases": [
+            "Capped Expert Choice",
+            "entropy-regularized linear programming",
+            "validation perplexity",
+            "load balanced training",
+            "fine-tuning results"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "e96a5505-edee-45dd-be65-af56dee281e4",
+        "properties": {
+          "page_content": "D Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [? ]. We use mod x to map a token ID to an expert ID. This in some way ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 2. Hashing based routing performs much worse than expert choice in terms of average scores and variance.\n\n",
+          "title": "D Comparison With Hash Layer",
+          "embedding": [
+            -0.015004903078079224,
+            -0.027099307626485825,
+            -0.030763821676373482,
+            -0.012313068844377995,
+            0.005327117629349232,
+            -0.009055723436176777,
+            -0.007592933718115091,
+            0.055405039340257645,
+            -0.039389755576848984,
+            -0.0020019104704260826,
+            -0.016120845451951027,
+            -0.011890820227563381,
+            -0.0219267625361681,
+            0.00947797205299139,
+            -0.006043432280421257,
+            0.06357856839895248,
+            0.02605876699090004,
+            0.014650516211986542,
+            -0.0008935080841183662,
+            -0.05407797545194626,
+            0.01372307725250721,
+            0.008814438246190548,
+            -0.03872622177004814,
+            -0.0012384700821712613,
+            0.021429114043712616,
+            0.0030575317796319723,
+            -0.0006979353493079543,
+            -0.023012544959783554,
+            -0.03145751729607582,
+            0.046960070729255676,
+            0.024007845669984818,
+            0.01857893541455269,
+            -0.006804987788200378,
+            -0.05121271684765816,
+            -0.024294370785355568,
+            -0.013783399015665054,
+            0.046990230679512024,
+            0.038092851638793945,
+            -0.036946747452020645,
+            0.04632669687271118,
+            0.04659814387559891,
+            0.062311820685863495,
+            -0.03190992400050163,
+            -0.010179205797612667,
+            0.000586718088015914,
+            -0.0037851566448807716,
+            -0.014424311928451061,
+            -0.0021338630467653275,
+            0.03184960409998894,
+            -0.0449996292591095,
+            -0.011257447302341461,
+            -0.07594441622495651,
+            0.025395233184099197,
+            0.012343229725956917,
+            -0.005538241937756538,
+            0.050549183040857315,
+            0.0019906002562493086,
+            0.027808083221316338,
+            0.023600677028298378,
+            -0.00661648390814662,
+            -0.004599492996931076,
+            -0.014469552785158157,
+            -3.43724932463374e-05,
+            0.015306509099900723,
+            0.010194285772740841,
+            0.0011376205366104841,
+            -0.013157566078007221,
+            -0.008015181869268417,
+            0.04210421070456505,
+            -0.015427151694893837,
+            -0.039118312299251556,
+            0.031216232106089592,
+            -0.0038567879237234592,
+            0.0019679798278957605,
+            -0.028697820380330086,
+            -0.00024034235684666783,
+            0.021821200847625732,
+            0.03218137100338936,
+            -0.00124601018615067,
+            0.020252849906682968,
+            0.012456332333385944,
+            0.0006861538859084249,
+            -0.002107472624629736,
+            0.0172820296138525,
+            0.018880542367696762,
+            -0.03504662960767746,
+            -0.02144419401884079,
+            -0.06122603639960289,
+            -0.015442232601344585,
+            -0.018232088536024094,
+            -0.01572875864803791,
+            -0.015909722074866295,
+            -0.007057582959532738,
+            -0.0011168851051479578,
+            -0.05905447527766228,
+            0.0761253759264946,
+            0.005873778834939003,
+            -0.0025391464587301016,
+            -0.023660998791456223,
+            -0.00031456572469323874,
+            -0.013579814694821835,
+            0.0022752410732209682,
+            -0.013421471230685711,
+            0.0437932051718235,
+            0.06086410954594612,
+            -0.029512157663702965,
+            0.010978462174534798,
+            0.022107727825641632,
+            -0.0007973711472004652,
+            0.1003141850233078,
+            0.006884159054607153,
+            0.005941640119999647,
+            -0.037549957633018494,
+            0.002186644123867154,
+            -0.01580416038632393,
+            -0.04240581765770912,
+            -0.03700706735253334,
+            -0.05407797545194626,
+            0.019619476050138474,
+            -0.046748943626880646,
+            0.00444114999845624,
+            0.04273758456110954,
+            -0.015140626579523087,
+            -0.0038096620701253414,
+            -0.01768919825553894,
+            0.013383770361542702,
+            -0.022861741483211517,
+            0.011189586482942104,
+            -0.0014024684205651283,
+            0.000858163577504456,
+            -0.004241335671395063,
+            -0.01000578235834837,
+            -0.029059747233986855,
+            -0.007672104984521866,
+            0.003811547067016363,
+            0.04047553986310959,
+            0.047653764486312866,
+            -0.0007059467607177794,
+            0.008761657401919365,
+            0.0011036897776648402,
+            0.012320608831942081,
+            -0.02206248603761196,
+            -0.018443213775753975,
+            0.010133964940905571,
+            0.0034119191113859415,
+            0.018322570249438286,
+            0.005353508051484823,
+            0.04521075263619423,
+            -0.03761028125882149,
+            0.0037681912072002888,
+            0.02813984826207161,
+            0.017870161682367325,
+            -0.006635334342718124,
+            0.0455123595893383,
+            -0.002667329041287303,
+            -0.015118005685508251,
+            -0.004588183015584946,
+            -0.01981552131474018,
+            0.0016041673952713609,
+            -0.0403548963367939,
+            0.014771158806979656,
+            0.00011899303353857249,
+            -6.232407031347975e-05,
+            0.02773268148303032,
+            -0.003076382214203477,
+            0.06930907815694809,
+            0.02446025423705578,
+            -0.0023204819299280643,
+            -0.07292835414409637,
+            0.027431074529886246,
+            0.0247920211404562,
+            0.01482393965125084,
+            0.049644362181425095,
+            0.06182925030589104,
+            -0.016799459233880043,
+            0.02823033183813095,
+            0.012810719199478626,
+            -0.007140524219721556,
+            -0.029994726181030273,
+            0.02352527529001236,
+            -0.04255662113428116,
+            -0.027114387601614,
+            0.046628303825855255,
+            -0.010450650937855244,
+            0.041229553520679474,
+            -0.007977481000125408,
+            0.04726167395710945,
+            0.03507678955793381,
+            -0.007902080193161964,
+            0.022198209539055824,
+            0.005432679783552885,
+            0.03423229232430458,
+            0.00433935783803463,
+            -0.007438360247761011,
+            -0.007193305529654026,
+            -0.03655466064810753,
+            -0.005575942806899548,
+            -0.024203889071941376,
+            0.033357635140419006,
+            0.006936940364539623,
+            -0.0062696365639567375,
+            -0.049855489283800125,
+            0.009251766838133335,
+            -0.014537413604557514,
+            0.01063915528357029,
+            -0.0010669315233826637,
+            0.051242876797914505,
+            -0.008957700803875923,
+            0.00243546930141747,
+            -0.021263230592012405,
+            0.002537261229008436,
+            -0.012177346274256706,
+            -0.006503381300717592,
+            0.014303669333457947,
+            -0.03100510686635971,
+            0.023600677028298378,
+            -0.006578783039003611,
+            -0.0004493459709919989,
+            -0.010986002162098885,
+            0.0029481996316462755,
+            0.040415216237306595,
+            0.015374370850622654,
+            0.017387591302394867,
+            -0.012426171451807022,
+            -0.004840777721256018,
+            0.01446201279759407,
+            -0.00019792901002801955,
+            0.009349789470434189,
+            0.007743736729025841,
+            -0.018533695489168167,
+            0.01826224848628044,
+            -0.020328251644968987,
+            0.014673137106001377,
+            0.0023921134416013956,
+            -0.006541082169860601,
+            -0.042948707938194275,
+            0.0064166695810854435,
+            -0.01943851262331009,
+            0.008188605308532715,
+            -0.002505215583369136,
+            -0.035559359937906265,
+            0.014469552785158157,
+            -0.03426245227456093,
+            0.009176366031169891,
+            -0.0007106593693606555,
+            0.011423330754041672,
+            -0.017719358205795288,
+            0.005127303767949343,
+            -0.0004592424083966762,
+            -0.01776459999382496,
+            0.01362505555152893,
+            -0.038002368062734604,
+            0.0039095692336559296,
+            0.016724057495594025,
+            -0.015743838623166084,
+            0.04246613755822182,
+            0.017824919894337654,
+            -0.03468470275402069,
+            0.00354764168150723,
+            -0.04116923362016678,
+            0.05814965441823006,
+            0.004049061797559261,
+            -0.005813457537442446,
+            0.020735418424010277,
+            -0.01219242624938488,
+            -0.024354692548513412,
+            0.020976703613996506,
+            0.0028464074712246656,
+            0.00209427741356194,
+            -0.027958884835243225,
+            -0.016347050666809082,
+            -0.043401118367910385,
+            -0.04892050847411156,
+            -0.004369518253952265,
+            0.04611557349562645,
+            -0.03272426128387451,
+            -0.022288691252470016,
+            -0.01286350004374981,
+            0.02528967149555683,
+            0.013361150398850441,
+            -0.007585393264889717,
+            0.030326493084430695,
+            0.02560635842382908,
+            -0.00020405538089107722,
+            -0.008761657401919365,
+            0.019906003028154373,
+            -0.04602508991956711,
+            0.002431699074804783,
+            -0.0633372813463211,
+            -0.010307388380169868,
+            -0.004475080408155918,
+            -0.03827381506562233,
+            -0.002079196972772479,
+            -0.0005141441361047328,
+            0.0239324439316988,
+            -0.02617940865457058,
+            -0.019091665744781494,
+            -0.0003159795014653355,
+            0.021006865426898003,
+            -0.019906003028154373,
+            -0.03010028786957264,
+            0.004007590934634209,
+            -0.03317667171359062,
+            -0.006620253901928663,
+            -0.052962031215429306,
+            -0.005176314618438482,
+            -0.009734337218105793,
+            0.01912182755768299,
+            -0.01895594410598278,
+            -0.000523569353390485,
+            0.00784175843000412,
+            0.02225852943956852,
+            0.035891126841306686,
+            -0.03896750882267952,
+            -0.007574083283543587,
+            -0.006273407023400068,
+            -0.017176467925310135,
+            0.02835097350180149,
+            -0.01207178458571434,
+            -0.008980321697890759,
+            -0.003890718799084425,
+            0.011363009922206402,
+            -0.046960070729255676,
+            0.0005061327246949077,
+            -0.013949282467365265,
+            -0.00020111000048927963,
+            0.01545731257647276,
+            0.04370272532105446,
+            -0.038424618542194366,
+            -0.0023751480039209127,
+            -0.04578380659222603,
+            -0.02859225869178772,
+            -0.038907185196876526,
+            -0.030763821676373482,
+            -0.023178428411483765,
+            -0.014575114473700523,
+            0.017538394778966904,
+            0.004173474386334419,
+            -0.021187828853726387,
+            0.009153745137155056,
+            -0.020901301875710487,
+            0.06828361749649048,
+            -0.007562772836536169,
+            0.015336669981479645,
+            0.001535363495349884,
+            -0.024580897763371468,
+            -0.009915300644934177,
+            -0.00974187720566988,
+            0.02385704219341278,
+            -0.007442130707204342,
+            0.0009670245344750583,
+            -0.0019246239680796862,
+            0.02560635842382908,
+            -0.0013713652733713388,
+            -0.005428909789770842,
+            -0.03721819445490837,
+            -0.0062168557196855545,
+            -0.010118884965777397,
+            0.030733661726117134,
+            -0.036132410168647766,
+            -7.999630179256201e-05,
+            -0.011468571610748768,
+            -0.0461457334458828,
+            0.00017471947649028152,
+            -0.04219469428062439,
+            0.003749340772628784,
+            -0.0007229121401906013,
+            -0.021534675732254982,
+            -0.022077566012740135,
+            0.01723678782582283,
+            -0.026858022436499596,
+            -0.022771259769797325,
+            0.006857768632471561,
+            0.0017804185627028346,
+            0.010020862333476543,
+            4.367986912257038e-05,
+            0.030055047944188118,
+            -0.043521761894226074,
+            -0.023133188486099243,
+            0.02761203795671463,
+            -0.022786341607570648,
+            -0.002165908692404628,
+            -0.04219469428062439,
+            -0.007347878534346819,
+            -0.040053289383649826,
+            0.020614776760339737,
+            0.02438485249876976,
+            -0.006861538626253605,
+            -0.0181416068226099,
+            0.026933424174785614,
+            -0.0037248353473842144,
+            -0.05555584281682968,
+            0.01296906266361475,
+            0.0052894167602062225,
+            0.022891903296113014,
+            -0.008279087021946907,
+            -0.024475334212183952,
+            -0.040626343339681625,
+            0.004689974710345268,
+            -0.01695026271045208,
+            -0.04919195547699928,
+            0.0204639732837677,
+            -0.032211530953645706,
+            -0.016573255881667137,
+            -0.007834218442440033,
+            0.0031461287289857864,
+            -0.002904843771830201,
+            0.019921083003282547,
+            0.001642810762859881,
+            -0.009847438894212246,
+            -0.03501646965742111,
+            -0.021474353969097137,
+            -0.03365924209356308,
+            -0.021745799109339714,
+            0.006529772188514471,
+            0.005892629269510508,
+            -0.014311209321022034,
+            0.04267726466059685,
+            -0.015487473458051682,
+            0.034775182604789734,
+            -0.0016041673952713609,
+            -0.016467692330479622,
+            0.05468118563294411,
+            0.012923821806907654,
+            -0.00233367714099586,
+            0.009937920607626438,
+            -0.00265036360360682,
+            0.0019472443964332342,
+            0.012637295760214329,
+            -0.004316737409681082,
+            0.031638480722904205,
+            -0.0005683390190824866,
+            0.009953001514077187,
+            -0.02708422765135765,
+            -0.013104785233736038,
+            -0.011574134230613708,
+            -0.040173932909965515,
+            0.02875814214348793,
+            -0.008037802763283253,
+            -0.002033956116065383,
+            0.018126526847481728,
+            0.012667456641793251,
+            -0.010948301292955875,
+            -0.03374972194433212,
+            -0.015291429124772549,
+            -0.00853545218706131,
+            -0.007879459299147129,
+            -0.012275367975234985,
+            -0.04457738250494003,
+            0.008550533093512058,
+            -0.0535954050719738,
+            -0.01996632292866707,
+            -0.05443990230560303,
+            -0.008867219090461731,
+            0.03038681484758854,
+            -4.72732208436355e-05,
+            0.012916280888020992,
+            0.01054113358259201,
+            0.0054062893614172935,
+            -0.03646417707204819,
+            -0.013677836395800114,
+            0.024897582828998566,
+            0.02136879228055477,
+            -0.01472591795027256,
+            -0.038092851638793945,
+            0.017432833090424538,
+            -0.01666373759508133,
+            -0.047925207763910294,
+            -0.011815418489277363,
+            0.011430870741605759,
+            -0.012554354034364223,
+            -0.060803789645433426,
+            -0.01764395646750927,
+            -0.018563855439424515,
+            -0.008610853925347328,
+            0.028893863782286644,
+            0.020403653383255005,
+            -0.004795536864548922,
+            -0.03679594397544861,
+            0.0018567625666037202,
+            0.015653356909751892,
+            0.006435520015656948,
+            0.032000407576560974,
+            -0.008550533093512058,
+            0.03803252801299095,
+            0.011355469934642315,
+            0.02589288353919983,
+            0.014213187620043755,
+            0.0016013398999348283,
+            0.0071970755234360695,
+            -0.01568351686000824,
+            -0.01362505555152893,
+            -0.011174505576491356,
+            -0.028320813551545143,
+            -0.009349789470434189,
+            -0.018292410299181938,
+            0.02124815061688423,
+            -0.0316988006234169,
+            0.04141051694750786,
+            0.01854877546429634,
+            -0.0007818195736035705,
+            0.010412951000034809,
+            0.001439226558431983,
+            0.00450524128973484,
+            0.007555232848972082,
+            -0.0016211328329518437,
+            0.025018226355314255,
+            0.023721320554614067,
+            -0.034051328897476196,
+            -0.013806018978357315,
+            0.014235807582736015,
+            -0.010103804059326649,
+            0.016889940947294235,
+            -0.005165004637092352,
+            -0.005662654526531696,
+            0.03160832077264786,
+            -0.024852342903614044,
+            0.02103702537715435,
+            0.003636238630861044,
+            -0.014243348501622677,
+            -0.02495790459215641,
+            0.0059944214299321175,
+            -0.023540355265140533,
+            0.04406465217471123,
+            -0.03531807288527489,
+            0.02756679803133011,
+            0.004497700836509466,
+            -0.015351750887930393,
+            0.03059793822467327,
+            -0.011166965588927269,
+            -0.017342351377010345,
+            -0.003223415231332183,
+            0.030160609632730484,
+            -0.010827658697962761,
+            -0.023600677028298378,
+            0.018699578940868378,
+            -0.007121674250811338,
+            0.027219951152801514,
+            -0.01413024589419365,
+            -0.0025466864462941885,
+            0.03332747519016266,
+            0.006303567439317703,
+            -0.0016456382581964135,
+            0.009266847744584084,
+            0.012086864560842514,
+            0.031186070293188095,
+            0.02969312109053135,
+            -0.025274591520428658,
+            -0.013361150398850441,
+            0.04032473638653755,
+            0.010978462174534798,
+            -0.002395883435383439,
+            0.012184886261820793,
+            -0.0032498056534677744,
+            -0.013594894669950008,
+            0.002346872352063656,
+            0.0007521302322857082,
+            -0.012109484523534775,
+            0.006420440040528774,
+            -0.028154930099844933,
+            -0.0017398901982232928,
+            0.01883530057966709,
+            0.013519493862986565,
+            -0.005915249697864056,
+            -0.013557193800807,
+            0.01525372825562954,
+            -0.01984568126499653,
+            0.024158649146556854,
+            -0.013821099884808064,
+            0.006582553032785654,
+            0.01425842847675085,
+            -0.010239526629447937,
+            -0.03281474485993385,
+            0.00847513135522604,
+            -3.4195771149825305e-05,
+            -0.009493052028119564,
+            0.003455274971202016,
+            -0.009794658049941063,
+            -0.008950160816311836,
+            -0.0032611158676445484,
+            0.011513812467455864,
+            -0.02940659411251545,
+            -0.0002133627567673102,
+            0.027762841433286667,
+            0.024988064542412758,
+            -0.013542113825678825,
+            -0.005741826258599758,
+            -0.010759797878563404,
+            0.009681555442512035,
+            -0.004282806534320116,
+            -0.009583533741533756,
+            0.027913644909858704,
+            0.02185136266052723,
+            -0.03212105110287666,
+            0.006420440040528774,
+            -0.005485461093485355,
+            0.0019547846168279648,
+            0.004999121185392141,
+            0.015238648280501366,
+            -0.018729738891124725,
+            0.00609621312469244,
+            -0.006454370450228453,
+            0.013994523324072361,
+            -0.014371531084179878,
+            0.023706238716840744,
+            0.017915401607751846,
+            0.009349789470434189,
+            -0.01482393965125084,
+            -0.006503381300717592,
+            0.010390330106019974,
+            -0.001478812424466014,
+            -0.010556213557720184,
+            0.035800643265247345,
+            0.00704627251252532,
+            -0.0011885165004059672,
+            0.01562319602817297,
+            0.028969265520572662,
+            0.02316334843635559,
+            -0.003754995996132493,
+            0.011853119358420372,
+            0.06774073094129562,
+            -0.018247168511152267,
+            0.005658884532749653,
+            0.0005353508167900145,
+            0.020041724666953087,
+            0.006130144000053406,
+            -0.032000407576560974,
+            -0.034051328897476196,
+            0.0001823774364311248,
+            -0.026073846966028214,
+            -0.04735215753316879,
+            -0.014047304168343544,
+            -0.01809636689722538,
+            -0.03172896057367325,
+            -0.0042639560997486115,
+            -0.00963631458580494,
+            0.005787067115306854,
+            0.023600677028298378,
+            0.004113153088837862,
+            -0.027385834604501724,
+            -0.016678817570209503,
+            0.02699374593794346,
+            0.025274591520428658,
+            0.03736899420619011,
+            -0.005681504961103201,
+            0.011174505576491356,
+            0.022529976442456245,
+            0.002050921553745866,
+            0.00927438773214817,
+            -0.004026441369205713,
+            -0.008052882738411427,
+            0.027174709364771843,
+            0.02703898772597313,
+            0.029270872473716736,
+            -0.0011461031390354037,
+            -0.034775182604789734,
+            -0.015298969112336636,
+            -0.018081285059452057,
+            -0.0006102810730226338,
+            0.008693795651197433,
+            -0.009033102542161942,
+            0.018352732062339783,
+            -0.0004740870790556073,
+            0.018518613651394844,
+            0.025591278448700905,
+            0.0022865512873977423,
+            -0.022394252941012383,
+            0.010495892725884914,
+            -0.0009608981781639159,
+            0.0071669151075184345,
+            -0.029843922704458237,
+            -0.04125971347093582,
+            -0.056430500000715256,
+            0.007068892940878868,
+            0.03254329785704613,
+            -0.042375657707452774,
+            0.01609068550169468,
+            0.04741247743368149,
+            -0.009018022567033768,
+            -0.023676078766584396,
+            -0.015864480286836624,
+            0.005410059355199337,
+            -0.022741099819540977,
+            -0.018307490274310112,
+            0.037067390978336334,
+            -0.015502553433179855,
+            -0.017870161682367325,
+            0.02079574018716812,
+            -0.02295222505927086,
+            0.02041873335838318,
+            -0.02818509005010128,
+            -0.002237540204077959,
+            -0.03311634808778763,
+            0.04014377295970917,
+            -0.012260288000106812,
+            0.00731017766520381,
+            -0.013097245246171951,
+            0.044456738978624344,
+            -0.008083043619990349,
+            0.017101066187024117,
+            0.0012997337616980076,
+            -0.001241297577507794,
+            -0.00830924790352583,
+            0.004893559031188488,
+            0.02838113345205784,
+            -0.017598716542124748,
+            0.0069030094891786575,
+            0.011264988221228123,
+            -0.03136703372001648,
+            -0.0187448188662529,
+            0.0037870416417717934,
+            0.021263230592012405,
+            0.015314050018787384,
+            0.04394400864839554,
+            0.028562096878886223,
+            -0.009930380620062351,
+            -0.008346948772668839,
+            0.009756957180798054,
+            -0.031276553869247437,
+            0.03709755092859268,
+            -0.013579814694821835,
+            0.031759120523929596,
+            0.028622418642044067,
+            -0.011264988221228123,
+            -0.001501432852819562,
+            -0.0027540407609194517,
+            0.02058461681008339,
+            0.03803252801299095,
+            0.007528842426836491,
+            0.005142384208738804,
+            0.02560635842382908,
+            0.010910600423812866,
+            -0.02626989036798477,
+            0.01826224848628044,
+            -0.004509011283516884,
+            -0.004780456889420748,
+            0.025259511545300484,
+            0.014265968464314938,
+            0.01409254502505064,
+            0.0017427176935598254,
+            0.028848623856902122,
+            -0.019875841215252876,
+            0.010118884965777397,
+            0.002904843771830201,
+            -0.012697616592049599,
+            0.015314050018787384,
+            0.0068276082165539265,
+            0.021263230592012405,
+            -0.004618343431502581,
+            0.02985900454223156,
+            0.01768919825553894,
+            0.01649785414338112,
+            0.0386357419192791,
+            0.006024581845849752,
+            -0.049041152000427246,
+            0.012049163691699505,
+            0.017176467925310135,
+            -0.0031423585023730993,
+            -0.021655317395925522,
+            -0.004041521809995174,
+            -0.002746500540524721,
+            0.007924700155854225,
+            0.0039057990070432425,
+            -0.01568351686000824,
+            -0.017824919894337654,
+            -0.004324277397245169,
+            0.010707016102969646,
+            0.021308470517396927,
+            0.010164125822484493,
+            0.005930330138653517,
+            0.006375198718160391,
+            0.040505699813365936,
+            0.02708422765135765,
+            -0.02495790459215641,
+            0.033387795090675354,
+            0.011543973349034786,
+            -0.015110465697944164,
+            -0.015223568305373192,
+            -0.022514894604682922,
+            0.0028690281324088573,
+            -0.02838113345205784,
+            -0.004659814294427633,
+            0.01883530057966709,
+            0.02209264598786831,
+            -0.013542113825678825,
+            -0.002821902045980096,
+            0.024701539427042007,
+            0.0009255537297576666,
+            -0.011363009922206402,
+            -0.021459273993968964,
+            0.011016163043677807,
+            0.0067785969004035,
+            -4.6360157284652814e-05,
+            0.008821978233754635,
+            0.04922211542725563,
+            -0.0055834827944636345,
+            0.027958884835243225,
+            0.049734845757484436,
+            0.0027917413972318172,
+            -0.02871290035545826,
+            0.02225852943956852,
+            0.007359188981354237,
+            -0.004169704392552376,
+            0.0221831277012825,
+            -0.005225325468927622,
+            -0.04819665476679802,
+            0.022861741483211517,
+            -0.028109688311815262,
+            -0.006005731411278248,
+            0.009289467707276344,
+            0.011385629884898663,
+            -0.002250735415145755,
+            -0.026692138984799385,
+            0.036253053694963455,
+            -0.006620253901928663,
+            0.015223568305373192,
+            -0.023374471813440323,
+            -0.027008825913071632,
+            -0.014974743127822876,
+            -0.007687185425311327,
+            -0.009749417193233967,
+            0.0031348182819783688,
+            -0.00901048257946968,
+            0.016241488978266716,
+            -0.03619273379445076,
+            0.003183829365298152,
+            0.012418631464242935,
+            0.00450524128973484,
+            0.04762360453605652,
+            0.00959861371666193,
+            0.00757785327732563,
+            -0.005915249697864056,
+            -0.012162266299128532,
+            0.023540355265140533,
+            0.010933221317827702,
+            0.03516726940870285,
+            -0.004399679135531187,
+            0.007223465945571661,
+            0.030447136610746384,
+            0.024927744641900063,
+            0.02201724611222744,
+            -0.016844701021909714,
+            -0.004181014373898506,
+            -0.00476914644241333,
+            -0.014590195380151272,
+            0.011822959408164024,
+            -0.005805917549878359,
+            0.008987861685454845,
+            0.012380930595099926,
+            0.007434590253978968,
+            0.011581674218177795,
+            0.06249278411269188,
+            0.015879560261964798,
+            -0.00888229999691248,
+            -0.026481015607714653,
+            -0.0032498056534677744,
+            0.007657025009393692,
+            0.00959861371666193,
+            0.0019736348185688257,
+            -0.009794658049941063,
+            0.014265968464314938,
+            0.00824892707169056,
+            -0.03402116894721985,
+            0.002678639255464077,
+            -0.004015131387859583,
+            0.006130144000053406,
+            0.01492950227111578,
+            -0.0403548963367939,
+            -0.00259381253272295,
+            -0.020720338448882103,
+            -0.004799306858330965,
+            0.022620458155870438,
+            -0.010412951000034809,
+            0.005979340989142656,
+            0.0016503508668392897,
+            -0.028803382068872452,
+            0.003958580084145069,
+            -0.0054364497773349285,
+            0.02777792140841484,
+            -0.028079528361558914,
+            -0.008821978233754635,
+            -0.013677836395800114,
+            -0.007600473705679178,
+            -0.017417753115296364,
+            0.029180390760302544,
+            -0.019589316099882126,
+            -0.002714454894885421,
+            -0.031155910342931747,
+            -0.014884261414408684,
+            -0.01476361881941557,
+            0.026375453919172287,
+            -0.01308970432728529,
+            -0.04481866583228111,
+            0.008482671342790127,
+            -0.012441251426935196,
+            0.00222057499922812,
+            -0.026194490492343903,
+            0.0030914624221622944,
+            0.028652578592300415,
+            0.02058461681008339,
+            0.006507151760160923,
+            -0.028411295264959335,
+            -0.010412951000034809,
+            0.01817176677286625,
+            0.04620605334639549,
+            -0.020765580236911774,
+            0.0128257991746068,
+            -0.012252748012542725,
+            0.01069947611540556,
+            -0.044004328548908234,
+            -0.0042337956838309765,
+            0.001988715259358287,
+            0.021821200847625732,
+            -0.015412071719765663,
+            0.002610777737572789,
+            0.05253978073596954,
+            0.017794759944081306,
+            -0.05163496360182762,
+            0.01866941712796688,
+            -0.012019002810120583,
+            0.034533899277448654,
+            0.05534471943974495,
+            -0.004535401705652475,
+            0.006936940364539623,
+            0.020765580236911774,
+            -0.022032326087355614,
+            0.010450650937855244,
+            0.014876720495522022,
+            -0.00576821668073535,
+            -0.02259029634296894,
+            -0.013579814694821835,
+            0.021640237420797348,
+            0.012592054903507233,
+            -0.012305528856813908,
+            0.03872622177004814,
+            -0.0007318660500459373,
+            0.029587557539343834,
+            -0.01678437925875187,
+            -0.03166864067316055,
+            -0.002508985809981823,
+            -0.018081285059452057,
+            0.025274591520428658,
+            0.010390330106019974,
+            0.0015985122881829739,
+            0.006699425633996725,
+            -0.035921286791563034,
+            0.01857893541455269,
+            0.02503330633044243,
+            0.0003843121521640569,
+            0.001393043203279376,
+            -0.02541031315922737,
+            0.02168547920882702,
+            -0.004761606454849243,
+            0.018186848610639572,
+            0.022605376318097115,
+            -0.0027955116238445044,
+            -0.019167067483067513,
+            0.005836077965795994,
+            0.0012083094334229827,
+            0.04346143826842308,
+            -0.001625845325179398,
+            0.0026239731814712286,
+            -0.01826224848628044,
+            -0.0032724260818213224,
+            -0.019001184031367302,
+            0.02572700008749962,
+            -0.005545781925320625,
+            -0.017432833090424538,
+            -0.018352732062339783,
+            0.03136703372001648,
+            -0.0007634404464624822,
+            -0.032332174479961395,
+            -0.003114083083346486,
+            -0.012433711439371109,
+            -0.0018379121320322156,
+            0.018639257177710533,
+            -0.03157815709710121,
+            -0.017251869663596153,
+            -0.0017493154155090451,
+            0.022077566012740135,
+            0.026948504149913788,
+            0.01883530057966709,
+            -0.0024712849408388138,
+            -0.0001750729134073481,
+            0.011076483875513077,
+            -0.0004952937597408891,
+            0.011506272479891777,
+            0.01299168262630701,
+            -0.007370498962700367,
+            0.03287506476044655,
+            -0.006367658730596304,
+            -0.0036173881962895393,
+            0.023887204006314278,
+            -0.01063915528357029,
+            -0.008731496520340443,
+            0.018654337152838707,
+            -0.018352732062339783,
+            -0.015502553433179855,
+            -0.007804058026522398,
+            0.005662654526531696,
+            -0.004886019043624401,
+            -0.0015080304583534598,
+            3.1014766136649996e-05,
+            0.03724835440516472,
+            -0.027265191078186035,
+            0.03178928419947624,
+            0.043220154941082,
+            -0.009847438894212246,
+            -0.02863749861717224,
+            -0.0016013398999348283,
+            -0.003496745601296425,
+            0.02455073595046997,
+            0.006631563883274794,
+            -0.0006592920981347561,
+            -0.04195340722799301,
+            0.015954962000250816,
+            -0.016226409003138542,
+            -0.011679695919156075,
+            -0.006024581845849752,
+            -0.004791766870766878,
+            0.009410110302269459,
+            0.028154930099844933,
+            -0.02801920659840107,
+            0.01190590113401413,
+            0.012418631464242935,
+            0.024083247408270836,
+            -0.005342198070138693,
+            -0.01409254502505064,
+            -0.01103124301880598,
+            -0.011287608183920383,
+            0.012531733140349388,
+            -0.0164224524050951,
+            -0.005319577641785145,
+            0.00519893504679203,
+            -0.005327117629349232,
+            0.01100862305611372,
+            -0.016920102760195732,
+            -0.06375952810049057,
+            -0.011114184744656086,
+            0.03667530044913292,
+            -0.014801319688558578,
+            -0.018880542367696762,
+            0.03712771087884903,
+            0.009545832872390747,
+            0.021640237420797348,
+            -0.023947523906826973,
+            0.0155628751963377,
+            -0.006767286919057369,
+            0.0029953254852443933,
+            -0.019785359501838684,
+            -0.010955841280519962,
+            0.00870133563876152,
+            -0.004429839551448822,
+            -0.004143313970416784,
+            -0.024249130859971046,
+            0.02617940865457058,
+            0.008264007046818733,
+            -0.02271093986928463,
+            0.007592933718115091,
+            0.008656094782054424,
+            0.005376128945499659,
+            -0.01604544371366501,
+            0.013572274707257748,
+            0.011822959408164024,
+            -0.01345917209982872,
+            0.02275617979466915,
+            -0.025063466280698776,
+            -0.004486390855163336,
+            0.013775859028100967,
+            0.030447136610746384,
+            0.014454471878707409,
+            -0.021670397371053696,
+            -0.02148943394422531,
+            0.011242367327213287,
+            0.0015909721842035651,
+            0.0034119191113859415,
+            0.000781348324380815,
+            -0.00709528336301446,
+            -0.04451705887913704,
+            0.01984568126499653,
+            -0.0020660017617046833,
+            -0.015035063959658146,
+            0.00412446353584528,
+            0.015909722074866295,
+            0.013609975576400757,
+            0.007487371563911438,
+            -0.006371428724378347,
+            0.01096338126808405,
+            0.033629078418016434,
+            -0.019001184031367302,
+            -0.022680778056383133,
+            -0.0063638887368142605,
+            -0.0010094379540532827,
+            0.01838289201259613,
+            -0.010360169224441051,
+            0.004395908676087856,
+            0.01809636689722538,
+            0.07220450043678284,
+            -0.014710837043821812,
+            0.00022785397595725954,
+            -0.03206072747707367,
+            -0.02340463362634182,
+            0.00108766695484519,
+            -0.0036569740623235703,
+            0.022198209539055824,
+            0.021293390542268753,
+            0.018066205084323883,
+            0.010691936127841473,
+            -0.007234776392579079,
+            0.002895418554544449,
+            0.025395233184099197,
+            -0.013353610411286354,
+            0.030809063464403152,
+            0.007909620180726051,
+            0.016316890716552734,
+            0.001337434514425695,
+            -0.015080304816365242,
+            0.012509113177657127,
+            0.033267151564359665,
+            -0.010857819579541683,
+            0.002940659411251545,
+            -0.004331817384809256,
+            0.0099454615265131,
+            -0.00028275573276914656,
+            -0.008158445358276367,
+            -0.009206525981426239,
+            -0.023057786747813225,
+            0.014115165919065475,
+            -0.014205647632479668,
+            -0.004735215567052364,
+            0.005263026338070631,
+            -0.016105765476822853,
+            0.062191177159547806,
+            -0.008769197389483452,
+            0.008324328809976578,
+            0.0077361962758004665,
+            -0.008497752249240875,
+            -0.005425139795988798,
+            0.008173525333404541,
+            0.005968030542135239,
+            0.011619375087320805,
+            -0.005149924196302891,
+            0.0037813864182680845,
+            -0.0024335843045264482,
+            -0.011189586482942104,
+            -0.00015857882681302726,
+            -0.023133188486099243,
+            0.016392290592193604,
+            0.006484531331807375,
+            -0.023615757003426552,
+            -0.0007064180099405348,
+            -0.0049425698816776276,
+            -0.00741196982562542,
+            0.002096162410452962,
+            -0.017251869663596153,
+            -0.0034119191113859415,
+            0.014612815342843533,
+            -0.004799306858330965,
+            -0.003898258786648512,
+            0.008075503632426262,
+            0.013127405196428299,
+            -0.008135824464261532,
+            -0.009183906018733978,
+            0.02156483568251133,
+            0.003148013725876808,
+            0.0034816653933376074,
+            0.006156534422188997,
+            -0.009726796299219131,
+            0.024988064542412758,
+            -0.01915198750793934,
+            -0.01435645017772913,
+            0.0050669824704527855,
+            0.010020862333476543,
+            0.019257549196481705,
+            -0.020328251644968987,
+            -0.003347827820107341,
+            -0.010940761305391788,
+            -0.008791817352175713,
+            0.022891903296113014,
+            -0.002870913129299879,
+            -0.023555437102913857,
+            -0.016482774168252945,
+            0.0025184110272675753,
+            -0.01833765022456646,
+            -0.005379898939281702,
+            -0.008339408785104752,
+            -0.005866238847374916,
+            -0.02103702537715435,
+            -0.018563855439424515,
+            0.0033384026028215885,
+            -0.029964566230773926,
+            -0.0036456636153161526,
+            -0.011046322993934155,
+            -0.0005975571111775935,
+            0.0076834154315292835,
+            -0.04578380659222603,
+            0.0005622126045636833,
+            -0.035227593034505844,
+            0.002356297569349408,
+            0.008656094782054424,
+            0.005304497200995684,
+            -0.013783399015665054,
+            0.020071886479854584,
+            0.017372511327266693,
+            -0.01981552131474018,
+            0.015713678672909737,
+            0.005334658082574606,
+            0.00018390902550891042,
+            0.0037983518559485674,
+            0.007672104984521866,
+            -0.007385579403489828,
+            0.0072272359393537045,
+            0.010186745785176754,
+            0.0028407524805516005,
+            -0.017025664448738098,
+            -0.012652375735342503,
+            -0.019167067483067513,
+            -0.008573153056204319,
+            0.006853998638689518,
+            0.021142587065696716,
+            -0.00959861371666193,
+            0.004901099018752575,
+            0.00671827606856823,
+            0.008625934831798077,
+            -0.016271648928523064,
+            0.01972503960132599,
+            -0.009900220669806004,
+            -0.0018906932091340423,
+            -0.018865462392568588,
+            -0.000623005093075335,
+            -0.00036051354254595935,
+            0.01372307725250721,
+            -0.009221606887876987,
+            -0.00444114999845624,
+            -0.0037531109992414713,
+            0.006337498314678669,
+            -0.010940761305391788,
+            -0.0013581699458882213,
+            0.00020865016267634928,
+            0.0267977025359869,
+            0.014363990165293217,
+            0.0008280029869638383,
+            -0.03966120257973671,
+            -0.027989046648144722,
+            0.025274591520428658,
+            0.00513484375551343,
+            0.012953981757164001,
+            -0.02409832738339901,
+            -0.018081285059452057,
+            0.0328449048101902,
+            0.0012808834435418248,
+            -0.000869473849888891,
+            -0.034986305981874466,
+            0.016603415831923485,
+            -0.015577955171465874,
+            0.02875814214348793,
+            0.025259511545300484,
+            -0.014801319688558578,
+            0.027702519670128822,
+            0.005391208920627832,
+            -0.022967305034399033,
+            0.008324328809976578,
+            0.006963330786675215,
+            0.019875841215252876,
+            -0.01515570655465126,
+            0.01601528376340866,
+            0.003886948572471738,
+            -0.017342351377010345,
+            -0.007502451539039612,
+            0.014205647632479668,
+            0.02756679803133011,
+            0.005802147556096315,
+            0.005485461093485355,
+            0.001331779407337308,
+            -0.036494337022304535,
+            0.015638276934623718,
+            -0.0019415892893448472,
+            0.01345917209982872,
+            -0.01637721061706543,
+            0.004181014373898506,
+            0.007924700155854225,
+            -0.015170786529779434,
+            -0.013270668685436249,
+            -0.029346274212002754,
+            0.005519391503185034,
+            -0.009116044268012047,
+            0.0020245308987796307,
+            -0.0033723332453519106,
+            0.0071669151075184345,
+            0.020569536834955215,
+            -0.014333830215036869,
+            -0.0035646071191877127,
+            0.005112223327159882,
+            0.008407269604504108,
+            -0.001880325493402779,
+            0.002735190326347947,
+            0.002011335687711835,
+            -0.00020111000048927963,
+            0.02835097350180149,
+            -0.016000203788280487,
+            -0.013134945183992386,
+            0.004101843107491732,
+            -0.00937240943312645,
+            0.002397768432274461,
+            -0.00646568089723587,
+            0.009628774598240852,
+            -0.0328449048101902,
+            -0.0018595900619402528,
+            -0.014409231022000313,
+            -0.021881522610783577,
+            -0.0013072739820927382,
+            0.008927540853619576,
+            0.009485512040555477,
+            -0.02099178545176983,
+            0.008422350510954857,
+            0.020433813333511353,
+            0.007702265866100788,
+            0.00867871567606926,
+            0.02299746498465538,
+            0.027793001383543015,
+            0.010028403252363205,
+            -0.008814438246190548,
+            -0.016000203788280487,
+            0.019830601289868355,
+            -0.015909722074866295,
+            0.013308369554579258,
+            0.00810566358268261,
+            0.004361978266388178,
+            0.03002488799393177,
+            0.018518613651394844,
+            0.00933470856398344,
+            0.011792798526585102,
+            0.004591953009366989,
+            0.01425842847675085,
+            -0.0042111752554774284,
+            0.01482393965125084,
+            0.00023786825477145612,
+            -0.00845251139253378,
+            0.00014538357208948582,
+            -0.0051876250654459,
+            0.026601657271385193,
+            -0.011755097657442093,
+            -0.00017742921772878617,
+            -0.011355469934642315,
+            -0.027340592816472054,
+            -0.010571293532848358,
+            0.005477920640259981,
+            -0.024249130859971046,
+            0.004637193866074085,
+            0.004486390855163336,
+            -0.01296906266361475,
+            -0.016482774168252945,
+            -0.015306509099900723,
+            0.010925681330263615,
+            -0.0006616484024561942,
+            0.008919999934732914,
+            0.0057003553956747055,
+            -0.011023703031241894,
+            0.003400608664378524,
+            0.020162368193268776,
+            0.021640237420797348,
+            0.0011074598878622055,
+            0.02646593563258648,
+            0.0013864454813301563,
+            -0.010616534389555454,
+            -0.0071669151075184345,
+            0.0017643957398831844,
+            -0.007962401024997234,
+            -0.005168774630874395,
+            -0.009093424305319786,
+            -0.0035551819019019604,
+            0.020252849906682968,
+            -0.004799306858330965,
+            -0.015758918598294258,
+            0.0006687172572128475,
+            -0.01649785414338112,
+            -0.0038831785786896944,
+            -0.008972781710326672,
+            0.04044537618756294,
+            -0.021459273993968964,
+            -0.007547692861407995,
+            -0.02981376275420189,
+            0.0024769401643425226,
+            -0.0015928571810945868,
+            0.00629979744553566,
+            0.004927489440888166,
+            -0.03402116894721985,
+            -0.05658130347728729,
+            0.027355672791600227,
+            0.002610777737572789,
+            0.01386634074151516,
+            0.022982385009527206,
+            -0.001093322061933577,
+            -0.033960845321416855,
+            0.010887980461120605,
+            -0.008497752249240875,
+            -0.007969941012561321,
+            0.005247946362942457,
+            -0.002974590053781867,
+            0.0007238546386361122,
+            0.02136879228055477,
+            -0.011958681978285313,
+            -0.030824143439531326,
+            -0.013187726959586143,
+            -0.0007653255015611649,
+            0.01425842847675085,
+            -0.004267726093530655,
+            0.03758012130856514,
+            0.01604544371366501,
+            -0.02013220638036728,
+            0.034986305981874466,
+            0.007102823816239834,
+            0.016075605526566505,
+            0.007751276716589928,
+            0.012305528856813908,
+            0.02642069384455681,
+            0.02237917296588421,
+            -0.04065650328993797,
+            0.008301707915961742,
+            -0.01403222419321537,
+            0.0008421407546848059,
+            0.010028403252363205,
+            0.029587557539343834,
+            0.000784647127147764,
+            0.0027615807484835386,
+            0.0011649535736069083,
+            -0.008716416545212269,
+            -0.038213491439819336,
+            0.0007181995315477252,
+            0.021911682561039925,
+            0.02111242711544037,
+            0.03480534255504608,
+            0.012652375735342503,
+            0.008188605308532715,
+            0.029512157663702965,
+            -0.005545781925320625,
+            -0.003053761785849929,
+            -0.014778698794543743,
+            -0.021655317395925522,
+            -0.004049061797559261,
+            -0.008098123595118523,
+            0.012614674866199493,
+            -0.0045731025747954845,
+            0.002378917997702956,
+            0.02177596092224121,
+            0.018774978816509247,
+            -0.0017568555194884539,
+            -0.011822959408164024,
+            0.02304270677268505,
+            -0.0025297212414443493,
+            -0.016151007264852524,
+            -0.009364869445562363,
+            0.015434691682457924,
+            0.03782140463590622,
+            0.0018954058177769184,
+            0.002769120968878269,
+            0.015080304816365242,
+            -0.008218766190111637,
+            0.024113407358527184,
+            -0.02103702537715435,
+            -0.03365924209356308,
+            -0.02340463362634182,
+            0.03854525834321976,
+            0.012901200912892818,
+            -0.008369569666683674,
+            0.011513812467455864,
+            0.004878478590399027,
+            0.03549903631210327,
+            0.0010339433792978525,
+            -0.04113907366991043,
+            0.006318647880107164,
+            0.007321488112211227,
+            -0.004026441369205713,
+            -0.026827862486243248,
+            0.0010782418539747596,
+            -0.02964787930250168,
+            0.017055824398994446,
+            -0.0014712722040712833,
+            0.015374370850622654,
+            0.02115766704082489,
+            0.009379949420690536,
+            0.00980219803750515,
+            0.006665494758635759,
+            -0.0008567498298361897,
+            0.009922840632498264,
+            0.008565613068640232,
+            -0.006737126037478447,
+            0.007087743375450373,
+            0.015713678672909737,
+            0.00824892707169056,
+            0.02364591881632805,
+            0.028215250000357628,
+            0.018729738891124725,
+            -0.014801319688558578,
+            -0.018729738891124725,
+            0.005511851515620947,
+            0.03236233443021774,
+            0.012757938355207443,
+            -0.001835084636695683,
+            -0.03209088742733002,
+            0.011574134230613708,
+            0.030567778274416924,
+            0.021519595757126808,
+            -0.004705055151134729,
+            -0.011566594243049622,
+            -0.024535655975341797,
+            -0.02720486931502819,
+            0.007381809409707785,
+            0.0069030094891786575,
+            -0.008218766190111637,
+            -0.019287709146738052,
+            -0.01149873249232769,
+            -0.019574236124753952,
+            0.03190992400050163,
+            0.021293390542268753,
+            -0.00450524128973484,
+            -0.009591073729097843,
+            0.0011281953193247318,
+            -0.004331817384809256,
+            0.013934201560914516,
+            -0.0003169220290146768,
+            -0.01229798886924982,
+            0.01160429511219263,
+            -0.0139869824051857,
+            -0.009379949420690536,
+            -0.01661849580705166,
+            -0.028577178716659546,
+            -0.004968960303813219,
+            0.016362130641937256,
+            0.008784277364611626,
+            0.011672155931591988,
+            -0.017749518156051636,
+            -0.002957624848932028,
+            0.014446931891143322,
+            0.0067333560436964035,
+            -0.010782417841255665,
+            0.0011187701020389795,
+            -0.0009953001281246543,
+            0.009470431134104729,
+            -0.028170010074973106,
+            0.012124565429985523,
+            -0.02782316319644451,
+            -0.0009792773053050041,
+            0.013572274707257748,
+            -0.0014703297056257725,
+            0.0007945435936562717,
+            -0.015608116053044796,
+            -0.010556213557720184,
+            -0.003159323940053582,
+            -0.008671175688505173,
+            -0.005210245493799448,
+            -0.02390228398144245,
+            0.01609068550169468,
+            -0.005990650970488787,
+            0.011106644757091999,
+            -0.038092851638793945,
+            0.001858647563494742,
+            0.010797498747706413,
+            -0.012237667106091976,
+            -0.024927744641900063,
+            0.003643778618425131,
+            0.040988270193338394,
+            0.0017691082321107388,
+            0.006876619067043066,
+            0.0011640110751613975,
+            0.0008100951090455055,
+            -0.021172748878598213,
+            -0.00022054946748539805,
+            -0.009455351158976555,
+            -0.008731496520340443,
+            -0.0077135758474469185,
+            0.011182046495378017,
+            0.006567472591996193,
+            -0.007393119391053915,
+            0.02299746498465538,
+            -0.0036494338419288397,
+            -0.002514640800654888,
+            0.002282781060785055,
+            0.023178428411483765,
+            -0.00608490314334631,
+            0.02503330633044243,
+            0.006869079079478979,
+            0.06514691561460495,
+            -0.0035947677679359913,
+            -0.005949180107563734,
+            0.0032592308707535267,
+            -0.015819240361452103,
+            0.0011008622823283076,
+            0.011875740252435207,
+            -0.022198209539055824,
+            0.015027523972094059,
+            0.0014175486285239458,
+            -0.010179205797612667,
+            -0.012199967168271542,
+            -0.002771005965769291,
+            0.012486492283642292,
+            0.022047406062483788,
+            0.021338632330298424,
+            0.001983060035854578,
+            0.01609068550169468,
+            0.03172896057367325,
+            0.013225427828729153,
+            -0.011378089897334576,
+            0.0013402621261775494,
+            -0.00019215607608202845,
+            0.017598716542124748,
+            -0.007819138467311859,
+            0.019694877788424492,
+            -0.00296705006621778,
+            0.011883280239999294,
+            0.010118884965777397,
+            0.025561116635799408,
+            0.01895594410598278,
+            0.0010179205564782023,
+            0.00916882511228323,
+            0.008301707915961742,
+            0.017734438180923462,
+            -0.001773820840753615,
+            0.011197126470506191,
+            -0.007280017249286175,
+            0.0012592055136337876,
+            0.019544076174497604,
+            0.02041873335838318,
+            -0.024520576000213623,
+            0.003958580084145069,
+            -0.008218766190111637,
+            -0.0024053086526691914,
+            -0.0005197992431931198,
+            0.0064166695810854435,
+            0.010103804059326649,
+            -0.0061942352913320065,
+            -0.02847161516547203,
+            -0.006122604012489319,
+            -0.004912409465759993,
+            0.005368588492274284,
+            -0.02263553813099861,
+            -0.013519493862986565,
+            0.00188126799184829,
+            0.01106894388794899,
+            -0.007114133797585964,
+            -0.0011592984665185213,
+            0.002567421877756715,
+            0.021534675732254982,
+            0.011528893373906612,
+            -0.0068426881916821,
+            -0.008799358271062374,
+            -0.001767223235219717,
+            0.006963330786675215,
+            0.022605376318097115,
+            0.002514640800654888,
+            0.012712697498500347,
+            0.014597735367715359,
+            0.006518461741507053,
+            -0.009658935479819775,
+            0.01446201279759407,
+            0.0005636264104396105,
+            0.001902946038171649,
+            0.0005537299439311028,
+            -0.0008228191290982068,
+            0.004784226883202791,
+            0.00927438773214817,
+            0.01972503960132599,
+            0.0071970755234360695,
+            0.00015186337986961007,
+            0.0028765681199729443,
+            0.014100085012614727,
+            -0.016105765476822853,
+            -0.0037926966324448586,
+            -0.0012771133333444595,
+            0.012056703679263592,
+            0.007042502518743277,
+            -0.0021489434875547886,
+            0.004561792127788067,
+            0.0123960105702281,
+            0.006503381300717592,
+            -0.008791817352175713,
+            -0.016407372429966927,
+            0.005643804091960192,
+            -0.007449670694768429,
+            -0.009206525981426239,
+            -0.007717345841228962,
+            0.009553372859954834,
+            0.005681504961103201,
+            0.010307388380169868,
+            -0.019694877788424492,
+            0.007528842426836491,
+            -0.0021998395677655935,
+            -0.009930380620062351,
+            -0.003935959655791521,
+            0.010737176984548569,
+            0.019860761240124702,
+            -0.0017945562722161412,
+            0.011709856800734997,
+            -0.01606052555143833,
+            0.0024222740903496742,
+            0.009093424305319786,
+            -0.0027766611892729998,
+            0.008859679102897644,
+            0.015668436884880066,
+            -0.008625934831798077,
+            -0.03483550623059273,
+            -0.007721116300672293,
+            0.015298969112336636,
+            -0.003477895399555564,
+            0.011219747364521027,
+            -0.01658833585679531,
+            0.013270668685436249,
+            -0.017251869663596153,
+            -0.018639257177710533,
+            -0.012999222613871098,
+            -0.01776459999382496,
+            -0.010631615296006203,
+            -0.028079528361558914,
+            -0.01190590113401413,
+            0.019106745719909668,
+            0.0027615807484835386,
+            0.004486390855163336,
+            -0.00036286984686739743,
+            7.49302635085769e-05,
+            0.0012884235475212336,
+            0.02067509852349758,
+            -0.010428030975162983,
+            0.0032139900140464306,
+            0.015879560261964798,
+            -0.025229349732398987,
+            -0.0054138293489813805,
+            0.011815418489277363,
+            -0.017251869663596153,
+            0.008618393912911415,
+            -0.024234050884842873,
+            0.002989670494571328,
+            -0.01562319602817297,
+            0.018201928585767746,
+            -0.020147288218140602,
+            -0.013730617240071297,
+            0.010020862333476543,
+            -0.015065224841237068,
+            -0.013142486102879047,
+            -0.024927744641900063,
+            -5.4754465963924304e-05,
+            -0.006032121833413839,
+            -0.0070274220779538155,
+            -0.020388571545481682,
+            -0.0058775488287210464,
+            -0.007721116300672293,
+            0.0042564161121845245,
+            -0.012418631464242935,
+            0.03447357565164566,
+            0.012705156579613686,
+            0.01695026271045208,
+            -0.00990776065737009,
+            -0.027144549414515495,
+            0.004328047391027212,
+            0.03190992400050163,
+            0.013632595539093018,
+            0.007404429838061333,
+            0.007811598014086485,
+            -0.0012884235475212336,
+            -0.004659814294427633,
+            0.02197200432419777,
+            0.00021937131532467902,
+            0.013949282467365265,
+            0.004331817384809256,
+            -0.015200947411358356,
+            -0.0028859933372586966,
+            0.021006865426898003,
+            0.021610077470541,
+            -0.0015174556756392121,
+            0.0006545794894918799,
+            -0.0049425698816776276,
+            0.01366275642067194,
+            -0.008935080841183662,
+            -0.009493052028119564,
+            0.007242316380143166,
+            -0.01752331480383873,
+            -0.01817176677286625,
+            -0.0024618597235530615,
+            0.017719358205795288,
+            0.004245105665177107,
+            -0.005394978914409876,
+            0.003647548845037818,
+            0.014756078831851482,
+            0.019227389246225357,
+            0.006326187867671251,
+            -0.02194184437394142,
+            -0.0033384026028215885,
+            0.018473373726010323,
+            -0.013006762601435184,
+            -0.010013322345912457,
+            0.01143841166049242,
+            0.026737380772829056,
+            -0.0024618597235530615,
+            -0.055525682866573334,
+            -0.0018360271351411939,
+            0.012554354034364223,
+            -0.009327168576419353,
+            -0.040505699813365936,
+            -0.004607032984495163,
+            2.5757279217941687e-05,
+            -0.023057786747813225,
+            0.029104989022016525,
+            -0.00974187720566988,
+            -0.008648554794490337,
+            0.015321590006351471,
+            0.01981552131474018,
+            -0.010345089249312878,
+            -0.0054590702056884766,
+            0.009877599775791168,
+            -0.015344209969043732,
+            -0.02263553813099861,
+            0.005168774630874395,
+            -0.023103026673197746,
+            -0.02438485249876976,
+            -0.009817278943955898,
+            0.005285646766424179,
+            0.01924246922135353,
+            0.013806018978357315,
+            0.01809636689722538,
+            0.019348030909895897,
+            0.0058775488287210464,
+            -0.011830499395728111,
+            -0.015713678672909737,
+            0.008754117414355278,
+            -0.014688217081129551,
+            -0.0007342223543673754,
+            -0.001240355079062283,
+            0.008527912199497223,
+            0.006213085725903511,
+            0.013165106065571308,
+            -0.015834320336580276,
+            -0.020840981975197792,
+            -2.8039941753377207e-05,
+            -0.004177244380116463,
+            0.007272477261722088,
+            -0.015879560261964798,
+            -0.013549653813242912,
+            0.006522231735289097,
+            0.0022054945584386587,
+            -0.010081184096634388,
+            -0.0009868174092844129,
+            0.02964787930250168,
+            -0.015608116053044796,
+            -0.005820997525006533,
+            -0.025500794872641563,
+            0.0007535439799539745,
+            -0.011981301940977573,
+            0.0239324439316988,
+            0.009183906018733978,
+            -0.023510195314884186,
+            0.009515671990811825,
+            -0.027355672791600227,
+            0.017779679968953133,
+            -0.006887929514050484,
+            0.0034062638878822327,
+            0.01891070231795311,
+            0.008218766190111637,
+            0.02177596092224121,
+            0.011551513336598873,
+            -0.0007243258878588676,
+            -0.0010066103423014283,
+            0.03390052542090416,
+            -0.009930380620062351,
+            0.016347050666809082,
+            -0.0027201101183891296,
+            -0.008754117414355278,
+            0.010216906666755676,
+            -0.004079222679138184,
+            -0.007404429838061333,
+            0.011325309053063393,
+            0.01785508170723915,
+            0.002763465978205204,
+            0.007457210682332516,
+            -0.021987084299325943,
+            -0.005538241937756538,
+            -0.010058563202619553,
+            0.009078343398869038,
+            -0.000991530017927289,
+            0.008060422725975513,
+            0.0034703551791608334,
+            -0.020388571545481682,
+            -0.01833765022456646,
+            0.0028181320521980524,
+            -0.015608116053044796,
+            -0.006009501405060291,
+            -0.0075891632586717606,
+            -0.0065561626106500626,
+            0.015298969112336636,
+            -0.017327269539237022,
+            -0.004248876124620438,
+            0.0068502286449074745,
+            0.016724057495594025,
+            -0.006126374006271362,
+            -0.016151007264852524,
+            0.019830601289868355,
+            0.013798478990793228,
+            0.0322718508541584,
+            0.002786086406558752,
+            0.01589464209973812,
+            -0.01711614616215229,
+            -0.014914421364665031,
+            -0.024897582828998566,
+            0.018699578940868378,
+            -0.006458140444010496,
+            0.014507253654301167,
+            -0.01654309406876564,
+            -0.0030462215654551983,
+            -0.018654337152838707,
+            0.0002771005965769291,
+            -0.0007808770169503987,
+            -0.020162368193268776,
+            0.011958681978285313,
+            -0.0016795690171420574,
+            0.023419713601469994,
+            -0.01621132716536522,
+            -0.014514793641865253,
+            0.002884108340367675,
+            -0.021459273993968964,
+            0.008339408785104752,
+            -0.0031875993590801954,
+            -0.006597633473575115,
+            0.0005037764203734696,
+            0.00910850428044796,
+            -0.006642874330282211,
+            -0.03803252801299095,
+            -0.007076432928442955,
+            -0.004000050947070122,
+            0.006963330786675215,
+            -0.006243246141821146,
+            -0.01756855472922325,
+            -0.018081285059452057,
+            0.013489332981407642,
+            0.02646593563258648,
+            -0.005741826258599758,
+            0.007246086373925209,
+            0.01026968751102686,
+            0.01964963786303997,
+            -0.01654309406876564,
+            0.019544076174497604,
+            0.0015721217496320605,
+            0.00870133563876152,
+            0.003347827820107341,
+            0.008482671342790127,
+            0.013730617240071297,
+            0.0023826882243156433,
+            -0.00746475113555789,
+            -0.00976449716836214,
+            -0.006759746931493282,
+            0.006133913993835449,
+            0.016648657619953156,
+            -0.002486365381628275,
+            -0.01478623878210783,
+            0.00413577351719141,
+            -0.003489205613732338,
+            -0.02168547920882702,
+            -0.002601352520287037,
+            -0.012855960056185722,
+            -0.014529873616993427,
+            0.015909722074866295,
+            -0.013406391255557537,
+            -0.017749518156051636,
+            -0.00034331256756559014,
+            0.007117903791368008,
+            -0.006303567439317703,
+            -0.03103526681661606,
+            0.005839847959578037,
+            -0.007758816704154015,
+            0.030416974797844887,
+            -0.0065109217539429665,
+            0.008188605308532715,
+            0.005655114538967609,
+            0.011121724732220173,
+            0.0267977025359869,
+            -0.0002858189109247178,
+            0.015819240361452103,
+            0.008098123595118523,
+            -0.006936940364539623,
+            -0.003790811635553837,
+            0.026435773819684982,
+            -0.01505014393478632,
+            0.02124815061688423,
+            0.003585342550650239,
+            -0.012117025442421436,
+            0.017794759944081306,
+            -0.0037418007850646973,
+            0.014688217081129551,
+            0.0007851183763705194,
+            0.0030160609167069197,
+            0.00608490314334631,
+            0.0017625106265768409,
+            -0.022560136392712593,
+            -0.010850279591977596,
+            0.0024392392951995134,
+            -0.011770177632570267,
+            0.023660998791456223,
+            0.015434691682457924,
+            -0.007506221998482943,
+            0.016331970691680908,
+            -0.027838243171572685,
+            0.009206525981426239,
+            -0.013127405196428299,
+            0.011076483875513077,
+            0.0031875993590801954,
+            -0.004007590934634209,
+            -0.007596703711897135,
+            -0.013376230373978615,
+            -0.006835148204118013,
+            -0.00609621312469244,
+            0.013406391255557537,
+            0.03317667171359062,
+            0.008301707915961742,
+            -0.0060698227025568485,
+            -0.010691936127841473,
+            -0.003943499643355608,
+            0.004497700836509466,
+            0.011657075956463814,
+            0.0052064755000174046,
+            -0.00963631458580494,
+            -0.005745596252381802,
+            -0.007012341637164354,
+            -0.013760778121650219,
+            -0.007170685101300478,
+            0.009153745137155056,
+            0.004181014373898506,
+            0.006397819612175226,
+            0.0053836689330637455,
+            -0.0025485714431852102,
+            -0.0010848394595086575,
+            0.010714557021856308,
+            -0.004644733853638172,
+            -0.004614573437720537,
+            0.006820067763328552,
+            0.026164328679442406,
+            -0.0013110439758747816,
+            0.01456003449857235,
+            -0.0032875065226107836,
+            -0.002005680464208126,
+            -0.001410951022990048,
+            -0.00970417633652687,
+            -0.017553474754095078,
+            0.002520296024158597,
+            0.048015691339969635,
+            -0.01472591795027256,
+            -0.014379071071743965,
+            -0.004859628155827522,
+            0.03026617132127285,
+            -0.0014118935214355588,
+            -0.006024581845849752,
+            0.01764395646750927,
+            -0.007939780130982399,
+            0.014167946763336658,
+            0.008995401673018932,
+            -0.026043687015771866,
+            0.0014345140662044287,
+            -0.00726116681471467,
+            -0.0006875676335766912,
+            0.011649535968899727,
+            0.025108708068728447,
+            0.012184886261820793,
+            0.00275215576402843,
+            0.002882223343476653,
+            0.020479053258895874,
+            0.015200947411358356,
+            -0.00540251936763525,
+            0.002458089729771018,
+            0.0017087870510295033,
+            0.005274336785078049,
+            -0.009447811171412468,
+            -0.011219747364521027,
+            -0.017387591302394867,
+            0.00014950709010008723,
+            -0.002073541982099414,
+            0.0005749366246163845,
+            -0.022439494729042053,
+            -0.00841481052339077,
+            0.013466712087392807,
+            -0.01262221485376358,
+            -0.00963631458580494,
+            0.002147058490663767,
+            -0.0011687236838042736,
+            -0.005240405909717083,
+            0.0055307019501924515,
+            0.02405308559536934,
+            -0.007442130707204342,
+            -0.014967203140258789,
+            0.003630583407357335,
+            0.01100862305611372,
+            0.013994523324072361,
+            -0.0005881318938918412,
+            0.008399729616940022,
+            -0.005251716356724501,
+            0.011589214205741882,
+            0.010737176984548569,
+            -0.005304497200995684,
+            -0.010593914426863194,
+            -0.007178225088864565,
+            0.00746475113555789,
+            -0.015427151694893837,
+            -0.004316737409681082,
+            -2.169266554119531e-05,
+            -0.005319577641785145,
+            0.014997363090515137,
+            0.021293390542268753,
+            -0.00672958604991436,
+            0.020192528143525124,
+            -0.006284717004746199,
+            0.00540251936763525,
+            -0.0034929756075143814,
+            -0.0011517582461237907,
+            -0.008980321697890759,
+            -0.03540855646133423,
+            -0.011951141990721226,
+            0.027355672791600227,
+            -0.01039787009358406,
+            -0.008256467059254646,
+            0.023314151912927628,
+            0.003951040096580982,
+            0.009417650289833546,
+            -0.008633474819362164,
+            0.009515671990811825,
+            -0.009342248551547527,
+            0.007581623271107674,
+            0.004316737409681082,
+            0.0009962426265701652,
+            -0.004361978266388178,
+            -0.024731699377298355,
+            -0.005809687543660402,
+            -0.006220625713467598,
+            -0.02450549602508545,
+            -0.006872849073261023,
+            0.004120693542063236,
+            -0.008927540853619576,
+            -0.022695859894156456,
+            0.0007804057677276433,
+            -0.005949180107563734,
+            0.0007690955535508692,
+            -0.020433813333511353,
+            0.01039787009358406,
+            0.020569536834955215,
+            0.0346243791282177,
+            -0.03230201452970505,
+            0.00651469174772501,
+            -0.0012752283364534378,
+            0.010586374439299107,
+            -0.026888184249401093,
+            -0.0009274387848563492,
+            0.008060422725975513,
+            0.007442130707204342,
+            0.001478812424466014,
+            -0.0026579038240015507,
+            -0.003954810090363026,
+            0.025877803564071655,
+            -0.002356297569349408,
+            0.0025693068746477365,
+            -0.008837059140205383,
+            -0.013594894669950008,
+            -0.02405308559536934,
+            0.012486492283642292,
+            0.012350769713521004,
+            0.01299168262630701,
+            -0.010940761305391788,
+            -0.0008407270070165396,
+            -0.006032121833413839,
+            -0.003853017929941416,
+            0.007879459299147129,
+            0.006612713914364576,
+            -0.01202654279768467,
+            0.023057786747813225,
+            -0.011815418489277363,
+            -0.0030462215654551983,
+            0.0029972107149660587,
+            -0.0039057990070432425,
+            0.001874670386314392,
+            0.011747557669878006,
+            -0.011393170803785324,
+            -0.018684497103095055,
+            -0.009357329457998276,
+            0.013248047791421413,
+            0.010827658697962761,
+            -0.00870133563876152,
+            -0.034292612224817276,
+            -0.02156483568251133,
+            0.03172896057367325,
+            -0.01345917209982872,
+            0.010654235258698463,
+            -0.00619800528511405,
+            0.011453491635620594,
+            0.012343229725956917,
+            0.0015655241440981627,
+            -0.0034194590989500284,
+            -0.01063915528357029,
+            -0.011385629884898663,
+            -0.00843743048608303,
+            -0.0003246978158131242,
+            -0.0029764752835035324,
+            -0.014612815342843533,
+            -0.005926559679210186,
+            0.017085986211895943,
+            -0.0015589265385642648,
+            0.007668334990739822,
+            0.0027201101183891296,
+            0.013134945183992386,
+            0.006940710358321667,
+            7.58727837819606e-05,
+            0.01802096515893936,
+            -0.014363990165293217,
+            0.012750397436320782,
+            0.041651804000139236,
+            -0.0020999324042350054,
+            -0.006669264752417803,
+            -0.019227389246225357,
+            -0.007083973381668329,
+            -0.029104989022016525,
+            0.005357278510928154,
+            -0.016362130641937256,
+            -0.026375453919172287,
+            0.011641995050013065,
+            -0.016271648928523064,
+            -0.012117025442421436,
+            -0.023555437102913857,
+            0.008165985345840454,
+            -0.0038944887928664684,
+            0.011151885613799095,
+            0.0022752410732209682,
+            0.01216980628669262,
+            -0.009146205149590969,
+            -0.020494135096669197,
+            0.005353508051484823,
+            0.013798478990793228,
+            -0.002884108340367675,
+            -0.02515394799411297,
+            -0.012908740900456905,
+            0.005296957213431597,
+            -0.017719358205795288,
+            -0.010111344046890736,
+            -0.03966120257973671,
+            -0.017659036442637444,
+            0.0207505002617836,
+            0.010593914426863194,
+            -0.007909620180726051,
+            0.005545781925320625,
+            -0.0012167921522632241,
+            0.004591953009366989,
+            0.0030236011371016502,
+            0.0038096620701253414,
+            0.013180186972022057,
+            -0.002090507186949253,
+            0.004354438278824091,
+            0.00976449716836214,
+            0.013617515563964844,
+            0.006590093486011028,
+            -0.01349687296897173,
+            -0.019785359501838684,
+            0.011657075956463814,
+            -0.005063212476670742,
+            0.005342198070138693,
+            -0.0019161412492394447,
+            -0.00466735428199172,
+            -0.012674996629357338,
+            0.010111344046890736,
+            0.01308970432728529,
+            0.016030363738536835,
+            0.017779679968953133,
+            0.014115165919065475,
+            0.005919019691646099,
+            -0.015427151694893837,
+            0.001035828492604196,
+            -7.470199761883123e-06,
+            0.004018901381641626,
+            -0.007083973381668329,
+            0.006224395707249641,
+            0.003477895399555564,
+            0.010201826691627502,
+            0.011355469934642315,
+            -0.0016701437998563051,
+            0.012750397436320782,
+            -0.007332798093557358,
+            -0.003246035659685731,
+            0.0022563906386494637,
+            0.003076382214203477,
+            -0.009319628588855267,
+            0.009568453766405582,
+            0.0017172697698697448,
+            -0.004780456889420748,
+            0.010578833520412445,
+            -0.02450549602508545,
+            -0.008181065320968628,
+            -0.01276547834277153,
+            0.0011866315035149455,
+            -0.023480035364627838,
+            0.01262221485376358,
+            0.03377988189458847,
+            0.0009773923084139824,
+            -0.00225262064486742,
+            -0.00038784657954238355,
+            0.010744716972112656,
+            -0.01666373759508133,
+            -0.009493052028119564,
+            -0.0007412912673316896,
+            -0.01862417720258236,
+            -0.024580897763371468,
+            0.01833765022456646,
+            0.017839999869465828,
+            -0.013308369554579258,
+            -0.016241488978266716,
+            -0.002853947691619396,
+            -0.01833765022456646,
+            -0.010020862333476543,
+            -0.0019180262461304665,
+            -0.012984142638742924,
+            -0.019574236124753952,
+            -0.002957624848932028,
+            -0.0291200689971447,
+            0.0021150128450244665,
+            0.010096264071762562,
+            0.011385629884898663,
+            -0.010744716972112656,
+            0.0062394761480391026,
+            -0.01190590113401413,
+            -0.0033685630187392235,
+            0.006593863479793072,
+            0.01256189402192831,
+            0.03390052542090416,
+            -0.006789907347410917,
+            -0.010737176984548569,
+            0.01525372825562954,
+            0.007634404581040144,
+            0.002652248600497842,
+            0.018729738891124725,
+            0.0013921007048338652,
+            0.010616534389555454,
+            0.0018614751752465963,
+            0.015412071719765663,
+            0.01349687296897173,
+            -0.008935080841183662,
+            0.017734438180923462,
+            0.0116646159440279,
+            -0.0014986053574830294,
+            -0.019393272697925568,
+            0.014477092772722244,
+            0.010330009274184704,
+            0.013014303520321846,
+            0.0009905875194817781,
+            0.014914421364665031,
+            -0.012463872320950031,
+            -0.01768919825553894,
+            -0.01431874930858612,
+            0.00016482302453368902,
+            -0.006929399911314249,
+            0.010111344046890736,
+            -0.0006734298658557236,
+            -0.02405308559536934,
+            -0.0012714582262560725,
+            0.0035928827710449696,
+            -0.002141403267160058,
+            0.0018944633193314075,
+            -0.005342198070138693,
+            0.011973761953413486,
+            0.013263127766549587,
+            0.007871919311583042,
+            0.012750397436320782,
+            -0.008927540853619576,
+            0.009447811171412468,
+            -0.004312967415899038,
+            0.012840880081057549,
+            -0.0025108708068728447,
+            -0.0020999324042350054,
+            -3.84960112569388e-05,
+            0.0003195139579474926,
+            0.007110363803803921,
+            0.00963631458580494,
+            0.008874759078025818,
+            0.026088926941156387,
+            -0.012780558317899704,
+            0.012584514915943146,
+            0.009530752897262573,
+            -0.013700457289814949,
+            0.03133687376976013,
+            0.017025664448738098,
+            0.011920981109142303,
+            -0.017101066187024117,
+            0.010518512688577175,
+            0.023781640455126762,
+            0.024188809096813202,
+            -0.00870133563876152,
+            -0.005964260548353195,
+            -0.0003664042742457241,
+            0.022967305034399033,
+            0.015374370850622654,
+            -0.017508234828710556,
+            0.01519340742379427,
+            0.011174505576491356,
+            0.009183906018733978,
+            -0.015057684853672981,
+            0.040777143090963364,
+            -0.008686255663633347,
+            -0.017387591302394867,
+            -0.011687235906720161,
+            -0.014160406775772572,
+            -0.016392290592193604,
+            -0.005176314618438482,
+            -0.006537312176078558,
+            -0.009953001514077187,
+            0.0018482798477634788,
+            0.0016239603282883763,
+            -0.031216232106089592,
+            -0.003966120071709156,
+            -0.006314877886325121,
+            -0.013655216433107853,
+            -0.015065224841237068,
+            0.008482671342790127,
+            -0.010812578722834587,
+            0.0030575317796319723,
+            0.004482620861381292,
+            -0.020690178498625755,
+            0.01744791306555271,
+            0.023148268461227417,
+            0.0033968386705964804,
+            0.011249907314777374,
+            -0.00466735428199172,
+            0.0058775488287210464,
+            -0.0025448014494031668,
+            0.025395233184099197,
+            -0.007479831110686064,
+            0.010345089249312878,
+            -0.002599467523396015,
+            -0.01519340742379427,
+            -0.04545203968882561,
+            -0.0024147338699549437,
+            0.006865308620035648,
+            0.0181416068226099,
+            -0.01356473471969366,
+            0.007649484556168318,
+            0.013519493862986565,
+            -0.013338529504835606,
+            0.0050368220545351505,
+            0.023314151912927628,
+            -0.006152764428406954,
+            0.013044463470578194,
+            0.00953829288482666,
+            0.02304270677268505,
+            -0.011626915074884892,
+            -0.02414356917142868,
+            0.006013271398842335,
+            0.013707997277379036,
+            -0.005255486350506544,
+            0.017146306112408638,
+            0.012207507155835629,
+            -0.0139869824051857,
+            -0.004478850401937962,
+            -0.014514793641865253,
+            -0.010345089249312878,
+            0.012848420068621635,
+            -0.0047502960078418255,
+            0.018036045134067535,
+            -0.002073541982099414,
+            -0.004825697746127844,
+            -0.016724057495594025,
+            0.011792798526585102,
+            0.007849298417568207,
+            -0.007212155964225531,
+            -0.004037751816213131,
+            -0.02124815061688423,
+            -0.021519595757126808,
+            0.003251690650358796,
+            -0.024897582828998566,
+            0.02622465044260025,
+            -0.0005909594474360347,
+            0.03218137100338936,
+            0.000863818742800504,
+            -0.002763465978205204,
+            0.019378192722797394,
+            0.01707090437412262,
+            -0.013383770361542702,
+            0.0038794083520770073,
+            0.009862519800662994,
+            0.0032931615132838488,
+            -0.0008280029869638383,
+            -0.00904818344861269,
+            -0.0006696598138660192,
+            -0.0007083030650392175,
+            0.009236686863005161,
+            0.0010527938138693571,
+            0.0009330938919447362,
+            -0.008181065320968628,
+            -0.024882502853870392,
+            -0.006597633473575115,
+            0.010767337866127491,
+            0.005734285805374384,
+            -0.006337498314678669,
+            -0.012878580018877983,
+            0.007415739819407463,
+            -0.006529772188514471,
+            -0.01084273960441351,
+            -0.00873903650790453,
+            0.009779578074812889,
+            0.0020660017617046833,
+            -0.01026968751102686,
+            0.021006865426898003,
+            0.0021432882640510798,
+            -0.006910549942404032,
+            0.014379071071743965,
+            0.001902946038171649,
+            -0.017387591302394867,
+            -0.00996808148920536,
+            0.009847438894212246,
+            -0.01000578235834837,
+            0.002179104136303067,
+            0.010058563202619553,
+            -0.0009604269289411604,
+            0.0136853763833642,
+            0.003257345873862505,
+            -0.011958681978285313,
+            -0.012690076604485512,
+            -0.0282604917883873,
+            -0.00847513135522604,
+            -0.028245411813259125,
+            0.012509113177657127,
+            0.0029764752835035324,
+            0.0017945562722161412,
+            0.013504412956535816,
+            0.0018209468107670546,
+            -0.0207505002617836,
+            0.017779679968953133,
+            -0.01768919825553894,
+            0.011257447302341461,
+            0.004716365598142147,
+            0.007962401024997234,
+            -0.006797447334975004,
+            0.011302688159048557,
+            0.009787118062376976,
+            -0.02185136266052723,
+            0.018156686797738075,
+            0.02173071913421154,
+            -0.02070525847375393,
+            0.010201826691627502,
+            0.009900220669806004,
+            0.007864379324018955,
+            -0.0006555219879373908,
+            -0.013971902430057526,
+            -0.008505292236804962,
+            -0.017960643395781517,
+            0.003438309533521533,
+            0.003342172596603632,
+            -0.006360118743032217,
+            -0.005455300211906433,
+            -0.015027523972094059,
+            0.011038783006370068,
+            0.00530072720721364,
+            -0.021640237420797348,
+            -0.017266949638724327,
+            -0.0038605581503361464,
+            -0.01924246922135353,
+            0.004803077317774296,
+            0.027551718056201935,
+            -0.008346948772668839,
+            -0.015268809162080288,
+            0.024611057713627815,
+            0.02630005218088627,
+            0.009440271183848381,
+            -0.010986002162098885,
+            0.0015410187188535929,
+            -0.004855858162045479,
+            -0.011340389028191566,
+            0.005312037654221058,
+            -0.004723905585706234,
+            -0.010035943239927292,
+            0.008542993105947971,
+            -0.003732375567778945,
+            0.00858069397509098,
+            0.022349011152982712,
+            -0.006642874330282211,
+            0.004459999967366457,
+            0.009123584255576134,
+            -0.0012422400759533048,
+            0.01339885126799345,
+            -0.02577224187552929,
+            0.0108050387352705,
+            -0.025455554947257042,
+            -0.002605122746899724,
+            0.01086535956710577,
+            0.01919722743332386,
+            0.025470634922385216,
+            -0.00963631458580494,
+            0.0008741864003241062,
+            -0.0009552430710755289,
+            0.027958884835243225,
+            0.021142587065696716,
+            -0.00422625569626689,
+            -0.01678437925875187,
+            -0.014017143286764622,
+            -0.004248876124620438,
+            -0.002265815855935216,
+            0.016648657619953156,
+            -0.024203889071941376,
+            0.0044336095452308655,
+            -0.018473373726010323,
+            -0.002045266330242157,
+            -0.0013402621261775494,
+            0.014115165919065475,
+            0.015261268243193626,
+            0.002590042306110263,
+            -0.0019566696137189865,
+            0.019875841215252876,
+            0.005647574085742235,
+            -0.008196145296096802,
+            0.0042564161121845245,
+            0.005805917549878359,
+            -0.001035828492604196,
+            0.007698495872318745,
+            -0.05235881730914116,
+            0.008203686214983463,
+            0.01780983991920948,
+            0.010292308405041695,
+            0.005168774630874395,
+            0.001149873249232769,
+            -0.005640034098178148,
+            -0.015366830863058567,
+            -0.010887980461120605,
+            0.017749518156051636,
+            -0.015472392551600933,
+            0.0064128995873034,
+            0.00392087921500206,
+            -0.007826678454875946,
+            0.021293390542268753,
+            0.008075503632426262,
+            0.015035063959658146,
+            0.009651395492255688,
+            0.03047729656100273,
+            0.011423330754041672,
+            0.010827658697962761,
+            -0.005624953657388687,
+            0.00304810656234622,
+            0.0010999197838827968,
+            0.005621183663606644,
+            0.0196043960750103,
+            -0.019634557887911797,
+            -0.004821927286684513,
+            0.006066052708774805,
+            -0.0279739648103714,
+            -0.005851158406585455,
+            0.012320608831942081,
+            -0.010081184096634388,
+            -0.006722046062350273,
+            0.007121674250811338,
+            -0.0014543068828061223,
+            0.0004229554324410856,
+            0.01572875864803791,
+            0.0022073795553296804,
+            -0.013285748660564423,
+            0.01568351686000824,
+            -0.002940659411251545,
+            -0.03136703372001648,
+            -0.011740017682313919,
+            -0.011845579370856285,
+            -0.007155604660511017,
+            -0.005191395059227943,
+            -0.03091462515294552,
+            0.0008732439018785954,
+            -0.011280068196356297,
+            0.016799459233880043,
+            -0.0010509087005630136,
+            0.006853998638689518,
+            0.006752206478267908,
+            -0.010797498747706413,
+            -0.011657075956463814,
+            -0.01797572337090969,
+            -0.0011395055335015059,
+            -0.01372307725250721,
+            -0.0075891632586717606,
+            0.01752331480383873,
+            -0.021217988803982735,
+            -0.007321488112211227,
+            0.025546036660671234,
+            -0.011310229077935219,
+            -0.0027653509750962257,
+            0.006691885180771351,
+            0.0026484786067157984,
+            0.0027069146744906902,
+            0.002339332364499569,
+            0.009033102542161942,
+            -0.0008756002062000334,
+            0.002906728768721223,
+            -0.02005680650472641,
+            0.03287506476044655,
+            0.011355469934642315,
+            -0.0011395055335015059,
+            0.010691936127841473,
+            -0.02687310427427292,
+            -0.025425394997000694,
+            0.019257549196481705,
+            -0.0311257503926754,
+            -0.006111293565481901,
+            -0.01250157319009304,
+            -0.0009213123703375459,
+            -0.03218137100338936,
+            -0.025108708068728447,
+            -0.029195470735430717,
+            -0.040746983140707016,
+            0.008482671342790127,
+            -0.00225262064486742,
+            0.005519391503185034,
+            0.007773897144943476,
+            -0.025636518374085426,
+            0.013021843507885933,
+            -0.008625934831798077,
+            -0.0013242393033578992,
+            0.018986104056239128,
+            -0.007434590253978968,
+            0.004486390855163336,
+            0.017025664448738098,
+            -0.0005692815175279975,
+            -0.006620253901928663,
+            -0.014145325869321823,
+            -0.006808757781982422,
+            0.026767540723085403,
+            0.025455554947257042,
+            0.022484734654426575,
+            -0.003114083083346486,
+            0.00249202037230134,
+            -0.003898258786648512,
+            0.019619476050138474,
+            -0.018066205084323883,
+            -0.008331868797540665,
+            0.003496745601296425,
+            0.007204615511000156,
+            0.008897379972040653,
+            0.004241335671395063,
+            0.03655466064810753,
+            0.020644936710596085,
+            0.015140626579523087,
+            -0.011740017682313919,
+            -0.013308369554579258,
+            -0.01103124301880598,
+            -0.013738158158957958,
+            0.02421896904706955,
+            0.024490416049957275,
+            -0.006925629917532206,
+            -0.02304270677268505,
+            -0.011461031623184681,
+            -0.00904818344861269,
+            -0.02053937502205372,
+            0.004901099018752575,
+            0.02254505641758442,
+            0.009990702383220196,
+            0.014326289296150208,
+            -0.010608994401991367,
+            0.011468571610748768,
+            0.03299570828676224,
+            -0.012124565429985523,
+            0.006118833553045988,
+            -0.0026465936098247766,
+            -0.016362130641937256,
+            0.012742857448756695,
+            -0.015208487398922443,
+            -0.019016264006495476,
+            -0.00466735428199172,
+            -0.021881522610783577,
+            0.018126526847481728,
+            0.00035768598900176585,
+            0.003087692428380251,
+            -0.012818259187042713,
+            -0.02443009428679943,
+            0.0060170418582856655,
+            0.00046089180978015065,
+            0.003898258786648512,
+            -0.00619800528511405,
+            0.00571920583024621,
+            -0.01711614616215229,
+            0.003393068676814437,
+            -0.017719358205795288,
+            0.0219267625361681,
+            -0.010043483227491379,
+            -0.011936061084270477,
+            0.004886019043624401,
+            0.008098123595118523,
+            0.009666475467383862,
+            0.0017879586666822433,
+            -0.0028690281324088573,
+            0.007562772836536169,
+            -0.017327269539237022,
+            0.017085986211895943,
+            0.01006610319018364,
+            0.011400710791349411,
+            0.003739915555343032,
+            0.002853947691619396,
+            -0.012546814046800137,
+            0.02801920659840107,
+            -0.0008986919419839978,
+            -0.008135824464261532,
+            0.00841481052339077,
+            -0.002000025473535061,
+            0.011958681978285313,
+            0.013263127766549587,
+            -0.00798502191901207,
+            0.0006074535194784403,
+            0.003754995996132493,
+            0.008369569666683674,
+            0.015321590006351471,
+            -0.0040339813567698,
+            0.020267929881811142,
+            0.0063337283208966255,
+            -0.00022243449348025024,
+            0.012275367975234985,
+            0.0037022149190306664,
+            0.015019983984529972,
+            -0.023962603881955147,
+            -0.009206525981426239,
+            -0.023721320554614067,
+            -0.0016739139100536704,
+            -0.00741196982562542,
+            0.008754117414355278,
+            -0.0029727050568908453,
+            -0.011521353386342525,
+            0.005455300211906433,
+            0.025907963514328003,
+            0.023299071937799454,
+            -0.004267726093530655,
+            0.016648657619953156,
+            -0.006744666490703821,
+            -0.023389553651213646,
+            -0.010707016102969646,
+            -0.006624023895710707,
+            -0.03145751729607582,
+            0.022514894604682922,
+            -0.007004801649600267,
+            0.005195165053009987,
+            0.02242441289126873,
+            -0.007238546386361122,
+            -0.0033384026028215885,
+            0.01202654279768467,
+            -0.004316737409681082,
+            -0.026330212131142616,
+            0.00830924790352583,
+            -0.013285748660564423,
+            0.014575114473700523,
+            0.011996382847428322,
+            0.031638480722904205,
+            -0.013225427828729153,
+            -0.002260160632431507,
+            0.009410110302269459,
+            0.012388470582664013,
+            0.020267929881811142,
+            -0.022695859894156456,
+            -0.003210220020264387,
+            -0.001869957777671516,
+            -0.0014656170969828963,
+            -0.0029255792032927275,
+            -0.004490160848945379,
+            0.008784277364611626,
+            -0.01972503960132599,
+            -0.002989670494571328,
+            -0.005082062911242247,
+            0.003183829365298152,
+            0.003858672920614481,
+            0.0026805242523550987,
+            0.002616432961076498,
+            -0.012516653165221214,
+            -0.008037802763283253,
+            -0.006526002194732428,
+            -0.015819240361452103,
+            0.012388470582664013,
+            0.000850152166094631,
+            -0.01597004383802414,
+            0.00847513135522604,
+            0.005673964973539114,
+            0.0030914624221622944,
+            0.003460929961875081,
+            0.0034703551791608334,
+            0.05558600276708603,
+            -0.008814438246190548,
+            -0.007140524219721556,
+            -0.019348030909895897,
+            0.011166965588927269,
+            -0.013904041610658169,
+            -0.0017417751951143146,
+            0.0007804057677276433,
+            0.02194184437394142,
+            -0.028969265520572662,
+            -0.018307490274310112,
+            0.010782417841255665,
+            -0.011423330754041672,
+            -0.01127252820879221,
+            -0.0005843618419021368,
+            0.0030613020062446594,
+            -0.03172896057367325,
+            0.003935959655791521,
+            0.005572172813117504,
+            -0.004154623951762915,
+            0.01462035533040762,
+            -0.004637193866074085,
+            -0.005481690634042025,
+            0.02230377122759819,
+            0.015057684853672981,
+            -0.006235706154257059,
+            0.002520296024158597,
+            -0.00810566358268261,
+            0.014273508451879025,
+            -0.024611057713627815,
+            -0.013557193800807,
+            0.002192299347370863,
+            0.0026258581783622503,
+            -0.017221707850694656,
+            -0.007404429838061333,
+            0.0009679670911282301,
+            -0.003574032336473465,
+            0.017598716542124748,
+            -0.004659814294427633,
+            0.015223568305373192,
+            -0.0119662219658494,
+            0.03305602818727493,
+            -0.004128233529627323,
+            -0.023253830149769783,
+            0.006390279158949852,
+            0.0029990957118570805,
+            0.011363009922206402,
+            0.0004262542352080345,
+            0.01833765022456646,
+            0.015268809162080288,
+            0.0019161412492394447,
+            0.006872849073261023,
+            0.005293187219649553,
+            -0.01744791306555271,
+            0.011920981109142303,
+            -0.008158445358276367,
+            0.008791817352175713,
+            -0.0108050387352705,
+            0.0030839224345982075,
+            -0.008316787891089916,
+            0.009093424305319786,
+            -0.004595723003149033,
+            0.0025221810210496187,
+            -0.0001197588280774653,
+            -0.001998140476644039,
+            -0.0016692013014107943,
+            -0.017659036442637444,
+            -0.010186745785176754,
+            0.02058461681008339,
+            -0.02115766704082489,
+            -0.007132984232157469,
+            0.00661648390814662,
+            -0.025048386305570602,
+            -0.010887980461120605,
+            0.017327269539237022,
+            -0.015140626579523087,
+            -0.007276247255504131,
+            0.011709856800734997,
+            0.0024844801519066095,
+            -0.01323296781629324,
+            0.014205647632479668,
+            0.002084852196276188,
+            -0.036946747452020645,
+            0.011385629884898663,
+            -0.003988740500062704,
+            0.00010674028453649953,
+            -0.01054113358259201,
+            -0.047653764486312866,
+            0.006861538626253605,
+            0.0006300740060396492,
+            -0.013994523324072361,
+            0.004784226883202791,
+            0.009116044268012047,
+            0.0155628751963377,
+            -0.0032875065226107836,
+            0.009425190277397633,
+            0.014605275355279446,
+            0.0007756931590847671,
+            -3.722950350493193e-05,
+            0.019875841215252876,
+            0.0010103804524987936,
+            0.022273611277341843,
+            -0.011672155931591988,
+            -0.005168774630874395,
+            -0.023434793576598167,
+            0.010051023215055466,
+            -0.0020584615413099527,
+            -0.0005358220660127699,
+            -0.019167067483067513,
+            -0.0032139900140464306,
+            -0.015577955171465874,
+            0.02859225869178772,
+            -0.003751225769519806,
+            -0.0033685630187392235,
+            0.013082164339721203,
+            -0.003721065353602171,
+            -0.007238546386361122,
+            0.014431851916015148,
+            0.030703501775860786,
+            0.00422625569626689,
+            -0.003709755139425397,
+            0.007325258105993271,
+            -0.006469450891017914,
+            -0.012886120937764645,
+            0.0010895520681515336,
+            -0.0008284742361865938,
+            -0.01086535956710577,
+            -0.0039057990070432425,
+            0.0011451606405898929,
+            -0.011755097657442093,
+            -0.01515570655465126,
+            -0.014348910190165043,
+            0.016241488978266716,
+            -0.003223415231332183,
+            -0.00413577351719141,
+            -0.021187828853726387,
+            0.0029104987625032663,
+            -0.015065224841237068,
+            -0.004874708596616983,
+            0.014627896249294281,
+            -0.011898360215127468,
+            0.03377988189458847,
+            -0.006631563883274794,
+            0.0239324439316988,
+            -0.004297886975109577,
+            -0.0011366780381649733,
+            0.0004455758898984641,
+            -0.004482620861381292,
+            -0.014884261414408684,
+            -0.005968030542135239,
+            0.008565613068640232,
+            0.0028916485607624054,
+            0.007102823816239834,
+            0.01826224848628044,
+            -0.0015118005685508251,
+            0.045844126492738724,
+            -0.006737126037478447,
+            0.02572700008749962,
+            -0.008158445358276367,
+            -0.002590042306110263,
+            0.00386432814411819,
+            0.012531733140349388,
+            0.005157464183866978,
+            0.002361952792853117,
+            -0.022906983271241188,
+            -0.02177596092224121,
+            0.020569536834955215,
+            -0.0062470161356031895,
+            -0.0011479882523417473,
+            -0.027069147676229477,
+            0.00037135250749997795,
+            0.009078343398869038,
+            0.0009217836195603013,
+            -0.001966094831004739,
+            0.00671827606856823,
+            0.015306509099900723,
+            0.003590997541323304,
+            -0.014680677093565464,
+            -0.004935029894113541,
+            -0.026692138984799385,
+            -0.004297886975109577,
+            0.010744716972112656,
+            0.006314877886325121,
+            0.012222587130963802,
+            -0.013481792993843555,
+            -0.031547997146844864,
+            0.007796517573297024,
+            -0.009251766838133335,
+            -0.007023652084171772,
+            0.03139719367027283,
+            -0.02103702537715435,
+            -0.010247067548334599,
+            -0.02099178545176983,
+            -0.018654337152838707,
+            -0.008075503632426262,
+            0.020690178498625755,
+            -0.010624075308442116,
+            0.0046975151635706425,
+            -0.02500314638018608,
+            -0.00152405328117311,
+            0.017055824398994446,
+            0.02053937502205372,
+            0.006205545272678137,
+            0.014326289296150208
+          ],
+          "keyphrases": [
+            "Hash Layers",
+            "token ID",
+            "expert ID",
+            "load balance",
+            "Hashing based routing"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "f5b60b6b-8419-43c7-a7e5-6f47cbd89582",
+      "type": "next",
+      "source": {
+        "id": "e96a5505-edee-45dd-be65-af56dee281e4",
+        "properties": {
+          "page_content": "D Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [? ]. We use mod x to map a token ID to an expert ID. This in some way ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 2. Hashing based routing performs much worse than expert choice in terms of average scores and variance.\n\n",
+          "title": "D Comparison With Hash Layer",
+          "embedding": [
+            -0.015004903078079224,
+            -0.027099307626485825,
+            -0.030763821676373482,
+            -0.012313068844377995,
+            0.005327117629349232,
+            -0.009055723436176777,
+            -0.007592933718115091,
+            0.055405039340257645,
+            -0.039389755576848984,
+            -0.0020019104704260826,
+            -0.016120845451951027,
+            -0.011890820227563381,
+            -0.0219267625361681,
+            0.00947797205299139,
+            -0.006043432280421257,
+            0.06357856839895248,
+            0.02605876699090004,
+            0.014650516211986542,
+            -0.0008935080841183662,
+            -0.05407797545194626,
+            0.01372307725250721,
+            0.008814438246190548,
+            -0.03872622177004814,
+            -0.0012384700821712613,
+            0.021429114043712616,
+            0.0030575317796319723,
+            -0.0006979353493079543,
+            -0.023012544959783554,
+            -0.03145751729607582,
+            0.046960070729255676,
+            0.024007845669984818,
+            0.01857893541455269,
+            -0.006804987788200378,
+            -0.05121271684765816,
+            -0.024294370785355568,
+            -0.013783399015665054,
+            0.046990230679512024,
+            0.038092851638793945,
+            -0.036946747452020645,
+            0.04632669687271118,
+            0.04659814387559891,
+            0.062311820685863495,
+            -0.03190992400050163,
+            -0.010179205797612667,
+            0.000586718088015914,
+            -0.0037851566448807716,
+            -0.014424311928451061,
+            -0.0021338630467653275,
+            0.03184960409998894,
+            -0.0449996292591095,
+            -0.011257447302341461,
+            -0.07594441622495651,
+            0.025395233184099197,
+            0.012343229725956917,
+            -0.005538241937756538,
+            0.050549183040857315,
+            0.0019906002562493086,
+            0.027808083221316338,
+            0.023600677028298378,
+            -0.00661648390814662,
+            -0.004599492996931076,
+            -0.014469552785158157,
+            -3.43724932463374e-05,
+            0.015306509099900723,
+            0.010194285772740841,
+            0.0011376205366104841,
+            -0.013157566078007221,
+            -0.008015181869268417,
+            0.04210421070456505,
+            -0.015427151694893837,
+            -0.039118312299251556,
+            0.031216232106089592,
+            -0.0038567879237234592,
+            0.0019679798278957605,
+            -0.028697820380330086,
+            -0.00024034235684666783,
+            0.021821200847625732,
+            0.03218137100338936,
+            -0.00124601018615067,
+            0.020252849906682968,
+            0.012456332333385944,
+            0.0006861538859084249,
+            -0.002107472624629736,
+            0.0172820296138525,
+            0.018880542367696762,
+            -0.03504662960767746,
+            -0.02144419401884079,
+            -0.06122603639960289,
+            -0.015442232601344585,
+            -0.018232088536024094,
+            -0.01572875864803791,
+            -0.015909722074866295,
+            -0.007057582959532738,
+            -0.0011168851051479578,
+            -0.05905447527766228,
+            0.0761253759264946,
+            0.005873778834939003,
+            -0.0025391464587301016,
+            -0.023660998791456223,
+            -0.00031456572469323874,
+            -0.013579814694821835,
+            0.0022752410732209682,
+            -0.013421471230685711,
+            0.0437932051718235,
+            0.06086410954594612,
+            -0.029512157663702965,
+            0.010978462174534798,
+            0.022107727825641632,
+            -0.0007973711472004652,
+            0.1003141850233078,
+            0.006884159054607153,
+            0.005941640119999647,
+            -0.037549957633018494,
+            0.002186644123867154,
+            -0.01580416038632393,
+            -0.04240581765770912,
+            -0.03700706735253334,
+            -0.05407797545194626,
+            0.019619476050138474,
+            -0.046748943626880646,
+            0.00444114999845624,
+            0.04273758456110954,
+            -0.015140626579523087,
+            -0.0038096620701253414,
+            -0.01768919825553894,
+            0.013383770361542702,
+            -0.022861741483211517,
+            0.011189586482942104,
+            -0.0014024684205651283,
+            0.000858163577504456,
+            -0.004241335671395063,
+            -0.01000578235834837,
+            -0.029059747233986855,
+            -0.007672104984521866,
+            0.003811547067016363,
+            0.04047553986310959,
+            0.047653764486312866,
+            -0.0007059467607177794,
+            0.008761657401919365,
+            0.0011036897776648402,
+            0.012320608831942081,
+            -0.02206248603761196,
+            -0.018443213775753975,
+            0.010133964940905571,
+            0.0034119191113859415,
+            0.018322570249438286,
+            0.005353508051484823,
+            0.04521075263619423,
+            -0.03761028125882149,
+            0.0037681912072002888,
+            0.02813984826207161,
+            0.017870161682367325,
+            -0.006635334342718124,
+            0.0455123595893383,
+            -0.002667329041287303,
+            -0.015118005685508251,
+            -0.004588183015584946,
+            -0.01981552131474018,
+            0.0016041673952713609,
+            -0.0403548963367939,
+            0.014771158806979656,
+            0.00011899303353857249,
+            -6.232407031347975e-05,
+            0.02773268148303032,
+            -0.003076382214203477,
+            0.06930907815694809,
+            0.02446025423705578,
+            -0.0023204819299280643,
+            -0.07292835414409637,
+            0.027431074529886246,
+            0.0247920211404562,
+            0.01482393965125084,
+            0.049644362181425095,
+            0.06182925030589104,
+            -0.016799459233880043,
+            0.02823033183813095,
+            0.012810719199478626,
+            -0.007140524219721556,
+            -0.029994726181030273,
+            0.02352527529001236,
+            -0.04255662113428116,
+            -0.027114387601614,
+            0.046628303825855255,
+            -0.010450650937855244,
+            0.041229553520679474,
+            -0.007977481000125408,
+            0.04726167395710945,
+            0.03507678955793381,
+            -0.007902080193161964,
+            0.022198209539055824,
+            0.005432679783552885,
+            0.03423229232430458,
+            0.00433935783803463,
+            -0.007438360247761011,
+            -0.007193305529654026,
+            -0.03655466064810753,
+            -0.005575942806899548,
+            -0.024203889071941376,
+            0.033357635140419006,
+            0.006936940364539623,
+            -0.0062696365639567375,
+            -0.049855489283800125,
+            0.009251766838133335,
+            -0.014537413604557514,
+            0.01063915528357029,
+            -0.0010669315233826637,
+            0.051242876797914505,
+            -0.008957700803875923,
+            0.00243546930141747,
+            -0.021263230592012405,
+            0.002537261229008436,
+            -0.012177346274256706,
+            -0.006503381300717592,
+            0.014303669333457947,
+            -0.03100510686635971,
+            0.023600677028298378,
+            -0.006578783039003611,
+            -0.0004493459709919989,
+            -0.010986002162098885,
+            0.0029481996316462755,
+            0.040415216237306595,
+            0.015374370850622654,
+            0.017387591302394867,
+            -0.012426171451807022,
+            -0.004840777721256018,
+            0.01446201279759407,
+            -0.00019792901002801955,
+            0.009349789470434189,
+            0.007743736729025841,
+            -0.018533695489168167,
+            0.01826224848628044,
+            -0.020328251644968987,
+            0.014673137106001377,
+            0.0023921134416013956,
+            -0.006541082169860601,
+            -0.042948707938194275,
+            0.0064166695810854435,
+            -0.01943851262331009,
+            0.008188605308532715,
+            -0.002505215583369136,
+            -0.035559359937906265,
+            0.014469552785158157,
+            -0.03426245227456093,
+            0.009176366031169891,
+            -0.0007106593693606555,
+            0.011423330754041672,
+            -0.017719358205795288,
+            0.005127303767949343,
+            -0.0004592424083966762,
+            -0.01776459999382496,
+            0.01362505555152893,
+            -0.038002368062734604,
+            0.0039095692336559296,
+            0.016724057495594025,
+            -0.015743838623166084,
+            0.04246613755822182,
+            0.017824919894337654,
+            -0.03468470275402069,
+            0.00354764168150723,
+            -0.04116923362016678,
+            0.05814965441823006,
+            0.004049061797559261,
+            -0.005813457537442446,
+            0.020735418424010277,
+            -0.01219242624938488,
+            -0.024354692548513412,
+            0.020976703613996506,
+            0.0028464074712246656,
+            0.00209427741356194,
+            -0.027958884835243225,
+            -0.016347050666809082,
+            -0.043401118367910385,
+            -0.04892050847411156,
+            -0.004369518253952265,
+            0.04611557349562645,
+            -0.03272426128387451,
+            -0.022288691252470016,
+            -0.01286350004374981,
+            0.02528967149555683,
+            0.013361150398850441,
+            -0.007585393264889717,
+            0.030326493084430695,
+            0.02560635842382908,
+            -0.00020405538089107722,
+            -0.008761657401919365,
+            0.019906003028154373,
+            -0.04602508991956711,
+            0.002431699074804783,
+            -0.0633372813463211,
+            -0.010307388380169868,
+            -0.004475080408155918,
+            -0.03827381506562233,
+            -0.002079196972772479,
+            -0.0005141441361047328,
+            0.0239324439316988,
+            -0.02617940865457058,
+            -0.019091665744781494,
+            -0.0003159795014653355,
+            0.021006865426898003,
+            -0.019906003028154373,
+            -0.03010028786957264,
+            0.004007590934634209,
+            -0.03317667171359062,
+            -0.006620253901928663,
+            -0.052962031215429306,
+            -0.005176314618438482,
+            -0.009734337218105793,
+            0.01912182755768299,
+            -0.01895594410598278,
+            -0.000523569353390485,
+            0.00784175843000412,
+            0.02225852943956852,
+            0.035891126841306686,
+            -0.03896750882267952,
+            -0.007574083283543587,
+            -0.006273407023400068,
+            -0.017176467925310135,
+            0.02835097350180149,
+            -0.01207178458571434,
+            -0.008980321697890759,
+            -0.003890718799084425,
+            0.011363009922206402,
+            -0.046960070729255676,
+            0.0005061327246949077,
+            -0.013949282467365265,
+            -0.00020111000048927963,
+            0.01545731257647276,
+            0.04370272532105446,
+            -0.038424618542194366,
+            -0.0023751480039209127,
+            -0.04578380659222603,
+            -0.02859225869178772,
+            -0.038907185196876526,
+            -0.030763821676373482,
+            -0.023178428411483765,
+            -0.014575114473700523,
+            0.017538394778966904,
+            0.004173474386334419,
+            -0.021187828853726387,
+            0.009153745137155056,
+            -0.020901301875710487,
+            0.06828361749649048,
+            -0.007562772836536169,
+            0.015336669981479645,
+            0.001535363495349884,
+            -0.024580897763371468,
+            -0.009915300644934177,
+            -0.00974187720566988,
+            0.02385704219341278,
+            -0.007442130707204342,
+            0.0009670245344750583,
+            -0.0019246239680796862,
+            0.02560635842382908,
+            -0.0013713652733713388,
+            -0.005428909789770842,
+            -0.03721819445490837,
+            -0.0062168557196855545,
+            -0.010118884965777397,
+            0.030733661726117134,
+            -0.036132410168647766,
+            -7.999630179256201e-05,
+            -0.011468571610748768,
+            -0.0461457334458828,
+            0.00017471947649028152,
+            -0.04219469428062439,
+            0.003749340772628784,
+            -0.0007229121401906013,
+            -0.021534675732254982,
+            -0.022077566012740135,
+            0.01723678782582283,
+            -0.026858022436499596,
+            -0.022771259769797325,
+            0.006857768632471561,
+            0.0017804185627028346,
+            0.010020862333476543,
+            4.367986912257038e-05,
+            0.030055047944188118,
+            -0.043521761894226074,
+            -0.023133188486099243,
+            0.02761203795671463,
+            -0.022786341607570648,
+            -0.002165908692404628,
+            -0.04219469428062439,
+            -0.007347878534346819,
+            -0.040053289383649826,
+            0.020614776760339737,
+            0.02438485249876976,
+            -0.006861538626253605,
+            -0.0181416068226099,
+            0.026933424174785614,
+            -0.0037248353473842144,
+            -0.05555584281682968,
+            0.01296906266361475,
+            0.0052894167602062225,
+            0.022891903296113014,
+            -0.008279087021946907,
+            -0.024475334212183952,
+            -0.040626343339681625,
+            0.004689974710345268,
+            -0.01695026271045208,
+            -0.04919195547699928,
+            0.0204639732837677,
+            -0.032211530953645706,
+            -0.016573255881667137,
+            -0.007834218442440033,
+            0.0031461287289857864,
+            -0.002904843771830201,
+            0.019921083003282547,
+            0.001642810762859881,
+            -0.009847438894212246,
+            -0.03501646965742111,
+            -0.021474353969097137,
+            -0.03365924209356308,
+            -0.021745799109339714,
+            0.006529772188514471,
+            0.005892629269510508,
+            -0.014311209321022034,
+            0.04267726466059685,
+            -0.015487473458051682,
+            0.034775182604789734,
+            -0.0016041673952713609,
+            -0.016467692330479622,
+            0.05468118563294411,
+            0.012923821806907654,
+            -0.00233367714099586,
+            0.009937920607626438,
+            -0.00265036360360682,
+            0.0019472443964332342,
+            0.012637295760214329,
+            -0.004316737409681082,
+            0.031638480722904205,
+            -0.0005683390190824866,
+            0.009953001514077187,
+            -0.02708422765135765,
+            -0.013104785233736038,
+            -0.011574134230613708,
+            -0.040173932909965515,
+            0.02875814214348793,
+            -0.008037802763283253,
+            -0.002033956116065383,
+            0.018126526847481728,
+            0.012667456641793251,
+            -0.010948301292955875,
+            -0.03374972194433212,
+            -0.015291429124772549,
+            -0.00853545218706131,
+            -0.007879459299147129,
+            -0.012275367975234985,
+            -0.04457738250494003,
+            0.008550533093512058,
+            -0.0535954050719738,
+            -0.01996632292866707,
+            -0.05443990230560303,
+            -0.008867219090461731,
+            0.03038681484758854,
+            -4.72732208436355e-05,
+            0.012916280888020992,
+            0.01054113358259201,
+            0.0054062893614172935,
+            -0.03646417707204819,
+            -0.013677836395800114,
+            0.024897582828998566,
+            0.02136879228055477,
+            -0.01472591795027256,
+            -0.038092851638793945,
+            0.017432833090424538,
+            -0.01666373759508133,
+            -0.047925207763910294,
+            -0.011815418489277363,
+            0.011430870741605759,
+            -0.012554354034364223,
+            -0.060803789645433426,
+            -0.01764395646750927,
+            -0.018563855439424515,
+            -0.008610853925347328,
+            0.028893863782286644,
+            0.020403653383255005,
+            -0.004795536864548922,
+            -0.03679594397544861,
+            0.0018567625666037202,
+            0.015653356909751892,
+            0.006435520015656948,
+            0.032000407576560974,
+            -0.008550533093512058,
+            0.03803252801299095,
+            0.011355469934642315,
+            0.02589288353919983,
+            0.014213187620043755,
+            0.0016013398999348283,
+            0.0071970755234360695,
+            -0.01568351686000824,
+            -0.01362505555152893,
+            -0.011174505576491356,
+            -0.028320813551545143,
+            -0.009349789470434189,
+            -0.018292410299181938,
+            0.02124815061688423,
+            -0.0316988006234169,
+            0.04141051694750786,
+            0.01854877546429634,
+            -0.0007818195736035705,
+            0.010412951000034809,
+            0.001439226558431983,
+            0.00450524128973484,
+            0.007555232848972082,
+            -0.0016211328329518437,
+            0.025018226355314255,
+            0.023721320554614067,
+            -0.034051328897476196,
+            -0.013806018978357315,
+            0.014235807582736015,
+            -0.010103804059326649,
+            0.016889940947294235,
+            -0.005165004637092352,
+            -0.005662654526531696,
+            0.03160832077264786,
+            -0.024852342903614044,
+            0.02103702537715435,
+            0.003636238630861044,
+            -0.014243348501622677,
+            -0.02495790459215641,
+            0.0059944214299321175,
+            -0.023540355265140533,
+            0.04406465217471123,
+            -0.03531807288527489,
+            0.02756679803133011,
+            0.004497700836509466,
+            -0.015351750887930393,
+            0.03059793822467327,
+            -0.011166965588927269,
+            -0.017342351377010345,
+            -0.003223415231332183,
+            0.030160609632730484,
+            -0.010827658697962761,
+            -0.023600677028298378,
+            0.018699578940868378,
+            -0.007121674250811338,
+            0.027219951152801514,
+            -0.01413024589419365,
+            -0.0025466864462941885,
+            0.03332747519016266,
+            0.006303567439317703,
+            -0.0016456382581964135,
+            0.009266847744584084,
+            0.012086864560842514,
+            0.031186070293188095,
+            0.02969312109053135,
+            -0.025274591520428658,
+            -0.013361150398850441,
+            0.04032473638653755,
+            0.010978462174534798,
+            -0.002395883435383439,
+            0.012184886261820793,
+            -0.0032498056534677744,
+            -0.013594894669950008,
+            0.002346872352063656,
+            0.0007521302322857082,
+            -0.012109484523534775,
+            0.006420440040528774,
+            -0.028154930099844933,
+            -0.0017398901982232928,
+            0.01883530057966709,
+            0.013519493862986565,
+            -0.005915249697864056,
+            -0.013557193800807,
+            0.01525372825562954,
+            -0.01984568126499653,
+            0.024158649146556854,
+            -0.013821099884808064,
+            0.006582553032785654,
+            0.01425842847675085,
+            -0.010239526629447937,
+            -0.03281474485993385,
+            0.00847513135522604,
+            -3.4195771149825305e-05,
+            -0.009493052028119564,
+            0.003455274971202016,
+            -0.009794658049941063,
+            -0.008950160816311836,
+            -0.0032611158676445484,
+            0.011513812467455864,
+            -0.02940659411251545,
+            -0.0002133627567673102,
+            0.027762841433286667,
+            0.024988064542412758,
+            -0.013542113825678825,
+            -0.005741826258599758,
+            -0.010759797878563404,
+            0.009681555442512035,
+            -0.004282806534320116,
+            -0.009583533741533756,
+            0.027913644909858704,
+            0.02185136266052723,
+            -0.03212105110287666,
+            0.006420440040528774,
+            -0.005485461093485355,
+            0.0019547846168279648,
+            0.004999121185392141,
+            0.015238648280501366,
+            -0.018729738891124725,
+            0.00609621312469244,
+            -0.006454370450228453,
+            0.013994523324072361,
+            -0.014371531084179878,
+            0.023706238716840744,
+            0.017915401607751846,
+            0.009349789470434189,
+            -0.01482393965125084,
+            -0.006503381300717592,
+            0.010390330106019974,
+            -0.001478812424466014,
+            -0.010556213557720184,
+            0.035800643265247345,
+            0.00704627251252532,
+            -0.0011885165004059672,
+            0.01562319602817297,
+            0.028969265520572662,
+            0.02316334843635559,
+            -0.003754995996132493,
+            0.011853119358420372,
+            0.06774073094129562,
+            -0.018247168511152267,
+            0.005658884532749653,
+            0.0005353508167900145,
+            0.020041724666953087,
+            0.006130144000053406,
+            -0.032000407576560974,
+            -0.034051328897476196,
+            0.0001823774364311248,
+            -0.026073846966028214,
+            -0.04735215753316879,
+            -0.014047304168343544,
+            -0.01809636689722538,
+            -0.03172896057367325,
+            -0.0042639560997486115,
+            -0.00963631458580494,
+            0.005787067115306854,
+            0.023600677028298378,
+            0.004113153088837862,
+            -0.027385834604501724,
+            -0.016678817570209503,
+            0.02699374593794346,
+            0.025274591520428658,
+            0.03736899420619011,
+            -0.005681504961103201,
+            0.011174505576491356,
+            0.022529976442456245,
+            0.002050921553745866,
+            0.00927438773214817,
+            -0.004026441369205713,
+            -0.008052882738411427,
+            0.027174709364771843,
+            0.02703898772597313,
+            0.029270872473716736,
+            -0.0011461031390354037,
+            -0.034775182604789734,
+            -0.015298969112336636,
+            -0.018081285059452057,
+            -0.0006102810730226338,
+            0.008693795651197433,
+            -0.009033102542161942,
+            0.018352732062339783,
+            -0.0004740870790556073,
+            0.018518613651394844,
+            0.025591278448700905,
+            0.0022865512873977423,
+            -0.022394252941012383,
+            0.010495892725884914,
+            -0.0009608981781639159,
+            0.0071669151075184345,
+            -0.029843922704458237,
+            -0.04125971347093582,
+            -0.056430500000715256,
+            0.007068892940878868,
+            0.03254329785704613,
+            -0.042375657707452774,
+            0.01609068550169468,
+            0.04741247743368149,
+            -0.009018022567033768,
+            -0.023676078766584396,
+            -0.015864480286836624,
+            0.005410059355199337,
+            -0.022741099819540977,
+            -0.018307490274310112,
+            0.037067390978336334,
+            -0.015502553433179855,
+            -0.017870161682367325,
+            0.02079574018716812,
+            -0.02295222505927086,
+            0.02041873335838318,
+            -0.02818509005010128,
+            -0.002237540204077959,
+            -0.03311634808778763,
+            0.04014377295970917,
+            -0.012260288000106812,
+            0.00731017766520381,
+            -0.013097245246171951,
+            0.044456738978624344,
+            -0.008083043619990349,
+            0.017101066187024117,
+            0.0012997337616980076,
+            -0.001241297577507794,
+            -0.00830924790352583,
+            0.004893559031188488,
+            0.02838113345205784,
+            -0.017598716542124748,
+            0.0069030094891786575,
+            0.011264988221228123,
+            -0.03136703372001648,
+            -0.0187448188662529,
+            0.0037870416417717934,
+            0.021263230592012405,
+            0.015314050018787384,
+            0.04394400864839554,
+            0.028562096878886223,
+            -0.009930380620062351,
+            -0.008346948772668839,
+            0.009756957180798054,
+            -0.031276553869247437,
+            0.03709755092859268,
+            -0.013579814694821835,
+            0.031759120523929596,
+            0.028622418642044067,
+            -0.011264988221228123,
+            -0.001501432852819562,
+            -0.0027540407609194517,
+            0.02058461681008339,
+            0.03803252801299095,
+            0.007528842426836491,
+            0.005142384208738804,
+            0.02560635842382908,
+            0.010910600423812866,
+            -0.02626989036798477,
+            0.01826224848628044,
+            -0.004509011283516884,
+            -0.004780456889420748,
+            0.025259511545300484,
+            0.014265968464314938,
+            0.01409254502505064,
+            0.0017427176935598254,
+            0.028848623856902122,
+            -0.019875841215252876,
+            0.010118884965777397,
+            0.002904843771830201,
+            -0.012697616592049599,
+            0.015314050018787384,
+            0.0068276082165539265,
+            0.021263230592012405,
+            -0.004618343431502581,
+            0.02985900454223156,
+            0.01768919825553894,
+            0.01649785414338112,
+            0.0386357419192791,
+            0.006024581845849752,
+            -0.049041152000427246,
+            0.012049163691699505,
+            0.017176467925310135,
+            -0.0031423585023730993,
+            -0.021655317395925522,
+            -0.004041521809995174,
+            -0.002746500540524721,
+            0.007924700155854225,
+            0.0039057990070432425,
+            -0.01568351686000824,
+            -0.017824919894337654,
+            -0.004324277397245169,
+            0.010707016102969646,
+            0.021308470517396927,
+            0.010164125822484493,
+            0.005930330138653517,
+            0.006375198718160391,
+            0.040505699813365936,
+            0.02708422765135765,
+            -0.02495790459215641,
+            0.033387795090675354,
+            0.011543973349034786,
+            -0.015110465697944164,
+            -0.015223568305373192,
+            -0.022514894604682922,
+            0.0028690281324088573,
+            -0.02838113345205784,
+            -0.004659814294427633,
+            0.01883530057966709,
+            0.02209264598786831,
+            -0.013542113825678825,
+            -0.002821902045980096,
+            0.024701539427042007,
+            0.0009255537297576666,
+            -0.011363009922206402,
+            -0.021459273993968964,
+            0.011016163043677807,
+            0.0067785969004035,
+            -4.6360157284652814e-05,
+            0.008821978233754635,
+            0.04922211542725563,
+            -0.0055834827944636345,
+            0.027958884835243225,
+            0.049734845757484436,
+            0.0027917413972318172,
+            -0.02871290035545826,
+            0.02225852943956852,
+            0.007359188981354237,
+            -0.004169704392552376,
+            0.0221831277012825,
+            -0.005225325468927622,
+            -0.04819665476679802,
+            0.022861741483211517,
+            -0.028109688311815262,
+            -0.006005731411278248,
+            0.009289467707276344,
+            0.011385629884898663,
+            -0.002250735415145755,
+            -0.026692138984799385,
+            0.036253053694963455,
+            -0.006620253901928663,
+            0.015223568305373192,
+            -0.023374471813440323,
+            -0.027008825913071632,
+            -0.014974743127822876,
+            -0.007687185425311327,
+            -0.009749417193233967,
+            0.0031348182819783688,
+            -0.00901048257946968,
+            0.016241488978266716,
+            -0.03619273379445076,
+            0.003183829365298152,
+            0.012418631464242935,
+            0.00450524128973484,
+            0.04762360453605652,
+            0.00959861371666193,
+            0.00757785327732563,
+            -0.005915249697864056,
+            -0.012162266299128532,
+            0.023540355265140533,
+            0.010933221317827702,
+            0.03516726940870285,
+            -0.004399679135531187,
+            0.007223465945571661,
+            0.030447136610746384,
+            0.024927744641900063,
+            0.02201724611222744,
+            -0.016844701021909714,
+            -0.004181014373898506,
+            -0.00476914644241333,
+            -0.014590195380151272,
+            0.011822959408164024,
+            -0.005805917549878359,
+            0.008987861685454845,
+            0.012380930595099926,
+            0.007434590253978968,
+            0.011581674218177795,
+            0.06249278411269188,
+            0.015879560261964798,
+            -0.00888229999691248,
+            -0.026481015607714653,
+            -0.0032498056534677744,
+            0.007657025009393692,
+            0.00959861371666193,
+            0.0019736348185688257,
+            -0.009794658049941063,
+            0.014265968464314938,
+            0.00824892707169056,
+            -0.03402116894721985,
+            0.002678639255464077,
+            -0.004015131387859583,
+            0.006130144000053406,
+            0.01492950227111578,
+            -0.0403548963367939,
+            -0.00259381253272295,
+            -0.020720338448882103,
+            -0.004799306858330965,
+            0.022620458155870438,
+            -0.010412951000034809,
+            0.005979340989142656,
+            0.0016503508668392897,
+            -0.028803382068872452,
+            0.003958580084145069,
+            -0.0054364497773349285,
+            0.02777792140841484,
+            -0.028079528361558914,
+            -0.008821978233754635,
+            -0.013677836395800114,
+            -0.007600473705679178,
+            -0.017417753115296364,
+            0.029180390760302544,
+            -0.019589316099882126,
+            -0.002714454894885421,
+            -0.031155910342931747,
+            -0.014884261414408684,
+            -0.01476361881941557,
+            0.026375453919172287,
+            -0.01308970432728529,
+            -0.04481866583228111,
+            0.008482671342790127,
+            -0.012441251426935196,
+            0.00222057499922812,
+            -0.026194490492343903,
+            0.0030914624221622944,
+            0.028652578592300415,
+            0.02058461681008339,
+            0.006507151760160923,
+            -0.028411295264959335,
+            -0.010412951000034809,
+            0.01817176677286625,
+            0.04620605334639549,
+            -0.020765580236911774,
+            0.0128257991746068,
+            -0.012252748012542725,
+            0.01069947611540556,
+            -0.044004328548908234,
+            -0.0042337956838309765,
+            0.001988715259358287,
+            0.021821200847625732,
+            -0.015412071719765663,
+            0.002610777737572789,
+            0.05253978073596954,
+            0.017794759944081306,
+            -0.05163496360182762,
+            0.01866941712796688,
+            -0.012019002810120583,
+            0.034533899277448654,
+            0.05534471943974495,
+            -0.004535401705652475,
+            0.006936940364539623,
+            0.020765580236911774,
+            -0.022032326087355614,
+            0.010450650937855244,
+            0.014876720495522022,
+            -0.00576821668073535,
+            -0.02259029634296894,
+            -0.013579814694821835,
+            0.021640237420797348,
+            0.012592054903507233,
+            -0.012305528856813908,
+            0.03872622177004814,
+            -0.0007318660500459373,
+            0.029587557539343834,
+            -0.01678437925875187,
+            -0.03166864067316055,
+            -0.002508985809981823,
+            -0.018081285059452057,
+            0.025274591520428658,
+            0.010390330106019974,
+            0.0015985122881829739,
+            0.006699425633996725,
+            -0.035921286791563034,
+            0.01857893541455269,
+            0.02503330633044243,
+            0.0003843121521640569,
+            0.001393043203279376,
+            -0.02541031315922737,
+            0.02168547920882702,
+            -0.004761606454849243,
+            0.018186848610639572,
+            0.022605376318097115,
+            -0.0027955116238445044,
+            -0.019167067483067513,
+            0.005836077965795994,
+            0.0012083094334229827,
+            0.04346143826842308,
+            -0.001625845325179398,
+            0.0026239731814712286,
+            -0.01826224848628044,
+            -0.0032724260818213224,
+            -0.019001184031367302,
+            0.02572700008749962,
+            -0.005545781925320625,
+            -0.017432833090424538,
+            -0.018352732062339783,
+            0.03136703372001648,
+            -0.0007634404464624822,
+            -0.032332174479961395,
+            -0.003114083083346486,
+            -0.012433711439371109,
+            -0.0018379121320322156,
+            0.018639257177710533,
+            -0.03157815709710121,
+            -0.017251869663596153,
+            -0.0017493154155090451,
+            0.022077566012740135,
+            0.026948504149913788,
+            0.01883530057966709,
+            -0.0024712849408388138,
+            -0.0001750729134073481,
+            0.011076483875513077,
+            -0.0004952937597408891,
+            0.011506272479891777,
+            0.01299168262630701,
+            -0.007370498962700367,
+            0.03287506476044655,
+            -0.006367658730596304,
+            -0.0036173881962895393,
+            0.023887204006314278,
+            -0.01063915528357029,
+            -0.008731496520340443,
+            0.018654337152838707,
+            -0.018352732062339783,
+            -0.015502553433179855,
+            -0.007804058026522398,
+            0.005662654526531696,
+            -0.004886019043624401,
+            -0.0015080304583534598,
+            3.1014766136649996e-05,
+            0.03724835440516472,
+            -0.027265191078186035,
+            0.03178928419947624,
+            0.043220154941082,
+            -0.009847438894212246,
+            -0.02863749861717224,
+            -0.0016013398999348283,
+            -0.003496745601296425,
+            0.02455073595046997,
+            0.006631563883274794,
+            -0.0006592920981347561,
+            -0.04195340722799301,
+            0.015954962000250816,
+            -0.016226409003138542,
+            -0.011679695919156075,
+            -0.006024581845849752,
+            -0.004791766870766878,
+            0.009410110302269459,
+            0.028154930099844933,
+            -0.02801920659840107,
+            0.01190590113401413,
+            0.012418631464242935,
+            0.024083247408270836,
+            -0.005342198070138693,
+            -0.01409254502505064,
+            -0.01103124301880598,
+            -0.011287608183920383,
+            0.012531733140349388,
+            -0.0164224524050951,
+            -0.005319577641785145,
+            0.00519893504679203,
+            -0.005327117629349232,
+            0.01100862305611372,
+            -0.016920102760195732,
+            -0.06375952810049057,
+            -0.011114184744656086,
+            0.03667530044913292,
+            -0.014801319688558578,
+            -0.018880542367696762,
+            0.03712771087884903,
+            0.009545832872390747,
+            0.021640237420797348,
+            -0.023947523906826973,
+            0.0155628751963377,
+            -0.006767286919057369,
+            0.0029953254852443933,
+            -0.019785359501838684,
+            -0.010955841280519962,
+            0.00870133563876152,
+            -0.004429839551448822,
+            -0.004143313970416784,
+            -0.024249130859971046,
+            0.02617940865457058,
+            0.008264007046818733,
+            -0.02271093986928463,
+            0.007592933718115091,
+            0.008656094782054424,
+            0.005376128945499659,
+            -0.01604544371366501,
+            0.013572274707257748,
+            0.011822959408164024,
+            -0.01345917209982872,
+            0.02275617979466915,
+            -0.025063466280698776,
+            -0.004486390855163336,
+            0.013775859028100967,
+            0.030447136610746384,
+            0.014454471878707409,
+            -0.021670397371053696,
+            -0.02148943394422531,
+            0.011242367327213287,
+            0.0015909721842035651,
+            0.0034119191113859415,
+            0.000781348324380815,
+            -0.00709528336301446,
+            -0.04451705887913704,
+            0.01984568126499653,
+            -0.0020660017617046833,
+            -0.015035063959658146,
+            0.00412446353584528,
+            0.015909722074866295,
+            0.013609975576400757,
+            0.007487371563911438,
+            -0.006371428724378347,
+            0.01096338126808405,
+            0.033629078418016434,
+            -0.019001184031367302,
+            -0.022680778056383133,
+            -0.0063638887368142605,
+            -0.0010094379540532827,
+            0.01838289201259613,
+            -0.010360169224441051,
+            0.004395908676087856,
+            0.01809636689722538,
+            0.07220450043678284,
+            -0.014710837043821812,
+            0.00022785397595725954,
+            -0.03206072747707367,
+            -0.02340463362634182,
+            0.00108766695484519,
+            -0.0036569740623235703,
+            0.022198209539055824,
+            0.021293390542268753,
+            0.018066205084323883,
+            0.010691936127841473,
+            -0.007234776392579079,
+            0.002895418554544449,
+            0.025395233184099197,
+            -0.013353610411286354,
+            0.030809063464403152,
+            0.007909620180726051,
+            0.016316890716552734,
+            0.001337434514425695,
+            -0.015080304816365242,
+            0.012509113177657127,
+            0.033267151564359665,
+            -0.010857819579541683,
+            0.002940659411251545,
+            -0.004331817384809256,
+            0.0099454615265131,
+            -0.00028275573276914656,
+            -0.008158445358276367,
+            -0.009206525981426239,
+            -0.023057786747813225,
+            0.014115165919065475,
+            -0.014205647632479668,
+            -0.004735215567052364,
+            0.005263026338070631,
+            -0.016105765476822853,
+            0.062191177159547806,
+            -0.008769197389483452,
+            0.008324328809976578,
+            0.0077361962758004665,
+            -0.008497752249240875,
+            -0.005425139795988798,
+            0.008173525333404541,
+            0.005968030542135239,
+            0.011619375087320805,
+            -0.005149924196302891,
+            0.0037813864182680845,
+            -0.0024335843045264482,
+            -0.011189586482942104,
+            -0.00015857882681302726,
+            -0.023133188486099243,
+            0.016392290592193604,
+            0.006484531331807375,
+            -0.023615757003426552,
+            -0.0007064180099405348,
+            -0.0049425698816776276,
+            -0.00741196982562542,
+            0.002096162410452962,
+            -0.017251869663596153,
+            -0.0034119191113859415,
+            0.014612815342843533,
+            -0.004799306858330965,
+            -0.003898258786648512,
+            0.008075503632426262,
+            0.013127405196428299,
+            -0.008135824464261532,
+            -0.009183906018733978,
+            0.02156483568251133,
+            0.003148013725876808,
+            0.0034816653933376074,
+            0.006156534422188997,
+            -0.009726796299219131,
+            0.024988064542412758,
+            -0.01915198750793934,
+            -0.01435645017772913,
+            0.0050669824704527855,
+            0.010020862333476543,
+            0.019257549196481705,
+            -0.020328251644968987,
+            -0.003347827820107341,
+            -0.010940761305391788,
+            -0.008791817352175713,
+            0.022891903296113014,
+            -0.002870913129299879,
+            -0.023555437102913857,
+            -0.016482774168252945,
+            0.0025184110272675753,
+            -0.01833765022456646,
+            -0.005379898939281702,
+            -0.008339408785104752,
+            -0.005866238847374916,
+            -0.02103702537715435,
+            -0.018563855439424515,
+            0.0033384026028215885,
+            -0.029964566230773926,
+            -0.0036456636153161526,
+            -0.011046322993934155,
+            -0.0005975571111775935,
+            0.0076834154315292835,
+            -0.04578380659222603,
+            0.0005622126045636833,
+            -0.035227593034505844,
+            0.002356297569349408,
+            0.008656094782054424,
+            0.005304497200995684,
+            -0.013783399015665054,
+            0.020071886479854584,
+            0.017372511327266693,
+            -0.01981552131474018,
+            0.015713678672909737,
+            0.005334658082574606,
+            0.00018390902550891042,
+            0.0037983518559485674,
+            0.007672104984521866,
+            -0.007385579403489828,
+            0.0072272359393537045,
+            0.010186745785176754,
+            0.0028407524805516005,
+            -0.017025664448738098,
+            -0.012652375735342503,
+            -0.019167067483067513,
+            -0.008573153056204319,
+            0.006853998638689518,
+            0.021142587065696716,
+            -0.00959861371666193,
+            0.004901099018752575,
+            0.00671827606856823,
+            0.008625934831798077,
+            -0.016271648928523064,
+            0.01972503960132599,
+            -0.009900220669806004,
+            -0.0018906932091340423,
+            -0.018865462392568588,
+            -0.000623005093075335,
+            -0.00036051354254595935,
+            0.01372307725250721,
+            -0.009221606887876987,
+            -0.00444114999845624,
+            -0.0037531109992414713,
+            0.006337498314678669,
+            -0.010940761305391788,
+            -0.0013581699458882213,
+            0.00020865016267634928,
+            0.0267977025359869,
+            0.014363990165293217,
+            0.0008280029869638383,
+            -0.03966120257973671,
+            -0.027989046648144722,
+            0.025274591520428658,
+            0.00513484375551343,
+            0.012953981757164001,
+            -0.02409832738339901,
+            -0.018081285059452057,
+            0.0328449048101902,
+            0.0012808834435418248,
+            -0.000869473849888891,
+            -0.034986305981874466,
+            0.016603415831923485,
+            -0.015577955171465874,
+            0.02875814214348793,
+            0.025259511545300484,
+            -0.014801319688558578,
+            0.027702519670128822,
+            0.005391208920627832,
+            -0.022967305034399033,
+            0.008324328809976578,
+            0.006963330786675215,
+            0.019875841215252876,
+            -0.01515570655465126,
+            0.01601528376340866,
+            0.003886948572471738,
+            -0.017342351377010345,
+            -0.007502451539039612,
+            0.014205647632479668,
+            0.02756679803133011,
+            0.005802147556096315,
+            0.005485461093485355,
+            0.001331779407337308,
+            -0.036494337022304535,
+            0.015638276934623718,
+            -0.0019415892893448472,
+            0.01345917209982872,
+            -0.01637721061706543,
+            0.004181014373898506,
+            0.007924700155854225,
+            -0.015170786529779434,
+            -0.013270668685436249,
+            -0.029346274212002754,
+            0.005519391503185034,
+            -0.009116044268012047,
+            0.0020245308987796307,
+            -0.0033723332453519106,
+            0.0071669151075184345,
+            0.020569536834955215,
+            -0.014333830215036869,
+            -0.0035646071191877127,
+            0.005112223327159882,
+            0.008407269604504108,
+            -0.001880325493402779,
+            0.002735190326347947,
+            0.002011335687711835,
+            -0.00020111000048927963,
+            0.02835097350180149,
+            -0.016000203788280487,
+            -0.013134945183992386,
+            0.004101843107491732,
+            -0.00937240943312645,
+            0.002397768432274461,
+            -0.00646568089723587,
+            0.009628774598240852,
+            -0.0328449048101902,
+            -0.0018595900619402528,
+            -0.014409231022000313,
+            -0.021881522610783577,
+            -0.0013072739820927382,
+            0.008927540853619576,
+            0.009485512040555477,
+            -0.02099178545176983,
+            0.008422350510954857,
+            0.020433813333511353,
+            0.007702265866100788,
+            0.00867871567606926,
+            0.02299746498465538,
+            0.027793001383543015,
+            0.010028403252363205,
+            -0.008814438246190548,
+            -0.016000203788280487,
+            0.019830601289868355,
+            -0.015909722074866295,
+            0.013308369554579258,
+            0.00810566358268261,
+            0.004361978266388178,
+            0.03002488799393177,
+            0.018518613651394844,
+            0.00933470856398344,
+            0.011792798526585102,
+            0.004591953009366989,
+            0.01425842847675085,
+            -0.0042111752554774284,
+            0.01482393965125084,
+            0.00023786825477145612,
+            -0.00845251139253378,
+            0.00014538357208948582,
+            -0.0051876250654459,
+            0.026601657271385193,
+            -0.011755097657442093,
+            -0.00017742921772878617,
+            -0.011355469934642315,
+            -0.027340592816472054,
+            -0.010571293532848358,
+            0.005477920640259981,
+            -0.024249130859971046,
+            0.004637193866074085,
+            0.004486390855163336,
+            -0.01296906266361475,
+            -0.016482774168252945,
+            -0.015306509099900723,
+            0.010925681330263615,
+            -0.0006616484024561942,
+            0.008919999934732914,
+            0.0057003553956747055,
+            -0.011023703031241894,
+            0.003400608664378524,
+            0.020162368193268776,
+            0.021640237420797348,
+            0.0011074598878622055,
+            0.02646593563258648,
+            0.0013864454813301563,
+            -0.010616534389555454,
+            -0.0071669151075184345,
+            0.0017643957398831844,
+            -0.007962401024997234,
+            -0.005168774630874395,
+            -0.009093424305319786,
+            -0.0035551819019019604,
+            0.020252849906682968,
+            -0.004799306858330965,
+            -0.015758918598294258,
+            0.0006687172572128475,
+            -0.01649785414338112,
+            -0.0038831785786896944,
+            -0.008972781710326672,
+            0.04044537618756294,
+            -0.021459273993968964,
+            -0.007547692861407995,
+            -0.02981376275420189,
+            0.0024769401643425226,
+            -0.0015928571810945868,
+            0.00629979744553566,
+            0.004927489440888166,
+            -0.03402116894721985,
+            -0.05658130347728729,
+            0.027355672791600227,
+            0.002610777737572789,
+            0.01386634074151516,
+            0.022982385009527206,
+            -0.001093322061933577,
+            -0.033960845321416855,
+            0.010887980461120605,
+            -0.008497752249240875,
+            -0.007969941012561321,
+            0.005247946362942457,
+            -0.002974590053781867,
+            0.0007238546386361122,
+            0.02136879228055477,
+            -0.011958681978285313,
+            -0.030824143439531326,
+            -0.013187726959586143,
+            -0.0007653255015611649,
+            0.01425842847675085,
+            -0.004267726093530655,
+            0.03758012130856514,
+            0.01604544371366501,
+            -0.02013220638036728,
+            0.034986305981874466,
+            0.007102823816239834,
+            0.016075605526566505,
+            0.007751276716589928,
+            0.012305528856813908,
+            0.02642069384455681,
+            0.02237917296588421,
+            -0.04065650328993797,
+            0.008301707915961742,
+            -0.01403222419321537,
+            0.0008421407546848059,
+            0.010028403252363205,
+            0.029587557539343834,
+            0.000784647127147764,
+            0.0027615807484835386,
+            0.0011649535736069083,
+            -0.008716416545212269,
+            -0.038213491439819336,
+            0.0007181995315477252,
+            0.021911682561039925,
+            0.02111242711544037,
+            0.03480534255504608,
+            0.012652375735342503,
+            0.008188605308532715,
+            0.029512157663702965,
+            -0.005545781925320625,
+            -0.003053761785849929,
+            -0.014778698794543743,
+            -0.021655317395925522,
+            -0.004049061797559261,
+            -0.008098123595118523,
+            0.012614674866199493,
+            -0.0045731025747954845,
+            0.002378917997702956,
+            0.02177596092224121,
+            0.018774978816509247,
+            -0.0017568555194884539,
+            -0.011822959408164024,
+            0.02304270677268505,
+            -0.0025297212414443493,
+            -0.016151007264852524,
+            -0.009364869445562363,
+            0.015434691682457924,
+            0.03782140463590622,
+            0.0018954058177769184,
+            0.002769120968878269,
+            0.015080304816365242,
+            -0.008218766190111637,
+            0.024113407358527184,
+            -0.02103702537715435,
+            -0.03365924209356308,
+            -0.02340463362634182,
+            0.03854525834321976,
+            0.012901200912892818,
+            -0.008369569666683674,
+            0.011513812467455864,
+            0.004878478590399027,
+            0.03549903631210327,
+            0.0010339433792978525,
+            -0.04113907366991043,
+            0.006318647880107164,
+            0.007321488112211227,
+            -0.004026441369205713,
+            -0.026827862486243248,
+            0.0010782418539747596,
+            -0.02964787930250168,
+            0.017055824398994446,
+            -0.0014712722040712833,
+            0.015374370850622654,
+            0.02115766704082489,
+            0.009379949420690536,
+            0.00980219803750515,
+            0.006665494758635759,
+            -0.0008567498298361897,
+            0.009922840632498264,
+            0.008565613068640232,
+            -0.006737126037478447,
+            0.007087743375450373,
+            0.015713678672909737,
+            0.00824892707169056,
+            0.02364591881632805,
+            0.028215250000357628,
+            0.018729738891124725,
+            -0.014801319688558578,
+            -0.018729738891124725,
+            0.005511851515620947,
+            0.03236233443021774,
+            0.012757938355207443,
+            -0.001835084636695683,
+            -0.03209088742733002,
+            0.011574134230613708,
+            0.030567778274416924,
+            0.021519595757126808,
+            -0.004705055151134729,
+            -0.011566594243049622,
+            -0.024535655975341797,
+            -0.02720486931502819,
+            0.007381809409707785,
+            0.0069030094891786575,
+            -0.008218766190111637,
+            -0.019287709146738052,
+            -0.01149873249232769,
+            -0.019574236124753952,
+            0.03190992400050163,
+            0.021293390542268753,
+            -0.00450524128973484,
+            -0.009591073729097843,
+            0.0011281953193247318,
+            -0.004331817384809256,
+            0.013934201560914516,
+            -0.0003169220290146768,
+            -0.01229798886924982,
+            0.01160429511219263,
+            -0.0139869824051857,
+            -0.009379949420690536,
+            -0.01661849580705166,
+            -0.028577178716659546,
+            -0.004968960303813219,
+            0.016362130641937256,
+            0.008784277364611626,
+            0.011672155931591988,
+            -0.017749518156051636,
+            -0.002957624848932028,
+            0.014446931891143322,
+            0.0067333560436964035,
+            -0.010782417841255665,
+            0.0011187701020389795,
+            -0.0009953001281246543,
+            0.009470431134104729,
+            -0.028170010074973106,
+            0.012124565429985523,
+            -0.02782316319644451,
+            -0.0009792773053050041,
+            0.013572274707257748,
+            -0.0014703297056257725,
+            0.0007945435936562717,
+            -0.015608116053044796,
+            -0.010556213557720184,
+            -0.003159323940053582,
+            -0.008671175688505173,
+            -0.005210245493799448,
+            -0.02390228398144245,
+            0.01609068550169468,
+            -0.005990650970488787,
+            0.011106644757091999,
+            -0.038092851638793945,
+            0.001858647563494742,
+            0.010797498747706413,
+            -0.012237667106091976,
+            -0.024927744641900063,
+            0.003643778618425131,
+            0.040988270193338394,
+            0.0017691082321107388,
+            0.006876619067043066,
+            0.0011640110751613975,
+            0.0008100951090455055,
+            -0.021172748878598213,
+            -0.00022054946748539805,
+            -0.009455351158976555,
+            -0.008731496520340443,
+            -0.0077135758474469185,
+            0.011182046495378017,
+            0.006567472591996193,
+            -0.007393119391053915,
+            0.02299746498465538,
+            -0.0036494338419288397,
+            -0.002514640800654888,
+            0.002282781060785055,
+            0.023178428411483765,
+            -0.00608490314334631,
+            0.02503330633044243,
+            0.006869079079478979,
+            0.06514691561460495,
+            -0.0035947677679359913,
+            -0.005949180107563734,
+            0.0032592308707535267,
+            -0.015819240361452103,
+            0.0011008622823283076,
+            0.011875740252435207,
+            -0.022198209539055824,
+            0.015027523972094059,
+            0.0014175486285239458,
+            -0.010179205797612667,
+            -0.012199967168271542,
+            -0.002771005965769291,
+            0.012486492283642292,
+            0.022047406062483788,
+            0.021338632330298424,
+            0.001983060035854578,
+            0.01609068550169468,
+            0.03172896057367325,
+            0.013225427828729153,
+            -0.011378089897334576,
+            0.0013402621261775494,
+            -0.00019215607608202845,
+            0.017598716542124748,
+            -0.007819138467311859,
+            0.019694877788424492,
+            -0.00296705006621778,
+            0.011883280239999294,
+            0.010118884965777397,
+            0.025561116635799408,
+            0.01895594410598278,
+            0.0010179205564782023,
+            0.00916882511228323,
+            0.008301707915961742,
+            0.017734438180923462,
+            -0.001773820840753615,
+            0.011197126470506191,
+            -0.007280017249286175,
+            0.0012592055136337876,
+            0.019544076174497604,
+            0.02041873335838318,
+            -0.024520576000213623,
+            0.003958580084145069,
+            -0.008218766190111637,
+            -0.0024053086526691914,
+            -0.0005197992431931198,
+            0.0064166695810854435,
+            0.010103804059326649,
+            -0.0061942352913320065,
+            -0.02847161516547203,
+            -0.006122604012489319,
+            -0.004912409465759993,
+            0.005368588492274284,
+            -0.02263553813099861,
+            -0.013519493862986565,
+            0.00188126799184829,
+            0.01106894388794899,
+            -0.007114133797585964,
+            -0.0011592984665185213,
+            0.002567421877756715,
+            0.021534675732254982,
+            0.011528893373906612,
+            -0.0068426881916821,
+            -0.008799358271062374,
+            -0.001767223235219717,
+            0.006963330786675215,
+            0.022605376318097115,
+            0.002514640800654888,
+            0.012712697498500347,
+            0.014597735367715359,
+            0.006518461741507053,
+            -0.009658935479819775,
+            0.01446201279759407,
+            0.0005636264104396105,
+            0.001902946038171649,
+            0.0005537299439311028,
+            -0.0008228191290982068,
+            0.004784226883202791,
+            0.00927438773214817,
+            0.01972503960132599,
+            0.0071970755234360695,
+            0.00015186337986961007,
+            0.0028765681199729443,
+            0.014100085012614727,
+            -0.016105765476822853,
+            -0.0037926966324448586,
+            -0.0012771133333444595,
+            0.012056703679263592,
+            0.007042502518743277,
+            -0.0021489434875547886,
+            0.004561792127788067,
+            0.0123960105702281,
+            0.006503381300717592,
+            -0.008791817352175713,
+            -0.016407372429966927,
+            0.005643804091960192,
+            -0.007449670694768429,
+            -0.009206525981426239,
+            -0.007717345841228962,
+            0.009553372859954834,
+            0.005681504961103201,
+            0.010307388380169868,
+            -0.019694877788424492,
+            0.007528842426836491,
+            -0.0021998395677655935,
+            -0.009930380620062351,
+            -0.003935959655791521,
+            0.010737176984548569,
+            0.019860761240124702,
+            -0.0017945562722161412,
+            0.011709856800734997,
+            -0.01606052555143833,
+            0.0024222740903496742,
+            0.009093424305319786,
+            -0.0027766611892729998,
+            0.008859679102897644,
+            0.015668436884880066,
+            -0.008625934831798077,
+            -0.03483550623059273,
+            -0.007721116300672293,
+            0.015298969112336636,
+            -0.003477895399555564,
+            0.011219747364521027,
+            -0.01658833585679531,
+            0.013270668685436249,
+            -0.017251869663596153,
+            -0.018639257177710533,
+            -0.012999222613871098,
+            -0.01776459999382496,
+            -0.010631615296006203,
+            -0.028079528361558914,
+            -0.01190590113401413,
+            0.019106745719909668,
+            0.0027615807484835386,
+            0.004486390855163336,
+            -0.00036286984686739743,
+            7.49302635085769e-05,
+            0.0012884235475212336,
+            0.02067509852349758,
+            -0.010428030975162983,
+            0.0032139900140464306,
+            0.015879560261964798,
+            -0.025229349732398987,
+            -0.0054138293489813805,
+            0.011815418489277363,
+            -0.017251869663596153,
+            0.008618393912911415,
+            -0.024234050884842873,
+            0.002989670494571328,
+            -0.01562319602817297,
+            0.018201928585767746,
+            -0.020147288218140602,
+            -0.013730617240071297,
+            0.010020862333476543,
+            -0.015065224841237068,
+            -0.013142486102879047,
+            -0.024927744641900063,
+            -5.4754465963924304e-05,
+            -0.006032121833413839,
+            -0.0070274220779538155,
+            -0.020388571545481682,
+            -0.0058775488287210464,
+            -0.007721116300672293,
+            0.0042564161121845245,
+            -0.012418631464242935,
+            0.03447357565164566,
+            0.012705156579613686,
+            0.01695026271045208,
+            -0.00990776065737009,
+            -0.027144549414515495,
+            0.004328047391027212,
+            0.03190992400050163,
+            0.013632595539093018,
+            0.007404429838061333,
+            0.007811598014086485,
+            -0.0012884235475212336,
+            -0.004659814294427633,
+            0.02197200432419777,
+            0.00021937131532467902,
+            0.013949282467365265,
+            0.004331817384809256,
+            -0.015200947411358356,
+            -0.0028859933372586966,
+            0.021006865426898003,
+            0.021610077470541,
+            -0.0015174556756392121,
+            0.0006545794894918799,
+            -0.0049425698816776276,
+            0.01366275642067194,
+            -0.008935080841183662,
+            -0.009493052028119564,
+            0.007242316380143166,
+            -0.01752331480383873,
+            -0.01817176677286625,
+            -0.0024618597235530615,
+            0.017719358205795288,
+            0.004245105665177107,
+            -0.005394978914409876,
+            0.003647548845037818,
+            0.014756078831851482,
+            0.019227389246225357,
+            0.006326187867671251,
+            -0.02194184437394142,
+            -0.0033384026028215885,
+            0.018473373726010323,
+            -0.013006762601435184,
+            -0.010013322345912457,
+            0.01143841166049242,
+            0.026737380772829056,
+            -0.0024618597235530615,
+            -0.055525682866573334,
+            -0.0018360271351411939,
+            0.012554354034364223,
+            -0.009327168576419353,
+            -0.040505699813365936,
+            -0.004607032984495163,
+            2.5757279217941687e-05,
+            -0.023057786747813225,
+            0.029104989022016525,
+            -0.00974187720566988,
+            -0.008648554794490337,
+            0.015321590006351471,
+            0.01981552131474018,
+            -0.010345089249312878,
+            -0.0054590702056884766,
+            0.009877599775791168,
+            -0.015344209969043732,
+            -0.02263553813099861,
+            0.005168774630874395,
+            -0.023103026673197746,
+            -0.02438485249876976,
+            -0.009817278943955898,
+            0.005285646766424179,
+            0.01924246922135353,
+            0.013806018978357315,
+            0.01809636689722538,
+            0.019348030909895897,
+            0.0058775488287210464,
+            -0.011830499395728111,
+            -0.015713678672909737,
+            0.008754117414355278,
+            -0.014688217081129551,
+            -0.0007342223543673754,
+            -0.001240355079062283,
+            0.008527912199497223,
+            0.006213085725903511,
+            0.013165106065571308,
+            -0.015834320336580276,
+            -0.020840981975197792,
+            -2.8039941753377207e-05,
+            -0.004177244380116463,
+            0.007272477261722088,
+            -0.015879560261964798,
+            -0.013549653813242912,
+            0.006522231735289097,
+            0.0022054945584386587,
+            -0.010081184096634388,
+            -0.0009868174092844129,
+            0.02964787930250168,
+            -0.015608116053044796,
+            -0.005820997525006533,
+            -0.025500794872641563,
+            0.0007535439799539745,
+            -0.011981301940977573,
+            0.0239324439316988,
+            0.009183906018733978,
+            -0.023510195314884186,
+            0.009515671990811825,
+            -0.027355672791600227,
+            0.017779679968953133,
+            -0.006887929514050484,
+            0.0034062638878822327,
+            0.01891070231795311,
+            0.008218766190111637,
+            0.02177596092224121,
+            0.011551513336598873,
+            -0.0007243258878588676,
+            -0.0010066103423014283,
+            0.03390052542090416,
+            -0.009930380620062351,
+            0.016347050666809082,
+            -0.0027201101183891296,
+            -0.008754117414355278,
+            0.010216906666755676,
+            -0.004079222679138184,
+            -0.007404429838061333,
+            0.011325309053063393,
+            0.01785508170723915,
+            0.002763465978205204,
+            0.007457210682332516,
+            -0.021987084299325943,
+            -0.005538241937756538,
+            -0.010058563202619553,
+            0.009078343398869038,
+            -0.000991530017927289,
+            0.008060422725975513,
+            0.0034703551791608334,
+            -0.020388571545481682,
+            -0.01833765022456646,
+            0.0028181320521980524,
+            -0.015608116053044796,
+            -0.006009501405060291,
+            -0.0075891632586717606,
+            -0.0065561626106500626,
+            0.015298969112336636,
+            -0.017327269539237022,
+            -0.004248876124620438,
+            0.0068502286449074745,
+            0.016724057495594025,
+            -0.006126374006271362,
+            -0.016151007264852524,
+            0.019830601289868355,
+            0.013798478990793228,
+            0.0322718508541584,
+            0.002786086406558752,
+            0.01589464209973812,
+            -0.01711614616215229,
+            -0.014914421364665031,
+            -0.024897582828998566,
+            0.018699578940868378,
+            -0.006458140444010496,
+            0.014507253654301167,
+            -0.01654309406876564,
+            -0.0030462215654551983,
+            -0.018654337152838707,
+            0.0002771005965769291,
+            -0.0007808770169503987,
+            -0.020162368193268776,
+            0.011958681978285313,
+            -0.0016795690171420574,
+            0.023419713601469994,
+            -0.01621132716536522,
+            -0.014514793641865253,
+            0.002884108340367675,
+            -0.021459273993968964,
+            0.008339408785104752,
+            -0.0031875993590801954,
+            -0.006597633473575115,
+            0.0005037764203734696,
+            0.00910850428044796,
+            -0.006642874330282211,
+            -0.03803252801299095,
+            -0.007076432928442955,
+            -0.004000050947070122,
+            0.006963330786675215,
+            -0.006243246141821146,
+            -0.01756855472922325,
+            -0.018081285059452057,
+            0.013489332981407642,
+            0.02646593563258648,
+            -0.005741826258599758,
+            0.007246086373925209,
+            0.01026968751102686,
+            0.01964963786303997,
+            -0.01654309406876564,
+            0.019544076174497604,
+            0.0015721217496320605,
+            0.00870133563876152,
+            0.003347827820107341,
+            0.008482671342790127,
+            0.013730617240071297,
+            0.0023826882243156433,
+            -0.00746475113555789,
+            -0.00976449716836214,
+            -0.006759746931493282,
+            0.006133913993835449,
+            0.016648657619953156,
+            -0.002486365381628275,
+            -0.01478623878210783,
+            0.00413577351719141,
+            -0.003489205613732338,
+            -0.02168547920882702,
+            -0.002601352520287037,
+            -0.012855960056185722,
+            -0.014529873616993427,
+            0.015909722074866295,
+            -0.013406391255557537,
+            -0.017749518156051636,
+            -0.00034331256756559014,
+            0.007117903791368008,
+            -0.006303567439317703,
+            -0.03103526681661606,
+            0.005839847959578037,
+            -0.007758816704154015,
+            0.030416974797844887,
+            -0.0065109217539429665,
+            0.008188605308532715,
+            0.005655114538967609,
+            0.011121724732220173,
+            0.0267977025359869,
+            -0.0002858189109247178,
+            0.015819240361452103,
+            0.008098123595118523,
+            -0.006936940364539623,
+            -0.003790811635553837,
+            0.026435773819684982,
+            -0.01505014393478632,
+            0.02124815061688423,
+            0.003585342550650239,
+            -0.012117025442421436,
+            0.017794759944081306,
+            -0.0037418007850646973,
+            0.014688217081129551,
+            0.0007851183763705194,
+            0.0030160609167069197,
+            0.00608490314334631,
+            0.0017625106265768409,
+            -0.022560136392712593,
+            -0.010850279591977596,
+            0.0024392392951995134,
+            -0.011770177632570267,
+            0.023660998791456223,
+            0.015434691682457924,
+            -0.007506221998482943,
+            0.016331970691680908,
+            -0.027838243171572685,
+            0.009206525981426239,
+            -0.013127405196428299,
+            0.011076483875513077,
+            0.0031875993590801954,
+            -0.004007590934634209,
+            -0.007596703711897135,
+            -0.013376230373978615,
+            -0.006835148204118013,
+            -0.00609621312469244,
+            0.013406391255557537,
+            0.03317667171359062,
+            0.008301707915961742,
+            -0.0060698227025568485,
+            -0.010691936127841473,
+            -0.003943499643355608,
+            0.004497700836509466,
+            0.011657075956463814,
+            0.0052064755000174046,
+            -0.00963631458580494,
+            -0.005745596252381802,
+            -0.007012341637164354,
+            -0.013760778121650219,
+            -0.007170685101300478,
+            0.009153745137155056,
+            0.004181014373898506,
+            0.006397819612175226,
+            0.0053836689330637455,
+            -0.0025485714431852102,
+            -0.0010848394595086575,
+            0.010714557021856308,
+            -0.004644733853638172,
+            -0.004614573437720537,
+            0.006820067763328552,
+            0.026164328679442406,
+            -0.0013110439758747816,
+            0.01456003449857235,
+            -0.0032875065226107836,
+            -0.002005680464208126,
+            -0.001410951022990048,
+            -0.00970417633652687,
+            -0.017553474754095078,
+            0.002520296024158597,
+            0.048015691339969635,
+            -0.01472591795027256,
+            -0.014379071071743965,
+            -0.004859628155827522,
+            0.03026617132127285,
+            -0.0014118935214355588,
+            -0.006024581845849752,
+            0.01764395646750927,
+            -0.007939780130982399,
+            0.014167946763336658,
+            0.008995401673018932,
+            -0.026043687015771866,
+            0.0014345140662044287,
+            -0.00726116681471467,
+            -0.0006875676335766912,
+            0.011649535968899727,
+            0.025108708068728447,
+            0.012184886261820793,
+            0.00275215576402843,
+            0.002882223343476653,
+            0.020479053258895874,
+            0.015200947411358356,
+            -0.00540251936763525,
+            0.002458089729771018,
+            0.0017087870510295033,
+            0.005274336785078049,
+            -0.009447811171412468,
+            -0.011219747364521027,
+            -0.017387591302394867,
+            0.00014950709010008723,
+            -0.002073541982099414,
+            0.0005749366246163845,
+            -0.022439494729042053,
+            -0.00841481052339077,
+            0.013466712087392807,
+            -0.01262221485376358,
+            -0.00963631458580494,
+            0.002147058490663767,
+            -0.0011687236838042736,
+            -0.005240405909717083,
+            0.0055307019501924515,
+            0.02405308559536934,
+            -0.007442130707204342,
+            -0.014967203140258789,
+            0.003630583407357335,
+            0.01100862305611372,
+            0.013994523324072361,
+            -0.0005881318938918412,
+            0.008399729616940022,
+            -0.005251716356724501,
+            0.011589214205741882,
+            0.010737176984548569,
+            -0.005304497200995684,
+            -0.010593914426863194,
+            -0.007178225088864565,
+            0.00746475113555789,
+            -0.015427151694893837,
+            -0.004316737409681082,
+            -2.169266554119531e-05,
+            -0.005319577641785145,
+            0.014997363090515137,
+            0.021293390542268753,
+            -0.00672958604991436,
+            0.020192528143525124,
+            -0.006284717004746199,
+            0.00540251936763525,
+            -0.0034929756075143814,
+            -0.0011517582461237907,
+            -0.008980321697890759,
+            -0.03540855646133423,
+            -0.011951141990721226,
+            0.027355672791600227,
+            -0.01039787009358406,
+            -0.008256467059254646,
+            0.023314151912927628,
+            0.003951040096580982,
+            0.009417650289833546,
+            -0.008633474819362164,
+            0.009515671990811825,
+            -0.009342248551547527,
+            0.007581623271107674,
+            0.004316737409681082,
+            0.0009962426265701652,
+            -0.004361978266388178,
+            -0.024731699377298355,
+            -0.005809687543660402,
+            -0.006220625713467598,
+            -0.02450549602508545,
+            -0.006872849073261023,
+            0.004120693542063236,
+            -0.008927540853619576,
+            -0.022695859894156456,
+            0.0007804057677276433,
+            -0.005949180107563734,
+            0.0007690955535508692,
+            -0.020433813333511353,
+            0.01039787009358406,
+            0.020569536834955215,
+            0.0346243791282177,
+            -0.03230201452970505,
+            0.00651469174772501,
+            -0.0012752283364534378,
+            0.010586374439299107,
+            -0.026888184249401093,
+            -0.0009274387848563492,
+            0.008060422725975513,
+            0.007442130707204342,
+            0.001478812424466014,
+            -0.0026579038240015507,
+            -0.003954810090363026,
+            0.025877803564071655,
+            -0.002356297569349408,
+            0.0025693068746477365,
+            -0.008837059140205383,
+            -0.013594894669950008,
+            -0.02405308559536934,
+            0.012486492283642292,
+            0.012350769713521004,
+            0.01299168262630701,
+            -0.010940761305391788,
+            -0.0008407270070165396,
+            -0.006032121833413839,
+            -0.003853017929941416,
+            0.007879459299147129,
+            0.006612713914364576,
+            -0.01202654279768467,
+            0.023057786747813225,
+            -0.011815418489277363,
+            -0.0030462215654551983,
+            0.0029972107149660587,
+            -0.0039057990070432425,
+            0.001874670386314392,
+            0.011747557669878006,
+            -0.011393170803785324,
+            -0.018684497103095055,
+            -0.009357329457998276,
+            0.013248047791421413,
+            0.010827658697962761,
+            -0.00870133563876152,
+            -0.034292612224817276,
+            -0.02156483568251133,
+            0.03172896057367325,
+            -0.01345917209982872,
+            0.010654235258698463,
+            -0.00619800528511405,
+            0.011453491635620594,
+            0.012343229725956917,
+            0.0015655241440981627,
+            -0.0034194590989500284,
+            -0.01063915528357029,
+            -0.011385629884898663,
+            -0.00843743048608303,
+            -0.0003246978158131242,
+            -0.0029764752835035324,
+            -0.014612815342843533,
+            -0.005926559679210186,
+            0.017085986211895943,
+            -0.0015589265385642648,
+            0.007668334990739822,
+            0.0027201101183891296,
+            0.013134945183992386,
+            0.006940710358321667,
+            7.58727837819606e-05,
+            0.01802096515893936,
+            -0.014363990165293217,
+            0.012750397436320782,
+            0.041651804000139236,
+            -0.0020999324042350054,
+            -0.006669264752417803,
+            -0.019227389246225357,
+            -0.007083973381668329,
+            -0.029104989022016525,
+            0.005357278510928154,
+            -0.016362130641937256,
+            -0.026375453919172287,
+            0.011641995050013065,
+            -0.016271648928523064,
+            -0.012117025442421436,
+            -0.023555437102913857,
+            0.008165985345840454,
+            -0.0038944887928664684,
+            0.011151885613799095,
+            0.0022752410732209682,
+            0.01216980628669262,
+            -0.009146205149590969,
+            -0.020494135096669197,
+            0.005353508051484823,
+            0.013798478990793228,
+            -0.002884108340367675,
+            -0.02515394799411297,
+            -0.012908740900456905,
+            0.005296957213431597,
+            -0.017719358205795288,
+            -0.010111344046890736,
+            -0.03966120257973671,
+            -0.017659036442637444,
+            0.0207505002617836,
+            0.010593914426863194,
+            -0.007909620180726051,
+            0.005545781925320625,
+            -0.0012167921522632241,
+            0.004591953009366989,
+            0.0030236011371016502,
+            0.0038096620701253414,
+            0.013180186972022057,
+            -0.002090507186949253,
+            0.004354438278824091,
+            0.00976449716836214,
+            0.013617515563964844,
+            0.006590093486011028,
+            -0.01349687296897173,
+            -0.019785359501838684,
+            0.011657075956463814,
+            -0.005063212476670742,
+            0.005342198070138693,
+            -0.0019161412492394447,
+            -0.00466735428199172,
+            -0.012674996629357338,
+            0.010111344046890736,
+            0.01308970432728529,
+            0.016030363738536835,
+            0.017779679968953133,
+            0.014115165919065475,
+            0.005919019691646099,
+            -0.015427151694893837,
+            0.001035828492604196,
+            -7.470199761883123e-06,
+            0.004018901381641626,
+            -0.007083973381668329,
+            0.006224395707249641,
+            0.003477895399555564,
+            0.010201826691627502,
+            0.011355469934642315,
+            -0.0016701437998563051,
+            0.012750397436320782,
+            -0.007332798093557358,
+            -0.003246035659685731,
+            0.0022563906386494637,
+            0.003076382214203477,
+            -0.009319628588855267,
+            0.009568453766405582,
+            0.0017172697698697448,
+            -0.004780456889420748,
+            0.010578833520412445,
+            -0.02450549602508545,
+            -0.008181065320968628,
+            -0.01276547834277153,
+            0.0011866315035149455,
+            -0.023480035364627838,
+            0.01262221485376358,
+            0.03377988189458847,
+            0.0009773923084139824,
+            -0.00225262064486742,
+            -0.00038784657954238355,
+            0.010744716972112656,
+            -0.01666373759508133,
+            -0.009493052028119564,
+            -0.0007412912673316896,
+            -0.01862417720258236,
+            -0.024580897763371468,
+            0.01833765022456646,
+            0.017839999869465828,
+            -0.013308369554579258,
+            -0.016241488978266716,
+            -0.002853947691619396,
+            -0.01833765022456646,
+            -0.010020862333476543,
+            -0.0019180262461304665,
+            -0.012984142638742924,
+            -0.019574236124753952,
+            -0.002957624848932028,
+            -0.0291200689971447,
+            0.0021150128450244665,
+            0.010096264071762562,
+            0.011385629884898663,
+            -0.010744716972112656,
+            0.0062394761480391026,
+            -0.01190590113401413,
+            -0.0033685630187392235,
+            0.006593863479793072,
+            0.01256189402192831,
+            0.03390052542090416,
+            -0.006789907347410917,
+            -0.010737176984548569,
+            0.01525372825562954,
+            0.007634404581040144,
+            0.002652248600497842,
+            0.018729738891124725,
+            0.0013921007048338652,
+            0.010616534389555454,
+            0.0018614751752465963,
+            0.015412071719765663,
+            0.01349687296897173,
+            -0.008935080841183662,
+            0.017734438180923462,
+            0.0116646159440279,
+            -0.0014986053574830294,
+            -0.019393272697925568,
+            0.014477092772722244,
+            0.010330009274184704,
+            0.013014303520321846,
+            0.0009905875194817781,
+            0.014914421364665031,
+            -0.012463872320950031,
+            -0.01768919825553894,
+            -0.01431874930858612,
+            0.00016482302453368902,
+            -0.006929399911314249,
+            0.010111344046890736,
+            -0.0006734298658557236,
+            -0.02405308559536934,
+            -0.0012714582262560725,
+            0.0035928827710449696,
+            -0.002141403267160058,
+            0.0018944633193314075,
+            -0.005342198070138693,
+            0.011973761953413486,
+            0.013263127766549587,
+            0.007871919311583042,
+            0.012750397436320782,
+            -0.008927540853619576,
+            0.009447811171412468,
+            -0.004312967415899038,
+            0.012840880081057549,
+            -0.0025108708068728447,
+            -0.0020999324042350054,
+            -3.84960112569388e-05,
+            0.0003195139579474926,
+            0.007110363803803921,
+            0.00963631458580494,
+            0.008874759078025818,
+            0.026088926941156387,
+            -0.012780558317899704,
+            0.012584514915943146,
+            0.009530752897262573,
+            -0.013700457289814949,
+            0.03133687376976013,
+            0.017025664448738098,
+            0.011920981109142303,
+            -0.017101066187024117,
+            0.010518512688577175,
+            0.023781640455126762,
+            0.024188809096813202,
+            -0.00870133563876152,
+            -0.005964260548353195,
+            -0.0003664042742457241,
+            0.022967305034399033,
+            0.015374370850622654,
+            -0.017508234828710556,
+            0.01519340742379427,
+            0.011174505576491356,
+            0.009183906018733978,
+            -0.015057684853672981,
+            0.040777143090963364,
+            -0.008686255663633347,
+            -0.017387591302394867,
+            -0.011687235906720161,
+            -0.014160406775772572,
+            -0.016392290592193604,
+            -0.005176314618438482,
+            -0.006537312176078558,
+            -0.009953001514077187,
+            0.0018482798477634788,
+            0.0016239603282883763,
+            -0.031216232106089592,
+            -0.003966120071709156,
+            -0.006314877886325121,
+            -0.013655216433107853,
+            -0.015065224841237068,
+            0.008482671342790127,
+            -0.010812578722834587,
+            0.0030575317796319723,
+            0.004482620861381292,
+            -0.020690178498625755,
+            0.01744791306555271,
+            0.023148268461227417,
+            0.0033968386705964804,
+            0.011249907314777374,
+            -0.00466735428199172,
+            0.0058775488287210464,
+            -0.0025448014494031668,
+            0.025395233184099197,
+            -0.007479831110686064,
+            0.010345089249312878,
+            -0.002599467523396015,
+            -0.01519340742379427,
+            -0.04545203968882561,
+            -0.0024147338699549437,
+            0.006865308620035648,
+            0.0181416068226099,
+            -0.01356473471969366,
+            0.007649484556168318,
+            0.013519493862986565,
+            -0.013338529504835606,
+            0.0050368220545351505,
+            0.023314151912927628,
+            -0.006152764428406954,
+            0.013044463470578194,
+            0.00953829288482666,
+            0.02304270677268505,
+            -0.011626915074884892,
+            -0.02414356917142868,
+            0.006013271398842335,
+            0.013707997277379036,
+            -0.005255486350506544,
+            0.017146306112408638,
+            0.012207507155835629,
+            -0.0139869824051857,
+            -0.004478850401937962,
+            -0.014514793641865253,
+            -0.010345089249312878,
+            0.012848420068621635,
+            -0.0047502960078418255,
+            0.018036045134067535,
+            -0.002073541982099414,
+            -0.004825697746127844,
+            -0.016724057495594025,
+            0.011792798526585102,
+            0.007849298417568207,
+            -0.007212155964225531,
+            -0.004037751816213131,
+            -0.02124815061688423,
+            -0.021519595757126808,
+            0.003251690650358796,
+            -0.024897582828998566,
+            0.02622465044260025,
+            -0.0005909594474360347,
+            0.03218137100338936,
+            0.000863818742800504,
+            -0.002763465978205204,
+            0.019378192722797394,
+            0.01707090437412262,
+            -0.013383770361542702,
+            0.0038794083520770073,
+            0.009862519800662994,
+            0.0032931615132838488,
+            -0.0008280029869638383,
+            -0.00904818344861269,
+            -0.0006696598138660192,
+            -0.0007083030650392175,
+            0.009236686863005161,
+            0.0010527938138693571,
+            0.0009330938919447362,
+            -0.008181065320968628,
+            -0.024882502853870392,
+            -0.006597633473575115,
+            0.010767337866127491,
+            0.005734285805374384,
+            -0.006337498314678669,
+            -0.012878580018877983,
+            0.007415739819407463,
+            -0.006529772188514471,
+            -0.01084273960441351,
+            -0.00873903650790453,
+            0.009779578074812889,
+            0.0020660017617046833,
+            -0.01026968751102686,
+            0.021006865426898003,
+            0.0021432882640510798,
+            -0.006910549942404032,
+            0.014379071071743965,
+            0.001902946038171649,
+            -0.017387591302394867,
+            -0.00996808148920536,
+            0.009847438894212246,
+            -0.01000578235834837,
+            0.002179104136303067,
+            0.010058563202619553,
+            -0.0009604269289411604,
+            0.0136853763833642,
+            0.003257345873862505,
+            -0.011958681978285313,
+            -0.012690076604485512,
+            -0.0282604917883873,
+            -0.00847513135522604,
+            -0.028245411813259125,
+            0.012509113177657127,
+            0.0029764752835035324,
+            0.0017945562722161412,
+            0.013504412956535816,
+            0.0018209468107670546,
+            -0.0207505002617836,
+            0.017779679968953133,
+            -0.01768919825553894,
+            0.011257447302341461,
+            0.004716365598142147,
+            0.007962401024997234,
+            -0.006797447334975004,
+            0.011302688159048557,
+            0.009787118062376976,
+            -0.02185136266052723,
+            0.018156686797738075,
+            0.02173071913421154,
+            -0.02070525847375393,
+            0.010201826691627502,
+            0.009900220669806004,
+            0.007864379324018955,
+            -0.0006555219879373908,
+            -0.013971902430057526,
+            -0.008505292236804962,
+            -0.017960643395781517,
+            0.003438309533521533,
+            0.003342172596603632,
+            -0.006360118743032217,
+            -0.005455300211906433,
+            -0.015027523972094059,
+            0.011038783006370068,
+            0.00530072720721364,
+            -0.021640237420797348,
+            -0.017266949638724327,
+            -0.0038605581503361464,
+            -0.01924246922135353,
+            0.004803077317774296,
+            0.027551718056201935,
+            -0.008346948772668839,
+            -0.015268809162080288,
+            0.024611057713627815,
+            0.02630005218088627,
+            0.009440271183848381,
+            -0.010986002162098885,
+            0.0015410187188535929,
+            -0.004855858162045479,
+            -0.011340389028191566,
+            0.005312037654221058,
+            -0.004723905585706234,
+            -0.010035943239927292,
+            0.008542993105947971,
+            -0.003732375567778945,
+            0.00858069397509098,
+            0.022349011152982712,
+            -0.006642874330282211,
+            0.004459999967366457,
+            0.009123584255576134,
+            -0.0012422400759533048,
+            0.01339885126799345,
+            -0.02577224187552929,
+            0.0108050387352705,
+            -0.025455554947257042,
+            -0.002605122746899724,
+            0.01086535956710577,
+            0.01919722743332386,
+            0.025470634922385216,
+            -0.00963631458580494,
+            0.0008741864003241062,
+            -0.0009552430710755289,
+            0.027958884835243225,
+            0.021142587065696716,
+            -0.00422625569626689,
+            -0.01678437925875187,
+            -0.014017143286764622,
+            -0.004248876124620438,
+            -0.002265815855935216,
+            0.016648657619953156,
+            -0.024203889071941376,
+            0.0044336095452308655,
+            -0.018473373726010323,
+            -0.002045266330242157,
+            -0.0013402621261775494,
+            0.014115165919065475,
+            0.015261268243193626,
+            0.002590042306110263,
+            -0.0019566696137189865,
+            0.019875841215252876,
+            0.005647574085742235,
+            -0.008196145296096802,
+            0.0042564161121845245,
+            0.005805917549878359,
+            -0.001035828492604196,
+            0.007698495872318745,
+            -0.05235881730914116,
+            0.008203686214983463,
+            0.01780983991920948,
+            0.010292308405041695,
+            0.005168774630874395,
+            0.001149873249232769,
+            -0.005640034098178148,
+            -0.015366830863058567,
+            -0.010887980461120605,
+            0.017749518156051636,
+            -0.015472392551600933,
+            0.0064128995873034,
+            0.00392087921500206,
+            -0.007826678454875946,
+            0.021293390542268753,
+            0.008075503632426262,
+            0.015035063959658146,
+            0.009651395492255688,
+            0.03047729656100273,
+            0.011423330754041672,
+            0.010827658697962761,
+            -0.005624953657388687,
+            0.00304810656234622,
+            0.0010999197838827968,
+            0.005621183663606644,
+            0.0196043960750103,
+            -0.019634557887911797,
+            -0.004821927286684513,
+            0.006066052708774805,
+            -0.0279739648103714,
+            -0.005851158406585455,
+            0.012320608831942081,
+            -0.010081184096634388,
+            -0.006722046062350273,
+            0.007121674250811338,
+            -0.0014543068828061223,
+            0.0004229554324410856,
+            0.01572875864803791,
+            0.0022073795553296804,
+            -0.013285748660564423,
+            0.01568351686000824,
+            -0.002940659411251545,
+            -0.03136703372001648,
+            -0.011740017682313919,
+            -0.011845579370856285,
+            -0.007155604660511017,
+            -0.005191395059227943,
+            -0.03091462515294552,
+            0.0008732439018785954,
+            -0.011280068196356297,
+            0.016799459233880043,
+            -0.0010509087005630136,
+            0.006853998638689518,
+            0.006752206478267908,
+            -0.010797498747706413,
+            -0.011657075956463814,
+            -0.01797572337090969,
+            -0.0011395055335015059,
+            -0.01372307725250721,
+            -0.0075891632586717606,
+            0.01752331480383873,
+            -0.021217988803982735,
+            -0.007321488112211227,
+            0.025546036660671234,
+            -0.011310229077935219,
+            -0.0027653509750962257,
+            0.006691885180771351,
+            0.0026484786067157984,
+            0.0027069146744906902,
+            0.002339332364499569,
+            0.009033102542161942,
+            -0.0008756002062000334,
+            0.002906728768721223,
+            -0.02005680650472641,
+            0.03287506476044655,
+            0.011355469934642315,
+            -0.0011395055335015059,
+            0.010691936127841473,
+            -0.02687310427427292,
+            -0.025425394997000694,
+            0.019257549196481705,
+            -0.0311257503926754,
+            -0.006111293565481901,
+            -0.01250157319009304,
+            -0.0009213123703375459,
+            -0.03218137100338936,
+            -0.025108708068728447,
+            -0.029195470735430717,
+            -0.040746983140707016,
+            0.008482671342790127,
+            -0.00225262064486742,
+            0.005519391503185034,
+            0.007773897144943476,
+            -0.025636518374085426,
+            0.013021843507885933,
+            -0.008625934831798077,
+            -0.0013242393033578992,
+            0.018986104056239128,
+            -0.007434590253978968,
+            0.004486390855163336,
+            0.017025664448738098,
+            -0.0005692815175279975,
+            -0.006620253901928663,
+            -0.014145325869321823,
+            -0.006808757781982422,
+            0.026767540723085403,
+            0.025455554947257042,
+            0.022484734654426575,
+            -0.003114083083346486,
+            0.00249202037230134,
+            -0.003898258786648512,
+            0.019619476050138474,
+            -0.018066205084323883,
+            -0.008331868797540665,
+            0.003496745601296425,
+            0.007204615511000156,
+            0.008897379972040653,
+            0.004241335671395063,
+            0.03655466064810753,
+            0.020644936710596085,
+            0.015140626579523087,
+            -0.011740017682313919,
+            -0.013308369554579258,
+            -0.01103124301880598,
+            -0.013738158158957958,
+            0.02421896904706955,
+            0.024490416049957275,
+            -0.006925629917532206,
+            -0.02304270677268505,
+            -0.011461031623184681,
+            -0.00904818344861269,
+            -0.02053937502205372,
+            0.004901099018752575,
+            0.02254505641758442,
+            0.009990702383220196,
+            0.014326289296150208,
+            -0.010608994401991367,
+            0.011468571610748768,
+            0.03299570828676224,
+            -0.012124565429985523,
+            0.006118833553045988,
+            -0.0026465936098247766,
+            -0.016362130641937256,
+            0.012742857448756695,
+            -0.015208487398922443,
+            -0.019016264006495476,
+            -0.00466735428199172,
+            -0.021881522610783577,
+            0.018126526847481728,
+            0.00035768598900176585,
+            0.003087692428380251,
+            -0.012818259187042713,
+            -0.02443009428679943,
+            0.0060170418582856655,
+            0.00046089180978015065,
+            0.003898258786648512,
+            -0.00619800528511405,
+            0.00571920583024621,
+            -0.01711614616215229,
+            0.003393068676814437,
+            -0.017719358205795288,
+            0.0219267625361681,
+            -0.010043483227491379,
+            -0.011936061084270477,
+            0.004886019043624401,
+            0.008098123595118523,
+            0.009666475467383862,
+            0.0017879586666822433,
+            -0.0028690281324088573,
+            0.007562772836536169,
+            -0.017327269539237022,
+            0.017085986211895943,
+            0.01006610319018364,
+            0.011400710791349411,
+            0.003739915555343032,
+            0.002853947691619396,
+            -0.012546814046800137,
+            0.02801920659840107,
+            -0.0008986919419839978,
+            -0.008135824464261532,
+            0.00841481052339077,
+            -0.002000025473535061,
+            0.011958681978285313,
+            0.013263127766549587,
+            -0.00798502191901207,
+            0.0006074535194784403,
+            0.003754995996132493,
+            0.008369569666683674,
+            0.015321590006351471,
+            -0.0040339813567698,
+            0.020267929881811142,
+            0.0063337283208966255,
+            -0.00022243449348025024,
+            0.012275367975234985,
+            0.0037022149190306664,
+            0.015019983984529972,
+            -0.023962603881955147,
+            -0.009206525981426239,
+            -0.023721320554614067,
+            -0.0016739139100536704,
+            -0.00741196982562542,
+            0.008754117414355278,
+            -0.0029727050568908453,
+            -0.011521353386342525,
+            0.005455300211906433,
+            0.025907963514328003,
+            0.023299071937799454,
+            -0.004267726093530655,
+            0.016648657619953156,
+            -0.006744666490703821,
+            -0.023389553651213646,
+            -0.010707016102969646,
+            -0.006624023895710707,
+            -0.03145751729607582,
+            0.022514894604682922,
+            -0.007004801649600267,
+            0.005195165053009987,
+            0.02242441289126873,
+            -0.007238546386361122,
+            -0.0033384026028215885,
+            0.01202654279768467,
+            -0.004316737409681082,
+            -0.026330212131142616,
+            0.00830924790352583,
+            -0.013285748660564423,
+            0.014575114473700523,
+            0.011996382847428322,
+            0.031638480722904205,
+            -0.013225427828729153,
+            -0.002260160632431507,
+            0.009410110302269459,
+            0.012388470582664013,
+            0.020267929881811142,
+            -0.022695859894156456,
+            -0.003210220020264387,
+            -0.001869957777671516,
+            -0.0014656170969828963,
+            -0.0029255792032927275,
+            -0.004490160848945379,
+            0.008784277364611626,
+            -0.01972503960132599,
+            -0.002989670494571328,
+            -0.005082062911242247,
+            0.003183829365298152,
+            0.003858672920614481,
+            0.0026805242523550987,
+            0.002616432961076498,
+            -0.012516653165221214,
+            -0.008037802763283253,
+            -0.006526002194732428,
+            -0.015819240361452103,
+            0.012388470582664013,
+            0.000850152166094631,
+            -0.01597004383802414,
+            0.00847513135522604,
+            0.005673964973539114,
+            0.0030914624221622944,
+            0.003460929961875081,
+            0.0034703551791608334,
+            0.05558600276708603,
+            -0.008814438246190548,
+            -0.007140524219721556,
+            -0.019348030909895897,
+            0.011166965588927269,
+            -0.013904041610658169,
+            -0.0017417751951143146,
+            0.0007804057677276433,
+            0.02194184437394142,
+            -0.028969265520572662,
+            -0.018307490274310112,
+            0.010782417841255665,
+            -0.011423330754041672,
+            -0.01127252820879221,
+            -0.0005843618419021368,
+            0.0030613020062446594,
+            -0.03172896057367325,
+            0.003935959655791521,
+            0.005572172813117504,
+            -0.004154623951762915,
+            0.01462035533040762,
+            -0.004637193866074085,
+            -0.005481690634042025,
+            0.02230377122759819,
+            0.015057684853672981,
+            -0.006235706154257059,
+            0.002520296024158597,
+            -0.00810566358268261,
+            0.014273508451879025,
+            -0.024611057713627815,
+            -0.013557193800807,
+            0.002192299347370863,
+            0.0026258581783622503,
+            -0.017221707850694656,
+            -0.007404429838061333,
+            0.0009679670911282301,
+            -0.003574032336473465,
+            0.017598716542124748,
+            -0.004659814294427633,
+            0.015223568305373192,
+            -0.0119662219658494,
+            0.03305602818727493,
+            -0.004128233529627323,
+            -0.023253830149769783,
+            0.006390279158949852,
+            0.0029990957118570805,
+            0.011363009922206402,
+            0.0004262542352080345,
+            0.01833765022456646,
+            0.015268809162080288,
+            0.0019161412492394447,
+            0.006872849073261023,
+            0.005293187219649553,
+            -0.01744791306555271,
+            0.011920981109142303,
+            -0.008158445358276367,
+            0.008791817352175713,
+            -0.0108050387352705,
+            0.0030839224345982075,
+            -0.008316787891089916,
+            0.009093424305319786,
+            -0.004595723003149033,
+            0.0025221810210496187,
+            -0.0001197588280774653,
+            -0.001998140476644039,
+            -0.0016692013014107943,
+            -0.017659036442637444,
+            -0.010186745785176754,
+            0.02058461681008339,
+            -0.02115766704082489,
+            -0.007132984232157469,
+            0.00661648390814662,
+            -0.025048386305570602,
+            -0.010887980461120605,
+            0.017327269539237022,
+            -0.015140626579523087,
+            -0.007276247255504131,
+            0.011709856800734997,
+            0.0024844801519066095,
+            -0.01323296781629324,
+            0.014205647632479668,
+            0.002084852196276188,
+            -0.036946747452020645,
+            0.011385629884898663,
+            -0.003988740500062704,
+            0.00010674028453649953,
+            -0.01054113358259201,
+            -0.047653764486312866,
+            0.006861538626253605,
+            0.0006300740060396492,
+            -0.013994523324072361,
+            0.004784226883202791,
+            0.009116044268012047,
+            0.0155628751963377,
+            -0.0032875065226107836,
+            0.009425190277397633,
+            0.014605275355279446,
+            0.0007756931590847671,
+            -3.722950350493193e-05,
+            0.019875841215252876,
+            0.0010103804524987936,
+            0.022273611277341843,
+            -0.011672155931591988,
+            -0.005168774630874395,
+            -0.023434793576598167,
+            0.010051023215055466,
+            -0.0020584615413099527,
+            -0.0005358220660127699,
+            -0.019167067483067513,
+            -0.0032139900140464306,
+            -0.015577955171465874,
+            0.02859225869178772,
+            -0.003751225769519806,
+            -0.0033685630187392235,
+            0.013082164339721203,
+            -0.003721065353602171,
+            -0.007238546386361122,
+            0.014431851916015148,
+            0.030703501775860786,
+            0.00422625569626689,
+            -0.003709755139425397,
+            0.007325258105993271,
+            -0.006469450891017914,
+            -0.012886120937764645,
+            0.0010895520681515336,
+            -0.0008284742361865938,
+            -0.01086535956710577,
+            -0.0039057990070432425,
+            0.0011451606405898929,
+            -0.011755097657442093,
+            -0.01515570655465126,
+            -0.014348910190165043,
+            0.016241488978266716,
+            -0.003223415231332183,
+            -0.00413577351719141,
+            -0.021187828853726387,
+            0.0029104987625032663,
+            -0.015065224841237068,
+            -0.004874708596616983,
+            0.014627896249294281,
+            -0.011898360215127468,
+            0.03377988189458847,
+            -0.006631563883274794,
+            0.0239324439316988,
+            -0.004297886975109577,
+            -0.0011366780381649733,
+            0.0004455758898984641,
+            -0.004482620861381292,
+            -0.014884261414408684,
+            -0.005968030542135239,
+            0.008565613068640232,
+            0.0028916485607624054,
+            0.007102823816239834,
+            0.01826224848628044,
+            -0.0015118005685508251,
+            0.045844126492738724,
+            -0.006737126037478447,
+            0.02572700008749962,
+            -0.008158445358276367,
+            -0.002590042306110263,
+            0.00386432814411819,
+            0.012531733140349388,
+            0.005157464183866978,
+            0.002361952792853117,
+            -0.022906983271241188,
+            -0.02177596092224121,
+            0.020569536834955215,
+            -0.0062470161356031895,
+            -0.0011479882523417473,
+            -0.027069147676229477,
+            0.00037135250749997795,
+            0.009078343398869038,
+            0.0009217836195603013,
+            -0.001966094831004739,
+            0.00671827606856823,
+            0.015306509099900723,
+            0.003590997541323304,
+            -0.014680677093565464,
+            -0.004935029894113541,
+            -0.026692138984799385,
+            -0.004297886975109577,
+            0.010744716972112656,
+            0.006314877886325121,
+            0.012222587130963802,
+            -0.013481792993843555,
+            -0.031547997146844864,
+            0.007796517573297024,
+            -0.009251766838133335,
+            -0.007023652084171772,
+            0.03139719367027283,
+            -0.02103702537715435,
+            -0.010247067548334599,
+            -0.02099178545176983,
+            -0.018654337152838707,
+            -0.008075503632426262,
+            0.020690178498625755,
+            -0.010624075308442116,
+            0.0046975151635706425,
+            -0.02500314638018608,
+            -0.00152405328117311,
+            0.017055824398994446,
+            0.02053937502205372,
+            0.006205545272678137,
+            0.014326289296150208
+          ],
+          "keyphrases": [
+            "Hash Layers",
+            "token ID",
+            "expert ID",
+            "load balance",
+            "Hashing based routing"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "2d39419a-1397-4001-8e1e-a802772c4f79",
+        "properties": {
+          "page_content": "E Fine-Tuning Details\n\nWe did a hyperparameter search for both baseline models and expert choice method. For fine-tuning of the 8B dense model, we use a constant learning rate of 0.0001 and a dropout rate of 0.1. We freeze the attention layer and feed-forward layer while leaving the embedding and layer normalization trainable. This setting has been found optimal for the 8B dense model. For MoE 8B/64E models including GShard top-2 gating and expert choice, we found continuing the learning rate from the pre-trained model while using a square root learning rate decay works better. In addition, we do not apply parameter freezing for fine-tuning MoE models. For models with 100M expert size, we use a constant learning rate of 0.0001 and no dropout is used.",
+          "title": "E Fine-Tuning Details",
+          "embedding": [
+            0.022390764206647873,
+            -0.0034248128067702055,
+            -0.016889112070202827,
+            -0.013610072433948517,
+            0.024709414690732956,
+            0.026149997487664223,
+            -0.02159501425921917,
+            0.05106521025300026,
+            -0.016985150054097176,
+            0.006918224971741438,
+            0.008664073422551155,
+            -0.005028318148106337,
+            0.013939348049461842,
+            0.006582088768482208,
+            0.030732421204447746,
+            0.0230218768119812,
+            0.008314217440783978,
+            0.024915212765336037,
+            0.007045133039355278,
+            -0.03493069112300873,
+            0.022267285734415054,
+            -0.0422845184803009,
+            0.0016892541898414493,
+            -0.017520224675536156,
+            0.0009578156750649214,
+            -0.003855272429063916,
+            0.014789978042244911,
+            0.014254904352128506,
+            -0.01691655069589615,
+            -0.0006456895498558879,
+            0.039293594658374786,
+            0.015421089716255665,
+            0.0020151000935584307,
+            0.005889237392693758,
+            -0.05144936591386795,
+            -0.019770275801420212,
+            0.042311958968639374,
+            0.03193976730108261,
+            -0.0038278328720480204,
+            0.021361777558922768,
+            0.03226904198527336,
+            0.015064374543726444,
+            -0.023460911586880684,
+            -0.014995775185525417,
+            -0.019139165058732033,
+            -0.019098004326224327,
+            -0.017163507640361786,
+            0.007635085843503475,
+            0.004493244457989931,
+            -0.06272706389427185,
+            -0.01129142101854086,
+            -0.024627096951007843,
+            0.009597022086381912,
+            0.02502497099339962,
+            -0.044781528413295746,
+            -0.0015443384181708097,
+            -0.00871209241449833,
+            0.06366001814603806,
+            0.05512627959251404,
+            0.0056354207918047905,
+            0.022527962923049927,
+            0.0003787959576584399,
+            -0.040638137608766556,
+            -0.015023214742541313,
+            0.025903040543198586,
+            0.011387459933757782,
+            0.027220144867897034,
+            -0.023309992626309395,
+            -0.024174341931939125,
+            0.003937591332942247,
+            -0.04969322681427002,
+            0.03051290474832058,
+            -0.018123896792531013,
+            0.014186304993927479,
+            -0.005940686911344528,
+            -0.04017166420817375,
+            0.020140711218118668,
+            0.023749027401208878,
+            0.039952147752046585,
+            -0.011373739689588547,
+            0.015187853015959263,
+            -0.026561593636870384,
+            -0.00991943757981062,
+            0.040308862924575806,
+            0.052601832896471024,
+            -0.006551219150424004,
+            -0.022774919867515564,
+            -0.036549631506204605,
+            -0.026410674676299095,
+            0.015352491289377213,
+            -0.02174593135714531,
+            -0.005419333465397358,
+            -0.019619358703494072,
+            -0.006074455101042986,
+            -0.029278120025992393,
+            0.047168780118227005,
+            -0.032735515385866165,
+            0.011908813379704952,
+            -0.02487405389547348,
+            -0.0023117915261536837,
+            -0.016710754483938217,
+            0.005851508118212223,
+            -0.0023615260142832994,
+            -0.002745681209489703,
+            0.02949763648211956,
+            -0.01580524444580078,
+            -0.00518266623839736,
+            -0.005628560669720173,
+            0.0011584681924432516,
+            0.05894039571285248,
+            -0.024627096951007843,
+            -0.04722365736961365,
+            -0.016271719709038734,
+            0.03597339615225792,
+            -0.043354667723178864,
+            0.01352775376290083,
+            -0.008101560175418854,
+            -0.013013259507715702,
+            0.012965240515768528,
+            -0.04494616761803627,
+            0.03976007178425789,
+            0.01974283717572689,
+            0.034162379801273346,
+            0.012690844014286995,
+            -0.04423273354768753,
+            0.03284527361392975,
+            0.021938009187579155,
+            0.014817417599260807,
+            0.004798511043190956,
+            -0.021732212975621223,
+            0.016930270940065384,
+            0.008163299411535263,
+            -0.022706320509314537,
+            -0.0387173630297184,
+            0.0006821328424848616,
+            -0.0073538292199373245,
+            0.04705902189016342,
+            0.05186096206307411,
+            -0.026945747435092926,
+            -0.03528740629553795,
+            0.014693939127027988,
+            -0.01673819310963154,
+            0.026712510734796524,
+            0.004249717574566603,
+            -0.03471117094159126,
+            0.021512694656848907,
+            0.027000626549124718,
+            0.06689789891242981,
+            0.008897310122847557,
+            0.03155561164021492,
+            -0.03427213802933693,
+            0.005693729966878891,
+            -0.03163792937994003,
+            0.00961760152131319,
+            -0.005920107010751963,
+            -0.007587066385895014,
+            -0.02540912665426731,
+            0.0006538357120007277,
+            -0.015174132771790028,
+            0.0062905424274504185,
+            -7.942495722090825e-05,
+            0.007401848677545786,
+            0.0049631488509476185,
+            0.03336662799119949,
+            0.011627556756138802,
+            0.06398928910493851,
+            0.007834023796021938,
+            0.01986631564795971,
+            -0.07842255383729935,
+            0.028317730873823166,
+            0.040994856506586075,
+            0.017739741131663322,
+            0.00991943757981062,
+            0.007984941825270653,
+            0.0012545069912448525,
+            0.025971639901399612,
+            0.016326598823070526,
+            0.023762747645378113,
+            -0.031418412923812866,
+            0.02243192307651043,
+            -0.01140803936868906,
+            0.009898858144879341,
+            0.021732212975621223,
+            0.00917170662432909,
+            0.0123341279104352,
+            -0.01002233661711216,
+            -0.007415568456053734,
+            0.030677542090415955,
+            -0.0015400510746985674,
+            0.00259133311919868,
+            0.0060950350016355515,
+            0.020209310576319695,
+            -0.005601121112704277,
+            -0.0056731500662863255,
+            0.007628226187080145,
+            -0.0274396613240242,
+            0.004613293334841728,
+            -0.014515581540763378,
+            0.008259338326752186,
+            -0.013678671792149544,
+            -0.010289872996509075,
+            -0.03328431025147438,
+            -0.007820303551852703,
+            -0.04431505501270294,
+            0.006654118187725544,
+            0.0023580959532409906,
+            0.01473509892821312,
+            -0.01882360875606537,
+            0.04211987927556038,
+            -0.006077885162085295,
+            -0.001488601672463119,
+            -0.018988246098160744,
+            -0.02525820955634117,
+            -0.01018697489053011,
+            -0.02695946767926216,
+            0.02335115149617195,
+            0.01634031906723976,
+            0.023159075528383255,
+            0.015325050801038742,
+            -0.012313548475503922,
+            0.02746710181236267,
+            0.029717154800891876,
+            0.0010469945846125484,
+            -0.006835905835032463,
+            -0.038168568164110184,
+            -0.03122633509337902,
+            0.008444556035101414,
+            -0.017726020887494087,
+            0.013431714847683907,
+            -0.00541247334331274,
+            -0.01422746479511261,
+            -0.004575563594698906,
+            -0.023762747645378113,
+            0.019043125212192535,
+            0.02012699283659458,
+            0.006005856208503246,
+            0.002215752610936761,
+            -0.032433681190013885,
+            0.038909438997507095,
+            -0.01458417996764183,
+            -0.00680160615593195,
+            0.011696156114339828,
+            -0.018590370193123817,
+            -0.04447969049215317,
+            0.014968335628509521,
+            -0.008135859854519367,
+            -0.01580524444580078,
+            6.827759352745488e-05,
+            0.007491027936339378,
+            -0.006427740678191185,
+            -0.030046429485082626,
+            -0.03558924049139023,
+            -0.005481072701513767,
+            0.00847199559211731,
+            -0.009775379672646523,
+            0.04601631313562393,
+            0.03797649219632149,
+            -0.012704563327133656,
+            0.002215752610936761,
+            -0.02090902253985405,
+            0.040967416018247604,
+            9.405587479704991e-05,
+            -0.00932262558490038,
+            -0.0003839409037027508,
+            -0.008094700053334236,
+            -0.005031748209148645,
+            0.024407578632235527,
+            -0.00319843553006649,
+            0.016024762764573097,
+            0.007676245644688606,
+            0.014597900211811066,
+            -0.006715857423841953,
+            -0.013863889500498772,
+            0.0021608732640743256,
+            0.02388622611761093,
+            -0.018384573981165886,
+            0.006842765491455793,
+            -0.027906136587262154,
+            0.0010975864715874195,
+            0.011744175106287003,
+            0.006232233252376318,
+            0.03454653546214104,
+            0.0224593635648489,
+            0.029634835198521614,
+            -0.03369590640068054,
+            -0.011627556756138802,
+            -0.03471117094159126,
+            -0.03907407820224762,
+            -0.015105534344911575,
+            0.004887689836323261,
+            -0.0009843978332355618,
+            -0.025189610198140144,
+            -0.009164847433567047,
+            -0.0026496422942727804,
+            -0.007662525400519371,
+            0.026835989207029343,
+            -0.015283891931176186,
+            0.007532187271863222,
+            0.00599556602537632,
+            -0.028207972645759583,
+            -0.03177512809634209,
+            -0.0002012956392718479,
+            0.003044087439775467,
+            -0.01789066009223461,
+            -0.0331745520234108,
+            -0.004390345886349678,
+            0.01697143167257309,
+            0.011174802668392658,
+            -0.041653405874967575,
+            0.00736754946410656,
+            0.021402936428785324,
+            -0.005913247354328632,
+            0.011325720697641373,
+            -0.05452260747551918,
+            0.007024553604424,
+            -0.005014598369598389,
+            -0.0038758523296564817,
+            0.016697034239768982,
+            -0.03117145597934723,
+            0.006071025040000677,
+            0.03292759507894516,
+            0.025422846898436546,
+            -0.03965030983090401,
+            0.00899334903806448,
+            -0.007168611511588097,
+            0.0097959591075778,
+            -0.007655665744096041,
+            0.055702514946460724,
+            -0.03641243278980255,
+            -0.014803697355091572,
+            -0.030595222488045692,
+            -0.0227886401116848,
+            -0.03490325063467026,
+            0.0001791081012925133,
+            -0.05101033300161362,
+            0.003769523464143276,
+            -0.0037043543998152018,
+            0.017465345561504364,
+            0.017588824033737183,
+            -0.008952190168201923,
+            -0.018947087228298187,
+            0.004757351242005825,
+            -0.04008934646844864,
+            -0.028976282104849815,
+            0.0049494290724396706,
+            0.026534153148531914,
+            -0.031363531947135925,
+            -0.013459154404699802,
+            -0.0084102563560009,
+            -0.04329978674650192,
+            0.0007151461904868484,
+            -0.0018041577422991395,
+            0.021814530715346336,
+            0.0021831681951880455,
+            -0.0057554692029953,
+            -0.03226904198527336,
+            -0.01306127943098545,
+            0.00024459886481054127,
+            -0.00391358183696866,
+            0.03366846591234207,
+            0.006746727041900158,
+            -0.02487405389547348,
+            -0.028235411271452904,
+            -0.006650688126683235,
+            -0.0272064246237278,
+            -0.031418412923812866,
+            0.027343623340129852,
+            0.015846405178308487,
+            -0.016326598823070526,
+            0.0035671561490744352,
+            -0.025121010839939117,
+            0.017698582261800766,
+            0.01223122887313366,
+            0.014776257798075676,
+            0.030677542090415955,
+            -0.009988036938011646,
+            -0.0024575646966695786,
+            -0.020991342142224312,
+            -0.05183352157473564,
+            0.024105742573738098,
+            -0.012368427589535713,
+            0.007470448035746813,
+            0.034189820289611816,
+            -0.008499435149133205,
+            0.03196720778942108,
+            0.028317730873823166,
+            0.0037043543998152018,
+            0.016203120350837708,
+            -0.0460437536239624,
+            0.03520508483052254,
+            0.018329694867134094,
+            -0.06069653108716011,
+            0.016326598823070526,
+            0.019070565700531006,
+            0.013219057582318783,
+            0.049857866019010544,
+            -0.028262851759791374,
+            -0.006492909975349903,
+            -0.004112519323825836,
+            0.004455515183508396,
+            -0.027576860040426254,
+            0.018453173339366913,
+            0.007415568456053734,
+            -0.03978750854730606,
+            0.007772284094244242,
+            -0.013925628736615181,
+            -0.014001087285578251,
+            -0.004304597154259682,
+            0.0400344654917717,
+            -0.03635755181312561,
+            -0.02261028066277504,
+            -0.013088718988001347,
+            -0.0036494750529527664,
+            -0.002870874712243676,
+            0.02812565304338932,
+            0.03709842264652252,
+            -0.018755009397864342,
+            0.010468230582773685,
+            -0.007443008478730917,
+            -0.014254904352128506,
+            0.00044632324716076255,
+            0.01163441687822342,
+            -0.016930270940065384,
+            0.003308194223791361,
+            0.007093152496963739,
+            -0.0071343122981488705,
+            -0.02121085859835148,
+            0.022267285734415054,
+            0.024256661534309387,
+            -0.013500313274562359,
+            0.00056208431487903,
+            -0.0007906052633188665,
+            0.030787300318479538,
+            -0.03811369091272354,
+            -0.03940335288643837,
+            -0.01440582238137722,
+            -0.00802610069513321,
+            0.034491654485464096,
+            -0.03794905170798302,
+            0.0024078304413706064,
+            -0.006842765491455793,
+            -0.005597691051661968,
+            -0.005782908760011196,
+            0.0013642656849697232,
+            -0.011956832371652126,
+            -0.01691655069589615,
+            -0.02782381698489189,
+            -0.016504956409335136,
+            -0.02723386511206627,
+            -0.007161751855164766,
+            -0.053287822753190994,
+            0.01197741273790598,
+            -0.010111515410244465,
+            -0.020826702937483788,
+            0.04072045907378197,
+            -0.00449667451903224,
+            0.006071025040000677,
+            -0.01583268493413925,
+            0.007189191412180662,
+            -0.050598736852407455,
+            -0.01819249615073204,
+            -0.0011936252703890204,
+            0.04316258803009987,
+            0.0065649389289319515,
+            -0.022171247750520706,
+            -0.0007284372695721686,
+            -0.022692600265145302,
+            -0.008863010443747044,
+            0.006355711724609137,
+            0.013520893640816212,
+            -0.027700338512659073,
+            -0.022692600265145302,
+            0.009562722407281399,
+            0.015407370403409004,
+            0.024613376706838608,
+            0.04428761452436447,
+            -0.011538377963006496,
+            -0.061190444976091385,
+            0.01717722788453102,
+            -0.015036934986710548,
+            0.028029615059494972,
+            -0.005323294550180435,
+            -0.0008073262870311737,
+            -0.01771230250597,
+            -0.014762538485229015,
+            -0.004976868629455566,
+            0.019948633387684822,
+            -0.006108754780143499,
+            0.0035568661987781525,
+            0.031390972435474396,
+            -0.016422636806964874,
+            0.0037146443501114845,
+            -0.0012545069912448525,
+            0.0039787511341273785,
+            -0.01640891842544079,
+            -0.05007738247513771,
+            0.025793282315135002,
+            -0.040939975529909134,
+            0.010536829940974712,
+            -0.0021385785657912493,
+            -0.02108738012611866,
+            -0.01575036533176899,
+            0.01370611134916544,
+            -0.015668047592043877,
+            0.013500313274562359,
+            0.030046429485082626,
+            0.02266516163945198,
+            0.045055925846099854,
+            -0.036933783441782,
+            -0.007285230327397585,
+            0.03844296559691429,
+            -0.014309783466160297,
+            -0.003944451455026865,
+            -0.013129877857863903,
+            0.0026324924547225237,
+            0.027576860040426254,
+            -0.01006349641829729,
+            0.04911699518561363,
+            -0.006359141785651445,
+            0.009631321765482426,
+            0.010660308413207531,
+            -0.00233408622443676,
+            -0.012642824091017246,
+            0.04192780330777168,
+            -0.029003722593188286,
+            0.012937800958752632,
+            0.012073451653122902,
+            -0.011099343188107014,
+            0.020442547276616096,
+            -0.0004986301064491272,
+            0.002894884441047907,
+            0.015517128631472588,
+            0.008616053499281406,
+            -0.04483640938997269,
+            -0.006372861564159393,
+            -0.01175103522837162,
+            -0.020497428253293037,
+            0.022500522434711456,
+            -0.013692391104996204,
+            -0.019619358703494072,
+            -0.03147329390048981,
+            -0.019770275801420212,
+            -0.0037180741783231497,
+            -0.003611745545640588,
+            0.00996059738099575,
+            -0.0018041577422991395,
+            0.026616472750902176,
+            -0.03605571761727333,
+            0.011435478925704956,
+            0.030458025634288788,
+            -0.005467352457344532,
+            -0.024297820404171944,
+            -0.03391542285680771,
+            0.025930481031537056,
+            -0.02073066495358944,
+            -0.019646797329187393,
+            0.004397206008434296,
+            0.027906136587262154,
+            0.012807462364435196,
+            0.002601622836664319,
+            -0.005957836750894785,
+            0.00736754946410656,
+            -0.01684795320034027,
+            -0.007840882986783981,
+            -0.024407578632235527,
+            -0.04244915768504143,
+            0.015489689074456692,
+            -0.004716191906481981,
+            -0.011428618803620338,
+            0.016065921634435654,
+            -0.02248680219054222,
+            -0.01577780582010746,
+            0.008458275347948074,
+            0.01261538453400135,
+            0.00167038943618536,
+            0.002545028692111373,
+            -0.006139624398201704,
+            0.0024181201588362455,
+            -0.0041605387814342976,
+            0.008375956676900387,
+            0.04516568407416344,
+            -0.016299158334732056,
+            -0.006609528791159391,
+            0.035863637924194336,
+            0.027576860040426254,
+            -0.008417116478085518,
+            -0.010660308413207531,
+            -0.013726690784096718,
+            0.0022929266560822725,
+            -0.033778224140405655,
+            0.023749027401208878,
+            0.051559124141931534,
+            0.007326389662921429,
+            -0.024531057104468346,
+            -0.030650101602077484,
+            -0.015709206461906433,
+            -0.005398753564804792,
+            0.049171872437000275,
+            0.02446245774626732,
+            -0.043683942407369614,
+            -0.02601279877126217,
+            0.03161048889160156,
+            0.04469921067357063,
+            0.0077517046593129635,
+            -0.011997992172837257,
+            0.0050557577051222324,
+            0.030293386429548264,
+            0.01628543995320797,
+            -0.03528740629553795,
+            0.011353160254657269,
+            0.028592128306627274,
+            -0.0013273936929181218,
+            -0.012265528552234173,
+            -0.004163968842476606,
+            -0.006300832144916058,
+            0.015366210602223873,
+            -0.008773831650614738,
+            -0.009302045218646526,
+            -0.016697034239768982,
+            -0.026712510734796524,
+            0.07606274634599686,
+            -0.00541247334331274,
+            0.022774919867515564,
+            0.008890450932085514,
+            0.00915798731148243,
+            0.04044606164097786,
+            0.003930731676518917,
+            -0.009494123049080372,
+            0.011867653578519821,
+            -0.006976534146815538,
+            -0.02621859684586525,
+            -0.011819634586572647,
+            -0.01605220139026642,
+            -0.0019053415162488818,
+            0.016450077295303345,
+            -0.006221943534910679,
+            0.03018362820148468,
+            0.03737282007932663,
+            0.013150458224117756,
+            -0.01854921132326126,
+            -0.01580524444580078,
+            0.00403363024815917,
+            0.01849433220922947,
+            0.03986983001232147,
+            -0.011373739689588547,
+            -0.0018436022801324725,
+            0.0026925168931484222,
+            -0.006414020899683237,
+            0.003155561164021492,
+            -0.011435478925704956,
+            -0.013712971471250057,
+            -0.008142719976603985,
+            0.022006608545780182,
+            0.0311988964676857,
+            -0.0031864307820796967,
+            -0.027508260682225227,
+            -0.004013050347566605,
+            0.022377043962478638,
+            0.01575036533176899,
+            0.0004411783011164516,
+            0.0066198185086250305,
+            0.014103986322879791,
+            0.006335131824016571,
+            -0.0015443384181708097,
+            0.02487405389547348,
+            -0.02645183354616165,
+            0.0016129376599565148,
+            0.021718492731451988,
+            -0.02431154064834118,
+            0.02436641976237297,
+            -0.011654996313154697,
+            -0.030759861692786217,
+            -0.027371061965823174,
+            -0.007141171954572201,
+            0.02764545939862728,
+            -0.02057974599301815,
+            -9.443102317163721e-05,
+            0.0246682558208704,
+            0.01406282652169466,
+            0.0007717405096627772,
+            0.0036494750529527664,
+            0.008506295271217823,
+            0.008835570886731148,
+            -0.02988179214298725,
+            -0.009535282850265503,
+            -0.00014116420061327517,
+            -0.013171037659049034,
+            0.04244915768504143,
+            -0.05485188588500023,
+            -0.013905048370361328,
+            -0.011847074143588543,
+            -0.02347462996840477,
+            -0.01819249615073204,
+            0.004479524679481983,
+            0.0017295562429353595,
+            -0.006379721220582724,
+            -0.024023424834012985,
+            0.0349581278860569,
+            0.00917170662432909,
+            -0.020113272592425346,
+            -0.019962353631854057,
+            0.03048546425998211,
+            -0.01867268979549408,
+            0.03254343941807747,
+            0.0033322039525955915,
+            -0.012622244656085968,
+            -0.005096917040646076,
+            -0.01267712377011776,
+            -0.02159501425921917,
+            -0.03572643920779228,
+            -0.020634625107049942,
+            0.04552239924669266,
+            0.015393650159239769,
+            0.003906721714884043,
+            -0.01572292670607567,
+            0.023282554000616074,
+            -0.008211318403482437,
+            0.009940017946064472,
+            0.011997992172837257,
+            0.05221767723560333,
+            0.0049631488509476185,
+            0.03630267456173897,
+            0.006163633894175291,
+            -0.06206851452589035,
+            0.0012793742353096604,
+            0.020236751064658165,
+            0.020140711218118668,
+            0.03397030010819435,
+            0.016450077295303345,
+            0.004887689836323261,
+            -0.021526414901018143,
+            -0.021197138354182243,
+            -0.04703158140182495,
+            -0.00927460566163063,
+            0.005168946459889412,
+            -0.011661856435239315,
+            -0.02812565304338932,
+            0.04722365736961365,
+            0.024736855179071426,
+            -0.003387083299458027,
+            0.018535491079092026,
+            -0.01009779516607523,
+            0.0016318024136126041,
+            0.01610708236694336,
+            -0.018425732851028442,
+            0.036220353096723557,
+            -0.005337014328688383,
+            0.03081474080681801,
+            -0.025546325370669365,
+            0.02034650929272175,
+            0.02182825095951557,
+            -0.012313548475503922,
+            0.034519094973802567,
+            0.005607980769127607,
+            -0.0623977929353714,
+            -0.0028760195709764957,
+            0.027906136587262154,
+            0.013452294282615185,
+            0.0015837829560041428,
+            0.0022826369386166334,
+            -0.01634031906723976,
+            0.019166603684425354,
+            -0.002447274746373296,
+            0.0115109384059906,
+            0.04524800181388855,
+            -0.0025621785316616297,
+            0.003800393082201481,
+            0.026794830337166786,
+            0.0058549377135932446,
+            -0.014296064153313637,
+            0.0036529048811644316,
+            -0.012039151974022388,
+            0.04524800181388855,
+            -0.03778441622853279,
+            -0.00043946332880295813,
+            0.0522451177239418,
+            -0.005186096299439669,
+            0.004208558239042759,
+            -0.015915004536509514,
+            -0.01867268979549408,
+            -0.005957836750894785,
+            0.02001723274588585,
+            0.004736771807074547,
+            0.04058326035737991,
+            -0.040336303412914276,
+            0.010145815089344978,
+            -0.020895302295684814,
+            0.0041742585599422455,
+            0.010776927694678307,
+            0.015764085575938225,
+            0.013390555046498775,
+            -0.038936879485845566,
+            -0.006983393803238869,
+            0.015283891931176186,
+            0.06398928910493851,
+            -0.006307692267000675,
+            -0.012491906061768532,
+            0.04601631313562393,
+            -0.005313004367053509,
+            -0.015078093856573105,
+            0.016422636806964874,
+            0.004225708078593016,
+            0.02848236821591854,
+            0.02001723274588585,
+            -0.02812565304338932,
+            -0.04137900844216347,
+            0.009137407876551151,
+            0.01473509892821312,
+            0.005309574771672487,
+            -0.001153323333710432,
+            -0.006846195552498102,
+            -0.013966787606477737,
+            -0.013054419308900833,
+            -0.0038209729827940464,
+            -0.007587066385895014,
+            0.023309992626309395,
+            -0.02352951094508171,
+            0.007635085843503475,
+            0.02949763648211956,
+            -0.013493454083800316,
+            -0.020360229536890984,
+            -0.005282134748995304,
+            -0.005127786658704281,
+            -0.002359810983762145,
+            -0.02481917478144169,
+            -0.010488810949027538,
+            -0.018988246098160744,
+            -0.019701676443219185,
+            0.04036374390125275,
+            0.016806792467832565,
+            0.005889237392693758,
+            0.009343205019831657,
+            -0.007292089983820915,
+            0.04500104486942291,
+            -0.028015894815325737,
+            0.05216279625892639,
+            -0.014872296713292599,
+            -0.003529426408931613,
+            -0.005518801975995302,
+            0.02654787339270115,
+            -0.014940896071493626,
+            0.019976073876023293,
+            -0.03215928375720978,
+            0.00703141326084733,
+            0.009144267067313194,
+            -0.005402183625847101,
+            -0.00858861394226551,
+            -0.01124340109527111,
+            0.001927636214531958,
+            -0.0026925168931484222,
+            0.02576584182679653,
+            0.033723343163728714,
+            0.010255573317408562,
+            0.004397206008434296,
+            -0.032790396362543106,
+            -0.005638850387185812,
+            -0.0007215773803181946,
+            -0.0030749570578336716,
+            -0.033723343163728714,
+            -0.018864767625927925,
+            -0.037510018795728683,
+            -0.013925628736615181,
+            0.009809679351747036,
+            -0.0035637260880321264,
+            0.021965449675917625,
+            0.019125444814562798,
+            -0.0017475634813308716,
+            -0.01605220139026642,
+            -0.013870748691260815,
+            0.006942234467715025,
+            0.013308236375451088,
+            -0.01565432734787464,
+            0.0015494833933189511,
+            0.0007811728864908218,
+            -0.010584849864244461,
+            0.021005060523748398,
+            -0.01989375427365303,
+            -0.025601204484701157,
+            -0.008273057639598846,
+            -0.03303735330700874,
+            0.00472648162394762,
+            0.0199211947619915,
+            -0.0038072532042860985,
+            0.01971539668738842,
+            0.009013929404318333,
+            0.002479859394952655,
+            -0.006678127683699131,
+            -0.03386054188013077,
+            -0.031418412923812866,
+            -0.034436777234077454,
+            -0.013857029378414154,
+            0.013994227163493633,
+            0.0008519157418049872,
+            0.02009955234825611,
+            0.008156439289450645,
+            0.00359802576713264,
+            0.018439453095197678,
+            -0.0007511607254855335,
+            -0.013356255367398262,
+            -0.013500313274562359,
+            -0.001734701101668179,
+            -0.01732814684510231,
+            0.0061979335732758045,
+            -0.03084217943251133,
+            0.0460437536239624,
+            -0.02230844460427761,
+            0.009343205019831657,
+            0.014940896071493626,
+            -0.009933157823979855,
+            -0.0026856570038944483,
+            -0.027247583493590355,
+            0.02248680219054222,
+            0.04596143215894699,
+            -0.013342536054551601,
+            0.033394068479537964,
+            0.03056778386235237,
+            0.0224593635648489,
+            -0.025450287386775017,
+            0.02759058028459549,
+            0.013664951547980309,
+            -0.008986488915979862,
+            0.030320826917886734,
+            -0.020442547276616096,
+            0.025875601917505264,
+            -0.007086292840540409,
+            -0.022884678095579147,
+            0.004294306971132755,
+            0.025367967784404755,
+            0.010866106487810612,
+            0.014186304993927479,
+            -0.01918032392859459,
+            0.004061069805175066,
+            0.0029669133946299553,
+            -0.019619358703494072,
+            0.04725109785795212,
+            0.002308361465111375,
+            0.021128538995981216,
+            0.0113943200558424,
+            -0.012773162685334682,
+            0.022034049034118652,
+            -0.016203120350837708,
+            -0.01933124102652073,
+            -0.0028365750331431627,
+            0.010927845723927021,
+            0.015379930846393108,
+            -0.048540763556957245,
+            0.03355870768427849,
+            0.028098214417696,
+            0.009645041078329086,
+            0.0058309282176196575,
+            0.008725812658667564,
+            -0.002208892721682787,
+            0.006352281663566828,
+            -0.00680160615593195,
+            -0.0009157987078651786,
+            0.044754087924957275,
+            -0.01175103522837162,
+            -0.0019019115716218948,
+            -0.0049871583469212055,
+            0.030101308599114418,
+            0.0017158363480120897,
+            -0.008252478204667568,
+            -0.008808131329715252,
+            -0.01458417996764183,
+            0.00013408990344032645,
+            0.00420512817800045,
+            -0.004887689836323261,
+            -0.016244279220700264,
+            -0.027096666395664215,
+            0.01319847721606493,
+            0.004808800760656595,
+            -0.03421725705265999,
+            -0.01060542929917574,
+            -0.02075810357928276,
+            0.010502530261874199,
+            -0.006544359494000673,
+            -0.0030458024702966213,
+            0.00018693269521463662,
+            -0.007923202589154243,
+            -0.014076546765863895,
+            0.03314711153507233,
+            0.003947881516069174,
+            0.04434249550104141,
+            -0.00342652783729136,
+            0.030979378148913383,
+            -0.02621859684586525,
+            0.03473861142992973,
+            -0.004194838460534811,
+            -0.015791526064276695,
+            0.02505241148173809,
+            -0.03498556837439537,
+            0.031089136376976967,
+            0.02225356549024582,
+            0.004983728751540184,
+            -0.0004171686014160514,
+            0.02949763648211956,
+            -0.04563215747475624,
+            -5.0806247600121424e-05,
+            -0.00553252175450325,
+            -0.0035843057557940483,
+            0.012148910202085972,
+            -0.01048195082694292,
+            -0.012965240515768528,
+            0.0027748357970267534,
+            0.015338771045207977,
+            0.02261028066277504,
+            0.03056778386235237,
+            0.00863663386553526,
+            -0.013116158545017242,
+            0.013520893640816212,
+            -0.0012802317505702376,
+            0.0026925168931484222,
+            -0.015969883650541306,
+            0.020181871950626373,
+            0.00204253988340497,
+            -0.03290015459060669,
+            -0.005786338821053505,
+            -0.06234291195869446,
+            -0.009075668640434742,
+            0.007154891733080149,
+            0.009151127189397812,
+            -0.004925419110804796,
+            -0.029717154800891876,
+            -0.038964319974184036,
+            0.017616262659430504,
+            0.02675366960465908,
+            -0.006623248569667339,
+            -0.008773831650614738,
+            -0.0224593635648489,
+            -0.016024762764573097,
+            0.03320199251174927,
+            -0.0007057138136588037,
+            0.012786882929503918,
+            0.004338896367698908,
+            0.03578132018446922,
+            -0.0038689924404025078,
+            -0.010715187527239323,
+            -0.04810172691941261,
+            -0.02949763648211956,
+            0.016244279220700264,
+            0.015475969761610031,
+            -0.009624461643397808,
+            0.019290082156658173,
+            -0.005782908760011196,
+            -0.02550516650080681,
+            -0.017245827242732048,
+            0.03594595566391945,
+            -0.012080310843884945,
+            0.028893964365124702,
+            0.01544852927327156,
+            -0.021512694656848907,
+            0.002509013982489705,
+            -0.0018195925513282418,
+            -0.009398084133863449,
+            0.005700589623302221,
+            0.02484661340713501,
+            0.0013719830894842744,
+            0.005611410830169916,
+            0.011922532692551613,
+            0.008252478204667568,
+            0.002426695078611374,
+            -0.022294726222753525,
+            -0.003380223410204053,
+            0.015215292572975159,
+            -0.03701610490679741,
+            0.008760112337768078,
+            -0.005282134748995304,
+            0.004908269736915827,
+            0.008382816798985004,
+            0.04346442595124245,
+            -0.005048897583037615,
+            0.005216965917497873,
+            -0.011483498848974705,
+            0.01699887029826641,
+            0.011606977321207523,
+            0.0033596435096114874,
+            -0.00541247334331274,
+            -0.010955285280942917,
+            -0.026794830337166786,
+            -0.008691512979567051,
+            0.01388446893543005,
+            0.012505626305937767,
+            0.01457046065479517,
+            0.007209771312773228,
+            0.004942568950355053,
+            -0.016834232956171036,
+            -0.006149914115667343,
+            0.010550550185143948,
+            0.01915288344025612,
+            -0.022061487659811974,
+            -0.02293955720961094,
+            0.01966051757335663,
+            -0.003011502791196108,
+            0.022322164848446846,
+            -0.021883130073547363,
+            -0.020675785839557648,
+            0.026287196204066277,
+            0.0712333619594574,
+            -0.0003322771517559886,
+            -0.010756347328424454,
+            -0.02228100597858429,
+            -0.011888233944773674,
+            -0.01664215512573719,
+            -0.017163507640361786,
+            0.027851257473230362,
+            0.03591851890087128,
+            -0.008238758891820908,
+            0.018466891720891,
+            -0.0020288201048970222,
+            0.0038038231432437897,
+            0.004218847956508398,
+            -0.030430585145950317,
+            0.01326021645218134,
+            -0.007552767172455788,
+            0.006366001442074776,
+            0.012999540194869041,
+            0.003989040851593018,
+            -0.008897310122847557,
+            -0.0018710419535636902,
+            0.009885137900710106,
+            0.012930940836668015,
+            -0.005299284588545561,
+            0.02126573771238327,
+            -0.0062253731302917,
+            0.009981176815927029,
+            -0.011318860575556755,
+            -0.018151337280869484,
+            0.00951470248401165,
+            0.01926264353096485,
+            -0.014474421739578247,
+            0.011936252936720848,
+            -0.007600786164402962,
+            0.03440933674573898,
+            -0.011421759612858295,
+            0.0029754883144050837,
+            -0.0011610407382249832,
+            -0.019454719498753548,
+            -0.002123143756762147,
+            0.0010718618286773562,
+            -0.002905174158513546,
+            0.02052486687898636,
+            0.004064499866217375,
+            0.03770209476351738,
+            0.021402936428785324,
+            -0.04862308129668236,
+            0.03633011132478714,
+            -0.009686200879514217,
+            -0.01388446893543005,
+            0.00857489462941885,
+            -0.009754800237715244,
+            -0.0019207763252779841,
+            -0.016271719709038734,
+            -0.014158865436911583,
+            0.013870748691260815,
+            -0.007086292840540409,
+            0.003745513968169689,
+            -0.004808800760656595,
+            0.015009495429694653,
+            0.0020545446313917637,
+            0.00772426463663578,
+            0.017259547486901283,
+            -0.006348851602524519,
+            -0.01661471463739872,
+            0.03520508483052254,
+            0.021869409829378128,
+            0.0004092368471901864,
+            0.0024215502198785543,
+            -0.004359476268291473,
+            0.02815309353172779,
+            0.009432383812963963,
+            -0.010042916052043438,
+            -0.015942443162202835,
+            -0.0019224913557991385,
+            -0.003222445258870721,
+            0.03657706826925278,
+            0.03180256858468056,
+            -0.02001723274588585,
+            0.0018933366518467665,
+            -0.0011258836602792144,
+            -0.009864558465778828,
+            0.009164847433567047,
+            -0.025903040543198586,
+            0.01816505566239357,
+            -0.006636968348175287,
+            0.007621366064995527,
+            0.02878420613706112,
+            -0.0044486550614237785,
+            0.012656544335186481,
+            0.013973647728562355,
+            0.007587066385895014,
+            0.003308194223791361,
+            0.006242522969841957,
+            0.018096456304192543,
+            -0.006667837966233492,
+            0.005230685696005821,
+            -0.029031163081526756,
+            -0.02075810357928276,
+            -0.034134939312934875,
+            0.01122968178242445,
+            0.011600117199122906,
+            -0.01343857403844595,
+            -0.014028526842594147,
+            0.009315765462815762,
+            0.026849709451198578,
+            0.005649140570312738,
+            -0.0046990420669317245,
+            -0.020620906725525856,
+            0.005553101655095816,
+            -0.005587401334196329,
+            -0.02711038663983345,
+            -0.01364437211304903,
+            0.02469569630920887,
+            0.0045275441370904446,
+            0.019070565700531006,
+            -0.008279917761683464,
+            0.0073538292199373245,
+            0.0004371052491478622,
+            -0.014611619524657726,
+            0.008279917761683464,
+            0.02845492959022522,
+            0.022870957851409912,
+            -0.008622913621366024,
+            0.0019224913557991385,
+            -0.0018333124462515116,
+            -0.007271510548889637,
+            0.021691052243113518,
+            -0.0008163299644365907,
+            -0.009974316693842411,
+            0.025079850107431412,
+            0.011668716557323933,
+            -0.016532396897673607,
+            0.014501861296594143,
+            -0.02594420127570629,
+            0.006297402549535036,
+            0.016175679862499237,
+            0.009603881277143955,
+            -0.005957836750894785,
+            -0.0030921068973839283,
+            0.016902832314372063,
+            0.03128121420741081,
+            0.006379721220582724,
+            0.0027782658580690622,
+            -0.021005060523748398,
+            -0.014639060012996197,
+            -1.834223439800553e-05,
+            0.00466131279245019,
+            0.042641233652830124,
+            -0.009775379672646523,
+            -0.018069017678499222,
+            0.013747270219027996,
+            -0.010433931834995747,
+            -0.0021728782448917627,
+            -0.014323503710329533,
+            0.01864525116980076,
+            0.005433053243905306,
+            0.015860125422477722,
+            0.007758564315736294,
+            -0.005210105795413256,
+            0.02314535528421402,
+            0.0026890868321061134,
+            -0.028674446046352386,
+            0.01834341324865818,
+            0.01613452099263668,
+            0.017012590542435646,
+            -0.006204793695360422,
+            -0.00229978677816689,
+            0.013685530982911587,
+            -0.007978081703186035,
+            0.016326598823070526,
+            0.030732421204447746,
+            0.03133609518408775,
+            -0.0006752729532308877,
+            -0.024119462817907333,
+            0.014611619524657726,
+            -0.003472832264378667,
+            -0.012046011164784431,
+            -0.0255600456148386,
+            0.018604090437293053,
+            -0.0039650313556194305,
+            -0.01631287857890129,
+            -0.009953737258911133,
+            -0.025614924728870392,
+            0.00036314676981419325,
+            -0.010598569177091122,
+            0.003944451455026865,
+            -0.011661856435239315,
+            0.03536972403526306,
+            0.012073451653122902,
+            0.019015686586499214,
+            0.0018607521196827292,
+            -0.02728874422609806,
+            -0.018178775906562805,
+            0.03226904198527336,
+            -0.012882920913398266,
+            -0.00656150933355093,
+            -0.0058309282176196575,
+            -0.0037146443501114845,
+            -0.03569899871945381,
+            0.0154348099604249,
+            0.027343623340129852,
+            0.013795290142297745,
+            0.0010615718783810735,
+            -0.024448739364743233,
+            -0.01557200774550438,
+            0.01249876618385315,
+            0.01422746479511261,
+            -0.026026519015431404,
+            0.027700338512659073,
+            -0.006441460456699133,
+            -0.016203120350837708,
+            0.011620696634054184,
+            -0.007813443429768085,
+            -0.010536829940974712,
+            0.00691136484965682,
+            -0.02767289988696575,
+            0.002752541098743677,
+            0.024435019120573997,
+            -0.019043125212192535,
+            0.00564571050927043,
+            0.012011712417006493,
+            0.035891078412532806,
+            0.013911908492445946,
+            0.01867268979549408,
+            -0.00025253064814023674,
+            -0.02914092130959034,
+            0.01291722059249878,
+            -0.00158464047126472,
+            -0.022377043962478638,
+            0.026794830337166786,
+            -0.015736646950244904,
+            0.017136069014668465,
+            -0.005803488660603762,
+            -0.011552097275853157,
+            0.020236751064658165,
+            0.013225916773080826,
+            0.01237528771162033,
+            -0.0002709666732698679,
+            0.004743631463497877,
+            0.007539046928286552,
+            -0.011346300132572651,
+            0.020181871950626373,
+            -0.013911908492445946,
+            -0.00023923955450300127,
+            -0.004709331784397364,
+            -0.008547455072402954,
+            -0.031418412923812866,
+            0.03427213802933693,
+            0.0014825991820544004,
+            -0.01492717582732439,
+            0.005913247354328632,
+            -0.025422846898436546,
+            -0.025971639901399612,
+            -0.006880495231598616,
+            -0.008725812658667564,
+            -0.0039787511341273785,
+            -0.0054227630607783794,
+            -0.007820303551852703,
+            -0.016889112070202827,
+            0.007792863994836807,
+            0.019399840384721756,
+            0.02764545939862728,
+            -0.005378173664212227,
+            0.0374002605676651,
+            0.01346601452678442,
+            0.02228100597858429,
+            -0.004397206008434296,
+            0.01006349641829729,
+            0.01565432734787464,
+            0.011819634586572647,
+            -0.015530848875641823,
+            0.03572643920779228,
+            -0.0014242900069803,
+            -0.03130865469574928,
+            0.002548458520323038,
+            -0.016957711428403854,
+            -0.012828041799366474,
+            -0.009603881277143955,
+            -0.004218847956508398,
+            0.025464005768299103,
+            0.0020785543601959944,
+            0.023995984345674515,
+            -0.011703016236424446,
+            -0.002551888581365347,
+            0.0007665955345146358,
+            -0.01557200774550438,
+            -0.01082494668662548,
+            -0.026739951223134995,
+            -0.009946877136826515,
+            -0.027096666395664215,
+            0.005443342961370945,
+            0.013514033518731594,
+            0.009020788595080376,
+            -0.00950098317116499,
+            -0.003594595706090331,
+            0.0013291086070239544,
+            -0.025464005768299103,
+            -0.02129317820072174,
+            0.01473509892821312,
+            -0.0007095725159160793,
+            0.004534404259175062,
+            0.010269293561577797,
+            0.015064374543726444,
+            -0.013054419308900833,
+            -0.013253356330096722,
+            -0.01337683480232954,
+            -0.007244070526212454,
+            -0.01974283717572689,
+            -0.0016472372226417065,
+            0.015297611244022846,
+            0.008266198448836803,
+            0.00876697152853012,
+            -0.014200025238096714,
+            0.01562688685953617,
+            0.0028468649834394455,
+            0.0038895721081644297,
+            0.014021667651832104,
+            0.034107498824596405,
+            -0.036522191017866135,
+            -0.0035071317106485367,
+            -0.029964111745357513,
+            0.02182825095951557,
+            0.012162630446255207,
+            0.020332789048552513,
+            -0.012299828231334686,
+            -0.008739531971514225,
+            -0.0362752340734005,
+            -0.0041605387814342976,
+            -0.018027858808636665,
+            0.00719605153426528,
+            0.013088718988001347,
+            -0.009192286990582943,
+            0.04672974348068237,
+            0.0196879580616951,
+            -0.025395406410098076,
+            -0.014446982182562351,
+            0.01989375427365303,
+            0.022857237607240677,
+            -0.006684987805783749,
+            0.02421550080180168,
+            0.0057280296459794044,
+            0.015036934986710548,
+            -0.012485045939683914,
+            -0.0071823312900960445,
+            -0.03081474080681801,
+            -0.013610072433948517,
+            0.007456728257238865,
+            0.005559961311519146,
+            0.011600117199122906,
+            0.003968460951000452,
+            0.009933157823979855,
+            -0.007676245644688606,
+            -0.02711038663983345,
+            0.027247583493590355,
+            0.018178775906562805,
+            0.022226126864552498,
+            -0.020716944709420204,
+            0.0006019576103426516,
+            -0.006770736537873745,
+            0.013932487927377224,
+            0.0048465305007994175,
+            0.0007541619706898928,
+            -0.02182825095951557,
+            0.035452041774988174,
+            -0.02979947254061699,
+            -0.016861671581864357,
+            -0.0007215773803181946,
+            -0.01789066009223461,
+            0.01664215512573719,
+            -0.013616932556033134,
+            -0.029415316879749298,
+            0.016573555767536163,
+            0.015544568188488483,
+            0.015311331488192081,
+            0.0035500063095241785,
+            0.015846405178308487,
+            -0.012416446581482887,
+            0.001987660536542535,
+            -0.022774919867515564,
+            -0.004122809041291475,
+            0.009507843293249607,
+            -0.025820722803473473,
+            -0.024078303948044777,
+            0.004119379445910454,
+            0.010502530261874199,
+            -0.017972977831959724,
+            0.039595432579517365,
+            0.011531517840921879,
+            -0.008828711695969105,
+            0.005649140570312738,
+            0.0005252122646197677,
+            -0.006383151281625032,
+            0.04700414091348648,
+            0.027014346793293953,
+            -0.022212406620383263,
+            0.013445434160530567,
+            -0.004071359988301992,
+            0.012828041799366474,
+            -0.004894549492746592,
+            0.017108628526329994,
+            -0.02012699283659458,
+            0.009850838221609592,
+            -0.011353160254657269,
+            0.006084744818508625,
+            0.007936921902000904,
+            -0.023872505873441696,
+            -0.02660275250673294,
+            -0.036467310041189194,
+            0.008965909481048584,
+            -0.023749027401208878,
+            -0.01312301866710186,
+            0.028619566932320595,
+            -0.011147363111376762,
+            -0.03161048889160156,
+            0.027700338512659073,
+            0.00910996738821268,
+            -0.014014807529747486,
+            0.024352699518203735,
+            0.019962353631854057,
+            0.0069628143683075905,
+            0.0067124273627996445,
+            -0.01048195082694292,
+            -0.013534612953662872,
+            0.0013625507708638906,
+            0.004215417895466089,
+            0.00524097541347146,
+            -0.029387878254055977,
+            -0.009905718266963959,
+            0.0016369473887607455,
+            0.01897452585399151,
+            0.00673300726339221,
+            0.01124340109527111,
+            -0.006029865704476833,
+            0.002486719284206629,
+            -6.742010737070814e-05,
+            0.010296733118593693,
+            -0.004287447314709425,
+            -0.0019259213004261255,
+            0.0127800228074193,
+            0.028647007420659065,
+            -0.024627096951007843,
+            0.008629773743450642,
+            -0.016655875369906425,
+            -0.015393650159239769,
+            0.003227590350434184,
+            0.007038273382931948,
+            0.0018093027174472809,
+            -0.007456728257238865,
+            -0.00771740498021245,
+            0.006897645071148872,
+            -0.008451416157186031,
+            0.0019173463806509972,
+            -0.02012699283659458,
+            0.00431831693276763,
+            0.027782658115029335,
+            0.007292089983820915,
+            -0.034491654485464096,
+            -0.0019430710235610604,
+            -0.015009495429694653,
+            -0.011895093135535717,
+            -0.009041368961334229,
+            0.018617810681462288,
+            0.01631287857890129,
+            -0.013562052510678768,
+            0.0067913164384663105,
+            0.002584473229944706,
+            0.027933575212955475,
+            -0.011483498848974705,
+            -0.007278370205312967,
+            0.01923520304262638,
+            -0.00697310408577323,
+            0.002812565304338932,
+            0.027810096740722656,
+            -0.003769523464143276,
+            -0.017040029168128967,
+            0.0008077550446614623,
+            0.00961760152131319,
+            0.0084102563560009,
+            -0.01798669807612896,
+            0.010344752110540867,
+            0.0020116702653467655,
+            -0.006071025040000677,
+            0.03421725705265999,
+            0.059544067829847336,
+            -0.0027542561292648315,
+            0.0035002718213945627,
+            0.04563215747475624,
+            -0.02034650929272175,
+            -0.027028067037463188,
+            -0.01006349641829729,
+            -0.014460701495409012,
+            0.012011712417006493,
+            0.011723595671355724,
+            -0.005168946459889412,
+            0.01336311548948288,
+            -4.842135604121722e-05,
+            0.030704982578754425,
+            -0.01819249615073204,
+            0.03888200223445892,
+            0.010852386243641376,
+            0.011312000453472137,
+            0.015160413458943367,
+            0.020030952990055084,
+            0.008540594950318336,
+            -0.0007533044554293156,
+            0.006180783733725548,
+            -0.014337223023176193,
+            0.006321412045508623,
+            -0.012800602242350578,
+            -0.012526205740869045,
+            0.014076546765863895,
+            0.0013908478431403637,
+            0.02797473594546318,
+            -0.007463587913662195,
+            -0.005103777162730694,
+            -0.014090266078710556,
+            -0.00472648162394762,
+            0.0034351027570664883,
+            0.012251809239387512,
+            -0.00010434574505779892,
+            -0.007648805622011423,
+            -0.025971639901399612,
+            0.00926088634878397,
+            -3.971247861045413e-05,
+            -0.014117705635726452,
+            0.0012853766093030572,
+            -0.0011653281981125474,
+            -0.014625339768826962,
+            0.026355795562267303,
+            -0.004942568950355053,
+            0.0037386538460850716,
+            0.011517798528075218,
+            0.0009732504840940237,
+            -0.0034848370123654604,
+            0.0023358012549579144,
+            0.011346300132572651,
+            -0.00023559521650895476,
+            -0.015640607103705406,
+            -0.004654452670365572,
+            -0.009226586669683456,
+            0.005529091693460941,
+            -0.006506629753857851,
+            0.009302045218646526,
+            0.003512276802212,
+            0.006307692267000675,
+            -0.01509181410074234,
+            -0.016573555767536163,
+            -0.012128330767154694,
+            -0.006784456316381693,
+            0.011620696634054184,
+            0.0010504245292395353,
+            -0.002776550827547908,
+            -0.004356046207249165,
+            -0.0005886664730496705,
+            -0.024256661534309387,
+            -0.007573346607387066,
+            0.003195005701854825,
+            -0.004441795405000448,
+            0.00019036265439353883,
+            0.003772953525185585,
+            -0.020264189690351486,
+            0.033064793795347214,
+            0.014090266078710556,
+            -0.008965909481048584,
+            0.006787886377424002,
+            -0.003872422268614173,
+            -0.00663010822609067,
+            -0.028619566932320595,
+            -0.0017681432655081153,
+            -0.01870013028383255,
+            0.019976073876023293,
+            0.025793282315135002,
+            -0.0227886401116848,
+            0.0012124900240451097,
+            -0.009013929404318333,
+            0.011456059291958809,
+            -0.0024747145362198353,
+            0.0008373384480364621,
+            0.028921402990818024,
+            -0.01679307222366333,
+            -0.019303802400827408,
+            -0.02350207045674324,
+            0.003423097776249051,
+            0.004486384801566601,
+            0.024887774139642715,
+            -0.0014148575719445944,
+            0.00871209241449833,
+            -0.011360020376741886,
+            -0.0012416447279974818,
+            0.016806792467832565,
+            0.01867268979549408,
+            0.021608734503388405,
+            -0.009254026226699352,
+            -0.0022003178019076586,
+            -0.011037603951990604,
+            -0.0013711255742236972,
+            0.026739951223134995,
+            0.003776383586227894,
+            -0.02540912665426731,
+            -0.008808131329715252,
+            -0.007051993161439896,
+            -0.005374743603169918,
+            0.004510394297540188,
+            -0.0026324924547225237,
+            0.002447274746373296,
+            0.001353118335828185,
+            -0.012210649438202381,
+            0.020442547276616096,
+            0.0015186137752607465,
+            0.0048396703787148,
+            -0.010728907771408558,
+            -0.013774710707366467,
+            -0.026657631620764732,
+            -0.01006349641829729,
+            -0.028647007420659065,
+            -0.003505416912958026,
+            0.0070657129399478436,
+            -0.005841217935085297,
+            0.0009269460570067167,
+            0.013959928415715694,
+            -0.0020408248528838158,
+            -0.005848078057169914,
+            0.0199211947619915,
+            -0.013095579110085964,
+            -0.010310453362762928,
+            -0.008197599090635777,
+            -0.017726020887494087,
+            0.005971556529402733,
+            -0.010296733118593693,
+            0.0003800822014454752,
+            -0.005144936498254538,
+            -0.0018264524405822158,
+            -0.04769013077020645,
+            -0.006715857423841953,
+            -0.014995775185525417,
+            -0.023364871740341187,
+            -0.006691847462207079,
+            0.018357133492827415,
+            0.007038273382931948,
+            -0.018206216394901276,
+            -0.008732672780752182,
+            -0.004747061524540186,
+            8.54139852890512e-06,
+            -0.03391542285680771,
+            0.023543229326605797,
+            0.0018213075818493962,
+            0.011922532692551613,
+            -0.012368427589535713,
+            0.012738863006234169,
+            0.007100012619048357,
+            0.020716944709420204,
+            -0.007326389662921429,
+            -0.01730070635676384,
+            -0.01920776255428791,
+            -0.0009861128637567163,
+            -0.010255573317408562,
+            -0.024325260892510414,
+            -0.01054369006305933,
+            0.0012836616951972246,
+            0.004342326428741217,
+            0.006846195552498102,
+            0.010818086564540863,
+            0.02558748424053192,
+            -0.012491906061768532,
+            0.002020245185121894,
+            0.019797716289758682,
+            0.002472999505698681,
+            0.02210264839231968,
+            0.01890592649579048,
+            0.007278370205312967,
+            -0.001317103742621839,
+            0.002605052897706628,
+            -0.0033390638418495655,
+            -0.01583268493413925,
+            0.007292089983820915,
+            -0.00865035317838192,
+            -0.03320199251174927,
+            0.025107290595769882,
+            0.027069225907325745,
+            0.010756347328424454,
+            -0.006897645071148872,
+            0.004397206008434296,
+            0.0056354207918047905,
+            0.008190738968551159,
+            0.02952507697045803,
+            -0.017383025959134102,
+            0.00253473874181509,
+            0.00788890291005373,
+            -0.011017024517059326,
+            -0.008321077562868595,
+            0.018933366984128952,
+            0.033064793795347214,
+            -0.005837787874042988,
+            -0.039924707263708115,
+            -0.00871209241449833,
+            0.016271719709038734,
+            -0.01406282652169466,
+            -0.015105534344911575,
+            -0.007552767172455788,
+            0.004613293334841728,
+            -0.020113272592425346,
+            0.0019619357772171497,
+            -0.003622035263106227,
+            -0.0028434349223971367,
+            0.008808131329715252,
+            0.028537249192595482,
+            0.003776383586227894,
+            0.02652043290436268,
+            -0.011805914342403412,
+            0.008019241504371166,
+            -0.026561593636870384,
+            0.011517798528075218,
+            -0.02073066495358944,
+            -0.021375495940446854,
+            0.006091604940593243,
+            0.0019173463806509972,
+            0.02807077392935753,
+            0.010516250506043434,
+            0.01923520304262638,
+            0.0023975404910743237,
+            0.00319843553006649,
+            -0.03358614444732666,
+            -0.022061487659811974,
+            -0.0015906429616734385,
+            -0.0195919182151556,
+            0.006890784949064255,
+            -0.006757016759365797,
+            0.014446982182562351,
+            0.016889112070202827,
+            0.015215292572975159,
+            -0.008760112337768078,
+            0.019276361912488937,
+            -2.2576088667847216e-05,
+            0.01712234877049923,
+            -0.02416062168776989,
+            0.00414681900292635,
+            0.02147153578698635,
+            -0.0009320910321548581,
+            0.017547663301229477,
+            0.0003359214751981199,
+            -0.0007018551114015281,
+            0.0069868238642811775,
+            -0.0046201529912650585,
+            -0.012903501279652119,
+            -0.006444890517741442,
+            0.023008156567811966,
+            -0.03163792937994003,
+            0.0110650435090065,
+            0.009020788595080376,
+            -0.006695277523249388,
+            -0.014954615384340286,
+            -0.007271510548889637,
+            0.03737282007932663,
+            0.004040490370243788,
+            -0.00011104487930424511,
+            0.014433261938393116,
+            0.019221482798457146,
+            0.007984941825270653,
+            0.012382147833704948,
+            -0.017081189900636673,
+            0.020854143425822258,
+            0.04041862115263939,
+            -0.007319530006498098,
+            -0.0038689924404025078,
+            0.007669385522603989,
+            0.008821851573884487,
+            -0.01037905178964138,
+            0.008444556035101414,
+            -0.004006190691143274,
+            0.002958338474854827,
+            0.009226586669683456,
+            -0.022212406620383263,
+            0.026438115164637566,
+            -0.022870957851409912,
+            -0.0019756555557250977,
+            -0.004510394297540188,
+            1.4858684153296053e-05,
+            0.004795080982148647,
+            -3.577338793547824e-05,
+            0.02022303082048893,
+            -0.003222445258870721,
+            -0.02197916992008686,
+            -0.006592378951609135,
+            -0.00663010822609067,
+            0.00737440912052989,
+            -0.02019559033215046,
+            0.0004523256793618202,
+            0.024352699518203735,
+            0.005223825573921204,
+            0.0037420839071273804,
+            -0.014001087285578251,
+            -0.00024116890563163906,
+            -0.0035534361377358437,
+            0.021348057314753532,
+            0.01676563359797001,
+            0.024942653253674507,
+            0.02639695443212986,
+            0.003330488922074437,
+            0.012416446581482887,
+            -0.007223491091281176,
+            0.003690634621307254,
+            -0.0243389792740345,
+            0.025985360145568848,
+            -0.011538377963006496,
+            -0.010468230582773685,
+            -0.003851842600852251,
+            -0.010056636296212673,
+            -0.010310453362762928,
+            0.004647592548280954,
+            0.01394620817154646,
+            0.010276153683662415,
+            0.026177437976002693,
+            -0.023762747645378113,
+            0.013342536054551601,
+            0.012135190889239311,
+            -0.018658969551324844,
+            0.025491446256637573,
+            -0.012423306703567505,
+            -0.006366001442074776,
+            -0.0016335173277184367,
+            -0.005961266346275806,
+            0.010811226442456245,
+            -0.021787092089653015,
+            0.009061948396265507,
+            -0.026136277243494987,
+            -0.006064165383577347,
+            -0.003941021393984556,
+            0.0014946040464565158,
+            -0.0014380097854882479,
+            -0.0005633705877698958,
+            -0.028592128306627274,
+            0.0006589806289412081,
+            0.008197599090635777,
+            5.399503788794391e-05,
+            0.014995775185525417,
+            -0.011085623875260353,
+            -0.008423976600170135,
+            -0.0052135358564555645,
+            -0.006201363634318113,
+            -0.008327937684953213,
+            0.03229648247361183,
+            0.003476262092590332,
+            -0.01122968178242445,
+            0.003707784228026867,
+            0.0034248128067702055,
+            -0.030979378148913383,
+            0.006472330074757338,
+            0.008259338326752186,
+            -0.017904380336403847,
+            0.020387668162584305,
+            0.002598193008452654,
+            0.015997322276234627,
+            -0.01813761703670025,
+            0.007957502268254757,
+            -0.017561383545398712,
+            -0.012052871286869049,
+            8.306928975798655e-07,
+            -0.019290082156658173,
+            0.012464466504752636,
+            -0.02352951094508171,
+            0.022130087018013,
+            -0.02070322446525097,
+            -0.003759233746677637,
+            -0.02344719134271145,
+            0.02055230736732483,
+            -0.0018779018428176641,
+            0.007504747714847326,
+            0.013500313274562359,
+            0.0017972978530451655,
+            0.004441795405000448,
+            0.016600996255874634,
+            0.0035431464202702045,
+            0.014515581540763378,
+            0.01798669807612896,
+            0.02233588509261608,
+            0.006811895873397589,
+            0.0033184841740876436,
+            -0.020319068804383278,
+            0.004229137673974037,
+            -0.007346969563513994,
+            0.05205303803086281,
+            0.005710879806429148,
+            -0.007984941825270653,
+            0.00466817244887352,
+            0.0035362865310162306,
+            0.013445434160530567,
+            0.0038141130935400724,
+            0.02129317820072174,
+            -0.016985150054097176,
+            0.014337223023176193,
+            -0.020826702937483788,
+            0.00656150933355093,
+            0.017643703147768974,
+            0.03193976730108261,
+            0.009391224011778831,
+            -0.012505626305937767,
+            -0.020140711218118668,
+            0.011353160254657269,
+            -0.008273057639598846,
+            0.00014716661826241761,
+            -0.018562931567430496,
+            0.007244070526212454,
+            -0.007744844537228346,
+            0.011593257077038288,
+            0.002666792133823037,
+            0.02317279390990734,
+            0.012279248796403408,
+            -0.013212197460234165,
+            0.00680160615593195,
+            -0.009507843293249607,
+            -0.010008616372942924,
+            0.006297402549535036,
+            -0.03462885320186615,
+            -0.02225356549024582,
+            0.009809679351747036,
+            0.030430585145950317,
+            -0.0020459697116166353,
+            -0.03144585341215134,
+            -0.0038758523296564817,
+            -0.015242732129991055,
+            -0.025971639901399612,
+            0.017780901864171028,
+            0.02952507697045803,
+            -0.011154222302138805,
+            0.0038278328720480204,
+            -0.00014041388931218535,
+            0.009603881277143955,
+            0.008135859854519367,
+            0.021348057314753532,
+            -0.008177019655704498,
+            -0.009624461643397808,
+            0.007244070526212454,
+            0.024242941290140152,
+            -0.00978223979473114,
+            0.022006608545780182,
+            -0.014103986322879791,
+            0.009597022086381912,
+            -0.018302254378795624,
+            -0.00991943757981062,
+            -0.01559944823384285,
+            -6.082815525587648e-05,
+            -0.0032567449379712343,
+            0.003855272429063916,
+            -0.00950098317116499,
+            -0.0010512820445001125,
+            0.01026243343949318,
+            0.003989040851593018,
+            -0.004352616146206856,
+            0.00851315539330244,
+            0.005234115291386843,
+            0.01666959375143051,
+            0.003155561164021492,
+            -0.0362752340734005,
+            0.005402183625847101,
+            0.004050780087709427,
+            -0.0024284101091325283,
+            0.01572292670607567,
+            0.02705550752580166,
+            0.00799180194735527,
+            0.014282343909144402,
+            0.015311331488192081,
+            0.018000418320298195,
+            0.014296064153313637,
+            -0.010420211590826511,
+            0.0020939891692250967,
+            0.016203120350837708,
+            -0.0017338435864076018,
+            -0.010660308413207531,
+            -0.004808800760656595,
+            -0.024229221045970917,
+            0.030320826917886734,
+            0.013678671792149544,
+            -0.008506295271217823,
+            -0.027727779000997543,
+            -0.00933634489774704,
+            0.020620906725525856,
+            0.002779980655759573,
+            -0.011462918482720852,
+            -0.006952524185180664,
+            -0.007216630969196558,
+            0.018947087228298187,
+            0.0019070565467700362,
+            0.0015417659888044,
+            -0.005203245673328638,
+            -0.007957502268254757,
+            0.0195919182151556,
+            -0.018425732851028442,
+            -0.006050445605069399,
+            -0.0074224285781383514,
+            0.00836909655481577,
+            0.00043989208643324673,
+            0.03904663771390915,
+            0.009425523690879345,
+            0.030650101602077484,
+            0.0009218011400662363,
+            -0.018466891720891,
+            0.007477307692170143,
+            0.008115280419588089,
+            0.00893160980194807,
+            0.0305403433740139,
+            -0.017904380336403847,
+            0.003666624892503023,
+            0.013534612953662872,
+            0.0007691680220887065,
+            0.0005582256126217544,
+            0.0005740891792811453,
+            0.004640732891857624,
+            -0.004657882731407881,
+            -0.015105534344911575,
+            -0.0008634918485768139,
+            -0.01780834048986435,
+            0.016011042520403862,
+            0.012491906061768532,
+            0.006760446820408106,
+            -0.01129142101854086,
+            0.003366503631696105,
+            0.0092334458604455,
+            -0.015544568188488483,
+            -0.00865035317838192,
+            0.007854603230953217,
+            -0.01216948963701725,
+            0.017437905073165894,
+            -0.00021179989562369883,
+            -0.018000418320298195,
+            -0.002262057038024068,
+            0.007635085843503475,
+            0.005974986590445042,
+            0.008087839931249619,
+            -0.02281607873737812,
+            -0.016600996255874634,
+            -0.0032207302283495665,
+            -0.02385878562927246,
+            -0.02764545939862728,
+            0.0025827581994235516,
+            0.009535282850265503,
+            -0.007518467493355274,
+            -0.00851315539330244,
+            0.01920776255428791,
+            0.011888233944773674,
+            0.023913664743304253,
+            -0.024640817195177078,
+            0.019468439742922783,
+            -0.00032477412605658174,
+            -0.016189400106668472,
+            0.004585853312164545,
+            -0.025285648182034492,
+            0.02436641976237297,
+            0.0011018739314749837,
+            0.00720291119068861,
+            -0.0031212614849209785,
+            -0.006482620257884264,
+            0.002944618696346879,
+            -0.007120592053979635,
+            0.009974316693842411,
+            0.0017921528778970242,
+            -0.020634625107049942,
+            -0.0010975864715874195,
+            -0.007285230327397585,
+            -0.009315765462815762,
+            0.011819634586572647,
+            -0.027631739154458046,
+            -0.0007005688967183232,
+            -0.00535416416823864,
+            -0.0031932906713336706,
+            -0.003143556183204055,
+            0.017136069014668465,
+            -0.022555401548743248,
+            0.035863637924194336,
+            0.008225038647651672,
+            0.012629104778170586,
+            -0.00316928094252944,
+            0.01607964187860489,
+            0.008938469924032688,
+            -0.0029497635550796986,
+            -0.0013676956295967102,
+            -0.007470448035746813,
+            -0.0061670639552176,
+            0.011545238085091114,
+            0.0025467437226325274,
+            -0.012457606382668018,
+            0.009912577457726002,
+            -0.0038038231432437897,
+            -0.02078554406762123,
+            -0.008773831650614738,
+            0.011799054220318794,
+            0.010975864715874195,
+            -0.002409545239061117,
+            0.005546241533011198,
+            0.0067364368587732315,
+            0.010111515410244465,
+            -0.004966578911989927,
+            -0.001727841212414205,
+            0.004486384801566601,
+            -0.017396746203303337,
+            -0.006407161243259907,
+            -0.014968335628509521,
+            -0.009631321765482426,
+            0.020799264311790466,
+            0.01849433220922947,
+            0.007497887592762709,
+            5.8094909036299214e-05,
+            -0.0019001965411007404,
+            -0.009116827510297298,
+            -0.0002758972404990345,
+            0.011826494708657265,
+            -0.02848236821591854,
+            0.019825154915452003,
+            0.01789066009223461,
+            0.022870957851409912,
+            -0.028180532157421112,
+            -0.007429288700222969,
+            0.0007117162458598614,
+            -0.007003973703831434,
+            -0.017314426600933075,
+            -0.0033973732497543097,
+            -0.008849291130900383,
+            0.014501861296594143,
+            -0.02004467323422432,
+            0.01509181410074234,
+            0.0022894968278706074,
+            0.0016755344113335013,
+            -0.0011944827856495976,
+            0.010838666930794716,
+            0.007175471633672714,
+            -0.0037180741783231497,
+            -0.002824570285156369,
+            -0.018274815753102303,
+            -0.007971221581101418,
+            0.02182825095951557,
+            0.002627347595989704,
+            -0.023433471098542213,
+            0.002495294203981757,
+            0.004558413755148649,
+            -0.025875601917505264,
+            0.006925084628164768,
+            -0.037564896047115326,
+            -0.009432383812963963,
+            -0.015407370403409004,
+            0.0019790856167674065,
+            -0.02009955234825611,
+            0.006221943534910679,
+            0.001680679270066321,
+            0.010132094845175743,
+            0.003201865591108799,
+            -0.007971221581101418,
+            -0.00945296324789524,
+            0.002495294203981757,
+            -0.004530974198132753,
+            0.0019756555557250977,
+            0.01044079102575779,
+            0.005916676949709654,
+            -0.02842748910188675,
+            -0.020538587123155594,
+            0.005388463847339153,
+            -0.014639060012996197,
+            -0.0007507320260629058,
+            -0.005446773022413254,
+            0.00019089858687948436,
+            -0.00044417951721698046,
+            -0.004589283373206854,
+            0.007587066385895014,
+            0.014625339768826962,
+            -0.00019818724831566215,
+            0.009754800237715244,
+            0.005326724145561457,
+            -0.0005595118855126202,
+            0.020497428253293037,
+            -9.680251423560549e-06,
+            0.0218968503177166,
+            0.024270381778478622,
+            -0.013342536054551601,
+            0.02314535528421402,
+            -0.009933157823979855,
+            0.013699251227080822,
+            -0.008698373101651669,
+            0.017040029168128967,
+            -0.0031006818171590567,
+            0.012121470645070076,
+            -0.006451750639826059,
+            -0.011757895350456238,
+            -0.006077885162085295,
+            0.009686200879514217,
+            -0.0006636968464590609,
+            -0.010193834081292152,
+            0.019962353631854057,
+            -0.014789978042244911,
+            0.011558957397937775,
+            -0.0038827122189104557,
+            0.001445727190002799,
+            0.0020442549139261246,
+            -0.0004028056573588401,
+            0.03125377371907234,
+            0.00919914711266756,
+            -0.00639001140370965,
+            -0.017822060734033585,
+            0.006846195552498102,
+            -0.029387878254055977,
+            -0.011737314984202385,
+            -0.004115949384868145,
+            -0.01474881824105978,
+            -0.0022963567171245813,
+            -0.0019979504868388176,
+            0.031363531947135925,
+            -0.005192955955862999,
+            -0.020675785839557648,
+            -0.005196386016905308,
+            -0.01664215512573719,
+            -0.00736754946410656,
+            -0.00799180194735527,
+            -0.0062528131529688835,
+            0.003152131102979183,
+            -8.617768617114052e-05,
+            -0.023378591984510422,
+            0.009116827510297298,
+            -0.0047127618454396725,
+            0.017437905073165894,
+            -0.001723553752526641,
+            0.0224593635648489,
+            -0.020154431462287903,
+            0.011888233944773674,
+            -0.009610741399228573,
+            0.002551888581365347,
+            0.018439453095197678,
+            0.0027748357970267534,
+            -0.0068153259344398975,
+            0.0040953694842755795,
+            0.015695486217737198,
+            -0.015517128631472588,
+            -0.0025467437226325274,
+            -0.011716735549271107,
+            0.02121085859835148,
+            3.5398235922912136e-05,
+            -0.007779144216328859,
+            -0.015050654299557209,
+            -0.02359810844063759,
+            0.016065921634435654,
+            -0.006321412045508623,
+            0.014200025238096714,
+            0.01181277446448803,
+            0.02362554892897606,
+            0.0022243275307118893,
+            -8.730314584681764e-05,
+            0.009103108197450638,
+            -0.007758564315736294,
+            0.008863010443747044,
+            0.021183418110013008,
+            0.0035362865310162306,
+            -0.005745179019868374,
+            0.006293972488492727,
+            -0.0014320074114948511,
+            0.02261028066277504,
+            -0.018562931567430496,
+            0.001973940758034587,
+            0.014762538485229015,
+            0.0020511148031800985,
+            -0.010639728978276253,
+            0.015942443162202835,
+            -0.0013462584465742111,
+            -0.0037180741783231497,
+            -0.01750650443136692,
+            0.0031349812634289265,
+            -0.00891789048910141,
+            0.009144267067313194,
+            -0.01352775376290083,
+            0.010015476495027542,
+            -0.02258284203708172,
+            0.007401848677545786,
+            0.00016388767107855529,
+            -0.002584473229944706,
+            0.02621859684586525,
+            -0.006465470418334007,
+            0.0009226586553268135,
+            0.030403144657611847,
+            -0.0035843057557940483,
+            -0.01966051757335663,
+            0.00133854104205966,
+            -0.0032790396362543106,
+            -0.004242857918143272,
+            0.00691136484965682,
+            0.0015572007978335023,
+            -0.016573555767536163,
+            -0.005491362418979406,
+            0.014213744550943375,
+            0.02565608359873295,
+            0.0015229012351483107,
+            -0.022377043962478638,
+            0.002049399772658944,
+            0.01572292670607567,
+            -0.0029617685358971357,
+            -0.0002458851085975766,
+            0.005820638500154018,
+            0.009789098985493183,
+            0.0006461183074861765,
+            -0.006595808546990156,
+            0.010866106487810612,
+            -0.02797473594546318,
+            -0.008554314263164997,
+            -0.01640891842544079,
+            -0.008760112337768078,
+            -0.04157108813524246,
+            0.024435019120573997,
+            0.00459957355633378,
+            0.003469402203336358,
+            0.008965909481048584,
+            0.005529091693460941,
+            -0.004009620752185583,
+            -0.018933366984128952,
+            -0.002056259661912918,
+            -0.006376291625201702,
+            0.01130514033138752,
+            -0.008828711695969105,
+            -0.007957502268254757,
+            0.005559961311519146,
+            0.0029754883144050837,
+            -0.005275275092571974,
+            0.024092022329568863,
+            -0.001313673797994852,
+            0.0031075417064130306,
+            0.0038106830324977636,
+            -0.004157108720391989,
+            -0.0013128162827342749,
+            0.015640607103705406,
+            0.014090266078710556,
+            -4.6465207560686395e-05,
+            0.009672480635344982,
+            0.008259338326752186,
+            0.001888191793113947,
+            -0.006403731182217598,
+            -0.009391224011778831,
+            0.007024553604424,
+            0.018206216394901276,
+            -0.0035465762484818697,
+            -0.004294306971132755,
+            0.019605638459324837,
+            -0.014529300853610039,
+            -0.0008008951554074883,
+            0.012642824091017246,
+            -0.015201572328805923,
+            0.017904380336403847,
+            0.008972769603133202,
+            0.011538377963006496,
+            -0.010509390383958817,
+            -0.04598887264728546,
+            0.004139958880841732,
+            -0.00737440912052989,
+            0.009727359749376774,
+            0.0072372108697891235,
+            0.02248680219054222,
+            -0.013486593961715698,
+            -0.03534228354692459,
+            0.004805370699614286,
+            0.00216258829459548,
+            0.020442547276616096,
+            0.008122139610350132,
+            0.018809888511896133,
+            0.008629773743450642,
+            0.01559944823384285,
+            -0.006674697622656822,
+            0.009638180956244469,
+            0.022857237607240677,
+            -0.011997992172837257,
+            -0.014213744550943375,
+            -0.004242857918143272,
+            0.0022792068775743246,
+            -0.005628560669720173,
+            -0.009768519550561905,
+            0.001305956393480301,
+            -0.013486593961715698,
+            0.0048465305007994175,
+            0.01422746479511261,
+            0.015078093856573105,
+            0.010502530261874199,
+            0.006492909975349903,
+            0.0033390638418495655,
+            -0.003954741172492504,
+            0.0006765591679140925,
+            0.006506629753857851,
+            0.018535491079092026,
+            -0.013877608813345432,
+            0.026163717731833458,
+            -0.0008446270949207246,
+            0.013074998743832111,
+            0.005621700547635555,
+            0.003694064449518919,
+            0.005162086337804794,
+            0.008698373101651669,
+            0.0032258753199130297,
+            0.013555193319916725,
+            -0.004122809041291475,
+            0.0014294348657131195,
+            0.0016095077153295279,
+            -0.008780691772699356,
+            0.007882042787969112,
+            -0.020113272592425346,
+            0.011757895350456238,
+            0.009528422728180885,
+            0.00034042331390082836,
+            0.010248714126646519,
+            0.018755009397864342,
+            0.015352491289377213,
+            -0.031390972435474396,
+            -0.0072372108697891235,
+            -0.007504747714847326,
+            -0.002130003646016121,
+            0.01676563359797001,
+            0.002109423978254199,
+            0.014076546765863895,
+            0.0424765981733799,
+            0.006129334680736065,
+            0.02335115149617195,
+            0.0017852929886430502,
+            -0.019317522644996643,
+            -0.00250558415427804,
+            0.0039821807295084,
+            -0.02147153578698635,
+            0.0016832518158480525,
+            -0.002937758807092905,
+            -0.01598360389471054,
+            -0.0028897393494844437,
+            -0.0028039903845638037,
+            0.0029960679821670055,
+            -0.00598870636895299,
+            -0.002406115410849452,
+            0.0039238715544342995,
+            0.0035431464202702045,
+            0.022088928148150444,
+            -0.019098004326224327,
+            0.0013333960669115186,
+            -0.006125904619693756,
+            0.006136194337159395,
+            -0.015517128631472588,
+            -0.010015476495027542,
+            -0.013857029378414154,
+            0.02944275736808777,
+            -0.021718492731451988,
+            -0.009549002163112164,
+            0.013006399385631084,
+            -0.012423306703567505,
+            -0.006582088768482208,
+            0.0013899903278797865,
+            -0.002208892721682787,
+            -0.026479274034500122,
+            -0.0006332559278234839,
+            -0.005144936498254538,
+            -0.015325050801038742,
+            -0.01343857403844595,
+            -0.04494616761803627,
+            0.021691052243113518,
+            0.004105659201741219,
+            -0.007854603230953217,
+            -0.0023495210334658623,
+            -0.020058393478393555,
+            0.014501861296594143,
+            0.0007142886752262712,
+            0.01920776255428791,
+            -0.011490358039736748,
+            -0.014350943267345428,
+            -0.004047350026667118,
+            0.009946877136826515,
+            -0.020291630178689957,
+            0.02034650929272175,
+            0.002613627817481756,
+            -0.024860333651304245,
+            -0.012608524411916733,
+            0.0040953694842755795,
+            -0.0020065251737833023,
+            -0.008389676921069622,
+            0.0038758523296564817,
+            -0.0040953694842755795,
+            0.00276283104903996,
+            -0.0058446479961276054,
+            0.014268624596297741,
+            -0.02293955720961094,
+            0.021608734503388405,
+            0.0013436859007924795,
+            0.0006739866803400218,
+            0.014529300853610039,
+            0.014872296713292599,
+            0.016203120350837708,
+            -0.014680218882858753,
+            -0.007593926507979631,
+            0.02009955234825611,
+            0.035808760672807693,
+            -0.009686200879514217,
+            0.007223491091281176,
+            0.011785334907472134,
+            0.02591676078736782,
+            0.011504078283905983,
+            -0.002399255521595478,
+            -0.015791526064276695,
+            0.014803697355091572,
+            0.004664742387831211,
+            0.004551554098725319,
+            -0.008540594950318336,
+            -0.031006818637251854,
+            0.0025930481497198343,
+            -0.008238758891820908,
+            -0.023419750854372978,
+            -0.0031281213741749525,
+            0.011023884639143944,
+            0.021787092089653015,
+            0.008780691772699356,
+            -0.004867109935730696,
+            0.01440582238137722,
+            -0.01849433220922947,
+            0.00805354118347168,
+            0.010392772033810616,
+            -0.011010164394974709,
+            0.014042247086763382,
+            -0.02675366960465908,
+            -0.042696114629507065,
+            0.0023718157317489386,
+            0.006684987805783749,
+            0.01257422473281622,
+            0.005717739462852478,
+            0.0070794327184557915,
+            -0.006441460456699133,
+            -0.008375956676900387,
+            -0.006136194337159395,
+            0.0023718157317489386,
+            -0.009082527831196785,
+            0.018384573981165886,
+            0.005436482839286327,
+            -0.008074120618402958,
+            0.0014217174611985683,
+            0.017945539206266403,
+            -0.007648805622011423,
+            0.009247166104614735,
+            0.008856151252985,
+            0.015393650159239769,
+            0.017726020887494087,
+            -0.008595474064350128,
+            0.038580164313316345,
+            0.010934704914689064,
+            -0.014543021097779274,
+            0.00045275440788827837,
+            -0.0044486550614237785,
+            0.030622662976384163,
+            -0.0003732222830876708,
+            -0.01583268493413925,
+            0.023927384987473488,
+            0.002188313053920865,
+            -0.00013923484948463738,
+            -0.014721378684043884,
+            0.0032498850487172604,
+            -0.006475760135799646,
+            0.01643635705113411,
+            -0.009192286990582943,
+            0.001149893389083445,
+            -0.020291630178689957,
+            -0.005611410830169916,
+            0.0004626155423466116,
+            -0.008897310122847557,
+            0.008677792735397816,
+            -0.0017750031547620893,
+            0.0038998620584607124,
+            -0.013459154404699802,
+            -0.018892208114266396,
+            0.0027079517021775246,
+            0.008938469924032688,
+            0.02012699283659458,
+            0.015105534344911575,
+            0.0004733341629616916,
+            -0.011881373822689056,
+            -0.00945296324789524,
+            -0.0035431464202702045,
+            0.005179236177355051,
+            0.0023049316368997097,
+            -0.01941356062889099,
+            -0.012004852294921875,
+            0.010132094845175743,
+            -0.0005805203691124916,
+            -0.003659765003249049,
+            0.00720291119068861,
+            -0.007209771312773228,
+            -0.00598870636895299,
+            0.009926297701895237,
+            -0.0018161626067012548,
+            0.0035105617716908455,
+            -0.006969674024730921,
+            -0.014268624596297741,
+            -0.00933634489774704,
+            0.003944451455026865,
+            -0.01664215512573719,
+            0.02797473594546318,
+            -0.00991943757981062,
+            0.0017518509412184358,
+            0.008876730687916279,
+            0.013171037659049034,
+            0.0017295562429353595,
+            0.02263772115111351,
+            -0.02446245774626732,
+            -0.006966243963688612,
+            0.0029926381539553404,
+            -0.022047769278287888,
+            -0.01780834048986435,
+            -0.017918098717927933,
+            -0.04151620715856552,
+            0.012656544335186481,
+            0.026561593636870384,
+            -0.01646379753947258,
+            0.02979947254061699,
+            -0.0002896170481108129,
+            0.005837787874042988,
+            0.011222821660339832,
+            -0.008314217440783978,
+            -0.003522566519677639,
+            -0.017753461375832558,
+            -0.014014807529747486,
+            0.02004467323422432,
+            -0.005186096299439669,
+            0.005148366559296846,
+            -0.01124340109527111,
+            -0.009747940115630627,
+            -0.0050660474225878716,
+            -0.0002876876969821751,
+            0.01628543995320797,
+            0.015421089716255665,
+            -0.005734889302402735,
+            -0.012313548475503922,
+            -0.020154431462287903,
+            0.025189610198140144,
+            -0.0020854142494499683,
+            -0.009837118908762932,
+            0.009432383812963963,
+            0.02317279390990734,
+            -0.013925628736615181,
+            -0.00645861029624939,
+            0.003143556183204055,
+            0.005676580127328634,
+            -0.0029531936161220074,
+            0.018933366984128952,
+            -0.011483498848974705,
+            0.0018813317874446511,
+            -0.010420211590826511,
+            0.016710754483938217,
+            0.02042882889509201,
+            0.021046221256256104,
+            -0.019756557419896126,
+            0.014501861296594143,
+            -0.01157267764210701,
+            -0.0038244028110057116,
+            0.014076546765863895,
+            0.014789978042244911,
+            0.002035679994150996,
+            0.007346969563513994,
+            0.025779562070965767,
+            -0.007024553604424,
+            0.02090902253985405,
+            -0.011113063432276249,
+            -0.015338771045207977,
+            -0.014433261938393116,
+            -0.02573840320110321,
+            0.003189860610291362,
+            -0.006451750639826059,
+            -0.018151337280869484,
+            -0.007298950105905533,
+            -0.006969674024730921,
+            0.004476095084100962,
+            -0.004801940638571978,
+            -0.02352951094508171,
+            0.01223122887313366,
+            -0.0202504713088274,
+            0.02276119962334633,
+            -0.023282554000616074,
+            -0.0097959591075778,
+            -0.018069017678499222,
+            -0.011133642867207527,
+            -0.008327937684953213,
+            -0.008952190168201923,
+            0.0079643614590168,
+            -0.006678127683699131,
+            0.005656000226736069,
+            -0.005247835535556078,
+            0.0027045216411352158,
+            -0.005467352457344532,
+            0.009363784454762936,
+            0.019701676443219185,
+            0.02162245288491249,
+            0.00472648162394762,
+            0.016065921634435654,
+            0.015489689074456692,
+            0.006623248569667339,
+            -0.016546115279197693,
+            0.006753586698323488,
+            -0.0075802067294716835,
+            0.008115280419588089,
+            0.020085832104086876,
+            -0.010735767893493176,
+            -0.00633856188505888,
+            -0.009686200879514217,
+            -0.0017870080191642046,
+            0.02398226410150528,
+            -0.004692182410508394,
+            -0.002123143756762147,
+            -0.001323963631875813,
+            0.016450077295303345,
+            0.0029549086466431618,
+            0.020058393478393555,
+            0.0019893755670636892,
+            0.01509181410074234,
+            -0.007868323475122452,
+            -0.019221482798457146,
+            -0.014872296713292599,
+            -0.02070322446525097,
+            0.010084075853228569,
+            -0.00909624807536602,
+            -0.01804157719016075,
+            -0.014995775185525417,
+            0.001756138401106,
+            -0.005930396728217602,
+            7.590121185785392e-06,
+            0.005515371914952993,
+            0.007477307692170143,
+            0.00945296324789524,
+            0.014172585681080818,
+            0.001274229260161519,
+            -0.0008952189818955958,
+            0.023556949570775032,
+            -0.015338771045207977,
+            0.00530271464958787,
+            -0.010310453362762928,
+            -0.01129142101854086,
+            -0.008266198448836803,
+            0.0006778453825972974,
+            0.006849625613540411,
+            -0.028537249192595482,
+            0.0061739240773022175,
+            -0.02807077392935753,
+            0.004194838460534811,
+            0.0071343122981488705,
+            -0.001848747255280614,
+            -0.007792863994836807,
+            0.0005758041515946388,
+            -0.009267745539546013,
+            0.011517798528075218,
+            0.021252017468214035,
+            0.021814530715346336,
+            -0.002745681209489703,
+            -0.005234115291386843,
+            0.013788430020213127,
+            0.020140711218118668,
+            0.016559835523366928,
+            0.003587735816836357,
+            0.005642280448228121,
+            -0.025491446256637573,
+            -0.0003918726579286158,
+            -0.012361567467451096,
+            -0.018123896792531013,
+            0.01918032392859459,
+            8.628487557871267e-05,
+            0.00598870636895299,
+            -0.008218178525567055,
+            -0.006726147141307592,
+            -0.004853390157222748,
+            0.01424118410795927,
+            0.010872965678572655,
+            -0.02842748910188675,
+            -0.004805370699614286,
+            -0.022171247750520706,
+            -0.004393775947391987,
+            0.023008156567811966,
+            0.0032207302283495665,
+            -0.003690634621307254,
+            -0.0011661855969578028,
+            -0.009864558465778828,
+            0.004386915825307369,
+            0.006071025040000677,
+            0.0006191073916852474,
+            0.04810172691941261,
+            0.008465135470032692,
+            -0.006894215010106564,
+            -0.00863663386553526,
+            0.0007318672141991556,
+            -0.008533734828233719,
+            0.004736771807074547,
+            0.01002233661711216,
+            0.0018967665964737535,
+            -0.013191617093980312,
+            -0.016381477937102318,
+            -0.009706780314445496,
+            -0.009555862285196781,
+            -0.004215417895466089,
+            -0.023515790700912476,
+            -0.00026753669953905046,
+            -0.009830258786678314,
+            0.002063119551166892,
+            -0.005117496941238642,
+            -0.011888233944773674,
+            0.01944100111722946,
+            -0.02619115822017193,
+            0.007271510548889637,
+            0.01768486201763153,
+            0.006726147141307592,
+            0.012560505419969559,
+            -0.005978416185826063,
+            -0.013520893640816212,
+            0.003927301615476608,
+            -0.014021667651832104,
+            -0.010557410307228565,
+            0.002400970319285989,
+            0.0027816956862807274,
+            -0.01966051757335663,
+            -0.01780834048986435,
+            0.002287781797349453,
+            0.011984271928668022,
+            0.00244898977689445,
+            0.005395323503762484,
+            0.012992680072784424,
+            -0.0004488957056310028,
+            0.02037394978106022,
+            0.0005959551781415939,
+            -0.02947019785642624,
+            0.006016145925968885,
+            0.010845526121556759,
+            0.005539381876587868,
+            -0.0004690467321779579,
+            0.010914125479757786,
+            0.008259338326752186,
+            0.0028674446512013674,
+            -0.02398226410150528,
+            0.0020974192302674055,
+            -0.015338771045207977,
+            0.010090935975313187,
+            -0.026355795562267303,
+            0.014995775185525417,
+            -0.02604023925960064,
+            0.01130514033138752,
+            -0.02850980870425701,
+            0.01225866936147213,
+            0.002647927263751626,
+            0.001377985579892993,
+            0.0071823312900960445,
+            -0.011545238085091114,
+            -0.00035114193451590836,
+            -0.015530848875641823,
+            0.0009037938434630632,
+            -0.0009681055671535432,
+            -0.0261225588619709,
+            -0.029223239049315453,
+            -0.0013376835267990828,
+            -0.007264650426805019,
+            0.004393775947391987,
+            0.009864558465778828,
+            -0.012018571607768536,
+            -0.02174593135714531,
+            0.007038273382931948,
+            0.002641067374497652,
+            -0.0014877441572025418,
+            0.026328355073928833,
+            -0.025875601917505264,
+            -0.02096390165388584,
+            0.003472832264378667,
+            0.01938612200319767,
+            0.005100347101688385,
+            0.006105324719101191,
+            -0.0193586815148592,
+            0.01831597462296486,
+            -0.0070657129399478436,
+            -0.02192429080605507,
+            -0.008595474064350128,
+            0.02096390165388584,
+            0.014378382824361324,
+            0.012382147833704948,
+            0.010475090704858303,
+            -0.0009543857304379344,
+            0.002027105074375868,
+            -0.0014980339910835028,
+            0.013109298422932625,
+            0.008924749679863453,
+            -0.004887689836323261,
+            -0.015174132771790028,
+            -0.0014011377934366465,
+            -0.008794412016868591,
+            0.0028640148229897022,
+            0.013630651868879795,
+            0.011346300132572651,
+            -0.017163507640361786,
+            -0.01583268493413925,
+            -0.03158305212855339,
+            0.002167733386158943,
+            -0.01983887515962124,
+            -0.008986488915979862,
+            -0.00034042331390082836,
+            0.006746727041900158,
+            0.008664073422551155,
+            0.008005521260201931,
+            0.015297611244022846,
+            0.01651867665350437,
+            0.019729116931557655,
+            0.027659179642796516,
+            -0.00056208431487903,
+            -0.004603003151714802,
+            0.004050780087709427,
+            0.0025107290130108595,
+            -0.028537249192595482,
+            -0.00927460566163063,
+            -0.012368427589535713,
+            -0.0006302547408267856,
+            0.016093362122774124,
+            0.001071004313416779,
+            0.007895763032138348,
+            -0.015119253657758236,
+            0.013829589821398258,
+            -0.010914125479757786,
+            -0.010584849864244461,
+            -0.02848236821591854,
+            0.0002185526245739311,
+            0.021608734503388405,
+            -0.025450287386775017,
+            0.013280796818435192,
+            -0.018027858808636665,
+            0.02004467323422432,
+            -0.008060400374233723,
+            -0.004736771807074547,
+            0.0014577320544049144,
+            0.005656000226736069,
+            -0.00611561443656683,
+            0.01020069420337677,
+            0.01136687956750393,
+            0.012704563327133656,
+            -0.013095579110085964,
+            0.009507843293249607,
+            -0.0021643033251166344,
+            0.02192429080605507,
+            0.024558497592806816,
+            0.015283891931176186,
+            -0.012340988032519817,
+            -0.017588824033737183,
+            0.00875325221568346,
+            0.008732672780752182,
+            -0.005803488660603762,
+            0.012649684213101864,
+            -0.01631287857890129,
+            0.01003605592995882,
+            0.01457046065479517,
+            0.005076337605714798,
+            0.020840423181653023,
+            -0.005381603725254536,
+            -0.011071903631091118,
+            0.01562688685953617,
+            0.012059731408953667,
+            -0.004263437353074551,
+            -0.011901953257620335,
+            0.009226586669683456,
+            0.011462918482720852,
+            0.008677792735397816,
+            0.016175679862499237,
+            0.009075668640434742,
+            0.0077517046593129635,
+            0.012251809239387512,
+            -0.028345171362161636,
+            0.004887689836323261,
+            -0.0031641360837966204,
+            -0.017396746203303337,
+            0.008396536111831665,
+            -0.01003605592995882,
+            -0.006496340036392212,
+            0.024407578632235527,
+            0.017314426600933075,
+            0.0020391098223626614,
+            -0.010132094845175743,
+            -0.0007327247294597328,
+            -0.00010531042062211782,
+            0.0023838207125663757,
+            0.0028897393494844437,
+            0.011860793456435204,
+            -0.01651867665350437,
+            -0.019495880231261253,
+            0.029031163081526756,
+            -0.007511607371270657,
+            0.006132764276117086,
+            0.001967080868780613
+          ],
+          "keyphrases": [
+            "hyperparameter search",
+            "8B dense model",
+            "constant learning rate",
+            "MoE 8B/64E models",
+            "square root learning rate decay"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "85fd3724-b6db-4994-a1ad-02ce9c98280c",
+      "type": "child",
+      "source": {
+        "id": "65742957-6bda-446d-a61a-0fe20313c760",
+        "properties": {
+          "page_content": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity\n\nWilliam Fedus∗ liamfedus@google.com Barret Zoph∗ barretzoph@google.com Noam Shazeer noam@google.com Google, Mountain View, CA 94043, USA Editor: Alexander Clark\n\nAbstract\n\nIn deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) models defy this and instead select different parameters for each incoming example. The result is a sparsely-activated model—with an outrageous number of parameters—but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs, and training instability. We address these with the introduction of the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques mitigate the instabilities, and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large (Raffel et al., 2019) to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the \"Colossal Clean Crawled Corpus\", and achieve a 4x speedup over the T5-XXL model.12 Keywords: mixture-of-experts, natural language processing, sparsity, large-scale machine learning, distributed computing arXiv:2101.03961v3 [cs.LG] 16 Jun 2022 Contents\n\n1 Introduction 3 2 Switch Transformer 4 2.1 Simplifying Sparse Routing 5 2.2 Efficient Sparse Routing 6 2.3 Putting It All Together: The Switch Transformer 8 2.4 Improved Training and Fine-Tuning Techniques 8 3 Scaling Properties 11 3.1 Scaling Results on a Step-Basis 12 3.2 Scaling Results on a Time-Basis 13 3.3 Scaling Versus a Larger Dense Model 13 4 Downstream Results 14 4.1 Fine-Tuning 14 4.2 Distillation 16 4.3 Multilingual Learning 17 5 Designing Models with Data, Model, and Expert-Parallelism 18 5.1 Data Parallelism 20 5.2 Model Parallelism 20 5.3 Model and Data Parallelism 21 5.4 Expert and Data Parallelism 22 5.5 Expert, Model and Data Parallelism 22 5.6 Towards Trillion Parameter Models 22 6 Related Work 24 7 Discussion 25 8 Future Work 26 9 Conclusion 27 A Switch for Attention 27 B Preventing Token Dropping with No-Token-Left-Behind 29\n\nC Encouraging Exploration Across Experts 29 D Switch Transformers in Lower Compute Regimes 29 E Relation of Upstream to Downstream Model Performance 32 F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/switch_transformers.md"
+          },
+          "summary": "The Switch Transformer is a sparsely-activated model that simplifies the Mixture of Experts (MoE) approach, allowing for efficient scaling to trillion-parameter models with reduced communication and computational costs. It achieves significant speedups in pre-training and fine-tuning by activating only a subset of neural network weights for each input, maintaining a constant computational cost. The model demonstrates superior scaling properties and improved performance across various natural language processing tasks, including multilingual settings. It also allows for effective distillation into smaller dense models, preserving a portion of the quality gains. The architecture is designed to maximize parameter count while keeping floating point operations per example constant, making it suitable for both large-scale and smaller-scale applications.",
+          "headlines": {
+            "1 Introduction": [],
+            "2 Switch Transformer": [
+              "2.1 Simplifying Sparse Routing",
+              "2.2 Efficient Sparse Routing",
+              "2.3 Putting It All Together: The Switch Transformer",
+              "2.4 Improved Training and Fine-Tuning Techniques"
+            ],
+            "3 Scaling Properties": [
+              "3.1 Scaling Results on a Step-Basis",
+              "3.2 Scaling Results on a Time-Basis",
+              "3.3 Scaling Versus a Larger Dense Model"
+            ],
+            "4 Downstream Results": [
+              "4.1 Fine-Tuning",
+              "4.2 Distillation",
+              "4.3 Multilingual Learning"
+            ],
+            "5 Designing Models with Data, Model, and Expert-Parallelism": [
+              "5.1 Data Parallelism",
+              "5.2 Model Parallelism",
+              "5.3 Model and Data Parallelism",
+              "5.4 Expert and Data Parallelism",
+              "5.5 Expert, Model and Data Parallelism",
+              "5.6 Towards Trillion Parameter Models"
+            ],
+            "6 Related Work": [],
+            "7 Discussion": [],
+            "8 Future Work": [],
+            "9 Conclusion": [],
+            "A Switch for Attention": [],
+            "B Preventing Token Dropping with No-Token-Left-Behind": [],
+            "C Encouraging Exploration Across Experts": [],
+            "D Switch Transformers in Lower Compute Regimes": [],
+            "E Relation of Upstream to Downstream Model Performance": [],
+            "F Pseudo Code for Switch Transformers": []
+          },
+          "summary_embedding": [
+            -0.0060447100549936295,
+            -0.00390331051312387,
+            -0.01784370467066765,
+            -0.0021820589900016785,
+            0.013227885589003563,
+            0.011338187381625175,
+            0.0165580902248621,
+            0.024240797385573387,
+            -0.005986624862998724,
+            0.03596621751785278,
+            -0.004852031823247671,
+            -0.0068811336532235146,
+            -0.003574162255972624,
+            0.014242436736822128,
+            0.011578272096812725,
+            -0.033302053809165955,
+            -0.024519605562090874,
+            0.01050176378339529,
+            0.016093410551548004,
+            -0.006795942317694426,
+            0.007473600562661886,
+            -0.030513975769281387,
+            -0.06124480068683624,
+            -0.006838538218289614,
+            -0.0026099516544491053,
+            0.01029265858232975,
+            0.0010329445358365774,
+            -0.010873507708311081,
+            -0.053469158709049225,
+            0.023311438038945198,
+            0.025294071063399315,
+            0.005092116538435221,
+            -0.029584616422653198,
+            0.0009361362317577004,
+            -0.05161044001579285,
+            -0.04448534920811653,
+            0.0379488542675972,
+            0.012561844661831856,
+            -0.04364892467856407,
+            0.039249956607818604,
+            -0.004875265993177891,
+            -0.0022440163884311914,
+            -0.021499186754226685,
+            -0.0034909071400761604,
+            -0.007039899472147226,
+            -0.009851212613284588,
+            -0.02874819189310074,
+            0.0014037203509360552,
+            0.021886419504880905,
+            -0.0056962003000080585,
+            0.023264970630407333,
+            -0.024271776899695396,
+            0.01936165988445282,
+            0.04240977764129639,
+            -0.06957805901765823,
+            0.024659009650349617,
+            0.017549408599734306,
+            0.05120771750807762,
+            0.06511713564395905,
+            0.018339363858103752,
+            0.021669568493962288,
+            -0.014141756109893322,
+            -0.017766257748007774,
+            -0.011872569099068642,
+            0.004673904739320278,
+            -0.014970434829592705,
+            -0.01891246810555458,
+            0.008449427783489227,
+            -0.020461400970816612,
+            0.008209343999624252,
+            -0.05576157942414284,
+            -0.014730350114405155,
+            -0.034200433641672134,
+            -0.01140014547854662,
+            0.017642343416810036,
+            -0.018215449526906013,
+            0.030126741155982018,
+            0.02997184917330742,
+            0.002638994250446558,
+            0.0007822110201232135,
+            0.021049996837973595,
+            0.006509389728307724,
+            -0.0053438181057572365,
+            0.012321759946644306,
+            0.028159597888588905,
+            -0.01450575515627861,
+            -0.010106786154210567,
+            -0.03454120084643364,
+            0.01861817203462124,
+            0.01589204929769039,
+            -0.04194509983062744,
+            -0.012569589540362358,
+            -0.013824224472045898,
+            0.032341714948415756,
+            -0.009549170732498169,
+            0.04098476096987724,
+            -0.010083552449941635,
+            0.001383390510454774,
+            -0.031149037182331085,
+            -0.009463978931307793,
+            0.017069239169359207,
+            0.006172496825456619,
+            -0.016589069738984108,
+            -0.0012846461031585932,
+            0.016170857474207878,
+            0.006617815233767033,
+            -0.030142230913043022,
+            0.04488807171583176,
+            0.023497309535741806,
+            0.055296897888183594,
+            0.05579255893826485,
+            -0.037391237914562225,
+            -0.002445377642288804,
+            0.0344172865152359,
+            -0.0228312686085701,
+            -0.015040136873722076,
+            -0.034014564007520676,
+            -0.01768881268799305,
+            0.035935238003730774,
+            -0.0318460576236248,
+            0.01556677371263504,
+            0.028066661208868027,
+            -0.025712283328175545,
+            0.003488970920443535,
+            -0.03943582624197006,
+            0.003622566582635045,
+            0.01220559049397707,
+            0.006095050368458033,
+            -0.0026293133851140738,
+            -0.013615119270980358,
+            0.008906363509595394,
+            0.003055269829928875,
+            0.00800798274576664,
+            -0.04615819454193115,
+            0.018478767946362495,
+            -0.00454224506393075,
+            0.04033420979976654,
+            0.042502716183662415,
+            -0.02730768360197544,
+            -0.016743963584303856,
+            0.008557853288948536,
+            0.012104909867048264,
+            0.04188314080238342,
+            -0.007287728600203991,
+            -0.00121784838847816,
+            -0.0007836631848476827,
+            0.007322579622268677,
+            0.047552235424518585,
+            0.004646798130124807,
+            0.0020697612781077623,
+            -0.027230238541960716,
+            0.014978179708123207,
+            -0.019733402878046036,
+            0.027648448944091797,
+            0.023373395204544067,
+            0.02942972257733345,
+            -0.04048910364508629,
+            0.017518429085612297,
+            -0.011818356812000275,
+            -0.005568413063883781,
+            -0.002726121572777629,
+            -0.02856232039630413,
+            -0.0074426219798624516,
+            0.033983584493398666,
+            0.015365412458777428,
+            0.012956822291016579,
+            0.03271345794200897,
+            0.0012052633101120591,
+            -0.042502716183662415,
+            0.03804178908467293,
+            0.024550583213567734,
+            0.04640602692961693,
+            0.023466331884264946,
+            0.010796061716973782,
+            0.006571347359567881,
+            0.013375034555792809,
+            -0.019377149641513824,
+            0.015907539054751396,
+            0.02560385875403881,
+            0.009750531986355782,
+            -0.03167567402124405,
+            -0.0009501734166406095,
+            0.01178737822920084,
+            -0.004546117503196001,
+            0.03701949119567871,
+            0.04104671627283096,
+            0.020569827407598495,
+            0.05030933395028114,
+            0.0020155487582087517,
+            0.017936641350388527,
+            0.005936284549534321,
+            0.02820606529712677,
+            0.030467506498098373,
+            -0.005285732913762331,
+            0.005076627247035503,
+            -0.01136916596442461,
+            -0.03385967016220093,
+            0.014110777527093887,
+            0.01561324205249548,
+            -0.008077683858573437,
+            -0.01697630248963833,
+            0.006033093202859163,
+            0.009169681929051876,
+            -0.01795213110744953,
+            -0.017456471920013428,
+            0.005971135571599007,
+            0.018866000697016716,
+            -0.013576395809650421,
+            0.024689987301826477,
+            -0.04442339017987251,
+            0.029119934886693954,
+            -0.033209118992090225,
+            -0.003384418087080121,
+            -0.010641167871654034,
+            -0.0030843124259263277,
+            0.0041666291654109955,
+            0.011314953677356243,
+            0.013406013138592243,
+            0.002042655134573579,
+            -0.013909416273236275,
+            0.024519605562090874,
+            0.018447788432240486,
+            0.012073930352926254,
+            -0.03148980438709259,
+            -0.014397329650819302,
+            -0.0015673263696953654,
+            -0.008743725717067719,
+            -0.034138478338718414,
+            0.026966920122504234,
+            0.014048820361495018,
+            -0.03844451159238815,
+            0.017797237262129784,
+            0.029538147151470184,
+            0.014420564286410809,
+            -0.004790074657648802,
+            -0.0463130883872509,
+            0.04045812413096428,
+            -0.030436528846621513,
+            -0.033333033323287964,
+            -0.010478530079126358,
+            -0.0003146269591525197,
+            0.02894955314695835,
+            -0.0064358157105743885,
+            -0.050835974514484406,
+            0.01184933539479971,
+            0.020631784573197365,
+            0.00010763872705865651,
+            0.005731051322072744,
+            -0.019593998789787292,
+            0.00028631053282879293,
+            -0.03395260497927666,
+            0.0090844901278615,
+            0.026750069111585617,
+            0.008867640048265457,
+            -0.016418687999248505,
+            0.03351890295743942,
+            -0.0004121613164898008,
+            -0.018695618957281113,
+            -0.003889757441356778,
+            0.006885006092488766,
+            0.034912943840026855,
+            0.004836542531847954,
+            -0.02946070022881031,
+            0.05665995925664902,
+            -0.02100352756679058,
+            0.021344292908906937,
+            0.05631919577717781,
+            -0.019439106807112694,
+            -0.008472662419080734,
+            -0.002065889071673155,
+            0.0026196325197815895,
+            -0.0260530486702919,
+            -0.008937342092394829,
+            0.032341714948415756,
+            0.041728246957063675,
+            -0.018742086365818977,
+            -0.001398879918269813,
+            0.006780453026294708,
+            0.009425255469977856,
+            0.004948840010911226,
+            0.02692045085132122,
+            -0.012128143571317196,
+            -0.014559967443346977,
+            0.05198218300938606,
+            0.028314489871263504,
+            -0.012081675231456757,
+            -0.0164031982421875,
+            0.019888296723365784,
+            -0.024844881147146225,
+            -0.020244551822543144,
+            -0.02431824430823326,
+            -0.020414933562278748,
+            -0.0042518205009400845,
+            0.009525936096906662,
+            0.037484172731637955,
+            -0.03968365862965584,
+            0.012747716158628464,
+            0.021700548008084297,
+            0.02981695532798767,
+            -0.019532041624188423,
+            -0.05910727381706238,
+            -0.0016128262504935265,
+            0.03296129032969475,
+            -0.015458349138498306,
+            -0.049379974603652954,
+            0.015164051204919815,
+            0.0260530486702919,
+            0.05957195535302162,
+            -0.001461805310100317,
+            -0.014954946003854275,
+            0.061926331371068954,
+            0.03026614524424076,
+            -0.004383479710668325,
+            -0.009494957514107227,
+            -0.0127864396199584,
+            0.003396035172045231,
+            0.00465454300865531,
+            -0.03364282101392746,
+            -0.041170634329319,
+            -0.011841590516269207,
+            -0.010284913703799248,
+            0.008108663372695446,
+            -0.040922801941633224,
+            0.03531566634774208,
+            -0.021917399019002914,
+            0.010950954630970955,
+            0.015473837964236736,
+            0.037391237914562225,
+            -0.021344292908906937,
+            -0.01284839678555727,
+            -0.01572941243648529,
+            -0.024968795478343964,
+            -0.014010096900165081,
+            0.021917399019002914,
+            0.007682706229388714,
+            3.4699722164077684e-05,
+            0.010075807571411133,
+            -0.024659009650349617,
+            -0.051052823662757874,
+            0.0291044469922781,
+            0.0015605497173964977,
+            -0.015938518568873405,
+            -0.004623564425855875,
+            -0.01977987214922905,
+            -0.01122201792895794,
+            -0.012608313001692295,
+            -0.039249956607818604,
+            0.021111954003572464,
+            -0.01135367713868618,
+            0.015876561403274536,
+            0.0009250032599084079,
+            0.003659353591501713,
+            0.028624277561903,
+            0.01116006076335907,
+            0.0051811798475682735,
+            -0.03255856782197952,
+            -0.018277406692504883,
+            0.010122275911271572,
+            -0.00561100896447897,
+            -0.018339363858103752,
+            -0.013351799920201302,
+            -0.05037129297852516,
+            -0.03305422514677048,
+            -0.008054450154304504,
+            -0.05415068939328194,
+            0.0020697612781077623,
+            -0.014296649023890495,
+            -0.02232012152671814,
+            -0.005990497302263975,
+            0.03107159025967121,
+            -0.003235333366319537,
+            0.025294071063399315,
+            0.0076594725251197815,
+            0.037360258400440216,
+            0.040117356926202774,
+            0.008426194079220295,
+            0.03065337985754013,
+            -0.0029991210903972387,
+            -0.016867877915501595,
+            0.05486319959163666,
+            -0.01881953328847885,
+            0.013529927469789982,
+            -0.008604321628808975,
+            0.008604321628808975,
+            0.03299226611852646,
+            -0.003794885240495205,
+            0.02007416822016239,
+            -0.0311955064535141,
+            -0.07998688519001007,
+            0.01637221872806549,
+            0.028159597888588905,
+            -0.06418777257204056,
+            0.014691627584397793,
+            -0.020275529474020004,
+            0.02052335813641548,
+            0.02411688305437565,
+            -0.0009114501299336553,
+            -0.014064309187233448,
+            0.005595519673079252,
+            0.0020833145827054977,
+            -0.01456771232187748,
+            -0.002691270550712943,
+            -0.009417511522769928,
+            0.008774704299867153,
+            -0.017007282003760338,
+            -0.013406013138592243,
+            -0.034231413155794144,
+            0.010873507708311081,
+            0.01233724970370531,
+            -0.003969140350818634,
+            0.0011500825639814138,
+            -0.03277541697025299,
+            -0.007876322604715824,
+            -0.02145271934568882,
+            0.03382869064807892,
+            0.012058441527187824,
+            -0.030699847266077995,
+            -0.007814365439116955,
+            0.015264731831848621,
+            -0.007876322604715824,
+            0.0008906363509595394,
+            0.024132372811436653,
+            0.02103450708091259,
+            0.0028364830650389194,
+            -0.0009700191440060735,
+            0.02438020147383213,
+            0.007427132688462734,
+            -0.01720864325761795,
+            0.028794659301638603,
+            0.014288905076682568,
+            0.060160547494888306,
+            -0.03271345794200897,
+            0.001297231181524694,
+            -0.04135650396347046,
+            -0.031118059530854225,
+            0.006408709101378918,
+            0.014350862242281437,
+            0.007597515359520912,
+            -0.028252532705664635,
+            0.005978880450129509,
+            0.02004319056868553,
+            0.01469937153160572,
+            0.009704063646495342,
+            0.0037580979987978935,
+            -0.036740683019161224,
+            -0.000691695255227387,
+            -0.011214273050427437,
+            -0.0038781403563916683,
+            0.035656433552503586,
+            -0.008186109364032745,
+            -0.05151750147342682,
+            -0.0006418389966711402,
+            -0.03255856782197952,
+            -0.0353776253759861,
+            0.021313315257430077,
+            -0.00839521549642086,
+            -0.0039207362569868565,
+            -0.013986863195896149,
+            -0.000537770101800561,
+            -0.010524998418986797,
+            0.027509044855833054,
+            0.028345469385385513,
+            0.022490503266453743,
+            -0.05384090170264244,
+            0.009479468688368797,
+            0.00023766436788719147,
+            -0.05817791447043419,
+            0.0009709871956147254,
+            -0.007117345929145813,
+            0.011191039346158504,
+            -0.020213572308421135,
+            -0.02414786070585251,
+            0.011500826105475426,
+            0.03392162546515465,
+            0.014304393902420998,
+            0.0321558453142643,
+            -0.013367289677262306,
+            -0.09250226616859436,
+            0.018292896449565887,
+            -0.008689512498676777,
+            0.04126356914639473,
+            0.00865078903734684,
+            -0.004278926644474268,
+            -0.015845581889152527,
+            -0.00865078903734684,
+            -0.006943091284483671,
+            0.01283290795981884,
+            0.004530628211796284,
+            -0.005626498255878687,
+            0.0016631665639579296,
+            0.013708055019378662,
+            -0.005967263597995043,
+            0.007481344975531101,
+            0.003651608945801854,
+            -0.013189162127673626,
+            -0.06059424951672554,
+            -0.04055105894804001,
+            -0.018540725111961365,
+            0.050835974514484406,
+            -0.0069159846752882,
+            -0.046684831380844116,
+            -0.010850274004042149,
+            0.05058814212679863,
+            0.012259802781045437,
+            -0.0334569476544857,
+            -0.0007841471815481782,
+            0.0050882440991699696,
+            0.015458349138498306,
+            0.004546117503196001,
+            -0.027292195707559586,
+            0.02820606529712677,
+            0.0008412640891037881,
+            0.02300165221095085,
+            -0.018339363858103752,
+            -0.005312839522957802,
+            0.035780347883701324,
+            -0.024844881147146225,
+            0.023791607469320297,
+            0.0055219451896846294,
+            0.031629208475351334,
+            0.002952652983367443,
+            0.010060317814350128,
+            0.004704883322119713,
+            0.0186491496860981,
+            0.004275054205209017,
+            0.0276639387011528,
+            0.011462102644145489,
+            0.006428070832043886,
+            0.021623101085424423,
+            -0.009378788061439991,
+            -0.014513500034809113,
+            0.01885051093995571,
+            -0.0014695499558001757,
+            0.025232113897800446,
+            -0.020538847893476486,
+            0.002726121572777629,
+            -0.0016883367206901312,
+            -0.017967619001865387,
+            -0.040922801941633224,
+            -0.0007405834621749818,
+            0.021747015416622162,
+            -0.011671207845211029,
+            -0.0228312686085701,
+            -0.013715799897909164,
+            -0.01585332676768303,
+            0.0038490979932248592,
+            0.034169457852840424,
+            -0.0398695282638073,
+            0.00963436160236597,
+            -0.000830615172162652,
+            -0.03125746175646782,
+            -0.0239619892090559,
+            -0.01093546487390995,
+            0.014614180661737919,
+            -0.01652711257338524,
+            -0.021173911169171333,
+            -0.028469383716583252,
+            -0.008410705253481865,
+            0.006048582494258881,
+            -0.05065010115504265,
+            -0.004054331686347723,
+            0.019965743646025658,
+            0.0276639387011528,
+            -0.003099801717326045,
+            -0.02177799493074417,
+            0.0012468908680602908,
+            0.022351099178195,
+            0.02936776541173458,
+            0.014226946979761124,
+            0.027586491778492928,
+            -0.002178186783567071,
+            -0.0011675080750137568,
+            -0.0073148347437381744,
+            0.01307299267500639,
+            0.0011287847300991416,
+            0.036616768687963486,
+            0.0122210793197155,
+            -0.020306508988142014,
+            0.0013959755888208747,
+            -0.006431943271309137,
+            0.029181893914937973,
+            -0.02270735427737236,
+            -9.832092473516241e-05,
+            0.03531566634774208,
+            0.027741385623812675,
+            0.004933350719511509,
+            -0.0143121387809515,
+            -0.05142456665635109,
+            0.021592123433947563,
+            -0.01765783317387104,
+            0.031025122851133347,
+            0.016666516661643982,
+            0.003397971158847213,
+            -0.0034212053287774324,
+            0.012368228286504745,
+            -0.022087780758738518,
+            -0.004406713880598545,
+            -0.021142931655049324,
+            0.04389675334095955,
+            -0.043184246867895126,
+            0.008472662419080734,
+            0.029445212334394455,
+            0.011926782317459583,
+            -0.019082851707935333,
+            -0.005425137002021074,
+            0.029212871566414833,
+            -0.00162637943867594,
+            0.01308848150074482,
+            0.0058665829710662365,
+            -0.026935940608382225,
+            0.01202746294438839,
+            -0.031660184264183044,
+            0.02177799493074417,
+            -0.013436991721391678,
+            0.018447788432240486,
+            0.03342596814036369,
+            -0.0030397805385291576,
+            -0.0038239278364926577,
+            -0.016263794153928757,
+            -0.00969631876796484,
+            0.0662323608994484,
+            0.01054823212325573,
+            0.03726732358336449,
+            0.0018974426202476025,
+            -0.009773765690624714,
+            0.007969259284436703,
+            0.0063041565008461475,
+            -0.012329504825174809,
+            0.010494019836187363,
+            0.024178840219974518,
+            0.0012730291346088052,
+            -0.002412462839856744,
+            -0.019408127292990685,
+            -0.03438630700111389,
+            0.003727119415998459,
+            -0.030932186171412468,
+            0.022691864520311356,
+            0.009208405390381813,
+            0.017456471920013428,
+            0.008240322582423687,
+            -0.015907539054751396,
+            0.03200095146894455,
+            0.05461536720395088,
+            0.0019768255297094584,
+            0.013375034555792809,
+            0.007938280701637268,
+            0.007036027032881975,
+            -0.011338187381625175,
+            0.04702559858560562,
+            0.000662652775645256,
+            0.005169562995433807,
+            0.022536970674991608,
+            0.02145271934568882,
+            0.022939693182706833,
+            0.002304037567228079,
+            -0.011694442480802536,
+            0.010478530079126358,
+            -0.0006302220281213522,
+            0.004445436876267195,
+            -0.010695381090044975,
+            -0.027865299955010414,
+            0.007492962293326855,
+            0.01302652433514595,
+            0.022598929703235626,
+            0.021592123433947563,
+            -0.03178409859538078,
+            -0.01855621486902237,
+            0.03364282101392746,
+            -0.02997184917330742,
+            0.03612111136317253,
+            -0.0014424435794353485,
+            -0.023807097226381302,
+            -0.0058356039226055145,
+            0.016929835081100464,
+            0.02656419761478901,
+            -0.029181893914937973,
+            0.01804506592452526,
+            0.029693040996789932,
+            -0.0074426219798624516,
+            0.0005939188995398581,
+            0.012112653814256191,
+            0.02447313815355301,
+            -0.009293596260249615,
+            -0.0024027819745242596,
+            -0.0013320822035893798,
+            -0.011926782317459583,
+            -0.018370341509580612,
+            -0.0044725434854626656,
+            -0.022629907354712486,
+            0.005564541090279818,
+            -0.007365175057202578,
+            -0.003481226274743676,
+            -0.008581086993217468,
+            0.01154729351401329,
+            0.001979729626327753,
+            -0.019857317209243774,
+            -0.022335609421133995,
+            0.01652711257338524,
+            0.03887821361422539,
+            -0.018215449526906013,
+            -0.02106548473238945,
+            0.024442158639431,
+            -0.01759587600827217,
+            -0.01601596549153328,
+            0.036678727716207504,
+            0.013739033602178097,
+            0.007736918982118368,
+            -0.03358086198568344,
+            -0.030049296095967293,
+            -0.03274443745613098,
+            0.006296411622315645,
+            0.03785591572523117,
+            -0.0028752065263688564,
+            0.02569679357111454,
+            0.03701949119567871,
+            0.0057891360484063625,
+            -0.028934063389897346,
+            -0.014823286794126034,
+            -0.045104920864105225,
+            0.060160547494888306,
+            0.0011481463443487883,
+            0.05161044001579285,
+            0.016294771805405617,
+            -0.01456771232187748,
+            0.00419760774821043,
+            -0.013251119293272495,
+            0.026192452758550644,
+            0.02280029095709324,
+            0.025929134339094162,
+            0.006582964211702347,
+            0.022505993023514748,
+            -0.0037522895727306604,
+            -0.012197845615446568,
+            0.007307090330868959,
+            0.014536733739078045,
+            0.0002497654058970511,
+            -0.011392400600016117,
+            0.015783624723553658,
+            -0.01307299267500639,
+            -0.011361422017216682,
+            0.023295948281884193,
+            0.01368482131510973,
+            -0.012344993650913239,
+            0.014064309187233448,
+            -0.008565598167479038,
+            0.01239920686930418,
+            -0.023915521800518036,
+            0.022691864520311356,
+            -0.03262052312493324,
+            0.009587894193828106,
+            0.0015053689712658525,
+            -0.0073148347437381744,
+            0.024550583213567734,
+            -0.0027454833034425974,
+            -0.05071205645799637,
+            0.03550153970718384,
+            -0.008015726692974567,
+            0.0037213109899312258,
+            0.006381602957844734,
+            0.011392400600016117,
+            -0.0012623801594600081,
+            0.03807276859879494,
+            0.01239920686930418,
+            0.015969496220350266,
+            0.02199484594166279,
+            0.011911292560398579,
+            -0.008433938957750797,
+            -0.004096927121281624,
+            -0.0016137943603098392,
+            -0.01784370467066765,
+            0.025944624096155167,
+            0.0018674320308491588,
+            0.03962169960141182,
+            -0.018168980255723,
+            -0.02300165221095085,
+            0.028066661208868027,
+            0.007597515359520912,
+            -0.02351279929280281,
+            -0.0017377089243382215,
+            0.01202746294438839,
+            -0.009185170754790306,
+            0.027385130524635315,
+            0.01196550577878952,
+            0.019113829359412193,
+            -0.018060555681586266,
+            -0.004340883810073137,
+            -0.005072754807770252,
+            -0.015938518568873405,
+            0.02238207869231701,
+            0.020771188661456108,
+            0.011152315884828568,
+            -0.011880313977599144,
+            -0.01307299267500639,
+            0.020600805059075356,
+            0.035036858171224594,
+            -0.0002744515077210963,
+            0.005672966130077839,
+            0.017487451434135437,
+            0.002373739378526807,
+            -0.04470219835639,
+            0.016232814639806747,
+            -0.014025585725903511,
+            0.011214273050427437,
+            0.019593998789787292,
+            -0.021917399019002914,
+            -0.03196997195482254,
+            -0.011539548635482788,
+            -0.0414804182946682,
+            0.00493722315877676,
+            -0.0069159846752882,
+            0.013576395809650421,
+            -0.010176488198339939,
+            -0.0014831031439825892,
+            0.016666516661643982,
+            0.0038103745318949223,
+            0.02177799493074417,
+            -0.011678952723741531,
+            -0.012073930352926254,
+            -0.020213572308421135,
+            -0.020786678418517113,
+            0.004278926644474268,
+            -0.030405549332499504,
+            -0.004488032776862383,
+            0.021375272423028946,
+            -0.04783104360103607,
+            -0.03689557686448097,
+            0.02898053266108036,
+            -0.008503641001880169,
+            0.020755698904395103,
+            0.022955182939767838,
+            -0.015094350092113018,
+            -0.0008717587334103882,
+            -0.018695618957281113,
+            0.02258343994617462,
+            -0.007837600074708462,
+            0.026300877332687378,
+            -0.0027551641687750816,
+            0.0006292539183050394,
+            -0.02659517526626587,
+            0.020306508988142014,
+            0.005227648187428713,
+            0.012275291606783867,
+            -0.02390003204345703,
+            0.006815304048359394,
+            -0.03133491054177284,
+            -0.01784370467066765,
+            0.0180140882730484,
+            -0.011152315884828568,
+            0.027989214286208153,
+            -0.006060199346393347,
+            0.028438406065106392,
+            0.022722844034433365,
+            0.024751944467425346,
+            0.02676555886864662,
+            -0.02363671362400055,
+            0.008495896123349667,
+            0.03726732358336449,
+            0.00865078903734684,
+            -0.015876561403274536,
+            0.008170620538294315,
+            -0.04863648861646652,
+            -0.008062195032835007,
+            0.05030933395028114,
+            0.002127846470102668,
+            0.023141054436564445,
+            0.016294771805405617,
+            -0.002873270306736231,
+            -0.005688455421477556,
+            0.010269423946738243,
+            -0.04296739399433136,
+            -0.023172033950686455,
+            0.022629907354712486,
+            -0.021561143919825554,
+            6.54212328754511e-07,
+            0.03122648410499096,
+            0.012120398692786694,
+            -0.0009109660750254989,
+            -0.05108380317687988,
+            0.02100352756679058,
+            0.01178737822920084,
+            0.014443797990679741,
+            -0.010300402529537678,
+            0.026068538427352905,
+            0.01733255758881569,
+            0.038413532078266144,
+            0.012755461037158966,
+            -0.002679653698578477,
+            -0.03782493621110916,
+            0.009177426807582378,
+            -0.021685058251023293,
+            0.0011471783509477973,
+            0.0074503663927316666,
+            -0.02273833192884922,
+            -0.024782923981547356,
+            0.0043912241235375404,
+            0.004588713403791189,
+            0.011880313977599144,
+            0.022087780758738518,
+            0.009417511522769928,
+            -0.021204888820648193,
+            0.015404135920107365,
+            -0.05446047708392143,
+            -0.015450604259967804,
+            -0.023884542286396027,
+            0.02566581591963768,
+            0.020012211054563522,
+            0.010184233076870441,
+            0.005545179359614849,
+            0.010904486291110516,
+            -0.020988039672374725,
+            -0.009239383973181248,
+            0.020631784573197365,
+            0.011074868962168694,
+            -0.012840652838349342,
+            0.04113965481519699,
+            -0.00011126903700642288,
+            0.031210994347929955,
+            -0.02280029095709324,
+            0.002941036131232977,
+            -0.001241082325577736,
+            -0.03649285435676575,
+            0.009130958467721939,
+            0.01981084980070591,
+            0.02151467651128769,
+            -0.01861817203462124,
+            0.0024143988266587257,
+            0.01048627495765686,
+            0.004766840487718582,
+            0.038351576775312424,
+            0.027121812105178833,
+            -0.044206541031599045,
+            0.011663463898003101,
+            0.012995545752346516,
+            -0.009363298304378986,
+            0.048419639468193054,
+            -0.017518429085612297,
+            0.017673322930932045,
+            0.008240322582423687,
+            -0.01798310875892639,
+            0.02318752370774746,
+            -0.01469937153160572,
+            -0.013282098807394505,
+            0.00650551775470376,
+            0.009192915633320808,
+            -0.0001770381786627695,
+            -0.037577107548713684,
+            -0.017859194427728653,
+            -0.005719434004276991,
+            0.0038297362625598907,
+            0.01562873087823391,
+            0.013498948886990547,
+            0.009967382065951824,
+            0.005161818582564592,
+            -0.00945623405277729,
+            -0.0016089539276435971,
+            0.008124152198433876,
+            -0.0005532593932002783,
+            0.0019216446671634912,
+            -0.01926872320473194,
+            0.02698240801692009,
+            0.02572777308523655,
+            -0.0018354853382334113,
+            -0.04652994126081467,
+            -0.0036709706764668226,
+            0.0037077576853334904,
+            -0.017735280096530914,
+            -0.0035315665882080793,
+            -0.04519785940647125,
+            -0.01810702309012413,
+            0.0143121387809515,
+            -0.014614180661737919,
+            -0.0460033044219017,
+            0.009076746180653572,
+            -0.013886182568967342,
+            0.0063970922492444515,
+            -0.00932457484304905,
+            0.005762029904872179,
+            0.003651608945801854,
+            0.005680711008608341,
+            -0.005909178406000137,
+            0.05108380317687988,
+            -0.03466511517763138,
+            0.028051171451807022,
+            0.008565598167479038,
+            -0.012437930330634117,
+            -0.02261441759765148,
+            0.00887538492679596,
+            -0.044981006532907486,
+            -0.033209118992090225,
+            0.016743963584303856,
+            -0.01781272701919079,
+            0.011074868962168694,
+            0.06161654368042946,
+            0.007330324035137892,
+            -0.023342415690422058,
+            -0.01233724970370531,
+            -0.021049996837973595,
+            -0.04191412031650543,
+            0.003442503046244383,
+            0.005463860463351011,
+            -0.004948840010911226,
+            -0.03807276859879494,
+            -0.021716037765145302,
+            0.04615819454193115,
+            -0.012004229240119457,
+            0.012422440573573112,
+            0.01080380566418171,
+            -0.0019410063978284597,
+            -0.04860550910234451,
+            -0.021019017323851585,
+            0.0395597442984581,
+            0.021886419504880905,
+            -0.008426194079220295,
+            0.0028364830650389194,
+            -0.018277406692504883,
+            -0.003289545886218548,
+            0.006296411622315645,
+            -0.009107724763453007,
+            -0.03562545403838158,
+            0.02067825198173523,
+            0.0054832217283546925,
+            -0.023373395204544067,
+            -0.025805220007896423,
+            -0.03296129032969475,
+            0.009944148361682892,
+            0.02453509531915188,
+            -0.01968693546950817,
+            0.005823987070471048,
+            0.00027203132049180567,
+            -0.000881923595443368,
+            0.00042740863864310086,
+            -0.009959637187421322,
+            -0.019764382392168045,
+            -0.020151615142822266,
+            0.03268248215317726,
+            -0.0060059865936636925,
+            0.0074387495405972,
+            -0.03630698472261429,
+            -0.010974188335239887,
+            0.011322698555886745,
+            -0.002395037328824401,
+            -0.013452480547130108,
+            -0.009092235006392002,
+            -0.01739451475441456,
+            -0.009882191196084023,
+            -0.007876322604715824,
+            0.022660886868834496,
+            0.006168624851852655,
+            0.005409647710621357,
+            0.0023795480374246836,
+            -0.008581086993217468,
+            0.02898053266108036,
+            0.017874684184789658,
+            -0.014188224449753761,
+            -0.00920066051185131,
+            0.03451022133231163,
+            0.02665713243186474,
+            -0.009115469641983509,
+            0.01798310875892639,
+            0.023264970630407333,
+            -0.0010484338272362947,
+            -0.01196550577878952,
+            0.001922612776979804,
+            -0.0009550138493068516,
+            -0.0032972905319184065,
+            0.00845717266201973,
+            -0.015365412458777428,
+            0.0315207801759243,
+            0.04287445917725563,
+            -0.002145271748304367,
+            0.009587894193828106,
+            -0.003727119415998459,
+            -0.008472662419080734,
+            0.017069239169359207,
+            0.009316830895841122,
+            0.04625113308429718,
+            0.01098193321377039,
+            0.005680711008608341,
+            -0.033983584493398666,
+            -0.001229465357027948,
+            -0.01184933539479971,
+            0.03308520466089249,
+            0.02061629481613636,
+            0.00889087375253439,
+            -0.008379725739359856,
+            0.01344473659992218,
+            0.008341003209352493,
+            0.00661007035523653,
+            0.020569827407598495,
+            -0.016542602330446243,
+            -0.017286090180277824,
+            0.030467506498098373,
+            -0.00672236829996109,
+            0.01289486512541771,
+            0.02029101923108101,
+            -0.002089123008772731,
+            0.010842529125511646,
+            0.07100307941436768,
+            -0.006184114143252373,
+            0.015713922679424286,
+            -0.019082851707935333,
+            0.0015789433382451534,
+            -0.007640110794454813,
+            -0.03370477631688118,
+            0.018246427178382874,
+            0.0039962464943528175,
+            0.00865078903734684,
+            0.01198099460452795,
+            0.009053511545062065,
+            -0.007946024648845196,
+            -0.00963436160236597,
+            0.015195030719041824,
+            0.0019080914789810777,
+            0.00033568276558071375,
+            0.001336922636255622,
+            -0.0065210070461034775,
+            -0.004011735785752535,
+            -0.021948376670479774,
+            0.02763296104967594,
+            0.007140580099076033,
+            0.030606910586357117,
+            -0.055296897888183594,
+            0.030095763504505157,
+            -0.00789568480104208,
+            0.02554190158843994,
+            -0.016201836988329887,
+            -0.011090358719229698,
+            0.001609921921044588,
+            0.018153492361307144,
+            0.004491904750466347,
+            0.006462921854108572,
+            0.011888058856129646,
+            0.023233991116285324,
+            -0.03444826602935791,
+            0.017053749412298203,
+            0.023110076785087585,
+            0.0048675211146473885,
+            0.0003535923024173826,
+            0.005165690556168556,
+            0.015458349138498306,
+            0.007198664825409651,
+            -0.023048119619488716,
+            -0.0025692922063171864,
+            0.017611365765333176,
+            -0.02306360937654972,
+            0.0036980770528316498,
+            -0.006923729553818703,
+            0.004623564425855875,
+            -0.0014705180656164885,
+            -0.03509881719946861,
+            -0.00100099784322083,
+            -0.025743262842297554,
+            0.01923774555325508,
+            0.009556914679706097,
+            -0.025867177173495293,
+            0.003932353109121323,
+            0.025650326162576675,
+            -0.0003695656778290868,
+            0.00821708794683218,
+            0.0004683101433329284,
+            0.00869725737720728,
+            -0.00974278710782528,
+            -0.01493945624679327,
+            0.004801691509783268,
+            0.01531120017170906,
+            -0.01493945624679327,
+            -0.007431004662066698,
+            -0.009510447271168232,
+            0.030560443177819252,
+            0.0006674932083114982,
+            0.004189862869679928,
+            -0.005045648664236069,
+            -0.01858719252049923,
+            -0.0032237162813544273,
+            -0.021142931655049324,
+            0.004367990419268608,
+            -0.0009046735358424485,
+            0.04197607934474945,
+            0.0038161829579621553,
+            -0.02029101923108101,
+            0.0032972905319184065,
+            -0.03136589005589485,
+            0.0005000148667022586,
+            -0.007276111748069525,
+            -0.02004319056868553,
+            0.0015005286550149322,
+            0.0016525175888091326,
+            0.012631546705961227,
+            -0.03255856782197952,
+            -0.009301341138780117,
+            0.018416810780763626,
+            -0.00020680672605521977,
+            -0.010687636211514473,
+            -0.017828216776251793,
+            0.00035189816844649613,
+            -0.011338187381625175,
+            -0.027973726391792297,
+            -0.02850036323070526,
+            -0.0026719090528786182,
+            0.013491204008460045,
+            -0.018277406692504883,
+            -0.014993669465184212,
+            0.0031249718740582466,
+            0.021592123433947563,
+            -0.009781510569155216,
+            0.008302279748022556,
+            0.02241305634379387,
+            -0.008255811408162117,
+            -0.005514200776815414,
+            -0.000941460661124438,
+            -0.009130958467721939,
+            0.004267309792339802,
+            0.006300284061580896,
+            0.043339136987924576,
+            -0.032403673976659775,
+            -0.0018238683696836233,
+            0.014815541915595531,
+            0.0034153966698795557,
+            0.0025905899237841368,
+            0.002503462601453066,
+            -0.0007841471815481782,
+            0.01344473659992218,
+            0.01178737822920084,
+            0.015466093085706234,
+            0.00100099784322083,
+            -0.0033011629711836576,
+            0.006408709101378918,
+            -0.021173911169171333,
+            0.015721667557954788,
+            0.027958236634731293,
+            -0.0033999073784798384,
+            0.005525817628949881,
+            -0.013731288723647594,
+            -0.008759214542806149,
+            0.013134949840605259,
+            0.0026641644071787596,
+            -0.005068882368505001,
+            -0.009394276887178421,
+            0.017967619001865387,
+            0.019082851707935333,
+            0.006668155547231436,
+            -0.037453193217515945,
+            -0.02898053266108036,
+            -0.009959637187421322,
+            0.020569827407598495,
+            0.014916222542524338,
+            0.0010339126456528902,
+            -0.010540487244725227,
+            -0.009595638141036034,
+            -0.011903548613190651,
+            0.010989678092300892,
+            0.02721474878489971,
+            -0.032372694462537766,
+            0.0008214184199459851,
+            -0.029383255168795586,
+            0.03068435750901699,
+            0.0020000594668090343,
+            0.011446612887084484,
+            -0.0008185141487047076,
+            -0.015140817500650883,
+            -0.029181893914937973,
+            0.013824224472045898,
+            -0.004395096562802792,
+            0.001024231780320406,
+            -0.0031753121875226498,
+            0.0201051477342844,
+            0.012887120246887207,
+            -0.0014472840121015906,
+            0.01968693546950817,
+            0.008178364485502243,
+            0.016480645164847374,
+            -0.010153254494071007,
+            -0.04135650396347046,
+            0.019965743646025658,
+            0.007562664337456226,
+            0.014219203032553196,
+            0.004739734344184399,
+            0.004588713403791189,
+            -0.020771188661456108,
+            0.002724185585975647,
+            0.004333139397203922,
+            0.014405074529349804,
+            0.011028401553630829,
+            0.0025615475606173277,
+            0.011678952723741531,
+            -0.02228914201259613,
+            0.01929970271885395,
+            -0.01631026156246662,
+            0.01006806269288063,
+            0.004445436876267195,
+            -0.026873983442783356,
+            0.007547175046056509,
+            0.007822110317647457,
+            -0.0019187404541298747,
+            -0.017549408599734306,
+            0.02160761132836342,
+            -0.0034831624943763018,
+            -0.03509881719946861,
+            0.0360281765460968,
+            -0.0043447562493383884,
+            0.019346170127391815,
+            0.03209388628602028,
+            -0.01771979033946991,
+            -0.01993476413190365,
+            0.00906900130212307,
+            0.010408828034996986,
+            -0.002873270306736231,
+            0.017859194427728653,
+            -0.002919738180935383,
+            -0.02232012152671814,
+            0.017766257748007774,
+            -0.028918573632836342,
+            -0.014234691858291626,
+            0.020337486639618874,
+            -0.03624502569437027,
+            -0.0021026763133704662,
+            0.03175312280654907,
+            -0.024070415645837784,
+            0.005734923295676708,
+            0.005196669604629278,
+            -0.0019351978553459048,
+            0.0012236569309607148,
+            -0.006451305001974106,
+            0.03801080957055092,
+            -0.010865762829780579,
+            0.0201051477342844,
+            0.01929970271885395,
+            -0.023388884961605072,
+            0.01977987214922905,
+            -0.001109423115849495,
+            0.01675945147871971,
+            -0.003940097521990538,
+            0.03466511517763138,
+            -0.006017603911459446,
+            -0.011330443434417248,
+            0.006431943271309137,
+            0.008488151244819164,
+            -0.009959637187421322,
+            0.008681767620146275,
+            -0.018122512847185135,
+            0.027152791619300842,
+            0.02091059274971485,
+            -0.0002691270783543587,
+            0.013800990767776966,
+            0.0004421719058882445,
+            -0.020833145827054977,
+            0.00789568480104208,
+            -0.027121812105178833,
+            -0.009967382065951824,
+            0.01646515540778637,
+            -0.005762029904872179,
+            0.007380664348602295,
+            -0.02064727433025837,
+            0.028004704043269157,
+            0.0077524082735180855,
+            -0.013591884635388851,
+            -0.0008824076503515244,
+            0.023760627955198288,
+            -0.005432881414890289,
+            0.009355553425848484,
+            0.02070923149585724,
+            0.015713922679424286,
+            0.013615119270980358,
+            0.016263794153928757,
+            0.03351890295743942,
+            0.005328328814357519,
+            0.005568413063883781,
+            0.003519949736073613,
+            0.016991792246699333,
+            -0.008557853288948536,
+            0.018029578030109406,
+            0.0008093173382803798,
+            -0.013576395809650421,
+            0.021499186754226685,
+            -0.0018684001406654716,
+            -0.013622863218188286,
+            -0.0040891822427511215,
+            -0.004337011836469173,
+            0.03271345794200897,
+            0.022041313350200653,
+            -0.004468671046197414,
+            -0.024442158639431,
+            -0.00394784240052104,
+            0.016682006418704987,
+            -0.003167567541822791,
+            0.011136826127767563,
+            0.0007952801533974707,
+            -0.01586107164621353,
+            -0.009595638141036034,
+            0.0063583687879145145,
+            0.010586955584585667,
+            -0.008921852335333824,
+            0.010346870869398117,
+            -0.0228312686085701,
+            -0.004011735785752535,
+            0.00046613195445388556,
+            -0.004844287410378456,
+            0.0053438181057572365,
+            -0.011314953677356243,
+            0.004948840010911226,
+            0.016062432900071144,
+            0.01026167906820774,
+            0.004832670092582703,
+            0.006989559158682823,
+            0.012066186405718327,
+            0.009161937050521374,
+            -0.01627928391098976,
+            0.00428667152300477,
+            -0.005072754807770252,
+            -0.02345084212720394,
+            0.005831731949001551,
+            0.00902253296226263,
+            -0.01122201792895794,
+            0.009386532939970493,
+            -0.010958699509501457,
+            0.03296129032969475,
+            0.01030814740806818,
+            -0.037391237914562225,
+            -0.0033379499800503254,
+            -0.0016244432190433145,
+            0.014885243959724903,
+            0.012298526242375374,
+            0.008550108410418034,
+            -0.017518429085612297,
+            -0.0047823297791182995,
+            -0.009835722856223583,
+            0.0012827098835259676,
+            -0.012306271120905876,
+            0.017456471920013428,
+            0.022567950189113617,
+            -0.036337960511446,
+            0.01968693546950817,
+            -0.010432061739265919,
+            -0.023652203381061554,
+            0.020476890727877617,
+            0.011500826105475426,
+            0.01942361705005169,
+            -0.009402021765708923,
+            0.018896980211138725,
+            -0.0001331114035565406,
+            0.025510922074317932,
+            -0.004228586331009865,
+            -0.022087780758738518,
+            -0.012499887496232986,
+            -0.03209388628602028,
+            -0.0051811798475682735,
+            0.006819176487624645,
+            -0.00045524101005867124,
+            0.013111716136336327,
+            0.014226946979761124,
+            -0.025712283328175545,
+            0.007295473478734493,
+            0.032589543610811234,
+            0.014637414366006851,
+            0.023311438038945198,
+            -0.011276230216026306,
+            -0.0010794125264510512,
+            -0.0026060794480144978,
+            -0.013770012184977531,
+            -0.03212486580014229,
+            -0.0017493258928880095,
+            -0.036585792899131775,
+            0.005680711008608341,
+            -0.016387708485126495,
+            -0.013096226379275322,
+            0.023264970630407333,
+            -0.034138478338718414,
+            0.0014608372002840042,
+            0.013708055019378662,
+            0.010540487244725227,
+            0.012391461990773678,
+            -0.00162637943867594,
+            0.015032391995191574,
+            -0.029243851080536842,
+            -0.010284913703799248,
+            -0.004491904750466347,
+            -0.01981084980070591,
+            -0.007338068913668394,
+            -0.018060555681586266,
+            -0.013127204962074757,
+            -0.003566417610272765,
+            -0.010060317814350128,
+            0.002247888594865799,
+            0.0032062907703220844,
+            -0.014714861288666725,
+            0.03206290677189827,
+            0.003157886676490307,
+            0.018958937376737595,
+            0.0033418224193155766,
+            0.004449309315532446,
+            0.018788553774356842,
+            -0.0026331855915486813,
+            -0.0058278595097362995,
+            0.030963165685534477,
+            -0.00032672801171429455,
+            0.008759214542806149,
+            0.019082851707935333,
+            0.004724245052784681,
+            0.0038529701996594667,
+            -0.0260530486702919,
+            0.012887120246887207,
+            -0.027447087690234184,
+            0.005080499220639467,
+            -0.0077175572514534,
+            -0.024302754551172256,
+            -0.028128618374466896,
+            -0.006571347359567881,
+            0.012058441527187824,
+            -0.02898053266108036,
+            -0.00613377382978797,
+            0.002315654419362545,
+            -0.00400399137288332,
+            -0.002673845272511244,
+            0.007241260726004839,
+            0.029150914400815964,
+            -0.033209118992090225,
+            0.008751469664275646,
+            0.005595519673079252,
+            -0.0005053392960689962,
+            0.013297587633132935,
+            0.008038961328566074,
+            0.003343758638948202,
+            0.020848635584115982,
+            0.006637176964432001,
+            0.002458930714055896,
+            -0.0025576751213520765,
+            -0.014242436736822128,
+            0.004000118933618069,
+            0.02154565416276455,
+            0.007562664337456226,
+            0.018385831266641617,
+            -0.008914108388125896,
+            0.002621568739414215,
+            -0.01455222349613905,
+            0.005552923772484064,
+            -0.002867461647838354,
+            0.004309905227273703,
+            -0.012236569076776505,
+            0.008093173615634441,
+            0.0028771425131708384,
+            -0.0043447562493383884,
+            0.018060555681586266,
+            -0.002319526858627796,
+            0.016480645164847374,
+            0.004015608225017786,
+            0.015620986931025982,
+            -0.0035412474535405636,
+            0.01178737822920084,
+            0.0005731051205657423,
+            -0.017905661836266518,
+            -0.017022771760821342,
+            -0.039249956607818604,
+            0.007702067960053682,
+            0.03531566634774208,
+            -0.0009564659558236599,
+            -0.03444826602935791,
+            0.010145509615540504,
+            -0.011942271143198013,
+            -0.009975126944482327,
+            -0.022862248122692108,
+            0.013862947933375835,
+            0.012430185452103615,
+            0.007415515370666981,
+            0.012252057902514935,
+            -0.006997303571552038,
+            0.01026167906820774,
+            -0.005216030869632959,
+            0.008356492035090923,
+            -0.0010823167394846678,
+            -0.037422213703393936,
+            0.012344993650913239,
+            0.02216522768139839,
+            -0.002635121811181307,
+            0.0016089539276435971,
+            -0.006145390681922436,
+            0.022103270515799522,
+            -0.013971373438835144,
+            -0.00650551775470376,
+            -0.00046443779137916863,
+            0.00797700323164463,
+            -0.01795213110744953,
+            0.004778457339853048,
+            0.0755879208445549,
+            -0.007775642443448305,
+            0.006122156511992216,
+            0.009270362555980682,
+            -0.015148562379181385,
+            -0.01637221872806549,
+            -0.004561606794595718,
+            -0.020260039716959,
+            0.008844406343996525,
+            -0.0018819533288478851,
+            -0.0110981035977602,
+            0.012468908913433552,
+            0.0053050946444272995,
+            0.002031038049608469,
+            -0.008379725739359856,
+            0.010811550542712212,
+            0.009525936096906662,
+            0.010432061739265919,
+            0.02360573597252369,
+            0.011477591469883919,
+            -0.00858883187174797,
+            -0.011795123107731342,
+            0.01129171997308731,
+            0.0028577810153365135,
+            0.016697494313120842,
+            0.007907302118837833,
+            0.0010648912284523249,
+            0.019501063972711563,
+            -0.002309845993295312,
+            0.02180897258222103,
+            -2.2689444449497387e-05,
+            -0.009301341138780117,
+            -0.004236331209540367,
+            -0.015349923633038998,
+            0.024271776899695396,
+            -0.0020155487582087517,
+            0.0015450604259967804,
+            -0.020213572308421135,
+            0.0013979118084535003,
+            0.013243375346064568,
+            0.011957760900259018,
+            -0.0020407189149409533,
+            -0.0051347119733691216,
+            0.0007154133054427803,
+            -0.016356728971004486,
+            0.032806396484375,
+            0.007578153628855944,
+            0.0007681738352403045,
+            0.0008838597568683326,
+            0.0009288755827583373,
+            -0.0032972905319184065,
+            0.0024608669336885214,
+            -0.004766840487718582,
+            0.009549170732498169,
+            -0.010943209752440453,
+            -0.010796061716973782,
+            0.010075807571411133,
+            -0.02402394637465477,
+            0.003566417610272765,
+            -0.00558777479454875,
+            -0.008062195032835007,
+            0.0076207490637898445,
+            0.00015537731815129519,
+            -0.007392281666398048,
+            -0.0030939930584281683,
+            0.0023408245760947466,
+            0.013429246842861176,
+            0.004406713880598545,
+            0.010850274004042149,
+            0.012956822291016579,
+            0.005359307397156954,
+            -0.006629432085901499,
+            0.015512561425566673,
+            -0.006946963258087635,
+            -0.004228586331009865,
+            -0.0012468908680602908,
+            -0.019996721297502518,
+            -0.0009419447160325944,
+            -0.004708755761384964,
+            0.0015092412941157818,
+            0.0001705036120256409,
+            -8.749050175538287e-05,
+            -0.0018490385264158249,
+            -0.011988739483058453,
+            -0.001831613015383482,
+            -0.011663463898003101,
+            -0.014498010277748108,
+            0.010253935120999813,
+            -0.0041201612912118435,
+            -0.022939693182706833,
+            -0.015349923633038998,
+            -0.020662762224674225,
+            0.02145271934568882,
+            -0.004387352149933577,
+            -0.005800753366202116,
+            0.006733985152095556,
+            -0.010215211659669876,
+            -0.015706177800893784,
+            -0.008069939911365509,
+            -0.0205853171646595,
+            0.010703125037252903,
+            0.003214035416021943,
+            0.0006791101768612862,
+            0.00882891658693552,
+            0.014397329650819302,
+            -0.01589204929769039,
+            0.0111755495890975,
+            0.012182355858385563,
+            0.011361422017216682,
+            -0.006854027509689331,
+            0.0008911203476600349,
+            -0.014103032648563385,
+            0.02596011385321617,
+            0.021623101085424423,
+            0.005169562995433807,
+            -0.04132552444934845,
+            -0.005862710531800985,
+            -0.013344055972993374,
+            -0.010524998418986797,
+            -0.012120398692786694,
+            0.011771888472139835,
+            -0.0038820127956569195,
+            0.0013117524795234203,
+            0.006288666743785143,
+            0.0015063370810821652,
+            0.004367990419268608,
+            -0.0022382077295333147,
+            -0.007709812838584185,
+            -0.03212486580014229,
+            -0.02817508764564991,
+            -0.00863530021160841,
+            0.020461400970816612,
+            0.002838419284671545,
+            -0.0009453330421820283,
+            -0.0036380558740347624,
+            0.0027048238553106785,
+            -0.005882072262465954,
+            -0.005018542055040598,
+            -0.014753584749996662,
+            -0.012739972211420536,
+            0.0101300198584795,
+            0.00802347157150507,
+            0.014195968396961689,
+            -0.011950016021728516,
+            0.0028190575540065765,
+            0.018788553774356842,
+            0.007690451107919216,
+            -0.023357905447483063,
+            0.01135367713868618,
+            -0.010912231169641018,
+            -0.009270362555980682,
+            -0.009433000348508358,
+            -0.009471723809838295,
+            0.0016980175860226154,
+            -0.008317768573760986,
+            0.003945906180888414,
+            -0.03615209087729454,
+            -0.011330443434417248,
+            -0.0013901671627536416,
+            -0.010346870869398117,
+            0.0009864765452221036,
+            0.02524760365486145,
+            0.006815304048359394,
+            0.01432762760668993,
+            -0.0006733016925863922,
+            0.017502939328551292,
+            -0.0018984107300639153,
+            0.018726596608757973,
+            0.012763205915689468,
+            -0.00797700323164463,
+            0.011717676185071468,
+            0.014738094992935658,
+            -0.0033863543067127466,
+            0.015620986931025982,
+            -0.010168743319809437,
+            0.005285732913762331,
+            -0.009556914679706097,
+            -0.003214035416021943,
+            0.002495717955753207,
+            0.030901208519935608,
+            -0.003618694143369794,
+            0.00858883187174797,
+            0.01098193321377039,
+            -0.015241498127579689,
+            0.0026951429899781942,
+            0.014962689951062202,
+            0.01141563430428505,
+            0.007760153152048588,
+            0.0024434414226561785,
+            -0.024225307628512383,
+            0.004193735308945179,
+            0.005037903785705566,
+            0.0031520782504230738,
+            -0.014366351068019867,
+            0.0021859314292669296,
+            0.015675200149416924,
+            0.008550108410418034,
+            -0.025325050577521324,
+            0.016418687999248505,
+            0.0009138703462667763,
+            -0.003909118939191103,
+            0.029119934886693954,
+            -0.011268485337495804,
+            -0.005723306443542242,
+            0.009463978931307793,
+            0.0009980935137718916,
+            0.0030029932968318462,
+            0.008937342092394829,
+            0.0228312686085701,
+            -0.003988501615822315,
+            -0.032341714948415756,
+            -0.022459525614976883,
+            0.015706177800893784,
+            -0.0047823297791182995,
+            -0.022939693182706833,
+            -0.026703599840402603,
+            -0.003113354789093137,
+            -0.02228914201259613,
+            -0.0022323993034660816,
+            -0.020058678463101387,
+            -0.014056564308702946,
+            0.001268188701942563,
+            0.029987338930368423,
+            0.016201836988329887,
+            0.008224832825362682,
+            -0.011051635257899761,
+            -0.029956359416246414,
+            -0.024271776899695396,
+            0.01810702309012413,
+            -0.007055388763546944,
+            -0.005541306920349598,
+            0.00591692328453064,
+            -0.009951893240213394,
+            -0.008480406366288662,
+            0.011802867986261845,
+            0.027137301862239838,
+            -0.002747419523075223,
+            -0.029987338930368423,
+            -0.02688947319984436,
+            -0.014335372485220432,
+            -0.01840132102370262,
+            -0.017704300582408905,
+            -0.008627555333077908,
+            0.0045035220682621,
+            0.00030300996149890125,
+            0.016186347231268883,
+            0.014962689951062202,
+            -0.000480653194244951,
+            0.0028597170021384954,
+            -2.7711375878425315e-05,
+            -0.001763847190886736,
+            0.006888878531754017,
+            -0.004933350719511509,
+            0.026997897773981094,
+            0.014614180661737919,
+            -0.006594581063836813,
+            0.018711106851696968,
+            0.028314489871263504,
+            0.020430423319339752,
+            0.003157886676490307,
+            -0.010695381090044975,
+            -0.014606435783207417,
+            0.009727297350764275,
+            -0.013421501964330673,
+            0.023264970630407333,
+            0.022552460432052612,
+            -0.01480779703706503,
+            -0.014885243959724903,
+            -0.017905661836266518,
+            0.015210519544780254,
+            -0.007562664337456226,
+            -0.008224832825362682,
+            0.012368228286504745,
+            0.022335609421133995,
+            -0.007024410180747509,
+            0.002371803391724825,
+            -0.0010232636705040932,
+            -0.021731525659561157,
+            -0.006122156511992216,
+            -0.004596457816660404,
+            -0.008666278794407845,
+            0.008681767620146275,
+            0.008131897076964378,
+            -0.001928421319462359,
+            -0.006319645792245865,
+            0.005471604876220226,
+            -0.013367289677262306,
+            0.01301103550940752,
+            -0.00789568480104208,
+            0.03379771113395691,
+            -0.009138703346252441,
+            -0.0006733016925863922,
+            -0.022598929703235626,
+            -0.011446612887084484,
+            0.00845717266201973,
+            0.0017967619933187962,
+            0.01682141050696373,
+            -0.0011626676423475146,
+            -0.005688455421477556,
+            0.009913169778883457,
+            -0.0010687635513022542,
+            -0.001450188341550529,
+            -0.006892750971019268,
+            -0.010401083156466484,
+            0.0002884887217078358,
+            0.013258864171802998,
+            -0.006025348324328661,
+            0.0018054747488349676,
+            0.006095050368458033,
+            -0.0175339188426733,
+            0.029027000069618225,
+            0.023884542286396027,
+            0.017255110666155815,
+            0.014660648070275784,
+            0.0006394187803380191,
+            0.011647974140942097,
+            -0.003572226269170642,
+            0.0038820127956569195,
+            -0.020337486639618874,
+            0.021080974489450455,
+            -0.010811550542712212,
+            0.01846327818930149,
+            0.011655719019472599,
+            -0.015233753249049187,
+            -0.005022414494305849,
+            0.021406250074505806,
+            0.009967382065951824,
+            0.00889087375253439,
+            0.0017541663255542517,
+            -0.02470547705888748,
+            0.017766257748007774,
+            0.010099041275680065,
+            -0.020399443805217743,
+            0.019547531381249428,
+            -0.009471723809838295,
+            0.01233724970370531,
+            -0.0005300253978930414,
+            0.004418330732733011,
+            0.0058743273839354515,
+            0.021499186754226685,
+            0.011570527218282223,
+            -0.03884723410010338,
+            -0.013615119270980358,
+            -0.0017357728211209178,
+            0.006857899948954582,
+            -0.0036303112283349037,
+            -0.015233753249049187,
+            -0.041635312139987946,
+            0.01604694314301014,
+            0.009998360648751259,
+            -0.01643417589366436,
+            0.004987563472241163,
+            -0.002427952131256461,
+            -0.0042053526267409325,
+            -0.008170620538294315,
+            0.003186929039657116,
+            -0.004170501604676247,
+            0.02324948087334633,
+            0.0018984107300639153,
+            -0.006946963258087635,
+            0.013258864171802998,
+            0.01344473659992218,
+            -0.021948376670479774,
+            0.0009874446550384164,
+            0.0021375271026045084,
+            0.000636998622212559,
+            0.023683181032538414,
+            -0.014707116410136223,
+            -0.0017657832941040397,
+            -0.01685238815844059,
+            -0.0013069120468571782,
+            -0.021592123433947563,
+            -0.001188805908896029,
+            -0.015187285840511322,
+            -0.008511385880410671,
+            0.025030752643942833,
+            -0.011005166918039322,
+            0.010997422970831394,
+            -0.006687517277896404,
+            0.006943091284483671,
+            -0.006904367823153734,
+            -0.01936165988445282,
+            -0.0008267428493127227,
+            0.009975126944482327,
+            -0.000512115890160203,
+            0.002941036131232977,
+            8.482827252009884e-05,
+            0.002464739140123129,
+            0.008867640048265457,
+            0.006234454456716776,
+            -0.0075278133153915405,
+            0.02489134855568409,
+            -0.012073930352926254,
+            0.0016399325104430318,
+            0.011911292560398579,
+            0.0004743606550619006,
+            -0.010424317792057991,
+            0.025588368996977806,
+            -0.02139076218008995,
+            -0.00800798274576664,
+            -0.00014799568452872336,
+            -0.0016544538084417582,
+            -0.006513262167572975,
+            0.006718495860695839,
+            0.02946070022881031,
+            0.0062654330395162106,
+            0.010447551496326923,
+            -0.0281441081315279,
+            -0.011493081226944923,
+            0.012538610957562923,
+            -0.022273652255535126,
+            -0.0007739823195151985,
+            -0.0004322490422055125,
+            -0.00619960343465209,
+            0.026378324255347252,
+            -0.02022906206548214,
+            -0.01093546487390995,
+            -0.018664639443159103,
+            0.005405775271356106,
+            -0.011121337302029133,
+            0.008294534869492054,
+            -0.0009777637897059321,
+            0.006540368776768446,
+            0.016805920749902725,
+            -0.019563021138310432,
+            0.016031453385949135,
+            0.005119222681969404,
+            -0.016217326745390892,
+            0.010222956538200378,
+            -0.017580386251211166,
+            -0.0008199662552215159,
+            0.020275529474020004,
+            -0.00400399137288332,
+            0.008751469664275646,
+            -0.003363120136782527,
+            0.005208286456763744,
+            -0.006513262167572975,
+            0.0001492057926952839,
+            -0.007249005138874054,
+            0.014428308233618736,
+            -0.016805920749902725,
+            -0.017828216776251793,
+            -0.008821171708405018,
+            0.0016709112096577883,
+            0.021297825500369072,
+            0.011136826127767563,
+            -0.010044828988611698,
+            -0.011059380136430264,
+            -0.011694442480802536,
+            0.041604332625865936,
+            0.005762029904872179,
+            0.027230238541960716,
+            -0.02261441759765148,
+            0.00545998802408576,
+            0.025170156732201576,
+            -0.014242436736822128,
+            0.0012788375606760383,
+            -0.0022788674104958773,
+            0.0053050946444272995,
+            0.011562783271074295,
+            -0.01819995976984501,
+            0.0049139889888465405,
+            -0.00019216447253711522,
+            -0.00782985519617796,
+            -0.000656844291370362,
+            0.003442503046244383,
+            0.00020196630794089288,
+            0.016325751319527626,
+            -0.004027225077152252,
+            -0.016511622816324234,
+            -0.004429947584867477,
+            -0.0003216455806978047,
+            -0.027896279469132423,
+            0.013514438644051552,
+            0.016589069738984108,
+            0.01566745527088642,
+            0.027989214286208153,
+            -0.014513500034809113,
+            0.016031453385949135,
+            0.009533680975437164,
+            0.013901671394705772,
+            -0.010610189288854599,
+            0.008952830918133259,
+            -0.01595400646328926,
+            0.007446494419127703,
+            -0.03454120084643364,
+            -0.007887939922511578,
+            -0.00802347157150507,
+            0.01984182931482792,
+            0.028763681650161743,
+            0.002923610620200634,
+            0.006493900436908007,
+            0.000196883876924403,
+            0.0024202074855566025,
+            -0.01714668609201908,
+            -0.020182594656944275,
+            -0.0003337466041557491,
+            0.0029332914855331182,
+            0.0006950835813768208,
+            -0.004495777189731598,
+            0.00472037261351943,
+            -0.012391461990773678,
+            0.0026835259050130844,
+            -0.013134949840605259,
+            0.005971135571599007,
+            -0.0059982421807944775,
+            0.009603383019566536,
+            0.0019429425010457635,
+            0.020864123478531837,
+            0.02300165221095085,
+            0.004100799560546875,
+            0.0046971384435892105,
+            -0.024333734065294266,
+            0.00920066051185131,
+            -0.013847459107637405,
+            0.012329504825174809,
+            0.019408127292990685,
+            0.008248066529631615,
+            0.006571347359567881,
+            0.01233724970370531,
+            0.012918098829686642,
+            -0.0031772484071552753,
+            0.024287264794111252,
+            0.005014669615775347,
+            -0.0009070937521755695,
+            0.002052335999906063,
+            -0.014954946003854275,
+            -0.0070979841984808445,
+            -0.0015286030247807503,
+            0.0018848575418815017,
+            -0.004731989465653896,
+            0.01906736195087433,
+            0.015256987884640694,
+            -0.01050176378339529,
+            0.017874684184789658,
+            -0.03937387093901634,
+            0.017379024997353554,
+            -0.0027377386577427387,
+            -0.0060911779291927814,
+            -0.00545998802408576,
+            0.00665653869509697,
+            0.004515138920396566,
+            0.003669034456834197,
+            0.03773200139403343,
+            -0.010408828034996986,
+            0.003963331691920757,
+            -0.005847221240401268,
+            0.007121218368411064,
+            -0.020864123478531837,
+            0.008302279748022556,
+            0.005336073227226734,
+            0.02160761132836342,
+            -0.0056497324258089066,
+            -0.0014482521219179034,
+            0.02328045852482319,
+            0.008859895169734955,
+            0.03469609469175339,
+            -0.01566745527088642,
+            -0.004418330732733011,
+            0.01048627495765686,
+            0.0207402091473341,
+            -0.00580462533980608,
+            -0.008976065553724766,
+            0.007903429679572582,
+            0.0035586729645729065,
+            0.021824462339282036,
+            -0.00515407370403409,
+            -0.0006650729919783771,
+            0.0054367538541555405,
+            -0.0060911779291927814,
+            0.009735042229294777,
+            0.011252996511757374,
+            -0.009076746180653572,
+            0.007539430167526007,
+            0.021530164405703545,
+            -0.009975126944482327,
+            -0.008658533915877342,
+            -0.009649851359426975,
+            0.0010658593382686377,
+            -0.03068435750901699,
+            0.001649613375775516,
+            0.005359307397156954,
+            0.015187285840511322,
+            -0.0004603234410751611,
+            -0.009014789015054703,
+            0.021019017323851585,
+            0.017255110666155815,
+            -0.005936284549534321,
+            -0.0028132491279393435,
+            0.0023795480374246836,
+            -0.0017154429806396365,
+            -0.006160879973322153,
+            -0.0006248975405469537,
+            -0.002656419761478901,
+            0.014730350114405155,
+            -0.0015469965292140841,
+            -0.006617815233767033,
+            -0.00756653631106019,
+            0.0033302053343504667,
+            0.010834784246981144,
+            -0.005661349277943373,
+            0.013057502917945385,
+            0.016805920749902725,
+            -0.00031704717548564076,
+            0.019671445712447166,
+            0.01037010457366705,
+            -0.010447551496326923,
+            0.015938518568873405,
+            -0.007984748110175133,
+            0.006563602481037378,
+            0.0164031982421875,
+            -0.01277095079421997,
+            -0.01246116403490305,
+            0.007210282143205404,
+            0.00665653869509697,
+            0.028918573632836342,
+            0.014792308211326599,
+            -0.02109646424651146,
+            0.014908477663993835,
+            0.015768134966492653,
+            -0.013018779456615448,
+            0.003537375247105956,
+            -0.01784370467066765,
+            0.020260039716959,
+            0.019051872193813324,
+            -0.015535795129835606,
+            -0.017224133014678955,
+            -0.014947201125323772,
+            -0.005092116538435221,
+            0.006331262644380331,
+            0.0033263331279158592,
+            -0.00617636926472187,
+            -0.027989214286208153,
+            -0.0049139889888465405,
+            -0.018633661791682243,
+            0.013870692811906338,
+            -0.008674023672938347,
+            0.0007715621031820774,
+            -0.004511266481131315,
+            0.007984748110175133,
+            -0.0033882902935147285,
+            0.032465629279613495,
+            -0.0051811798475682735,
+            -0.02832997962832451,
+            0.004398969002068043,
+            0.016356728971004486,
+            0.012623801827430725,
+            -0.029243851080536842,
+            0.010463040322065353,
+            0.004805563949048519,
+            -0.012530866079032421,
+            0.005990497302263975,
+            -0.02258343994617462,
+            -0.007632365915924311,
+            -0.0009395244996994734,
+            -0.016743963584303856,
+            -0.027121812105178833,
+            -0.008433938957750797,
+            -0.006993431597948074,
+            -0.005401902832090855,
+            -0.009525936096906662,
+            -0.008720491081476212,
+            -0.011477591469883919,
+            0.02303262986242771,
+            -0.007899557240307331,
+            -0.001866463921032846,
+            -0.0014637414133176208,
+            0.00930908601731062,
+            -0.023776117712259293,
+            -0.015605497173964977,
+            0.006389347370713949,
+            -0.002594462363049388,
+            0.022815778851509094,
+            -0.012747716158628464,
+            0.005940156988799572,
+            -0.009316830895841122,
+            0.009851212613284588,
+            0.015241498127579689,
+            0.004956584889441729,
+            0.017440982162952423,
+            -0.009889936074614525,
+            0.0021026763133704662,
+            -0.008627555333077908,
+            -0.00018986526993103325,
+            -1.027075541060185e-05,
+            0.014428308233618736,
+            0.007624621503055096,
+            0.014141756109893322,
+            0.017487451434135437,
+            -0.013553161174058914,
+            0.005003052763640881,
+            -0.0003274540649726987,
+            -0.017255110666155815,
+            0.0027725896798074245,
+            -0.007984748110175133,
+            -0.0008896682411432266,
+            0.001758038648404181,
+            6.395397940650582e-05,
+            -0.02112744189798832,
+            -0.0009124181815423071,
+            0.0018809852190315723,
+            0.0010222955606877804,
+            -0.025201136246323586,
+            0.00889087375253439,
+            -0.00889087375253439,
+            -0.0014918158994987607,
+            -0.019113829359412193,
+            0.019795360043644905,
+            0.026300877332687378,
+            -0.004619691986590624,
+            0.00036206303047947586,
+            -0.013537672348320484,
+            -0.02049238048493862,
+            0.0001489637652412057,
+            -0.011314953677356243,
+            -0.011345932260155678,
+            -0.0022169100120663643,
+            -0.003214035416021943,
+            0.008805682882666588,
+            0.0004189379105810076,
+            -0.009433000348508358,
+            -0.004956584889441729,
+            -0.011764144524931908,
+            -0.017549408599734306,
+            -0.010401083156466484,
+            0.01933068037033081,
+            -0.0018016024259850383,
+            0.0020794421434402466,
+            -0.0032508226577192545,
+            0.0005271211848594248,
+            0.011748654767870903,
+            0.013475715182721615,
+            0.013344055972993374,
+            -0.023915521800518036,
+            -0.017890173941850662,
+            -0.013320821337401867,
+            0.0029487807769328356,
+            -0.007187047973275185,
+            0.004844287410378456,
+            0.026362836360931396,
+            -0.0009695350890979171,
+            -0.01091997604817152,
+            0.002327271504327655,
+            0.019051872193813324,
+            -0.031272951513528824,
+            -0.013390523381531239,
+            -0.025077221915125847,
+            0.02225816436111927,
+            -0.013351799920201302,
+            0.011516314931213856,
+            0.010191977955400944,
+            -0.025944624096155167,
+            0.006768836174160242,
+            -0.007686578668653965,
+            0.026037558913230896,
+            0.01634124107658863,
+            0.0005934348446317017,
+            -0.00944074522703886,
+            0.013622863218188286,
+            -0.0008722427301108837,
+            -0.013576395809650421,
+            0.002201420720666647,
+            0.014645159244537354,
+            -0.014691627584397793,
+            -0.009510447271168232,
+            -0.004507394507527351,
+            0.004313777666538954,
+            0.012763205915689468,
+            -0.0030339721124619246,
+            -0.004031097516417503,
+            -0.011624740436673164,
+            0.013344055972993374,
+            -0.003930416889488697,
+            0.012383717112243176,
+            0.007380664348602295,
+            -0.014056564308702946,
+            -0.019532041624188423,
+            0.033333033323287964,
+            0.00400399137288332,
+            0.00338054564781487,
+            -0.010997422970831394,
+            0.02740062028169632,
+            -0.009897680021822453,
+            0.010571465827524662,
+            -0.006408709101378918,
+            0.0031985461246222258,
+            0.012321759946644306,
+            -0.012104909867048264,
+            -0.01981084980070591,
+            0.021468207240104675,
+            -0.00364192808046937,
+            -0.0010193913476541638,
+            0.008310024626553059,
+            -0.020182594656944275,
+            -0.00663330452516675,
+            0.002180122770369053,
+            0.009541425853967667,
+            -0.011818356812000275,
+            0.006369986105710268,
+            0.017905661836266518,
+            0.003912991378456354,
+            -0.0007672057254239917,
+            0.019996721297502518,
+            -0.01672847382724285,
+            0.011717676185071468,
+            -0.010850274004042149,
+            0.0016825281782075763,
+            0.0006065039779059589,
+            -0.0048597767017781734,
+            0.0058743273839354515,
+            -0.014536733739078045,
+            0.011988739483058453,
+            -6.244135147426277e-05,
+            -0.013979118317365646,
+            0.007601387333124876,
+            -0.013359544798731804,
+            -0.016836898401379585,
+            0.041542377322912216,
+            -0.005301222205162048,
+            -0.0038665232714265585,
+            -0.0056962003000080585,
+            0.01350669376552105,
+            0.02823704481124878,
+            -0.005336073227226734,
+            -0.01153180468827486,
+            -0.010780571959912777,
+            -0.016573579981923103,
+            0.015907539054751396,
+            -0.01810702309012413,
+            0.002495717955753207,
+            -0.010532742366194725,
+            -0.007903429679572582,
+            0.004302160814404488,
+            -0.003516077296808362,
+            0.014722606167197227,
+            0.00950270239263773,
+            0.023357905447483063,
+            0.00797700323164463,
+            -0.00027348342700861394,
+            0.010393339209258556,
+            0.011663463898003101,
+            0.003243078012019396,
+            -0.010509508661925793,
+            -0.00011949774489039555,
+            -0.02894955314695835,
+            -0.008449427783489227,
+            -0.004863648675382137,
+            -0.001485039247199893,
+            -0.00428667152300477,
+            -0.01239920686930418,
+            0.006617815233767033,
+            -0.0022052929271012545,
+            0.013909416273236275,
+            -0.009254872798919678,
+            -0.017766257748007774,
+            0.03389064967632294,
+            0.004495777189731598,
+            0.006466794293373823,
+            -0.008286789990961552,
+            -0.03705047070980072,
+            -0.0032566310837864876,
+            -0.008116407319903374,
+            -0.013529927469789982,
+            0.004538373090326786,
+            0.01283290795981884,
+            -0.014831030741333961,
+            -0.028097640722990036,
+            0.0019390701781958342,
+            -0.006683644838631153,
+            0.02978597767651081,
+            -0.007798876613378525,
+            0.00419760774821043,
+            0.0037251831963658333,
+            0.0011558911064639688,
+            0.01239920686930418,
+            -0.00845717266201973,
+            -0.0022536972537636757,
+            0.016387708485126495,
+            -0.010858018882572651,
+            0.011051635257899761,
+            -0.010966443456709385,
+            -0.02554190158843994,
+            -0.01456771232187748,
+            0.021344292908906937,
+            -0.0004823473282158375,
+            -0.00024903935263864696,
+            0.014823286794126034,
+            0.02354377694427967,
+            0.011260741390287876,
+            0.00969631876796484,
+            0.006408709101378918,
+            -0.001035848748870194,
+            0.005975008010864258,
+            0.006095050368458033,
+            -0.0024163350462913513,
+            -0.013398268260061741,
+            0.017564896494150162,
+            0.009773765690624714,
+            -0.01810702309012413,
+            0.0008514289511367679,
+            0.0036612898111343384,
+            0.0168833676725626,
+            0.006935346405953169,
+            0.009316830895841122,
+            -0.0143121387809515,
+            -0.023760627955198288,
+            -0.00014932679187040776,
+            0.002873270306736231,
+            0.015721667557954788,
+            0.010579210706055164,
+            -0.014319883659482002,
+            0.0005963391158729792,
+            0.008712747134268284,
+            0.020213572308421135,
+            -0.024209819734096527,
+            0.019702425226569176,
+            -0.004673904739320278,
+            -0.0021084847394376993,
+            0.007946024648845196,
+            0.011632485315203667,
+            -0.011872569099068642,
+            -0.00650551775470376,
+            0.006249943748116493,
+            0.003157886676490307,
+            0.03172214329242706,
+            -0.0017260919557884336,
+            0.009665340185165405,
+            -0.010997422970831394,
+            -0.0002405686245765537,
+            0.0006728176376782358,
+            -0.0006820144481025636,
+            -0.002884887158870697,
+            -0.011020656675100327,
+            0.0018974426202476025,
+            -0.020306508988142014,
+            0.0024473138619214296,
+            0.00463130883872509,
+            -0.005684582982212305,
+            -0.0032972905319184065,
+            -0.019222255796194077,
+            -0.0022672503255307674,
+            0.003293418325483799,
+            0.014335372485220432,
+            -0.031691163778305054,
+            0.004972074180841446,
+            0.010672146454453468,
+            -0.002309845993295312,
+            -0.010416572913527489,
+            0.011423379182815552,
+            -0.0026060794480144978,
+            0.011988739483058453,
+            -0.0038142469711601734,
+            -0.0004905760288238525,
+            0.025681305676698685,
+            -0.005308967083692551,
+            0.014707116410136223,
+            -0.01026167906820774,
+            -0.002993312431499362,
+            -0.028252532705664635,
+            0.005235392600297928,
+            -0.007403898518532515,
+            -0.0012081675231456757,
+            -0.004402841441333294,
+            -0.01130720879882574,
+            -0.0026990154292434454,
+            0.002122037811204791,
+            0.008000237867236137,
+            0.012910354882478714,
+            -0.0007725302129983902,
+            -0.010075807571411133,
+            0.004832670092582703,
+            0.0075355577282607555,
+            -0.015520306304097176,
+            -0.0070437719114124775,
+            -0.020306508988142014,
+            0.029662061482667923,
+            -0.013731288723647594,
+            -0.0011123273288831115,
+            -0.020864123478531837,
+            0.010160998441278934,
+            -0.0063119009137153625,
+            -0.002782270545139909,
+            0.004708755761384964,
+            -0.015109838917851448,
+            0.01344473659992218,
+            0.008751469664275646,
+            0.01450575515627861,
+            -0.02151467651128769,
+            -0.01762685552239418,
+            -0.0044725434854626656,
+            -0.002735802438110113,
+            -0.010648912750184536,
+            -0.0005334137240424752,
+            0.002013612538576126,
+            0.021251358091831207,
+            -0.00032503384863957763,
+            0.0042053526267409325,
+            -0.003214035416021943,
+            0.017224133014678955,
+            -0.010137764737010002,
+            -0.013057502917945385,
+            0.0024260159116238356,
+            0.005994369741529226,
+            0.016186347231268883,
+            0.03158273920416832,
+            -0.020693741738796234,
+            -0.00272999401204288,
+            0.0032740565948188305,
+            0.010656657628715038,
+            -0.004085310269147158,
+            6.304640555754304e-05,
+            -0.027803342789411545,
+            -0.04464024305343628,
+            -0.002422143705189228,
+            0.013057502917945385,
+            -0.015636475756764412,
+            0.012670270167291164,
+            0.008743725717067719,
+            0.010021595284342766,
+            -0.005196669604629278,
+            0.02620794251561165,
+            -0.004793947096914053,
+            0.014304393902420998,
+            0.011733165942132473,
+            0.011911292560398579,
+            0.0009874446550384164,
+            0.0007459079497493804,
+            -0.018354853615164757,
+            -0.008255811408162117,
+            -0.0015954007394611835,
+            0.01765783317387104,
+            -0.027478067204356194,
+            0.008526874706149101,
+            -0.011516314931213856,
+            0.013839714229106903,
+            0.0003666614356916398,
+            0.005661349277943373,
+            -0.01771979033946991,
+            0.012445674277842045,
+            -0.0009337160154245794,
+            -0.025325050577521324,
+            0.004805563949048519,
+            0.011276230216026306,
+            -0.010362359695136547,
+            0.006877261213958263,
+            0.019950253888964653,
+            0.0023079097736626863,
+            0.008495896123349667,
+            -0.014126266352832317,
+            0.010703125037252903,
+            0.009649851359426975,
+            -0.021080974489450455,
+            -0.01233724970370531,
+            -0.007059261202812195,
+            -0.006335135083645582,
+            -0.0035470561124384403,
+            -0.00013105422840453684,
+            0.010176488198339939,
+            -0.016000475734472275,
+            0.010099041275680065,
+            -0.006559730041772127,
+            0.006400964688509703,
+            0.00944074522703886,
+            0.001115231541916728,
+            -0.013576395809650421,
+            0.012716737575829029,
+            -0.0218709297478199,
+            0.008426194079220295,
+            -0.04287445917725563,
+            -0.005847221240401268,
+            -0.015078860335052013,
+            0.005204414017498493,
+            -0.003798757679760456,
+            -0.004453181754797697,
+            -0.006203475873917341,
+            -0.005061137955635786,
+            0.00920066051185131,
+            0.021747015416622162,
+            0.012135888449847698,
+            -0.016620049253106117,
+            -0.007551047019660473,
+            0.005425137002021074,
+            0.0004854935978073627,
+            -0.013336311094462872,
+            0.009316830895841122,
+            -0.01576039008796215,
+            0.0009259713697247207,
+            0.006339007057249546,
+            -0.01072635967284441,
+            0.013452480547130108,
+            0.004731989465653896,
+            -0.003401843598112464,
+            0.013390523381531239,
+            -0.0050417762249708176,
+            -0.0057891360484063625,
+            -0.007272239308804274,
+            -0.013739033602178097,
+            -0.003818119177594781,
+            -0.003072695340961218,
+            0.012747716158628464,
+            0.001529571134597063,
+            -0.0003243077953811735,
+            -0.00782985519617796,
+            0.009154192171990871,
+            0.002220782218500972,
+            -0.0013146566925570369,
+            0.005773646757006645,
+            -0.02300165221095085,
+            0.0001287550403503701,
+            0.004755223635584116,
+            -0.0009487213101238012,
+            -0.006246071308851242,
+            -0.012151377275586128,
+            -0.011903548613190651,
+            -0.038165703415870667,
+            0.009363298304378986,
+            0.0019197085639461875,
+            -0.0076207490637898445,
+            0.0064822835847735405,
+            0.017255110666155815,
+            0.01456771232187748,
+            0.016743963584303856,
+            -0.020693741738796234,
+            -0.02303262986242771,
+            0.005556796211749315,
+            -0.009402021765708923,
+            0.02261441759765148,
+            0.004046586807817221,
+            -0.005847221240401268,
+            0.0013243375578895211,
+            -0.013258864171802998,
+            0.008929597213864326,
+            0.005723306443542242,
+            -0.001592496526427567,
+            0.005425137002021074,
+            -0.013692565262317657,
+            -0.010904486291110516,
+            0.01048627495765686,
+            0.032496608793735504,
+            -0.0004261985304765403,
+            0.0024105266202241182,
+            0.023295948281884193,
+            0.015466093085706234,
+            0.016356728971004486,
+            -0.021111954003572464,
+            0.011167804710566998,
+            0.010060317814350128,
+            -0.0018645278178155422,
+            -0.014892988838255405,
+            0.00513083953410387,
+            -0.012716737575829029,
+            -0.010664401575922966,
+            0.014993669465184212,
+            0.007853088900446892,
+            0.017735280096530914,
+            -0.02369867078959942,
+            0.00789568480104208,
+            0.00042765066609717906,
+            0.00013710475468542427,
+            0.011524059809744358,
+            0.012956822291016579,
+            0.03370477631688118,
+            0.027152791619300842,
+            0.020662762224674225,
+            -0.009285852313041687,
+            0.014366351068019867,
+            -0.009192915633320808,
+            -0.011090358719229698,
+            -0.019470084458589554,
+            -0.015249243006110191,
+            -0.011136826127767563,
+            0.017998598515987396,
+            0.008790193125605583,
+            -0.009355553425848484,
+            -0.005947901867330074,
+            0.01586107164621353,
+            -0.003909118939191103,
+            -0.004635181277990341,
+            0.01646515540778637,
+            -0.026269899681210518,
+            0.016836898401379585,
+            -0.000745423894841224,
+            -0.014668392948806286,
+            -0.014691627584397793,
+            -0.004182118456810713,
+            -0.012670270167291164,
+            0.0003964299685321748,
+            -0.020817656069993973,
+            0.013181418180465698,
+            0.024597052484750748,
+            -0.0030920570716261864,
+            -0.004100799560546875,
+            -0.011299464851617813,
+            -0.0005111477803438902,
+            0.018385831266641617,
+            0.0034773540683090687,
+            0.009192915633320808,
+            -0.0023872926831245422,
+            0.013924905098974705,
+            0.00524313747882843,
+            0.0027958236169070005,
+            -0.0007565568666905165,
+            -0.016263794153928757,
+            0.012197845615446568,
+            0.0026835259050130844,
+            0.012135888449847698,
+            -0.022242674604058266,
+            0.007938280701637268,
+            -0.011764144524931908,
+            -0.0023698671720921993,
+            -0.02228914201259613,
+            0.0189744271337986,
+            -0.01610890030860901,
+            0.001046497724018991,
+            0.003841353114694357,
+            0.0006224773242138326,
+            0.009580149315297604,
+            0.01610890030860901,
+            -0.023388884961605072,
+            -0.0003460896550677717,
+            -0.014908477663993835,
+            -0.021917399019002914,
+            -0.01607792265713215,
+            0.004728117026388645,
+            -0.019501063972711563,
+            0.005444498732686043,
+            -0.0025557391345500946,
+            -0.015125328674912453,
+            0.018478767946362495,
+            0.014691627584397793,
+            -0.004379607271403074,
+            0.002412462839856744,
+            0.005425137002021074,
+            -0.0014405074762180448,
+            0.01971791312098503,
+            0.00013274837692733854,
+            -0.0010048701660707593,
+            -0.009425255469977856,
+            -0.0168833676725626,
+            -0.0065752193331718445,
+            -0.035005878657102585,
+            0.0260530486702919,
+            -0.004670032300055027,
+            -0.016511622816324234,
+            0.014319883659482002,
+            0.009115469641983509,
+            -0.014629669487476349,
+            -0.007059261202812195,
+            0.016294771805405617,
+            -0.024287264794111252,
+            0.02244403585791588,
+            -0.009611127898097038,
+            -0.008774704299867153,
+            0.00784921646118164,
+            0.05154848098754883,
+            0.009146448224782944,
+            0.003492843359708786,
+            0.011036145500838757,
+            0.005839476361870766,
+            -0.0013165927957743406,
+            -0.023683181032538414,
+            0.005568413063883781,
+            -0.007190920412540436,
+            -0.003601268632337451,
+            0.002536377403885126,
+            -0.004801691509783268,
+            0.031149037182331085,
+            -0.004379607271403074,
+            -0.005289605353027582,
+            0.004484160337597132,
+            -0.02196386642754078,
+            -0.0001332324172835797,
+            0.023203013464808464,
+            -2.3128106477088295e-05,
+            -0.023590246215462685,
+            -0.006571347359567881,
+            -0.015071115456521511,
+            -0.013770012184977531,
+            -0.018989915028214455,
+            0.005231520161032677,
+            -0.012755461037158966,
+            -0.0033940989524126053,
+            0.0016660707769915462,
+            -0.005940156988799572,
+            -0.002582845278084278,
+            0.009603383019566536,
+            0.05362405255436897,
+            0.014103032648563385,
+            -0.006335135083645582,
+            -0.005854965653270483,
+            0.009192915633320808,
+            -0.010524998418986797,
+            -0.019795360043644905,
+            -0.009820234030485153,
+            -0.029507169499993324,
+            -0.012747716158628464,
+            -0.0143121387809515,
+            -0.010191977955400944,
+            -0.010277168825268745,
+            0.00963436160236597,
+            -0.018122512847185135,
+            -0.007628493942320347,
+            -0.020337486639618874,
+            0.002941036131232977,
+            -0.011268485337495804,
+            -0.014714861288666725,
+            -0.008054450154304504,
+            -0.04807887226343155,
+            0.005796880926936865,
+            0.019593998789787292,
+            0.01263929158449173,
+            0.002515079453587532,
+            0.004383479710668325,
+            -0.01993476413190365,
+            -0.030792783945798874,
+            0.013839714229106903,
+            0.011283975094556808,
+            0.011733165942132473,
+            0.03283737599849701,
+            -0.011709931306540966,
+            -0.00813964195549488,
+            -0.004693266469985247,
+            -0.0017251238459721208,
+            0.012066186405718327,
+            0.00969631876796484,
+            -0.004917861428111792,
+            -0.012375972233712673,
+            0.016217326745390892,
+            -0.01981084980070591,
+            -0.023683181032538414,
+            -0.01672847382724285,
+            0.006157007534056902,
+            0.00813964195549488,
+            -0.008054450154304504,
+            0.002973950933665037,
+            0.0056962003000080585,
+            -0.007159941829741001,
+            -0.00022423222253564745,
+            0.004890755284577608,
+            -0.023048119619488716,
+            0.0008490087930113077,
+            -0.002257569460198283,
+            0.0061376458033919334,
+            0.021266845986247063,
+            -0.010168743319809437,
+            -0.0014075926737859845,
+            -0.0053902859799563885,
+            -0.0028984404634684324,
+            -0.006764963734894991,
+            0.02620794251561165,
+            0.009154192171990871,
+            -0.009882191196084023,
+            -0.01011453103274107,
+            -0.01129171997308731,
+            -0.0011278166202828288,
+            -0.017255110666155815,
+            -0.016186347231268883,
+            0.0015566773945465684,
+            -0.016712984070181847,
+            0.013971373438835144,
+            -0.00876695942133665,
+            0.00400399137288332,
+            -0.014110777527093887,
+            0.02019808255136013,
+            0.01929970271885395,
+            -0.009711808525025845,
+            0.023776117712259293,
+            -0.007709812838584185,
+            -0.038382552564144135,
+            -0.00800798274576664,
+            0.01843230053782463,
+            -0.006118284538388252,
+            0.010037084110081196,
+            -0.011314953677356243,
+            0.008209343999624252,
+            0.00015319912927225232,
+            0.0018122512847185135,
+            -0.0003112386621069163,
+            0.027555514127016068,
+            0.015535795129835606,
+            -0.005560668651014566,
+            -0.005494839046150446,
+            0.004255692940205336,
+            0.009541425853967667,
+            0.017580386251211166,
+            0.012151377275586128,
+            -0.021019017323851585,
+            -0.014257925562560558,
+            -0.016263794153928757,
+            0.011012911796569824,
+            0.015125328674912453,
+            -0.0040891822427511215,
+            0.01691434532403946,
+            0.014126266352832317,
+            -0.019098341464996338,
+            0.0019884423818439245,
+            -0.008193854242563248,
+            -0.00158184755127877,
+            -0.03029712475836277,
+            0.011043890379369259,
+            0.0074387495405972,
+            -0.0072218989953398705,
+            0.036616768687963486,
+            0.009858956560492516,
+            0.037577107548713684,
+            0.018680129200220108,
+            0.0008974128868430853,
+            0.009045767597854137,
+            -0.018370341509580612,
+            0.013065247796475887,
+            0.01500141341239214,
+            -0.012724482454359531,
+            -0.005734923295676708,
+            0.0013446672819554806,
+            0.0007551047019660473,
+            0.012631546705961227,
+            0.02004319056868553,
+            -0.01795213110744953,
+            0.009773765690624714,
+            -0.015032391995191574,
+            -0.005250881891697645,
+            0.006555858068168163,
+            -0.013932649977505207,
+            -0.008495896123349667,
+            -0.00963436160236597,
+            0.019408127292990685,
+            -0.012600568123161793,
+            0.016573579981923103,
+            -0.009595638141036034,
+            0.015520306304097176,
+            -0.029724018648266792,
+            -0.003669034456834197,
+            -0.006679772399365902,
+            -0.019593998789787292,
+            -0.012887120246887207,
+            0.009820234030485153,
+            0.010633422993123531,
+            -0.009301341138780117,
+            -0.011562783271074295,
+            0.020368466153740883,
+            -0.004185990896075964,
+            0.013529927469789982,
+            0.021189400926232338,
+            0.00737291993573308,
+            0.024132372811436653,
+            0.0005605200421996415,
+            0.014164989814162254,
+            0.005463860463351011,
+            0.010362359695136547,
+            -0.004677776712924242,
+            0.005161818582564592,
+            0.00932457484304905,
+            0.014203713275492191,
+            0.017766257748007774,
+            0.013196907006204128,
+            -0.00598275288939476,
+            0.007419387809932232,
+            0.02016710489988327,
+            0.0016254113288596272,
+            -0.030312614515423775,
+            -0.009719553403556347,
+            0.0001736498816171661,
+            0.00924712885171175,
+            0.009185170754790306,
+            3.817877222900279e-05,
+            -0.01093546487390995,
+            0.011144571006298065,
+            0.00969631876796484,
+            0.002286612056195736,
+            0.006280922330915928,
+            -0.01456771232187748,
+            -0.020957060158252716,
+            0.006106667220592499,
+            0.01798310875892639,
+            0.013917161151766777,
+            -0.014722606167197227,
+            0.022784801200032234,
+            0.006145390681922436,
+            -0.021406250074505806,
+            -0.01840132102370262,
+            -0.0009559819591231644,
+            0.0004542729293461889,
+            -0.002265314105898142,
+            0.03720536455512047,
+            -0.0011539548868313432,
+            -0.005955646280199289,
+            -0.0054754773154854774,
+            0.02775687538087368,
+            0.02049238048493862,
+            0.009177426807582378
+          ],
+          "embedding": [
+            0.0008215984936325879,
+            -0.019349418284688353,
+            -0.02322660633029795,
+            0.009747820845194852,
+            0.007353771220903641,
+            0.015252908783199839,
+            -0.012176175829980202,
+            0.051898051094966605,
+            -0.0068130353886146165,
+            0.008189212816890595,
+            0.008102287831931905,
+            0.01682801136011614,
+            0.02005557996049711,
+            -0.020779879580497183,
+            0.002332165268329781,
+            0.02337485065616377,
+            0.006074992895540845,
+            0.02576575146535689,
+            0.0007517286429767504,
+            -0.03763909942314134,
+            0.0011204729720314833,
+            0.00041212613589632806,
+            -0.06089176800314795,
+            0.012852035873395388,
+            -0.008901127092058543,
+            0.01702347055484216,
+            0.005803446395466281,
+            0.017889570275475296,
+            -0.05375047382413552,
+            0.021623778774252724,
+            0.014784317465883424,
+            -0.0018881880857885708,
+            0.0004735328554290247,
+            0.012393802850407656,
+            -0.019234442968055952,
+            -0.019566283611638122,
+            0.04407789373296535,
+            -0.0053565873209102995,
+            -0.026240866282480922,
+            0.048107902270426675,
+            -0.0054681314939387235,
+            -0.0009441397948041965,
+            -0.015205079626883548,
+            0.006885981841893229,
+            0.01403724256866746,
+            -0.026086548246792754,
+            0.0034778946051893067,
+            0.00044587533948775473,
+            0.032851328435891634,
+            -0.03882097333255334,
+            -0.011126344604254894,
+            -0.01648943336384904,
+            0.035810456555802166,
+            -0.019384589526165754,
+            -0.019430702768170285,
+            0.014297440272239852,
+            -0.007883453204663151,
+            0.050996900793086844,
+            0.06007149434153715,
+            0.03406832053483379,
+            0.025193439934671852,
+            0.00027785129663604897,
+            -0.008014391056168083,
+            -0.018806730908490114,
+            0.027193788903098558,
+            -0.006195006670328595,
+            -0.020466000344352826,
+            0.0068939884219589185,
+            0.017252276167086602,
+            -0.045073444648704136,
+            -0.049242269917908754,
+            0.023021140866008313,
+            -0.015676872472071106,
+            -0.024497517908898602,
+            0.005873633718973869,
+            -0.022828585938203683,
+            0.011748341004940436,
+            0.04624205727927451,
+            -0.015442165499674641,
+            0.018565551071491767,
+            0.015246505395507096,
+            0.004119269796224144,
+            0.004862735250819252,
+            -0.000145323321960046,
+            0.05155838952887077,
+            0.0027052423345323936,
+            -0.016218063670586664,
+            -0.07462470920944148,
+            0.043695974746295135,
+            -0.027608273720377708,
+            -0.03860912774695494,
+            0.002823110565150435,
+            0.026471469849508766,
+            -0.0026063577189074103,
+            -0.029230652014151735,
+            0.05684164470477843,
+            0.00022182868300572677,
+            0.015590845552956364,
+            -0.024939617549529242,
+            0.01035803286203917,
+            -0.010225298012384327,
+            0.019571847648759157,
+            0.01841613041441054,
+            0.0156981470538354,
+            0.021764297694793343,
+            -0.018831230007033416,
+            -0.014419529638789993,
+            -0.0029965475899652273,
+            0.028881185012095884,
+            0.09163526460163937,
+            0.022644302207811818,
+            -0.017933909936330133,
+            -0.04663928709016869,
+            0.004962951444105128,
+            -0.041333041174526026,
+            -0.027545628007617527,
+            -0.037142394579894954,
+            -0.013358143162788402,
+            0.02719686432396677,
+            -0.0034971800937504205,
+            0.04413635817371732,
+            0.04903857110328917,
+            0.03186067155630788,
+            0.007831051050269414,
+            -0.025257538623313574,
+            0.023517513773109414,
+            -0.0016527403209505964,
+            -0.0016101860707429347,
+            0.014448515185903624,
+            0.009905018312477684,
+            0.0024114582235163955,
+            0.00172043851384822,
+            -0.03818316221102173,
+            -0.02304537008110578,
+            0.01024969233570089,
+            -0.01901116356327339,
+            0.019791121460848986,
+            0.07985067340821365,
+            -0.051903191272236665,
+            -0.025924646976556942,
+            0.014766511532849252,
+            -0.002073086076894999,
+            0.030959269926800852,
+            0.006394417986617089,
+            -0.028177586693713442,
+            0.010725285336674526,
+            0.0009614052276924192,
+            0.07500253788978564,
+            -0.00036794182149960157,
+            0.004058458332826519,
+            -0.013218269166951234,
+            0.027613539268683245,
+            -0.012995170442480582,
+            0.023388707018488407,
+            0.026389423313105165,
+            -0.003068639956061266,
+            -0.02661967649632026,
+            0.026145274570635913,
+            -0.03300591322174331,
+            -0.004556072066733024,
+            0.020045722084112896,
+            0.0033437343914784865,
+            -0.010224220905483085,
+            -0.0025025498684079703,
+            0.03460103245395426,
+            0.04514362357278504,
+            0.03797031988058962,
+            -0.004586662475576956,
+            -0.06351723722596923,
+            0.06037983484369874,
+            0.019415753124363902,
+            0.028850618647994476,
+            -0.0034810237045849006,
+            0.02957938154808314,
+            0.02642652523231673,
+            0.013187653724133275,
+            -0.007856320804109569,
+            -0.007065882659482447,
+            -0.016672784625197085,
+            0.029020161154209707,
+            -0.013243058921970134,
+            -0.014900932834241732,
+            0.006001769901896033,
+            0.008648456186944271,
+            0.03484763413464497,
+            0.012279368984306564,
+            0.02379643508687133,
+            0.04933044959399941,
+            -0.021047723935647383,
+            -0.007264823701706426,
+            0.0007058774795716792,
+            -0.010512071832195085,
+            0.027489671668749157,
+            0.009731157570361245,
+            -0.03195919640758579,
+            -0.0033626031972691915,
+            -0.04282419484651207,
+            0.01873977910485268,
+            -0.014398285316695653,
+            0.012854631447494207,
+            -0.0017372959978092432,
+            -0.037331867825735776,
+            0.001088209292238142,
+            -0.012345684323522151,
+            0.037096063326414734,
+            0.005050219690024652,
+            0.0008906051515353004,
+            -0.003697709449630797,
+            0.006548479142737107,
+            -0.010727663712299397,
+            -0.007486226350610755,
+            -0.02704618481069485,
+            -0.019752552016312926,
+            -0.014632581296303486,
+            -0.0037501138119724717,
+            0.014588411795703245,
+            -0.007119137071719376,
+            0.016440258608998535,
+            -0.03686645352111013,
+            -0.023546402448713415,
+            0.005628703432152666,
+            0.024988145751166143,
+            0.011971199595808573,
+            -0.010868778329190568,
+            -0.03270046433624425,
+            -0.021950559285357267,
+            -0.011889914022657738,
+            -0.005123971523701009,
+            0.02619713302280345,
+            -0.021990810305904974,
+            0.01165257474485436,
+            -0.009117901804315863,
+            -0.007492082755249399,
+            0.010725902070291232,
+            0.01862811996181951,
+            -0.026694281868322685,
+            -0.01623776415010199,
+            -0.009403997538684658,
+            -0.02207329027097839,
+            0.025939540497196416,
+            0.014178803271471762,
+            -0.001141821318145744,
+            -0.006281187017652108,
+            -0.035993365198554964,
+            0.0026192776111449633,
+            -0.015254976771874396,
+            -0.0116872453009606,
+            -0.015780016491993135,
+            0.021421328283124968,
+            0.02889432250124136,
+            -0.01871009416296692,
+            -0.026540472640514536,
+            0.015703370682110186,
+            -0.0002849235245164438,
+            -0.02594930889995505,
+            0.04491828744809679,
+            0.044063522197410734,
+            -0.037441864491547405,
+            -0.007005339187980476,
+            -0.030381794393931814,
+            0.06901083845539985,
+            -0.017604082400603037,
+            -0.04060591381445593,
+            0.06044850935418443,
+            -0.01756490171206357,
+            0.025539717592705295,
+            0.03452018789549939,
+            0.0016217577522523927,
+            0.00018825054264059277,
+            -0.002563606092068712,
+            0.04770277039047993,
+            -0.0032010143255574255,
+            -0.03425328601475849,
+            -0.02105909767072799,
+            -0.00840881022563103,
+            -0.028699447677549002,
+            -0.015925160861589006,
+            -0.0055750742281166,
+            0.017513987082479483,
+            0.010399718005952185,
+            0.02267201252273823,
+            0.01258511878601025,
+            0.0015081978484714903,
+            0.01874981572091931,
+            0.011158889421057756,
+            -0.002607810686437211,
+            -0.003732023411827874,
+            -0.007787189972165309,
+            -0.05524012485122728,
+            -0.025110689689279583,
+            -0.03946383236703947,
+            0.004443572529325475,
+            -0.018745077598477586,
+            0.005926160888274727,
+            0.039430331582190524,
+            -0.0049500324516682,
+            0.004352795787899792,
+            0.015566139991780173,
+            0.013148156356408315,
+            -0.015279560337042242,
+            -0.03316846614871165,
+            0.0001592209771257332,
+            0.018362411161743347,
+            -0.012706582418607377,
+            -0.03424477956681259,
+            -0.001991022841226589,
+            0.0033467286047492175,
+            0.03357019284647071,
+            -0.024359110796680374,
+            0.00359117029951049,
+            0.005061818038334802,
+            0.011359582931760315,
+            -0.026793992746244076,
+            -0.018170136079817746,
+            -0.017421603453826397,
+            0.03996608926471779,
+            0.02587013058343799,
+            0.015763410145648683,
+            -0.04512536449983358,
+            -0.02293473890636517,
+            0.048575083783480195,
+            -0.009589860756342604,
+            -0.06486262416944208,
+            0.03447666009581266,
+            0.016105654870253654,
+            0.003526582643722421,
+            -0.01354754623151814,
+            0.05401575248587799,
+            -0.03563190295559527,
+            -0.017365166993914202,
+            -0.0046288709698227735,
+            -0.033657212157212645,
+            -0.007686252059880836,
+            -0.001385273642139782,
+            0.048229062789530236,
+            -0.029083855744728694,
+            -0.030465559177859865,
+            0.0024368661249980416,
+            -0.024144031154493057,
+            0.021693278579520123,
+            -0.016405088055634124,
+            -0.018439935250344958,
+            0.006948603359995389,
+            -0.033943037395199695,
+            -0.003007796423804453,
+            -0.016783195426536633,
+            -0.02270589491206155,
+            0.020874438753828814,
+            0.016724666903417947,
+            0.027033184180344852,
+            0.0019847647852695785,
+            0.017749524526371133,
+            0.0204661893602285,
+            -0.005604929968008821,
+            0.028462234633072944,
+            -0.020790459496727066,
+            -0.04183816355941877,
+            0.03946487892663695,
+            0.01192861758684541,
+            -0.0018024841275380242,
+            0.01077082591611702,
+            -0.01475239020295694,
+            -0.03952356245715099,
+            0.03683950018557526,
+            -0.056767672022551764,
+            -0.007507600090017693,
+            0.009530544352246083,
+            -0.00788856051420074,
+            -0.011204694027562521,
+            -0.010587701466318598,
+            -0.028604940377315504,
+            0.023749537635431462,
+            -0.013672419168235691,
+            0.01953449229707597,
+            -0.011449939491554354,
+            -0.012238180529977605,
+            0.05795536247491174,
+            -0.021518581774137322,
+            -0.040672955795925836,
+            0.026066659238628543,
+            -0.014061106927896723,
+            0.03328357230532424,
+            0.03957107406164427,
+            0.030670947051333917,
+            -0.004278577327978564,
+            0.025885946540364826,
+            0.0319106101668716,
+            -0.003462160036198642,
+            -0.04686868330851868,
+            0.00555303933159587,
+            0.02156143971867743,
+            -0.03739529489075782,
+            -0.01025626126223168,
+            -0.02141970191890338,
+            0.022134910980548407,
+            0.017807484142010496,
+            -0.007232643256393455,
+            0.011318041192005274,
+            -0.010489012862455913,
+            -0.013003217163997554,
+            -0.00847279880392288,
+            0.02357274819910594,
+            -0.0016485066139744577,
+            -0.005873113634992993,
+            0.006424249429163587,
+            0.004008703814301658,
+            0.0054639858216457724,
+            0.010198079500408441,
+            0.017694720520798957,
+            -0.017087021465208255,
+            -0.0025558869389290554,
+            -0.016503579212015177,
+            -0.0420082404922411,
+            -0.003563173424359332,
+            0.022924161700566468,
+            0.012345933179990547,
+            -0.00729908668365632,
+            -0.004234683369642856,
+            -0.0022962545569187934,
+            -0.020666205489143392,
+            -0.014796149043465308,
+            0.024562653223873056,
+            0.030217494971086438,
+            0.007810961433007407,
+            0.007781263640449456,
+            -0.0030348559388780655,
+            0.007310876971216526,
+            0.016099591640649783,
+            0.02311929597780174,
+            -0.04160162038353416,
+            0.027152100845897007,
+            -0.00025396315502635146,
+            0.023717123370023598,
+            -0.04186918759097667,
+            -0.017988531857836703,
+            -0.03106663560912813,
+            -0.023595079699181436,
+            0.02437134274097885,
+            -0.02317559303912261,
+            -0.012327263775257306,
+            0.000989566600374836,
+            0.00954921216572729,
+            -0.005155079395685017,
+            -0.017508404061379164,
+            -0.04582321461886958,
+            -0.0014367264231457263,
+            -0.030417229142190933,
+            -0.017829710557332023,
+            -0.001056472801554354,
+            0.005016912084754242,
+            -0.06276644565618253,
+            0.0109965681725148,
+            -0.07270669271977365,
+            -0.004542796125637244,
+            0.040139998232966566,
+            -0.017890441783630643,
+            0.055909102081208445,
+            -0.05029104308417623,
+            0.01673962668706085,
+            -0.04507293803055318,
+            0.015385506084493484,
+            0.025677372277539627,
+            0.030235171586316813,
+            -0.022189978375852563,
+            0.008461188938299745,
+            0.008033707483259569,
+            -0.045089505755429415,
+            -0.04144698743458133,
+            -0.013640256640978024,
+            0.01036782197518379,
+            0.0012696958402094232,
+            -0.016090728932978384,
+            0.023838437877040163,
+            0.011946482002767135,
+            0.03342604921859589,
+            0.04211801335157251,
+            0.002708015395637621,
+            -0.047156646610976634,
+            -0.013104464812562618,
+            -0.003933116891120681,
+            0.03155321708116329,
+            -0.018866109579446184,
+            0.015376812010630058,
+            -0.01955557376085726,
+            0.016255217383093355,
+            0.017868332102274922,
+            -0.020325179506828335,
+            0.013434364256590097,
+            0.004176904701252156,
+            -0.03877060944639854,
+            0.0006475542749369515,
+            0.00042468300294954125,
+            -0.02116992978038693,
+            0.0032654387271108293,
+            -0.00575003786159731,
+            -0.05149954071159492,
+            -0.031382926943128234,
+            -0.015730563640253065,
+            0.0382312502769352,
+            0.004297939848910064,
+            0.019724983545557365,
+            -0.017907869879607064,
+            -0.005165628493894463,
+            0.03698976707579373,
+            0.008501001373266799,
+            -0.01725527821298066,
+            0.006879620077191753,
+            0.02455792200929138,
+            -0.03224193972301503,
+            -0.039181328884413266,
+            -0.0008206986106407542,
+            -0.023306758893533952,
+            0.01672946960876128,
+            -0.02658587698869307,
+            0.005873620623736886,
+            0.0025331170979801232,
+            -0.03490809296591325,
+            0.02456764314036355,
+            0.003291113462353163,
+            -0.004303423672882112,
+            -0.020359544091059775,
+            -0.016892415165120203,
+            0.017174797487220006,
+            0.04391489511478065,
+            0.007327321684948843,
+            -0.003943713832544148,
+            0.00862684231313155,
+            -0.006361172785753631,
+            0.05826688726534012,
+            -0.02284956969636637,
+            -0.026680023999604643,
+            0.027527436347931256,
+            -0.022597669279069723,
+            -0.006277863878949772,
+            -0.017343446085471857,
+            -0.012065192152203348,
+            -0.0004183712490442143,
+            0.019339311388446943,
+            -0.019423562920897294,
+            -0.010212527447243049,
+            0.004783006556748757,
+            0.012824133870627983,
+            -0.004855945108275706,
+            0.006239949458562236,
+            0.006847581690471813,
+            0.03094005096378827,
+            0.026993131409761086,
+            -0.011331881651539968,
+            0.03297014291876675,
+            0.010265600643742144,
+            4.666113318251278e-05,
+            0.001970879281172325,
+            0.002523664335535017,
+            0.0034107811520368574,
+            -0.03143890864030425,
+            -0.01624584278940617,
+            0.013468729048885906,
+            0.005233309338223403,
+            -0.004550544967856448,
+            -0.023783653541634683,
+            -0.009402494239255264,
+            0.03334923712760375,
+            -0.003260424721959948,
+            0.02748395114281811,
+            -0.0058374897745283065,
+            -0.01985840930273332,
+            0.017595418271610163,
+            0.02881743246631328,
+            -0.05928552440605384,
+            0.017062540894682354,
+            -0.0011508547522925793,
+            0.008953761538519997,
+            0.01610561914042553,
+            -0.008894160513648953,
+            0.013301991014533684,
+            0.04080863883897929,
+            0.010738051773755976,
+            -0.023345002072886145,
+            0.004215726310838674,
+            0.01541749448570608,
+            0.01291813299324752,
+            -0.022747942296528025,
+            0.0008640791861357522,
+            0.011008522066277018,
+            0.005893793007012561,
+            0.0026106450193140156,
+            -0.006278394080583205,
+            -0.025659188287146268,
+            0.024180550333596625,
+            -0.0044491292740074925,
+            0.026967973346698617,
+            0.04390909273429254,
+            0.009596855239692042,
+            -0.03686020608700941,
+            -0.013109951664921729,
+            0.0002799680359670127,
+            -0.0038537577602593366,
+            -0.0037552094562213916,
+            0.011105295862811114,
+            -0.04185615640149602,
+            0.019950890919369135,
+            0.006453241355231516,
+            0.015314645427201533,
+            -0.020031193483734364,
+            -0.024530173488182337,
+            0.0031949828520724337,
+            -0.01492645564652531,
+            0.012875827757040941,
+            0.005057170413681063,
+            -0.004319834710426384,
+            0.005386862713494232,
+            0.00030848054013413157,
+            0.008384941028326983,
+            -0.010811526993301901,
+            0.03175068297908541,
+            0.004680987487229685,
+            0.009264547472281849,
+            -0.02856314241700808,
+            -0.021083721348429348,
+            0.020113396771629096,
+            0.03572366771051171,
+            -0.01708529918264109,
+            0.03346921724975398,
+            -0.009042046660366718,
+            0.0033219155387035227,
+            0.0151720168318272,
+            -0.015805318210244414,
+            -0.008057236707694303,
+            -0.0005483009439015931,
+            -0.002917527004423299,
+            -0.014538511124345527,
+            0.007302979537905532,
+            -0.0208529958115422,
+            -0.02965227242147972,
+            0.020064178921373875,
+            -0.00011892622932349096,
+            0.016455188521316146,
+            0.010335044657747514,
+            -0.007816155992852236,
+            -0.0052262330850905165,
+            0.006959344748883874,
+            0.04432147882994085,
+            0.026734484553994495,
+            0.05547661378166294,
+            -0.005787997952888261,
+            -0.004559444854138438,
+            -0.0082295581123363,
+            -0.006588172755707144,
+            0.04524009609473585,
+            0.015915003261638767,
+            -0.004910285095602864,
+            0.017495607570191165,
+            0.01557886571266685,
+            0.03858259260136037,
+            0.0027648525902558316,
+            -0.016912796336835408,
+            0.04014755669075221,
+            -0.004077405782937414,
+            0.008616827127381418,
+            0.011102692199778469,
+            0.0024185742218798562,
+            0.00807141794538637,
+            0.02996274720815659,
+            -0.012429186019907761,
+            0.01680942269506771,
+            -0.04571916592313758,
+            -0.02775530246832996,
+            0.014370280587419414,
+            -0.004931905540824582,
+            0.03419959665162407,
+            0.01668521021731861,
+            -0.023105305446763272,
+            -0.024761385385514877,
+            0.021529428247380984,
+            -0.002603042335561961,
+            -0.024135326707477325,
+            -0.015906833215844473,
+            0.003286605488908195,
+            -0.0023437997044735617,
+            0.0060251848371355166,
+            -0.013481652076194643,
+            0.0002908426939460218,
+            0.01953362396253743,
+            0.0038544889405965057,
+            0.020760460182873287,
+            -0.016224396121239458,
+            -0.00259233446682179,
+            -0.009282861237653619,
+            -0.02952130765234087,
+            0.017988370145301916,
+            0.006138236653154911,
+            -0.007607182955958238,
+            -0.009516767050028553,
+            0.0048347778733230365,
+            -0.005819209642901871,
+            -0.0025851701689263706,
+            -0.01073014664868326,
+            0.012433465684360061,
+            0.025784014942833097,
+            -0.016875624785552303,
+            0.007944349714123104,
+            8.651589872318479e-05,
+            -0.022306044741692753,
+            -0.022127016745066307,
+            0.020143472307319017,
+            0.002792646249762062,
+            0.008768813346304779,
+            -0.004074214276300631,
+            -0.00015024341785520542,
+            0.002990550758627699,
+            0.01951746553838439,
+            0.03896451381932467,
+            0.004232673242619471,
+            0.027876586543280867,
+            0.0145860303663199,
+            0.03994234200854897,
+            0.009657017763624678,
+            0.00696607158576712,
+            -0.026055387694656488,
+            0.02819890357011778,
+            0.00018297502510753014,
+            0.043190622821775236,
+            0.022488402221902314,
+            -0.0407730101161195,
+            -0.0021946518523759066,
+            0.01669520746276111,
+            0.013212623257794862,
+            0.017009844136059122,
+            -0.004537512994041156,
+            -0.014041213424740049,
+            0.028135404528975894,
+            -0.022859586430516003,
+            0.0008637006862481223,
+            -0.004486332934188017,
+            -0.025031795510001887,
+            0.007351370254175368,
+            -0.004567816916440959,
+            0.030025413528233364,
+            0.0014976740344892735,
+            -0.022285096307825804,
+            0.022837170656498543,
+            -0.009551269792616045,
+            0.011045530488684207,
+            0.008809047185288755,
+            0.020065001354321994,
+            0.031993362733068575,
+            0.009143210434415064,
+            0.02133640814803879,
+            0.012241777474889248,
+            0.014771538127546474,
+            -0.008804192497858033,
+            -0.02341427260922065,
+            0.023167424359504862,
+            -0.0011598649565273595,
+            -0.029627604902846688,
+            0.0021991474725787303,
+            0.003261878079344105,
+            -0.0005935713331682358,
+            -0.009144689470851402,
+            -0.005580557862988033,
+            -0.007240836451875213,
+            0.019631377011298905,
+            -0.007085213886352692,
+            0.007017573363709754,
+            -0.004074171791476129,
+            -0.00822871835665378,
+            -0.0421101573155962,
+            0.004210914075286222,
+            0.003408427700622687,
+            -0.01752179330665445,
+            0.006384440404174512,
+            0.011121481025573962,
+            0.043020575455262985,
+            -0.022809979913698254,
+            -0.0005737138409049003,
+            0.019206623212867476,
+            0.0009562551889820241,
+            -0.023427467810802533,
+            -0.038931688376706186,
+            0.007172345672967499,
+            -0.03467050750542982,
+            0.023185301620612612,
+            -0.0007075596339842658,
+            0.02387977204565927,
+            -0.03043468414449851,
+            0.013909304572370437,
+            0.004784668158421625,
+            0.012524879174804915,
+            0.021654512058639427,
+            0.03142146922870715,
+            -0.02053473475540882,
+            -0.050409862638621725,
+            -0.005406667316025549,
+            0.00035138762584085976,
+            0.022018487016014856,
+            -0.026014580970974616,
+            0.00025633004699302783,
+            -0.003349094027291921,
+            -0.02401366769212778,
+            -0.01735996745614562,
+            0.01962340791446025,
+            -0.050011480783403936,
+            0.015042913576492952,
+            0.0262737424555697,
+            -0.02507473901367252,
+            -0.05408889321034101,
+            -0.018988587082325706,
+            -0.028217893770659686,
+            -0.009285770094319318,
+            -0.00661159230987788,
+            0.028597405154740078,
+            0.003525576256054959,
+            -0.011462219358952466,
+            0.025401668390029348,
+            0.006155837251830549,
+            0.0026050693705924593,
+            0.015985722392086222,
+            0.007289910674212967,
+            -0.011452511453623235,
+            -0.014599577191293152,
+            -0.04558584556100621,
+            0.010197536827229795,
+            -0.0045431962637690155,
+            -0.019372598050508005,
+            -0.030147301333718016,
+            -0.0137828704002156,
+            -0.027060515748528666,
+            -0.0017291966950107505,
+            0.053783684195381654,
+            0.03292494290301388,
+            0.005560172296221419,
+            -0.0026085670565688026,
+            -0.010466890489356092,
+            0.036247483457995895,
+            -0.006789746401727701,
+            0.03662690472838588,
+            0.001784522073210422,
+            -0.0242673253380092,
+            0.01169382884548301,
+            0.02892627181542449,
+            0.009858451662166387,
+            -0.0005592283645355213,
+            -0.006160132931009508,
+            -0.008782830470079547,
+            0.0031207630972208394,
+            0.025613233596604418,
+            0.003268170392795548,
+            -0.010062991425127038,
+            0.018106967407336815,
+            0.004480291699976396,
+            0.02010374270557836,
+            0.03555357587371324,
+            0.023489837757983054,
+            -0.021782173652026497,
+            -0.026581762796336082,
+            -0.0013561486781077014,
+            0.0072266106510934505,
+            0.03551967295942857,
+            -0.01651315514034218,
+            0.0001274396319698452,
+            -0.008272735507120127,
+            -0.00894931703949637,
+            0.011149952898504355,
+            -0.0029671593121986116,
+            -0.0030752010285951576,
+            0.01662711184497578,
+            -0.0011672263615879709,
+            -0.033448222499147764,
+            0.0078288560210653,
+            -0.023496331346917084,
+            0.0051040606404415545,
+            -0.004113104164416575,
+            -0.007076525381595441,
+            -0.002157548492138289,
+            0.006302962169276006,
+            -0.008065841090139872,
+            -0.00035242603988955366,
+            -0.01586406637763306,
+            0.008271894439780969,
+            0.007438312999409211,
+            -0.0069617481460406095,
+            0.011402932583388205,
+            0.002666895909532698,
+            0.0005219881149353855,
+            -0.012159425670629751,
+            -0.012042145306711322,
+            -0.00957276588761916,
+            -0.03401863879157551,
+            -0.02604353712472079,
+            -0.024949533021906844,
+            -0.0014209149651479556,
+            -0.006566685925666972,
+            -0.03187796739029736,
+            -0.01209152137926458,
+            -0.011110763124407752,
+            0.007756961097537267,
+            -0.013260548545917543,
+            0.005433285079114047,
+            -6.399098520041351e-05,
+            -0.0044960488826674555,
+            0.016615636438440516,
+            -0.003643291599919672,
+            -0.019488010211707343,
+            -0.03927818878149417,
+            0.008209336177358258,
+            -0.02805593125673831,
+            -0.0038816771550190788,
+            0.007607119219721711,
+            -0.00528681677614909,
+            -0.029502897838743516,
+            -0.009149794737430706,
+            0.008767001199311985,
+            0.003507217008280066,
+            -0.01773349756494436,
+            0.03579670773605949,
+            0.026971617790626075,
+            0.011875179879209696,
+            -0.019891342676443626,
+            0.0508929813275565,
+            -0.0047010735337056275,
+            -0.012544935684845649,
+            0.018297503869683027,
+            0.02355022268131519,
+            0.03273167043015095,
+            -0.024830243320788693,
+            0.0009370814985214882,
+            -0.021937793653233454,
+            0.00535662615596573,
+            -0.035009915581788176,
+            0.0028595659314725012,
+            -0.014395027018699446,
+            0.015596459769560532,
+            0.03397372881171701,
+            -0.013254226072981457,
+            0.011195747181764677,
+            -0.016299384375589934,
+            0.00730244731485402,
+            0.00569944090461748,
+            -0.012916284280345044,
+            -0.005812311882399539,
+            -0.022902815152116493,
+            0.020140276669979015,
+            0.005809799950176385,
+            -0.0008130655714656685,
+            0.0104085826652653,
+            -0.044723432238900435,
+            0.03286429597337671,
+            0.008730225944501239,
+            0.017599189944921396,
+            0.0029762446259779998,
+            -0.0003068671951232415,
+            0.011611551606224214,
+            0.011228401175698422,
+            -0.016029727902740706,
+            0.004345913009102136,
+            -0.004613061137378433,
+            -0.03850533909474302,
+            0.009363351681220008,
+            -0.03631184057978666,
+            0.014427784677340159,
+            0.016026872002233575,
+            -0.004170582842643461,
+            -0.025166189571409366,
+            -0.005428471012333673,
+            0.010827910811085384,
+            -0.009783069664327864,
+            -0.009987805234276443,
+            -0.04163358183060102,
+            -0.025236440364963034,
+            0.034097508113604576,
+            -0.002119474613596975,
+            -0.025826403511707147,
+            -0.010830738112107807,
+            -0.011351132636491575,
+            0.020207687645133,
+            -0.006542026283946858,
+            -0.0006263975804936843,
+            -0.04150011545311013,
+            -0.0073396962917864515,
+            0.011192225907726928,
+            0.027746213794734322,
+            -0.035970017302005566,
+            0.02708126943501373,
+            0.002105633542220187,
+            0.014262674845755191,
+            -0.01626664983003942,
+            -0.005631805361903626,
+            -0.013990539604372344,
+            -0.035877093361375685,
+            0.016487723013906077,
+            -0.008607758248055054,
+            0.004714436224135091,
+            0.04458009880881818,
+            -0.0036872068506712727,
+            -0.009466125348499464,
+            0.007600705817195143,
+            -0.0034590883553048063,
+            -0.010351576085287273,
+            -0.0028564505793999295,
+            0.001959661298507,
+            -0.007080697863126256,
+            -0.04183476693971616,
+            -0.00499888976530217,
+            0.04684391782633148,
+            0.006361744381879857,
+            0.013276793765092465,
+            0.013516220521120684,
+            -0.018335746557932003,
+            -0.045577810515431215,
+            -0.0071616111758423025,
+            -0.0032763880176386808,
+            -0.022708028392758567,
+            -0.0240284945459861,
+            -0.008735040856657758,
+            0.005844145402902923,
+            0.006359368165686724,
+            -0.01120763599008222,
+            -0.022676131361171057,
+            -0.027085968700452136,
+            -0.010526103176253857,
+            -0.02123944716136792,
+            -0.010792145796733375,
+            -0.027142270190771887,
+            -0.028742234538796804,
+            0.02880426953386151,
+            0.007575182232859564,
+            -0.004084088301176117,
+            -0.010005765476121966,
+            -0.01255640546813973,
+            -0.0004774761617900028,
+            -0.0025307994917461232,
+            -0.005822065709999144,
+            0.0038614286313282164,
+            -0.011390482620811868,
+            0.023152086963976554,
+            0.0021220701260546895,
+            0.018329162172128138,
+            -0.05772324571231113,
+            -0.002709083851090071,
+            0.021497896395795905,
+            0.0033025410488410913,
+            -0.005412445141910314,
+            0.029352650459792132,
+            0.011223947499453582,
+            -0.006539460282439711,
+            -0.031355428613718234,
+            0.020942976176820262,
+            0.024122993515785947,
+            0.00817615634503138,
+            -0.012994160510030828,
+            0.01221095914384204,
+            0.01629778384790159,
+            -0.004494985018246821,
+            -0.011161098655072933,
+            0.005586857967039181,
+            0.020941039642372112,
+            0.013580622629911925,
+            0.0013398071867077221,
+            0.030905022925366163,
+            0.0006862084391508121,
+            -0.03521231612862609,
+            -0.01908191995763771,
+            -0.01275995164030872,
+            -0.003584331305986364,
+            -0.015264425022763126,
+            0.005294329654809353,
+            -0.002355465010876059,
+            0.006335052149475905,
+            0.022655379849785256,
+            0.03593231939948758,
+            0.0146937743203089,
+            0.00937330328448886,
+            0.003167462724350819,
+            -0.0013235176101076284,
+            0.012102872918129403,
+            0.008638011773075119,
+            0.0350701240268404,
+            -0.014653221697693864,
+            -0.011003994230902835,
+            -0.0037767288519369537,
+            0.010100131061186566,
+            -0.004113527518662366,
+            0.020414393256136372,
+            0.03274709615649469,
+            -0.0037222949335916876,
+            -0.005604552180246868,
+            -0.006160118231975786,
+            0.0041467454414452565,
+            0.01807879127455274,
+            -0.01252134965321004,
+            0.004364858987501461,
+            0.005549300665159658,
+            -0.0071897162833861,
+            0.018567059255813558,
+            0.0002622074720860951,
+            -0.012983550838417323,
+            0.03266376564402191,
+            0.11449509387240629,
+            0.0017251916835047628,
+            0.007125215166916139,
+            -0.040229997815548584,
+            -0.020537987527148888,
+            -0.00245728735683473,
+            0.005779972828836593,
+            0.007004732428236457,
+            0.02225914982152608,
+            0.006505827715591914,
+            0.0006708045351142873,
+            0.031708522931495536,
+            -0.017334083511709092,
+            -0.002522861743921599,
+            -0.01745656559688129,
+            0.013890451996979048,
+            -0.014312605700272691,
+            -0.013784963572164409,
+            -0.01572417853955075,
+            0.004507425871748977,
+            -0.000601158156253668,
+            0.0016158265715634729,
+            0.01791303234734911,
+            0.0011926917387973374,
+            -0.026393351252156246,
+            0.0006694008039871878,
+            0.001026031515214626,
+            -0.004514419527575117,
+            -0.011320929719992686,
+            -0.02223001723942815,
+            0.016235841442653918,
+            0.014188311619681008,
+            -0.004895218832118094,
+            0.009244697376301004,
+            0.010558796153959677,
+            0.02707485418319703,
+            -0.028380342401606758,
+            0.0040204430418629405,
+            0.007023041225985294,
+            0.01606556581447933,
+            0.017617862031190843,
+            0.0034870847111552862,
+            -0.011253315705053224,
+            0.03238289399766116,
+            -0.010563282687352038,
+            0.0065093536270233385,
+            0.014966802208644462,
+            -0.03541611540945151,
+            -5.821682910348165e-05,
+            0.0061985444938402745,
+            -0.011716201661363997,
+            0.009580035961315467,
+            -0.03120692534097433,
+            -0.0039291243987978845,
+            0.009317468331974374,
+            0.017611312558635673,
+            0.008804996982574765,
+            -0.0034129169537151925,
+            0.010822226698822738,
+            0.012510661458054741,
+            -0.0027147975086409966,
+            -0.008855213644027594,
+            0.01238502686635956,
+            -0.0007687053446954701,
+            0.00450108863711973,
+            -0.00799170686059026,
+            0.0029755595812552293,
+            0.002290650917244691,
+            -0.01554856300008412,
+            -0.011779430686394945,
+            0.00438393126679289,
+            0.0036931891075561957,
+            0.001535662168147833,
+            0.012014408636737167,
+            -0.006087255045485002,
+            -0.0077816220325470226,
+            0.0040573724156794284,
+            0.003136943055301706,
+            -5.505014316363865e-05,
+            -0.030625245024561508,
+            0.01585923084309149,
+            0.0067595435845396405,
+            0.010013308598626031,
+            -0.005315626260228517,
+            -0.00903142414091241,
+            -0.007240920015018893,
+            -0.01228521172101564,
+            -0.010657894223173453,
+            0.007658605592936815,
+            -0.00724924993432376,
+            0.011124174211132038,
+            -0.03128027787357322,
+            -0.013988715435985316,
+            0.01231792993245938,
+            -0.01528161242159377,
+            0.0007675821323979293,
+            -0.008389914312201402,
+            0.0009417242487395995,
+            -0.005119304379761705,
+            0.007887436211143328,
+            -0.03731503169547854,
+            0.007878001990271541,
+            0.02456170736765724,
+            -0.00330924625934981,
+            -0.020259771366167557,
+            -0.009613930457931786,
+            0.0241314385136084,
+            -0.008267831641886344,
+            -0.01645888287387047,
+            0.008421304862524506,
+            0.010140813149093993,
+            -0.0006309118552875393,
+            0.0008237784055549769,
+            -0.003726984191590917,
+            0.01669125388798323,
+            0.02054777616420997,
+            0.041930770964107515,
+            -0.014789610432004559,
+            0.01092349703703152,
+            0.02866938974792608,
+            -0.03378576980722847,
+            0.014728113846952417,
+            0.02518409645155824,
+            0.014590531327650454,
+            -0.0004446956315190408,
+            0.001609777186372437,
+            0.0068939321280085475,
+            -0.014273492115311677,
+            0.009247676157551494,
+            0.013380104440744367,
+            -0.018407746260686045,
+            0.020444430509441124,
+            0.03140071554008003,
+            0.0025839998101985736,
+            0.01355205268903058,
+            -0.006268939830271642,
+            -0.013860692186849113,
+            0.014717364068116991,
+            0.031429975488060896,
+            0.0006919565473572909,
+            -0.018561500446336806,
+            0.019128244642842268,
+            0.006988095806264863,
+            -0.003678529269242034,
+            -0.0021993078588142664,
+            -0.021334919445200388,
+            -0.021003586791369228,
+            0.017304350704792804,
+            0.013508779408333964,
+            0.027794485858413365,
+            -0.010594702782952029,
+            -0.007818089330662161,
+            0.014293688127901867,
+            -0.022368568887004094,
+            0.015426482139881204,
+            -0.02160820059598751,
+            -0.00571499813703566,
+            -0.0077402210272634235,
+            0.03274360648629609,
+            0.001441304806200403,
+            0.02734331677589297,
+            0.016457027764064002,
+            -0.033609420134045206,
+            -0.029790535553657092,
+            -0.00712106584238484,
+            -0.02307933297713104,
+            0.010356561171963577,
+            -0.03901183838032092,
+            -0.00071735150189382,
+            0.008251646656432826,
+            -0.008079917288471352,
+            -0.0007087526632966007,
+            -0.006419191399519291,
+            0.013989344185652506,
+            0.013661633160336152,
+            -0.0228103143826663,
+            0.023644639659577426,
+            0.008343481940133684,
+            0.025998483906975782,
+            -0.02076055376006461,
+            0.019148568238437927,
+            -0.017705363646416224,
+            -0.02777313933338929,
+            0.014358130220942483,
+            0.023585731219788467,
+            0.01635364038245971,
+            -0.007122853965024528,
+            0.007110003583832299,
+            -0.03556761320536796,
+            0.027585457849964405,
+            -0.013064062623866305,
+            -0.012504161289952458,
+            -0.0032379276562094555,
+            -0.011931827532360686,
+            -0.002574161005800279,
+            0.0008533907465694845,
+            0.0034345232246102806,
+            -0.03131462673429805,
+            -0.0002450959360520896,
+            -0.03727416416766574,
+            -0.028944814604815072,
+            0.01360126708805471,
+            -0.00036361284498363273,
+            -0.0049097813612814616,
+            0.02933242705521636,
+            -0.026339136251046315,
+            -0.02693295650849956,
+            -0.009590247544093352,
+            0.02853096456887074,
+            -0.0220673798697511,
+            -0.004519050393279848,
+            0.0015034014871316069,
+            -0.01828226786700525,
+            0.004436383319946364,
+            -0.0012748481042235446,
+            -0.010416065158876917,
+            -0.01105146139270597,
+            -0.03229479099927737,
+            -0.005822133182096844,
+            0.02106215163620738,
+            -0.006957125617688484,
+            0.002746241228040847,
+            0.013312248331456934,
+            -0.010968629118224582,
+            -0.0029946900704643812,
+            0.0017875284441794235,
+            0.023304175718480834,
+            -0.021033941064173063,
+            0.007714334541806199,
+            0.005955442031111047,
+            0.0016405586860131207,
+            0.018406562522853148,
+            -0.0054922197063461825,
+            0.0037690571959124617,
+            0.0162126534419016,
+            -0.009969304596173302,
+            0.017959307638726878,
+            0.007691549802475525,
+            0.006681837561745192,
+            0.002775700618222622,
+            0.007712770283579111,
+            0.009752652973700739,
+            -0.013506020126078112,
+            0.016911141425375933,
+            0.022468715726349706,
+            0.00023233555209372766,
+            0.021066580799163234,
+            -0.02302516292580905,
+            -0.028618544628453015,
+            0.018297878810036087,
+            -0.0225078941299981,
+            -0.012735208680311652,
+            0.016459925496937065,
+            -0.0043032724578332855,
+            -0.00797609922711709,
+            -0.014986057073351055,
+            0.02260618561608301,
+            0.02132930921464508,
+            -0.020249788476562527,
+            -0.012385936327068884,
+            0.0072304431443959505,
+            -0.004373475447051833,
+            0.007086656081865733,
+            0.03195810137269375,
+            -0.015982210897392764,
+            0.006818890093901413,
+            -0.0038306138017521626,
+            0.0030753642449792215,
+            0.017830546304667783,
+            0.009470018737705222,
+            -0.0019815241970236146,
+            0.030100238416495066,
+            -0.001330614013444443,
+            0.016392226282459177,
+            0.0004888028477871524,
+            -0.010948833692356176,
+            0.006343890495626752,
+            -0.026652806306173643,
+            -0.008627570055007363,
+            -0.001203530223345183,
+            -0.009581494451831351,
+            0.07598616930784373,
+            0.003930153552069478,
+            -0.01295957554290434,
+            -0.001673007115291578,
+            -0.001368287102181597,
+            -0.006897179139643672,
+            -0.00960679451603541,
+            0.0287166325268306,
+            -0.016613613331623346,
+            -0.01894527592131826,
+            0.0008370014132002703,
+            -0.006785618888421608,
+            0.00919924251249592,
+            0.01816363936999796,
+            -0.010225961928154075,
+            -0.01604759761423727,
+            0.001577138242530474,
+            -0.003520564936698765,
+            -0.004395434431595494,
+            0.0052399043934902014,
+            -0.010127122101423486,
+            0.006456662777583096,
+            0.026560491731327886,
+            -0.013684536319659296,
+            -0.01425196172663986,
+            0.004084982328707292,
+            -0.0010682084059972982,
+            0.009758641913073978,
+            -0.00864879062601467,
+            -0.00481222050200678,
+            -0.014375330954464725,
+            -0.011378890549403524,
+            0.02505641780042516,
+            -0.02861164504722546,
+            0.0034060847953066354,
+            0.014937227531980433,
+            -0.00016904224334342915,
+            0.01541381851648916,
+            0.0033028855218525644,
+            -0.03615935765718852,
+            -0.013677774380513626,
+            -0.02073839691250734,
+            0.019821074228011413,
+            0.01688422012576042,
+            -0.0033731840788147442,
+            -0.013840993586441808,
+            -0.026149880178874887,
+            -0.007430563876740068,
+            -0.02466234818567362,
+            -0.020708282040996014,
+            -0.019276991448222817,
+            0.006137002799644838,
+            0.005162892426268572,
+            0.022881759407670583,
+            0.0025854880108157857,
+            -0.015058889448404231,
+            0.0029948608544278386,
+            -0.006722836436935071,
+            0.011845298212977357,
+            -0.02747944486818378,
+            0.015311272959213262,
+            -0.01610493464983465,
+            0.025239768880017402,
+            0.007425208902253236,
+            0.0010569008485112994,
+            -0.008822681490988575,
+            0.00010704167529046774,
+            0.014883787023445807,
+            -0.004186968199973372,
+            -0.0007781259002896265,
+            0.02787555889813697,
+            0.02087373813833492,
+            -0.0017783425135865238,
+            0.008743718448136432,
+            0.028026471206552342,
+            0.019081083283793984,
+            0.0325261942213039,
+            -0.013815526479760766,
+            0.005113676604826083,
+            -0.028873600114081546,
+            0.010510475843707614,
+            -0.010871039729515013,
+            -0.010201824683340651,
+            -0.0067946599549165735,
+            0.027756887504389347,
+            -0.007962601205446472,
+            -0.0006687976124826922,
+            0.03469894503563926,
+            -0.014185294847328063,
+            -0.00010543064621374546,
+            0.0007386418834840544,
+            -0.009759481309152999,
+            0.02742722730386853,
+            -0.0073590367291510046,
+            0.018456032721630843,
+            -0.019560882612141943,
+            -0.013705917411254969,
+            -0.011894560208199602,
+            -0.001142768182039817,
+            -0.02576754877141608,
+            -0.008755240066826174,
+            -0.02249287216087102,
+            0.00859172262856532,
+            -0.0046012388553094375,
+            0.018845920253446127,
+            0.0011617844393195566,
+            -0.005275799428018491,
+            0.029069367036565145,
+            0.0088080893110627,
+            0.018100411389202663,
+            -0.0004971948011851863,
+            0.01216566778850985,
+            0.02369448391321832,
+            0.002637001729688794,
+            0.005012948614448268,
+            -0.024213704079796478,
+            0.003420184574622472,
+            -0.0004359900462023086,
+            8.265488264269535e-05,
+            0.016267948861008616,
+            -0.019572032453100303,
+            -0.02216264534756073,
+            0.021871880626992544,
+            -0.012742926070754964,
+            0.011676769226089007,
+            0.0035362870996336194,
+            -0.010732229460038356,
+            0.0006838777488770268,
+            -0.03083438636386314,
+            0.008820746379709355,
+            -0.03216685858285627,
+            -0.032230808828686675,
+            -0.013547681710236494,
+            -0.009994106571421729,
+            0.0008758088338659712,
+            0.028447187234357855,
+            -0.00795317029726568,
+            -0.015797892954136448,
+            0.005363565557248085,
+            -0.011199252697624111,
+            -0.00817162221789202,
+            -0.0007867892785855576,
+            0.005856939999700809,
+            0.009461959355310273,
+            0.005997804620690314,
+            0.011648335019189696,
+            0.007491946658046298,
+            -0.008028800679169293,
+            -0.008837725220736191,
+            0.015889923814605387,
+            -0.0014409716087425222,
+            -0.01176424391171685,
+            0.030430130082762363,
+            -0.013300620185884817,
+            -0.0005592926632214984,
+            -0.0021975669766032472,
+            -0.0015687208316663597,
+            -0.0021652076181411244,
+            -0.014225289143358926,
+            0.01870568404189425,
+            8.334501002782455e-05,
+            -0.014288806218067678,
+            0.0032841743905975634,
+            0.003841780428153241,
+            -0.0036498339483411074,
+            0.008388922960269383,
+            -0.008147275785908849,
+            0.017348147498892,
+            -0.015113395971380741,
+            0.008740452410210708,
+            0.0038335161236289234,
+            0.0043482078469644385,
+            -0.0065755348640655865,
+            -0.01592809184384243,
+            0.009642593996701513,
+            0.012051872087663658,
+            0.02173264012652684,
+            -0.02525660485859878,
+            0.00876161653901566,
+            -0.005374533920781607,
+            -0.014534179652013885,
+            -0.016127866637369738,
+            0.00982303248417706,
+            0.022322910435388785,
+            0.0063086346898091296,
+            0.012139263756190026,
+            0.00013383660542132047,
+            0.005848259030010402,
+            -0.014767198851314148,
+            0.010570158368602513,
+            -0.011355774405066537,
+            -0.018439261453084426,
+            -0.005243587266408389,
+            0.028894120979627323,
+            -0.005492024421080143,
+            0.017938747701972086,
+            0.0021350711250093858,
+            0.029007356490801176,
+            -0.008686781541806358,
+            0.0014883465513564527,
+            0.0039866022469891155,
+            -0.01295271525889559,
+            -0.004272008629260297,
+            0.01976273843646234,
+            0.04882868638601926,
+            0.010204475902027572,
+            -0.0009155109545661691,
+            0.010069071220988797,
+            0.005626418787776309,
+            0.002148101019776153,
+            -0.00805781810841286,
+            -0.030431071067211577,
+            0.008883707378543252,
+            0.005628218215212441,
+            -0.01997825125961566,
+            0.019838234890736068,
+            0.026483365072033395,
+            0.023040623182975964,
+            -0.011585643598171132,
+            0.016539930468533123,
+            -0.010092529089936188,
+            0.0177008735686489,
+            0.028665424114962996,
+            0.013810061902648515,
+            -0.010487150104822674,
+            -0.0020847870330548534,
+            0.006329342599564054,
+            0.004186981876004016,
+            0.00026239408223817384,
+            0.015190614519766204,
+            -0.019155180136775043,
+            0.01005728016778505,
+            0.014725826697964637,
+            0.03322436221319799,
+            8.049930133627564e-05,
+            -0.01645191835557271,
+            0.0016249281996128196,
+            -0.009976155662939693,
+            0.00856399492362911,
+            0.000632427127231252,
+            0.0032242260692672124,
+            -0.009688119824276678,
+            -0.012682779503915694,
+            -0.002851918183031152,
+            0.009403205508474241,
+            0.0021188716361101698,
+            0.002759004011231831,
+            0.007641925150177999,
+            0.0022830062031146204,
+            0.01713024474384216,
+            -0.0011181980841546123,
+            -0.00029141889484139217,
+            0.004601090256674874,
+            0.010509199590505558,
+            -0.01649852835819883,
+            -0.017548369964263514,
+            0.004337237475053179,
+            0.012217494210546133,
+            -0.013159179948224596,
+            -0.013241981632059755,
+            0.015704543608959117,
+            -0.01913575130892795,
+            0.024934102868065366,
+            -0.00942381892138331,
+            -0.004543611336526587,
+            0.000343200128934031,
+            -0.00727538880611703,
+            -0.0038751118497522673,
+            -0.0030064279504724797,
+            0.003984715814505561,
+            0.0019261818110367147,
+            0.007472350699291607,
+            0.008472369788801792,
+            0.014568547873264247,
+            0.006043895352236181,
+            0.0032580830408778395,
+            0.009109104498825864,
+            -0.014523846294121576,
+            -0.014800649887278973,
+            -0.002704791598011488,
+            -0.008895394038700809,
+            0.00902016867266151,
+            -0.006607182154874112,
+            0.00845408382182461,
+            -0.015501867835442603,
+            0.015573642225745032,
+            -0.012200552219016517,
+            -0.0008900215712478553,
+            -0.009666233545144905,
+            -2.7470277092642562e-05,
+            0.008191697904514857,
+            0.014605287653054787,
+            0.0070552942843382215,
+            -0.031921405225964206,
+            0.0017403427534633655,
+            -0.008846568352320488,
+            0.03446319360336241,
+            0.002612606662691094,
+            -0.01060863123202008,
+            0.020352181356568008,
+            -0.01592124132155158,
+            -0.00027704668715660693,
+            -0.01602075523652062,
+            -0.004797557413047212,
+            0.00707939446796679,
+            0.016399017978990946,
+            0.02759049512664933,
+            -0.005697020356865688,
+            -0.007995305872076221,
+            0.006384047872862793,
+            0.029590696670334232,
+            0.012246955508890425,
+            0.024748351124415785,
+            -0.007633712004672087,
+            0.014348228716100414,
+            0.007773260552660029,
+            0.0076045488653348975,
+            0.004321689250440021,
+            -0.002591801797145718,
+            -0.011698574606499644,
+            -0.014974971814360001,
+            -0.0037563183330623384,
+            -0.016987915346053498,
+            0.00012247833530362867,
+            0.021344400276501117,
+            -0.00475351423959752,
+            -0.0033847161875730026,
+            -0.01874186437694691,
+            0.0033461670218611346,
+            -0.010029445302549983,
+            -0.00367421246775698,
+            -0.013133469163793961,
+            -0.0254122035745965,
+            -0.018556369080030412,
+            -0.002155244024999317,
+            -0.00929309186646207,
+            -0.015194269781263309,
+            0.0070354690219415535,
+            -0.016305117309271634,
+            0.0080837166473439,
+            -0.00448473417024141,
+            -0.016508775226236883,
+            -0.00958925666995396,
+            0.01100335153658195,
+            -0.0041819993634961685,
+            -0.01929493648733792,
+            -0.01282300765645703,
+            -0.011300260564838231,
+            0.0051733704121938884,
+            0.005996780699975213,
+            -0.0038111648304178544,
+            -0.0028279639348479987,
+            0.016640121845366295,
+            -0.010197365811240557,
+            0.009878730312873492,
+            -0.03335474019383245,
+            -0.03551826040406959,
+            0.004497847402931096,
+            0.01507007013703286,
+            0.0013085881889541252,
+            -0.02078680101065954,
+            0.010806315988966003,
+            -0.016499186313125556,
+            -0.00858013792516617,
+            -0.008522534127498645,
+            0.009649521088411411,
+            -0.007283003380127847,
+            0.01299542385123007,
+            -5.3226845309105124e-05,
+            0.030477683597373687,
+            0.0074153455615142755,
+            0.004161881195438522,
+            -0.00679081048132963,
+            0.008354912600159931,
+            -0.001621776416108105,
+            0.004080191054849395,
+            0.02120524874226696,
+            0.012440848502230418,
+            -0.00218041171518331,
+            0.006458415098595872,
+            0.005678055538564355,
+            -0.017186537377121235,
+            0.02507960402044456,
+            0.024678178106806115,
+            -0.008205288425409454,
+            0.004108402466376321,
+            0.005783335302169668,
+            -0.008846663924040241,
+            -0.006897707231869481,
+            0.006527795262026734,
+            -0.018474181903007085,
+            -0.0008721381678008483,
+            0.010598084358464908,
+            -0.02255441909004843,
+            -0.004840511019512733,
+            0.021745651362109403,
+            0.004631567631335321,
+            -0.018558972627958834,
+            0.02764788406651861,
+            -0.004942361238163609,
+            -0.0027290999934532494,
+            -0.010876135202650048,
+            0.01671497143541799,
+            -0.012173527207375091,
+            0.008443317346655462,
+            0.011021541861818043,
+            -0.006295639322287843,
+            0.003083685676540999,
+            0.008277206401503182,
+            -0.004504302913715969,
+            -0.0068075424927125676,
+            -0.0010508416962215848,
+            0.010167160518175114,
+            0.0036980815941666336,
+            -0.022704963865525812,
+            -0.006020073261002279,
+            0.02934227992246887,
+            -0.015082922782242507,
+            -0.0050916398557337705,
+            0.005376294716056329,
+            0.007709342013963906,
+            -0.003476236561444464,
+            -0.0023399991233097337,
+            -0.00787486645740744,
+            0.010356414979428607,
+            -0.015950083845400274,
+            0.02076360898078935,
+            0.025608263494550438,
+            0.024504236990840842,
+            -0.014814235990238182,
+            0.010828584133720797,
+            -0.011703992394890898,
+            -0.011047470905668532,
+            -0.015542835222173864,
+            -0.001658296303013357,
+            0.001970208852236451,
+            0.013107754130616248,
+            0.009666887546609378,
+            0.0235853683657408,
+            0.006473544089109194,
+            0.008973300312205858,
+            0.014118601565410305,
+            -0.003902202854013708,
+            -0.018303438244719996,
+            -0.014584226585460755,
+            -0.011977613285028946,
+            -0.01756345491020528,
+            0.0004563602312336091,
+            0.020461659996271687,
+            0.012417744207683839,
+            0.024136626592375227,
+            0.002119213701333967,
+            0.013759902247467922,
+            -2.9525392347047554e-05,
+            -0.000892319792935443,
+            0.00045783825646242434,
+            -0.005238483120312843,
+            0.019255756091120846,
+            -0.004599752813817219,
+            0.01656589522110913,
+            0.0006443773300684925,
+            0.007133940816771615,
+            0.013554761340256566,
+            -0.01704388669741371,
+            -0.003443100892108368,
+            -0.010029620309829285,
+            0.000941391966877521,
+            -0.019633111750884034,
+            0.01934571074127927,
+            0.010273330977233852,
+            0.0017121345993533963,
+            -0.018327189396014198,
+            -0.0029706385099949693,
+            0.03448635131998772,
+            0.019458999453011265,
+            0.0050734354363614525,
+            0.02632443373898792,
+            0.013018762352749236,
+            0.014443611411502168,
+            0.010153370950346075,
+            0.02356766253713842,
+            -0.023479811300626968,
+            0.02095255383200973,
+            -0.011590347850340786,
+            -0.007414136601134069,
+            -0.0035774888741646705,
+            0.0025141508183687115,
+            -0.013588691733097813,
+            0.00982877622088977,
+            -0.0017916539399655826,
+            -0.025386211797192578,
+            0.009164316443206016,
+            -0.008811052979979271,
+            0.010464442616928296,
+            -0.007774569252594923,
+            -0.0002806136360286729,
+            -0.0127084026190163,
+            -0.015184317235990752,
+            0.001259201880504347,
+            -0.009158429132281081,
+            0.023908150915412617,
+            -0.021210848953875233,
+            -0.01321384014436847,
+            0.013997502713857512,
+            -0.006869443694669701,
+            -0.011364239609894453,
+            -0.00374449980963666,
+            0.023673050288879224,
+            -0.009243341446512421,
+            -0.01359367519009039,
+            0.014464166953530989,
+            0.003834735113400469,
+            0.0059713317014342395,
+            -0.007508862694272452,
+            -0.008428705474135382,
+            -0.003280788036976103,
+            0.005998840043285335,
+            0.026615077617284676,
+            -0.014059930972065399,
+            -0.0032833264512892022,
+            -0.009189999799498163,
+            0.004118204537632102,
+            -0.02359889635281631,
+            0.02771075351384059,
+            -0.008871395303200375,
+            0.021705177758560575,
+            -0.0015924449308442096,
+            -0.006239188158271628,
+            -0.024231036074349822,
+            0.027522945599928003,
+            0.003683870966132199,
+            -0.010221180899173243,
+            0.0012768725460662862,
+            -0.052005292736885915,
+            0.016636109796947025,
+            0.008429895913163985,
+            -0.019507994429834353,
+            0.003697519981610208,
+            -0.004773584856005853,
+            0.018983847580558853,
+            -0.022966993228850697,
+            -0.014078986007109447,
+            0.008283766047449966,
+            -0.01153285124362746,
+            0.0031439180465649996,
+            -0.02527749501821793,
+            -0.005754537200570427,
+            0.0011760345167229269,
+            -0.005276936074280004,
+            -0.001705933750221904,
+            -0.020022761608132548,
+            -0.025972227850795115,
+            0.017666017379157135,
+            0.015820875579600418,
+            -0.002278262448951789,
+            0.01320818011561616,
+            0.008709899932124187,
+            0.00836573261091119,
+            0.016034263394940008,
+            0.007124509234103267,
+            0.0081473711608784,
+            0.045498533239850446,
+            -0.019846323044145485,
+            0.007371629414298193,
+            -0.0014854149174929534,
+            0.013951741898629716,
+            -0.030467924817555014,
+            -0.0055144480998344705,
+            0.0062802744565489945,
+            -0.0049373689340857915,
+            0.03510454786147091,
+            -0.017326094216003073,
+            0.01694160526735264,
+            -0.0026075809870445238,
+            0.0007420394265014909,
+            -0.02042003877666893,
+            -0.026156937726134134,
+            -0.014294938857971834,
+            0.0011857406889443987,
+            -0.0010047941261608975,
+            -0.011946583993069242,
+            0.006797001710574735,
+            0.004917161312925866,
+            -0.0020713379451245515,
+            0.015652523103715316,
+            -0.02952061566240878,
+            0.007356800041369564,
+            -0.010733578620243498,
+            0.020008504830359524,
+            -0.00990331048284697,
+            -0.0013319400098308093,
+            0.007236960885181047,
+            -0.004122868383988748,
+            0.013216043472071927,
+            -0.0038337251451538677,
+            0.004227115653354611,
+            -0.005221321656766545,
+            -0.01794343569722877,
+            -0.0016435261495179767,
+            0.021429008432348098,
+            -0.008349684096540855,
+            0.031864497493526205,
+            -0.01264743878037828,
+            0.009174982784686633,
+            0.010692990337462762,
+            0.0006628831036334152,
+            -0.005286285851702483,
+            0.009052971814211658,
+            0.0021431970274545893,
+            -0.008690520448278274,
+            0.006934310986093342,
+            -0.018349449937310143,
+            0.008494451785507706,
+            0.03394632739506833,
+            0.0018511328780472266,
+            -0.0007519661670070526,
+            -0.007266663179613392,
+            -0.01335844201823745,
+            0.018540713029383066,
+            0.005488273906733518,
+            -0.002546531152114063,
+            -0.027536660869590543,
+            0.02007746621157705,
+            0.0012265798861291223,
+            -0.004755325069578531,
+            0.012466557901411086,
+            0.0020495867632654885,
+            0.019989531151046736,
+            -0.010171678639113316,
+            0.030757957899447096,
+            0.010050036220124393,
+            0.002959833051227505,
+            0.0038556928367700113,
+            -0.036655749074851536,
+            -0.01652152778069314,
+            -0.00019357502472740266,
+            0.014271581054865031,
+            -0.012430647252826173,
+            -0.022432580814844155,
+            0.0010346169306977334,
+            0.01148004051449691,
+            -0.0033898642762881046,
+            -0.009650665305271706,
+            0.01560757541276278,
+            -0.020342213902670575,
+            0.006701302394408393,
+            0.007693672045981921,
+            0.0287100214318104,
+            -0.0020735384554022153,
+            -0.0012324838849990056,
+            0.000626416803904792,
+            -0.0055669860801192935,
+            -0.0008037366126870824,
+            0.017251369734470945,
+            -0.013661316947002466,
+            0.0175926211217534,
+            -0.012735241490550407,
+            0.0064713656749838585,
+            -0.0021092378432037563,
+            -0.010629579860050812,
+            -0.020114807971202787,
+            0.0029609495764307357,
+            0.009378003163375899,
+            0.006920144698428806,
+            -0.010438984881449953,
+            0.010715409812532968,
+            0.012197673689706352,
+            -0.00753962861484439,
+            0.014519905132928041,
+            0.022173989895384192,
+            -0.007687503607783572,
+            0.013633321414009471,
+            -0.0041446334728745095,
+            -0.014056264816568229,
+            -0.009985463824440833,
+            0.023929750617160402,
+            -0.0005563806942667597,
+            0.008756997373618809,
+            0.00388125051641897,
+            0.0007856447663040696,
+            -0.011226530517316183,
+            -0.001653447552601681,
+            0.019294448360333633,
+            0.011831934675073843,
+            -0.0013888534822008192,
+            -0.0010525764356699885,
+            -0.0014566936525653407,
+            -0.009427557739483988,
+            0.02049188741459728,
+            -0.022030783927622997,
+            -0.0019604940727076573,
+            0.00461018148439991,
+            -0.022589816803875618,
+            0.0001863117949313752,
+            -0.024570217962989905,
+            -0.01569952241187498,
+            0.011763033781122343,
+            -0.0022530631823737554,
+            -0.016028810867854907,
+            -0.0003242701040942633,
+            7.424127301806556e-05,
+            -0.007815789940508527,
+            -0.004135500768181258,
+            0.006115909992703774,
+            -0.00034193217510010117,
+            -0.0032262189374874842,
+            0.013430154310506884,
+            -0.009996622196870008,
+            0.015431528911575157,
+            0.0067029980830614145,
+            -0.015702167721800483,
+            0.0021406610839557003,
+            0.018500219544264753,
+            -0.0058910087864916085,
+            0.00787021425357441,
+            -0.014681907249972176,
+            -0.0037495551894426303,
+            -0.005787661351944086,
+            0.006051351977821761,
+            0.024849195780885833,
+            0.012969434450643372,
+            0.01139340445077682,
+            0.017405361843075933,
+            0.004325000715054734,
+            -0.01572297405297993,
+            0.0007416798945675923,
+            0.01682727189828363,
+            0.010464036837245312,
+            0.009804326468113703,
+            0.0008975556667684365,
+            0.0043704860361487635,
+            -0.005142176468800175,
+            0.009317064072855551,
+            0.02915482449687158,
+            0.0013434689612997423,
+            0.008696141010760192,
+            0.017276730345147134,
+            0.027038741396131265,
+            -0.0035088997194497725,
+            -0.007469614319788499,
+            0.008739942310300076,
+            -0.009657991604173173,
+            -0.001081629174976152,
+            -0.017000351183542026,
+            -0.012310275736750868,
+            0.016510505259358398,
+            -0.006084627062230492,
+            0.02848913486719956,
+            -0.0010655806791907225,
+            -0.02026498848361482,
+            -0.01565505608194376,
+            0.001404549479440637,
+            -0.01362132895080652,
+            0.005287566256051299,
+            0.014187265685427415,
+            0.01917046513020831,
+            0.004687191682875024,
+            -0.022258337721085656,
+            0.037364962197399806,
+            0.01757108963349427,
+            0.019377870795760123,
+            -0.019422840506328382,
+            -0.006159311237412833,
+            -0.009077095160676276,
+            0.02116697012944498,
+            -0.00880692095488465,
+            -0.009875674856043954,
+            0.024014332004451508,
+            0.017812606845656484,
+            0.005679806286464216,
+            0.006736609904079586,
+            0.0020048663439319585,
+            0.0025414630253661444,
+            -0.015831593851012047,
+            0.017957902498542212,
+            -0.005858125698347523,
+            -0.0058068554681761945,
+            0.017529514573571298,
+            -0.005989581792993925,
+            -0.0030306021203732347,
+            0.00015839997806822045,
+            -0.019697052145531482,
+            0.011618112886207201,
+            0.008256007601576028,
+            -0.017229774429089857,
+            -0.0014757560402567304,
+            -0.0002404480326945295,
+            -0.014118694740134205,
+            0.02004885146995733,
+            0.01694893356262835,
+            -0.0013248884139405398,
+            -0.0051752815372013315,
+            0.008223868147393114,
+            0.007000679132597393,
+            0.005256194890343006,
+            -0.019496098876977206,
+            0.005230597149342019,
+            0.005103909294824105,
+            -0.0014716649591412774,
+            0.010274346323051634,
+            -0.009030594825985526,
+            0.033740380586893826,
+            -0.006505788197604522,
+            0.013065658941136721,
+            -0.0043634731684193745,
+            -0.0055529072007985125,
+            0.001816020676232658,
+            0.013054468478598695,
+            0.009753501030700457,
+            -0.00970446155762939,
+            -0.004172354962129865,
+            0.018697982092775737,
+            -0.011983156146516074,
+            0.0013300791598046338,
+            0.01564072865804035,
+            -0.004858814041438713,
+            0.004708235083775013,
+            0.008635204790345423,
+            -0.004489505237783172,
+            0.01801512858970922,
+            -0.016482572999490366,
+            0.0023953436660054026,
+            0.010694869684962088,
+            0.02390864159830096,
+            -0.019152187046410703,
+            0.021786596486708207,
+            -0.017951050280323393,
+            0.026538582634436314,
+            0.014484894489694262,
+            -0.0035222259457311444,
+            -0.01946200885147006,
+            -0.00850690913394181,
+            0.011654565129063417,
+            -0.021423803876241532,
+            -0.0007985948078290641,
+            -0.0007290222944686444,
+            -0.021367628543092133,
+            -0.011189364017231132,
+            -0.009961779129778331,
+            0.010701637745810138,
+            0.001918633898887013,
+            0.010380983868185359,
+            0.008002182827196827,
+            0.010784760574518087,
+            0.014405640442760822,
+            0.0008958107769384412,
+            -0.010743001445440941,
+            -0.023795589003655273,
+            0.016987708698679597,
+            0.00438463274382093,
+            0.0007862212695139549,
+            -0.01627468994515667,
+            0.011936651269737248,
+            0.013250643815768723,
+            -0.009385605548926286,
+            0.009121882469425302,
+            -0.03716901103204463,
+            -0.004552808410398574,
+            -0.0035704811925430366,
+            -0.0020309197365173475,
+            -0.00908678158053104,
+            0.01559445415882485,
+            -0.013682507836005851,
+            -0.0009590922904844632,
+            0.00011696985057536548,
+            -0.0074756526903548915,
+            -1.5467890432254378e-05,
+            0.020649928837031005,
+            0.008438785939908428,
+            -0.0037382105257999757,
+            0.010376213224654074,
+            0.008826031303751312,
+            -0.013900359449287676,
+            -0.01527474712786084,
+            0.019035366035884273,
+            -0.018776918681908793,
+            0.0063098418544296165,
+            0.01677916431403883,
+            0.020778813096283988,
+            -0.02891862501830075,
+            -0.0007231469961304435,
+            0.022093939332021185,
+            -0.0012807635799950273,
+            0.006370841109790851,
+            0.008781853138377092,
+            0.006754082980602544,
+            0.0029102159062355374,
+            -0.003711173531879644,
+            -4.843770941622283e-06,
+            -0.003940474588075707,
+            0.007155539022862352,
+            0.004366433868349658,
+            0.036453728071874844,
+            -0.013661129348165794,
+            0.002541308805735001,
+            -0.0058838191638835155,
+            -0.00347401304572066,
+            -0.00593469688967835,
+            0.004407665856950772,
+            -0.010055727994798024,
+            0.004330827357536078,
+            -0.001570861535288093,
+            -0.0003731129422346693,
+            0.004883969598300302,
+            -0.001701673379646408,
+            0.007036892564844683,
+            -0.011863650867472625,
+            -0.0003979727750011258,
+            -0.020174851412609312,
+            0.021117284557875267,
+            0.00011666551282792189,
+            0.003662462447326642,
+            0.03723741246290771,
+            0.00476064510662738,
+            -0.00238355131307445,
+            -0.012880057949435798,
+            -0.013551338766321525,
+            -0.012986444917747435,
+            -0.020703821629987552,
+            -0.0036665673548525388,
+            -0.021006311026247164,
+            -0.0052061406234528915,
+            0.010362862594785442,
+            -0.011540184420787572,
+            -0.004023659825622843,
+            -0.015881589631889473,
+            -0.006531981517815174,
+            -0.03114211633687585,
+            -0.01124607488650473,
+            0.007493228366452009,
+            -0.019149285949049886,
+            -0.0039932870926865805,
+            0.026850121590606017,
+            0.001095360635760988,
+            0.01387696424607248,
+            0.0038058248676879625,
+            0.008117686785954085,
+            -0.0045937472017383955,
+            -0.0005669025036141417,
+            -0.01928244877716143,
+            0.003391980330529026,
+            0.008525354854576925,
+            -0.00361507909530282,
+            0.0070008404454106905,
+            -0.008835727374865623,
+            -0.016950017602980382,
+            0.0021654956715219517,
+            0.009334379529385142,
+            -0.006639140003365139,
+            -0.017587472740268683,
+            -0.011049929670847685,
+            0.010401189523318744,
+            -0.02283429470064187,
+            0.006257145380266984,
+            0.016439581169238607,
+            -0.009363030375624585,
+            0.004651575425738153,
+            0.019809183749755425,
+            -0.011810674576781702,
+            -0.005811328501528397,
+            0.008471992070998505,
+            0.013891259999721588,
+            0.0095606134736748,
+            0.02954498207274139,
+            0.0026596809804008996,
+            0.000561870602466584,
+            0.004889263753490895,
+            -0.004277061142504552,
+            -0.017423200644235665,
+            -0.006243787897080996,
+            0.0038507494787521856,
+            0.012873996671952184,
+            -0.009879417960879032,
+            -0.00048575277472139055,
+            0.01682604801673206,
+            0.004530939990187663,
+            -0.0033901248079550284,
+            -0.011868399508738274,
+            -0.020080331108085103,
+            0.0015580259874809898,
+            -0.01362200460317948,
+            0.015354347040143093,
+            0.003390601069331352,
+            0.0015284610287850575,
+            -0.004239880336806429,
+            0.01828645373058079,
+            -0.005013418527935815,
+            0.012749833447770729,
+            -0.006148890994661865,
+            -0.004587831231448011,
+            0.007986533783001116,
+            0.009166578151006778,
+            0.0004939857995863664,
+            0.05421019830078428,
+            0.002838056604877719,
+            0.01333205648597435,
+            0.013419127695838135,
+            -0.014523733369484934,
+            0.014086525147734823,
+            0.012688569853552613,
+            0.018497344547821693,
+            0.013037819487160016,
+            -0.012353776077273462,
+            0.017354801119939964,
+            0.0024084723134891487,
+            0.003909681065813095,
+            0.0022202438064449653,
+            -0.004528576079628071,
+            0.019594683950225115,
+            0.004070222328547378,
+            -0.0074362443729372105,
+            0.004453618452623641,
+            0.005007543168224267,
+            -0.013105524222311019,
+            -0.0112958915514782,
+            0.027105208848863218,
+            -0.01219417557128091,
+            -0.018749971743148304,
+            0.006706594269933445,
+            -0.02008458683853808,
+            -0.03763686535076864,
+            -0.0021332074289244973,
+            0.009992361098878196,
+            0.002795183524495193,
+            -0.00253695604043486,
+            0.01614298064601032,
+            0.016730443047719683,
+            -0.016985296937992726,
+            -0.018245902728910347,
+            0.0025257780765481825,
+            0.0016333325293570182,
+            0.01861807236979066,
+            -0.023168663546030196,
+            -0.006378864850313035,
+            -1.3795465857380962e-05,
+            -0.006829008132963306,
+            0.004850306884033312,
+            0.004703280112191218,
+            -0.0071714759384722965,
+            -0.0027773519096287217,
+            0.002256846645341137,
+            0.01975383140755998,
+            0.013198754584066554,
+            0.01572971337890945,
+            0.002600741959577911,
+            -0.005033508062449482,
+            -0.016817359058724685,
+            -0.005450011991586895,
+            -0.013478330855299777,
+            -0.009091854345715177,
+            -0.003042052092498012,
+            0.008217376329142566,
+            -0.025053469059115735,
+            -0.011965083373220982,
+            0.0119294363235832,
+            0.00391377331459168,
+            0.0003099203772539202,
+            0.007607672728136279,
+            -0.003830132495364688,
+            0.010331053246451333,
+            0.013823890473448383,
+            0.0076112887952959766,
+            -0.004723377809559611,
+            0.009600411526173948,
+            0.011460781436250457,
+            0.006015477068316837,
+            -0.005072711638895421,
+            0.023859899048048694,
+            0.01821920335673539,
+            -0.01680365156728093,
+            -0.002149107383573353,
+            -0.0027829676356423997,
+            -0.01884308738652242,
+            0.009665830801645276,
+            0.00880322398336952,
+            0.008308268605449346,
+            0.003906369351868086,
+            0.00974531700889854,
+            -0.013875904819275925,
+            -0.0050570811397453235,
+            0.002870120865445989,
+            -0.005729556771404781,
+            -0.01901337075686543,
+            0.000297242372701423,
+            -0.011987192261104132,
+            -0.011327157550660895,
+            -0.01409710922165158,
+            -0.003221240464411861,
+            0.005283472511900512,
+            0.021472187640986972,
+            -0.004410881709548286,
+            -0.001855852749039012,
+            0.01351697599517034,
+            0.010060125186506398,
+            0.0021658738823899905,
+            0.0002206474541239167,
+            0.005418828324605768,
+            -0.0027919130944783397,
+            0.0008094042430809068,
+            -0.0015826797905407809,
+            0.021089276765434876,
+            -0.02016611086095107,
+            -0.009413977569213423,
+            -0.009288078098692112,
+            0.005842197094339927,
+            0.028793032462603636,
+            -0.007165634502373454,
+            -0.010104002083017078,
+            -0.01994315293556678,
+            0.0003786527922888834,
+            -0.07810609175343784,
+            -0.005210636756258548,
+            0.001870513618910238,
+            0.03126380113105983,
+            -0.010817888476436834,
+            0.0033673768058629823,
+            -0.002204487854177595,
+            0.014542299655712928,
+            -0.001760107965206874,
+            -0.0013283746982194259,
+            0.0071954806737858056,
+            -0.005640400818010697,
+            -0.006210989097047217,
+            0.005904163661619133,
+            -9.945941028848348e-05,
+            -0.010266938705916303,
+            0.00057263080955709,
+            0.02209498395848068,
+            0.0159398346608872,
+            -0.006575439759804175,
+            0.01816748552021211,
+            -0.00719294553484007,
+            -0.009838738592324234,
+            0.005639288780502127,
+            -0.0019007683459126327,
+            -0.015369619176147573,
+            -0.011475977270803784,
+            0.009862808084641291,
+            -0.030069890644150284,
+            -0.00010725753918890802,
+            0.00943698424646706,
+            -0.013540245857270492,
+            -0.008293042205008774,
+            0.007212974960781723,
+            0.006708343474532583,
+            -0.0012017968151025369,
+            -0.0078026000239507145,
+            -0.02527526731750743,
+            0.0027578400032925625,
+            -0.002446811971857853,
+            -0.004899702349784173,
+            0.0019960465173086183,
+            0.001479503476855694,
+            -0.012645905858466798,
+            0.01216748847452763,
+            -0.013378877368040743,
+            -0.012053251571418766,
+            0.012508842412556306,
+            -0.00027167241745418584,
+            0.01682680930386839,
+            -0.020607299598265426,
+            0.0010702151063620052,
+            -0.028075439824932304,
+            -0.007671848153932823,
+            -0.015486630894165054,
+            -0.0009776136330459065,
+            -0.0012939921569540803,
+            -0.03481880531471923,
+            0.012889447665794033,
+            -0.005200149829198305,
+            -0.006094702500154477,
+            -0.00459778303079732,
+            -0.024583504817899775,
+            0.006072281238752663,
+            0.00607350167867964,
+            -0.006506508970816635,
+            -0.016816439635445445,
+            0.008428048142588516,
+            -0.0011040593013238872,
+            0.004873670535200424,
+            -0.007475800310154908,
+            -0.0005638920132076605,
+            -0.013047135093134946,
+            -0.005463454269764494,
+            -0.0008522562324122834,
+            0.01827123304871433,
+            0.0044055851550317835,
+            -0.014526381222031981,
+            0.016862445236481526,
+            0.013250692156820124,
+            0.01602714293656341,
+            -0.0071089293427753115,
+            -0.0004265739315250755,
+            -0.02750630108787212,
+            -0.00327225092954903,
+            -0.023687644154808972,
+            0.00041213204365507744,
+            0.009447660230507965,
+            0.0203905264889617,
+            0.012694421477479161,
+            0.004620617778996077,
+            -0.001557511764988156,
+            0.0066751090724502,
+            0.02696959560423252,
+            -0.01099183126742224,
+            -0.006969111598423376,
+            -0.01884504192789845,
+            0.01113023717919988,
+            0.025135748541199255,
+            0.003381303139273845,
+            -0.011317957771935042,
+            0.008112598754655146,
+            -0.0009247845526290428,
+            -0.010302980598863016,
+            0.02889587382004902,
+            -0.0391134481322538,
+            -0.015599591108049343,
+            -0.004605824192962023,
+            0.007406520608110251,
+            -0.015596917115435033,
+            0.00770468362805995,
+            -0.012614339656105264,
+            0.001699049329817941,
+            -0.005851477164035241,
+            0.03535087340659861,
+            -0.0031950960727572925,
+            0.0006909856233673865,
+            -0.003395233050301427,
+            0.003531135044131077,
+            0.012355948349937309,
+            -0.004444221317023453,
+            -0.030726300006909638,
+            -0.0022983766869702346,
+            -0.011198252640110788,
+            0.009274666307515955,
+            -0.016207714291658314,
+            -0.017342435976846468,
+            -0.009424737744066259,
+            -0.021827243671837953,
+            -0.012403084982789827,
+            0.029747453318486338,
+            -0.005824001042844527,
+            0.01747569398142269,
+            -0.005219060548639168,
+            -0.008365003057800934,
+            0.00021272411948388247,
+            0.01344335658390732,
+            -0.028498240040284714,
+            0.00041963499785238845,
+            -0.002322931702641349,
+            0.0126393550661767,
+            -0.001595134597082556,
+            -0.00983073906977773,
+            0.006903313245694627,
+            0.0010522208685011098,
+            -0.015648940549713764,
+            -0.0007573245570947632,
+            -0.008533504878052644,
+            0.020475329445149498,
+            0.00027791387213409814,
+            0.001074213629646005,
+            0.013864331671732542,
+            -0.02102352993295736,
+            0.01576938071930521,
+            -0.021722775924283996,
+            0.01893267442161547,
+            -0.006577103101038035,
+            0.005772016664149475,
+            0.008283223343661555,
+            0.014068789332250016,
+            -0.014030042849491258,
+            -0.00663752723080222,
+            -0.034436220972634726,
+            -0.0122621879114314,
+            -0.001946072976826902,
+            -0.005308941654794638,
+            -0.014773123927909993,
+            -0.0011054822266082056,
+            -0.026205461509573584,
+            -0.006460373065662098,
+            0.012822188046808264,
+            0.019900312098359022,
+            -0.008409070094265829,
+            0.014163132434889265,
+            -0.006743711872055091,
+            -0.0028230771810882492,
+            -0.012652021749331811,
+            -0.018061180380009395,
+            -0.01083672324723357,
+            -0.01890966636827258,
+            -0.0018635408070944168,
+            0.004709222372611061,
+            -0.015760618199235518,
+            0.011435131737642627,
+            0.03198874888333405,
+            -0.01567271472650171,
+            0.008243406188694909,
+            0.016411496328333205,
+            0.0034236971031973503,
+            0.011728079139322837,
+            0.0038311415233052795,
+            0.004621789116267879,
+            -0.011109450058055078,
+            -0.011473158133621661,
+            0.006524490579223986,
+            0.00954511918454185,
+            -0.004913142601134428,
+            0.024347072255007807,
+            0.0008745419206472714,
+            0.005045794664611216,
+            0.005304506902446091,
+            0.011488187043525824,
+            -0.013135271908307747,
+            0.019746484188715552,
+            -0.002055210086078267,
+            -0.015571665869889405,
+            -0.01535499169234956,
+            -0.01105419264826655,
+            -0.018626399757822328,
+            0.0017108083455869296,
+            0.0181439079855921,
+            -0.0012493542718277685,
+            0.012137753522284387,
+            -0.00924460959150499,
+            -0.005019437549649743,
+            -0.0027156377206998474,
+            -0.038211954658821295,
+            -0.0021878970332906457,
+            -0.018809487178512628,
+            -0.02498945889484935,
+            0.03505216568400238,
+            -4.26477081213836e-05,
+            0.00997643255528116,
+            -0.010310197159232386,
+            -0.0002454902098986129,
+            -0.011065403358802515,
+            0.014954998647970473,
+            0.004624123403100598,
+            0.008872923678983478,
+            0.008560280683876702,
+            -0.016419500515801093,
+            -0.002997557798375716,
+            0.007472390947008284,
+            -0.00374822278027057,
+            0.012680407710694952,
+            0.015353684671120731,
+            0.020886474047003733,
+            0.011161262397502876,
+            0.002045862158732102,
+            0.00887271411363814,
+            0.009563361150297699,
+            -0.0003513237086717377,
+            -0.010459621030424415,
+            0.015419039131459582,
+            -0.02204848739950703,
+            0.012172257374410352,
+            0.04346634265794738,
+            0.019197433209794286,
+            0.029488039318675732,
+            -0.02272810439246353,
+            0.01448092574459596,
+            -0.003801043409441206,
+            0.009476229966122152,
+            0.0049309968883087685,
+            0.02346191840355209,
+            0.003973840862678076,
+            0.0073546443568171405,
+            0.010647581332452525,
+            -0.00972051412996416,
+            0.025644022498404723,
+            -0.00846669916801108,
+            5.1518540791212095e-05,
+            -0.004407265804145216,
+            -0.026400029105519137,
+            0.01615193008014211,
+            0.0081722536401398,
+            0.0010702314763197463,
+            0.0010263623479474385,
+            -0.0019566604229607643,
+            0.007982949510899019,
+            0.007079322685771456,
+            -0.015274148711633536,
+            -0.003886053737590683,
+            -0.018527455865296014,
+            0.0012864569064749438,
+            -0.024337133883771334,
+            0.010466651250597847,
+            -0.017568706814132484,
+            -0.014822354159709854,
+            -0.021666681078831483,
+            -0.013232016867264097,
+            -0.02011653224168991,
+            0.0035858355517491515,
+            -0.008632755352062642,
+            0.013025728891203198,
+            -0.016962939287139683,
+            -0.021494971929779273,
+            0.008430887589398292,
+            0.00870802359711456,
+            0.007481282677955534,
+            0.009728368603458082,
+            0.014996157358749115,
+            0.01736292711410622,
+            -0.022712386143092265,
+            -0.023163001611599246,
+            -0.006682209717364464,
+            -0.003531495447167166,
+            0.006762379833125297,
+            0.00933681196321128,
+            0.012884185500343249,
+            -0.007077895849501418,
+            -0.007554762054641417,
+            -0.013283067570021503,
+            0.006197966297581871,
+            -0.003993067576677973,
+            -0.009464789349849612,
+            -0.0015662739825834614,
+            -0.0061389795593180686,
+            0.002753664732282926,
+            -0.0001791088888074743,
+            0.0007595261237141144,
+            0.017122834676182343,
+            -0.008176319370751018,
+            -0.009787747261447871,
+            -0.02027228453943018,
+            -0.010683273300243025,
+            -0.015604745521156111,
+            -0.0016279406232527138,
+            -0.006506713191861036,
+            -0.014420323628589382,
+            0.010888206569219648,
+            -0.011930708571093279,
+            -0.00887210117616784,
+            0.010807147705360856,
+            0.0073900910661282935,
+            -0.003431679006289002,
+            0.037804020313161775,
+            0.0011945597874820288,
+            0.00778778394048267,
+            -0.003366543306746973,
+            0.005296727102519836,
+            -0.017761251541598876,
+            -0.026382400396890007,
+            0.0010357716464853195,
+            -0.01775777668381489,
+            0.010807713750049491,
+            -0.008515368407459745,
+            0.0016051389544388124,
+            -0.00034958600424912995,
+            -0.009585854918521076,
+            0.0036549720706351473,
+            -0.008110927907950148,
+            0.004212219431447508,
+            -0.03507890639477636,
+            0.011409377585617926,
+            -0.01819120811524444,
+            -0.008962242411035718,
+            0.01710177594046791,
+            0.037560936862121835,
+            0.0028513388998398104,
+            0.00030058353811717895,
+            0.0014792184333546447,
+            0.004427921639326168,
+            0.021438127785303025,
+            0.006339483221420334,
+            -0.001187747087595264,
+            -0.02237982599829809,
+            0.0062435650294483695,
+            0.002200578793697352,
+            0.010092828474934744,
+            0.023105023225115266,
+            -0.016797326021070008,
+            -0.015378520287395565,
+            -0.012732527281000097,
+            -0.010797463778182385,
+            0.009881499418260473,
+            0.02801767062785128,
+            0.01811139137849365,
+            -0.017358098300028903,
+            -0.020207173948076473,
+            0.0016242901745373126,
+            -0.026331540602868762,
+            0.007269751661479859,
+            0.11940631114242901,
+            -0.0045382135734049305,
+            -0.005994407567723939,
+            -0.008679712231646732,
+            -0.004482553208064535,
+            0.016709571294385293,
+            -0.013387043582971358,
+            0.069179848974036,
+            -0.007221545718220735,
+            0.0037392229364642915,
+            0.0028916224964057385,
+            0.00462729158956839,
+            -0.009727339449635306,
+            0.011570145487622916,
+            -0.011287365791681587,
+            -0.012991994493845187,
+            -0.013076194601468383,
+            -0.0180830261130804,
+            -0.007272768410218907,
+            -0.004155564814758654,
+            0.004204993170487392,
+            -0.005525575983182816,
+            0.008051108079520902,
+            -0.017551645696802648,
+            -0.010800070994485792,
+            0.013007176804761548,
+            -0.0023627485852619817,
+            0.005390986239687555,
+            -0.029668390278675554,
+            -0.0007650391259819867,
+            -0.0002446394046947245,
+            0.021280292985495608,
+            0.003137771886236958,
+            -0.0019077298815862946,
+            -0.010810159675564049,
+            0.007040107942856957,
+            0.023835993694726953,
+            0.021875366566457236,
+            0.0180511566130312,
+            0.012135351541044354,
+            0.020483975362807574,
+            -0.008667079096167644,
+            0.026324312888512866,
+            -0.001552207146726937,
+            -0.00397810465151493,
+            0.0077577136352257004,
+            -0.009849600842081875,
+            -0.00816392268697345,
+            0.00677353736462267,
+            -0.017288900282504148,
+            -0.020418071729964713,
+            -0.020383762873109595,
+            0.025741408097358233,
+            -0.011755094311186617,
+            0.002737129201159793,
+            -0.008266210469668866,
+            0.027287436144516414,
+            -0.006283892307996516,
+            -0.0029440642954701632,
+            0.009428876742171362,
+            -0.03788574758692254,
+            0.008834605607057885,
+            0.006463342392514337,
+            0.026711196829464153,
+            -3.9827332346130654e-05,
+            0.005565561841469247,
+            -0.0006738085601962865,
+            -0.006104320995853506,
+            -0.010792830169647851,
+            2.4565453208679464e-05,
+            0.017983908748172493,
+            0.0035181716362611077,
+            0.016970206528324653,
+            -0.005449808667944701,
+            -0.023171388778833197,
+            0.007092509502738752,
+            -0.01555320330471131,
+            -0.023306285898839978,
+            -0.002028783456704567,
+            -0.03196436161255752,
+            0.0033412618908711163,
+            0.007527633574188688,
+            -0.01890570959143866,
+            -0.017859735533232564,
+            0.010329340599904344,
+            0.004382396539605985,
+            -0.0016746397858953155,
+            0.006643785064338525,
+            -0.03307812909454471,
+            -0.01927475592253288,
+            0.004686970010943116,
+            0.012733525168279355,
+            -0.012993441126970916,
+            0.015291832952199082,
+            -0.020491167895268936,
+            0.016451976170508816,
+            -0.012259387189535237,
+            0.018393274368854132,
+            -0.006997668664226754,
+            0.011843450716200625,
+            0.011457415642860215,
+            0.009560877634897626,
+            -0.01790001115302499,
+            0.0214879415944361,
+            0.0066829070506954115,
+            0.03900547021123392,
+            0.02512484499576633,
+            -0.012651294056575184,
+            -0.0018425680226742782,
+            -0.00011361458320146007,
+            0.014982647266052349,
+            0.0061874424115559,
+            -0.003689464024837091,
+            0.02705902205734107,
+            0.018239469373522537,
+            -0.021070704644859155,
+            0.020037508334843183,
+            -0.00973804771680803,
+            0.018876283979591423,
+            -0.013303080480396818,
+            0.010592424494628194,
+            0.004717870260467274,
+            0.01070131238869639,
+            0.009087656015197565,
+            0.006708673669681232,
+            0.025928065153829043,
+            0.03663367449576964,
+            -0.00511073108390551,
+            0.002161381031392467,
+            0.004864092591944181,
+            0.012814518637357098,
+            0.001556833001747549,
+            0.007915766323215207,
+            -0.007012785514562176,
+            0.005834786334750152,
+            -0.002311533032619821,
+            -0.00122162928193159,
+            0.019525999280692766,
+            0.000824163362963678,
+            0.012875575199417254,
+            0.013259376059185821,
+            -0.002390133687709065,
+            -0.005682644446302278,
+            0.0019262218417782657,
+            -0.018193478045923098,
+            -0.015131696847666434,
+            0.029463723011600933,
+            -0.011447039651934148,
+            -0.008083114389964213,
+            0.0020642857121204753,
+            0.012160202510990566,
+            -0.00866944494225225,
+            -0.005378300635997713,
+            0.0038280364584418335,
+            0.014427318067525318,
+            -0.01799358399064332,
+            0.003381165206832161,
+            0.005252552821854107,
+            0.019433554824232185,
+            -0.00028791985674147885,
+            -0.013487678738391956,
+            0.0005317922815671783,
+            0.014982152801567637,
+            0.011902348165369078,
+            0.0016056390299304837,
+            -0.006935777197137747,
+            -0.00446584737370504,
+            0.004586814866182051,
+            0.006269466956110759,
+            0.0027516492952993445,
+            0.008157662045793032,
+            -0.0025001885993292637,
+            -0.016648747436203294,
+            0.017735757469839637,
+            -0.0030669873770852725,
+            0.018058248627079793,
+            -0.0038238312452176745,
+            -0.0147172470474118,
+            0.0043328417868095355,
+            -0.001888507803054499,
+            -0.018556943283494755,
+            0.009407930200720534,
+            0.002560949424273885,
+            0.010411159836451958,
+            0.0052571230238294825,
+            -0.007499470181994448,
+            -6.145889479758644e-05,
+            -0.015163819825329975,
+            0.022836682800304944,
+            0.016390944816242627,
+            0.001713572907204073,
+            -0.0035687505973373878,
+            -0.031210299824387475,
+            0.0029076511444359886,
+            -0.007822231980296341,
+            0.017821806420465356,
+            0.008250605126658402,
+            0.007908773724009905,
+            0.008351355849444162,
+            -0.00743970642051451,
+            -0.002411981861500092,
+            0.012543476434788293,
+            0.0202407777715137,
+            -0.0024635818448627447,
+            0.02535771413579989,
+            -0.012533449594824386,
+            -0.0022489292647974992,
+            0.0012362917358624392,
+            0.01578939075186126,
+            -0.0008001783766371555,
+            0.006311837812042848
+          ],
+          "keyphrases": [
+            "Switch Transformers",
+            "Mixture of Experts",
+            "sparsely-activated model",
+            "training instability",
+            "multilingual settings"
+          ],
+          "title": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "23ceada7-a437-4b2d-b989-f16e928f439a",
+        "properties": {
+          "page_content": "1 Introduction 3 ",
+          "embedding": [
+            0.009373236447572708,
+            -0.00019147605053149164,
+            -0.011275587603449821,
+            0.02236648090183735,
+            -0.016502436250448227,
+            0.024490466341376305,
+            -0.009955023415386677,
+            0.09330756217241287,
+            0.00554082915186882,
+            0.0107676787301898,
+            0.006459683645516634,
+            -0.0031074818689376116,
+            0.005069858860224485,
+            -0.029274048283696175,
+            -0.0028881572652608156,
+            0.012771611101925373,
+            0.004903634078800678,
+            0.0010071611031889915,
+            -0.00228790077380836,
+            -0.00550850760191679,
+            0.008833005093038082,
+            0.009594869799911976,
+            -0.036237023770809174,
+            0.07295425236225128,
+            0.023493116721510887,
+            -0.021886276081204414,
+            0.03270936384797096,
+            0.024564342573285103,
+            -0.023142196238040924,
+            -0.0063627189956605434,
+            0.006759811658412218,
+            0.008625224232673645,
+            0.0010533347958698869,
+            -0.021516887471079826,
+            -0.06157708168029785,
+            0.0061087640933692455,
+            0.01108165830373764,
+            0.020574945956468582,
+            -0.03252466768026352,
+            -0.003042838769033551,
+            -0.011358699761331081,
+            -0.008699102327227592,
+            -0.01935596391558647,
+            0.006593586876988411,
+            0.04015254229307175,
+            -0.0013990594306960702,
+            -0.043292347341775894,
+            -0.018090806901454926,
+            -0.023991791531443596,
+            -0.014581616036593914,
+            0.029440272599458694,
+            -0.0307700727134943,
+            0.045434799045324326,
+            0.010758443735539913,
+            0.027667207643389702,
+            -0.0015398887917399406,
+            0.01821085810661316,
+            0.09737084060907364,
+            0.003970927558839321,
+            -0.02190474607050419,
+            0.005037537310272455,
+            -0.05751380696892738,
+            -0.011543394066393375,
+            -0.022107908502221107,
+            -0.010001196525990963,
+            -0.0067828986793756485,
+            -0.016742538660764694,
+            0.04137152433395386,
+            -0.02829516865313053,
+            -0.02114749886095524,
+            0.02690996043384075,
+            -0.002694228198379278,
+            0.0043634031899273396,
+            0.03933988884091377,
+            0.002493373118340969,
+            -0.005203762091696262,
+            0.069408118724823,
+            0.05537135526537895,
+            -0.001163574168458581,
+            0.031915176659822464,
+            -0.00717537384480238,
+            -0.0074200937524437904,
+            0.0161607526242733,
+            0.005069858860224485,
+            0.022772807627916336,
+            -0.01561590377241373,
+            -0.07964018732309341,
+            -0.10867413133382797,
+            0.0017811458092182875,
+            -0.0003122488269582391,
+            0.006099529564380646,
+            -0.008445147424936295,
+            -0.008514408022165298,
+            -0.04657990485429764,
+            -0.01678871363401413,
+            -0.003474561730399728,
+            0.0430707149207592,
+            0.00898999534547329,
+            -0.008334331214427948,
+            -0.01108165830373764,
+            0.03139803186058998,
+            -0.027556391432881355,
+            -0.01748131588101387,
+            -0.007743308786302805,
+            -0.020870456472039223,
+            -0.015976058319211006,
+            0.043255407363176346,
+            0.020741170272231102,
+            -0.0013759726425632834,
+            0.022348010912537575,
+            -0.03315262869000435,
+            0.00933167990297079,
+            -0.013057887554168701,
+            -0.01091543398797512,
+            0.01863565668463707,
+            -0.006025651935487986,
+            -0.03577528893947601,
+            -0.014886361546814442,
+            0.015458913519978523,
+            0.02661444991827011,
+            -0.013741256669163704,
+            -0.010287472978234291,
+            -0.009424027055501938,
+            -0.014535441994667053,
+            -0.027593329548835754,
+            0.0079002995043993,
+            -0.004626592621207237,
+            0.003342967014759779,
+            -0.007669431157410145,
+            0.04181479290127754,
+            0.02417648583650589,
+            0.04436357319355011,
+            -0.02628200128674507,
+            0.003675416810438037,
+            0.017398204654455185,
+            0.019503720104694366,
+            0.021627703681588173,
+            -0.006436596624553204,
+            0.00369388610124588,
+            0.02476750686764717,
+            -0.07927079498767853,
+            -0.010028901509940624,
+            -0.017527490854263306,
+            0.025709448382258415,
+            -0.009433262050151825,
+            0.018728002905845642,
+            -0.04750337824225426,
+            0.019485250115394592,
+            0.007235399447381496,
+            0.010730739682912827,
+            -0.01968841440975666,
+            0.006709021050482988,
+            -0.02216331660747528,
+            -0.012660794891417027,
+            0.004765113350003958,
+            0.011423342861235142,
+            0.03219221904873848,
+            0.017065754160284996,
+            0.03143497183918953,
+            0.03298640251159668,
+            -0.004991363734006882,
+            0.034020692110061646,
+            0.026429755613207817,
+            -0.02631893940269947,
+            0.026134245097637177,
+            -0.019208207726478577,
+            0.012134416028857231,
+            -0.04089131951332092,
+            -0.03645865619182587,
+            0.025044549256563187,
+            0.01748131588101387,
+            -0.006067208014428616,
+            0.040780503302812576,
+            0.0038855066522955894,
+            -0.013131765648722649,
+            -0.00817734096199274,
+            0.017499785870313644,
+            0.01250380463898182,
+            -0.014997177757322788,
+            0.005356134846806526,
+            0.006769046653062105,
+            -0.02308678813278675,
+            0.0019115861505270004,
+            0.0045896535739302635,
+            -0.027002308517694473,
+            -0.010416759178042412,
+            0.020925864577293396,
+            -0.030751602724194527,
+            0.006810602732002735,
+            -0.0014486960135400295,
+            0.05489115044474602,
+            0.0032944846898317337,
+            0.028442922979593277,
+            0.034149978309869766,
+            -0.01456314604729414,
+            -0.04414194077253342,
+            -0.003213681047782302,
+            0.004421120043843985,
+            0.0161607526242733,
+            -0.010970842093229294,
+            -0.0005338820046745241,
+            -0.01942984201014042,
+            0.01731509156525135,
+            0.03333732485771179,
+            0.0022463444620370865,
+            -0.019023513421416283,
+            0.010860025882720947,
+            0.038083966821432114,
+            0.02090739645063877,
+            -0.002989739179611206,
+            0.027500983327627182,
+            0.008200427517294884,
+            -0.03113946132361889,
+            -0.009890380315482616,
+            0.01893116720020771,
+            0.009308593347668648,
+            -0.0269469004124403,
+            0.05725523456931114,
+            0.0020755024161189795,
+            -0.021129028871655464,
+            0.026743736118078232,
+            0.012180590070784092,
+            0.049461137503385544,
+            0.021812397986650467,
+            -0.022311072796583176,
+            0.026577511802315712,
+            0.008893030695617199,
+            0.032469261437654495,
+            0.005651645828038454,
+            -0.00040199869545176625,
+            0.002104360843077302,
+            0.03051150031387806,
+            -0.0017707566730678082,
+            0.007401624694466591,
+            -0.03723437339067459,
+            0.01070303563028574,
+            0.0019427533261477947,
+            0.00814501941204071,
+            0.021165968850255013,
+            0.022384950891137123,
+            -0.006778281182050705,
+            0.008223514072597027,
+            -0.009770329110324383,
+            -0.021221376955509186,
+            -0.018986575305461884,
+            0.005268405191600323,
+            -0.0123375803232193,
+            0.08414673060178757,
+            0.006907567381858826,
+            -0.05141889676451683,
+            0.06157708168029785,
+            0.010823086835443974,
+            0.014156819321215153,
+            -0.007627875078469515,
+            0.011164771392941475,
+            0.0864369347691536,
+            -0.0008519025286659598,
+            -0.00424566026777029,
+            -0.03786233440041542,
+            0.04454826936125755,
+            -0.020593415945768356,
+            0.037751518189907074,
+            -0.0031813594978302717,
+            -0.012032833881676197,
+            0.019780760630965233,
+            -0.060801368206739426,
+            0.043920308351516724,
+            -0.008976143784821033,
+            -0.0009165455121546984,
+            0.03198905289173126,
+            0.06349790096282959,
+            -0.009040786884725094,
+            0.019337493926286697,
+            0.018247798085212708,
+            0.015569730661809444,
+            -0.021258315071463585,
+            0.020796580240130424,
+            -0.0016149209113791585,
+            -0.0142491664737463,
+            -0.033004872500896454,
+            0.004381872713565826,
+            0.026706798002123833,
+            0.025173833593726158,
+            -0.027427105233073235,
+            0.00783565640449524,
+            -0.024194953963160515,
+            0.013482684269547462,
+            0.020593415945768356,
+            -0.03904437646269798,
+            -0.026411285623908043,
+            0.004485763143748045,
+            -0.003559982869774103,
+            -0.04905480891466141,
+            -0.011958956718444824,
+            0.01857101358473301,
+            0.0034791789948940277,
+            0.013381103053689003,
+            -0.0331895686686039,
+            -0.036828044801950455,
+            -0.006542795803397894,
+            0.004314920864999294,
+            -0.0001486211986048147,
+            -0.006436596624553204,
+            -0.035701408982276917,
+            0.0030012824572622776,
+            0.07155057787895203,
+            -0.01592988334596157,
+            0.0028350576758384705,
+            -0.03740059956908226,
+            0.0222926028072834,
+            0.03392834588885307,
+            -0.03908131644129753,
+            0.017324326559901237,
+            -0.06593587249517441,
+            -0.021572295576334,
+            0.023659341037273407,
+            -0.007928003557026386,
+            -0.011718854308128357,
+            -0.03937682881951332,
+            -0.008929969742894173,
+            -0.004912868607789278,
+            -0.003446857677772641,
+            -0.024619750678539276,
+            -0.016040701419115067,
+            0.009622573852539062,
+            -0.007928003557026386,
+            -0.00038872379809617996,
+            0.015246515162289143,
+            -0.01649320125579834,
+            -0.01435074768960476,
+            -0.0038624198641628027,
+            0.027057716622948647,
+            0.03344814106822014,
+            0.01913432963192463,
+            -0.016881059855222702,
+            -0.055629927664995193,
+            0.0015387344174087048,
+            -0.0015075673582032323,
+            -0.004356476943939924,
+            -0.02711312472820282,
+            0.01589294523000717,
+            0.010813851840794086,
+            -0.014443094842135906,
+            0.026392817497253418,
+            0.0029112440533936024,
+            -0.016253098845481873,
+            0.016465498134493828,
+            -0.039561521261930466,
+            -0.006492005195468664,
+            0.013011714443564415,
+            0.0008392047602683306,
+            -0.009548695757985115,
+            0.014212227426469326,
+            -0.02328995242714882,
+            0.01676100865006447,
+            -0.03703121095895767,
+            0.021756989881396294,
+            -0.02354852482676506,
+            0.025783326476812363,
+            0.004104831255972385,
+            0.004903634078800678,
+            0.0001385207287967205,
+            0.02236648090183735,
+            0.006662847474217415,
+            -0.0007001068443059921,
+            -0.06257443130016327,
+            0.01767524518072605,
+            -0.026669858023524284,
+            0.02090739645063877,
+            -0.006542795803397894,
+            -0.04247969016432762,
+            0.018792646005749702,
+            0.0180723387748003,
+            0.008717571385204792,
+            -0.04414194077253342,
+            0.0005454253987409174,
+            0.0029320220928639174,
+            -0.0011347156250849366,
+            -0.04229499772191048,
+            0.01797999069094658,
+            0.004199486691504717,
+            -0.023160666227340698,
+            -0.004656605422496796,
+            0.008435912430286407,
+            0.001627618563361466,
+            -0.030991705134510994,
+            0.005356134846806526,
+            0.004040188156068325,
+            -0.0046242838725447655,
+            -0.008985377848148346,
+            -0.026134245097637177,
+            -0.002957417629659176,
+            -0.020131679251790047,
+            -0.037382129579782486,
+            0.042073365300893784,
+            0.0010117784840986133,
+            -0.0060164169408380985,
+            -0.01735202968120575,
+            0.0010550662409514189,
+            0.01141410879790783,
+            -0.009733390063047409,
+            0.0033845233265310526,
+            -0.014904830604791641,
+            0.027057716622948647,
+            0.02354852482676506,
+            -0.00550850760191679,
+            -0.03512885794043541,
+            -0.02774108573794365,
+            0.0052360836416482925,
+            0.0026064985431730747,
+            -0.03546130657196045,
+            -0.05289645120501518,
+            0.023326890543103218,
+            -0.021923214197158813,
+            0.020131679251790047,
+            0.010481402277946472,
+            -0.030456092208623886,
+            0.028239760547876358,
+            -0.022772807627916336,
+            -0.02061188593506813,
+            -0.001733817858621478,
+            0.005840957630425692,
+            0.0088560925796628,
+            0.002608807059004903,
+            -0.04137152433395386,
+            0.024749036878347397,
+            -0.03134262561798096,
+            -0.007904916070401669,
+            0.024952201172709465,
+            -0.006348866969347,
+            -0.014258400537073612,
+            -0.007119965739548206,
+            0.003691577585414052,
+            0.030659254640340805,
+            0.0076140230521559715,
+            -0.03139803186058998,
+            -0.010213595815002918,
+            -0.002126293256878853,
+            0.023105258122086525,
+            0.005559298675507307,
+            0.021055150777101517,
+            -0.041482340544462204,
+            0.016437793150544167,
+            0.0362185537815094,
+            -0.027685677632689476,
+            0.0021874732337892056,
+            -0.03919213265180588,
+            -0.026429755613207817,
+            0.026429755613207817,
+            -0.010536810383200645,
+            0.0200208630412817,
+            0.0041694738902151585,
+            -0.024121077731251717,
+            0.005577768199145794,
+            0.026411285623908043,
+            -0.00010345767077524215,
+            -0.000978302676230669,
+            0.05452176183462143,
+            -0.008689867332577705,
+            0.021258315071463585,
+            -0.00016622488328721374,
+            0.014433860778808594,
+            0.011349465698003769,
+            -0.017204275354743004,
+            -0.01873723790049553,
+            0.004361094441264868,
+            -0.02282821759581566,
+            -0.013279520906507969,
+            -0.00630731089040637,
+            0.011931252665817738,
+            -0.016345446929335594,
+            -0.002054724143818021,
+            -0.014849422499537468,
+            0.028018126264214516,
+            -0.026429755613207817,
+            -0.029015475884079933,
+            -0.010878494940698147,
+            -0.010112013667821884,
+            -0.006672082003206015,
+            -0.02203403040766716,
+            -0.015218811109662056,
+            0.04092825949192047,
+            0.012097476981580257,
+            -0.0010181274265050888,
+            0.0024218042381107807,
+            0.014221461489796638,
+            0.044806838035583496,
+            -0.026374347507953644,
+            0.010758443735539913,
+            0.027168532833456993,
+            -0.012411457486450672,
+            0.004467293620109558,
+            0.004324155393987894,
+            0.019282085821032524,
+            -0.01784146949648857,
+            -0.07816263288259506,
+            -0.020002393051981926,
+            -0.0386011116206646,
+            -0.01360273640602827,
+            0.04558255523443222,
+            -0.002902009291574359,
+            0.016668662428855896,
+            -0.006007182411849499,
+            -0.011155536398291588,
+            0.010795382782816887,
+            0.03394681587815285,
+            0.0025418554432690144,
+            -0.0037054296117275953,
+            -0.004377255216240883,
+            0.01626233384013176,
+            -0.004307994619011879,
+            -0.02740863524377346,
+            0.008657545782625675,
+            0.004963659681379795,
+            0.008851475082337856,
+            -0.019577596336603165,
+            -0.005088328383862972,
+            0.005887131206691265,
+            0.017887644469738007,
+            -0.02005780301988125,
+            0.0027080802246928215,
+            0.03241385146975517,
+            -0.033004872500896454,
+            0.015477383509278297,
+            -0.022551175206899643,
+            0.026245061308145523,
+            0.022551175206899643,
+            -0.0063581014983356,
+            0.013528858311474323,
+            -0.018552543595433235,
+            -0.004201795440167189,
+            0.0053976913914084435,
+            0.018063103780150414,
+            0.009816502220928669,
+            0.013418042100965977,
+            -0.019078921526670456,
+            -0.024065667763352394,
+            0.03549824655056,
+            0.011469516903162003,
+            0.00992731936275959,
+            -0.050938691943883896,
+            0.008223514072597027,
+            0.010398290120065212,
+            -0.026559041813015938,
+            -0.025210773572325706,
+            0.03559059277176857,
+            0.010610688477754593,
+            -0.02295750193297863,
+            0.01391671597957611,
+            0.05444788187742233,
+            -0.008625224232673645,
+            0.02325301431119442,
+            0.01254997868090868,
+            0.010906198993325233,
+            -4.3900970922550187e-05,
+            0.022218724712729454,
+            -0.0075124409049749374,
+            0.0037746899761259556,
+            0.0031328771729022264,
+            -0.0238809734582901,
+            0.026928430423140526,
+            -0.008560581132769585,
+            -0.01689029484987259,
+            0.001642625080421567,
+            0.028018126264214516,
+            -0.0199839249253273,
+            0.00649662222713232,
+            0.015135698951780796,
+            -0.014997177757322788,
+            0.009622573852539062,
+            0.02437964826822281,
+            0.010970842093229294,
+            -0.04454826936125755,
+            -0.01976229064166546,
+            0.012429927475750446,
+            0.012799316085875034,
+            0.02661444991827011,
+            0.009110046550631523,
+            0.006196494214236736,
+            0.0003523621126078069,
+            -0.010047370567917824,
+            -0.01873723790049553,
+            0.014951004646718502,
+            -0.015375801362097263,
+            -0.029828131198883057,
+            -0.028646087273955345,
+            -0.005850192159414291,
+            -0.004857460502535105,
+            -0.013141000643372536,
+            -0.009733390063047409,
+            0.0158005990087986,
+            -0.015135698951780796,
+            0.01959606632590294,
+            0.006602821405977011,
+            -0.00042508551268838346,
+            0.033300384879112244,
+            0.03064078651368618,
+            0.02144300937652588,
+            -0.00038497219793498516,
+            -0.04281214252114296,
+            0.004885164555162191,
+            0.0005820756778120995,
+            0.007678666152060032,
+            0.0036130824591964483,
+            0.00820966251194477,
+            0.04809439927339554,
+            0.004931338131427765,
+            0.013390337117016315,
+            0.006921419408172369,
+            0.03732671961188316,
+            -0.029865069314837456,
+            0.0032044462859630585,
+            0.032007522881031036,
+            -0.017961522564291954,
+            -0.03165660426020622,
+            -0.014590850099921227,
+            0.006944505963474512,
+            0.007794099859893322,
+            -0.005222231615334749,
+            -0.00103255663998425,
+            0.02111055888235569,
+            -0.0013344164472073317,
+            0.008265070617198944,
+            0.020297905430197716,
+            0.0007843736675567925,
+            0.01606840454041958,
+            -0.016345446929335594,
+            0.0039432235062122345,
+            -0.009816502220928669,
+            -0.02988353930413723,
+            -0.00913775060325861,
+            0.002717314986512065,
+            0.004215647466480732,
+            0.004573492798954248,
+            -0.0077479262836277485,
+            0.013270285911858082,
+            0.03592304512858391,
+            -0.0024633603170514107,
+            0.010739973746240139,
+            -0.02018708735704422,
+            0.017398204654455185,
+            0.0026388198602944613,
+            0.006602821405977011,
+            0.01985463872551918,
+            -0.02476750686764717,
+            0.007443180773407221,
+            -0.011672680266201496,
+            0.017296621575951576,
+            -0.005600854754447937,
+            0.03952458128333092,
+            0.0018192389979958534,
+            0.014877126552164555,
+            0.016151517629623413,
+            0.029440272599458694,
+            0.01764754205942154,
+            -0.039302948862314224,
+            -0.014803249388933182,
+            0.008232749067246914,
+            -0.0063211629167199135,
+            -0.012513039633631706,
+            0.0017615220276638865,
+            -0.011783497408032417,
+            0.032044462859630585,
+            -0.010204360820353031,
+            0.012743907049298286,
+            -0.014840187504887581,
+            -0.01439692173153162,
+            -0.04824215546250343,
+            -0.0015548951923847198,
+            0.00037256305222399533,
+            0.0019970072899013758,
+            -0.036292433738708496,
+            0.002897392027080059,
+            0.024989139288663864,
+            -0.02870149537920952,
+            0.002140145283192396,
+            -0.04318153113126755,
+            0.01896810531616211,
+            0.005854809656739235,
+            -0.021165968850255013,
+            -0.01606840454041958,
+            -0.012679263949394226,
+            0.0038947411812841892,
+            0.04488071799278259,
+            0.00553159462288022,
+            0.008902265690267086,
+            -0.03291252627968788,
+            -0.00022769345378037542,
+            -0.0012697733473032713,
+            0.003028986742720008,
+            0.010315177030861378,
+            0.014424625784158707,
+            0.03206293284893036,
+            0.00712458323687315,
+            -0.024435056373476982,
+            -0.010130482725799084,
+            0.01385207287967205,
+            0.021646173670887947,
+            0.016945702955126762,
+            0.008006498217582703,
+            -0.007567849475890398,
+            -0.005633176304399967,
+            -0.030234457924962044,
+            0.01441539078950882,
+            0.016871824860572815,
+            0.007138435263186693,
+            0.006750577129423618,
+            -0.004326464142650366,
+            0.01546814851462841,
+            -0.03150884807109833,
+            -0.026411285623908043,
+            0.023659341037273407,
+            -0.01883881911635399,
+            0.010370585136115551,
+            -0.006986062508076429,
+            0.020371781662106514,
+            -0.02328995242714882,
+            -0.02279127761721611,
+            -0.004040188156068325,
+            0.012697733938694,
+            -0.0001104558541555889,
+            -0.007364685647189617,
+            -0.004379563964903355,
+            -0.006233432795852423,
+            -0.0046750749461352825,
+            -3.780461702262983e-05,
+            -0.0146277891471982,
+            0.03812090680003166,
+            0.004192560911178589,
+            -0.00451115844771266,
+            -0.018063103780150414,
+            -0.018820350989699364,
+            0.00834356527775526,
+            0.013168704695999622,
+            0.03126874566078186,
+            0.001201667357236147,
+            -0.010629157535731792,
+            0.015643607825040817,
+            0.0024148779921233654,
+            -0.005171440541744232,
+            0.03185976669192314,
+            -0.0030751603189855814,
+            -0.01968841440975666,
+            0.006764429155737162,
+            -0.025210773572325706,
+            0.007660196628421545,
+            0.02437964826822281,
+            -0.0008998076082207263,
+            -0.020168619230389595,
+            0.02262505330145359,
+            -0.02064882405102253,
+            -0.03014211170375347,
+            0.010416759178042412,
+            -0.034648653119802475,
+            -0.01867259480059147,
+            0.01001043152064085,
+            0.02321607433259487,
+            0.007623257581144571,
+            0.007350833620876074,
+            0.002272894373163581,
+            -0.011201709508895874,
+            0.01826626807451248,
+            -0.03856417164206505,
+            0.009982727468013763,
+            0.025321589782834053,
+            0.0061087640933692455,
+            -0.013113295659422874,
+            -0.014258400537073612,
+            -0.012226763181388378,
+            0.003841641591861844,
+            0.022440358996391296,
+            0.031675074249506,
+            0.027390167117118835,
+            -0.009834972210228443,
+            0.0014302266063168645,
+            -0.002456434303894639,
+            0.009493287652730942,
+            -0.040189482271671295,
+            -0.013870542868971825,
+            0.003945532254874706,
+            -0.04096519947052002,
+            -0.0082789221778512,
+            -0.0030543820466846228,
+            0.007558614481240511,
+            0.022348010912537575,
+            0.00971492100507021,
+            -0.018386319279670715,
+            -0.037788454443216324,
+            -0.0238440353423357,
+            0.0051437364891171455,
+            0.04303377494215965,
+            0.03021598979830742,
+            -0.016908764839172363,
+            0.011238648556172848,
+            0.045434799045324326,
+            -0.005799401551485062,
+            -0.015911415219306946,
+            0.021627703681588173,
+            -0.012346814386546612,
+            0.016086874529719353,
+            0.016530141234397888,
+            0.0024218042381107807,
+            -0.04875929653644562,
+            0.029329456388950348,
+            0.017952287569642067,
+            0.006935271434485912,
+            0.02628200128674507,
+            -0.023068320006132126,
+            -0.03512885794043541,
+            -0.002382556675001979,
+            -0.0009690679726190865,
+            0.014138349331915379,
+            -0.009770329110324383,
+            -0.0013436510926112533,
+            0.003975545056164265,
+            0.052231550216674805,
+            0.010305942967534065,
+            -0.002495681867003441,
+            -0.01265155989676714,
+            0.02338230051100254,
+            0.013805899769067764,
+            -0.010204360820353031,
+            0.015578964725136757,
+            -0.019337493926286697,
+            0.02413954585790634,
+            0.046653781086206436,
+            0.007493971846997738,
+            0.009465583600103855,
+            0.018257033079862595,
+            0.010749208740890026,
+            -0.014932534657418728,
+            0.00727233849465847,
+            -0.013547327369451523,
+            0.006949123460799456,
+            0.010416759178042412,
+            0.007184608839452267,
+            -0.015791364014148712,
+            0.016216160729527473,
+            -0.017952287569642067,
+            0.012162120081484318,
+            0.00358768692240119,
+            -0.014932534657418728,
+            0.009290123358368874,
+            -0.0123375803232193,
+            -0.025026079267263412,
+            -0.0044626761227846146,
+            -0.00965027790516615,
+            0.0034999572671949863,
+            0.024749036878347397,
+            -0.021775459870696068,
+            0.016206925734877586,
+            0.014295339584350586,
+            0.0030520735308527946,
+            0.017823001369833946,
+            -0.04137152433395386,
+            0.009871911257505417,
+            0.010970842093229294,
+            0.05167746916413307,
+            0.03322650492191315,
+            -0.015542026609182358,
+            0.014101410284638405,
+            0.04159316048026085,
+            -0.019078921526670456,
+            -0.016142282634973526,
+            0.012217528186738491,
+            -0.005614707246422768,
+            -0.040189482271671295,
+            -0.04838990792632103,
+            -0.03021598979830742,
+            -0.003042838769033551,
+            -0.015246515162289143,
+            -0.0037262076511979103,
+            -0.01376896072179079,
+            0.0005832299939356744,
+            0.008717571385204792,
+            0.00018700299551710486,
+            0.015117228962481022,
+            -0.009484052658081055,
+            -0.0018411714117974043,
+            -0.023807097226381302,
+            -0.019005045294761658,
+            0.0009771483018994331,
+            -0.0177121851593256,
+            0.009364001452922821,
+            -0.020297905430197716,
+            -0.011174005456268787,
+            0.0038855066522955894,
+            0.024435056373476982,
+            0.03437161073088646,
+            -0.01870029978454113,
+            0.011451046913862228,
+            0.010481402277946472,
+            -0.006958357989788055,
+            -0.04613664001226425,
+            0.01185737457126379,
+            0.008583667688071728,
+            -0.0037192816380411386,
+            0.008943822234869003,
+            -0.009622573852539062,
+            0.001551432185806334,
+            -0.006427362095564604,
+            0.029643436893820763,
+            0.018487900495529175,
+            -0.02391791343688965,
+            -0.004525010474026203,
+            0.015578964725136757,
+            0.01857101358473301,
+            0.005134501960128546,
+            -0.011663445271551609,
+            -0.010555279441177845,
+            -0.01708422414958477,
+            -0.027353227138519287,
+            -0.04074356332421303,
+            -0.049128685146570206,
+            0.006593586876988411,
+            -0.0061087640933692455,
+            0.0055546811781823635,
+            0.029015475884079933,
+            -0.0002024422719841823,
+            0.041777852922677994,
+            0.0063858055509626865,
+            -0.0017072680639103055,
+            -0.013150234706699848,
+            0.009705686010420322,
+            0.021572295576334,
+            -0.05267481878399849,
+            -0.02354852482676506,
+            0.0022070968989282846,
+            0.017536723986268044,
+            0.004187943413853645,
+            -0.0009448268101550639,
+            0.020556475967168808,
+            -0.00783565640449524,
+            -0.00719846086576581,
+            0.04779888689517975,
+            -0.010509106330573559,
+            -0.007526292931288481,
+            0.011321761645376682,
+            -0.010656861588358879,
+            0.020371781662106514,
+            0.023954851552844048,
+            0.03331885486841202,
+            0.016373150050640106,
+            -0.008833005093038082,
+            0.022107908502221107,
+            -0.0215353574603796,
+            0.004848225507885218,
+            0.025450875982642174,
+            0.001217828132212162,
+            0.0021066695917397738,
+            -0.025395467877388,
+            0.025284651666879654,
+            -0.004425737541168928,
+            0.014092176221311092,
+            -0.0017188114579766989,
+            0.013371868059039116,
+            -0.014738606289029121,
+            0.025450875982642174,
+            0.0010175502393394709,
+            0.03136109560728073,
+            0.017065754160284996,
+            -0.029902009293437004,
+            -0.015218811109662056,
+            -0.012032833881676197,
+            -0.04151928052306175,
+            0.007544762454926968,
+            0.006713638082146645,
+            0.021516887471079826,
+            -0.02245882898569107,
+            -0.004580419044941664,
+            -0.01896810531616211,
+            -0.015643607825040817,
+            0.0051437364891171455,
+            -0.028683027252554893,
+            0.000924048712477088,
+            -0.010112013667821884,
+            -0.0034076101146638393,
+            0.0009985036449506879,
+            -0.02962496690452099,
+            0.020390251651406288,
+            0.0013782812748104334,
+            -0.016336211934685707,
+            0.02681761421263218,
+            0.019614536315202713,
+            -0.00643197912722826,
+            -0.016317741945385933,
+            0.003005899954587221,
+            -0.00890688318759203,
+            -0.001478708814829588,
+            0.04802051931619644,
+            0.010084309615194798,
+            0.012300641275942326,
+            0.004732791800051928,
+            -0.014064472168684006,
+            -0.00013296547695063055,
+            0.0014475416392087936,
+            0.010102778673171997,
+            -0.010333647020161152,
+            -0.008057288825511932,
+            0.005947156809270382,
+            -0.0045896535739302635,
+            -0.012697733938694,
+            0.01454467698931694,
+            0.016604019328951836,
+            -0.0036061564460396767,
+            -0.0193005558103323,
+            0.023345360532402992,
+            0.010305942967534065,
+            0.0048666950315237045,
+            0.0042133391834795475,
+            -0.0018053868552669883,
+            -0.006926036439836025,
+            0.01750902086496353,
+            -0.008555963635444641,
+            0.013547327369451523,
+            0.019023513421416283,
+            -0.03664335235953331,
+            0.018127746880054474,
+            0.030696194618940353,
+            0.04310765117406845,
+            -0.0011774261947721243,
+            -0.01449850294739008,
+            -0.018090806901454926,
+            -0.01555126067250967,
+            0.004229499492794275,
+            0.015643607825040817,
+            0.02015014924108982,
+            0.005379221867769957,
+            0.0068244547583162785,
+            -0.011460281908512115,
+            -0.018386319279670715,
+            0.009502522647380829,
+            -0.0032991021871566772,
+            -0.0200208630412817,
+            0.032303035259246826,
+            0.0003789119073189795,
+            -0.008966908790171146,
+            -0.03051150031387806,
+            0.006141085643321276,
+            0.003566908882930875,
+            0.017370499670505524,
+            0.007558614481240511,
+            -0.001290551503188908,
+            -0.008158870972692966,
+            0.0035415133461356163,
+            0.008943822234869003,
+            -0.008999230340123177,
+            0.04181479290127754,
+            -0.0009627190884202719,
+            0.013464215211570263,
+            0.006699786055833101,
+            -0.021036682650446892,
+            0.0002056167140835896,
+            -0.005900983233004808,
+            -0.01689029484987259,
+            0.001274390728212893,
+            -0.02077811025083065,
+            -0.0027311670128256083,
+            0.025118425488471985,
+            -0.00929935835301876,
+            -0.029661906883120537,
+            0.030234457924962044,
+            -0.012513039633631706,
+            -0.02123984508216381,
+            0.0044395895674824715,
+            0.006879862863570452,
+            0.013944420032203197,
+            0.00624266779050231,
+            -0.004894399084150791,
+            0.01590218022465706,
+            0.008315861225128174,
+            0.001127789611928165,
+            0.02328995242714882,
+            -0.019633004441857338,
+            -0.00524531863629818,
+            -0.0022244120482355356,
+            -0.020888926461338997,
+            -0.01635468192398548,
+            0.006335014943033457,
+            -0.023696279153227806,
+            -0.018792646005749702,
+            -0.001254766946658492,
+            -0.0092347152531147,
+            0.016049936413764954,
+            0.03051150031387806,
+            0.0035368960816413164,
+            -0.01254997868090868,
+            0.007974176667630672,
+            -0.00046606454998254776,
+            -0.0038831979036331177,
+            -0.05042154714465141,
+            -0.021258315071463585,
+            0.011829670518636703,
+            0.007909533567726612,
+            -0.01681641675531864,
+            -0.001661094487644732,
+            0.0036061564460396767,
+            0.06763505935668945,
+            -0.02232954278588295,
+            -0.016576314345002174,
+            -0.030603846535086632,
+            0.002384865190833807,
+            0.003539204830303788,
+            0.003673108061775565,
+            0.003989397082477808,
+            -0.014830953441560268,
+            0.030252927914261818,
+            -0.0012963231420144439,
+            0.04613664001226425,
+            -0.009604103863239288,
+            0.008860709145665169,
+            0.028110474348068237,
+            0.0199839249253273,
+            0.015163403004407883,
+            -0.0238440353423357,
+            -0.000534170598257333,
+            -0.00905463844537735,
+            0.0022417271975427866,
+            -0.01439692173153162,
+            0.008200427517294884,
+            0.02774108573794365,
+            -0.016447028145194054,
+            0.00649662222713232,
+            0.017102692276239395,
+            -0.020427191630005836,
+            -0.022569645196199417,
+            -0.0011924325954169035,
+            -0.0008091919589787722,
+            0.02668832801282406,
+            0.01299324445426464,
+            -0.005600854754447937,
+            0.010619922541081905,
+            0.022089440375566483,
+            -0.007678666152060032,
+            0.01728738658130169,
+            0.014119880273938179,
+            -0.014489268884062767,
+            0.011811201460659504,
+            0.007295425049960613,
+            0.0029158613178879023,
+            -0.03147191181778908,
+            -0.022237194702029228,
+            -0.0018965797498822212,
+            0.007946472615003586,
+            -0.001829628017731011,
+            -0.02236648090183735,
+            -0.008389739319682121,
+            0.02777802385389805,
+            -0.00840820837765932,
+            0.011663445271551609,
+            0.0002845446579158306,
+            -0.009936554357409477,
+            0.029902009293437004,
+            0.03457477316260338,
+            -0.002913552802056074,
+            0.007041470613330603,
+            -0.029809661209583282,
+            0.019743822515010834,
+            -0.018100041896104813,
+            -0.0038139375392347574,
+            -0.03571987897157669,
+            0.03856417164206505,
+            0.016280803829431534,
+            -0.0028535271994769573,
+            -0.004751261323690414,
+            0.028516801074147224,
+            0.02299444191157818,
+            -0.0386011116206646,
+            0.03274630010128021,
+            0.008135784417390823,
+            0.0031651987228542566,
+            -0.0038624198641628027,
+            -0.020722702145576477,
+            0.009114664047956467,
+            -0.033725179731845856,
+            0.0020974348299205303,
+            -0.01533886231482029,
+            0.005619324278086424,
+            -0.0246751606464386,
+            -0.0146277891471982,
+            0.0036823428235948086,
+            -0.006699786055833101,
+            0.005259170662611723,
+            -0.0036615647841244936,
+            0.003359127789735794,
+            -0.002754253800958395,
+            -0.02203403040766716,
+            -0.033060282468795776,
+            0.026115775108337402,
+            -0.01250380463898182,
+            0.004513467196375132,
+            -0.009484052658081055,
+            -0.03106558322906494,
+            0.02005780301988125,
+            -0.013205642811954021,
+            -0.03248772770166397,
+            -0.009303975850343704,
+            0.026041897013783455,
+            0.000664899533148855,
+            -0.008980761282145977,
+            0.00920239370316267,
+            0.04377255216240883,
+            0.03634783998131752,
+            -0.017952287569642067,
+            0.014239931479096413,
+            0.019928516820073128,
+            0.0010221676202490926,
+            0.013242581859230995,
+            -0.019614536315202713,
+            -0.013362633064389229,
+            0.004407268017530441,
+            -0.019448310136795044,
+            -0.014757075347006321,
+            -0.040817443281412125,
+            -0.00986267626285553,
+            0.02602342888712883,
+            -0.01544967945665121,
+            -0.015357332304120064,
+            0.03919213265180588,
+            0.007036853116005659,
+            -0.04613664001226425,
+            -0.00831124372780323,
+            0.027851901948451996,
+            0.00270115421153605,
+            -0.012429927475750446,
+            0.004880547057837248,
+            0.013353399001061916,
+            -0.027131594717502594,
+            -0.001456776401028037,
+            0.023622402921319008,
+            0.03054843842983246,
+            -0.03440855070948601,
+            -0.01519110705703497,
+            0.014812483452260494,
+            -0.0010071611031889915,
+            -0.00793262105435133,
+            -0.014747840352356434,
+            -0.0068244547583162785,
+            0.022274134680628777,
+            0.006515091750770807,
+            -0.01286395825445652,
+            0.006196494214236736,
+            -0.011755792424082756,
+            -0.03472252935171127,
+            0.02266199141740799,
+            -0.01286395825445652,
+            -0.00313749467022717,
+            0.01735202968120575,
+            0.003996323328465223,
+            0.007023001089692116,
+            0.012245233170688152,
+            -0.012706968933343887,
+            -0.017102692276239395,
+            0.028849251568317413,
+            0.02460128255188465,
+            -0.022384950891137123,
+            0.015772894024848938,
+            -0.0045550232753157616,
+            0.01918973959982395,
+            -0.012808550149202347,
+            -0.007309277541935444,
+            -0.0019646857399493456,
+            0.0003399529668968171,
+            0.003322188975289464,
+            0.06257443130016327,
+            -0.03974621742963791,
+            -0.019466780126094818,
+            0.008597520180046558,
+            0.016973407939076424,
+            -0.02321607433259487,
+            -0.01456314604729414,
+            0.028719965368509293,
+            -0.025487814098596573,
+            0.02484138496220112,
+            0.014507737942039967,
+            -0.014535441994667053,
+            -0.0188295841217041,
+            0.015301923267543316,
+            0.02668832801282406,
+            -0.010444463230669498,
+            0.007309277541935444,
+            -0.01326105184853077,
+            0.004903634078800678,
+            0.006468918174505234,
+            0.010324412025511265,
+            -0.008597520180046558,
+            0.005199145060032606,
+            0.021516887471079826,
+            0.0002038852107943967,
+            -0.00269191968254745,
+            -0.00965027790516615,
+            0.014618555083870888,
+            0.007803334388881922,
+            0.000943095306865871,
+            -0.0005249358364380896,
+            -3.19607715937309e-05,
+            -0.006616673897951841,
+            0.021332193166017532,
+            -0.026559041813015938,
+            -0.002375630661845207,
+            -0.007669431157410145,
+            0.024102607741951942,
+            -0.01685335673391819,
+            0.02022402733564377,
+            -0.0070091490633785725,
+            0.026669858023524284,
+            0.00461735762655735,
+            0.00012438007979653776,
+            0.012300641275942326,
+            0.018395552411675453,
+            -0.02292056381702423,
+            0.002876613987609744,
+            -0.024989139288663864,
+            -0.0009125053184106946,
+            -0.0027265497483313084,
+            0.0269469004124403,
+            -0.00020720393513329327,
+            -0.02783343195915222,
+            0.03165660426020622,
+            -0.006048738490790129,
+            0.007323129568248987,
+            -0.012457631528377533,
+            -0.007757160812616348,
+            0.017407439649105072,
+            -0.03361436352133751,
+            0.015006412751972675,
+            0.020002393051981926,
+            0.012817785143852234,
+            -0.017527490854263306,
+            0.030197519809007645,
+            -0.001605686149559915,
+            -0.01935596391558647,
+            0.011534160003066063,
+            0.00199585291557014,
+            0.01070303563028574,
+            -0.0044557503424584866,
+            0.00849132053554058,
+            -0.02127678506076336,
+            -0.012577682733535767,
+            0.019743822515010834,
+            0.018681829795241356,
+            -0.004564258269965649,
+            -0.0005191641394048929,
+            -0.0017672936664894223,
+            -0.024435056373476982,
+            -0.018035398796200752,
+            0.001420991844497621,
+            -0.0008836468332447112,
+            0.0029851216822862625,
+            -0.0051437364891171455,
+            -0.03492569550871849,
+            0.004130226559937,
+            0.021646173670887947,
+            0.014600085094571114,
+            0.0067828986793756485,
+            -0.021313723176717758,
+            0.008320478722453117,
+            0.013990594074130058,
+            -0.023825565353035927,
+            0.03139803186058998,
+            -0.00910542905330658,
+            -0.03719743341207504,
+            0.004407268017530441,
+            0.0051206499338150024,
+            -0.016410090029239655,
+            -0.0004279713612049818,
+            0.01183890551328659,
+            -0.0061780246905982494,
+            0.016945702955126762,
+            0.021036682650446892,
+            -0.016908764839172363,
+            0.0088560925796628,
+            0.013381103053689003,
+            -0.012365284375846386,
+            -0.0014613937819376588,
+            -0.0006545104552060366,
+            -0.00712458323687315,
+            -0.0014867892023175955,
+            0.059323813766241074,
+            -0.012171355076134205,
+            -0.0038901239167898893,
+            -0.025968020781874657,
+            0.024952201172709465,
+            -0.01989157684147358,
+            -0.009668746963143349,
+            -0.001956605352461338,
+            -0.010887729935348034,
+            -0.009982727468013763,
+            0.005956391338258982,
+            -0.014323043636977673,
+            0.018293971195816994,
+            0.03054843842983246,
+            0.027334757149219513,
+            0.003559982869774103,
+            0.014193757437169552,
+            -0.00628422386944294,
+            -0.004693544004112482,
+            0.02434271015226841,
+            -0.019448310136795044,
+            0.0064781527034938335,
+            0.0007786019705235958,
+            -0.0007474347949028015,
+            -0.004432663321495056,
+            -0.010832320898771286,
+            -0.04100213572382927,
+            -0.0292555782943964,
+            0.02249576710164547,
+            0.006173407193273306,
+            0.020888926461338997,
+            0.002327148336917162,
+            0.009677981957793236,
+            -0.0028004273772239685,
+            0.012032833881676197,
+            0.011303291656076908,
+            -0.02391791343688965,
+            -0.014516972936689854,
+            -0.0033337322529405355,
+            -0.004559640772640705,
+            -0.012559212744235992,
+            -0.010555279441177845,
+            0.018949635326862335,
+            0.01800769567489624,
+            0.009548695757985115,
+            -0.015837537124753,
+            0.00036159681621938944,
+            -0.0007029927219264209,
+            -0.0013702008873224258,
+            0.0008253526757471263,
+            -0.007083026692271233,
+            -0.033466607332229614,
+            0.009354766458272934,
+            -0.016770243644714355,
+            0.006196494214236736,
+            0.015412740409374237,
+            -0.010943138040602207,
+            -0.0032575458753854036,
+            0.0046312096528708935,
+            -0.02183086797595024,
+            -0.007817186415195465,
+            -0.018358614295721054,
+            0.006704403553158045,
+            -0.027851901948451996,
+            0.0012097477447241545,
+            0.00803420227020979,
+            0.0010573748731985688,
+            0.026263531297445297,
+            0.00920239370316267,
+            -0.030880888924002647,
+            0.02321607433259487,
+            -0.003158272709697485,
+            0.008805301040410995,
+            -0.0037700727116316557,
+            -0.012402222491800785,
+            0.01636391505599022,
+            0.0074200937524437904,
+            0.00840820837765932,
+            -0.008126549422740936,
+            0.019651474431157112,
+            -0.0001968148717423901,
+            0.007138435263186693,
+            0.017195040360093117,
+            -0.022975971922278404,
+            -0.006085677538067102,
+            -0.0026896109338849783,
+            0.017933817580342293,
+            0.026540571823716164,
+            -0.01368584856390953,
+            -0.017407439649105072,
+            -0.009585634805262089,
+            0.023991791531443596,
+            0.014600085094571114,
+            0.015874475240707397,
+            0.002256733598187566,
+            -0.023142196238040924,
+            -0.0199839249253273,
+            0.02186780609190464,
+            -0.004266438540071249,
+            0.0017430525040253997,
+            0.026374347507953644,
+            -0.028405984863638878,
+            -0.014406155794858932,
+            -0.022089440375566483,
+            0.006034886464476585,
+            0.0012293715262785554,
+            0.009050020948052406,
+            -0.003119025146588683,
+            -0.015883710235357285,
+            -0.003446857677772641,
+            0.0013517314800992608,
+            -0.005240701138973236,
+            0.003998631611466408,
+            0.010130482725799084,
+            -0.003250619862228632,
+            0.005610089749097824,
+            0.011617272160947323,
+            -0.0238440353423357,
+            -0.004372637718915939,
+            0.010305942967534065,
+            -0.02262505330145359,
+            -0.02598648890852928,
+            -0.0027219324838370085,
+            0.014313808642327785,
+            0.016668662428855896,
+            0.006399658042937517,
+            0.006538178771734238,
+            -0.006399658042937517,
+            -0.027981188148260117,
+            0.01860795170068741,
+            -0.004351859912276268,
+            0.00405634893104434,
+            -0.013778195716440678,
+            0.020888926461338997,
+            0.017592133954167366,
+            0.007646344602108002,
+            0.009677981957793236,
+            -0.013519623316824436,
+            -0.03680957481265068,
+            -0.00034803335438482463,
+            0.00910542905330658,
+            -0.006335014943033457,
+            0.003012825967743993,
+            -0.008006498217582703,
+            -0.017832236364483833,
+            0.01883881911635399,
+            -0.007863360457122326,
+            0.0021932448726147413,
+            0.004642753396183252,
+            -0.032044462859630585,
+            0.008763745427131653,
+            0.019540658220648766,
+            0.01857101358473301,
+            -0.01446156483143568,
+            -0.015680547803640366,
+            -0.01276237703859806,
+            0.0107676787301898,
+            0.002585720270872116,
+            0.000738777220249176,
+            -0.0188295841217041,
+            -0.021720051765441895,
+            -0.028405984863638878,
+            0.006256519816815853,
+            0.0005335934110917151,
+            -0.022902093827724457,
+            -0.011238648556172848,
+            -0.005503890570253134,
+            0.01896810531616211,
+            0.01276237703859806,
+            0.007489354349672794,
+            0.02421342395246029,
+            -0.008306626230478287,
+            0.017130397260189056,
+            0.004940572660416365,
+            0.01368584856390953,
+            -0.0142491664737463,
+            -0.0002904606517404318,
+            -0.00988114532083273,
+            0.01926361583173275,
+            0.029477212578058243,
+            -0.008833005093038082,
+            -0.004679691977798939,
+            0.011330995708703995,
+            -0.004598888102918863,
+            0.022107908502221107,
+            0.02341923862695694,
+            -0.01085079088807106,
+            0.03198905289173126,
+            0.0008190038497559726,
+            0.015181872062385082,
+            -0.010527575388550758,
+            -0.013085591606795788,
+            -0.009585634805262089,
+            0.00042508551268838346,
+            -0.028812311589717865,
+            -0.027464043349027634,
+            0.009290123358368874,
+            0.0075216758996248245,
+            0.003876271890476346,
+            -0.015458913519978523,
+            0.009188542142510414,
+            0.0030913210939615965,
+            0.01613304764032364,
+            0.009377853944897652,
+            0.09301205724477768,
+            0.012106711976230145,
+            0.01566207781434059,
+            0.013667378574609756,
+            0.0011185549665242434,
+            -0.013150234706699848,
+            0.005042154807597399,
+            -0.0016368533251807094,
+            9.480012522544712e-05,
+            -0.005910217761993408,
+            -0.014166053384542465,
+            -0.01774912327528,
+            -0.008588285185396671,
+            0.025968020781874657,
+            0.015292689204216003,
+            -0.014877126552164555,
+            -0.004912868607789278,
+            -0.020833518356084824,
+            -0.003654638770967722,
+            -0.013177938759326935,
+            0.009073108434677124,
+            -0.0216092336922884,
+            -0.006501239724457264,
+            0.013565797358751297,
+            0.021184436976909637,
+            0.004109448287636042,
+            -0.015689780935645103,
+            -0.006191876716911793,
+            0.027223940938711166,
+            0.015588199719786644,
+            0.009322444908320904,
+            -0.006542795803397894,
+            -0.004328772891312838,
+            0.025376997888088226,
+            -0.006939888931810856,
+            0.018811115995049477,
+            0.01009354367852211,
+            -0.008523642085492611,
+            -0.0034607097040861845,
+            0.0010025438386946917,
+            0.006787515711039305,
+            0.003964001778513193,
+            0.01446156483143568,
+            0.018644889816641808,
+            0.012365284375846386,
+            -0.009613338857889175,
+            -0.008676014840602875,
+            0.004268747288733721,
+            -0.01913432963192463,
+            0.020205557346343994,
+            -0.019540658220648766,
+            -0.011635741218924522,
+            -0.02345617674291134,
+            0.0006851004436612129,
+            -0.0014579306589439511,
+            0.005637793801724911,
+            -0.006972210016101599,
+            -0.01859871670603752,
+            -0.029366394504904747,
+            0.01935596391558647,
+            0.0013690466294065118,
+            -0.01162650715559721,
+            -0.011700384318828583,
+            0.016807181760668755,
+            -0.005157588515430689,
+            0.01368584856390953,
+            0.0034768704790621996,
+            0.011469516903162003,
+            -0.008648310787975788,
+            0.024859854951500893,
+            0.0030913210939615965,
+            0.0005491769989021122,
+            0.007276955991983414,
+            -0.008648310787975788,
+            0.014064472168684006,
+            0.019466780126094818,
+            0.015819067135453224,
+            -0.0010123556712642312,
+            0.015717485919594765,
+            -0.00630731089040637,
+            -0.005817870609462261,
+            0.012457631528377533,
+            -0.003691577585414052,
+            0.0077987173572182655,
+            0.004176400136202574,
+            0.0026849934365600348,
+            0.020851988345384598,
+            0.005152971483767033,
+            0.020242497324943542,
+            0.02018708735704422,
+            -0.036754168570041656,
+            -0.003506883280351758,
+            -0.012235998176038265,
+            -0.016382385045289993,
+            0.034390080720186234,
+            0.006353484466671944,
+            -0.01009354367852211,
+            0.003511500544846058,
+            -0.01385207287967205,
+            -0.00631654541939497,
+            0.010989311151206493,
+            -0.005623941775411367,
+            0.006588969379663467,
+            0.006579734850674868,
+            -0.01242069248110056,
+            -0.004236425738781691,
+            0.018460195511579514,
+            -8.520467963535339e-05,
+            -0.007914151065051556,
+            -0.01242069248110056,
+            -0.0034837964922189713,
+            0.005222231615334749,
+            -0.04281214252114296,
+            -0.00917007215321064,
+            0.0011710773687809706,
+            0.0009078879957087338,
+            0.006866010837256908,
+            0.009146985597908497,
+            -0.02027943544089794,
+            0.0027773405890911818,
+            0.01893116720020771,
+            -0.03195211663842201,
+            -0.0032390763517469168,
+            -0.01206977292895317,
+            0.010509106330573559,
+            0.010795382782816887,
+            0.0011618426069617271,
+            -0.004658914171159267,
+            -0.0024310387670993805,
+            0.001940444577485323,
+            0.011201709508895874,
+            -0.0031282599084079266,
+            0.015505087561905384,
+            -0.0007918768678791821,
+            0.010758443735539913,
+            -0.00044788370723836124,
+            0.004721248522400856,
+            0.017730653285980225,
+            0.016280803829431534,
+            -0.002516459906473756,
+            -0.014378451742231846,
+            0.009548695757985115,
+            0.002673450158908963,
+            0.013020948506891727,
+            -0.0068244547583162785,
+            0.01939290203154087,
+            -0.02476750686764717,
+            0.0026665241457521915,
+            -0.001568747335113585,
+            0.003139803186058998,
+            -0.0066397604532539845,
+            -0.013981359079480171,
+            -0.001972766127437353,
+            -0.031878236681222916,
+            -0.03298640251159668,
+            -0.01649320125579834,
+            0.00920239370316267,
+            0.005102180410176516,
+            0.011109362356364727,
+            -0.005840957630425692,
+            -0.008740657940506935,
+            0.005166823510080576,
+            0.031157929450273514,
+            0.0008772979490458965,
+            -0.005060624331235886,
+            0.0028142796363681555,
+            0.013122530654072762,
+            0.02552475407719612,
+            -0.007410859223455191,
+            -0.028092004358768463,
+            -0.0006781744305044413,
+            -0.0001631947379792109,
+            -0.02375168912112713,
+            -0.02807353436946869,
+            -0.009013081900775433,
+            -0.0054207779467105865,
+            -0.0020778109319508076,
+            0.0020697305444628,
+            0.01099854614585638,
+            0.007397007197141647,
+            -0.004229499492794275,
+            0.005157588515430689,
+            -0.00984420720487833,
+            0.006898332387208939,
+            0.0036223172210156918,
+            0.01169115025550127,
+            -0.026466695591807365,
+            -0.007974176667630672,
+            -0.025469345971941948,
+            -0.007438563276082277,
+            0.006889097858220339,
+            -0.01446156483143568,
+            0.015301923267543316,
+            0.022052500396966934,
+            0.018589481711387634,
+            -0.010666096583008766,
+            0.029532620683312416,
+            0.016770243644714355,
+            -0.03461171314120293,
+            0.021221376955509186,
+            0.03014211170375347,
+            -0.01681641675531864,
+            -0.0005194527329877019,
+            -0.00033158401492983103,
+            0.019633004441857338,
+            0.012919367291033268,
+            0.010361351072788239,
+            -0.013399572111666203,
+            0.00460812309756875,
+            -0.01517263799905777,
+            -0.05197297781705856,
+            0.0039085932075977325,
+            0.024398118257522583,
+            0.004668148700147867,
+            -0.004483454395085573,
+            0.016253098845481873,
+            0.0005252244300208986,
+            -0.006787515711039305,
+            0.03169354423880577,
+            0.02744557522237301,
+            0.01922667771577835,
+            0.0016841812757775187,
+            -0.006884480360895395,
+            -0.029403334483504295,
+            -0.01676100865006447,
+            0.006404275074601173,
+            0.014110645279288292,
+            0.006990679539740086,
+            -0.01790611259639263,
+            -0.02430577203631401,
+            0.005356134846806526,
+            0.023530054837465286,
+            0.004211030434817076,
+            0.017859939485788345,
+            0.021018212661147118,
+            5.732016870751977e-05,
+            0.01582830213010311,
+            -0.008648310787975788,
+            -0.0006066053756512702,
+            -0.015754424035549164,
+            -0.00013974722241982818,
+            -0.014055237174034119,
+            0.0012859341222792864,
+            0.002643437357619405,
+            0.006173407193273306,
+            0.006621290929615498,
+            0.006672082003206015,
+            -0.009257802739739418,
+            -0.004968277178704739,
+            -3.21411389450077e-05,
+            0.016465498134493828,
+            -0.0068244547583162785,
+            0.004707396030426025,
+            -0.012143651023507118,
+            -0.004317229613661766,
+            0.008546729572117329,
+            0.0007001068443059921,
+            0.007840273901820183,
+            0.012152886018157005,
+            0.004116374533623457,
+            0.010112013667821884,
+            -0.0010764215840026736,
+            0.0024125694762915373,
+            -0.004737408831715584,
+            0.004280290566384792,
+            0.01198666077107191,
+            -0.004174091387540102,
+            -0.006187259219586849,
+            -0.02450893446803093,
+            0.037751518189907074,
+            -0.023179136216640472,
+            -0.002844292437657714,
+            -0.01057374943047762,
+            0.022514237090945244,
+            -0.0042548952624201775,
+            0.01877417601644993,
+            -0.011174005456268787,
+            -0.009368618950247765,
+            0.014784779399633408,
+            -0.006810602732002735,
+            -0.025709448382258415,
+            -0.01015818677842617,
+            -0.005707054398953915,
+            0.02421342395246029,
+            -0.007216930389404297,
+            -0.012688498944044113,
+            0.009142368100583553,
+            -0.0035761436447501183,
+            0.004940572660416365,
+            -0.020334843546152115,
+            -0.006588969379663467,
+            0.00010259191913064569,
+            -0.007553997449576855,
+            -0.009770329110324383,
+            -0.016216160729527473,
+            0.009327062405645847,
+            -0.003246002597734332,
+            0.0037423684261739254,
+            0.014156819321215153,
+            -0.008676014840602875,
+            -0.0014325352385640144,
+            0.021129028871655464,
+            0.0033129542134702206,
+            -0.004628901369869709,
+            0.009908849373459816,
+            -0.01169115025550127,
+            0.01952218823134899,
+            -0.015412740409374237,
+            -0.002085891319438815,
+            -0.007710987236350775,
+            -0.0077063702046871185,
+            -0.006584352347999811,
+            0.007004531566053629,
+            0.0031074818689376116,
+            0.021036682650446892,
+            -0.003495339769870043,
+            -0.009724155068397522,
+            -0.017832236364483833,
+            0.006805985234677792,
+            0.018848054111003876,
+            -0.000645852938760072,
+            -0.001607994781807065,
+            -0.0123375803232193,
+            -0.013141000643372536,
+            0.005268405191600323,
+            -0.0039940145798027515,
+            0.00269191968254745,
+            -0.026245061308145523,
+            -0.010028901509940624,
+            0.0045388625003397465,
+            0.0315273180603981,
+            -0.0075124409049749374,
+            -0.010527575388550758,
+            0.005106797907501459,
+            0.008075758814811707,
+            -0.01959606632590294,
+            -0.016945702955126762,
+            0.003282941412180662,
+            0.018183154985308647,
+            -0.0012697733473032713,
+            -0.0032021375373005867,
+            0.0013794356491416693,
+            -0.0016483967192471027,
+            -0.015200342051684856,
+            0.002309833187609911,
+            0.011090893298387527,
+            -0.030529970303177834,
+            0.003931680228561163,
+            -0.049498073756694794,
+            0.027464043349027634,
+            0.013713552616536617,
+            0.002952800365164876,
+            -0.008837622590363026,
+            -0.022551175206899643,
+            -0.023659341037273407,
+            -0.002936639590188861,
+            0.042073365300893784,
+            0.009096194989979267,
+            -0.010952373035252094,
+            0.022846685722470284,
+            -0.0017118854448199272,
+            -0.004234116990119219,
+            -0.01347345020622015,
+            0.0077479262836277485,
+            -0.0062011112459003925,
+            0.008117315359413624,
+            0.009373236447572708,
+            0.008232749067246914,
+            0.01800769567489624,
+            0.0010452544083818793,
+            0.010582984425127506,
+            -0.02870149537920952,
+            -0.0035899956710636616,
+            0.010906198993325233,
+            0.008528259582817554,
+            0.001420991844497621,
+            0.02031637355685234,
+            0.013584266416728497,
+            -0.00829277466982603,
+            -0.002938948106020689,
+            0.019152799621224403,
+            -0.014618555083870888,
+            0.014960238710045815,
+            0.004155621863901615,
+            0.009724155068397522,
+            0.0007053014123812318,
+            -0.001627618563361466,
+            0.025340059772133827,
+            0.003255237126722932,
+            -0.026392817497253418,
+            -0.021627703681588173,
+            0.010666096583008766,
+            -0.017869174480438232,
+            -0.00643197912722826,
+            -0.005836340133100748,
+            0.0038023940287530422,
+            0.010786147788167,
+            0.010462932288646698,
+            0.01926361583173275,
+            0.011044720187783241,
+            0.0016599401133134961,
+            -0.0030382215045392513,
+            0.013214877806603909,
+            -0.01804463379085064,
+            0.0013055579038336873,
+            -0.018654124811291695,
+            0.00895767379552126,
+            0.012125181034207344,
+            -0.00037631465238519013,
+            0.007295425049960613,
+            0.002249807585030794,
+            -0.0028373664245009422,
+            0.014877126552164555,
+            0.0079002995043993,
+            0.011257118545472622,
+            -0.027500983327627182,
+            0.0024541255552321672,
+            -0.02572791650891304,
+            0.0200208630412817,
+            -0.016530141234397888,
+            0.008463616482913494,
+            0.013094826601445675,
+            -0.0031467291992157698,
+            0.012956305406987667,
+            0.010980077087879181,
+            -0.002865070477128029,
+            0.004982129205018282,
+            0.023991791531443596,
+            -0.013861307874321938,
+            -0.05721829831600189,
+            0.0028927745297551155,
+            0.004825138952583075,
+            0.007854125462472439,
+            0.012439161539077759,
+            0.017231978476047516,
+            -0.0056978194043040276,
+            -0.014867891557514668,
+            -0.006898332387208939,
+            -0.002091662958264351,
+            -0.004252586513757706,
+            -0.0015918341232463717,
+            -0.0003933411499019712,
+            0.016253098845481873,
+            -0.0017038050573319197,
+            -0.017268918454647064,
+            0.001054489053785801,
+            -0.0022094056475907564,
+            0.004414193797856569,
+            0.01827550120651722,
+            -0.015782129019498825,
+            0.0057670800015330315,
+            0.01632697694003582,
+            0.0161976907402277,
+            0.006792133208364248,
+            0.008615989238023758,
+            -0.006191876716911793,
+            0.004991363734006882,
+            0.009456348605453968,
+            0.013944420032203197,
+            -0.006122616585344076,
+            -0.012079007923603058,
+            0.007881829515099525,
+            -0.01072150468826294,
+            0.02127678506076336,
+            -0.0030728515703231096,
+            -0.013676613569259644,
+            -0.01538503635674715,
+            -0.009761094115674496,
+            -0.009770329110324383,
+            0.005448481999337673,
+            0.001754595898091793,
+            0.003354510525241494,
+            -0.0028696877416223288,
+            0.01242069248110056,
+            0.0014094484504312277,
+            -0.016567079350352287,
+            0.0004233539802953601,
+            0.010416759178042412,
+            0.02905241586267948,
+            -0.0009754167986102402,
+            -0.018164684996008873,
+            -0.010204360820353031,
+            -0.01985463872551918,
+            0.02988353930413723,
+            0.0020085505675524473,
+            -0.0003895895497407764,
+            0.005157588515430689,
+            -0.006556647829711437,
+            -0.005517742596566677,
+            -0.007881829515099525,
+            0.01141410879790783,
+            0.0021389909088611603,
+            0.02220025658607483,
+            0.0008761436329223216,
+            0.0074708848260343075,
+            -0.006704403553158045,
+            0.004700470250099897,
+            -0.002366395900025964,
+            -0.004208721686154604,
+            0.003599230432882905,
+            0.0066397604532539845,
+            -0.0029597263783216476,
+            0.0034768704790621996,
+            0.005665497854351997,
+            -0.013547327369451523,
+            0.004811286926269531,
+            -0.02199709229171276,
+            0.030566908419132233,
+            0.006395040545612574,
+            0.015874475240707397,
+            0.004114065784960985,
+            -0.017952287569642067,
+            0.006727490108460188,
+            0.01183890551328659,
+            -0.0041209920309484005,
+            0.031213337555527687,
+            -0.002458743052557111,
+            -0.043550919741392136,
+            -0.00550850760191679,
+            0.02005780301988125,
+            0.03307875245809555,
+            0.0011791577562689781,
+            -0.01705651916563511,
+            0.0024471995420753956,
+            -0.0011197092244401574,
+            0.009142368100583553,
+            0.019577596336603165,
+            0.00720769539475441,
+            0.0040771267376840115,
+            -0.003915519453585148,
+            -0.014073706232011318,
+            -0.0031282599084079266,
+            0.020796580240130424,
+            -0.005845575127750635,
+            -0.008394356817007065,
+            0.009031551890075207,
+            0.01668713055551052,
+            -0.01863565668463707,
+            0.01070303563028574,
+            0.007443180773407221,
+            -0.02790731005370617,
+            -0.017102692276239395,
+            0.012365284375846386,
+            0.023437708616256714,
+            0.002456434303894639,
+            -0.014701667241752148,
+            -0.012079007923603058,
+            0.02245882898569107,
+            0.01642855815589428,
+            0.0061364686116576195,
+            -0.010610688477754593,
+            -0.0017430525040253997,
+            -0.011007781140506268,
+            -0.02114749886095524,
+            -0.0006423898739740252,
+            0.018349379301071167,
+            -0.014323043636977673,
+            0.008814536035060883,
+            -0.0005018490483053029,
+            0.0024310387670993805,
+            0.006339631974697113,
+            0.005799401551485062,
+            -0.011894313618540764,
+            0.005180675536394119,
+            0.0058317226357758045,
+            -0.005928687285631895,
+            0.004176400136202574,
+            0.0065058572217822075,
+            0.0012339889071881771,
+            -0.025081487372517586,
+            -0.003864728379994631,
+            -8.809052815195173e-05,
+            0.002162077696993947,
+            0.0010816160356625915,
+            -0.026965368539094925,
+            -0.012402222491800785,
+            -0.014840187504887581,
+            -0.025340059772133827,
+            -0.003996323328465223,
+            -0.01456314604729414,
+            -0.02031637355685234,
+            -0.001719965715892613,
+            -0.0035692176315933466,
+            0.0008086147718131542,
+            -0.0006602821522392333,
+            0.011072424240410328,
+            0.014036767184734344,
+            -0.0067321076057851315,
+            0.003070542821660638,
+            -0.0027911928482353687,
+            0.0323953814804554,
+            0.008583667688071728,
+            0.003964001778513193,
+            0.010832320898771286,
+            0.011183240450918674,
+            0.004116374533623457,
+            -0.006755194626748562,
+            -0.012910132296383381,
+            0.011487985961139202,
+            -0.00905463844537735,
+            -0.015985293313860893,
+            0.013464215211570263,
+            -0.010259768925607204,
+            0.0074708848260343075,
+            -0.0010060068452730775,
+            -0.015726720914244652,
+            -0.008315861225128174,
+            -0.0015952971298247576,
+            -0.01622539572417736,
+            -0.03507344797253609,
+            0.009631807915866375,
+            0.013972125016152859,
+            0.0013932876754552126,
+            -0.007397007197141647,
+            -0.028405984863638878,
+            -0.007424711249768734,
+            -0.006939888931810856,
+            0.004471911117434502,
+            0.01830320619046688,
+            0.00926703680306673,
+            -0.011469516903162003,
+            -0.008435912430286407,
+            -0.010361351072788239,
+            -0.003213681047782302,
+            0.005743992980569601,
+            0.0025372381787747145,
+            -0.000359576748451218,
+            -0.010361351072788239,
+            0.02173851989209652,
+            -0.01108165830373764,
+            0.0008034202619455755,
+            0.01448003388941288,
+            -0.026078836992383003,
+            0.0024841385893523693,
+            -0.02908935397863388,
+            -0.01681641675531864,
+            0.02648516371846199,
+            0.0025372381787747145,
+            0.013935185968875885,
+            -0.017333561554551125,
+            -0.004176400136202574,
+            0.018450962379574776,
+            -0.006921419408172369,
+            -0.025672508403658867,
+            -0.005748610477894545,
+            -0.010555279441177845,
+            -0.018432492390275,
+            0.012272937223315239,
+            0.0026064985431730747,
+            -0.014489268884062767,
+            -0.01542197447270155,
+            0.011801966466009617,
+            -0.005448481999337673,
+            -0.007807951886206865,
+            -0.02262505330145359,
+            0.026632919907569885,
+            -0.005268405191600323,
+            -0.019208207726478577,
+            -0.00136673788074404,
+            -0.014258400537073612,
+            0.0058825137093663216,
+            -0.00915160309523344,
+            0.005550064146518707,
+            0.010250533930957317,
+            -0.030326806008815765,
+            -0.009761094115674496,
+            -0.012836254201829433,
+            -0.01418452337384224,
+            -0.03911825641989708,
+            -0.005171440541744232,
+            0.011737323366105556,
+            -0.001036019646562636,
+            -0.01827550120651722,
+            -0.007609405554831028,
+            -0.013353399001061916,
+            -0.005009833257645369,
+            -0.0030543820466846228,
+            0.0073046600446105,
+            -0.007784865330904722,
+            -0.010619922541081905,
+            0.018524838611483574,
+            0.014433860778808594,
+            0.012125181034207344,
+            -0.019873108714818954,
+            -2.845266408257885e-06,
+            -0.00039622699841856956,
+            0.015135698951780796,
+            0.001699187676422298,
+            0.015034116804599762,
+            0.01057374943047762,
+            -0.020408721640706062,
+            -0.004926720634102821,
+            0.007632492575794458,
+            -0.013177938759326935,
+            -0.009419409558176994,
+            -0.0034791789948940277,
+            0.031915176659822464,
+            -0.00627498934045434,
+            0.027556391432881355,
+            -0.004074818454682827,
+            0.008994612842798233,
+            0.01527421921491623,
+            -0.0025187686551362276,
+            -0.026245061308145523,
+            -0.0038831979036331177,
+            -0.005799401551485062,
+            -0.010232064872980118,
+            -0.0006856776308268309,
+            -0.007821803912520409,
+            0.020574945956468582,
+            -0.005693201906979084,
+            0.011404873803257942,
+            0.0019889269024133682,
+            0.010813851840794086,
+            -0.016779478639364243,
+            -0.024749036878347397,
+            0.011912782676517963,
+            0.010102778673171997,
+            -0.0022255664225667715,
+            0.020925864577293396,
+            0.005448481999337673,
+            -0.0019150491571053863,
+            0.024121077731251717,
+            -0.004418811295181513,
+            -0.006667464505881071,
+            -0.01731509156525135,
+            0.0034214621409773827,
+            0.01599452644586563,
+            0.019208207726478577,
+            -0.019633004441857338,
+            0.011811201460659504,
+            0.038970500230789185,
+            0.00713381776586175,
+            -0.016945702955126762,
+            -0.026411285623908043,
+            -0.005730140954256058,
+            -0.008837622590363026,
+            0.005314578767865896,
+            -0.03274630010128021,
+            0.0002919035905506462,
+            -0.005157588515430689,
+            -0.005725523456931114,
+            0.016105344519019127,
+            0.011718854308128357,
+            -0.0038624198641628027,
+            -0.0014302266063168645,
+            0.012965540401637554,
+            -0.029957417398691177,
+            0.0003561137127690017,
+            0.008708336390554905,
+            -0.016142282634973526,
+            -0.007332364097237587,
+            0.007877212017774582,
+            -0.013353399001061916,
+            -0.004550406243652105,
+            -0.013399572111666203,
+            0.01265155989676714,
+            -0.00977956410497427,
+            -0.010398290120065212,
+            0.019651474431157112,
+            -0.004116374533623457,
+            -0.013307224959135056,
+            -0.0021828559692949057,
+            0.004026336129754782,
+            -0.03446395695209503,
+            0.020999742671847343,
+            0.005840957630425692,
+            0.00155258656013757,
+            -0.01164497621357441,
+            0.011995895765721798,
+            0.007766395807266235,
+            0.005323813296854496,
+            -0.009936554357409477,
+            -0.008126549422740936,
+            -0.008980761282145977,
+            0.003707738360390067,
+            0.011423342861235142,
+            -0.004164856858551502,
+            -0.009696451015770435,
+            0.0034122273791581392,
+            0.010582984425127506,
+            0.016964172944426537,
+            -0.0064781527034938335,
+            -0.013889011926949024,
+            0.0011878153309226036,
+            -0.002876613987609744,
+            -0.014637024141848087,
+            -0.0035345873329788446,
+            0.009123899042606354,
+            -0.016345446929335594,
+            -0.000193640444194898,
+            0.011580333113670349,
+            -0.010185890831053257,
+            -0.01918973959982395,
+            -0.025210773572325706,
+            -0.000573129509575665,
+            -0.006561265327036381,
+            0.000608914066106081,
+            -0.021553825587034225,
+            0.014720136299729347,
+            -0.0006037194980308414,
+            -0.022052500396966934,
+            0.017398204654455185,
+            -0.005069858860224485,
+            -0.00199585291557014,
+            0.030289866030216217,
+            0.00716613931581378,
+            -0.0015006412286311388,
+            -0.01972535252571106,
+            0.007821803912520409,
+            -0.005660880822688341,
+            -0.0067321076057851315,
+            -0.0007705215830355883,
+            0.02044565975666046,
+            -0.005106797907501459,
+            -0.005993330385535955,
+            0.025026079267263412,
+            0.02081504836678505,
+            0.013307224959135056,
+            -0.014904830604791641,
+            0.004610431846231222,
+            -0.026337409391999245,
+            0.022274134680628777,
+            0.009705686010420322,
+            -0.011358699761331081,
+            -0.007794099859893322,
+            0.0008917272207327187,
+            -0.014683197252452374,
+            0.025949550792574883,
+            0.011174005456268787,
+            -0.011958956718444824,
+            -0.008883796632289886,
+            -0.0023029071744531393,
+            0.0030797775834798813,
+            -0.008288157172501087,
+            -0.008551346138119698,
+            -0.004458059091120958,
+            0.018718767911195755,
+            0.005517742596566677,
+            -0.001165305613540113,
+            -0.012559212744235992,
+            -0.014646259136497974,
+            0.011958956718444824,
+            0.014341513626277447,
+            -0.004615048877894878,
+            -0.028092004358768463,
+            0.010564514435827732,
+            0.016530141234397888,
+            -0.00984420720487833,
+            0.004400341771543026,
+            -0.005730140954256058,
+            0.031675074249506,
+            -0.0019877725280821323,
+            -0.01754595898091793,
+            -0.008061906322836876,
+            0.026078836992383003,
+            0.0023410003632307053,
+            0.022237194702029228,
+            0.005891748704016209,
+            -0.0054900385439395905,
+            -0.014673963189125061,
+            0.0011179777793586254,
+            0.005864044185727835,
+            0.01198666077107191,
+            0.021479947492480278,
+            0.00395014975219965,
+            -0.006279606372117996,
+            0.00994578842073679,
+            -0.013399572111666203,
+            -0.012679263949394226,
+            0.00181231286842376,
+            -0.017435142770409584,
+            0.007600171025842428,
+            -0.023899443447589874,
+            -0.0014613937819376588,
+            0.013011714443564415,
+            0.009410175494849682,
+            0.0011231722310185432,
+            0.006099529564380646,
+            0.008385121822357178,
+            0.0008888413431122899,
+            -0.014655493199825287,
+            -0.005374604370445013,
+            -0.009225481189787388,
+            0.011377169750630856,
+            0.010435228236019611,
+            -0.010407524183392525,
+            -0.022255664691329002,
+            0.018284736201167107,
+            -0.016243863850831985,
+            0.007447798270732164,
+            -0.005032919812947512,
+            -0.0016022231429815292,
+            0.006127233617007732,
+            0.013205642811954021,
+            -0.02661444991827011,
+            0.012799316085875034,
+            -0.0021666951943188906,
+            -0.03017904981970787,
+            0.0025557074695825577,
+            -0.004317229613661766,
+            -0.0004957888158969581,
+            -0.014997177757322788,
+            0.01527421921491623,
+            -0.010619922541081905,
+            0.009350149892270565,
+            0.00536075234413147,
+            -0.0005206071073189378,
+            -0.003250619862228632,
+            0.004712013527750969,
+            -0.0014579306589439511,
+            -0.012790081091225147,
+            0.016613252460956573,
+            0.011765027418732643,
+            -0.005263787694275379,
+            0.007258486468344927,
+            -0.004656605422496796,
+            -0.014313808642327785,
+            -0.030234457924962044,
+            0.005979478359222412,
+            0.010213595815002918,
+            0.002398717449977994,
+            -0.00888841412961483,
+            -0.0008876870269887149,
+            0.0038531851023435593,
+            0.013759725727140903,
+            0.004485763143748045,
+            -0.010555279441177845,
+            -0.02476750686764717,
+            0.011866609565913677,
+            0.013002479448914528,
+            -0.012079007923603058,
+            -0.005887131206691265,
+            0.013787430711090565,
+            -0.006427362095564604,
+            0.001054489053785801,
+            0.0037746899761259556,
+            8.174165850505233e-05,
+            0.01162650715559721,
+            0.014710902236402035,
+            -0.012134416028857231,
+            0.022643523290753365,
+            -0.0015572039410471916,
+            -0.004610431846231222,
+            -0.006085677538067102,
+            0.0023525438737124205,
+            -0.011044720187783241,
+            0.0007659042021259665,
+            -0.0025764857418835163,
+            0.01698264107108116,
+            -0.0033291149884462357,
+            0.011875844560563564,
+            -0.008154253475368023,
+            0.00728157302364707,
+            -0.0005664920900017023,
+            -0.01106318924576044,
+            0.003303719451650977,
+            -0.01248533558100462,
+            0.011958956718444824,
+            -0.005600854754447937,
+            0.004275673069059849,
+            -0.022939033806324005,
+            0.017527490854263306,
+            0.009918084368109703,
+            -0.012374518439173698,
+            -0.017111927270889282,
+            0.013732021674513817,
+            -0.006048738490790129,
+            -0.047614194452762604,
+            -0.011894313618540764,
+            -0.02833210676908493,
+            0.00019970072025898844,
+            0.014720136299729347,
+            0.02816588245332241,
+            -0.012965540401637554,
+            -0.00265959813259542,
+            0.01748131588101387,
+            0.01072150468826294,
+            0.0009956177091225982,
+            0.01596682332456112,
+            -0.006939888931810856,
+            0.00965027790516615,
+            0.0034607097040861845,
+            0.0022982899099588394,
+            -0.001016973052173853,
+            -0.00042421973193995655,
+            -0.007406241726130247,
+            0.029606498777866364,
+            -0.008168105967342854,
+            0.004778965376317501,
+            -0.012162120081484318,
+            0.00811269786208868,
+            -0.0222926028072834,
+            0.0017257374711334705,
+            0.014710902236402035,
+            -0.021720051765441895,
+            0.006468918174505234,
+            0.003730825148522854,
+            -0.015680547803640366,
+            -0.0036500212736427784,
+            0.013962890021502972,
+            -0.01880188100039959,
+            -0.0025372381787747145,
+            0.004912868607789278,
+            0.012984010390937328,
+            0.008929969742894173,
+            0.004760495852679014,
+            -0.005762462504208088,
+            0.020999742671847343,
+            0.0011185549665242434,
+            0.008357417769730091,
+            0.004058657679706812,
+            0.00638118851929903,
+            0.0006348866736516356,
+            0.01599452644586563,
+            -0.009257802739739418,
+            0.003559982869774103,
+            0.019910046830773354,
+            -0.02522924356162548,
+            -0.01699187606573105,
+            0.03274630010128021,
+            0.0016726378817111254,
+            0.021978622302412987,
+            -0.0013690466294065118,
+            0.0031351859215646982,
+            0.0066397604532539845,
+            0.006214963737875223,
+            -0.0033129542134702206,
+            0.00624266779050231,
+            0.007276955991983414,
+            0.011238648556172848,
+            0.01741667278110981,
+            8.679189340909943e-05,
+            0.006057973485440016,
+            0.01347345020622015,
+            -0.02358546294271946,
+            0.006076443009078503,
+            0.005032919812947512,
+            0.007752543780952692,
+            -0.004238734487444162,
+            0.01926361583173275,
+            -0.003153655445203185,
+            -0.005254553165286779,
+            0.017527490854263306,
+            0.028646087273955345,
+            -0.0004600042593665421,
+            0.01566207781434059,
+            0.026503633707761765,
+            -0.004522701725363731,
+            -0.009936554357409477,
+            -0.006418127100914717,
+            -0.018284736201167107,
+            -0.0013078665360808372,
+            -0.0008201581658795476,
+            -0.0017569046467542648,
+            -0.01553279161453247,
+            0.0009119281312450767,
+            0.01622539572417736,
+            0.01353809330612421,
+            -0.025949550792574883,
+            0.01952218823134899,
+            0.0006902950117364526,
+            0.00324831111356616,
+            -0.00979803316295147,
+            0.014747840352356434,
+            -0.024250362068414688,
+            -0.002902009291574359,
+            0.017305856570601463,
+            -0.0161607526242733,
+            -0.012392988428473473,
+            0.0073138945735991,
+            -0.009211628697812557,
+            -0.011635741218924522,
+            0.006750577129423618,
+            0.0038624198641628027,
+            0.0108046168461442,
+            -0.00038843523361720145,
+            -0.00716613931581378,
+            -0.006588969379663467,
+            0.025173833593726158,
+            0.007383155170828104,
+            -0.005670115351676941,
+            -0.020297905430197716,
+            -0.009825737215578556,
+            -0.011977425776422024,
+            0.016650192439556122,
+            0.0017222744645550847,
+            -0.00720769539475441,
+            -0.0022648139856755733,
+            -0.013741256669163704,
+            -0.007831038907170296,
+            -0.02249576710164547,
+            0.009853441268205643,
+            0.014701667241752148,
+            0.009096194989979267,
+            0.010121248662471771,
+            0.0024218042381107807,
+            0.012300641275942326,
+            -0.000943095306865871,
+            -0.0007555151823908091,
+            0.009114664047956467,
+            0.014369217678904533,
+            -0.003474561730399728,
+            0.019946984946727753,
+            -0.012402222491800785,
+            0.009687216952443123,
+            -0.022089440375566483,
+            0.005517742596566677,
+            -0.00977956410497427,
+            0.005730140954256058,
+            0.005176058039069176,
+            -0.007503206375986338,
+            0.019873108714818954,
+            0.00545771699398756,
+            0.0006037194980308414,
+            -0.0003411073121242225,
+            0.014175288379192352,
+            0.0057901665568351746,
+            0.00893920473754406,
+            -0.01918973959982395,
+            0.0017661392921581864,
+            0.0014279178576543927,
+            0.010518341325223446,
+            -0.0028489097021520138,
+            0.010462932288646698,
+            -0.007567849475890398,
+            0.012365284375846386,
+            0.004995981231331825,
+            -0.00709226168692112,
+            0.026263531297445297,
+            0.0064781527034938335,
+            -0.0011947413440793753,
+            0.01022282987833023,
+            0.015440444462001324,
+            -0.015708250924944878,
+            -0.0012524583144113421,
+            -0.020427191630005836,
+            0.00012603943469002843,
+            0.0022255664225667715,
+            -0.02484138496220112,
+            0.0238809734582901,
+            0.023530054837465286,
+            0.0177121851593256,
+            -0.008948439732193947,
+            -0.004940572660416365,
+            -0.007928003557026386,
+            -0.0054623340256512165,
+            0.0018515604315325618,
+            -0.0005136810359545052,
+            0.023142196238040924,
+            0.009303975850343704,
+            0.035867635160684586,
+            0.01326105184853077,
+            -0.0020893544424325228,
+            0.0005399422952905297,
+            -0.004557332023978233,
+            -0.021258315071463585,
+            0.00941479206085205,
+            -0.01555126067250967,
+            -0.019041983410716057,
+            0.013205642811954021,
+            -0.00550850760191679,
+            -0.0007543608080595732,
+            -0.004416502546519041,
+            -0.008814536035060883,
+            0.015569730661809444,
+            -0.024102607741951942,
+            -0.014646259136497974,
+            0.016234630718827248,
+            -0.004765113350003958,
+            -0.017730653285980225,
+            -0.009276271797716618,
+            -0.02988353930413723,
+            -0.0038831979036331177,
+            0.02236648090183735,
+            0.002456434303894639,
+            0.0013182556722313166,
+            -0.001848097424954176,
+            -0.007678666152060032,
+            -0.006898332387208939,
+            0.002126293256878853,
+            0.021627703681588173,
+            -0.017989225685596466,
+            0.01101701520383358,
+            -0.01741667278110981,
+            -0.010001196525990963,
+            -0.0029089353047311306,
+            -0.019946984946727753,
+            0.009276271797716618,
+            0.004688926972448826,
+            -0.011127832345664501,
+            -0.0018746472196653485,
+            -0.017896879464387894,
+            -0.021165968850255013,
+            0.013667378574609756,
+            -0.003936297260224819,
+            0.012522274628281593,
+            0.008712953887879848,
+            0.006953740958124399,
+            0.002585720270872116,
+            0.058363400399684906,
+            -0.012291406281292439,
+            0.021018212661147118,
+            -0.013048653490841389,
+            0.03017904981970787,
+            -0.005365369841456413,
+            0.01781376637518406,
+            0.00918392464518547,
+            0.00460812309756875,
+            0.0018931166268885136,
+            0.0292555782943964,
+            0.022440358996391296,
+            -0.016465498134493828,
+            0.013233347795903683,
+            0.00042941427091136575,
+            0.009031551890075207,
+            -0.004982129205018282,
+            -0.011072424240410328,
+            -0.011487985961139202,
+            -0.020427191630005836,
+            -0.008509790524840355,
+            -0.00817734096199274,
+            -0.0032021375373005867,
+            -0.002143608406186104,
+            -0.003262163372710347,
+            -0.009918084368109703,
+            0.00938247051090002,
+            -0.02170158177614212,
+            0.028110474348068237,
+            0.01790611259639263,
+            0.003031295258551836,
+            0.0009298204095102847,
+            -0.003749294439330697,
+            0.007517058402299881,
+            0.0030405300203710794,
+            -0.0053699868731200695,
+            0.0003194634336978197,
+            0.01942984201014042,
+            -0.006935271434485912,
+            0.002050106879323721,
+            -0.009340914897620678,
+            0.024896793067455292,
+            -0.020168619230389595,
+            -0.0014579306589439511,
+            -0.002174775581806898,
+            -0.0004100790829397738,
+            -0.004116374533623457,
+            0.011783497408032417,
+            -0.0007058785413391888,
+            0.004146387334913015,
+            0.012263702228665352,
+            0.008482086472213268,
+            0.019005045294761658,
+            0.0216092336922884,
+            -0.005134501960128546,
+            0.0044049592688679695,
+            -0.007461650297045708,
+            -0.007226164918392897,
+            -0.012272937223315239,
+            -0.0008438221411779523,
+            -0.002327148336917162,
+            -0.022588113322854042,
+            -0.02820282056927681,
+            -0.0005673578125424683,
+            0.00369388610124588,
+            0.003262163372710347,
+            0.0034791789948940277,
+            -0.014304574579000473,
+            -0.01972535252571106,
+            0.023234544321894646,
+            -0.009834972210228443,
+            -0.02354852482676506,
+            0.013445746153593063,
+            -0.0011023941915482283,
+            -0.012900897301733494,
+            0.0027034629601985216,
+            -0.007590936031192541,
+            -0.032598547637462616,
+            0.004146387334913015,
+            -0.006150320637971163,
+            0.007447798270732164,
+            -0.033097218722105026,
+            -0.017222745344042778,
+            -0.0038347155787050724,
+            0.007475502323359251,
+            0.0015745189739391208,
+            0.062352798879146576,
+            -0.005651645828038454,
+            0.017231978476047516,
+            -0.015264985151588917,
+            -0.0012997862650081515,
+            -0.004875930026173592,
+            0.004940572660416365,
+            0.013104061596095562,
+            -0.01658554933965206,
+            -0.012134416028857231,
+            0.010961607098579407,
+            0.009017699398100376,
+            0.01276237703859806,
+            0.026208123192191124,
+            0.001071804203093052,
+            -0.014018298126757145,
+            -0.0033498930279165506,
+            -0.004520393442362547,
+            0.006621290929615498,
+            -0.006709021050482988,
+            0.0246751606464386,
+            -0.0005027148290537298,
+            -0.022846685722470284,
+            0.0003916096466127783,
+            -0.012725437991321087,
+            0.016899529844522476,
+            -0.013676613569259644,
+            -0.005217614118009806,
+            -0.017924582585692406,
+            -0.0254324059933424,
+            -0.018894227221608162,
+            0.007637109607458115,
+            0.0021701580844819546,
+            0.012291406281292439,
+            -0.0016680205008015037,
+            0.004857460502535105,
+            -0.007720222231000662,
+            0.0238809734582901,
+            -0.012291406281292439,
+            -0.010564514435827732,
+            0.002860453212633729,
+            -0.026559041813015938,
+            0.0019970072899013758,
+            0.01708422414958477,
+            -0.002918170066550374,
+            -0.008366651833057404,
+            -0.020870456472039223,
+            -0.03097323514521122,
+            0.016040701419115067,
+            -0.004843608476221561,
+            -0.003841641591861844,
+            -0.009567164815962315,
+            0.017102692276239395,
+            0.02035331353545189,
+            -0.002636511344462633,
+            0.03937682881951332,
+            0.02220025658607483,
+            -0.014747840352356434,
+            0.01658554933965206,
+            -0.016317741945385933,
+            -0.002957417629659176,
+            -0.011608037166297436,
+            0.014978708699345589,
+            -0.004079435486346483,
+            -0.030271397903561592,
+            0.025284651666879654,
+            0.0026526721194386482,
+            -0.0008380504441447556,
+            -0.006935271434485912,
+            0.0006423898739740252,
+            0.022218724712729454,
+            0.0013009405229240656,
+            -0.010379820130765438,
+            -0.010121248662471771,
+            0.009243950247764587,
+            0.009511756710708141,
+            0.0009482898749411106,
+            -0.03134262561798096,
+            0.013104061596095562,
+            0.005563916172832251,
+            -0.003269089385867119,
+            0.02123984508216381,
+            -0.012734672985970974,
+            0.021498417481780052,
+            0.020094741135835648,
+            -0.022237194702029228,
+            0.0007347370265051723,
+            -0.03387293592095375,
+            -0.014757075347006321,
+            0.00905463844537735,
+            -0.010213595815002918,
+            0.012476100586354733,
+            -0.0054023084230721,
+            -0.005291492212563753,
+            0.008925352245569229,
+            -0.01093390304595232,
+            -0.011266352608799934,
+            0.006958357989788055,
+            -0.0034122273791581392,
+            -0.007115348242223263,
+            0.003271397901698947,
+            0.012282171286642551,
+            0.020925864577293396,
+            -0.010730739682912827,
+            -0.0062934583984315395,
+            -0.009368618950247765,
+            0.028904659673571587,
+            0.00917007215321064,
+            -0.007623257581144571,
+            0.0039847795851528645,
+            0.005009833257645369,
+            0.006667464505881071,
+            0.00632577994838357,
+            0.004695852752774954,
+            0.013205642811954021,
+            0.007817186415195465,
+            0.023179136216640472,
+            -0.01446156483143568,
+            0.02345617674291134,
+            0.00627498934045434,
+            -0.010730739682912827,
+            -0.0012028217315673828,
+            0.0009032706147991121,
+            0.009567164815962315,
+            -0.022181786596775055,
+            -0.011358699761331081,
+            -0.005448481999337673,
+            -0.024859854951500893,
+            0.01926361583173275,
+            0.003287558676674962,
+            -0.0007843736675567925,
+            0.006528943777084351,
+            0.0025187686551362276,
+            0.0016934159211814404,
+            0.0018769559683278203,
+            -0.00024255557218566537,
+            -0.004224882461130619,
+            0.004252586513757706,
+            0.022902093827724457,
+            0.004321846645325422,
+            0.00014443672262132168,
+            0.0554082915186882,
+            -0.0029481828678399324,
+            -0.01141410879790783,
+            0.02245882898569107,
+            0.0018365540308877826,
+            0.011958956718444824,
+            -0.006298075895756483,
+            0.012429927475750446,
+            -0.026559041813015938,
+            -0.01099854614585638,
+            -0.03265395388007164,
+            0.017259683459997177,
+            0.002401025965809822,
+            -0.030419152230024338,
+            -0.007650961633771658,
+            0.0051852925680577755,
+            0.017158102244138718,
+            -0.011672680266201496,
+            0.026337409391999245,
+            0.007466267328709364,
+            -0.008994612842798233,
+            -0.00020330803818069398,
+            -0.01070303563028574,
+            0.006104147061705589,
+            -0.01689029484987259,
+            -0.01286395825445652,
+            -0.005356134846806526,
+            0.0019854637794196606,
+            0.00979803316295147,
+            0.02652210369706154,
+            0.022680461406707764,
+            -0.012208294123411179,
+            0.015652842819690704,
+            0.022865155711770058,
+            0.0018908079946413636,
+            0.012429927475750446,
+            -0.006376571021974087,
+            -0.024582812562584877,
+            -0.0378069244325161,
+            0.00639042304828763,
+            0.04159316048026085,
+            -0.005065241362899542,
+            0.004848225507885218,
+            0.027630269527435303,
+            0.00046029285294935107,
+            0.01714886724948883,
+            -0.005494655575603247,
+            0.005845575127750635,
+            0.01676100865006447,
+            0.0050513893365859985,
+            -0.04369867220520973,
+            0.011571098119020462,
+            0.0034976485185325146,
+            -0.005342282820492983,
+            0.014895596541464329,
+            -0.022311072796583176,
+            0.00012365861039143056,
+            0.0075632319785654545,
+            0.01334416400641203,
+            0.008200427517294884,
+            -0.004834373481571674,
+            0.004728174302726984,
+            -0.005868661683052778
+          ],
+          "title": "1 Introduction 3",
+          "keyphrases": [
+            "Introduction"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "84c5c313-9fbd-4bc2-a85f-3b62a64ae3db",
+      "type": "child",
+      "source": {
+        "id": "65742957-6bda-446d-a61a-0fe20313c760",
+        "properties": {
+          "page_content": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity\n\nWilliam Fedus∗ liamfedus@google.com Barret Zoph∗ barretzoph@google.com Noam Shazeer noam@google.com Google, Mountain View, CA 94043, USA Editor: Alexander Clark\n\nAbstract\n\nIn deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) models defy this and instead select different parameters for each incoming example. The result is a sparsely-activated model—with an outrageous number of parameters—but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs, and training instability. We address these with the introduction of the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques mitigate the instabilities, and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large (Raffel et al., 2019) to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the \"Colossal Clean Crawled Corpus\", and achieve a 4x speedup over the T5-XXL model.12 Keywords: mixture-of-experts, natural language processing, sparsity, large-scale machine learning, distributed computing arXiv:2101.03961v3 [cs.LG] 16 Jun 2022 Contents\n\n1 Introduction 3 2 Switch Transformer 4 2.1 Simplifying Sparse Routing 5 2.2 Efficient Sparse Routing 6 2.3 Putting It All Together: The Switch Transformer 8 2.4 Improved Training and Fine-Tuning Techniques 8 3 Scaling Properties 11 3.1 Scaling Results on a Step-Basis 12 3.2 Scaling Results on a Time-Basis 13 3.3 Scaling Versus a Larger Dense Model 13 4 Downstream Results 14 4.1 Fine-Tuning 14 4.2 Distillation 16 4.3 Multilingual Learning 17 5 Designing Models with Data, Model, and Expert-Parallelism 18 5.1 Data Parallelism 20 5.2 Model Parallelism 20 5.3 Model and Data Parallelism 21 5.4 Expert and Data Parallelism 22 5.5 Expert, Model and Data Parallelism 22 5.6 Towards Trillion Parameter Models 22 6 Related Work 24 7 Discussion 25 8 Future Work 26 9 Conclusion 27 A Switch for Attention 27 B Preventing Token Dropping with No-Token-Left-Behind 29\n\nC Encouraging Exploration Across Experts 29 D Switch Transformers in Lower Compute Regimes 29 E Relation of Upstream to Downstream Model Performance 32 F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/switch_transformers.md"
+          },
+          "summary": "The Switch Transformer is a sparsely-activated model that simplifies the Mixture of Experts (MoE) approach, allowing for efficient scaling to trillion-parameter models with reduced communication and computational costs. It achieves significant speedups in pre-training and fine-tuning by activating only a subset of neural network weights for each input, maintaining a constant computational cost. The model demonstrates superior scaling properties and improved performance across various natural language processing tasks, including multilingual settings. It also allows for effective distillation into smaller dense models, preserving a portion of the quality gains. The architecture is designed to maximize parameter count while keeping floating point operations per example constant, making it suitable for both large-scale and smaller-scale applications.",
+          "headlines": {
+            "1 Introduction": [],
+            "2 Switch Transformer": [
+              "2.1 Simplifying Sparse Routing",
+              "2.2 Efficient Sparse Routing",
+              "2.3 Putting It All Together: The Switch Transformer",
+              "2.4 Improved Training and Fine-Tuning Techniques"
+            ],
+            "3 Scaling Properties": [
+              "3.1 Scaling Results on a Step-Basis",
+              "3.2 Scaling Results on a Time-Basis",
+              "3.3 Scaling Versus a Larger Dense Model"
+            ],
+            "4 Downstream Results": [
+              "4.1 Fine-Tuning",
+              "4.2 Distillation",
+              "4.3 Multilingual Learning"
+            ],
+            "5 Designing Models with Data, Model, and Expert-Parallelism": [
+              "5.1 Data Parallelism",
+              "5.2 Model Parallelism",
+              "5.3 Model and Data Parallelism",
+              "5.4 Expert and Data Parallelism",
+              "5.5 Expert, Model and Data Parallelism",
+              "5.6 Towards Trillion Parameter Models"
+            ],
+            "6 Related Work": [],
+            "7 Discussion": [],
+            "8 Future Work": [],
+            "9 Conclusion": [],
+            "A Switch for Attention": [],
+            "B Preventing Token Dropping with No-Token-Left-Behind": [],
+            "C Encouraging Exploration Across Experts": [],
+            "D Switch Transformers in Lower Compute Regimes": [],
+            "E Relation of Upstream to Downstream Model Performance": [],
+            "F Pseudo Code for Switch Transformers": []
+          },
+          "summary_embedding": [
+            -0.0060447100549936295,
+            -0.00390331051312387,
+            -0.01784370467066765,
+            -0.0021820589900016785,
+            0.013227885589003563,
+            0.011338187381625175,
+            0.0165580902248621,
+            0.024240797385573387,
+            -0.005986624862998724,
+            0.03596621751785278,
+            -0.004852031823247671,
+            -0.0068811336532235146,
+            -0.003574162255972624,
+            0.014242436736822128,
+            0.011578272096812725,
+            -0.033302053809165955,
+            -0.024519605562090874,
+            0.01050176378339529,
+            0.016093410551548004,
+            -0.006795942317694426,
+            0.007473600562661886,
+            -0.030513975769281387,
+            -0.06124480068683624,
+            -0.006838538218289614,
+            -0.0026099516544491053,
+            0.01029265858232975,
+            0.0010329445358365774,
+            -0.010873507708311081,
+            -0.053469158709049225,
+            0.023311438038945198,
+            0.025294071063399315,
+            0.005092116538435221,
+            -0.029584616422653198,
+            0.0009361362317577004,
+            -0.05161044001579285,
+            -0.04448534920811653,
+            0.0379488542675972,
+            0.012561844661831856,
+            -0.04364892467856407,
+            0.039249956607818604,
+            -0.004875265993177891,
+            -0.0022440163884311914,
+            -0.021499186754226685,
+            -0.0034909071400761604,
+            -0.007039899472147226,
+            -0.009851212613284588,
+            -0.02874819189310074,
+            0.0014037203509360552,
+            0.021886419504880905,
+            -0.0056962003000080585,
+            0.023264970630407333,
+            -0.024271776899695396,
+            0.01936165988445282,
+            0.04240977764129639,
+            -0.06957805901765823,
+            0.024659009650349617,
+            0.017549408599734306,
+            0.05120771750807762,
+            0.06511713564395905,
+            0.018339363858103752,
+            0.021669568493962288,
+            -0.014141756109893322,
+            -0.017766257748007774,
+            -0.011872569099068642,
+            0.004673904739320278,
+            -0.014970434829592705,
+            -0.01891246810555458,
+            0.008449427783489227,
+            -0.020461400970816612,
+            0.008209343999624252,
+            -0.05576157942414284,
+            -0.014730350114405155,
+            -0.034200433641672134,
+            -0.01140014547854662,
+            0.017642343416810036,
+            -0.018215449526906013,
+            0.030126741155982018,
+            0.02997184917330742,
+            0.002638994250446558,
+            0.0007822110201232135,
+            0.021049996837973595,
+            0.006509389728307724,
+            -0.0053438181057572365,
+            0.012321759946644306,
+            0.028159597888588905,
+            -0.01450575515627861,
+            -0.010106786154210567,
+            -0.03454120084643364,
+            0.01861817203462124,
+            0.01589204929769039,
+            -0.04194509983062744,
+            -0.012569589540362358,
+            -0.013824224472045898,
+            0.032341714948415756,
+            -0.009549170732498169,
+            0.04098476096987724,
+            -0.010083552449941635,
+            0.001383390510454774,
+            -0.031149037182331085,
+            -0.009463978931307793,
+            0.017069239169359207,
+            0.006172496825456619,
+            -0.016589069738984108,
+            -0.0012846461031585932,
+            0.016170857474207878,
+            0.006617815233767033,
+            -0.030142230913043022,
+            0.04488807171583176,
+            0.023497309535741806,
+            0.055296897888183594,
+            0.05579255893826485,
+            -0.037391237914562225,
+            -0.002445377642288804,
+            0.0344172865152359,
+            -0.0228312686085701,
+            -0.015040136873722076,
+            -0.034014564007520676,
+            -0.01768881268799305,
+            0.035935238003730774,
+            -0.0318460576236248,
+            0.01556677371263504,
+            0.028066661208868027,
+            -0.025712283328175545,
+            0.003488970920443535,
+            -0.03943582624197006,
+            0.003622566582635045,
+            0.01220559049397707,
+            0.006095050368458033,
+            -0.0026293133851140738,
+            -0.013615119270980358,
+            0.008906363509595394,
+            0.003055269829928875,
+            0.00800798274576664,
+            -0.04615819454193115,
+            0.018478767946362495,
+            -0.00454224506393075,
+            0.04033420979976654,
+            0.042502716183662415,
+            -0.02730768360197544,
+            -0.016743963584303856,
+            0.008557853288948536,
+            0.012104909867048264,
+            0.04188314080238342,
+            -0.007287728600203991,
+            -0.00121784838847816,
+            -0.0007836631848476827,
+            0.007322579622268677,
+            0.047552235424518585,
+            0.004646798130124807,
+            0.0020697612781077623,
+            -0.027230238541960716,
+            0.014978179708123207,
+            -0.019733402878046036,
+            0.027648448944091797,
+            0.023373395204544067,
+            0.02942972257733345,
+            -0.04048910364508629,
+            0.017518429085612297,
+            -0.011818356812000275,
+            -0.005568413063883781,
+            -0.002726121572777629,
+            -0.02856232039630413,
+            -0.0074426219798624516,
+            0.033983584493398666,
+            0.015365412458777428,
+            0.012956822291016579,
+            0.03271345794200897,
+            0.0012052633101120591,
+            -0.042502716183662415,
+            0.03804178908467293,
+            0.024550583213567734,
+            0.04640602692961693,
+            0.023466331884264946,
+            0.010796061716973782,
+            0.006571347359567881,
+            0.013375034555792809,
+            -0.019377149641513824,
+            0.015907539054751396,
+            0.02560385875403881,
+            0.009750531986355782,
+            -0.03167567402124405,
+            -0.0009501734166406095,
+            0.01178737822920084,
+            -0.004546117503196001,
+            0.03701949119567871,
+            0.04104671627283096,
+            0.020569827407598495,
+            0.05030933395028114,
+            0.0020155487582087517,
+            0.017936641350388527,
+            0.005936284549534321,
+            0.02820606529712677,
+            0.030467506498098373,
+            -0.005285732913762331,
+            0.005076627247035503,
+            -0.01136916596442461,
+            -0.03385967016220093,
+            0.014110777527093887,
+            0.01561324205249548,
+            -0.008077683858573437,
+            -0.01697630248963833,
+            0.006033093202859163,
+            0.009169681929051876,
+            -0.01795213110744953,
+            -0.017456471920013428,
+            0.005971135571599007,
+            0.018866000697016716,
+            -0.013576395809650421,
+            0.024689987301826477,
+            -0.04442339017987251,
+            0.029119934886693954,
+            -0.033209118992090225,
+            -0.003384418087080121,
+            -0.010641167871654034,
+            -0.0030843124259263277,
+            0.0041666291654109955,
+            0.011314953677356243,
+            0.013406013138592243,
+            0.002042655134573579,
+            -0.013909416273236275,
+            0.024519605562090874,
+            0.018447788432240486,
+            0.012073930352926254,
+            -0.03148980438709259,
+            -0.014397329650819302,
+            -0.0015673263696953654,
+            -0.008743725717067719,
+            -0.034138478338718414,
+            0.026966920122504234,
+            0.014048820361495018,
+            -0.03844451159238815,
+            0.017797237262129784,
+            0.029538147151470184,
+            0.014420564286410809,
+            -0.004790074657648802,
+            -0.0463130883872509,
+            0.04045812413096428,
+            -0.030436528846621513,
+            -0.033333033323287964,
+            -0.010478530079126358,
+            -0.0003146269591525197,
+            0.02894955314695835,
+            -0.0064358157105743885,
+            -0.050835974514484406,
+            0.01184933539479971,
+            0.020631784573197365,
+            0.00010763872705865651,
+            0.005731051322072744,
+            -0.019593998789787292,
+            0.00028631053282879293,
+            -0.03395260497927666,
+            0.0090844901278615,
+            0.026750069111585617,
+            0.008867640048265457,
+            -0.016418687999248505,
+            0.03351890295743942,
+            -0.0004121613164898008,
+            -0.018695618957281113,
+            -0.003889757441356778,
+            0.006885006092488766,
+            0.034912943840026855,
+            0.004836542531847954,
+            -0.02946070022881031,
+            0.05665995925664902,
+            -0.02100352756679058,
+            0.021344292908906937,
+            0.05631919577717781,
+            -0.019439106807112694,
+            -0.008472662419080734,
+            -0.002065889071673155,
+            0.0026196325197815895,
+            -0.0260530486702919,
+            -0.008937342092394829,
+            0.032341714948415756,
+            0.041728246957063675,
+            -0.018742086365818977,
+            -0.001398879918269813,
+            0.006780453026294708,
+            0.009425255469977856,
+            0.004948840010911226,
+            0.02692045085132122,
+            -0.012128143571317196,
+            -0.014559967443346977,
+            0.05198218300938606,
+            0.028314489871263504,
+            -0.012081675231456757,
+            -0.0164031982421875,
+            0.019888296723365784,
+            -0.024844881147146225,
+            -0.020244551822543144,
+            -0.02431824430823326,
+            -0.020414933562278748,
+            -0.0042518205009400845,
+            0.009525936096906662,
+            0.037484172731637955,
+            -0.03968365862965584,
+            0.012747716158628464,
+            0.021700548008084297,
+            0.02981695532798767,
+            -0.019532041624188423,
+            -0.05910727381706238,
+            -0.0016128262504935265,
+            0.03296129032969475,
+            -0.015458349138498306,
+            -0.049379974603652954,
+            0.015164051204919815,
+            0.0260530486702919,
+            0.05957195535302162,
+            -0.001461805310100317,
+            -0.014954946003854275,
+            0.061926331371068954,
+            0.03026614524424076,
+            -0.004383479710668325,
+            -0.009494957514107227,
+            -0.0127864396199584,
+            0.003396035172045231,
+            0.00465454300865531,
+            -0.03364282101392746,
+            -0.041170634329319,
+            -0.011841590516269207,
+            -0.010284913703799248,
+            0.008108663372695446,
+            -0.040922801941633224,
+            0.03531566634774208,
+            -0.021917399019002914,
+            0.010950954630970955,
+            0.015473837964236736,
+            0.037391237914562225,
+            -0.021344292908906937,
+            -0.01284839678555727,
+            -0.01572941243648529,
+            -0.024968795478343964,
+            -0.014010096900165081,
+            0.021917399019002914,
+            0.007682706229388714,
+            3.4699722164077684e-05,
+            0.010075807571411133,
+            -0.024659009650349617,
+            -0.051052823662757874,
+            0.0291044469922781,
+            0.0015605497173964977,
+            -0.015938518568873405,
+            -0.004623564425855875,
+            -0.01977987214922905,
+            -0.01122201792895794,
+            -0.012608313001692295,
+            -0.039249956607818604,
+            0.021111954003572464,
+            -0.01135367713868618,
+            0.015876561403274536,
+            0.0009250032599084079,
+            0.003659353591501713,
+            0.028624277561903,
+            0.01116006076335907,
+            0.0051811798475682735,
+            -0.03255856782197952,
+            -0.018277406692504883,
+            0.010122275911271572,
+            -0.00561100896447897,
+            -0.018339363858103752,
+            -0.013351799920201302,
+            -0.05037129297852516,
+            -0.03305422514677048,
+            -0.008054450154304504,
+            -0.05415068939328194,
+            0.0020697612781077623,
+            -0.014296649023890495,
+            -0.02232012152671814,
+            -0.005990497302263975,
+            0.03107159025967121,
+            -0.003235333366319537,
+            0.025294071063399315,
+            0.0076594725251197815,
+            0.037360258400440216,
+            0.040117356926202774,
+            0.008426194079220295,
+            0.03065337985754013,
+            -0.0029991210903972387,
+            -0.016867877915501595,
+            0.05486319959163666,
+            -0.01881953328847885,
+            0.013529927469789982,
+            -0.008604321628808975,
+            0.008604321628808975,
+            0.03299226611852646,
+            -0.003794885240495205,
+            0.02007416822016239,
+            -0.0311955064535141,
+            -0.07998688519001007,
+            0.01637221872806549,
+            0.028159597888588905,
+            -0.06418777257204056,
+            0.014691627584397793,
+            -0.020275529474020004,
+            0.02052335813641548,
+            0.02411688305437565,
+            -0.0009114501299336553,
+            -0.014064309187233448,
+            0.005595519673079252,
+            0.0020833145827054977,
+            -0.01456771232187748,
+            -0.002691270550712943,
+            -0.009417511522769928,
+            0.008774704299867153,
+            -0.017007282003760338,
+            -0.013406013138592243,
+            -0.034231413155794144,
+            0.010873507708311081,
+            0.01233724970370531,
+            -0.003969140350818634,
+            0.0011500825639814138,
+            -0.03277541697025299,
+            -0.007876322604715824,
+            -0.02145271934568882,
+            0.03382869064807892,
+            0.012058441527187824,
+            -0.030699847266077995,
+            -0.007814365439116955,
+            0.015264731831848621,
+            -0.007876322604715824,
+            0.0008906363509595394,
+            0.024132372811436653,
+            0.02103450708091259,
+            0.0028364830650389194,
+            -0.0009700191440060735,
+            0.02438020147383213,
+            0.007427132688462734,
+            -0.01720864325761795,
+            0.028794659301638603,
+            0.014288905076682568,
+            0.060160547494888306,
+            -0.03271345794200897,
+            0.001297231181524694,
+            -0.04135650396347046,
+            -0.031118059530854225,
+            0.006408709101378918,
+            0.014350862242281437,
+            0.007597515359520912,
+            -0.028252532705664635,
+            0.005978880450129509,
+            0.02004319056868553,
+            0.01469937153160572,
+            0.009704063646495342,
+            0.0037580979987978935,
+            -0.036740683019161224,
+            -0.000691695255227387,
+            -0.011214273050427437,
+            -0.0038781403563916683,
+            0.035656433552503586,
+            -0.008186109364032745,
+            -0.05151750147342682,
+            -0.0006418389966711402,
+            -0.03255856782197952,
+            -0.0353776253759861,
+            0.021313315257430077,
+            -0.00839521549642086,
+            -0.0039207362569868565,
+            -0.013986863195896149,
+            -0.000537770101800561,
+            -0.010524998418986797,
+            0.027509044855833054,
+            0.028345469385385513,
+            0.022490503266453743,
+            -0.05384090170264244,
+            0.009479468688368797,
+            0.00023766436788719147,
+            -0.05817791447043419,
+            0.0009709871956147254,
+            -0.007117345929145813,
+            0.011191039346158504,
+            -0.020213572308421135,
+            -0.02414786070585251,
+            0.011500826105475426,
+            0.03392162546515465,
+            0.014304393902420998,
+            0.0321558453142643,
+            -0.013367289677262306,
+            -0.09250226616859436,
+            0.018292896449565887,
+            -0.008689512498676777,
+            0.04126356914639473,
+            0.00865078903734684,
+            -0.004278926644474268,
+            -0.015845581889152527,
+            -0.00865078903734684,
+            -0.006943091284483671,
+            0.01283290795981884,
+            0.004530628211796284,
+            -0.005626498255878687,
+            0.0016631665639579296,
+            0.013708055019378662,
+            -0.005967263597995043,
+            0.007481344975531101,
+            0.003651608945801854,
+            -0.013189162127673626,
+            -0.06059424951672554,
+            -0.04055105894804001,
+            -0.018540725111961365,
+            0.050835974514484406,
+            -0.0069159846752882,
+            -0.046684831380844116,
+            -0.010850274004042149,
+            0.05058814212679863,
+            0.012259802781045437,
+            -0.0334569476544857,
+            -0.0007841471815481782,
+            0.0050882440991699696,
+            0.015458349138498306,
+            0.004546117503196001,
+            -0.027292195707559586,
+            0.02820606529712677,
+            0.0008412640891037881,
+            0.02300165221095085,
+            -0.018339363858103752,
+            -0.005312839522957802,
+            0.035780347883701324,
+            -0.024844881147146225,
+            0.023791607469320297,
+            0.0055219451896846294,
+            0.031629208475351334,
+            0.002952652983367443,
+            0.010060317814350128,
+            0.004704883322119713,
+            0.0186491496860981,
+            0.004275054205209017,
+            0.0276639387011528,
+            0.011462102644145489,
+            0.006428070832043886,
+            0.021623101085424423,
+            -0.009378788061439991,
+            -0.014513500034809113,
+            0.01885051093995571,
+            -0.0014695499558001757,
+            0.025232113897800446,
+            -0.020538847893476486,
+            0.002726121572777629,
+            -0.0016883367206901312,
+            -0.017967619001865387,
+            -0.040922801941633224,
+            -0.0007405834621749818,
+            0.021747015416622162,
+            -0.011671207845211029,
+            -0.0228312686085701,
+            -0.013715799897909164,
+            -0.01585332676768303,
+            0.0038490979932248592,
+            0.034169457852840424,
+            -0.0398695282638073,
+            0.00963436160236597,
+            -0.000830615172162652,
+            -0.03125746175646782,
+            -0.0239619892090559,
+            -0.01093546487390995,
+            0.014614180661737919,
+            -0.01652711257338524,
+            -0.021173911169171333,
+            -0.028469383716583252,
+            -0.008410705253481865,
+            0.006048582494258881,
+            -0.05065010115504265,
+            -0.004054331686347723,
+            0.019965743646025658,
+            0.0276639387011528,
+            -0.003099801717326045,
+            -0.02177799493074417,
+            0.0012468908680602908,
+            0.022351099178195,
+            0.02936776541173458,
+            0.014226946979761124,
+            0.027586491778492928,
+            -0.002178186783567071,
+            -0.0011675080750137568,
+            -0.0073148347437381744,
+            0.01307299267500639,
+            0.0011287847300991416,
+            0.036616768687963486,
+            0.0122210793197155,
+            -0.020306508988142014,
+            0.0013959755888208747,
+            -0.006431943271309137,
+            0.029181893914937973,
+            -0.02270735427737236,
+            -9.832092473516241e-05,
+            0.03531566634774208,
+            0.027741385623812675,
+            0.004933350719511509,
+            -0.0143121387809515,
+            -0.05142456665635109,
+            0.021592123433947563,
+            -0.01765783317387104,
+            0.031025122851133347,
+            0.016666516661643982,
+            0.003397971158847213,
+            -0.0034212053287774324,
+            0.012368228286504745,
+            -0.022087780758738518,
+            -0.004406713880598545,
+            -0.021142931655049324,
+            0.04389675334095955,
+            -0.043184246867895126,
+            0.008472662419080734,
+            0.029445212334394455,
+            0.011926782317459583,
+            -0.019082851707935333,
+            -0.005425137002021074,
+            0.029212871566414833,
+            -0.00162637943867594,
+            0.01308848150074482,
+            0.0058665829710662365,
+            -0.026935940608382225,
+            0.01202746294438839,
+            -0.031660184264183044,
+            0.02177799493074417,
+            -0.013436991721391678,
+            0.018447788432240486,
+            0.03342596814036369,
+            -0.0030397805385291576,
+            -0.0038239278364926577,
+            -0.016263794153928757,
+            -0.00969631876796484,
+            0.0662323608994484,
+            0.01054823212325573,
+            0.03726732358336449,
+            0.0018974426202476025,
+            -0.009773765690624714,
+            0.007969259284436703,
+            0.0063041565008461475,
+            -0.012329504825174809,
+            0.010494019836187363,
+            0.024178840219974518,
+            0.0012730291346088052,
+            -0.002412462839856744,
+            -0.019408127292990685,
+            -0.03438630700111389,
+            0.003727119415998459,
+            -0.030932186171412468,
+            0.022691864520311356,
+            0.009208405390381813,
+            0.017456471920013428,
+            0.008240322582423687,
+            -0.015907539054751396,
+            0.03200095146894455,
+            0.05461536720395088,
+            0.0019768255297094584,
+            0.013375034555792809,
+            0.007938280701637268,
+            0.007036027032881975,
+            -0.011338187381625175,
+            0.04702559858560562,
+            0.000662652775645256,
+            0.005169562995433807,
+            0.022536970674991608,
+            0.02145271934568882,
+            0.022939693182706833,
+            0.002304037567228079,
+            -0.011694442480802536,
+            0.010478530079126358,
+            -0.0006302220281213522,
+            0.004445436876267195,
+            -0.010695381090044975,
+            -0.027865299955010414,
+            0.007492962293326855,
+            0.01302652433514595,
+            0.022598929703235626,
+            0.021592123433947563,
+            -0.03178409859538078,
+            -0.01855621486902237,
+            0.03364282101392746,
+            -0.02997184917330742,
+            0.03612111136317253,
+            -0.0014424435794353485,
+            -0.023807097226381302,
+            -0.0058356039226055145,
+            0.016929835081100464,
+            0.02656419761478901,
+            -0.029181893914937973,
+            0.01804506592452526,
+            0.029693040996789932,
+            -0.0074426219798624516,
+            0.0005939188995398581,
+            0.012112653814256191,
+            0.02447313815355301,
+            -0.009293596260249615,
+            -0.0024027819745242596,
+            -0.0013320822035893798,
+            -0.011926782317459583,
+            -0.018370341509580612,
+            -0.0044725434854626656,
+            -0.022629907354712486,
+            0.005564541090279818,
+            -0.007365175057202578,
+            -0.003481226274743676,
+            -0.008581086993217468,
+            0.01154729351401329,
+            0.001979729626327753,
+            -0.019857317209243774,
+            -0.022335609421133995,
+            0.01652711257338524,
+            0.03887821361422539,
+            -0.018215449526906013,
+            -0.02106548473238945,
+            0.024442158639431,
+            -0.01759587600827217,
+            -0.01601596549153328,
+            0.036678727716207504,
+            0.013739033602178097,
+            0.007736918982118368,
+            -0.03358086198568344,
+            -0.030049296095967293,
+            -0.03274443745613098,
+            0.006296411622315645,
+            0.03785591572523117,
+            -0.0028752065263688564,
+            0.02569679357111454,
+            0.03701949119567871,
+            0.0057891360484063625,
+            -0.028934063389897346,
+            -0.014823286794126034,
+            -0.045104920864105225,
+            0.060160547494888306,
+            0.0011481463443487883,
+            0.05161044001579285,
+            0.016294771805405617,
+            -0.01456771232187748,
+            0.00419760774821043,
+            -0.013251119293272495,
+            0.026192452758550644,
+            0.02280029095709324,
+            0.025929134339094162,
+            0.006582964211702347,
+            0.022505993023514748,
+            -0.0037522895727306604,
+            -0.012197845615446568,
+            0.007307090330868959,
+            0.014536733739078045,
+            0.0002497654058970511,
+            -0.011392400600016117,
+            0.015783624723553658,
+            -0.01307299267500639,
+            -0.011361422017216682,
+            0.023295948281884193,
+            0.01368482131510973,
+            -0.012344993650913239,
+            0.014064309187233448,
+            -0.008565598167479038,
+            0.01239920686930418,
+            -0.023915521800518036,
+            0.022691864520311356,
+            -0.03262052312493324,
+            0.009587894193828106,
+            0.0015053689712658525,
+            -0.0073148347437381744,
+            0.024550583213567734,
+            -0.0027454833034425974,
+            -0.05071205645799637,
+            0.03550153970718384,
+            -0.008015726692974567,
+            0.0037213109899312258,
+            0.006381602957844734,
+            0.011392400600016117,
+            -0.0012623801594600081,
+            0.03807276859879494,
+            0.01239920686930418,
+            0.015969496220350266,
+            0.02199484594166279,
+            0.011911292560398579,
+            -0.008433938957750797,
+            -0.004096927121281624,
+            -0.0016137943603098392,
+            -0.01784370467066765,
+            0.025944624096155167,
+            0.0018674320308491588,
+            0.03962169960141182,
+            -0.018168980255723,
+            -0.02300165221095085,
+            0.028066661208868027,
+            0.007597515359520912,
+            -0.02351279929280281,
+            -0.0017377089243382215,
+            0.01202746294438839,
+            -0.009185170754790306,
+            0.027385130524635315,
+            0.01196550577878952,
+            0.019113829359412193,
+            -0.018060555681586266,
+            -0.004340883810073137,
+            -0.005072754807770252,
+            -0.015938518568873405,
+            0.02238207869231701,
+            0.020771188661456108,
+            0.011152315884828568,
+            -0.011880313977599144,
+            -0.01307299267500639,
+            0.020600805059075356,
+            0.035036858171224594,
+            -0.0002744515077210963,
+            0.005672966130077839,
+            0.017487451434135437,
+            0.002373739378526807,
+            -0.04470219835639,
+            0.016232814639806747,
+            -0.014025585725903511,
+            0.011214273050427437,
+            0.019593998789787292,
+            -0.021917399019002914,
+            -0.03196997195482254,
+            -0.011539548635482788,
+            -0.0414804182946682,
+            0.00493722315877676,
+            -0.0069159846752882,
+            0.013576395809650421,
+            -0.010176488198339939,
+            -0.0014831031439825892,
+            0.016666516661643982,
+            0.0038103745318949223,
+            0.02177799493074417,
+            -0.011678952723741531,
+            -0.012073930352926254,
+            -0.020213572308421135,
+            -0.020786678418517113,
+            0.004278926644474268,
+            -0.030405549332499504,
+            -0.004488032776862383,
+            0.021375272423028946,
+            -0.04783104360103607,
+            -0.03689557686448097,
+            0.02898053266108036,
+            -0.008503641001880169,
+            0.020755698904395103,
+            0.022955182939767838,
+            -0.015094350092113018,
+            -0.0008717587334103882,
+            -0.018695618957281113,
+            0.02258343994617462,
+            -0.007837600074708462,
+            0.026300877332687378,
+            -0.0027551641687750816,
+            0.0006292539183050394,
+            -0.02659517526626587,
+            0.020306508988142014,
+            0.005227648187428713,
+            0.012275291606783867,
+            -0.02390003204345703,
+            0.006815304048359394,
+            -0.03133491054177284,
+            -0.01784370467066765,
+            0.0180140882730484,
+            -0.011152315884828568,
+            0.027989214286208153,
+            -0.006060199346393347,
+            0.028438406065106392,
+            0.022722844034433365,
+            0.024751944467425346,
+            0.02676555886864662,
+            -0.02363671362400055,
+            0.008495896123349667,
+            0.03726732358336449,
+            0.00865078903734684,
+            -0.015876561403274536,
+            0.008170620538294315,
+            -0.04863648861646652,
+            -0.008062195032835007,
+            0.05030933395028114,
+            0.002127846470102668,
+            0.023141054436564445,
+            0.016294771805405617,
+            -0.002873270306736231,
+            -0.005688455421477556,
+            0.010269423946738243,
+            -0.04296739399433136,
+            -0.023172033950686455,
+            0.022629907354712486,
+            -0.021561143919825554,
+            6.54212328754511e-07,
+            0.03122648410499096,
+            0.012120398692786694,
+            -0.0009109660750254989,
+            -0.05108380317687988,
+            0.02100352756679058,
+            0.01178737822920084,
+            0.014443797990679741,
+            -0.010300402529537678,
+            0.026068538427352905,
+            0.01733255758881569,
+            0.038413532078266144,
+            0.012755461037158966,
+            -0.002679653698578477,
+            -0.03782493621110916,
+            0.009177426807582378,
+            -0.021685058251023293,
+            0.0011471783509477973,
+            0.0074503663927316666,
+            -0.02273833192884922,
+            -0.024782923981547356,
+            0.0043912241235375404,
+            0.004588713403791189,
+            0.011880313977599144,
+            0.022087780758738518,
+            0.009417511522769928,
+            -0.021204888820648193,
+            0.015404135920107365,
+            -0.05446047708392143,
+            -0.015450604259967804,
+            -0.023884542286396027,
+            0.02566581591963768,
+            0.020012211054563522,
+            0.010184233076870441,
+            0.005545179359614849,
+            0.010904486291110516,
+            -0.020988039672374725,
+            -0.009239383973181248,
+            0.020631784573197365,
+            0.011074868962168694,
+            -0.012840652838349342,
+            0.04113965481519699,
+            -0.00011126903700642288,
+            0.031210994347929955,
+            -0.02280029095709324,
+            0.002941036131232977,
+            -0.001241082325577736,
+            -0.03649285435676575,
+            0.009130958467721939,
+            0.01981084980070591,
+            0.02151467651128769,
+            -0.01861817203462124,
+            0.0024143988266587257,
+            0.01048627495765686,
+            0.004766840487718582,
+            0.038351576775312424,
+            0.027121812105178833,
+            -0.044206541031599045,
+            0.011663463898003101,
+            0.012995545752346516,
+            -0.009363298304378986,
+            0.048419639468193054,
+            -0.017518429085612297,
+            0.017673322930932045,
+            0.008240322582423687,
+            -0.01798310875892639,
+            0.02318752370774746,
+            -0.01469937153160572,
+            -0.013282098807394505,
+            0.00650551775470376,
+            0.009192915633320808,
+            -0.0001770381786627695,
+            -0.037577107548713684,
+            -0.017859194427728653,
+            -0.005719434004276991,
+            0.0038297362625598907,
+            0.01562873087823391,
+            0.013498948886990547,
+            0.009967382065951824,
+            0.005161818582564592,
+            -0.00945623405277729,
+            -0.0016089539276435971,
+            0.008124152198433876,
+            -0.0005532593932002783,
+            0.0019216446671634912,
+            -0.01926872320473194,
+            0.02698240801692009,
+            0.02572777308523655,
+            -0.0018354853382334113,
+            -0.04652994126081467,
+            -0.0036709706764668226,
+            0.0037077576853334904,
+            -0.017735280096530914,
+            -0.0035315665882080793,
+            -0.04519785940647125,
+            -0.01810702309012413,
+            0.0143121387809515,
+            -0.014614180661737919,
+            -0.0460033044219017,
+            0.009076746180653572,
+            -0.013886182568967342,
+            0.0063970922492444515,
+            -0.00932457484304905,
+            0.005762029904872179,
+            0.003651608945801854,
+            0.005680711008608341,
+            -0.005909178406000137,
+            0.05108380317687988,
+            -0.03466511517763138,
+            0.028051171451807022,
+            0.008565598167479038,
+            -0.012437930330634117,
+            -0.02261441759765148,
+            0.00887538492679596,
+            -0.044981006532907486,
+            -0.033209118992090225,
+            0.016743963584303856,
+            -0.01781272701919079,
+            0.011074868962168694,
+            0.06161654368042946,
+            0.007330324035137892,
+            -0.023342415690422058,
+            -0.01233724970370531,
+            -0.021049996837973595,
+            -0.04191412031650543,
+            0.003442503046244383,
+            0.005463860463351011,
+            -0.004948840010911226,
+            -0.03807276859879494,
+            -0.021716037765145302,
+            0.04615819454193115,
+            -0.012004229240119457,
+            0.012422440573573112,
+            0.01080380566418171,
+            -0.0019410063978284597,
+            -0.04860550910234451,
+            -0.021019017323851585,
+            0.0395597442984581,
+            0.021886419504880905,
+            -0.008426194079220295,
+            0.0028364830650389194,
+            -0.018277406692504883,
+            -0.003289545886218548,
+            0.006296411622315645,
+            -0.009107724763453007,
+            -0.03562545403838158,
+            0.02067825198173523,
+            0.0054832217283546925,
+            -0.023373395204544067,
+            -0.025805220007896423,
+            -0.03296129032969475,
+            0.009944148361682892,
+            0.02453509531915188,
+            -0.01968693546950817,
+            0.005823987070471048,
+            0.00027203132049180567,
+            -0.000881923595443368,
+            0.00042740863864310086,
+            -0.009959637187421322,
+            -0.019764382392168045,
+            -0.020151615142822266,
+            0.03268248215317726,
+            -0.0060059865936636925,
+            0.0074387495405972,
+            -0.03630698472261429,
+            -0.010974188335239887,
+            0.011322698555886745,
+            -0.002395037328824401,
+            -0.013452480547130108,
+            -0.009092235006392002,
+            -0.01739451475441456,
+            -0.009882191196084023,
+            -0.007876322604715824,
+            0.022660886868834496,
+            0.006168624851852655,
+            0.005409647710621357,
+            0.0023795480374246836,
+            -0.008581086993217468,
+            0.02898053266108036,
+            0.017874684184789658,
+            -0.014188224449753761,
+            -0.00920066051185131,
+            0.03451022133231163,
+            0.02665713243186474,
+            -0.009115469641983509,
+            0.01798310875892639,
+            0.023264970630407333,
+            -0.0010484338272362947,
+            -0.01196550577878952,
+            0.001922612776979804,
+            -0.0009550138493068516,
+            -0.0032972905319184065,
+            0.00845717266201973,
+            -0.015365412458777428,
+            0.0315207801759243,
+            0.04287445917725563,
+            -0.002145271748304367,
+            0.009587894193828106,
+            -0.003727119415998459,
+            -0.008472662419080734,
+            0.017069239169359207,
+            0.009316830895841122,
+            0.04625113308429718,
+            0.01098193321377039,
+            0.005680711008608341,
+            -0.033983584493398666,
+            -0.001229465357027948,
+            -0.01184933539479971,
+            0.03308520466089249,
+            0.02061629481613636,
+            0.00889087375253439,
+            -0.008379725739359856,
+            0.01344473659992218,
+            0.008341003209352493,
+            0.00661007035523653,
+            0.020569827407598495,
+            -0.016542602330446243,
+            -0.017286090180277824,
+            0.030467506498098373,
+            -0.00672236829996109,
+            0.01289486512541771,
+            0.02029101923108101,
+            -0.002089123008772731,
+            0.010842529125511646,
+            0.07100307941436768,
+            -0.006184114143252373,
+            0.015713922679424286,
+            -0.019082851707935333,
+            0.0015789433382451534,
+            -0.007640110794454813,
+            -0.03370477631688118,
+            0.018246427178382874,
+            0.0039962464943528175,
+            0.00865078903734684,
+            0.01198099460452795,
+            0.009053511545062065,
+            -0.007946024648845196,
+            -0.00963436160236597,
+            0.015195030719041824,
+            0.0019080914789810777,
+            0.00033568276558071375,
+            0.001336922636255622,
+            -0.0065210070461034775,
+            -0.004011735785752535,
+            -0.021948376670479774,
+            0.02763296104967594,
+            0.007140580099076033,
+            0.030606910586357117,
+            -0.055296897888183594,
+            0.030095763504505157,
+            -0.00789568480104208,
+            0.02554190158843994,
+            -0.016201836988329887,
+            -0.011090358719229698,
+            0.001609921921044588,
+            0.018153492361307144,
+            0.004491904750466347,
+            0.006462921854108572,
+            0.011888058856129646,
+            0.023233991116285324,
+            -0.03444826602935791,
+            0.017053749412298203,
+            0.023110076785087585,
+            0.0048675211146473885,
+            0.0003535923024173826,
+            0.005165690556168556,
+            0.015458349138498306,
+            0.007198664825409651,
+            -0.023048119619488716,
+            -0.0025692922063171864,
+            0.017611365765333176,
+            -0.02306360937654972,
+            0.0036980770528316498,
+            -0.006923729553818703,
+            0.004623564425855875,
+            -0.0014705180656164885,
+            -0.03509881719946861,
+            -0.00100099784322083,
+            -0.025743262842297554,
+            0.01923774555325508,
+            0.009556914679706097,
+            -0.025867177173495293,
+            0.003932353109121323,
+            0.025650326162576675,
+            -0.0003695656778290868,
+            0.00821708794683218,
+            0.0004683101433329284,
+            0.00869725737720728,
+            -0.00974278710782528,
+            -0.01493945624679327,
+            0.004801691509783268,
+            0.01531120017170906,
+            -0.01493945624679327,
+            -0.007431004662066698,
+            -0.009510447271168232,
+            0.030560443177819252,
+            0.0006674932083114982,
+            0.004189862869679928,
+            -0.005045648664236069,
+            -0.01858719252049923,
+            -0.0032237162813544273,
+            -0.021142931655049324,
+            0.004367990419268608,
+            -0.0009046735358424485,
+            0.04197607934474945,
+            0.0038161829579621553,
+            -0.02029101923108101,
+            0.0032972905319184065,
+            -0.03136589005589485,
+            0.0005000148667022586,
+            -0.007276111748069525,
+            -0.02004319056868553,
+            0.0015005286550149322,
+            0.0016525175888091326,
+            0.012631546705961227,
+            -0.03255856782197952,
+            -0.009301341138780117,
+            0.018416810780763626,
+            -0.00020680672605521977,
+            -0.010687636211514473,
+            -0.017828216776251793,
+            0.00035189816844649613,
+            -0.011338187381625175,
+            -0.027973726391792297,
+            -0.02850036323070526,
+            -0.0026719090528786182,
+            0.013491204008460045,
+            -0.018277406692504883,
+            -0.014993669465184212,
+            0.0031249718740582466,
+            0.021592123433947563,
+            -0.009781510569155216,
+            0.008302279748022556,
+            0.02241305634379387,
+            -0.008255811408162117,
+            -0.005514200776815414,
+            -0.000941460661124438,
+            -0.009130958467721939,
+            0.004267309792339802,
+            0.006300284061580896,
+            0.043339136987924576,
+            -0.032403673976659775,
+            -0.0018238683696836233,
+            0.014815541915595531,
+            0.0034153966698795557,
+            0.0025905899237841368,
+            0.002503462601453066,
+            -0.0007841471815481782,
+            0.01344473659992218,
+            0.01178737822920084,
+            0.015466093085706234,
+            0.00100099784322083,
+            -0.0033011629711836576,
+            0.006408709101378918,
+            -0.021173911169171333,
+            0.015721667557954788,
+            0.027958236634731293,
+            -0.0033999073784798384,
+            0.005525817628949881,
+            -0.013731288723647594,
+            -0.008759214542806149,
+            0.013134949840605259,
+            0.0026641644071787596,
+            -0.005068882368505001,
+            -0.009394276887178421,
+            0.017967619001865387,
+            0.019082851707935333,
+            0.006668155547231436,
+            -0.037453193217515945,
+            -0.02898053266108036,
+            -0.009959637187421322,
+            0.020569827407598495,
+            0.014916222542524338,
+            0.0010339126456528902,
+            -0.010540487244725227,
+            -0.009595638141036034,
+            -0.011903548613190651,
+            0.010989678092300892,
+            0.02721474878489971,
+            -0.032372694462537766,
+            0.0008214184199459851,
+            -0.029383255168795586,
+            0.03068435750901699,
+            0.0020000594668090343,
+            0.011446612887084484,
+            -0.0008185141487047076,
+            -0.015140817500650883,
+            -0.029181893914937973,
+            0.013824224472045898,
+            -0.004395096562802792,
+            0.001024231780320406,
+            -0.0031753121875226498,
+            0.0201051477342844,
+            0.012887120246887207,
+            -0.0014472840121015906,
+            0.01968693546950817,
+            0.008178364485502243,
+            0.016480645164847374,
+            -0.010153254494071007,
+            -0.04135650396347046,
+            0.019965743646025658,
+            0.007562664337456226,
+            0.014219203032553196,
+            0.004739734344184399,
+            0.004588713403791189,
+            -0.020771188661456108,
+            0.002724185585975647,
+            0.004333139397203922,
+            0.014405074529349804,
+            0.011028401553630829,
+            0.0025615475606173277,
+            0.011678952723741531,
+            -0.02228914201259613,
+            0.01929970271885395,
+            -0.01631026156246662,
+            0.01006806269288063,
+            0.004445436876267195,
+            -0.026873983442783356,
+            0.007547175046056509,
+            0.007822110317647457,
+            -0.0019187404541298747,
+            -0.017549408599734306,
+            0.02160761132836342,
+            -0.0034831624943763018,
+            -0.03509881719946861,
+            0.0360281765460968,
+            -0.0043447562493383884,
+            0.019346170127391815,
+            0.03209388628602028,
+            -0.01771979033946991,
+            -0.01993476413190365,
+            0.00906900130212307,
+            0.010408828034996986,
+            -0.002873270306736231,
+            0.017859194427728653,
+            -0.002919738180935383,
+            -0.02232012152671814,
+            0.017766257748007774,
+            -0.028918573632836342,
+            -0.014234691858291626,
+            0.020337486639618874,
+            -0.03624502569437027,
+            -0.0021026763133704662,
+            0.03175312280654907,
+            -0.024070415645837784,
+            0.005734923295676708,
+            0.005196669604629278,
+            -0.0019351978553459048,
+            0.0012236569309607148,
+            -0.006451305001974106,
+            0.03801080957055092,
+            -0.010865762829780579,
+            0.0201051477342844,
+            0.01929970271885395,
+            -0.023388884961605072,
+            0.01977987214922905,
+            -0.001109423115849495,
+            0.01675945147871971,
+            -0.003940097521990538,
+            0.03466511517763138,
+            -0.006017603911459446,
+            -0.011330443434417248,
+            0.006431943271309137,
+            0.008488151244819164,
+            -0.009959637187421322,
+            0.008681767620146275,
+            -0.018122512847185135,
+            0.027152791619300842,
+            0.02091059274971485,
+            -0.0002691270783543587,
+            0.013800990767776966,
+            0.0004421719058882445,
+            -0.020833145827054977,
+            0.00789568480104208,
+            -0.027121812105178833,
+            -0.009967382065951824,
+            0.01646515540778637,
+            -0.005762029904872179,
+            0.007380664348602295,
+            -0.02064727433025837,
+            0.028004704043269157,
+            0.0077524082735180855,
+            -0.013591884635388851,
+            -0.0008824076503515244,
+            0.023760627955198288,
+            -0.005432881414890289,
+            0.009355553425848484,
+            0.02070923149585724,
+            0.015713922679424286,
+            0.013615119270980358,
+            0.016263794153928757,
+            0.03351890295743942,
+            0.005328328814357519,
+            0.005568413063883781,
+            0.003519949736073613,
+            0.016991792246699333,
+            -0.008557853288948536,
+            0.018029578030109406,
+            0.0008093173382803798,
+            -0.013576395809650421,
+            0.021499186754226685,
+            -0.0018684001406654716,
+            -0.013622863218188286,
+            -0.0040891822427511215,
+            -0.004337011836469173,
+            0.03271345794200897,
+            0.022041313350200653,
+            -0.004468671046197414,
+            -0.024442158639431,
+            -0.00394784240052104,
+            0.016682006418704987,
+            -0.003167567541822791,
+            0.011136826127767563,
+            0.0007952801533974707,
+            -0.01586107164621353,
+            -0.009595638141036034,
+            0.0063583687879145145,
+            0.010586955584585667,
+            -0.008921852335333824,
+            0.010346870869398117,
+            -0.0228312686085701,
+            -0.004011735785752535,
+            0.00046613195445388556,
+            -0.004844287410378456,
+            0.0053438181057572365,
+            -0.011314953677356243,
+            0.004948840010911226,
+            0.016062432900071144,
+            0.01026167906820774,
+            0.004832670092582703,
+            0.006989559158682823,
+            0.012066186405718327,
+            0.009161937050521374,
+            -0.01627928391098976,
+            0.00428667152300477,
+            -0.005072754807770252,
+            -0.02345084212720394,
+            0.005831731949001551,
+            0.00902253296226263,
+            -0.01122201792895794,
+            0.009386532939970493,
+            -0.010958699509501457,
+            0.03296129032969475,
+            0.01030814740806818,
+            -0.037391237914562225,
+            -0.0033379499800503254,
+            -0.0016244432190433145,
+            0.014885243959724903,
+            0.012298526242375374,
+            0.008550108410418034,
+            -0.017518429085612297,
+            -0.0047823297791182995,
+            -0.009835722856223583,
+            0.0012827098835259676,
+            -0.012306271120905876,
+            0.017456471920013428,
+            0.022567950189113617,
+            -0.036337960511446,
+            0.01968693546950817,
+            -0.010432061739265919,
+            -0.023652203381061554,
+            0.020476890727877617,
+            0.011500826105475426,
+            0.01942361705005169,
+            -0.009402021765708923,
+            0.018896980211138725,
+            -0.0001331114035565406,
+            0.025510922074317932,
+            -0.004228586331009865,
+            -0.022087780758738518,
+            -0.012499887496232986,
+            -0.03209388628602028,
+            -0.0051811798475682735,
+            0.006819176487624645,
+            -0.00045524101005867124,
+            0.013111716136336327,
+            0.014226946979761124,
+            -0.025712283328175545,
+            0.007295473478734493,
+            0.032589543610811234,
+            0.014637414366006851,
+            0.023311438038945198,
+            -0.011276230216026306,
+            -0.0010794125264510512,
+            -0.0026060794480144978,
+            -0.013770012184977531,
+            -0.03212486580014229,
+            -0.0017493258928880095,
+            -0.036585792899131775,
+            0.005680711008608341,
+            -0.016387708485126495,
+            -0.013096226379275322,
+            0.023264970630407333,
+            -0.034138478338718414,
+            0.0014608372002840042,
+            0.013708055019378662,
+            0.010540487244725227,
+            0.012391461990773678,
+            -0.00162637943867594,
+            0.015032391995191574,
+            -0.029243851080536842,
+            -0.010284913703799248,
+            -0.004491904750466347,
+            -0.01981084980070591,
+            -0.007338068913668394,
+            -0.018060555681586266,
+            -0.013127204962074757,
+            -0.003566417610272765,
+            -0.010060317814350128,
+            0.002247888594865799,
+            0.0032062907703220844,
+            -0.014714861288666725,
+            0.03206290677189827,
+            0.003157886676490307,
+            0.018958937376737595,
+            0.0033418224193155766,
+            0.004449309315532446,
+            0.018788553774356842,
+            -0.0026331855915486813,
+            -0.0058278595097362995,
+            0.030963165685534477,
+            -0.00032672801171429455,
+            0.008759214542806149,
+            0.019082851707935333,
+            0.004724245052784681,
+            0.0038529701996594667,
+            -0.0260530486702919,
+            0.012887120246887207,
+            -0.027447087690234184,
+            0.005080499220639467,
+            -0.0077175572514534,
+            -0.024302754551172256,
+            -0.028128618374466896,
+            -0.006571347359567881,
+            0.012058441527187824,
+            -0.02898053266108036,
+            -0.00613377382978797,
+            0.002315654419362545,
+            -0.00400399137288332,
+            -0.002673845272511244,
+            0.007241260726004839,
+            0.029150914400815964,
+            -0.033209118992090225,
+            0.008751469664275646,
+            0.005595519673079252,
+            -0.0005053392960689962,
+            0.013297587633132935,
+            0.008038961328566074,
+            0.003343758638948202,
+            0.020848635584115982,
+            0.006637176964432001,
+            0.002458930714055896,
+            -0.0025576751213520765,
+            -0.014242436736822128,
+            0.004000118933618069,
+            0.02154565416276455,
+            0.007562664337456226,
+            0.018385831266641617,
+            -0.008914108388125896,
+            0.002621568739414215,
+            -0.01455222349613905,
+            0.005552923772484064,
+            -0.002867461647838354,
+            0.004309905227273703,
+            -0.012236569076776505,
+            0.008093173615634441,
+            0.0028771425131708384,
+            -0.0043447562493383884,
+            0.018060555681586266,
+            -0.002319526858627796,
+            0.016480645164847374,
+            0.004015608225017786,
+            0.015620986931025982,
+            -0.0035412474535405636,
+            0.01178737822920084,
+            0.0005731051205657423,
+            -0.017905661836266518,
+            -0.017022771760821342,
+            -0.039249956607818604,
+            0.007702067960053682,
+            0.03531566634774208,
+            -0.0009564659558236599,
+            -0.03444826602935791,
+            0.010145509615540504,
+            -0.011942271143198013,
+            -0.009975126944482327,
+            -0.022862248122692108,
+            0.013862947933375835,
+            0.012430185452103615,
+            0.007415515370666981,
+            0.012252057902514935,
+            -0.006997303571552038,
+            0.01026167906820774,
+            -0.005216030869632959,
+            0.008356492035090923,
+            -0.0010823167394846678,
+            -0.037422213703393936,
+            0.012344993650913239,
+            0.02216522768139839,
+            -0.002635121811181307,
+            0.0016089539276435971,
+            -0.006145390681922436,
+            0.022103270515799522,
+            -0.013971373438835144,
+            -0.00650551775470376,
+            -0.00046443779137916863,
+            0.00797700323164463,
+            -0.01795213110744953,
+            0.004778457339853048,
+            0.0755879208445549,
+            -0.007775642443448305,
+            0.006122156511992216,
+            0.009270362555980682,
+            -0.015148562379181385,
+            -0.01637221872806549,
+            -0.004561606794595718,
+            -0.020260039716959,
+            0.008844406343996525,
+            -0.0018819533288478851,
+            -0.0110981035977602,
+            0.012468908913433552,
+            0.0053050946444272995,
+            0.002031038049608469,
+            -0.008379725739359856,
+            0.010811550542712212,
+            0.009525936096906662,
+            0.010432061739265919,
+            0.02360573597252369,
+            0.011477591469883919,
+            -0.00858883187174797,
+            -0.011795123107731342,
+            0.01129171997308731,
+            0.0028577810153365135,
+            0.016697494313120842,
+            0.007907302118837833,
+            0.0010648912284523249,
+            0.019501063972711563,
+            -0.002309845993295312,
+            0.02180897258222103,
+            -2.2689444449497387e-05,
+            -0.009301341138780117,
+            -0.004236331209540367,
+            -0.015349923633038998,
+            0.024271776899695396,
+            -0.0020155487582087517,
+            0.0015450604259967804,
+            -0.020213572308421135,
+            0.0013979118084535003,
+            0.013243375346064568,
+            0.011957760900259018,
+            -0.0020407189149409533,
+            -0.0051347119733691216,
+            0.0007154133054427803,
+            -0.016356728971004486,
+            0.032806396484375,
+            0.007578153628855944,
+            0.0007681738352403045,
+            0.0008838597568683326,
+            0.0009288755827583373,
+            -0.0032972905319184065,
+            0.0024608669336885214,
+            -0.004766840487718582,
+            0.009549170732498169,
+            -0.010943209752440453,
+            -0.010796061716973782,
+            0.010075807571411133,
+            -0.02402394637465477,
+            0.003566417610272765,
+            -0.00558777479454875,
+            -0.008062195032835007,
+            0.0076207490637898445,
+            0.00015537731815129519,
+            -0.007392281666398048,
+            -0.0030939930584281683,
+            0.0023408245760947466,
+            0.013429246842861176,
+            0.004406713880598545,
+            0.010850274004042149,
+            0.012956822291016579,
+            0.005359307397156954,
+            -0.006629432085901499,
+            0.015512561425566673,
+            -0.006946963258087635,
+            -0.004228586331009865,
+            -0.0012468908680602908,
+            -0.019996721297502518,
+            -0.0009419447160325944,
+            -0.004708755761384964,
+            0.0015092412941157818,
+            0.0001705036120256409,
+            -8.749050175538287e-05,
+            -0.0018490385264158249,
+            -0.011988739483058453,
+            -0.001831613015383482,
+            -0.011663463898003101,
+            -0.014498010277748108,
+            0.010253935120999813,
+            -0.0041201612912118435,
+            -0.022939693182706833,
+            -0.015349923633038998,
+            -0.020662762224674225,
+            0.02145271934568882,
+            -0.004387352149933577,
+            -0.005800753366202116,
+            0.006733985152095556,
+            -0.010215211659669876,
+            -0.015706177800893784,
+            -0.008069939911365509,
+            -0.0205853171646595,
+            0.010703125037252903,
+            0.003214035416021943,
+            0.0006791101768612862,
+            0.00882891658693552,
+            0.014397329650819302,
+            -0.01589204929769039,
+            0.0111755495890975,
+            0.012182355858385563,
+            0.011361422017216682,
+            -0.006854027509689331,
+            0.0008911203476600349,
+            -0.014103032648563385,
+            0.02596011385321617,
+            0.021623101085424423,
+            0.005169562995433807,
+            -0.04132552444934845,
+            -0.005862710531800985,
+            -0.013344055972993374,
+            -0.010524998418986797,
+            -0.012120398692786694,
+            0.011771888472139835,
+            -0.0038820127956569195,
+            0.0013117524795234203,
+            0.006288666743785143,
+            0.0015063370810821652,
+            0.004367990419268608,
+            -0.0022382077295333147,
+            -0.007709812838584185,
+            -0.03212486580014229,
+            -0.02817508764564991,
+            -0.00863530021160841,
+            0.020461400970816612,
+            0.002838419284671545,
+            -0.0009453330421820283,
+            -0.0036380558740347624,
+            0.0027048238553106785,
+            -0.005882072262465954,
+            -0.005018542055040598,
+            -0.014753584749996662,
+            -0.012739972211420536,
+            0.0101300198584795,
+            0.00802347157150507,
+            0.014195968396961689,
+            -0.011950016021728516,
+            0.0028190575540065765,
+            0.018788553774356842,
+            0.007690451107919216,
+            -0.023357905447483063,
+            0.01135367713868618,
+            -0.010912231169641018,
+            -0.009270362555980682,
+            -0.009433000348508358,
+            -0.009471723809838295,
+            0.0016980175860226154,
+            -0.008317768573760986,
+            0.003945906180888414,
+            -0.03615209087729454,
+            -0.011330443434417248,
+            -0.0013901671627536416,
+            -0.010346870869398117,
+            0.0009864765452221036,
+            0.02524760365486145,
+            0.006815304048359394,
+            0.01432762760668993,
+            -0.0006733016925863922,
+            0.017502939328551292,
+            -0.0018984107300639153,
+            0.018726596608757973,
+            0.012763205915689468,
+            -0.00797700323164463,
+            0.011717676185071468,
+            0.014738094992935658,
+            -0.0033863543067127466,
+            0.015620986931025982,
+            -0.010168743319809437,
+            0.005285732913762331,
+            -0.009556914679706097,
+            -0.003214035416021943,
+            0.002495717955753207,
+            0.030901208519935608,
+            -0.003618694143369794,
+            0.00858883187174797,
+            0.01098193321377039,
+            -0.015241498127579689,
+            0.0026951429899781942,
+            0.014962689951062202,
+            0.01141563430428505,
+            0.007760153152048588,
+            0.0024434414226561785,
+            -0.024225307628512383,
+            0.004193735308945179,
+            0.005037903785705566,
+            0.0031520782504230738,
+            -0.014366351068019867,
+            0.0021859314292669296,
+            0.015675200149416924,
+            0.008550108410418034,
+            -0.025325050577521324,
+            0.016418687999248505,
+            0.0009138703462667763,
+            -0.003909118939191103,
+            0.029119934886693954,
+            -0.011268485337495804,
+            -0.005723306443542242,
+            0.009463978931307793,
+            0.0009980935137718916,
+            0.0030029932968318462,
+            0.008937342092394829,
+            0.0228312686085701,
+            -0.003988501615822315,
+            -0.032341714948415756,
+            -0.022459525614976883,
+            0.015706177800893784,
+            -0.0047823297791182995,
+            -0.022939693182706833,
+            -0.026703599840402603,
+            -0.003113354789093137,
+            -0.02228914201259613,
+            -0.0022323993034660816,
+            -0.020058678463101387,
+            -0.014056564308702946,
+            0.001268188701942563,
+            0.029987338930368423,
+            0.016201836988329887,
+            0.008224832825362682,
+            -0.011051635257899761,
+            -0.029956359416246414,
+            -0.024271776899695396,
+            0.01810702309012413,
+            -0.007055388763546944,
+            -0.005541306920349598,
+            0.00591692328453064,
+            -0.009951893240213394,
+            -0.008480406366288662,
+            0.011802867986261845,
+            0.027137301862239838,
+            -0.002747419523075223,
+            -0.029987338930368423,
+            -0.02688947319984436,
+            -0.014335372485220432,
+            -0.01840132102370262,
+            -0.017704300582408905,
+            -0.008627555333077908,
+            0.0045035220682621,
+            0.00030300996149890125,
+            0.016186347231268883,
+            0.014962689951062202,
+            -0.000480653194244951,
+            0.0028597170021384954,
+            -2.7711375878425315e-05,
+            -0.001763847190886736,
+            0.006888878531754017,
+            -0.004933350719511509,
+            0.026997897773981094,
+            0.014614180661737919,
+            -0.006594581063836813,
+            0.018711106851696968,
+            0.028314489871263504,
+            0.020430423319339752,
+            0.003157886676490307,
+            -0.010695381090044975,
+            -0.014606435783207417,
+            0.009727297350764275,
+            -0.013421501964330673,
+            0.023264970630407333,
+            0.022552460432052612,
+            -0.01480779703706503,
+            -0.014885243959724903,
+            -0.017905661836266518,
+            0.015210519544780254,
+            -0.007562664337456226,
+            -0.008224832825362682,
+            0.012368228286504745,
+            0.022335609421133995,
+            -0.007024410180747509,
+            0.002371803391724825,
+            -0.0010232636705040932,
+            -0.021731525659561157,
+            -0.006122156511992216,
+            -0.004596457816660404,
+            -0.008666278794407845,
+            0.008681767620146275,
+            0.008131897076964378,
+            -0.001928421319462359,
+            -0.006319645792245865,
+            0.005471604876220226,
+            -0.013367289677262306,
+            0.01301103550940752,
+            -0.00789568480104208,
+            0.03379771113395691,
+            -0.009138703346252441,
+            -0.0006733016925863922,
+            -0.022598929703235626,
+            -0.011446612887084484,
+            0.00845717266201973,
+            0.0017967619933187962,
+            0.01682141050696373,
+            -0.0011626676423475146,
+            -0.005688455421477556,
+            0.009913169778883457,
+            -0.0010687635513022542,
+            -0.001450188341550529,
+            -0.006892750971019268,
+            -0.010401083156466484,
+            0.0002884887217078358,
+            0.013258864171802998,
+            -0.006025348324328661,
+            0.0018054747488349676,
+            0.006095050368458033,
+            -0.0175339188426733,
+            0.029027000069618225,
+            0.023884542286396027,
+            0.017255110666155815,
+            0.014660648070275784,
+            0.0006394187803380191,
+            0.011647974140942097,
+            -0.003572226269170642,
+            0.0038820127956569195,
+            -0.020337486639618874,
+            0.021080974489450455,
+            -0.010811550542712212,
+            0.01846327818930149,
+            0.011655719019472599,
+            -0.015233753249049187,
+            -0.005022414494305849,
+            0.021406250074505806,
+            0.009967382065951824,
+            0.00889087375253439,
+            0.0017541663255542517,
+            -0.02470547705888748,
+            0.017766257748007774,
+            0.010099041275680065,
+            -0.020399443805217743,
+            0.019547531381249428,
+            -0.009471723809838295,
+            0.01233724970370531,
+            -0.0005300253978930414,
+            0.004418330732733011,
+            0.0058743273839354515,
+            0.021499186754226685,
+            0.011570527218282223,
+            -0.03884723410010338,
+            -0.013615119270980358,
+            -0.0017357728211209178,
+            0.006857899948954582,
+            -0.0036303112283349037,
+            -0.015233753249049187,
+            -0.041635312139987946,
+            0.01604694314301014,
+            0.009998360648751259,
+            -0.01643417589366436,
+            0.004987563472241163,
+            -0.002427952131256461,
+            -0.0042053526267409325,
+            -0.008170620538294315,
+            0.003186929039657116,
+            -0.004170501604676247,
+            0.02324948087334633,
+            0.0018984107300639153,
+            -0.006946963258087635,
+            0.013258864171802998,
+            0.01344473659992218,
+            -0.021948376670479774,
+            0.0009874446550384164,
+            0.0021375271026045084,
+            0.000636998622212559,
+            0.023683181032538414,
+            -0.014707116410136223,
+            -0.0017657832941040397,
+            -0.01685238815844059,
+            -0.0013069120468571782,
+            -0.021592123433947563,
+            -0.001188805908896029,
+            -0.015187285840511322,
+            -0.008511385880410671,
+            0.025030752643942833,
+            -0.011005166918039322,
+            0.010997422970831394,
+            -0.006687517277896404,
+            0.006943091284483671,
+            -0.006904367823153734,
+            -0.01936165988445282,
+            -0.0008267428493127227,
+            0.009975126944482327,
+            -0.000512115890160203,
+            0.002941036131232977,
+            8.482827252009884e-05,
+            0.002464739140123129,
+            0.008867640048265457,
+            0.006234454456716776,
+            -0.0075278133153915405,
+            0.02489134855568409,
+            -0.012073930352926254,
+            0.0016399325104430318,
+            0.011911292560398579,
+            0.0004743606550619006,
+            -0.010424317792057991,
+            0.025588368996977806,
+            -0.02139076218008995,
+            -0.00800798274576664,
+            -0.00014799568452872336,
+            -0.0016544538084417582,
+            -0.006513262167572975,
+            0.006718495860695839,
+            0.02946070022881031,
+            0.0062654330395162106,
+            0.010447551496326923,
+            -0.0281441081315279,
+            -0.011493081226944923,
+            0.012538610957562923,
+            -0.022273652255535126,
+            -0.0007739823195151985,
+            -0.0004322490422055125,
+            -0.00619960343465209,
+            0.026378324255347252,
+            -0.02022906206548214,
+            -0.01093546487390995,
+            -0.018664639443159103,
+            0.005405775271356106,
+            -0.011121337302029133,
+            0.008294534869492054,
+            -0.0009777637897059321,
+            0.006540368776768446,
+            0.016805920749902725,
+            -0.019563021138310432,
+            0.016031453385949135,
+            0.005119222681969404,
+            -0.016217326745390892,
+            0.010222956538200378,
+            -0.017580386251211166,
+            -0.0008199662552215159,
+            0.020275529474020004,
+            -0.00400399137288332,
+            0.008751469664275646,
+            -0.003363120136782527,
+            0.005208286456763744,
+            -0.006513262167572975,
+            0.0001492057926952839,
+            -0.007249005138874054,
+            0.014428308233618736,
+            -0.016805920749902725,
+            -0.017828216776251793,
+            -0.008821171708405018,
+            0.0016709112096577883,
+            0.021297825500369072,
+            0.011136826127767563,
+            -0.010044828988611698,
+            -0.011059380136430264,
+            -0.011694442480802536,
+            0.041604332625865936,
+            0.005762029904872179,
+            0.027230238541960716,
+            -0.02261441759765148,
+            0.00545998802408576,
+            0.025170156732201576,
+            -0.014242436736822128,
+            0.0012788375606760383,
+            -0.0022788674104958773,
+            0.0053050946444272995,
+            0.011562783271074295,
+            -0.01819995976984501,
+            0.0049139889888465405,
+            -0.00019216447253711522,
+            -0.00782985519617796,
+            -0.000656844291370362,
+            0.003442503046244383,
+            0.00020196630794089288,
+            0.016325751319527626,
+            -0.004027225077152252,
+            -0.016511622816324234,
+            -0.004429947584867477,
+            -0.0003216455806978047,
+            -0.027896279469132423,
+            0.013514438644051552,
+            0.016589069738984108,
+            0.01566745527088642,
+            0.027989214286208153,
+            -0.014513500034809113,
+            0.016031453385949135,
+            0.009533680975437164,
+            0.013901671394705772,
+            -0.010610189288854599,
+            0.008952830918133259,
+            -0.01595400646328926,
+            0.007446494419127703,
+            -0.03454120084643364,
+            -0.007887939922511578,
+            -0.00802347157150507,
+            0.01984182931482792,
+            0.028763681650161743,
+            0.002923610620200634,
+            0.006493900436908007,
+            0.000196883876924403,
+            0.0024202074855566025,
+            -0.01714668609201908,
+            -0.020182594656944275,
+            -0.0003337466041557491,
+            0.0029332914855331182,
+            0.0006950835813768208,
+            -0.004495777189731598,
+            0.00472037261351943,
+            -0.012391461990773678,
+            0.0026835259050130844,
+            -0.013134949840605259,
+            0.005971135571599007,
+            -0.0059982421807944775,
+            0.009603383019566536,
+            0.0019429425010457635,
+            0.020864123478531837,
+            0.02300165221095085,
+            0.004100799560546875,
+            0.0046971384435892105,
+            -0.024333734065294266,
+            0.00920066051185131,
+            -0.013847459107637405,
+            0.012329504825174809,
+            0.019408127292990685,
+            0.008248066529631615,
+            0.006571347359567881,
+            0.01233724970370531,
+            0.012918098829686642,
+            -0.0031772484071552753,
+            0.024287264794111252,
+            0.005014669615775347,
+            -0.0009070937521755695,
+            0.002052335999906063,
+            -0.014954946003854275,
+            -0.0070979841984808445,
+            -0.0015286030247807503,
+            0.0018848575418815017,
+            -0.004731989465653896,
+            0.01906736195087433,
+            0.015256987884640694,
+            -0.01050176378339529,
+            0.017874684184789658,
+            -0.03937387093901634,
+            0.017379024997353554,
+            -0.0027377386577427387,
+            -0.0060911779291927814,
+            -0.00545998802408576,
+            0.00665653869509697,
+            0.004515138920396566,
+            0.003669034456834197,
+            0.03773200139403343,
+            -0.010408828034996986,
+            0.003963331691920757,
+            -0.005847221240401268,
+            0.007121218368411064,
+            -0.020864123478531837,
+            0.008302279748022556,
+            0.005336073227226734,
+            0.02160761132836342,
+            -0.0056497324258089066,
+            -0.0014482521219179034,
+            0.02328045852482319,
+            0.008859895169734955,
+            0.03469609469175339,
+            -0.01566745527088642,
+            -0.004418330732733011,
+            0.01048627495765686,
+            0.0207402091473341,
+            -0.00580462533980608,
+            -0.008976065553724766,
+            0.007903429679572582,
+            0.0035586729645729065,
+            0.021824462339282036,
+            -0.00515407370403409,
+            -0.0006650729919783771,
+            0.0054367538541555405,
+            -0.0060911779291927814,
+            0.009735042229294777,
+            0.011252996511757374,
+            -0.009076746180653572,
+            0.007539430167526007,
+            0.021530164405703545,
+            -0.009975126944482327,
+            -0.008658533915877342,
+            -0.009649851359426975,
+            0.0010658593382686377,
+            -0.03068435750901699,
+            0.001649613375775516,
+            0.005359307397156954,
+            0.015187285840511322,
+            -0.0004603234410751611,
+            -0.009014789015054703,
+            0.021019017323851585,
+            0.017255110666155815,
+            -0.005936284549534321,
+            -0.0028132491279393435,
+            0.0023795480374246836,
+            -0.0017154429806396365,
+            -0.006160879973322153,
+            -0.0006248975405469537,
+            -0.002656419761478901,
+            0.014730350114405155,
+            -0.0015469965292140841,
+            -0.006617815233767033,
+            -0.00756653631106019,
+            0.0033302053343504667,
+            0.010834784246981144,
+            -0.005661349277943373,
+            0.013057502917945385,
+            0.016805920749902725,
+            -0.00031704717548564076,
+            0.019671445712447166,
+            0.01037010457366705,
+            -0.010447551496326923,
+            0.015938518568873405,
+            -0.007984748110175133,
+            0.006563602481037378,
+            0.0164031982421875,
+            -0.01277095079421997,
+            -0.01246116403490305,
+            0.007210282143205404,
+            0.00665653869509697,
+            0.028918573632836342,
+            0.014792308211326599,
+            -0.02109646424651146,
+            0.014908477663993835,
+            0.015768134966492653,
+            -0.013018779456615448,
+            0.003537375247105956,
+            -0.01784370467066765,
+            0.020260039716959,
+            0.019051872193813324,
+            -0.015535795129835606,
+            -0.017224133014678955,
+            -0.014947201125323772,
+            -0.005092116538435221,
+            0.006331262644380331,
+            0.0033263331279158592,
+            -0.00617636926472187,
+            -0.027989214286208153,
+            -0.0049139889888465405,
+            -0.018633661791682243,
+            0.013870692811906338,
+            -0.008674023672938347,
+            0.0007715621031820774,
+            -0.004511266481131315,
+            0.007984748110175133,
+            -0.0033882902935147285,
+            0.032465629279613495,
+            -0.0051811798475682735,
+            -0.02832997962832451,
+            0.004398969002068043,
+            0.016356728971004486,
+            0.012623801827430725,
+            -0.029243851080536842,
+            0.010463040322065353,
+            0.004805563949048519,
+            -0.012530866079032421,
+            0.005990497302263975,
+            -0.02258343994617462,
+            -0.007632365915924311,
+            -0.0009395244996994734,
+            -0.016743963584303856,
+            -0.027121812105178833,
+            -0.008433938957750797,
+            -0.006993431597948074,
+            -0.005401902832090855,
+            -0.009525936096906662,
+            -0.008720491081476212,
+            -0.011477591469883919,
+            0.02303262986242771,
+            -0.007899557240307331,
+            -0.001866463921032846,
+            -0.0014637414133176208,
+            0.00930908601731062,
+            -0.023776117712259293,
+            -0.015605497173964977,
+            0.006389347370713949,
+            -0.002594462363049388,
+            0.022815778851509094,
+            -0.012747716158628464,
+            0.005940156988799572,
+            -0.009316830895841122,
+            0.009851212613284588,
+            0.015241498127579689,
+            0.004956584889441729,
+            0.017440982162952423,
+            -0.009889936074614525,
+            0.0021026763133704662,
+            -0.008627555333077908,
+            -0.00018986526993103325,
+            -1.027075541060185e-05,
+            0.014428308233618736,
+            0.007624621503055096,
+            0.014141756109893322,
+            0.017487451434135437,
+            -0.013553161174058914,
+            0.005003052763640881,
+            -0.0003274540649726987,
+            -0.017255110666155815,
+            0.0027725896798074245,
+            -0.007984748110175133,
+            -0.0008896682411432266,
+            0.001758038648404181,
+            6.395397940650582e-05,
+            -0.02112744189798832,
+            -0.0009124181815423071,
+            0.0018809852190315723,
+            0.0010222955606877804,
+            -0.025201136246323586,
+            0.00889087375253439,
+            -0.00889087375253439,
+            -0.0014918158994987607,
+            -0.019113829359412193,
+            0.019795360043644905,
+            0.026300877332687378,
+            -0.004619691986590624,
+            0.00036206303047947586,
+            -0.013537672348320484,
+            -0.02049238048493862,
+            0.0001489637652412057,
+            -0.011314953677356243,
+            -0.011345932260155678,
+            -0.0022169100120663643,
+            -0.003214035416021943,
+            0.008805682882666588,
+            0.0004189379105810076,
+            -0.009433000348508358,
+            -0.004956584889441729,
+            -0.011764144524931908,
+            -0.017549408599734306,
+            -0.010401083156466484,
+            0.01933068037033081,
+            -0.0018016024259850383,
+            0.0020794421434402466,
+            -0.0032508226577192545,
+            0.0005271211848594248,
+            0.011748654767870903,
+            0.013475715182721615,
+            0.013344055972993374,
+            -0.023915521800518036,
+            -0.017890173941850662,
+            -0.013320821337401867,
+            0.0029487807769328356,
+            -0.007187047973275185,
+            0.004844287410378456,
+            0.026362836360931396,
+            -0.0009695350890979171,
+            -0.01091997604817152,
+            0.002327271504327655,
+            0.019051872193813324,
+            -0.031272951513528824,
+            -0.013390523381531239,
+            -0.025077221915125847,
+            0.02225816436111927,
+            -0.013351799920201302,
+            0.011516314931213856,
+            0.010191977955400944,
+            -0.025944624096155167,
+            0.006768836174160242,
+            -0.007686578668653965,
+            0.026037558913230896,
+            0.01634124107658863,
+            0.0005934348446317017,
+            -0.00944074522703886,
+            0.013622863218188286,
+            -0.0008722427301108837,
+            -0.013576395809650421,
+            0.002201420720666647,
+            0.014645159244537354,
+            -0.014691627584397793,
+            -0.009510447271168232,
+            -0.004507394507527351,
+            0.004313777666538954,
+            0.012763205915689468,
+            -0.0030339721124619246,
+            -0.004031097516417503,
+            -0.011624740436673164,
+            0.013344055972993374,
+            -0.003930416889488697,
+            0.012383717112243176,
+            0.007380664348602295,
+            -0.014056564308702946,
+            -0.019532041624188423,
+            0.033333033323287964,
+            0.00400399137288332,
+            0.00338054564781487,
+            -0.010997422970831394,
+            0.02740062028169632,
+            -0.009897680021822453,
+            0.010571465827524662,
+            -0.006408709101378918,
+            0.0031985461246222258,
+            0.012321759946644306,
+            -0.012104909867048264,
+            -0.01981084980070591,
+            0.021468207240104675,
+            -0.00364192808046937,
+            -0.0010193913476541638,
+            0.008310024626553059,
+            -0.020182594656944275,
+            -0.00663330452516675,
+            0.002180122770369053,
+            0.009541425853967667,
+            -0.011818356812000275,
+            0.006369986105710268,
+            0.017905661836266518,
+            0.003912991378456354,
+            -0.0007672057254239917,
+            0.019996721297502518,
+            -0.01672847382724285,
+            0.011717676185071468,
+            -0.010850274004042149,
+            0.0016825281782075763,
+            0.0006065039779059589,
+            -0.0048597767017781734,
+            0.0058743273839354515,
+            -0.014536733739078045,
+            0.011988739483058453,
+            -6.244135147426277e-05,
+            -0.013979118317365646,
+            0.007601387333124876,
+            -0.013359544798731804,
+            -0.016836898401379585,
+            0.041542377322912216,
+            -0.005301222205162048,
+            -0.0038665232714265585,
+            -0.0056962003000080585,
+            0.01350669376552105,
+            0.02823704481124878,
+            -0.005336073227226734,
+            -0.01153180468827486,
+            -0.010780571959912777,
+            -0.016573579981923103,
+            0.015907539054751396,
+            -0.01810702309012413,
+            0.002495717955753207,
+            -0.010532742366194725,
+            -0.007903429679572582,
+            0.004302160814404488,
+            -0.003516077296808362,
+            0.014722606167197227,
+            0.00950270239263773,
+            0.023357905447483063,
+            0.00797700323164463,
+            -0.00027348342700861394,
+            0.010393339209258556,
+            0.011663463898003101,
+            0.003243078012019396,
+            -0.010509508661925793,
+            -0.00011949774489039555,
+            -0.02894955314695835,
+            -0.008449427783489227,
+            -0.004863648675382137,
+            -0.001485039247199893,
+            -0.00428667152300477,
+            -0.01239920686930418,
+            0.006617815233767033,
+            -0.0022052929271012545,
+            0.013909416273236275,
+            -0.009254872798919678,
+            -0.017766257748007774,
+            0.03389064967632294,
+            0.004495777189731598,
+            0.006466794293373823,
+            -0.008286789990961552,
+            -0.03705047070980072,
+            -0.0032566310837864876,
+            -0.008116407319903374,
+            -0.013529927469789982,
+            0.004538373090326786,
+            0.01283290795981884,
+            -0.014831030741333961,
+            -0.028097640722990036,
+            0.0019390701781958342,
+            -0.006683644838631153,
+            0.02978597767651081,
+            -0.007798876613378525,
+            0.00419760774821043,
+            0.0037251831963658333,
+            0.0011558911064639688,
+            0.01239920686930418,
+            -0.00845717266201973,
+            -0.0022536972537636757,
+            0.016387708485126495,
+            -0.010858018882572651,
+            0.011051635257899761,
+            -0.010966443456709385,
+            -0.02554190158843994,
+            -0.01456771232187748,
+            0.021344292908906937,
+            -0.0004823473282158375,
+            -0.00024903935263864696,
+            0.014823286794126034,
+            0.02354377694427967,
+            0.011260741390287876,
+            0.00969631876796484,
+            0.006408709101378918,
+            -0.001035848748870194,
+            0.005975008010864258,
+            0.006095050368458033,
+            -0.0024163350462913513,
+            -0.013398268260061741,
+            0.017564896494150162,
+            0.009773765690624714,
+            -0.01810702309012413,
+            0.0008514289511367679,
+            0.0036612898111343384,
+            0.0168833676725626,
+            0.006935346405953169,
+            0.009316830895841122,
+            -0.0143121387809515,
+            -0.023760627955198288,
+            -0.00014932679187040776,
+            0.002873270306736231,
+            0.015721667557954788,
+            0.010579210706055164,
+            -0.014319883659482002,
+            0.0005963391158729792,
+            0.008712747134268284,
+            0.020213572308421135,
+            -0.024209819734096527,
+            0.019702425226569176,
+            -0.004673904739320278,
+            -0.0021084847394376993,
+            0.007946024648845196,
+            0.011632485315203667,
+            -0.011872569099068642,
+            -0.00650551775470376,
+            0.006249943748116493,
+            0.003157886676490307,
+            0.03172214329242706,
+            -0.0017260919557884336,
+            0.009665340185165405,
+            -0.010997422970831394,
+            -0.0002405686245765537,
+            0.0006728176376782358,
+            -0.0006820144481025636,
+            -0.002884887158870697,
+            -0.011020656675100327,
+            0.0018974426202476025,
+            -0.020306508988142014,
+            0.0024473138619214296,
+            0.00463130883872509,
+            -0.005684582982212305,
+            -0.0032972905319184065,
+            -0.019222255796194077,
+            -0.0022672503255307674,
+            0.003293418325483799,
+            0.014335372485220432,
+            -0.031691163778305054,
+            0.004972074180841446,
+            0.010672146454453468,
+            -0.002309845993295312,
+            -0.010416572913527489,
+            0.011423379182815552,
+            -0.0026060794480144978,
+            0.011988739483058453,
+            -0.0038142469711601734,
+            -0.0004905760288238525,
+            0.025681305676698685,
+            -0.005308967083692551,
+            0.014707116410136223,
+            -0.01026167906820774,
+            -0.002993312431499362,
+            -0.028252532705664635,
+            0.005235392600297928,
+            -0.007403898518532515,
+            -0.0012081675231456757,
+            -0.004402841441333294,
+            -0.01130720879882574,
+            -0.0026990154292434454,
+            0.002122037811204791,
+            0.008000237867236137,
+            0.012910354882478714,
+            -0.0007725302129983902,
+            -0.010075807571411133,
+            0.004832670092582703,
+            0.0075355577282607555,
+            -0.015520306304097176,
+            -0.0070437719114124775,
+            -0.020306508988142014,
+            0.029662061482667923,
+            -0.013731288723647594,
+            -0.0011123273288831115,
+            -0.020864123478531837,
+            0.010160998441278934,
+            -0.0063119009137153625,
+            -0.002782270545139909,
+            0.004708755761384964,
+            -0.015109838917851448,
+            0.01344473659992218,
+            0.008751469664275646,
+            0.01450575515627861,
+            -0.02151467651128769,
+            -0.01762685552239418,
+            -0.0044725434854626656,
+            -0.002735802438110113,
+            -0.010648912750184536,
+            -0.0005334137240424752,
+            0.002013612538576126,
+            0.021251358091831207,
+            -0.00032503384863957763,
+            0.0042053526267409325,
+            -0.003214035416021943,
+            0.017224133014678955,
+            -0.010137764737010002,
+            -0.013057502917945385,
+            0.0024260159116238356,
+            0.005994369741529226,
+            0.016186347231268883,
+            0.03158273920416832,
+            -0.020693741738796234,
+            -0.00272999401204288,
+            0.0032740565948188305,
+            0.010656657628715038,
+            -0.004085310269147158,
+            6.304640555754304e-05,
+            -0.027803342789411545,
+            -0.04464024305343628,
+            -0.002422143705189228,
+            0.013057502917945385,
+            -0.015636475756764412,
+            0.012670270167291164,
+            0.008743725717067719,
+            0.010021595284342766,
+            -0.005196669604629278,
+            0.02620794251561165,
+            -0.004793947096914053,
+            0.014304393902420998,
+            0.011733165942132473,
+            0.011911292560398579,
+            0.0009874446550384164,
+            0.0007459079497493804,
+            -0.018354853615164757,
+            -0.008255811408162117,
+            -0.0015954007394611835,
+            0.01765783317387104,
+            -0.027478067204356194,
+            0.008526874706149101,
+            -0.011516314931213856,
+            0.013839714229106903,
+            0.0003666614356916398,
+            0.005661349277943373,
+            -0.01771979033946991,
+            0.012445674277842045,
+            -0.0009337160154245794,
+            -0.025325050577521324,
+            0.004805563949048519,
+            0.011276230216026306,
+            -0.010362359695136547,
+            0.006877261213958263,
+            0.019950253888964653,
+            0.0023079097736626863,
+            0.008495896123349667,
+            -0.014126266352832317,
+            0.010703125037252903,
+            0.009649851359426975,
+            -0.021080974489450455,
+            -0.01233724970370531,
+            -0.007059261202812195,
+            -0.006335135083645582,
+            -0.0035470561124384403,
+            -0.00013105422840453684,
+            0.010176488198339939,
+            -0.016000475734472275,
+            0.010099041275680065,
+            -0.006559730041772127,
+            0.006400964688509703,
+            0.00944074522703886,
+            0.001115231541916728,
+            -0.013576395809650421,
+            0.012716737575829029,
+            -0.0218709297478199,
+            0.008426194079220295,
+            -0.04287445917725563,
+            -0.005847221240401268,
+            -0.015078860335052013,
+            0.005204414017498493,
+            -0.003798757679760456,
+            -0.004453181754797697,
+            -0.006203475873917341,
+            -0.005061137955635786,
+            0.00920066051185131,
+            0.021747015416622162,
+            0.012135888449847698,
+            -0.016620049253106117,
+            -0.007551047019660473,
+            0.005425137002021074,
+            0.0004854935978073627,
+            -0.013336311094462872,
+            0.009316830895841122,
+            -0.01576039008796215,
+            0.0009259713697247207,
+            0.006339007057249546,
+            -0.01072635967284441,
+            0.013452480547130108,
+            0.004731989465653896,
+            -0.003401843598112464,
+            0.013390523381531239,
+            -0.0050417762249708176,
+            -0.0057891360484063625,
+            -0.007272239308804274,
+            -0.013739033602178097,
+            -0.003818119177594781,
+            -0.003072695340961218,
+            0.012747716158628464,
+            0.001529571134597063,
+            -0.0003243077953811735,
+            -0.00782985519617796,
+            0.009154192171990871,
+            0.002220782218500972,
+            -0.0013146566925570369,
+            0.005773646757006645,
+            -0.02300165221095085,
+            0.0001287550403503701,
+            0.004755223635584116,
+            -0.0009487213101238012,
+            -0.006246071308851242,
+            -0.012151377275586128,
+            -0.011903548613190651,
+            -0.038165703415870667,
+            0.009363298304378986,
+            0.0019197085639461875,
+            -0.0076207490637898445,
+            0.0064822835847735405,
+            0.017255110666155815,
+            0.01456771232187748,
+            0.016743963584303856,
+            -0.020693741738796234,
+            -0.02303262986242771,
+            0.005556796211749315,
+            -0.009402021765708923,
+            0.02261441759765148,
+            0.004046586807817221,
+            -0.005847221240401268,
+            0.0013243375578895211,
+            -0.013258864171802998,
+            0.008929597213864326,
+            0.005723306443542242,
+            -0.001592496526427567,
+            0.005425137002021074,
+            -0.013692565262317657,
+            -0.010904486291110516,
+            0.01048627495765686,
+            0.032496608793735504,
+            -0.0004261985304765403,
+            0.0024105266202241182,
+            0.023295948281884193,
+            0.015466093085706234,
+            0.016356728971004486,
+            -0.021111954003572464,
+            0.011167804710566998,
+            0.010060317814350128,
+            -0.0018645278178155422,
+            -0.014892988838255405,
+            0.00513083953410387,
+            -0.012716737575829029,
+            -0.010664401575922966,
+            0.014993669465184212,
+            0.007853088900446892,
+            0.017735280096530914,
+            -0.02369867078959942,
+            0.00789568480104208,
+            0.00042765066609717906,
+            0.00013710475468542427,
+            0.011524059809744358,
+            0.012956822291016579,
+            0.03370477631688118,
+            0.027152791619300842,
+            0.020662762224674225,
+            -0.009285852313041687,
+            0.014366351068019867,
+            -0.009192915633320808,
+            -0.011090358719229698,
+            -0.019470084458589554,
+            -0.015249243006110191,
+            -0.011136826127767563,
+            0.017998598515987396,
+            0.008790193125605583,
+            -0.009355553425848484,
+            -0.005947901867330074,
+            0.01586107164621353,
+            -0.003909118939191103,
+            -0.004635181277990341,
+            0.01646515540778637,
+            -0.026269899681210518,
+            0.016836898401379585,
+            -0.000745423894841224,
+            -0.014668392948806286,
+            -0.014691627584397793,
+            -0.004182118456810713,
+            -0.012670270167291164,
+            0.0003964299685321748,
+            -0.020817656069993973,
+            0.013181418180465698,
+            0.024597052484750748,
+            -0.0030920570716261864,
+            -0.004100799560546875,
+            -0.011299464851617813,
+            -0.0005111477803438902,
+            0.018385831266641617,
+            0.0034773540683090687,
+            0.009192915633320808,
+            -0.0023872926831245422,
+            0.013924905098974705,
+            0.00524313747882843,
+            0.0027958236169070005,
+            -0.0007565568666905165,
+            -0.016263794153928757,
+            0.012197845615446568,
+            0.0026835259050130844,
+            0.012135888449847698,
+            -0.022242674604058266,
+            0.007938280701637268,
+            -0.011764144524931908,
+            -0.0023698671720921993,
+            -0.02228914201259613,
+            0.0189744271337986,
+            -0.01610890030860901,
+            0.001046497724018991,
+            0.003841353114694357,
+            0.0006224773242138326,
+            0.009580149315297604,
+            0.01610890030860901,
+            -0.023388884961605072,
+            -0.0003460896550677717,
+            -0.014908477663993835,
+            -0.021917399019002914,
+            -0.01607792265713215,
+            0.004728117026388645,
+            -0.019501063972711563,
+            0.005444498732686043,
+            -0.0025557391345500946,
+            -0.015125328674912453,
+            0.018478767946362495,
+            0.014691627584397793,
+            -0.004379607271403074,
+            0.002412462839856744,
+            0.005425137002021074,
+            -0.0014405074762180448,
+            0.01971791312098503,
+            0.00013274837692733854,
+            -0.0010048701660707593,
+            -0.009425255469977856,
+            -0.0168833676725626,
+            -0.0065752193331718445,
+            -0.035005878657102585,
+            0.0260530486702919,
+            -0.004670032300055027,
+            -0.016511622816324234,
+            0.014319883659482002,
+            0.009115469641983509,
+            -0.014629669487476349,
+            -0.007059261202812195,
+            0.016294771805405617,
+            -0.024287264794111252,
+            0.02244403585791588,
+            -0.009611127898097038,
+            -0.008774704299867153,
+            0.00784921646118164,
+            0.05154848098754883,
+            0.009146448224782944,
+            0.003492843359708786,
+            0.011036145500838757,
+            0.005839476361870766,
+            -0.0013165927957743406,
+            -0.023683181032538414,
+            0.005568413063883781,
+            -0.007190920412540436,
+            -0.003601268632337451,
+            0.002536377403885126,
+            -0.004801691509783268,
+            0.031149037182331085,
+            -0.004379607271403074,
+            -0.005289605353027582,
+            0.004484160337597132,
+            -0.02196386642754078,
+            -0.0001332324172835797,
+            0.023203013464808464,
+            -2.3128106477088295e-05,
+            -0.023590246215462685,
+            -0.006571347359567881,
+            -0.015071115456521511,
+            -0.013770012184977531,
+            -0.018989915028214455,
+            0.005231520161032677,
+            -0.012755461037158966,
+            -0.0033940989524126053,
+            0.0016660707769915462,
+            -0.005940156988799572,
+            -0.002582845278084278,
+            0.009603383019566536,
+            0.05362405255436897,
+            0.014103032648563385,
+            -0.006335135083645582,
+            -0.005854965653270483,
+            0.009192915633320808,
+            -0.010524998418986797,
+            -0.019795360043644905,
+            -0.009820234030485153,
+            -0.029507169499993324,
+            -0.012747716158628464,
+            -0.0143121387809515,
+            -0.010191977955400944,
+            -0.010277168825268745,
+            0.00963436160236597,
+            -0.018122512847185135,
+            -0.007628493942320347,
+            -0.020337486639618874,
+            0.002941036131232977,
+            -0.011268485337495804,
+            -0.014714861288666725,
+            -0.008054450154304504,
+            -0.04807887226343155,
+            0.005796880926936865,
+            0.019593998789787292,
+            0.01263929158449173,
+            0.002515079453587532,
+            0.004383479710668325,
+            -0.01993476413190365,
+            -0.030792783945798874,
+            0.013839714229106903,
+            0.011283975094556808,
+            0.011733165942132473,
+            0.03283737599849701,
+            -0.011709931306540966,
+            -0.00813964195549488,
+            -0.004693266469985247,
+            -0.0017251238459721208,
+            0.012066186405718327,
+            0.00969631876796484,
+            -0.004917861428111792,
+            -0.012375972233712673,
+            0.016217326745390892,
+            -0.01981084980070591,
+            -0.023683181032538414,
+            -0.01672847382724285,
+            0.006157007534056902,
+            0.00813964195549488,
+            -0.008054450154304504,
+            0.002973950933665037,
+            0.0056962003000080585,
+            -0.007159941829741001,
+            -0.00022423222253564745,
+            0.004890755284577608,
+            -0.023048119619488716,
+            0.0008490087930113077,
+            -0.002257569460198283,
+            0.0061376458033919334,
+            0.021266845986247063,
+            -0.010168743319809437,
+            -0.0014075926737859845,
+            -0.0053902859799563885,
+            -0.0028984404634684324,
+            -0.006764963734894991,
+            0.02620794251561165,
+            0.009154192171990871,
+            -0.009882191196084023,
+            -0.01011453103274107,
+            -0.01129171997308731,
+            -0.0011278166202828288,
+            -0.017255110666155815,
+            -0.016186347231268883,
+            0.0015566773945465684,
+            -0.016712984070181847,
+            0.013971373438835144,
+            -0.00876695942133665,
+            0.00400399137288332,
+            -0.014110777527093887,
+            0.02019808255136013,
+            0.01929970271885395,
+            -0.009711808525025845,
+            0.023776117712259293,
+            -0.007709812838584185,
+            -0.038382552564144135,
+            -0.00800798274576664,
+            0.01843230053782463,
+            -0.006118284538388252,
+            0.010037084110081196,
+            -0.011314953677356243,
+            0.008209343999624252,
+            0.00015319912927225232,
+            0.0018122512847185135,
+            -0.0003112386621069163,
+            0.027555514127016068,
+            0.015535795129835606,
+            -0.005560668651014566,
+            -0.005494839046150446,
+            0.004255692940205336,
+            0.009541425853967667,
+            0.017580386251211166,
+            0.012151377275586128,
+            -0.021019017323851585,
+            -0.014257925562560558,
+            -0.016263794153928757,
+            0.011012911796569824,
+            0.015125328674912453,
+            -0.0040891822427511215,
+            0.01691434532403946,
+            0.014126266352832317,
+            -0.019098341464996338,
+            0.0019884423818439245,
+            -0.008193854242563248,
+            -0.00158184755127877,
+            -0.03029712475836277,
+            0.011043890379369259,
+            0.0074387495405972,
+            -0.0072218989953398705,
+            0.036616768687963486,
+            0.009858956560492516,
+            0.037577107548713684,
+            0.018680129200220108,
+            0.0008974128868430853,
+            0.009045767597854137,
+            -0.018370341509580612,
+            0.013065247796475887,
+            0.01500141341239214,
+            -0.012724482454359531,
+            -0.005734923295676708,
+            0.0013446672819554806,
+            0.0007551047019660473,
+            0.012631546705961227,
+            0.02004319056868553,
+            -0.01795213110744953,
+            0.009773765690624714,
+            -0.015032391995191574,
+            -0.005250881891697645,
+            0.006555858068168163,
+            -0.013932649977505207,
+            -0.008495896123349667,
+            -0.00963436160236597,
+            0.019408127292990685,
+            -0.012600568123161793,
+            0.016573579981923103,
+            -0.009595638141036034,
+            0.015520306304097176,
+            -0.029724018648266792,
+            -0.003669034456834197,
+            -0.006679772399365902,
+            -0.019593998789787292,
+            -0.012887120246887207,
+            0.009820234030485153,
+            0.010633422993123531,
+            -0.009301341138780117,
+            -0.011562783271074295,
+            0.020368466153740883,
+            -0.004185990896075964,
+            0.013529927469789982,
+            0.021189400926232338,
+            0.00737291993573308,
+            0.024132372811436653,
+            0.0005605200421996415,
+            0.014164989814162254,
+            0.005463860463351011,
+            0.010362359695136547,
+            -0.004677776712924242,
+            0.005161818582564592,
+            0.00932457484304905,
+            0.014203713275492191,
+            0.017766257748007774,
+            0.013196907006204128,
+            -0.00598275288939476,
+            0.007419387809932232,
+            0.02016710489988327,
+            0.0016254113288596272,
+            -0.030312614515423775,
+            -0.009719553403556347,
+            0.0001736498816171661,
+            0.00924712885171175,
+            0.009185170754790306,
+            3.817877222900279e-05,
+            -0.01093546487390995,
+            0.011144571006298065,
+            0.00969631876796484,
+            0.002286612056195736,
+            0.006280922330915928,
+            -0.01456771232187748,
+            -0.020957060158252716,
+            0.006106667220592499,
+            0.01798310875892639,
+            0.013917161151766777,
+            -0.014722606167197227,
+            0.022784801200032234,
+            0.006145390681922436,
+            -0.021406250074505806,
+            -0.01840132102370262,
+            -0.0009559819591231644,
+            0.0004542729293461889,
+            -0.002265314105898142,
+            0.03720536455512047,
+            -0.0011539548868313432,
+            -0.005955646280199289,
+            -0.0054754773154854774,
+            0.02775687538087368,
+            0.02049238048493862,
+            0.009177426807582378
+          ],
+          "embedding": [
+            0.0008215984936325879,
+            -0.019349418284688353,
+            -0.02322660633029795,
+            0.009747820845194852,
+            0.007353771220903641,
+            0.015252908783199839,
+            -0.012176175829980202,
+            0.051898051094966605,
+            -0.0068130353886146165,
+            0.008189212816890595,
+            0.008102287831931905,
+            0.01682801136011614,
+            0.02005557996049711,
+            -0.020779879580497183,
+            0.002332165268329781,
+            0.02337485065616377,
+            0.006074992895540845,
+            0.02576575146535689,
+            0.0007517286429767504,
+            -0.03763909942314134,
+            0.0011204729720314833,
+            0.00041212613589632806,
+            -0.06089176800314795,
+            0.012852035873395388,
+            -0.008901127092058543,
+            0.01702347055484216,
+            0.005803446395466281,
+            0.017889570275475296,
+            -0.05375047382413552,
+            0.021623778774252724,
+            0.014784317465883424,
+            -0.0018881880857885708,
+            0.0004735328554290247,
+            0.012393802850407656,
+            -0.019234442968055952,
+            -0.019566283611638122,
+            0.04407789373296535,
+            -0.0053565873209102995,
+            -0.026240866282480922,
+            0.048107902270426675,
+            -0.0054681314939387235,
+            -0.0009441397948041965,
+            -0.015205079626883548,
+            0.006885981841893229,
+            0.01403724256866746,
+            -0.026086548246792754,
+            0.0034778946051893067,
+            0.00044587533948775473,
+            0.032851328435891634,
+            -0.03882097333255334,
+            -0.011126344604254894,
+            -0.01648943336384904,
+            0.035810456555802166,
+            -0.019384589526165754,
+            -0.019430702768170285,
+            0.014297440272239852,
+            -0.007883453204663151,
+            0.050996900793086844,
+            0.06007149434153715,
+            0.03406832053483379,
+            0.025193439934671852,
+            0.00027785129663604897,
+            -0.008014391056168083,
+            -0.018806730908490114,
+            0.027193788903098558,
+            -0.006195006670328595,
+            -0.020466000344352826,
+            0.0068939884219589185,
+            0.017252276167086602,
+            -0.045073444648704136,
+            -0.049242269917908754,
+            0.023021140866008313,
+            -0.015676872472071106,
+            -0.024497517908898602,
+            0.005873633718973869,
+            -0.022828585938203683,
+            0.011748341004940436,
+            0.04624205727927451,
+            -0.015442165499674641,
+            0.018565551071491767,
+            0.015246505395507096,
+            0.004119269796224144,
+            0.004862735250819252,
+            -0.000145323321960046,
+            0.05155838952887077,
+            0.0027052423345323936,
+            -0.016218063670586664,
+            -0.07462470920944148,
+            0.043695974746295135,
+            -0.027608273720377708,
+            -0.03860912774695494,
+            0.002823110565150435,
+            0.026471469849508766,
+            -0.0026063577189074103,
+            -0.029230652014151735,
+            0.05684164470477843,
+            0.00022182868300572677,
+            0.015590845552956364,
+            -0.024939617549529242,
+            0.01035803286203917,
+            -0.010225298012384327,
+            0.019571847648759157,
+            0.01841613041441054,
+            0.0156981470538354,
+            0.021764297694793343,
+            -0.018831230007033416,
+            -0.014419529638789993,
+            -0.0029965475899652273,
+            0.028881185012095884,
+            0.09163526460163937,
+            0.022644302207811818,
+            -0.017933909936330133,
+            -0.04663928709016869,
+            0.004962951444105128,
+            -0.041333041174526026,
+            -0.027545628007617527,
+            -0.037142394579894954,
+            -0.013358143162788402,
+            0.02719686432396677,
+            -0.0034971800937504205,
+            0.04413635817371732,
+            0.04903857110328917,
+            0.03186067155630788,
+            0.007831051050269414,
+            -0.025257538623313574,
+            0.023517513773109414,
+            -0.0016527403209505964,
+            -0.0016101860707429347,
+            0.014448515185903624,
+            0.009905018312477684,
+            0.0024114582235163955,
+            0.00172043851384822,
+            -0.03818316221102173,
+            -0.02304537008110578,
+            0.01024969233570089,
+            -0.01901116356327339,
+            0.019791121460848986,
+            0.07985067340821365,
+            -0.051903191272236665,
+            -0.025924646976556942,
+            0.014766511532849252,
+            -0.002073086076894999,
+            0.030959269926800852,
+            0.006394417986617089,
+            -0.028177586693713442,
+            0.010725285336674526,
+            0.0009614052276924192,
+            0.07500253788978564,
+            -0.00036794182149960157,
+            0.004058458332826519,
+            -0.013218269166951234,
+            0.027613539268683245,
+            -0.012995170442480582,
+            0.023388707018488407,
+            0.026389423313105165,
+            -0.003068639956061266,
+            -0.02661967649632026,
+            0.026145274570635913,
+            -0.03300591322174331,
+            -0.004556072066733024,
+            0.020045722084112896,
+            0.0033437343914784865,
+            -0.010224220905483085,
+            -0.0025025498684079703,
+            0.03460103245395426,
+            0.04514362357278504,
+            0.03797031988058962,
+            -0.004586662475576956,
+            -0.06351723722596923,
+            0.06037983484369874,
+            0.019415753124363902,
+            0.028850618647994476,
+            -0.0034810237045849006,
+            0.02957938154808314,
+            0.02642652523231673,
+            0.013187653724133275,
+            -0.007856320804109569,
+            -0.007065882659482447,
+            -0.016672784625197085,
+            0.029020161154209707,
+            -0.013243058921970134,
+            -0.014900932834241732,
+            0.006001769901896033,
+            0.008648456186944271,
+            0.03484763413464497,
+            0.012279368984306564,
+            0.02379643508687133,
+            0.04933044959399941,
+            -0.021047723935647383,
+            -0.007264823701706426,
+            0.0007058774795716792,
+            -0.010512071832195085,
+            0.027489671668749157,
+            0.009731157570361245,
+            -0.03195919640758579,
+            -0.0033626031972691915,
+            -0.04282419484651207,
+            0.01873977910485268,
+            -0.014398285316695653,
+            0.012854631447494207,
+            -0.0017372959978092432,
+            -0.037331867825735776,
+            0.001088209292238142,
+            -0.012345684323522151,
+            0.037096063326414734,
+            0.005050219690024652,
+            0.0008906051515353004,
+            -0.003697709449630797,
+            0.006548479142737107,
+            -0.010727663712299397,
+            -0.007486226350610755,
+            -0.02704618481069485,
+            -0.019752552016312926,
+            -0.014632581296303486,
+            -0.0037501138119724717,
+            0.014588411795703245,
+            -0.007119137071719376,
+            0.016440258608998535,
+            -0.03686645352111013,
+            -0.023546402448713415,
+            0.005628703432152666,
+            0.024988145751166143,
+            0.011971199595808573,
+            -0.010868778329190568,
+            -0.03270046433624425,
+            -0.021950559285357267,
+            -0.011889914022657738,
+            -0.005123971523701009,
+            0.02619713302280345,
+            -0.021990810305904974,
+            0.01165257474485436,
+            -0.009117901804315863,
+            -0.007492082755249399,
+            0.010725902070291232,
+            0.01862811996181951,
+            -0.026694281868322685,
+            -0.01623776415010199,
+            -0.009403997538684658,
+            -0.02207329027097839,
+            0.025939540497196416,
+            0.014178803271471762,
+            -0.001141821318145744,
+            -0.006281187017652108,
+            -0.035993365198554964,
+            0.0026192776111449633,
+            -0.015254976771874396,
+            -0.0116872453009606,
+            -0.015780016491993135,
+            0.021421328283124968,
+            0.02889432250124136,
+            -0.01871009416296692,
+            -0.026540472640514536,
+            0.015703370682110186,
+            -0.0002849235245164438,
+            -0.02594930889995505,
+            0.04491828744809679,
+            0.044063522197410734,
+            -0.037441864491547405,
+            -0.007005339187980476,
+            -0.030381794393931814,
+            0.06901083845539985,
+            -0.017604082400603037,
+            -0.04060591381445593,
+            0.06044850935418443,
+            -0.01756490171206357,
+            0.025539717592705295,
+            0.03452018789549939,
+            0.0016217577522523927,
+            0.00018825054264059277,
+            -0.002563606092068712,
+            0.04770277039047993,
+            -0.0032010143255574255,
+            -0.03425328601475849,
+            -0.02105909767072799,
+            -0.00840881022563103,
+            -0.028699447677549002,
+            -0.015925160861589006,
+            -0.0055750742281166,
+            0.017513987082479483,
+            0.010399718005952185,
+            0.02267201252273823,
+            0.01258511878601025,
+            0.0015081978484714903,
+            0.01874981572091931,
+            0.011158889421057756,
+            -0.002607810686437211,
+            -0.003732023411827874,
+            -0.007787189972165309,
+            -0.05524012485122728,
+            -0.025110689689279583,
+            -0.03946383236703947,
+            0.004443572529325475,
+            -0.018745077598477586,
+            0.005926160888274727,
+            0.039430331582190524,
+            -0.0049500324516682,
+            0.004352795787899792,
+            0.015566139991780173,
+            0.013148156356408315,
+            -0.015279560337042242,
+            -0.03316846614871165,
+            0.0001592209771257332,
+            0.018362411161743347,
+            -0.012706582418607377,
+            -0.03424477956681259,
+            -0.001991022841226589,
+            0.0033467286047492175,
+            0.03357019284647071,
+            -0.024359110796680374,
+            0.00359117029951049,
+            0.005061818038334802,
+            0.011359582931760315,
+            -0.026793992746244076,
+            -0.018170136079817746,
+            -0.017421603453826397,
+            0.03996608926471779,
+            0.02587013058343799,
+            0.015763410145648683,
+            -0.04512536449983358,
+            -0.02293473890636517,
+            0.048575083783480195,
+            -0.009589860756342604,
+            -0.06486262416944208,
+            0.03447666009581266,
+            0.016105654870253654,
+            0.003526582643722421,
+            -0.01354754623151814,
+            0.05401575248587799,
+            -0.03563190295559527,
+            -0.017365166993914202,
+            -0.0046288709698227735,
+            -0.033657212157212645,
+            -0.007686252059880836,
+            -0.001385273642139782,
+            0.048229062789530236,
+            -0.029083855744728694,
+            -0.030465559177859865,
+            0.0024368661249980416,
+            -0.024144031154493057,
+            0.021693278579520123,
+            -0.016405088055634124,
+            -0.018439935250344958,
+            0.006948603359995389,
+            -0.033943037395199695,
+            -0.003007796423804453,
+            -0.016783195426536633,
+            -0.02270589491206155,
+            0.020874438753828814,
+            0.016724666903417947,
+            0.027033184180344852,
+            0.0019847647852695785,
+            0.017749524526371133,
+            0.0204661893602285,
+            -0.005604929968008821,
+            0.028462234633072944,
+            -0.020790459496727066,
+            -0.04183816355941877,
+            0.03946487892663695,
+            0.01192861758684541,
+            -0.0018024841275380242,
+            0.01077082591611702,
+            -0.01475239020295694,
+            -0.03952356245715099,
+            0.03683950018557526,
+            -0.056767672022551764,
+            -0.007507600090017693,
+            0.009530544352246083,
+            -0.00788856051420074,
+            -0.011204694027562521,
+            -0.010587701466318598,
+            -0.028604940377315504,
+            0.023749537635431462,
+            -0.013672419168235691,
+            0.01953449229707597,
+            -0.011449939491554354,
+            -0.012238180529977605,
+            0.05795536247491174,
+            -0.021518581774137322,
+            -0.040672955795925836,
+            0.026066659238628543,
+            -0.014061106927896723,
+            0.03328357230532424,
+            0.03957107406164427,
+            0.030670947051333917,
+            -0.004278577327978564,
+            0.025885946540364826,
+            0.0319106101668716,
+            -0.003462160036198642,
+            -0.04686868330851868,
+            0.00555303933159587,
+            0.02156143971867743,
+            -0.03739529489075782,
+            -0.01025626126223168,
+            -0.02141970191890338,
+            0.022134910980548407,
+            0.017807484142010496,
+            -0.007232643256393455,
+            0.011318041192005274,
+            -0.010489012862455913,
+            -0.013003217163997554,
+            -0.00847279880392288,
+            0.02357274819910594,
+            -0.0016485066139744577,
+            -0.005873113634992993,
+            0.006424249429163587,
+            0.004008703814301658,
+            0.0054639858216457724,
+            0.010198079500408441,
+            0.017694720520798957,
+            -0.017087021465208255,
+            -0.0025558869389290554,
+            -0.016503579212015177,
+            -0.0420082404922411,
+            -0.003563173424359332,
+            0.022924161700566468,
+            0.012345933179990547,
+            -0.00729908668365632,
+            -0.004234683369642856,
+            -0.0022962545569187934,
+            -0.020666205489143392,
+            -0.014796149043465308,
+            0.024562653223873056,
+            0.030217494971086438,
+            0.007810961433007407,
+            0.007781263640449456,
+            -0.0030348559388780655,
+            0.007310876971216526,
+            0.016099591640649783,
+            0.02311929597780174,
+            -0.04160162038353416,
+            0.027152100845897007,
+            -0.00025396315502635146,
+            0.023717123370023598,
+            -0.04186918759097667,
+            -0.017988531857836703,
+            -0.03106663560912813,
+            -0.023595079699181436,
+            0.02437134274097885,
+            -0.02317559303912261,
+            -0.012327263775257306,
+            0.000989566600374836,
+            0.00954921216572729,
+            -0.005155079395685017,
+            -0.017508404061379164,
+            -0.04582321461886958,
+            -0.0014367264231457263,
+            -0.030417229142190933,
+            -0.017829710557332023,
+            -0.001056472801554354,
+            0.005016912084754242,
+            -0.06276644565618253,
+            0.0109965681725148,
+            -0.07270669271977365,
+            -0.004542796125637244,
+            0.040139998232966566,
+            -0.017890441783630643,
+            0.055909102081208445,
+            -0.05029104308417623,
+            0.01673962668706085,
+            -0.04507293803055318,
+            0.015385506084493484,
+            0.025677372277539627,
+            0.030235171586316813,
+            -0.022189978375852563,
+            0.008461188938299745,
+            0.008033707483259569,
+            -0.045089505755429415,
+            -0.04144698743458133,
+            -0.013640256640978024,
+            0.01036782197518379,
+            0.0012696958402094232,
+            -0.016090728932978384,
+            0.023838437877040163,
+            0.011946482002767135,
+            0.03342604921859589,
+            0.04211801335157251,
+            0.002708015395637621,
+            -0.047156646610976634,
+            -0.013104464812562618,
+            -0.003933116891120681,
+            0.03155321708116329,
+            -0.018866109579446184,
+            0.015376812010630058,
+            -0.01955557376085726,
+            0.016255217383093355,
+            0.017868332102274922,
+            -0.020325179506828335,
+            0.013434364256590097,
+            0.004176904701252156,
+            -0.03877060944639854,
+            0.0006475542749369515,
+            0.00042468300294954125,
+            -0.02116992978038693,
+            0.0032654387271108293,
+            -0.00575003786159731,
+            -0.05149954071159492,
+            -0.031382926943128234,
+            -0.015730563640253065,
+            0.0382312502769352,
+            0.004297939848910064,
+            0.019724983545557365,
+            -0.017907869879607064,
+            -0.005165628493894463,
+            0.03698976707579373,
+            0.008501001373266799,
+            -0.01725527821298066,
+            0.006879620077191753,
+            0.02455792200929138,
+            -0.03224193972301503,
+            -0.039181328884413266,
+            -0.0008206986106407542,
+            -0.023306758893533952,
+            0.01672946960876128,
+            -0.02658587698869307,
+            0.005873620623736886,
+            0.0025331170979801232,
+            -0.03490809296591325,
+            0.02456764314036355,
+            0.003291113462353163,
+            -0.004303423672882112,
+            -0.020359544091059775,
+            -0.016892415165120203,
+            0.017174797487220006,
+            0.04391489511478065,
+            0.007327321684948843,
+            -0.003943713832544148,
+            0.00862684231313155,
+            -0.006361172785753631,
+            0.05826688726534012,
+            -0.02284956969636637,
+            -0.026680023999604643,
+            0.027527436347931256,
+            -0.022597669279069723,
+            -0.006277863878949772,
+            -0.017343446085471857,
+            -0.012065192152203348,
+            -0.0004183712490442143,
+            0.019339311388446943,
+            -0.019423562920897294,
+            -0.010212527447243049,
+            0.004783006556748757,
+            0.012824133870627983,
+            -0.004855945108275706,
+            0.006239949458562236,
+            0.006847581690471813,
+            0.03094005096378827,
+            0.026993131409761086,
+            -0.011331881651539968,
+            0.03297014291876675,
+            0.010265600643742144,
+            4.666113318251278e-05,
+            0.001970879281172325,
+            0.002523664335535017,
+            0.0034107811520368574,
+            -0.03143890864030425,
+            -0.01624584278940617,
+            0.013468729048885906,
+            0.005233309338223403,
+            -0.004550544967856448,
+            -0.023783653541634683,
+            -0.009402494239255264,
+            0.03334923712760375,
+            -0.003260424721959948,
+            0.02748395114281811,
+            -0.0058374897745283065,
+            -0.01985840930273332,
+            0.017595418271610163,
+            0.02881743246631328,
+            -0.05928552440605384,
+            0.017062540894682354,
+            -0.0011508547522925793,
+            0.008953761538519997,
+            0.01610561914042553,
+            -0.008894160513648953,
+            0.013301991014533684,
+            0.04080863883897929,
+            0.010738051773755976,
+            -0.023345002072886145,
+            0.004215726310838674,
+            0.01541749448570608,
+            0.01291813299324752,
+            -0.022747942296528025,
+            0.0008640791861357522,
+            0.011008522066277018,
+            0.005893793007012561,
+            0.0026106450193140156,
+            -0.006278394080583205,
+            -0.025659188287146268,
+            0.024180550333596625,
+            -0.0044491292740074925,
+            0.026967973346698617,
+            0.04390909273429254,
+            0.009596855239692042,
+            -0.03686020608700941,
+            -0.013109951664921729,
+            0.0002799680359670127,
+            -0.0038537577602593366,
+            -0.0037552094562213916,
+            0.011105295862811114,
+            -0.04185615640149602,
+            0.019950890919369135,
+            0.006453241355231516,
+            0.015314645427201533,
+            -0.020031193483734364,
+            -0.024530173488182337,
+            0.0031949828520724337,
+            -0.01492645564652531,
+            0.012875827757040941,
+            0.005057170413681063,
+            -0.004319834710426384,
+            0.005386862713494232,
+            0.00030848054013413157,
+            0.008384941028326983,
+            -0.010811526993301901,
+            0.03175068297908541,
+            0.004680987487229685,
+            0.009264547472281849,
+            -0.02856314241700808,
+            -0.021083721348429348,
+            0.020113396771629096,
+            0.03572366771051171,
+            -0.01708529918264109,
+            0.03346921724975398,
+            -0.009042046660366718,
+            0.0033219155387035227,
+            0.0151720168318272,
+            -0.015805318210244414,
+            -0.008057236707694303,
+            -0.0005483009439015931,
+            -0.002917527004423299,
+            -0.014538511124345527,
+            0.007302979537905532,
+            -0.0208529958115422,
+            -0.02965227242147972,
+            0.020064178921373875,
+            -0.00011892622932349096,
+            0.016455188521316146,
+            0.010335044657747514,
+            -0.007816155992852236,
+            -0.0052262330850905165,
+            0.006959344748883874,
+            0.04432147882994085,
+            0.026734484553994495,
+            0.05547661378166294,
+            -0.005787997952888261,
+            -0.004559444854138438,
+            -0.0082295581123363,
+            -0.006588172755707144,
+            0.04524009609473585,
+            0.015915003261638767,
+            -0.004910285095602864,
+            0.017495607570191165,
+            0.01557886571266685,
+            0.03858259260136037,
+            0.0027648525902558316,
+            -0.016912796336835408,
+            0.04014755669075221,
+            -0.004077405782937414,
+            0.008616827127381418,
+            0.011102692199778469,
+            0.0024185742218798562,
+            0.00807141794538637,
+            0.02996274720815659,
+            -0.012429186019907761,
+            0.01680942269506771,
+            -0.04571916592313758,
+            -0.02775530246832996,
+            0.014370280587419414,
+            -0.004931905540824582,
+            0.03419959665162407,
+            0.01668521021731861,
+            -0.023105305446763272,
+            -0.024761385385514877,
+            0.021529428247380984,
+            -0.002603042335561961,
+            -0.024135326707477325,
+            -0.015906833215844473,
+            0.003286605488908195,
+            -0.0023437997044735617,
+            0.0060251848371355166,
+            -0.013481652076194643,
+            0.0002908426939460218,
+            0.01953362396253743,
+            0.0038544889405965057,
+            0.020760460182873287,
+            -0.016224396121239458,
+            -0.00259233446682179,
+            -0.009282861237653619,
+            -0.02952130765234087,
+            0.017988370145301916,
+            0.006138236653154911,
+            -0.007607182955958238,
+            -0.009516767050028553,
+            0.0048347778733230365,
+            -0.005819209642901871,
+            -0.0025851701689263706,
+            -0.01073014664868326,
+            0.012433465684360061,
+            0.025784014942833097,
+            -0.016875624785552303,
+            0.007944349714123104,
+            8.651589872318479e-05,
+            -0.022306044741692753,
+            -0.022127016745066307,
+            0.020143472307319017,
+            0.002792646249762062,
+            0.008768813346304779,
+            -0.004074214276300631,
+            -0.00015024341785520542,
+            0.002990550758627699,
+            0.01951746553838439,
+            0.03896451381932467,
+            0.004232673242619471,
+            0.027876586543280867,
+            0.0145860303663199,
+            0.03994234200854897,
+            0.009657017763624678,
+            0.00696607158576712,
+            -0.026055387694656488,
+            0.02819890357011778,
+            0.00018297502510753014,
+            0.043190622821775236,
+            0.022488402221902314,
+            -0.0407730101161195,
+            -0.0021946518523759066,
+            0.01669520746276111,
+            0.013212623257794862,
+            0.017009844136059122,
+            -0.004537512994041156,
+            -0.014041213424740049,
+            0.028135404528975894,
+            -0.022859586430516003,
+            0.0008637006862481223,
+            -0.004486332934188017,
+            -0.025031795510001887,
+            0.007351370254175368,
+            -0.004567816916440959,
+            0.030025413528233364,
+            0.0014976740344892735,
+            -0.022285096307825804,
+            0.022837170656498543,
+            -0.009551269792616045,
+            0.011045530488684207,
+            0.008809047185288755,
+            0.020065001354321994,
+            0.031993362733068575,
+            0.009143210434415064,
+            0.02133640814803879,
+            0.012241777474889248,
+            0.014771538127546474,
+            -0.008804192497858033,
+            -0.02341427260922065,
+            0.023167424359504862,
+            -0.0011598649565273595,
+            -0.029627604902846688,
+            0.0021991474725787303,
+            0.003261878079344105,
+            -0.0005935713331682358,
+            -0.009144689470851402,
+            -0.005580557862988033,
+            -0.007240836451875213,
+            0.019631377011298905,
+            -0.007085213886352692,
+            0.007017573363709754,
+            -0.004074171791476129,
+            -0.00822871835665378,
+            -0.0421101573155962,
+            0.004210914075286222,
+            0.003408427700622687,
+            -0.01752179330665445,
+            0.006384440404174512,
+            0.011121481025573962,
+            0.043020575455262985,
+            -0.022809979913698254,
+            -0.0005737138409049003,
+            0.019206623212867476,
+            0.0009562551889820241,
+            -0.023427467810802533,
+            -0.038931688376706186,
+            0.007172345672967499,
+            -0.03467050750542982,
+            0.023185301620612612,
+            -0.0007075596339842658,
+            0.02387977204565927,
+            -0.03043468414449851,
+            0.013909304572370437,
+            0.004784668158421625,
+            0.012524879174804915,
+            0.021654512058639427,
+            0.03142146922870715,
+            -0.02053473475540882,
+            -0.050409862638621725,
+            -0.005406667316025549,
+            0.00035138762584085976,
+            0.022018487016014856,
+            -0.026014580970974616,
+            0.00025633004699302783,
+            -0.003349094027291921,
+            -0.02401366769212778,
+            -0.01735996745614562,
+            0.01962340791446025,
+            -0.050011480783403936,
+            0.015042913576492952,
+            0.0262737424555697,
+            -0.02507473901367252,
+            -0.05408889321034101,
+            -0.018988587082325706,
+            -0.028217893770659686,
+            -0.009285770094319318,
+            -0.00661159230987788,
+            0.028597405154740078,
+            0.003525576256054959,
+            -0.011462219358952466,
+            0.025401668390029348,
+            0.006155837251830549,
+            0.0026050693705924593,
+            0.015985722392086222,
+            0.007289910674212967,
+            -0.011452511453623235,
+            -0.014599577191293152,
+            -0.04558584556100621,
+            0.010197536827229795,
+            -0.0045431962637690155,
+            -0.019372598050508005,
+            -0.030147301333718016,
+            -0.0137828704002156,
+            -0.027060515748528666,
+            -0.0017291966950107505,
+            0.053783684195381654,
+            0.03292494290301388,
+            0.005560172296221419,
+            -0.0026085670565688026,
+            -0.010466890489356092,
+            0.036247483457995895,
+            -0.006789746401727701,
+            0.03662690472838588,
+            0.001784522073210422,
+            -0.0242673253380092,
+            0.01169382884548301,
+            0.02892627181542449,
+            0.009858451662166387,
+            -0.0005592283645355213,
+            -0.006160132931009508,
+            -0.008782830470079547,
+            0.0031207630972208394,
+            0.025613233596604418,
+            0.003268170392795548,
+            -0.010062991425127038,
+            0.018106967407336815,
+            0.004480291699976396,
+            0.02010374270557836,
+            0.03555357587371324,
+            0.023489837757983054,
+            -0.021782173652026497,
+            -0.026581762796336082,
+            -0.0013561486781077014,
+            0.0072266106510934505,
+            0.03551967295942857,
+            -0.01651315514034218,
+            0.0001274396319698452,
+            -0.008272735507120127,
+            -0.00894931703949637,
+            0.011149952898504355,
+            -0.0029671593121986116,
+            -0.0030752010285951576,
+            0.01662711184497578,
+            -0.0011672263615879709,
+            -0.033448222499147764,
+            0.0078288560210653,
+            -0.023496331346917084,
+            0.0051040606404415545,
+            -0.004113104164416575,
+            -0.007076525381595441,
+            -0.002157548492138289,
+            0.006302962169276006,
+            -0.008065841090139872,
+            -0.00035242603988955366,
+            -0.01586406637763306,
+            0.008271894439780969,
+            0.007438312999409211,
+            -0.0069617481460406095,
+            0.011402932583388205,
+            0.002666895909532698,
+            0.0005219881149353855,
+            -0.012159425670629751,
+            -0.012042145306711322,
+            -0.00957276588761916,
+            -0.03401863879157551,
+            -0.02604353712472079,
+            -0.024949533021906844,
+            -0.0014209149651479556,
+            -0.006566685925666972,
+            -0.03187796739029736,
+            -0.01209152137926458,
+            -0.011110763124407752,
+            0.007756961097537267,
+            -0.013260548545917543,
+            0.005433285079114047,
+            -6.399098520041351e-05,
+            -0.0044960488826674555,
+            0.016615636438440516,
+            -0.003643291599919672,
+            -0.019488010211707343,
+            -0.03927818878149417,
+            0.008209336177358258,
+            -0.02805593125673831,
+            -0.0038816771550190788,
+            0.007607119219721711,
+            -0.00528681677614909,
+            -0.029502897838743516,
+            -0.009149794737430706,
+            0.008767001199311985,
+            0.003507217008280066,
+            -0.01773349756494436,
+            0.03579670773605949,
+            0.026971617790626075,
+            0.011875179879209696,
+            -0.019891342676443626,
+            0.0508929813275565,
+            -0.0047010735337056275,
+            -0.012544935684845649,
+            0.018297503869683027,
+            0.02355022268131519,
+            0.03273167043015095,
+            -0.024830243320788693,
+            0.0009370814985214882,
+            -0.021937793653233454,
+            0.00535662615596573,
+            -0.035009915581788176,
+            0.0028595659314725012,
+            -0.014395027018699446,
+            0.015596459769560532,
+            0.03397372881171701,
+            -0.013254226072981457,
+            0.011195747181764677,
+            -0.016299384375589934,
+            0.00730244731485402,
+            0.00569944090461748,
+            -0.012916284280345044,
+            -0.005812311882399539,
+            -0.022902815152116493,
+            0.020140276669979015,
+            0.005809799950176385,
+            -0.0008130655714656685,
+            0.0104085826652653,
+            -0.044723432238900435,
+            0.03286429597337671,
+            0.008730225944501239,
+            0.017599189944921396,
+            0.0029762446259779998,
+            -0.0003068671951232415,
+            0.011611551606224214,
+            0.011228401175698422,
+            -0.016029727902740706,
+            0.004345913009102136,
+            -0.004613061137378433,
+            -0.03850533909474302,
+            0.009363351681220008,
+            -0.03631184057978666,
+            0.014427784677340159,
+            0.016026872002233575,
+            -0.004170582842643461,
+            -0.025166189571409366,
+            -0.005428471012333673,
+            0.010827910811085384,
+            -0.009783069664327864,
+            -0.009987805234276443,
+            -0.04163358183060102,
+            -0.025236440364963034,
+            0.034097508113604576,
+            -0.002119474613596975,
+            -0.025826403511707147,
+            -0.010830738112107807,
+            -0.011351132636491575,
+            0.020207687645133,
+            -0.006542026283946858,
+            -0.0006263975804936843,
+            -0.04150011545311013,
+            -0.0073396962917864515,
+            0.011192225907726928,
+            0.027746213794734322,
+            -0.035970017302005566,
+            0.02708126943501373,
+            0.002105633542220187,
+            0.014262674845755191,
+            -0.01626664983003942,
+            -0.005631805361903626,
+            -0.013990539604372344,
+            -0.035877093361375685,
+            0.016487723013906077,
+            -0.008607758248055054,
+            0.004714436224135091,
+            0.04458009880881818,
+            -0.0036872068506712727,
+            -0.009466125348499464,
+            0.007600705817195143,
+            -0.0034590883553048063,
+            -0.010351576085287273,
+            -0.0028564505793999295,
+            0.001959661298507,
+            -0.007080697863126256,
+            -0.04183476693971616,
+            -0.00499888976530217,
+            0.04684391782633148,
+            0.006361744381879857,
+            0.013276793765092465,
+            0.013516220521120684,
+            -0.018335746557932003,
+            -0.045577810515431215,
+            -0.0071616111758423025,
+            -0.0032763880176386808,
+            -0.022708028392758567,
+            -0.0240284945459861,
+            -0.008735040856657758,
+            0.005844145402902923,
+            0.006359368165686724,
+            -0.01120763599008222,
+            -0.022676131361171057,
+            -0.027085968700452136,
+            -0.010526103176253857,
+            -0.02123944716136792,
+            -0.010792145796733375,
+            -0.027142270190771887,
+            -0.028742234538796804,
+            0.02880426953386151,
+            0.007575182232859564,
+            -0.004084088301176117,
+            -0.010005765476121966,
+            -0.01255640546813973,
+            -0.0004774761617900028,
+            -0.0025307994917461232,
+            -0.005822065709999144,
+            0.0038614286313282164,
+            -0.011390482620811868,
+            0.023152086963976554,
+            0.0021220701260546895,
+            0.018329162172128138,
+            -0.05772324571231113,
+            -0.002709083851090071,
+            0.021497896395795905,
+            0.0033025410488410913,
+            -0.005412445141910314,
+            0.029352650459792132,
+            0.011223947499453582,
+            -0.006539460282439711,
+            -0.031355428613718234,
+            0.020942976176820262,
+            0.024122993515785947,
+            0.00817615634503138,
+            -0.012994160510030828,
+            0.01221095914384204,
+            0.01629778384790159,
+            -0.004494985018246821,
+            -0.011161098655072933,
+            0.005586857967039181,
+            0.020941039642372112,
+            0.013580622629911925,
+            0.0013398071867077221,
+            0.030905022925366163,
+            0.0006862084391508121,
+            -0.03521231612862609,
+            -0.01908191995763771,
+            -0.01275995164030872,
+            -0.003584331305986364,
+            -0.015264425022763126,
+            0.005294329654809353,
+            -0.002355465010876059,
+            0.006335052149475905,
+            0.022655379849785256,
+            0.03593231939948758,
+            0.0146937743203089,
+            0.00937330328448886,
+            0.003167462724350819,
+            -0.0013235176101076284,
+            0.012102872918129403,
+            0.008638011773075119,
+            0.0350701240268404,
+            -0.014653221697693864,
+            -0.011003994230902835,
+            -0.0037767288519369537,
+            0.010100131061186566,
+            -0.004113527518662366,
+            0.020414393256136372,
+            0.03274709615649469,
+            -0.0037222949335916876,
+            -0.005604552180246868,
+            -0.006160118231975786,
+            0.0041467454414452565,
+            0.01807879127455274,
+            -0.01252134965321004,
+            0.004364858987501461,
+            0.005549300665159658,
+            -0.0071897162833861,
+            0.018567059255813558,
+            0.0002622074720860951,
+            -0.012983550838417323,
+            0.03266376564402191,
+            0.11449509387240629,
+            0.0017251916835047628,
+            0.007125215166916139,
+            -0.040229997815548584,
+            -0.020537987527148888,
+            -0.00245728735683473,
+            0.005779972828836593,
+            0.007004732428236457,
+            0.02225914982152608,
+            0.006505827715591914,
+            0.0006708045351142873,
+            0.031708522931495536,
+            -0.017334083511709092,
+            -0.002522861743921599,
+            -0.01745656559688129,
+            0.013890451996979048,
+            -0.014312605700272691,
+            -0.013784963572164409,
+            -0.01572417853955075,
+            0.004507425871748977,
+            -0.000601158156253668,
+            0.0016158265715634729,
+            0.01791303234734911,
+            0.0011926917387973374,
+            -0.026393351252156246,
+            0.0006694008039871878,
+            0.001026031515214626,
+            -0.004514419527575117,
+            -0.011320929719992686,
+            -0.02223001723942815,
+            0.016235841442653918,
+            0.014188311619681008,
+            -0.004895218832118094,
+            0.009244697376301004,
+            0.010558796153959677,
+            0.02707485418319703,
+            -0.028380342401606758,
+            0.0040204430418629405,
+            0.007023041225985294,
+            0.01606556581447933,
+            0.017617862031190843,
+            0.0034870847111552862,
+            -0.011253315705053224,
+            0.03238289399766116,
+            -0.010563282687352038,
+            0.0065093536270233385,
+            0.014966802208644462,
+            -0.03541611540945151,
+            -5.821682910348165e-05,
+            0.0061985444938402745,
+            -0.011716201661363997,
+            0.009580035961315467,
+            -0.03120692534097433,
+            -0.0039291243987978845,
+            0.009317468331974374,
+            0.017611312558635673,
+            0.008804996982574765,
+            -0.0034129169537151925,
+            0.010822226698822738,
+            0.012510661458054741,
+            -0.0027147975086409966,
+            -0.008855213644027594,
+            0.01238502686635956,
+            -0.0007687053446954701,
+            0.00450108863711973,
+            -0.00799170686059026,
+            0.0029755595812552293,
+            0.002290650917244691,
+            -0.01554856300008412,
+            -0.011779430686394945,
+            0.00438393126679289,
+            0.0036931891075561957,
+            0.001535662168147833,
+            0.012014408636737167,
+            -0.006087255045485002,
+            -0.0077816220325470226,
+            0.0040573724156794284,
+            0.003136943055301706,
+            -5.505014316363865e-05,
+            -0.030625245024561508,
+            0.01585923084309149,
+            0.0067595435845396405,
+            0.010013308598626031,
+            -0.005315626260228517,
+            -0.00903142414091241,
+            -0.007240920015018893,
+            -0.01228521172101564,
+            -0.010657894223173453,
+            0.007658605592936815,
+            -0.00724924993432376,
+            0.011124174211132038,
+            -0.03128027787357322,
+            -0.013988715435985316,
+            0.01231792993245938,
+            -0.01528161242159377,
+            0.0007675821323979293,
+            -0.008389914312201402,
+            0.0009417242487395995,
+            -0.005119304379761705,
+            0.007887436211143328,
+            -0.03731503169547854,
+            0.007878001990271541,
+            0.02456170736765724,
+            -0.00330924625934981,
+            -0.020259771366167557,
+            -0.009613930457931786,
+            0.0241314385136084,
+            -0.008267831641886344,
+            -0.01645888287387047,
+            0.008421304862524506,
+            0.010140813149093993,
+            -0.0006309118552875393,
+            0.0008237784055549769,
+            -0.003726984191590917,
+            0.01669125388798323,
+            0.02054777616420997,
+            0.041930770964107515,
+            -0.014789610432004559,
+            0.01092349703703152,
+            0.02866938974792608,
+            -0.03378576980722847,
+            0.014728113846952417,
+            0.02518409645155824,
+            0.014590531327650454,
+            -0.0004446956315190408,
+            0.001609777186372437,
+            0.0068939321280085475,
+            -0.014273492115311677,
+            0.009247676157551494,
+            0.013380104440744367,
+            -0.018407746260686045,
+            0.020444430509441124,
+            0.03140071554008003,
+            0.0025839998101985736,
+            0.01355205268903058,
+            -0.006268939830271642,
+            -0.013860692186849113,
+            0.014717364068116991,
+            0.031429975488060896,
+            0.0006919565473572909,
+            -0.018561500446336806,
+            0.019128244642842268,
+            0.006988095806264863,
+            -0.003678529269242034,
+            -0.0021993078588142664,
+            -0.021334919445200388,
+            -0.021003586791369228,
+            0.017304350704792804,
+            0.013508779408333964,
+            0.027794485858413365,
+            -0.010594702782952029,
+            -0.007818089330662161,
+            0.014293688127901867,
+            -0.022368568887004094,
+            0.015426482139881204,
+            -0.02160820059598751,
+            -0.00571499813703566,
+            -0.0077402210272634235,
+            0.03274360648629609,
+            0.001441304806200403,
+            0.02734331677589297,
+            0.016457027764064002,
+            -0.033609420134045206,
+            -0.029790535553657092,
+            -0.00712106584238484,
+            -0.02307933297713104,
+            0.010356561171963577,
+            -0.03901183838032092,
+            -0.00071735150189382,
+            0.008251646656432826,
+            -0.008079917288471352,
+            -0.0007087526632966007,
+            -0.006419191399519291,
+            0.013989344185652506,
+            0.013661633160336152,
+            -0.0228103143826663,
+            0.023644639659577426,
+            0.008343481940133684,
+            0.025998483906975782,
+            -0.02076055376006461,
+            0.019148568238437927,
+            -0.017705363646416224,
+            -0.02777313933338929,
+            0.014358130220942483,
+            0.023585731219788467,
+            0.01635364038245971,
+            -0.007122853965024528,
+            0.007110003583832299,
+            -0.03556761320536796,
+            0.027585457849964405,
+            -0.013064062623866305,
+            -0.012504161289952458,
+            -0.0032379276562094555,
+            -0.011931827532360686,
+            -0.002574161005800279,
+            0.0008533907465694845,
+            0.0034345232246102806,
+            -0.03131462673429805,
+            -0.0002450959360520896,
+            -0.03727416416766574,
+            -0.028944814604815072,
+            0.01360126708805471,
+            -0.00036361284498363273,
+            -0.0049097813612814616,
+            0.02933242705521636,
+            -0.026339136251046315,
+            -0.02693295650849956,
+            -0.009590247544093352,
+            0.02853096456887074,
+            -0.0220673798697511,
+            -0.004519050393279848,
+            0.0015034014871316069,
+            -0.01828226786700525,
+            0.004436383319946364,
+            -0.0012748481042235446,
+            -0.010416065158876917,
+            -0.01105146139270597,
+            -0.03229479099927737,
+            -0.005822133182096844,
+            0.02106215163620738,
+            -0.006957125617688484,
+            0.002746241228040847,
+            0.013312248331456934,
+            -0.010968629118224582,
+            -0.0029946900704643812,
+            0.0017875284441794235,
+            0.023304175718480834,
+            -0.021033941064173063,
+            0.007714334541806199,
+            0.005955442031111047,
+            0.0016405586860131207,
+            0.018406562522853148,
+            -0.0054922197063461825,
+            0.0037690571959124617,
+            0.0162126534419016,
+            -0.009969304596173302,
+            0.017959307638726878,
+            0.007691549802475525,
+            0.006681837561745192,
+            0.002775700618222622,
+            0.007712770283579111,
+            0.009752652973700739,
+            -0.013506020126078112,
+            0.016911141425375933,
+            0.022468715726349706,
+            0.00023233555209372766,
+            0.021066580799163234,
+            -0.02302516292580905,
+            -0.028618544628453015,
+            0.018297878810036087,
+            -0.0225078941299981,
+            -0.012735208680311652,
+            0.016459925496937065,
+            -0.0043032724578332855,
+            -0.00797609922711709,
+            -0.014986057073351055,
+            0.02260618561608301,
+            0.02132930921464508,
+            -0.020249788476562527,
+            -0.012385936327068884,
+            0.0072304431443959505,
+            -0.004373475447051833,
+            0.007086656081865733,
+            0.03195810137269375,
+            -0.015982210897392764,
+            0.006818890093901413,
+            -0.0038306138017521626,
+            0.0030753642449792215,
+            0.017830546304667783,
+            0.009470018737705222,
+            -0.0019815241970236146,
+            0.030100238416495066,
+            -0.001330614013444443,
+            0.016392226282459177,
+            0.0004888028477871524,
+            -0.010948833692356176,
+            0.006343890495626752,
+            -0.026652806306173643,
+            -0.008627570055007363,
+            -0.001203530223345183,
+            -0.009581494451831351,
+            0.07598616930784373,
+            0.003930153552069478,
+            -0.01295957554290434,
+            -0.001673007115291578,
+            -0.001368287102181597,
+            -0.006897179139643672,
+            -0.00960679451603541,
+            0.0287166325268306,
+            -0.016613613331623346,
+            -0.01894527592131826,
+            0.0008370014132002703,
+            -0.006785618888421608,
+            0.00919924251249592,
+            0.01816363936999796,
+            -0.010225961928154075,
+            -0.01604759761423727,
+            0.001577138242530474,
+            -0.003520564936698765,
+            -0.004395434431595494,
+            0.0052399043934902014,
+            -0.010127122101423486,
+            0.006456662777583096,
+            0.026560491731327886,
+            -0.013684536319659296,
+            -0.01425196172663986,
+            0.004084982328707292,
+            -0.0010682084059972982,
+            0.009758641913073978,
+            -0.00864879062601467,
+            -0.00481222050200678,
+            -0.014375330954464725,
+            -0.011378890549403524,
+            0.02505641780042516,
+            -0.02861164504722546,
+            0.0034060847953066354,
+            0.014937227531980433,
+            -0.00016904224334342915,
+            0.01541381851648916,
+            0.0033028855218525644,
+            -0.03615935765718852,
+            -0.013677774380513626,
+            -0.02073839691250734,
+            0.019821074228011413,
+            0.01688422012576042,
+            -0.0033731840788147442,
+            -0.013840993586441808,
+            -0.026149880178874887,
+            -0.007430563876740068,
+            -0.02466234818567362,
+            -0.020708282040996014,
+            -0.019276991448222817,
+            0.006137002799644838,
+            0.005162892426268572,
+            0.022881759407670583,
+            0.0025854880108157857,
+            -0.015058889448404231,
+            0.0029948608544278386,
+            -0.006722836436935071,
+            0.011845298212977357,
+            -0.02747944486818378,
+            0.015311272959213262,
+            -0.01610493464983465,
+            0.025239768880017402,
+            0.007425208902253236,
+            0.0010569008485112994,
+            -0.008822681490988575,
+            0.00010704167529046774,
+            0.014883787023445807,
+            -0.004186968199973372,
+            -0.0007781259002896265,
+            0.02787555889813697,
+            0.02087373813833492,
+            -0.0017783425135865238,
+            0.008743718448136432,
+            0.028026471206552342,
+            0.019081083283793984,
+            0.0325261942213039,
+            -0.013815526479760766,
+            0.005113676604826083,
+            -0.028873600114081546,
+            0.010510475843707614,
+            -0.010871039729515013,
+            -0.010201824683340651,
+            -0.0067946599549165735,
+            0.027756887504389347,
+            -0.007962601205446472,
+            -0.0006687976124826922,
+            0.03469894503563926,
+            -0.014185294847328063,
+            -0.00010543064621374546,
+            0.0007386418834840544,
+            -0.009759481309152999,
+            0.02742722730386853,
+            -0.0073590367291510046,
+            0.018456032721630843,
+            -0.019560882612141943,
+            -0.013705917411254969,
+            -0.011894560208199602,
+            -0.001142768182039817,
+            -0.02576754877141608,
+            -0.008755240066826174,
+            -0.02249287216087102,
+            0.00859172262856532,
+            -0.0046012388553094375,
+            0.018845920253446127,
+            0.0011617844393195566,
+            -0.005275799428018491,
+            0.029069367036565145,
+            0.0088080893110627,
+            0.018100411389202663,
+            -0.0004971948011851863,
+            0.01216566778850985,
+            0.02369448391321832,
+            0.002637001729688794,
+            0.005012948614448268,
+            -0.024213704079796478,
+            0.003420184574622472,
+            -0.0004359900462023086,
+            8.265488264269535e-05,
+            0.016267948861008616,
+            -0.019572032453100303,
+            -0.02216264534756073,
+            0.021871880626992544,
+            -0.012742926070754964,
+            0.011676769226089007,
+            0.0035362870996336194,
+            -0.010732229460038356,
+            0.0006838777488770268,
+            -0.03083438636386314,
+            0.008820746379709355,
+            -0.03216685858285627,
+            -0.032230808828686675,
+            -0.013547681710236494,
+            -0.009994106571421729,
+            0.0008758088338659712,
+            0.028447187234357855,
+            -0.00795317029726568,
+            -0.015797892954136448,
+            0.005363565557248085,
+            -0.011199252697624111,
+            -0.00817162221789202,
+            -0.0007867892785855576,
+            0.005856939999700809,
+            0.009461959355310273,
+            0.005997804620690314,
+            0.011648335019189696,
+            0.007491946658046298,
+            -0.008028800679169293,
+            -0.008837725220736191,
+            0.015889923814605387,
+            -0.0014409716087425222,
+            -0.01176424391171685,
+            0.030430130082762363,
+            -0.013300620185884817,
+            -0.0005592926632214984,
+            -0.0021975669766032472,
+            -0.0015687208316663597,
+            -0.0021652076181411244,
+            -0.014225289143358926,
+            0.01870568404189425,
+            8.334501002782455e-05,
+            -0.014288806218067678,
+            0.0032841743905975634,
+            0.003841780428153241,
+            -0.0036498339483411074,
+            0.008388922960269383,
+            -0.008147275785908849,
+            0.017348147498892,
+            -0.015113395971380741,
+            0.008740452410210708,
+            0.0038335161236289234,
+            0.0043482078469644385,
+            -0.0065755348640655865,
+            -0.01592809184384243,
+            0.009642593996701513,
+            0.012051872087663658,
+            0.02173264012652684,
+            -0.02525660485859878,
+            0.00876161653901566,
+            -0.005374533920781607,
+            -0.014534179652013885,
+            -0.016127866637369738,
+            0.00982303248417706,
+            0.022322910435388785,
+            0.0063086346898091296,
+            0.012139263756190026,
+            0.00013383660542132047,
+            0.005848259030010402,
+            -0.014767198851314148,
+            0.010570158368602513,
+            -0.011355774405066537,
+            -0.018439261453084426,
+            -0.005243587266408389,
+            0.028894120979627323,
+            -0.005492024421080143,
+            0.017938747701972086,
+            0.0021350711250093858,
+            0.029007356490801176,
+            -0.008686781541806358,
+            0.0014883465513564527,
+            0.0039866022469891155,
+            -0.01295271525889559,
+            -0.004272008629260297,
+            0.01976273843646234,
+            0.04882868638601926,
+            0.010204475902027572,
+            -0.0009155109545661691,
+            0.010069071220988797,
+            0.005626418787776309,
+            0.002148101019776153,
+            -0.00805781810841286,
+            -0.030431071067211577,
+            0.008883707378543252,
+            0.005628218215212441,
+            -0.01997825125961566,
+            0.019838234890736068,
+            0.026483365072033395,
+            0.023040623182975964,
+            -0.011585643598171132,
+            0.016539930468533123,
+            -0.010092529089936188,
+            0.0177008735686489,
+            0.028665424114962996,
+            0.013810061902648515,
+            -0.010487150104822674,
+            -0.0020847870330548534,
+            0.006329342599564054,
+            0.004186981876004016,
+            0.00026239408223817384,
+            0.015190614519766204,
+            -0.019155180136775043,
+            0.01005728016778505,
+            0.014725826697964637,
+            0.03322436221319799,
+            8.049930133627564e-05,
+            -0.01645191835557271,
+            0.0016249281996128196,
+            -0.009976155662939693,
+            0.00856399492362911,
+            0.000632427127231252,
+            0.0032242260692672124,
+            -0.009688119824276678,
+            -0.012682779503915694,
+            -0.002851918183031152,
+            0.009403205508474241,
+            0.0021188716361101698,
+            0.002759004011231831,
+            0.007641925150177999,
+            0.0022830062031146204,
+            0.01713024474384216,
+            -0.0011181980841546123,
+            -0.00029141889484139217,
+            0.004601090256674874,
+            0.010509199590505558,
+            -0.01649852835819883,
+            -0.017548369964263514,
+            0.004337237475053179,
+            0.012217494210546133,
+            -0.013159179948224596,
+            -0.013241981632059755,
+            0.015704543608959117,
+            -0.01913575130892795,
+            0.024934102868065366,
+            -0.00942381892138331,
+            -0.004543611336526587,
+            0.000343200128934031,
+            -0.00727538880611703,
+            -0.0038751118497522673,
+            -0.0030064279504724797,
+            0.003984715814505561,
+            0.0019261818110367147,
+            0.007472350699291607,
+            0.008472369788801792,
+            0.014568547873264247,
+            0.006043895352236181,
+            0.0032580830408778395,
+            0.009109104498825864,
+            -0.014523846294121576,
+            -0.014800649887278973,
+            -0.002704791598011488,
+            -0.008895394038700809,
+            0.00902016867266151,
+            -0.006607182154874112,
+            0.00845408382182461,
+            -0.015501867835442603,
+            0.015573642225745032,
+            -0.012200552219016517,
+            -0.0008900215712478553,
+            -0.009666233545144905,
+            -2.7470277092642562e-05,
+            0.008191697904514857,
+            0.014605287653054787,
+            0.0070552942843382215,
+            -0.031921405225964206,
+            0.0017403427534633655,
+            -0.008846568352320488,
+            0.03446319360336241,
+            0.002612606662691094,
+            -0.01060863123202008,
+            0.020352181356568008,
+            -0.01592124132155158,
+            -0.00027704668715660693,
+            -0.01602075523652062,
+            -0.004797557413047212,
+            0.00707939446796679,
+            0.016399017978990946,
+            0.02759049512664933,
+            -0.005697020356865688,
+            -0.007995305872076221,
+            0.006384047872862793,
+            0.029590696670334232,
+            0.012246955508890425,
+            0.024748351124415785,
+            -0.007633712004672087,
+            0.014348228716100414,
+            0.007773260552660029,
+            0.0076045488653348975,
+            0.004321689250440021,
+            -0.002591801797145718,
+            -0.011698574606499644,
+            -0.014974971814360001,
+            -0.0037563183330623384,
+            -0.016987915346053498,
+            0.00012247833530362867,
+            0.021344400276501117,
+            -0.00475351423959752,
+            -0.0033847161875730026,
+            -0.01874186437694691,
+            0.0033461670218611346,
+            -0.010029445302549983,
+            -0.00367421246775698,
+            -0.013133469163793961,
+            -0.0254122035745965,
+            -0.018556369080030412,
+            -0.002155244024999317,
+            -0.00929309186646207,
+            -0.015194269781263309,
+            0.0070354690219415535,
+            -0.016305117309271634,
+            0.0080837166473439,
+            -0.00448473417024141,
+            -0.016508775226236883,
+            -0.00958925666995396,
+            0.01100335153658195,
+            -0.0041819993634961685,
+            -0.01929493648733792,
+            -0.01282300765645703,
+            -0.011300260564838231,
+            0.0051733704121938884,
+            0.005996780699975213,
+            -0.0038111648304178544,
+            -0.0028279639348479987,
+            0.016640121845366295,
+            -0.010197365811240557,
+            0.009878730312873492,
+            -0.03335474019383245,
+            -0.03551826040406959,
+            0.004497847402931096,
+            0.01507007013703286,
+            0.0013085881889541252,
+            -0.02078680101065954,
+            0.010806315988966003,
+            -0.016499186313125556,
+            -0.00858013792516617,
+            -0.008522534127498645,
+            0.009649521088411411,
+            -0.007283003380127847,
+            0.01299542385123007,
+            -5.3226845309105124e-05,
+            0.030477683597373687,
+            0.0074153455615142755,
+            0.004161881195438522,
+            -0.00679081048132963,
+            0.008354912600159931,
+            -0.001621776416108105,
+            0.004080191054849395,
+            0.02120524874226696,
+            0.012440848502230418,
+            -0.00218041171518331,
+            0.006458415098595872,
+            0.005678055538564355,
+            -0.017186537377121235,
+            0.02507960402044456,
+            0.024678178106806115,
+            -0.008205288425409454,
+            0.004108402466376321,
+            0.005783335302169668,
+            -0.008846663924040241,
+            -0.006897707231869481,
+            0.006527795262026734,
+            -0.018474181903007085,
+            -0.0008721381678008483,
+            0.010598084358464908,
+            -0.02255441909004843,
+            -0.004840511019512733,
+            0.021745651362109403,
+            0.004631567631335321,
+            -0.018558972627958834,
+            0.02764788406651861,
+            -0.004942361238163609,
+            -0.0027290999934532494,
+            -0.010876135202650048,
+            0.01671497143541799,
+            -0.012173527207375091,
+            0.008443317346655462,
+            0.011021541861818043,
+            -0.006295639322287843,
+            0.003083685676540999,
+            0.008277206401503182,
+            -0.004504302913715969,
+            -0.0068075424927125676,
+            -0.0010508416962215848,
+            0.010167160518175114,
+            0.0036980815941666336,
+            -0.022704963865525812,
+            -0.006020073261002279,
+            0.02934227992246887,
+            -0.015082922782242507,
+            -0.0050916398557337705,
+            0.005376294716056329,
+            0.007709342013963906,
+            -0.003476236561444464,
+            -0.0023399991233097337,
+            -0.00787486645740744,
+            0.010356414979428607,
+            -0.015950083845400274,
+            0.02076360898078935,
+            0.025608263494550438,
+            0.024504236990840842,
+            -0.014814235990238182,
+            0.010828584133720797,
+            -0.011703992394890898,
+            -0.011047470905668532,
+            -0.015542835222173864,
+            -0.001658296303013357,
+            0.001970208852236451,
+            0.013107754130616248,
+            0.009666887546609378,
+            0.0235853683657408,
+            0.006473544089109194,
+            0.008973300312205858,
+            0.014118601565410305,
+            -0.003902202854013708,
+            -0.018303438244719996,
+            -0.014584226585460755,
+            -0.011977613285028946,
+            -0.01756345491020528,
+            0.0004563602312336091,
+            0.020461659996271687,
+            0.012417744207683839,
+            0.024136626592375227,
+            0.002119213701333967,
+            0.013759902247467922,
+            -2.9525392347047554e-05,
+            -0.000892319792935443,
+            0.00045783825646242434,
+            -0.005238483120312843,
+            0.019255756091120846,
+            -0.004599752813817219,
+            0.01656589522110913,
+            0.0006443773300684925,
+            0.007133940816771615,
+            0.013554761340256566,
+            -0.01704388669741371,
+            -0.003443100892108368,
+            -0.010029620309829285,
+            0.000941391966877521,
+            -0.019633111750884034,
+            0.01934571074127927,
+            0.010273330977233852,
+            0.0017121345993533963,
+            -0.018327189396014198,
+            -0.0029706385099949693,
+            0.03448635131998772,
+            0.019458999453011265,
+            0.0050734354363614525,
+            0.02632443373898792,
+            0.013018762352749236,
+            0.014443611411502168,
+            0.010153370950346075,
+            0.02356766253713842,
+            -0.023479811300626968,
+            0.02095255383200973,
+            -0.011590347850340786,
+            -0.007414136601134069,
+            -0.0035774888741646705,
+            0.0025141508183687115,
+            -0.013588691733097813,
+            0.00982877622088977,
+            -0.0017916539399655826,
+            -0.025386211797192578,
+            0.009164316443206016,
+            -0.008811052979979271,
+            0.010464442616928296,
+            -0.007774569252594923,
+            -0.0002806136360286729,
+            -0.0127084026190163,
+            -0.015184317235990752,
+            0.001259201880504347,
+            -0.009158429132281081,
+            0.023908150915412617,
+            -0.021210848953875233,
+            -0.01321384014436847,
+            0.013997502713857512,
+            -0.006869443694669701,
+            -0.011364239609894453,
+            -0.00374449980963666,
+            0.023673050288879224,
+            -0.009243341446512421,
+            -0.01359367519009039,
+            0.014464166953530989,
+            0.003834735113400469,
+            0.0059713317014342395,
+            -0.007508862694272452,
+            -0.008428705474135382,
+            -0.003280788036976103,
+            0.005998840043285335,
+            0.026615077617284676,
+            -0.014059930972065399,
+            -0.0032833264512892022,
+            -0.009189999799498163,
+            0.004118204537632102,
+            -0.02359889635281631,
+            0.02771075351384059,
+            -0.008871395303200375,
+            0.021705177758560575,
+            -0.0015924449308442096,
+            -0.006239188158271628,
+            -0.024231036074349822,
+            0.027522945599928003,
+            0.003683870966132199,
+            -0.010221180899173243,
+            0.0012768725460662862,
+            -0.052005292736885915,
+            0.016636109796947025,
+            0.008429895913163985,
+            -0.019507994429834353,
+            0.003697519981610208,
+            -0.004773584856005853,
+            0.018983847580558853,
+            -0.022966993228850697,
+            -0.014078986007109447,
+            0.008283766047449966,
+            -0.01153285124362746,
+            0.0031439180465649996,
+            -0.02527749501821793,
+            -0.005754537200570427,
+            0.0011760345167229269,
+            -0.005276936074280004,
+            -0.001705933750221904,
+            -0.020022761608132548,
+            -0.025972227850795115,
+            0.017666017379157135,
+            0.015820875579600418,
+            -0.002278262448951789,
+            0.01320818011561616,
+            0.008709899932124187,
+            0.00836573261091119,
+            0.016034263394940008,
+            0.007124509234103267,
+            0.0081473711608784,
+            0.045498533239850446,
+            -0.019846323044145485,
+            0.007371629414298193,
+            -0.0014854149174929534,
+            0.013951741898629716,
+            -0.030467924817555014,
+            -0.0055144480998344705,
+            0.0062802744565489945,
+            -0.0049373689340857915,
+            0.03510454786147091,
+            -0.017326094216003073,
+            0.01694160526735264,
+            -0.0026075809870445238,
+            0.0007420394265014909,
+            -0.02042003877666893,
+            -0.026156937726134134,
+            -0.014294938857971834,
+            0.0011857406889443987,
+            -0.0010047941261608975,
+            -0.011946583993069242,
+            0.006797001710574735,
+            0.004917161312925866,
+            -0.0020713379451245515,
+            0.015652523103715316,
+            -0.02952061566240878,
+            0.007356800041369564,
+            -0.010733578620243498,
+            0.020008504830359524,
+            -0.00990331048284697,
+            -0.0013319400098308093,
+            0.007236960885181047,
+            -0.004122868383988748,
+            0.013216043472071927,
+            -0.0038337251451538677,
+            0.004227115653354611,
+            -0.005221321656766545,
+            -0.01794343569722877,
+            -0.0016435261495179767,
+            0.021429008432348098,
+            -0.008349684096540855,
+            0.031864497493526205,
+            -0.01264743878037828,
+            0.009174982784686633,
+            0.010692990337462762,
+            0.0006628831036334152,
+            -0.005286285851702483,
+            0.009052971814211658,
+            0.0021431970274545893,
+            -0.008690520448278274,
+            0.006934310986093342,
+            -0.018349449937310143,
+            0.008494451785507706,
+            0.03394632739506833,
+            0.0018511328780472266,
+            -0.0007519661670070526,
+            -0.007266663179613392,
+            -0.01335844201823745,
+            0.018540713029383066,
+            0.005488273906733518,
+            -0.002546531152114063,
+            -0.027536660869590543,
+            0.02007746621157705,
+            0.0012265798861291223,
+            -0.004755325069578531,
+            0.012466557901411086,
+            0.0020495867632654885,
+            0.019989531151046736,
+            -0.010171678639113316,
+            0.030757957899447096,
+            0.010050036220124393,
+            0.002959833051227505,
+            0.0038556928367700113,
+            -0.036655749074851536,
+            -0.01652152778069314,
+            -0.00019357502472740266,
+            0.014271581054865031,
+            -0.012430647252826173,
+            -0.022432580814844155,
+            0.0010346169306977334,
+            0.01148004051449691,
+            -0.0033898642762881046,
+            -0.009650665305271706,
+            0.01560757541276278,
+            -0.020342213902670575,
+            0.006701302394408393,
+            0.007693672045981921,
+            0.0287100214318104,
+            -0.0020735384554022153,
+            -0.0012324838849990056,
+            0.000626416803904792,
+            -0.0055669860801192935,
+            -0.0008037366126870824,
+            0.017251369734470945,
+            -0.013661316947002466,
+            0.0175926211217534,
+            -0.012735241490550407,
+            0.0064713656749838585,
+            -0.0021092378432037563,
+            -0.010629579860050812,
+            -0.020114807971202787,
+            0.0029609495764307357,
+            0.009378003163375899,
+            0.006920144698428806,
+            -0.010438984881449953,
+            0.010715409812532968,
+            0.012197673689706352,
+            -0.00753962861484439,
+            0.014519905132928041,
+            0.022173989895384192,
+            -0.007687503607783572,
+            0.013633321414009471,
+            -0.0041446334728745095,
+            -0.014056264816568229,
+            -0.009985463824440833,
+            0.023929750617160402,
+            -0.0005563806942667597,
+            0.008756997373618809,
+            0.00388125051641897,
+            0.0007856447663040696,
+            -0.011226530517316183,
+            -0.001653447552601681,
+            0.019294448360333633,
+            0.011831934675073843,
+            -0.0013888534822008192,
+            -0.0010525764356699885,
+            -0.0014566936525653407,
+            -0.009427557739483988,
+            0.02049188741459728,
+            -0.022030783927622997,
+            -0.0019604940727076573,
+            0.00461018148439991,
+            -0.022589816803875618,
+            0.0001863117949313752,
+            -0.024570217962989905,
+            -0.01569952241187498,
+            0.011763033781122343,
+            -0.0022530631823737554,
+            -0.016028810867854907,
+            -0.0003242701040942633,
+            7.424127301806556e-05,
+            -0.007815789940508527,
+            -0.004135500768181258,
+            0.006115909992703774,
+            -0.00034193217510010117,
+            -0.0032262189374874842,
+            0.013430154310506884,
+            -0.009996622196870008,
+            0.015431528911575157,
+            0.0067029980830614145,
+            -0.015702167721800483,
+            0.0021406610839557003,
+            0.018500219544264753,
+            -0.0058910087864916085,
+            0.00787021425357441,
+            -0.014681907249972176,
+            -0.0037495551894426303,
+            -0.005787661351944086,
+            0.006051351977821761,
+            0.024849195780885833,
+            0.012969434450643372,
+            0.01139340445077682,
+            0.017405361843075933,
+            0.004325000715054734,
+            -0.01572297405297993,
+            0.0007416798945675923,
+            0.01682727189828363,
+            0.010464036837245312,
+            0.009804326468113703,
+            0.0008975556667684365,
+            0.0043704860361487635,
+            -0.005142176468800175,
+            0.009317064072855551,
+            0.02915482449687158,
+            0.0013434689612997423,
+            0.008696141010760192,
+            0.017276730345147134,
+            0.027038741396131265,
+            -0.0035088997194497725,
+            -0.007469614319788499,
+            0.008739942310300076,
+            -0.009657991604173173,
+            -0.001081629174976152,
+            -0.017000351183542026,
+            -0.012310275736750868,
+            0.016510505259358398,
+            -0.006084627062230492,
+            0.02848913486719956,
+            -0.0010655806791907225,
+            -0.02026498848361482,
+            -0.01565505608194376,
+            0.001404549479440637,
+            -0.01362132895080652,
+            0.005287566256051299,
+            0.014187265685427415,
+            0.01917046513020831,
+            0.004687191682875024,
+            -0.022258337721085656,
+            0.037364962197399806,
+            0.01757108963349427,
+            0.019377870795760123,
+            -0.019422840506328382,
+            -0.006159311237412833,
+            -0.009077095160676276,
+            0.02116697012944498,
+            -0.00880692095488465,
+            -0.009875674856043954,
+            0.024014332004451508,
+            0.017812606845656484,
+            0.005679806286464216,
+            0.006736609904079586,
+            0.0020048663439319585,
+            0.0025414630253661444,
+            -0.015831593851012047,
+            0.017957902498542212,
+            -0.005858125698347523,
+            -0.0058068554681761945,
+            0.017529514573571298,
+            -0.005989581792993925,
+            -0.0030306021203732347,
+            0.00015839997806822045,
+            -0.019697052145531482,
+            0.011618112886207201,
+            0.008256007601576028,
+            -0.017229774429089857,
+            -0.0014757560402567304,
+            -0.0002404480326945295,
+            -0.014118694740134205,
+            0.02004885146995733,
+            0.01694893356262835,
+            -0.0013248884139405398,
+            -0.0051752815372013315,
+            0.008223868147393114,
+            0.007000679132597393,
+            0.005256194890343006,
+            -0.019496098876977206,
+            0.005230597149342019,
+            0.005103909294824105,
+            -0.0014716649591412774,
+            0.010274346323051634,
+            -0.009030594825985526,
+            0.033740380586893826,
+            -0.006505788197604522,
+            0.013065658941136721,
+            -0.0043634731684193745,
+            -0.0055529072007985125,
+            0.001816020676232658,
+            0.013054468478598695,
+            0.009753501030700457,
+            -0.00970446155762939,
+            -0.004172354962129865,
+            0.018697982092775737,
+            -0.011983156146516074,
+            0.0013300791598046338,
+            0.01564072865804035,
+            -0.004858814041438713,
+            0.004708235083775013,
+            0.008635204790345423,
+            -0.004489505237783172,
+            0.01801512858970922,
+            -0.016482572999490366,
+            0.0023953436660054026,
+            0.010694869684962088,
+            0.02390864159830096,
+            -0.019152187046410703,
+            0.021786596486708207,
+            -0.017951050280323393,
+            0.026538582634436314,
+            0.014484894489694262,
+            -0.0035222259457311444,
+            -0.01946200885147006,
+            -0.00850690913394181,
+            0.011654565129063417,
+            -0.021423803876241532,
+            -0.0007985948078290641,
+            -0.0007290222944686444,
+            -0.021367628543092133,
+            -0.011189364017231132,
+            -0.009961779129778331,
+            0.010701637745810138,
+            0.001918633898887013,
+            0.010380983868185359,
+            0.008002182827196827,
+            0.010784760574518087,
+            0.014405640442760822,
+            0.0008958107769384412,
+            -0.010743001445440941,
+            -0.023795589003655273,
+            0.016987708698679597,
+            0.00438463274382093,
+            0.0007862212695139549,
+            -0.01627468994515667,
+            0.011936651269737248,
+            0.013250643815768723,
+            -0.009385605548926286,
+            0.009121882469425302,
+            -0.03716901103204463,
+            -0.004552808410398574,
+            -0.0035704811925430366,
+            -0.0020309197365173475,
+            -0.00908678158053104,
+            0.01559445415882485,
+            -0.013682507836005851,
+            -0.0009590922904844632,
+            0.00011696985057536548,
+            -0.0074756526903548915,
+            -1.5467890432254378e-05,
+            0.020649928837031005,
+            0.008438785939908428,
+            -0.0037382105257999757,
+            0.010376213224654074,
+            0.008826031303751312,
+            -0.013900359449287676,
+            -0.01527474712786084,
+            0.019035366035884273,
+            -0.018776918681908793,
+            0.0063098418544296165,
+            0.01677916431403883,
+            0.020778813096283988,
+            -0.02891862501830075,
+            -0.0007231469961304435,
+            0.022093939332021185,
+            -0.0012807635799950273,
+            0.006370841109790851,
+            0.008781853138377092,
+            0.006754082980602544,
+            0.0029102159062355374,
+            -0.003711173531879644,
+            -4.843770941622283e-06,
+            -0.003940474588075707,
+            0.007155539022862352,
+            0.004366433868349658,
+            0.036453728071874844,
+            -0.013661129348165794,
+            0.002541308805735001,
+            -0.0058838191638835155,
+            -0.00347401304572066,
+            -0.00593469688967835,
+            0.004407665856950772,
+            -0.010055727994798024,
+            0.004330827357536078,
+            -0.001570861535288093,
+            -0.0003731129422346693,
+            0.004883969598300302,
+            -0.001701673379646408,
+            0.007036892564844683,
+            -0.011863650867472625,
+            -0.0003979727750011258,
+            -0.020174851412609312,
+            0.021117284557875267,
+            0.00011666551282792189,
+            0.003662462447326642,
+            0.03723741246290771,
+            0.00476064510662738,
+            -0.00238355131307445,
+            -0.012880057949435798,
+            -0.013551338766321525,
+            -0.012986444917747435,
+            -0.020703821629987552,
+            -0.0036665673548525388,
+            -0.021006311026247164,
+            -0.0052061406234528915,
+            0.010362862594785442,
+            -0.011540184420787572,
+            -0.004023659825622843,
+            -0.015881589631889473,
+            -0.006531981517815174,
+            -0.03114211633687585,
+            -0.01124607488650473,
+            0.007493228366452009,
+            -0.019149285949049886,
+            -0.0039932870926865805,
+            0.026850121590606017,
+            0.001095360635760988,
+            0.01387696424607248,
+            0.0038058248676879625,
+            0.008117686785954085,
+            -0.0045937472017383955,
+            -0.0005669025036141417,
+            -0.01928244877716143,
+            0.003391980330529026,
+            0.008525354854576925,
+            -0.00361507909530282,
+            0.0070008404454106905,
+            -0.008835727374865623,
+            -0.016950017602980382,
+            0.0021654956715219517,
+            0.009334379529385142,
+            -0.006639140003365139,
+            -0.017587472740268683,
+            -0.011049929670847685,
+            0.010401189523318744,
+            -0.02283429470064187,
+            0.006257145380266984,
+            0.016439581169238607,
+            -0.009363030375624585,
+            0.004651575425738153,
+            0.019809183749755425,
+            -0.011810674576781702,
+            -0.005811328501528397,
+            0.008471992070998505,
+            0.013891259999721588,
+            0.0095606134736748,
+            0.02954498207274139,
+            0.0026596809804008996,
+            0.000561870602466584,
+            0.004889263753490895,
+            -0.004277061142504552,
+            -0.017423200644235665,
+            -0.006243787897080996,
+            0.0038507494787521856,
+            0.012873996671952184,
+            -0.009879417960879032,
+            -0.00048575277472139055,
+            0.01682604801673206,
+            0.004530939990187663,
+            -0.0033901248079550284,
+            -0.011868399508738274,
+            -0.020080331108085103,
+            0.0015580259874809898,
+            -0.01362200460317948,
+            0.015354347040143093,
+            0.003390601069331352,
+            0.0015284610287850575,
+            -0.004239880336806429,
+            0.01828645373058079,
+            -0.005013418527935815,
+            0.012749833447770729,
+            -0.006148890994661865,
+            -0.004587831231448011,
+            0.007986533783001116,
+            0.009166578151006778,
+            0.0004939857995863664,
+            0.05421019830078428,
+            0.002838056604877719,
+            0.01333205648597435,
+            0.013419127695838135,
+            -0.014523733369484934,
+            0.014086525147734823,
+            0.012688569853552613,
+            0.018497344547821693,
+            0.013037819487160016,
+            -0.012353776077273462,
+            0.017354801119939964,
+            0.0024084723134891487,
+            0.003909681065813095,
+            0.0022202438064449653,
+            -0.004528576079628071,
+            0.019594683950225115,
+            0.004070222328547378,
+            -0.0074362443729372105,
+            0.004453618452623641,
+            0.005007543168224267,
+            -0.013105524222311019,
+            -0.0112958915514782,
+            0.027105208848863218,
+            -0.01219417557128091,
+            -0.018749971743148304,
+            0.006706594269933445,
+            -0.02008458683853808,
+            -0.03763686535076864,
+            -0.0021332074289244973,
+            0.009992361098878196,
+            0.002795183524495193,
+            -0.00253695604043486,
+            0.01614298064601032,
+            0.016730443047719683,
+            -0.016985296937992726,
+            -0.018245902728910347,
+            0.0025257780765481825,
+            0.0016333325293570182,
+            0.01861807236979066,
+            -0.023168663546030196,
+            -0.006378864850313035,
+            -1.3795465857380962e-05,
+            -0.006829008132963306,
+            0.004850306884033312,
+            0.004703280112191218,
+            -0.0071714759384722965,
+            -0.0027773519096287217,
+            0.002256846645341137,
+            0.01975383140755998,
+            0.013198754584066554,
+            0.01572971337890945,
+            0.002600741959577911,
+            -0.005033508062449482,
+            -0.016817359058724685,
+            -0.005450011991586895,
+            -0.013478330855299777,
+            -0.009091854345715177,
+            -0.003042052092498012,
+            0.008217376329142566,
+            -0.025053469059115735,
+            -0.011965083373220982,
+            0.0119294363235832,
+            0.00391377331459168,
+            0.0003099203772539202,
+            0.007607672728136279,
+            -0.003830132495364688,
+            0.010331053246451333,
+            0.013823890473448383,
+            0.0076112887952959766,
+            -0.004723377809559611,
+            0.009600411526173948,
+            0.011460781436250457,
+            0.006015477068316837,
+            -0.005072711638895421,
+            0.023859899048048694,
+            0.01821920335673539,
+            -0.01680365156728093,
+            -0.002149107383573353,
+            -0.0027829676356423997,
+            -0.01884308738652242,
+            0.009665830801645276,
+            0.00880322398336952,
+            0.008308268605449346,
+            0.003906369351868086,
+            0.00974531700889854,
+            -0.013875904819275925,
+            -0.0050570811397453235,
+            0.002870120865445989,
+            -0.005729556771404781,
+            -0.01901337075686543,
+            0.000297242372701423,
+            -0.011987192261104132,
+            -0.011327157550660895,
+            -0.01409710922165158,
+            -0.003221240464411861,
+            0.005283472511900512,
+            0.021472187640986972,
+            -0.004410881709548286,
+            -0.001855852749039012,
+            0.01351697599517034,
+            0.010060125186506398,
+            0.0021658738823899905,
+            0.0002206474541239167,
+            0.005418828324605768,
+            -0.0027919130944783397,
+            0.0008094042430809068,
+            -0.0015826797905407809,
+            0.021089276765434876,
+            -0.02016611086095107,
+            -0.009413977569213423,
+            -0.009288078098692112,
+            0.005842197094339927,
+            0.028793032462603636,
+            -0.007165634502373454,
+            -0.010104002083017078,
+            -0.01994315293556678,
+            0.0003786527922888834,
+            -0.07810609175343784,
+            -0.005210636756258548,
+            0.001870513618910238,
+            0.03126380113105983,
+            -0.010817888476436834,
+            0.0033673768058629823,
+            -0.002204487854177595,
+            0.014542299655712928,
+            -0.001760107965206874,
+            -0.0013283746982194259,
+            0.0071954806737858056,
+            -0.005640400818010697,
+            -0.006210989097047217,
+            0.005904163661619133,
+            -9.945941028848348e-05,
+            -0.010266938705916303,
+            0.00057263080955709,
+            0.02209498395848068,
+            0.0159398346608872,
+            -0.006575439759804175,
+            0.01816748552021211,
+            -0.00719294553484007,
+            -0.009838738592324234,
+            0.005639288780502127,
+            -0.0019007683459126327,
+            -0.015369619176147573,
+            -0.011475977270803784,
+            0.009862808084641291,
+            -0.030069890644150284,
+            -0.00010725753918890802,
+            0.00943698424646706,
+            -0.013540245857270492,
+            -0.008293042205008774,
+            0.007212974960781723,
+            0.006708343474532583,
+            -0.0012017968151025369,
+            -0.0078026000239507145,
+            -0.02527526731750743,
+            0.0027578400032925625,
+            -0.002446811971857853,
+            -0.004899702349784173,
+            0.0019960465173086183,
+            0.001479503476855694,
+            -0.012645905858466798,
+            0.01216748847452763,
+            -0.013378877368040743,
+            -0.012053251571418766,
+            0.012508842412556306,
+            -0.00027167241745418584,
+            0.01682680930386839,
+            -0.020607299598265426,
+            0.0010702151063620052,
+            -0.028075439824932304,
+            -0.007671848153932823,
+            -0.015486630894165054,
+            -0.0009776136330459065,
+            -0.0012939921569540803,
+            -0.03481880531471923,
+            0.012889447665794033,
+            -0.005200149829198305,
+            -0.006094702500154477,
+            -0.00459778303079732,
+            -0.024583504817899775,
+            0.006072281238752663,
+            0.00607350167867964,
+            -0.006506508970816635,
+            -0.016816439635445445,
+            0.008428048142588516,
+            -0.0011040593013238872,
+            0.004873670535200424,
+            -0.007475800310154908,
+            -0.0005638920132076605,
+            -0.013047135093134946,
+            -0.005463454269764494,
+            -0.0008522562324122834,
+            0.01827123304871433,
+            0.0044055851550317835,
+            -0.014526381222031981,
+            0.016862445236481526,
+            0.013250692156820124,
+            0.01602714293656341,
+            -0.0071089293427753115,
+            -0.0004265739315250755,
+            -0.02750630108787212,
+            -0.00327225092954903,
+            -0.023687644154808972,
+            0.00041213204365507744,
+            0.009447660230507965,
+            0.0203905264889617,
+            0.012694421477479161,
+            0.004620617778996077,
+            -0.001557511764988156,
+            0.0066751090724502,
+            0.02696959560423252,
+            -0.01099183126742224,
+            -0.006969111598423376,
+            -0.01884504192789845,
+            0.01113023717919988,
+            0.025135748541199255,
+            0.003381303139273845,
+            -0.011317957771935042,
+            0.008112598754655146,
+            -0.0009247845526290428,
+            -0.010302980598863016,
+            0.02889587382004902,
+            -0.0391134481322538,
+            -0.015599591108049343,
+            -0.004605824192962023,
+            0.007406520608110251,
+            -0.015596917115435033,
+            0.00770468362805995,
+            -0.012614339656105264,
+            0.001699049329817941,
+            -0.005851477164035241,
+            0.03535087340659861,
+            -0.0031950960727572925,
+            0.0006909856233673865,
+            -0.003395233050301427,
+            0.003531135044131077,
+            0.012355948349937309,
+            -0.004444221317023453,
+            -0.030726300006909638,
+            -0.0022983766869702346,
+            -0.011198252640110788,
+            0.009274666307515955,
+            -0.016207714291658314,
+            -0.017342435976846468,
+            -0.009424737744066259,
+            -0.021827243671837953,
+            -0.012403084982789827,
+            0.029747453318486338,
+            -0.005824001042844527,
+            0.01747569398142269,
+            -0.005219060548639168,
+            -0.008365003057800934,
+            0.00021272411948388247,
+            0.01344335658390732,
+            -0.028498240040284714,
+            0.00041963499785238845,
+            -0.002322931702641349,
+            0.0126393550661767,
+            -0.001595134597082556,
+            -0.00983073906977773,
+            0.006903313245694627,
+            0.0010522208685011098,
+            -0.015648940549713764,
+            -0.0007573245570947632,
+            -0.008533504878052644,
+            0.020475329445149498,
+            0.00027791387213409814,
+            0.001074213629646005,
+            0.013864331671732542,
+            -0.02102352993295736,
+            0.01576938071930521,
+            -0.021722775924283996,
+            0.01893267442161547,
+            -0.006577103101038035,
+            0.005772016664149475,
+            0.008283223343661555,
+            0.014068789332250016,
+            -0.014030042849491258,
+            -0.00663752723080222,
+            -0.034436220972634726,
+            -0.0122621879114314,
+            -0.001946072976826902,
+            -0.005308941654794638,
+            -0.014773123927909993,
+            -0.0011054822266082056,
+            -0.026205461509573584,
+            -0.006460373065662098,
+            0.012822188046808264,
+            0.019900312098359022,
+            -0.008409070094265829,
+            0.014163132434889265,
+            -0.006743711872055091,
+            -0.0028230771810882492,
+            -0.012652021749331811,
+            -0.018061180380009395,
+            -0.01083672324723357,
+            -0.01890966636827258,
+            -0.0018635408070944168,
+            0.004709222372611061,
+            -0.015760618199235518,
+            0.011435131737642627,
+            0.03198874888333405,
+            -0.01567271472650171,
+            0.008243406188694909,
+            0.016411496328333205,
+            0.0034236971031973503,
+            0.011728079139322837,
+            0.0038311415233052795,
+            0.004621789116267879,
+            -0.011109450058055078,
+            -0.011473158133621661,
+            0.006524490579223986,
+            0.00954511918454185,
+            -0.004913142601134428,
+            0.024347072255007807,
+            0.0008745419206472714,
+            0.005045794664611216,
+            0.005304506902446091,
+            0.011488187043525824,
+            -0.013135271908307747,
+            0.019746484188715552,
+            -0.002055210086078267,
+            -0.015571665869889405,
+            -0.01535499169234956,
+            -0.01105419264826655,
+            -0.018626399757822328,
+            0.0017108083455869296,
+            0.0181439079855921,
+            -0.0012493542718277685,
+            0.012137753522284387,
+            -0.00924460959150499,
+            -0.005019437549649743,
+            -0.0027156377206998474,
+            -0.038211954658821295,
+            -0.0021878970332906457,
+            -0.018809487178512628,
+            -0.02498945889484935,
+            0.03505216568400238,
+            -4.26477081213836e-05,
+            0.00997643255528116,
+            -0.010310197159232386,
+            -0.0002454902098986129,
+            -0.011065403358802515,
+            0.014954998647970473,
+            0.004624123403100598,
+            0.008872923678983478,
+            0.008560280683876702,
+            -0.016419500515801093,
+            -0.002997557798375716,
+            0.007472390947008284,
+            -0.00374822278027057,
+            0.012680407710694952,
+            0.015353684671120731,
+            0.020886474047003733,
+            0.011161262397502876,
+            0.002045862158732102,
+            0.00887271411363814,
+            0.009563361150297699,
+            -0.0003513237086717377,
+            -0.010459621030424415,
+            0.015419039131459582,
+            -0.02204848739950703,
+            0.012172257374410352,
+            0.04346634265794738,
+            0.019197433209794286,
+            0.029488039318675732,
+            -0.02272810439246353,
+            0.01448092574459596,
+            -0.003801043409441206,
+            0.009476229966122152,
+            0.0049309968883087685,
+            0.02346191840355209,
+            0.003973840862678076,
+            0.0073546443568171405,
+            0.010647581332452525,
+            -0.00972051412996416,
+            0.025644022498404723,
+            -0.00846669916801108,
+            5.1518540791212095e-05,
+            -0.004407265804145216,
+            -0.026400029105519137,
+            0.01615193008014211,
+            0.0081722536401398,
+            0.0010702314763197463,
+            0.0010263623479474385,
+            -0.0019566604229607643,
+            0.007982949510899019,
+            0.007079322685771456,
+            -0.015274148711633536,
+            -0.003886053737590683,
+            -0.018527455865296014,
+            0.0012864569064749438,
+            -0.024337133883771334,
+            0.010466651250597847,
+            -0.017568706814132484,
+            -0.014822354159709854,
+            -0.021666681078831483,
+            -0.013232016867264097,
+            -0.02011653224168991,
+            0.0035858355517491515,
+            -0.008632755352062642,
+            0.013025728891203198,
+            -0.016962939287139683,
+            -0.021494971929779273,
+            0.008430887589398292,
+            0.00870802359711456,
+            0.007481282677955534,
+            0.009728368603458082,
+            0.014996157358749115,
+            0.01736292711410622,
+            -0.022712386143092265,
+            -0.023163001611599246,
+            -0.006682209717364464,
+            -0.003531495447167166,
+            0.006762379833125297,
+            0.00933681196321128,
+            0.012884185500343249,
+            -0.007077895849501418,
+            -0.007554762054641417,
+            -0.013283067570021503,
+            0.006197966297581871,
+            -0.003993067576677973,
+            -0.009464789349849612,
+            -0.0015662739825834614,
+            -0.0061389795593180686,
+            0.002753664732282926,
+            -0.0001791088888074743,
+            0.0007595261237141144,
+            0.017122834676182343,
+            -0.008176319370751018,
+            -0.009787747261447871,
+            -0.02027228453943018,
+            -0.010683273300243025,
+            -0.015604745521156111,
+            -0.0016279406232527138,
+            -0.006506713191861036,
+            -0.014420323628589382,
+            0.010888206569219648,
+            -0.011930708571093279,
+            -0.00887210117616784,
+            0.010807147705360856,
+            0.0073900910661282935,
+            -0.003431679006289002,
+            0.037804020313161775,
+            0.0011945597874820288,
+            0.00778778394048267,
+            -0.003366543306746973,
+            0.005296727102519836,
+            -0.017761251541598876,
+            -0.026382400396890007,
+            0.0010357716464853195,
+            -0.01775777668381489,
+            0.010807713750049491,
+            -0.008515368407459745,
+            0.0016051389544388124,
+            -0.00034958600424912995,
+            -0.009585854918521076,
+            0.0036549720706351473,
+            -0.008110927907950148,
+            0.004212219431447508,
+            -0.03507890639477636,
+            0.011409377585617926,
+            -0.01819120811524444,
+            -0.008962242411035718,
+            0.01710177594046791,
+            0.037560936862121835,
+            0.0028513388998398104,
+            0.00030058353811717895,
+            0.0014792184333546447,
+            0.004427921639326168,
+            0.021438127785303025,
+            0.006339483221420334,
+            -0.001187747087595264,
+            -0.02237982599829809,
+            0.0062435650294483695,
+            0.002200578793697352,
+            0.010092828474934744,
+            0.023105023225115266,
+            -0.016797326021070008,
+            -0.015378520287395565,
+            -0.012732527281000097,
+            -0.010797463778182385,
+            0.009881499418260473,
+            0.02801767062785128,
+            0.01811139137849365,
+            -0.017358098300028903,
+            -0.020207173948076473,
+            0.0016242901745373126,
+            -0.026331540602868762,
+            0.007269751661479859,
+            0.11940631114242901,
+            -0.0045382135734049305,
+            -0.005994407567723939,
+            -0.008679712231646732,
+            -0.004482553208064535,
+            0.016709571294385293,
+            -0.013387043582971358,
+            0.069179848974036,
+            -0.007221545718220735,
+            0.0037392229364642915,
+            0.0028916224964057385,
+            0.00462729158956839,
+            -0.009727339449635306,
+            0.011570145487622916,
+            -0.011287365791681587,
+            -0.012991994493845187,
+            -0.013076194601468383,
+            -0.0180830261130804,
+            -0.007272768410218907,
+            -0.004155564814758654,
+            0.004204993170487392,
+            -0.005525575983182816,
+            0.008051108079520902,
+            -0.017551645696802648,
+            -0.010800070994485792,
+            0.013007176804761548,
+            -0.0023627485852619817,
+            0.005390986239687555,
+            -0.029668390278675554,
+            -0.0007650391259819867,
+            -0.0002446394046947245,
+            0.021280292985495608,
+            0.003137771886236958,
+            -0.0019077298815862946,
+            -0.010810159675564049,
+            0.007040107942856957,
+            0.023835993694726953,
+            0.021875366566457236,
+            0.0180511566130312,
+            0.012135351541044354,
+            0.020483975362807574,
+            -0.008667079096167644,
+            0.026324312888512866,
+            -0.001552207146726937,
+            -0.00397810465151493,
+            0.0077577136352257004,
+            -0.009849600842081875,
+            -0.00816392268697345,
+            0.00677353736462267,
+            -0.017288900282504148,
+            -0.020418071729964713,
+            -0.020383762873109595,
+            0.025741408097358233,
+            -0.011755094311186617,
+            0.002737129201159793,
+            -0.008266210469668866,
+            0.027287436144516414,
+            -0.006283892307996516,
+            -0.0029440642954701632,
+            0.009428876742171362,
+            -0.03788574758692254,
+            0.008834605607057885,
+            0.006463342392514337,
+            0.026711196829464153,
+            -3.9827332346130654e-05,
+            0.005565561841469247,
+            -0.0006738085601962865,
+            -0.006104320995853506,
+            -0.010792830169647851,
+            2.4565453208679464e-05,
+            0.017983908748172493,
+            0.0035181716362611077,
+            0.016970206528324653,
+            -0.005449808667944701,
+            -0.023171388778833197,
+            0.007092509502738752,
+            -0.01555320330471131,
+            -0.023306285898839978,
+            -0.002028783456704567,
+            -0.03196436161255752,
+            0.0033412618908711163,
+            0.007527633574188688,
+            -0.01890570959143866,
+            -0.017859735533232564,
+            0.010329340599904344,
+            0.004382396539605985,
+            -0.0016746397858953155,
+            0.006643785064338525,
+            -0.03307812909454471,
+            -0.01927475592253288,
+            0.004686970010943116,
+            0.012733525168279355,
+            -0.012993441126970916,
+            0.015291832952199082,
+            -0.020491167895268936,
+            0.016451976170508816,
+            -0.012259387189535237,
+            0.018393274368854132,
+            -0.006997668664226754,
+            0.011843450716200625,
+            0.011457415642860215,
+            0.009560877634897626,
+            -0.01790001115302499,
+            0.0214879415944361,
+            0.0066829070506954115,
+            0.03900547021123392,
+            0.02512484499576633,
+            -0.012651294056575184,
+            -0.0018425680226742782,
+            -0.00011361458320146007,
+            0.014982647266052349,
+            0.0061874424115559,
+            -0.003689464024837091,
+            0.02705902205734107,
+            0.018239469373522537,
+            -0.021070704644859155,
+            0.020037508334843183,
+            -0.00973804771680803,
+            0.018876283979591423,
+            -0.013303080480396818,
+            0.010592424494628194,
+            0.004717870260467274,
+            0.01070131238869639,
+            0.009087656015197565,
+            0.006708673669681232,
+            0.025928065153829043,
+            0.03663367449576964,
+            -0.00511073108390551,
+            0.002161381031392467,
+            0.004864092591944181,
+            0.012814518637357098,
+            0.001556833001747549,
+            0.007915766323215207,
+            -0.007012785514562176,
+            0.005834786334750152,
+            -0.002311533032619821,
+            -0.00122162928193159,
+            0.019525999280692766,
+            0.000824163362963678,
+            0.012875575199417254,
+            0.013259376059185821,
+            -0.002390133687709065,
+            -0.005682644446302278,
+            0.0019262218417782657,
+            -0.018193478045923098,
+            -0.015131696847666434,
+            0.029463723011600933,
+            -0.011447039651934148,
+            -0.008083114389964213,
+            0.0020642857121204753,
+            0.012160202510990566,
+            -0.00866944494225225,
+            -0.005378300635997713,
+            0.0038280364584418335,
+            0.014427318067525318,
+            -0.01799358399064332,
+            0.003381165206832161,
+            0.005252552821854107,
+            0.019433554824232185,
+            -0.00028791985674147885,
+            -0.013487678738391956,
+            0.0005317922815671783,
+            0.014982152801567637,
+            0.011902348165369078,
+            0.0016056390299304837,
+            -0.006935777197137747,
+            -0.00446584737370504,
+            0.004586814866182051,
+            0.006269466956110759,
+            0.0027516492952993445,
+            0.008157662045793032,
+            -0.0025001885993292637,
+            -0.016648747436203294,
+            0.017735757469839637,
+            -0.0030669873770852725,
+            0.018058248627079793,
+            -0.0038238312452176745,
+            -0.0147172470474118,
+            0.0043328417868095355,
+            -0.001888507803054499,
+            -0.018556943283494755,
+            0.009407930200720534,
+            0.002560949424273885,
+            0.010411159836451958,
+            0.0052571230238294825,
+            -0.007499470181994448,
+            -6.145889479758644e-05,
+            -0.015163819825329975,
+            0.022836682800304944,
+            0.016390944816242627,
+            0.001713572907204073,
+            -0.0035687505973373878,
+            -0.031210299824387475,
+            0.0029076511444359886,
+            -0.007822231980296341,
+            0.017821806420465356,
+            0.008250605126658402,
+            0.007908773724009905,
+            0.008351355849444162,
+            -0.00743970642051451,
+            -0.002411981861500092,
+            0.012543476434788293,
+            0.0202407777715137,
+            -0.0024635818448627447,
+            0.02535771413579989,
+            -0.012533449594824386,
+            -0.0022489292647974992,
+            0.0012362917358624392,
+            0.01578939075186126,
+            -0.0008001783766371555,
+            0.006311837812042848
+          ],
+          "keyphrases": [
+            "Switch Transformers",
+            "Mixture of Experts",
+            "sparsely-activated model",
+            "training instability",
+            "multilingual settings"
+          ],
+          "title": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "47c02fce-72d4-4324-913d-f36ecb5160fd",
+        "properties": {
+          "page_content": "2 Switch Transformer 4 2.1 Simplifying Sparse Routing 5 2.2 Efficient Sparse Routing 6 2.3 Putting It All Together: The Switch Transformer 8 2.4 Improved Training and Fine-Tuning Techniques 8 ",
+          "embedding": [
+            0.03316810354590416,
+            0.0006708452710881829,
+            -0.017672521993517876,
+            0.002751410473138094,
+            -0.007615511305630207,
+            0.019335461780428886,
+            0.034437984228134155,
+            0.057870324701070786,
+            -0.03555668890476227,
+            0.048315975815057755,
+            -0.011119023896753788,
+            -0.004195145331323147,
+            0.03028063289821148,
+            0.0179899912327528,
+            -0.011209730058908463,
+            0.001036503235809505,
+            0.0006793489446863532,
+            0.0007010805420577526,
+            0.017612051218748093,
+            -0.04595762491226196,
+            0.018035344779491425,
+            0.0009675289620645344,
+            -0.059351854026317596,
+            0.008488554507493973,
+            -0.024520812556147575,
+            -0.01762716844677925,
+            0.012955817393958569,
+            0.012260406278073788,
+            -0.03809645399451256,
+            0.034891512244939804,
+            0.024278929457068443,
+            0.003327770624309778,
+            -0.005680453032255173,
+            -0.009335142560303211,
+            -0.03256339579820633,
+            -0.025382516905665398,
+            0.021603107452392578,
+            0.03576833754777908,
+            -0.049616094678640366,
+            0.04940444603562355,
+            0.019698284566402435,
+            -0.0011952384375035763,
+            -0.008284466341137886,
+            0.006708452943712473,
+            0.023900989443063736,
+            -0.014611199498176575,
+            -0.017581814900040627,
+            -0.004780953750014305,
+            0.02580581232905388,
+            -0.012744170613586903,
+            0.0013350766384974122,
+            -0.07056914269924164,
+            0.026440752670168877,
+            0.009962525218725204,
+            -0.013636111281812191,
+            0.03818715736269951,
+            0.009312466718256474,
+            0.05723538622260094,
+            0.035859040915966034,
+            0.03422633558511734,
+            0.017158521339297295,
+            0.002755190012976527,
+            -0.014467581175267696,
+            -0.03377280756831169,
+            0.01649334467947483,
+            -0.0113684656098485,
+            -0.03395421802997589,
+            0.013137228786945343,
+            -0.006243585143238306,
+            0.004047748167067766,
+            -0.05055338889360428,
+            -0.003365564625710249,
+            -0.05070456489920616,
+            -0.0017942748963832855,
+            -0.017218992114067078,
+            -0.003101005917415023,
+            0.03486127778887749,
+            0.03897327557206154,
+            -0.004614659585058689,
+            0.01395358145236969,
+            0.02783157490193844,
+            -0.01479261089116335,
+            0.005737144500017166,
+            0.00222796224988997,
+            0.030462045222520828,
+            -0.013583199121057987,
+            -0.01650846377015114,
+            -0.05324432998895645,
+            0.05019056424498558,
+            -0.004943468142300844,
+            -0.03510316088795662,
+            -0.027695516124367714,
+            -0.009131054393947124,
+            0.003350446932017803,
+            -0.011973170563578606,
+            0.05324432998895645,
+            0.029010750353336334,
+            -0.014240817166864872,
+            -0.0005191964446566999,
+            0.009592142887413502,
+            0.0290409866720438,
+            0.0069012027233839035,
+            -0.036191631108522415,
+            0.004875438753515482,
+            0.06062173470854759,
+            0.007555040530860424,
+            0.00041620753472670913,
+            0.0489509180188179,
+            0.025775576010346413,
+            0.0024244915693998337,
+            0.04616927355527878,
+            -0.026652399450540543,
+            -0.031898219138383865,
+            0.03431704267859459,
+            -0.004985041916370392,
+            -0.05448397621512413,
+            -0.03126328065991402,
+            -0.04299456998705864,
+            0.014127434231340885,
+            -0.022615989670157433,
+            0.039638452231884,
+            0.045413389801979065,
+            -0.037794098258018494,
+            -0.021119343116879463,
+            -0.07449973374605179,
+            0.006077291443943977,
+            0.033137865364551544,
+            0.005703129805624485,
+            -0.011277759447693825,
+            0.023326518014073372,
+            0.003941924776881933,
+            -0.03126328065991402,
+            0.00961481872946024,
+            -0.03731033578515053,
+            -0.0009084756602533162,
+            -0.022268284112215042,
+            -0.00158546247985214,
+            0.0030953367240726948,
+            -0.015162993222475052,
+            0.0004497497866395861,
+            0.0014427897986024618,
+            -0.017234109342098236,
+            0.02633492834866047,
+            0.012547641061246395,
+            -0.02922239899635315,
+            0.03516362980008125,
+            -0.013749493286013603,
+            0.0394570417702198,
+            0.021724049001932144,
+            0.028058340772986412,
+            -0.05230703577399254,
+            0.004981262609362602,
+            -0.021104225888848305,
+            0.037340570241212845,
+            0.032865747809410095,
+            0.016478227451443672,
+            -0.02264622412621975,
+            -0.0021051312796771526,
+            0.0319586917757988,
+            0.004334983415901661,
+            0.054090917110443115,
+            -0.02650122344493866,
+            -0.002630469389259815,
+            0.029116574674844742,
+            0.024263812229037285,
+            -0.0018538006115704775,
+            0.029796868562698364,
+            0.02043904922902584,
+            -0.042964331805706024,
+            0.01938081532716751,
+            0.025563929229974747,
+            0.05176280066370964,
+            0.04142233356833458,
+            0.02940380945801735,
+            -0.014550728723406792,
+            -0.004678909666836262,
+            -0.02931310422718525,
+            0.013280847109854221,
+            0.007120408583432436,
+            -0.016266580671072006,
+            -0.021391460672020912,
+            -0.023024166002869606,
+            -0.01171617116779089,
+            0.007422761060297489,
+            0.019773872569203377,
+            0.01912381500005722,
+            0.014641433954238892,
+            0.015427551232278347,
+            -0.011610347777605057,
+            0.0028893589042127132,
+            0.0160700511187315,
+            0.01514031644910574,
+            0.017234109342098236,
+            -0.0007648580940440297,
+            -0.011527200229465961,
+            -0.05650973692536354,
+            -0.01764228567481041,
+            0.004017512779682875,
+            0.005332747474312782,
+            -0.013855316676199436,
+            -0.013159905560314655,
+            0.024777812883257866,
+            0.02288810722529888,
+            -0.0030065206810832024,
+            -0.025065047666430473,
+            -0.03123304434120655,
+            -0.0004428996180649847,
+            -0.005578408949077129,
+            -0.009713083505630493,
+            -0.05061385780572891,
+            0.019531991332769394,
+            -0.020559990778565407,
+            0.015767699107527733,
+            0.0018566352082416415,
+            0.014951346442103386,
+            -0.010355583392083645,
+            -0.014656552113592625,
+            -0.004531512502580881,
+            0.009584583342075348,
+            -0.02343234233558178,
+            0.02879910357296467,
+            0.027861811220645905,
+            0.03906398266553879,
+            -0.02482316456735134,
+            -0.02157287299633026,
+            -0.015200787223875523,
+            -0.035919513553380966,
+            -0.02142169512808323,
+            -0.011232406832277775,
+            -0.01685616932809353,
+            -0.0354357473552227,
+            0.008140848949551582,
+            0.02254040166735649,
+            -0.007687320001423359,
+            -0.002987623680382967,
+            -0.05254891514778137,
+            0.0034184763208031654,
+            0.0072526875883340836,
+            -0.005121100693941116,
+            -0.02359863556921482,
+            -0.004996379837393761,
+            0.004758276976644993,
+            -0.004860321059823036,
+            -0.02025763690471649,
+            -0.004229160025715828,
+            0.014316405169665813,
+            -0.036100924015045166,
+            0.04299456998705864,
+            -0.016961991786956787,
+            -0.0025322046130895615,
+            -0.005873203277587891,
+            0.025548812001943588,
+            0.018035344779491425,
+            -0.0014890875900164247,
+            -0.004100659862160683,
+            0.03165633976459503,
+            -0.017869049683213234,
+            -0.04710656777024269,
+            -0.02063557878136635,
+            0.018730755895376205,
+            0.040908332914114,
+            -0.002668263390660286,
+            -0.0011149259516969323,
+            0.02306951954960823,
+            0.0037624025717377663,
+            -0.007793143391609192,
+            0.032775044441223145,
+            0.020454166457057,
+            -0.01676546223461628,
+            1.5036442164273467e-05,
+            0.04003151133656502,
+            -0.022872989997267723,
+            0.005820291582494974,
+            0.030114339664578438,
+            0.026984987780451775,
+            -0.037431277334690094,
+            0.002757079666480422,
+            0.026652399450540543,
+            -0.005812732502818108,
+            -0.014029170386493206,
+            0.02588140033185482,
+            0.006784040946513414,
+            -0.030507396906614304,
+            -0.00842052511870861,
+            0.016720110550522804,
+            -0.03002363257110119,
+            -0.033833280205726624,
+            0.057265620678663254,
+            -0.02560928277671337,
+            -0.03815692290663719,
+            -0.008435643278062344,
+            -0.014097199775278568,
+            -0.02069604955613613,
+            -0.010075907222926617,
+            0.05366762354969978,
+            -0.025639517232775688,
+            0.02359863556921482,
+            -0.04245033487677574,
+            0.021829871460795403,
+            -0.01729458011686802,
+            -0.008722878061234951,
+            0.010620142333209515,
+            0.005688012111932039,
+            -0.03712892532348633,
+            -0.05929138511419296,
+            0.033047161996364594,
+            0.029328221455216408,
+            0.045141272246837616,
+            -0.02385563589632511,
+            -0.001808447646908462,
+            0.0442039780318737,
+            0.02317534200847149,
+            0.010068348608911037,
+            -0.0412711575627327,
+            -0.004750718362629414,
+            -0.011360906064510345,
+            0.022570636123418808,
+            -0.013719257898628712,
+            -0.03220057487487793,
+            -0.008556583896279335,
+            -0.04934397712349892,
+            -0.011958053335547447,
+            -0.029267750680446625,
+            0.06845267117023468,
+            0.008866495452821255,
+            0.01175396516919136,
+            0.024006811901926994,
+            0.02783157490193844,
+            -0.03186798468232155,
+            0.0007024978403933346,
+            -0.008972319774329662,
+            -0.009440965950489044,
+            0.008186202496290207,
+            -0.00898743700236082,
+            -0.0029214839451014996,
+            -0.02589651755988598,
+            -0.016130521893501282,
+            0.003979718778282404,
+            -0.036463748663663864,
+            -0.018231874331831932,
+            -0.009365377947688103,
+            0.005846747197210789,
+            0.016387522220611572,
+            -0.016100287437438965,
+            -0.03537527844309807,
+            -0.03186798468232155,
+            -0.018821461126208305,
+            -0.012010964564979076,
+            -0.009289789944887161,
+            0.003407138166949153,
+            0.0008442256948910654,
+            0.023039283230900764,
+            0.023825401440262794,
+            0.002604013541713357,
+            0.028118809685111046,
+            -0.03867092356085777,
+            -0.01650846377015114,
+            0.01570722833275795,
+            0.015034493058919907,
+            -0.04284339025616646,
+            0.0247475765645504,
+            -0.03344022110104561,
+            -0.03939656913280487,
+            -0.021996166557073593,
+            -0.053123388439416885,
+            -0.04021292179822922,
+            0.0015656206523999572,
+            -0.04737868532538414,
+            -0.026607045903801918,
+            0.02940380945801735,
+            -0.0289502814412117,
+            0.005827850196510553,
+            0.010468965396285057,
+            -0.014150111004710197,
+            -0.010650377720594406,
+            -0.019017990678548813,
+            0.05212562158703804,
+            -0.011285318061709404,
+            -0.001033668639138341,
+            0.02763504534959793,
+            -0.017702756449580193,
+            0.005162674002349377,
+            -0.005812732502818108,
+            -0.004282071720808744,
+            0.022071754559874535,
+            0.024278929457068443,
+            0.04344809800386429,
+            -0.007097731810063124,
+            -0.08961737155914307,
+            0.016735227778553963,
+            0.040575746446847916,
+            -0.014807728119194508,
+            0.00541967386379838,
+            -0.023659106343984604,
+            0.05626785755157471,
+            0.011345788836479187,
+            -0.013537846505641937,
+            -0.0009372936910949647,
+            0.012668582610785961,
+            -0.005098423920571804,
+            -0.028315339237451553,
+            -0.016614286229014397,
+            -0.042329393327236176,
+            -0.006855849642306566,
+            -0.009055466391146183,
+            0.018095815554261208,
+            -0.02186010777950287,
+            0.022253166884183884,
+            0.007914084941148758,
+            0.008934524841606617,
+            0.013515169732272625,
+            -0.026108164340257645,
+            0.03247269243001938,
+            -0.04178515821695328,
+            0.00832981988787651,
+            0.012222612276673317,
+            -0.03924539312720299,
+            0.016432873904705048,
+            -0.02195081301033497,
+            -0.030144574120640755,
+            -0.011648141779005527,
+            0.030205044895410538,
+            0.024324283003807068,
+            -0.014452463947236538,
+            0.008148407563567162,
+            0.008284466341137886,
+            0.0229183416813612,
+            -0.03464962914586067,
+            0.003811534959822893,
+            0.030235279351472855,
+            0.03903374820947647,
+            0.0062398058362305164,
+            0.00339957932010293,
+            -0.004554189275950193,
+            -0.04450633376836777,
+            0.0027608589734882116,
+            -0.029978280887007713,
+            -0.008269349113106728,
+            -0.035224102437496185,
+            0.024157987907528877,
+            0.04580644890666008,
+            0.023281166329979897,
+            0.014974022284150124,
+            0.00031628936994820833,
+            -0.028738632798194885,
+            0.0008456429932266474,
+            -0.029857339337468147,
+            0.006198232527822256,
+            0.031112102791666985,
+            -0.0010667385067790747,
+            -0.028738632798194885,
+            -0.00025251181796193123,
+            -0.047167036682367325,
+            -0.03933610022068024,
+            0.017324814572930336,
+            -0.02780134044587612,
+            0.021527519449591637,
+            0.00541589455679059,
+            0.003002741141244769,
+            -0.022268284112215042,
+            0.005910997278988361,
+            0.01904822699725628,
+            0.02089257910847664,
+            -0.054000210016965866,
+            0.024067282676696777,
+            -0.024233577772974968,
+            -0.05309315025806427,
+            0.0020503299310803413,
+            -0.027332693338394165,
+            0.003017858834937215,
+            -0.0003706183924805373,
+            -0.026002340018749237,
+            0.00039353108149953187,
+            0.019169166684150696,
+            -0.014611199498176575,
+            0.033651866018772125,
+            -0.01649334467947483,
+            -0.09403172135353088,
+            0.0020408814307302237,
+            -0.025034811347723007,
+            -0.011338230222463608,
+            0.03885233402252197,
+            -0.0027627486269921064,
+            0.0054612476378679276,
+            -0.007226231973618269,
+            0.012781964614987373,
+            0.017612051218748093,
+            0.029615456238389015,
+            0.00965261273086071,
+            0.00511354161426425,
+            0.030068986117839813,
+            0.005665335804224014,
+            9.962288459064439e-05,
+            -0.0040061743929982185,
+            -0.020393695682287216,
+            -0.05152091756463051,
+            -0.022872989997267723,
+            -0.05396997556090355,
+            0.020030872896313667,
+            0.0071733202785253525,
+            -0.030129456892609596,
+            -0.0049888212233781815,
+            0.04870903491973877,
+            0.020484402775764465,
+            -0.02176940254867077,
+            -0.007056158501654863,
+            0.019108697772026062,
+            0.024157987907528877,
+            0.005593526642769575,
+            -0.001357753062620759,
+            0.010620142333209515,
+            0.0015826279995962977,
+            0.011897582560777664,
+            -0.02195081301033497,
+            -0.03090045601129532,
+            0.006889864336699247,
+            -0.028390927240252495,
+            0.022782282903790474,
+            0.006470350082963705,
+            -0.0012490949593484402,
+            0.00991717167198658,
+            0.006421217694878578,
+            -0.0027117265854030848,
+            0.023976577445864677,
+            -0.01087714172899723,
+            0.015004257671535015,
+            0.037854570895433426,
+            0.014467581175267696,
+            0.05294197425246239,
+            -0.0076684230007231236,
+            -0.010854465886950493,
+            0.008397849276661873,
+            -0.0028251090552657843,
+            -0.017778344452381134,
+            -0.00031487210071645677,
+            -0.007675981614738703,
+            0.0089949956163764,
+            0.013515169732272625,
+            -0.0398198626935482,
+            -0.011958053335547447,
+            -0.002184499055147171,
+            -0.01649334467947483,
+            -0.0004769143124576658,
+            -0.0031085647642612457,
+            8.887518924893811e-05,
+            0.03906398266553879,
+            -0.0022090652491897345,
+            -0.016977109014987946,
+            -0.00726402597501874,
+            0.02000063844025135,
+            -0.05245821177959442,
+            -0.00741520244628191,
+            0.003911689389497042,
+            0.03359139710664749,
+            -0.011217288672924042,
+            0.005767379887402058,
+            -0.028904927894473076,
+            0.01694687455892563,
+            -0.01394602283835411,
+            -0.026304693892598152,
+            -0.0039494833908975124,
+            0.0024887414183467627,
+            0.01416522916406393,
+            0.01518566906452179,
+            -0.01079399511218071,
+            0.009849142283201218,
+            0.017869049683213234,
+            0.03331927955150604,
+            -0.021300755441188812,
+            0.011255082674324512,
+            -0.004013733472675085,
+            0.03099116124212742,
+            -0.007093952503055334,
+            -0.00710907019674778,
+            0.017445756122469902,
+            0.015253698453307152,
+            -0.00209379312582314,
+            0.010998083278536797,
+            -0.013689023442566395,
+            -0.0068785264156758785,
+            0.01684105023741722,
+            -0.03168657422065735,
+            0.015321727842092514,
+            0.01139870099723339,
+            0.005332747474312782,
+            -0.009191525168716908,
+            -0.02297881245613098,
+            -0.020998401567339897,
+            0.003896571695804596,
+            -0.016357285901904106,
+            0.03368210047483444,
+            0.013809964060783386,
+            0.007354731671512127,
+            -0.020136697217822075,
+            -0.0016043595969676971,
+            0.011580112390220165,
+            0.009433407336473465,
+            0.005030394531786442,
+            0.03537527844309807,
+            -0.0637359693646431,
+            0.02684892900288105,
+            0.011232406832277775,
+            -0.004818747751414776,
+            -0.01417278777807951,
+            0.0011470509925857186,
+            -0.004542850889265537,
+            0.0102270832285285,
+            0.01056723017245531,
+            0.02368934266269207,
+            0.0020087563898414373,
+            0.014633875340223312,
+            -0.03168657422065735,
+            0.035586923360824585,
+            -0.009131054393947124,
+            0.0034241455141454935,
+            -0.0006117919692769647,
+            -0.012003405950963497,
+            0.00869264267385006,
+            -0.009645054116845131,
+            0.004576865583658218,
+            0.08411455154418945,
+            0.013568081893026829,
+            0.0101363779976964,
+            0.005291174165904522,
+            -0.027967633679509163,
+            0.03546598181128502,
+            -0.014860640279948711,
+            -0.021376343443989754,
+            0.0014824735699221492,
+            0.0035658732522279024,
+            0.009622378274798393,
+            -0.02595698833465576,
+            -0.007921643555164337,
+            -0.03897327557206154,
+            -0.010778876952826977,
+            -0.018307462334632874,
+            0.010559671558439732,
+            -0.0035450865980237722,
+            0.036826569586992264,
+            -0.01948663778603077,
+            -0.01913893222808838,
+            0.03437751531600952,
+            0.0582633838057518,
+            0.03386351466178894,
+            -0.00868508405983448,
+            0.008707760833203793,
+            0.004512615501880646,
+            -0.013106994330883026,
+            0.02124028466641903,
+            0.015813050791621208,
+            0.008155967108905315,
+            0.026879163458943367,
+            -0.010091024450957775,
+            0.03625210002064705,
+            0.023900989443063736,
+            -0.017082933336496353,
+            -0.02692451700568199,
+            -0.0052382624708116055,
+            -0.02604769356548786,
+            -0.0040666451677680016,
+            -0.00307266041636467,
+            0.006965452805161476,
+            0.005820291582494974,
+            1.740595507726539e-05,
+            0.01718875765800476,
+            -0.023326518014073372,
+            -0.018035344779491425,
+            0.007400084752589464,
+            -0.022752048447728157,
+            0.01746087335050106,
+            0.0035791012924164534,
+            -0.03138422220945358,
+            -0.009153731167316437,
+            0.028300222009420395,
+            0.000411955697927624,
+            -0.030386457219719887,
+            -0.006039496976882219,
+            0.005521717946976423,
+            -0.028103692457079887,
+            0.0013946022372692823,
+            -0.025911634787917137,
+            -0.0071959965862333775,
+            -0.004474821500480175,
+            0.008881613612174988,
+            0.009395613335072994,
+            -0.00038762573967687786,
+            -0.02394634112715721,
+            -0.015200787223875523,
+            -0.029721280559897423,
+            0.027075693011283875,
+            -0.02631981112062931,
+            0.015079845674335957,
+            -0.02553369477391243,
+            0.02246481366455555,
+            -0.01737016811966896,
+            -0.007449217140674591,
+            -0.004365218337625265,
+            0.030855102464556694,
+            0.061468325555324554,
+            -0.030598104000091553,
+            0.0010865803342312574,
+            0.010098583996295929,
+            0.003289976390078664,
+            -0.017869049683213234,
+            0.03649398311972618,
+            0.02589651755988598,
+            -3.9772385207470506e-05,
+            -0.0032918660435825586,
+            -0.013016288168728352,
+            -0.041906099766492844,
+            -0.0024263812229037285,
+            0.011648141779005527,
+            -0.0197285208851099,
+            0.02210198901593685,
+            0.031081868335604668,
+            0.01684105023741722,
+            -0.03044692613184452,
+            -0.023659106343984604,
+            -0.03625210002064705,
+            0.032593630254268646,
+            -0.013122111558914185,
+            0.043478332459926605,
+            0.014807728119194508,
+            -0.025140635669231415,
+            -0.010899818502366543,
+            -0.008722878061234951,
+            0.014399551786482334,
+            -0.02263110689818859,
+            0.027151281014084816,
+            -0.012177258729934692,
+            0.02192057855427265,
+            -0.007033482193946838,
+            -0.003972159698605537,
+            0.011617906391620636,
+            -0.007097731810063124,
+            -0.02043904922902584,
+            -0.0012557089794427156,
+            0.005975247360765934,
+            0.017581814900040627,
+            -0.0066064088605344296,
+            0.034619394689798355,
+            0.03062833845615387,
+            -0.0040326304733753204,
+            0.023356754332780838,
+            0.02774086967110634,
+            0.00572202680632472,
+            0.01966805011034012,
+            0.011557435616850853,
+            -0.001473969896323979,
+            -0.021542636677622795,
+            0.001912381500005722,
+            -0.032865747809410095,
+            0.010544554330408573,
+            0.005465026944875717,
+            -0.02279740199446678,
+            0.021542636677622795,
+            0.01702246256172657,
+            -0.019184285774827003,
+            -0.00036636655568145216,
+            -0.024898752570152283,
+            0.0007379298331215978,
+            0.02668263390660286,
+            -0.025669753551483154,
+            0.027362927794456482,
+            0.020650696009397507,
+            0.010363142006099224,
+            -0.024535929784178734,
+            -0.005619982723146677,
+            0.0035299689043313265,
+            -0.007698658388108015,
+            0.024157987907528877,
+            0.027952516451478004,
+            0.03174704313278198,
+            -0.016629403457045555,
+            0.0036811453755944967,
+            0.013061640784144402,
+            0.0036282336805015802,
+            -0.019320344552397728,
+            -0.0264861062169075,
+            0.013832640834152699,
+            0.005907217971980572,
+            0.0009306797292083502,
+            0.020862342789769173,
+            0.003520520403981209,
+            -0.004195145331323147,
+            -0.0012292531318962574,
+            -0.007127967197448015,
+            -0.029948044568300247,
+            -0.0075248051434755325,
+            0.02545810490846634,
+            0.012139464728534222,
+            0.03017481043934822,
+            -0.026380281895399094,
+            0.011239965446293354,
+            0.042510803788900375,
+            -0.018987756222486496,
+            -0.009380496107041836,
+            0.017869049683213234,
+            0.003953262697905302,
+            -0.015321727842092514,
+            -0.014308846555650234,
+            -0.009818906895816326,
+            -0.004414350725710392,
+            0.03310763090848923,
+            -0.00987937767058611,
+            -0.046108800917863846,
+            -0.004021292086690664,
+            -0.033289045095443726,
+            -0.006746246945112944,
+            7.907234248705208e-05,
+            0.023659106343984604,
+            -0.02052975445985794,
+            0.007929202169179916,
+            0.025473223999142647,
+            -0.005427232943475246,
+            0.0356171615421772,
+            0.012388906441628933,
+            -0.02580581232905388,
+            0.011821994557976723,
+            -0.025155752897262573,
+            0.01417278777807951,
+            -0.006825614254921675,
+            0.005313850473612547,
+            -0.013099434785544872,
+            -0.04411327466368675,
+            -0.05170232802629471,
+            0.0030896677635610104,
+            -0.02518598921597004,
+            0.04142233356833458,
+            0.01938081532716751,
+            -0.02105887234210968,
+            0.0025643296539783478,
+            -0.017355050891637802,
+            0.017324814572930336,
+            0.04680421203374863,
+            0.011678377166390419,
+            -0.009667730890214443,
+            0.01966805011034012,
+            -0.014059404842555523,
+            0.02376493066549301,
+            0.020121578127145767,
+            0.003779409918934107,
+            -0.03477057069540024,
+            -0.007793143391609192,
+            -0.06434067338705063,
+            0.0033806823194026947,
+            0.013076758943498135,
+            0.008246672339737415,
+            0.03809645399451256,
+            -0.0009150896221399307,
+            0.03099116124212742,
+            0.02692451700568199,
+            0.024324283003807068,
+            0.001319014118053019,
+            -0.03368210047483444,
+            -0.0024055945686995983,
+            0.04641115665435791,
+            0.038640689104795456,
+            -0.012842435389757156,
+            -0.014648993499577045,
+            -0.016735227778553963,
+            -0.005170233082026243,
+            0.026470987126231194,
+            -0.018972638994455338,
+            0.011648141779005527,
+            -0.008019908331334591,
+            -0.004353880416601896,
+            -0.010657936334609985,
+            0.007154423277825117,
+            -0.020862342789769173,
+            -0.03885233402252197,
+            -0.0290409866720438,
+            -0.03229127824306488,
+            0.02615351788699627,
+            0.02677334100008011,
+            -0.042601510882377625,
+            -0.008239113725721836,
+            -0.04226892068982124,
+            0.010771318338811398,
+            0.021451931446790695,
+            0.0014409000286832452,
+            0.01665963977575302,
+            -0.013167464174330235,
+            0.03791503980755806,
+            0.02352304756641388,
+            0.0016922308132052422,
+            -0.005986585281789303,
+            -0.007543702609837055,
+            -0.00833737850189209,
+            -0.007936760783195496,
+            0.00445592449977994,
+            0.03386351466178894,
+            -0.01888193190097809,
+            -0.008193761110305786,
+            0.004743159748613834,
+            0.0003840825520455837,
+            -0.0012216942850500345,
+            0.0161607563495636,
+            0.011965611949563026,
+            -0.004784733057022095,
+            -0.0023848076816648245,
+            -0.01364366989582777,
+            -0.004055306781083345,
+            -0.005533056333661079,
+            0.007071276195347309,
+            -0.009743318893015385,
+            -0.006666879169642925,
+            0.031172573566436768,
+            0.0013246831949800253,
+            -0.026289576664566994,
+            -0.014815287664532661,
+            0.0032351750414818525,
+            -0.0013624773127958179,
+            -0.009456084109842777,
+            0.03380304202437401,
+            0.052609387785196304,
+            -0.0049548065289855,
+            -0.023659106343984604,
+            0.010121259838342667,
+            -0.001774432952515781,
+            0.002562440000474453,
+            0.0005012442707084119,
+            0.01876099221408367,
+            0.02668263390660286,
+            -0.0038927923887968063,
+            -0.004395453725010157,
+            -0.01913893222808838,
+            0.008624613285064697,
+            0.022600872442126274,
+            0.025140635669231415,
+            -0.022041520103812218,
+            0.014921111054718494,
+            0.0019596240017563105,
+            -0.0197285208851099,
+            0.015767699107527733,
+            0.0160700511187315,
+            -0.007233790587633848,
+            0.012509847059845924,
+            -0.005174012389034033,
+            0.00825423188507557,
+            -0.010431171394884586,
+            0.010990524664521217,
+            0.01975875534117222,
+            0.008299584500491619,
+            -0.010242201387882233,
+            -0.017687639221549034,
+            0.007517246529459953,
+            0.005623762030154467,
+            -0.0018358484376221895,
+            0.017173638567328453,
+            -0.005706909112632275,
+            0.004890556447207928,
+            -0.005427232943475246,
+            -0.024551047012209892,
+            -0.004176247864961624,
+            -0.004229160025715828,
+            -0.010809112340211868,
+            0.009758437052369118,
+            -0.026365164667367935,
+            0.016901521012187004,
+            0.023235812783241272,
+            -0.0057787178084254265,
+            0.0007317882846109569,
+            -0.0089042903855443,
+            0.001393657410517335,
+            -0.017475992441177368,
+            0.017339933663606644,
+            -0.029539868235588074,
+            -0.02615351788699627,
+            0.04311550781130791,
+            -0.025684870779514313,
+            -0.026395399123430252,
+            0.0006264371913857758,
+            -0.01824699155986309,
+            -0.0022525284439325333,
+            0.009675289504230022,
+            0.006859628949314356,
+            -0.015419992618262768,
+            -0.017143404111266136,
+            0.015412434004247189,
+            0.013689023442566395,
+            -0.03238198533654213,
+            0.022963695228099823,
+            -0.014958905056118965,
+            -0.019169166684150696,
+            -0.00019806470663752407,
+            -0.0031936015002429485,
+            -0.03570786491036415,
+            -0.02562440000474453,
+            -0.006088629364967346,
+            -0.005041732918471098,
+            -0.0012708266731351614,
+            0.032865747809410095,
+            0.0005503766005858779,
+            -0.0008706816006451845,
+            -0.01605493389070034,
+            -0.009592142887413502,
+            -0.04411327466368675,
+            0.006436335388571024,
+            0.01650846377015114,
+            -0.021739166229963303,
+            -0.01334131695330143,
+            -0.0006311614997684956,
+            0.059442561119794846,
+            -0.02001575566828251,
+            0.005139997694641352,
+            0.01139870099723339,
+            -0.018715638667345047,
+            -0.030764397233724594,
+            0.014679228886961937,
+            0.03289598599076271,
+            0.0009817017707973719,
+            -0.02201128378510475,
+            -0.014323963783681393,
+            -0.011376024223864079,
+            -0.0024396092630922794,
+            0.017778344452381134,
+            0.0068180556409060955,
+            -0.022510165348649025,
+            0.025140635669231415,
+            -0.002588895848020911,
+            0.0007237570243887603,
+            -0.03192845731973648,
+            -4.650445771403611e-05,
+            0.010151495225727558,
+            0.02299392968416214,
+            -0.013099434785544872,
+            0.0008319426560774446,
+            -0.005676673725247383,
+            -0.002031432930380106,
+            -0.00829202588647604,
+            -0.0070901731960475445,
+            -0.026516340672969818,
+            -0.024263812229037285,
+            0.026108164340257645,
+            -0.005593526642769575,
+            0.012555199675261974,
+            -0.03271457180380821,
+            -0.01483040489256382,
+            0.032261043787002563,
+            0.0014616867993026972,
+            -0.025231340900063515,
+            0.019365696236491203,
+            -0.025503458455204964,
+            -0.00900255423039198,
+            0.006005482282489538,
+            0.0051551153883337975,
+            0.006859628949314356,
+            0.019259873777627945,
+            0.015949109569191933,
+            0.023190459236502647,
+            4.854179860558361e-05,
+            0.015933992341160774,
+            -0.022328754886984825,
+            -0.018201638013124466,
+            0.02798275090754032,
+            -0.0021278078202158213,
+            0.008564143441617489,
+            0.02808857522904873,
+            0.0037246085703372955,
+            -0.015737462788820267,
+            8.344228990608826e-05,
+            -0.004561747889965773,
+            0.010302671231329441,
+            0.0010998082580044866,
+            0.014958905056118965,
+            -0.005994144361466169,
+            -0.0002851092431228608,
+            0.015858404338359833,
+            0.01452805195003748,
+            0.007921643555164337,
+            -0.011572553776204586,
+            -0.01894240267574787,
+            0.0021693813614547253,
+            -0.0005267552915029228,
+            0.04281315580010414,
+            0.007222452666610479,
+            0.004985041916370392,
+            -0.05433279648423195,
+            0.004678909666836262,
+            -0.01842840388417244,
+            0.004788512364029884,
+            0.024460341781377792,
+            0.002974395640194416,
+            -0.00863973144441843,
+            0.011224847286939621,
+            -0.0015457787085324526,
+            -0.0029233735986053944,
+            0.006254923529922962,
+            -0.043024804443120956,
+            -0.02246481366455555,
+            0.01623634621500969,
+            0.008896730840206146,
+            0.022767165675759315,
+            0.016478227451443672,
+            -0.004969924222677946,
+            0.00691254111006856,
+            0.05956350266933441,
+            0.0008007625001482666,
+            -0.009425848722457886,
+            -0.027060575783252716,
+            0.00016924670489970595,
+            -0.006988129112869501,
+            -0.03422633558511734,
+            0.03906398266553879,
+            0.008435643278062344,
+            0.02001575566828251,
+            0.009690407663583755,
+            0.02439987100660801,
+            -0.011852229945361614,
+            0.011746406555175781,
+            0.014580964110791683,
+            -0.009856700897216797,
+            -0.006825614254921675,
+            -0.0015542823821306229,
+            -0.015004257671535015,
+            0.010990524664521217,
+            -0.0031180132646113634,
+            0.014399551786482334,
+            -0.0074454378336668015,
+            0.03703821823000908,
+            -0.0315958671271801,
+            0.024944106116890907,
+            0.02482316456735134,
+            0.007449217140674591,
+            -0.02421845868229866,
+            -0.007407643832266331,
+            0.012827317230403423,
+            0.017657402902841568,
+            0.021346107125282288,
+            0.025034811347723007,
+            0.015004257671535015,
+            0.03721962869167328,
+            -0.03344022110104561,
+            0.015676992014050484,
+            0.009191525168716908,
+            0.028920045122504234,
+            0.013106994330883026,
+            0.014301287941634655,
+            0.02052975445985794,
+            -0.006621526554226875,
+            -0.021618224680423737,
+            -0.0015608962858095765,
+            0.006595070473849773,
+            -0.00929734855890274,
+            -0.01545022800564766,
+            -0.01762716844677925,
+            -0.0020068667363375425,
+            0.00934270117431879,
+            -0.029721280559897423,
+            -0.020590225234627724,
+            -0.022842753678560257,
+            0.0077629080042243,
+            0.01675034500658512,
+            -0.026607045903801918,
+            0.010159053839743137,
+            0.011557435616850853,
+            0.016206109896302223,
+            0.009796231053769588,
+            0.009040349163115025,
+            0.004285851027816534,
+            0.01588863879442215,
+            -0.013673905283212662,
+            0.019637813791632652,
+            0.010476524941623211,
+            0.011625465005636215,
+            -0.012169700115919113,
+            -0.035405512899160385,
+            0.007464334834367037,
+            0.01703757978975773,
+            0.0003247930435463786,
+            -0.018549343571066856,
+            -0.0040061743929982185,
+            -0.0014862529933452606,
+            -0.0291316919028759,
+            -0.0013294074451550841,
+            -0.0020068667363375425,
+            0.01329596433788538,
+            -0.008760672062635422,
+            -0.0009996539447456598,
+            0.004656233359128237,
+            -0.020212285220623016,
+            -0.0032276161946356297,
+            -0.010559671558439732,
+            0.010385818779468536,
+            0.001661050715483725,
+            -0.005782497115433216,
+            0.010506759397685528,
+            -0.018927285447716713,
+            -0.01694687455892563,
+            0.023039283230900764,
+            0.006375864613801241,
+            -0.019320344552397728,
+            0.01167081855237484,
+            -0.016568932682275772,
+            -0.005563291721045971,
+            -0.005971467588096857,
+            -0.03168657422065735,
+            0.004690248053520918,
+            0.018398167565464973,
+            -0.005638879723846912,
+            0.011633023619651794,
+            0.009577024728059769,
+            0.018473755568265915,
+            -0.013507611118257046,
+            0.023039283230900764,
+            0.004236718639731407,
+            0.004391674418002367,
+            -0.005087085999548435,
+            -0.008662407286465168,
+            0.005162674002349377,
+            -0.0009509940282441676,
+            -1.5781988622620702e-05,
+            0.0223741065710783,
+            -0.0438411571085453,
+            0.005929894279688597,
+            0.006270041223615408,
+            0.009992759674787521,
+            0.022661343216896057,
+            0.010249760001897812,
+            -0.022600872442126274,
+            0.0049283504486083984,
+            0.005899658892303705,
+            0.006942776031792164,
+            -4.81284259876702e-05,
+            0.0014049956807866693,
+            -0.00842052511870861,
+            -0.011746406555175781,
+            0.016614286229014397,
+            0.023099754005670547,
+            -0.01148940622806549,
+            0.014853081665933132,
+            -0.024883635342121124,
+            -0.0031917118467390537,
+            0.013129670172929764,
+            0.03676610067486763,
+            -0.035859040915966034,
+            -0.010605024173855782,
+            0.020302990451455116,
+            0.005064409226179123,
+            0.011942935176193714,
+            -0.03694751113653183,
+            -0.02385563589632511,
+            -0.02193569578230381,
+            0.01788416877388954,
+            0.0003573904687073082,
+            -0.0022487489040941,
+            -0.01170861255377531,
+            -0.02010646089911461,
+            -0.010091024450957775,
+            0.012283082120120525,
+            0.00025794474640861154,
+            -0.011897582560777664,
+            -0.001873642555437982,
+            0.006069732364267111,
+            0.03555668890476227,
+            -0.011995847336947918,
+            0.02113446034491062,
+            -0.008322261273860931,
+            -0.019456403329968452,
+            -0.020514637231826782,
+            -0.010506759397685528,
+            0.0010327238123863935,
+            0.005037953611463308,
+            0.012388906441628933,
+            0.010234642773866653,
+            0.005480144638568163,
+            -0.007241349667310715,
+            0.004357659723609686,
+            -0.0060735116712749004,
+            0.0019426167709752917,
+            -0.008246672339737415,
+            -0.025911634787917137,
+            0.02492898888885975,
+            0.00829202588647604,
+            0.008586819283664227,
+            -0.006065953057259321,
+            0.0032295058481395245,
+            -0.022147342562675476,
+            0.004406792111694813,
+            0.018564462661743164,
+            -0.003560204291716218,
+            0.002891248557716608,
+            0.013424464501440525,
+            -0.02343234233558178,
+            -0.008042584173381329,
+            0.006504364777356386,
+            0.00034912300179712474,
+            0.010703288950026035,
+            0.014271052554249763,
+            -0.017959756776690483,
+            0.006077291443943977,
+            0.010552112944424152,
+            -0.009577024728059769,
+            -0.005714467726647854,
+            0.009176407940685749,
+            0.006844511721283197,
+            -0.03658469021320343,
+            0.023538164794445038,
+            -0.0037321674171835184,
+            -0.0027325134724378586,
+            0.02774086967110634,
+            -0.019879696890711784,
+            -0.019017990678548813,
+            -0.005166453309357166,
+            0.015918875113129616,
+            -0.0013719258131459355,
+            0.01048408355563879,
+            -0.0034865057095885277,
+            0.003429814474657178,
+            -0.0005040788091719151,
+            0.004818747751414776,
+            -0.01057478878647089,
+            0.00075399229535833,
+            -0.008140848949551582,
+            0.001432396355085075,
+            0.009289789944887161,
+            -0.007604172918945551,
+            -0.008186202496290207,
+            0.003972159698605537,
+            -0.0011952384375035763,
+            0.003414697013795376,
+            -0.007014585193246603,
+            0.018972638994455338,
+            -0.03126328065991402,
+            -0.0003994363942183554,
+            0.00568423280492425,
+            -0.009289789944887161,
+            0.023296283558011055,
+            0.011013200506567955,
+            0.010385818779468536,
+            0.005117321386933327,
+            0.012131906114518642,
+            -0.013258170336484909,
+            0.0008073764620348811,
+            -0.006304055918008089,
+            -0.009335142560303211,
+            0.004686468280851841,
+            0.026213988661766052,
+            0.016100287437438965,
+            0.026093047112226486,
+            0.018201638013124466,
+            -0.0002548739721532911,
+            0.004002395085990429,
+            0.004731821361929178,
+            -0.0054234531708061695,
+            0.01720387488603592,
+            -0.019078461453318596,
+            0.0028496752493083477,
+            0.007868731394410133,
+            -0.023825401440262794,
+            -0.003443042514845729,
+            -0.003862557001411915,
+            0.031535398215055466,
+            0.01851910911500454,
+            0.015102522447705269,
+            0.010763759724795818,
+            0.006738687865436077,
+            -0.0160700511187315,
+            0.03044692613184452,
+            -0.007861172780394554,
+            0.016463110223412514,
+            0.010884701274335384,
+            -0.007177099585533142,
+            0.0030839985702186823,
+            0.010703288950026035,
+            0.002562440000474453,
+            -0.022479930892586708,
+            0.020318107679486275,
+            -0.018624933436512947,
+            0.0076684230007231236,
+            0.009501436725258827,
+            -0.008866495452821255,
+            0.023296283558011055,
+            -0.01079399511218071,
+            -0.00863973144441843,
+            -0.005389438476413488,
+            -0.005272277165204287,
+            0.05578409135341644,
+            0.002758969319984317,
+            -0.019320344552397728,
+            -0.024611517786979675,
+            0.024551047012209892,
+            0.0039834980852901936,
+            -0.0061906734481453896,
+            0.021965930238366127,
+            0.012010964564979076,
+            -0.03480080887675285,
+            0.015586286783218384,
+            0.004225380253046751,
+            -0.008012348785996437,
+            0.020151814445853233,
+            0.0031028955709189177,
+            -0.006141541060060263,
+            0.030401574447751045,
+            -0.003520520403981209,
+            -0.001451293472200632,
+            0.008737996220588684,
+            -0.001432396355085075,
+            -0.017869049683213234,
+            0.05327456444501877,
+            0.001982300542294979,
+            0.025216223672032356,
+            -0.0056124236434698105,
+            -0.009100819006562233,
+            -0.017566697672009468,
+            -0.011028318665921688,
+            0.024233577772974968,
+            -0.012184818275272846,
+            -0.015110081061720848,
+            0.0011602789163589478,
+            0.013696582056581974,
+            0.0032654101960361004,
+            -0.0013568082358688116,
+            -0.006243585143238306,
+            0.03431704267859459,
+            -0.005136218387633562,
+            -0.018095815554261208,
+            -0.016085168346762657,
+            -0.0024887414183467627,
+            0.020998401567339897,
+            0.019017990678548813,
+            0.015737462788820267,
+            -0.02069604955613613,
+            -0.004890556447207928,
+            -0.013311082497239113,
+            -0.03362163156270981,
+            0.010620142333209515,
+            0.0004606155853252858,
+            0.023825401440262794,
+            -0.02579069323837757,
+            0.006825614254921675,
+            0.003658468835055828,
+            -0.04589715600013733,
+            0.019350579008460045,
+            0.014467581175267696,
+            0.00031416345154866576,
+            -0.014762375503778458,
+            0.0025529915001243353,
+            -0.0028950280975550413,
+            0.006266261916607618,
+            -0.012600553222000599,
+            -0.017869049683213234,
+            0.0024660651106387377,
+            0.01056723017245531,
+            0.023129988461732864,
+            0.001661050715483725,
+            -0.021557753905653954,
+            -0.0009731980971992016,
+            0.00042234905413351953,
+            -0.013159905560314655,
+            0.0009486319031566381,
+            0.02124028466641903,
+            0.02255551889538765,
+            0.02211710810661316,
+            -0.004274512641131878,
+            0.005703129805624485,
+            0.013144788332283497,
+            -0.007573937531560659,
+            -0.026531457901000977,
+            -0.008239113725721836,
+            -0.024112636223435402,
+            0.021633343771100044,
+            0.004414350725710392,
+            -0.00011361851647961885,
+            0.026440752670168877,
+            -0.023900989443063736,
+            0.012086553499102592,
+            0.009629936888813972,
+            -0.001953955041244626,
+            0.010053230449557304,
+            -0.00024920483701862395,
+            0.019244756549596786,
+            -0.025488341227173805,
+            0.012849994003772736,
+            -0.002855344209820032,
+            0.0067349085584282875,
+            0.015933992341160774,
+            -0.011942935176193714,
+            0.009486319497227669,
+            0.00047148141311481595,
+            -0.00418758625164628,
+            -0.011678377166390419,
+            -0.000992095097899437,
+            0.003769961418583989,
+            0.03079463355243206,
+            -0.011307994835078716,
+            0.004130895249545574,
+            0.015737462788820267,
+            0.009886936284601688,
+            0.014150111004710197,
+            0.01782369799911976,
+            0.0016667197924107313,
+            0.019002873450517654,
+            -0.014633875340223312,
+            0.006591291166841984,
+            0.009516553953289986,
+            -0.01522346306592226,
+            -0.006829394027590752,
+            -0.026289576664566994,
+            -0.0005437626386992633,
+            -0.003998615778982639,
+            -0.011610347777605057,
+            -0.009047907777130604,
+            -0.028496751561760902,
+            -0.007800702471286058,
+            -0.026213988661766052,
+            -0.007313158363103867,
+            -0.008019908331334591,
+            0.000944380066357553,
+            -0.000724229437764734,
+            0.01782369799911976,
+            -0.004678909666836262,
+            -0.004875438753515482,
+            0.021542636677622795,
+            -0.011421376839280128,
+            -0.00038644467713311315,
+            -0.012041199952363968,
+            0.0013596427161246538,
+            -0.0072526875883340836,
+            0.011549877002835274,
+            0.005752262193709612,
+            -0.00023007158597465605,
+            -0.002220403403043747,
+            0.0080803781747818,
+            -0.005465026944875717,
+            0.019002873450517654,
+            -0.03386351466178894,
+            0.01975875534117222,
+            0.010053230449557304,
+            -0.004678909666836262,
+            -0.020166931673884392,
+            -0.02052975445985794,
+            -0.021739166229963303,
+            0.005408335942775011,
+            0.026712870225310326,
+            -0.013386670500040054,
+            -0.006304055918008089,
+            -0.00480363005772233,
+            -0.01079399511218071,
+            0.0024944106116890907,
+            0.015231022611260414,
+            -0.027257105335593224,
+            0.02387075312435627,
+            -0.007914084941148758,
+            0.009902054443955421,
+            -0.008851378224790096,
+            0.003518630750477314,
+            0.00126704724971205,
+            -0.009803789667785168,
+            -0.0031917118467390537,
+            -0.018050462007522583,
+            0.010302671231329441,
+            0.025291811674833298,
+            -0.008934524841606617,
+            -0.025760458782315254,
+            0.007997231557965279,
+            -0.011005641892552376,
+            -0.012668582610785961,
+            -0.015556051395833492,
+            0.0027816458605229855,
+            0.015858404338359833,
+            0.002666373737156391,
+            0.022525284439325333,
+            0.0009061135351657867,
+            -0.0053176297806203365,
+            -0.006198232527822256,
+            0.0136058758944273,
+            0.010544554330408573,
+            -0.033289045095443726,
+            -0.008783348836004734,
+            0.0290409866720438,
+            -0.020061109215021133,
+            -0.004395453725010157,
+            0.003435483667999506,
+            0.026455869898200035,
+            -0.01171617116779089,
+            0.009629936888813972,
+            0.0003321156546007842,
+            0.001567510305903852,
+            -0.02009134367108345,
+            0.0010034333681687713,
+            0.07752326130867004,
+            0.0179899912327528,
+            -0.017128286883234978,
+            -0.00349784386344254,
+            -0.013658788055181503,
+            -0.012842435389757156,
+            0.00894964300096035,
+            -0.025382516905665398,
+            0.00860193744301796,
+            -0.00961481872946024,
+            -0.021890342235565186,
+            0.015752580016851425,
+            -0.00013605876301880926,
+            0.020408814772963524,
+            -0.03359139710664749,
+            0.012215052731335163,
+            0.015737462788820267,
+            0.01631193421781063,
+            0.01676546223461628,
+            0.0009358763927593827,
+            -0.013930905610322952,
+            -7.009624823695049e-05,
+            0.00649680569767952,
+            0.017566697672009468,
+            0.025397635996341705,
+            0.011534759774804115,
+            -0.012555199675261974,
+            0.03126328065991402,
+            0.005204247776418924,
+            0.0229183416813612,
+            0.013001170009374619,
+            -0.0021807195153087378,
+            -0.007808261085301638,
+            -0.007604172918945551,
+            0.014883317053318024,
+            0.01833769679069519,
+            -0.0036471306812018156,
+            -0.014490257948637009,
+            0.005960129667073488,
+            0.004985041916370392,
+            -0.006481688003987074,
+            -0.005438570864498615,
+            -0.0011205950286239386,
+            0.013840199448168278,
+            -0.009395613335072994,
+            0.026727987453341484,
+            0.00771755538880825,
+            0.0014087751042097807,
+            -0.015246139839291573,
+            -0.0015911315567791462,
+            -0.006617746781557798,
+            0.007728893309831619,
+            0.004796071443706751,
+            0.009350260719656944,
+            -0.0005513214273378253,
+            0.00126704724971205,
+            0.012925582006573677,
+            -0.03480080887675285,
+            -0.0007577717187814415,
+            0.023114871233701706,
+            -0.0026210208889096975,
+            -0.006889864336699247,
+            -0.009592142887413502,
+            -0.018352815881371498,
+            0.001622311770915985,
+            0.002823219168931246,
+            0.010582348331809044,
+            -0.0019917490426450968,
+            0.010703288950026035,
+            0.02885957434773445,
+            0.006727349944412708,
+            -0.016357285901904106,
+            0.0228276364505291,
+            0.006258702836930752,
+            0.008722878061234951,
+            0.014225699007511139,
+            -0.006236026529222727,
+            0.006118864752352238,
+            -0.010242201387882233,
+            -0.0066290851682424545,
+            0.02359863556921482,
+            0.01044628955423832,
+            -0.011549877002835274,
+            -0.004765836056321859,
+            0.014618758112192154,
+            -0.0265768114477396,
+            0.012744170613586903,
+            0.0019218300003558397,
+            0.013719257898628712,
+            1.0371232747274917e-05,
+            -0.016281697899103165,
+            -0.020076226443052292,
+            0.00599036505445838,
+            0.0032502925023436546,
+            -0.013054082170128822,
+            -0.018972638994455338,
+            0.0008801301009953022,
+            -0.014709463343024254,
+            -0.030159691348671913,
+            -0.021028637886047363,
+            0.014732140116393566,
+            0.0007105291006155312,
+            0.0076684230007231236,
+            0.010045671835541725,
+            -0.0025246459990739822,
+            0.007876290008425713,
+            0.02754434011876583,
+            0.014051846228539944,
+            0.018912168219685555,
+            -0.00829202588647604,
+            -0.002518976805731654,
+            0.011043435893952847,
+            0.011542318388819695,
+            0.01737016811966896,
+            -0.0018755322089418769,
+            -0.01983434334397316,
+            -0.013681463897228241,
+            0.01762716844677925,
+            -0.023371871560811996,
+            -0.02306951954960823,
+            0.013144788332283497,
+            -0.02341722510755062,
+            0.0002966836909763515,
+            -0.002492520958185196,
+            0.007573937531560659,
+            -0.003401468973606825,
+            -0.006001702975481749,
+            0.0018027785699814558,
+            -0.028451398015022278,
+            -0.03607068955898285,
+            -0.0024471678771078587,
+            0.009017672389745712,
+            -0.001181065570563078,
+            0.004221600946038961,
+            0.005933673586696386,
+            0.009070583619177341,
+            -0.01570722833275795,
+            -0.011610347777605057,
+            -0.007384967058897018,
+            0.010166613385081291,
+            0.008012348785996437,
+            0.027876928448677063,
+            -0.005449909251183271,
+            0.0035734320990741253,
+            0.01800510846078396,
+            0.009773554280400276,
+            -0.002562440000474453,
+            -0.004350101109594107,
+            0.012933140620589256,
+            -0.011345788836479187,
+            0.006292717531323433,
+            -0.019516874104738235,
+            0.0011782310903072357,
+            0.0018897049594670534,
+            0.009992759674787521,
+            0.019531991332769394,
+            -0.019199403002858162,
+            -0.0007086393889039755,
+            -0.008276907727122307,
+            -0.016795698553323746,
+            0.005669115111231804,
+            0.025216223672032356,
+            -0.004675130359828472,
+            0.018640050664544106,
+            -0.005994144361466169,
+            0.026365164667367935,
+            -0.01044628955423832,
+            0.013794846832752228,
+            0.0047167036682367325,
+            -0.004353880416601896,
+            0.008057702332735062,
+            0.01426349300891161,
+            -0.005204247776418924,
+            0.01570722833275795,
+            -0.012880229391157627,
+            0.016281697899103165,
+            0.0032011603470891714,
+            -0.0014805839164182544,
+            0.0018452969379723072,
+            0.0064627910032868385,
+            -0.02254040166735649,
+            0.011958053335547447,
+            0.01202608272433281,
+            -0.003779409918934107,
+            0.02195081301033497,
+            -0.011973170563578606,
+            -0.006973011419177055,
+            -0.0014730250695720315,
+            0.021905461326241493,
+            -0.02929798699915409,
+            0.01860981434583664,
+            0.0223741065710783,
+            -0.0005872258334420621,
+            -0.011935376562178135,
+            0.02124028466641903,
+            0.011345788836479187,
+            0.002981954487040639,
+            -0.009388054721057415,
+            0.020030872896313667,
+            -0.0006160438060760498,
+            -0.0027627486269921064,
+            0.009750877507030964,
+            -0.01087714172899723,
+            0.007577716838568449,
+            0.014769934117794037,
+            -0.00952411349862814,
+            0.0017215212574228644,
+            0.006220908835530281,
+            0.016251463443040848,
+            -0.015661874786019325,
+            -0.025820929557085037,
+            -0.018972638994455338,
+            0.024808047339320183,
+            -0.0022808739449828863,
+            -0.0064627910032868385,
+            -0.024883635342121124,
+            0.008186202496290207,
+            -0.026002340018749237,
+            0.013084317557513714,
+            -0.011519641615450382,
+            -0.002025763737037778,
+            -0.0007865896914154291,
+            0.005344085860997438,
+            0.00807281956076622,
+            0.018141167238354683,
+            0.0015618412289768457,
+            -0.014694346114993095,
+            -0.010914935730397701,
+            0.010302671231329441,
+            -0.001181065570563078,
+            -0.007048599887639284,
+            0.005657776724547148,
+            -0.005929894279688597,
+            0.0006406100001186132,
+            0.02378004789352417,
+            0.007403864059597254,
+            -0.0020975726656615734,
+            -0.005087085999548435,
+            -0.011436494998633862,
+            -0.006213350221514702,
+            -0.00021152885165065527,
+            -0.029615456238389015,
+            -0.0005579353892244399,
+            0.007815820164978504,
+            0.0039268070831894875,
+            0.0010818560840561986,
+            -0.007282922975718975,
+            -0.016916640102863312,
+            0.008473437279462814,
+            -2.8035545255988836e-05,
+            -0.02071116678416729,
+            -0.018624933436512947,
+            0.0003132185956928879,
+            0.010340466164052486,
+            0.0046335565857589245,
+            0.010506759397685528,
+            0.014543170109391212,
+            0.018035344779491425,
+            0.01746087335050106,
+            -0.002492520958185196,
+            -0.01895751990377903,
+            -0.006084850057959557,
+            0.013628552667796612,
+            -0.02833045832812786,
+            0.001375705236569047,
+            0.013636111281812191,
+            -0.0034997337497770786,
+            -0.005453688558191061,
+            -0.023674223572015762,
+            0.019093578681349754,
+            0.005933673586696386,
+            -0.004138453863561153,
+            0.00837517250329256,
+            0.010211966000497341,
+            -0.016523580998182297,
+            -0.00286101340316236,
+            0.003199270460754633,
+            -0.007294261362403631,
+            0.003316432237625122,
+            -0.01334131695330143,
+            -0.0058845411986112595,
+            -0.005079526919871569,
+            -0.002764638513326645,
+            0.003686814568936825,
+            0.012713935226202011,
+            -0.0014427897986024618,
+            -0.012245288118720055,
+            0.012910464778542519,
+            0.0003134548314847052,
+            0.019184285774827003,
+            -0.008186202496290207,
+            0.008768231607973576,
+            -0.020151814445853233,
+            -0.012751729227602482,
+            0.000203733827220276,
+            -0.012736611999571323,
+            0.015072287060320377,
+            -0.0049056741409003735,
+            0.006368305999785662,
+            0.0025511018466204405,
+            -0.01456584595143795,
+            -0.007426540832966566,
+            0.003635792527347803,
+            0.000584391294978559,
+            0.01622122712433338,
+            0.014898434281349182,
+            0.015367081388831139,
+            0.0012717714998871088,
+            0.004350101109594107,
+            -0.02938869222998619,
+            -0.0074454378336668015,
+            0.013711699284613132,
+            0.022404342889785767,
+            0.0019284439040347934,
+            0.008511231280863285,
+            0.013469817116856575,
+            -0.0009835914243012667,
+            -0.0113684656098485,
+            -0.019290108233690262,
+            0.016916640102863312,
+            -0.025911634787917137,
+            0.01417278777807951,
+            0.006640423554927111,
+            -0.003197380807250738,
+            0.0028458957094699144,
+            0.010204407386481762,
+            0.019940167665481567,
+            -0.01207143533974886,
+            -0.011806877329945564,
+            -0.012116788886487484,
+            0.019547108560800552,
+            -0.008458320051431656,
+            -0.009599701501429081,
+            0.000633523624856025,
+            -0.02080187201499939,
+            -0.003979718778282404,
+            0.01421814039349556,
+            -0.02264622412621975,
+            -0.006285158917307854,
+            0.006613967474550009,
+            -0.00873043667525053,
+            -0.02449057623744011,
+            0.00832981988787651,
+            0.006870967335999012,
+            -0.0026474767364561558,
+            0.007630628999322653,
+            -0.017702756449580193,
+            -0.04359927400946617,
+            0.010559671558439732,
+            0.0019426167709752917,
+            -0.013477375730872154,
+            0.0014342861250042915,
+            -0.003274858696386218,
+            -0.006474129389971495,
+            -0.004168689250946045,
+            -0.011648141779005527,
+            0.01965293288230896,
+            0.023009048774838448,
+            -0.007959437556564808,
+            0.0020427710842341185,
+            -0.027045458555221558,
+            0.01983434334397316,
+            -0.007709996309131384,
+            -0.0011999626876786351,
+            -0.010597465559840202,
+            0.0031784838065505028,
+            0.04311550781130791,
+            -0.015382198616862297,
+            -0.011300436221063137,
+            -0.014512934722006321,
+            0.0008928856113925576,
+            -0.02124028466641903,
+            -0.017687639221549034,
+            0.0012472053058445454,
+            -0.021210048347711563,
+            0.012600553222000599,
+            0.0022298519033938646,
+            0.013507611118257046,
+            0.00108563550747931,
+            -0.008397849276661873,
+            -0.005427232943475246,
+            -0.0223741065710783,
+            0.005831629503518343,
+            0.011814435943961143,
+            0.009221760556101799,
+            0.013855316676199436,
+            0.006198232527822256,
+            0.005616203416138887,
+            -0.0018538006115704775,
+            0.0033806823194026947,
+            -0.016266580671072006,
+            0.014588522724807262,
+            -0.018352815881371498,
+            -0.004516394808888435,
+            -0.004380336031317711,
+            -0.012003405950963497,
+            -0.0015542823821306229,
+            0.029086340218782425,
+            -0.004746939055621624,
+            -0.00023727607913315296,
+            -0.004350101109594107,
+            0.0006084850174374878,
+            0.017612051218748093,
+            0.00868508405983448,
+            0.02677334100008011,
+            -0.008889172226190567,
+            0.00495858583599329,
+            -0.022842753678560257,
+            0.004391674418002367,
+            0.006455232389271259,
+            -0.011360906064510345,
+            0.008065260946750641,
+            0.017052698880434036,
+            0.005790056195110083,
+            0.04459703713655472,
+            -0.01895751990377903,
+            0.0017479771049693227,
+            -0.03555668890476227,
+            0.023371871560811996,
+            -0.0007969830767251551,
+            0.004486159887164831,
+            -0.0028062118217349052,
+            0.011685935780405998,
+            -0.000542817753739655,
+            -0.008926966227591038,
+            0.010922495275735855,
+            0.013235493563115597,
+            -0.02140657790005207,
+            0.0011952384375035763,
+            -0.035314805805683136,
+            0.0029233735986053944,
+            0.0004955751355737448,
+            0.006326732225716114,
+            -0.0066442028619349,
+            -0.01800510846078396,
+            0.01021952461451292,
+            -0.00873043667525053,
+            0.00952411349862814,
+            0.0008295804727822542,
+            0.023825401440262794,
+            -0.020680932328104973,
+            0.03156563267111778,
+            -0.0004497497866395861,
+            -0.005117321386933327,
+            0.013106994330883026,
+            -0.011111465282738209,
+            -0.004599541891366243,
+            -0.01563164032995701,
+            -0.008216436952352524,
+            0.013122111558914185,
+            0.002330006333068013,
+            0.015321727842092514,
+            -0.011935376562178135,
+            0.009660172276198864,
+            0.017566697672009468,
+            -0.007838496007025242,
+            0.0007502128719352186,
+            0.004229160025715828,
+            0.011050994507968426,
+            0.005805173888802528,
+            -0.00930490717291832,
+            -0.006334291305392981,
+            -0.006750026252120733,
+            -0.022071754559874535,
+            -0.01667475700378418,
+            -0.012222612276673317,
+            -0.00526471808552742,
+            0.004391674418002367,
+            0.0061566587537527084,
+            -0.0018178962636739016,
+            0.008488554507493973,
+            -0.0006769868195988238,
+            -0.016553815454244614,
+            0.0006580897606909275,
+            0.013182582333683968,
+            0.015155433677136898,
+            0.007740231696516275,
+            0.0011309884721413255,
+            0.029086340218782425,
+            0.0013794846599921584,
+            -0.007861172780394554,
+            0.005060629919171333,
+            0.0003578628820832819,
+            -0.025034811347723007,
+            0.007543702609837055,
+            -0.019531991332769394,
+            0.005083306692540646,
+            -0.016372404992580414,
+            0.007929202169179916,
+            0.028632810339331627,
+            0.009115937165915966,
+            -0.0006354133365675807,
+            -0.012479611672461033,
+            0.007997231557965279,
+            -0.01596422865986824,
+            -0.01727946288883686,
+            0.011081229895353317,
+            5.4535707022296265e-05,
+            0.011194612830877304,
+            0.014694346114993095,
+            -0.008737996220588684,
+            -0.02043904922902584,
+            -0.00034061932819895446,
+            -0.015193228609859943,
+            0.00864729005843401,
+            0.01074864249676466,
+            0.0011461060494184494,
+            -0.014905992895364761,
+            0.014989140443503857,
+            0.0042782919481396675,
+            -0.00987181905657053,
+            -0.016735227778553963,
+            -0.03090045601129532,
+            0.01210922934114933,
+            0.011171936057507992,
+            0.025563929229974747,
+            -0.009645054116845131,
+            -0.005941232666373253,
+            -0.011648141779005527,
+            0.004323645029217005,
+            0.005948791280388832,
+            -0.019637813791632652,
+            0.014905992895364761,
+            -0.01382508222013712,
+            -0.0024963002651929855,
+            0.0027533001266419888,
+            -0.0024944106116890907,
+            -0.00895720161497593,
+            -0.005465026944875717,
+            0.010400936007499695,
+            -0.0032238366547971964,
+            0.0014853081665933132,
+            0.013212817721068859,
+            -0.0006325787398964167,
+            0.0199099313467741,
+            -0.02376493066549301,
+            0.03271457180380821,
+            0.006202011834830046,
+            3.097344597335905e-05,
+            0.00033376915962435305,
+            -0.008019908331334591,
+            -0.017491109669208527,
+            0.00215048436075449,
+            0.02299392968416214,
+            -0.004471042193472385,
+            -0.0113684656098485,
+            0.00046510365791618824,
+            0.0060735116712749004,
+            -0.013673905283212662,
+            -0.012970935553312302,
+            -0.01267614122480154,
+            0.01359831728041172,
+            -0.00378318945877254,
+            -0.005158894695341587,
+            0.029902692884206772,
+            0.0063002766110002995,
+            0.030945809558033943,
+            -0.013537846505641937,
+            -0.005412115249782801,
+            4.6858778659952804e-05,
+            0.013893111608922482,
+            0.004308527335524559,
+            -0.02122516743838787,
+            0.009403171949088573,
+            0.00011208312935195863,
+            0.017218992114067078,
+            -0.0033353292383253574,
+            -0.009697966277599335,
+            -0.005865644197911024,
+            0.015276375226676464,
+            -0.003953262697905302,
+            0.011436494998633862,
+            0.006149100139737129,
+            0.016568932682275772,
+            0.012623229064047337,
+            -0.01232843566685915,
+            0.005858085583895445,
+            -0.009939848445355892,
+            0.011897582560777664,
+            -0.02107398957014084,
+            -0.007539922837167978,
+            0.01175396516919136,
+            0.01390822883695364,
+            0.00245661661028862,
+            -0.0011621685698628426,
+            0.0009722532122395933,
+            0.0043387627229094505,
+            -0.005653997417539358,
+            0.012865111231803894,
+            -0.01176152378320694,
+            0.013938464224338531,
+            0.004731821361929178,
+            -0.007460555527359247,
+            -0.000914617208763957,
+            0.0007818654412403703,
+            -0.004463483113795519,
+            -0.006538379471749067,
+            -0.01176152378320694,
+            -0.016689874231815338,
+            -0.014490257948637009,
+            -0.01202608272433281,
+            0.014701904729008675,
+            0.011897582560777664,
+            0.00860193744301796,
+            0.008662407286465168,
+            0.011376024223864079,
+            -0.01885169744491577,
+            0.0071733202785253525,
+            -0.002624800195917487,
+            -0.003779409918934107,
+            0.02087746001780033,
+            -0.00232622679322958,
+            -0.014293728396296501,
+            0.0023791384883224964,
+            0.01718875765800476,
+            0.01513275783509016,
+            -0.00019511203572619706,
+            0.006663099862635136,
+            0.004387895110994577,
+            0.02281251922249794,
+            -0.019426167011260986,
+            -0.0032219470012933016,
+            -0.007815820164978504,
+            0.012192376889288425,
+            0.02263110689818859,
+            -0.004523953888565302,
+            -0.015276375226676464,
+            -0.03458916023373604,
+            0.013832640834152699,
+            -0.014671669341623783,
+            0.0014390103751793504,
+            0.013568081893026829,
+            -0.028315339237451553,
+            0.013749493286013603,
+            -0.022495048120617867,
+            0.007675981614738703,
+            0.017793461680412292,
+            -0.016977109014987946,
+            -0.016130521893501282,
+            -0.0036849246826022863,
+            0.004244277253746986,
+            0.025594163686037064,
+            -0.020590225234627724,
+            -0.029706163331866264,
+            -0.0022525284439325333,
+            -0.007808261085301638,
+            -0.011897582560777664,
+            -0.008004790171980858,
+            -0.009554348886013031,
+            0.02492898888885975,
+            -0.019093578681349754,
+            0.003085888223722577,
+            -0.02993292734026909,
+            -0.0066064088605344296,
+            -0.011315553449094296,
+            -0.007573937531560659,
+            -0.019199403002858162,
+            -0.005665335804224014,
+            -0.010725965723395348,
+            -0.0004521119117271155,
+            -0.008050143718719482,
+            0.0004632139462046325,
+            -0.019199403002858162,
+            0.012131906114518642,
+            0.008231555111706257,
+            0.011685935780405998,
+            0.013507611118257046,
+            -0.007951878942549229,
+            -0.01966805011034012,
+            -0.00599036505445838,
+            0.01114925928413868,
+            0.0017319145845249295,
+            0.009085701778531075,
+            -0.0057295854203403,
+            0.016901521012187004,
+            -0.025775576010346413,
+            0.004119556862860918,
+            0.018806343898177147,
+            0.007400084752589464,
+            0.01539731677621603,
+            -0.02054487355053425,
+            0.011648141779005527,
+            -0.00618311483412981,
+            -0.0016988448332995176,
+            -6.315010068647098e-06,
+            -0.003911689389497042,
+            -0.005676673725247383,
+            0.019093578681349754,
+            0.016190992668271065,
+            -0.010393377393484116,
+            -0.002549211960285902,
+            0.0016260911943390965,
+            -0.015556051395833492,
+            -0.004433247726410627,
+            -0.00013015343574807048,
+            -0.0034090278204530478,
+            0.005548174027353525,
+            0.0041497922502458096,
+            -0.012842435389757156,
+            -0.005053071305155754,
+            0.0019445064244791865,
+            -0.00023975632211659104,
+            -0.01711316779255867,
+            -0.007959437556564808,
+            -0.00476205674931407,
+            -0.009894495829939842,
+            -0.004637335892766714,
+            0.004183806944638491,
+            0.02210198901593685,
+            -0.0010005987714976072,
+            -0.005767379887402058,
+            -0.001888760132715106,
+            -0.0203634612262249,
+            0.0026153516955673695,
+            0.00038644467713311315,
+            -0.0020919034723192453,
+            -0.016100287437438965,
+            0.008715319447219372,
+            0.002876131096854806,
+            0.017581814900040627,
+            -0.026546575129032135,
+            -0.002184499055147171,
+            -0.014550728723406792,
+            -0.01426349300891161,
+            -0.02615351788699627,
+            0.01417278777807951,
+            -0.011685935780405998,
+            -0.0012207494582980871,
+            -0.01879122667014599,
+            -0.011481847614049911,
+            0.0073622907511889935,
+            0.003960821777582169,
+            0.022268284112215042,
+            -0.03582880645990372,
+            -0.028239751234650612,
+            0.00726402597501874,
+            -0.006761364638805389,
+            0.006304055918008089,
+            -0.019244756549596786,
+            0.010597465559840202,
+            -0.009697966277599335,
+            -0.021965930238366127,
+            -0.0006708452710881829,
+            0.02308463677763939,
+            -0.029086340218782425,
+            -0.04021292179822922,
+            -0.02569998800754547,
+            0.016266580671072006,
+            0.006836952641606331,
+            0.022389225661754608,
+            0.014082081615924835,
+            -0.0040666451677680016,
+            -0.007298040669411421,
+            -0.010544554330408573,
+            0.0036509099882096052,
+            0.019804108887910843,
+            -0.011890023946762085,
+            -0.0061075263656675816,
+            0.00015991629334166646,
+            0.021346107125282288,
+            -0.0009604425868019462,
+            0.013938464224338531,
+            0.006213350221514702,
+            -0.017400404438376427,
+            -0.004637335892766714,
+            0.0001971198507817462,
+            -0.005170233082026243,
+            0.0018972638063132763,
+            0.0011744516668841243,
+            -0.0025038591120392084,
+            0.010862024500966072,
+            0.007974554784595966,
+            -0.004622218664735556,
+            -0.013575640507042408,
+            -0.003023528028279543,
+            0.0017999439733102918,
+            -0.018730755895376205,
+            0.021799637004733086,
+            0.008889172226190567,
+            -0.0054839239455759525,
+            -0.004773394670337439,
+            0.0052382624708116055,
+            0.022147342562675476,
+            0.004263174720108509,
+            0.005298732779920101,
+            0.007638187613338232,
+            0.007071276195347309,
+            0.00863973144441843,
+            -0.0014796389732509851,
+            0.01052943617105484,
+            -0.003289976390078664,
+            0.017400404438376427,
+            -0.002144815167412162,
+            -0.015737462788820267,
+            -0.0009599701152183115,
+            -0.007475673221051693,
+            0.009622378274798393,
+            -0.00995496567338705,
+            0.011474289000034332,
+            0.015427551232278347,
+            0.008881613612174988,
+            0.0011961832642555237,
+            0.010189289227128029,
+            -0.014323963783681393,
+            0.027514105662703514,
+            -0.00039683806244283915,
+            -0.0010270546190440655,
+            0.009645054116845131,
+            0.005544394254684448,
+            -0.00041526267887093127,
+            -0.0011054774513468146,
+            0.018473755568265915,
+            -0.006663099862635136,
+            -0.02606281079351902,
+            0.022434577345848083,
+            -0.01974363811314106,
+            -0.018640050664544106,
+            0.010007877834141254,
+            -0.006867188028991222,
+            0.009577024728059769,
+            0.004459703806787729,
+            0.009206642396748066,
+            0.03168657422065735,
+            0.005438570864498615,
+            -0.01966805011034012,
+            -0.002471734071150422,
+            -0.010824230499565601,
+            0.008571702055633068,
+            -0.02176940254867077,
+            -0.0059223356656730175,
+            -0.009947407059371471,
+            0.004051527474075556,
+            -0.0021183593198657036,
+            -0.0018434072844684124,
+            0.017959756776690483,
+            0.0021674917079508305,
+            0.012131906114518642,
+            -0.001987969735637307,
+            -0.004769615363329649,
+            0.03186798468232155,
+            -0.0008933580247685313,
+            -0.014233258552849293,
+            -0.004951027221977711,
+            -0.0053176297806203365,
+            -0.039789628237485886,
+            -0.01083178911358118,
+            -0.0023696899879723787,
+            0.01703757978975773,
+            -0.0011120913550257683,
+            0.00833737850189209,
+            -0.007698658388108015,
+            -0.00695789372548461,
+            0.0004176248039584607,
+            0.003666027681902051,
+            -0.0009273727191612124,
+            0.02149728499352932,
+            -0.007793143391609192,
+            0.002653145929798484,
+            -0.002394256182014942,
+            -0.02817928045988083,
+            -0.02527669444680214,
+            -0.002486851764842868,
+            -0.022510165348649025,
+            0.0024981899186968803,
+            0.013205258175730705,
+            -0.012131906114518642,
+            -0.026017459109425545,
+            0.0031104544177651405,
+            -0.02379516512155533,
+            0.03613115847110748,
+            0.011655700393021107,
+            -0.0028591237496584654,
+            -0.014603639952838421,
+            -0.007581496611237526,
+            -0.0037718513049185276,
+            -0.020937930792570114,
+            -0.006791599560528994,
+            0.009289789944887161,
+            -0.0014210582012310624,
+            0.0016147529240697622,
+            0.0021618225146085024,
+            -0.04220845177769661,
+            -0.016251463443040848,
+            0.013961140997707844,
+            -0.0319586917757988,
+            0.018670285120606422,
+            0.03088533878326416,
+            -0.0011215399717912078,
+            0.0031028955709189177,
+            -0.0033258807379752398,
+            0.005257159471511841,
+            0.010295112617313862,
+            0.005646438337862492,
+            -0.00903278961777687,
+            -0.010967847891151905,
+            -0.015949109569191933,
+            0.025140635669231415,
+            0.009108378551900387,
+            -0.01263078860938549,
+            0.008768231607973576,
+            0.011353347450494766,
+            0.004002395085990429,
+            -0.004603321198374033,
+            0.0035715424455702305,
+            -0.02501969411969185,
+            -0.005895879585295916,
+            -0.0032219470012933016,
+            0.013099434785544872,
+            0.01267614122480154,
+            0.0037246085703372955,
+            0.00495858583599329,
+            0.007736452389508486,
+            -0.002653145929798484,
+            -0.0060735116712749004,
+            -0.02387075312435627,
+            0.020061109215021133,
+            -0.00042943545849993825,
+            0.0036981527227908373,
+            0.0080803781747818,
+            0.0025246459990739822,
+            0.002176940208300948,
+            -0.008012348785996437,
+            0.011519641615450382,
+            0.01903310790657997,
+            0.016115404665470123,
+            -0.006220908835530281,
+            0.007589055225253105,
+            -0.01259299460798502,
+            0.003051873529329896,
+            0.010635259561240673,
+            -0.002365910680964589,
+            -0.008307143114507198,
+            -0.025473223999142647,
+            0.00033754855394363403,
+            -0.01083934772759676,
+            -0.0007251743227243423,
+            -0.0002626689965836704,
+            0.0010629590833559632,
+            -0.00726402597501874,
+            -0.018141167238354683,
+            0.0011952384375035763,
+            0.0026588148903101683,
+            0.020302990451455116,
+            -0.027317576110363007,
+            -0.002471734071150422,
+            0.017763227224349976,
+            0.009886936284601688,
+            0.007778025697916746,
+            0.019290108233690262,
+            0.005982805974781513,
+            0.0013719258131459355,
+            -0.007033482193946838,
+            -0.002777866320684552,
+            0.0197285208851099,
+            -0.002549211960285902,
+            0.008276907727122307,
+            -0.0145204933360219,
+            -0.014550728723406792,
+            -0.016871286556124687,
+            0.010514318943023682,
+            -0.013507611118257046,
+            -0.008261790499091148,
+            0.016992228105664253,
+            -0.002679601777344942,
+            -0.011958053335547447,
+            0.01718875765800476,
+            0.0015892419032752514,
+            0.0070674968883395195,
+            -0.003622564487159252,
+            0.0023791384883224964,
+            0.006326732225716114,
+            -0.0019019880564883351,
+            0.0035016234032809734,
+            0.005381879862397909,
+            -0.026032576337456703,
+            0.019154049456119537,
+            0.004372777417302132,
+            0.020665813237428665,
+            -0.009478759951889515,
+            0.017687639221549034,
+            -0.002324337139725685,
+            0.00802746694535017,
+            0.020378578454256058,
+            -0.016977109014987946,
+            0.008397849276661873,
+            0.003934365697205067,
+            0.015556051395833492,
+            -0.010378260165452957,
+            -0.0022449695970863104,
+            -0.012691258452832699,
+            -0.00199552858248353,
+            -0.006780261639505625,
+            -0.0024755136109888554,
+            -0.0010185509454458952,
+            -0.0005735254962928593,
+            -0.0030613220296800137,
+            -0.006602629087865353,
+            0.005249600391834974,
+            0.021875225007534027,
+            -0.009320025332272053,
+            -0.016478227451443672,
+            0.004017512779682875,
+            0.0018452969379723072,
+            0.021708931773900986,
+            0.0401826873421669,
+            -0.011890023946762085,
+            0.02376493066549301,
+            -0.019063344225287437,
+            -0.011481847614049911,
+            0.008798466064035892,
+            0.00307266041636467,
+            -0.013054082170128822,
+            -0.035586923360824585,
+            0.00587698258459568,
+            0.012645905837416649,
+            -0.009660172276198864,
+            0.02113446034491062,
+            0.014384434558451176,
+            0.005449909251183271,
+            -0.017869049683213234,
+            0.020151814445853233,
+            -0.006674438249319792,
+            0.0055670710280537605,
+            -0.00898743700236082,
+            0.016175875440239906,
+            -0.00648924708366394,
+            0.013666346669197083,
+            -0.016342168673872948,
+            0.004618438892066479,
+            0.008964760228991508,
+            0.02843628078699112,
+            -0.02133098989725113,
+            0.014497816562652588,
+            -0.016432873904705048,
+            -0.004875438753515482,
+            -0.005442350637167692,
+            0.009697966277599335,
+            -0.0001299172145081684,
+            0.006228467915207148,
+            -0.00011285082291578874,
+            -0.016357285901904106,
+            -0.0007147809374146163,
+            -0.005257159471511841,
+            -0.010703288950026035,
+            -0.0003616423055063933,
+            0.0022865431383252144,
+            0.005918555893003941,
+            0.01298605278134346,
+            0.011421376839280128,
+            -0.01201852411031723,
+            0.012736611999571323,
+            -0.041633982211351395,
+            0.001619477174244821,
+            -0.004289630334824324,
+            -0.0005791945732198656,
+            0.0007459610351361334,
+            -0.015661874786019325,
+            -0.012124347500503063,
+            -0.0070674968883395195,
+            0.010196847841143608,
+            -0.00552927702665329,
+            -0.009765995666384697,
+            -0.011769082397222519,
+            0.007592834532260895,
+            0.001629870617762208,
+            0.006625305861234665,
+            0.010552112944424152,
+            0.012260406278073788,
+            -0.033047161996364594,
+            -0.01263834722340107,
+            -0.019607579335570335,
+            0.00012212219007778913,
+            -0.005623762030154467,
+            -0.0010818560840561986,
+            -0.014422228559851646,
+            -0.02553369477391243,
+            -0.0010998082580044866,
+            0.0154351107776165,
+            0.013175023719668388,
+            -0.004206483252346516,
+            0.009796231053769588,
+            -0.0034676087088882923,
+            0.004180027637630701,
+            -0.018141167238354683,
+            0.002394256182014942,
+            0.004100659862160683,
+            0.0036905938759446144,
+            0.020726284012198448,
+            -0.019788991659879684,
+            0.01921452023088932,
+            0.014278611168265343,
+            0.0019936386961489916,
+            0.011028318665921688,
+            -0.0011801207438111305,
+            0.0066442028619349,
+            -0.008465878665447235,
+            -0.012608111836016178,
+            0.005559511948376894,
+            -0.0010506759863346815,
+            0.008405407890677452,
+            0.006504364777356386,
+            0.005627541337162256,
+            0.008087937720119953,
+            0.020302990451455116,
+            -0.008163525722920895,
+            0.0028043221682310104,
+            -0.01596422865986824,
+            -0.0006836007814854383,
+            -0.0009590252884663641,
+            -0.004414350725710392,
+            0.013205258175730705,
+            0.0024736239574849606,
+            -0.013885552063584328,
+            -0.011391141451895237,
+            -0.02405216544866562,
+            0.011973170563578606,
+            0.014437345787882805,
+            0.003580990945920348,
+            0.010620142333209515,
+            -0.004951027221977711,
+            0.001301061944104731,
+            0.01545022800564766,
+            -0.007033482193946838,
+            -0.028904927894473076,
+            0.0055670710280537605,
+            0.010234642773866653,
+            0.023553283885121346,
+            -0.0020767857786267996,
+            -0.004845203831791878,
+            0.00418758625164628,
+            0.0017914402997121215,
+            0.014414669945836067,
+            0.005638879723846912,
+            0.012230170890688896,
+            0.019063344225287437,
+            -0.006013041362166405,
+            0.007010805420577526,
+            0.005752262193709612,
+            0.01694687455892563,
+            -0.0160700511187315,
+            -7.039151387289166e-05,
+            0.02122516743838787,
+            0.009486319497227669,
+            0.01206387672573328,
+            -0.010068348608911037,
+            0.014588522724807262,
+            0.0036093364469707012,
+            0.0019142711535096169,
+            -0.004860321059823036,
+            0.0018065579934045672,
+            0.00434632133692503,
+            0.0047167036682367325,
+            0.02842116355895996,
+            -0.0028723515570163727,
+            0.008050143718719482,
+            -0.040243156254291534,
+            0.016810815781354904,
+            0.00032148606260307133,
+            0.009131054393947124,
+            -0.0036527998745441437,
+            0.018367933109402657,
+            0.03088533878326416,
+            0.014104758389294147,
+            0.00511354161426425,
+            0.005771159194409847,
+            0.0034524910151958466,
+            -0.006413658615201712,
+            -0.011028318665921688,
+            0.004323645029217005,
+            0.004667571280151606,
+            -0.0074454378336668015,
+            0.014505375176668167,
+            0.0032994248904287815,
+            -0.014286169782280922,
+            -0.004992600530385971,
+            0.013273287564516068,
+            0.0006491136737167835,
+            -0.010680613107979298,
+            0.03483104333281517,
+            -0.012426700443029404,
+            -0.0016100286738947034,
+            -0.0001238937838934362,
+            -0.01563164032995701,
+            -0.0069843498058617115,
+            0.020333226770162582,
+            -0.014490257948637009,
+            -0.010884701274335384,
+            -0.037340570241212845,
+            0.012237729504704475,
+            0.0029082559049129486,
+            0.003282417543232441,
+            -0.014724581502377987,
+            -0.03942680358886719,
+            0.005022835917770863,
+            0.011194612830877304,
+            -0.007846055552363396,
+            0.009229319170117378,
+            0.00679915864020586,
+            -0.01198073010891676,
+            0.012789523229002953,
+            0.029691044241189957,
+            0.01817140355706215,
+            -0.01210922934114933,
+            0.004773394670337439,
+            0.024702223017811775,
+            0.008677525445818901,
+            -0.013522729277610779,
+            0.020756520330905914,
+            -0.02798275090754032,
+            0.007543702609837055,
+            0.0159793458878994,
+            -0.0014191684313118458,
+            0.0035847704857587814,
+            -8.899329259293154e-05,
+            0.015933992341160774,
+            0.004478600807487965,
+            0.008889172226190567,
+            0.03235175088047981,
+            -0.010673053562641144,
+            -0.00025322046712972224,
+            -0.013734376057982445,
+            -0.009463642723858356,
+            -0.0015391646884381771,
+            0.00645145308226347,
+            -0.02254040166735649,
+            -0.0007809206144884229,
+            0.002675822237506509,
+            -0.026274457573890686,
+            0.010642818175256252,
+            0.020680932328104973,
+            -0.016825933009386063,
+            -2.1598738385364413e-05,
+            0.014550728723406792,
+            0.017067816108465195,
+            0.010673053562641144,
+            -0.012509847059845924,
+            -0.0007492680451832712,
+            -0.009493878111243248,
+            -0.008722878061234951,
+            0.004229160025715828,
+            -0.02394634112715721,
+            0.03833833336830139,
+            -0.01229064166545868,
+            0.006024379748851061,
+            0.015601404942572117,
+            -0.004387895110994577,
+            -0.004973703529685736,
+            0.004667571280151606,
+            0.011557435616850853,
+            -0.023659106343984604,
+            0.03340998291969299,
+            -0.016024697571992874,
+            0.00491323322057724,
+            0.01703757978975773,
+            0.027075693011283875,
+            -0.007638187613338232,
+            0.0013416905421763659,
+            -0.00026030687149614096,
+            0.028889810666441917,
+            -4.538244684226811e-05,
+            -0.02325093001127243,
+            -0.01895751990377903,
+            -0.005136218387633562,
+            0.01938081532716751,
+            0.00757015822455287,
+            -0.015752580016851425,
+            0.028935162350535393,
+            -0.004939688835293055,
+            -0.01956222578883171,
+            -0.002256307750940323,
+            -0.0008097385871224105,
+            -0.0036206748336553574,
+            0.0317772775888443,
+            0.01017417199909687,
+            -0.01817140355706215,
+            -0.01965293288230896,
+            -0.008760672062635422,
+            -0.02686404623091221,
+            -0.0022789842914789915,
+            0.0004492773732636124,
+            -0.008874054998159409,
+            0.00301596918143332,
+            -0.012494729831814766,
+            -0.019622696563601494,
+            0.0012320876121520996,
+            0.014089640229940414,
+            0.04746938869357109,
+            0.001130043645389378,
+            0.007275364361703396,
+            -0.02624422311782837,
+            0.01272905245423317,
+            -0.011685935780405998,
+            -0.0033409984316676855,
+            -0.0027344031259417534,
+            -0.00925199594348669,
+            -0.00039022407145239413,
+            -0.006285158917307854,
+            -0.015646757557988167,
+            -0.029328221455216408,
+            0.01879122667014599,
+            0.002141035860404372,
+            -0.006311614532023668,
+            -0.013537846505641937,
+            -0.02387075312435627,
+            0.012759287841618061,
+            -0.01675034500658512,
+            -0.008624613285064697,
+            -0.0356171615421772,
+            -0.024989459663629532,
+            0.011285318061709404,
+            0.006882305722683668,
+            -0.0017583705484867096,
+            -0.0028855795972049236,
+            -0.013726817443966866,
+            -0.02124028466641903,
+            -0.03640327602624893,
+            0.009115937165915966,
+            -0.005355423782020807,
+            0.01622122712433338,
+            -0.011247524060308933,
+            -5.229168164078146e-05,
+            -0.006319173611700535,
+            0.008458320051431656,
+            0.020665813237428665,
+            0.01140625961124897,
+            0.003021638374775648,
+            0.002367800334468484,
+            0.01700734533369541,
+            -0.004463483113795519,
+            -0.02376493066549301,
+            -0.02518598921597004,
+            0.00286479271017015,
+            0.00599036505445838,
+            -0.01782369799911976,
+            0.009463642723858356,
+            0.0007790309027768672,
+            0.0019237196538597345,
+            0.009267113171517849,
+            0.0034317043609917164,
+            -0.0069843498058617115,
+            -0.010400936007499695,
+            -0.001878366805613041,
+            0.026969870552420616,
+            0.007815820164978504,
+            -0.010249760001897812,
+            -0.010718407109379768,
+            0.0023848076816648245,
+            -0.007203555665910244,
+            0.0011961832642555237,
+            0.012222612276673317,
+            -0.013794846832752228,
+            -0.001549558131955564,
+            -0.011307994835078716,
+            -0.010000319220125675,
+            -0.011988288722932339,
+            -0.017475992441177368,
+            0.0011961832642555237,
+            -0.000834304781164974,
+            -0.03854998201131821,
+            0.013855316676199436,
+            -0.0006779316463507712,
+            -0.007124187890440226,
+            -0.0015949109802022576,
+            0.007936760783195496,
+            0.0005286450032144785,
+            -0.003512961557134986,
+            -0.0002489686303306371,
+            0.018201638013124466,
+            -0.027181517332792282,
+            0.012970935553312302,
+            0.021618224680423737,
+            -0.017143404111266136,
+            -0.001795219723135233,
+            -0.009206642396748066,
+            0.010113701224327087,
+            0.009161289781332016,
+            0.012494729831814766,
+            -0.001033668639138341,
+            0.003919248003512621,
+            4.290220749680884e-05,
+            -0.0010431171394884586,
+            0.006999467499554157,
+            0.0028118810150772333,
+            0.010982965119183064,
+            0.00807281956076622,
+            0.006364526227116585,
+            -0.01026487722992897,
+            0.010453848168253899,
+            0.0025945650413632393,
+            -0.0006132092676125467,
+            0.0011253192787989974,
+            0.01390067022293806,
+            0.023659106343984604,
+            0.012351111508905888,
+            -0.015540934167802334,
+            0.019819226115942,
+            -0.013205258175730705,
+            0.006663099862635136,
+            0.00278731482103467,
+            0.0002310164418304339,
+            0.03477057069540024,
+            0.012555199675261974,
+            0.024021929129958153,
+            0.025639517232775688,
+            0.026652399450540543,
+            0.019970402121543884,
+            -0.0020030871964991093,
+            -0.015057169832289219,
+            -0.01623634621500969,
+            -0.002135366667062044,
+            0.019879696890711784,
+            0.0018481315346434712,
+            0.00027849528123624623,
+            -0.010461406782269478,
+            -0.002199616516008973,
+            -0.004304748028516769,
+            0.010642818175256252,
+            0.0053176297806203365,
+            -0.0034543806686997414,
+            -0.0037510644178837538,
+            -0.0007029702537693083,
+            0.0008220216841436923,
+            -0.008133290335536003,
+            -0.007400084752589464,
+            -0.004924571141600609,
+            0.0021334770135581493,
+            -0.03053763322532177,
+            0.05000915378332138,
+            -0.00803502555936575,
+            0.014354199171066284,
+            -0.009728201664984226,
+            0.003121792571619153,
+            0.004901894833892584,
+            -0.013507611118257046,
+            -0.002435829723253846,
+            -0.003199270460754633,
+            0.0026909399311989546,
+            -0.0147472582757473,
+            0.002702278085052967,
+            0.008571702055633068,
+            -0.014596081338822842,
+            0.0032011603470891714,
+            0.01149696484208107,
+            0.006753805559128523,
+            0.005933673586696386,
+            0.009025231003761292,
+            0.015193228609859943,
+            0.02464175410568714,
+            -0.00073462282307446,
+            -0.0005399832152761519,
+            0.0025322046130895615,
+            0.010695730336010456,
+            0.015828169882297516,
+            0.001142326625995338,
+            0.024883635342121124,
+            -0.0021126901265233755,
+            0.019063344225287437,
+            0.028194399550557137,
+            -0.010673053562641144,
+            -0.007490790449082851,
+            0.001940727001056075,
+            -0.01365122850984335,
+            0.008737996220588684,
+            -0.0019426167709752917,
+            -0.008019908331334591,
+            -0.013477375730872154,
+            0.001699789660051465,
+            0.02201128378510475,
+            0.00832981988787651,
+            -0.006511923391371965,
+            0.01263078860938549,
+            -0.002673932584002614,
+            0.0030197484884411097,
+            0.002269535791128874,
+            -0.0013917677570134401,
+            -0.0030367558356374502,
+            -0.010695730336010456,
+            0.01587352156639099,
+            -0.005230703391134739,
+            -0.021376343443989754,
+            0.009765995666384697,
+            -0.00434632133692503,
+            -0.008322261273860931,
+            0.03235175088047981,
+            -0.01622122712433338,
+            -0.006731129251420498,
+            -0.003365564625710249,
+            0.016296815127134323,
+            0.008019908331334591,
+            -0.0007880069897510111
+          ],
+          "title": "Switch Transformer",
+          "keyphrases": [
+            "Switch Transformer",
+            "Simplifying Sparse Routing",
+            "Efficient Sparse Routing",
+            "Improved Training",
+            "Fine-Tuning Techniques"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "6b0c0b81-ce21-4498-976c-884e9649b996",
+      "type": "child",
+      "source": {
+        "id": "65742957-6bda-446d-a61a-0fe20313c760",
+        "properties": {
+          "page_content": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity\n\nWilliam Fedus∗ liamfedus@google.com Barret Zoph∗ barretzoph@google.com Noam Shazeer noam@google.com Google, Mountain View, CA 94043, USA Editor: Alexander Clark\n\nAbstract\n\nIn deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) models defy this and instead select different parameters for each incoming example. The result is a sparsely-activated model—with an outrageous number of parameters—but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs, and training instability. We address these with the introduction of the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques mitigate the instabilities, and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large (Raffel et al., 2019) to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the \"Colossal Clean Crawled Corpus\", and achieve a 4x speedup over the T5-XXL model.12 Keywords: mixture-of-experts, natural language processing, sparsity, large-scale machine learning, distributed computing arXiv:2101.03961v3 [cs.LG] 16 Jun 2022 Contents\n\n1 Introduction 3 2 Switch Transformer 4 2.1 Simplifying Sparse Routing 5 2.2 Efficient Sparse Routing 6 2.3 Putting It All Together: The Switch Transformer 8 2.4 Improved Training and Fine-Tuning Techniques 8 3 Scaling Properties 11 3.1 Scaling Results on a Step-Basis 12 3.2 Scaling Results on a Time-Basis 13 3.3 Scaling Versus a Larger Dense Model 13 4 Downstream Results 14 4.1 Fine-Tuning 14 4.2 Distillation 16 4.3 Multilingual Learning 17 5 Designing Models with Data, Model, and Expert-Parallelism 18 5.1 Data Parallelism 20 5.2 Model Parallelism 20 5.3 Model and Data Parallelism 21 5.4 Expert and Data Parallelism 22 5.5 Expert, Model and Data Parallelism 22 5.6 Towards Trillion Parameter Models 22 6 Related Work 24 7 Discussion 25 8 Future Work 26 9 Conclusion 27 A Switch for Attention 27 B Preventing Token Dropping with No-Token-Left-Behind 29\n\nC Encouraging Exploration Across Experts 29 D Switch Transformers in Lower Compute Regimes 29 E Relation of Upstream to Downstream Model Performance 32 F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/switch_transformers.md"
+          },
+          "summary": "The Switch Transformer is a sparsely-activated model that simplifies the Mixture of Experts (MoE) approach, allowing for efficient scaling to trillion-parameter models with reduced communication and computational costs. It achieves significant speedups in pre-training and fine-tuning by activating only a subset of neural network weights for each input, maintaining a constant computational cost. The model demonstrates superior scaling properties and improved performance across various natural language processing tasks, including multilingual settings. It also allows for effective distillation into smaller dense models, preserving a portion of the quality gains. The architecture is designed to maximize parameter count while keeping floating point operations per example constant, making it suitable for both large-scale and smaller-scale applications.",
+          "headlines": {
+            "1 Introduction": [],
+            "2 Switch Transformer": [
+              "2.1 Simplifying Sparse Routing",
+              "2.2 Efficient Sparse Routing",
+              "2.3 Putting It All Together: The Switch Transformer",
+              "2.4 Improved Training and Fine-Tuning Techniques"
+            ],
+            "3 Scaling Properties": [
+              "3.1 Scaling Results on a Step-Basis",
+              "3.2 Scaling Results on a Time-Basis",
+              "3.3 Scaling Versus a Larger Dense Model"
+            ],
+            "4 Downstream Results": [
+              "4.1 Fine-Tuning",
+              "4.2 Distillation",
+              "4.3 Multilingual Learning"
+            ],
+            "5 Designing Models with Data, Model, and Expert-Parallelism": [
+              "5.1 Data Parallelism",
+              "5.2 Model Parallelism",
+              "5.3 Model and Data Parallelism",
+              "5.4 Expert and Data Parallelism",
+              "5.5 Expert, Model and Data Parallelism",
+              "5.6 Towards Trillion Parameter Models"
+            ],
+            "6 Related Work": [],
+            "7 Discussion": [],
+            "8 Future Work": [],
+            "9 Conclusion": [],
+            "A Switch for Attention": [],
+            "B Preventing Token Dropping with No-Token-Left-Behind": [],
+            "C Encouraging Exploration Across Experts": [],
+            "D Switch Transformers in Lower Compute Regimes": [],
+            "E Relation of Upstream to Downstream Model Performance": [],
+            "F Pseudo Code for Switch Transformers": []
+          },
+          "summary_embedding": [
+            -0.0060447100549936295,
+            -0.00390331051312387,
+            -0.01784370467066765,
+            -0.0021820589900016785,
+            0.013227885589003563,
+            0.011338187381625175,
+            0.0165580902248621,
+            0.024240797385573387,
+            -0.005986624862998724,
+            0.03596621751785278,
+            -0.004852031823247671,
+            -0.0068811336532235146,
+            -0.003574162255972624,
+            0.014242436736822128,
+            0.011578272096812725,
+            -0.033302053809165955,
+            -0.024519605562090874,
+            0.01050176378339529,
+            0.016093410551548004,
+            -0.006795942317694426,
+            0.007473600562661886,
+            -0.030513975769281387,
+            -0.06124480068683624,
+            -0.006838538218289614,
+            -0.0026099516544491053,
+            0.01029265858232975,
+            0.0010329445358365774,
+            -0.010873507708311081,
+            -0.053469158709049225,
+            0.023311438038945198,
+            0.025294071063399315,
+            0.005092116538435221,
+            -0.029584616422653198,
+            0.0009361362317577004,
+            -0.05161044001579285,
+            -0.04448534920811653,
+            0.0379488542675972,
+            0.012561844661831856,
+            -0.04364892467856407,
+            0.039249956607818604,
+            -0.004875265993177891,
+            -0.0022440163884311914,
+            -0.021499186754226685,
+            -0.0034909071400761604,
+            -0.007039899472147226,
+            -0.009851212613284588,
+            -0.02874819189310074,
+            0.0014037203509360552,
+            0.021886419504880905,
+            -0.0056962003000080585,
+            0.023264970630407333,
+            -0.024271776899695396,
+            0.01936165988445282,
+            0.04240977764129639,
+            -0.06957805901765823,
+            0.024659009650349617,
+            0.017549408599734306,
+            0.05120771750807762,
+            0.06511713564395905,
+            0.018339363858103752,
+            0.021669568493962288,
+            -0.014141756109893322,
+            -0.017766257748007774,
+            -0.011872569099068642,
+            0.004673904739320278,
+            -0.014970434829592705,
+            -0.01891246810555458,
+            0.008449427783489227,
+            -0.020461400970816612,
+            0.008209343999624252,
+            -0.05576157942414284,
+            -0.014730350114405155,
+            -0.034200433641672134,
+            -0.01140014547854662,
+            0.017642343416810036,
+            -0.018215449526906013,
+            0.030126741155982018,
+            0.02997184917330742,
+            0.002638994250446558,
+            0.0007822110201232135,
+            0.021049996837973595,
+            0.006509389728307724,
+            -0.0053438181057572365,
+            0.012321759946644306,
+            0.028159597888588905,
+            -0.01450575515627861,
+            -0.010106786154210567,
+            -0.03454120084643364,
+            0.01861817203462124,
+            0.01589204929769039,
+            -0.04194509983062744,
+            -0.012569589540362358,
+            -0.013824224472045898,
+            0.032341714948415756,
+            -0.009549170732498169,
+            0.04098476096987724,
+            -0.010083552449941635,
+            0.001383390510454774,
+            -0.031149037182331085,
+            -0.009463978931307793,
+            0.017069239169359207,
+            0.006172496825456619,
+            -0.016589069738984108,
+            -0.0012846461031585932,
+            0.016170857474207878,
+            0.006617815233767033,
+            -0.030142230913043022,
+            0.04488807171583176,
+            0.023497309535741806,
+            0.055296897888183594,
+            0.05579255893826485,
+            -0.037391237914562225,
+            -0.002445377642288804,
+            0.0344172865152359,
+            -0.0228312686085701,
+            -0.015040136873722076,
+            -0.034014564007520676,
+            -0.01768881268799305,
+            0.035935238003730774,
+            -0.0318460576236248,
+            0.01556677371263504,
+            0.028066661208868027,
+            -0.025712283328175545,
+            0.003488970920443535,
+            -0.03943582624197006,
+            0.003622566582635045,
+            0.01220559049397707,
+            0.006095050368458033,
+            -0.0026293133851140738,
+            -0.013615119270980358,
+            0.008906363509595394,
+            0.003055269829928875,
+            0.00800798274576664,
+            -0.04615819454193115,
+            0.018478767946362495,
+            -0.00454224506393075,
+            0.04033420979976654,
+            0.042502716183662415,
+            -0.02730768360197544,
+            -0.016743963584303856,
+            0.008557853288948536,
+            0.012104909867048264,
+            0.04188314080238342,
+            -0.007287728600203991,
+            -0.00121784838847816,
+            -0.0007836631848476827,
+            0.007322579622268677,
+            0.047552235424518585,
+            0.004646798130124807,
+            0.0020697612781077623,
+            -0.027230238541960716,
+            0.014978179708123207,
+            -0.019733402878046036,
+            0.027648448944091797,
+            0.023373395204544067,
+            0.02942972257733345,
+            -0.04048910364508629,
+            0.017518429085612297,
+            -0.011818356812000275,
+            -0.005568413063883781,
+            -0.002726121572777629,
+            -0.02856232039630413,
+            -0.0074426219798624516,
+            0.033983584493398666,
+            0.015365412458777428,
+            0.012956822291016579,
+            0.03271345794200897,
+            0.0012052633101120591,
+            -0.042502716183662415,
+            0.03804178908467293,
+            0.024550583213567734,
+            0.04640602692961693,
+            0.023466331884264946,
+            0.010796061716973782,
+            0.006571347359567881,
+            0.013375034555792809,
+            -0.019377149641513824,
+            0.015907539054751396,
+            0.02560385875403881,
+            0.009750531986355782,
+            -0.03167567402124405,
+            -0.0009501734166406095,
+            0.01178737822920084,
+            -0.004546117503196001,
+            0.03701949119567871,
+            0.04104671627283096,
+            0.020569827407598495,
+            0.05030933395028114,
+            0.0020155487582087517,
+            0.017936641350388527,
+            0.005936284549534321,
+            0.02820606529712677,
+            0.030467506498098373,
+            -0.005285732913762331,
+            0.005076627247035503,
+            -0.01136916596442461,
+            -0.03385967016220093,
+            0.014110777527093887,
+            0.01561324205249548,
+            -0.008077683858573437,
+            -0.01697630248963833,
+            0.006033093202859163,
+            0.009169681929051876,
+            -0.01795213110744953,
+            -0.017456471920013428,
+            0.005971135571599007,
+            0.018866000697016716,
+            -0.013576395809650421,
+            0.024689987301826477,
+            -0.04442339017987251,
+            0.029119934886693954,
+            -0.033209118992090225,
+            -0.003384418087080121,
+            -0.010641167871654034,
+            -0.0030843124259263277,
+            0.0041666291654109955,
+            0.011314953677356243,
+            0.013406013138592243,
+            0.002042655134573579,
+            -0.013909416273236275,
+            0.024519605562090874,
+            0.018447788432240486,
+            0.012073930352926254,
+            -0.03148980438709259,
+            -0.014397329650819302,
+            -0.0015673263696953654,
+            -0.008743725717067719,
+            -0.034138478338718414,
+            0.026966920122504234,
+            0.014048820361495018,
+            -0.03844451159238815,
+            0.017797237262129784,
+            0.029538147151470184,
+            0.014420564286410809,
+            -0.004790074657648802,
+            -0.0463130883872509,
+            0.04045812413096428,
+            -0.030436528846621513,
+            -0.033333033323287964,
+            -0.010478530079126358,
+            -0.0003146269591525197,
+            0.02894955314695835,
+            -0.0064358157105743885,
+            -0.050835974514484406,
+            0.01184933539479971,
+            0.020631784573197365,
+            0.00010763872705865651,
+            0.005731051322072744,
+            -0.019593998789787292,
+            0.00028631053282879293,
+            -0.03395260497927666,
+            0.0090844901278615,
+            0.026750069111585617,
+            0.008867640048265457,
+            -0.016418687999248505,
+            0.03351890295743942,
+            -0.0004121613164898008,
+            -0.018695618957281113,
+            -0.003889757441356778,
+            0.006885006092488766,
+            0.034912943840026855,
+            0.004836542531847954,
+            -0.02946070022881031,
+            0.05665995925664902,
+            -0.02100352756679058,
+            0.021344292908906937,
+            0.05631919577717781,
+            -0.019439106807112694,
+            -0.008472662419080734,
+            -0.002065889071673155,
+            0.0026196325197815895,
+            -0.0260530486702919,
+            -0.008937342092394829,
+            0.032341714948415756,
+            0.041728246957063675,
+            -0.018742086365818977,
+            -0.001398879918269813,
+            0.006780453026294708,
+            0.009425255469977856,
+            0.004948840010911226,
+            0.02692045085132122,
+            -0.012128143571317196,
+            -0.014559967443346977,
+            0.05198218300938606,
+            0.028314489871263504,
+            -0.012081675231456757,
+            -0.0164031982421875,
+            0.019888296723365784,
+            -0.024844881147146225,
+            -0.020244551822543144,
+            -0.02431824430823326,
+            -0.020414933562278748,
+            -0.0042518205009400845,
+            0.009525936096906662,
+            0.037484172731637955,
+            -0.03968365862965584,
+            0.012747716158628464,
+            0.021700548008084297,
+            0.02981695532798767,
+            -0.019532041624188423,
+            -0.05910727381706238,
+            -0.0016128262504935265,
+            0.03296129032969475,
+            -0.015458349138498306,
+            -0.049379974603652954,
+            0.015164051204919815,
+            0.0260530486702919,
+            0.05957195535302162,
+            -0.001461805310100317,
+            -0.014954946003854275,
+            0.061926331371068954,
+            0.03026614524424076,
+            -0.004383479710668325,
+            -0.009494957514107227,
+            -0.0127864396199584,
+            0.003396035172045231,
+            0.00465454300865531,
+            -0.03364282101392746,
+            -0.041170634329319,
+            -0.011841590516269207,
+            -0.010284913703799248,
+            0.008108663372695446,
+            -0.040922801941633224,
+            0.03531566634774208,
+            -0.021917399019002914,
+            0.010950954630970955,
+            0.015473837964236736,
+            0.037391237914562225,
+            -0.021344292908906937,
+            -0.01284839678555727,
+            -0.01572941243648529,
+            -0.024968795478343964,
+            -0.014010096900165081,
+            0.021917399019002914,
+            0.007682706229388714,
+            3.4699722164077684e-05,
+            0.010075807571411133,
+            -0.024659009650349617,
+            -0.051052823662757874,
+            0.0291044469922781,
+            0.0015605497173964977,
+            -0.015938518568873405,
+            -0.004623564425855875,
+            -0.01977987214922905,
+            -0.01122201792895794,
+            -0.012608313001692295,
+            -0.039249956607818604,
+            0.021111954003572464,
+            -0.01135367713868618,
+            0.015876561403274536,
+            0.0009250032599084079,
+            0.003659353591501713,
+            0.028624277561903,
+            0.01116006076335907,
+            0.0051811798475682735,
+            -0.03255856782197952,
+            -0.018277406692504883,
+            0.010122275911271572,
+            -0.00561100896447897,
+            -0.018339363858103752,
+            -0.013351799920201302,
+            -0.05037129297852516,
+            -0.03305422514677048,
+            -0.008054450154304504,
+            -0.05415068939328194,
+            0.0020697612781077623,
+            -0.014296649023890495,
+            -0.02232012152671814,
+            -0.005990497302263975,
+            0.03107159025967121,
+            -0.003235333366319537,
+            0.025294071063399315,
+            0.0076594725251197815,
+            0.037360258400440216,
+            0.040117356926202774,
+            0.008426194079220295,
+            0.03065337985754013,
+            -0.0029991210903972387,
+            -0.016867877915501595,
+            0.05486319959163666,
+            -0.01881953328847885,
+            0.013529927469789982,
+            -0.008604321628808975,
+            0.008604321628808975,
+            0.03299226611852646,
+            -0.003794885240495205,
+            0.02007416822016239,
+            -0.0311955064535141,
+            -0.07998688519001007,
+            0.01637221872806549,
+            0.028159597888588905,
+            -0.06418777257204056,
+            0.014691627584397793,
+            -0.020275529474020004,
+            0.02052335813641548,
+            0.02411688305437565,
+            -0.0009114501299336553,
+            -0.014064309187233448,
+            0.005595519673079252,
+            0.0020833145827054977,
+            -0.01456771232187748,
+            -0.002691270550712943,
+            -0.009417511522769928,
+            0.008774704299867153,
+            -0.017007282003760338,
+            -0.013406013138592243,
+            -0.034231413155794144,
+            0.010873507708311081,
+            0.01233724970370531,
+            -0.003969140350818634,
+            0.0011500825639814138,
+            -0.03277541697025299,
+            -0.007876322604715824,
+            -0.02145271934568882,
+            0.03382869064807892,
+            0.012058441527187824,
+            -0.030699847266077995,
+            -0.007814365439116955,
+            0.015264731831848621,
+            -0.007876322604715824,
+            0.0008906363509595394,
+            0.024132372811436653,
+            0.02103450708091259,
+            0.0028364830650389194,
+            -0.0009700191440060735,
+            0.02438020147383213,
+            0.007427132688462734,
+            -0.01720864325761795,
+            0.028794659301638603,
+            0.014288905076682568,
+            0.060160547494888306,
+            -0.03271345794200897,
+            0.001297231181524694,
+            -0.04135650396347046,
+            -0.031118059530854225,
+            0.006408709101378918,
+            0.014350862242281437,
+            0.007597515359520912,
+            -0.028252532705664635,
+            0.005978880450129509,
+            0.02004319056868553,
+            0.01469937153160572,
+            0.009704063646495342,
+            0.0037580979987978935,
+            -0.036740683019161224,
+            -0.000691695255227387,
+            -0.011214273050427437,
+            -0.0038781403563916683,
+            0.035656433552503586,
+            -0.008186109364032745,
+            -0.05151750147342682,
+            -0.0006418389966711402,
+            -0.03255856782197952,
+            -0.0353776253759861,
+            0.021313315257430077,
+            -0.00839521549642086,
+            -0.0039207362569868565,
+            -0.013986863195896149,
+            -0.000537770101800561,
+            -0.010524998418986797,
+            0.027509044855833054,
+            0.028345469385385513,
+            0.022490503266453743,
+            -0.05384090170264244,
+            0.009479468688368797,
+            0.00023766436788719147,
+            -0.05817791447043419,
+            0.0009709871956147254,
+            -0.007117345929145813,
+            0.011191039346158504,
+            -0.020213572308421135,
+            -0.02414786070585251,
+            0.011500826105475426,
+            0.03392162546515465,
+            0.014304393902420998,
+            0.0321558453142643,
+            -0.013367289677262306,
+            -0.09250226616859436,
+            0.018292896449565887,
+            -0.008689512498676777,
+            0.04126356914639473,
+            0.00865078903734684,
+            -0.004278926644474268,
+            -0.015845581889152527,
+            -0.00865078903734684,
+            -0.006943091284483671,
+            0.01283290795981884,
+            0.004530628211796284,
+            -0.005626498255878687,
+            0.0016631665639579296,
+            0.013708055019378662,
+            -0.005967263597995043,
+            0.007481344975531101,
+            0.003651608945801854,
+            -0.013189162127673626,
+            -0.06059424951672554,
+            -0.04055105894804001,
+            -0.018540725111961365,
+            0.050835974514484406,
+            -0.0069159846752882,
+            -0.046684831380844116,
+            -0.010850274004042149,
+            0.05058814212679863,
+            0.012259802781045437,
+            -0.0334569476544857,
+            -0.0007841471815481782,
+            0.0050882440991699696,
+            0.015458349138498306,
+            0.004546117503196001,
+            -0.027292195707559586,
+            0.02820606529712677,
+            0.0008412640891037881,
+            0.02300165221095085,
+            -0.018339363858103752,
+            -0.005312839522957802,
+            0.035780347883701324,
+            -0.024844881147146225,
+            0.023791607469320297,
+            0.0055219451896846294,
+            0.031629208475351334,
+            0.002952652983367443,
+            0.010060317814350128,
+            0.004704883322119713,
+            0.0186491496860981,
+            0.004275054205209017,
+            0.0276639387011528,
+            0.011462102644145489,
+            0.006428070832043886,
+            0.021623101085424423,
+            -0.009378788061439991,
+            -0.014513500034809113,
+            0.01885051093995571,
+            -0.0014695499558001757,
+            0.025232113897800446,
+            -0.020538847893476486,
+            0.002726121572777629,
+            -0.0016883367206901312,
+            -0.017967619001865387,
+            -0.040922801941633224,
+            -0.0007405834621749818,
+            0.021747015416622162,
+            -0.011671207845211029,
+            -0.0228312686085701,
+            -0.013715799897909164,
+            -0.01585332676768303,
+            0.0038490979932248592,
+            0.034169457852840424,
+            -0.0398695282638073,
+            0.00963436160236597,
+            -0.000830615172162652,
+            -0.03125746175646782,
+            -0.0239619892090559,
+            -0.01093546487390995,
+            0.014614180661737919,
+            -0.01652711257338524,
+            -0.021173911169171333,
+            -0.028469383716583252,
+            -0.008410705253481865,
+            0.006048582494258881,
+            -0.05065010115504265,
+            -0.004054331686347723,
+            0.019965743646025658,
+            0.0276639387011528,
+            -0.003099801717326045,
+            -0.02177799493074417,
+            0.0012468908680602908,
+            0.022351099178195,
+            0.02936776541173458,
+            0.014226946979761124,
+            0.027586491778492928,
+            -0.002178186783567071,
+            -0.0011675080750137568,
+            -0.0073148347437381744,
+            0.01307299267500639,
+            0.0011287847300991416,
+            0.036616768687963486,
+            0.0122210793197155,
+            -0.020306508988142014,
+            0.0013959755888208747,
+            -0.006431943271309137,
+            0.029181893914937973,
+            -0.02270735427737236,
+            -9.832092473516241e-05,
+            0.03531566634774208,
+            0.027741385623812675,
+            0.004933350719511509,
+            -0.0143121387809515,
+            -0.05142456665635109,
+            0.021592123433947563,
+            -0.01765783317387104,
+            0.031025122851133347,
+            0.016666516661643982,
+            0.003397971158847213,
+            -0.0034212053287774324,
+            0.012368228286504745,
+            -0.022087780758738518,
+            -0.004406713880598545,
+            -0.021142931655049324,
+            0.04389675334095955,
+            -0.043184246867895126,
+            0.008472662419080734,
+            0.029445212334394455,
+            0.011926782317459583,
+            -0.019082851707935333,
+            -0.005425137002021074,
+            0.029212871566414833,
+            -0.00162637943867594,
+            0.01308848150074482,
+            0.0058665829710662365,
+            -0.026935940608382225,
+            0.01202746294438839,
+            -0.031660184264183044,
+            0.02177799493074417,
+            -0.013436991721391678,
+            0.018447788432240486,
+            0.03342596814036369,
+            -0.0030397805385291576,
+            -0.0038239278364926577,
+            -0.016263794153928757,
+            -0.00969631876796484,
+            0.0662323608994484,
+            0.01054823212325573,
+            0.03726732358336449,
+            0.0018974426202476025,
+            -0.009773765690624714,
+            0.007969259284436703,
+            0.0063041565008461475,
+            -0.012329504825174809,
+            0.010494019836187363,
+            0.024178840219974518,
+            0.0012730291346088052,
+            -0.002412462839856744,
+            -0.019408127292990685,
+            -0.03438630700111389,
+            0.003727119415998459,
+            -0.030932186171412468,
+            0.022691864520311356,
+            0.009208405390381813,
+            0.017456471920013428,
+            0.008240322582423687,
+            -0.015907539054751396,
+            0.03200095146894455,
+            0.05461536720395088,
+            0.0019768255297094584,
+            0.013375034555792809,
+            0.007938280701637268,
+            0.007036027032881975,
+            -0.011338187381625175,
+            0.04702559858560562,
+            0.000662652775645256,
+            0.005169562995433807,
+            0.022536970674991608,
+            0.02145271934568882,
+            0.022939693182706833,
+            0.002304037567228079,
+            -0.011694442480802536,
+            0.010478530079126358,
+            -0.0006302220281213522,
+            0.004445436876267195,
+            -0.010695381090044975,
+            -0.027865299955010414,
+            0.007492962293326855,
+            0.01302652433514595,
+            0.022598929703235626,
+            0.021592123433947563,
+            -0.03178409859538078,
+            -0.01855621486902237,
+            0.03364282101392746,
+            -0.02997184917330742,
+            0.03612111136317253,
+            -0.0014424435794353485,
+            -0.023807097226381302,
+            -0.0058356039226055145,
+            0.016929835081100464,
+            0.02656419761478901,
+            -0.029181893914937973,
+            0.01804506592452526,
+            0.029693040996789932,
+            -0.0074426219798624516,
+            0.0005939188995398581,
+            0.012112653814256191,
+            0.02447313815355301,
+            -0.009293596260249615,
+            -0.0024027819745242596,
+            -0.0013320822035893798,
+            -0.011926782317459583,
+            -0.018370341509580612,
+            -0.0044725434854626656,
+            -0.022629907354712486,
+            0.005564541090279818,
+            -0.007365175057202578,
+            -0.003481226274743676,
+            -0.008581086993217468,
+            0.01154729351401329,
+            0.001979729626327753,
+            -0.019857317209243774,
+            -0.022335609421133995,
+            0.01652711257338524,
+            0.03887821361422539,
+            -0.018215449526906013,
+            -0.02106548473238945,
+            0.024442158639431,
+            -0.01759587600827217,
+            -0.01601596549153328,
+            0.036678727716207504,
+            0.013739033602178097,
+            0.007736918982118368,
+            -0.03358086198568344,
+            -0.030049296095967293,
+            -0.03274443745613098,
+            0.006296411622315645,
+            0.03785591572523117,
+            -0.0028752065263688564,
+            0.02569679357111454,
+            0.03701949119567871,
+            0.0057891360484063625,
+            -0.028934063389897346,
+            -0.014823286794126034,
+            -0.045104920864105225,
+            0.060160547494888306,
+            0.0011481463443487883,
+            0.05161044001579285,
+            0.016294771805405617,
+            -0.01456771232187748,
+            0.00419760774821043,
+            -0.013251119293272495,
+            0.026192452758550644,
+            0.02280029095709324,
+            0.025929134339094162,
+            0.006582964211702347,
+            0.022505993023514748,
+            -0.0037522895727306604,
+            -0.012197845615446568,
+            0.007307090330868959,
+            0.014536733739078045,
+            0.0002497654058970511,
+            -0.011392400600016117,
+            0.015783624723553658,
+            -0.01307299267500639,
+            -0.011361422017216682,
+            0.023295948281884193,
+            0.01368482131510973,
+            -0.012344993650913239,
+            0.014064309187233448,
+            -0.008565598167479038,
+            0.01239920686930418,
+            -0.023915521800518036,
+            0.022691864520311356,
+            -0.03262052312493324,
+            0.009587894193828106,
+            0.0015053689712658525,
+            -0.0073148347437381744,
+            0.024550583213567734,
+            -0.0027454833034425974,
+            -0.05071205645799637,
+            0.03550153970718384,
+            -0.008015726692974567,
+            0.0037213109899312258,
+            0.006381602957844734,
+            0.011392400600016117,
+            -0.0012623801594600081,
+            0.03807276859879494,
+            0.01239920686930418,
+            0.015969496220350266,
+            0.02199484594166279,
+            0.011911292560398579,
+            -0.008433938957750797,
+            -0.004096927121281624,
+            -0.0016137943603098392,
+            -0.01784370467066765,
+            0.025944624096155167,
+            0.0018674320308491588,
+            0.03962169960141182,
+            -0.018168980255723,
+            -0.02300165221095085,
+            0.028066661208868027,
+            0.007597515359520912,
+            -0.02351279929280281,
+            -0.0017377089243382215,
+            0.01202746294438839,
+            -0.009185170754790306,
+            0.027385130524635315,
+            0.01196550577878952,
+            0.019113829359412193,
+            -0.018060555681586266,
+            -0.004340883810073137,
+            -0.005072754807770252,
+            -0.015938518568873405,
+            0.02238207869231701,
+            0.020771188661456108,
+            0.011152315884828568,
+            -0.011880313977599144,
+            -0.01307299267500639,
+            0.020600805059075356,
+            0.035036858171224594,
+            -0.0002744515077210963,
+            0.005672966130077839,
+            0.017487451434135437,
+            0.002373739378526807,
+            -0.04470219835639,
+            0.016232814639806747,
+            -0.014025585725903511,
+            0.011214273050427437,
+            0.019593998789787292,
+            -0.021917399019002914,
+            -0.03196997195482254,
+            -0.011539548635482788,
+            -0.0414804182946682,
+            0.00493722315877676,
+            -0.0069159846752882,
+            0.013576395809650421,
+            -0.010176488198339939,
+            -0.0014831031439825892,
+            0.016666516661643982,
+            0.0038103745318949223,
+            0.02177799493074417,
+            -0.011678952723741531,
+            -0.012073930352926254,
+            -0.020213572308421135,
+            -0.020786678418517113,
+            0.004278926644474268,
+            -0.030405549332499504,
+            -0.004488032776862383,
+            0.021375272423028946,
+            -0.04783104360103607,
+            -0.03689557686448097,
+            0.02898053266108036,
+            -0.008503641001880169,
+            0.020755698904395103,
+            0.022955182939767838,
+            -0.015094350092113018,
+            -0.0008717587334103882,
+            -0.018695618957281113,
+            0.02258343994617462,
+            -0.007837600074708462,
+            0.026300877332687378,
+            -0.0027551641687750816,
+            0.0006292539183050394,
+            -0.02659517526626587,
+            0.020306508988142014,
+            0.005227648187428713,
+            0.012275291606783867,
+            -0.02390003204345703,
+            0.006815304048359394,
+            -0.03133491054177284,
+            -0.01784370467066765,
+            0.0180140882730484,
+            -0.011152315884828568,
+            0.027989214286208153,
+            -0.006060199346393347,
+            0.028438406065106392,
+            0.022722844034433365,
+            0.024751944467425346,
+            0.02676555886864662,
+            -0.02363671362400055,
+            0.008495896123349667,
+            0.03726732358336449,
+            0.00865078903734684,
+            -0.015876561403274536,
+            0.008170620538294315,
+            -0.04863648861646652,
+            -0.008062195032835007,
+            0.05030933395028114,
+            0.002127846470102668,
+            0.023141054436564445,
+            0.016294771805405617,
+            -0.002873270306736231,
+            -0.005688455421477556,
+            0.010269423946738243,
+            -0.04296739399433136,
+            -0.023172033950686455,
+            0.022629907354712486,
+            -0.021561143919825554,
+            6.54212328754511e-07,
+            0.03122648410499096,
+            0.012120398692786694,
+            -0.0009109660750254989,
+            -0.05108380317687988,
+            0.02100352756679058,
+            0.01178737822920084,
+            0.014443797990679741,
+            -0.010300402529537678,
+            0.026068538427352905,
+            0.01733255758881569,
+            0.038413532078266144,
+            0.012755461037158966,
+            -0.002679653698578477,
+            -0.03782493621110916,
+            0.009177426807582378,
+            -0.021685058251023293,
+            0.0011471783509477973,
+            0.0074503663927316666,
+            -0.02273833192884922,
+            -0.024782923981547356,
+            0.0043912241235375404,
+            0.004588713403791189,
+            0.011880313977599144,
+            0.022087780758738518,
+            0.009417511522769928,
+            -0.021204888820648193,
+            0.015404135920107365,
+            -0.05446047708392143,
+            -0.015450604259967804,
+            -0.023884542286396027,
+            0.02566581591963768,
+            0.020012211054563522,
+            0.010184233076870441,
+            0.005545179359614849,
+            0.010904486291110516,
+            -0.020988039672374725,
+            -0.009239383973181248,
+            0.020631784573197365,
+            0.011074868962168694,
+            -0.012840652838349342,
+            0.04113965481519699,
+            -0.00011126903700642288,
+            0.031210994347929955,
+            -0.02280029095709324,
+            0.002941036131232977,
+            -0.001241082325577736,
+            -0.03649285435676575,
+            0.009130958467721939,
+            0.01981084980070591,
+            0.02151467651128769,
+            -0.01861817203462124,
+            0.0024143988266587257,
+            0.01048627495765686,
+            0.004766840487718582,
+            0.038351576775312424,
+            0.027121812105178833,
+            -0.044206541031599045,
+            0.011663463898003101,
+            0.012995545752346516,
+            -0.009363298304378986,
+            0.048419639468193054,
+            -0.017518429085612297,
+            0.017673322930932045,
+            0.008240322582423687,
+            -0.01798310875892639,
+            0.02318752370774746,
+            -0.01469937153160572,
+            -0.013282098807394505,
+            0.00650551775470376,
+            0.009192915633320808,
+            -0.0001770381786627695,
+            -0.037577107548713684,
+            -0.017859194427728653,
+            -0.005719434004276991,
+            0.0038297362625598907,
+            0.01562873087823391,
+            0.013498948886990547,
+            0.009967382065951824,
+            0.005161818582564592,
+            -0.00945623405277729,
+            -0.0016089539276435971,
+            0.008124152198433876,
+            -0.0005532593932002783,
+            0.0019216446671634912,
+            -0.01926872320473194,
+            0.02698240801692009,
+            0.02572777308523655,
+            -0.0018354853382334113,
+            -0.04652994126081467,
+            -0.0036709706764668226,
+            0.0037077576853334904,
+            -0.017735280096530914,
+            -0.0035315665882080793,
+            -0.04519785940647125,
+            -0.01810702309012413,
+            0.0143121387809515,
+            -0.014614180661737919,
+            -0.0460033044219017,
+            0.009076746180653572,
+            -0.013886182568967342,
+            0.0063970922492444515,
+            -0.00932457484304905,
+            0.005762029904872179,
+            0.003651608945801854,
+            0.005680711008608341,
+            -0.005909178406000137,
+            0.05108380317687988,
+            -0.03466511517763138,
+            0.028051171451807022,
+            0.008565598167479038,
+            -0.012437930330634117,
+            -0.02261441759765148,
+            0.00887538492679596,
+            -0.044981006532907486,
+            -0.033209118992090225,
+            0.016743963584303856,
+            -0.01781272701919079,
+            0.011074868962168694,
+            0.06161654368042946,
+            0.007330324035137892,
+            -0.023342415690422058,
+            -0.01233724970370531,
+            -0.021049996837973595,
+            -0.04191412031650543,
+            0.003442503046244383,
+            0.005463860463351011,
+            -0.004948840010911226,
+            -0.03807276859879494,
+            -0.021716037765145302,
+            0.04615819454193115,
+            -0.012004229240119457,
+            0.012422440573573112,
+            0.01080380566418171,
+            -0.0019410063978284597,
+            -0.04860550910234451,
+            -0.021019017323851585,
+            0.0395597442984581,
+            0.021886419504880905,
+            -0.008426194079220295,
+            0.0028364830650389194,
+            -0.018277406692504883,
+            -0.003289545886218548,
+            0.006296411622315645,
+            -0.009107724763453007,
+            -0.03562545403838158,
+            0.02067825198173523,
+            0.0054832217283546925,
+            -0.023373395204544067,
+            -0.025805220007896423,
+            -0.03296129032969475,
+            0.009944148361682892,
+            0.02453509531915188,
+            -0.01968693546950817,
+            0.005823987070471048,
+            0.00027203132049180567,
+            -0.000881923595443368,
+            0.00042740863864310086,
+            -0.009959637187421322,
+            -0.019764382392168045,
+            -0.020151615142822266,
+            0.03268248215317726,
+            -0.0060059865936636925,
+            0.0074387495405972,
+            -0.03630698472261429,
+            -0.010974188335239887,
+            0.011322698555886745,
+            -0.002395037328824401,
+            -0.013452480547130108,
+            -0.009092235006392002,
+            -0.01739451475441456,
+            -0.009882191196084023,
+            -0.007876322604715824,
+            0.022660886868834496,
+            0.006168624851852655,
+            0.005409647710621357,
+            0.0023795480374246836,
+            -0.008581086993217468,
+            0.02898053266108036,
+            0.017874684184789658,
+            -0.014188224449753761,
+            -0.00920066051185131,
+            0.03451022133231163,
+            0.02665713243186474,
+            -0.009115469641983509,
+            0.01798310875892639,
+            0.023264970630407333,
+            -0.0010484338272362947,
+            -0.01196550577878952,
+            0.001922612776979804,
+            -0.0009550138493068516,
+            -0.0032972905319184065,
+            0.00845717266201973,
+            -0.015365412458777428,
+            0.0315207801759243,
+            0.04287445917725563,
+            -0.002145271748304367,
+            0.009587894193828106,
+            -0.003727119415998459,
+            -0.008472662419080734,
+            0.017069239169359207,
+            0.009316830895841122,
+            0.04625113308429718,
+            0.01098193321377039,
+            0.005680711008608341,
+            -0.033983584493398666,
+            -0.001229465357027948,
+            -0.01184933539479971,
+            0.03308520466089249,
+            0.02061629481613636,
+            0.00889087375253439,
+            -0.008379725739359856,
+            0.01344473659992218,
+            0.008341003209352493,
+            0.00661007035523653,
+            0.020569827407598495,
+            -0.016542602330446243,
+            -0.017286090180277824,
+            0.030467506498098373,
+            -0.00672236829996109,
+            0.01289486512541771,
+            0.02029101923108101,
+            -0.002089123008772731,
+            0.010842529125511646,
+            0.07100307941436768,
+            -0.006184114143252373,
+            0.015713922679424286,
+            -0.019082851707935333,
+            0.0015789433382451534,
+            -0.007640110794454813,
+            -0.03370477631688118,
+            0.018246427178382874,
+            0.0039962464943528175,
+            0.00865078903734684,
+            0.01198099460452795,
+            0.009053511545062065,
+            -0.007946024648845196,
+            -0.00963436160236597,
+            0.015195030719041824,
+            0.0019080914789810777,
+            0.00033568276558071375,
+            0.001336922636255622,
+            -0.0065210070461034775,
+            -0.004011735785752535,
+            -0.021948376670479774,
+            0.02763296104967594,
+            0.007140580099076033,
+            0.030606910586357117,
+            -0.055296897888183594,
+            0.030095763504505157,
+            -0.00789568480104208,
+            0.02554190158843994,
+            -0.016201836988329887,
+            -0.011090358719229698,
+            0.001609921921044588,
+            0.018153492361307144,
+            0.004491904750466347,
+            0.006462921854108572,
+            0.011888058856129646,
+            0.023233991116285324,
+            -0.03444826602935791,
+            0.017053749412298203,
+            0.023110076785087585,
+            0.0048675211146473885,
+            0.0003535923024173826,
+            0.005165690556168556,
+            0.015458349138498306,
+            0.007198664825409651,
+            -0.023048119619488716,
+            -0.0025692922063171864,
+            0.017611365765333176,
+            -0.02306360937654972,
+            0.0036980770528316498,
+            -0.006923729553818703,
+            0.004623564425855875,
+            -0.0014705180656164885,
+            -0.03509881719946861,
+            -0.00100099784322083,
+            -0.025743262842297554,
+            0.01923774555325508,
+            0.009556914679706097,
+            -0.025867177173495293,
+            0.003932353109121323,
+            0.025650326162576675,
+            -0.0003695656778290868,
+            0.00821708794683218,
+            0.0004683101433329284,
+            0.00869725737720728,
+            -0.00974278710782528,
+            -0.01493945624679327,
+            0.004801691509783268,
+            0.01531120017170906,
+            -0.01493945624679327,
+            -0.007431004662066698,
+            -0.009510447271168232,
+            0.030560443177819252,
+            0.0006674932083114982,
+            0.004189862869679928,
+            -0.005045648664236069,
+            -0.01858719252049923,
+            -0.0032237162813544273,
+            -0.021142931655049324,
+            0.004367990419268608,
+            -0.0009046735358424485,
+            0.04197607934474945,
+            0.0038161829579621553,
+            -0.02029101923108101,
+            0.0032972905319184065,
+            -0.03136589005589485,
+            0.0005000148667022586,
+            -0.007276111748069525,
+            -0.02004319056868553,
+            0.0015005286550149322,
+            0.0016525175888091326,
+            0.012631546705961227,
+            -0.03255856782197952,
+            -0.009301341138780117,
+            0.018416810780763626,
+            -0.00020680672605521977,
+            -0.010687636211514473,
+            -0.017828216776251793,
+            0.00035189816844649613,
+            -0.011338187381625175,
+            -0.027973726391792297,
+            -0.02850036323070526,
+            -0.0026719090528786182,
+            0.013491204008460045,
+            -0.018277406692504883,
+            -0.014993669465184212,
+            0.0031249718740582466,
+            0.021592123433947563,
+            -0.009781510569155216,
+            0.008302279748022556,
+            0.02241305634379387,
+            -0.008255811408162117,
+            -0.005514200776815414,
+            -0.000941460661124438,
+            -0.009130958467721939,
+            0.004267309792339802,
+            0.006300284061580896,
+            0.043339136987924576,
+            -0.032403673976659775,
+            -0.0018238683696836233,
+            0.014815541915595531,
+            0.0034153966698795557,
+            0.0025905899237841368,
+            0.002503462601453066,
+            -0.0007841471815481782,
+            0.01344473659992218,
+            0.01178737822920084,
+            0.015466093085706234,
+            0.00100099784322083,
+            -0.0033011629711836576,
+            0.006408709101378918,
+            -0.021173911169171333,
+            0.015721667557954788,
+            0.027958236634731293,
+            -0.0033999073784798384,
+            0.005525817628949881,
+            -0.013731288723647594,
+            -0.008759214542806149,
+            0.013134949840605259,
+            0.0026641644071787596,
+            -0.005068882368505001,
+            -0.009394276887178421,
+            0.017967619001865387,
+            0.019082851707935333,
+            0.006668155547231436,
+            -0.037453193217515945,
+            -0.02898053266108036,
+            -0.009959637187421322,
+            0.020569827407598495,
+            0.014916222542524338,
+            0.0010339126456528902,
+            -0.010540487244725227,
+            -0.009595638141036034,
+            -0.011903548613190651,
+            0.010989678092300892,
+            0.02721474878489971,
+            -0.032372694462537766,
+            0.0008214184199459851,
+            -0.029383255168795586,
+            0.03068435750901699,
+            0.0020000594668090343,
+            0.011446612887084484,
+            -0.0008185141487047076,
+            -0.015140817500650883,
+            -0.029181893914937973,
+            0.013824224472045898,
+            -0.004395096562802792,
+            0.001024231780320406,
+            -0.0031753121875226498,
+            0.0201051477342844,
+            0.012887120246887207,
+            -0.0014472840121015906,
+            0.01968693546950817,
+            0.008178364485502243,
+            0.016480645164847374,
+            -0.010153254494071007,
+            -0.04135650396347046,
+            0.019965743646025658,
+            0.007562664337456226,
+            0.014219203032553196,
+            0.004739734344184399,
+            0.004588713403791189,
+            -0.020771188661456108,
+            0.002724185585975647,
+            0.004333139397203922,
+            0.014405074529349804,
+            0.011028401553630829,
+            0.0025615475606173277,
+            0.011678952723741531,
+            -0.02228914201259613,
+            0.01929970271885395,
+            -0.01631026156246662,
+            0.01006806269288063,
+            0.004445436876267195,
+            -0.026873983442783356,
+            0.007547175046056509,
+            0.007822110317647457,
+            -0.0019187404541298747,
+            -0.017549408599734306,
+            0.02160761132836342,
+            -0.0034831624943763018,
+            -0.03509881719946861,
+            0.0360281765460968,
+            -0.0043447562493383884,
+            0.019346170127391815,
+            0.03209388628602028,
+            -0.01771979033946991,
+            -0.01993476413190365,
+            0.00906900130212307,
+            0.010408828034996986,
+            -0.002873270306736231,
+            0.017859194427728653,
+            -0.002919738180935383,
+            -0.02232012152671814,
+            0.017766257748007774,
+            -0.028918573632836342,
+            -0.014234691858291626,
+            0.020337486639618874,
+            -0.03624502569437027,
+            -0.0021026763133704662,
+            0.03175312280654907,
+            -0.024070415645837784,
+            0.005734923295676708,
+            0.005196669604629278,
+            -0.0019351978553459048,
+            0.0012236569309607148,
+            -0.006451305001974106,
+            0.03801080957055092,
+            -0.010865762829780579,
+            0.0201051477342844,
+            0.01929970271885395,
+            -0.023388884961605072,
+            0.01977987214922905,
+            -0.001109423115849495,
+            0.01675945147871971,
+            -0.003940097521990538,
+            0.03466511517763138,
+            -0.006017603911459446,
+            -0.011330443434417248,
+            0.006431943271309137,
+            0.008488151244819164,
+            -0.009959637187421322,
+            0.008681767620146275,
+            -0.018122512847185135,
+            0.027152791619300842,
+            0.02091059274971485,
+            -0.0002691270783543587,
+            0.013800990767776966,
+            0.0004421719058882445,
+            -0.020833145827054977,
+            0.00789568480104208,
+            -0.027121812105178833,
+            -0.009967382065951824,
+            0.01646515540778637,
+            -0.005762029904872179,
+            0.007380664348602295,
+            -0.02064727433025837,
+            0.028004704043269157,
+            0.0077524082735180855,
+            -0.013591884635388851,
+            -0.0008824076503515244,
+            0.023760627955198288,
+            -0.005432881414890289,
+            0.009355553425848484,
+            0.02070923149585724,
+            0.015713922679424286,
+            0.013615119270980358,
+            0.016263794153928757,
+            0.03351890295743942,
+            0.005328328814357519,
+            0.005568413063883781,
+            0.003519949736073613,
+            0.016991792246699333,
+            -0.008557853288948536,
+            0.018029578030109406,
+            0.0008093173382803798,
+            -0.013576395809650421,
+            0.021499186754226685,
+            -0.0018684001406654716,
+            -0.013622863218188286,
+            -0.0040891822427511215,
+            -0.004337011836469173,
+            0.03271345794200897,
+            0.022041313350200653,
+            -0.004468671046197414,
+            -0.024442158639431,
+            -0.00394784240052104,
+            0.016682006418704987,
+            -0.003167567541822791,
+            0.011136826127767563,
+            0.0007952801533974707,
+            -0.01586107164621353,
+            -0.009595638141036034,
+            0.0063583687879145145,
+            0.010586955584585667,
+            -0.008921852335333824,
+            0.010346870869398117,
+            -0.0228312686085701,
+            -0.004011735785752535,
+            0.00046613195445388556,
+            -0.004844287410378456,
+            0.0053438181057572365,
+            -0.011314953677356243,
+            0.004948840010911226,
+            0.016062432900071144,
+            0.01026167906820774,
+            0.004832670092582703,
+            0.006989559158682823,
+            0.012066186405718327,
+            0.009161937050521374,
+            -0.01627928391098976,
+            0.00428667152300477,
+            -0.005072754807770252,
+            -0.02345084212720394,
+            0.005831731949001551,
+            0.00902253296226263,
+            -0.01122201792895794,
+            0.009386532939970493,
+            -0.010958699509501457,
+            0.03296129032969475,
+            0.01030814740806818,
+            -0.037391237914562225,
+            -0.0033379499800503254,
+            -0.0016244432190433145,
+            0.014885243959724903,
+            0.012298526242375374,
+            0.008550108410418034,
+            -0.017518429085612297,
+            -0.0047823297791182995,
+            -0.009835722856223583,
+            0.0012827098835259676,
+            -0.012306271120905876,
+            0.017456471920013428,
+            0.022567950189113617,
+            -0.036337960511446,
+            0.01968693546950817,
+            -0.010432061739265919,
+            -0.023652203381061554,
+            0.020476890727877617,
+            0.011500826105475426,
+            0.01942361705005169,
+            -0.009402021765708923,
+            0.018896980211138725,
+            -0.0001331114035565406,
+            0.025510922074317932,
+            -0.004228586331009865,
+            -0.022087780758738518,
+            -0.012499887496232986,
+            -0.03209388628602028,
+            -0.0051811798475682735,
+            0.006819176487624645,
+            -0.00045524101005867124,
+            0.013111716136336327,
+            0.014226946979761124,
+            -0.025712283328175545,
+            0.007295473478734493,
+            0.032589543610811234,
+            0.014637414366006851,
+            0.023311438038945198,
+            -0.011276230216026306,
+            -0.0010794125264510512,
+            -0.0026060794480144978,
+            -0.013770012184977531,
+            -0.03212486580014229,
+            -0.0017493258928880095,
+            -0.036585792899131775,
+            0.005680711008608341,
+            -0.016387708485126495,
+            -0.013096226379275322,
+            0.023264970630407333,
+            -0.034138478338718414,
+            0.0014608372002840042,
+            0.013708055019378662,
+            0.010540487244725227,
+            0.012391461990773678,
+            -0.00162637943867594,
+            0.015032391995191574,
+            -0.029243851080536842,
+            -0.010284913703799248,
+            -0.004491904750466347,
+            -0.01981084980070591,
+            -0.007338068913668394,
+            -0.018060555681586266,
+            -0.013127204962074757,
+            -0.003566417610272765,
+            -0.010060317814350128,
+            0.002247888594865799,
+            0.0032062907703220844,
+            -0.014714861288666725,
+            0.03206290677189827,
+            0.003157886676490307,
+            0.018958937376737595,
+            0.0033418224193155766,
+            0.004449309315532446,
+            0.018788553774356842,
+            -0.0026331855915486813,
+            -0.0058278595097362995,
+            0.030963165685534477,
+            -0.00032672801171429455,
+            0.008759214542806149,
+            0.019082851707935333,
+            0.004724245052784681,
+            0.0038529701996594667,
+            -0.0260530486702919,
+            0.012887120246887207,
+            -0.027447087690234184,
+            0.005080499220639467,
+            -0.0077175572514534,
+            -0.024302754551172256,
+            -0.028128618374466896,
+            -0.006571347359567881,
+            0.012058441527187824,
+            -0.02898053266108036,
+            -0.00613377382978797,
+            0.002315654419362545,
+            -0.00400399137288332,
+            -0.002673845272511244,
+            0.007241260726004839,
+            0.029150914400815964,
+            -0.033209118992090225,
+            0.008751469664275646,
+            0.005595519673079252,
+            -0.0005053392960689962,
+            0.013297587633132935,
+            0.008038961328566074,
+            0.003343758638948202,
+            0.020848635584115982,
+            0.006637176964432001,
+            0.002458930714055896,
+            -0.0025576751213520765,
+            -0.014242436736822128,
+            0.004000118933618069,
+            0.02154565416276455,
+            0.007562664337456226,
+            0.018385831266641617,
+            -0.008914108388125896,
+            0.002621568739414215,
+            -0.01455222349613905,
+            0.005552923772484064,
+            -0.002867461647838354,
+            0.004309905227273703,
+            -0.012236569076776505,
+            0.008093173615634441,
+            0.0028771425131708384,
+            -0.0043447562493383884,
+            0.018060555681586266,
+            -0.002319526858627796,
+            0.016480645164847374,
+            0.004015608225017786,
+            0.015620986931025982,
+            -0.0035412474535405636,
+            0.01178737822920084,
+            0.0005731051205657423,
+            -0.017905661836266518,
+            -0.017022771760821342,
+            -0.039249956607818604,
+            0.007702067960053682,
+            0.03531566634774208,
+            -0.0009564659558236599,
+            -0.03444826602935791,
+            0.010145509615540504,
+            -0.011942271143198013,
+            -0.009975126944482327,
+            -0.022862248122692108,
+            0.013862947933375835,
+            0.012430185452103615,
+            0.007415515370666981,
+            0.012252057902514935,
+            -0.006997303571552038,
+            0.01026167906820774,
+            -0.005216030869632959,
+            0.008356492035090923,
+            -0.0010823167394846678,
+            -0.037422213703393936,
+            0.012344993650913239,
+            0.02216522768139839,
+            -0.002635121811181307,
+            0.0016089539276435971,
+            -0.006145390681922436,
+            0.022103270515799522,
+            -0.013971373438835144,
+            -0.00650551775470376,
+            -0.00046443779137916863,
+            0.00797700323164463,
+            -0.01795213110744953,
+            0.004778457339853048,
+            0.0755879208445549,
+            -0.007775642443448305,
+            0.006122156511992216,
+            0.009270362555980682,
+            -0.015148562379181385,
+            -0.01637221872806549,
+            -0.004561606794595718,
+            -0.020260039716959,
+            0.008844406343996525,
+            -0.0018819533288478851,
+            -0.0110981035977602,
+            0.012468908913433552,
+            0.0053050946444272995,
+            0.002031038049608469,
+            -0.008379725739359856,
+            0.010811550542712212,
+            0.009525936096906662,
+            0.010432061739265919,
+            0.02360573597252369,
+            0.011477591469883919,
+            -0.00858883187174797,
+            -0.011795123107731342,
+            0.01129171997308731,
+            0.0028577810153365135,
+            0.016697494313120842,
+            0.007907302118837833,
+            0.0010648912284523249,
+            0.019501063972711563,
+            -0.002309845993295312,
+            0.02180897258222103,
+            -2.2689444449497387e-05,
+            -0.009301341138780117,
+            -0.004236331209540367,
+            -0.015349923633038998,
+            0.024271776899695396,
+            -0.0020155487582087517,
+            0.0015450604259967804,
+            -0.020213572308421135,
+            0.0013979118084535003,
+            0.013243375346064568,
+            0.011957760900259018,
+            -0.0020407189149409533,
+            -0.0051347119733691216,
+            0.0007154133054427803,
+            -0.016356728971004486,
+            0.032806396484375,
+            0.007578153628855944,
+            0.0007681738352403045,
+            0.0008838597568683326,
+            0.0009288755827583373,
+            -0.0032972905319184065,
+            0.0024608669336885214,
+            -0.004766840487718582,
+            0.009549170732498169,
+            -0.010943209752440453,
+            -0.010796061716973782,
+            0.010075807571411133,
+            -0.02402394637465477,
+            0.003566417610272765,
+            -0.00558777479454875,
+            -0.008062195032835007,
+            0.0076207490637898445,
+            0.00015537731815129519,
+            -0.007392281666398048,
+            -0.0030939930584281683,
+            0.0023408245760947466,
+            0.013429246842861176,
+            0.004406713880598545,
+            0.010850274004042149,
+            0.012956822291016579,
+            0.005359307397156954,
+            -0.006629432085901499,
+            0.015512561425566673,
+            -0.006946963258087635,
+            -0.004228586331009865,
+            -0.0012468908680602908,
+            -0.019996721297502518,
+            -0.0009419447160325944,
+            -0.004708755761384964,
+            0.0015092412941157818,
+            0.0001705036120256409,
+            -8.749050175538287e-05,
+            -0.0018490385264158249,
+            -0.011988739483058453,
+            -0.001831613015383482,
+            -0.011663463898003101,
+            -0.014498010277748108,
+            0.010253935120999813,
+            -0.0041201612912118435,
+            -0.022939693182706833,
+            -0.015349923633038998,
+            -0.020662762224674225,
+            0.02145271934568882,
+            -0.004387352149933577,
+            -0.005800753366202116,
+            0.006733985152095556,
+            -0.010215211659669876,
+            -0.015706177800893784,
+            -0.008069939911365509,
+            -0.0205853171646595,
+            0.010703125037252903,
+            0.003214035416021943,
+            0.0006791101768612862,
+            0.00882891658693552,
+            0.014397329650819302,
+            -0.01589204929769039,
+            0.0111755495890975,
+            0.012182355858385563,
+            0.011361422017216682,
+            -0.006854027509689331,
+            0.0008911203476600349,
+            -0.014103032648563385,
+            0.02596011385321617,
+            0.021623101085424423,
+            0.005169562995433807,
+            -0.04132552444934845,
+            -0.005862710531800985,
+            -0.013344055972993374,
+            -0.010524998418986797,
+            -0.012120398692786694,
+            0.011771888472139835,
+            -0.0038820127956569195,
+            0.0013117524795234203,
+            0.006288666743785143,
+            0.0015063370810821652,
+            0.004367990419268608,
+            -0.0022382077295333147,
+            -0.007709812838584185,
+            -0.03212486580014229,
+            -0.02817508764564991,
+            -0.00863530021160841,
+            0.020461400970816612,
+            0.002838419284671545,
+            -0.0009453330421820283,
+            -0.0036380558740347624,
+            0.0027048238553106785,
+            -0.005882072262465954,
+            -0.005018542055040598,
+            -0.014753584749996662,
+            -0.012739972211420536,
+            0.0101300198584795,
+            0.00802347157150507,
+            0.014195968396961689,
+            -0.011950016021728516,
+            0.0028190575540065765,
+            0.018788553774356842,
+            0.007690451107919216,
+            -0.023357905447483063,
+            0.01135367713868618,
+            -0.010912231169641018,
+            -0.009270362555980682,
+            -0.009433000348508358,
+            -0.009471723809838295,
+            0.0016980175860226154,
+            -0.008317768573760986,
+            0.003945906180888414,
+            -0.03615209087729454,
+            -0.011330443434417248,
+            -0.0013901671627536416,
+            -0.010346870869398117,
+            0.0009864765452221036,
+            0.02524760365486145,
+            0.006815304048359394,
+            0.01432762760668993,
+            -0.0006733016925863922,
+            0.017502939328551292,
+            -0.0018984107300639153,
+            0.018726596608757973,
+            0.012763205915689468,
+            -0.00797700323164463,
+            0.011717676185071468,
+            0.014738094992935658,
+            -0.0033863543067127466,
+            0.015620986931025982,
+            -0.010168743319809437,
+            0.005285732913762331,
+            -0.009556914679706097,
+            -0.003214035416021943,
+            0.002495717955753207,
+            0.030901208519935608,
+            -0.003618694143369794,
+            0.00858883187174797,
+            0.01098193321377039,
+            -0.015241498127579689,
+            0.0026951429899781942,
+            0.014962689951062202,
+            0.01141563430428505,
+            0.007760153152048588,
+            0.0024434414226561785,
+            -0.024225307628512383,
+            0.004193735308945179,
+            0.005037903785705566,
+            0.0031520782504230738,
+            -0.014366351068019867,
+            0.0021859314292669296,
+            0.015675200149416924,
+            0.008550108410418034,
+            -0.025325050577521324,
+            0.016418687999248505,
+            0.0009138703462667763,
+            -0.003909118939191103,
+            0.029119934886693954,
+            -0.011268485337495804,
+            -0.005723306443542242,
+            0.009463978931307793,
+            0.0009980935137718916,
+            0.0030029932968318462,
+            0.008937342092394829,
+            0.0228312686085701,
+            -0.003988501615822315,
+            -0.032341714948415756,
+            -0.022459525614976883,
+            0.015706177800893784,
+            -0.0047823297791182995,
+            -0.022939693182706833,
+            -0.026703599840402603,
+            -0.003113354789093137,
+            -0.02228914201259613,
+            -0.0022323993034660816,
+            -0.020058678463101387,
+            -0.014056564308702946,
+            0.001268188701942563,
+            0.029987338930368423,
+            0.016201836988329887,
+            0.008224832825362682,
+            -0.011051635257899761,
+            -0.029956359416246414,
+            -0.024271776899695396,
+            0.01810702309012413,
+            -0.007055388763546944,
+            -0.005541306920349598,
+            0.00591692328453064,
+            -0.009951893240213394,
+            -0.008480406366288662,
+            0.011802867986261845,
+            0.027137301862239838,
+            -0.002747419523075223,
+            -0.029987338930368423,
+            -0.02688947319984436,
+            -0.014335372485220432,
+            -0.01840132102370262,
+            -0.017704300582408905,
+            -0.008627555333077908,
+            0.0045035220682621,
+            0.00030300996149890125,
+            0.016186347231268883,
+            0.014962689951062202,
+            -0.000480653194244951,
+            0.0028597170021384954,
+            -2.7711375878425315e-05,
+            -0.001763847190886736,
+            0.006888878531754017,
+            -0.004933350719511509,
+            0.026997897773981094,
+            0.014614180661737919,
+            -0.006594581063836813,
+            0.018711106851696968,
+            0.028314489871263504,
+            0.020430423319339752,
+            0.003157886676490307,
+            -0.010695381090044975,
+            -0.014606435783207417,
+            0.009727297350764275,
+            -0.013421501964330673,
+            0.023264970630407333,
+            0.022552460432052612,
+            -0.01480779703706503,
+            -0.014885243959724903,
+            -0.017905661836266518,
+            0.015210519544780254,
+            -0.007562664337456226,
+            -0.008224832825362682,
+            0.012368228286504745,
+            0.022335609421133995,
+            -0.007024410180747509,
+            0.002371803391724825,
+            -0.0010232636705040932,
+            -0.021731525659561157,
+            -0.006122156511992216,
+            -0.004596457816660404,
+            -0.008666278794407845,
+            0.008681767620146275,
+            0.008131897076964378,
+            -0.001928421319462359,
+            -0.006319645792245865,
+            0.005471604876220226,
+            -0.013367289677262306,
+            0.01301103550940752,
+            -0.00789568480104208,
+            0.03379771113395691,
+            -0.009138703346252441,
+            -0.0006733016925863922,
+            -0.022598929703235626,
+            -0.011446612887084484,
+            0.00845717266201973,
+            0.0017967619933187962,
+            0.01682141050696373,
+            -0.0011626676423475146,
+            -0.005688455421477556,
+            0.009913169778883457,
+            -0.0010687635513022542,
+            -0.001450188341550529,
+            -0.006892750971019268,
+            -0.010401083156466484,
+            0.0002884887217078358,
+            0.013258864171802998,
+            -0.006025348324328661,
+            0.0018054747488349676,
+            0.006095050368458033,
+            -0.0175339188426733,
+            0.029027000069618225,
+            0.023884542286396027,
+            0.017255110666155815,
+            0.014660648070275784,
+            0.0006394187803380191,
+            0.011647974140942097,
+            -0.003572226269170642,
+            0.0038820127956569195,
+            -0.020337486639618874,
+            0.021080974489450455,
+            -0.010811550542712212,
+            0.01846327818930149,
+            0.011655719019472599,
+            -0.015233753249049187,
+            -0.005022414494305849,
+            0.021406250074505806,
+            0.009967382065951824,
+            0.00889087375253439,
+            0.0017541663255542517,
+            -0.02470547705888748,
+            0.017766257748007774,
+            0.010099041275680065,
+            -0.020399443805217743,
+            0.019547531381249428,
+            -0.009471723809838295,
+            0.01233724970370531,
+            -0.0005300253978930414,
+            0.004418330732733011,
+            0.0058743273839354515,
+            0.021499186754226685,
+            0.011570527218282223,
+            -0.03884723410010338,
+            -0.013615119270980358,
+            -0.0017357728211209178,
+            0.006857899948954582,
+            -0.0036303112283349037,
+            -0.015233753249049187,
+            -0.041635312139987946,
+            0.01604694314301014,
+            0.009998360648751259,
+            -0.01643417589366436,
+            0.004987563472241163,
+            -0.002427952131256461,
+            -0.0042053526267409325,
+            -0.008170620538294315,
+            0.003186929039657116,
+            -0.004170501604676247,
+            0.02324948087334633,
+            0.0018984107300639153,
+            -0.006946963258087635,
+            0.013258864171802998,
+            0.01344473659992218,
+            -0.021948376670479774,
+            0.0009874446550384164,
+            0.0021375271026045084,
+            0.000636998622212559,
+            0.023683181032538414,
+            -0.014707116410136223,
+            -0.0017657832941040397,
+            -0.01685238815844059,
+            -0.0013069120468571782,
+            -0.021592123433947563,
+            -0.001188805908896029,
+            -0.015187285840511322,
+            -0.008511385880410671,
+            0.025030752643942833,
+            -0.011005166918039322,
+            0.010997422970831394,
+            -0.006687517277896404,
+            0.006943091284483671,
+            -0.006904367823153734,
+            -0.01936165988445282,
+            -0.0008267428493127227,
+            0.009975126944482327,
+            -0.000512115890160203,
+            0.002941036131232977,
+            8.482827252009884e-05,
+            0.002464739140123129,
+            0.008867640048265457,
+            0.006234454456716776,
+            -0.0075278133153915405,
+            0.02489134855568409,
+            -0.012073930352926254,
+            0.0016399325104430318,
+            0.011911292560398579,
+            0.0004743606550619006,
+            -0.010424317792057991,
+            0.025588368996977806,
+            -0.02139076218008995,
+            -0.00800798274576664,
+            -0.00014799568452872336,
+            -0.0016544538084417582,
+            -0.006513262167572975,
+            0.006718495860695839,
+            0.02946070022881031,
+            0.0062654330395162106,
+            0.010447551496326923,
+            -0.0281441081315279,
+            -0.011493081226944923,
+            0.012538610957562923,
+            -0.022273652255535126,
+            -0.0007739823195151985,
+            -0.0004322490422055125,
+            -0.00619960343465209,
+            0.026378324255347252,
+            -0.02022906206548214,
+            -0.01093546487390995,
+            -0.018664639443159103,
+            0.005405775271356106,
+            -0.011121337302029133,
+            0.008294534869492054,
+            -0.0009777637897059321,
+            0.006540368776768446,
+            0.016805920749902725,
+            -0.019563021138310432,
+            0.016031453385949135,
+            0.005119222681969404,
+            -0.016217326745390892,
+            0.010222956538200378,
+            -0.017580386251211166,
+            -0.0008199662552215159,
+            0.020275529474020004,
+            -0.00400399137288332,
+            0.008751469664275646,
+            -0.003363120136782527,
+            0.005208286456763744,
+            -0.006513262167572975,
+            0.0001492057926952839,
+            -0.007249005138874054,
+            0.014428308233618736,
+            -0.016805920749902725,
+            -0.017828216776251793,
+            -0.008821171708405018,
+            0.0016709112096577883,
+            0.021297825500369072,
+            0.011136826127767563,
+            -0.010044828988611698,
+            -0.011059380136430264,
+            -0.011694442480802536,
+            0.041604332625865936,
+            0.005762029904872179,
+            0.027230238541960716,
+            -0.02261441759765148,
+            0.00545998802408576,
+            0.025170156732201576,
+            -0.014242436736822128,
+            0.0012788375606760383,
+            -0.0022788674104958773,
+            0.0053050946444272995,
+            0.011562783271074295,
+            -0.01819995976984501,
+            0.0049139889888465405,
+            -0.00019216447253711522,
+            -0.00782985519617796,
+            -0.000656844291370362,
+            0.003442503046244383,
+            0.00020196630794089288,
+            0.016325751319527626,
+            -0.004027225077152252,
+            -0.016511622816324234,
+            -0.004429947584867477,
+            -0.0003216455806978047,
+            -0.027896279469132423,
+            0.013514438644051552,
+            0.016589069738984108,
+            0.01566745527088642,
+            0.027989214286208153,
+            -0.014513500034809113,
+            0.016031453385949135,
+            0.009533680975437164,
+            0.013901671394705772,
+            -0.010610189288854599,
+            0.008952830918133259,
+            -0.01595400646328926,
+            0.007446494419127703,
+            -0.03454120084643364,
+            -0.007887939922511578,
+            -0.00802347157150507,
+            0.01984182931482792,
+            0.028763681650161743,
+            0.002923610620200634,
+            0.006493900436908007,
+            0.000196883876924403,
+            0.0024202074855566025,
+            -0.01714668609201908,
+            -0.020182594656944275,
+            -0.0003337466041557491,
+            0.0029332914855331182,
+            0.0006950835813768208,
+            -0.004495777189731598,
+            0.00472037261351943,
+            -0.012391461990773678,
+            0.0026835259050130844,
+            -0.013134949840605259,
+            0.005971135571599007,
+            -0.0059982421807944775,
+            0.009603383019566536,
+            0.0019429425010457635,
+            0.020864123478531837,
+            0.02300165221095085,
+            0.004100799560546875,
+            0.0046971384435892105,
+            -0.024333734065294266,
+            0.00920066051185131,
+            -0.013847459107637405,
+            0.012329504825174809,
+            0.019408127292990685,
+            0.008248066529631615,
+            0.006571347359567881,
+            0.01233724970370531,
+            0.012918098829686642,
+            -0.0031772484071552753,
+            0.024287264794111252,
+            0.005014669615775347,
+            -0.0009070937521755695,
+            0.002052335999906063,
+            -0.014954946003854275,
+            -0.0070979841984808445,
+            -0.0015286030247807503,
+            0.0018848575418815017,
+            -0.004731989465653896,
+            0.01906736195087433,
+            0.015256987884640694,
+            -0.01050176378339529,
+            0.017874684184789658,
+            -0.03937387093901634,
+            0.017379024997353554,
+            -0.0027377386577427387,
+            -0.0060911779291927814,
+            -0.00545998802408576,
+            0.00665653869509697,
+            0.004515138920396566,
+            0.003669034456834197,
+            0.03773200139403343,
+            -0.010408828034996986,
+            0.003963331691920757,
+            -0.005847221240401268,
+            0.007121218368411064,
+            -0.020864123478531837,
+            0.008302279748022556,
+            0.005336073227226734,
+            0.02160761132836342,
+            -0.0056497324258089066,
+            -0.0014482521219179034,
+            0.02328045852482319,
+            0.008859895169734955,
+            0.03469609469175339,
+            -0.01566745527088642,
+            -0.004418330732733011,
+            0.01048627495765686,
+            0.0207402091473341,
+            -0.00580462533980608,
+            -0.008976065553724766,
+            0.007903429679572582,
+            0.0035586729645729065,
+            0.021824462339282036,
+            -0.00515407370403409,
+            -0.0006650729919783771,
+            0.0054367538541555405,
+            -0.0060911779291927814,
+            0.009735042229294777,
+            0.011252996511757374,
+            -0.009076746180653572,
+            0.007539430167526007,
+            0.021530164405703545,
+            -0.009975126944482327,
+            -0.008658533915877342,
+            -0.009649851359426975,
+            0.0010658593382686377,
+            -0.03068435750901699,
+            0.001649613375775516,
+            0.005359307397156954,
+            0.015187285840511322,
+            -0.0004603234410751611,
+            -0.009014789015054703,
+            0.021019017323851585,
+            0.017255110666155815,
+            -0.005936284549534321,
+            -0.0028132491279393435,
+            0.0023795480374246836,
+            -0.0017154429806396365,
+            -0.006160879973322153,
+            -0.0006248975405469537,
+            -0.002656419761478901,
+            0.014730350114405155,
+            -0.0015469965292140841,
+            -0.006617815233767033,
+            -0.00756653631106019,
+            0.0033302053343504667,
+            0.010834784246981144,
+            -0.005661349277943373,
+            0.013057502917945385,
+            0.016805920749902725,
+            -0.00031704717548564076,
+            0.019671445712447166,
+            0.01037010457366705,
+            -0.010447551496326923,
+            0.015938518568873405,
+            -0.007984748110175133,
+            0.006563602481037378,
+            0.0164031982421875,
+            -0.01277095079421997,
+            -0.01246116403490305,
+            0.007210282143205404,
+            0.00665653869509697,
+            0.028918573632836342,
+            0.014792308211326599,
+            -0.02109646424651146,
+            0.014908477663993835,
+            0.015768134966492653,
+            -0.013018779456615448,
+            0.003537375247105956,
+            -0.01784370467066765,
+            0.020260039716959,
+            0.019051872193813324,
+            -0.015535795129835606,
+            -0.017224133014678955,
+            -0.014947201125323772,
+            -0.005092116538435221,
+            0.006331262644380331,
+            0.0033263331279158592,
+            -0.00617636926472187,
+            -0.027989214286208153,
+            -0.0049139889888465405,
+            -0.018633661791682243,
+            0.013870692811906338,
+            -0.008674023672938347,
+            0.0007715621031820774,
+            -0.004511266481131315,
+            0.007984748110175133,
+            -0.0033882902935147285,
+            0.032465629279613495,
+            -0.0051811798475682735,
+            -0.02832997962832451,
+            0.004398969002068043,
+            0.016356728971004486,
+            0.012623801827430725,
+            -0.029243851080536842,
+            0.010463040322065353,
+            0.004805563949048519,
+            -0.012530866079032421,
+            0.005990497302263975,
+            -0.02258343994617462,
+            -0.007632365915924311,
+            -0.0009395244996994734,
+            -0.016743963584303856,
+            -0.027121812105178833,
+            -0.008433938957750797,
+            -0.006993431597948074,
+            -0.005401902832090855,
+            -0.009525936096906662,
+            -0.008720491081476212,
+            -0.011477591469883919,
+            0.02303262986242771,
+            -0.007899557240307331,
+            -0.001866463921032846,
+            -0.0014637414133176208,
+            0.00930908601731062,
+            -0.023776117712259293,
+            -0.015605497173964977,
+            0.006389347370713949,
+            -0.002594462363049388,
+            0.022815778851509094,
+            -0.012747716158628464,
+            0.005940156988799572,
+            -0.009316830895841122,
+            0.009851212613284588,
+            0.015241498127579689,
+            0.004956584889441729,
+            0.017440982162952423,
+            -0.009889936074614525,
+            0.0021026763133704662,
+            -0.008627555333077908,
+            -0.00018986526993103325,
+            -1.027075541060185e-05,
+            0.014428308233618736,
+            0.007624621503055096,
+            0.014141756109893322,
+            0.017487451434135437,
+            -0.013553161174058914,
+            0.005003052763640881,
+            -0.0003274540649726987,
+            -0.017255110666155815,
+            0.0027725896798074245,
+            -0.007984748110175133,
+            -0.0008896682411432266,
+            0.001758038648404181,
+            6.395397940650582e-05,
+            -0.02112744189798832,
+            -0.0009124181815423071,
+            0.0018809852190315723,
+            0.0010222955606877804,
+            -0.025201136246323586,
+            0.00889087375253439,
+            -0.00889087375253439,
+            -0.0014918158994987607,
+            -0.019113829359412193,
+            0.019795360043644905,
+            0.026300877332687378,
+            -0.004619691986590624,
+            0.00036206303047947586,
+            -0.013537672348320484,
+            -0.02049238048493862,
+            0.0001489637652412057,
+            -0.011314953677356243,
+            -0.011345932260155678,
+            -0.0022169100120663643,
+            -0.003214035416021943,
+            0.008805682882666588,
+            0.0004189379105810076,
+            -0.009433000348508358,
+            -0.004956584889441729,
+            -0.011764144524931908,
+            -0.017549408599734306,
+            -0.010401083156466484,
+            0.01933068037033081,
+            -0.0018016024259850383,
+            0.0020794421434402466,
+            -0.0032508226577192545,
+            0.0005271211848594248,
+            0.011748654767870903,
+            0.013475715182721615,
+            0.013344055972993374,
+            -0.023915521800518036,
+            -0.017890173941850662,
+            -0.013320821337401867,
+            0.0029487807769328356,
+            -0.007187047973275185,
+            0.004844287410378456,
+            0.026362836360931396,
+            -0.0009695350890979171,
+            -0.01091997604817152,
+            0.002327271504327655,
+            0.019051872193813324,
+            -0.031272951513528824,
+            -0.013390523381531239,
+            -0.025077221915125847,
+            0.02225816436111927,
+            -0.013351799920201302,
+            0.011516314931213856,
+            0.010191977955400944,
+            -0.025944624096155167,
+            0.006768836174160242,
+            -0.007686578668653965,
+            0.026037558913230896,
+            0.01634124107658863,
+            0.0005934348446317017,
+            -0.00944074522703886,
+            0.013622863218188286,
+            -0.0008722427301108837,
+            -0.013576395809650421,
+            0.002201420720666647,
+            0.014645159244537354,
+            -0.014691627584397793,
+            -0.009510447271168232,
+            -0.004507394507527351,
+            0.004313777666538954,
+            0.012763205915689468,
+            -0.0030339721124619246,
+            -0.004031097516417503,
+            -0.011624740436673164,
+            0.013344055972993374,
+            -0.003930416889488697,
+            0.012383717112243176,
+            0.007380664348602295,
+            -0.014056564308702946,
+            -0.019532041624188423,
+            0.033333033323287964,
+            0.00400399137288332,
+            0.00338054564781487,
+            -0.010997422970831394,
+            0.02740062028169632,
+            -0.009897680021822453,
+            0.010571465827524662,
+            -0.006408709101378918,
+            0.0031985461246222258,
+            0.012321759946644306,
+            -0.012104909867048264,
+            -0.01981084980070591,
+            0.021468207240104675,
+            -0.00364192808046937,
+            -0.0010193913476541638,
+            0.008310024626553059,
+            -0.020182594656944275,
+            -0.00663330452516675,
+            0.002180122770369053,
+            0.009541425853967667,
+            -0.011818356812000275,
+            0.006369986105710268,
+            0.017905661836266518,
+            0.003912991378456354,
+            -0.0007672057254239917,
+            0.019996721297502518,
+            -0.01672847382724285,
+            0.011717676185071468,
+            -0.010850274004042149,
+            0.0016825281782075763,
+            0.0006065039779059589,
+            -0.0048597767017781734,
+            0.0058743273839354515,
+            -0.014536733739078045,
+            0.011988739483058453,
+            -6.244135147426277e-05,
+            -0.013979118317365646,
+            0.007601387333124876,
+            -0.013359544798731804,
+            -0.016836898401379585,
+            0.041542377322912216,
+            -0.005301222205162048,
+            -0.0038665232714265585,
+            -0.0056962003000080585,
+            0.01350669376552105,
+            0.02823704481124878,
+            -0.005336073227226734,
+            -0.01153180468827486,
+            -0.010780571959912777,
+            -0.016573579981923103,
+            0.015907539054751396,
+            -0.01810702309012413,
+            0.002495717955753207,
+            -0.010532742366194725,
+            -0.007903429679572582,
+            0.004302160814404488,
+            -0.003516077296808362,
+            0.014722606167197227,
+            0.00950270239263773,
+            0.023357905447483063,
+            0.00797700323164463,
+            -0.00027348342700861394,
+            0.010393339209258556,
+            0.011663463898003101,
+            0.003243078012019396,
+            -0.010509508661925793,
+            -0.00011949774489039555,
+            -0.02894955314695835,
+            -0.008449427783489227,
+            -0.004863648675382137,
+            -0.001485039247199893,
+            -0.00428667152300477,
+            -0.01239920686930418,
+            0.006617815233767033,
+            -0.0022052929271012545,
+            0.013909416273236275,
+            -0.009254872798919678,
+            -0.017766257748007774,
+            0.03389064967632294,
+            0.004495777189731598,
+            0.006466794293373823,
+            -0.008286789990961552,
+            -0.03705047070980072,
+            -0.0032566310837864876,
+            -0.008116407319903374,
+            -0.013529927469789982,
+            0.004538373090326786,
+            0.01283290795981884,
+            -0.014831030741333961,
+            -0.028097640722990036,
+            0.0019390701781958342,
+            -0.006683644838631153,
+            0.02978597767651081,
+            -0.007798876613378525,
+            0.00419760774821043,
+            0.0037251831963658333,
+            0.0011558911064639688,
+            0.01239920686930418,
+            -0.00845717266201973,
+            -0.0022536972537636757,
+            0.016387708485126495,
+            -0.010858018882572651,
+            0.011051635257899761,
+            -0.010966443456709385,
+            -0.02554190158843994,
+            -0.01456771232187748,
+            0.021344292908906937,
+            -0.0004823473282158375,
+            -0.00024903935263864696,
+            0.014823286794126034,
+            0.02354377694427967,
+            0.011260741390287876,
+            0.00969631876796484,
+            0.006408709101378918,
+            -0.001035848748870194,
+            0.005975008010864258,
+            0.006095050368458033,
+            -0.0024163350462913513,
+            -0.013398268260061741,
+            0.017564896494150162,
+            0.009773765690624714,
+            -0.01810702309012413,
+            0.0008514289511367679,
+            0.0036612898111343384,
+            0.0168833676725626,
+            0.006935346405953169,
+            0.009316830895841122,
+            -0.0143121387809515,
+            -0.023760627955198288,
+            -0.00014932679187040776,
+            0.002873270306736231,
+            0.015721667557954788,
+            0.010579210706055164,
+            -0.014319883659482002,
+            0.0005963391158729792,
+            0.008712747134268284,
+            0.020213572308421135,
+            -0.024209819734096527,
+            0.019702425226569176,
+            -0.004673904739320278,
+            -0.0021084847394376993,
+            0.007946024648845196,
+            0.011632485315203667,
+            -0.011872569099068642,
+            -0.00650551775470376,
+            0.006249943748116493,
+            0.003157886676490307,
+            0.03172214329242706,
+            -0.0017260919557884336,
+            0.009665340185165405,
+            -0.010997422970831394,
+            -0.0002405686245765537,
+            0.0006728176376782358,
+            -0.0006820144481025636,
+            -0.002884887158870697,
+            -0.011020656675100327,
+            0.0018974426202476025,
+            -0.020306508988142014,
+            0.0024473138619214296,
+            0.00463130883872509,
+            -0.005684582982212305,
+            -0.0032972905319184065,
+            -0.019222255796194077,
+            -0.0022672503255307674,
+            0.003293418325483799,
+            0.014335372485220432,
+            -0.031691163778305054,
+            0.004972074180841446,
+            0.010672146454453468,
+            -0.002309845993295312,
+            -0.010416572913527489,
+            0.011423379182815552,
+            -0.0026060794480144978,
+            0.011988739483058453,
+            -0.0038142469711601734,
+            -0.0004905760288238525,
+            0.025681305676698685,
+            -0.005308967083692551,
+            0.014707116410136223,
+            -0.01026167906820774,
+            -0.002993312431499362,
+            -0.028252532705664635,
+            0.005235392600297928,
+            -0.007403898518532515,
+            -0.0012081675231456757,
+            -0.004402841441333294,
+            -0.01130720879882574,
+            -0.0026990154292434454,
+            0.002122037811204791,
+            0.008000237867236137,
+            0.012910354882478714,
+            -0.0007725302129983902,
+            -0.010075807571411133,
+            0.004832670092582703,
+            0.0075355577282607555,
+            -0.015520306304097176,
+            -0.0070437719114124775,
+            -0.020306508988142014,
+            0.029662061482667923,
+            -0.013731288723647594,
+            -0.0011123273288831115,
+            -0.020864123478531837,
+            0.010160998441278934,
+            -0.0063119009137153625,
+            -0.002782270545139909,
+            0.004708755761384964,
+            -0.015109838917851448,
+            0.01344473659992218,
+            0.008751469664275646,
+            0.01450575515627861,
+            -0.02151467651128769,
+            -0.01762685552239418,
+            -0.0044725434854626656,
+            -0.002735802438110113,
+            -0.010648912750184536,
+            -0.0005334137240424752,
+            0.002013612538576126,
+            0.021251358091831207,
+            -0.00032503384863957763,
+            0.0042053526267409325,
+            -0.003214035416021943,
+            0.017224133014678955,
+            -0.010137764737010002,
+            -0.013057502917945385,
+            0.0024260159116238356,
+            0.005994369741529226,
+            0.016186347231268883,
+            0.03158273920416832,
+            -0.020693741738796234,
+            -0.00272999401204288,
+            0.0032740565948188305,
+            0.010656657628715038,
+            -0.004085310269147158,
+            6.304640555754304e-05,
+            -0.027803342789411545,
+            -0.04464024305343628,
+            -0.002422143705189228,
+            0.013057502917945385,
+            -0.015636475756764412,
+            0.012670270167291164,
+            0.008743725717067719,
+            0.010021595284342766,
+            -0.005196669604629278,
+            0.02620794251561165,
+            -0.004793947096914053,
+            0.014304393902420998,
+            0.011733165942132473,
+            0.011911292560398579,
+            0.0009874446550384164,
+            0.0007459079497493804,
+            -0.018354853615164757,
+            -0.008255811408162117,
+            -0.0015954007394611835,
+            0.01765783317387104,
+            -0.027478067204356194,
+            0.008526874706149101,
+            -0.011516314931213856,
+            0.013839714229106903,
+            0.0003666614356916398,
+            0.005661349277943373,
+            -0.01771979033946991,
+            0.012445674277842045,
+            -0.0009337160154245794,
+            -0.025325050577521324,
+            0.004805563949048519,
+            0.011276230216026306,
+            -0.010362359695136547,
+            0.006877261213958263,
+            0.019950253888964653,
+            0.0023079097736626863,
+            0.008495896123349667,
+            -0.014126266352832317,
+            0.010703125037252903,
+            0.009649851359426975,
+            -0.021080974489450455,
+            -0.01233724970370531,
+            -0.007059261202812195,
+            -0.006335135083645582,
+            -0.0035470561124384403,
+            -0.00013105422840453684,
+            0.010176488198339939,
+            -0.016000475734472275,
+            0.010099041275680065,
+            -0.006559730041772127,
+            0.006400964688509703,
+            0.00944074522703886,
+            0.001115231541916728,
+            -0.013576395809650421,
+            0.012716737575829029,
+            -0.0218709297478199,
+            0.008426194079220295,
+            -0.04287445917725563,
+            -0.005847221240401268,
+            -0.015078860335052013,
+            0.005204414017498493,
+            -0.003798757679760456,
+            -0.004453181754797697,
+            -0.006203475873917341,
+            -0.005061137955635786,
+            0.00920066051185131,
+            0.021747015416622162,
+            0.012135888449847698,
+            -0.016620049253106117,
+            -0.007551047019660473,
+            0.005425137002021074,
+            0.0004854935978073627,
+            -0.013336311094462872,
+            0.009316830895841122,
+            -0.01576039008796215,
+            0.0009259713697247207,
+            0.006339007057249546,
+            -0.01072635967284441,
+            0.013452480547130108,
+            0.004731989465653896,
+            -0.003401843598112464,
+            0.013390523381531239,
+            -0.0050417762249708176,
+            -0.0057891360484063625,
+            -0.007272239308804274,
+            -0.013739033602178097,
+            -0.003818119177594781,
+            -0.003072695340961218,
+            0.012747716158628464,
+            0.001529571134597063,
+            -0.0003243077953811735,
+            -0.00782985519617796,
+            0.009154192171990871,
+            0.002220782218500972,
+            -0.0013146566925570369,
+            0.005773646757006645,
+            -0.02300165221095085,
+            0.0001287550403503701,
+            0.004755223635584116,
+            -0.0009487213101238012,
+            -0.006246071308851242,
+            -0.012151377275586128,
+            -0.011903548613190651,
+            -0.038165703415870667,
+            0.009363298304378986,
+            0.0019197085639461875,
+            -0.0076207490637898445,
+            0.0064822835847735405,
+            0.017255110666155815,
+            0.01456771232187748,
+            0.016743963584303856,
+            -0.020693741738796234,
+            -0.02303262986242771,
+            0.005556796211749315,
+            -0.009402021765708923,
+            0.02261441759765148,
+            0.004046586807817221,
+            -0.005847221240401268,
+            0.0013243375578895211,
+            -0.013258864171802998,
+            0.008929597213864326,
+            0.005723306443542242,
+            -0.001592496526427567,
+            0.005425137002021074,
+            -0.013692565262317657,
+            -0.010904486291110516,
+            0.01048627495765686,
+            0.032496608793735504,
+            -0.0004261985304765403,
+            0.0024105266202241182,
+            0.023295948281884193,
+            0.015466093085706234,
+            0.016356728971004486,
+            -0.021111954003572464,
+            0.011167804710566998,
+            0.010060317814350128,
+            -0.0018645278178155422,
+            -0.014892988838255405,
+            0.00513083953410387,
+            -0.012716737575829029,
+            -0.010664401575922966,
+            0.014993669465184212,
+            0.007853088900446892,
+            0.017735280096530914,
+            -0.02369867078959942,
+            0.00789568480104208,
+            0.00042765066609717906,
+            0.00013710475468542427,
+            0.011524059809744358,
+            0.012956822291016579,
+            0.03370477631688118,
+            0.027152791619300842,
+            0.020662762224674225,
+            -0.009285852313041687,
+            0.014366351068019867,
+            -0.009192915633320808,
+            -0.011090358719229698,
+            -0.019470084458589554,
+            -0.015249243006110191,
+            -0.011136826127767563,
+            0.017998598515987396,
+            0.008790193125605583,
+            -0.009355553425848484,
+            -0.005947901867330074,
+            0.01586107164621353,
+            -0.003909118939191103,
+            -0.004635181277990341,
+            0.01646515540778637,
+            -0.026269899681210518,
+            0.016836898401379585,
+            -0.000745423894841224,
+            -0.014668392948806286,
+            -0.014691627584397793,
+            -0.004182118456810713,
+            -0.012670270167291164,
+            0.0003964299685321748,
+            -0.020817656069993973,
+            0.013181418180465698,
+            0.024597052484750748,
+            -0.0030920570716261864,
+            -0.004100799560546875,
+            -0.011299464851617813,
+            -0.0005111477803438902,
+            0.018385831266641617,
+            0.0034773540683090687,
+            0.009192915633320808,
+            -0.0023872926831245422,
+            0.013924905098974705,
+            0.00524313747882843,
+            0.0027958236169070005,
+            -0.0007565568666905165,
+            -0.016263794153928757,
+            0.012197845615446568,
+            0.0026835259050130844,
+            0.012135888449847698,
+            -0.022242674604058266,
+            0.007938280701637268,
+            -0.011764144524931908,
+            -0.0023698671720921993,
+            -0.02228914201259613,
+            0.0189744271337986,
+            -0.01610890030860901,
+            0.001046497724018991,
+            0.003841353114694357,
+            0.0006224773242138326,
+            0.009580149315297604,
+            0.01610890030860901,
+            -0.023388884961605072,
+            -0.0003460896550677717,
+            -0.014908477663993835,
+            -0.021917399019002914,
+            -0.01607792265713215,
+            0.004728117026388645,
+            -0.019501063972711563,
+            0.005444498732686043,
+            -0.0025557391345500946,
+            -0.015125328674912453,
+            0.018478767946362495,
+            0.014691627584397793,
+            -0.004379607271403074,
+            0.002412462839856744,
+            0.005425137002021074,
+            -0.0014405074762180448,
+            0.01971791312098503,
+            0.00013274837692733854,
+            -0.0010048701660707593,
+            -0.009425255469977856,
+            -0.0168833676725626,
+            -0.0065752193331718445,
+            -0.035005878657102585,
+            0.0260530486702919,
+            -0.004670032300055027,
+            -0.016511622816324234,
+            0.014319883659482002,
+            0.009115469641983509,
+            -0.014629669487476349,
+            -0.007059261202812195,
+            0.016294771805405617,
+            -0.024287264794111252,
+            0.02244403585791588,
+            -0.009611127898097038,
+            -0.008774704299867153,
+            0.00784921646118164,
+            0.05154848098754883,
+            0.009146448224782944,
+            0.003492843359708786,
+            0.011036145500838757,
+            0.005839476361870766,
+            -0.0013165927957743406,
+            -0.023683181032538414,
+            0.005568413063883781,
+            -0.007190920412540436,
+            -0.003601268632337451,
+            0.002536377403885126,
+            -0.004801691509783268,
+            0.031149037182331085,
+            -0.004379607271403074,
+            -0.005289605353027582,
+            0.004484160337597132,
+            -0.02196386642754078,
+            -0.0001332324172835797,
+            0.023203013464808464,
+            -2.3128106477088295e-05,
+            -0.023590246215462685,
+            -0.006571347359567881,
+            -0.015071115456521511,
+            -0.013770012184977531,
+            -0.018989915028214455,
+            0.005231520161032677,
+            -0.012755461037158966,
+            -0.0033940989524126053,
+            0.0016660707769915462,
+            -0.005940156988799572,
+            -0.002582845278084278,
+            0.009603383019566536,
+            0.05362405255436897,
+            0.014103032648563385,
+            -0.006335135083645582,
+            -0.005854965653270483,
+            0.009192915633320808,
+            -0.010524998418986797,
+            -0.019795360043644905,
+            -0.009820234030485153,
+            -0.029507169499993324,
+            -0.012747716158628464,
+            -0.0143121387809515,
+            -0.010191977955400944,
+            -0.010277168825268745,
+            0.00963436160236597,
+            -0.018122512847185135,
+            -0.007628493942320347,
+            -0.020337486639618874,
+            0.002941036131232977,
+            -0.011268485337495804,
+            -0.014714861288666725,
+            -0.008054450154304504,
+            -0.04807887226343155,
+            0.005796880926936865,
+            0.019593998789787292,
+            0.01263929158449173,
+            0.002515079453587532,
+            0.004383479710668325,
+            -0.01993476413190365,
+            -0.030792783945798874,
+            0.013839714229106903,
+            0.011283975094556808,
+            0.011733165942132473,
+            0.03283737599849701,
+            -0.011709931306540966,
+            -0.00813964195549488,
+            -0.004693266469985247,
+            -0.0017251238459721208,
+            0.012066186405718327,
+            0.00969631876796484,
+            -0.004917861428111792,
+            -0.012375972233712673,
+            0.016217326745390892,
+            -0.01981084980070591,
+            -0.023683181032538414,
+            -0.01672847382724285,
+            0.006157007534056902,
+            0.00813964195549488,
+            -0.008054450154304504,
+            0.002973950933665037,
+            0.0056962003000080585,
+            -0.007159941829741001,
+            -0.00022423222253564745,
+            0.004890755284577608,
+            -0.023048119619488716,
+            0.0008490087930113077,
+            -0.002257569460198283,
+            0.0061376458033919334,
+            0.021266845986247063,
+            -0.010168743319809437,
+            -0.0014075926737859845,
+            -0.0053902859799563885,
+            -0.0028984404634684324,
+            -0.006764963734894991,
+            0.02620794251561165,
+            0.009154192171990871,
+            -0.009882191196084023,
+            -0.01011453103274107,
+            -0.01129171997308731,
+            -0.0011278166202828288,
+            -0.017255110666155815,
+            -0.016186347231268883,
+            0.0015566773945465684,
+            -0.016712984070181847,
+            0.013971373438835144,
+            -0.00876695942133665,
+            0.00400399137288332,
+            -0.014110777527093887,
+            0.02019808255136013,
+            0.01929970271885395,
+            -0.009711808525025845,
+            0.023776117712259293,
+            -0.007709812838584185,
+            -0.038382552564144135,
+            -0.00800798274576664,
+            0.01843230053782463,
+            -0.006118284538388252,
+            0.010037084110081196,
+            -0.011314953677356243,
+            0.008209343999624252,
+            0.00015319912927225232,
+            0.0018122512847185135,
+            -0.0003112386621069163,
+            0.027555514127016068,
+            0.015535795129835606,
+            -0.005560668651014566,
+            -0.005494839046150446,
+            0.004255692940205336,
+            0.009541425853967667,
+            0.017580386251211166,
+            0.012151377275586128,
+            -0.021019017323851585,
+            -0.014257925562560558,
+            -0.016263794153928757,
+            0.011012911796569824,
+            0.015125328674912453,
+            -0.0040891822427511215,
+            0.01691434532403946,
+            0.014126266352832317,
+            -0.019098341464996338,
+            0.0019884423818439245,
+            -0.008193854242563248,
+            -0.00158184755127877,
+            -0.03029712475836277,
+            0.011043890379369259,
+            0.0074387495405972,
+            -0.0072218989953398705,
+            0.036616768687963486,
+            0.009858956560492516,
+            0.037577107548713684,
+            0.018680129200220108,
+            0.0008974128868430853,
+            0.009045767597854137,
+            -0.018370341509580612,
+            0.013065247796475887,
+            0.01500141341239214,
+            -0.012724482454359531,
+            -0.005734923295676708,
+            0.0013446672819554806,
+            0.0007551047019660473,
+            0.012631546705961227,
+            0.02004319056868553,
+            -0.01795213110744953,
+            0.009773765690624714,
+            -0.015032391995191574,
+            -0.005250881891697645,
+            0.006555858068168163,
+            -0.013932649977505207,
+            -0.008495896123349667,
+            -0.00963436160236597,
+            0.019408127292990685,
+            -0.012600568123161793,
+            0.016573579981923103,
+            -0.009595638141036034,
+            0.015520306304097176,
+            -0.029724018648266792,
+            -0.003669034456834197,
+            -0.006679772399365902,
+            -0.019593998789787292,
+            -0.012887120246887207,
+            0.009820234030485153,
+            0.010633422993123531,
+            -0.009301341138780117,
+            -0.011562783271074295,
+            0.020368466153740883,
+            -0.004185990896075964,
+            0.013529927469789982,
+            0.021189400926232338,
+            0.00737291993573308,
+            0.024132372811436653,
+            0.0005605200421996415,
+            0.014164989814162254,
+            0.005463860463351011,
+            0.010362359695136547,
+            -0.004677776712924242,
+            0.005161818582564592,
+            0.00932457484304905,
+            0.014203713275492191,
+            0.017766257748007774,
+            0.013196907006204128,
+            -0.00598275288939476,
+            0.007419387809932232,
+            0.02016710489988327,
+            0.0016254113288596272,
+            -0.030312614515423775,
+            -0.009719553403556347,
+            0.0001736498816171661,
+            0.00924712885171175,
+            0.009185170754790306,
+            3.817877222900279e-05,
+            -0.01093546487390995,
+            0.011144571006298065,
+            0.00969631876796484,
+            0.002286612056195736,
+            0.006280922330915928,
+            -0.01456771232187748,
+            -0.020957060158252716,
+            0.006106667220592499,
+            0.01798310875892639,
+            0.013917161151766777,
+            -0.014722606167197227,
+            0.022784801200032234,
+            0.006145390681922436,
+            -0.021406250074505806,
+            -0.01840132102370262,
+            -0.0009559819591231644,
+            0.0004542729293461889,
+            -0.002265314105898142,
+            0.03720536455512047,
+            -0.0011539548868313432,
+            -0.005955646280199289,
+            -0.0054754773154854774,
+            0.02775687538087368,
+            0.02049238048493862,
+            0.009177426807582378
+          ],
+          "embedding": [
+            0.0008215984936325879,
+            -0.019349418284688353,
+            -0.02322660633029795,
+            0.009747820845194852,
+            0.007353771220903641,
+            0.015252908783199839,
+            -0.012176175829980202,
+            0.051898051094966605,
+            -0.0068130353886146165,
+            0.008189212816890595,
+            0.008102287831931905,
+            0.01682801136011614,
+            0.02005557996049711,
+            -0.020779879580497183,
+            0.002332165268329781,
+            0.02337485065616377,
+            0.006074992895540845,
+            0.02576575146535689,
+            0.0007517286429767504,
+            -0.03763909942314134,
+            0.0011204729720314833,
+            0.00041212613589632806,
+            -0.06089176800314795,
+            0.012852035873395388,
+            -0.008901127092058543,
+            0.01702347055484216,
+            0.005803446395466281,
+            0.017889570275475296,
+            -0.05375047382413552,
+            0.021623778774252724,
+            0.014784317465883424,
+            -0.0018881880857885708,
+            0.0004735328554290247,
+            0.012393802850407656,
+            -0.019234442968055952,
+            -0.019566283611638122,
+            0.04407789373296535,
+            -0.0053565873209102995,
+            -0.026240866282480922,
+            0.048107902270426675,
+            -0.0054681314939387235,
+            -0.0009441397948041965,
+            -0.015205079626883548,
+            0.006885981841893229,
+            0.01403724256866746,
+            -0.026086548246792754,
+            0.0034778946051893067,
+            0.00044587533948775473,
+            0.032851328435891634,
+            -0.03882097333255334,
+            -0.011126344604254894,
+            -0.01648943336384904,
+            0.035810456555802166,
+            -0.019384589526165754,
+            -0.019430702768170285,
+            0.014297440272239852,
+            -0.007883453204663151,
+            0.050996900793086844,
+            0.06007149434153715,
+            0.03406832053483379,
+            0.025193439934671852,
+            0.00027785129663604897,
+            -0.008014391056168083,
+            -0.018806730908490114,
+            0.027193788903098558,
+            -0.006195006670328595,
+            -0.020466000344352826,
+            0.0068939884219589185,
+            0.017252276167086602,
+            -0.045073444648704136,
+            -0.049242269917908754,
+            0.023021140866008313,
+            -0.015676872472071106,
+            -0.024497517908898602,
+            0.005873633718973869,
+            -0.022828585938203683,
+            0.011748341004940436,
+            0.04624205727927451,
+            -0.015442165499674641,
+            0.018565551071491767,
+            0.015246505395507096,
+            0.004119269796224144,
+            0.004862735250819252,
+            -0.000145323321960046,
+            0.05155838952887077,
+            0.0027052423345323936,
+            -0.016218063670586664,
+            -0.07462470920944148,
+            0.043695974746295135,
+            -0.027608273720377708,
+            -0.03860912774695494,
+            0.002823110565150435,
+            0.026471469849508766,
+            -0.0026063577189074103,
+            -0.029230652014151735,
+            0.05684164470477843,
+            0.00022182868300572677,
+            0.015590845552956364,
+            -0.024939617549529242,
+            0.01035803286203917,
+            -0.010225298012384327,
+            0.019571847648759157,
+            0.01841613041441054,
+            0.0156981470538354,
+            0.021764297694793343,
+            -0.018831230007033416,
+            -0.014419529638789993,
+            -0.0029965475899652273,
+            0.028881185012095884,
+            0.09163526460163937,
+            0.022644302207811818,
+            -0.017933909936330133,
+            -0.04663928709016869,
+            0.004962951444105128,
+            -0.041333041174526026,
+            -0.027545628007617527,
+            -0.037142394579894954,
+            -0.013358143162788402,
+            0.02719686432396677,
+            -0.0034971800937504205,
+            0.04413635817371732,
+            0.04903857110328917,
+            0.03186067155630788,
+            0.007831051050269414,
+            -0.025257538623313574,
+            0.023517513773109414,
+            -0.0016527403209505964,
+            -0.0016101860707429347,
+            0.014448515185903624,
+            0.009905018312477684,
+            0.0024114582235163955,
+            0.00172043851384822,
+            -0.03818316221102173,
+            -0.02304537008110578,
+            0.01024969233570089,
+            -0.01901116356327339,
+            0.019791121460848986,
+            0.07985067340821365,
+            -0.051903191272236665,
+            -0.025924646976556942,
+            0.014766511532849252,
+            -0.002073086076894999,
+            0.030959269926800852,
+            0.006394417986617089,
+            -0.028177586693713442,
+            0.010725285336674526,
+            0.0009614052276924192,
+            0.07500253788978564,
+            -0.00036794182149960157,
+            0.004058458332826519,
+            -0.013218269166951234,
+            0.027613539268683245,
+            -0.012995170442480582,
+            0.023388707018488407,
+            0.026389423313105165,
+            -0.003068639956061266,
+            -0.02661967649632026,
+            0.026145274570635913,
+            -0.03300591322174331,
+            -0.004556072066733024,
+            0.020045722084112896,
+            0.0033437343914784865,
+            -0.010224220905483085,
+            -0.0025025498684079703,
+            0.03460103245395426,
+            0.04514362357278504,
+            0.03797031988058962,
+            -0.004586662475576956,
+            -0.06351723722596923,
+            0.06037983484369874,
+            0.019415753124363902,
+            0.028850618647994476,
+            -0.0034810237045849006,
+            0.02957938154808314,
+            0.02642652523231673,
+            0.013187653724133275,
+            -0.007856320804109569,
+            -0.007065882659482447,
+            -0.016672784625197085,
+            0.029020161154209707,
+            -0.013243058921970134,
+            -0.014900932834241732,
+            0.006001769901896033,
+            0.008648456186944271,
+            0.03484763413464497,
+            0.012279368984306564,
+            0.02379643508687133,
+            0.04933044959399941,
+            -0.021047723935647383,
+            -0.007264823701706426,
+            0.0007058774795716792,
+            -0.010512071832195085,
+            0.027489671668749157,
+            0.009731157570361245,
+            -0.03195919640758579,
+            -0.0033626031972691915,
+            -0.04282419484651207,
+            0.01873977910485268,
+            -0.014398285316695653,
+            0.012854631447494207,
+            -0.0017372959978092432,
+            -0.037331867825735776,
+            0.001088209292238142,
+            -0.012345684323522151,
+            0.037096063326414734,
+            0.005050219690024652,
+            0.0008906051515353004,
+            -0.003697709449630797,
+            0.006548479142737107,
+            -0.010727663712299397,
+            -0.007486226350610755,
+            -0.02704618481069485,
+            -0.019752552016312926,
+            -0.014632581296303486,
+            -0.0037501138119724717,
+            0.014588411795703245,
+            -0.007119137071719376,
+            0.016440258608998535,
+            -0.03686645352111013,
+            -0.023546402448713415,
+            0.005628703432152666,
+            0.024988145751166143,
+            0.011971199595808573,
+            -0.010868778329190568,
+            -0.03270046433624425,
+            -0.021950559285357267,
+            -0.011889914022657738,
+            -0.005123971523701009,
+            0.02619713302280345,
+            -0.021990810305904974,
+            0.01165257474485436,
+            -0.009117901804315863,
+            -0.007492082755249399,
+            0.010725902070291232,
+            0.01862811996181951,
+            -0.026694281868322685,
+            -0.01623776415010199,
+            -0.009403997538684658,
+            -0.02207329027097839,
+            0.025939540497196416,
+            0.014178803271471762,
+            -0.001141821318145744,
+            -0.006281187017652108,
+            -0.035993365198554964,
+            0.0026192776111449633,
+            -0.015254976771874396,
+            -0.0116872453009606,
+            -0.015780016491993135,
+            0.021421328283124968,
+            0.02889432250124136,
+            -0.01871009416296692,
+            -0.026540472640514536,
+            0.015703370682110186,
+            -0.0002849235245164438,
+            -0.02594930889995505,
+            0.04491828744809679,
+            0.044063522197410734,
+            -0.037441864491547405,
+            -0.007005339187980476,
+            -0.030381794393931814,
+            0.06901083845539985,
+            -0.017604082400603037,
+            -0.04060591381445593,
+            0.06044850935418443,
+            -0.01756490171206357,
+            0.025539717592705295,
+            0.03452018789549939,
+            0.0016217577522523927,
+            0.00018825054264059277,
+            -0.002563606092068712,
+            0.04770277039047993,
+            -0.0032010143255574255,
+            -0.03425328601475849,
+            -0.02105909767072799,
+            -0.00840881022563103,
+            -0.028699447677549002,
+            -0.015925160861589006,
+            -0.0055750742281166,
+            0.017513987082479483,
+            0.010399718005952185,
+            0.02267201252273823,
+            0.01258511878601025,
+            0.0015081978484714903,
+            0.01874981572091931,
+            0.011158889421057756,
+            -0.002607810686437211,
+            -0.003732023411827874,
+            -0.007787189972165309,
+            -0.05524012485122728,
+            -0.025110689689279583,
+            -0.03946383236703947,
+            0.004443572529325475,
+            -0.018745077598477586,
+            0.005926160888274727,
+            0.039430331582190524,
+            -0.0049500324516682,
+            0.004352795787899792,
+            0.015566139991780173,
+            0.013148156356408315,
+            -0.015279560337042242,
+            -0.03316846614871165,
+            0.0001592209771257332,
+            0.018362411161743347,
+            -0.012706582418607377,
+            -0.03424477956681259,
+            -0.001991022841226589,
+            0.0033467286047492175,
+            0.03357019284647071,
+            -0.024359110796680374,
+            0.00359117029951049,
+            0.005061818038334802,
+            0.011359582931760315,
+            -0.026793992746244076,
+            -0.018170136079817746,
+            -0.017421603453826397,
+            0.03996608926471779,
+            0.02587013058343799,
+            0.015763410145648683,
+            -0.04512536449983358,
+            -0.02293473890636517,
+            0.048575083783480195,
+            -0.009589860756342604,
+            -0.06486262416944208,
+            0.03447666009581266,
+            0.016105654870253654,
+            0.003526582643722421,
+            -0.01354754623151814,
+            0.05401575248587799,
+            -0.03563190295559527,
+            -0.017365166993914202,
+            -0.0046288709698227735,
+            -0.033657212157212645,
+            -0.007686252059880836,
+            -0.001385273642139782,
+            0.048229062789530236,
+            -0.029083855744728694,
+            -0.030465559177859865,
+            0.0024368661249980416,
+            -0.024144031154493057,
+            0.021693278579520123,
+            -0.016405088055634124,
+            -0.018439935250344958,
+            0.006948603359995389,
+            -0.033943037395199695,
+            -0.003007796423804453,
+            -0.016783195426536633,
+            -0.02270589491206155,
+            0.020874438753828814,
+            0.016724666903417947,
+            0.027033184180344852,
+            0.0019847647852695785,
+            0.017749524526371133,
+            0.0204661893602285,
+            -0.005604929968008821,
+            0.028462234633072944,
+            -0.020790459496727066,
+            -0.04183816355941877,
+            0.03946487892663695,
+            0.01192861758684541,
+            -0.0018024841275380242,
+            0.01077082591611702,
+            -0.01475239020295694,
+            -0.03952356245715099,
+            0.03683950018557526,
+            -0.056767672022551764,
+            -0.007507600090017693,
+            0.009530544352246083,
+            -0.00788856051420074,
+            -0.011204694027562521,
+            -0.010587701466318598,
+            -0.028604940377315504,
+            0.023749537635431462,
+            -0.013672419168235691,
+            0.01953449229707597,
+            -0.011449939491554354,
+            -0.012238180529977605,
+            0.05795536247491174,
+            -0.021518581774137322,
+            -0.040672955795925836,
+            0.026066659238628543,
+            -0.014061106927896723,
+            0.03328357230532424,
+            0.03957107406164427,
+            0.030670947051333917,
+            -0.004278577327978564,
+            0.025885946540364826,
+            0.0319106101668716,
+            -0.003462160036198642,
+            -0.04686868330851868,
+            0.00555303933159587,
+            0.02156143971867743,
+            -0.03739529489075782,
+            -0.01025626126223168,
+            -0.02141970191890338,
+            0.022134910980548407,
+            0.017807484142010496,
+            -0.007232643256393455,
+            0.011318041192005274,
+            -0.010489012862455913,
+            -0.013003217163997554,
+            -0.00847279880392288,
+            0.02357274819910594,
+            -0.0016485066139744577,
+            -0.005873113634992993,
+            0.006424249429163587,
+            0.004008703814301658,
+            0.0054639858216457724,
+            0.010198079500408441,
+            0.017694720520798957,
+            -0.017087021465208255,
+            -0.0025558869389290554,
+            -0.016503579212015177,
+            -0.0420082404922411,
+            -0.003563173424359332,
+            0.022924161700566468,
+            0.012345933179990547,
+            -0.00729908668365632,
+            -0.004234683369642856,
+            -0.0022962545569187934,
+            -0.020666205489143392,
+            -0.014796149043465308,
+            0.024562653223873056,
+            0.030217494971086438,
+            0.007810961433007407,
+            0.007781263640449456,
+            -0.0030348559388780655,
+            0.007310876971216526,
+            0.016099591640649783,
+            0.02311929597780174,
+            -0.04160162038353416,
+            0.027152100845897007,
+            -0.00025396315502635146,
+            0.023717123370023598,
+            -0.04186918759097667,
+            -0.017988531857836703,
+            -0.03106663560912813,
+            -0.023595079699181436,
+            0.02437134274097885,
+            -0.02317559303912261,
+            -0.012327263775257306,
+            0.000989566600374836,
+            0.00954921216572729,
+            -0.005155079395685017,
+            -0.017508404061379164,
+            -0.04582321461886958,
+            -0.0014367264231457263,
+            -0.030417229142190933,
+            -0.017829710557332023,
+            -0.001056472801554354,
+            0.005016912084754242,
+            -0.06276644565618253,
+            0.0109965681725148,
+            -0.07270669271977365,
+            -0.004542796125637244,
+            0.040139998232966566,
+            -0.017890441783630643,
+            0.055909102081208445,
+            -0.05029104308417623,
+            0.01673962668706085,
+            -0.04507293803055318,
+            0.015385506084493484,
+            0.025677372277539627,
+            0.030235171586316813,
+            -0.022189978375852563,
+            0.008461188938299745,
+            0.008033707483259569,
+            -0.045089505755429415,
+            -0.04144698743458133,
+            -0.013640256640978024,
+            0.01036782197518379,
+            0.0012696958402094232,
+            -0.016090728932978384,
+            0.023838437877040163,
+            0.011946482002767135,
+            0.03342604921859589,
+            0.04211801335157251,
+            0.002708015395637621,
+            -0.047156646610976634,
+            -0.013104464812562618,
+            -0.003933116891120681,
+            0.03155321708116329,
+            -0.018866109579446184,
+            0.015376812010630058,
+            -0.01955557376085726,
+            0.016255217383093355,
+            0.017868332102274922,
+            -0.020325179506828335,
+            0.013434364256590097,
+            0.004176904701252156,
+            -0.03877060944639854,
+            0.0006475542749369515,
+            0.00042468300294954125,
+            -0.02116992978038693,
+            0.0032654387271108293,
+            -0.00575003786159731,
+            -0.05149954071159492,
+            -0.031382926943128234,
+            -0.015730563640253065,
+            0.0382312502769352,
+            0.004297939848910064,
+            0.019724983545557365,
+            -0.017907869879607064,
+            -0.005165628493894463,
+            0.03698976707579373,
+            0.008501001373266799,
+            -0.01725527821298066,
+            0.006879620077191753,
+            0.02455792200929138,
+            -0.03224193972301503,
+            -0.039181328884413266,
+            -0.0008206986106407542,
+            -0.023306758893533952,
+            0.01672946960876128,
+            -0.02658587698869307,
+            0.005873620623736886,
+            0.0025331170979801232,
+            -0.03490809296591325,
+            0.02456764314036355,
+            0.003291113462353163,
+            -0.004303423672882112,
+            -0.020359544091059775,
+            -0.016892415165120203,
+            0.017174797487220006,
+            0.04391489511478065,
+            0.007327321684948843,
+            -0.003943713832544148,
+            0.00862684231313155,
+            -0.006361172785753631,
+            0.05826688726534012,
+            -0.02284956969636637,
+            -0.026680023999604643,
+            0.027527436347931256,
+            -0.022597669279069723,
+            -0.006277863878949772,
+            -0.017343446085471857,
+            -0.012065192152203348,
+            -0.0004183712490442143,
+            0.019339311388446943,
+            -0.019423562920897294,
+            -0.010212527447243049,
+            0.004783006556748757,
+            0.012824133870627983,
+            -0.004855945108275706,
+            0.006239949458562236,
+            0.006847581690471813,
+            0.03094005096378827,
+            0.026993131409761086,
+            -0.011331881651539968,
+            0.03297014291876675,
+            0.010265600643742144,
+            4.666113318251278e-05,
+            0.001970879281172325,
+            0.002523664335535017,
+            0.0034107811520368574,
+            -0.03143890864030425,
+            -0.01624584278940617,
+            0.013468729048885906,
+            0.005233309338223403,
+            -0.004550544967856448,
+            -0.023783653541634683,
+            -0.009402494239255264,
+            0.03334923712760375,
+            -0.003260424721959948,
+            0.02748395114281811,
+            -0.0058374897745283065,
+            -0.01985840930273332,
+            0.017595418271610163,
+            0.02881743246631328,
+            -0.05928552440605384,
+            0.017062540894682354,
+            -0.0011508547522925793,
+            0.008953761538519997,
+            0.01610561914042553,
+            -0.008894160513648953,
+            0.013301991014533684,
+            0.04080863883897929,
+            0.010738051773755976,
+            -0.023345002072886145,
+            0.004215726310838674,
+            0.01541749448570608,
+            0.01291813299324752,
+            -0.022747942296528025,
+            0.0008640791861357522,
+            0.011008522066277018,
+            0.005893793007012561,
+            0.0026106450193140156,
+            -0.006278394080583205,
+            -0.025659188287146268,
+            0.024180550333596625,
+            -0.0044491292740074925,
+            0.026967973346698617,
+            0.04390909273429254,
+            0.009596855239692042,
+            -0.03686020608700941,
+            -0.013109951664921729,
+            0.0002799680359670127,
+            -0.0038537577602593366,
+            -0.0037552094562213916,
+            0.011105295862811114,
+            -0.04185615640149602,
+            0.019950890919369135,
+            0.006453241355231516,
+            0.015314645427201533,
+            -0.020031193483734364,
+            -0.024530173488182337,
+            0.0031949828520724337,
+            -0.01492645564652531,
+            0.012875827757040941,
+            0.005057170413681063,
+            -0.004319834710426384,
+            0.005386862713494232,
+            0.00030848054013413157,
+            0.008384941028326983,
+            -0.010811526993301901,
+            0.03175068297908541,
+            0.004680987487229685,
+            0.009264547472281849,
+            -0.02856314241700808,
+            -0.021083721348429348,
+            0.020113396771629096,
+            0.03572366771051171,
+            -0.01708529918264109,
+            0.03346921724975398,
+            -0.009042046660366718,
+            0.0033219155387035227,
+            0.0151720168318272,
+            -0.015805318210244414,
+            -0.008057236707694303,
+            -0.0005483009439015931,
+            -0.002917527004423299,
+            -0.014538511124345527,
+            0.007302979537905532,
+            -0.0208529958115422,
+            -0.02965227242147972,
+            0.020064178921373875,
+            -0.00011892622932349096,
+            0.016455188521316146,
+            0.010335044657747514,
+            -0.007816155992852236,
+            -0.0052262330850905165,
+            0.006959344748883874,
+            0.04432147882994085,
+            0.026734484553994495,
+            0.05547661378166294,
+            -0.005787997952888261,
+            -0.004559444854138438,
+            -0.0082295581123363,
+            -0.006588172755707144,
+            0.04524009609473585,
+            0.015915003261638767,
+            -0.004910285095602864,
+            0.017495607570191165,
+            0.01557886571266685,
+            0.03858259260136037,
+            0.0027648525902558316,
+            -0.016912796336835408,
+            0.04014755669075221,
+            -0.004077405782937414,
+            0.008616827127381418,
+            0.011102692199778469,
+            0.0024185742218798562,
+            0.00807141794538637,
+            0.02996274720815659,
+            -0.012429186019907761,
+            0.01680942269506771,
+            -0.04571916592313758,
+            -0.02775530246832996,
+            0.014370280587419414,
+            -0.004931905540824582,
+            0.03419959665162407,
+            0.01668521021731861,
+            -0.023105305446763272,
+            -0.024761385385514877,
+            0.021529428247380984,
+            -0.002603042335561961,
+            -0.024135326707477325,
+            -0.015906833215844473,
+            0.003286605488908195,
+            -0.0023437997044735617,
+            0.0060251848371355166,
+            -0.013481652076194643,
+            0.0002908426939460218,
+            0.01953362396253743,
+            0.0038544889405965057,
+            0.020760460182873287,
+            -0.016224396121239458,
+            -0.00259233446682179,
+            -0.009282861237653619,
+            -0.02952130765234087,
+            0.017988370145301916,
+            0.006138236653154911,
+            -0.007607182955958238,
+            -0.009516767050028553,
+            0.0048347778733230365,
+            -0.005819209642901871,
+            -0.0025851701689263706,
+            -0.01073014664868326,
+            0.012433465684360061,
+            0.025784014942833097,
+            -0.016875624785552303,
+            0.007944349714123104,
+            8.651589872318479e-05,
+            -0.022306044741692753,
+            -0.022127016745066307,
+            0.020143472307319017,
+            0.002792646249762062,
+            0.008768813346304779,
+            -0.004074214276300631,
+            -0.00015024341785520542,
+            0.002990550758627699,
+            0.01951746553838439,
+            0.03896451381932467,
+            0.004232673242619471,
+            0.027876586543280867,
+            0.0145860303663199,
+            0.03994234200854897,
+            0.009657017763624678,
+            0.00696607158576712,
+            -0.026055387694656488,
+            0.02819890357011778,
+            0.00018297502510753014,
+            0.043190622821775236,
+            0.022488402221902314,
+            -0.0407730101161195,
+            -0.0021946518523759066,
+            0.01669520746276111,
+            0.013212623257794862,
+            0.017009844136059122,
+            -0.004537512994041156,
+            -0.014041213424740049,
+            0.028135404528975894,
+            -0.022859586430516003,
+            0.0008637006862481223,
+            -0.004486332934188017,
+            -0.025031795510001887,
+            0.007351370254175368,
+            -0.004567816916440959,
+            0.030025413528233364,
+            0.0014976740344892735,
+            -0.022285096307825804,
+            0.022837170656498543,
+            -0.009551269792616045,
+            0.011045530488684207,
+            0.008809047185288755,
+            0.020065001354321994,
+            0.031993362733068575,
+            0.009143210434415064,
+            0.02133640814803879,
+            0.012241777474889248,
+            0.014771538127546474,
+            -0.008804192497858033,
+            -0.02341427260922065,
+            0.023167424359504862,
+            -0.0011598649565273595,
+            -0.029627604902846688,
+            0.0021991474725787303,
+            0.003261878079344105,
+            -0.0005935713331682358,
+            -0.009144689470851402,
+            -0.005580557862988033,
+            -0.007240836451875213,
+            0.019631377011298905,
+            -0.007085213886352692,
+            0.007017573363709754,
+            -0.004074171791476129,
+            -0.00822871835665378,
+            -0.0421101573155962,
+            0.004210914075286222,
+            0.003408427700622687,
+            -0.01752179330665445,
+            0.006384440404174512,
+            0.011121481025573962,
+            0.043020575455262985,
+            -0.022809979913698254,
+            -0.0005737138409049003,
+            0.019206623212867476,
+            0.0009562551889820241,
+            -0.023427467810802533,
+            -0.038931688376706186,
+            0.007172345672967499,
+            -0.03467050750542982,
+            0.023185301620612612,
+            -0.0007075596339842658,
+            0.02387977204565927,
+            -0.03043468414449851,
+            0.013909304572370437,
+            0.004784668158421625,
+            0.012524879174804915,
+            0.021654512058639427,
+            0.03142146922870715,
+            -0.02053473475540882,
+            -0.050409862638621725,
+            -0.005406667316025549,
+            0.00035138762584085976,
+            0.022018487016014856,
+            -0.026014580970974616,
+            0.00025633004699302783,
+            -0.003349094027291921,
+            -0.02401366769212778,
+            -0.01735996745614562,
+            0.01962340791446025,
+            -0.050011480783403936,
+            0.015042913576492952,
+            0.0262737424555697,
+            -0.02507473901367252,
+            -0.05408889321034101,
+            -0.018988587082325706,
+            -0.028217893770659686,
+            -0.009285770094319318,
+            -0.00661159230987788,
+            0.028597405154740078,
+            0.003525576256054959,
+            -0.011462219358952466,
+            0.025401668390029348,
+            0.006155837251830549,
+            0.0026050693705924593,
+            0.015985722392086222,
+            0.007289910674212967,
+            -0.011452511453623235,
+            -0.014599577191293152,
+            -0.04558584556100621,
+            0.010197536827229795,
+            -0.0045431962637690155,
+            -0.019372598050508005,
+            -0.030147301333718016,
+            -0.0137828704002156,
+            -0.027060515748528666,
+            -0.0017291966950107505,
+            0.053783684195381654,
+            0.03292494290301388,
+            0.005560172296221419,
+            -0.0026085670565688026,
+            -0.010466890489356092,
+            0.036247483457995895,
+            -0.006789746401727701,
+            0.03662690472838588,
+            0.001784522073210422,
+            -0.0242673253380092,
+            0.01169382884548301,
+            0.02892627181542449,
+            0.009858451662166387,
+            -0.0005592283645355213,
+            -0.006160132931009508,
+            -0.008782830470079547,
+            0.0031207630972208394,
+            0.025613233596604418,
+            0.003268170392795548,
+            -0.010062991425127038,
+            0.018106967407336815,
+            0.004480291699976396,
+            0.02010374270557836,
+            0.03555357587371324,
+            0.023489837757983054,
+            -0.021782173652026497,
+            -0.026581762796336082,
+            -0.0013561486781077014,
+            0.0072266106510934505,
+            0.03551967295942857,
+            -0.01651315514034218,
+            0.0001274396319698452,
+            -0.008272735507120127,
+            -0.00894931703949637,
+            0.011149952898504355,
+            -0.0029671593121986116,
+            -0.0030752010285951576,
+            0.01662711184497578,
+            -0.0011672263615879709,
+            -0.033448222499147764,
+            0.0078288560210653,
+            -0.023496331346917084,
+            0.0051040606404415545,
+            -0.004113104164416575,
+            -0.007076525381595441,
+            -0.002157548492138289,
+            0.006302962169276006,
+            -0.008065841090139872,
+            -0.00035242603988955366,
+            -0.01586406637763306,
+            0.008271894439780969,
+            0.007438312999409211,
+            -0.0069617481460406095,
+            0.011402932583388205,
+            0.002666895909532698,
+            0.0005219881149353855,
+            -0.012159425670629751,
+            -0.012042145306711322,
+            -0.00957276588761916,
+            -0.03401863879157551,
+            -0.02604353712472079,
+            -0.024949533021906844,
+            -0.0014209149651479556,
+            -0.006566685925666972,
+            -0.03187796739029736,
+            -0.01209152137926458,
+            -0.011110763124407752,
+            0.007756961097537267,
+            -0.013260548545917543,
+            0.005433285079114047,
+            -6.399098520041351e-05,
+            -0.0044960488826674555,
+            0.016615636438440516,
+            -0.003643291599919672,
+            -0.019488010211707343,
+            -0.03927818878149417,
+            0.008209336177358258,
+            -0.02805593125673831,
+            -0.0038816771550190788,
+            0.007607119219721711,
+            -0.00528681677614909,
+            -0.029502897838743516,
+            -0.009149794737430706,
+            0.008767001199311985,
+            0.003507217008280066,
+            -0.01773349756494436,
+            0.03579670773605949,
+            0.026971617790626075,
+            0.011875179879209696,
+            -0.019891342676443626,
+            0.0508929813275565,
+            -0.0047010735337056275,
+            -0.012544935684845649,
+            0.018297503869683027,
+            0.02355022268131519,
+            0.03273167043015095,
+            -0.024830243320788693,
+            0.0009370814985214882,
+            -0.021937793653233454,
+            0.00535662615596573,
+            -0.035009915581788176,
+            0.0028595659314725012,
+            -0.014395027018699446,
+            0.015596459769560532,
+            0.03397372881171701,
+            -0.013254226072981457,
+            0.011195747181764677,
+            -0.016299384375589934,
+            0.00730244731485402,
+            0.00569944090461748,
+            -0.012916284280345044,
+            -0.005812311882399539,
+            -0.022902815152116493,
+            0.020140276669979015,
+            0.005809799950176385,
+            -0.0008130655714656685,
+            0.0104085826652653,
+            -0.044723432238900435,
+            0.03286429597337671,
+            0.008730225944501239,
+            0.017599189944921396,
+            0.0029762446259779998,
+            -0.0003068671951232415,
+            0.011611551606224214,
+            0.011228401175698422,
+            -0.016029727902740706,
+            0.004345913009102136,
+            -0.004613061137378433,
+            -0.03850533909474302,
+            0.009363351681220008,
+            -0.03631184057978666,
+            0.014427784677340159,
+            0.016026872002233575,
+            -0.004170582842643461,
+            -0.025166189571409366,
+            -0.005428471012333673,
+            0.010827910811085384,
+            -0.009783069664327864,
+            -0.009987805234276443,
+            -0.04163358183060102,
+            -0.025236440364963034,
+            0.034097508113604576,
+            -0.002119474613596975,
+            -0.025826403511707147,
+            -0.010830738112107807,
+            -0.011351132636491575,
+            0.020207687645133,
+            -0.006542026283946858,
+            -0.0006263975804936843,
+            -0.04150011545311013,
+            -0.0073396962917864515,
+            0.011192225907726928,
+            0.027746213794734322,
+            -0.035970017302005566,
+            0.02708126943501373,
+            0.002105633542220187,
+            0.014262674845755191,
+            -0.01626664983003942,
+            -0.005631805361903626,
+            -0.013990539604372344,
+            -0.035877093361375685,
+            0.016487723013906077,
+            -0.008607758248055054,
+            0.004714436224135091,
+            0.04458009880881818,
+            -0.0036872068506712727,
+            -0.009466125348499464,
+            0.007600705817195143,
+            -0.0034590883553048063,
+            -0.010351576085287273,
+            -0.0028564505793999295,
+            0.001959661298507,
+            -0.007080697863126256,
+            -0.04183476693971616,
+            -0.00499888976530217,
+            0.04684391782633148,
+            0.006361744381879857,
+            0.013276793765092465,
+            0.013516220521120684,
+            -0.018335746557932003,
+            -0.045577810515431215,
+            -0.0071616111758423025,
+            -0.0032763880176386808,
+            -0.022708028392758567,
+            -0.0240284945459861,
+            -0.008735040856657758,
+            0.005844145402902923,
+            0.006359368165686724,
+            -0.01120763599008222,
+            -0.022676131361171057,
+            -0.027085968700452136,
+            -0.010526103176253857,
+            -0.02123944716136792,
+            -0.010792145796733375,
+            -0.027142270190771887,
+            -0.028742234538796804,
+            0.02880426953386151,
+            0.007575182232859564,
+            -0.004084088301176117,
+            -0.010005765476121966,
+            -0.01255640546813973,
+            -0.0004774761617900028,
+            -0.0025307994917461232,
+            -0.005822065709999144,
+            0.0038614286313282164,
+            -0.011390482620811868,
+            0.023152086963976554,
+            0.0021220701260546895,
+            0.018329162172128138,
+            -0.05772324571231113,
+            -0.002709083851090071,
+            0.021497896395795905,
+            0.0033025410488410913,
+            -0.005412445141910314,
+            0.029352650459792132,
+            0.011223947499453582,
+            -0.006539460282439711,
+            -0.031355428613718234,
+            0.020942976176820262,
+            0.024122993515785947,
+            0.00817615634503138,
+            -0.012994160510030828,
+            0.01221095914384204,
+            0.01629778384790159,
+            -0.004494985018246821,
+            -0.011161098655072933,
+            0.005586857967039181,
+            0.020941039642372112,
+            0.013580622629911925,
+            0.0013398071867077221,
+            0.030905022925366163,
+            0.0006862084391508121,
+            -0.03521231612862609,
+            -0.01908191995763771,
+            -0.01275995164030872,
+            -0.003584331305986364,
+            -0.015264425022763126,
+            0.005294329654809353,
+            -0.002355465010876059,
+            0.006335052149475905,
+            0.022655379849785256,
+            0.03593231939948758,
+            0.0146937743203089,
+            0.00937330328448886,
+            0.003167462724350819,
+            -0.0013235176101076284,
+            0.012102872918129403,
+            0.008638011773075119,
+            0.0350701240268404,
+            -0.014653221697693864,
+            -0.011003994230902835,
+            -0.0037767288519369537,
+            0.010100131061186566,
+            -0.004113527518662366,
+            0.020414393256136372,
+            0.03274709615649469,
+            -0.0037222949335916876,
+            -0.005604552180246868,
+            -0.006160118231975786,
+            0.0041467454414452565,
+            0.01807879127455274,
+            -0.01252134965321004,
+            0.004364858987501461,
+            0.005549300665159658,
+            -0.0071897162833861,
+            0.018567059255813558,
+            0.0002622074720860951,
+            -0.012983550838417323,
+            0.03266376564402191,
+            0.11449509387240629,
+            0.0017251916835047628,
+            0.007125215166916139,
+            -0.040229997815548584,
+            -0.020537987527148888,
+            -0.00245728735683473,
+            0.005779972828836593,
+            0.007004732428236457,
+            0.02225914982152608,
+            0.006505827715591914,
+            0.0006708045351142873,
+            0.031708522931495536,
+            -0.017334083511709092,
+            -0.002522861743921599,
+            -0.01745656559688129,
+            0.013890451996979048,
+            -0.014312605700272691,
+            -0.013784963572164409,
+            -0.01572417853955075,
+            0.004507425871748977,
+            -0.000601158156253668,
+            0.0016158265715634729,
+            0.01791303234734911,
+            0.0011926917387973374,
+            -0.026393351252156246,
+            0.0006694008039871878,
+            0.001026031515214626,
+            -0.004514419527575117,
+            -0.011320929719992686,
+            -0.02223001723942815,
+            0.016235841442653918,
+            0.014188311619681008,
+            -0.004895218832118094,
+            0.009244697376301004,
+            0.010558796153959677,
+            0.02707485418319703,
+            -0.028380342401606758,
+            0.0040204430418629405,
+            0.007023041225985294,
+            0.01606556581447933,
+            0.017617862031190843,
+            0.0034870847111552862,
+            -0.011253315705053224,
+            0.03238289399766116,
+            -0.010563282687352038,
+            0.0065093536270233385,
+            0.014966802208644462,
+            -0.03541611540945151,
+            -5.821682910348165e-05,
+            0.0061985444938402745,
+            -0.011716201661363997,
+            0.009580035961315467,
+            -0.03120692534097433,
+            -0.0039291243987978845,
+            0.009317468331974374,
+            0.017611312558635673,
+            0.008804996982574765,
+            -0.0034129169537151925,
+            0.010822226698822738,
+            0.012510661458054741,
+            -0.0027147975086409966,
+            -0.008855213644027594,
+            0.01238502686635956,
+            -0.0007687053446954701,
+            0.00450108863711973,
+            -0.00799170686059026,
+            0.0029755595812552293,
+            0.002290650917244691,
+            -0.01554856300008412,
+            -0.011779430686394945,
+            0.00438393126679289,
+            0.0036931891075561957,
+            0.001535662168147833,
+            0.012014408636737167,
+            -0.006087255045485002,
+            -0.0077816220325470226,
+            0.0040573724156794284,
+            0.003136943055301706,
+            -5.505014316363865e-05,
+            -0.030625245024561508,
+            0.01585923084309149,
+            0.0067595435845396405,
+            0.010013308598626031,
+            -0.005315626260228517,
+            -0.00903142414091241,
+            -0.007240920015018893,
+            -0.01228521172101564,
+            -0.010657894223173453,
+            0.007658605592936815,
+            -0.00724924993432376,
+            0.011124174211132038,
+            -0.03128027787357322,
+            -0.013988715435985316,
+            0.01231792993245938,
+            -0.01528161242159377,
+            0.0007675821323979293,
+            -0.008389914312201402,
+            0.0009417242487395995,
+            -0.005119304379761705,
+            0.007887436211143328,
+            -0.03731503169547854,
+            0.007878001990271541,
+            0.02456170736765724,
+            -0.00330924625934981,
+            -0.020259771366167557,
+            -0.009613930457931786,
+            0.0241314385136084,
+            -0.008267831641886344,
+            -0.01645888287387047,
+            0.008421304862524506,
+            0.010140813149093993,
+            -0.0006309118552875393,
+            0.0008237784055549769,
+            -0.003726984191590917,
+            0.01669125388798323,
+            0.02054777616420997,
+            0.041930770964107515,
+            -0.014789610432004559,
+            0.01092349703703152,
+            0.02866938974792608,
+            -0.03378576980722847,
+            0.014728113846952417,
+            0.02518409645155824,
+            0.014590531327650454,
+            -0.0004446956315190408,
+            0.001609777186372437,
+            0.0068939321280085475,
+            -0.014273492115311677,
+            0.009247676157551494,
+            0.013380104440744367,
+            -0.018407746260686045,
+            0.020444430509441124,
+            0.03140071554008003,
+            0.0025839998101985736,
+            0.01355205268903058,
+            -0.006268939830271642,
+            -0.013860692186849113,
+            0.014717364068116991,
+            0.031429975488060896,
+            0.0006919565473572909,
+            -0.018561500446336806,
+            0.019128244642842268,
+            0.006988095806264863,
+            -0.003678529269242034,
+            -0.0021993078588142664,
+            -0.021334919445200388,
+            -0.021003586791369228,
+            0.017304350704792804,
+            0.013508779408333964,
+            0.027794485858413365,
+            -0.010594702782952029,
+            -0.007818089330662161,
+            0.014293688127901867,
+            -0.022368568887004094,
+            0.015426482139881204,
+            -0.02160820059598751,
+            -0.00571499813703566,
+            -0.0077402210272634235,
+            0.03274360648629609,
+            0.001441304806200403,
+            0.02734331677589297,
+            0.016457027764064002,
+            -0.033609420134045206,
+            -0.029790535553657092,
+            -0.00712106584238484,
+            -0.02307933297713104,
+            0.010356561171963577,
+            -0.03901183838032092,
+            -0.00071735150189382,
+            0.008251646656432826,
+            -0.008079917288471352,
+            -0.0007087526632966007,
+            -0.006419191399519291,
+            0.013989344185652506,
+            0.013661633160336152,
+            -0.0228103143826663,
+            0.023644639659577426,
+            0.008343481940133684,
+            0.025998483906975782,
+            -0.02076055376006461,
+            0.019148568238437927,
+            -0.017705363646416224,
+            -0.02777313933338929,
+            0.014358130220942483,
+            0.023585731219788467,
+            0.01635364038245971,
+            -0.007122853965024528,
+            0.007110003583832299,
+            -0.03556761320536796,
+            0.027585457849964405,
+            -0.013064062623866305,
+            -0.012504161289952458,
+            -0.0032379276562094555,
+            -0.011931827532360686,
+            -0.002574161005800279,
+            0.0008533907465694845,
+            0.0034345232246102806,
+            -0.03131462673429805,
+            -0.0002450959360520896,
+            -0.03727416416766574,
+            -0.028944814604815072,
+            0.01360126708805471,
+            -0.00036361284498363273,
+            -0.0049097813612814616,
+            0.02933242705521636,
+            -0.026339136251046315,
+            -0.02693295650849956,
+            -0.009590247544093352,
+            0.02853096456887074,
+            -0.0220673798697511,
+            -0.004519050393279848,
+            0.0015034014871316069,
+            -0.01828226786700525,
+            0.004436383319946364,
+            -0.0012748481042235446,
+            -0.010416065158876917,
+            -0.01105146139270597,
+            -0.03229479099927737,
+            -0.005822133182096844,
+            0.02106215163620738,
+            -0.006957125617688484,
+            0.002746241228040847,
+            0.013312248331456934,
+            -0.010968629118224582,
+            -0.0029946900704643812,
+            0.0017875284441794235,
+            0.023304175718480834,
+            -0.021033941064173063,
+            0.007714334541806199,
+            0.005955442031111047,
+            0.0016405586860131207,
+            0.018406562522853148,
+            -0.0054922197063461825,
+            0.0037690571959124617,
+            0.0162126534419016,
+            -0.009969304596173302,
+            0.017959307638726878,
+            0.007691549802475525,
+            0.006681837561745192,
+            0.002775700618222622,
+            0.007712770283579111,
+            0.009752652973700739,
+            -0.013506020126078112,
+            0.016911141425375933,
+            0.022468715726349706,
+            0.00023233555209372766,
+            0.021066580799163234,
+            -0.02302516292580905,
+            -0.028618544628453015,
+            0.018297878810036087,
+            -0.0225078941299981,
+            -0.012735208680311652,
+            0.016459925496937065,
+            -0.0043032724578332855,
+            -0.00797609922711709,
+            -0.014986057073351055,
+            0.02260618561608301,
+            0.02132930921464508,
+            -0.020249788476562527,
+            -0.012385936327068884,
+            0.0072304431443959505,
+            -0.004373475447051833,
+            0.007086656081865733,
+            0.03195810137269375,
+            -0.015982210897392764,
+            0.006818890093901413,
+            -0.0038306138017521626,
+            0.0030753642449792215,
+            0.017830546304667783,
+            0.009470018737705222,
+            -0.0019815241970236146,
+            0.030100238416495066,
+            -0.001330614013444443,
+            0.016392226282459177,
+            0.0004888028477871524,
+            -0.010948833692356176,
+            0.006343890495626752,
+            -0.026652806306173643,
+            -0.008627570055007363,
+            -0.001203530223345183,
+            -0.009581494451831351,
+            0.07598616930784373,
+            0.003930153552069478,
+            -0.01295957554290434,
+            -0.001673007115291578,
+            -0.001368287102181597,
+            -0.006897179139643672,
+            -0.00960679451603541,
+            0.0287166325268306,
+            -0.016613613331623346,
+            -0.01894527592131826,
+            0.0008370014132002703,
+            -0.006785618888421608,
+            0.00919924251249592,
+            0.01816363936999796,
+            -0.010225961928154075,
+            -0.01604759761423727,
+            0.001577138242530474,
+            -0.003520564936698765,
+            -0.004395434431595494,
+            0.0052399043934902014,
+            -0.010127122101423486,
+            0.006456662777583096,
+            0.026560491731327886,
+            -0.013684536319659296,
+            -0.01425196172663986,
+            0.004084982328707292,
+            -0.0010682084059972982,
+            0.009758641913073978,
+            -0.00864879062601467,
+            -0.00481222050200678,
+            -0.014375330954464725,
+            -0.011378890549403524,
+            0.02505641780042516,
+            -0.02861164504722546,
+            0.0034060847953066354,
+            0.014937227531980433,
+            -0.00016904224334342915,
+            0.01541381851648916,
+            0.0033028855218525644,
+            -0.03615935765718852,
+            -0.013677774380513626,
+            -0.02073839691250734,
+            0.019821074228011413,
+            0.01688422012576042,
+            -0.0033731840788147442,
+            -0.013840993586441808,
+            -0.026149880178874887,
+            -0.007430563876740068,
+            -0.02466234818567362,
+            -0.020708282040996014,
+            -0.019276991448222817,
+            0.006137002799644838,
+            0.005162892426268572,
+            0.022881759407670583,
+            0.0025854880108157857,
+            -0.015058889448404231,
+            0.0029948608544278386,
+            -0.006722836436935071,
+            0.011845298212977357,
+            -0.02747944486818378,
+            0.015311272959213262,
+            -0.01610493464983465,
+            0.025239768880017402,
+            0.007425208902253236,
+            0.0010569008485112994,
+            -0.008822681490988575,
+            0.00010704167529046774,
+            0.014883787023445807,
+            -0.004186968199973372,
+            -0.0007781259002896265,
+            0.02787555889813697,
+            0.02087373813833492,
+            -0.0017783425135865238,
+            0.008743718448136432,
+            0.028026471206552342,
+            0.019081083283793984,
+            0.0325261942213039,
+            -0.013815526479760766,
+            0.005113676604826083,
+            -0.028873600114081546,
+            0.010510475843707614,
+            -0.010871039729515013,
+            -0.010201824683340651,
+            -0.0067946599549165735,
+            0.027756887504389347,
+            -0.007962601205446472,
+            -0.0006687976124826922,
+            0.03469894503563926,
+            -0.014185294847328063,
+            -0.00010543064621374546,
+            0.0007386418834840544,
+            -0.009759481309152999,
+            0.02742722730386853,
+            -0.0073590367291510046,
+            0.018456032721630843,
+            -0.019560882612141943,
+            -0.013705917411254969,
+            -0.011894560208199602,
+            -0.001142768182039817,
+            -0.02576754877141608,
+            -0.008755240066826174,
+            -0.02249287216087102,
+            0.00859172262856532,
+            -0.0046012388553094375,
+            0.018845920253446127,
+            0.0011617844393195566,
+            -0.005275799428018491,
+            0.029069367036565145,
+            0.0088080893110627,
+            0.018100411389202663,
+            -0.0004971948011851863,
+            0.01216566778850985,
+            0.02369448391321832,
+            0.002637001729688794,
+            0.005012948614448268,
+            -0.024213704079796478,
+            0.003420184574622472,
+            -0.0004359900462023086,
+            8.265488264269535e-05,
+            0.016267948861008616,
+            -0.019572032453100303,
+            -0.02216264534756073,
+            0.021871880626992544,
+            -0.012742926070754964,
+            0.011676769226089007,
+            0.0035362870996336194,
+            -0.010732229460038356,
+            0.0006838777488770268,
+            -0.03083438636386314,
+            0.008820746379709355,
+            -0.03216685858285627,
+            -0.032230808828686675,
+            -0.013547681710236494,
+            -0.009994106571421729,
+            0.0008758088338659712,
+            0.028447187234357855,
+            -0.00795317029726568,
+            -0.015797892954136448,
+            0.005363565557248085,
+            -0.011199252697624111,
+            -0.00817162221789202,
+            -0.0007867892785855576,
+            0.005856939999700809,
+            0.009461959355310273,
+            0.005997804620690314,
+            0.011648335019189696,
+            0.007491946658046298,
+            -0.008028800679169293,
+            -0.008837725220736191,
+            0.015889923814605387,
+            -0.0014409716087425222,
+            -0.01176424391171685,
+            0.030430130082762363,
+            -0.013300620185884817,
+            -0.0005592926632214984,
+            -0.0021975669766032472,
+            -0.0015687208316663597,
+            -0.0021652076181411244,
+            -0.014225289143358926,
+            0.01870568404189425,
+            8.334501002782455e-05,
+            -0.014288806218067678,
+            0.0032841743905975634,
+            0.003841780428153241,
+            -0.0036498339483411074,
+            0.008388922960269383,
+            -0.008147275785908849,
+            0.017348147498892,
+            -0.015113395971380741,
+            0.008740452410210708,
+            0.0038335161236289234,
+            0.0043482078469644385,
+            -0.0065755348640655865,
+            -0.01592809184384243,
+            0.009642593996701513,
+            0.012051872087663658,
+            0.02173264012652684,
+            -0.02525660485859878,
+            0.00876161653901566,
+            -0.005374533920781607,
+            -0.014534179652013885,
+            -0.016127866637369738,
+            0.00982303248417706,
+            0.022322910435388785,
+            0.0063086346898091296,
+            0.012139263756190026,
+            0.00013383660542132047,
+            0.005848259030010402,
+            -0.014767198851314148,
+            0.010570158368602513,
+            -0.011355774405066537,
+            -0.018439261453084426,
+            -0.005243587266408389,
+            0.028894120979627323,
+            -0.005492024421080143,
+            0.017938747701972086,
+            0.0021350711250093858,
+            0.029007356490801176,
+            -0.008686781541806358,
+            0.0014883465513564527,
+            0.0039866022469891155,
+            -0.01295271525889559,
+            -0.004272008629260297,
+            0.01976273843646234,
+            0.04882868638601926,
+            0.010204475902027572,
+            -0.0009155109545661691,
+            0.010069071220988797,
+            0.005626418787776309,
+            0.002148101019776153,
+            -0.00805781810841286,
+            -0.030431071067211577,
+            0.008883707378543252,
+            0.005628218215212441,
+            -0.01997825125961566,
+            0.019838234890736068,
+            0.026483365072033395,
+            0.023040623182975964,
+            -0.011585643598171132,
+            0.016539930468533123,
+            -0.010092529089936188,
+            0.0177008735686489,
+            0.028665424114962996,
+            0.013810061902648515,
+            -0.010487150104822674,
+            -0.0020847870330548534,
+            0.006329342599564054,
+            0.004186981876004016,
+            0.00026239408223817384,
+            0.015190614519766204,
+            -0.019155180136775043,
+            0.01005728016778505,
+            0.014725826697964637,
+            0.03322436221319799,
+            8.049930133627564e-05,
+            -0.01645191835557271,
+            0.0016249281996128196,
+            -0.009976155662939693,
+            0.00856399492362911,
+            0.000632427127231252,
+            0.0032242260692672124,
+            -0.009688119824276678,
+            -0.012682779503915694,
+            -0.002851918183031152,
+            0.009403205508474241,
+            0.0021188716361101698,
+            0.002759004011231831,
+            0.007641925150177999,
+            0.0022830062031146204,
+            0.01713024474384216,
+            -0.0011181980841546123,
+            -0.00029141889484139217,
+            0.004601090256674874,
+            0.010509199590505558,
+            -0.01649852835819883,
+            -0.017548369964263514,
+            0.004337237475053179,
+            0.012217494210546133,
+            -0.013159179948224596,
+            -0.013241981632059755,
+            0.015704543608959117,
+            -0.01913575130892795,
+            0.024934102868065366,
+            -0.00942381892138331,
+            -0.004543611336526587,
+            0.000343200128934031,
+            -0.00727538880611703,
+            -0.0038751118497522673,
+            -0.0030064279504724797,
+            0.003984715814505561,
+            0.0019261818110367147,
+            0.007472350699291607,
+            0.008472369788801792,
+            0.014568547873264247,
+            0.006043895352236181,
+            0.0032580830408778395,
+            0.009109104498825864,
+            -0.014523846294121576,
+            -0.014800649887278973,
+            -0.002704791598011488,
+            -0.008895394038700809,
+            0.00902016867266151,
+            -0.006607182154874112,
+            0.00845408382182461,
+            -0.015501867835442603,
+            0.015573642225745032,
+            -0.012200552219016517,
+            -0.0008900215712478553,
+            -0.009666233545144905,
+            -2.7470277092642562e-05,
+            0.008191697904514857,
+            0.014605287653054787,
+            0.0070552942843382215,
+            -0.031921405225964206,
+            0.0017403427534633655,
+            -0.008846568352320488,
+            0.03446319360336241,
+            0.002612606662691094,
+            -0.01060863123202008,
+            0.020352181356568008,
+            -0.01592124132155158,
+            -0.00027704668715660693,
+            -0.01602075523652062,
+            -0.004797557413047212,
+            0.00707939446796679,
+            0.016399017978990946,
+            0.02759049512664933,
+            -0.005697020356865688,
+            -0.007995305872076221,
+            0.006384047872862793,
+            0.029590696670334232,
+            0.012246955508890425,
+            0.024748351124415785,
+            -0.007633712004672087,
+            0.014348228716100414,
+            0.007773260552660029,
+            0.0076045488653348975,
+            0.004321689250440021,
+            -0.002591801797145718,
+            -0.011698574606499644,
+            -0.014974971814360001,
+            -0.0037563183330623384,
+            -0.016987915346053498,
+            0.00012247833530362867,
+            0.021344400276501117,
+            -0.00475351423959752,
+            -0.0033847161875730026,
+            -0.01874186437694691,
+            0.0033461670218611346,
+            -0.010029445302549983,
+            -0.00367421246775698,
+            -0.013133469163793961,
+            -0.0254122035745965,
+            -0.018556369080030412,
+            -0.002155244024999317,
+            -0.00929309186646207,
+            -0.015194269781263309,
+            0.0070354690219415535,
+            -0.016305117309271634,
+            0.0080837166473439,
+            -0.00448473417024141,
+            -0.016508775226236883,
+            -0.00958925666995396,
+            0.01100335153658195,
+            -0.0041819993634961685,
+            -0.01929493648733792,
+            -0.01282300765645703,
+            -0.011300260564838231,
+            0.0051733704121938884,
+            0.005996780699975213,
+            -0.0038111648304178544,
+            -0.0028279639348479987,
+            0.016640121845366295,
+            -0.010197365811240557,
+            0.009878730312873492,
+            -0.03335474019383245,
+            -0.03551826040406959,
+            0.004497847402931096,
+            0.01507007013703286,
+            0.0013085881889541252,
+            -0.02078680101065954,
+            0.010806315988966003,
+            -0.016499186313125556,
+            -0.00858013792516617,
+            -0.008522534127498645,
+            0.009649521088411411,
+            -0.007283003380127847,
+            0.01299542385123007,
+            -5.3226845309105124e-05,
+            0.030477683597373687,
+            0.0074153455615142755,
+            0.004161881195438522,
+            -0.00679081048132963,
+            0.008354912600159931,
+            -0.001621776416108105,
+            0.004080191054849395,
+            0.02120524874226696,
+            0.012440848502230418,
+            -0.00218041171518331,
+            0.006458415098595872,
+            0.005678055538564355,
+            -0.017186537377121235,
+            0.02507960402044456,
+            0.024678178106806115,
+            -0.008205288425409454,
+            0.004108402466376321,
+            0.005783335302169668,
+            -0.008846663924040241,
+            -0.006897707231869481,
+            0.006527795262026734,
+            -0.018474181903007085,
+            -0.0008721381678008483,
+            0.010598084358464908,
+            -0.02255441909004843,
+            -0.004840511019512733,
+            0.021745651362109403,
+            0.004631567631335321,
+            -0.018558972627958834,
+            0.02764788406651861,
+            -0.004942361238163609,
+            -0.0027290999934532494,
+            -0.010876135202650048,
+            0.01671497143541799,
+            -0.012173527207375091,
+            0.008443317346655462,
+            0.011021541861818043,
+            -0.006295639322287843,
+            0.003083685676540999,
+            0.008277206401503182,
+            -0.004504302913715969,
+            -0.0068075424927125676,
+            -0.0010508416962215848,
+            0.010167160518175114,
+            0.0036980815941666336,
+            -0.022704963865525812,
+            -0.006020073261002279,
+            0.02934227992246887,
+            -0.015082922782242507,
+            -0.0050916398557337705,
+            0.005376294716056329,
+            0.007709342013963906,
+            -0.003476236561444464,
+            -0.0023399991233097337,
+            -0.00787486645740744,
+            0.010356414979428607,
+            -0.015950083845400274,
+            0.02076360898078935,
+            0.025608263494550438,
+            0.024504236990840842,
+            -0.014814235990238182,
+            0.010828584133720797,
+            -0.011703992394890898,
+            -0.011047470905668532,
+            -0.015542835222173864,
+            -0.001658296303013357,
+            0.001970208852236451,
+            0.013107754130616248,
+            0.009666887546609378,
+            0.0235853683657408,
+            0.006473544089109194,
+            0.008973300312205858,
+            0.014118601565410305,
+            -0.003902202854013708,
+            -0.018303438244719996,
+            -0.014584226585460755,
+            -0.011977613285028946,
+            -0.01756345491020528,
+            0.0004563602312336091,
+            0.020461659996271687,
+            0.012417744207683839,
+            0.024136626592375227,
+            0.002119213701333967,
+            0.013759902247467922,
+            -2.9525392347047554e-05,
+            -0.000892319792935443,
+            0.00045783825646242434,
+            -0.005238483120312843,
+            0.019255756091120846,
+            -0.004599752813817219,
+            0.01656589522110913,
+            0.0006443773300684925,
+            0.007133940816771615,
+            0.013554761340256566,
+            -0.01704388669741371,
+            -0.003443100892108368,
+            -0.010029620309829285,
+            0.000941391966877521,
+            -0.019633111750884034,
+            0.01934571074127927,
+            0.010273330977233852,
+            0.0017121345993533963,
+            -0.018327189396014198,
+            -0.0029706385099949693,
+            0.03448635131998772,
+            0.019458999453011265,
+            0.0050734354363614525,
+            0.02632443373898792,
+            0.013018762352749236,
+            0.014443611411502168,
+            0.010153370950346075,
+            0.02356766253713842,
+            -0.023479811300626968,
+            0.02095255383200973,
+            -0.011590347850340786,
+            -0.007414136601134069,
+            -0.0035774888741646705,
+            0.0025141508183687115,
+            -0.013588691733097813,
+            0.00982877622088977,
+            -0.0017916539399655826,
+            -0.025386211797192578,
+            0.009164316443206016,
+            -0.008811052979979271,
+            0.010464442616928296,
+            -0.007774569252594923,
+            -0.0002806136360286729,
+            -0.0127084026190163,
+            -0.015184317235990752,
+            0.001259201880504347,
+            -0.009158429132281081,
+            0.023908150915412617,
+            -0.021210848953875233,
+            -0.01321384014436847,
+            0.013997502713857512,
+            -0.006869443694669701,
+            -0.011364239609894453,
+            -0.00374449980963666,
+            0.023673050288879224,
+            -0.009243341446512421,
+            -0.01359367519009039,
+            0.014464166953530989,
+            0.003834735113400469,
+            0.0059713317014342395,
+            -0.007508862694272452,
+            -0.008428705474135382,
+            -0.003280788036976103,
+            0.005998840043285335,
+            0.026615077617284676,
+            -0.014059930972065399,
+            -0.0032833264512892022,
+            -0.009189999799498163,
+            0.004118204537632102,
+            -0.02359889635281631,
+            0.02771075351384059,
+            -0.008871395303200375,
+            0.021705177758560575,
+            -0.0015924449308442096,
+            -0.006239188158271628,
+            -0.024231036074349822,
+            0.027522945599928003,
+            0.003683870966132199,
+            -0.010221180899173243,
+            0.0012768725460662862,
+            -0.052005292736885915,
+            0.016636109796947025,
+            0.008429895913163985,
+            -0.019507994429834353,
+            0.003697519981610208,
+            -0.004773584856005853,
+            0.018983847580558853,
+            -0.022966993228850697,
+            -0.014078986007109447,
+            0.008283766047449966,
+            -0.01153285124362746,
+            0.0031439180465649996,
+            -0.02527749501821793,
+            -0.005754537200570427,
+            0.0011760345167229269,
+            -0.005276936074280004,
+            -0.001705933750221904,
+            -0.020022761608132548,
+            -0.025972227850795115,
+            0.017666017379157135,
+            0.015820875579600418,
+            -0.002278262448951789,
+            0.01320818011561616,
+            0.008709899932124187,
+            0.00836573261091119,
+            0.016034263394940008,
+            0.007124509234103267,
+            0.0081473711608784,
+            0.045498533239850446,
+            -0.019846323044145485,
+            0.007371629414298193,
+            -0.0014854149174929534,
+            0.013951741898629716,
+            -0.030467924817555014,
+            -0.0055144480998344705,
+            0.0062802744565489945,
+            -0.0049373689340857915,
+            0.03510454786147091,
+            -0.017326094216003073,
+            0.01694160526735264,
+            -0.0026075809870445238,
+            0.0007420394265014909,
+            -0.02042003877666893,
+            -0.026156937726134134,
+            -0.014294938857971834,
+            0.0011857406889443987,
+            -0.0010047941261608975,
+            -0.011946583993069242,
+            0.006797001710574735,
+            0.004917161312925866,
+            -0.0020713379451245515,
+            0.015652523103715316,
+            -0.02952061566240878,
+            0.007356800041369564,
+            -0.010733578620243498,
+            0.020008504830359524,
+            -0.00990331048284697,
+            -0.0013319400098308093,
+            0.007236960885181047,
+            -0.004122868383988748,
+            0.013216043472071927,
+            -0.0038337251451538677,
+            0.004227115653354611,
+            -0.005221321656766545,
+            -0.01794343569722877,
+            -0.0016435261495179767,
+            0.021429008432348098,
+            -0.008349684096540855,
+            0.031864497493526205,
+            -0.01264743878037828,
+            0.009174982784686633,
+            0.010692990337462762,
+            0.0006628831036334152,
+            -0.005286285851702483,
+            0.009052971814211658,
+            0.0021431970274545893,
+            -0.008690520448278274,
+            0.006934310986093342,
+            -0.018349449937310143,
+            0.008494451785507706,
+            0.03394632739506833,
+            0.0018511328780472266,
+            -0.0007519661670070526,
+            -0.007266663179613392,
+            -0.01335844201823745,
+            0.018540713029383066,
+            0.005488273906733518,
+            -0.002546531152114063,
+            -0.027536660869590543,
+            0.02007746621157705,
+            0.0012265798861291223,
+            -0.004755325069578531,
+            0.012466557901411086,
+            0.0020495867632654885,
+            0.019989531151046736,
+            -0.010171678639113316,
+            0.030757957899447096,
+            0.010050036220124393,
+            0.002959833051227505,
+            0.0038556928367700113,
+            -0.036655749074851536,
+            -0.01652152778069314,
+            -0.00019357502472740266,
+            0.014271581054865031,
+            -0.012430647252826173,
+            -0.022432580814844155,
+            0.0010346169306977334,
+            0.01148004051449691,
+            -0.0033898642762881046,
+            -0.009650665305271706,
+            0.01560757541276278,
+            -0.020342213902670575,
+            0.006701302394408393,
+            0.007693672045981921,
+            0.0287100214318104,
+            -0.0020735384554022153,
+            -0.0012324838849990056,
+            0.000626416803904792,
+            -0.0055669860801192935,
+            -0.0008037366126870824,
+            0.017251369734470945,
+            -0.013661316947002466,
+            0.0175926211217534,
+            -0.012735241490550407,
+            0.0064713656749838585,
+            -0.0021092378432037563,
+            -0.010629579860050812,
+            -0.020114807971202787,
+            0.0029609495764307357,
+            0.009378003163375899,
+            0.006920144698428806,
+            -0.010438984881449953,
+            0.010715409812532968,
+            0.012197673689706352,
+            -0.00753962861484439,
+            0.014519905132928041,
+            0.022173989895384192,
+            -0.007687503607783572,
+            0.013633321414009471,
+            -0.0041446334728745095,
+            -0.014056264816568229,
+            -0.009985463824440833,
+            0.023929750617160402,
+            -0.0005563806942667597,
+            0.008756997373618809,
+            0.00388125051641897,
+            0.0007856447663040696,
+            -0.011226530517316183,
+            -0.001653447552601681,
+            0.019294448360333633,
+            0.011831934675073843,
+            -0.0013888534822008192,
+            -0.0010525764356699885,
+            -0.0014566936525653407,
+            -0.009427557739483988,
+            0.02049188741459728,
+            -0.022030783927622997,
+            -0.0019604940727076573,
+            0.00461018148439991,
+            -0.022589816803875618,
+            0.0001863117949313752,
+            -0.024570217962989905,
+            -0.01569952241187498,
+            0.011763033781122343,
+            -0.0022530631823737554,
+            -0.016028810867854907,
+            -0.0003242701040942633,
+            7.424127301806556e-05,
+            -0.007815789940508527,
+            -0.004135500768181258,
+            0.006115909992703774,
+            -0.00034193217510010117,
+            -0.0032262189374874842,
+            0.013430154310506884,
+            -0.009996622196870008,
+            0.015431528911575157,
+            0.0067029980830614145,
+            -0.015702167721800483,
+            0.0021406610839557003,
+            0.018500219544264753,
+            -0.0058910087864916085,
+            0.00787021425357441,
+            -0.014681907249972176,
+            -0.0037495551894426303,
+            -0.005787661351944086,
+            0.006051351977821761,
+            0.024849195780885833,
+            0.012969434450643372,
+            0.01139340445077682,
+            0.017405361843075933,
+            0.004325000715054734,
+            -0.01572297405297993,
+            0.0007416798945675923,
+            0.01682727189828363,
+            0.010464036837245312,
+            0.009804326468113703,
+            0.0008975556667684365,
+            0.0043704860361487635,
+            -0.005142176468800175,
+            0.009317064072855551,
+            0.02915482449687158,
+            0.0013434689612997423,
+            0.008696141010760192,
+            0.017276730345147134,
+            0.027038741396131265,
+            -0.0035088997194497725,
+            -0.007469614319788499,
+            0.008739942310300076,
+            -0.009657991604173173,
+            -0.001081629174976152,
+            -0.017000351183542026,
+            -0.012310275736750868,
+            0.016510505259358398,
+            -0.006084627062230492,
+            0.02848913486719956,
+            -0.0010655806791907225,
+            -0.02026498848361482,
+            -0.01565505608194376,
+            0.001404549479440637,
+            -0.01362132895080652,
+            0.005287566256051299,
+            0.014187265685427415,
+            0.01917046513020831,
+            0.004687191682875024,
+            -0.022258337721085656,
+            0.037364962197399806,
+            0.01757108963349427,
+            0.019377870795760123,
+            -0.019422840506328382,
+            -0.006159311237412833,
+            -0.009077095160676276,
+            0.02116697012944498,
+            -0.00880692095488465,
+            -0.009875674856043954,
+            0.024014332004451508,
+            0.017812606845656484,
+            0.005679806286464216,
+            0.006736609904079586,
+            0.0020048663439319585,
+            0.0025414630253661444,
+            -0.015831593851012047,
+            0.017957902498542212,
+            -0.005858125698347523,
+            -0.0058068554681761945,
+            0.017529514573571298,
+            -0.005989581792993925,
+            -0.0030306021203732347,
+            0.00015839997806822045,
+            -0.019697052145531482,
+            0.011618112886207201,
+            0.008256007601576028,
+            -0.017229774429089857,
+            -0.0014757560402567304,
+            -0.0002404480326945295,
+            -0.014118694740134205,
+            0.02004885146995733,
+            0.01694893356262835,
+            -0.0013248884139405398,
+            -0.0051752815372013315,
+            0.008223868147393114,
+            0.007000679132597393,
+            0.005256194890343006,
+            -0.019496098876977206,
+            0.005230597149342019,
+            0.005103909294824105,
+            -0.0014716649591412774,
+            0.010274346323051634,
+            -0.009030594825985526,
+            0.033740380586893826,
+            -0.006505788197604522,
+            0.013065658941136721,
+            -0.0043634731684193745,
+            -0.0055529072007985125,
+            0.001816020676232658,
+            0.013054468478598695,
+            0.009753501030700457,
+            -0.00970446155762939,
+            -0.004172354962129865,
+            0.018697982092775737,
+            -0.011983156146516074,
+            0.0013300791598046338,
+            0.01564072865804035,
+            -0.004858814041438713,
+            0.004708235083775013,
+            0.008635204790345423,
+            -0.004489505237783172,
+            0.01801512858970922,
+            -0.016482572999490366,
+            0.0023953436660054026,
+            0.010694869684962088,
+            0.02390864159830096,
+            -0.019152187046410703,
+            0.021786596486708207,
+            -0.017951050280323393,
+            0.026538582634436314,
+            0.014484894489694262,
+            -0.0035222259457311444,
+            -0.01946200885147006,
+            -0.00850690913394181,
+            0.011654565129063417,
+            -0.021423803876241532,
+            -0.0007985948078290641,
+            -0.0007290222944686444,
+            -0.021367628543092133,
+            -0.011189364017231132,
+            -0.009961779129778331,
+            0.010701637745810138,
+            0.001918633898887013,
+            0.010380983868185359,
+            0.008002182827196827,
+            0.010784760574518087,
+            0.014405640442760822,
+            0.0008958107769384412,
+            -0.010743001445440941,
+            -0.023795589003655273,
+            0.016987708698679597,
+            0.00438463274382093,
+            0.0007862212695139549,
+            -0.01627468994515667,
+            0.011936651269737248,
+            0.013250643815768723,
+            -0.009385605548926286,
+            0.009121882469425302,
+            -0.03716901103204463,
+            -0.004552808410398574,
+            -0.0035704811925430366,
+            -0.0020309197365173475,
+            -0.00908678158053104,
+            0.01559445415882485,
+            -0.013682507836005851,
+            -0.0009590922904844632,
+            0.00011696985057536548,
+            -0.0074756526903548915,
+            -1.5467890432254378e-05,
+            0.020649928837031005,
+            0.008438785939908428,
+            -0.0037382105257999757,
+            0.010376213224654074,
+            0.008826031303751312,
+            -0.013900359449287676,
+            -0.01527474712786084,
+            0.019035366035884273,
+            -0.018776918681908793,
+            0.0063098418544296165,
+            0.01677916431403883,
+            0.020778813096283988,
+            -0.02891862501830075,
+            -0.0007231469961304435,
+            0.022093939332021185,
+            -0.0012807635799950273,
+            0.006370841109790851,
+            0.008781853138377092,
+            0.006754082980602544,
+            0.0029102159062355374,
+            -0.003711173531879644,
+            -4.843770941622283e-06,
+            -0.003940474588075707,
+            0.007155539022862352,
+            0.004366433868349658,
+            0.036453728071874844,
+            -0.013661129348165794,
+            0.002541308805735001,
+            -0.0058838191638835155,
+            -0.00347401304572066,
+            -0.00593469688967835,
+            0.004407665856950772,
+            -0.010055727994798024,
+            0.004330827357536078,
+            -0.001570861535288093,
+            -0.0003731129422346693,
+            0.004883969598300302,
+            -0.001701673379646408,
+            0.007036892564844683,
+            -0.011863650867472625,
+            -0.0003979727750011258,
+            -0.020174851412609312,
+            0.021117284557875267,
+            0.00011666551282792189,
+            0.003662462447326642,
+            0.03723741246290771,
+            0.00476064510662738,
+            -0.00238355131307445,
+            -0.012880057949435798,
+            -0.013551338766321525,
+            -0.012986444917747435,
+            -0.020703821629987552,
+            -0.0036665673548525388,
+            -0.021006311026247164,
+            -0.0052061406234528915,
+            0.010362862594785442,
+            -0.011540184420787572,
+            -0.004023659825622843,
+            -0.015881589631889473,
+            -0.006531981517815174,
+            -0.03114211633687585,
+            -0.01124607488650473,
+            0.007493228366452009,
+            -0.019149285949049886,
+            -0.0039932870926865805,
+            0.026850121590606017,
+            0.001095360635760988,
+            0.01387696424607248,
+            0.0038058248676879625,
+            0.008117686785954085,
+            -0.0045937472017383955,
+            -0.0005669025036141417,
+            -0.01928244877716143,
+            0.003391980330529026,
+            0.008525354854576925,
+            -0.00361507909530282,
+            0.0070008404454106905,
+            -0.008835727374865623,
+            -0.016950017602980382,
+            0.0021654956715219517,
+            0.009334379529385142,
+            -0.006639140003365139,
+            -0.017587472740268683,
+            -0.011049929670847685,
+            0.010401189523318744,
+            -0.02283429470064187,
+            0.006257145380266984,
+            0.016439581169238607,
+            -0.009363030375624585,
+            0.004651575425738153,
+            0.019809183749755425,
+            -0.011810674576781702,
+            -0.005811328501528397,
+            0.008471992070998505,
+            0.013891259999721588,
+            0.0095606134736748,
+            0.02954498207274139,
+            0.0026596809804008996,
+            0.000561870602466584,
+            0.004889263753490895,
+            -0.004277061142504552,
+            -0.017423200644235665,
+            -0.006243787897080996,
+            0.0038507494787521856,
+            0.012873996671952184,
+            -0.009879417960879032,
+            -0.00048575277472139055,
+            0.01682604801673206,
+            0.004530939990187663,
+            -0.0033901248079550284,
+            -0.011868399508738274,
+            -0.020080331108085103,
+            0.0015580259874809898,
+            -0.01362200460317948,
+            0.015354347040143093,
+            0.003390601069331352,
+            0.0015284610287850575,
+            -0.004239880336806429,
+            0.01828645373058079,
+            -0.005013418527935815,
+            0.012749833447770729,
+            -0.006148890994661865,
+            -0.004587831231448011,
+            0.007986533783001116,
+            0.009166578151006778,
+            0.0004939857995863664,
+            0.05421019830078428,
+            0.002838056604877719,
+            0.01333205648597435,
+            0.013419127695838135,
+            -0.014523733369484934,
+            0.014086525147734823,
+            0.012688569853552613,
+            0.018497344547821693,
+            0.013037819487160016,
+            -0.012353776077273462,
+            0.017354801119939964,
+            0.0024084723134891487,
+            0.003909681065813095,
+            0.0022202438064449653,
+            -0.004528576079628071,
+            0.019594683950225115,
+            0.004070222328547378,
+            -0.0074362443729372105,
+            0.004453618452623641,
+            0.005007543168224267,
+            -0.013105524222311019,
+            -0.0112958915514782,
+            0.027105208848863218,
+            -0.01219417557128091,
+            -0.018749971743148304,
+            0.006706594269933445,
+            -0.02008458683853808,
+            -0.03763686535076864,
+            -0.0021332074289244973,
+            0.009992361098878196,
+            0.002795183524495193,
+            -0.00253695604043486,
+            0.01614298064601032,
+            0.016730443047719683,
+            -0.016985296937992726,
+            -0.018245902728910347,
+            0.0025257780765481825,
+            0.0016333325293570182,
+            0.01861807236979066,
+            -0.023168663546030196,
+            -0.006378864850313035,
+            -1.3795465857380962e-05,
+            -0.006829008132963306,
+            0.004850306884033312,
+            0.004703280112191218,
+            -0.0071714759384722965,
+            -0.0027773519096287217,
+            0.002256846645341137,
+            0.01975383140755998,
+            0.013198754584066554,
+            0.01572971337890945,
+            0.002600741959577911,
+            -0.005033508062449482,
+            -0.016817359058724685,
+            -0.005450011991586895,
+            -0.013478330855299777,
+            -0.009091854345715177,
+            -0.003042052092498012,
+            0.008217376329142566,
+            -0.025053469059115735,
+            -0.011965083373220982,
+            0.0119294363235832,
+            0.00391377331459168,
+            0.0003099203772539202,
+            0.007607672728136279,
+            -0.003830132495364688,
+            0.010331053246451333,
+            0.013823890473448383,
+            0.0076112887952959766,
+            -0.004723377809559611,
+            0.009600411526173948,
+            0.011460781436250457,
+            0.006015477068316837,
+            -0.005072711638895421,
+            0.023859899048048694,
+            0.01821920335673539,
+            -0.01680365156728093,
+            -0.002149107383573353,
+            -0.0027829676356423997,
+            -0.01884308738652242,
+            0.009665830801645276,
+            0.00880322398336952,
+            0.008308268605449346,
+            0.003906369351868086,
+            0.00974531700889854,
+            -0.013875904819275925,
+            -0.0050570811397453235,
+            0.002870120865445989,
+            -0.005729556771404781,
+            -0.01901337075686543,
+            0.000297242372701423,
+            -0.011987192261104132,
+            -0.011327157550660895,
+            -0.01409710922165158,
+            -0.003221240464411861,
+            0.005283472511900512,
+            0.021472187640986972,
+            -0.004410881709548286,
+            -0.001855852749039012,
+            0.01351697599517034,
+            0.010060125186506398,
+            0.0021658738823899905,
+            0.0002206474541239167,
+            0.005418828324605768,
+            -0.0027919130944783397,
+            0.0008094042430809068,
+            -0.0015826797905407809,
+            0.021089276765434876,
+            -0.02016611086095107,
+            -0.009413977569213423,
+            -0.009288078098692112,
+            0.005842197094339927,
+            0.028793032462603636,
+            -0.007165634502373454,
+            -0.010104002083017078,
+            -0.01994315293556678,
+            0.0003786527922888834,
+            -0.07810609175343784,
+            -0.005210636756258548,
+            0.001870513618910238,
+            0.03126380113105983,
+            -0.010817888476436834,
+            0.0033673768058629823,
+            -0.002204487854177595,
+            0.014542299655712928,
+            -0.001760107965206874,
+            -0.0013283746982194259,
+            0.0071954806737858056,
+            -0.005640400818010697,
+            -0.006210989097047217,
+            0.005904163661619133,
+            -9.945941028848348e-05,
+            -0.010266938705916303,
+            0.00057263080955709,
+            0.02209498395848068,
+            0.0159398346608872,
+            -0.006575439759804175,
+            0.01816748552021211,
+            -0.00719294553484007,
+            -0.009838738592324234,
+            0.005639288780502127,
+            -0.0019007683459126327,
+            -0.015369619176147573,
+            -0.011475977270803784,
+            0.009862808084641291,
+            -0.030069890644150284,
+            -0.00010725753918890802,
+            0.00943698424646706,
+            -0.013540245857270492,
+            -0.008293042205008774,
+            0.007212974960781723,
+            0.006708343474532583,
+            -0.0012017968151025369,
+            -0.0078026000239507145,
+            -0.02527526731750743,
+            0.0027578400032925625,
+            -0.002446811971857853,
+            -0.004899702349784173,
+            0.0019960465173086183,
+            0.001479503476855694,
+            -0.012645905858466798,
+            0.01216748847452763,
+            -0.013378877368040743,
+            -0.012053251571418766,
+            0.012508842412556306,
+            -0.00027167241745418584,
+            0.01682680930386839,
+            -0.020607299598265426,
+            0.0010702151063620052,
+            -0.028075439824932304,
+            -0.007671848153932823,
+            -0.015486630894165054,
+            -0.0009776136330459065,
+            -0.0012939921569540803,
+            -0.03481880531471923,
+            0.012889447665794033,
+            -0.005200149829198305,
+            -0.006094702500154477,
+            -0.00459778303079732,
+            -0.024583504817899775,
+            0.006072281238752663,
+            0.00607350167867964,
+            -0.006506508970816635,
+            -0.016816439635445445,
+            0.008428048142588516,
+            -0.0011040593013238872,
+            0.004873670535200424,
+            -0.007475800310154908,
+            -0.0005638920132076605,
+            -0.013047135093134946,
+            -0.005463454269764494,
+            -0.0008522562324122834,
+            0.01827123304871433,
+            0.0044055851550317835,
+            -0.014526381222031981,
+            0.016862445236481526,
+            0.013250692156820124,
+            0.01602714293656341,
+            -0.0071089293427753115,
+            -0.0004265739315250755,
+            -0.02750630108787212,
+            -0.00327225092954903,
+            -0.023687644154808972,
+            0.00041213204365507744,
+            0.009447660230507965,
+            0.0203905264889617,
+            0.012694421477479161,
+            0.004620617778996077,
+            -0.001557511764988156,
+            0.0066751090724502,
+            0.02696959560423252,
+            -0.01099183126742224,
+            -0.006969111598423376,
+            -0.01884504192789845,
+            0.01113023717919988,
+            0.025135748541199255,
+            0.003381303139273845,
+            -0.011317957771935042,
+            0.008112598754655146,
+            -0.0009247845526290428,
+            -0.010302980598863016,
+            0.02889587382004902,
+            -0.0391134481322538,
+            -0.015599591108049343,
+            -0.004605824192962023,
+            0.007406520608110251,
+            -0.015596917115435033,
+            0.00770468362805995,
+            -0.012614339656105264,
+            0.001699049329817941,
+            -0.005851477164035241,
+            0.03535087340659861,
+            -0.0031950960727572925,
+            0.0006909856233673865,
+            -0.003395233050301427,
+            0.003531135044131077,
+            0.012355948349937309,
+            -0.004444221317023453,
+            -0.030726300006909638,
+            -0.0022983766869702346,
+            -0.011198252640110788,
+            0.009274666307515955,
+            -0.016207714291658314,
+            -0.017342435976846468,
+            -0.009424737744066259,
+            -0.021827243671837953,
+            -0.012403084982789827,
+            0.029747453318486338,
+            -0.005824001042844527,
+            0.01747569398142269,
+            -0.005219060548639168,
+            -0.008365003057800934,
+            0.00021272411948388247,
+            0.01344335658390732,
+            -0.028498240040284714,
+            0.00041963499785238845,
+            -0.002322931702641349,
+            0.0126393550661767,
+            -0.001595134597082556,
+            -0.00983073906977773,
+            0.006903313245694627,
+            0.0010522208685011098,
+            -0.015648940549713764,
+            -0.0007573245570947632,
+            -0.008533504878052644,
+            0.020475329445149498,
+            0.00027791387213409814,
+            0.001074213629646005,
+            0.013864331671732542,
+            -0.02102352993295736,
+            0.01576938071930521,
+            -0.021722775924283996,
+            0.01893267442161547,
+            -0.006577103101038035,
+            0.005772016664149475,
+            0.008283223343661555,
+            0.014068789332250016,
+            -0.014030042849491258,
+            -0.00663752723080222,
+            -0.034436220972634726,
+            -0.0122621879114314,
+            -0.001946072976826902,
+            -0.005308941654794638,
+            -0.014773123927909993,
+            -0.0011054822266082056,
+            -0.026205461509573584,
+            -0.006460373065662098,
+            0.012822188046808264,
+            0.019900312098359022,
+            -0.008409070094265829,
+            0.014163132434889265,
+            -0.006743711872055091,
+            -0.0028230771810882492,
+            -0.012652021749331811,
+            -0.018061180380009395,
+            -0.01083672324723357,
+            -0.01890966636827258,
+            -0.0018635408070944168,
+            0.004709222372611061,
+            -0.015760618199235518,
+            0.011435131737642627,
+            0.03198874888333405,
+            -0.01567271472650171,
+            0.008243406188694909,
+            0.016411496328333205,
+            0.0034236971031973503,
+            0.011728079139322837,
+            0.0038311415233052795,
+            0.004621789116267879,
+            -0.011109450058055078,
+            -0.011473158133621661,
+            0.006524490579223986,
+            0.00954511918454185,
+            -0.004913142601134428,
+            0.024347072255007807,
+            0.0008745419206472714,
+            0.005045794664611216,
+            0.005304506902446091,
+            0.011488187043525824,
+            -0.013135271908307747,
+            0.019746484188715552,
+            -0.002055210086078267,
+            -0.015571665869889405,
+            -0.01535499169234956,
+            -0.01105419264826655,
+            -0.018626399757822328,
+            0.0017108083455869296,
+            0.0181439079855921,
+            -0.0012493542718277685,
+            0.012137753522284387,
+            -0.00924460959150499,
+            -0.005019437549649743,
+            -0.0027156377206998474,
+            -0.038211954658821295,
+            -0.0021878970332906457,
+            -0.018809487178512628,
+            -0.02498945889484935,
+            0.03505216568400238,
+            -4.26477081213836e-05,
+            0.00997643255528116,
+            -0.010310197159232386,
+            -0.0002454902098986129,
+            -0.011065403358802515,
+            0.014954998647970473,
+            0.004624123403100598,
+            0.008872923678983478,
+            0.008560280683876702,
+            -0.016419500515801093,
+            -0.002997557798375716,
+            0.007472390947008284,
+            -0.00374822278027057,
+            0.012680407710694952,
+            0.015353684671120731,
+            0.020886474047003733,
+            0.011161262397502876,
+            0.002045862158732102,
+            0.00887271411363814,
+            0.009563361150297699,
+            -0.0003513237086717377,
+            -0.010459621030424415,
+            0.015419039131459582,
+            -0.02204848739950703,
+            0.012172257374410352,
+            0.04346634265794738,
+            0.019197433209794286,
+            0.029488039318675732,
+            -0.02272810439246353,
+            0.01448092574459596,
+            -0.003801043409441206,
+            0.009476229966122152,
+            0.0049309968883087685,
+            0.02346191840355209,
+            0.003973840862678076,
+            0.0073546443568171405,
+            0.010647581332452525,
+            -0.00972051412996416,
+            0.025644022498404723,
+            -0.00846669916801108,
+            5.1518540791212095e-05,
+            -0.004407265804145216,
+            -0.026400029105519137,
+            0.01615193008014211,
+            0.0081722536401398,
+            0.0010702314763197463,
+            0.0010263623479474385,
+            -0.0019566604229607643,
+            0.007982949510899019,
+            0.007079322685771456,
+            -0.015274148711633536,
+            -0.003886053737590683,
+            -0.018527455865296014,
+            0.0012864569064749438,
+            -0.024337133883771334,
+            0.010466651250597847,
+            -0.017568706814132484,
+            -0.014822354159709854,
+            -0.021666681078831483,
+            -0.013232016867264097,
+            -0.02011653224168991,
+            0.0035858355517491515,
+            -0.008632755352062642,
+            0.013025728891203198,
+            -0.016962939287139683,
+            -0.021494971929779273,
+            0.008430887589398292,
+            0.00870802359711456,
+            0.007481282677955534,
+            0.009728368603458082,
+            0.014996157358749115,
+            0.01736292711410622,
+            -0.022712386143092265,
+            -0.023163001611599246,
+            -0.006682209717364464,
+            -0.003531495447167166,
+            0.006762379833125297,
+            0.00933681196321128,
+            0.012884185500343249,
+            -0.007077895849501418,
+            -0.007554762054641417,
+            -0.013283067570021503,
+            0.006197966297581871,
+            -0.003993067576677973,
+            -0.009464789349849612,
+            -0.0015662739825834614,
+            -0.0061389795593180686,
+            0.002753664732282926,
+            -0.0001791088888074743,
+            0.0007595261237141144,
+            0.017122834676182343,
+            -0.008176319370751018,
+            -0.009787747261447871,
+            -0.02027228453943018,
+            -0.010683273300243025,
+            -0.015604745521156111,
+            -0.0016279406232527138,
+            -0.006506713191861036,
+            -0.014420323628589382,
+            0.010888206569219648,
+            -0.011930708571093279,
+            -0.00887210117616784,
+            0.010807147705360856,
+            0.0073900910661282935,
+            -0.003431679006289002,
+            0.037804020313161775,
+            0.0011945597874820288,
+            0.00778778394048267,
+            -0.003366543306746973,
+            0.005296727102519836,
+            -0.017761251541598876,
+            -0.026382400396890007,
+            0.0010357716464853195,
+            -0.01775777668381489,
+            0.010807713750049491,
+            -0.008515368407459745,
+            0.0016051389544388124,
+            -0.00034958600424912995,
+            -0.009585854918521076,
+            0.0036549720706351473,
+            -0.008110927907950148,
+            0.004212219431447508,
+            -0.03507890639477636,
+            0.011409377585617926,
+            -0.01819120811524444,
+            -0.008962242411035718,
+            0.01710177594046791,
+            0.037560936862121835,
+            0.0028513388998398104,
+            0.00030058353811717895,
+            0.0014792184333546447,
+            0.004427921639326168,
+            0.021438127785303025,
+            0.006339483221420334,
+            -0.001187747087595264,
+            -0.02237982599829809,
+            0.0062435650294483695,
+            0.002200578793697352,
+            0.010092828474934744,
+            0.023105023225115266,
+            -0.016797326021070008,
+            -0.015378520287395565,
+            -0.012732527281000097,
+            -0.010797463778182385,
+            0.009881499418260473,
+            0.02801767062785128,
+            0.01811139137849365,
+            -0.017358098300028903,
+            -0.020207173948076473,
+            0.0016242901745373126,
+            -0.026331540602868762,
+            0.007269751661479859,
+            0.11940631114242901,
+            -0.0045382135734049305,
+            -0.005994407567723939,
+            -0.008679712231646732,
+            -0.004482553208064535,
+            0.016709571294385293,
+            -0.013387043582971358,
+            0.069179848974036,
+            -0.007221545718220735,
+            0.0037392229364642915,
+            0.0028916224964057385,
+            0.00462729158956839,
+            -0.009727339449635306,
+            0.011570145487622916,
+            -0.011287365791681587,
+            -0.012991994493845187,
+            -0.013076194601468383,
+            -0.0180830261130804,
+            -0.007272768410218907,
+            -0.004155564814758654,
+            0.004204993170487392,
+            -0.005525575983182816,
+            0.008051108079520902,
+            -0.017551645696802648,
+            -0.010800070994485792,
+            0.013007176804761548,
+            -0.0023627485852619817,
+            0.005390986239687555,
+            -0.029668390278675554,
+            -0.0007650391259819867,
+            -0.0002446394046947245,
+            0.021280292985495608,
+            0.003137771886236958,
+            -0.0019077298815862946,
+            -0.010810159675564049,
+            0.007040107942856957,
+            0.023835993694726953,
+            0.021875366566457236,
+            0.0180511566130312,
+            0.012135351541044354,
+            0.020483975362807574,
+            -0.008667079096167644,
+            0.026324312888512866,
+            -0.001552207146726937,
+            -0.00397810465151493,
+            0.0077577136352257004,
+            -0.009849600842081875,
+            -0.00816392268697345,
+            0.00677353736462267,
+            -0.017288900282504148,
+            -0.020418071729964713,
+            -0.020383762873109595,
+            0.025741408097358233,
+            -0.011755094311186617,
+            0.002737129201159793,
+            -0.008266210469668866,
+            0.027287436144516414,
+            -0.006283892307996516,
+            -0.0029440642954701632,
+            0.009428876742171362,
+            -0.03788574758692254,
+            0.008834605607057885,
+            0.006463342392514337,
+            0.026711196829464153,
+            -3.9827332346130654e-05,
+            0.005565561841469247,
+            -0.0006738085601962865,
+            -0.006104320995853506,
+            -0.010792830169647851,
+            2.4565453208679464e-05,
+            0.017983908748172493,
+            0.0035181716362611077,
+            0.016970206528324653,
+            -0.005449808667944701,
+            -0.023171388778833197,
+            0.007092509502738752,
+            -0.01555320330471131,
+            -0.023306285898839978,
+            -0.002028783456704567,
+            -0.03196436161255752,
+            0.0033412618908711163,
+            0.007527633574188688,
+            -0.01890570959143866,
+            -0.017859735533232564,
+            0.010329340599904344,
+            0.004382396539605985,
+            -0.0016746397858953155,
+            0.006643785064338525,
+            -0.03307812909454471,
+            -0.01927475592253288,
+            0.004686970010943116,
+            0.012733525168279355,
+            -0.012993441126970916,
+            0.015291832952199082,
+            -0.020491167895268936,
+            0.016451976170508816,
+            -0.012259387189535237,
+            0.018393274368854132,
+            -0.006997668664226754,
+            0.011843450716200625,
+            0.011457415642860215,
+            0.009560877634897626,
+            -0.01790001115302499,
+            0.0214879415944361,
+            0.0066829070506954115,
+            0.03900547021123392,
+            0.02512484499576633,
+            -0.012651294056575184,
+            -0.0018425680226742782,
+            -0.00011361458320146007,
+            0.014982647266052349,
+            0.0061874424115559,
+            -0.003689464024837091,
+            0.02705902205734107,
+            0.018239469373522537,
+            -0.021070704644859155,
+            0.020037508334843183,
+            -0.00973804771680803,
+            0.018876283979591423,
+            -0.013303080480396818,
+            0.010592424494628194,
+            0.004717870260467274,
+            0.01070131238869639,
+            0.009087656015197565,
+            0.006708673669681232,
+            0.025928065153829043,
+            0.03663367449576964,
+            -0.00511073108390551,
+            0.002161381031392467,
+            0.004864092591944181,
+            0.012814518637357098,
+            0.001556833001747549,
+            0.007915766323215207,
+            -0.007012785514562176,
+            0.005834786334750152,
+            -0.002311533032619821,
+            -0.00122162928193159,
+            0.019525999280692766,
+            0.000824163362963678,
+            0.012875575199417254,
+            0.013259376059185821,
+            -0.002390133687709065,
+            -0.005682644446302278,
+            0.0019262218417782657,
+            -0.018193478045923098,
+            -0.015131696847666434,
+            0.029463723011600933,
+            -0.011447039651934148,
+            -0.008083114389964213,
+            0.0020642857121204753,
+            0.012160202510990566,
+            -0.00866944494225225,
+            -0.005378300635997713,
+            0.0038280364584418335,
+            0.014427318067525318,
+            -0.01799358399064332,
+            0.003381165206832161,
+            0.005252552821854107,
+            0.019433554824232185,
+            -0.00028791985674147885,
+            -0.013487678738391956,
+            0.0005317922815671783,
+            0.014982152801567637,
+            0.011902348165369078,
+            0.0016056390299304837,
+            -0.006935777197137747,
+            -0.00446584737370504,
+            0.004586814866182051,
+            0.006269466956110759,
+            0.0027516492952993445,
+            0.008157662045793032,
+            -0.0025001885993292637,
+            -0.016648747436203294,
+            0.017735757469839637,
+            -0.0030669873770852725,
+            0.018058248627079793,
+            -0.0038238312452176745,
+            -0.0147172470474118,
+            0.0043328417868095355,
+            -0.001888507803054499,
+            -0.018556943283494755,
+            0.009407930200720534,
+            0.002560949424273885,
+            0.010411159836451958,
+            0.0052571230238294825,
+            -0.007499470181994448,
+            -6.145889479758644e-05,
+            -0.015163819825329975,
+            0.022836682800304944,
+            0.016390944816242627,
+            0.001713572907204073,
+            -0.0035687505973373878,
+            -0.031210299824387475,
+            0.0029076511444359886,
+            -0.007822231980296341,
+            0.017821806420465356,
+            0.008250605126658402,
+            0.007908773724009905,
+            0.008351355849444162,
+            -0.00743970642051451,
+            -0.002411981861500092,
+            0.012543476434788293,
+            0.0202407777715137,
+            -0.0024635818448627447,
+            0.02535771413579989,
+            -0.012533449594824386,
+            -0.0022489292647974992,
+            0.0012362917358624392,
+            0.01578939075186126,
+            -0.0008001783766371555,
+            0.006311837812042848
+          ],
+          "keyphrases": [
+            "Switch Transformers",
+            "Mixture of Experts",
+            "sparsely-activated model",
+            "training instability",
+            "multilingual settings"
+          ],
+          "title": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "d17e0745-ef0b-4687-8d99-5321eb6cab7e",
+        "properties": {
+          "page_content": "3 Scaling Properties 11 3.1 Scaling Results on a Step-Basis 12 3.2 Scaling Results on a Time-Basis 13 3.3 Scaling Versus a Larger Dense Model 13 ",
+          "keyphrases": [
+            "Scaling Properties",
+            "Scaling Results on a Step-Basis",
+            "Scaling Results on a Time-Basis",
+            "Scaling Versus a Larger Dense Model"
+          ],
+          "title": "Scaling Properties",
+          "embedding": [
+            -0.020178575068712234,
+            0.010542738251388073,
+            -0.025771139189600945,
+            0.015961479395627975,
+            0.009152155369520187,
+            -0.016702115535736084,
+            -0.001157245016656816,
+            0.07732853293418884,
+            -0.004825475625693798,
+            0.01630912534892559,
+            0.00782203208655119,
+            0.006609050091356039,
+            0.04265463352203369,
+            -0.01061831321567297,
+            0.03403906524181366,
+            0.045133501291275024,
+            0.008094102144241333,
+            0.006782873068004847,
+            -0.0005828736466355622,
+            -0.042836014181375504,
+            -0.026269935071468353,
+            -0.020103000104427338,
+            -0.027645403519272804,
+            0.029096446931362152,
+            0.025499068200588226,
+            0.0028454058337956667,
+            0.03932932764291763,
+            -0.0032780736219137907,
+            -0.010232879780232906,
+            0.0061329263262450695,
+            -0.011638578958809376,
+            0.002119883894920349,
+            -0.04081059992313385,
+            -0.002091543283313513,
+            -0.04446844011545181,
+            -0.02823488973081112,
+            0.06620386242866516,
+            -0.015024347230792046,
+            -0.0017486208816990256,
+            0.0305474903434515,
+            0.011358950287103653,
+            0.01002126932144165,
+            -0.01330123282968998,
+            -0.00594398844987154,
+            0.01584055833518505,
+            -0.04274532571434975,
+            -0.048398349434137344,
+            0.013391923159360886,
+            0.007130519021302462,
+            0.006314306985586882,
+            0.014230807311832905,
+            -0.0615786612033844,
+            0.040961749851703644,
+            -0.01794154942035675,
+            -0.02935340255498886,
+            0.01768459379673004,
+            -0.00886496901512146,
+            0.0859743282198906,
+            0.023745723068714142,
+            0.01330123282968998,
+            0.025272341445088387,
+            -0.028113968670368195,
+            -0.03201364725828171,
+            -0.006930244620889425,
+            0.0015370103064924479,
+            -0.006253846455365419,
+            -0.035550568252801895,
+            0.003850555745884776,
+            -0.03279963135719299,
+            -0.007655766326934099,
+            -0.033464692533016205,
+            -0.0024070697836577892,
+            0.0023749503307044506,
+            -0.01073167659342289,
+            0.006182050332427025,
+            -0.01117756962776184,
+            0.030834676697850227,
+            0.010504950769245625,
+            -0.014759833924472332,
+            0.029020871967077255,
+            0.052811939269304276,
+            0.039057254791259766,
+            -0.04205003380775452,
+            0.02893018163740635,
+            0.01701953262090683,
+            -0.03422044590115547,
+            -0.033011242747306824,
+            -0.055865176022052765,
+            0.009061465039849281,
+            -0.021266857162117958,
+            -0.0681990459561348,
+            -0.033736761659383774,
+            0.012983816675841808,
+            -0.03210433945059776,
+            -0.026919880881905556,
+            0.050967905670404434,
+            -0.01117001287639141,
+            0.010421818122267723,
+            -0.02122151292860508,
+            0.0011147339828312397,
+            -0.002512875013053417,
+            0.03597378730773926,
+            0.01862172596156597,
+            -0.01503946166485548,
+            0.02306554652750492,
+            -0.011600791476666927,
+            -0.0017051651375368237,
+            0.030199844390153885,
+            -0.006102696061134338,
+            0.05272125080227852,
+            0.0037315248046070337,
+            -0.01797178015112877,
+            -0.03198342025279999,
+            -0.03736437112092972,
+            -0.060036927461624146,
+            0.009363765828311443,
+            -0.00215578218922019,
+            -0.04594971239566803,
+            0.010527623817324638,
+            -0.02753959782421589,
+            0.02655711956322193,
+            0.02265744097530842,
+            -0.007481943350285292,
+            0.005112661514431238,
+            -0.032527558505535126,
+            0.04428705945611,
+            0.02823488973081112,
+            -0.02261209487915039,
+            -0.001841200515627861,
+            -0.02264232560992241,
+            0.004058387596160173,
+            0.019241442903876305,
+            -0.02985219843685627,
+            -0.007198536302894354,
+            -0.01142696850001812,
+            -0.016959071159362793,
+            0.050030771642923355,
+            0.025000272318720818,
+            -0.005399846937507391,
+            0.005161785054951906,
+            -0.011154897511005402,
+            -0.04071991145610809,
+            0.040689680725336075,
+            0.013452382758259773,
+            -0.03385768458247185,
+            0.0025582199450582266,
+            -0.028386039659380913,
+            0.062334414571523666,
+            -0.0035784849897027016,
+            -0.006261404138058424,
+            -0.02492469549179077,
+            -0.01747298240661621,
+            0.007746456656605005,
+            0.0467054657638073,
+            -0.0063445367850363255,
+            0.005543440114706755,
+            -0.018939141184091568,
+            0.0396013967692852,
+            -0.013928506523370743,
+            0.034432053565979004,
+            0.011661251075565815,
+            -0.007716226391494274,
+            0.014034312218427658,
+            0.01913563720881939,
+            0.028159314766526222,
+            0.00200652121566236,
+            -0.0012942251050844789,
+            0.022551635280251503,
+            -0.029217366129159927,
+            0.0015946364728733897,
+            0.02794770337641239,
+            0.011729269288480282,
+            0.012998932041227818,
+            0.01797178015112877,
+            -0.029459208250045776,
+            0.01653585024178028,
+            -0.02264232560992241,
+            0.026164129376411438,
+            -0.014880754053592682,
+            0.04005485028028488,
+            -0.003034343710169196,
+            -0.02941386215388775,
+            -0.011812401935458183,
+            0.021599387750029564,
+            0.030214959755539894,
+            -0.03056260570883751,
+            0.030456800013780594,
+            0.03518780693411827,
+            0.006627943832427263,
+            0.007213651668280363,
+            0.02353411354124546,
+            0.03476458415389061,
+            0.05646977946162224,
+            -0.00728922663256526,
+            -0.03286009281873703,
+            -0.027932588011026382,
+            0.010346243157982826,
+            -0.0043493518605828285,
+            0.01748809777200222,
+            0.026073439046740532,
+            -0.008320827968418598,
+            -0.02283882163465023,
+            0.0023995121009647846,
+            0.008373730815947056,
+            0.02237025462090969,
+            -0.011200242675840855,
+            -0.02078317664563656,
+            0.00010415205179015175,
+            0.017216026782989502,
+            -0.07104067504405975,
+            -0.010784579440951347,
+            -0.01410232950001955,
+            -0.024864235892891884,
+            -0.024380555376410484,
+            -0.008948101662099361,
+            0.013150081969797611,
+            0.01469937339425087,
+            0.02447124570608139,
+            0.0008743104408495128,
+            -0.02051110565662384,
+            -0.0035520337987691164,
+            -0.01797178015112877,
+            0.02985219843685627,
+            -0.04546603187918663,
+            -0.022294679656624794,
+            0.026405969634652138,
+            0.01073167659342289,
+            -0.0017287824302911758,
+            0.009650951251387596,
+            -0.039268866181373596,
+            0.037001609802246094,
+            -0.018304310739040375,
+            -0.001512448419816792,
+            0.009356208145618439,
+            0.009227730333805084,
+            -0.017548557370901108,
+            0.02332250215113163,
+            0.007973182015120983,
+            -0.021750537678599358,
+            0.03146950900554657,
+            0.022143529728055,
+            0.014631356112658978,
+            0.011661251075565815,
+            -0.03295078128576279,
+            0.015764983370900154,
+            0.013429710641503334,
+            -0.00931842066347599,
+            0.05486758425831795,
+            0.018833335489034653,
+            -0.004228431731462479,
+            0.022899281233549118,
+            -0.023216696456074715,
+            0.011381623335182667,
+            0.01455578114837408,
+            -0.0037031841930001974,
+            0.029020871967077255,
+            0.02543860860168934,
+            0.022521406412124634,
+            -0.04894249141216278,
+            0.01937747746706009,
+            0.03219502791762352,
+            -0.00419064424932003,
+            -0.0698314756155014,
+            0.025952519848942757,
+            0.0018770986935123801,
+            0.01748809777200222,
+            0.017352063208818436,
+            0.02144823782145977,
+            0.01674746163189411,
+            0.002259697997942567,
+            0.043924298137426376,
+            -0.039057254791259766,
+            0.019044946879148483,
+            0.0010344353504478931,
+            -0.026859421283006668,
+            -0.014638913795351982,
+            -0.007379916962236166,
+            -0.01726137287914753,
+            -0.010580525733530521,
+            0.026723386719822884,
+            0.02049599029123783,
+            0.033676303923130035,
+            -0.01191064901649952,
+            0.028809260576963425,
+            -0.0014926099684089422,
+            0.00100326060783118,
+            -0.0124850207939744,
+            -0.00945445615798235,
+            -0.017442753538489342,
+            -0.016732346266508102,
+            0.003291299333795905,
+            0.0034613434690982103,
+            -0.03875495493412018,
+            0.01202401239424944,
+            -0.007024713326245546,
+            -0.01751832850277424,
+            0.023458538576960564,
+            -0.033887915313243866,
+            0.0400850772857666,
+            -0.015780098736286163,
+            -0.05078652501106262,
+            0.006786651909351349,
+            0.01837988570332527,
+            -0.030381225049495697,
+            0.00044494890607893467,
+            0.004428706131875515,
+            0.030668411403894424,
+            -0.0060875811614096165,
+            -0.01933213323354721,
+            0.001051439787261188,
+            0.002350388327613473,
+            0.024622395634651184,
+            -0.0022294679656624794,
+            -0.016898611560463905,
+            -0.006306749302893877,
+            0.0045609623193740845,
+            0.020888982340693474,
+            0.0007798414444550872,
+            -0.025015385821461678,
+            0.02170519344508648,
+            0.025483952835202217,
+            -0.014412187971174717,
+            -0.05244917795062065,
+            0.03824104368686676,
+            0.033706534653902054,
+            -0.007478164508938789,
+            -0.04226164519786835,
+            0.016006823629140854,
+            -0.029081331565976143,
+            0.01798689365386963,
+            0.00021491693041753024,
+            -0.00454206857830286,
+            -0.016248665750026703,
+            0.019014716148376465,
+            0.021599387750029564,
+            -0.013271003030240536,
+            -0.005369617138057947,
+            -0.042291875928640366,
+            -0.036487698554992676,
+            0.017140451818704605,
+            -0.01304427720606327,
+            0.00782203208655119,
+            -0.012326313182711601,
+            -0.02421429008245468,
+            0.009605606086552143,
+            -0.05263055860996246,
+            -0.04640316590666771,
+            0.0015011121286079288,
+            0.024894466623663902,
+            -0.0001595344947418198,
+            0.0009215449099428952,
+            -0.010172420181334019,
+            0.01961931772530079,
+            0.020375071093440056,
+            0.024894466623663902,
+            -0.0358830988407135,
+            -0.002492091851308942,
+            0.028113968670368195,
+            0.0014963886933401227,
+            -0.018153158947825432,
+            0.014517993666231632,
+            -0.025499068200588226,
+            -0.006265182979404926,
+            0.0008100715349428356,
+            -0.0326484814286232,
+            0.004504281096160412,
+            -0.005407404620200396,
+            -0.0013424042845144868,
+            0.010247995145618916,
+            0.0316811166703701,
+            -0.021145937964320183,
+            0.004625201225280762,
+            0.040447838604450226,
+            -0.0043040066957473755,
+            -0.013134967535734177,
+            0.008260367438197136,
+            0.017911318689584732,
+            -0.017125336453318596,
+            -0.022083070129156113,
+            0.02329227328300476,
+            -0.02379106916487217,
+            0.014805179089307785,
+            0.045828793197870255,
+            -0.012696631252765656,
+            0.006484351120889187,
+            0.042291875928640366,
+            0.02540837787091732,
+            0.013679108582437038,
+            -0.01423836499452591,
+            -0.008676031604409218,
+            0.012054242193698883,
+            -0.059099797159433365,
+            9.500037413090467e-05,
+            -0.005857076961547136,
+            0.022899281233549118,
+            -0.002246472518891096,
+            0.014087215065956116,
+            -0.021372662857174873,
+            0.007104067597538233,
+            0.03009403869509697,
+            -0.02333761751651764,
+            0.014381958171725273,
+            -0.057678982615470886,
+            0.016369584947824478,
+            0.012394330464303493,
+            0.012160047888755798,
+            -0.0045118387788534164,
+            -0.0124850207939744,
+            0.027207067236304283,
+            -0.007920279167592525,
+            0.009280633181333542,
+            0.01745786890387535,
+            -0.022324910387396812,
+            -0.053023550659418106,
+            0.017805512994527817,
+            -0.010323570109903812,
+            -0.030653296038508415,
+            0.007485722191631794,
+            -0.02167496271431446,
+            -0.021312203258275986,
+            -0.044438209384679794,
+            0.012046684511005878,
+            0.037666670978069305,
+            -0.016732346266508102,
+            -0.04274532571434975,
+            -0.0004893493023701012,
+            -0.022249335423111916,
+            -0.013822701759636402,
+            0.010746791027486324,
+            -0.03225548937916756,
+            0.020178575068712234,
+            0.018138045445084572,
+            0.048882029950618744,
+            -0.010316013358533382,
+            -0.004345573019236326,
+            0.02823488973081112,
+            -0.029217366129159927,
+            0.005097546149045229,
+            -0.02471308596432209,
+            -0.03446228429675102,
+            0.02888483554124832,
+            0.0050597586669027805,
+            0.029020871967077255,
+            0.015235957689583302,
+            -0.007988297380506992,
+            0.016974186524748802,
+            -0.021281972527503967,
+            -0.027690747752785683,
+            -0.018062468618154526,
+            -0.026980342343449593,
+            -0.04504281282424927,
+            0.028355810791254044,
+            -0.05734645202755928,
+            -0.019966963678598404,
+            0.028612766414880753,
+            0.009552703239023685,
+            -0.0021274415776133537,
+            0.005853298120200634,
+            -0.019453052431344986,
+            -0.036517929285764694,
+            -0.011374065652489662,
+            0.028794145211577415,
+            0.009598048403859138,
+            -0.010187535546720028,
+            0.005762607790529728,
+            0.009598048403859138,
+            -0.04407544806599617,
+            -0.016898611560463905,
+            -0.00010893454600591213,
+            0.013694223947823048,
+            -0.02170519344508648,
+            -0.04425682872533798,
+            -0.04211049526929855,
+            0.009386437945067883,
+            -0.02729775756597519,
+            0.007912722416222095,
+            -0.030411455780267715,
+            -0.04845881089568138,
+            -0.004073502495884895,
+            -0.0071909790858626366,
+            0.02382129803299904,
+            0.02584671415388584,
+            -2.5447581720072776e-05,
+            0.03201364725828171,
+            -0.032527558505535126,
+            0.011389181017875671,
+            0.006571262609213591,
+            -0.021538928151130676,
+            -0.009008562192320824,
+            -0.0022294679656624794,
+            -0.029897544533014297,
+            0.029292942956089973,
+            0.01913563720881939,
+            0.00495017459616065,
+            -0.0025941182393580675,
+            -0.027146607637405396,
+            -0.02149358205497265,
+            -0.05789059400558472,
+            0.014752276241779327,
+            0.05380953103303909,
+            -0.030502144247293472,
+            -0.01051250845193863,
+            -0.017835743725299835,
+            0.019740238785743713,
+            0.04827743023633957,
+            0.007950509898364544,
+            0.03149973601102829,
+            -0.023050431162118912,
+            0.009514915756881237,
+            0.006975589785724878,
+            0.011018862016499043,
+            -0.044861432164907455,
+            -0.0014614351093769073,
+            -0.013376807793974876,
+            -0.0042586615309119225,
+            -0.003671064740046859,
+            -0.015598718076944351,
+            -0.025000272318720818,
+            -0.0045005022548139095,
+            0.006911350879818201,
+            -0.0050710951909422874,
+            0.002527989912778139,
+            -0.018017124384641647,
+            0.03479481488466263,
+            -0.04017576947808266,
+            0.03827127441763878,
+            -0.006427669432014227,
+            0.004568520002067089,
+            0.04981916397809982,
+            -0.018077583983540535,
+            -0.007648208644241095,
+            -0.008033642545342445,
+            -0.0038524451665580273,
+            -0.01141185313463211,
+            0.010096845217049122,
+            -0.014608683064579964,
+            0.0017864084802567959,
+            0.020465759560465813,
+            -0.02584671415388584,
+            -0.020344840362668037,
+            -0.005841962061822414,
+            0.0031590426806360483,
+            -0.0405687615275383,
+            0.006170713808387518,
+            0.0012951696990057826,
+            0.030169613659381866,
+            0.014155232347548008,
+            -0.03128812834620476,
+            0.000974919879809022,
+            0.01565917767584324,
+            -0.012492578476667404,
+            -0.02330738678574562,
+            0.03225548937916756,
+            0.003327197628095746,
+            -0.004972847178578377,
+            0.003485905472189188,
+            0.01769970916211605,
+            -0.013732011429965496,
+            0.01863684132695198,
+            -0.006522138603031635,
+            0.012265852652490139,
+            0.017397407442331314,
+            -0.03618539869785309,
+            0.016173088923096657,
+            0.007950509898364544,
+            -0.019241442903876305,
+            0.008139447309076786,
+            0.019044946879148483,
+            -0.025499068200588226,
+            0.021145937964320183,
+            -0.02097967080771923,
+            0.016626540571451187,
+            -0.04537534341216087,
+            -0.008192350156605244,
+            0.02896041050553322,
+            0.009802102111279964,
+            0.0013943621888756752,
+            -0.022234220057725906,
+            -0.0013839706080034375,
+            0.04809604957699776,
+            0.005229802802205086,
+            -0.011525215581059456,
+            -0.011850189417600632,
+            0.00513911247253418,
+            0.00043975311564281583,
+            0.0009342982666566968,
+            -0.02753959782421589,
+            -0.04588925465941429,
+            -0.006393660791218281,
+            -0.036941152065992355,
+            0.0326484814286232,
+            0.027343101799488068,
+            -0.0037126310635358095,
+            -0.016868380829691887,
+            0.0189089123159647,
+            -0.0006636445759795606,
+            0.03282986208796501,
+            -0.007478164508938789,
+            0.04150589182972908,
+            -0.052842170000076294,
+            0.0018478132551535964,
+            -0.011517658829689026,
+            -0.02193191833794117,
+            -0.007406368386000395,
+            -0.014638913795351982,
+            -0.018002009019255638,
+            0.01892402581870556,
+            0.01083748135715723,
+            0.0031533746514469385,
+            0.016974186524748802,
+            -0.0028246226720511913,
+            0.0035274717956781387,
+            0.012409445829689503,
+            -0.012084471993148327,
+            -0.003589821280911565,
+            0.04458935931324959,
+            0.009530031122267246,
+            -0.00979454442858696,
+            -0.017321832478046417,
+            -0.0024864235892891884,
+            0.052116647362709045,
+            -0.008494650945067406,
+            0.021569158881902695,
+            0.03606447950005531,
+            0.007958066649734974,
+            0.03074398636817932,
+            -0.004685661755502224,
+            0.026678040623664856,
+            0.003858113195747137,
+            -0.019180981442332268,
+            -0.006310528144240379,
+            -0.020405299961566925,
+            -0.01340703759342432,
+            -0.005849519278854132,
+            -0.023186467587947845,
+            0.026058323681354523,
+            -0.006525917444378138,
+            -0.00215578218922019,
+            0.0006490019150078297,
+            -0.030623065307736397,
+            0.021418007090687752,
+            0.018334539607167244,
+            0.014722046442329884,
+            0.04099198058247566,
+            0.007164527662098408,
+            0.02403290942311287,
+            -0.002614901401102543,
+            -0.004761236719787121,
+            0.014004082418978214,
+            0.024818891659379005,
+            0.024380555376410484,
+            0.03606447950005531,
+            -0.005350723396986723,
+            0.028567420318722725,
+            0.032527558505535126,
+            -0.027464022859930992,
+            0.014419745653867722,
+            0.016883496195077896,
+            -0.006008227355778217,
+            -0.021614503115415573,
+            -0.0022559193894267082,
+            0.02073783054947853,
+            0.002458082977682352,
+            -0.00838128849864006,
+            0.0363667793571949,
+            -0.02498515695333481,
+            -0.015946364030241966,
+            0.0031514852307736874,
+            -0.03267871215939522,
+            0.02495492622256279,
+            0.0028246226720511913,
+            -0.00792783685028553,
+            -0.02608855441212654,
+            -0.0076595451682806015,
+            0.0029757730662822723,
+            -0.011381623335182667,
+            -0.0038543345872312784,
+            0.04189888387918472,
+            -0.006654395256191492,
+            0.01318031270056963,
+            -0.0022389148361980915,
+            0.004957732278853655,
+            -0.0058306255377829075,
+            0.028854606673121452,
+            0.0006863171583972871,
+            0.0049199447967112064,
+            -0.022098183631896973,
+            -0.007580191362649202,
+            0.0049652899615466595,
+            0.015538258478045464,
+            0.027826784178614616,
+            0.019649548456072807,
+            0.010127075016498566,
+            0.027010571211576462,
+            -0.02172030881047249,
+            0.00020688706717919558,
+            -0.017397407442331314,
+            0.050726063549518585,
+            0.036034248769283295,
+            -0.023005086928606033,
+            -0.012787321582436562,
+            0.010323570109903812,
+            -0.03594356030225754,
+            0.012137374840676785,
+            0.02445613034069538,
+            -0.0005446136929094791,
+            -0.0065863775089383125,
+            0.0048632631078362465,
+            -0.008101659826934338,
+            -0.026028094813227654,
+            -0.00919750053435564,
+            0.02654200606048107,
+            0.0002616790879983455,
+            0.0062425103969872,
+            0.011827516369521618,
+            0.016173088923096657,
+            -0.017442753538489342,
+            -0.029338287189602852,
+            0.000420150812715292,
+            0.037424832582473755,
+            0.01109443698078394,
+            0.004372024443000555,
+            0.019286787137389183,
+            -0.009280633181333542,
+            -0.014140116982161999,
+            0.02474331669509411,
+            0.019906504079699516,
+            -0.0013216211227700114,
+            0.004938838537782431,
+            -0.007973182015120983,
+            0.00968118105083704,
+            -0.059553246945142746,
+            0.03258801996707916,
+            0.008570225909352303,
+            0.017079992219805717,
+            -0.0009522473555989563,
+            -0.007988297380506992,
+            0.028159314766526222,
+            -0.0003171795979142189,
+            0.008056314662098885,
+            0.016943957656621933,
+            -0.030895136296749115,
+            0.008366173133254051,
+            -0.018531035631895065,
+            -0.0010495503665879369,
+            0.019286787137389183,
+            0.01013463269919157,
+            -0.013021604157984257,
+            -0.012636170722544193,
+            -0.009371323511004448,
+            0.023443423211574554,
+            -0.007153191603720188,
+            0.015077250078320503,
+            0.0004227487079333514,
+            -0.009734083898365498,
+            -0.02241560071706772,
+            0.006733749061822891,
+            0.026647809892892838,
+            -0.025090962648391724,
+            0.01700441725552082,
+            -0.01529641728848219,
+            0.01435928512364626,
+            -0.029247596859931946,
+            0.005486758425831795,
+            0.04507303982973099,
+            0.011812401935458183,
+            -0.040659449994564056,
+            -0.010890384204685688,
+            -0.0003204860258847475,
+            -0.01154788862913847,
+            -0.007617978844791651,
+            0.012318755500018597,
+            0.05692322924733162,
+            -0.03051725961267948,
+            0.008532438427209854,
+            0.028975525870919228,
+            -0.014034312218427658,
+            0.016671886667609215,
+            -0.019966963678598404,
+            0.033222850412130356,
+            0.006850890815258026,
+            0.008539996109902859,
+            0.001844034530222416,
+            -0.0016796585405245423,
+            -0.0096585089340806,
+            0.0041037327609956264,
+            -0.018954256549477577,
+            -0.017064876854419708,
+            0.010172420181334019,
+            0.013429710641503334,
+            0.006714855320751667,
+            -0.00838128849864006,
+            -0.027222182601690292,
+            -0.001923388452269137,
+            0.019710008054971695,
+            -0.016671886667609215,
+            -0.032497331500053406,
+            0.014404630288481712,
+            0.012855338864028454,
+            -0.015538258478045464,
+            0.0391177162528038,
+            0.0041490779258310795,
+            0.01037647295743227,
+            0.015167939476668835,
+            -0.006321864202618599,
+            -0.008736491203308105,
+            0.024441014975309372,
+            0.0170497614890337,
+            -0.004334236960858107,
+            -0.016278894618153572,
+            0.015455125831067562,
+            0.030623065307736397,
+            0.013331462629139423,
+            0.005947767291218042,
+            -0.012401888146996498,
+            0.0378178246319294,
+            -0.004390918184071779,
+            -0.011555446311831474,
+            -0.03171134740114212,
+            -0.014034312218427658,
+            -0.02049599029123783,
+            -0.0033555382397025824,
+            -0.0039412458427250385,
+            0.012273410335183144,
+            -0.024531705304980278,
+            0.004594971425831318,
+            0.008509766310453415,
+            0.00636343052610755,
+            0.08131889998912811,
+            0.01798689365386963,
+            0.007633093744516373,
+            0.023413192480802536,
+            -0.01154788862913847,
+            0.029474321752786636,
+            -0.02708614617586136,
+            0.017427638173103333,
+            -0.013278559781610966,
+            0.011109552346169949,
+            -0.005350723396986723,
+            0.02590717375278473,
+            0.02332250215113163,
+            -0.0007576412172056735,
+            0.002580892527475953,
+            0.013225657865405083,
+            -0.0034084408544003963,
+            -0.019740238785743713,
+            0.012076914310455322,
+            0.030880020931363106,
+            0.03654816001653671,
+            0.012961144559085369,
+            0.02169007807970047,
+            0.008441748097538948,
+            -0.01049739308655262,
+            -0.02026926539838314,
+            -0.006866005714982748,
+            0.005524545907974243,
+            0.021614503115415573,
+            0.008358615450561047,
+            0.009582933969795704,
+            0.00642011221498251,
+            -0.008751606568694115,
+            -0.006091360002756119,
+            0.00636343052610755,
+            -0.014147674664855003,
+            0.03225548937916756,
+            0.005977997090667486,
+            -0.018092699348926544,
+            -0.028809260576963425,
+            -0.009137040004134178,
+            -0.01605216972529888,
+            -0.033041469752788544,
+            -0.01015730481594801,
+            0.008547553792595863,
+            -0.03482504561543465,
+            -0.008471978828310966,
+            0.01051250845193863,
+            -0.018077583983540535,
+            -0.03775736317038536,
+            0.014744718559086323,
+            -0.028129084035754204,
+            0.023911988362669945,
+            -0.024622395634651184,
+            0.010089287534356117,
+            0.002883193315938115,
+            -0.006457899697124958,
+            -0.011563003063201904,
+            -0.0022823705803602934,
+            -0.002686697756871581,
+            -0.0020103000570088625,
+            -0.03140904754400253,
+            -0.014321497641503811,
+            -0.017367178574204445,
+            -0.002590339398011565,
+            -0.033706534653902054,
+            0.025090962648391724,
+            0.030411455780267715,
+            0.012658843770623207,
+            0.007323235739022493,
+            -0.008751606568694115,
+            -0.01841011457145214,
+            -0.007874934002757072,
+            -0.033011242747306824,
+            -0.0064805722795426846,
+            -0.05928117781877518,
+            -0.010905499570071697,
+            -0.008932987228035927,
+            0.01772993803024292,
+            -0.013202984817326069,
+            -0.010814809240400791,
+            -0.02100990153849125,
+            -0.04099198058247566,
+            -0.019014716148376465,
+            0.007202315144240856,
+            -0.03379722312092781,
+            0.05925094708800316,
+            0.013210542500019073,
+            0.006968032103031874,
+            -0.024577051401138306,
+            0.010051500052213669,
+            -0.000378112104954198,
+            0.0096585089340806,
+            -0.005902422126382589,
+            -0.010913057252764702,
+            0.013339020311832428,
+            0.01698930189013481,
+            -0.008517323061823845,
+            0.0012687183916568756,
+            0.016173088923096657,
+            0.01682303659617901,
+            0.02614901401102543,
+            0.022249335423111916,
+            0.019014716148376465,
+            0.032769400626420975,
+            -0.029232481494545937,
+            0.06293901801109314,
+            -0.011479870416224003,
+            0.005369617138057947,
+            0.016928842291235924,
+            -0.019211212173104286,
+            0.029791738837957382,
+            -0.015251072123646736,
+            -0.02143312245607376,
+            0.00335175939835608,
+            0.005607679020613432,
+            0.013943621888756752,
+            -0.05217710882425308,
+            0.033434461802244186,
+            0.03963162750005722,
+            0.017352063208818436,
+            -0.00019826677453238517,
+            0.009174827486276627,
+            -0.0190751776099205,
+            0.038120124489068985,
+            -0.007342129480093718,
+            0.019997194409370422,
+            0.03727368265390396,
+            -0.0008039310341700912,
+            0.029292942956089973,
+            -0.012326313182711601,
+            0.02894529700279236,
+            -0.006605271250009537,
+            0.012666401453316212,
+            0.01198622491210699,
+            -0.009325978346168995,
+            -0.023262042552232742,
+            -0.015326648019254208,
+            0.0017731827683746815,
+            0.0012384883593767881,
+            -0.02843138575553894,
+            0.030245188623666763,
+            -0.019241442903876305,
+            -0.0007533901371061802,
+            0.006095138844102621,
+            -0.017412522807717323,
+            0.000406925129937008,
+            -0.0006593934958800673,
+            0.009507358074188232,
+            -0.005441413260996342,
+            0.010686331428587437,
+            0.01584055833518505,
+            -0.011842631734907627,
+            -0.02240048535168171,
+            0.01701953262090683,
+            -0.010066614486277103,
+            0.006968032103031874,
+            -0.03310193121433258,
+            -0.003370653372257948,
+            -0.03775736317038536,
+            -0.02847672998905182,
+            0.03948047757148743,
+            -0.0163242407143116,
+            -0.02265744097530842,
+            0.021599387750029564,
+            -0.03524826839566231,
+            -0.010111959651112556,
+            0.021508697420358658,
+            -0.010988632217049599,
+            -0.011736826039850712,
+            0.027554713189601898,
+            0.002395733492448926,
+            -0.009401553310453892,
+            -0.04990985244512558,
+            -0.026012979447841644,
+            0.004870820790529251,
+            0.011215358041226864,
+            -0.013308790512382984,
+            0.04546603187918663,
+            -0.01479762140661478,
+            -0.042352333664894104,
+            -0.015228400006890297,
+            0.01529641728848219,
+            -0.0099003491923213,
+            -0.019468167796730995,
+            -0.014230807311832905,
+            0.009726526215672493,
+            0.0024070697836577892,
+            -0.03149973601102829,
+            -0.01108688022941351,
+            0.02169007807970047,
+            -0.009847446344792843,
+            0.011744383722543716,
+            -0.03585286810994148,
+            -0.021856343373656273,
+            -0.01844034530222416,
+            0.020677370950579643,
+            0.029202252626419067,
+            -0.00115252158138901,
+            0.010368915274739265,
+            0.011895534582436085,
+            0.024652626365423203,
+            0.010452047921717167,
+            -0.0195588581264019,
+            0.015183054842054844,
+            -0.016037054359912872,
+            0.04032691940665245,
+            -0.0027830563485622406,
+            0.004417369607836008,
+            -0.03355538100004196,
+            0.0029323173221200705,
+            0.009303305298089981,
+            0.020223919302225113,
+            -0.009091694839298725,
+            0.03821081295609474,
+            -0.024335209280252457,
+            0.002350388327613473,
+            -0.009514915756881237,
+            0.01352040097117424,
+            0.03727368265390396,
+            0.002677250886335969,
+            -0.005146670155227184,
+            -0.007164527662098408,
+            0.019483283162117004,
+            0.024818891659379005,
+            -0.03760621324181557,
+            -0.013694223947823048,
+            0.039027027785778046,
+            0.00967362429946661,
+            -0.00875916425138712,
+            0.00459875026717782,
+            0.014737160876393318,
+            -0.023050431162118912,
+            0.017412522807717323,
+            0.015364435501396656,
+            0.009023677557706833,
+            -0.01889379695057869,
+            0.017352063208818436,
+            -0.016868380829691887,
+            -0.011419410817325115,
+            0.02753959782421589,
+            0.026285050436854362,
+            -0.025483952835202217,
+            0.016913726925849915,
+            -0.02401779405772686,
+            0.028068624436855316,
+            0.002153892768546939,
+            0.002055644989013672,
+            0.00874404888600111,
+            -0.004096175078302622,
+            -0.02867322601377964,
+            0.0012375436490401626,
+            0.006571262609213591,
+            0.006616607774049044,
+            0.0036238301545381546,
+            0.010028827004134655,
+            0.014926099218428135,
+            -0.008449305780231953,
+            0.008434190414845943,
+            0.00838884525001049,
+            0.02917202189564705,
+            -0.009356208145618439,
+            -0.027373332530260086,
+            -0.006102696061134338,
+            -0.0029512110631912947,
+            0.028506960719823837,
+            0.01292335707694292,
+            -0.00814700499176979,
+            0.02167496271431446,
+            0.09613163769245148,
+            0.006185829173773527,
+            -0.020163459703326225,
+            -0.02353411354124546,
+            -0.013437268324196339,
+            0.005891085602343082,
+            -0.006684625055640936,
+            0.009371323511004448,
+            0.004704555496573448,
+            -0.013815144076943398,
+            -0.009862561710178852,
+            0.03748529404401779,
+            -0.002057534409686923,
+            -0.008298155851662159,
+            -0.012885569594800472,
+            0.01841011457145214,
+            0.00780691672116518,
+            -0.008993446826934814,
+            0.004145299084484577,
+            -0.02519676648080349,
+            -0.03171134740114212,
+            -0.010656101629137993,
+            -0.005123997572809458,
+            0.02329227328300476,
+            -0.006423891056329012,
+            0.019014716148376465,
+            0.0054225195199251175,
+            0.012099587358534336,
+            -0.009991039521992207,
+            -0.0012252627639099956,
+            -0.018939141184091568,
+            0.019060062244534492,
+            -0.009371323511004448,
+            -0.0009862561710178852,
+            -0.01176705677062273,
+            0.003833551425486803,
+            -0.019634433090686798,
+            -0.0015766873257234693,
+            -0.029066216200590134,
+            0.016475390642881393,
+            -0.0054225195199251175,
+            0.0069491383619606495,
+            0.011842631734907627,
+            -0.012401888146996498,
+            0.0007992075989022851,
+            0.006166934967041016,
+            -0.0027981712482869625,
+            -0.03703184053301811,
+            0.011797286570072174,
+            0.007274111732840538,
+            0.02147846855223179,
+            -0.020405299961566925,
+            -0.034371595829725266,
+            -0.012076914310455322,
+            -0.0017467314610257745,
+            0.013369250111281872,
+            0.014109887182712555,
+            0.01059564109891653,
+            -0.025952519848942757,
+            -0.008426633663475513,
+            0.018153158947825432,
+            0.012144932523369789,
+            0.023141121491789818,
+            -0.006964253261685371,
+            0.041626811027526855,
+            -0.001176138874143362,
+            0.024335209280252457,
+            -0.006695961579680443,
+            -0.0018657624023035169,
+            -0.01934724859893322,
+            0.00885741226375103,
+            0.031348586082458496,
+            0.010860154405236244,
+            -0.015303974971175194,
+            -0.022944627329707146,
+            0.00953758880496025,
+            0.011245587840676308,
+            -0.006231173872947693,
+            0.009038791991770267,
+            -0.024365440011024475,
+            -0.00780691672116518,
+            0.0011780281784012914,
+            0.0012687183916568756,
+            0.021085476502776146,
+            -0.033918142318725586,
+            -0.01984604448080063,
+            -0.008373730815947056,
+            -0.014842966571450233,
+            -0.005003077443689108,
+            0.002580892527475953,
+            0.004020600114017725,
+            0.01200889702886343,
+            -0.012568153440952301,
+            -0.002246472518891096,
+            -0.024773545563220978,
+            0.00021161051699891686,
+            0.019271671772003174,
+            -0.007769129239022732,
+            -0.023201582953333855,
+            0.012658843770623207,
+            -0.016838151961565018,
+            0.010300897993147373,
+            0.026209475472569466,
+            -0.010399146005511284,
+            0.014412187971174717,
+            -0.015961479395627975,
+            -0.01936236210167408,
+            0.01328611746430397,
+            -0.005709705408662558,
+            -0.002926649060100317,
+            0.009975924156606197,
+            0.005796616896986961,
+            -0.00010568717698333785,
+            0.02982196770608425,
+            0.034432053565979004,
+            -0.023262042552232742,
+            0.013233215548098087,
+            -0.008124332875013351,
+            0.006427669432014227,
+            -0.0009295747731812298,
+            -0.02212841436266899,
+            0.014344170689582825,
+            0.014132560230791569,
+            -0.0022237999364733696,
+            -0.002837848151102662,
+            0.023624803870916367,
+            -0.0008270759717561305,
+            -0.01027822494506836,
+            0.031590428203344345,
+            0.002308822004124522,
+            0.0005960993003100157,
+            0.009016119875013828,
+            0.021508697420358658,
+            0.04171750321984291,
+            0.012658843770623207,
+            -0.00953758880496025,
+            0.0013452382991090417,
+            -0.005294041708111763,
+            0.009325978346168995,
+            -0.012069357559084892,
+            0.02072271518409252,
+            0.00896321702748537,
+            0.0244863610714674,
+            -0.003982812166213989,
+            -0.004791466984897852,
+            -0.02001230977475643,
+            -0.01537954993546009,
+            0.00814700499176979,
+            0.033192623406648636,
+            -0.000691512948833406,
+            -0.015674293041229248,
+            -0.0030362331308424473,
+            0.033041469752788544,
+            -0.0033668745309114456,
+            -0.030955595895648003,
+            -0.03122766688466072,
+            -0.00484059052541852,
+            -0.0019167757127434015,
+            0.012915799394249916,
+            -0.0017618464771658182,
+            0.007172085344791412,
+            0.015349320136010647,
+            -0.04576833173632622,
+            -0.046977534890174866,
+            0.015901019796729088,
+            -0.010777021758258343,
+            0.007247660309076309,
+            0.019407708197832108,
+            -0.033948373049497604,
+            -0.008789394050836563,
+            0.0060271210968494415,
+            0.012787321582436562,
+            0.008177234791219234,
+            0.006310528144240379,
+            0.015183054842054844,
+            -0.027872128412127495,
+            -0.019286787137389183,
+            0.020103000104427338,
+            0.0011317384196445346,
+            -0.004341794643551111,
+            0.0024788661394268274,
+            0.01132872048765421,
+            -0.009152155369520187,
+            0.01352040097117424,
+            0.0006357762613333762,
+            0.019301902502775192,
+            -0.011215358041226864,
+            -0.006333200726658106,
+            -0.010920614004135132,
+            0.0016258112154901028,
+            0.014147674664855003,
+            0.01724625751376152,
+            0.0027887243777513504,
+            0.006831996608525515,
+            0.001246045925654471,
+            0.00991546455770731,
+            -0.015016789548099041,
+            -0.0006310528260655701,
+            0.01071656122803688,
+            0.014019196853041649,
+            0.0035161355044692755,
+            0.009718968532979488,
+            0.006605271250009537,
+            0.0025393262039870024,
+            0.0024335209745913744,
+            -0.025967633351683617,
+            -0.009605606086552143,
+            -0.013905834406614304,
+            0.011804844252765179,
+            -0.004402254708111286,
+            0.0057852803729474545,
+            -0.010293340310454369,
+            -0.012968702241778374,
+            0.020118115469813347,
+            -0.0048292544670403,
+            -0.010520066134631634,
+            -0.026466431096196175,
+            -0.039571166038513184,
+            0.027630288153886795,
+            -0.009492243640124798,
+            0.0037579762283712626,
+            -0.01039158832281828,
+            0.022793475538492203,
+            0.0052713691256940365,
+            0.007814474403858185,
+            0.006166934967041016,
+            0.0067375279031693935,
+            -0.0031873835250735283,
+            -0.02097967080771923,
+            -0.0024316315539181232,
+            -0.01703464612364769,
+            0.017533443868160248,
+            -0.010210207663476467,
+            0.021735424175858498,
+            0.015719639137387276,
+            0.00023640862491447479,
+            0.006268961820751429,
+            0.02613389864563942,
+            0.012757091782987118,
+            0.0160824004560709,
+            -0.025816483423113823,
+            -0.0037749805487692356,
+            -0.01035380084067583,
+            0.024652626365423203,
+            -0.005505652166903019,
+            -0.00023735332069918513,
+            0.014865638688206673,
+            -0.022264450788497925,
+            -0.009870119392871857,
+            0.012878011912107468,
+            -0.006722413003444672,
+            -0.02234002575278282,
+            -0.001692884135991335,
+            -0.019664663821458817,
+            -0.007198536302894354,
+            -0.022143529728055,
+            0.005312935449182987,
+            0.018984487280249596,
+            -0.008721376769244671,
+            0.016399815678596497,
+            0.02915690653026104,
+            -0.006695961579680443,
+            0.024818891659379005,
+            0.0160824004560709,
+            0.0010372694814577699,
+            -0.00046148098772391677,
+            -0.005894864443689585,
+            0.004477829672396183,
+            -0.009333535097539425,
+            -0.012817551381886005,
+            -0.01748809777200222,
+            0.009061465039849281,
+            -0.018304310739040375,
+            0.015470240265130997,
+            0.03473435714840889,
+            -0.004020600114017725,
+            0.0042586615309119225,
+            -0.022959740832448006,
+            -0.016611425206065178,
+            -0.004447599872946739,
+            -0.014850524254143238,
+            0.06517603993415833,
+            -0.004795245360583067,
+            -0.014502878300845623,
+            -0.012666401453316212,
+            0.0036672858987003565,
+            -0.0028869721572846174,
+            -0.0036691753193736076,
+            -0.012182720005512238,
+            -0.015137709677219391,
+            -0.006964253261685371,
+            0.022445829585194588,
+            -0.032769400626420975,
+            0.005475422367453575,
+            0.030728871002793312,
+            -0.005641687661409378,
+            -0.0037579762283712626,
+            0.0023787289392203093,
+            -0.017397407442331314,
+            -0.012447233311831951,
+            -0.013565746136009693,
+            0.0008299099863506854,
+            0.0005710650002583861,
+            -0.0033215293660759926,
+            -0.020420415326952934,
+            0.01913563720881939,
+            -0.00850220862776041,
+            0.005309156607836485,
+            -0.029716163873672485,
+            -0.0055283247493207455,
+            0.012651286087930202,
+            0.0045005022548139095,
+            -0.019891388714313507,
+            0.013225657865405083,
+            -0.017790399491786957,
+            -0.0017023311229422688,
+            0.009356208145618439,
+            -0.01768459379673004,
+            -0.007391253020614386,
+            -0.024879351258277893,
+            -0.045798562467098236,
+            0.0036937373224645853,
+            -0.005743714049458504,
+            0.04011530801653862,
+            0.010452047921717167,
+            0.025816483423113823,
+            -0.011751941405236721,
+            -0.01798689365386963,
+            -0.008275482803583145,
+            -0.010897941887378693,
+            -0.015387107618153095,
+            -0.014193019829690456,
+            0.005683253984898329,
+            0.0029474322218447924,
+            -0.004885935690253973,
+            0.024380555376410484,
+            -0.006930244620889425,
+            -0.02188657410442829,
+            -0.003960139583796263,
+            -0.022037724032998085,
+            -0.01682303659617901,
+            0.006000669673085213,
+            -0.001827030093409121,
+            0.008630686439573765,
+            -0.005592563655227423,
+            -0.004398475866764784,
+            -0.016414931043982506,
+            0.00762931490316987,
+            0.020647140219807625,
+            0.010897941887378693,
+            -0.03691092133522034,
+            0.026511775329709053,
+            0.005849519278854132,
+            0.0007633093628101051,
+            0.017624134197831154,
+            0.004904829431325197,
+            0.007255217991769314,
+            0.01792643405497074,
+            -0.024773545563220978,
+            -0.004016821272671223,
+            -0.011880419217050076,
+            -0.025045616552233696,
+            -0.00943934079259634,
+            0.007799359038472176,
+            -0.020178575068712234,
+            -0.015689408406615257,
+            0.008366173133254051,
+            -0.02167496271431446,
+            0.01443486101925373,
+            -0.03503665700554848,
+            0.02026926539838314,
+            0.037878282368183136,
+            0.0072967843152582645,
+            0.0190751776099205,
+            0.002031083218753338,
+            0.00969629641622305,
+            -0.0014718268066644669,
+            -0.008328385651111603,
+            -0.02191680483520031,
+            0.01340703759342432,
+            -0.009257960133254528,
+            -0.003017339389771223,
+            -0.0030456800013780594,
+            0.00804120022803545,
+            -0.0009701964445412159,
+            -0.010127075016498566,
+            -0.0026111227925866842,
+            -0.015092364512383938,
+            0.008320827968418598,
+            -0.005660581402480602,
+            0.032315950840711594,
+            0.0008440803503617644,
+            0.018062468618154526,
+            0.0015332315815612674,
+            0.031620658934116364,
+            0.01863684132695198,
+            0.021841228008270264,
+            -0.0016891054110601544,
+            -0.008298155851662159,
+            0.00850220862776041,
+            -0.0019630654715001583,
+            -0.019906504079699516,
+            -0.01700441725552082,
+            -0.0028170649893581867,
+            0.0033196399454027414,
+            0.0006197165348567069,
+            0.007572633679956198,
+            0.006125368643552065,
+            -0.024093369022011757,
+            -0.01129093300551176,
+            0.003548254957422614,
+            0.0007727562915533781,
+            -0.005600121337920427,
+            0.009983481839299202,
+            -0.020647140219807625,
+            -0.0044060335494577885,
+            0.025317687541246414,
+            0.019755354151129723,
+            -0.025831598788499832,
+            -0.01549291331321001,
+            -0.005660581402480602,
+            0.0007500837091356516,
+            -0.018787991255521774,
+            -0.014389515854418278,
+            0.0037390822544693947,
+            -0.007504615932703018,
+            0.006779094226658344,
+            -0.01772993803024292,
+            -0.009514915756881237,
+            -0.0051202187314629555,
+            -0.030154500156641006,
+            0.012326313182711601,
+            0.013694223947823048,
+            0.030245188623666763,
+            -0.008676031604409218,
+            -0.01635446958243847,
+            -0.029323171824216843,
+            -0.006336979568004608,
+            -0.004765015561133623,
+            0.0033007462043315172,
+            0.004659210331737995,
+            -0.015115037560462952,
+            -0.025468837469816208,
+            -0.013376807793974876,
+            0.0063898819498717785,
+            -0.020571565255522728,
+            0.015251072123646736,
+            -0.008660916239023209,
+            0.03146950900554657,
+            -0.017805512994527817,
+            0.001601249212399125,
+            0.01222806517034769,
+            0.0005838182987645268,
+            0.007565075997263193,
+            0.0014954439830034971,
+            0.008071430027484894,
+            0.021130822598934174,
+            0.00826792512089014,
+            -0.01841011457145214,
+            0.007122961338609457,
+            0.006423891056329012,
+            -0.019422823563218117,
+            -0.0005328050465323031,
+            0.02052622102200985,
+            0.0030966931954026222,
+            0.00015670042193960398,
+            0.0021595610305666924,
+            0.004016821272671223,
+            0.01674746163189411,
+            -0.010436933487653732,
+            0.02495492622256279,
+            -0.00806387234479189,
+            -0.02732798643410206,
+            -0.01469181664288044,
+            -0.004077281337231398,
+            -0.03104628622531891,
+            -0.01108688022941351,
+            0.0009371322812512517,
+            -0.001392472768202424,
+            -0.006934023462235928,
+            0.0020443086978048086,
+            0.023866644129157066,
+            -0.003994148690253496,
+            -0.0021671184804290533,
+            0.01912052184343338,
+            0.09897326678037643,
+            0.002624348271638155,
+            -0.007746456656605005,
+            0.00437958212569356,
+            -0.011260703206062317,
+            -0.00647301459684968,
+            -0.018833335489034653,
+            -0.04924479126930237,
+            0.0016125855036079884,
+            1.3513492376659997e-05,
+            -0.004247325472533703,
+            -0.017125336453318596,
+            -0.013822701759636402,
+            0.013467498123645782,
+            -0.0060875811614096165,
+            0.01306694932281971,
+            -0.0052222455851733685,
+            0.0032686267513781786,
+            0.02329227328300476,
+            0.017881089821457863,
+            -0.006639279890805483,
+            0.009628279134631157,
+            -0.0029058658983558416,
+            0.008955659344792366,
+            0.018032239750027657,
+            0.014193019829690456,
+            -0.00769733265042305,
+            0.011661251075565815,
+            0.004946395754814148,
+            0.03981300815939903,
+            -0.007308120373636484,
+            -0.016928842291235924,
+            -0.00953758880496025,
+            -0.0009928690269589424,
+            -0.0053620594553649426,
+            0.011238030157983303,
+            -0.002766051795333624,
+            -0.016838151961565018,
+            -0.0017174461390823126,
+            -0.00780691672116518,
+            0.014306383207440376,
+            0.01117756962776184,
+            0.020163459703326225,
+            0.011457198299467564,
+            -0.002864299574866891,
+            0.028053509071469307,
+            -0.006129147484898567,
+            5.936785237281583e-05,
+            -0.013868046924471855,
+            0.004825475625693798,
+            -0.018017124384641647,
+            -0.002760383766144514,
+            0.005891085602343082,
+            -0.005078652407974005,
+            -0.0011563003063201904,
+            -0.0096585089340806,
+            0.005709705408662558,
+            -0.0027868349570780993,
+            0.010149747133255005,
+            -0.0036200513131916523,
+            -0.0018497026758268476,
+            0.009666066616773605,
+            0.00722120888531208,
+            -0.008056314662098885,
+            0.00922017265111208,
+            -0.021342432126402855,
+            -0.0007930670981295407,
+            -0.0074025895446538925,
+            0.003123144619166851,
+            -0.0019989637657999992,
+            -0.0030192285776138306,
+            -0.005569891072809696,
+            0.02031460963189602,
+            0.0017769616097211838,
+            -0.007980739697813988,
+            -0.022264450788497925,
+            0.022113298997282982,
+            -0.0031590426806360483,
+            0.014601126313209534,
+            -0.008827181532979012,
+            0.008925429545342922,
+            0.0035161355044692755,
+            -0.003438670886680484,
+            -0.002607343951240182,
+            -0.0018232513684779406,
+            -0.00361060444265604,
+            0.004307785537093878,
+            -0.001872375258244574,
+            0.0004959621583111584,
+            0.02311089262366295,
+            0.0009701964445412159,
+            -0.0027509366627782583,
+            0.012658843770623207,
+            0.006318085361272097,
+            0.0017004417022690177,
+            0.023881759494543076,
+            -0.008592898957431316,
+            -0.007727562915533781,
+            0.00012221925135236233,
+            -0.007258996833115816,
+            0.031590428203344345,
+            -0.009749199263751507,
+            0.00227103428915143,
+            0.020798292011022568,
+            -0.010285782627761364,
+            -0.003967697266489267,
+            0.003967697266489267,
+            -0.015553372912108898,
+            0.020344840362668037,
+            -0.023655032739043236,
+            0.022566750645637512,
+            0.012001339346170425,
+            -0.021372662857174873,
+            0.013913392089307308,
+            -0.00045439580571837723,
+            -0.013021604157984257,
+            0.006370988208800554,
+            0.007980739697813988,
+            -0.0420198030769825,
+            -0.008222579956054688,
+            -0.007882491685450077,
+            -0.03340423107147217,
+            0.009635835886001587,
+            -0.016430046409368515,
+            -0.0064956871792674065,
+            -0.0034084408544003963,
+            -0.006439005956053734,
+            -0.002450525527819991,
+            -0.02261209487915039,
+            -0.02327715791761875,
+            0.00642011221498251,
+            -0.02732798643410206,
+            -0.008184792473912239,
+            -0.0003235562762711197,
+            -0.002083985833451152,
+            -0.005668139085173607,
+            0.0052260239608585835,
+            0.016475390642881393,
+            0.0034821266308426857,
+            0.014049426652491093,
+            -0.005448970943689346,
+            0.011449640616774559,
+            -0.005626572761684656,
+            -0.01260594092309475,
+            -0.002288038842380047,
+            0.0027414897922426462,
+            -0.014011639170348644,
+            0.004251104313880205,
+            -0.008139447309076786,
+            -0.0060724662616848946,
+            0.012681515887379646,
+            -0.018939141184091568,
+            0.0018100256565958261,
+            0.008766721934080124,
+            7.917681796243414e-05,
+            -0.01258326880633831,
+            -0.028401155024766922,
+            0.00256955623626709,
+            -0.0054149623028934,
+            -0.002854852704331279,
+            0.00033772661117836833,
+            0.012417003512382507,
+            0.011540330946445465,
+            0.010701446793973446,
+            -0.002269144868478179,
+            0.007081395015120506,
+            0.028975525870919228,
+            0.013376807793974876,
+            -0.0074630496092140675,
+            0.0002914368233177811,
+            0.0022502511274069548,
+            0.0068886782974004745,
+            -0.020178575068712234,
+            -0.010323570109903812,
+            -0.0272372979670763,
+            -0.008766721934080124,
+            0.0014708820963278413,
+            0.0013594086049124599,
+            0.0244863610714674,
+            0.007493279874324799,
+            0.0018279748037457466,
+            -0.0028095075394958258,
+            0.010520066134631634,
+            0.007149412762373686,
+            0.0048028030432760715,
+            0.007009598426520824,
+            -0.018017124384641647,
+            -0.011593233793973923,
+            0.02146335318684578,
+            -0.0025884502101689577,
+            0.005543440114706755,
+            -0.0015200059860944748,
+            -0.0038203257136046886,
+            -0.004285112954676151,
+            0.01961931772530079,
+            -0.005671917926520109,
+            0.00012599800538737327,
+            0.029444092884659767,
+            -0.0035350292455404997,
+            0.0036238301545381546,
+            -0.006397439632564783,
+            -0.007224987726658583,
+            -0.04002461954951286,
+            0.01633935607969761,
+            -0.006983147002756596,
+            0.009431783109903336,
+            -0.010119517333805561,
+            0.009817216545343399,
+            0.010988632217049599,
+            0.0045609623193740845,
+            -0.010066614486277103,
+            0.0023995121009647846,
+            0.015047019347548485,
+            -0.011147339828312397,
+            -0.008313270285725594,
+            0.02100990153849125,
+            -0.006370988208800554,
+            -0.001745786750689149,
+            -0.0019177203066647053,
+            0.011948436498641968,
+            0.0027452686335891485,
+            0.018319424241781235,
+            0.005123997572809458,
+            -0.012076914310455322,
+            -0.0012970591196790338,
+            0.010308455675840378,
+            0.001875209272839129,
+            -0.006775315385311842,
+            0.005894864443689585,
+            -0.021554043516516685,
+            -0.0011270149843767285,
+            0.0018024682067334652,
+            0.023428307846188545,
+            0.005078652407974005,
+            0.026889652013778687,
+            -0.00035520337405614555,
+            0.0012167604872956872,
+            0.007727562915533781,
+            -0.007092731073498726,
+            -0.027146607637405396,
+            -0.01282510906457901,
+            -0.00828304048627615,
+            0.00018822944548446685,
+            0.0023749503307044506,
+            0.021735424175858498,
+            0.01841011457145214,
+            -0.02167496271431446,
+            0.007935394532978535,
+            -0.0047574578784406185,
+            -2.664320527401287e-05,
+            0.00888008438050747,
+            -0.00601200619712472,
+            0.003933688625693321,
+            0.00038921221857890487,
+            -0.0014944992726668715,
+            -0.007784244138747454,
+            -0.00886496901512146,
+            -0.0027112597599625587,
+            0.012084471993148327,
+            -0.003540697507560253,
+            0.008713819086551666,
+            -0.017654363065958023,
+            0.012386772781610489,
+            -0.008479535579681396,
+            0.03428090363740921,
+            0.010459605604410172,
+            -0.009061465039849281,
+            -0.013724453747272491,
+            -0.022083070129156113,
+            0.048851799219846725,
+            0.010346243157982826,
+            -0.0010835592402145267,
+            -0.00513911247253418,
+            -0.0077540138736367226,
+            -0.00036866520531475544,
+            -0.009582933969795704,
+            0.004315343219786882,
+            -0.004443821031600237,
+            0.006707297638058662,
+            0.0014160900609567761,
+            -0.0349157378077507,
+            0.0048179179430007935,
+            -0.014979002065956593,
+            -0.0012743865372613072,
+            -0.0008365228422917426,
+            -0.006892457138746977,
+            -0.00413396256044507,
+            -0.004625201225280762,
+            -0.008630686439573765,
+            0.01435928512364626,
+            0.0028850827366113663,
+            0.00026970895123668015,
+            -0.005585006438195705,
+            -0.013475055806338787,
+            0.008056314662098885,
+            -0.012386772781610489,
+            0.012462347745895386,
+            -0.006930244620889425,
+            -0.008668473921716213,
+            0.018304310739040375,
+            0.008252810686826706,
+            -0.0032421755604445934,
+            -0.02146335318684578,
+            -0.012333869934082031,
+            0.030895136296749115,
+            -0.010814809240400791,
+            0.005970439873635769,
+            -0.004451378248631954,
+            0.001610696199350059,
+            -0.006627943832427263,
+            0.008154562674462795,
+            0.014049426652491093,
+            0.01772993803024292,
+            0.017639247700572014,
+            -0.01771482452750206,
+            -0.0014689926756545901,
+            0.004799024201929569,
+            -0.02028438076376915,
+            -0.022959740832448006,
+            0.03682022914290428,
+            -0.002288038842380047,
+            0.014994116500020027,
+            0.0015162271447479725,
+            0.016369584947824478,
+            -0.011540330946445465,
+            0.017881089821457863,
+            0.004659210331737995,
+            -0.012280968017876148,
+            0.0019479504553601146,
+            -0.022007493302226067,
+            0.0396013967692852,
+            -0.010520066134631634,
+            -0.013210542500019073,
+            0.00734590832144022,
+            -0.02096455730497837,
+            -0.005093767773360014,
+            -0.024395670741796494,
+            -0.006080023944377899,
+            -0.00045415962813422084,
+            0.0004756513226311654,
+            0.007542403414845467,
+            -0.015523143112659454,
+            -0.028325580060482025,
+            -0.001077891094610095,
+            -0.011774614453315735,
+            -0.006148041225969791,
+            -0.01306694932281971,
+            -0.029761508107185364,
+            -0.011736826039850712,
+            0.039299096912145615,
+            -0.006979368627071381,
+            -0.0028964190278202295,
+            -0.0008029863820411265,
+            -0.0008398292702622712,
+            0.0038184362929314375,
+            0.019483283162117004,
+            -0.005887307226657867,
+            0.023609688505530357,
+            -0.005777723155915737,
+            0.010096845217049122,
+            0.004821696784347296,
+            -0.013822701759636402,
+            -0.013384365476667881,
+            0.014004082418978214,
+            0.003289409913122654,
+            -0.020390184596180916,
+            0.023670148104429245,
+            -0.008358615450561047,
+            0.01866707019507885,
+            -0.017216026782989502,
+            0.012099587358534336,
+            -0.02661758102476597,
+            -0.012757091782987118,
+            -0.014593568630516529,
+            -0.017805512994527817,
+            0.008819623850286007,
+            -0.011835074052214622,
+            0.01653585024178028,
+            -0.005743714049458504,
+            -0.0018837115494534373,
+            -0.0011723600327968597,
+            -0.037001609802246094,
+            0.0032591798808425665,
+            0.026693155989050865,
+            0.03675977140665054,
+            0.012311197817325592,
+            0.0030097817070782185,
+            -0.007950509898364544,
+            0.03654816001653671,
+            -0.010913057252764702,
+            -0.0122431805357337,
+            0.013187870383262634,
+            -0.006355873309075832,
+            -5.260741454549134e-05,
+            -0.01002126932144165,
+            0.02191680483520031,
+            0.013369250111281872,
+            0.04171750321984291,
+            -0.007319456897675991,
+            -0.000410467735491693,
+            -0.00897833239287138,
+            0.0007189089665189385,
+            0.007122961338609457,
+            0.0036616178695112467,
+            0.0017968000611290336,
+            -0.015704523772001266,
+            0.0067526428028941154,
+            -0.010867712087929249,
+            0.009046349674463272,
+            -0.0007529177819378674,
+            0.001029711915180087,
+            0.010043942369520664,
+            0.009348650462925434,
+            0.009076579473912716,
+            0.01958908885717392,
+            -0.012870454229414463,
+            -0.0059213158674538136,
+            -0.008895199745893478,
+            0.015024347230792046,
+            -0.0124850207939744,
+            0.0160824004560709,
+            0.006790430285036564,
+            -0.006091360002756119,
+            0.008411518298089504,
+            0.003138259518891573,
+            -0.0036559496074914932,
+            -0.0035180249251425266,
+            0.02422940544784069,
+            -0.007164527662098408,
+            -0.016399815678596497,
+            -0.00943934079259634,
+            0.018818221986293793,
+            -0.006114032585173845,
+            0.005214687902480364,
+            -0.005709705408662558,
+            -0.003502909792587161,
+            -0.013830259442329407,
+            -0.003361206501722336,
+            -0.010603198781609535,
+            0.0021274415776133537,
+            0.02028438076376915,
+            -0.0005294986767694354,
+            0.0034273345954716206,
+            0.0001297767594223842,
+            0.00804875697940588,
+            0.010913057252764702,
+            0.011472313664853573,
+            0.004387139342725277,
+            0.02333761751651764,
+            0.04126405343413353,
+            -0.00022826067288406193,
+            0.0005219411104917526,
+            -0.012537923641502857,
+            0.013928506523370743,
+            -0.0018374216742813587,
+            0.028340695425868034,
+            -0.01960420422255993,
+            0.0026848085690289736,
+            0.006125368643552065,
+            -0.01794154942035675,
+            -0.0003625247336458415,
+            0.001641870941966772,
+            0.00031694344943389297,
+            0.014147674664855003,
+            0.0051051038317382336,
+            -0.006627943832427263,
+            0.005199573002755642,
+            0.00804875697940588,
+            -0.0022237999364733696,
+            -0.015991710126399994,
+            0.012409445829689503,
+            0.0020461981184780598,
+            -0.0003955888678319752,
+            0.010127075016498566,
+            0.027887243777513504,
+            0.006185829173773527,
+            0.005566112697124481,
+            0.028053509071469307,
+            0.0027641623746603727,
+            0.010006154887378216,
+            0.00484059052541852,
+            0.007738898973912001,
+            0.007580191362649202,
+            -0.006321864202618599,
+            -0.014283710159361362,
+            -0.016974186524748802,
+            -0.01723114214837551,
+            -0.0030097817070782185,
+            -0.00016000683535821736,
+            0.03222525864839554,
+            0.001197866746224463,
+            0.01073167659342289,
+            0.01866707019507885,
+            -0.0017901872051879764,
+            -0.004885935690253973,
+            -0.025499068200588226,
+            -0.00489727221429348,
+            -0.0003348925383761525,
+            0.007073837332427502,
+            0.0005772055010311306,
+            0.004753679037094116,
+            -0.007799359038472176,
+            -0.00035071608726866543,
+            0.01655096560716629,
+            0.0014293156564235687,
+            0.013097180053591728,
+            -0.02351899817585945,
+            -0.023231811821460724,
+            0.010006154887378216,
+            0.020465759560465813,
+            0.005048422608524561,
+            -0.011661251075565815,
+            0.0046667675487697124,
+            0.01039158832281828,
+            0.0021425564773380756,
+            0.026753615587949753,
+            0.007323235739022493,
+            -0.018606610596179962,
+            0.008608013391494751,
+            0.010663658380508423,
+            0.01234898529946804,
+            -0.01176705677062273,
+            -0.0035010206047445536,
+            -0.007217430509626865,
+            -0.005690811667591333,
+            0.004107511602342129,
+            0.004353130701929331,
+            0.020934326574206352,
+            0.008509766310453415,
+            0.02008788473904133,
+            0.006151820067316294,
+            0.002475087298080325,
+            0.017382292076945305,
+            0.008441748097538948,
+            0.012069357559084892,
+            0.012757091782987118,
+            0.018712416291236877,
+            -0.02400267869234085,
+            0.017110222950577736,
+            -0.014963886700570583,
+            -0.007051164750009775,
+            0.0011336278403177857,
+            0.0048179179430007935,
+            0.004020600114017725,
+            -0.008237695321440697,
+            0.001068444224074483,
+            -0.010769464075565338,
+            0.00554721849039197,
+            -0.020919211208820343,
+            -0.00011064679711125791,
+            0.012160047888755798,
+            -0.003557701827958226,
+            -0.014767391607165337,
+            -0.010179977864027023,
+            0.02636062540113926,
+            0.011804844252765179,
+            0.012061799876391888,
+            -0.006544811185449362,
+            0.002210574224591255,
+            -0.015440010465681553,
+            0.013920948840677738,
+            0.018727531656622887,
+            -0.008872526697814465,
+            0.023246927186846733,
+            -0.014449975453317165,
+            0.006499466020613909,
+            -0.010149747133255005,
+            -0.006268961820751429,
+            -0.009212614968419075,
+            0.004292670637369156,
+            0.0041037327609956264,
+            0.016717230901122093,
+            0.008366173133254051,
+            -0.008940544910728931,
+            0.0086458008736372,
+            -0.0041982014663517475,
+            -0.008698703721165657,
+            -0.0172916017472744,
+            -0.014125002548098564,
+            -0.010754348710179329,
+            -0.0018138044979423285,
+            -0.0004062166262883693,
+            -0.022748131304979324,
+            0.00751973083242774,
+            0.0005649244994856417,
+            0.009114367887377739,
+            -0.001027822494506836,
+            -0.006873563397675753,
+            0.016974186524748802,
+            0.012076914310455322,
+            0.006896235514432192,
+            0.009552703239023685,
+            0.034643664956092834,
+            0.00945445615798235,
+            -0.007633093744516373,
+            0.013097180053591728,
+            -0.009446898475289345,
+            -0.027569828554987907,
+            0.0030910251662135124,
+            0.00241462723352015,
+            -0.000520996458362788,
+            0.0008346334798261523,
+            -0.005664360243827105,
+            0.0037220779340714216,
+            0.002367392648011446,
+            0.004062166437506676,
+            -0.0027981712482869625,
+            0.010671216063201427,
+            -0.026889652013778687,
+            0.014404630288481712,
+            0.014827851206064224,
+            0.011396737769246101,
+            -0.013482613489031792,
+            0.008260367438197136,
+            0.000955553783569485,
+            -0.017200913280248642,
+            -0.03243687003850937,
+            -0.01002126932144165,
+            0.01305939257144928,
+            0.011162455193698406,
+            0.004364467225968838,
+            0.0009560260805301368,
+            -0.0006178271723911166,
+            0.013792471028864384,
+            0.015213284641504288,
+            -0.004330458119511604,
+            -0.028310464695096016,
+            -0.023050431162118912,
+            0.032739169895648956,
+            0.007799359038472176,
+            0.006280297879129648,
+            0.019982079043984413,
+            0.006503244861960411,
+            0.0008549442864023149,
+            -0.02590717375278473,
+            0.014049426652491093,
+            -0.013505285605788231,
+            -0.004579856060445309,
+            0.006986925844103098,
+            -0.006499466020613909,
+            -0.016641655936837196,
+            0.013898276723921299,
+            0.001303671975620091,
+            -0.007243881467729807,
+            0.006677067838609219,
+            -0.008222579956054688,
+            -0.01001371257007122,
+            -0.01635446958243847,
+            -0.0008351058349944651,
+            -3.5425870009930804e-05,
+            -0.032074108719825745,
+            0.008721376769244671,
+            -0.012598383240401745,
+            0.007769129239022732,
+            0.014570895582437515,
+            0.004224652890115976,
+            -0.021750537678599358,
+            0.005562333855777979,
+            -0.012500136159360409,
+            -0.013497727923095226,
+            0.0069491383619606495,
+            -0.007334571797400713,
+            -0.00636343052610755,
+            -0.015024347230792046,
+            0.0051655638962984085,
+            0.006412554532289505,
+            0.019437937065958977,
+            -0.009552703239023685,
+            -0.0008813955937512219,
+            -0.014563338831067085,
+            0.014253480359911919,
+            -0.015455125831067562,
+            0.016278894618153572,
+            0.001953618600964546,
+            -0.004583634901791811,
+            -0.03237640857696533,
+            -0.003948803525418043,
+            0.0026281271129846573,
+            -0.006257625296711922,
+            -0.007723784074187279,
+            0.019241442903876305,
+            0.02005765400826931,
+            0.005101324990391731,
+            -0.01481273677200079,
+            -8.428404726146255e-06,
+            0.008192350156605244,
+            0.008018527179956436,
+            0.002970104804262519,
+            0.001875209272839129,
+            -0.016384700313210487,
+            -0.003960139583796263,
+            -0.022672556340694427,
+            -0.032557789236307144,
+            -0.006257625296711922,
+            0.0010920614004135132,
+            0.004084839019924402,
+            0.023216696456074715,
+            0.013905834406614304,
+            0.013527958653867245,
+            -0.006918908096849918,
+            0.021614503115415573,
+            0.009567818604409695,
+            -0.018092699348926544,
+            -0.009862561710178852,
+            -0.010043942369520664,
+            0.01629400998353958,
+            0.007213651668280363,
+            0.011147339828312397,
+            0.02823488973081112,
+            0.017352063208818436,
+            0.005558555014431477,
+            0.002766051795333624,
+            -0.007980739697813988,
+            -0.01364132110029459,
+            -0.01199378166347742,
+            -0.009257960133254528,
+            -0.0228690505027771,
+            0.009182385168969631,
+            -0.010520066134631634,
+            0.00665061641484499,
+            -0.008698703721165657,
+            -0.0022219105158001184,
+            0.0060271210968494415,
+            -0.01582544483244419,
+            0.0004508532292675227,
+            -0.01202401239424944,
+            -0.0008365228422917426,
+            0.00802608486264944,
+            0.010059057734906673,
+            -0.01889379695057869,
+            0.00792783685028553,
+            0.005150448996573687,
+            0.010920614004135132,
+            -0.02964058890938759,
+            -0.029323171824216843,
+            0.008320827968418598,
+            -0.01165369339287281,
+            -0.009824774228036404,
+            -0.026239704340696335,
+            -0.009250402450561523,
+            -0.013603533618152142,
+            -0.001517171855084598,
+            0.0019290565978735685,
+            0.021735424175858498,
+            -0.009159712120890617,
+            -0.03518780693411827,
+            -0.0252118818461895,
+            -0.015931248664855957,
+            -0.011215358041226864,
+            0.0019800697918981314,
+            -0.0023466094862669706,
+            -0.001889379695057869,
+            0.006896235514432192,
+            -0.02308066189289093,
+            -0.02049599029123783,
+            0.0006617552135139704,
+            0.010452047921717167,
+            -0.013339020311832428,
+            -0.010119517333805561,
+            0.026421085000038147,
+            -0.015553372912108898,
+            0.01001371257007122,
+            -0.013225657865405083,
+            -0.005925094708800316,
+            0.0010117627680301666,
+            -0.001555904163978994,
+            -0.004145299084484577,
+            0.00558878481388092,
+            -0.03403906524181366,
+            0.00513911247253418,
+            0.031832266598939896,
+            0.02382129803299904,
+            -0.004791466984897852,
+            -0.002518543042242527,
+            -0.03521803766489029,
+            -0.002465640427544713,
+            -0.012689073570072651,
+            0.02753959782421589,
+            -0.003945024684071541,
+            0.007474386133253574,
+            0.001541733741760254,
+            0.0036162727046757936,
+            -0.005600121337920427,
+            0.003094803774729371,
+            0.0005252475384622812,
+            -0.005857076961547136,
+            0.005671917926520109,
+            0.014963886700570583,
+            0.0009494132827967405,
+            0.012938471511006355,
+            -0.01635446958243847,
+            -0.008993446826934814,
+            -0.008872526697814465,
+            -0.024592164903879166,
+            -0.00558878481388092,
+            0.009839889593422413,
+            0.019226327538490295,
+            -0.01467670127749443,
+            -0.015946364030241966,
+            0.01527374517172575,
+            0.006624164991080761,
+            -0.02658735029399395,
+            -0.0007855095900595188,
+            -0.02403290942311287,
+            0.026436200365424156,
+            0.0163242407143116,
+            -0.0023371626157313585,
+            0.00244863610714674,
+            -0.006900014355778694,
+            -0.011691480875015259,
+            -0.013936064206063747,
+            0.00029309000819921494,
+            0.007867377251386642,
+            -0.032315950840711594,
+            0.0076142000034451485,
+            0.009159712120890617,
+            -0.0013546851696446538,
+            -0.02218887396156788,
+            0.017200913280248642,
+            0.00838884525001049,
+            0.0016569859581068158,
+            -0.019876273348927498,
+            0.008101659826934338,
+            0.0014264816418290138,
+            -0.004693218972533941,
+            -0.0049539534375071526,
+            -0.010406702756881714,
+            4.165491554886103e-05,
+            -0.023851528763771057,
+            -0.005974218249320984,
+            -0.007531067356467247,
+            -0.0029946668073534966,
+            0.017352063208818436,
+            -0.014510435983538628,
+            0.00020676899293903261,
+            0.009590490721166134,
+            0.018848450854420662,
+            0.010164862498641014,
+            -0.014593568630516529,
+            0.023216696456074715,
+            0.0019800697918981314,
+            -0.007364802062511444,
+            0.007047385908663273,
+            -0.005112661514431238,
+            -0.009303305298089981,
+            0.009431783109903336,
+            -0.0029927773866802454,
+            0.006563704926520586,
+            -0.008335943333804607,
+            -0.007281669415533543,
+            -0.013527958653867245,
+            -0.0013764130417257547,
+            0.014895869418978691,
+            0.003892122069373727,
+            -0.023896872997283936,
+            0.01528886053711176,
+            -0.002108547603711486,
+            -0.0024335209745913744,
+            -0.009998597204685211,
+            -0.019437937065958977,
+            -0.012976259924471378,
+            -0.009152155369520187,
+            -0.004776351619511843,
+            0.013210542500019073,
+            0.012968702241778374,
+            -0.011623463593423367,
+            -0.009567818604409695,
+            -0.004995519760996103,
+            -0.030487030744552612,
+            0.035792406648397446,
+            -0.019483283162117004,
+            0.02008788473904133,
+            -0.01910540647804737,
+            0.005192015320062637,
+            -0.001130793709307909,
+            0.0021614502184093,
+            -0.007799359038472176,
+            0.004995519760996103,
+            -0.019649548456072807,
+            -0.015961479395627975,
+            -0.006605271250009537,
+            -0.008033642545342445,
+            -0.021130822598934174,
+            -0.014608683064579964,
+            -0.007070058491080999,
+            0.012794879265129566,
+            -0.0011988113401457667,
+            -0.0014633245300501585,
+            0.020919211208820343,
+            0.007814474403858185,
+            0.011162455193698406,
+            0.026164129376411438,
+            0.022566750645637512,
+            0.03149973601102829,
+            -0.002148224739357829,
+            -0.01676257699728012,
+            0.006918908096849918,
+            0.0005550052737817168,
+            0.0007992075989022851,
+            0.002907755319029093,
+            4.006075323559344e-05,
+            0.011132225394248962,
+            -0.01469937339425087,
+            -0.003922352101653814,
+            -0.007640651427209377,
+            -0.010043942369520664,
+            0.006465457379817963,
+            0.007217430509626865,
+            0.0008539995760656893,
+            0.022476060315966606,
+            -0.017790399491786957,
+            0.007890049368143082,
+            -0.0018478132551535964,
+            0.01612774468958378,
+            0.01676257699728012,
+            0.011442082934081554,
+            -0.004477829672396183,
+            -0.005687032826244831,
+            -0.015916133299469948,
+            0.04226164519786835,
+            -0.005033307243138552,
+            0.00028010053210891783,
+            0.0086458008736372,
+            0.008351057767868042,
+            0.0008747827960178256,
+            -0.0095753762871027,
+            -0.010572968982160091,
+            -0.009038791991770267,
+            0.012326313182711601,
+            0.004734785296022892,
+            -0.011600791476666927,
+            -0.016414931043982506,
+            -0.006219837814569473,
+            0.021145937964320183,
+            0.009938136674463749,
+            -0.0077200052328407764,
+            0.001680603250861168,
+            -0.009076579473912716,
+            0.006268961820751429,
+            0.015266187489032745,
+            0.00838128849864006,
+            0.02821977436542511,
+            0.018818221986293793,
+            -0.030003348365426064,
+            0.02637574076652527,
+            0.009605606086552143,
+            0.005063537508249283,
+            0.0035236929543316364,
+            0.016868380829691887,
+            0.010777021758258343,
+            -0.009416667744517326,
+            -0.009666066616773605,
+            0.0013915281742811203,
+            0.0077200052328407764,
+            0.004564741160720587,
+            -0.016399815678596497,
+            0.0006296357605606318,
+            -0.0038108788430690765,
+            -0.014427303336560726,
+            -0.009152155369520187,
+            -0.010081729851663113,
+            -0.005350723396986723,
+            0.02237025462090969,
+            0.00523736048489809,
+            -0.009741641581058502,
+            -0.03213457018136978,
+            0.005101324990391731,
+            -0.016218435019254684,
+            -0.020858751609921455,
+            -0.014427303336560726,
+            -0.008766721934080124,
+            0.00010432918497826904,
+            -0.0113136051222682,
+            -0.022113298997282982,
+            -0.016218435019254684,
+            0.0007222153944894671,
+            -0.004621422849595547,
+            0.00804120022803545,
+            -0.005690811667591333,
+            0.031167207285761833,
+            -0.005932651925832033,
+            0.00530537823215127,
+            -0.006563704926520586,
+            0.004984183702617884,
+            0.004848148208111525,
+            -0.023186467587947845,
+            -0.005728599149733782,
+            0.006359652150422335,
+            -0.0013244551373645663,
+            0.00010574621410341933,
+            0.00646167853847146,
+            -0.012099587358534336,
+            0.005539661273360252,
+            0.003895900910720229,
+            -0.0006830107304267585,
+            -0.0014510435285046697,
+            0.005762607790529728,
+            0.005959103349596262,
+            0.015946364030241966,
+            -0.007890049368143082,
+            -0.017669478431344032,
+            -0.009144597686827183,
+            0.00047187256859615445,
+            -0.002110437024384737,
+            0.027675632387399673,
+            0.010565411299467087,
+            0.005516988690942526,
+            0.018138045445084572,
+            0.012477463111281395,
+            -0.0022200210951268673,
+            0.022989971563220024,
+            -0.0015974704874679446,
+            -0.015341762453317642,
+            0.009288189932703972,
+            -0.006609050091356039,
+            0.0014680479653179646,
+            -0.016717230901122093,
+            0.001082614529877901,
+            -0.007081395015120506,
+            0.0034159983042627573,
+            0.010240437462925911,
+            -0.001951729180291295,
+            0.013202984817326069,
+            -0.00954514555633068,
+            0.004889714531600475,
+            -0.0029852199368178844,
+            0.005789059214293957,
+            -0.023836413398385048,
+            0.006159377750009298,
+            -0.004821696784347296,
+            -0.0043002283200621605,
+            -0.002405180362984538,
+            0.006280297879129648,
+            -0.0032931887544691563,
+            0.013255887664854527,
+            -0.003586042672395706,
+            0.025529297068715096,
+            0.003841108875349164,
+            -0.0026999234687536955,
+            0.01410232950001955,
+            -0.0021274415776133537,
+            0.01036135759204626,
+            0.017200913280248642,
+            -0.02164473384618759,
+            -0.001862928387708962,
+            -0.01771482452750206,
+            0.00402437848970294,
+            0.00397147610783577,
+            0.006805545650422573,
+            -0.0016862713964655995,
+            0.0042435466311872005,
+            -0.02655711956322193,
+            -0.005411183461546898,
+            -0.006454120855778456,
+            0.014510435983538628,
+            -0.017064876854419708,
+            -0.018833335489034653,
+            -0.002526100492104888,
+            0.0037806485779583454,
+            0.0001020265044644475,
+            0.0008993447409011424,
+            0.0055736699141561985,
+            0.013490171171724796,
+            -0.001459545805118978,
+            -0.006302970461547375,
+            -0.011185127310454845,
+            -0.0017571230418980122,
+            0.01582544483244419,
+            -0.004194422625005245,
+            -0.015462683513760567,
+            -0.006794209126383066,
+            -0.013966294005513191,
+            -0.02702568657696247,
+            -0.00242029526270926,
+            -0.007281669415533543,
+            0.015228400006890297,
+            0.008970774710178375,
+            -0.0006919853040017188,
+            0.014261038042604923,
+            0.00027088981005363166,
+            0.010232879780232906,
+            -0.020148344337940216,
+            -0.024622395634651184,
+            -0.02311089262366295,
+            -0.010905499570071697,
+            0.003536918666213751,
+            -0.021176166832447052,
+            -0.01605216972529888,
+            0.02052622102200985,
+            -0.0048292544670403,
+            -0.0034802372101694345,
+            0.01073167659342289,
+            -0.004931280855089426,
+            -0.009114367887377739,
+            -0.007304341532289982,
+            -0.016248665750026703,
+            -0.0165660809725523,
+            0.01469181664288044,
+            0.003967697266489267,
+            -0.01549291331321001,
+            0.02658735029399395,
+            0.030487030744552612,
+            0.00922017265111208,
+            -0.008411518298089504,
+            -0.002813286380842328,
+            -0.009847446344792843,
+            -0.025136306881904602,
+            -0.0014633245300501585,
+            -0.007429040968418121,
+            -0.0040017059072852135,
+            -0.015764983370900154,
+            -0.005864634644240141,
+            -0.002218131674453616,
+            -0.005539661273360252,
+            -0.007995855063199997,
+            -0.002146335318684578,
+            0.03210433945059776,
+            0.009643393568694592,
+            0.010784579440951347,
+            0.0018393110949546099,
+            -0.010875268839299679,
+            0.014230807311832905,
+            -0.030623065307736397,
+            -0.016142860054969788,
+            0.013134967535734177,
+            0.001925277872942388,
+            0.010225323028862476,
+            -0.0023296051658689976,
+            0.0030437905807048082,
+            0.013051834888756275,
+            -0.004972847178578377,
+            0.015024347230792046,
+            0.00886496901512146,
+            -0.0017873531905934215,
+            -0.002206795383244753,
+            0.002907755319029093,
+            -0.002322047483175993,
+            0.008812067098915577,
+            0.01635446958243847,
+            -0.014971444383263588,
+            -0.00279439240694046,
+            0.007500837091356516,
+            0.0247282013297081,
+            -0.0026470208540558815,
+            -0.013021604157984257,
+            -0.0069907046854496,
+            0.021826114505529404,
+            0.007077616173774004,
+            -0.014638913795351982,
+            -0.0032610693015158176,
+            -0.006336979568004608,
+            0.031590428203344345,
+            0.02589205838739872,
+            0.02102501690387726,
+            -0.0013896387536078691,
+            -0.01865195669233799,
+            -0.004681882914155722,
+            -0.016974186524748802,
+            0.016248665750026703,
+            0.0027112597599625587,
+            -0.008910314179956913,
+            -0.0039412458427250385,
+            -0.015107479877769947,
+            -0.010043942369520664,
+            -0.0032535118516534567,
+            0.007421483285725117,
+            -0.017548557370901108,
+            -0.009242845699191093,
+            -0.003255401039496064,
+            -0.015417338348925114,
+            -0.002473197877407074,
+            0.01015730481594801,
+            -0.0065863775089383125,
+            -0.013679108582437038,
+            0.004976626019924879,
+            0.0007236324017867446,
+            0.012998932041227818,
+            -0.011925764381885529,
+            -0.026511775329709053,
+            -0.0021746759302914143,
+            0.011759499087929726,
+            0.0189089123159647,
+            -0.0008625017944723368,
+            -0.01862172596156597,
+            -0.03225548937916756,
+            -0.017427638173103333,
+            -0.012235622853040695,
+            -0.025529297068715096,
+            0.001894103130325675,
+            0.0020726495422422886,
+            0.0086458008736372,
+            -0.01505457703024149,
+            -0.043742917478084564,
+            0.005475422367453575,
+            -0.004326679278165102,
+            0.006336979568004608,
+            0.003963918425142765,
+            0.008018527179956436,
+            0.017397407442331314,
+            0.00670351879671216,
+            0.012273410335183144,
+            -0.00420198030769825,
+            -0.011819958686828613,
+            0.00852488074451685,
+            0.03470412641763687,
+            0.0057701654732227325,
+            -0.011880419217050076,
+            0.008955659344792366,
+            -0.014669143594801426,
+            -0.0009654730092734098,
+            0.003629498416557908,
+            -0.0012120370520278811,
+            -0.004043272230774164,
+            0.014654028229415417,
+            0.0009919243166223168,
+            -0.008494650945067406,
+            0.0008034586790017784,
+            0.02354922890663147,
+            -0.02007276937365532,
+            -0.001997074345126748,
+            -0.0067979879677295685,
+            -0.02631527930498123,
+            0.010709003545343876,
+            0.0014718268066644669,
+            -0.019543742761015892,
+            -0.01839500106871128,
+            0.0009612218709662557,
+            -0.014835408888757229,
+            0.03219502791762352,
+            0.01957397349178791,
+            -0.0038259937427937984,
+            -0.004296449478715658,
+            0.018969371914863586,
+            0.033192623406648636,
+            0.005093767773360014,
+            0.00524869654327631,
+            -0.011117110028862953,
+            0.009975924156606197,
+            0.01260594092309475,
+            0.0041528563015162945,
+            -0.000520996458362788,
+            0.017185797914862633,
+            -0.010293340310454369,
+            -0.015311532653868198,
+            -0.010875268839299679,
+            0.00218979106284678,
+            -0.014759833924472332,
+            -0.0045005022548139095,
+            -0.0030211179982870817,
+            -0.0009853114606812596,
+            0.02983708307147026,
+            -0.004999298602342606,
+            0.01798689365386963,
+            0.024320095777511597,
+            0.006571262609213591,
+            0.018757760524749756,
+            -0.016868380829691887,
+            -0.00021290946460794657,
+            0.0006858448032289743,
+            0.01909029111266136,
+            -0.0029058658983558416,
+            -0.00601200619712472,
+            -0.016460275277495384,
+            -0.005751271732151508,
+            -0.01176705677062273,
+            -0.03612493723630905,
+            0.02705591730773449,
+            -0.009061465039849281,
+            -0.024637511000037193,
+            -0.012658843770623207,
+            -0.028053509071469307,
+            -0.01366399321705103,
+            0.02353411354124546,
+            0.01844034530222416,
+            -0.0124850207939744,
+            -0.00780691672116518,
+            -0.02099478617310524,
+            -0.011404295451939106,
+            0.04873088002204895,
+            -0.0035255823750048876,
+            -0.02029949426651001,
+            0.009975924156606197,
+            -0.01889379695057869,
+            -0.015266187489032745,
+            0.012855338864028454,
+            0.006412554532289505,
+            0.04383360967040062,
+            0.011366507969796658,
+            0.00022991388686932623,
+            -0.01027822494506836,
+            0.000404563412303105,
+            -0.028809260576963425,
+            0.011638578958809376,
+            -0.014132560230791569,
+            -0.017382292076945305,
+            -0.0018648176919668913,
+            -0.011336278170347214,
+            -0.015311532653868198,
+            -0.0037107416428625584,
+            0.001894103130325675,
+            0.01236410066485405,
+            0.007217430509626865,
+            -0.008902756497263908,
+            0.01527374517172575,
+            0.018470576032996178,
+            -0.010127075016498566,
+            0.01933213323354721,
+            0.0025525519158691168,
+            0.006627943832427263,
+            -0.003502909792587161,
+            -0.01061831321567297,
+            -0.015508027747273445,
+            0.013852931559085846,
+            0.004523174837231636,
+            -0.016490506008267403,
+            0.0066014924086630344,
+            0.016384700313210487,
+            0.014858081936836243,
+            -0.008078987710177898,
+            -0.01198622491210699,
+            0.0007396921282634139,
+            0.005063537508249283,
+            -0.003856223775073886,
+            0.005033307243138552,
+            0.006378545891493559,
+            -0.006068687420338392,
+            -0.008517323061823845,
+            -0.0027377111837267876,
+            0.010965959168970585,
+            -0.0034802372101694345,
+            -0.018848450854420662,
+            0.003138259518891573,
+            -0.007651987485587597,
+            -0.008305712603032589,
+            0.02636062540113926,
+            0.0363667793571949,
+            -0.003128812648355961,
+            0.003111808327957988,
+            0.0034745691809803247,
+            0.0060422359965741634,
+            -0.010164862498641014,
+            0.012039126828312874,
+            0.00909925252199173,
+            -0.01609751395881176,
+            0.00792783685028553,
+            -0.015235957689583302,
+            -0.002306932583451271,
+            -0.012190277688205242,
+            0.008245253004133701,
+            -0.0004520340880844742,
+            0.004345573019236326,
+            0.014404630288481712,
+            -0.01505457703024149,
+            -0.01165369339287281,
+            -0.008585341274738312,
+            -0.009008562192320824,
+            0.027630288153886795,
+            0.01769970916211605,
+            -0.027615172788500786,
+            -0.000787398952525109,
+            -0.002512875013053417,
+            -0.023685263469815254,
+            0.0027641623746603727,
+            -0.0007160748937167227,
+            0.0017306717345491052,
+            0.0018251407891511917,
+            0.003895900910720229,
+            0.004644094966351986,
+            -0.00991546455770731,
+            0.013611091300845146,
+            0.013331462629139423,
+            -0.005611457396298647,
+            0.01445753313601017,
+            -0.01958908885717392,
+            0.007209872826933861,
+            -0.01585567370057106,
+            -0.008139447309076786,
+            0.006782873068004847,
+            0.024199174717068672,
+            0.0045458474196493626,
+            0.002652689116075635,
+            0.01517549715936184,
+            -1.436961792933289e-05,
+            0.0216598492115736,
+            0.010527623817324638,
+            0.0022237999364733696,
+            0.009650951251387596,
+            0.0025790033396333456,
+            0.005350723396986723,
+            -0.00977942906320095,
+            0.0009650006541050971,
+            -0.010542738251388073,
+            0.016490506008267403,
+            0.01457845326513052,
+            -0.0027112597599625587,
+            0.012507692910730839,
+            -0.006144262850284576,
+            -0.006488129962235689,
+            -0.01410232950001955,
+            0.005112661514431238,
+            0.009885234758257866,
+            -0.0038108788430690765,
+            0.002359835198149085,
+            0.005501873325556517,
+            -0.005135334096848965,
+            0.01339948084205389,
+            -0.005524545907974243,
+            0.003389547113329172,
+            0.02980685420334339,
+            -0.004345573019236326,
+            -0.0013556298799812794,
+            0.015568488277494907,
+            0.0029474322218447924,
+            -0.0036653964780271053,
+            0.003438670886680484,
+            -0.02608855441212654,
+            0.022022608667612076,
+            0.019725123420357704,
+            0.019468167796730995,
+            0.02421429008245468,
+            0.02335273288190365,
+            -0.007572633679956198,
+            -0.0016796585405245423,
+            0.009061465039849281,
+            0.0037220779340714216,
+            0.009688738733530045,
+            0.011381623335182667,
+            0.03594356030225754,
+            -0.002499649301171303,
+            0.005490537267178297,
+            -0.01961931772530079,
+            0.012961144559085369,
+            0.012681515887379646,
+            -0.004356909543275833,
+            0.0007760627195239067,
+            -0.022249335423111916,
+            0.013414595276117325,
+            -0.024395670741796494,
+            -0.0077540138736367226,
+            0.007791801821440458,
+            -0.01797178015112877,
+            0.031348586082458496,
+            -0.00478768814355135,
+            -0.012530365958809853,
+            0.009069022722542286,
+            0.00576638663187623,
+            0.005093767773360014,
+            -0.018561266362667084,
+            0.008728934451937675,
+            0.0011685813078656793,
+            -0.0035614806693047285,
+            -0.01039158832281828,
+            0.010467163287103176,
+            0.015976594761013985,
+            0.008827181532979012,
+            -0.00075150077464059,
+            -0.00804875697940588,
+            0.03621562942862511,
+            0.0008757274481467903,
+            0.0003894483670592308,
+            -0.0043946970254182816,
+            -0.0027282643131911755,
+            0.007882491685450077,
+            -0.00717964256182313,
+            -0.016188204288482666,
+            -0.014888311736285686,
+            -0.005267590284347534,
+            0.028083739802241325,
+            0.0036238301545381546,
+            0.013293675146996975,
+            -0.0075083947740495205,
+            -0.013293675146996975,
+            0.008351057767868042,
+            0.00518067879602313,
+            -0.0020972113125026226,
+            8.472686749882996e-05,
+            0.0027188172098249197,
+            -0.009378880262374878,
+            -0.034159984439611435,
+            -0.004235989414155483,
+            0.01676257699728012,
+            0.006733749061822891,
+            -0.006854669190943241,
+            0.01837988570332527,
+            -0.016702115535736084,
+            0.010671216063201427,
+            -0.0005975163076072931,
+            0.02283882163465023,
+            0.0163242407143116,
+            0.003030564868822694
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "f9e3ee71-ae4c-42ee-b161-771c512ad2b1",
+      "type": "child",
+      "source": {
+        "id": "65742957-6bda-446d-a61a-0fe20313c760",
+        "properties": {
+          "page_content": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity\n\nWilliam Fedus∗ liamfedus@google.com Barret Zoph∗ barretzoph@google.com Noam Shazeer noam@google.com Google, Mountain View, CA 94043, USA Editor: Alexander Clark\n\nAbstract\n\nIn deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) models defy this and instead select different parameters for each incoming example. The result is a sparsely-activated model—with an outrageous number of parameters—but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs, and training instability. We address these with the introduction of the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques mitigate the instabilities, and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large (Raffel et al., 2019) to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the \"Colossal Clean Crawled Corpus\", and achieve a 4x speedup over the T5-XXL model.12 Keywords: mixture-of-experts, natural language processing, sparsity, large-scale machine learning, distributed computing arXiv:2101.03961v3 [cs.LG] 16 Jun 2022 Contents\n\n1 Introduction 3 2 Switch Transformer 4 2.1 Simplifying Sparse Routing 5 2.2 Efficient Sparse Routing 6 2.3 Putting It All Together: The Switch Transformer 8 2.4 Improved Training and Fine-Tuning Techniques 8 3 Scaling Properties 11 3.1 Scaling Results on a Step-Basis 12 3.2 Scaling Results on a Time-Basis 13 3.3 Scaling Versus a Larger Dense Model 13 4 Downstream Results 14 4.1 Fine-Tuning 14 4.2 Distillation 16 4.3 Multilingual Learning 17 5 Designing Models with Data, Model, and Expert-Parallelism 18 5.1 Data Parallelism 20 5.2 Model Parallelism 20 5.3 Model and Data Parallelism 21 5.4 Expert and Data Parallelism 22 5.5 Expert, Model and Data Parallelism 22 5.6 Towards Trillion Parameter Models 22 6 Related Work 24 7 Discussion 25 8 Future Work 26 9 Conclusion 27 A Switch for Attention 27 B Preventing Token Dropping with No-Token-Left-Behind 29\n\nC Encouraging Exploration Across Experts 29 D Switch Transformers in Lower Compute Regimes 29 E Relation of Upstream to Downstream Model Performance 32 F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/switch_transformers.md"
+          },
+          "summary": "The Switch Transformer is a sparsely-activated model that simplifies the Mixture of Experts (MoE) approach, allowing for efficient scaling to trillion-parameter models with reduced communication and computational costs. It achieves significant speedups in pre-training and fine-tuning by activating only a subset of neural network weights for each input, maintaining a constant computational cost. The model demonstrates superior scaling properties and improved performance across various natural language processing tasks, including multilingual settings. It also allows for effective distillation into smaller dense models, preserving a portion of the quality gains. The architecture is designed to maximize parameter count while keeping floating point operations per example constant, making it suitable for both large-scale and smaller-scale applications.",
+          "headlines": {
+            "1 Introduction": [],
+            "2 Switch Transformer": [
+              "2.1 Simplifying Sparse Routing",
+              "2.2 Efficient Sparse Routing",
+              "2.3 Putting It All Together: The Switch Transformer",
+              "2.4 Improved Training and Fine-Tuning Techniques"
+            ],
+            "3 Scaling Properties": [
+              "3.1 Scaling Results on a Step-Basis",
+              "3.2 Scaling Results on a Time-Basis",
+              "3.3 Scaling Versus a Larger Dense Model"
+            ],
+            "4 Downstream Results": [
+              "4.1 Fine-Tuning",
+              "4.2 Distillation",
+              "4.3 Multilingual Learning"
+            ],
+            "5 Designing Models with Data, Model, and Expert-Parallelism": [
+              "5.1 Data Parallelism",
+              "5.2 Model Parallelism",
+              "5.3 Model and Data Parallelism",
+              "5.4 Expert and Data Parallelism",
+              "5.5 Expert, Model and Data Parallelism",
+              "5.6 Towards Trillion Parameter Models"
+            ],
+            "6 Related Work": [],
+            "7 Discussion": [],
+            "8 Future Work": [],
+            "9 Conclusion": [],
+            "A Switch for Attention": [],
+            "B Preventing Token Dropping with No-Token-Left-Behind": [],
+            "C Encouraging Exploration Across Experts": [],
+            "D Switch Transformers in Lower Compute Regimes": [],
+            "E Relation of Upstream to Downstream Model Performance": [],
+            "F Pseudo Code for Switch Transformers": []
+          },
+          "summary_embedding": [
+            -0.0060447100549936295,
+            -0.00390331051312387,
+            -0.01784370467066765,
+            -0.0021820589900016785,
+            0.013227885589003563,
+            0.011338187381625175,
+            0.0165580902248621,
+            0.024240797385573387,
+            -0.005986624862998724,
+            0.03596621751785278,
+            -0.004852031823247671,
+            -0.0068811336532235146,
+            -0.003574162255972624,
+            0.014242436736822128,
+            0.011578272096812725,
+            -0.033302053809165955,
+            -0.024519605562090874,
+            0.01050176378339529,
+            0.016093410551548004,
+            -0.006795942317694426,
+            0.007473600562661886,
+            -0.030513975769281387,
+            -0.06124480068683624,
+            -0.006838538218289614,
+            -0.0026099516544491053,
+            0.01029265858232975,
+            0.0010329445358365774,
+            -0.010873507708311081,
+            -0.053469158709049225,
+            0.023311438038945198,
+            0.025294071063399315,
+            0.005092116538435221,
+            -0.029584616422653198,
+            0.0009361362317577004,
+            -0.05161044001579285,
+            -0.04448534920811653,
+            0.0379488542675972,
+            0.012561844661831856,
+            -0.04364892467856407,
+            0.039249956607818604,
+            -0.004875265993177891,
+            -0.0022440163884311914,
+            -0.021499186754226685,
+            -0.0034909071400761604,
+            -0.007039899472147226,
+            -0.009851212613284588,
+            -0.02874819189310074,
+            0.0014037203509360552,
+            0.021886419504880905,
+            -0.0056962003000080585,
+            0.023264970630407333,
+            -0.024271776899695396,
+            0.01936165988445282,
+            0.04240977764129639,
+            -0.06957805901765823,
+            0.024659009650349617,
+            0.017549408599734306,
+            0.05120771750807762,
+            0.06511713564395905,
+            0.018339363858103752,
+            0.021669568493962288,
+            -0.014141756109893322,
+            -0.017766257748007774,
+            -0.011872569099068642,
+            0.004673904739320278,
+            -0.014970434829592705,
+            -0.01891246810555458,
+            0.008449427783489227,
+            -0.020461400970816612,
+            0.008209343999624252,
+            -0.05576157942414284,
+            -0.014730350114405155,
+            -0.034200433641672134,
+            -0.01140014547854662,
+            0.017642343416810036,
+            -0.018215449526906013,
+            0.030126741155982018,
+            0.02997184917330742,
+            0.002638994250446558,
+            0.0007822110201232135,
+            0.021049996837973595,
+            0.006509389728307724,
+            -0.0053438181057572365,
+            0.012321759946644306,
+            0.028159597888588905,
+            -0.01450575515627861,
+            -0.010106786154210567,
+            -0.03454120084643364,
+            0.01861817203462124,
+            0.01589204929769039,
+            -0.04194509983062744,
+            -0.012569589540362358,
+            -0.013824224472045898,
+            0.032341714948415756,
+            -0.009549170732498169,
+            0.04098476096987724,
+            -0.010083552449941635,
+            0.001383390510454774,
+            -0.031149037182331085,
+            -0.009463978931307793,
+            0.017069239169359207,
+            0.006172496825456619,
+            -0.016589069738984108,
+            -0.0012846461031585932,
+            0.016170857474207878,
+            0.006617815233767033,
+            -0.030142230913043022,
+            0.04488807171583176,
+            0.023497309535741806,
+            0.055296897888183594,
+            0.05579255893826485,
+            -0.037391237914562225,
+            -0.002445377642288804,
+            0.0344172865152359,
+            -0.0228312686085701,
+            -0.015040136873722076,
+            -0.034014564007520676,
+            -0.01768881268799305,
+            0.035935238003730774,
+            -0.0318460576236248,
+            0.01556677371263504,
+            0.028066661208868027,
+            -0.025712283328175545,
+            0.003488970920443535,
+            -0.03943582624197006,
+            0.003622566582635045,
+            0.01220559049397707,
+            0.006095050368458033,
+            -0.0026293133851140738,
+            -0.013615119270980358,
+            0.008906363509595394,
+            0.003055269829928875,
+            0.00800798274576664,
+            -0.04615819454193115,
+            0.018478767946362495,
+            -0.00454224506393075,
+            0.04033420979976654,
+            0.042502716183662415,
+            -0.02730768360197544,
+            -0.016743963584303856,
+            0.008557853288948536,
+            0.012104909867048264,
+            0.04188314080238342,
+            -0.007287728600203991,
+            -0.00121784838847816,
+            -0.0007836631848476827,
+            0.007322579622268677,
+            0.047552235424518585,
+            0.004646798130124807,
+            0.0020697612781077623,
+            -0.027230238541960716,
+            0.014978179708123207,
+            -0.019733402878046036,
+            0.027648448944091797,
+            0.023373395204544067,
+            0.02942972257733345,
+            -0.04048910364508629,
+            0.017518429085612297,
+            -0.011818356812000275,
+            -0.005568413063883781,
+            -0.002726121572777629,
+            -0.02856232039630413,
+            -0.0074426219798624516,
+            0.033983584493398666,
+            0.015365412458777428,
+            0.012956822291016579,
+            0.03271345794200897,
+            0.0012052633101120591,
+            -0.042502716183662415,
+            0.03804178908467293,
+            0.024550583213567734,
+            0.04640602692961693,
+            0.023466331884264946,
+            0.010796061716973782,
+            0.006571347359567881,
+            0.013375034555792809,
+            -0.019377149641513824,
+            0.015907539054751396,
+            0.02560385875403881,
+            0.009750531986355782,
+            -0.03167567402124405,
+            -0.0009501734166406095,
+            0.01178737822920084,
+            -0.004546117503196001,
+            0.03701949119567871,
+            0.04104671627283096,
+            0.020569827407598495,
+            0.05030933395028114,
+            0.0020155487582087517,
+            0.017936641350388527,
+            0.005936284549534321,
+            0.02820606529712677,
+            0.030467506498098373,
+            -0.005285732913762331,
+            0.005076627247035503,
+            -0.01136916596442461,
+            -0.03385967016220093,
+            0.014110777527093887,
+            0.01561324205249548,
+            -0.008077683858573437,
+            -0.01697630248963833,
+            0.006033093202859163,
+            0.009169681929051876,
+            -0.01795213110744953,
+            -0.017456471920013428,
+            0.005971135571599007,
+            0.018866000697016716,
+            -0.013576395809650421,
+            0.024689987301826477,
+            -0.04442339017987251,
+            0.029119934886693954,
+            -0.033209118992090225,
+            -0.003384418087080121,
+            -0.010641167871654034,
+            -0.0030843124259263277,
+            0.0041666291654109955,
+            0.011314953677356243,
+            0.013406013138592243,
+            0.002042655134573579,
+            -0.013909416273236275,
+            0.024519605562090874,
+            0.018447788432240486,
+            0.012073930352926254,
+            -0.03148980438709259,
+            -0.014397329650819302,
+            -0.0015673263696953654,
+            -0.008743725717067719,
+            -0.034138478338718414,
+            0.026966920122504234,
+            0.014048820361495018,
+            -0.03844451159238815,
+            0.017797237262129784,
+            0.029538147151470184,
+            0.014420564286410809,
+            -0.004790074657648802,
+            -0.0463130883872509,
+            0.04045812413096428,
+            -0.030436528846621513,
+            -0.033333033323287964,
+            -0.010478530079126358,
+            -0.0003146269591525197,
+            0.02894955314695835,
+            -0.0064358157105743885,
+            -0.050835974514484406,
+            0.01184933539479971,
+            0.020631784573197365,
+            0.00010763872705865651,
+            0.005731051322072744,
+            -0.019593998789787292,
+            0.00028631053282879293,
+            -0.03395260497927666,
+            0.0090844901278615,
+            0.026750069111585617,
+            0.008867640048265457,
+            -0.016418687999248505,
+            0.03351890295743942,
+            -0.0004121613164898008,
+            -0.018695618957281113,
+            -0.003889757441356778,
+            0.006885006092488766,
+            0.034912943840026855,
+            0.004836542531847954,
+            -0.02946070022881031,
+            0.05665995925664902,
+            -0.02100352756679058,
+            0.021344292908906937,
+            0.05631919577717781,
+            -0.019439106807112694,
+            -0.008472662419080734,
+            -0.002065889071673155,
+            0.0026196325197815895,
+            -0.0260530486702919,
+            -0.008937342092394829,
+            0.032341714948415756,
+            0.041728246957063675,
+            -0.018742086365818977,
+            -0.001398879918269813,
+            0.006780453026294708,
+            0.009425255469977856,
+            0.004948840010911226,
+            0.02692045085132122,
+            -0.012128143571317196,
+            -0.014559967443346977,
+            0.05198218300938606,
+            0.028314489871263504,
+            -0.012081675231456757,
+            -0.0164031982421875,
+            0.019888296723365784,
+            -0.024844881147146225,
+            -0.020244551822543144,
+            -0.02431824430823326,
+            -0.020414933562278748,
+            -0.0042518205009400845,
+            0.009525936096906662,
+            0.037484172731637955,
+            -0.03968365862965584,
+            0.012747716158628464,
+            0.021700548008084297,
+            0.02981695532798767,
+            -0.019532041624188423,
+            -0.05910727381706238,
+            -0.0016128262504935265,
+            0.03296129032969475,
+            -0.015458349138498306,
+            -0.049379974603652954,
+            0.015164051204919815,
+            0.0260530486702919,
+            0.05957195535302162,
+            -0.001461805310100317,
+            -0.014954946003854275,
+            0.061926331371068954,
+            0.03026614524424076,
+            -0.004383479710668325,
+            -0.009494957514107227,
+            -0.0127864396199584,
+            0.003396035172045231,
+            0.00465454300865531,
+            -0.03364282101392746,
+            -0.041170634329319,
+            -0.011841590516269207,
+            -0.010284913703799248,
+            0.008108663372695446,
+            -0.040922801941633224,
+            0.03531566634774208,
+            -0.021917399019002914,
+            0.010950954630970955,
+            0.015473837964236736,
+            0.037391237914562225,
+            -0.021344292908906937,
+            -0.01284839678555727,
+            -0.01572941243648529,
+            -0.024968795478343964,
+            -0.014010096900165081,
+            0.021917399019002914,
+            0.007682706229388714,
+            3.4699722164077684e-05,
+            0.010075807571411133,
+            -0.024659009650349617,
+            -0.051052823662757874,
+            0.0291044469922781,
+            0.0015605497173964977,
+            -0.015938518568873405,
+            -0.004623564425855875,
+            -0.01977987214922905,
+            -0.01122201792895794,
+            -0.012608313001692295,
+            -0.039249956607818604,
+            0.021111954003572464,
+            -0.01135367713868618,
+            0.015876561403274536,
+            0.0009250032599084079,
+            0.003659353591501713,
+            0.028624277561903,
+            0.01116006076335907,
+            0.0051811798475682735,
+            -0.03255856782197952,
+            -0.018277406692504883,
+            0.010122275911271572,
+            -0.00561100896447897,
+            -0.018339363858103752,
+            -0.013351799920201302,
+            -0.05037129297852516,
+            -0.03305422514677048,
+            -0.008054450154304504,
+            -0.05415068939328194,
+            0.0020697612781077623,
+            -0.014296649023890495,
+            -0.02232012152671814,
+            -0.005990497302263975,
+            0.03107159025967121,
+            -0.003235333366319537,
+            0.025294071063399315,
+            0.0076594725251197815,
+            0.037360258400440216,
+            0.040117356926202774,
+            0.008426194079220295,
+            0.03065337985754013,
+            -0.0029991210903972387,
+            -0.016867877915501595,
+            0.05486319959163666,
+            -0.01881953328847885,
+            0.013529927469789982,
+            -0.008604321628808975,
+            0.008604321628808975,
+            0.03299226611852646,
+            -0.003794885240495205,
+            0.02007416822016239,
+            -0.0311955064535141,
+            -0.07998688519001007,
+            0.01637221872806549,
+            0.028159597888588905,
+            -0.06418777257204056,
+            0.014691627584397793,
+            -0.020275529474020004,
+            0.02052335813641548,
+            0.02411688305437565,
+            -0.0009114501299336553,
+            -0.014064309187233448,
+            0.005595519673079252,
+            0.0020833145827054977,
+            -0.01456771232187748,
+            -0.002691270550712943,
+            -0.009417511522769928,
+            0.008774704299867153,
+            -0.017007282003760338,
+            -0.013406013138592243,
+            -0.034231413155794144,
+            0.010873507708311081,
+            0.01233724970370531,
+            -0.003969140350818634,
+            0.0011500825639814138,
+            -0.03277541697025299,
+            -0.007876322604715824,
+            -0.02145271934568882,
+            0.03382869064807892,
+            0.012058441527187824,
+            -0.030699847266077995,
+            -0.007814365439116955,
+            0.015264731831848621,
+            -0.007876322604715824,
+            0.0008906363509595394,
+            0.024132372811436653,
+            0.02103450708091259,
+            0.0028364830650389194,
+            -0.0009700191440060735,
+            0.02438020147383213,
+            0.007427132688462734,
+            -0.01720864325761795,
+            0.028794659301638603,
+            0.014288905076682568,
+            0.060160547494888306,
+            -0.03271345794200897,
+            0.001297231181524694,
+            -0.04135650396347046,
+            -0.031118059530854225,
+            0.006408709101378918,
+            0.014350862242281437,
+            0.007597515359520912,
+            -0.028252532705664635,
+            0.005978880450129509,
+            0.02004319056868553,
+            0.01469937153160572,
+            0.009704063646495342,
+            0.0037580979987978935,
+            -0.036740683019161224,
+            -0.000691695255227387,
+            -0.011214273050427437,
+            -0.0038781403563916683,
+            0.035656433552503586,
+            -0.008186109364032745,
+            -0.05151750147342682,
+            -0.0006418389966711402,
+            -0.03255856782197952,
+            -0.0353776253759861,
+            0.021313315257430077,
+            -0.00839521549642086,
+            -0.0039207362569868565,
+            -0.013986863195896149,
+            -0.000537770101800561,
+            -0.010524998418986797,
+            0.027509044855833054,
+            0.028345469385385513,
+            0.022490503266453743,
+            -0.05384090170264244,
+            0.009479468688368797,
+            0.00023766436788719147,
+            -0.05817791447043419,
+            0.0009709871956147254,
+            -0.007117345929145813,
+            0.011191039346158504,
+            -0.020213572308421135,
+            -0.02414786070585251,
+            0.011500826105475426,
+            0.03392162546515465,
+            0.014304393902420998,
+            0.0321558453142643,
+            -0.013367289677262306,
+            -0.09250226616859436,
+            0.018292896449565887,
+            -0.008689512498676777,
+            0.04126356914639473,
+            0.00865078903734684,
+            -0.004278926644474268,
+            -0.015845581889152527,
+            -0.00865078903734684,
+            -0.006943091284483671,
+            0.01283290795981884,
+            0.004530628211796284,
+            -0.005626498255878687,
+            0.0016631665639579296,
+            0.013708055019378662,
+            -0.005967263597995043,
+            0.007481344975531101,
+            0.003651608945801854,
+            -0.013189162127673626,
+            -0.06059424951672554,
+            -0.04055105894804001,
+            -0.018540725111961365,
+            0.050835974514484406,
+            -0.0069159846752882,
+            -0.046684831380844116,
+            -0.010850274004042149,
+            0.05058814212679863,
+            0.012259802781045437,
+            -0.0334569476544857,
+            -0.0007841471815481782,
+            0.0050882440991699696,
+            0.015458349138498306,
+            0.004546117503196001,
+            -0.027292195707559586,
+            0.02820606529712677,
+            0.0008412640891037881,
+            0.02300165221095085,
+            -0.018339363858103752,
+            -0.005312839522957802,
+            0.035780347883701324,
+            -0.024844881147146225,
+            0.023791607469320297,
+            0.0055219451896846294,
+            0.031629208475351334,
+            0.002952652983367443,
+            0.010060317814350128,
+            0.004704883322119713,
+            0.0186491496860981,
+            0.004275054205209017,
+            0.0276639387011528,
+            0.011462102644145489,
+            0.006428070832043886,
+            0.021623101085424423,
+            -0.009378788061439991,
+            -0.014513500034809113,
+            0.01885051093995571,
+            -0.0014695499558001757,
+            0.025232113897800446,
+            -0.020538847893476486,
+            0.002726121572777629,
+            -0.0016883367206901312,
+            -0.017967619001865387,
+            -0.040922801941633224,
+            -0.0007405834621749818,
+            0.021747015416622162,
+            -0.011671207845211029,
+            -0.0228312686085701,
+            -0.013715799897909164,
+            -0.01585332676768303,
+            0.0038490979932248592,
+            0.034169457852840424,
+            -0.0398695282638073,
+            0.00963436160236597,
+            -0.000830615172162652,
+            -0.03125746175646782,
+            -0.0239619892090559,
+            -0.01093546487390995,
+            0.014614180661737919,
+            -0.01652711257338524,
+            -0.021173911169171333,
+            -0.028469383716583252,
+            -0.008410705253481865,
+            0.006048582494258881,
+            -0.05065010115504265,
+            -0.004054331686347723,
+            0.019965743646025658,
+            0.0276639387011528,
+            -0.003099801717326045,
+            -0.02177799493074417,
+            0.0012468908680602908,
+            0.022351099178195,
+            0.02936776541173458,
+            0.014226946979761124,
+            0.027586491778492928,
+            -0.002178186783567071,
+            -0.0011675080750137568,
+            -0.0073148347437381744,
+            0.01307299267500639,
+            0.0011287847300991416,
+            0.036616768687963486,
+            0.0122210793197155,
+            -0.020306508988142014,
+            0.0013959755888208747,
+            -0.006431943271309137,
+            0.029181893914937973,
+            -0.02270735427737236,
+            -9.832092473516241e-05,
+            0.03531566634774208,
+            0.027741385623812675,
+            0.004933350719511509,
+            -0.0143121387809515,
+            -0.05142456665635109,
+            0.021592123433947563,
+            -0.01765783317387104,
+            0.031025122851133347,
+            0.016666516661643982,
+            0.003397971158847213,
+            -0.0034212053287774324,
+            0.012368228286504745,
+            -0.022087780758738518,
+            -0.004406713880598545,
+            -0.021142931655049324,
+            0.04389675334095955,
+            -0.043184246867895126,
+            0.008472662419080734,
+            0.029445212334394455,
+            0.011926782317459583,
+            -0.019082851707935333,
+            -0.005425137002021074,
+            0.029212871566414833,
+            -0.00162637943867594,
+            0.01308848150074482,
+            0.0058665829710662365,
+            -0.026935940608382225,
+            0.01202746294438839,
+            -0.031660184264183044,
+            0.02177799493074417,
+            -0.013436991721391678,
+            0.018447788432240486,
+            0.03342596814036369,
+            -0.0030397805385291576,
+            -0.0038239278364926577,
+            -0.016263794153928757,
+            -0.00969631876796484,
+            0.0662323608994484,
+            0.01054823212325573,
+            0.03726732358336449,
+            0.0018974426202476025,
+            -0.009773765690624714,
+            0.007969259284436703,
+            0.0063041565008461475,
+            -0.012329504825174809,
+            0.010494019836187363,
+            0.024178840219974518,
+            0.0012730291346088052,
+            -0.002412462839856744,
+            -0.019408127292990685,
+            -0.03438630700111389,
+            0.003727119415998459,
+            -0.030932186171412468,
+            0.022691864520311356,
+            0.009208405390381813,
+            0.017456471920013428,
+            0.008240322582423687,
+            -0.015907539054751396,
+            0.03200095146894455,
+            0.05461536720395088,
+            0.0019768255297094584,
+            0.013375034555792809,
+            0.007938280701637268,
+            0.007036027032881975,
+            -0.011338187381625175,
+            0.04702559858560562,
+            0.000662652775645256,
+            0.005169562995433807,
+            0.022536970674991608,
+            0.02145271934568882,
+            0.022939693182706833,
+            0.002304037567228079,
+            -0.011694442480802536,
+            0.010478530079126358,
+            -0.0006302220281213522,
+            0.004445436876267195,
+            -0.010695381090044975,
+            -0.027865299955010414,
+            0.007492962293326855,
+            0.01302652433514595,
+            0.022598929703235626,
+            0.021592123433947563,
+            -0.03178409859538078,
+            -0.01855621486902237,
+            0.03364282101392746,
+            -0.02997184917330742,
+            0.03612111136317253,
+            -0.0014424435794353485,
+            -0.023807097226381302,
+            -0.0058356039226055145,
+            0.016929835081100464,
+            0.02656419761478901,
+            -0.029181893914937973,
+            0.01804506592452526,
+            0.029693040996789932,
+            -0.0074426219798624516,
+            0.0005939188995398581,
+            0.012112653814256191,
+            0.02447313815355301,
+            -0.009293596260249615,
+            -0.0024027819745242596,
+            -0.0013320822035893798,
+            -0.011926782317459583,
+            -0.018370341509580612,
+            -0.0044725434854626656,
+            -0.022629907354712486,
+            0.005564541090279818,
+            -0.007365175057202578,
+            -0.003481226274743676,
+            -0.008581086993217468,
+            0.01154729351401329,
+            0.001979729626327753,
+            -0.019857317209243774,
+            -0.022335609421133995,
+            0.01652711257338524,
+            0.03887821361422539,
+            -0.018215449526906013,
+            -0.02106548473238945,
+            0.024442158639431,
+            -0.01759587600827217,
+            -0.01601596549153328,
+            0.036678727716207504,
+            0.013739033602178097,
+            0.007736918982118368,
+            -0.03358086198568344,
+            -0.030049296095967293,
+            -0.03274443745613098,
+            0.006296411622315645,
+            0.03785591572523117,
+            -0.0028752065263688564,
+            0.02569679357111454,
+            0.03701949119567871,
+            0.0057891360484063625,
+            -0.028934063389897346,
+            -0.014823286794126034,
+            -0.045104920864105225,
+            0.060160547494888306,
+            0.0011481463443487883,
+            0.05161044001579285,
+            0.016294771805405617,
+            -0.01456771232187748,
+            0.00419760774821043,
+            -0.013251119293272495,
+            0.026192452758550644,
+            0.02280029095709324,
+            0.025929134339094162,
+            0.006582964211702347,
+            0.022505993023514748,
+            -0.0037522895727306604,
+            -0.012197845615446568,
+            0.007307090330868959,
+            0.014536733739078045,
+            0.0002497654058970511,
+            -0.011392400600016117,
+            0.015783624723553658,
+            -0.01307299267500639,
+            -0.011361422017216682,
+            0.023295948281884193,
+            0.01368482131510973,
+            -0.012344993650913239,
+            0.014064309187233448,
+            -0.008565598167479038,
+            0.01239920686930418,
+            -0.023915521800518036,
+            0.022691864520311356,
+            -0.03262052312493324,
+            0.009587894193828106,
+            0.0015053689712658525,
+            -0.0073148347437381744,
+            0.024550583213567734,
+            -0.0027454833034425974,
+            -0.05071205645799637,
+            0.03550153970718384,
+            -0.008015726692974567,
+            0.0037213109899312258,
+            0.006381602957844734,
+            0.011392400600016117,
+            -0.0012623801594600081,
+            0.03807276859879494,
+            0.01239920686930418,
+            0.015969496220350266,
+            0.02199484594166279,
+            0.011911292560398579,
+            -0.008433938957750797,
+            -0.004096927121281624,
+            -0.0016137943603098392,
+            -0.01784370467066765,
+            0.025944624096155167,
+            0.0018674320308491588,
+            0.03962169960141182,
+            -0.018168980255723,
+            -0.02300165221095085,
+            0.028066661208868027,
+            0.007597515359520912,
+            -0.02351279929280281,
+            -0.0017377089243382215,
+            0.01202746294438839,
+            -0.009185170754790306,
+            0.027385130524635315,
+            0.01196550577878952,
+            0.019113829359412193,
+            -0.018060555681586266,
+            -0.004340883810073137,
+            -0.005072754807770252,
+            -0.015938518568873405,
+            0.02238207869231701,
+            0.020771188661456108,
+            0.011152315884828568,
+            -0.011880313977599144,
+            -0.01307299267500639,
+            0.020600805059075356,
+            0.035036858171224594,
+            -0.0002744515077210963,
+            0.005672966130077839,
+            0.017487451434135437,
+            0.002373739378526807,
+            -0.04470219835639,
+            0.016232814639806747,
+            -0.014025585725903511,
+            0.011214273050427437,
+            0.019593998789787292,
+            -0.021917399019002914,
+            -0.03196997195482254,
+            -0.011539548635482788,
+            -0.0414804182946682,
+            0.00493722315877676,
+            -0.0069159846752882,
+            0.013576395809650421,
+            -0.010176488198339939,
+            -0.0014831031439825892,
+            0.016666516661643982,
+            0.0038103745318949223,
+            0.02177799493074417,
+            -0.011678952723741531,
+            -0.012073930352926254,
+            -0.020213572308421135,
+            -0.020786678418517113,
+            0.004278926644474268,
+            -0.030405549332499504,
+            -0.004488032776862383,
+            0.021375272423028946,
+            -0.04783104360103607,
+            -0.03689557686448097,
+            0.02898053266108036,
+            -0.008503641001880169,
+            0.020755698904395103,
+            0.022955182939767838,
+            -0.015094350092113018,
+            -0.0008717587334103882,
+            -0.018695618957281113,
+            0.02258343994617462,
+            -0.007837600074708462,
+            0.026300877332687378,
+            -0.0027551641687750816,
+            0.0006292539183050394,
+            -0.02659517526626587,
+            0.020306508988142014,
+            0.005227648187428713,
+            0.012275291606783867,
+            -0.02390003204345703,
+            0.006815304048359394,
+            -0.03133491054177284,
+            -0.01784370467066765,
+            0.0180140882730484,
+            -0.011152315884828568,
+            0.027989214286208153,
+            -0.006060199346393347,
+            0.028438406065106392,
+            0.022722844034433365,
+            0.024751944467425346,
+            0.02676555886864662,
+            -0.02363671362400055,
+            0.008495896123349667,
+            0.03726732358336449,
+            0.00865078903734684,
+            -0.015876561403274536,
+            0.008170620538294315,
+            -0.04863648861646652,
+            -0.008062195032835007,
+            0.05030933395028114,
+            0.002127846470102668,
+            0.023141054436564445,
+            0.016294771805405617,
+            -0.002873270306736231,
+            -0.005688455421477556,
+            0.010269423946738243,
+            -0.04296739399433136,
+            -0.023172033950686455,
+            0.022629907354712486,
+            -0.021561143919825554,
+            6.54212328754511e-07,
+            0.03122648410499096,
+            0.012120398692786694,
+            -0.0009109660750254989,
+            -0.05108380317687988,
+            0.02100352756679058,
+            0.01178737822920084,
+            0.014443797990679741,
+            -0.010300402529537678,
+            0.026068538427352905,
+            0.01733255758881569,
+            0.038413532078266144,
+            0.012755461037158966,
+            -0.002679653698578477,
+            -0.03782493621110916,
+            0.009177426807582378,
+            -0.021685058251023293,
+            0.0011471783509477973,
+            0.0074503663927316666,
+            -0.02273833192884922,
+            -0.024782923981547356,
+            0.0043912241235375404,
+            0.004588713403791189,
+            0.011880313977599144,
+            0.022087780758738518,
+            0.009417511522769928,
+            -0.021204888820648193,
+            0.015404135920107365,
+            -0.05446047708392143,
+            -0.015450604259967804,
+            -0.023884542286396027,
+            0.02566581591963768,
+            0.020012211054563522,
+            0.010184233076870441,
+            0.005545179359614849,
+            0.010904486291110516,
+            -0.020988039672374725,
+            -0.009239383973181248,
+            0.020631784573197365,
+            0.011074868962168694,
+            -0.012840652838349342,
+            0.04113965481519699,
+            -0.00011126903700642288,
+            0.031210994347929955,
+            -0.02280029095709324,
+            0.002941036131232977,
+            -0.001241082325577736,
+            -0.03649285435676575,
+            0.009130958467721939,
+            0.01981084980070591,
+            0.02151467651128769,
+            -0.01861817203462124,
+            0.0024143988266587257,
+            0.01048627495765686,
+            0.004766840487718582,
+            0.038351576775312424,
+            0.027121812105178833,
+            -0.044206541031599045,
+            0.011663463898003101,
+            0.012995545752346516,
+            -0.009363298304378986,
+            0.048419639468193054,
+            -0.017518429085612297,
+            0.017673322930932045,
+            0.008240322582423687,
+            -0.01798310875892639,
+            0.02318752370774746,
+            -0.01469937153160572,
+            -0.013282098807394505,
+            0.00650551775470376,
+            0.009192915633320808,
+            -0.0001770381786627695,
+            -0.037577107548713684,
+            -0.017859194427728653,
+            -0.005719434004276991,
+            0.0038297362625598907,
+            0.01562873087823391,
+            0.013498948886990547,
+            0.009967382065951824,
+            0.005161818582564592,
+            -0.00945623405277729,
+            -0.0016089539276435971,
+            0.008124152198433876,
+            -0.0005532593932002783,
+            0.0019216446671634912,
+            -0.01926872320473194,
+            0.02698240801692009,
+            0.02572777308523655,
+            -0.0018354853382334113,
+            -0.04652994126081467,
+            -0.0036709706764668226,
+            0.0037077576853334904,
+            -0.017735280096530914,
+            -0.0035315665882080793,
+            -0.04519785940647125,
+            -0.01810702309012413,
+            0.0143121387809515,
+            -0.014614180661737919,
+            -0.0460033044219017,
+            0.009076746180653572,
+            -0.013886182568967342,
+            0.0063970922492444515,
+            -0.00932457484304905,
+            0.005762029904872179,
+            0.003651608945801854,
+            0.005680711008608341,
+            -0.005909178406000137,
+            0.05108380317687988,
+            -0.03466511517763138,
+            0.028051171451807022,
+            0.008565598167479038,
+            -0.012437930330634117,
+            -0.02261441759765148,
+            0.00887538492679596,
+            -0.044981006532907486,
+            -0.033209118992090225,
+            0.016743963584303856,
+            -0.01781272701919079,
+            0.011074868962168694,
+            0.06161654368042946,
+            0.007330324035137892,
+            -0.023342415690422058,
+            -0.01233724970370531,
+            -0.021049996837973595,
+            -0.04191412031650543,
+            0.003442503046244383,
+            0.005463860463351011,
+            -0.004948840010911226,
+            -0.03807276859879494,
+            -0.021716037765145302,
+            0.04615819454193115,
+            -0.012004229240119457,
+            0.012422440573573112,
+            0.01080380566418171,
+            -0.0019410063978284597,
+            -0.04860550910234451,
+            -0.021019017323851585,
+            0.0395597442984581,
+            0.021886419504880905,
+            -0.008426194079220295,
+            0.0028364830650389194,
+            -0.018277406692504883,
+            -0.003289545886218548,
+            0.006296411622315645,
+            -0.009107724763453007,
+            -0.03562545403838158,
+            0.02067825198173523,
+            0.0054832217283546925,
+            -0.023373395204544067,
+            -0.025805220007896423,
+            -0.03296129032969475,
+            0.009944148361682892,
+            0.02453509531915188,
+            -0.01968693546950817,
+            0.005823987070471048,
+            0.00027203132049180567,
+            -0.000881923595443368,
+            0.00042740863864310086,
+            -0.009959637187421322,
+            -0.019764382392168045,
+            -0.020151615142822266,
+            0.03268248215317726,
+            -0.0060059865936636925,
+            0.0074387495405972,
+            -0.03630698472261429,
+            -0.010974188335239887,
+            0.011322698555886745,
+            -0.002395037328824401,
+            -0.013452480547130108,
+            -0.009092235006392002,
+            -0.01739451475441456,
+            -0.009882191196084023,
+            -0.007876322604715824,
+            0.022660886868834496,
+            0.006168624851852655,
+            0.005409647710621357,
+            0.0023795480374246836,
+            -0.008581086993217468,
+            0.02898053266108036,
+            0.017874684184789658,
+            -0.014188224449753761,
+            -0.00920066051185131,
+            0.03451022133231163,
+            0.02665713243186474,
+            -0.009115469641983509,
+            0.01798310875892639,
+            0.023264970630407333,
+            -0.0010484338272362947,
+            -0.01196550577878952,
+            0.001922612776979804,
+            -0.0009550138493068516,
+            -0.0032972905319184065,
+            0.00845717266201973,
+            -0.015365412458777428,
+            0.0315207801759243,
+            0.04287445917725563,
+            -0.002145271748304367,
+            0.009587894193828106,
+            -0.003727119415998459,
+            -0.008472662419080734,
+            0.017069239169359207,
+            0.009316830895841122,
+            0.04625113308429718,
+            0.01098193321377039,
+            0.005680711008608341,
+            -0.033983584493398666,
+            -0.001229465357027948,
+            -0.01184933539479971,
+            0.03308520466089249,
+            0.02061629481613636,
+            0.00889087375253439,
+            -0.008379725739359856,
+            0.01344473659992218,
+            0.008341003209352493,
+            0.00661007035523653,
+            0.020569827407598495,
+            -0.016542602330446243,
+            -0.017286090180277824,
+            0.030467506498098373,
+            -0.00672236829996109,
+            0.01289486512541771,
+            0.02029101923108101,
+            -0.002089123008772731,
+            0.010842529125511646,
+            0.07100307941436768,
+            -0.006184114143252373,
+            0.015713922679424286,
+            -0.019082851707935333,
+            0.0015789433382451534,
+            -0.007640110794454813,
+            -0.03370477631688118,
+            0.018246427178382874,
+            0.0039962464943528175,
+            0.00865078903734684,
+            0.01198099460452795,
+            0.009053511545062065,
+            -0.007946024648845196,
+            -0.00963436160236597,
+            0.015195030719041824,
+            0.0019080914789810777,
+            0.00033568276558071375,
+            0.001336922636255622,
+            -0.0065210070461034775,
+            -0.004011735785752535,
+            -0.021948376670479774,
+            0.02763296104967594,
+            0.007140580099076033,
+            0.030606910586357117,
+            -0.055296897888183594,
+            0.030095763504505157,
+            -0.00789568480104208,
+            0.02554190158843994,
+            -0.016201836988329887,
+            -0.011090358719229698,
+            0.001609921921044588,
+            0.018153492361307144,
+            0.004491904750466347,
+            0.006462921854108572,
+            0.011888058856129646,
+            0.023233991116285324,
+            -0.03444826602935791,
+            0.017053749412298203,
+            0.023110076785087585,
+            0.0048675211146473885,
+            0.0003535923024173826,
+            0.005165690556168556,
+            0.015458349138498306,
+            0.007198664825409651,
+            -0.023048119619488716,
+            -0.0025692922063171864,
+            0.017611365765333176,
+            -0.02306360937654972,
+            0.0036980770528316498,
+            -0.006923729553818703,
+            0.004623564425855875,
+            -0.0014705180656164885,
+            -0.03509881719946861,
+            -0.00100099784322083,
+            -0.025743262842297554,
+            0.01923774555325508,
+            0.009556914679706097,
+            -0.025867177173495293,
+            0.003932353109121323,
+            0.025650326162576675,
+            -0.0003695656778290868,
+            0.00821708794683218,
+            0.0004683101433329284,
+            0.00869725737720728,
+            -0.00974278710782528,
+            -0.01493945624679327,
+            0.004801691509783268,
+            0.01531120017170906,
+            -0.01493945624679327,
+            -0.007431004662066698,
+            -0.009510447271168232,
+            0.030560443177819252,
+            0.0006674932083114982,
+            0.004189862869679928,
+            -0.005045648664236069,
+            -0.01858719252049923,
+            -0.0032237162813544273,
+            -0.021142931655049324,
+            0.004367990419268608,
+            -0.0009046735358424485,
+            0.04197607934474945,
+            0.0038161829579621553,
+            -0.02029101923108101,
+            0.0032972905319184065,
+            -0.03136589005589485,
+            0.0005000148667022586,
+            -0.007276111748069525,
+            -0.02004319056868553,
+            0.0015005286550149322,
+            0.0016525175888091326,
+            0.012631546705961227,
+            -0.03255856782197952,
+            -0.009301341138780117,
+            0.018416810780763626,
+            -0.00020680672605521977,
+            -0.010687636211514473,
+            -0.017828216776251793,
+            0.00035189816844649613,
+            -0.011338187381625175,
+            -0.027973726391792297,
+            -0.02850036323070526,
+            -0.0026719090528786182,
+            0.013491204008460045,
+            -0.018277406692504883,
+            -0.014993669465184212,
+            0.0031249718740582466,
+            0.021592123433947563,
+            -0.009781510569155216,
+            0.008302279748022556,
+            0.02241305634379387,
+            -0.008255811408162117,
+            -0.005514200776815414,
+            -0.000941460661124438,
+            -0.009130958467721939,
+            0.004267309792339802,
+            0.006300284061580896,
+            0.043339136987924576,
+            -0.032403673976659775,
+            -0.0018238683696836233,
+            0.014815541915595531,
+            0.0034153966698795557,
+            0.0025905899237841368,
+            0.002503462601453066,
+            -0.0007841471815481782,
+            0.01344473659992218,
+            0.01178737822920084,
+            0.015466093085706234,
+            0.00100099784322083,
+            -0.0033011629711836576,
+            0.006408709101378918,
+            -0.021173911169171333,
+            0.015721667557954788,
+            0.027958236634731293,
+            -0.0033999073784798384,
+            0.005525817628949881,
+            -0.013731288723647594,
+            -0.008759214542806149,
+            0.013134949840605259,
+            0.0026641644071787596,
+            -0.005068882368505001,
+            -0.009394276887178421,
+            0.017967619001865387,
+            0.019082851707935333,
+            0.006668155547231436,
+            -0.037453193217515945,
+            -0.02898053266108036,
+            -0.009959637187421322,
+            0.020569827407598495,
+            0.014916222542524338,
+            0.0010339126456528902,
+            -0.010540487244725227,
+            -0.009595638141036034,
+            -0.011903548613190651,
+            0.010989678092300892,
+            0.02721474878489971,
+            -0.032372694462537766,
+            0.0008214184199459851,
+            -0.029383255168795586,
+            0.03068435750901699,
+            0.0020000594668090343,
+            0.011446612887084484,
+            -0.0008185141487047076,
+            -0.015140817500650883,
+            -0.029181893914937973,
+            0.013824224472045898,
+            -0.004395096562802792,
+            0.001024231780320406,
+            -0.0031753121875226498,
+            0.0201051477342844,
+            0.012887120246887207,
+            -0.0014472840121015906,
+            0.01968693546950817,
+            0.008178364485502243,
+            0.016480645164847374,
+            -0.010153254494071007,
+            -0.04135650396347046,
+            0.019965743646025658,
+            0.007562664337456226,
+            0.014219203032553196,
+            0.004739734344184399,
+            0.004588713403791189,
+            -0.020771188661456108,
+            0.002724185585975647,
+            0.004333139397203922,
+            0.014405074529349804,
+            0.011028401553630829,
+            0.0025615475606173277,
+            0.011678952723741531,
+            -0.02228914201259613,
+            0.01929970271885395,
+            -0.01631026156246662,
+            0.01006806269288063,
+            0.004445436876267195,
+            -0.026873983442783356,
+            0.007547175046056509,
+            0.007822110317647457,
+            -0.0019187404541298747,
+            -0.017549408599734306,
+            0.02160761132836342,
+            -0.0034831624943763018,
+            -0.03509881719946861,
+            0.0360281765460968,
+            -0.0043447562493383884,
+            0.019346170127391815,
+            0.03209388628602028,
+            -0.01771979033946991,
+            -0.01993476413190365,
+            0.00906900130212307,
+            0.010408828034996986,
+            -0.002873270306736231,
+            0.017859194427728653,
+            -0.002919738180935383,
+            -0.02232012152671814,
+            0.017766257748007774,
+            -0.028918573632836342,
+            -0.014234691858291626,
+            0.020337486639618874,
+            -0.03624502569437027,
+            -0.0021026763133704662,
+            0.03175312280654907,
+            -0.024070415645837784,
+            0.005734923295676708,
+            0.005196669604629278,
+            -0.0019351978553459048,
+            0.0012236569309607148,
+            -0.006451305001974106,
+            0.03801080957055092,
+            -0.010865762829780579,
+            0.0201051477342844,
+            0.01929970271885395,
+            -0.023388884961605072,
+            0.01977987214922905,
+            -0.001109423115849495,
+            0.01675945147871971,
+            -0.003940097521990538,
+            0.03466511517763138,
+            -0.006017603911459446,
+            -0.011330443434417248,
+            0.006431943271309137,
+            0.008488151244819164,
+            -0.009959637187421322,
+            0.008681767620146275,
+            -0.018122512847185135,
+            0.027152791619300842,
+            0.02091059274971485,
+            -0.0002691270783543587,
+            0.013800990767776966,
+            0.0004421719058882445,
+            -0.020833145827054977,
+            0.00789568480104208,
+            -0.027121812105178833,
+            -0.009967382065951824,
+            0.01646515540778637,
+            -0.005762029904872179,
+            0.007380664348602295,
+            -0.02064727433025837,
+            0.028004704043269157,
+            0.0077524082735180855,
+            -0.013591884635388851,
+            -0.0008824076503515244,
+            0.023760627955198288,
+            -0.005432881414890289,
+            0.009355553425848484,
+            0.02070923149585724,
+            0.015713922679424286,
+            0.013615119270980358,
+            0.016263794153928757,
+            0.03351890295743942,
+            0.005328328814357519,
+            0.005568413063883781,
+            0.003519949736073613,
+            0.016991792246699333,
+            -0.008557853288948536,
+            0.018029578030109406,
+            0.0008093173382803798,
+            -0.013576395809650421,
+            0.021499186754226685,
+            -0.0018684001406654716,
+            -0.013622863218188286,
+            -0.0040891822427511215,
+            -0.004337011836469173,
+            0.03271345794200897,
+            0.022041313350200653,
+            -0.004468671046197414,
+            -0.024442158639431,
+            -0.00394784240052104,
+            0.016682006418704987,
+            -0.003167567541822791,
+            0.011136826127767563,
+            0.0007952801533974707,
+            -0.01586107164621353,
+            -0.009595638141036034,
+            0.0063583687879145145,
+            0.010586955584585667,
+            -0.008921852335333824,
+            0.010346870869398117,
+            -0.0228312686085701,
+            -0.004011735785752535,
+            0.00046613195445388556,
+            -0.004844287410378456,
+            0.0053438181057572365,
+            -0.011314953677356243,
+            0.004948840010911226,
+            0.016062432900071144,
+            0.01026167906820774,
+            0.004832670092582703,
+            0.006989559158682823,
+            0.012066186405718327,
+            0.009161937050521374,
+            -0.01627928391098976,
+            0.00428667152300477,
+            -0.005072754807770252,
+            -0.02345084212720394,
+            0.005831731949001551,
+            0.00902253296226263,
+            -0.01122201792895794,
+            0.009386532939970493,
+            -0.010958699509501457,
+            0.03296129032969475,
+            0.01030814740806818,
+            -0.037391237914562225,
+            -0.0033379499800503254,
+            -0.0016244432190433145,
+            0.014885243959724903,
+            0.012298526242375374,
+            0.008550108410418034,
+            -0.017518429085612297,
+            -0.0047823297791182995,
+            -0.009835722856223583,
+            0.0012827098835259676,
+            -0.012306271120905876,
+            0.017456471920013428,
+            0.022567950189113617,
+            -0.036337960511446,
+            0.01968693546950817,
+            -0.010432061739265919,
+            -0.023652203381061554,
+            0.020476890727877617,
+            0.011500826105475426,
+            0.01942361705005169,
+            -0.009402021765708923,
+            0.018896980211138725,
+            -0.0001331114035565406,
+            0.025510922074317932,
+            -0.004228586331009865,
+            -0.022087780758738518,
+            -0.012499887496232986,
+            -0.03209388628602028,
+            -0.0051811798475682735,
+            0.006819176487624645,
+            -0.00045524101005867124,
+            0.013111716136336327,
+            0.014226946979761124,
+            -0.025712283328175545,
+            0.007295473478734493,
+            0.032589543610811234,
+            0.014637414366006851,
+            0.023311438038945198,
+            -0.011276230216026306,
+            -0.0010794125264510512,
+            -0.0026060794480144978,
+            -0.013770012184977531,
+            -0.03212486580014229,
+            -0.0017493258928880095,
+            -0.036585792899131775,
+            0.005680711008608341,
+            -0.016387708485126495,
+            -0.013096226379275322,
+            0.023264970630407333,
+            -0.034138478338718414,
+            0.0014608372002840042,
+            0.013708055019378662,
+            0.010540487244725227,
+            0.012391461990773678,
+            -0.00162637943867594,
+            0.015032391995191574,
+            -0.029243851080536842,
+            -0.010284913703799248,
+            -0.004491904750466347,
+            -0.01981084980070591,
+            -0.007338068913668394,
+            -0.018060555681586266,
+            -0.013127204962074757,
+            -0.003566417610272765,
+            -0.010060317814350128,
+            0.002247888594865799,
+            0.0032062907703220844,
+            -0.014714861288666725,
+            0.03206290677189827,
+            0.003157886676490307,
+            0.018958937376737595,
+            0.0033418224193155766,
+            0.004449309315532446,
+            0.018788553774356842,
+            -0.0026331855915486813,
+            -0.0058278595097362995,
+            0.030963165685534477,
+            -0.00032672801171429455,
+            0.008759214542806149,
+            0.019082851707935333,
+            0.004724245052784681,
+            0.0038529701996594667,
+            -0.0260530486702919,
+            0.012887120246887207,
+            -0.027447087690234184,
+            0.005080499220639467,
+            -0.0077175572514534,
+            -0.024302754551172256,
+            -0.028128618374466896,
+            -0.006571347359567881,
+            0.012058441527187824,
+            -0.02898053266108036,
+            -0.00613377382978797,
+            0.002315654419362545,
+            -0.00400399137288332,
+            -0.002673845272511244,
+            0.007241260726004839,
+            0.029150914400815964,
+            -0.033209118992090225,
+            0.008751469664275646,
+            0.005595519673079252,
+            -0.0005053392960689962,
+            0.013297587633132935,
+            0.008038961328566074,
+            0.003343758638948202,
+            0.020848635584115982,
+            0.006637176964432001,
+            0.002458930714055896,
+            -0.0025576751213520765,
+            -0.014242436736822128,
+            0.004000118933618069,
+            0.02154565416276455,
+            0.007562664337456226,
+            0.018385831266641617,
+            -0.008914108388125896,
+            0.002621568739414215,
+            -0.01455222349613905,
+            0.005552923772484064,
+            -0.002867461647838354,
+            0.004309905227273703,
+            -0.012236569076776505,
+            0.008093173615634441,
+            0.0028771425131708384,
+            -0.0043447562493383884,
+            0.018060555681586266,
+            -0.002319526858627796,
+            0.016480645164847374,
+            0.004015608225017786,
+            0.015620986931025982,
+            -0.0035412474535405636,
+            0.01178737822920084,
+            0.0005731051205657423,
+            -0.017905661836266518,
+            -0.017022771760821342,
+            -0.039249956607818604,
+            0.007702067960053682,
+            0.03531566634774208,
+            -0.0009564659558236599,
+            -0.03444826602935791,
+            0.010145509615540504,
+            -0.011942271143198013,
+            -0.009975126944482327,
+            -0.022862248122692108,
+            0.013862947933375835,
+            0.012430185452103615,
+            0.007415515370666981,
+            0.012252057902514935,
+            -0.006997303571552038,
+            0.01026167906820774,
+            -0.005216030869632959,
+            0.008356492035090923,
+            -0.0010823167394846678,
+            -0.037422213703393936,
+            0.012344993650913239,
+            0.02216522768139839,
+            -0.002635121811181307,
+            0.0016089539276435971,
+            -0.006145390681922436,
+            0.022103270515799522,
+            -0.013971373438835144,
+            -0.00650551775470376,
+            -0.00046443779137916863,
+            0.00797700323164463,
+            -0.01795213110744953,
+            0.004778457339853048,
+            0.0755879208445549,
+            -0.007775642443448305,
+            0.006122156511992216,
+            0.009270362555980682,
+            -0.015148562379181385,
+            -0.01637221872806549,
+            -0.004561606794595718,
+            -0.020260039716959,
+            0.008844406343996525,
+            -0.0018819533288478851,
+            -0.0110981035977602,
+            0.012468908913433552,
+            0.0053050946444272995,
+            0.002031038049608469,
+            -0.008379725739359856,
+            0.010811550542712212,
+            0.009525936096906662,
+            0.010432061739265919,
+            0.02360573597252369,
+            0.011477591469883919,
+            -0.00858883187174797,
+            -0.011795123107731342,
+            0.01129171997308731,
+            0.0028577810153365135,
+            0.016697494313120842,
+            0.007907302118837833,
+            0.0010648912284523249,
+            0.019501063972711563,
+            -0.002309845993295312,
+            0.02180897258222103,
+            -2.2689444449497387e-05,
+            -0.009301341138780117,
+            -0.004236331209540367,
+            -0.015349923633038998,
+            0.024271776899695396,
+            -0.0020155487582087517,
+            0.0015450604259967804,
+            -0.020213572308421135,
+            0.0013979118084535003,
+            0.013243375346064568,
+            0.011957760900259018,
+            -0.0020407189149409533,
+            -0.0051347119733691216,
+            0.0007154133054427803,
+            -0.016356728971004486,
+            0.032806396484375,
+            0.007578153628855944,
+            0.0007681738352403045,
+            0.0008838597568683326,
+            0.0009288755827583373,
+            -0.0032972905319184065,
+            0.0024608669336885214,
+            -0.004766840487718582,
+            0.009549170732498169,
+            -0.010943209752440453,
+            -0.010796061716973782,
+            0.010075807571411133,
+            -0.02402394637465477,
+            0.003566417610272765,
+            -0.00558777479454875,
+            -0.008062195032835007,
+            0.0076207490637898445,
+            0.00015537731815129519,
+            -0.007392281666398048,
+            -0.0030939930584281683,
+            0.0023408245760947466,
+            0.013429246842861176,
+            0.004406713880598545,
+            0.010850274004042149,
+            0.012956822291016579,
+            0.005359307397156954,
+            -0.006629432085901499,
+            0.015512561425566673,
+            -0.006946963258087635,
+            -0.004228586331009865,
+            -0.0012468908680602908,
+            -0.019996721297502518,
+            -0.0009419447160325944,
+            -0.004708755761384964,
+            0.0015092412941157818,
+            0.0001705036120256409,
+            -8.749050175538287e-05,
+            -0.0018490385264158249,
+            -0.011988739483058453,
+            -0.001831613015383482,
+            -0.011663463898003101,
+            -0.014498010277748108,
+            0.010253935120999813,
+            -0.0041201612912118435,
+            -0.022939693182706833,
+            -0.015349923633038998,
+            -0.020662762224674225,
+            0.02145271934568882,
+            -0.004387352149933577,
+            -0.005800753366202116,
+            0.006733985152095556,
+            -0.010215211659669876,
+            -0.015706177800893784,
+            -0.008069939911365509,
+            -0.0205853171646595,
+            0.010703125037252903,
+            0.003214035416021943,
+            0.0006791101768612862,
+            0.00882891658693552,
+            0.014397329650819302,
+            -0.01589204929769039,
+            0.0111755495890975,
+            0.012182355858385563,
+            0.011361422017216682,
+            -0.006854027509689331,
+            0.0008911203476600349,
+            -0.014103032648563385,
+            0.02596011385321617,
+            0.021623101085424423,
+            0.005169562995433807,
+            -0.04132552444934845,
+            -0.005862710531800985,
+            -0.013344055972993374,
+            -0.010524998418986797,
+            -0.012120398692786694,
+            0.011771888472139835,
+            -0.0038820127956569195,
+            0.0013117524795234203,
+            0.006288666743785143,
+            0.0015063370810821652,
+            0.004367990419268608,
+            -0.0022382077295333147,
+            -0.007709812838584185,
+            -0.03212486580014229,
+            -0.02817508764564991,
+            -0.00863530021160841,
+            0.020461400970816612,
+            0.002838419284671545,
+            -0.0009453330421820283,
+            -0.0036380558740347624,
+            0.0027048238553106785,
+            -0.005882072262465954,
+            -0.005018542055040598,
+            -0.014753584749996662,
+            -0.012739972211420536,
+            0.0101300198584795,
+            0.00802347157150507,
+            0.014195968396961689,
+            -0.011950016021728516,
+            0.0028190575540065765,
+            0.018788553774356842,
+            0.007690451107919216,
+            -0.023357905447483063,
+            0.01135367713868618,
+            -0.010912231169641018,
+            -0.009270362555980682,
+            -0.009433000348508358,
+            -0.009471723809838295,
+            0.0016980175860226154,
+            -0.008317768573760986,
+            0.003945906180888414,
+            -0.03615209087729454,
+            -0.011330443434417248,
+            -0.0013901671627536416,
+            -0.010346870869398117,
+            0.0009864765452221036,
+            0.02524760365486145,
+            0.006815304048359394,
+            0.01432762760668993,
+            -0.0006733016925863922,
+            0.017502939328551292,
+            -0.0018984107300639153,
+            0.018726596608757973,
+            0.012763205915689468,
+            -0.00797700323164463,
+            0.011717676185071468,
+            0.014738094992935658,
+            -0.0033863543067127466,
+            0.015620986931025982,
+            -0.010168743319809437,
+            0.005285732913762331,
+            -0.009556914679706097,
+            -0.003214035416021943,
+            0.002495717955753207,
+            0.030901208519935608,
+            -0.003618694143369794,
+            0.00858883187174797,
+            0.01098193321377039,
+            -0.015241498127579689,
+            0.0026951429899781942,
+            0.014962689951062202,
+            0.01141563430428505,
+            0.007760153152048588,
+            0.0024434414226561785,
+            -0.024225307628512383,
+            0.004193735308945179,
+            0.005037903785705566,
+            0.0031520782504230738,
+            -0.014366351068019867,
+            0.0021859314292669296,
+            0.015675200149416924,
+            0.008550108410418034,
+            -0.025325050577521324,
+            0.016418687999248505,
+            0.0009138703462667763,
+            -0.003909118939191103,
+            0.029119934886693954,
+            -0.011268485337495804,
+            -0.005723306443542242,
+            0.009463978931307793,
+            0.0009980935137718916,
+            0.0030029932968318462,
+            0.008937342092394829,
+            0.0228312686085701,
+            -0.003988501615822315,
+            -0.032341714948415756,
+            -0.022459525614976883,
+            0.015706177800893784,
+            -0.0047823297791182995,
+            -0.022939693182706833,
+            -0.026703599840402603,
+            -0.003113354789093137,
+            -0.02228914201259613,
+            -0.0022323993034660816,
+            -0.020058678463101387,
+            -0.014056564308702946,
+            0.001268188701942563,
+            0.029987338930368423,
+            0.016201836988329887,
+            0.008224832825362682,
+            -0.011051635257899761,
+            -0.029956359416246414,
+            -0.024271776899695396,
+            0.01810702309012413,
+            -0.007055388763546944,
+            -0.005541306920349598,
+            0.00591692328453064,
+            -0.009951893240213394,
+            -0.008480406366288662,
+            0.011802867986261845,
+            0.027137301862239838,
+            -0.002747419523075223,
+            -0.029987338930368423,
+            -0.02688947319984436,
+            -0.014335372485220432,
+            -0.01840132102370262,
+            -0.017704300582408905,
+            -0.008627555333077908,
+            0.0045035220682621,
+            0.00030300996149890125,
+            0.016186347231268883,
+            0.014962689951062202,
+            -0.000480653194244951,
+            0.0028597170021384954,
+            -2.7711375878425315e-05,
+            -0.001763847190886736,
+            0.006888878531754017,
+            -0.004933350719511509,
+            0.026997897773981094,
+            0.014614180661737919,
+            -0.006594581063836813,
+            0.018711106851696968,
+            0.028314489871263504,
+            0.020430423319339752,
+            0.003157886676490307,
+            -0.010695381090044975,
+            -0.014606435783207417,
+            0.009727297350764275,
+            -0.013421501964330673,
+            0.023264970630407333,
+            0.022552460432052612,
+            -0.01480779703706503,
+            -0.014885243959724903,
+            -0.017905661836266518,
+            0.015210519544780254,
+            -0.007562664337456226,
+            -0.008224832825362682,
+            0.012368228286504745,
+            0.022335609421133995,
+            -0.007024410180747509,
+            0.002371803391724825,
+            -0.0010232636705040932,
+            -0.021731525659561157,
+            -0.006122156511992216,
+            -0.004596457816660404,
+            -0.008666278794407845,
+            0.008681767620146275,
+            0.008131897076964378,
+            -0.001928421319462359,
+            -0.006319645792245865,
+            0.005471604876220226,
+            -0.013367289677262306,
+            0.01301103550940752,
+            -0.00789568480104208,
+            0.03379771113395691,
+            -0.009138703346252441,
+            -0.0006733016925863922,
+            -0.022598929703235626,
+            -0.011446612887084484,
+            0.00845717266201973,
+            0.0017967619933187962,
+            0.01682141050696373,
+            -0.0011626676423475146,
+            -0.005688455421477556,
+            0.009913169778883457,
+            -0.0010687635513022542,
+            -0.001450188341550529,
+            -0.006892750971019268,
+            -0.010401083156466484,
+            0.0002884887217078358,
+            0.013258864171802998,
+            -0.006025348324328661,
+            0.0018054747488349676,
+            0.006095050368458033,
+            -0.0175339188426733,
+            0.029027000069618225,
+            0.023884542286396027,
+            0.017255110666155815,
+            0.014660648070275784,
+            0.0006394187803380191,
+            0.011647974140942097,
+            -0.003572226269170642,
+            0.0038820127956569195,
+            -0.020337486639618874,
+            0.021080974489450455,
+            -0.010811550542712212,
+            0.01846327818930149,
+            0.011655719019472599,
+            -0.015233753249049187,
+            -0.005022414494305849,
+            0.021406250074505806,
+            0.009967382065951824,
+            0.00889087375253439,
+            0.0017541663255542517,
+            -0.02470547705888748,
+            0.017766257748007774,
+            0.010099041275680065,
+            -0.020399443805217743,
+            0.019547531381249428,
+            -0.009471723809838295,
+            0.01233724970370531,
+            -0.0005300253978930414,
+            0.004418330732733011,
+            0.0058743273839354515,
+            0.021499186754226685,
+            0.011570527218282223,
+            -0.03884723410010338,
+            -0.013615119270980358,
+            -0.0017357728211209178,
+            0.006857899948954582,
+            -0.0036303112283349037,
+            -0.015233753249049187,
+            -0.041635312139987946,
+            0.01604694314301014,
+            0.009998360648751259,
+            -0.01643417589366436,
+            0.004987563472241163,
+            -0.002427952131256461,
+            -0.0042053526267409325,
+            -0.008170620538294315,
+            0.003186929039657116,
+            -0.004170501604676247,
+            0.02324948087334633,
+            0.0018984107300639153,
+            -0.006946963258087635,
+            0.013258864171802998,
+            0.01344473659992218,
+            -0.021948376670479774,
+            0.0009874446550384164,
+            0.0021375271026045084,
+            0.000636998622212559,
+            0.023683181032538414,
+            -0.014707116410136223,
+            -0.0017657832941040397,
+            -0.01685238815844059,
+            -0.0013069120468571782,
+            -0.021592123433947563,
+            -0.001188805908896029,
+            -0.015187285840511322,
+            -0.008511385880410671,
+            0.025030752643942833,
+            -0.011005166918039322,
+            0.010997422970831394,
+            -0.006687517277896404,
+            0.006943091284483671,
+            -0.006904367823153734,
+            -0.01936165988445282,
+            -0.0008267428493127227,
+            0.009975126944482327,
+            -0.000512115890160203,
+            0.002941036131232977,
+            8.482827252009884e-05,
+            0.002464739140123129,
+            0.008867640048265457,
+            0.006234454456716776,
+            -0.0075278133153915405,
+            0.02489134855568409,
+            -0.012073930352926254,
+            0.0016399325104430318,
+            0.011911292560398579,
+            0.0004743606550619006,
+            -0.010424317792057991,
+            0.025588368996977806,
+            -0.02139076218008995,
+            -0.00800798274576664,
+            -0.00014799568452872336,
+            -0.0016544538084417582,
+            -0.006513262167572975,
+            0.006718495860695839,
+            0.02946070022881031,
+            0.0062654330395162106,
+            0.010447551496326923,
+            -0.0281441081315279,
+            -0.011493081226944923,
+            0.012538610957562923,
+            -0.022273652255535126,
+            -0.0007739823195151985,
+            -0.0004322490422055125,
+            -0.00619960343465209,
+            0.026378324255347252,
+            -0.02022906206548214,
+            -0.01093546487390995,
+            -0.018664639443159103,
+            0.005405775271356106,
+            -0.011121337302029133,
+            0.008294534869492054,
+            -0.0009777637897059321,
+            0.006540368776768446,
+            0.016805920749902725,
+            -0.019563021138310432,
+            0.016031453385949135,
+            0.005119222681969404,
+            -0.016217326745390892,
+            0.010222956538200378,
+            -0.017580386251211166,
+            -0.0008199662552215159,
+            0.020275529474020004,
+            -0.00400399137288332,
+            0.008751469664275646,
+            -0.003363120136782527,
+            0.005208286456763744,
+            -0.006513262167572975,
+            0.0001492057926952839,
+            -0.007249005138874054,
+            0.014428308233618736,
+            -0.016805920749902725,
+            -0.017828216776251793,
+            -0.008821171708405018,
+            0.0016709112096577883,
+            0.021297825500369072,
+            0.011136826127767563,
+            -0.010044828988611698,
+            -0.011059380136430264,
+            -0.011694442480802536,
+            0.041604332625865936,
+            0.005762029904872179,
+            0.027230238541960716,
+            -0.02261441759765148,
+            0.00545998802408576,
+            0.025170156732201576,
+            -0.014242436736822128,
+            0.0012788375606760383,
+            -0.0022788674104958773,
+            0.0053050946444272995,
+            0.011562783271074295,
+            -0.01819995976984501,
+            0.0049139889888465405,
+            -0.00019216447253711522,
+            -0.00782985519617796,
+            -0.000656844291370362,
+            0.003442503046244383,
+            0.00020196630794089288,
+            0.016325751319527626,
+            -0.004027225077152252,
+            -0.016511622816324234,
+            -0.004429947584867477,
+            -0.0003216455806978047,
+            -0.027896279469132423,
+            0.013514438644051552,
+            0.016589069738984108,
+            0.01566745527088642,
+            0.027989214286208153,
+            -0.014513500034809113,
+            0.016031453385949135,
+            0.009533680975437164,
+            0.013901671394705772,
+            -0.010610189288854599,
+            0.008952830918133259,
+            -0.01595400646328926,
+            0.007446494419127703,
+            -0.03454120084643364,
+            -0.007887939922511578,
+            -0.00802347157150507,
+            0.01984182931482792,
+            0.028763681650161743,
+            0.002923610620200634,
+            0.006493900436908007,
+            0.000196883876924403,
+            0.0024202074855566025,
+            -0.01714668609201908,
+            -0.020182594656944275,
+            -0.0003337466041557491,
+            0.0029332914855331182,
+            0.0006950835813768208,
+            -0.004495777189731598,
+            0.00472037261351943,
+            -0.012391461990773678,
+            0.0026835259050130844,
+            -0.013134949840605259,
+            0.005971135571599007,
+            -0.0059982421807944775,
+            0.009603383019566536,
+            0.0019429425010457635,
+            0.020864123478531837,
+            0.02300165221095085,
+            0.004100799560546875,
+            0.0046971384435892105,
+            -0.024333734065294266,
+            0.00920066051185131,
+            -0.013847459107637405,
+            0.012329504825174809,
+            0.019408127292990685,
+            0.008248066529631615,
+            0.006571347359567881,
+            0.01233724970370531,
+            0.012918098829686642,
+            -0.0031772484071552753,
+            0.024287264794111252,
+            0.005014669615775347,
+            -0.0009070937521755695,
+            0.002052335999906063,
+            -0.014954946003854275,
+            -0.0070979841984808445,
+            -0.0015286030247807503,
+            0.0018848575418815017,
+            -0.004731989465653896,
+            0.01906736195087433,
+            0.015256987884640694,
+            -0.01050176378339529,
+            0.017874684184789658,
+            -0.03937387093901634,
+            0.017379024997353554,
+            -0.0027377386577427387,
+            -0.0060911779291927814,
+            -0.00545998802408576,
+            0.00665653869509697,
+            0.004515138920396566,
+            0.003669034456834197,
+            0.03773200139403343,
+            -0.010408828034996986,
+            0.003963331691920757,
+            -0.005847221240401268,
+            0.007121218368411064,
+            -0.020864123478531837,
+            0.008302279748022556,
+            0.005336073227226734,
+            0.02160761132836342,
+            -0.0056497324258089066,
+            -0.0014482521219179034,
+            0.02328045852482319,
+            0.008859895169734955,
+            0.03469609469175339,
+            -0.01566745527088642,
+            -0.004418330732733011,
+            0.01048627495765686,
+            0.0207402091473341,
+            -0.00580462533980608,
+            -0.008976065553724766,
+            0.007903429679572582,
+            0.0035586729645729065,
+            0.021824462339282036,
+            -0.00515407370403409,
+            -0.0006650729919783771,
+            0.0054367538541555405,
+            -0.0060911779291927814,
+            0.009735042229294777,
+            0.011252996511757374,
+            -0.009076746180653572,
+            0.007539430167526007,
+            0.021530164405703545,
+            -0.009975126944482327,
+            -0.008658533915877342,
+            -0.009649851359426975,
+            0.0010658593382686377,
+            -0.03068435750901699,
+            0.001649613375775516,
+            0.005359307397156954,
+            0.015187285840511322,
+            -0.0004603234410751611,
+            -0.009014789015054703,
+            0.021019017323851585,
+            0.017255110666155815,
+            -0.005936284549534321,
+            -0.0028132491279393435,
+            0.0023795480374246836,
+            -0.0017154429806396365,
+            -0.006160879973322153,
+            -0.0006248975405469537,
+            -0.002656419761478901,
+            0.014730350114405155,
+            -0.0015469965292140841,
+            -0.006617815233767033,
+            -0.00756653631106019,
+            0.0033302053343504667,
+            0.010834784246981144,
+            -0.005661349277943373,
+            0.013057502917945385,
+            0.016805920749902725,
+            -0.00031704717548564076,
+            0.019671445712447166,
+            0.01037010457366705,
+            -0.010447551496326923,
+            0.015938518568873405,
+            -0.007984748110175133,
+            0.006563602481037378,
+            0.0164031982421875,
+            -0.01277095079421997,
+            -0.01246116403490305,
+            0.007210282143205404,
+            0.00665653869509697,
+            0.028918573632836342,
+            0.014792308211326599,
+            -0.02109646424651146,
+            0.014908477663993835,
+            0.015768134966492653,
+            -0.013018779456615448,
+            0.003537375247105956,
+            -0.01784370467066765,
+            0.020260039716959,
+            0.019051872193813324,
+            -0.015535795129835606,
+            -0.017224133014678955,
+            -0.014947201125323772,
+            -0.005092116538435221,
+            0.006331262644380331,
+            0.0033263331279158592,
+            -0.00617636926472187,
+            -0.027989214286208153,
+            -0.0049139889888465405,
+            -0.018633661791682243,
+            0.013870692811906338,
+            -0.008674023672938347,
+            0.0007715621031820774,
+            -0.004511266481131315,
+            0.007984748110175133,
+            -0.0033882902935147285,
+            0.032465629279613495,
+            -0.0051811798475682735,
+            -0.02832997962832451,
+            0.004398969002068043,
+            0.016356728971004486,
+            0.012623801827430725,
+            -0.029243851080536842,
+            0.010463040322065353,
+            0.004805563949048519,
+            -0.012530866079032421,
+            0.005990497302263975,
+            -0.02258343994617462,
+            -0.007632365915924311,
+            -0.0009395244996994734,
+            -0.016743963584303856,
+            -0.027121812105178833,
+            -0.008433938957750797,
+            -0.006993431597948074,
+            -0.005401902832090855,
+            -0.009525936096906662,
+            -0.008720491081476212,
+            -0.011477591469883919,
+            0.02303262986242771,
+            -0.007899557240307331,
+            -0.001866463921032846,
+            -0.0014637414133176208,
+            0.00930908601731062,
+            -0.023776117712259293,
+            -0.015605497173964977,
+            0.006389347370713949,
+            -0.002594462363049388,
+            0.022815778851509094,
+            -0.012747716158628464,
+            0.005940156988799572,
+            -0.009316830895841122,
+            0.009851212613284588,
+            0.015241498127579689,
+            0.004956584889441729,
+            0.017440982162952423,
+            -0.009889936074614525,
+            0.0021026763133704662,
+            -0.008627555333077908,
+            -0.00018986526993103325,
+            -1.027075541060185e-05,
+            0.014428308233618736,
+            0.007624621503055096,
+            0.014141756109893322,
+            0.017487451434135437,
+            -0.013553161174058914,
+            0.005003052763640881,
+            -0.0003274540649726987,
+            -0.017255110666155815,
+            0.0027725896798074245,
+            -0.007984748110175133,
+            -0.0008896682411432266,
+            0.001758038648404181,
+            6.395397940650582e-05,
+            -0.02112744189798832,
+            -0.0009124181815423071,
+            0.0018809852190315723,
+            0.0010222955606877804,
+            -0.025201136246323586,
+            0.00889087375253439,
+            -0.00889087375253439,
+            -0.0014918158994987607,
+            -0.019113829359412193,
+            0.019795360043644905,
+            0.026300877332687378,
+            -0.004619691986590624,
+            0.00036206303047947586,
+            -0.013537672348320484,
+            -0.02049238048493862,
+            0.0001489637652412057,
+            -0.011314953677356243,
+            -0.011345932260155678,
+            -0.0022169100120663643,
+            -0.003214035416021943,
+            0.008805682882666588,
+            0.0004189379105810076,
+            -0.009433000348508358,
+            -0.004956584889441729,
+            -0.011764144524931908,
+            -0.017549408599734306,
+            -0.010401083156466484,
+            0.01933068037033081,
+            -0.0018016024259850383,
+            0.0020794421434402466,
+            -0.0032508226577192545,
+            0.0005271211848594248,
+            0.011748654767870903,
+            0.013475715182721615,
+            0.013344055972993374,
+            -0.023915521800518036,
+            -0.017890173941850662,
+            -0.013320821337401867,
+            0.0029487807769328356,
+            -0.007187047973275185,
+            0.004844287410378456,
+            0.026362836360931396,
+            -0.0009695350890979171,
+            -0.01091997604817152,
+            0.002327271504327655,
+            0.019051872193813324,
+            -0.031272951513528824,
+            -0.013390523381531239,
+            -0.025077221915125847,
+            0.02225816436111927,
+            -0.013351799920201302,
+            0.011516314931213856,
+            0.010191977955400944,
+            -0.025944624096155167,
+            0.006768836174160242,
+            -0.007686578668653965,
+            0.026037558913230896,
+            0.01634124107658863,
+            0.0005934348446317017,
+            -0.00944074522703886,
+            0.013622863218188286,
+            -0.0008722427301108837,
+            -0.013576395809650421,
+            0.002201420720666647,
+            0.014645159244537354,
+            -0.014691627584397793,
+            -0.009510447271168232,
+            -0.004507394507527351,
+            0.004313777666538954,
+            0.012763205915689468,
+            -0.0030339721124619246,
+            -0.004031097516417503,
+            -0.011624740436673164,
+            0.013344055972993374,
+            -0.003930416889488697,
+            0.012383717112243176,
+            0.007380664348602295,
+            -0.014056564308702946,
+            -0.019532041624188423,
+            0.033333033323287964,
+            0.00400399137288332,
+            0.00338054564781487,
+            -0.010997422970831394,
+            0.02740062028169632,
+            -0.009897680021822453,
+            0.010571465827524662,
+            -0.006408709101378918,
+            0.0031985461246222258,
+            0.012321759946644306,
+            -0.012104909867048264,
+            -0.01981084980070591,
+            0.021468207240104675,
+            -0.00364192808046937,
+            -0.0010193913476541638,
+            0.008310024626553059,
+            -0.020182594656944275,
+            -0.00663330452516675,
+            0.002180122770369053,
+            0.009541425853967667,
+            -0.011818356812000275,
+            0.006369986105710268,
+            0.017905661836266518,
+            0.003912991378456354,
+            -0.0007672057254239917,
+            0.019996721297502518,
+            -0.01672847382724285,
+            0.011717676185071468,
+            -0.010850274004042149,
+            0.0016825281782075763,
+            0.0006065039779059589,
+            -0.0048597767017781734,
+            0.0058743273839354515,
+            -0.014536733739078045,
+            0.011988739483058453,
+            -6.244135147426277e-05,
+            -0.013979118317365646,
+            0.007601387333124876,
+            -0.013359544798731804,
+            -0.016836898401379585,
+            0.041542377322912216,
+            -0.005301222205162048,
+            -0.0038665232714265585,
+            -0.0056962003000080585,
+            0.01350669376552105,
+            0.02823704481124878,
+            -0.005336073227226734,
+            -0.01153180468827486,
+            -0.010780571959912777,
+            -0.016573579981923103,
+            0.015907539054751396,
+            -0.01810702309012413,
+            0.002495717955753207,
+            -0.010532742366194725,
+            -0.007903429679572582,
+            0.004302160814404488,
+            -0.003516077296808362,
+            0.014722606167197227,
+            0.00950270239263773,
+            0.023357905447483063,
+            0.00797700323164463,
+            -0.00027348342700861394,
+            0.010393339209258556,
+            0.011663463898003101,
+            0.003243078012019396,
+            -0.010509508661925793,
+            -0.00011949774489039555,
+            -0.02894955314695835,
+            -0.008449427783489227,
+            -0.004863648675382137,
+            -0.001485039247199893,
+            -0.00428667152300477,
+            -0.01239920686930418,
+            0.006617815233767033,
+            -0.0022052929271012545,
+            0.013909416273236275,
+            -0.009254872798919678,
+            -0.017766257748007774,
+            0.03389064967632294,
+            0.004495777189731598,
+            0.006466794293373823,
+            -0.008286789990961552,
+            -0.03705047070980072,
+            -0.0032566310837864876,
+            -0.008116407319903374,
+            -0.013529927469789982,
+            0.004538373090326786,
+            0.01283290795981884,
+            -0.014831030741333961,
+            -0.028097640722990036,
+            0.0019390701781958342,
+            -0.006683644838631153,
+            0.02978597767651081,
+            -0.007798876613378525,
+            0.00419760774821043,
+            0.0037251831963658333,
+            0.0011558911064639688,
+            0.01239920686930418,
+            -0.00845717266201973,
+            -0.0022536972537636757,
+            0.016387708485126495,
+            -0.010858018882572651,
+            0.011051635257899761,
+            -0.010966443456709385,
+            -0.02554190158843994,
+            -0.01456771232187748,
+            0.021344292908906937,
+            -0.0004823473282158375,
+            -0.00024903935263864696,
+            0.014823286794126034,
+            0.02354377694427967,
+            0.011260741390287876,
+            0.00969631876796484,
+            0.006408709101378918,
+            -0.001035848748870194,
+            0.005975008010864258,
+            0.006095050368458033,
+            -0.0024163350462913513,
+            -0.013398268260061741,
+            0.017564896494150162,
+            0.009773765690624714,
+            -0.01810702309012413,
+            0.0008514289511367679,
+            0.0036612898111343384,
+            0.0168833676725626,
+            0.006935346405953169,
+            0.009316830895841122,
+            -0.0143121387809515,
+            -0.023760627955198288,
+            -0.00014932679187040776,
+            0.002873270306736231,
+            0.015721667557954788,
+            0.010579210706055164,
+            -0.014319883659482002,
+            0.0005963391158729792,
+            0.008712747134268284,
+            0.020213572308421135,
+            -0.024209819734096527,
+            0.019702425226569176,
+            -0.004673904739320278,
+            -0.0021084847394376993,
+            0.007946024648845196,
+            0.011632485315203667,
+            -0.011872569099068642,
+            -0.00650551775470376,
+            0.006249943748116493,
+            0.003157886676490307,
+            0.03172214329242706,
+            -0.0017260919557884336,
+            0.009665340185165405,
+            -0.010997422970831394,
+            -0.0002405686245765537,
+            0.0006728176376782358,
+            -0.0006820144481025636,
+            -0.002884887158870697,
+            -0.011020656675100327,
+            0.0018974426202476025,
+            -0.020306508988142014,
+            0.0024473138619214296,
+            0.00463130883872509,
+            -0.005684582982212305,
+            -0.0032972905319184065,
+            -0.019222255796194077,
+            -0.0022672503255307674,
+            0.003293418325483799,
+            0.014335372485220432,
+            -0.031691163778305054,
+            0.004972074180841446,
+            0.010672146454453468,
+            -0.002309845993295312,
+            -0.010416572913527489,
+            0.011423379182815552,
+            -0.0026060794480144978,
+            0.011988739483058453,
+            -0.0038142469711601734,
+            -0.0004905760288238525,
+            0.025681305676698685,
+            -0.005308967083692551,
+            0.014707116410136223,
+            -0.01026167906820774,
+            -0.002993312431499362,
+            -0.028252532705664635,
+            0.005235392600297928,
+            -0.007403898518532515,
+            -0.0012081675231456757,
+            -0.004402841441333294,
+            -0.01130720879882574,
+            -0.0026990154292434454,
+            0.002122037811204791,
+            0.008000237867236137,
+            0.012910354882478714,
+            -0.0007725302129983902,
+            -0.010075807571411133,
+            0.004832670092582703,
+            0.0075355577282607555,
+            -0.015520306304097176,
+            -0.0070437719114124775,
+            -0.020306508988142014,
+            0.029662061482667923,
+            -0.013731288723647594,
+            -0.0011123273288831115,
+            -0.020864123478531837,
+            0.010160998441278934,
+            -0.0063119009137153625,
+            -0.002782270545139909,
+            0.004708755761384964,
+            -0.015109838917851448,
+            0.01344473659992218,
+            0.008751469664275646,
+            0.01450575515627861,
+            -0.02151467651128769,
+            -0.01762685552239418,
+            -0.0044725434854626656,
+            -0.002735802438110113,
+            -0.010648912750184536,
+            -0.0005334137240424752,
+            0.002013612538576126,
+            0.021251358091831207,
+            -0.00032503384863957763,
+            0.0042053526267409325,
+            -0.003214035416021943,
+            0.017224133014678955,
+            -0.010137764737010002,
+            -0.013057502917945385,
+            0.0024260159116238356,
+            0.005994369741529226,
+            0.016186347231268883,
+            0.03158273920416832,
+            -0.020693741738796234,
+            -0.00272999401204288,
+            0.0032740565948188305,
+            0.010656657628715038,
+            -0.004085310269147158,
+            6.304640555754304e-05,
+            -0.027803342789411545,
+            -0.04464024305343628,
+            -0.002422143705189228,
+            0.013057502917945385,
+            -0.015636475756764412,
+            0.012670270167291164,
+            0.008743725717067719,
+            0.010021595284342766,
+            -0.005196669604629278,
+            0.02620794251561165,
+            -0.004793947096914053,
+            0.014304393902420998,
+            0.011733165942132473,
+            0.011911292560398579,
+            0.0009874446550384164,
+            0.0007459079497493804,
+            -0.018354853615164757,
+            -0.008255811408162117,
+            -0.0015954007394611835,
+            0.01765783317387104,
+            -0.027478067204356194,
+            0.008526874706149101,
+            -0.011516314931213856,
+            0.013839714229106903,
+            0.0003666614356916398,
+            0.005661349277943373,
+            -0.01771979033946991,
+            0.012445674277842045,
+            -0.0009337160154245794,
+            -0.025325050577521324,
+            0.004805563949048519,
+            0.011276230216026306,
+            -0.010362359695136547,
+            0.006877261213958263,
+            0.019950253888964653,
+            0.0023079097736626863,
+            0.008495896123349667,
+            -0.014126266352832317,
+            0.010703125037252903,
+            0.009649851359426975,
+            -0.021080974489450455,
+            -0.01233724970370531,
+            -0.007059261202812195,
+            -0.006335135083645582,
+            -0.0035470561124384403,
+            -0.00013105422840453684,
+            0.010176488198339939,
+            -0.016000475734472275,
+            0.010099041275680065,
+            -0.006559730041772127,
+            0.006400964688509703,
+            0.00944074522703886,
+            0.001115231541916728,
+            -0.013576395809650421,
+            0.012716737575829029,
+            -0.0218709297478199,
+            0.008426194079220295,
+            -0.04287445917725563,
+            -0.005847221240401268,
+            -0.015078860335052013,
+            0.005204414017498493,
+            -0.003798757679760456,
+            -0.004453181754797697,
+            -0.006203475873917341,
+            -0.005061137955635786,
+            0.00920066051185131,
+            0.021747015416622162,
+            0.012135888449847698,
+            -0.016620049253106117,
+            -0.007551047019660473,
+            0.005425137002021074,
+            0.0004854935978073627,
+            -0.013336311094462872,
+            0.009316830895841122,
+            -0.01576039008796215,
+            0.0009259713697247207,
+            0.006339007057249546,
+            -0.01072635967284441,
+            0.013452480547130108,
+            0.004731989465653896,
+            -0.003401843598112464,
+            0.013390523381531239,
+            -0.0050417762249708176,
+            -0.0057891360484063625,
+            -0.007272239308804274,
+            -0.013739033602178097,
+            -0.003818119177594781,
+            -0.003072695340961218,
+            0.012747716158628464,
+            0.001529571134597063,
+            -0.0003243077953811735,
+            -0.00782985519617796,
+            0.009154192171990871,
+            0.002220782218500972,
+            -0.0013146566925570369,
+            0.005773646757006645,
+            -0.02300165221095085,
+            0.0001287550403503701,
+            0.004755223635584116,
+            -0.0009487213101238012,
+            -0.006246071308851242,
+            -0.012151377275586128,
+            -0.011903548613190651,
+            -0.038165703415870667,
+            0.009363298304378986,
+            0.0019197085639461875,
+            -0.0076207490637898445,
+            0.0064822835847735405,
+            0.017255110666155815,
+            0.01456771232187748,
+            0.016743963584303856,
+            -0.020693741738796234,
+            -0.02303262986242771,
+            0.005556796211749315,
+            -0.009402021765708923,
+            0.02261441759765148,
+            0.004046586807817221,
+            -0.005847221240401268,
+            0.0013243375578895211,
+            -0.013258864171802998,
+            0.008929597213864326,
+            0.005723306443542242,
+            -0.001592496526427567,
+            0.005425137002021074,
+            -0.013692565262317657,
+            -0.010904486291110516,
+            0.01048627495765686,
+            0.032496608793735504,
+            -0.0004261985304765403,
+            0.0024105266202241182,
+            0.023295948281884193,
+            0.015466093085706234,
+            0.016356728971004486,
+            -0.021111954003572464,
+            0.011167804710566998,
+            0.010060317814350128,
+            -0.0018645278178155422,
+            -0.014892988838255405,
+            0.00513083953410387,
+            -0.012716737575829029,
+            -0.010664401575922966,
+            0.014993669465184212,
+            0.007853088900446892,
+            0.017735280096530914,
+            -0.02369867078959942,
+            0.00789568480104208,
+            0.00042765066609717906,
+            0.00013710475468542427,
+            0.011524059809744358,
+            0.012956822291016579,
+            0.03370477631688118,
+            0.027152791619300842,
+            0.020662762224674225,
+            -0.009285852313041687,
+            0.014366351068019867,
+            -0.009192915633320808,
+            -0.011090358719229698,
+            -0.019470084458589554,
+            -0.015249243006110191,
+            -0.011136826127767563,
+            0.017998598515987396,
+            0.008790193125605583,
+            -0.009355553425848484,
+            -0.005947901867330074,
+            0.01586107164621353,
+            -0.003909118939191103,
+            -0.004635181277990341,
+            0.01646515540778637,
+            -0.026269899681210518,
+            0.016836898401379585,
+            -0.000745423894841224,
+            -0.014668392948806286,
+            -0.014691627584397793,
+            -0.004182118456810713,
+            -0.012670270167291164,
+            0.0003964299685321748,
+            -0.020817656069993973,
+            0.013181418180465698,
+            0.024597052484750748,
+            -0.0030920570716261864,
+            -0.004100799560546875,
+            -0.011299464851617813,
+            -0.0005111477803438902,
+            0.018385831266641617,
+            0.0034773540683090687,
+            0.009192915633320808,
+            -0.0023872926831245422,
+            0.013924905098974705,
+            0.00524313747882843,
+            0.0027958236169070005,
+            -0.0007565568666905165,
+            -0.016263794153928757,
+            0.012197845615446568,
+            0.0026835259050130844,
+            0.012135888449847698,
+            -0.022242674604058266,
+            0.007938280701637268,
+            -0.011764144524931908,
+            -0.0023698671720921993,
+            -0.02228914201259613,
+            0.0189744271337986,
+            -0.01610890030860901,
+            0.001046497724018991,
+            0.003841353114694357,
+            0.0006224773242138326,
+            0.009580149315297604,
+            0.01610890030860901,
+            -0.023388884961605072,
+            -0.0003460896550677717,
+            -0.014908477663993835,
+            -0.021917399019002914,
+            -0.01607792265713215,
+            0.004728117026388645,
+            -0.019501063972711563,
+            0.005444498732686043,
+            -0.0025557391345500946,
+            -0.015125328674912453,
+            0.018478767946362495,
+            0.014691627584397793,
+            -0.004379607271403074,
+            0.002412462839856744,
+            0.005425137002021074,
+            -0.0014405074762180448,
+            0.01971791312098503,
+            0.00013274837692733854,
+            -0.0010048701660707593,
+            -0.009425255469977856,
+            -0.0168833676725626,
+            -0.0065752193331718445,
+            -0.035005878657102585,
+            0.0260530486702919,
+            -0.004670032300055027,
+            -0.016511622816324234,
+            0.014319883659482002,
+            0.009115469641983509,
+            -0.014629669487476349,
+            -0.007059261202812195,
+            0.016294771805405617,
+            -0.024287264794111252,
+            0.02244403585791588,
+            -0.009611127898097038,
+            -0.008774704299867153,
+            0.00784921646118164,
+            0.05154848098754883,
+            0.009146448224782944,
+            0.003492843359708786,
+            0.011036145500838757,
+            0.005839476361870766,
+            -0.0013165927957743406,
+            -0.023683181032538414,
+            0.005568413063883781,
+            -0.007190920412540436,
+            -0.003601268632337451,
+            0.002536377403885126,
+            -0.004801691509783268,
+            0.031149037182331085,
+            -0.004379607271403074,
+            -0.005289605353027582,
+            0.004484160337597132,
+            -0.02196386642754078,
+            -0.0001332324172835797,
+            0.023203013464808464,
+            -2.3128106477088295e-05,
+            -0.023590246215462685,
+            -0.006571347359567881,
+            -0.015071115456521511,
+            -0.013770012184977531,
+            -0.018989915028214455,
+            0.005231520161032677,
+            -0.012755461037158966,
+            -0.0033940989524126053,
+            0.0016660707769915462,
+            -0.005940156988799572,
+            -0.002582845278084278,
+            0.009603383019566536,
+            0.05362405255436897,
+            0.014103032648563385,
+            -0.006335135083645582,
+            -0.005854965653270483,
+            0.009192915633320808,
+            -0.010524998418986797,
+            -0.019795360043644905,
+            -0.009820234030485153,
+            -0.029507169499993324,
+            -0.012747716158628464,
+            -0.0143121387809515,
+            -0.010191977955400944,
+            -0.010277168825268745,
+            0.00963436160236597,
+            -0.018122512847185135,
+            -0.007628493942320347,
+            -0.020337486639618874,
+            0.002941036131232977,
+            -0.011268485337495804,
+            -0.014714861288666725,
+            -0.008054450154304504,
+            -0.04807887226343155,
+            0.005796880926936865,
+            0.019593998789787292,
+            0.01263929158449173,
+            0.002515079453587532,
+            0.004383479710668325,
+            -0.01993476413190365,
+            -0.030792783945798874,
+            0.013839714229106903,
+            0.011283975094556808,
+            0.011733165942132473,
+            0.03283737599849701,
+            -0.011709931306540966,
+            -0.00813964195549488,
+            -0.004693266469985247,
+            -0.0017251238459721208,
+            0.012066186405718327,
+            0.00969631876796484,
+            -0.004917861428111792,
+            -0.012375972233712673,
+            0.016217326745390892,
+            -0.01981084980070591,
+            -0.023683181032538414,
+            -0.01672847382724285,
+            0.006157007534056902,
+            0.00813964195549488,
+            -0.008054450154304504,
+            0.002973950933665037,
+            0.0056962003000080585,
+            -0.007159941829741001,
+            -0.00022423222253564745,
+            0.004890755284577608,
+            -0.023048119619488716,
+            0.0008490087930113077,
+            -0.002257569460198283,
+            0.0061376458033919334,
+            0.021266845986247063,
+            -0.010168743319809437,
+            -0.0014075926737859845,
+            -0.0053902859799563885,
+            -0.0028984404634684324,
+            -0.006764963734894991,
+            0.02620794251561165,
+            0.009154192171990871,
+            -0.009882191196084023,
+            -0.01011453103274107,
+            -0.01129171997308731,
+            -0.0011278166202828288,
+            -0.017255110666155815,
+            -0.016186347231268883,
+            0.0015566773945465684,
+            -0.016712984070181847,
+            0.013971373438835144,
+            -0.00876695942133665,
+            0.00400399137288332,
+            -0.014110777527093887,
+            0.02019808255136013,
+            0.01929970271885395,
+            -0.009711808525025845,
+            0.023776117712259293,
+            -0.007709812838584185,
+            -0.038382552564144135,
+            -0.00800798274576664,
+            0.01843230053782463,
+            -0.006118284538388252,
+            0.010037084110081196,
+            -0.011314953677356243,
+            0.008209343999624252,
+            0.00015319912927225232,
+            0.0018122512847185135,
+            -0.0003112386621069163,
+            0.027555514127016068,
+            0.015535795129835606,
+            -0.005560668651014566,
+            -0.005494839046150446,
+            0.004255692940205336,
+            0.009541425853967667,
+            0.017580386251211166,
+            0.012151377275586128,
+            -0.021019017323851585,
+            -0.014257925562560558,
+            -0.016263794153928757,
+            0.011012911796569824,
+            0.015125328674912453,
+            -0.0040891822427511215,
+            0.01691434532403946,
+            0.014126266352832317,
+            -0.019098341464996338,
+            0.0019884423818439245,
+            -0.008193854242563248,
+            -0.00158184755127877,
+            -0.03029712475836277,
+            0.011043890379369259,
+            0.0074387495405972,
+            -0.0072218989953398705,
+            0.036616768687963486,
+            0.009858956560492516,
+            0.037577107548713684,
+            0.018680129200220108,
+            0.0008974128868430853,
+            0.009045767597854137,
+            -0.018370341509580612,
+            0.013065247796475887,
+            0.01500141341239214,
+            -0.012724482454359531,
+            -0.005734923295676708,
+            0.0013446672819554806,
+            0.0007551047019660473,
+            0.012631546705961227,
+            0.02004319056868553,
+            -0.01795213110744953,
+            0.009773765690624714,
+            -0.015032391995191574,
+            -0.005250881891697645,
+            0.006555858068168163,
+            -0.013932649977505207,
+            -0.008495896123349667,
+            -0.00963436160236597,
+            0.019408127292990685,
+            -0.012600568123161793,
+            0.016573579981923103,
+            -0.009595638141036034,
+            0.015520306304097176,
+            -0.029724018648266792,
+            -0.003669034456834197,
+            -0.006679772399365902,
+            -0.019593998789787292,
+            -0.012887120246887207,
+            0.009820234030485153,
+            0.010633422993123531,
+            -0.009301341138780117,
+            -0.011562783271074295,
+            0.020368466153740883,
+            -0.004185990896075964,
+            0.013529927469789982,
+            0.021189400926232338,
+            0.00737291993573308,
+            0.024132372811436653,
+            0.0005605200421996415,
+            0.014164989814162254,
+            0.005463860463351011,
+            0.010362359695136547,
+            -0.004677776712924242,
+            0.005161818582564592,
+            0.00932457484304905,
+            0.014203713275492191,
+            0.017766257748007774,
+            0.013196907006204128,
+            -0.00598275288939476,
+            0.007419387809932232,
+            0.02016710489988327,
+            0.0016254113288596272,
+            -0.030312614515423775,
+            -0.009719553403556347,
+            0.0001736498816171661,
+            0.00924712885171175,
+            0.009185170754790306,
+            3.817877222900279e-05,
+            -0.01093546487390995,
+            0.011144571006298065,
+            0.00969631876796484,
+            0.002286612056195736,
+            0.006280922330915928,
+            -0.01456771232187748,
+            -0.020957060158252716,
+            0.006106667220592499,
+            0.01798310875892639,
+            0.013917161151766777,
+            -0.014722606167197227,
+            0.022784801200032234,
+            0.006145390681922436,
+            -0.021406250074505806,
+            -0.01840132102370262,
+            -0.0009559819591231644,
+            0.0004542729293461889,
+            -0.002265314105898142,
+            0.03720536455512047,
+            -0.0011539548868313432,
+            -0.005955646280199289,
+            -0.0054754773154854774,
+            0.02775687538087368,
+            0.02049238048493862,
+            0.009177426807582378
+          ],
+          "embedding": [
+            0.0008215984936325879,
+            -0.019349418284688353,
+            -0.02322660633029795,
+            0.009747820845194852,
+            0.007353771220903641,
+            0.015252908783199839,
+            -0.012176175829980202,
+            0.051898051094966605,
+            -0.0068130353886146165,
+            0.008189212816890595,
+            0.008102287831931905,
+            0.01682801136011614,
+            0.02005557996049711,
+            -0.020779879580497183,
+            0.002332165268329781,
+            0.02337485065616377,
+            0.006074992895540845,
+            0.02576575146535689,
+            0.0007517286429767504,
+            -0.03763909942314134,
+            0.0011204729720314833,
+            0.00041212613589632806,
+            -0.06089176800314795,
+            0.012852035873395388,
+            -0.008901127092058543,
+            0.01702347055484216,
+            0.005803446395466281,
+            0.017889570275475296,
+            -0.05375047382413552,
+            0.021623778774252724,
+            0.014784317465883424,
+            -0.0018881880857885708,
+            0.0004735328554290247,
+            0.012393802850407656,
+            -0.019234442968055952,
+            -0.019566283611638122,
+            0.04407789373296535,
+            -0.0053565873209102995,
+            -0.026240866282480922,
+            0.048107902270426675,
+            -0.0054681314939387235,
+            -0.0009441397948041965,
+            -0.015205079626883548,
+            0.006885981841893229,
+            0.01403724256866746,
+            -0.026086548246792754,
+            0.0034778946051893067,
+            0.00044587533948775473,
+            0.032851328435891634,
+            -0.03882097333255334,
+            -0.011126344604254894,
+            -0.01648943336384904,
+            0.035810456555802166,
+            -0.019384589526165754,
+            -0.019430702768170285,
+            0.014297440272239852,
+            -0.007883453204663151,
+            0.050996900793086844,
+            0.06007149434153715,
+            0.03406832053483379,
+            0.025193439934671852,
+            0.00027785129663604897,
+            -0.008014391056168083,
+            -0.018806730908490114,
+            0.027193788903098558,
+            -0.006195006670328595,
+            -0.020466000344352826,
+            0.0068939884219589185,
+            0.017252276167086602,
+            -0.045073444648704136,
+            -0.049242269917908754,
+            0.023021140866008313,
+            -0.015676872472071106,
+            -0.024497517908898602,
+            0.005873633718973869,
+            -0.022828585938203683,
+            0.011748341004940436,
+            0.04624205727927451,
+            -0.015442165499674641,
+            0.018565551071491767,
+            0.015246505395507096,
+            0.004119269796224144,
+            0.004862735250819252,
+            -0.000145323321960046,
+            0.05155838952887077,
+            0.0027052423345323936,
+            -0.016218063670586664,
+            -0.07462470920944148,
+            0.043695974746295135,
+            -0.027608273720377708,
+            -0.03860912774695494,
+            0.002823110565150435,
+            0.026471469849508766,
+            -0.0026063577189074103,
+            -0.029230652014151735,
+            0.05684164470477843,
+            0.00022182868300572677,
+            0.015590845552956364,
+            -0.024939617549529242,
+            0.01035803286203917,
+            -0.010225298012384327,
+            0.019571847648759157,
+            0.01841613041441054,
+            0.0156981470538354,
+            0.021764297694793343,
+            -0.018831230007033416,
+            -0.014419529638789993,
+            -0.0029965475899652273,
+            0.028881185012095884,
+            0.09163526460163937,
+            0.022644302207811818,
+            -0.017933909936330133,
+            -0.04663928709016869,
+            0.004962951444105128,
+            -0.041333041174526026,
+            -0.027545628007617527,
+            -0.037142394579894954,
+            -0.013358143162788402,
+            0.02719686432396677,
+            -0.0034971800937504205,
+            0.04413635817371732,
+            0.04903857110328917,
+            0.03186067155630788,
+            0.007831051050269414,
+            -0.025257538623313574,
+            0.023517513773109414,
+            -0.0016527403209505964,
+            -0.0016101860707429347,
+            0.014448515185903624,
+            0.009905018312477684,
+            0.0024114582235163955,
+            0.00172043851384822,
+            -0.03818316221102173,
+            -0.02304537008110578,
+            0.01024969233570089,
+            -0.01901116356327339,
+            0.019791121460848986,
+            0.07985067340821365,
+            -0.051903191272236665,
+            -0.025924646976556942,
+            0.014766511532849252,
+            -0.002073086076894999,
+            0.030959269926800852,
+            0.006394417986617089,
+            -0.028177586693713442,
+            0.010725285336674526,
+            0.0009614052276924192,
+            0.07500253788978564,
+            -0.00036794182149960157,
+            0.004058458332826519,
+            -0.013218269166951234,
+            0.027613539268683245,
+            -0.012995170442480582,
+            0.023388707018488407,
+            0.026389423313105165,
+            -0.003068639956061266,
+            -0.02661967649632026,
+            0.026145274570635913,
+            -0.03300591322174331,
+            -0.004556072066733024,
+            0.020045722084112896,
+            0.0033437343914784865,
+            -0.010224220905483085,
+            -0.0025025498684079703,
+            0.03460103245395426,
+            0.04514362357278504,
+            0.03797031988058962,
+            -0.004586662475576956,
+            -0.06351723722596923,
+            0.06037983484369874,
+            0.019415753124363902,
+            0.028850618647994476,
+            -0.0034810237045849006,
+            0.02957938154808314,
+            0.02642652523231673,
+            0.013187653724133275,
+            -0.007856320804109569,
+            -0.007065882659482447,
+            -0.016672784625197085,
+            0.029020161154209707,
+            -0.013243058921970134,
+            -0.014900932834241732,
+            0.006001769901896033,
+            0.008648456186944271,
+            0.03484763413464497,
+            0.012279368984306564,
+            0.02379643508687133,
+            0.04933044959399941,
+            -0.021047723935647383,
+            -0.007264823701706426,
+            0.0007058774795716792,
+            -0.010512071832195085,
+            0.027489671668749157,
+            0.009731157570361245,
+            -0.03195919640758579,
+            -0.0033626031972691915,
+            -0.04282419484651207,
+            0.01873977910485268,
+            -0.014398285316695653,
+            0.012854631447494207,
+            -0.0017372959978092432,
+            -0.037331867825735776,
+            0.001088209292238142,
+            -0.012345684323522151,
+            0.037096063326414734,
+            0.005050219690024652,
+            0.0008906051515353004,
+            -0.003697709449630797,
+            0.006548479142737107,
+            -0.010727663712299397,
+            -0.007486226350610755,
+            -0.02704618481069485,
+            -0.019752552016312926,
+            -0.014632581296303486,
+            -0.0037501138119724717,
+            0.014588411795703245,
+            -0.007119137071719376,
+            0.016440258608998535,
+            -0.03686645352111013,
+            -0.023546402448713415,
+            0.005628703432152666,
+            0.024988145751166143,
+            0.011971199595808573,
+            -0.010868778329190568,
+            -0.03270046433624425,
+            -0.021950559285357267,
+            -0.011889914022657738,
+            -0.005123971523701009,
+            0.02619713302280345,
+            -0.021990810305904974,
+            0.01165257474485436,
+            -0.009117901804315863,
+            -0.007492082755249399,
+            0.010725902070291232,
+            0.01862811996181951,
+            -0.026694281868322685,
+            -0.01623776415010199,
+            -0.009403997538684658,
+            -0.02207329027097839,
+            0.025939540497196416,
+            0.014178803271471762,
+            -0.001141821318145744,
+            -0.006281187017652108,
+            -0.035993365198554964,
+            0.0026192776111449633,
+            -0.015254976771874396,
+            -0.0116872453009606,
+            -0.015780016491993135,
+            0.021421328283124968,
+            0.02889432250124136,
+            -0.01871009416296692,
+            -0.026540472640514536,
+            0.015703370682110186,
+            -0.0002849235245164438,
+            -0.02594930889995505,
+            0.04491828744809679,
+            0.044063522197410734,
+            -0.037441864491547405,
+            -0.007005339187980476,
+            -0.030381794393931814,
+            0.06901083845539985,
+            -0.017604082400603037,
+            -0.04060591381445593,
+            0.06044850935418443,
+            -0.01756490171206357,
+            0.025539717592705295,
+            0.03452018789549939,
+            0.0016217577522523927,
+            0.00018825054264059277,
+            -0.002563606092068712,
+            0.04770277039047993,
+            -0.0032010143255574255,
+            -0.03425328601475849,
+            -0.02105909767072799,
+            -0.00840881022563103,
+            -0.028699447677549002,
+            -0.015925160861589006,
+            -0.0055750742281166,
+            0.017513987082479483,
+            0.010399718005952185,
+            0.02267201252273823,
+            0.01258511878601025,
+            0.0015081978484714903,
+            0.01874981572091931,
+            0.011158889421057756,
+            -0.002607810686437211,
+            -0.003732023411827874,
+            -0.007787189972165309,
+            -0.05524012485122728,
+            -0.025110689689279583,
+            -0.03946383236703947,
+            0.004443572529325475,
+            -0.018745077598477586,
+            0.005926160888274727,
+            0.039430331582190524,
+            -0.0049500324516682,
+            0.004352795787899792,
+            0.015566139991780173,
+            0.013148156356408315,
+            -0.015279560337042242,
+            -0.03316846614871165,
+            0.0001592209771257332,
+            0.018362411161743347,
+            -0.012706582418607377,
+            -0.03424477956681259,
+            -0.001991022841226589,
+            0.0033467286047492175,
+            0.03357019284647071,
+            -0.024359110796680374,
+            0.00359117029951049,
+            0.005061818038334802,
+            0.011359582931760315,
+            -0.026793992746244076,
+            -0.018170136079817746,
+            -0.017421603453826397,
+            0.03996608926471779,
+            0.02587013058343799,
+            0.015763410145648683,
+            -0.04512536449983358,
+            -0.02293473890636517,
+            0.048575083783480195,
+            -0.009589860756342604,
+            -0.06486262416944208,
+            0.03447666009581266,
+            0.016105654870253654,
+            0.003526582643722421,
+            -0.01354754623151814,
+            0.05401575248587799,
+            -0.03563190295559527,
+            -0.017365166993914202,
+            -0.0046288709698227735,
+            -0.033657212157212645,
+            -0.007686252059880836,
+            -0.001385273642139782,
+            0.048229062789530236,
+            -0.029083855744728694,
+            -0.030465559177859865,
+            0.0024368661249980416,
+            -0.024144031154493057,
+            0.021693278579520123,
+            -0.016405088055634124,
+            -0.018439935250344958,
+            0.006948603359995389,
+            -0.033943037395199695,
+            -0.003007796423804453,
+            -0.016783195426536633,
+            -0.02270589491206155,
+            0.020874438753828814,
+            0.016724666903417947,
+            0.027033184180344852,
+            0.0019847647852695785,
+            0.017749524526371133,
+            0.0204661893602285,
+            -0.005604929968008821,
+            0.028462234633072944,
+            -0.020790459496727066,
+            -0.04183816355941877,
+            0.03946487892663695,
+            0.01192861758684541,
+            -0.0018024841275380242,
+            0.01077082591611702,
+            -0.01475239020295694,
+            -0.03952356245715099,
+            0.03683950018557526,
+            -0.056767672022551764,
+            -0.007507600090017693,
+            0.009530544352246083,
+            -0.00788856051420074,
+            -0.011204694027562521,
+            -0.010587701466318598,
+            -0.028604940377315504,
+            0.023749537635431462,
+            -0.013672419168235691,
+            0.01953449229707597,
+            -0.011449939491554354,
+            -0.012238180529977605,
+            0.05795536247491174,
+            -0.021518581774137322,
+            -0.040672955795925836,
+            0.026066659238628543,
+            -0.014061106927896723,
+            0.03328357230532424,
+            0.03957107406164427,
+            0.030670947051333917,
+            -0.004278577327978564,
+            0.025885946540364826,
+            0.0319106101668716,
+            -0.003462160036198642,
+            -0.04686868330851868,
+            0.00555303933159587,
+            0.02156143971867743,
+            -0.03739529489075782,
+            -0.01025626126223168,
+            -0.02141970191890338,
+            0.022134910980548407,
+            0.017807484142010496,
+            -0.007232643256393455,
+            0.011318041192005274,
+            -0.010489012862455913,
+            -0.013003217163997554,
+            -0.00847279880392288,
+            0.02357274819910594,
+            -0.0016485066139744577,
+            -0.005873113634992993,
+            0.006424249429163587,
+            0.004008703814301658,
+            0.0054639858216457724,
+            0.010198079500408441,
+            0.017694720520798957,
+            -0.017087021465208255,
+            -0.0025558869389290554,
+            -0.016503579212015177,
+            -0.0420082404922411,
+            -0.003563173424359332,
+            0.022924161700566468,
+            0.012345933179990547,
+            -0.00729908668365632,
+            -0.004234683369642856,
+            -0.0022962545569187934,
+            -0.020666205489143392,
+            -0.014796149043465308,
+            0.024562653223873056,
+            0.030217494971086438,
+            0.007810961433007407,
+            0.007781263640449456,
+            -0.0030348559388780655,
+            0.007310876971216526,
+            0.016099591640649783,
+            0.02311929597780174,
+            -0.04160162038353416,
+            0.027152100845897007,
+            -0.00025396315502635146,
+            0.023717123370023598,
+            -0.04186918759097667,
+            -0.017988531857836703,
+            -0.03106663560912813,
+            -0.023595079699181436,
+            0.02437134274097885,
+            -0.02317559303912261,
+            -0.012327263775257306,
+            0.000989566600374836,
+            0.00954921216572729,
+            -0.005155079395685017,
+            -0.017508404061379164,
+            -0.04582321461886958,
+            -0.0014367264231457263,
+            -0.030417229142190933,
+            -0.017829710557332023,
+            -0.001056472801554354,
+            0.005016912084754242,
+            -0.06276644565618253,
+            0.0109965681725148,
+            -0.07270669271977365,
+            -0.004542796125637244,
+            0.040139998232966566,
+            -0.017890441783630643,
+            0.055909102081208445,
+            -0.05029104308417623,
+            0.01673962668706085,
+            -0.04507293803055318,
+            0.015385506084493484,
+            0.025677372277539627,
+            0.030235171586316813,
+            -0.022189978375852563,
+            0.008461188938299745,
+            0.008033707483259569,
+            -0.045089505755429415,
+            -0.04144698743458133,
+            -0.013640256640978024,
+            0.01036782197518379,
+            0.0012696958402094232,
+            -0.016090728932978384,
+            0.023838437877040163,
+            0.011946482002767135,
+            0.03342604921859589,
+            0.04211801335157251,
+            0.002708015395637621,
+            -0.047156646610976634,
+            -0.013104464812562618,
+            -0.003933116891120681,
+            0.03155321708116329,
+            -0.018866109579446184,
+            0.015376812010630058,
+            -0.01955557376085726,
+            0.016255217383093355,
+            0.017868332102274922,
+            -0.020325179506828335,
+            0.013434364256590097,
+            0.004176904701252156,
+            -0.03877060944639854,
+            0.0006475542749369515,
+            0.00042468300294954125,
+            -0.02116992978038693,
+            0.0032654387271108293,
+            -0.00575003786159731,
+            -0.05149954071159492,
+            -0.031382926943128234,
+            -0.015730563640253065,
+            0.0382312502769352,
+            0.004297939848910064,
+            0.019724983545557365,
+            -0.017907869879607064,
+            -0.005165628493894463,
+            0.03698976707579373,
+            0.008501001373266799,
+            -0.01725527821298066,
+            0.006879620077191753,
+            0.02455792200929138,
+            -0.03224193972301503,
+            -0.039181328884413266,
+            -0.0008206986106407542,
+            -0.023306758893533952,
+            0.01672946960876128,
+            -0.02658587698869307,
+            0.005873620623736886,
+            0.0025331170979801232,
+            -0.03490809296591325,
+            0.02456764314036355,
+            0.003291113462353163,
+            -0.004303423672882112,
+            -0.020359544091059775,
+            -0.016892415165120203,
+            0.017174797487220006,
+            0.04391489511478065,
+            0.007327321684948843,
+            -0.003943713832544148,
+            0.00862684231313155,
+            -0.006361172785753631,
+            0.05826688726534012,
+            -0.02284956969636637,
+            -0.026680023999604643,
+            0.027527436347931256,
+            -0.022597669279069723,
+            -0.006277863878949772,
+            -0.017343446085471857,
+            -0.012065192152203348,
+            -0.0004183712490442143,
+            0.019339311388446943,
+            -0.019423562920897294,
+            -0.010212527447243049,
+            0.004783006556748757,
+            0.012824133870627983,
+            -0.004855945108275706,
+            0.006239949458562236,
+            0.006847581690471813,
+            0.03094005096378827,
+            0.026993131409761086,
+            -0.011331881651539968,
+            0.03297014291876675,
+            0.010265600643742144,
+            4.666113318251278e-05,
+            0.001970879281172325,
+            0.002523664335535017,
+            0.0034107811520368574,
+            -0.03143890864030425,
+            -0.01624584278940617,
+            0.013468729048885906,
+            0.005233309338223403,
+            -0.004550544967856448,
+            -0.023783653541634683,
+            -0.009402494239255264,
+            0.03334923712760375,
+            -0.003260424721959948,
+            0.02748395114281811,
+            -0.0058374897745283065,
+            -0.01985840930273332,
+            0.017595418271610163,
+            0.02881743246631328,
+            -0.05928552440605384,
+            0.017062540894682354,
+            -0.0011508547522925793,
+            0.008953761538519997,
+            0.01610561914042553,
+            -0.008894160513648953,
+            0.013301991014533684,
+            0.04080863883897929,
+            0.010738051773755976,
+            -0.023345002072886145,
+            0.004215726310838674,
+            0.01541749448570608,
+            0.01291813299324752,
+            -0.022747942296528025,
+            0.0008640791861357522,
+            0.011008522066277018,
+            0.005893793007012561,
+            0.0026106450193140156,
+            -0.006278394080583205,
+            -0.025659188287146268,
+            0.024180550333596625,
+            -0.0044491292740074925,
+            0.026967973346698617,
+            0.04390909273429254,
+            0.009596855239692042,
+            -0.03686020608700941,
+            -0.013109951664921729,
+            0.0002799680359670127,
+            -0.0038537577602593366,
+            -0.0037552094562213916,
+            0.011105295862811114,
+            -0.04185615640149602,
+            0.019950890919369135,
+            0.006453241355231516,
+            0.015314645427201533,
+            -0.020031193483734364,
+            -0.024530173488182337,
+            0.0031949828520724337,
+            -0.01492645564652531,
+            0.012875827757040941,
+            0.005057170413681063,
+            -0.004319834710426384,
+            0.005386862713494232,
+            0.00030848054013413157,
+            0.008384941028326983,
+            -0.010811526993301901,
+            0.03175068297908541,
+            0.004680987487229685,
+            0.009264547472281849,
+            -0.02856314241700808,
+            -0.021083721348429348,
+            0.020113396771629096,
+            0.03572366771051171,
+            -0.01708529918264109,
+            0.03346921724975398,
+            -0.009042046660366718,
+            0.0033219155387035227,
+            0.0151720168318272,
+            -0.015805318210244414,
+            -0.008057236707694303,
+            -0.0005483009439015931,
+            -0.002917527004423299,
+            -0.014538511124345527,
+            0.007302979537905532,
+            -0.0208529958115422,
+            -0.02965227242147972,
+            0.020064178921373875,
+            -0.00011892622932349096,
+            0.016455188521316146,
+            0.010335044657747514,
+            -0.007816155992852236,
+            -0.0052262330850905165,
+            0.006959344748883874,
+            0.04432147882994085,
+            0.026734484553994495,
+            0.05547661378166294,
+            -0.005787997952888261,
+            -0.004559444854138438,
+            -0.0082295581123363,
+            -0.006588172755707144,
+            0.04524009609473585,
+            0.015915003261638767,
+            -0.004910285095602864,
+            0.017495607570191165,
+            0.01557886571266685,
+            0.03858259260136037,
+            0.0027648525902558316,
+            -0.016912796336835408,
+            0.04014755669075221,
+            -0.004077405782937414,
+            0.008616827127381418,
+            0.011102692199778469,
+            0.0024185742218798562,
+            0.00807141794538637,
+            0.02996274720815659,
+            -0.012429186019907761,
+            0.01680942269506771,
+            -0.04571916592313758,
+            -0.02775530246832996,
+            0.014370280587419414,
+            -0.004931905540824582,
+            0.03419959665162407,
+            0.01668521021731861,
+            -0.023105305446763272,
+            -0.024761385385514877,
+            0.021529428247380984,
+            -0.002603042335561961,
+            -0.024135326707477325,
+            -0.015906833215844473,
+            0.003286605488908195,
+            -0.0023437997044735617,
+            0.0060251848371355166,
+            -0.013481652076194643,
+            0.0002908426939460218,
+            0.01953362396253743,
+            0.0038544889405965057,
+            0.020760460182873287,
+            -0.016224396121239458,
+            -0.00259233446682179,
+            -0.009282861237653619,
+            -0.02952130765234087,
+            0.017988370145301916,
+            0.006138236653154911,
+            -0.007607182955958238,
+            -0.009516767050028553,
+            0.0048347778733230365,
+            -0.005819209642901871,
+            -0.0025851701689263706,
+            -0.01073014664868326,
+            0.012433465684360061,
+            0.025784014942833097,
+            -0.016875624785552303,
+            0.007944349714123104,
+            8.651589872318479e-05,
+            -0.022306044741692753,
+            -0.022127016745066307,
+            0.020143472307319017,
+            0.002792646249762062,
+            0.008768813346304779,
+            -0.004074214276300631,
+            -0.00015024341785520542,
+            0.002990550758627699,
+            0.01951746553838439,
+            0.03896451381932467,
+            0.004232673242619471,
+            0.027876586543280867,
+            0.0145860303663199,
+            0.03994234200854897,
+            0.009657017763624678,
+            0.00696607158576712,
+            -0.026055387694656488,
+            0.02819890357011778,
+            0.00018297502510753014,
+            0.043190622821775236,
+            0.022488402221902314,
+            -0.0407730101161195,
+            -0.0021946518523759066,
+            0.01669520746276111,
+            0.013212623257794862,
+            0.017009844136059122,
+            -0.004537512994041156,
+            -0.014041213424740049,
+            0.028135404528975894,
+            -0.022859586430516003,
+            0.0008637006862481223,
+            -0.004486332934188017,
+            -0.025031795510001887,
+            0.007351370254175368,
+            -0.004567816916440959,
+            0.030025413528233364,
+            0.0014976740344892735,
+            -0.022285096307825804,
+            0.022837170656498543,
+            -0.009551269792616045,
+            0.011045530488684207,
+            0.008809047185288755,
+            0.020065001354321994,
+            0.031993362733068575,
+            0.009143210434415064,
+            0.02133640814803879,
+            0.012241777474889248,
+            0.014771538127546474,
+            -0.008804192497858033,
+            -0.02341427260922065,
+            0.023167424359504862,
+            -0.0011598649565273595,
+            -0.029627604902846688,
+            0.0021991474725787303,
+            0.003261878079344105,
+            -0.0005935713331682358,
+            -0.009144689470851402,
+            -0.005580557862988033,
+            -0.007240836451875213,
+            0.019631377011298905,
+            -0.007085213886352692,
+            0.007017573363709754,
+            -0.004074171791476129,
+            -0.00822871835665378,
+            -0.0421101573155962,
+            0.004210914075286222,
+            0.003408427700622687,
+            -0.01752179330665445,
+            0.006384440404174512,
+            0.011121481025573962,
+            0.043020575455262985,
+            -0.022809979913698254,
+            -0.0005737138409049003,
+            0.019206623212867476,
+            0.0009562551889820241,
+            -0.023427467810802533,
+            -0.038931688376706186,
+            0.007172345672967499,
+            -0.03467050750542982,
+            0.023185301620612612,
+            -0.0007075596339842658,
+            0.02387977204565927,
+            -0.03043468414449851,
+            0.013909304572370437,
+            0.004784668158421625,
+            0.012524879174804915,
+            0.021654512058639427,
+            0.03142146922870715,
+            -0.02053473475540882,
+            -0.050409862638621725,
+            -0.005406667316025549,
+            0.00035138762584085976,
+            0.022018487016014856,
+            -0.026014580970974616,
+            0.00025633004699302783,
+            -0.003349094027291921,
+            -0.02401366769212778,
+            -0.01735996745614562,
+            0.01962340791446025,
+            -0.050011480783403936,
+            0.015042913576492952,
+            0.0262737424555697,
+            -0.02507473901367252,
+            -0.05408889321034101,
+            -0.018988587082325706,
+            -0.028217893770659686,
+            -0.009285770094319318,
+            -0.00661159230987788,
+            0.028597405154740078,
+            0.003525576256054959,
+            -0.011462219358952466,
+            0.025401668390029348,
+            0.006155837251830549,
+            0.0026050693705924593,
+            0.015985722392086222,
+            0.007289910674212967,
+            -0.011452511453623235,
+            -0.014599577191293152,
+            -0.04558584556100621,
+            0.010197536827229795,
+            -0.0045431962637690155,
+            -0.019372598050508005,
+            -0.030147301333718016,
+            -0.0137828704002156,
+            -0.027060515748528666,
+            -0.0017291966950107505,
+            0.053783684195381654,
+            0.03292494290301388,
+            0.005560172296221419,
+            -0.0026085670565688026,
+            -0.010466890489356092,
+            0.036247483457995895,
+            -0.006789746401727701,
+            0.03662690472838588,
+            0.001784522073210422,
+            -0.0242673253380092,
+            0.01169382884548301,
+            0.02892627181542449,
+            0.009858451662166387,
+            -0.0005592283645355213,
+            -0.006160132931009508,
+            -0.008782830470079547,
+            0.0031207630972208394,
+            0.025613233596604418,
+            0.003268170392795548,
+            -0.010062991425127038,
+            0.018106967407336815,
+            0.004480291699976396,
+            0.02010374270557836,
+            0.03555357587371324,
+            0.023489837757983054,
+            -0.021782173652026497,
+            -0.026581762796336082,
+            -0.0013561486781077014,
+            0.0072266106510934505,
+            0.03551967295942857,
+            -0.01651315514034218,
+            0.0001274396319698452,
+            -0.008272735507120127,
+            -0.00894931703949637,
+            0.011149952898504355,
+            -0.0029671593121986116,
+            -0.0030752010285951576,
+            0.01662711184497578,
+            -0.0011672263615879709,
+            -0.033448222499147764,
+            0.0078288560210653,
+            -0.023496331346917084,
+            0.0051040606404415545,
+            -0.004113104164416575,
+            -0.007076525381595441,
+            -0.002157548492138289,
+            0.006302962169276006,
+            -0.008065841090139872,
+            -0.00035242603988955366,
+            -0.01586406637763306,
+            0.008271894439780969,
+            0.007438312999409211,
+            -0.0069617481460406095,
+            0.011402932583388205,
+            0.002666895909532698,
+            0.0005219881149353855,
+            -0.012159425670629751,
+            -0.012042145306711322,
+            -0.00957276588761916,
+            -0.03401863879157551,
+            -0.02604353712472079,
+            -0.024949533021906844,
+            -0.0014209149651479556,
+            -0.006566685925666972,
+            -0.03187796739029736,
+            -0.01209152137926458,
+            -0.011110763124407752,
+            0.007756961097537267,
+            -0.013260548545917543,
+            0.005433285079114047,
+            -6.399098520041351e-05,
+            -0.0044960488826674555,
+            0.016615636438440516,
+            -0.003643291599919672,
+            -0.019488010211707343,
+            -0.03927818878149417,
+            0.008209336177358258,
+            -0.02805593125673831,
+            -0.0038816771550190788,
+            0.007607119219721711,
+            -0.00528681677614909,
+            -0.029502897838743516,
+            -0.009149794737430706,
+            0.008767001199311985,
+            0.003507217008280066,
+            -0.01773349756494436,
+            0.03579670773605949,
+            0.026971617790626075,
+            0.011875179879209696,
+            -0.019891342676443626,
+            0.0508929813275565,
+            -0.0047010735337056275,
+            -0.012544935684845649,
+            0.018297503869683027,
+            0.02355022268131519,
+            0.03273167043015095,
+            -0.024830243320788693,
+            0.0009370814985214882,
+            -0.021937793653233454,
+            0.00535662615596573,
+            -0.035009915581788176,
+            0.0028595659314725012,
+            -0.014395027018699446,
+            0.015596459769560532,
+            0.03397372881171701,
+            -0.013254226072981457,
+            0.011195747181764677,
+            -0.016299384375589934,
+            0.00730244731485402,
+            0.00569944090461748,
+            -0.012916284280345044,
+            -0.005812311882399539,
+            -0.022902815152116493,
+            0.020140276669979015,
+            0.005809799950176385,
+            -0.0008130655714656685,
+            0.0104085826652653,
+            -0.044723432238900435,
+            0.03286429597337671,
+            0.008730225944501239,
+            0.017599189944921396,
+            0.0029762446259779998,
+            -0.0003068671951232415,
+            0.011611551606224214,
+            0.011228401175698422,
+            -0.016029727902740706,
+            0.004345913009102136,
+            -0.004613061137378433,
+            -0.03850533909474302,
+            0.009363351681220008,
+            -0.03631184057978666,
+            0.014427784677340159,
+            0.016026872002233575,
+            -0.004170582842643461,
+            -0.025166189571409366,
+            -0.005428471012333673,
+            0.010827910811085384,
+            -0.009783069664327864,
+            -0.009987805234276443,
+            -0.04163358183060102,
+            -0.025236440364963034,
+            0.034097508113604576,
+            -0.002119474613596975,
+            -0.025826403511707147,
+            -0.010830738112107807,
+            -0.011351132636491575,
+            0.020207687645133,
+            -0.006542026283946858,
+            -0.0006263975804936843,
+            -0.04150011545311013,
+            -0.0073396962917864515,
+            0.011192225907726928,
+            0.027746213794734322,
+            -0.035970017302005566,
+            0.02708126943501373,
+            0.002105633542220187,
+            0.014262674845755191,
+            -0.01626664983003942,
+            -0.005631805361903626,
+            -0.013990539604372344,
+            -0.035877093361375685,
+            0.016487723013906077,
+            -0.008607758248055054,
+            0.004714436224135091,
+            0.04458009880881818,
+            -0.0036872068506712727,
+            -0.009466125348499464,
+            0.007600705817195143,
+            -0.0034590883553048063,
+            -0.010351576085287273,
+            -0.0028564505793999295,
+            0.001959661298507,
+            -0.007080697863126256,
+            -0.04183476693971616,
+            -0.00499888976530217,
+            0.04684391782633148,
+            0.006361744381879857,
+            0.013276793765092465,
+            0.013516220521120684,
+            -0.018335746557932003,
+            -0.045577810515431215,
+            -0.0071616111758423025,
+            -0.0032763880176386808,
+            -0.022708028392758567,
+            -0.0240284945459861,
+            -0.008735040856657758,
+            0.005844145402902923,
+            0.006359368165686724,
+            -0.01120763599008222,
+            -0.022676131361171057,
+            -0.027085968700452136,
+            -0.010526103176253857,
+            -0.02123944716136792,
+            -0.010792145796733375,
+            -0.027142270190771887,
+            -0.028742234538796804,
+            0.02880426953386151,
+            0.007575182232859564,
+            -0.004084088301176117,
+            -0.010005765476121966,
+            -0.01255640546813973,
+            -0.0004774761617900028,
+            -0.0025307994917461232,
+            -0.005822065709999144,
+            0.0038614286313282164,
+            -0.011390482620811868,
+            0.023152086963976554,
+            0.0021220701260546895,
+            0.018329162172128138,
+            -0.05772324571231113,
+            -0.002709083851090071,
+            0.021497896395795905,
+            0.0033025410488410913,
+            -0.005412445141910314,
+            0.029352650459792132,
+            0.011223947499453582,
+            -0.006539460282439711,
+            -0.031355428613718234,
+            0.020942976176820262,
+            0.024122993515785947,
+            0.00817615634503138,
+            -0.012994160510030828,
+            0.01221095914384204,
+            0.01629778384790159,
+            -0.004494985018246821,
+            -0.011161098655072933,
+            0.005586857967039181,
+            0.020941039642372112,
+            0.013580622629911925,
+            0.0013398071867077221,
+            0.030905022925366163,
+            0.0006862084391508121,
+            -0.03521231612862609,
+            -0.01908191995763771,
+            -0.01275995164030872,
+            -0.003584331305986364,
+            -0.015264425022763126,
+            0.005294329654809353,
+            -0.002355465010876059,
+            0.006335052149475905,
+            0.022655379849785256,
+            0.03593231939948758,
+            0.0146937743203089,
+            0.00937330328448886,
+            0.003167462724350819,
+            -0.0013235176101076284,
+            0.012102872918129403,
+            0.008638011773075119,
+            0.0350701240268404,
+            -0.014653221697693864,
+            -0.011003994230902835,
+            -0.0037767288519369537,
+            0.010100131061186566,
+            -0.004113527518662366,
+            0.020414393256136372,
+            0.03274709615649469,
+            -0.0037222949335916876,
+            -0.005604552180246868,
+            -0.006160118231975786,
+            0.0041467454414452565,
+            0.01807879127455274,
+            -0.01252134965321004,
+            0.004364858987501461,
+            0.005549300665159658,
+            -0.0071897162833861,
+            0.018567059255813558,
+            0.0002622074720860951,
+            -0.012983550838417323,
+            0.03266376564402191,
+            0.11449509387240629,
+            0.0017251916835047628,
+            0.007125215166916139,
+            -0.040229997815548584,
+            -0.020537987527148888,
+            -0.00245728735683473,
+            0.005779972828836593,
+            0.007004732428236457,
+            0.02225914982152608,
+            0.006505827715591914,
+            0.0006708045351142873,
+            0.031708522931495536,
+            -0.017334083511709092,
+            -0.002522861743921599,
+            -0.01745656559688129,
+            0.013890451996979048,
+            -0.014312605700272691,
+            -0.013784963572164409,
+            -0.01572417853955075,
+            0.004507425871748977,
+            -0.000601158156253668,
+            0.0016158265715634729,
+            0.01791303234734911,
+            0.0011926917387973374,
+            -0.026393351252156246,
+            0.0006694008039871878,
+            0.001026031515214626,
+            -0.004514419527575117,
+            -0.011320929719992686,
+            -0.02223001723942815,
+            0.016235841442653918,
+            0.014188311619681008,
+            -0.004895218832118094,
+            0.009244697376301004,
+            0.010558796153959677,
+            0.02707485418319703,
+            -0.028380342401606758,
+            0.0040204430418629405,
+            0.007023041225985294,
+            0.01606556581447933,
+            0.017617862031190843,
+            0.0034870847111552862,
+            -0.011253315705053224,
+            0.03238289399766116,
+            -0.010563282687352038,
+            0.0065093536270233385,
+            0.014966802208644462,
+            -0.03541611540945151,
+            -5.821682910348165e-05,
+            0.0061985444938402745,
+            -0.011716201661363997,
+            0.009580035961315467,
+            -0.03120692534097433,
+            -0.0039291243987978845,
+            0.009317468331974374,
+            0.017611312558635673,
+            0.008804996982574765,
+            -0.0034129169537151925,
+            0.010822226698822738,
+            0.012510661458054741,
+            -0.0027147975086409966,
+            -0.008855213644027594,
+            0.01238502686635956,
+            -0.0007687053446954701,
+            0.00450108863711973,
+            -0.00799170686059026,
+            0.0029755595812552293,
+            0.002290650917244691,
+            -0.01554856300008412,
+            -0.011779430686394945,
+            0.00438393126679289,
+            0.0036931891075561957,
+            0.001535662168147833,
+            0.012014408636737167,
+            -0.006087255045485002,
+            -0.0077816220325470226,
+            0.0040573724156794284,
+            0.003136943055301706,
+            -5.505014316363865e-05,
+            -0.030625245024561508,
+            0.01585923084309149,
+            0.0067595435845396405,
+            0.010013308598626031,
+            -0.005315626260228517,
+            -0.00903142414091241,
+            -0.007240920015018893,
+            -0.01228521172101564,
+            -0.010657894223173453,
+            0.007658605592936815,
+            -0.00724924993432376,
+            0.011124174211132038,
+            -0.03128027787357322,
+            -0.013988715435985316,
+            0.01231792993245938,
+            -0.01528161242159377,
+            0.0007675821323979293,
+            -0.008389914312201402,
+            0.0009417242487395995,
+            -0.005119304379761705,
+            0.007887436211143328,
+            -0.03731503169547854,
+            0.007878001990271541,
+            0.02456170736765724,
+            -0.00330924625934981,
+            -0.020259771366167557,
+            -0.009613930457931786,
+            0.0241314385136084,
+            -0.008267831641886344,
+            -0.01645888287387047,
+            0.008421304862524506,
+            0.010140813149093993,
+            -0.0006309118552875393,
+            0.0008237784055549769,
+            -0.003726984191590917,
+            0.01669125388798323,
+            0.02054777616420997,
+            0.041930770964107515,
+            -0.014789610432004559,
+            0.01092349703703152,
+            0.02866938974792608,
+            -0.03378576980722847,
+            0.014728113846952417,
+            0.02518409645155824,
+            0.014590531327650454,
+            -0.0004446956315190408,
+            0.001609777186372437,
+            0.0068939321280085475,
+            -0.014273492115311677,
+            0.009247676157551494,
+            0.013380104440744367,
+            -0.018407746260686045,
+            0.020444430509441124,
+            0.03140071554008003,
+            0.0025839998101985736,
+            0.01355205268903058,
+            -0.006268939830271642,
+            -0.013860692186849113,
+            0.014717364068116991,
+            0.031429975488060896,
+            0.0006919565473572909,
+            -0.018561500446336806,
+            0.019128244642842268,
+            0.006988095806264863,
+            -0.003678529269242034,
+            -0.0021993078588142664,
+            -0.021334919445200388,
+            -0.021003586791369228,
+            0.017304350704792804,
+            0.013508779408333964,
+            0.027794485858413365,
+            -0.010594702782952029,
+            -0.007818089330662161,
+            0.014293688127901867,
+            -0.022368568887004094,
+            0.015426482139881204,
+            -0.02160820059598751,
+            -0.00571499813703566,
+            -0.0077402210272634235,
+            0.03274360648629609,
+            0.001441304806200403,
+            0.02734331677589297,
+            0.016457027764064002,
+            -0.033609420134045206,
+            -0.029790535553657092,
+            -0.00712106584238484,
+            -0.02307933297713104,
+            0.010356561171963577,
+            -0.03901183838032092,
+            -0.00071735150189382,
+            0.008251646656432826,
+            -0.008079917288471352,
+            -0.0007087526632966007,
+            -0.006419191399519291,
+            0.013989344185652506,
+            0.013661633160336152,
+            -0.0228103143826663,
+            0.023644639659577426,
+            0.008343481940133684,
+            0.025998483906975782,
+            -0.02076055376006461,
+            0.019148568238437927,
+            -0.017705363646416224,
+            -0.02777313933338929,
+            0.014358130220942483,
+            0.023585731219788467,
+            0.01635364038245971,
+            -0.007122853965024528,
+            0.007110003583832299,
+            -0.03556761320536796,
+            0.027585457849964405,
+            -0.013064062623866305,
+            -0.012504161289952458,
+            -0.0032379276562094555,
+            -0.011931827532360686,
+            -0.002574161005800279,
+            0.0008533907465694845,
+            0.0034345232246102806,
+            -0.03131462673429805,
+            -0.0002450959360520896,
+            -0.03727416416766574,
+            -0.028944814604815072,
+            0.01360126708805471,
+            -0.00036361284498363273,
+            -0.0049097813612814616,
+            0.02933242705521636,
+            -0.026339136251046315,
+            -0.02693295650849956,
+            -0.009590247544093352,
+            0.02853096456887074,
+            -0.0220673798697511,
+            -0.004519050393279848,
+            0.0015034014871316069,
+            -0.01828226786700525,
+            0.004436383319946364,
+            -0.0012748481042235446,
+            -0.010416065158876917,
+            -0.01105146139270597,
+            -0.03229479099927737,
+            -0.005822133182096844,
+            0.02106215163620738,
+            -0.006957125617688484,
+            0.002746241228040847,
+            0.013312248331456934,
+            -0.010968629118224582,
+            -0.0029946900704643812,
+            0.0017875284441794235,
+            0.023304175718480834,
+            -0.021033941064173063,
+            0.007714334541806199,
+            0.005955442031111047,
+            0.0016405586860131207,
+            0.018406562522853148,
+            -0.0054922197063461825,
+            0.0037690571959124617,
+            0.0162126534419016,
+            -0.009969304596173302,
+            0.017959307638726878,
+            0.007691549802475525,
+            0.006681837561745192,
+            0.002775700618222622,
+            0.007712770283579111,
+            0.009752652973700739,
+            -0.013506020126078112,
+            0.016911141425375933,
+            0.022468715726349706,
+            0.00023233555209372766,
+            0.021066580799163234,
+            -0.02302516292580905,
+            -0.028618544628453015,
+            0.018297878810036087,
+            -0.0225078941299981,
+            -0.012735208680311652,
+            0.016459925496937065,
+            -0.0043032724578332855,
+            -0.00797609922711709,
+            -0.014986057073351055,
+            0.02260618561608301,
+            0.02132930921464508,
+            -0.020249788476562527,
+            -0.012385936327068884,
+            0.0072304431443959505,
+            -0.004373475447051833,
+            0.007086656081865733,
+            0.03195810137269375,
+            -0.015982210897392764,
+            0.006818890093901413,
+            -0.0038306138017521626,
+            0.0030753642449792215,
+            0.017830546304667783,
+            0.009470018737705222,
+            -0.0019815241970236146,
+            0.030100238416495066,
+            -0.001330614013444443,
+            0.016392226282459177,
+            0.0004888028477871524,
+            -0.010948833692356176,
+            0.006343890495626752,
+            -0.026652806306173643,
+            -0.008627570055007363,
+            -0.001203530223345183,
+            -0.009581494451831351,
+            0.07598616930784373,
+            0.003930153552069478,
+            -0.01295957554290434,
+            -0.001673007115291578,
+            -0.001368287102181597,
+            -0.006897179139643672,
+            -0.00960679451603541,
+            0.0287166325268306,
+            -0.016613613331623346,
+            -0.01894527592131826,
+            0.0008370014132002703,
+            -0.006785618888421608,
+            0.00919924251249592,
+            0.01816363936999796,
+            -0.010225961928154075,
+            -0.01604759761423727,
+            0.001577138242530474,
+            -0.003520564936698765,
+            -0.004395434431595494,
+            0.0052399043934902014,
+            -0.010127122101423486,
+            0.006456662777583096,
+            0.026560491731327886,
+            -0.013684536319659296,
+            -0.01425196172663986,
+            0.004084982328707292,
+            -0.0010682084059972982,
+            0.009758641913073978,
+            -0.00864879062601467,
+            -0.00481222050200678,
+            -0.014375330954464725,
+            -0.011378890549403524,
+            0.02505641780042516,
+            -0.02861164504722546,
+            0.0034060847953066354,
+            0.014937227531980433,
+            -0.00016904224334342915,
+            0.01541381851648916,
+            0.0033028855218525644,
+            -0.03615935765718852,
+            -0.013677774380513626,
+            -0.02073839691250734,
+            0.019821074228011413,
+            0.01688422012576042,
+            -0.0033731840788147442,
+            -0.013840993586441808,
+            -0.026149880178874887,
+            -0.007430563876740068,
+            -0.02466234818567362,
+            -0.020708282040996014,
+            -0.019276991448222817,
+            0.006137002799644838,
+            0.005162892426268572,
+            0.022881759407670583,
+            0.0025854880108157857,
+            -0.015058889448404231,
+            0.0029948608544278386,
+            -0.006722836436935071,
+            0.011845298212977357,
+            -0.02747944486818378,
+            0.015311272959213262,
+            -0.01610493464983465,
+            0.025239768880017402,
+            0.007425208902253236,
+            0.0010569008485112994,
+            -0.008822681490988575,
+            0.00010704167529046774,
+            0.014883787023445807,
+            -0.004186968199973372,
+            -0.0007781259002896265,
+            0.02787555889813697,
+            0.02087373813833492,
+            -0.0017783425135865238,
+            0.008743718448136432,
+            0.028026471206552342,
+            0.019081083283793984,
+            0.0325261942213039,
+            -0.013815526479760766,
+            0.005113676604826083,
+            -0.028873600114081546,
+            0.010510475843707614,
+            -0.010871039729515013,
+            -0.010201824683340651,
+            -0.0067946599549165735,
+            0.027756887504389347,
+            -0.007962601205446472,
+            -0.0006687976124826922,
+            0.03469894503563926,
+            -0.014185294847328063,
+            -0.00010543064621374546,
+            0.0007386418834840544,
+            -0.009759481309152999,
+            0.02742722730386853,
+            -0.0073590367291510046,
+            0.018456032721630843,
+            -0.019560882612141943,
+            -0.013705917411254969,
+            -0.011894560208199602,
+            -0.001142768182039817,
+            -0.02576754877141608,
+            -0.008755240066826174,
+            -0.02249287216087102,
+            0.00859172262856532,
+            -0.0046012388553094375,
+            0.018845920253446127,
+            0.0011617844393195566,
+            -0.005275799428018491,
+            0.029069367036565145,
+            0.0088080893110627,
+            0.018100411389202663,
+            -0.0004971948011851863,
+            0.01216566778850985,
+            0.02369448391321832,
+            0.002637001729688794,
+            0.005012948614448268,
+            -0.024213704079796478,
+            0.003420184574622472,
+            -0.0004359900462023086,
+            8.265488264269535e-05,
+            0.016267948861008616,
+            -0.019572032453100303,
+            -0.02216264534756073,
+            0.021871880626992544,
+            -0.012742926070754964,
+            0.011676769226089007,
+            0.0035362870996336194,
+            -0.010732229460038356,
+            0.0006838777488770268,
+            -0.03083438636386314,
+            0.008820746379709355,
+            -0.03216685858285627,
+            -0.032230808828686675,
+            -0.013547681710236494,
+            -0.009994106571421729,
+            0.0008758088338659712,
+            0.028447187234357855,
+            -0.00795317029726568,
+            -0.015797892954136448,
+            0.005363565557248085,
+            -0.011199252697624111,
+            -0.00817162221789202,
+            -0.0007867892785855576,
+            0.005856939999700809,
+            0.009461959355310273,
+            0.005997804620690314,
+            0.011648335019189696,
+            0.007491946658046298,
+            -0.008028800679169293,
+            -0.008837725220736191,
+            0.015889923814605387,
+            -0.0014409716087425222,
+            -0.01176424391171685,
+            0.030430130082762363,
+            -0.013300620185884817,
+            -0.0005592926632214984,
+            -0.0021975669766032472,
+            -0.0015687208316663597,
+            -0.0021652076181411244,
+            -0.014225289143358926,
+            0.01870568404189425,
+            8.334501002782455e-05,
+            -0.014288806218067678,
+            0.0032841743905975634,
+            0.003841780428153241,
+            -0.0036498339483411074,
+            0.008388922960269383,
+            -0.008147275785908849,
+            0.017348147498892,
+            -0.015113395971380741,
+            0.008740452410210708,
+            0.0038335161236289234,
+            0.0043482078469644385,
+            -0.0065755348640655865,
+            -0.01592809184384243,
+            0.009642593996701513,
+            0.012051872087663658,
+            0.02173264012652684,
+            -0.02525660485859878,
+            0.00876161653901566,
+            -0.005374533920781607,
+            -0.014534179652013885,
+            -0.016127866637369738,
+            0.00982303248417706,
+            0.022322910435388785,
+            0.0063086346898091296,
+            0.012139263756190026,
+            0.00013383660542132047,
+            0.005848259030010402,
+            -0.014767198851314148,
+            0.010570158368602513,
+            -0.011355774405066537,
+            -0.018439261453084426,
+            -0.005243587266408389,
+            0.028894120979627323,
+            -0.005492024421080143,
+            0.017938747701972086,
+            0.0021350711250093858,
+            0.029007356490801176,
+            -0.008686781541806358,
+            0.0014883465513564527,
+            0.0039866022469891155,
+            -0.01295271525889559,
+            -0.004272008629260297,
+            0.01976273843646234,
+            0.04882868638601926,
+            0.010204475902027572,
+            -0.0009155109545661691,
+            0.010069071220988797,
+            0.005626418787776309,
+            0.002148101019776153,
+            -0.00805781810841286,
+            -0.030431071067211577,
+            0.008883707378543252,
+            0.005628218215212441,
+            -0.01997825125961566,
+            0.019838234890736068,
+            0.026483365072033395,
+            0.023040623182975964,
+            -0.011585643598171132,
+            0.016539930468533123,
+            -0.010092529089936188,
+            0.0177008735686489,
+            0.028665424114962996,
+            0.013810061902648515,
+            -0.010487150104822674,
+            -0.0020847870330548534,
+            0.006329342599564054,
+            0.004186981876004016,
+            0.00026239408223817384,
+            0.015190614519766204,
+            -0.019155180136775043,
+            0.01005728016778505,
+            0.014725826697964637,
+            0.03322436221319799,
+            8.049930133627564e-05,
+            -0.01645191835557271,
+            0.0016249281996128196,
+            -0.009976155662939693,
+            0.00856399492362911,
+            0.000632427127231252,
+            0.0032242260692672124,
+            -0.009688119824276678,
+            -0.012682779503915694,
+            -0.002851918183031152,
+            0.009403205508474241,
+            0.0021188716361101698,
+            0.002759004011231831,
+            0.007641925150177999,
+            0.0022830062031146204,
+            0.01713024474384216,
+            -0.0011181980841546123,
+            -0.00029141889484139217,
+            0.004601090256674874,
+            0.010509199590505558,
+            -0.01649852835819883,
+            -0.017548369964263514,
+            0.004337237475053179,
+            0.012217494210546133,
+            -0.013159179948224596,
+            -0.013241981632059755,
+            0.015704543608959117,
+            -0.01913575130892795,
+            0.024934102868065366,
+            -0.00942381892138331,
+            -0.004543611336526587,
+            0.000343200128934031,
+            -0.00727538880611703,
+            -0.0038751118497522673,
+            -0.0030064279504724797,
+            0.003984715814505561,
+            0.0019261818110367147,
+            0.007472350699291607,
+            0.008472369788801792,
+            0.014568547873264247,
+            0.006043895352236181,
+            0.0032580830408778395,
+            0.009109104498825864,
+            -0.014523846294121576,
+            -0.014800649887278973,
+            -0.002704791598011488,
+            -0.008895394038700809,
+            0.00902016867266151,
+            -0.006607182154874112,
+            0.00845408382182461,
+            -0.015501867835442603,
+            0.015573642225745032,
+            -0.012200552219016517,
+            -0.0008900215712478553,
+            -0.009666233545144905,
+            -2.7470277092642562e-05,
+            0.008191697904514857,
+            0.014605287653054787,
+            0.0070552942843382215,
+            -0.031921405225964206,
+            0.0017403427534633655,
+            -0.008846568352320488,
+            0.03446319360336241,
+            0.002612606662691094,
+            -0.01060863123202008,
+            0.020352181356568008,
+            -0.01592124132155158,
+            -0.00027704668715660693,
+            -0.01602075523652062,
+            -0.004797557413047212,
+            0.00707939446796679,
+            0.016399017978990946,
+            0.02759049512664933,
+            -0.005697020356865688,
+            -0.007995305872076221,
+            0.006384047872862793,
+            0.029590696670334232,
+            0.012246955508890425,
+            0.024748351124415785,
+            -0.007633712004672087,
+            0.014348228716100414,
+            0.007773260552660029,
+            0.0076045488653348975,
+            0.004321689250440021,
+            -0.002591801797145718,
+            -0.011698574606499644,
+            -0.014974971814360001,
+            -0.0037563183330623384,
+            -0.016987915346053498,
+            0.00012247833530362867,
+            0.021344400276501117,
+            -0.00475351423959752,
+            -0.0033847161875730026,
+            -0.01874186437694691,
+            0.0033461670218611346,
+            -0.010029445302549983,
+            -0.00367421246775698,
+            -0.013133469163793961,
+            -0.0254122035745965,
+            -0.018556369080030412,
+            -0.002155244024999317,
+            -0.00929309186646207,
+            -0.015194269781263309,
+            0.0070354690219415535,
+            -0.016305117309271634,
+            0.0080837166473439,
+            -0.00448473417024141,
+            -0.016508775226236883,
+            -0.00958925666995396,
+            0.01100335153658195,
+            -0.0041819993634961685,
+            -0.01929493648733792,
+            -0.01282300765645703,
+            -0.011300260564838231,
+            0.0051733704121938884,
+            0.005996780699975213,
+            -0.0038111648304178544,
+            -0.0028279639348479987,
+            0.016640121845366295,
+            -0.010197365811240557,
+            0.009878730312873492,
+            -0.03335474019383245,
+            -0.03551826040406959,
+            0.004497847402931096,
+            0.01507007013703286,
+            0.0013085881889541252,
+            -0.02078680101065954,
+            0.010806315988966003,
+            -0.016499186313125556,
+            -0.00858013792516617,
+            -0.008522534127498645,
+            0.009649521088411411,
+            -0.007283003380127847,
+            0.01299542385123007,
+            -5.3226845309105124e-05,
+            0.030477683597373687,
+            0.0074153455615142755,
+            0.004161881195438522,
+            -0.00679081048132963,
+            0.008354912600159931,
+            -0.001621776416108105,
+            0.004080191054849395,
+            0.02120524874226696,
+            0.012440848502230418,
+            -0.00218041171518331,
+            0.006458415098595872,
+            0.005678055538564355,
+            -0.017186537377121235,
+            0.02507960402044456,
+            0.024678178106806115,
+            -0.008205288425409454,
+            0.004108402466376321,
+            0.005783335302169668,
+            -0.008846663924040241,
+            -0.006897707231869481,
+            0.006527795262026734,
+            -0.018474181903007085,
+            -0.0008721381678008483,
+            0.010598084358464908,
+            -0.02255441909004843,
+            -0.004840511019512733,
+            0.021745651362109403,
+            0.004631567631335321,
+            -0.018558972627958834,
+            0.02764788406651861,
+            -0.004942361238163609,
+            -0.0027290999934532494,
+            -0.010876135202650048,
+            0.01671497143541799,
+            -0.012173527207375091,
+            0.008443317346655462,
+            0.011021541861818043,
+            -0.006295639322287843,
+            0.003083685676540999,
+            0.008277206401503182,
+            -0.004504302913715969,
+            -0.0068075424927125676,
+            -0.0010508416962215848,
+            0.010167160518175114,
+            0.0036980815941666336,
+            -0.022704963865525812,
+            -0.006020073261002279,
+            0.02934227992246887,
+            -0.015082922782242507,
+            -0.0050916398557337705,
+            0.005376294716056329,
+            0.007709342013963906,
+            -0.003476236561444464,
+            -0.0023399991233097337,
+            -0.00787486645740744,
+            0.010356414979428607,
+            -0.015950083845400274,
+            0.02076360898078935,
+            0.025608263494550438,
+            0.024504236990840842,
+            -0.014814235990238182,
+            0.010828584133720797,
+            -0.011703992394890898,
+            -0.011047470905668532,
+            -0.015542835222173864,
+            -0.001658296303013357,
+            0.001970208852236451,
+            0.013107754130616248,
+            0.009666887546609378,
+            0.0235853683657408,
+            0.006473544089109194,
+            0.008973300312205858,
+            0.014118601565410305,
+            -0.003902202854013708,
+            -0.018303438244719996,
+            -0.014584226585460755,
+            -0.011977613285028946,
+            -0.01756345491020528,
+            0.0004563602312336091,
+            0.020461659996271687,
+            0.012417744207683839,
+            0.024136626592375227,
+            0.002119213701333967,
+            0.013759902247467922,
+            -2.9525392347047554e-05,
+            -0.000892319792935443,
+            0.00045783825646242434,
+            -0.005238483120312843,
+            0.019255756091120846,
+            -0.004599752813817219,
+            0.01656589522110913,
+            0.0006443773300684925,
+            0.007133940816771615,
+            0.013554761340256566,
+            -0.01704388669741371,
+            -0.003443100892108368,
+            -0.010029620309829285,
+            0.000941391966877521,
+            -0.019633111750884034,
+            0.01934571074127927,
+            0.010273330977233852,
+            0.0017121345993533963,
+            -0.018327189396014198,
+            -0.0029706385099949693,
+            0.03448635131998772,
+            0.019458999453011265,
+            0.0050734354363614525,
+            0.02632443373898792,
+            0.013018762352749236,
+            0.014443611411502168,
+            0.010153370950346075,
+            0.02356766253713842,
+            -0.023479811300626968,
+            0.02095255383200973,
+            -0.011590347850340786,
+            -0.007414136601134069,
+            -0.0035774888741646705,
+            0.0025141508183687115,
+            -0.013588691733097813,
+            0.00982877622088977,
+            -0.0017916539399655826,
+            -0.025386211797192578,
+            0.009164316443206016,
+            -0.008811052979979271,
+            0.010464442616928296,
+            -0.007774569252594923,
+            -0.0002806136360286729,
+            -0.0127084026190163,
+            -0.015184317235990752,
+            0.001259201880504347,
+            -0.009158429132281081,
+            0.023908150915412617,
+            -0.021210848953875233,
+            -0.01321384014436847,
+            0.013997502713857512,
+            -0.006869443694669701,
+            -0.011364239609894453,
+            -0.00374449980963666,
+            0.023673050288879224,
+            -0.009243341446512421,
+            -0.01359367519009039,
+            0.014464166953530989,
+            0.003834735113400469,
+            0.0059713317014342395,
+            -0.007508862694272452,
+            -0.008428705474135382,
+            -0.003280788036976103,
+            0.005998840043285335,
+            0.026615077617284676,
+            -0.014059930972065399,
+            -0.0032833264512892022,
+            -0.009189999799498163,
+            0.004118204537632102,
+            -0.02359889635281631,
+            0.02771075351384059,
+            -0.008871395303200375,
+            0.021705177758560575,
+            -0.0015924449308442096,
+            -0.006239188158271628,
+            -0.024231036074349822,
+            0.027522945599928003,
+            0.003683870966132199,
+            -0.010221180899173243,
+            0.0012768725460662862,
+            -0.052005292736885915,
+            0.016636109796947025,
+            0.008429895913163985,
+            -0.019507994429834353,
+            0.003697519981610208,
+            -0.004773584856005853,
+            0.018983847580558853,
+            -0.022966993228850697,
+            -0.014078986007109447,
+            0.008283766047449966,
+            -0.01153285124362746,
+            0.0031439180465649996,
+            -0.02527749501821793,
+            -0.005754537200570427,
+            0.0011760345167229269,
+            -0.005276936074280004,
+            -0.001705933750221904,
+            -0.020022761608132548,
+            -0.025972227850795115,
+            0.017666017379157135,
+            0.015820875579600418,
+            -0.002278262448951789,
+            0.01320818011561616,
+            0.008709899932124187,
+            0.00836573261091119,
+            0.016034263394940008,
+            0.007124509234103267,
+            0.0081473711608784,
+            0.045498533239850446,
+            -0.019846323044145485,
+            0.007371629414298193,
+            -0.0014854149174929534,
+            0.013951741898629716,
+            -0.030467924817555014,
+            -0.0055144480998344705,
+            0.0062802744565489945,
+            -0.0049373689340857915,
+            0.03510454786147091,
+            -0.017326094216003073,
+            0.01694160526735264,
+            -0.0026075809870445238,
+            0.0007420394265014909,
+            -0.02042003877666893,
+            -0.026156937726134134,
+            -0.014294938857971834,
+            0.0011857406889443987,
+            -0.0010047941261608975,
+            -0.011946583993069242,
+            0.006797001710574735,
+            0.004917161312925866,
+            -0.0020713379451245515,
+            0.015652523103715316,
+            -0.02952061566240878,
+            0.007356800041369564,
+            -0.010733578620243498,
+            0.020008504830359524,
+            -0.00990331048284697,
+            -0.0013319400098308093,
+            0.007236960885181047,
+            -0.004122868383988748,
+            0.013216043472071927,
+            -0.0038337251451538677,
+            0.004227115653354611,
+            -0.005221321656766545,
+            -0.01794343569722877,
+            -0.0016435261495179767,
+            0.021429008432348098,
+            -0.008349684096540855,
+            0.031864497493526205,
+            -0.01264743878037828,
+            0.009174982784686633,
+            0.010692990337462762,
+            0.0006628831036334152,
+            -0.005286285851702483,
+            0.009052971814211658,
+            0.0021431970274545893,
+            -0.008690520448278274,
+            0.006934310986093342,
+            -0.018349449937310143,
+            0.008494451785507706,
+            0.03394632739506833,
+            0.0018511328780472266,
+            -0.0007519661670070526,
+            -0.007266663179613392,
+            -0.01335844201823745,
+            0.018540713029383066,
+            0.005488273906733518,
+            -0.002546531152114063,
+            -0.027536660869590543,
+            0.02007746621157705,
+            0.0012265798861291223,
+            -0.004755325069578531,
+            0.012466557901411086,
+            0.0020495867632654885,
+            0.019989531151046736,
+            -0.010171678639113316,
+            0.030757957899447096,
+            0.010050036220124393,
+            0.002959833051227505,
+            0.0038556928367700113,
+            -0.036655749074851536,
+            -0.01652152778069314,
+            -0.00019357502472740266,
+            0.014271581054865031,
+            -0.012430647252826173,
+            -0.022432580814844155,
+            0.0010346169306977334,
+            0.01148004051449691,
+            -0.0033898642762881046,
+            -0.009650665305271706,
+            0.01560757541276278,
+            -0.020342213902670575,
+            0.006701302394408393,
+            0.007693672045981921,
+            0.0287100214318104,
+            -0.0020735384554022153,
+            -0.0012324838849990056,
+            0.000626416803904792,
+            -0.0055669860801192935,
+            -0.0008037366126870824,
+            0.017251369734470945,
+            -0.013661316947002466,
+            0.0175926211217534,
+            -0.012735241490550407,
+            0.0064713656749838585,
+            -0.0021092378432037563,
+            -0.010629579860050812,
+            -0.020114807971202787,
+            0.0029609495764307357,
+            0.009378003163375899,
+            0.006920144698428806,
+            -0.010438984881449953,
+            0.010715409812532968,
+            0.012197673689706352,
+            -0.00753962861484439,
+            0.014519905132928041,
+            0.022173989895384192,
+            -0.007687503607783572,
+            0.013633321414009471,
+            -0.0041446334728745095,
+            -0.014056264816568229,
+            -0.009985463824440833,
+            0.023929750617160402,
+            -0.0005563806942667597,
+            0.008756997373618809,
+            0.00388125051641897,
+            0.0007856447663040696,
+            -0.011226530517316183,
+            -0.001653447552601681,
+            0.019294448360333633,
+            0.011831934675073843,
+            -0.0013888534822008192,
+            -0.0010525764356699885,
+            -0.0014566936525653407,
+            -0.009427557739483988,
+            0.02049188741459728,
+            -0.022030783927622997,
+            -0.0019604940727076573,
+            0.00461018148439991,
+            -0.022589816803875618,
+            0.0001863117949313752,
+            -0.024570217962989905,
+            -0.01569952241187498,
+            0.011763033781122343,
+            -0.0022530631823737554,
+            -0.016028810867854907,
+            -0.0003242701040942633,
+            7.424127301806556e-05,
+            -0.007815789940508527,
+            -0.004135500768181258,
+            0.006115909992703774,
+            -0.00034193217510010117,
+            -0.0032262189374874842,
+            0.013430154310506884,
+            -0.009996622196870008,
+            0.015431528911575157,
+            0.0067029980830614145,
+            -0.015702167721800483,
+            0.0021406610839557003,
+            0.018500219544264753,
+            -0.0058910087864916085,
+            0.00787021425357441,
+            -0.014681907249972176,
+            -0.0037495551894426303,
+            -0.005787661351944086,
+            0.006051351977821761,
+            0.024849195780885833,
+            0.012969434450643372,
+            0.01139340445077682,
+            0.017405361843075933,
+            0.004325000715054734,
+            -0.01572297405297993,
+            0.0007416798945675923,
+            0.01682727189828363,
+            0.010464036837245312,
+            0.009804326468113703,
+            0.0008975556667684365,
+            0.0043704860361487635,
+            -0.005142176468800175,
+            0.009317064072855551,
+            0.02915482449687158,
+            0.0013434689612997423,
+            0.008696141010760192,
+            0.017276730345147134,
+            0.027038741396131265,
+            -0.0035088997194497725,
+            -0.007469614319788499,
+            0.008739942310300076,
+            -0.009657991604173173,
+            -0.001081629174976152,
+            -0.017000351183542026,
+            -0.012310275736750868,
+            0.016510505259358398,
+            -0.006084627062230492,
+            0.02848913486719956,
+            -0.0010655806791907225,
+            -0.02026498848361482,
+            -0.01565505608194376,
+            0.001404549479440637,
+            -0.01362132895080652,
+            0.005287566256051299,
+            0.014187265685427415,
+            0.01917046513020831,
+            0.004687191682875024,
+            -0.022258337721085656,
+            0.037364962197399806,
+            0.01757108963349427,
+            0.019377870795760123,
+            -0.019422840506328382,
+            -0.006159311237412833,
+            -0.009077095160676276,
+            0.02116697012944498,
+            -0.00880692095488465,
+            -0.009875674856043954,
+            0.024014332004451508,
+            0.017812606845656484,
+            0.005679806286464216,
+            0.006736609904079586,
+            0.0020048663439319585,
+            0.0025414630253661444,
+            -0.015831593851012047,
+            0.017957902498542212,
+            -0.005858125698347523,
+            -0.0058068554681761945,
+            0.017529514573571298,
+            -0.005989581792993925,
+            -0.0030306021203732347,
+            0.00015839997806822045,
+            -0.019697052145531482,
+            0.011618112886207201,
+            0.008256007601576028,
+            -0.017229774429089857,
+            -0.0014757560402567304,
+            -0.0002404480326945295,
+            -0.014118694740134205,
+            0.02004885146995733,
+            0.01694893356262835,
+            -0.0013248884139405398,
+            -0.0051752815372013315,
+            0.008223868147393114,
+            0.007000679132597393,
+            0.005256194890343006,
+            -0.019496098876977206,
+            0.005230597149342019,
+            0.005103909294824105,
+            -0.0014716649591412774,
+            0.010274346323051634,
+            -0.009030594825985526,
+            0.033740380586893826,
+            -0.006505788197604522,
+            0.013065658941136721,
+            -0.0043634731684193745,
+            -0.0055529072007985125,
+            0.001816020676232658,
+            0.013054468478598695,
+            0.009753501030700457,
+            -0.00970446155762939,
+            -0.004172354962129865,
+            0.018697982092775737,
+            -0.011983156146516074,
+            0.0013300791598046338,
+            0.01564072865804035,
+            -0.004858814041438713,
+            0.004708235083775013,
+            0.008635204790345423,
+            -0.004489505237783172,
+            0.01801512858970922,
+            -0.016482572999490366,
+            0.0023953436660054026,
+            0.010694869684962088,
+            0.02390864159830096,
+            -0.019152187046410703,
+            0.021786596486708207,
+            -0.017951050280323393,
+            0.026538582634436314,
+            0.014484894489694262,
+            -0.0035222259457311444,
+            -0.01946200885147006,
+            -0.00850690913394181,
+            0.011654565129063417,
+            -0.021423803876241532,
+            -0.0007985948078290641,
+            -0.0007290222944686444,
+            -0.021367628543092133,
+            -0.011189364017231132,
+            -0.009961779129778331,
+            0.010701637745810138,
+            0.001918633898887013,
+            0.010380983868185359,
+            0.008002182827196827,
+            0.010784760574518087,
+            0.014405640442760822,
+            0.0008958107769384412,
+            -0.010743001445440941,
+            -0.023795589003655273,
+            0.016987708698679597,
+            0.00438463274382093,
+            0.0007862212695139549,
+            -0.01627468994515667,
+            0.011936651269737248,
+            0.013250643815768723,
+            -0.009385605548926286,
+            0.009121882469425302,
+            -0.03716901103204463,
+            -0.004552808410398574,
+            -0.0035704811925430366,
+            -0.0020309197365173475,
+            -0.00908678158053104,
+            0.01559445415882485,
+            -0.013682507836005851,
+            -0.0009590922904844632,
+            0.00011696985057536548,
+            -0.0074756526903548915,
+            -1.5467890432254378e-05,
+            0.020649928837031005,
+            0.008438785939908428,
+            -0.0037382105257999757,
+            0.010376213224654074,
+            0.008826031303751312,
+            -0.013900359449287676,
+            -0.01527474712786084,
+            0.019035366035884273,
+            -0.018776918681908793,
+            0.0063098418544296165,
+            0.01677916431403883,
+            0.020778813096283988,
+            -0.02891862501830075,
+            -0.0007231469961304435,
+            0.022093939332021185,
+            -0.0012807635799950273,
+            0.006370841109790851,
+            0.008781853138377092,
+            0.006754082980602544,
+            0.0029102159062355374,
+            -0.003711173531879644,
+            -4.843770941622283e-06,
+            -0.003940474588075707,
+            0.007155539022862352,
+            0.004366433868349658,
+            0.036453728071874844,
+            -0.013661129348165794,
+            0.002541308805735001,
+            -0.0058838191638835155,
+            -0.00347401304572066,
+            -0.00593469688967835,
+            0.004407665856950772,
+            -0.010055727994798024,
+            0.004330827357536078,
+            -0.001570861535288093,
+            -0.0003731129422346693,
+            0.004883969598300302,
+            -0.001701673379646408,
+            0.007036892564844683,
+            -0.011863650867472625,
+            -0.0003979727750011258,
+            -0.020174851412609312,
+            0.021117284557875267,
+            0.00011666551282792189,
+            0.003662462447326642,
+            0.03723741246290771,
+            0.00476064510662738,
+            -0.00238355131307445,
+            -0.012880057949435798,
+            -0.013551338766321525,
+            -0.012986444917747435,
+            -0.020703821629987552,
+            -0.0036665673548525388,
+            -0.021006311026247164,
+            -0.0052061406234528915,
+            0.010362862594785442,
+            -0.011540184420787572,
+            -0.004023659825622843,
+            -0.015881589631889473,
+            -0.006531981517815174,
+            -0.03114211633687585,
+            -0.01124607488650473,
+            0.007493228366452009,
+            -0.019149285949049886,
+            -0.0039932870926865805,
+            0.026850121590606017,
+            0.001095360635760988,
+            0.01387696424607248,
+            0.0038058248676879625,
+            0.008117686785954085,
+            -0.0045937472017383955,
+            -0.0005669025036141417,
+            -0.01928244877716143,
+            0.003391980330529026,
+            0.008525354854576925,
+            -0.00361507909530282,
+            0.0070008404454106905,
+            -0.008835727374865623,
+            -0.016950017602980382,
+            0.0021654956715219517,
+            0.009334379529385142,
+            -0.006639140003365139,
+            -0.017587472740268683,
+            -0.011049929670847685,
+            0.010401189523318744,
+            -0.02283429470064187,
+            0.006257145380266984,
+            0.016439581169238607,
+            -0.009363030375624585,
+            0.004651575425738153,
+            0.019809183749755425,
+            -0.011810674576781702,
+            -0.005811328501528397,
+            0.008471992070998505,
+            0.013891259999721588,
+            0.0095606134736748,
+            0.02954498207274139,
+            0.0026596809804008996,
+            0.000561870602466584,
+            0.004889263753490895,
+            -0.004277061142504552,
+            -0.017423200644235665,
+            -0.006243787897080996,
+            0.0038507494787521856,
+            0.012873996671952184,
+            -0.009879417960879032,
+            -0.00048575277472139055,
+            0.01682604801673206,
+            0.004530939990187663,
+            -0.0033901248079550284,
+            -0.011868399508738274,
+            -0.020080331108085103,
+            0.0015580259874809898,
+            -0.01362200460317948,
+            0.015354347040143093,
+            0.003390601069331352,
+            0.0015284610287850575,
+            -0.004239880336806429,
+            0.01828645373058079,
+            -0.005013418527935815,
+            0.012749833447770729,
+            -0.006148890994661865,
+            -0.004587831231448011,
+            0.007986533783001116,
+            0.009166578151006778,
+            0.0004939857995863664,
+            0.05421019830078428,
+            0.002838056604877719,
+            0.01333205648597435,
+            0.013419127695838135,
+            -0.014523733369484934,
+            0.014086525147734823,
+            0.012688569853552613,
+            0.018497344547821693,
+            0.013037819487160016,
+            -0.012353776077273462,
+            0.017354801119939964,
+            0.0024084723134891487,
+            0.003909681065813095,
+            0.0022202438064449653,
+            -0.004528576079628071,
+            0.019594683950225115,
+            0.004070222328547378,
+            -0.0074362443729372105,
+            0.004453618452623641,
+            0.005007543168224267,
+            -0.013105524222311019,
+            -0.0112958915514782,
+            0.027105208848863218,
+            -0.01219417557128091,
+            -0.018749971743148304,
+            0.006706594269933445,
+            -0.02008458683853808,
+            -0.03763686535076864,
+            -0.0021332074289244973,
+            0.009992361098878196,
+            0.002795183524495193,
+            -0.00253695604043486,
+            0.01614298064601032,
+            0.016730443047719683,
+            -0.016985296937992726,
+            -0.018245902728910347,
+            0.0025257780765481825,
+            0.0016333325293570182,
+            0.01861807236979066,
+            -0.023168663546030196,
+            -0.006378864850313035,
+            -1.3795465857380962e-05,
+            -0.006829008132963306,
+            0.004850306884033312,
+            0.004703280112191218,
+            -0.0071714759384722965,
+            -0.0027773519096287217,
+            0.002256846645341137,
+            0.01975383140755998,
+            0.013198754584066554,
+            0.01572971337890945,
+            0.002600741959577911,
+            -0.005033508062449482,
+            -0.016817359058724685,
+            -0.005450011991586895,
+            -0.013478330855299777,
+            -0.009091854345715177,
+            -0.003042052092498012,
+            0.008217376329142566,
+            -0.025053469059115735,
+            -0.011965083373220982,
+            0.0119294363235832,
+            0.00391377331459168,
+            0.0003099203772539202,
+            0.007607672728136279,
+            -0.003830132495364688,
+            0.010331053246451333,
+            0.013823890473448383,
+            0.0076112887952959766,
+            -0.004723377809559611,
+            0.009600411526173948,
+            0.011460781436250457,
+            0.006015477068316837,
+            -0.005072711638895421,
+            0.023859899048048694,
+            0.01821920335673539,
+            -0.01680365156728093,
+            -0.002149107383573353,
+            -0.0027829676356423997,
+            -0.01884308738652242,
+            0.009665830801645276,
+            0.00880322398336952,
+            0.008308268605449346,
+            0.003906369351868086,
+            0.00974531700889854,
+            -0.013875904819275925,
+            -0.0050570811397453235,
+            0.002870120865445989,
+            -0.005729556771404781,
+            -0.01901337075686543,
+            0.000297242372701423,
+            -0.011987192261104132,
+            -0.011327157550660895,
+            -0.01409710922165158,
+            -0.003221240464411861,
+            0.005283472511900512,
+            0.021472187640986972,
+            -0.004410881709548286,
+            -0.001855852749039012,
+            0.01351697599517034,
+            0.010060125186506398,
+            0.0021658738823899905,
+            0.0002206474541239167,
+            0.005418828324605768,
+            -0.0027919130944783397,
+            0.0008094042430809068,
+            -0.0015826797905407809,
+            0.021089276765434876,
+            -0.02016611086095107,
+            -0.009413977569213423,
+            -0.009288078098692112,
+            0.005842197094339927,
+            0.028793032462603636,
+            -0.007165634502373454,
+            -0.010104002083017078,
+            -0.01994315293556678,
+            0.0003786527922888834,
+            -0.07810609175343784,
+            -0.005210636756258548,
+            0.001870513618910238,
+            0.03126380113105983,
+            -0.010817888476436834,
+            0.0033673768058629823,
+            -0.002204487854177595,
+            0.014542299655712928,
+            -0.001760107965206874,
+            -0.0013283746982194259,
+            0.0071954806737858056,
+            -0.005640400818010697,
+            -0.006210989097047217,
+            0.005904163661619133,
+            -9.945941028848348e-05,
+            -0.010266938705916303,
+            0.00057263080955709,
+            0.02209498395848068,
+            0.0159398346608872,
+            -0.006575439759804175,
+            0.01816748552021211,
+            -0.00719294553484007,
+            -0.009838738592324234,
+            0.005639288780502127,
+            -0.0019007683459126327,
+            -0.015369619176147573,
+            -0.011475977270803784,
+            0.009862808084641291,
+            -0.030069890644150284,
+            -0.00010725753918890802,
+            0.00943698424646706,
+            -0.013540245857270492,
+            -0.008293042205008774,
+            0.007212974960781723,
+            0.006708343474532583,
+            -0.0012017968151025369,
+            -0.0078026000239507145,
+            -0.02527526731750743,
+            0.0027578400032925625,
+            -0.002446811971857853,
+            -0.004899702349784173,
+            0.0019960465173086183,
+            0.001479503476855694,
+            -0.012645905858466798,
+            0.01216748847452763,
+            -0.013378877368040743,
+            -0.012053251571418766,
+            0.012508842412556306,
+            -0.00027167241745418584,
+            0.01682680930386839,
+            -0.020607299598265426,
+            0.0010702151063620052,
+            -0.028075439824932304,
+            -0.007671848153932823,
+            -0.015486630894165054,
+            -0.0009776136330459065,
+            -0.0012939921569540803,
+            -0.03481880531471923,
+            0.012889447665794033,
+            -0.005200149829198305,
+            -0.006094702500154477,
+            -0.00459778303079732,
+            -0.024583504817899775,
+            0.006072281238752663,
+            0.00607350167867964,
+            -0.006506508970816635,
+            -0.016816439635445445,
+            0.008428048142588516,
+            -0.0011040593013238872,
+            0.004873670535200424,
+            -0.007475800310154908,
+            -0.0005638920132076605,
+            -0.013047135093134946,
+            -0.005463454269764494,
+            -0.0008522562324122834,
+            0.01827123304871433,
+            0.0044055851550317835,
+            -0.014526381222031981,
+            0.016862445236481526,
+            0.013250692156820124,
+            0.01602714293656341,
+            -0.0071089293427753115,
+            -0.0004265739315250755,
+            -0.02750630108787212,
+            -0.00327225092954903,
+            -0.023687644154808972,
+            0.00041213204365507744,
+            0.009447660230507965,
+            0.0203905264889617,
+            0.012694421477479161,
+            0.004620617778996077,
+            -0.001557511764988156,
+            0.0066751090724502,
+            0.02696959560423252,
+            -0.01099183126742224,
+            -0.006969111598423376,
+            -0.01884504192789845,
+            0.01113023717919988,
+            0.025135748541199255,
+            0.003381303139273845,
+            -0.011317957771935042,
+            0.008112598754655146,
+            -0.0009247845526290428,
+            -0.010302980598863016,
+            0.02889587382004902,
+            -0.0391134481322538,
+            -0.015599591108049343,
+            -0.004605824192962023,
+            0.007406520608110251,
+            -0.015596917115435033,
+            0.00770468362805995,
+            -0.012614339656105264,
+            0.001699049329817941,
+            -0.005851477164035241,
+            0.03535087340659861,
+            -0.0031950960727572925,
+            0.0006909856233673865,
+            -0.003395233050301427,
+            0.003531135044131077,
+            0.012355948349937309,
+            -0.004444221317023453,
+            -0.030726300006909638,
+            -0.0022983766869702346,
+            -0.011198252640110788,
+            0.009274666307515955,
+            -0.016207714291658314,
+            -0.017342435976846468,
+            -0.009424737744066259,
+            -0.021827243671837953,
+            -0.012403084982789827,
+            0.029747453318486338,
+            -0.005824001042844527,
+            0.01747569398142269,
+            -0.005219060548639168,
+            -0.008365003057800934,
+            0.00021272411948388247,
+            0.01344335658390732,
+            -0.028498240040284714,
+            0.00041963499785238845,
+            -0.002322931702641349,
+            0.0126393550661767,
+            -0.001595134597082556,
+            -0.00983073906977773,
+            0.006903313245694627,
+            0.0010522208685011098,
+            -0.015648940549713764,
+            -0.0007573245570947632,
+            -0.008533504878052644,
+            0.020475329445149498,
+            0.00027791387213409814,
+            0.001074213629646005,
+            0.013864331671732542,
+            -0.02102352993295736,
+            0.01576938071930521,
+            -0.021722775924283996,
+            0.01893267442161547,
+            -0.006577103101038035,
+            0.005772016664149475,
+            0.008283223343661555,
+            0.014068789332250016,
+            -0.014030042849491258,
+            -0.00663752723080222,
+            -0.034436220972634726,
+            -0.0122621879114314,
+            -0.001946072976826902,
+            -0.005308941654794638,
+            -0.014773123927909993,
+            -0.0011054822266082056,
+            -0.026205461509573584,
+            -0.006460373065662098,
+            0.012822188046808264,
+            0.019900312098359022,
+            -0.008409070094265829,
+            0.014163132434889265,
+            -0.006743711872055091,
+            -0.0028230771810882492,
+            -0.012652021749331811,
+            -0.018061180380009395,
+            -0.01083672324723357,
+            -0.01890966636827258,
+            -0.0018635408070944168,
+            0.004709222372611061,
+            -0.015760618199235518,
+            0.011435131737642627,
+            0.03198874888333405,
+            -0.01567271472650171,
+            0.008243406188694909,
+            0.016411496328333205,
+            0.0034236971031973503,
+            0.011728079139322837,
+            0.0038311415233052795,
+            0.004621789116267879,
+            -0.011109450058055078,
+            -0.011473158133621661,
+            0.006524490579223986,
+            0.00954511918454185,
+            -0.004913142601134428,
+            0.024347072255007807,
+            0.0008745419206472714,
+            0.005045794664611216,
+            0.005304506902446091,
+            0.011488187043525824,
+            -0.013135271908307747,
+            0.019746484188715552,
+            -0.002055210086078267,
+            -0.015571665869889405,
+            -0.01535499169234956,
+            -0.01105419264826655,
+            -0.018626399757822328,
+            0.0017108083455869296,
+            0.0181439079855921,
+            -0.0012493542718277685,
+            0.012137753522284387,
+            -0.00924460959150499,
+            -0.005019437549649743,
+            -0.0027156377206998474,
+            -0.038211954658821295,
+            -0.0021878970332906457,
+            -0.018809487178512628,
+            -0.02498945889484935,
+            0.03505216568400238,
+            -4.26477081213836e-05,
+            0.00997643255528116,
+            -0.010310197159232386,
+            -0.0002454902098986129,
+            -0.011065403358802515,
+            0.014954998647970473,
+            0.004624123403100598,
+            0.008872923678983478,
+            0.008560280683876702,
+            -0.016419500515801093,
+            -0.002997557798375716,
+            0.007472390947008284,
+            -0.00374822278027057,
+            0.012680407710694952,
+            0.015353684671120731,
+            0.020886474047003733,
+            0.011161262397502876,
+            0.002045862158732102,
+            0.00887271411363814,
+            0.009563361150297699,
+            -0.0003513237086717377,
+            -0.010459621030424415,
+            0.015419039131459582,
+            -0.02204848739950703,
+            0.012172257374410352,
+            0.04346634265794738,
+            0.019197433209794286,
+            0.029488039318675732,
+            -0.02272810439246353,
+            0.01448092574459596,
+            -0.003801043409441206,
+            0.009476229966122152,
+            0.0049309968883087685,
+            0.02346191840355209,
+            0.003973840862678076,
+            0.0073546443568171405,
+            0.010647581332452525,
+            -0.00972051412996416,
+            0.025644022498404723,
+            -0.00846669916801108,
+            5.1518540791212095e-05,
+            -0.004407265804145216,
+            -0.026400029105519137,
+            0.01615193008014211,
+            0.0081722536401398,
+            0.0010702314763197463,
+            0.0010263623479474385,
+            -0.0019566604229607643,
+            0.007982949510899019,
+            0.007079322685771456,
+            -0.015274148711633536,
+            -0.003886053737590683,
+            -0.018527455865296014,
+            0.0012864569064749438,
+            -0.024337133883771334,
+            0.010466651250597847,
+            -0.017568706814132484,
+            -0.014822354159709854,
+            -0.021666681078831483,
+            -0.013232016867264097,
+            -0.02011653224168991,
+            0.0035858355517491515,
+            -0.008632755352062642,
+            0.013025728891203198,
+            -0.016962939287139683,
+            -0.021494971929779273,
+            0.008430887589398292,
+            0.00870802359711456,
+            0.007481282677955534,
+            0.009728368603458082,
+            0.014996157358749115,
+            0.01736292711410622,
+            -0.022712386143092265,
+            -0.023163001611599246,
+            -0.006682209717364464,
+            -0.003531495447167166,
+            0.006762379833125297,
+            0.00933681196321128,
+            0.012884185500343249,
+            -0.007077895849501418,
+            -0.007554762054641417,
+            -0.013283067570021503,
+            0.006197966297581871,
+            -0.003993067576677973,
+            -0.009464789349849612,
+            -0.0015662739825834614,
+            -0.0061389795593180686,
+            0.002753664732282926,
+            -0.0001791088888074743,
+            0.0007595261237141144,
+            0.017122834676182343,
+            -0.008176319370751018,
+            -0.009787747261447871,
+            -0.02027228453943018,
+            -0.010683273300243025,
+            -0.015604745521156111,
+            -0.0016279406232527138,
+            -0.006506713191861036,
+            -0.014420323628589382,
+            0.010888206569219648,
+            -0.011930708571093279,
+            -0.00887210117616784,
+            0.010807147705360856,
+            0.0073900910661282935,
+            -0.003431679006289002,
+            0.037804020313161775,
+            0.0011945597874820288,
+            0.00778778394048267,
+            -0.003366543306746973,
+            0.005296727102519836,
+            -0.017761251541598876,
+            -0.026382400396890007,
+            0.0010357716464853195,
+            -0.01775777668381489,
+            0.010807713750049491,
+            -0.008515368407459745,
+            0.0016051389544388124,
+            -0.00034958600424912995,
+            -0.009585854918521076,
+            0.0036549720706351473,
+            -0.008110927907950148,
+            0.004212219431447508,
+            -0.03507890639477636,
+            0.011409377585617926,
+            -0.01819120811524444,
+            -0.008962242411035718,
+            0.01710177594046791,
+            0.037560936862121835,
+            0.0028513388998398104,
+            0.00030058353811717895,
+            0.0014792184333546447,
+            0.004427921639326168,
+            0.021438127785303025,
+            0.006339483221420334,
+            -0.001187747087595264,
+            -0.02237982599829809,
+            0.0062435650294483695,
+            0.002200578793697352,
+            0.010092828474934744,
+            0.023105023225115266,
+            -0.016797326021070008,
+            -0.015378520287395565,
+            -0.012732527281000097,
+            -0.010797463778182385,
+            0.009881499418260473,
+            0.02801767062785128,
+            0.01811139137849365,
+            -0.017358098300028903,
+            -0.020207173948076473,
+            0.0016242901745373126,
+            -0.026331540602868762,
+            0.007269751661479859,
+            0.11940631114242901,
+            -0.0045382135734049305,
+            -0.005994407567723939,
+            -0.008679712231646732,
+            -0.004482553208064535,
+            0.016709571294385293,
+            -0.013387043582971358,
+            0.069179848974036,
+            -0.007221545718220735,
+            0.0037392229364642915,
+            0.0028916224964057385,
+            0.00462729158956839,
+            -0.009727339449635306,
+            0.011570145487622916,
+            -0.011287365791681587,
+            -0.012991994493845187,
+            -0.013076194601468383,
+            -0.0180830261130804,
+            -0.007272768410218907,
+            -0.004155564814758654,
+            0.004204993170487392,
+            -0.005525575983182816,
+            0.008051108079520902,
+            -0.017551645696802648,
+            -0.010800070994485792,
+            0.013007176804761548,
+            -0.0023627485852619817,
+            0.005390986239687555,
+            -0.029668390278675554,
+            -0.0007650391259819867,
+            -0.0002446394046947245,
+            0.021280292985495608,
+            0.003137771886236958,
+            -0.0019077298815862946,
+            -0.010810159675564049,
+            0.007040107942856957,
+            0.023835993694726953,
+            0.021875366566457236,
+            0.0180511566130312,
+            0.012135351541044354,
+            0.020483975362807574,
+            -0.008667079096167644,
+            0.026324312888512866,
+            -0.001552207146726937,
+            -0.00397810465151493,
+            0.0077577136352257004,
+            -0.009849600842081875,
+            -0.00816392268697345,
+            0.00677353736462267,
+            -0.017288900282504148,
+            -0.020418071729964713,
+            -0.020383762873109595,
+            0.025741408097358233,
+            -0.011755094311186617,
+            0.002737129201159793,
+            -0.008266210469668866,
+            0.027287436144516414,
+            -0.006283892307996516,
+            -0.0029440642954701632,
+            0.009428876742171362,
+            -0.03788574758692254,
+            0.008834605607057885,
+            0.006463342392514337,
+            0.026711196829464153,
+            -3.9827332346130654e-05,
+            0.005565561841469247,
+            -0.0006738085601962865,
+            -0.006104320995853506,
+            -0.010792830169647851,
+            2.4565453208679464e-05,
+            0.017983908748172493,
+            0.0035181716362611077,
+            0.016970206528324653,
+            -0.005449808667944701,
+            -0.023171388778833197,
+            0.007092509502738752,
+            -0.01555320330471131,
+            -0.023306285898839978,
+            -0.002028783456704567,
+            -0.03196436161255752,
+            0.0033412618908711163,
+            0.007527633574188688,
+            -0.01890570959143866,
+            -0.017859735533232564,
+            0.010329340599904344,
+            0.004382396539605985,
+            -0.0016746397858953155,
+            0.006643785064338525,
+            -0.03307812909454471,
+            -0.01927475592253288,
+            0.004686970010943116,
+            0.012733525168279355,
+            -0.012993441126970916,
+            0.015291832952199082,
+            -0.020491167895268936,
+            0.016451976170508816,
+            -0.012259387189535237,
+            0.018393274368854132,
+            -0.006997668664226754,
+            0.011843450716200625,
+            0.011457415642860215,
+            0.009560877634897626,
+            -0.01790001115302499,
+            0.0214879415944361,
+            0.0066829070506954115,
+            0.03900547021123392,
+            0.02512484499576633,
+            -0.012651294056575184,
+            -0.0018425680226742782,
+            -0.00011361458320146007,
+            0.014982647266052349,
+            0.0061874424115559,
+            -0.003689464024837091,
+            0.02705902205734107,
+            0.018239469373522537,
+            -0.021070704644859155,
+            0.020037508334843183,
+            -0.00973804771680803,
+            0.018876283979591423,
+            -0.013303080480396818,
+            0.010592424494628194,
+            0.004717870260467274,
+            0.01070131238869639,
+            0.009087656015197565,
+            0.006708673669681232,
+            0.025928065153829043,
+            0.03663367449576964,
+            -0.00511073108390551,
+            0.002161381031392467,
+            0.004864092591944181,
+            0.012814518637357098,
+            0.001556833001747549,
+            0.007915766323215207,
+            -0.007012785514562176,
+            0.005834786334750152,
+            -0.002311533032619821,
+            -0.00122162928193159,
+            0.019525999280692766,
+            0.000824163362963678,
+            0.012875575199417254,
+            0.013259376059185821,
+            -0.002390133687709065,
+            -0.005682644446302278,
+            0.0019262218417782657,
+            -0.018193478045923098,
+            -0.015131696847666434,
+            0.029463723011600933,
+            -0.011447039651934148,
+            -0.008083114389964213,
+            0.0020642857121204753,
+            0.012160202510990566,
+            -0.00866944494225225,
+            -0.005378300635997713,
+            0.0038280364584418335,
+            0.014427318067525318,
+            -0.01799358399064332,
+            0.003381165206832161,
+            0.005252552821854107,
+            0.019433554824232185,
+            -0.00028791985674147885,
+            -0.013487678738391956,
+            0.0005317922815671783,
+            0.014982152801567637,
+            0.011902348165369078,
+            0.0016056390299304837,
+            -0.006935777197137747,
+            -0.00446584737370504,
+            0.004586814866182051,
+            0.006269466956110759,
+            0.0027516492952993445,
+            0.008157662045793032,
+            -0.0025001885993292637,
+            -0.016648747436203294,
+            0.017735757469839637,
+            -0.0030669873770852725,
+            0.018058248627079793,
+            -0.0038238312452176745,
+            -0.0147172470474118,
+            0.0043328417868095355,
+            -0.001888507803054499,
+            -0.018556943283494755,
+            0.009407930200720534,
+            0.002560949424273885,
+            0.010411159836451958,
+            0.0052571230238294825,
+            -0.007499470181994448,
+            -6.145889479758644e-05,
+            -0.015163819825329975,
+            0.022836682800304944,
+            0.016390944816242627,
+            0.001713572907204073,
+            -0.0035687505973373878,
+            -0.031210299824387475,
+            0.0029076511444359886,
+            -0.007822231980296341,
+            0.017821806420465356,
+            0.008250605126658402,
+            0.007908773724009905,
+            0.008351355849444162,
+            -0.00743970642051451,
+            -0.002411981861500092,
+            0.012543476434788293,
+            0.0202407777715137,
+            -0.0024635818448627447,
+            0.02535771413579989,
+            -0.012533449594824386,
+            -0.0022489292647974992,
+            0.0012362917358624392,
+            0.01578939075186126,
+            -0.0008001783766371555,
+            0.006311837812042848
+          ],
+          "keyphrases": [
+            "Switch Transformers",
+            "Mixture of Experts",
+            "sparsely-activated model",
+            "training instability",
+            "multilingual settings"
+          ],
+          "title": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "8f4b7200-614f-45df-b481-ec48e0567e94",
+        "properties": {
+          "page_content": "4 Downstream Results 14 4.1 Fine-Tuning 14 4.2 Distillation 16 4.3 Multilingual Learning 17 ",
+          "keyphrases": [
+            "Downstream Results",
+            "Fine-Tuning",
+            "Distillation",
+            "Multilingual Learning"
+          ],
+          "embedding": [
+            0.007475246209651232,
+            0.018637971952557564,
+            -0.022418024018406868,
+            -0.003915054257959127,
+            -0.025920359417796135,
+            -0.01815968006849289,
+            -0.013469329103827477,
+            0.07850165665149689,
+            -0.036689650267362595,
+            0.014703632332384586,
+            0.017773959785699844,
+            -0.022819172590970993,
+            -0.010807863436639309,
+            -0.039034824818372726,
+            -0.02303517609834671,
+            0.02710837498307228,
+            0.017064236104488373,
+            -0.007691249251365662,
+            0.015552215278148651,
+            -0.02539578080177307,
+            -0.010514716617763042,
+            -0.0059323678724467754,
+            -0.04039255902171135,
+            0.03233873471617699,
+            -0.02058199979364872,
+            0.002258388325572014,
+            0.04107142612338066,
+            -0.006360516417771578,
+            -0.03289416804909706,
+            0.017465384677052498,
+            -0.020350567996501923,
+            -0.00360647845081985,
+            0.01323789730668068,
+            -0.029422692954540253,
+            -0.022634027525782585,
+            0.012574459426105022,
+            0.06967639178037643,
+            0.011378728784620762,
+            0.010777005925774574,
+            0.03610335662961006,
+            0.009380700998008251,
+            0.007938110269606113,
+            0.008539832197129726,
+            -0.007521532475948334,
+            -0.002765609649941325,
+            -0.047397226095199585,
+            -0.0303792767226696,
+            -0.04141085967421532,
+            0.011671875603497028,
+            -0.05992539972066879,
+            0.01718766614794731,
+            -0.060480836778879166,
+            0.016693945974111557,
+            -0.014703632332384586,
+            -0.002127243671566248,
+            0.057271648198366165,
+            0.014317912049591541,
+            0.05264301598072052,
+            0.05779622867703438,
+            0.020921431481838226,
+            0.030240418389439583,
+            -0.005635363515466452,
+            0.014194482006132603,
+            -0.025920359417796135,
+            0.002937254961580038,
+            -0.04468176141381264,
+            -0.009534989483654499,
+            -0.028512394055724144,
+            -0.029129546135663986,
+            0.025735212489962578,
+            -0.003972912207245827,
+            0.04144171625375748,
+            -0.007787679322063923,
+            -0.0030394706409424543,
+            -0.01823682337999344,
+            0.014634202234447002,
+            0.00830840039998293,
+            0.00699695385992527,
+            -0.010993009433150291,
+            0.03243130445480347,
+            0.0004541848611552268,
+            -0.0026498937513679266,
+            0.012991037219762802,
+            -0.016786517575383186,
+            0.03557877987623215,
+            -0.016925377771258354,
+            -0.04036170244216919,
+            -0.049773260951042175,
+            0.02891354262828827,
+            -0.026151791214942932,
+            -0.02970041148364544,
+            -0.02309689112007618,
+            0.0036836224608123302,
+            -0.013939906843006611,
+            -0.033449605107307434,
+            0.02380661480128765,
+            -0.007452103309333324,
+            0.001406912342645228,
+            -0.021631157025694847,
+            0.011209012009203434,
+            -0.003843696089461446,
+            0.005519647616893053,
+            0.004609349649399519,
+            0.0007053847657516599,
+            0.02632150799036026,
+            -0.017928248271346092,
+            0.00661894865334034,
+            -0.016678515821695328,
+            0.004844638518989086,
+            0.07368787378072739,
+            0.01564478874206543,
+            0.007540818769484758,
+            -0.02072085812687874,
+            0.025087203830480576,
+            -0.02229459397494793,
+            -0.018745973706245422,
+            -0.03391246870160103,
+            -0.04693436250090599,
+            0.0011436586501076818,
+            -0.027432380244135857,
+            0.04838467016816139,
+            0.025056347250938416,
+            0.006591948214918375,
+            -0.046378929167985916,
+            -0.053167592734098434,
+            0.034221045672893524,
+            0.025904929265379906,
+            0.031582724303007126,
+            -0.01393219269812107,
+            0.02979298308491707,
+            0.019393982365727425,
+            0.0023509610909968615,
+            -0.06424546241760254,
+            -0.0021214578300714493,
+            0.00862469058483839,
+            -0.0044126324355602264,
+            0.01812882162630558,
+            0.011733590625226498,
+            0.017912819981575012,
+            0.001125336973927915,
+            0.03468390926718712,
+            -0.03631936013698578,
+            0.02301974780857563,
+            -0.006086655426770449,
+            -0.022448882460594177,
+            0.0007304565515369177,
+            -0.009596704505383968,
+            0.09985509514808655,
+            0.018607115373015404,
+            -0.006129084620624781,
+            -0.015968792140483856,
+            0.003992198035120964,
+            -0.012682461179792881,
+            0.026861514896154404,
+            0.018406540155410767,
+            0.010159854777157307,
+            0.01717223785817623,
+            0.004825352691113949,
+            0.0030587567016482353,
+            0.0013577330391854048,
+            0.020767144858837128,
+            -0.007043240126222372,
+            0.019378554075956345,
+            0.006148370914161205,
+            0.07208327949047089,
+            0.03388161212205887,
+            -0.0052612158469855785,
+            0.026614654809236526,
+            -0.04964983090758324,
+            0.021044863387942314,
+            0.02553463913500309,
+            0.030857570469379425,
+            0.04526805505156517,
+            0.0030356133356690407,
+            -0.01653965748846531,
+            -0.016987092792987823,
+            -0.020828859880566597,
+            0.03162900730967522,
+            -0.06344316154718399,
+            0.01741909794509411,
+            -0.013153038918972015,
+            -0.014549343846738338,
+            0.020906003192067146,
+            -0.0063450876623392105,
+            0.01806710660457611,
+            -0.023621469736099243,
+            0.04292288050055504,
+            -0.00019466786761768162,
+            0.027324378490447998,
+            0.006726950407028198,
+            0.0067732366733253,
+            0.010908151045441628,
+            0.027386093512177467,
+            0.016940806061029434,
+            -0.0021947447676211596,
+            -0.007737535517662764,
+            -0.013716190122067928,
+            0.03545534610748291,
+            -0.0014773061266168952,
+            0.010576431639492512,
+            -0.019440269097685814,
+            -0.05048298463225365,
+            -0.0035293344408273697,
+            0.05329102277755737,
+            0.005943939555436373,
+            0.020412283018231392,
+            0.023528896272182465,
+            0.05344530940055847,
+            0.0020018848590552807,
+            -0.03142843395471573,
+            0.01661680079996586,
+            -0.01977970264852047,
+            -0.052735585719347,
+            0.026136361062526703,
+            -0.0027559667360037565,
+            -0.0024358194787055254,
+            -0.008269828744232655,
+            0.012867606244981289,
+            -0.04230572655797005,
+            -0.017650529742240906,
+            0.02061285637319088,
+            -0.0011561946012079716,
+            0.013461614958941936,
+            -0.01480391900986433,
+            -0.032554734498262405,
+            0.0017675601411610842,
+            -0.020427711308002472,
+            3.8240483263507485e-05,
+            0.009689277037978172,
+            -0.02221745066344738,
+            0.01800539158284664,
+            0.03056442365050316,
+            -0.022803744301199913,
+            -0.019270552322268486,
+            0.01731109619140625,
+            -0.018761401996016502,
+            0.014194482006132603,
+            0.00035775493597611785,
+            0.003957483451813459,
+            0.02880554087460041,
+            0.0005978654371574521,
+            -0.033418748527765274,
+            -0.02209402061998844,
+            -0.032585594803094864,
+            0.02317403443157673,
+            -0.03468390926718712,
+            -0.024207763373851776,
+            0.01882311701774597,
+            -0.008362401276826859,
+            0.012481886893510818,
+            0.006757807917892933,
+            0.005187929142266512,
+            0.016231082379817963,
+            0.01397847943007946,
+            -0.021584870293736458,
+            0.05446361005306244,
+            0.03206101432442665,
+            -0.024269478395581245,
+            -0.0248403437435627,
+            -0.04853895679116249,
+            0.018360253423452377,
+            -0.018483683466911316,
+            -0.051501285284757614,
+            0.0009676741319708526,
+            -0.003297902876511216,
+            -0.007864823564887047,
+            0.021260865032672882,
+            0.03378903865814209,
+            0.005824366118758917,
+            0.025704355910420418,
+            0.0031937584280967712,
+            0.004107913933694363,
+            -0.014927349053323269,
+            -0.021152863278985023,
+            -0.008015253581106663,
+            -0.03388161212205887,
+            0.019270552322268486,
+            -0.014024765230715275,
+            -0.01820596680045128,
+            0.022541454061865807,
+            0.008586118929088116,
+            0.04875496029853821,
+            0.01447991468012333,
+            0.0017743102507665753,
+            -0.022757457569241524,
+            -0.01633908413350582,
+            -0.014078766107559204,
+            0.02485577203333378,
+            -0.05665449798107147,
+            -0.0132224690169096,
+            0.011579303070902824,
+            -0.015544501133263111,
+            -0.05514247715473175,
+            0.005191786214709282,
+            -0.029808413237333298,
+            0.012782747857272625,
+            0.00558136310428381,
+            -0.005284358747303486,
+            0.010939008556306362,
+            0.01324561145156622,
+            -0.022464310750365257,
+            0.009635276161134243,
+            0.02153858356177807,
+            -0.008524403907358646,
+            -0.03366560861468315,
+            0.027648381888866425,
+            0.019424840807914734,
+            0.028651254251599312,
+            -0.058845385909080505,
+            0.00558907724916935,
+            0.028543252497911453,
+            -0.02470148541033268,
+            -0.013523329980671406,
+            -0.011533016338944435,
+            -0.03366560861468315,
+            0.007120384369045496,
+            0.02971583977341652,
+            0.007575533352792263,
+            -0.04468176141381264,
+            0.007679677568376064,
+            -0.011602446436882019,
+            0.009295842610299587,
+            -0.04212058335542679,
+            0.05168642848730087,
+            0.007108812686055899,
+            0.004316202364861965,
+            0.013052752241492271,
+            0.005288215819746256,
+            -0.011293870396912098,
+            0.03854110464453697,
+            -0.024207763373851776,
+            -0.010206141509115696,
+            -0.017465384677052498,
+            0.022526025772094727,
+            0.021662013605237007,
+            -0.04366346076130867,
+            -0.049032676964998245,
+            0.006044226698577404,
+            0.010661290027201176,
+            -0.015444213524460793,
+            -0.03527020290493965,
+            0.02459348365664482,
+            -0.007394245360046625,
+            -0.02064371481537819,
+            0.006414517294615507,
+            -0.029299262911081314,
+            -0.008964124135673046,
+            0.0019999563228338957,
+            0.0058937957510352135,
+            0.015536786057054996,
+            0.0009117447771131992,
+            0.010946722701191902,
+            -0.005990225821733475,
+            0.03780052438378334,
+            -0.012875321321189404,
+            -0.02951526641845703,
+            -0.020967718213796616,
+            0.019501984119415283,
+            0.024917487055063248,
+            0.03162900730967522,
+            -0.008022967725992203,
+            -0.04039255902171135,
+            -0.035825639963150024,
+            -0.003938197158277035,
+            -0.025843214243650436,
+            -0.035856496542692184,
+            -0.005191786214709282,
+            -0.003957483451813459,
+            -0.004836923908442259,
+            0.01324561145156622,
+            -0.02542663738131523,
+            -0.002941112034022808,
+            0.0429537370800972,
+            0.02146144025027752,
+            -0.0017280238680541515,
+            -0.033418748527765274,
+            -0.014271626248955727,
+            -0.03443704918026924,
+            -0.036597076803445816,
+            0.020319709554314613,
+            -0.0013972693122923374,
+            0.020273422822356224,
+            -0.021260865032672882,
+            -0.009249556809663773,
+            0.016061365604400635,
+            0.018283110111951828,
+            0.0148656340315938,
+            0.007347958628088236,
+            -0.039806265383958817,
+            -0.030641566962003708,
+            -0.01643165573477745,
+            -0.009851278737187386,
+            -0.021692872047424316,
+            -0.029283834621310234,
+            0.012682461179792881,
+            0.040917135775089264,
+            -0.02468605525791645,
+            0.005211072042584419,
+            0.015467356890439987,
+            -0.021708300337195396,
+            -0.057395078241825104,
+            0.03289416804909706,
+            -0.015220495872199535,
+            -0.02062828466296196,
+            0.0032169015612453222,
+            0.01279046293348074,
+            0.005014355294406414,
+            -0.022047733888030052,
+            0.015251354314386845,
+            0.01982598938047886,
+            0.009974709711968899,
+            -0.017743103206157684,
+            0.019285980612039566,
+            -0.03193758428096771,
+            0.02479405701160431,
+            0.0015737360808998346,
+            -0.022711170837283134,
+            -0.001737666898407042,
+            -0.06436888873577118,
+            -0.003911197185516357,
+            0.0006523483316414058,
+            0.020227137953042984,
+            0.03357303515076637,
+            -0.003241973463445902,
+            0.01581450365483761,
+            0.03369646519422531,
+            0.02561178244650364,
+            -0.0017135593807324767,
+            0.0020906003192067146,
+            -0.01655508577823639,
+            0.02309689112007618,
+            0.018622543662786484,
+            0.05415503680706024,
+            -0.01823682337999344,
+            -0.008370115421712399,
+            -0.017496241256594658,
+            -0.014634202234447002,
+            0.03755366429686546,
+            -0.0181133933365345,
+            -0.005079927388578653,
+            0.0010925508104264736,
+            -0.04853895679116249,
+            0.016771089285612106,
+            0.011239869520068169,
+            0.014209911227226257,
+            0.017588814720511436,
+            -0.00450520496815443,
+            0.013484758324921131,
+            -0.022587740793824196,
+            -0.017588814720511436,
+            -0.013253326527774334,
+            0.054031603038311005,
+            -0.07307072728872299,
+            -0.04116399958729744,
+            0.05032869800925255,
+            -0.02803410217165947,
+            0.02806496061384678,
+            0.003409761469811201,
+            0.024207763373851776,
+            -0.05208757892251015,
+            -0.013531044125556946,
+            0.02883639931678772,
+            0.014989065006375313,
+            -0.03465305268764496,
+            -0.0110624385997653,
+            -0.03378903865814209,
+            -0.051470424979925156,
+            -0.044002894312143326,
+            -0.022865459322929382,
+            0.009843564592301846,
+            0.006009511649608612,
+            -0.025102633982896805,
+            -0.012613032013177872,
+            -0.01561393029987812,
+            -0.021677441895008087,
+            0.00471735093742609,
+            -0.027401521801948547,
+            -0.03551706299185753,
+            -0.012512744404375553,
+            0.022896315902471542,
+            -0.0012661246582865715,
+            0.01993398927152157,
+            -0.017635101452469826,
+            -0.006931381765753031,
+            -0.024161476641893387,
+            -0.0010607289150357246,
+            0.03675136715173721,
+            0.009820421226322651,
+            0.04668750241398811,
+            -0.0010134782642126083,
+            0.009311271831393242,
+            -0.03354217857122421,
+            -0.000637883844319731,
+            0.008439545519649982,
+            0.005176357459276915,
+            -0.04934125393629074,
+            -0.003911197185516357,
+            -0.04764408618211746,
+            0.01885397545993328,
+            0.002808038843795657,
+            -0.01198045164346695,
+            -0.002281531458720565,
+            -0.012952464632689953,
+            -0.0004305595357436687,
+            0.026969516649842262,
+            -0.0102987140417099,
+            -0.0075948191806674,
+            -0.009357557632029057,
+            -0.034221045672893524,
+            0.03486905246973038,
+            -0.014348769560456276,
+            -0.017743103206157684,
+            -0.008732692338526249,
+            -0.019270552322268486,
+            -0.018375683575868607,
+            0.011425015516579151,
+            0.0012391243362799287,
+            -0.00016139955550897866,
+            -0.01235074270516634,
+            -0.005083784461021423,
+            0.02152315527200699,
+            0.01800539158284664,
+            -0.0023509610909968615,
+            0.03273988142609596,
+            -0.04175029322504997,
+            0.019100835546851158,
+            0.0014223410980775952,
+            -0.015637073665857315,
+            0.03709080070257187,
+            0.01435648463666439,
+            -0.02562721259891987,
+            0.001886168960481882,
+            0.03397418558597565,
+            -0.0020057419314980507,
+            0.010753863491117954,
+            -0.01074614841490984,
+            -0.011695018969476223,
+            0.025133490562438965,
+            -0.016678515821695328,
+            -0.0015814504586160183,
+            -0.021245436742901802,
+            0.014171338640153408,
+            -0.014873349107801914,
+            0.029052402824163437,
+            0.009396130219101906,
+            0.022803744301199913,
+            0.020998576655983925,
+            -0.013862763531506062,
+            0.006129084620624781,
+            0.019147122278809547,
+            -0.03456047922372818,
+            0.013099038042128086,
+            -0.009141555055975914,
+            0.004084771033376455,
+            -0.01147901639342308,
+            -0.007228385657072067,
+            0.012096167542040348,
+            -0.015073922462761402,
+            0.014564773067831993,
+            -0.013986193574965,
+            0.024207763373851776,
+            0.039219971746206284,
+            -0.026599224656820297,
+            0.007664248812943697,
+            0.001095443731173873,
+            0.0008090469054877758,
+            0.03480733931064606,
+            0.004443489946424961,
+            -0.03277073800563812,
+            -0.006946810521185398,
+            -0.0072823865339159966,
+            0.027756383642554283,
+            -0.026676369830965996,
+            -0.01720309443771839,
+            0.03807824105024338,
+            0.030795855447649956,
+            -0.002358675468713045,
+            -0.03227701783180237,
+            -0.019208837300539017,
+            0.003047185018658638,
+            0.01893111877143383,
+            -0.025735212489962578,
+            -0.029422692954540253,
+            -0.009619846940040588,
+            0.011841592378914356,
+            0.017511671409010887,
+            -0.00911841168999672,
+            -0.01280589122325182,
+            0.0030992571264505386,
+            -0.006811808329075575,
+            0.010784721001982689,
+            -0.004269916098564863,
+            0.0074598174542188644,
+            -0.05878366902470589,
+            -0.006645949091762304,
+            0.011232155375182629,
+            0.04067027568817139,
+            0.00450134789571166,
+            0.03452961891889572,
+            -0.06273344159126282,
+            -0.0036122642923146486,
+            -0.0038880538195371628,
+            -0.01735738292336464,
+            -0.015698788687586784,
+            -0.016832804307341576,
+            -0.004455061629414558,
+            -0.013847334310412407,
+            0.02058199979364872,
+            0.024500910192728043,
+            0.012520458549261093,
+            0.014742203988134861,
+            -0.0058976528234779835,
+            0.042490873485803604,
+            -0.009372986853122711,
+            -0.014186767861247063,
+            0.00041947010322473943,
+            -0.002705823164433241,
+            0.004146486055105925,
+            0.007818536832928658,
+            -0.006572662387043238,
+            0.035825639963150024,
+            0.020890574902296066,
+            0.008832979016005993,
+            0.00911841168999672,
+            0.005693221464753151,
+            0.0458543486893177,
+            0.027432380244135857,
+            -0.045761775225400925,
+            -0.013153038918972015,
+            -0.0098589938133955,
+            -0.017758531495928764,
+            -0.01808253675699234,
+            0.006534090265631676,
+            -0.029823841527104378,
+            -0.00904898252338171,
+            0.008262114599347115,
+            0.019301410764455795,
+            0.015876220539212227,
+            0.016801945865154266,
+            -0.01825225166976452,
+            -0.04684179276227951,
+            -0.01720309443771839,
+            -0.004208201076835394,
+            0.0246089119464159,
+            -0.008030682802200317,
+            -0.0034599050413817167,
+            -0.003587192390114069,
+            -4.5985012548044324e-05,
+            0.01820596680045128,
+            0.01360818836838007,
+            -0.0073248157277703285,
+            -0.010831006802618504,
+            0.012960179708898067,
+            0.037461090832948685,
+            0.028543252497911453,
+            -0.02294260263442993,
+            0.012620746158063412,
+            0.01393219269812107,
+            -0.022850031033158302,
+            0.02237173728644848,
+            0.04051598906517029,
+            0.020196279510855675,
+            0.0015100922901183367,
+            -0.02641407959163189,
+            0.014950492419302464,
+            -0.01068443339318037,
+            0.014773061498999596,
+            0.0029681124724447727,
+            0.009226413443684578,
+            0.025997502729296684,
+            0.02632150799036026,
+            -0.0033962612506002188,
+            -0.020288852974772453,
+            0.023220321163535118,
+            0.0009445309406146407,
+            -0.0037183372769504786,
+            0.012173310853540897,
+            0.030703281983733177,
+            -0.009897565469145775,
+            0.004424204118549824,
+            -0.03403589874505997,
+            0.006742379162460566,
+            0.011371014639735222,
+            0.02465519867837429,
+            -0.004300773609429598,
+            0.004293059464544058,
+            0.004624778404831886,
+            0.009743277914822102,
+            -0.02798781543970108,
+            -0.026815228164196014,
+            -0.007201385218650103,
+            0.016508799046278,
+            -0.016092222183942795,
+            0.02301974780857563,
+            -0.014695917256176472,
+            0.017851104959845543,
+            -0.03471476584672928,
+            0.04144171625375748,
+            0.024392908439040184,
+            -0.03372732549905777,
+            -0.020134564489126205,
+            -0.013376756571233273,
+            -0.05677792802453041,
+            0.011671875603497028,
+            0.03604163974523544,
+            -0.010074996389448643,
+            -0.0062910872511565685,
+            -0.017449956387281418,
+            0.007313244044780731,
+            -0.013299612328410149,
+            -0.014773061498999596,
+            0.007475246209651232,
+            0.011810734868049622,
+            0.018452826887369156,
+            -0.008755835704505444,
+            0.029006116092205048,
+            0.012775033712387085,
+            -0.03511591628193855,
+            0.016508799046278,
+            0.030101560056209564,
+            0.010098139755427837,
+            0.024917487055063248,
+            -0.001642201328650117,
+            -0.010599575005471706,
+            -0.007598676718771458,
+            0.03597992658615112,
+            0.022479739040136337,
+            0.01562164444476366,
+            0.010838720947504044,
+            -0.010591860860586166,
+            0.026043789461255074,
+            -0.012235026806592941,
+            -0.0222020223736763,
+            0.00449749082326889,
+            -0.018375683575868607,
+            -0.00942698772996664,
+            -0.00900269579142332,
+            0.018375683575868607,
+            -0.001383769093081355,
+            -0.010429858230054379,
+            0.010260142385959625,
+            0.0023914617486298084,
+            -0.003322974545881152,
+            -0.006456946488469839,
+            -0.004354774486273527,
+            0.004744351375848055,
+            0.03397418558597565,
+            0.009218699298799038,
+            -0.010476144962012768,
+            0.03687479719519615,
+            0.007386530749499798,
+            -0.027463236823678017,
+            0.01973341591656208,
+            -0.01902369223535061,
+            -0.019949419423937798,
+            -5.614028486888856e-05,
+            0.020844288170337677,
+            -0.012289026752114296,
+            -0.02465519867837429,
+            0.0018485613400116563,
+            -0.009797278791666031,
+            0.007309386972337961,
+            0.022726600989699364,
+            0.00014801989891566336,
+            0.028496965765953064,
+            -0.017048807814717293,
+            -0.016184795647859573,
+            0.024300336837768555,
+            0.011155011132359505,
+            -0.012150168418884277,
+            0.034221045672893524,
+            -0.006279515568166971,
+            0.05955510959029198,
+            -0.0363810732960701,
+            0.003913125488907099,
+            -0.024423766881227493,
+            -0.01806710660457611,
+            -0.014155910350382328,
+            -0.040886279195547104,
+            0.029345549643039703,
+            -0.05350702628493309,
+            0.019579127430915833,
+            0.0058089373633265495,
+            -0.016678515821695328,
+            -0.0376770943403244,
+            0.0015978434821590781,
+            0.017789389938116074,
+            -0.008053826168179512,
+            -0.02808038890361786,
+            0.004204344004392624,
+            -0.0008818514761514962,
+            0.006468518171459436,
+            -0.04273773357272148,
+            0.00906441081315279,
+            0.05292073264718056,
+            -0.03363475203514099,
+            0.010993009433150291,
+            0.009635276161134243,
+            0.0032496878411620855,
+            -0.016987092792987823,
+            0.03613421320915222,
+            0.003791623981669545,
+            -0.003820552956312895,
+            0.02295803278684616,
+            -0.03443704918026924,
+            -0.04474347457289696,
+            0.019316839054226875,
+            -0.0011455873027443886,
+            0.03320274502038956,
+            -0.003004755824804306,
+            0.026938658207654953,
+            0.0062949443235993385,
+            0.023420896381139755,
+            0.027740955352783203,
+            -0.0019594556652009487,
+            0.010954436846077442,
+            -0.00828525796532631,
+            -0.014402770437300205,
+            -0.010445287451148033,
+            -0.02954612299799919,
+            0.0102987140417099,
+            0.027000373229384422,
+            -0.025133490562438965,
+            0.02048942632973194,
+            -0.03613421320915222,
+            0.007749107200652361,
+            -0.017527099698781967,
+            0.011016152799129486,
+            0.051532141864299774,
+            0.047335512936115265,
+            0.0026981087867170572,
+            -0.025287779048085213,
+            0.009642990306019783,
+            0.01743452623486519,
+            -0.024932917207479477,
+            0.031814154237508774,
+            -0.0024628196842968464,
+            -0.007976681925356388,
+            -0.013099038042128086,
+            0.031845010817050934,
+            -0.0038282673340290785,
+            -0.0073711019940674305,
+            -0.000637883844319731,
+            -0.01242017187178135,
+            0.01324561145156622,
+            -0.006090512964874506,
+            -0.01657051593065262,
+            -0.017002521082758904,
+            0.012620746158063412,
+            -0.012582174502313137,
+            0.012065310031175613,
+            0.013014179654419422,
+            0.03388161212205887,
+            0.013407614082098007,
+            -0.019116263836622238,
+            0.023405466228723526,
+            0.02638322301208973,
+            0.0657266229391098,
+            -0.012952464632689953,
+            0.012435601092875004,
+            0.007614105474203825,
+            -0.021168293431401253,
+            -0.002835039282217622,
+            -0.02895982936024666,
+            0.0015014136442914605,
+            -0.028543252497911453,
+            -0.008917837403714657,
+            -0.034190189093351364,
+            -0.023606041446328163,
+            -0.03770795091986656,
+            0.01110101118683815,
+            -0.001489842077717185,
+            -0.026198076084256172,
+            0.01902369223535061,
+            -0.006495518609881401,
+            -0.010753863491117954,
+            -0.013477043248713017,
+            -0.026136361062526703,
+            0.010090425610542297,
+            -0.029345549643039703,
+            0.003562120720744133,
+            0.006237086374312639,
+            0.012119310908019543,
+            0.00829297211021185,
+            0.019548270851373672,
+            0.00910298340022564,
+            0.018468255177140236,
+            0.0013934121234342456,
+            -0.02726266346871853,
+            -0.026876943185925484,
+            0.010931294411420822,
+            0.011201297864317894,
+            0.014896491542458534,
+            -0.013947621919214725,
+            -0.006807951256632805,
+            0.02894440107047558,
+            0.0014638060238212347,
+            -0.010792435146868229,
+            -0.022016875445842743,
+            -0.014981349930167198,
+            -0.02153858356177807,
+            -0.025688927620649338,
+            -0.001375090447254479,
+            -0.02298888936638832,
+            -0.00899498164653778,
+            -0.007116526830941439,
+            -0.0008288150420412421,
+            -0.0005269894609227777,
+            0.0005901510012336075,
+            0.01817510835826397,
+            -0.03968283534049988,
+            -0.0009941923199221492,
+            0.0003018256102222949,
+            -0.04705779254436493,
+            0.024130620062351227,
+            -0.004995069000869989,
+            0.013014179654419422,
+            -0.027355235069990158,
+            0.020335137844085693,
+            0.007922681048512459,
+            0.011749019846320152,
+            -0.0015428784536197782,
+            -0.01718766614794731,
+            0.014742203988134861,
+            0.00575107941403985,
+            -0.018730545416474342,
+            0.0016566658159717917,
+            0.028203818947076797,
+            0.016693945974111557,
+            0.00786096602678299,
+            0.014240768738090992,
+            0.01198045164346695,
+            0.005241929553449154,
+            -0.00236060400493443,
+            0.047335512936115265,
+            -0.012967893853783607,
+            0.012204169295728207,
+            0.017527099698781967,
+            -0.005203357897698879,
+            -0.0003975322761107236,
+            0.006510947365313768,
+            -0.004740494303405285,
+            0.0062949443235993385,
+            0.003673979314044118,
+            0.029438121244311333,
+            -0.00906441081315279,
+            0.018452826887369156,
+            0.034961625933647156,
+            -0.014734489843249321,
+            0.0008148326887749135,
+            -0.005461789667606354,
+            0.007020097225904465,
+            0.009704705327749252,
+            -0.014256197027862072,
+            -0.014911920763552189,
+            0.028157532215118408,
+            -0.01199587993323803,
+            0.02953069470822811,
+            -0.004705779254436493,
+            0.014626488089561462,
+            0.03527020290493965,
+            0.00764882005751133,
+            0.004250630270689726,
+            -0.00861697643995285,
+            -0.0078146792948246,
+            0.013315041549503803,
+            -0.021260865032672882,
+            0.011679589748382568,
+            -0.021245436742901802,
+            0.027679240331053734,
+            -0.031690724194049835,
+            -0.005049069877713919,
+            -0.012636174447834492,
+            -0.01112415362149477,
+            0.02474777027964592,
+            0.0017627386841922998,
+            0.0034714764915406704,
+            -0.0057587940245866776,
+            -0.013369042426347733,
+            0.01110872533172369,
+            0.01965627260506153,
+            -0.007166670635342598,
+            0.012960179708898067,
+            0.004632492549717426,
+            -0.030286705121397972,
+            -0.014965921640396118,
+            0.030147844925522804,
+            0.0029777553863823414,
+            -0.021862588822841644,
+            0.014379628002643585,
+            -0.0062910872511565685,
+            -0.004401060752570629,
+            0.036535363644361496,
+            -0.009851278737187386,
+            -0.0074328170157969,
+            -0.0009223520755767822,
+            -0.001847596955485642,
+            -0.001201516599394381,
+            0.02394547499716282,
+            0.021785443648695946,
+            -0.02641407959163189,
+            -0.007999825291335583,
+            -0.014009336940944195,
+            0.007498389575630426,
+            -0.013816476799547672,
+            0.02371404320001602,
+            0.031783297657966614,
+            -0.00948870275169611,
+            0.00596708245575428,
+            0.004536062944680452,
+            0.013199325650930405,
+            -0.01517421007156372,
+            -0.023374609649181366,
+            -0.0017212738748639822,
+            0.005943939555436373,
+            -0.01569107361137867,
+            -0.006746236234903336,
+            -0.04384860768914223,
+            -0.011749019846320152,
+            -0.0026730368845164776,
+            -0.029268404468894005,
+            -0.04011484235525131,
+            -0.031845010817050934,
+            -0.005546648055315018,
+            0.02052028477191925,
+            0.016771089285612106,
+            0.0017193452222272754,
+            0.02386832982301712,
+            -0.005724078975617886,
+            0.005623792298138142,
+            0.021924303844571114,
+            0.00011264218483120203,
+            0.015073922462761402,
+            -0.0006962239276617765,
+            0.038170814514160156,
+            0.014665059745311737,
+            -0.0013162682298570871,
+            -0.05671621486544609,
+            -0.0188385471701622,
+            0.012937036342918873,
+            0.021168293431401253,
+            -0.0041271997615695,
+            0.02139972522854805,
+            -0.005245787091553211,
+            -0.020859716460108757,
+            -0.012543601915240288,
+            0.007035525981336832,
+            0.0037067655939608812,
+            0.007733678445219994,
+            0.0048176380805671215,
+            0.0036103357560932636,
+            0.029345549643039703,
+            0.026753513142466545,
+            0.00048239060561172664,
+            0.008940980769693851,
+            0.04224401339888573,
+            -0.006121370475739241,
+            0.0008119398262351751,
+            -0.00742510287091136,
+            0.022402595728635788,
+            -0.018653400242328644,
+            0.02377575822174549,
+            0.019995706155896187,
+            0.008771263994276524,
+            0.023467181250452995,
+            0.016169367358088493,
+            -0.00515707116574049,
+            -0.009226413443684578,
+            0.032647307962179184,
+            -0.010437573306262493,
+            -0.029098687693476677,
+            -0.007228385657072067,
+            -0.012975607998669147,
+            0.011826164089143276,
+            0.005608363542705774,
+            0.01191873662173748,
+            0.0033789039589464664,
+            0.0006263122195377946,
+            -0.02129172347486019,
+            -0.011949594132602215,
+            -0.00022540490317624062,
+            0.007760678883641958,
+            0.011193583719432354,
+            0.017465384677052498,
+            -0.0034406189806759357,
+            -0.018607115373015404,
+            0.033387891948223114,
+            -0.020782573148608208,
+            0.041565146297216415,
+            0.00011203949543414637,
+            -0.023189464583992958,
+            0.004038484301418066,
+            -0.00030399527167901397,
+            0.023405466228723526,
+            -0.009982423856854439,
+            -0.019069978967308998,
+            0.017943676561117172,
+            0.06066598370671272,
+            0.012273598462343216,
+            -0.016231082379817963,
+            -0.00787639431655407,
+            -0.02970041148364544,
+            0.01806710660457611,
+            0.006171513814479113,
+            0.026630083099007607,
+            0.02800324372947216,
+            0.021662013605237007,
+            -0.0042776307091116905,
+            0.02314317785203457,
+            -0.014626488089561462,
+            0.010530145838856697,
+            -0.004532205406576395,
+            0.02295803278684616,
+            0.0036450503394007683,
+            -0.013500186614692211,
+            0.034282758831977844,
+            0.0021793157793581486,
+            -0.002719323383644223,
+            -0.015590786933898926,
+            0.00575107941403985,
+            0.004875496029853821,
+            -0.00640294561162591,
+            0.02633693628013134,
+            0.03059528023004532,
+            0.003114685881882906,
+            -0.021631157025694847,
+            -0.010993009433150291,
+            0.016801945865154266,
+            0.020319709554314613,
+            0.0015920577570796013,
+            0.016925377771258354,
+            0.013816476799547672,
+            0.033295318484306335,
+            -0.0029989699833095074,
+            -0.0008153148810379207,
+            0.0041271997615695,
+            -0.0058937957510352135,
+            0.005461789667606354,
+            0.011417300440371037,
+            0.034190189093351364,
+            -0.00020033313194289804,
+            -0.018499113619327545,
+            0.009581275284290314,
+            0.011826164089143276,
+            -0.039034824818372726,
+            0.004393346607685089,
+            -0.022510597482323647,
+            0.0006914024124853313,
+            0.00992070883512497,
+            -0.022865459322929382,
+            -0.021615726873278618,
+            0.004046198911964893,
+            0.013029608875513077,
+            0.018360253423452377,
+            0.0156293585896492,
+            -0.019378554075956345,
+            -0.014194482006132603,
+            0.02718551829457283,
+            0.002271888544782996,
+            0.009781849570572376,
+            0.024346621707081795,
+            -0.006599662825465202,
+            0.0015824147267267108,
+            0.019116263836622238,
+            0.004138771444559097,
+            0.03150557726621628,
+            -0.002854325110092759,
+            -0.028311820700764656,
+            0.012466458603739738,
+            0.006210085935890675,
+            -0.009650705382227898,
+            -0.006896666716784239,
+            -0.027694668620824814,
+            -0.015035350807011127,
+            -0.030240418389439583,
+            0.003689408302307129,
+            -0.02305060438811779,
+            -0.0016373797552660108,
+            0.012235026806592941,
+            -0.011471301317214966,
+            0.006437660660594702,
+            -0.013777905143797398,
+            -0.02389918826520443,
+            -0.003689408302307129,
+            -0.0001803239283617586,
+            -0.012975607998669147,
+            -0.03033299185335636,
+            -0.02132258005440235,
+            -0.017835674807429314,
+            -0.014317912049591541,
+            -0.012073024176061153,
+            -0.019532842561602592,
+            -0.0008567796903662384,
+            0.02056656964123249,
+            0.007907251827418804,
+            -0.035085055977106094,
+            0.004208201076835394,
+            -0.004289202392101288,
+            -0.010128997266292572,
+            0.008871551603078842,
+            -0.02135343849658966,
+            -0.00038403208600357175,
+            0.02223287895321846,
+            0.011440443806350231,
+            -0.0165859442204237,
+            -0.004188915248960257,
+            -0.010954436846077442,
+            0.03059528023004532,
+            0.0094192735850811,
+            -0.008223542012274265,
+            -0.0058860816061496735,
+            0.024192335084080696,
+            -0.0023471040185540915,
+            -0.00023818186309654266,
+            -0.0024821057450026274,
+            -0.0009724955889396369,
+            0.0008399044745601714,
+            -0.024269478395581245,
+            0.030857570469379425,
+            0.00450520496815443,
+            0.002742466516792774,
+            -0.024979202076792717,
+            0.024315765127539635,
+            0.02314317785203457,
+            -0.030764997005462646,
+            0.0073749590665102005,
+            0.0012931249802932143,
+            0.010653575882315636,
+            0.00905669666826725,
+            0.0026267506182193756,
+            -0.008809836581349373,
+            0.03221530467271805,
+            -0.025256920605897903,
+            0.008123255334794521,
+            0.012929322198033333,
+            0.040053125470876694,
+            0.02226373739540577,
+            -0.007301672361791134,
+            0.009473273530602455,
+            0.020119136199355125,
+            0.013515615835785866,
+            0.017835674807429314,
+            -0.017665958032011986,
+            -0.01272874791175127,
+            0.0001393412094330415,
+            -0.0038109098095446825,
+            0.010468430817127228,
+            0.0013490543933585286,
+            -0.03155186399817467,
+            0.005631506443023682,
+            0.01067671924829483,
+            -0.007560104597359896,
+            -0.019285980612039566,
+            0.023667756468057632,
+            -0.004019198473542929,
+            0.007020097225904465,
+            -0.003754980629310012,
+            -0.006499375682324171,
+            0.011571588926017284,
+            -0.05936996266245842,
+            -0.020057421177625656,
+            0.008871551603078842,
+            0.0053036450408399105,
+            0.01447220053523779,
+            0.010020995512604713,
+            -0.02963869646191597,
+            -0.007197528146207333,
+            -0.02874382585287094,
+            0.018344825133681297,
+            -0.00987442210316658,
+            0.020057421177625656,
+            0.0221248771995306,
+            -0.008493546396493912,
+            0.0058436524122953415,
+            -0.003068399615585804,
+            0.018637971952557564,
+            -0.012451029382646084,
+            0.022603169083595276,
+            -0.016801945865154266,
+            -0.015004493296146393,
+            -0.00944241601973772,
+            0.011386442929506302,
+            -0.007938110269606113,
+            0.01449534296989441,
+            -0.024238621816039085,
+            -0.02141515351831913,
+            0.007922681048512459,
+            0.013531044125556946,
+            0.014603344723582268,
+            -0.01806710660457611,
+            0.007999825291335583,
+            -0.005616077687591314,
+            0.01105472445487976,
+            0.006152227986603975,
+            0.009712420403957367,
+            -0.0003524513158481568,
+            -0.022649455815553665,
+            -0.008354687131941319,
+            0.007563961669802666,
+            0.0008644941262900829,
+            0.004250630270689726,
+            -0.008794407360255718,
+            -0.029021544381976128,
+            0.007787679322063923,
+            -0.014896491542458534,
+            -0.019347695633769035,
+            -0.01653965748846531,
+            -0.009689277037978172,
+            -0.010838720947504044,
+            0.0018543470650911331,
+            0.02389918826520443,
+            -0.020072849467396736,
+            -0.010607290081679821,
+            0.01323789730668068,
+            -0.013083609752357006,
+            0.016153937205672264,
+            0.0035949070006608963,
+            -0.007629534229636192,
+            0.00552350515499711,
+            0.0003452190721873194,
+            0.03684393689036369,
+            0.0019903131760656834,
+            -0.02152315527200699,
+            0.01312218140810728,
+            -0.015498214401304722,
+            0.02382204309105873,
+            0.007976681925356388,
+            -0.0021349580492824316,
+            0.011571588926017284,
+            0.024886630475521088,
+            0.0027366806752979755,
+            0.020443139597773552,
+            -0.011008437722921371,
+            -0.029484407976269722,
+            0.007961252704262733,
+            -0.008817550726234913,
+            -0.01991856098175049,
+            -0.0198722742497921,
+            0.003754980629310012,
+            -0.010931294411420822,
+            0.023575183004140854,
+            -0.022649455815553665,
+            -0.0003042363387066871,
+            0.00038089812733232975,
+            0.0015814504586160183,
+            -0.014888777397572994,
+            0.013585045002400875,
+            0.007968967780470848,
+            -0.017542527988553047,
+            0.00779925100505352,
+            -0.01652422919869423,
+            -0.02130715176463127,
+            -2.7196247174288146e-05,
+            -0.008516688831150532,
+            -0.0022622456308454275,
+            -0.017064236104488373,
+            -0.0030201845802366734,
+            0.00033726359833963215,
+            -0.0006306516006588936,
+            0.016956234350800514,
+            0.019162550568580627,
+            -0.007745250128209591,
+            0.01894654706120491,
+            0.003976769279688597,
+            0.02209402061998844,
+            0.0031648294534534216,
+            -0.006985382176935673,
+            -0.022402595728635788,
+            0.02642950788140297,
+            -0.02567349746823311,
+            -0.0024338907096534967,
+            0.01235074270516634,
+            -0.014904206618666649,
+            -0.0012005523312836885,
+            -0.0297312680631876,
+            -0.004678778816014528,
+            -0.0014011265011504292,
+            -0.0018177037127315998,
+            0.04551491513848305,
+            -0.003259330755099654,
+            -0.012967893853783607,
+            -0.0131067531183362,
+            0.0006841701688244939,
+            0.0066112345084548,
+            0.005006640683859587,
+            0.01640079915523529,
+            -0.01154073141515255,
+            -0.02391461655497551,
+            0.016030507162213326,
+            -0.005855224095284939,
+            0.01647794246673584,
+            0.020196279510855675,
+            0.0026286791544407606,
+            -0.03307931497693062,
+            0.012505030259490013,
+            -0.01905454881489277,
+            -0.00944241601973772,
+            0.019270552322268486,
+            0.02801867388188839,
+            -0.00867869146168232,
+            -0.01072300598025322,
+            0.011633303947746754,
+            -0.000418505776906386,
+            -0.01066900510340929,
+            -0.02479405701160431,
+            -0.009403844363987446,
+            -0.013762475922703743,
+            0.013731618411839008,
+            0.006113655865192413,
+            -0.02065914310514927,
+            0.006483946926891804,
+            -0.006414517294615507,
+            0.01230445597320795,
+            -0.01154073141515255,
+            0.003432904602959752,
+            0.018514541909098625,
+            0.02150772698223591,
+            -0.0230814628303051,
+            0.025827785953879356,
+            -0.027648381888866425,
+            0.021075719967484474,
+            -0.0008799228817224503,
+            0.026707226410508156,
+            -0.029484407976269722,
+            -0.018313968554139137,
+            -0.013006465509533882,
+            -0.011378728784620762,
+            -0.025195205584168434,
+            -0.006078941281884909,
+            0.01441048551350832,
+            0.013970764353871346,
+            0.009311271831393242,
+            0.026799799874424934,
+            -0.012096167542040348,
+            0.018653400242328644,
+            0.02064371481537819,
+            -0.01193416491150856,
+            0.005681650247424841,
+            0.011733590625226498,
+            -0.009558131918311119,
+            0.012813606299459934,
+            0.001909312093630433,
+            0.010445287451148033,
+            -0.033264461904764175,
+            0.022078590467572212,
+            0.014819348230957985,
+            -0.0037299087271094322,
+            -0.03872625157237053,
+            -0.0016798089491203427,
+            -0.020057421177625656,
+            -0.02315860614180565,
+            -0.0014310197439044714,
+            0.011957308277487755,
+            0.028682110831141472,
+            0.018622543662786484,
+            -0.036689650267362595,
+            -0.000610401330050081,
+            -0.01579907536506653,
+            -0.011232155375182629,
+            -0.014024765230715275,
+            0.012921607121825218,
+            -0.009018125012516975,
+            0.014958206564188004,
+            -0.0004170593456365168,
+            0.002379890065640211,
+            -0.003363475203514099,
+            -0.01231217011809349,
+            0.016138508915901184,
+            0.010275570675730705,
+            -0.009457845240831375,
+            0.010429858230054379,
+            0.013816476799547672,
+            0.0026074645575135946,
+            -0.004644064232707024,
+            -0.012597602792084217,
+            -0.02485577203333378,
+            0.011617874726653099,
+            -0.0018205966334789991,
+            0.007498389575630426,
+            -0.009797278791666031,
+            0.0034676194190979004,
+            -0.047366369515657425,
+            0.003726051654666662,
+            0.0032265447080135345,
+            0.030641566962003708,
+            0.011749019846320152,
+            0.018622543662786484,
+            0.014919634908437729,
+            0.0005795437609776855,
+            0.010591860860586166,
+            0.02127629518508911,
+            0.03786223754286766,
+            0.00537307420745492,
+            0.014896491542458534,
+            -5.394048002926866e-06,
+            -0.007563961669802666,
+            0.0074559603817760944,
+            -0.006125227548182011,
+            -0.02062828466296196,
+            -0.006360516417771578,
+            -0.00020177957776468247,
+            0.0008881194517016411,
+            -0.00860926229506731,
+            0.004308488219976425,
+            -0.017465384677052498,
+            -0.010838720947504044,
+            -0.02794152870774269,
+            0.00216581579297781,
+            -0.025118062272667885,
+            -0.015891648828983307,
+            -0.016308225691318512,
+            -0.010869579389691353,
+            -0.003953625913709402,
+            0.022772885859012604,
+            -0.0049063535407185555,
+            -0.033418748527765274,
+            -0.0037029082886874676,
+            0.02727809175848961,
+            -0.008269828744232655,
+            -0.010530145838856697,
+            0.011725876480340958,
+            0.010059568099677563,
+            -0.008022967725992203,
+            0.01362361665815115,
+            0.010229283943772316,
+            -0.009357557632029057,
+            -0.0013104823883622885,
+            -0.01560621615499258,
+            0.02963869646191597,
+            0.017110522836446762,
+            0.002422319259494543,
+            -0.004528348334133625,
+            -0.017449956387281418,
+            -0.0016084507806226611,
+            -0.0013914835872128606,
+            0.02480948530137539,
+            -0.021569442003965378,
+            0.027756383642554283,
+            -0.0031474721617996693,
+            -0.02227916568517685,
+            0.013747047632932663,
+            0.02055114135146141,
+            -0.02721637673676014,
+            0.0010732648661360145,
+            -0.006588091142475605,
+            0.015505928546190262,
+            -0.008223542012274265,
+            -0.0057665081694722176,
+            -0.010622718371450901,
+            0.007517675403505564,
+            0.013214753940701485,
+            -0.01967170089483261,
+            0.017542527988553047,
+            0.0110624385997653,
+            0.01280589122325182,
+            -0.02547292411327362,
+            0.013307327404618263,
+            -0.006213943008333445,
+            -0.01275189034640789,
+            -0.0020751715637743473,
+            -0.004825352691113949,
+            0.017573386430740356,
+            0.006059655454009771,
+            0.002231388119980693,
+            -0.00758324796333909,
+            0.003984483890235424,
+            -0.006244800519198179,
+            0.010514716617763042,
+            0.011432729661464691,
+            -0.020273422822356224,
+            -0.004377917852252722,
+            -0.013006465509533882,
+            -0.009349843487143517,
+            0.003344189142808318,
+            -0.007178242318332195,
+            0.03650450333952904,
+            0.00986670795828104,
+            0.004296916536986828,
+            0.02231002226471901,
+            -0.011139582842588425,
+            -0.012913892976939678,
+            0.01660137251019478,
+            0.09436244517564774,
+            3.441342414589599e-05,
+            -0.003969055134803057,
+            0.002773324027657509,
+            -0.010476144962012768,
+            -0.0008866729913279414,
+            -0.01235074270516634,
+            -0.018468255177140236,
+            0.008123255334794521,
+            -0.00514549994841218,
+            -0.021739158779382706,
+            -0.009149269200861454,
+            -0.020011134445667267,
+            0.017141379415988922,
+            0.003322974545881152,
+            0.012242740951478481,
+            0.013939906843006611,
+            0.01905454881489277,
+            0.0031899013556540012,
+            0.014819348230957985,
+            -0.009519560262560844,
+            0.007243814412504435,
+            -0.011347871273756027,
+            0.007263100706040859,
+            0.008123255334794521,
+            0.009210984222590923,
+            0.006244800519198179,
+            0.015012207441031933,
+            0.012003595009446144,
+            0.029006116092205048,
+            0.010283284820616245,
+            -0.0012179097393527627,
+            -0.005049069877713919,
+            -0.016693945974111557,
+            -0.003693265374749899,
+            0.00904898252338171,
+            -0.01984141767024994,
+            -0.022634027525782585,
+            -0.013693046756088734,
+            -0.002790681552141905,
+            0.0019382410682737827,
+            -0.008940980769693851,
+            0.023405466228723526,
+            0.026954086497426033,
+            -0.006800237111747265,
+            0.004269916098564863,
+            -0.007479103282094002,
+            -0.0011108724866062403,
+            -0.00866326317191124,
+            -0.010514716617763042,
+            -0.0013066251995041966,
+            -0.0013596616918221116,
+            0.005295930430293083,
+            -0.021939732134342194,
+            -0.013677617534995079,
+            -0.004107913933694363,
+            0.009766421280801296,
+            -0.00040548774995841086,
+            0.0024956059642136097,
+            0.011363300494849682,
+            -0.01393219269812107,
+            -0.004944925662130117,
+            0.00558136310428381,
+            -0.009550417773425579,
+            -0.008385544642806053,
+            0.006275658495724201,
+            0.012134739197790623,
+            -0.0010173354530707002,
+            -0.011725876480340958,
+            -0.012613032013177872,
+            -0.010453001596033573,
+            0.006109798792749643,
+            0.0221248771995306,
+            -0.007548532914370298,
+            0.008501260541379452,
+            -0.008424116298556328,
+            0.003648907644674182,
+            -0.017727673053741455,
+            0.003157115075737238,
+            -0.0038880538195371628,
+            -0.0021542441099882126,
+            0.00992842298001051,
+            -0.006607376970350742,
+            -0.00817725621163845,
+            0.00011372702283551916,
+            0.0006518661393783987,
+            -0.007228385657072067,
+            0.02132258005440235,
+            0.028203818947076797,
+            0.029931843280792236,
+            0.002574678510427475,
+            -0.02629064954817295,
+            0.02312774769961834,
+            -0.009681562893092632,
+            -0.009573561139404774,
+            0.009156984277069569,
+            0.0005901510012336075,
+            -0.009411558508872986,
+            -0.01896197721362114,
+            -0.002981612691655755,
+            0.005230357870459557,
+            0.019162550568580627,
+            0.00224681687541306,
+            -0.0018312039319425821,
+            -0.022880887612700462,
+            0.003407832933589816,
+            0.02727809175848961,
+            0.0030780425295233727,
+            -0.009789563715457916,
+            -0.008339257910847664,
+            0.020813431590795517,
+            -0.003297902876511216,
+            0.004624778404831886,
+            0.003791623981669545,
+            -0.008339257910847664,
+            -0.003342260606586933,
+            -0.004941068589687347,
+            0.0026518225204199553,
+            -0.01805167831480503,
+            0.009758706204593182,
+            0.01116272620856762,
+            -0.010499288327991962,
+            0.004215915687382221,
+            -0.012705604545772076,
+            0.006210085935890675,
+            -0.0004380328464321792,
+            0.0077761076390743256,
+            -0.0015698788920417428,
+            -0.012289026752114296,
+            -0.02971583977341652,
+            -0.01185702160000801,
+            -0.01817510835826397,
+            -0.014634202234447002,
+            0.0007473317673429847,
+            0.02784895710647106,
+            -0.004266059026122093,
+            0.007563961669802666,
+            0.001351947314105928,
+            -0.009974709711968899,
+            0.012188740074634552,
+            -0.020026562735438347,
+            0.01480391900986433,
+            -0.013639045879244804,
+            0.013091323897242546,
+            0.015451927669346333,
+            0.01797453500330448,
+            -0.009396130219101906,
+            0.0004013894940726459,
+            -0.0020983146969228983,
+            -0.008100111968815327,
+            -0.00537307420745492,
+            -0.012813606299459934,
+            -0.008053826168179512,
+            -0.017635101452469826,
+            0.015328497625887394,
+            0.024099761620163918,
+            0.0009990137768909335,
+            0.01728023961186409,
+            0.0008259221212938428,
+            -1.7749129256117158e-05,
+            -0.01111643947660923,
+            -0.004092485178261995,
+            0.014348769560456276,
+            0.005068355705589056,
+            -0.025210633873939514,
+            -0.0036334788892418146,
+            0.01652422919869423,
+            0.006102084647864103,
+            -0.01393219269812107,
+            0.013469329103827477,
+            -0.003799338359385729,
+            -0.0021812445484101772,
+            -0.01355418749153614,
+            -0.0009030660730786622,
+            0.007675820495933294,
+            0.0052727870643138885,
+            -0.021122006699442863,
+            -0.008786693215370178,
+            0.0032496878411620855,
+            0.012435601092875004,
+            -0.004258344415575266,
+            0.00309347128495574,
+            0.01885397545993328,
+            -0.00536150299012661,
+            0.03465305268764496,
+            -0.012520458549261093,
+            -0.014503058046102524,
+            0.0020713144913315773,
+            0.020134564489126205,
+            -0.0062563722021877766,
+            -0.0007400995236821473,
+            0.011340157128870487,
+            0.0005506147863343358,
+            -0.008161826990544796,
+            0.02974669821560383,
+            0.006040369160473347,
+            0.01738824136555195,
+            -0.0010877293534576893,
+            0.014595630578696728,
+            -0.001114729675464332,
+            0.0002372175658820197,
+            -0.0009802100248634815,
+            -0.01035271491855383,
+            -0.005068355705589056,
+            -0.020026562735438347,
+            -0.006009511649608612,
+            -0.020273422822356224,
+            0.029160402715206146,
+            0.012389314360916615,
+            0.007494532503187656,
+            -0.02638322301208973,
+            0.002289246069267392,
+            0.006441517733037472,
+            -0.007170527707785368,
+            -0.006900524254888296,
+            -0.012204169295728207,
+            -0.022032305598258972,
+            0.009318985976278782,
+            0.007259243167936802,
+            0.010568717494606972,
+            0.004250630270689726,
+            -0.010182998143136501,
+            0.0032034015748649836,
+            -0.018529970198869705,
+            0.012705604545772076,
+            0.0024801772087812424,
+            0.003133971942588687,
+            0.0016942734364420176,
+            -0.0034483333583921194,
+            0.014665059745311737,
+            -3.730631942744367e-05,
+            0.011039295233786106,
+            0.014094195328652859,
+            0.01111643947660923,
+            0.007976681925356388,
+            0.004401060752570629,
+            0.010576431639492512,
+            -0.00816954206675291,
+            -0.030734140425920486,
+            -0.0264449380338192,
+            0.004080913495272398,
+            -0.01110872533172369,
+            -0.009627562016248703,
+            0.007984396070241928,
+            0.011139582842588425,
+            0.01069214753806591,
+            -0.016323653981089592,
+            -0.005666221491992474,
+            -0.0043200599029660225,
+            -2.2917169189895503e-05,
+            -0.011787591502070427,
+            -0.0007203313871286809,
+            0.012157882563769817,
+            0.002040456747636199,
+            0.006491661071777344,
+            0.01894654706120491,
+            -0.00906441081315279,
+            -0.007826250977814198,
+            0.01567564532160759,
+            -0.0074328170157969,
+            -0.019625414162874222,
+            -0.008910123258829117,
+            0.016215652227401733,
+            -0.008215827867388725,
+            0.011286156252026558,
+            0.009565846994519234,
+            0.010491574183106422,
+            -0.016632230952382088,
+            -0.007999825291335583,
+            0.02804953046143055,
+            0.03230787441134453,
+            -0.010807863436639309,
+            -0.002273817081004381,
+            0.002725109225139022,
+            -0.012127025052905083,
+            0.021122006699442863,
+            0.00860926229506731,
+            -0.010113568976521492,
+            0.016987092792987823,
+            0.0026903944090008736,
+            0.008138683624565601,
+            -0.0036238357424736023,
+            0.0012304455740377307,
+            -0.017604243010282516,
+            -0.018375683575868607,
+            -0.00948098860681057,
+            0.009164698421955109,
+            0.006742379162460566,
+            0.0073749590665102005,
+            8.919765969039872e-05,
+            0.011093296110630035,
+            -0.011332442983984947,
+            0.005604506004601717,
+            0.0008119398262351751,
+            0.027740955352783203,
+            -0.029885556548833847,
+            0.019903132691979408,
+            -0.003267045132815838,
+            -0.0018283110111951828,
+            -0.013585045002400875,
+            -0.012844463810324669,
+            -0.020828859880566597,
+            -0.011209012009203434,
+            0.014973635785281658,
+            0.016076793894171715,
+            -0.0005472397315315902,
+            -0.006908238399773836,
+            -0.002619036240503192,
+            0.0057973661459982395,
+            -0.006861952133476734,
+            0.002761752577498555,
+            -0.0017646672204136848,
+            0.0014956278027966619,
+            -0.0020751715637743473,
+            -0.015559929423034191,
+            0.007444388698786497,
+            -0.003652764717116952,
+            -0.0015669859712943435,
+            -0.018761401996016502,
+            0.03604163974523544,
+            -0.013060466386377811,
+            0.00904126837849617,
+            0.014155910350382328,
+            0.0018128822557628155,
+            -0.011471301317214966,
+            -0.013777905143797398,
+            0.011216727085411549,
+            -0.02149229682981968,
+            -0.021955160424113274,
+            -0.004096342250704765,
+            0.0223563089966774,
+            -0.015760503709316254,
+            -0.004046198911964893,
+            0.019532842561602592,
+            -0.006271800957620144,
+            -0.009658419527113438,
+            -0.022680314257740974,
+            -0.016231082379817963,
+            -1.1903066479135305e-05,
+            -0.016801945865154266,
+            -0.017912819981575012,
+            -0.017079664394259453,
+            -0.007864823564887047,
+            -0.01970255747437477,
+            -0.0006914024124853313,
+            0.0033924041781574488,
+            -0.01805167831480503,
+            -0.022695742547512054,
+            0.01275189034640789,
+            0.00818497035652399,
+            -0.0026306079234927893,
+            0.008524403907358646,
+            0.00596322538331151,
+            0.0027231804560869932,
+            -2.3640393919777125e-05,
+            0.007035525981336832,
+            0.02892897091805935,
+            0.017017949372529984,
+            0.006599662825465202,
+            0.017002521082758904,
+            -0.0062988013960421085,
+            -0.01029099989682436,
+            -0.010321857407689095,
+            -0.004169628955423832,
+            0.010445287451148033,
+            -0.019964847713708878,
+            0.033357031643390656,
+            -0.006202371325343847,
+            -0.018730545416474342,
+            -0.006410660222172737,
+            0.00987442210316658,
+            -0.010476144962012768,
+            -0.019532842561602592,
+            -0.007922681048512459,
+            -0.006885095499455929,
+            0.016092222183942795,
+            -0.0032901884987950325,
+            0.00898726750165224,
+            0.010113568976521492,
+            -0.015421070158481598,
+            0.010823292657732964,
+            -0.006240943446755409,
+            -0.014271626248955727,
+            0.025195205584168434,
+            0.022448882460594177,
+            -0.005064498633146286,
+            -0.007826250977814198,
+            -0.002401104662567377,
+            0.021245436742901802,
+            0.008940980769693851,
+            -0.0003459422732703388,
+            0.009172412566840649,
+            0.005608363542705774,
+            0.003905411111190915,
+            0.0024608911480754614,
+            0.001322053954936564,
+            0.011224441230297089,
+            0.03311017155647278,
+            0.030147844925522804,
+            -0.008046111091971397,
+            -0.002551535377278924,
+            0.01528992597013712,
+            0.00899498164653778,
+            0.028280962258577347,
+            -0.0031455436255782843,
+            -0.035023342818021774,
+            0.001998027553781867,
+            -0.008447259664535522,
+            0.014703632332384586,
+            0.0006648842245340347,
+            0.015876220539212227,
+            0.0017540600383654237,
+            -0.0014879134250804782,
+            0.018514541909098625,
+            0.019316839054226875,
+            0.00617922842502594,
+            -0.017635101452469826,
+            -0.01312989555299282,
+            0.008740406483411789,
+            0.006514804437756538,
+            -0.005685507319867611,
+            -0.012327599339187145,
+            -0.00015826558228582144,
+            0.029808413237333298,
+            -0.010198426432907581,
+            -0.017449956387281418,
+            0.01242017187178135,
+            0.011525302194058895,
+            0.007787679322063923,
+            -0.025951215997338295,
+            0.004169628955423832,
+            0.008354687131941319,
+            0.014819348230957985,
+            -0.00899498164653778,
+            -0.015004493296146393,
+            -0.000610401330050081,
+            0.011193583719432354,
+            -0.016261938959360123,
+            0.001589164836332202,
+            0.02218659222126007,
+            -0.02053571306169033,
+            0.03122786059975624,
+            -0.010059568099677563,
+            0.017804818227887154,
+            0.003477262333035469,
+            0.020088277757167816,
+            0.006495518609881401,
+            0.0012304455740377307,
+            0.004161914810538292,
+            0.013885905966162682,
+            -0.003245830535888672,
+            0.011795305646955967,
+            -0.01485020574182272,
+            -0.004536062944680452,
+            0.0003758355451282114,
+            -0.0021581011824309826,
+            -0.0024512482341378927,
+            0.008215827867388725,
+            0.02226373739540577,
+            -0.01581450365483761,
+            0.0015843433793634176,
+            -0.005284358747303486,
+            0.006800237111747265,
+            0.023312894627451897,
+            -0.011903307400643826,
+            -0.009519560262560844,
+            0.012890749610960484,
+            0.019085407257080078,
+            -0.0053537883795797825,
+            -0.027478665113449097,
+            -0.0015043065650388598,
+            0.014680488966405392,
+            0.017789389938116074,
+            -0.0031821869779378176,
+            0.036658793687820435,
+            -0.013060466386377811,
+            -0.024038046598434448,
+            -0.008856122381985188,
+            0.03789309412240982,
+            0.004605492111295462,
+            -0.005095356144011021,
+            0.0015264854300767183,
+            -0.010152140632271767,
+            0.008393258787691593,
+            0.015066208317875862,
+            -0.00662280572578311,
+            -0.01976427435874939,
+            0.01726480945944786,
+            -0.0153053542599082,
+            -0.009835850447416306,
+            -0.002852396573871374,
+            -0.005477218423038721,
+            -0.009450131095945835,
+            0.012481886893510818,
+            -0.01449534296989441,
+            0.0003293081244919449,
+            0.000391987559851259,
+            0.0029063974507153034,
+            0.004636350087821484,
+            0.013855048455297947,
+            -0.010831006802618504,
+            0.007679677568376064,
+            0.016061365604400635,
+            -0.0013828048249706626,
+            0.016724802553653717,
+            0.007131956052035093,
+            -0.007093383930623531,
+            -0.013446185737848282,
+            0.03212273120880127,
+            -0.0024531767703592777,
+            0.016076793894171715,
+            -0.011401872150599957,
+            -0.008215827867388725,
+            -0.02559635415673256,
+            -0.010576431639492512,
+            0.028358107432723045,
+            0.004952639807015657,
+            -0.024099761620163918,
+            0.003521620063111186,
+            0.0347764827311039,
+            0.00823125708848238,
+            0.006507089827209711,
+            0.008431831374764442,
+            0.009542703628540039,
+            0.005137785337865353,
+            -0.0022873173002153635,
+            -0.0016113437013700604,
+            -0.00553121929988265,
+            0.009581275284290314,
+            0.0021214578300714493,
+            -0.005130071192979813,
+            -0.013099038042128086,
+            -0.01360818836838007,
+            -0.028280962258577347,
+            0.005928510800004005,
+            -0.03517762944102287,
+            0.015876220539212227,
+            -0.02136886678636074,
+            0.02385290153324604,
+            -0.004871638957411051,
+            -0.005616077687591314,
+            -0.020211707800626755,
+            0.008539832197129726,
+            -0.002129172207787633,
+            0.006352802272886038,
+            0.002316246274858713,
+            -0.004316202364861965,
+            0.012204169295728207,
+            -0.012188740074634552,
+            -0.024254050105810165,
+            0.028527824208140373,
+            -0.006067369598895311,
+            -0.02480948530137539,
+            0.0025843214243650436,
+            0.003236187621951103,
+            0.006499375682324171,
+            0.005635363515466452,
+            -0.03227701783180237,
+            0.0025669641327112913,
+            -0.003905411111190915,
+            0.009234127588570118,
+            0.006171513814479113,
+            -0.028666682541370392,
+            0.023544326424598694,
+            0.008917837403714657,
+            -0.0013567687710747123,
+            -0.007891823537647724,
+            -0.018715115264058113,
+            0.00014283679774962366,
+            0.0042390585877001286,
+            0.011687304824590683,
+            0.004354774486273527,
+            0.0015592715935781598,
+            -0.011093296110630035,
+            0.0029642554000020027,
+            -0.011216727085411549,
+            0.016647659242153168,
+            -0.020134564489126205,
+            0.011633303947746754,
+            -0.005018212366849184,
+            -0.005110784899443388,
+            -0.0036373359616845846,
+            0.007899537682533264,
+            -0.01404019445180893,
+            0.02872839756309986,
+            0.005133928265422583,
+            -0.014533915556967258,
+            -0.012065310031175613,
+            0.012026737444102764,
+            0.0069930967874825,
+            0.011031581088900566,
+            0.0014358413172885776,
+            -0.005677792709320784,
+            0.0035794780123978853,
+            0.0027771813329309225,
+            0.01231217011809349,
+            -0.0030298277270048857,
+            0.0188385471701622,
+            0.011401872150599957,
+            -0.0030626137740910053,
+            -0.0058976528234779835,
+            -0.00829297211021185,
+            2.452935768815223e-05,
+            -0.01715680956840515,
+            0.011371014639735222,
+            0.0068349516950547695,
+            -0.010460715740919113,
+            0.002080957405269146,
+            0.005006640683859587,
+            -0.001408840878866613,
+            0.013361327350139618,
+            -0.001426198286935687,
+            0.0005809901631437242,
+            1.5458919733646326e-05,
+            0.007633391302078962,
+            -0.010576431639492512,
+            -0.024485481902956963,
+            -0.031042715534567833,
+            0.004034627228975296,
+            -0.0005313287838362157,
+            0.015081637538969517,
+            0.021260865032672882,
+            -0.015374784357845783,
+            -0.00021925750479567796,
+            0.005943939555436373,
+            -0.011332442983984947,
+            -0.015012207441031933,
+            -0.017727673053741455,
+            0.013214753940701485,
+            -0.016740230843424797,
+            0.002659536898136139,
+            0.019548270851373672,
+            -0.025226064026355743,
+            0.012975607998669147,
+            -0.034251902252435684,
+            -0.007228385657072067,
+            -0.011039295233786106,
+            -0.014209911227226257,
+            0.005045212805271149,
+            0.016894519329071045,
+            -0.014842490665614605,
+            -0.0005539897829294205,
+            -0.004528348334133625,
+            0.0012420171406120062,
+            -0.022896315902471542,
+            -0.02136886678636074,
+            0.016123080626130104,
+            -0.01888483203947544,
+            -0.02135343849658966,
+            -0.009558131918311119,
+            -0.0014609131030738354,
+            -0.001360625959932804,
+            -0.014711346477270126,
+            0.019594557583332062,
+            0.0005718293250538409,
+            0.012466458603739738,
+            6.394508091034368e-05,
+            0.015845362097024918,
+            -0.00558136310428381,
+            -0.006865809205919504,
+            -0.006703807041049004,
+            0.0036643364001065493,
+            -0.011201297864317894,
+            0.0029873985331505537,
+            -0.0005356681067496538,
+            0.0015168423997238278,
+            0.0024724628310650587,
+            -0.004358631558716297,
+            0.008061540313065052,
+            0.0003941572504118085,
+            0.016879091039299965,
+            -0.0017627386841922998,
+            0.008470403030514717,
+            0.014541629701852798,
+            0.0030433277133852243,
+            -0.013585045002400875,
+            -0.0038282673340290785,
+            0.011795305646955967,
+            0.002084814477711916,
+            0.006896666716784239,
+            0.022742029279470444,
+            -0.0031436148565262556,
+            0.014163624495267868,
+            0.0012092309771105647,
+            -8.505418918502983e-06,
+            0.005118499509990215,
+            -0.008979552425444126,
+            0.004246773198246956,
+            -0.0013500186614692211,
+            -0.0006889916840009391,
+            -0.002725109225139022,
+            0.014749918133020401,
+            0.001008656807243824,
+            -0.013392184861004353,
+            -0.010221569798886776,
+            -0.0018890618812292814,
+            0.03203015774488449,
+            0.00025168206775560975,
+            0.010807863436639309,
+            -0.02882097102701664,
+            -0.00742895994335413,
+            0.009380700998008251,
+            -0.025148918852210045,
+            -0.020921431481838226,
+            0.005650792270898819,
+            0.012682461179792881,
+            -0.003945911768823862,
+            -0.0004481579817365855,
+            0.04890924692153931,
+            0.020165421068668365,
+            -0.0035987640731036663,
+            -0.004802209325134754,
+            -0.0063026584684848785,
+            -0.0026884658727794886,
+            0.011617874726653099,
+            -0.012204169295728207,
+            -0.006148370914161205,
+            -0.0069275242276489735,
+            0.004474347457289696,
+            0.007830108515918255,
+            -0.013662189245223999,
+            -0.006044226698577404,
+            0.002401104662567377,
+            -0.011147296987473965,
+            -0.013723904266953468,
+            0.006588091142475605,
+            -0.0063412305898964405,
+            3.5859869967680424e-05,
+            0.006453089416027069,
+            -0.012250455096364021,
+            0.01155615970492363,
+            0.004983497317880392,
+            0.019100835546851158,
+            -0.01653965748846531,
+            -0.001184159191325307,
+            0.00904126837849617,
+            -0.015953363850712776,
+            0.0031841155141592026,
+            -0.013145324774086475,
+            0.006167656742036343,
+            -0.0012063381727784872,
+            0.0058513665571808815,
+            -0.002252602484077215,
+            -0.00450520496815443,
+            0.007189814001321793,
+            -0.03320274502038956,
+            -0.009812707081437111,
+            0.013685332611203194,
+            -0.025874072685837746,
+            0.010483859106898308,
+            0.004748208448290825,
+            0.009511846117675304,
+            -0.007961252704262733,
+            0.011949594132602215,
+            0.00112340843770653,
+            -0.015374784357845783,
+            0.020890574902296066,
+            0.007675820495933294,
+            0.013762475922703743,
+            0.005708650220185518,
+            0.005928510800004005,
+            0.009990138001739979,
+            0.013515615835785866,
+            -0.012119310908019543,
+            -0.01636994071304798,
+            0.011455873027443886,
+            0.003450262127444148,
+            -5.23433591297362e-05,
+            -0.01230445597320795,
+            -0.0148656340315938,
+            0.01066900510340929,
+            0.01279046293348074,
+            0.012921607121825218,
+            0.01275189034640789,
+            0.006981525104492903,
+            0.015343926846981049,
+            -0.02062828466296196,
+            -0.012767319567501545,
+            -0.025904929265379906,
+            0.014965921640396118,
+            -0.0026576081290841103,
+            -0.001909312093630433,
+            -0.017496241256594658,
+            -0.005072213243693113,
+            -0.02237173728644848,
+            0.007344101555645466,
+            0.016123080626130104,
+            0.0009990137768909335,
+            -0.0024551053065806627,
+            0.02389918826520443,
+            -0.013739332556724548,
+            0.04357088729739189,
+            -0.007521532475948334,
+            -0.019193409010767937,
+            -0.010831006802618504,
+            -0.004593920893967152,
+            0.013893621042370796,
+            0.005338359624147415,
+            -0.00617922842502594,
+            0.003953625913709402,
+            0.020149992778897285,
+            -0.001371233258396387,
+            -0.002293103141710162,
+            -0.011965022422373295,
+            0.01441048551350832,
+            -0.006534090265631676,
+            0.008539832197129726,
+            -0.011085581965744495,
+            0.019471125677227974,
+            0.00515707116574049,
+            0.010699862614274025,
+            0.009588989429175854,
+            -0.014348769560456276,
+            -0.014603344723582268,
+            -0.00595936831086874,
+            0.00558136310428381,
+            -0.019316839054226875,
+            -0.009095268324017525,
+            0.03465305268764496,
+            -7.075544272083789e-05,
+            -0.006676806602627039,
+            -0.005114641971886158,
+            0.011471301317214966,
+            -0.009288128465414047,
+            -0.001129194162786007,
+            -0.005716364830732346,
+            -0.010877293534576893,
+            -0.001112801139242947,
+            -0.005785794463008642,
+            -0.006526376120746136,
+            -0.013261040672659874,
+            -0.0058976528234779835,
+            0.0014483771519735456,
+            -0.007668105885386467,
+            -0.0006889916840009391,
+            0.004532205406576395,
+            0.012512744404375553,
+            0.0110624385997653,
+            -0.00276753818616271,
+            0.006865809205919504,
+            -0.0012603388167917728,
+            0.018637971952557564,
+            0.0135464733466506,
+            -0.013739332556724548,
+            -0.019378554075956345,
+            -0.0028793970122933388,
+            0.00617537135258317,
+            0.014611058868467808,
+            0.012890749610960484,
+            -0.006032655015587807,
+            -0.0003430493816267699,
+            -0.024485481902956963,
+            -0.005176357459276915,
+            0.008547547273337841,
+            -0.01273646205663681,
+            -0.007579390425235033,
+            -0.0029951129108667374,
+            -0.0010154069168493152,
+            -0.004485919140279293,
+            -0.016925377771258354,
+            -0.000276512757409364,
+            0.010877293534576893,
+            -0.006483946926891804,
+            0.017110522836446762,
+            0.018622543662786484,
+            -0.014973635785281658,
+            -0.007251529023051262,
+            0.0009271735325455666,
+            -0.03236959129571915,
+            0.0024146048817783594,
+            -0.000498060486279428,
+            0.02067457139492035,
+            0.0053152162581682205,
+            0.0005901510012336075,
+            0.002809967380017042,
+            -0.006460803560912609,
+            0.002123386599123478,
+            0.007309386972337961,
+            -0.008840694092214108,
+            -0.01073072012513876,
+            -0.010468430817127228,
+            -0.011039295233786106,
+            -0.009735562838613987,
+            -0.0025631068274378777,
+            -0.022572312504053116,
+            0.029931843280792236,
+            0.011795305646955967,
+            0.003629621583968401,
+            -0.01069214753806591,
+            -0.0010134782642126083,
+            0.018637971952557564,
+            -0.0035235488321632147,
+            0.023729471489787102,
+            0.013006465509533882,
+            -0.011826164089143276,
+            -0.017125951126217842,
+            0.023467181250452995,
+            -0.020983148366212845,
+            0.0062525151297450066,
+            -0.003961340524256229,
+            0.0019044906366616488,
+            0.003965197596698999,
+            -0.011417300440371037,
+            0.008022967725992203,
+            -0.016940806061029434,
+            -0.00298354122787714,
+            -0.00433934573084116,
+            0.006217800546437502,
+            -0.002638322301208973,
+            0.026768941432237625,
+            -0.0019739202689379454,
+            -0.004042341839522123,
+            0.013099038042128086,
+            0.039127398282289505,
+            0.029283834621310234,
+            0.018761401996016502,
+            -0.006665234919637442,
+            -0.006194657180458307,
+            -0.012705604545772076,
+            0.004678778816014528,
+            -0.017095094546675682,
+            -0.003265116596594453,
+            0.004663350060582161,
+            0.0010684434091672301,
+            -0.009704705327749252,
+            -0.0003309956518933177,
+            -0.019162550568580627,
+            0.0239609032869339,
+            -0.01236617099493742,
+            0.00011173815437359735,
+            -0.010962151922285557,
+            -0.01404790859669447,
+            -0.018468255177140236,
+            0.0023876044433563948,
+            -0.0072823865339159966,
+            -0.01891569048166275,
+            0.0024782486725598574,
+            -0.019579127430915833,
+            -0.03690565377473831,
+            0.021986018866300583,
+            0.0029970414470881224,
+            0.011664161458611488,
+            0.025025488808751106,
+            -0.011371014639735222,
+            0.01578364707529545,
+            0.02713923342525959,
+            0.017002521082758904,
+            0.004566920455545187,
+            0.011756733991205692,
+            0.01023699901998043,
+            0.012597602792084217,
+            -0.03227701783180237,
+            -0.00698923971503973,
+            0.015027636662125587,
+            0.009558131918311119,
+            -0.008323829621076584,
+            -0.010861864313483238,
+            0.0030548993963748217,
+            -0.02798781543970108,
+            0.006476232316344976,
+            -0.022495169192552567,
+            -0.007444388698786497,
+            0.012659317813813686,
+            -0.009781849570572376,
+            0.0008239935268647969,
+            0.004875496029853821,
+            0.005415503401309252,
+            -0.011702733114361763,
+            -0.01229674182832241,
+            -0.007714392617344856,
+            0.00742510287091136,
+            0.01316846814006567,
+            -0.004875496029853821,
+            -0.0001758640428306535,
+            -0.014950492419302464,
+            0.003351903520524502,
+            0.011625589802861214,
+            0.015482786111533642,
+            0.015729647129774094,
+            -0.00287361117079854,
+            -0.0024705342948436737,
+            0.0181133933365345,
+            0.00044237219844944775,
+            -0.007201385218650103,
+            0.008532118052244186,
+            0.010846436023712158,
+            0.000777707202360034,
+            0.017449956387281418,
+            0.009581275284290314,
+            -0.01657051593065262,
+            0.004809923470020294,
+            0.004995069000869989,
+            0.03292502835392952,
+            0.010105853900313377,
+            0.00742895994335413,
+            -0.0031243290286511183,
+            -0.0016412369441241026,
+            -0.009596704505383968,
+            0.012659317813813686,
+            0.017959104850888252,
+            -0.0068388087674975395,
+            0.004686493426561356,
+            0.02556549571454525,
+            0.004177343565970659,
+            0.044866908341646194,
+            0.004852352663874626,
+            0.0035274059046059847,
+            -0.006048083771020174,
+            -0.009018125012516975,
+            -0.0015804861905053258,
+            -0.0018958119908347726,
+            -0.004077056422829628,
+            -0.006206228863447905,
+            -0.0016711303032934666,
+            0.019316839054226875,
+            -0.004088628105819225,
+            0.00595936831086874,
+            0.009897565469145775,
+            -0.00216967286542058,
+            -0.006931381765753031,
+            0.014503058046102524,
+            -0.01155615970492363,
+            0.010522431693971157,
+            -0.021569442003965378,
+            -0.002233316656202078,
+            0.00411562854424119,
+            0.014148196205496788,
+            -0.010977580212056637,
+            0.00021202526113484055,
+            -0.003330688923597336,
+            0.015112495049834251,
+            0.004532205406576395,
+            -0.0032940455712378025,
+            0.0011947664897888899,
+            -0.01903912052512169,
+            -0.008084683679044247,
+            0.03230787441134453,
+            0.004007626790553331,
+            0.015390212647616863,
+            -0.005508076399564743,
+            0.008200399577617645,
+            0.005866795312613249,
+            0.013824190944433212,
+            -0.02794152870774269,
+            0.011664161458611488,
+            0.020196279510855675,
+            0.0013345899060368538,
+            0.00780696514993906,
+            -0.001996099017560482,
+            -0.01566021703183651,
+            0.01638536900281906,
+            -0.026799799874424934,
+            0.0049024964682757854,
+            -0.014287054538726807,
+            0.012011309154331684,
+            0.004212058149278164,
+            -0.004146486055105925,
+            0.007992110215127468,
+            0.006966096349060535,
+            0.017743103206157684,
+            0.025750642642378807,
+            0.026568368077278137,
+            0.0002040697872871533,
+            0.0148656340315938,
+            -0.007031668908894062,
+            -3.0887702450854704e-05,
+            -0.026059217751026154,
+            -0.01715680956840515,
+            0.003988340962678194,
+            -0.0363810732960701,
+            -7.202108827186748e-05,
+            -0.008030682802200317,
+            -0.018576256930828094,
+            0.041688576340675354,
+            -0.006117513403296471,
+            -0.013569616712629795,
+            0.007197528146207333,
+            -0.005724078975617886,
+            0.004836923908442259,
+            -0.009550417773425579,
+            0.030194131657481194,
+            -0.002079028869047761,
+            -0.017604243010282516,
+            -0.03314103186130524,
+            -0.00574336526915431,
+            -0.025766070932149887,
+            -0.00450520496815443,
+            -0.01633908413350582,
+            0.014109623618423939,
+            -0.020828859880566597,
+            0.002302746055647731,
+            0.0037974095903337,
+            0.0020539569668471813,
+            0.01965627260506153,
+            0.0014638060238212347,
+            -0.007386530749499798,
+            0.006267943885177374,
+            -0.005106927827000618,
+            -0.012651603668928146,
+            -0.025087203830480576,
+            0.0011369085405021906,
+            0.00449363375082612,
+            0.00041971117025241256,
+            0.013808762654662132,
+            -0.0037279801908880472,
+            0.022495169192552567,
+            -0.016323653981089592,
+            0.012775033712387085,
+            -0.01517421007156372,
+            0.0007073133601807058,
+            0.004875496029853821,
+            0.0006644020322710276,
+            -7.636043301317841e-05,
+            -0.00536150299012661,
+            -0.02558092586696148,
+            0.007725963834673166,
+            0.021708300337195396,
+            -0.0025708212051540613,
+            0.0063450876623392105,
+            -0.006213943008333445,
+            0.000337986828526482,
+            0.0035061913076788187,
+            0.008138683624565601,
+            -0.015382498502731323,
+            0.0012535888236016035,
+            0.03369646519422531,
+            0.013855048455297947,
+            -0.007984396070241928,
+            -0.01230445597320795,
+            0.0021388153545558453,
+            0.013515615835785866,
+            -0.016076793894171715,
+            0.004586206283420324,
+            0.01279817707836628,
+            0.024315765127539635,
+            0.0020944576244801283,
+            -0.00553121929988265,
+            -0.028219247236847878,
+            0.021230008453130722,
+            -0.011810734868049622,
+            0.015205067582428455,
+            0.0018620614428073168,
+            0.028358107432723045,
+            0.002680751495063305,
+            0.0014271625550463796,
+            0.0010935150785371661,
+            0.03351132199168205,
+            0.00942698772996664,
+            -0.023590611293911934,
+            -0.0024281051009893417,
+            -0.0009517631842754781,
+            0.003047185018658638,
+            0.029268404468894005,
+            0.03153643757104874,
+            -0.004755923058837652,
+            -0.039281684905290604,
+            0.008802121505141258,
+            -0.024115189909934998,
+            0.002595893107354641,
+            0.011000723578035831,
+            0.00617922842502594,
+            -0.01360047422349453,
+            -0.0012748033041134477,
+            0.0012034452520310879,
+            0.0038514104671776295,
+            0.014950492419302464,
+            -0.011448157951235771,
+            -0.002935326425358653,
+            0.009735562838613987,
+            0.0032130444888025522,
+            -0.012543601915240288,
+            0.017557958140969276,
+            0.0020867432467639446,
+            -0.003022113349288702,
+            -0.004208201076835394,
+            0.03357303515076637,
+            0.008532118052244186,
+            -0.016308225691318512,
+            -0.0054039317183196545,
+            -0.026028359308838844,
+            -0.009234127588570118,
+            -0.013531044125556946,
+            -0.006904381327331066,
+            -0.022772885859012604,
+            -0.007679677568376064,
+            -0.019347695633769035,
+            -0.007513818331062794,
+            -0.02895982936024666,
+            0.006117513403296471,
+            -0.0047752088867127895,
+            0.03375818207859993,
+            -0.0035505490377545357,
+            -0.031582724303007126,
+            0.01741909794509411,
+            -0.0035717636346817017,
+            0.019594557583332062,
+            -0.010777005925774574,
+            0.009156984277069569,
+            0.01069214753806591,
+            0.01312989555299282,
+            0.02152315527200699,
+            0.00026566439191810787,
+            0.006503232754766941,
+            0.0030857569072395563,
+            0.025010060518980026,
+            -0.0033576893620193005,
+            -0.007818536832928658,
+            0.002827324904501438,
+            -0.011571588926017284,
+            0.019255124032497406,
+            0.0030317562632262707,
+            0.014503058046102524,
+            -0.022526025772094727,
+            0.022418024018406868,
+            -0.004570777527987957,
+            0.013801048509776592,
+            0.0012468387139961123,
+            0.005330645013600588,
+            -0.012582174502313137,
+            -0.007351816166192293,
+            -0.004269916098564863,
+            -0.0035949070006608963,
+            0.015220495872199535,
+            -0.01361590251326561,
+            0.005130071192979813,
+            -0.03523934632539749,
+            0.027015801519155502,
+            0.008601547218859196,
+            0.003278616815805435,
+            -0.0017087379237636924,
+            0.010121283121407032,
+            0.005924653261899948,
+            0.01235074270516634,
+            0.005727936513721943,
+            -0.013099038042128086,
+            0.017465384677052498,
+            0.001782024628482759,
+            0.008370115421712399,
+            0.0036874795332551003,
+            -0.016771089285612106,
+            -0.010645861737430096,
+            0.01324561145156622,
+            0.01819053664803505,
+            0.003342260606586933,
+            0.0013596616918221116,
+            -0.03145929425954819,
+            0.030086129903793335,
+            0.0017289882525801659,
+            0.008562975563108921,
+            -0.007891823537647724,
+            0.02894440107047558,
+            -0.007444388698786497,
+            0.010622718371450901,
+            0.022603169083595276,
+            0.01567564532160759,
+            0.007128098513931036,
+            -0.008416402153670788,
+            0.007668105885386467,
+            0.027632953599095345,
+            0.007976681925356388,
+            0.005573648493736982,
+            0.018298538401722908,
+            0.009156984277069569,
+            0.012049880810081959,
+            -0.01576821878552437,
+            -0.04017655551433563,
+            0.001536128344014287,
+            -0.015444213524460793,
+            -0.018668830394744873,
+            0.011949594132602215,
+            0.011949594132602215,
+            0.004917925223708153,
+            0.02477862872183323,
+            0.015652501955628395,
+            -0.006858095061033964,
+            0.0018466326873749495,
+            -0.013006465509533882,
+            -0.019903132691979408,
+            0.005770365707576275,
+            -0.003265116596594453,
+            -0.023220321163535118,
+            -0.001962348585948348,
+            -0.011031581088900566,
+            1.3439917893265374e-05,
+            -0.006024940405040979,
+            -0.0069699534215033054,
+            0.04431147128343582,
+            0.008200399577617645,
+            -0.010321857407689095,
+            -0.014387342147529125,
+            0.013577330857515335,
+            -0.011841592378914356,
+            0.029175832867622375,
+            -0.0034155473113059998,
+            -0.001836989657022059,
+            -0.025982074439525604,
+            -0.0010607289150357246,
+            -0.009465559385716915,
+            -0.005496504716575146,
+            0.005230357870459557,
+            -0.01198045164346695,
+            -0.009789563715457916,
+            0.0028813255485147238,
+            -0.008709548972547054,
+            0.013307327404618263,
+            -0.009087554179131985,
+            0.006599662825465202,
+            -0.01272874791175127,
+            -0.0007323851459659636,
+            0.018684258684515953,
+            -4.550286030280404e-05,
+            0.016308225691318512,
+            -0.015104779973626137,
+            -0.0005366324330680072,
+            -0.00022648973390460014,
+            -0.03323360159993172,
+            0.016693945974111557,
+            0.012273598462343216,
+            0.0001559753727633506,
+            -0.02312774769961834,
+            0.002248745411634445,
+            0.024207763373851776,
+            0.0014117337996140122,
+            0.004354774486273527,
+            0.025040917098522186,
+            -0.005091499071568251,
+            0.0074598174542188644,
+            0.013446185737848282,
+            0.007922681048512459,
+            -0.020026562735438347,
+            -0.0020925288554280996,
+            0.009226413443684578,
+            0.007668105885386467,
+            0.007710535079240799,
+            -0.0008996910182759166,
+            0.0165859442204237,
+            0.0067848083563148975,
+            -0.019285980612039566,
+            -0.017696816474199295,
+            0.015120209194719791,
+            0.010098139755427837,
+            -0.0009392272913828492,
+            0.004393346607685089,
+            0.005731793586164713,
+            0.011394158005714417,
+            -0.011695018969476223,
+            -0.0013037322787567973,
+            -0.02298888936638832,
+            0.021662013605237007,
+            0.016663087531924248,
+            -0.016045935451984406,
+            0.034961625933647156,
+            -0.004250630270689726,
+            -0.007363387383520603,
+            -0.019239693880081177,
+            -0.02541120909154415,
+            -0.0021966733038425446,
+            0.015359355136752129,
+            -0.004458918701857328,
+            0.002956541022285819,
+            0.007961252704262733,
+            -0.032523877918720245,
+            -0.020365996286273003,
+            -0.011671875603497028,
+            0.0005154178361408412,
+            0.0052612158469855785,
+            0.0027347521390765905,
+            0.009704705327749252,
+            0.005018212366849184,
+            0.01116272620856762,
+            0.005862938240170479,
+            0.007706678006798029,
+            -0.017959104850888252,
+            -0.02135343849658966,
+            0.008161826990544796,
+            0.00027048587799072266,
+            -0.016107652336359024,
+            -0.021631157025694847,
+            0.007023954298347235,
+            0.004077056422829628,
+            -0.001974884420633316,
+            0.013492472469806671,
+            -0.020458567887544632,
+            0.017665958032011986,
+            0.009596704505383968,
+            -0.003650836180895567,
+            0.022556884214282036,
+            0.013137610629200935,
+            0.01318389642983675,
+            0.00037053192500025034,
+            -0.016925377771258354,
+            -0.0017810603603720665,
+            0.02649122290313244,
+            0.0009932280518114567,
+            0.0014628416392952204,
+            0.012528173625469208,
+            0.004543777089565992,
+            0.020288852974772453,
+            0.0034849767107516527,
+            -0.018715115264058113,
+            0.029484407976269722,
+            0.00019563216483220458,
+            0.012119310908019543,
+            0.018730545416474342,
+            0.021862588822841644,
+            0.0062525151297450066,
+            0.004632492549717426,
+            0.007097241003066301,
+            0.01903912052512169,
+            0.006545661948621273,
+            -0.013075895607471466,
+            -0.004185057710856199,
+            0.0069699534215033054,
+            0.0020057419314980507,
+            -0.004755923058837652,
+            -0.012196454219520092,
+            0.018298538401722908,
+            -0.007043240126222372,
+            0.0018292752793058753,
+            -0.022664885967969894,
+            0.0069930967874825,
+            -0.008848408237099648,
+            -0.004038484301418066,
+            -0.00823897123336792,
+            0.004786780569702387,
+            0.015876220539212227,
+            -0.004678778816014528,
+            0.03400504216551781,
+            -0.010098139755427837,
+            0.014595630578696728,
+            -0.009342129342257977,
+            -0.02227916568517685,
+            0.018483683466911316,
+            -0.01116272620856762,
+            -0.0014011265011504292,
+            0.00012547941878437996,
+            0.0022391024976968765,
+            -0.009596704505383968,
+            -0.014842490665614605,
+            0.004667207598686218,
+            -0.010869579389691353,
+            0.0303792767226696,
+            -0.0004961318918503821,
+            0.02315860614180565,
+            0.003916982561349869,
+            0.0021735301706939936,
+            0.01731109619140625,
+            -0.005558219738304615,
+            -0.003257402218878269,
+            0.011664161458611488,
+            -0.0072129569016397,
+            0.020011134445667267,
+            -0.00020913235493935645,
+            -0.001489842077717185,
+            0.016940806061029434,
+            -0.011772163212299347,
+            -0.012481886893510818,
+            0.018545400351285934,
+            0.012659317813813686,
+            0.0014657345600426197,
+            0.006144513376057148,
+            0.002813824685290456,
+            -0.0002004536654567346,
+            -0.003560192184522748,
+            -0.020921431481838226,
+            -0.005704793147742748,
+            0.002511034719645977,
+            0.002727037761360407,
+            -0.00455920584499836,
+            0.004401060752570629,
+            0.0012593745486810803,
+            -0.0004539437941275537,
+            -0.002684608567506075,
+            0.0025418922305107117,
+            -0.0031841155141592026,
+            0.004690350499004126,
+            0.005878366995602846,
+            -0.014395056292414665,
+            -0.02391461655497551,
+            0.0065533765591681,
+            0.013631331734359264,
+            -0.010020995512604713,
+            0.0019806702621281147,
+            0.03150557726621628,
+            -0.03786223754286766,
+            0.01738824136555195,
+            0.005434789694845676,
+            -0.007336387410759926,
+            0.005704793147742748,
+            -0.014287054538726807
+          ],
+          "title": "4 Downstream Results"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "f6c6b9b0-8ec4-4329-994f-66b2f9a43c88",
+      "type": "child",
+      "source": {
+        "id": "65742957-6bda-446d-a61a-0fe20313c760",
+        "properties": {
+          "page_content": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity\n\nWilliam Fedus∗ liamfedus@google.com Barret Zoph∗ barretzoph@google.com Noam Shazeer noam@google.com Google, Mountain View, CA 94043, USA Editor: Alexander Clark\n\nAbstract\n\nIn deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) models defy this and instead select different parameters for each incoming example. The result is a sparsely-activated model—with an outrageous number of parameters—but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs, and training instability. We address these with the introduction of the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques mitigate the instabilities, and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large (Raffel et al., 2019) to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the \"Colossal Clean Crawled Corpus\", and achieve a 4x speedup over the T5-XXL model.12 Keywords: mixture-of-experts, natural language processing, sparsity, large-scale machine learning, distributed computing arXiv:2101.03961v3 [cs.LG] 16 Jun 2022 Contents\n\n1 Introduction 3 2 Switch Transformer 4 2.1 Simplifying Sparse Routing 5 2.2 Efficient Sparse Routing 6 2.3 Putting It All Together: The Switch Transformer 8 2.4 Improved Training and Fine-Tuning Techniques 8 3 Scaling Properties 11 3.1 Scaling Results on a Step-Basis 12 3.2 Scaling Results on a Time-Basis 13 3.3 Scaling Versus a Larger Dense Model 13 4 Downstream Results 14 4.1 Fine-Tuning 14 4.2 Distillation 16 4.3 Multilingual Learning 17 5 Designing Models with Data, Model, and Expert-Parallelism 18 5.1 Data Parallelism 20 5.2 Model Parallelism 20 5.3 Model and Data Parallelism 21 5.4 Expert and Data Parallelism 22 5.5 Expert, Model and Data Parallelism 22 5.6 Towards Trillion Parameter Models 22 6 Related Work 24 7 Discussion 25 8 Future Work 26 9 Conclusion 27 A Switch for Attention 27 B Preventing Token Dropping with No-Token-Left-Behind 29\n\nC Encouraging Exploration Across Experts 29 D Switch Transformers in Lower Compute Regimes 29 E Relation of Upstream to Downstream Model Performance 32 F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/switch_transformers.md"
+          },
+          "summary": "The Switch Transformer is a sparsely-activated model that simplifies the Mixture of Experts (MoE) approach, allowing for efficient scaling to trillion-parameter models with reduced communication and computational costs. It achieves significant speedups in pre-training and fine-tuning by activating only a subset of neural network weights for each input, maintaining a constant computational cost. The model demonstrates superior scaling properties and improved performance across various natural language processing tasks, including multilingual settings. It also allows for effective distillation into smaller dense models, preserving a portion of the quality gains. The architecture is designed to maximize parameter count while keeping floating point operations per example constant, making it suitable for both large-scale and smaller-scale applications.",
+          "headlines": {
+            "1 Introduction": [],
+            "2 Switch Transformer": [
+              "2.1 Simplifying Sparse Routing",
+              "2.2 Efficient Sparse Routing",
+              "2.3 Putting It All Together: The Switch Transformer",
+              "2.4 Improved Training and Fine-Tuning Techniques"
+            ],
+            "3 Scaling Properties": [
+              "3.1 Scaling Results on a Step-Basis",
+              "3.2 Scaling Results on a Time-Basis",
+              "3.3 Scaling Versus a Larger Dense Model"
+            ],
+            "4 Downstream Results": [
+              "4.1 Fine-Tuning",
+              "4.2 Distillation",
+              "4.3 Multilingual Learning"
+            ],
+            "5 Designing Models with Data, Model, and Expert-Parallelism": [
+              "5.1 Data Parallelism",
+              "5.2 Model Parallelism",
+              "5.3 Model and Data Parallelism",
+              "5.4 Expert and Data Parallelism",
+              "5.5 Expert, Model and Data Parallelism",
+              "5.6 Towards Trillion Parameter Models"
+            ],
+            "6 Related Work": [],
+            "7 Discussion": [],
+            "8 Future Work": [],
+            "9 Conclusion": [],
+            "A Switch for Attention": [],
+            "B Preventing Token Dropping with No-Token-Left-Behind": [],
+            "C Encouraging Exploration Across Experts": [],
+            "D Switch Transformers in Lower Compute Regimes": [],
+            "E Relation of Upstream to Downstream Model Performance": [],
+            "F Pseudo Code for Switch Transformers": []
+          },
+          "summary_embedding": [
+            -0.0060447100549936295,
+            -0.00390331051312387,
+            -0.01784370467066765,
+            -0.0021820589900016785,
+            0.013227885589003563,
+            0.011338187381625175,
+            0.0165580902248621,
+            0.024240797385573387,
+            -0.005986624862998724,
+            0.03596621751785278,
+            -0.004852031823247671,
+            -0.0068811336532235146,
+            -0.003574162255972624,
+            0.014242436736822128,
+            0.011578272096812725,
+            -0.033302053809165955,
+            -0.024519605562090874,
+            0.01050176378339529,
+            0.016093410551548004,
+            -0.006795942317694426,
+            0.007473600562661886,
+            -0.030513975769281387,
+            -0.06124480068683624,
+            -0.006838538218289614,
+            -0.0026099516544491053,
+            0.01029265858232975,
+            0.0010329445358365774,
+            -0.010873507708311081,
+            -0.053469158709049225,
+            0.023311438038945198,
+            0.025294071063399315,
+            0.005092116538435221,
+            -0.029584616422653198,
+            0.0009361362317577004,
+            -0.05161044001579285,
+            -0.04448534920811653,
+            0.0379488542675972,
+            0.012561844661831856,
+            -0.04364892467856407,
+            0.039249956607818604,
+            -0.004875265993177891,
+            -0.0022440163884311914,
+            -0.021499186754226685,
+            -0.0034909071400761604,
+            -0.007039899472147226,
+            -0.009851212613284588,
+            -0.02874819189310074,
+            0.0014037203509360552,
+            0.021886419504880905,
+            -0.0056962003000080585,
+            0.023264970630407333,
+            -0.024271776899695396,
+            0.01936165988445282,
+            0.04240977764129639,
+            -0.06957805901765823,
+            0.024659009650349617,
+            0.017549408599734306,
+            0.05120771750807762,
+            0.06511713564395905,
+            0.018339363858103752,
+            0.021669568493962288,
+            -0.014141756109893322,
+            -0.017766257748007774,
+            -0.011872569099068642,
+            0.004673904739320278,
+            -0.014970434829592705,
+            -0.01891246810555458,
+            0.008449427783489227,
+            -0.020461400970816612,
+            0.008209343999624252,
+            -0.05576157942414284,
+            -0.014730350114405155,
+            -0.034200433641672134,
+            -0.01140014547854662,
+            0.017642343416810036,
+            -0.018215449526906013,
+            0.030126741155982018,
+            0.02997184917330742,
+            0.002638994250446558,
+            0.0007822110201232135,
+            0.021049996837973595,
+            0.006509389728307724,
+            -0.0053438181057572365,
+            0.012321759946644306,
+            0.028159597888588905,
+            -0.01450575515627861,
+            -0.010106786154210567,
+            -0.03454120084643364,
+            0.01861817203462124,
+            0.01589204929769039,
+            -0.04194509983062744,
+            -0.012569589540362358,
+            -0.013824224472045898,
+            0.032341714948415756,
+            -0.009549170732498169,
+            0.04098476096987724,
+            -0.010083552449941635,
+            0.001383390510454774,
+            -0.031149037182331085,
+            -0.009463978931307793,
+            0.017069239169359207,
+            0.006172496825456619,
+            -0.016589069738984108,
+            -0.0012846461031585932,
+            0.016170857474207878,
+            0.006617815233767033,
+            -0.030142230913043022,
+            0.04488807171583176,
+            0.023497309535741806,
+            0.055296897888183594,
+            0.05579255893826485,
+            -0.037391237914562225,
+            -0.002445377642288804,
+            0.0344172865152359,
+            -0.0228312686085701,
+            -0.015040136873722076,
+            -0.034014564007520676,
+            -0.01768881268799305,
+            0.035935238003730774,
+            -0.0318460576236248,
+            0.01556677371263504,
+            0.028066661208868027,
+            -0.025712283328175545,
+            0.003488970920443535,
+            -0.03943582624197006,
+            0.003622566582635045,
+            0.01220559049397707,
+            0.006095050368458033,
+            -0.0026293133851140738,
+            -0.013615119270980358,
+            0.008906363509595394,
+            0.003055269829928875,
+            0.00800798274576664,
+            -0.04615819454193115,
+            0.018478767946362495,
+            -0.00454224506393075,
+            0.04033420979976654,
+            0.042502716183662415,
+            -0.02730768360197544,
+            -0.016743963584303856,
+            0.008557853288948536,
+            0.012104909867048264,
+            0.04188314080238342,
+            -0.007287728600203991,
+            -0.00121784838847816,
+            -0.0007836631848476827,
+            0.007322579622268677,
+            0.047552235424518585,
+            0.004646798130124807,
+            0.0020697612781077623,
+            -0.027230238541960716,
+            0.014978179708123207,
+            -0.019733402878046036,
+            0.027648448944091797,
+            0.023373395204544067,
+            0.02942972257733345,
+            -0.04048910364508629,
+            0.017518429085612297,
+            -0.011818356812000275,
+            -0.005568413063883781,
+            -0.002726121572777629,
+            -0.02856232039630413,
+            -0.0074426219798624516,
+            0.033983584493398666,
+            0.015365412458777428,
+            0.012956822291016579,
+            0.03271345794200897,
+            0.0012052633101120591,
+            -0.042502716183662415,
+            0.03804178908467293,
+            0.024550583213567734,
+            0.04640602692961693,
+            0.023466331884264946,
+            0.010796061716973782,
+            0.006571347359567881,
+            0.013375034555792809,
+            -0.019377149641513824,
+            0.015907539054751396,
+            0.02560385875403881,
+            0.009750531986355782,
+            -0.03167567402124405,
+            -0.0009501734166406095,
+            0.01178737822920084,
+            -0.004546117503196001,
+            0.03701949119567871,
+            0.04104671627283096,
+            0.020569827407598495,
+            0.05030933395028114,
+            0.0020155487582087517,
+            0.017936641350388527,
+            0.005936284549534321,
+            0.02820606529712677,
+            0.030467506498098373,
+            -0.005285732913762331,
+            0.005076627247035503,
+            -0.01136916596442461,
+            -0.03385967016220093,
+            0.014110777527093887,
+            0.01561324205249548,
+            -0.008077683858573437,
+            -0.01697630248963833,
+            0.006033093202859163,
+            0.009169681929051876,
+            -0.01795213110744953,
+            -0.017456471920013428,
+            0.005971135571599007,
+            0.018866000697016716,
+            -0.013576395809650421,
+            0.024689987301826477,
+            -0.04442339017987251,
+            0.029119934886693954,
+            -0.033209118992090225,
+            -0.003384418087080121,
+            -0.010641167871654034,
+            -0.0030843124259263277,
+            0.0041666291654109955,
+            0.011314953677356243,
+            0.013406013138592243,
+            0.002042655134573579,
+            -0.013909416273236275,
+            0.024519605562090874,
+            0.018447788432240486,
+            0.012073930352926254,
+            -0.03148980438709259,
+            -0.014397329650819302,
+            -0.0015673263696953654,
+            -0.008743725717067719,
+            -0.034138478338718414,
+            0.026966920122504234,
+            0.014048820361495018,
+            -0.03844451159238815,
+            0.017797237262129784,
+            0.029538147151470184,
+            0.014420564286410809,
+            -0.004790074657648802,
+            -0.0463130883872509,
+            0.04045812413096428,
+            -0.030436528846621513,
+            -0.033333033323287964,
+            -0.010478530079126358,
+            -0.0003146269591525197,
+            0.02894955314695835,
+            -0.0064358157105743885,
+            -0.050835974514484406,
+            0.01184933539479971,
+            0.020631784573197365,
+            0.00010763872705865651,
+            0.005731051322072744,
+            -0.019593998789787292,
+            0.00028631053282879293,
+            -0.03395260497927666,
+            0.0090844901278615,
+            0.026750069111585617,
+            0.008867640048265457,
+            -0.016418687999248505,
+            0.03351890295743942,
+            -0.0004121613164898008,
+            -0.018695618957281113,
+            -0.003889757441356778,
+            0.006885006092488766,
+            0.034912943840026855,
+            0.004836542531847954,
+            -0.02946070022881031,
+            0.05665995925664902,
+            -0.02100352756679058,
+            0.021344292908906937,
+            0.05631919577717781,
+            -0.019439106807112694,
+            -0.008472662419080734,
+            -0.002065889071673155,
+            0.0026196325197815895,
+            -0.0260530486702919,
+            -0.008937342092394829,
+            0.032341714948415756,
+            0.041728246957063675,
+            -0.018742086365818977,
+            -0.001398879918269813,
+            0.006780453026294708,
+            0.009425255469977856,
+            0.004948840010911226,
+            0.02692045085132122,
+            -0.012128143571317196,
+            -0.014559967443346977,
+            0.05198218300938606,
+            0.028314489871263504,
+            -0.012081675231456757,
+            -0.0164031982421875,
+            0.019888296723365784,
+            -0.024844881147146225,
+            -0.020244551822543144,
+            -0.02431824430823326,
+            -0.020414933562278748,
+            -0.0042518205009400845,
+            0.009525936096906662,
+            0.037484172731637955,
+            -0.03968365862965584,
+            0.012747716158628464,
+            0.021700548008084297,
+            0.02981695532798767,
+            -0.019532041624188423,
+            -0.05910727381706238,
+            -0.0016128262504935265,
+            0.03296129032969475,
+            -0.015458349138498306,
+            -0.049379974603652954,
+            0.015164051204919815,
+            0.0260530486702919,
+            0.05957195535302162,
+            -0.001461805310100317,
+            -0.014954946003854275,
+            0.061926331371068954,
+            0.03026614524424076,
+            -0.004383479710668325,
+            -0.009494957514107227,
+            -0.0127864396199584,
+            0.003396035172045231,
+            0.00465454300865531,
+            -0.03364282101392746,
+            -0.041170634329319,
+            -0.011841590516269207,
+            -0.010284913703799248,
+            0.008108663372695446,
+            -0.040922801941633224,
+            0.03531566634774208,
+            -0.021917399019002914,
+            0.010950954630970955,
+            0.015473837964236736,
+            0.037391237914562225,
+            -0.021344292908906937,
+            -0.01284839678555727,
+            -0.01572941243648529,
+            -0.024968795478343964,
+            -0.014010096900165081,
+            0.021917399019002914,
+            0.007682706229388714,
+            3.4699722164077684e-05,
+            0.010075807571411133,
+            -0.024659009650349617,
+            -0.051052823662757874,
+            0.0291044469922781,
+            0.0015605497173964977,
+            -0.015938518568873405,
+            -0.004623564425855875,
+            -0.01977987214922905,
+            -0.01122201792895794,
+            -0.012608313001692295,
+            -0.039249956607818604,
+            0.021111954003572464,
+            -0.01135367713868618,
+            0.015876561403274536,
+            0.0009250032599084079,
+            0.003659353591501713,
+            0.028624277561903,
+            0.01116006076335907,
+            0.0051811798475682735,
+            -0.03255856782197952,
+            -0.018277406692504883,
+            0.010122275911271572,
+            -0.00561100896447897,
+            -0.018339363858103752,
+            -0.013351799920201302,
+            -0.05037129297852516,
+            -0.03305422514677048,
+            -0.008054450154304504,
+            -0.05415068939328194,
+            0.0020697612781077623,
+            -0.014296649023890495,
+            -0.02232012152671814,
+            -0.005990497302263975,
+            0.03107159025967121,
+            -0.003235333366319537,
+            0.025294071063399315,
+            0.0076594725251197815,
+            0.037360258400440216,
+            0.040117356926202774,
+            0.008426194079220295,
+            0.03065337985754013,
+            -0.0029991210903972387,
+            -0.016867877915501595,
+            0.05486319959163666,
+            -0.01881953328847885,
+            0.013529927469789982,
+            -0.008604321628808975,
+            0.008604321628808975,
+            0.03299226611852646,
+            -0.003794885240495205,
+            0.02007416822016239,
+            -0.0311955064535141,
+            -0.07998688519001007,
+            0.01637221872806549,
+            0.028159597888588905,
+            -0.06418777257204056,
+            0.014691627584397793,
+            -0.020275529474020004,
+            0.02052335813641548,
+            0.02411688305437565,
+            -0.0009114501299336553,
+            -0.014064309187233448,
+            0.005595519673079252,
+            0.0020833145827054977,
+            -0.01456771232187748,
+            -0.002691270550712943,
+            -0.009417511522769928,
+            0.008774704299867153,
+            -0.017007282003760338,
+            -0.013406013138592243,
+            -0.034231413155794144,
+            0.010873507708311081,
+            0.01233724970370531,
+            -0.003969140350818634,
+            0.0011500825639814138,
+            -0.03277541697025299,
+            -0.007876322604715824,
+            -0.02145271934568882,
+            0.03382869064807892,
+            0.012058441527187824,
+            -0.030699847266077995,
+            -0.007814365439116955,
+            0.015264731831848621,
+            -0.007876322604715824,
+            0.0008906363509595394,
+            0.024132372811436653,
+            0.02103450708091259,
+            0.0028364830650389194,
+            -0.0009700191440060735,
+            0.02438020147383213,
+            0.007427132688462734,
+            -0.01720864325761795,
+            0.028794659301638603,
+            0.014288905076682568,
+            0.060160547494888306,
+            -0.03271345794200897,
+            0.001297231181524694,
+            -0.04135650396347046,
+            -0.031118059530854225,
+            0.006408709101378918,
+            0.014350862242281437,
+            0.007597515359520912,
+            -0.028252532705664635,
+            0.005978880450129509,
+            0.02004319056868553,
+            0.01469937153160572,
+            0.009704063646495342,
+            0.0037580979987978935,
+            -0.036740683019161224,
+            -0.000691695255227387,
+            -0.011214273050427437,
+            -0.0038781403563916683,
+            0.035656433552503586,
+            -0.008186109364032745,
+            -0.05151750147342682,
+            -0.0006418389966711402,
+            -0.03255856782197952,
+            -0.0353776253759861,
+            0.021313315257430077,
+            -0.00839521549642086,
+            -0.0039207362569868565,
+            -0.013986863195896149,
+            -0.000537770101800561,
+            -0.010524998418986797,
+            0.027509044855833054,
+            0.028345469385385513,
+            0.022490503266453743,
+            -0.05384090170264244,
+            0.009479468688368797,
+            0.00023766436788719147,
+            -0.05817791447043419,
+            0.0009709871956147254,
+            -0.007117345929145813,
+            0.011191039346158504,
+            -0.020213572308421135,
+            -0.02414786070585251,
+            0.011500826105475426,
+            0.03392162546515465,
+            0.014304393902420998,
+            0.0321558453142643,
+            -0.013367289677262306,
+            -0.09250226616859436,
+            0.018292896449565887,
+            -0.008689512498676777,
+            0.04126356914639473,
+            0.00865078903734684,
+            -0.004278926644474268,
+            -0.015845581889152527,
+            -0.00865078903734684,
+            -0.006943091284483671,
+            0.01283290795981884,
+            0.004530628211796284,
+            -0.005626498255878687,
+            0.0016631665639579296,
+            0.013708055019378662,
+            -0.005967263597995043,
+            0.007481344975531101,
+            0.003651608945801854,
+            -0.013189162127673626,
+            -0.06059424951672554,
+            -0.04055105894804001,
+            -0.018540725111961365,
+            0.050835974514484406,
+            -0.0069159846752882,
+            -0.046684831380844116,
+            -0.010850274004042149,
+            0.05058814212679863,
+            0.012259802781045437,
+            -0.0334569476544857,
+            -0.0007841471815481782,
+            0.0050882440991699696,
+            0.015458349138498306,
+            0.004546117503196001,
+            -0.027292195707559586,
+            0.02820606529712677,
+            0.0008412640891037881,
+            0.02300165221095085,
+            -0.018339363858103752,
+            -0.005312839522957802,
+            0.035780347883701324,
+            -0.024844881147146225,
+            0.023791607469320297,
+            0.0055219451896846294,
+            0.031629208475351334,
+            0.002952652983367443,
+            0.010060317814350128,
+            0.004704883322119713,
+            0.0186491496860981,
+            0.004275054205209017,
+            0.0276639387011528,
+            0.011462102644145489,
+            0.006428070832043886,
+            0.021623101085424423,
+            -0.009378788061439991,
+            -0.014513500034809113,
+            0.01885051093995571,
+            -0.0014695499558001757,
+            0.025232113897800446,
+            -0.020538847893476486,
+            0.002726121572777629,
+            -0.0016883367206901312,
+            -0.017967619001865387,
+            -0.040922801941633224,
+            -0.0007405834621749818,
+            0.021747015416622162,
+            -0.011671207845211029,
+            -0.0228312686085701,
+            -0.013715799897909164,
+            -0.01585332676768303,
+            0.0038490979932248592,
+            0.034169457852840424,
+            -0.0398695282638073,
+            0.00963436160236597,
+            -0.000830615172162652,
+            -0.03125746175646782,
+            -0.0239619892090559,
+            -0.01093546487390995,
+            0.014614180661737919,
+            -0.01652711257338524,
+            -0.021173911169171333,
+            -0.028469383716583252,
+            -0.008410705253481865,
+            0.006048582494258881,
+            -0.05065010115504265,
+            -0.004054331686347723,
+            0.019965743646025658,
+            0.0276639387011528,
+            -0.003099801717326045,
+            -0.02177799493074417,
+            0.0012468908680602908,
+            0.022351099178195,
+            0.02936776541173458,
+            0.014226946979761124,
+            0.027586491778492928,
+            -0.002178186783567071,
+            -0.0011675080750137568,
+            -0.0073148347437381744,
+            0.01307299267500639,
+            0.0011287847300991416,
+            0.036616768687963486,
+            0.0122210793197155,
+            -0.020306508988142014,
+            0.0013959755888208747,
+            -0.006431943271309137,
+            0.029181893914937973,
+            -0.02270735427737236,
+            -9.832092473516241e-05,
+            0.03531566634774208,
+            0.027741385623812675,
+            0.004933350719511509,
+            -0.0143121387809515,
+            -0.05142456665635109,
+            0.021592123433947563,
+            -0.01765783317387104,
+            0.031025122851133347,
+            0.016666516661643982,
+            0.003397971158847213,
+            -0.0034212053287774324,
+            0.012368228286504745,
+            -0.022087780758738518,
+            -0.004406713880598545,
+            -0.021142931655049324,
+            0.04389675334095955,
+            -0.043184246867895126,
+            0.008472662419080734,
+            0.029445212334394455,
+            0.011926782317459583,
+            -0.019082851707935333,
+            -0.005425137002021074,
+            0.029212871566414833,
+            -0.00162637943867594,
+            0.01308848150074482,
+            0.0058665829710662365,
+            -0.026935940608382225,
+            0.01202746294438839,
+            -0.031660184264183044,
+            0.02177799493074417,
+            -0.013436991721391678,
+            0.018447788432240486,
+            0.03342596814036369,
+            -0.0030397805385291576,
+            -0.0038239278364926577,
+            -0.016263794153928757,
+            -0.00969631876796484,
+            0.0662323608994484,
+            0.01054823212325573,
+            0.03726732358336449,
+            0.0018974426202476025,
+            -0.009773765690624714,
+            0.007969259284436703,
+            0.0063041565008461475,
+            -0.012329504825174809,
+            0.010494019836187363,
+            0.024178840219974518,
+            0.0012730291346088052,
+            -0.002412462839856744,
+            -0.019408127292990685,
+            -0.03438630700111389,
+            0.003727119415998459,
+            -0.030932186171412468,
+            0.022691864520311356,
+            0.009208405390381813,
+            0.017456471920013428,
+            0.008240322582423687,
+            -0.015907539054751396,
+            0.03200095146894455,
+            0.05461536720395088,
+            0.0019768255297094584,
+            0.013375034555792809,
+            0.007938280701637268,
+            0.007036027032881975,
+            -0.011338187381625175,
+            0.04702559858560562,
+            0.000662652775645256,
+            0.005169562995433807,
+            0.022536970674991608,
+            0.02145271934568882,
+            0.022939693182706833,
+            0.002304037567228079,
+            -0.011694442480802536,
+            0.010478530079126358,
+            -0.0006302220281213522,
+            0.004445436876267195,
+            -0.010695381090044975,
+            -0.027865299955010414,
+            0.007492962293326855,
+            0.01302652433514595,
+            0.022598929703235626,
+            0.021592123433947563,
+            -0.03178409859538078,
+            -0.01855621486902237,
+            0.03364282101392746,
+            -0.02997184917330742,
+            0.03612111136317253,
+            -0.0014424435794353485,
+            -0.023807097226381302,
+            -0.0058356039226055145,
+            0.016929835081100464,
+            0.02656419761478901,
+            -0.029181893914937973,
+            0.01804506592452526,
+            0.029693040996789932,
+            -0.0074426219798624516,
+            0.0005939188995398581,
+            0.012112653814256191,
+            0.02447313815355301,
+            -0.009293596260249615,
+            -0.0024027819745242596,
+            -0.0013320822035893798,
+            -0.011926782317459583,
+            -0.018370341509580612,
+            -0.0044725434854626656,
+            -0.022629907354712486,
+            0.005564541090279818,
+            -0.007365175057202578,
+            -0.003481226274743676,
+            -0.008581086993217468,
+            0.01154729351401329,
+            0.001979729626327753,
+            -0.019857317209243774,
+            -0.022335609421133995,
+            0.01652711257338524,
+            0.03887821361422539,
+            -0.018215449526906013,
+            -0.02106548473238945,
+            0.024442158639431,
+            -0.01759587600827217,
+            -0.01601596549153328,
+            0.036678727716207504,
+            0.013739033602178097,
+            0.007736918982118368,
+            -0.03358086198568344,
+            -0.030049296095967293,
+            -0.03274443745613098,
+            0.006296411622315645,
+            0.03785591572523117,
+            -0.0028752065263688564,
+            0.02569679357111454,
+            0.03701949119567871,
+            0.0057891360484063625,
+            -0.028934063389897346,
+            -0.014823286794126034,
+            -0.045104920864105225,
+            0.060160547494888306,
+            0.0011481463443487883,
+            0.05161044001579285,
+            0.016294771805405617,
+            -0.01456771232187748,
+            0.00419760774821043,
+            -0.013251119293272495,
+            0.026192452758550644,
+            0.02280029095709324,
+            0.025929134339094162,
+            0.006582964211702347,
+            0.022505993023514748,
+            -0.0037522895727306604,
+            -0.012197845615446568,
+            0.007307090330868959,
+            0.014536733739078045,
+            0.0002497654058970511,
+            -0.011392400600016117,
+            0.015783624723553658,
+            -0.01307299267500639,
+            -0.011361422017216682,
+            0.023295948281884193,
+            0.01368482131510973,
+            -0.012344993650913239,
+            0.014064309187233448,
+            -0.008565598167479038,
+            0.01239920686930418,
+            -0.023915521800518036,
+            0.022691864520311356,
+            -0.03262052312493324,
+            0.009587894193828106,
+            0.0015053689712658525,
+            -0.0073148347437381744,
+            0.024550583213567734,
+            -0.0027454833034425974,
+            -0.05071205645799637,
+            0.03550153970718384,
+            -0.008015726692974567,
+            0.0037213109899312258,
+            0.006381602957844734,
+            0.011392400600016117,
+            -0.0012623801594600081,
+            0.03807276859879494,
+            0.01239920686930418,
+            0.015969496220350266,
+            0.02199484594166279,
+            0.011911292560398579,
+            -0.008433938957750797,
+            -0.004096927121281624,
+            -0.0016137943603098392,
+            -0.01784370467066765,
+            0.025944624096155167,
+            0.0018674320308491588,
+            0.03962169960141182,
+            -0.018168980255723,
+            -0.02300165221095085,
+            0.028066661208868027,
+            0.007597515359520912,
+            -0.02351279929280281,
+            -0.0017377089243382215,
+            0.01202746294438839,
+            -0.009185170754790306,
+            0.027385130524635315,
+            0.01196550577878952,
+            0.019113829359412193,
+            -0.018060555681586266,
+            -0.004340883810073137,
+            -0.005072754807770252,
+            -0.015938518568873405,
+            0.02238207869231701,
+            0.020771188661456108,
+            0.011152315884828568,
+            -0.011880313977599144,
+            -0.01307299267500639,
+            0.020600805059075356,
+            0.035036858171224594,
+            -0.0002744515077210963,
+            0.005672966130077839,
+            0.017487451434135437,
+            0.002373739378526807,
+            -0.04470219835639,
+            0.016232814639806747,
+            -0.014025585725903511,
+            0.011214273050427437,
+            0.019593998789787292,
+            -0.021917399019002914,
+            -0.03196997195482254,
+            -0.011539548635482788,
+            -0.0414804182946682,
+            0.00493722315877676,
+            -0.0069159846752882,
+            0.013576395809650421,
+            -0.010176488198339939,
+            -0.0014831031439825892,
+            0.016666516661643982,
+            0.0038103745318949223,
+            0.02177799493074417,
+            -0.011678952723741531,
+            -0.012073930352926254,
+            -0.020213572308421135,
+            -0.020786678418517113,
+            0.004278926644474268,
+            -0.030405549332499504,
+            -0.004488032776862383,
+            0.021375272423028946,
+            -0.04783104360103607,
+            -0.03689557686448097,
+            0.02898053266108036,
+            -0.008503641001880169,
+            0.020755698904395103,
+            0.022955182939767838,
+            -0.015094350092113018,
+            -0.0008717587334103882,
+            -0.018695618957281113,
+            0.02258343994617462,
+            -0.007837600074708462,
+            0.026300877332687378,
+            -0.0027551641687750816,
+            0.0006292539183050394,
+            -0.02659517526626587,
+            0.020306508988142014,
+            0.005227648187428713,
+            0.012275291606783867,
+            -0.02390003204345703,
+            0.006815304048359394,
+            -0.03133491054177284,
+            -0.01784370467066765,
+            0.0180140882730484,
+            -0.011152315884828568,
+            0.027989214286208153,
+            -0.006060199346393347,
+            0.028438406065106392,
+            0.022722844034433365,
+            0.024751944467425346,
+            0.02676555886864662,
+            -0.02363671362400055,
+            0.008495896123349667,
+            0.03726732358336449,
+            0.00865078903734684,
+            -0.015876561403274536,
+            0.008170620538294315,
+            -0.04863648861646652,
+            -0.008062195032835007,
+            0.05030933395028114,
+            0.002127846470102668,
+            0.023141054436564445,
+            0.016294771805405617,
+            -0.002873270306736231,
+            -0.005688455421477556,
+            0.010269423946738243,
+            -0.04296739399433136,
+            -0.023172033950686455,
+            0.022629907354712486,
+            -0.021561143919825554,
+            6.54212328754511e-07,
+            0.03122648410499096,
+            0.012120398692786694,
+            -0.0009109660750254989,
+            -0.05108380317687988,
+            0.02100352756679058,
+            0.01178737822920084,
+            0.014443797990679741,
+            -0.010300402529537678,
+            0.026068538427352905,
+            0.01733255758881569,
+            0.038413532078266144,
+            0.012755461037158966,
+            -0.002679653698578477,
+            -0.03782493621110916,
+            0.009177426807582378,
+            -0.021685058251023293,
+            0.0011471783509477973,
+            0.0074503663927316666,
+            -0.02273833192884922,
+            -0.024782923981547356,
+            0.0043912241235375404,
+            0.004588713403791189,
+            0.011880313977599144,
+            0.022087780758738518,
+            0.009417511522769928,
+            -0.021204888820648193,
+            0.015404135920107365,
+            -0.05446047708392143,
+            -0.015450604259967804,
+            -0.023884542286396027,
+            0.02566581591963768,
+            0.020012211054563522,
+            0.010184233076870441,
+            0.005545179359614849,
+            0.010904486291110516,
+            -0.020988039672374725,
+            -0.009239383973181248,
+            0.020631784573197365,
+            0.011074868962168694,
+            -0.012840652838349342,
+            0.04113965481519699,
+            -0.00011126903700642288,
+            0.031210994347929955,
+            -0.02280029095709324,
+            0.002941036131232977,
+            -0.001241082325577736,
+            -0.03649285435676575,
+            0.009130958467721939,
+            0.01981084980070591,
+            0.02151467651128769,
+            -0.01861817203462124,
+            0.0024143988266587257,
+            0.01048627495765686,
+            0.004766840487718582,
+            0.038351576775312424,
+            0.027121812105178833,
+            -0.044206541031599045,
+            0.011663463898003101,
+            0.012995545752346516,
+            -0.009363298304378986,
+            0.048419639468193054,
+            -0.017518429085612297,
+            0.017673322930932045,
+            0.008240322582423687,
+            -0.01798310875892639,
+            0.02318752370774746,
+            -0.01469937153160572,
+            -0.013282098807394505,
+            0.00650551775470376,
+            0.009192915633320808,
+            -0.0001770381786627695,
+            -0.037577107548713684,
+            -0.017859194427728653,
+            -0.005719434004276991,
+            0.0038297362625598907,
+            0.01562873087823391,
+            0.013498948886990547,
+            0.009967382065951824,
+            0.005161818582564592,
+            -0.00945623405277729,
+            -0.0016089539276435971,
+            0.008124152198433876,
+            -0.0005532593932002783,
+            0.0019216446671634912,
+            -0.01926872320473194,
+            0.02698240801692009,
+            0.02572777308523655,
+            -0.0018354853382334113,
+            -0.04652994126081467,
+            -0.0036709706764668226,
+            0.0037077576853334904,
+            -0.017735280096530914,
+            -0.0035315665882080793,
+            -0.04519785940647125,
+            -0.01810702309012413,
+            0.0143121387809515,
+            -0.014614180661737919,
+            -0.0460033044219017,
+            0.009076746180653572,
+            -0.013886182568967342,
+            0.0063970922492444515,
+            -0.00932457484304905,
+            0.005762029904872179,
+            0.003651608945801854,
+            0.005680711008608341,
+            -0.005909178406000137,
+            0.05108380317687988,
+            -0.03466511517763138,
+            0.028051171451807022,
+            0.008565598167479038,
+            -0.012437930330634117,
+            -0.02261441759765148,
+            0.00887538492679596,
+            -0.044981006532907486,
+            -0.033209118992090225,
+            0.016743963584303856,
+            -0.01781272701919079,
+            0.011074868962168694,
+            0.06161654368042946,
+            0.007330324035137892,
+            -0.023342415690422058,
+            -0.01233724970370531,
+            -0.021049996837973595,
+            -0.04191412031650543,
+            0.003442503046244383,
+            0.005463860463351011,
+            -0.004948840010911226,
+            -0.03807276859879494,
+            -0.021716037765145302,
+            0.04615819454193115,
+            -0.012004229240119457,
+            0.012422440573573112,
+            0.01080380566418171,
+            -0.0019410063978284597,
+            -0.04860550910234451,
+            -0.021019017323851585,
+            0.0395597442984581,
+            0.021886419504880905,
+            -0.008426194079220295,
+            0.0028364830650389194,
+            -0.018277406692504883,
+            -0.003289545886218548,
+            0.006296411622315645,
+            -0.009107724763453007,
+            -0.03562545403838158,
+            0.02067825198173523,
+            0.0054832217283546925,
+            -0.023373395204544067,
+            -0.025805220007896423,
+            -0.03296129032969475,
+            0.009944148361682892,
+            0.02453509531915188,
+            -0.01968693546950817,
+            0.005823987070471048,
+            0.00027203132049180567,
+            -0.000881923595443368,
+            0.00042740863864310086,
+            -0.009959637187421322,
+            -0.019764382392168045,
+            -0.020151615142822266,
+            0.03268248215317726,
+            -0.0060059865936636925,
+            0.0074387495405972,
+            -0.03630698472261429,
+            -0.010974188335239887,
+            0.011322698555886745,
+            -0.002395037328824401,
+            -0.013452480547130108,
+            -0.009092235006392002,
+            -0.01739451475441456,
+            -0.009882191196084023,
+            -0.007876322604715824,
+            0.022660886868834496,
+            0.006168624851852655,
+            0.005409647710621357,
+            0.0023795480374246836,
+            -0.008581086993217468,
+            0.02898053266108036,
+            0.017874684184789658,
+            -0.014188224449753761,
+            -0.00920066051185131,
+            0.03451022133231163,
+            0.02665713243186474,
+            -0.009115469641983509,
+            0.01798310875892639,
+            0.023264970630407333,
+            -0.0010484338272362947,
+            -0.01196550577878952,
+            0.001922612776979804,
+            -0.0009550138493068516,
+            -0.0032972905319184065,
+            0.00845717266201973,
+            -0.015365412458777428,
+            0.0315207801759243,
+            0.04287445917725563,
+            -0.002145271748304367,
+            0.009587894193828106,
+            -0.003727119415998459,
+            -0.008472662419080734,
+            0.017069239169359207,
+            0.009316830895841122,
+            0.04625113308429718,
+            0.01098193321377039,
+            0.005680711008608341,
+            -0.033983584493398666,
+            -0.001229465357027948,
+            -0.01184933539479971,
+            0.03308520466089249,
+            0.02061629481613636,
+            0.00889087375253439,
+            -0.008379725739359856,
+            0.01344473659992218,
+            0.008341003209352493,
+            0.00661007035523653,
+            0.020569827407598495,
+            -0.016542602330446243,
+            -0.017286090180277824,
+            0.030467506498098373,
+            -0.00672236829996109,
+            0.01289486512541771,
+            0.02029101923108101,
+            -0.002089123008772731,
+            0.010842529125511646,
+            0.07100307941436768,
+            -0.006184114143252373,
+            0.015713922679424286,
+            -0.019082851707935333,
+            0.0015789433382451534,
+            -0.007640110794454813,
+            -0.03370477631688118,
+            0.018246427178382874,
+            0.0039962464943528175,
+            0.00865078903734684,
+            0.01198099460452795,
+            0.009053511545062065,
+            -0.007946024648845196,
+            -0.00963436160236597,
+            0.015195030719041824,
+            0.0019080914789810777,
+            0.00033568276558071375,
+            0.001336922636255622,
+            -0.0065210070461034775,
+            -0.004011735785752535,
+            -0.021948376670479774,
+            0.02763296104967594,
+            0.007140580099076033,
+            0.030606910586357117,
+            -0.055296897888183594,
+            0.030095763504505157,
+            -0.00789568480104208,
+            0.02554190158843994,
+            -0.016201836988329887,
+            -0.011090358719229698,
+            0.001609921921044588,
+            0.018153492361307144,
+            0.004491904750466347,
+            0.006462921854108572,
+            0.011888058856129646,
+            0.023233991116285324,
+            -0.03444826602935791,
+            0.017053749412298203,
+            0.023110076785087585,
+            0.0048675211146473885,
+            0.0003535923024173826,
+            0.005165690556168556,
+            0.015458349138498306,
+            0.007198664825409651,
+            -0.023048119619488716,
+            -0.0025692922063171864,
+            0.017611365765333176,
+            -0.02306360937654972,
+            0.0036980770528316498,
+            -0.006923729553818703,
+            0.004623564425855875,
+            -0.0014705180656164885,
+            -0.03509881719946861,
+            -0.00100099784322083,
+            -0.025743262842297554,
+            0.01923774555325508,
+            0.009556914679706097,
+            -0.025867177173495293,
+            0.003932353109121323,
+            0.025650326162576675,
+            -0.0003695656778290868,
+            0.00821708794683218,
+            0.0004683101433329284,
+            0.00869725737720728,
+            -0.00974278710782528,
+            -0.01493945624679327,
+            0.004801691509783268,
+            0.01531120017170906,
+            -0.01493945624679327,
+            -0.007431004662066698,
+            -0.009510447271168232,
+            0.030560443177819252,
+            0.0006674932083114982,
+            0.004189862869679928,
+            -0.005045648664236069,
+            -0.01858719252049923,
+            -0.0032237162813544273,
+            -0.021142931655049324,
+            0.004367990419268608,
+            -0.0009046735358424485,
+            0.04197607934474945,
+            0.0038161829579621553,
+            -0.02029101923108101,
+            0.0032972905319184065,
+            -0.03136589005589485,
+            0.0005000148667022586,
+            -0.007276111748069525,
+            -0.02004319056868553,
+            0.0015005286550149322,
+            0.0016525175888091326,
+            0.012631546705961227,
+            -0.03255856782197952,
+            -0.009301341138780117,
+            0.018416810780763626,
+            -0.00020680672605521977,
+            -0.010687636211514473,
+            -0.017828216776251793,
+            0.00035189816844649613,
+            -0.011338187381625175,
+            -0.027973726391792297,
+            -0.02850036323070526,
+            -0.0026719090528786182,
+            0.013491204008460045,
+            -0.018277406692504883,
+            -0.014993669465184212,
+            0.0031249718740582466,
+            0.021592123433947563,
+            -0.009781510569155216,
+            0.008302279748022556,
+            0.02241305634379387,
+            -0.008255811408162117,
+            -0.005514200776815414,
+            -0.000941460661124438,
+            -0.009130958467721939,
+            0.004267309792339802,
+            0.006300284061580896,
+            0.043339136987924576,
+            -0.032403673976659775,
+            -0.0018238683696836233,
+            0.014815541915595531,
+            0.0034153966698795557,
+            0.0025905899237841368,
+            0.002503462601453066,
+            -0.0007841471815481782,
+            0.01344473659992218,
+            0.01178737822920084,
+            0.015466093085706234,
+            0.00100099784322083,
+            -0.0033011629711836576,
+            0.006408709101378918,
+            -0.021173911169171333,
+            0.015721667557954788,
+            0.027958236634731293,
+            -0.0033999073784798384,
+            0.005525817628949881,
+            -0.013731288723647594,
+            -0.008759214542806149,
+            0.013134949840605259,
+            0.0026641644071787596,
+            -0.005068882368505001,
+            -0.009394276887178421,
+            0.017967619001865387,
+            0.019082851707935333,
+            0.006668155547231436,
+            -0.037453193217515945,
+            -0.02898053266108036,
+            -0.009959637187421322,
+            0.020569827407598495,
+            0.014916222542524338,
+            0.0010339126456528902,
+            -0.010540487244725227,
+            -0.009595638141036034,
+            -0.011903548613190651,
+            0.010989678092300892,
+            0.02721474878489971,
+            -0.032372694462537766,
+            0.0008214184199459851,
+            -0.029383255168795586,
+            0.03068435750901699,
+            0.0020000594668090343,
+            0.011446612887084484,
+            -0.0008185141487047076,
+            -0.015140817500650883,
+            -0.029181893914937973,
+            0.013824224472045898,
+            -0.004395096562802792,
+            0.001024231780320406,
+            -0.0031753121875226498,
+            0.0201051477342844,
+            0.012887120246887207,
+            -0.0014472840121015906,
+            0.01968693546950817,
+            0.008178364485502243,
+            0.016480645164847374,
+            -0.010153254494071007,
+            -0.04135650396347046,
+            0.019965743646025658,
+            0.007562664337456226,
+            0.014219203032553196,
+            0.004739734344184399,
+            0.004588713403791189,
+            -0.020771188661456108,
+            0.002724185585975647,
+            0.004333139397203922,
+            0.014405074529349804,
+            0.011028401553630829,
+            0.0025615475606173277,
+            0.011678952723741531,
+            -0.02228914201259613,
+            0.01929970271885395,
+            -0.01631026156246662,
+            0.01006806269288063,
+            0.004445436876267195,
+            -0.026873983442783356,
+            0.007547175046056509,
+            0.007822110317647457,
+            -0.0019187404541298747,
+            -0.017549408599734306,
+            0.02160761132836342,
+            -0.0034831624943763018,
+            -0.03509881719946861,
+            0.0360281765460968,
+            -0.0043447562493383884,
+            0.019346170127391815,
+            0.03209388628602028,
+            -0.01771979033946991,
+            -0.01993476413190365,
+            0.00906900130212307,
+            0.010408828034996986,
+            -0.002873270306736231,
+            0.017859194427728653,
+            -0.002919738180935383,
+            -0.02232012152671814,
+            0.017766257748007774,
+            -0.028918573632836342,
+            -0.014234691858291626,
+            0.020337486639618874,
+            -0.03624502569437027,
+            -0.0021026763133704662,
+            0.03175312280654907,
+            -0.024070415645837784,
+            0.005734923295676708,
+            0.005196669604629278,
+            -0.0019351978553459048,
+            0.0012236569309607148,
+            -0.006451305001974106,
+            0.03801080957055092,
+            -0.010865762829780579,
+            0.0201051477342844,
+            0.01929970271885395,
+            -0.023388884961605072,
+            0.01977987214922905,
+            -0.001109423115849495,
+            0.01675945147871971,
+            -0.003940097521990538,
+            0.03466511517763138,
+            -0.006017603911459446,
+            -0.011330443434417248,
+            0.006431943271309137,
+            0.008488151244819164,
+            -0.009959637187421322,
+            0.008681767620146275,
+            -0.018122512847185135,
+            0.027152791619300842,
+            0.02091059274971485,
+            -0.0002691270783543587,
+            0.013800990767776966,
+            0.0004421719058882445,
+            -0.020833145827054977,
+            0.00789568480104208,
+            -0.027121812105178833,
+            -0.009967382065951824,
+            0.01646515540778637,
+            -0.005762029904872179,
+            0.007380664348602295,
+            -0.02064727433025837,
+            0.028004704043269157,
+            0.0077524082735180855,
+            -0.013591884635388851,
+            -0.0008824076503515244,
+            0.023760627955198288,
+            -0.005432881414890289,
+            0.009355553425848484,
+            0.02070923149585724,
+            0.015713922679424286,
+            0.013615119270980358,
+            0.016263794153928757,
+            0.03351890295743942,
+            0.005328328814357519,
+            0.005568413063883781,
+            0.003519949736073613,
+            0.016991792246699333,
+            -0.008557853288948536,
+            0.018029578030109406,
+            0.0008093173382803798,
+            -0.013576395809650421,
+            0.021499186754226685,
+            -0.0018684001406654716,
+            -0.013622863218188286,
+            -0.0040891822427511215,
+            -0.004337011836469173,
+            0.03271345794200897,
+            0.022041313350200653,
+            -0.004468671046197414,
+            -0.024442158639431,
+            -0.00394784240052104,
+            0.016682006418704987,
+            -0.003167567541822791,
+            0.011136826127767563,
+            0.0007952801533974707,
+            -0.01586107164621353,
+            -0.009595638141036034,
+            0.0063583687879145145,
+            0.010586955584585667,
+            -0.008921852335333824,
+            0.010346870869398117,
+            -0.0228312686085701,
+            -0.004011735785752535,
+            0.00046613195445388556,
+            -0.004844287410378456,
+            0.0053438181057572365,
+            -0.011314953677356243,
+            0.004948840010911226,
+            0.016062432900071144,
+            0.01026167906820774,
+            0.004832670092582703,
+            0.006989559158682823,
+            0.012066186405718327,
+            0.009161937050521374,
+            -0.01627928391098976,
+            0.00428667152300477,
+            -0.005072754807770252,
+            -0.02345084212720394,
+            0.005831731949001551,
+            0.00902253296226263,
+            -0.01122201792895794,
+            0.009386532939970493,
+            -0.010958699509501457,
+            0.03296129032969475,
+            0.01030814740806818,
+            -0.037391237914562225,
+            -0.0033379499800503254,
+            -0.0016244432190433145,
+            0.014885243959724903,
+            0.012298526242375374,
+            0.008550108410418034,
+            -0.017518429085612297,
+            -0.0047823297791182995,
+            -0.009835722856223583,
+            0.0012827098835259676,
+            -0.012306271120905876,
+            0.017456471920013428,
+            0.022567950189113617,
+            -0.036337960511446,
+            0.01968693546950817,
+            -0.010432061739265919,
+            -0.023652203381061554,
+            0.020476890727877617,
+            0.011500826105475426,
+            0.01942361705005169,
+            -0.009402021765708923,
+            0.018896980211138725,
+            -0.0001331114035565406,
+            0.025510922074317932,
+            -0.004228586331009865,
+            -0.022087780758738518,
+            -0.012499887496232986,
+            -0.03209388628602028,
+            -0.0051811798475682735,
+            0.006819176487624645,
+            -0.00045524101005867124,
+            0.013111716136336327,
+            0.014226946979761124,
+            -0.025712283328175545,
+            0.007295473478734493,
+            0.032589543610811234,
+            0.014637414366006851,
+            0.023311438038945198,
+            -0.011276230216026306,
+            -0.0010794125264510512,
+            -0.0026060794480144978,
+            -0.013770012184977531,
+            -0.03212486580014229,
+            -0.0017493258928880095,
+            -0.036585792899131775,
+            0.005680711008608341,
+            -0.016387708485126495,
+            -0.013096226379275322,
+            0.023264970630407333,
+            -0.034138478338718414,
+            0.0014608372002840042,
+            0.013708055019378662,
+            0.010540487244725227,
+            0.012391461990773678,
+            -0.00162637943867594,
+            0.015032391995191574,
+            -0.029243851080536842,
+            -0.010284913703799248,
+            -0.004491904750466347,
+            -0.01981084980070591,
+            -0.007338068913668394,
+            -0.018060555681586266,
+            -0.013127204962074757,
+            -0.003566417610272765,
+            -0.010060317814350128,
+            0.002247888594865799,
+            0.0032062907703220844,
+            -0.014714861288666725,
+            0.03206290677189827,
+            0.003157886676490307,
+            0.018958937376737595,
+            0.0033418224193155766,
+            0.004449309315532446,
+            0.018788553774356842,
+            -0.0026331855915486813,
+            -0.0058278595097362995,
+            0.030963165685534477,
+            -0.00032672801171429455,
+            0.008759214542806149,
+            0.019082851707935333,
+            0.004724245052784681,
+            0.0038529701996594667,
+            -0.0260530486702919,
+            0.012887120246887207,
+            -0.027447087690234184,
+            0.005080499220639467,
+            -0.0077175572514534,
+            -0.024302754551172256,
+            -0.028128618374466896,
+            -0.006571347359567881,
+            0.012058441527187824,
+            -0.02898053266108036,
+            -0.00613377382978797,
+            0.002315654419362545,
+            -0.00400399137288332,
+            -0.002673845272511244,
+            0.007241260726004839,
+            0.029150914400815964,
+            -0.033209118992090225,
+            0.008751469664275646,
+            0.005595519673079252,
+            -0.0005053392960689962,
+            0.013297587633132935,
+            0.008038961328566074,
+            0.003343758638948202,
+            0.020848635584115982,
+            0.006637176964432001,
+            0.002458930714055896,
+            -0.0025576751213520765,
+            -0.014242436736822128,
+            0.004000118933618069,
+            0.02154565416276455,
+            0.007562664337456226,
+            0.018385831266641617,
+            -0.008914108388125896,
+            0.002621568739414215,
+            -0.01455222349613905,
+            0.005552923772484064,
+            -0.002867461647838354,
+            0.004309905227273703,
+            -0.012236569076776505,
+            0.008093173615634441,
+            0.0028771425131708384,
+            -0.0043447562493383884,
+            0.018060555681586266,
+            -0.002319526858627796,
+            0.016480645164847374,
+            0.004015608225017786,
+            0.015620986931025982,
+            -0.0035412474535405636,
+            0.01178737822920084,
+            0.0005731051205657423,
+            -0.017905661836266518,
+            -0.017022771760821342,
+            -0.039249956607818604,
+            0.007702067960053682,
+            0.03531566634774208,
+            -0.0009564659558236599,
+            -0.03444826602935791,
+            0.010145509615540504,
+            -0.011942271143198013,
+            -0.009975126944482327,
+            -0.022862248122692108,
+            0.013862947933375835,
+            0.012430185452103615,
+            0.007415515370666981,
+            0.012252057902514935,
+            -0.006997303571552038,
+            0.01026167906820774,
+            -0.005216030869632959,
+            0.008356492035090923,
+            -0.0010823167394846678,
+            -0.037422213703393936,
+            0.012344993650913239,
+            0.02216522768139839,
+            -0.002635121811181307,
+            0.0016089539276435971,
+            -0.006145390681922436,
+            0.022103270515799522,
+            -0.013971373438835144,
+            -0.00650551775470376,
+            -0.00046443779137916863,
+            0.00797700323164463,
+            -0.01795213110744953,
+            0.004778457339853048,
+            0.0755879208445549,
+            -0.007775642443448305,
+            0.006122156511992216,
+            0.009270362555980682,
+            -0.015148562379181385,
+            -0.01637221872806549,
+            -0.004561606794595718,
+            -0.020260039716959,
+            0.008844406343996525,
+            -0.0018819533288478851,
+            -0.0110981035977602,
+            0.012468908913433552,
+            0.0053050946444272995,
+            0.002031038049608469,
+            -0.008379725739359856,
+            0.010811550542712212,
+            0.009525936096906662,
+            0.010432061739265919,
+            0.02360573597252369,
+            0.011477591469883919,
+            -0.00858883187174797,
+            -0.011795123107731342,
+            0.01129171997308731,
+            0.0028577810153365135,
+            0.016697494313120842,
+            0.007907302118837833,
+            0.0010648912284523249,
+            0.019501063972711563,
+            -0.002309845993295312,
+            0.02180897258222103,
+            -2.2689444449497387e-05,
+            -0.009301341138780117,
+            -0.004236331209540367,
+            -0.015349923633038998,
+            0.024271776899695396,
+            -0.0020155487582087517,
+            0.0015450604259967804,
+            -0.020213572308421135,
+            0.0013979118084535003,
+            0.013243375346064568,
+            0.011957760900259018,
+            -0.0020407189149409533,
+            -0.0051347119733691216,
+            0.0007154133054427803,
+            -0.016356728971004486,
+            0.032806396484375,
+            0.007578153628855944,
+            0.0007681738352403045,
+            0.0008838597568683326,
+            0.0009288755827583373,
+            -0.0032972905319184065,
+            0.0024608669336885214,
+            -0.004766840487718582,
+            0.009549170732498169,
+            -0.010943209752440453,
+            -0.010796061716973782,
+            0.010075807571411133,
+            -0.02402394637465477,
+            0.003566417610272765,
+            -0.00558777479454875,
+            -0.008062195032835007,
+            0.0076207490637898445,
+            0.00015537731815129519,
+            -0.007392281666398048,
+            -0.0030939930584281683,
+            0.0023408245760947466,
+            0.013429246842861176,
+            0.004406713880598545,
+            0.010850274004042149,
+            0.012956822291016579,
+            0.005359307397156954,
+            -0.006629432085901499,
+            0.015512561425566673,
+            -0.006946963258087635,
+            -0.004228586331009865,
+            -0.0012468908680602908,
+            -0.019996721297502518,
+            -0.0009419447160325944,
+            -0.004708755761384964,
+            0.0015092412941157818,
+            0.0001705036120256409,
+            -8.749050175538287e-05,
+            -0.0018490385264158249,
+            -0.011988739483058453,
+            -0.001831613015383482,
+            -0.011663463898003101,
+            -0.014498010277748108,
+            0.010253935120999813,
+            -0.0041201612912118435,
+            -0.022939693182706833,
+            -0.015349923633038998,
+            -0.020662762224674225,
+            0.02145271934568882,
+            -0.004387352149933577,
+            -0.005800753366202116,
+            0.006733985152095556,
+            -0.010215211659669876,
+            -0.015706177800893784,
+            -0.008069939911365509,
+            -0.0205853171646595,
+            0.010703125037252903,
+            0.003214035416021943,
+            0.0006791101768612862,
+            0.00882891658693552,
+            0.014397329650819302,
+            -0.01589204929769039,
+            0.0111755495890975,
+            0.012182355858385563,
+            0.011361422017216682,
+            -0.006854027509689331,
+            0.0008911203476600349,
+            -0.014103032648563385,
+            0.02596011385321617,
+            0.021623101085424423,
+            0.005169562995433807,
+            -0.04132552444934845,
+            -0.005862710531800985,
+            -0.013344055972993374,
+            -0.010524998418986797,
+            -0.012120398692786694,
+            0.011771888472139835,
+            -0.0038820127956569195,
+            0.0013117524795234203,
+            0.006288666743785143,
+            0.0015063370810821652,
+            0.004367990419268608,
+            -0.0022382077295333147,
+            -0.007709812838584185,
+            -0.03212486580014229,
+            -0.02817508764564991,
+            -0.00863530021160841,
+            0.020461400970816612,
+            0.002838419284671545,
+            -0.0009453330421820283,
+            -0.0036380558740347624,
+            0.0027048238553106785,
+            -0.005882072262465954,
+            -0.005018542055040598,
+            -0.014753584749996662,
+            -0.012739972211420536,
+            0.0101300198584795,
+            0.00802347157150507,
+            0.014195968396961689,
+            -0.011950016021728516,
+            0.0028190575540065765,
+            0.018788553774356842,
+            0.007690451107919216,
+            -0.023357905447483063,
+            0.01135367713868618,
+            -0.010912231169641018,
+            -0.009270362555980682,
+            -0.009433000348508358,
+            -0.009471723809838295,
+            0.0016980175860226154,
+            -0.008317768573760986,
+            0.003945906180888414,
+            -0.03615209087729454,
+            -0.011330443434417248,
+            -0.0013901671627536416,
+            -0.010346870869398117,
+            0.0009864765452221036,
+            0.02524760365486145,
+            0.006815304048359394,
+            0.01432762760668993,
+            -0.0006733016925863922,
+            0.017502939328551292,
+            -0.0018984107300639153,
+            0.018726596608757973,
+            0.012763205915689468,
+            -0.00797700323164463,
+            0.011717676185071468,
+            0.014738094992935658,
+            -0.0033863543067127466,
+            0.015620986931025982,
+            -0.010168743319809437,
+            0.005285732913762331,
+            -0.009556914679706097,
+            -0.003214035416021943,
+            0.002495717955753207,
+            0.030901208519935608,
+            -0.003618694143369794,
+            0.00858883187174797,
+            0.01098193321377039,
+            -0.015241498127579689,
+            0.0026951429899781942,
+            0.014962689951062202,
+            0.01141563430428505,
+            0.007760153152048588,
+            0.0024434414226561785,
+            -0.024225307628512383,
+            0.004193735308945179,
+            0.005037903785705566,
+            0.0031520782504230738,
+            -0.014366351068019867,
+            0.0021859314292669296,
+            0.015675200149416924,
+            0.008550108410418034,
+            -0.025325050577521324,
+            0.016418687999248505,
+            0.0009138703462667763,
+            -0.003909118939191103,
+            0.029119934886693954,
+            -0.011268485337495804,
+            -0.005723306443542242,
+            0.009463978931307793,
+            0.0009980935137718916,
+            0.0030029932968318462,
+            0.008937342092394829,
+            0.0228312686085701,
+            -0.003988501615822315,
+            -0.032341714948415756,
+            -0.022459525614976883,
+            0.015706177800893784,
+            -0.0047823297791182995,
+            -0.022939693182706833,
+            -0.026703599840402603,
+            -0.003113354789093137,
+            -0.02228914201259613,
+            -0.0022323993034660816,
+            -0.020058678463101387,
+            -0.014056564308702946,
+            0.001268188701942563,
+            0.029987338930368423,
+            0.016201836988329887,
+            0.008224832825362682,
+            -0.011051635257899761,
+            -0.029956359416246414,
+            -0.024271776899695396,
+            0.01810702309012413,
+            -0.007055388763546944,
+            -0.005541306920349598,
+            0.00591692328453064,
+            -0.009951893240213394,
+            -0.008480406366288662,
+            0.011802867986261845,
+            0.027137301862239838,
+            -0.002747419523075223,
+            -0.029987338930368423,
+            -0.02688947319984436,
+            -0.014335372485220432,
+            -0.01840132102370262,
+            -0.017704300582408905,
+            -0.008627555333077908,
+            0.0045035220682621,
+            0.00030300996149890125,
+            0.016186347231268883,
+            0.014962689951062202,
+            -0.000480653194244951,
+            0.0028597170021384954,
+            -2.7711375878425315e-05,
+            -0.001763847190886736,
+            0.006888878531754017,
+            -0.004933350719511509,
+            0.026997897773981094,
+            0.014614180661737919,
+            -0.006594581063836813,
+            0.018711106851696968,
+            0.028314489871263504,
+            0.020430423319339752,
+            0.003157886676490307,
+            -0.010695381090044975,
+            -0.014606435783207417,
+            0.009727297350764275,
+            -0.013421501964330673,
+            0.023264970630407333,
+            0.022552460432052612,
+            -0.01480779703706503,
+            -0.014885243959724903,
+            -0.017905661836266518,
+            0.015210519544780254,
+            -0.007562664337456226,
+            -0.008224832825362682,
+            0.012368228286504745,
+            0.022335609421133995,
+            -0.007024410180747509,
+            0.002371803391724825,
+            -0.0010232636705040932,
+            -0.021731525659561157,
+            -0.006122156511992216,
+            -0.004596457816660404,
+            -0.008666278794407845,
+            0.008681767620146275,
+            0.008131897076964378,
+            -0.001928421319462359,
+            -0.006319645792245865,
+            0.005471604876220226,
+            -0.013367289677262306,
+            0.01301103550940752,
+            -0.00789568480104208,
+            0.03379771113395691,
+            -0.009138703346252441,
+            -0.0006733016925863922,
+            -0.022598929703235626,
+            -0.011446612887084484,
+            0.00845717266201973,
+            0.0017967619933187962,
+            0.01682141050696373,
+            -0.0011626676423475146,
+            -0.005688455421477556,
+            0.009913169778883457,
+            -0.0010687635513022542,
+            -0.001450188341550529,
+            -0.006892750971019268,
+            -0.010401083156466484,
+            0.0002884887217078358,
+            0.013258864171802998,
+            -0.006025348324328661,
+            0.0018054747488349676,
+            0.006095050368458033,
+            -0.0175339188426733,
+            0.029027000069618225,
+            0.023884542286396027,
+            0.017255110666155815,
+            0.014660648070275784,
+            0.0006394187803380191,
+            0.011647974140942097,
+            -0.003572226269170642,
+            0.0038820127956569195,
+            -0.020337486639618874,
+            0.021080974489450455,
+            -0.010811550542712212,
+            0.01846327818930149,
+            0.011655719019472599,
+            -0.015233753249049187,
+            -0.005022414494305849,
+            0.021406250074505806,
+            0.009967382065951824,
+            0.00889087375253439,
+            0.0017541663255542517,
+            -0.02470547705888748,
+            0.017766257748007774,
+            0.010099041275680065,
+            -0.020399443805217743,
+            0.019547531381249428,
+            -0.009471723809838295,
+            0.01233724970370531,
+            -0.0005300253978930414,
+            0.004418330732733011,
+            0.0058743273839354515,
+            0.021499186754226685,
+            0.011570527218282223,
+            -0.03884723410010338,
+            -0.013615119270980358,
+            -0.0017357728211209178,
+            0.006857899948954582,
+            -0.0036303112283349037,
+            -0.015233753249049187,
+            -0.041635312139987946,
+            0.01604694314301014,
+            0.009998360648751259,
+            -0.01643417589366436,
+            0.004987563472241163,
+            -0.002427952131256461,
+            -0.0042053526267409325,
+            -0.008170620538294315,
+            0.003186929039657116,
+            -0.004170501604676247,
+            0.02324948087334633,
+            0.0018984107300639153,
+            -0.006946963258087635,
+            0.013258864171802998,
+            0.01344473659992218,
+            -0.021948376670479774,
+            0.0009874446550384164,
+            0.0021375271026045084,
+            0.000636998622212559,
+            0.023683181032538414,
+            -0.014707116410136223,
+            -0.0017657832941040397,
+            -0.01685238815844059,
+            -0.0013069120468571782,
+            -0.021592123433947563,
+            -0.001188805908896029,
+            -0.015187285840511322,
+            -0.008511385880410671,
+            0.025030752643942833,
+            -0.011005166918039322,
+            0.010997422970831394,
+            -0.006687517277896404,
+            0.006943091284483671,
+            -0.006904367823153734,
+            -0.01936165988445282,
+            -0.0008267428493127227,
+            0.009975126944482327,
+            -0.000512115890160203,
+            0.002941036131232977,
+            8.482827252009884e-05,
+            0.002464739140123129,
+            0.008867640048265457,
+            0.006234454456716776,
+            -0.0075278133153915405,
+            0.02489134855568409,
+            -0.012073930352926254,
+            0.0016399325104430318,
+            0.011911292560398579,
+            0.0004743606550619006,
+            -0.010424317792057991,
+            0.025588368996977806,
+            -0.02139076218008995,
+            -0.00800798274576664,
+            -0.00014799568452872336,
+            -0.0016544538084417582,
+            -0.006513262167572975,
+            0.006718495860695839,
+            0.02946070022881031,
+            0.0062654330395162106,
+            0.010447551496326923,
+            -0.0281441081315279,
+            -0.011493081226944923,
+            0.012538610957562923,
+            -0.022273652255535126,
+            -0.0007739823195151985,
+            -0.0004322490422055125,
+            -0.00619960343465209,
+            0.026378324255347252,
+            -0.02022906206548214,
+            -0.01093546487390995,
+            -0.018664639443159103,
+            0.005405775271356106,
+            -0.011121337302029133,
+            0.008294534869492054,
+            -0.0009777637897059321,
+            0.006540368776768446,
+            0.016805920749902725,
+            -0.019563021138310432,
+            0.016031453385949135,
+            0.005119222681969404,
+            -0.016217326745390892,
+            0.010222956538200378,
+            -0.017580386251211166,
+            -0.0008199662552215159,
+            0.020275529474020004,
+            -0.00400399137288332,
+            0.008751469664275646,
+            -0.003363120136782527,
+            0.005208286456763744,
+            -0.006513262167572975,
+            0.0001492057926952839,
+            -0.007249005138874054,
+            0.014428308233618736,
+            -0.016805920749902725,
+            -0.017828216776251793,
+            -0.008821171708405018,
+            0.0016709112096577883,
+            0.021297825500369072,
+            0.011136826127767563,
+            -0.010044828988611698,
+            -0.011059380136430264,
+            -0.011694442480802536,
+            0.041604332625865936,
+            0.005762029904872179,
+            0.027230238541960716,
+            -0.02261441759765148,
+            0.00545998802408576,
+            0.025170156732201576,
+            -0.014242436736822128,
+            0.0012788375606760383,
+            -0.0022788674104958773,
+            0.0053050946444272995,
+            0.011562783271074295,
+            -0.01819995976984501,
+            0.0049139889888465405,
+            -0.00019216447253711522,
+            -0.00782985519617796,
+            -0.000656844291370362,
+            0.003442503046244383,
+            0.00020196630794089288,
+            0.016325751319527626,
+            -0.004027225077152252,
+            -0.016511622816324234,
+            -0.004429947584867477,
+            -0.0003216455806978047,
+            -0.027896279469132423,
+            0.013514438644051552,
+            0.016589069738984108,
+            0.01566745527088642,
+            0.027989214286208153,
+            -0.014513500034809113,
+            0.016031453385949135,
+            0.009533680975437164,
+            0.013901671394705772,
+            -0.010610189288854599,
+            0.008952830918133259,
+            -0.01595400646328926,
+            0.007446494419127703,
+            -0.03454120084643364,
+            -0.007887939922511578,
+            -0.00802347157150507,
+            0.01984182931482792,
+            0.028763681650161743,
+            0.002923610620200634,
+            0.006493900436908007,
+            0.000196883876924403,
+            0.0024202074855566025,
+            -0.01714668609201908,
+            -0.020182594656944275,
+            -0.0003337466041557491,
+            0.0029332914855331182,
+            0.0006950835813768208,
+            -0.004495777189731598,
+            0.00472037261351943,
+            -0.012391461990773678,
+            0.0026835259050130844,
+            -0.013134949840605259,
+            0.005971135571599007,
+            -0.0059982421807944775,
+            0.009603383019566536,
+            0.0019429425010457635,
+            0.020864123478531837,
+            0.02300165221095085,
+            0.004100799560546875,
+            0.0046971384435892105,
+            -0.024333734065294266,
+            0.00920066051185131,
+            -0.013847459107637405,
+            0.012329504825174809,
+            0.019408127292990685,
+            0.008248066529631615,
+            0.006571347359567881,
+            0.01233724970370531,
+            0.012918098829686642,
+            -0.0031772484071552753,
+            0.024287264794111252,
+            0.005014669615775347,
+            -0.0009070937521755695,
+            0.002052335999906063,
+            -0.014954946003854275,
+            -0.0070979841984808445,
+            -0.0015286030247807503,
+            0.0018848575418815017,
+            -0.004731989465653896,
+            0.01906736195087433,
+            0.015256987884640694,
+            -0.01050176378339529,
+            0.017874684184789658,
+            -0.03937387093901634,
+            0.017379024997353554,
+            -0.0027377386577427387,
+            -0.0060911779291927814,
+            -0.00545998802408576,
+            0.00665653869509697,
+            0.004515138920396566,
+            0.003669034456834197,
+            0.03773200139403343,
+            -0.010408828034996986,
+            0.003963331691920757,
+            -0.005847221240401268,
+            0.007121218368411064,
+            -0.020864123478531837,
+            0.008302279748022556,
+            0.005336073227226734,
+            0.02160761132836342,
+            -0.0056497324258089066,
+            -0.0014482521219179034,
+            0.02328045852482319,
+            0.008859895169734955,
+            0.03469609469175339,
+            -0.01566745527088642,
+            -0.004418330732733011,
+            0.01048627495765686,
+            0.0207402091473341,
+            -0.00580462533980608,
+            -0.008976065553724766,
+            0.007903429679572582,
+            0.0035586729645729065,
+            0.021824462339282036,
+            -0.00515407370403409,
+            -0.0006650729919783771,
+            0.0054367538541555405,
+            -0.0060911779291927814,
+            0.009735042229294777,
+            0.011252996511757374,
+            -0.009076746180653572,
+            0.007539430167526007,
+            0.021530164405703545,
+            -0.009975126944482327,
+            -0.008658533915877342,
+            -0.009649851359426975,
+            0.0010658593382686377,
+            -0.03068435750901699,
+            0.001649613375775516,
+            0.005359307397156954,
+            0.015187285840511322,
+            -0.0004603234410751611,
+            -0.009014789015054703,
+            0.021019017323851585,
+            0.017255110666155815,
+            -0.005936284549534321,
+            -0.0028132491279393435,
+            0.0023795480374246836,
+            -0.0017154429806396365,
+            -0.006160879973322153,
+            -0.0006248975405469537,
+            -0.002656419761478901,
+            0.014730350114405155,
+            -0.0015469965292140841,
+            -0.006617815233767033,
+            -0.00756653631106019,
+            0.0033302053343504667,
+            0.010834784246981144,
+            -0.005661349277943373,
+            0.013057502917945385,
+            0.016805920749902725,
+            -0.00031704717548564076,
+            0.019671445712447166,
+            0.01037010457366705,
+            -0.010447551496326923,
+            0.015938518568873405,
+            -0.007984748110175133,
+            0.006563602481037378,
+            0.0164031982421875,
+            -0.01277095079421997,
+            -0.01246116403490305,
+            0.007210282143205404,
+            0.00665653869509697,
+            0.028918573632836342,
+            0.014792308211326599,
+            -0.02109646424651146,
+            0.014908477663993835,
+            0.015768134966492653,
+            -0.013018779456615448,
+            0.003537375247105956,
+            -0.01784370467066765,
+            0.020260039716959,
+            0.019051872193813324,
+            -0.015535795129835606,
+            -0.017224133014678955,
+            -0.014947201125323772,
+            -0.005092116538435221,
+            0.006331262644380331,
+            0.0033263331279158592,
+            -0.00617636926472187,
+            -0.027989214286208153,
+            -0.0049139889888465405,
+            -0.018633661791682243,
+            0.013870692811906338,
+            -0.008674023672938347,
+            0.0007715621031820774,
+            -0.004511266481131315,
+            0.007984748110175133,
+            -0.0033882902935147285,
+            0.032465629279613495,
+            -0.0051811798475682735,
+            -0.02832997962832451,
+            0.004398969002068043,
+            0.016356728971004486,
+            0.012623801827430725,
+            -0.029243851080536842,
+            0.010463040322065353,
+            0.004805563949048519,
+            -0.012530866079032421,
+            0.005990497302263975,
+            -0.02258343994617462,
+            -0.007632365915924311,
+            -0.0009395244996994734,
+            -0.016743963584303856,
+            -0.027121812105178833,
+            -0.008433938957750797,
+            -0.006993431597948074,
+            -0.005401902832090855,
+            -0.009525936096906662,
+            -0.008720491081476212,
+            -0.011477591469883919,
+            0.02303262986242771,
+            -0.007899557240307331,
+            -0.001866463921032846,
+            -0.0014637414133176208,
+            0.00930908601731062,
+            -0.023776117712259293,
+            -0.015605497173964977,
+            0.006389347370713949,
+            -0.002594462363049388,
+            0.022815778851509094,
+            -0.012747716158628464,
+            0.005940156988799572,
+            -0.009316830895841122,
+            0.009851212613284588,
+            0.015241498127579689,
+            0.004956584889441729,
+            0.017440982162952423,
+            -0.009889936074614525,
+            0.0021026763133704662,
+            -0.008627555333077908,
+            -0.00018986526993103325,
+            -1.027075541060185e-05,
+            0.014428308233618736,
+            0.007624621503055096,
+            0.014141756109893322,
+            0.017487451434135437,
+            -0.013553161174058914,
+            0.005003052763640881,
+            -0.0003274540649726987,
+            -0.017255110666155815,
+            0.0027725896798074245,
+            -0.007984748110175133,
+            -0.0008896682411432266,
+            0.001758038648404181,
+            6.395397940650582e-05,
+            -0.02112744189798832,
+            -0.0009124181815423071,
+            0.0018809852190315723,
+            0.0010222955606877804,
+            -0.025201136246323586,
+            0.00889087375253439,
+            -0.00889087375253439,
+            -0.0014918158994987607,
+            -0.019113829359412193,
+            0.019795360043644905,
+            0.026300877332687378,
+            -0.004619691986590624,
+            0.00036206303047947586,
+            -0.013537672348320484,
+            -0.02049238048493862,
+            0.0001489637652412057,
+            -0.011314953677356243,
+            -0.011345932260155678,
+            -0.0022169100120663643,
+            -0.003214035416021943,
+            0.008805682882666588,
+            0.0004189379105810076,
+            -0.009433000348508358,
+            -0.004956584889441729,
+            -0.011764144524931908,
+            -0.017549408599734306,
+            -0.010401083156466484,
+            0.01933068037033081,
+            -0.0018016024259850383,
+            0.0020794421434402466,
+            -0.0032508226577192545,
+            0.0005271211848594248,
+            0.011748654767870903,
+            0.013475715182721615,
+            0.013344055972993374,
+            -0.023915521800518036,
+            -0.017890173941850662,
+            -0.013320821337401867,
+            0.0029487807769328356,
+            -0.007187047973275185,
+            0.004844287410378456,
+            0.026362836360931396,
+            -0.0009695350890979171,
+            -0.01091997604817152,
+            0.002327271504327655,
+            0.019051872193813324,
+            -0.031272951513528824,
+            -0.013390523381531239,
+            -0.025077221915125847,
+            0.02225816436111927,
+            -0.013351799920201302,
+            0.011516314931213856,
+            0.010191977955400944,
+            -0.025944624096155167,
+            0.006768836174160242,
+            -0.007686578668653965,
+            0.026037558913230896,
+            0.01634124107658863,
+            0.0005934348446317017,
+            -0.00944074522703886,
+            0.013622863218188286,
+            -0.0008722427301108837,
+            -0.013576395809650421,
+            0.002201420720666647,
+            0.014645159244537354,
+            -0.014691627584397793,
+            -0.009510447271168232,
+            -0.004507394507527351,
+            0.004313777666538954,
+            0.012763205915689468,
+            -0.0030339721124619246,
+            -0.004031097516417503,
+            -0.011624740436673164,
+            0.013344055972993374,
+            -0.003930416889488697,
+            0.012383717112243176,
+            0.007380664348602295,
+            -0.014056564308702946,
+            -0.019532041624188423,
+            0.033333033323287964,
+            0.00400399137288332,
+            0.00338054564781487,
+            -0.010997422970831394,
+            0.02740062028169632,
+            -0.009897680021822453,
+            0.010571465827524662,
+            -0.006408709101378918,
+            0.0031985461246222258,
+            0.012321759946644306,
+            -0.012104909867048264,
+            -0.01981084980070591,
+            0.021468207240104675,
+            -0.00364192808046937,
+            -0.0010193913476541638,
+            0.008310024626553059,
+            -0.020182594656944275,
+            -0.00663330452516675,
+            0.002180122770369053,
+            0.009541425853967667,
+            -0.011818356812000275,
+            0.006369986105710268,
+            0.017905661836266518,
+            0.003912991378456354,
+            -0.0007672057254239917,
+            0.019996721297502518,
+            -0.01672847382724285,
+            0.011717676185071468,
+            -0.010850274004042149,
+            0.0016825281782075763,
+            0.0006065039779059589,
+            -0.0048597767017781734,
+            0.0058743273839354515,
+            -0.014536733739078045,
+            0.011988739483058453,
+            -6.244135147426277e-05,
+            -0.013979118317365646,
+            0.007601387333124876,
+            -0.013359544798731804,
+            -0.016836898401379585,
+            0.041542377322912216,
+            -0.005301222205162048,
+            -0.0038665232714265585,
+            -0.0056962003000080585,
+            0.01350669376552105,
+            0.02823704481124878,
+            -0.005336073227226734,
+            -0.01153180468827486,
+            -0.010780571959912777,
+            -0.016573579981923103,
+            0.015907539054751396,
+            -0.01810702309012413,
+            0.002495717955753207,
+            -0.010532742366194725,
+            -0.007903429679572582,
+            0.004302160814404488,
+            -0.003516077296808362,
+            0.014722606167197227,
+            0.00950270239263773,
+            0.023357905447483063,
+            0.00797700323164463,
+            -0.00027348342700861394,
+            0.010393339209258556,
+            0.011663463898003101,
+            0.003243078012019396,
+            -0.010509508661925793,
+            -0.00011949774489039555,
+            -0.02894955314695835,
+            -0.008449427783489227,
+            -0.004863648675382137,
+            -0.001485039247199893,
+            -0.00428667152300477,
+            -0.01239920686930418,
+            0.006617815233767033,
+            -0.0022052929271012545,
+            0.013909416273236275,
+            -0.009254872798919678,
+            -0.017766257748007774,
+            0.03389064967632294,
+            0.004495777189731598,
+            0.006466794293373823,
+            -0.008286789990961552,
+            -0.03705047070980072,
+            -0.0032566310837864876,
+            -0.008116407319903374,
+            -0.013529927469789982,
+            0.004538373090326786,
+            0.01283290795981884,
+            -0.014831030741333961,
+            -0.028097640722990036,
+            0.0019390701781958342,
+            -0.006683644838631153,
+            0.02978597767651081,
+            -0.007798876613378525,
+            0.00419760774821043,
+            0.0037251831963658333,
+            0.0011558911064639688,
+            0.01239920686930418,
+            -0.00845717266201973,
+            -0.0022536972537636757,
+            0.016387708485126495,
+            -0.010858018882572651,
+            0.011051635257899761,
+            -0.010966443456709385,
+            -0.02554190158843994,
+            -0.01456771232187748,
+            0.021344292908906937,
+            -0.0004823473282158375,
+            -0.00024903935263864696,
+            0.014823286794126034,
+            0.02354377694427967,
+            0.011260741390287876,
+            0.00969631876796484,
+            0.006408709101378918,
+            -0.001035848748870194,
+            0.005975008010864258,
+            0.006095050368458033,
+            -0.0024163350462913513,
+            -0.013398268260061741,
+            0.017564896494150162,
+            0.009773765690624714,
+            -0.01810702309012413,
+            0.0008514289511367679,
+            0.0036612898111343384,
+            0.0168833676725626,
+            0.006935346405953169,
+            0.009316830895841122,
+            -0.0143121387809515,
+            -0.023760627955198288,
+            -0.00014932679187040776,
+            0.002873270306736231,
+            0.015721667557954788,
+            0.010579210706055164,
+            -0.014319883659482002,
+            0.0005963391158729792,
+            0.008712747134268284,
+            0.020213572308421135,
+            -0.024209819734096527,
+            0.019702425226569176,
+            -0.004673904739320278,
+            -0.0021084847394376993,
+            0.007946024648845196,
+            0.011632485315203667,
+            -0.011872569099068642,
+            -0.00650551775470376,
+            0.006249943748116493,
+            0.003157886676490307,
+            0.03172214329242706,
+            -0.0017260919557884336,
+            0.009665340185165405,
+            -0.010997422970831394,
+            -0.0002405686245765537,
+            0.0006728176376782358,
+            -0.0006820144481025636,
+            -0.002884887158870697,
+            -0.011020656675100327,
+            0.0018974426202476025,
+            -0.020306508988142014,
+            0.0024473138619214296,
+            0.00463130883872509,
+            -0.005684582982212305,
+            -0.0032972905319184065,
+            -0.019222255796194077,
+            -0.0022672503255307674,
+            0.003293418325483799,
+            0.014335372485220432,
+            -0.031691163778305054,
+            0.004972074180841446,
+            0.010672146454453468,
+            -0.002309845993295312,
+            -0.010416572913527489,
+            0.011423379182815552,
+            -0.0026060794480144978,
+            0.011988739483058453,
+            -0.0038142469711601734,
+            -0.0004905760288238525,
+            0.025681305676698685,
+            -0.005308967083692551,
+            0.014707116410136223,
+            -0.01026167906820774,
+            -0.002993312431499362,
+            -0.028252532705664635,
+            0.005235392600297928,
+            -0.007403898518532515,
+            -0.0012081675231456757,
+            -0.004402841441333294,
+            -0.01130720879882574,
+            -0.0026990154292434454,
+            0.002122037811204791,
+            0.008000237867236137,
+            0.012910354882478714,
+            -0.0007725302129983902,
+            -0.010075807571411133,
+            0.004832670092582703,
+            0.0075355577282607555,
+            -0.015520306304097176,
+            -0.0070437719114124775,
+            -0.020306508988142014,
+            0.029662061482667923,
+            -0.013731288723647594,
+            -0.0011123273288831115,
+            -0.020864123478531837,
+            0.010160998441278934,
+            -0.0063119009137153625,
+            -0.002782270545139909,
+            0.004708755761384964,
+            -0.015109838917851448,
+            0.01344473659992218,
+            0.008751469664275646,
+            0.01450575515627861,
+            -0.02151467651128769,
+            -0.01762685552239418,
+            -0.0044725434854626656,
+            -0.002735802438110113,
+            -0.010648912750184536,
+            -0.0005334137240424752,
+            0.002013612538576126,
+            0.021251358091831207,
+            -0.00032503384863957763,
+            0.0042053526267409325,
+            -0.003214035416021943,
+            0.017224133014678955,
+            -0.010137764737010002,
+            -0.013057502917945385,
+            0.0024260159116238356,
+            0.005994369741529226,
+            0.016186347231268883,
+            0.03158273920416832,
+            -0.020693741738796234,
+            -0.00272999401204288,
+            0.0032740565948188305,
+            0.010656657628715038,
+            -0.004085310269147158,
+            6.304640555754304e-05,
+            -0.027803342789411545,
+            -0.04464024305343628,
+            -0.002422143705189228,
+            0.013057502917945385,
+            -0.015636475756764412,
+            0.012670270167291164,
+            0.008743725717067719,
+            0.010021595284342766,
+            -0.005196669604629278,
+            0.02620794251561165,
+            -0.004793947096914053,
+            0.014304393902420998,
+            0.011733165942132473,
+            0.011911292560398579,
+            0.0009874446550384164,
+            0.0007459079497493804,
+            -0.018354853615164757,
+            -0.008255811408162117,
+            -0.0015954007394611835,
+            0.01765783317387104,
+            -0.027478067204356194,
+            0.008526874706149101,
+            -0.011516314931213856,
+            0.013839714229106903,
+            0.0003666614356916398,
+            0.005661349277943373,
+            -0.01771979033946991,
+            0.012445674277842045,
+            -0.0009337160154245794,
+            -0.025325050577521324,
+            0.004805563949048519,
+            0.011276230216026306,
+            -0.010362359695136547,
+            0.006877261213958263,
+            0.019950253888964653,
+            0.0023079097736626863,
+            0.008495896123349667,
+            -0.014126266352832317,
+            0.010703125037252903,
+            0.009649851359426975,
+            -0.021080974489450455,
+            -0.01233724970370531,
+            -0.007059261202812195,
+            -0.006335135083645582,
+            -0.0035470561124384403,
+            -0.00013105422840453684,
+            0.010176488198339939,
+            -0.016000475734472275,
+            0.010099041275680065,
+            -0.006559730041772127,
+            0.006400964688509703,
+            0.00944074522703886,
+            0.001115231541916728,
+            -0.013576395809650421,
+            0.012716737575829029,
+            -0.0218709297478199,
+            0.008426194079220295,
+            -0.04287445917725563,
+            -0.005847221240401268,
+            -0.015078860335052013,
+            0.005204414017498493,
+            -0.003798757679760456,
+            -0.004453181754797697,
+            -0.006203475873917341,
+            -0.005061137955635786,
+            0.00920066051185131,
+            0.021747015416622162,
+            0.012135888449847698,
+            -0.016620049253106117,
+            -0.007551047019660473,
+            0.005425137002021074,
+            0.0004854935978073627,
+            -0.013336311094462872,
+            0.009316830895841122,
+            -0.01576039008796215,
+            0.0009259713697247207,
+            0.006339007057249546,
+            -0.01072635967284441,
+            0.013452480547130108,
+            0.004731989465653896,
+            -0.003401843598112464,
+            0.013390523381531239,
+            -0.0050417762249708176,
+            -0.0057891360484063625,
+            -0.007272239308804274,
+            -0.013739033602178097,
+            -0.003818119177594781,
+            -0.003072695340961218,
+            0.012747716158628464,
+            0.001529571134597063,
+            -0.0003243077953811735,
+            -0.00782985519617796,
+            0.009154192171990871,
+            0.002220782218500972,
+            -0.0013146566925570369,
+            0.005773646757006645,
+            -0.02300165221095085,
+            0.0001287550403503701,
+            0.004755223635584116,
+            -0.0009487213101238012,
+            -0.006246071308851242,
+            -0.012151377275586128,
+            -0.011903548613190651,
+            -0.038165703415870667,
+            0.009363298304378986,
+            0.0019197085639461875,
+            -0.0076207490637898445,
+            0.0064822835847735405,
+            0.017255110666155815,
+            0.01456771232187748,
+            0.016743963584303856,
+            -0.020693741738796234,
+            -0.02303262986242771,
+            0.005556796211749315,
+            -0.009402021765708923,
+            0.02261441759765148,
+            0.004046586807817221,
+            -0.005847221240401268,
+            0.0013243375578895211,
+            -0.013258864171802998,
+            0.008929597213864326,
+            0.005723306443542242,
+            -0.001592496526427567,
+            0.005425137002021074,
+            -0.013692565262317657,
+            -0.010904486291110516,
+            0.01048627495765686,
+            0.032496608793735504,
+            -0.0004261985304765403,
+            0.0024105266202241182,
+            0.023295948281884193,
+            0.015466093085706234,
+            0.016356728971004486,
+            -0.021111954003572464,
+            0.011167804710566998,
+            0.010060317814350128,
+            -0.0018645278178155422,
+            -0.014892988838255405,
+            0.00513083953410387,
+            -0.012716737575829029,
+            -0.010664401575922966,
+            0.014993669465184212,
+            0.007853088900446892,
+            0.017735280096530914,
+            -0.02369867078959942,
+            0.00789568480104208,
+            0.00042765066609717906,
+            0.00013710475468542427,
+            0.011524059809744358,
+            0.012956822291016579,
+            0.03370477631688118,
+            0.027152791619300842,
+            0.020662762224674225,
+            -0.009285852313041687,
+            0.014366351068019867,
+            -0.009192915633320808,
+            -0.011090358719229698,
+            -0.019470084458589554,
+            -0.015249243006110191,
+            -0.011136826127767563,
+            0.017998598515987396,
+            0.008790193125605583,
+            -0.009355553425848484,
+            -0.005947901867330074,
+            0.01586107164621353,
+            -0.003909118939191103,
+            -0.004635181277990341,
+            0.01646515540778637,
+            -0.026269899681210518,
+            0.016836898401379585,
+            -0.000745423894841224,
+            -0.014668392948806286,
+            -0.014691627584397793,
+            -0.004182118456810713,
+            -0.012670270167291164,
+            0.0003964299685321748,
+            -0.020817656069993973,
+            0.013181418180465698,
+            0.024597052484750748,
+            -0.0030920570716261864,
+            -0.004100799560546875,
+            -0.011299464851617813,
+            -0.0005111477803438902,
+            0.018385831266641617,
+            0.0034773540683090687,
+            0.009192915633320808,
+            -0.0023872926831245422,
+            0.013924905098974705,
+            0.00524313747882843,
+            0.0027958236169070005,
+            -0.0007565568666905165,
+            -0.016263794153928757,
+            0.012197845615446568,
+            0.0026835259050130844,
+            0.012135888449847698,
+            -0.022242674604058266,
+            0.007938280701637268,
+            -0.011764144524931908,
+            -0.0023698671720921993,
+            -0.02228914201259613,
+            0.0189744271337986,
+            -0.01610890030860901,
+            0.001046497724018991,
+            0.003841353114694357,
+            0.0006224773242138326,
+            0.009580149315297604,
+            0.01610890030860901,
+            -0.023388884961605072,
+            -0.0003460896550677717,
+            -0.014908477663993835,
+            -0.021917399019002914,
+            -0.01607792265713215,
+            0.004728117026388645,
+            -0.019501063972711563,
+            0.005444498732686043,
+            -0.0025557391345500946,
+            -0.015125328674912453,
+            0.018478767946362495,
+            0.014691627584397793,
+            -0.004379607271403074,
+            0.002412462839856744,
+            0.005425137002021074,
+            -0.0014405074762180448,
+            0.01971791312098503,
+            0.00013274837692733854,
+            -0.0010048701660707593,
+            -0.009425255469977856,
+            -0.0168833676725626,
+            -0.0065752193331718445,
+            -0.035005878657102585,
+            0.0260530486702919,
+            -0.004670032300055027,
+            -0.016511622816324234,
+            0.014319883659482002,
+            0.009115469641983509,
+            -0.014629669487476349,
+            -0.007059261202812195,
+            0.016294771805405617,
+            -0.024287264794111252,
+            0.02244403585791588,
+            -0.009611127898097038,
+            -0.008774704299867153,
+            0.00784921646118164,
+            0.05154848098754883,
+            0.009146448224782944,
+            0.003492843359708786,
+            0.011036145500838757,
+            0.005839476361870766,
+            -0.0013165927957743406,
+            -0.023683181032538414,
+            0.005568413063883781,
+            -0.007190920412540436,
+            -0.003601268632337451,
+            0.002536377403885126,
+            -0.004801691509783268,
+            0.031149037182331085,
+            -0.004379607271403074,
+            -0.005289605353027582,
+            0.004484160337597132,
+            -0.02196386642754078,
+            -0.0001332324172835797,
+            0.023203013464808464,
+            -2.3128106477088295e-05,
+            -0.023590246215462685,
+            -0.006571347359567881,
+            -0.015071115456521511,
+            -0.013770012184977531,
+            -0.018989915028214455,
+            0.005231520161032677,
+            -0.012755461037158966,
+            -0.0033940989524126053,
+            0.0016660707769915462,
+            -0.005940156988799572,
+            -0.002582845278084278,
+            0.009603383019566536,
+            0.05362405255436897,
+            0.014103032648563385,
+            -0.006335135083645582,
+            -0.005854965653270483,
+            0.009192915633320808,
+            -0.010524998418986797,
+            -0.019795360043644905,
+            -0.009820234030485153,
+            -0.029507169499993324,
+            -0.012747716158628464,
+            -0.0143121387809515,
+            -0.010191977955400944,
+            -0.010277168825268745,
+            0.00963436160236597,
+            -0.018122512847185135,
+            -0.007628493942320347,
+            -0.020337486639618874,
+            0.002941036131232977,
+            -0.011268485337495804,
+            -0.014714861288666725,
+            -0.008054450154304504,
+            -0.04807887226343155,
+            0.005796880926936865,
+            0.019593998789787292,
+            0.01263929158449173,
+            0.002515079453587532,
+            0.004383479710668325,
+            -0.01993476413190365,
+            -0.030792783945798874,
+            0.013839714229106903,
+            0.011283975094556808,
+            0.011733165942132473,
+            0.03283737599849701,
+            -0.011709931306540966,
+            -0.00813964195549488,
+            -0.004693266469985247,
+            -0.0017251238459721208,
+            0.012066186405718327,
+            0.00969631876796484,
+            -0.004917861428111792,
+            -0.012375972233712673,
+            0.016217326745390892,
+            -0.01981084980070591,
+            -0.023683181032538414,
+            -0.01672847382724285,
+            0.006157007534056902,
+            0.00813964195549488,
+            -0.008054450154304504,
+            0.002973950933665037,
+            0.0056962003000080585,
+            -0.007159941829741001,
+            -0.00022423222253564745,
+            0.004890755284577608,
+            -0.023048119619488716,
+            0.0008490087930113077,
+            -0.002257569460198283,
+            0.0061376458033919334,
+            0.021266845986247063,
+            -0.010168743319809437,
+            -0.0014075926737859845,
+            -0.0053902859799563885,
+            -0.0028984404634684324,
+            -0.006764963734894991,
+            0.02620794251561165,
+            0.009154192171990871,
+            -0.009882191196084023,
+            -0.01011453103274107,
+            -0.01129171997308731,
+            -0.0011278166202828288,
+            -0.017255110666155815,
+            -0.016186347231268883,
+            0.0015566773945465684,
+            -0.016712984070181847,
+            0.013971373438835144,
+            -0.00876695942133665,
+            0.00400399137288332,
+            -0.014110777527093887,
+            0.02019808255136013,
+            0.01929970271885395,
+            -0.009711808525025845,
+            0.023776117712259293,
+            -0.007709812838584185,
+            -0.038382552564144135,
+            -0.00800798274576664,
+            0.01843230053782463,
+            -0.006118284538388252,
+            0.010037084110081196,
+            -0.011314953677356243,
+            0.008209343999624252,
+            0.00015319912927225232,
+            0.0018122512847185135,
+            -0.0003112386621069163,
+            0.027555514127016068,
+            0.015535795129835606,
+            -0.005560668651014566,
+            -0.005494839046150446,
+            0.004255692940205336,
+            0.009541425853967667,
+            0.017580386251211166,
+            0.012151377275586128,
+            -0.021019017323851585,
+            -0.014257925562560558,
+            -0.016263794153928757,
+            0.011012911796569824,
+            0.015125328674912453,
+            -0.0040891822427511215,
+            0.01691434532403946,
+            0.014126266352832317,
+            -0.019098341464996338,
+            0.0019884423818439245,
+            -0.008193854242563248,
+            -0.00158184755127877,
+            -0.03029712475836277,
+            0.011043890379369259,
+            0.0074387495405972,
+            -0.0072218989953398705,
+            0.036616768687963486,
+            0.009858956560492516,
+            0.037577107548713684,
+            0.018680129200220108,
+            0.0008974128868430853,
+            0.009045767597854137,
+            -0.018370341509580612,
+            0.013065247796475887,
+            0.01500141341239214,
+            -0.012724482454359531,
+            -0.005734923295676708,
+            0.0013446672819554806,
+            0.0007551047019660473,
+            0.012631546705961227,
+            0.02004319056868553,
+            -0.01795213110744953,
+            0.009773765690624714,
+            -0.015032391995191574,
+            -0.005250881891697645,
+            0.006555858068168163,
+            -0.013932649977505207,
+            -0.008495896123349667,
+            -0.00963436160236597,
+            0.019408127292990685,
+            -0.012600568123161793,
+            0.016573579981923103,
+            -0.009595638141036034,
+            0.015520306304097176,
+            -0.029724018648266792,
+            -0.003669034456834197,
+            -0.006679772399365902,
+            -0.019593998789787292,
+            -0.012887120246887207,
+            0.009820234030485153,
+            0.010633422993123531,
+            -0.009301341138780117,
+            -0.011562783271074295,
+            0.020368466153740883,
+            -0.004185990896075964,
+            0.013529927469789982,
+            0.021189400926232338,
+            0.00737291993573308,
+            0.024132372811436653,
+            0.0005605200421996415,
+            0.014164989814162254,
+            0.005463860463351011,
+            0.010362359695136547,
+            -0.004677776712924242,
+            0.005161818582564592,
+            0.00932457484304905,
+            0.014203713275492191,
+            0.017766257748007774,
+            0.013196907006204128,
+            -0.00598275288939476,
+            0.007419387809932232,
+            0.02016710489988327,
+            0.0016254113288596272,
+            -0.030312614515423775,
+            -0.009719553403556347,
+            0.0001736498816171661,
+            0.00924712885171175,
+            0.009185170754790306,
+            3.817877222900279e-05,
+            -0.01093546487390995,
+            0.011144571006298065,
+            0.00969631876796484,
+            0.002286612056195736,
+            0.006280922330915928,
+            -0.01456771232187748,
+            -0.020957060158252716,
+            0.006106667220592499,
+            0.01798310875892639,
+            0.013917161151766777,
+            -0.014722606167197227,
+            0.022784801200032234,
+            0.006145390681922436,
+            -0.021406250074505806,
+            -0.01840132102370262,
+            -0.0009559819591231644,
+            0.0004542729293461889,
+            -0.002265314105898142,
+            0.03720536455512047,
+            -0.0011539548868313432,
+            -0.005955646280199289,
+            -0.0054754773154854774,
+            0.02775687538087368,
+            0.02049238048493862,
+            0.009177426807582378
+          ],
+          "embedding": [
+            0.0008215984936325879,
+            -0.019349418284688353,
+            -0.02322660633029795,
+            0.009747820845194852,
+            0.007353771220903641,
+            0.015252908783199839,
+            -0.012176175829980202,
+            0.051898051094966605,
+            -0.0068130353886146165,
+            0.008189212816890595,
+            0.008102287831931905,
+            0.01682801136011614,
+            0.02005557996049711,
+            -0.020779879580497183,
+            0.002332165268329781,
+            0.02337485065616377,
+            0.006074992895540845,
+            0.02576575146535689,
+            0.0007517286429767504,
+            -0.03763909942314134,
+            0.0011204729720314833,
+            0.00041212613589632806,
+            -0.06089176800314795,
+            0.012852035873395388,
+            -0.008901127092058543,
+            0.01702347055484216,
+            0.005803446395466281,
+            0.017889570275475296,
+            -0.05375047382413552,
+            0.021623778774252724,
+            0.014784317465883424,
+            -0.0018881880857885708,
+            0.0004735328554290247,
+            0.012393802850407656,
+            -0.019234442968055952,
+            -0.019566283611638122,
+            0.04407789373296535,
+            -0.0053565873209102995,
+            -0.026240866282480922,
+            0.048107902270426675,
+            -0.0054681314939387235,
+            -0.0009441397948041965,
+            -0.015205079626883548,
+            0.006885981841893229,
+            0.01403724256866746,
+            -0.026086548246792754,
+            0.0034778946051893067,
+            0.00044587533948775473,
+            0.032851328435891634,
+            -0.03882097333255334,
+            -0.011126344604254894,
+            -0.01648943336384904,
+            0.035810456555802166,
+            -0.019384589526165754,
+            -0.019430702768170285,
+            0.014297440272239852,
+            -0.007883453204663151,
+            0.050996900793086844,
+            0.06007149434153715,
+            0.03406832053483379,
+            0.025193439934671852,
+            0.00027785129663604897,
+            -0.008014391056168083,
+            -0.018806730908490114,
+            0.027193788903098558,
+            -0.006195006670328595,
+            -0.020466000344352826,
+            0.0068939884219589185,
+            0.017252276167086602,
+            -0.045073444648704136,
+            -0.049242269917908754,
+            0.023021140866008313,
+            -0.015676872472071106,
+            -0.024497517908898602,
+            0.005873633718973869,
+            -0.022828585938203683,
+            0.011748341004940436,
+            0.04624205727927451,
+            -0.015442165499674641,
+            0.018565551071491767,
+            0.015246505395507096,
+            0.004119269796224144,
+            0.004862735250819252,
+            -0.000145323321960046,
+            0.05155838952887077,
+            0.0027052423345323936,
+            -0.016218063670586664,
+            -0.07462470920944148,
+            0.043695974746295135,
+            -0.027608273720377708,
+            -0.03860912774695494,
+            0.002823110565150435,
+            0.026471469849508766,
+            -0.0026063577189074103,
+            -0.029230652014151735,
+            0.05684164470477843,
+            0.00022182868300572677,
+            0.015590845552956364,
+            -0.024939617549529242,
+            0.01035803286203917,
+            -0.010225298012384327,
+            0.019571847648759157,
+            0.01841613041441054,
+            0.0156981470538354,
+            0.021764297694793343,
+            -0.018831230007033416,
+            -0.014419529638789993,
+            -0.0029965475899652273,
+            0.028881185012095884,
+            0.09163526460163937,
+            0.022644302207811818,
+            -0.017933909936330133,
+            -0.04663928709016869,
+            0.004962951444105128,
+            -0.041333041174526026,
+            -0.027545628007617527,
+            -0.037142394579894954,
+            -0.013358143162788402,
+            0.02719686432396677,
+            -0.0034971800937504205,
+            0.04413635817371732,
+            0.04903857110328917,
+            0.03186067155630788,
+            0.007831051050269414,
+            -0.025257538623313574,
+            0.023517513773109414,
+            -0.0016527403209505964,
+            -0.0016101860707429347,
+            0.014448515185903624,
+            0.009905018312477684,
+            0.0024114582235163955,
+            0.00172043851384822,
+            -0.03818316221102173,
+            -0.02304537008110578,
+            0.01024969233570089,
+            -0.01901116356327339,
+            0.019791121460848986,
+            0.07985067340821365,
+            -0.051903191272236665,
+            -0.025924646976556942,
+            0.014766511532849252,
+            -0.002073086076894999,
+            0.030959269926800852,
+            0.006394417986617089,
+            -0.028177586693713442,
+            0.010725285336674526,
+            0.0009614052276924192,
+            0.07500253788978564,
+            -0.00036794182149960157,
+            0.004058458332826519,
+            -0.013218269166951234,
+            0.027613539268683245,
+            -0.012995170442480582,
+            0.023388707018488407,
+            0.026389423313105165,
+            -0.003068639956061266,
+            -0.02661967649632026,
+            0.026145274570635913,
+            -0.03300591322174331,
+            -0.004556072066733024,
+            0.020045722084112896,
+            0.0033437343914784865,
+            -0.010224220905483085,
+            -0.0025025498684079703,
+            0.03460103245395426,
+            0.04514362357278504,
+            0.03797031988058962,
+            -0.004586662475576956,
+            -0.06351723722596923,
+            0.06037983484369874,
+            0.019415753124363902,
+            0.028850618647994476,
+            -0.0034810237045849006,
+            0.02957938154808314,
+            0.02642652523231673,
+            0.013187653724133275,
+            -0.007856320804109569,
+            -0.007065882659482447,
+            -0.016672784625197085,
+            0.029020161154209707,
+            -0.013243058921970134,
+            -0.014900932834241732,
+            0.006001769901896033,
+            0.008648456186944271,
+            0.03484763413464497,
+            0.012279368984306564,
+            0.02379643508687133,
+            0.04933044959399941,
+            -0.021047723935647383,
+            -0.007264823701706426,
+            0.0007058774795716792,
+            -0.010512071832195085,
+            0.027489671668749157,
+            0.009731157570361245,
+            -0.03195919640758579,
+            -0.0033626031972691915,
+            -0.04282419484651207,
+            0.01873977910485268,
+            -0.014398285316695653,
+            0.012854631447494207,
+            -0.0017372959978092432,
+            -0.037331867825735776,
+            0.001088209292238142,
+            -0.012345684323522151,
+            0.037096063326414734,
+            0.005050219690024652,
+            0.0008906051515353004,
+            -0.003697709449630797,
+            0.006548479142737107,
+            -0.010727663712299397,
+            -0.007486226350610755,
+            -0.02704618481069485,
+            -0.019752552016312926,
+            -0.014632581296303486,
+            -0.0037501138119724717,
+            0.014588411795703245,
+            -0.007119137071719376,
+            0.016440258608998535,
+            -0.03686645352111013,
+            -0.023546402448713415,
+            0.005628703432152666,
+            0.024988145751166143,
+            0.011971199595808573,
+            -0.010868778329190568,
+            -0.03270046433624425,
+            -0.021950559285357267,
+            -0.011889914022657738,
+            -0.005123971523701009,
+            0.02619713302280345,
+            -0.021990810305904974,
+            0.01165257474485436,
+            -0.009117901804315863,
+            -0.007492082755249399,
+            0.010725902070291232,
+            0.01862811996181951,
+            -0.026694281868322685,
+            -0.01623776415010199,
+            -0.009403997538684658,
+            -0.02207329027097839,
+            0.025939540497196416,
+            0.014178803271471762,
+            -0.001141821318145744,
+            -0.006281187017652108,
+            -0.035993365198554964,
+            0.0026192776111449633,
+            -0.015254976771874396,
+            -0.0116872453009606,
+            -0.015780016491993135,
+            0.021421328283124968,
+            0.02889432250124136,
+            -0.01871009416296692,
+            -0.026540472640514536,
+            0.015703370682110186,
+            -0.0002849235245164438,
+            -0.02594930889995505,
+            0.04491828744809679,
+            0.044063522197410734,
+            -0.037441864491547405,
+            -0.007005339187980476,
+            -0.030381794393931814,
+            0.06901083845539985,
+            -0.017604082400603037,
+            -0.04060591381445593,
+            0.06044850935418443,
+            -0.01756490171206357,
+            0.025539717592705295,
+            0.03452018789549939,
+            0.0016217577522523927,
+            0.00018825054264059277,
+            -0.002563606092068712,
+            0.04770277039047993,
+            -0.0032010143255574255,
+            -0.03425328601475849,
+            -0.02105909767072799,
+            -0.00840881022563103,
+            -0.028699447677549002,
+            -0.015925160861589006,
+            -0.0055750742281166,
+            0.017513987082479483,
+            0.010399718005952185,
+            0.02267201252273823,
+            0.01258511878601025,
+            0.0015081978484714903,
+            0.01874981572091931,
+            0.011158889421057756,
+            -0.002607810686437211,
+            -0.003732023411827874,
+            -0.007787189972165309,
+            -0.05524012485122728,
+            -0.025110689689279583,
+            -0.03946383236703947,
+            0.004443572529325475,
+            -0.018745077598477586,
+            0.005926160888274727,
+            0.039430331582190524,
+            -0.0049500324516682,
+            0.004352795787899792,
+            0.015566139991780173,
+            0.013148156356408315,
+            -0.015279560337042242,
+            -0.03316846614871165,
+            0.0001592209771257332,
+            0.018362411161743347,
+            -0.012706582418607377,
+            -0.03424477956681259,
+            -0.001991022841226589,
+            0.0033467286047492175,
+            0.03357019284647071,
+            -0.024359110796680374,
+            0.00359117029951049,
+            0.005061818038334802,
+            0.011359582931760315,
+            -0.026793992746244076,
+            -0.018170136079817746,
+            -0.017421603453826397,
+            0.03996608926471779,
+            0.02587013058343799,
+            0.015763410145648683,
+            -0.04512536449983358,
+            -0.02293473890636517,
+            0.048575083783480195,
+            -0.009589860756342604,
+            -0.06486262416944208,
+            0.03447666009581266,
+            0.016105654870253654,
+            0.003526582643722421,
+            -0.01354754623151814,
+            0.05401575248587799,
+            -0.03563190295559527,
+            -0.017365166993914202,
+            -0.0046288709698227735,
+            -0.033657212157212645,
+            -0.007686252059880836,
+            -0.001385273642139782,
+            0.048229062789530236,
+            -0.029083855744728694,
+            -0.030465559177859865,
+            0.0024368661249980416,
+            -0.024144031154493057,
+            0.021693278579520123,
+            -0.016405088055634124,
+            -0.018439935250344958,
+            0.006948603359995389,
+            -0.033943037395199695,
+            -0.003007796423804453,
+            -0.016783195426536633,
+            -0.02270589491206155,
+            0.020874438753828814,
+            0.016724666903417947,
+            0.027033184180344852,
+            0.0019847647852695785,
+            0.017749524526371133,
+            0.0204661893602285,
+            -0.005604929968008821,
+            0.028462234633072944,
+            -0.020790459496727066,
+            -0.04183816355941877,
+            0.03946487892663695,
+            0.01192861758684541,
+            -0.0018024841275380242,
+            0.01077082591611702,
+            -0.01475239020295694,
+            -0.03952356245715099,
+            0.03683950018557526,
+            -0.056767672022551764,
+            -0.007507600090017693,
+            0.009530544352246083,
+            -0.00788856051420074,
+            -0.011204694027562521,
+            -0.010587701466318598,
+            -0.028604940377315504,
+            0.023749537635431462,
+            -0.013672419168235691,
+            0.01953449229707597,
+            -0.011449939491554354,
+            -0.012238180529977605,
+            0.05795536247491174,
+            -0.021518581774137322,
+            -0.040672955795925836,
+            0.026066659238628543,
+            -0.014061106927896723,
+            0.03328357230532424,
+            0.03957107406164427,
+            0.030670947051333917,
+            -0.004278577327978564,
+            0.025885946540364826,
+            0.0319106101668716,
+            -0.003462160036198642,
+            -0.04686868330851868,
+            0.00555303933159587,
+            0.02156143971867743,
+            -0.03739529489075782,
+            -0.01025626126223168,
+            -0.02141970191890338,
+            0.022134910980548407,
+            0.017807484142010496,
+            -0.007232643256393455,
+            0.011318041192005274,
+            -0.010489012862455913,
+            -0.013003217163997554,
+            -0.00847279880392288,
+            0.02357274819910594,
+            -0.0016485066139744577,
+            -0.005873113634992993,
+            0.006424249429163587,
+            0.004008703814301658,
+            0.0054639858216457724,
+            0.010198079500408441,
+            0.017694720520798957,
+            -0.017087021465208255,
+            -0.0025558869389290554,
+            -0.016503579212015177,
+            -0.0420082404922411,
+            -0.003563173424359332,
+            0.022924161700566468,
+            0.012345933179990547,
+            -0.00729908668365632,
+            -0.004234683369642856,
+            -0.0022962545569187934,
+            -0.020666205489143392,
+            -0.014796149043465308,
+            0.024562653223873056,
+            0.030217494971086438,
+            0.007810961433007407,
+            0.007781263640449456,
+            -0.0030348559388780655,
+            0.007310876971216526,
+            0.016099591640649783,
+            0.02311929597780174,
+            -0.04160162038353416,
+            0.027152100845897007,
+            -0.00025396315502635146,
+            0.023717123370023598,
+            -0.04186918759097667,
+            -0.017988531857836703,
+            -0.03106663560912813,
+            -0.023595079699181436,
+            0.02437134274097885,
+            -0.02317559303912261,
+            -0.012327263775257306,
+            0.000989566600374836,
+            0.00954921216572729,
+            -0.005155079395685017,
+            -0.017508404061379164,
+            -0.04582321461886958,
+            -0.0014367264231457263,
+            -0.030417229142190933,
+            -0.017829710557332023,
+            -0.001056472801554354,
+            0.005016912084754242,
+            -0.06276644565618253,
+            0.0109965681725148,
+            -0.07270669271977365,
+            -0.004542796125637244,
+            0.040139998232966566,
+            -0.017890441783630643,
+            0.055909102081208445,
+            -0.05029104308417623,
+            0.01673962668706085,
+            -0.04507293803055318,
+            0.015385506084493484,
+            0.025677372277539627,
+            0.030235171586316813,
+            -0.022189978375852563,
+            0.008461188938299745,
+            0.008033707483259569,
+            -0.045089505755429415,
+            -0.04144698743458133,
+            -0.013640256640978024,
+            0.01036782197518379,
+            0.0012696958402094232,
+            -0.016090728932978384,
+            0.023838437877040163,
+            0.011946482002767135,
+            0.03342604921859589,
+            0.04211801335157251,
+            0.002708015395637621,
+            -0.047156646610976634,
+            -0.013104464812562618,
+            -0.003933116891120681,
+            0.03155321708116329,
+            -0.018866109579446184,
+            0.015376812010630058,
+            -0.01955557376085726,
+            0.016255217383093355,
+            0.017868332102274922,
+            -0.020325179506828335,
+            0.013434364256590097,
+            0.004176904701252156,
+            -0.03877060944639854,
+            0.0006475542749369515,
+            0.00042468300294954125,
+            -0.02116992978038693,
+            0.0032654387271108293,
+            -0.00575003786159731,
+            -0.05149954071159492,
+            -0.031382926943128234,
+            -0.015730563640253065,
+            0.0382312502769352,
+            0.004297939848910064,
+            0.019724983545557365,
+            -0.017907869879607064,
+            -0.005165628493894463,
+            0.03698976707579373,
+            0.008501001373266799,
+            -0.01725527821298066,
+            0.006879620077191753,
+            0.02455792200929138,
+            -0.03224193972301503,
+            -0.039181328884413266,
+            -0.0008206986106407542,
+            -0.023306758893533952,
+            0.01672946960876128,
+            -0.02658587698869307,
+            0.005873620623736886,
+            0.0025331170979801232,
+            -0.03490809296591325,
+            0.02456764314036355,
+            0.003291113462353163,
+            -0.004303423672882112,
+            -0.020359544091059775,
+            -0.016892415165120203,
+            0.017174797487220006,
+            0.04391489511478065,
+            0.007327321684948843,
+            -0.003943713832544148,
+            0.00862684231313155,
+            -0.006361172785753631,
+            0.05826688726534012,
+            -0.02284956969636637,
+            -0.026680023999604643,
+            0.027527436347931256,
+            -0.022597669279069723,
+            -0.006277863878949772,
+            -0.017343446085471857,
+            -0.012065192152203348,
+            -0.0004183712490442143,
+            0.019339311388446943,
+            -0.019423562920897294,
+            -0.010212527447243049,
+            0.004783006556748757,
+            0.012824133870627983,
+            -0.004855945108275706,
+            0.006239949458562236,
+            0.006847581690471813,
+            0.03094005096378827,
+            0.026993131409761086,
+            -0.011331881651539968,
+            0.03297014291876675,
+            0.010265600643742144,
+            4.666113318251278e-05,
+            0.001970879281172325,
+            0.002523664335535017,
+            0.0034107811520368574,
+            -0.03143890864030425,
+            -0.01624584278940617,
+            0.013468729048885906,
+            0.005233309338223403,
+            -0.004550544967856448,
+            -0.023783653541634683,
+            -0.009402494239255264,
+            0.03334923712760375,
+            -0.003260424721959948,
+            0.02748395114281811,
+            -0.0058374897745283065,
+            -0.01985840930273332,
+            0.017595418271610163,
+            0.02881743246631328,
+            -0.05928552440605384,
+            0.017062540894682354,
+            -0.0011508547522925793,
+            0.008953761538519997,
+            0.01610561914042553,
+            -0.008894160513648953,
+            0.013301991014533684,
+            0.04080863883897929,
+            0.010738051773755976,
+            -0.023345002072886145,
+            0.004215726310838674,
+            0.01541749448570608,
+            0.01291813299324752,
+            -0.022747942296528025,
+            0.0008640791861357522,
+            0.011008522066277018,
+            0.005893793007012561,
+            0.0026106450193140156,
+            -0.006278394080583205,
+            -0.025659188287146268,
+            0.024180550333596625,
+            -0.0044491292740074925,
+            0.026967973346698617,
+            0.04390909273429254,
+            0.009596855239692042,
+            -0.03686020608700941,
+            -0.013109951664921729,
+            0.0002799680359670127,
+            -0.0038537577602593366,
+            -0.0037552094562213916,
+            0.011105295862811114,
+            -0.04185615640149602,
+            0.019950890919369135,
+            0.006453241355231516,
+            0.015314645427201533,
+            -0.020031193483734364,
+            -0.024530173488182337,
+            0.0031949828520724337,
+            -0.01492645564652531,
+            0.012875827757040941,
+            0.005057170413681063,
+            -0.004319834710426384,
+            0.005386862713494232,
+            0.00030848054013413157,
+            0.008384941028326983,
+            -0.010811526993301901,
+            0.03175068297908541,
+            0.004680987487229685,
+            0.009264547472281849,
+            -0.02856314241700808,
+            -0.021083721348429348,
+            0.020113396771629096,
+            0.03572366771051171,
+            -0.01708529918264109,
+            0.03346921724975398,
+            -0.009042046660366718,
+            0.0033219155387035227,
+            0.0151720168318272,
+            -0.015805318210244414,
+            -0.008057236707694303,
+            -0.0005483009439015931,
+            -0.002917527004423299,
+            -0.014538511124345527,
+            0.007302979537905532,
+            -0.0208529958115422,
+            -0.02965227242147972,
+            0.020064178921373875,
+            -0.00011892622932349096,
+            0.016455188521316146,
+            0.010335044657747514,
+            -0.007816155992852236,
+            -0.0052262330850905165,
+            0.006959344748883874,
+            0.04432147882994085,
+            0.026734484553994495,
+            0.05547661378166294,
+            -0.005787997952888261,
+            -0.004559444854138438,
+            -0.0082295581123363,
+            -0.006588172755707144,
+            0.04524009609473585,
+            0.015915003261638767,
+            -0.004910285095602864,
+            0.017495607570191165,
+            0.01557886571266685,
+            0.03858259260136037,
+            0.0027648525902558316,
+            -0.016912796336835408,
+            0.04014755669075221,
+            -0.004077405782937414,
+            0.008616827127381418,
+            0.011102692199778469,
+            0.0024185742218798562,
+            0.00807141794538637,
+            0.02996274720815659,
+            -0.012429186019907761,
+            0.01680942269506771,
+            -0.04571916592313758,
+            -0.02775530246832996,
+            0.014370280587419414,
+            -0.004931905540824582,
+            0.03419959665162407,
+            0.01668521021731861,
+            -0.023105305446763272,
+            -0.024761385385514877,
+            0.021529428247380984,
+            -0.002603042335561961,
+            -0.024135326707477325,
+            -0.015906833215844473,
+            0.003286605488908195,
+            -0.0023437997044735617,
+            0.0060251848371355166,
+            -0.013481652076194643,
+            0.0002908426939460218,
+            0.01953362396253743,
+            0.0038544889405965057,
+            0.020760460182873287,
+            -0.016224396121239458,
+            -0.00259233446682179,
+            -0.009282861237653619,
+            -0.02952130765234087,
+            0.017988370145301916,
+            0.006138236653154911,
+            -0.007607182955958238,
+            -0.009516767050028553,
+            0.0048347778733230365,
+            -0.005819209642901871,
+            -0.0025851701689263706,
+            -0.01073014664868326,
+            0.012433465684360061,
+            0.025784014942833097,
+            -0.016875624785552303,
+            0.007944349714123104,
+            8.651589872318479e-05,
+            -0.022306044741692753,
+            -0.022127016745066307,
+            0.020143472307319017,
+            0.002792646249762062,
+            0.008768813346304779,
+            -0.004074214276300631,
+            -0.00015024341785520542,
+            0.002990550758627699,
+            0.01951746553838439,
+            0.03896451381932467,
+            0.004232673242619471,
+            0.027876586543280867,
+            0.0145860303663199,
+            0.03994234200854897,
+            0.009657017763624678,
+            0.00696607158576712,
+            -0.026055387694656488,
+            0.02819890357011778,
+            0.00018297502510753014,
+            0.043190622821775236,
+            0.022488402221902314,
+            -0.0407730101161195,
+            -0.0021946518523759066,
+            0.01669520746276111,
+            0.013212623257794862,
+            0.017009844136059122,
+            -0.004537512994041156,
+            -0.014041213424740049,
+            0.028135404528975894,
+            -0.022859586430516003,
+            0.0008637006862481223,
+            -0.004486332934188017,
+            -0.025031795510001887,
+            0.007351370254175368,
+            -0.004567816916440959,
+            0.030025413528233364,
+            0.0014976740344892735,
+            -0.022285096307825804,
+            0.022837170656498543,
+            -0.009551269792616045,
+            0.011045530488684207,
+            0.008809047185288755,
+            0.020065001354321994,
+            0.031993362733068575,
+            0.009143210434415064,
+            0.02133640814803879,
+            0.012241777474889248,
+            0.014771538127546474,
+            -0.008804192497858033,
+            -0.02341427260922065,
+            0.023167424359504862,
+            -0.0011598649565273595,
+            -0.029627604902846688,
+            0.0021991474725787303,
+            0.003261878079344105,
+            -0.0005935713331682358,
+            -0.009144689470851402,
+            -0.005580557862988033,
+            -0.007240836451875213,
+            0.019631377011298905,
+            -0.007085213886352692,
+            0.007017573363709754,
+            -0.004074171791476129,
+            -0.00822871835665378,
+            -0.0421101573155962,
+            0.004210914075286222,
+            0.003408427700622687,
+            -0.01752179330665445,
+            0.006384440404174512,
+            0.011121481025573962,
+            0.043020575455262985,
+            -0.022809979913698254,
+            -0.0005737138409049003,
+            0.019206623212867476,
+            0.0009562551889820241,
+            -0.023427467810802533,
+            -0.038931688376706186,
+            0.007172345672967499,
+            -0.03467050750542982,
+            0.023185301620612612,
+            -0.0007075596339842658,
+            0.02387977204565927,
+            -0.03043468414449851,
+            0.013909304572370437,
+            0.004784668158421625,
+            0.012524879174804915,
+            0.021654512058639427,
+            0.03142146922870715,
+            -0.02053473475540882,
+            -0.050409862638621725,
+            -0.005406667316025549,
+            0.00035138762584085976,
+            0.022018487016014856,
+            -0.026014580970974616,
+            0.00025633004699302783,
+            -0.003349094027291921,
+            -0.02401366769212778,
+            -0.01735996745614562,
+            0.01962340791446025,
+            -0.050011480783403936,
+            0.015042913576492952,
+            0.0262737424555697,
+            -0.02507473901367252,
+            -0.05408889321034101,
+            -0.018988587082325706,
+            -0.028217893770659686,
+            -0.009285770094319318,
+            -0.00661159230987788,
+            0.028597405154740078,
+            0.003525576256054959,
+            -0.011462219358952466,
+            0.025401668390029348,
+            0.006155837251830549,
+            0.0026050693705924593,
+            0.015985722392086222,
+            0.007289910674212967,
+            -0.011452511453623235,
+            -0.014599577191293152,
+            -0.04558584556100621,
+            0.010197536827229795,
+            -0.0045431962637690155,
+            -0.019372598050508005,
+            -0.030147301333718016,
+            -0.0137828704002156,
+            -0.027060515748528666,
+            -0.0017291966950107505,
+            0.053783684195381654,
+            0.03292494290301388,
+            0.005560172296221419,
+            -0.0026085670565688026,
+            -0.010466890489356092,
+            0.036247483457995895,
+            -0.006789746401727701,
+            0.03662690472838588,
+            0.001784522073210422,
+            -0.0242673253380092,
+            0.01169382884548301,
+            0.02892627181542449,
+            0.009858451662166387,
+            -0.0005592283645355213,
+            -0.006160132931009508,
+            -0.008782830470079547,
+            0.0031207630972208394,
+            0.025613233596604418,
+            0.003268170392795548,
+            -0.010062991425127038,
+            0.018106967407336815,
+            0.004480291699976396,
+            0.02010374270557836,
+            0.03555357587371324,
+            0.023489837757983054,
+            -0.021782173652026497,
+            -0.026581762796336082,
+            -0.0013561486781077014,
+            0.0072266106510934505,
+            0.03551967295942857,
+            -0.01651315514034218,
+            0.0001274396319698452,
+            -0.008272735507120127,
+            -0.00894931703949637,
+            0.011149952898504355,
+            -0.0029671593121986116,
+            -0.0030752010285951576,
+            0.01662711184497578,
+            -0.0011672263615879709,
+            -0.033448222499147764,
+            0.0078288560210653,
+            -0.023496331346917084,
+            0.0051040606404415545,
+            -0.004113104164416575,
+            -0.007076525381595441,
+            -0.002157548492138289,
+            0.006302962169276006,
+            -0.008065841090139872,
+            -0.00035242603988955366,
+            -0.01586406637763306,
+            0.008271894439780969,
+            0.007438312999409211,
+            -0.0069617481460406095,
+            0.011402932583388205,
+            0.002666895909532698,
+            0.0005219881149353855,
+            -0.012159425670629751,
+            -0.012042145306711322,
+            -0.00957276588761916,
+            -0.03401863879157551,
+            -0.02604353712472079,
+            -0.024949533021906844,
+            -0.0014209149651479556,
+            -0.006566685925666972,
+            -0.03187796739029736,
+            -0.01209152137926458,
+            -0.011110763124407752,
+            0.007756961097537267,
+            -0.013260548545917543,
+            0.005433285079114047,
+            -6.399098520041351e-05,
+            -0.0044960488826674555,
+            0.016615636438440516,
+            -0.003643291599919672,
+            -0.019488010211707343,
+            -0.03927818878149417,
+            0.008209336177358258,
+            -0.02805593125673831,
+            -0.0038816771550190788,
+            0.007607119219721711,
+            -0.00528681677614909,
+            -0.029502897838743516,
+            -0.009149794737430706,
+            0.008767001199311985,
+            0.003507217008280066,
+            -0.01773349756494436,
+            0.03579670773605949,
+            0.026971617790626075,
+            0.011875179879209696,
+            -0.019891342676443626,
+            0.0508929813275565,
+            -0.0047010735337056275,
+            -0.012544935684845649,
+            0.018297503869683027,
+            0.02355022268131519,
+            0.03273167043015095,
+            -0.024830243320788693,
+            0.0009370814985214882,
+            -0.021937793653233454,
+            0.00535662615596573,
+            -0.035009915581788176,
+            0.0028595659314725012,
+            -0.014395027018699446,
+            0.015596459769560532,
+            0.03397372881171701,
+            -0.013254226072981457,
+            0.011195747181764677,
+            -0.016299384375589934,
+            0.00730244731485402,
+            0.00569944090461748,
+            -0.012916284280345044,
+            -0.005812311882399539,
+            -0.022902815152116493,
+            0.020140276669979015,
+            0.005809799950176385,
+            -0.0008130655714656685,
+            0.0104085826652653,
+            -0.044723432238900435,
+            0.03286429597337671,
+            0.008730225944501239,
+            0.017599189944921396,
+            0.0029762446259779998,
+            -0.0003068671951232415,
+            0.011611551606224214,
+            0.011228401175698422,
+            -0.016029727902740706,
+            0.004345913009102136,
+            -0.004613061137378433,
+            -0.03850533909474302,
+            0.009363351681220008,
+            -0.03631184057978666,
+            0.014427784677340159,
+            0.016026872002233575,
+            -0.004170582842643461,
+            -0.025166189571409366,
+            -0.005428471012333673,
+            0.010827910811085384,
+            -0.009783069664327864,
+            -0.009987805234276443,
+            -0.04163358183060102,
+            -0.025236440364963034,
+            0.034097508113604576,
+            -0.002119474613596975,
+            -0.025826403511707147,
+            -0.010830738112107807,
+            -0.011351132636491575,
+            0.020207687645133,
+            -0.006542026283946858,
+            -0.0006263975804936843,
+            -0.04150011545311013,
+            -0.0073396962917864515,
+            0.011192225907726928,
+            0.027746213794734322,
+            -0.035970017302005566,
+            0.02708126943501373,
+            0.002105633542220187,
+            0.014262674845755191,
+            -0.01626664983003942,
+            -0.005631805361903626,
+            -0.013990539604372344,
+            -0.035877093361375685,
+            0.016487723013906077,
+            -0.008607758248055054,
+            0.004714436224135091,
+            0.04458009880881818,
+            -0.0036872068506712727,
+            -0.009466125348499464,
+            0.007600705817195143,
+            -0.0034590883553048063,
+            -0.010351576085287273,
+            -0.0028564505793999295,
+            0.001959661298507,
+            -0.007080697863126256,
+            -0.04183476693971616,
+            -0.00499888976530217,
+            0.04684391782633148,
+            0.006361744381879857,
+            0.013276793765092465,
+            0.013516220521120684,
+            -0.018335746557932003,
+            -0.045577810515431215,
+            -0.0071616111758423025,
+            -0.0032763880176386808,
+            -0.022708028392758567,
+            -0.0240284945459861,
+            -0.008735040856657758,
+            0.005844145402902923,
+            0.006359368165686724,
+            -0.01120763599008222,
+            -0.022676131361171057,
+            -0.027085968700452136,
+            -0.010526103176253857,
+            -0.02123944716136792,
+            -0.010792145796733375,
+            -0.027142270190771887,
+            -0.028742234538796804,
+            0.02880426953386151,
+            0.007575182232859564,
+            -0.004084088301176117,
+            -0.010005765476121966,
+            -0.01255640546813973,
+            -0.0004774761617900028,
+            -0.0025307994917461232,
+            -0.005822065709999144,
+            0.0038614286313282164,
+            -0.011390482620811868,
+            0.023152086963976554,
+            0.0021220701260546895,
+            0.018329162172128138,
+            -0.05772324571231113,
+            -0.002709083851090071,
+            0.021497896395795905,
+            0.0033025410488410913,
+            -0.005412445141910314,
+            0.029352650459792132,
+            0.011223947499453582,
+            -0.006539460282439711,
+            -0.031355428613718234,
+            0.020942976176820262,
+            0.024122993515785947,
+            0.00817615634503138,
+            -0.012994160510030828,
+            0.01221095914384204,
+            0.01629778384790159,
+            -0.004494985018246821,
+            -0.011161098655072933,
+            0.005586857967039181,
+            0.020941039642372112,
+            0.013580622629911925,
+            0.0013398071867077221,
+            0.030905022925366163,
+            0.0006862084391508121,
+            -0.03521231612862609,
+            -0.01908191995763771,
+            -0.01275995164030872,
+            -0.003584331305986364,
+            -0.015264425022763126,
+            0.005294329654809353,
+            -0.002355465010876059,
+            0.006335052149475905,
+            0.022655379849785256,
+            0.03593231939948758,
+            0.0146937743203089,
+            0.00937330328448886,
+            0.003167462724350819,
+            -0.0013235176101076284,
+            0.012102872918129403,
+            0.008638011773075119,
+            0.0350701240268404,
+            -0.014653221697693864,
+            -0.011003994230902835,
+            -0.0037767288519369537,
+            0.010100131061186566,
+            -0.004113527518662366,
+            0.020414393256136372,
+            0.03274709615649469,
+            -0.0037222949335916876,
+            -0.005604552180246868,
+            -0.006160118231975786,
+            0.0041467454414452565,
+            0.01807879127455274,
+            -0.01252134965321004,
+            0.004364858987501461,
+            0.005549300665159658,
+            -0.0071897162833861,
+            0.018567059255813558,
+            0.0002622074720860951,
+            -0.012983550838417323,
+            0.03266376564402191,
+            0.11449509387240629,
+            0.0017251916835047628,
+            0.007125215166916139,
+            -0.040229997815548584,
+            -0.020537987527148888,
+            -0.00245728735683473,
+            0.005779972828836593,
+            0.007004732428236457,
+            0.02225914982152608,
+            0.006505827715591914,
+            0.0006708045351142873,
+            0.031708522931495536,
+            -0.017334083511709092,
+            -0.002522861743921599,
+            -0.01745656559688129,
+            0.013890451996979048,
+            -0.014312605700272691,
+            -0.013784963572164409,
+            -0.01572417853955075,
+            0.004507425871748977,
+            -0.000601158156253668,
+            0.0016158265715634729,
+            0.01791303234734911,
+            0.0011926917387973374,
+            -0.026393351252156246,
+            0.0006694008039871878,
+            0.001026031515214626,
+            -0.004514419527575117,
+            -0.011320929719992686,
+            -0.02223001723942815,
+            0.016235841442653918,
+            0.014188311619681008,
+            -0.004895218832118094,
+            0.009244697376301004,
+            0.010558796153959677,
+            0.02707485418319703,
+            -0.028380342401606758,
+            0.0040204430418629405,
+            0.007023041225985294,
+            0.01606556581447933,
+            0.017617862031190843,
+            0.0034870847111552862,
+            -0.011253315705053224,
+            0.03238289399766116,
+            -0.010563282687352038,
+            0.0065093536270233385,
+            0.014966802208644462,
+            -0.03541611540945151,
+            -5.821682910348165e-05,
+            0.0061985444938402745,
+            -0.011716201661363997,
+            0.009580035961315467,
+            -0.03120692534097433,
+            -0.0039291243987978845,
+            0.009317468331974374,
+            0.017611312558635673,
+            0.008804996982574765,
+            -0.0034129169537151925,
+            0.010822226698822738,
+            0.012510661458054741,
+            -0.0027147975086409966,
+            -0.008855213644027594,
+            0.01238502686635956,
+            -0.0007687053446954701,
+            0.00450108863711973,
+            -0.00799170686059026,
+            0.0029755595812552293,
+            0.002290650917244691,
+            -0.01554856300008412,
+            -0.011779430686394945,
+            0.00438393126679289,
+            0.0036931891075561957,
+            0.001535662168147833,
+            0.012014408636737167,
+            -0.006087255045485002,
+            -0.0077816220325470226,
+            0.0040573724156794284,
+            0.003136943055301706,
+            -5.505014316363865e-05,
+            -0.030625245024561508,
+            0.01585923084309149,
+            0.0067595435845396405,
+            0.010013308598626031,
+            -0.005315626260228517,
+            -0.00903142414091241,
+            -0.007240920015018893,
+            -0.01228521172101564,
+            -0.010657894223173453,
+            0.007658605592936815,
+            -0.00724924993432376,
+            0.011124174211132038,
+            -0.03128027787357322,
+            -0.013988715435985316,
+            0.01231792993245938,
+            -0.01528161242159377,
+            0.0007675821323979293,
+            -0.008389914312201402,
+            0.0009417242487395995,
+            -0.005119304379761705,
+            0.007887436211143328,
+            -0.03731503169547854,
+            0.007878001990271541,
+            0.02456170736765724,
+            -0.00330924625934981,
+            -0.020259771366167557,
+            -0.009613930457931786,
+            0.0241314385136084,
+            -0.008267831641886344,
+            -0.01645888287387047,
+            0.008421304862524506,
+            0.010140813149093993,
+            -0.0006309118552875393,
+            0.0008237784055549769,
+            -0.003726984191590917,
+            0.01669125388798323,
+            0.02054777616420997,
+            0.041930770964107515,
+            -0.014789610432004559,
+            0.01092349703703152,
+            0.02866938974792608,
+            -0.03378576980722847,
+            0.014728113846952417,
+            0.02518409645155824,
+            0.014590531327650454,
+            -0.0004446956315190408,
+            0.001609777186372437,
+            0.0068939321280085475,
+            -0.014273492115311677,
+            0.009247676157551494,
+            0.013380104440744367,
+            -0.018407746260686045,
+            0.020444430509441124,
+            0.03140071554008003,
+            0.0025839998101985736,
+            0.01355205268903058,
+            -0.006268939830271642,
+            -0.013860692186849113,
+            0.014717364068116991,
+            0.031429975488060896,
+            0.0006919565473572909,
+            -0.018561500446336806,
+            0.019128244642842268,
+            0.006988095806264863,
+            -0.003678529269242034,
+            -0.0021993078588142664,
+            -0.021334919445200388,
+            -0.021003586791369228,
+            0.017304350704792804,
+            0.013508779408333964,
+            0.027794485858413365,
+            -0.010594702782952029,
+            -0.007818089330662161,
+            0.014293688127901867,
+            -0.022368568887004094,
+            0.015426482139881204,
+            -0.02160820059598751,
+            -0.00571499813703566,
+            -0.0077402210272634235,
+            0.03274360648629609,
+            0.001441304806200403,
+            0.02734331677589297,
+            0.016457027764064002,
+            -0.033609420134045206,
+            -0.029790535553657092,
+            -0.00712106584238484,
+            -0.02307933297713104,
+            0.010356561171963577,
+            -0.03901183838032092,
+            -0.00071735150189382,
+            0.008251646656432826,
+            -0.008079917288471352,
+            -0.0007087526632966007,
+            -0.006419191399519291,
+            0.013989344185652506,
+            0.013661633160336152,
+            -0.0228103143826663,
+            0.023644639659577426,
+            0.008343481940133684,
+            0.025998483906975782,
+            -0.02076055376006461,
+            0.019148568238437927,
+            -0.017705363646416224,
+            -0.02777313933338929,
+            0.014358130220942483,
+            0.023585731219788467,
+            0.01635364038245971,
+            -0.007122853965024528,
+            0.007110003583832299,
+            -0.03556761320536796,
+            0.027585457849964405,
+            -0.013064062623866305,
+            -0.012504161289952458,
+            -0.0032379276562094555,
+            -0.011931827532360686,
+            -0.002574161005800279,
+            0.0008533907465694845,
+            0.0034345232246102806,
+            -0.03131462673429805,
+            -0.0002450959360520896,
+            -0.03727416416766574,
+            -0.028944814604815072,
+            0.01360126708805471,
+            -0.00036361284498363273,
+            -0.0049097813612814616,
+            0.02933242705521636,
+            -0.026339136251046315,
+            -0.02693295650849956,
+            -0.009590247544093352,
+            0.02853096456887074,
+            -0.0220673798697511,
+            -0.004519050393279848,
+            0.0015034014871316069,
+            -0.01828226786700525,
+            0.004436383319946364,
+            -0.0012748481042235446,
+            -0.010416065158876917,
+            -0.01105146139270597,
+            -0.03229479099927737,
+            -0.005822133182096844,
+            0.02106215163620738,
+            -0.006957125617688484,
+            0.002746241228040847,
+            0.013312248331456934,
+            -0.010968629118224582,
+            -0.0029946900704643812,
+            0.0017875284441794235,
+            0.023304175718480834,
+            -0.021033941064173063,
+            0.007714334541806199,
+            0.005955442031111047,
+            0.0016405586860131207,
+            0.018406562522853148,
+            -0.0054922197063461825,
+            0.0037690571959124617,
+            0.0162126534419016,
+            -0.009969304596173302,
+            0.017959307638726878,
+            0.007691549802475525,
+            0.006681837561745192,
+            0.002775700618222622,
+            0.007712770283579111,
+            0.009752652973700739,
+            -0.013506020126078112,
+            0.016911141425375933,
+            0.022468715726349706,
+            0.00023233555209372766,
+            0.021066580799163234,
+            -0.02302516292580905,
+            -0.028618544628453015,
+            0.018297878810036087,
+            -0.0225078941299981,
+            -0.012735208680311652,
+            0.016459925496937065,
+            -0.0043032724578332855,
+            -0.00797609922711709,
+            -0.014986057073351055,
+            0.02260618561608301,
+            0.02132930921464508,
+            -0.020249788476562527,
+            -0.012385936327068884,
+            0.0072304431443959505,
+            -0.004373475447051833,
+            0.007086656081865733,
+            0.03195810137269375,
+            -0.015982210897392764,
+            0.006818890093901413,
+            -0.0038306138017521626,
+            0.0030753642449792215,
+            0.017830546304667783,
+            0.009470018737705222,
+            -0.0019815241970236146,
+            0.030100238416495066,
+            -0.001330614013444443,
+            0.016392226282459177,
+            0.0004888028477871524,
+            -0.010948833692356176,
+            0.006343890495626752,
+            -0.026652806306173643,
+            -0.008627570055007363,
+            -0.001203530223345183,
+            -0.009581494451831351,
+            0.07598616930784373,
+            0.003930153552069478,
+            -0.01295957554290434,
+            -0.001673007115291578,
+            -0.001368287102181597,
+            -0.006897179139643672,
+            -0.00960679451603541,
+            0.0287166325268306,
+            -0.016613613331623346,
+            -0.01894527592131826,
+            0.0008370014132002703,
+            -0.006785618888421608,
+            0.00919924251249592,
+            0.01816363936999796,
+            -0.010225961928154075,
+            -0.01604759761423727,
+            0.001577138242530474,
+            -0.003520564936698765,
+            -0.004395434431595494,
+            0.0052399043934902014,
+            -0.010127122101423486,
+            0.006456662777583096,
+            0.026560491731327886,
+            -0.013684536319659296,
+            -0.01425196172663986,
+            0.004084982328707292,
+            -0.0010682084059972982,
+            0.009758641913073978,
+            -0.00864879062601467,
+            -0.00481222050200678,
+            -0.014375330954464725,
+            -0.011378890549403524,
+            0.02505641780042516,
+            -0.02861164504722546,
+            0.0034060847953066354,
+            0.014937227531980433,
+            -0.00016904224334342915,
+            0.01541381851648916,
+            0.0033028855218525644,
+            -0.03615935765718852,
+            -0.013677774380513626,
+            -0.02073839691250734,
+            0.019821074228011413,
+            0.01688422012576042,
+            -0.0033731840788147442,
+            -0.013840993586441808,
+            -0.026149880178874887,
+            -0.007430563876740068,
+            -0.02466234818567362,
+            -0.020708282040996014,
+            -0.019276991448222817,
+            0.006137002799644838,
+            0.005162892426268572,
+            0.022881759407670583,
+            0.0025854880108157857,
+            -0.015058889448404231,
+            0.0029948608544278386,
+            -0.006722836436935071,
+            0.011845298212977357,
+            -0.02747944486818378,
+            0.015311272959213262,
+            -0.01610493464983465,
+            0.025239768880017402,
+            0.007425208902253236,
+            0.0010569008485112994,
+            -0.008822681490988575,
+            0.00010704167529046774,
+            0.014883787023445807,
+            -0.004186968199973372,
+            -0.0007781259002896265,
+            0.02787555889813697,
+            0.02087373813833492,
+            -0.0017783425135865238,
+            0.008743718448136432,
+            0.028026471206552342,
+            0.019081083283793984,
+            0.0325261942213039,
+            -0.013815526479760766,
+            0.005113676604826083,
+            -0.028873600114081546,
+            0.010510475843707614,
+            -0.010871039729515013,
+            -0.010201824683340651,
+            -0.0067946599549165735,
+            0.027756887504389347,
+            -0.007962601205446472,
+            -0.0006687976124826922,
+            0.03469894503563926,
+            -0.014185294847328063,
+            -0.00010543064621374546,
+            0.0007386418834840544,
+            -0.009759481309152999,
+            0.02742722730386853,
+            -0.0073590367291510046,
+            0.018456032721630843,
+            -0.019560882612141943,
+            -0.013705917411254969,
+            -0.011894560208199602,
+            -0.001142768182039817,
+            -0.02576754877141608,
+            -0.008755240066826174,
+            -0.02249287216087102,
+            0.00859172262856532,
+            -0.0046012388553094375,
+            0.018845920253446127,
+            0.0011617844393195566,
+            -0.005275799428018491,
+            0.029069367036565145,
+            0.0088080893110627,
+            0.018100411389202663,
+            -0.0004971948011851863,
+            0.01216566778850985,
+            0.02369448391321832,
+            0.002637001729688794,
+            0.005012948614448268,
+            -0.024213704079796478,
+            0.003420184574622472,
+            -0.0004359900462023086,
+            8.265488264269535e-05,
+            0.016267948861008616,
+            -0.019572032453100303,
+            -0.02216264534756073,
+            0.021871880626992544,
+            -0.012742926070754964,
+            0.011676769226089007,
+            0.0035362870996336194,
+            -0.010732229460038356,
+            0.0006838777488770268,
+            -0.03083438636386314,
+            0.008820746379709355,
+            -0.03216685858285627,
+            -0.032230808828686675,
+            -0.013547681710236494,
+            -0.009994106571421729,
+            0.0008758088338659712,
+            0.028447187234357855,
+            -0.00795317029726568,
+            -0.015797892954136448,
+            0.005363565557248085,
+            -0.011199252697624111,
+            -0.00817162221789202,
+            -0.0007867892785855576,
+            0.005856939999700809,
+            0.009461959355310273,
+            0.005997804620690314,
+            0.011648335019189696,
+            0.007491946658046298,
+            -0.008028800679169293,
+            -0.008837725220736191,
+            0.015889923814605387,
+            -0.0014409716087425222,
+            -0.01176424391171685,
+            0.030430130082762363,
+            -0.013300620185884817,
+            -0.0005592926632214984,
+            -0.0021975669766032472,
+            -0.0015687208316663597,
+            -0.0021652076181411244,
+            -0.014225289143358926,
+            0.01870568404189425,
+            8.334501002782455e-05,
+            -0.014288806218067678,
+            0.0032841743905975634,
+            0.003841780428153241,
+            -0.0036498339483411074,
+            0.008388922960269383,
+            -0.008147275785908849,
+            0.017348147498892,
+            -0.015113395971380741,
+            0.008740452410210708,
+            0.0038335161236289234,
+            0.0043482078469644385,
+            -0.0065755348640655865,
+            -0.01592809184384243,
+            0.009642593996701513,
+            0.012051872087663658,
+            0.02173264012652684,
+            -0.02525660485859878,
+            0.00876161653901566,
+            -0.005374533920781607,
+            -0.014534179652013885,
+            -0.016127866637369738,
+            0.00982303248417706,
+            0.022322910435388785,
+            0.0063086346898091296,
+            0.012139263756190026,
+            0.00013383660542132047,
+            0.005848259030010402,
+            -0.014767198851314148,
+            0.010570158368602513,
+            -0.011355774405066537,
+            -0.018439261453084426,
+            -0.005243587266408389,
+            0.028894120979627323,
+            -0.005492024421080143,
+            0.017938747701972086,
+            0.0021350711250093858,
+            0.029007356490801176,
+            -0.008686781541806358,
+            0.0014883465513564527,
+            0.0039866022469891155,
+            -0.01295271525889559,
+            -0.004272008629260297,
+            0.01976273843646234,
+            0.04882868638601926,
+            0.010204475902027572,
+            -0.0009155109545661691,
+            0.010069071220988797,
+            0.005626418787776309,
+            0.002148101019776153,
+            -0.00805781810841286,
+            -0.030431071067211577,
+            0.008883707378543252,
+            0.005628218215212441,
+            -0.01997825125961566,
+            0.019838234890736068,
+            0.026483365072033395,
+            0.023040623182975964,
+            -0.011585643598171132,
+            0.016539930468533123,
+            -0.010092529089936188,
+            0.0177008735686489,
+            0.028665424114962996,
+            0.013810061902648515,
+            -0.010487150104822674,
+            -0.0020847870330548534,
+            0.006329342599564054,
+            0.004186981876004016,
+            0.00026239408223817384,
+            0.015190614519766204,
+            -0.019155180136775043,
+            0.01005728016778505,
+            0.014725826697964637,
+            0.03322436221319799,
+            8.049930133627564e-05,
+            -0.01645191835557271,
+            0.0016249281996128196,
+            -0.009976155662939693,
+            0.00856399492362911,
+            0.000632427127231252,
+            0.0032242260692672124,
+            -0.009688119824276678,
+            -0.012682779503915694,
+            -0.002851918183031152,
+            0.009403205508474241,
+            0.0021188716361101698,
+            0.002759004011231831,
+            0.007641925150177999,
+            0.0022830062031146204,
+            0.01713024474384216,
+            -0.0011181980841546123,
+            -0.00029141889484139217,
+            0.004601090256674874,
+            0.010509199590505558,
+            -0.01649852835819883,
+            -0.017548369964263514,
+            0.004337237475053179,
+            0.012217494210546133,
+            -0.013159179948224596,
+            -0.013241981632059755,
+            0.015704543608959117,
+            -0.01913575130892795,
+            0.024934102868065366,
+            -0.00942381892138331,
+            -0.004543611336526587,
+            0.000343200128934031,
+            -0.00727538880611703,
+            -0.0038751118497522673,
+            -0.0030064279504724797,
+            0.003984715814505561,
+            0.0019261818110367147,
+            0.007472350699291607,
+            0.008472369788801792,
+            0.014568547873264247,
+            0.006043895352236181,
+            0.0032580830408778395,
+            0.009109104498825864,
+            -0.014523846294121576,
+            -0.014800649887278973,
+            -0.002704791598011488,
+            -0.008895394038700809,
+            0.00902016867266151,
+            -0.006607182154874112,
+            0.00845408382182461,
+            -0.015501867835442603,
+            0.015573642225745032,
+            -0.012200552219016517,
+            -0.0008900215712478553,
+            -0.009666233545144905,
+            -2.7470277092642562e-05,
+            0.008191697904514857,
+            0.014605287653054787,
+            0.0070552942843382215,
+            -0.031921405225964206,
+            0.0017403427534633655,
+            -0.008846568352320488,
+            0.03446319360336241,
+            0.002612606662691094,
+            -0.01060863123202008,
+            0.020352181356568008,
+            -0.01592124132155158,
+            -0.00027704668715660693,
+            -0.01602075523652062,
+            -0.004797557413047212,
+            0.00707939446796679,
+            0.016399017978990946,
+            0.02759049512664933,
+            -0.005697020356865688,
+            -0.007995305872076221,
+            0.006384047872862793,
+            0.029590696670334232,
+            0.012246955508890425,
+            0.024748351124415785,
+            -0.007633712004672087,
+            0.014348228716100414,
+            0.007773260552660029,
+            0.0076045488653348975,
+            0.004321689250440021,
+            -0.002591801797145718,
+            -0.011698574606499644,
+            -0.014974971814360001,
+            -0.0037563183330623384,
+            -0.016987915346053498,
+            0.00012247833530362867,
+            0.021344400276501117,
+            -0.00475351423959752,
+            -0.0033847161875730026,
+            -0.01874186437694691,
+            0.0033461670218611346,
+            -0.010029445302549983,
+            -0.00367421246775698,
+            -0.013133469163793961,
+            -0.0254122035745965,
+            -0.018556369080030412,
+            -0.002155244024999317,
+            -0.00929309186646207,
+            -0.015194269781263309,
+            0.0070354690219415535,
+            -0.016305117309271634,
+            0.0080837166473439,
+            -0.00448473417024141,
+            -0.016508775226236883,
+            -0.00958925666995396,
+            0.01100335153658195,
+            -0.0041819993634961685,
+            -0.01929493648733792,
+            -0.01282300765645703,
+            -0.011300260564838231,
+            0.0051733704121938884,
+            0.005996780699975213,
+            -0.0038111648304178544,
+            -0.0028279639348479987,
+            0.016640121845366295,
+            -0.010197365811240557,
+            0.009878730312873492,
+            -0.03335474019383245,
+            -0.03551826040406959,
+            0.004497847402931096,
+            0.01507007013703286,
+            0.0013085881889541252,
+            -0.02078680101065954,
+            0.010806315988966003,
+            -0.016499186313125556,
+            -0.00858013792516617,
+            -0.008522534127498645,
+            0.009649521088411411,
+            -0.007283003380127847,
+            0.01299542385123007,
+            -5.3226845309105124e-05,
+            0.030477683597373687,
+            0.0074153455615142755,
+            0.004161881195438522,
+            -0.00679081048132963,
+            0.008354912600159931,
+            -0.001621776416108105,
+            0.004080191054849395,
+            0.02120524874226696,
+            0.012440848502230418,
+            -0.00218041171518331,
+            0.006458415098595872,
+            0.005678055538564355,
+            -0.017186537377121235,
+            0.02507960402044456,
+            0.024678178106806115,
+            -0.008205288425409454,
+            0.004108402466376321,
+            0.005783335302169668,
+            -0.008846663924040241,
+            -0.006897707231869481,
+            0.006527795262026734,
+            -0.018474181903007085,
+            -0.0008721381678008483,
+            0.010598084358464908,
+            -0.02255441909004843,
+            -0.004840511019512733,
+            0.021745651362109403,
+            0.004631567631335321,
+            -0.018558972627958834,
+            0.02764788406651861,
+            -0.004942361238163609,
+            -0.0027290999934532494,
+            -0.010876135202650048,
+            0.01671497143541799,
+            -0.012173527207375091,
+            0.008443317346655462,
+            0.011021541861818043,
+            -0.006295639322287843,
+            0.003083685676540999,
+            0.008277206401503182,
+            -0.004504302913715969,
+            -0.0068075424927125676,
+            -0.0010508416962215848,
+            0.010167160518175114,
+            0.0036980815941666336,
+            -0.022704963865525812,
+            -0.006020073261002279,
+            0.02934227992246887,
+            -0.015082922782242507,
+            -0.0050916398557337705,
+            0.005376294716056329,
+            0.007709342013963906,
+            -0.003476236561444464,
+            -0.0023399991233097337,
+            -0.00787486645740744,
+            0.010356414979428607,
+            -0.015950083845400274,
+            0.02076360898078935,
+            0.025608263494550438,
+            0.024504236990840842,
+            -0.014814235990238182,
+            0.010828584133720797,
+            -0.011703992394890898,
+            -0.011047470905668532,
+            -0.015542835222173864,
+            -0.001658296303013357,
+            0.001970208852236451,
+            0.013107754130616248,
+            0.009666887546609378,
+            0.0235853683657408,
+            0.006473544089109194,
+            0.008973300312205858,
+            0.014118601565410305,
+            -0.003902202854013708,
+            -0.018303438244719996,
+            -0.014584226585460755,
+            -0.011977613285028946,
+            -0.01756345491020528,
+            0.0004563602312336091,
+            0.020461659996271687,
+            0.012417744207683839,
+            0.024136626592375227,
+            0.002119213701333967,
+            0.013759902247467922,
+            -2.9525392347047554e-05,
+            -0.000892319792935443,
+            0.00045783825646242434,
+            -0.005238483120312843,
+            0.019255756091120846,
+            -0.004599752813817219,
+            0.01656589522110913,
+            0.0006443773300684925,
+            0.007133940816771615,
+            0.013554761340256566,
+            -0.01704388669741371,
+            -0.003443100892108368,
+            -0.010029620309829285,
+            0.000941391966877521,
+            -0.019633111750884034,
+            0.01934571074127927,
+            0.010273330977233852,
+            0.0017121345993533963,
+            -0.018327189396014198,
+            -0.0029706385099949693,
+            0.03448635131998772,
+            0.019458999453011265,
+            0.0050734354363614525,
+            0.02632443373898792,
+            0.013018762352749236,
+            0.014443611411502168,
+            0.010153370950346075,
+            0.02356766253713842,
+            -0.023479811300626968,
+            0.02095255383200973,
+            -0.011590347850340786,
+            -0.007414136601134069,
+            -0.0035774888741646705,
+            0.0025141508183687115,
+            -0.013588691733097813,
+            0.00982877622088977,
+            -0.0017916539399655826,
+            -0.025386211797192578,
+            0.009164316443206016,
+            -0.008811052979979271,
+            0.010464442616928296,
+            -0.007774569252594923,
+            -0.0002806136360286729,
+            -0.0127084026190163,
+            -0.015184317235990752,
+            0.001259201880504347,
+            -0.009158429132281081,
+            0.023908150915412617,
+            -0.021210848953875233,
+            -0.01321384014436847,
+            0.013997502713857512,
+            -0.006869443694669701,
+            -0.011364239609894453,
+            -0.00374449980963666,
+            0.023673050288879224,
+            -0.009243341446512421,
+            -0.01359367519009039,
+            0.014464166953530989,
+            0.003834735113400469,
+            0.0059713317014342395,
+            -0.007508862694272452,
+            -0.008428705474135382,
+            -0.003280788036976103,
+            0.005998840043285335,
+            0.026615077617284676,
+            -0.014059930972065399,
+            -0.0032833264512892022,
+            -0.009189999799498163,
+            0.004118204537632102,
+            -0.02359889635281631,
+            0.02771075351384059,
+            -0.008871395303200375,
+            0.021705177758560575,
+            -0.0015924449308442096,
+            -0.006239188158271628,
+            -0.024231036074349822,
+            0.027522945599928003,
+            0.003683870966132199,
+            -0.010221180899173243,
+            0.0012768725460662862,
+            -0.052005292736885915,
+            0.016636109796947025,
+            0.008429895913163985,
+            -0.019507994429834353,
+            0.003697519981610208,
+            -0.004773584856005853,
+            0.018983847580558853,
+            -0.022966993228850697,
+            -0.014078986007109447,
+            0.008283766047449966,
+            -0.01153285124362746,
+            0.0031439180465649996,
+            -0.02527749501821793,
+            -0.005754537200570427,
+            0.0011760345167229269,
+            -0.005276936074280004,
+            -0.001705933750221904,
+            -0.020022761608132548,
+            -0.025972227850795115,
+            0.017666017379157135,
+            0.015820875579600418,
+            -0.002278262448951789,
+            0.01320818011561616,
+            0.008709899932124187,
+            0.00836573261091119,
+            0.016034263394940008,
+            0.007124509234103267,
+            0.0081473711608784,
+            0.045498533239850446,
+            -0.019846323044145485,
+            0.007371629414298193,
+            -0.0014854149174929534,
+            0.013951741898629716,
+            -0.030467924817555014,
+            -0.0055144480998344705,
+            0.0062802744565489945,
+            -0.0049373689340857915,
+            0.03510454786147091,
+            -0.017326094216003073,
+            0.01694160526735264,
+            -0.0026075809870445238,
+            0.0007420394265014909,
+            -0.02042003877666893,
+            -0.026156937726134134,
+            -0.014294938857971834,
+            0.0011857406889443987,
+            -0.0010047941261608975,
+            -0.011946583993069242,
+            0.006797001710574735,
+            0.004917161312925866,
+            -0.0020713379451245515,
+            0.015652523103715316,
+            -0.02952061566240878,
+            0.007356800041369564,
+            -0.010733578620243498,
+            0.020008504830359524,
+            -0.00990331048284697,
+            -0.0013319400098308093,
+            0.007236960885181047,
+            -0.004122868383988748,
+            0.013216043472071927,
+            -0.0038337251451538677,
+            0.004227115653354611,
+            -0.005221321656766545,
+            -0.01794343569722877,
+            -0.0016435261495179767,
+            0.021429008432348098,
+            -0.008349684096540855,
+            0.031864497493526205,
+            -0.01264743878037828,
+            0.009174982784686633,
+            0.010692990337462762,
+            0.0006628831036334152,
+            -0.005286285851702483,
+            0.009052971814211658,
+            0.0021431970274545893,
+            -0.008690520448278274,
+            0.006934310986093342,
+            -0.018349449937310143,
+            0.008494451785507706,
+            0.03394632739506833,
+            0.0018511328780472266,
+            -0.0007519661670070526,
+            -0.007266663179613392,
+            -0.01335844201823745,
+            0.018540713029383066,
+            0.005488273906733518,
+            -0.002546531152114063,
+            -0.027536660869590543,
+            0.02007746621157705,
+            0.0012265798861291223,
+            -0.004755325069578531,
+            0.012466557901411086,
+            0.0020495867632654885,
+            0.019989531151046736,
+            -0.010171678639113316,
+            0.030757957899447096,
+            0.010050036220124393,
+            0.002959833051227505,
+            0.0038556928367700113,
+            -0.036655749074851536,
+            -0.01652152778069314,
+            -0.00019357502472740266,
+            0.014271581054865031,
+            -0.012430647252826173,
+            -0.022432580814844155,
+            0.0010346169306977334,
+            0.01148004051449691,
+            -0.0033898642762881046,
+            -0.009650665305271706,
+            0.01560757541276278,
+            -0.020342213902670575,
+            0.006701302394408393,
+            0.007693672045981921,
+            0.0287100214318104,
+            -0.0020735384554022153,
+            -0.0012324838849990056,
+            0.000626416803904792,
+            -0.0055669860801192935,
+            -0.0008037366126870824,
+            0.017251369734470945,
+            -0.013661316947002466,
+            0.0175926211217534,
+            -0.012735241490550407,
+            0.0064713656749838585,
+            -0.0021092378432037563,
+            -0.010629579860050812,
+            -0.020114807971202787,
+            0.0029609495764307357,
+            0.009378003163375899,
+            0.006920144698428806,
+            -0.010438984881449953,
+            0.010715409812532968,
+            0.012197673689706352,
+            -0.00753962861484439,
+            0.014519905132928041,
+            0.022173989895384192,
+            -0.007687503607783572,
+            0.013633321414009471,
+            -0.0041446334728745095,
+            -0.014056264816568229,
+            -0.009985463824440833,
+            0.023929750617160402,
+            -0.0005563806942667597,
+            0.008756997373618809,
+            0.00388125051641897,
+            0.0007856447663040696,
+            -0.011226530517316183,
+            -0.001653447552601681,
+            0.019294448360333633,
+            0.011831934675073843,
+            -0.0013888534822008192,
+            -0.0010525764356699885,
+            -0.0014566936525653407,
+            -0.009427557739483988,
+            0.02049188741459728,
+            -0.022030783927622997,
+            -0.0019604940727076573,
+            0.00461018148439991,
+            -0.022589816803875618,
+            0.0001863117949313752,
+            -0.024570217962989905,
+            -0.01569952241187498,
+            0.011763033781122343,
+            -0.0022530631823737554,
+            -0.016028810867854907,
+            -0.0003242701040942633,
+            7.424127301806556e-05,
+            -0.007815789940508527,
+            -0.004135500768181258,
+            0.006115909992703774,
+            -0.00034193217510010117,
+            -0.0032262189374874842,
+            0.013430154310506884,
+            -0.009996622196870008,
+            0.015431528911575157,
+            0.0067029980830614145,
+            -0.015702167721800483,
+            0.0021406610839557003,
+            0.018500219544264753,
+            -0.0058910087864916085,
+            0.00787021425357441,
+            -0.014681907249972176,
+            -0.0037495551894426303,
+            -0.005787661351944086,
+            0.006051351977821761,
+            0.024849195780885833,
+            0.012969434450643372,
+            0.01139340445077682,
+            0.017405361843075933,
+            0.004325000715054734,
+            -0.01572297405297993,
+            0.0007416798945675923,
+            0.01682727189828363,
+            0.010464036837245312,
+            0.009804326468113703,
+            0.0008975556667684365,
+            0.0043704860361487635,
+            -0.005142176468800175,
+            0.009317064072855551,
+            0.02915482449687158,
+            0.0013434689612997423,
+            0.008696141010760192,
+            0.017276730345147134,
+            0.027038741396131265,
+            -0.0035088997194497725,
+            -0.007469614319788499,
+            0.008739942310300076,
+            -0.009657991604173173,
+            -0.001081629174976152,
+            -0.017000351183542026,
+            -0.012310275736750868,
+            0.016510505259358398,
+            -0.006084627062230492,
+            0.02848913486719956,
+            -0.0010655806791907225,
+            -0.02026498848361482,
+            -0.01565505608194376,
+            0.001404549479440637,
+            -0.01362132895080652,
+            0.005287566256051299,
+            0.014187265685427415,
+            0.01917046513020831,
+            0.004687191682875024,
+            -0.022258337721085656,
+            0.037364962197399806,
+            0.01757108963349427,
+            0.019377870795760123,
+            -0.019422840506328382,
+            -0.006159311237412833,
+            -0.009077095160676276,
+            0.02116697012944498,
+            -0.00880692095488465,
+            -0.009875674856043954,
+            0.024014332004451508,
+            0.017812606845656484,
+            0.005679806286464216,
+            0.006736609904079586,
+            0.0020048663439319585,
+            0.0025414630253661444,
+            -0.015831593851012047,
+            0.017957902498542212,
+            -0.005858125698347523,
+            -0.0058068554681761945,
+            0.017529514573571298,
+            -0.005989581792993925,
+            -0.0030306021203732347,
+            0.00015839997806822045,
+            -0.019697052145531482,
+            0.011618112886207201,
+            0.008256007601576028,
+            -0.017229774429089857,
+            -0.0014757560402567304,
+            -0.0002404480326945295,
+            -0.014118694740134205,
+            0.02004885146995733,
+            0.01694893356262835,
+            -0.0013248884139405398,
+            -0.0051752815372013315,
+            0.008223868147393114,
+            0.007000679132597393,
+            0.005256194890343006,
+            -0.019496098876977206,
+            0.005230597149342019,
+            0.005103909294824105,
+            -0.0014716649591412774,
+            0.010274346323051634,
+            -0.009030594825985526,
+            0.033740380586893826,
+            -0.006505788197604522,
+            0.013065658941136721,
+            -0.0043634731684193745,
+            -0.0055529072007985125,
+            0.001816020676232658,
+            0.013054468478598695,
+            0.009753501030700457,
+            -0.00970446155762939,
+            -0.004172354962129865,
+            0.018697982092775737,
+            -0.011983156146516074,
+            0.0013300791598046338,
+            0.01564072865804035,
+            -0.004858814041438713,
+            0.004708235083775013,
+            0.008635204790345423,
+            -0.004489505237783172,
+            0.01801512858970922,
+            -0.016482572999490366,
+            0.0023953436660054026,
+            0.010694869684962088,
+            0.02390864159830096,
+            -0.019152187046410703,
+            0.021786596486708207,
+            -0.017951050280323393,
+            0.026538582634436314,
+            0.014484894489694262,
+            -0.0035222259457311444,
+            -0.01946200885147006,
+            -0.00850690913394181,
+            0.011654565129063417,
+            -0.021423803876241532,
+            -0.0007985948078290641,
+            -0.0007290222944686444,
+            -0.021367628543092133,
+            -0.011189364017231132,
+            -0.009961779129778331,
+            0.010701637745810138,
+            0.001918633898887013,
+            0.010380983868185359,
+            0.008002182827196827,
+            0.010784760574518087,
+            0.014405640442760822,
+            0.0008958107769384412,
+            -0.010743001445440941,
+            -0.023795589003655273,
+            0.016987708698679597,
+            0.00438463274382093,
+            0.0007862212695139549,
+            -0.01627468994515667,
+            0.011936651269737248,
+            0.013250643815768723,
+            -0.009385605548926286,
+            0.009121882469425302,
+            -0.03716901103204463,
+            -0.004552808410398574,
+            -0.0035704811925430366,
+            -0.0020309197365173475,
+            -0.00908678158053104,
+            0.01559445415882485,
+            -0.013682507836005851,
+            -0.0009590922904844632,
+            0.00011696985057536548,
+            -0.0074756526903548915,
+            -1.5467890432254378e-05,
+            0.020649928837031005,
+            0.008438785939908428,
+            -0.0037382105257999757,
+            0.010376213224654074,
+            0.008826031303751312,
+            -0.013900359449287676,
+            -0.01527474712786084,
+            0.019035366035884273,
+            -0.018776918681908793,
+            0.0063098418544296165,
+            0.01677916431403883,
+            0.020778813096283988,
+            -0.02891862501830075,
+            -0.0007231469961304435,
+            0.022093939332021185,
+            -0.0012807635799950273,
+            0.006370841109790851,
+            0.008781853138377092,
+            0.006754082980602544,
+            0.0029102159062355374,
+            -0.003711173531879644,
+            -4.843770941622283e-06,
+            -0.003940474588075707,
+            0.007155539022862352,
+            0.004366433868349658,
+            0.036453728071874844,
+            -0.013661129348165794,
+            0.002541308805735001,
+            -0.0058838191638835155,
+            -0.00347401304572066,
+            -0.00593469688967835,
+            0.004407665856950772,
+            -0.010055727994798024,
+            0.004330827357536078,
+            -0.001570861535288093,
+            -0.0003731129422346693,
+            0.004883969598300302,
+            -0.001701673379646408,
+            0.007036892564844683,
+            -0.011863650867472625,
+            -0.0003979727750011258,
+            -0.020174851412609312,
+            0.021117284557875267,
+            0.00011666551282792189,
+            0.003662462447326642,
+            0.03723741246290771,
+            0.00476064510662738,
+            -0.00238355131307445,
+            -0.012880057949435798,
+            -0.013551338766321525,
+            -0.012986444917747435,
+            -0.020703821629987552,
+            -0.0036665673548525388,
+            -0.021006311026247164,
+            -0.0052061406234528915,
+            0.010362862594785442,
+            -0.011540184420787572,
+            -0.004023659825622843,
+            -0.015881589631889473,
+            -0.006531981517815174,
+            -0.03114211633687585,
+            -0.01124607488650473,
+            0.007493228366452009,
+            -0.019149285949049886,
+            -0.0039932870926865805,
+            0.026850121590606017,
+            0.001095360635760988,
+            0.01387696424607248,
+            0.0038058248676879625,
+            0.008117686785954085,
+            -0.0045937472017383955,
+            -0.0005669025036141417,
+            -0.01928244877716143,
+            0.003391980330529026,
+            0.008525354854576925,
+            -0.00361507909530282,
+            0.0070008404454106905,
+            -0.008835727374865623,
+            -0.016950017602980382,
+            0.0021654956715219517,
+            0.009334379529385142,
+            -0.006639140003365139,
+            -0.017587472740268683,
+            -0.011049929670847685,
+            0.010401189523318744,
+            -0.02283429470064187,
+            0.006257145380266984,
+            0.016439581169238607,
+            -0.009363030375624585,
+            0.004651575425738153,
+            0.019809183749755425,
+            -0.011810674576781702,
+            -0.005811328501528397,
+            0.008471992070998505,
+            0.013891259999721588,
+            0.0095606134736748,
+            0.02954498207274139,
+            0.0026596809804008996,
+            0.000561870602466584,
+            0.004889263753490895,
+            -0.004277061142504552,
+            -0.017423200644235665,
+            -0.006243787897080996,
+            0.0038507494787521856,
+            0.012873996671952184,
+            -0.009879417960879032,
+            -0.00048575277472139055,
+            0.01682604801673206,
+            0.004530939990187663,
+            -0.0033901248079550284,
+            -0.011868399508738274,
+            -0.020080331108085103,
+            0.0015580259874809898,
+            -0.01362200460317948,
+            0.015354347040143093,
+            0.003390601069331352,
+            0.0015284610287850575,
+            -0.004239880336806429,
+            0.01828645373058079,
+            -0.005013418527935815,
+            0.012749833447770729,
+            -0.006148890994661865,
+            -0.004587831231448011,
+            0.007986533783001116,
+            0.009166578151006778,
+            0.0004939857995863664,
+            0.05421019830078428,
+            0.002838056604877719,
+            0.01333205648597435,
+            0.013419127695838135,
+            -0.014523733369484934,
+            0.014086525147734823,
+            0.012688569853552613,
+            0.018497344547821693,
+            0.013037819487160016,
+            -0.012353776077273462,
+            0.017354801119939964,
+            0.0024084723134891487,
+            0.003909681065813095,
+            0.0022202438064449653,
+            -0.004528576079628071,
+            0.019594683950225115,
+            0.004070222328547378,
+            -0.0074362443729372105,
+            0.004453618452623641,
+            0.005007543168224267,
+            -0.013105524222311019,
+            -0.0112958915514782,
+            0.027105208848863218,
+            -0.01219417557128091,
+            -0.018749971743148304,
+            0.006706594269933445,
+            -0.02008458683853808,
+            -0.03763686535076864,
+            -0.0021332074289244973,
+            0.009992361098878196,
+            0.002795183524495193,
+            -0.00253695604043486,
+            0.01614298064601032,
+            0.016730443047719683,
+            -0.016985296937992726,
+            -0.018245902728910347,
+            0.0025257780765481825,
+            0.0016333325293570182,
+            0.01861807236979066,
+            -0.023168663546030196,
+            -0.006378864850313035,
+            -1.3795465857380962e-05,
+            -0.006829008132963306,
+            0.004850306884033312,
+            0.004703280112191218,
+            -0.0071714759384722965,
+            -0.0027773519096287217,
+            0.002256846645341137,
+            0.01975383140755998,
+            0.013198754584066554,
+            0.01572971337890945,
+            0.002600741959577911,
+            -0.005033508062449482,
+            -0.016817359058724685,
+            -0.005450011991586895,
+            -0.013478330855299777,
+            -0.009091854345715177,
+            -0.003042052092498012,
+            0.008217376329142566,
+            -0.025053469059115735,
+            -0.011965083373220982,
+            0.0119294363235832,
+            0.00391377331459168,
+            0.0003099203772539202,
+            0.007607672728136279,
+            -0.003830132495364688,
+            0.010331053246451333,
+            0.013823890473448383,
+            0.0076112887952959766,
+            -0.004723377809559611,
+            0.009600411526173948,
+            0.011460781436250457,
+            0.006015477068316837,
+            -0.005072711638895421,
+            0.023859899048048694,
+            0.01821920335673539,
+            -0.01680365156728093,
+            -0.002149107383573353,
+            -0.0027829676356423997,
+            -0.01884308738652242,
+            0.009665830801645276,
+            0.00880322398336952,
+            0.008308268605449346,
+            0.003906369351868086,
+            0.00974531700889854,
+            -0.013875904819275925,
+            -0.0050570811397453235,
+            0.002870120865445989,
+            -0.005729556771404781,
+            -0.01901337075686543,
+            0.000297242372701423,
+            -0.011987192261104132,
+            -0.011327157550660895,
+            -0.01409710922165158,
+            -0.003221240464411861,
+            0.005283472511900512,
+            0.021472187640986972,
+            -0.004410881709548286,
+            -0.001855852749039012,
+            0.01351697599517034,
+            0.010060125186506398,
+            0.0021658738823899905,
+            0.0002206474541239167,
+            0.005418828324605768,
+            -0.0027919130944783397,
+            0.0008094042430809068,
+            -0.0015826797905407809,
+            0.021089276765434876,
+            -0.02016611086095107,
+            -0.009413977569213423,
+            -0.009288078098692112,
+            0.005842197094339927,
+            0.028793032462603636,
+            -0.007165634502373454,
+            -0.010104002083017078,
+            -0.01994315293556678,
+            0.0003786527922888834,
+            -0.07810609175343784,
+            -0.005210636756258548,
+            0.001870513618910238,
+            0.03126380113105983,
+            -0.010817888476436834,
+            0.0033673768058629823,
+            -0.002204487854177595,
+            0.014542299655712928,
+            -0.001760107965206874,
+            -0.0013283746982194259,
+            0.0071954806737858056,
+            -0.005640400818010697,
+            -0.006210989097047217,
+            0.005904163661619133,
+            -9.945941028848348e-05,
+            -0.010266938705916303,
+            0.00057263080955709,
+            0.02209498395848068,
+            0.0159398346608872,
+            -0.006575439759804175,
+            0.01816748552021211,
+            -0.00719294553484007,
+            -0.009838738592324234,
+            0.005639288780502127,
+            -0.0019007683459126327,
+            -0.015369619176147573,
+            -0.011475977270803784,
+            0.009862808084641291,
+            -0.030069890644150284,
+            -0.00010725753918890802,
+            0.00943698424646706,
+            -0.013540245857270492,
+            -0.008293042205008774,
+            0.007212974960781723,
+            0.006708343474532583,
+            -0.0012017968151025369,
+            -0.0078026000239507145,
+            -0.02527526731750743,
+            0.0027578400032925625,
+            -0.002446811971857853,
+            -0.004899702349784173,
+            0.0019960465173086183,
+            0.001479503476855694,
+            -0.012645905858466798,
+            0.01216748847452763,
+            -0.013378877368040743,
+            -0.012053251571418766,
+            0.012508842412556306,
+            -0.00027167241745418584,
+            0.01682680930386839,
+            -0.020607299598265426,
+            0.0010702151063620052,
+            -0.028075439824932304,
+            -0.007671848153932823,
+            -0.015486630894165054,
+            -0.0009776136330459065,
+            -0.0012939921569540803,
+            -0.03481880531471923,
+            0.012889447665794033,
+            -0.005200149829198305,
+            -0.006094702500154477,
+            -0.00459778303079732,
+            -0.024583504817899775,
+            0.006072281238752663,
+            0.00607350167867964,
+            -0.006506508970816635,
+            -0.016816439635445445,
+            0.008428048142588516,
+            -0.0011040593013238872,
+            0.004873670535200424,
+            -0.007475800310154908,
+            -0.0005638920132076605,
+            -0.013047135093134946,
+            -0.005463454269764494,
+            -0.0008522562324122834,
+            0.01827123304871433,
+            0.0044055851550317835,
+            -0.014526381222031981,
+            0.016862445236481526,
+            0.013250692156820124,
+            0.01602714293656341,
+            -0.0071089293427753115,
+            -0.0004265739315250755,
+            -0.02750630108787212,
+            -0.00327225092954903,
+            -0.023687644154808972,
+            0.00041213204365507744,
+            0.009447660230507965,
+            0.0203905264889617,
+            0.012694421477479161,
+            0.004620617778996077,
+            -0.001557511764988156,
+            0.0066751090724502,
+            0.02696959560423252,
+            -0.01099183126742224,
+            -0.006969111598423376,
+            -0.01884504192789845,
+            0.01113023717919988,
+            0.025135748541199255,
+            0.003381303139273845,
+            -0.011317957771935042,
+            0.008112598754655146,
+            -0.0009247845526290428,
+            -0.010302980598863016,
+            0.02889587382004902,
+            -0.0391134481322538,
+            -0.015599591108049343,
+            -0.004605824192962023,
+            0.007406520608110251,
+            -0.015596917115435033,
+            0.00770468362805995,
+            -0.012614339656105264,
+            0.001699049329817941,
+            -0.005851477164035241,
+            0.03535087340659861,
+            -0.0031950960727572925,
+            0.0006909856233673865,
+            -0.003395233050301427,
+            0.003531135044131077,
+            0.012355948349937309,
+            -0.004444221317023453,
+            -0.030726300006909638,
+            -0.0022983766869702346,
+            -0.011198252640110788,
+            0.009274666307515955,
+            -0.016207714291658314,
+            -0.017342435976846468,
+            -0.009424737744066259,
+            -0.021827243671837953,
+            -0.012403084982789827,
+            0.029747453318486338,
+            -0.005824001042844527,
+            0.01747569398142269,
+            -0.005219060548639168,
+            -0.008365003057800934,
+            0.00021272411948388247,
+            0.01344335658390732,
+            -0.028498240040284714,
+            0.00041963499785238845,
+            -0.002322931702641349,
+            0.0126393550661767,
+            -0.001595134597082556,
+            -0.00983073906977773,
+            0.006903313245694627,
+            0.0010522208685011098,
+            -0.015648940549713764,
+            -0.0007573245570947632,
+            -0.008533504878052644,
+            0.020475329445149498,
+            0.00027791387213409814,
+            0.001074213629646005,
+            0.013864331671732542,
+            -0.02102352993295736,
+            0.01576938071930521,
+            -0.021722775924283996,
+            0.01893267442161547,
+            -0.006577103101038035,
+            0.005772016664149475,
+            0.008283223343661555,
+            0.014068789332250016,
+            -0.014030042849491258,
+            -0.00663752723080222,
+            -0.034436220972634726,
+            -0.0122621879114314,
+            -0.001946072976826902,
+            -0.005308941654794638,
+            -0.014773123927909993,
+            -0.0011054822266082056,
+            -0.026205461509573584,
+            -0.006460373065662098,
+            0.012822188046808264,
+            0.019900312098359022,
+            -0.008409070094265829,
+            0.014163132434889265,
+            -0.006743711872055091,
+            -0.0028230771810882492,
+            -0.012652021749331811,
+            -0.018061180380009395,
+            -0.01083672324723357,
+            -0.01890966636827258,
+            -0.0018635408070944168,
+            0.004709222372611061,
+            -0.015760618199235518,
+            0.011435131737642627,
+            0.03198874888333405,
+            -0.01567271472650171,
+            0.008243406188694909,
+            0.016411496328333205,
+            0.0034236971031973503,
+            0.011728079139322837,
+            0.0038311415233052795,
+            0.004621789116267879,
+            -0.011109450058055078,
+            -0.011473158133621661,
+            0.006524490579223986,
+            0.00954511918454185,
+            -0.004913142601134428,
+            0.024347072255007807,
+            0.0008745419206472714,
+            0.005045794664611216,
+            0.005304506902446091,
+            0.011488187043525824,
+            -0.013135271908307747,
+            0.019746484188715552,
+            -0.002055210086078267,
+            -0.015571665869889405,
+            -0.01535499169234956,
+            -0.01105419264826655,
+            -0.018626399757822328,
+            0.0017108083455869296,
+            0.0181439079855921,
+            -0.0012493542718277685,
+            0.012137753522284387,
+            -0.00924460959150499,
+            -0.005019437549649743,
+            -0.0027156377206998474,
+            -0.038211954658821295,
+            -0.0021878970332906457,
+            -0.018809487178512628,
+            -0.02498945889484935,
+            0.03505216568400238,
+            -4.26477081213836e-05,
+            0.00997643255528116,
+            -0.010310197159232386,
+            -0.0002454902098986129,
+            -0.011065403358802515,
+            0.014954998647970473,
+            0.004624123403100598,
+            0.008872923678983478,
+            0.008560280683876702,
+            -0.016419500515801093,
+            -0.002997557798375716,
+            0.007472390947008284,
+            -0.00374822278027057,
+            0.012680407710694952,
+            0.015353684671120731,
+            0.020886474047003733,
+            0.011161262397502876,
+            0.002045862158732102,
+            0.00887271411363814,
+            0.009563361150297699,
+            -0.0003513237086717377,
+            -0.010459621030424415,
+            0.015419039131459582,
+            -0.02204848739950703,
+            0.012172257374410352,
+            0.04346634265794738,
+            0.019197433209794286,
+            0.029488039318675732,
+            -0.02272810439246353,
+            0.01448092574459596,
+            -0.003801043409441206,
+            0.009476229966122152,
+            0.0049309968883087685,
+            0.02346191840355209,
+            0.003973840862678076,
+            0.0073546443568171405,
+            0.010647581332452525,
+            -0.00972051412996416,
+            0.025644022498404723,
+            -0.00846669916801108,
+            5.1518540791212095e-05,
+            -0.004407265804145216,
+            -0.026400029105519137,
+            0.01615193008014211,
+            0.0081722536401398,
+            0.0010702314763197463,
+            0.0010263623479474385,
+            -0.0019566604229607643,
+            0.007982949510899019,
+            0.007079322685771456,
+            -0.015274148711633536,
+            -0.003886053737590683,
+            -0.018527455865296014,
+            0.0012864569064749438,
+            -0.024337133883771334,
+            0.010466651250597847,
+            -0.017568706814132484,
+            -0.014822354159709854,
+            -0.021666681078831483,
+            -0.013232016867264097,
+            -0.02011653224168991,
+            0.0035858355517491515,
+            -0.008632755352062642,
+            0.013025728891203198,
+            -0.016962939287139683,
+            -0.021494971929779273,
+            0.008430887589398292,
+            0.00870802359711456,
+            0.007481282677955534,
+            0.009728368603458082,
+            0.014996157358749115,
+            0.01736292711410622,
+            -0.022712386143092265,
+            -0.023163001611599246,
+            -0.006682209717364464,
+            -0.003531495447167166,
+            0.006762379833125297,
+            0.00933681196321128,
+            0.012884185500343249,
+            -0.007077895849501418,
+            -0.007554762054641417,
+            -0.013283067570021503,
+            0.006197966297581871,
+            -0.003993067576677973,
+            -0.009464789349849612,
+            -0.0015662739825834614,
+            -0.0061389795593180686,
+            0.002753664732282926,
+            -0.0001791088888074743,
+            0.0007595261237141144,
+            0.017122834676182343,
+            -0.008176319370751018,
+            -0.009787747261447871,
+            -0.02027228453943018,
+            -0.010683273300243025,
+            -0.015604745521156111,
+            -0.0016279406232527138,
+            -0.006506713191861036,
+            -0.014420323628589382,
+            0.010888206569219648,
+            -0.011930708571093279,
+            -0.00887210117616784,
+            0.010807147705360856,
+            0.0073900910661282935,
+            -0.003431679006289002,
+            0.037804020313161775,
+            0.0011945597874820288,
+            0.00778778394048267,
+            -0.003366543306746973,
+            0.005296727102519836,
+            -0.017761251541598876,
+            -0.026382400396890007,
+            0.0010357716464853195,
+            -0.01775777668381489,
+            0.010807713750049491,
+            -0.008515368407459745,
+            0.0016051389544388124,
+            -0.00034958600424912995,
+            -0.009585854918521076,
+            0.0036549720706351473,
+            -0.008110927907950148,
+            0.004212219431447508,
+            -0.03507890639477636,
+            0.011409377585617926,
+            -0.01819120811524444,
+            -0.008962242411035718,
+            0.01710177594046791,
+            0.037560936862121835,
+            0.0028513388998398104,
+            0.00030058353811717895,
+            0.0014792184333546447,
+            0.004427921639326168,
+            0.021438127785303025,
+            0.006339483221420334,
+            -0.001187747087595264,
+            -0.02237982599829809,
+            0.0062435650294483695,
+            0.002200578793697352,
+            0.010092828474934744,
+            0.023105023225115266,
+            -0.016797326021070008,
+            -0.015378520287395565,
+            -0.012732527281000097,
+            -0.010797463778182385,
+            0.009881499418260473,
+            0.02801767062785128,
+            0.01811139137849365,
+            -0.017358098300028903,
+            -0.020207173948076473,
+            0.0016242901745373126,
+            -0.026331540602868762,
+            0.007269751661479859,
+            0.11940631114242901,
+            -0.0045382135734049305,
+            -0.005994407567723939,
+            -0.008679712231646732,
+            -0.004482553208064535,
+            0.016709571294385293,
+            -0.013387043582971358,
+            0.069179848974036,
+            -0.007221545718220735,
+            0.0037392229364642915,
+            0.0028916224964057385,
+            0.00462729158956839,
+            -0.009727339449635306,
+            0.011570145487622916,
+            -0.011287365791681587,
+            -0.012991994493845187,
+            -0.013076194601468383,
+            -0.0180830261130804,
+            -0.007272768410218907,
+            -0.004155564814758654,
+            0.004204993170487392,
+            -0.005525575983182816,
+            0.008051108079520902,
+            -0.017551645696802648,
+            -0.010800070994485792,
+            0.013007176804761548,
+            -0.0023627485852619817,
+            0.005390986239687555,
+            -0.029668390278675554,
+            -0.0007650391259819867,
+            -0.0002446394046947245,
+            0.021280292985495608,
+            0.003137771886236958,
+            -0.0019077298815862946,
+            -0.010810159675564049,
+            0.007040107942856957,
+            0.023835993694726953,
+            0.021875366566457236,
+            0.0180511566130312,
+            0.012135351541044354,
+            0.020483975362807574,
+            -0.008667079096167644,
+            0.026324312888512866,
+            -0.001552207146726937,
+            -0.00397810465151493,
+            0.0077577136352257004,
+            -0.009849600842081875,
+            -0.00816392268697345,
+            0.00677353736462267,
+            -0.017288900282504148,
+            -0.020418071729964713,
+            -0.020383762873109595,
+            0.025741408097358233,
+            -0.011755094311186617,
+            0.002737129201159793,
+            -0.008266210469668866,
+            0.027287436144516414,
+            -0.006283892307996516,
+            -0.0029440642954701632,
+            0.009428876742171362,
+            -0.03788574758692254,
+            0.008834605607057885,
+            0.006463342392514337,
+            0.026711196829464153,
+            -3.9827332346130654e-05,
+            0.005565561841469247,
+            -0.0006738085601962865,
+            -0.006104320995853506,
+            -0.010792830169647851,
+            2.4565453208679464e-05,
+            0.017983908748172493,
+            0.0035181716362611077,
+            0.016970206528324653,
+            -0.005449808667944701,
+            -0.023171388778833197,
+            0.007092509502738752,
+            -0.01555320330471131,
+            -0.023306285898839978,
+            -0.002028783456704567,
+            -0.03196436161255752,
+            0.0033412618908711163,
+            0.007527633574188688,
+            -0.01890570959143866,
+            -0.017859735533232564,
+            0.010329340599904344,
+            0.004382396539605985,
+            -0.0016746397858953155,
+            0.006643785064338525,
+            -0.03307812909454471,
+            -0.01927475592253288,
+            0.004686970010943116,
+            0.012733525168279355,
+            -0.012993441126970916,
+            0.015291832952199082,
+            -0.020491167895268936,
+            0.016451976170508816,
+            -0.012259387189535237,
+            0.018393274368854132,
+            -0.006997668664226754,
+            0.011843450716200625,
+            0.011457415642860215,
+            0.009560877634897626,
+            -0.01790001115302499,
+            0.0214879415944361,
+            0.0066829070506954115,
+            0.03900547021123392,
+            0.02512484499576633,
+            -0.012651294056575184,
+            -0.0018425680226742782,
+            -0.00011361458320146007,
+            0.014982647266052349,
+            0.0061874424115559,
+            -0.003689464024837091,
+            0.02705902205734107,
+            0.018239469373522537,
+            -0.021070704644859155,
+            0.020037508334843183,
+            -0.00973804771680803,
+            0.018876283979591423,
+            -0.013303080480396818,
+            0.010592424494628194,
+            0.004717870260467274,
+            0.01070131238869639,
+            0.009087656015197565,
+            0.006708673669681232,
+            0.025928065153829043,
+            0.03663367449576964,
+            -0.00511073108390551,
+            0.002161381031392467,
+            0.004864092591944181,
+            0.012814518637357098,
+            0.001556833001747549,
+            0.007915766323215207,
+            -0.007012785514562176,
+            0.005834786334750152,
+            -0.002311533032619821,
+            -0.00122162928193159,
+            0.019525999280692766,
+            0.000824163362963678,
+            0.012875575199417254,
+            0.013259376059185821,
+            -0.002390133687709065,
+            -0.005682644446302278,
+            0.0019262218417782657,
+            -0.018193478045923098,
+            -0.015131696847666434,
+            0.029463723011600933,
+            -0.011447039651934148,
+            -0.008083114389964213,
+            0.0020642857121204753,
+            0.012160202510990566,
+            -0.00866944494225225,
+            -0.005378300635997713,
+            0.0038280364584418335,
+            0.014427318067525318,
+            -0.01799358399064332,
+            0.003381165206832161,
+            0.005252552821854107,
+            0.019433554824232185,
+            -0.00028791985674147885,
+            -0.013487678738391956,
+            0.0005317922815671783,
+            0.014982152801567637,
+            0.011902348165369078,
+            0.0016056390299304837,
+            -0.006935777197137747,
+            -0.00446584737370504,
+            0.004586814866182051,
+            0.006269466956110759,
+            0.0027516492952993445,
+            0.008157662045793032,
+            -0.0025001885993292637,
+            -0.016648747436203294,
+            0.017735757469839637,
+            -0.0030669873770852725,
+            0.018058248627079793,
+            -0.0038238312452176745,
+            -0.0147172470474118,
+            0.0043328417868095355,
+            -0.001888507803054499,
+            -0.018556943283494755,
+            0.009407930200720534,
+            0.002560949424273885,
+            0.010411159836451958,
+            0.0052571230238294825,
+            -0.007499470181994448,
+            -6.145889479758644e-05,
+            -0.015163819825329975,
+            0.022836682800304944,
+            0.016390944816242627,
+            0.001713572907204073,
+            -0.0035687505973373878,
+            -0.031210299824387475,
+            0.0029076511444359886,
+            -0.007822231980296341,
+            0.017821806420465356,
+            0.008250605126658402,
+            0.007908773724009905,
+            0.008351355849444162,
+            -0.00743970642051451,
+            -0.002411981861500092,
+            0.012543476434788293,
+            0.0202407777715137,
+            -0.0024635818448627447,
+            0.02535771413579989,
+            -0.012533449594824386,
+            -0.0022489292647974992,
+            0.0012362917358624392,
+            0.01578939075186126,
+            -0.0008001783766371555,
+            0.006311837812042848
+          ],
+          "keyphrases": [
+            "Switch Transformers",
+            "Mixture of Experts",
+            "sparsely-activated model",
+            "training instability",
+            "multilingual settings"
+          ],
+          "title": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "9cc3633d-79f3-4045-b6a2-d3df9e569d10",
+        "properties": {
+          "page_content": "5 Designing Models with Data, Model, and Expert-Parallelism 18 5.1 Data Parallelism 20 5.2 Model Parallelism 20 5.3 Model and Data Parallelism 21 5.4 Expert and Data Parallelism 22 5.5 Expert, Model and Data Parallelism 22 5.6 Towards Trillion Parameter Models 22 ",
+          "keyphrases": [
+            "Designing Models",
+            "Data Parallelism",
+            "Model Parallelism",
+            "Expert Parallelism",
+            "Trillion Parameter Models"
+          ],
+          "embedding": [
+            -0.004642441403120756,
+            0.005233492702245712,
+            -0.022710707038640976,
+            0.004320049658417702,
+            0.027123892679810524,
+            -0.0023946543224155903,
+            -0.029717355966567993,
+            0.04969131574034691,
+            -0.019472461193799973,
+            0.01448613591492176,
+            0.021908311173319817,
+            -0.0279119610786438,
+            0.043014224618673325,
+            -0.02931615710258484,
+            -0.037196844816207886,
+            0.025519099086523056,
+            0.024114903062582016,
+            0.01901394873857498,
+            -0.010932662524282932,
+            -0.05880425497889519,
+            0.020575758069753647,
+            -0.04052106291055679,
+            -0.04877429082989693,
+            0.019085591658949852,
+            0.03307022899389267,
+            0.024630729109048843,
+            0.0260635819286108,
+            0.01029504369944334,
+            -0.05450569838285446,
+            0.01792498119175434,
+            0.012745220214128494,
+            -0.014686735346913338,
+            0.02420087344944477,
+            -0.021263526752591133,
+            -0.03960403427481651,
+            -0.017409155145287514,
+            0.05966396629810333,
+            -0.004216167610138655,
+            -0.01887066476047039,
+            0.022108910605311394,
+            0.006763062439858913,
+            0.0028657044749706984,
+            -0.030462438240647316,
+            0.002539730630815029,
+            0.006548134610056877,
+            -0.02255309373140335,
+            -0.047570694237947464,
+            0.003571384120732546,
+            0.01928619109094143,
+            -0.024874314665794373,
+            0.008267557248473167,
+            -0.046309784054756165,
+            0.025963282212615013,
+            0.00751530984416604,
+            -0.022108910605311394,
+            4.404901119414717e-05,
+            0.028943615034222603,
+            0.037196844816207886,
+            0.007013811729848385,
+            0.017695724964141846,
+            0.00950697436928749,
+            -0.033901285380125046,
+            -0.05513615533709526,
+            0.018369166180491447,
+            -0.012265214696526527,
+            -0.035477422177791595,
+            -0.017967967316508293,
+            0.017595425248146057,
+            -0.026450451463460922,
+            -0.011283711530268192,
+            -0.0028764507733285427,
+            0.014801363460719585,
+            0.003943925723433495,
+            -0.006902765482664108,
+            0.009463989175856113,
+            -0.014521957375109196,
+            0.029975268989801407,
+            0.040062546730041504,
+            0.027654048055410385,
+            -0.001376433647237718,
+            -0.02430117316544056,
+            0.004194675013422966,
+            -0.010445492342114449,
+            0.02546178363263607,
+            0.02044680155813694,
+            -0.00542692793533206,
+            -0.005899769254028797,
+            -0.06487955152988434,
+            -0.017265869304537773,
+            0.010130265727639198,
+            -0.04218316823244095,
+            0.005924844183027744,
+            0.025934625416994095,
+            -0.009299211204051971,
+            -0.0171512421220541,
+            0.027553748339414597,
+            0.028069574385881424,
+            0.007192918099462986,
+            -0.01659242808818817,
+            -0.017409155145287514,
+            0.0003219439822714776,
+            0.047398753464221954,
+            0.0451061874628067,
+            -0.01139833964407444,
+            0.04106554388999939,
+            -0.009227568283677101,
+            0.005846037063747644,
+            0.03155140578746796,
+            -0.013518961146473885,
+            0.03499025106430054,
+            -0.025791339576244354,
+            -0.009772052057087421,
+            -0.016950642690062523,
+            0.008518306538462639,
+            -0.038772981613874435,
+            -0.007185753900557756,
+            -0.017867667600512505,
+            -0.04335810989141464,
+            0.020719043910503387,
+            -0.04120882973074913,
+            0.013017462566494942,
+            0.017982294782996178,
+            -0.02114889957010746,
+            0.03243977576494217,
+            -0.04295691102743149,
+            0.029574070125818253,
+            -0.013239555060863495,
+            -0.021335169672966003,
+            -0.015431818552315235,
+            -0.02100561372935772,
+            0.02230951003730297,
+            0.018039610236883163,
+            -0.03513353690505028,
+            -0.027768677100539207,
+            0.008819205686450005,
+            -0.032353803515434265,
+            0.025691039860248566,
+            0.0003951537946704775,
+            -0.006573209539055824,
+            0.002344504464417696,
+            -0.0020274859853088856,
+            -0.018612749874591827,
+            0.0076227737590670586,
+            -0.053502701222896576,
+            -0.020246202126145363,
+            -0.026536423712968826,
+            0.01940082013607025,
+            0.042068541049957275,
+            -0.0010889676632359624,
+            0.026822993531823158,
+            -0.013318361714482307,
+            0.0033672028221189976,
+            0.010115936398506165,
+            0.018999621272087097,
+            -0.00021448006737045944,
+            -0.006286639254540205,
+            -0.007938001304864883,
+            0.022982949391007423,
+            -0.004549305886030197,
+            0.001577928545884788,
+            0.009292046539485455,
+            0.009901008568704128,
+            0.00019175592751707882,
+            0.05324478819966316,
+            0.0067773908376693726,
+            0.000919712008908391,
+            0.061039503663778305,
+            0.0037110873963683844,
+            -0.04668232426047325,
+            0.024043260142207146,
+            0.0024286846164613962,
+            -0.00724306795746088,
+            0.03946075215935707,
+            0.012451485730707645,
+            -0.0010235938243567944,
+            0.017208555713295937,
+            0.013339853845536709,
+            0.01571838930249214,
+            -0.004660351667553186,
+            0.03467502444982529,
+            -0.0015474804677069187,
+            -0.017796024680137634,
+            -0.0029695862904191017,
+            0.0010003099450841546,
+            0.04794323444366455,
+            0.0009313539485447109,
+            0.029545413330197334,
+            0.009693245403468609,
+            0.01960141770541668,
+            -0.017380498349666595,
+            0.02044680155813694,
+            0.02574835531413555,
+            0.015016291290521622,
+            0.012465814128518105,
+            -0.029330484569072723,
+            -0.03258306160569191,
+            0.01591898873448372,
+            -0.010946990922093391,
+            -0.01806826703250408,
+            0.010861019603908062,
+            0.006372610107064247,
+            -0.0444757342338562,
+            -0.0026239105500280857,
+            -0.01911424845457077,
+            -0.000753590720705688,
+            0.0021922639571130276,
+            -0.006401267368346453,
+            0.011090275831520557,
+            0.03828581050038338,
+            -0.0343024842441082,
+            -0.006039472296833992,
+            0.016004959121346474,
+            -0.024917300790548325,
+            -0.006129025481641293,
+            -0.003829297609627247,
+            0.0020471876487135887,
+            -0.03355740010738373,
+            0.0253901407122612,
+            0.013332690112292767,
+            -0.04043509066104889,
+            -0.008969655260443687,
+            -0.008145765401422977,
+            0.006798883900046349,
+            -0.02054710127413273,
+            0.003039437811821699,
+            -0.02093397080898285,
+            0.006709330715239048,
+            -0.015517789870500565,
+            -0.02230951003730297,
+            -0.02633582428097725,
+            0.017280198633670807,
+            0.003111080499365926,
+            -0.00244122208096087,
+            -0.010932662524282932,
+            0.030978266149759293,
+            -0.013848517090082169,
+            -0.0075368029065430164,
+            -0.020805014297366142,
+            -0.03857238218188286,
+            0.03261171653866768,
+            0.01520256232470274,
+            -0.022395480424165726,
+            -0.04120882973074913,
+            -0.01995963230729103,
+            0.008718905970454216,
+            0.009026968851685524,
+            -0.015804359689354897,
+            0.052557021379470825,
+            -0.014758378267288208,
+            -0.001351358718238771,
+            0.0011588193010538816,
+            -0.0005270209512673318,
+            0.015575103461742401,
+            0.024731028825044632,
+            -0.010846691206097603,
+            0.006917094346135855,
+            0.013934488408267498,
+            0.0006443357560783625,
+            -0.005061550531536341,
+            -0.0090484619140625,
+            0.05284358933568001,
+            -0.030491095036268234,
+            -0.047742635011672974,
+            0.04545007273554802,
+            -0.032840974628925323,
+            0.009772052057087421,
+            0.016778700053691864,
+            0.001771363546140492,
+            0.03524816408753395,
+            0.007343367673456669,
+            -0.00307705020532012,
+            -0.015317190438508987,
+            0.0018662899965420365,
+            -0.012673578225076199,
+            0.02412923239171505,
+            -0.03587862104177475,
+            0.0015277786878868937,
+            0.007644266821444035,
+            0.02437281608581543,
+            0.0067021665163338184,
+            -0.021822338923811913,
+            0.02655075117945671,
+            -0.008102779276669025,
+            0.002068680478259921,
+            -0.005036475602537394,
+            -0.0005041848635300994,
+            -0.01732318289577961,
+            -0.005151103716343641,
+            -0.015876002609729767,
+            -0.02357041835784912,
+            -0.024487445130944252,
+            -0.01950111798942089,
+            -0.01676437072455883,
+            0.01785333827137947,
+            0.00706754345446825,
+            0.005398270674049854,
+            0.007572623901069164,
+            0.012995969504117966,
+            0.007196500431746244,
+            0.005982158239930868,
+            -0.03272634372115135,
+            0.013712395913898945,
+            0.007823373191058636,
+            -0.004277063999325037,
+            -0.023341162130236626,
+            -0.0018430062336847186,
+            0.008704577572643757,
+            0.012687906622886658,
+            0.0016218096716329455,
+            -0.009549960494041443,
+            0.012129094451665878,
+            0.0023803256917744875,
+            -0.00688127288594842,
+            0.0008588157943449914,
+            -0.06619777530431747,
+            0.05814514309167862,
+            -0.0012635965831577778,
+            0.009528467431664467,
+            -0.057887230068445206,
+            -0.018913649022579193,
+            0.05909082666039467,
+            0.008109943941235542,
+            -0.048487719148397446,
+            -0.011899838224053383,
+            0.016563771292567253,
+            -0.02391430363059044,
+            -0.00041127335862256587,
+            0.029129885137081146,
+            -0.03143677860498428,
+            -0.04163868725299835,
+            -0.050837598741054535,
+            -0.040033891797065735,
+            -0.035477422177791595,
+            0.012172079645097256,
+            -0.011584610678255558,
+            -0.005043639801442623,
+            -0.026751350611448288,
+            0.006845451425760984,
+            -0.00787352304905653,
+            -0.03037646785378456,
+            0.0027152549009770155,
+            -0.02784031815826893,
+            0.0009698618669062853,
+            0.010158922523260117,
+            -0.005982158239930868,
+            -0.004366617184132338,
+            -0.03209589049220085,
+            -0.012917162850499153,
+            0.006064547225832939,
+            0.0251179002225399,
+            0.0007222470594570041,
+            0.02100561372935772,
+            0.021349499002099037,
+            0.010352357290685177,
+            -0.007099783048033714,
+            -0.0015698687639087439,
+            0.0029731683898717165,
+            0.004764233715832233,
+            0.003082423470914364,
+            -0.009951158426702023,
+            0.035162195563316345,
+            -0.05676960572600365,
+            -0.029860639944672585,
+            0.006770226638764143,
+            -0.01111893355846405,
+            -0.024071916937828064,
+            0.020174559205770493,
+            -0.0023713705595582724,
+            -0.04275631159543991,
+            0.01176371704787016,
+            -0.025590740144252777,
+            0.033299487084150314,
+            0.02742479182779789,
+            0.02524685673415661,
+            -0.010130265727639198,
+            -0.024745358154177666,
+            -0.004506320226937532,
+            -0.02216622419655323,
+            -0.015976302325725555,
+            0.020088588818907738,
+            -0.010173250921070576,
+            -0.005824544467031956,
+            0.011656252667307854,
+            -0.002566596493124962,
+            -0.0036824301350861788,
+            0.014385836198925972,
+            0.02441580221056938,
+            0.009986979886889458,
+            -0.04665366932749748,
+            0.00562036270275712,
+            -0.01613391563296318,
+            -0.029516756534576416,
+            -0.0067845555022358894,
+            0.011727895587682724,
+            0.0015833016950637102,
+            -0.010338028892874718,
+            -0.011835359036922455,
+            -0.025819996371865273,
+            0.014729720540344715,
+            -0.0009125477517955005,
+            -0.017265869304537773,
+            0.024143559858202934,
+            0.00978638045489788,
+            0.02381400391459465,
+            0.05378927290439606,
+            -0.004954086616635323,
+            0.009170254692435265,
+            0.021335169672966003,
+            0.021765025332570076,
+            0.0260635819286108,
+            -0.009893844835460186,
+            -0.04014851897954941,
+            -0.01577570289373398,
+            0.003718251595273614,
+            0.03711087256669998,
+            -0.018598422408103943,
+            -0.0006416491232812405,
+            0.027310162782669067,
+            -0.04023449122905731,
+            -0.001844797283411026,
+            0.007672923617064953,
+            0.011849688366055489,
+            0.05548003688454628,
+            -0.020790686830878258,
+            -0.013526124879717827,
+            0.016105258837342262,
+            -0.01596197299659252,
+            -0.018956635147333145,
+            -0.007844866253435612,
+            -0.020174559205770493,
+            0.012172079645097256,
+            -0.007350531872361898,
+            0.034245166927576065,
+            -0.00932786799967289,
+            0.0020794267766177654,
+            -0.04765666648745537,
+            0.022223537787795067,
+            0.05049371346831322,
+            -0.026149554178118706,
+            -0.020532771944999695,
+            -0.029330484569072723,
+            -0.013361346907913685,
+            0.011362518183887005,
+            -0.004449006170034409,
+            -0.058030515909194946,
+            -0.015245547518134117,
+            -0.02136382646858692,
+            -0.030032582581043243,
+            -0.04481961950659752,
+            -0.03413053974509239,
+            -0.0667136013507843,
+            0.021062927320599556,
+            -0.07714476436376572,
+            -0.032955601811409,
+            0.031236179172992706,
+            -0.016033615916967392,
+            -0.029574070125818253,
+            0.013318361714482307,
+            -0.008776219561696053,
+            -0.056339748203754425,
+            -0.03885895386338234,
+            0.05249970406293869,
+            0.016921984031796455,
+            -0.0033313813619315624,
+            -0.029803326353430748,
+            0.0007710536010563374,
+            -0.03166603296995163,
+            -0.028212860226631165,
+            0.004731994587928057,
+            0.012609099969267845,
+            0.023341162130236626,
+            -0.013261047191917896,
+            0.009972651489078999,
+            -0.002568387659266591,
+            0.013970308937132359,
+            0.02332683466374874,
+            0.02507491409778595,
+            -0.014357179403305054,
+            -0.004914683289825916,
+            -0.020260531455278397,
+            0.04398856312036514,
+            0.0203035157173872,
+            0.014149416238069534,
+            -0.006816794630140066,
+            -0.01901394873857498,
+            0.025275513529777527,
+            0.010775049217045307,
+            -0.005233492702245712,
+            1.3845774446963333e-05,
+            -0.006748734042048454,
+            0.03771267086267471,
+            -0.015603761188685894,
+            0.0014167326735332608,
+            0.002708090702071786,
+            -0.005896186921745539,
+            -0.02536148391664028,
+            -0.053502701222896576,
+            -0.016606757417321205,
+            0.0203035157173872,
+            -0.0028997347690165043,
+            -0.010108772665262222,
+            0.0023373402655124664,
+            -0.0171512421220541,
+            -0.02497461438179016,
+            0.025017600506544113,
+            -0.003610787680372596,
+            -0.01527420524507761,
+            -0.04633844271302223,
+            -0.0219226386398077,
+            -0.015446146950125694,
+            -0.015259875915944576,
+            0.025203870609402657,
+            0.028356146067380905,
+            0.009542795829474926,
+            0.01283835619688034,
+            0.017065269872546196,
+            -0.03272634372115135,
+            0.0004092584131285548,
+            -0.02563372626900673,
+            -0.0067845555022358894,
+            -0.005785140674561262,
+            -0.009822201915085316,
+            -0.01732318289577961,
+            0.02276802249252796,
+            -0.015302862040698528,
+            0.035964589565992355,
+            0.0021367408335208893,
+            -0.009263389743864536,
+            0.04851637780666351,
+            -0.016406157985329628,
+            0.002040023449808359,
+            0.008539799600839615,
+            -0.003204215783625841,
+            -2.553667400206905e-05,
+            -0.0020991284400224686,
+            -0.035018909722566605,
+            0.018383493646979332,
+            0.04843040555715561,
+            -0.0009501601452939212,
+            -0.030290495604276657,
+            -0.022524436935782433,
+            0.02563372626900673,
+            0.004309303127229214,
+            -0.006000068504363298,
+            0.01952977664768696,
+            -0.013963145203888416,
+            0.028069574385881424,
+            -0.048802945762872696,
+            0.0008507560123689473,
+            0.02461640164256096,
+            0.013347018510103226,
+            -0.04169600084424019,
+            0.048344433307647705,
+            -0.00925622507929802,
+            -0.038916267454624176,
+            -0.02240980975329876,
+            0.03762669861316681,
+            0.014371507801115513,
+            0.02100561372935772,
+            -0.018111253157258034,
+            0.012186408042907715,
+            0.009206075221300125,
+            -0.03708221763372421,
+            0.017523782327771187,
+            -0.029660040512681007,
+            0.004001239780336618,
+            0.004058553837239742,
+            0.02868570201098919,
+            -0.01623421534895897,
+            0.01816856674849987,
+            0.02556208334863186,
+            -0.006229325197637081,
+            -0.005118864588439465,
+            0.006662762723863125,
+            0.021091584116220474,
+            0.03742609918117523,
+            0.034846965223550797,
+            -0.02391430363059044,
+            0.012881341390311718,
+            0.013024627231061459,
+            0.03132215142250061,
+            -0.0009779216488823295,
+            0.0019397237338125706,
+            0.02020321600139141,
+            0.025590740144252777,
+            -0.003840043907985091,
+            -0.007307546213269234,
+            -0.008812041021883488,
+            0.0013379257870838046,
+            -0.00915592536330223,
+            0.0036860124673694372,
+            0.03289828822016716,
+            -0.010001308284699917,
+            -0.03928880766034126,
+            -0.0025271931663155556,
+            0.001104191760532558,
+            0.010273550637066364,
+            0.0023319669999182224,
+            0.05705617740750313,
+            -0.016606757417321205,
+            -0.006437088828533888,
+            0.013755381107330322,
+            -0.015732716768980026,
+            -0.024788344278931618,
+            -0.013769710436463356,
+            0.005043639801442623,
+            -0.023126235231757164,
+            -0.032525744289159775,
+            0.00379705848172307,
+            0.04387393593788147,
+            -0.016921984031796455,
+            0.013253883458673954,
+            0.03341411426663399,
+            -0.014385836198925972,
+            0.006494402885437012,
+            -0.0009349361062049866,
+            0.003689594566822052,
+            -0.00751530984416604,
+            -0.04871697723865509,
+            0.0206187441945076,
+            0.078864187002182,
+            0.005863947793841362,
+            0.02963138371706009,
+            -0.011964316479861736,
+            0.012702235020697117,
+            -0.013841352425515652,
+            0.024774014949798584,
+            9.951606625691056e-05,
+            0.014858677983283997,
+            -0.008045465685427189,
+            -0.019902316853404045,
+            -0.003714669495820999,
+            -0.004825130105018616,
+            -0.01394165214151144,
+            0.009356524795293808,
+            -0.014873006381094456,
+            -0.023226534947752953,
+            0.006652016658335924,
+            0.0009465779876336455,
+            -0.03441711142659187,
+            -0.0020203215535730124,
+            -0.02153576910495758,
+            0.023197878152132034,
+            0.026221195235848427,
+            -0.0019253952195867896,
+            0.0018125580390915275,
+            0.005254985764622688,
+            -0.010409671813249588,
+            0.01952977664768696,
+            -0.003539144992828369,
+            0.03759804368019104,
+            0.01328970491886139,
+            0.006046636495739222,
+            0.041466742753982544,
+            0.01816856674849987,
+            -0.03473233804106712,
+            -0.014930319972336292,
+            0.012422828935086727,
+            -0.005072297062724829,
+            0.0164778009057045,
+            -0.001104191760532558,
+            0.007866358384490013,
+            -0.0036519821733236313,
+            0.02384266071021557,
+            0.012931491248309612,
+            -0.028943615034222603,
+            -0.004463334567844868,
+            0.006261564325541258,
+            -0.018741706386208534,
+            0.01409926638007164,
+            -0.000238435561186634,
+            -0.04983460158109665,
+            0.0016110632568597794,
+            0.034560397267341614,
+            0.004470498766750097,
+            -0.014844349585473537,
+            0.006394103169441223,
+            0.04467633366584778,
+            0.012387007474899292,
+            0.010008472949266434,
+            0.0066126128658652306,
+            0.014686735346913338,
+            -0.010982812382280827,
+            -0.015317190438508987,
+            -0.028399132192134857,
+            -0.007422174792736769,
+            -0.003703922964632511,
+            -0.02430117316544056,
+            -0.010015636682510376,
+            -0.03292694315314293,
+            -0.0036681017372757196,
+            0.014973306097090244,
+            -0.006917094346135855,
+            -0.005183342844247818,
+            -0.015560775063931942,
+            -0.010230564512312412,
+            -0.004183928482234478,
+            0.039403434842824936,
+            0.025060584768652916,
+            0.01366224605590105,
+            -0.004932594019919634,
+            -0.00029597352840937674,
+            -0.035792648792266846,
+            -0.0056955874897539616,
+            0.021550098434090614,
+            0.026149554178118706,
+            0.006852615624666214,
+            -0.00939951092004776,
+            -0.05292956158518791,
+            -0.015618089586496353,
+            0.0007746357587166131,
+            -0.00939951092004776,
+            0.02296862192451954,
+            0.015861675143241882,
+            0.012730891816318035,
+            0.03384396806359291,
+            -0.0060323080979287624,
+            -0.01746646873652935,
+            -0.010818034410476685,
+            0.03665236011147499,
+            -0.004388109780848026,
+            0.004237660672515631,
+            0.016262872144579887,
+            -0.005849619396030903,
+            -0.035792648792266846,
+            0.023541761562228203,
+            0.0064478348940610886,
+            0.0024143559858202934,
+            0.016090931370854378,
+            -0.0026418212801218033,
+            -0.001410463941283524,
+            -0.019200220704078674,
+            0.012702235020697117,
+            0.023197878152132034,
+            0.035334136337041855,
+            -0.0018859917763620615,
+            0.013898666948080063,
+            0.043186165392398834,
+            0.004656769800931215,
+            -0.008790548890829086,
+            0.02676567994058132,
+            -0.019099920988082886,
+            -0.003883029567077756,
+            0.017480798065662384,
+            -0.01564674638211727,
+            0.04671098291873932,
+            0.011670581065118313,
+            0.005208417773246765,
+            -0.040492404252290726,
+            0.003263321006670594,
+            -0.003961836453527212,
+            -0.014142251573503017,
+            0.03160871937870979,
+            0.015689732506871223,
+            -0.04381662234663963,
+            0.014034787192940712,
+            0.016893327236175537,
+            0.006902765482664108,
+            0.012086108326911926,
+            0.022882649675011635,
+            -0.0038113868795335293,
+            0.03665236011147499,
+            0.017380498349666595,
+            -0.006086039822548628,
+            0.034560397267341614,
+            -0.01265924982726574,
+            0.003514070063829422,
+            0.01422105822712183,
+            -0.007758894935250282,
+            -0.001464195898734033,
+            -0.006175593007355928,
+            0.015517789870500565,
+            0.03619384765625,
+            -0.007680087815970182,
+            -0.03811386972665787,
+            0.03983329236507416,
+            -0.012193572707474232,
+            0.005885440390557051,
+            -0.04986325651407242,
+            0.010051458142697811,
+            -0.01858409307897091,
+            -0.007415010128170252,
+            9.391898493049666e-05,
+            0.008260393515229225,
+            -0.010359521955251694,
+            -0.0012224020902067423,
+            0.015661075711250305,
+            0.00903413351625204,
+            0.014958977699279785,
+            0.024286845698952675,
+            0.02994661219418049,
+            0.008704577572643757,
+            -0.015460475347936153,
+            -0.006064547225832939,
+            0.03903089463710785,
+            -0.01222222950309515,
+            -0.013225225731730461,
+            0.04530678689479828,
+            -0.025447456166148186,
+            -0.027697034180164337,
+            0.03665236011147499,
+            -0.03539144992828369,
+            0.021306512877345085,
+            0.04608052968978882,
+            -0.004029897041618824,
+            -0.02974601276218891,
+            0.01527420524507761,
+            -0.030920950695872307,
+            0.010911169461905956,
+            -0.0026633141096681356,
+            0.0012098646257072687,
+            0.005294389091432095,
+            0.016320187598466873,
+            0.005358867347240448,
+            0.007332621142268181,
+            0.011197740212082863,
+            -0.01792498119175434,
+            0.02107725664973259,
+            -0.03146543353796005,
+            -0.013769710436463356,
+            0.01057444978505373,
+            -0.027510762214660645,
+            -0.0005879171658307314,
+            0.013053284026682377,
+            -0.016563771292567253,
+            -0.030433781445026398,
+            0.018111253157258034,
+            0.009184583090245724,
+            0.06522343307733536,
+            0.01995963230729103,
+            0.016750043258070946,
+            0.009277718141674995,
+            -0.01577570289373398,
+            0.05049371346831322,
+            -0.000637171498965472,
+            0.03215320408344269,
+            0.020117245614528656,
+            -0.016248544678092003,
+            -0.016406157985329628,
+            0.014744049869477749,
+            0.015345847234129906,
+            0.008281885646283627,
+            -0.008016807958483696,
+            0.003514070063829422,
+            0.008955326862633228,
+            0.0017525573493912816,
+            0.02689463645219803,
+            0.006408431567251682,
+            0.02321220561861992,
+            0.03229648992419243,
+            0.0035570557229220867,
+            0.025232527405023575,
+            0.001274342997930944,
+            0.015790032222867012,
+            -0.01914290525019169,
+            0.014958977699279785,
+            0.01901394873857498,
+            0.00896249059587717,
+            0.0193578340113163,
+            -0.006103950552642345,
+            -0.013719560578465462,
+            0.008253228850662708,
+            0.005799469538033009,
+            0.01058877818286419,
+            0.010166086256504059,
+            -0.013726724311709404,
+            0.014887334778904915,
+            -0.032038576900959015,
+            0.014006130397319794,
+            -0.050522368401288986,
+            -0.005785140674561262,
+            0.030204525217413902,
+            0.017108255997300148,
+            -0.025304170325398445,
+            0.018053937703371048,
+            -0.015345847234129906,
+            -0.007852029986679554,
+            -0.038601040840148926,
+            -0.014421657659113407,
+            -0.003650191007182002,
+            0.022710707038640976,
+            0.011226397007703781,
+            0.015603761188685894,
+            0.017308855429291725,
+            -0.011247890070080757,
+            -0.007715909276157618,
+            -0.00841800682246685,
+            -0.03911686688661575,
+            -0.04120882973074913,
+            -0.012193572707474232,
+            0.0006353803910315037,
+            0.00048045325092971325,
+            -0.010875348001718521,
+            -0.02226652391254902,
+            0.009499810636043549,
+            0.022839665412902832,
+            0.0010853855637833476,
+            -0.020074259489774704,
+            0.007393517531454563,
+            -0.022710707038640976,
+            -0.006143353879451752,
+            -0.0424983985722065,
+            -0.020045602694153786,
+            -0.02819853276014328,
+            0.013504632748663425,
+            -0.0126950703561306,
+            0.026665380224585533,
+            -0.003232872812077403,
+            0.0014373299200087786,
+            -0.006444253027439117,
+            0.009707573801279068,
+            -0.008826369419693947,
+            0.006100368220359087,
+            -0.03656638786196709,
+            0.05568063631653786,
+            -0.006125443149358034,
+            0.029831983149051666,
+            -0.016004959121346474,
+            0.05158267915248871,
+            0.00843233522027731,
+            -0.008919505402445793,
+            0.03762669861316681,
+            -0.003850790439173579,
+            0.010882512666285038,
+            0.008797712624073029,
+            -0.03246843069791794,
+            -0.005441256333142519,
+            -0.01834050938487053,
+            0.0008409051806665957,
+            0.010345193557441235,
+            -0.017552439123392105,
+            0.031150206923484802,
+            0.030175868421792984,
+            -0.004663933999836445,
+            0.058976199477910995,
+            0.003345709992572665,
+            -0.018927978351712227,
+            0.008511141873896122,
+            -0.059606652706861496,
+            -0.009772052057087421,
+            -0.028571072965860367,
+            0.011391174979507923,
+            0.015546446666121483,
+            0.026364481076598167,
+            0.0051905070431530476,
+            -0.014185236766934395,
+            0.027611061930656433,
+            0.04281362518668175,
+            0.0002937347162514925,
+            0.020031273365020752,
+            0.012351186014711857,
+            0.00960727408528328,
+            0.009091447107493877,
+            0.0021080838050693274,
+            0.017122583463788033,
+            0.01869872212409973,
+            -0.004921847488731146,
+            -0.0012752384645864367,
+            -0.01581868901848793,
+            -0.006078875623643398,
+            0.0036663105711340904,
+            -0.01732318289577961,
+            -0.008704577572643757,
+            -0.003494368400424719,
+            0.014958977699279785,
+            -0.012723728083074093,
+            0.017738711088895798,
+            -0.02384266071021557,
+            -0.0539325587451458,
+            0.00913443323224783,
+            -0.010789377614855766,
+            -0.02490297146141529,
+            0.009005476720631123,
+            -0.014973306097090244,
+            0.01401329506188631,
+            0.004506320226937532,
+            -0.006172011140733957,
+            -0.014127923175692558,
+            0.010373850353062153,
+            0.003519443329423666,
+            0.040807630866765976,
+            0.006315296050161123,
+            0.006766644772142172,
+            -0.01124072540551424,
+            0.013332690112292767,
+            -0.025304170325398445,
+            0.02342713437974453,
+            -0.010732063092291355,
+            -0.06098219007253647,
+            0.03086363710463047,
+            -0.012623428367078304,
+            -0.010674749501049519,
+            0.03413053974509239,
+            -0.012100436724722385,
+            -0.0006783659919165075,
+            -0.002858540276065469,
+            0.006641270127147436,
+            -0.02286832220852375,
+            0.008396513760089874,
+            -0.04226914048194885,
+            0.010954155586659908,
+            -0.04522081837058067,
+            -0.019945302978157997,
+            0.045020218938589096,
+            0.02149278298020363,
+            0.00312182679772377,
+            0.05103819817304611,
+            0.017652738839387894,
+            0.013776874169707298,
+            0.0035212342627346516,
+            0.014285536482930183,
+            -0.016076602041721344,
+            -0.014242551289498806,
+            -0.006856197956949472,
+            0.019372161477804184,
+            -0.005785140674561262,
+            -0.020776357501745224,
+            0.015016291290521622,
+            0.0035176521632820368,
+            0.0029534667264670134,
+            0.010087279602885246,
+            0.0010773257818073034,
+            -0.013139255344867706,
+            -0.013017462566494942,
+            0.013798367232084274,
+            0.032067231833934784,
+            -0.03395859897136688,
+            -0.0020077840890735388,
+            0.010889677330851555,
+            0.00779471592977643,
+            -0.003854372538626194,
+            -0.03117886371910572,
+            0.01304611936211586,
+            -0.02209458127617836,
+            0.006884855218231678,
+            -0.009026968851685524,
+            -0.013698067516088486,
+            -0.056483034044504166,
+            0.001492852927185595,
+            0.019902316853404045,
+            0.0005453793564811349,
+            -0.009865188039839268,
+            0.0406930036842823,
+            0.016463471576571465,
+            0.021378155797719955,
+            0.0018483793828636408,
+            0.013805530965328217,
+            0.014887334778904915,
+            -0.0006734405760653317,
+            3.0084299851296237e-06,
+            0.005118864588439465,
+            -0.0004316467384342104,
+            -0.0038615367375314236,
+            -0.0015510625671595335,
+            -0.0039725829847157,
+            0.03054841049015522,
+            0.02739613503217697,
+            0.006859780289232731,
+            0.003499741666018963,
+            0.020260531455278397,
+            -0.008998312056064606,
+            -0.005029311403632164,
+            0.010037129744887352,
+            0.013719560578465462,
+            0.006931422743946314,
+            0.007823373191058636,
+            0.01627720147371292,
+            0.002285399241372943,
+            0.003814968978986144,
+            0.013490303419530392,
+            0.012630592100322247,
+            0.0073720249347388744,
+            0.011541624553501606,
+            -0.008797712624073029,
+            0.019759032875299454,
+            -0.022639065980911255,
+            -0.0016880790935829282,
+            0.0008315020822919905,
+            -0.018842006102204323,
+            -0.0004016463935840875,
+            0.00967891700565815,
+            0.012816863134503365,
+            0.011462817899882793,
+            -0.002675851574167609,
+            0.02497461438179016,
+            -0.03630847483873367,
+            -0.011899838224053383,
+            0.004452588502317667,
+            0.028628388419747353,
+            -0.01360493153333664,
+            -0.01957276090979576,
+            0.013368511572480202,
+            -0.008231735788285732,
+            0.02700926549732685,
+            -0.01050280686467886,
+            0.0017534529324620962,
+            -0.013640752993524075,
+            0.05722811818122864,
+            -0.006132607348263264,
+            0.01367657445371151,
+            -0.0012071779929101467,
+            -0.00041284054168500006,
+            0.011011469177901745,
+            0.0015295697376132011,
+            0.04493424668908119,
+            0.026522094383835793,
+            0.009091447107493877,
+            0.018670065328478813,
+            0.02276802249252796,
+            0.016921984031796455,
+            0.005520063452422619,
+            -0.005788723006844521,
+            0.006401267368346453,
+            0.0006600075867027044,
+            -0.011305204592645168,
+            0.02132084034383297,
+            -0.007296800147742033,
+            -0.016721386462450027,
+            0.004011986311525106,
+            -0.010524299927055836,
+            -0.006343953311443329,
+            -0.013647917658090591,
+            0.007758894935250282,
+            0.03679564595222473,
+            -0.001645989017561078,
+            -0.009284882806241512,
+            -0.03928880766034126,
+            0.008281885646283627,
+            0.01693631336092949,
+            0.013733888976275921,
+            -0.020647400990128517,
+            0.005036475602537394,
+            0.020217545330524445,
+            -0.035792648792266846,
+            0.005409017205238342,
+            -0.021578755229711533,
+            0.011169083416461945,
+            -0.010424000211060047,
+            0.027367478236556053,
+            0.0006586642703041434,
+            0.003519443329423666,
+            0.0067881373688578606,
+            -0.009772052057087421,
+            0.02381400391459465,
+            -0.02167905494570732,
+            -0.0003752281772904098,
+            0.005434092134237289,
+            0.008718905970454216,
+            -0.029373470693826675,
+            0.006354699842631817,
+            0.018354836851358414,
+            -0.01591898873448372,
+            0.024817001074552536,
+            0.0285854022949934,
+            -0.008009644225239754,
+            -0.009105775505304337,
+            -0.010911169461905956,
+            0.005828126333653927,
+            0.002023903885856271,
+            0.029187200590968132,
+            -0.003188096219673753,
+            -0.0006156786694191396,
+            -0.007895016111433506,
+            0.010667584836483002,
+            0.0025522680953145027,
+            0.015001962892711163,
+            -0.008561291731894016,
+            -0.03143677860498428,
+            0.039403434842824936,
+            -0.0009385182056576014,
+            -0.008561291731894016,
+            -0.01690765656530857,
+            -0.007415010128170252,
+            0.0015958391595631838,
+            -0.008059794083237648,
+            0.00271346396766603,
+            -0.03476099669933319,
+            0.02146412618458271,
+            0.0020937554072588682,
+            -0.0005014982889406383,
+            0.0013191195903345942,
+            -0.01437867246568203,
+            -0.006849033758044243,
+            -0.0024089827202260494,
+            -0.013225225731730461,
+            0.012480142526328564,
+            0.004574380815029144,
+            -0.018526779487729073,
+            0.013991801999509335,
+            0.009428167715668678,
+            0.016821684315800667,
+            -0.0032973513007164,
+            0.005570212844759226,
+            0.020948300138115883,
+            -0.011348189786076546,
+            -0.03590727597475052,
+            -0.0036197430454194546,
+            -0.00271346396766603,
+            0.030118554830551147,
+            0.006268728524446487,
+            -0.023484447970986366,
+            0.01981634646654129,
+            0.004388109780848026,
+            0.007128439843654633,
+            0.0009922501631081104,
+            -0.008661591447889805,
+            -0.004739158786833286,
+            0.002597044687718153,
+            -0.002835256280377507,
+            -0.00011233337863814086,
+            -0.01022340077906847,
+            0.009478317573666573,
+            -0.00980070885270834,
+            0.005871111992746592,
+            -0.00884069874882698,
+            0.013597767800092697,
+            -0.006791719701141119,
+            -0.026665380224585533,
+            0.004051389638334513,
+            -0.002688389038667083,
+            -0.05235642194747925,
+            -0.0036448179744184017,
+            0.02013157308101654,
+            -0.0074579957872629166,
+            0.005681259091943502,
+            -0.0013862844789400697,
+            0.0028889882378280163,
+            0.017409155145287514,
+            0.008804877288639545,
+            0.026178210973739624,
+            0.016033615916967392,
+            0.01921454817056656,
+            -0.01623421534895897,
+            0.011054455302655697,
+            -0.009564288891851902,
+            0.039260152727365494,
+            0.002247786847874522,
+            -0.0068669444881379604,
+            -0.01736616902053356,
+            0.04418916255235672,
+            -0.00967891700565815,
+            -0.0015304653206840158,
+            -0.01455777883529663,
+            -0.014142251573503017,
+            0.03269768878817558,
+            -0.002256742212921381,
+            0.011512967757880688,
+            -0.004760651383548975,
+            0.008031136356294155,
+            0.0206617284566164,
+            -0.02556208334863186,
+            -0.02766837738454342,
+            -0.017767367884516716,
+            0.007214411161839962,
+            -0.004112286027520895,
+            0.04106554388999939,
+            -0.017910653725266457,
+            -0.0016836014110594988,
+            0.02718120627105236,
+            -0.0269662793725729,
+            -0.03418785333633423,
+            -0.010158922523260117,
+            0.014615092426538467,
+            -0.0010486687533557415,
+            -0.005057968199253082,
+            -0.01942947693169117,
+            -0.020647400990128517,
+            -0.000571797601878643,
+            0.0021868906915187836,
+            0.015603761188685894,
+            0.008862190879881382,
+            0.015388833358883858,
+            0.0019827093929052353,
+            0.0031576480250805616,
+            -0.012236557900905609,
+            0.013683739118278027,
+            0.010810869745910168,
+            0.002262115478515625,
+            -0.014521957375109196,
+            -0.011011469177901745,
+            -0.022366823628544807,
+            -0.008847862482070923,
+            -0.00808128621429205,
+            -0.00023933110060170293,
+            -0.0034531739074736834,
+            -0.015317190438508987,
+            0.02805524691939354,
+            0.00480363704264164,
+            0.00878338422626257,
+            0.015804359689354897,
+            -0.023584747686982155,
+            0.019099920988082886,
+            0.013920159079134464,
+            -0.010739227756857872,
+            0.003189887385815382,
+            -0.006770226638764143,
+            -0.012229394167661667,
+            -0.014858677983283997,
+            0.0038078047800809145,
+            0.015087934210896492,
+            0.011921330355107784,
+            -0.0003962731861975044,
+            -0.015345847234129906,
+            -0.01268074195832014,
+            -0.011713567189872265,
+            0.02563372626900673,
+            -0.030634380877017975,
+            -0.002115248003974557,
+            0.023942960426211357,
+            -0.01995963230729103,
+            -0.011899838224053383,
+            -0.03143677860498428,
+            0.0027994350530207157,
+            -0.02960272692143917,
+            -0.029402127489447594,
+            0.01897096447646618,
+            0.049146831035614014,
+            -0.013261047191917896,
+            -0.010345193557441235,
+            0.01627720147371292,
+            -0.016463471576571465,
+            -0.0021671890281140804,
+            -0.033614713698625565,
+            0.02616388164460659,
+            -0.021664725616574287,
+            0.0038472083397209644,
+            -0.008740399032831192,
+            -0.03499025106430054,
+            0.012687906622886658,
+            0.023541761562228203,
+            0.03338545560836792,
+            -0.001543898251838982,
+            0.00379705848172307,
+            -0.007637102622538805,
+            0.017839010804891586,
+            0.018111253157258034,
+            0.002573760924860835,
+            0.03301291540265083,
+            0.0011032961774617434,
+            -0.04642441123723984,
+            -0.00022880859614815563,
+            0.005151103716343641,
+            -0.0003194812743458897,
+            0.002509282436221838,
+            -0.01366224605590105,
+            -0.026808666065335274,
+            0.015059277415275574,
+            -0.026536423712968826,
+            -0.00040813899249769747,
+            -0.016434814780950546,
+            -0.010818034410476685,
+            -0.007780387531965971,
+            0.005509316921234131,
+            0.004495573695749044,
+            0.024000274017453194,
+            0.009557124227285385,
+            -0.0006985154468566179,
+            0.015016291290521622,
+            0.014529122039675713,
+            0.03069169446825981,
+            -0.0006680673686787486,
+            0.02219488099217415,
+            0.02686597965657711,
+            -0.013110597617924213,
+            0.007343367673456669,
+            0.0028083904180675745,
+            0.026350151747465134,
+            -0.03163737803697586,
+            0.02517521381378174,
+            0.01250163558870554,
+            0.003983329050242901,
+            0.02657940983772278,
+            -0.011985808610916138,
+            -0.00912010483443737,
+            -0.00014709123934153467,
+            0.004001239780336618,
+            -0.003247201442718506,
+            -0.0046925912611186504,
+            0.040635690093040466,
+            0.011491474695503712,
+            -0.024186545982956886,
+            -0.012401335872709751,
+            -0.0158903319388628,
+            0.028040917590260506,
+            -0.019773360341787338,
+            0.00017183032468892634,
+            -0.016492128372192383,
+            0.004918265156447887,
+            0.014314194209873676,
+            0.0014597182162106037,
+            0.01879902184009552,
+            0.016019288450479507,
+            -0.0006586642703041434,
+            0.018354836851358414,
+            0.007751730736345053,
+            -0.026393137872219086,
+            0.004839458502829075,
+            0.007436503190547228,
+            -0.014020458795130253,
+            0.007529638707637787,
+            0.015503461472690105,
+            -0.00407646456733346,
+            -0.013683739118278027,
+            -0.01230820082128048,
+            -0.00862576998770237,
+            0.004595873411744833,
+            -0.010638928040862083,
+            0.019300520420074463,
+            0.0018806185107678175,
+            -0.0038113868795335293,
+            0.019859332591295242,
+            -0.006734405644237995,
+            -0.023025935515761375,
+            0.01897096447646618,
+            -0.01437867246568203,
+            0.02441580221056938,
+            0.012530292384326458,
+            -0.022123238071799278,
+            0.00579588720574975,
+            0.009929666295647621,
+            0.04226914048194885,
+            -0.005878276191651821,
+            0.00615410041064024,
+            -0.02703792229294777,
+            -0.005584541708230972,
+            -0.02609223872423172,
+            0.0067845555022358894,
+            0.003377949120476842,
+            -0.011276546865701675,
+            -0.00021190541156101972,
+            -0.0012447903864085674,
+            0.014142251573503017,
+            0.012809699401259422,
+            -0.014873006381094456,
+            -0.0011149381753057241,
+            0.0036860124673694372,
+            -0.01528853364288807,
+            -0.0024698791094124317,
+            0.0011955361114814878,
+            -0.008317707106471062,
+            0.02293996512889862,
+            0.0393461212515831,
+            -0.021693382412195206,
+            -0.01760975457727909,
+            0.016878999769687653,
+            -0.003571384120732546,
+            -0.022438466548919678,
+            -0.007508145645260811,
+            0.01113326195627451,
+            0.0054376740008592606,
+            -0.0025558501947671175,
+            -0.03372934088110924,
+            0.021507112309336662,
+            0.006472909823060036,
+            0.014958977699279785,
+            0.00788785144686699,
+            0.0026955532375723124,
+            0.030347811058163643,
+            -0.020002616569399834,
+            -0.018770365044474602,
+            -0.00569916982203722,
+            -0.020188888534903526,
+            0.019099920988082886,
+            0.008790548890829086,
+            -0.011770880781114101,
+            0.02829883247613907,
+            -0.026493437588214874,
+            0.015560775063931942,
+            0.009299211204051971,
+            -0.0051976717077195644,
+            0.010603106580674648,
+            0.0028227188158780336,
+            -0.004914683289825916,
+            -0.04854503273963928,
+            -0.005100953858345747,
+            0.008074122481048107,
+            0.02104859985411167,
+            -0.0060251434333622456,
+            -0.031408119946718216,
+            0.02974601276218891,
+            0.0008785175159573555,
+            -0.002616746351122856,
+            -0.010058622807264328,
+            0.00706754345446825,
+            -0.020561428740620613,
+            -0.006863362155854702,
+            0.007053215056657791,
+            0.020461129024624825,
+            0.0059427544474601746,
+            0.028485102578997612,
+            -0.006028725765645504,
+            0.05232776328921318,
+            -0.0006617986364290118,
+            0.009435332380235195,
+            0.0026418212801218033,
+            -0.002097337506711483,
+            -0.0017946474254131317,
+            -0.013504632748663425,
+            -0.026292838156223297,
+            -0.021120242774486542,
+            -0.006010815035551786,
+            -0.00208300887607038,
+            0.010316535830497742,
+            0.008704577572643757,
+            -0.02034650184214115,
+            0.0009967278456315398,
+            -0.039546720683574677,
+            0.009170254692435265,
+            -0.027023592963814735,
+            -0.009843694977462292,
+            -0.0035122791305184364,
+            -0.004481245297938585,
+            -0.022567423060536385,
+            0.022366823628544807,
+            0.019930975511670113,
+            -0.032181862741708755,
+            0.002450177213177085,
+            0.008847862482070923,
+            0.0079451659694314,
+            0.01339716836810112,
+            2.5690593247418292e-05,
+            -0.02412923239171505,
+            -0.014744049869477749,
+            -0.0006953810807317495,
+            0.01029504369944334,
+            -0.008740399032831192,
+            -0.026421794667840004,
+            0.0019253952195867896,
+            0.03788461163640022,
+            -0.007017393596470356,
+            0.013461646623909473,
+            -0.00047776667634025216,
+            -0.00869741290807724,
+            -0.0222378671169281,
+            0.004080046899616718,
+            0.019701717421412468,
+            0.025576412677764893,
+            -0.009378017857670784,
+            0.004954086616635323,
+            -0.0029928700532764196,
+            0.006343953311443329,
+            -0.00632246071472764,
+            0.0006474701222032309,
+            0.01826886646449566,
+            -0.029545413330197334,
+            0.004739158786833286,
+            -0.008031136356294155,
+            -0.011978644877672195,
+            0.010832362808287144,
+            -0.0004674680531024933,
+            0.014256879687309265,
+            -0.016506457701325417,
+            -0.0018573347479104996,
+            0.030003925785422325,
+            0.00407646456733346,
+            -0.025017600506544113,
+            0.012752384878695011,
+            -0.02556208334863186,
+            -0.007192918099462986,
+            -0.0103093720972538,
+            0.023685047402977943,
+            0.01331119704991579,
+            0.00925622507929802,
+            0.027152549475431442,
+            -0.0006125443032942712,
+            0.01792498119175434,
+            -0.004878861829638481,
+            -0.0033277992624789476,
+            0.0016853924607858062,
+            -0.03209589049220085,
+            0.0047785621136426926,
+            0.029889296740293503,
+            -0.03395859897136688,
+            -0.004402438644319773,
+            0.010524299927055836,
+            0.02359907701611519,
+            -0.016993626952171326,
+            0.008281885646283627,
+            0.014092101715505123,
+            -0.006326042581349611,
+            0.00897681899368763,
+            0.025089243426918983,
+            0.049462057650089264,
+            -0.00248599867336452,
+            0.006175593007355928,
+            0.009578617289662361,
+            0.013626424595713615,
+            -0.0007370233652181923,
+            0.008812041021883488,
+            -0.05089491233229637,
+            0.01258760690689087,
+            -0.006394103169441223,
+            -0.022682050243020058,
+            -0.0053624496795237064,
+            -0.016248544678092003,
+            0.017093926668167114,
+            0.026135224848985672,
+            0.02352743409574032,
+            0.01518823392689228,
+            0.013339853845536709,
+            0.02841345965862274,
+            0.012007301673293114,
+            0.0052728960290551186,
+            0.004689008928835392,
+            -0.003426307812333107,
+            0.0022065923549234867,
+            -0.004126614425331354,
+            0.009557124227285385,
+            -0.005287224892526865,
+            0.02819853276014328,
+            0.007723073475062847,
+            0.043472737073898315,
+            -0.03401591256260872,
+            -0.006397685036063194,
+            0.015790032222867012,
+            -0.016735713928937912,
+            -0.0017821099609136581,
+            0.007687252014875412,
+            0.02027485892176628,
+            -0.014543450437486172,
+            0.011985808610916138,
+            -0.0033994419500231743,
+            0.010323700495064259,
+            -0.0042698998004198074,
+            0.01339716836810112,
+            -0.002792270854115486,
+            0.016004959121346474,
+            0.01824020966887474,
+            0.006576791871339083,
+            0.005738573148846626,
+            0.006017979234457016,
+            0.00019522612274158746,
+            -0.022366823628544807,
+            -0.011892673559486866,
+            0.00208300887607038,
+            0.004230496007949114,
+            -0.00625081779435277,
+            0.004456170368939638,
+            0.014622257091104984,
+            -0.02034650184214115,
+            0.016578100621700287,
+            0.019515447318553925,
+            -0.004753487184643745,
+            -0.00034903385676443577,
+            0.0015304653206840158,
+            -0.011641924269497395,
+            -0.015245547518134117,
+            -0.022180551663041115,
+            -0.011161918751895428,
+            -0.012859849259257317,
+            -0.005681259091943502,
+            0.01654944382607937,
+            0.00994399469345808,
+            -0.014729720540344715,
+            0.019271861761808395,
+            -0.027768677100539207,
+            0.019071262329816818,
+            0.004742741119116545,
+            -0.0055308095179498196,
+            -0.014593600295484066,
+            -0.00994399469345808,
+            0.04416050761938095,
+            0.00787352304905653,
+            0.002009575255215168,
+            -0.010882512666285038,
+            -0.004201839212328196,
+            -0.018741706386208534,
+            -0.002480625407770276,
+            -0.0090484619140625,
+            0.024774014949798584,
+            0.01630585826933384,
+            0.008074122481048107,
+            -0.020160231739282608,
+            0.008281885646283627,
+            0.005462748929858208,
+            0.016176901757717133,
+            0.012394172139465809,
+            0.01819722354412079,
+            -0.010402507148683071,
+            -0.021765025332570076,
+            0.0010424000211060047,
+            -0.007880687713623047,
+            0.017767367884516716,
+            0.008983983658254147,
+            0.0038221334107220173,
+            0.006182757206261158,
+            -0.014299864880740643,
+            0.012773877941071987,
+            0.027969276532530785,
+            -0.0004370199458207935,
+            0.021750696003437042,
+            -0.00995832309126854,
+            -0.00834636390209198,
+            0.006852615624666214,
+            0.017409155145287514,
+            0.018354836851358414,
+            -0.03137946501374245,
+            -0.009471152909100056,
+            -0.008862190879881382,
+            -0.011075947433710098,
+            -0.011706402525305748,
+            0.0003340336843393743,
+            -0.019973959773778915,
+            -0.0126950703561306,
+            -0.0104884784668684,
+            -0.01749512553215027,
+            0.005401853006333113,
+            -0.009041297249495983,
+            0.004320049658417702,
+            -0.013089105486869812,
+            -0.034388452768325806,
+            -0.01504494808614254,
+            0.020074259489774704,
+            -0.002005993155762553,
+            -0.009793545119464397,
+            0.008145765401422977,
+            -0.0016997209750115871,
+            0.010373850353062153,
+            0.007629937957972288,
+            -0.013812695629894733,
+            -0.014350014738738537,
+            0.00013623290578834713,
+            -0.004474081099033356,
+            0.010538628324866295,
+            -0.010595941916108131,
+            -0.004581545013934374,
+            -0.012974477373063564,
+            -0.004277063999325037,
+            -0.017452139407396317,
+            -0.015790032222867012,
+            0.006730823311954737,
+            -0.018956635147333145,
+            0.01360493153333664,
+            -0.023728033527731895,
+            -0.012387007474899292,
+            0.020074259489774704,
+            0.002625701716169715,
+            -0.016850342974066734,
+            -0.01869872212409973,
+            -0.008045465685427189,
+            -0.019615747034549713,
+            -0.009456824511289597,
+            0.003207797883078456,
+            0.02686597965657711,
+            0.005838872864842415,
+            0.014314194209873676,
+            -0.0017068851739168167,
+            0.032038576900959015,
+            0.006286639254540205,
+            0.029889296740293503,
+            -0.005863947793841362,
+            0.009363689459860325,
+            -0.02458774484694004,
+            -0.003499741666018963,
+            -0.009112940169870853,
+            0.0036340714432299137,
+            0.00044642301509156823,
+            -0.006852615624666214,
+            -0.0012734474148601294,
+            -0.0023283849004656076,
+            -0.007400681730359793,
+            -0.009585781022906303,
+            0.015575103461742401,
+            0.006591120269149542,
+            0.005129611119627953,
+            0.004183928482234478,
+            0.03332814201712608,
+            -0.006684255786240101,
+            -0.013655081391334534,
+            0.01366224605590105,
+            0.023470120504498482,
+            -0.016893327236175537,
+            -0.000970757391769439,
+            0.009084283374249935,
+            0.011290875263512135,
+            -0.023871317505836487,
+            0.01940082013607025,
+            -0.004696173127740622,
+            -0.004871697630733252,
+            0.008074122481048107,
+            0.002426893450319767,
+            0.01705094240605831,
+            -0.0029212275985628366,
+            -0.0019988289568573236,
+            -0.009936830028891563,
+            0.003943925723433495,
+            -0.015102262608706951,
+            -0.0007929941639304161,
+            0.01255895011126995,
+            -0.011405503377318382,
+            0.008088450878858566,
+            -0.027825990691781044,
+            -0.029831983149051666,
+            -0.01705094240605831,
+            0.01050280686467886,
+            0.002369579393416643,
+            -0.022395480424165726,
+            0.018956635147333145,
+            -0.0038113868795335293,
+            -0.017280198633670807,
+            0.018598422408103943,
+            -0.006501567084342241,
+            -0.003922433126717806,
+            -0.0005767230177298188,
+            0.020633071660995483,
+            0.005842455197125673,
+            -0.005623945035040379,
+            -0.002880033105611801,
+            -0.0069529153406620026,
+            -0.00317376758903265,
+            -0.002838838379830122,
+            0.001708676340058446,
+            -0.0206187441945076,
+            0.00434154225513339,
+            0.01268074195832014,
+            0.01513091940432787,
+            -0.0007625460275448859,
+            0.02829883247613907,
+            -0.005086625460535288,
+            0.00397974718362093,
+            0.006852615624666214,
+            -0.0393461212515831,
+            5.3284191380953416e-05,
+            -0.018569765612483025,
+            -0.008582784794270992,
+            0.010044294409453869,
+            0.009478317573666573,
+            0.001631660503335297,
+            -0.008016807958483696,
+            0.0028065992519259453,
+            -0.009213239885866642,
+            -2.209446938650217e-05,
+            0.007286053616553545,
+            0.022080253809690475,
+            0.013196568936109543,
+            -0.004434677772223949,
+            0.02991795539855957,
+            -0.011434161104261875,
+            0.007443667389452457,
+            0.012322529219090939,
+            0.008898012340068817,
+            -0.0026382391806691885,
+            0.02076202817261219,
+            -0.009198911488056183,
+            -0.009772052057087421,
+            -0.01241566427052021,
+            0.014163744635879993,
+            0.01571838930249214,
+            -0.025261184200644493,
+            -0.0024770433083176613,
+            -0.003428098978474736,
+            0.005262149963527918,
+            -0.002656149910762906,
+            0.012974477373063564,
+            -0.01882767863571644,
+            0.025160884484648705,
+            0.031150206923484802,
+            -0.008704577572643757,
+            -0.011584610678255558,
+            0.0019110667053610086,
+            0.00267406040802598,
+            0.01268074195832014,
+            0.0033385457936674356,
+            0.006519477814435959,
+            -0.003358247457072139,
+            -0.02626418136060238,
+            -0.021836668252944946,
+            0.008403678424656391,
+            -0.01436434406787157,
+            -0.004072882700711489,
+            0.00931353960186243,
+            0.016076602041721344,
+            0.0010352357057854533,
+            0.0060358899645507336,
+            0.022452794015407562,
+            -0.02514655701816082,
+            0.0036824301350861788,
+            -0.01083952747285366,
+            -0.0023892810568213463,
+            -0.006017979234457016,
+            -0.034531738609075546,
+            0.011713567189872265,
+            0.006168428808450699,
+            -0.017667068168520927,
+            0.0023767435923218727,
+            -0.014063444919884205,
+            0.016463471576571465,
+            -0.0007370233652181923,
+            0.03160871937870979,
+            0.008195914328098297,
+            -0.006813212297856808,
+            0.00489319022744894,
+            0.02332683466374874,
+            0.04083628952503204,
+            0.010037129744887352,
+            0.004835876170545816,
+            -0.019300520420074463,
+            -0.00960727408528328,
+            -0.010746391490101814,
+            -0.007708745077252388,
+            -0.01942947693169117,
+            0.009528467431664467,
+            -0.012816863134503365,
+            -0.002960630925372243,
+            0.01213625818490982,
+            -0.0010952363954856992,
+            -0.020991284400224686,
+            0.014729720540344715,
+            0.014271208085119724,
+            -0.0316946916282177,
+            -0.013948816806077957,
+            -0.031236179172992706,
+            0.03054841049015522,
+            -0.020919643342494965,
+            -0.0257053691893816,
+            0.024501772597432137,
+            -0.007572623901069164,
+            0.018498122692108154,
+            0.010595941916108131,
+            0.005842455197125673,
+            -0.0002697792078834027,
+            0.004731994587928057,
+            0.007175007369369268,
+            -0.025576412677764893,
+            0.028786001726984978,
+            0.006759480573236942,
+            -0.016420487314462662,
+            0.008554127998650074,
+            -0.025347156450152397,
+            -0.024114903062582016,
+            0.02170771174132824,
+            0.019773360341787338,
+            0.00048358761705458164,
+            0.01078221295028925,
+            -0.0007495608297176659,
+            -0.0009940412128344178,
+            -0.013970308937132359,
+            -0.008940997533500195,
+            -0.000573140918277204,
+            0.02132084034383297,
+            -0.017036613076925278,
+            0.008446663618087769,
+            0.01255895011126995,
+            8.630695811007172e-05,
+            -0.02805524691939354,
+            -0.00924189668148756,
+            -0.004491991829127073,
+            -0.028800329193472862,
+            0.0076227737590670586,
+            -0.00469975546002388,
+            0.003743326524272561,
+            -0.02381400391459465,
+            0.0019468879327178001,
+            -0.04201122745871544,
+            -0.03089229390025139,
+            0.00140509067568928,
+            -0.029101228341460228,
+            -0.012250886298716068,
+            -0.02371370419859886,
+            0.01131236832588911,
+            -0.01503061968833208,
+            -0.0006752316257916391,
+            0.007665759418159723,
+            -0.01725153997540474,
+            0.006583956070244312,
+            -0.019859332591295242,
+            0.004363034851849079,
+            -0.011441324837505817,
+            0.012673578225076199,
+            0.017595425248146057,
+            0.020174559205770493,
+            0.005333792418241501,
+            -0.002863913308829069,
+            0.03103557974100113,
+            -0.008403678424656391,
+            -0.016363171860575676,
+            -0.006476492155343294,
+            0.017910653725266457,
+            -0.00014597181871067733,
+            0.03576399013400078,
+            0.0016558398492634296,
+            -0.0008068748866207898,
+            0.0036000411491841078,
+            0.007175007369369268,
+            0.0047785621136426926,
+            -0.006831123027950525,
+            0.0269662793725729,
+            0.010352357290685177,
+            0.012752384878695011,
+            -0.006150518078356981,
+            -0.009657423943281174,
+            -0.006759480573236942,
+            0.014077773317694664,
+            0.016463471576571465,
+            0.012107601389288902,
+            0.010409671813249588,
+            0.00489319022744894,
+            -0.017065269872546196,
+            0.01894230581820011,
+            -0.02422953210771084,
+            0.011369682848453522,
+            -0.020890984684228897,
+            0.007343367673456669,
+            -0.0007303068996407092,
+            0.00834636390209198,
+            -0.0020167394541203976,
+            -0.005745737347751856,
+            0.00823890045285225,
+            0.012236557900905609,
+            -0.01230820082128048,
+            -0.0014015085762366652,
+            -0.0253901407122612,
+            -0.01201446633785963,
+            0.0067057483829557896,
+            0.018297523260116577,
+            0.024243859574198723,
+            -0.03089229390025139,
+            0.0038078047800809145,
+            -0.000754934037104249,
+            -0.020719043910503387,
+            0.0035212342627346516,
+            0.013282540254294872,
+            -0.015546446666121483,
+            -0.0009528467198833823,
+            0.023054592311382294,
+            -0.002648985479027033,
+            0.009198911488056183,
+            0.013182240538299084,
+            -0.009299211204051971,
+            0.015317190438508987,
+            0.013741052709519863,
+            0.01729452610015869,
+            -0.009277718141674995,
+            0.016176901757717133,
+            -0.018541108816862106,
+            0.008647263050079346,
+            0.02059008739888668,
+            0.020461129024624825,
+            -0.02069038711488247,
+            0.026636723428964615,
+            0.018254537135362625,
+            -0.020174559205770493,
+            -0.0016925566596910357,
+            -0.007773223333060741,
+            0.011842523701488972,
+            0.014235386624932289,
+            -0.004312884993851185,
+            -0.025089243426918983,
+            -0.01346881128847599,
+            0.010280714370310307,
+            -0.010509971529245377,
+            -0.004739158786833286,
+            -0.015259875915944576,
+            0.010366685688495636,
+            0.0007634415524080396,
+            0.020676057785749435,
+            0.03258306160569191,
+            0.014335686340928078,
+            -0.006662762723863125,
+            0.02599193900823593,
+            0.03215320408344269,
+            0.00541976373642683,
+            -0.005946336779743433,
+            0.010022801347076893,
+            0.0010567285353317857,
+            -0.009213239885866642,
+            -0.003437054343521595,
+            -0.001859125797636807,
+            -0.0142927011474967,
+            -0.0064263422973454,
+            0.014930319972336292,
+            0.03100692294538021,
+            -0.005222746171057224,
+            -0.020575758069753647,
+            0.00858994945883751,
+            0.00912010483443737,
+            0.014564942568540573,
+            0.004997072275727987,
+            0.018999621272087097,
+            0.015489133074879646,
+            0.016219887882471085,
+            0.005835290532559156,
+            -0.023685047402977943,
+            -0.008031136356294155,
+            0.009750558994710445,
+            -0.02114889957010746,
+            -0.006451417226344347,
+            -0.004843040369451046,
+            -0.039747320115566254,
+            -0.021550098434090614,
+            0.014550614170730114,
+            0.026436123996973038,
+            0.01167774572968483,
+            -0.0009832947980612516,
+            -0.015073605813086033,
+            0.0029140631668269634,
+            0.0038113868795335293,
+            0.01346881128847599,
+            0.012601935304701328,
+            0.0013746425975114107,
+            0.00011899389937752858,
+            0.02301160618662834,
+            0.0028674956411123276,
+            -0.012580442242324352,
+            0.005785140674561262,
+            -0.008181585930287838,
+            -0.007923672907054424,
+            0.003062721574679017,
+            -0.005161850247532129,
+            -0.004452588502317667,
+            0.010775049217045307,
+            0.006096786353737116,
+            -0.003505114698782563,
+            -0.010194743983447552,
+            0.0011641924502328038,
+            0.010638928040862083,
+            0.002199428156018257,
+            0.0020453964825719595,
+            0.008847862482070923,
+            -0.011534460820257664,
+            -0.01043116394430399,
+            -0.015503461472690105,
+            0.007694416679441929,
+            -0.015016291290521622,
+            0.006902765482664108,
+            0.022108910605311394,
+            -0.004488409496843815,
+            -0.026708366349339485,
+            -0.008253228850662708,
+            0.008203078992664814,
+            -0.014507628977298737,
+            0.005745737347751856,
+            0.018096923828125,
+            -0.0009582199272699654,
+            0.009850858710706234,
+            -0.01346881128847599,
+            0.015489133074879646,
+            0.013253883458673954,
+            0.02139248326420784,
+            -0.0004813488048966974,
+            -0.007758894935250282,
+            0.011706402525305748,
+            0.0036681017372757196,
+            -0.017122583463788033,
+            0.009714738465845585,
+            0.03427382558584213,
+            -0.001524196588434279,
+            0.005964247509837151,
+            0.0016943477094173431,
+            0.0043558706529438496,
+            -0.001223297556862235,
+            -0.0010379223385825753,
+            0.004724830389022827,
+            0.012816863134503365,
+            0.005677676759660244,
+            -0.0038758653681725264,
+            0.009721902199089527,
+            0.008804877288639545,
+            -0.004735576454550028,
+            -0.02703792229294777,
+            -0.004499156028032303,
+            -0.0008422484388574958,
+            -0.0067773908376693726,
+            0.007386353332549334,
+            -0.016721386462450027,
+            0.007099783048033714,
+            -0.006043054163455963,
+            0.014350014738738537,
+            -0.0033134708646684885,
+            0.009736230596899986,
+            0.019128577783703804,
+            0.01422105822712183,
+            0.01237984374165535,
+            -0.007286053616553545,
+            0.010080115869641304,
+            0.007952329702675343,
+            0.009793545119464397,
+            0.0027403298299759626,
+            -0.006071711424738169,
+            0.0021421140991151333,
+            -0.004835876170545816,
+            0.0064263422973454,
+            0.0034191436134278774,
+            0.015603761188685894,
+            -0.01558943185955286,
+            -0.017308855429291725,
+            -0.000402318051783368,
+            0.032840974628925323,
+            0.001868081046268344,
+            0.0019182309042662382,
+            -0.009234732948243618,
+            0.010266385972499847,
+            0.000753590720705688,
+            -0.012859849259257317,
+            -0.007257396355271339,
+            0.026350151747465134,
+            -0.0021295766346156597,
+            0.0017749457620084286,
+            -0.0034728755708783865,
+            -0.0016692728968337178,
+            0.012745220214128494,
+            0.012365514412522316,
+            -0.011684910394251347,
+            0.018053937703371048,
+            -0.004585127346217632,
+            0.02293996512889862,
+            0.013225225731730461,
+            0.0018376329680904746,
+            -0.0005776185425929725,
+            -0.008038301020860672,
+            0.0074579957872629166,
+            -0.02339847758412361,
+            0.002009575255215168,
+            -0.0090484619140625,
+            -0.010445492342114449,
+            -0.0055308095179498196,
+            -0.025791339576244354,
+            0.00451706675812602,
+            0.002878241939470172,
+            -0.011684910394251347,
+            0.0016979299252852798,
+            0.005645437631756067,
+            0.007637102622538805,
+            0.013425825163722038,
+            -0.01420672982931137,
+            -0.005749319680035114,
+            0.010724899359047413,
+            0.0009832947980612516,
+            0.005520063452422619,
+            -0.020776357501745224,
+            0.01676437072455883,
+            -0.0065947026014328,
+            -0.02689463645219803,
+            0.007063961587846279,
+            -0.00406930036842823,
+            0.01662108674645424,
+            0.00806695781648159,
+            0.002824509982019663,
+            -0.020532771944999695,
+            0.018713049590587616,
+            -0.004284228198230267,
+            -0.012494471855461597,
+            -0.019343504682183266,
+            -0.0030179449822753668,
+            -0.007973822765052319,
+            0.018713049590587616,
+            -0.007651431020349264,
+            -0.0336720272898674,
+            0.004660351667553186,
+            -0.022223537787795067,
+            -0.017480798065662384,
+            -0.022223537787795067,
+            0.010323700495064259,
+            -0.005627527367323637,
+            0.011004305444657803,
+            -0.009714738465845585,
+            0.007551131304353476,
+            -0.029402127489447594,
+            0.019372161477804184,
+            0.007020975928753614,
+            0.017896324396133423,
+            0.019687389954924583,
+            0.0273244921118021,
+            -0.011649088934063911,
+            0.0048609510995447636,
+            -0.022710707038640976,
+            -1.0858333553187549e-05,
+            -0.019372161477804184,
+            -0.007472324185073376,
+            0.003976164851337671,
+            0.017738711088895798,
+            0.00680246576666832,
+            0.003290186868980527,
+            -0.016076602041721344,
+            -0.022037267684936523,
+            -0.03215320408344269,
+            -0.027123892679810524,
+            -0.004230496007949114,
+            0.006139772012829781,
+            0.013741052709519863,
+            -0.002645403379574418,
+            -0.015904659405350685,
+            3.201529034413397e-05,
+            0.012759549543261528,
+            -0.01960141770541668,
+            -0.002000619890168309,
+            -0.03278365731239319,
+            -0.010072951205074787,
+            -0.029545413330197334,
+            0.018211551010608673,
+            0.021936967968940735,
+            0.004760651383548975,
+            -0.011534460820257664,
+            -0.012967312708497047,
+            -0.018383493646979332,
+            0.0016504667000845075,
+            -0.006361864041537046,
+            -0.013003134168684483,
+            -0.03490427881479263,
+            -0.012143422849476337,
+            0.011935658752918243,
+            0.010087279602885246,
+            -0.006394103169441223,
+            -0.0018331552855670452,
+            0.011462817899882793,
+            -0.005681259091943502,
+            -0.005237075034528971,
+            -0.0036967587657272816,
+            -0.02146412618458271,
+            -0.015302862040698528,
+            0.006644852459430695,
+            -0.006390520837157965,
+            0.01257327850908041,
+            -0.013898666948080063,
+            0.0016003168420866132,
+            -0.008675919845700264,
+            0.006755898240953684,
+            0.00275824056006968,
+            -0.010072951205074787,
+            -0.010617434978485107,
+            0.016707057133316994,
+            0.010825199075043201,
+            -0.016878999769687653,
+            -0.003650191007182002,
+            0.01023772917687893,
+            0.00939951092004776,
+            -0.009700410068035126,
+            -0.028313159942626953,
+            -0.013432989828288555,
+            0.014242551289498806,
+            -0.01103296224027872,
+            0.01887066476047039,
+            -0.012093272991478443,
+            -0.00831054337322712,
+            0.020432472229003906,
+            0.008246064186096191,
+            -0.003408397315070033,
+            0.0007831432740204036,
+            -0.004123032093048096,
+            -0.003528398694470525,
+            -0.006956497672945261,
+            0.015016291290521622,
+            -0.015102262608706951,
+            0.008940997533500195,
+            -0.007096200715750456,
+            0.014314194209873676,
+            -0.00787352304905653,
+            0.0034782488364726305,
+            0.023341162130236626,
+            -0.006723659113049507,
+            -0.013411496765911579,
+            0.008389350026845932,
+            0.01448613591492176,
+            0.02122054249048233,
+            0.00913443323224783,
+            -0.003068094840273261,
+            -0.01296014804393053,
+            -0.014271208085119724,
+            -0.00896249059587717,
+            0.025203870609402657,
+            -0.01518823392689228,
+            -0.006652016658335924,
+            0.009456824511289597,
+            -0.0005064237047918141,
+            -0.011971480213105679,
+            -0.002588089322671294,
+            -0.025719698518514633,
+            -0.005351703148335218,
+            0.02286832220852375,
+            -0.03258306160569191,
+            -0.005910515319555998,
+            0.017065269872546196,
+            -0.01402762345969677,
+            -0.017423482611775398,
+            0.010982812382280827,
+            -0.015159576199948788,
+            0.009220404550433159,
+            -0.008038301020860672,
+            0.01430702954530716,
+            0.000650156696792692,
+            -0.02367071807384491,
+            0.005143939517438412,
+            0.021206213161349297,
+            -0.02437281608581543,
+            0.004577962681651115,
+            -0.0443897619843483,
+            0.002100919606164098,
+            0.003943925723433495,
+            0.014607928693294525,
+            -0.00925622507929802,
+            0.0002228756929980591,
+            -0.004044225439429283,
+            -0.012279544025659561,
+            0.017552439123392105,
+            -0.008948162198066711,
+            -0.02230951003730297,
+            0.002145696198567748,
+            0.005176178645342588,
+            0.007038886658847332,
+            -0.009743395261466503,
+            0.013239555060863495,
+            -0.0019146488048136234,
+            -0.010037129744887352,
+            0.016262872144579887,
+            -0.0007576206116937101,
+            -0.020604414865374565,
+            -0.02013157308101654,
+            -0.00244122208096087,
+            -0.024358488619327545,
+            0.012071779929101467,
+            -0.004760651383548975,
+            -0.0015743464464321733,
+            -0.005480659659951925,
+            0.016535114496946335,
+            0.00027179415337741375,
+            0.017867667600512505,
+            -0.0027295835316181183,
+            0.012200736440718174,
+            0.00022567423002328724,
+            0.00352481659501791,
+            -0.02325519174337387,
+            0.0005046326550655067,
+            -0.003368993755429983,
+            -0.012472978793084621,
+            -0.016334515064954758,
+            -0.01957276090979576,
+            -0.018010953441262245,
+            -0.010058622807264328,
+            0.016606757417321205,
+            0.025619398802518845,
+            0.006863362155854702,
+            -0.020432472229003906,
+            0.013755381107330322,
+            -0.004667516332119703,
+            0.00959294568747282,
+            0.0013459855690598488,
+            -0.009478317573666573,
+            0.029688699170947075,
+            0.013110597617924213,
+            0.018741706386208534,
+            -0.003703922964632511,
+            -0.028169874101877213,
+            -0.01819722354412079,
+            -0.0177816953510046,
+            0.010810869745910168,
+            -0.010825199075043201,
+            0.010338028892874718,
+            -0.011727895587682724,
+            -0.00787352304905653,
+            -0.001703303074464202,
+            -0.021234869956970215,
+            0.032181862741708755,
+            -0.024988943710923195,
+            0.0046102022752165794,
+            0.017409155145287514,
+            -0.0015886749606579542,
+            -0.010323700495064259,
+            0.023556090891361237,
+            0.002005993155762553,
+            -0.022925635799765587,
+            -0.00966458860784769,
+            0.010037129744887352,
+            -0.012988805770874023,
+            0.005176178645342588,
+            -1.1285110304015689e-05,
+            0.007171425502747297,
+            -0.028327489271759987,
+            0.012387007474899292,
+            0.018053937703371048,
+            0.013490303419530392,
+            0.02276802249252796,
+            0.0023122653365135193,
+            0.013003134168684483,
+            -0.012286707758903503,
+            0.006386938970535994,
+            0.01988798938691616,
+            0.0079451659694314,
+            -0.028040917590260506,
+            0.020676057785749435,
+            -0.021693382412195206,
+            0.018598422408103943,
+            -0.0042698998004198074,
+            -0.0031003339681774378,
+            0.03387262672185898,
+            -0.010667584836483002,
+            -0.004456170368939638,
+            -0.01069624163210392,
+            -0.004989908076822758,
+            -0.003675265936180949,
+            -0.011634760536253452,
+            0.02633582428097725,
+            0.024716701358556747,
+            -0.021019943058490753,
+            0.01211476605385542,
+            0.002511073602363467,
+            0.012243722565472126,
+            -0.00697082607075572,
+            0.023441461846232414,
+            0.014507628977298737,
+            -0.007314710877835751,
+            0.009019805118441582,
+            0.004051389638334513,
+            -0.003911686595529318,
+            0.0006891123484820127,
+            -0.013447318226099014,
+            -0.007386353332549334,
+            -0.006053800694644451,
+            0.011555952951312065,
+            0.008554127998650074,
+            -0.008016807958483696,
+            0.003718251595273614,
+            0.0003978403692599386,
+            0.0012931491946801543,
+            -0.01002996601164341,
+            0.0074579957872629166,
+            0.0035122791305184364,
+            0.00869741290807724,
+            0.011212068609893322,
+            0.0033672028221189976,
+            -0.009213239885866642,
+            -0.00451706675812602,
+            0.01824020966887474,
+            0.016191231086850166,
+            0.00976488832384348,
+            0.014958977699279785,
+            -0.01652078703045845,
+            0.0253901407122612,
+            0.024530429393053055,
+            0.0007558295619674027,
+            -0.01176371704787016,
+            0.0053552850149571896,
+            0.0028137636836618185,
+            -0.008632934652268887,
+            0.011054455302655697,
+            -0.006017979234457016,
+            0.015216890722513199,
+            0.003354665357619524,
+            -0.00806695781648159,
+            -0.006161264609545469,
+            -0.027095235884189606,
+            -0.037540730088949203,
+            0.002767195925116539,
+            -0.004599455744028091,
+            -0.006723659113049507,
+            -0.0026579408440738916,
+            -0.009986979886889458,
+            0.0014865841949358582,
+            0.006261564325541258,
+            -0.006010815035551786,
+            0.01202879473567009,
+            -0.023226534947752953,
+            -0.005423345603048801,
+            0.000444184202933684,
+            -0.010459821671247482,
+            -2.4809052774799056e-05,
+            -0.007046050857752562,
+            -0.010051458142697811,
+            0.005906933452934027,
+            -0.022782349959015846,
+            -0.000383287959266454,
+            -0.01763841137290001,
+            0.01889932155609131,
+            -0.008632934652268887,
+            0.0045206486247479916,
+            0.0016504667000845075,
+            -0.013074777089059353,
+            0.01504494808614254,
+            0.011491474695503712,
+            -0.010509971529245377,
+            0.002151069464161992,
+            0.008876519277691841,
+            0.0025558501947671175,
+            0.0011767299147322774,
+            -0.021793682128190994,
+            -0.01139833964407444,
+            0.02289697900414467,
+            0.002111665904521942,
+            -0.017423482611775398,
+            -9.363912977278233e-05,
+            0.009686080738902092,
+            0.003426307812333107,
+            0.016019288450479507,
+            0.00607529329136014,
+            0.013741052709519863,
+            0.014256879687309265,
+            -0.0008301587658934295,
+            0.020160231739282608,
+            -0.0009206075337715447,
+            -0.002340922364965081,
+            0.0024251025170087814,
+            0.006770226638764143,
+            0.0007200082764029503,
+            0.004807219374924898,
+            -0.010015636682510376,
+            -0.010739227756857872,
+            -0.0019021113403141499,
+            0.0014695690479129553,
+            0.00425198907032609,
+            0.016076602041721344,
+            0.0008444872801192105,
+            -0.01383418869227171,
+            0.014077773317694664,
+            0.04267033934593201,
+            0.0034191436134278774,
+            0.021062927320599556,
+            0.0010970274452120066,
+            -0.0005614989786408842,
+            0.009077118709683418,
+            0.00858994945883751,
+            -0.026636723428964615,
+            0.021621739491820335,
+            -0.006809630431234837,
+            0.00551648112013936,
+            -0.030175868421792984,
+            0.006286639254540205,
+            -0.017208555713295937,
+            0.002500327071174979,
+            -0.01166341733187437,
+            0.006394103169441223,
+            0.012315364554524422,
+            -0.006616195198148489,
+            0.017523782327771187,
+            -0.0008525470620952547,
+            0.016047945246100426,
+            0.0012421037536114454,
+            -0.0021188303362578154,
+            -0.00931353960186243,
+            0.012480142526328564,
+            0.018913649022579193,
+            0.018440809100866318,
+            0.0005261254264041781,
+            0.01222222950309515,
+            -0.005262149963527918,
+            -0.010015636682510376,
+            0.009284882806241512,
+            -0.004047807771712542,
+            -0.0005579168209806085,
+            0.0037934763822704554,
+            -0.011197740212082863,
+            -0.0033564562909305096,
+            0.01676437072455883,
+            0.019515447318553925,
+            0.0028657044749706984,
+            -0.011577446013689041,
+            -0.02037515863776207,
+            0.0006282161339186132,
+            0.014350014738738537,
+            -0.003111080499365926,
+            0.013683739118278027,
+            -0.0036716838367283344,
+            -0.0029695862904191017,
+            -0.008647263050079346,
+            0.0004466469108592719,
+            0.020074259489774704,
+            0.0051546860486269,
+            -0.014034787192940712,
+            -0.004534977488219738,
+            0.021936967968940735,
+            0.010610271245241165,
+            0.008145765401422977,
+            0.006576791871339083,
+            -0.007529638707637787,
+            0.01659242808818817,
+            -0.011169083416461945,
+            -0.00037948196404613554,
+            -0.012666413560509682,
+            -0.003236454911530018,
+            -0.005856783594936132,
+            -0.008496813476085663,
+            -0.007350531872361898,
+            -0.008568456396460533,
+            0.005487823858857155,
+            0.011161918751895428,
+            -0.01596197299659252,
+            -0.012236557900905609,
+            0.012250886298716068,
+            -0.002503909170627594,
+            -0.0004728412313852459,
+            -0.01620555855333805,
+            0.014156579971313477,
+            0.008059794083237648,
+            0.0033851133193820715,
+            -0.009779216721653938,
+            0.01558943185955286,
+            -0.005817380268126726,
+            0.0002970929490402341,
+            -0.0060323080979287624,
+            -0.0004099300713278353,
+            0.0014982260763645172,
+            -0.004090792965143919,
+            -0.005548720248043537,
+            -0.0018967381911352277,
+            0.007701580878347158,
+            -0.01952977664768696,
+            0.009686080738902092,
+            -0.010832362808287144,
+            -0.026565080508589745,
+            -0.013827024027705193,
+            0.022223537787795067,
+            0.025805668905377388,
+            0.027410462498664856,
+            0.007701580878347158,
+            0.002815554616972804,
+            0.00562036270275712,
+            0.0034155615139752626,
+            -0.0025845072232186794,
+            0.005376778077334166,
+            0.009026968851685524,
+            -0.0038078047800809145,
+            0.03057706728577614,
+            0.005641855765134096,
+            -0.004545723553746939,
+            0.014328522607684135,
+            0.018369166180491447,
+            0.01831185072660446,
+            -0.007687252014875412,
+            0.020174559205770493,
+            -0.016148244962096214,
+            -0.0008395618642680347,
+            -0.002708090702071786,
+            0.03507622331380844,
+            -0.029373470693826675,
+            -0.0012644920498132706,
+            -0.0027510763611644506,
+            0.02041814476251602,
+            0.006028725765645504,
+            -0.02304026484489441,
+            0.010531463660299778,
+            0.028112560510635376,
+            0.0025791339576244354,
+            0.012895669788122177,
+            -0.02051844447851181,
+            -0.009700410068035126,
+            0.00017686770297586918,
+            0.005308717489242554,
+            0.0029964521527290344,
+            0.005505734588950872,
+            -0.022323837503790855,
+            0.01623421534895897,
+            -0.01457210723310709,
+            0.019873660057783127,
+            0.0037218336947262287,
+            0.026292838156223297,
+            0.005366031546145678,
+            0.006064547225832939,
+            0.006960079539567232,
+            -0.013877173885703087,
+            0.014987634494900703,
+            0.0007853821152821183,
+            -0.013862845487892628,
+            0.002903316868469119,
+            -0.016721386462450027,
+            0.01167774572968483,
+            -0.0168360136449337,
+            0.015947645530104637,
+            -0.004359452985227108,
+            0.007565459702163935,
+            -0.0036089965142309666,
+            -0.0009138910681940615,
+            -0.013755381107330322,
+            0.012666413560509682,
+            -0.030032582581043243,
+            0.010287879034876823,
+            -0.010954155586659908,
+            -0.031092893332242966,
+            0.0037504907231777906,
+            0.002140322932973504,
+            0.01131236832588911,
+            -0.013633589260280132,
+            -0.017982294782996178,
+            0.01630585826933384,
+            -0.0019880824256688356,
+            0.004481245297938585,
+            -0.020819343626499176,
+            -0.007085454184561968,
+            0.003800640581175685,
+            0.024945957586169243,
+            -0.0017928563756868243,
+            -0.014536285772919655,
+            0.007325456943362951,
+            0.014371507801115513,
+            -0.009614438749849796,
+            0.020461129024624825,
+            0.005634691566228867,
+            -0.007701580878347158,
+            0.007253814488649368,
+            -0.0025468948297202587,
+            0.022954292595386505,
+            -0.00470691965892911,
+            -0.008496813476085663,
+            0.013905830681324005,
+            0.008375021629035473,
+            -0.02328384853899479,
+            0.005186925176531076,
+            -0.008726069703698158,
+            -0.0013871800620108843,
+            -0.0007809044909663498,
+            -0.001941514783538878,
+            0.010037129744887352,
+            0.028485102578997612,
+            -0.00834636390209198,
+            0.008948162198066711,
+            0.003689594566822052,
+            -0.00317376758903265,
+            5.6950279940792825e-06,
+            0.0006322460249066353,
+            -0.019830675795674324,
+            -0.00903413351625204,
+            -0.004749905318021774,
+            -0.010395343415439129,
+            0.0007531429291702807,
+            0.016463471576571465,
+            0.027582405135035515,
+            -0.0073720249347388744,
+            0.007167843170464039,
+            -0.0007070230203680694,
+            0.020461129024624825,
+            0.031752005219459534,
+            -0.014328522607684135,
+            0.004588709212839603,
+            -0.009370853193104267,
+            -0.012867012992501259,
+            -0.020561428740620613,
+            0.004259153269231319,
+            -0.0014668825315311551,
+            -0.02304026484489441,
+            -0.0009725484414957464,
+            0.00046657249913550913,
+            -0.005176178645342588,
+            -0.009549960494041443,
+            0.00488244416192174,
+            -0.011885508894920349,
+            0.0045278132893145084,
+            -0.002625701716169715,
+            0.004889608360826969,
+            0.010911169461905956,
+            0.034875623881816864,
+            -0.005946336779743433,
+            -0.002511073602363467,
+            -0.011498639360070229,
+            0.0033224262297153473,
+            0.020160231739282608,
+            0.000652843271382153,
+            0.008195914328098297,
+            -0.0066197775304317474,
+            0.00834636390209198,
+            -0.006841869559139013,
+            -0.012401335872709751,
+            0.01520256232470274,
+            0.008948162198066711,
+            0.003125408897176385,
+            -0.01076788455247879,
+            -0.0501498281955719,
+            -0.0015143457567319274,
+            0.032525744289159775,
+            0.010531463660299778,
+            -0.005720662418752909,
+            -0.012652085162699223,
+            -0.018541108816862106,
+            -0.008754727430641651,
+            -0.0020704714115709066,
+            0.0013638961827382445,
+            -0.023455791175365448,
+            0.0024645058438181877,
+            -0.005305135622620583,
+            -0.003789894049987197,
+            -0.0037289978936314583,
+            0.011441324837505817,
+            0.04390259087085724,
+            -0.015431818552315235,
+            0.00813860073685646,
+            -0.009657423943281174,
+            -0.01527420524507761,
+            -0.0006761271506547928,
+            0.018598422408103943,
+            -0.010216236114501953,
+            -0.012494471855461597,
+            -0.016964970156550407,
+            -0.010674749501049519,
+            -0.006472909823060036,
+            0.009979816153645515,
+            0.009628767147660255,
+            -0.011649088934063911,
+            -0.007751730736345053,
+            -0.004491991829127073,
+            -0.007053215056657791,
+            0.013511796481907368,
+            0.006193503737449646,
+            0.021091584116220474,
+            -0.0161625724285841,
+            -0.01185685209929943,
+            0.012021630071103573,
+            0.008754727430641651,
+            0.0030698860064148903,
+            -0.015360175631940365,
+            -0.005541556049138308,
+            -0.007465159986168146,
+            -0.0023122653365135193,
+            0.025662383064627647,
+            -0.004864533431828022,
+            0.016463471576571465,
+            -0.0003806013846769929,
+            -0.005584541708230972,
+            0.004574380815029144,
+            0.005545137915760279,
+            0.025232527405023575,
+            0.011806702241301537,
+            -0.0026042088866233826,
+            -0.01338283997029066,
+            0.0023283849004656076,
+            -0.010624599643051624,
+            -0.0022191298194229603,
+            -0.009356524795293808,
+            0.006086039822548628,
+            0.01574704609811306,
+            0.0034334722440689802,
+            0.021979954093694687,
+            -0.00030358557705767453,
+            -0.020848000422120094,
+            -0.004330795723944902,
+            0.00831054337322712,
+            0.0021779353264719248,
+            0.017437811940908432,
+            0.009707573801279068,
+            0.013432989828288555,
+            0.0058675301261246204,
+            0.012472978793084621,
+            -0.011298039928078651,
+            -0.004316467326134443,
+            -0.006086039822548628,
+            -0.002656149910762906,
+            0.014163744635879993,
+            6.481417221948504e-05,
+            0.03819984197616577,
+            -0.015073605813086033,
+            -0.019687389954924583,
+            0.002557641128078103,
+            -0.011520132422447205,
+            -0.012422828935086727,
+            0.00462811253964901,
+            -0.012809699401259422,
+            -0.0017874831100925803,
+            0.013167912140488625,
+            -0.020676057785749435,
+            -0.022452794015407562,
+            -0.013175075873732567,
+            0.011262218467891216,
+            -0.005183342844247818,
+            -0.005577377509325743,
+            -0.00020630833751056343,
+            0.010946990922093391,
+            0.0024698791094124317,
+            -0.007303964346647263,
+            0.018397822976112366,
+            -0.007680087815970182,
+            -0.007809044793248177,
+            0.013705231249332428,
+            -0.003943925723433495,
+            0.007114111445844173,
+            0.002369579393416643,
+            0.015933316200971603,
+            0.022925635799765587,
+            -0.004782144445925951,
+            0.0034746667370200157,
+            0.007880687713623047,
+            0.014701063744723797,
+            0.0020597251132130623,
+            0.008453828282654285,
+            -0.023785347118973732,
+            0.01360493153333664,
+            0.007164261303842068,
+            -0.007930837571620941,
+            0.013755381107330322,
+            0.01518823392689228,
+            0.0019182309042662382,
+            0.005459167063236237,
+            -0.005050804000347853,
+            0.01964440383017063,
+            -0.0009895635303109884,
+            0.002396445255726576,
+            -0.01967306062579155,
+            0.012738056480884552,
+            -0.004391692113131285,
+            0.011190575547516346,
+            0.009886680170893669,
+            0.01627720147371292,
+            0.029086900874972343,
+            -0.0010692659998312593,
+            0.00294630229473114,
+            -0.008955326862633228,
+            0.011498639360070229,
+            0.0038615367375314236,
+            -0.009901008568704128,
+            -0.0024286846164613962,
+            0.00641917809844017,
+            0.01792498119175434,
+            0.022753693163394928,
+            0.019314847886562347,
+            0.004255570936948061,
+            -0.0029588397592306137,
+            0.026923293247818947,
+            0.0013603140832856297,
+            -0.00029351082048378885,
+            -0.0027546584606170654,
+            -0.009177418425679207,
+            -0.002872868673875928,
+            -0.014744049869477749,
+            0.011985808610916138,
+            -0.0020848000422120094,
+            0.019730376079678535,
+            -0.000688216823618859,
+            0.02342713437974453,
+            0.010244893841445446,
+            -0.015431818552315235,
+            -0.0068669444881379604,
+            0.0017149450723081827,
+            0.002801226219162345,
+            -0.0171512421220541,
+            -0.0021725622937083244,
+            -0.003159439191222191,
+            0.0017015120247378945,
+            0.010825199075043201,
+            8.04300289019011e-05,
+            9.75011134869419e-05,
+            0.001118520274758339,
+            -0.0015600178157910705,
+            -0.002203010255470872,
+            0.0011821030639111996,
+            -0.0007885164814069867,
+            -0.005598870106041431,
+            0.008790548890829086,
+            0.013139255344867706,
+            0.01436434406787157,
+            0.015059277415275574,
+            0.02391430363059044,
+            0.01644914411008358,
+            0.013905830681324005,
+            -0.0027600317262113094,
+            -0.0025074915029108524,
+            0.024774014949798584,
+            0.003986911382526159,
+            -0.004710501525551081,
+            0.004581545013934374,
+            0.0026668962091207504,
+            0.0019343504682183266,
+            -0.006086039822548628,
+            0.0055308095179498196,
+            -0.016090931370854378,
+            5.056960799265653e-05,
+            -0.0036143697798252106,
+            -0.002208383521065116,
+            -0.010617434978485107,
+            0.0009152343845926225,
+            -0.017208555713295937,
+            0.009127268567681313,
+            0.01571838930249214,
+            0.003442427609115839,
+            0.01058877818286419,
+            0.007708745077252388,
+            -0.030949607491493225,
+            -0.009299211204051971,
+            0.011362518183887005,
+            0.003689594566822052,
+            0.011226397007703781,
+            0.012358350679278374,
+            0.008181585930287838,
+            -0.017910653725266457,
+            0.008246064186096191,
+            -0.002319429535418749,
+            0.011806702241301537,
+            0.010424000211060047,
+            -0.008948162198066711
+          ],
+          "title": "Designing Models with Data, Model, and Expert-Parallelism"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "fa9b8012-51a7-4f3b-8612-1d02446d4c9d",
+      "type": "child",
+      "source": {
+        "id": "65742957-6bda-446d-a61a-0fe20313c760",
+        "properties": {
+          "page_content": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity\n\nWilliam Fedus∗ liamfedus@google.com Barret Zoph∗ barretzoph@google.com Noam Shazeer noam@google.com Google, Mountain View, CA 94043, USA Editor: Alexander Clark\n\nAbstract\n\nIn deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) models defy this and instead select different parameters for each incoming example. The result is a sparsely-activated model—with an outrageous number of parameters—but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs, and training instability. We address these with the introduction of the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques mitigate the instabilities, and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large (Raffel et al., 2019) to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the \"Colossal Clean Crawled Corpus\", and achieve a 4x speedup over the T5-XXL model.12 Keywords: mixture-of-experts, natural language processing, sparsity, large-scale machine learning, distributed computing arXiv:2101.03961v3 [cs.LG] 16 Jun 2022 Contents\n\n1 Introduction 3 2 Switch Transformer 4 2.1 Simplifying Sparse Routing 5 2.2 Efficient Sparse Routing 6 2.3 Putting It All Together: The Switch Transformer 8 2.4 Improved Training and Fine-Tuning Techniques 8 3 Scaling Properties 11 3.1 Scaling Results on a Step-Basis 12 3.2 Scaling Results on a Time-Basis 13 3.3 Scaling Versus a Larger Dense Model 13 4 Downstream Results 14 4.1 Fine-Tuning 14 4.2 Distillation 16 4.3 Multilingual Learning 17 5 Designing Models with Data, Model, and Expert-Parallelism 18 5.1 Data Parallelism 20 5.2 Model Parallelism 20 5.3 Model and Data Parallelism 21 5.4 Expert and Data Parallelism 22 5.5 Expert, Model and Data Parallelism 22 5.6 Towards Trillion Parameter Models 22 6 Related Work 24 7 Discussion 25 8 Future Work 26 9 Conclusion 27 A Switch for Attention 27 B Preventing Token Dropping with No-Token-Left-Behind 29\n\nC Encouraging Exploration Across Experts 29 D Switch Transformers in Lower Compute Regimes 29 E Relation of Upstream to Downstream Model Performance 32 F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/switch_transformers.md"
+          },
+          "summary": "The Switch Transformer is a sparsely-activated model that simplifies the Mixture of Experts (MoE) approach, allowing for efficient scaling to trillion-parameter models with reduced communication and computational costs. It achieves significant speedups in pre-training and fine-tuning by activating only a subset of neural network weights for each input, maintaining a constant computational cost. The model demonstrates superior scaling properties and improved performance across various natural language processing tasks, including multilingual settings. It also allows for effective distillation into smaller dense models, preserving a portion of the quality gains. The architecture is designed to maximize parameter count while keeping floating point operations per example constant, making it suitable for both large-scale and smaller-scale applications.",
+          "headlines": {
+            "1 Introduction": [],
+            "2 Switch Transformer": [
+              "2.1 Simplifying Sparse Routing",
+              "2.2 Efficient Sparse Routing",
+              "2.3 Putting It All Together: The Switch Transformer",
+              "2.4 Improved Training and Fine-Tuning Techniques"
+            ],
+            "3 Scaling Properties": [
+              "3.1 Scaling Results on a Step-Basis",
+              "3.2 Scaling Results on a Time-Basis",
+              "3.3 Scaling Versus a Larger Dense Model"
+            ],
+            "4 Downstream Results": [
+              "4.1 Fine-Tuning",
+              "4.2 Distillation",
+              "4.3 Multilingual Learning"
+            ],
+            "5 Designing Models with Data, Model, and Expert-Parallelism": [
+              "5.1 Data Parallelism",
+              "5.2 Model Parallelism",
+              "5.3 Model and Data Parallelism",
+              "5.4 Expert and Data Parallelism",
+              "5.5 Expert, Model and Data Parallelism",
+              "5.6 Towards Trillion Parameter Models"
+            ],
+            "6 Related Work": [],
+            "7 Discussion": [],
+            "8 Future Work": [],
+            "9 Conclusion": [],
+            "A Switch for Attention": [],
+            "B Preventing Token Dropping with No-Token-Left-Behind": [],
+            "C Encouraging Exploration Across Experts": [],
+            "D Switch Transformers in Lower Compute Regimes": [],
+            "E Relation of Upstream to Downstream Model Performance": [],
+            "F Pseudo Code for Switch Transformers": []
+          },
+          "summary_embedding": [
+            -0.0060447100549936295,
+            -0.00390331051312387,
+            -0.01784370467066765,
+            -0.0021820589900016785,
+            0.013227885589003563,
+            0.011338187381625175,
+            0.0165580902248621,
+            0.024240797385573387,
+            -0.005986624862998724,
+            0.03596621751785278,
+            -0.004852031823247671,
+            -0.0068811336532235146,
+            -0.003574162255972624,
+            0.014242436736822128,
+            0.011578272096812725,
+            -0.033302053809165955,
+            -0.024519605562090874,
+            0.01050176378339529,
+            0.016093410551548004,
+            -0.006795942317694426,
+            0.007473600562661886,
+            -0.030513975769281387,
+            -0.06124480068683624,
+            -0.006838538218289614,
+            -0.0026099516544491053,
+            0.01029265858232975,
+            0.0010329445358365774,
+            -0.010873507708311081,
+            -0.053469158709049225,
+            0.023311438038945198,
+            0.025294071063399315,
+            0.005092116538435221,
+            -0.029584616422653198,
+            0.0009361362317577004,
+            -0.05161044001579285,
+            -0.04448534920811653,
+            0.0379488542675972,
+            0.012561844661831856,
+            -0.04364892467856407,
+            0.039249956607818604,
+            -0.004875265993177891,
+            -0.0022440163884311914,
+            -0.021499186754226685,
+            -0.0034909071400761604,
+            -0.007039899472147226,
+            -0.009851212613284588,
+            -0.02874819189310074,
+            0.0014037203509360552,
+            0.021886419504880905,
+            -0.0056962003000080585,
+            0.023264970630407333,
+            -0.024271776899695396,
+            0.01936165988445282,
+            0.04240977764129639,
+            -0.06957805901765823,
+            0.024659009650349617,
+            0.017549408599734306,
+            0.05120771750807762,
+            0.06511713564395905,
+            0.018339363858103752,
+            0.021669568493962288,
+            -0.014141756109893322,
+            -0.017766257748007774,
+            -0.011872569099068642,
+            0.004673904739320278,
+            -0.014970434829592705,
+            -0.01891246810555458,
+            0.008449427783489227,
+            -0.020461400970816612,
+            0.008209343999624252,
+            -0.05576157942414284,
+            -0.014730350114405155,
+            -0.034200433641672134,
+            -0.01140014547854662,
+            0.017642343416810036,
+            -0.018215449526906013,
+            0.030126741155982018,
+            0.02997184917330742,
+            0.002638994250446558,
+            0.0007822110201232135,
+            0.021049996837973595,
+            0.006509389728307724,
+            -0.0053438181057572365,
+            0.012321759946644306,
+            0.028159597888588905,
+            -0.01450575515627861,
+            -0.010106786154210567,
+            -0.03454120084643364,
+            0.01861817203462124,
+            0.01589204929769039,
+            -0.04194509983062744,
+            -0.012569589540362358,
+            -0.013824224472045898,
+            0.032341714948415756,
+            -0.009549170732498169,
+            0.04098476096987724,
+            -0.010083552449941635,
+            0.001383390510454774,
+            -0.031149037182331085,
+            -0.009463978931307793,
+            0.017069239169359207,
+            0.006172496825456619,
+            -0.016589069738984108,
+            -0.0012846461031585932,
+            0.016170857474207878,
+            0.006617815233767033,
+            -0.030142230913043022,
+            0.04488807171583176,
+            0.023497309535741806,
+            0.055296897888183594,
+            0.05579255893826485,
+            -0.037391237914562225,
+            -0.002445377642288804,
+            0.0344172865152359,
+            -0.0228312686085701,
+            -0.015040136873722076,
+            -0.034014564007520676,
+            -0.01768881268799305,
+            0.035935238003730774,
+            -0.0318460576236248,
+            0.01556677371263504,
+            0.028066661208868027,
+            -0.025712283328175545,
+            0.003488970920443535,
+            -0.03943582624197006,
+            0.003622566582635045,
+            0.01220559049397707,
+            0.006095050368458033,
+            -0.0026293133851140738,
+            -0.013615119270980358,
+            0.008906363509595394,
+            0.003055269829928875,
+            0.00800798274576664,
+            -0.04615819454193115,
+            0.018478767946362495,
+            -0.00454224506393075,
+            0.04033420979976654,
+            0.042502716183662415,
+            -0.02730768360197544,
+            -0.016743963584303856,
+            0.008557853288948536,
+            0.012104909867048264,
+            0.04188314080238342,
+            -0.007287728600203991,
+            -0.00121784838847816,
+            -0.0007836631848476827,
+            0.007322579622268677,
+            0.047552235424518585,
+            0.004646798130124807,
+            0.0020697612781077623,
+            -0.027230238541960716,
+            0.014978179708123207,
+            -0.019733402878046036,
+            0.027648448944091797,
+            0.023373395204544067,
+            0.02942972257733345,
+            -0.04048910364508629,
+            0.017518429085612297,
+            -0.011818356812000275,
+            -0.005568413063883781,
+            -0.002726121572777629,
+            -0.02856232039630413,
+            -0.0074426219798624516,
+            0.033983584493398666,
+            0.015365412458777428,
+            0.012956822291016579,
+            0.03271345794200897,
+            0.0012052633101120591,
+            -0.042502716183662415,
+            0.03804178908467293,
+            0.024550583213567734,
+            0.04640602692961693,
+            0.023466331884264946,
+            0.010796061716973782,
+            0.006571347359567881,
+            0.013375034555792809,
+            -0.019377149641513824,
+            0.015907539054751396,
+            0.02560385875403881,
+            0.009750531986355782,
+            -0.03167567402124405,
+            -0.0009501734166406095,
+            0.01178737822920084,
+            -0.004546117503196001,
+            0.03701949119567871,
+            0.04104671627283096,
+            0.020569827407598495,
+            0.05030933395028114,
+            0.0020155487582087517,
+            0.017936641350388527,
+            0.005936284549534321,
+            0.02820606529712677,
+            0.030467506498098373,
+            -0.005285732913762331,
+            0.005076627247035503,
+            -0.01136916596442461,
+            -0.03385967016220093,
+            0.014110777527093887,
+            0.01561324205249548,
+            -0.008077683858573437,
+            -0.01697630248963833,
+            0.006033093202859163,
+            0.009169681929051876,
+            -0.01795213110744953,
+            -0.017456471920013428,
+            0.005971135571599007,
+            0.018866000697016716,
+            -0.013576395809650421,
+            0.024689987301826477,
+            -0.04442339017987251,
+            0.029119934886693954,
+            -0.033209118992090225,
+            -0.003384418087080121,
+            -0.010641167871654034,
+            -0.0030843124259263277,
+            0.0041666291654109955,
+            0.011314953677356243,
+            0.013406013138592243,
+            0.002042655134573579,
+            -0.013909416273236275,
+            0.024519605562090874,
+            0.018447788432240486,
+            0.012073930352926254,
+            -0.03148980438709259,
+            -0.014397329650819302,
+            -0.0015673263696953654,
+            -0.008743725717067719,
+            -0.034138478338718414,
+            0.026966920122504234,
+            0.014048820361495018,
+            -0.03844451159238815,
+            0.017797237262129784,
+            0.029538147151470184,
+            0.014420564286410809,
+            -0.004790074657648802,
+            -0.0463130883872509,
+            0.04045812413096428,
+            -0.030436528846621513,
+            -0.033333033323287964,
+            -0.010478530079126358,
+            -0.0003146269591525197,
+            0.02894955314695835,
+            -0.0064358157105743885,
+            -0.050835974514484406,
+            0.01184933539479971,
+            0.020631784573197365,
+            0.00010763872705865651,
+            0.005731051322072744,
+            -0.019593998789787292,
+            0.00028631053282879293,
+            -0.03395260497927666,
+            0.0090844901278615,
+            0.026750069111585617,
+            0.008867640048265457,
+            -0.016418687999248505,
+            0.03351890295743942,
+            -0.0004121613164898008,
+            -0.018695618957281113,
+            -0.003889757441356778,
+            0.006885006092488766,
+            0.034912943840026855,
+            0.004836542531847954,
+            -0.02946070022881031,
+            0.05665995925664902,
+            -0.02100352756679058,
+            0.021344292908906937,
+            0.05631919577717781,
+            -0.019439106807112694,
+            -0.008472662419080734,
+            -0.002065889071673155,
+            0.0026196325197815895,
+            -0.0260530486702919,
+            -0.008937342092394829,
+            0.032341714948415756,
+            0.041728246957063675,
+            -0.018742086365818977,
+            -0.001398879918269813,
+            0.006780453026294708,
+            0.009425255469977856,
+            0.004948840010911226,
+            0.02692045085132122,
+            -0.012128143571317196,
+            -0.014559967443346977,
+            0.05198218300938606,
+            0.028314489871263504,
+            -0.012081675231456757,
+            -0.0164031982421875,
+            0.019888296723365784,
+            -0.024844881147146225,
+            -0.020244551822543144,
+            -0.02431824430823326,
+            -0.020414933562278748,
+            -0.0042518205009400845,
+            0.009525936096906662,
+            0.037484172731637955,
+            -0.03968365862965584,
+            0.012747716158628464,
+            0.021700548008084297,
+            0.02981695532798767,
+            -0.019532041624188423,
+            -0.05910727381706238,
+            -0.0016128262504935265,
+            0.03296129032969475,
+            -0.015458349138498306,
+            -0.049379974603652954,
+            0.015164051204919815,
+            0.0260530486702919,
+            0.05957195535302162,
+            -0.001461805310100317,
+            -0.014954946003854275,
+            0.061926331371068954,
+            0.03026614524424076,
+            -0.004383479710668325,
+            -0.009494957514107227,
+            -0.0127864396199584,
+            0.003396035172045231,
+            0.00465454300865531,
+            -0.03364282101392746,
+            -0.041170634329319,
+            -0.011841590516269207,
+            -0.010284913703799248,
+            0.008108663372695446,
+            -0.040922801941633224,
+            0.03531566634774208,
+            -0.021917399019002914,
+            0.010950954630970955,
+            0.015473837964236736,
+            0.037391237914562225,
+            -0.021344292908906937,
+            -0.01284839678555727,
+            -0.01572941243648529,
+            -0.024968795478343964,
+            -0.014010096900165081,
+            0.021917399019002914,
+            0.007682706229388714,
+            3.4699722164077684e-05,
+            0.010075807571411133,
+            -0.024659009650349617,
+            -0.051052823662757874,
+            0.0291044469922781,
+            0.0015605497173964977,
+            -0.015938518568873405,
+            -0.004623564425855875,
+            -0.01977987214922905,
+            -0.01122201792895794,
+            -0.012608313001692295,
+            -0.039249956607818604,
+            0.021111954003572464,
+            -0.01135367713868618,
+            0.015876561403274536,
+            0.0009250032599084079,
+            0.003659353591501713,
+            0.028624277561903,
+            0.01116006076335907,
+            0.0051811798475682735,
+            -0.03255856782197952,
+            -0.018277406692504883,
+            0.010122275911271572,
+            -0.00561100896447897,
+            -0.018339363858103752,
+            -0.013351799920201302,
+            -0.05037129297852516,
+            -0.03305422514677048,
+            -0.008054450154304504,
+            -0.05415068939328194,
+            0.0020697612781077623,
+            -0.014296649023890495,
+            -0.02232012152671814,
+            -0.005990497302263975,
+            0.03107159025967121,
+            -0.003235333366319537,
+            0.025294071063399315,
+            0.0076594725251197815,
+            0.037360258400440216,
+            0.040117356926202774,
+            0.008426194079220295,
+            0.03065337985754013,
+            -0.0029991210903972387,
+            -0.016867877915501595,
+            0.05486319959163666,
+            -0.01881953328847885,
+            0.013529927469789982,
+            -0.008604321628808975,
+            0.008604321628808975,
+            0.03299226611852646,
+            -0.003794885240495205,
+            0.02007416822016239,
+            -0.0311955064535141,
+            -0.07998688519001007,
+            0.01637221872806549,
+            0.028159597888588905,
+            -0.06418777257204056,
+            0.014691627584397793,
+            -0.020275529474020004,
+            0.02052335813641548,
+            0.02411688305437565,
+            -0.0009114501299336553,
+            -0.014064309187233448,
+            0.005595519673079252,
+            0.0020833145827054977,
+            -0.01456771232187748,
+            -0.002691270550712943,
+            -0.009417511522769928,
+            0.008774704299867153,
+            -0.017007282003760338,
+            -0.013406013138592243,
+            -0.034231413155794144,
+            0.010873507708311081,
+            0.01233724970370531,
+            -0.003969140350818634,
+            0.0011500825639814138,
+            -0.03277541697025299,
+            -0.007876322604715824,
+            -0.02145271934568882,
+            0.03382869064807892,
+            0.012058441527187824,
+            -0.030699847266077995,
+            -0.007814365439116955,
+            0.015264731831848621,
+            -0.007876322604715824,
+            0.0008906363509595394,
+            0.024132372811436653,
+            0.02103450708091259,
+            0.0028364830650389194,
+            -0.0009700191440060735,
+            0.02438020147383213,
+            0.007427132688462734,
+            -0.01720864325761795,
+            0.028794659301638603,
+            0.014288905076682568,
+            0.060160547494888306,
+            -0.03271345794200897,
+            0.001297231181524694,
+            -0.04135650396347046,
+            -0.031118059530854225,
+            0.006408709101378918,
+            0.014350862242281437,
+            0.007597515359520912,
+            -0.028252532705664635,
+            0.005978880450129509,
+            0.02004319056868553,
+            0.01469937153160572,
+            0.009704063646495342,
+            0.0037580979987978935,
+            -0.036740683019161224,
+            -0.000691695255227387,
+            -0.011214273050427437,
+            -0.0038781403563916683,
+            0.035656433552503586,
+            -0.008186109364032745,
+            -0.05151750147342682,
+            -0.0006418389966711402,
+            -0.03255856782197952,
+            -0.0353776253759861,
+            0.021313315257430077,
+            -0.00839521549642086,
+            -0.0039207362569868565,
+            -0.013986863195896149,
+            -0.000537770101800561,
+            -0.010524998418986797,
+            0.027509044855833054,
+            0.028345469385385513,
+            0.022490503266453743,
+            -0.05384090170264244,
+            0.009479468688368797,
+            0.00023766436788719147,
+            -0.05817791447043419,
+            0.0009709871956147254,
+            -0.007117345929145813,
+            0.011191039346158504,
+            -0.020213572308421135,
+            -0.02414786070585251,
+            0.011500826105475426,
+            0.03392162546515465,
+            0.014304393902420998,
+            0.0321558453142643,
+            -0.013367289677262306,
+            -0.09250226616859436,
+            0.018292896449565887,
+            -0.008689512498676777,
+            0.04126356914639473,
+            0.00865078903734684,
+            -0.004278926644474268,
+            -0.015845581889152527,
+            -0.00865078903734684,
+            -0.006943091284483671,
+            0.01283290795981884,
+            0.004530628211796284,
+            -0.005626498255878687,
+            0.0016631665639579296,
+            0.013708055019378662,
+            -0.005967263597995043,
+            0.007481344975531101,
+            0.003651608945801854,
+            -0.013189162127673626,
+            -0.06059424951672554,
+            -0.04055105894804001,
+            -0.018540725111961365,
+            0.050835974514484406,
+            -0.0069159846752882,
+            -0.046684831380844116,
+            -0.010850274004042149,
+            0.05058814212679863,
+            0.012259802781045437,
+            -0.0334569476544857,
+            -0.0007841471815481782,
+            0.0050882440991699696,
+            0.015458349138498306,
+            0.004546117503196001,
+            -0.027292195707559586,
+            0.02820606529712677,
+            0.0008412640891037881,
+            0.02300165221095085,
+            -0.018339363858103752,
+            -0.005312839522957802,
+            0.035780347883701324,
+            -0.024844881147146225,
+            0.023791607469320297,
+            0.0055219451896846294,
+            0.031629208475351334,
+            0.002952652983367443,
+            0.010060317814350128,
+            0.004704883322119713,
+            0.0186491496860981,
+            0.004275054205209017,
+            0.0276639387011528,
+            0.011462102644145489,
+            0.006428070832043886,
+            0.021623101085424423,
+            -0.009378788061439991,
+            -0.014513500034809113,
+            0.01885051093995571,
+            -0.0014695499558001757,
+            0.025232113897800446,
+            -0.020538847893476486,
+            0.002726121572777629,
+            -0.0016883367206901312,
+            -0.017967619001865387,
+            -0.040922801941633224,
+            -0.0007405834621749818,
+            0.021747015416622162,
+            -0.011671207845211029,
+            -0.0228312686085701,
+            -0.013715799897909164,
+            -0.01585332676768303,
+            0.0038490979932248592,
+            0.034169457852840424,
+            -0.0398695282638073,
+            0.00963436160236597,
+            -0.000830615172162652,
+            -0.03125746175646782,
+            -0.0239619892090559,
+            -0.01093546487390995,
+            0.014614180661737919,
+            -0.01652711257338524,
+            -0.021173911169171333,
+            -0.028469383716583252,
+            -0.008410705253481865,
+            0.006048582494258881,
+            -0.05065010115504265,
+            -0.004054331686347723,
+            0.019965743646025658,
+            0.0276639387011528,
+            -0.003099801717326045,
+            -0.02177799493074417,
+            0.0012468908680602908,
+            0.022351099178195,
+            0.02936776541173458,
+            0.014226946979761124,
+            0.027586491778492928,
+            -0.002178186783567071,
+            -0.0011675080750137568,
+            -0.0073148347437381744,
+            0.01307299267500639,
+            0.0011287847300991416,
+            0.036616768687963486,
+            0.0122210793197155,
+            -0.020306508988142014,
+            0.0013959755888208747,
+            -0.006431943271309137,
+            0.029181893914937973,
+            -0.02270735427737236,
+            -9.832092473516241e-05,
+            0.03531566634774208,
+            0.027741385623812675,
+            0.004933350719511509,
+            -0.0143121387809515,
+            -0.05142456665635109,
+            0.021592123433947563,
+            -0.01765783317387104,
+            0.031025122851133347,
+            0.016666516661643982,
+            0.003397971158847213,
+            -0.0034212053287774324,
+            0.012368228286504745,
+            -0.022087780758738518,
+            -0.004406713880598545,
+            -0.021142931655049324,
+            0.04389675334095955,
+            -0.043184246867895126,
+            0.008472662419080734,
+            0.029445212334394455,
+            0.011926782317459583,
+            -0.019082851707935333,
+            -0.005425137002021074,
+            0.029212871566414833,
+            -0.00162637943867594,
+            0.01308848150074482,
+            0.0058665829710662365,
+            -0.026935940608382225,
+            0.01202746294438839,
+            -0.031660184264183044,
+            0.02177799493074417,
+            -0.013436991721391678,
+            0.018447788432240486,
+            0.03342596814036369,
+            -0.0030397805385291576,
+            -0.0038239278364926577,
+            -0.016263794153928757,
+            -0.00969631876796484,
+            0.0662323608994484,
+            0.01054823212325573,
+            0.03726732358336449,
+            0.0018974426202476025,
+            -0.009773765690624714,
+            0.007969259284436703,
+            0.0063041565008461475,
+            -0.012329504825174809,
+            0.010494019836187363,
+            0.024178840219974518,
+            0.0012730291346088052,
+            -0.002412462839856744,
+            -0.019408127292990685,
+            -0.03438630700111389,
+            0.003727119415998459,
+            -0.030932186171412468,
+            0.022691864520311356,
+            0.009208405390381813,
+            0.017456471920013428,
+            0.008240322582423687,
+            -0.015907539054751396,
+            0.03200095146894455,
+            0.05461536720395088,
+            0.0019768255297094584,
+            0.013375034555792809,
+            0.007938280701637268,
+            0.007036027032881975,
+            -0.011338187381625175,
+            0.04702559858560562,
+            0.000662652775645256,
+            0.005169562995433807,
+            0.022536970674991608,
+            0.02145271934568882,
+            0.022939693182706833,
+            0.002304037567228079,
+            -0.011694442480802536,
+            0.010478530079126358,
+            -0.0006302220281213522,
+            0.004445436876267195,
+            -0.010695381090044975,
+            -0.027865299955010414,
+            0.007492962293326855,
+            0.01302652433514595,
+            0.022598929703235626,
+            0.021592123433947563,
+            -0.03178409859538078,
+            -0.01855621486902237,
+            0.03364282101392746,
+            -0.02997184917330742,
+            0.03612111136317253,
+            -0.0014424435794353485,
+            -0.023807097226381302,
+            -0.0058356039226055145,
+            0.016929835081100464,
+            0.02656419761478901,
+            -0.029181893914937973,
+            0.01804506592452526,
+            0.029693040996789932,
+            -0.0074426219798624516,
+            0.0005939188995398581,
+            0.012112653814256191,
+            0.02447313815355301,
+            -0.009293596260249615,
+            -0.0024027819745242596,
+            -0.0013320822035893798,
+            -0.011926782317459583,
+            -0.018370341509580612,
+            -0.0044725434854626656,
+            -0.022629907354712486,
+            0.005564541090279818,
+            -0.007365175057202578,
+            -0.003481226274743676,
+            -0.008581086993217468,
+            0.01154729351401329,
+            0.001979729626327753,
+            -0.019857317209243774,
+            -0.022335609421133995,
+            0.01652711257338524,
+            0.03887821361422539,
+            -0.018215449526906013,
+            -0.02106548473238945,
+            0.024442158639431,
+            -0.01759587600827217,
+            -0.01601596549153328,
+            0.036678727716207504,
+            0.013739033602178097,
+            0.007736918982118368,
+            -0.03358086198568344,
+            -0.030049296095967293,
+            -0.03274443745613098,
+            0.006296411622315645,
+            0.03785591572523117,
+            -0.0028752065263688564,
+            0.02569679357111454,
+            0.03701949119567871,
+            0.0057891360484063625,
+            -0.028934063389897346,
+            -0.014823286794126034,
+            -0.045104920864105225,
+            0.060160547494888306,
+            0.0011481463443487883,
+            0.05161044001579285,
+            0.016294771805405617,
+            -0.01456771232187748,
+            0.00419760774821043,
+            -0.013251119293272495,
+            0.026192452758550644,
+            0.02280029095709324,
+            0.025929134339094162,
+            0.006582964211702347,
+            0.022505993023514748,
+            -0.0037522895727306604,
+            -0.012197845615446568,
+            0.007307090330868959,
+            0.014536733739078045,
+            0.0002497654058970511,
+            -0.011392400600016117,
+            0.015783624723553658,
+            -0.01307299267500639,
+            -0.011361422017216682,
+            0.023295948281884193,
+            0.01368482131510973,
+            -0.012344993650913239,
+            0.014064309187233448,
+            -0.008565598167479038,
+            0.01239920686930418,
+            -0.023915521800518036,
+            0.022691864520311356,
+            -0.03262052312493324,
+            0.009587894193828106,
+            0.0015053689712658525,
+            -0.0073148347437381744,
+            0.024550583213567734,
+            -0.0027454833034425974,
+            -0.05071205645799637,
+            0.03550153970718384,
+            -0.008015726692974567,
+            0.0037213109899312258,
+            0.006381602957844734,
+            0.011392400600016117,
+            -0.0012623801594600081,
+            0.03807276859879494,
+            0.01239920686930418,
+            0.015969496220350266,
+            0.02199484594166279,
+            0.011911292560398579,
+            -0.008433938957750797,
+            -0.004096927121281624,
+            -0.0016137943603098392,
+            -0.01784370467066765,
+            0.025944624096155167,
+            0.0018674320308491588,
+            0.03962169960141182,
+            -0.018168980255723,
+            -0.02300165221095085,
+            0.028066661208868027,
+            0.007597515359520912,
+            -0.02351279929280281,
+            -0.0017377089243382215,
+            0.01202746294438839,
+            -0.009185170754790306,
+            0.027385130524635315,
+            0.01196550577878952,
+            0.019113829359412193,
+            -0.018060555681586266,
+            -0.004340883810073137,
+            -0.005072754807770252,
+            -0.015938518568873405,
+            0.02238207869231701,
+            0.020771188661456108,
+            0.011152315884828568,
+            -0.011880313977599144,
+            -0.01307299267500639,
+            0.020600805059075356,
+            0.035036858171224594,
+            -0.0002744515077210963,
+            0.005672966130077839,
+            0.017487451434135437,
+            0.002373739378526807,
+            -0.04470219835639,
+            0.016232814639806747,
+            -0.014025585725903511,
+            0.011214273050427437,
+            0.019593998789787292,
+            -0.021917399019002914,
+            -0.03196997195482254,
+            -0.011539548635482788,
+            -0.0414804182946682,
+            0.00493722315877676,
+            -0.0069159846752882,
+            0.013576395809650421,
+            -0.010176488198339939,
+            -0.0014831031439825892,
+            0.016666516661643982,
+            0.0038103745318949223,
+            0.02177799493074417,
+            -0.011678952723741531,
+            -0.012073930352926254,
+            -0.020213572308421135,
+            -0.020786678418517113,
+            0.004278926644474268,
+            -0.030405549332499504,
+            -0.004488032776862383,
+            0.021375272423028946,
+            -0.04783104360103607,
+            -0.03689557686448097,
+            0.02898053266108036,
+            -0.008503641001880169,
+            0.020755698904395103,
+            0.022955182939767838,
+            -0.015094350092113018,
+            -0.0008717587334103882,
+            -0.018695618957281113,
+            0.02258343994617462,
+            -0.007837600074708462,
+            0.026300877332687378,
+            -0.0027551641687750816,
+            0.0006292539183050394,
+            -0.02659517526626587,
+            0.020306508988142014,
+            0.005227648187428713,
+            0.012275291606783867,
+            -0.02390003204345703,
+            0.006815304048359394,
+            -0.03133491054177284,
+            -0.01784370467066765,
+            0.0180140882730484,
+            -0.011152315884828568,
+            0.027989214286208153,
+            -0.006060199346393347,
+            0.028438406065106392,
+            0.022722844034433365,
+            0.024751944467425346,
+            0.02676555886864662,
+            -0.02363671362400055,
+            0.008495896123349667,
+            0.03726732358336449,
+            0.00865078903734684,
+            -0.015876561403274536,
+            0.008170620538294315,
+            -0.04863648861646652,
+            -0.008062195032835007,
+            0.05030933395028114,
+            0.002127846470102668,
+            0.023141054436564445,
+            0.016294771805405617,
+            -0.002873270306736231,
+            -0.005688455421477556,
+            0.010269423946738243,
+            -0.04296739399433136,
+            -0.023172033950686455,
+            0.022629907354712486,
+            -0.021561143919825554,
+            6.54212328754511e-07,
+            0.03122648410499096,
+            0.012120398692786694,
+            -0.0009109660750254989,
+            -0.05108380317687988,
+            0.02100352756679058,
+            0.01178737822920084,
+            0.014443797990679741,
+            -0.010300402529537678,
+            0.026068538427352905,
+            0.01733255758881569,
+            0.038413532078266144,
+            0.012755461037158966,
+            -0.002679653698578477,
+            -0.03782493621110916,
+            0.009177426807582378,
+            -0.021685058251023293,
+            0.0011471783509477973,
+            0.0074503663927316666,
+            -0.02273833192884922,
+            -0.024782923981547356,
+            0.0043912241235375404,
+            0.004588713403791189,
+            0.011880313977599144,
+            0.022087780758738518,
+            0.009417511522769928,
+            -0.021204888820648193,
+            0.015404135920107365,
+            -0.05446047708392143,
+            -0.015450604259967804,
+            -0.023884542286396027,
+            0.02566581591963768,
+            0.020012211054563522,
+            0.010184233076870441,
+            0.005545179359614849,
+            0.010904486291110516,
+            -0.020988039672374725,
+            -0.009239383973181248,
+            0.020631784573197365,
+            0.011074868962168694,
+            -0.012840652838349342,
+            0.04113965481519699,
+            -0.00011126903700642288,
+            0.031210994347929955,
+            -0.02280029095709324,
+            0.002941036131232977,
+            -0.001241082325577736,
+            -0.03649285435676575,
+            0.009130958467721939,
+            0.01981084980070591,
+            0.02151467651128769,
+            -0.01861817203462124,
+            0.0024143988266587257,
+            0.01048627495765686,
+            0.004766840487718582,
+            0.038351576775312424,
+            0.027121812105178833,
+            -0.044206541031599045,
+            0.011663463898003101,
+            0.012995545752346516,
+            -0.009363298304378986,
+            0.048419639468193054,
+            -0.017518429085612297,
+            0.017673322930932045,
+            0.008240322582423687,
+            -0.01798310875892639,
+            0.02318752370774746,
+            -0.01469937153160572,
+            -0.013282098807394505,
+            0.00650551775470376,
+            0.009192915633320808,
+            -0.0001770381786627695,
+            -0.037577107548713684,
+            -0.017859194427728653,
+            -0.005719434004276991,
+            0.0038297362625598907,
+            0.01562873087823391,
+            0.013498948886990547,
+            0.009967382065951824,
+            0.005161818582564592,
+            -0.00945623405277729,
+            -0.0016089539276435971,
+            0.008124152198433876,
+            -0.0005532593932002783,
+            0.0019216446671634912,
+            -0.01926872320473194,
+            0.02698240801692009,
+            0.02572777308523655,
+            -0.0018354853382334113,
+            -0.04652994126081467,
+            -0.0036709706764668226,
+            0.0037077576853334904,
+            -0.017735280096530914,
+            -0.0035315665882080793,
+            -0.04519785940647125,
+            -0.01810702309012413,
+            0.0143121387809515,
+            -0.014614180661737919,
+            -0.0460033044219017,
+            0.009076746180653572,
+            -0.013886182568967342,
+            0.0063970922492444515,
+            -0.00932457484304905,
+            0.005762029904872179,
+            0.003651608945801854,
+            0.005680711008608341,
+            -0.005909178406000137,
+            0.05108380317687988,
+            -0.03466511517763138,
+            0.028051171451807022,
+            0.008565598167479038,
+            -0.012437930330634117,
+            -0.02261441759765148,
+            0.00887538492679596,
+            -0.044981006532907486,
+            -0.033209118992090225,
+            0.016743963584303856,
+            -0.01781272701919079,
+            0.011074868962168694,
+            0.06161654368042946,
+            0.007330324035137892,
+            -0.023342415690422058,
+            -0.01233724970370531,
+            -0.021049996837973595,
+            -0.04191412031650543,
+            0.003442503046244383,
+            0.005463860463351011,
+            -0.004948840010911226,
+            -0.03807276859879494,
+            -0.021716037765145302,
+            0.04615819454193115,
+            -0.012004229240119457,
+            0.012422440573573112,
+            0.01080380566418171,
+            -0.0019410063978284597,
+            -0.04860550910234451,
+            -0.021019017323851585,
+            0.0395597442984581,
+            0.021886419504880905,
+            -0.008426194079220295,
+            0.0028364830650389194,
+            -0.018277406692504883,
+            -0.003289545886218548,
+            0.006296411622315645,
+            -0.009107724763453007,
+            -0.03562545403838158,
+            0.02067825198173523,
+            0.0054832217283546925,
+            -0.023373395204544067,
+            -0.025805220007896423,
+            -0.03296129032969475,
+            0.009944148361682892,
+            0.02453509531915188,
+            -0.01968693546950817,
+            0.005823987070471048,
+            0.00027203132049180567,
+            -0.000881923595443368,
+            0.00042740863864310086,
+            -0.009959637187421322,
+            -0.019764382392168045,
+            -0.020151615142822266,
+            0.03268248215317726,
+            -0.0060059865936636925,
+            0.0074387495405972,
+            -0.03630698472261429,
+            -0.010974188335239887,
+            0.011322698555886745,
+            -0.002395037328824401,
+            -0.013452480547130108,
+            -0.009092235006392002,
+            -0.01739451475441456,
+            -0.009882191196084023,
+            -0.007876322604715824,
+            0.022660886868834496,
+            0.006168624851852655,
+            0.005409647710621357,
+            0.0023795480374246836,
+            -0.008581086993217468,
+            0.02898053266108036,
+            0.017874684184789658,
+            -0.014188224449753761,
+            -0.00920066051185131,
+            0.03451022133231163,
+            0.02665713243186474,
+            -0.009115469641983509,
+            0.01798310875892639,
+            0.023264970630407333,
+            -0.0010484338272362947,
+            -0.01196550577878952,
+            0.001922612776979804,
+            -0.0009550138493068516,
+            -0.0032972905319184065,
+            0.00845717266201973,
+            -0.015365412458777428,
+            0.0315207801759243,
+            0.04287445917725563,
+            -0.002145271748304367,
+            0.009587894193828106,
+            -0.003727119415998459,
+            -0.008472662419080734,
+            0.017069239169359207,
+            0.009316830895841122,
+            0.04625113308429718,
+            0.01098193321377039,
+            0.005680711008608341,
+            -0.033983584493398666,
+            -0.001229465357027948,
+            -0.01184933539479971,
+            0.03308520466089249,
+            0.02061629481613636,
+            0.00889087375253439,
+            -0.008379725739359856,
+            0.01344473659992218,
+            0.008341003209352493,
+            0.00661007035523653,
+            0.020569827407598495,
+            -0.016542602330446243,
+            -0.017286090180277824,
+            0.030467506498098373,
+            -0.00672236829996109,
+            0.01289486512541771,
+            0.02029101923108101,
+            -0.002089123008772731,
+            0.010842529125511646,
+            0.07100307941436768,
+            -0.006184114143252373,
+            0.015713922679424286,
+            -0.019082851707935333,
+            0.0015789433382451534,
+            -0.007640110794454813,
+            -0.03370477631688118,
+            0.018246427178382874,
+            0.0039962464943528175,
+            0.00865078903734684,
+            0.01198099460452795,
+            0.009053511545062065,
+            -0.007946024648845196,
+            -0.00963436160236597,
+            0.015195030719041824,
+            0.0019080914789810777,
+            0.00033568276558071375,
+            0.001336922636255622,
+            -0.0065210070461034775,
+            -0.004011735785752535,
+            -0.021948376670479774,
+            0.02763296104967594,
+            0.007140580099076033,
+            0.030606910586357117,
+            -0.055296897888183594,
+            0.030095763504505157,
+            -0.00789568480104208,
+            0.02554190158843994,
+            -0.016201836988329887,
+            -0.011090358719229698,
+            0.001609921921044588,
+            0.018153492361307144,
+            0.004491904750466347,
+            0.006462921854108572,
+            0.011888058856129646,
+            0.023233991116285324,
+            -0.03444826602935791,
+            0.017053749412298203,
+            0.023110076785087585,
+            0.0048675211146473885,
+            0.0003535923024173826,
+            0.005165690556168556,
+            0.015458349138498306,
+            0.007198664825409651,
+            -0.023048119619488716,
+            -0.0025692922063171864,
+            0.017611365765333176,
+            -0.02306360937654972,
+            0.0036980770528316498,
+            -0.006923729553818703,
+            0.004623564425855875,
+            -0.0014705180656164885,
+            -0.03509881719946861,
+            -0.00100099784322083,
+            -0.025743262842297554,
+            0.01923774555325508,
+            0.009556914679706097,
+            -0.025867177173495293,
+            0.003932353109121323,
+            0.025650326162576675,
+            -0.0003695656778290868,
+            0.00821708794683218,
+            0.0004683101433329284,
+            0.00869725737720728,
+            -0.00974278710782528,
+            -0.01493945624679327,
+            0.004801691509783268,
+            0.01531120017170906,
+            -0.01493945624679327,
+            -0.007431004662066698,
+            -0.009510447271168232,
+            0.030560443177819252,
+            0.0006674932083114982,
+            0.004189862869679928,
+            -0.005045648664236069,
+            -0.01858719252049923,
+            -0.0032237162813544273,
+            -0.021142931655049324,
+            0.004367990419268608,
+            -0.0009046735358424485,
+            0.04197607934474945,
+            0.0038161829579621553,
+            -0.02029101923108101,
+            0.0032972905319184065,
+            -0.03136589005589485,
+            0.0005000148667022586,
+            -0.007276111748069525,
+            -0.02004319056868553,
+            0.0015005286550149322,
+            0.0016525175888091326,
+            0.012631546705961227,
+            -0.03255856782197952,
+            -0.009301341138780117,
+            0.018416810780763626,
+            -0.00020680672605521977,
+            -0.010687636211514473,
+            -0.017828216776251793,
+            0.00035189816844649613,
+            -0.011338187381625175,
+            -0.027973726391792297,
+            -0.02850036323070526,
+            -0.0026719090528786182,
+            0.013491204008460045,
+            -0.018277406692504883,
+            -0.014993669465184212,
+            0.0031249718740582466,
+            0.021592123433947563,
+            -0.009781510569155216,
+            0.008302279748022556,
+            0.02241305634379387,
+            -0.008255811408162117,
+            -0.005514200776815414,
+            -0.000941460661124438,
+            -0.009130958467721939,
+            0.004267309792339802,
+            0.006300284061580896,
+            0.043339136987924576,
+            -0.032403673976659775,
+            -0.0018238683696836233,
+            0.014815541915595531,
+            0.0034153966698795557,
+            0.0025905899237841368,
+            0.002503462601453066,
+            -0.0007841471815481782,
+            0.01344473659992218,
+            0.01178737822920084,
+            0.015466093085706234,
+            0.00100099784322083,
+            -0.0033011629711836576,
+            0.006408709101378918,
+            -0.021173911169171333,
+            0.015721667557954788,
+            0.027958236634731293,
+            -0.0033999073784798384,
+            0.005525817628949881,
+            -0.013731288723647594,
+            -0.008759214542806149,
+            0.013134949840605259,
+            0.0026641644071787596,
+            -0.005068882368505001,
+            -0.009394276887178421,
+            0.017967619001865387,
+            0.019082851707935333,
+            0.006668155547231436,
+            -0.037453193217515945,
+            -0.02898053266108036,
+            -0.009959637187421322,
+            0.020569827407598495,
+            0.014916222542524338,
+            0.0010339126456528902,
+            -0.010540487244725227,
+            -0.009595638141036034,
+            -0.011903548613190651,
+            0.010989678092300892,
+            0.02721474878489971,
+            -0.032372694462537766,
+            0.0008214184199459851,
+            -0.029383255168795586,
+            0.03068435750901699,
+            0.0020000594668090343,
+            0.011446612887084484,
+            -0.0008185141487047076,
+            -0.015140817500650883,
+            -0.029181893914937973,
+            0.013824224472045898,
+            -0.004395096562802792,
+            0.001024231780320406,
+            -0.0031753121875226498,
+            0.0201051477342844,
+            0.012887120246887207,
+            -0.0014472840121015906,
+            0.01968693546950817,
+            0.008178364485502243,
+            0.016480645164847374,
+            -0.010153254494071007,
+            -0.04135650396347046,
+            0.019965743646025658,
+            0.007562664337456226,
+            0.014219203032553196,
+            0.004739734344184399,
+            0.004588713403791189,
+            -0.020771188661456108,
+            0.002724185585975647,
+            0.004333139397203922,
+            0.014405074529349804,
+            0.011028401553630829,
+            0.0025615475606173277,
+            0.011678952723741531,
+            -0.02228914201259613,
+            0.01929970271885395,
+            -0.01631026156246662,
+            0.01006806269288063,
+            0.004445436876267195,
+            -0.026873983442783356,
+            0.007547175046056509,
+            0.007822110317647457,
+            -0.0019187404541298747,
+            -0.017549408599734306,
+            0.02160761132836342,
+            -0.0034831624943763018,
+            -0.03509881719946861,
+            0.0360281765460968,
+            -0.0043447562493383884,
+            0.019346170127391815,
+            0.03209388628602028,
+            -0.01771979033946991,
+            -0.01993476413190365,
+            0.00906900130212307,
+            0.010408828034996986,
+            -0.002873270306736231,
+            0.017859194427728653,
+            -0.002919738180935383,
+            -0.02232012152671814,
+            0.017766257748007774,
+            -0.028918573632836342,
+            -0.014234691858291626,
+            0.020337486639618874,
+            -0.03624502569437027,
+            -0.0021026763133704662,
+            0.03175312280654907,
+            -0.024070415645837784,
+            0.005734923295676708,
+            0.005196669604629278,
+            -0.0019351978553459048,
+            0.0012236569309607148,
+            -0.006451305001974106,
+            0.03801080957055092,
+            -0.010865762829780579,
+            0.0201051477342844,
+            0.01929970271885395,
+            -0.023388884961605072,
+            0.01977987214922905,
+            -0.001109423115849495,
+            0.01675945147871971,
+            -0.003940097521990538,
+            0.03466511517763138,
+            -0.006017603911459446,
+            -0.011330443434417248,
+            0.006431943271309137,
+            0.008488151244819164,
+            -0.009959637187421322,
+            0.008681767620146275,
+            -0.018122512847185135,
+            0.027152791619300842,
+            0.02091059274971485,
+            -0.0002691270783543587,
+            0.013800990767776966,
+            0.0004421719058882445,
+            -0.020833145827054977,
+            0.00789568480104208,
+            -0.027121812105178833,
+            -0.009967382065951824,
+            0.01646515540778637,
+            -0.005762029904872179,
+            0.007380664348602295,
+            -0.02064727433025837,
+            0.028004704043269157,
+            0.0077524082735180855,
+            -0.013591884635388851,
+            -0.0008824076503515244,
+            0.023760627955198288,
+            -0.005432881414890289,
+            0.009355553425848484,
+            0.02070923149585724,
+            0.015713922679424286,
+            0.013615119270980358,
+            0.016263794153928757,
+            0.03351890295743942,
+            0.005328328814357519,
+            0.005568413063883781,
+            0.003519949736073613,
+            0.016991792246699333,
+            -0.008557853288948536,
+            0.018029578030109406,
+            0.0008093173382803798,
+            -0.013576395809650421,
+            0.021499186754226685,
+            -0.0018684001406654716,
+            -0.013622863218188286,
+            -0.0040891822427511215,
+            -0.004337011836469173,
+            0.03271345794200897,
+            0.022041313350200653,
+            -0.004468671046197414,
+            -0.024442158639431,
+            -0.00394784240052104,
+            0.016682006418704987,
+            -0.003167567541822791,
+            0.011136826127767563,
+            0.0007952801533974707,
+            -0.01586107164621353,
+            -0.009595638141036034,
+            0.0063583687879145145,
+            0.010586955584585667,
+            -0.008921852335333824,
+            0.010346870869398117,
+            -0.0228312686085701,
+            -0.004011735785752535,
+            0.00046613195445388556,
+            -0.004844287410378456,
+            0.0053438181057572365,
+            -0.011314953677356243,
+            0.004948840010911226,
+            0.016062432900071144,
+            0.01026167906820774,
+            0.004832670092582703,
+            0.006989559158682823,
+            0.012066186405718327,
+            0.009161937050521374,
+            -0.01627928391098976,
+            0.00428667152300477,
+            -0.005072754807770252,
+            -0.02345084212720394,
+            0.005831731949001551,
+            0.00902253296226263,
+            -0.01122201792895794,
+            0.009386532939970493,
+            -0.010958699509501457,
+            0.03296129032969475,
+            0.01030814740806818,
+            -0.037391237914562225,
+            -0.0033379499800503254,
+            -0.0016244432190433145,
+            0.014885243959724903,
+            0.012298526242375374,
+            0.008550108410418034,
+            -0.017518429085612297,
+            -0.0047823297791182995,
+            -0.009835722856223583,
+            0.0012827098835259676,
+            -0.012306271120905876,
+            0.017456471920013428,
+            0.022567950189113617,
+            -0.036337960511446,
+            0.01968693546950817,
+            -0.010432061739265919,
+            -0.023652203381061554,
+            0.020476890727877617,
+            0.011500826105475426,
+            0.01942361705005169,
+            -0.009402021765708923,
+            0.018896980211138725,
+            -0.0001331114035565406,
+            0.025510922074317932,
+            -0.004228586331009865,
+            -0.022087780758738518,
+            -0.012499887496232986,
+            -0.03209388628602028,
+            -0.0051811798475682735,
+            0.006819176487624645,
+            -0.00045524101005867124,
+            0.013111716136336327,
+            0.014226946979761124,
+            -0.025712283328175545,
+            0.007295473478734493,
+            0.032589543610811234,
+            0.014637414366006851,
+            0.023311438038945198,
+            -0.011276230216026306,
+            -0.0010794125264510512,
+            -0.0026060794480144978,
+            -0.013770012184977531,
+            -0.03212486580014229,
+            -0.0017493258928880095,
+            -0.036585792899131775,
+            0.005680711008608341,
+            -0.016387708485126495,
+            -0.013096226379275322,
+            0.023264970630407333,
+            -0.034138478338718414,
+            0.0014608372002840042,
+            0.013708055019378662,
+            0.010540487244725227,
+            0.012391461990773678,
+            -0.00162637943867594,
+            0.015032391995191574,
+            -0.029243851080536842,
+            -0.010284913703799248,
+            -0.004491904750466347,
+            -0.01981084980070591,
+            -0.007338068913668394,
+            -0.018060555681586266,
+            -0.013127204962074757,
+            -0.003566417610272765,
+            -0.010060317814350128,
+            0.002247888594865799,
+            0.0032062907703220844,
+            -0.014714861288666725,
+            0.03206290677189827,
+            0.003157886676490307,
+            0.018958937376737595,
+            0.0033418224193155766,
+            0.004449309315532446,
+            0.018788553774356842,
+            -0.0026331855915486813,
+            -0.0058278595097362995,
+            0.030963165685534477,
+            -0.00032672801171429455,
+            0.008759214542806149,
+            0.019082851707935333,
+            0.004724245052784681,
+            0.0038529701996594667,
+            -0.0260530486702919,
+            0.012887120246887207,
+            -0.027447087690234184,
+            0.005080499220639467,
+            -0.0077175572514534,
+            -0.024302754551172256,
+            -0.028128618374466896,
+            -0.006571347359567881,
+            0.012058441527187824,
+            -0.02898053266108036,
+            -0.00613377382978797,
+            0.002315654419362545,
+            -0.00400399137288332,
+            -0.002673845272511244,
+            0.007241260726004839,
+            0.029150914400815964,
+            -0.033209118992090225,
+            0.008751469664275646,
+            0.005595519673079252,
+            -0.0005053392960689962,
+            0.013297587633132935,
+            0.008038961328566074,
+            0.003343758638948202,
+            0.020848635584115982,
+            0.006637176964432001,
+            0.002458930714055896,
+            -0.0025576751213520765,
+            -0.014242436736822128,
+            0.004000118933618069,
+            0.02154565416276455,
+            0.007562664337456226,
+            0.018385831266641617,
+            -0.008914108388125896,
+            0.002621568739414215,
+            -0.01455222349613905,
+            0.005552923772484064,
+            -0.002867461647838354,
+            0.004309905227273703,
+            -0.012236569076776505,
+            0.008093173615634441,
+            0.0028771425131708384,
+            -0.0043447562493383884,
+            0.018060555681586266,
+            -0.002319526858627796,
+            0.016480645164847374,
+            0.004015608225017786,
+            0.015620986931025982,
+            -0.0035412474535405636,
+            0.01178737822920084,
+            0.0005731051205657423,
+            -0.017905661836266518,
+            -0.017022771760821342,
+            -0.039249956607818604,
+            0.007702067960053682,
+            0.03531566634774208,
+            -0.0009564659558236599,
+            -0.03444826602935791,
+            0.010145509615540504,
+            -0.011942271143198013,
+            -0.009975126944482327,
+            -0.022862248122692108,
+            0.013862947933375835,
+            0.012430185452103615,
+            0.007415515370666981,
+            0.012252057902514935,
+            -0.006997303571552038,
+            0.01026167906820774,
+            -0.005216030869632959,
+            0.008356492035090923,
+            -0.0010823167394846678,
+            -0.037422213703393936,
+            0.012344993650913239,
+            0.02216522768139839,
+            -0.002635121811181307,
+            0.0016089539276435971,
+            -0.006145390681922436,
+            0.022103270515799522,
+            -0.013971373438835144,
+            -0.00650551775470376,
+            -0.00046443779137916863,
+            0.00797700323164463,
+            -0.01795213110744953,
+            0.004778457339853048,
+            0.0755879208445549,
+            -0.007775642443448305,
+            0.006122156511992216,
+            0.009270362555980682,
+            -0.015148562379181385,
+            -0.01637221872806549,
+            -0.004561606794595718,
+            -0.020260039716959,
+            0.008844406343996525,
+            -0.0018819533288478851,
+            -0.0110981035977602,
+            0.012468908913433552,
+            0.0053050946444272995,
+            0.002031038049608469,
+            -0.008379725739359856,
+            0.010811550542712212,
+            0.009525936096906662,
+            0.010432061739265919,
+            0.02360573597252369,
+            0.011477591469883919,
+            -0.00858883187174797,
+            -0.011795123107731342,
+            0.01129171997308731,
+            0.0028577810153365135,
+            0.016697494313120842,
+            0.007907302118837833,
+            0.0010648912284523249,
+            0.019501063972711563,
+            -0.002309845993295312,
+            0.02180897258222103,
+            -2.2689444449497387e-05,
+            -0.009301341138780117,
+            -0.004236331209540367,
+            -0.015349923633038998,
+            0.024271776899695396,
+            -0.0020155487582087517,
+            0.0015450604259967804,
+            -0.020213572308421135,
+            0.0013979118084535003,
+            0.013243375346064568,
+            0.011957760900259018,
+            -0.0020407189149409533,
+            -0.0051347119733691216,
+            0.0007154133054427803,
+            -0.016356728971004486,
+            0.032806396484375,
+            0.007578153628855944,
+            0.0007681738352403045,
+            0.0008838597568683326,
+            0.0009288755827583373,
+            -0.0032972905319184065,
+            0.0024608669336885214,
+            -0.004766840487718582,
+            0.009549170732498169,
+            -0.010943209752440453,
+            -0.010796061716973782,
+            0.010075807571411133,
+            -0.02402394637465477,
+            0.003566417610272765,
+            -0.00558777479454875,
+            -0.008062195032835007,
+            0.0076207490637898445,
+            0.00015537731815129519,
+            -0.007392281666398048,
+            -0.0030939930584281683,
+            0.0023408245760947466,
+            0.013429246842861176,
+            0.004406713880598545,
+            0.010850274004042149,
+            0.012956822291016579,
+            0.005359307397156954,
+            -0.006629432085901499,
+            0.015512561425566673,
+            -0.006946963258087635,
+            -0.004228586331009865,
+            -0.0012468908680602908,
+            -0.019996721297502518,
+            -0.0009419447160325944,
+            -0.004708755761384964,
+            0.0015092412941157818,
+            0.0001705036120256409,
+            -8.749050175538287e-05,
+            -0.0018490385264158249,
+            -0.011988739483058453,
+            -0.001831613015383482,
+            -0.011663463898003101,
+            -0.014498010277748108,
+            0.010253935120999813,
+            -0.0041201612912118435,
+            -0.022939693182706833,
+            -0.015349923633038998,
+            -0.020662762224674225,
+            0.02145271934568882,
+            -0.004387352149933577,
+            -0.005800753366202116,
+            0.006733985152095556,
+            -0.010215211659669876,
+            -0.015706177800893784,
+            -0.008069939911365509,
+            -0.0205853171646595,
+            0.010703125037252903,
+            0.003214035416021943,
+            0.0006791101768612862,
+            0.00882891658693552,
+            0.014397329650819302,
+            -0.01589204929769039,
+            0.0111755495890975,
+            0.012182355858385563,
+            0.011361422017216682,
+            -0.006854027509689331,
+            0.0008911203476600349,
+            -0.014103032648563385,
+            0.02596011385321617,
+            0.021623101085424423,
+            0.005169562995433807,
+            -0.04132552444934845,
+            -0.005862710531800985,
+            -0.013344055972993374,
+            -0.010524998418986797,
+            -0.012120398692786694,
+            0.011771888472139835,
+            -0.0038820127956569195,
+            0.0013117524795234203,
+            0.006288666743785143,
+            0.0015063370810821652,
+            0.004367990419268608,
+            -0.0022382077295333147,
+            -0.007709812838584185,
+            -0.03212486580014229,
+            -0.02817508764564991,
+            -0.00863530021160841,
+            0.020461400970816612,
+            0.002838419284671545,
+            -0.0009453330421820283,
+            -0.0036380558740347624,
+            0.0027048238553106785,
+            -0.005882072262465954,
+            -0.005018542055040598,
+            -0.014753584749996662,
+            -0.012739972211420536,
+            0.0101300198584795,
+            0.00802347157150507,
+            0.014195968396961689,
+            -0.011950016021728516,
+            0.0028190575540065765,
+            0.018788553774356842,
+            0.007690451107919216,
+            -0.023357905447483063,
+            0.01135367713868618,
+            -0.010912231169641018,
+            -0.009270362555980682,
+            -0.009433000348508358,
+            -0.009471723809838295,
+            0.0016980175860226154,
+            -0.008317768573760986,
+            0.003945906180888414,
+            -0.03615209087729454,
+            -0.011330443434417248,
+            -0.0013901671627536416,
+            -0.010346870869398117,
+            0.0009864765452221036,
+            0.02524760365486145,
+            0.006815304048359394,
+            0.01432762760668993,
+            -0.0006733016925863922,
+            0.017502939328551292,
+            -0.0018984107300639153,
+            0.018726596608757973,
+            0.012763205915689468,
+            -0.00797700323164463,
+            0.011717676185071468,
+            0.014738094992935658,
+            -0.0033863543067127466,
+            0.015620986931025982,
+            -0.010168743319809437,
+            0.005285732913762331,
+            -0.009556914679706097,
+            -0.003214035416021943,
+            0.002495717955753207,
+            0.030901208519935608,
+            -0.003618694143369794,
+            0.00858883187174797,
+            0.01098193321377039,
+            -0.015241498127579689,
+            0.0026951429899781942,
+            0.014962689951062202,
+            0.01141563430428505,
+            0.007760153152048588,
+            0.0024434414226561785,
+            -0.024225307628512383,
+            0.004193735308945179,
+            0.005037903785705566,
+            0.0031520782504230738,
+            -0.014366351068019867,
+            0.0021859314292669296,
+            0.015675200149416924,
+            0.008550108410418034,
+            -0.025325050577521324,
+            0.016418687999248505,
+            0.0009138703462667763,
+            -0.003909118939191103,
+            0.029119934886693954,
+            -0.011268485337495804,
+            -0.005723306443542242,
+            0.009463978931307793,
+            0.0009980935137718916,
+            0.0030029932968318462,
+            0.008937342092394829,
+            0.0228312686085701,
+            -0.003988501615822315,
+            -0.032341714948415756,
+            -0.022459525614976883,
+            0.015706177800893784,
+            -0.0047823297791182995,
+            -0.022939693182706833,
+            -0.026703599840402603,
+            -0.003113354789093137,
+            -0.02228914201259613,
+            -0.0022323993034660816,
+            -0.020058678463101387,
+            -0.014056564308702946,
+            0.001268188701942563,
+            0.029987338930368423,
+            0.016201836988329887,
+            0.008224832825362682,
+            -0.011051635257899761,
+            -0.029956359416246414,
+            -0.024271776899695396,
+            0.01810702309012413,
+            -0.007055388763546944,
+            -0.005541306920349598,
+            0.00591692328453064,
+            -0.009951893240213394,
+            -0.008480406366288662,
+            0.011802867986261845,
+            0.027137301862239838,
+            -0.002747419523075223,
+            -0.029987338930368423,
+            -0.02688947319984436,
+            -0.014335372485220432,
+            -0.01840132102370262,
+            -0.017704300582408905,
+            -0.008627555333077908,
+            0.0045035220682621,
+            0.00030300996149890125,
+            0.016186347231268883,
+            0.014962689951062202,
+            -0.000480653194244951,
+            0.0028597170021384954,
+            -2.7711375878425315e-05,
+            -0.001763847190886736,
+            0.006888878531754017,
+            -0.004933350719511509,
+            0.026997897773981094,
+            0.014614180661737919,
+            -0.006594581063836813,
+            0.018711106851696968,
+            0.028314489871263504,
+            0.020430423319339752,
+            0.003157886676490307,
+            -0.010695381090044975,
+            -0.014606435783207417,
+            0.009727297350764275,
+            -0.013421501964330673,
+            0.023264970630407333,
+            0.022552460432052612,
+            -0.01480779703706503,
+            -0.014885243959724903,
+            -0.017905661836266518,
+            0.015210519544780254,
+            -0.007562664337456226,
+            -0.008224832825362682,
+            0.012368228286504745,
+            0.022335609421133995,
+            -0.007024410180747509,
+            0.002371803391724825,
+            -0.0010232636705040932,
+            -0.021731525659561157,
+            -0.006122156511992216,
+            -0.004596457816660404,
+            -0.008666278794407845,
+            0.008681767620146275,
+            0.008131897076964378,
+            -0.001928421319462359,
+            -0.006319645792245865,
+            0.005471604876220226,
+            -0.013367289677262306,
+            0.01301103550940752,
+            -0.00789568480104208,
+            0.03379771113395691,
+            -0.009138703346252441,
+            -0.0006733016925863922,
+            -0.022598929703235626,
+            -0.011446612887084484,
+            0.00845717266201973,
+            0.0017967619933187962,
+            0.01682141050696373,
+            -0.0011626676423475146,
+            -0.005688455421477556,
+            0.009913169778883457,
+            -0.0010687635513022542,
+            -0.001450188341550529,
+            -0.006892750971019268,
+            -0.010401083156466484,
+            0.0002884887217078358,
+            0.013258864171802998,
+            -0.006025348324328661,
+            0.0018054747488349676,
+            0.006095050368458033,
+            -0.0175339188426733,
+            0.029027000069618225,
+            0.023884542286396027,
+            0.017255110666155815,
+            0.014660648070275784,
+            0.0006394187803380191,
+            0.011647974140942097,
+            -0.003572226269170642,
+            0.0038820127956569195,
+            -0.020337486639618874,
+            0.021080974489450455,
+            -0.010811550542712212,
+            0.01846327818930149,
+            0.011655719019472599,
+            -0.015233753249049187,
+            -0.005022414494305849,
+            0.021406250074505806,
+            0.009967382065951824,
+            0.00889087375253439,
+            0.0017541663255542517,
+            -0.02470547705888748,
+            0.017766257748007774,
+            0.010099041275680065,
+            -0.020399443805217743,
+            0.019547531381249428,
+            -0.009471723809838295,
+            0.01233724970370531,
+            -0.0005300253978930414,
+            0.004418330732733011,
+            0.0058743273839354515,
+            0.021499186754226685,
+            0.011570527218282223,
+            -0.03884723410010338,
+            -0.013615119270980358,
+            -0.0017357728211209178,
+            0.006857899948954582,
+            -0.0036303112283349037,
+            -0.015233753249049187,
+            -0.041635312139987946,
+            0.01604694314301014,
+            0.009998360648751259,
+            -0.01643417589366436,
+            0.004987563472241163,
+            -0.002427952131256461,
+            -0.0042053526267409325,
+            -0.008170620538294315,
+            0.003186929039657116,
+            -0.004170501604676247,
+            0.02324948087334633,
+            0.0018984107300639153,
+            -0.006946963258087635,
+            0.013258864171802998,
+            0.01344473659992218,
+            -0.021948376670479774,
+            0.0009874446550384164,
+            0.0021375271026045084,
+            0.000636998622212559,
+            0.023683181032538414,
+            -0.014707116410136223,
+            -0.0017657832941040397,
+            -0.01685238815844059,
+            -0.0013069120468571782,
+            -0.021592123433947563,
+            -0.001188805908896029,
+            -0.015187285840511322,
+            -0.008511385880410671,
+            0.025030752643942833,
+            -0.011005166918039322,
+            0.010997422970831394,
+            -0.006687517277896404,
+            0.006943091284483671,
+            -0.006904367823153734,
+            -0.01936165988445282,
+            -0.0008267428493127227,
+            0.009975126944482327,
+            -0.000512115890160203,
+            0.002941036131232977,
+            8.482827252009884e-05,
+            0.002464739140123129,
+            0.008867640048265457,
+            0.006234454456716776,
+            -0.0075278133153915405,
+            0.02489134855568409,
+            -0.012073930352926254,
+            0.0016399325104430318,
+            0.011911292560398579,
+            0.0004743606550619006,
+            -0.010424317792057991,
+            0.025588368996977806,
+            -0.02139076218008995,
+            -0.00800798274576664,
+            -0.00014799568452872336,
+            -0.0016544538084417582,
+            -0.006513262167572975,
+            0.006718495860695839,
+            0.02946070022881031,
+            0.0062654330395162106,
+            0.010447551496326923,
+            -0.0281441081315279,
+            -0.011493081226944923,
+            0.012538610957562923,
+            -0.022273652255535126,
+            -0.0007739823195151985,
+            -0.0004322490422055125,
+            -0.00619960343465209,
+            0.026378324255347252,
+            -0.02022906206548214,
+            -0.01093546487390995,
+            -0.018664639443159103,
+            0.005405775271356106,
+            -0.011121337302029133,
+            0.008294534869492054,
+            -0.0009777637897059321,
+            0.006540368776768446,
+            0.016805920749902725,
+            -0.019563021138310432,
+            0.016031453385949135,
+            0.005119222681969404,
+            -0.016217326745390892,
+            0.010222956538200378,
+            -0.017580386251211166,
+            -0.0008199662552215159,
+            0.020275529474020004,
+            -0.00400399137288332,
+            0.008751469664275646,
+            -0.003363120136782527,
+            0.005208286456763744,
+            -0.006513262167572975,
+            0.0001492057926952839,
+            -0.007249005138874054,
+            0.014428308233618736,
+            -0.016805920749902725,
+            -0.017828216776251793,
+            -0.008821171708405018,
+            0.0016709112096577883,
+            0.021297825500369072,
+            0.011136826127767563,
+            -0.010044828988611698,
+            -0.011059380136430264,
+            -0.011694442480802536,
+            0.041604332625865936,
+            0.005762029904872179,
+            0.027230238541960716,
+            -0.02261441759765148,
+            0.00545998802408576,
+            0.025170156732201576,
+            -0.014242436736822128,
+            0.0012788375606760383,
+            -0.0022788674104958773,
+            0.0053050946444272995,
+            0.011562783271074295,
+            -0.01819995976984501,
+            0.0049139889888465405,
+            -0.00019216447253711522,
+            -0.00782985519617796,
+            -0.000656844291370362,
+            0.003442503046244383,
+            0.00020196630794089288,
+            0.016325751319527626,
+            -0.004027225077152252,
+            -0.016511622816324234,
+            -0.004429947584867477,
+            -0.0003216455806978047,
+            -0.027896279469132423,
+            0.013514438644051552,
+            0.016589069738984108,
+            0.01566745527088642,
+            0.027989214286208153,
+            -0.014513500034809113,
+            0.016031453385949135,
+            0.009533680975437164,
+            0.013901671394705772,
+            -0.010610189288854599,
+            0.008952830918133259,
+            -0.01595400646328926,
+            0.007446494419127703,
+            -0.03454120084643364,
+            -0.007887939922511578,
+            -0.00802347157150507,
+            0.01984182931482792,
+            0.028763681650161743,
+            0.002923610620200634,
+            0.006493900436908007,
+            0.000196883876924403,
+            0.0024202074855566025,
+            -0.01714668609201908,
+            -0.020182594656944275,
+            -0.0003337466041557491,
+            0.0029332914855331182,
+            0.0006950835813768208,
+            -0.004495777189731598,
+            0.00472037261351943,
+            -0.012391461990773678,
+            0.0026835259050130844,
+            -0.013134949840605259,
+            0.005971135571599007,
+            -0.0059982421807944775,
+            0.009603383019566536,
+            0.0019429425010457635,
+            0.020864123478531837,
+            0.02300165221095085,
+            0.004100799560546875,
+            0.0046971384435892105,
+            -0.024333734065294266,
+            0.00920066051185131,
+            -0.013847459107637405,
+            0.012329504825174809,
+            0.019408127292990685,
+            0.008248066529631615,
+            0.006571347359567881,
+            0.01233724970370531,
+            0.012918098829686642,
+            -0.0031772484071552753,
+            0.024287264794111252,
+            0.005014669615775347,
+            -0.0009070937521755695,
+            0.002052335999906063,
+            -0.014954946003854275,
+            -0.0070979841984808445,
+            -0.0015286030247807503,
+            0.0018848575418815017,
+            -0.004731989465653896,
+            0.01906736195087433,
+            0.015256987884640694,
+            -0.01050176378339529,
+            0.017874684184789658,
+            -0.03937387093901634,
+            0.017379024997353554,
+            -0.0027377386577427387,
+            -0.0060911779291927814,
+            -0.00545998802408576,
+            0.00665653869509697,
+            0.004515138920396566,
+            0.003669034456834197,
+            0.03773200139403343,
+            -0.010408828034996986,
+            0.003963331691920757,
+            -0.005847221240401268,
+            0.007121218368411064,
+            -0.020864123478531837,
+            0.008302279748022556,
+            0.005336073227226734,
+            0.02160761132836342,
+            -0.0056497324258089066,
+            -0.0014482521219179034,
+            0.02328045852482319,
+            0.008859895169734955,
+            0.03469609469175339,
+            -0.01566745527088642,
+            -0.004418330732733011,
+            0.01048627495765686,
+            0.0207402091473341,
+            -0.00580462533980608,
+            -0.008976065553724766,
+            0.007903429679572582,
+            0.0035586729645729065,
+            0.021824462339282036,
+            -0.00515407370403409,
+            -0.0006650729919783771,
+            0.0054367538541555405,
+            -0.0060911779291927814,
+            0.009735042229294777,
+            0.011252996511757374,
+            -0.009076746180653572,
+            0.007539430167526007,
+            0.021530164405703545,
+            -0.009975126944482327,
+            -0.008658533915877342,
+            -0.009649851359426975,
+            0.0010658593382686377,
+            -0.03068435750901699,
+            0.001649613375775516,
+            0.005359307397156954,
+            0.015187285840511322,
+            -0.0004603234410751611,
+            -0.009014789015054703,
+            0.021019017323851585,
+            0.017255110666155815,
+            -0.005936284549534321,
+            -0.0028132491279393435,
+            0.0023795480374246836,
+            -0.0017154429806396365,
+            -0.006160879973322153,
+            -0.0006248975405469537,
+            -0.002656419761478901,
+            0.014730350114405155,
+            -0.0015469965292140841,
+            -0.006617815233767033,
+            -0.00756653631106019,
+            0.0033302053343504667,
+            0.010834784246981144,
+            -0.005661349277943373,
+            0.013057502917945385,
+            0.016805920749902725,
+            -0.00031704717548564076,
+            0.019671445712447166,
+            0.01037010457366705,
+            -0.010447551496326923,
+            0.015938518568873405,
+            -0.007984748110175133,
+            0.006563602481037378,
+            0.0164031982421875,
+            -0.01277095079421997,
+            -0.01246116403490305,
+            0.007210282143205404,
+            0.00665653869509697,
+            0.028918573632836342,
+            0.014792308211326599,
+            -0.02109646424651146,
+            0.014908477663993835,
+            0.015768134966492653,
+            -0.013018779456615448,
+            0.003537375247105956,
+            -0.01784370467066765,
+            0.020260039716959,
+            0.019051872193813324,
+            -0.015535795129835606,
+            -0.017224133014678955,
+            -0.014947201125323772,
+            -0.005092116538435221,
+            0.006331262644380331,
+            0.0033263331279158592,
+            -0.00617636926472187,
+            -0.027989214286208153,
+            -0.0049139889888465405,
+            -0.018633661791682243,
+            0.013870692811906338,
+            -0.008674023672938347,
+            0.0007715621031820774,
+            -0.004511266481131315,
+            0.007984748110175133,
+            -0.0033882902935147285,
+            0.032465629279613495,
+            -0.0051811798475682735,
+            -0.02832997962832451,
+            0.004398969002068043,
+            0.016356728971004486,
+            0.012623801827430725,
+            -0.029243851080536842,
+            0.010463040322065353,
+            0.004805563949048519,
+            -0.012530866079032421,
+            0.005990497302263975,
+            -0.02258343994617462,
+            -0.007632365915924311,
+            -0.0009395244996994734,
+            -0.016743963584303856,
+            -0.027121812105178833,
+            -0.008433938957750797,
+            -0.006993431597948074,
+            -0.005401902832090855,
+            -0.009525936096906662,
+            -0.008720491081476212,
+            -0.011477591469883919,
+            0.02303262986242771,
+            -0.007899557240307331,
+            -0.001866463921032846,
+            -0.0014637414133176208,
+            0.00930908601731062,
+            -0.023776117712259293,
+            -0.015605497173964977,
+            0.006389347370713949,
+            -0.002594462363049388,
+            0.022815778851509094,
+            -0.012747716158628464,
+            0.005940156988799572,
+            -0.009316830895841122,
+            0.009851212613284588,
+            0.015241498127579689,
+            0.004956584889441729,
+            0.017440982162952423,
+            -0.009889936074614525,
+            0.0021026763133704662,
+            -0.008627555333077908,
+            -0.00018986526993103325,
+            -1.027075541060185e-05,
+            0.014428308233618736,
+            0.007624621503055096,
+            0.014141756109893322,
+            0.017487451434135437,
+            -0.013553161174058914,
+            0.005003052763640881,
+            -0.0003274540649726987,
+            -0.017255110666155815,
+            0.0027725896798074245,
+            -0.007984748110175133,
+            -0.0008896682411432266,
+            0.001758038648404181,
+            6.395397940650582e-05,
+            -0.02112744189798832,
+            -0.0009124181815423071,
+            0.0018809852190315723,
+            0.0010222955606877804,
+            -0.025201136246323586,
+            0.00889087375253439,
+            -0.00889087375253439,
+            -0.0014918158994987607,
+            -0.019113829359412193,
+            0.019795360043644905,
+            0.026300877332687378,
+            -0.004619691986590624,
+            0.00036206303047947586,
+            -0.013537672348320484,
+            -0.02049238048493862,
+            0.0001489637652412057,
+            -0.011314953677356243,
+            -0.011345932260155678,
+            -0.0022169100120663643,
+            -0.003214035416021943,
+            0.008805682882666588,
+            0.0004189379105810076,
+            -0.009433000348508358,
+            -0.004956584889441729,
+            -0.011764144524931908,
+            -0.017549408599734306,
+            -0.010401083156466484,
+            0.01933068037033081,
+            -0.0018016024259850383,
+            0.0020794421434402466,
+            -0.0032508226577192545,
+            0.0005271211848594248,
+            0.011748654767870903,
+            0.013475715182721615,
+            0.013344055972993374,
+            -0.023915521800518036,
+            -0.017890173941850662,
+            -0.013320821337401867,
+            0.0029487807769328356,
+            -0.007187047973275185,
+            0.004844287410378456,
+            0.026362836360931396,
+            -0.0009695350890979171,
+            -0.01091997604817152,
+            0.002327271504327655,
+            0.019051872193813324,
+            -0.031272951513528824,
+            -0.013390523381531239,
+            -0.025077221915125847,
+            0.02225816436111927,
+            -0.013351799920201302,
+            0.011516314931213856,
+            0.010191977955400944,
+            -0.025944624096155167,
+            0.006768836174160242,
+            -0.007686578668653965,
+            0.026037558913230896,
+            0.01634124107658863,
+            0.0005934348446317017,
+            -0.00944074522703886,
+            0.013622863218188286,
+            -0.0008722427301108837,
+            -0.013576395809650421,
+            0.002201420720666647,
+            0.014645159244537354,
+            -0.014691627584397793,
+            -0.009510447271168232,
+            -0.004507394507527351,
+            0.004313777666538954,
+            0.012763205915689468,
+            -0.0030339721124619246,
+            -0.004031097516417503,
+            -0.011624740436673164,
+            0.013344055972993374,
+            -0.003930416889488697,
+            0.012383717112243176,
+            0.007380664348602295,
+            -0.014056564308702946,
+            -0.019532041624188423,
+            0.033333033323287964,
+            0.00400399137288332,
+            0.00338054564781487,
+            -0.010997422970831394,
+            0.02740062028169632,
+            -0.009897680021822453,
+            0.010571465827524662,
+            -0.006408709101378918,
+            0.0031985461246222258,
+            0.012321759946644306,
+            -0.012104909867048264,
+            -0.01981084980070591,
+            0.021468207240104675,
+            -0.00364192808046937,
+            -0.0010193913476541638,
+            0.008310024626553059,
+            -0.020182594656944275,
+            -0.00663330452516675,
+            0.002180122770369053,
+            0.009541425853967667,
+            -0.011818356812000275,
+            0.006369986105710268,
+            0.017905661836266518,
+            0.003912991378456354,
+            -0.0007672057254239917,
+            0.019996721297502518,
+            -0.01672847382724285,
+            0.011717676185071468,
+            -0.010850274004042149,
+            0.0016825281782075763,
+            0.0006065039779059589,
+            -0.0048597767017781734,
+            0.0058743273839354515,
+            -0.014536733739078045,
+            0.011988739483058453,
+            -6.244135147426277e-05,
+            -0.013979118317365646,
+            0.007601387333124876,
+            -0.013359544798731804,
+            -0.016836898401379585,
+            0.041542377322912216,
+            -0.005301222205162048,
+            -0.0038665232714265585,
+            -0.0056962003000080585,
+            0.01350669376552105,
+            0.02823704481124878,
+            -0.005336073227226734,
+            -0.01153180468827486,
+            -0.010780571959912777,
+            -0.016573579981923103,
+            0.015907539054751396,
+            -0.01810702309012413,
+            0.002495717955753207,
+            -0.010532742366194725,
+            -0.007903429679572582,
+            0.004302160814404488,
+            -0.003516077296808362,
+            0.014722606167197227,
+            0.00950270239263773,
+            0.023357905447483063,
+            0.00797700323164463,
+            -0.00027348342700861394,
+            0.010393339209258556,
+            0.011663463898003101,
+            0.003243078012019396,
+            -0.010509508661925793,
+            -0.00011949774489039555,
+            -0.02894955314695835,
+            -0.008449427783489227,
+            -0.004863648675382137,
+            -0.001485039247199893,
+            -0.00428667152300477,
+            -0.01239920686930418,
+            0.006617815233767033,
+            -0.0022052929271012545,
+            0.013909416273236275,
+            -0.009254872798919678,
+            -0.017766257748007774,
+            0.03389064967632294,
+            0.004495777189731598,
+            0.006466794293373823,
+            -0.008286789990961552,
+            -0.03705047070980072,
+            -0.0032566310837864876,
+            -0.008116407319903374,
+            -0.013529927469789982,
+            0.004538373090326786,
+            0.01283290795981884,
+            -0.014831030741333961,
+            -0.028097640722990036,
+            0.0019390701781958342,
+            -0.006683644838631153,
+            0.02978597767651081,
+            -0.007798876613378525,
+            0.00419760774821043,
+            0.0037251831963658333,
+            0.0011558911064639688,
+            0.01239920686930418,
+            -0.00845717266201973,
+            -0.0022536972537636757,
+            0.016387708485126495,
+            -0.010858018882572651,
+            0.011051635257899761,
+            -0.010966443456709385,
+            -0.02554190158843994,
+            -0.01456771232187748,
+            0.021344292908906937,
+            -0.0004823473282158375,
+            -0.00024903935263864696,
+            0.014823286794126034,
+            0.02354377694427967,
+            0.011260741390287876,
+            0.00969631876796484,
+            0.006408709101378918,
+            -0.001035848748870194,
+            0.005975008010864258,
+            0.006095050368458033,
+            -0.0024163350462913513,
+            -0.013398268260061741,
+            0.017564896494150162,
+            0.009773765690624714,
+            -0.01810702309012413,
+            0.0008514289511367679,
+            0.0036612898111343384,
+            0.0168833676725626,
+            0.006935346405953169,
+            0.009316830895841122,
+            -0.0143121387809515,
+            -0.023760627955198288,
+            -0.00014932679187040776,
+            0.002873270306736231,
+            0.015721667557954788,
+            0.010579210706055164,
+            -0.014319883659482002,
+            0.0005963391158729792,
+            0.008712747134268284,
+            0.020213572308421135,
+            -0.024209819734096527,
+            0.019702425226569176,
+            -0.004673904739320278,
+            -0.0021084847394376993,
+            0.007946024648845196,
+            0.011632485315203667,
+            -0.011872569099068642,
+            -0.00650551775470376,
+            0.006249943748116493,
+            0.003157886676490307,
+            0.03172214329242706,
+            -0.0017260919557884336,
+            0.009665340185165405,
+            -0.010997422970831394,
+            -0.0002405686245765537,
+            0.0006728176376782358,
+            -0.0006820144481025636,
+            -0.002884887158870697,
+            -0.011020656675100327,
+            0.0018974426202476025,
+            -0.020306508988142014,
+            0.0024473138619214296,
+            0.00463130883872509,
+            -0.005684582982212305,
+            -0.0032972905319184065,
+            -0.019222255796194077,
+            -0.0022672503255307674,
+            0.003293418325483799,
+            0.014335372485220432,
+            -0.031691163778305054,
+            0.004972074180841446,
+            0.010672146454453468,
+            -0.002309845993295312,
+            -0.010416572913527489,
+            0.011423379182815552,
+            -0.0026060794480144978,
+            0.011988739483058453,
+            -0.0038142469711601734,
+            -0.0004905760288238525,
+            0.025681305676698685,
+            -0.005308967083692551,
+            0.014707116410136223,
+            -0.01026167906820774,
+            -0.002993312431499362,
+            -0.028252532705664635,
+            0.005235392600297928,
+            -0.007403898518532515,
+            -0.0012081675231456757,
+            -0.004402841441333294,
+            -0.01130720879882574,
+            -0.0026990154292434454,
+            0.002122037811204791,
+            0.008000237867236137,
+            0.012910354882478714,
+            -0.0007725302129983902,
+            -0.010075807571411133,
+            0.004832670092582703,
+            0.0075355577282607555,
+            -0.015520306304097176,
+            -0.0070437719114124775,
+            -0.020306508988142014,
+            0.029662061482667923,
+            -0.013731288723647594,
+            -0.0011123273288831115,
+            -0.020864123478531837,
+            0.010160998441278934,
+            -0.0063119009137153625,
+            -0.002782270545139909,
+            0.004708755761384964,
+            -0.015109838917851448,
+            0.01344473659992218,
+            0.008751469664275646,
+            0.01450575515627861,
+            -0.02151467651128769,
+            -0.01762685552239418,
+            -0.0044725434854626656,
+            -0.002735802438110113,
+            -0.010648912750184536,
+            -0.0005334137240424752,
+            0.002013612538576126,
+            0.021251358091831207,
+            -0.00032503384863957763,
+            0.0042053526267409325,
+            -0.003214035416021943,
+            0.017224133014678955,
+            -0.010137764737010002,
+            -0.013057502917945385,
+            0.0024260159116238356,
+            0.005994369741529226,
+            0.016186347231268883,
+            0.03158273920416832,
+            -0.020693741738796234,
+            -0.00272999401204288,
+            0.0032740565948188305,
+            0.010656657628715038,
+            -0.004085310269147158,
+            6.304640555754304e-05,
+            -0.027803342789411545,
+            -0.04464024305343628,
+            -0.002422143705189228,
+            0.013057502917945385,
+            -0.015636475756764412,
+            0.012670270167291164,
+            0.008743725717067719,
+            0.010021595284342766,
+            -0.005196669604629278,
+            0.02620794251561165,
+            -0.004793947096914053,
+            0.014304393902420998,
+            0.011733165942132473,
+            0.011911292560398579,
+            0.0009874446550384164,
+            0.0007459079497493804,
+            -0.018354853615164757,
+            -0.008255811408162117,
+            -0.0015954007394611835,
+            0.01765783317387104,
+            -0.027478067204356194,
+            0.008526874706149101,
+            -0.011516314931213856,
+            0.013839714229106903,
+            0.0003666614356916398,
+            0.005661349277943373,
+            -0.01771979033946991,
+            0.012445674277842045,
+            -0.0009337160154245794,
+            -0.025325050577521324,
+            0.004805563949048519,
+            0.011276230216026306,
+            -0.010362359695136547,
+            0.006877261213958263,
+            0.019950253888964653,
+            0.0023079097736626863,
+            0.008495896123349667,
+            -0.014126266352832317,
+            0.010703125037252903,
+            0.009649851359426975,
+            -0.021080974489450455,
+            -0.01233724970370531,
+            -0.007059261202812195,
+            -0.006335135083645582,
+            -0.0035470561124384403,
+            -0.00013105422840453684,
+            0.010176488198339939,
+            -0.016000475734472275,
+            0.010099041275680065,
+            -0.006559730041772127,
+            0.006400964688509703,
+            0.00944074522703886,
+            0.001115231541916728,
+            -0.013576395809650421,
+            0.012716737575829029,
+            -0.0218709297478199,
+            0.008426194079220295,
+            -0.04287445917725563,
+            -0.005847221240401268,
+            -0.015078860335052013,
+            0.005204414017498493,
+            -0.003798757679760456,
+            -0.004453181754797697,
+            -0.006203475873917341,
+            -0.005061137955635786,
+            0.00920066051185131,
+            0.021747015416622162,
+            0.012135888449847698,
+            -0.016620049253106117,
+            -0.007551047019660473,
+            0.005425137002021074,
+            0.0004854935978073627,
+            -0.013336311094462872,
+            0.009316830895841122,
+            -0.01576039008796215,
+            0.0009259713697247207,
+            0.006339007057249546,
+            -0.01072635967284441,
+            0.013452480547130108,
+            0.004731989465653896,
+            -0.003401843598112464,
+            0.013390523381531239,
+            -0.0050417762249708176,
+            -0.0057891360484063625,
+            -0.007272239308804274,
+            -0.013739033602178097,
+            -0.003818119177594781,
+            -0.003072695340961218,
+            0.012747716158628464,
+            0.001529571134597063,
+            -0.0003243077953811735,
+            -0.00782985519617796,
+            0.009154192171990871,
+            0.002220782218500972,
+            -0.0013146566925570369,
+            0.005773646757006645,
+            -0.02300165221095085,
+            0.0001287550403503701,
+            0.004755223635584116,
+            -0.0009487213101238012,
+            -0.006246071308851242,
+            -0.012151377275586128,
+            -0.011903548613190651,
+            -0.038165703415870667,
+            0.009363298304378986,
+            0.0019197085639461875,
+            -0.0076207490637898445,
+            0.0064822835847735405,
+            0.017255110666155815,
+            0.01456771232187748,
+            0.016743963584303856,
+            -0.020693741738796234,
+            -0.02303262986242771,
+            0.005556796211749315,
+            -0.009402021765708923,
+            0.02261441759765148,
+            0.004046586807817221,
+            -0.005847221240401268,
+            0.0013243375578895211,
+            -0.013258864171802998,
+            0.008929597213864326,
+            0.005723306443542242,
+            -0.001592496526427567,
+            0.005425137002021074,
+            -0.013692565262317657,
+            -0.010904486291110516,
+            0.01048627495765686,
+            0.032496608793735504,
+            -0.0004261985304765403,
+            0.0024105266202241182,
+            0.023295948281884193,
+            0.015466093085706234,
+            0.016356728971004486,
+            -0.021111954003572464,
+            0.011167804710566998,
+            0.010060317814350128,
+            -0.0018645278178155422,
+            -0.014892988838255405,
+            0.00513083953410387,
+            -0.012716737575829029,
+            -0.010664401575922966,
+            0.014993669465184212,
+            0.007853088900446892,
+            0.017735280096530914,
+            -0.02369867078959942,
+            0.00789568480104208,
+            0.00042765066609717906,
+            0.00013710475468542427,
+            0.011524059809744358,
+            0.012956822291016579,
+            0.03370477631688118,
+            0.027152791619300842,
+            0.020662762224674225,
+            -0.009285852313041687,
+            0.014366351068019867,
+            -0.009192915633320808,
+            -0.011090358719229698,
+            -0.019470084458589554,
+            -0.015249243006110191,
+            -0.011136826127767563,
+            0.017998598515987396,
+            0.008790193125605583,
+            -0.009355553425848484,
+            -0.005947901867330074,
+            0.01586107164621353,
+            -0.003909118939191103,
+            -0.004635181277990341,
+            0.01646515540778637,
+            -0.026269899681210518,
+            0.016836898401379585,
+            -0.000745423894841224,
+            -0.014668392948806286,
+            -0.014691627584397793,
+            -0.004182118456810713,
+            -0.012670270167291164,
+            0.0003964299685321748,
+            -0.020817656069993973,
+            0.013181418180465698,
+            0.024597052484750748,
+            -0.0030920570716261864,
+            -0.004100799560546875,
+            -0.011299464851617813,
+            -0.0005111477803438902,
+            0.018385831266641617,
+            0.0034773540683090687,
+            0.009192915633320808,
+            -0.0023872926831245422,
+            0.013924905098974705,
+            0.00524313747882843,
+            0.0027958236169070005,
+            -0.0007565568666905165,
+            -0.016263794153928757,
+            0.012197845615446568,
+            0.0026835259050130844,
+            0.012135888449847698,
+            -0.022242674604058266,
+            0.007938280701637268,
+            -0.011764144524931908,
+            -0.0023698671720921993,
+            -0.02228914201259613,
+            0.0189744271337986,
+            -0.01610890030860901,
+            0.001046497724018991,
+            0.003841353114694357,
+            0.0006224773242138326,
+            0.009580149315297604,
+            0.01610890030860901,
+            -0.023388884961605072,
+            -0.0003460896550677717,
+            -0.014908477663993835,
+            -0.021917399019002914,
+            -0.01607792265713215,
+            0.004728117026388645,
+            -0.019501063972711563,
+            0.005444498732686043,
+            -0.0025557391345500946,
+            -0.015125328674912453,
+            0.018478767946362495,
+            0.014691627584397793,
+            -0.004379607271403074,
+            0.002412462839856744,
+            0.005425137002021074,
+            -0.0014405074762180448,
+            0.01971791312098503,
+            0.00013274837692733854,
+            -0.0010048701660707593,
+            -0.009425255469977856,
+            -0.0168833676725626,
+            -0.0065752193331718445,
+            -0.035005878657102585,
+            0.0260530486702919,
+            -0.004670032300055027,
+            -0.016511622816324234,
+            0.014319883659482002,
+            0.009115469641983509,
+            -0.014629669487476349,
+            -0.007059261202812195,
+            0.016294771805405617,
+            -0.024287264794111252,
+            0.02244403585791588,
+            -0.009611127898097038,
+            -0.008774704299867153,
+            0.00784921646118164,
+            0.05154848098754883,
+            0.009146448224782944,
+            0.003492843359708786,
+            0.011036145500838757,
+            0.005839476361870766,
+            -0.0013165927957743406,
+            -0.023683181032538414,
+            0.005568413063883781,
+            -0.007190920412540436,
+            -0.003601268632337451,
+            0.002536377403885126,
+            -0.004801691509783268,
+            0.031149037182331085,
+            -0.004379607271403074,
+            -0.005289605353027582,
+            0.004484160337597132,
+            -0.02196386642754078,
+            -0.0001332324172835797,
+            0.023203013464808464,
+            -2.3128106477088295e-05,
+            -0.023590246215462685,
+            -0.006571347359567881,
+            -0.015071115456521511,
+            -0.013770012184977531,
+            -0.018989915028214455,
+            0.005231520161032677,
+            -0.012755461037158966,
+            -0.0033940989524126053,
+            0.0016660707769915462,
+            -0.005940156988799572,
+            -0.002582845278084278,
+            0.009603383019566536,
+            0.05362405255436897,
+            0.014103032648563385,
+            -0.006335135083645582,
+            -0.005854965653270483,
+            0.009192915633320808,
+            -0.010524998418986797,
+            -0.019795360043644905,
+            -0.009820234030485153,
+            -0.029507169499993324,
+            -0.012747716158628464,
+            -0.0143121387809515,
+            -0.010191977955400944,
+            -0.010277168825268745,
+            0.00963436160236597,
+            -0.018122512847185135,
+            -0.007628493942320347,
+            -0.020337486639618874,
+            0.002941036131232977,
+            -0.011268485337495804,
+            -0.014714861288666725,
+            -0.008054450154304504,
+            -0.04807887226343155,
+            0.005796880926936865,
+            0.019593998789787292,
+            0.01263929158449173,
+            0.002515079453587532,
+            0.004383479710668325,
+            -0.01993476413190365,
+            -0.030792783945798874,
+            0.013839714229106903,
+            0.011283975094556808,
+            0.011733165942132473,
+            0.03283737599849701,
+            -0.011709931306540966,
+            -0.00813964195549488,
+            -0.004693266469985247,
+            -0.0017251238459721208,
+            0.012066186405718327,
+            0.00969631876796484,
+            -0.004917861428111792,
+            -0.012375972233712673,
+            0.016217326745390892,
+            -0.01981084980070591,
+            -0.023683181032538414,
+            -0.01672847382724285,
+            0.006157007534056902,
+            0.00813964195549488,
+            -0.008054450154304504,
+            0.002973950933665037,
+            0.0056962003000080585,
+            -0.007159941829741001,
+            -0.00022423222253564745,
+            0.004890755284577608,
+            -0.023048119619488716,
+            0.0008490087930113077,
+            -0.002257569460198283,
+            0.0061376458033919334,
+            0.021266845986247063,
+            -0.010168743319809437,
+            -0.0014075926737859845,
+            -0.0053902859799563885,
+            -0.0028984404634684324,
+            -0.006764963734894991,
+            0.02620794251561165,
+            0.009154192171990871,
+            -0.009882191196084023,
+            -0.01011453103274107,
+            -0.01129171997308731,
+            -0.0011278166202828288,
+            -0.017255110666155815,
+            -0.016186347231268883,
+            0.0015566773945465684,
+            -0.016712984070181847,
+            0.013971373438835144,
+            -0.00876695942133665,
+            0.00400399137288332,
+            -0.014110777527093887,
+            0.02019808255136013,
+            0.01929970271885395,
+            -0.009711808525025845,
+            0.023776117712259293,
+            -0.007709812838584185,
+            -0.038382552564144135,
+            -0.00800798274576664,
+            0.01843230053782463,
+            -0.006118284538388252,
+            0.010037084110081196,
+            -0.011314953677356243,
+            0.008209343999624252,
+            0.00015319912927225232,
+            0.0018122512847185135,
+            -0.0003112386621069163,
+            0.027555514127016068,
+            0.015535795129835606,
+            -0.005560668651014566,
+            -0.005494839046150446,
+            0.004255692940205336,
+            0.009541425853967667,
+            0.017580386251211166,
+            0.012151377275586128,
+            -0.021019017323851585,
+            -0.014257925562560558,
+            -0.016263794153928757,
+            0.011012911796569824,
+            0.015125328674912453,
+            -0.0040891822427511215,
+            0.01691434532403946,
+            0.014126266352832317,
+            -0.019098341464996338,
+            0.0019884423818439245,
+            -0.008193854242563248,
+            -0.00158184755127877,
+            -0.03029712475836277,
+            0.011043890379369259,
+            0.0074387495405972,
+            -0.0072218989953398705,
+            0.036616768687963486,
+            0.009858956560492516,
+            0.037577107548713684,
+            0.018680129200220108,
+            0.0008974128868430853,
+            0.009045767597854137,
+            -0.018370341509580612,
+            0.013065247796475887,
+            0.01500141341239214,
+            -0.012724482454359531,
+            -0.005734923295676708,
+            0.0013446672819554806,
+            0.0007551047019660473,
+            0.012631546705961227,
+            0.02004319056868553,
+            -0.01795213110744953,
+            0.009773765690624714,
+            -0.015032391995191574,
+            -0.005250881891697645,
+            0.006555858068168163,
+            -0.013932649977505207,
+            -0.008495896123349667,
+            -0.00963436160236597,
+            0.019408127292990685,
+            -0.012600568123161793,
+            0.016573579981923103,
+            -0.009595638141036034,
+            0.015520306304097176,
+            -0.029724018648266792,
+            -0.003669034456834197,
+            -0.006679772399365902,
+            -0.019593998789787292,
+            -0.012887120246887207,
+            0.009820234030485153,
+            0.010633422993123531,
+            -0.009301341138780117,
+            -0.011562783271074295,
+            0.020368466153740883,
+            -0.004185990896075964,
+            0.013529927469789982,
+            0.021189400926232338,
+            0.00737291993573308,
+            0.024132372811436653,
+            0.0005605200421996415,
+            0.014164989814162254,
+            0.005463860463351011,
+            0.010362359695136547,
+            -0.004677776712924242,
+            0.005161818582564592,
+            0.00932457484304905,
+            0.014203713275492191,
+            0.017766257748007774,
+            0.013196907006204128,
+            -0.00598275288939476,
+            0.007419387809932232,
+            0.02016710489988327,
+            0.0016254113288596272,
+            -0.030312614515423775,
+            -0.009719553403556347,
+            0.0001736498816171661,
+            0.00924712885171175,
+            0.009185170754790306,
+            3.817877222900279e-05,
+            -0.01093546487390995,
+            0.011144571006298065,
+            0.00969631876796484,
+            0.002286612056195736,
+            0.006280922330915928,
+            -0.01456771232187748,
+            -0.020957060158252716,
+            0.006106667220592499,
+            0.01798310875892639,
+            0.013917161151766777,
+            -0.014722606167197227,
+            0.022784801200032234,
+            0.006145390681922436,
+            -0.021406250074505806,
+            -0.01840132102370262,
+            -0.0009559819591231644,
+            0.0004542729293461889,
+            -0.002265314105898142,
+            0.03720536455512047,
+            -0.0011539548868313432,
+            -0.005955646280199289,
+            -0.0054754773154854774,
+            0.02775687538087368,
+            0.02049238048493862,
+            0.009177426807582378
+          ],
+          "embedding": [
+            0.0008215984936325879,
+            -0.019349418284688353,
+            -0.02322660633029795,
+            0.009747820845194852,
+            0.007353771220903641,
+            0.015252908783199839,
+            -0.012176175829980202,
+            0.051898051094966605,
+            -0.0068130353886146165,
+            0.008189212816890595,
+            0.008102287831931905,
+            0.01682801136011614,
+            0.02005557996049711,
+            -0.020779879580497183,
+            0.002332165268329781,
+            0.02337485065616377,
+            0.006074992895540845,
+            0.02576575146535689,
+            0.0007517286429767504,
+            -0.03763909942314134,
+            0.0011204729720314833,
+            0.00041212613589632806,
+            -0.06089176800314795,
+            0.012852035873395388,
+            -0.008901127092058543,
+            0.01702347055484216,
+            0.005803446395466281,
+            0.017889570275475296,
+            -0.05375047382413552,
+            0.021623778774252724,
+            0.014784317465883424,
+            -0.0018881880857885708,
+            0.0004735328554290247,
+            0.012393802850407656,
+            -0.019234442968055952,
+            -0.019566283611638122,
+            0.04407789373296535,
+            -0.0053565873209102995,
+            -0.026240866282480922,
+            0.048107902270426675,
+            -0.0054681314939387235,
+            -0.0009441397948041965,
+            -0.015205079626883548,
+            0.006885981841893229,
+            0.01403724256866746,
+            -0.026086548246792754,
+            0.0034778946051893067,
+            0.00044587533948775473,
+            0.032851328435891634,
+            -0.03882097333255334,
+            -0.011126344604254894,
+            -0.01648943336384904,
+            0.035810456555802166,
+            -0.019384589526165754,
+            -0.019430702768170285,
+            0.014297440272239852,
+            -0.007883453204663151,
+            0.050996900793086844,
+            0.06007149434153715,
+            0.03406832053483379,
+            0.025193439934671852,
+            0.00027785129663604897,
+            -0.008014391056168083,
+            -0.018806730908490114,
+            0.027193788903098558,
+            -0.006195006670328595,
+            -0.020466000344352826,
+            0.0068939884219589185,
+            0.017252276167086602,
+            -0.045073444648704136,
+            -0.049242269917908754,
+            0.023021140866008313,
+            -0.015676872472071106,
+            -0.024497517908898602,
+            0.005873633718973869,
+            -0.022828585938203683,
+            0.011748341004940436,
+            0.04624205727927451,
+            -0.015442165499674641,
+            0.018565551071491767,
+            0.015246505395507096,
+            0.004119269796224144,
+            0.004862735250819252,
+            -0.000145323321960046,
+            0.05155838952887077,
+            0.0027052423345323936,
+            -0.016218063670586664,
+            -0.07462470920944148,
+            0.043695974746295135,
+            -0.027608273720377708,
+            -0.03860912774695494,
+            0.002823110565150435,
+            0.026471469849508766,
+            -0.0026063577189074103,
+            -0.029230652014151735,
+            0.05684164470477843,
+            0.00022182868300572677,
+            0.015590845552956364,
+            -0.024939617549529242,
+            0.01035803286203917,
+            -0.010225298012384327,
+            0.019571847648759157,
+            0.01841613041441054,
+            0.0156981470538354,
+            0.021764297694793343,
+            -0.018831230007033416,
+            -0.014419529638789993,
+            -0.0029965475899652273,
+            0.028881185012095884,
+            0.09163526460163937,
+            0.022644302207811818,
+            -0.017933909936330133,
+            -0.04663928709016869,
+            0.004962951444105128,
+            -0.041333041174526026,
+            -0.027545628007617527,
+            -0.037142394579894954,
+            -0.013358143162788402,
+            0.02719686432396677,
+            -0.0034971800937504205,
+            0.04413635817371732,
+            0.04903857110328917,
+            0.03186067155630788,
+            0.007831051050269414,
+            -0.025257538623313574,
+            0.023517513773109414,
+            -0.0016527403209505964,
+            -0.0016101860707429347,
+            0.014448515185903624,
+            0.009905018312477684,
+            0.0024114582235163955,
+            0.00172043851384822,
+            -0.03818316221102173,
+            -0.02304537008110578,
+            0.01024969233570089,
+            -0.01901116356327339,
+            0.019791121460848986,
+            0.07985067340821365,
+            -0.051903191272236665,
+            -0.025924646976556942,
+            0.014766511532849252,
+            -0.002073086076894999,
+            0.030959269926800852,
+            0.006394417986617089,
+            -0.028177586693713442,
+            0.010725285336674526,
+            0.0009614052276924192,
+            0.07500253788978564,
+            -0.00036794182149960157,
+            0.004058458332826519,
+            -0.013218269166951234,
+            0.027613539268683245,
+            -0.012995170442480582,
+            0.023388707018488407,
+            0.026389423313105165,
+            -0.003068639956061266,
+            -0.02661967649632026,
+            0.026145274570635913,
+            -0.03300591322174331,
+            -0.004556072066733024,
+            0.020045722084112896,
+            0.0033437343914784865,
+            -0.010224220905483085,
+            -0.0025025498684079703,
+            0.03460103245395426,
+            0.04514362357278504,
+            0.03797031988058962,
+            -0.004586662475576956,
+            -0.06351723722596923,
+            0.06037983484369874,
+            0.019415753124363902,
+            0.028850618647994476,
+            -0.0034810237045849006,
+            0.02957938154808314,
+            0.02642652523231673,
+            0.013187653724133275,
+            -0.007856320804109569,
+            -0.007065882659482447,
+            -0.016672784625197085,
+            0.029020161154209707,
+            -0.013243058921970134,
+            -0.014900932834241732,
+            0.006001769901896033,
+            0.008648456186944271,
+            0.03484763413464497,
+            0.012279368984306564,
+            0.02379643508687133,
+            0.04933044959399941,
+            -0.021047723935647383,
+            -0.007264823701706426,
+            0.0007058774795716792,
+            -0.010512071832195085,
+            0.027489671668749157,
+            0.009731157570361245,
+            -0.03195919640758579,
+            -0.0033626031972691915,
+            -0.04282419484651207,
+            0.01873977910485268,
+            -0.014398285316695653,
+            0.012854631447494207,
+            -0.0017372959978092432,
+            -0.037331867825735776,
+            0.001088209292238142,
+            -0.012345684323522151,
+            0.037096063326414734,
+            0.005050219690024652,
+            0.0008906051515353004,
+            -0.003697709449630797,
+            0.006548479142737107,
+            -0.010727663712299397,
+            -0.007486226350610755,
+            -0.02704618481069485,
+            -0.019752552016312926,
+            -0.014632581296303486,
+            -0.0037501138119724717,
+            0.014588411795703245,
+            -0.007119137071719376,
+            0.016440258608998535,
+            -0.03686645352111013,
+            -0.023546402448713415,
+            0.005628703432152666,
+            0.024988145751166143,
+            0.011971199595808573,
+            -0.010868778329190568,
+            -0.03270046433624425,
+            -0.021950559285357267,
+            -0.011889914022657738,
+            -0.005123971523701009,
+            0.02619713302280345,
+            -0.021990810305904974,
+            0.01165257474485436,
+            -0.009117901804315863,
+            -0.007492082755249399,
+            0.010725902070291232,
+            0.01862811996181951,
+            -0.026694281868322685,
+            -0.01623776415010199,
+            -0.009403997538684658,
+            -0.02207329027097839,
+            0.025939540497196416,
+            0.014178803271471762,
+            -0.001141821318145744,
+            -0.006281187017652108,
+            -0.035993365198554964,
+            0.0026192776111449633,
+            -0.015254976771874396,
+            -0.0116872453009606,
+            -0.015780016491993135,
+            0.021421328283124968,
+            0.02889432250124136,
+            -0.01871009416296692,
+            -0.026540472640514536,
+            0.015703370682110186,
+            -0.0002849235245164438,
+            -0.02594930889995505,
+            0.04491828744809679,
+            0.044063522197410734,
+            -0.037441864491547405,
+            -0.007005339187980476,
+            -0.030381794393931814,
+            0.06901083845539985,
+            -0.017604082400603037,
+            -0.04060591381445593,
+            0.06044850935418443,
+            -0.01756490171206357,
+            0.025539717592705295,
+            0.03452018789549939,
+            0.0016217577522523927,
+            0.00018825054264059277,
+            -0.002563606092068712,
+            0.04770277039047993,
+            -0.0032010143255574255,
+            -0.03425328601475849,
+            -0.02105909767072799,
+            -0.00840881022563103,
+            -0.028699447677549002,
+            -0.015925160861589006,
+            -0.0055750742281166,
+            0.017513987082479483,
+            0.010399718005952185,
+            0.02267201252273823,
+            0.01258511878601025,
+            0.0015081978484714903,
+            0.01874981572091931,
+            0.011158889421057756,
+            -0.002607810686437211,
+            -0.003732023411827874,
+            -0.007787189972165309,
+            -0.05524012485122728,
+            -0.025110689689279583,
+            -0.03946383236703947,
+            0.004443572529325475,
+            -0.018745077598477586,
+            0.005926160888274727,
+            0.039430331582190524,
+            -0.0049500324516682,
+            0.004352795787899792,
+            0.015566139991780173,
+            0.013148156356408315,
+            -0.015279560337042242,
+            -0.03316846614871165,
+            0.0001592209771257332,
+            0.018362411161743347,
+            -0.012706582418607377,
+            -0.03424477956681259,
+            -0.001991022841226589,
+            0.0033467286047492175,
+            0.03357019284647071,
+            -0.024359110796680374,
+            0.00359117029951049,
+            0.005061818038334802,
+            0.011359582931760315,
+            -0.026793992746244076,
+            -0.018170136079817746,
+            -0.017421603453826397,
+            0.03996608926471779,
+            0.02587013058343799,
+            0.015763410145648683,
+            -0.04512536449983358,
+            -0.02293473890636517,
+            0.048575083783480195,
+            -0.009589860756342604,
+            -0.06486262416944208,
+            0.03447666009581266,
+            0.016105654870253654,
+            0.003526582643722421,
+            -0.01354754623151814,
+            0.05401575248587799,
+            -0.03563190295559527,
+            -0.017365166993914202,
+            -0.0046288709698227735,
+            -0.033657212157212645,
+            -0.007686252059880836,
+            -0.001385273642139782,
+            0.048229062789530236,
+            -0.029083855744728694,
+            -0.030465559177859865,
+            0.0024368661249980416,
+            -0.024144031154493057,
+            0.021693278579520123,
+            -0.016405088055634124,
+            -0.018439935250344958,
+            0.006948603359995389,
+            -0.033943037395199695,
+            -0.003007796423804453,
+            -0.016783195426536633,
+            -0.02270589491206155,
+            0.020874438753828814,
+            0.016724666903417947,
+            0.027033184180344852,
+            0.0019847647852695785,
+            0.017749524526371133,
+            0.0204661893602285,
+            -0.005604929968008821,
+            0.028462234633072944,
+            -0.020790459496727066,
+            -0.04183816355941877,
+            0.03946487892663695,
+            0.01192861758684541,
+            -0.0018024841275380242,
+            0.01077082591611702,
+            -0.01475239020295694,
+            -0.03952356245715099,
+            0.03683950018557526,
+            -0.056767672022551764,
+            -0.007507600090017693,
+            0.009530544352246083,
+            -0.00788856051420074,
+            -0.011204694027562521,
+            -0.010587701466318598,
+            -0.028604940377315504,
+            0.023749537635431462,
+            -0.013672419168235691,
+            0.01953449229707597,
+            -0.011449939491554354,
+            -0.012238180529977605,
+            0.05795536247491174,
+            -0.021518581774137322,
+            -0.040672955795925836,
+            0.026066659238628543,
+            -0.014061106927896723,
+            0.03328357230532424,
+            0.03957107406164427,
+            0.030670947051333917,
+            -0.004278577327978564,
+            0.025885946540364826,
+            0.0319106101668716,
+            -0.003462160036198642,
+            -0.04686868330851868,
+            0.00555303933159587,
+            0.02156143971867743,
+            -0.03739529489075782,
+            -0.01025626126223168,
+            -0.02141970191890338,
+            0.022134910980548407,
+            0.017807484142010496,
+            -0.007232643256393455,
+            0.011318041192005274,
+            -0.010489012862455913,
+            -0.013003217163997554,
+            -0.00847279880392288,
+            0.02357274819910594,
+            -0.0016485066139744577,
+            -0.005873113634992993,
+            0.006424249429163587,
+            0.004008703814301658,
+            0.0054639858216457724,
+            0.010198079500408441,
+            0.017694720520798957,
+            -0.017087021465208255,
+            -0.0025558869389290554,
+            -0.016503579212015177,
+            -0.0420082404922411,
+            -0.003563173424359332,
+            0.022924161700566468,
+            0.012345933179990547,
+            -0.00729908668365632,
+            -0.004234683369642856,
+            -0.0022962545569187934,
+            -0.020666205489143392,
+            -0.014796149043465308,
+            0.024562653223873056,
+            0.030217494971086438,
+            0.007810961433007407,
+            0.007781263640449456,
+            -0.0030348559388780655,
+            0.007310876971216526,
+            0.016099591640649783,
+            0.02311929597780174,
+            -0.04160162038353416,
+            0.027152100845897007,
+            -0.00025396315502635146,
+            0.023717123370023598,
+            -0.04186918759097667,
+            -0.017988531857836703,
+            -0.03106663560912813,
+            -0.023595079699181436,
+            0.02437134274097885,
+            -0.02317559303912261,
+            -0.012327263775257306,
+            0.000989566600374836,
+            0.00954921216572729,
+            -0.005155079395685017,
+            -0.017508404061379164,
+            -0.04582321461886958,
+            -0.0014367264231457263,
+            -0.030417229142190933,
+            -0.017829710557332023,
+            -0.001056472801554354,
+            0.005016912084754242,
+            -0.06276644565618253,
+            0.0109965681725148,
+            -0.07270669271977365,
+            -0.004542796125637244,
+            0.040139998232966566,
+            -0.017890441783630643,
+            0.055909102081208445,
+            -0.05029104308417623,
+            0.01673962668706085,
+            -0.04507293803055318,
+            0.015385506084493484,
+            0.025677372277539627,
+            0.030235171586316813,
+            -0.022189978375852563,
+            0.008461188938299745,
+            0.008033707483259569,
+            -0.045089505755429415,
+            -0.04144698743458133,
+            -0.013640256640978024,
+            0.01036782197518379,
+            0.0012696958402094232,
+            -0.016090728932978384,
+            0.023838437877040163,
+            0.011946482002767135,
+            0.03342604921859589,
+            0.04211801335157251,
+            0.002708015395637621,
+            -0.047156646610976634,
+            -0.013104464812562618,
+            -0.003933116891120681,
+            0.03155321708116329,
+            -0.018866109579446184,
+            0.015376812010630058,
+            -0.01955557376085726,
+            0.016255217383093355,
+            0.017868332102274922,
+            -0.020325179506828335,
+            0.013434364256590097,
+            0.004176904701252156,
+            -0.03877060944639854,
+            0.0006475542749369515,
+            0.00042468300294954125,
+            -0.02116992978038693,
+            0.0032654387271108293,
+            -0.00575003786159731,
+            -0.05149954071159492,
+            -0.031382926943128234,
+            -0.015730563640253065,
+            0.0382312502769352,
+            0.004297939848910064,
+            0.019724983545557365,
+            -0.017907869879607064,
+            -0.005165628493894463,
+            0.03698976707579373,
+            0.008501001373266799,
+            -0.01725527821298066,
+            0.006879620077191753,
+            0.02455792200929138,
+            -0.03224193972301503,
+            -0.039181328884413266,
+            -0.0008206986106407542,
+            -0.023306758893533952,
+            0.01672946960876128,
+            -0.02658587698869307,
+            0.005873620623736886,
+            0.0025331170979801232,
+            -0.03490809296591325,
+            0.02456764314036355,
+            0.003291113462353163,
+            -0.004303423672882112,
+            -0.020359544091059775,
+            -0.016892415165120203,
+            0.017174797487220006,
+            0.04391489511478065,
+            0.007327321684948843,
+            -0.003943713832544148,
+            0.00862684231313155,
+            -0.006361172785753631,
+            0.05826688726534012,
+            -0.02284956969636637,
+            -0.026680023999604643,
+            0.027527436347931256,
+            -0.022597669279069723,
+            -0.006277863878949772,
+            -0.017343446085471857,
+            -0.012065192152203348,
+            -0.0004183712490442143,
+            0.019339311388446943,
+            -0.019423562920897294,
+            -0.010212527447243049,
+            0.004783006556748757,
+            0.012824133870627983,
+            -0.004855945108275706,
+            0.006239949458562236,
+            0.006847581690471813,
+            0.03094005096378827,
+            0.026993131409761086,
+            -0.011331881651539968,
+            0.03297014291876675,
+            0.010265600643742144,
+            4.666113318251278e-05,
+            0.001970879281172325,
+            0.002523664335535017,
+            0.0034107811520368574,
+            -0.03143890864030425,
+            -0.01624584278940617,
+            0.013468729048885906,
+            0.005233309338223403,
+            -0.004550544967856448,
+            -0.023783653541634683,
+            -0.009402494239255264,
+            0.03334923712760375,
+            -0.003260424721959948,
+            0.02748395114281811,
+            -0.0058374897745283065,
+            -0.01985840930273332,
+            0.017595418271610163,
+            0.02881743246631328,
+            -0.05928552440605384,
+            0.017062540894682354,
+            -0.0011508547522925793,
+            0.008953761538519997,
+            0.01610561914042553,
+            -0.008894160513648953,
+            0.013301991014533684,
+            0.04080863883897929,
+            0.010738051773755976,
+            -0.023345002072886145,
+            0.004215726310838674,
+            0.01541749448570608,
+            0.01291813299324752,
+            -0.022747942296528025,
+            0.0008640791861357522,
+            0.011008522066277018,
+            0.005893793007012561,
+            0.0026106450193140156,
+            -0.006278394080583205,
+            -0.025659188287146268,
+            0.024180550333596625,
+            -0.0044491292740074925,
+            0.026967973346698617,
+            0.04390909273429254,
+            0.009596855239692042,
+            -0.03686020608700941,
+            -0.013109951664921729,
+            0.0002799680359670127,
+            -0.0038537577602593366,
+            -0.0037552094562213916,
+            0.011105295862811114,
+            -0.04185615640149602,
+            0.019950890919369135,
+            0.006453241355231516,
+            0.015314645427201533,
+            -0.020031193483734364,
+            -0.024530173488182337,
+            0.0031949828520724337,
+            -0.01492645564652531,
+            0.012875827757040941,
+            0.005057170413681063,
+            -0.004319834710426384,
+            0.005386862713494232,
+            0.00030848054013413157,
+            0.008384941028326983,
+            -0.010811526993301901,
+            0.03175068297908541,
+            0.004680987487229685,
+            0.009264547472281849,
+            -0.02856314241700808,
+            -0.021083721348429348,
+            0.020113396771629096,
+            0.03572366771051171,
+            -0.01708529918264109,
+            0.03346921724975398,
+            -0.009042046660366718,
+            0.0033219155387035227,
+            0.0151720168318272,
+            -0.015805318210244414,
+            -0.008057236707694303,
+            -0.0005483009439015931,
+            -0.002917527004423299,
+            -0.014538511124345527,
+            0.007302979537905532,
+            -0.0208529958115422,
+            -0.02965227242147972,
+            0.020064178921373875,
+            -0.00011892622932349096,
+            0.016455188521316146,
+            0.010335044657747514,
+            -0.007816155992852236,
+            -0.0052262330850905165,
+            0.006959344748883874,
+            0.04432147882994085,
+            0.026734484553994495,
+            0.05547661378166294,
+            -0.005787997952888261,
+            -0.004559444854138438,
+            -0.0082295581123363,
+            -0.006588172755707144,
+            0.04524009609473585,
+            0.015915003261638767,
+            -0.004910285095602864,
+            0.017495607570191165,
+            0.01557886571266685,
+            0.03858259260136037,
+            0.0027648525902558316,
+            -0.016912796336835408,
+            0.04014755669075221,
+            -0.004077405782937414,
+            0.008616827127381418,
+            0.011102692199778469,
+            0.0024185742218798562,
+            0.00807141794538637,
+            0.02996274720815659,
+            -0.012429186019907761,
+            0.01680942269506771,
+            -0.04571916592313758,
+            -0.02775530246832996,
+            0.014370280587419414,
+            -0.004931905540824582,
+            0.03419959665162407,
+            0.01668521021731861,
+            -0.023105305446763272,
+            -0.024761385385514877,
+            0.021529428247380984,
+            -0.002603042335561961,
+            -0.024135326707477325,
+            -0.015906833215844473,
+            0.003286605488908195,
+            -0.0023437997044735617,
+            0.0060251848371355166,
+            -0.013481652076194643,
+            0.0002908426939460218,
+            0.01953362396253743,
+            0.0038544889405965057,
+            0.020760460182873287,
+            -0.016224396121239458,
+            -0.00259233446682179,
+            -0.009282861237653619,
+            -0.02952130765234087,
+            0.017988370145301916,
+            0.006138236653154911,
+            -0.007607182955958238,
+            -0.009516767050028553,
+            0.0048347778733230365,
+            -0.005819209642901871,
+            -0.0025851701689263706,
+            -0.01073014664868326,
+            0.012433465684360061,
+            0.025784014942833097,
+            -0.016875624785552303,
+            0.007944349714123104,
+            8.651589872318479e-05,
+            -0.022306044741692753,
+            -0.022127016745066307,
+            0.020143472307319017,
+            0.002792646249762062,
+            0.008768813346304779,
+            -0.004074214276300631,
+            -0.00015024341785520542,
+            0.002990550758627699,
+            0.01951746553838439,
+            0.03896451381932467,
+            0.004232673242619471,
+            0.027876586543280867,
+            0.0145860303663199,
+            0.03994234200854897,
+            0.009657017763624678,
+            0.00696607158576712,
+            -0.026055387694656488,
+            0.02819890357011778,
+            0.00018297502510753014,
+            0.043190622821775236,
+            0.022488402221902314,
+            -0.0407730101161195,
+            -0.0021946518523759066,
+            0.01669520746276111,
+            0.013212623257794862,
+            0.017009844136059122,
+            -0.004537512994041156,
+            -0.014041213424740049,
+            0.028135404528975894,
+            -0.022859586430516003,
+            0.0008637006862481223,
+            -0.004486332934188017,
+            -0.025031795510001887,
+            0.007351370254175368,
+            -0.004567816916440959,
+            0.030025413528233364,
+            0.0014976740344892735,
+            -0.022285096307825804,
+            0.022837170656498543,
+            -0.009551269792616045,
+            0.011045530488684207,
+            0.008809047185288755,
+            0.020065001354321994,
+            0.031993362733068575,
+            0.009143210434415064,
+            0.02133640814803879,
+            0.012241777474889248,
+            0.014771538127546474,
+            -0.008804192497858033,
+            -0.02341427260922065,
+            0.023167424359504862,
+            -0.0011598649565273595,
+            -0.029627604902846688,
+            0.0021991474725787303,
+            0.003261878079344105,
+            -0.0005935713331682358,
+            -0.009144689470851402,
+            -0.005580557862988033,
+            -0.007240836451875213,
+            0.019631377011298905,
+            -0.007085213886352692,
+            0.007017573363709754,
+            -0.004074171791476129,
+            -0.00822871835665378,
+            -0.0421101573155962,
+            0.004210914075286222,
+            0.003408427700622687,
+            -0.01752179330665445,
+            0.006384440404174512,
+            0.011121481025573962,
+            0.043020575455262985,
+            -0.022809979913698254,
+            -0.0005737138409049003,
+            0.019206623212867476,
+            0.0009562551889820241,
+            -0.023427467810802533,
+            -0.038931688376706186,
+            0.007172345672967499,
+            -0.03467050750542982,
+            0.023185301620612612,
+            -0.0007075596339842658,
+            0.02387977204565927,
+            -0.03043468414449851,
+            0.013909304572370437,
+            0.004784668158421625,
+            0.012524879174804915,
+            0.021654512058639427,
+            0.03142146922870715,
+            -0.02053473475540882,
+            -0.050409862638621725,
+            -0.005406667316025549,
+            0.00035138762584085976,
+            0.022018487016014856,
+            -0.026014580970974616,
+            0.00025633004699302783,
+            -0.003349094027291921,
+            -0.02401366769212778,
+            -0.01735996745614562,
+            0.01962340791446025,
+            -0.050011480783403936,
+            0.015042913576492952,
+            0.0262737424555697,
+            -0.02507473901367252,
+            -0.05408889321034101,
+            -0.018988587082325706,
+            -0.028217893770659686,
+            -0.009285770094319318,
+            -0.00661159230987788,
+            0.028597405154740078,
+            0.003525576256054959,
+            -0.011462219358952466,
+            0.025401668390029348,
+            0.006155837251830549,
+            0.0026050693705924593,
+            0.015985722392086222,
+            0.007289910674212967,
+            -0.011452511453623235,
+            -0.014599577191293152,
+            -0.04558584556100621,
+            0.010197536827229795,
+            -0.0045431962637690155,
+            -0.019372598050508005,
+            -0.030147301333718016,
+            -0.0137828704002156,
+            -0.027060515748528666,
+            -0.0017291966950107505,
+            0.053783684195381654,
+            0.03292494290301388,
+            0.005560172296221419,
+            -0.0026085670565688026,
+            -0.010466890489356092,
+            0.036247483457995895,
+            -0.006789746401727701,
+            0.03662690472838588,
+            0.001784522073210422,
+            -0.0242673253380092,
+            0.01169382884548301,
+            0.02892627181542449,
+            0.009858451662166387,
+            -0.0005592283645355213,
+            -0.006160132931009508,
+            -0.008782830470079547,
+            0.0031207630972208394,
+            0.025613233596604418,
+            0.003268170392795548,
+            -0.010062991425127038,
+            0.018106967407336815,
+            0.004480291699976396,
+            0.02010374270557836,
+            0.03555357587371324,
+            0.023489837757983054,
+            -0.021782173652026497,
+            -0.026581762796336082,
+            -0.0013561486781077014,
+            0.0072266106510934505,
+            0.03551967295942857,
+            -0.01651315514034218,
+            0.0001274396319698452,
+            -0.008272735507120127,
+            -0.00894931703949637,
+            0.011149952898504355,
+            -0.0029671593121986116,
+            -0.0030752010285951576,
+            0.01662711184497578,
+            -0.0011672263615879709,
+            -0.033448222499147764,
+            0.0078288560210653,
+            -0.023496331346917084,
+            0.0051040606404415545,
+            -0.004113104164416575,
+            -0.007076525381595441,
+            -0.002157548492138289,
+            0.006302962169276006,
+            -0.008065841090139872,
+            -0.00035242603988955366,
+            -0.01586406637763306,
+            0.008271894439780969,
+            0.007438312999409211,
+            -0.0069617481460406095,
+            0.011402932583388205,
+            0.002666895909532698,
+            0.0005219881149353855,
+            -0.012159425670629751,
+            -0.012042145306711322,
+            -0.00957276588761916,
+            -0.03401863879157551,
+            -0.02604353712472079,
+            -0.024949533021906844,
+            -0.0014209149651479556,
+            -0.006566685925666972,
+            -0.03187796739029736,
+            -0.01209152137926458,
+            -0.011110763124407752,
+            0.007756961097537267,
+            -0.013260548545917543,
+            0.005433285079114047,
+            -6.399098520041351e-05,
+            -0.0044960488826674555,
+            0.016615636438440516,
+            -0.003643291599919672,
+            -0.019488010211707343,
+            -0.03927818878149417,
+            0.008209336177358258,
+            -0.02805593125673831,
+            -0.0038816771550190788,
+            0.007607119219721711,
+            -0.00528681677614909,
+            -0.029502897838743516,
+            -0.009149794737430706,
+            0.008767001199311985,
+            0.003507217008280066,
+            -0.01773349756494436,
+            0.03579670773605949,
+            0.026971617790626075,
+            0.011875179879209696,
+            -0.019891342676443626,
+            0.0508929813275565,
+            -0.0047010735337056275,
+            -0.012544935684845649,
+            0.018297503869683027,
+            0.02355022268131519,
+            0.03273167043015095,
+            -0.024830243320788693,
+            0.0009370814985214882,
+            -0.021937793653233454,
+            0.00535662615596573,
+            -0.035009915581788176,
+            0.0028595659314725012,
+            -0.014395027018699446,
+            0.015596459769560532,
+            0.03397372881171701,
+            -0.013254226072981457,
+            0.011195747181764677,
+            -0.016299384375589934,
+            0.00730244731485402,
+            0.00569944090461748,
+            -0.012916284280345044,
+            -0.005812311882399539,
+            -0.022902815152116493,
+            0.020140276669979015,
+            0.005809799950176385,
+            -0.0008130655714656685,
+            0.0104085826652653,
+            -0.044723432238900435,
+            0.03286429597337671,
+            0.008730225944501239,
+            0.017599189944921396,
+            0.0029762446259779998,
+            -0.0003068671951232415,
+            0.011611551606224214,
+            0.011228401175698422,
+            -0.016029727902740706,
+            0.004345913009102136,
+            -0.004613061137378433,
+            -0.03850533909474302,
+            0.009363351681220008,
+            -0.03631184057978666,
+            0.014427784677340159,
+            0.016026872002233575,
+            -0.004170582842643461,
+            -0.025166189571409366,
+            -0.005428471012333673,
+            0.010827910811085384,
+            -0.009783069664327864,
+            -0.009987805234276443,
+            -0.04163358183060102,
+            -0.025236440364963034,
+            0.034097508113604576,
+            -0.002119474613596975,
+            -0.025826403511707147,
+            -0.010830738112107807,
+            -0.011351132636491575,
+            0.020207687645133,
+            -0.006542026283946858,
+            -0.0006263975804936843,
+            -0.04150011545311013,
+            -0.0073396962917864515,
+            0.011192225907726928,
+            0.027746213794734322,
+            -0.035970017302005566,
+            0.02708126943501373,
+            0.002105633542220187,
+            0.014262674845755191,
+            -0.01626664983003942,
+            -0.005631805361903626,
+            -0.013990539604372344,
+            -0.035877093361375685,
+            0.016487723013906077,
+            -0.008607758248055054,
+            0.004714436224135091,
+            0.04458009880881818,
+            -0.0036872068506712727,
+            -0.009466125348499464,
+            0.007600705817195143,
+            -0.0034590883553048063,
+            -0.010351576085287273,
+            -0.0028564505793999295,
+            0.001959661298507,
+            -0.007080697863126256,
+            -0.04183476693971616,
+            -0.00499888976530217,
+            0.04684391782633148,
+            0.006361744381879857,
+            0.013276793765092465,
+            0.013516220521120684,
+            -0.018335746557932003,
+            -0.045577810515431215,
+            -0.0071616111758423025,
+            -0.0032763880176386808,
+            -0.022708028392758567,
+            -0.0240284945459861,
+            -0.008735040856657758,
+            0.005844145402902923,
+            0.006359368165686724,
+            -0.01120763599008222,
+            -0.022676131361171057,
+            -0.027085968700452136,
+            -0.010526103176253857,
+            -0.02123944716136792,
+            -0.010792145796733375,
+            -0.027142270190771887,
+            -0.028742234538796804,
+            0.02880426953386151,
+            0.007575182232859564,
+            -0.004084088301176117,
+            -0.010005765476121966,
+            -0.01255640546813973,
+            -0.0004774761617900028,
+            -0.0025307994917461232,
+            -0.005822065709999144,
+            0.0038614286313282164,
+            -0.011390482620811868,
+            0.023152086963976554,
+            0.0021220701260546895,
+            0.018329162172128138,
+            -0.05772324571231113,
+            -0.002709083851090071,
+            0.021497896395795905,
+            0.0033025410488410913,
+            -0.005412445141910314,
+            0.029352650459792132,
+            0.011223947499453582,
+            -0.006539460282439711,
+            -0.031355428613718234,
+            0.020942976176820262,
+            0.024122993515785947,
+            0.00817615634503138,
+            -0.012994160510030828,
+            0.01221095914384204,
+            0.01629778384790159,
+            -0.004494985018246821,
+            -0.011161098655072933,
+            0.005586857967039181,
+            0.020941039642372112,
+            0.013580622629911925,
+            0.0013398071867077221,
+            0.030905022925366163,
+            0.0006862084391508121,
+            -0.03521231612862609,
+            -0.01908191995763771,
+            -0.01275995164030872,
+            -0.003584331305986364,
+            -0.015264425022763126,
+            0.005294329654809353,
+            -0.002355465010876059,
+            0.006335052149475905,
+            0.022655379849785256,
+            0.03593231939948758,
+            0.0146937743203089,
+            0.00937330328448886,
+            0.003167462724350819,
+            -0.0013235176101076284,
+            0.012102872918129403,
+            0.008638011773075119,
+            0.0350701240268404,
+            -0.014653221697693864,
+            -0.011003994230902835,
+            -0.0037767288519369537,
+            0.010100131061186566,
+            -0.004113527518662366,
+            0.020414393256136372,
+            0.03274709615649469,
+            -0.0037222949335916876,
+            -0.005604552180246868,
+            -0.006160118231975786,
+            0.0041467454414452565,
+            0.01807879127455274,
+            -0.01252134965321004,
+            0.004364858987501461,
+            0.005549300665159658,
+            -0.0071897162833861,
+            0.018567059255813558,
+            0.0002622074720860951,
+            -0.012983550838417323,
+            0.03266376564402191,
+            0.11449509387240629,
+            0.0017251916835047628,
+            0.007125215166916139,
+            -0.040229997815548584,
+            -0.020537987527148888,
+            -0.00245728735683473,
+            0.005779972828836593,
+            0.007004732428236457,
+            0.02225914982152608,
+            0.006505827715591914,
+            0.0006708045351142873,
+            0.031708522931495536,
+            -0.017334083511709092,
+            -0.002522861743921599,
+            -0.01745656559688129,
+            0.013890451996979048,
+            -0.014312605700272691,
+            -0.013784963572164409,
+            -0.01572417853955075,
+            0.004507425871748977,
+            -0.000601158156253668,
+            0.0016158265715634729,
+            0.01791303234734911,
+            0.0011926917387973374,
+            -0.026393351252156246,
+            0.0006694008039871878,
+            0.001026031515214626,
+            -0.004514419527575117,
+            -0.011320929719992686,
+            -0.02223001723942815,
+            0.016235841442653918,
+            0.014188311619681008,
+            -0.004895218832118094,
+            0.009244697376301004,
+            0.010558796153959677,
+            0.02707485418319703,
+            -0.028380342401606758,
+            0.0040204430418629405,
+            0.007023041225985294,
+            0.01606556581447933,
+            0.017617862031190843,
+            0.0034870847111552862,
+            -0.011253315705053224,
+            0.03238289399766116,
+            -0.010563282687352038,
+            0.0065093536270233385,
+            0.014966802208644462,
+            -0.03541611540945151,
+            -5.821682910348165e-05,
+            0.0061985444938402745,
+            -0.011716201661363997,
+            0.009580035961315467,
+            -0.03120692534097433,
+            -0.0039291243987978845,
+            0.009317468331974374,
+            0.017611312558635673,
+            0.008804996982574765,
+            -0.0034129169537151925,
+            0.010822226698822738,
+            0.012510661458054741,
+            -0.0027147975086409966,
+            -0.008855213644027594,
+            0.01238502686635956,
+            -0.0007687053446954701,
+            0.00450108863711973,
+            -0.00799170686059026,
+            0.0029755595812552293,
+            0.002290650917244691,
+            -0.01554856300008412,
+            -0.011779430686394945,
+            0.00438393126679289,
+            0.0036931891075561957,
+            0.001535662168147833,
+            0.012014408636737167,
+            -0.006087255045485002,
+            -0.0077816220325470226,
+            0.0040573724156794284,
+            0.003136943055301706,
+            -5.505014316363865e-05,
+            -0.030625245024561508,
+            0.01585923084309149,
+            0.0067595435845396405,
+            0.010013308598626031,
+            -0.005315626260228517,
+            -0.00903142414091241,
+            -0.007240920015018893,
+            -0.01228521172101564,
+            -0.010657894223173453,
+            0.007658605592936815,
+            -0.00724924993432376,
+            0.011124174211132038,
+            -0.03128027787357322,
+            -0.013988715435985316,
+            0.01231792993245938,
+            -0.01528161242159377,
+            0.0007675821323979293,
+            -0.008389914312201402,
+            0.0009417242487395995,
+            -0.005119304379761705,
+            0.007887436211143328,
+            -0.03731503169547854,
+            0.007878001990271541,
+            0.02456170736765724,
+            -0.00330924625934981,
+            -0.020259771366167557,
+            -0.009613930457931786,
+            0.0241314385136084,
+            -0.008267831641886344,
+            -0.01645888287387047,
+            0.008421304862524506,
+            0.010140813149093993,
+            -0.0006309118552875393,
+            0.0008237784055549769,
+            -0.003726984191590917,
+            0.01669125388798323,
+            0.02054777616420997,
+            0.041930770964107515,
+            -0.014789610432004559,
+            0.01092349703703152,
+            0.02866938974792608,
+            -0.03378576980722847,
+            0.014728113846952417,
+            0.02518409645155824,
+            0.014590531327650454,
+            -0.0004446956315190408,
+            0.001609777186372437,
+            0.0068939321280085475,
+            -0.014273492115311677,
+            0.009247676157551494,
+            0.013380104440744367,
+            -0.018407746260686045,
+            0.020444430509441124,
+            0.03140071554008003,
+            0.0025839998101985736,
+            0.01355205268903058,
+            -0.006268939830271642,
+            -0.013860692186849113,
+            0.014717364068116991,
+            0.031429975488060896,
+            0.0006919565473572909,
+            -0.018561500446336806,
+            0.019128244642842268,
+            0.006988095806264863,
+            -0.003678529269242034,
+            -0.0021993078588142664,
+            -0.021334919445200388,
+            -0.021003586791369228,
+            0.017304350704792804,
+            0.013508779408333964,
+            0.027794485858413365,
+            -0.010594702782952029,
+            -0.007818089330662161,
+            0.014293688127901867,
+            -0.022368568887004094,
+            0.015426482139881204,
+            -0.02160820059598751,
+            -0.00571499813703566,
+            -0.0077402210272634235,
+            0.03274360648629609,
+            0.001441304806200403,
+            0.02734331677589297,
+            0.016457027764064002,
+            -0.033609420134045206,
+            -0.029790535553657092,
+            -0.00712106584238484,
+            -0.02307933297713104,
+            0.010356561171963577,
+            -0.03901183838032092,
+            -0.00071735150189382,
+            0.008251646656432826,
+            -0.008079917288471352,
+            -0.0007087526632966007,
+            -0.006419191399519291,
+            0.013989344185652506,
+            0.013661633160336152,
+            -0.0228103143826663,
+            0.023644639659577426,
+            0.008343481940133684,
+            0.025998483906975782,
+            -0.02076055376006461,
+            0.019148568238437927,
+            -0.017705363646416224,
+            -0.02777313933338929,
+            0.014358130220942483,
+            0.023585731219788467,
+            0.01635364038245971,
+            -0.007122853965024528,
+            0.007110003583832299,
+            -0.03556761320536796,
+            0.027585457849964405,
+            -0.013064062623866305,
+            -0.012504161289952458,
+            -0.0032379276562094555,
+            -0.011931827532360686,
+            -0.002574161005800279,
+            0.0008533907465694845,
+            0.0034345232246102806,
+            -0.03131462673429805,
+            -0.0002450959360520896,
+            -0.03727416416766574,
+            -0.028944814604815072,
+            0.01360126708805471,
+            -0.00036361284498363273,
+            -0.0049097813612814616,
+            0.02933242705521636,
+            -0.026339136251046315,
+            -0.02693295650849956,
+            -0.009590247544093352,
+            0.02853096456887074,
+            -0.0220673798697511,
+            -0.004519050393279848,
+            0.0015034014871316069,
+            -0.01828226786700525,
+            0.004436383319946364,
+            -0.0012748481042235446,
+            -0.010416065158876917,
+            -0.01105146139270597,
+            -0.03229479099927737,
+            -0.005822133182096844,
+            0.02106215163620738,
+            -0.006957125617688484,
+            0.002746241228040847,
+            0.013312248331456934,
+            -0.010968629118224582,
+            -0.0029946900704643812,
+            0.0017875284441794235,
+            0.023304175718480834,
+            -0.021033941064173063,
+            0.007714334541806199,
+            0.005955442031111047,
+            0.0016405586860131207,
+            0.018406562522853148,
+            -0.0054922197063461825,
+            0.0037690571959124617,
+            0.0162126534419016,
+            -0.009969304596173302,
+            0.017959307638726878,
+            0.007691549802475525,
+            0.006681837561745192,
+            0.002775700618222622,
+            0.007712770283579111,
+            0.009752652973700739,
+            -0.013506020126078112,
+            0.016911141425375933,
+            0.022468715726349706,
+            0.00023233555209372766,
+            0.021066580799163234,
+            -0.02302516292580905,
+            -0.028618544628453015,
+            0.018297878810036087,
+            -0.0225078941299981,
+            -0.012735208680311652,
+            0.016459925496937065,
+            -0.0043032724578332855,
+            -0.00797609922711709,
+            -0.014986057073351055,
+            0.02260618561608301,
+            0.02132930921464508,
+            -0.020249788476562527,
+            -0.012385936327068884,
+            0.0072304431443959505,
+            -0.004373475447051833,
+            0.007086656081865733,
+            0.03195810137269375,
+            -0.015982210897392764,
+            0.006818890093901413,
+            -0.0038306138017521626,
+            0.0030753642449792215,
+            0.017830546304667783,
+            0.009470018737705222,
+            -0.0019815241970236146,
+            0.030100238416495066,
+            -0.001330614013444443,
+            0.016392226282459177,
+            0.0004888028477871524,
+            -0.010948833692356176,
+            0.006343890495626752,
+            -0.026652806306173643,
+            -0.008627570055007363,
+            -0.001203530223345183,
+            -0.009581494451831351,
+            0.07598616930784373,
+            0.003930153552069478,
+            -0.01295957554290434,
+            -0.001673007115291578,
+            -0.001368287102181597,
+            -0.006897179139643672,
+            -0.00960679451603541,
+            0.0287166325268306,
+            -0.016613613331623346,
+            -0.01894527592131826,
+            0.0008370014132002703,
+            -0.006785618888421608,
+            0.00919924251249592,
+            0.01816363936999796,
+            -0.010225961928154075,
+            -0.01604759761423727,
+            0.001577138242530474,
+            -0.003520564936698765,
+            -0.004395434431595494,
+            0.0052399043934902014,
+            -0.010127122101423486,
+            0.006456662777583096,
+            0.026560491731327886,
+            -0.013684536319659296,
+            -0.01425196172663986,
+            0.004084982328707292,
+            -0.0010682084059972982,
+            0.009758641913073978,
+            -0.00864879062601467,
+            -0.00481222050200678,
+            -0.014375330954464725,
+            -0.011378890549403524,
+            0.02505641780042516,
+            -0.02861164504722546,
+            0.0034060847953066354,
+            0.014937227531980433,
+            -0.00016904224334342915,
+            0.01541381851648916,
+            0.0033028855218525644,
+            -0.03615935765718852,
+            -0.013677774380513626,
+            -0.02073839691250734,
+            0.019821074228011413,
+            0.01688422012576042,
+            -0.0033731840788147442,
+            -0.013840993586441808,
+            -0.026149880178874887,
+            -0.007430563876740068,
+            -0.02466234818567362,
+            -0.020708282040996014,
+            -0.019276991448222817,
+            0.006137002799644838,
+            0.005162892426268572,
+            0.022881759407670583,
+            0.0025854880108157857,
+            -0.015058889448404231,
+            0.0029948608544278386,
+            -0.006722836436935071,
+            0.011845298212977357,
+            -0.02747944486818378,
+            0.015311272959213262,
+            -0.01610493464983465,
+            0.025239768880017402,
+            0.007425208902253236,
+            0.0010569008485112994,
+            -0.008822681490988575,
+            0.00010704167529046774,
+            0.014883787023445807,
+            -0.004186968199973372,
+            -0.0007781259002896265,
+            0.02787555889813697,
+            0.02087373813833492,
+            -0.0017783425135865238,
+            0.008743718448136432,
+            0.028026471206552342,
+            0.019081083283793984,
+            0.0325261942213039,
+            -0.013815526479760766,
+            0.005113676604826083,
+            -0.028873600114081546,
+            0.010510475843707614,
+            -0.010871039729515013,
+            -0.010201824683340651,
+            -0.0067946599549165735,
+            0.027756887504389347,
+            -0.007962601205446472,
+            -0.0006687976124826922,
+            0.03469894503563926,
+            -0.014185294847328063,
+            -0.00010543064621374546,
+            0.0007386418834840544,
+            -0.009759481309152999,
+            0.02742722730386853,
+            -0.0073590367291510046,
+            0.018456032721630843,
+            -0.019560882612141943,
+            -0.013705917411254969,
+            -0.011894560208199602,
+            -0.001142768182039817,
+            -0.02576754877141608,
+            -0.008755240066826174,
+            -0.02249287216087102,
+            0.00859172262856532,
+            -0.0046012388553094375,
+            0.018845920253446127,
+            0.0011617844393195566,
+            -0.005275799428018491,
+            0.029069367036565145,
+            0.0088080893110627,
+            0.018100411389202663,
+            -0.0004971948011851863,
+            0.01216566778850985,
+            0.02369448391321832,
+            0.002637001729688794,
+            0.005012948614448268,
+            -0.024213704079796478,
+            0.003420184574622472,
+            -0.0004359900462023086,
+            8.265488264269535e-05,
+            0.016267948861008616,
+            -0.019572032453100303,
+            -0.02216264534756073,
+            0.021871880626992544,
+            -0.012742926070754964,
+            0.011676769226089007,
+            0.0035362870996336194,
+            -0.010732229460038356,
+            0.0006838777488770268,
+            -0.03083438636386314,
+            0.008820746379709355,
+            -0.03216685858285627,
+            -0.032230808828686675,
+            -0.013547681710236494,
+            -0.009994106571421729,
+            0.0008758088338659712,
+            0.028447187234357855,
+            -0.00795317029726568,
+            -0.015797892954136448,
+            0.005363565557248085,
+            -0.011199252697624111,
+            -0.00817162221789202,
+            -0.0007867892785855576,
+            0.005856939999700809,
+            0.009461959355310273,
+            0.005997804620690314,
+            0.011648335019189696,
+            0.007491946658046298,
+            -0.008028800679169293,
+            -0.008837725220736191,
+            0.015889923814605387,
+            -0.0014409716087425222,
+            -0.01176424391171685,
+            0.030430130082762363,
+            -0.013300620185884817,
+            -0.0005592926632214984,
+            -0.0021975669766032472,
+            -0.0015687208316663597,
+            -0.0021652076181411244,
+            -0.014225289143358926,
+            0.01870568404189425,
+            8.334501002782455e-05,
+            -0.014288806218067678,
+            0.0032841743905975634,
+            0.003841780428153241,
+            -0.0036498339483411074,
+            0.008388922960269383,
+            -0.008147275785908849,
+            0.017348147498892,
+            -0.015113395971380741,
+            0.008740452410210708,
+            0.0038335161236289234,
+            0.0043482078469644385,
+            -0.0065755348640655865,
+            -0.01592809184384243,
+            0.009642593996701513,
+            0.012051872087663658,
+            0.02173264012652684,
+            -0.02525660485859878,
+            0.00876161653901566,
+            -0.005374533920781607,
+            -0.014534179652013885,
+            -0.016127866637369738,
+            0.00982303248417706,
+            0.022322910435388785,
+            0.0063086346898091296,
+            0.012139263756190026,
+            0.00013383660542132047,
+            0.005848259030010402,
+            -0.014767198851314148,
+            0.010570158368602513,
+            -0.011355774405066537,
+            -0.018439261453084426,
+            -0.005243587266408389,
+            0.028894120979627323,
+            -0.005492024421080143,
+            0.017938747701972086,
+            0.0021350711250093858,
+            0.029007356490801176,
+            -0.008686781541806358,
+            0.0014883465513564527,
+            0.0039866022469891155,
+            -0.01295271525889559,
+            -0.004272008629260297,
+            0.01976273843646234,
+            0.04882868638601926,
+            0.010204475902027572,
+            -0.0009155109545661691,
+            0.010069071220988797,
+            0.005626418787776309,
+            0.002148101019776153,
+            -0.00805781810841286,
+            -0.030431071067211577,
+            0.008883707378543252,
+            0.005628218215212441,
+            -0.01997825125961566,
+            0.019838234890736068,
+            0.026483365072033395,
+            0.023040623182975964,
+            -0.011585643598171132,
+            0.016539930468533123,
+            -0.010092529089936188,
+            0.0177008735686489,
+            0.028665424114962996,
+            0.013810061902648515,
+            -0.010487150104822674,
+            -0.0020847870330548534,
+            0.006329342599564054,
+            0.004186981876004016,
+            0.00026239408223817384,
+            0.015190614519766204,
+            -0.019155180136775043,
+            0.01005728016778505,
+            0.014725826697964637,
+            0.03322436221319799,
+            8.049930133627564e-05,
+            -0.01645191835557271,
+            0.0016249281996128196,
+            -0.009976155662939693,
+            0.00856399492362911,
+            0.000632427127231252,
+            0.0032242260692672124,
+            -0.009688119824276678,
+            -0.012682779503915694,
+            -0.002851918183031152,
+            0.009403205508474241,
+            0.0021188716361101698,
+            0.002759004011231831,
+            0.007641925150177999,
+            0.0022830062031146204,
+            0.01713024474384216,
+            -0.0011181980841546123,
+            -0.00029141889484139217,
+            0.004601090256674874,
+            0.010509199590505558,
+            -0.01649852835819883,
+            -0.017548369964263514,
+            0.004337237475053179,
+            0.012217494210546133,
+            -0.013159179948224596,
+            -0.013241981632059755,
+            0.015704543608959117,
+            -0.01913575130892795,
+            0.024934102868065366,
+            -0.00942381892138331,
+            -0.004543611336526587,
+            0.000343200128934031,
+            -0.00727538880611703,
+            -0.0038751118497522673,
+            -0.0030064279504724797,
+            0.003984715814505561,
+            0.0019261818110367147,
+            0.007472350699291607,
+            0.008472369788801792,
+            0.014568547873264247,
+            0.006043895352236181,
+            0.0032580830408778395,
+            0.009109104498825864,
+            -0.014523846294121576,
+            -0.014800649887278973,
+            -0.002704791598011488,
+            -0.008895394038700809,
+            0.00902016867266151,
+            -0.006607182154874112,
+            0.00845408382182461,
+            -0.015501867835442603,
+            0.015573642225745032,
+            -0.012200552219016517,
+            -0.0008900215712478553,
+            -0.009666233545144905,
+            -2.7470277092642562e-05,
+            0.008191697904514857,
+            0.014605287653054787,
+            0.0070552942843382215,
+            -0.031921405225964206,
+            0.0017403427534633655,
+            -0.008846568352320488,
+            0.03446319360336241,
+            0.002612606662691094,
+            -0.01060863123202008,
+            0.020352181356568008,
+            -0.01592124132155158,
+            -0.00027704668715660693,
+            -0.01602075523652062,
+            -0.004797557413047212,
+            0.00707939446796679,
+            0.016399017978990946,
+            0.02759049512664933,
+            -0.005697020356865688,
+            -0.007995305872076221,
+            0.006384047872862793,
+            0.029590696670334232,
+            0.012246955508890425,
+            0.024748351124415785,
+            -0.007633712004672087,
+            0.014348228716100414,
+            0.007773260552660029,
+            0.0076045488653348975,
+            0.004321689250440021,
+            -0.002591801797145718,
+            -0.011698574606499644,
+            -0.014974971814360001,
+            -0.0037563183330623384,
+            -0.016987915346053498,
+            0.00012247833530362867,
+            0.021344400276501117,
+            -0.00475351423959752,
+            -0.0033847161875730026,
+            -0.01874186437694691,
+            0.0033461670218611346,
+            -0.010029445302549983,
+            -0.00367421246775698,
+            -0.013133469163793961,
+            -0.0254122035745965,
+            -0.018556369080030412,
+            -0.002155244024999317,
+            -0.00929309186646207,
+            -0.015194269781263309,
+            0.0070354690219415535,
+            -0.016305117309271634,
+            0.0080837166473439,
+            -0.00448473417024141,
+            -0.016508775226236883,
+            -0.00958925666995396,
+            0.01100335153658195,
+            -0.0041819993634961685,
+            -0.01929493648733792,
+            -0.01282300765645703,
+            -0.011300260564838231,
+            0.0051733704121938884,
+            0.005996780699975213,
+            -0.0038111648304178544,
+            -0.0028279639348479987,
+            0.016640121845366295,
+            -0.010197365811240557,
+            0.009878730312873492,
+            -0.03335474019383245,
+            -0.03551826040406959,
+            0.004497847402931096,
+            0.01507007013703286,
+            0.0013085881889541252,
+            -0.02078680101065954,
+            0.010806315988966003,
+            -0.016499186313125556,
+            -0.00858013792516617,
+            -0.008522534127498645,
+            0.009649521088411411,
+            -0.007283003380127847,
+            0.01299542385123007,
+            -5.3226845309105124e-05,
+            0.030477683597373687,
+            0.0074153455615142755,
+            0.004161881195438522,
+            -0.00679081048132963,
+            0.008354912600159931,
+            -0.001621776416108105,
+            0.004080191054849395,
+            0.02120524874226696,
+            0.012440848502230418,
+            -0.00218041171518331,
+            0.006458415098595872,
+            0.005678055538564355,
+            -0.017186537377121235,
+            0.02507960402044456,
+            0.024678178106806115,
+            -0.008205288425409454,
+            0.004108402466376321,
+            0.005783335302169668,
+            -0.008846663924040241,
+            -0.006897707231869481,
+            0.006527795262026734,
+            -0.018474181903007085,
+            -0.0008721381678008483,
+            0.010598084358464908,
+            -0.02255441909004843,
+            -0.004840511019512733,
+            0.021745651362109403,
+            0.004631567631335321,
+            -0.018558972627958834,
+            0.02764788406651861,
+            -0.004942361238163609,
+            -0.0027290999934532494,
+            -0.010876135202650048,
+            0.01671497143541799,
+            -0.012173527207375091,
+            0.008443317346655462,
+            0.011021541861818043,
+            -0.006295639322287843,
+            0.003083685676540999,
+            0.008277206401503182,
+            -0.004504302913715969,
+            -0.0068075424927125676,
+            -0.0010508416962215848,
+            0.010167160518175114,
+            0.0036980815941666336,
+            -0.022704963865525812,
+            -0.006020073261002279,
+            0.02934227992246887,
+            -0.015082922782242507,
+            -0.0050916398557337705,
+            0.005376294716056329,
+            0.007709342013963906,
+            -0.003476236561444464,
+            -0.0023399991233097337,
+            -0.00787486645740744,
+            0.010356414979428607,
+            -0.015950083845400274,
+            0.02076360898078935,
+            0.025608263494550438,
+            0.024504236990840842,
+            -0.014814235990238182,
+            0.010828584133720797,
+            -0.011703992394890898,
+            -0.011047470905668532,
+            -0.015542835222173864,
+            -0.001658296303013357,
+            0.001970208852236451,
+            0.013107754130616248,
+            0.009666887546609378,
+            0.0235853683657408,
+            0.006473544089109194,
+            0.008973300312205858,
+            0.014118601565410305,
+            -0.003902202854013708,
+            -0.018303438244719996,
+            -0.014584226585460755,
+            -0.011977613285028946,
+            -0.01756345491020528,
+            0.0004563602312336091,
+            0.020461659996271687,
+            0.012417744207683839,
+            0.024136626592375227,
+            0.002119213701333967,
+            0.013759902247467922,
+            -2.9525392347047554e-05,
+            -0.000892319792935443,
+            0.00045783825646242434,
+            -0.005238483120312843,
+            0.019255756091120846,
+            -0.004599752813817219,
+            0.01656589522110913,
+            0.0006443773300684925,
+            0.007133940816771615,
+            0.013554761340256566,
+            -0.01704388669741371,
+            -0.003443100892108368,
+            -0.010029620309829285,
+            0.000941391966877521,
+            -0.019633111750884034,
+            0.01934571074127927,
+            0.010273330977233852,
+            0.0017121345993533963,
+            -0.018327189396014198,
+            -0.0029706385099949693,
+            0.03448635131998772,
+            0.019458999453011265,
+            0.0050734354363614525,
+            0.02632443373898792,
+            0.013018762352749236,
+            0.014443611411502168,
+            0.010153370950346075,
+            0.02356766253713842,
+            -0.023479811300626968,
+            0.02095255383200973,
+            -0.011590347850340786,
+            -0.007414136601134069,
+            -0.0035774888741646705,
+            0.0025141508183687115,
+            -0.013588691733097813,
+            0.00982877622088977,
+            -0.0017916539399655826,
+            -0.025386211797192578,
+            0.009164316443206016,
+            -0.008811052979979271,
+            0.010464442616928296,
+            -0.007774569252594923,
+            -0.0002806136360286729,
+            -0.0127084026190163,
+            -0.015184317235990752,
+            0.001259201880504347,
+            -0.009158429132281081,
+            0.023908150915412617,
+            -0.021210848953875233,
+            -0.01321384014436847,
+            0.013997502713857512,
+            -0.006869443694669701,
+            -0.011364239609894453,
+            -0.00374449980963666,
+            0.023673050288879224,
+            -0.009243341446512421,
+            -0.01359367519009039,
+            0.014464166953530989,
+            0.003834735113400469,
+            0.0059713317014342395,
+            -0.007508862694272452,
+            -0.008428705474135382,
+            -0.003280788036976103,
+            0.005998840043285335,
+            0.026615077617284676,
+            -0.014059930972065399,
+            -0.0032833264512892022,
+            -0.009189999799498163,
+            0.004118204537632102,
+            -0.02359889635281631,
+            0.02771075351384059,
+            -0.008871395303200375,
+            0.021705177758560575,
+            -0.0015924449308442096,
+            -0.006239188158271628,
+            -0.024231036074349822,
+            0.027522945599928003,
+            0.003683870966132199,
+            -0.010221180899173243,
+            0.0012768725460662862,
+            -0.052005292736885915,
+            0.016636109796947025,
+            0.008429895913163985,
+            -0.019507994429834353,
+            0.003697519981610208,
+            -0.004773584856005853,
+            0.018983847580558853,
+            -0.022966993228850697,
+            -0.014078986007109447,
+            0.008283766047449966,
+            -0.01153285124362746,
+            0.0031439180465649996,
+            -0.02527749501821793,
+            -0.005754537200570427,
+            0.0011760345167229269,
+            -0.005276936074280004,
+            -0.001705933750221904,
+            -0.020022761608132548,
+            -0.025972227850795115,
+            0.017666017379157135,
+            0.015820875579600418,
+            -0.002278262448951789,
+            0.01320818011561616,
+            0.008709899932124187,
+            0.00836573261091119,
+            0.016034263394940008,
+            0.007124509234103267,
+            0.0081473711608784,
+            0.045498533239850446,
+            -0.019846323044145485,
+            0.007371629414298193,
+            -0.0014854149174929534,
+            0.013951741898629716,
+            -0.030467924817555014,
+            -0.0055144480998344705,
+            0.0062802744565489945,
+            -0.0049373689340857915,
+            0.03510454786147091,
+            -0.017326094216003073,
+            0.01694160526735264,
+            -0.0026075809870445238,
+            0.0007420394265014909,
+            -0.02042003877666893,
+            -0.026156937726134134,
+            -0.014294938857971834,
+            0.0011857406889443987,
+            -0.0010047941261608975,
+            -0.011946583993069242,
+            0.006797001710574735,
+            0.004917161312925866,
+            -0.0020713379451245515,
+            0.015652523103715316,
+            -0.02952061566240878,
+            0.007356800041369564,
+            -0.010733578620243498,
+            0.020008504830359524,
+            -0.00990331048284697,
+            -0.0013319400098308093,
+            0.007236960885181047,
+            -0.004122868383988748,
+            0.013216043472071927,
+            -0.0038337251451538677,
+            0.004227115653354611,
+            -0.005221321656766545,
+            -0.01794343569722877,
+            -0.0016435261495179767,
+            0.021429008432348098,
+            -0.008349684096540855,
+            0.031864497493526205,
+            -0.01264743878037828,
+            0.009174982784686633,
+            0.010692990337462762,
+            0.0006628831036334152,
+            -0.005286285851702483,
+            0.009052971814211658,
+            0.0021431970274545893,
+            -0.008690520448278274,
+            0.006934310986093342,
+            -0.018349449937310143,
+            0.008494451785507706,
+            0.03394632739506833,
+            0.0018511328780472266,
+            -0.0007519661670070526,
+            -0.007266663179613392,
+            -0.01335844201823745,
+            0.018540713029383066,
+            0.005488273906733518,
+            -0.002546531152114063,
+            -0.027536660869590543,
+            0.02007746621157705,
+            0.0012265798861291223,
+            -0.004755325069578531,
+            0.012466557901411086,
+            0.0020495867632654885,
+            0.019989531151046736,
+            -0.010171678639113316,
+            0.030757957899447096,
+            0.010050036220124393,
+            0.002959833051227505,
+            0.0038556928367700113,
+            -0.036655749074851536,
+            -0.01652152778069314,
+            -0.00019357502472740266,
+            0.014271581054865031,
+            -0.012430647252826173,
+            -0.022432580814844155,
+            0.0010346169306977334,
+            0.01148004051449691,
+            -0.0033898642762881046,
+            -0.009650665305271706,
+            0.01560757541276278,
+            -0.020342213902670575,
+            0.006701302394408393,
+            0.007693672045981921,
+            0.0287100214318104,
+            -0.0020735384554022153,
+            -0.0012324838849990056,
+            0.000626416803904792,
+            -0.0055669860801192935,
+            -0.0008037366126870824,
+            0.017251369734470945,
+            -0.013661316947002466,
+            0.0175926211217534,
+            -0.012735241490550407,
+            0.0064713656749838585,
+            -0.0021092378432037563,
+            -0.010629579860050812,
+            -0.020114807971202787,
+            0.0029609495764307357,
+            0.009378003163375899,
+            0.006920144698428806,
+            -0.010438984881449953,
+            0.010715409812532968,
+            0.012197673689706352,
+            -0.00753962861484439,
+            0.014519905132928041,
+            0.022173989895384192,
+            -0.007687503607783572,
+            0.013633321414009471,
+            -0.0041446334728745095,
+            -0.014056264816568229,
+            -0.009985463824440833,
+            0.023929750617160402,
+            -0.0005563806942667597,
+            0.008756997373618809,
+            0.00388125051641897,
+            0.0007856447663040696,
+            -0.011226530517316183,
+            -0.001653447552601681,
+            0.019294448360333633,
+            0.011831934675073843,
+            -0.0013888534822008192,
+            -0.0010525764356699885,
+            -0.0014566936525653407,
+            -0.009427557739483988,
+            0.02049188741459728,
+            -0.022030783927622997,
+            -0.0019604940727076573,
+            0.00461018148439991,
+            -0.022589816803875618,
+            0.0001863117949313752,
+            -0.024570217962989905,
+            -0.01569952241187498,
+            0.011763033781122343,
+            -0.0022530631823737554,
+            -0.016028810867854907,
+            -0.0003242701040942633,
+            7.424127301806556e-05,
+            -0.007815789940508527,
+            -0.004135500768181258,
+            0.006115909992703774,
+            -0.00034193217510010117,
+            -0.0032262189374874842,
+            0.013430154310506884,
+            -0.009996622196870008,
+            0.015431528911575157,
+            0.0067029980830614145,
+            -0.015702167721800483,
+            0.0021406610839557003,
+            0.018500219544264753,
+            -0.0058910087864916085,
+            0.00787021425357441,
+            -0.014681907249972176,
+            -0.0037495551894426303,
+            -0.005787661351944086,
+            0.006051351977821761,
+            0.024849195780885833,
+            0.012969434450643372,
+            0.01139340445077682,
+            0.017405361843075933,
+            0.004325000715054734,
+            -0.01572297405297993,
+            0.0007416798945675923,
+            0.01682727189828363,
+            0.010464036837245312,
+            0.009804326468113703,
+            0.0008975556667684365,
+            0.0043704860361487635,
+            -0.005142176468800175,
+            0.009317064072855551,
+            0.02915482449687158,
+            0.0013434689612997423,
+            0.008696141010760192,
+            0.017276730345147134,
+            0.027038741396131265,
+            -0.0035088997194497725,
+            -0.007469614319788499,
+            0.008739942310300076,
+            -0.009657991604173173,
+            -0.001081629174976152,
+            -0.017000351183542026,
+            -0.012310275736750868,
+            0.016510505259358398,
+            -0.006084627062230492,
+            0.02848913486719956,
+            -0.0010655806791907225,
+            -0.02026498848361482,
+            -0.01565505608194376,
+            0.001404549479440637,
+            -0.01362132895080652,
+            0.005287566256051299,
+            0.014187265685427415,
+            0.01917046513020831,
+            0.004687191682875024,
+            -0.022258337721085656,
+            0.037364962197399806,
+            0.01757108963349427,
+            0.019377870795760123,
+            -0.019422840506328382,
+            -0.006159311237412833,
+            -0.009077095160676276,
+            0.02116697012944498,
+            -0.00880692095488465,
+            -0.009875674856043954,
+            0.024014332004451508,
+            0.017812606845656484,
+            0.005679806286464216,
+            0.006736609904079586,
+            0.0020048663439319585,
+            0.0025414630253661444,
+            -0.015831593851012047,
+            0.017957902498542212,
+            -0.005858125698347523,
+            -0.0058068554681761945,
+            0.017529514573571298,
+            -0.005989581792993925,
+            -0.0030306021203732347,
+            0.00015839997806822045,
+            -0.019697052145531482,
+            0.011618112886207201,
+            0.008256007601576028,
+            -0.017229774429089857,
+            -0.0014757560402567304,
+            -0.0002404480326945295,
+            -0.014118694740134205,
+            0.02004885146995733,
+            0.01694893356262835,
+            -0.0013248884139405398,
+            -0.0051752815372013315,
+            0.008223868147393114,
+            0.007000679132597393,
+            0.005256194890343006,
+            -0.019496098876977206,
+            0.005230597149342019,
+            0.005103909294824105,
+            -0.0014716649591412774,
+            0.010274346323051634,
+            -0.009030594825985526,
+            0.033740380586893826,
+            -0.006505788197604522,
+            0.013065658941136721,
+            -0.0043634731684193745,
+            -0.0055529072007985125,
+            0.001816020676232658,
+            0.013054468478598695,
+            0.009753501030700457,
+            -0.00970446155762939,
+            -0.004172354962129865,
+            0.018697982092775737,
+            -0.011983156146516074,
+            0.0013300791598046338,
+            0.01564072865804035,
+            -0.004858814041438713,
+            0.004708235083775013,
+            0.008635204790345423,
+            -0.004489505237783172,
+            0.01801512858970922,
+            -0.016482572999490366,
+            0.0023953436660054026,
+            0.010694869684962088,
+            0.02390864159830096,
+            -0.019152187046410703,
+            0.021786596486708207,
+            -0.017951050280323393,
+            0.026538582634436314,
+            0.014484894489694262,
+            -0.0035222259457311444,
+            -0.01946200885147006,
+            -0.00850690913394181,
+            0.011654565129063417,
+            -0.021423803876241532,
+            -0.0007985948078290641,
+            -0.0007290222944686444,
+            -0.021367628543092133,
+            -0.011189364017231132,
+            -0.009961779129778331,
+            0.010701637745810138,
+            0.001918633898887013,
+            0.010380983868185359,
+            0.008002182827196827,
+            0.010784760574518087,
+            0.014405640442760822,
+            0.0008958107769384412,
+            -0.010743001445440941,
+            -0.023795589003655273,
+            0.016987708698679597,
+            0.00438463274382093,
+            0.0007862212695139549,
+            -0.01627468994515667,
+            0.011936651269737248,
+            0.013250643815768723,
+            -0.009385605548926286,
+            0.009121882469425302,
+            -0.03716901103204463,
+            -0.004552808410398574,
+            -0.0035704811925430366,
+            -0.0020309197365173475,
+            -0.00908678158053104,
+            0.01559445415882485,
+            -0.013682507836005851,
+            -0.0009590922904844632,
+            0.00011696985057536548,
+            -0.0074756526903548915,
+            -1.5467890432254378e-05,
+            0.020649928837031005,
+            0.008438785939908428,
+            -0.0037382105257999757,
+            0.010376213224654074,
+            0.008826031303751312,
+            -0.013900359449287676,
+            -0.01527474712786084,
+            0.019035366035884273,
+            -0.018776918681908793,
+            0.0063098418544296165,
+            0.01677916431403883,
+            0.020778813096283988,
+            -0.02891862501830075,
+            -0.0007231469961304435,
+            0.022093939332021185,
+            -0.0012807635799950273,
+            0.006370841109790851,
+            0.008781853138377092,
+            0.006754082980602544,
+            0.0029102159062355374,
+            -0.003711173531879644,
+            -4.843770941622283e-06,
+            -0.003940474588075707,
+            0.007155539022862352,
+            0.004366433868349658,
+            0.036453728071874844,
+            -0.013661129348165794,
+            0.002541308805735001,
+            -0.0058838191638835155,
+            -0.00347401304572066,
+            -0.00593469688967835,
+            0.004407665856950772,
+            -0.010055727994798024,
+            0.004330827357536078,
+            -0.001570861535288093,
+            -0.0003731129422346693,
+            0.004883969598300302,
+            -0.001701673379646408,
+            0.007036892564844683,
+            -0.011863650867472625,
+            -0.0003979727750011258,
+            -0.020174851412609312,
+            0.021117284557875267,
+            0.00011666551282792189,
+            0.003662462447326642,
+            0.03723741246290771,
+            0.00476064510662738,
+            -0.00238355131307445,
+            -0.012880057949435798,
+            -0.013551338766321525,
+            -0.012986444917747435,
+            -0.020703821629987552,
+            -0.0036665673548525388,
+            -0.021006311026247164,
+            -0.0052061406234528915,
+            0.010362862594785442,
+            -0.011540184420787572,
+            -0.004023659825622843,
+            -0.015881589631889473,
+            -0.006531981517815174,
+            -0.03114211633687585,
+            -0.01124607488650473,
+            0.007493228366452009,
+            -0.019149285949049886,
+            -0.0039932870926865805,
+            0.026850121590606017,
+            0.001095360635760988,
+            0.01387696424607248,
+            0.0038058248676879625,
+            0.008117686785954085,
+            -0.0045937472017383955,
+            -0.0005669025036141417,
+            -0.01928244877716143,
+            0.003391980330529026,
+            0.008525354854576925,
+            -0.00361507909530282,
+            0.0070008404454106905,
+            -0.008835727374865623,
+            -0.016950017602980382,
+            0.0021654956715219517,
+            0.009334379529385142,
+            -0.006639140003365139,
+            -0.017587472740268683,
+            -0.011049929670847685,
+            0.010401189523318744,
+            -0.02283429470064187,
+            0.006257145380266984,
+            0.016439581169238607,
+            -0.009363030375624585,
+            0.004651575425738153,
+            0.019809183749755425,
+            -0.011810674576781702,
+            -0.005811328501528397,
+            0.008471992070998505,
+            0.013891259999721588,
+            0.0095606134736748,
+            0.02954498207274139,
+            0.0026596809804008996,
+            0.000561870602466584,
+            0.004889263753490895,
+            -0.004277061142504552,
+            -0.017423200644235665,
+            -0.006243787897080996,
+            0.0038507494787521856,
+            0.012873996671952184,
+            -0.009879417960879032,
+            -0.00048575277472139055,
+            0.01682604801673206,
+            0.004530939990187663,
+            -0.0033901248079550284,
+            -0.011868399508738274,
+            -0.020080331108085103,
+            0.0015580259874809898,
+            -0.01362200460317948,
+            0.015354347040143093,
+            0.003390601069331352,
+            0.0015284610287850575,
+            -0.004239880336806429,
+            0.01828645373058079,
+            -0.005013418527935815,
+            0.012749833447770729,
+            -0.006148890994661865,
+            -0.004587831231448011,
+            0.007986533783001116,
+            0.009166578151006778,
+            0.0004939857995863664,
+            0.05421019830078428,
+            0.002838056604877719,
+            0.01333205648597435,
+            0.013419127695838135,
+            -0.014523733369484934,
+            0.014086525147734823,
+            0.012688569853552613,
+            0.018497344547821693,
+            0.013037819487160016,
+            -0.012353776077273462,
+            0.017354801119939964,
+            0.0024084723134891487,
+            0.003909681065813095,
+            0.0022202438064449653,
+            -0.004528576079628071,
+            0.019594683950225115,
+            0.004070222328547378,
+            -0.0074362443729372105,
+            0.004453618452623641,
+            0.005007543168224267,
+            -0.013105524222311019,
+            -0.0112958915514782,
+            0.027105208848863218,
+            -0.01219417557128091,
+            -0.018749971743148304,
+            0.006706594269933445,
+            -0.02008458683853808,
+            -0.03763686535076864,
+            -0.0021332074289244973,
+            0.009992361098878196,
+            0.002795183524495193,
+            -0.00253695604043486,
+            0.01614298064601032,
+            0.016730443047719683,
+            -0.016985296937992726,
+            -0.018245902728910347,
+            0.0025257780765481825,
+            0.0016333325293570182,
+            0.01861807236979066,
+            -0.023168663546030196,
+            -0.006378864850313035,
+            -1.3795465857380962e-05,
+            -0.006829008132963306,
+            0.004850306884033312,
+            0.004703280112191218,
+            -0.0071714759384722965,
+            -0.0027773519096287217,
+            0.002256846645341137,
+            0.01975383140755998,
+            0.013198754584066554,
+            0.01572971337890945,
+            0.002600741959577911,
+            -0.005033508062449482,
+            -0.016817359058724685,
+            -0.005450011991586895,
+            -0.013478330855299777,
+            -0.009091854345715177,
+            -0.003042052092498012,
+            0.008217376329142566,
+            -0.025053469059115735,
+            -0.011965083373220982,
+            0.0119294363235832,
+            0.00391377331459168,
+            0.0003099203772539202,
+            0.007607672728136279,
+            -0.003830132495364688,
+            0.010331053246451333,
+            0.013823890473448383,
+            0.0076112887952959766,
+            -0.004723377809559611,
+            0.009600411526173948,
+            0.011460781436250457,
+            0.006015477068316837,
+            -0.005072711638895421,
+            0.023859899048048694,
+            0.01821920335673539,
+            -0.01680365156728093,
+            -0.002149107383573353,
+            -0.0027829676356423997,
+            -0.01884308738652242,
+            0.009665830801645276,
+            0.00880322398336952,
+            0.008308268605449346,
+            0.003906369351868086,
+            0.00974531700889854,
+            -0.013875904819275925,
+            -0.0050570811397453235,
+            0.002870120865445989,
+            -0.005729556771404781,
+            -0.01901337075686543,
+            0.000297242372701423,
+            -0.011987192261104132,
+            -0.011327157550660895,
+            -0.01409710922165158,
+            -0.003221240464411861,
+            0.005283472511900512,
+            0.021472187640986972,
+            -0.004410881709548286,
+            -0.001855852749039012,
+            0.01351697599517034,
+            0.010060125186506398,
+            0.0021658738823899905,
+            0.0002206474541239167,
+            0.005418828324605768,
+            -0.0027919130944783397,
+            0.0008094042430809068,
+            -0.0015826797905407809,
+            0.021089276765434876,
+            -0.02016611086095107,
+            -0.009413977569213423,
+            -0.009288078098692112,
+            0.005842197094339927,
+            0.028793032462603636,
+            -0.007165634502373454,
+            -0.010104002083017078,
+            -0.01994315293556678,
+            0.0003786527922888834,
+            -0.07810609175343784,
+            -0.005210636756258548,
+            0.001870513618910238,
+            0.03126380113105983,
+            -0.010817888476436834,
+            0.0033673768058629823,
+            -0.002204487854177595,
+            0.014542299655712928,
+            -0.001760107965206874,
+            -0.0013283746982194259,
+            0.0071954806737858056,
+            -0.005640400818010697,
+            -0.006210989097047217,
+            0.005904163661619133,
+            -9.945941028848348e-05,
+            -0.010266938705916303,
+            0.00057263080955709,
+            0.02209498395848068,
+            0.0159398346608872,
+            -0.006575439759804175,
+            0.01816748552021211,
+            -0.00719294553484007,
+            -0.009838738592324234,
+            0.005639288780502127,
+            -0.0019007683459126327,
+            -0.015369619176147573,
+            -0.011475977270803784,
+            0.009862808084641291,
+            -0.030069890644150284,
+            -0.00010725753918890802,
+            0.00943698424646706,
+            -0.013540245857270492,
+            -0.008293042205008774,
+            0.007212974960781723,
+            0.006708343474532583,
+            -0.0012017968151025369,
+            -0.0078026000239507145,
+            -0.02527526731750743,
+            0.0027578400032925625,
+            -0.002446811971857853,
+            -0.004899702349784173,
+            0.0019960465173086183,
+            0.001479503476855694,
+            -0.012645905858466798,
+            0.01216748847452763,
+            -0.013378877368040743,
+            -0.012053251571418766,
+            0.012508842412556306,
+            -0.00027167241745418584,
+            0.01682680930386839,
+            -0.020607299598265426,
+            0.0010702151063620052,
+            -0.028075439824932304,
+            -0.007671848153932823,
+            -0.015486630894165054,
+            -0.0009776136330459065,
+            -0.0012939921569540803,
+            -0.03481880531471923,
+            0.012889447665794033,
+            -0.005200149829198305,
+            -0.006094702500154477,
+            -0.00459778303079732,
+            -0.024583504817899775,
+            0.006072281238752663,
+            0.00607350167867964,
+            -0.006506508970816635,
+            -0.016816439635445445,
+            0.008428048142588516,
+            -0.0011040593013238872,
+            0.004873670535200424,
+            -0.007475800310154908,
+            -0.0005638920132076605,
+            -0.013047135093134946,
+            -0.005463454269764494,
+            -0.0008522562324122834,
+            0.01827123304871433,
+            0.0044055851550317835,
+            -0.014526381222031981,
+            0.016862445236481526,
+            0.013250692156820124,
+            0.01602714293656341,
+            -0.0071089293427753115,
+            -0.0004265739315250755,
+            -0.02750630108787212,
+            -0.00327225092954903,
+            -0.023687644154808972,
+            0.00041213204365507744,
+            0.009447660230507965,
+            0.0203905264889617,
+            0.012694421477479161,
+            0.004620617778996077,
+            -0.001557511764988156,
+            0.0066751090724502,
+            0.02696959560423252,
+            -0.01099183126742224,
+            -0.006969111598423376,
+            -0.01884504192789845,
+            0.01113023717919988,
+            0.025135748541199255,
+            0.003381303139273845,
+            -0.011317957771935042,
+            0.008112598754655146,
+            -0.0009247845526290428,
+            -0.010302980598863016,
+            0.02889587382004902,
+            -0.0391134481322538,
+            -0.015599591108049343,
+            -0.004605824192962023,
+            0.007406520608110251,
+            -0.015596917115435033,
+            0.00770468362805995,
+            -0.012614339656105264,
+            0.001699049329817941,
+            -0.005851477164035241,
+            0.03535087340659861,
+            -0.0031950960727572925,
+            0.0006909856233673865,
+            -0.003395233050301427,
+            0.003531135044131077,
+            0.012355948349937309,
+            -0.004444221317023453,
+            -0.030726300006909638,
+            -0.0022983766869702346,
+            -0.011198252640110788,
+            0.009274666307515955,
+            -0.016207714291658314,
+            -0.017342435976846468,
+            -0.009424737744066259,
+            -0.021827243671837953,
+            -0.012403084982789827,
+            0.029747453318486338,
+            -0.005824001042844527,
+            0.01747569398142269,
+            -0.005219060548639168,
+            -0.008365003057800934,
+            0.00021272411948388247,
+            0.01344335658390732,
+            -0.028498240040284714,
+            0.00041963499785238845,
+            -0.002322931702641349,
+            0.0126393550661767,
+            -0.001595134597082556,
+            -0.00983073906977773,
+            0.006903313245694627,
+            0.0010522208685011098,
+            -0.015648940549713764,
+            -0.0007573245570947632,
+            -0.008533504878052644,
+            0.020475329445149498,
+            0.00027791387213409814,
+            0.001074213629646005,
+            0.013864331671732542,
+            -0.02102352993295736,
+            0.01576938071930521,
+            -0.021722775924283996,
+            0.01893267442161547,
+            -0.006577103101038035,
+            0.005772016664149475,
+            0.008283223343661555,
+            0.014068789332250016,
+            -0.014030042849491258,
+            -0.00663752723080222,
+            -0.034436220972634726,
+            -0.0122621879114314,
+            -0.001946072976826902,
+            -0.005308941654794638,
+            -0.014773123927909993,
+            -0.0011054822266082056,
+            -0.026205461509573584,
+            -0.006460373065662098,
+            0.012822188046808264,
+            0.019900312098359022,
+            -0.008409070094265829,
+            0.014163132434889265,
+            -0.006743711872055091,
+            -0.0028230771810882492,
+            -0.012652021749331811,
+            -0.018061180380009395,
+            -0.01083672324723357,
+            -0.01890966636827258,
+            -0.0018635408070944168,
+            0.004709222372611061,
+            -0.015760618199235518,
+            0.011435131737642627,
+            0.03198874888333405,
+            -0.01567271472650171,
+            0.008243406188694909,
+            0.016411496328333205,
+            0.0034236971031973503,
+            0.011728079139322837,
+            0.0038311415233052795,
+            0.004621789116267879,
+            -0.011109450058055078,
+            -0.011473158133621661,
+            0.006524490579223986,
+            0.00954511918454185,
+            -0.004913142601134428,
+            0.024347072255007807,
+            0.0008745419206472714,
+            0.005045794664611216,
+            0.005304506902446091,
+            0.011488187043525824,
+            -0.013135271908307747,
+            0.019746484188715552,
+            -0.002055210086078267,
+            -0.015571665869889405,
+            -0.01535499169234956,
+            -0.01105419264826655,
+            -0.018626399757822328,
+            0.0017108083455869296,
+            0.0181439079855921,
+            -0.0012493542718277685,
+            0.012137753522284387,
+            -0.00924460959150499,
+            -0.005019437549649743,
+            -0.0027156377206998474,
+            -0.038211954658821295,
+            -0.0021878970332906457,
+            -0.018809487178512628,
+            -0.02498945889484935,
+            0.03505216568400238,
+            -4.26477081213836e-05,
+            0.00997643255528116,
+            -0.010310197159232386,
+            -0.0002454902098986129,
+            -0.011065403358802515,
+            0.014954998647970473,
+            0.004624123403100598,
+            0.008872923678983478,
+            0.008560280683876702,
+            -0.016419500515801093,
+            -0.002997557798375716,
+            0.007472390947008284,
+            -0.00374822278027057,
+            0.012680407710694952,
+            0.015353684671120731,
+            0.020886474047003733,
+            0.011161262397502876,
+            0.002045862158732102,
+            0.00887271411363814,
+            0.009563361150297699,
+            -0.0003513237086717377,
+            -0.010459621030424415,
+            0.015419039131459582,
+            -0.02204848739950703,
+            0.012172257374410352,
+            0.04346634265794738,
+            0.019197433209794286,
+            0.029488039318675732,
+            -0.02272810439246353,
+            0.01448092574459596,
+            -0.003801043409441206,
+            0.009476229966122152,
+            0.0049309968883087685,
+            0.02346191840355209,
+            0.003973840862678076,
+            0.0073546443568171405,
+            0.010647581332452525,
+            -0.00972051412996416,
+            0.025644022498404723,
+            -0.00846669916801108,
+            5.1518540791212095e-05,
+            -0.004407265804145216,
+            -0.026400029105519137,
+            0.01615193008014211,
+            0.0081722536401398,
+            0.0010702314763197463,
+            0.0010263623479474385,
+            -0.0019566604229607643,
+            0.007982949510899019,
+            0.007079322685771456,
+            -0.015274148711633536,
+            -0.003886053737590683,
+            -0.018527455865296014,
+            0.0012864569064749438,
+            -0.024337133883771334,
+            0.010466651250597847,
+            -0.017568706814132484,
+            -0.014822354159709854,
+            -0.021666681078831483,
+            -0.013232016867264097,
+            -0.02011653224168991,
+            0.0035858355517491515,
+            -0.008632755352062642,
+            0.013025728891203198,
+            -0.016962939287139683,
+            -0.021494971929779273,
+            0.008430887589398292,
+            0.00870802359711456,
+            0.007481282677955534,
+            0.009728368603458082,
+            0.014996157358749115,
+            0.01736292711410622,
+            -0.022712386143092265,
+            -0.023163001611599246,
+            -0.006682209717364464,
+            -0.003531495447167166,
+            0.006762379833125297,
+            0.00933681196321128,
+            0.012884185500343249,
+            -0.007077895849501418,
+            -0.007554762054641417,
+            -0.013283067570021503,
+            0.006197966297581871,
+            -0.003993067576677973,
+            -0.009464789349849612,
+            -0.0015662739825834614,
+            -0.0061389795593180686,
+            0.002753664732282926,
+            -0.0001791088888074743,
+            0.0007595261237141144,
+            0.017122834676182343,
+            -0.008176319370751018,
+            -0.009787747261447871,
+            -0.02027228453943018,
+            -0.010683273300243025,
+            -0.015604745521156111,
+            -0.0016279406232527138,
+            -0.006506713191861036,
+            -0.014420323628589382,
+            0.010888206569219648,
+            -0.011930708571093279,
+            -0.00887210117616784,
+            0.010807147705360856,
+            0.0073900910661282935,
+            -0.003431679006289002,
+            0.037804020313161775,
+            0.0011945597874820288,
+            0.00778778394048267,
+            -0.003366543306746973,
+            0.005296727102519836,
+            -0.017761251541598876,
+            -0.026382400396890007,
+            0.0010357716464853195,
+            -0.01775777668381489,
+            0.010807713750049491,
+            -0.008515368407459745,
+            0.0016051389544388124,
+            -0.00034958600424912995,
+            -0.009585854918521076,
+            0.0036549720706351473,
+            -0.008110927907950148,
+            0.004212219431447508,
+            -0.03507890639477636,
+            0.011409377585617926,
+            -0.01819120811524444,
+            -0.008962242411035718,
+            0.01710177594046791,
+            0.037560936862121835,
+            0.0028513388998398104,
+            0.00030058353811717895,
+            0.0014792184333546447,
+            0.004427921639326168,
+            0.021438127785303025,
+            0.006339483221420334,
+            -0.001187747087595264,
+            -0.02237982599829809,
+            0.0062435650294483695,
+            0.002200578793697352,
+            0.010092828474934744,
+            0.023105023225115266,
+            -0.016797326021070008,
+            -0.015378520287395565,
+            -0.012732527281000097,
+            -0.010797463778182385,
+            0.009881499418260473,
+            0.02801767062785128,
+            0.01811139137849365,
+            -0.017358098300028903,
+            -0.020207173948076473,
+            0.0016242901745373126,
+            -0.026331540602868762,
+            0.007269751661479859,
+            0.11940631114242901,
+            -0.0045382135734049305,
+            -0.005994407567723939,
+            -0.008679712231646732,
+            -0.004482553208064535,
+            0.016709571294385293,
+            -0.013387043582971358,
+            0.069179848974036,
+            -0.007221545718220735,
+            0.0037392229364642915,
+            0.0028916224964057385,
+            0.00462729158956839,
+            -0.009727339449635306,
+            0.011570145487622916,
+            -0.011287365791681587,
+            -0.012991994493845187,
+            -0.013076194601468383,
+            -0.0180830261130804,
+            -0.007272768410218907,
+            -0.004155564814758654,
+            0.004204993170487392,
+            -0.005525575983182816,
+            0.008051108079520902,
+            -0.017551645696802648,
+            -0.010800070994485792,
+            0.013007176804761548,
+            -0.0023627485852619817,
+            0.005390986239687555,
+            -0.029668390278675554,
+            -0.0007650391259819867,
+            -0.0002446394046947245,
+            0.021280292985495608,
+            0.003137771886236958,
+            -0.0019077298815862946,
+            -0.010810159675564049,
+            0.007040107942856957,
+            0.023835993694726953,
+            0.021875366566457236,
+            0.0180511566130312,
+            0.012135351541044354,
+            0.020483975362807574,
+            -0.008667079096167644,
+            0.026324312888512866,
+            -0.001552207146726937,
+            -0.00397810465151493,
+            0.0077577136352257004,
+            -0.009849600842081875,
+            -0.00816392268697345,
+            0.00677353736462267,
+            -0.017288900282504148,
+            -0.020418071729964713,
+            -0.020383762873109595,
+            0.025741408097358233,
+            -0.011755094311186617,
+            0.002737129201159793,
+            -0.008266210469668866,
+            0.027287436144516414,
+            -0.006283892307996516,
+            -0.0029440642954701632,
+            0.009428876742171362,
+            -0.03788574758692254,
+            0.008834605607057885,
+            0.006463342392514337,
+            0.026711196829464153,
+            -3.9827332346130654e-05,
+            0.005565561841469247,
+            -0.0006738085601962865,
+            -0.006104320995853506,
+            -0.010792830169647851,
+            2.4565453208679464e-05,
+            0.017983908748172493,
+            0.0035181716362611077,
+            0.016970206528324653,
+            -0.005449808667944701,
+            -0.023171388778833197,
+            0.007092509502738752,
+            -0.01555320330471131,
+            -0.023306285898839978,
+            -0.002028783456704567,
+            -0.03196436161255752,
+            0.0033412618908711163,
+            0.007527633574188688,
+            -0.01890570959143866,
+            -0.017859735533232564,
+            0.010329340599904344,
+            0.004382396539605985,
+            -0.0016746397858953155,
+            0.006643785064338525,
+            -0.03307812909454471,
+            -0.01927475592253288,
+            0.004686970010943116,
+            0.012733525168279355,
+            -0.012993441126970916,
+            0.015291832952199082,
+            -0.020491167895268936,
+            0.016451976170508816,
+            -0.012259387189535237,
+            0.018393274368854132,
+            -0.006997668664226754,
+            0.011843450716200625,
+            0.011457415642860215,
+            0.009560877634897626,
+            -0.01790001115302499,
+            0.0214879415944361,
+            0.0066829070506954115,
+            0.03900547021123392,
+            0.02512484499576633,
+            -0.012651294056575184,
+            -0.0018425680226742782,
+            -0.00011361458320146007,
+            0.014982647266052349,
+            0.0061874424115559,
+            -0.003689464024837091,
+            0.02705902205734107,
+            0.018239469373522537,
+            -0.021070704644859155,
+            0.020037508334843183,
+            -0.00973804771680803,
+            0.018876283979591423,
+            -0.013303080480396818,
+            0.010592424494628194,
+            0.004717870260467274,
+            0.01070131238869639,
+            0.009087656015197565,
+            0.006708673669681232,
+            0.025928065153829043,
+            0.03663367449576964,
+            -0.00511073108390551,
+            0.002161381031392467,
+            0.004864092591944181,
+            0.012814518637357098,
+            0.001556833001747549,
+            0.007915766323215207,
+            -0.007012785514562176,
+            0.005834786334750152,
+            -0.002311533032619821,
+            -0.00122162928193159,
+            0.019525999280692766,
+            0.000824163362963678,
+            0.012875575199417254,
+            0.013259376059185821,
+            -0.002390133687709065,
+            -0.005682644446302278,
+            0.0019262218417782657,
+            -0.018193478045923098,
+            -0.015131696847666434,
+            0.029463723011600933,
+            -0.011447039651934148,
+            -0.008083114389964213,
+            0.0020642857121204753,
+            0.012160202510990566,
+            -0.00866944494225225,
+            -0.005378300635997713,
+            0.0038280364584418335,
+            0.014427318067525318,
+            -0.01799358399064332,
+            0.003381165206832161,
+            0.005252552821854107,
+            0.019433554824232185,
+            -0.00028791985674147885,
+            -0.013487678738391956,
+            0.0005317922815671783,
+            0.014982152801567637,
+            0.011902348165369078,
+            0.0016056390299304837,
+            -0.006935777197137747,
+            -0.00446584737370504,
+            0.004586814866182051,
+            0.006269466956110759,
+            0.0027516492952993445,
+            0.008157662045793032,
+            -0.0025001885993292637,
+            -0.016648747436203294,
+            0.017735757469839637,
+            -0.0030669873770852725,
+            0.018058248627079793,
+            -0.0038238312452176745,
+            -0.0147172470474118,
+            0.0043328417868095355,
+            -0.001888507803054499,
+            -0.018556943283494755,
+            0.009407930200720534,
+            0.002560949424273885,
+            0.010411159836451958,
+            0.0052571230238294825,
+            -0.007499470181994448,
+            -6.145889479758644e-05,
+            -0.015163819825329975,
+            0.022836682800304944,
+            0.016390944816242627,
+            0.001713572907204073,
+            -0.0035687505973373878,
+            -0.031210299824387475,
+            0.0029076511444359886,
+            -0.007822231980296341,
+            0.017821806420465356,
+            0.008250605126658402,
+            0.007908773724009905,
+            0.008351355849444162,
+            -0.00743970642051451,
+            -0.002411981861500092,
+            0.012543476434788293,
+            0.0202407777715137,
+            -0.0024635818448627447,
+            0.02535771413579989,
+            -0.012533449594824386,
+            -0.0022489292647974992,
+            0.0012362917358624392,
+            0.01578939075186126,
+            -0.0008001783766371555,
+            0.006311837812042848
+          ],
+          "keyphrases": [
+            "Switch Transformers",
+            "Mixture of Experts",
+            "sparsely-activated model",
+            "training instability",
+            "multilingual settings"
+          ],
+          "title": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "9204ecdb-832f-4b40-b0fe-5cc34ec75d3b",
+        "properties": {
+          "page_content": "6 Related Work 24 ",
+          "keyphrases": [
+            "Related Work"
+          ],
+          "embedding": [
+            0.002109470311552286,
+            0.024481380358338356,
+            -0.01675625890493393,
+            0.01014935877174139,
+            0.03144679218530655,
+            0.0017690955428406596,
+            0.0074690403416752815,
+            0.09587687253952026,
+            0.01939389668405056,
+            0.014093012548983097,
+            0.001160902320407331,
+            -0.013068686239421368,
+            -0.015330738388001919,
+            -0.026103228330612183,
+            0.03235161304473877,
+            0.036431845277547836,
+            -0.005620986223220825,
+            -0.00097150873625651,
+            -0.033068642020225525,
+            0.002234310144558549,
+            0.0004380058671813458,
+            0.024430163204669952,
+            -0.015834365040063858,
+            0.04742627218365669,
+            0.02104988880455494,
+            0.008480561897158623,
+            0.051182132214307785,
+            0.01811349019408226,
+            -0.03865121677517891,
+            0.0019728937186300755,
+            -0.008903096430003643,
+            0.010678594000637531,
+            8.36265753605403e-05,
+            -0.0034208204597234726,
+            -0.028698187321424484,
+            -0.018420787528157234,
+            0.014630783349275589,
+            0.01811349019408226,
+            -0.0005980567075312138,
+            0.012718708254396915,
+            -0.036909863352775574,
+            -0.027776293456554413,
+            -0.030541973188519478,
+            0.0029363997746258974,
+            0.03361494839191437,
+            -0.029739584773778915,
+            -0.01758425496518612,
+            -0.044626448303461075,
+            0.005258204415440559,
+            0.03020053170621395,
+            0.029500575736165047,
+            -0.023098541423678398,
+            0.01559535600244999,
+            -0.0012782729463651776,
+            -0.01341866422444582,
+            0.003841220634058118,
+            -0.0015738335205242038,
+            0.07300026714801788,
+            0.023798497393727303,
+            0.019069526344537735,
+            0.001734951394610107,
+            -0.016662361100316048,
+            -0.0010467326501384377,
+            -0.03286377713084221,
+            0.009978638030588627,
+            -0.01825006678700447,
+            0.018045201897621155,
+            -0.005023463163524866,
+            0.020264573395252228,
+            -0.008489097468554974,
+            -0.0041101062670350075,
+            -0.007831822149455547,
+            -0.026905616745352745,
+            0.05551844462752342,
+            -0.0061459532007575035,
+            0.00418479647487402,
+            0.042850952595472336,
+            0.027008049190044403,
+            -0.015689251944422722,
+            0.04995294287800789,
+            -0.005655130371451378,
+            -0.024703318253159523,
+            -0.012863821350038052,
+            -0.01522830594331026,
+            0.019701194018125534,
+            -0.005778903141617775,
+            -0.056850068271160126,
+            -0.13206970691680908,
+            0.004724701400846243,
+            -0.0011918453965336084,
+            0.011318797245621681,
+            0.014229589141905308,
+            -0.002349546644836664,
+            -0.03841220587491989,
+            -0.051182132214307785,
+            0.04595807194709778,
+            -0.008531778119504452,
+            -0.021391332149505615,
+            0.009551835246384144,
+            -0.01522830594331026,
+            0.008096439763903618,
+            0.016414817422628403,
+            0.0179598405957222,
+            -0.03494657203555107,
+            0.02656417526304722,
+            -0.03824148699641228,
+            -0.005292348563671112,
+            -0.00846775807440281,
+            0.026291022077202797,
+            0.03970968723297119,
+            -0.02081087976694107,
+            -0.025249624624848366,
+            -0.033444229513406754,
+            0.004285095259547234,
+            -0.0017040082020685077,
+            -0.029995666816830635,
+            0.011079788208007812,
+            -0.046162936836481094,
+            -0.021203538402915,
+            0.012360194697976112,
+            0.020486511290073395,
+            -0.0011032839538529515,
+            0.004673485178500414,
+            -0.02465210109949112,
+            -0.02274002693593502,
+            0.021664485335350037,
+            0.003265037667006254,
+            -0.01618434302508831,
+            -0.01535634696483612,
+            0.0282543133944273,
+            0.03605625778436661,
+            -0.00591121194884181,
+            0.0010622042464092374,
+            -0.03387103229761124,
+            0.01354670524597168,
+            -0.018574437126517296,
+            0.011933392845094204,
+            0.03660256415605545,
+            -0.022022999823093414,
+            -0.004733237437903881,
+            0.005288080777972937,
+            -0.019359752535820007,
+            -0.0030473684892058372,
+            0.022415656596422195,
+            -0.007490380667150021,
+            0.014912472106516361,
+            -0.03055904433131218,
+            0.012991862371563911,
+            -0.017789119854569435,
+            0.04548005387187004,
+            -0.03424661606550217,
+            0.017319638282060623,
+            -0.018284210935235023,
+            0.032027244567871094,
+            -0.019735338166356087,
+            -0.0157319325953722,
+            0.009287218563258648,
+            0.0017146782483905554,
+            0.016355063766241074,
+            -0.02403750643134117,
+            0.04097302258014679,
+            0.023320477455854416,
+            0.024447236210107803,
+            -0.01644042506814003,
+            0.021630341187119484,
+            -0.03378567099571228,
+            0.015211233869194984,
+            -0.014681999571621418,
+            -0.017908625304698944,
+            0.026512959972023964,
+            0.02734949253499508,
+            0.0033183880150318146,
+            0.018420787528157234,
+            0.0003475771227385849,
+            -0.026683680713176727,
+            -0.007976935245096684,
+            -0.007575741037726402,
+            0.03168580308556557,
+            -0.013956435024738312,
+            0.003331192070618272,
+            -0.021288899704813957,
+            0.03762689232826233,
+            -0.0013903084909543395,
+            -0.00194301747251302,
+            0.011071251705288887,
+            -0.013393056578934193,
+            0.029381070286035538,
+            -0.030132243409752846,
+            0.041314464062452316,
+            0.024378947913646698,
+            -0.0024221031926572323,
+            -0.019052455201745033,
+            0.01622702367603779,
+            0.012752852402627468,
+            0.005198452156037092,
+            -0.057737816125154495,
+            0.011199292726814747,
+            -0.006252653896808624,
+            -0.03568067401647568,
+            -0.021220611408352852,
+            -0.00606912886723876,
+            -0.022057143971323967,
+            0.007443432230502367,
+            0.004575320519506931,
+            -0.025847148150205612,
+            -0.009201857261359692,
+            -0.0024583812337368727,
+            0.002289794385433197,
+            -0.008659819141030312,
+            -0.03516850993037224,
+            0.00754159688949585,
+            0.0003875898546539247,
+            0.0006898192223161459,
+            -0.016406280919909477,
+            -0.0011064850259572268,
+            -0.023559486493468285,
+            -0.0224839448928833,
+            0.06524953991174698,
+            -0.0035787373781204224,
+            -0.0295347198843956,
+            0.03892436996102333,
+            -0.0030708424746990204,
+            0.007925719022750854,
+            0.008809199556708336,
+            -0.02647881582379341,
+            -0.023439982905983925,
+            -0.0022663201671093702,
+            0.05275276303291321,
+            0.0034955109003931284,
+            -0.01458810269832611,
+            0.0006594095611944795,
+            0.008292769081890583,
+            -0.004041817970573902,
+            -0.052616186439991,
+            -0.0314980112016201,
+            -0.01964997872710228,
+            -0.00800254289060831,
+            -0.030354179441928864,
+            -0.012300441972911358,
+            0.02801530249416828,
+            0.030763909220695496,
+            -0.0016613280167803168,
+            -0.042953383177518845,
+            0.017123308032751083,
+            -0.02466917410492897,
+            -0.020349934697151184,
+            -0.03163458779454231,
+            0.05312835052609444,
+            0.019871914759278297,
+            -0.02620566077530384,
+            0.0058045112527906895,
+            0.012616275809705257,
+            -0.009048208594322205,
+            0.02079380862414837,
+            0.019342679530382156,
+            0.09280389547348022,
+            0.026666607707738876,
+            -0.0018149768002331257,
+            -0.0026653804816305637,
+            -0.015211233869194984,
+            -0.015194161795079708,
+            -0.01675625890493393,
+            -0.04561663046479225,
+            -0.021783988922834396,
+            0.005377708934247494,
+            -0.0013038810575380921,
+            0.039368245750665665,
+            0.0044302078895270824,
+            0.022415656596422195,
+            -0.01001278217881918,
+            0.024686245247721672,
+            -0.013145511038601398,
+            -0.002503195544704795,
+            -0.02622273378074169,
+            -0.009603051468729973,
+            -0.030115170404314995,
+            -0.003077244618907571,
+            -0.015142945572733879,
+            0.016372136771678925,
+            -0.004588124807924032,
+            -0.009893277660012245,
+            0.04466059431433678,
+            0.008937240578234196,
+            -0.018984166905283928,
+            -0.021527908742427826,
+            -0.03557823970913887,
+            0.009449402801692486,
+            -0.0026995246298611164,
+            -0.0031199248041957617,
+            -0.019564617425203323,
+            0.01213825773447752,
+            -0.027912870049476624,
+            -0.040529146790504456,
+            0.044899601489305496,
+            0.006828837096691132,
+            -0.025198407471179962,
+            -0.01381132286041975,
+            0.006905661430209875,
+            -0.0014351228019222617,
+            -0.026410527527332306,
+            -0.001253731781616807,
+            0.012838213704526424,
+            -0.00121211854275316,
+            0.007302587386220694,
+            -0.019718267023563385,
+            0.008331180550158024,
+            0.02106696180999279,
+            0.03177116438746452,
+            -0.029978593811392784,
+            -0.026871472597122192,
+            0.01149805448949337,
+            -0.04868960753083229,
+            -0.0007074248278513551,
+            -0.04063158109784126,
+            0.012069969438016415,
+            0.024430163204669952,
+            0.005902675911784172,
+            0.021732773631811142,
+            -0.02874940261244774,
+            -0.04612879455089569,
+            0.0013113500317558646,
+            0.01078102644532919,
+            -0.04353383556008339,
+            0.024481380358338356,
+            -0.014724679291248322,
+            0.022296153008937836,
+            -0.003619283437728882,
+            0.04059743508696556,
+            0.01580875739455223,
+            -0.010763954371213913,
+            0.03006395511329174,
+            -0.00937257893383503,
+            -6.443714482884388e-06,
+            0.018967093899846077,
+            -0.011941928416490555,
+            -0.03156629949808121,
+            -0.03557823970913887,
+            0.015475851483643055,
+            0.007567205000668764,
+            -0.017823264002799988,
+            0.009662804193794727,
+            0.01554413978010416,
+            0.007511720526963472,
+            0.02451552450656891,
+            0.019086599349975586,
+            -0.05367465689778328,
+            0.035373374819755554,
+            -0.04029013589024544,
+            -0.012923574075102806,
+            0.007780605927109718,
+            0.0009704417316243052,
+            -0.03554409742355347,
+            -0.0024370411410927773,
+            0.012104113586246967,
+            -0.014238124713301659,
+            -0.047221407294273376,
+            -0.017703760415315628,
+            -0.029244493693113327,
+            0.020076779648661613,
+            -0.021801061928272247,
+            0.00022700548288412392,
+            -0.04022184759378433,
+            -0.009125033393502235,
+            -0.016022158786654472,
+            -0.05278690904378891,
+            -0.036670852452516556,
+            0.024464307352900505,
+            -0.005795975215733051,
+            -0.02145962044596672,
+            -0.0009016198455356061,
+            -0.059410881251096725,
+            0.007989739067852497,
+            0.0016175807686522603,
+            -0.006777620408684015,
+            -0.025095975026488304,
+            -0.011148076504468918,
+            0.014571030624210835,
+            0.017635472118854523,
+            -0.0397779755294323,
+            0.005680738482624292,
+            -0.01373449806123972,
+            0.01014935877174139,
+            0.005706346593797207,
+            -0.017635472118854523,
+            -0.024088721722364426,
+            0.0036897058598697186,
+            -0.0037665304262191057,
+            0.01168584730476141,
+            0.0030644405633211136,
+            -0.010678594000637531,
+            0.02480575069785118,
+            -0.04520690068602562,
+            0.003469902789220214,
+            -0.03632941097021103,
+            0.039880406111478806,
+            -0.006495931185781956,
+            0.02992737665772438,
+            0.015100265853106976,
+            -0.0030793785117566586,
+            -0.005181380081921816,
+            -0.0501236617565155,
+            0.005595378112047911,
+            0.005138699896633625,
+            0.006700796075165272,
+            0.04435329511761665,
+            -0.003719581989571452,
+            -0.017925696447491646,
+            0.01232605054974556,
+            0.016628216952085495,
+            -0.011643166653811932,
+            0.03168580308556557,
+            0.015023441053926945,
+            -0.000559644540771842,
+            -0.011216364800930023,
+            0.03574896231293678,
+            -0.002466917270794511,
+            0.009150641039013863,
+            0.061220522969961166,
+            -0.023542415350675583,
+            -0.010311543941497803,
+            -0.004592392593622208,
+            -0.008732374757528305,
+            0.07368315011262894,
+            -0.009611587971448898,
+            -0.008472025394439697,
+            0.0179598405957222,
+            -0.015663644298911095,
+            -0.011916320770978928,
+            0.01600508578121662,
+            -0.02569349855184555,
+            -0.02169862948358059,
+            0.012505307793617249,
+            0.026273949071764946,
+            -0.021101105958223343,
+            0.028032375499606133,
+            -0.015689251944422722,
+            -0.011139540001749992,
+            0.01201021671295166,
+            -0.016329456120729446,
+            -0.012300441972911358,
+            0.035407520830631256,
+            -0.004592392593622208,
+            0.03366616740822792,
+            0.03660256415605545,
+            0.006192901637405157,
+            -0.03131021559238434,
+            -0.04548005387187004,
+            -0.010670057497918606,
+            -0.03480999544262886,
+            0.017413534224033356,
+            -0.0003048969083465636,
+            0.013691818341612816,
+            -0.04469473659992218,
+            -0.00844641774892807,
+            0.05510871484875679,
+            -0.02209128811955452,
+            0.029090845957398415,
+            0.04343140125274658,
+            -0.03185652568936348,
+            -0.02412286587059498,
+            -0.021510835736989975,
+            0.08549703657627106,
+            0.040665723383426666,
+            -0.010055461898446083,
+            -0.0359879694879055,
+            0.01592826284468174,
+            -0.026666607707738876,
+            -0.025078903883695602,
+            2.39409419009462e-05,
+            0.04339725896716118,
+            0.00754159688949585,
+            -0.015527067705988884,
+            -0.017908625304698944,
+            0.01937682554125786,
+            -0.033853959292173386,
+            0.001105417963117361,
+            0.019991420209407806,
+            0.008087903261184692,
+            -0.04510446637868881,
+            0.012069969438016415,
+            0.017891552299261093,
+            0.02031579054892063,
+            -0.02504475973546505,
+            0.015322202816605568,
+            0.026512959972023964,
+            0.027451924979686737,
+            0.027947014197707176,
+            -0.005676470696926117,
+            0.015339274890720844,
+            -0.006965413689613342,
+            0.03865121677517891,
+            -0.0018619250040501356,
+            -0.007840358652174473,
+            0.01695258729159832,
+            -0.029483502730727196,
+            -0.04005112871527672,
+            -0.017080627381801605,
+            -0.07129305601119995,
+            0.015834365040063858,
+            0.027178769931197166,
+            -0.014272268861532211,
+            -0.00818180013448,
+            -0.008058027364313602,
+            0.0450020357966423,
+            0.010251791216433048,
+            -0.021869350224733353,
+            0.01799398474395275,
+            -0.018950022757053375,
+            -0.01213825773447752,
+            0.013008934445679188,
+            0.028220169246196747,
+            0.016090447083115578,
+            0.019479257985949516,
+            -0.0237472802400589,
+            0.003971395548433065,
+            0.006991021800786257,
+            0.012658956460654736,
+            -0.0007340999436564744,
+            -0.01065298542380333,
+            0.030115170404314995,
+            0.0077379257418215275,
+            0.0027976890560239553,
+            0.001323087140917778,
+            0.04855303093791008,
+            -0.03187359496951103,
+            0.008971384726464748,
+            -0.010584697127342224,
+            -0.02236444130539894,
+            0.05346979200839996,
+            -0.003363202093169093,
+            -0.012744316831231117,
+            0.031259000301361084,
+            0.027588501572608948,
+            -0.013316231779754162,
+            -0.0118821756914258,
+            -0.0011043510166928172,
+            -0.027008049190044403,
+            0.020725520327687263,
+            -0.02067430317401886,
+            -0.0068587129935622215,
+            0.01007253397256136,
+            0.016824547201395035,
+            0.011660238727927208,
+            -0.017754975706338882,
+            -0.011848031543195248,
+            0.05125042051076889,
+            0.041280318051576614,
+            -0.010883458890020847,
+            0.007140402682125568,
+            -0.0028446372598409653,
+            0.014161300845444202,
+            0.028442105278372765,
+            0.006534343119710684,
+            -0.003913776949048042,
+            -0.002244980074465275,
+            -0.010951747186481953,
+            -0.03236868605017662,
+            0.019957276061177254,
+            0.025864219292998314,
+            0.012223618105053902,
+            0.028100663796067238,
+            -0.03779761120676994,
+            -0.00943233072757721,
+            -0.0053819771856069565,
+            0.006555683445185423,
+            -0.04670924320816994,
+            0.014101548120379448,
+            0.021544979885220528,
+            -0.03520265221595764,
+            -0.021408403292298317,
+            -0.0025330716744065285,
+            0.011259044520556927,
+            -0.03096877597272396,
+            -0.035509951412677765,
+            -0.032163821160793304,
+            0.04742627218365669,
+            -0.0021820268593728542,
+            -0.0063508180901408195,
+            -0.04059743508696556,
+            -0.01738792657852173,
+            0.015646573156118393,
+            -0.02429358661174774,
+            -0.006760548334568739,
+            0.025744715705513954,
+            -0.006133148912340403,
+            0.006167293526232243,
+            -0.024447236210107803,
+            -0.015262450091540813,
+            -0.03622698038816452,
+            0.02878354676067829,
+            0.012846749275922775,
+            0.0025800198782235384,
+            -0.01966704986989498,
+            -0.024327730759978294,
+            -0.025539850816130638,
+            0.004728969652205706,
+            0.010891994461417198,
+            0.0009005528409034014,
+            0.0289030522108078,
+            -0.05428925156593323,
+            0.030729765072464943,
+            0.006235581822693348,
+            -0.026239804923534393,
+            0.006389230489730835,
+            0.010644449852406979,
+            -0.023030253127217293,
+            0.003160471096634865,
+            0.01677333004772663,
+            -0.009346970357000828,
+            0.011446838267147541,
+            0.02555692195892334,
+            -0.014093012548983097,
+            0.048484742641448975,
+            -0.004524104297161102,
+            0.009918885305523872,
+            -0.025915436446666718,
+            0.041690047830343246,
+            0.008587262593209743,
+            -0.0009405656019225717,
+            0.0015044781612232327,
+            0.03892436996102333,
+            -0.01886466145515442,
+            -0.0224839448928833,
+            -0.0112334368750453,
+            0.013828394934535027,
+            -0.033973462879657745,
+            -0.008886024355888367,
+            -0.007848894223570824,
+            -0.017439141869544983,
+            -0.011933392845094204,
+            0.005783171392977238,
+            0.010746882297098637,
+            0.005492945667356253,
+            0.010635913349688053,
+            0.018011057749390602,
+            0.008143387734889984,
+            0.038104910403490067,
+            -0.009346970357000828,
+            0.03687571734189987,
+            -0.012360194697976112,
+            -0.01600508578121662,
+            -0.016858691349625587,
+            0.019223175942897797,
+            0.041177887469530106,
+            -0.040255993604660034,
+            0.029568864032626152,
+            -0.014442989602684975,
+            0.0147673599421978,
+            0.005253936164081097,
+            0.01838664337992668,
+            0.011165148578584194,
+            -0.003881766926497221,
+            -0.0050405352376401424,
+            0.022893676534295082,
+            0.011020035482943058,
+            -0.00015231507131829858,
+            -0.0012836079113185406,
+            -0.02425944246351719,
+            -0.0474945604801178,
+            0.020127996802330017,
+            -0.012940646149218082,
+            -0.022808315232396126,
+            -0.005467337556183338,
+            0.010490800254046917,
+            0.005224060267210007,
+            0.029722511768341064,
+            0.015211233869194984,
+            0.005224060267210007,
+            -0.014442989602684975,
+            0.006577023304998875,
+            -0.004024745896458626,
+            -0.035885538905858994,
+            0.0050917514599859715,
+            0.020503582432866096,
+            -0.03131021559238434,
+            -0.018403714522719383,
+            -0.004159188363701105,
+            0.03337594121694565,
+            -0.0031113887671381235,
+            0.004270156845450401,
+            -0.016568465158343315,
+            -0.004618000704795122,
+            -0.021118177101016045,
+            0.017217205837368965,
+            0.002994018141180277,
+            0.01503197755664587,
+            -0.010542016476392746,
+            0.0003016958653461188,
+            0.006623971741646528,
+            -0.00018725951667875051,
+            -0.0041762604378163815,
+            0.005744758993387222,
+            -0.006700796075165272,
+            -0.0016666629817336798,
+            0.0013561643427237868,
+            0.010695666074752808,
+            -0.014272268861532211,
+            0.00950915552675724,
+            0.007405019830912352,
+            0.013512561097741127,
+            -0.02799823135137558,
+            -0.025761786848306656,
+            -0.055689163506031036,
+            0.01675625890493393,
+            -0.009355506859719753,
+            -0.011284653097391129,
+            -0.006585559342056513,
+            -0.006081932689994574,
+            -0.025130119174718857,
+            -0.014391773380339146,
+            -0.014622246846556664,
+            -0.011122467927634716,
+            0.010328616015613079,
+            -0.0007314324611797929,
+            -0.0015524934278801084,
+            0.01744767837226391,
+            0.002784885000437498,
+            -0.022569306194782257,
+            -0.01746474951505661,
+            0.013589384965598583,
+            -0.01876222901046276,
+            -0.0006722136167809367,
+            -0.034588057547807693,
+            -0.04923591390252113,
+            -0.019308535382151604,
+            0.006141684949398041,
+            0.026530031114816666,
+            -0.0002678184537217021,
+            -0.004404599778354168,
+            -0.003990601748228073,
+            0.005714882630854845,
+            -0.01118222065269947,
+            -0.000805055839009583,
+            0.005859995726495981,
+            0.0036342216189950705,
+            0.020264573395252228,
+            0.039368245750665665,
+            0.018523219972848892,
+            0.0006119278259575367,
+            -0.016337992623448372,
+            0.012547987513244152,
+            0.016269704326987267,
+            -0.006116076838225126,
+            0.015919726341962814,
+            0.00982498936355114,
+            -0.004507032223045826,
+            0.046060506254434586,
+            0.029227422550320625,
+            -0.04349968954920769,
+            0.009517691098153591,
+            0.014622246846556664,
+            -0.0250276867300272,
+            0.03404175117611885,
+            -0.017618399113416672,
+            -0.02028164640069008,
+            0.0032500994857400656,
+            0.021101105958223343,
+            -0.0002234043349744752,
+            -0.015783149749040604,
+            0.022022999823093414,
+            -0.01002131775021553,
+            -0.0014329887926578522,
+            0.007601349148899317,
+            0.026530031114816666,
+            -0.0011822424130514264,
+            -0.03356373310089111,
+            -0.002336742589250207,
+            -0.009475011378526688,
+            0.004985050763934851,
+            0.011224900372326374,
+            0.01811349019408226,
+            0.002705926541239023,
+            0.009398186579346657,
+            -0.0231156125664711,
+            0.012573596090078354,
+            -0.02337169460952282,
+            -0.008019614964723587,
+            -0.02400336228311062,
+            -0.011626094579696655,
+            0.01450274232774973,
+            0.006141684949398041,
+            0.01657700166106224,
+            -0.02053772658109665,
+            -0.009483546949923038,
+            -0.014280805364251137,
+            0.024447236210107803,
+            0.012419947423040867,
+            -0.02760557271540165,
+            -0.016585538163781166,
+            -0.012437019497156143,
+            -0.015211233869194984,
+            0.012437019497156143,
+            -0.01966704986989498,
+            0.02053772658109665,
+            0.04817744344472885,
+            -0.03568067401647568,
+            -0.011702919378876686,
+            0.03387103229761124,
+            -0.004997855052351952,
+            -0.013136974535882473,
+            0.02991030551493168,
+            0.01027739979326725,
+            -0.01014935877174139,
+            0.005172844044864178,
+            0.012633347883820534,
+            0.01400765124708414,
+            -0.020759664475917816,
+            -0.020998673513531685,
+            -0.019735338166356087,
+            -0.0031498009338974953,
+            -0.007951326668262482,
+            -0.02965422347187996,
+            0.017498893663287163,
+            0.0166794341057539,
+            -0.004447279963642359,
+            0.01650017686188221,
+            -0.028066519647836685,
+            -0.01457956712692976,
+            -0.01142976526170969,
+            -0.01387107465416193,
+            0.0314980112016201,
+            -0.0068587129935622215,
+            0.05749880522489548,
+            -0.006939805578440428,
+            0.00023194037203211337,
+            0.061083946377038956,
+            0.011404157616198063,
+            -0.0015354212373495102,
+            0.037456169724464417,
+            0.027434851974248886,
+            0.0093213627114892,
+            0.018215922638773918,
+            -0.01773790456354618,
+            0.013538168743252754,
+            -0.02081087976694107,
+            0.00026995246298611164,
+            -0.0068074967712163925,
+            0.016133127734065056,
+            -0.0385146401822567,
+            0.0359538272023201,
+            0.017200132831931114,
+            -0.002206567907705903,
+            0.01142976526170969,
+            -0.023303406313061714,
+            -0.010303007438778877,
+            0.016022158786654472,
+            -0.02197178266942501,
+            0.015424635261297226,
+            -0.004131446126848459,
+            -0.004946638830006123,
+            0.024720389395952225,
+            0.033819813281297684,
+            -0.016978194937109947,
+            -0.009893277660012245,
+            -0.042338788509368896,
+            0.0023068664595484734,
+            -0.018335426226258278,
+            0.02258637733757496,
+            0.010354223661124706,
+            -0.013854002580046654,
+            0.0009037538548000157,
+            -0.009227465838193893,
+            -0.016431888565421104,
+            0.004865546245127916,
+            0.008408005349338055,
+            -0.02492525428533554,
+            -0.039641398936510086,
+            -0.011404157616198063,
+            -0.012932109646499157,
+            -0.013136974535882473,
+            -0.016978194937109947,
+            0.0186427254229784,
+            0.0073836799710989,
+            0.011796815320849419,
+            0.009893277660012245,
+            -0.010610305704176426,
+            -0.019479257985949516,
+            0.011515126563608646,
+            0.005919747985899448,
+            0.0023900929372757673,
+            -0.026291022077202797,
+            0.003915911074727774,
+            -0.029585935175418854,
+            0.003687571967020631,
+            -0.010874922387301922,
+            0.03375152498483658,
+            -0.0027507408522069454,
+            -0.029363999143242836,
+            -0.026154445484280586,
+            -0.02492525428533554,
+            0.016884298995137215,
+            -0.010243255645036697,
+            0.00846775807440281,
+            -0.029961522668600082,
+            0.019103670492768288,
+            -0.008937240578234196,
+            -0.027025122195482254,
+            0.00028835830744355917,
+            0.004193332511931658,
+            -0.010874922387301922,
+            -0.021493764594197273,
+            -0.011327332817018032,
+            -0.011941928416490555,
+            -0.052343033254146576,
+            0.0030366983264684677,
+            -0.009389651007950306,
+            0.008766518905758858,
+            -0.0109176030382514,
+            0.00855311844497919,
+            -0.008681158535182476,
+            -0.004142116289585829,
+            -0.021510835736989975,
+            -0.014485670253634453,
+            -0.022296153008937836,
+            0.033068642020225525,
+            0.017353782430291176,
+            -0.023593630641698837,
+            -0.006713600363582373,
+            -0.008689695037901402,
+            -0.004310703370720148,
+            0.0166965052485466,
+            0.012923574075102806,
+            -0.02593250758945942,
+            0.0029022556263953447,
+            0.023474127054214478,
+            -0.003399480367079377,
+            -0.008548850193619728,
+            0.02903962880373001,
+            0.005518553778529167,
+            0.009142105467617512,
+            0.004101570229977369,
+            0.025232551619410515,
+            -0.005347833037376404,
+            0.004575320519506931,
+            0.054903849959373474,
+            0.0026355041190981865,
+            -0.005667934659868479,
+            0.03786589950323105,
+            0.0004075962060596794,
+            -0.01548438798636198,
+            0.024361874908208847,
+            0.012249225750565529,
+            0.025352057069540024,
+            0.013444272801280022,
+            -0.001758425496518612,
+            0.002910791663452983,
+            -0.00014244526391848922,
+            0.03416125476360321,
+            0.0029684100300073624,
+            0.01600508578121662,
+            -0.03947067633271217,
+            0.02195470966398716,
+            0.015689251944422722,
+            -0.0014649989316239953,
+            0.016944050788879395,
+            -0.004024745896458626,
+            -0.029620079323649406,
+            0.0031391310039907694,
+            0.0011331600835546851,
+            0.02390092983841896,
+            0.024617956951260567,
+            0.008343985304236412,
+            -0.019445113837718964,
+            -0.007823286578059196,
+            -0.002910791663452983,
+            0.05971817672252655,
+            0.011856568045914173,
+            0.03136143460869789,
+            -0.018693940714001656,
+            -0.00846775807440281,
+            -0.02106696180999279,
+            0.0021809597965329885,
+            0.025659354403614998,
+            -0.0029769460670650005,
+            -0.01540756318718195,
+            -0.010798098519444466,
+            -0.001411648583598435,
+            -0.004767381586134434,
+            -0.001800038735382259,
+            0.009142105467617512,
+            0.02079380862414837,
+            -0.013563777320086956,
+            0.01059323363006115,
+            0.028851835057139397,
+            -0.007784874178469181,
+            0.01915488764643669,
+            -0.027264131233096123,
+            -0.02746899612247944,
+            -0.010046926327049732,
+            0.012505307793617249,
+            0.009082352742552757,
+            0.013119902461767197,
+            0.020059708505868912,
+            0.008642747066915035,
+            0.008809199556708336,
+            0.014946616254746914,
+            0.015697788447141647,
+            -0.014084476046264172,
+            0.04698239639401436,
+            -0.00393298314884305,
+            -0.002027310896664858,
+            -0.010584697127342224,
+            -0.01065298542380333,
+            0.012266297824680805,
+            -0.0007298319251276553,
+            0.013478416949510574,
+            0.013085758313536644,
+            -0.02326926216483116,
+            0.01937682554125786,
+            0.00982498936355114,
+            0.0006257988861761987,
+            -0.0007986538112163544,
+            0.0166794341057539,
+            -0.013290624134242535,
+            0.026120301336050034,
+            -0.005949624348431826,
+            -0.01912074349820614,
+            0.010303007438778877,
+            0.01710623688995838,
+            0.0028659775853157043,
+            -0.02579593099653721,
+            -0.029995666816830635,
+            -0.0031049868557602167,
+            -0.03240283206105232,
+            0.02144254744052887,
+            0.007456236053258181,
+            0.014332021586596966,
+            0.014878327958285809,
+            -0.01336744800209999,
+            -0.000287024537101388,
+            0.003220223356038332,
+            0.016056302934885025,
+            0.0052454001270234585,
+            -0.05865970999002457,
+            0.004421671852469444,
+            -0.007823286578059196,
+            -0.005031999200582504,
+            -0.06091322377324104,
+            0.009500619024038315,
+            -0.006598363630473614,
+            0.022825386375188828,
+            -0.006692260038107634,
+            0.03803662210702896,
+            0.00777206989005208,
+            -0.0019302134169265628,
+            0.007819018326699734,
+            0.008220212534070015,
+            0.022944891825318336,
+            -0.007183082867413759,
+            0.00777206989005208,
+            0.0030751104932278395,
+            0.01720866933465004,
+            -0.0033525321632623672,
+            -0.012547987513244152,
+            0.0011875773780047894,
+            0.020981600508093834,
+            -0.006222777534276247,
+            0.009065280668437481,
+            0.03574896231293678,
+            0.002298330422490835,
+            -0.029773728922009468,
+            0.013666209764778614,
+            0.0020614550448954105,
+            -0.017891552299261093,
+            0.01964997872710228,
+            -0.005680738482624292,
+            0.011250508949160576,
+            -0.002041182015091181,
+            0.002332474570721388,
+            0.023525342345237732,
+            0.001920610317029059,
+            0.018540292978286743,
+            -0.0014820710057392716,
+            -0.010755417868494987,
+            0.027315348386764526,
+            -0.008766518905758858,
+            -0.022244935855269432,
+            -0.015381954610347748,
+            -0.03817319869995117,
+            -0.010268863290548325,
+            -0.0056508625857532024,
+            -0.03042246773838997,
+            0.005078947637230158,
+            0.027793366461992264,
+            0.019103670492768288,
+            -0.023730207234621048,
+            -0.023542415350675583,
+            -0.018198849633336067,
+            -0.004167724400758743,
+            0.007144670467823744,
+            -0.03322229161858559,
+            -0.012300441972911358,
+            0.012923574075102806,
+            -0.006056324578821659,
+            0.004511300474405289,
+            -0.009927421808242798,
+            -0.03096877597272396,
+            0.06944926828145981,
+            -0.015620964579284191,
+            -0.013896683230996132,
+            -0.02567642740905285,
+            -0.003478438826277852,
+            -0.011668775230646133,
+            0.023474127054214478,
+            -0.0034250884782522917,
+            0.005403317045420408,
+            0.015774613246321678,
+            0.005155771970748901,
+            0.011207828298211098,
+            -0.00033610680839046836,
+            0.009551835246384144,
+            0.014656390994787216,
+            0.005360636860132217,
+            -0.003478438826277852,
+            0.00591121194884181,
+            0.015262450091540813,
+            -0.027947014197707176,
+            0.008685426786541939,
+            -0.019428040832281113,
+            0.008681158535182476,
+            0.000938965065870434,
+            0.011617559008300304,
+            0.0031988832633942366,
+            -0.01290650200098753,
+            -0.005437461193650961,
+            -0.04073401167988777,
+            -0.009594515897333622,
+            0.023542415350675583,
+            0.017942769452929497,
+            0.03544166311621666,
+            0.0022321760188788176,
+            0.009253074415028095,
+            0.022671738639473915,
+            0.03841220587491989,
+            -0.018198849633336067,
+            0.0015813024947419763,
+            0.010550552979111671,
+            -0.0015770344762131572,
+            0.04916762560606003,
+            0.02362777478992939,
+            0.0012644018279388547,
+            -0.01149805448949337,
+            0.0006620771018788218,
+            0.00023580827109981328,
+            0.006154489237815142,
+            0.006611167918890715,
+            -0.0018437859835103154,
+            -0.002035846933722496,
+            -0.017669616267085075,
+            0.005539894104003906,
+            -0.006542879156768322,
+            -0.019735338166356087,
+            -0.0068587129935622215,
+            0.004835669882595539,
+            0.0289030522108078,
+            0.0002462115662638098,
+            0.004950906615704298,
+            -0.0005164307658560574,
+            0.0007132933242246509,
+            0.026786113157868385,
+            -0.009210393764078617,
+            0.01588558219373226,
+            0.002345278626307845,
+            -0.013597921468317509,
+            0.010174967348575592,
+            -0.008459221571683884,
+            0.0021574855782091618,
+            -0.012974790297448635,
+            0.014878327958285809,
+            0.015569748356938362,
+            -0.010192039422690868,
+            -0.025095975026488304,
+            -0.011762671172618866,
+            0.026769040152430534,
+            -0.015552676282823086,
+            0.011327332817018032,
+            0.003011090215295553,
+            0.0007778471917845309,
+            -0.00181390973739326,
+            -0.0009202924557030201,
+            0.03827562928199768,
+            -0.00943233072757721,
+            -0.016158735379576683,
+            -0.005749027244746685,
+            0.012479699216783047,
+            0.008343985304236412,
+            -0.014972224831581116,
+            -0.01925732009112835,
+            0.018796373158693314,
+            -0.017908625304698944,
+            0.015697788447141647,
+            -0.04182662442326546,
+            -0.023303406313061714,
+            0.026615392416715622,
+            -9.683077223598957e-05,
+            -0.03503193333745003,
+            0.01168584730476141,
+            0.016329456120729446,
+            0.0028617095667868853,
+            0.008271428756415844,
+            -0.008702498860657215,
+            0.0028190291486680508,
+            -0.007754997815936804,
+            -0.010849314741790295,
+            0.0013636333169415593,
+            0.007114794570952654,
+            -0.007955594919621944,
+            0.02082795277237892,
+            -0.014195444993674755,
+            -0.015211233869194984,
+            -0.005680738482624292,
+            -0.0128552857786417,
+            0.01553560420870781,
+            -0.00581731554120779,
+            -0.013256479986011982,
+            0.017473286017775536,
+            0.014485670253634453,
+            0.015433171764016151,
+            0.02478867769241333,
+            0.018301282078027725,
+            -0.02250101789832115,
+            -0.002558679785579443,
+            3.8512240280397236e-05,
+            -0.0024733194150030613,
+            0.01227483432739973,
+            0.035236798226833344,
+            0.0016421218169853091,
+            -0.009816452860832214,
+            -0.02364484779536724,
+            0.01277846097946167,
+            0.01309429481625557,
+            -0.014041795395314693,
+            -0.018711013719439507,
+            -0.004114374052733183,
+            -0.013964971527457237,
+            0.00652153929695487,
+            -0.0023111344780772924,
+            0.000925094005651772,
+            -0.0009939158335328102,
+            -0.003484840737655759,
+            -0.02091331221163273,
+            0.005356369074434042,
+            -0.01953047327697277,
+            -0.031907740980386734,
+            -0.012530915439128876,
+            0.008036687038838863,
+            0.024754533544182777,
+            0.002035846933722496,
+            -0.0224839448928833,
+            -0.0014415248297154903,
+            -0.00016365200281143188,
+            -0.0093213627114892,
+            -0.01495515275746584,
+            0.0016197147779166698,
+            0.0005118960398249328,
+            0.010038389824330807,
+            -0.007682441268116236,
+            -0.01722574047744274,
+            0.005471605341881514,
+            0.01734524592757225,
+            -0.022381512448191643,
+            -0.0025202676188200712,
+            -0.017174525186419487,
+            -0.01559535600244999,
+            0.0873408243060112,
+            -0.010891994461417198,
+            -0.012539451941847801,
+            -0.01418690849095583,
+            0.0038092106115072966,
+            -0.006333746016025543,
+            0.010678594000637531,
+            0.03264183923602104,
+            -0.006026448681950569,
+            -0.006338014267385006,
+            -0.0026717823930084705,
+            -0.0008578726556152105,
+            -0.019359752535820007,
+            0.001124624046497047,
+            0.007921450771391392,
+            -0.004566784482449293,
+            -0.003316253889352083,
+            0.01450274232774973,
+            -0.04435329511761665,
+            0.0021916297264397144,
+            0.005096019711345434,
+            -0.02260345034301281,
+            0.02002556435763836,
+            0.015919726341962814,
+            -0.022057143971323967,
+            0.003002554178237915,
+            0.00606912886723876,
+            0.03311985731124878,
+            0.022296153008937836,
+            0.009218929335474968,
+            0.015723396092653275,
+            -0.01836957037448883,
+            0.012863821350038052,
+            -0.002146815648302436,
+            -0.0002396761701675132,
+            0.008954312652349472,
+            0.004472888074815273,
+            0.013145511038601398,
+            0.0016581269446760416,
+            0.0009016198455356061,
+            -0.022688809782266617,
+            -0.006850176956504583,
+            -0.026820257306098938,
+            -0.027895798906683922,
+            -0.010379832237958908,
+            0.03182237967848778,
+            -0.008988456800580025,
+            -0.014332021586596966,
+            0.006628239993005991,
+            -0.008292769081890583,
+            -0.002063589170575142,
+            -0.024617956951260567,
+            0.016337992623448372,
+            -0.007272711489349604,
+            0.0024413091596215963,
+            0.0005537759861908853,
+            0.0112334368750453,
+            -0.0064660548232495785,
+            -0.022808315232396126,
+            0.010584697127342224,
+            -0.0024583812337368727,
+            0.03426368907094002,
+            -0.00924453791230917,
+            -0.01836957037448883,
+            0.004780185874551535,
+            0.00041613224311731756,
+            -0.01941096968948841,
+            -0.005962428171187639,
+            -0.007793410215526819,
+            -0.021425476297736168,
+            -0.014613711275160313,
+            -0.013597921468317509,
+            -0.012027288787066936,
+            -0.01655992865562439,
+            0.006278262007981539,
+            0.00014591304352506995,
+            0.008574457839131355,
+            -0.01027739979326725,
+            -0.000391057605156675,
+            0.012539451941847801,
+            -0.015433171764016151,
+            -0.019308535382151604,
+            -0.007784874178469181,
+            -0.0041976007632911205,
+            -0.0013294891687110066,
+            0.005915479734539986,
+            -0.0218522772192955,
+            -0.037319593131542206,
+            0.011335869319736958,
+            -0.011002963408827782,
+            -0.004342713393270969,
+            0.022791242226958275,
+            0.000883480766788125,
+            -0.022705882787704468,
+            0.009415258653461933,
+            0.012599203735589981,
+            0.0045027644373476505,
+            -0.015612428076565266,
+            0.005510017741471529,
+            0.0034677686635404825,
+            -0.01110539585351944,
+            -0.01860858127474785,
+            0.0034528307151049376,
+            -0.008685426786541939,
+            0.005731955170631409,
+            -0.02786165475845337,
+            0.013657674193382263,
+            0.014784432016313076,
+            0.01084077823907137,
+            0.00021033351367805153,
+            -0.00475457776337862,
+            -0.01719159632921219,
+            -0.002552277874201536,
+            -0.00642764288932085,
+            0.07013215124607086,
+            -0.012752852402627468,
+            -0.017635472118854523,
+            0.016056302934885025,
+            0.0011384951649233699,
+            -0.011617559008300304,
+            0.009662804193794727,
+            -0.01848907582461834,
+            0.020623087882995605,
+            -0.03868535906076431,
+            0.012291906401515007,
+            -0.006060592830181122,
+            0.0075842770747840405,
+            -0.008655550889670849,
+            0.0006172628491185606,
+            -0.021408403292298317,
+            0.023918000981211662,
+            0.027656789869070053,
+            0.010431048460304737,
+            -0.01847200281918049,
+            -0.027895798906683922,
+            0.016824547201395035,
+            0.017003804445266724,
+            0.002428505104035139,
+            -0.009637195616960526,
+            0.011754135601222515,
+            -0.023952145129442215,
+            0.008945776149630547,
+            -0.006346550304442644,
+            -0.0032372954301536083,
+            0.009218929335474968,
+            0.007814750075340271,
+            0.007951326668262482,
+            0.009364042431116104,
+            -0.03235161304473877,
+            0.016798939555883408,
+            0.028800619766116142,
+            0.004592392593622208,
+            0.028800619766116142,
+            -0.018796373158693314,
+            0.01823299378156662,
+            -0.006752012297511101,
+            0.009022600948810577,
+            0.021254755556583405,
+            0.011463910341262817,
+            -0.030081026256084442,
+            0.0027678129263222218,
+            -0.03380274400115013,
+            0.0017712295521050692,
+            0.02673489600419998,
+            -0.0044302078895270824,
+            0.001891801250167191,
+            -0.02284245938062668,
+            0.008860415779054165,
+            0.004443012177944183,
+            -0.020059708505868912,
+            -0.014357629232108593,
+            0.00600510835647583,
+            0.0025800198782235384,
+            -0.026547104120254517,
+            0.022518089041113853,
+            -0.023815568536520004,
+            -0.0010488666594028473,
+            0.017618399113416672,
+            0.0006231313454918563,
+            -0.01579168438911438,
+            0.008420809172093868,
+            0.02159619703888893,
+            -0.02579593099653721,
+            -0.032027244567871094,
+            0.006431910675019026,
+            0.0075842770747840405,
+            0.021630341187119484,
+            0.01978655532002449,
+            -0.008723839186131954,
+            0.02287660352885723,
+            0.007840358652174473,
+            0.0038177466485649347,
+            -0.0372513048350811,
+            -0.017635472118854523,
+            0.019957276061177254,
+            -0.014536886475980282,
+            0.018454931676387787,
+            -0.011412693187594414,
+            -0.001605843659490347,
+            0.009355506859719753,
+            -0.008139119483530521,
+            0.027793366461992264,
+            -0.01405886746942997,
+            -0.0036576958373188972,
+            0.004425940103828907,
+            -0.03341008350253105,
+            -0.0024178349412977695,
+            0.025727642700076103,
+            0.001300679985433817,
+            -0.02454966865479946,
+            -0.0030132243409752846,
+            -0.0023580826818943024,
+            0.003779334481805563,
+            -0.005475873593240976,
+            -0.007733657956123352,
+            -0.010644449852406979,
+            0.003572335233911872,
+            -0.006453251000493765,
+            -0.011352941393852234,
+            0.00214574858546257,
+            0.04985050857067108,
+            0.015817293897271156,
+            -0.007123330608010292,
+            0.0005857861833646894,
+            0.01728549413383007,
+            0.042953383177518845,
+            -0.022825386375188828,
+            0.03122485615313053,
+            0.0009693747269921005,
+            0.0038540249224752188,
+            -0.02478867769241333,
+            -0.026052013039588928,
+            0.02632516622543335,
+            0.010721273720264435,
+            -0.02542034536600113,
+            -0.002863843459635973,
+            0.00415492057800293,
+            0.012223618105053902,
+            0.014929544180631638,
+            0.014690535143017769,
+            -0.031241929158568382,
+            -0.015475851483643055,
+            -0.003343996126204729,
+            -0.008954312652349472,
+            -0.00606912886723876,
+            -0.005322224926203489,
+            0.013478416949510574,
+            0.008143387734889984,
+            -0.025778859853744507,
+            -0.0013454942964017391,
+            -0.005390513222664595,
+            -0.005343564786016941,
+            -0.01033715158700943,
+            0.009278682060539722,
+            -0.009227465838193893,
+            0.0040610237047076225,
+            -0.010448120534420013,
+            0.010832242667675018,
+            -0.015074657276272774,
+            0.010516408830881119,
+            0.004186930600553751,
+            -0.02094745635986328,
+            -0.012496771290898323,
+            -0.01838664337992668,
+            0.007183082867413759,
+            0.01937682554125786,
+            0.01835249923169613,
+            0.0062996018677949905,
+            -0.014084476046264172,
+            -0.016039229929447174,
+            -0.015151482075452805,
+            0.017908625304698944,
+            0.01014935877174139,
+            -0.005501481704413891,
+            -0.018028128892183304,
+            -0.025659354403614998,
+            0.022518089041113853,
+            0.005988036282360554,
+            -0.03612454608082771,
+            0.02250101789832115,
+            -0.008326913230121136,
+            -6.268659035413293e-06,
+            0.012761388905346394,
+            -0.010900530964136124,
+            -0.004400331526994705,
+            -0.0023474127519875765,
+            0.019206102937459946,
+            -0.0014916741056367755,
+            -0.0008450685418210924,
+            0.005706346593797207,
+            -0.00011263579654041678,
+            -0.008817736059427261,
+            -0.013213799335062504,
+            -0.0037537263706326485,
+            0.001452194876037538,
+            -0.0033226560335606337,
+            -0.0024498451966792345,
+            0.004545444622635841,
+            0.014681999571621418,
+            0.013802786357700825,
+            0.005578306037932634,
+            0.004746041726320982,
+            0.0007917182520031929,
+            -0.007242835126817226,
+            0.01059323363006115,
+            -0.021408403292298317,
+            0.006816032808274031,
+            -0.005569770000874996,
+            -0.020725520327687263,
+            -0.005902675911784172,
+            0.00018845988961402327,
+            0.02157912403345108,
+            -0.03431490436196327,
+            0.0038198805414140224,
+            0.017294028773903847,
+            0.009346970357000828,
+            -0.01836957037448883,
+            0.010883458890020847,
+            0.06518124788999557,
+            -0.005271008238196373,
+            0.008284232579171658,
+            -0.0010888794204220176,
+            0.011779743246734142,
+            -0.006184365600347519,
+            0.005322224926203489,
+            -0.0017360183410346508,
+            0.0346904918551445,
+            -0.007823286578059196,
+            -0.012940646149218082,
+            -0.023183900862932205,
+            -0.012496771290898323,
+            -0.016397744417190552,
+            0.02786165475845337,
+            -0.014477133750915527,
+            0.002011305885389447,
+            -0.012078505009412766,
+            0.0065386113710701466,
+            0.002077460289001465,
+            0.0014596638502553105,
+            -0.018215922638773918,
+            -0.009927421808242798,
+            0.016747722402215004,
+            0.0034442946780472994,
+            0.0023666187189519405,
+            0.00530088460072875,
+            -0.014528350904583931,
+            0.030524900183081627,
+            -0.0074477000162005424,
+            0.020059708505868912,
+            0.004391795489937067,
+            -0.005232596304267645,
+            0.015185626223683357,
+            -0.009466474875807762,
+            0.015859974548220634,
+            -0.0118821756914258,
+            -0.003529655048623681,
+            -0.030149314552545547,
+            -0.008565922267735004,
+            0.020623087882995605,
+            -0.014519814401865005,
+            0.0005495079676620662,
+            0.016525784507393837,
+            0.005014927126467228,
+            -0.007959863170981407,
+            0.011224900372326374,
+            0.003382408292964101,
+            -0.01758425496518612,
+            -0.010832242667675018,
+            -0.004647877067327499,
+            -0.01072981022298336,
+            0.011301725171506405,
+            0.013589384965598583,
+            -0.022398585453629494,
+            0.0015290192095562816,
+            -0.006914197467267513,
+            -0.008207408711314201,
+            0.00716174254193902,
+            0.013162583112716675,
+            -0.027571428567171097,
+            0.017370853573083878,
+            0.0021254755556583405,
+            -0.0002107336331391707,
+            0.004250951111316681,
+            -0.0004980249213986099,
+            0.014605174772441387,
+            -0.007575741037726402,
+            -0.002740070689469576,
+            -0.006845909170806408,
+            0.010934675112366676,
+            -0.0004094634496141225,
+            0.0157319325953722,
+            -0.00683737313374877,
+            -0.002915059681981802,
+            0.007234299089759588,
+            0.016286775469779968,
+            -0.02557399496436119,
+            0.015424635261297226,
+            -0.00810070801526308,
+            0.0002974278468172997,
+            0.00031343294540420175,
+            0.009253074415028095,
+            0.005591110326349735,
+            -0.015902653336524963,
+            -0.0072300308384001255,
+            0.015569748356938362,
+            0.005117359571158886,
+            0.018950022757053375,
+            0.021476691588759422,
+            0.007033701986074448,
+            0.023866785690188408,
+            -0.005744758993387222,
+            -0.013085758313536644,
+            -0.0020561201963573694,
+            -0.009338434785604477,
+            -0.002368752844631672,
+            0.006286798045039177,
+            -0.004054621793329716,
+            -0.025915436446666718,
+            -0.012821141630411148,
+            -0.007276979275047779,
+            -0.00038012079312466085,
+            0.009475011378526688,
+            -0.010311543941497803,
+            0.01707209274172783,
+            0.02236444130539894,
+            7.760800144751556e-06,
+            -0.006175829563289881,
+            -0.02094745635986328,
+            0.007251371163874865,
+            -0.007763533852994442,
+            -0.011737063527107239,
+            -0.010098142549395561,
+            -0.0028702456038445234,
+            0.0018395179649814963,
+            0.005407585296779871,
+            0.004775917623192072,
+            -0.017772048711776733,
+            -0.006862981244921684,
+            0.009423795156180859,
+            -0.01799398474395275,
+            -0.0023239385336637497,
+            0.00031343294540420175,
+            0.006666651926934719,
+            0.012872357852756977,
+            0.01758425496518612,
+            -0.013614993542432785,
+            -0.022449800744652748,
+            0.003002554178237915,
+            -0.009739628992974758,
+            0.03274427354335785,
+            -0.008168996311724186,
+            -0.005514285992830992,
+            -0.018284210935235023,
+            -0.006175829563289881,
+            -0.009346970357000828,
+            -0.00956890732049942,
+            -0.0004897556500509381,
+            0.014861255884170532,
+            -0.009304290637373924,
+            0.01482711173593998,
+            -0.016491640359163284,
+            -0.003205285407602787,
+            0.00409090006724,
+            0.00969694834202528,
+            -0.030012737959623337,
+            -0.006278262007981539,
+            -0.00799400731921196,
+            -0.02145962044596672,
+            -0.013708890415728092,
+            -0.012471163645386696,
+            0.007588544860482216,
+            0.0014073805650696158,
+            0.01642335206270218,
+            -0.007187350653111935,
+            -0.014195444993674755,
+            0.009978638030588627,
+            -0.02031579054892063,
+            -0.012983325868844986,
+            -0.008292769081890583,
+            -0.04073401167988777,
+            0.020469438284635544,
+            0.01354670524597168,
+            0.023337550461292267,
+            -0.0024434432853013277,
+            0.020964529365301132,
+            0.008723839186131954,
+            -0.01742207072675228,
+            0.006372158415615559,
+            -0.012505307793617249,
+            0.005744758993387222,
+            -0.02506183087825775,
+            -0.0066026318818330765,
+            0.005505749955773354,
+            0.01246262714266777,
+            -0.0037665304262191057,
+            2.7758822398027405e-05,
+            0.014280805364251137,
+            -0.01041397638618946,
+            -0.01787448115646839,
+            0.004067426081746817,
+            0.0067562805488705635,
+            -0.01713184453547001,
+            0.03670499846339226,
+            -0.00796413142234087,
+            -0.00017445544654037803,
+            0.013179655186831951,
+            0.02104988880455494,
+            -0.020264573395252228,
+            -0.01450274232774973,
+            0.01937682554125786,
+            -0.0010200574761256576,
+            -0.0004940236685797572,
+            0.017183061689138412,
+            -0.006828837096691132,
+            0.008032419718801975,
+            -0.005766099318861961,
+            -0.00484847417101264,
+            -0.00503626698628068,
+            0.0011683712946251035,
+            -0.016901372000575066,
+            0.0042637549340724945,
+            -0.000693553767632693,
+            -0.0006658115889877081,
+            0.019581690430641174,
+            -0.006453251000493765,
+            0.0020102388225495815,
+            0.011976072564721107,
+            0.00038332180702127516,
+            -0.05049924924969673,
+            -0.0047204336151480675,
+            0.0034869748633354902,
+            -0.003279975615441799,
+            -0.003469902789220214,
+            0.009807917289435863,
+            0.01195046491920948,
+            0.01110539585351944,
+            -0.005130163859575987,
+            0.013452808372676373,
+            0.00938111450523138,
+            0.006632507778704166,
+            0.0007287649204954505,
+            -0.004329909570515156,
+            0.011813887394964695,
+            -0.004588124807924032,
+            0.020127996802330017,
+            -0.004421671852469444,
+            -0.01889880560338497,
+            -0.016414817422628403,
+            0.0010739413555711508,
+            0.027025122195482254,
+            -0.0037665304262191057,
+            0.026103228330612183,
+            -0.0013839064631611109,
+            0.0033205219078809023,
+            -0.0018032396910712123,
+            -0.0032223574817180634,
+            -0.004827133845537901,
+            -0.035134363919496536,
+            0.009039673022925854,
+            -0.007084918208420277,
+            0.020008491352200508,
+            -0.02943228743970394,
+            0.003026028396561742,
+            -0.006175829563289881,
+            -0.005787439178675413,
+            -0.0025608139112591743,
+            0.005031999200582504,
+            -2.869245145120658e-05,
+            0.03803662210702896,
+            -0.014844183810055256,
+            -0.021271826699376106,
+            -0.010396904312074184,
+            0.006995289586484432,
+            0.02798115834593773,
+            -0.0004838871245738119,
+            0.01913781464099884,
+            0.020896241068840027,
+            -0.013068686239421368,
+            0.004844206385314465,
+            -0.0031156567856669426,
+            -0.002942801918834448,
+            0.00011677043949021026,
+            -0.009440867230296135,
+            0.005198452156037092,
+            -0.03530508652329445,
+            0.007008093874901533,
+            -0.0005225660861469805,
+            0.02427651546895504,
+            5.1282964705023915e-05,
+            0.004814330022782087,
+            -0.003311985870823264,
+            0.02274002693593502,
+            0.003945787437260151,
+            0.01719159632921219,
+            -0.0013017470482736826,
+            0.004767381586134434,
+            0.010396904312074184,
+            -0.019086599349975586,
+            -0.019820699468255043,
+            -0.015142945572733879,
+            0.001055268687196076,
+            0.0011448971927165985,
+            -0.015270986594259739,
+            -0.018830517306923866,
+            -0.0059923045337200165,
+            0.017515966668725014,
+            -0.011702919378876686,
+            0.01264188438653946,
+            0.013538168743252754,
+            0.009159177541732788,
+            -0.001976094674319029,
+            0.01227483432739973,
+            0.01592826284468174,
+            -0.018215922638773918,
+            0.009022600948810577,
+            -0.009577443823218346,
+            -0.008937240578234196,
+            -0.010960282757878304,
+            -0.01811349019408226,
+            0.03438319265842438,
+            0.014750287868082523,
+            -0.011677310802042484,
+            -0.008587262593209743,
+            -0.0010019184555858374,
+            0.004758845549076796,
+            0.00044013987644575536,
+            0.005663666408509016,
+            -0.004647877067327499,
+            -0.026137372478842735,
+            -0.014673463068902493,
+            -0.016466032713651657,
+            -0.005719150882214308,
+            0.002590690040960908,
+            0.014801504090428352,
+            -0.021732773631811142,
+            -0.016397744417190552,
+            -0.0263080932199955,
+            0.01925732009112835,
+            -0.017533037811517715,
+            0.01651724986732006,
+            0.012249225750565529,
+            0.002076393226161599,
+            -0.01265041995793581,
+            -0.02801530249416828,
+            0.014741751365363598,
+            -0.024361874908208847,
+            0.00670933211222291,
+            -0.0016261168057098985,
+            0.0500553734600544,
+            -0.00277634896337986,
+            -0.006192901637405157,
+            0.009500619024038315,
+            0.003561665304005146,
+            0.013845467008650303,
+            -0.02197178266942501,
+            -0.005403317045420408,
+            -0.003525387030094862,
+            0.011566342785954475,
+            -0.004571052733808756,
+            -0.0056892745196819305,
+            -0.002590690040960908,
+            -0.007375143934041262,
+            0.001402045600116253,
+            0.019291464239358902,
+            -0.006880053319036961,
+            -0.007008093874901533,
+            -0.0030921828001737595,
+            0.029842017218470573,
+            0.01963290572166443,
+            -0.008548850193619728,
+            -0.006444714963436127,
+            0.008365325629711151,
+            -0.01787448115646839,
+            -0.0014916741056367755,
+            0.017370853573083878,
+            0.021220611408352852,
+            -0.011224900372326374,
+            0.006999557837843895,
+            0.002942801918834448,
+            -0.0033269240520894527,
+            -0.021510835736989975,
+            -0.01137001346796751,
+            -0.0075202565640211105,
+            -0.0018896672409027815,
+            0.01632091961801052,
+            -0.011310260742902756,
+            0.02325218915939331,
+            0.01567218080163002,
+            0.009517691098153591,
+            0.0010883458890020847,
+            -0.041280318051576614,
+            -0.01548438798636198,
+            0.01687576249241829,
+            -0.0034613667521625757,
+            0.017362317070364952,
+            0.013649137690663338,
+            0.0031007188372313976,
+            -0.019479257985949516,
+            0.023952145129442215,
+            0.0008594731334596872,
+            0.0014233856927603483,
+            -0.001049933722242713,
+            0.015330738388001919,
+            0.0019632906187325716,
+            0.00962012354284525,
+            0.02106696180999279,
+            0.020640159025788307,
+            -0.01797691360116005,
+            -0.009987173601984978,
+            0.011301725171506405,
+            -0.017037948593497276,
+            0.019684122875332832,
+            0.001101149944588542,
+            0.003862560959532857,
+            0.015501460060477257,
+            0.027025122195482254,
+            0.03277841582894325,
+            0.011062716133892536,
+            -0.004933834541589022,
+            -0.007959863170981407,
+            -0.001679467037320137,
+            0.004545444622635841,
+            0.014041795395314693,
+            -0.02337169460952282,
+            -0.031054135411977768,
+            0.003213821444660425,
+            0.013529633171856403,
+            -0.02506183087825775,
+            0.007725121919065714,
+            0.01913781464099884,
+            0.02181813307106495,
+            0.0050917514599859715,
+            0.015552676282823086,
+            0.014519814401865005,
+            0.0034528307151049376,
+            -0.002049718052148819,
+            0.010046926327049732,
+            -0.013623529113829136,
+            0.005155771970748901,
+            0.0022833922412246466,
+            -0.015509995631873608,
+            0.02053772658109665,
+            -0.00037345202872529626,
+            0.00026034939219243824,
+            0.003395212348550558,
+            0.001364700379781425,
+            -0.00950915552675724,
+            -0.03426368907094002,
+            0.016551394015550613,
+            -0.004159188363701105,
+            0.010439584031701088,
+            -0.017020875588059425,
+            -0.0075714727863669395,
+            -0.007511720526963472,
+            0.03421247377991676,
+            0.009304290637373924,
+            0.007631225045770407,
+            -0.004293631296604872,
+            0.0008301304769702256,
+            0.003205285407602787,
+            0.007699513342231512,
+            -0.01799398474395275,
+            -0.018045201897621155,
+            0.009662804193794727,
+            0.004903958644717932,
+            8.744445949560031e-06,
+            0.01642335206270218,
+            0.009073817171156406,
+            0.012078505009412766,
+            0.02299610897898674,
+            0.008040955290198326,
+            -0.01728549413383007,
+            0.0033013159409165382,
+            0.019957276061177254,
+            -0.0008957513491623104,
+            0.01707209274172783,
+            0.007486112415790558,
+            -0.007008093874901533,
+            -0.026103228330612183,
+            0.00020659899746533483,
+            0.018164705485105515,
+            0.023952145129442215,
+            -0.023969218134880066,
+            -0.019052455201745033,
+            0.01850614883005619,
+            -0.0121980095282197,
+            0.010627377778291702,
+            -0.002761411014944315,
+            0.000420933763962239,
+            0.0031754090450704098,
+            -0.017217205837368965,
+            0.004562516696751118,
+            0.017037948593497276,
+            0.01014935877174139,
+            0.00015778347733430564,
+            0.005386244971305132,
+            0.01809641718864441,
+            0.022449800744652748,
+            -0.030661476776003838,
+            -0.010968819260597229,
+            -0.0029769460670650005,
+            0.004579588770866394,
+            -0.014613711275160313,
+            -0.0028531732968986034,
+            0.005108823534101248,
+            0.00937257893383503,
+            -0.022057143971323967,
+            -0.0109176030382514,
+            -0.0023025984410196543,
+            -0.0024626492522656918,
+            0.008903096430003643,
+            -0.004733237437903881,
+            0.0030559045262634754,
+            -0.016798939555883408,
+            0.005492945667356253,
+            -0.019615834578871727,
+            0.017183061689138412,
+            -0.0049252985045313835,
+            -0.0008722771890461445,
+            -0.010217647068202496,
+            0.0021062693558633327,
+            -0.0028702456038445234,
+            -0.02390092983841896,
+            0.022893676534295082,
+            -0.0157148614525795,
+            -0.0036897058598697186,
+            0.043738700449466705,
+            0.002827565185725689,
+            0.006828837096691132,
+            0.02364484779536724,
+            0.002174557652324438,
+            -0.004647877067327499,
+            0.026103228330612183,
+            0.03175409138202667,
+            -0.010550552979111671,
+            -0.009748164564371109,
+            0.017618399113416672,
+            0.009278682060539722,
+            0.006935537327080965,
+            0.011489517986774445,
+            -0.0027784830890595913,
+            -0.017686687409877777,
+            -0.00911649689078331,
+            0.0011790413409471512,
+            0.015492923557758331,
+            -0.0032308935187757015,
+            -0.0020059708040207624,
+            0.011924856342375278,
+            7.60908515076153e-05,
+            -0.007524524815380573,
+            0.02837381698191166,
+            0.002735802670940757,
+            -0.01784033700823784,
+            -0.029210349544882774,
+            0.0118821756914258,
+            -0.0125138433650136,
+            -0.018796373158693314,
+            -0.017294028773903847,
+            -0.0021030684001743793,
+            -0.011301725171506405,
+            -0.002526669530197978,
+            0.012829677201807499,
+            0.01163463108241558,
+            -0.005211255978792906,
+            0.01620141603052616,
+            0.006329478230327368,
+            -0.015902653336524963,
+            -0.01612459123134613,
+            -0.004276559222489595,
+            -0.003909509163349867,
+            -0.0035040469374507666,
+            -0.003911643289029598,
+            0.018198849633336067,
+            -0.009090889245271683,
+            -0.0044302078895270824,
+            -0.018557364121079445,
+            0.01760132610797882,
+            0.013700353913009167,
+            -0.009398186579346657,
+            -0.001582369557581842,
+            -0.001988898729905486,
+            0.028203096240758896,
+            0.0062996018677949905,
+            0.0017947036540135741,
+            0.004297899082303047,
+            -0.00968841277062893,
+            -0.0010846114018931985,
+            -0.02463502809405327,
+            0.020093852654099464,
+            -0.018437858670949936,
+            -0.026939760893583298,
+            -0.007170278578996658,
+            0.024959398433566093,
+            -0.0018395179649814963,
+            0.0015076791169121861,
+            -0.00831837672740221,
+            0.018932949751615524,
+            0.0029812140855938196,
+            -0.02823724038898945,
+            -0.0125138433650136,
+            -0.010183502919971943,
+            -0.015552676282823086,
+            -0.0013518963241949677,
+            0.020332861691713333,
+            -0.01360645703971386,
+            0.00035237864358350635,
+            0.01671357825398445,
+            -0.01149805448949337,
+            0.013469880446791649,
+            -0.003260769648477435,
+            -0.009125033393502235,
+            0.013845467008650303,
+            -0.008536046370863914,
+            -0.00408022990450263,
+            -0.003094316693022847,
+            0.005275276489555836,
+            -0.0012387937167659402,
+            -0.019086599349975586,
+            -0.009440867230296135,
+            0.01084077823907137,
+            0.002966275904327631,
+            0.008728107437491417,
+            -0.014050331898033619,
+            -0.005629522260278463,
+            0.001096881926059723,
+            -7.188950985437259e-05,
+            -0.0018960692686960101,
+            -0.012582131661474705,
+            -0.011327332817018032,
+            -0.020879168063402176,
+            -0.007665369194000959,
+            -0.01206143293529749,
+            0.011173684149980545,
+            0.015245378017425537,
+            -0.010738345794379711,
+            -0.005642326548695564,
+            0.008305572904646397,
+            0.04100716486573219,
+            -0.009782308712601662,
+            0.004383259452879429,
+            0.021613268181681633,
+            0.0032372954301536083,
+            0.007592813111841679,
+            0.02874940261244774,
+            -0.0017701626056805253,
+            -0.010704201646149158,
+            0.005117359571158886,
+            -0.01825006678700447,
+            -0.002957739867269993,
+            0.027554357424378395,
+            -0.021271826699376106,
+            0.005787439178675413,
+            -0.023730207234621048,
+            -0.008079367689788342,
+            -0.0033888102043420076,
+            -0.0121980095282197,
+            -0.014238124713301659,
+            -0.002049718052148819,
+            0.0006028582574799657,
+            0.023713136091828346,
+            -0.007439163979142904,
+            0.0050405352376401424,
+            0.009159177541732788,
+            -0.025130119174718857,
+            0.007426360156387091,
+            -0.010994426906108856,
+            -0.010166430845856667,
+            -0.004916762467473745,
+            -0.026803184300661087,
+            0.015501460060477257,
+            -0.011540734209120274,
+            -0.006717868149280548,
+            -0.018557364121079445,
+            0.033205218613147736,
+            0.017635472118854523,
+            0.011276116594672203,
+            -0.02313268557190895,
+            -0.015014905482530594,
+            -0.020110923796892166,
+            -0.037046439945697784,
+            -0.00030409664032049477,
+            0.0008386665140278637,
+            -0.002919327700510621,
+            0.007716585882008076,
+            0.025864219292998314,
+            0.0017701626056805253,
+            -0.006444714963436127,
+            -0.004575320519506931,
+            0.007375143934041262,
+            0.009005528874695301,
+            -0.014417381957173347,
+            0.00911649689078331,
+            0.018301282078027725,
+            0.0002280724875163287,
+            0.007725121919065714,
+            0.00813058391213417,
+            0.0020305118523538113,
+            0.00548867741599679,
+            0.028407961130142212,
+            0.0034122844226658344,
+            0.003109254874289036,
+            -0.0001357764849672094,
+            -7.043905043246923e-06,
+            -0.011148076504468918,
+            0.0010947479167953134,
+            -0.018591508269309998,
+            0.00600510835647583,
+            -0.01860858127474785,
+            0.0199231319129467,
+            -0.004127178341150284,
+            0.008092171512544155,
+            -0.02207421511411667,
+            -0.0243960190564394,
+            0.004268023185431957,
+            0.027178769931197166,
+            0.01259066816419363,
+            -0.007878770120441914,
+            -0.015859974548220634,
+            0.01888173446059227,
+            0.01104564405977726,
+            -0.011267581023275852,
+            -0.012505307793617249,
+            -0.017789119854569435,
+            -9.489682270213962e-05,
+            -0.02144254744052887,
+            0.02528376877307892,
+            0.015902653336524963,
+            0.0038881690707057714,
+            -0.013171118684113026,
+            0.02425944246351719,
+            -0.011549270711839199,
+            0.0070465062744915485,
+            -0.01503197755664587,
+            -0.015561211854219437,
+            -0.0186427254229784,
+            -0.005945356097072363,
+            -0.001342293224297464,
+            0.011651703156530857,
+            -0.027571428567171097,
+            0.013768642209470272,
+            0.014391773380339146,
+            -0.002705926541239023,
+            0.0022215060889720917,
+            -0.0055356258526444435,
+            0.014571030624210835,
+            -0.006875785067677498,
+            0.007336731534451246,
+            -0.006991021800786257,
+            -0.0021873617079108953,
+            -0.0065172710455954075,
+            -0.01418690849095583,
+            -0.014656390994787216,
+            -0.0053307609632611275,
+            -0.014110084623098373,
+            0.0035125829745084047,
+            0.019342679530382156,
+            -0.026052013039588928,
+            -0.00018205786182079464,
+            -0.001230257679708302,
+            -6.235315231606364e-05,
+            -0.0032735737040638924,
+            0.012795533053576946,
+            0.016918443143367767,
+            -0.0067861564457416534,
+            -0.005117359571158886,
+            -0.0007324994658119977,
+            -0.017498893663287163,
+            -0.0004924231325276196,
+            0.021203538402915,
+            -0.0031433990225195885,
+            -0.005962428171187639,
+            -0.010934675112366676,
+            -0.038856081664562225,
+            -0.01078956201672554,
+            -0.01862565241754055,
+            -0.021544979885220528,
+            -0.006939805578440428,
+            0.0007244969019666314,
+            0.008028151467442513,
+            0.01381132286041975,
+            -0.010712738148868084,
+            -0.011096860282123089,
+            0.0033290579449385405,
+            -0.01622702367603779,
+            0.0016463898355141282,
+            -0.019342679530382156,
+            -0.0013614993076771498,
+            0.008783590979874134,
+            0.014212517067790031,
+            0.0044515482150018215,
+            0.00632094219326973,
+            0.0019995688926428556,
+            -0.011301725171506405,
+            -0.009978638030588627,
+            0.019189031794667244,
+            -0.022552233189344406,
+            0.003687571967020631,
+            -0.004865546245127916,
+            0.0065386113710701466,
+            -0.0034400266595184803,
+            -0.02159619703888893,
+            0.006367890164256096,
+            -0.02069137617945671,
+            0.010362760163843632,
+            -0.0010099209612235427,
+            -0.004950906615704298,
+            0.006658115889877081,
+            -0.0021521507296711206,
+            -0.011719991452991962,
+            0.01772083155810833,
+            -0.006777620408684015,
+            -0.004511300474405289,
+            -0.007191618904471397,
+            0.006244117859750986,
+            0.01588558219373226,
+            0.011037107557058334,
+            -0.0053819771856069565,
+            0.0062483856454491615,
+            0.011455373838543892,
+            -0.0026995246298611164,
+            -0.00767817348241806,
+            -0.01760132610797882,
+            0.00048788837739266455,
+            0.013956435024738312,
+            -0.020657232031226158,
+            0.0059282840229570866,
+            0.011967536993324757,
+            0.024310659617185593,
+            -0.007494648452848196,
+            0.006991021800786257,
+            -0.01913781464099884,
+            0.015561211854219437,
+            -0.00414425041526556,
+            -0.02250101789832115,
+            -0.0015524934278801084,
+            -0.013802786357700825,
+            0.0009058878640644252,
+            -0.005305152852088213,
+            0.00010189905151491985,
+            -0.0060435207560658455,
+            -0.0010563357500359416,
+            -0.004724701400846243,
+            -0.015262450091540813,
+            0.0013454942964017391,
+            -0.01610751822590828,
+            0.006790424697101116,
+            0.015288058668375015,
+            0.0144515261054039,
+            0.0028489052783697844,
+            0.00439606374129653,
+            -0.0030196262523531914,
+            0.002827565185725689,
+            -0.011916320770978928,
+            -0.004750309512019157,
+            0.007481844164431095,
+            -0.018284210935235023,
+            0.0005361703806556761,
+            -0.009338434785604477,
+            0.009099424816668034,
+            -0.022927820682525635,
+            0.003696108004078269,
+            -0.00207212520763278,
+            -0.014545422978699207,
+            -0.007072114385664463,
+            -0.007908646948635578,
+            0.023013180121779442,
+            0.010285935364663601,
+            0.01458810269832611,
+            0.005049071274697781,
+            -0.018130561336874962,
+            -0.01567218080163002,
+            0.0008263959898613393,
+            -0.001281473902054131,
+            0.02209128811955452,
+            -0.018932949751615524,
+            0.004464352037757635,
+            0.00844641774892807,
+            -0.02878354676067829,
+            -0.012283369898796082,
+            0.001568498439155519,
+            -0.01335891243070364,
+            0.009483546949923038,
+            0.009850597009062767,
+            -0.00800254289060831,
+            -0.0037729323375970125,
+            -0.004417404066771269,
+            -0.01630384847521782,
+            -0.006854445207864046,
+            -0.002769947052001953,
+            0.01169438287615776,
+            -0.009731092490255833,
+            0.0021916297264397144,
+            0.005445997230708599,
+            0.0121980095282197,
+            -0.007853162474930286,
+            -0.011207828298211098,
+            -0.010687129572033882,
+            0.002074259100481868,
+            0.015416099689900875,
+            -0.01823299378156662,
+            -0.010260327719151974,
+            0.015347810462117195,
+            -0.017515966668725014,
+            0.009893277660012245,
+            -0.00962012354284525,
+            -0.021732773631811142,
+            -0.006730672437697649,
+            0.016978194937109947,
+            0.000151514817844145,
+            0.02453259564936161,
+            0.01124197244644165,
+            -0.00622704578563571,
+            -0.006397766526788473,
+            -0.02349119819700718,
+            0.01201021671295166,
+            -0.006901393178850412,
+            0.0069739497266709805,
+            -0.000578317150939256,
+            -6.068595394026488e-05,
+            0.011711454950273037,
+            -0.001123557100072503,
+            -0.004107972141355276,
+            -0.011907784268260002,
+            -0.008087903261184692,
+            0.02951764687895775,
+            -0.007554400712251663,
+            0.017686687409877777,
+            0.01259066816419363,
+            -0.006884321104735136,
+            0.008361057378351688,
+            0.009048208594322205,
+            0.00388603494502604,
+            -0.010823706164956093,
+            0.01458810269832611,
+            -0.010806634090840816,
+            -0.016337992623448372,
+            0.008373861201107502,
+            0.007106258533895016,
+            -0.012027288787066936,
+            -0.008971384726464748,
+            0.0018619250040501356,
+            -0.0011854433687403798,
+            0.0003201017389073968,
+            -0.011011499911546707,
+            -0.009150641039013863,
+            -0.002554411767050624,
+            -0.005416121333837509,
+            0.0021830936893820763,
+            0.0027976890560239553,
+            0.008480561897158623,
+            0.002078527119010687,
+            -0.01645749621093273,
+            -0.01976948231458664,
+            -0.022279080003499985,
+            -0.015629500150680542,
+            0.00491249468177557,
+            0.0001392442500218749,
+            0.010268863290548325,
+            -0.007716585882008076,
+            0.014758823439478874,
+            -0.02079380862414837,
+            0.011412693187594414,
+            0.013324768282473087,
+            0.006508735008537769,
+            -0.013760106638073921,
+            -0.023098541423678398,
+            -0.012573596090078354,
+            0.0030004202853888273,
+            0.006952609401196241,
+            0.015125873498618603,
+            -0.00032917127828113735,
+            -0.008757983334362507,
+            0.00399700365960598,
+            -0.012573596090078354,
+            0.006769084371626377,
+            -0.00616302527487278,
+            0.0069227335043251514,
+            0.0011267580557614565,
+            -0.006901393178850412,
+            0.007985470816493034,
+            -0.005740491207689047,
+            0.0033717381302267313,
+            -0.008655550889670849,
+            -0.01663675345480442,
+            0.016431888565421104,
+            -0.0013518963241949677,
+            -0.007434896193444729,
+            -0.00860433466732502,
+            -0.004257353022694588,
+            -0.0015055451076477766,
+            -0.0036790359299629927,
+            0.009526227600872517,
+            0.011335869319736958,
+            0.027042193338274956,
+            0.00886895228177309,
+            0.008164728060364723,
+            0.004287228919565678,
+            -0.0005313688889145851,
+            0.013162583112716675,
+            0.018830517306923866,
+            0.00020873300672974437,
+            0.002744338707998395,
+            -0.0045539806596934795,
+            -0.02168155647814274,
+            -0.004562516696751118,
+            0.017686687409877777,
+            0.00937257893383503,
+            0.002082795137539506,
+            -0.007302587386220694,
+            0.0013999115908518434,
+            0.0019451514817774296,
+            -0.02299610897898674,
+            0.015919726341962814,
+            -0.005919747985899448,
+            -0.0041613224893808365,
+            0.015014905482530594,
+            -0.016594072803854942,
+            0.0052667404524981976,
+            -0.009176249615848064,
+            0.0031988832633942366,
+            0.0023346086964011192,
+            0.008267160505056381,
+            0.03762689232826233,
+            -0.01496368832886219,
+            0.006534343119710684,
+            0.021135250106453896,
+            -0.008629942312836647,
+            -0.017772048711776733,
+            -0.003045234363526106,
+            -0.006000840570777655,
+            0.004191198851913214,
+            0.012223618105053902,
+            0.0030921828001737595,
+            -0.0007869167602621019,
+            0.017515966668725014,
+            0.001877930131740868,
+            0.01206143293529749,
+            -0.014852720312774181,
+            -1.1286920198472217e-05,
+            -0.008937240578234196,
+            0.013410128653049469,
+            0.0019728937186300755,
+            -0.015902653336524963,
+            -0.002094532363116741,
+            0.02043529413640499,
+            -0.018215922638773918,
+            0.013717425987124443,
+            -0.006090468727052212,
+            0.02673489600419998,
+            0.004711897578090429,
+            0.0073153916746377945,
+            -0.004818597808480263,
+            0.009705484844744205,
+            0.013614993542432785,
+            0.024361874908208847,
+            -0.0015610294649377465,
+            0.002044382970780134,
+            0.0058557274751365185,
+            0.013691818341612816,
+            0.01610751822590828,
+            0.006478859111666679,
+            -0.0016549259889870882,
+            0.029978593811392784,
+            0.01720866933465004,
+            0.007306855637580156,
+            0.004438743926584721,
+            0.0001512480666860938,
+            -0.017200132831931114,
+            -0.005889871623367071,
+            -0.03168580308556557,
+            0.0036342216189950705,
+            0.013947899453341961,
+            0.010371295735239983,
+            -0.015108801424503326,
+            0.0083482526242733,
+            0.007050774060189724,
+            -0.018335426226258278,
+            0.013196727260947227,
+            0.0012217216426506639,
+            0.008826271630823612,
+            0.004387527704238892,
+            0.008006811141967773,
+            0.008510437794029713,
+            0.022159576416015625,
+            0.008783590979874134,
+            0.0063508180901408195,
+            -0.008006811141967773,
+            -0.01912074349820614,
+            0.0026995246298611164,
+            -0.016926979646086693,
+            0.001235592644661665,
+            0.010388367809355259,
+            -0.006141684949398041,
+            0.002078527119010687,
+            0.014357629232108593,
+            0.011113932356238365,
+            0.00018219123012386262,
+            -0.009807917289435863,
+            0.021374259144067764,
+            -0.0026162981521338224,
+            0.002697390504181385,
+            -0.02134011499583721,
+            0.0044515482150018215,
+            0.013649137690663338,
+            -0.00956037174910307,
+            0.012018753215670586,
+            -0.01951340213418007,
+            0.013469880446791649,
+            0.001887533231638372,
+            -0.01213825773447752,
+            -0.001184376422315836,
+            -0.010772489942610264,
+            0.023218045011162758,
+            -0.01574900560081005,
+            0.014084476046264172,
+            0.02490818314254284,
+            -0.002483989344909787,
+            -0.005569770000874996,
+            0.0022684542927891016,
+            0.012727244757115841,
+            -0.014605174772441387,
+            -0.006329478230327368,
+            0.0015738335205242038,
+            0.030354179441928864,
+            -0.009765236638486385,
+            0.002919327700510621,
+            -0.00886895228177309,
+            0.0199231319129467,
+            -0.01315404660999775,
+            -0.006760548334568739,
+            0.005612450186163187,
+            0.02673489600419998,
+            -0.00268245255574584,
+            0.027554357424378395,
+            0.016722114756703377,
+            0.0010526011465117335,
+            0.006171561311930418,
+            -0.01521977037191391,
+            0.006905661430209875,
+            0.017686687409877777,
+            0.01246262714266777,
+            0.017396461218595505,
+            -0.009048208594322205,
+            0.009543299674987793,
+            0.027537284418940544,
+            -0.00713186664506793,
+            0.0038177466485649347,
+            0.012308978475630283,
+            0.011156612075865269,
+            0.003747324226424098,
+            -0.0017541574779897928,
+            0.004259487148374319,
+            -0.015236842446029186,
+            0.013136974535882473,
+            -0.00943233072757721,
+            0.006393498741090298,
+            0.021510835736989975,
+            0.007925719022750854,
+            9.296288044424728e-05,
+            0.0006012577214278281,
+            0.012445555068552494,
+            0.03315400332212448,
+            -0.003975663334131241,
+            0.013768642209470272,
+            -0.0067861564457416534,
+            0.004250951111316681,
+            -0.01007253397256136,
+            -0.012872357852756977,
+            -0.0001271737419301644,
+            -0.014630783349275589,
+            -0.0010915468446910381,
+            7.54239663365297e-05,
+            0.008783590979874134,
+            -0.017618399113416672,
+            0.0015279522631317377,
+            -0.006833104882389307,
+            -0.007187350653111935,
+            -0.011865103617310524,
+            0.01137001346796751,
+            -0.010968819260597229,
+            -0.007332463748753071,
+            0.007281247526407242,
+            0.0076739052310585976,
+            -6.271993333939463e-05,
+            -0.02156205289065838,
+            -0.01059323363006115,
+            0.002249248092994094,
+            0.0036064793821424246,
+            0.014750287868082523,
+            -0.02067430317401886,
+            -0.004289363045245409,
+            0.01002131775021553,
+            -0.009133568964898586,
+            -0.008224480785429478,
+            0.0033717381302267313,
+            -0.01052494440227747,
+            -0.017703760415315628,
+            -0.012078505009412766,
+            0.00831837672740221,
+            -0.019803626462817192,
+            -0.029876161366701126,
+            -0.017635472118854523,
+            0.009329898282885551,
+            -0.016448961570858955,
+            -0.004519836511462927,
+            -0.025352057069540024,
+            0.02181813307106495,
+            0.008489097468554974,
+            0.011173684149980545,
+            0.006811764556914568,
+            -0.006547147408127785,
+            0.012949181720614433,
+            0.0010008513927459717,
+            -0.004289363045245409,
+            0.014050331898033619,
+            0.015424635261297226,
+            -0.010900530964136124,
+            -0.010866386815905571,
+            0.048757895827293396,
+            -0.004455816000699997,
+            0.008809199556708336,
+            -0.0011715723667293787,
+            0.02555692195892334,
+            0.022381512448191643,
+            0.01118222065269947,
+            0.009944493882358074,
+            0.026171516627073288,
+            -0.005813047289848328,
+            0.0008845477714203298,
+            0.018711013719439507,
+            0.002041182015091181,
+            0.028425034135580063,
+            0.016209950670599937,
+            -0.003617149544879794,
+            -0.008412273600697517,
+            -0.0032436975743621588,
+            0.0024818554520606995,
+            0.017353782430291176,
+            -0.003011090215295553,
+            -0.00116517033893615,
+            0.011976072564721107,
+            -0.018796373158693314,
+            -0.02606908418238163,
+            -0.028544537723064423,
+            -0.008096439763903618,
+            -0.003841220634058118,
+            0.0006578090251423419,
+            0.021135250106453896,
+            0.0029641420114785433,
+            0.0074690403416752815,
+            0.01937682554125786,
+            0.005283812526613474,
+            0.0044302078895270824,
+            0.004472888074815273,
+            -0.0020977333188056946,
+            -0.01002131775021553,
+            0.004835669882595539,
+            0.004472888074815273,
+            -0.0012729378649964929,
+            0.01953047327697277,
+            0.008126315660774708,
+            0.008736643008887768,
+            -0.003049502382054925,
+            0.002296196296811104,
+            0.006598363630473614,
+            0.018454931676387787,
+            0.0014372568111866713,
+            -0.026171516627073288,
+            0.008011079393327236,
+            0.009867669083178043,
+            -0.004639341030269861,
+            -0.004054621793329716,
+            0.003964993637055159,
+            -0.003877498907968402,
+            -0.004150652326643467,
+            -0.003719581989571452,
+            0.021203538402915,
+            0.022313224151730537,
+            0.02017921209335327,
+            0.003395212348550558,
+            0.012377266772091389,
+            0.0010584697593003511,
+            -0.02750314027070999,
+            0.007648297119885683,
+            -0.007055042311549187,
+            0.01418690849095583,
+            -0.0359196811914444,
+            -0.00670933211222291,
+            -0.004733237437903881,
+            0.001975027611479163,
+            0.03120778501033783,
+            0.0021478827111423016,
+            0.006564219482243061,
+            0.00552282202988863,
+            0.011959000490605831,
+            -0.004310703370720148,
+            0.024378947913646698,
+            -0.002678184537217021,
+            0.004737505689263344,
+            -0.002415701048448682,
+            -0.01084077823907137,
+            -0.020110923796892166,
+            0.01110539585351944,
+            0.008689695037901402,
+            0.0616643987596035,
+            -0.006513003259897232,
+            0.023047324270009995,
+            -0.01373449806123972,
+            0.007234299089759588,
+            0.0218522772192955,
+            0.024447236210107803,
+            0.02748606912791729,
+            -0.0010691398056223989,
+            0.004707629326730967,
+            -0.014647855423390865,
+            -0.018523219972848892,
+            0.009585979394614697,
+            -0.007242835126817226,
+            0.00597523245960474,
+            -0.022415656596422195,
+            -0.02016214095056057,
+            -0.008988456800580025,
+            -0.0057532950304448605,
+            -0.001239860663190484,
+            0.004618000704795122,
+            0.005309420637786388,
+            -0.0034549646079540253,
+            0.025864219292998314,
+            -0.0005332361324690282,
+            -0.0033141199965029955,
+            -0.028595754876732826,
+            -0.009184785187244415,
+            0.011890712194144726,
+            -0.05049924924969673,
+            0.018779302015900612,
+            -0.00272940075956285,
+            -0.02748606912791729,
+            -0.01744767837226391,
+            -0.0035595311783254147,
+            0.015629500150680542,
+            -0.012334587052464485,
+            0.010610305704176426,
+            0.0026205661706626415,
+            -0.0006882187444716692,
+            0.0037772003561258316,
+            0.014545422978699207,
+            0.0224839448928833,
+            0.004370455630123615,
+            0.009329898282885551,
+            0.005556966178119183,
+            0.009978638030588627,
+            -0.007580008823424578,
+            0.019974347203969955,
+            0.001809641718864441,
+            0.0045326403342187405,
+            0.0020337130408734083,
+            -0.016961123794317245,
+            -0.004536908585578203,
+            -0.006017912644892931,
+            0.028083590790629387,
+            0.015339274890720844,
+            -0.01041397638618946,
+            -0.005347833037376404,
+            -0.005569770000874996,
+            -0.011924856342375278,
+            0.0013028139946982265,
+            0.013512561097741127,
+            0.010448120534420013,
+            -0.038856081664562225,
+            0.03317107632756233,
+            0.01728549413383007,
+            -0.01848907582461834,
+            -0.024208227172493935,
+            -0.002460515359416604,
+            0.020708447322249413,
+            0.023832641541957855,
+            -0.03271012753248215,
+            -0.013427200727164745,
+            0.014468598179519176,
+            -0.014383237808942795,
+            6.825502350693569e-05,
+            -0.0005495079676620662,
+            -0.0205547995865345,
+            0.011250508949160576,
+            -0.014810039661824703,
+            0.019291464239358902,
+            -0.0025928239338099957,
+            0.022518089041113853,
+            0.025727642700076103,
+            -0.004528372548520565,
+            0.0018352499464526772,
+            -0.01355524081736803,
+            -0.008207408711314201,
+            0.03277841582894325,
+            0.0018053737003356218,
+            -0.005428925156593323,
+            -0.007917182520031929,
+            -0.0023986289743334055,
+            -0.012676028534770012,
+            0.01490393653512001,
+            0.007349535822868347,
+            0.0012035825056955218,
+            0.005241132341325283,
+            -0.0010878123575821519,
+            0.025215480476617813,
+            0.004763113800436258,
+            -0.004763113800436258,
+            0.008834808133542538,
+            -0.015108801424503326,
+            0.01937682554125786,
+            -0.0029022556263953447,
+            -0.008361057378351688,
+            0.0053179566748440266,
+            0.014656390994787216,
+            0.004792989697307348,
+            -0.0038518907967954874,
+            -0.010482264682650566,
+            -0.004857010208070278,
+            0.007669637445360422,
+            0.027059266343712807,
+            0.014519814401865005,
+            -0.007597080897539854,
+            -0.011412693187594414,
+            0.01451127789914608,
+            0.007883038371801376,
+            0.0006924867630004883,
+            -0.0071788146160542965,
+            0.012095577083528042,
+            -0.01554413978010416,
+            0.014681999571621418,
+            -0.009346970357000828,
+            -0.020862096920609474,
+            0.008988456800580025,
+            -0.02506183087825775,
+            0.0017125442391261458,
+            -0.01835249923169613,
+            0.002155351685360074,
+            -0.007968398742377758,
+            -0.011395621113479137,
+            0.009500619024038315,
+            -0.000326237000990659,
+            0.03878779336810112,
+            0.011190756224095821,
+            0.009679876267910004,
+            -0.0014489938039332628,
+            0.07416117191314697,
+            -0.023439982905983925,
+            -0.01033715158700943,
+            -0.028954267501831055,
+            -0.012949181720614433,
+            -0.0063636223785579205,
+            0.01002131775021553,
+            0.008104975335299969,
+            -0.0004057289334014058,
+            -0.011301725171506405,
+            -0.007072114385664463,
+            0.015783149749040604,
+            -0.026615392416715622,
+            -0.0007479710620827973,
+            0.02159619703888893,
+            0.0030729766003787518,
+            0.006564219482243061,
+            0.015006368979811668,
+            -0.007268443237990141,
+            0.01966704986989498,
+            -0.016346529126167297,
+            -0.012744316831231117,
+            0.005894139874726534,
+            0.016329456120729446,
+            0.005339297000318766,
+            0.0008904163260012865,
+            0.011848031543195248,
+            -0.0006097938166931272,
+            0.005108823534101248,
+            0.02323511801660061,
+            -0.02014506794512272,
+            0.014417381957173347,
+            -0.008412273600697517,
+            0.01825006678700447,
+            0.007644029334187508,
+            0.018028128892183304,
+            -0.005134431645274162,
+            -0.008877487853169441,
+            -0.008442149497568607,
+            0.010943210683763027,
+            0.015962406992912292,
+            -0.012215081602334976,
+            -0.0019686257001012564,
+            0.004048219881951809,
+            -0.018028128892183304,
+            -0.006897125393152237,
+            -0.012180937454104424,
+            0.008049491792917252,
+            0.021510835736989975,
+            -0.012471163645386696,
+            -0.01675625890493393,
+            -0.022176647558808327,
+            0.012317514047026634,
+            -0.008190336637198925,
+            0.02772507816553116,
+            -0.013623529113829136,
+            -0.00523686408996582,
+            0.0211864672601223,
+            0.0004182662523817271,
+            0.0012174536241218448,
+            -0.027776293456554413,
+            -0.010166430845856667,
+            -0.014997833408415318
+          ],
+          "title": "6 Related Work"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "7b418b19-51bf-4af0-8323-70dc89f40077",
+      "type": "child",
+      "source": {
+        "id": "65742957-6bda-446d-a61a-0fe20313c760",
+        "properties": {
+          "page_content": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity\n\nWilliam Fedus∗ liamfedus@google.com Barret Zoph∗ barretzoph@google.com Noam Shazeer noam@google.com Google, Mountain View, CA 94043, USA Editor: Alexander Clark\n\nAbstract\n\nIn deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) models defy this and instead select different parameters for each incoming example. The result is a sparsely-activated model—with an outrageous number of parameters—but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs, and training instability. We address these with the introduction of the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques mitigate the instabilities, and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large (Raffel et al., 2019) to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the \"Colossal Clean Crawled Corpus\", and achieve a 4x speedup over the T5-XXL model.12 Keywords: mixture-of-experts, natural language processing, sparsity, large-scale machine learning, distributed computing arXiv:2101.03961v3 [cs.LG] 16 Jun 2022 Contents\n\n1 Introduction 3 2 Switch Transformer 4 2.1 Simplifying Sparse Routing 5 2.2 Efficient Sparse Routing 6 2.3 Putting It All Together: The Switch Transformer 8 2.4 Improved Training and Fine-Tuning Techniques 8 3 Scaling Properties 11 3.1 Scaling Results on a Step-Basis 12 3.2 Scaling Results on a Time-Basis 13 3.3 Scaling Versus a Larger Dense Model 13 4 Downstream Results 14 4.1 Fine-Tuning 14 4.2 Distillation 16 4.3 Multilingual Learning 17 5 Designing Models with Data, Model, and Expert-Parallelism 18 5.1 Data Parallelism 20 5.2 Model Parallelism 20 5.3 Model and Data Parallelism 21 5.4 Expert and Data Parallelism 22 5.5 Expert, Model and Data Parallelism 22 5.6 Towards Trillion Parameter Models 22 6 Related Work 24 7 Discussion 25 8 Future Work 26 9 Conclusion 27 A Switch for Attention 27 B Preventing Token Dropping with No-Token-Left-Behind 29\n\nC Encouraging Exploration Across Experts 29 D Switch Transformers in Lower Compute Regimes 29 E Relation of Upstream to Downstream Model Performance 32 F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/switch_transformers.md"
+          },
+          "summary": "The Switch Transformer is a sparsely-activated model that simplifies the Mixture of Experts (MoE) approach, allowing for efficient scaling to trillion-parameter models with reduced communication and computational costs. It achieves significant speedups in pre-training and fine-tuning by activating only a subset of neural network weights for each input, maintaining a constant computational cost. The model demonstrates superior scaling properties and improved performance across various natural language processing tasks, including multilingual settings. It also allows for effective distillation into smaller dense models, preserving a portion of the quality gains. The architecture is designed to maximize parameter count while keeping floating point operations per example constant, making it suitable for both large-scale and smaller-scale applications.",
+          "headlines": {
+            "1 Introduction": [],
+            "2 Switch Transformer": [
+              "2.1 Simplifying Sparse Routing",
+              "2.2 Efficient Sparse Routing",
+              "2.3 Putting It All Together: The Switch Transformer",
+              "2.4 Improved Training and Fine-Tuning Techniques"
+            ],
+            "3 Scaling Properties": [
+              "3.1 Scaling Results on a Step-Basis",
+              "3.2 Scaling Results on a Time-Basis",
+              "3.3 Scaling Versus a Larger Dense Model"
+            ],
+            "4 Downstream Results": [
+              "4.1 Fine-Tuning",
+              "4.2 Distillation",
+              "4.3 Multilingual Learning"
+            ],
+            "5 Designing Models with Data, Model, and Expert-Parallelism": [
+              "5.1 Data Parallelism",
+              "5.2 Model Parallelism",
+              "5.3 Model and Data Parallelism",
+              "5.4 Expert and Data Parallelism",
+              "5.5 Expert, Model and Data Parallelism",
+              "5.6 Towards Trillion Parameter Models"
+            ],
+            "6 Related Work": [],
+            "7 Discussion": [],
+            "8 Future Work": [],
+            "9 Conclusion": [],
+            "A Switch for Attention": [],
+            "B Preventing Token Dropping with No-Token-Left-Behind": [],
+            "C Encouraging Exploration Across Experts": [],
+            "D Switch Transformers in Lower Compute Regimes": [],
+            "E Relation of Upstream to Downstream Model Performance": [],
+            "F Pseudo Code for Switch Transformers": []
+          },
+          "summary_embedding": [
+            -0.0060447100549936295,
+            -0.00390331051312387,
+            -0.01784370467066765,
+            -0.0021820589900016785,
+            0.013227885589003563,
+            0.011338187381625175,
+            0.0165580902248621,
+            0.024240797385573387,
+            -0.005986624862998724,
+            0.03596621751785278,
+            -0.004852031823247671,
+            -0.0068811336532235146,
+            -0.003574162255972624,
+            0.014242436736822128,
+            0.011578272096812725,
+            -0.033302053809165955,
+            -0.024519605562090874,
+            0.01050176378339529,
+            0.016093410551548004,
+            -0.006795942317694426,
+            0.007473600562661886,
+            -0.030513975769281387,
+            -0.06124480068683624,
+            -0.006838538218289614,
+            -0.0026099516544491053,
+            0.01029265858232975,
+            0.0010329445358365774,
+            -0.010873507708311081,
+            -0.053469158709049225,
+            0.023311438038945198,
+            0.025294071063399315,
+            0.005092116538435221,
+            -0.029584616422653198,
+            0.0009361362317577004,
+            -0.05161044001579285,
+            -0.04448534920811653,
+            0.0379488542675972,
+            0.012561844661831856,
+            -0.04364892467856407,
+            0.039249956607818604,
+            -0.004875265993177891,
+            -0.0022440163884311914,
+            -0.021499186754226685,
+            -0.0034909071400761604,
+            -0.007039899472147226,
+            -0.009851212613284588,
+            -0.02874819189310074,
+            0.0014037203509360552,
+            0.021886419504880905,
+            -0.0056962003000080585,
+            0.023264970630407333,
+            -0.024271776899695396,
+            0.01936165988445282,
+            0.04240977764129639,
+            -0.06957805901765823,
+            0.024659009650349617,
+            0.017549408599734306,
+            0.05120771750807762,
+            0.06511713564395905,
+            0.018339363858103752,
+            0.021669568493962288,
+            -0.014141756109893322,
+            -0.017766257748007774,
+            -0.011872569099068642,
+            0.004673904739320278,
+            -0.014970434829592705,
+            -0.01891246810555458,
+            0.008449427783489227,
+            -0.020461400970816612,
+            0.008209343999624252,
+            -0.05576157942414284,
+            -0.014730350114405155,
+            -0.034200433641672134,
+            -0.01140014547854662,
+            0.017642343416810036,
+            -0.018215449526906013,
+            0.030126741155982018,
+            0.02997184917330742,
+            0.002638994250446558,
+            0.0007822110201232135,
+            0.021049996837973595,
+            0.006509389728307724,
+            -0.0053438181057572365,
+            0.012321759946644306,
+            0.028159597888588905,
+            -0.01450575515627861,
+            -0.010106786154210567,
+            -0.03454120084643364,
+            0.01861817203462124,
+            0.01589204929769039,
+            -0.04194509983062744,
+            -0.012569589540362358,
+            -0.013824224472045898,
+            0.032341714948415756,
+            -0.009549170732498169,
+            0.04098476096987724,
+            -0.010083552449941635,
+            0.001383390510454774,
+            -0.031149037182331085,
+            -0.009463978931307793,
+            0.017069239169359207,
+            0.006172496825456619,
+            -0.016589069738984108,
+            -0.0012846461031585932,
+            0.016170857474207878,
+            0.006617815233767033,
+            -0.030142230913043022,
+            0.04488807171583176,
+            0.023497309535741806,
+            0.055296897888183594,
+            0.05579255893826485,
+            -0.037391237914562225,
+            -0.002445377642288804,
+            0.0344172865152359,
+            -0.0228312686085701,
+            -0.015040136873722076,
+            -0.034014564007520676,
+            -0.01768881268799305,
+            0.035935238003730774,
+            -0.0318460576236248,
+            0.01556677371263504,
+            0.028066661208868027,
+            -0.025712283328175545,
+            0.003488970920443535,
+            -0.03943582624197006,
+            0.003622566582635045,
+            0.01220559049397707,
+            0.006095050368458033,
+            -0.0026293133851140738,
+            -0.013615119270980358,
+            0.008906363509595394,
+            0.003055269829928875,
+            0.00800798274576664,
+            -0.04615819454193115,
+            0.018478767946362495,
+            -0.00454224506393075,
+            0.04033420979976654,
+            0.042502716183662415,
+            -0.02730768360197544,
+            -0.016743963584303856,
+            0.008557853288948536,
+            0.012104909867048264,
+            0.04188314080238342,
+            -0.007287728600203991,
+            -0.00121784838847816,
+            -0.0007836631848476827,
+            0.007322579622268677,
+            0.047552235424518585,
+            0.004646798130124807,
+            0.0020697612781077623,
+            -0.027230238541960716,
+            0.014978179708123207,
+            -0.019733402878046036,
+            0.027648448944091797,
+            0.023373395204544067,
+            0.02942972257733345,
+            -0.04048910364508629,
+            0.017518429085612297,
+            -0.011818356812000275,
+            -0.005568413063883781,
+            -0.002726121572777629,
+            -0.02856232039630413,
+            -0.0074426219798624516,
+            0.033983584493398666,
+            0.015365412458777428,
+            0.012956822291016579,
+            0.03271345794200897,
+            0.0012052633101120591,
+            -0.042502716183662415,
+            0.03804178908467293,
+            0.024550583213567734,
+            0.04640602692961693,
+            0.023466331884264946,
+            0.010796061716973782,
+            0.006571347359567881,
+            0.013375034555792809,
+            -0.019377149641513824,
+            0.015907539054751396,
+            0.02560385875403881,
+            0.009750531986355782,
+            -0.03167567402124405,
+            -0.0009501734166406095,
+            0.01178737822920084,
+            -0.004546117503196001,
+            0.03701949119567871,
+            0.04104671627283096,
+            0.020569827407598495,
+            0.05030933395028114,
+            0.0020155487582087517,
+            0.017936641350388527,
+            0.005936284549534321,
+            0.02820606529712677,
+            0.030467506498098373,
+            -0.005285732913762331,
+            0.005076627247035503,
+            -0.01136916596442461,
+            -0.03385967016220093,
+            0.014110777527093887,
+            0.01561324205249548,
+            -0.008077683858573437,
+            -0.01697630248963833,
+            0.006033093202859163,
+            0.009169681929051876,
+            -0.01795213110744953,
+            -0.017456471920013428,
+            0.005971135571599007,
+            0.018866000697016716,
+            -0.013576395809650421,
+            0.024689987301826477,
+            -0.04442339017987251,
+            0.029119934886693954,
+            -0.033209118992090225,
+            -0.003384418087080121,
+            -0.010641167871654034,
+            -0.0030843124259263277,
+            0.0041666291654109955,
+            0.011314953677356243,
+            0.013406013138592243,
+            0.002042655134573579,
+            -0.013909416273236275,
+            0.024519605562090874,
+            0.018447788432240486,
+            0.012073930352926254,
+            -0.03148980438709259,
+            -0.014397329650819302,
+            -0.0015673263696953654,
+            -0.008743725717067719,
+            -0.034138478338718414,
+            0.026966920122504234,
+            0.014048820361495018,
+            -0.03844451159238815,
+            0.017797237262129784,
+            0.029538147151470184,
+            0.014420564286410809,
+            -0.004790074657648802,
+            -0.0463130883872509,
+            0.04045812413096428,
+            -0.030436528846621513,
+            -0.033333033323287964,
+            -0.010478530079126358,
+            -0.0003146269591525197,
+            0.02894955314695835,
+            -0.0064358157105743885,
+            -0.050835974514484406,
+            0.01184933539479971,
+            0.020631784573197365,
+            0.00010763872705865651,
+            0.005731051322072744,
+            -0.019593998789787292,
+            0.00028631053282879293,
+            -0.03395260497927666,
+            0.0090844901278615,
+            0.026750069111585617,
+            0.008867640048265457,
+            -0.016418687999248505,
+            0.03351890295743942,
+            -0.0004121613164898008,
+            -0.018695618957281113,
+            -0.003889757441356778,
+            0.006885006092488766,
+            0.034912943840026855,
+            0.004836542531847954,
+            -0.02946070022881031,
+            0.05665995925664902,
+            -0.02100352756679058,
+            0.021344292908906937,
+            0.05631919577717781,
+            -0.019439106807112694,
+            -0.008472662419080734,
+            -0.002065889071673155,
+            0.0026196325197815895,
+            -0.0260530486702919,
+            -0.008937342092394829,
+            0.032341714948415756,
+            0.041728246957063675,
+            -0.018742086365818977,
+            -0.001398879918269813,
+            0.006780453026294708,
+            0.009425255469977856,
+            0.004948840010911226,
+            0.02692045085132122,
+            -0.012128143571317196,
+            -0.014559967443346977,
+            0.05198218300938606,
+            0.028314489871263504,
+            -0.012081675231456757,
+            -0.0164031982421875,
+            0.019888296723365784,
+            -0.024844881147146225,
+            -0.020244551822543144,
+            -0.02431824430823326,
+            -0.020414933562278748,
+            -0.0042518205009400845,
+            0.009525936096906662,
+            0.037484172731637955,
+            -0.03968365862965584,
+            0.012747716158628464,
+            0.021700548008084297,
+            0.02981695532798767,
+            -0.019532041624188423,
+            -0.05910727381706238,
+            -0.0016128262504935265,
+            0.03296129032969475,
+            -0.015458349138498306,
+            -0.049379974603652954,
+            0.015164051204919815,
+            0.0260530486702919,
+            0.05957195535302162,
+            -0.001461805310100317,
+            -0.014954946003854275,
+            0.061926331371068954,
+            0.03026614524424076,
+            -0.004383479710668325,
+            -0.009494957514107227,
+            -0.0127864396199584,
+            0.003396035172045231,
+            0.00465454300865531,
+            -0.03364282101392746,
+            -0.041170634329319,
+            -0.011841590516269207,
+            -0.010284913703799248,
+            0.008108663372695446,
+            -0.040922801941633224,
+            0.03531566634774208,
+            -0.021917399019002914,
+            0.010950954630970955,
+            0.015473837964236736,
+            0.037391237914562225,
+            -0.021344292908906937,
+            -0.01284839678555727,
+            -0.01572941243648529,
+            -0.024968795478343964,
+            -0.014010096900165081,
+            0.021917399019002914,
+            0.007682706229388714,
+            3.4699722164077684e-05,
+            0.010075807571411133,
+            -0.024659009650349617,
+            -0.051052823662757874,
+            0.0291044469922781,
+            0.0015605497173964977,
+            -0.015938518568873405,
+            -0.004623564425855875,
+            -0.01977987214922905,
+            -0.01122201792895794,
+            -0.012608313001692295,
+            -0.039249956607818604,
+            0.021111954003572464,
+            -0.01135367713868618,
+            0.015876561403274536,
+            0.0009250032599084079,
+            0.003659353591501713,
+            0.028624277561903,
+            0.01116006076335907,
+            0.0051811798475682735,
+            -0.03255856782197952,
+            -0.018277406692504883,
+            0.010122275911271572,
+            -0.00561100896447897,
+            -0.018339363858103752,
+            -0.013351799920201302,
+            -0.05037129297852516,
+            -0.03305422514677048,
+            -0.008054450154304504,
+            -0.05415068939328194,
+            0.0020697612781077623,
+            -0.014296649023890495,
+            -0.02232012152671814,
+            -0.005990497302263975,
+            0.03107159025967121,
+            -0.003235333366319537,
+            0.025294071063399315,
+            0.0076594725251197815,
+            0.037360258400440216,
+            0.040117356926202774,
+            0.008426194079220295,
+            0.03065337985754013,
+            -0.0029991210903972387,
+            -0.016867877915501595,
+            0.05486319959163666,
+            -0.01881953328847885,
+            0.013529927469789982,
+            -0.008604321628808975,
+            0.008604321628808975,
+            0.03299226611852646,
+            -0.003794885240495205,
+            0.02007416822016239,
+            -0.0311955064535141,
+            -0.07998688519001007,
+            0.01637221872806549,
+            0.028159597888588905,
+            -0.06418777257204056,
+            0.014691627584397793,
+            -0.020275529474020004,
+            0.02052335813641548,
+            0.02411688305437565,
+            -0.0009114501299336553,
+            -0.014064309187233448,
+            0.005595519673079252,
+            0.0020833145827054977,
+            -0.01456771232187748,
+            -0.002691270550712943,
+            -0.009417511522769928,
+            0.008774704299867153,
+            -0.017007282003760338,
+            -0.013406013138592243,
+            -0.034231413155794144,
+            0.010873507708311081,
+            0.01233724970370531,
+            -0.003969140350818634,
+            0.0011500825639814138,
+            -0.03277541697025299,
+            -0.007876322604715824,
+            -0.02145271934568882,
+            0.03382869064807892,
+            0.012058441527187824,
+            -0.030699847266077995,
+            -0.007814365439116955,
+            0.015264731831848621,
+            -0.007876322604715824,
+            0.0008906363509595394,
+            0.024132372811436653,
+            0.02103450708091259,
+            0.0028364830650389194,
+            -0.0009700191440060735,
+            0.02438020147383213,
+            0.007427132688462734,
+            -0.01720864325761795,
+            0.028794659301638603,
+            0.014288905076682568,
+            0.060160547494888306,
+            -0.03271345794200897,
+            0.001297231181524694,
+            -0.04135650396347046,
+            -0.031118059530854225,
+            0.006408709101378918,
+            0.014350862242281437,
+            0.007597515359520912,
+            -0.028252532705664635,
+            0.005978880450129509,
+            0.02004319056868553,
+            0.01469937153160572,
+            0.009704063646495342,
+            0.0037580979987978935,
+            -0.036740683019161224,
+            -0.000691695255227387,
+            -0.011214273050427437,
+            -0.0038781403563916683,
+            0.035656433552503586,
+            -0.008186109364032745,
+            -0.05151750147342682,
+            -0.0006418389966711402,
+            -0.03255856782197952,
+            -0.0353776253759861,
+            0.021313315257430077,
+            -0.00839521549642086,
+            -0.0039207362569868565,
+            -0.013986863195896149,
+            -0.000537770101800561,
+            -0.010524998418986797,
+            0.027509044855833054,
+            0.028345469385385513,
+            0.022490503266453743,
+            -0.05384090170264244,
+            0.009479468688368797,
+            0.00023766436788719147,
+            -0.05817791447043419,
+            0.0009709871956147254,
+            -0.007117345929145813,
+            0.011191039346158504,
+            -0.020213572308421135,
+            -0.02414786070585251,
+            0.011500826105475426,
+            0.03392162546515465,
+            0.014304393902420998,
+            0.0321558453142643,
+            -0.013367289677262306,
+            -0.09250226616859436,
+            0.018292896449565887,
+            -0.008689512498676777,
+            0.04126356914639473,
+            0.00865078903734684,
+            -0.004278926644474268,
+            -0.015845581889152527,
+            -0.00865078903734684,
+            -0.006943091284483671,
+            0.01283290795981884,
+            0.004530628211796284,
+            -0.005626498255878687,
+            0.0016631665639579296,
+            0.013708055019378662,
+            -0.005967263597995043,
+            0.007481344975531101,
+            0.003651608945801854,
+            -0.013189162127673626,
+            -0.06059424951672554,
+            -0.04055105894804001,
+            -0.018540725111961365,
+            0.050835974514484406,
+            -0.0069159846752882,
+            -0.046684831380844116,
+            -0.010850274004042149,
+            0.05058814212679863,
+            0.012259802781045437,
+            -0.0334569476544857,
+            -0.0007841471815481782,
+            0.0050882440991699696,
+            0.015458349138498306,
+            0.004546117503196001,
+            -0.027292195707559586,
+            0.02820606529712677,
+            0.0008412640891037881,
+            0.02300165221095085,
+            -0.018339363858103752,
+            -0.005312839522957802,
+            0.035780347883701324,
+            -0.024844881147146225,
+            0.023791607469320297,
+            0.0055219451896846294,
+            0.031629208475351334,
+            0.002952652983367443,
+            0.010060317814350128,
+            0.004704883322119713,
+            0.0186491496860981,
+            0.004275054205209017,
+            0.0276639387011528,
+            0.011462102644145489,
+            0.006428070832043886,
+            0.021623101085424423,
+            -0.009378788061439991,
+            -0.014513500034809113,
+            0.01885051093995571,
+            -0.0014695499558001757,
+            0.025232113897800446,
+            -0.020538847893476486,
+            0.002726121572777629,
+            -0.0016883367206901312,
+            -0.017967619001865387,
+            -0.040922801941633224,
+            -0.0007405834621749818,
+            0.021747015416622162,
+            -0.011671207845211029,
+            -0.0228312686085701,
+            -0.013715799897909164,
+            -0.01585332676768303,
+            0.0038490979932248592,
+            0.034169457852840424,
+            -0.0398695282638073,
+            0.00963436160236597,
+            -0.000830615172162652,
+            -0.03125746175646782,
+            -0.0239619892090559,
+            -0.01093546487390995,
+            0.014614180661737919,
+            -0.01652711257338524,
+            -0.021173911169171333,
+            -0.028469383716583252,
+            -0.008410705253481865,
+            0.006048582494258881,
+            -0.05065010115504265,
+            -0.004054331686347723,
+            0.019965743646025658,
+            0.0276639387011528,
+            -0.003099801717326045,
+            -0.02177799493074417,
+            0.0012468908680602908,
+            0.022351099178195,
+            0.02936776541173458,
+            0.014226946979761124,
+            0.027586491778492928,
+            -0.002178186783567071,
+            -0.0011675080750137568,
+            -0.0073148347437381744,
+            0.01307299267500639,
+            0.0011287847300991416,
+            0.036616768687963486,
+            0.0122210793197155,
+            -0.020306508988142014,
+            0.0013959755888208747,
+            -0.006431943271309137,
+            0.029181893914937973,
+            -0.02270735427737236,
+            -9.832092473516241e-05,
+            0.03531566634774208,
+            0.027741385623812675,
+            0.004933350719511509,
+            -0.0143121387809515,
+            -0.05142456665635109,
+            0.021592123433947563,
+            -0.01765783317387104,
+            0.031025122851133347,
+            0.016666516661643982,
+            0.003397971158847213,
+            -0.0034212053287774324,
+            0.012368228286504745,
+            -0.022087780758738518,
+            -0.004406713880598545,
+            -0.021142931655049324,
+            0.04389675334095955,
+            -0.043184246867895126,
+            0.008472662419080734,
+            0.029445212334394455,
+            0.011926782317459583,
+            -0.019082851707935333,
+            -0.005425137002021074,
+            0.029212871566414833,
+            -0.00162637943867594,
+            0.01308848150074482,
+            0.0058665829710662365,
+            -0.026935940608382225,
+            0.01202746294438839,
+            -0.031660184264183044,
+            0.02177799493074417,
+            -0.013436991721391678,
+            0.018447788432240486,
+            0.03342596814036369,
+            -0.0030397805385291576,
+            -0.0038239278364926577,
+            -0.016263794153928757,
+            -0.00969631876796484,
+            0.0662323608994484,
+            0.01054823212325573,
+            0.03726732358336449,
+            0.0018974426202476025,
+            -0.009773765690624714,
+            0.007969259284436703,
+            0.0063041565008461475,
+            -0.012329504825174809,
+            0.010494019836187363,
+            0.024178840219974518,
+            0.0012730291346088052,
+            -0.002412462839856744,
+            -0.019408127292990685,
+            -0.03438630700111389,
+            0.003727119415998459,
+            -0.030932186171412468,
+            0.022691864520311356,
+            0.009208405390381813,
+            0.017456471920013428,
+            0.008240322582423687,
+            -0.015907539054751396,
+            0.03200095146894455,
+            0.05461536720395088,
+            0.0019768255297094584,
+            0.013375034555792809,
+            0.007938280701637268,
+            0.007036027032881975,
+            -0.011338187381625175,
+            0.04702559858560562,
+            0.000662652775645256,
+            0.005169562995433807,
+            0.022536970674991608,
+            0.02145271934568882,
+            0.022939693182706833,
+            0.002304037567228079,
+            -0.011694442480802536,
+            0.010478530079126358,
+            -0.0006302220281213522,
+            0.004445436876267195,
+            -0.010695381090044975,
+            -0.027865299955010414,
+            0.007492962293326855,
+            0.01302652433514595,
+            0.022598929703235626,
+            0.021592123433947563,
+            -0.03178409859538078,
+            -0.01855621486902237,
+            0.03364282101392746,
+            -0.02997184917330742,
+            0.03612111136317253,
+            -0.0014424435794353485,
+            -0.023807097226381302,
+            -0.0058356039226055145,
+            0.016929835081100464,
+            0.02656419761478901,
+            -0.029181893914937973,
+            0.01804506592452526,
+            0.029693040996789932,
+            -0.0074426219798624516,
+            0.0005939188995398581,
+            0.012112653814256191,
+            0.02447313815355301,
+            -0.009293596260249615,
+            -0.0024027819745242596,
+            -0.0013320822035893798,
+            -0.011926782317459583,
+            -0.018370341509580612,
+            -0.0044725434854626656,
+            -0.022629907354712486,
+            0.005564541090279818,
+            -0.007365175057202578,
+            -0.003481226274743676,
+            -0.008581086993217468,
+            0.01154729351401329,
+            0.001979729626327753,
+            -0.019857317209243774,
+            -0.022335609421133995,
+            0.01652711257338524,
+            0.03887821361422539,
+            -0.018215449526906013,
+            -0.02106548473238945,
+            0.024442158639431,
+            -0.01759587600827217,
+            -0.01601596549153328,
+            0.036678727716207504,
+            0.013739033602178097,
+            0.007736918982118368,
+            -0.03358086198568344,
+            -0.030049296095967293,
+            -0.03274443745613098,
+            0.006296411622315645,
+            0.03785591572523117,
+            -0.0028752065263688564,
+            0.02569679357111454,
+            0.03701949119567871,
+            0.0057891360484063625,
+            -0.028934063389897346,
+            -0.014823286794126034,
+            -0.045104920864105225,
+            0.060160547494888306,
+            0.0011481463443487883,
+            0.05161044001579285,
+            0.016294771805405617,
+            -0.01456771232187748,
+            0.00419760774821043,
+            -0.013251119293272495,
+            0.026192452758550644,
+            0.02280029095709324,
+            0.025929134339094162,
+            0.006582964211702347,
+            0.022505993023514748,
+            -0.0037522895727306604,
+            -0.012197845615446568,
+            0.007307090330868959,
+            0.014536733739078045,
+            0.0002497654058970511,
+            -0.011392400600016117,
+            0.015783624723553658,
+            -0.01307299267500639,
+            -0.011361422017216682,
+            0.023295948281884193,
+            0.01368482131510973,
+            -0.012344993650913239,
+            0.014064309187233448,
+            -0.008565598167479038,
+            0.01239920686930418,
+            -0.023915521800518036,
+            0.022691864520311356,
+            -0.03262052312493324,
+            0.009587894193828106,
+            0.0015053689712658525,
+            -0.0073148347437381744,
+            0.024550583213567734,
+            -0.0027454833034425974,
+            -0.05071205645799637,
+            0.03550153970718384,
+            -0.008015726692974567,
+            0.0037213109899312258,
+            0.006381602957844734,
+            0.011392400600016117,
+            -0.0012623801594600081,
+            0.03807276859879494,
+            0.01239920686930418,
+            0.015969496220350266,
+            0.02199484594166279,
+            0.011911292560398579,
+            -0.008433938957750797,
+            -0.004096927121281624,
+            -0.0016137943603098392,
+            -0.01784370467066765,
+            0.025944624096155167,
+            0.0018674320308491588,
+            0.03962169960141182,
+            -0.018168980255723,
+            -0.02300165221095085,
+            0.028066661208868027,
+            0.007597515359520912,
+            -0.02351279929280281,
+            -0.0017377089243382215,
+            0.01202746294438839,
+            -0.009185170754790306,
+            0.027385130524635315,
+            0.01196550577878952,
+            0.019113829359412193,
+            -0.018060555681586266,
+            -0.004340883810073137,
+            -0.005072754807770252,
+            -0.015938518568873405,
+            0.02238207869231701,
+            0.020771188661456108,
+            0.011152315884828568,
+            -0.011880313977599144,
+            -0.01307299267500639,
+            0.020600805059075356,
+            0.035036858171224594,
+            -0.0002744515077210963,
+            0.005672966130077839,
+            0.017487451434135437,
+            0.002373739378526807,
+            -0.04470219835639,
+            0.016232814639806747,
+            -0.014025585725903511,
+            0.011214273050427437,
+            0.019593998789787292,
+            -0.021917399019002914,
+            -0.03196997195482254,
+            -0.011539548635482788,
+            -0.0414804182946682,
+            0.00493722315877676,
+            -0.0069159846752882,
+            0.013576395809650421,
+            -0.010176488198339939,
+            -0.0014831031439825892,
+            0.016666516661643982,
+            0.0038103745318949223,
+            0.02177799493074417,
+            -0.011678952723741531,
+            -0.012073930352926254,
+            -0.020213572308421135,
+            -0.020786678418517113,
+            0.004278926644474268,
+            -0.030405549332499504,
+            -0.004488032776862383,
+            0.021375272423028946,
+            -0.04783104360103607,
+            -0.03689557686448097,
+            0.02898053266108036,
+            -0.008503641001880169,
+            0.020755698904395103,
+            0.022955182939767838,
+            -0.015094350092113018,
+            -0.0008717587334103882,
+            -0.018695618957281113,
+            0.02258343994617462,
+            -0.007837600074708462,
+            0.026300877332687378,
+            -0.0027551641687750816,
+            0.0006292539183050394,
+            -0.02659517526626587,
+            0.020306508988142014,
+            0.005227648187428713,
+            0.012275291606783867,
+            -0.02390003204345703,
+            0.006815304048359394,
+            -0.03133491054177284,
+            -0.01784370467066765,
+            0.0180140882730484,
+            -0.011152315884828568,
+            0.027989214286208153,
+            -0.006060199346393347,
+            0.028438406065106392,
+            0.022722844034433365,
+            0.024751944467425346,
+            0.02676555886864662,
+            -0.02363671362400055,
+            0.008495896123349667,
+            0.03726732358336449,
+            0.00865078903734684,
+            -0.015876561403274536,
+            0.008170620538294315,
+            -0.04863648861646652,
+            -0.008062195032835007,
+            0.05030933395028114,
+            0.002127846470102668,
+            0.023141054436564445,
+            0.016294771805405617,
+            -0.002873270306736231,
+            -0.005688455421477556,
+            0.010269423946738243,
+            -0.04296739399433136,
+            -0.023172033950686455,
+            0.022629907354712486,
+            -0.021561143919825554,
+            6.54212328754511e-07,
+            0.03122648410499096,
+            0.012120398692786694,
+            -0.0009109660750254989,
+            -0.05108380317687988,
+            0.02100352756679058,
+            0.01178737822920084,
+            0.014443797990679741,
+            -0.010300402529537678,
+            0.026068538427352905,
+            0.01733255758881569,
+            0.038413532078266144,
+            0.012755461037158966,
+            -0.002679653698578477,
+            -0.03782493621110916,
+            0.009177426807582378,
+            -0.021685058251023293,
+            0.0011471783509477973,
+            0.0074503663927316666,
+            -0.02273833192884922,
+            -0.024782923981547356,
+            0.0043912241235375404,
+            0.004588713403791189,
+            0.011880313977599144,
+            0.022087780758738518,
+            0.009417511522769928,
+            -0.021204888820648193,
+            0.015404135920107365,
+            -0.05446047708392143,
+            -0.015450604259967804,
+            -0.023884542286396027,
+            0.02566581591963768,
+            0.020012211054563522,
+            0.010184233076870441,
+            0.005545179359614849,
+            0.010904486291110516,
+            -0.020988039672374725,
+            -0.009239383973181248,
+            0.020631784573197365,
+            0.011074868962168694,
+            -0.012840652838349342,
+            0.04113965481519699,
+            -0.00011126903700642288,
+            0.031210994347929955,
+            -0.02280029095709324,
+            0.002941036131232977,
+            -0.001241082325577736,
+            -0.03649285435676575,
+            0.009130958467721939,
+            0.01981084980070591,
+            0.02151467651128769,
+            -0.01861817203462124,
+            0.0024143988266587257,
+            0.01048627495765686,
+            0.004766840487718582,
+            0.038351576775312424,
+            0.027121812105178833,
+            -0.044206541031599045,
+            0.011663463898003101,
+            0.012995545752346516,
+            -0.009363298304378986,
+            0.048419639468193054,
+            -0.017518429085612297,
+            0.017673322930932045,
+            0.008240322582423687,
+            -0.01798310875892639,
+            0.02318752370774746,
+            -0.01469937153160572,
+            -0.013282098807394505,
+            0.00650551775470376,
+            0.009192915633320808,
+            -0.0001770381786627695,
+            -0.037577107548713684,
+            -0.017859194427728653,
+            -0.005719434004276991,
+            0.0038297362625598907,
+            0.01562873087823391,
+            0.013498948886990547,
+            0.009967382065951824,
+            0.005161818582564592,
+            -0.00945623405277729,
+            -0.0016089539276435971,
+            0.008124152198433876,
+            -0.0005532593932002783,
+            0.0019216446671634912,
+            -0.01926872320473194,
+            0.02698240801692009,
+            0.02572777308523655,
+            -0.0018354853382334113,
+            -0.04652994126081467,
+            -0.0036709706764668226,
+            0.0037077576853334904,
+            -0.017735280096530914,
+            -0.0035315665882080793,
+            -0.04519785940647125,
+            -0.01810702309012413,
+            0.0143121387809515,
+            -0.014614180661737919,
+            -0.0460033044219017,
+            0.009076746180653572,
+            -0.013886182568967342,
+            0.0063970922492444515,
+            -0.00932457484304905,
+            0.005762029904872179,
+            0.003651608945801854,
+            0.005680711008608341,
+            -0.005909178406000137,
+            0.05108380317687988,
+            -0.03466511517763138,
+            0.028051171451807022,
+            0.008565598167479038,
+            -0.012437930330634117,
+            -0.02261441759765148,
+            0.00887538492679596,
+            -0.044981006532907486,
+            -0.033209118992090225,
+            0.016743963584303856,
+            -0.01781272701919079,
+            0.011074868962168694,
+            0.06161654368042946,
+            0.007330324035137892,
+            -0.023342415690422058,
+            -0.01233724970370531,
+            -0.021049996837973595,
+            -0.04191412031650543,
+            0.003442503046244383,
+            0.005463860463351011,
+            -0.004948840010911226,
+            -0.03807276859879494,
+            -0.021716037765145302,
+            0.04615819454193115,
+            -0.012004229240119457,
+            0.012422440573573112,
+            0.01080380566418171,
+            -0.0019410063978284597,
+            -0.04860550910234451,
+            -0.021019017323851585,
+            0.0395597442984581,
+            0.021886419504880905,
+            -0.008426194079220295,
+            0.0028364830650389194,
+            -0.018277406692504883,
+            -0.003289545886218548,
+            0.006296411622315645,
+            -0.009107724763453007,
+            -0.03562545403838158,
+            0.02067825198173523,
+            0.0054832217283546925,
+            -0.023373395204544067,
+            -0.025805220007896423,
+            -0.03296129032969475,
+            0.009944148361682892,
+            0.02453509531915188,
+            -0.01968693546950817,
+            0.005823987070471048,
+            0.00027203132049180567,
+            -0.000881923595443368,
+            0.00042740863864310086,
+            -0.009959637187421322,
+            -0.019764382392168045,
+            -0.020151615142822266,
+            0.03268248215317726,
+            -0.0060059865936636925,
+            0.0074387495405972,
+            -0.03630698472261429,
+            -0.010974188335239887,
+            0.011322698555886745,
+            -0.002395037328824401,
+            -0.013452480547130108,
+            -0.009092235006392002,
+            -0.01739451475441456,
+            -0.009882191196084023,
+            -0.007876322604715824,
+            0.022660886868834496,
+            0.006168624851852655,
+            0.005409647710621357,
+            0.0023795480374246836,
+            -0.008581086993217468,
+            0.02898053266108036,
+            0.017874684184789658,
+            -0.014188224449753761,
+            -0.00920066051185131,
+            0.03451022133231163,
+            0.02665713243186474,
+            -0.009115469641983509,
+            0.01798310875892639,
+            0.023264970630407333,
+            -0.0010484338272362947,
+            -0.01196550577878952,
+            0.001922612776979804,
+            -0.0009550138493068516,
+            -0.0032972905319184065,
+            0.00845717266201973,
+            -0.015365412458777428,
+            0.0315207801759243,
+            0.04287445917725563,
+            -0.002145271748304367,
+            0.009587894193828106,
+            -0.003727119415998459,
+            -0.008472662419080734,
+            0.017069239169359207,
+            0.009316830895841122,
+            0.04625113308429718,
+            0.01098193321377039,
+            0.005680711008608341,
+            -0.033983584493398666,
+            -0.001229465357027948,
+            -0.01184933539479971,
+            0.03308520466089249,
+            0.02061629481613636,
+            0.00889087375253439,
+            -0.008379725739359856,
+            0.01344473659992218,
+            0.008341003209352493,
+            0.00661007035523653,
+            0.020569827407598495,
+            -0.016542602330446243,
+            -0.017286090180277824,
+            0.030467506498098373,
+            -0.00672236829996109,
+            0.01289486512541771,
+            0.02029101923108101,
+            -0.002089123008772731,
+            0.010842529125511646,
+            0.07100307941436768,
+            -0.006184114143252373,
+            0.015713922679424286,
+            -0.019082851707935333,
+            0.0015789433382451534,
+            -0.007640110794454813,
+            -0.03370477631688118,
+            0.018246427178382874,
+            0.0039962464943528175,
+            0.00865078903734684,
+            0.01198099460452795,
+            0.009053511545062065,
+            -0.007946024648845196,
+            -0.00963436160236597,
+            0.015195030719041824,
+            0.0019080914789810777,
+            0.00033568276558071375,
+            0.001336922636255622,
+            -0.0065210070461034775,
+            -0.004011735785752535,
+            -0.021948376670479774,
+            0.02763296104967594,
+            0.007140580099076033,
+            0.030606910586357117,
+            -0.055296897888183594,
+            0.030095763504505157,
+            -0.00789568480104208,
+            0.02554190158843994,
+            -0.016201836988329887,
+            -0.011090358719229698,
+            0.001609921921044588,
+            0.018153492361307144,
+            0.004491904750466347,
+            0.006462921854108572,
+            0.011888058856129646,
+            0.023233991116285324,
+            -0.03444826602935791,
+            0.017053749412298203,
+            0.023110076785087585,
+            0.0048675211146473885,
+            0.0003535923024173826,
+            0.005165690556168556,
+            0.015458349138498306,
+            0.007198664825409651,
+            -0.023048119619488716,
+            -0.0025692922063171864,
+            0.017611365765333176,
+            -0.02306360937654972,
+            0.0036980770528316498,
+            -0.006923729553818703,
+            0.004623564425855875,
+            -0.0014705180656164885,
+            -0.03509881719946861,
+            -0.00100099784322083,
+            -0.025743262842297554,
+            0.01923774555325508,
+            0.009556914679706097,
+            -0.025867177173495293,
+            0.003932353109121323,
+            0.025650326162576675,
+            -0.0003695656778290868,
+            0.00821708794683218,
+            0.0004683101433329284,
+            0.00869725737720728,
+            -0.00974278710782528,
+            -0.01493945624679327,
+            0.004801691509783268,
+            0.01531120017170906,
+            -0.01493945624679327,
+            -0.007431004662066698,
+            -0.009510447271168232,
+            0.030560443177819252,
+            0.0006674932083114982,
+            0.004189862869679928,
+            -0.005045648664236069,
+            -0.01858719252049923,
+            -0.0032237162813544273,
+            -0.021142931655049324,
+            0.004367990419268608,
+            -0.0009046735358424485,
+            0.04197607934474945,
+            0.0038161829579621553,
+            -0.02029101923108101,
+            0.0032972905319184065,
+            -0.03136589005589485,
+            0.0005000148667022586,
+            -0.007276111748069525,
+            -0.02004319056868553,
+            0.0015005286550149322,
+            0.0016525175888091326,
+            0.012631546705961227,
+            -0.03255856782197952,
+            -0.009301341138780117,
+            0.018416810780763626,
+            -0.00020680672605521977,
+            -0.010687636211514473,
+            -0.017828216776251793,
+            0.00035189816844649613,
+            -0.011338187381625175,
+            -0.027973726391792297,
+            -0.02850036323070526,
+            -0.0026719090528786182,
+            0.013491204008460045,
+            -0.018277406692504883,
+            -0.014993669465184212,
+            0.0031249718740582466,
+            0.021592123433947563,
+            -0.009781510569155216,
+            0.008302279748022556,
+            0.02241305634379387,
+            -0.008255811408162117,
+            -0.005514200776815414,
+            -0.000941460661124438,
+            -0.009130958467721939,
+            0.004267309792339802,
+            0.006300284061580896,
+            0.043339136987924576,
+            -0.032403673976659775,
+            -0.0018238683696836233,
+            0.014815541915595531,
+            0.0034153966698795557,
+            0.0025905899237841368,
+            0.002503462601453066,
+            -0.0007841471815481782,
+            0.01344473659992218,
+            0.01178737822920084,
+            0.015466093085706234,
+            0.00100099784322083,
+            -0.0033011629711836576,
+            0.006408709101378918,
+            -0.021173911169171333,
+            0.015721667557954788,
+            0.027958236634731293,
+            -0.0033999073784798384,
+            0.005525817628949881,
+            -0.013731288723647594,
+            -0.008759214542806149,
+            0.013134949840605259,
+            0.0026641644071787596,
+            -0.005068882368505001,
+            -0.009394276887178421,
+            0.017967619001865387,
+            0.019082851707935333,
+            0.006668155547231436,
+            -0.037453193217515945,
+            -0.02898053266108036,
+            -0.009959637187421322,
+            0.020569827407598495,
+            0.014916222542524338,
+            0.0010339126456528902,
+            -0.010540487244725227,
+            -0.009595638141036034,
+            -0.011903548613190651,
+            0.010989678092300892,
+            0.02721474878489971,
+            -0.032372694462537766,
+            0.0008214184199459851,
+            -0.029383255168795586,
+            0.03068435750901699,
+            0.0020000594668090343,
+            0.011446612887084484,
+            -0.0008185141487047076,
+            -0.015140817500650883,
+            -0.029181893914937973,
+            0.013824224472045898,
+            -0.004395096562802792,
+            0.001024231780320406,
+            -0.0031753121875226498,
+            0.0201051477342844,
+            0.012887120246887207,
+            -0.0014472840121015906,
+            0.01968693546950817,
+            0.008178364485502243,
+            0.016480645164847374,
+            -0.010153254494071007,
+            -0.04135650396347046,
+            0.019965743646025658,
+            0.007562664337456226,
+            0.014219203032553196,
+            0.004739734344184399,
+            0.004588713403791189,
+            -0.020771188661456108,
+            0.002724185585975647,
+            0.004333139397203922,
+            0.014405074529349804,
+            0.011028401553630829,
+            0.0025615475606173277,
+            0.011678952723741531,
+            -0.02228914201259613,
+            0.01929970271885395,
+            -0.01631026156246662,
+            0.01006806269288063,
+            0.004445436876267195,
+            -0.026873983442783356,
+            0.007547175046056509,
+            0.007822110317647457,
+            -0.0019187404541298747,
+            -0.017549408599734306,
+            0.02160761132836342,
+            -0.0034831624943763018,
+            -0.03509881719946861,
+            0.0360281765460968,
+            -0.0043447562493383884,
+            0.019346170127391815,
+            0.03209388628602028,
+            -0.01771979033946991,
+            -0.01993476413190365,
+            0.00906900130212307,
+            0.010408828034996986,
+            -0.002873270306736231,
+            0.017859194427728653,
+            -0.002919738180935383,
+            -0.02232012152671814,
+            0.017766257748007774,
+            -0.028918573632836342,
+            -0.014234691858291626,
+            0.020337486639618874,
+            -0.03624502569437027,
+            -0.0021026763133704662,
+            0.03175312280654907,
+            -0.024070415645837784,
+            0.005734923295676708,
+            0.005196669604629278,
+            -0.0019351978553459048,
+            0.0012236569309607148,
+            -0.006451305001974106,
+            0.03801080957055092,
+            -0.010865762829780579,
+            0.0201051477342844,
+            0.01929970271885395,
+            -0.023388884961605072,
+            0.01977987214922905,
+            -0.001109423115849495,
+            0.01675945147871971,
+            -0.003940097521990538,
+            0.03466511517763138,
+            -0.006017603911459446,
+            -0.011330443434417248,
+            0.006431943271309137,
+            0.008488151244819164,
+            -0.009959637187421322,
+            0.008681767620146275,
+            -0.018122512847185135,
+            0.027152791619300842,
+            0.02091059274971485,
+            -0.0002691270783543587,
+            0.013800990767776966,
+            0.0004421719058882445,
+            -0.020833145827054977,
+            0.00789568480104208,
+            -0.027121812105178833,
+            -0.009967382065951824,
+            0.01646515540778637,
+            -0.005762029904872179,
+            0.007380664348602295,
+            -0.02064727433025837,
+            0.028004704043269157,
+            0.0077524082735180855,
+            -0.013591884635388851,
+            -0.0008824076503515244,
+            0.023760627955198288,
+            -0.005432881414890289,
+            0.009355553425848484,
+            0.02070923149585724,
+            0.015713922679424286,
+            0.013615119270980358,
+            0.016263794153928757,
+            0.03351890295743942,
+            0.005328328814357519,
+            0.005568413063883781,
+            0.003519949736073613,
+            0.016991792246699333,
+            -0.008557853288948536,
+            0.018029578030109406,
+            0.0008093173382803798,
+            -0.013576395809650421,
+            0.021499186754226685,
+            -0.0018684001406654716,
+            -0.013622863218188286,
+            -0.0040891822427511215,
+            -0.004337011836469173,
+            0.03271345794200897,
+            0.022041313350200653,
+            -0.004468671046197414,
+            -0.024442158639431,
+            -0.00394784240052104,
+            0.016682006418704987,
+            -0.003167567541822791,
+            0.011136826127767563,
+            0.0007952801533974707,
+            -0.01586107164621353,
+            -0.009595638141036034,
+            0.0063583687879145145,
+            0.010586955584585667,
+            -0.008921852335333824,
+            0.010346870869398117,
+            -0.0228312686085701,
+            -0.004011735785752535,
+            0.00046613195445388556,
+            -0.004844287410378456,
+            0.0053438181057572365,
+            -0.011314953677356243,
+            0.004948840010911226,
+            0.016062432900071144,
+            0.01026167906820774,
+            0.004832670092582703,
+            0.006989559158682823,
+            0.012066186405718327,
+            0.009161937050521374,
+            -0.01627928391098976,
+            0.00428667152300477,
+            -0.005072754807770252,
+            -0.02345084212720394,
+            0.005831731949001551,
+            0.00902253296226263,
+            -0.01122201792895794,
+            0.009386532939970493,
+            -0.010958699509501457,
+            0.03296129032969475,
+            0.01030814740806818,
+            -0.037391237914562225,
+            -0.0033379499800503254,
+            -0.0016244432190433145,
+            0.014885243959724903,
+            0.012298526242375374,
+            0.008550108410418034,
+            -0.017518429085612297,
+            -0.0047823297791182995,
+            -0.009835722856223583,
+            0.0012827098835259676,
+            -0.012306271120905876,
+            0.017456471920013428,
+            0.022567950189113617,
+            -0.036337960511446,
+            0.01968693546950817,
+            -0.010432061739265919,
+            -0.023652203381061554,
+            0.020476890727877617,
+            0.011500826105475426,
+            0.01942361705005169,
+            -0.009402021765708923,
+            0.018896980211138725,
+            -0.0001331114035565406,
+            0.025510922074317932,
+            -0.004228586331009865,
+            -0.022087780758738518,
+            -0.012499887496232986,
+            -0.03209388628602028,
+            -0.0051811798475682735,
+            0.006819176487624645,
+            -0.00045524101005867124,
+            0.013111716136336327,
+            0.014226946979761124,
+            -0.025712283328175545,
+            0.007295473478734493,
+            0.032589543610811234,
+            0.014637414366006851,
+            0.023311438038945198,
+            -0.011276230216026306,
+            -0.0010794125264510512,
+            -0.0026060794480144978,
+            -0.013770012184977531,
+            -0.03212486580014229,
+            -0.0017493258928880095,
+            -0.036585792899131775,
+            0.005680711008608341,
+            -0.016387708485126495,
+            -0.013096226379275322,
+            0.023264970630407333,
+            -0.034138478338718414,
+            0.0014608372002840042,
+            0.013708055019378662,
+            0.010540487244725227,
+            0.012391461990773678,
+            -0.00162637943867594,
+            0.015032391995191574,
+            -0.029243851080536842,
+            -0.010284913703799248,
+            -0.004491904750466347,
+            -0.01981084980070591,
+            -0.007338068913668394,
+            -0.018060555681586266,
+            -0.013127204962074757,
+            -0.003566417610272765,
+            -0.010060317814350128,
+            0.002247888594865799,
+            0.0032062907703220844,
+            -0.014714861288666725,
+            0.03206290677189827,
+            0.003157886676490307,
+            0.018958937376737595,
+            0.0033418224193155766,
+            0.004449309315532446,
+            0.018788553774356842,
+            -0.0026331855915486813,
+            -0.0058278595097362995,
+            0.030963165685534477,
+            -0.00032672801171429455,
+            0.008759214542806149,
+            0.019082851707935333,
+            0.004724245052784681,
+            0.0038529701996594667,
+            -0.0260530486702919,
+            0.012887120246887207,
+            -0.027447087690234184,
+            0.005080499220639467,
+            -0.0077175572514534,
+            -0.024302754551172256,
+            -0.028128618374466896,
+            -0.006571347359567881,
+            0.012058441527187824,
+            -0.02898053266108036,
+            -0.00613377382978797,
+            0.002315654419362545,
+            -0.00400399137288332,
+            -0.002673845272511244,
+            0.007241260726004839,
+            0.029150914400815964,
+            -0.033209118992090225,
+            0.008751469664275646,
+            0.005595519673079252,
+            -0.0005053392960689962,
+            0.013297587633132935,
+            0.008038961328566074,
+            0.003343758638948202,
+            0.020848635584115982,
+            0.006637176964432001,
+            0.002458930714055896,
+            -0.0025576751213520765,
+            -0.014242436736822128,
+            0.004000118933618069,
+            0.02154565416276455,
+            0.007562664337456226,
+            0.018385831266641617,
+            -0.008914108388125896,
+            0.002621568739414215,
+            -0.01455222349613905,
+            0.005552923772484064,
+            -0.002867461647838354,
+            0.004309905227273703,
+            -0.012236569076776505,
+            0.008093173615634441,
+            0.0028771425131708384,
+            -0.0043447562493383884,
+            0.018060555681586266,
+            -0.002319526858627796,
+            0.016480645164847374,
+            0.004015608225017786,
+            0.015620986931025982,
+            -0.0035412474535405636,
+            0.01178737822920084,
+            0.0005731051205657423,
+            -0.017905661836266518,
+            -0.017022771760821342,
+            -0.039249956607818604,
+            0.007702067960053682,
+            0.03531566634774208,
+            -0.0009564659558236599,
+            -0.03444826602935791,
+            0.010145509615540504,
+            -0.011942271143198013,
+            -0.009975126944482327,
+            -0.022862248122692108,
+            0.013862947933375835,
+            0.012430185452103615,
+            0.007415515370666981,
+            0.012252057902514935,
+            -0.006997303571552038,
+            0.01026167906820774,
+            -0.005216030869632959,
+            0.008356492035090923,
+            -0.0010823167394846678,
+            -0.037422213703393936,
+            0.012344993650913239,
+            0.02216522768139839,
+            -0.002635121811181307,
+            0.0016089539276435971,
+            -0.006145390681922436,
+            0.022103270515799522,
+            -0.013971373438835144,
+            -0.00650551775470376,
+            -0.00046443779137916863,
+            0.00797700323164463,
+            -0.01795213110744953,
+            0.004778457339853048,
+            0.0755879208445549,
+            -0.007775642443448305,
+            0.006122156511992216,
+            0.009270362555980682,
+            -0.015148562379181385,
+            -0.01637221872806549,
+            -0.004561606794595718,
+            -0.020260039716959,
+            0.008844406343996525,
+            -0.0018819533288478851,
+            -0.0110981035977602,
+            0.012468908913433552,
+            0.0053050946444272995,
+            0.002031038049608469,
+            -0.008379725739359856,
+            0.010811550542712212,
+            0.009525936096906662,
+            0.010432061739265919,
+            0.02360573597252369,
+            0.011477591469883919,
+            -0.00858883187174797,
+            -0.011795123107731342,
+            0.01129171997308731,
+            0.0028577810153365135,
+            0.016697494313120842,
+            0.007907302118837833,
+            0.0010648912284523249,
+            0.019501063972711563,
+            -0.002309845993295312,
+            0.02180897258222103,
+            -2.2689444449497387e-05,
+            -0.009301341138780117,
+            -0.004236331209540367,
+            -0.015349923633038998,
+            0.024271776899695396,
+            -0.0020155487582087517,
+            0.0015450604259967804,
+            -0.020213572308421135,
+            0.0013979118084535003,
+            0.013243375346064568,
+            0.011957760900259018,
+            -0.0020407189149409533,
+            -0.0051347119733691216,
+            0.0007154133054427803,
+            -0.016356728971004486,
+            0.032806396484375,
+            0.007578153628855944,
+            0.0007681738352403045,
+            0.0008838597568683326,
+            0.0009288755827583373,
+            -0.0032972905319184065,
+            0.0024608669336885214,
+            -0.004766840487718582,
+            0.009549170732498169,
+            -0.010943209752440453,
+            -0.010796061716973782,
+            0.010075807571411133,
+            -0.02402394637465477,
+            0.003566417610272765,
+            -0.00558777479454875,
+            -0.008062195032835007,
+            0.0076207490637898445,
+            0.00015537731815129519,
+            -0.007392281666398048,
+            -0.0030939930584281683,
+            0.0023408245760947466,
+            0.013429246842861176,
+            0.004406713880598545,
+            0.010850274004042149,
+            0.012956822291016579,
+            0.005359307397156954,
+            -0.006629432085901499,
+            0.015512561425566673,
+            -0.006946963258087635,
+            -0.004228586331009865,
+            -0.0012468908680602908,
+            -0.019996721297502518,
+            -0.0009419447160325944,
+            -0.004708755761384964,
+            0.0015092412941157818,
+            0.0001705036120256409,
+            -8.749050175538287e-05,
+            -0.0018490385264158249,
+            -0.011988739483058453,
+            -0.001831613015383482,
+            -0.011663463898003101,
+            -0.014498010277748108,
+            0.010253935120999813,
+            -0.0041201612912118435,
+            -0.022939693182706833,
+            -0.015349923633038998,
+            -0.020662762224674225,
+            0.02145271934568882,
+            -0.004387352149933577,
+            -0.005800753366202116,
+            0.006733985152095556,
+            -0.010215211659669876,
+            -0.015706177800893784,
+            -0.008069939911365509,
+            -0.0205853171646595,
+            0.010703125037252903,
+            0.003214035416021943,
+            0.0006791101768612862,
+            0.00882891658693552,
+            0.014397329650819302,
+            -0.01589204929769039,
+            0.0111755495890975,
+            0.012182355858385563,
+            0.011361422017216682,
+            -0.006854027509689331,
+            0.0008911203476600349,
+            -0.014103032648563385,
+            0.02596011385321617,
+            0.021623101085424423,
+            0.005169562995433807,
+            -0.04132552444934845,
+            -0.005862710531800985,
+            -0.013344055972993374,
+            -0.010524998418986797,
+            -0.012120398692786694,
+            0.011771888472139835,
+            -0.0038820127956569195,
+            0.0013117524795234203,
+            0.006288666743785143,
+            0.0015063370810821652,
+            0.004367990419268608,
+            -0.0022382077295333147,
+            -0.007709812838584185,
+            -0.03212486580014229,
+            -0.02817508764564991,
+            -0.00863530021160841,
+            0.020461400970816612,
+            0.002838419284671545,
+            -0.0009453330421820283,
+            -0.0036380558740347624,
+            0.0027048238553106785,
+            -0.005882072262465954,
+            -0.005018542055040598,
+            -0.014753584749996662,
+            -0.012739972211420536,
+            0.0101300198584795,
+            0.00802347157150507,
+            0.014195968396961689,
+            -0.011950016021728516,
+            0.0028190575540065765,
+            0.018788553774356842,
+            0.007690451107919216,
+            -0.023357905447483063,
+            0.01135367713868618,
+            -0.010912231169641018,
+            -0.009270362555980682,
+            -0.009433000348508358,
+            -0.009471723809838295,
+            0.0016980175860226154,
+            -0.008317768573760986,
+            0.003945906180888414,
+            -0.03615209087729454,
+            -0.011330443434417248,
+            -0.0013901671627536416,
+            -0.010346870869398117,
+            0.0009864765452221036,
+            0.02524760365486145,
+            0.006815304048359394,
+            0.01432762760668993,
+            -0.0006733016925863922,
+            0.017502939328551292,
+            -0.0018984107300639153,
+            0.018726596608757973,
+            0.012763205915689468,
+            -0.00797700323164463,
+            0.011717676185071468,
+            0.014738094992935658,
+            -0.0033863543067127466,
+            0.015620986931025982,
+            -0.010168743319809437,
+            0.005285732913762331,
+            -0.009556914679706097,
+            -0.003214035416021943,
+            0.002495717955753207,
+            0.030901208519935608,
+            -0.003618694143369794,
+            0.00858883187174797,
+            0.01098193321377039,
+            -0.015241498127579689,
+            0.0026951429899781942,
+            0.014962689951062202,
+            0.01141563430428505,
+            0.007760153152048588,
+            0.0024434414226561785,
+            -0.024225307628512383,
+            0.004193735308945179,
+            0.005037903785705566,
+            0.0031520782504230738,
+            -0.014366351068019867,
+            0.0021859314292669296,
+            0.015675200149416924,
+            0.008550108410418034,
+            -0.025325050577521324,
+            0.016418687999248505,
+            0.0009138703462667763,
+            -0.003909118939191103,
+            0.029119934886693954,
+            -0.011268485337495804,
+            -0.005723306443542242,
+            0.009463978931307793,
+            0.0009980935137718916,
+            0.0030029932968318462,
+            0.008937342092394829,
+            0.0228312686085701,
+            -0.003988501615822315,
+            -0.032341714948415756,
+            -0.022459525614976883,
+            0.015706177800893784,
+            -0.0047823297791182995,
+            -0.022939693182706833,
+            -0.026703599840402603,
+            -0.003113354789093137,
+            -0.02228914201259613,
+            -0.0022323993034660816,
+            -0.020058678463101387,
+            -0.014056564308702946,
+            0.001268188701942563,
+            0.029987338930368423,
+            0.016201836988329887,
+            0.008224832825362682,
+            -0.011051635257899761,
+            -0.029956359416246414,
+            -0.024271776899695396,
+            0.01810702309012413,
+            -0.007055388763546944,
+            -0.005541306920349598,
+            0.00591692328453064,
+            -0.009951893240213394,
+            -0.008480406366288662,
+            0.011802867986261845,
+            0.027137301862239838,
+            -0.002747419523075223,
+            -0.029987338930368423,
+            -0.02688947319984436,
+            -0.014335372485220432,
+            -0.01840132102370262,
+            -0.017704300582408905,
+            -0.008627555333077908,
+            0.0045035220682621,
+            0.00030300996149890125,
+            0.016186347231268883,
+            0.014962689951062202,
+            -0.000480653194244951,
+            0.0028597170021384954,
+            -2.7711375878425315e-05,
+            -0.001763847190886736,
+            0.006888878531754017,
+            -0.004933350719511509,
+            0.026997897773981094,
+            0.014614180661737919,
+            -0.006594581063836813,
+            0.018711106851696968,
+            0.028314489871263504,
+            0.020430423319339752,
+            0.003157886676490307,
+            -0.010695381090044975,
+            -0.014606435783207417,
+            0.009727297350764275,
+            -0.013421501964330673,
+            0.023264970630407333,
+            0.022552460432052612,
+            -0.01480779703706503,
+            -0.014885243959724903,
+            -0.017905661836266518,
+            0.015210519544780254,
+            -0.007562664337456226,
+            -0.008224832825362682,
+            0.012368228286504745,
+            0.022335609421133995,
+            -0.007024410180747509,
+            0.002371803391724825,
+            -0.0010232636705040932,
+            -0.021731525659561157,
+            -0.006122156511992216,
+            -0.004596457816660404,
+            -0.008666278794407845,
+            0.008681767620146275,
+            0.008131897076964378,
+            -0.001928421319462359,
+            -0.006319645792245865,
+            0.005471604876220226,
+            -0.013367289677262306,
+            0.01301103550940752,
+            -0.00789568480104208,
+            0.03379771113395691,
+            -0.009138703346252441,
+            -0.0006733016925863922,
+            -0.022598929703235626,
+            -0.011446612887084484,
+            0.00845717266201973,
+            0.0017967619933187962,
+            0.01682141050696373,
+            -0.0011626676423475146,
+            -0.005688455421477556,
+            0.009913169778883457,
+            -0.0010687635513022542,
+            -0.001450188341550529,
+            -0.006892750971019268,
+            -0.010401083156466484,
+            0.0002884887217078358,
+            0.013258864171802998,
+            -0.006025348324328661,
+            0.0018054747488349676,
+            0.006095050368458033,
+            -0.0175339188426733,
+            0.029027000069618225,
+            0.023884542286396027,
+            0.017255110666155815,
+            0.014660648070275784,
+            0.0006394187803380191,
+            0.011647974140942097,
+            -0.003572226269170642,
+            0.0038820127956569195,
+            -0.020337486639618874,
+            0.021080974489450455,
+            -0.010811550542712212,
+            0.01846327818930149,
+            0.011655719019472599,
+            -0.015233753249049187,
+            -0.005022414494305849,
+            0.021406250074505806,
+            0.009967382065951824,
+            0.00889087375253439,
+            0.0017541663255542517,
+            -0.02470547705888748,
+            0.017766257748007774,
+            0.010099041275680065,
+            -0.020399443805217743,
+            0.019547531381249428,
+            -0.009471723809838295,
+            0.01233724970370531,
+            -0.0005300253978930414,
+            0.004418330732733011,
+            0.0058743273839354515,
+            0.021499186754226685,
+            0.011570527218282223,
+            -0.03884723410010338,
+            -0.013615119270980358,
+            -0.0017357728211209178,
+            0.006857899948954582,
+            -0.0036303112283349037,
+            -0.015233753249049187,
+            -0.041635312139987946,
+            0.01604694314301014,
+            0.009998360648751259,
+            -0.01643417589366436,
+            0.004987563472241163,
+            -0.002427952131256461,
+            -0.0042053526267409325,
+            -0.008170620538294315,
+            0.003186929039657116,
+            -0.004170501604676247,
+            0.02324948087334633,
+            0.0018984107300639153,
+            -0.006946963258087635,
+            0.013258864171802998,
+            0.01344473659992218,
+            -0.021948376670479774,
+            0.0009874446550384164,
+            0.0021375271026045084,
+            0.000636998622212559,
+            0.023683181032538414,
+            -0.014707116410136223,
+            -0.0017657832941040397,
+            -0.01685238815844059,
+            -0.0013069120468571782,
+            -0.021592123433947563,
+            -0.001188805908896029,
+            -0.015187285840511322,
+            -0.008511385880410671,
+            0.025030752643942833,
+            -0.011005166918039322,
+            0.010997422970831394,
+            -0.006687517277896404,
+            0.006943091284483671,
+            -0.006904367823153734,
+            -0.01936165988445282,
+            -0.0008267428493127227,
+            0.009975126944482327,
+            -0.000512115890160203,
+            0.002941036131232977,
+            8.482827252009884e-05,
+            0.002464739140123129,
+            0.008867640048265457,
+            0.006234454456716776,
+            -0.0075278133153915405,
+            0.02489134855568409,
+            -0.012073930352926254,
+            0.0016399325104430318,
+            0.011911292560398579,
+            0.0004743606550619006,
+            -0.010424317792057991,
+            0.025588368996977806,
+            -0.02139076218008995,
+            -0.00800798274576664,
+            -0.00014799568452872336,
+            -0.0016544538084417582,
+            -0.006513262167572975,
+            0.006718495860695839,
+            0.02946070022881031,
+            0.0062654330395162106,
+            0.010447551496326923,
+            -0.0281441081315279,
+            -0.011493081226944923,
+            0.012538610957562923,
+            -0.022273652255535126,
+            -0.0007739823195151985,
+            -0.0004322490422055125,
+            -0.00619960343465209,
+            0.026378324255347252,
+            -0.02022906206548214,
+            -0.01093546487390995,
+            -0.018664639443159103,
+            0.005405775271356106,
+            -0.011121337302029133,
+            0.008294534869492054,
+            -0.0009777637897059321,
+            0.006540368776768446,
+            0.016805920749902725,
+            -0.019563021138310432,
+            0.016031453385949135,
+            0.005119222681969404,
+            -0.016217326745390892,
+            0.010222956538200378,
+            -0.017580386251211166,
+            -0.0008199662552215159,
+            0.020275529474020004,
+            -0.00400399137288332,
+            0.008751469664275646,
+            -0.003363120136782527,
+            0.005208286456763744,
+            -0.006513262167572975,
+            0.0001492057926952839,
+            -0.007249005138874054,
+            0.014428308233618736,
+            -0.016805920749902725,
+            -0.017828216776251793,
+            -0.008821171708405018,
+            0.0016709112096577883,
+            0.021297825500369072,
+            0.011136826127767563,
+            -0.010044828988611698,
+            -0.011059380136430264,
+            -0.011694442480802536,
+            0.041604332625865936,
+            0.005762029904872179,
+            0.027230238541960716,
+            -0.02261441759765148,
+            0.00545998802408576,
+            0.025170156732201576,
+            -0.014242436736822128,
+            0.0012788375606760383,
+            -0.0022788674104958773,
+            0.0053050946444272995,
+            0.011562783271074295,
+            -0.01819995976984501,
+            0.0049139889888465405,
+            -0.00019216447253711522,
+            -0.00782985519617796,
+            -0.000656844291370362,
+            0.003442503046244383,
+            0.00020196630794089288,
+            0.016325751319527626,
+            -0.004027225077152252,
+            -0.016511622816324234,
+            -0.004429947584867477,
+            -0.0003216455806978047,
+            -0.027896279469132423,
+            0.013514438644051552,
+            0.016589069738984108,
+            0.01566745527088642,
+            0.027989214286208153,
+            -0.014513500034809113,
+            0.016031453385949135,
+            0.009533680975437164,
+            0.013901671394705772,
+            -0.010610189288854599,
+            0.008952830918133259,
+            -0.01595400646328926,
+            0.007446494419127703,
+            -0.03454120084643364,
+            -0.007887939922511578,
+            -0.00802347157150507,
+            0.01984182931482792,
+            0.028763681650161743,
+            0.002923610620200634,
+            0.006493900436908007,
+            0.000196883876924403,
+            0.0024202074855566025,
+            -0.01714668609201908,
+            -0.020182594656944275,
+            -0.0003337466041557491,
+            0.0029332914855331182,
+            0.0006950835813768208,
+            -0.004495777189731598,
+            0.00472037261351943,
+            -0.012391461990773678,
+            0.0026835259050130844,
+            -0.013134949840605259,
+            0.005971135571599007,
+            -0.0059982421807944775,
+            0.009603383019566536,
+            0.0019429425010457635,
+            0.020864123478531837,
+            0.02300165221095085,
+            0.004100799560546875,
+            0.0046971384435892105,
+            -0.024333734065294266,
+            0.00920066051185131,
+            -0.013847459107637405,
+            0.012329504825174809,
+            0.019408127292990685,
+            0.008248066529631615,
+            0.006571347359567881,
+            0.01233724970370531,
+            0.012918098829686642,
+            -0.0031772484071552753,
+            0.024287264794111252,
+            0.005014669615775347,
+            -0.0009070937521755695,
+            0.002052335999906063,
+            -0.014954946003854275,
+            -0.0070979841984808445,
+            -0.0015286030247807503,
+            0.0018848575418815017,
+            -0.004731989465653896,
+            0.01906736195087433,
+            0.015256987884640694,
+            -0.01050176378339529,
+            0.017874684184789658,
+            -0.03937387093901634,
+            0.017379024997353554,
+            -0.0027377386577427387,
+            -0.0060911779291927814,
+            -0.00545998802408576,
+            0.00665653869509697,
+            0.004515138920396566,
+            0.003669034456834197,
+            0.03773200139403343,
+            -0.010408828034996986,
+            0.003963331691920757,
+            -0.005847221240401268,
+            0.007121218368411064,
+            -0.020864123478531837,
+            0.008302279748022556,
+            0.005336073227226734,
+            0.02160761132836342,
+            -0.0056497324258089066,
+            -0.0014482521219179034,
+            0.02328045852482319,
+            0.008859895169734955,
+            0.03469609469175339,
+            -0.01566745527088642,
+            -0.004418330732733011,
+            0.01048627495765686,
+            0.0207402091473341,
+            -0.00580462533980608,
+            -0.008976065553724766,
+            0.007903429679572582,
+            0.0035586729645729065,
+            0.021824462339282036,
+            -0.00515407370403409,
+            -0.0006650729919783771,
+            0.0054367538541555405,
+            -0.0060911779291927814,
+            0.009735042229294777,
+            0.011252996511757374,
+            -0.009076746180653572,
+            0.007539430167526007,
+            0.021530164405703545,
+            -0.009975126944482327,
+            -0.008658533915877342,
+            -0.009649851359426975,
+            0.0010658593382686377,
+            -0.03068435750901699,
+            0.001649613375775516,
+            0.005359307397156954,
+            0.015187285840511322,
+            -0.0004603234410751611,
+            -0.009014789015054703,
+            0.021019017323851585,
+            0.017255110666155815,
+            -0.005936284549534321,
+            -0.0028132491279393435,
+            0.0023795480374246836,
+            -0.0017154429806396365,
+            -0.006160879973322153,
+            -0.0006248975405469537,
+            -0.002656419761478901,
+            0.014730350114405155,
+            -0.0015469965292140841,
+            -0.006617815233767033,
+            -0.00756653631106019,
+            0.0033302053343504667,
+            0.010834784246981144,
+            -0.005661349277943373,
+            0.013057502917945385,
+            0.016805920749902725,
+            -0.00031704717548564076,
+            0.019671445712447166,
+            0.01037010457366705,
+            -0.010447551496326923,
+            0.015938518568873405,
+            -0.007984748110175133,
+            0.006563602481037378,
+            0.0164031982421875,
+            -0.01277095079421997,
+            -0.01246116403490305,
+            0.007210282143205404,
+            0.00665653869509697,
+            0.028918573632836342,
+            0.014792308211326599,
+            -0.02109646424651146,
+            0.014908477663993835,
+            0.015768134966492653,
+            -0.013018779456615448,
+            0.003537375247105956,
+            -0.01784370467066765,
+            0.020260039716959,
+            0.019051872193813324,
+            -0.015535795129835606,
+            -0.017224133014678955,
+            -0.014947201125323772,
+            -0.005092116538435221,
+            0.006331262644380331,
+            0.0033263331279158592,
+            -0.00617636926472187,
+            -0.027989214286208153,
+            -0.0049139889888465405,
+            -0.018633661791682243,
+            0.013870692811906338,
+            -0.008674023672938347,
+            0.0007715621031820774,
+            -0.004511266481131315,
+            0.007984748110175133,
+            -0.0033882902935147285,
+            0.032465629279613495,
+            -0.0051811798475682735,
+            -0.02832997962832451,
+            0.004398969002068043,
+            0.016356728971004486,
+            0.012623801827430725,
+            -0.029243851080536842,
+            0.010463040322065353,
+            0.004805563949048519,
+            -0.012530866079032421,
+            0.005990497302263975,
+            -0.02258343994617462,
+            -0.007632365915924311,
+            -0.0009395244996994734,
+            -0.016743963584303856,
+            -0.027121812105178833,
+            -0.008433938957750797,
+            -0.006993431597948074,
+            -0.005401902832090855,
+            -0.009525936096906662,
+            -0.008720491081476212,
+            -0.011477591469883919,
+            0.02303262986242771,
+            -0.007899557240307331,
+            -0.001866463921032846,
+            -0.0014637414133176208,
+            0.00930908601731062,
+            -0.023776117712259293,
+            -0.015605497173964977,
+            0.006389347370713949,
+            -0.002594462363049388,
+            0.022815778851509094,
+            -0.012747716158628464,
+            0.005940156988799572,
+            -0.009316830895841122,
+            0.009851212613284588,
+            0.015241498127579689,
+            0.004956584889441729,
+            0.017440982162952423,
+            -0.009889936074614525,
+            0.0021026763133704662,
+            -0.008627555333077908,
+            -0.00018986526993103325,
+            -1.027075541060185e-05,
+            0.014428308233618736,
+            0.007624621503055096,
+            0.014141756109893322,
+            0.017487451434135437,
+            -0.013553161174058914,
+            0.005003052763640881,
+            -0.0003274540649726987,
+            -0.017255110666155815,
+            0.0027725896798074245,
+            -0.007984748110175133,
+            -0.0008896682411432266,
+            0.001758038648404181,
+            6.395397940650582e-05,
+            -0.02112744189798832,
+            -0.0009124181815423071,
+            0.0018809852190315723,
+            0.0010222955606877804,
+            -0.025201136246323586,
+            0.00889087375253439,
+            -0.00889087375253439,
+            -0.0014918158994987607,
+            -0.019113829359412193,
+            0.019795360043644905,
+            0.026300877332687378,
+            -0.004619691986590624,
+            0.00036206303047947586,
+            -0.013537672348320484,
+            -0.02049238048493862,
+            0.0001489637652412057,
+            -0.011314953677356243,
+            -0.011345932260155678,
+            -0.0022169100120663643,
+            -0.003214035416021943,
+            0.008805682882666588,
+            0.0004189379105810076,
+            -0.009433000348508358,
+            -0.004956584889441729,
+            -0.011764144524931908,
+            -0.017549408599734306,
+            -0.010401083156466484,
+            0.01933068037033081,
+            -0.0018016024259850383,
+            0.0020794421434402466,
+            -0.0032508226577192545,
+            0.0005271211848594248,
+            0.011748654767870903,
+            0.013475715182721615,
+            0.013344055972993374,
+            -0.023915521800518036,
+            -0.017890173941850662,
+            -0.013320821337401867,
+            0.0029487807769328356,
+            -0.007187047973275185,
+            0.004844287410378456,
+            0.026362836360931396,
+            -0.0009695350890979171,
+            -0.01091997604817152,
+            0.002327271504327655,
+            0.019051872193813324,
+            -0.031272951513528824,
+            -0.013390523381531239,
+            -0.025077221915125847,
+            0.02225816436111927,
+            -0.013351799920201302,
+            0.011516314931213856,
+            0.010191977955400944,
+            -0.025944624096155167,
+            0.006768836174160242,
+            -0.007686578668653965,
+            0.026037558913230896,
+            0.01634124107658863,
+            0.0005934348446317017,
+            -0.00944074522703886,
+            0.013622863218188286,
+            -0.0008722427301108837,
+            -0.013576395809650421,
+            0.002201420720666647,
+            0.014645159244537354,
+            -0.014691627584397793,
+            -0.009510447271168232,
+            -0.004507394507527351,
+            0.004313777666538954,
+            0.012763205915689468,
+            -0.0030339721124619246,
+            -0.004031097516417503,
+            -0.011624740436673164,
+            0.013344055972993374,
+            -0.003930416889488697,
+            0.012383717112243176,
+            0.007380664348602295,
+            -0.014056564308702946,
+            -0.019532041624188423,
+            0.033333033323287964,
+            0.00400399137288332,
+            0.00338054564781487,
+            -0.010997422970831394,
+            0.02740062028169632,
+            -0.009897680021822453,
+            0.010571465827524662,
+            -0.006408709101378918,
+            0.0031985461246222258,
+            0.012321759946644306,
+            -0.012104909867048264,
+            -0.01981084980070591,
+            0.021468207240104675,
+            -0.00364192808046937,
+            -0.0010193913476541638,
+            0.008310024626553059,
+            -0.020182594656944275,
+            -0.00663330452516675,
+            0.002180122770369053,
+            0.009541425853967667,
+            -0.011818356812000275,
+            0.006369986105710268,
+            0.017905661836266518,
+            0.003912991378456354,
+            -0.0007672057254239917,
+            0.019996721297502518,
+            -0.01672847382724285,
+            0.011717676185071468,
+            -0.010850274004042149,
+            0.0016825281782075763,
+            0.0006065039779059589,
+            -0.0048597767017781734,
+            0.0058743273839354515,
+            -0.014536733739078045,
+            0.011988739483058453,
+            -6.244135147426277e-05,
+            -0.013979118317365646,
+            0.007601387333124876,
+            -0.013359544798731804,
+            -0.016836898401379585,
+            0.041542377322912216,
+            -0.005301222205162048,
+            -0.0038665232714265585,
+            -0.0056962003000080585,
+            0.01350669376552105,
+            0.02823704481124878,
+            -0.005336073227226734,
+            -0.01153180468827486,
+            -0.010780571959912777,
+            -0.016573579981923103,
+            0.015907539054751396,
+            -0.01810702309012413,
+            0.002495717955753207,
+            -0.010532742366194725,
+            -0.007903429679572582,
+            0.004302160814404488,
+            -0.003516077296808362,
+            0.014722606167197227,
+            0.00950270239263773,
+            0.023357905447483063,
+            0.00797700323164463,
+            -0.00027348342700861394,
+            0.010393339209258556,
+            0.011663463898003101,
+            0.003243078012019396,
+            -0.010509508661925793,
+            -0.00011949774489039555,
+            -0.02894955314695835,
+            -0.008449427783489227,
+            -0.004863648675382137,
+            -0.001485039247199893,
+            -0.00428667152300477,
+            -0.01239920686930418,
+            0.006617815233767033,
+            -0.0022052929271012545,
+            0.013909416273236275,
+            -0.009254872798919678,
+            -0.017766257748007774,
+            0.03389064967632294,
+            0.004495777189731598,
+            0.006466794293373823,
+            -0.008286789990961552,
+            -0.03705047070980072,
+            -0.0032566310837864876,
+            -0.008116407319903374,
+            -0.013529927469789982,
+            0.004538373090326786,
+            0.01283290795981884,
+            -0.014831030741333961,
+            -0.028097640722990036,
+            0.0019390701781958342,
+            -0.006683644838631153,
+            0.02978597767651081,
+            -0.007798876613378525,
+            0.00419760774821043,
+            0.0037251831963658333,
+            0.0011558911064639688,
+            0.01239920686930418,
+            -0.00845717266201973,
+            -0.0022536972537636757,
+            0.016387708485126495,
+            -0.010858018882572651,
+            0.011051635257899761,
+            -0.010966443456709385,
+            -0.02554190158843994,
+            -0.01456771232187748,
+            0.021344292908906937,
+            -0.0004823473282158375,
+            -0.00024903935263864696,
+            0.014823286794126034,
+            0.02354377694427967,
+            0.011260741390287876,
+            0.00969631876796484,
+            0.006408709101378918,
+            -0.001035848748870194,
+            0.005975008010864258,
+            0.006095050368458033,
+            -0.0024163350462913513,
+            -0.013398268260061741,
+            0.017564896494150162,
+            0.009773765690624714,
+            -0.01810702309012413,
+            0.0008514289511367679,
+            0.0036612898111343384,
+            0.0168833676725626,
+            0.006935346405953169,
+            0.009316830895841122,
+            -0.0143121387809515,
+            -0.023760627955198288,
+            -0.00014932679187040776,
+            0.002873270306736231,
+            0.015721667557954788,
+            0.010579210706055164,
+            -0.014319883659482002,
+            0.0005963391158729792,
+            0.008712747134268284,
+            0.020213572308421135,
+            -0.024209819734096527,
+            0.019702425226569176,
+            -0.004673904739320278,
+            -0.0021084847394376993,
+            0.007946024648845196,
+            0.011632485315203667,
+            -0.011872569099068642,
+            -0.00650551775470376,
+            0.006249943748116493,
+            0.003157886676490307,
+            0.03172214329242706,
+            -0.0017260919557884336,
+            0.009665340185165405,
+            -0.010997422970831394,
+            -0.0002405686245765537,
+            0.0006728176376782358,
+            -0.0006820144481025636,
+            -0.002884887158870697,
+            -0.011020656675100327,
+            0.0018974426202476025,
+            -0.020306508988142014,
+            0.0024473138619214296,
+            0.00463130883872509,
+            -0.005684582982212305,
+            -0.0032972905319184065,
+            -0.019222255796194077,
+            -0.0022672503255307674,
+            0.003293418325483799,
+            0.014335372485220432,
+            -0.031691163778305054,
+            0.004972074180841446,
+            0.010672146454453468,
+            -0.002309845993295312,
+            -0.010416572913527489,
+            0.011423379182815552,
+            -0.0026060794480144978,
+            0.011988739483058453,
+            -0.0038142469711601734,
+            -0.0004905760288238525,
+            0.025681305676698685,
+            -0.005308967083692551,
+            0.014707116410136223,
+            -0.01026167906820774,
+            -0.002993312431499362,
+            -0.028252532705664635,
+            0.005235392600297928,
+            -0.007403898518532515,
+            -0.0012081675231456757,
+            -0.004402841441333294,
+            -0.01130720879882574,
+            -0.0026990154292434454,
+            0.002122037811204791,
+            0.008000237867236137,
+            0.012910354882478714,
+            -0.0007725302129983902,
+            -0.010075807571411133,
+            0.004832670092582703,
+            0.0075355577282607555,
+            -0.015520306304097176,
+            -0.0070437719114124775,
+            -0.020306508988142014,
+            0.029662061482667923,
+            -0.013731288723647594,
+            -0.0011123273288831115,
+            -0.020864123478531837,
+            0.010160998441278934,
+            -0.0063119009137153625,
+            -0.002782270545139909,
+            0.004708755761384964,
+            -0.015109838917851448,
+            0.01344473659992218,
+            0.008751469664275646,
+            0.01450575515627861,
+            -0.02151467651128769,
+            -0.01762685552239418,
+            -0.0044725434854626656,
+            -0.002735802438110113,
+            -0.010648912750184536,
+            -0.0005334137240424752,
+            0.002013612538576126,
+            0.021251358091831207,
+            -0.00032503384863957763,
+            0.0042053526267409325,
+            -0.003214035416021943,
+            0.017224133014678955,
+            -0.010137764737010002,
+            -0.013057502917945385,
+            0.0024260159116238356,
+            0.005994369741529226,
+            0.016186347231268883,
+            0.03158273920416832,
+            -0.020693741738796234,
+            -0.00272999401204288,
+            0.0032740565948188305,
+            0.010656657628715038,
+            -0.004085310269147158,
+            6.304640555754304e-05,
+            -0.027803342789411545,
+            -0.04464024305343628,
+            -0.002422143705189228,
+            0.013057502917945385,
+            -0.015636475756764412,
+            0.012670270167291164,
+            0.008743725717067719,
+            0.010021595284342766,
+            -0.005196669604629278,
+            0.02620794251561165,
+            -0.004793947096914053,
+            0.014304393902420998,
+            0.011733165942132473,
+            0.011911292560398579,
+            0.0009874446550384164,
+            0.0007459079497493804,
+            -0.018354853615164757,
+            -0.008255811408162117,
+            -0.0015954007394611835,
+            0.01765783317387104,
+            -0.027478067204356194,
+            0.008526874706149101,
+            -0.011516314931213856,
+            0.013839714229106903,
+            0.0003666614356916398,
+            0.005661349277943373,
+            -0.01771979033946991,
+            0.012445674277842045,
+            -0.0009337160154245794,
+            -0.025325050577521324,
+            0.004805563949048519,
+            0.011276230216026306,
+            -0.010362359695136547,
+            0.006877261213958263,
+            0.019950253888964653,
+            0.0023079097736626863,
+            0.008495896123349667,
+            -0.014126266352832317,
+            0.010703125037252903,
+            0.009649851359426975,
+            -0.021080974489450455,
+            -0.01233724970370531,
+            -0.007059261202812195,
+            -0.006335135083645582,
+            -0.0035470561124384403,
+            -0.00013105422840453684,
+            0.010176488198339939,
+            -0.016000475734472275,
+            0.010099041275680065,
+            -0.006559730041772127,
+            0.006400964688509703,
+            0.00944074522703886,
+            0.001115231541916728,
+            -0.013576395809650421,
+            0.012716737575829029,
+            -0.0218709297478199,
+            0.008426194079220295,
+            -0.04287445917725563,
+            -0.005847221240401268,
+            -0.015078860335052013,
+            0.005204414017498493,
+            -0.003798757679760456,
+            -0.004453181754797697,
+            -0.006203475873917341,
+            -0.005061137955635786,
+            0.00920066051185131,
+            0.021747015416622162,
+            0.012135888449847698,
+            -0.016620049253106117,
+            -0.007551047019660473,
+            0.005425137002021074,
+            0.0004854935978073627,
+            -0.013336311094462872,
+            0.009316830895841122,
+            -0.01576039008796215,
+            0.0009259713697247207,
+            0.006339007057249546,
+            -0.01072635967284441,
+            0.013452480547130108,
+            0.004731989465653896,
+            -0.003401843598112464,
+            0.013390523381531239,
+            -0.0050417762249708176,
+            -0.0057891360484063625,
+            -0.007272239308804274,
+            -0.013739033602178097,
+            -0.003818119177594781,
+            -0.003072695340961218,
+            0.012747716158628464,
+            0.001529571134597063,
+            -0.0003243077953811735,
+            -0.00782985519617796,
+            0.009154192171990871,
+            0.002220782218500972,
+            -0.0013146566925570369,
+            0.005773646757006645,
+            -0.02300165221095085,
+            0.0001287550403503701,
+            0.004755223635584116,
+            -0.0009487213101238012,
+            -0.006246071308851242,
+            -0.012151377275586128,
+            -0.011903548613190651,
+            -0.038165703415870667,
+            0.009363298304378986,
+            0.0019197085639461875,
+            -0.0076207490637898445,
+            0.0064822835847735405,
+            0.017255110666155815,
+            0.01456771232187748,
+            0.016743963584303856,
+            -0.020693741738796234,
+            -0.02303262986242771,
+            0.005556796211749315,
+            -0.009402021765708923,
+            0.02261441759765148,
+            0.004046586807817221,
+            -0.005847221240401268,
+            0.0013243375578895211,
+            -0.013258864171802998,
+            0.008929597213864326,
+            0.005723306443542242,
+            -0.001592496526427567,
+            0.005425137002021074,
+            -0.013692565262317657,
+            -0.010904486291110516,
+            0.01048627495765686,
+            0.032496608793735504,
+            -0.0004261985304765403,
+            0.0024105266202241182,
+            0.023295948281884193,
+            0.015466093085706234,
+            0.016356728971004486,
+            -0.021111954003572464,
+            0.011167804710566998,
+            0.010060317814350128,
+            -0.0018645278178155422,
+            -0.014892988838255405,
+            0.00513083953410387,
+            -0.012716737575829029,
+            -0.010664401575922966,
+            0.014993669465184212,
+            0.007853088900446892,
+            0.017735280096530914,
+            -0.02369867078959942,
+            0.00789568480104208,
+            0.00042765066609717906,
+            0.00013710475468542427,
+            0.011524059809744358,
+            0.012956822291016579,
+            0.03370477631688118,
+            0.027152791619300842,
+            0.020662762224674225,
+            -0.009285852313041687,
+            0.014366351068019867,
+            -0.009192915633320808,
+            -0.011090358719229698,
+            -0.019470084458589554,
+            -0.015249243006110191,
+            -0.011136826127767563,
+            0.017998598515987396,
+            0.008790193125605583,
+            -0.009355553425848484,
+            -0.005947901867330074,
+            0.01586107164621353,
+            -0.003909118939191103,
+            -0.004635181277990341,
+            0.01646515540778637,
+            -0.026269899681210518,
+            0.016836898401379585,
+            -0.000745423894841224,
+            -0.014668392948806286,
+            -0.014691627584397793,
+            -0.004182118456810713,
+            -0.012670270167291164,
+            0.0003964299685321748,
+            -0.020817656069993973,
+            0.013181418180465698,
+            0.024597052484750748,
+            -0.0030920570716261864,
+            -0.004100799560546875,
+            -0.011299464851617813,
+            -0.0005111477803438902,
+            0.018385831266641617,
+            0.0034773540683090687,
+            0.009192915633320808,
+            -0.0023872926831245422,
+            0.013924905098974705,
+            0.00524313747882843,
+            0.0027958236169070005,
+            -0.0007565568666905165,
+            -0.016263794153928757,
+            0.012197845615446568,
+            0.0026835259050130844,
+            0.012135888449847698,
+            -0.022242674604058266,
+            0.007938280701637268,
+            -0.011764144524931908,
+            -0.0023698671720921993,
+            -0.02228914201259613,
+            0.0189744271337986,
+            -0.01610890030860901,
+            0.001046497724018991,
+            0.003841353114694357,
+            0.0006224773242138326,
+            0.009580149315297604,
+            0.01610890030860901,
+            -0.023388884961605072,
+            -0.0003460896550677717,
+            -0.014908477663993835,
+            -0.021917399019002914,
+            -0.01607792265713215,
+            0.004728117026388645,
+            -0.019501063972711563,
+            0.005444498732686043,
+            -0.0025557391345500946,
+            -0.015125328674912453,
+            0.018478767946362495,
+            0.014691627584397793,
+            -0.004379607271403074,
+            0.002412462839856744,
+            0.005425137002021074,
+            -0.0014405074762180448,
+            0.01971791312098503,
+            0.00013274837692733854,
+            -0.0010048701660707593,
+            -0.009425255469977856,
+            -0.0168833676725626,
+            -0.0065752193331718445,
+            -0.035005878657102585,
+            0.0260530486702919,
+            -0.004670032300055027,
+            -0.016511622816324234,
+            0.014319883659482002,
+            0.009115469641983509,
+            -0.014629669487476349,
+            -0.007059261202812195,
+            0.016294771805405617,
+            -0.024287264794111252,
+            0.02244403585791588,
+            -0.009611127898097038,
+            -0.008774704299867153,
+            0.00784921646118164,
+            0.05154848098754883,
+            0.009146448224782944,
+            0.003492843359708786,
+            0.011036145500838757,
+            0.005839476361870766,
+            -0.0013165927957743406,
+            -0.023683181032538414,
+            0.005568413063883781,
+            -0.007190920412540436,
+            -0.003601268632337451,
+            0.002536377403885126,
+            -0.004801691509783268,
+            0.031149037182331085,
+            -0.004379607271403074,
+            -0.005289605353027582,
+            0.004484160337597132,
+            -0.02196386642754078,
+            -0.0001332324172835797,
+            0.023203013464808464,
+            -2.3128106477088295e-05,
+            -0.023590246215462685,
+            -0.006571347359567881,
+            -0.015071115456521511,
+            -0.013770012184977531,
+            -0.018989915028214455,
+            0.005231520161032677,
+            -0.012755461037158966,
+            -0.0033940989524126053,
+            0.0016660707769915462,
+            -0.005940156988799572,
+            -0.002582845278084278,
+            0.009603383019566536,
+            0.05362405255436897,
+            0.014103032648563385,
+            -0.006335135083645582,
+            -0.005854965653270483,
+            0.009192915633320808,
+            -0.010524998418986797,
+            -0.019795360043644905,
+            -0.009820234030485153,
+            -0.029507169499993324,
+            -0.012747716158628464,
+            -0.0143121387809515,
+            -0.010191977955400944,
+            -0.010277168825268745,
+            0.00963436160236597,
+            -0.018122512847185135,
+            -0.007628493942320347,
+            -0.020337486639618874,
+            0.002941036131232977,
+            -0.011268485337495804,
+            -0.014714861288666725,
+            -0.008054450154304504,
+            -0.04807887226343155,
+            0.005796880926936865,
+            0.019593998789787292,
+            0.01263929158449173,
+            0.002515079453587532,
+            0.004383479710668325,
+            -0.01993476413190365,
+            -0.030792783945798874,
+            0.013839714229106903,
+            0.011283975094556808,
+            0.011733165942132473,
+            0.03283737599849701,
+            -0.011709931306540966,
+            -0.00813964195549488,
+            -0.004693266469985247,
+            -0.0017251238459721208,
+            0.012066186405718327,
+            0.00969631876796484,
+            -0.004917861428111792,
+            -0.012375972233712673,
+            0.016217326745390892,
+            -0.01981084980070591,
+            -0.023683181032538414,
+            -0.01672847382724285,
+            0.006157007534056902,
+            0.00813964195549488,
+            -0.008054450154304504,
+            0.002973950933665037,
+            0.0056962003000080585,
+            -0.007159941829741001,
+            -0.00022423222253564745,
+            0.004890755284577608,
+            -0.023048119619488716,
+            0.0008490087930113077,
+            -0.002257569460198283,
+            0.0061376458033919334,
+            0.021266845986247063,
+            -0.010168743319809437,
+            -0.0014075926737859845,
+            -0.0053902859799563885,
+            -0.0028984404634684324,
+            -0.006764963734894991,
+            0.02620794251561165,
+            0.009154192171990871,
+            -0.009882191196084023,
+            -0.01011453103274107,
+            -0.01129171997308731,
+            -0.0011278166202828288,
+            -0.017255110666155815,
+            -0.016186347231268883,
+            0.0015566773945465684,
+            -0.016712984070181847,
+            0.013971373438835144,
+            -0.00876695942133665,
+            0.00400399137288332,
+            -0.014110777527093887,
+            0.02019808255136013,
+            0.01929970271885395,
+            -0.009711808525025845,
+            0.023776117712259293,
+            -0.007709812838584185,
+            -0.038382552564144135,
+            -0.00800798274576664,
+            0.01843230053782463,
+            -0.006118284538388252,
+            0.010037084110081196,
+            -0.011314953677356243,
+            0.008209343999624252,
+            0.00015319912927225232,
+            0.0018122512847185135,
+            -0.0003112386621069163,
+            0.027555514127016068,
+            0.015535795129835606,
+            -0.005560668651014566,
+            -0.005494839046150446,
+            0.004255692940205336,
+            0.009541425853967667,
+            0.017580386251211166,
+            0.012151377275586128,
+            -0.021019017323851585,
+            -0.014257925562560558,
+            -0.016263794153928757,
+            0.011012911796569824,
+            0.015125328674912453,
+            -0.0040891822427511215,
+            0.01691434532403946,
+            0.014126266352832317,
+            -0.019098341464996338,
+            0.0019884423818439245,
+            -0.008193854242563248,
+            -0.00158184755127877,
+            -0.03029712475836277,
+            0.011043890379369259,
+            0.0074387495405972,
+            -0.0072218989953398705,
+            0.036616768687963486,
+            0.009858956560492516,
+            0.037577107548713684,
+            0.018680129200220108,
+            0.0008974128868430853,
+            0.009045767597854137,
+            -0.018370341509580612,
+            0.013065247796475887,
+            0.01500141341239214,
+            -0.012724482454359531,
+            -0.005734923295676708,
+            0.0013446672819554806,
+            0.0007551047019660473,
+            0.012631546705961227,
+            0.02004319056868553,
+            -0.01795213110744953,
+            0.009773765690624714,
+            -0.015032391995191574,
+            -0.005250881891697645,
+            0.006555858068168163,
+            -0.013932649977505207,
+            -0.008495896123349667,
+            -0.00963436160236597,
+            0.019408127292990685,
+            -0.012600568123161793,
+            0.016573579981923103,
+            -0.009595638141036034,
+            0.015520306304097176,
+            -0.029724018648266792,
+            -0.003669034456834197,
+            -0.006679772399365902,
+            -0.019593998789787292,
+            -0.012887120246887207,
+            0.009820234030485153,
+            0.010633422993123531,
+            -0.009301341138780117,
+            -0.011562783271074295,
+            0.020368466153740883,
+            -0.004185990896075964,
+            0.013529927469789982,
+            0.021189400926232338,
+            0.00737291993573308,
+            0.024132372811436653,
+            0.0005605200421996415,
+            0.014164989814162254,
+            0.005463860463351011,
+            0.010362359695136547,
+            -0.004677776712924242,
+            0.005161818582564592,
+            0.00932457484304905,
+            0.014203713275492191,
+            0.017766257748007774,
+            0.013196907006204128,
+            -0.00598275288939476,
+            0.007419387809932232,
+            0.02016710489988327,
+            0.0016254113288596272,
+            -0.030312614515423775,
+            -0.009719553403556347,
+            0.0001736498816171661,
+            0.00924712885171175,
+            0.009185170754790306,
+            3.817877222900279e-05,
+            -0.01093546487390995,
+            0.011144571006298065,
+            0.00969631876796484,
+            0.002286612056195736,
+            0.006280922330915928,
+            -0.01456771232187748,
+            -0.020957060158252716,
+            0.006106667220592499,
+            0.01798310875892639,
+            0.013917161151766777,
+            -0.014722606167197227,
+            0.022784801200032234,
+            0.006145390681922436,
+            -0.021406250074505806,
+            -0.01840132102370262,
+            -0.0009559819591231644,
+            0.0004542729293461889,
+            -0.002265314105898142,
+            0.03720536455512047,
+            -0.0011539548868313432,
+            -0.005955646280199289,
+            -0.0054754773154854774,
+            0.02775687538087368,
+            0.02049238048493862,
+            0.009177426807582378
+          ],
+          "embedding": [
+            0.0008215984936325879,
+            -0.019349418284688353,
+            -0.02322660633029795,
+            0.009747820845194852,
+            0.007353771220903641,
+            0.015252908783199839,
+            -0.012176175829980202,
+            0.051898051094966605,
+            -0.0068130353886146165,
+            0.008189212816890595,
+            0.008102287831931905,
+            0.01682801136011614,
+            0.02005557996049711,
+            -0.020779879580497183,
+            0.002332165268329781,
+            0.02337485065616377,
+            0.006074992895540845,
+            0.02576575146535689,
+            0.0007517286429767504,
+            -0.03763909942314134,
+            0.0011204729720314833,
+            0.00041212613589632806,
+            -0.06089176800314795,
+            0.012852035873395388,
+            -0.008901127092058543,
+            0.01702347055484216,
+            0.005803446395466281,
+            0.017889570275475296,
+            -0.05375047382413552,
+            0.021623778774252724,
+            0.014784317465883424,
+            -0.0018881880857885708,
+            0.0004735328554290247,
+            0.012393802850407656,
+            -0.019234442968055952,
+            -0.019566283611638122,
+            0.04407789373296535,
+            -0.0053565873209102995,
+            -0.026240866282480922,
+            0.048107902270426675,
+            -0.0054681314939387235,
+            -0.0009441397948041965,
+            -0.015205079626883548,
+            0.006885981841893229,
+            0.01403724256866746,
+            -0.026086548246792754,
+            0.0034778946051893067,
+            0.00044587533948775473,
+            0.032851328435891634,
+            -0.03882097333255334,
+            -0.011126344604254894,
+            -0.01648943336384904,
+            0.035810456555802166,
+            -0.019384589526165754,
+            -0.019430702768170285,
+            0.014297440272239852,
+            -0.007883453204663151,
+            0.050996900793086844,
+            0.06007149434153715,
+            0.03406832053483379,
+            0.025193439934671852,
+            0.00027785129663604897,
+            -0.008014391056168083,
+            -0.018806730908490114,
+            0.027193788903098558,
+            -0.006195006670328595,
+            -0.020466000344352826,
+            0.0068939884219589185,
+            0.017252276167086602,
+            -0.045073444648704136,
+            -0.049242269917908754,
+            0.023021140866008313,
+            -0.015676872472071106,
+            -0.024497517908898602,
+            0.005873633718973869,
+            -0.022828585938203683,
+            0.011748341004940436,
+            0.04624205727927451,
+            -0.015442165499674641,
+            0.018565551071491767,
+            0.015246505395507096,
+            0.004119269796224144,
+            0.004862735250819252,
+            -0.000145323321960046,
+            0.05155838952887077,
+            0.0027052423345323936,
+            -0.016218063670586664,
+            -0.07462470920944148,
+            0.043695974746295135,
+            -0.027608273720377708,
+            -0.03860912774695494,
+            0.002823110565150435,
+            0.026471469849508766,
+            -0.0026063577189074103,
+            -0.029230652014151735,
+            0.05684164470477843,
+            0.00022182868300572677,
+            0.015590845552956364,
+            -0.024939617549529242,
+            0.01035803286203917,
+            -0.010225298012384327,
+            0.019571847648759157,
+            0.01841613041441054,
+            0.0156981470538354,
+            0.021764297694793343,
+            -0.018831230007033416,
+            -0.014419529638789993,
+            -0.0029965475899652273,
+            0.028881185012095884,
+            0.09163526460163937,
+            0.022644302207811818,
+            -0.017933909936330133,
+            -0.04663928709016869,
+            0.004962951444105128,
+            -0.041333041174526026,
+            -0.027545628007617527,
+            -0.037142394579894954,
+            -0.013358143162788402,
+            0.02719686432396677,
+            -0.0034971800937504205,
+            0.04413635817371732,
+            0.04903857110328917,
+            0.03186067155630788,
+            0.007831051050269414,
+            -0.025257538623313574,
+            0.023517513773109414,
+            -0.0016527403209505964,
+            -0.0016101860707429347,
+            0.014448515185903624,
+            0.009905018312477684,
+            0.0024114582235163955,
+            0.00172043851384822,
+            -0.03818316221102173,
+            -0.02304537008110578,
+            0.01024969233570089,
+            -0.01901116356327339,
+            0.019791121460848986,
+            0.07985067340821365,
+            -0.051903191272236665,
+            -0.025924646976556942,
+            0.014766511532849252,
+            -0.002073086076894999,
+            0.030959269926800852,
+            0.006394417986617089,
+            -0.028177586693713442,
+            0.010725285336674526,
+            0.0009614052276924192,
+            0.07500253788978564,
+            -0.00036794182149960157,
+            0.004058458332826519,
+            -0.013218269166951234,
+            0.027613539268683245,
+            -0.012995170442480582,
+            0.023388707018488407,
+            0.026389423313105165,
+            -0.003068639956061266,
+            -0.02661967649632026,
+            0.026145274570635913,
+            -0.03300591322174331,
+            -0.004556072066733024,
+            0.020045722084112896,
+            0.0033437343914784865,
+            -0.010224220905483085,
+            -0.0025025498684079703,
+            0.03460103245395426,
+            0.04514362357278504,
+            0.03797031988058962,
+            -0.004586662475576956,
+            -0.06351723722596923,
+            0.06037983484369874,
+            0.019415753124363902,
+            0.028850618647994476,
+            -0.0034810237045849006,
+            0.02957938154808314,
+            0.02642652523231673,
+            0.013187653724133275,
+            -0.007856320804109569,
+            -0.007065882659482447,
+            -0.016672784625197085,
+            0.029020161154209707,
+            -0.013243058921970134,
+            -0.014900932834241732,
+            0.006001769901896033,
+            0.008648456186944271,
+            0.03484763413464497,
+            0.012279368984306564,
+            0.02379643508687133,
+            0.04933044959399941,
+            -0.021047723935647383,
+            -0.007264823701706426,
+            0.0007058774795716792,
+            -0.010512071832195085,
+            0.027489671668749157,
+            0.009731157570361245,
+            -0.03195919640758579,
+            -0.0033626031972691915,
+            -0.04282419484651207,
+            0.01873977910485268,
+            -0.014398285316695653,
+            0.012854631447494207,
+            -0.0017372959978092432,
+            -0.037331867825735776,
+            0.001088209292238142,
+            -0.012345684323522151,
+            0.037096063326414734,
+            0.005050219690024652,
+            0.0008906051515353004,
+            -0.003697709449630797,
+            0.006548479142737107,
+            -0.010727663712299397,
+            -0.007486226350610755,
+            -0.02704618481069485,
+            -0.019752552016312926,
+            -0.014632581296303486,
+            -0.0037501138119724717,
+            0.014588411795703245,
+            -0.007119137071719376,
+            0.016440258608998535,
+            -0.03686645352111013,
+            -0.023546402448713415,
+            0.005628703432152666,
+            0.024988145751166143,
+            0.011971199595808573,
+            -0.010868778329190568,
+            -0.03270046433624425,
+            -0.021950559285357267,
+            -0.011889914022657738,
+            -0.005123971523701009,
+            0.02619713302280345,
+            -0.021990810305904974,
+            0.01165257474485436,
+            -0.009117901804315863,
+            -0.007492082755249399,
+            0.010725902070291232,
+            0.01862811996181951,
+            -0.026694281868322685,
+            -0.01623776415010199,
+            -0.009403997538684658,
+            -0.02207329027097839,
+            0.025939540497196416,
+            0.014178803271471762,
+            -0.001141821318145744,
+            -0.006281187017652108,
+            -0.035993365198554964,
+            0.0026192776111449633,
+            -0.015254976771874396,
+            -0.0116872453009606,
+            -0.015780016491993135,
+            0.021421328283124968,
+            0.02889432250124136,
+            -0.01871009416296692,
+            -0.026540472640514536,
+            0.015703370682110186,
+            -0.0002849235245164438,
+            -0.02594930889995505,
+            0.04491828744809679,
+            0.044063522197410734,
+            -0.037441864491547405,
+            -0.007005339187980476,
+            -0.030381794393931814,
+            0.06901083845539985,
+            -0.017604082400603037,
+            -0.04060591381445593,
+            0.06044850935418443,
+            -0.01756490171206357,
+            0.025539717592705295,
+            0.03452018789549939,
+            0.0016217577522523927,
+            0.00018825054264059277,
+            -0.002563606092068712,
+            0.04770277039047993,
+            -0.0032010143255574255,
+            -0.03425328601475849,
+            -0.02105909767072799,
+            -0.00840881022563103,
+            -0.028699447677549002,
+            -0.015925160861589006,
+            -0.0055750742281166,
+            0.017513987082479483,
+            0.010399718005952185,
+            0.02267201252273823,
+            0.01258511878601025,
+            0.0015081978484714903,
+            0.01874981572091931,
+            0.011158889421057756,
+            -0.002607810686437211,
+            -0.003732023411827874,
+            -0.007787189972165309,
+            -0.05524012485122728,
+            -0.025110689689279583,
+            -0.03946383236703947,
+            0.004443572529325475,
+            -0.018745077598477586,
+            0.005926160888274727,
+            0.039430331582190524,
+            -0.0049500324516682,
+            0.004352795787899792,
+            0.015566139991780173,
+            0.013148156356408315,
+            -0.015279560337042242,
+            -0.03316846614871165,
+            0.0001592209771257332,
+            0.018362411161743347,
+            -0.012706582418607377,
+            -0.03424477956681259,
+            -0.001991022841226589,
+            0.0033467286047492175,
+            0.03357019284647071,
+            -0.024359110796680374,
+            0.00359117029951049,
+            0.005061818038334802,
+            0.011359582931760315,
+            -0.026793992746244076,
+            -0.018170136079817746,
+            -0.017421603453826397,
+            0.03996608926471779,
+            0.02587013058343799,
+            0.015763410145648683,
+            -0.04512536449983358,
+            -0.02293473890636517,
+            0.048575083783480195,
+            -0.009589860756342604,
+            -0.06486262416944208,
+            0.03447666009581266,
+            0.016105654870253654,
+            0.003526582643722421,
+            -0.01354754623151814,
+            0.05401575248587799,
+            -0.03563190295559527,
+            -0.017365166993914202,
+            -0.0046288709698227735,
+            -0.033657212157212645,
+            -0.007686252059880836,
+            -0.001385273642139782,
+            0.048229062789530236,
+            -0.029083855744728694,
+            -0.030465559177859865,
+            0.0024368661249980416,
+            -0.024144031154493057,
+            0.021693278579520123,
+            -0.016405088055634124,
+            -0.018439935250344958,
+            0.006948603359995389,
+            -0.033943037395199695,
+            -0.003007796423804453,
+            -0.016783195426536633,
+            -0.02270589491206155,
+            0.020874438753828814,
+            0.016724666903417947,
+            0.027033184180344852,
+            0.0019847647852695785,
+            0.017749524526371133,
+            0.0204661893602285,
+            -0.005604929968008821,
+            0.028462234633072944,
+            -0.020790459496727066,
+            -0.04183816355941877,
+            0.03946487892663695,
+            0.01192861758684541,
+            -0.0018024841275380242,
+            0.01077082591611702,
+            -0.01475239020295694,
+            -0.03952356245715099,
+            0.03683950018557526,
+            -0.056767672022551764,
+            -0.007507600090017693,
+            0.009530544352246083,
+            -0.00788856051420074,
+            -0.011204694027562521,
+            -0.010587701466318598,
+            -0.028604940377315504,
+            0.023749537635431462,
+            -0.013672419168235691,
+            0.01953449229707597,
+            -0.011449939491554354,
+            -0.012238180529977605,
+            0.05795536247491174,
+            -0.021518581774137322,
+            -0.040672955795925836,
+            0.026066659238628543,
+            -0.014061106927896723,
+            0.03328357230532424,
+            0.03957107406164427,
+            0.030670947051333917,
+            -0.004278577327978564,
+            0.025885946540364826,
+            0.0319106101668716,
+            -0.003462160036198642,
+            -0.04686868330851868,
+            0.00555303933159587,
+            0.02156143971867743,
+            -0.03739529489075782,
+            -0.01025626126223168,
+            -0.02141970191890338,
+            0.022134910980548407,
+            0.017807484142010496,
+            -0.007232643256393455,
+            0.011318041192005274,
+            -0.010489012862455913,
+            -0.013003217163997554,
+            -0.00847279880392288,
+            0.02357274819910594,
+            -0.0016485066139744577,
+            -0.005873113634992993,
+            0.006424249429163587,
+            0.004008703814301658,
+            0.0054639858216457724,
+            0.010198079500408441,
+            0.017694720520798957,
+            -0.017087021465208255,
+            -0.0025558869389290554,
+            -0.016503579212015177,
+            -0.0420082404922411,
+            -0.003563173424359332,
+            0.022924161700566468,
+            0.012345933179990547,
+            -0.00729908668365632,
+            -0.004234683369642856,
+            -0.0022962545569187934,
+            -0.020666205489143392,
+            -0.014796149043465308,
+            0.024562653223873056,
+            0.030217494971086438,
+            0.007810961433007407,
+            0.007781263640449456,
+            -0.0030348559388780655,
+            0.007310876971216526,
+            0.016099591640649783,
+            0.02311929597780174,
+            -0.04160162038353416,
+            0.027152100845897007,
+            -0.00025396315502635146,
+            0.023717123370023598,
+            -0.04186918759097667,
+            -0.017988531857836703,
+            -0.03106663560912813,
+            -0.023595079699181436,
+            0.02437134274097885,
+            -0.02317559303912261,
+            -0.012327263775257306,
+            0.000989566600374836,
+            0.00954921216572729,
+            -0.005155079395685017,
+            -0.017508404061379164,
+            -0.04582321461886958,
+            -0.0014367264231457263,
+            -0.030417229142190933,
+            -0.017829710557332023,
+            -0.001056472801554354,
+            0.005016912084754242,
+            -0.06276644565618253,
+            0.0109965681725148,
+            -0.07270669271977365,
+            -0.004542796125637244,
+            0.040139998232966566,
+            -0.017890441783630643,
+            0.055909102081208445,
+            -0.05029104308417623,
+            0.01673962668706085,
+            -0.04507293803055318,
+            0.015385506084493484,
+            0.025677372277539627,
+            0.030235171586316813,
+            -0.022189978375852563,
+            0.008461188938299745,
+            0.008033707483259569,
+            -0.045089505755429415,
+            -0.04144698743458133,
+            -0.013640256640978024,
+            0.01036782197518379,
+            0.0012696958402094232,
+            -0.016090728932978384,
+            0.023838437877040163,
+            0.011946482002767135,
+            0.03342604921859589,
+            0.04211801335157251,
+            0.002708015395637621,
+            -0.047156646610976634,
+            -0.013104464812562618,
+            -0.003933116891120681,
+            0.03155321708116329,
+            -0.018866109579446184,
+            0.015376812010630058,
+            -0.01955557376085726,
+            0.016255217383093355,
+            0.017868332102274922,
+            -0.020325179506828335,
+            0.013434364256590097,
+            0.004176904701252156,
+            -0.03877060944639854,
+            0.0006475542749369515,
+            0.00042468300294954125,
+            -0.02116992978038693,
+            0.0032654387271108293,
+            -0.00575003786159731,
+            -0.05149954071159492,
+            -0.031382926943128234,
+            -0.015730563640253065,
+            0.0382312502769352,
+            0.004297939848910064,
+            0.019724983545557365,
+            -0.017907869879607064,
+            -0.005165628493894463,
+            0.03698976707579373,
+            0.008501001373266799,
+            -0.01725527821298066,
+            0.006879620077191753,
+            0.02455792200929138,
+            -0.03224193972301503,
+            -0.039181328884413266,
+            -0.0008206986106407542,
+            -0.023306758893533952,
+            0.01672946960876128,
+            -0.02658587698869307,
+            0.005873620623736886,
+            0.0025331170979801232,
+            -0.03490809296591325,
+            0.02456764314036355,
+            0.003291113462353163,
+            -0.004303423672882112,
+            -0.020359544091059775,
+            -0.016892415165120203,
+            0.017174797487220006,
+            0.04391489511478065,
+            0.007327321684948843,
+            -0.003943713832544148,
+            0.00862684231313155,
+            -0.006361172785753631,
+            0.05826688726534012,
+            -0.02284956969636637,
+            -0.026680023999604643,
+            0.027527436347931256,
+            -0.022597669279069723,
+            -0.006277863878949772,
+            -0.017343446085471857,
+            -0.012065192152203348,
+            -0.0004183712490442143,
+            0.019339311388446943,
+            -0.019423562920897294,
+            -0.010212527447243049,
+            0.004783006556748757,
+            0.012824133870627983,
+            -0.004855945108275706,
+            0.006239949458562236,
+            0.006847581690471813,
+            0.03094005096378827,
+            0.026993131409761086,
+            -0.011331881651539968,
+            0.03297014291876675,
+            0.010265600643742144,
+            4.666113318251278e-05,
+            0.001970879281172325,
+            0.002523664335535017,
+            0.0034107811520368574,
+            -0.03143890864030425,
+            -0.01624584278940617,
+            0.013468729048885906,
+            0.005233309338223403,
+            -0.004550544967856448,
+            -0.023783653541634683,
+            -0.009402494239255264,
+            0.03334923712760375,
+            -0.003260424721959948,
+            0.02748395114281811,
+            -0.0058374897745283065,
+            -0.01985840930273332,
+            0.017595418271610163,
+            0.02881743246631328,
+            -0.05928552440605384,
+            0.017062540894682354,
+            -0.0011508547522925793,
+            0.008953761538519997,
+            0.01610561914042553,
+            -0.008894160513648953,
+            0.013301991014533684,
+            0.04080863883897929,
+            0.010738051773755976,
+            -0.023345002072886145,
+            0.004215726310838674,
+            0.01541749448570608,
+            0.01291813299324752,
+            -0.022747942296528025,
+            0.0008640791861357522,
+            0.011008522066277018,
+            0.005893793007012561,
+            0.0026106450193140156,
+            -0.006278394080583205,
+            -0.025659188287146268,
+            0.024180550333596625,
+            -0.0044491292740074925,
+            0.026967973346698617,
+            0.04390909273429254,
+            0.009596855239692042,
+            -0.03686020608700941,
+            -0.013109951664921729,
+            0.0002799680359670127,
+            -0.0038537577602593366,
+            -0.0037552094562213916,
+            0.011105295862811114,
+            -0.04185615640149602,
+            0.019950890919369135,
+            0.006453241355231516,
+            0.015314645427201533,
+            -0.020031193483734364,
+            -0.024530173488182337,
+            0.0031949828520724337,
+            -0.01492645564652531,
+            0.012875827757040941,
+            0.005057170413681063,
+            -0.004319834710426384,
+            0.005386862713494232,
+            0.00030848054013413157,
+            0.008384941028326983,
+            -0.010811526993301901,
+            0.03175068297908541,
+            0.004680987487229685,
+            0.009264547472281849,
+            -0.02856314241700808,
+            -0.021083721348429348,
+            0.020113396771629096,
+            0.03572366771051171,
+            -0.01708529918264109,
+            0.03346921724975398,
+            -0.009042046660366718,
+            0.0033219155387035227,
+            0.0151720168318272,
+            -0.015805318210244414,
+            -0.008057236707694303,
+            -0.0005483009439015931,
+            -0.002917527004423299,
+            -0.014538511124345527,
+            0.007302979537905532,
+            -0.0208529958115422,
+            -0.02965227242147972,
+            0.020064178921373875,
+            -0.00011892622932349096,
+            0.016455188521316146,
+            0.010335044657747514,
+            -0.007816155992852236,
+            -0.0052262330850905165,
+            0.006959344748883874,
+            0.04432147882994085,
+            0.026734484553994495,
+            0.05547661378166294,
+            -0.005787997952888261,
+            -0.004559444854138438,
+            -0.0082295581123363,
+            -0.006588172755707144,
+            0.04524009609473585,
+            0.015915003261638767,
+            -0.004910285095602864,
+            0.017495607570191165,
+            0.01557886571266685,
+            0.03858259260136037,
+            0.0027648525902558316,
+            -0.016912796336835408,
+            0.04014755669075221,
+            -0.004077405782937414,
+            0.008616827127381418,
+            0.011102692199778469,
+            0.0024185742218798562,
+            0.00807141794538637,
+            0.02996274720815659,
+            -0.012429186019907761,
+            0.01680942269506771,
+            -0.04571916592313758,
+            -0.02775530246832996,
+            0.014370280587419414,
+            -0.004931905540824582,
+            0.03419959665162407,
+            0.01668521021731861,
+            -0.023105305446763272,
+            -0.024761385385514877,
+            0.021529428247380984,
+            -0.002603042335561961,
+            -0.024135326707477325,
+            -0.015906833215844473,
+            0.003286605488908195,
+            -0.0023437997044735617,
+            0.0060251848371355166,
+            -0.013481652076194643,
+            0.0002908426939460218,
+            0.01953362396253743,
+            0.0038544889405965057,
+            0.020760460182873287,
+            -0.016224396121239458,
+            -0.00259233446682179,
+            -0.009282861237653619,
+            -0.02952130765234087,
+            0.017988370145301916,
+            0.006138236653154911,
+            -0.007607182955958238,
+            -0.009516767050028553,
+            0.0048347778733230365,
+            -0.005819209642901871,
+            -0.0025851701689263706,
+            -0.01073014664868326,
+            0.012433465684360061,
+            0.025784014942833097,
+            -0.016875624785552303,
+            0.007944349714123104,
+            8.651589872318479e-05,
+            -0.022306044741692753,
+            -0.022127016745066307,
+            0.020143472307319017,
+            0.002792646249762062,
+            0.008768813346304779,
+            -0.004074214276300631,
+            -0.00015024341785520542,
+            0.002990550758627699,
+            0.01951746553838439,
+            0.03896451381932467,
+            0.004232673242619471,
+            0.027876586543280867,
+            0.0145860303663199,
+            0.03994234200854897,
+            0.009657017763624678,
+            0.00696607158576712,
+            -0.026055387694656488,
+            0.02819890357011778,
+            0.00018297502510753014,
+            0.043190622821775236,
+            0.022488402221902314,
+            -0.0407730101161195,
+            -0.0021946518523759066,
+            0.01669520746276111,
+            0.013212623257794862,
+            0.017009844136059122,
+            -0.004537512994041156,
+            -0.014041213424740049,
+            0.028135404528975894,
+            -0.022859586430516003,
+            0.0008637006862481223,
+            -0.004486332934188017,
+            -0.025031795510001887,
+            0.007351370254175368,
+            -0.004567816916440959,
+            0.030025413528233364,
+            0.0014976740344892735,
+            -0.022285096307825804,
+            0.022837170656498543,
+            -0.009551269792616045,
+            0.011045530488684207,
+            0.008809047185288755,
+            0.020065001354321994,
+            0.031993362733068575,
+            0.009143210434415064,
+            0.02133640814803879,
+            0.012241777474889248,
+            0.014771538127546474,
+            -0.008804192497858033,
+            -0.02341427260922065,
+            0.023167424359504862,
+            -0.0011598649565273595,
+            -0.029627604902846688,
+            0.0021991474725787303,
+            0.003261878079344105,
+            -0.0005935713331682358,
+            -0.009144689470851402,
+            -0.005580557862988033,
+            -0.007240836451875213,
+            0.019631377011298905,
+            -0.007085213886352692,
+            0.007017573363709754,
+            -0.004074171791476129,
+            -0.00822871835665378,
+            -0.0421101573155962,
+            0.004210914075286222,
+            0.003408427700622687,
+            -0.01752179330665445,
+            0.006384440404174512,
+            0.011121481025573962,
+            0.043020575455262985,
+            -0.022809979913698254,
+            -0.0005737138409049003,
+            0.019206623212867476,
+            0.0009562551889820241,
+            -0.023427467810802533,
+            -0.038931688376706186,
+            0.007172345672967499,
+            -0.03467050750542982,
+            0.023185301620612612,
+            -0.0007075596339842658,
+            0.02387977204565927,
+            -0.03043468414449851,
+            0.013909304572370437,
+            0.004784668158421625,
+            0.012524879174804915,
+            0.021654512058639427,
+            0.03142146922870715,
+            -0.02053473475540882,
+            -0.050409862638621725,
+            -0.005406667316025549,
+            0.00035138762584085976,
+            0.022018487016014856,
+            -0.026014580970974616,
+            0.00025633004699302783,
+            -0.003349094027291921,
+            -0.02401366769212778,
+            -0.01735996745614562,
+            0.01962340791446025,
+            -0.050011480783403936,
+            0.015042913576492952,
+            0.0262737424555697,
+            -0.02507473901367252,
+            -0.05408889321034101,
+            -0.018988587082325706,
+            -0.028217893770659686,
+            -0.009285770094319318,
+            -0.00661159230987788,
+            0.028597405154740078,
+            0.003525576256054959,
+            -0.011462219358952466,
+            0.025401668390029348,
+            0.006155837251830549,
+            0.0026050693705924593,
+            0.015985722392086222,
+            0.007289910674212967,
+            -0.011452511453623235,
+            -0.014599577191293152,
+            -0.04558584556100621,
+            0.010197536827229795,
+            -0.0045431962637690155,
+            -0.019372598050508005,
+            -0.030147301333718016,
+            -0.0137828704002156,
+            -0.027060515748528666,
+            -0.0017291966950107505,
+            0.053783684195381654,
+            0.03292494290301388,
+            0.005560172296221419,
+            -0.0026085670565688026,
+            -0.010466890489356092,
+            0.036247483457995895,
+            -0.006789746401727701,
+            0.03662690472838588,
+            0.001784522073210422,
+            -0.0242673253380092,
+            0.01169382884548301,
+            0.02892627181542449,
+            0.009858451662166387,
+            -0.0005592283645355213,
+            -0.006160132931009508,
+            -0.008782830470079547,
+            0.0031207630972208394,
+            0.025613233596604418,
+            0.003268170392795548,
+            -0.010062991425127038,
+            0.018106967407336815,
+            0.004480291699976396,
+            0.02010374270557836,
+            0.03555357587371324,
+            0.023489837757983054,
+            -0.021782173652026497,
+            -0.026581762796336082,
+            -0.0013561486781077014,
+            0.0072266106510934505,
+            0.03551967295942857,
+            -0.01651315514034218,
+            0.0001274396319698452,
+            -0.008272735507120127,
+            -0.00894931703949637,
+            0.011149952898504355,
+            -0.0029671593121986116,
+            -0.0030752010285951576,
+            0.01662711184497578,
+            -0.0011672263615879709,
+            -0.033448222499147764,
+            0.0078288560210653,
+            -0.023496331346917084,
+            0.0051040606404415545,
+            -0.004113104164416575,
+            -0.007076525381595441,
+            -0.002157548492138289,
+            0.006302962169276006,
+            -0.008065841090139872,
+            -0.00035242603988955366,
+            -0.01586406637763306,
+            0.008271894439780969,
+            0.007438312999409211,
+            -0.0069617481460406095,
+            0.011402932583388205,
+            0.002666895909532698,
+            0.0005219881149353855,
+            -0.012159425670629751,
+            -0.012042145306711322,
+            -0.00957276588761916,
+            -0.03401863879157551,
+            -0.02604353712472079,
+            -0.024949533021906844,
+            -0.0014209149651479556,
+            -0.006566685925666972,
+            -0.03187796739029736,
+            -0.01209152137926458,
+            -0.011110763124407752,
+            0.007756961097537267,
+            -0.013260548545917543,
+            0.005433285079114047,
+            -6.399098520041351e-05,
+            -0.0044960488826674555,
+            0.016615636438440516,
+            -0.003643291599919672,
+            -0.019488010211707343,
+            -0.03927818878149417,
+            0.008209336177358258,
+            -0.02805593125673831,
+            -0.0038816771550190788,
+            0.007607119219721711,
+            -0.00528681677614909,
+            -0.029502897838743516,
+            -0.009149794737430706,
+            0.008767001199311985,
+            0.003507217008280066,
+            -0.01773349756494436,
+            0.03579670773605949,
+            0.026971617790626075,
+            0.011875179879209696,
+            -0.019891342676443626,
+            0.0508929813275565,
+            -0.0047010735337056275,
+            -0.012544935684845649,
+            0.018297503869683027,
+            0.02355022268131519,
+            0.03273167043015095,
+            -0.024830243320788693,
+            0.0009370814985214882,
+            -0.021937793653233454,
+            0.00535662615596573,
+            -0.035009915581788176,
+            0.0028595659314725012,
+            -0.014395027018699446,
+            0.015596459769560532,
+            0.03397372881171701,
+            -0.013254226072981457,
+            0.011195747181764677,
+            -0.016299384375589934,
+            0.00730244731485402,
+            0.00569944090461748,
+            -0.012916284280345044,
+            -0.005812311882399539,
+            -0.022902815152116493,
+            0.020140276669979015,
+            0.005809799950176385,
+            -0.0008130655714656685,
+            0.0104085826652653,
+            -0.044723432238900435,
+            0.03286429597337671,
+            0.008730225944501239,
+            0.017599189944921396,
+            0.0029762446259779998,
+            -0.0003068671951232415,
+            0.011611551606224214,
+            0.011228401175698422,
+            -0.016029727902740706,
+            0.004345913009102136,
+            -0.004613061137378433,
+            -0.03850533909474302,
+            0.009363351681220008,
+            -0.03631184057978666,
+            0.014427784677340159,
+            0.016026872002233575,
+            -0.004170582842643461,
+            -0.025166189571409366,
+            -0.005428471012333673,
+            0.010827910811085384,
+            -0.009783069664327864,
+            -0.009987805234276443,
+            -0.04163358183060102,
+            -0.025236440364963034,
+            0.034097508113604576,
+            -0.002119474613596975,
+            -0.025826403511707147,
+            -0.010830738112107807,
+            -0.011351132636491575,
+            0.020207687645133,
+            -0.006542026283946858,
+            -0.0006263975804936843,
+            -0.04150011545311013,
+            -0.0073396962917864515,
+            0.011192225907726928,
+            0.027746213794734322,
+            -0.035970017302005566,
+            0.02708126943501373,
+            0.002105633542220187,
+            0.014262674845755191,
+            -0.01626664983003942,
+            -0.005631805361903626,
+            -0.013990539604372344,
+            -0.035877093361375685,
+            0.016487723013906077,
+            -0.008607758248055054,
+            0.004714436224135091,
+            0.04458009880881818,
+            -0.0036872068506712727,
+            -0.009466125348499464,
+            0.007600705817195143,
+            -0.0034590883553048063,
+            -0.010351576085287273,
+            -0.0028564505793999295,
+            0.001959661298507,
+            -0.007080697863126256,
+            -0.04183476693971616,
+            -0.00499888976530217,
+            0.04684391782633148,
+            0.006361744381879857,
+            0.013276793765092465,
+            0.013516220521120684,
+            -0.018335746557932003,
+            -0.045577810515431215,
+            -0.0071616111758423025,
+            -0.0032763880176386808,
+            -0.022708028392758567,
+            -0.0240284945459861,
+            -0.008735040856657758,
+            0.005844145402902923,
+            0.006359368165686724,
+            -0.01120763599008222,
+            -0.022676131361171057,
+            -0.027085968700452136,
+            -0.010526103176253857,
+            -0.02123944716136792,
+            -0.010792145796733375,
+            -0.027142270190771887,
+            -0.028742234538796804,
+            0.02880426953386151,
+            0.007575182232859564,
+            -0.004084088301176117,
+            -0.010005765476121966,
+            -0.01255640546813973,
+            -0.0004774761617900028,
+            -0.0025307994917461232,
+            -0.005822065709999144,
+            0.0038614286313282164,
+            -0.011390482620811868,
+            0.023152086963976554,
+            0.0021220701260546895,
+            0.018329162172128138,
+            -0.05772324571231113,
+            -0.002709083851090071,
+            0.021497896395795905,
+            0.0033025410488410913,
+            -0.005412445141910314,
+            0.029352650459792132,
+            0.011223947499453582,
+            -0.006539460282439711,
+            -0.031355428613718234,
+            0.020942976176820262,
+            0.024122993515785947,
+            0.00817615634503138,
+            -0.012994160510030828,
+            0.01221095914384204,
+            0.01629778384790159,
+            -0.004494985018246821,
+            -0.011161098655072933,
+            0.005586857967039181,
+            0.020941039642372112,
+            0.013580622629911925,
+            0.0013398071867077221,
+            0.030905022925366163,
+            0.0006862084391508121,
+            -0.03521231612862609,
+            -0.01908191995763771,
+            -0.01275995164030872,
+            -0.003584331305986364,
+            -0.015264425022763126,
+            0.005294329654809353,
+            -0.002355465010876059,
+            0.006335052149475905,
+            0.022655379849785256,
+            0.03593231939948758,
+            0.0146937743203089,
+            0.00937330328448886,
+            0.003167462724350819,
+            -0.0013235176101076284,
+            0.012102872918129403,
+            0.008638011773075119,
+            0.0350701240268404,
+            -0.014653221697693864,
+            -0.011003994230902835,
+            -0.0037767288519369537,
+            0.010100131061186566,
+            -0.004113527518662366,
+            0.020414393256136372,
+            0.03274709615649469,
+            -0.0037222949335916876,
+            -0.005604552180246868,
+            -0.006160118231975786,
+            0.0041467454414452565,
+            0.01807879127455274,
+            -0.01252134965321004,
+            0.004364858987501461,
+            0.005549300665159658,
+            -0.0071897162833861,
+            0.018567059255813558,
+            0.0002622074720860951,
+            -0.012983550838417323,
+            0.03266376564402191,
+            0.11449509387240629,
+            0.0017251916835047628,
+            0.007125215166916139,
+            -0.040229997815548584,
+            -0.020537987527148888,
+            -0.00245728735683473,
+            0.005779972828836593,
+            0.007004732428236457,
+            0.02225914982152608,
+            0.006505827715591914,
+            0.0006708045351142873,
+            0.031708522931495536,
+            -0.017334083511709092,
+            -0.002522861743921599,
+            -0.01745656559688129,
+            0.013890451996979048,
+            -0.014312605700272691,
+            -0.013784963572164409,
+            -0.01572417853955075,
+            0.004507425871748977,
+            -0.000601158156253668,
+            0.0016158265715634729,
+            0.01791303234734911,
+            0.0011926917387973374,
+            -0.026393351252156246,
+            0.0006694008039871878,
+            0.001026031515214626,
+            -0.004514419527575117,
+            -0.011320929719992686,
+            -0.02223001723942815,
+            0.016235841442653918,
+            0.014188311619681008,
+            -0.004895218832118094,
+            0.009244697376301004,
+            0.010558796153959677,
+            0.02707485418319703,
+            -0.028380342401606758,
+            0.0040204430418629405,
+            0.007023041225985294,
+            0.01606556581447933,
+            0.017617862031190843,
+            0.0034870847111552862,
+            -0.011253315705053224,
+            0.03238289399766116,
+            -0.010563282687352038,
+            0.0065093536270233385,
+            0.014966802208644462,
+            -0.03541611540945151,
+            -5.821682910348165e-05,
+            0.0061985444938402745,
+            -0.011716201661363997,
+            0.009580035961315467,
+            -0.03120692534097433,
+            -0.0039291243987978845,
+            0.009317468331974374,
+            0.017611312558635673,
+            0.008804996982574765,
+            -0.0034129169537151925,
+            0.010822226698822738,
+            0.012510661458054741,
+            -0.0027147975086409966,
+            -0.008855213644027594,
+            0.01238502686635956,
+            -0.0007687053446954701,
+            0.00450108863711973,
+            -0.00799170686059026,
+            0.0029755595812552293,
+            0.002290650917244691,
+            -0.01554856300008412,
+            -0.011779430686394945,
+            0.00438393126679289,
+            0.0036931891075561957,
+            0.001535662168147833,
+            0.012014408636737167,
+            -0.006087255045485002,
+            -0.0077816220325470226,
+            0.0040573724156794284,
+            0.003136943055301706,
+            -5.505014316363865e-05,
+            -0.030625245024561508,
+            0.01585923084309149,
+            0.0067595435845396405,
+            0.010013308598626031,
+            -0.005315626260228517,
+            -0.00903142414091241,
+            -0.007240920015018893,
+            -0.01228521172101564,
+            -0.010657894223173453,
+            0.007658605592936815,
+            -0.00724924993432376,
+            0.011124174211132038,
+            -0.03128027787357322,
+            -0.013988715435985316,
+            0.01231792993245938,
+            -0.01528161242159377,
+            0.0007675821323979293,
+            -0.008389914312201402,
+            0.0009417242487395995,
+            -0.005119304379761705,
+            0.007887436211143328,
+            -0.03731503169547854,
+            0.007878001990271541,
+            0.02456170736765724,
+            -0.00330924625934981,
+            -0.020259771366167557,
+            -0.009613930457931786,
+            0.0241314385136084,
+            -0.008267831641886344,
+            -0.01645888287387047,
+            0.008421304862524506,
+            0.010140813149093993,
+            -0.0006309118552875393,
+            0.0008237784055549769,
+            -0.003726984191590917,
+            0.01669125388798323,
+            0.02054777616420997,
+            0.041930770964107515,
+            -0.014789610432004559,
+            0.01092349703703152,
+            0.02866938974792608,
+            -0.03378576980722847,
+            0.014728113846952417,
+            0.02518409645155824,
+            0.014590531327650454,
+            -0.0004446956315190408,
+            0.001609777186372437,
+            0.0068939321280085475,
+            -0.014273492115311677,
+            0.009247676157551494,
+            0.013380104440744367,
+            -0.018407746260686045,
+            0.020444430509441124,
+            0.03140071554008003,
+            0.0025839998101985736,
+            0.01355205268903058,
+            -0.006268939830271642,
+            -0.013860692186849113,
+            0.014717364068116991,
+            0.031429975488060896,
+            0.0006919565473572909,
+            -0.018561500446336806,
+            0.019128244642842268,
+            0.006988095806264863,
+            -0.003678529269242034,
+            -0.0021993078588142664,
+            -0.021334919445200388,
+            -0.021003586791369228,
+            0.017304350704792804,
+            0.013508779408333964,
+            0.027794485858413365,
+            -0.010594702782952029,
+            -0.007818089330662161,
+            0.014293688127901867,
+            -0.022368568887004094,
+            0.015426482139881204,
+            -0.02160820059598751,
+            -0.00571499813703566,
+            -0.0077402210272634235,
+            0.03274360648629609,
+            0.001441304806200403,
+            0.02734331677589297,
+            0.016457027764064002,
+            -0.033609420134045206,
+            -0.029790535553657092,
+            -0.00712106584238484,
+            -0.02307933297713104,
+            0.010356561171963577,
+            -0.03901183838032092,
+            -0.00071735150189382,
+            0.008251646656432826,
+            -0.008079917288471352,
+            -0.0007087526632966007,
+            -0.006419191399519291,
+            0.013989344185652506,
+            0.013661633160336152,
+            -0.0228103143826663,
+            0.023644639659577426,
+            0.008343481940133684,
+            0.025998483906975782,
+            -0.02076055376006461,
+            0.019148568238437927,
+            -0.017705363646416224,
+            -0.02777313933338929,
+            0.014358130220942483,
+            0.023585731219788467,
+            0.01635364038245971,
+            -0.007122853965024528,
+            0.007110003583832299,
+            -0.03556761320536796,
+            0.027585457849964405,
+            -0.013064062623866305,
+            -0.012504161289952458,
+            -0.0032379276562094555,
+            -0.011931827532360686,
+            -0.002574161005800279,
+            0.0008533907465694845,
+            0.0034345232246102806,
+            -0.03131462673429805,
+            -0.0002450959360520896,
+            -0.03727416416766574,
+            -0.028944814604815072,
+            0.01360126708805471,
+            -0.00036361284498363273,
+            -0.0049097813612814616,
+            0.02933242705521636,
+            -0.026339136251046315,
+            -0.02693295650849956,
+            -0.009590247544093352,
+            0.02853096456887074,
+            -0.0220673798697511,
+            -0.004519050393279848,
+            0.0015034014871316069,
+            -0.01828226786700525,
+            0.004436383319946364,
+            -0.0012748481042235446,
+            -0.010416065158876917,
+            -0.01105146139270597,
+            -0.03229479099927737,
+            -0.005822133182096844,
+            0.02106215163620738,
+            -0.006957125617688484,
+            0.002746241228040847,
+            0.013312248331456934,
+            -0.010968629118224582,
+            -0.0029946900704643812,
+            0.0017875284441794235,
+            0.023304175718480834,
+            -0.021033941064173063,
+            0.007714334541806199,
+            0.005955442031111047,
+            0.0016405586860131207,
+            0.018406562522853148,
+            -0.0054922197063461825,
+            0.0037690571959124617,
+            0.0162126534419016,
+            -0.009969304596173302,
+            0.017959307638726878,
+            0.007691549802475525,
+            0.006681837561745192,
+            0.002775700618222622,
+            0.007712770283579111,
+            0.009752652973700739,
+            -0.013506020126078112,
+            0.016911141425375933,
+            0.022468715726349706,
+            0.00023233555209372766,
+            0.021066580799163234,
+            -0.02302516292580905,
+            -0.028618544628453015,
+            0.018297878810036087,
+            -0.0225078941299981,
+            -0.012735208680311652,
+            0.016459925496937065,
+            -0.0043032724578332855,
+            -0.00797609922711709,
+            -0.014986057073351055,
+            0.02260618561608301,
+            0.02132930921464508,
+            -0.020249788476562527,
+            -0.012385936327068884,
+            0.0072304431443959505,
+            -0.004373475447051833,
+            0.007086656081865733,
+            0.03195810137269375,
+            -0.015982210897392764,
+            0.006818890093901413,
+            -0.0038306138017521626,
+            0.0030753642449792215,
+            0.017830546304667783,
+            0.009470018737705222,
+            -0.0019815241970236146,
+            0.030100238416495066,
+            -0.001330614013444443,
+            0.016392226282459177,
+            0.0004888028477871524,
+            -0.010948833692356176,
+            0.006343890495626752,
+            -0.026652806306173643,
+            -0.008627570055007363,
+            -0.001203530223345183,
+            -0.009581494451831351,
+            0.07598616930784373,
+            0.003930153552069478,
+            -0.01295957554290434,
+            -0.001673007115291578,
+            -0.001368287102181597,
+            -0.006897179139643672,
+            -0.00960679451603541,
+            0.0287166325268306,
+            -0.016613613331623346,
+            -0.01894527592131826,
+            0.0008370014132002703,
+            -0.006785618888421608,
+            0.00919924251249592,
+            0.01816363936999796,
+            -0.010225961928154075,
+            -0.01604759761423727,
+            0.001577138242530474,
+            -0.003520564936698765,
+            -0.004395434431595494,
+            0.0052399043934902014,
+            -0.010127122101423486,
+            0.006456662777583096,
+            0.026560491731327886,
+            -0.013684536319659296,
+            -0.01425196172663986,
+            0.004084982328707292,
+            -0.0010682084059972982,
+            0.009758641913073978,
+            -0.00864879062601467,
+            -0.00481222050200678,
+            -0.014375330954464725,
+            -0.011378890549403524,
+            0.02505641780042516,
+            -0.02861164504722546,
+            0.0034060847953066354,
+            0.014937227531980433,
+            -0.00016904224334342915,
+            0.01541381851648916,
+            0.0033028855218525644,
+            -0.03615935765718852,
+            -0.013677774380513626,
+            -0.02073839691250734,
+            0.019821074228011413,
+            0.01688422012576042,
+            -0.0033731840788147442,
+            -0.013840993586441808,
+            -0.026149880178874887,
+            -0.007430563876740068,
+            -0.02466234818567362,
+            -0.020708282040996014,
+            -0.019276991448222817,
+            0.006137002799644838,
+            0.005162892426268572,
+            0.022881759407670583,
+            0.0025854880108157857,
+            -0.015058889448404231,
+            0.0029948608544278386,
+            -0.006722836436935071,
+            0.011845298212977357,
+            -0.02747944486818378,
+            0.015311272959213262,
+            -0.01610493464983465,
+            0.025239768880017402,
+            0.007425208902253236,
+            0.0010569008485112994,
+            -0.008822681490988575,
+            0.00010704167529046774,
+            0.014883787023445807,
+            -0.004186968199973372,
+            -0.0007781259002896265,
+            0.02787555889813697,
+            0.02087373813833492,
+            -0.0017783425135865238,
+            0.008743718448136432,
+            0.028026471206552342,
+            0.019081083283793984,
+            0.0325261942213039,
+            -0.013815526479760766,
+            0.005113676604826083,
+            -0.028873600114081546,
+            0.010510475843707614,
+            -0.010871039729515013,
+            -0.010201824683340651,
+            -0.0067946599549165735,
+            0.027756887504389347,
+            -0.007962601205446472,
+            -0.0006687976124826922,
+            0.03469894503563926,
+            -0.014185294847328063,
+            -0.00010543064621374546,
+            0.0007386418834840544,
+            -0.009759481309152999,
+            0.02742722730386853,
+            -0.0073590367291510046,
+            0.018456032721630843,
+            -0.019560882612141943,
+            -0.013705917411254969,
+            -0.011894560208199602,
+            -0.001142768182039817,
+            -0.02576754877141608,
+            -0.008755240066826174,
+            -0.02249287216087102,
+            0.00859172262856532,
+            -0.0046012388553094375,
+            0.018845920253446127,
+            0.0011617844393195566,
+            -0.005275799428018491,
+            0.029069367036565145,
+            0.0088080893110627,
+            0.018100411389202663,
+            -0.0004971948011851863,
+            0.01216566778850985,
+            0.02369448391321832,
+            0.002637001729688794,
+            0.005012948614448268,
+            -0.024213704079796478,
+            0.003420184574622472,
+            -0.0004359900462023086,
+            8.265488264269535e-05,
+            0.016267948861008616,
+            -0.019572032453100303,
+            -0.02216264534756073,
+            0.021871880626992544,
+            -0.012742926070754964,
+            0.011676769226089007,
+            0.0035362870996336194,
+            -0.010732229460038356,
+            0.0006838777488770268,
+            -0.03083438636386314,
+            0.008820746379709355,
+            -0.03216685858285627,
+            -0.032230808828686675,
+            -0.013547681710236494,
+            -0.009994106571421729,
+            0.0008758088338659712,
+            0.028447187234357855,
+            -0.00795317029726568,
+            -0.015797892954136448,
+            0.005363565557248085,
+            -0.011199252697624111,
+            -0.00817162221789202,
+            -0.0007867892785855576,
+            0.005856939999700809,
+            0.009461959355310273,
+            0.005997804620690314,
+            0.011648335019189696,
+            0.007491946658046298,
+            -0.008028800679169293,
+            -0.008837725220736191,
+            0.015889923814605387,
+            -0.0014409716087425222,
+            -0.01176424391171685,
+            0.030430130082762363,
+            -0.013300620185884817,
+            -0.0005592926632214984,
+            -0.0021975669766032472,
+            -0.0015687208316663597,
+            -0.0021652076181411244,
+            -0.014225289143358926,
+            0.01870568404189425,
+            8.334501002782455e-05,
+            -0.014288806218067678,
+            0.0032841743905975634,
+            0.003841780428153241,
+            -0.0036498339483411074,
+            0.008388922960269383,
+            -0.008147275785908849,
+            0.017348147498892,
+            -0.015113395971380741,
+            0.008740452410210708,
+            0.0038335161236289234,
+            0.0043482078469644385,
+            -0.0065755348640655865,
+            -0.01592809184384243,
+            0.009642593996701513,
+            0.012051872087663658,
+            0.02173264012652684,
+            -0.02525660485859878,
+            0.00876161653901566,
+            -0.005374533920781607,
+            -0.014534179652013885,
+            -0.016127866637369738,
+            0.00982303248417706,
+            0.022322910435388785,
+            0.0063086346898091296,
+            0.012139263756190026,
+            0.00013383660542132047,
+            0.005848259030010402,
+            -0.014767198851314148,
+            0.010570158368602513,
+            -0.011355774405066537,
+            -0.018439261453084426,
+            -0.005243587266408389,
+            0.028894120979627323,
+            -0.005492024421080143,
+            0.017938747701972086,
+            0.0021350711250093858,
+            0.029007356490801176,
+            -0.008686781541806358,
+            0.0014883465513564527,
+            0.0039866022469891155,
+            -0.01295271525889559,
+            -0.004272008629260297,
+            0.01976273843646234,
+            0.04882868638601926,
+            0.010204475902027572,
+            -0.0009155109545661691,
+            0.010069071220988797,
+            0.005626418787776309,
+            0.002148101019776153,
+            -0.00805781810841286,
+            -0.030431071067211577,
+            0.008883707378543252,
+            0.005628218215212441,
+            -0.01997825125961566,
+            0.019838234890736068,
+            0.026483365072033395,
+            0.023040623182975964,
+            -0.011585643598171132,
+            0.016539930468533123,
+            -0.010092529089936188,
+            0.0177008735686489,
+            0.028665424114962996,
+            0.013810061902648515,
+            -0.010487150104822674,
+            -0.0020847870330548534,
+            0.006329342599564054,
+            0.004186981876004016,
+            0.00026239408223817384,
+            0.015190614519766204,
+            -0.019155180136775043,
+            0.01005728016778505,
+            0.014725826697964637,
+            0.03322436221319799,
+            8.049930133627564e-05,
+            -0.01645191835557271,
+            0.0016249281996128196,
+            -0.009976155662939693,
+            0.00856399492362911,
+            0.000632427127231252,
+            0.0032242260692672124,
+            -0.009688119824276678,
+            -0.012682779503915694,
+            -0.002851918183031152,
+            0.009403205508474241,
+            0.0021188716361101698,
+            0.002759004011231831,
+            0.007641925150177999,
+            0.0022830062031146204,
+            0.01713024474384216,
+            -0.0011181980841546123,
+            -0.00029141889484139217,
+            0.004601090256674874,
+            0.010509199590505558,
+            -0.01649852835819883,
+            -0.017548369964263514,
+            0.004337237475053179,
+            0.012217494210546133,
+            -0.013159179948224596,
+            -0.013241981632059755,
+            0.015704543608959117,
+            -0.01913575130892795,
+            0.024934102868065366,
+            -0.00942381892138331,
+            -0.004543611336526587,
+            0.000343200128934031,
+            -0.00727538880611703,
+            -0.0038751118497522673,
+            -0.0030064279504724797,
+            0.003984715814505561,
+            0.0019261818110367147,
+            0.007472350699291607,
+            0.008472369788801792,
+            0.014568547873264247,
+            0.006043895352236181,
+            0.0032580830408778395,
+            0.009109104498825864,
+            -0.014523846294121576,
+            -0.014800649887278973,
+            -0.002704791598011488,
+            -0.008895394038700809,
+            0.00902016867266151,
+            -0.006607182154874112,
+            0.00845408382182461,
+            -0.015501867835442603,
+            0.015573642225745032,
+            -0.012200552219016517,
+            -0.0008900215712478553,
+            -0.009666233545144905,
+            -2.7470277092642562e-05,
+            0.008191697904514857,
+            0.014605287653054787,
+            0.0070552942843382215,
+            -0.031921405225964206,
+            0.0017403427534633655,
+            -0.008846568352320488,
+            0.03446319360336241,
+            0.002612606662691094,
+            -0.01060863123202008,
+            0.020352181356568008,
+            -0.01592124132155158,
+            -0.00027704668715660693,
+            -0.01602075523652062,
+            -0.004797557413047212,
+            0.00707939446796679,
+            0.016399017978990946,
+            0.02759049512664933,
+            -0.005697020356865688,
+            -0.007995305872076221,
+            0.006384047872862793,
+            0.029590696670334232,
+            0.012246955508890425,
+            0.024748351124415785,
+            -0.007633712004672087,
+            0.014348228716100414,
+            0.007773260552660029,
+            0.0076045488653348975,
+            0.004321689250440021,
+            -0.002591801797145718,
+            -0.011698574606499644,
+            -0.014974971814360001,
+            -0.0037563183330623384,
+            -0.016987915346053498,
+            0.00012247833530362867,
+            0.021344400276501117,
+            -0.00475351423959752,
+            -0.0033847161875730026,
+            -0.01874186437694691,
+            0.0033461670218611346,
+            -0.010029445302549983,
+            -0.00367421246775698,
+            -0.013133469163793961,
+            -0.0254122035745965,
+            -0.018556369080030412,
+            -0.002155244024999317,
+            -0.00929309186646207,
+            -0.015194269781263309,
+            0.0070354690219415535,
+            -0.016305117309271634,
+            0.0080837166473439,
+            -0.00448473417024141,
+            -0.016508775226236883,
+            -0.00958925666995396,
+            0.01100335153658195,
+            -0.0041819993634961685,
+            -0.01929493648733792,
+            -0.01282300765645703,
+            -0.011300260564838231,
+            0.0051733704121938884,
+            0.005996780699975213,
+            -0.0038111648304178544,
+            -0.0028279639348479987,
+            0.016640121845366295,
+            -0.010197365811240557,
+            0.009878730312873492,
+            -0.03335474019383245,
+            -0.03551826040406959,
+            0.004497847402931096,
+            0.01507007013703286,
+            0.0013085881889541252,
+            -0.02078680101065954,
+            0.010806315988966003,
+            -0.016499186313125556,
+            -0.00858013792516617,
+            -0.008522534127498645,
+            0.009649521088411411,
+            -0.007283003380127847,
+            0.01299542385123007,
+            -5.3226845309105124e-05,
+            0.030477683597373687,
+            0.0074153455615142755,
+            0.004161881195438522,
+            -0.00679081048132963,
+            0.008354912600159931,
+            -0.001621776416108105,
+            0.004080191054849395,
+            0.02120524874226696,
+            0.012440848502230418,
+            -0.00218041171518331,
+            0.006458415098595872,
+            0.005678055538564355,
+            -0.017186537377121235,
+            0.02507960402044456,
+            0.024678178106806115,
+            -0.008205288425409454,
+            0.004108402466376321,
+            0.005783335302169668,
+            -0.008846663924040241,
+            -0.006897707231869481,
+            0.006527795262026734,
+            -0.018474181903007085,
+            -0.0008721381678008483,
+            0.010598084358464908,
+            -0.02255441909004843,
+            -0.004840511019512733,
+            0.021745651362109403,
+            0.004631567631335321,
+            -0.018558972627958834,
+            0.02764788406651861,
+            -0.004942361238163609,
+            -0.0027290999934532494,
+            -0.010876135202650048,
+            0.01671497143541799,
+            -0.012173527207375091,
+            0.008443317346655462,
+            0.011021541861818043,
+            -0.006295639322287843,
+            0.003083685676540999,
+            0.008277206401503182,
+            -0.004504302913715969,
+            -0.0068075424927125676,
+            -0.0010508416962215848,
+            0.010167160518175114,
+            0.0036980815941666336,
+            -0.022704963865525812,
+            -0.006020073261002279,
+            0.02934227992246887,
+            -0.015082922782242507,
+            -0.0050916398557337705,
+            0.005376294716056329,
+            0.007709342013963906,
+            -0.003476236561444464,
+            -0.0023399991233097337,
+            -0.00787486645740744,
+            0.010356414979428607,
+            -0.015950083845400274,
+            0.02076360898078935,
+            0.025608263494550438,
+            0.024504236990840842,
+            -0.014814235990238182,
+            0.010828584133720797,
+            -0.011703992394890898,
+            -0.011047470905668532,
+            -0.015542835222173864,
+            -0.001658296303013357,
+            0.001970208852236451,
+            0.013107754130616248,
+            0.009666887546609378,
+            0.0235853683657408,
+            0.006473544089109194,
+            0.008973300312205858,
+            0.014118601565410305,
+            -0.003902202854013708,
+            -0.018303438244719996,
+            -0.014584226585460755,
+            -0.011977613285028946,
+            -0.01756345491020528,
+            0.0004563602312336091,
+            0.020461659996271687,
+            0.012417744207683839,
+            0.024136626592375227,
+            0.002119213701333967,
+            0.013759902247467922,
+            -2.9525392347047554e-05,
+            -0.000892319792935443,
+            0.00045783825646242434,
+            -0.005238483120312843,
+            0.019255756091120846,
+            -0.004599752813817219,
+            0.01656589522110913,
+            0.0006443773300684925,
+            0.007133940816771615,
+            0.013554761340256566,
+            -0.01704388669741371,
+            -0.003443100892108368,
+            -0.010029620309829285,
+            0.000941391966877521,
+            -0.019633111750884034,
+            0.01934571074127927,
+            0.010273330977233852,
+            0.0017121345993533963,
+            -0.018327189396014198,
+            -0.0029706385099949693,
+            0.03448635131998772,
+            0.019458999453011265,
+            0.0050734354363614525,
+            0.02632443373898792,
+            0.013018762352749236,
+            0.014443611411502168,
+            0.010153370950346075,
+            0.02356766253713842,
+            -0.023479811300626968,
+            0.02095255383200973,
+            -0.011590347850340786,
+            -0.007414136601134069,
+            -0.0035774888741646705,
+            0.0025141508183687115,
+            -0.013588691733097813,
+            0.00982877622088977,
+            -0.0017916539399655826,
+            -0.025386211797192578,
+            0.009164316443206016,
+            -0.008811052979979271,
+            0.010464442616928296,
+            -0.007774569252594923,
+            -0.0002806136360286729,
+            -0.0127084026190163,
+            -0.015184317235990752,
+            0.001259201880504347,
+            -0.009158429132281081,
+            0.023908150915412617,
+            -0.021210848953875233,
+            -0.01321384014436847,
+            0.013997502713857512,
+            -0.006869443694669701,
+            -0.011364239609894453,
+            -0.00374449980963666,
+            0.023673050288879224,
+            -0.009243341446512421,
+            -0.01359367519009039,
+            0.014464166953530989,
+            0.003834735113400469,
+            0.0059713317014342395,
+            -0.007508862694272452,
+            -0.008428705474135382,
+            -0.003280788036976103,
+            0.005998840043285335,
+            0.026615077617284676,
+            -0.014059930972065399,
+            -0.0032833264512892022,
+            -0.009189999799498163,
+            0.004118204537632102,
+            -0.02359889635281631,
+            0.02771075351384059,
+            -0.008871395303200375,
+            0.021705177758560575,
+            -0.0015924449308442096,
+            -0.006239188158271628,
+            -0.024231036074349822,
+            0.027522945599928003,
+            0.003683870966132199,
+            -0.010221180899173243,
+            0.0012768725460662862,
+            -0.052005292736885915,
+            0.016636109796947025,
+            0.008429895913163985,
+            -0.019507994429834353,
+            0.003697519981610208,
+            -0.004773584856005853,
+            0.018983847580558853,
+            -0.022966993228850697,
+            -0.014078986007109447,
+            0.008283766047449966,
+            -0.01153285124362746,
+            0.0031439180465649996,
+            -0.02527749501821793,
+            -0.005754537200570427,
+            0.0011760345167229269,
+            -0.005276936074280004,
+            -0.001705933750221904,
+            -0.020022761608132548,
+            -0.025972227850795115,
+            0.017666017379157135,
+            0.015820875579600418,
+            -0.002278262448951789,
+            0.01320818011561616,
+            0.008709899932124187,
+            0.00836573261091119,
+            0.016034263394940008,
+            0.007124509234103267,
+            0.0081473711608784,
+            0.045498533239850446,
+            -0.019846323044145485,
+            0.007371629414298193,
+            -0.0014854149174929534,
+            0.013951741898629716,
+            -0.030467924817555014,
+            -0.0055144480998344705,
+            0.0062802744565489945,
+            -0.0049373689340857915,
+            0.03510454786147091,
+            -0.017326094216003073,
+            0.01694160526735264,
+            -0.0026075809870445238,
+            0.0007420394265014909,
+            -0.02042003877666893,
+            -0.026156937726134134,
+            -0.014294938857971834,
+            0.0011857406889443987,
+            -0.0010047941261608975,
+            -0.011946583993069242,
+            0.006797001710574735,
+            0.004917161312925866,
+            -0.0020713379451245515,
+            0.015652523103715316,
+            -0.02952061566240878,
+            0.007356800041369564,
+            -0.010733578620243498,
+            0.020008504830359524,
+            -0.00990331048284697,
+            -0.0013319400098308093,
+            0.007236960885181047,
+            -0.004122868383988748,
+            0.013216043472071927,
+            -0.0038337251451538677,
+            0.004227115653354611,
+            -0.005221321656766545,
+            -0.01794343569722877,
+            -0.0016435261495179767,
+            0.021429008432348098,
+            -0.008349684096540855,
+            0.031864497493526205,
+            -0.01264743878037828,
+            0.009174982784686633,
+            0.010692990337462762,
+            0.0006628831036334152,
+            -0.005286285851702483,
+            0.009052971814211658,
+            0.0021431970274545893,
+            -0.008690520448278274,
+            0.006934310986093342,
+            -0.018349449937310143,
+            0.008494451785507706,
+            0.03394632739506833,
+            0.0018511328780472266,
+            -0.0007519661670070526,
+            -0.007266663179613392,
+            -0.01335844201823745,
+            0.018540713029383066,
+            0.005488273906733518,
+            -0.002546531152114063,
+            -0.027536660869590543,
+            0.02007746621157705,
+            0.0012265798861291223,
+            -0.004755325069578531,
+            0.012466557901411086,
+            0.0020495867632654885,
+            0.019989531151046736,
+            -0.010171678639113316,
+            0.030757957899447096,
+            0.010050036220124393,
+            0.002959833051227505,
+            0.0038556928367700113,
+            -0.036655749074851536,
+            -0.01652152778069314,
+            -0.00019357502472740266,
+            0.014271581054865031,
+            -0.012430647252826173,
+            -0.022432580814844155,
+            0.0010346169306977334,
+            0.01148004051449691,
+            -0.0033898642762881046,
+            -0.009650665305271706,
+            0.01560757541276278,
+            -0.020342213902670575,
+            0.006701302394408393,
+            0.007693672045981921,
+            0.0287100214318104,
+            -0.0020735384554022153,
+            -0.0012324838849990056,
+            0.000626416803904792,
+            -0.0055669860801192935,
+            -0.0008037366126870824,
+            0.017251369734470945,
+            -0.013661316947002466,
+            0.0175926211217534,
+            -0.012735241490550407,
+            0.0064713656749838585,
+            -0.0021092378432037563,
+            -0.010629579860050812,
+            -0.020114807971202787,
+            0.0029609495764307357,
+            0.009378003163375899,
+            0.006920144698428806,
+            -0.010438984881449953,
+            0.010715409812532968,
+            0.012197673689706352,
+            -0.00753962861484439,
+            0.014519905132928041,
+            0.022173989895384192,
+            -0.007687503607783572,
+            0.013633321414009471,
+            -0.0041446334728745095,
+            -0.014056264816568229,
+            -0.009985463824440833,
+            0.023929750617160402,
+            -0.0005563806942667597,
+            0.008756997373618809,
+            0.00388125051641897,
+            0.0007856447663040696,
+            -0.011226530517316183,
+            -0.001653447552601681,
+            0.019294448360333633,
+            0.011831934675073843,
+            -0.0013888534822008192,
+            -0.0010525764356699885,
+            -0.0014566936525653407,
+            -0.009427557739483988,
+            0.02049188741459728,
+            -0.022030783927622997,
+            -0.0019604940727076573,
+            0.00461018148439991,
+            -0.022589816803875618,
+            0.0001863117949313752,
+            -0.024570217962989905,
+            -0.01569952241187498,
+            0.011763033781122343,
+            -0.0022530631823737554,
+            -0.016028810867854907,
+            -0.0003242701040942633,
+            7.424127301806556e-05,
+            -0.007815789940508527,
+            -0.004135500768181258,
+            0.006115909992703774,
+            -0.00034193217510010117,
+            -0.0032262189374874842,
+            0.013430154310506884,
+            -0.009996622196870008,
+            0.015431528911575157,
+            0.0067029980830614145,
+            -0.015702167721800483,
+            0.0021406610839557003,
+            0.018500219544264753,
+            -0.0058910087864916085,
+            0.00787021425357441,
+            -0.014681907249972176,
+            -0.0037495551894426303,
+            -0.005787661351944086,
+            0.006051351977821761,
+            0.024849195780885833,
+            0.012969434450643372,
+            0.01139340445077682,
+            0.017405361843075933,
+            0.004325000715054734,
+            -0.01572297405297993,
+            0.0007416798945675923,
+            0.01682727189828363,
+            0.010464036837245312,
+            0.009804326468113703,
+            0.0008975556667684365,
+            0.0043704860361487635,
+            -0.005142176468800175,
+            0.009317064072855551,
+            0.02915482449687158,
+            0.0013434689612997423,
+            0.008696141010760192,
+            0.017276730345147134,
+            0.027038741396131265,
+            -0.0035088997194497725,
+            -0.007469614319788499,
+            0.008739942310300076,
+            -0.009657991604173173,
+            -0.001081629174976152,
+            -0.017000351183542026,
+            -0.012310275736750868,
+            0.016510505259358398,
+            -0.006084627062230492,
+            0.02848913486719956,
+            -0.0010655806791907225,
+            -0.02026498848361482,
+            -0.01565505608194376,
+            0.001404549479440637,
+            -0.01362132895080652,
+            0.005287566256051299,
+            0.014187265685427415,
+            0.01917046513020831,
+            0.004687191682875024,
+            -0.022258337721085656,
+            0.037364962197399806,
+            0.01757108963349427,
+            0.019377870795760123,
+            -0.019422840506328382,
+            -0.006159311237412833,
+            -0.009077095160676276,
+            0.02116697012944498,
+            -0.00880692095488465,
+            -0.009875674856043954,
+            0.024014332004451508,
+            0.017812606845656484,
+            0.005679806286464216,
+            0.006736609904079586,
+            0.0020048663439319585,
+            0.0025414630253661444,
+            -0.015831593851012047,
+            0.017957902498542212,
+            -0.005858125698347523,
+            -0.0058068554681761945,
+            0.017529514573571298,
+            -0.005989581792993925,
+            -0.0030306021203732347,
+            0.00015839997806822045,
+            -0.019697052145531482,
+            0.011618112886207201,
+            0.008256007601576028,
+            -0.017229774429089857,
+            -0.0014757560402567304,
+            -0.0002404480326945295,
+            -0.014118694740134205,
+            0.02004885146995733,
+            0.01694893356262835,
+            -0.0013248884139405398,
+            -0.0051752815372013315,
+            0.008223868147393114,
+            0.007000679132597393,
+            0.005256194890343006,
+            -0.019496098876977206,
+            0.005230597149342019,
+            0.005103909294824105,
+            -0.0014716649591412774,
+            0.010274346323051634,
+            -0.009030594825985526,
+            0.033740380586893826,
+            -0.006505788197604522,
+            0.013065658941136721,
+            -0.0043634731684193745,
+            -0.0055529072007985125,
+            0.001816020676232658,
+            0.013054468478598695,
+            0.009753501030700457,
+            -0.00970446155762939,
+            -0.004172354962129865,
+            0.018697982092775737,
+            -0.011983156146516074,
+            0.0013300791598046338,
+            0.01564072865804035,
+            -0.004858814041438713,
+            0.004708235083775013,
+            0.008635204790345423,
+            -0.004489505237783172,
+            0.01801512858970922,
+            -0.016482572999490366,
+            0.0023953436660054026,
+            0.010694869684962088,
+            0.02390864159830096,
+            -0.019152187046410703,
+            0.021786596486708207,
+            -0.017951050280323393,
+            0.026538582634436314,
+            0.014484894489694262,
+            -0.0035222259457311444,
+            -0.01946200885147006,
+            -0.00850690913394181,
+            0.011654565129063417,
+            -0.021423803876241532,
+            -0.0007985948078290641,
+            -0.0007290222944686444,
+            -0.021367628543092133,
+            -0.011189364017231132,
+            -0.009961779129778331,
+            0.010701637745810138,
+            0.001918633898887013,
+            0.010380983868185359,
+            0.008002182827196827,
+            0.010784760574518087,
+            0.014405640442760822,
+            0.0008958107769384412,
+            -0.010743001445440941,
+            -0.023795589003655273,
+            0.016987708698679597,
+            0.00438463274382093,
+            0.0007862212695139549,
+            -0.01627468994515667,
+            0.011936651269737248,
+            0.013250643815768723,
+            -0.009385605548926286,
+            0.009121882469425302,
+            -0.03716901103204463,
+            -0.004552808410398574,
+            -0.0035704811925430366,
+            -0.0020309197365173475,
+            -0.00908678158053104,
+            0.01559445415882485,
+            -0.013682507836005851,
+            -0.0009590922904844632,
+            0.00011696985057536548,
+            -0.0074756526903548915,
+            -1.5467890432254378e-05,
+            0.020649928837031005,
+            0.008438785939908428,
+            -0.0037382105257999757,
+            0.010376213224654074,
+            0.008826031303751312,
+            -0.013900359449287676,
+            -0.01527474712786084,
+            0.019035366035884273,
+            -0.018776918681908793,
+            0.0063098418544296165,
+            0.01677916431403883,
+            0.020778813096283988,
+            -0.02891862501830075,
+            -0.0007231469961304435,
+            0.022093939332021185,
+            -0.0012807635799950273,
+            0.006370841109790851,
+            0.008781853138377092,
+            0.006754082980602544,
+            0.0029102159062355374,
+            -0.003711173531879644,
+            -4.843770941622283e-06,
+            -0.003940474588075707,
+            0.007155539022862352,
+            0.004366433868349658,
+            0.036453728071874844,
+            -0.013661129348165794,
+            0.002541308805735001,
+            -0.0058838191638835155,
+            -0.00347401304572066,
+            -0.00593469688967835,
+            0.004407665856950772,
+            -0.010055727994798024,
+            0.004330827357536078,
+            -0.001570861535288093,
+            -0.0003731129422346693,
+            0.004883969598300302,
+            -0.001701673379646408,
+            0.007036892564844683,
+            -0.011863650867472625,
+            -0.0003979727750011258,
+            -0.020174851412609312,
+            0.021117284557875267,
+            0.00011666551282792189,
+            0.003662462447326642,
+            0.03723741246290771,
+            0.00476064510662738,
+            -0.00238355131307445,
+            -0.012880057949435798,
+            -0.013551338766321525,
+            -0.012986444917747435,
+            -0.020703821629987552,
+            -0.0036665673548525388,
+            -0.021006311026247164,
+            -0.0052061406234528915,
+            0.010362862594785442,
+            -0.011540184420787572,
+            -0.004023659825622843,
+            -0.015881589631889473,
+            -0.006531981517815174,
+            -0.03114211633687585,
+            -0.01124607488650473,
+            0.007493228366452009,
+            -0.019149285949049886,
+            -0.0039932870926865805,
+            0.026850121590606017,
+            0.001095360635760988,
+            0.01387696424607248,
+            0.0038058248676879625,
+            0.008117686785954085,
+            -0.0045937472017383955,
+            -0.0005669025036141417,
+            -0.01928244877716143,
+            0.003391980330529026,
+            0.008525354854576925,
+            -0.00361507909530282,
+            0.0070008404454106905,
+            -0.008835727374865623,
+            -0.016950017602980382,
+            0.0021654956715219517,
+            0.009334379529385142,
+            -0.006639140003365139,
+            -0.017587472740268683,
+            -0.011049929670847685,
+            0.010401189523318744,
+            -0.02283429470064187,
+            0.006257145380266984,
+            0.016439581169238607,
+            -0.009363030375624585,
+            0.004651575425738153,
+            0.019809183749755425,
+            -0.011810674576781702,
+            -0.005811328501528397,
+            0.008471992070998505,
+            0.013891259999721588,
+            0.0095606134736748,
+            0.02954498207274139,
+            0.0026596809804008996,
+            0.000561870602466584,
+            0.004889263753490895,
+            -0.004277061142504552,
+            -0.017423200644235665,
+            -0.006243787897080996,
+            0.0038507494787521856,
+            0.012873996671952184,
+            -0.009879417960879032,
+            -0.00048575277472139055,
+            0.01682604801673206,
+            0.004530939990187663,
+            -0.0033901248079550284,
+            -0.011868399508738274,
+            -0.020080331108085103,
+            0.0015580259874809898,
+            -0.01362200460317948,
+            0.015354347040143093,
+            0.003390601069331352,
+            0.0015284610287850575,
+            -0.004239880336806429,
+            0.01828645373058079,
+            -0.005013418527935815,
+            0.012749833447770729,
+            -0.006148890994661865,
+            -0.004587831231448011,
+            0.007986533783001116,
+            0.009166578151006778,
+            0.0004939857995863664,
+            0.05421019830078428,
+            0.002838056604877719,
+            0.01333205648597435,
+            0.013419127695838135,
+            -0.014523733369484934,
+            0.014086525147734823,
+            0.012688569853552613,
+            0.018497344547821693,
+            0.013037819487160016,
+            -0.012353776077273462,
+            0.017354801119939964,
+            0.0024084723134891487,
+            0.003909681065813095,
+            0.0022202438064449653,
+            -0.004528576079628071,
+            0.019594683950225115,
+            0.004070222328547378,
+            -0.0074362443729372105,
+            0.004453618452623641,
+            0.005007543168224267,
+            -0.013105524222311019,
+            -0.0112958915514782,
+            0.027105208848863218,
+            -0.01219417557128091,
+            -0.018749971743148304,
+            0.006706594269933445,
+            -0.02008458683853808,
+            -0.03763686535076864,
+            -0.0021332074289244973,
+            0.009992361098878196,
+            0.002795183524495193,
+            -0.00253695604043486,
+            0.01614298064601032,
+            0.016730443047719683,
+            -0.016985296937992726,
+            -0.018245902728910347,
+            0.0025257780765481825,
+            0.0016333325293570182,
+            0.01861807236979066,
+            -0.023168663546030196,
+            -0.006378864850313035,
+            -1.3795465857380962e-05,
+            -0.006829008132963306,
+            0.004850306884033312,
+            0.004703280112191218,
+            -0.0071714759384722965,
+            -0.0027773519096287217,
+            0.002256846645341137,
+            0.01975383140755998,
+            0.013198754584066554,
+            0.01572971337890945,
+            0.002600741959577911,
+            -0.005033508062449482,
+            -0.016817359058724685,
+            -0.005450011991586895,
+            -0.013478330855299777,
+            -0.009091854345715177,
+            -0.003042052092498012,
+            0.008217376329142566,
+            -0.025053469059115735,
+            -0.011965083373220982,
+            0.0119294363235832,
+            0.00391377331459168,
+            0.0003099203772539202,
+            0.007607672728136279,
+            -0.003830132495364688,
+            0.010331053246451333,
+            0.013823890473448383,
+            0.0076112887952959766,
+            -0.004723377809559611,
+            0.009600411526173948,
+            0.011460781436250457,
+            0.006015477068316837,
+            -0.005072711638895421,
+            0.023859899048048694,
+            0.01821920335673539,
+            -0.01680365156728093,
+            -0.002149107383573353,
+            -0.0027829676356423997,
+            -0.01884308738652242,
+            0.009665830801645276,
+            0.00880322398336952,
+            0.008308268605449346,
+            0.003906369351868086,
+            0.00974531700889854,
+            -0.013875904819275925,
+            -0.0050570811397453235,
+            0.002870120865445989,
+            -0.005729556771404781,
+            -0.01901337075686543,
+            0.000297242372701423,
+            -0.011987192261104132,
+            -0.011327157550660895,
+            -0.01409710922165158,
+            -0.003221240464411861,
+            0.005283472511900512,
+            0.021472187640986972,
+            -0.004410881709548286,
+            -0.001855852749039012,
+            0.01351697599517034,
+            0.010060125186506398,
+            0.0021658738823899905,
+            0.0002206474541239167,
+            0.005418828324605768,
+            -0.0027919130944783397,
+            0.0008094042430809068,
+            -0.0015826797905407809,
+            0.021089276765434876,
+            -0.02016611086095107,
+            -0.009413977569213423,
+            -0.009288078098692112,
+            0.005842197094339927,
+            0.028793032462603636,
+            -0.007165634502373454,
+            -0.010104002083017078,
+            -0.01994315293556678,
+            0.0003786527922888834,
+            -0.07810609175343784,
+            -0.005210636756258548,
+            0.001870513618910238,
+            0.03126380113105983,
+            -0.010817888476436834,
+            0.0033673768058629823,
+            -0.002204487854177595,
+            0.014542299655712928,
+            -0.001760107965206874,
+            -0.0013283746982194259,
+            0.0071954806737858056,
+            -0.005640400818010697,
+            -0.006210989097047217,
+            0.005904163661619133,
+            -9.945941028848348e-05,
+            -0.010266938705916303,
+            0.00057263080955709,
+            0.02209498395848068,
+            0.0159398346608872,
+            -0.006575439759804175,
+            0.01816748552021211,
+            -0.00719294553484007,
+            -0.009838738592324234,
+            0.005639288780502127,
+            -0.0019007683459126327,
+            -0.015369619176147573,
+            -0.011475977270803784,
+            0.009862808084641291,
+            -0.030069890644150284,
+            -0.00010725753918890802,
+            0.00943698424646706,
+            -0.013540245857270492,
+            -0.008293042205008774,
+            0.007212974960781723,
+            0.006708343474532583,
+            -0.0012017968151025369,
+            -0.0078026000239507145,
+            -0.02527526731750743,
+            0.0027578400032925625,
+            -0.002446811971857853,
+            -0.004899702349784173,
+            0.0019960465173086183,
+            0.001479503476855694,
+            -0.012645905858466798,
+            0.01216748847452763,
+            -0.013378877368040743,
+            -0.012053251571418766,
+            0.012508842412556306,
+            -0.00027167241745418584,
+            0.01682680930386839,
+            -0.020607299598265426,
+            0.0010702151063620052,
+            -0.028075439824932304,
+            -0.007671848153932823,
+            -0.015486630894165054,
+            -0.0009776136330459065,
+            -0.0012939921569540803,
+            -0.03481880531471923,
+            0.012889447665794033,
+            -0.005200149829198305,
+            -0.006094702500154477,
+            -0.00459778303079732,
+            -0.024583504817899775,
+            0.006072281238752663,
+            0.00607350167867964,
+            -0.006506508970816635,
+            -0.016816439635445445,
+            0.008428048142588516,
+            -0.0011040593013238872,
+            0.004873670535200424,
+            -0.007475800310154908,
+            -0.0005638920132076605,
+            -0.013047135093134946,
+            -0.005463454269764494,
+            -0.0008522562324122834,
+            0.01827123304871433,
+            0.0044055851550317835,
+            -0.014526381222031981,
+            0.016862445236481526,
+            0.013250692156820124,
+            0.01602714293656341,
+            -0.0071089293427753115,
+            -0.0004265739315250755,
+            -0.02750630108787212,
+            -0.00327225092954903,
+            -0.023687644154808972,
+            0.00041213204365507744,
+            0.009447660230507965,
+            0.0203905264889617,
+            0.012694421477479161,
+            0.004620617778996077,
+            -0.001557511764988156,
+            0.0066751090724502,
+            0.02696959560423252,
+            -0.01099183126742224,
+            -0.006969111598423376,
+            -0.01884504192789845,
+            0.01113023717919988,
+            0.025135748541199255,
+            0.003381303139273845,
+            -0.011317957771935042,
+            0.008112598754655146,
+            -0.0009247845526290428,
+            -0.010302980598863016,
+            0.02889587382004902,
+            -0.0391134481322538,
+            -0.015599591108049343,
+            -0.004605824192962023,
+            0.007406520608110251,
+            -0.015596917115435033,
+            0.00770468362805995,
+            -0.012614339656105264,
+            0.001699049329817941,
+            -0.005851477164035241,
+            0.03535087340659861,
+            -0.0031950960727572925,
+            0.0006909856233673865,
+            -0.003395233050301427,
+            0.003531135044131077,
+            0.012355948349937309,
+            -0.004444221317023453,
+            -0.030726300006909638,
+            -0.0022983766869702346,
+            -0.011198252640110788,
+            0.009274666307515955,
+            -0.016207714291658314,
+            -0.017342435976846468,
+            -0.009424737744066259,
+            -0.021827243671837953,
+            -0.012403084982789827,
+            0.029747453318486338,
+            -0.005824001042844527,
+            0.01747569398142269,
+            -0.005219060548639168,
+            -0.008365003057800934,
+            0.00021272411948388247,
+            0.01344335658390732,
+            -0.028498240040284714,
+            0.00041963499785238845,
+            -0.002322931702641349,
+            0.0126393550661767,
+            -0.001595134597082556,
+            -0.00983073906977773,
+            0.006903313245694627,
+            0.0010522208685011098,
+            -0.015648940549713764,
+            -0.0007573245570947632,
+            -0.008533504878052644,
+            0.020475329445149498,
+            0.00027791387213409814,
+            0.001074213629646005,
+            0.013864331671732542,
+            -0.02102352993295736,
+            0.01576938071930521,
+            -0.021722775924283996,
+            0.01893267442161547,
+            -0.006577103101038035,
+            0.005772016664149475,
+            0.008283223343661555,
+            0.014068789332250016,
+            -0.014030042849491258,
+            -0.00663752723080222,
+            -0.034436220972634726,
+            -0.0122621879114314,
+            -0.001946072976826902,
+            -0.005308941654794638,
+            -0.014773123927909993,
+            -0.0011054822266082056,
+            -0.026205461509573584,
+            -0.006460373065662098,
+            0.012822188046808264,
+            0.019900312098359022,
+            -0.008409070094265829,
+            0.014163132434889265,
+            -0.006743711872055091,
+            -0.0028230771810882492,
+            -0.012652021749331811,
+            -0.018061180380009395,
+            -0.01083672324723357,
+            -0.01890966636827258,
+            -0.0018635408070944168,
+            0.004709222372611061,
+            -0.015760618199235518,
+            0.011435131737642627,
+            0.03198874888333405,
+            -0.01567271472650171,
+            0.008243406188694909,
+            0.016411496328333205,
+            0.0034236971031973503,
+            0.011728079139322837,
+            0.0038311415233052795,
+            0.004621789116267879,
+            -0.011109450058055078,
+            -0.011473158133621661,
+            0.006524490579223986,
+            0.00954511918454185,
+            -0.004913142601134428,
+            0.024347072255007807,
+            0.0008745419206472714,
+            0.005045794664611216,
+            0.005304506902446091,
+            0.011488187043525824,
+            -0.013135271908307747,
+            0.019746484188715552,
+            -0.002055210086078267,
+            -0.015571665869889405,
+            -0.01535499169234956,
+            -0.01105419264826655,
+            -0.018626399757822328,
+            0.0017108083455869296,
+            0.0181439079855921,
+            -0.0012493542718277685,
+            0.012137753522284387,
+            -0.00924460959150499,
+            -0.005019437549649743,
+            -0.0027156377206998474,
+            -0.038211954658821295,
+            -0.0021878970332906457,
+            -0.018809487178512628,
+            -0.02498945889484935,
+            0.03505216568400238,
+            -4.26477081213836e-05,
+            0.00997643255528116,
+            -0.010310197159232386,
+            -0.0002454902098986129,
+            -0.011065403358802515,
+            0.014954998647970473,
+            0.004624123403100598,
+            0.008872923678983478,
+            0.008560280683876702,
+            -0.016419500515801093,
+            -0.002997557798375716,
+            0.007472390947008284,
+            -0.00374822278027057,
+            0.012680407710694952,
+            0.015353684671120731,
+            0.020886474047003733,
+            0.011161262397502876,
+            0.002045862158732102,
+            0.00887271411363814,
+            0.009563361150297699,
+            -0.0003513237086717377,
+            -0.010459621030424415,
+            0.015419039131459582,
+            -0.02204848739950703,
+            0.012172257374410352,
+            0.04346634265794738,
+            0.019197433209794286,
+            0.029488039318675732,
+            -0.02272810439246353,
+            0.01448092574459596,
+            -0.003801043409441206,
+            0.009476229966122152,
+            0.0049309968883087685,
+            0.02346191840355209,
+            0.003973840862678076,
+            0.0073546443568171405,
+            0.010647581332452525,
+            -0.00972051412996416,
+            0.025644022498404723,
+            -0.00846669916801108,
+            5.1518540791212095e-05,
+            -0.004407265804145216,
+            -0.026400029105519137,
+            0.01615193008014211,
+            0.0081722536401398,
+            0.0010702314763197463,
+            0.0010263623479474385,
+            -0.0019566604229607643,
+            0.007982949510899019,
+            0.007079322685771456,
+            -0.015274148711633536,
+            -0.003886053737590683,
+            -0.018527455865296014,
+            0.0012864569064749438,
+            -0.024337133883771334,
+            0.010466651250597847,
+            -0.017568706814132484,
+            -0.014822354159709854,
+            -0.021666681078831483,
+            -0.013232016867264097,
+            -0.02011653224168991,
+            0.0035858355517491515,
+            -0.008632755352062642,
+            0.013025728891203198,
+            -0.016962939287139683,
+            -0.021494971929779273,
+            0.008430887589398292,
+            0.00870802359711456,
+            0.007481282677955534,
+            0.009728368603458082,
+            0.014996157358749115,
+            0.01736292711410622,
+            -0.022712386143092265,
+            -0.023163001611599246,
+            -0.006682209717364464,
+            -0.003531495447167166,
+            0.006762379833125297,
+            0.00933681196321128,
+            0.012884185500343249,
+            -0.007077895849501418,
+            -0.007554762054641417,
+            -0.013283067570021503,
+            0.006197966297581871,
+            -0.003993067576677973,
+            -0.009464789349849612,
+            -0.0015662739825834614,
+            -0.0061389795593180686,
+            0.002753664732282926,
+            -0.0001791088888074743,
+            0.0007595261237141144,
+            0.017122834676182343,
+            -0.008176319370751018,
+            -0.009787747261447871,
+            -0.02027228453943018,
+            -0.010683273300243025,
+            -0.015604745521156111,
+            -0.0016279406232527138,
+            -0.006506713191861036,
+            -0.014420323628589382,
+            0.010888206569219648,
+            -0.011930708571093279,
+            -0.00887210117616784,
+            0.010807147705360856,
+            0.0073900910661282935,
+            -0.003431679006289002,
+            0.037804020313161775,
+            0.0011945597874820288,
+            0.00778778394048267,
+            -0.003366543306746973,
+            0.005296727102519836,
+            -0.017761251541598876,
+            -0.026382400396890007,
+            0.0010357716464853195,
+            -0.01775777668381489,
+            0.010807713750049491,
+            -0.008515368407459745,
+            0.0016051389544388124,
+            -0.00034958600424912995,
+            -0.009585854918521076,
+            0.0036549720706351473,
+            -0.008110927907950148,
+            0.004212219431447508,
+            -0.03507890639477636,
+            0.011409377585617926,
+            -0.01819120811524444,
+            -0.008962242411035718,
+            0.01710177594046791,
+            0.037560936862121835,
+            0.0028513388998398104,
+            0.00030058353811717895,
+            0.0014792184333546447,
+            0.004427921639326168,
+            0.021438127785303025,
+            0.006339483221420334,
+            -0.001187747087595264,
+            -0.02237982599829809,
+            0.0062435650294483695,
+            0.002200578793697352,
+            0.010092828474934744,
+            0.023105023225115266,
+            -0.016797326021070008,
+            -0.015378520287395565,
+            -0.012732527281000097,
+            -0.010797463778182385,
+            0.009881499418260473,
+            0.02801767062785128,
+            0.01811139137849365,
+            -0.017358098300028903,
+            -0.020207173948076473,
+            0.0016242901745373126,
+            -0.026331540602868762,
+            0.007269751661479859,
+            0.11940631114242901,
+            -0.0045382135734049305,
+            -0.005994407567723939,
+            -0.008679712231646732,
+            -0.004482553208064535,
+            0.016709571294385293,
+            -0.013387043582971358,
+            0.069179848974036,
+            -0.007221545718220735,
+            0.0037392229364642915,
+            0.0028916224964057385,
+            0.00462729158956839,
+            -0.009727339449635306,
+            0.011570145487622916,
+            -0.011287365791681587,
+            -0.012991994493845187,
+            -0.013076194601468383,
+            -0.0180830261130804,
+            -0.007272768410218907,
+            -0.004155564814758654,
+            0.004204993170487392,
+            -0.005525575983182816,
+            0.008051108079520902,
+            -0.017551645696802648,
+            -0.010800070994485792,
+            0.013007176804761548,
+            -0.0023627485852619817,
+            0.005390986239687555,
+            -0.029668390278675554,
+            -0.0007650391259819867,
+            -0.0002446394046947245,
+            0.021280292985495608,
+            0.003137771886236958,
+            -0.0019077298815862946,
+            -0.010810159675564049,
+            0.007040107942856957,
+            0.023835993694726953,
+            0.021875366566457236,
+            0.0180511566130312,
+            0.012135351541044354,
+            0.020483975362807574,
+            -0.008667079096167644,
+            0.026324312888512866,
+            -0.001552207146726937,
+            -0.00397810465151493,
+            0.0077577136352257004,
+            -0.009849600842081875,
+            -0.00816392268697345,
+            0.00677353736462267,
+            -0.017288900282504148,
+            -0.020418071729964713,
+            -0.020383762873109595,
+            0.025741408097358233,
+            -0.011755094311186617,
+            0.002737129201159793,
+            -0.008266210469668866,
+            0.027287436144516414,
+            -0.006283892307996516,
+            -0.0029440642954701632,
+            0.009428876742171362,
+            -0.03788574758692254,
+            0.008834605607057885,
+            0.006463342392514337,
+            0.026711196829464153,
+            -3.9827332346130654e-05,
+            0.005565561841469247,
+            -0.0006738085601962865,
+            -0.006104320995853506,
+            -0.010792830169647851,
+            2.4565453208679464e-05,
+            0.017983908748172493,
+            0.0035181716362611077,
+            0.016970206528324653,
+            -0.005449808667944701,
+            -0.023171388778833197,
+            0.007092509502738752,
+            -0.01555320330471131,
+            -0.023306285898839978,
+            -0.002028783456704567,
+            -0.03196436161255752,
+            0.0033412618908711163,
+            0.007527633574188688,
+            -0.01890570959143866,
+            -0.017859735533232564,
+            0.010329340599904344,
+            0.004382396539605985,
+            -0.0016746397858953155,
+            0.006643785064338525,
+            -0.03307812909454471,
+            -0.01927475592253288,
+            0.004686970010943116,
+            0.012733525168279355,
+            -0.012993441126970916,
+            0.015291832952199082,
+            -0.020491167895268936,
+            0.016451976170508816,
+            -0.012259387189535237,
+            0.018393274368854132,
+            -0.006997668664226754,
+            0.011843450716200625,
+            0.011457415642860215,
+            0.009560877634897626,
+            -0.01790001115302499,
+            0.0214879415944361,
+            0.0066829070506954115,
+            0.03900547021123392,
+            0.02512484499576633,
+            -0.012651294056575184,
+            -0.0018425680226742782,
+            -0.00011361458320146007,
+            0.014982647266052349,
+            0.0061874424115559,
+            -0.003689464024837091,
+            0.02705902205734107,
+            0.018239469373522537,
+            -0.021070704644859155,
+            0.020037508334843183,
+            -0.00973804771680803,
+            0.018876283979591423,
+            -0.013303080480396818,
+            0.010592424494628194,
+            0.004717870260467274,
+            0.01070131238869639,
+            0.009087656015197565,
+            0.006708673669681232,
+            0.025928065153829043,
+            0.03663367449576964,
+            -0.00511073108390551,
+            0.002161381031392467,
+            0.004864092591944181,
+            0.012814518637357098,
+            0.001556833001747549,
+            0.007915766323215207,
+            -0.007012785514562176,
+            0.005834786334750152,
+            -0.002311533032619821,
+            -0.00122162928193159,
+            0.019525999280692766,
+            0.000824163362963678,
+            0.012875575199417254,
+            0.013259376059185821,
+            -0.002390133687709065,
+            -0.005682644446302278,
+            0.0019262218417782657,
+            -0.018193478045923098,
+            -0.015131696847666434,
+            0.029463723011600933,
+            -0.011447039651934148,
+            -0.008083114389964213,
+            0.0020642857121204753,
+            0.012160202510990566,
+            -0.00866944494225225,
+            -0.005378300635997713,
+            0.0038280364584418335,
+            0.014427318067525318,
+            -0.01799358399064332,
+            0.003381165206832161,
+            0.005252552821854107,
+            0.019433554824232185,
+            -0.00028791985674147885,
+            -0.013487678738391956,
+            0.0005317922815671783,
+            0.014982152801567637,
+            0.011902348165369078,
+            0.0016056390299304837,
+            -0.006935777197137747,
+            -0.00446584737370504,
+            0.004586814866182051,
+            0.006269466956110759,
+            0.0027516492952993445,
+            0.008157662045793032,
+            -0.0025001885993292637,
+            -0.016648747436203294,
+            0.017735757469839637,
+            -0.0030669873770852725,
+            0.018058248627079793,
+            -0.0038238312452176745,
+            -0.0147172470474118,
+            0.0043328417868095355,
+            -0.001888507803054499,
+            -0.018556943283494755,
+            0.009407930200720534,
+            0.002560949424273885,
+            0.010411159836451958,
+            0.0052571230238294825,
+            -0.007499470181994448,
+            -6.145889479758644e-05,
+            -0.015163819825329975,
+            0.022836682800304944,
+            0.016390944816242627,
+            0.001713572907204073,
+            -0.0035687505973373878,
+            -0.031210299824387475,
+            0.0029076511444359886,
+            -0.007822231980296341,
+            0.017821806420465356,
+            0.008250605126658402,
+            0.007908773724009905,
+            0.008351355849444162,
+            -0.00743970642051451,
+            -0.002411981861500092,
+            0.012543476434788293,
+            0.0202407777715137,
+            -0.0024635818448627447,
+            0.02535771413579989,
+            -0.012533449594824386,
+            -0.0022489292647974992,
+            0.0012362917358624392,
+            0.01578939075186126,
+            -0.0008001783766371555,
+            0.006311837812042848
+          ],
+          "keyphrases": [
+            "Switch Transformers",
+            "Mixture of Experts",
+            "sparsely-activated model",
+            "training instability",
+            "multilingual settings"
+          ],
+          "title": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "be3079d3-622e-4848-bff6-15e3ac00f52b",
+        "properties": {
+          "page_content": "7 Discussion 25 ",
+          "keyphrases": [
+            "Discussion"
+          ],
+          "title": "7 Discussion",
+          "embedding": [
+            -0.015314480289816856,
+            0.01669096015393734,
+            -0.018997248262166977,
+            0.02089332602918148,
+            -0.021677281707525253,
+            -0.005907013546675444,
+            -0.010774830356240273,
+            0.0937100350856781,
+            -0.00431403424590826,
+            0.05294434353709221,
+            0.0064402855932712555,
+            -0.027529601007699966,
+            -0.014475829899311066,
+            -0.01842295378446579,
+            -0.024448473006486893,
+            0.03755693882703781,
+            -0.012051037512719631,
+            0.033710088580846786,
+            -0.006408380344510078,
+            -0.030100245028734207,
+            0.01741110533475876,
+            -0.012816760689020157,
+            0.011850490234792233,
+            0.07365535199642181,
+            -0.013801263645291328,
+            0.025287123396992683,
+            0.04072922468185425,
+            0.02200544998049736,
+            -0.03755693882703781,
+            0.04565173387527466,
+            -0.010009106248617172,
+            0.004895164165645838,
+            -0.0015485400799661875,
+            -0.04878755658864975,
+            -0.024029148742556572,
+            0.00932086631655693,
+            0.03347307816147804,
+            -0.010191421955823898,
+            0.033710088580846786,
+            0.02512303926050663,
+            -0.030592497438192368,
+            -0.039453018456697464,
+            -0.0070100207813084126,
+            0.0217137448489666,
+            0.010382852517068386,
+            -0.006080213002860546,
+            -0.025505902245640755,
+            -0.018559690564870834,
+            0.0007360977469943464,
+            0.012707372196018696,
+            -0.016827696934342384,
+            -0.04269822686910629,
+            0.01159524917602539,
+            0.0005062096752226353,
+            0.014184125699102879,
+            -0.00020211352966725826,
+            0.011440280824899673,
+            0.034111179411411285,
+            0.00420920317992568,
+            -0.02865995466709137,
+            0.003798993770033121,
+            -0.044375527650117874,
+            -0.004564717877656221,
+            -0.004081582184880972,
+            -0.016408370807766914,
+            -0.02123972587287426,
+            -0.013500443659722805,
+            0.01520509086549282,
+            -0.02880580723285675,
+            -0.021130336448550224,
+            -0.014421135187149048,
+            -0.008003639057278633,
+            -0.0213126502931118,
+            0.04120324179530144,
+            -0.017447568476200104,
+            -0.012944381684064865,
+            0.03330899402499199,
+            0.01408385206013918,
+            -0.019799435511231422,
+            0.044448453933000565,
+            0.02428438887000084,
+            -0.019070174545049667,
+            -0.025378281250596046,
+            0.030829505994915962,
+            0.029790310189127922,
+            -0.01874200627207756,
+            -0.06716493517160416,
+            -0.11777564138174057,
+            0.026891497895121574,
+            0.022114837542176247,
+            0.006271644029766321,
+            -0.012962613254785538,
+            -0.012224236503243446,
+            -0.0341658741235733,
+            -0.009398350492119789,
+            -0.00598905561491847,
+            -0.01561530027538538,
+            -0.0026207815390080214,
+            0.023099340498447418,
+            -0.012780298478901386,
+            0.03454873710870743,
+            0.0032497690990567207,
+            0.0071695465594530106,
+            0.024685483425855637,
+            0.022461237385869026,
+            -0.00037574031739495695,
+            0.02880580723285675,
+            0.016700075939297676,
+            -0.0015508190263062716,
+            0.007880575954914093,
+            -0.024612557142972946,
+            0.022953487932682037,
+            -0.030009087175130844,
+            -0.010428431443870068,
+            0.00737465126439929,
+            -0.015505910851061344,
+            0.006800358183681965,
+            -0.04754781350493431,
+            -0.017155863344669342,
+            0.016453949734568596,
+            -0.0017046474386006594,
+            -0.005405646748840809,
+            -0.03053780272603035,
+            -0.02102094702422619,
+            -0.027183201164007187,
+            0.006253412459045649,
+            4.162627010373399e-05,
+            -0.011121229268610477,
+            -0.005282584112137556,
+            0.016435718163847923,
+            0.006668179761618376,
+            0.04557880759239197,
+            -0.02499541826546192,
+            0.029188670217990875,
+            0.002563808113336563,
+            -0.005733814090490341,
+            0.002456697868183255,
+            0.02665448747575283,
+            -0.008040102198719978,
+            -0.01068367250263691,
+            -0.02235184796154499,
+            -0.003769367467612028,
+            0.006385591346770525,
+            0.02900635451078415,
+            -0.01597081497311592,
+            0.023901527747511864,
+            -0.0013240644475445151,
+            0.021640818566083908,
+            0.006157697178423405,
+            0.021549660712480545,
+            0.0255423653870821,
+            -0.012470361776649952,
+            -0.0430263951420784,
+            0.03489513695240021,
+            0.006426611915230751,
+            -0.010510473512113094,
+            -0.009380118921399117,
+            0.014931618236005306,
+            -0.012133078649640083,
+            0.010865988209843636,
+            0.015414752997457981,
+            0.06450313329696655,
+            -0.022096605971455574,
+            -0.0035095682833343744,
+            0.04204189404845238,
+            -0.01353690680116415,
+            0.02561529166996479,
+            -0.008409290574491024,
+            -0.04204189404845238,
+            0.019544193521142006,
+            0.04448491707444191,
+            0.008309016935527325,
+            0.004799448419362307,
+            -0.012333625927567482,
+            0.008514121174812317,
+            0.02242477424442768,
+            -0.003452594857662916,
+            -0.0010278021218255162,
+            -0.014074736274778843,
+            0.023281656205654144,
+            -0.04393797367811203,
+            0.003309021471068263,
+            0.012762066908180714,
+            -0.0032725585624575615,
+            -0.034311726689338684,
+            -0.01408385206013918,
+            0.04587051272392273,
+            -0.02650863490998745,
+            0.04572466015815735,
+            0.010182306170463562,
+            -0.0075751980766654015,
+            -0.002725612837821245,
+            -0.007570640183985233,
+            -0.01061986293643713,
+            -0.014995427802205086,
+            -0.06351862847805023,
+            0.01214219443500042,
+            0.011686407029628754,
+            0.0002243331982754171,
+            0.01436644047498703,
+            -0.021677281707525253,
+            -0.013646295294165611,
+            -0.00037232189788483083,
+            0.007383767049759626,
+            -0.011987227015197277,
+            -0.0016602081013843417,
+            0.010364620946347713,
+            0.022734710946679115,
+            0.03617134317755699,
+            -0.024867799133062363,
+            0.010701904073357582,
+            0.02435731515288353,
+            -0.022151300683617592,
+            0.04054690897464752,
+            0.01696443371474743,
+            0.008126701228320599,
+            -0.019197793677449226,
+            0.06319046020507812,
+            0.01896078512072563,
+            -0.040218740701675415,
+            0.025432975962758064,
+            -0.000753759581129998,
+            0.01708293706178665,
+            0.026745645329356194,
+            -0.012634445913136005,
+            0.026071079075336456,
+            -0.02359159104526043,
+            0.021549660712480545,
+            -0.01819506101310253,
+            -0.052907880395650864,
+            0.02603461593389511,
+            0.014566987752914429,
+            -0.01453964039683342,
+            -0.010100264102220535,
+            -0.041677262634038925,
+            -0.026198700070381165,
+            -0.025761142373085022,
+            0.0063126650638878345,
+            0.024047380313277245,
+            0.03068365529179573,
+            0.029735615476965904,
+            -0.010109379887580872,
+            -0.040984466671943665,
+            0.010364620946347713,
+            -0.054803960025310516,
+            0.020783936604857445,
+            -0.025141270831227303,
+            0.029553299769759178,
+            0.0002243331982754171,
+            -0.010920682922005653,
+            0.020072907209396362,
+            -0.007365535479038954,
+            0.012406552210450172,
+            0.029808541759848595,
+            -0.008204185403883457,
+            0.045979902148246765,
+            0.04481308534741402,
+            -0.0034844998735934496,
+            -0.0023837718181312084,
+            0.004040561616420746,
+            -0.018204176798462868,
+            -0.010437547229230404,
+            -0.032379187643527985,
+            -0.014074736274778843,
+            0.02138557657599449,
+            -0.01175933238118887,
+            0.021458502858877182,
+            0.004427981097251177,
+            0.026472171768546104,
+            -0.009024604223668575,
+            0.02568821609020233,
+            -0.01901547983288765,
+            0.010391968302428722,
+            0.005587961990386248,
+            -0.027092043310403824,
+            -0.014402903616428375,
+            0.01896078512072563,
+            -0.005309931468218565,
+            0.004931627307087183,
+            0.032999057322740555,
+            0.00916134100407362,
+            0.03061072900891304,
+            -0.0073883249424397945,
+            -0.025943458080291748,
+            -0.010556052438914776,
+            -0.035223305225372314,
+            -0.016052857041358948,
+            0.024813104420900345,
+            -0.012716487981379032,
+            -0.0217137448489666,
+            0.009999990463256836,
+            -0.005027342587709427,
+            -0.0037306256126612425,
+            0.017228789627552032,
+            0.011431165039539337,
+            -0.009234266355633736,
+            0.021330881863832474,
+            0.005095710977911949,
+            -0.0022117116022855043,
+            0.005628983024507761,
+            -0.009735633619129658,
+            0.021257957443594933,
+            -0.024667251855134964,
+            0.001667044940404594,
+            -0.009516855701804161,
+            0.01627163402736187,
+            0.005911571439355612,
+            0.009343655779957771,
+            -0.0852506011724472,
+            -0.020236991345882416,
+            0.06257058680057526,
+            -0.03839558735489845,
+            -0.009361887350678444,
+            0.009972643107175827,
+            -0.017000894993543625,
+            0.019252488389611244,
+            0.02497718669474125,
+            0.013710105791687965,
+            -0.01644483394920826,
+            0.0024361873511224985,
+            0.02082039974629879,
+            0.005670004058629274,
+            -0.029735615476965904,
+            -0.004749312065541744,
+            0.0068185897544026375,
+            0.002308566588908434,
+            -0.024047380313277245,
+            0.027092043310403824,
+            0.006882400251924992,
+            0.0013126697158440948,
+            0.023792138323187828,
+            -0.03234272450208664,
+            0.005856877192854881,
+            -0.004876932594925165,
+            -0.00793527066707611,
+            0.004968089982867241,
+            -0.041677262634038925,
+            0.03624426946043968,
+            0.002105741063132882,
+            -0.030993590131402016,
+            0.01408385206013918,
+            0.009817675687372684,
+            0.011777563951909542,
+            0.02220599539577961,
+            -0.012579751200973988,
+            -0.006855052895843983,
+            0.01461256667971611,
+            -0.023281656205654144,
+            4.2480874981265515e-05,
+            0.010364620946347713,
+            0.000994757516309619,
+            0.006595253478735685,
+            -0.013582484796643257,
+            0.014949849806725979,
+            -0.0014881481183692813,
+            -0.035843174904584885,
+            -0.05560614541172981,
+            -0.014448482543230057,
+            0.0003068023652303964,
+            9.008936467580497e-05,
+            0.04477662220597267,
+            -0.013381938450038433,
+            -0.00829534325748682,
+            0.0055834040977060795,
+            -0.029024586081504822,
+            -0.07099355012178421,
+            0.035369157791137695,
+            0.022661784663796425,
+            -0.024940723553299904,
+            0.014758418314158916,
+            -0.03228802978992462,
+            -0.02789423055946827,
+            0.01589788869023323,
+            -0.020492233335971832,
+            -0.017119400203227997,
+            0.0018675917526707053,
+            -0.007629892788827419,
+            -0.005888781975954771,
+            -0.03201455622911453,
+            0.026417477056384087,
+            -0.02269824780523777,
+            0.01121238712221384,
+            -0.040911540389060974,
+            -0.013281664811074734,
+            0.005510477814823389,
+            -2.746301333900192e-06,
+            0.01567910984158516,
+            0.026289857923984528,
+            -0.02116679958999157,
+            0.00796261802315712,
+            -0.04754781350493431,
+            -0.012379204854369164,
+            -0.006859610788524151,
+            -0.017265252768993378,
+            0.03108474798500538,
+            0.006203276105225086,
+            0.01651776023209095,
+            -0.00870555266737938,
+            0.036262501031160355,
+            -0.030446644872426987,
+            -0.00527346832677722,
+            0.018577922135591507,
+            -0.006581580266356468,
+            -0.017401989549398422,
+            0.04196896776556969,
+            0.00760254543274641,
+            -0.019598888233304024,
+            -0.0020943463314324617,
+            -0.005838645622134209,
+            0.0072287991642951965,
+            0.009726517833769321,
+            -0.009917949326336384,
+            0.036116648465394974,
+            -0.011185039766132832,
+            0.06468544900417328,
+            0.0025205081328749657,
+            0.04565173387527466,
+            0.05367360636591911,
+            -0.02971738390624523,
+            -0.01061986293643713,
+            0.007739281747490168,
+            0.0238103698939085,
+            0.026818571612238884,
+            -0.017219673842191696,
+            -0.025232428684830666,
+            0.03615311160683632,
+            -0.015788499265909195,
+            -0.0039106616750359535,
+            0.010319042019546032,
+            -0.030556034296751022,
+            -0.028714649379253387,
+            0.04200543090701103,
+            -0.020528696477413177,
+            -0.007775744888931513,
+            0.018632616847753525,
+            -0.01936187781393528,
+            0.005574288312345743,
+            0.008422964252531528,
+            -0.03174108266830444,
+            0.024029148742556572,
+            0.010154958814382553,
+            -0.005246120970696211,
+            -0.015569721348583698,
+            0.046052828431129456,
+            -0.017593421041965485,
+            0.0016066529788076878,
+            -0.031412914395332336,
+            -0.021203262731432915,
+            -0.04477662220597267,
+            0.035843174904584885,
+            -0.03547854349017143,
+            0.029899699613451958,
+            -0.038213275372982025,
+            0.010574284009635448,
+            0.036061953753232956,
+            -0.00958066526800394,
+            0.0019553308375179768,
+            0.018924321979284286,
+            -0.020510464906692505,
+            -0.06891515851020813,
+            0.0106563251465559,
+            0.022734710946679115,
+            0.021002715453505516,
+            -0.015405637212097645,
+            -0.0364995114505291,
+            0.03330899402499199,
+            -0.0038628040347248316,
+            -0.03163169324398041,
+            0.005783950909972191,
+            0.018796700984239578,
+            -0.015834078192710876,
+            -0.010382852517068386,
+            -0.05425701290369034,
+            -0.005382857285439968,
+            -0.0376298651099205,
+            -0.024047380313277245,
+            0.0347675159573555,
+            -0.013117581605911255,
+            -0.04510479047894478,
+            0.03901546075940132,
+            -0.033837709575891495,
+            0.011996342800557613,
+            -0.0033477635588496923,
+            0.022224226966500282,
+            -0.02242477424442768,
+            0.015587952919304371,
+            0.042370062321424484,
+            -0.041458483785390854,
+            0.004712848924100399,
+            0.039380092173814774,
+            0.03296259418129921,
+            -0.01585230976343155,
+            0.06618043035268784,
+            -0.005820414051413536,
+            -0.013728337362408638,
+            -0.06771188229322433,
+            -0.00018388200260233134,
+            -0.07817677408456802,
+            -0.001375340623781085,
+            0.03365539386868477,
+            -0.008901541121304035,
+            -0.019325414672493935,
+            -0.033071983605623245,
+            0.03312667831778526,
+            0.009206918999552727,
+            -0.025177733972668648,
+            0.011713754385709763,
+            -0.04579758644104004,
+            -0.010802177712321281,
+            0.0192160252481699,
+            0.027985388413071632,
+            0.005551498848944902,
+            0.00091556430561468,
+            -0.012105731293559074,
+            -0.019197793677449226,
+            -0.007543292827904224,
+            -0.016645381227135658,
+            0.03132175654172897,
+            -0.004416586831212044,
+            0.012652677483856678,
+            0.0019040547776967287,
+            -0.007648124359548092,
+            -0.03387417271733284,
+            0.04324517399072647,
+            -0.01773015595972538,
+            0.012570635415613651,
+            0.013181391172111034,
+            -0.008810384199023247,
+            -0.0016602081013843417,
+            0.011257966049015522,
+            0.0009468997595831752,
+            0.004343660548329353,
+            0.006030076649039984,
+            0.022187763825058937,
+            0.031212368980050087,
+            -0.0015850031049922109,
+            0.0024908818304538727,
+            0.01916133053600788,
+            0.007939828559756279,
+            0.026490403339266777,
+            -0.0023814928717911243,
+            0.024193231016397476,
+            -0.030628960579633713,
+            -0.028295325115323067,
+            0.007643566466867924,
+            0.01916133053600788,
+            -0.00929807685315609,
+            -0.00896535161882639,
+            -0.015861425548791885,
+            -0.02915220707654953,
+            0.02386506460607052,
+            -0.0023792139254510403,
+            0.008290785364806652,
+            -0.01992705464363098,
+            0.0021137173753231764,
+            -0.0013696432579308748,
+            0.0315040722489357,
+            0.019416572526097298,
+            0.013728337362408638,
+            -0.004619412124156952,
+            0.011084766127169132,
+            0.0028988122940063477,
+            -0.027164969593286514,
+            0.0028646280989050865,
+            -0.011048302985727787,
+            -0.007023694459348917,
+            -0.014348208904266357,
+            0.043682731688022614,
+            0.004453049506992102,
+            -0.025031881406903267,
+            0.009571549482643604,
+            0.00199065450578928,
+            -0.01679123379290104,
+            -0.009097530506551266,
+            -0.0017673183465376496,
+            0.033965326845645905,
+            0.027875998988747597,
+            -0.028204167261719704,
+            -0.02499541826546192,
+            0.006207833997905254,
+            0.013783032074570656,
+            0.002743844408541918,
+            0.009872370399534702,
+            -0.004425702150911093,
+            -5.647499347105622e-05,
+            -0.01207838486880064,
+            -0.009389234706759453,
+            -0.02025522291660309,
+            -0.009617128409445286,
+            0.020802168175578117,
+            0.025998152792453766,
+            0.020364612340927124,
+            0.031066516414284706,
+            -0.04634453356266022,
+            -0.010957146063446999,
+            0.016453949734568596,
+            0.04856877774000168,
+            0.025159502401947975,
+            -0.006408380344510078,
+            -0.03290789946913719,
+            -0.008787594735622406,
+            0.010893335565924644,
+            -0.010200537741184235,
+            0.012461245991289616,
+            -0.01731083169579506,
+            0.004026887938380241,
+            -0.03194162994623184,
+            0.013427517376840115,
+            -0.0015154953580349684,
+            0.02700088731944561,
+            -0.02020052820444107,
+            0.012169541791081429,
+            0.01894255355000496,
+            -0.02957153134047985,
+            -0.011987227015197277,
+            -0.0315040722489357,
+            -0.003691883524879813,
+            0.001616908237338066,
+            -0.01589788869023323,
+            0.0061440235003829,
+            0.014402903616428375,
+            -0.010492241941392422,
+            0.0011018677614629269,
+            0.024466704577207565,
+            0.019106637686491013,
+            0.0077939764596521854,
+            -0.00453053368255496,
+            0.0063400124199688435,
+            -0.009361887350678444,
+            -0.028623491525650024,
+            -0.03219687193632126,
+            0.018331797793507576,
+            0.02180490270256996,
+            0.003411573823541403,
+            -0.006175928749144077,
+            0.005095710977911949,
+            0.057283446192741394,
+            -0.026891497895121574,
+            0.018833164125680923,
+            0.0016043740324676037,
+            -0.03901546075940132,
+            -0.031832240521907806,
+            -0.003343205666169524,
+            0.06089328974485397,
+            -0.028222398832440376,
+            0.01121238712221384,
+            0.015606184490025043,
+            0.01817682944238186,
+            0.002261848421767354,
+            0.0003395621315576136,
+            -0.034184105694293976,
+            0.014329977333545685,
+            -0.01207838486880064,
+            0.01894255355000496,
+            0.031394682824611664,
+            -0.04072922468185425,
+            -0.004804006312042475,
+            -0.021950755268335342,
+            -0.0012830435298383236,
+            -0.024029148742556572,
+            -0.024175001308321953,
+            0.00619416031986475,
+            0.00550136249512434,
+            0.0071695465594530106,
+            -0.0047265226021409035,
+            -0.006444843485951424,
+            -0.0015838636318221688,
+            -0.00231198500841856,
+            0.00924794003367424,
+            -0.007014578673988581,
+            0.0008050356991589069,
+            -0.01159524917602539,
+            0.01718321070075035,
+            0.0007492016884498298,
+            -0.03832266107201576,
+            -0.013491327874362469,
+            -0.020036444067955017,
+            0.011950763873755932,
+            -0.0008283948409371078,
+            -0.01908840611577034,
+            -0.020036444067955017,
+            -0.011340008117258549,
+            0.010948030278086662,
+            0.01696443371474743,
+            -0.02123972587287426,
+            0.0044781179167330265,
+            0.008669089525938034,
+            -0.0035095682833343744,
+            -0.007680029142647982,
+            -0.0018493601819500327,
+            0.0215131975710392,
+            0.04940742999315262,
+            0.004020051099359989,
+            0.006016402970999479,
+            -0.007771186996251345,
+            -0.017338179051876068,
+            -0.04102092981338501,
+            0.03923423960804939,
+            0.01339105423539877,
+            0.029735615476965904,
+            -0.00248176627792418,
+            -0.04058337211608887,
+            -0.0016624870477244258,
+            -0.01783042959868908,
+            0.004172740038484335,
+            -0.0034298053942620754,
+            -0.008637184277176857,
+            0.004475838970392942,
+            -0.017319947481155396,
+            -0.0326344259083271,
+            -0.016873275861144066,
+            -0.029334520921111107,
+            0.0009634220623411238,
+            0.022661784663796425,
+            -0.021604355424642563,
+            0.009361887350678444,
+            -0.0025979920756071806,
+            -0.039963498711586,
+            -0.007433903869241476,
+            -0.003502731444314122,
+            -0.01398357842117548,
+            0.013719221577048302,
+            -0.014220588840544224,
+            0.0023245192132890224,
+            -0.005054689943790436,
+            0.01467637624591589,
+            0.025232428684830666,
+            -0.02138557657599449,
+            0.005373741500079632,
+            -0.0001051873477990739,
+            -0.00041220334242098033,
+            -0.013609832152724266,
+            0.0005908153252676129,
+            -0.0012819040566682816,
+            -0.020109370350837708,
+            0.018131250515580177,
+            0.004927069414407015,
+            0.02776661142706871,
+            0.016636265441775322,
+            -0.03312667831778526,
+            0.010072916746139526,
+            -0.002771191531792283,
+            -0.01696443371474743,
+            0.015496795065701008,
+            -0.012898802757263184,
+            0.0008266856311820447,
+            0.02422969415783882,
+            0.017620768398046494,
+            -0.018377376720309258,
+            0.003115311497822404,
+            0.00978121254593134,
+            -0.04247945174574852,
+            0.062351811677217484,
+            -0.00193937832955271,
+            -0.03347307816147804,
+            0.016499528661370277,
+            0.018304450437426567,
+            0.01673653908073902,
+            -0.0045669968239963055,
+            0.00888786744326353,
+            -0.002016862388700247,
+            0.03555146977305412,
+            0.012944381684064865,
+            0.04776659235358238,
+            0.029407447203993797,
+            -0.014566987752914429,
+            -0.006458517163991928,
+            -0.01819506101310253,
+            -0.00801275484263897,
+            0.01683681271970272,
+            0.008960793726146221,
+            0.01742933690547943,
+            0.021458502858877182,
+            -0.013546021655201912,
+            0.015861425548791885,
+            0.005382857285439968,
+            0.00898814108222723,
+            -0.05330897495150566,
+            -0.0430263951420784,
+            0.007543292827904224,
+            -0.01565176248550415,
+            -0.033071983605623245,
+            -0.005929803010076284,
+            -0.004872374702244997,
+            -0.022607089951634407,
+            0.0025205081328749657,
+            0.041057392954826355,
+            -0.024175001308321953,
+            0.027711916714906693,
+            0.007306283339858055,
+            0.006303549278527498,
+            0.0255423653870821,
+            -0.02200544998049736,
+            -0.00387875665910542,
+            0.043974436819553375,
+            -0.04933450371026993,
+            0.009516855701804161,
+            -0.0050455741584300995,
+            -0.009197804145514965,
+            -0.015050122514367104,
+            0.0315040722489357,
+            0.009653591550886631,
+            -0.0064721908420324326,
+            0.015761151909828186,
+            0.01950773037970066,
+            0.0006551953847520053,
+            0.016326328739523888,
+            -0.020528696477413177,
+            -0.034238800406455994,
+            0.030027318745851517,
+            0.0009104367345571518,
+            -0.008144932799041271,
+            0.0414949469268322,
+            0.009945295751094818,
+            0.010975377634167671,
+            0.0034252475015819073,
+            -0.007392882835119963,
+            -0.026125773787498474,
+            0.00699178921058774,
+            0.023226961493492126,
+            0.025925226509571075,
+            0.011768448166549206,
+            0.020419307053089142,
+            -0.006025518756359816,
+            0.008286227472126484,
+            0.04732903465628624,
+            0.008760247379541397,
+            0.006900631822645664,
+            0.042588841170072556,
+            0.035369157791137695,
+            -0.02484956756234169,
+            -0.020984483882784843,
+            -0.00975386518985033,
+            0.022807635366916656,
+            0.0006956465658731759,
+            0.02054692804813385,
+            0.011221502907574177,
+            -0.0035665417090058327,
+            -0.005460341461002827,
+            0.030501339584589005,
+            -0.014393787831068039,
+            0.002228803699836135,
+            -0.0006791242631152272,
+            -0.017392873764038086,
+            -0.003037827555090189,
+            0.004402913153171539,
+            -0.020437538623809814,
+            0.009279845282435417,
+            0.02422969415783882,
+            0.00045806702109985054,
+            0.018979016691446304,
+            0.019124869257211685,
+            0.006358243990689516,
+            -0.0029717383440583944,
+            -0.024302620440721512,
+            -0.05061070993542671,
+            0.021257957443594933,
+            0.019343646243214607,
+            0.0026891496963799,
+            0.0029238807037472725,
+            0.012944381684064865,
+            0.024685483425855637,
+            -0.00985413882881403,
+            -0.02032814919948578,
+            0.03773925453424454,
+            -0.02319049835205078,
+            -0.008824057877063751,
+            -0.03159523010253906,
+            0.006791242398321629,
+            -0.023500433191657066,
+            -0.01207838486880064,
+            -0.020164065062999725,
+            -0.0066317166201770306,
+            0.0058340877294540405,
+            0.002695986535400152,
+            -0.0036235153675079346,
+            -0.012552403844892979,
+            0.001857336494140327,
+            0.004516860004514456,
+            -0.031759314239025116,
+            -0.02123972587287426,
+            -0.027456674724817276,
+            -0.0253418181091547,
+            -0.006207833997905254,
+            -0.02242477424442768,
+            0.003976751118898392,
+            -0.01965358294546604,
+            -0.046818554401397705,
+            -0.008546026423573494,
+            -0.010583399794995785,
+            0.0036736519541591406,
+            -0.006367359776049852,
+            0.014958965592086315,
+            -0.03157699853181839,
+            0.0030606170184910297,
+            0.003632631152868271,
+            -0.032105714082717896,
+            0.011896069161593914,
+            -0.011750216595828533,
+            -0.005565172526985407,
+            -0.00603919243440032,
+            0.0063400124199688435,
+            0.019471267238259315,
+            -0.032105714082717896,
+            -0.010391968302428722,
+            -0.028550565242767334,
+            0.024594325572252274,
+            0.004010935313999653,
+            0.014557871967554092,
+            0.006125791929662228,
+            -0.027365516871213913,
+            -0.02275294065475464,
+            -0.03713761270046234,
+            -0.02816770412027836,
+            0.010975377634167671,
+            0.02282586693763733,
+            -0.0024133978877216578,
+            0.016390139237046242,
+            -0.0016271634958684444,
+            0.024138538166880608,
+            0.010273464024066925,
+            -0.0008152909576892853,
+            -0.029024586081504822,
+            0.009644475765526295,
+            0.00855514220893383,
+            -0.04350041598081589,
+            -0.030009087175130844,
+            0.023026414215564728,
+            -0.013400170020759106,
+            0.009945295751094818,
+            0.021494966000318527,
+            0.017839545384049416,
+            0.026526866480708122,
+            0.006941652856767178,
+            0.027748379856348038,
+            -0.0115223228931427,
+            -0.016973547637462616,
+            0.031194137409329414,
+            0.015551489777863026,
+            -0.012725603766739368,
+            0.02712850645184517,
+            0.01391065213829279,
+            0.0023792139254510403,
+            0.008318132720887661,
+            0.021494966000318527,
+            0.005305373575538397,
+            0.0013343195896595716,
+            0.04977206140756607,
+            0.01817682944238186,
+            -0.008842289447784424,
+            -0.019525961950421333,
+            0.007429345976561308,
+            0.00982679147273302,
+            -0.007338188122957945,
+            0.021987218409776688,
+            0.00015268901188392192,
+            -0.04503186419606209,
+            0.01824975572526455,
+            0.019124869257211685,
+            0.024065611883997917,
+            0.0017057869117707014,
+            0.008277111686766148,
+            -0.027930693700909615,
+            0.00732451444491744,
+            -0.01023700088262558,
+            0.01985413022339344,
+            0.008381943218410015,
+            0.03226979821920395,
+            -0.03886960819363594,
+            0.020929789170622826,
+            -0.001477892859838903,
+            0.002021420281380415,
+            0.02601638436317444,
+            -0.020109370350837708,
+            -0.029735615476965904,
+            0.0013912931317463517,
+            -0.028678186237812042,
+            -0.011896069161593914,
+            0.004143113736063242,
+            0.015688225626945496,
+            0.004525975789874792,
+            -0.013719221577048302,
+            0.01617136225104332,
+            -0.008117586374282837,
+            -0.01589788869023323,
+            -0.004799448419362307,
+            -0.0009816535748541355,
+            0.043208710849285126,
+            0.016882391646504402,
+            0.013327243737876415,
+            -0.024466704577207565,
+            -0.03828619793057442,
+            -0.0034753840882331133,
+            -0.00029070733580738306,
+            0.03573378548026085,
+            0.007005462888628244,
+            0.007055599708110094,
+            -0.012233352288603783,
+            -0.011804911307990551,
+            0.0008768223342485726,
+            -0.016262518242001534,
+            0.0013081118231639266,
+            -0.011431165039539337,
+            -0.011494975537061691,
+            -0.004047398455440998,
+            0.004104371648281813,
+            0.010255232453346252,
+            0.013974462635815144,
+            0.012935265898704529,
+            0.006540559232234955,
+            -0.0032953477930277586,
+            -0.012689140625298023,
+            -0.0116225965321064,
+            -0.007488598581403494,
+            0.006271644029766321,
+            0.031066516414284706,
+            0.0032657217234373093,
+            -0.007023694459348917,
+            0.0047265226021409035,
+            0.032999057322740555,
+            -0.011504091322422028,
+            0.001646534539759159,
+            0.017474915832281113,
+            -0.02497718669474125,
+            0.016973547637462616,
+            -0.0008699854952283204,
+            0.015150396153330803,
+            0.006139465607702732,
+            0.015013659372925758,
+            -0.013518675230443478,
+            0.0009047393687069416,
+            0.011886953376233578,
+            0.008076565340161324,
+            -0.04886048287153244,
+            -0.00996352732181549,
+            0.009170456789433956,
+            -0.02173197641968727,
+            -0.01505923829972744,
+            -0.002342750784009695,
+            -0.0019188678124919534,
+            0.013345475308597088,
+            -0.017292600125074387,
+            0.014111199416220188,
+            0.03704645484685898,
+            -0.015086585655808449,
+            -0.02178667113184929,
+            0.008983583189547062,
+            0.02658156119287014,
+            0.013655411079525948,
+            -0.004981763660907745,
+            -0.008842289447784424,
+            0.014721955172717571,
+            0.009325424209237099,
+            -0.02116679958999157,
+            -0.013454864732921124,
+            0.040838614106178284,
+            -0.013044655323028564,
+            -0.016253402456641197,
+            0.023336350917816162,
+            -0.013309012167155743,
+            -0.02317226678133011,
+            0.017520494759082794,
+            0.014284398406744003,
+            -0.0061531392857432365,
+            0.020437538623809814,
+            0.0220783744007349,
+            0.003309021471068263,
+            -0.004156787414103746,
+            0.019945286214351654,
+            -0.006431169807910919,
+            -0.015350942499935627,
+            -0.007319956552237272,
+            -0.011303544975817204,
+            -0.012479477562010288,
+            0.014512293040752411,
+            0.008008196949958801,
+            -0.01928895153105259,
+            -0.005893339868634939,
+            -0.020638084039092064,
+            -0.004149950575083494,
+            0.04627160727977753,
+            -0.028222398832440376,
+            -0.0023336349986493587,
+            0.04051044583320618,
+            -0.003101638052612543,
+            -0.04470369592308998,
+            -0.01864173263311386,
+            0.007762071210891008,
+            -0.01752961054444313,
+            -0.009024604223668575,
+            -0.0422242097556591,
+            -0.005820414051413536,
+            0.020492233335971832,
+            0.015250669792294502,
+            -0.015697341412305832,
+            0.010082032531499863,
+            -0.020455770194530487,
+            0.06388325989246368,
+            0.0004905419191345572,
+            -0.009047393687069416,
+            -0.01092979870736599,
+            -0.02650863490998745,
+            0.011968995444476604,
+            0.02650863490998745,
+            0.014503177255392075,
+            0.012543288059532642,
+            0.022041911259293556,
+            0.02047400176525116,
+            0.012069269083440304,
+            0.025852300226688385,
+            0.02123972587287426,
+            0.030902432277798653,
+            0.024266157299280167,
+            -0.0003660548245534301,
+            0.01807655580341816,
+            -0.008509563282132149,
+            -0.04656331241130829,
+            -0.008573373779654503,
+            -0.003154053585603833,
+            0.00038371660048142076,
+            -0.011704638600349426,
+            -0.008855962194502354,
+            -0.00014143674343358725,
+            -0.020528696477413177,
+            -0.019416572526097298,
+            -0.041130319237709045,
+            -0.006490422412753105,
+            0.0028919754549860954,
+            0.03447581082582474,
+            0.011677291244268417,
+            -0.0007338188006542623,
+            -0.01835002936422825,
+            0.018577922135591507,
+            0.017228789627552032,
+            0.010145843029022217,
+            0.010692788287997246,
+            -0.00012954352132510394,
+            -0.014904270879924297,
+            0.050428394228219986,
+            0.021476734429597855,
+            0.0075159454718232155,
+            -0.027219664305448532,
+            -0.01492250245064497,
+            0.004343660548329353,
+            -0.005255236756056547,
+            -0.008477658964693546,
+            -0.0130537711083889,
+            -0.019270719960331917,
+            0.0033272530417889357,
+            0.014949849806725979,
+            -0.018304450437426567,
+            -0.015961699187755585,
+            0.003577936440706253,
+            0.003413852769881487,
+            0.024794872850179672,
+            -0.006148581393063068,
+            0.015907004475593567,
+            -0.012762066908180714,
+            -0.02025522291660309,
+            0.028422946110367775,
+            0.02061985246837139,
+            0.007857786491513252,
+            0.007392882835119963,
+            -0.006554232910275459,
+            0.006385591346770525,
+            -0.0011622596066445112,
+            0.012707372196018696,
+            -0.03810388594865799,
+            0.029370984062552452,
+            -0.004181855823844671,
+            5.131177022121847e-05,
+            -0.012278931215405464,
+            0.0220783744007349,
+            0.006517769768834114,
+            -0.0004099244251847267,
+            0.007739281747490168,
+            0.0025227870792150497,
+            -0.010419315658509731,
+            -0.01582496240735054,
+            0.002310845535248518,
+            0.036535974591970444,
+            -0.0031654483173042536,
+            0.016973547637462616,
+            -0.0038559671957045794,
+            0.02255239523947239,
+            0.007926154881715775,
+            -0.022807635366916656,
+            -0.023281656205654144,
+            -0.006025518756359816,
+            0.0015929794171825051,
+            0.03197809308767319,
+            -0.016134899109601974,
+            -0.03325429931282997,
+            0.022789403796195984,
+            0.01711028441786766,
+            -0.020145833492279053,
+            0.03578848019242287,
+            0.013454864732921124,
+            -0.0035095682833343744,
+            0.032999057322740555,
+            -0.0033454846125096083,
+            0.0018664522795006633,
+            0.009863254614174366,
+            0.014630797319114208,
+            0.0032360954210162163,
+            -0.012561519630253315,
+            -0.00414767162874341,
+            -0.009047393687069416,
+            -0.01595258340239525,
+            -0.006390149239450693,
+            0.004345939494669437,
+            -0.025943458080291748,
+            -0.0025729236658662558,
+            -0.0008483355632051826,
+            -0.001078508561477065,
+            0.015396521426737309,
+            0.01283499225974083,
+            -0.017119400203227997,
+            0.020729241892695427,
+            -0.007784860674291849,
+            -0.025906994938850403,
+            0.02393799088895321,
+            0.015405637212097645,
+            -0.00026165085728280246,
+            0.005268910434097052,
+            0.036955296993255615,
+            0.02395622245967388,
+            -0.009845023043453693,
+            -2.443629455228802e-06,
+            -0.001749086775816977,
+            0.009407466277480125,
+            -0.005979939829558134,
+            0.0009018906857818365,
+            0.00822241697460413,
+            0.0027005444280803204,
+            0.024375546723604202,
+            -0.008017312735319138,
+            -0.01575203612446785,
+            0.005838645622134209,
+            -0.017292600125074387,
+            0.01137647032737732,
+            -0.0018527786014601588,
+            -0.006567906588315964,
+            -0.01357336901128292,
+            0.008336364291608334,
+            0.014484945684671402,
+            0.047292571514844894,
+            0.014092967845499516,
+            0.0056153093464672565,
+            -0.017247021198272705,
+            0.01283499225974083,
+            0.02783953584730625,
+            0.001371922204270959,
+            0.012069269083440304,
+            -0.02811300940811634,
+            0.021968986839056015,
+            -0.002764354692772031,
+            0.003744299290701747,
+            0.0065998113714158535,
+            -0.0012978565646335483,
+            -0.0001713478413876146,
+            0.03440288454294205,
+            0.020145833492279053,
+            0.021950755268335342,
+            0.08576108515262604,
+            0.009644475765526295,
+            -0.016927970573306084,
+            -0.010774830356240273,
+            -0.0032429322600364685,
+            -0.028678186237812042,
+            -0.013454864732921124,
+            0.044521380215883255,
+            -0.004936185199767351,
+            0.004553323145955801,
+            -0.00706471549347043,
+            -0.0006426612380892038,
+            -0.013655411079525948,
+            0.009462160989642143,
+            0.010391968302428722,
+            0.0036508627235889435,
+            -0.0020054676569998264,
+            -0.0018641733331605792,
+            -0.023208729922771454,
+            -0.003976751118898392,
+            -0.025177733972668648,
+            -0.0072287991642951965,
+            -0.0035938890650868416,
+            0.011431165039539337,
+            0.0018687312258407474,
+            0.023482201620936394,
+            -1.8819062461261638e-05,
+            -0.007885133847594261,
+            0.00307201175019145,
+            0.017985397949814796,
+            -0.020382843911647797,
+            0.010018222033977509,
+            0.019179562106728554,
+            -0.0031358220148831606,
+            -0.0007053320878185332,
+            0.0021444829180836678,
+            -0.008500447496771812,
+            -0.009626244194805622,
+            -0.012306278571486473,
+            0.02415676973760128,
+            -0.006517769768834114,
+            -0.01810390315949917,
+            -0.010884219780564308,
+            0.00918413046747446,
+            0.000970258901361376,
+            0.01864173263311386,
+            0.00678668450564146,
+            -0.005460341461002827,
+            0.014886039309203625,
+            -0.01183225866407156,
+            -0.006927979178726673,
+            -0.00954420305788517,
+            0.010774830356240273,
+            -0.009726517833769321,
+            0.02242477424442768,
+            -0.013473096303641796,
+            0.009115762077271938,
+            -0.01881493255496025,
+            0.0065587908029556274,
+            0.00035124170244671404,
+            -0.024448473006486893,
+            0.011613480746746063,
+            -0.0036690940614789724,
+            -0.004936185199767351,
+            -0.008322690613567829,
+            0.005806740373373032,
+            0.015578837133944035,
+            -0.016982663422822952,
+            0.00034810815122909844,
+            -0.014138546772301197,
+            -0.02776661142706871,
+            -0.006134907715022564,
+            0.006955326069146395,
+            -0.008017312735319138,
+            -0.02464902028441429,
+            0.02054692804813385,
+            0.014594335108995438,
+            -0.02499541826546192,
+            -0.0005406786222010851,
+            -0.0014676376013085246,
+            -0.01723790541291237,
+            -0.025560596957802773,
+            0.0240109171718359,
+            -0.003445758018642664,
+            -0.007525061257183552,
+            0.020729241892695427,
+            -0.0003711824247147888,
+            -0.03551500663161278,
+            0.02929805777966976,
+            -0.004589786287397146,
+            0.00822241697460413,
+            0.022114837542176247,
+            0.003903825068846345,
+            -0.0007594568887725472,
+            0.006800358183681965,
+            0.02206014283001423,
+            0.008436637930572033,
+            -0.02756606414914131,
+            0.005164078902453184,
+            -0.014557871967554092,
+            -0.0027552389074116945,
+            -0.0035391945857554674,
+            -0.015907004475593567,
+            -0.03402002155780792,
+            0.0066590639762580395,
+            0.011531438678503036,
+            0.011358239687979221,
+            0.004120324272662401,
+            0.0054557835683226585,
+            0.018687311559915543,
+            0.0003073721018154174,
+            0.02082039974629879,
+            -0.0021365066058933735,
+            0.016581570729613304,
+            0.047365497797727585,
+            0.00021920558356214315,
+            -0.03394709527492523,
+            0.0007190057076513767,
+            0.00652688555419445,
+            -0.005200542043894529,
+            0.0028691859915852547,
+            0.03680944815278053,
+            0.01708293706178665,
+            -0.021968986839056015,
+            0.014010925777256489,
+            0.0013833168195560575,
+            -0.013208738528192043,
+            -0.019525961950421333,
+            0.0011759332846850157,
+            -0.02670918218791485,
+            0.007584313862025738,
+            -0.0025888762902468443,
+            -0.009229708462953568,
+            -0.020948020741343498,
+            -0.004721964709460735,
+            0.043062858283519745,
+            0.019307183101773262,
+            0.009001814760267735,
+            -0.001074520405381918,
+            -0.011048302985727787,
+            -0.03817681223154068,
+            0.010355505160987377,
+            0.029535068199038506,
+            -0.0015553769189864397,
+            0.006080213002860546,
+            -0.017000894993543625,
+            0.009589781053364277,
+            -0.0077347238548099995,
+            0.006335454527288675,
+            0.011467628180980682,
+            0.013409285806119442,
+            -0.0021102989558130503,
+            0.011823142878711224,
+            0.009762980975210667,
+            0.01006380096077919,
+            -0.0406927615404129,
+            0.007338188122957945,
+            0.0057429298758506775,
+            0.008568815886974335,
+            -0.00844575371593237,
+            0.016061972826719284,
+            -0.01970827765762806,
+            0.016326328739523888,
+            -0.03290789946913719,
+            -0.003418410662561655,
+            0.015733804553747177,
+            -0.01829533465206623,
+            -0.02443024143576622,
+            0.007702818606048822,
+            -0.0030287117697298527,
+            0.005984497722238302,
+            -0.006476748734712601,
+            0.00960801262408495,
+            -0.04557880759239197,
+            0.011075650341808796,
+            -0.02317226678133011,
+            -0.005578846205025911,
+            0.03617134317755699,
+            -0.004854143131524324,
+            -0.020291686058044434,
+            -0.00022817892022430897,
+            0.019489498808979988,
+            -0.020729241892695427,
+            -0.04350041598081589,
+            0.012725603766739368,
+            -0.014065620489418507,
+            0.013272549025714397,
+            -0.015843193978071213,
+            -0.007885133847594261,
+            0.015606184490025043,
+            0.013418401591479778,
+            0.009434813633561134,
+            -0.0028144915122538805,
+            -0.000781106820795685,
+            0.016034625470638275,
+            -0.03374655172228813,
+            0.006166812963783741,
+            -0.018723774701356888,
+            -0.03941655531525612,
+            -0.0012499988079071045,
+            0.023755675181746483,
+            -0.0042205979116261005,
+            0.02408384345471859,
+            0.01467637624591589,
+            -0.030063781887292862,
+            -0.02109387330710888,
+            -0.014484945684671402,
+            0.039671797305345535,
+            0.007971733808517456,
+            -0.023281656205654144,
+            -0.0027871441561728716,
+            -0.0007218543905764818,
+            0.014092967845499516,
+            -0.017985397949814796,
+            -0.0030879643745720387,
+            0.004181855823844671,
+            -0.004703733138740063,
+            -0.005975381936877966,
+            -0.03927070274949074,
+            0.012570635415613651,
+            0.019179562106728554,
+            -0.012935265898704529,
+            0.006271644029766321,
+            0.012844108045101166,
+            -0.01859615370631218,
+            0.012816760689020157,
+            -0.004300360567867756,
+            0.02796715684235096,
+            0.01680946536362171,
+            0.014594335108995438,
+            -0.009498624131083488,
+            -0.010993609204888344,
+            0.0011394702596589923,
+            -0.0007719910936430097,
+            -0.011285313405096531,
+            0.004767543636262417,
+            -0.003935730084776878,
+            0.020018212497234344,
+            0.017839545384049416,
+            0.029954394325613976,
+            -0.019179562106728554,
+            0.012798530049622059,
+            -0.03664536401629448,
+            0.00568823516368866,
+            0.0010044430382549763,
+            -0.00441430788487196,
+            -0.007739281747490168,
+            0.020236991345882416,
+            -0.01965358294546604,
+            0.0046672699972987175,
+            -0.011577017605304718,
+            -0.026599792763590813,
+            0.0009919088333845139,
+            0.007438461761921644,
+            -0.015961699187755585,
+            0.0018926600459963083,
+            -0.013546021655201912,
+            -0.01985413022339344,
+            -0.007497713901102543,
+            0.026563329622149467,
+            -0.008413848467171192,
+            -0.0005198832950554788,
+            -0.0005409634904935956,
+            -0.01391065213829279,
+            0.007333630230277777,
+            -0.003530078800395131,
+            -0.006567906588315964,
+            0.0017080658581107855,
+            -0.02428438887000084,
+            -0.007443019654601812,
+            0.005041016265749931,
+            0.02339104562997818,
+            0.015551489777863026,
+            -0.018441185355186462,
+            -0.024029148742556572,
+            -0.002853233367204666,
+            0.026526866480708122,
+            0.014247936196625233,
+            -0.026891497895121574,
+            0.01846853271126747,
+            -0.003641746938228607,
+            0.020638084039092064,
+            0.006216949783265591,
+            0.006618042942136526,
+            -0.0032657217234373093,
+            -0.008199627511203289,
+            0.007119410205632448,
+            -0.009525971487164497,
+            -0.02138557657599449,
+            -0.009735633619129658,
+            0.015496795065701008,
+            -0.010647210292518139,
+            0.003821783233433962,
+            0.000417046103393659,
+            -0.015916120260953903,
+            0.006289875600486994,
+            -0.009790328331291676,
+            0.015432984568178654,
+            0.011996342800557613,
+            0.00036747913691215217,
+            0.00036235153675079346,
+            0.030483108013868332,
+            -0.012297162786126137,
+            -0.007917039096355438,
+            0.00522333150729537,
+            -0.009047393687069416,
+            0.004010935313999653,
+            -0.007834997028112411,
+            -0.03241565078496933,
+            -0.0031859588343650103,
+            0.014311745762825012,
+            0.018605269491672516,
+            -0.01044666301459074,
+            -0.007958060130476952,
+            0.028058314695954323,
+            0.006285317707806826,
+            -0.007306283339858055,
+            -0.005018226802349091,
+            0.04718318209052086,
+            -0.0003959659079555422,
+            0.020984483882784843,
+            0.009535087272524834,
+            0.003142658853903413,
+            -0.004181855823844671,
+            -0.0003748857125174254,
+            0.008846846409142017,
+            -0.003760251682251692,
+            0.0021718302741646767,
+            -0.012415667995810509,
+            0.013254317454993725,
+            -0.0037078361492604017,
+            -0.028404714539647102,
+            0.008327248506247997,
+            -0.014603450894355774,
+            0.0001921431685332209,
+            -0.018377376720309258,
+            0.0027005444280803204,
+            0.012798530049622059,
+            -0.012561519630253315,
+            -0.0017525051953271031,
+            -0.007101178634911776,
+            -0.00947127677500248,
+            -0.0021923407912254333,
+            -0.0010414757998660207,
+            -0.02027345448732376,
+            -0.010893335565924644,
+            0.0008016172796487808,
+            -0.012762066908180714,
+            0.011886953376233578,
+            0.0002697695745155215,
+            -0.0006933676195330918,
+            0.00975386518985033,
+            -0.0009805141016840935,
+            0.019544193521142006,
+            0.00699178921058774,
+            0.0047265226021409035,
+            -0.006089328788220882,
+            0.004712848924100399,
+            0.020091138780117035,
+            -0.035770248621702194,
+            0.004095255862921476,
+            0.0005959429545328021,
+            0.009120319969952106,
+            -0.006604369264096022,
+            -0.016216941177845,
+            -0.002313124481588602,
+            -0.019598888233304024,
+            0.009899717755615711,
+            -0.01595258340239525,
+            -0.012680024839937687,
+            -0.000525010924320668,
+            0.0003065174969378859,
+            -0.004571554716676474,
+            0.006157697178423405,
+            0.011850490234792233,
+            0.004327707923948765,
+            0.0032133059576153755,
+            0.013937999494373798,
+            -0.020018212497234344,
+            0.004459886346012354,
+            -0.007944386452436447,
+            -0.0019747018814086914,
+            0.015195975080132484,
+            -0.0020134439691901207,
+            0.0006033494719304144,
+            -0.014895155094563961,
+            -0.002074975287541747,
+            0.005765719339251518,
+            -0.0012830435298383236,
+            -0.0036987203639000654,
+            0.013381938450038433,
+            -0.004958974663168192,
+            0.00013353166286833584,
+            -0.008587047457695007,
+            -0.00039767511771060526,
+            -0.004949858877807856,
+            0.006467632949352264,
+            -0.016982663422822952,
+            0.009735633619129658,
+            0.01721055805683136,
+            0.0061440235003829,
+            -0.002125111874192953,
+            -0.003600725904107094,
+            -0.00301959621720016,
+            0.03309021517634392,
+            0.006704642903059721,
+            0.02060162089765072,
+            0.029334520921111107,
+            0.008896983228623867,
+            -0.0037853200919926167,
+            0.0013092512963339686,
+            -0.008331806398928165,
+            -0.007456693332642317,
+            -0.0230446457862854,
+            -0.013299896381795406,
+            0.01464902888983488,
+            -0.006084770895540714,
+            0.012470361776649952,
+            -5.740081178373657e-05,
+            0.007383767049759626,
+            -0.00607109721750021,
+            -0.01896078512072563,
+            -0.030902432277798653,
+            -0.002962622558698058,
+            0.022114837542176247,
+            -0.005405646748840809,
+            -0.026964424178004265,
+            0.0004817110311705619,
+            0.010474010370671749,
+            -0.018094787374138832,
+            -0.015669994056224823,
+            0.010273464024066925,
+            -0.007429345976561308,
+            0.01886962726712227,
+            -0.008641742169857025,
+            0.017739271745085716,
+            -0.0009041696321219206,
+            -0.008641742169857025,
+            0.022114837542176247,
+            -0.024339083582162857,
+            0.013299896381795406,
+            -0.0004179007082711905,
+            -0.0031677272636443377,
+            -0.0003310161118861288,
+            0.010528705082833767,
+            -0.010765714570879936,
+            -0.0184503011405468,
+            0.005478573031723499,
+            -0.01346398051828146,
+            0.009352771565318108,
+            -0.008896983228623867,
+            0.011148576624691486,
+            0.02470371499657631,
+            -0.014475829899311066,
+            0.004977205768227577,
+            0.007206009700894356,
+            0.0012534172274172306,
+            0.0014949849573895335,
+            -0.005793066695332527,
+            0.013199622742831707,
+            0.0069781155325472355,
+            -0.010647210292518139,
+            0.02692796103656292,
+            0.010993609204888344,
+            -0.012752951122820377,
+            -0.0035847732797265053,
+            -0.007301725447177887,
+            -0.018796700984239578,
+            -0.000998745672404766,
+            -0.018997248262166977,
+            -0.011048302985727787,
+            -0.008587047457695007,
+            -0.011112113483250141,
+            0.0019507730612531304,
+            -0.005528709385544062,
+            -0.01259798277169466,
+            -0.0038377356249839067,
+            0.011804911307990551,
+            -0.006704642903059721,
+            -0.03318137302994728,
+            0.01812213473021984,
+            0.019762972369790077,
+            0.008140374906361103,
+            -0.03792157024145126,
+            0.019872359931468964,
+            0.00757975596934557,
+            -0.002256151055917144,
+            -0.012853223830461502,
+            0.0012841830030083656,
+            -0.01896078512072563,
+            0.013263433240354061,
+            -0.019872359931468964,
+            -0.001451685093343258,
+            0.0016818580916151404,
+            0.007958060130476952,
+            0.005100268870592117,
+            0.022880561649799347,
+            -0.005195984151214361,
+            -0.004268455319106579,
+            0.005478573031723499,
+            0.0038103885017335415,
+            -0.0025524133816361427,
+            0.030829505994915962,
+            -0.01051958929747343,
+            -0.023154035210609436,
+            -0.002796259941533208,
+            0.014521408826112747,
+            -0.009735633619129658,
+            -0.015387405641376972,
+            0.020437538623809814,
+            -0.01797628216445446,
+            0.0022401984315365553,
+            0.014731070958077908,
+            0.004284407943487167,
+            0.030720116570591927,
+            0.01290791854262352,
+            -0.005346394143998623,
+            -0.000578565988689661,
+            0.00010262354044243693,
+            0.0010055825114250183,
+            0.011613480746746063,
+            -0.004895164165645838,
+            -0.004762985743582249,
+            -0.005815856158733368,
+            0.012725603766739368,
+            -9.649888670537621e-05,
+            -0.0031996325124055147,
+            -0.00015781662659719586,
+            -0.015414752997457981,
+            0.014876923523843288,
+            0.0026731970719993114,
+            0.0036280732601881027,
+            0.007420230191200972,
+            -6.366789602907375e-05,
+            -0.0059617082588374615,
+            -0.01384684257209301,
+            0.006567906588315964,
+            -0.011340008117258549,
+            -0.0115223228931427,
+            -0.028696417808532715,
+            0.014949849806725979,
+            -0.02040107548236847,
+            0.00958066526800394,
+            -0.002073835814371705,
+            0.009863254614174366,
+            -0.02054692804813385,
+            -0.015788499265909195,
+            -0.009325424209237099,
+            0.012607098557054996,
+            -0.009662707336246967,
+            0.02144027128815651,
+            0.007174104452133179,
+            0.015770267695188522,
+            0.021257957443594933,
+            0.024247925728559494,
+            0.004867816809564829,
+            0.01629898138344288,
+            -0.02818593569099903,
+            -0.0007919318159110844,
+            -0.007748397532850504,
+            0.012014574371278286,
+            -0.0036166785284876823,
+            0.006394707132130861,
+            0.013108465820550919,
+            0.01723790541291237,
+            -0.010938914492726326,
+            0.002903370186686516,
+            -2.533540828153491e-05,
+            -0.00958066526800394,
+            -0.00711485231295228,
+            -0.017784850671887398,
+            -0.01859615370631218,
+            0.01631721295416355,
+            0.007511387579143047,
+            0.012780298478901386,
+            0.016052857041358948,
+            0.016289865598082542,
+            -0.0023199613206088543,
+            0.0021091594826430082,
+            -0.01943480409681797,
+            -0.0025934341829270124,
+            -0.012917034327983856,
+            -0.0013593879994004965,
+            0.025560596957802773,
+            -0.0017069263849407434,
+            0.017939819023013115,
+            0.007433903869241476,
+            0.034731052815914154,
+            0.0010015943553298712,
+            0.006927979178726673,
+            0.005278026219457388,
+            0.015633530914783478,
+            0.0027119391597807407,
+            0.02386506460607052,
+            -0.02040107548236847,
+            0.0013001355109736323,
+            0.03691883385181427,
+            0.022862330079078674,
+            -0.008195069618523121,
+            -0.01790335588157177,
+            -0.01943480409681797,
+            0.016681844368577003,
+            -0.03019140288233757,
+            -0.00757975596934557,
+            0.02386506460607052,
+            0.00624885456636548,
+            0.01319050695747137,
+            -0.0006648809066973627,
+            -0.018650848418474197,
+            0.012889686971902847,
+            0.008195069618523121,
+            -0.009498624131083488,
+            0.004423423204571009,
+            -0.006381033454090357,
+            -0.009489508345723152,
+            -0.0027825862634927034,
+            -0.0027871441561728716,
+            -0.017474915832281113,
+            -0.00865541584789753,
+            0.0004555032355710864,
+            -0.004541928414255381,
+            0.0021102989558130503,
+            -0.003470826428383589,
+            -0.0217137448489666,
+            0.0058340877294540405,
+            -0.012087500654160976,
+            -0.0025683660060167313,
+            0.0018128971569240093,
+            -0.009498624131083488,
+            -0.024667251855134964,
+            -0.00650409609079361,
+            -0.007142199203372002,
+            -0.005268910434097052,
+            0.01277118269354105,
+            -0.015460331924259663,
+            0.009257055819034576,
+            -0.014876923523843288,
+            0.023354582488536835,
+            -0.0005976521642878652,
+            0.008609836921095848,
+            0.013965346850454807,
+            -0.013454864732921124,
+            -0.011494975537061691,
+            -0.03442111611366272,
+            -0.0044781179167330265,
+            -0.020455770194530487,
+            -0.00565633038058877,
+            -0.00660892715677619,
+            0.01689150743186474,
+            0.0018197339959442616,
+            -0.017301715910434723,
+            0.01696443371474743,
+            0.004854143131524324,
+            -0.0013138091890141368,
+            -0.01561530027538538,
+            -0.0044484916143119335,
+            -9.236830373993143e-05,
+            -0.007452135439962149,
+            0.01956242509186268,
+            0.00161348981782794,
+            -0.002983133075758815,
+            0.0025136712938547134,
+            0.007803092245012522,
+            -0.007092062849551439,
+            -0.03389240428805351,
+            -0.012579751200973988,
+            -0.002392887370660901,
+            6.17094337940216e-05,
+            -0.007493156008422375,
+            0.003685046685859561,
+            0.014339093118906021,
+            -0.011185039766132832,
+            -0.007921596989035606,
+            -0.00013324679457582533,
+            0.009808559902012348,
+            0.015451216138899326,
+            -0.009334539994597435,
+            0.018094787374138832,
+            -0.01567910984158516,
+            0.01874200627207756,
+            0.0007093202439136803,
+            -0.002285777358338237,
+            -0.0022162694949656725,
+            0.010282579809427261,
+            -0.008473101072013378,
+            -0.006918863393366337,
+            -0.009211476892232895,
+            -0.008814942091703415,
+            -0.0073290723375976086,
+            0.0013046934036538005,
+            -0.020291686058044434,
+            -0.020528696477413177,
+            -0.006230622995644808,
+            0.0030013646464794874,
+            -0.006125791929662228,
+            0.011567901819944382,
+            -0.01211484707891941,
+            -0.014348208904266357,
+            0.014147662557661533,
+            0.007392882835119963,
+            0.0029216017574071884,
+            -0.017092052847146988,
+            -0.007274378091096878,
+            -0.007661798037588596,
+            0.008299901150166988,
+            -0.0031700062099844217,
+            0.034803979098796844,
+            -0.003343205666169524,
+            -0.010455778799951077,
+            0.003015038324519992,
+            -0.001371922204270959,
+            0.0058112982660532,
+            -0.016499528661370277,
+            0.005701908841729164,
+            0.0011172505328431726,
+            0.006814031861722469,
+            0.020528696477413177,
+            -0.0016294424422085285,
+            -0.012871455401182175,
+            0.008796710520982742,
+            0.010191421955823898,
+            0.002445303136482835,
+            -0.0016009557293727994,
+            -0.0016898344038054347,
+            -0.00801275484263897,
+            0.019945286214351654,
+            0.041604336351156235,
+            0.0035164051223546267,
+            0.01037373673170805,
+            0.009389234706759453,
+            -0.01641748659312725,
+            -0.009489508345723152,
+            0.014685492031276226,
+            -0.008436637930572033,
+            0.005683677736669779,
+            -0.006253412459045649,
+            0.019981749355793,
+            -0.011458512395620346,
+            -0.005232447292655706,
+            0.009981758892536163,
+            -0.003575657494366169,
+            0.014904270879924297,
+            0.006107560358941555,
+            -0.019817667081952095,
+            0.008600721135735512,
+            0.0014482666738331318,
+            -0.0035072893369942904,
+            -0.026818571612238884,
+            0.0024703715462237597,
+            0.002686870750039816,
+            0.009945295751094818,
+            -0.0232451930642128,
+            0.01252505648881197,
+            0.010145843029022217,
+            0.004908837843686342,
+            0.004651317372918129,
+            0.0037511358968913555,
+            -0.009329982101917267,
+            0.005369183607399464,
+            0.01923425681889057,
+            -0.008154048584401608,
+            -0.025852300226688385,
+            -0.0004278710694052279,
+            0.001534866401925683,
+            -0.010811293497681618,
+            -0.0058112982660532,
+            0.010528705082833767,
+            -0.009480392560362816,
+            -0.005337278358638287,
+            0.015569721348583698,
+            0.014138546772301197,
+            -0.012570635415613651,
+            0.02816770412027836,
+            -0.006203276105225086,
+            0.0061804866418242455,
+            0.025724679231643677,
+            -0.007365535479038954,
+            -0.012178657576441765,
+            -0.025141270831227303,
+            0.013245201669633389,
+            0.0010956006590276957,
+            0.019398340955376625,
+            -0.02747490629553795,
+            -0.021640818566083908,
+            -0.018359145149588585,
+            0.008359153755009174,
+            0.008208743296563625,
+            0.0027301707305014133,
+            0.02277117222547531,
+            -0.00683682132512331,
+            -0.0010357784340158105,
+            -0.0013331801164895296,
+            0.019270719960331917,
+            0.00732451444491744,
+            -0.0015758873196318746,
+            0.011686407029628754,
+            0.012789414264261723,
+            0.01567910984158516,
+            -0.011558786034584045,
+            -0.014594335108995438,
+            -0.014968081377446651,
+            0.00016992349992506206,
+            -0.01044666301459074,
+            -0.004863258916884661,
+            0.006513211876153946,
+            0.026271626353263855,
+            -0.021203262731432915,
+            0.010018222033977509,
+            0.006376475561410189,
+            0.002236780012026429,
+            -0.0016647659940645099,
+            -0.004175018984824419,
+            0.018113018944859505,
+            -0.017721040174365044,
+            0.00532816257327795,
+            -0.0010152680333703756,
+            0.0027073812671005726,
+            0.012470361776649952,
+            -0.023135803639888763,
+            -0.005565172526985407,
+            0.015341827645897865,
+            -0.004122603219002485,
+            -0.00947127677500248,
+            0.026052847504615784,
+            -0.03700999170541763,
+            0.021276189014315605,
+            0.015423868782818317,
+            -0.0024065610487014055,
+            -0.011896069161593914,
+            -0.01881493255496025,
+            0.0034024580381810665,
+            -0.027019118890166283,
+            -0.0018687312258407474,
+            0.005405646748840809,
+            -0.01267090905457735,
+            -0.010173190385103226,
+            0.009589781053364277,
+            0.011786679737269878,
+            -0.005515035707503557,
+            -0.009936179965734482,
+            -0.008003639057278633,
+            -0.009507739916443825,
+            -0.011996342800557613,
+            0.0057429298758506775,
+            -0.00898814108222723,
+            -0.003976751118898392,
+            0.0007685726741328835,
+            0.02678210847079754,
+            -4.931769581162371e-06,
+            -0.012661793269217014,
+            0.013892421498894691,
+            0.02541474439203739,
+            -0.009726517833769321,
+            -0.02235184796154499,
+            -0.008983583189547062,
+            -0.0009565852815285325,
+            -0.0037055572029203176,
+            -0.008263438008725643,
+            0.0009070183150470257,
+            -0.011804911307990551,
+            0.0040701874531805515,
+            0.003933451138436794,
+            -0.0009845022577792406,
+            -0.022151300683617592,
+            0.008331806398928165,
+            0.007999081164598465,
+            0.011440280824899673,
+            -0.0414949469268322,
+            -0.002910207025706768,
+            -0.012880571186542511,
+            -0.0042205979116261005,
+            -0.01429351419210434,
+            0.025232428684830666,
+            0.005847761407494545,
+            0.0065770223736763,
+            -0.02470371499657631,
+            0.008103912696242332,
+            0.022242458537220955,
+            -0.0014699165476486087,
+            0.0018698706990107894,
+            -0.01159524917602539,
+            -0.00257520261220634,
+            0.028751112520694733,
+            -0.007803092245012522,
+            0.00506836362183094,
+            -0.003101638052612543,
+            0.00027831559418700635,
+            0.004441654775291681,
+            0.013372822664678097,
+            0.0007588871521875262,
+            -0.00982679147273302,
+            0.014494061470031738,
+            0.00021607204689644277,
+            0.005756603553891182,
+            -0.0031107538379728794,
+            -0.01329078059643507,
+            0.0025432975962758064,
+            -0.009899717755615711,
+            -0.03144937753677368,
+            -0.01579761505126953,
+            -0.020510464906692505,
+            -0.013427517376840115,
+            -0.002880580723285675,
+            0.0334366150200367,
+            0.014776649884879589,
+            -0.014010925777256489,
+            -0.0012955776182934642,
+            -0.0034047369845211506,
+            0.027784842997789383,
+            0.011485859751701355,
+            -0.008030986413359642,
+            0.02020052820444107,
+            -0.0017593420343473554,
+            -0.00285551231354475,
+            0.018277103081345558,
+            -0.004343660548329353,
+            0.010802177712321281,
+            0.0024726504925638437,
+            -0.02421146258711815,
+            0.012743835337460041,
+            -0.002525066025555134,
+            0.001482450752519071,
+            0.010957146063446999,
+            -0.00960801262408495,
+            -0.01290791854262352,
+            0.0015417032409459352,
+            0.008523236960172653,
+            -0.012196889147162437,
+            -0.024375546723604202,
+            -0.0033477635588496923,
+            -0.008550584316253662,
+            -0.007948944345116615,
+            0.0016191871836781502,
+            0.014703723601996899,
+            -0.0004247375181876123,
+            0.006522327661514282,
+            -0.004936185199767351,
+            -0.014165894128382206,
+            -0.004694617353379726,
+            0.02144027128815651,
+            0.01866907998919487,
+            0.00954420305788517,
+            -4.436812378116883e-05,
+            0.016882391646504402,
+            -0.010255232453346252,
+            -0.005875108297914267,
+            -0.016927970573306084,
+            -0.02484956756234169,
+            0.0009457602864131331,
+            0.0106563251465559,
+            -0.024120306596159935,
+            0.0034844998735934496,
+            -0.048678167164325714,
+            -0.008700994774699211,
+            0.017174094915390015,
+            -0.01908840611577034,
+            0.005059247836470604,
+            0.00242023472674191,
+            -0.03041018173098564,
+            0.015268901363015175,
+            -0.019762972369790077,
+            0.01061986293643713,
+            -0.009311750531196594,
+            -0.02588876336812973,
+            0.016818581148982048,
+            -0.006677295546978712,
+            0.016335444524884224,
+            -0.009079298935830593,
+            -0.008304459042847157,
+            -0.0056426567025482655,
+            -0.006221507675945759,
+            0.00036690940032713115,
+            0.0006888097268529236,
+            0.001456242986023426,
+            0.0036736519541591406,
+            0.004899722058326006,
+            -0.01750226318836212,
+            -0.009945295751094818,
+            -0.01551502663642168,
+            -0.014895155094563961,
+            -0.01810390315949917,
+            0.009083856828510761,
+            -0.01520509086549282,
+            0.010027337819337845,
+            -0.0034252475015819073,
+            -0.017438452690839767,
+            0.007092062849551439,
+            -0.006809473969042301,
+            0.005232447292655706,
+            -0.005970824044197798,
+            -0.006595253478735685,
+            0.009899717755615711,
+            0.006905189715325832,
+            0.007871460169553757,
+            0.00706471549347043,
+            -0.012844108045101166,
+            -0.004193250555545092,
+            -0.003876477712765336,
+            0.03318137302994728,
+            0.0005891061155125499,
+            -0.01023700088262558,
+            -0.005788508802652359,
+            -1.0931792530755047e-05,
+            7.641572301508859e-05,
+            0.005086595192551613,
+            0.0007127386052161455,
+            0.016499528661370277,
+            -0.019197793677449226,
+            -0.0051367320120334625,
+            -0.0060209608636796474,
+            -0.005697350949048996,
+            -0.012534172274172306,
+            -0.016918854787945747,
+            0.002527344971895218,
+            0.01916133053600788,
+            0.009261613711714745,
+            0.01259798277169466,
+            -0.03110297955572605,
+            0.0029671804513782263,
+            0.021494966000318527,
+            -0.025706447660923004,
+            -0.010474010370671749,
+            -0.00793527066707611,
+            -0.028277093544602394,
+            -0.015551489777863026,
+            0.013855958357453346,
+            0.014311745762825012,
+            0.023208729922771454,
+            -0.017319947481155396,
+            -0.0007452135323546827,
+            -0.03274381533265114,
+            -0.0012978565646335483,
+            -0.0030811275355517864,
+            -0.008381943218410015,
+            -0.024375546723604202,
+            0.025834068655967712,
+            0.015979930758476257,
+            0.008559700101613998,
+            0.004355055280029774,
+            -0.000295692530926317,
+            0.024831335991621017,
+            -0.004131719004362822,
+            0.007629892788827419,
+            -0.001804920844733715,
+            0.017347294837236404,
+            -0.009553318843245506,
+            -0.012889686971902847,
+            0.005109384655952454,
+            -0.01068367250263691,
+            -7.24987912690267e-05,
+            -0.00688695814460516,
+            -0.027456674724817276,
+            -0.029443910345435143,
+            0.0053965309634804726,
+            0.0013400169555097818,
+            0.009872370399534702,
+            -0.022935256361961365,
+            0.018240639939904213,
+            -0.0066590639762580395,
+            -0.004744754172861576,
+            -0.0006774150533601642,
+            -0.0023906086571514606,
+            0.02720143273472786,
+            -0.02220599539577961,
+            -0.0026230604853481054,
+            -0.0015439821872860193,
+            -0.010027337819337845,
+            0.006595253478735685,
+            0.0008027567528188229,
+            0.013372822664678097,
+            0.016818581148982048,
+            -0.015004543587565422,
+            -0.04123970493674278,
+            -0.002021420281380415,
+            -0.035095684230327606,
+            -0.0077347238548099995,
+            -0.002716497052460909,
+            0.021768439561128616,
+            -0.006435727700591087,
+            0.037082917988300323,
+            -0.0014414298348128796,
+            -0.02464902028441429,
+            -0.0270738136023283,
+            0.01599816232919693,
+            0.023135803639888763,
+            -0.014457598328590393,
+            -0.0026549657341092825,
+            0.019525961950421333,
+            -0.004120324272662401,
+            0.01121238712221384,
+            0.012160426005721092,
+            0.015323596075177193,
+            -0.004138555843383074,
+            0.0018687312258407474,
+            0.008267995901405811,
+            -0.019690046086907387,
+            -0.028751112520694733,
+            -0.0036189574748277664,
+            0.00030025039450265467,
+            -0.015323596075177193,
+            0.0074475775472819805,
+            -0.01106653455644846,
+            -0.007060157600790262,
+            0.006016402970999479,
+            0.007876018062233925,
+            -0.007944386452436447,
+            0.025925226509571075,
+            -0.014193241484463215,
+            -0.002069277921691537,
+            0.023901527747511864,
+            0.0006523467018269002,
+            -0.016873275861144066,
+            -0.0047265226021409035,
+            0.0032452112063765526,
+            0.027948925271630287,
+            0.01679123379290104,
+            -0.007201451808214188,
+            -0.017848661169409752,
+            0.02443024143576622,
+            -0.011230618692934513,
+            -0.007629892788827419,
+            0.0001251992944162339,
+            0.01200545858591795,
+            0.01842295378446579,
+            -0.004234271589666605,
+            0.004813122097402811,
+            -0.0009480392327532172,
+            0.033491309732198715,
+            0.002347308676689863,
+            0.009453045204281807,
+            -0.01644483394920826,
+            0.009307192638516426,
+            -0.0023974452633410692,
+            -0.00021179902250878513,
+            -0.0021410644985735416,
+            -0.034731052815914154,
+            -0.01530536450445652,
+            -0.0042205979116261005,
+            0.003985866904258728,
+            -0.021914292126893997,
+            0.0041294400580227375,
+            0.001722879009321332,
+            -0.02665448747575283,
+            0.016216941177845,
+            -0.016918854787945747,
+            -0.00944392941892147,
+            0.017821313813328743,
+            -0.0070692733861505985,
+            -0.0073883249424397945,
+            -0.012206004932522774,
+            0.022716479375958443,
+            0.00975386518985033,
+            -0.008846846409142017,
+            0.009890601970255375,
+            -0.01482222881168127,
+            0.008158606477081776,
+            -0.0015451216604560614,
+            0.007834997028112411,
+            0.021185031160712242,
+            -0.017638999968767166,
+            0.017438452690839767,
+            0.00011950194311793894,
+            -0.009744749404489994,
+            -0.008523236960172653,
+            0.00436417106539011,
+            0.027037350460886955,
+            0.00035893311724066734,
+            0.017693692818284035,
+            0.007429345976561308,
+            0.03301728889346123,
+            -0.0034867788199335337,
+            0.004471281077712774,
+            0.012133078649640083,
+            0.016089320182800293,
+            -0.017174094915390015,
+            -0.013026423752307892,
+            0.0020795331802219152,
+            -0.009790328331291676,
+            0.0013650853652507067,
+            -0.009762980975210667,
+            -0.02165905013680458,
+            0.02623516321182251,
+            0.021531429141759872,
+            -0.017019126564264297,
+            0.0008016172796487808,
+            -0.008810384199023247,
+            -0.0351686105132103,
+            -0.018404724076390266,
+            0.0108568724244833,
+            0.015241554006934166,
+            0.0010813572444021702,
+            0.010309926234185696,
+            -0.01016407459974289,
+            -0.00021920558356214315,
+            -0.0036189574748277664,
+            -0.01723790541291237,
+            -0.01793070323765278,
+            -0.015405637212097645,
+            -0.009671823121607304,
+            -0.009029162116348743,
+            0.0039699142798781395,
+            -0.011303544975817204,
+            -0.006536001339554787,
+            0.004514581058174372,
+            0.004731080494821072,
+            -0.009735633619129658,
+            -0.011850490234792233,
+            0.013126697391271591,
+            -0.001399269443936646,
+            0.00046974659198895097,
+            0.016581570729613304,
+            -0.0035961680114269257,
+            -0.0213126502931118,
+            0.00387875665910542,
+            0.006025518756359816,
+            0.012515940703451633,
+            0.00429124478250742,
+            -0.010483126156032085,
+            0.01599816232919693,
+            0.00568823516368866,
+            -0.012333625927567482,
+            0.017146747559309006,
+            -0.022023679688572884,
+            -0.007538734935224056,
+            0.03296259418129921,
+            -0.013637179508805275,
+            0.009316308423876762,
+            0.0003042385505978018,
+            -0.006431169807910919,
+            0.02206014283001423,
+            0.026545098051428795,
+            0.01627163402736187,
+            0.00515040522441268,
+            0.012798530049622059,
+            0.00031363917514681816,
+            -0.0011656780261546373,
+            0.015843193978071213,
+            -0.0030765696428716183,
+            -0.026198700070381165,
+            -0.0025501344352960587,
+            0.0009018906857818365,
+            0.002495439723134041,
+            0.0029808541294187307,
+            -0.010628978721797466,
+            -0.005715582519769669,
+            -0.0010101404041051865,
+            -0.006226065568625927,
+            -0.0034092948772013187,
+            0.001910891616716981,
+            -0.011850490234792233,
+            0.0011998621048405766,
+            -0.02650863490998745,
+            0.00031335430685430765,
+            -0.012570635415613651,
+            -0.027912462130188942,
+            0.021075641736388206,
+            -0.005984497722238302,
+            0.011148576624691486,
+            0.022917024791240692,
+            0.019252488389611244,
+            -0.008792152628302574,
+            -0.004981763660907745,
+            0.0012978565646335483,
+            0.008673647418618202,
+            -0.014202357269823551,
+            -0.014594335108995438,
+            -0.009808559902012348,
+            -0.0019439362222328782,
+            -0.020218759775161743,
+            -0.021257957443594933,
+            -0.018231524154543877,
+            0.0010454639559611678,
+            0.01513216458261013,
+            0.0006540559115819633,
+            0.00459206523373723,
+            -0.016399255022406578,
+            0.00898814108222723,
+            -0.011030071415007114,
+            -0.011987227015197277,
+            0.003685046685859561,
+            0.004494070541113615,
+            -0.010793061926960945,
+            0.004712848924100399,
+            -0.005715582519769669,
+            -0.00301959621720016,
+            0.002584318397566676,
+            -0.014211473055183887,
+            -0.002613944699987769,
+            0.0006301270332187414,
+            0.025451207533478737,
+            0.0005329871783033013,
+            0.009772096760571003,
+            0.004045119509100914,
+            0.009188688360154629,
+            0.008267995901405811,
+            0.0003355739754624665,
+            0.0071695465594530106,
+            0.0014049668097868562,
+            -0.005401088856160641,
+            0.017620768398046494,
+            0.0011964436853304505,
+            -0.006586138159036636,
+            -0.007926154881715775,
+            -0.021403808146715164,
+            -0.003926614299416542,
+            0.0011098439572378993,
+            0.02180490270256996,
+            0.002579760504886508,
+            0.004731080494821072,
+            -0.013318127952516079,
+            -0.004443933721631765,
+            -0.004279850050806999,
+            -0.0008688460220582783,
+            0.0037830411456525326,
+            0.025651752948760986,
+            -0.004168182145804167,
+            -0.008819499984383583,
+            -0.01894255355000496,
+            -0.017711924389004707,
+            0.020145833492279053,
+            -0.006007287185639143,
+            0.006394707132130861,
+            -0.017420221120119095,
+            0.007789418566972017,
+            0.0064995381981134415,
+            0.018340913578867912,
+            -0.009717402048408985,
+            0.011704638600349426,
+            0.005246120970696211,
+            -0.0020077466033399105,
+            0.011959879659116268,
+            -0.0064995381981134415,
+            0.034038253128528595,
+            -0.014876923523843288,
+            0.009708286263048649,
+            0.0022048749960958958,
+            0.004213761072605848,
+            0.007844112813472748,
+            0.019070174545049667,
+            -0.004863258916884661,
+            -0.012762066908180714,
+            -0.005195984151214361,
+            -0.004863258916884661,
+            0.016800349578261375,
+            0.021841365844011307,
+            0.0046353647485375404,
+            -0.0008283948409371078,
+            0.0027415654622018337,
+            0.006171370856463909,
+            0.009726517833769321,
+            0.019070174545049667,
+            -0.01669096015393734,
+            0.005350952036678791,
+            -0.016353676095604897,
+            -0.009662707336246967,
+            0.00836826954036951,
+            -0.005843203514814377,
+            0.014147662557661533,
+            0.012616214342415333,
+            -0.000646079599391669,
+            -0.0004167612351011485,
+            0.015834078192710876,
+            -0.0021091594826430082,
+            0.006157697178423405,
+            0.012178657576441765,
+            -0.01100272499024868,
+            -0.001858475967310369,
+            -0.02680034004151821,
+            0.008053775876760483,
+            -0.023500433191657066,
+            -0.019471267238259315,
+            0.003286232240498066,
+            -0.025177733972668648,
+            0.005191426258534193,
+            0.018632616847753525,
+            0.01758430525660515,
+            -0.009143109433352947,
+            -0.010191421955823898,
+            -0.0009360748226754367,
+            0.02027345448732376,
+            -0.013555137440562248,
+            0.02032814919948578,
+            -0.002256151055917144,
+            0.004931627307087183,
+            0.007648124359548092,
+            -0.004412028938531876,
+            0.016216941177845,
+            -0.012734719552099705,
+            0.0024430241901427507,
+            -0.012588866986334324,
+            -0.0032702796161174774,
+            0.022151300683617592,
+            -0.02102094702422619,
+            0.027237895876169205,
+            0.009257055819034576,
+            -0.010109379887580872,
+            -0.0008483355632051826,
+            0.004177297931164503,
+            0.0215131975710392,
+            -0.03287143632769585,
+            -0.0010705322492867708,
+            -0.03239741921424866,
+            -0.001720600062981248,
+            -0.00011544257722562179,
+            0.03288966789841652,
+            -0.012251583859324455,
+            -0.019981749355793,
+            -0.002964901505038142,
+            0.006782126612961292,
+            -0.03305375203490257,
+            -0.001533726928755641,
+            0.007133083418011665,
+            -0.0037260677199810743,
+            0.01741110533475876,
+            0.0037807621993124485,
+            0.0022983115632086992,
+            -0.015186859294772148,
+            -0.012452131137251854,
+            0.0192160252481699,
+            0.016007278114557266,
+            -0.007351861800998449,
+            0.013509559445083141,
+            -0.022041911259293556,
+            -0.00624885456636548,
+            0.004491791594773531,
+            0.0030082014854997396,
+            -0.0008990420028567314,
+            0.034603431820869446,
+            0.018532343208789825,
+            -0.005829529836773872,
+            0.007456693332642317,
+            -0.0009639917989261448,
+            0.022096605971455574,
+            0.011841374449431896,
+            0.032452113926410675,
+            -0.00053782993927598,
+            0.03004555031657219,
+            -0.02858702838420868,
+            -0.00673199025914073,
+            -0.005259794648736715,
+            0.01412943098694086,
+            -0.025068344548344612,
+            0.015287132933735847,
+            -3.9169288356788456e-05,
+            0.028915196657180786,
+            0.009562434628605843,
+            0.0014608008787035942,
+            0.004250223748385906,
+            0.006344570312649012,
+            -0.0017376920441165566,
+            0.005241563078016043,
+            -0.0016909738769754767,
+            -0.026399245485663414,
+            -0.023445740342140198,
+            -0.0028122125659137964,
+            -0.018650848418474197,
+            0.014521408826112747,
+            -0.005770277231931686,
+            -0.006773010827600956,
+            -0.011795795522630215,
+            0.015688225626945496,
+            0.010638094507157803,
+            0.00916134100407362,
+            0.001238604192622006,
+            0.021185031160712242,
+            0.01832268200814724,
+            0.01577938348054886,
+            -0.003901546122506261,
+            0.0099908746778965,
+            -0.000999315408989787,
+            0.011485859751701355,
+            0.009699170477688313,
+            -0.011367355473339558,
+            0.020510464906692505,
+            -0.0032748375087976456,
+            -0.01415677834302187,
+            0.0075159454718232155,
+            -0.001373061677441001,
+            0.010638094507157803,
+            -0.02818593569099903,
+            0.0026754760183393955,
+            0.008591605350375175,
+            0.011412933468818665,
+            0.009917949326336384,
+            -0.01565176248550415,
+            -0.014895155094563961,
+            0.02450316771864891,
+            -0.005282584112137556,
+            0.01819506101310253,
+            0.0029762962367385626,
+            0.017912471666932106,
+            0.011914300732314587,
+            0.002582039451226592,
+            -0.021458502858877182,
+            -0.0015006823232397437,
+            0.0022686852607876062,
+            -0.0005760021740570664,
+            -0.008021870627999306,
+            -0.01054693665355444,
+            0.005597077775746584,
+            -0.036535974591970444,
+            0.004371007904410362,
+            -0.015031890943646431,
+            -0.02588876336812973,
+            0.004560159984976053,
+            -0.012935265898704529,
+            0.002186643425375223,
+            0.0032429322600364685,
+            0.02895165979862213,
+            -0.009999990463256836,
+            -0.02824063040316105,
+            1.6219646568060853e-05,
+            0.0030856854282319546,
+            -0.004113487433642149,
+            0.013309012167155743,
+            0.015788499265909195,
+            -0.0020054676569998264,
+            0.0026230604853481054,
+            0.042370062321424484,
+            -0.001320646028034389,
+            -0.006804916076362133,
+            0.00527346832677722,
+            -0.003721509827300906,
+            -0.012297162786126137,
+            -0.0003475384146440774,
+            0.008518679067492485,
+            0.011549670249223709,
+            -0.018587037920951843,
+            -0.00565633038058877,
+            0.010082032531499863,
+            0.030227866023778915,
+            0.017301715910434723,
+            0.013236085884273052,
+            -0.0024863239377737045,
+            0.005054689943790436,
+            -0.0018402443965896964,
+            -0.00023558546672575176,
+            -0.015578837133944035,
+            -0.029334520921111107,
+            -0.01970827765762806,
+            -0.0046991752460598946,
+            -0.01214219443500042,
+            -0.009908833540976048,
+            -0.011631712317466736,
+            0.0077347238548099995,
+            -0.0014608008787035942,
+            0.007014578673988581,
+            0.004357334226369858,
+            -0.010738367214798927,
+            0.010765714570879936,
+            0.022716479375958443,
+            0.017292600125074387,
+            -0.0018470812356099486,
+            0.01269825641065836,
+            -0.003842293517664075,
+            0.004008656367659569,
+            0.002835002029314637,
+            -0.010665440931916237,
+            -0.00978121254593134,
+            -0.0006865308387205005,
+            0.012415667995810509,
+            0.011349123902618885,
+            -0.01513216458261013,
+            0.002525066025555134,
+            0.011896069161593914,
+            0.010720135644078255,
+            0.0032953477930277586,
+            -0.020309917628765106,
+            0.015123048797249794,
+            0.0021205542143434286,
+            0.004703733138740063,
+            -0.0017638999270275235,
+            0.0008722644415684044,
+            -0.004024608992040157,
+            0.014211473055183887,
+            -0.01211484707891941,
+            0.014329977333545685,
+            0.024302620440721512,
+            0.01725613698363304,
+            0.005469457246363163,
+            0.007142199203372002,
+            0.008354595862329006,
+            0.0027278917841613293,
+            -0.009220592677593231,
+            -0.003871919820085168,
+            -0.0018413838697597384,
+            -0.00596626615151763,
+            0.023099340498447418,
+            -0.0092251505702734,
+            0.005414762534201145,
+            0.018906090408563614,
+            -0.008997256867587566,
+            -0.006467632949352264,
+            0.009279845282435417,
+            -0.002962622558698058,
+            -0.012187773361802101,
+            0.013263433240354061,
+            -0.002878301776945591,
+            0.01030081044882536,
+            0.014530524611473083,
+            -0.0013844562927260995,
+            -0.0022504536900669336,
+            -0.00414767162874341,
+            -0.0074475775472819805,
+            0.06745664030313492,
+            -0.015123048797249794,
+            0.004466723185032606,
+            -0.010984493419528008,
+            -0.017155863344669342,
+            0.014530524611473083,
+            0.02027345448732376,
+            0.02942567877471447,
+            -0.010701904073357582,
+            0.003972193226218224,
+            0.000644940126221627,
+            0.0017958050593733788,
+            0.018085671588778496,
+            0.006800358183681965,
+            -0.0009081577882170677,
+            -0.017265252768993378,
+            -0.01842295378446579,
+            -0.015241554006934166,
+            0.00473563838750124,
+            -0.01881493255496025,
+            0.0038149463944137096,
+            0.0046991752460598946,
+            -0.001347993267700076,
+            -0.013783032074570656,
+            0.005879666190594435,
+            0.003363716183230281,
+            -0.020510464906692505,
+            -0.0018744284752756357,
+            -0.008049217984080315,
+            -0.04740196093916893,
+            0.017721040174365044,
+            -0.005820414051413536,
+            0.011722870171070099,
+            -0.0032429322600364685,
+            0.016800349578261375,
+            0.007292609661817551,
+            0.004040561616420746,
+            0.005241563078016043,
+            0.009635359980165958,
+            0.009562434628605843,
+            0.013090234249830246,
+            -0.00982679147273302,
+            0.013072002679109573,
+            0.013637179508805275,
+            -0.007939828559756279,
+            -0.004498628433793783,
+            0.013454864732921124,
+            -0.006125791929662228,
+            -0.0011434584157541394,
+            0.0047584278509020805,
+            0.009744749404489994,
+            0.0062807598151266575,
+            -0.0013685037847608328,
+            0.004858701024204493,
+            -0.0004552183672785759,
+            0.02824063040316105,
+            0.0019040547776967287,
+            -0.00737465126439929,
+            -0.015496795065701008,
+            -0.01800362952053547,
+            -0.022096605971455574,
+            -0.010802177712321281,
+            -0.014074736274778843,
+            0.002639013109728694,
+            -0.05677296593785286,
+            0.022716479375958443,
+            0.0028144915122538805,
+            0.0037465780042111874,
+            -0.0024339084047824144,
+            -0.013810379430651665,
+            0.030665423721075058,
+            -0.004749312065541744,
+            -0.022096605971455574,
+            -0.01602550968527794,
+            0.011941648088395596,
+            0.002258430002257228,
+            0.0018539180746302009,
+            -0.009863254614174366,
+            -0.0026731970719993114,
+            -0.0013673643115907907,
+            0.006736548151820898,
+            0.014038273133337498,
+            0.005478573031723499,
+            0.007164988666772842,
+            0.02393799088895321,
+            0.007201451808214188,
+            -0.01711028441786766,
+            0.007538734935224056,
+            -0.008281669579446316,
+            0.0039995405822992325,
+            -0.005765719339251518,
+            -0.002180946059525013,
+            0.003687325632199645,
+            -0.00339334225282073,
+            -0.0075478507205843925,
+            -0.011349123902618885,
+            0.002987690968438983,
+            0.004881490487605333,
+            -0.003343205666169524,
+            -0.002072696341201663,
+            0.01467637624591589,
+            0.0217137448489666,
+            0.015423868782818317,
+            -0.0008209883235394955,
+            -0.005628983024507761,
+            0.014320861548185349,
+            0.0107565987855196,
+            0.005319047253578901,
+            -0.007898807525634766,
+            -0.006513211876153946,
+            0.007092062849551439,
+            -0.01738375797867775,
+            0.0050774794071912766,
+            -0.01950773037970066,
+            0.0003783041029237211,
+            0.005852319300174713,
+            0.016681844368577003,
+            -0.0005318477051332593,
+            -0.006162255071103573,
+            0.0037374624516814947,
+            0.0077666291035711765,
+            0.003691883524879813,
+            0.01515951193869114,
+            0.014357324689626694,
+            -0.010273464024066925,
+            0.022114837542176247,
+            0.000282588618574664,
+            -0.007493156008422375,
+            -0.0023814928717911243,
+            -0.0027051023207604885,
+            0.01985413022339344,
+            -0.0009167037787847221,
+            0.015123048797249794,
+            -0.006435727700591087,
+            0.020802168175578117,
+            0.007206009700894356,
+            0.004767543636262417,
+            0.025797605514526367,
+            0.01651776023209095,
+            0.002832723082974553,
+            0.012360973283648491,
+            0.08350037783384323,
+            0.018240639939904213,
+            -0.010975377634167671,
+            -0.017547842115163803,
+            -0.0047265226021409035,
+            -0.0071695465594530106,
+            0.003413852769881487,
+            0.017493147403001785,
+            -0.017292600125074387,
+            -0.023354582488536835,
+            0.0006494980189017951,
+            0.000889356539119035,
+            0.0037875990383327007,
+            -0.01554237399250269,
+            0.01780308224260807,
+            0.015296248719096184,
+            -0.0013821773463860154,
+            0.0046376436948776245,
+            0.005765719339251518,
+            -0.007593429647386074,
+            -0.007620777003467083,
+            0.010401084087789059,
+            -0.015387405641376972,
+            -0.014102083630859852,
+            0.01190518494695425,
+            -0.004876932594925165,
+            -0.00021450527128763497,
+            0.004020051099359989,
+            0.010082032531499863,
+            -0.00183112861122936,
+            0.015287132933735847,
+            0.02353689633309841,
+            0.004532812628895044,
+            0.003794435877352953,
+            0.015268901363015175,
+            0.01530536450445652,
+            -0.00046746767475269735,
+            -0.0036166785284876823,
+            -0.01671830751001835,
+            0.009626244194805622,
+            0.006845937110483646,
+            -0.015323596075177193,
+            0.016216941177845,
+            0.02193252369761467,
+            0.006257970351725817,
+            0.006485864520072937,
+            0.00468550156801939,
+            -0.0004227434692438692,
+            0.01916133053600788,
+            0.000195134271052666,
+            -0.0034571527503430843,
+            0.006230622995644808,
+            0.007520503364503384,
+            -0.012898802757263184,
+            0.017839545384049416,
+            0.013664526864886284,
+            0.012379204854369164,
+            0.007319956552237272,
+            0.0037853200919926167,
+            0.010720135644078255,
+            -0.022443005815148354,
+            0.016508644446730614,
+            -0.018514111638069153
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "93502993-0a3b-485e-9d40-e77a14173ff7",
+      "type": "child",
+      "source": {
+        "id": "65742957-6bda-446d-a61a-0fe20313c760",
+        "properties": {
+          "page_content": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity\n\nWilliam Fedus∗ liamfedus@google.com Barret Zoph∗ barretzoph@google.com Noam Shazeer noam@google.com Google, Mountain View, CA 94043, USA Editor: Alexander Clark\n\nAbstract\n\nIn deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) models defy this and instead select different parameters for each incoming example. The result is a sparsely-activated model—with an outrageous number of parameters—but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs, and training instability. We address these with the introduction of the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques mitigate the instabilities, and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large (Raffel et al., 2019) to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the \"Colossal Clean Crawled Corpus\", and achieve a 4x speedup over the T5-XXL model.12 Keywords: mixture-of-experts, natural language processing, sparsity, large-scale machine learning, distributed computing arXiv:2101.03961v3 [cs.LG] 16 Jun 2022 Contents\n\n1 Introduction 3 2 Switch Transformer 4 2.1 Simplifying Sparse Routing 5 2.2 Efficient Sparse Routing 6 2.3 Putting It All Together: The Switch Transformer 8 2.4 Improved Training and Fine-Tuning Techniques 8 3 Scaling Properties 11 3.1 Scaling Results on a Step-Basis 12 3.2 Scaling Results on a Time-Basis 13 3.3 Scaling Versus a Larger Dense Model 13 4 Downstream Results 14 4.1 Fine-Tuning 14 4.2 Distillation 16 4.3 Multilingual Learning 17 5 Designing Models with Data, Model, and Expert-Parallelism 18 5.1 Data Parallelism 20 5.2 Model Parallelism 20 5.3 Model and Data Parallelism 21 5.4 Expert and Data Parallelism 22 5.5 Expert, Model and Data Parallelism 22 5.6 Towards Trillion Parameter Models 22 6 Related Work 24 7 Discussion 25 8 Future Work 26 9 Conclusion 27 A Switch for Attention 27 B Preventing Token Dropping with No-Token-Left-Behind 29\n\nC Encouraging Exploration Across Experts 29 D Switch Transformers in Lower Compute Regimes 29 E Relation of Upstream to Downstream Model Performance 32 F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/switch_transformers.md"
+          },
+          "summary": "The Switch Transformer is a sparsely-activated model that simplifies the Mixture of Experts (MoE) approach, allowing for efficient scaling to trillion-parameter models with reduced communication and computational costs. It achieves significant speedups in pre-training and fine-tuning by activating only a subset of neural network weights for each input, maintaining a constant computational cost. The model demonstrates superior scaling properties and improved performance across various natural language processing tasks, including multilingual settings. It also allows for effective distillation into smaller dense models, preserving a portion of the quality gains. The architecture is designed to maximize parameter count while keeping floating point operations per example constant, making it suitable for both large-scale and smaller-scale applications.",
+          "headlines": {
+            "1 Introduction": [],
+            "2 Switch Transformer": [
+              "2.1 Simplifying Sparse Routing",
+              "2.2 Efficient Sparse Routing",
+              "2.3 Putting It All Together: The Switch Transformer",
+              "2.4 Improved Training and Fine-Tuning Techniques"
+            ],
+            "3 Scaling Properties": [
+              "3.1 Scaling Results on a Step-Basis",
+              "3.2 Scaling Results on a Time-Basis",
+              "3.3 Scaling Versus a Larger Dense Model"
+            ],
+            "4 Downstream Results": [
+              "4.1 Fine-Tuning",
+              "4.2 Distillation",
+              "4.3 Multilingual Learning"
+            ],
+            "5 Designing Models with Data, Model, and Expert-Parallelism": [
+              "5.1 Data Parallelism",
+              "5.2 Model Parallelism",
+              "5.3 Model and Data Parallelism",
+              "5.4 Expert and Data Parallelism",
+              "5.5 Expert, Model and Data Parallelism",
+              "5.6 Towards Trillion Parameter Models"
+            ],
+            "6 Related Work": [],
+            "7 Discussion": [],
+            "8 Future Work": [],
+            "9 Conclusion": [],
+            "A Switch for Attention": [],
+            "B Preventing Token Dropping with No-Token-Left-Behind": [],
+            "C Encouraging Exploration Across Experts": [],
+            "D Switch Transformers in Lower Compute Regimes": [],
+            "E Relation of Upstream to Downstream Model Performance": [],
+            "F Pseudo Code for Switch Transformers": []
+          },
+          "summary_embedding": [
+            -0.0060447100549936295,
+            -0.00390331051312387,
+            -0.01784370467066765,
+            -0.0021820589900016785,
+            0.013227885589003563,
+            0.011338187381625175,
+            0.0165580902248621,
+            0.024240797385573387,
+            -0.005986624862998724,
+            0.03596621751785278,
+            -0.004852031823247671,
+            -0.0068811336532235146,
+            -0.003574162255972624,
+            0.014242436736822128,
+            0.011578272096812725,
+            -0.033302053809165955,
+            -0.024519605562090874,
+            0.01050176378339529,
+            0.016093410551548004,
+            -0.006795942317694426,
+            0.007473600562661886,
+            -0.030513975769281387,
+            -0.06124480068683624,
+            -0.006838538218289614,
+            -0.0026099516544491053,
+            0.01029265858232975,
+            0.0010329445358365774,
+            -0.010873507708311081,
+            -0.053469158709049225,
+            0.023311438038945198,
+            0.025294071063399315,
+            0.005092116538435221,
+            -0.029584616422653198,
+            0.0009361362317577004,
+            -0.05161044001579285,
+            -0.04448534920811653,
+            0.0379488542675972,
+            0.012561844661831856,
+            -0.04364892467856407,
+            0.039249956607818604,
+            -0.004875265993177891,
+            -0.0022440163884311914,
+            -0.021499186754226685,
+            -0.0034909071400761604,
+            -0.007039899472147226,
+            -0.009851212613284588,
+            -0.02874819189310074,
+            0.0014037203509360552,
+            0.021886419504880905,
+            -0.0056962003000080585,
+            0.023264970630407333,
+            -0.024271776899695396,
+            0.01936165988445282,
+            0.04240977764129639,
+            -0.06957805901765823,
+            0.024659009650349617,
+            0.017549408599734306,
+            0.05120771750807762,
+            0.06511713564395905,
+            0.018339363858103752,
+            0.021669568493962288,
+            -0.014141756109893322,
+            -0.017766257748007774,
+            -0.011872569099068642,
+            0.004673904739320278,
+            -0.014970434829592705,
+            -0.01891246810555458,
+            0.008449427783489227,
+            -0.020461400970816612,
+            0.008209343999624252,
+            -0.05576157942414284,
+            -0.014730350114405155,
+            -0.034200433641672134,
+            -0.01140014547854662,
+            0.017642343416810036,
+            -0.018215449526906013,
+            0.030126741155982018,
+            0.02997184917330742,
+            0.002638994250446558,
+            0.0007822110201232135,
+            0.021049996837973595,
+            0.006509389728307724,
+            -0.0053438181057572365,
+            0.012321759946644306,
+            0.028159597888588905,
+            -0.01450575515627861,
+            -0.010106786154210567,
+            -0.03454120084643364,
+            0.01861817203462124,
+            0.01589204929769039,
+            -0.04194509983062744,
+            -0.012569589540362358,
+            -0.013824224472045898,
+            0.032341714948415756,
+            -0.009549170732498169,
+            0.04098476096987724,
+            -0.010083552449941635,
+            0.001383390510454774,
+            -0.031149037182331085,
+            -0.009463978931307793,
+            0.017069239169359207,
+            0.006172496825456619,
+            -0.016589069738984108,
+            -0.0012846461031585932,
+            0.016170857474207878,
+            0.006617815233767033,
+            -0.030142230913043022,
+            0.04488807171583176,
+            0.023497309535741806,
+            0.055296897888183594,
+            0.05579255893826485,
+            -0.037391237914562225,
+            -0.002445377642288804,
+            0.0344172865152359,
+            -0.0228312686085701,
+            -0.015040136873722076,
+            -0.034014564007520676,
+            -0.01768881268799305,
+            0.035935238003730774,
+            -0.0318460576236248,
+            0.01556677371263504,
+            0.028066661208868027,
+            -0.025712283328175545,
+            0.003488970920443535,
+            -0.03943582624197006,
+            0.003622566582635045,
+            0.01220559049397707,
+            0.006095050368458033,
+            -0.0026293133851140738,
+            -0.013615119270980358,
+            0.008906363509595394,
+            0.003055269829928875,
+            0.00800798274576664,
+            -0.04615819454193115,
+            0.018478767946362495,
+            -0.00454224506393075,
+            0.04033420979976654,
+            0.042502716183662415,
+            -0.02730768360197544,
+            -0.016743963584303856,
+            0.008557853288948536,
+            0.012104909867048264,
+            0.04188314080238342,
+            -0.007287728600203991,
+            -0.00121784838847816,
+            -0.0007836631848476827,
+            0.007322579622268677,
+            0.047552235424518585,
+            0.004646798130124807,
+            0.0020697612781077623,
+            -0.027230238541960716,
+            0.014978179708123207,
+            -0.019733402878046036,
+            0.027648448944091797,
+            0.023373395204544067,
+            0.02942972257733345,
+            -0.04048910364508629,
+            0.017518429085612297,
+            -0.011818356812000275,
+            -0.005568413063883781,
+            -0.002726121572777629,
+            -0.02856232039630413,
+            -0.0074426219798624516,
+            0.033983584493398666,
+            0.015365412458777428,
+            0.012956822291016579,
+            0.03271345794200897,
+            0.0012052633101120591,
+            -0.042502716183662415,
+            0.03804178908467293,
+            0.024550583213567734,
+            0.04640602692961693,
+            0.023466331884264946,
+            0.010796061716973782,
+            0.006571347359567881,
+            0.013375034555792809,
+            -0.019377149641513824,
+            0.015907539054751396,
+            0.02560385875403881,
+            0.009750531986355782,
+            -0.03167567402124405,
+            -0.0009501734166406095,
+            0.01178737822920084,
+            -0.004546117503196001,
+            0.03701949119567871,
+            0.04104671627283096,
+            0.020569827407598495,
+            0.05030933395028114,
+            0.0020155487582087517,
+            0.017936641350388527,
+            0.005936284549534321,
+            0.02820606529712677,
+            0.030467506498098373,
+            -0.005285732913762331,
+            0.005076627247035503,
+            -0.01136916596442461,
+            -0.03385967016220093,
+            0.014110777527093887,
+            0.01561324205249548,
+            -0.008077683858573437,
+            -0.01697630248963833,
+            0.006033093202859163,
+            0.009169681929051876,
+            -0.01795213110744953,
+            -0.017456471920013428,
+            0.005971135571599007,
+            0.018866000697016716,
+            -0.013576395809650421,
+            0.024689987301826477,
+            -0.04442339017987251,
+            0.029119934886693954,
+            -0.033209118992090225,
+            -0.003384418087080121,
+            -0.010641167871654034,
+            -0.0030843124259263277,
+            0.0041666291654109955,
+            0.011314953677356243,
+            0.013406013138592243,
+            0.002042655134573579,
+            -0.013909416273236275,
+            0.024519605562090874,
+            0.018447788432240486,
+            0.012073930352926254,
+            -0.03148980438709259,
+            -0.014397329650819302,
+            -0.0015673263696953654,
+            -0.008743725717067719,
+            -0.034138478338718414,
+            0.026966920122504234,
+            0.014048820361495018,
+            -0.03844451159238815,
+            0.017797237262129784,
+            0.029538147151470184,
+            0.014420564286410809,
+            -0.004790074657648802,
+            -0.0463130883872509,
+            0.04045812413096428,
+            -0.030436528846621513,
+            -0.033333033323287964,
+            -0.010478530079126358,
+            -0.0003146269591525197,
+            0.02894955314695835,
+            -0.0064358157105743885,
+            -0.050835974514484406,
+            0.01184933539479971,
+            0.020631784573197365,
+            0.00010763872705865651,
+            0.005731051322072744,
+            -0.019593998789787292,
+            0.00028631053282879293,
+            -0.03395260497927666,
+            0.0090844901278615,
+            0.026750069111585617,
+            0.008867640048265457,
+            -0.016418687999248505,
+            0.03351890295743942,
+            -0.0004121613164898008,
+            -0.018695618957281113,
+            -0.003889757441356778,
+            0.006885006092488766,
+            0.034912943840026855,
+            0.004836542531847954,
+            -0.02946070022881031,
+            0.05665995925664902,
+            -0.02100352756679058,
+            0.021344292908906937,
+            0.05631919577717781,
+            -0.019439106807112694,
+            -0.008472662419080734,
+            -0.002065889071673155,
+            0.0026196325197815895,
+            -0.0260530486702919,
+            -0.008937342092394829,
+            0.032341714948415756,
+            0.041728246957063675,
+            -0.018742086365818977,
+            -0.001398879918269813,
+            0.006780453026294708,
+            0.009425255469977856,
+            0.004948840010911226,
+            0.02692045085132122,
+            -0.012128143571317196,
+            -0.014559967443346977,
+            0.05198218300938606,
+            0.028314489871263504,
+            -0.012081675231456757,
+            -0.0164031982421875,
+            0.019888296723365784,
+            -0.024844881147146225,
+            -0.020244551822543144,
+            -0.02431824430823326,
+            -0.020414933562278748,
+            -0.0042518205009400845,
+            0.009525936096906662,
+            0.037484172731637955,
+            -0.03968365862965584,
+            0.012747716158628464,
+            0.021700548008084297,
+            0.02981695532798767,
+            -0.019532041624188423,
+            -0.05910727381706238,
+            -0.0016128262504935265,
+            0.03296129032969475,
+            -0.015458349138498306,
+            -0.049379974603652954,
+            0.015164051204919815,
+            0.0260530486702919,
+            0.05957195535302162,
+            -0.001461805310100317,
+            -0.014954946003854275,
+            0.061926331371068954,
+            0.03026614524424076,
+            -0.004383479710668325,
+            -0.009494957514107227,
+            -0.0127864396199584,
+            0.003396035172045231,
+            0.00465454300865531,
+            -0.03364282101392746,
+            -0.041170634329319,
+            -0.011841590516269207,
+            -0.010284913703799248,
+            0.008108663372695446,
+            -0.040922801941633224,
+            0.03531566634774208,
+            -0.021917399019002914,
+            0.010950954630970955,
+            0.015473837964236736,
+            0.037391237914562225,
+            -0.021344292908906937,
+            -0.01284839678555727,
+            -0.01572941243648529,
+            -0.024968795478343964,
+            -0.014010096900165081,
+            0.021917399019002914,
+            0.007682706229388714,
+            3.4699722164077684e-05,
+            0.010075807571411133,
+            -0.024659009650349617,
+            -0.051052823662757874,
+            0.0291044469922781,
+            0.0015605497173964977,
+            -0.015938518568873405,
+            -0.004623564425855875,
+            -0.01977987214922905,
+            -0.01122201792895794,
+            -0.012608313001692295,
+            -0.039249956607818604,
+            0.021111954003572464,
+            -0.01135367713868618,
+            0.015876561403274536,
+            0.0009250032599084079,
+            0.003659353591501713,
+            0.028624277561903,
+            0.01116006076335907,
+            0.0051811798475682735,
+            -0.03255856782197952,
+            -0.018277406692504883,
+            0.010122275911271572,
+            -0.00561100896447897,
+            -0.018339363858103752,
+            -0.013351799920201302,
+            -0.05037129297852516,
+            -0.03305422514677048,
+            -0.008054450154304504,
+            -0.05415068939328194,
+            0.0020697612781077623,
+            -0.014296649023890495,
+            -0.02232012152671814,
+            -0.005990497302263975,
+            0.03107159025967121,
+            -0.003235333366319537,
+            0.025294071063399315,
+            0.0076594725251197815,
+            0.037360258400440216,
+            0.040117356926202774,
+            0.008426194079220295,
+            0.03065337985754013,
+            -0.0029991210903972387,
+            -0.016867877915501595,
+            0.05486319959163666,
+            -0.01881953328847885,
+            0.013529927469789982,
+            -0.008604321628808975,
+            0.008604321628808975,
+            0.03299226611852646,
+            -0.003794885240495205,
+            0.02007416822016239,
+            -0.0311955064535141,
+            -0.07998688519001007,
+            0.01637221872806549,
+            0.028159597888588905,
+            -0.06418777257204056,
+            0.014691627584397793,
+            -0.020275529474020004,
+            0.02052335813641548,
+            0.02411688305437565,
+            -0.0009114501299336553,
+            -0.014064309187233448,
+            0.005595519673079252,
+            0.0020833145827054977,
+            -0.01456771232187748,
+            -0.002691270550712943,
+            -0.009417511522769928,
+            0.008774704299867153,
+            -0.017007282003760338,
+            -0.013406013138592243,
+            -0.034231413155794144,
+            0.010873507708311081,
+            0.01233724970370531,
+            -0.003969140350818634,
+            0.0011500825639814138,
+            -0.03277541697025299,
+            -0.007876322604715824,
+            -0.02145271934568882,
+            0.03382869064807892,
+            0.012058441527187824,
+            -0.030699847266077995,
+            -0.007814365439116955,
+            0.015264731831848621,
+            -0.007876322604715824,
+            0.0008906363509595394,
+            0.024132372811436653,
+            0.02103450708091259,
+            0.0028364830650389194,
+            -0.0009700191440060735,
+            0.02438020147383213,
+            0.007427132688462734,
+            -0.01720864325761795,
+            0.028794659301638603,
+            0.014288905076682568,
+            0.060160547494888306,
+            -0.03271345794200897,
+            0.001297231181524694,
+            -0.04135650396347046,
+            -0.031118059530854225,
+            0.006408709101378918,
+            0.014350862242281437,
+            0.007597515359520912,
+            -0.028252532705664635,
+            0.005978880450129509,
+            0.02004319056868553,
+            0.01469937153160572,
+            0.009704063646495342,
+            0.0037580979987978935,
+            -0.036740683019161224,
+            -0.000691695255227387,
+            -0.011214273050427437,
+            -0.0038781403563916683,
+            0.035656433552503586,
+            -0.008186109364032745,
+            -0.05151750147342682,
+            -0.0006418389966711402,
+            -0.03255856782197952,
+            -0.0353776253759861,
+            0.021313315257430077,
+            -0.00839521549642086,
+            -0.0039207362569868565,
+            -0.013986863195896149,
+            -0.000537770101800561,
+            -0.010524998418986797,
+            0.027509044855833054,
+            0.028345469385385513,
+            0.022490503266453743,
+            -0.05384090170264244,
+            0.009479468688368797,
+            0.00023766436788719147,
+            -0.05817791447043419,
+            0.0009709871956147254,
+            -0.007117345929145813,
+            0.011191039346158504,
+            -0.020213572308421135,
+            -0.02414786070585251,
+            0.011500826105475426,
+            0.03392162546515465,
+            0.014304393902420998,
+            0.0321558453142643,
+            -0.013367289677262306,
+            -0.09250226616859436,
+            0.018292896449565887,
+            -0.008689512498676777,
+            0.04126356914639473,
+            0.00865078903734684,
+            -0.004278926644474268,
+            -0.015845581889152527,
+            -0.00865078903734684,
+            -0.006943091284483671,
+            0.01283290795981884,
+            0.004530628211796284,
+            -0.005626498255878687,
+            0.0016631665639579296,
+            0.013708055019378662,
+            -0.005967263597995043,
+            0.007481344975531101,
+            0.003651608945801854,
+            -0.013189162127673626,
+            -0.06059424951672554,
+            -0.04055105894804001,
+            -0.018540725111961365,
+            0.050835974514484406,
+            -0.0069159846752882,
+            -0.046684831380844116,
+            -0.010850274004042149,
+            0.05058814212679863,
+            0.012259802781045437,
+            -0.0334569476544857,
+            -0.0007841471815481782,
+            0.0050882440991699696,
+            0.015458349138498306,
+            0.004546117503196001,
+            -0.027292195707559586,
+            0.02820606529712677,
+            0.0008412640891037881,
+            0.02300165221095085,
+            -0.018339363858103752,
+            -0.005312839522957802,
+            0.035780347883701324,
+            -0.024844881147146225,
+            0.023791607469320297,
+            0.0055219451896846294,
+            0.031629208475351334,
+            0.002952652983367443,
+            0.010060317814350128,
+            0.004704883322119713,
+            0.0186491496860981,
+            0.004275054205209017,
+            0.0276639387011528,
+            0.011462102644145489,
+            0.006428070832043886,
+            0.021623101085424423,
+            -0.009378788061439991,
+            -0.014513500034809113,
+            0.01885051093995571,
+            -0.0014695499558001757,
+            0.025232113897800446,
+            -0.020538847893476486,
+            0.002726121572777629,
+            -0.0016883367206901312,
+            -0.017967619001865387,
+            -0.040922801941633224,
+            -0.0007405834621749818,
+            0.021747015416622162,
+            -0.011671207845211029,
+            -0.0228312686085701,
+            -0.013715799897909164,
+            -0.01585332676768303,
+            0.0038490979932248592,
+            0.034169457852840424,
+            -0.0398695282638073,
+            0.00963436160236597,
+            -0.000830615172162652,
+            -0.03125746175646782,
+            -0.0239619892090559,
+            -0.01093546487390995,
+            0.014614180661737919,
+            -0.01652711257338524,
+            -0.021173911169171333,
+            -0.028469383716583252,
+            -0.008410705253481865,
+            0.006048582494258881,
+            -0.05065010115504265,
+            -0.004054331686347723,
+            0.019965743646025658,
+            0.0276639387011528,
+            -0.003099801717326045,
+            -0.02177799493074417,
+            0.0012468908680602908,
+            0.022351099178195,
+            0.02936776541173458,
+            0.014226946979761124,
+            0.027586491778492928,
+            -0.002178186783567071,
+            -0.0011675080750137568,
+            -0.0073148347437381744,
+            0.01307299267500639,
+            0.0011287847300991416,
+            0.036616768687963486,
+            0.0122210793197155,
+            -0.020306508988142014,
+            0.0013959755888208747,
+            -0.006431943271309137,
+            0.029181893914937973,
+            -0.02270735427737236,
+            -9.832092473516241e-05,
+            0.03531566634774208,
+            0.027741385623812675,
+            0.004933350719511509,
+            -0.0143121387809515,
+            -0.05142456665635109,
+            0.021592123433947563,
+            -0.01765783317387104,
+            0.031025122851133347,
+            0.016666516661643982,
+            0.003397971158847213,
+            -0.0034212053287774324,
+            0.012368228286504745,
+            -0.022087780758738518,
+            -0.004406713880598545,
+            -0.021142931655049324,
+            0.04389675334095955,
+            -0.043184246867895126,
+            0.008472662419080734,
+            0.029445212334394455,
+            0.011926782317459583,
+            -0.019082851707935333,
+            -0.005425137002021074,
+            0.029212871566414833,
+            -0.00162637943867594,
+            0.01308848150074482,
+            0.0058665829710662365,
+            -0.026935940608382225,
+            0.01202746294438839,
+            -0.031660184264183044,
+            0.02177799493074417,
+            -0.013436991721391678,
+            0.018447788432240486,
+            0.03342596814036369,
+            -0.0030397805385291576,
+            -0.0038239278364926577,
+            -0.016263794153928757,
+            -0.00969631876796484,
+            0.0662323608994484,
+            0.01054823212325573,
+            0.03726732358336449,
+            0.0018974426202476025,
+            -0.009773765690624714,
+            0.007969259284436703,
+            0.0063041565008461475,
+            -0.012329504825174809,
+            0.010494019836187363,
+            0.024178840219974518,
+            0.0012730291346088052,
+            -0.002412462839856744,
+            -0.019408127292990685,
+            -0.03438630700111389,
+            0.003727119415998459,
+            -0.030932186171412468,
+            0.022691864520311356,
+            0.009208405390381813,
+            0.017456471920013428,
+            0.008240322582423687,
+            -0.015907539054751396,
+            0.03200095146894455,
+            0.05461536720395088,
+            0.0019768255297094584,
+            0.013375034555792809,
+            0.007938280701637268,
+            0.007036027032881975,
+            -0.011338187381625175,
+            0.04702559858560562,
+            0.000662652775645256,
+            0.005169562995433807,
+            0.022536970674991608,
+            0.02145271934568882,
+            0.022939693182706833,
+            0.002304037567228079,
+            -0.011694442480802536,
+            0.010478530079126358,
+            -0.0006302220281213522,
+            0.004445436876267195,
+            -0.010695381090044975,
+            -0.027865299955010414,
+            0.007492962293326855,
+            0.01302652433514595,
+            0.022598929703235626,
+            0.021592123433947563,
+            -0.03178409859538078,
+            -0.01855621486902237,
+            0.03364282101392746,
+            -0.02997184917330742,
+            0.03612111136317253,
+            -0.0014424435794353485,
+            -0.023807097226381302,
+            -0.0058356039226055145,
+            0.016929835081100464,
+            0.02656419761478901,
+            -0.029181893914937973,
+            0.01804506592452526,
+            0.029693040996789932,
+            -0.0074426219798624516,
+            0.0005939188995398581,
+            0.012112653814256191,
+            0.02447313815355301,
+            -0.009293596260249615,
+            -0.0024027819745242596,
+            -0.0013320822035893798,
+            -0.011926782317459583,
+            -0.018370341509580612,
+            -0.0044725434854626656,
+            -0.022629907354712486,
+            0.005564541090279818,
+            -0.007365175057202578,
+            -0.003481226274743676,
+            -0.008581086993217468,
+            0.01154729351401329,
+            0.001979729626327753,
+            -0.019857317209243774,
+            -0.022335609421133995,
+            0.01652711257338524,
+            0.03887821361422539,
+            -0.018215449526906013,
+            -0.02106548473238945,
+            0.024442158639431,
+            -0.01759587600827217,
+            -0.01601596549153328,
+            0.036678727716207504,
+            0.013739033602178097,
+            0.007736918982118368,
+            -0.03358086198568344,
+            -0.030049296095967293,
+            -0.03274443745613098,
+            0.006296411622315645,
+            0.03785591572523117,
+            -0.0028752065263688564,
+            0.02569679357111454,
+            0.03701949119567871,
+            0.0057891360484063625,
+            -0.028934063389897346,
+            -0.014823286794126034,
+            -0.045104920864105225,
+            0.060160547494888306,
+            0.0011481463443487883,
+            0.05161044001579285,
+            0.016294771805405617,
+            -0.01456771232187748,
+            0.00419760774821043,
+            -0.013251119293272495,
+            0.026192452758550644,
+            0.02280029095709324,
+            0.025929134339094162,
+            0.006582964211702347,
+            0.022505993023514748,
+            -0.0037522895727306604,
+            -0.012197845615446568,
+            0.007307090330868959,
+            0.014536733739078045,
+            0.0002497654058970511,
+            -0.011392400600016117,
+            0.015783624723553658,
+            -0.01307299267500639,
+            -0.011361422017216682,
+            0.023295948281884193,
+            0.01368482131510973,
+            -0.012344993650913239,
+            0.014064309187233448,
+            -0.008565598167479038,
+            0.01239920686930418,
+            -0.023915521800518036,
+            0.022691864520311356,
+            -0.03262052312493324,
+            0.009587894193828106,
+            0.0015053689712658525,
+            -0.0073148347437381744,
+            0.024550583213567734,
+            -0.0027454833034425974,
+            -0.05071205645799637,
+            0.03550153970718384,
+            -0.008015726692974567,
+            0.0037213109899312258,
+            0.006381602957844734,
+            0.011392400600016117,
+            -0.0012623801594600081,
+            0.03807276859879494,
+            0.01239920686930418,
+            0.015969496220350266,
+            0.02199484594166279,
+            0.011911292560398579,
+            -0.008433938957750797,
+            -0.004096927121281624,
+            -0.0016137943603098392,
+            -0.01784370467066765,
+            0.025944624096155167,
+            0.0018674320308491588,
+            0.03962169960141182,
+            -0.018168980255723,
+            -0.02300165221095085,
+            0.028066661208868027,
+            0.007597515359520912,
+            -0.02351279929280281,
+            -0.0017377089243382215,
+            0.01202746294438839,
+            -0.009185170754790306,
+            0.027385130524635315,
+            0.01196550577878952,
+            0.019113829359412193,
+            -0.018060555681586266,
+            -0.004340883810073137,
+            -0.005072754807770252,
+            -0.015938518568873405,
+            0.02238207869231701,
+            0.020771188661456108,
+            0.011152315884828568,
+            -0.011880313977599144,
+            -0.01307299267500639,
+            0.020600805059075356,
+            0.035036858171224594,
+            -0.0002744515077210963,
+            0.005672966130077839,
+            0.017487451434135437,
+            0.002373739378526807,
+            -0.04470219835639,
+            0.016232814639806747,
+            -0.014025585725903511,
+            0.011214273050427437,
+            0.019593998789787292,
+            -0.021917399019002914,
+            -0.03196997195482254,
+            -0.011539548635482788,
+            -0.0414804182946682,
+            0.00493722315877676,
+            -0.0069159846752882,
+            0.013576395809650421,
+            -0.010176488198339939,
+            -0.0014831031439825892,
+            0.016666516661643982,
+            0.0038103745318949223,
+            0.02177799493074417,
+            -0.011678952723741531,
+            -0.012073930352926254,
+            -0.020213572308421135,
+            -0.020786678418517113,
+            0.004278926644474268,
+            -0.030405549332499504,
+            -0.004488032776862383,
+            0.021375272423028946,
+            -0.04783104360103607,
+            -0.03689557686448097,
+            0.02898053266108036,
+            -0.008503641001880169,
+            0.020755698904395103,
+            0.022955182939767838,
+            -0.015094350092113018,
+            -0.0008717587334103882,
+            -0.018695618957281113,
+            0.02258343994617462,
+            -0.007837600074708462,
+            0.026300877332687378,
+            -0.0027551641687750816,
+            0.0006292539183050394,
+            -0.02659517526626587,
+            0.020306508988142014,
+            0.005227648187428713,
+            0.012275291606783867,
+            -0.02390003204345703,
+            0.006815304048359394,
+            -0.03133491054177284,
+            -0.01784370467066765,
+            0.0180140882730484,
+            -0.011152315884828568,
+            0.027989214286208153,
+            -0.006060199346393347,
+            0.028438406065106392,
+            0.022722844034433365,
+            0.024751944467425346,
+            0.02676555886864662,
+            -0.02363671362400055,
+            0.008495896123349667,
+            0.03726732358336449,
+            0.00865078903734684,
+            -0.015876561403274536,
+            0.008170620538294315,
+            -0.04863648861646652,
+            -0.008062195032835007,
+            0.05030933395028114,
+            0.002127846470102668,
+            0.023141054436564445,
+            0.016294771805405617,
+            -0.002873270306736231,
+            -0.005688455421477556,
+            0.010269423946738243,
+            -0.04296739399433136,
+            -0.023172033950686455,
+            0.022629907354712486,
+            -0.021561143919825554,
+            6.54212328754511e-07,
+            0.03122648410499096,
+            0.012120398692786694,
+            -0.0009109660750254989,
+            -0.05108380317687988,
+            0.02100352756679058,
+            0.01178737822920084,
+            0.014443797990679741,
+            -0.010300402529537678,
+            0.026068538427352905,
+            0.01733255758881569,
+            0.038413532078266144,
+            0.012755461037158966,
+            -0.002679653698578477,
+            -0.03782493621110916,
+            0.009177426807582378,
+            -0.021685058251023293,
+            0.0011471783509477973,
+            0.0074503663927316666,
+            -0.02273833192884922,
+            -0.024782923981547356,
+            0.0043912241235375404,
+            0.004588713403791189,
+            0.011880313977599144,
+            0.022087780758738518,
+            0.009417511522769928,
+            -0.021204888820648193,
+            0.015404135920107365,
+            -0.05446047708392143,
+            -0.015450604259967804,
+            -0.023884542286396027,
+            0.02566581591963768,
+            0.020012211054563522,
+            0.010184233076870441,
+            0.005545179359614849,
+            0.010904486291110516,
+            -0.020988039672374725,
+            -0.009239383973181248,
+            0.020631784573197365,
+            0.011074868962168694,
+            -0.012840652838349342,
+            0.04113965481519699,
+            -0.00011126903700642288,
+            0.031210994347929955,
+            -0.02280029095709324,
+            0.002941036131232977,
+            -0.001241082325577736,
+            -0.03649285435676575,
+            0.009130958467721939,
+            0.01981084980070591,
+            0.02151467651128769,
+            -0.01861817203462124,
+            0.0024143988266587257,
+            0.01048627495765686,
+            0.004766840487718582,
+            0.038351576775312424,
+            0.027121812105178833,
+            -0.044206541031599045,
+            0.011663463898003101,
+            0.012995545752346516,
+            -0.009363298304378986,
+            0.048419639468193054,
+            -0.017518429085612297,
+            0.017673322930932045,
+            0.008240322582423687,
+            -0.01798310875892639,
+            0.02318752370774746,
+            -0.01469937153160572,
+            -0.013282098807394505,
+            0.00650551775470376,
+            0.009192915633320808,
+            -0.0001770381786627695,
+            -0.037577107548713684,
+            -0.017859194427728653,
+            -0.005719434004276991,
+            0.0038297362625598907,
+            0.01562873087823391,
+            0.013498948886990547,
+            0.009967382065951824,
+            0.005161818582564592,
+            -0.00945623405277729,
+            -0.0016089539276435971,
+            0.008124152198433876,
+            -0.0005532593932002783,
+            0.0019216446671634912,
+            -0.01926872320473194,
+            0.02698240801692009,
+            0.02572777308523655,
+            -0.0018354853382334113,
+            -0.04652994126081467,
+            -0.0036709706764668226,
+            0.0037077576853334904,
+            -0.017735280096530914,
+            -0.0035315665882080793,
+            -0.04519785940647125,
+            -0.01810702309012413,
+            0.0143121387809515,
+            -0.014614180661737919,
+            -0.0460033044219017,
+            0.009076746180653572,
+            -0.013886182568967342,
+            0.0063970922492444515,
+            -0.00932457484304905,
+            0.005762029904872179,
+            0.003651608945801854,
+            0.005680711008608341,
+            -0.005909178406000137,
+            0.05108380317687988,
+            -0.03466511517763138,
+            0.028051171451807022,
+            0.008565598167479038,
+            -0.012437930330634117,
+            -0.02261441759765148,
+            0.00887538492679596,
+            -0.044981006532907486,
+            -0.033209118992090225,
+            0.016743963584303856,
+            -0.01781272701919079,
+            0.011074868962168694,
+            0.06161654368042946,
+            0.007330324035137892,
+            -0.023342415690422058,
+            -0.01233724970370531,
+            -0.021049996837973595,
+            -0.04191412031650543,
+            0.003442503046244383,
+            0.005463860463351011,
+            -0.004948840010911226,
+            -0.03807276859879494,
+            -0.021716037765145302,
+            0.04615819454193115,
+            -0.012004229240119457,
+            0.012422440573573112,
+            0.01080380566418171,
+            -0.0019410063978284597,
+            -0.04860550910234451,
+            -0.021019017323851585,
+            0.0395597442984581,
+            0.021886419504880905,
+            -0.008426194079220295,
+            0.0028364830650389194,
+            -0.018277406692504883,
+            -0.003289545886218548,
+            0.006296411622315645,
+            -0.009107724763453007,
+            -0.03562545403838158,
+            0.02067825198173523,
+            0.0054832217283546925,
+            -0.023373395204544067,
+            -0.025805220007896423,
+            -0.03296129032969475,
+            0.009944148361682892,
+            0.02453509531915188,
+            -0.01968693546950817,
+            0.005823987070471048,
+            0.00027203132049180567,
+            -0.000881923595443368,
+            0.00042740863864310086,
+            -0.009959637187421322,
+            -0.019764382392168045,
+            -0.020151615142822266,
+            0.03268248215317726,
+            -0.0060059865936636925,
+            0.0074387495405972,
+            -0.03630698472261429,
+            -0.010974188335239887,
+            0.011322698555886745,
+            -0.002395037328824401,
+            -0.013452480547130108,
+            -0.009092235006392002,
+            -0.01739451475441456,
+            -0.009882191196084023,
+            -0.007876322604715824,
+            0.022660886868834496,
+            0.006168624851852655,
+            0.005409647710621357,
+            0.0023795480374246836,
+            -0.008581086993217468,
+            0.02898053266108036,
+            0.017874684184789658,
+            -0.014188224449753761,
+            -0.00920066051185131,
+            0.03451022133231163,
+            0.02665713243186474,
+            -0.009115469641983509,
+            0.01798310875892639,
+            0.023264970630407333,
+            -0.0010484338272362947,
+            -0.01196550577878952,
+            0.001922612776979804,
+            -0.0009550138493068516,
+            -0.0032972905319184065,
+            0.00845717266201973,
+            -0.015365412458777428,
+            0.0315207801759243,
+            0.04287445917725563,
+            -0.002145271748304367,
+            0.009587894193828106,
+            -0.003727119415998459,
+            -0.008472662419080734,
+            0.017069239169359207,
+            0.009316830895841122,
+            0.04625113308429718,
+            0.01098193321377039,
+            0.005680711008608341,
+            -0.033983584493398666,
+            -0.001229465357027948,
+            -0.01184933539479971,
+            0.03308520466089249,
+            0.02061629481613636,
+            0.00889087375253439,
+            -0.008379725739359856,
+            0.01344473659992218,
+            0.008341003209352493,
+            0.00661007035523653,
+            0.020569827407598495,
+            -0.016542602330446243,
+            -0.017286090180277824,
+            0.030467506498098373,
+            -0.00672236829996109,
+            0.01289486512541771,
+            0.02029101923108101,
+            -0.002089123008772731,
+            0.010842529125511646,
+            0.07100307941436768,
+            -0.006184114143252373,
+            0.015713922679424286,
+            -0.019082851707935333,
+            0.0015789433382451534,
+            -0.007640110794454813,
+            -0.03370477631688118,
+            0.018246427178382874,
+            0.0039962464943528175,
+            0.00865078903734684,
+            0.01198099460452795,
+            0.009053511545062065,
+            -0.007946024648845196,
+            -0.00963436160236597,
+            0.015195030719041824,
+            0.0019080914789810777,
+            0.00033568276558071375,
+            0.001336922636255622,
+            -0.0065210070461034775,
+            -0.004011735785752535,
+            -0.021948376670479774,
+            0.02763296104967594,
+            0.007140580099076033,
+            0.030606910586357117,
+            -0.055296897888183594,
+            0.030095763504505157,
+            -0.00789568480104208,
+            0.02554190158843994,
+            -0.016201836988329887,
+            -0.011090358719229698,
+            0.001609921921044588,
+            0.018153492361307144,
+            0.004491904750466347,
+            0.006462921854108572,
+            0.011888058856129646,
+            0.023233991116285324,
+            -0.03444826602935791,
+            0.017053749412298203,
+            0.023110076785087585,
+            0.0048675211146473885,
+            0.0003535923024173826,
+            0.005165690556168556,
+            0.015458349138498306,
+            0.007198664825409651,
+            -0.023048119619488716,
+            -0.0025692922063171864,
+            0.017611365765333176,
+            -0.02306360937654972,
+            0.0036980770528316498,
+            -0.006923729553818703,
+            0.004623564425855875,
+            -0.0014705180656164885,
+            -0.03509881719946861,
+            -0.00100099784322083,
+            -0.025743262842297554,
+            0.01923774555325508,
+            0.009556914679706097,
+            -0.025867177173495293,
+            0.003932353109121323,
+            0.025650326162576675,
+            -0.0003695656778290868,
+            0.00821708794683218,
+            0.0004683101433329284,
+            0.00869725737720728,
+            -0.00974278710782528,
+            -0.01493945624679327,
+            0.004801691509783268,
+            0.01531120017170906,
+            -0.01493945624679327,
+            -0.007431004662066698,
+            -0.009510447271168232,
+            0.030560443177819252,
+            0.0006674932083114982,
+            0.004189862869679928,
+            -0.005045648664236069,
+            -0.01858719252049923,
+            -0.0032237162813544273,
+            -0.021142931655049324,
+            0.004367990419268608,
+            -0.0009046735358424485,
+            0.04197607934474945,
+            0.0038161829579621553,
+            -0.02029101923108101,
+            0.0032972905319184065,
+            -0.03136589005589485,
+            0.0005000148667022586,
+            -0.007276111748069525,
+            -0.02004319056868553,
+            0.0015005286550149322,
+            0.0016525175888091326,
+            0.012631546705961227,
+            -0.03255856782197952,
+            -0.009301341138780117,
+            0.018416810780763626,
+            -0.00020680672605521977,
+            -0.010687636211514473,
+            -0.017828216776251793,
+            0.00035189816844649613,
+            -0.011338187381625175,
+            -0.027973726391792297,
+            -0.02850036323070526,
+            -0.0026719090528786182,
+            0.013491204008460045,
+            -0.018277406692504883,
+            -0.014993669465184212,
+            0.0031249718740582466,
+            0.021592123433947563,
+            -0.009781510569155216,
+            0.008302279748022556,
+            0.02241305634379387,
+            -0.008255811408162117,
+            -0.005514200776815414,
+            -0.000941460661124438,
+            -0.009130958467721939,
+            0.004267309792339802,
+            0.006300284061580896,
+            0.043339136987924576,
+            -0.032403673976659775,
+            -0.0018238683696836233,
+            0.014815541915595531,
+            0.0034153966698795557,
+            0.0025905899237841368,
+            0.002503462601453066,
+            -0.0007841471815481782,
+            0.01344473659992218,
+            0.01178737822920084,
+            0.015466093085706234,
+            0.00100099784322083,
+            -0.0033011629711836576,
+            0.006408709101378918,
+            -0.021173911169171333,
+            0.015721667557954788,
+            0.027958236634731293,
+            -0.0033999073784798384,
+            0.005525817628949881,
+            -0.013731288723647594,
+            -0.008759214542806149,
+            0.013134949840605259,
+            0.0026641644071787596,
+            -0.005068882368505001,
+            -0.009394276887178421,
+            0.017967619001865387,
+            0.019082851707935333,
+            0.006668155547231436,
+            -0.037453193217515945,
+            -0.02898053266108036,
+            -0.009959637187421322,
+            0.020569827407598495,
+            0.014916222542524338,
+            0.0010339126456528902,
+            -0.010540487244725227,
+            -0.009595638141036034,
+            -0.011903548613190651,
+            0.010989678092300892,
+            0.02721474878489971,
+            -0.032372694462537766,
+            0.0008214184199459851,
+            -0.029383255168795586,
+            0.03068435750901699,
+            0.0020000594668090343,
+            0.011446612887084484,
+            -0.0008185141487047076,
+            -0.015140817500650883,
+            -0.029181893914937973,
+            0.013824224472045898,
+            -0.004395096562802792,
+            0.001024231780320406,
+            -0.0031753121875226498,
+            0.0201051477342844,
+            0.012887120246887207,
+            -0.0014472840121015906,
+            0.01968693546950817,
+            0.008178364485502243,
+            0.016480645164847374,
+            -0.010153254494071007,
+            -0.04135650396347046,
+            0.019965743646025658,
+            0.007562664337456226,
+            0.014219203032553196,
+            0.004739734344184399,
+            0.004588713403791189,
+            -0.020771188661456108,
+            0.002724185585975647,
+            0.004333139397203922,
+            0.014405074529349804,
+            0.011028401553630829,
+            0.0025615475606173277,
+            0.011678952723741531,
+            -0.02228914201259613,
+            0.01929970271885395,
+            -0.01631026156246662,
+            0.01006806269288063,
+            0.004445436876267195,
+            -0.026873983442783356,
+            0.007547175046056509,
+            0.007822110317647457,
+            -0.0019187404541298747,
+            -0.017549408599734306,
+            0.02160761132836342,
+            -0.0034831624943763018,
+            -0.03509881719946861,
+            0.0360281765460968,
+            -0.0043447562493383884,
+            0.019346170127391815,
+            0.03209388628602028,
+            -0.01771979033946991,
+            -0.01993476413190365,
+            0.00906900130212307,
+            0.010408828034996986,
+            -0.002873270306736231,
+            0.017859194427728653,
+            -0.002919738180935383,
+            -0.02232012152671814,
+            0.017766257748007774,
+            -0.028918573632836342,
+            -0.014234691858291626,
+            0.020337486639618874,
+            -0.03624502569437027,
+            -0.0021026763133704662,
+            0.03175312280654907,
+            -0.024070415645837784,
+            0.005734923295676708,
+            0.005196669604629278,
+            -0.0019351978553459048,
+            0.0012236569309607148,
+            -0.006451305001974106,
+            0.03801080957055092,
+            -0.010865762829780579,
+            0.0201051477342844,
+            0.01929970271885395,
+            -0.023388884961605072,
+            0.01977987214922905,
+            -0.001109423115849495,
+            0.01675945147871971,
+            -0.003940097521990538,
+            0.03466511517763138,
+            -0.006017603911459446,
+            -0.011330443434417248,
+            0.006431943271309137,
+            0.008488151244819164,
+            -0.009959637187421322,
+            0.008681767620146275,
+            -0.018122512847185135,
+            0.027152791619300842,
+            0.02091059274971485,
+            -0.0002691270783543587,
+            0.013800990767776966,
+            0.0004421719058882445,
+            -0.020833145827054977,
+            0.00789568480104208,
+            -0.027121812105178833,
+            -0.009967382065951824,
+            0.01646515540778637,
+            -0.005762029904872179,
+            0.007380664348602295,
+            -0.02064727433025837,
+            0.028004704043269157,
+            0.0077524082735180855,
+            -0.013591884635388851,
+            -0.0008824076503515244,
+            0.023760627955198288,
+            -0.005432881414890289,
+            0.009355553425848484,
+            0.02070923149585724,
+            0.015713922679424286,
+            0.013615119270980358,
+            0.016263794153928757,
+            0.03351890295743942,
+            0.005328328814357519,
+            0.005568413063883781,
+            0.003519949736073613,
+            0.016991792246699333,
+            -0.008557853288948536,
+            0.018029578030109406,
+            0.0008093173382803798,
+            -0.013576395809650421,
+            0.021499186754226685,
+            -0.0018684001406654716,
+            -0.013622863218188286,
+            -0.0040891822427511215,
+            -0.004337011836469173,
+            0.03271345794200897,
+            0.022041313350200653,
+            -0.004468671046197414,
+            -0.024442158639431,
+            -0.00394784240052104,
+            0.016682006418704987,
+            -0.003167567541822791,
+            0.011136826127767563,
+            0.0007952801533974707,
+            -0.01586107164621353,
+            -0.009595638141036034,
+            0.0063583687879145145,
+            0.010586955584585667,
+            -0.008921852335333824,
+            0.010346870869398117,
+            -0.0228312686085701,
+            -0.004011735785752535,
+            0.00046613195445388556,
+            -0.004844287410378456,
+            0.0053438181057572365,
+            -0.011314953677356243,
+            0.004948840010911226,
+            0.016062432900071144,
+            0.01026167906820774,
+            0.004832670092582703,
+            0.006989559158682823,
+            0.012066186405718327,
+            0.009161937050521374,
+            -0.01627928391098976,
+            0.00428667152300477,
+            -0.005072754807770252,
+            -0.02345084212720394,
+            0.005831731949001551,
+            0.00902253296226263,
+            -0.01122201792895794,
+            0.009386532939970493,
+            -0.010958699509501457,
+            0.03296129032969475,
+            0.01030814740806818,
+            -0.037391237914562225,
+            -0.0033379499800503254,
+            -0.0016244432190433145,
+            0.014885243959724903,
+            0.012298526242375374,
+            0.008550108410418034,
+            -0.017518429085612297,
+            -0.0047823297791182995,
+            -0.009835722856223583,
+            0.0012827098835259676,
+            -0.012306271120905876,
+            0.017456471920013428,
+            0.022567950189113617,
+            -0.036337960511446,
+            0.01968693546950817,
+            -0.010432061739265919,
+            -0.023652203381061554,
+            0.020476890727877617,
+            0.011500826105475426,
+            0.01942361705005169,
+            -0.009402021765708923,
+            0.018896980211138725,
+            -0.0001331114035565406,
+            0.025510922074317932,
+            -0.004228586331009865,
+            -0.022087780758738518,
+            -0.012499887496232986,
+            -0.03209388628602028,
+            -0.0051811798475682735,
+            0.006819176487624645,
+            -0.00045524101005867124,
+            0.013111716136336327,
+            0.014226946979761124,
+            -0.025712283328175545,
+            0.007295473478734493,
+            0.032589543610811234,
+            0.014637414366006851,
+            0.023311438038945198,
+            -0.011276230216026306,
+            -0.0010794125264510512,
+            -0.0026060794480144978,
+            -0.013770012184977531,
+            -0.03212486580014229,
+            -0.0017493258928880095,
+            -0.036585792899131775,
+            0.005680711008608341,
+            -0.016387708485126495,
+            -0.013096226379275322,
+            0.023264970630407333,
+            -0.034138478338718414,
+            0.0014608372002840042,
+            0.013708055019378662,
+            0.010540487244725227,
+            0.012391461990773678,
+            -0.00162637943867594,
+            0.015032391995191574,
+            -0.029243851080536842,
+            -0.010284913703799248,
+            -0.004491904750466347,
+            -0.01981084980070591,
+            -0.007338068913668394,
+            -0.018060555681586266,
+            -0.013127204962074757,
+            -0.003566417610272765,
+            -0.010060317814350128,
+            0.002247888594865799,
+            0.0032062907703220844,
+            -0.014714861288666725,
+            0.03206290677189827,
+            0.003157886676490307,
+            0.018958937376737595,
+            0.0033418224193155766,
+            0.004449309315532446,
+            0.018788553774356842,
+            -0.0026331855915486813,
+            -0.0058278595097362995,
+            0.030963165685534477,
+            -0.00032672801171429455,
+            0.008759214542806149,
+            0.019082851707935333,
+            0.004724245052784681,
+            0.0038529701996594667,
+            -0.0260530486702919,
+            0.012887120246887207,
+            -0.027447087690234184,
+            0.005080499220639467,
+            -0.0077175572514534,
+            -0.024302754551172256,
+            -0.028128618374466896,
+            -0.006571347359567881,
+            0.012058441527187824,
+            -0.02898053266108036,
+            -0.00613377382978797,
+            0.002315654419362545,
+            -0.00400399137288332,
+            -0.002673845272511244,
+            0.007241260726004839,
+            0.029150914400815964,
+            -0.033209118992090225,
+            0.008751469664275646,
+            0.005595519673079252,
+            -0.0005053392960689962,
+            0.013297587633132935,
+            0.008038961328566074,
+            0.003343758638948202,
+            0.020848635584115982,
+            0.006637176964432001,
+            0.002458930714055896,
+            -0.0025576751213520765,
+            -0.014242436736822128,
+            0.004000118933618069,
+            0.02154565416276455,
+            0.007562664337456226,
+            0.018385831266641617,
+            -0.008914108388125896,
+            0.002621568739414215,
+            -0.01455222349613905,
+            0.005552923772484064,
+            -0.002867461647838354,
+            0.004309905227273703,
+            -0.012236569076776505,
+            0.008093173615634441,
+            0.0028771425131708384,
+            -0.0043447562493383884,
+            0.018060555681586266,
+            -0.002319526858627796,
+            0.016480645164847374,
+            0.004015608225017786,
+            0.015620986931025982,
+            -0.0035412474535405636,
+            0.01178737822920084,
+            0.0005731051205657423,
+            -0.017905661836266518,
+            -0.017022771760821342,
+            -0.039249956607818604,
+            0.007702067960053682,
+            0.03531566634774208,
+            -0.0009564659558236599,
+            -0.03444826602935791,
+            0.010145509615540504,
+            -0.011942271143198013,
+            -0.009975126944482327,
+            -0.022862248122692108,
+            0.013862947933375835,
+            0.012430185452103615,
+            0.007415515370666981,
+            0.012252057902514935,
+            -0.006997303571552038,
+            0.01026167906820774,
+            -0.005216030869632959,
+            0.008356492035090923,
+            -0.0010823167394846678,
+            -0.037422213703393936,
+            0.012344993650913239,
+            0.02216522768139839,
+            -0.002635121811181307,
+            0.0016089539276435971,
+            -0.006145390681922436,
+            0.022103270515799522,
+            -0.013971373438835144,
+            -0.00650551775470376,
+            -0.00046443779137916863,
+            0.00797700323164463,
+            -0.01795213110744953,
+            0.004778457339853048,
+            0.0755879208445549,
+            -0.007775642443448305,
+            0.006122156511992216,
+            0.009270362555980682,
+            -0.015148562379181385,
+            -0.01637221872806549,
+            -0.004561606794595718,
+            -0.020260039716959,
+            0.008844406343996525,
+            -0.0018819533288478851,
+            -0.0110981035977602,
+            0.012468908913433552,
+            0.0053050946444272995,
+            0.002031038049608469,
+            -0.008379725739359856,
+            0.010811550542712212,
+            0.009525936096906662,
+            0.010432061739265919,
+            0.02360573597252369,
+            0.011477591469883919,
+            -0.00858883187174797,
+            -0.011795123107731342,
+            0.01129171997308731,
+            0.0028577810153365135,
+            0.016697494313120842,
+            0.007907302118837833,
+            0.0010648912284523249,
+            0.019501063972711563,
+            -0.002309845993295312,
+            0.02180897258222103,
+            -2.2689444449497387e-05,
+            -0.009301341138780117,
+            -0.004236331209540367,
+            -0.015349923633038998,
+            0.024271776899695396,
+            -0.0020155487582087517,
+            0.0015450604259967804,
+            -0.020213572308421135,
+            0.0013979118084535003,
+            0.013243375346064568,
+            0.011957760900259018,
+            -0.0020407189149409533,
+            -0.0051347119733691216,
+            0.0007154133054427803,
+            -0.016356728971004486,
+            0.032806396484375,
+            0.007578153628855944,
+            0.0007681738352403045,
+            0.0008838597568683326,
+            0.0009288755827583373,
+            -0.0032972905319184065,
+            0.0024608669336885214,
+            -0.004766840487718582,
+            0.009549170732498169,
+            -0.010943209752440453,
+            -0.010796061716973782,
+            0.010075807571411133,
+            -0.02402394637465477,
+            0.003566417610272765,
+            -0.00558777479454875,
+            -0.008062195032835007,
+            0.0076207490637898445,
+            0.00015537731815129519,
+            -0.007392281666398048,
+            -0.0030939930584281683,
+            0.0023408245760947466,
+            0.013429246842861176,
+            0.004406713880598545,
+            0.010850274004042149,
+            0.012956822291016579,
+            0.005359307397156954,
+            -0.006629432085901499,
+            0.015512561425566673,
+            -0.006946963258087635,
+            -0.004228586331009865,
+            -0.0012468908680602908,
+            -0.019996721297502518,
+            -0.0009419447160325944,
+            -0.004708755761384964,
+            0.0015092412941157818,
+            0.0001705036120256409,
+            -8.749050175538287e-05,
+            -0.0018490385264158249,
+            -0.011988739483058453,
+            -0.001831613015383482,
+            -0.011663463898003101,
+            -0.014498010277748108,
+            0.010253935120999813,
+            -0.0041201612912118435,
+            -0.022939693182706833,
+            -0.015349923633038998,
+            -0.020662762224674225,
+            0.02145271934568882,
+            -0.004387352149933577,
+            -0.005800753366202116,
+            0.006733985152095556,
+            -0.010215211659669876,
+            -0.015706177800893784,
+            -0.008069939911365509,
+            -0.0205853171646595,
+            0.010703125037252903,
+            0.003214035416021943,
+            0.0006791101768612862,
+            0.00882891658693552,
+            0.014397329650819302,
+            -0.01589204929769039,
+            0.0111755495890975,
+            0.012182355858385563,
+            0.011361422017216682,
+            -0.006854027509689331,
+            0.0008911203476600349,
+            -0.014103032648563385,
+            0.02596011385321617,
+            0.021623101085424423,
+            0.005169562995433807,
+            -0.04132552444934845,
+            -0.005862710531800985,
+            -0.013344055972993374,
+            -0.010524998418986797,
+            -0.012120398692786694,
+            0.011771888472139835,
+            -0.0038820127956569195,
+            0.0013117524795234203,
+            0.006288666743785143,
+            0.0015063370810821652,
+            0.004367990419268608,
+            -0.0022382077295333147,
+            -0.007709812838584185,
+            -0.03212486580014229,
+            -0.02817508764564991,
+            -0.00863530021160841,
+            0.020461400970816612,
+            0.002838419284671545,
+            -0.0009453330421820283,
+            -0.0036380558740347624,
+            0.0027048238553106785,
+            -0.005882072262465954,
+            -0.005018542055040598,
+            -0.014753584749996662,
+            -0.012739972211420536,
+            0.0101300198584795,
+            0.00802347157150507,
+            0.014195968396961689,
+            -0.011950016021728516,
+            0.0028190575540065765,
+            0.018788553774356842,
+            0.007690451107919216,
+            -0.023357905447483063,
+            0.01135367713868618,
+            -0.010912231169641018,
+            -0.009270362555980682,
+            -0.009433000348508358,
+            -0.009471723809838295,
+            0.0016980175860226154,
+            -0.008317768573760986,
+            0.003945906180888414,
+            -0.03615209087729454,
+            -0.011330443434417248,
+            -0.0013901671627536416,
+            -0.010346870869398117,
+            0.0009864765452221036,
+            0.02524760365486145,
+            0.006815304048359394,
+            0.01432762760668993,
+            -0.0006733016925863922,
+            0.017502939328551292,
+            -0.0018984107300639153,
+            0.018726596608757973,
+            0.012763205915689468,
+            -0.00797700323164463,
+            0.011717676185071468,
+            0.014738094992935658,
+            -0.0033863543067127466,
+            0.015620986931025982,
+            -0.010168743319809437,
+            0.005285732913762331,
+            -0.009556914679706097,
+            -0.003214035416021943,
+            0.002495717955753207,
+            0.030901208519935608,
+            -0.003618694143369794,
+            0.00858883187174797,
+            0.01098193321377039,
+            -0.015241498127579689,
+            0.0026951429899781942,
+            0.014962689951062202,
+            0.01141563430428505,
+            0.007760153152048588,
+            0.0024434414226561785,
+            -0.024225307628512383,
+            0.004193735308945179,
+            0.005037903785705566,
+            0.0031520782504230738,
+            -0.014366351068019867,
+            0.0021859314292669296,
+            0.015675200149416924,
+            0.008550108410418034,
+            -0.025325050577521324,
+            0.016418687999248505,
+            0.0009138703462667763,
+            -0.003909118939191103,
+            0.029119934886693954,
+            -0.011268485337495804,
+            -0.005723306443542242,
+            0.009463978931307793,
+            0.0009980935137718916,
+            0.0030029932968318462,
+            0.008937342092394829,
+            0.0228312686085701,
+            -0.003988501615822315,
+            -0.032341714948415756,
+            -0.022459525614976883,
+            0.015706177800893784,
+            -0.0047823297791182995,
+            -0.022939693182706833,
+            -0.026703599840402603,
+            -0.003113354789093137,
+            -0.02228914201259613,
+            -0.0022323993034660816,
+            -0.020058678463101387,
+            -0.014056564308702946,
+            0.001268188701942563,
+            0.029987338930368423,
+            0.016201836988329887,
+            0.008224832825362682,
+            -0.011051635257899761,
+            -0.029956359416246414,
+            -0.024271776899695396,
+            0.01810702309012413,
+            -0.007055388763546944,
+            -0.005541306920349598,
+            0.00591692328453064,
+            -0.009951893240213394,
+            -0.008480406366288662,
+            0.011802867986261845,
+            0.027137301862239838,
+            -0.002747419523075223,
+            -0.029987338930368423,
+            -0.02688947319984436,
+            -0.014335372485220432,
+            -0.01840132102370262,
+            -0.017704300582408905,
+            -0.008627555333077908,
+            0.0045035220682621,
+            0.00030300996149890125,
+            0.016186347231268883,
+            0.014962689951062202,
+            -0.000480653194244951,
+            0.0028597170021384954,
+            -2.7711375878425315e-05,
+            -0.001763847190886736,
+            0.006888878531754017,
+            -0.004933350719511509,
+            0.026997897773981094,
+            0.014614180661737919,
+            -0.006594581063836813,
+            0.018711106851696968,
+            0.028314489871263504,
+            0.020430423319339752,
+            0.003157886676490307,
+            -0.010695381090044975,
+            -0.014606435783207417,
+            0.009727297350764275,
+            -0.013421501964330673,
+            0.023264970630407333,
+            0.022552460432052612,
+            -0.01480779703706503,
+            -0.014885243959724903,
+            -0.017905661836266518,
+            0.015210519544780254,
+            -0.007562664337456226,
+            -0.008224832825362682,
+            0.012368228286504745,
+            0.022335609421133995,
+            -0.007024410180747509,
+            0.002371803391724825,
+            -0.0010232636705040932,
+            -0.021731525659561157,
+            -0.006122156511992216,
+            -0.004596457816660404,
+            -0.008666278794407845,
+            0.008681767620146275,
+            0.008131897076964378,
+            -0.001928421319462359,
+            -0.006319645792245865,
+            0.005471604876220226,
+            -0.013367289677262306,
+            0.01301103550940752,
+            -0.00789568480104208,
+            0.03379771113395691,
+            -0.009138703346252441,
+            -0.0006733016925863922,
+            -0.022598929703235626,
+            -0.011446612887084484,
+            0.00845717266201973,
+            0.0017967619933187962,
+            0.01682141050696373,
+            -0.0011626676423475146,
+            -0.005688455421477556,
+            0.009913169778883457,
+            -0.0010687635513022542,
+            -0.001450188341550529,
+            -0.006892750971019268,
+            -0.010401083156466484,
+            0.0002884887217078358,
+            0.013258864171802998,
+            -0.006025348324328661,
+            0.0018054747488349676,
+            0.006095050368458033,
+            -0.0175339188426733,
+            0.029027000069618225,
+            0.023884542286396027,
+            0.017255110666155815,
+            0.014660648070275784,
+            0.0006394187803380191,
+            0.011647974140942097,
+            -0.003572226269170642,
+            0.0038820127956569195,
+            -0.020337486639618874,
+            0.021080974489450455,
+            -0.010811550542712212,
+            0.01846327818930149,
+            0.011655719019472599,
+            -0.015233753249049187,
+            -0.005022414494305849,
+            0.021406250074505806,
+            0.009967382065951824,
+            0.00889087375253439,
+            0.0017541663255542517,
+            -0.02470547705888748,
+            0.017766257748007774,
+            0.010099041275680065,
+            -0.020399443805217743,
+            0.019547531381249428,
+            -0.009471723809838295,
+            0.01233724970370531,
+            -0.0005300253978930414,
+            0.004418330732733011,
+            0.0058743273839354515,
+            0.021499186754226685,
+            0.011570527218282223,
+            -0.03884723410010338,
+            -0.013615119270980358,
+            -0.0017357728211209178,
+            0.006857899948954582,
+            -0.0036303112283349037,
+            -0.015233753249049187,
+            -0.041635312139987946,
+            0.01604694314301014,
+            0.009998360648751259,
+            -0.01643417589366436,
+            0.004987563472241163,
+            -0.002427952131256461,
+            -0.0042053526267409325,
+            -0.008170620538294315,
+            0.003186929039657116,
+            -0.004170501604676247,
+            0.02324948087334633,
+            0.0018984107300639153,
+            -0.006946963258087635,
+            0.013258864171802998,
+            0.01344473659992218,
+            -0.021948376670479774,
+            0.0009874446550384164,
+            0.0021375271026045084,
+            0.000636998622212559,
+            0.023683181032538414,
+            -0.014707116410136223,
+            -0.0017657832941040397,
+            -0.01685238815844059,
+            -0.0013069120468571782,
+            -0.021592123433947563,
+            -0.001188805908896029,
+            -0.015187285840511322,
+            -0.008511385880410671,
+            0.025030752643942833,
+            -0.011005166918039322,
+            0.010997422970831394,
+            -0.006687517277896404,
+            0.006943091284483671,
+            -0.006904367823153734,
+            -0.01936165988445282,
+            -0.0008267428493127227,
+            0.009975126944482327,
+            -0.000512115890160203,
+            0.002941036131232977,
+            8.482827252009884e-05,
+            0.002464739140123129,
+            0.008867640048265457,
+            0.006234454456716776,
+            -0.0075278133153915405,
+            0.02489134855568409,
+            -0.012073930352926254,
+            0.0016399325104430318,
+            0.011911292560398579,
+            0.0004743606550619006,
+            -0.010424317792057991,
+            0.025588368996977806,
+            -0.02139076218008995,
+            -0.00800798274576664,
+            -0.00014799568452872336,
+            -0.0016544538084417582,
+            -0.006513262167572975,
+            0.006718495860695839,
+            0.02946070022881031,
+            0.0062654330395162106,
+            0.010447551496326923,
+            -0.0281441081315279,
+            -0.011493081226944923,
+            0.012538610957562923,
+            -0.022273652255535126,
+            -0.0007739823195151985,
+            -0.0004322490422055125,
+            -0.00619960343465209,
+            0.026378324255347252,
+            -0.02022906206548214,
+            -0.01093546487390995,
+            -0.018664639443159103,
+            0.005405775271356106,
+            -0.011121337302029133,
+            0.008294534869492054,
+            -0.0009777637897059321,
+            0.006540368776768446,
+            0.016805920749902725,
+            -0.019563021138310432,
+            0.016031453385949135,
+            0.005119222681969404,
+            -0.016217326745390892,
+            0.010222956538200378,
+            -0.017580386251211166,
+            -0.0008199662552215159,
+            0.020275529474020004,
+            -0.00400399137288332,
+            0.008751469664275646,
+            -0.003363120136782527,
+            0.005208286456763744,
+            -0.006513262167572975,
+            0.0001492057926952839,
+            -0.007249005138874054,
+            0.014428308233618736,
+            -0.016805920749902725,
+            -0.017828216776251793,
+            -0.008821171708405018,
+            0.0016709112096577883,
+            0.021297825500369072,
+            0.011136826127767563,
+            -0.010044828988611698,
+            -0.011059380136430264,
+            -0.011694442480802536,
+            0.041604332625865936,
+            0.005762029904872179,
+            0.027230238541960716,
+            -0.02261441759765148,
+            0.00545998802408576,
+            0.025170156732201576,
+            -0.014242436736822128,
+            0.0012788375606760383,
+            -0.0022788674104958773,
+            0.0053050946444272995,
+            0.011562783271074295,
+            -0.01819995976984501,
+            0.0049139889888465405,
+            -0.00019216447253711522,
+            -0.00782985519617796,
+            -0.000656844291370362,
+            0.003442503046244383,
+            0.00020196630794089288,
+            0.016325751319527626,
+            -0.004027225077152252,
+            -0.016511622816324234,
+            -0.004429947584867477,
+            -0.0003216455806978047,
+            -0.027896279469132423,
+            0.013514438644051552,
+            0.016589069738984108,
+            0.01566745527088642,
+            0.027989214286208153,
+            -0.014513500034809113,
+            0.016031453385949135,
+            0.009533680975437164,
+            0.013901671394705772,
+            -0.010610189288854599,
+            0.008952830918133259,
+            -0.01595400646328926,
+            0.007446494419127703,
+            -0.03454120084643364,
+            -0.007887939922511578,
+            -0.00802347157150507,
+            0.01984182931482792,
+            0.028763681650161743,
+            0.002923610620200634,
+            0.006493900436908007,
+            0.000196883876924403,
+            0.0024202074855566025,
+            -0.01714668609201908,
+            -0.020182594656944275,
+            -0.0003337466041557491,
+            0.0029332914855331182,
+            0.0006950835813768208,
+            -0.004495777189731598,
+            0.00472037261351943,
+            -0.012391461990773678,
+            0.0026835259050130844,
+            -0.013134949840605259,
+            0.005971135571599007,
+            -0.0059982421807944775,
+            0.009603383019566536,
+            0.0019429425010457635,
+            0.020864123478531837,
+            0.02300165221095085,
+            0.004100799560546875,
+            0.0046971384435892105,
+            -0.024333734065294266,
+            0.00920066051185131,
+            -0.013847459107637405,
+            0.012329504825174809,
+            0.019408127292990685,
+            0.008248066529631615,
+            0.006571347359567881,
+            0.01233724970370531,
+            0.012918098829686642,
+            -0.0031772484071552753,
+            0.024287264794111252,
+            0.005014669615775347,
+            -0.0009070937521755695,
+            0.002052335999906063,
+            -0.014954946003854275,
+            -0.0070979841984808445,
+            -0.0015286030247807503,
+            0.0018848575418815017,
+            -0.004731989465653896,
+            0.01906736195087433,
+            0.015256987884640694,
+            -0.01050176378339529,
+            0.017874684184789658,
+            -0.03937387093901634,
+            0.017379024997353554,
+            -0.0027377386577427387,
+            -0.0060911779291927814,
+            -0.00545998802408576,
+            0.00665653869509697,
+            0.004515138920396566,
+            0.003669034456834197,
+            0.03773200139403343,
+            -0.010408828034996986,
+            0.003963331691920757,
+            -0.005847221240401268,
+            0.007121218368411064,
+            -0.020864123478531837,
+            0.008302279748022556,
+            0.005336073227226734,
+            0.02160761132836342,
+            -0.0056497324258089066,
+            -0.0014482521219179034,
+            0.02328045852482319,
+            0.008859895169734955,
+            0.03469609469175339,
+            -0.01566745527088642,
+            -0.004418330732733011,
+            0.01048627495765686,
+            0.0207402091473341,
+            -0.00580462533980608,
+            -0.008976065553724766,
+            0.007903429679572582,
+            0.0035586729645729065,
+            0.021824462339282036,
+            -0.00515407370403409,
+            -0.0006650729919783771,
+            0.0054367538541555405,
+            -0.0060911779291927814,
+            0.009735042229294777,
+            0.011252996511757374,
+            -0.009076746180653572,
+            0.007539430167526007,
+            0.021530164405703545,
+            -0.009975126944482327,
+            -0.008658533915877342,
+            -0.009649851359426975,
+            0.0010658593382686377,
+            -0.03068435750901699,
+            0.001649613375775516,
+            0.005359307397156954,
+            0.015187285840511322,
+            -0.0004603234410751611,
+            -0.009014789015054703,
+            0.021019017323851585,
+            0.017255110666155815,
+            -0.005936284549534321,
+            -0.0028132491279393435,
+            0.0023795480374246836,
+            -0.0017154429806396365,
+            -0.006160879973322153,
+            -0.0006248975405469537,
+            -0.002656419761478901,
+            0.014730350114405155,
+            -0.0015469965292140841,
+            -0.006617815233767033,
+            -0.00756653631106019,
+            0.0033302053343504667,
+            0.010834784246981144,
+            -0.005661349277943373,
+            0.013057502917945385,
+            0.016805920749902725,
+            -0.00031704717548564076,
+            0.019671445712447166,
+            0.01037010457366705,
+            -0.010447551496326923,
+            0.015938518568873405,
+            -0.007984748110175133,
+            0.006563602481037378,
+            0.0164031982421875,
+            -0.01277095079421997,
+            -0.01246116403490305,
+            0.007210282143205404,
+            0.00665653869509697,
+            0.028918573632836342,
+            0.014792308211326599,
+            -0.02109646424651146,
+            0.014908477663993835,
+            0.015768134966492653,
+            -0.013018779456615448,
+            0.003537375247105956,
+            -0.01784370467066765,
+            0.020260039716959,
+            0.019051872193813324,
+            -0.015535795129835606,
+            -0.017224133014678955,
+            -0.014947201125323772,
+            -0.005092116538435221,
+            0.006331262644380331,
+            0.0033263331279158592,
+            -0.00617636926472187,
+            -0.027989214286208153,
+            -0.0049139889888465405,
+            -0.018633661791682243,
+            0.013870692811906338,
+            -0.008674023672938347,
+            0.0007715621031820774,
+            -0.004511266481131315,
+            0.007984748110175133,
+            -0.0033882902935147285,
+            0.032465629279613495,
+            -0.0051811798475682735,
+            -0.02832997962832451,
+            0.004398969002068043,
+            0.016356728971004486,
+            0.012623801827430725,
+            -0.029243851080536842,
+            0.010463040322065353,
+            0.004805563949048519,
+            -0.012530866079032421,
+            0.005990497302263975,
+            -0.02258343994617462,
+            -0.007632365915924311,
+            -0.0009395244996994734,
+            -0.016743963584303856,
+            -0.027121812105178833,
+            -0.008433938957750797,
+            -0.006993431597948074,
+            -0.005401902832090855,
+            -0.009525936096906662,
+            -0.008720491081476212,
+            -0.011477591469883919,
+            0.02303262986242771,
+            -0.007899557240307331,
+            -0.001866463921032846,
+            -0.0014637414133176208,
+            0.00930908601731062,
+            -0.023776117712259293,
+            -0.015605497173964977,
+            0.006389347370713949,
+            -0.002594462363049388,
+            0.022815778851509094,
+            -0.012747716158628464,
+            0.005940156988799572,
+            -0.009316830895841122,
+            0.009851212613284588,
+            0.015241498127579689,
+            0.004956584889441729,
+            0.017440982162952423,
+            -0.009889936074614525,
+            0.0021026763133704662,
+            -0.008627555333077908,
+            -0.00018986526993103325,
+            -1.027075541060185e-05,
+            0.014428308233618736,
+            0.007624621503055096,
+            0.014141756109893322,
+            0.017487451434135437,
+            -0.013553161174058914,
+            0.005003052763640881,
+            -0.0003274540649726987,
+            -0.017255110666155815,
+            0.0027725896798074245,
+            -0.007984748110175133,
+            -0.0008896682411432266,
+            0.001758038648404181,
+            6.395397940650582e-05,
+            -0.02112744189798832,
+            -0.0009124181815423071,
+            0.0018809852190315723,
+            0.0010222955606877804,
+            -0.025201136246323586,
+            0.00889087375253439,
+            -0.00889087375253439,
+            -0.0014918158994987607,
+            -0.019113829359412193,
+            0.019795360043644905,
+            0.026300877332687378,
+            -0.004619691986590624,
+            0.00036206303047947586,
+            -0.013537672348320484,
+            -0.02049238048493862,
+            0.0001489637652412057,
+            -0.011314953677356243,
+            -0.011345932260155678,
+            -0.0022169100120663643,
+            -0.003214035416021943,
+            0.008805682882666588,
+            0.0004189379105810076,
+            -0.009433000348508358,
+            -0.004956584889441729,
+            -0.011764144524931908,
+            -0.017549408599734306,
+            -0.010401083156466484,
+            0.01933068037033081,
+            -0.0018016024259850383,
+            0.0020794421434402466,
+            -0.0032508226577192545,
+            0.0005271211848594248,
+            0.011748654767870903,
+            0.013475715182721615,
+            0.013344055972993374,
+            -0.023915521800518036,
+            -0.017890173941850662,
+            -0.013320821337401867,
+            0.0029487807769328356,
+            -0.007187047973275185,
+            0.004844287410378456,
+            0.026362836360931396,
+            -0.0009695350890979171,
+            -0.01091997604817152,
+            0.002327271504327655,
+            0.019051872193813324,
+            -0.031272951513528824,
+            -0.013390523381531239,
+            -0.025077221915125847,
+            0.02225816436111927,
+            -0.013351799920201302,
+            0.011516314931213856,
+            0.010191977955400944,
+            -0.025944624096155167,
+            0.006768836174160242,
+            -0.007686578668653965,
+            0.026037558913230896,
+            0.01634124107658863,
+            0.0005934348446317017,
+            -0.00944074522703886,
+            0.013622863218188286,
+            -0.0008722427301108837,
+            -0.013576395809650421,
+            0.002201420720666647,
+            0.014645159244537354,
+            -0.014691627584397793,
+            -0.009510447271168232,
+            -0.004507394507527351,
+            0.004313777666538954,
+            0.012763205915689468,
+            -0.0030339721124619246,
+            -0.004031097516417503,
+            -0.011624740436673164,
+            0.013344055972993374,
+            -0.003930416889488697,
+            0.012383717112243176,
+            0.007380664348602295,
+            -0.014056564308702946,
+            -0.019532041624188423,
+            0.033333033323287964,
+            0.00400399137288332,
+            0.00338054564781487,
+            -0.010997422970831394,
+            0.02740062028169632,
+            -0.009897680021822453,
+            0.010571465827524662,
+            -0.006408709101378918,
+            0.0031985461246222258,
+            0.012321759946644306,
+            -0.012104909867048264,
+            -0.01981084980070591,
+            0.021468207240104675,
+            -0.00364192808046937,
+            -0.0010193913476541638,
+            0.008310024626553059,
+            -0.020182594656944275,
+            -0.00663330452516675,
+            0.002180122770369053,
+            0.009541425853967667,
+            -0.011818356812000275,
+            0.006369986105710268,
+            0.017905661836266518,
+            0.003912991378456354,
+            -0.0007672057254239917,
+            0.019996721297502518,
+            -0.01672847382724285,
+            0.011717676185071468,
+            -0.010850274004042149,
+            0.0016825281782075763,
+            0.0006065039779059589,
+            -0.0048597767017781734,
+            0.0058743273839354515,
+            -0.014536733739078045,
+            0.011988739483058453,
+            -6.244135147426277e-05,
+            -0.013979118317365646,
+            0.007601387333124876,
+            -0.013359544798731804,
+            -0.016836898401379585,
+            0.041542377322912216,
+            -0.005301222205162048,
+            -0.0038665232714265585,
+            -0.0056962003000080585,
+            0.01350669376552105,
+            0.02823704481124878,
+            -0.005336073227226734,
+            -0.01153180468827486,
+            -0.010780571959912777,
+            -0.016573579981923103,
+            0.015907539054751396,
+            -0.01810702309012413,
+            0.002495717955753207,
+            -0.010532742366194725,
+            -0.007903429679572582,
+            0.004302160814404488,
+            -0.003516077296808362,
+            0.014722606167197227,
+            0.00950270239263773,
+            0.023357905447483063,
+            0.00797700323164463,
+            -0.00027348342700861394,
+            0.010393339209258556,
+            0.011663463898003101,
+            0.003243078012019396,
+            -0.010509508661925793,
+            -0.00011949774489039555,
+            -0.02894955314695835,
+            -0.008449427783489227,
+            -0.004863648675382137,
+            -0.001485039247199893,
+            -0.00428667152300477,
+            -0.01239920686930418,
+            0.006617815233767033,
+            -0.0022052929271012545,
+            0.013909416273236275,
+            -0.009254872798919678,
+            -0.017766257748007774,
+            0.03389064967632294,
+            0.004495777189731598,
+            0.006466794293373823,
+            -0.008286789990961552,
+            -0.03705047070980072,
+            -0.0032566310837864876,
+            -0.008116407319903374,
+            -0.013529927469789982,
+            0.004538373090326786,
+            0.01283290795981884,
+            -0.014831030741333961,
+            -0.028097640722990036,
+            0.0019390701781958342,
+            -0.006683644838631153,
+            0.02978597767651081,
+            -0.007798876613378525,
+            0.00419760774821043,
+            0.0037251831963658333,
+            0.0011558911064639688,
+            0.01239920686930418,
+            -0.00845717266201973,
+            -0.0022536972537636757,
+            0.016387708485126495,
+            -0.010858018882572651,
+            0.011051635257899761,
+            -0.010966443456709385,
+            -0.02554190158843994,
+            -0.01456771232187748,
+            0.021344292908906937,
+            -0.0004823473282158375,
+            -0.00024903935263864696,
+            0.014823286794126034,
+            0.02354377694427967,
+            0.011260741390287876,
+            0.00969631876796484,
+            0.006408709101378918,
+            -0.001035848748870194,
+            0.005975008010864258,
+            0.006095050368458033,
+            -0.0024163350462913513,
+            -0.013398268260061741,
+            0.017564896494150162,
+            0.009773765690624714,
+            -0.01810702309012413,
+            0.0008514289511367679,
+            0.0036612898111343384,
+            0.0168833676725626,
+            0.006935346405953169,
+            0.009316830895841122,
+            -0.0143121387809515,
+            -0.023760627955198288,
+            -0.00014932679187040776,
+            0.002873270306736231,
+            0.015721667557954788,
+            0.010579210706055164,
+            -0.014319883659482002,
+            0.0005963391158729792,
+            0.008712747134268284,
+            0.020213572308421135,
+            -0.024209819734096527,
+            0.019702425226569176,
+            -0.004673904739320278,
+            -0.0021084847394376993,
+            0.007946024648845196,
+            0.011632485315203667,
+            -0.011872569099068642,
+            -0.00650551775470376,
+            0.006249943748116493,
+            0.003157886676490307,
+            0.03172214329242706,
+            -0.0017260919557884336,
+            0.009665340185165405,
+            -0.010997422970831394,
+            -0.0002405686245765537,
+            0.0006728176376782358,
+            -0.0006820144481025636,
+            -0.002884887158870697,
+            -0.011020656675100327,
+            0.0018974426202476025,
+            -0.020306508988142014,
+            0.0024473138619214296,
+            0.00463130883872509,
+            -0.005684582982212305,
+            -0.0032972905319184065,
+            -0.019222255796194077,
+            -0.0022672503255307674,
+            0.003293418325483799,
+            0.014335372485220432,
+            -0.031691163778305054,
+            0.004972074180841446,
+            0.010672146454453468,
+            -0.002309845993295312,
+            -0.010416572913527489,
+            0.011423379182815552,
+            -0.0026060794480144978,
+            0.011988739483058453,
+            -0.0038142469711601734,
+            -0.0004905760288238525,
+            0.025681305676698685,
+            -0.005308967083692551,
+            0.014707116410136223,
+            -0.01026167906820774,
+            -0.002993312431499362,
+            -0.028252532705664635,
+            0.005235392600297928,
+            -0.007403898518532515,
+            -0.0012081675231456757,
+            -0.004402841441333294,
+            -0.01130720879882574,
+            -0.0026990154292434454,
+            0.002122037811204791,
+            0.008000237867236137,
+            0.012910354882478714,
+            -0.0007725302129983902,
+            -0.010075807571411133,
+            0.004832670092582703,
+            0.0075355577282607555,
+            -0.015520306304097176,
+            -0.0070437719114124775,
+            -0.020306508988142014,
+            0.029662061482667923,
+            -0.013731288723647594,
+            -0.0011123273288831115,
+            -0.020864123478531837,
+            0.010160998441278934,
+            -0.0063119009137153625,
+            -0.002782270545139909,
+            0.004708755761384964,
+            -0.015109838917851448,
+            0.01344473659992218,
+            0.008751469664275646,
+            0.01450575515627861,
+            -0.02151467651128769,
+            -0.01762685552239418,
+            -0.0044725434854626656,
+            -0.002735802438110113,
+            -0.010648912750184536,
+            -0.0005334137240424752,
+            0.002013612538576126,
+            0.021251358091831207,
+            -0.00032503384863957763,
+            0.0042053526267409325,
+            -0.003214035416021943,
+            0.017224133014678955,
+            -0.010137764737010002,
+            -0.013057502917945385,
+            0.0024260159116238356,
+            0.005994369741529226,
+            0.016186347231268883,
+            0.03158273920416832,
+            -0.020693741738796234,
+            -0.00272999401204288,
+            0.0032740565948188305,
+            0.010656657628715038,
+            -0.004085310269147158,
+            6.304640555754304e-05,
+            -0.027803342789411545,
+            -0.04464024305343628,
+            -0.002422143705189228,
+            0.013057502917945385,
+            -0.015636475756764412,
+            0.012670270167291164,
+            0.008743725717067719,
+            0.010021595284342766,
+            -0.005196669604629278,
+            0.02620794251561165,
+            -0.004793947096914053,
+            0.014304393902420998,
+            0.011733165942132473,
+            0.011911292560398579,
+            0.0009874446550384164,
+            0.0007459079497493804,
+            -0.018354853615164757,
+            -0.008255811408162117,
+            -0.0015954007394611835,
+            0.01765783317387104,
+            -0.027478067204356194,
+            0.008526874706149101,
+            -0.011516314931213856,
+            0.013839714229106903,
+            0.0003666614356916398,
+            0.005661349277943373,
+            -0.01771979033946991,
+            0.012445674277842045,
+            -0.0009337160154245794,
+            -0.025325050577521324,
+            0.004805563949048519,
+            0.011276230216026306,
+            -0.010362359695136547,
+            0.006877261213958263,
+            0.019950253888964653,
+            0.0023079097736626863,
+            0.008495896123349667,
+            -0.014126266352832317,
+            0.010703125037252903,
+            0.009649851359426975,
+            -0.021080974489450455,
+            -0.01233724970370531,
+            -0.007059261202812195,
+            -0.006335135083645582,
+            -0.0035470561124384403,
+            -0.00013105422840453684,
+            0.010176488198339939,
+            -0.016000475734472275,
+            0.010099041275680065,
+            -0.006559730041772127,
+            0.006400964688509703,
+            0.00944074522703886,
+            0.001115231541916728,
+            -0.013576395809650421,
+            0.012716737575829029,
+            -0.0218709297478199,
+            0.008426194079220295,
+            -0.04287445917725563,
+            -0.005847221240401268,
+            -0.015078860335052013,
+            0.005204414017498493,
+            -0.003798757679760456,
+            -0.004453181754797697,
+            -0.006203475873917341,
+            -0.005061137955635786,
+            0.00920066051185131,
+            0.021747015416622162,
+            0.012135888449847698,
+            -0.016620049253106117,
+            -0.007551047019660473,
+            0.005425137002021074,
+            0.0004854935978073627,
+            -0.013336311094462872,
+            0.009316830895841122,
+            -0.01576039008796215,
+            0.0009259713697247207,
+            0.006339007057249546,
+            -0.01072635967284441,
+            0.013452480547130108,
+            0.004731989465653896,
+            -0.003401843598112464,
+            0.013390523381531239,
+            -0.0050417762249708176,
+            -0.0057891360484063625,
+            -0.007272239308804274,
+            -0.013739033602178097,
+            -0.003818119177594781,
+            -0.003072695340961218,
+            0.012747716158628464,
+            0.001529571134597063,
+            -0.0003243077953811735,
+            -0.00782985519617796,
+            0.009154192171990871,
+            0.002220782218500972,
+            -0.0013146566925570369,
+            0.005773646757006645,
+            -0.02300165221095085,
+            0.0001287550403503701,
+            0.004755223635584116,
+            -0.0009487213101238012,
+            -0.006246071308851242,
+            -0.012151377275586128,
+            -0.011903548613190651,
+            -0.038165703415870667,
+            0.009363298304378986,
+            0.0019197085639461875,
+            -0.0076207490637898445,
+            0.0064822835847735405,
+            0.017255110666155815,
+            0.01456771232187748,
+            0.016743963584303856,
+            -0.020693741738796234,
+            -0.02303262986242771,
+            0.005556796211749315,
+            -0.009402021765708923,
+            0.02261441759765148,
+            0.004046586807817221,
+            -0.005847221240401268,
+            0.0013243375578895211,
+            -0.013258864171802998,
+            0.008929597213864326,
+            0.005723306443542242,
+            -0.001592496526427567,
+            0.005425137002021074,
+            -0.013692565262317657,
+            -0.010904486291110516,
+            0.01048627495765686,
+            0.032496608793735504,
+            -0.0004261985304765403,
+            0.0024105266202241182,
+            0.023295948281884193,
+            0.015466093085706234,
+            0.016356728971004486,
+            -0.021111954003572464,
+            0.011167804710566998,
+            0.010060317814350128,
+            -0.0018645278178155422,
+            -0.014892988838255405,
+            0.00513083953410387,
+            -0.012716737575829029,
+            -0.010664401575922966,
+            0.014993669465184212,
+            0.007853088900446892,
+            0.017735280096530914,
+            -0.02369867078959942,
+            0.00789568480104208,
+            0.00042765066609717906,
+            0.00013710475468542427,
+            0.011524059809744358,
+            0.012956822291016579,
+            0.03370477631688118,
+            0.027152791619300842,
+            0.020662762224674225,
+            -0.009285852313041687,
+            0.014366351068019867,
+            -0.009192915633320808,
+            -0.011090358719229698,
+            -0.019470084458589554,
+            -0.015249243006110191,
+            -0.011136826127767563,
+            0.017998598515987396,
+            0.008790193125605583,
+            -0.009355553425848484,
+            -0.005947901867330074,
+            0.01586107164621353,
+            -0.003909118939191103,
+            -0.004635181277990341,
+            0.01646515540778637,
+            -0.026269899681210518,
+            0.016836898401379585,
+            -0.000745423894841224,
+            -0.014668392948806286,
+            -0.014691627584397793,
+            -0.004182118456810713,
+            -0.012670270167291164,
+            0.0003964299685321748,
+            -0.020817656069993973,
+            0.013181418180465698,
+            0.024597052484750748,
+            -0.0030920570716261864,
+            -0.004100799560546875,
+            -0.011299464851617813,
+            -0.0005111477803438902,
+            0.018385831266641617,
+            0.0034773540683090687,
+            0.009192915633320808,
+            -0.0023872926831245422,
+            0.013924905098974705,
+            0.00524313747882843,
+            0.0027958236169070005,
+            -0.0007565568666905165,
+            -0.016263794153928757,
+            0.012197845615446568,
+            0.0026835259050130844,
+            0.012135888449847698,
+            -0.022242674604058266,
+            0.007938280701637268,
+            -0.011764144524931908,
+            -0.0023698671720921993,
+            -0.02228914201259613,
+            0.0189744271337986,
+            -0.01610890030860901,
+            0.001046497724018991,
+            0.003841353114694357,
+            0.0006224773242138326,
+            0.009580149315297604,
+            0.01610890030860901,
+            -0.023388884961605072,
+            -0.0003460896550677717,
+            -0.014908477663993835,
+            -0.021917399019002914,
+            -0.01607792265713215,
+            0.004728117026388645,
+            -0.019501063972711563,
+            0.005444498732686043,
+            -0.0025557391345500946,
+            -0.015125328674912453,
+            0.018478767946362495,
+            0.014691627584397793,
+            -0.004379607271403074,
+            0.002412462839856744,
+            0.005425137002021074,
+            -0.0014405074762180448,
+            0.01971791312098503,
+            0.00013274837692733854,
+            -0.0010048701660707593,
+            -0.009425255469977856,
+            -0.0168833676725626,
+            -0.0065752193331718445,
+            -0.035005878657102585,
+            0.0260530486702919,
+            -0.004670032300055027,
+            -0.016511622816324234,
+            0.014319883659482002,
+            0.009115469641983509,
+            -0.014629669487476349,
+            -0.007059261202812195,
+            0.016294771805405617,
+            -0.024287264794111252,
+            0.02244403585791588,
+            -0.009611127898097038,
+            -0.008774704299867153,
+            0.00784921646118164,
+            0.05154848098754883,
+            0.009146448224782944,
+            0.003492843359708786,
+            0.011036145500838757,
+            0.005839476361870766,
+            -0.0013165927957743406,
+            -0.023683181032538414,
+            0.005568413063883781,
+            -0.007190920412540436,
+            -0.003601268632337451,
+            0.002536377403885126,
+            -0.004801691509783268,
+            0.031149037182331085,
+            -0.004379607271403074,
+            -0.005289605353027582,
+            0.004484160337597132,
+            -0.02196386642754078,
+            -0.0001332324172835797,
+            0.023203013464808464,
+            -2.3128106477088295e-05,
+            -0.023590246215462685,
+            -0.006571347359567881,
+            -0.015071115456521511,
+            -0.013770012184977531,
+            -0.018989915028214455,
+            0.005231520161032677,
+            -0.012755461037158966,
+            -0.0033940989524126053,
+            0.0016660707769915462,
+            -0.005940156988799572,
+            -0.002582845278084278,
+            0.009603383019566536,
+            0.05362405255436897,
+            0.014103032648563385,
+            -0.006335135083645582,
+            -0.005854965653270483,
+            0.009192915633320808,
+            -0.010524998418986797,
+            -0.019795360043644905,
+            -0.009820234030485153,
+            -0.029507169499993324,
+            -0.012747716158628464,
+            -0.0143121387809515,
+            -0.010191977955400944,
+            -0.010277168825268745,
+            0.00963436160236597,
+            -0.018122512847185135,
+            -0.007628493942320347,
+            -0.020337486639618874,
+            0.002941036131232977,
+            -0.011268485337495804,
+            -0.014714861288666725,
+            -0.008054450154304504,
+            -0.04807887226343155,
+            0.005796880926936865,
+            0.019593998789787292,
+            0.01263929158449173,
+            0.002515079453587532,
+            0.004383479710668325,
+            -0.01993476413190365,
+            -0.030792783945798874,
+            0.013839714229106903,
+            0.011283975094556808,
+            0.011733165942132473,
+            0.03283737599849701,
+            -0.011709931306540966,
+            -0.00813964195549488,
+            -0.004693266469985247,
+            -0.0017251238459721208,
+            0.012066186405718327,
+            0.00969631876796484,
+            -0.004917861428111792,
+            -0.012375972233712673,
+            0.016217326745390892,
+            -0.01981084980070591,
+            -0.023683181032538414,
+            -0.01672847382724285,
+            0.006157007534056902,
+            0.00813964195549488,
+            -0.008054450154304504,
+            0.002973950933665037,
+            0.0056962003000080585,
+            -0.007159941829741001,
+            -0.00022423222253564745,
+            0.004890755284577608,
+            -0.023048119619488716,
+            0.0008490087930113077,
+            -0.002257569460198283,
+            0.0061376458033919334,
+            0.021266845986247063,
+            -0.010168743319809437,
+            -0.0014075926737859845,
+            -0.0053902859799563885,
+            -0.0028984404634684324,
+            -0.006764963734894991,
+            0.02620794251561165,
+            0.009154192171990871,
+            -0.009882191196084023,
+            -0.01011453103274107,
+            -0.01129171997308731,
+            -0.0011278166202828288,
+            -0.017255110666155815,
+            -0.016186347231268883,
+            0.0015566773945465684,
+            -0.016712984070181847,
+            0.013971373438835144,
+            -0.00876695942133665,
+            0.00400399137288332,
+            -0.014110777527093887,
+            0.02019808255136013,
+            0.01929970271885395,
+            -0.009711808525025845,
+            0.023776117712259293,
+            -0.007709812838584185,
+            -0.038382552564144135,
+            -0.00800798274576664,
+            0.01843230053782463,
+            -0.006118284538388252,
+            0.010037084110081196,
+            -0.011314953677356243,
+            0.008209343999624252,
+            0.00015319912927225232,
+            0.0018122512847185135,
+            -0.0003112386621069163,
+            0.027555514127016068,
+            0.015535795129835606,
+            -0.005560668651014566,
+            -0.005494839046150446,
+            0.004255692940205336,
+            0.009541425853967667,
+            0.017580386251211166,
+            0.012151377275586128,
+            -0.021019017323851585,
+            -0.014257925562560558,
+            -0.016263794153928757,
+            0.011012911796569824,
+            0.015125328674912453,
+            -0.0040891822427511215,
+            0.01691434532403946,
+            0.014126266352832317,
+            -0.019098341464996338,
+            0.0019884423818439245,
+            -0.008193854242563248,
+            -0.00158184755127877,
+            -0.03029712475836277,
+            0.011043890379369259,
+            0.0074387495405972,
+            -0.0072218989953398705,
+            0.036616768687963486,
+            0.009858956560492516,
+            0.037577107548713684,
+            0.018680129200220108,
+            0.0008974128868430853,
+            0.009045767597854137,
+            -0.018370341509580612,
+            0.013065247796475887,
+            0.01500141341239214,
+            -0.012724482454359531,
+            -0.005734923295676708,
+            0.0013446672819554806,
+            0.0007551047019660473,
+            0.012631546705961227,
+            0.02004319056868553,
+            -0.01795213110744953,
+            0.009773765690624714,
+            -0.015032391995191574,
+            -0.005250881891697645,
+            0.006555858068168163,
+            -0.013932649977505207,
+            -0.008495896123349667,
+            -0.00963436160236597,
+            0.019408127292990685,
+            -0.012600568123161793,
+            0.016573579981923103,
+            -0.009595638141036034,
+            0.015520306304097176,
+            -0.029724018648266792,
+            -0.003669034456834197,
+            -0.006679772399365902,
+            -0.019593998789787292,
+            -0.012887120246887207,
+            0.009820234030485153,
+            0.010633422993123531,
+            -0.009301341138780117,
+            -0.011562783271074295,
+            0.020368466153740883,
+            -0.004185990896075964,
+            0.013529927469789982,
+            0.021189400926232338,
+            0.00737291993573308,
+            0.024132372811436653,
+            0.0005605200421996415,
+            0.014164989814162254,
+            0.005463860463351011,
+            0.010362359695136547,
+            -0.004677776712924242,
+            0.005161818582564592,
+            0.00932457484304905,
+            0.014203713275492191,
+            0.017766257748007774,
+            0.013196907006204128,
+            -0.00598275288939476,
+            0.007419387809932232,
+            0.02016710489988327,
+            0.0016254113288596272,
+            -0.030312614515423775,
+            -0.009719553403556347,
+            0.0001736498816171661,
+            0.00924712885171175,
+            0.009185170754790306,
+            3.817877222900279e-05,
+            -0.01093546487390995,
+            0.011144571006298065,
+            0.00969631876796484,
+            0.002286612056195736,
+            0.006280922330915928,
+            -0.01456771232187748,
+            -0.020957060158252716,
+            0.006106667220592499,
+            0.01798310875892639,
+            0.013917161151766777,
+            -0.014722606167197227,
+            0.022784801200032234,
+            0.006145390681922436,
+            -0.021406250074505806,
+            -0.01840132102370262,
+            -0.0009559819591231644,
+            0.0004542729293461889,
+            -0.002265314105898142,
+            0.03720536455512047,
+            -0.0011539548868313432,
+            -0.005955646280199289,
+            -0.0054754773154854774,
+            0.02775687538087368,
+            0.02049238048493862,
+            0.009177426807582378
+          ],
+          "embedding": [
+            0.0008215984936325879,
+            -0.019349418284688353,
+            -0.02322660633029795,
+            0.009747820845194852,
+            0.007353771220903641,
+            0.015252908783199839,
+            -0.012176175829980202,
+            0.051898051094966605,
+            -0.0068130353886146165,
+            0.008189212816890595,
+            0.008102287831931905,
+            0.01682801136011614,
+            0.02005557996049711,
+            -0.020779879580497183,
+            0.002332165268329781,
+            0.02337485065616377,
+            0.006074992895540845,
+            0.02576575146535689,
+            0.0007517286429767504,
+            -0.03763909942314134,
+            0.0011204729720314833,
+            0.00041212613589632806,
+            -0.06089176800314795,
+            0.012852035873395388,
+            -0.008901127092058543,
+            0.01702347055484216,
+            0.005803446395466281,
+            0.017889570275475296,
+            -0.05375047382413552,
+            0.021623778774252724,
+            0.014784317465883424,
+            -0.0018881880857885708,
+            0.0004735328554290247,
+            0.012393802850407656,
+            -0.019234442968055952,
+            -0.019566283611638122,
+            0.04407789373296535,
+            -0.0053565873209102995,
+            -0.026240866282480922,
+            0.048107902270426675,
+            -0.0054681314939387235,
+            -0.0009441397948041965,
+            -0.015205079626883548,
+            0.006885981841893229,
+            0.01403724256866746,
+            -0.026086548246792754,
+            0.0034778946051893067,
+            0.00044587533948775473,
+            0.032851328435891634,
+            -0.03882097333255334,
+            -0.011126344604254894,
+            -0.01648943336384904,
+            0.035810456555802166,
+            -0.019384589526165754,
+            -0.019430702768170285,
+            0.014297440272239852,
+            -0.007883453204663151,
+            0.050996900793086844,
+            0.06007149434153715,
+            0.03406832053483379,
+            0.025193439934671852,
+            0.00027785129663604897,
+            -0.008014391056168083,
+            -0.018806730908490114,
+            0.027193788903098558,
+            -0.006195006670328595,
+            -0.020466000344352826,
+            0.0068939884219589185,
+            0.017252276167086602,
+            -0.045073444648704136,
+            -0.049242269917908754,
+            0.023021140866008313,
+            -0.015676872472071106,
+            -0.024497517908898602,
+            0.005873633718973869,
+            -0.022828585938203683,
+            0.011748341004940436,
+            0.04624205727927451,
+            -0.015442165499674641,
+            0.018565551071491767,
+            0.015246505395507096,
+            0.004119269796224144,
+            0.004862735250819252,
+            -0.000145323321960046,
+            0.05155838952887077,
+            0.0027052423345323936,
+            -0.016218063670586664,
+            -0.07462470920944148,
+            0.043695974746295135,
+            -0.027608273720377708,
+            -0.03860912774695494,
+            0.002823110565150435,
+            0.026471469849508766,
+            -0.0026063577189074103,
+            -0.029230652014151735,
+            0.05684164470477843,
+            0.00022182868300572677,
+            0.015590845552956364,
+            -0.024939617549529242,
+            0.01035803286203917,
+            -0.010225298012384327,
+            0.019571847648759157,
+            0.01841613041441054,
+            0.0156981470538354,
+            0.021764297694793343,
+            -0.018831230007033416,
+            -0.014419529638789993,
+            -0.0029965475899652273,
+            0.028881185012095884,
+            0.09163526460163937,
+            0.022644302207811818,
+            -0.017933909936330133,
+            -0.04663928709016869,
+            0.004962951444105128,
+            -0.041333041174526026,
+            -0.027545628007617527,
+            -0.037142394579894954,
+            -0.013358143162788402,
+            0.02719686432396677,
+            -0.0034971800937504205,
+            0.04413635817371732,
+            0.04903857110328917,
+            0.03186067155630788,
+            0.007831051050269414,
+            -0.025257538623313574,
+            0.023517513773109414,
+            -0.0016527403209505964,
+            -0.0016101860707429347,
+            0.014448515185903624,
+            0.009905018312477684,
+            0.0024114582235163955,
+            0.00172043851384822,
+            -0.03818316221102173,
+            -0.02304537008110578,
+            0.01024969233570089,
+            -0.01901116356327339,
+            0.019791121460848986,
+            0.07985067340821365,
+            -0.051903191272236665,
+            -0.025924646976556942,
+            0.014766511532849252,
+            -0.002073086076894999,
+            0.030959269926800852,
+            0.006394417986617089,
+            -0.028177586693713442,
+            0.010725285336674526,
+            0.0009614052276924192,
+            0.07500253788978564,
+            -0.00036794182149960157,
+            0.004058458332826519,
+            -0.013218269166951234,
+            0.027613539268683245,
+            -0.012995170442480582,
+            0.023388707018488407,
+            0.026389423313105165,
+            -0.003068639956061266,
+            -0.02661967649632026,
+            0.026145274570635913,
+            -0.03300591322174331,
+            -0.004556072066733024,
+            0.020045722084112896,
+            0.0033437343914784865,
+            -0.010224220905483085,
+            -0.0025025498684079703,
+            0.03460103245395426,
+            0.04514362357278504,
+            0.03797031988058962,
+            -0.004586662475576956,
+            -0.06351723722596923,
+            0.06037983484369874,
+            0.019415753124363902,
+            0.028850618647994476,
+            -0.0034810237045849006,
+            0.02957938154808314,
+            0.02642652523231673,
+            0.013187653724133275,
+            -0.007856320804109569,
+            -0.007065882659482447,
+            -0.016672784625197085,
+            0.029020161154209707,
+            -0.013243058921970134,
+            -0.014900932834241732,
+            0.006001769901896033,
+            0.008648456186944271,
+            0.03484763413464497,
+            0.012279368984306564,
+            0.02379643508687133,
+            0.04933044959399941,
+            -0.021047723935647383,
+            -0.007264823701706426,
+            0.0007058774795716792,
+            -0.010512071832195085,
+            0.027489671668749157,
+            0.009731157570361245,
+            -0.03195919640758579,
+            -0.0033626031972691915,
+            -0.04282419484651207,
+            0.01873977910485268,
+            -0.014398285316695653,
+            0.012854631447494207,
+            -0.0017372959978092432,
+            -0.037331867825735776,
+            0.001088209292238142,
+            -0.012345684323522151,
+            0.037096063326414734,
+            0.005050219690024652,
+            0.0008906051515353004,
+            -0.003697709449630797,
+            0.006548479142737107,
+            -0.010727663712299397,
+            -0.007486226350610755,
+            -0.02704618481069485,
+            -0.019752552016312926,
+            -0.014632581296303486,
+            -0.0037501138119724717,
+            0.014588411795703245,
+            -0.007119137071719376,
+            0.016440258608998535,
+            -0.03686645352111013,
+            -0.023546402448713415,
+            0.005628703432152666,
+            0.024988145751166143,
+            0.011971199595808573,
+            -0.010868778329190568,
+            -0.03270046433624425,
+            -0.021950559285357267,
+            -0.011889914022657738,
+            -0.005123971523701009,
+            0.02619713302280345,
+            -0.021990810305904974,
+            0.01165257474485436,
+            -0.009117901804315863,
+            -0.007492082755249399,
+            0.010725902070291232,
+            0.01862811996181951,
+            -0.026694281868322685,
+            -0.01623776415010199,
+            -0.009403997538684658,
+            -0.02207329027097839,
+            0.025939540497196416,
+            0.014178803271471762,
+            -0.001141821318145744,
+            -0.006281187017652108,
+            -0.035993365198554964,
+            0.0026192776111449633,
+            -0.015254976771874396,
+            -0.0116872453009606,
+            -0.015780016491993135,
+            0.021421328283124968,
+            0.02889432250124136,
+            -0.01871009416296692,
+            -0.026540472640514536,
+            0.015703370682110186,
+            -0.0002849235245164438,
+            -0.02594930889995505,
+            0.04491828744809679,
+            0.044063522197410734,
+            -0.037441864491547405,
+            -0.007005339187980476,
+            -0.030381794393931814,
+            0.06901083845539985,
+            -0.017604082400603037,
+            -0.04060591381445593,
+            0.06044850935418443,
+            -0.01756490171206357,
+            0.025539717592705295,
+            0.03452018789549939,
+            0.0016217577522523927,
+            0.00018825054264059277,
+            -0.002563606092068712,
+            0.04770277039047993,
+            -0.0032010143255574255,
+            -0.03425328601475849,
+            -0.02105909767072799,
+            -0.00840881022563103,
+            -0.028699447677549002,
+            -0.015925160861589006,
+            -0.0055750742281166,
+            0.017513987082479483,
+            0.010399718005952185,
+            0.02267201252273823,
+            0.01258511878601025,
+            0.0015081978484714903,
+            0.01874981572091931,
+            0.011158889421057756,
+            -0.002607810686437211,
+            -0.003732023411827874,
+            -0.007787189972165309,
+            -0.05524012485122728,
+            -0.025110689689279583,
+            -0.03946383236703947,
+            0.004443572529325475,
+            -0.018745077598477586,
+            0.005926160888274727,
+            0.039430331582190524,
+            -0.0049500324516682,
+            0.004352795787899792,
+            0.015566139991780173,
+            0.013148156356408315,
+            -0.015279560337042242,
+            -0.03316846614871165,
+            0.0001592209771257332,
+            0.018362411161743347,
+            -0.012706582418607377,
+            -0.03424477956681259,
+            -0.001991022841226589,
+            0.0033467286047492175,
+            0.03357019284647071,
+            -0.024359110796680374,
+            0.00359117029951049,
+            0.005061818038334802,
+            0.011359582931760315,
+            -0.026793992746244076,
+            -0.018170136079817746,
+            -0.017421603453826397,
+            0.03996608926471779,
+            0.02587013058343799,
+            0.015763410145648683,
+            -0.04512536449983358,
+            -0.02293473890636517,
+            0.048575083783480195,
+            -0.009589860756342604,
+            -0.06486262416944208,
+            0.03447666009581266,
+            0.016105654870253654,
+            0.003526582643722421,
+            -0.01354754623151814,
+            0.05401575248587799,
+            -0.03563190295559527,
+            -0.017365166993914202,
+            -0.0046288709698227735,
+            -0.033657212157212645,
+            -0.007686252059880836,
+            -0.001385273642139782,
+            0.048229062789530236,
+            -0.029083855744728694,
+            -0.030465559177859865,
+            0.0024368661249980416,
+            -0.024144031154493057,
+            0.021693278579520123,
+            -0.016405088055634124,
+            -0.018439935250344958,
+            0.006948603359995389,
+            -0.033943037395199695,
+            -0.003007796423804453,
+            -0.016783195426536633,
+            -0.02270589491206155,
+            0.020874438753828814,
+            0.016724666903417947,
+            0.027033184180344852,
+            0.0019847647852695785,
+            0.017749524526371133,
+            0.0204661893602285,
+            -0.005604929968008821,
+            0.028462234633072944,
+            -0.020790459496727066,
+            -0.04183816355941877,
+            0.03946487892663695,
+            0.01192861758684541,
+            -0.0018024841275380242,
+            0.01077082591611702,
+            -0.01475239020295694,
+            -0.03952356245715099,
+            0.03683950018557526,
+            -0.056767672022551764,
+            -0.007507600090017693,
+            0.009530544352246083,
+            -0.00788856051420074,
+            -0.011204694027562521,
+            -0.010587701466318598,
+            -0.028604940377315504,
+            0.023749537635431462,
+            -0.013672419168235691,
+            0.01953449229707597,
+            -0.011449939491554354,
+            -0.012238180529977605,
+            0.05795536247491174,
+            -0.021518581774137322,
+            -0.040672955795925836,
+            0.026066659238628543,
+            -0.014061106927896723,
+            0.03328357230532424,
+            0.03957107406164427,
+            0.030670947051333917,
+            -0.004278577327978564,
+            0.025885946540364826,
+            0.0319106101668716,
+            -0.003462160036198642,
+            -0.04686868330851868,
+            0.00555303933159587,
+            0.02156143971867743,
+            -0.03739529489075782,
+            -0.01025626126223168,
+            -0.02141970191890338,
+            0.022134910980548407,
+            0.017807484142010496,
+            -0.007232643256393455,
+            0.011318041192005274,
+            -0.010489012862455913,
+            -0.013003217163997554,
+            -0.00847279880392288,
+            0.02357274819910594,
+            -0.0016485066139744577,
+            -0.005873113634992993,
+            0.006424249429163587,
+            0.004008703814301658,
+            0.0054639858216457724,
+            0.010198079500408441,
+            0.017694720520798957,
+            -0.017087021465208255,
+            -0.0025558869389290554,
+            -0.016503579212015177,
+            -0.0420082404922411,
+            -0.003563173424359332,
+            0.022924161700566468,
+            0.012345933179990547,
+            -0.00729908668365632,
+            -0.004234683369642856,
+            -0.0022962545569187934,
+            -0.020666205489143392,
+            -0.014796149043465308,
+            0.024562653223873056,
+            0.030217494971086438,
+            0.007810961433007407,
+            0.007781263640449456,
+            -0.0030348559388780655,
+            0.007310876971216526,
+            0.016099591640649783,
+            0.02311929597780174,
+            -0.04160162038353416,
+            0.027152100845897007,
+            -0.00025396315502635146,
+            0.023717123370023598,
+            -0.04186918759097667,
+            -0.017988531857836703,
+            -0.03106663560912813,
+            -0.023595079699181436,
+            0.02437134274097885,
+            -0.02317559303912261,
+            -0.012327263775257306,
+            0.000989566600374836,
+            0.00954921216572729,
+            -0.005155079395685017,
+            -0.017508404061379164,
+            -0.04582321461886958,
+            -0.0014367264231457263,
+            -0.030417229142190933,
+            -0.017829710557332023,
+            -0.001056472801554354,
+            0.005016912084754242,
+            -0.06276644565618253,
+            0.0109965681725148,
+            -0.07270669271977365,
+            -0.004542796125637244,
+            0.040139998232966566,
+            -0.017890441783630643,
+            0.055909102081208445,
+            -0.05029104308417623,
+            0.01673962668706085,
+            -0.04507293803055318,
+            0.015385506084493484,
+            0.025677372277539627,
+            0.030235171586316813,
+            -0.022189978375852563,
+            0.008461188938299745,
+            0.008033707483259569,
+            -0.045089505755429415,
+            -0.04144698743458133,
+            -0.013640256640978024,
+            0.01036782197518379,
+            0.0012696958402094232,
+            -0.016090728932978384,
+            0.023838437877040163,
+            0.011946482002767135,
+            0.03342604921859589,
+            0.04211801335157251,
+            0.002708015395637621,
+            -0.047156646610976634,
+            -0.013104464812562618,
+            -0.003933116891120681,
+            0.03155321708116329,
+            -0.018866109579446184,
+            0.015376812010630058,
+            -0.01955557376085726,
+            0.016255217383093355,
+            0.017868332102274922,
+            -0.020325179506828335,
+            0.013434364256590097,
+            0.004176904701252156,
+            -0.03877060944639854,
+            0.0006475542749369515,
+            0.00042468300294954125,
+            -0.02116992978038693,
+            0.0032654387271108293,
+            -0.00575003786159731,
+            -0.05149954071159492,
+            -0.031382926943128234,
+            -0.015730563640253065,
+            0.0382312502769352,
+            0.004297939848910064,
+            0.019724983545557365,
+            -0.017907869879607064,
+            -0.005165628493894463,
+            0.03698976707579373,
+            0.008501001373266799,
+            -0.01725527821298066,
+            0.006879620077191753,
+            0.02455792200929138,
+            -0.03224193972301503,
+            -0.039181328884413266,
+            -0.0008206986106407542,
+            -0.023306758893533952,
+            0.01672946960876128,
+            -0.02658587698869307,
+            0.005873620623736886,
+            0.0025331170979801232,
+            -0.03490809296591325,
+            0.02456764314036355,
+            0.003291113462353163,
+            -0.004303423672882112,
+            -0.020359544091059775,
+            -0.016892415165120203,
+            0.017174797487220006,
+            0.04391489511478065,
+            0.007327321684948843,
+            -0.003943713832544148,
+            0.00862684231313155,
+            -0.006361172785753631,
+            0.05826688726534012,
+            -0.02284956969636637,
+            -0.026680023999604643,
+            0.027527436347931256,
+            -0.022597669279069723,
+            -0.006277863878949772,
+            -0.017343446085471857,
+            -0.012065192152203348,
+            -0.0004183712490442143,
+            0.019339311388446943,
+            -0.019423562920897294,
+            -0.010212527447243049,
+            0.004783006556748757,
+            0.012824133870627983,
+            -0.004855945108275706,
+            0.006239949458562236,
+            0.006847581690471813,
+            0.03094005096378827,
+            0.026993131409761086,
+            -0.011331881651539968,
+            0.03297014291876675,
+            0.010265600643742144,
+            4.666113318251278e-05,
+            0.001970879281172325,
+            0.002523664335535017,
+            0.0034107811520368574,
+            -0.03143890864030425,
+            -0.01624584278940617,
+            0.013468729048885906,
+            0.005233309338223403,
+            -0.004550544967856448,
+            -0.023783653541634683,
+            -0.009402494239255264,
+            0.03334923712760375,
+            -0.003260424721959948,
+            0.02748395114281811,
+            -0.0058374897745283065,
+            -0.01985840930273332,
+            0.017595418271610163,
+            0.02881743246631328,
+            -0.05928552440605384,
+            0.017062540894682354,
+            -0.0011508547522925793,
+            0.008953761538519997,
+            0.01610561914042553,
+            -0.008894160513648953,
+            0.013301991014533684,
+            0.04080863883897929,
+            0.010738051773755976,
+            -0.023345002072886145,
+            0.004215726310838674,
+            0.01541749448570608,
+            0.01291813299324752,
+            -0.022747942296528025,
+            0.0008640791861357522,
+            0.011008522066277018,
+            0.005893793007012561,
+            0.0026106450193140156,
+            -0.006278394080583205,
+            -0.025659188287146268,
+            0.024180550333596625,
+            -0.0044491292740074925,
+            0.026967973346698617,
+            0.04390909273429254,
+            0.009596855239692042,
+            -0.03686020608700941,
+            -0.013109951664921729,
+            0.0002799680359670127,
+            -0.0038537577602593366,
+            -0.0037552094562213916,
+            0.011105295862811114,
+            -0.04185615640149602,
+            0.019950890919369135,
+            0.006453241355231516,
+            0.015314645427201533,
+            -0.020031193483734364,
+            -0.024530173488182337,
+            0.0031949828520724337,
+            -0.01492645564652531,
+            0.012875827757040941,
+            0.005057170413681063,
+            -0.004319834710426384,
+            0.005386862713494232,
+            0.00030848054013413157,
+            0.008384941028326983,
+            -0.010811526993301901,
+            0.03175068297908541,
+            0.004680987487229685,
+            0.009264547472281849,
+            -0.02856314241700808,
+            -0.021083721348429348,
+            0.020113396771629096,
+            0.03572366771051171,
+            -0.01708529918264109,
+            0.03346921724975398,
+            -0.009042046660366718,
+            0.0033219155387035227,
+            0.0151720168318272,
+            -0.015805318210244414,
+            -0.008057236707694303,
+            -0.0005483009439015931,
+            -0.002917527004423299,
+            -0.014538511124345527,
+            0.007302979537905532,
+            -0.0208529958115422,
+            -0.02965227242147972,
+            0.020064178921373875,
+            -0.00011892622932349096,
+            0.016455188521316146,
+            0.010335044657747514,
+            -0.007816155992852236,
+            -0.0052262330850905165,
+            0.006959344748883874,
+            0.04432147882994085,
+            0.026734484553994495,
+            0.05547661378166294,
+            -0.005787997952888261,
+            -0.004559444854138438,
+            -0.0082295581123363,
+            -0.006588172755707144,
+            0.04524009609473585,
+            0.015915003261638767,
+            -0.004910285095602864,
+            0.017495607570191165,
+            0.01557886571266685,
+            0.03858259260136037,
+            0.0027648525902558316,
+            -0.016912796336835408,
+            0.04014755669075221,
+            -0.004077405782937414,
+            0.008616827127381418,
+            0.011102692199778469,
+            0.0024185742218798562,
+            0.00807141794538637,
+            0.02996274720815659,
+            -0.012429186019907761,
+            0.01680942269506771,
+            -0.04571916592313758,
+            -0.02775530246832996,
+            0.014370280587419414,
+            -0.004931905540824582,
+            0.03419959665162407,
+            0.01668521021731861,
+            -0.023105305446763272,
+            -0.024761385385514877,
+            0.021529428247380984,
+            -0.002603042335561961,
+            -0.024135326707477325,
+            -0.015906833215844473,
+            0.003286605488908195,
+            -0.0023437997044735617,
+            0.0060251848371355166,
+            -0.013481652076194643,
+            0.0002908426939460218,
+            0.01953362396253743,
+            0.0038544889405965057,
+            0.020760460182873287,
+            -0.016224396121239458,
+            -0.00259233446682179,
+            -0.009282861237653619,
+            -0.02952130765234087,
+            0.017988370145301916,
+            0.006138236653154911,
+            -0.007607182955958238,
+            -0.009516767050028553,
+            0.0048347778733230365,
+            -0.005819209642901871,
+            -0.0025851701689263706,
+            -0.01073014664868326,
+            0.012433465684360061,
+            0.025784014942833097,
+            -0.016875624785552303,
+            0.007944349714123104,
+            8.651589872318479e-05,
+            -0.022306044741692753,
+            -0.022127016745066307,
+            0.020143472307319017,
+            0.002792646249762062,
+            0.008768813346304779,
+            -0.004074214276300631,
+            -0.00015024341785520542,
+            0.002990550758627699,
+            0.01951746553838439,
+            0.03896451381932467,
+            0.004232673242619471,
+            0.027876586543280867,
+            0.0145860303663199,
+            0.03994234200854897,
+            0.009657017763624678,
+            0.00696607158576712,
+            -0.026055387694656488,
+            0.02819890357011778,
+            0.00018297502510753014,
+            0.043190622821775236,
+            0.022488402221902314,
+            -0.0407730101161195,
+            -0.0021946518523759066,
+            0.01669520746276111,
+            0.013212623257794862,
+            0.017009844136059122,
+            -0.004537512994041156,
+            -0.014041213424740049,
+            0.028135404528975894,
+            -0.022859586430516003,
+            0.0008637006862481223,
+            -0.004486332934188017,
+            -0.025031795510001887,
+            0.007351370254175368,
+            -0.004567816916440959,
+            0.030025413528233364,
+            0.0014976740344892735,
+            -0.022285096307825804,
+            0.022837170656498543,
+            -0.009551269792616045,
+            0.011045530488684207,
+            0.008809047185288755,
+            0.020065001354321994,
+            0.031993362733068575,
+            0.009143210434415064,
+            0.02133640814803879,
+            0.012241777474889248,
+            0.014771538127546474,
+            -0.008804192497858033,
+            -0.02341427260922065,
+            0.023167424359504862,
+            -0.0011598649565273595,
+            -0.029627604902846688,
+            0.0021991474725787303,
+            0.003261878079344105,
+            -0.0005935713331682358,
+            -0.009144689470851402,
+            -0.005580557862988033,
+            -0.007240836451875213,
+            0.019631377011298905,
+            -0.007085213886352692,
+            0.007017573363709754,
+            -0.004074171791476129,
+            -0.00822871835665378,
+            -0.0421101573155962,
+            0.004210914075286222,
+            0.003408427700622687,
+            -0.01752179330665445,
+            0.006384440404174512,
+            0.011121481025573962,
+            0.043020575455262985,
+            -0.022809979913698254,
+            -0.0005737138409049003,
+            0.019206623212867476,
+            0.0009562551889820241,
+            -0.023427467810802533,
+            -0.038931688376706186,
+            0.007172345672967499,
+            -0.03467050750542982,
+            0.023185301620612612,
+            -0.0007075596339842658,
+            0.02387977204565927,
+            -0.03043468414449851,
+            0.013909304572370437,
+            0.004784668158421625,
+            0.012524879174804915,
+            0.021654512058639427,
+            0.03142146922870715,
+            -0.02053473475540882,
+            -0.050409862638621725,
+            -0.005406667316025549,
+            0.00035138762584085976,
+            0.022018487016014856,
+            -0.026014580970974616,
+            0.00025633004699302783,
+            -0.003349094027291921,
+            -0.02401366769212778,
+            -0.01735996745614562,
+            0.01962340791446025,
+            -0.050011480783403936,
+            0.015042913576492952,
+            0.0262737424555697,
+            -0.02507473901367252,
+            -0.05408889321034101,
+            -0.018988587082325706,
+            -0.028217893770659686,
+            -0.009285770094319318,
+            -0.00661159230987788,
+            0.028597405154740078,
+            0.003525576256054959,
+            -0.011462219358952466,
+            0.025401668390029348,
+            0.006155837251830549,
+            0.0026050693705924593,
+            0.015985722392086222,
+            0.007289910674212967,
+            -0.011452511453623235,
+            -0.014599577191293152,
+            -0.04558584556100621,
+            0.010197536827229795,
+            -0.0045431962637690155,
+            -0.019372598050508005,
+            -0.030147301333718016,
+            -0.0137828704002156,
+            -0.027060515748528666,
+            -0.0017291966950107505,
+            0.053783684195381654,
+            0.03292494290301388,
+            0.005560172296221419,
+            -0.0026085670565688026,
+            -0.010466890489356092,
+            0.036247483457995895,
+            -0.006789746401727701,
+            0.03662690472838588,
+            0.001784522073210422,
+            -0.0242673253380092,
+            0.01169382884548301,
+            0.02892627181542449,
+            0.009858451662166387,
+            -0.0005592283645355213,
+            -0.006160132931009508,
+            -0.008782830470079547,
+            0.0031207630972208394,
+            0.025613233596604418,
+            0.003268170392795548,
+            -0.010062991425127038,
+            0.018106967407336815,
+            0.004480291699976396,
+            0.02010374270557836,
+            0.03555357587371324,
+            0.023489837757983054,
+            -0.021782173652026497,
+            -0.026581762796336082,
+            -0.0013561486781077014,
+            0.0072266106510934505,
+            0.03551967295942857,
+            -0.01651315514034218,
+            0.0001274396319698452,
+            -0.008272735507120127,
+            -0.00894931703949637,
+            0.011149952898504355,
+            -0.0029671593121986116,
+            -0.0030752010285951576,
+            0.01662711184497578,
+            -0.0011672263615879709,
+            -0.033448222499147764,
+            0.0078288560210653,
+            -0.023496331346917084,
+            0.0051040606404415545,
+            -0.004113104164416575,
+            -0.007076525381595441,
+            -0.002157548492138289,
+            0.006302962169276006,
+            -0.008065841090139872,
+            -0.00035242603988955366,
+            -0.01586406637763306,
+            0.008271894439780969,
+            0.007438312999409211,
+            -0.0069617481460406095,
+            0.011402932583388205,
+            0.002666895909532698,
+            0.0005219881149353855,
+            -0.012159425670629751,
+            -0.012042145306711322,
+            -0.00957276588761916,
+            -0.03401863879157551,
+            -0.02604353712472079,
+            -0.024949533021906844,
+            -0.0014209149651479556,
+            -0.006566685925666972,
+            -0.03187796739029736,
+            -0.01209152137926458,
+            -0.011110763124407752,
+            0.007756961097537267,
+            -0.013260548545917543,
+            0.005433285079114047,
+            -6.399098520041351e-05,
+            -0.0044960488826674555,
+            0.016615636438440516,
+            -0.003643291599919672,
+            -0.019488010211707343,
+            -0.03927818878149417,
+            0.008209336177358258,
+            -0.02805593125673831,
+            -0.0038816771550190788,
+            0.007607119219721711,
+            -0.00528681677614909,
+            -0.029502897838743516,
+            -0.009149794737430706,
+            0.008767001199311985,
+            0.003507217008280066,
+            -0.01773349756494436,
+            0.03579670773605949,
+            0.026971617790626075,
+            0.011875179879209696,
+            -0.019891342676443626,
+            0.0508929813275565,
+            -0.0047010735337056275,
+            -0.012544935684845649,
+            0.018297503869683027,
+            0.02355022268131519,
+            0.03273167043015095,
+            -0.024830243320788693,
+            0.0009370814985214882,
+            -0.021937793653233454,
+            0.00535662615596573,
+            -0.035009915581788176,
+            0.0028595659314725012,
+            -0.014395027018699446,
+            0.015596459769560532,
+            0.03397372881171701,
+            -0.013254226072981457,
+            0.011195747181764677,
+            -0.016299384375589934,
+            0.00730244731485402,
+            0.00569944090461748,
+            -0.012916284280345044,
+            -0.005812311882399539,
+            -0.022902815152116493,
+            0.020140276669979015,
+            0.005809799950176385,
+            -0.0008130655714656685,
+            0.0104085826652653,
+            -0.044723432238900435,
+            0.03286429597337671,
+            0.008730225944501239,
+            0.017599189944921396,
+            0.0029762446259779998,
+            -0.0003068671951232415,
+            0.011611551606224214,
+            0.011228401175698422,
+            -0.016029727902740706,
+            0.004345913009102136,
+            -0.004613061137378433,
+            -0.03850533909474302,
+            0.009363351681220008,
+            -0.03631184057978666,
+            0.014427784677340159,
+            0.016026872002233575,
+            -0.004170582842643461,
+            -0.025166189571409366,
+            -0.005428471012333673,
+            0.010827910811085384,
+            -0.009783069664327864,
+            -0.009987805234276443,
+            -0.04163358183060102,
+            -0.025236440364963034,
+            0.034097508113604576,
+            -0.002119474613596975,
+            -0.025826403511707147,
+            -0.010830738112107807,
+            -0.011351132636491575,
+            0.020207687645133,
+            -0.006542026283946858,
+            -0.0006263975804936843,
+            -0.04150011545311013,
+            -0.0073396962917864515,
+            0.011192225907726928,
+            0.027746213794734322,
+            -0.035970017302005566,
+            0.02708126943501373,
+            0.002105633542220187,
+            0.014262674845755191,
+            -0.01626664983003942,
+            -0.005631805361903626,
+            -0.013990539604372344,
+            -0.035877093361375685,
+            0.016487723013906077,
+            -0.008607758248055054,
+            0.004714436224135091,
+            0.04458009880881818,
+            -0.0036872068506712727,
+            -0.009466125348499464,
+            0.007600705817195143,
+            -0.0034590883553048063,
+            -0.010351576085287273,
+            -0.0028564505793999295,
+            0.001959661298507,
+            -0.007080697863126256,
+            -0.04183476693971616,
+            -0.00499888976530217,
+            0.04684391782633148,
+            0.006361744381879857,
+            0.013276793765092465,
+            0.013516220521120684,
+            -0.018335746557932003,
+            -0.045577810515431215,
+            -0.0071616111758423025,
+            -0.0032763880176386808,
+            -0.022708028392758567,
+            -0.0240284945459861,
+            -0.008735040856657758,
+            0.005844145402902923,
+            0.006359368165686724,
+            -0.01120763599008222,
+            -0.022676131361171057,
+            -0.027085968700452136,
+            -0.010526103176253857,
+            -0.02123944716136792,
+            -0.010792145796733375,
+            -0.027142270190771887,
+            -0.028742234538796804,
+            0.02880426953386151,
+            0.007575182232859564,
+            -0.004084088301176117,
+            -0.010005765476121966,
+            -0.01255640546813973,
+            -0.0004774761617900028,
+            -0.0025307994917461232,
+            -0.005822065709999144,
+            0.0038614286313282164,
+            -0.011390482620811868,
+            0.023152086963976554,
+            0.0021220701260546895,
+            0.018329162172128138,
+            -0.05772324571231113,
+            -0.002709083851090071,
+            0.021497896395795905,
+            0.0033025410488410913,
+            -0.005412445141910314,
+            0.029352650459792132,
+            0.011223947499453582,
+            -0.006539460282439711,
+            -0.031355428613718234,
+            0.020942976176820262,
+            0.024122993515785947,
+            0.00817615634503138,
+            -0.012994160510030828,
+            0.01221095914384204,
+            0.01629778384790159,
+            -0.004494985018246821,
+            -0.011161098655072933,
+            0.005586857967039181,
+            0.020941039642372112,
+            0.013580622629911925,
+            0.0013398071867077221,
+            0.030905022925366163,
+            0.0006862084391508121,
+            -0.03521231612862609,
+            -0.01908191995763771,
+            -0.01275995164030872,
+            -0.003584331305986364,
+            -0.015264425022763126,
+            0.005294329654809353,
+            -0.002355465010876059,
+            0.006335052149475905,
+            0.022655379849785256,
+            0.03593231939948758,
+            0.0146937743203089,
+            0.00937330328448886,
+            0.003167462724350819,
+            -0.0013235176101076284,
+            0.012102872918129403,
+            0.008638011773075119,
+            0.0350701240268404,
+            -0.014653221697693864,
+            -0.011003994230902835,
+            -0.0037767288519369537,
+            0.010100131061186566,
+            -0.004113527518662366,
+            0.020414393256136372,
+            0.03274709615649469,
+            -0.0037222949335916876,
+            -0.005604552180246868,
+            -0.006160118231975786,
+            0.0041467454414452565,
+            0.01807879127455274,
+            -0.01252134965321004,
+            0.004364858987501461,
+            0.005549300665159658,
+            -0.0071897162833861,
+            0.018567059255813558,
+            0.0002622074720860951,
+            -0.012983550838417323,
+            0.03266376564402191,
+            0.11449509387240629,
+            0.0017251916835047628,
+            0.007125215166916139,
+            -0.040229997815548584,
+            -0.020537987527148888,
+            -0.00245728735683473,
+            0.005779972828836593,
+            0.007004732428236457,
+            0.02225914982152608,
+            0.006505827715591914,
+            0.0006708045351142873,
+            0.031708522931495536,
+            -0.017334083511709092,
+            -0.002522861743921599,
+            -0.01745656559688129,
+            0.013890451996979048,
+            -0.014312605700272691,
+            -0.013784963572164409,
+            -0.01572417853955075,
+            0.004507425871748977,
+            -0.000601158156253668,
+            0.0016158265715634729,
+            0.01791303234734911,
+            0.0011926917387973374,
+            -0.026393351252156246,
+            0.0006694008039871878,
+            0.001026031515214626,
+            -0.004514419527575117,
+            -0.011320929719992686,
+            -0.02223001723942815,
+            0.016235841442653918,
+            0.014188311619681008,
+            -0.004895218832118094,
+            0.009244697376301004,
+            0.010558796153959677,
+            0.02707485418319703,
+            -0.028380342401606758,
+            0.0040204430418629405,
+            0.007023041225985294,
+            0.01606556581447933,
+            0.017617862031190843,
+            0.0034870847111552862,
+            -0.011253315705053224,
+            0.03238289399766116,
+            -0.010563282687352038,
+            0.0065093536270233385,
+            0.014966802208644462,
+            -0.03541611540945151,
+            -5.821682910348165e-05,
+            0.0061985444938402745,
+            -0.011716201661363997,
+            0.009580035961315467,
+            -0.03120692534097433,
+            -0.0039291243987978845,
+            0.009317468331974374,
+            0.017611312558635673,
+            0.008804996982574765,
+            -0.0034129169537151925,
+            0.010822226698822738,
+            0.012510661458054741,
+            -0.0027147975086409966,
+            -0.008855213644027594,
+            0.01238502686635956,
+            -0.0007687053446954701,
+            0.00450108863711973,
+            -0.00799170686059026,
+            0.0029755595812552293,
+            0.002290650917244691,
+            -0.01554856300008412,
+            -0.011779430686394945,
+            0.00438393126679289,
+            0.0036931891075561957,
+            0.001535662168147833,
+            0.012014408636737167,
+            -0.006087255045485002,
+            -0.0077816220325470226,
+            0.0040573724156794284,
+            0.003136943055301706,
+            -5.505014316363865e-05,
+            -0.030625245024561508,
+            0.01585923084309149,
+            0.0067595435845396405,
+            0.010013308598626031,
+            -0.005315626260228517,
+            -0.00903142414091241,
+            -0.007240920015018893,
+            -0.01228521172101564,
+            -0.010657894223173453,
+            0.007658605592936815,
+            -0.00724924993432376,
+            0.011124174211132038,
+            -0.03128027787357322,
+            -0.013988715435985316,
+            0.01231792993245938,
+            -0.01528161242159377,
+            0.0007675821323979293,
+            -0.008389914312201402,
+            0.0009417242487395995,
+            -0.005119304379761705,
+            0.007887436211143328,
+            -0.03731503169547854,
+            0.007878001990271541,
+            0.02456170736765724,
+            -0.00330924625934981,
+            -0.020259771366167557,
+            -0.009613930457931786,
+            0.0241314385136084,
+            -0.008267831641886344,
+            -0.01645888287387047,
+            0.008421304862524506,
+            0.010140813149093993,
+            -0.0006309118552875393,
+            0.0008237784055549769,
+            -0.003726984191590917,
+            0.01669125388798323,
+            0.02054777616420997,
+            0.041930770964107515,
+            -0.014789610432004559,
+            0.01092349703703152,
+            0.02866938974792608,
+            -0.03378576980722847,
+            0.014728113846952417,
+            0.02518409645155824,
+            0.014590531327650454,
+            -0.0004446956315190408,
+            0.001609777186372437,
+            0.0068939321280085475,
+            -0.014273492115311677,
+            0.009247676157551494,
+            0.013380104440744367,
+            -0.018407746260686045,
+            0.020444430509441124,
+            0.03140071554008003,
+            0.0025839998101985736,
+            0.01355205268903058,
+            -0.006268939830271642,
+            -0.013860692186849113,
+            0.014717364068116991,
+            0.031429975488060896,
+            0.0006919565473572909,
+            -0.018561500446336806,
+            0.019128244642842268,
+            0.006988095806264863,
+            -0.003678529269242034,
+            -0.0021993078588142664,
+            -0.021334919445200388,
+            -0.021003586791369228,
+            0.017304350704792804,
+            0.013508779408333964,
+            0.027794485858413365,
+            -0.010594702782952029,
+            -0.007818089330662161,
+            0.014293688127901867,
+            -0.022368568887004094,
+            0.015426482139881204,
+            -0.02160820059598751,
+            -0.00571499813703566,
+            -0.0077402210272634235,
+            0.03274360648629609,
+            0.001441304806200403,
+            0.02734331677589297,
+            0.016457027764064002,
+            -0.033609420134045206,
+            -0.029790535553657092,
+            -0.00712106584238484,
+            -0.02307933297713104,
+            0.010356561171963577,
+            -0.03901183838032092,
+            -0.00071735150189382,
+            0.008251646656432826,
+            -0.008079917288471352,
+            -0.0007087526632966007,
+            -0.006419191399519291,
+            0.013989344185652506,
+            0.013661633160336152,
+            -0.0228103143826663,
+            0.023644639659577426,
+            0.008343481940133684,
+            0.025998483906975782,
+            -0.02076055376006461,
+            0.019148568238437927,
+            -0.017705363646416224,
+            -0.02777313933338929,
+            0.014358130220942483,
+            0.023585731219788467,
+            0.01635364038245971,
+            -0.007122853965024528,
+            0.007110003583832299,
+            -0.03556761320536796,
+            0.027585457849964405,
+            -0.013064062623866305,
+            -0.012504161289952458,
+            -0.0032379276562094555,
+            -0.011931827532360686,
+            -0.002574161005800279,
+            0.0008533907465694845,
+            0.0034345232246102806,
+            -0.03131462673429805,
+            -0.0002450959360520896,
+            -0.03727416416766574,
+            -0.028944814604815072,
+            0.01360126708805471,
+            -0.00036361284498363273,
+            -0.0049097813612814616,
+            0.02933242705521636,
+            -0.026339136251046315,
+            -0.02693295650849956,
+            -0.009590247544093352,
+            0.02853096456887074,
+            -0.0220673798697511,
+            -0.004519050393279848,
+            0.0015034014871316069,
+            -0.01828226786700525,
+            0.004436383319946364,
+            -0.0012748481042235446,
+            -0.010416065158876917,
+            -0.01105146139270597,
+            -0.03229479099927737,
+            -0.005822133182096844,
+            0.02106215163620738,
+            -0.006957125617688484,
+            0.002746241228040847,
+            0.013312248331456934,
+            -0.010968629118224582,
+            -0.0029946900704643812,
+            0.0017875284441794235,
+            0.023304175718480834,
+            -0.021033941064173063,
+            0.007714334541806199,
+            0.005955442031111047,
+            0.0016405586860131207,
+            0.018406562522853148,
+            -0.0054922197063461825,
+            0.0037690571959124617,
+            0.0162126534419016,
+            -0.009969304596173302,
+            0.017959307638726878,
+            0.007691549802475525,
+            0.006681837561745192,
+            0.002775700618222622,
+            0.007712770283579111,
+            0.009752652973700739,
+            -0.013506020126078112,
+            0.016911141425375933,
+            0.022468715726349706,
+            0.00023233555209372766,
+            0.021066580799163234,
+            -0.02302516292580905,
+            -0.028618544628453015,
+            0.018297878810036087,
+            -0.0225078941299981,
+            -0.012735208680311652,
+            0.016459925496937065,
+            -0.0043032724578332855,
+            -0.00797609922711709,
+            -0.014986057073351055,
+            0.02260618561608301,
+            0.02132930921464508,
+            -0.020249788476562527,
+            -0.012385936327068884,
+            0.0072304431443959505,
+            -0.004373475447051833,
+            0.007086656081865733,
+            0.03195810137269375,
+            -0.015982210897392764,
+            0.006818890093901413,
+            -0.0038306138017521626,
+            0.0030753642449792215,
+            0.017830546304667783,
+            0.009470018737705222,
+            -0.0019815241970236146,
+            0.030100238416495066,
+            -0.001330614013444443,
+            0.016392226282459177,
+            0.0004888028477871524,
+            -0.010948833692356176,
+            0.006343890495626752,
+            -0.026652806306173643,
+            -0.008627570055007363,
+            -0.001203530223345183,
+            -0.009581494451831351,
+            0.07598616930784373,
+            0.003930153552069478,
+            -0.01295957554290434,
+            -0.001673007115291578,
+            -0.001368287102181597,
+            -0.006897179139643672,
+            -0.00960679451603541,
+            0.0287166325268306,
+            -0.016613613331623346,
+            -0.01894527592131826,
+            0.0008370014132002703,
+            -0.006785618888421608,
+            0.00919924251249592,
+            0.01816363936999796,
+            -0.010225961928154075,
+            -0.01604759761423727,
+            0.001577138242530474,
+            -0.003520564936698765,
+            -0.004395434431595494,
+            0.0052399043934902014,
+            -0.010127122101423486,
+            0.006456662777583096,
+            0.026560491731327886,
+            -0.013684536319659296,
+            -0.01425196172663986,
+            0.004084982328707292,
+            -0.0010682084059972982,
+            0.009758641913073978,
+            -0.00864879062601467,
+            -0.00481222050200678,
+            -0.014375330954464725,
+            -0.011378890549403524,
+            0.02505641780042516,
+            -0.02861164504722546,
+            0.0034060847953066354,
+            0.014937227531980433,
+            -0.00016904224334342915,
+            0.01541381851648916,
+            0.0033028855218525644,
+            -0.03615935765718852,
+            -0.013677774380513626,
+            -0.02073839691250734,
+            0.019821074228011413,
+            0.01688422012576042,
+            -0.0033731840788147442,
+            -0.013840993586441808,
+            -0.026149880178874887,
+            -0.007430563876740068,
+            -0.02466234818567362,
+            -0.020708282040996014,
+            -0.019276991448222817,
+            0.006137002799644838,
+            0.005162892426268572,
+            0.022881759407670583,
+            0.0025854880108157857,
+            -0.015058889448404231,
+            0.0029948608544278386,
+            -0.006722836436935071,
+            0.011845298212977357,
+            -0.02747944486818378,
+            0.015311272959213262,
+            -0.01610493464983465,
+            0.025239768880017402,
+            0.007425208902253236,
+            0.0010569008485112994,
+            -0.008822681490988575,
+            0.00010704167529046774,
+            0.014883787023445807,
+            -0.004186968199973372,
+            -0.0007781259002896265,
+            0.02787555889813697,
+            0.02087373813833492,
+            -0.0017783425135865238,
+            0.008743718448136432,
+            0.028026471206552342,
+            0.019081083283793984,
+            0.0325261942213039,
+            -0.013815526479760766,
+            0.005113676604826083,
+            -0.028873600114081546,
+            0.010510475843707614,
+            -0.010871039729515013,
+            -0.010201824683340651,
+            -0.0067946599549165735,
+            0.027756887504389347,
+            -0.007962601205446472,
+            -0.0006687976124826922,
+            0.03469894503563926,
+            -0.014185294847328063,
+            -0.00010543064621374546,
+            0.0007386418834840544,
+            -0.009759481309152999,
+            0.02742722730386853,
+            -0.0073590367291510046,
+            0.018456032721630843,
+            -0.019560882612141943,
+            -0.013705917411254969,
+            -0.011894560208199602,
+            -0.001142768182039817,
+            -0.02576754877141608,
+            -0.008755240066826174,
+            -0.02249287216087102,
+            0.00859172262856532,
+            -0.0046012388553094375,
+            0.018845920253446127,
+            0.0011617844393195566,
+            -0.005275799428018491,
+            0.029069367036565145,
+            0.0088080893110627,
+            0.018100411389202663,
+            -0.0004971948011851863,
+            0.01216566778850985,
+            0.02369448391321832,
+            0.002637001729688794,
+            0.005012948614448268,
+            -0.024213704079796478,
+            0.003420184574622472,
+            -0.0004359900462023086,
+            8.265488264269535e-05,
+            0.016267948861008616,
+            -0.019572032453100303,
+            -0.02216264534756073,
+            0.021871880626992544,
+            -0.012742926070754964,
+            0.011676769226089007,
+            0.0035362870996336194,
+            -0.010732229460038356,
+            0.0006838777488770268,
+            -0.03083438636386314,
+            0.008820746379709355,
+            -0.03216685858285627,
+            -0.032230808828686675,
+            -0.013547681710236494,
+            -0.009994106571421729,
+            0.0008758088338659712,
+            0.028447187234357855,
+            -0.00795317029726568,
+            -0.015797892954136448,
+            0.005363565557248085,
+            -0.011199252697624111,
+            -0.00817162221789202,
+            -0.0007867892785855576,
+            0.005856939999700809,
+            0.009461959355310273,
+            0.005997804620690314,
+            0.011648335019189696,
+            0.007491946658046298,
+            -0.008028800679169293,
+            -0.008837725220736191,
+            0.015889923814605387,
+            -0.0014409716087425222,
+            -0.01176424391171685,
+            0.030430130082762363,
+            -0.013300620185884817,
+            -0.0005592926632214984,
+            -0.0021975669766032472,
+            -0.0015687208316663597,
+            -0.0021652076181411244,
+            -0.014225289143358926,
+            0.01870568404189425,
+            8.334501002782455e-05,
+            -0.014288806218067678,
+            0.0032841743905975634,
+            0.003841780428153241,
+            -0.0036498339483411074,
+            0.008388922960269383,
+            -0.008147275785908849,
+            0.017348147498892,
+            -0.015113395971380741,
+            0.008740452410210708,
+            0.0038335161236289234,
+            0.0043482078469644385,
+            -0.0065755348640655865,
+            -0.01592809184384243,
+            0.009642593996701513,
+            0.012051872087663658,
+            0.02173264012652684,
+            -0.02525660485859878,
+            0.00876161653901566,
+            -0.005374533920781607,
+            -0.014534179652013885,
+            -0.016127866637369738,
+            0.00982303248417706,
+            0.022322910435388785,
+            0.0063086346898091296,
+            0.012139263756190026,
+            0.00013383660542132047,
+            0.005848259030010402,
+            -0.014767198851314148,
+            0.010570158368602513,
+            -0.011355774405066537,
+            -0.018439261453084426,
+            -0.005243587266408389,
+            0.028894120979627323,
+            -0.005492024421080143,
+            0.017938747701972086,
+            0.0021350711250093858,
+            0.029007356490801176,
+            -0.008686781541806358,
+            0.0014883465513564527,
+            0.0039866022469891155,
+            -0.01295271525889559,
+            -0.004272008629260297,
+            0.01976273843646234,
+            0.04882868638601926,
+            0.010204475902027572,
+            -0.0009155109545661691,
+            0.010069071220988797,
+            0.005626418787776309,
+            0.002148101019776153,
+            -0.00805781810841286,
+            -0.030431071067211577,
+            0.008883707378543252,
+            0.005628218215212441,
+            -0.01997825125961566,
+            0.019838234890736068,
+            0.026483365072033395,
+            0.023040623182975964,
+            -0.011585643598171132,
+            0.016539930468533123,
+            -0.010092529089936188,
+            0.0177008735686489,
+            0.028665424114962996,
+            0.013810061902648515,
+            -0.010487150104822674,
+            -0.0020847870330548534,
+            0.006329342599564054,
+            0.004186981876004016,
+            0.00026239408223817384,
+            0.015190614519766204,
+            -0.019155180136775043,
+            0.01005728016778505,
+            0.014725826697964637,
+            0.03322436221319799,
+            8.049930133627564e-05,
+            -0.01645191835557271,
+            0.0016249281996128196,
+            -0.009976155662939693,
+            0.00856399492362911,
+            0.000632427127231252,
+            0.0032242260692672124,
+            -0.009688119824276678,
+            -0.012682779503915694,
+            -0.002851918183031152,
+            0.009403205508474241,
+            0.0021188716361101698,
+            0.002759004011231831,
+            0.007641925150177999,
+            0.0022830062031146204,
+            0.01713024474384216,
+            -0.0011181980841546123,
+            -0.00029141889484139217,
+            0.004601090256674874,
+            0.010509199590505558,
+            -0.01649852835819883,
+            -0.017548369964263514,
+            0.004337237475053179,
+            0.012217494210546133,
+            -0.013159179948224596,
+            -0.013241981632059755,
+            0.015704543608959117,
+            -0.01913575130892795,
+            0.024934102868065366,
+            -0.00942381892138331,
+            -0.004543611336526587,
+            0.000343200128934031,
+            -0.00727538880611703,
+            -0.0038751118497522673,
+            -0.0030064279504724797,
+            0.003984715814505561,
+            0.0019261818110367147,
+            0.007472350699291607,
+            0.008472369788801792,
+            0.014568547873264247,
+            0.006043895352236181,
+            0.0032580830408778395,
+            0.009109104498825864,
+            -0.014523846294121576,
+            -0.014800649887278973,
+            -0.002704791598011488,
+            -0.008895394038700809,
+            0.00902016867266151,
+            -0.006607182154874112,
+            0.00845408382182461,
+            -0.015501867835442603,
+            0.015573642225745032,
+            -0.012200552219016517,
+            -0.0008900215712478553,
+            -0.009666233545144905,
+            -2.7470277092642562e-05,
+            0.008191697904514857,
+            0.014605287653054787,
+            0.0070552942843382215,
+            -0.031921405225964206,
+            0.0017403427534633655,
+            -0.008846568352320488,
+            0.03446319360336241,
+            0.002612606662691094,
+            -0.01060863123202008,
+            0.020352181356568008,
+            -0.01592124132155158,
+            -0.00027704668715660693,
+            -0.01602075523652062,
+            -0.004797557413047212,
+            0.00707939446796679,
+            0.016399017978990946,
+            0.02759049512664933,
+            -0.005697020356865688,
+            -0.007995305872076221,
+            0.006384047872862793,
+            0.029590696670334232,
+            0.012246955508890425,
+            0.024748351124415785,
+            -0.007633712004672087,
+            0.014348228716100414,
+            0.007773260552660029,
+            0.0076045488653348975,
+            0.004321689250440021,
+            -0.002591801797145718,
+            -0.011698574606499644,
+            -0.014974971814360001,
+            -0.0037563183330623384,
+            -0.016987915346053498,
+            0.00012247833530362867,
+            0.021344400276501117,
+            -0.00475351423959752,
+            -0.0033847161875730026,
+            -0.01874186437694691,
+            0.0033461670218611346,
+            -0.010029445302549983,
+            -0.00367421246775698,
+            -0.013133469163793961,
+            -0.0254122035745965,
+            -0.018556369080030412,
+            -0.002155244024999317,
+            -0.00929309186646207,
+            -0.015194269781263309,
+            0.0070354690219415535,
+            -0.016305117309271634,
+            0.0080837166473439,
+            -0.00448473417024141,
+            -0.016508775226236883,
+            -0.00958925666995396,
+            0.01100335153658195,
+            -0.0041819993634961685,
+            -0.01929493648733792,
+            -0.01282300765645703,
+            -0.011300260564838231,
+            0.0051733704121938884,
+            0.005996780699975213,
+            -0.0038111648304178544,
+            -0.0028279639348479987,
+            0.016640121845366295,
+            -0.010197365811240557,
+            0.009878730312873492,
+            -0.03335474019383245,
+            -0.03551826040406959,
+            0.004497847402931096,
+            0.01507007013703286,
+            0.0013085881889541252,
+            -0.02078680101065954,
+            0.010806315988966003,
+            -0.016499186313125556,
+            -0.00858013792516617,
+            -0.008522534127498645,
+            0.009649521088411411,
+            -0.007283003380127847,
+            0.01299542385123007,
+            -5.3226845309105124e-05,
+            0.030477683597373687,
+            0.0074153455615142755,
+            0.004161881195438522,
+            -0.00679081048132963,
+            0.008354912600159931,
+            -0.001621776416108105,
+            0.004080191054849395,
+            0.02120524874226696,
+            0.012440848502230418,
+            -0.00218041171518331,
+            0.006458415098595872,
+            0.005678055538564355,
+            -0.017186537377121235,
+            0.02507960402044456,
+            0.024678178106806115,
+            -0.008205288425409454,
+            0.004108402466376321,
+            0.005783335302169668,
+            -0.008846663924040241,
+            -0.006897707231869481,
+            0.006527795262026734,
+            -0.018474181903007085,
+            -0.0008721381678008483,
+            0.010598084358464908,
+            -0.02255441909004843,
+            -0.004840511019512733,
+            0.021745651362109403,
+            0.004631567631335321,
+            -0.018558972627958834,
+            0.02764788406651861,
+            -0.004942361238163609,
+            -0.0027290999934532494,
+            -0.010876135202650048,
+            0.01671497143541799,
+            -0.012173527207375091,
+            0.008443317346655462,
+            0.011021541861818043,
+            -0.006295639322287843,
+            0.003083685676540999,
+            0.008277206401503182,
+            -0.004504302913715969,
+            -0.0068075424927125676,
+            -0.0010508416962215848,
+            0.010167160518175114,
+            0.0036980815941666336,
+            -0.022704963865525812,
+            -0.006020073261002279,
+            0.02934227992246887,
+            -0.015082922782242507,
+            -0.0050916398557337705,
+            0.005376294716056329,
+            0.007709342013963906,
+            -0.003476236561444464,
+            -0.0023399991233097337,
+            -0.00787486645740744,
+            0.010356414979428607,
+            -0.015950083845400274,
+            0.02076360898078935,
+            0.025608263494550438,
+            0.024504236990840842,
+            -0.014814235990238182,
+            0.010828584133720797,
+            -0.011703992394890898,
+            -0.011047470905668532,
+            -0.015542835222173864,
+            -0.001658296303013357,
+            0.001970208852236451,
+            0.013107754130616248,
+            0.009666887546609378,
+            0.0235853683657408,
+            0.006473544089109194,
+            0.008973300312205858,
+            0.014118601565410305,
+            -0.003902202854013708,
+            -0.018303438244719996,
+            -0.014584226585460755,
+            -0.011977613285028946,
+            -0.01756345491020528,
+            0.0004563602312336091,
+            0.020461659996271687,
+            0.012417744207683839,
+            0.024136626592375227,
+            0.002119213701333967,
+            0.013759902247467922,
+            -2.9525392347047554e-05,
+            -0.000892319792935443,
+            0.00045783825646242434,
+            -0.005238483120312843,
+            0.019255756091120846,
+            -0.004599752813817219,
+            0.01656589522110913,
+            0.0006443773300684925,
+            0.007133940816771615,
+            0.013554761340256566,
+            -0.01704388669741371,
+            -0.003443100892108368,
+            -0.010029620309829285,
+            0.000941391966877521,
+            -0.019633111750884034,
+            0.01934571074127927,
+            0.010273330977233852,
+            0.0017121345993533963,
+            -0.018327189396014198,
+            -0.0029706385099949693,
+            0.03448635131998772,
+            0.019458999453011265,
+            0.0050734354363614525,
+            0.02632443373898792,
+            0.013018762352749236,
+            0.014443611411502168,
+            0.010153370950346075,
+            0.02356766253713842,
+            -0.023479811300626968,
+            0.02095255383200973,
+            -0.011590347850340786,
+            -0.007414136601134069,
+            -0.0035774888741646705,
+            0.0025141508183687115,
+            -0.013588691733097813,
+            0.00982877622088977,
+            -0.0017916539399655826,
+            -0.025386211797192578,
+            0.009164316443206016,
+            -0.008811052979979271,
+            0.010464442616928296,
+            -0.007774569252594923,
+            -0.0002806136360286729,
+            -0.0127084026190163,
+            -0.015184317235990752,
+            0.001259201880504347,
+            -0.009158429132281081,
+            0.023908150915412617,
+            -0.021210848953875233,
+            -0.01321384014436847,
+            0.013997502713857512,
+            -0.006869443694669701,
+            -0.011364239609894453,
+            -0.00374449980963666,
+            0.023673050288879224,
+            -0.009243341446512421,
+            -0.01359367519009039,
+            0.014464166953530989,
+            0.003834735113400469,
+            0.0059713317014342395,
+            -0.007508862694272452,
+            -0.008428705474135382,
+            -0.003280788036976103,
+            0.005998840043285335,
+            0.026615077617284676,
+            -0.014059930972065399,
+            -0.0032833264512892022,
+            -0.009189999799498163,
+            0.004118204537632102,
+            -0.02359889635281631,
+            0.02771075351384059,
+            -0.008871395303200375,
+            0.021705177758560575,
+            -0.0015924449308442096,
+            -0.006239188158271628,
+            -0.024231036074349822,
+            0.027522945599928003,
+            0.003683870966132199,
+            -0.010221180899173243,
+            0.0012768725460662862,
+            -0.052005292736885915,
+            0.016636109796947025,
+            0.008429895913163985,
+            -0.019507994429834353,
+            0.003697519981610208,
+            -0.004773584856005853,
+            0.018983847580558853,
+            -0.022966993228850697,
+            -0.014078986007109447,
+            0.008283766047449966,
+            -0.01153285124362746,
+            0.0031439180465649996,
+            -0.02527749501821793,
+            -0.005754537200570427,
+            0.0011760345167229269,
+            -0.005276936074280004,
+            -0.001705933750221904,
+            -0.020022761608132548,
+            -0.025972227850795115,
+            0.017666017379157135,
+            0.015820875579600418,
+            -0.002278262448951789,
+            0.01320818011561616,
+            0.008709899932124187,
+            0.00836573261091119,
+            0.016034263394940008,
+            0.007124509234103267,
+            0.0081473711608784,
+            0.045498533239850446,
+            -0.019846323044145485,
+            0.007371629414298193,
+            -0.0014854149174929534,
+            0.013951741898629716,
+            -0.030467924817555014,
+            -0.0055144480998344705,
+            0.0062802744565489945,
+            -0.0049373689340857915,
+            0.03510454786147091,
+            -0.017326094216003073,
+            0.01694160526735264,
+            -0.0026075809870445238,
+            0.0007420394265014909,
+            -0.02042003877666893,
+            -0.026156937726134134,
+            -0.014294938857971834,
+            0.0011857406889443987,
+            -0.0010047941261608975,
+            -0.011946583993069242,
+            0.006797001710574735,
+            0.004917161312925866,
+            -0.0020713379451245515,
+            0.015652523103715316,
+            -0.02952061566240878,
+            0.007356800041369564,
+            -0.010733578620243498,
+            0.020008504830359524,
+            -0.00990331048284697,
+            -0.0013319400098308093,
+            0.007236960885181047,
+            -0.004122868383988748,
+            0.013216043472071927,
+            -0.0038337251451538677,
+            0.004227115653354611,
+            -0.005221321656766545,
+            -0.01794343569722877,
+            -0.0016435261495179767,
+            0.021429008432348098,
+            -0.008349684096540855,
+            0.031864497493526205,
+            -0.01264743878037828,
+            0.009174982784686633,
+            0.010692990337462762,
+            0.0006628831036334152,
+            -0.005286285851702483,
+            0.009052971814211658,
+            0.0021431970274545893,
+            -0.008690520448278274,
+            0.006934310986093342,
+            -0.018349449937310143,
+            0.008494451785507706,
+            0.03394632739506833,
+            0.0018511328780472266,
+            -0.0007519661670070526,
+            -0.007266663179613392,
+            -0.01335844201823745,
+            0.018540713029383066,
+            0.005488273906733518,
+            -0.002546531152114063,
+            -0.027536660869590543,
+            0.02007746621157705,
+            0.0012265798861291223,
+            -0.004755325069578531,
+            0.012466557901411086,
+            0.0020495867632654885,
+            0.019989531151046736,
+            -0.010171678639113316,
+            0.030757957899447096,
+            0.010050036220124393,
+            0.002959833051227505,
+            0.0038556928367700113,
+            -0.036655749074851536,
+            -0.01652152778069314,
+            -0.00019357502472740266,
+            0.014271581054865031,
+            -0.012430647252826173,
+            -0.022432580814844155,
+            0.0010346169306977334,
+            0.01148004051449691,
+            -0.0033898642762881046,
+            -0.009650665305271706,
+            0.01560757541276278,
+            -0.020342213902670575,
+            0.006701302394408393,
+            0.007693672045981921,
+            0.0287100214318104,
+            -0.0020735384554022153,
+            -0.0012324838849990056,
+            0.000626416803904792,
+            -0.0055669860801192935,
+            -0.0008037366126870824,
+            0.017251369734470945,
+            -0.013661316947002466,
+            0.0175926211217534,
+            -0.012735241490550407,
+            0.0064713656749838585,
+            -0.0021092378432037563,
+            -0.010629579860050812,
+            -0.020114807971202787,
+            0.0029609495764307357,
+            0.009378003163375899,
+            0.006920144698428806,
+            -0.010438984881449953,
+            0.010715409812532968,
+            0.012197673689706352,
+            -0.00753962861484439,
+            0.014519905132928041,
+            0.022173989895384192,
+            -0.007687503607783572,
+            0.013633321414009471,
+            -0.0041446334728745095,
+            -0.014056264816568229,
+            -0.009985463824440833,
+            0.023929750617160402,
+            -0.0005563806942667597,
+            0.008756997373618809,
+            0.00388125051641897,
+            0.0007856447663040696,
+            -0.011226530517316183,
+            -0.001653447552601681,
+            0.019294448360333633,
+            0.011831934675073843,
+            -0.0013888534822008192,
+            -0.0010525764356699885,
+            -0.0014566936525653407,
+            -0.009427557739483988,
+            0.02049188741459728,
+            -0.022030783927622997,
+            -0.0019604940727076573,
+            0.00461018148439991,
+            -0.022589816803875618,
+            0.0001863117949313752,
+            -0.024570217962989905,
+            -0.01569952241187498,
+            0.011763033781122343,
+            -0.0022530631823737554,
+            -0.016028810867854907,
+            -0.0003242701040942633,
+            7.424127301806556e-05,
+            -0.007815789940508527,
+            -0.004135500768181258,
+            0.006115909992703774,
+            -0.00034193217510010117,
+            -0.0032262189374874842,
+            0.013430154310506884,
+            -0.009996622196870008,
+            0.015431528911575157,
+            0.0067029980830614145,
+            -0.015702167721800483,
+            0.0021406610839557003,
+            0.018500219544264753,
+            -0.0058910087864916085,
+            0.00787021425357441,
+            -0.014681907249972176,
+            -0.0037495551894426303,
+            -0.005787661351944086,
+            0.006051351977821761,
+            0.024849195780885833,
+            0.012969434450643372,
+            0.01139340445077682,
+            0.017405361843075933,
+            0.004325000715054734,
+            -0.01572297405297993,
+            0.0007416798945675923,
+            0.01682727189828363,
+            0.010464036837245312,
+            0.009804326468113703,
+            0.0008975556667684365,
+            0.0043704860361487635,
+            -0.005142176468800175,
+            0.009317064072855551,
+            0.02915482449687158,
+            0.0013434689612997423,
+            0.008696141010760192,
+            0.017276730345147134,
+            0.027038741396131265,
+            -0.0035088997194497725,
+            -0.007469614319788499,
+            0.008739942310300076,
+            -0.009657991604173173,
+            -0.001081629174976152,
+            -0.017000351183542026,
+            -0.012310275736750868,
+            0.016510505259358398,
+            -0.006084627062230492,
+            0.02848913486719956,
+            -0.0010655806791907225,
+            -0.02026498848361482,
+            -0.01565505608194376,
+            0.001404549479440637,
+            -0.01362132895080652,
+            0.005287566256051299,
+            0.014187265685427415,
+            0.01917046513020831,
+            0.004687191682875024,
+            -0.022258337721085656,
+            0.037364962197399806,
+            0.01757108963349427,
+            0.019377870795760123,
+            -0.019422840506328382,
+            -0.006159311237412833,
+            -0.009077095160676276,
+            0.02116697012944498,
+            -0.00880692095488465,
+            -0.009875674856043954,
+            0.024014332004451508,
+            0.017812606845656484,
+            0.005679806286464216,
+            0.006736609904079586,
+            0.0020048663439319585,
+            0.0025414630253661444,
+            -0.015831593851012047,
+            0.017957902498542212,
+            -0.005858125698347523,
+            -0.0058068554681761945,
+            0.017529514573571298,
+            -0.005989581792993925,
+            -0.0030306021203732347,
+            0.00015839997806822045,
+            -0.019697052145531482,
+            0.011618112886207201,
+            0.008256007601576028,
+            -0.017229774429089857,
+            -0.0014757560402567304,
+            -0.0002404480326945295,
+            -0.014118694740134205,
+            0.02004885146995733,
+            0.01694893356262835,
+            -0.0013248884139405398,
+            -0.0051752815372013315,
+            0.008223868147393114,
+            0.007000679132597393,
+            0.005256194890343006,
+            -0.019496098876977206,
+            0.005230597149342019,
+            0.005103909294824105,
+            -0.0014716649591412774,
+            0.010274346323051634,
+            -0.009030594825985526,
+            0.033740380586893826,
+            -0.006505788197604522,
+            0.013065658941136721,
+            -0.0043634731684193745,
+            -0.0055529072007985125,
+            0.001816020676232658,
+            0.013054468478598695,
+            0.009753501030700457,
+            -0.00970446155762939,
+            -0.004172354962129865,
+            0.018697982092775737,
+            -0.011983156146516074,
+            0.0013300791598046338,
+            0.01564072865804035,
+            -0.004858814041438713,
+            0.004708235083775013,
+            0.008635204790345423,
+            -0.004489505237783172,
+            0.01801512858970922,
+            -0.016482572999490366,
+            0.0023953436660054026,
+            0.010694869684962088,
+            0.02390864159830096,
+            -0.019152187046410703,
+            0.021786596486708207,
+            -0.017951050280323393,
+            0.026538582634436314,
+            0.014484894489694262,
+            -0.0035222259457311444,
+            -0.01946200885147006,
+            -0.00850690913394181,
+            0.011654565129063417,
+            -0.021423803876241532,
+            -0.0007985948078290641,
+            -0.0007290222944686444,
+            -0.021367628543092133,
+            -0.011189364017231132,
+            -0.009961779129778331,
+            0.010701637745810138,
+            0.001918633898887013,
+            0.010380983868185359,
+            0.008002182827196827,
+            0.010784760574518087,
+            0.014405640442760822,
+            0.0008958107769384412,
+            -0.010743001445440941,
+            -0.023795589003655273,
+            0.016987708698679597,
+            0.00438463274382093,
+            0.0007862212695139549,
+            -0.01627468994515667,
+            0.011936651269737248,
+            0.013250643815768723,
+            -0.009385605548926286,
+            0.009121882469425302,
+            -0.03716901103204463,
+            -0.004552808410398574,
+            -0.0035704811925430366,
+            -0.0020309197365173475,
+            -0.00908678158053104,
+            0.01559445415882485,
+            -0.013682507836005851,
+            -0.0009590922904844632,
+            0.00011696985057536548,
+            -0.0074756526903548915,
+            -1.5467890432254378e-05,
+            0.020649928837031005,
+            0.008438785939908428,
+            -0.0037382105257999757,
+            0.010376213224654074,
+            0.008826031303751312,
+            -0.013900359449287676,
+            -0.01527474712786084,
+            0.019035366035884273,
+            -0.018776918681908793,
+            0.0063098418544296165,
+            0.01677916431403883,
+            0.020778813096283988,
+            -0.02891862501830075,
+            -0.0007231469961304435,
+            0.022093939332021185,
+            -0.0012807635799950273,
+            0.006370841109790851,
+            0.008781853138377092,
+            0.006754082980602544,
+            0.0029102159062355374,
+            -0.003711173531879644,
+            -4.843770941622283e-06,
+            -0.003940474588075707,
+            0.007155539022862352,
+            0.004366433868349658,
+            0.036453728071874844,
+            -0.013661129348165794,
+            0.002541308805735001,
+            -0.0058838191638835155,
+            -0.00347401304572066,
+            -0.00593469688967835,
+            0.004407665856950772,
+            -0.010055727994798024,
+            0.004330827357536078,
+            -0.001570861535288093,
+            -0.0003731129422346693,
+            0.004883969598300302,
+            -0.001701673379646408,
+            0.007036892564844683,
+            -0.011863650867472625,
+            -0.0003979727750011258,
+            -0.020174851412609312,
+            0.021117284557875267,
+            0.00011666551282792189,
+            0.003662462447326642,
+            0.03723741246290771,
+            0.00476064510662738,
+            -0.00238355131307445,
+            -0.012880057949435798,
+            -0.013551338766321525,
+            -0.012986444917747435,
+            -0.020703821629987552,
+            -0.0036665673548525388,
+            -0.021006311026247164,
+            -0.0052061406234528915,
+            0.010362862594785442,
+            -0.011540184420787572,
+            -0.004023659825622843,
+            -0.015881589631889473,
+            -0.006531981517815174,
+            -0.03114211633687585,
+            -0.01124607488650473,
+            0.007493228366452009,
+            -0.019149285949049886,
+            -0.0039932870926865805,
+            0.026850121590606017,
+            0.001095360635760988,
+            0.01387696424607248,
+            0.0038058248676879625,
+            0.008117686785954085,
+            -0.0045937472017383955,
+            -0.0005669025036141417,
+            -0.01928244877716143,
+            0.003391980330529026,
+            0.008525354854576925,
+            -0.00361507909530282,
+            0.0070008404454106905,
+            -0.008835727374865623,
+            -0.016950017602980382,
+            0.0021654956715219517,
+            0.009334379529385142,
+            -0.006639140003365139,
+            -0.017587472740268683,
+            -0.011049929670847685,
+            0.010401189523318744,
+            -0.02283429470064187,
+            0.006257145380266984,
+            0.016439581169238607,
+            -0.009363030375624585,
+            0.004651575425738153,
+            0.019809183749755425,
+            -0.011810674576781702,
+            -0.005811328501528397,
+            0.008471992070998505,
+            0.013891259999721588,
+            0.0095606134736748,
+            0.02954498207274139,
+            0.0026596809804008996,
+            0.000561870602466584,
+            0.004889263753490895,
+            -0.004277061142504552,
+            -0.017423200644235665,
+            -0.006243787897080996,
+            0.0038507494787521856,
+            0.012873996671952184,
+            -0.009879417960879032,
+            -0.00048575277472139055,
+            0.01682604801673206,
+            0.004530939990187663,
+            -0.0033901248079550284,
+            -0.011868399508738274,
+            -0.020080331108085103,
+            0.0015580259874809898,
+            -0.01362200460317948,
+            0.015354347040143093,
+            0.003390601069331352,
+            0.0015284610287850575,
+            -0.004239880336806429,
+            0.01828645373058079,
+            -0.005013418527935815,
+            0.012749833447770729,
+            -0.006148890994661865,
+            -0.004587831231448011,
+            0.007986533783001116,
+            0.009166578151006778,
+            0.0004939857995863664,
+            0.05421019830078428,
+            0.002838056604877719,
+            0.01333205648597435,
+            0.013419127695838135,
+            -0.014523733369484934,
+            0.014086525147734823,
+            0.012688569853552613,
+            0.018497344547821693,
+            0.013037819487160016,
+            -0.012353776077273462,
+            0.017354801119939964,
+            0.0024084723134891487,
+            0.003909681065813095,
+            0.0022202438064449653,
+            -0.004528576079628071,
+            0.019594683950225115,
+            0.004070222328547378,
+            -0.0074362443729372105,
+            0.004453618452623641,
+            0.005007543168224267,
+            -0.013105524222311019,
+            -0.0112958915514782,
+            0.027105208848863218,
+            -0.01219417557128091,
+            -0.018749971743148304,
+            0.006706594269933445,
+            -0.02008458683853808,
+            -0.03763686535076864,
+            -0.0021332074289244973,
+            0.009992361098878196,
+            0.002795183524495193,
+            -0.00253695604043486,
+            0.01614298064601032,
+            0.016730443047719683,
+            -0.016985296937992726,
+            -0.018245902728910347,
+            0.0025257780765481825,
+            0.0016333325293570182,
+            0.01861807236979066,
+            -0.023168663546030196,
+            -0.006378864850313035,
+            -1.3795465857380962e-05,
+            -0.006829008132963306,
+            0.004850306884033312,
+            0.004703280112191218,
+            -0.0071714759384722965,
+            -0.0027773519096287217,
+            0.002256846645341137,
+            0.01975383140755998,
+            0.013198754584066554,
+            0.01572971337890945,
+            0.002600741959577911,
+            -0.005033508062449482,
+            -0.016817359058724685,
+            -0.005450011991586895,
+            -0.013478330855299777,
+            -0.009091854345715177,
+            -0.003042052092498012,
+            0.008217376329142566,
+            -0.025053469059115735,
+            -0.011965083373220982,
+            0.0119294363235832,
+            0.00391377331459168,
+            0.0003099203772539202,
+            0.007607672728136279,
+            -0.003830132495364688,
+            0.010331053246451333,
+            0.013823890473448383,
+            0.0076112887952959766,
+            -0.004723377809559611,
+            0.009600411526173948,
+            0.011460781436250457,
+            0.006015477068316837,
+            -0.005072711638895421,
+            0.023859899048048694,
+            0.01821920335673539,
+            -0.01680365156728093,
+            -0.002149107383573353,
+            -0.0027829676356423997,
+            -0.01884308738652242,
+            0.009665830801645276,
+            0.00880322398336952,
+            0.008308268605449346,
+            0.003906369351868086,
+            0.00974531700889854,
+            -0.013875904819275925,
+            -0.0050570811397453235,
+            0.002870120865445989,
+            -0.005729556771404781,
+            -0.01901337075686543,
+            0.000297242372701423,
+            -0.011987192261104132,
+            -0.011327157550660895,
+            -0.01409710922165158,
+            -0.003221240464411861,
+            0.005283472511900512,
+            0.021472187640986972,
+            -0.004410881709548286,
+            -0.001855852749039012,
+            0.01351697599517034,
+            0.010060125186506398,
+            0.0021658738823899905,
+            0.0002206474541239167,
+            0.005418828324605768,
+            -0.0027919130944783397,
+            0.0008094042430809068,
+            -0.0015826797905407809,
+            0.021089276765434876,
+            -0.02016611086095107,
+            -0.009413977569213423,
+            -0.009288078098692112,
+            0.005842197094339927,
+            0.028793032462603636,
+            -0.007165634502373454,
+            -0.010104002083017078,
+            -0.01994315293556678,
+            0.0003786527922888834,
+            -0.07810609175343784,
+            -0.005210636756258548,
+            0.001870513618910238,
+            0.03126380113105983,
+            -0.010817888476436834,
+            0.0033673768058629823,
+            -0.002204487854177595,
+            0.014542299655712928,
+            -0.001760107965206874,
+            -0.0013283746982194259,
+            0.0071954806737858056,
+            -0.005640400818010697,
+            -0.006210989097047217,
+            0.005904163661619133,
+            -9.945941028848348e-05,
+            -0.010266938705916303,
+            0.00057263080955709,
+            0.02209498395848068,
+            0.0159398346608872,
+            -0.006575439759804175,
+            0.01816748552021211,
+            -0.00719294553484007,
+            -0.009838738592324234,
+            0.005639288780502127,
+            -0.0019007683459126327,
+            -0.015369619176147573,
+            -0.011475977270803784,
+            0.009862808084641291,
+            -0.030069890644150284,
+            -0.00010725753918890802,
+            0.00943698424646706,
+            -0.013540245857270492,
+            -0.008293042205008774,
+            0.007212974960781723,
+            0.006708343474532583,
+            -0.0012017968151025369,
+            -0.0078026000239507145,
+            -0.02527526731750743,
+            0.0027578400032925625,
+            -0.002446811971857853,
+            -0.004899702349784173,
+            0.0019960465173086183,
+            0.001479503476855694,
+            -0.012645905858466798,
+            0.01216748847452763,
+            -0.013378877368040743,
+            -0.012053251571418766,
+            0.012508842412556306,
+            -0.00027167241745418584,
+            0.01682680930386839,
+            -0.020607299598265426,
+            0.0010702151063620052,
+            -0.028075439824932304,
+            -0.007671848153932823,
+            -0.015486630894165054,
+            -0.0009776136330459065,
+            -0.0012939921569540803,
+            -0.03481880531471923,
+            0.012889447665794033,
+            -0.005200149829198305,
+            -0.006094702500154477,
+            -0.00459778303079732,
+            -0.024583504817899775,
+            0.006072281238752663,
+            0.00607350167867964,
+            -0.006506508970816635,
+            -0.016816439635445445,
+            0.008428048142588516,
+            -0.0011040593013238872,
+            0.004873670535200424,
+            -0.007475800310154908,
+            -0.0005638920132076605,
+            -0.013047135093134946,
+            -0.005463454269764494,
+            -0.0008522562324122834,
+            0.01827123304871433,
+            0.0044055851550317835,
+            -0.014526381222031981,
+            0.016862445236481526,
+            0.013250692156820124,
+            0.01602714293656341,
+            -0.0071089293427753115,
+            -0.0004265739315250755,
+            -0.02750630108787212,
+            -0.00327225092954903,
+            -0.023687644154808972,
+            0.00041213204365507744,
+            0.009447660230507965,
+            0.0203905264889617,
+            0.012694421477479161,
+            0.004620617778996077,
+            -0.001557511764988156,
+            0.0066751090724502,
+            0.02696959560423252,
+            -0.01099183126742224,
+            -0.006969111598423376,
+            -0.01884504192789845,
+            0.01113023717919988,
+            0.025135748541199255,
+            0.003381303139273845,
+            -0.011317957771935042,
+            0.008112598754655146,
+            -0.0009247845526290428,
+            -0.010302980598863016,
+            0.02889587382004902,
+            -0.0391134481322538,
+            -0.015599591108049343,
+            -0.004605824192962023,
+            0.007406520608110251,
+            -0.015596917115435033,
+            0.00770468362805995,
+            -0.012614339656105264,
+            0.001699049329817941,
+            -0.005851477164035241,
+            0.03535087340659861,
+            -0.0031950960727572925,
+            0.0006909856233673865,
+            -0.003395233050301427,
+            0.003531135044131077,
+            0.012355948349937309,
+            -0.004444221317023453,
+            -0.030726300006909638,
+            -0.0022983766869702346,
+            -0.011198252640110788,
+            0.009274666307515955,
+            -0.016207714291658314,
+            -0.017342435976846468,
+            -0.009424737744066259,
+            -0.021827243671837953,
+            -0.012403084982789827,
+            0.029747453318486338,
+            -0.005824001042844527,
+            0.01747569398142269,
+            -0.005219060548639168,
+            -0.008365003057800934,
+            0.00021272411948388247,
+            0.01344335658390732,
+            -0.028498240040284714,
+            0.00041963499785238845,
+            -0.002322931702641349,
+            0.0126393550661767,
+            -0.001595134597082556,
+            -0.00983073906977773,
+            0.006903313245694627,
+            0.0010522208685011098,
+            -0.015648940549713764,
+            -0.0007573245570947632,
+            -0.008533504878052644,
+            0.020475329445149498,
+            0.00027791387213409814,
+            0.001074213629646005,
+            0.013864331671732542,
+            -0.02102352993295736,
+            0.01576938071930521,
+            -0.021722775924283996,
+            0.01893267442161547,
+            -0.006577103101038035,
+            0.005772016664149475,
+            0.008283223343661555,
+            0.014068789332250016,
+            -0.014030042849491258,
+            -0.00663752723080222,
+            -0.034436220972634726,
+            -0.0122621879114314,
+            -0.001946072976826902,
+            -0.005308941654794638,
+            -0.014773123927909993,
+            -0.0011054822266082056,
+            -0.026205461509573584,
+            -0.006460373065662098,
+            0.012822188046808264,
+            0.019900312098359022,
+            -0.008409070094265829,
+            0.014163132434889265,
+            -0.006743711872055091,
+            -0.0028230771810882492,
+            -0.012652021749331811,
+            -0.018061180380009395,
+            -0.01083672324723357,
+            -0.01890966636827258,
+            -0.0018635408070944168,
+            0.004709222372611061,
+            -0.015760618199235518,
+            0.011435131737642627,
+            0.03198874888333405,
+            -0.01567271472650171,
+            0.008243406188694909,
+            0.016411496328333205,
+            0.0034236971031973503,
+            0.011728079139322837,
+            0.0038311415233052795,
+            0.004621789116267879,
+            -0.011109450058055078,
+            -0.011473158133621661,
+            0.006524490579223986,
+            0.00954511918454185,
+            -0.004913142601134428,
+            0.024347072255007807,
+            0.0008745419206472714,
+            0.005045794664611216,
+            0.005304506902446091,
+            0.011488187043525824,
+            -0.013135271908307747,
+            0.019746484188715552,
+            -0.002055210086078267,
+            -0.015571665869889405,
+            -0.01535499169234956,
+            -0.01105419264826655,
+            -0.018626399757822328,
+            0.0017108083455869296,
+            0.0181439079855921,
+            -0.0012493542718277685,
+            0.012137753522284387,
+            -0.00924460959150499,
+            -0.005019437549649743,
+            -0.0027156377206998474,
+            -0.038211954658821295,
+            -0.0021878970332906457,
+            -0.018809487178512628,
+            -0.02498945889484935,
+            0.03505216568400238,
+            -4.26477081213836e-05,
+            0.00997643255528116,
+            -0.010310197159232386,
+            -0.0002454902098986129,
+            -0.011065403358802515,
+            0.014954998647970473,
+            0.004624123403100598,
+            0.008872923678983478,
+            0.008560280683876702,
+            -0.016419500515801093,
+            -0.002997557798375716,
+            0.007472390947008284,
+            -0.00374822278027057,
+            0.012680407710694952,
+            0.015353684671120731,
+            0.020886474047003733,
+            0.011161262397502876,
+            0.002045862158732102,
+            0.00887271411363814,
+            0.009563361150297699,
+            -0.0003513237086717377,
+            -0.010459621030424415,
+            0.015419039131459582,
+            -0.02204848739950703,
+            0.012172257374410352,
+            0.04346634265794738,
+            0.019197433209794286,
+            0.029488039318675732,
+            -0.02272810439246353,
+            0.01448092574459596,
+            -0.003801043409441206,
+            0.009476229966122152,
+            0.0049309968883087685,
+            0.02346191840355209,
+            0.003973840862678076,
+            0.0073546443568171405,
+            0.010647581332452525,
+            -0.00972051412996416,
+            0.025644022498404723,
+            -0.00846669916801108,
+            5.1518540791212095e-05,
+            -0.004407265804145216,
+            -0.026400029105519137,
+            0.01615193008014211,
+            0.0081722536401398,
+            0.0010702314763197463,
+            0.0010263623479474385,
+            -0.0019566604229607643,
+            0.007982949510899019,
+            0.007079322685771456,
+            -0.015274148711633536,
+            -0.003886053737590683,
+            -0.018527455865296014,
+            0.0012864569064749438,
+            -0.024337133883771334,
+            0.010466651250597847,
+            -0.017568706814132484,
+            -0.014822354159709854,
+            -0.021666681078831483,
+            -0.013232016867264097,
+            -0.02011653224168991,
+            0.0035858355517491515,
+            -0.008632755352062642,
+            0.013025728891203198,
+            -0.016962939287139683,
+            -0.021494971929779273,
+            0.008430887589398292,
+            0.00870802359711456,
+            0.007481282677955534,
+            0.009728368603458082,
+            0.014996157358749115,
+            0.01736292711410622,
+            -0.022712386143092265,
+            -0.023163001611599246,
+            -0.006682209717364464,
+            -0.003531495447167166,
+            0.006762379833125297,
+            0.00933681196321128,
+            0.012884185500343249,
+            -0.007077895849501418,
+            -0.007554762054641417,
+            -0.013283067570021503,
+            0.006197966297581871,
+            -0.003993067576677973,
+            -0.009464789349849612,
+            -0.0015662739825834614,
+            -0.0061389795593180686,
+            0.002753664732282926,
+            -0.0001791088888074743,
+            0.0007595261237141144,
+            0.017122834676182343,
+            -0.008176319370751018,
+            -0.009787747261447871,
+            -0.02027228453943018,
+            -0.010683273300243025,
+            -0.015604745521156111,
+            -0.0016279406232527138,
+            -0.006506713191861036,
+            -0.014420323628589382,
+            0.010888206569219648,
+            -0.011930708571093279,
+            -0.00887210117616784,
+            0.010807147705360856,
+            0.0073900910661282935,
+            -0.003431679006289002,
+            0.037804020313161775,
+            0.0011945597874820288,
+            0.00778778394048267,
+            -0.003366543306746973,
+            0.005296727102519836,
+            -0.017761251541598876,
+            -0.026382400396890007,
+            0.0010357716464853195,
+            -0.01775777668381489,
+            0.010807713750049491,
+            -0.008515368407459745,
+            0.0016051389544388124,
+            -0.00034958600424912995,
+            -0.009585854918521076,
+            0.0036549720706351473,
+            -0.008110927907950148,
+            0.004212219431447508,
+            -0.03507890639477636,
+            0.011409377585617926,
+            -0.01819120811524444,
+            -0.008962242411035718,
+            0.01710177594046791,
+            0.037560936862121835,
+            0.0028513388998398104,
+            0.00030058353811717895,
+            0.0014792184333546447,
+            0.004427921639326168,
+            0.021438127785303025,
+            0.006339483221420334,
+            -0.001187747087595264,
+            -0.02237982599829809,
+            0.0062435650294483695,
+            0.002200578793697352,
+            0.010092828474934744,
+            0.023105023225115266,
+            -0.016797326021070008,
+            -0.015378520287395565,
+            -0.012732527281000097,
+            -0.010797463778182385,
+            0.009881499418260473,
+            0.02801767062785128,
+            0.01811139137849365,
+            -0.017358098300028903,
+            -0.020207173948076473,
+            0.0016242901745373126,
+            -0.026331540602868762,
+            0.007269751661479859,
+            0.11940631114242901,
+            -0.0045382135734049305,
+            -0.005994407567723939,
+            -0.008679712231646732,
+            -0.004482553208064535,
+            0.016709571294385293,
+            -0.013387043582971358,
+            0.069179848974036,
+            -0.007221545718220735,
+            0.0037392229364642915,
+            0.0028916224964057385,
+            0.00462729158956839,
+            -0.009727339449635306,
+            0.011570145487622916,
+            -0.011287365791681587,
+            -0.012991994493845187,
+            -0.013076194601468383,
+            -0.0180830261130804,
+            -0.007272768410218907,
+            -0.004155564814758654,
+            0.004204993170487392,
+            -0.005525575983182816,
+            0.008051108079520902,
+            -0.017551645696802648,
+            -0.010800070994485792,
+            0.013007176804761548,
+            -0.0023627485852619817,
+            0.005390986239687555,
+            -0.029668390278675554,
+            -0.0007650391259819867,
+            -0.0002446394046947245,
+            0.021280292985495608,
+            0.003137771886236958,
+            -0.0019077298815862946,
+            -0.010810159675564049,
+            0.007040107942856957,
+            0.023835993694726953,
+            0.021875366566457236,
+            0.0180511566130312,
+            0.012135351541044354,
+            0.020483975362807574,
+            -0.008667079096167644,
+            0.026324312888512866,
+            -0.001552207146726937,
+            -0.00397810465151493,
+            0.0077577136352257004,
+            -0.009849600842081875,
+            -0.00816392268697345,
+            0.00677353736462267,
+            -0.017288900282504148,
+            -0.020418071729964713,
+            -0.020383762873109595,
+            0.025741408097358233,
+            -0.011755094311186617,
+            0.002737129201159793,
+            -0.008266210469668866,
+            0.027287436144516414,
+            -0.006283892307996516,
+            -0.0029440642954701632,
+            0.009428876742171362,
+            -0.03788574758692254,
+            0.008834605607057885,
+            0.006463342392514337,
+            0.026711196829464153,
+            -3.9827332346130654e-05,
+            0.005565561841469247,
+            -0.0006738085601962865,
+            -0.006104320995853506,
+            -0.010792830169647851,
+            2.4565453208679464e-05,
+            0.017983908748172493,
+            0.0035181716362611077,
+            0.016970206528324653,
+            -0.005449808667944701,
+            -0.023171388778833197,
+            0.007092509502738752,
+            -0.01555320330471131,
+            -0.023306285898839978,
+            -0.002028783456704567,
+            -0.03196436161255752,
+            0.0033412618908711163,
+            0.007527633574188688,
+            -0.01890570959143866,
+            -0.017859735533232564,
+            0.010329340599904344,
+            0.004382396539605985,
+            -0.0016746397858953155,
+            0.006643785064338525,
+            -0.03307812909454471,
+            -0.01927475592253288,
+            0.004686970010943116,
+            0.012733525168279355,
+            -0.012993441126970916,
+            0.015291832952199082,
+            -0.020491167895268936,
+            0.016451976170508816,
+            -0.012259387189535237,
+            0.018393274368854132,
+            -0.006997668664226754,
+            0.011843450716200625,
+            0.011457415642860215,
+            0.009560877634897626,
+            -0.01790001115302499,
+            0.0214879415944361,
+            0.0066829070506954115,
+            0.03900547021123392,
+            0.02512484499576633,
+            -0.012651294056575184,
+            -0.0018425680226742782,
+            -0.00011361458320146007,
+            0.014982647266052349,
+            0.0061874424115559,
+            -0.003689464024837091,
+            0.02705902205734107,
+            0.018239469373522537,
+            -0.021070704644859155,
+            0.020037508334843183,
+            -0.00973804771680803,
+            0.018876283979591423,
+            -0.013303080480396818,
+            0.010592424494628194,
+            0.004717870260467274,
+            0.01070131238869639,
+            0.009087656015197565,
+            0.006708673669681232,
+            0.025928065153829043,
+            0.03663367449576964,
+            -0.00511073108390551,
+            0.002161381031392467,
+            0.004864092591944181,
+            0.012814518637357098,
+            0.001556833001747549,
+            0.007915766323215207,
+            -0.007012785514562176,
+            0.005834786334750152,
+            -0.002311533032619821,
+            -0.00122162928193159,
+            0.019525999280692766,
+            0.000824163362963678,
+            0.012875575199417254,
+            0.013259376059185821,
+            -0.002390133687709065,
+            -0.005682644446302278,
+            0.0019262218417782657,
+            -0.018193478045923098,
+            -0.015131696847666434,
+            0.029463723011600933,
+            -0.011447039651934148,
+            -0.008083114389964213,
+            0.0020642857121204753,
+            0.012160202510990566,
+            -0.00866944494225225,
+            -0.005378300635997713,
+            0.0038280364584418335,
+            0.014427318067525318,
+            -0.01799358399064332,
+            0.003381165206832161,
+            0.005252552821854107,
+            0.019433554824232185,
+            -0.00028791985674147885,
+            -0.013487678738391956,
+            0.0005317922815671783,
+            0.014982152801567637,
+            0.011902348165369078,
+            0.0016056390299304837,
+            -0.006935777197137747,
+            -0.00446584737370504,
+            0.004586814866182051,
+            0.006269466956110759,
+            0.0027516492952993445,
+            0.008157662045793032,
+            -0.0025001885993292637,
+            -0.016648747436203294,
+            0.017735757469839637,
+            -0.0030669873770852725,
+            0.018058248627079793,
+            -0.0038238312452176745,
+            -0.0147172470474118,
+            0.0043328417868095355,
+            -0.001888507803054499,
+            -0.018556943283494755,
+            0.009407930200720534,
+            0.002560949424273885,
+            0.010411159836451958,
+            0.0052571230238294825,
+            -0.007499470181994448,
+            -6.145889479758644e-05,
+            -0.015163819825329975,
+            0.022836682800304944,
+            0.016390944816242627,
+            0.001713572907204073,
+            -0.0035687505973373878,
+            -0.031210299824387475,
+            0.0029076511444359886,
+            -0.007822231980296341,
+            0.017821806420465356,
+            0.008250605126658402,
+            0.007908773724009905,
+            0.008351355849444162,
+            -0.00743970642051451,
+            -0.002411981861500092,
+            0.012543476434788293,
+            0.0202407777715137,
+            -0.0024635818448627447,
+            0.02535771413579989,
+            -0.012533449594824386,
+            -0.0022489292647974992,
+            0.0012362917358624392,
+            0.01578939075186126,
+            -0.0008001783766371555,
+            0.006311837812042848
+          ],
+          "keyphrases": [
+            "Switch Transformers",
+            "Mixture of Experts",
+            "sparsely-activated model",
+            "training instability",
+            "multilingual settings"
+          ],
+          "title": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "8f4dd4ba-f551-4c79-af63-003b9ccf3976",
+        "properties": {
+          "page_content": "8 Future Work 26 ",
+          "keyphrases": [],
+          "embedding": [
+            -0.024294501170516014,
+            0.01452074758708477,
+            -0.01423195656388998,
+            0.00711597828194499,
+            -0.015378094278275967,
+            0.004546194337308407,
+            -0.010901841334998608,
+            0.11226730793714523,
+            0.006362415850162506,
+            0.04544840380549431,
+            0.02425840124487877,
+            5.9330373915145174e-05,
+            -0.00413782661780715,
+            -0.018969925120472908,
+            -0.018825531005859375,
+            0.03312066197395325,
+            -0.018969925120472908,
+            0.014204883016645908,
+            -0.029601026326417923,
+            -0.0012081821914762259,
+            -0.01162381749600172,
+            -0.0036031126510351896,
+            -0.03490755334496498,
+            0.01664155349135399,
+            -0.001300685340538621,
+            0.011109409853816032,
+            0.04822801426053047,
+            0.030413249507546425,
+            -0.05245157331228256,
+            0.018013307824730873,
+            -0.007968813180923462,
+            0.023933513090014458,
+            0.019258717074990273,
+            -0.005049321334809065,
+            0.012463115155696869,
+            0.006443637888878584,
+            0.027326801791787148,
+            -0.008451634086668491,
+            0.0013762671733275056,
+            0.012914350256323814,
+            -0.0050673708319664,
+            -0.00494553754106164,
+            -0.009110437706112862,
+            0.026081392541527748,
+            0.035539280623197556,
+            -0.01277897972613573,
+            -0.012950449250638485,
+            -0.04808361828327179,
+            0.0016458802856504917,
+            0.007197200786322355,
+            0.0050809080712497234,
+            -0.022200770676136017,
+            0.001969641540199518,
+            0.0006689561414532363,
+            0.02433059923350811,
+            -0.00736415758728981,
+            0.012986548244953156,
+            0.07804563641548157,
+            0.020558273419737816,
+            0.007833442650735378,
+            -0.022200770676136017,
+            -0.02465548925101757,
+            -0.0032579179387539625,
+            -0.03779545798897743,
+            0.016109095886349678,
+            -0.02696581371128559,
+            0.029564928263425827,
+            -0.011903583072125912,
+            0.009882049635052681,
+            0.0025224045384675264,
+            -0.0248179342597723,
+            -0.005893131252378225,
+            -0.007359645329415798,
+            0.04508741572499275,
+            0.00730549730360508,
+            -0.017210109159350395,
+            0.026370182633399963,
+            0.018753333017230034,
+            -0.02730875089764595,
+            0.046711862087249756,
+            0.01647910848259926,
+            -0.03963649645447731,
+            -0.005058346316218376,
+            0.021496843546628952,
+            0.024529142305254936,
+            -0.004372468683868647,
+            -0.04992465674877167,
+            -0.11732114106416702,
+            0.016452033072710037,
+            0.0011562900617718697,
+            -0.005071883089840412,
+            0.015332970768213272,
+            0.0010463015642017126,
+            -0.028680507093667984,
+            -0.04238000512123108,
+            0.04259659722447395,
+            -0.00870432611554861,
+            0.010405482724308968,
+            -0.0004030094132758677,
+            -0.020215334370732307,
+            0.01059500128030777,
+            0.005374210886657238,
+            0.005992402788251638,
+            -0.02342812903225422,
+            0.0422356091439724,
+            -0.02474573627114296,
+            0.019872397184371948,
+            0.015026130713522434,
+            0.0034542051143944263,
+            -0.002572040306404233,
+            0.005545679945498705,
+            -0.033644095063209534,
+            -0.022327115759253502,
+            0.018148677423596382,
+            -0.006168384570628405,
+            -0.01757109723985195,
+            0.013952190987765789,
+            -0.03044934757053852,
+            -0.029474681243300438,
+            0.04494302347302437,
+            0.04559279978275299,
+            0.019511409103870392,
+            0.027687789872288704,
+            -0.03633345663547516,
+            -0.032073795795440674,
+            0.02905954420566559,
+            0.012327744625508785,
+            0.00182637432590127,
+            -0.003952819854021072,
+            0.03254307806491852,
+            0.02994396537542343,
+            0.004620647989213467,
+            -0.02211052179336548,
+            -0.03440216928720474,
+            0.021478792652487755,
+            -0.02243541181087494,
+            -0.009548136033117771,
+            0.024186205118894577,
+            -0.0227603018283844,
+            0.01911432109773159,
+            0.004505583085119724,
+            0.004850777797400951,
+            -0.011632842011749744,
+            0.0054193343967199326,
+            -0.006348878610879183,
+            0.03633345663547516,
+            -0.014078537002205849,
+            0.023067140951752663,
+            0.003984406590461731,
+            0.050466138869524,
+            -0.01692131906747818,
+            0.00340231298469007,
+            -0.01360925193876028,
+            0.015712007880210876,
+            0.0010214835638180375,
+            0.0005175103433430195,
+            0.000724796496797353,
+            0.024872081354260445,
+            0.014502697624266148,
+            0.012291645631194115,
+            0.02424035221338272,
+            0.03005226142704487,
+            -0.013672425411641598,
+            0.0012894044630229473,
+            0.005112494342029095,
+            -0.020504126325249672,
+            0.0355934277176857,
+            -0.01125380489975214,
+            -0.04353516921401024,
+            0.03377043828368187,
+            0.05858837440609932,
+            -0.01819380186498165,
+            0.01096501387655735,
+            -0.01952945813536644,
+            0.000879344530403614,
+            -0.0008167356136254966,
+            -0.014917834661900997,
+            0.03696518391370773,
+            -0.028102925047278404,
+            -0.019998742267489433,
+            -0.010802569799125195,
+            0.016677651554346085,
+            -0.007828930392861366,
+            0.003255661576986313,
+            0.01038743369281292,
+            -0.006457175128161907,
+            0.027417048811912537,
+            -0.037614963948726654,
+            0.052595969289541245,
+            0.013573153875768185,
+            0.01576615683734417,
+            -0.030503496527671814,
+            0.026009194552898407,
+            0.013419733382761478,
+            0.00960228405892849,
+            -0.06082649901509285,
+            -0.0021546478383243084,
+            0.026514578610658646,
+            0.004909438546746969,
+            -0.015937626361846924,
+            -0.014782464131712914,
+            -0.01398828998208046,
+            0.012273596599698067,
+            -0.007657460402697325,
+            -0.000787969387602061,
+            0.02068462036550045,
+            0.018500640988349915,
+            0.023969611153006554,
+            0.00773868290707469,
+            -0.024619389325380325,
+            0.023049091920256615,
+            -0.0033887759782373905,
+            -0.010910865850746632,
+            0.005360673647373915,
+            -0.008514807559549809,
+            -0.005405797157436609,
+            -0.02044997736811638,
+            0.04905828461050987,
+            -0.00019558223721105605,
+            -0.04313807934522629,
+            -0.007720633409917355,
+            0.0005521990242414176,
+            0.009854976087808609,
+            -0.004986148327589035,
+            -0.004424360580742359,
+            -0.03593636676669121,
+            -0.009854976087808609,
+            0.06310072541236877,
+            -0.02582870051264763,
+            -0.02730875089764595,
+            -0.01687619462609291,
+            0.004620647989213467,
+            0.0031721831765025854,
+            -0.028157074004411697,
+            -0.018807481974363327,
+            -0.0032285875640809536,
+            -0.012057003565132618,
+            -0.011524545960128307,
+            -0.025341367349028587,
+            0.022724201902747154,
+            0.05815518647432327,
+            0.013879992999136448,
+            -0.038842324167490005,
+            0.015278822742402554,
+            -0.013022646307945251,
+            -0.02059437334537506,
+            -0.04526790976524353,
+            0.09941612929105759,
+            0.023807166144251823,
+            0.003623418277129531,
+            0.0062270453199744225,
+            -0.00327822333201766,
+            0.0001518688368378207,
+            0.018717234954237938,
+            0.04006968066096306,
+            0.07746805250644684,
+            0.04956366866827011,
+            -0.007409281097352505,
+            -0.0322723388671875,
+            0.003966357093304396,
+            -0.021226102486252785,
+            -0.019006025046110153,
+            -0.045556701719760895,
+            -0.0063308291137218475,
+            -0.00036803868715651333,
+            -0.016605453565716743,
+            0.01588347740471363,
+            0.033553846180438995,
+            0.01112745888531208,
+            -0.005292988382279873,
+            0.04429324343800545,
+            0.005144080612808466,
+            0.014250006526708603,
+            -0.0033210907131433487,
+            -0.011831386014819145,
+            -0.02012508735060692,
+            -0.014376352541148663,
+            -0.0018714978359639645,
+            -0.008961530402302742,
+            0.015296871773898602,
+            -0.013419733382761478,
+            0.05848007649183273,
+            0.006195458583533764,
+            -0.006764015182852745,
+            -0.05191009119153023,
+            -0.03360799327492714,
+            -0.010540853254497051,
+            0.021749533712863922,
+            -0.0235725250095129,
+            -0.012381892651319504,
+            0.02202027477324009,
+            -0.0120028555393219,
+            -0.020486075431108475,
+            0.03530463948845863,
+            0.007066342514008284,
+            -0.028915148228406906,
+            0.005202741362154484,
+            0.020215334370732307,
+            -0.011867485009133816,
+            -0.0019312864169478416,
+            0.02341008000075817,
+            0.003925745841115713,
+            -0.025233069434762,
+            -0.004307039547711611,
+            0.005947279278188944,
+            0.021984176710247993,
+            0.015405167825520039,
+            0.025178922340273857,
+            -0.05801079049706459,
+            -0.014430500566959381,
+            0.043823957443237305,
+            -0.04173022881150246,
+            -0.020702669396996498,
+            -0.045881591737270355,
+            -0.016127144917845726,
+            0.03407727926969528,
+            0.028915148228406906,
+            0.02738094888627529,
+            -0.04660356789827347,
+            -0.03783155605196953,
+            -0.031063027679920197,
+            -0.00024550012312829494,
+            -0.05306525528430939,
+            0.030178606510162354,
+            0.0006040910957381129,
+            -0.0036865912843495607,
+            -0.023049091920256615,
+            0.049527570605278015,
+            0.01749889925122261,
+            -0.005482506938278675,
+            0.004674796015024185,
+            -0.007652948144823313,
+            0.018058430403470993,
+            -0.006592545658349991,
+            0.010640124790370464,
+            -0.034384120255708694,
+            -0.02878880314528942,
+            0.03631540387868881,
+            0.00046590028796344995,
+            -0.018112579360604286,
+            0.040502868592739105,
+            0.005446408409625292,
+            0.015179550275206566,
+            0.02218271978199482,
+            -0.018347222357988358,
+            -0.03573782369494438,
+            0.041405338793992996,
+            -0.05086322873830795,
+            0.004361187573522329,
+            0.0067188916727900505,
+            0.000985948834568262,
+            -0.04006968066096306,
+            -0.00813576951622963,
+            0.019655803218483925,
+            0.010179865173995495,
+            -0.044401541352272034,
+            -0.015305896289646626,
+            -0.004252891521900892,
+            0.021785633638501167,
+            -0.014340253546833992,
+            0.01498100720345974,
+            -0.04772263020277023,
+            -0.030070310458540916,
+            0.0056900754570961,
+            -0.05819128453731537,
+            -0.021406596526503563,
+            0.029691273346543312,
+            0.008726888336241245,
+            -0.013961215503513813,
+            -0.005094444844871759,
+            -0.03469095751643181,
+            0.013816820457577705,
+            0.02664092369377613,
+            -0.02804877795279026,
+            -0.00885323341935873,
+            -0.010739396326243877,
+            0.01009864266961813,
+            0.017950134351849556,
+            -0.04003358259797096,
+            0.022146621719002724,
+            -0.021659286692738533,
+            0.01274288073182106,
+            -0.014502697624266148,
+            -0.02994396537542343,
+            -0.01675887405872345,
+            -0.01637081243097782,
+            -0.01224652212113142,
+            0.02010703831911087,
+            -0.006646693684160709,
+            0.0017428958090022206,
+            0.002045223256573081,
+            -0.0347631573677063,
+            -0.0073867193423211575,
+            -0.027435097843408585,
+            0.03667639195919037,
+            0.027001911774277687,
+            0.02954687736928463,
+            -0.0042754532769322395,
+            0.018157701939344406,
+            -0.02979956939816475,
+            -0.028500013053417206,
+            -0.008420048281550407,
+            -0.0028269882313907146,
+            0.020052891224622726,
+            0.0463508740067482,
+            0.021875880658626556,
+            -0.0029646148905158043,
+            0.011632842011749744,
+            0.006800113711506128,
+            0.004968099296092987,
+            0.025251120328903198,
+            -0.03467290848493576,
+            0.011867485009133816,
+            -0.0421995110809803,
+            0.04981636255979538,
+            0.037542764097452164,
+            0.04310198128223419,
+            0.04104435071349144,
+            -0.03171280771493912,
+            0.0012860201532021165,
+            -0.0017541766865178943,
+            -0.01046865526586771,
+            0.03180305287241936,
+            -0.008650178089737892,
+            0.008907382376492023,
+            0.034546565264463425,
+            0.00457101222127676,
+            -0.013663400895893574,
+            -0.011578693985939026,
+            -0.023969611153006554,
+            -0.021550990641117096,
+            -0.000988204963505268,
+            0.025846749544143677,
+            -0.018347222357988358,
+            0.012327744625508785,
+            -0.02416815422475338,
+            0.004546194337308407,
+            -0.0022155647166073322,
+            -0.005121518857777119,
+            -0.025467712432146072,
+            0.014376352541148663,
+            -0.02631603367626667,
+            -0.002851806115359068,
+            0.03534073755145073,
+            0.006633156910538673,
+            -0.0163437370210886,
+            -0.04526790976524353,
+            -0.005414821673184633,
+            0.0007496143807657063,
+            0.00883067212998867,
+            -0.008478708565235138,
+            0.03772325813770294,
+            -0.040827758610248566,
+            -0.023319832980632782,
+            0.039744794368743896,
+            -0.015468341298401356,
+            -0.004737969022244215,
+            0.007720633409917355,
+            -0.02211052179336548,
+            -0.038842324167490005,
+            -0.0004580036911647767,
+            0.05786639824509621,
+            0.034709006547927856,
+            -0.002942053135484457,
+            -0.022363213822245598,
+            0.0013954447349533439,
+            -0.03815644606947899,
+            -0.014944908209145069,
+            -0.0017113093053922057,
+            0.05107982084155083,
+            0.003921233583241701,
+            -0.009575209580361843,
+            0.0023464227560907602,
+            -0.003269198816269636,
+            -0.021966127678751945,
+            0.016397885978221893,
+            0.01187650952488184,
+            -0.005450920667499304,
+            -0.018347222357988358,
+            0.005008710082620382,
+            -0.008911894634366035,
+            0.009412765502929688,
+            -0.027579491958022118,
+            -0.016072995960712433,
+            0.017354503273963928,
+            0.025052575394511223,
+            0.03180305287241936,
+            0.02017923630774021,
+            0.018590888008475304,
+            0.011118434369564056,
+            0.045809391885995865,
+            -0.013419733382761478,
+            0.001081836293451488,
+            0.006033014040440321,
+            -0.01724620722234249,
+            -0.042740993201732635,
+            0.006412051618099213,
+            -0.07840662449598312,
+            -0.004950049798935652,
+            0.03180305287241936,
+            -0.022236868739128113,
+            0.017607195302844048,
+            -0.0008167356136254966,
+            0.02880685217678547,
+            0.016632527112960815,
+            -0.027778036892414093,
+            -0.004611623473465443,
+            -0.015134426765143871,
+            0.011010137386620045,
+            0.012174324132502079,
+            0.0008443737751804292,
+            0.021984176710247993,
+            0.00013847279478795826,
+            0.00972863007336855,
+            -0.009295444004237652,
+            0.012625560164451599,
+            0.005144080612808466,
+            0.011037211865186691,
+            -0.021226102486252785,
+            0.006109723821282387,
+            -0.00761233689263463,
+            -0.000742281845305115,
+            0.008451634086668491,
+            0.03624320775270462,
+            -0.03862572833895683,
+            0.0006170640699565411,
+            -0.03873402625322342,
+            -0.008189918473362923,
+            0.06941801309585571,
+            -0.03167670965194702,
+            -0.01505320519208908,
+            0.006610595155507326,
+            -0.00080432667164132,
+            0.003090960904955864,
+            0.002966871252283454,
+            0.027850233018398285,
+            -0.043499067425727844,
+            0.019060172140598297,
+            -0.015332970768213272,
+            -0.026279935613274574,
+            -0.02433059923350811,
+            0.021370496600866318,
+            0.03275967389345169,
+            0.00581190874800086,
+            0.00792820192873478,
+            0.0388784222304821,
+            0.01985434629023075,
+            -0.031135225668549538,
+            0.01411463599652052,
+            -0.014222932048141956,
+            -0.0028269882313907146,
+            0.014268055558204651,
+            0.015802254900336266,
+            -0.014674167148768902,
+            0.012047979049384594,
+            0.0005457125371322036,
+            0.0025878336746245623,
+            0.015035155229270458,
+            0.021695386618375778,
+            0.0051892041228711605,
+            0.012842152267694473,
+            -0.020251434296369553,
+            -0.006633156910538673,
+            0.019150419160723686,
+            -0.010360359214246273,
+            -0.03923941031098366,
+            -0.0016537768533453345,
+            0.048877790570259094,
+            -0.03481730446219444,
+            -0.019511409103870392,
+            0.004349906928837299,
+            0.032055746763944626,
+            -0.009647407568991184,
+            -0.03481730446219444,
+            -0.012201398611068726,
+            0.022200770676136017,
+            0.014439525082707405,
+            -0.016722774133086205,
+            -0.03552123159170151,
+            0.0044401539489626884,
+            -0.004094959236681461,
+            -0.007440867833793163,
+            -0.004056604113429785,
+            0.016776923090219498,
+            0.005617877934128046,
+            -0.0017801227513700724,
+            -0.011768212541937828,
+            -0.02564820647239685,
+            -0.04945537447929382,
+            0.02308518998324871,
+            -0.0063082673586905,
+            0.019475309178233147,
+            0.01059500128030777,
+            -0.003131571924313903,
+            -0.03840913623571396,
+            0.010053519159555435,
+            -0.002939797006547451,
+            0.004234842024743557,
+            0.042488303035497665,
+            -0.037037380039691925,
+            0.00377683830447495,
+            0.023049091920256615,
+            -0.016154218465089798,
+            -0.0248179342597723,
+            0.00680913869291544,
+            -0.01299557276070118,
+            0.002973639639094472,
+            0.03451046347618103,
+            -0.01604592241346836,
+            0.0260633435100317,
+            0.02035973034799099,
+            0.0007941738585941494,
+            0.05006905272603035,
+            0.00674596568569541,
+            0.006118748802691698,
+            -0.006290218327194452,
+            0.014719290658831596,
+            -0.0037226900458335876,
+            -0.014349278062582016,
+            0.0008415535558015108,
+            0.0310269296169281,
+            -0.013067769818007946,
+            -0.01514345221221447,
+            -0.006204483564943075,
+            0.03461876139044762,
+            -0.005960816517472267,
+            0.007720633409917355,
+            -0.03797595202922821,
+            -0.006064600311219692,
+            0.00780185591429472,
+            0.020756816491484642,
+            -0.00798686221241951,
+            0.012463115155696869,
+            0.0011811080621555448,
+            -0.010161816142499447,
+            0.010441581718623638,
+            0.033553846180438995,
+            -0.00027962480089627206,
+            0.012806054204702377,
+            -0.011641867458820343,
+            -0.017137911170721054,
+            -0.011046236380934715,
+            0.018518690019845963,
+            0.0612235851585865,
+            -0.03321090713143349,
+            0.020305581390857697,
+            0.010640124790370464,
+            0.0044446662068367004,
+            0.015179550275206566,
+            0.016975466161966324,
+            0.020666569471359253,
+            0.0019245179137215018,
+            -0.007337083574384451,
+            0.03400508314371109,
+            0.018735283985733986,
+            -0.014583920128643513,
+            -0.0007501784712076187,
+            0.0025946020614355803,
+            -0.05219888314604759,
+            0.0003074039414059371,
+            -0.02216467075049877,
+            0.0019211337203159928,
+            -0.000984820770099759,
+            -0.011632842011749744,
+            0.002883392618969083,
+            0.017688417807221413,
+            0.004706382751464844,
+            0.01162381749600172,
+            -0.029853718355298042,
+            -0.009322518482804298,
+            0.0017553047509863973,
+            -0.021460743620991707,
+            0.04313807934522629,
+            0.037290073931217194,
+            -0.04653136804699898,
+            -0.01795915886759758,
+            0.004638697486370802,
+            0.049274880439043045,
+            0.0011342923389747739,
+            0.002632957184687257,
+            -0.029492730274796486,
+            -0.010567927733063698,
+            0.015278822742402554,
+            0.006705354433506727,
+            0.0028630869928747416,
+            0.017950134351849556,
+            -0.013455832377076149,
+            0.018040381371974945,
+            -0.005893131252378225,
+            0.008808109909296036,
+            -0.015666885301470757,
+            0.029727371409535408,
+            -0.011118434369564056,
+            0.000694338115863502,
+            0.01662350259721279,
+            0.018572838976979256,
+            -0.031171325594186783,
+            0.013356560841202736,
+            0.03176695480942726,
+            0.013870968483388424,
+            -0.018184777349233627,
+            -0.015594687312841415,
+            -0.044473737478256226,
+            0.026334084570407867,
+            0.007648435886949301,
+            0.0016470083501189947,
+            -0.00259685842320323,
+            -0.020161187276244164,
+            -0.018275024369359016,
+            -0.027038009837269783,
+            -0.01567590981721878,
+            -0.010694272816181183,
+            0.005139568354934454,
+            0.00446045957505703,
+            0.003928001970052719,
+            0.017814764752984047,
+            0.007034756243228912,
+            -0.02059437334537506,
+            -0.016100069507956505,
+            -0.014710266143083572,
+            0.0045304009690880775,
+            -0.024384748190641403,
+            -0.048047520220279694,
+            -0.03452851250767708,
+            -0.015450291335582733,
+            0.020504126325249672,
+            0.030918633565306664,
+            -0.006353390868753195,
+            0.011262829415500164,
+            -0.010730371810495853,
+            0.01472831517457962,
+            -0.0003917285066563636,
+            -0.0055772666819393635,
+            0.003946051467210054,
+            -0.012896301224827766,
+            0.03270552307367325,
+            0.01022498868405819,
+            0.021893929690122604,
+            0.010071569122374058,
+            -0.004316064063459635,
+            0.029402483254671097,
+            0.013636326417326927,
+            -0.013248263858258724,
+            0.0024005710147321224,
+            0.013645350933074951,
+            -0.006533884909003973,
+            0.048191916197538376,
+            -0.0014304154319688678,
+            -0.019078223034739494,
+            0.009313493967056274,
+            0.007562701124697924,
+            -0.0363876037299633,
+            0.05389552563428879,
+            -0.01178626250475645,
+            -0.0281209759414196,
+            0.010802569799125195,
+            0.01763427071273327,
+            -0.02581065148115158,
+            -0.006023989524692297,
+            0.015170525759458542,
+            -0.021388545632362366,
+            -0.0033662142232060432,
+            0.019258717074990273,
+            0.03617101162672043,
+            0.014944908209145069,
+            0.0013785234186798334,
+            -0.026677021756768227,
+            -0.017020590603351593,
+            0.023861315101385117,
+            -0.022200770676136017,
+            0.02516087330877781,
+            0.0013401684118434787,
+            -0.002479537157341838,
+            -0.02664092369377613,
+            0.009845951572060585,
+            -0.014565871097147465,
+            0.015666885301470757,
+            -0.025142822414636612,
+            0.015044179745018482,
+            0.008988603949546814,
+            0.00914202444255352,
+            0.015360044315457344,
+            -0.05277646332979202,
+            -0.03174890577793121,
+            -0.030900582671165466,
+            0.014746365137398243,
+            0.028355617076158524,
+            -0.03093668259680271,
+            -0.016569355502724648,
+            0.012887275777757168,
+            0.006452662870287895,
+            0.015278822742402554,
+            -0.012273596599698067,
+            4.801424074685201e-05,
+            0.02342812903225422,
+            -0.03270552307367325,
+            -0.013762672431766987,
+            0.0297815203666687,
+            0.006890360731631517,
+            -0.012562386691570282,
+            0.040936052799224854,
+            0.007571726106107235,
+            -0.03467290848493576,
+            -0.0036008565220981836,
+            -0.01303167175501585,
+            -0.0023396543692797422,
+            -0.005942767020314932,
+            0.005256889387965202,
+            0.008528344333171844,
+            0.012174324132502079,
+            0.0060736252926290035,
+            -0.023482277989387512,
+            0.02781413495540619,
+            0.005974353291094303,
+            0.014629043638706207,
+            0.01812160387635231,
+            -0.014638068154454231,
+            0.007513065356761217,
+            0.005653976462781429,
+            0.0037881191819906235,
+            0.013248263858258724,
+            0.0013954447349533439,
+            0.05891326069831848,
+            -0.014015363529324532,
+            0.005536655429750681,
+            0.0513325110077858,
+            0.011795287020504475,
+            -0.002567528048530221,
+            0.020468026399612427,
+            0.023789117112755775,
+            -0.013167042285203934,
+            -0.01766134425997734,
+            -0.011686990968883038,
+            -0.0031744393054395914,
+            -0.011389175429940224,
+            0.03146011382341385,
+            0.002407339634373784,
+            0.02158709056675434,
+            -0.03153231367468834,
+            0.03568367660045624,
+            0.006425588857382536,
+            -0.011307952925562859,
+            0.0132392393425107,
+            -0.0244208462536335,
+            -0.019493358209729195,
+            -0.006646693684160709,
+            -0.02472768723964691,
+            -0.006159360054880381,
+            -0.007106953766196966,
+            0.0310269296169281,
+            0.025918947532773018,
+            0.008537368848919868,
+            0.005667513702064753,
+            0.003379751229658723,
+            -0.0239876601845026,
+            -0.013230214826762676,
+            0.005568241700530052,
+            0.011064286343753338,
+            0.03451046347618103,
+            0.021334398537874222,
+            -0.0032669424545019865,
+            -0.006136798299849033,
+            -0.007806368172168732,
+            -0.011524545960128307,
+            0.017002541571855545,
+            -0.02887905016541481,
+            -0.03261527791619301,
+            -0.014078537002205849,
+            -0.003988918848335743,
+            -0.0025788089260458946,
+            0.00024634620058350265,
+            0.021226102486252785,
+            0.004410823807120323,
+            0.021532941609621048,
+            0.0016210622852668166,
+            0.012093102559447289,
+            -0.02664092369377613,
+            -0.007355133071541786,
+            0.031893301755189896,
+            0.0021501355804502964,
+            -0.014313179068267345,
+            0.013879992999136448,
+            -0.012860202230513096,
+            -0.012842152267694473,
+            -0.016749849542975426,
+            0.029925916343927383,
+            0.0033910321071743965,
+            -0.03423972427845001,
+            -0.010712322778999805,
+            -0.023698870092630386,
+            0.0020170211791992188,
+            -0.004401798825711012,
+            0.02615359053015709,
+            -0.03855353221297264,
+            0.0010874767322093248,
+            0.014881735667586327,
+            -0.005667513702064753,
+            -0.010306211188435555,
+            0.00447174021974206,
+            0.004006968345493078,
+            -0.017923060804605484,
+            -0.010495729744434357,
+            -0.012950449250638485,
+            -0.020468026399612427,
+            -0.009132999926805496,
+            -0.010243037715554237,
+            0.013690474443137646,
+            0.010784519836306572,
+            -0.008266627788543701,
+            0.008893844671547413,
+            -0.002168184844776988,
+            -0.02622578665614128,
+            -0.020088989287614822,
+            -0.022886646911501884,
+            0.03552123159170151,
+            -0.0030413249041885138,
+            -0.03837303817272186,
+            -0.0033323715906590223,
+            -0.0018940595909953117,
+            0.0011529058683663607,
+            0.0067279161885380745,
+            -0.01352803036570549,
+            -0.0182118508964777,
+            0.013753647916018963,
+            0.007237812038511038,
+            -0.013546079397201538,
+            0.011650891974568367,
+            0.026514578610658646,
+            0.017787689343094826,
+            0.02341008000075817,
+            0.011226730421185493,
+            0.037614963948726654,
+            0.017300356179475784,
+            -0.002073425566777587,
+            0.043065883219242096,
+            -0.001112294616177678,
+            -0.01844649389386177,
+            0.046314775943756104,
+            0.00023196307301986963,
+            -0.0017214621184393764,
+            0.027579491958022118,
+            -0.00013339638826437294,
+            0.0037046405486762524,
+            0.011668941006064415,
+            -0.027272652834653854,
+            0.005117006599903107,
+            0.004706382751464844,
+            0.03655004873871803,
+            0.005717149470001459,
+            -0.017760615795850754,
+            -0.022381264716386795,
+            0.012643609195947647,
+            0.018663085997104645,
+            -0.0011258317390456796,
+            0.014006339013576508,
+            0.006786576937884092,
+            -0.03920331224799156,
+            0.015784205868840218,
+            -0.0069760954938828945,
+            0.010622075758874416,
+            0.005762272980064154,
+            0.005153105594217777,
+            -0.016172267496585846,
+            0.0007783806649968028,
+            0.009575209580361843,
+            0.03815644606947899,
+            -0.013094844296574593,
+            0.03331920504570007,
+            -0.032308436930179596,
+            -0.009818877093493938,
+            -0.025846749544143677,
+            0.00782441720366478,
+            0.011542594991624355,
+            0.007355133071541786,
+            -0.042416103184223175,
+            0.005103469826281071,
+            -0.011479422450065613,
+            0.0010835283901542425,
+            -0.002407339634373784,
+            0.008203455246984959,
+            0.04097215086221695,
+            -0.02299494296312332,
+            0.025955045595765114,
+            -0.007350620813667774,
+            0.01837429590523243,
+            -0.0002003766130656004,
+            -0.009191660210490227,
+            -0.019728001207113266,
+            -0.004864315036684275,
+            0.003530915128067136,
+            -0.000880472594872117,
+            -0.007567213382571936,
+            -3.902479147654958e-05,
+            -0.005035784561187029,
+            0.0045394254848361015,
+            -0.006520348135381937,
+            0.01597372442483902,
+            -0.01729133166372776,
+            0.04227171093225479,
+            0.001362730166874826,
+            -0.011190632358193398,
+            -0.0151073532178998,
+            -0.03402313217520714,
+            0.0027705838438123465,
+            -0.006655718665570021,
+            0.02696581371128559,
+            0.014674167148768902,
+            0.020919261500239372,
+            -0.0008658074657432735,
+            0.008785548619925976,
+            0.020648520439863205,
+            -0.017850862815976143,
+            0.024944279342889786,
+            -0.01807648129761219,
+            0.002287762239575386,
+            -0.0040430668741464615,
+            0.016533255577087402,
+            0.01869918406009674,
+            0.01398828998208046,
+            0.0020847064442932606,
+            -0.008591517806053162,
+            -0.032001595944166183,
+            -0.008095158264040947,
+            -0.005893131252378225,
+            0.014583920128643513,
+            0.010279136709868908,
+            0.019637754186987877,
+            0.0015691702719777822,
+            0.010360359214246273,
+            0.00032376122544519603,
+            0.004873339552432299,
+            0.02456524223089218,
+            0.006777551956474781,
+            -0.03978089243173599,
+            0.0005428923177532852,
+            0.001857960713095963,
+            0.009182635694742203,
+            -0.021623188629746437,
+            0.014908809214830399,
+            -0.00537872314453125,
+            0.028752705082297325,
+            -0.007007681764662266,
+            0.02084706351161003,
+            -0.006136798299849033,
+            -0.004661259241402149,
+            -0.003783606691285968,
+            0.006524860393255949,
+            0.030846435576677322,
+            -0.00328047969378531,
+            0.024854032322764397,
+            0.023103240877389908,
+            0.0074724541045725346,
+            0.007851491682231426,
+            -0.017923060804605484,
+            0.0030774236656725407,
+            0.022904695942997932,
+            -0.0017146936152130365,
+            0.007887590676546097,
+            -0.004877852275967598,
+            0.006303755100816488,
+            -0.016677651554346085,
+            0.01627153903245926,
+            0.01452074758708477,
+            -0.0058209337294101715,
+            0.02781413495540619,
+            0.015567612834274769,
+            0.009737654589116573,
+            -0.002993945265188813,
+            0.004408567678183317,
+            0.014538796618580818,
+            0.00432283291593194,
+            0.004140082746744156,
+            -0.015657860785722733,
+            0.013573153875768185,
+            0.010802569799125195,
+            0.013130943290889263,
+            -0.03687493875622749,
+            -0.016036897897720337,
+            -0.02326568402349949,
+            -0.0019121089717373252,
+            0.0194572601467371,
+            0.00011612254456849769,
+            0.001237512449733913,
+            0.025612108409404755,
+            -0.0003043017059098929,
+            -0.008162843994796276,
+            -0.007075367495417595,
+            -0.020973410457372665,
+            0.0017135655507445335,
+            -0.0001287430350203067,
+            -0.04270489513874054,
+            0.007296472322195768,
+            0.007025731261819601,
+            0.001146137248724699,
+            -0.007910151965916157,
+            -0.02572040446102619,
+            -0.01134405191987753,
+            0.08793670684099197,
+            -0.012345793657004833,
+            -0.02283249981701374,
+            -0.008591517806053162,
+            -0.004950049798935652,
+            -0.008659202605485916,
+            -0.003582807257771492,
+            0.021370496600866318,
+            0.01860893703997135,
+            0.0015624017687514424,
+            0.015179550275206566,
+            -0.008108695968985558,
+            0.02936638332903385,
+            0.01208407711237669,
+            0.019998742267489433,
+            -0.0028044264763593674,
+            0.007856003940105438,
+            -0.0044401539489626884,
+            0.013546079397201538,
+            -0.022128572687506676,
+            -0.0011641867458820343,
+            -0.0031383405439555645,
+            0.00407465361058712,
+            -0.02202027477324009,
+            0.007883078418672085,
+            -0.011515521444380283,
+            -0.013302412815392017,
+            -0.0032015135511755943,
+            -0.030467398464679718,
+            -0.013320461846888065,
+            -0.002380265388637781,
+            0.039095014333724976,
+            0.009620334021747112,
+            0.01804940588772297,
+            0.0012442809529602528,
+            0.02267005480825901,
+            0.0072197625413537025,
+            0.0017428958090022206,
+            0.014096586033701897,
+            0.009137512184679508,
+            -0.00923227146267891,
+            0.04624257981777191,
+            0.015405167825520039,
+            -0.0012262315722182393,
+            -0.03277772292494774,
+            0.003679822664707899,
+            0.016424959525465965,
+            0.009295444004237652,
+            -0.0028021703474223614,
+            -0.0017451519379392266,
+            0.00537872314453125,
+            -0.0046793087385594845,
+            0.017399627715349197,
+            0.015008081682026386,
+            -0.02227296680212021,
+            0.017562072724103928,
+            -0.0018331428291276097,
+            0.02259785681962967,
+            0.0006396258249878883,
+            -0.00021673388255294412,
+            0.02837366610765457,
+            -0.0018444237066432834,
+            0.024150105193257332,
+            0.00885323341935873,
+            -0.002257303800433874,
+            0.0033684703521430492,
+            0.017300356179475784,
+            0.004404054954648018,
+            0.006655718665570021,
+            -0.004176181275397539,
+            -0.017968183383345604,
+            0.023067140951752663,
+            0.01650618202984333,
+            0.002729972591623664,
+            -0.01763427071273327,
+            -0.006786576937884092,
+            0.009182635694742203,
+            0.0013807795476168394,
+            0.006651206407696009,
+            -0.005459945183247328,
+            0.01026108767837286,
+            0.014908809214830399,
+            -0.006939996499568224,
+            0.03241673484444618,
+            -0.010992088355123997,
+            -0.02299494296312332,
+            -0.011885534040629864,
+            0.023121289908885956,
+            0.01943921111524105,
+            -0.004207768011838198,
+            -0.015134426765143871,
+            0.031315721571445465,
+            -0.011533570475876331,
+            -0.0036143935285508633,
+            -0.030900582671165466,
+            -0.036189060658216476,
+            0.04501521959900856,
+            -0.0004785912751685828,
+            -0.0446903295814991,
+            0.027453146874904633,
+            0.01692131906747818,
+            0.010694272816181183,
+            0.010243037715554237,
+            -0.022128572687506676,
+            0.015657860785722733,
+            -0.0049274880439043045,
+            0.0045349132269620895,
+            -0.008244066499173641,
+            0.006664743181318045,
+            -0.011010137386620045,
+            0.027669738978147507,
+            -0.01221042312681675,
+            -0.002286634175106883,
+            -0.0010615306673571467,
+            0.00028300905250944197,
+            0.005870569497346878,
+            -0.005062858574092388,
+            -0.014656118117272854,
+            0.011930657550692558,
+            0.01389804296195507,
+            -0.0018703697714954615,
+            0.0179862342774868,
+            0.020648520439863205,
+            -0.03581002354621887,
+            -0.0014416963094845414,
+            0.0012826359597966075,
+            -0.0005392259918153286,
+            0.008826159872114658,
+            0.012075052596628666,
+            0.023211536929011345,
+            -0.0138619439676404,
+            -0.0368388369679451,
+            0.008762986399233341,
+            0.023644722998142242,
+            -0.0022990431170910597,
+            -0.023446178063750267,
+            -0.0020858345087617636,
+            0.00829821452498436,
+            0.020540224388241768,
+            -0.0012905325274914503,
+            0.008059060201048851,
+            0.0077702696435153484,
+            -0.017210109159350395,
+            -0.009999371133744717,
+            -0.005766785237938166,
+            -0.015332970768213272,
+            -0.027778036892414093,
+            0.010775495320558548,
+            0.005978866014629602,
+            0.01911432109773159,
+            -0.022868597880005836,
+            -0.021370496600866318,
+            -0.021911978721618652,
+            0.0009171354467980564,
+            0.006926459725946188,
+            -0.020973410457372665,
+            0.0014687704388052225,
+            -0.010766470804810524,
+            0.025792602449655533,
+            -0.013636326417326927,
+            0.0014394401805475354,
+            0.0020486076828092337,
+            0.028319519013166428,
+            0.002041839063167572,
+            0.008401998318731785,
+            -0.020504126325249672,
+            -0.0035106095019727945,
+            0.07266691327095032,
+            -0.013943166472017765,
+            -0.0045258887112140656,
+            -0.01175016351044178,
+            0.017020590603351593,
+            -0.01009864266961813,
+            0.0036707979161292315,
+            0.04772263020277023,
+            -0.014403426088392735,
+            0.004473996814340353,
+            0.021911978721618652,
+            0.008415536023676395,
+            0.000976360053755343,
+            -0.00967448204755783,
+            -0.012932399287819862,
+            -0.0023464227560907602,
+            -0.017697442322969437,
+            0.02301299385726452,
+            -0.021081706508994102,
+            0.027074109762907028,
+            -0.015784205868840218,
+            -0.034131426364183426,
+            0.01712888665497303,
+            0.029240038245916367,
+            -0.015594687312841415,
+            -0.009909124113619328,
+            -0.000990461208857596,
+            0.010983063839375973,
+            0.035629529505968094,
+            0.014448549598455429,
+            0.01887967810034752,
+            0.005320062395185232,
+            -0.003212794428691268,
+            0.0018410393968224525,
+            0.02012508735060692,
+            0.015892501920461655,
+            -0.0013345279730856419,
+            0.0064707123674452305,
+            -0.003801656188443303,
+            0.02548576146364212,
+            -0.014845636673271656,
+            -0.02846391312777996,
+            -0.00235995976254344,
+            -0.026370182633399963,
+            -0.007576238363981247,
+            0.029438581317663193,
+            -0.009448864497244358,
+            0.012445065192878246,
+            0.025431614369153976,
+            -0.019475309178233147,
+            -0.027326801791787148,
+            -0.02117195352911949,
+            0.01729133166372776,
+            -0.005915693007409573,
+            0.005288476124405861,
+            0.0017293586861342192,
+            0.015892501920461655,
+            -0.016118120402097702,
+            -0.01514345221221447,
+            0.0020294301211833954,
+            -0.020829014480113983,
+            0.024150105193257332,
+            0.018365271389484406,
+            -0.004354419186711311,
+            0.008492245338857174,
+            0.019728001207113266,
+            -0.010549877770245075,
+            0.0012713550822809339,
+            -0.017905011773109436,
+            -0.004877852275967598,
+            -0.0004585677233990282,
+            0.009936198592185974,
+            -0.02458329126238823,
+            -0.007707096636295319,
+            -0.004516863729804754,
+            -0.00182637432590127,
+            0.01522467378526926,
+            -0.015621760860085487,
+            -0.0004402362974360585,
+            0.008550906553864479,
+            0.0022144364193081856,
+            -0.017065713182091713,
+            -0.0015274309553205967,
+            0.00039313864544965327,
+            -0.011154533363878727,
+            0.0017169497441500425,
+            -0.010279136709868908,
+            -0.023879364132881165,
+            -0.004006968345493078,
+            -0.01141624990850687,
+            -0.0045326570980250835,
+            0.005789346992969513,
+            -0.00901567842811346,
+            0.0007039268384687603,
+            0.020666569471359253,
+            0.016316663473844528,
+            0.009304468519985676,
+            -0.029149791225790977,
+            0.003840011078864336,
+            -0.0037001282908022404,
+            -0.0027412534691393375,
+            -0.01692131906747818,
+            0.0030210192780941725,
+            -0.023536425083875656,
+            0.005089932586997747,
+            -0.018275024369359016,
+            0.014746365137398243,
+            0.005802884232252836,
+            -0.0032579179387539625,
+            -0.0006774167995899916,
+            -0.0002590371877886355,
+            -0.0010288162156939507,
+            0.0007428458775393665,
+            0.01675887405872345,
+            0.050430040806531906,
+            0.0032082819379866123,
+            -0.034456316381692886,
+            -0.004128801636397839,
+            0.006813650950789452,
+            0.0019380550365895033,
+            0.0089931171387434,
+            0.012634584680199623,
+            0.04021407663822174,
+            -0.03732617199420929,
+            0.01564883440732956,
+            -0.021298298612236977,
+            0.005396772641688585,
+            -0.014177808538079262,
+            0.0049229757860302925,
+            -0.025088675320148468,
+            0.01009864266961813,
+            0.012237497605383396,
+            -0.0019910752307623625,
+            -0.005247864872217178,
+            -0.015576637350022793,
+            0.016975466161966324,
+            0.010937940329313278,
+            -0.019998742267489433,
+            -0.009218734689056873,
+            0.001989946933463216,
+            -0.03416752442717552,
+            -0.01618129201233387,
+            -0.005198229104280472,
+            -0.009710581041872501,
+            0.0042867339216172695,
+            -0.00875847414135933,
+            0.012327744625508785,
+            0.013852919451892376,
+            -0.008031985722482204,
+            -0.002881136490032077,
+            0.014683191664516926,
+            -0.003145109163597226,
+            0.0037204339168965816,
+            0.00328047969378531,
+            0.021803682669997215,
+            -0.03490755334496498,
+            0.019132370129227638,
+            -0.003659517038613558,
+            -0.0015939881559461355,
+            -0.008677251636981964,
+            -0.0006633157026953995,
+            -0.022796399891376495,
+            -0.007711608894169331,
+            0.004473996814340353,
+            -0.005234327632933855,
+            0.0025539910420775414,
+            -0.0190240740776062,
+            0.021893929690122604,
+            0.010567927733063698,
+            0.008388461545109749,
+            -0.021081706508994102,
+            -0.005302012898027897,
+            -0.014096586033701897,
+            -0.01786891184747219,
+            0.007549164351075888,
+            -0.013744622468948364,
+            0.0009086747886613011,
+            0.004415336064994335,
+            0.016406910493969917,
+            0.0015285591362044215,
+            0.003025531768798828,
+            0.009069826453924179,
+            -0.015757132321596146,
+            -0.03317480906844139,
+            0.008478708565235138,
+            -0.0101347416639328,
+            0.026695072650909424,
+            0.0256662555038929,
+            0.015576637350022793,
+            0.009493988007307053,
+            0.006430101115256548,
+            -0.00327822333201766,
+            -0.03535878658294678,
+            -0.023283734917640686,
+            0.011921633034944534,
+            -0.020468026399612427,
+            0.017598170787096024,
+            0.004169412888586521,
+            -0.003429387230426073,
+            0.012318720109760761,
+            0.01671374961733818,
+            0.028193172067403793,
+            0.025359416380524635,
+            -0.01657838001847267,
+            0.01687619462609291,
+            -0.019565556198358536,
+            -0.006366928108036518,
+            0.02862635813653469,
+            -0.0040362984873354435,
+            -0.011172582395374775,
+            0.007643923629075289,
+            -0.0095120370388031,
+            0.01918651908636093,
+            0.000988204963505268,
+            -0.010495729744434357,
+            -0.01038743369281292,
+            0.0036346991546452045,
+            -0.016397885978221893,
+            -0.01911432109773159,
+            0.01554956380277872,
+            0.027525344863533974,
+            0.01369949895888567,
+            -0.020305581390857697,
+            0.008663714863359928,
+            0.002093731192871928,
+            0.024962328374385834,
+            0.005667513702064753,
+            0.03718177601695061,
+            -0.0056765382178127766,
+            0.00643461337313056,
+            -0.016452033072710037,
+            -0.005392259918153286,
+            0.01766134425997734,
+            -0.003770069684833288,
+            -0.007467941846698523,
+            -0.007851491682231426,
+            0.0030165070202201605,
+            0.016289589926600456,
+            0.01224652212113142,
+            0.0024321575183421373,
+            -0.01729133166372776,
+            -0.0031044979114085436,
+            -0.010035470128059387,
+            -0.0036031126510351896,
+            -0.0010271240025758743,
+            -0.018681135028600693,
+            -0.0007941738585941494,
+            0.012165299616754055,
+            -0.04537620767951012,
+            -0.001970769604668021,
+            -0.015910550951957703,
+            -0.022056374698877335,
+            -0.024132056161761284,
+            0.023771068081259727,
+            0.009620334021747112,
+            0.013645350933074951,
+            0.007319034077227116,
+            0.01604592241346836,
+            -0.041008252650499344,
+            0.01960165612399578,
+            0.01017084065824747,
+            -0.005644951947033405,
+            0.0022730971686542034,
+            -0.02705606073141098,
+            0.006254119332879782,
+            0.008216992020606995,
+            -0.003864829195663333,
+            0.01551346480846405,
+            -0.026911664754152298,
+            -0.002569784177467227,
+            -0.011885534040629864,
+            0.011831386014819145,
+            0.027344850823283195,
+            -0.006145822815597057,
+            3.934206688427366e-05,
+            -0.015161501243710518,
+            0.005207253620028496,
+            0.002380265388637781,
+            -0.01837429590523243,
+            0.006637669168412685,
+            -0.014204883016645908,
+            -0.0036031126510351896,
+            0.00458229286596179,
+            -0.0012916605919599533,
+            -0.01683107204735279,
+            0.0019301583524793386,
+            -0.0011935170041397214,
+            -0.004171669017523527,
+            -0.005103469826281071,
+            -0.0022674566134810448,
+            0.012643609195947647,
+            -0.008925431407988071,
+            -0.008623103611171246,
+            -0.0035444521345198154,
+            -0.022363213822245598,
+            -0.0011991574428975582,
+            -0.00114275305531919,
+            -0.007070854771882296,
+            0.00494553754106164,
+            0.002865343354642391,
+            -0.004661259241402149,
+            0.01597372442483902,
+            -0.008203455246984959,
+            -0.00552763044834137,
+            0.008230529725551605,
+            -0.02259785681962967,
+            0.005685563199222088,
+            0.012390917167067528,
+            -0.04089995473623276,
+            0.0030480935238301754,
+            -0.004550706595182419,
+            0.028427815064787865,
+            -0.011425274424254894,
+            0.0042799655348062515,
+            0.021460743620991707,
+            -0.009011166170239449,
+            -0.026171639561653137,
+            0.00989107508212328,
+            0.053570639342069626,
+            0.004886876791715622,
+            -0.0008945736917667091,
+            -0.003650492522865534,
+            0.00023027094721328467,
+            -0.011975781060755253,
+            -0.0064797368831932545,
+            -0.014159759506583214,
+            0.016984490677714348,
+            0.005112494342029095,
+            -0.015784205868840218,
+            -0.020052891224622726,
+            -0.015558588318526745,
+            -0.013148992322385311,
+            0.023608623072504997,
+            -0.010585976764559746,
+            -0.01579323038458824,
+            0.004232585895806551,
+            0.02259785681962967,
+            0.010306211188435555,
+            -0.00575324846431613,
+            -0.02109975554049015,
+            -0.019096272066235542,
+            0.018067454919219017,
+            0.006303755100816488,
+            0.014331229031085968,
+            0.0024908180348575115,
+            -0.013708524405956268,
+            0.014177808538079262,
+            0.003025531768798828,
+            0.021081706508994102,
+            -0.0005378159112296999,
+            -0.007052805740386248,
+            0.005013222806155682,
+            -0.02456524223089218,
+            0.020666569471359253,
+            -0.008695301599800587,
+            0.013148992322385311,
+            -0.020648520439863205,
+            -0.008623103611171246,
+            0.004480765201151371,
+            -0.0030864484142512083,
+            -0.012634584680199623,
+            0.021496843546628952,
+            0.0024028271436691284,
+            -0.008271140046417713,
+            -0.0032895042095333338,
+            0.0008240682072937489,
+            -0.027778036892414093,
+            -0.001954976236447692,
+            0.006394002120941877,
+            -0.02465548925101757,
+            0.015405167825520039,
+            0.0015533770201727748,
+            -0.014240982010960579,
+            0.004015992861241102,
+            -0.00792820192873478,
+            -0.005274938885122538,
+            0.0003812937065958977,
+            0.013654375448822975,
+            -0.029907865449786186,
+            0.015982748940587044,
+            -0.0031766954343765974,
+            -0.0071295155212283134,
+            0.006935484241694212,
+            0.0018365271389484406,
+            0.00864566583186388,
+            -0.008027473464608192,
+            1.942426388268359e-05,
+            0.017778664827346802,
+            0.0063263168558478355,
+            0.008568955585360527,
+            0.00513054383918643,
+            -0.0031766954343765974,
+            -0.003219562815502286,
+            0.0006644437671639025,
+            0.002129829954355955,
+            -0.020810965448617935,
+            -0.005911180749535561,
+            -0.003986662719398737,
+            0.0014089817414060235,
+            0.01026108767837286,
+            -0.003939283080399036,
+            0.0013593458570539951,
+            -0.02573845349252224,
+            -0.004268684424459934,
+            0.005911180749535561,
+            0.0030187631491571665,
+            0.007580750621855259,
+            0.03028690442442894,
+            0.015396143309772015,
+            0.019258717074990273,
+            0.0020215334370732307,
+            -0.0015240467619150877,
+            -0.011948706582188606,
+            -0.0011399327777326107,
+            0.00792820192873478,
+            -0.00844712182879448,
+            0.009827901609241962,
+            -0.022814448922872543,
+            0.01072134729474783,
+            0.005261402111500502,
+            0.003183464054018259,
+            -0.000386370113119483,
+            -0.014096586033701897,
+            0.012129200622439384,
+            0.021821731701493263,
+            -0.021496843546628952,
+            0.001207054010592401,
+            -0.013780721463263035,
+            -0.00023619340208824724,
+            -0.011569669470191002,
+            -0.03656809777021408,
+            -0.008144794963300228,
+            -0.030665941536426544,
+            -0.00015750927559565753,
+            -0.014033413492143154,
+            -0.0030119947623461485,
+            -0.01079354528337717,
+            -0.010694272816181183,
+            0.001760945189744234,
+            -0.009791802614927292,
+            0.009692531079053879,
+            0.010414507240056992,
+            0.004737969022244215,
+            -0.0007857132004573941,
+            0.011235755868256092,
+            -0.009367641992866993,
+            -0.014872711151838303,
+            -0.010414507240056992,
+            -0.002763815224170685,
+            0.024547193199396133,
+            -0.010766470804810524,
+            -0.014845636673271656,
+            -0.00612326106056571,
+            -0.006430101115256548,
+            -0.011605768464505672,
+            -0.01837429590523243,
+            0.014195858500897884,
+            0.02754339389503002,
+            -0.013690474443137646,
+            0.0023667283821851015,
+            -0.010847693309187889,
+            0.013266313821077347,
+            0.023969611153006554,
+            -0.001973025733605027,
+            -0.0289692971855402,
+            -0.018717234954237938,
+            0.0012521775206550956,
+            -0.011425274424254894,
+            -0.0072287870571017265,
+            -0.01432220358401537,
+            -0.004706382751464844,
+            -0.015206624753773212,
+            0.017769640311598778,
+            -0.00926837045699358,
+            -0.02028753235936165,
+            -0.008456147275865078,
+            -0.0030165070202201605,
+            -0.00600142776966095,
+            -0.008690789341926575,
+            -0.022525658831000328,
+            0.011434298940002918,
+            0.008253091014921665,
+            0.009593259543180466,
+            -0.01567590981721878,
+            0.019673852249979973,
+            0.013383634388446808,
+            -0.024799883365631104,
+            -0.003431643359363079,
+            -0.01464709360152483,
+            -0.0020666569471359253,
+            -0.013843894936144352,
+            -0.011271853931248188,
+            0.0006982863997109234,
+            0.0014394401805475354,
+            0.010802569799125195,
+            0.0024231327697634697,
+            0.022633954882621765,
+            -4.344900844444055e-06,
+            -0.013627301901578903,
+            0.00643461337313056,
+            0.01522467378526926,
+            -0.014538796618580818,
+            0.02806682698428631,
+            0.012914350256323814,
+            -0.017670368775725365,
+            0.015937626361846924,
+            0.013049720786511898,
+            -0.015964699909090996,
+            0.005762272980064154,
+            0.004977123811841011,
+            -0.00854188110679388,
+            -0.01361827738583088,
+            0.018320146948099136,
+            0.01175016351044178,
+            0.021532941609621048,
+            0.001636855537071824,
+            0.012508238665759563,
+            -0.00686779897660017,
+            0.007409281097352505,
+            -0.022200770676136017,
+            0.01985434629023075,
+            -0.00963838305324316,
+            -0.02010703831911087,
+            0.014926859177649021,
+            -0.0026103954296559095,
+            0.012896301224827766,
+            0.011262829415500164,
+            -0.004832728765904903,
+            -0.05097152292728424,
+            -0.01038743369281292,
+            0.011470397934317589,
+            0.003955075982958078,
+            -0.014990031719207764,
+            0.019565556198358536,
+            -0.0036911035422235727,
+            0.030268853530287743,
+            -0.005049321334809065,
+            0.02631603367626667,
+            -0.005256889387965202,
+            0.018735283985733986,
+            -0.015125402249395847,
+            -0.016993515193462372,
+            0.0071295155212283134,
+            0.00026777986204251647,
+            -0.004674796015024185,
+            0.008307239040732384,
+            -0.030918633565306664,
+            -0.010531828738749027,
+            -0.007147565018385649,
+            0.01886162906885147,
+            -0.006177409086376429,
+            0.026604825630784035,
+            0.0019233898492529988,
+            0.009466913528740406,
+            0.017580121755599976,
+            -0.01352803036570549,
+            0.019565556198358536,
+            -0.05479799583554268,
+            -0.0020316862501204014,
+            -0.0044333855621516705,
+            0.027669738978147507,
+            -0.004205511882901192,
+            -0.0012668427079916,
+            0.004088190384209156,
+            3.040549358956923e-07,
+            -0.005017735064029694,
+            0.007661973126232624,
+            -3.456531703704968e-05,
+            0.01361827738583088,
+            -0.027272652834653854,
+            -0.02739899791777134,
+            0.00661961967125535,
+            0.004828216042369604,
+            0.016975466161966324,
+            -0.010306211188435555,
+            -0.001425903057679534,
+            0.017850862815976143,
+            -0.01439440157264471,
+            0.00471991952508688,
+            -0.010495729744434357,
+            -0.016316663473844528,
+            -0.018455518409609795,
+            0.0018771382747218013,
+            0.019493358209729195,
+            -0.006412051618099213,
+            0.005536655429750681,
+            0.001989946933463216,
+            0.013627301901578903,
+            0.0012352562043815851,
+            0.011425274424254894,
+            -0.011777237989008427,
+            0.027705838903784752,
+            -0.009710581041872501,
+            0.015982748940587044,
+            0.004088190384209156,
+            -0.0017146936152130365,
+            0.006168384570628405,
+            -0.007251348812133074,
+            -0.029655175283551216,
+            -0.004467227961868048,
+            -0.010243037715554237,
+            -0.0018342708935961127,
+            -0.02433059923350811,
+            -0.024799883365631104,
+            0.0040430668741464615,
+            0.0030390687752515078,
+            0.00963838305324316,
+            0.016100069507956505,
+            0.022236868739128113,
+            0.007589775137603283,
+            0.008731400594115257,
+            -0.0021726973354816437,
+            0.004106239881366491,
+            -0.020901212468743324,
+            0.008302726782858372,
+            -0.016614478081464767,
+            -0.010576952248811722,
+            -0.0019211337203159928,
+            -0.02597309648990631,
+            0.002477281028404832,
+            -0.008131257258355618,
+            0.007319034077227116,
+            -0.00026369054103270173,
+            0.0021140368189662695,
+            0.016749849542975426,
+            -0.00894348043948412,
+            0.010613051243126392,
+            -0.005089932586997747,
+            0.00327822333201766,
+            -0.004904926288872957,
+            -0.014583920128643513,
+            -0.0011207553325220942,
+            -0.014421476051211357,
+            0.012102127075195312,
+            -0.023139338940382004,
+            -0.0045213764533400536,
+            -0.05512288585305214,
+            -0.006096187047660351,
+            -0.021911978721618652,
+            0.005978866014629602,
+            0.013519004918634892,
+            -0.012273596599698067,
+            -0.01590152643620968,
+            -0.02043192833662033,
+            0.022561758756637573,
+            -0.012038953602313995,
+            -0.0012228472623974085,
+            -0.01059500128030777,
+            0.035467084497213364,
+            -0.030431298539042473,
+            -0.001330015598796308,
+            -0.0032263314351439476,
+            0.008916406892240047,
+            -0.006615107413381338,
+            -0.024294501170516014,
+            -0.005324574653059244,
+            -0.012806054204702377,
+            0.004453691188246012,
+            -0.007661973126232624,
+            -0.017814764752984047,
+            0.005324574653059244,
+            -0.022778350859880447,
+            0.0031947449315339327,
+            0.012562386691570282,
+            -0.006800113711506128,
+            -0.00527945114299655,
+            0.004900414030998945,
+            0.007711608894169331,
+            0.009773753583431244,
+            0.0076303863897919655,
+            -0.0014890760648995638,
+            0.006344366353005171,
+            -0.009385691024363041,
+            -0.009277394972741604,
+            0.012824103236198425,
+            0.02102755755186081,
+            -0.002097115386277437,
+            0.006046551279723644,
+            -0.019385062158107758,
+            -0.011957732029259205,
+            -0.02747119590640068,
+            -0.013546079397201538,
+            -0.02100950852036476,
+            0.0009092388208955526,
+            0.01125380489975214,
+            0.00643461337313056,
+            0.01432220358401537,
+            -0.00019713335495907813,
+            -0.0030526057817041874,
+            0.0016278307884931564,
+            -0.046964555978775024,
+            -0.010215964168310165,
+            0.006380465347319841,
+            -0.005306525621563196,
+            0.009038240648806095,
+            0.020829014480113983,
+            -0.007684534881263971,
+            -0.021984176710247993,
+            0.0028540624771267176,
+            -0.0006937740836292505,
+            -0.0010694273514673114,
+            0.014078537002205849,
+            0.014069512486457825,
+            -0.006678280420601368,
+            0.013654375448822975,
+            0.004363444168120623,
+            0.022399313747882843,
+            -0.029330285266041756,
+            0.008826159872114658,
+            0.011235755868256092,
+            -0.024781834334135056,
+            0.018347222357988358,
+            0.004832728765904903,
+            -0.018094530329108238,
+            0.004155875649303198,
+            0.015712007880210876,
+            0.019709952175617218,
+            -0.002291146432980895,
+            0.012354818172752857,
+            -0.005103469826281071,
+            0.003623418277129531,
+            0.019312864169478416,
+            0.004977123811841011,
+            -0.021081706508994102,
+            -0.017589146271348,
+            -0.011641867458820343,
+            0.018500640988349915,
+            -0.005969841033220291,
+            0.012390917167067528,
+            0.018365271389484406,
+            -0.0007405896903946996,
+            -0.009909124113619328,
+            0.022706152871251106,
+            -0.0006666999543085694,
+            0.006831700447946787,
+            0.0014180064899846911,
+            0.0004551834717858583,
+            -0.019006025046110153,
+            0.000448414939455688,
+            0.008153819479048252,
+            0.012815078720450401,
+            0.015919575467705727,
+            0.01398828998208046,
+            -0.01597372442483902,
+            -0.003149621421471238,
+            -0.001759817125275731,
+            -0.012463115155696869,
+            -0.023373981937766075,
+            0.014953932724893093,
+            -0.006588033400475979,
+            0.018085505813360214,
+            -0.005906668026000261,
+            -0.001572554581798613,
+            -0.01303167175501585,
+            0.018265999853610992,
+            0.001610909472219646,
+            0.01109135989099741,
+            -0.00012359612446743995,
+            -0.002317092614248395,
+            0.00717915128916502,
+            0.0005073575302958488,
+            -0.017228158190846443,
+            -0.013248263858258724,
+            0.013455832377076149,
+            0.005780322477221489,
+            -0.0029127229936420918,
+            0.02216467075049877,
+            -0.0010936812032014132,
+            0.024366699159145355,
+            0.019836297258734703,
+            0.01277897972613573,
+            -0.006520348135381937,
+            0.008456147275865078,
+            0.026081392541527748,
+            -0.021659286692738533,
+            0.013943166472017765,
+            0.004891389049589634,
+            -0.02061242237687111,
+            -0.03183915093541145,
+            0.0007620233809575438,
+            -0.0006949021480977535,
+            0.010495729744434357,
+            -0.02167733758687973,
+            -0.00934959203004837,
+            -0.015964699909090996,
+            -0.009909124113619328,
+            0.008388461545109749,
+            0.00376330129802227,
+            0.008587004616856575,
+            0.008546394295990467,
+            -0.01327533833682537,
+            0.0032105380669236183,
+            0.0165513064712286,
+            0.0033210907131433487,
+            -0.012255546636879444,
+            0.014006339013576508,
+            0.03220013901591301,
+            0.005450920667499304,
+            -0.031063027679920197,
+            0.016948392614722252,
+            0.0012792516499757767,
+            0.017697442322969437,
+            -0.004329601302742958,
+            -0.007968813180923462,
+            -0.008442609570920467,
+            0.01472831517457962,
+            -0.03218208998441696,
+            -0.0022753532975912094,
+            0.018933827057480812,
+            0.003181207925081253,
+            0.008559931069612503,
+            0.009430814534425735,
+            0.001052506035193801,
+            -0.017905011773109436,
+            0.004864315036684275,
+            -0.014403426088392735,
+            -0.0007287447806447744,
+            0.011795287020504475,
+            -0.01360925193876028,
+            -0.009191660210490227,
+            0.014331229031085968,
+            -0.00767550989985466,
+            -0.02821122296154499,
+            0.03292211890220642,
+            -0.013347536325454712,
+            -0.008013936690986156,
+            0.0372539758682251,
+            0.004769555758684874,
+            -0.005789346992969513,
+            0.011073310859501362,
+            0.015712007880210876,
+            -0.007314521819353104,
+            0.004113008268177509,
+            0.018320146948099136,
+            -0.019980693235993385,
+            -0.011280879378318787,
+            0.018482591956853867,
+            0.03019665740430355,
+            -0.0013221190311014652,
+            0.0007479222840629518,
+            -0.017589146271348,
+            -0.015423217788338661,
+            -0.005455432925373316,
+            0.008361387066543102,
+            0.01804940588772297,
+            -0.012544337660074234,
+            -0.01733645424246788,
+            0.02373497001826763,
+            0.0045236325822770596,
+            0.005062858574092388,
+            0.0071340277791023254,
+            0.022525658831000328,
+            -0.013049720786511898,
+            -0.025594057515263557,
+            0.014656118117272854,
+            -0.010360359214246273,
+            -0.01782378926873207,
+            -0.017760615795850754,
+            0.014087561517953873,
+            -0.012634584680199623,
+            -0.015531513839960098,
+            0.021641237661242485,
+            0.0013367841020226479,
+            -0.0032060258090496063,
+            0.02225491777062416,
+            0.010748421773314476,
+            -0.0167408250272274,
+            -0.03783155605196953,
+            -0.021370496600866318,
+            -0.00038298583240248263,
+            0.009584235027432442,
+            0.006538397632539272,
+            0.009029215201735497,
+            -0.025449663400650024,
+            -0.0029826643876731396,
+            -0.025052575394511223,
+            -0.0005820933729410172,
+            -0.006849749479442835,
+            -0.010694272816181183,
+            -0.004122033249586821,
+            -0.00376330129802227,
+            0.021695386618375778,
+            0.01647910848259926,
+            -0.0009487218922004104,
+            0.006344366353005171,
+            -0.002160288393497467,
+            -0.008623103611171246,
+            -0.011578693985939026,
+            0.01646105945110321,
+            0.016903268173336983,
+            -0.02017923630774021,
+            -0.0028224759735167027,
+            0.011190632358193398,
+            0.0026690559461712837,
+            -0.013509980402886868,
+            -0.019096272066235542,
+            -0.009493988007307053,
+            -0.009173611178994179,
+            -0.024547193199396133,
+            -0.002287762239575386,
+            -0.006023989524692297,
+            -0.025449663400650024,
+            0.012842152267694473,
+            0.04230780899524689,
+            -0.01606397144496441,
+            -0.0032285875640809536,
+            0.006664743181318045,
+            -0.0028292443603277206,
+            0.0072107380256056786,
+            0.005311037879437208,
+            -0.008975067175924778,
+            0.02052217535674572,
+            -0.004376980941742659,
+            -0.004986148327589035,
+            -0.002004612237215042,
+            0.005148593336343765,
+            -0.011389175429940224,
+            -0.02458329126238823,
+            -0.025016477331519127,
+            0.006935484241694212,
+            -0.0006949021480977535,
+            0.0011822361266240478,
+            -0.009087876416742802,
+            0.0012871483340859413,
+            -0.01749889925122261,
+            0.009137512184679508,
+            0.012941424734890461,
+            -0.0024592315312474966,
+            -0.0045258887112140656,
+            -0.013519004918634892,
+            -0.011136483401060104,
+            0.005771297495812178,
+            0.017047664150595665,
+            0.006127773318439722,
+            -0.01800428330898285,
+            0.0007157717482186854,
+            0.012661658227443695,
+            0.02680336870253086,
+            -0.023482277989387512,
+            0.013717548921704292,
+            0.012733856216073036,
+            0.001949335914105177,
+            0.0020666569471359253,
+            0.019728001207113266,
+            -0.007007681764662266,
+            -0.004864315036684275,
+            0.00606008805334568,
+            -0.015242723748087883,
+            0.0029578465037047863,
+            0.026099441573023796,
+            -0.008117720484733582,
+            0.014286105521023273,
+            -0.05353453755378723,
+            -0.010405482724308968,
+            0.0009977937443181872,
+            -0.016352761536836624,
+            -0.0055817789398133755,
+            0.024095958098769188,
+            -0.00013755621330346912,
+            -0.003215050557628274,
+            -0.0055772666819393635,
+            0.012237497605383396,
+            -0.02176758460700512,
+            -0.04353516921401024,
+            0.0018748820293694735,
+            -0.01590152643620968,
+            -0.016361786052584648,
+            0.00014249159721657634,
+            -0.018933827057480812,
+            0.007454404607415199,
+            -0.00963838305324316,
+            -0.008496757596731186,
+            -0.014665142633020878,
+            0.027290701866149902,
+            -0.000774996355175972,
+            -0.0012668427079916,
+            -0.01812160387635231,
+            -0.01184041053056717,
+            -0.015784205868840218,
+            -0.011849435046315193,
+            -0.008749449625611305,
+            -0.0028247321024537086,
+            -0.008226016536355019,
+            -0.0034451803658157587,
+            0.0045258887112140656,
+            0.017444750294089317,
+            -0.020413879305124283,
+            -0.02035973034799099,
+            -0.0028946734964847565,
+            -0.01009864266961813,
+            0.004832728765904903,
+            0.0138619439676404,
+            0.015125402249395847,
+            -0.0054328711703419685,
+            0.007359645329415798,
+            -0.007846979424357414,
+            0.0016571611631661654,
+            -0.010243037715554237,
+            0.03862572833895683,
+            0.0027322289533913136,
+            0.0008325288654305041,
+            -0.0019042124040424824,
+            -8.156604053510819e-06,
+            0.005446408409625292,
+            -0.004977123811841011,
+            -0.017507923766970634,
+            0.010549877770245075,
+            -0.02326568402349949,
+            0.021406596526503563,
+            -0.010279136709868908,
+            0.01017084065824747,
+            -0.005762272980064154,
+            -0.026586774736642838,
+            -0.002233613980934024,
+            0.024781834334135056,
+            0.0010919891064986587,
+            0.009227759204804897,
+            -0.01782378926873207,
+            -0.008636641316115856,
+            -0.0030751675367355347,
+            -0.022291017696261406,
+            -0.009584235027432442,
+            -0.00901567842811346,
+            0.002319348743185401,
+            -0.010576952248811722,
+            0.014935883693397045,
+            0.017580121755599976,
+            0.005053833592683077,
+            -0.006831700447946787,
+            0.009710581041872501,
+            -0.02979956939816475,
+            0.008781036362051964,
+            -0.003436155617237091,
+            -0.005473482422530651,
+            -0.019583605229854584,
+            0.019421162083745003,
+            0.0001576502836542204,
+            -0.0027389973402023315,
+            -0.002817963482812047,
+            0.02100950852036476,
+            0.008361387066543102,
+            0.009683506563305855,
+            -0.005468970164656639,
+            -0.004638697486370802,
+            -0.01221042312681675,
+            0.003219562815502286,
+            -0.00019558223721105605,
+            -0.0002899185928981751,
+            -0.0016176780918613076,
+            0.005320062395185232,
+            -0.010486705228686333,
+            -0.025142822414636612,
+            0.017192060127854347,
+            -0.00894348043948412,
+            -0.006664743181318045,
+            0.017155960202217102,
+            -0.01514345221221447,
+            0.0011213193647563457,
+            0.004088190384209156,
+            0.0004159824165981263,
+            0.004162644501775503,
+            0.004760530777275562,
+            0.008884820155799389,
+            -0.0063082673586905,
+            -0.020558273419737816,
+            0.011235755868256092,
+            -0.013482906855642796,
+            -0.005139568354934454,
+            0.021911978721618652,
+            -0.006529372651129961,
+            -0.002199771348387003,
+            -0.010486705228686333,
+            -0.02183978073298931,
+            0.01481856219470501,
+            -0.02689361572265625,
+            0.0016481364145874977,
+            -0.0029804082587361336,
+            0.01588347740471363,
+            -0.011939682066440582,
+            0.02368082106113434,
+            -0.006655718665570021,
+            -0.006263143848627806,
+            0.0021208052057772875,
+            -0.0260633435100317,
+            0.008198942989110947,
+            -0.035449035465717316,
+            0.004338625818490982,
+            -0.01125380489975214,
+            0.029258087277412415,
+            0.018645036965608597,
+            -0.006425588857382536,
+            0.011398199945688248,
+            -0.02904149517416954,
+            0.004214536398649216,
+            0.01411463599652052,
+            0.0038783662021160126,
+            -0.0040295301005244255,
+            0.007449892349541187,
+            -0.010865742340683937,
+            -0.023536425083875656,
+            -0.02954687736928463,
+            0.001267970772460103,
+            -0.03481730446219444,
+            0.0013785234186798334,
+            0.0012916605919599533,
+            -0.016912294551730156,
+            0.010324260219931602,
+            -0.006759502459317446,
+            -0.020666569471359253,
+            0.017228158190846443,
+            0.0031157787889242172,
+            -0.012878251262009144,
+            -0.0042777094058692455,
+            0.006750477943569422,
+            0.02160513959825039,
+            0.014638068154454231,
+            -0.009737654589116573,
+            0.0008404254913330078,
+            0.010585976764559746,
+            -0.006389489863067865,
+            -0.010197914205491543,
+            -0.022796399891376495,
+            -0.00883067212998867,
+            0.004004712216556072,
+            0.006443637888878584,
+            -0.006664743181318045,
+            0.02001679129898548,
+            0.011226730421185493,
+            0.0012205911334604025,
+            0.01352803036570549,
+            -0.01389804296195507,
+            0.018338195979595184,
+            -0.0031090101692825556,
+            -0.014629043638706207,
+            0.0019290302880108356,
+            -0.03593636676669121,
+            0.005965328775346279,
+            -0.006213508080691099,
+            0.025756502524018288,
+            -0.01439440157264471,
+            -0.0017530486220493913,
+            -0.018455518409609795,
+            -0.0025336854159832,
+            0.015684934332966805,
+            -0.012598485685884953,
+            -0.011280879378318787,
+            0.013537054881453514,
+            -0.013176066800951958,
+            -0.0020305581856518984,
+            0.016930343583226204,
+            -0.010089618153870106,
+            0.01233676914125681,
+            -0.007188176270574331,
+            -0.00027215119916945696,
+            -0.00182637432590127,
+            -0.0012566898949444294,
+            -0.004566499963402748,
+            -0.0013412964763119817,
+            0.0028337568510323763,
+            -0.020143138244748116,
+            0.029582977294921875,
+            -0.013509980402886868,
+            -0.010766470804810524,
+            0.004250635392963886,
+            -0.003032300155609846,
+            0.021875880658626556,
+            -0.011650891974568367,
+            0.00391220860183239,
+            0.000977488118223846,
+            -0.010342310182750225,
+            -0.008167356252670288,
+            -0.004559731110930443,
+            0.013013621792197227,
+            0.026279935613274574,
+            -0.019980693235993385,
+            0.009394715540111065,
+            0.00997229665517807,
+            -0.02350032702088356,
+            -0.01869918406009674,
+            -0.004543938208371401,
+            -0.013446807861328125,
+            0.01788696087896824,
+            -0.00039454875513911247,
+            -0.005658488720655441,
+            0.0036843351554125547,
+            0.005035784561187029,
+            -0.014340253546833992,
+            -0.000682493147905916,
+            -0.003898671828210354,
+            0.004345394670963287,
+            -0.006281193345785141,
+            0.011768212541937828,
+            0.0006520348251797259,
+            0.02059437334537506,
+            -0.00926837045699358,
+            -0.014132685028016567,
+            0.0064842491410672665,
+            -0.00997229665517807,
+            0.009882049635052681,
+            -0.021208051592111588,
+            0.0010152790928259492,
+            0.0074724541045725346,
+            -0.005098957102745771,
+            0.004841753281652927,
+            0.009484962560236454,
+            -0.02739899791777134,
+            0.005969841033220291,
+            0.005482506938278675,
+            -0.0015691702719777822,
+            0.016497157514095306,
+            0.009503012523055077,
+            -0.0007095672772265971,
+            0.0006988504319451749,
+            -0.014719290658831596,
+            0.019385062158107758,
+            0.006921947468072176,
+            0.019096272066235542,
+            0.006773039698600769,
+            0.015522489324212074,
+            -0.00013586408749688417,
+            -0.005622390192002058,
+            0.006782064214348793,
+            -0.006064600311219692,
+            -0.004595830105245113,
+            0.03557537868618965,
+            -0.0006255247280932963,
+            0.012300670146942139,
+            0.006696329917758703,
+            -0.00030599383171647787,
+            0.0056765382178127766,
+            0.0120028555393219,
+            0.0012454090174287558,
+            -0.017697442322969437,
+            0.011235755868256092,
+            -0.015991773456335068,
+            -0.019637754186987877,
+            0.0019380550365895033,
+            0.009394715540111065,
+            -0.00780185591429472,
+            -0.0026081393007189035,
+            0.026099441573023796,
+            0.008550906553864479,
+            0.009358617477118969,
+            0.003530915128067136,
+            -0.007901127450168133,
+            -0.002162544522434473,
+            0.005956304259598255,
+            0.000220682195504196,
+            0.001759817125275731,
+            -0.004805654287338257,
+            -0.02581065148115158,
+            0.003303041448816657,
+            -0.005446408409625292,
+            -0.00340456934645772,
+            -0.0015026130713522434,
+            0.018383320420980453,
+            -0.01352803036570549,
+            0.01544126681983471,
+            0.006033014040440321,
+            -0.004113008268177509,
+            -0.014304154552519321,
+            0.0062315575778484344,
+            0.004081421997398138,
+            0.020883163437247276,
+            -0.001704540802165866,
+            -0.019746050238609314,
+            -0.0227603018283844,
+            0.003797143930569291,
+            -5.8026020269608125e-05,
+            -0.0034880477469414473,
+            0.0022945308592170477,
+            -0.002249407349154353,
+            0.008216992020606995,
+            -0.0034857916180044413,
+            -0.006673768162727356,
+            0.002287762239575386,
+            -0.006154847331345081,
+            0.02017923630774021,
+            -0.002326117129996419,
+            -0.0006571111734956503,
+            0.0023667283821851015,
+            -0.004013736732304096,
+            0.001205925946123898,
+            -0.002102755941450596,
+            0.016794972121715546,
+            -0.017192060127854347,
+            -0.00393702695146203,
+            -0.009078850969672203,
+            -0.017480850219726562,
+            0.0014721547486260533,
+            -0.009259345009922981,
+            0.006664743181318045,
+            0.013943166472017765,
+            0.014764414168894291,
+            -0.004990661051124334,
+            -0.0017846350092440844,
+            0.01918651908636093,
+            0.011677965521812439,
+            0.0010474296286702156,
+            0.011353076435625553,
+            0.0009002141305245459,
+            -0.010992088355123997,
+            -0.0006266528507694602,
+            -0.010910865850746632,
+            -0.007607824634760618,
+            0.023554475978016853,
+            0.02274225279688835,
+            0.012318720109760761,
+            0.0025742966681718826,
+            -0.0023486788850277662,
+            -0.0029285161290317774,
+            0.006294730585068464,
+            0.000976360053755343,
+            -0.011885534040629864,
+            0.0021456230897456408,
+            0.0006243966636247933,
+            -0.022615905851125717,
+            0.013925116509199142,
+            -0.02929418720304966,
+            0.0240057110786438,
+            -0.0018365271389484406,
+            0.009187147952616215,
+            0.019547507166862488,
+            -0.0011280878679826856,
+            0.013464856892824173,
+            0.01096501387655735,
+            0.0019414392299950123,
+            -0.02465548925101757,
+            -0.00811320822685957,
+            -0.010378408245742321,
+            -0.005256889387965202,
+            0.006989632733166218,
+            0.009114949963986874,
+            0.000154830064275302,
+            0.0030841922853142023,
+            -0.0021241893991827965,
+            0.001891803345642984,
+            -0.01969190314412117,
+            0.011903583072125912,
+            0.0022178208455443382,
+            -0.010215964168310165,
+            -0.023933513090014458,
+            -0.005703612230718136,
+            0.009864000603556633,
+            0.0022099241614341736,
+            -0.003519634250551462,
+            0.008365900255739689,
+            -0.0063308291137218475,
+            -0.0008658074657432735,
+            0.00024973045219667256,
+            -0.003395544597879052,
+            0.0016526487888768315,
+            -0.005866057239472866,
+            0.005117006599903107,
+            0.005766785237938166,
+            -0.0008415535558015108,
+            0.008230529725551605,
+            -0.01046865526586771,
+            0.013952190987765789,
+            -0.011407224461436272,
+            0.005613365210592747,
+            -0.008537368848919868,
+            0.006831700447946787,
+            0.008893844671547413,
+            0.005861544515937567,
+            0.013148992322385311,
+            -0.0003739611420314759,
+            -0.0013164785923436284,
+            -0.016217391937971115,
+            -0.03485340252518654,
+            0.006366928108036518,
+            -0.0009064186015166342,
+            0.006588033400475979,
+            -0.02160513959825039,
+            0.0050809080712497234,
+            0.012580436654388905,
+            -0.017255231738090515,
+            0.0010169711895287037,
+            0.005784834735095501,
+            0.015910550951957703,
+            -0.011154533363878727,
+            0.000819555833004415,
+            0.007043780758976936,
+            0.018825531005859375,
+            0.0031202910467982292,
+            0.004189718514680862,
+            -0.02225491777062416,
+            -0.012670683674514294,
+            0.015341995283961296,
+            -0.030738139525055885,
+            0.008641153573989868,
+            0.020702669396996498,
+            -0.009286419488489628,
+            0.03109912760555744,
+            0.01788696087896824,
+            -0.013573153875768185,
+            -0.0073731825686991215,
+            -0.008573467843234539,
+            0.022453460842370987,
+            -0.0067324284464120865,
+            0.009683506563305855,
+            -0.014620019122958183,
+            0.011037211865186691,
+            0.0046838209964334965,
+            0.005202741362154484,
+            0.010983063839375973,
+            -0.027128256857395172,
+            0.007702583912760019,
+            4.857828389503993e-05,
+            -0.026586774736642838,
+            0.00804552249610424,
+            -0.005771297495812178,
+            0.020919261500239372,
+            -0.001577066839672625,
+            0.012607510201632977,
+            0.024962328374385834,
+            0.00027088209753856063,
+            -0.012381892651319504,
+            -0.0073686703108251095,
+            0.009900099597871304,
+            -0.012860202230513096,
+            0.028355617076158524,
+            -0.00042472509085200727,
+            0.0009227758855558932,
+            -0.005049321334809065,
+            0.008352362550795078,
+            -0.017598170787096024,
+            0.004701870493590832,
+            -0.013970240019261837,
+            -0.016731800511479378,
+            0.0011574181262403727,
+            0.012147250585258007,
+            -0.0006035270052962005,
+            0.013257289305329323,
+            0.011073310859501362,
+            0.0040385546162724495,
+            0.014195858500897884,
+            0.003397800726816058,
+            0.0023283734917640686,
+            0.022309066727757454,
+            0.010450606234371662,
+            0.001882778713479638,
+            -0.0018940595909953117,
+            -0.0022798655554652214,
+            0.03169475868344307,
+            -6.616940663661808e-05,
+            -0.004652234259992838,
+            0.010874766856431961,
+            0.01432220358401537,
+            0.006384977605193853,
+            0.01088379230350256,
+            0.015712007880210876,
+            0.0012713550822809339,
+            0.003219562815502286,
+            0.01046865526586771,
+            0.002475024899467826,
+            0.01240896712988615,
+            0.003131571924313903,
+            -0.006493274122476578,
+            -0.0038084248080849648,
+            0.006218020338565111,
+            0.013284362852573395,
+            -0.01000839564949274,
+            0.017489874735474586,
+            0.009493988007307053,
+            0.00943983905017376,
+            -0.005044809076935053,
+            0.007070854771882296,
+            0.012950449250638485,
+            0.0034384119790047407,
+            -0.007291960064321756,
+            -0.011389175429940224,
+            0.002443438395857811,
+            -0.027759985998272896,
+            0.012363843619823456,
+            0.0014349278062582016,
+            -0.016822047531604767,
+            -0.007314521819353104,
+            -0.003946051467210054,
+            -0.015919575467705727,
+            0.0032872480805963278,
+            -0.002628444693982601,
+            0.026117490604519844,
+            -0.0007608952582813799,
+            -0.02532331645488739,
+            -0.008411023765802383,
+            0.016993515193462372,
+            -0.007341595832258463,
+            0.005153105594217777,
+            0.012255546636879444,
+            0.028084876015782356,
+            0.017020590603351593,
+            -0.0006706482381559908,
+            -0.003896415699273348,
+            -0.004904926288872957,
+            -0.003957332111895084,
+            0.010775495320558548,
+            -0.0159286018460989,
+            -0.008591517806053162,
+            -0.01647910848259926,
+            -0.016587404534220695,
+            0.0023712406400591135,
+            0.009575209580361843,
+            -0.004505583085119724,
+            -0.007201713044196367,
+            -0.0020192773081362247,
+            0.001828630454838276,
+            -0.003657260909676552,
+            0.02043192833662033,
+            -0.01860893703997135,
+            -0.02781413495540619,
+            0.0031586461700499058,
+            -0.0038851348217576742,
+            0.0073776948265731335,
+            0.006660230923444033,
+            0.014656118117272854,
+            -0.002908210502937436,
+            -0.012616534717381,
+            0.04371566325426102,
+            -0.009277394972741604,
+            0.017074737697839737,
+            -0.006750477943569422,
+            0.016325687989592552,
+            0.015062229707837105,
+            0.0060871620662510395,
+            0.0054238466545939445,
+            0.019258717074990273,
+            -0.01647910848259926,
+            -0.002655518939718604,
+            0.007639411371201277,
+            0.009864000603556633,
+            0.02498037740588188,
+            0.008474196307361126,
+            0.0010474296286702156,
+            -0.02044997736811638,
+            -0.003066142788156867,
+            -0.019258717074990273,
+            -0.0009453376405872405,
+            -0.015260772779583931,
+            -0.003970869351178408,
+            0.01233676914125681,
+            -0.02218271978199482,
+            -0.013248263858258724,
+            -0.02160513959825039,
+            -0.006375952623784542,
+            0.009818877093493938,
+            0.0006221404764801264,
+            0.018437469378113747,
+            0.00897957943379879,
+            0.00789661519229412,
+            0.007869540713727474,
+            0.015161501243710518,
+            -0.006014964543282986,
+            0.002693873830139637,
+            0.00550055643543601,
+            0.0009013422532007098,
+            0.011578693985939026,
+            -0.00045151717495173216,
+            -0.013762672431766987,
+            0.012580436654388905,
+            0.0045281448401510715,
+            0.009236783720552921,
+            -0.02582870051264763,
+            -0.023789117112755775,
+            0.01112745888531208,
+            0.0067369407042860985,
+            0.004092703107744455,
+            -0.013356560841202736,
+            0.010658174753189087,
+            -0.003269198816269636,
+            -0.006466199643909931,
+            -0.005762272980064154,
+            0.0036346991546452045,
+            -0.016993515193462372,
+            -0.007783806417137384,
+            0.00637144036591053,
+            0.01664155349135399,
+            0.000557275430765003,
+            0.014881735667586327,
+            0.010288161225616932,
+            0.02283249981701374,
+            0.012733856216073036,
+            -0.026099441573023796,
+            -0.002752534346655011,
+            0.007143052760511637,
+            0.002515635918825865,
+            -0.023608623072504997,
+            -0.009340567514300346,
+            -0.002231357851997018,
+            -0.0010158431250602007,
+            0.026947762817144394,
+            -0.0025449662934988737,
+            0.011362100951373577,
+            -0.009295444004237652,
+            -0.004399542696774006,
+            -0.018843580037355423,
+            0.031008880585432053,
+            -0.003934770356863737,
+            0.005793859250843525,
+            -0.007607824634760618,
+            -0.013762672431766987,
+            -0.012165299616754055,
+            -0.004742481280118227,
+            0.00123638438526541,
+            0.06407539546489716,
+            -0.017562072724103928,
+            0.010477680712938309,
+            -0.008456147275865078,
+            0.006046551279723644,
+            0.022236868739128113,
+            0.010279136709868908,
+            0.0060826498083770275,
+            -0.011163557879626751,
+            -0.01472831517457962,
+            -0.004823703784495592,
+            -0.002286634175106883,
+            0.008451634086668491,
+            0.005157617852091789,
+            -0.0058344705030322075,
+            -0.02144269458949566,
+            -0.025016477331519127,
+            -0.023554475978016853,
+            -0.0012172068236395717,
+            -0.01022498868405819,
+            -0.003677566535770893,
+            0.00041711051017045975,
+            0.0013548335991799831,
+            0.008794573135674,
+            0.01960165612399578,
+            -0.007643923629075289,
+            -0.02391546405851841,
+            -0.027850233018398285,
+            0.014574895612895489,
+            -0.04227171093225479,
+            0.01860893703997135,
+            0.0004881800268776715,
+            -0.027453146874904633,
+            -0.012643609195947647,
+            0.013049720786511898,
+            0.017092788591980934,
+            -0.010901841334998608,
+            0.021731484681367874,
+            0.010685248300433159,
+            -0.011470397934317589,
+            0.0029307722579687834,
+            0.022633954882621765,
+            0.026586774736642838,
+            0.012634584680199623,
+            0.0032060258090496063,
+            0.01853674091398716,
+            0.013221190311014652,
+            -0.018013307824730873,
+            0.007698071654886007,
+            0.01567590981721878,
+            -0.012463115155696869,
+            0.008122232742607594,
+            -0.010432557202875614,
+            0.012986548244953156,
+            -0.005351649131625891,
+            -0.001114550861530006,
+            0.030665941536426544,
+            -0.028608309105038643,
+            0.021208051592111588,
+            -0.031171325594186783,
+            0.008000398986041546,
+            -0.004990661051124334,
+            -0.0004560295201372355,
+            0.010712322778999805,
+            -0.051513005048036575,
+            0.03245283290743828,
+            0.018717234954237938,
+            -5.53468125872314e-05,
+            -0.026297984644770622,
+            -0.01312191877514124,
+            0.021460743620991707,
+            0.007792831398546696,
+            -0.029907865449786186,
+            -0.019475309178233147,
+            0.009548136033117771,
+            -0.0010711194481700659,
+            0.0011500855907797813,
+            -0.007779294159263372,
+            -0.009173611178994179,
+            0.003406825475394726,
+            -0.02615359053015709,
+            0.02781413495540619,
+            -0.018455518409609795,
+            0.02285054884850979,
+            0.007670997641980648,
+            0.006384977605193853,
+            -0.0005039732786826789,
+            -0.008510295301675797,
+            -0.009620334021747112,
+            0.01680399663746357,
+            -0.0053290873765945435,
+            -0.002637469442561269,
+            -0.0173635296523571,
+            -0.0045349132269620895,
+            0.0067188916727900505,
+            0.0027886333409696817,
+            0.0022166927810758352,
+            0.011118434369564056,
+            -0.001003434183076024,
+            0.014051462523639202,
+            0.017372554168105125,
+            0.02622578665614128,
+            0.002996201394125819,
+            0.001920005539432168,
+            0.004189718514680862,
+            0.015991773456335068,
+            -0.0003677566419355571,
+            -0.00531555013731122,
+            0.0037858630530536175,
+            0.01088379230350256,
+            0.013879992999136448,
+            0.01175016351044178,
+            0.008284677751362324,
+            -0.00035986004513688385,
+            0.0044424100778996944,
+            0.016948392614722252,
+            0.020233385264873505,
+            -0.01026108767837286,
+            0.00556372944265604,
+            0.009069826453924179,
+            -0.00795076321810484,
+            0.0038512919563800097,
+            -0.007287447806447744,
+            0.007061830256134272,
+            -0.03245283290743828,
+            0.02936638332903385,
+            -0.0020666569471359253,
+            -0.008550906553864479,
+            -0.006168384570628405,
+            -0.013672425411641598,
+            0.013194115832448006,
+            -0.019637754186987877,
+            0.004367956425994635,
+            -0.02507062442600727,
+            0.008365900255739689,
+            0.002070041373372078,
+            -0.00022604061814490706,
+            0.030575694516301155,
+            0.012688732706010342,
+            0.01795915886759758,
+            -0.00787405390292406,
+            0.06663841009140015,
+            -0.004361187573522329,
+            -0.0046928455121815205,
+            -0.029276136308908463,
+            -0.018518690019845963,
+            -0.009412765502929688,
+            0.0005434563499875367,
+            0.0006892617093399167,
+            0.007413793820887804,
+            -0.0017293586861342192,
+            -0.00839297380298376,
+            0.012417991645634174,
+            -0.01141624990850687,
+            -0.013004597276449203,
+            -0.0007862772908993065,
+            0.015802254900336266,
+            0.005022247321903706,
+            -0.008383949287235737,
+            0.016054946929216385,
+            0.0030187631491571665,
+            0.0011709552491083741,
+            -0.006949021480977535,
+            0.012201398611068726,
+            -0.005107982084155083,
+            0.009078850969672203,
+            0.02573845349252224,
+            -0.003400056855753064,
+            -0.0020170211791992188,
+            0.004103983752429485,
+            -0.01187650952488184,
+            0.0018004282610490918,
+            0.006660230923444033,
+            -0.011037211865186691,
+            0.002903698245063424,
+            0.002323861001059413,
+            0.005536655429750681,
+            -0.010243037715554237,
+            -0.011407224461436272,
+            -0.0008404254913330078,
+            0.005649464204907417,
+            0.01554956380277872,
+            0.0001692131772870198,
+            0.0042777094058692455,
+            0.022652005776762962,
+            -0.01680399663746357,
+            -0.012075052596628666,
+            -0.02283249981701374,
+            0.007147565018385649,
+            0.01423195656388998,
+            0.002506611403077841,
+            -0.018987976014614105,
+            -0.012896301224827766,
+            0.02234516479074955,
+            0.009530086070299149,
+            0.034456316381692886,
+            0.006375952623784542,
+            0.0017146936152130365,
+            0.011154533363878727,
+            0.02756144292652607,
+            0.010423531755805016,
+            -0.019764099270105362,
+            0.0020621446892619133,
+            -0.012156275101006031
+          ],
+          "title": "8 Future Work"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "1f214258-3b4f-4a21-93fa-000d69b344a1",
+      "type": "child",
+      "source": {
+        "id": "65742957-6bda-446d-a61a-0fe20313c760",
+        "properties": {
+          "page_content": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity\n\nWilliam Fedus∗ liamfedus@google.com Barret Zoph∗ barretzoph@google.com Noam Shazeer noam@google.com Google, Mountain View, CA 94043, USA Editor: Alexander Clark\n\nAbstract\n\nIn deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) models defy this and instead select different parameters for each incoming example. The result is a sparsely-activated model—with an outrageous number of parameters—but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs, and training instability. We address these with the introduction of the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques mitigate the instabilities, and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large (Raffel et al., 2019) to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the \"Colossal Clean Crawled Corpus\", and achieve a 4x speedup over the T5-XXL model.12 Keywords: mixture-of-experts, natural language processing, sparsity, large-scale machine learning, distributed computing arXiv:2101.03961v3 [cs.LG] 16 Jun 2022 Contents\n\n1 Introduction 3 2 Switch Transformer 4 2.1 Simplifying Sparse Routing 5 2.2 Efficient Sparse Routing 6 2.3 Putting It All Together: The Switch Transformer 8 2.4 Improved Training and Fine-Tuning Techniques 8 3 Scaling Properties 11 3.1 Scaling Results on a Step-Basis 12 3.2 Scaling Results on a Time-Basis 13 3.3 Scaling Versus a Larger Dense Model 13 4 Downstream Results 14 4.1 Fine-Tuning 14 4.2 Distillation 16 4.3 Multilingual Learning 17 5 Designing Models with Data, Model, and Expert-Parallelism 18 5.1 Data Parallelism 20 5.2 Model Parallelism 20 5.3 Model and Data Parallelism 21 5.4 Expert and Data Parallelism 22 5.5 Expert, Model and Data Parallelism 22 5.6 Towards Trillion Parameter Models 22 6 Related Work 24 7 Discussion 25 8 Future Work 26 9 Conclusion 27 A Switch for Attention 27 B Preventing Token Dropping with No-Token-Left-Behind 29\n\nC Encouraging Exploration Across Experts 29 D Switch Transformers in Lower Compute Regimes 29 E Relation of Upstream to Downstream Model Performance 32 F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/switch_transformers.md"
+          },
+          "summary": "The Switch Transformer is a sparsely-activated model that simplifies the Mixture of Experts (MoE) approach, allowing for efficient scaling to trillion-parameter models with reduced communication and computational costs. It achieves significant speedups in pre-training and fine-tuning by activating only a subset of neural network weights for each input, maintaining a constant computational cost. The model demonstrates superior scaling properties and improved performance across various natural language processing tasks, including multilingual settings. It also allows for effective distillation into smaller dense models, preserving a portion of the quality gains. The architecture is designed to maximize parameter count while keeping floating point operations per example constant, making it suitable for both large-scale and smaller-scale applications.",
+          "headlines": {
+            "1 Introduction": [],
+            "2 Switch Transformer": [
+              "2.1 Simplifying Sparse Routing",
+              "2.2 Efficient Sparse Routing",
+              "2.3 Putting It All Together: The Switch Transformer",
+              "2.4 Improved Training and Fine-Tuning Techniques"
+            ],
+            "3 Scaling Properties": [
+              "3.1 Scaling Results on a Step-Basis",
+              "3.2 Scaling Results on a Time-Basis",
+              "3.3 Scaling Versus a Larger Dense Model"
+            ],
+            "4 Downstream Results": [
+              "4.1 Fine-Tuning",
+              "4.2 Distillation",
+              "4.3 Multilingual Learning"
+            ],
+            "5 Designing Models with Data, Model, and Expert-Parallelism": [
+              "5.1 Data Parallelism",
+              "5.2 Model Parallelism",
+              "5.3 Model and Data Parallelism",
+              "5.4 Expert and Data Parallelism",
+              "5.5 Expert, Model and Data Parallelism",
+              "5.6 Towards Trillion Parameter Models"
+            ],
+            "6 Related Work": [],
+            "7 Discussion": [],
+            "8 Future Work": [],
+            "9 Conclusion": [],
+            "A Switch for Attention": [],
+            "B Preventing Token Dropping with No-Token-Left-Behind": [],
+            "C Encouraging Exploration Across Experts": [],
+            "D Switch Transformers in Lower Compute Regimes": [],
+            "E Relation of Upstream to Downstream Model Performance": [],
+            "F Pseudo Code for Switch Transformers": []
+          },
+          "summary_embedding": [
+            -0.0060447100549936295,
+            -0.00390331051312387,
+            -0.01784370467066765,
+            -0.0021820589900016785,
+            0.013227885589003563,
+            0.011338187381625175,
+            0.0165580902248621,
+            0.024240797385573387,
+            -0.005986624862998724,
+            0.03596621751785278,
+            -0.004852031823247671,
+            -0.0068811336532235146,
+            -0.003574162255972624,
+            0.014242436736822128,
+            0.011578272096812725,
+            -0.033302053809165955,
+            -0.024519605562090874,
+            0.01050176378339529,
+            0.016093410551548004,
+            -0.006795942317694426,
+            0.007473600562661886,
+            -0.030513975769281387,
+            -0.06124480068683624,
+            -0.006838538218289614,
+            -0.0026099516544491053,
+            0.01029265858232975,
+            0.0010329445358365774,
+            -0.010873507708311081,
+            -0.053469158709049225,
+            0.023311438038945198,
+            0.025294071063399315,
+            0.005092116538435221,
+            -0.029584616422653198,
+            0.0009361362317577004,
+            -0.05161044001579285,
+            -0.04448534920811653,
+            0.0379488542675972,
+            0.012561844661831856,
+            -0.04364892467856407,
+            0.039249956607818604,
+            -0.004875265993177891,
+            -0.0022440163884311914,
+            -0.021499186754226685,
+            -0.0034909071400761604,
+            -0.007039899472147226,
+            -0.009851212613284588,
+            -0.02874819189310074,
+            0.0014037203509360552,
+            0.021886419504880905,
+            -0.0056962003000080585,
+            0.023264970630407333,
+            -0.024271776899695396,
+            0.01936165988445282,
+            0.04240977764129639,
+            -0.06957805901765823,
+            0.024659009650349617,
+            0.017549408599734306,
+            0.05120771750807762,
+            0.06511713564395905,
+            0.018339363858103752,
+            0.021669568493962288,
+            -0.014141756109893322,
+            -0.017766257748007774,
+            -0.011872569099068642,
+            0.004673904739320278,
+            -0.014970434829592705,
+            -0.01891246810555458,
+            0.008449427783489227,
+            -0.020461400970816612,
+            0.008209343999624252,
+            -0.05576157942414284,
+            -0.014730350114405155,
+            -0.034200433641672134,
+            -0.01140014547854662,
+            0.017642343416810036,
+            -0.018215449526906013,
+            0.030126741155982018,
+            0.02997184917330742,
+            0.002638994250446558,
+            0.0007822110201232135,
+            0.021049996837973595,
+            0.006509389728307724,
+            -0.0053438181057572365,
+            0.012321759946644306,
+            0.028159597888588905,
+            -0.01450575515627861,
+            -0.010106786154210567,
+            -0.03454120084643364,
+            0.01861817203462124,
+            0.01589204929769039,
+            -0.04194509983062744,
+            -0.012569589540362358,
+            -0.013824224472045898,
+            0.032341714948415756,
+            -0.009549170732498169,
+            0.04098476096987724,
+            -0.010083552449941635,
+            0.001383390510454774,
+            -0.031149037182331085,
+            -0.009463978931307793,
+            0.017069239169359207,
+            0.006172496825456619,
+            -0.016589069738984108,
+            -0.0012846461031585932,
+            0.016170857474207878,
+            0.006617815233767033,
+            -0.030142230913043022,
+            0.04488807171583176,
+            0.023497309535741806,
+            0.055296897888183594,
+            0.05579255893826485,
+            -0.037391237914562225,
+            -0.002445377642288804,
+            0.0344172865152359,
+            -0.0228312686085701,
+            -0.015040136873722076,
+            -0.034014564007520676,
+            -0.01768881268799305,
+            0.035935238003730774,
+            -0.0318460576236248,
+            0.01556677371263504,
+            0.028066661208868027,
+            -0.025712283328175545,
+            0.003488970920443535,
+            -0.03943582624197006,
+            0.003622566582635045,
+            0.01220559049397707,
+            0.006095050368458033,
+            -0.0026293133851140738,
+            -0.013615119270980358,
+            0.008906363509595394,
+            0.003055269829928875,
+            0.00800798274576664,
+            -0.04615819454193115,
+            0.018478767946362495,
+            -0.00454224506393075,
+            0.04033420979976654,
+            0.042502716183662415,
+            -0.02730768360197544,
+            -0.016743963584303856,
+            0.008557853288948536,
+            0.012104909867048264,
+            0.04188314080238342,
+            -0.007287728600203991,
+            -0.00121784838847816,
+            -0.0007836631848476827,
+            0.007322579622268677,
+            0.047552235424518585,
+            0.004646798130124807,
+            0.0020697612781077623,
+            -0.027230238541960716,
+            0.014978179708123207,
+            -0.019733402878046036,
+            0.027648448944091797,
+            0.023373395204544067,
+            0.02942972257733345,
+            -0.04048910364508629,
+            0.017518429085612297,
+            -0.011818356812000275,
+            -0.005568413063883781,
+            -0.002726121572777629,
+            -0.02856232039630413,
+            -0.0074426219798624516,
+            0.033983584493398666,
+            0.015365412458777428,
+            0.012956822291016579,
+            0.03271345794200897,
+            0.0012052633101120591,
+            -0.042502716183662415,
+            0.03804178908467293,
+            0.024550583213567734,
+            0.04640602692961693,
+            0.023466331884264946,
+            0.010796061716973782,
+            0.006571347359567881,
+            0.013375034555792809,
+            -0.019377149641513824,
+            0.015907539054751396,
+            0.02560385875403881,
+            0.009750531986355782,
+            -0.03167567402124405,
+            -0.0009501734166406095,
+            0.01178737822920084,
+            -0.004546117503196001,
+            0.03701949119567871,
+            0.04104671627283096,
+            0.020569827407598495,
+            0.05030933395028114,
+            0.0020155487582087517,
+            0.017936641350388527,
+            0.005936284549534321,
+            0.02820606529712677,
+            0.030467506498098373,
+            -0.005285732913762331,
+            0.005076627247035503,
+            -0.01136916596442461,
+            -0.03385967016220093,
+            0.014110777527093887,
+            0.01561324205249548,
+            -0.008077683858573437,
+            -0.01697630248963833,
+            0.006033093202859163,
+            0.009169681929051876,
+            -0.01795213110744953,
+            -0.017456471920013428,
+            0.005971135571599007,
+            0.018866000697016716,
+            -0.013576395809650421,
+            0.024689987301826477,
+            -0.04442339017987251,
+            0.029119934886693954,
+            -0.033209118992090225,
+            -0.003384418087080121,
+            -0.010641167871654034,
+            -0.0030843124259263277,
+            0.0041666291654109955,
+            0.011314953677356243,
+            0.013406013138592243,
+            0.002042655134573579,
+            -0.013909416273236275,
+            0.024519605562090874,
+            0.018447788432240486,
+            0.012073930352926254,
+            -0.03148980438709259,
+            -0.014397329650819302,
+            -0.0015673263696953654,
+            -0.008743725717067719,
+            -0.034138478338718414,
+            0.026966920122504234,
+            0.014048820361495018,
+            -0.03844451159238815,
+            0.017797237262129784,
+            0.029538147151470184,
+            0.014420564286410809,
+            -0.004790074657648802,
+            -0.0463130883872509,
+            0.04045812413096428,
+            -0.030436528846621513,
+            -0.033333033323287964,
+            -0.010478530079126358,
+            -0.0003146269591525197,
+            0.02894955314695835,
+            -0.0064358157105743885,
+            -0.050835974514484406,
+            0.01184933539479971,
+            0.020631784573197365,
+            0.00010763872705865651,
+            0.005731051322072744,
+            -0.019593998789787292,
+            0.00028631053282879293,
+            -0.03395260497927666,
+            0.0090844901278615,
+            0.026750069111585617,
+            0.008867640048265457,
+            -0.016418687999248505,
+            0.03351890295743942,
+            -0.0004121613164898008,
+            -0.018695618957281113,
+            -0.003889757441356778,
+            0.006885006092488766,
+            0.034912943840026855,
+            0.004836542531847954,
+            -0.02946070022881031,
+            0.05665995925664902,
+            -0.02100352756679058,
+            0.021344292908906937,
+            0.05631919577717781,
+            -0.019439106807112694,
+            -0.008472662419080734,
+            -0.002065889071673155,
+            0.0026196325197815895,
+            -0.0260530486702919,
+            -0.008937342092394829,
+            0.032341714948415756,
+            0.041728246957063675,
+            -0.018742086365818977,
+            -0.001398879918269813,
+            0.006780453026294708,
+            0.009425255469977856,
+            0.004948840010911226,
+            0.02692045085132122,
+            -0.012128143571317196,
+            -0.014559967443346977,
+            0.05198218300938606,
+            0.028314489871263504,
+            -0.012081675231456757,
+            -0.0164031982421875,
+            0.019888296723365784,
+            -0.024844881147146225,
+            -0.020244551822543144,
+            -0.02431824430823326,
+            -0.020414933562278748,
+            -0.0042518205009400845,
+            0.009525936096906662,
+            0.037484172731637955,
+            -0.03968365862965584,
+            0.012747716158628464,
+            0.021700548008084297,
+            0.02981695532798767,
+            -0.019532041624188423,
+            -0.05910727381706238,
+            -0.0016128262504935265,
+            0.03296129032969475,
+            -0.015458349138498306,
+            -0.049379974603652954,
+            0.015164051204919815,
+            0.0260530486702919,
+            0.05957195535302162,
+            -0.001461805310100317,
+            -0.014954946003854275,
+            0.061926331371068954,
+            0.03026614524424076,
+            -0.004383479710668325,
+            -0.009494957514107227,
+            -0.0127864396199584,
+            0.003396035172045231,
+            0.00465454300865531,
+            -0.03364282101392746,
+            -0.041170634329319,
+            -0.011841590516269207,
+            -0.010284913703799248,
+            0.008108663372695446,
+            -0.040922801941633224,
+            0.03531566634774208,
+            -0.021917399019002914,
+            0.010950954630970955,
+            0.015473837964236736,
+            0.037391237914562225,
+            -0.021344292908906937,
+            -0.01284839678555727,
+            -0.01572941243648529,
+            -0.024968795478343964,
+            -0.014010096900165081,
+            0.021917399019002914,
+            0.007682706229388714,
+            3.4699722164077684e-05,
+            0.010075807571411133,
+            -0.024659009650349617,
+            -0.051052823662757874,
+            0.0291044469922781,
+            0.0015605497173964977,
+            -0.015938518568873405,
+            -0.004623564425855875,
+            -0.01977987214922905,
+            -0.01122201792895794,
+            -0.012608313001692295,
+            -0.039249956607818604,
+            0.021111954003572464,
+            -0.01135367713868618,
+            0.015876561403274536,
+            0.0009250032599084079,
+            0.003659353591501713,
+            0.028624277561903,
+            0.01116006076335907,
+            0.0051811798475682735,
+            -0.03255856782197952,
+            -0.018277406692504883,
+            0.010122275911271572,
+            -0.00561100896447897,
+            -0.018339363858103752,
+            -0.013351799920201302,
+            -0.05037129297852516,
+            -0.03305422514677048,
+            -0.008054450154304504,
+            -0.05415068939328194,
+            0.0020697612781077623,
+            -0.014296649023890495,
+            -0.02232012152671814,
+            -0.005990497302263975,
+            0.03107159025967121,
+            -0.003235333366319537,
+            0.025294071063399315,
+            0.0076594725251197815,
+            0.037360258400440216,
+            0.040117356926202774,
+            0.008426194079220295,
+            0.03065337985754013,
+            -0.0029991210903972387,
+            -0.016867877915501595,
+            0.05486319959163666,
+            -0.01881953328847885,
+            0.013529927469789982,
+            -0.008604321628808975,
+            0.008604321628808975,
+            0.03299226611852646,
+            -0.003794885240495205,
+            0.02007416822016239,
+            -0.0311955064535141,
+            -0.07998688519001007,
+            0.01637221872806549,
+            0.028159597888588905,
+            -0.06418777257204056,
+            0.014691627584397793,
+            -0.020275529474020004,
+            0.02052335813641548,
+            0.02411688305437565,
+            -0.0009114501299336553,
+            -0.014064309187233448,
+            0.005595519673079252,
+            0.0020833145827054977,
+            -0.01456771232187748,
+            -0.002691270550712943,
+            -0.009417511522769928,
+            0.008774704299867153,
+            -0.017007282003760338,
+            -0.013406013138592243,
+            -0.034231413155794144,
+            0.010873507708311081,
+            0.01233724970370531,
+            -0.003969140350818634,
+            0.0011500825639814138,
+            -0.03277541697025299,
+            -0.007876322604715824,
+            -0.02145271934568882,
+            0.03382869064807892,
+            0.012058441527187824,
+            -0.030699847266077995,
+            -0.007814365439116955,
+            0.015264731831848621,
+            -0.007876322604715824,
+            0.0008906363509595394,
+            0.024132372811436653,
+            0.02103450708091259,
+            0.0028364830650389194,
+            -0.0009700191440060735,
+            0.02438020147383213,
+            0.007427132688462734,
+            -0.01720864325761795,
+            0.028794659301638603,
+            0.014288905076682568,
+            0.060160547494888306,
+            -0.03271345794200897,
+            0.001297231181524694,
+            -0.04135650396347046,
+            -0.031118059530854225,
+            0.006408709101378918,
+            0.014350862242281437,
+            0.007597515359520912,
+            -0.028252532705664635,
+            0.005978880450129509,
+            0.02004319056868553,
+            0.01469937153160572,
+            0.009704063646495342,
+            0.0037580979987978935,
+            -0.036740683019161224,
+            -0.000691695255227387,
+            -0.011214273050427437,
+            -0.0038781403563916683,
+            0.035656433552503586,
+            -0.008186109364032745,
+            -0.05151750147342682,
+            -0.0006418389966711402,
+            -0.03255856782197952,
+            -0.0353776253759861,
+            0.021313315257430077,
+            -0.00839521549642086,
+            -0.0039207362569868565,
+            -0.013986863195896149,
+            -0.000537770101800561,
+            -0.010524998418986797,
+            0.027509044855833054,
+            0.028345469385385513,
+            0.022490503266453743,
+            -0.05384090170264244,
+            0.009479468688368797,
+            0.00023766436788719147,
+            -0.05817791447043419,
+            0.0009709871956147254,
+            -0.007117345929145813,
+            0.011191039346158504,
+            -0.020213572308421135,
+            -0.02414786070585251,
+            0.011500826105475426,
+            0.03392162546515465,
+            0.014304393902420998,
+            0.0321558453142643,
+            -0.013367289677262306,
+            -0.09250226616859436,
+            0.018292896449565887,
+            -0.008689512498676777,
+            0.04126356914639473,
+            0.00865078903734684,
+            -0.004278926644474268,
+            -0.015845581889152527,
+            -0.00865078903734684,
+            -0.006943091284483671,
+            0.01283290795981884,
+            0.004530628211796284,
+            -0.005626498255878687,
+            0.0016631665639579296,
+            0.013708055019378662,
+            -0.005967263597995043,
+            0.007481344975531101,
+            0.003651608945801854,
+            -0.013189162127673626,
+            -0.06059424951672554,
+            -0.04055105894804001,
+            -0.018540725111961365,
+            0.050835974514484406,
+            -0.0069159846752882,
+            -0.046684831380844116,
+            -0.010850274004042149,
+            0.05058814212679863,
+            0.012259802781045437,
+            -0.0334569476544857,
+            -0.0007841471815481782,
+            0.0050882440991699696,
+            0.015458349138498306,
+            0.004546117503196001,
+            -0.027292195707559586,
+            0.02820606529712677,
+            0.0008412640891037881,
+            0.02300165221095085,
+            -0.018339363858103752,
+            -0.005312839522957802,
+            0.035780347883701324,
+            -0.024844881147146225,
+            0.023791607469320297,
+            0.0055219451896846294,
+            0.031629208475351334,
+            0.002952652983367443,
+            0.010060317814350128,
+            0.004704883322119713,
+            0.0186491496860981,
+            0.004275054205209017,
+            0.0276639387011528,
+            0.011462102644145489,
+            0.006428070832043886,
+            0.021623101085424423,
+            -0.009378788061439991,
+            -0.014513500034809113,
+            0.01885051093995571,
+            -0.0014695499558001757,
+            0.025232113897800446,
+            -0.020538847893476486,
+            0.002726121572777629,
+            -0.0016883367206901312,
+            -0.017967619001865387,
+            -0.040922801941633224,
+            -0.0007405834621749818,
+            0.021747015416622162,
+            -0.011671207845211029,
+            -0.0228312686085701,
+            -0.013715799897909164,
+            -0.01585332676768303,
+            0.0038490979932248592,
+            0.034169457852840424,
+            -0.0398695282638073,
+            0.00963436160236597,
+            -0.000830615172162652,
+            -0.03125746175646782,
+            -0.0239619892090559,
+            -0.01093546487390995,
+            0.014614180661737919,
+            -0.01652711257338524,
+            -0.021173911169171333,
+            -0.028469383716583252,
+            -0.008410705253481865,
+            0.006048582494258881,
+            -0.05065010115504265,
+            -0.004054331686347723,
+            0.019965743646025658,
+            0.0276639387011528,
+            -0.003099801717326045,
+            -0.02177799493074417,
+            0.0012468908680602908,
+            0.022351099178195,
+            0.02936776541173458,
+            0.014226946979761124,
+            0.027586491778492928,
+            -0.002178186783567071,
+            -0.0011675080750137568,
+            -0.0073148347437381744,
+            0.01307299267500639,
+            0.0011287847300991416,
+            0.036616768687963486,
+            0.0122210793197155,
+            -0.020306508988142014,
+            0.0013959755888208747,
+            -0.006431943271309137,
+            0.029181893914937973,
+            -0.02270735427737236,
+            -9.832092473516241e-05,
+            0.03531566634774208,
+            0.027741385623812675,
+            0.004933350719511509,
+            -0.0143121387809515,
+            -0.05142456665635109,
+            0.021592123433947563,
+            -0.01765783317387104,
+            0.031025122851133347,
+            0.016666516661643982,
+            0.003397971158847213,
+            -0.0034212053287774324,
+            0.012368228286504745,
+            -0.022087780758738518,
+            -0.004406713880598545,
+            -0.021142931655049324,
+            0.04389675334095955,
+            -0.043184246867895126,
+            0.008472662419080734,
+            0.029445212334394455,
+            0.011926782317459583,
+            -0.019082851707935333,
+            -0.005425137002021074,
+            0.029212871566414833,
+            -0.00162637943867594,
+            0.01308848150074482,
+            0.0058665829710662365,
+            -0.026935940608382225,
+            0.01202746294438839,
+            -0.031660184264183044,
+            0.02177799493074417,
+            -0.013436991721391678,
+            0.018447788432240486,
+            0.03342596814036369,
+            -0.0030397805385291576,
+            -0.0038239278364926577,
+            -0.016263794153928757,
+            -0.00969631876796484,
+            0.0662323608994484,
+            0.01054823212325573,
+            0.03726732358336449,
+            0.0018974426202476025,
+            -0.009773765690624714,
+            0.007969259284436703,
+            0.0063041565008461475,
+            -0.012329504825174809,
+            0.010494019836187363,
+            0.024178840219974518,
+            0.0012730291346088052,
+            -0.002412462839856744,
+            -0.019408127292990685,
+            -0.03438630700111389,
+            0.003727119415998459,
+            -0.030932186171412468,
+            0.022691864520311356,
+            0.009208405390381813,
+            0.017456471920013428,
+            0.008240322582423687,
+            -0.015907539054751396,
+            0.03200095146894455,
+            0.05461536720395088,
+            0.0019768255297094584,
+            0.013375034555792809,
+            0.007938280701637268,
+            0.007036027032881975,
+            -0.011338187381625175,
+            0.04702559858560562,
+            0.000662652775645256,
+            0.005169562995433807,
+            0.022536970674991608,
+            0.02145271934568882,
+            0.022939693182706833,
+            0.002304037567228079,
+            -0.011694442480802536,
+            0.010478530079126358,
+            -0.0006302220281213522,
+            0.004445436876267195,
+            -0.010695381090044975,
+            -0.027865299955010414,
+            0.007492962293326855,
+            0.01302652433514595,
+            0.022598929703235626,
+            0.021592123433947563,
+            -0.03178409859538078,
+            -0.01855621486902237,
+            0.03364282101392746,
+            -0.02997184917330742,
+            0.03612111136317253,
+            -0.0014424435794353485,
+            -0.023807097226381302,
+            -0.0058356039226055145,
+            0.016929835081100464,
+            0.02656419761478901,
+            -0.029181893914937973,
+            0.01804506592452526,
+            0.029693040996789932,
+            -0.0074426219798624516,
+            0.0005939188995398581,
+            0.012112653814256191,
+            0.02447313815355301,
+            -0.009293596260249615,
+            -0.0024027819745242596,
+            -0.0013320822035893798,
+            -0.011926782317459583,
+            -0.018370341509580612,
+            -0.0044725434854626656,
+            -0.022629907354712486,
+            0.005564541090279818,
+            -0.007365175057202578,
+            -0.003481226274743676,
+            -0.008581086993217468,
+            0.01154729351401329,
+            0.001979729626327753,
+            -0.019857317209243774,
+            -0.022335609421133995,
+            0.01652711257338524,
+            0.03887821361422539,
+            -0.018215449526906013,
+            -0.02106548473238945,
+            0.024442158639431,
+            -0.01759587600827217,
+            -0.01601596549153328,
+            0.036678727716207504,
+            0.013739033602178097,
+            0.007736918982118368,
+            -0.03358086198568344,
+            -0.030049296095967293,
+            -0.03274443745613098,
+            0.006296411622315645,
+            0.03785591572523117,
+            -0.0028752065263688564,
+            0.02569679357111454,
+            0.03701949119567871,
+            0.0057891360484063625,
+            -0.028934063389897346,
+            -0.014823286794126034,
+            -0.045104920864105225,
+            0.060160547494888306,
+            0.0011481463443487883,
+            0.05161044001579285,
+            0.016294771805405617,
+            -0.01456771232187748,
+            0.00419760774821043,
+            -0.013251119293272495,
+            0.026192452758550644,
+            0.02280029095709324,
+            0.025929134339094162,
+            0.006582964211702347,
+            0.022505993023514748,
+            -0.0037522895727306604,
+            -0.012197845615446568,
+            0.007307090330868959,
+            0.014536733739078045,
+            0.0002497654058970511,
+            -0.011392400600016117,
+            0.015783624723553658,
+            -0.01307299267500639,
+            -0.011361422017216682,
+            0.023295948281884193,
+            0.01368482131510973,
+            -0.012344993650913239,
+            0.014064309187233448,
+            -0.008565598167479038,
+            0.01239920686930418,
+            -0.023915521800518036,
+            0.022691864520311356,
+            -0.03262052312493324,
+            0.009587894193828106,
+            0.0015053689712658525,
+            -0.0073148347437381744,
+            0.024550583213567734,
+            -0.0027454833034425974,
+            -0.05071205645799637,
+            0.03550153970718384,
+            -0.008015726692974567,
+            0.0037213109899312258,
+            0.006381602957844734,
+            0.011392400600016117,
+            -0.0012623801594600081,
+            0.03807276859879494,
+            0.01239920686930418,
+            0.015969496220350266,
+            0.02199484594166279,
+            0.011911292560398579,
+            -0.008433938957750797,
+            -0.004096927121281624,
+            -0.0016137943603098392,
+            -0.01784370467066765,
+            0.025944624096155167,
+            0.0018674320308491588,
+            0.03962169960141182,
+            -0.018168980255723,
+            -0.02300165221095085,
+            0.028066661208868027,
+            0.007597515359520912,
+            -0.02351279929280281,
+            -0.0017377089243382215,
+            0.01202746294438839,
+            -0.009185170754790306,
+            0.027385130524635315,
+            0.01196550577878952,
+            0.019113829359412193,
+            -0.018060555681586266,
+            -0.004340883810073137,
+            -0.005072754807770252,
+            -0.015938518568873405,
+            0.02238207869231701,
+            0.020771188661456108,
+            0.011152315884828568,
+            -0.011880313977599144,
+            -0.01307299267500639,
+            0.020600805059075356,
+            0.035036858171224594,
+            -0.0002744515077210963,
+            0.005672966130077839,
+            0.017487451434135437,
+            0.002373739378526807,
+            -0.04470219835639,
+            0.016232814639806747,
+            -0.014025585725903511,
+            0.011214273050427437,
+            0.019593998789787292,
+            -0.021917399019002914,
+            -0.03196997195482254,
+            -0.011539548635482788,
+            -0.0414804182946682,
+            0.00493722315877676,
+            -0.0069159846752882,
+            0.013576395809650421,
+            -0.010176488198339939,
+            -0.0014831031439825892,
+            0.016666516661643982,
+            0.0038103745318949223,
+            0.02177799493074417,
+            -0.011678952723741531,
+            -0.012073930352926254,
+            -0.020213572308421135,
+            -0.020786678418517113,
+            0.004278926644474268,
+            -0.030405549332499504,
+            -0.004488032776862383,
+            0.021375272423028946,
+            -0.04783104360103607,
+            -0.03689557686448097,
+            0.02898053266108036,
+            -0.008503641001880169,
+            0.020755698904395103,
+            0.022955182939767838,
+            -0.015094350092113018,
+            -0.0008717587334103882,
+            -0.018695618957281113,
+            0.02258343994617462,
+            -0.007837600074708462,
+            0.026300877332687378,
+            -0.0027551641687750816,
+            0.0006292539183050394,
+            -0.02659517526626587,
+            0.020306508988142014,
+            0.005227648187428713,
+            0.012275291606783867,
+            -0.02390003204345703,
+            0.006815304048359394,
+            -0.03133491054177284,
+            -0.01784370467066765,
+            0.0180140882730484,
+            -0.011152315884828568,
+            0.027989214286208153,
+            -0.006060199346393347,
+            0.028438406065106392,
+            0.022722844034433365,
+            0.024751944467425346,
+            0.02676555886864662,
+            -0.02363671362400055,
+            0.008495896123349667,
+            0.03726732358336449,
+            0.00865078903734684,
+            -0.015876561403274536,
+            0.008170620538294315,
+            -0.04863648861646652,
+            -0.008062195032835007,
+            0.05030933395028114,
+            0.002127846470102668,
+            0.023141054436564445,
+            0.016294771805405617,
+            -0.002873270306736231,
+            -0.005688455421477556,
+            0.010269423946738243,
+            -0.04296739399433136,
+            -0.023172033950686455,
+            0.022629907354712486,
+            -0.021561143919825554,
+            6.54212328754511e-07,
+            0.03122648410499096,
+            0.012120398692786694,
+            -0.0009109660750254989,
+            -0.05108380317687988,
+            0.02100352756679058,
+            0.01178737822920084,
+            0.014443797990679741,
+            -0.010300402529537678,
+            0.026068538427352905,
+            0.01733255758881569,
+            0.038413532078266144,
+            0.012755461037158966,
+            -0.002679653698578477,
+            -0.03782493621110916,
+            0.009177426807582378,
+            -0.021685058251023293,
+            0.0011471783509477973,
+            0.0074503663927316666,
+            -0.02273833192884922,
+            -0.024782923981547356,
+            0.0043912241235375404,
+            0.004588713403791189,
+            0.011880313977599144,
+            0.022087780758738518,
+            0.009417511522769928,
+            -0.021204888820648193,
+            0.015404135920107365,
+            -0.05446047708392143,
+            -0.015450604259967804,
+            -0.023884542286396027,
+            0.02566581591963768,
+            0.020012211054563522,
+            0.010184233076870441,
+            0.005545179359614849,
+            0.010904486291110516,
+            -0.020988039672374725,
+            -0.009239383973181248,
+            0.020631784573197365,
+            0.011074868962168694,
+            -0.012840652838349342,
+            0.04113965481519699,
+            -0.00011126903700642288,
+            0.031210994347929955,
+            -0.02280029095709324,
+            0.002941036131232977,
+            -0.001241082325577736,
+            -0.03649285435676575,
+            0.009130958467721939,
+            0.01981084980070591,
+            0.02151467651128769,
+            -0.01861817203462124,
+            0.0024143988266587257,
+            0.01048627495765686,
+            0.004766840487718582,
+            0.038351576775312424,
+            0.027121812105178833,
+            -0.044206541031599045,
+            0.011663463898003101,
+            0.012995545752346516,
+            -0.009363298304378986,
+            0.048419639468193054,
+            -0.017518429085612297,
+            0.017673322930932045,
+            0.008240322582423687,
+            -0.01798310875892639,
+            0.02318752370774746,
+            -0.01469937153160572,
+            -0.013282098807394505,
+            0.00650551775470376,
+            0.009192915633320808,
+            -0.0001770381786627695,
+            -0.037577107548713684,
+            -0.017859194427728653,
+            -0.005719434004276991,
+            0.0038297362625598907,
+            0.01562873087823391,
+            0.013498948886990547,
+            0.009967382065951824,
+            0.005161818582564592,
+            -0.00945623405277729,
+            -0.0016089539276435971,
+            0.008124152198433876,
+            -0.0005532593932002783,
+            0.0019216446671634912,
+            -0.01926872320473194,
+            0.02698240801692009,
+            0.02572777308523655,
+            -0.0018354853382334113,
+            -0.04652994126081467,
+            -0.0036709706764668226,
+            0.0037077576853334904,
+            -0.017735280096530914,
+            -0.0035315665882080793,
+            -0.04519785940647125,
+            -0.01810702309012413,
+            0.0143121387809515,
+            -0.014614180661737919,
+            -0.0460033044219017,
+            0.009076746180653572,
+            -0.013886182568967342,
+            0.0063970922492444515,
+            -0.00932457484304905,
+            0.005762029904872179,
+            0.003651608945801854,
+            0.005680711008608341,
+            -0.005909178406000137,
+            0.05108380317687988,
+            -0.03466511517763138,
+            0.028051171451807022,
+            0.008565598167479038,
+            -0.012437930330634117,
+            -0.02261441759765148,
+            0.00887538492679596,
+            -0.044981006532907486,
+            -0.033209118992090225,
+            0.016743963584303856,
+            -0.01781272701919079,
+            0.011074868962168694,
+            0.06161654368042946,
+            0.007330324035137892,
+            -0.023342415690422058,
+            -0.01233724970370531,
+            -0.021049996837973595,
+            -0.04191412031650543,
+            0.003442503046244383,
+            0.005463860463351011,
+            -0.004948840010911226,
+            -0.03807276859879494,
+            -0.021716037765145302,
+            0.04615819454193115,
+            -0.012004229240119457,
+            0.012422440573573112,
+            0.01080380566418171,
+            -0.0019410063978284597,
+            -0.04860550910234451,
+            -0.021019017323851585,
+            0.0395597442984581,
+            0.021886419504880905,
+            -0.008426194079220295,
+            0.0028364830650389194,
+            -0.018277406692504883,
+            -0.003289545886218548,
+            0.006296411622315645,
+            -0.009107724763453007,
+            -0.03562545403838158,
+            0.02067825198173523,
+            0.0054832217283546925,
+            -0.023373395204544067,
+            -0.025805220007896423,
+            -0.03296129032969475,
+            0.009944148361682892,
+            0.02453509531915188,
+            -0.01968693546950817,
+            0.005823987070471048,
+            0.00027203132049180567,
+            -0.000881923595443368,
+            0.00042740863864310086,
+            -0.009959637187421322,
+            -0.019764382392168045,
+            -0.020151615142822266,
+            0.03268248215317726,
+            -0.0060059865936636925,
+            0.0074387495405972,
+            -0.03630698472261429,
+            -0.010974188335239887,
+            0.011322698555886745,
+            -0.002395037328824401,
+            -0.013452480547130108,
+            -0.009092235006392002,
+            -0.01739451475441456,
+            -0.009882191196084023,
+            -0.007876322604715824,
+            0.022660886868834496,
+            0.006168624851852655,
+            0.005409647710621357,
+            0.0023795480374246836,
+            -0.008581086993217468,
+            0.02898053266108036,
+            0.017874684184789658,
+            -0.014188224449753761,
+            -0.00920066051185131,
+            0.03451022133231163,
+            0.02665713243186474,
+            -0.009115469641983509,
+            0.01798310875892639,
+            0.023264970630407333,
+            -0.0010484338272362947,
+            -0.01196550577878952,
+            0.001922612776979804,
+            -0.0009550138493068516,
+            -0.0032972905319184065,
+            0.00845717266201973,
+            -0.015365412458777428,
+            0.0315207801759243,
+            0.04287445917725563,
+            -0.002145271748304367,
+            0.009587894193828106,
+            -0.003727119415998459,
+            -0.008472662419080734,
+            0.017069239169359207,
+            0.009316830895841122,
+            0.04625113308429718,
+            0.01098193321377039,
+            0.005680711008608341,
+            -0.033983584493398666,
+            -0.001229465357027948,
+            -0.01184933539479971,
+            0.03308520466089249,
+            0.02061629481613636,
+            0.00889087375253439,
+            -0.008379725739359856,
+            0.01344473659992218,
+            0.008341003209352493,
+            0.00661007035523653,
+            0.020569827407598495,
+            -0.016542602330446243,
+            -0.017286090180277824,
+            0.030467506498098373,
+            -0.00672236829996109,
+            0.01289486512541771,
+            0.02029101923108101,
+            -0.002089123008772731,
+            0.010842529125511646,
+            0.07100307941436768,
+            -0.006184114143252373,
+            0.015713922679424286,
+            -0.019082851707935333,
+            0.0015789433382451534,
+            -0.007640110794454813,
+            -0.03370477631688118,
+            0.018246427178382874,
+            0.0039962464943528175,
+            0.00865078903734684,
+            0.01198099460452795,
+            0.009053511545062065,
+            -0.007946024648845196,
+            -0.00963436160236597,
+            0.015195030719041824,
+            0.0019080914789810777,
+            0.00033568276558071375,
+            0.001336922636255622,
+            -0.0065210070461034775,
+            -0.004011735785752535,
+            -0.021948376670479774,
+            0.02763296104967594,
+            0.007140580099076033,
+            0.030606910586357117,
+            -0.055296897888183594,
+            0.030095763504505157,
+            -0.00789568480104208,
+            0.02554190158843994,
+            -0.016201836988329887,
+            -0.011090358719229698,
+            0.001609921921044588,
+            0.018153492361307144,
+            0.004491904750466347,
+            0.006462921854108572,
+            0.011888058856129646,
+            0.023233991116285324,
+            -0.03444826602935791,
+            0.017053749412298203,
+            0.023110076785087585,
+            0.0048675211146473885,
+            0.0003535923024173826,
+            0.005165690556168556,
+            0.015458349138498306,
+            0.007198664825409651,
+            -0.023048119619488716,
+            -0.0025692922063171864,
+            0.017611365765333176,
+            -0.02306360937654972,
+            0.0036980770528316498,
+            -0.006923729553818703,
+            0.004623564425855875,
+            -0.0014705180656164885,
+            -0.03509881719946861,
+            -0.00100099784322083,
+            -0.025743262842297554,
+            0.01923774555325508,
+            0.009556914679706097,
+            -0.025867177173495293,
+            0.003932353109121323,
+            0.025650326162576675,
+            -0.0003695656778290868,
+            0.00821708794683218,
+            0.0004683101433329284,
+            0.00869725737720728,
+            -0.00974278710782528,
+            -0.01493945624679327,
+            0.004801691509783268,
+            0.01531120017170906,
+            -0.01493945624679327,
+            -0.007431004662066698,
+            -0.009510447271168232,
+            0.030560443177819252,
+            0.0006674932083114982,
+            0.004189862869679928,
+            -0.005045648664236069,
+            -0.01858719252049923,
+            -0.0032237162813544273,
+            -0.021142931655049324,
+            0.004367990419268608,
+            -0.0009046735358424485,
+            0.04197607934474945,
+            0.0038161829579621553,
+            -0.02029101923108101,
+            0.0032972905319184065,
+            -0.03136589005589485,
+            0.0005000148667022586,
+            -0.007276111748069525,
+            -0.02004319056868553,
+            0.0015005286550149322,
+            0.0016525175888091326,
+            0.012631546705961227,
+            -0.03255856782197952,
+            -0.009301341138780117,
+            0.018416810780763626,
+            -0.00020680672605521977,
+            -0.010687636211514473,
+            -0.017828216776251793,
+            0.00035189816844649613,
+            -0.011338187381625175,
+            -0.027973726391792297,
+            -0.02850036323070526,
+            -0.0026719090528786182,
+            0.013491204008460045,
+            -0.018277406692504883,
+            -0.014993669465184212,
+            0.0031249718740582466,
+            0.021592123433947563,
+            -0.009781510569155216,
+            0.008302279748022556,
+            0.02241305634379387,
+            -0.008255811408162117,
+            -0.005514200776815414,
+            -0.000941460661124438,
+            -0.009130958467721939,
+            0.004267309792339802,
+            0.006300284061580896,
+            0.043339136987924576,
+            -0.032403673976659775,
+            -0.0018238683696836233,
+            0.014815541915595531,
+            0.0034153966698795557,
+            0.0025905899237841368,
+            0.002503462601453066,
+            -0.0007841471815481782,
+            0.01344473659992218,
+            0.01178737822920084,
+            0.015466093085706234,
+            0.00100099784322083,
+            -0.0033011629711836576,
+            0.006408709101378918,
+            -0.021173911169171333,
+            0.015721667557954788,
+            0.027958236634731293,
+            -0.0033999073784798384,
+            0.005525817628949881,
+            -0.013731288723647594,
+            -0.008759214542806149,
+            0.013134949840605259,
+            0.0026641644071787596,
+            -0.005068882368505001,
+            -0.009394276887178421,
+            0.017967619001865387,
+            0.019082851707935333,
+            0.006668155547231436,
+            -0.037453193217515945,
+            -0.02898053266108036,
+            -0.009959637187421322,
+            0.020569827407598495,
+            0.014916222542524338,
+            0.0010339126456528902,
+            -0.010540487244725227,
+            -0.009595638141036034,
+            -0.011903548613190651,
+            0.010989678092300892,
+            0.02721474878489971,
+            -0.032372694462537766,
+            0.0008214184199459851,
+            -0.029383255168795586,
+            0.03068435750901699,
+            0.0020000594668090343,
+            0.011446612887084484,
+            -0.0008185141487047076,
+            -0.015140817500650883,
+            -0.029181893914937973,
+            0.013824224472045898,
+            -0.004395096562802792,
+            0.001024231780320406,
+            -0.0031753121875226498,
+            0.0201051477342844,
+            0.012887120246887207,
+            -0.0014472840121015906,
+            0.01968693546950817,
+            0.008178364485502243,
+            0.016480645164847374,
+            -0.010153254494071007,
+            -0.04135650396347046,
+            0.019965743646025658,
+            0.007562664337456226,
+            0.014219203032553196,
+            0.004739734344184399,
+            0.004588713403791189,
+            -0.020771188661456108,
+            0.002724185585975647,
+            0.004333139397203922,
+            0.014405074529349804,
+            0.011028401553630829,
+            0.0025615475606173277,
+            0.011678952723741531,
+            -0.02228914201259613,
+            0.01929970271885395,
+            -0.01631026156246662,
+            0.01006806269288063,
+            0.004445436876267195,
+            -0.026873983442783356,
+            0.007547175046056509,
+            0.007822110317647457,
+            -0.0019187404541298747,
+            -0.017549408599734306,
+            0.02160761132836342,
+            -0.0034831624943763018,
+            -0.03509881719946861,
+            0.0360281765460968,
+            -0.0043447562493383884,
+            0.019346170127391815,
+            0.03209388628602028,
+            -0.01771979033946991,
+            -0.01993476413190365,
+            0.00906900130212307,
+            0.010408828034996986,
+            -0.002873270306736231,
+            0.017859194427728653,
+            -0.002919738180935383,
+            -0.02232012152671814,
+            0.017766257748007774,
+            -0.028918573632836342,
+            -0.014234691858291626,
+            0.020337486639618874,
+            -0.03624502569437027,
+            -0.0021026763133704662,
+            0.03175312280654907,
+            -0.024070415645837784,
+            0.005734923295676708,
+            0.005196669604629278,
+            -0.0019351978553459048,
+            0.0012236569309607148,
+            -0.006451305001974106,
+            0.03801080957055092,
+            -0.010865762829780579,
+            0.0201051477342844,
+            0.01929970271885395,
+            -0.023388884961605072,
+            0.01977987214922905,
+            -0.001109423115849495,
+            0.01675945147871971,
+            -0.003940097521990538,
+            0.03466511517763138,
+            -0.006017603911459446,
+            -0.011330443434417248,
+            0.006431943271309137,
+            0.008488151244819164,
+            -0.009959637187421322,
+            0.008681767620146275,
+            -0.018122512847185135,
+            0.027152791619300842,
+            0.02091059274971485,
+            -0.0002691270783543587,
+            0.013800990767776966,
+            0.0004421719058882445,
+            -0.020833145827054977,
+            0.00789568480104208,
+            -0.027121812105178833,
+            -0.009967382065951824,
+            0.01646515540778637,
+            -0.005762029904872179,
+            0.007380664348602295,
+            -0.02064727433025837,
+            0.028004704043269157,
+            0.0077524082735180855,
+            -0.013591884635388851,
+            -0.0008824076503515244,
+            0.023760627955198288,
+            -0.005432881414890289,
+            0.009355553425848484,
+            0.02070923149585724,
+            0.015713922679424286,
+            0.013615119270980358,
+            0.016263794153928757,
+            0.03351890295743942,
+            0.005328328814357519,
+            0.005568413063883781,
+            0.003519949736073613,
+            0.016991792246699333,
+            -0.008557853288948536,
+            0.018029578030109406,
+            0.0008093173382803798,
+            -0.013576395809650421,
+            0.021499186754226685,
+            -0.0018684001406654716,
+            -0.013622863218188286,
+            -0.0040891822427511215,
+            -0.004337011836469173,
+            0.03271345794200897,
+            0.022041313350200653,
+            -0.004468671046197414,
+            -0.024442158639431,
+            -0.00394784240052104,
+            0.016682006418704987,
+            -0.003167567541822791,
+            0.011136826127767563,
+            0.0007952801533974707,
+            -0.01586107164621353,
+            -0.009595638141036034,
+            0.0063583687879145145,
+            0.010586955584585667,
+            -0.008921852335333824,
+            0.010346870869398117,
+            -0.0228312686085701,
+            -0.004011735785752535,
+            0.00046613195445388556,
+            -0.004844287410378456,
+            0.0053438181057572365,
+            -0.011314953677356243,
+            0.004948840010911226,
+            0.016062432900071144,
+            0.01026167906820774,
+            0.004832670092582703,
+            0.006989559158682823,
+            0.012066186405718327,
+            0.009161937050521374,
+            -0.01627928391098976,
+            0.00428667152300477,
+            -0.005072754807770252,
+            -0.02345084212720394,
+            0.005831731949001551,
+            0.00902253296226263,
+            -0.01122201792895794,
+            0.009386532939970493,
+            -0.010958699509501457,
+            0.03296129032969475,
+            0.01030814740806818,
+            -0.037391237914562225,
+            -0.0033379499800503254,
+            -0.0016244432190433145,
+            0.014885243959724903,
+            0.012298526242375374,
+            0.008550108410418034,
+            -0.017518429085612297,
+            -0.0047823297791182995,
+            -0.009835722856223583,
+            0.0012827098835259676,
+            -0.012306271120905876,
+            0.017456471920013428,
+            0.022567950189113617,
+            -0.036337960511446,
+            0.01968693546950817,
+            -0.010432061739265919,
+            -0.023652203381061554,
+            0.020476890727877617,
+            0.011500826105475426,
+            0.01942361705005169,
+            -0.009402021765708923,
+            0.018896980211138725,
+            -0.0001331114035565406,
+            0.025510922074317932,
+            -0.004228586331009865,
+            -0.022087780758738518,
+            -0.012499887496232986,
+            -0.03209388628602028,
+            -0.0051811798475682735,
+            0.006819176487624645,
+            -0.00045524101005867124,
+            0.013111716136336327,
+            0.014226946979761124,
+            -0.025712283328175545,
+            0.007295473478734493,
+            0.032589543610811234,
+            0.014637414366006851,
+            0.023311438038945198,
+            -0.011276230216026306,
+            -0.0010794125264510512,
+            -0.0026060794480144978,
+            -0.013770012184977531,
+            -0.03212486580014229,
+            -0.0017493258928880095,
+            -0.036585792899131775,
+            0.005680711008608341,
+            -0.016387708485126495,
+            -0.013096226379275322,
+            0.023264970630407333,
+            -0.034138478338718414,
+            0.0014608372002840042,
+            0.013708055019378662,
+            0.010540487244725227,
+            0.012391461990773678,
+            -0.00162637943867594,
+            0.015032391995191574,
+            -0.029243851080536842,
+            -0.010284913703799248,
+            -0.004491904750466347,
+            -0.01981084980070591,
+            -0.007338068913668394,
+            -0.018060555681586266,
+            -0.013127204962074757,
+            -0.003566417610272765,
+            -0.010060317814350128,
+            0.002247888594865799,
+            0.0032062907703220844,
+            -0.014714861288666725,
+            0.03206290677189827,
+            0.003157886676490307,
+            0.018958937376737595,
+            0.0033418224193155766,
+            0.004449309315532446,
+            0.018788553774356842,
+            -0.0026331855915486813,
+            -0.0058278595097362995,
+            0.030963165685534477,
+            -0.00032672801171429455,
+            0.008759214542806149,
+            0.019082851707935333,
+            0.004724245052784681,
+            0.0038529701996594667,
+            -0.0260530486702919,
+            0.012887120246887207,
+            -0.027447087690234184,
+            0.005080499220639467,
+            -0.0077175572514534,
+            -0.024302754551172256,
+            -0.028128618374466896,
+            -0.006571347359567881,
+            0.012058441527187824,
+            -0.02898053266108036,
+            -0.00613377382978797,
+            0.002315654419362545,
+            -0.00400399137288332,
+            -0.002673845272511244,
+            0.007241260726004839,
+            0.029150914400815964,
+            -0.033209118992090225,
+            0.008751469664275646,
+            0.005595519673079252,
+            -0.0005053392960689962,
+            0.013297587633132935,
+            0.008038961328566074,
+            0.003343758638948202,
+            0.020848635584115982,
+            0.006637176964432001,
+            0.002458930714055896,
+            -0.0025576751213520765,
+            -0.014242436736822128,
+            0.004000118933618069,
+            0.02154565416276455,
+            0.007562664337456226,
+            0.018385831266641617,
+            -0.008914108388125896,
+            0.002621568739414215,
+            -0.01455222349613905,
+            0.005552923772484064,
+            -0.002867461647838354,
+            0.004309905227273703,
+            -0.012236569076776505,
+            0.008093173615634441,
+            0.0028771425131708384,
+            -0.0043447562493383884,
+            0.018060555681586266,
+            -0.002319526858627796,
+            0.016480645164847374,
+            0.004015608225017786,
+            0.015620986931025982,
+            -0.0035412474535405636,
+            0.01178737822920084,
+            0.0005731051205657423,
+            -0.017905661836266518,
+            -0.017022771760821342,
+            -0.039249956607818604,
+            0.007702067960053682,
+            0.03531566634774208,
+            -0.0009564659558236599,
+            -0.03444826602935791,
+            0.010145509615540504,
+            -0.011942271143198013,
+            -0.009975126944482327,
+            -0.022862248122692108,
+            0.013862947933375835,
+            0.012430185452103615,
+            0.007415515370666981,
+            0.012252057902514935,
+            -0.006997303571552038,
+            0.01026167906820774,
+            -0.005216030869632959,
+            0.008356492035090923,
+            -0.0010823167394846678,
+            -0.037422213703393936,
+            0.012344993650913239,
+            0.02216522768139839,
+            -0.002635121811181307,
+            0.0016089539276435971,
+            -0.006145390681922436,
+            0.022103270515799522,
+            -0.013971373438835144,
+            -0.00650551775470376,
+            -0.00046443779137916863,
+            0.00797700323164463,
+            -0.01795213110744953,
+            0.004778457339853048,
+            0.0755879208445549,
+            -0.007775642443448305,
+            0.006122156511992216,
+            0.009270362555980682,
+            -0.015148562379181385,
+            -0.01637221872806549,
+            -0.004561606794595718,
+            -0.020260039716959,
+            0.008844406343996525,
+            -0.0018819533288478851,
+            -0.0110981035977602,
+            0.012468908913433552,
+            0.0053050946444272995,
+            0.002031038049608469,
+            -0.008379725739359856,
+            0.010811550542712212,
+            0.009525936096906662,
+            0.010432061739265919,
+            0.02360573597252369,
+            0.011477591469883919,
+            -0.00858883187174797,
+            -0.011795123107731342,
+            0.01129171997308731,
+            0.0028577810153365135,
+            0.016697494313120842,
+            0.007907302118837833,
+            0.0010648912284523249,
+            0.019501063972711563,
+            -0.002309845993295312,
+            0.02180897258222103,
+            -2.2689444449497387e-05,
+            -0.009301341138780117,
+            -0.004236331209540367,
+            -0.015349923633038998,
+            0.024271776899695396,
+            -0.0020155487582087517,
+            0.0015450604259967804,
+            -0.020213572308421135,
+            0.0013979118084535003,
+            0.013243375346064568,
+            0.011957760900259018,
+            -0.0020407189149409533,
+            -0.0051347119733691216,
+            0.0007154133054427803,
+            -0.016356728971004486,
+            0.032806396484375,
+            0.007578153628855944,
+            0.0007681738352403045,
+            0.0008838597568683326,
+            0.0009288755827583373,
+            -0.0032972905319184065,
+            0.0024608669336885214,
+            -0.004766840487718582,
+            0.009549170732498169,
+            -0.010943209752440453,
+            -0.010796061716973782,
+            0.010075807571411133,
+            -0.02402394637465477,
+            0.003566417610272765,
+            -0.00558777479454875,
+            -0.008062195032835007,
+            0.0076207490637898445,
+            0.00015537731815129519,
+            -0.007392281666398048,
+            -0.0030939930584281683,
+            0.0023408245760947466,
+            0.013429246842861176,
+            0.004406713880598545,
+            0.010850274004042149,
+            0.012956822291016579,
+            0.005359307397156954,
+            -0.006629432085901499,
+            0.015512561425566673,
+            -0.006946963258087635,
+            -0.004228586331009865,
+            -0.0012468908680602908,
+            -0.019996721297502518,
+            -0.0009419447160325944,
+            -0.004708755761384964,
+            0.0015092412941157818,
+            0.0001705036120256409,
+            -8.749050175538287e-05,
+            -0.0018490385264158249,
+            -0.011988739483058453,
+            -0.001831613015383482,
+            -0.011663463898003101,
+            -0.014498010277748108,
+            0.010253935120999813,
+            -0.0041201612912118435,
+            -0.022939693182706833,
+            -0.015349923633038998,
+            -0.020662762224674225,
+            0.02145271934568882,
+            -0.004387352149933577,
+            -0.005800753366202116,
+            0.006733985152095556,
+            -0.010215211659669876,
+            -0.015706177800893784,
+            -0.008069939911365509,
+            -0.0205853171646595,
+            0.010703125037252903,
+            0.003214035416021943,
+            0.0006791101768612862,
+            0.00882891658693552,
+            0.014397329650819302,
+            -0.01589204929769039,
+            0.0111755495890975,
+            0.012182355858385563,
+            0.011361422017216682,
+            -0.006854027509689331,
+            0.0008911203476600349,
+            -0.014103032648563385,
+            0.02596011385321617,
+            0.021623101085424423,
+            0.005169562995433807,
+            -0.04132552444934845,
+            -0.005862710531800985,
+            -0.013344055972993374,
+            -0.010524998418986797,
+            -0.012120398692786694,
+            0.011771888472139835,
+            -0.0038820127956569195,
+            0.0013117524795234203,
+            0.006288666743785143,
+            0.0015063370810821652,
+            0.004367990419268608,
+            -0.0022382077295333147,
+            -0.007709812838584185,
+            -0.03212486580014229,
+            -0.02817508764564991,
+            -0.00863530021160841,
+            0.020461400970816612,
+            0.002838419284671545,
+            -0.0009453330421820283,
+            -0.0036380558740347624,
+            0.0027048238553106785,
+            -0.005882072262465954,
+            -0.005018542055040598,
+            -0.014753584749996662,
+            -0.012739972211420536,
+            0.0101300198584795,
+            0.00802347157150507,
+            0.014195968396961689,
+            -0.011950016021728516,
+            0.0028190575540065765,
+            0.018788553774356842,
+            0.007690451107919216,
+            -0.023357905447483063,
+            0.01135367713868618,
+            -0.010912231169641018,
+            -0.009270362555980682,
+            -0.009433000348508358,
+            -0.009471723809838295,
+            0.0016980175860226154,
+            -0.008317768573760986,
+            0.003945906180888414,
+            -0.03615209087729454,
+            -0.011330443434417248,
+            -0.0013901671627536416,
+            -0.010346870869398117,
+            0.0009864765452221036,
+            0.02524760365486145,
+            0.006815304048359394,
+            0.01432762760668993,
+            -0.0006733016925863922,
+            0.017502939328551292,
+            -0.0018984107300639153,
+            0.018726596608757973,
+            0.012763205915689468,
+            -0.00797700323164463,
+            0.011717676185071468,
+            0.014738094992935658,
+            -0.0033863543067127466,
+            0.015620986931025982,
+            -0.010168743319809437,
+            0.005285732913762331,
+            -0.009556914679706097,
+            -0.003214035416021943,
+            0.002495717955753207,
+            0.030901208519935608,
+            -0.003618694143369794,
+            0.00858883187174797,
+            0.01098193321377039,
+            -0.015241498127579689,
+            0.0026951429899781942,
+            0.014962689951062202,
+            0.01141563430428505,
+            0.007760153152048588,
+            0.0024434414226561785,
+            -0.024225307628512383,
+            0.004193735308945179,
+            0.005037903785705566,
+            0.0031520782504230738,
+            -0.014366351068019867,
+            0.0021859314292669296,
+            0.015675200149416924,
+            0.008550108410418034,
+            -0.025325050577521324,
+            0.016418687999248505,
+            0.0009138703462667763,
+            -0.003909118939191103,
+            0.029119934886693954,
+            -0.011268485337495804,
+            -0.005723306443542242,
+            0.009463978931307793,
+            0.0009980935137718916,
+            0.0030029932968318462,
+            0.008937342092394829,
+            0.0228312686085701,
+            -0.003988501615822315,
+            -0.032341714948415756,
+            -0.022459525614976883,
+            0.015706177800893784,
+            -0.0047823297791182995,
+            -0.022939693182706833,
+            -0.026703599840402603,
+            -0.003113354789093137,
+            -0.02228914201259613,
+            -0.0022323993034660816,
+            -0.020058678463101387,
+            -0.014056564308702946,
+            0.001268188701942563,
+            0.029987338930368423,
+            0.016201836988329887,
+            0.008224832825362682,
+            -0.011051635257899761,
+            -0.029956359416246414,
+            -0.024271776899695396,
+            0.01810702309012413,
+            -0.007055388763546944,
+            -0.005541306920349598,
+            0.00591692328453064,
+            -0.009951893240213394,
+            -0.008480406366288662,
+            0.011802867986261845,
+            0.027137301862239838,
+            -0.002747419523075223,
+            -0.029987338930368423,
+            -0.02688947319984436,
+            -0.014335372485220432,
+            -0.01840132102370262,
+            -0.017704300582408905,
+            -0.008627555333077908,
+            0.0045035220682621,
+            0.00030300996149890125,
+            0.016186347231268883,
+            0.014962689951062202,
+            -0.000480653194244951,
+            0.0028597170021384954,
+            -2.7711375878425315e-05,
+            -0.001763847190886736,
+            0.006888878531754017,
+            -0.004933350719511509,
+            0.026997897773981094,
+            0.014614180661737919,
+            -0.006594581063836813,
+            0.018711106851696968,
+            0.028314489871263504,
+            0.020430423319339752,
+            0.003157886676490307,
+            -0.010695381090044975,
+            -0.014606435783207417,
+            0.009727297350764275,
+            -0.013421501964330673,
+            0.023264970630407333,
+            0.022552460432052612,
+            -0.01480779703706503,
+            -0.014885243959724903,
+            -0.017905661836266518,
+            0.015210519544780254,
+            -0.007562664337456226,
+            -0.008224832825362682,
+            0.012368228286504745,
+            0.022335609421133995,
+            -0.007024410180747509,
+            0.002371803391724825,
+            -0.0010232636705040932,
+            -0.021731525659561157,
+            -0.006122156511992216,
+            -0.004596457816660404,
+            -0.008666278794407845,
+            0.008681767620146275,
+            0.008131897076964378,
+            -0.001928421319462359,
+            -0.006319645792245865,
+            0.005471604876220226,
+            -0.013367289677262306,
+            0.01301103550940752,
+            -0.00789568480104208,
+            0.03379771113395691,
+            -0.009138703346252441,
+            -0.0006733016925863922,
+            -0.022598929703235626,
+            -0.011446612887084484,
+            0.00845717266201973,
+            0.0017967619933187962,
+            0.01682141050696373,
+            -0.0011626676423475146,
+            -0.005688455421477556,
+            0.009913169778883457,
+            -0.0010687635513022542,
+            -0.001450188341550529,
+            -0.006892750971019268,
+            -0.010401083156466484,
+            0.0002884887217078358,
+            0.013258864171802998,
+            -0.006025348324328661,
+            0.0018054747488349676,
+            0.006095050368458033,
+            -0.0175339188426733,
+            0.029027000069618225,
+            0.023884542286396027,
+            0.017255110666155815,
+            0.014660648070275784,
+            0.0006394187803380191,
+            0.011647974140942097,
+            -0.003572226269170642,
+            0.0038820127956569195,
+            -0.020337486639618874,
+            0.021080974489450455,
+            -0.010811550542712212,
+            0.01846327818930149,
+            0.011655719019472599,
+            -0.015233753249049187,
+            -0.005022414494305849,
+            0.021406250074505806,
+            0.009967382065951824,
+            0.00889087375253439,
+            0.0017541663255542517,
+            -0.02470547705888748,
+            0.017766257748007774,
+            0.010099041275680065,
+            -0.020399443805217743,
+            0.019547531381249428,
+            -0.009471723809838295,
+            0.01233724970370531,
+            -0.0005300253978930414,
+            0.004418330732733011,
+            0.0058743273839354515,
+            0.021499186754226685,
+            0.011570527218282223,
+            -0.03884723410010338,
+            -0.013615119270980358,
+            -0.0017357728211209178,
+            0.006857899948954582,
+            -0.0036303112283349037,
+            -0.015233753249049187,
+            -0.041635312139987946,
+            0.01604694314301014,
+            0.009998360648751259,
+            -0.01643417589366436,
+            0.004987563472241163,
+            -0.002427952131256461,
+            -0.0042053526267409325,
+            -0.008170620538294315,
+            0.003186929039657116,
+            -0.004170501604676247,
+            0.02324948087334633,
+            0.0018984107300639153,
+            -0.006946963258087635,
+            0.013258864171802998,
+            0.01344473659992218,
+            -0.021948376670479774,
+            0.0009874446550384164,
+            0.0021375271026045084,
+            0.000636998622212559,
+            0.023683181032538414,
+            -0.014707116410136223,
+            -0.0017657832941040397,
+            -0.01685238815844059,
+            -0.0013069120468571782,
+            -0.021592123433947563,
+            -0.001188805908896029,
+            -0.015187285840511322,
+            -0.008511385880410671,
+            0.025030752643942833,
+            -0.011005166918039322,
+            0.010997422970831394,
+            -0.006687517277896404,
+            0.006943091284483671,
+            -0.006904367823153734,
+            -0.01936165988445282,
+            -0.0008267428493127227,
+            0.009975126944482327,
+            -0.000512115890160203,
+            0.002941036131232977,
+            8.482827252009884e-05,
+            0.002464739140123129,
+            0.008867640048265457,
+            0.006234454456716776,
+            -0.0075278133153915405,
+            0.02489134855568409,
+            -0.012073930352926254,
+            0.0016399325104430318,
+            0.011911292560398579,
+            0.0004743606550619006,
+            -0.010424317792057991,
+            0.025588368996977806,
+            -0.02139076218008995,
+            -0.00800798274576664,
+            -0.00014799568452872336,
+            -0.0016544538084417582,
+            -0.006513262167572975,
+            0.006718495860695839,
+            0.02946070022881031,
+            0.0062654330395162106,
+            0.010447551496326923,
+            -0.0281441081315279,
+            -0.011493081226944923,
+            0.012538610957562923,
+            -0.022273652255535126,
+            -0.0007739823195151985,
+            -0.0004322490422055125,
+            -0.00619960343465209,
+            0.026378324255347252,
+            -0.02022906206548214,
+            -0.01093546487390995,
+            -0.018664639443159103,
+            0.005405775271356106,
+            -0.011121337302029133,
+            0.008294534869492054,
+            -0.0009777637897059321,
+            0.006540368776768446,
+            0.016805920749902725,
+            -0.019563021138310432,
+            0.016031453385949135,
+            0.005119222681969404,
+            -0.016217326745390892,
+            0.010222956538200378,
+            -0.017580386251211166,
+            -0.0008199662552215159,
+            0.020275529474020004,
+            -0.00400399137288332,
+            0.008751469664275646,
+            -0.003363120136782527,
+            0.005208286456763744,
+            -0.006513262167572975,
+            0.0001492057926952839,
+            -0.007249005138874054,
+            0.014428308233618736,
+            -0.016805920749902725,
+            -0.017828216776251793,
+            -0.008821171708405018,
+            0.0016709112096577883,
+            0.021297825500369072,
+            0.011136826127767563,
+            -0.010044828988611698,
+            -0.011059380136430264,
+            -0.011694442480802536,
+            0.041604332625865936,
+            0.005762029904872179,
+            0.027230238541960716,
+            -0.02261441759765148,
+            0.00545998802408576,
+            0.025170156732201576,
+            -0.014242436736822128,
+            0.0012788375606760383,
+            -0.0022788674104958773,
+            0.0053050946444272995,
+            0.011562783271074295,
+            -0.01819995976984501,
+            0.0049139889888465405,
+            -0.00019216447253711522,
+            -0.00782985519617796,
+            -0.000656844291370362,
+            0.003442503046244383,
+            0.00020196630794089288,
+            0.016325751319527626,
+            -0.004027225077152252,
+            -0.016511622816324234,
+            -0.004429947584867477,
+            -0.0003216455806978047,
+            -0.027896279469132423,
+            0.013514438644051552,
+            0.016589069738984108,
+            0.01566745527088642,
+            0.027989214286208153,
+            -0.014513500034809113,
+            0.016031453385949135,
+            0.009533680975437164,
+            0.013901671394705772,
+            -0.010610189288854599,
+            0.008952830918133259,
+            -0.01595400646328926,
+            0.007446494419127703,
+            -0.03454120084643364,
+            -0.007887939922511578,
+            -0.00802347157150507,
+            0.01984182931482792,
+            0.028763681650161743,
+            0.002923610620200634,
+            0.006493900436908007,
+            0.000196883876924403,
+            0.0024202074855566025,
+            -0.01714668609201908,
+            -0.020182594656944275,
+            -0.0003337466041557491,
+            0.0029332914855331182,
+            0.0006950835813768208,
+            -0.004495777189731598,
+            0.00472037261351943,
+            -0.012391461990773678,
+            0.0026835259050130844,
+            -0.013134949840605259,
+            0.005971135571599007,
+            -0.0059982421807944775,
+            0.009603383019566536,
+            0.0019429425010457635,
+            0.020864123478531837,
+            0.02300165221095085,
+            0.004100799560546875,
+            0.0046971384435892105,
+            -0.024333734065294266,
+            0.00920066051185131,
+            -0.013847459107637405,
+            0.012329504825174809,
+            0.019408127292990685,
+            0.008248066529631615,
+            0.006571347359567881,
+            0.01233724970370531,
+            0.012918098829686642,
+            -0.0031772484071552753,
+            0.024287264794111252,
+            0.005014669615775347,
+            -0.0009070937521755695,
+            0.002052335999906063,
+            -0.014954946003854275,
+            -0.0070979841984808445,
+            -0.0015286030247807503,
+            0.0018848575418815017,
+            -0.004731989465653896,
+            0.01906736195087433,
+            0.015256987884640694,
+            -0.01050176378339529,
+            0.017874684184789658,
+            -0.03937387093901634,
+            0.017379024997353554,
+            -0.0027377386577427387,
+            -0.0060911779291927814,
+            -0.00545998802408576,
+            0.00665653869509697,
+            0.004515138920396566,
+            0.003669034456834197,
+            0.03773200139403343,
+            -0.010408828034996986,
+            0.003963331691920757,
+            -0.005847221240401268,
+            0.007121218368411064,
+            -0.020864123478531837,
+            0.008302279748022556,
+            0.005336073227226734,
+            0.02160761132836342,
+            -0.0056497324258089066,
+            -0.0014482521219179034,
+            0.02328045852482319,
+            0.008859895169734955,
+            0.03469609469175339,
+            -0.01566745527088642,
+            -0.004418330732733011,
+            0.01048627495765686,
+            0.0207402091473341,
+            -0.00580462533980608,
+            -0.008976065553724766,
+            0.007903429679572582,
+            0.0035586729645729065,
+            0.021824462339282036,
+            -0.00515407370403409,
+            -0.0006650729919783771,
+            0.0054367538541555405,
+            -0.0060911779291927814,
+            0.009735042229294777,
+            0.011252996511757374,
+            -0.009076746180653572,
+            0.007539430167526007,
+            0.021530164405703545,
+            -0.009975126944482327,
+            -0.008658533915877342,
+            -0.009649851359426975,
+            0.0010658593382686377,
+            -0.03068435750901699,
+            0.001649613375775516,
+            0.005359307397156954,
+            0.015187285840511322,
+            -0.0004603234410751611,
+            -0.009014789015054703,
+            0.021019017323851585,
+            0.017255110666155815,
+            -0.005936284549534321,
+            -0.0028132491279393435,
+            0.0023795480374246836,
+            -0.0017154429806396365,
+            -0.006160879973322153,
+            -0.0006248975405469537,
+            -0.002656419761478901,
+            0.014730350114405155,
+            -0.0015469965292140841,
+            -0.006617815233767033,
+            -0.00756653631106019,
+            0.0033302053343504667,
+            0.010834784246981144,
+            -0.005661349277943373,
+            0.013057502917945385,
+            0.016805920749902725,
+            -0.00031704717548564076,
+            0.019671445712447166,
+            0.01037010457366705,
+            -0.010447551496326923,
+            0.015938518568873405,
+            -0.007984748110175133,
+            0.006563602481037378,
+            0.0164031982421875,
+            -0.01277095079421997,
+            -0.01246116403490305,
+            0.007210282143205404,
+            0.00665653869509697,
+            0.028918573632836342,
+            0.014792308211326599,
+            -0.02109646424651146,
+            0.014908477663993835,
+            0.015768134966492653,
+            -0.013018779456615448,
+            0.003537375247105956,
+            -0.01784370467066765,
+            0.020260039716959,
+            0.019051872193813324,
+            -0.015535795129835606,
+            -0.017224133014678955,
+            -0.014947201125323772,
+            -0.005092116538435221,
+            0.006331262644380331,
+            0.0033263331279158592,
+            -0.00617636926472187,
+            -0.027989214286208153,
+            -0.0049139889888465405,
+            -0.018633661791682243,
+            0.013870692811906338,
+            -0.008674023672938347,
+            0.0007715621031820774,
+            -0.004511266481131315,
+            0.007984748110175133,
+            -0.0033882902935147285,
+            0.032465629279613495,
+            -0.0051811798475682735,
+            -0.02832997962832451,
+            0.004398969002068043,
+            0.016356728971004486,
+            0.012623801827430725,
+            -0.029243851080536842,
+            0.010463040322065353,
+            0.004805563949048519,
+            -0.012530866079032421,
+            0.005990497302263975,
+            -0.02258343994617462,
+            -0.007632365915924311,
+            -0.0009395244996994734,
+            -0.016743963584303856,
+            -0.027121812105178833,
+            -0.008433938957750797,
+            -0.006993431597948074,
+            -0.005401902832090855,
+            -0.009525936096906662,
+            -0.008720491081476212,
+            -0.011477591469883919,
+            0.02303262986242771,
+            -0.007899557240307331,
+            -0.001866463921032846,
+            -0.0014637414133176208,
+            0.00930908601731062,
+            -0.023776117712259293,
+            -0.015605497173964977,
+            0.006389347370713949,
+            -0.002594462363049388,
+            0.022815778851509094,
+            -0.012747716158628464,
+            0.005940156988799572,
+            -0.009316830895841122,
+            0.009851212613284588,
+            0.015241498127579689,
+            0.004956584889441729,
+            0.017440982162952423,
+            -0.009889936074614525,
+            0.0021026763133704662,
+            -0.008627555333077908,
+            -0.00018986526993103325,
+            -1.027075541060185e-05,
+            0.014428308233618736,
+            0.007624621503055096,
+            0.014141756109893322,
+            0.017487451434135437,
+            -0.013553161174058914,
+            0.005003052763640881,
+            -0.0003274540649726987,
+            -0.017255110666155815,
+            0.0027725896798074245,
+            -0.007984748110175133,
+            -0.0008896682411432266,
+            0.001758038648404181,
+            6.395397940650582e-05,
+            -0.02112744189798832,
+            -0.0009124181815423071,
+            0.0018809852190315723,
+            0.0010222955606877804,
+            -0.025201136246323586,
+            0.00889087375253439,
+            -0.00889087375253439,
+            -0.0014918158994987607,
+            -0.019113829359412193,
+            0.019795360043644905,
+            0.026300877332687378,
+            -0.004619691986590624,
+            0.00036206303047947586,
+            -0.013537672348320484,
+            -0.02049238048493862,
+            0.0001489637652412057,
+            -0.011314953677356243,
+            -0.011345932260155678,
+            -0.0022169100120663643,
+            -0.003214035416021943,
+            0.008805682882666588,
+            0.0004189379105810076,
+            -0.009433000348508358,
+            -0.004956584889441729,
+            -0.011764144524931908,
+            -0.017549408599734306,
+            -0.010401083156466484,
+            0.01933068037033081,
+            -0.0018016024259850383,
+            0.0020794421434402466,
+            -0.0032508226577192545,
+            0.0005271211848594248,
+            0.011748654767870903,
+            0.013475715182721615,
+            0.013344055972993374,
+            -0.023915521800518036,
+            -0.017890173941850662,
+            -0.013320821337401867,
+            0.0029487807769328356,
+            -0.007187047973275185,
+            0.004844287410378456,
+            0.026362836360931396,
+            -0.0009695350890979171,
+            -0.01091997604817152,
+            0.002327271504327655,
+            0.019051872193813324,
+            -0.031272951513528824,
+            -0.013390523381531239,
+            -0.025077221915125847,
+            0.02225816436111927,
+            -0.013351799920201302,
+            0.011516314931213856,
+            0.010191977955400944,
+            -0.025944624096155167,
+            0.006768836174160242,
+            -0.007686578668653965,
+            0.026037558913230896,
+            0.01634124107658863,
+            0.0005934348446317017,
+            -0.00944074522703886,
+            0.013622863218188286,
+            -0.0008722427301108837,
+            -0.013576395809650421,
+            0.002201420720666647,
+            0.014645159244537354,
+            -0.014691627584397793,
+            -0.009510447271168232,
+            -0.004507394507527351,
+            0.004313777666538954,
+            0.012763205915689468,
+            -0.0030339721124619246,
+            -0.004031097516417503,
+            -0.011624740436673164,
+            0.013344055972993374,
+            -0.003930416889488697,
+            0.012383717112243176,
+            0.007380664348602295,
+            -0.014056564308702946,
+            -0.019532041624188423,
+            0.033333033323287964,
+            0.00400399137288332,
+            0.00338054564781487,
+            -0.010997422970831394,
+            0.02740062028169632,
+            -0.009897680021822453,
+            0.010571465827524662,
+            -0.006408709101378918,
+            0.0031985461246222258,
+            0.012321759946644306,
+            -0.012104909867048264,
+            -0.01981084980070591,
+            0.021468207240104675,
+            -0.00364192808046937,
+            -0.0010193913476541638,
+            0.008310024626553059,
+            -0.020182594656944275,
+            -0.00663330452516675,
+            0.002180122770369053,
+            0.009541425853967667,
+            -0.011818356812000275,
+            0.006369986105710268,
+            0.017905661836266518,
+            0.003912991378456354,
+            -0.0007672057254239917,
+            0.019996721297502518,
+            -0.01672847382724285,
+            0.011717676185071468,
+            -0.010850274004042149,
+            0.0016825281782075763,
+            0.0006065039779059589,
+            -0.0048597767017781734,
+            0.0058743273839354515,
+            -0.014536733739078045,
+            0.011988739483058453,
+            -6.244135147426277e-05,
+            -0.013979118317365646,
+            0.007601387333124876,
+            -0.013359544798731804,
+            -0.016836898401379585,
+            0.041542377322912216,
+            -0.005301222205162048,
+            -0.0038665232714265585,
+            -0.0056962003000080585,
+            0.01350669376552105,
+            0.02823704481124878,
+            -0.005336073227226734,
+            -0.01153180468827486,
+            -0.010780571959912777,
+            -0.016573579981923103,
+            0.015907539054751396,
+            -0.01810702309012413,
+            0.002495717955753207,
+            -0.010532742366194725,
+            -0.007903429679572582,
+            0.004302160814404488,
+            -0.003516077296808362,
+            0.014722606167197227,
+            0.00950270239263773,
+            0.023357905447483063,
+            0.00797700323164463,
+            -0.00027348342700861394,
+            0.010393339209258556,
+            0.011663463898003101,
+            0.003243078012019396,
+            -0.010509508661925793,
+            -0.00011949774489039555,
+            -0.02894955314695835,
+            -0.008449427783489227,
+            -0.004863648675382137,
+            -0.001485039247199893,
+            -0.00428667152300477,
+            -0.01239920686930418,
+            0.006617815233767033,
+            -0.0022052929271012545,
+            0.013909416273236275,
+            -0.009254872798919678,
+            -0.017766257748007774,
+            0.03389064967632294,
+            0.004495777189731598,
+            0.006466794293373823,
+            -0.008286789990961552,
+            -0.03705047070980072,
+            -0.0032566310837864876,
+            -0.008116407319903374,
+            -0.013529927469789982,
+            0.004538373090326786,
+            0.01283290795981884,
+            -0.014831030741333961,
+            -0.028097640722990036,
+            0.0019390701781958342,
+            -0.006683644838631153,
+            0.02978597767651081,
+            -0.007798876613378525,
+            0.00419760774821043,
+            0.0037251831963658333,
+            0.0011558911064639688,
+            0.01239920686930418,
+            -0.00845717266201973,
+            -0.0022536972537636757,
+            0.016387708485126495,
+            -0.010858018882572651,
+            0.011051635257899761,
+            -0.010966443456709385,
+            -0.02554190158843994,
+            -0.01456771232187748,
+            0.021344292908906937,
+            -0.0004823473282158375,
+            -0.00024903935263864696,
+            0.014823286794126034,
+            0.02354377694427967,
+            0.011260741390287876,
+            0.00969631876796484,
+            0.006408709101378918,
+            -0.001035848748870194,
+            0.005975008010864258,
+            0.006095050368458033,
+            -0.0024163350462913513,
+            -0.013398268260061741,
+            0.017564896494150162,
+            0.009773765690624714,
+            -0.01810702309012413,
+            0.0008514289511367679,
+            0.0036612898111343384,
+            0.0168833676725626,
+            0.006935346405953169,
+            0.009316830895841122,
+            -0.0143121387809515,
+            -0.023760627955198288,
+            -0.00014932679187040776,
+            0.002873270306736231,
+            0.015721667557954788,
+            0.010579210706055164,
+            -0.014319883659482002,
+            0.0005963391158729792,
+            0.008712747134268284,
+            0.020213572308421135,
+            -0.024209819734096527,
+            0.019702425226569176,
+            -0.004673904739320278,
+            -0.0021084847394376993,
+            0.007946024648845196,
+            0.011632485315203667,
+            -0.011872569099068642,
+            -0.00650551775470376,
+            0.006249943748116493,
+            0.003157886676490307,
+            0.03172214329242706,
+            -0.0017260919557884336,
+            0.009665340185165405,
+            -0.010997422970831394,
+            -0.0002405686245765537,
+            0.0006728176376782358,
+            -0.0006820144481025636,
+            -0.002884887158870697,
+            -0.011020656675100327,
+            0.0018974426202476025,
+            -0.020306508988142014,
+            0.0024473138619214296,
+            0.00463130883872509,
+            -0.005684582982212305,
+            -0.0032972905319184065,
+            -0.019222255796194077,
+            -0.0022672503255307674,
+            0.003293418325483799,
+            0.014335372485220432,
+            -0.031691163778305054,
+            0.004972074180841446,
+            0.010672146454453468,
+            -0.002309845993295312,
+            -0.010416572913527489,
+            0.011423379182815552,
+            -0.0026060794480144978,
+            0.011988739483058453,
+            -0.0038142469711601734,
+            -0.0004905760288238525,
+            0.025681305676698685,
+            -0.005308967083692551,
+            0.014707116410136223,
+            -0.01026167906820774,
+            -0.002993312431499362,
+            -0.028252532705664635,
+            0.005235392600297928,
+            -0.007403898518532515,
+            -0.0012081675231456757,
+            -0.004402841441333294,
+            -0.01130720879882574,
+            -0.0026990154292434454,
+            0.002122037811204791,
+            0.008000237867236137,
+            0.012910354882478714,
+            -0.0007725302129983902,
+            -0.010075807571411133,
+            0.004832670092582703,
+            0.0075355577282607555,
+            -0.015520306304097176,
+            -0.0070437719114124775,
+            -0.020306508988142014,
+            0.029662061482667923,
+            -0.013731288723647594,
+            -0.0011123273288831115,
+            -0.020864123478531837,
+            0.010160998441278934,
+            -0.0063119009137153625,
+            -0.002782270545139909,
+            0.004708755761384964,
+            -0.015109838917851448,
+            0.01344473659992218,
+            0.008751469664275646,
+            0.01450575515627861,
+            -0.02151467651128769,
+            -0.01762685552239418,
+            -0.0044725434854626656,
+            -0.002735802438110113,
+            -0.010648912750184536,
+            -0.0005334137240424752,
+            0.002013612538576126,
+            0.021251358091831207,
+            -0.00032503384863957763,
+            0.0042053526267409325,
+            -0.003214035416021943,
+            0.017224133014678955,
+            -0.010137764737010002,
+            -0.013057502917945385,
+            0.0024260159116238356,
+            0.005994369741529226,
+            0.016186347231268883,
+            0.03158273920416832,
+            -0.020693741738796234,
+            -0.00272999401204288,
+            0.0032740565948188305,
+            0.010656657628715038,
+            -0.004085310269147158,
+            6.304640555754304e-05,
+            -0.027803342789411545,
+            -0.04464024305343628,
+            -0.002422143705189228,
+            0.013057502917945385,
+            -0.015636475756764412,
+            0.012670270167291164,
+            0.008743725717067719,
+            0.010021595284342766,
+            -0.005196669604629278,
+            0.02620794251561165,
+            -0.004793947096914053,
+            0.014304393902420998,
+            0.011733165942132473,
+            0.011911292560398579,
+            0.0009874446550384164,
+            0.0007459079497493804,
+            -0.018354853615164757,
+            -0.008255811408162117,
+            -0.0015954007394611835,
+            0.01765783317387104,
+            -0.027478067204356194,
+            0.008526874706149101,
+            -0.011516314931213856,
+            0.013839714229106903,
+            0.0003666614356916398,
+            0.005661349277943373,
+            -0.01771979033946991,
+            0.012445674277842045,
+            -0.0009337160154245794,
+            -0.025325050577521324,
+            0.004805563949048519,
+            0.011276230216026306,
+            -0.010362359695136547,
+            0.006877261213958263,
+            0.019950253888964653,
+            0.0023079097736626863,
+            0.008495896123349667,
+            -0.014126266352832317,
+            0.010703125037252903,
+            0.009649851359426975,
+            -0.021080974489450455,
+            -0.01233724970370531,
+            -0.007059261202812195,
+            -0.006335135083645582,
+            -0.0035470561124384403,
+            -0.00013105422840453684,
+            0.010176488198339939,
+            -0.016000475734472275,
+            0.010099041275680065,
+            -0.006559730041772127,
+            0.006400964688509703,
+            0.00944074522703886,
+            0.001115231541916728,
+            -0.013576395809650421,
+            0.012716737575829029,
+            -0.0218709297478199,
+            0.008426194079220295,
+            -0.04287445917725563,
+            -0.005847221240401268,
+            -0.015078860335052013,
+            0.005204414017498493,
+            -0.003798757679760456,
+            -0.004453181754797697,
+            -0.006203475873917341,
+            -0.005061137955635786,
+            0.00920066051185131,
+            0.021747015416622162,
+            0.012135888449847698,
+            -0.016620049253106117,
+            -0.007551047019660473,
+            0.005425137002021074,
+            0.0004854935978073627,
+            -0.013336311094462872,
+            0.009316830895841122,
+            -0.01576039008796215,
+            0.0009259713697247207,
+            0.006339007057249546,
+            -0.01072635967284441,
+            0.013452480547130108,
+            0.004731989465653896,
+            -0.003401843598112464,
+            0.013390523381531239,
+            -0.0050417762249708176,
+            -0.0057891360484063625,
+            -0.007272239308804274,
+            -0.013739033602178097,
+            -0.003818119177594781,
+            -0.003072695340961218,
+            0.012747716158628464,
+            0.001529571134597063,
+            -0.0003243077953811735,
+            -0.00782985519617796,
+            0.009154192171990871,
+            0.002220782218500972,
+            -0.0013146566925570369,
+            0.005773646757006645,
+            -0.02300165221095085,
+            0.0001287550403503701,
+            0.004755223635584116,
+            -0.0009487213101238012,
+            -0.006246071308851242,
+            -0.012151377275586128,
+            -0.011903548613190651,
+            -0.038165703415870667,
+            0.009363298304378986,
+            0.0019197085639461875,
+            -0.0076207490637898445,
+            0.0064822835847735405,
+            0.017255110666155815,
+            0.01456771232187748,
+            0.016743963584303856,
+            -0.020693741738796234,
+            -0.02303262986242771,
+            0.005556796211749315,
+            -0.009402021765708923,
+            0.02261441759765148,
+            0.004046586807817221,
+            -0.005847221240401268,
+            0.0013243375578895211,
+            -0.013258864171802998,
+            0.008929597213864326,
+            0.005723306443542242,
+            -0.001592496526427567,
+            0.005425137002021074,
+            -0.013692565262317657,
+            -0.010904486291110516,
+            0.01048627495765686,
+            0.032496608793735504,
+            -0.0004261985304765403,
+            0.0024105266202241182,
+            0.023295948281884193,
+            0.015466093085706234,
+            0.016356728971004486,
+            -0.021111954003572464,
+            0.011167804710566998,
+            0.010060317814350128,
+            -0.0018645278178155422,
+            -0.014892988838255405,
+            0.00513083953410387,
+            -0.012716737575829029,
+            -0.010664401575922966,
+            0.014993669465184212,
+            0.007853088900446892,
+            0.017735280096530914,
+            -0.02369867078959942,
+            0.00789568480104208,
+            0.00042765066609717906,
+            0.00013710475468542427,
+            0.011524059809744358,
+            0.012956822291016579,
+            0.03370477631688118,
+            0.027152791619300842,
+            0.020662762224674225,
+            -0.009285852313041687,
+            0.014366351068019867,
+            -0.009192915633320808,
+            -0.011090358719229698,
+            -0.019470084458589554,
+            -0.015249243006110191,
+            -0.011136826127767563,
+            0.017998598515987396,
+            0.008790193125605583,
+            -0.009355553425848484,
+            -0.005947901867330074,
+            0.01586107164621353,
+            -0.003909118939191103,
+            -0.004635181277990341,
+            0.01646515540778637,
+            -0.026269899681210518,
+            0.016836898401379585,
+            -0.000745423894841224,
+            -0.014668392948806286,
+            -0.014691627584397793,
+            -0.004182118456810713,
+            -0.012670270167291164,
+            0.0003964299685321748,
+            -0.020817656069993973,
+            0.013181418180465698,
+            0.024597052484750748,
+            -0.0030920570716261864,
+            -0.004100799560546875,
+            -0.011299464851617813,
+            -0.0005111477803438902,
+            0.018385831266641617,
+            0.0034773540683090687,
+            0.009192915633320808,
+            -0.0023872926831245422,
+            0.013924905098974705,
+            0.00524313747882843,
+            0.0027958236169070005,
+            -0.0007565568666905165,
+            -0.016263794153928757,
+            0.012197845615446568,
+            0.0026835259050130844,
+            0.012135888449847698,
+            -0.022242674604058266,
+            0.007938280701637268,
+            -0.011764144524931908,
+            -0.0023698671720921993,
+            -0.02228914201259613,
+            0.0189744271337986,
+            -0.01610890030860901,
+            0.001046497724018991,
+            0.003841353114694357,
+            0.0006224773242138326,
+            0.009580149315297604,
+            0.01610890030860901,
+            -0.023388884961605072,
+            -0.0003460896550677717,
+            -0.014908477663993835,
+            -0.021917399019002914,
+            -0.01607792265713215,
+            0.004728117026388645,
+            -0.019501063972711563,
+            0.005444498732686043,
+            -0.0025557391345500946,
+            -0.015125328674912453,
+            0.018478767946362495,
+            0.014691627584397793,
+            -0.004379607271403074,
+            0.002412462839856744,
+            0.005425137002021074,
+            -0.0014405074762180448,
+            0.01971791312098503,
+            0.00013274837692733854,
+            -0.0010048701660707593,
+            -0.009425255469977856,
+            -0.0168833676725626,
+            -0.0065752193331718445,
+            -0.035005878657102585,
+            0.0260530486702919,
+            -0.004670032300055027,
+            -0.016511622816324234,
+            0.014319883659482002,
+            0.009115469641983509,
+            -0.014629669487476349,
+            -0.007059261202812195,
+            0.016294771805405617,
+            -0.024287264794111252,
+            0.02244403585791588,
+            -0.009611127898097038,
+            -0.008774704299867153,
+            0.00784921646118164,
+            0.05154848098754883,
+            0.009146448224782944,
+            0.003492843359708786,
+            0.011036145500838757,
+            0.005839476361870766,
+            -0.0013165927957743406,
+            -0.023683181032538414,
+            0.005568413063883781,
+            -0.007190920412540436,
+            -0.003601268632337451,
+            0.002536377403885126,
+            -0.004801691509783268,
+            0.031149037182331085,
+            -0.004379607271403074,
+            -0.005289605353027582,
+            0.004484160337597132,
+            -0.02196386642754078,
+            -0.0001332324172835797,
+            0.023203013464808464,
+            -2.3128106477088295e-05,
+            -0.023590246215462685,
+            -0.006571347359567881,
+            -0.015071115456521511,
+            -0.013770012184977531,
+            -0.018989915028214455,
+            0.005231520161032677,
+            -0.012755461037158966,
+            -0.0033940989524126053,
+            0.0016660707769915462,
+            -0.005940156988799572,
+            -0.002582845278084278,
+            0.009603383019566536,
+            0.05362405255436897,
+            0.014103032648563385,
+            -0.006335135083645582,
+            -0.005854965653270483,
+            0.009192915633320808,
+            -0.010524998418986797,
+            -0.019795360043644905,
+            -0.009820234030485153,
+            -0.029507169499993324,
+            -0.012747716158628464,
+            -0.0143121387809515,
+            -0.010191977955400944,
+            -0.010277168825268745,
+            0.00963436160236597,
+            -0.018122512847185135,
+            -0.007628493942320347,
+            -0.020337486639618874,
+            0.002941036131232977,
+            -0.011268485337495804,
+            -0.014714861288666725,
+            -0.008054450154304504,
+            -0.04807887226343155,
+            0.005796880926936865,
+            0.019593998789787292,
+            0.01263929158449173,
+            0.002515079453587532,
+            0.004383479710668325,
+            -0.01993476413190365,
+            -0.030792783945798874,
+            0.013839714229106903,
+            0.011283975094556808,
+            0.011733165942132473,
+            0.03283737599849701,
+            -0.011709931306540966,
+            -0.00813964195549488,
+            -0.004693266469985247,
+            -0.0017251238459721208,
+            0.012066186405718327,
+            0.00969631876796484,
+            -0.004917861428111792,
+            -0.012375972233712673,
+            0.016217326745390892,
+            -0.01981084980070591,
+            -0.023683181032538414,
+            -0.01672847382724285,
+            0.006157007534056902,
+            0.00813964195549488,
+            -0.008054450154304504,
+            0.002973950933665037,
+            0.0056962003000080585,
+            -0.007159941829741001,
+            -0.00022423222253564745,
+            0.004890755284577608,
+            -0.023048119619488716,
+            0.0008490087930113077,
+            -0.002257569460198283,
+            0.0061376458033919334,
+            0.021266845986247063,
+            -0.010168743319809437,
+            -0.0014075926737859845,
+            -0.0053902859799563885,
+            -0.0028984404634684324,
+            -0.006764963734894991,
+            0.02620794251561165,
+            0.009154192171990871,
+            -0.009882191196084023,
+            -0.01011453103274107,
+            -0.01129171997308731,
+            -0.0011278166202828288,
+            -0.017255110666155815,
+            -0.016186347231268883,
+            0.0015566773945465684,
+            -0.016712984070181847,
+            0.013971373438835144,
+            -0.00876695942133665,
+            0.00400399137288332,
+            -0.014110777527093887,
+            0.02019808255136013,
+            0.01929970271885395,
+            -0.009711808525025845,
+            0.023776117712259293,
+            -0.007709812838584185,
+            -0.038382552564144135,
+            -0.00800798274576664,
+            0.01843230053782463,
+            -0.006118284538388252,
+            0.010037084110081196,
+            -0.011314953677356243,
+            0.008209343999624252,
+            0.00015319912927225232,
+            0.0018122512847185135,
+            -0.0003112386621069163,
+            0.027555514127016068,
+            0.015535795129835606,
+            -0.005560668651014566,
+            -0.005494839046150446,
+            0.004255692940205336,
+            0.009541425853967667,
+            0.017580386251211166,
+            0.012151377275586128,
+            -0.021019017323851585,
+            -0.014257925562560558,
+            -0.016263794153928757,
+            0.011012911796569824,
+            0.015125328674912453,
+            -0.0040891822427511215,
+            0.01691434532403946,
+            0.014126266352832317,
+            -0.019098341464996338,
+            0.0019884423818439245,
+            -0.008193854242563248,
+            -0.00158184755127877,
+            -0.03029712475836277,
+            0.011043890379369259,
+            0.0074387495405972,
+            -0.0072218989953398705,
+            0.036616768687963486,
+            0.009858956560492516,
+            0.037577107548713684,
+            0.018680129200220108,
+            0.0008974128868430853,
+            0.009045767597854137,
+            -0.018370341509580612,
+            0.013065247796475887,
+            0.01500141341239214,
+            -0.012724482454359531,
+            -0.005734923295676708,
+            0.0013446672819554806,
+            0.0007551047019660473,
+            0.012631546705961227,
+            0.02004319056868553,
+            -0.01795213110744953,
+            0.009773765690624714,
+            -0.015032391995191574,
+            -0.005250881891697645,
+            0.006555858068168163,
+            -0.013932649977505207,
+            -0.008495896123349667,
+            -0.00963436160236597,
+            0.019408127292990685,
+            -0.012600568123161793,
+            0.016573579981923103,
+            -0.009595638141036034,
+            0.015520306304097176,
+            -0.029724018648266792,
+            -0.003669034456834197,
+            -0.006679772399365902,
+            -0.019593998789787292,
+            -0.012887120246887207,
+            0.009820234030485153,
+            0.010633422993123531,
+            -0.009301341138780117,
+            -0.011562783271074295,
+            0.020368466153740883,
+            -0.004185990896075964,
+            0.013529927469789982,
+            0.021189400926232338,
+            0.00737291993573308,
+            0.024132372811436653,
+            0.0005605200421996415,
+            0.014164989814162254,
+            0.005463860463351011,
+            0.010362359695136547,
+            -0.004677776712924242,
+            0.005161818582564592,
+            0.00932457484304905,
+            0.014203713275492191,
+            0.017766257748007774,
+            0.013196907006204128,
+            -0.00598275288939476,
+            0.007419387809932232,
+            0.02016710489988327,
+            0.0016254113288596272,
+            -0.030312614515423775,
+            -0.009719553403556347,
+            0.0001736498816171661,
+            0.00924712885171175,
+            0.009185170754790306,
+            3.817877222900279e-05,
+            -0.01093546487390995,
+            0.011144571006298065,
+            0.00969631876796484,
+            0.002286612056195736,
+            0.006280922330915928,
+            -0.01456771232187748,
+            -0.020957060158252716,
+            0.006106667220592499,
+            0.01798310875892639,
+            0.013917161151766777,
+            -0.014722606167197227,
+            0.022784801200032234,
+            0.006145390681922436,
+            -0.021406250074505806,
+            -0.01840132102370262,
+            -0.0009559819591231644,
+            0.0004542729293461889,
+            -0.002265314105898142,
+            0.03720536455512047,
+            -0.0011539548868313432,
+            -0.005955646280199289,
+            -0.0054754773154854774,
+            0.02775687538087368,
+            0.02049238048493862,
+            0.009177426807582378
+          ],
+          "embedding": [
+            0.0008215984936325879,
+            -0.019349418284688353,
+            -0.02322660633029795,
+            0.009747820845194852,
+            0.007353771220903641,
+            0.015252908783199839,
+            -0.012176175829980202,
+            0.051898051094966605,
+            -0.0068130353886146165,
+            0.008189212816890595,
+            0.008102287831931905,
+            0.01682801136011614,
+            0.02005557996049711,
+            -0.020779879580497183,
+            0.002332165268329781,
+            0.02337485065616377,
+            0.006074992895540845,
+            0.02576575146535689,
+            0.0007517286429767504,
+            -0.03763909942314134,
+            0.0011204729720314833,
+            0.00041212613589632806,
+            -0.06089176800314795,
+            0.012852035873395388,
+            -0.008901127092058543,
+            0.01702347055484216,
+            0.005803446395466281,
+            0.017889570275475296,
+            -0.05375047382413552,
+            0.021623778774252724,
+            0.014784317465883424,
+            -0.0018881880857885708,
+            0.0004735328554290247,
+            0.012393802850407656,
+            -0.019234442968055952,
+            -0.019566283611638122,
+            0.04407789373296535,
+            -0.0053565873209102995,
+            -0.026240866282480922,
+            0.048107902270426675,
+            -0.0054681314939387235,
+            -0.0009441397948041965,
+            -0.015205079626883548,
+            0.006885981841893229,
+            0.01403724256866746,
+            -0.026086548246792754,
+            0.0034778946051893067,
+            0.00044587533948775473,
+            0.032851328435891634,
+            -0.03882097333255334,
+            -0.011126344604254894,
+            -0.01648943336384904,
+            0.035810456555802166,
+            -0.019384589526165754,
+            -0.019430702768170285,
+            0.014297440272239852,
+            -0.007883453204663151,
+            0.050996900793086844,
+            0.06007149434153715,
+            0.03406832053483379,
+            0.025193439934671852,
+            0.00027785129663604897,
+            -0.008014391056168083,
+            -0.018806730908490114,
+            0.027193788903098558,
+            -0.006195006670328595,
+            -0.020466000344352826,
+            0.0068939884219589185,
+            0.017252276167086602,
+            -0.045073444648704136,
+            -0.049242269917908754,
+            0.023021140866008313,
+            -0.015676872472071106,
+            -0.024497517908898602,
+            0.005873633718973869,
+            -0.022828585938203683,
+            0.011748341004940436,
+            0.04624205727927451,
+            -0.015442165499674641,
+            0.018565551071491767,
+            0.015246505395507096,
+            0.004119269796224144,
+            0.004862735250819252,
+            -0.000145323321960046,
+            0.05155838952887077,
+            0.0027052423345323936,
+            -0.016218063670586664,
+            -0.07462470920944148,
+            0.043695974746295135,
+            -0.027608273720377708,
+            -0.03860912774695494,
+            0.002823110565150435,
+            0.026471469849508766,
+            -0.0026063577189074103,
+            -0.029230652014151735,
+            0.05684164470477843,
+            0.00022182868300572677,
+            0.015590845552956364,
+            -0.024939617549529242,
+            0.01035803286203917,
+            -0.010225298012384327,
+            0.019571847648759157,
+            0.01841613041441054,
+            0.0156981470538354,
+            0.021764297694793343,
+            -0.018831230007033416,
+            -0.014419529638789993,
+            -0.0029965475899652273,
+            0.028881185012095884,
+            0.09163526460163937,
+            0.022644302207811818,
+            -0.017933909936330133,
+            -0.04663928709016869,
+            0.004962951444105128,
+            -0.041333041174526026,
+            -0.027545628007617527,
+            -0.037142394579894954,
+            -0.013358143162788402,
+            0.02719686432396677,
+            -0.0034971800937504205,
+            0.04413635817371732,
+            0.04903857110328917,
+            0.03186067155630788,
+            0.007831051050269414,
+            -0.025257538623313574,
+            0.023517513773109414,
+            -0.0016527403209505964,
+            -0.0016101860707429347,
+            0.014448515185903624,
+            0.009905018312477684,
+            0.0024114582235163955,
+            0.00172043851384822,
+            -0.03818316221102173,
+            -0.02304537008110578,
+            0.01024969233570089,
+            -0.01901116356327339,
+            0.019791121460848986,
+            0.07985067340821365,
+            -0.051903191272236665,
+            -0.025924646976556942,
+            0.014766511532849252,
+            -0.002073086076894999,
+            0.030959269926800852,
+            0.006394417986617089,
+            -0.028177586693713442,
+            0.010725285336674526,
+            0.0009614052276924192,
+            0.07500253788978564,
+            -0.00036794182149960157,
+            0.004058458332826519,
+            -0.013218269166951234,
+            0.027613539268683245,
+            -0.012995170442480582,
+            0.023388707018488407,
+            0.026389423313105165,
+            -0.003068639956061266,
+            -0.02661967649632026,
+            0.026145274570635913,
+            -0.03300591322174331,
+            -0.004556072066733024,
+            0.020045722084112896,
+            0.0033437343914784865,
+            -0.010224220905483085,
+            -0.0025025498684079703,
+            0.03460103245395426,
+            0.04514362357278504,
+            0.03797031988058962,
+            -0.004586662475576956,
+            -0.06351723722596923,
+            0.06037983484369874,
+            0.019415753124363902,
+            0.028850618647994476,
+            -0.0034810237045849006,
+            0.02957938154808314,
+            0.02642652523231673,
+            0.013187653724133275,
+            -0.007856320804109569,
+            -0.007065882659482447,
+            -0.016672784625197085,
+            0.029020161154209707,
+            -0.013243058921970134,
+            -0.014900932834241732,
+            0.006001769901896033,
+            0.008648456186944271,
+            0.03484763413464497,
+            0.012279368984306564,
+            0.02379643508687133,
+            0.04933044959399941,
+            -0.021047723935647383,
+            -0.007264823701706426,
+            0.0007058774795716792,
+            -0.010512071832195085,
+            0.027489671668749157,
+            0.009731157570361245,
+            -0.03195919640758579,
+            -0.0033626031972691915,
+            -0.04282419484651207,
+            0.01873977910485268,
+            -0.014398285316695653,
+            0.012854631447494207,
+            -0.0017372959978092432,
+            -0.037331867825735776,
+            0.001088209292238142,
+            -0.012345684323522151,
+            0.037096063326414734,
+            0.005050219690024652,
+            0.0008906051515353004,
+            -0.003697709449630797,
+            0.006548479142737107,
+            -0.010727663712299397,
+            -0.007486226350610755,
+            -0.02704618481069485,
+            -0.019752552016312926,
+            -0.014632581296303486,
+            -0.0037501138119724717,
+            0.014588411795703245,
+            -0.007119137071719376,
+            0.016440258608998535,
+            -0.03686645352111013,
+            -0.023546402448713415,
+            0.005628703432152666,
+            0.024988145751166143,
+            0.011971199595808573,
+            -0.010868778329190568,
+            -0.03270046433624425,
+            -0.021950559285357267,
+            -0.011889914022657738,
+            -0.005123971523701009,
+            0.02619713302280345,
+            -0.021990810305904974,
+            0.01165257474485436,
+            -0.009117901804315863,
+            -0.007492082755249399,
+            0.010725902070291232,
+            0.01862811996181951,
+            -0.026694281868322685,
+            -0.01623776415010199,
+            -0.009403997538684658,
+            -0.02207329027097839,
+            0.025939540497196416,
+            0.014178803271471762,
+            -0.001141821318145744,
+            -0.006281187017652108,
+            -0.035993365198554964,
+            0.0026192776111449633,
+            -0.015254976771874396,
+            -0.0116872453009606,
+            -0.015780016491993135,
+            0.021421328283124968,
+            0.02889432250124136,
+            -0.01871009416296692,
+            -0.026540472640514536,
+            0.015703370682110186,
+            -0.0002849235245164438,
+            -0.02594930889995505,
+            0.04491828744809679,
+            0.044063522197410734,
+            -0.037441864491547405,
+            -0.007005339187980476,
+            -0.030381794393931814,
+            0.06901083845539985,
+            -0.017604082400603037,
+            -0.04060591381445593,
+            0.06044850935418443,
+            -0.01756490171206357,
+            0.025539717592705295,
+            0.03452018789549939,
+            0.0016217577522523927,
+            0.00018825054264059277,
+            -0.002563606092068712,
+            0.04770277039047993,
+            -0.0032010143255574255,
+            -0.03425328601475849,
+            -0.02105909767072799,
+            -0.00840881022563103,
+            -0.028699447677549002,
+            -0.015925160861589006,
+            -0.0055750742281166,
+            0.017513987082479483,
+            0.010399718005952185,
+            0.02267201252273823,
+            0.01258511878601025,
+            0.0015081978484714903,
+            0.01874981572091931,
+            0.011158889421057756,
+            -0.002607810686437211,
+            -0.003732023411827874,
+            -0.007787189972165309,
+            -0.05524012485122728,
+            -0.025110689689279583,
+            -0.03946383236703947,
+            0.004443572529325475,
+            -0.018745077598477586,
+            0.005926160888274727,
+            0.039430331582190524,
+            -0.0049500324516682,
+            0.004352795787899792,
+            0.015566139991780173,
+            0.013148156356408315,
+            -0.015279560337042242,
+            -0.03316846614871165,
+            0.0001592209771257332,
+            0.018362411161743347,
+            -0.012706582418607377,
+            -0.03424477956681259,
+            -0.001991022841226589,
+            0.0033467286047492175,
+            0.03357019284647071,
+            -0.024359110796680374,
+            0.00359117029951049,
+            0.005061818038334802,
+            0.011359582931760315,
+            -0.026793992746244076,
+            -0.018170136079817746,
+            -0.017421603453826397,
+            0.03996608926471779,
+            0.02587013058343799,
+            0.015763410145648683,
+            -0.04512536449983358,
+            -0.02293473890636517,
+            0.048575083783480195,
+            -0.009589860756342604,
+            -0.06486262416944208,
+            0.03447666009581266,
+            0.016105654870253654,
+            0.003526582643722421,
+            -0.01354754623151814,
+            0.05401575248587799,
+            -0.03563190295559527,
+            -0.017365166993914202,
+            -0.0046288709698227735,
+            -0.033657212157212645,
+            -0.007686252059880836,
+            -0.001385273642139782,
+            0.048229062789530236,
+            -0.029083855744728694,
+            -0.030465559177859865,
+            0.0024368661249980416,
+            -0.024144031154493057,
+            0.021693278579520123,
+            -0.016405088055634124,
+            -0.018439935250344958,
+            0.006948603359995389,
+            -0.033943037395199695,
+            -0.003007796423804453,
+            -0.016783195426536633,
+            -0.02270589491206155,
+            0.020874438753828814,
+            0.016724666903417947,
+            0.027033184180344852,
+            0.0019847647852695785,
+            0.017749524526371133,
+            0.0204661893602285,
+            -0.005604929968008821,
+            0.028462234633072944,
+            -0.020790459496727066,
+            -0.04183816355941877,
+            0.03946487892663695,
+            0.01192861758684541,
+            -0.0018024841275380242,
+            0.01077082591611702,
+            -0.01475239020295694,
+            -0.03952356245715099,
+            0.03683950018557526,
+            -0.056767672022551764,
+            -0.007507600090017693,
+            0.009530544352246083,
+            -0.00788856051420074,
+            -0.011204694027562521,
+            -0.010587701466318598,
+            -0.028604940377315504,
+            0.023749537635431462,
+            -0.013672419168235691,
+            0.01953449229707597,
+            -0.011449939491554354,
+            -0.012238180529977605,
+            0.05795536247491174,
+            -0.021518581774137322,
+            -0.040672955795925836,
+            0.026066659238628543,
+            -0.014061106927896723,
+            0.03328357230532424,
+            0.03957107406164427,
+            0.030670947051333917,
+            -0.004278577327978564,
+            0.025885946540364826,
+            0.0319106101668716,
+            -0.003462160036198642,
+            -0.04686868330851868,
+            0.00555303933159587,
+            0.02156143971867743,
+            -0.03739529489075782,
+            -0.01025626126223168,
+            -0.02141970191890338,
+            0.022134910980548407,
+            0.017807484142010496,
+            -0.007232643256393455,
+            0.011318041192005274,
+            -0.010489012862455913,
+            -0.013003217163997554,
+            -0.00847279880392288,
+            0.02357274819910594,
+            -0.0016485066139744577,
+            -0.005873113634992993,
+            0.006424249429163587,
+            0.004008703814301658,
+            0.0054639858216457724,
+            0.010198079500408441,
+            0.017694720520798957,
+            -0.017087021465208255,
+            -0.0025558869389290554,
+            -0.016503579212015177,
+            -0.0420082404922411,
+            -0.003563173424359332,
+            0.022924161700566468,
+            0.012345933179990547,
+            -0.00729908668365632,
+            -0.004234683369642856,
+            -0.0022962545569187934,
+            -0.020666205489143392,
+            -0.014796149043465308,
+            0.024562653223873056,
+            0.030217494971086438,
+            0.007810961433007407,
+            0.007781263640449456,
+            -0.0030348559388780655,
+            0.007310876971216526,
+            0.016099591640649783,
+            0.02311929597780174,
+            -0.04160162038353416,
+            0.027152100845897007,
+            -0.00025396315502635146,
+            0.023717123370023598,
+            -0.04186918759097667,
+            -0.017988531857836703,
+            -0.03106663560912813,
+            -0.023595079699181436,
+            0.02437134274097885,
+            -0.02317559303912261,
+            -0.012327263775257306,
+            0.000989566600374836,
+            0.00954921216572729,
+            -0.005155079395685017,
+            -0.017508404061379164,
+            -0.04582321461886958,
+            -0.0014367264231457263,
+            -0.030417229142190933,
+            -0.017829710557332023,
+            -0.001056472801554354,
+            0.005016912084754242,
+            -0.06276644565618253,
+            0.0109965681725148,
+            -0.07270669271977365,
+            -0.004542796125637244,
+            0.040139998232966566,
+            -0.017890441783630643,
+            0.055909102081208445,
+            -0.05029104308417623,
+            0.01673962668706085,
+            -0.04507293803055318,
+            0.015385506084493484,
+            0.025677372277539627,
+            0.030235171586316813,
+            -0.022189978375852563,
+            0.008461188938299745,
+            0.008033707483259569,
+            -0.045089505755429415,
+            -0.04144698743458133,
+            -0.013640256640978024,
+            0.01036782197518379,
+            0.0012696958402094232,
+            -0.016090728932978384,
+            0.023838437877040163,
+            0.011946482002767135,
+            0.03342604921859589,
+            0.04211801335157251,
+            0.002708015395637621,
+            -0.047156646610976634,
+            -0.013104464812562618,
+            -0.003933116891120681,
+            0.03155321708116329,
+            -0.018866109579446184,
+            0.015376812010630058,
+            -0.01955557376085726,
+            0.016255217383093355,
+            0.017868332102274922,
+            -0.020325179506828335,
+            0.013434364256590097,
+            0.004176904701252156,
+            -0.03877060944639854,
+            0.0006475542749369515,
+            0.00042468300294954125,
+            -0.02116992978038693,
+            0.0032654387271108293,
+            -0.00575003786159731,
+            -0.05149954071159492,
+            -0.031382926943128234,
+            -0.015730563640253065,
+            0.0382312502769352,
+            0.004297939848910064,
+            0.019724983545557365,
+            -0.017907869879607064,
+            -0.005165628493894463,
+            0.03698976707579373,
+            0.008501001373266799,
+            -0.01725527821298066,
+            0.006879620077191753,
+            0.02455792200929138,
+            -0.03224193972301503,
+            -0.039181328884413266,
+            -0.0008206986106407542,
+            -0.023306758893533952,
+            0.01672946960876128,
+            -0.02658587698869307,
+            0.005873620623736886,
+            0.0025331170979801232,
+            -0.03490809296591325,
+            0.02456764314036355,
+            0.003291113462353163,
+            -0.004303423672882112,
+            -0.020359544091059775,
+            -0.016892415165120203,
+            0.017174797487220006,
+            0.04391489511478065,
+            0.007327321684948843,
+            -0.003943713832544148,
+            0.00862684231313155,
+            -0.006361172785753631,
+            0.05826688726534012,
+            -0.02284956969636637,
+            -0.026680023999604643,
+            0.027527436347931256,
+            -0.022597669279069723,
+            -0.006277863878949772,
+            -0.017343446085471857,
+            -0.012065192152203348,
+            -0.0004183712490442143,
+            0.019339311388446943,
+            -0.019423562920897294,
+            -0.010212527447243049,
+            0.004783006556748757,
+            0.012824133870627983,
+            -0.004855945108275706,
+            0.006239949458562236,
+            0.006847581690471813,
+            0.03094005096378827,
+            0.026993131409761086,
+            -0.011331881651539968,
+            0.03297014291876675,
+            0.010265600643742144,
+            4.666113318251278e-05,
+            0.001970879281172325,
+            0.002523664335535017,
+            0.0034107811520368574,
+            -0.03143890864030425,
+            -0.01624584278940617,
+            0.013468729048885906,
+            0.005233309338223403,
+            -0.004550544967856448,
+            -0.023783653541634683,
+            -0.009402494239255264,
+            0.03334923712760375,
+            -0.003260424721959948,
+            0.02748395114281811,
+            -0.0058374897745283065,
+            -0.01985840930273332,
+            0.017595418271610163,
+            0.02881743246631328,
+            -0.05928552440605384,
+            0.017062540894682354,
+            -0.0011508547522925793,
+            0.008953761538519997,
+            0.01610561914042553,
+            -0.008894160513648953,
+            0.013301991014533684,
+            0.04080863883897929,
+            0.010738051773755976,
+            -0.023345002072886145,
+            0.004215726310838674,
+            0.01541749448570608,
+            0.01291813299324752,
+            -0.022747942296528025,
+            0.0008640791861357522,
+            0.011008522066277018,
+            0.005893793007012561,
+            0.0026106450193140156,
+            -0.006278394080583205,
+            -0.025659188287146268,
+            0.024180550333596625,
+            -0.0044491292740074925,
+            0.026967973346698617,
+            0.04390909273429254,
+            0.009596855239692042,
+            -0.03686020608700941,
+            -0.013109951664921729,
+            0.0002799680359670127,
+            -0.0038537577602593366,
+            -0.0037552094562213916,
+            0.011105295862811114,
+            -0.04185615640149602,
+            0.019950890919369135,
+            0.006453241355231516,
+            0.015314645427201533,
+            -0.020031193483734364,
+            -0.024530173488182337,
+            0.0031949828520724337,
+            -0.01492645564652531,
+            0.012875827757040941,
+            0.005057170413681063,
+            -0.004319834710426384,
+            0.005386862713494232,
+            0.00030848054013413157,
+            0.008384941028326983,
+            -0.010811526993301901,
+            0.03175068297908541,
+            0.004680987487229685,
+            0.009264547472281849,
+            -0.02856314241700808,
+            -0.021083721348429348,
+            0.020113396771629096,
+            0.03572366771051171,
+            -0.01708529918264109,
+            0.03346921724975398,
+            -0.009042046660366718,
+            0.0033219155387035227,
+            0.0151720168318272,
+            -0.015805318210244414,
+            -0.008057236707694303,
+            -0.0005483009439015931,
+            -0.002917527004423299,
+            -0.014538511124345527,
+            0.007302979537905532,
+            -0.0208529958115422,
+            -0.02965227242147972,
+            0.020064178921373875,
+            -0.00011892622932349096,
+            0.016455188521316146,
+            0.010335044657747514,
+            -0.007816155992852236,
+            -0.0052262330850905165,
+            0.006959344748883874,
+            0.04432147882994085,
+            0.026734484553994495,
+            0.05547661378166294,
+            -0.005787997952888261,
+            -0.004559444854138438,
+            -0.0082295581123363,
+            -0.006588172755707144,
+            0.04524009609473585,
+            0.015915003261638767,
+            -0.004910285095602864,
+            0.017495607570191165,
+            0.01557886571266685,
+            0.03858259260136037,
+            0.0027648525902558316,
+            -0.016912796336835408,
+            0.04014755669075221,
+            -0.004077405782937414,
+            0.008616827127381418,
+            0.011102692199778469,
+            0.0024185742218798562,
+            0.00807141794538637,
+            0.02996274720815659,
+            -0.012429186019907761,
+            0.01680942269506771,
+            -0.04571916592313758,
+            -0.02775530246832996,
+            0.014370280587419414,
+            -0.004931905540824582,
+            0.03419959665162407,
+            0.01668521021731861,
+            -0.023105305446763272,
+            -0.024761385385514877,
+            0.021529428247380984,
+            -0.002603042335561961,
+            -0.024135326707477325,
+            -0.015906833215844473,
+            0.003286605488908195,
+            -0.0023437997044735617,
+            0.0060251848371355166,
+            -0.013481652076194643,
+            0.0002908426939460218,
+            0.01953362396253743,
+            0.0038544889405965057,
+            0.020760460182873287,
+            -0.016224396121239458,
+            -0.00259233446682179,
+            -0.009282861237653619,
+            -0.02952130765234087,
+            0.017988370145301916,
+            0.006138236653154911,
+            -0.007607182955958238,
+            -0.009516767050028553,
+            0.0048347778733230365,
+            -0.005819209642901871,
+            -0.0025851701689263706,
+            -0.01073014664868326,
+            0.012433465684360061,
+            0.025784014942833097,
+            -0.016875624785552303,
+            0.007944349714123104,
+            8.651589872318479e-05,
+            -0.022306044741692753,
+            -0.022127016745066307,
+            0.020143472307319017,
+            0.002792646249762062,
+            0.008768813346304779,
+            -0.004074214276300631,
+            -0.00015024341785520542,
+            0.002990550758627699,
+            0.01951746553838439,
+            0.03896451381932467,
+            0.004232673242619471,
+            0.027876586543280867,
+            0.0145860303663199,
+            0.03994234200854897,
+            0.009657017763624678,
+            0.00696607158576712,
+            -0.026055387694656488,
+            0.02819890357011778,
+            0.00018297502510753014,
+            0.043190622821775236,
+            0.022488402221902314,
+            -0.0407730101161195,
+            -0.0021946518523759066,
+            0.01669520746276111,
+            0.013212623257794862,
+            0.017009844136059122,
+            -0.004537512994041156,
+            -0.014041213424740049,
+            0.028135404528975894,
+            -0.022859586430516003,
+            0.0008637006862481223,
+            -0.004486332934188017,
+            -0.025031795510001887,
+            0.007351370254175368,
+            -0.004567816916440959,
+            0.030025413528233364,
+            0.0014976740344892735,
+            -0.022285096307825804,
+            0.022837170656498543,
+            -0.009551269792616045,
+            0.011045530488684207,
+            0.008809047185288755,
+            0.020065001354321994,
+            0.031993362733068575,
+            0.009143210434415064,
+            0.02133640814803879,
+            0.012241777474889248,
+            0.014771538127546474,
+            -0.008804192497858033,
+            -0.02341427260922065,
+            0.023167424359504862,
+            -0.0011598649565273595,
+            -0.029627604902846688,
+            0.0021991474725787303,
+            0.003261878079344105,
+            -0.0005935713331682358,
+            -0.009144689470851402,
+            -0.005580557862988033,
+            -0.007240836451875213,
+            0.019631377011298905,
+            -0.007085213886352692,
+            0.007017573363709754,
+            -0.004074171791476129,
+            -0.00822871835665378,
+            -0.0421101573155962,
+            0.004210914075286222,
+            0.003408427700622687,
+            -0.01752179330665445,
+            0.006384440404174512,
+            0.011121481025573962,
+            0.043020575455262985,
+            -0.022809979913698254,
+            -0.0005737138409049003,
+            0.019206623212867476,
+            0.0009562551889820241,
+            -0.023427467810802533,
+            -0.038931688376706186,
+            0.007172345672967499,
+            -0.03467050750542982,
+            0.023185301620612612,
+            -0.0007075596339842658,
+            0.02387977204565927,
+            -0.03043468414449851,
+            0.013909304572370437,
+            0.004784668158421625,
+            0.012524879174804915,
+            0.021654512058639427,
+            0.03142146922870715,
+            -0.02053473475540882,
+            -0.050409862638621725,
+            -0.005406667316025549,
+            0.00035138762584085976,
+            0.022018487016014856,
+            -0.026014580970974616,
+            0.00025633004699302783,
+            -0.003349094027291921,
+            -0.02401366769212778,
+            -0.01735996745614562,
+            0.01962340791446025,
+            -0.050011480783403936,
+            0.015042913576492952,
+            0.0262737424555697,
+            -0.02507473901367252,
+            -0.05408889321034101,
+            -0.018988587082325706,
+            -0.028217893770659686,
+            -0.009285770094319318,
+            -0.00661159230987788,
+            0.028597405154740078,
+            0.003525576256054959,
+            -0.011462219358952466,
+            0.025401668390029348,
+            0.006155837251830549,
+            0.0026050693705924593,
+            0.015985722392086222,
+            0.007289910674212967,
+            -0.011452511453623235,
+            -0.014599577191293152,
+            -0.04558584556100621,
+            0.010197536827229795,
+            -0.0045431962637690155,
+            -0.019372598050508005,
+            -0.030147301333718016,
+            -0.0137828704002156,
+            -0.027060515748528666,
+            -0.0017291966950107505,
+            0.053783684195381654,
+            0.03292494290301388,
+            0.005560172296221419,
+            -0.0026085670565688026,
+            -0.010466890489356092,
+            0.036247483457995895,
+            -0.006789746401727701,
+            0.03662690472838588,
+            0.001784522073210422,
+            -0.0242673253380092,
+            0.01169382884548301,
+            0.02892627181542449,
+            0.009858451662166387,
+            -0.0005592283645355213,
+            -0.006160132931009508,
+            -0.008782830470079547,
+            0.0031207630972208394,
+            0.025613233596604418,
+            0.003268170392795548,
+            -0.010062991425127038,
+            0.018106967407336815,
+            0.004480291699976396,
+            0.02010374270557836,
+            0.03555357587371324,
+            0.023489837757983054,
+            -0.021782173652026497,
+            -0.026581762796336082,
+            -0.0013561486781077014,
+            0.0072266106510934505,
+            0.03551967295942857,
+            -0.01651315514034218,
+            0.0001274396319698452,
+            -0.008272735507120127,
+            -0.00894931703949637,
+            0.011149952898504355,
+            -0.0029671593121986116,
+            -0.0030752010285951576,
+            0.01662711184497578,
+            -0.0011672263615879709,
+            -0.033448222499147764,
+            0.0078288560210653,
+            -0.023496331346917084,
+            0.0051040606404415545,
+            -0.004113104164416575,
+            -0.007076525381595441,
+            -0.002157548492138289,
+            0.006302962169276006,
+            -0.008065841090139872,
+            -0.00035242603988955366,
+            -0.01586406637763306,
+            0.008271894439780969,
+            0.007438312999409211,
+            -0.0069617481460406095,
+            0.011402932583388205,
+            0.002666895909532698,
+            0.0005219881149353855,
+            -0.012159425670629751,
+            -0.012042145306711322,
+            -0.00957276588761916,
+            -0.03401863879157551,
+            -0.02604353712472079,
+            -0.024949533021906844,
+            -0.0014209149651479556,
+            -0.006566685925666972,
+            -0.03187796739029736,
+            -0.01209152137926458,
+            -0.011110763124407752,
+            0.007756961097537267,
+            -0.013260548545917543,
+            0.005433285079114047,
+            -6.399098520041351e-05,
+            -0.0044960488826674555,
+            0.016615636438440516,
+            -0.003643291599919672,
+            -0.019488010211707343,
+            -0.03927818878149417,
+            0.008209336177358258,
+            -0.02805593125673831,
+            -0.0038816771550190788,
+            0.007607119219721711,
+            -0.00528681677614909,
+            -0.029502897838743516,
+            -0.009149794737430706,
+            0.008767001199311985,
+            0.003507217008280066,
+            -0.01773349756494436,
+            0.03579670773605949,
+            0.026971617790626075,
+            0.011875179879209696,
+            -0.019891342676443626,
+            0.0508929813275565,
+            -0.0047010735337056275,
+            -0.012544935684845649,
+            0.018297503869683027,
+            0.02355022268131519,
+            0.03273167043015095,
+            -0.024830243320788693,
+            0.0009370814985214882,
+            -0.021937793653233454,
+            0.00535662615596573,
+            -0.035009915581788176,
+            0.0028595659314725012,
+            -0.014395027018699446,
+            0.015596459769560532,
+            0.03397372881171701,
+            -0.013254226072981457,
+            0.011195747181764677,
+            -0.016299384375589934,
+            0.00730244731485402,
+            0.00569944090461748,
+            -0.012916284280345044,
+            -0.005812311882399539,
+            -0.022902815152116493,
+            0.020140276669979015,
+            0.005809799950176385,
+            -0.0008130655714656685,
+            0.0104085826652653,
+            -0.044723432238900435,
+            0.03286429597337671,
+            0.008730225944501239,
+            0.017599189944921396,
+            0.0029762446259779998,
+            -0.0003068671951232415,
+            0.011611551606224214,
+            0.011228401175698422,
+            -0.016029727902740706,
+            0.004345913009102136,
+            -0.004613061137378433,
+            -0.03850533909474302,
+            0.009363351681220008,
+            -0.03631184057978666,
+            0.014427784677340159,
+            0.016026872002233575,
+            -0.004170582842643461,
+            -0.025166189571409366,
+            -0.005428471012333673,
+            0.010827910811085384,
+            -0.009783069664327864,
+            -0.009987805234276443,
+            -0.04163358183060102,
+            -0.025236440364963034,
+            0.034097508113604576,
+            -0.002119474613596975,
+            -0.025826403511707147,
+            -0.010830738112107807,
+            -0.011351132636491575,
+            0.020207687645133,
+            -0.006542026283946858,
+            -0.0006263975804936843,
+            -0.04150011545311013,
+            -0.0073396962917864515,
+            0.011192225907726928,
+            0.027746213794734322,
+            -0.035970017302005566,
+            0.02708126943501373,
+            0.002105633542220187,
+            0.014262674845755191,
+            -0.01626664983003942,
+            -0.005631805361903626,
+            -0.013990539604372344,
+            -0.035877093361375685,
+            0.016487723013906077,
+            -0.008607758248055054,
+            0.004714436224135091,
+            0.04458009880881818,
+            -0.0036872068506712727,
+            -0.009466125348499464,
+            0.007600705817195143,
+            -0.0034590883553048063,
+            -0.010351576085287273,
+            -0.0028564505793999295,
+            0.001959661298507,
+            -0.007080697863126256,
+            -0.04183476693971616,
+            -0.00499888976530217,
+            0.04684391782633148,
+            0.006361744381879857,
+            0.013276793765092465,
+            0.013516220521120684,
+            -0.018335746557932003,
+            -0.045577810515431215,
+            -0.0071616111758423025,
+            -0.0032763880176386808,
+            -0.022708028392758567,
+            -0.0240284945459861,
+            -0.008735040856657758,
+            0.005844145402902923,
+            0.006359368165686724,
+            -0.01120763599008222,
+            -0.022676131361171057,
+            -0.027085968700452136,
+            -0.010526103176253857,
+            -0.02123944716136792,
+            -0.010792145796733375,
+            -0.027142270190771887,
+            -0.028742234538796804,
+            0.02880426953386151,
+            0.007575182232859564,
+            -0.004084088301176117,
+            -0.010005765476121966,
+            -0.01255640546813973,
+            -0.0004774761617900028,
+            -0.0025307994917461232,
+            -0.005822065709999144,
+            0.0038614286313282164,
+            -0.011390482620811868,
+            0.023152086963976554,
+            0.0021220701260546895,
+            0.018329162172128138,
+            -0.05772324571231113,
+            -0.002709083851090071,
+            0.021497896395795905,
+            0.0033025410488410913,
+            -0.005412445141910314,
+            0.029352650459792132,
+            0.011223947499453582,
+            -0.006539460282439711,
+            -0.031355428613718234,
+            0.020942976176820262,
+            0.024122993515785947,
+            0.00817615634503138,
+            -0.012994160510030828,
+            0.01221095914384204,
+            0.01629778384790159,
+            -0.004494985018246821,
+            -0.011161098655072933,
+            0.005586857967039181,
+            0.020941039642372112,
+            0.013580622629911925,
+            0.0013398071867077221,
+            0.030905022925366163,
+            0.0006862084391508121,
+            -0.03521231612862609,
+            -0.01908191995763771,
+            -0.01275995164030872,
+            -0.003584331305986364,
+            -0.015264425022763126,
+            0.005294329654809353,
+            -0.002355465010876059,
+            0.006335052149475905,
+            0.022655379849785256,
+            0.03593231939948758,
+            0.0146937743203089,
+            0.00937330328448886,
+            0.003167462724350819,
+            -0.0013235176101076284,
+            0.012102872918129403,
+            0.008638011773075119,
+            0.0350701240268404,
+            -0.014653221697693864,
+            -0.011003994230902835,
+            -0.0037767288519369537,
+            0.010100131061186566,
+            -0.004113527518662366,
+            0.020414393256136372,
+            0.03274709615649469,
+            -0.0037222949335916876,
+            -0.005604552180246868,
+            -0.006160118231975786,
+            0.0041467454414452565,
+            0.01807879127455274,
+            -0.01252134965321004,
+            0.004364858987501461,
+            0.005549300665159658,
+            -0.0071897162833861,
+            0.018567059255813558,
+            0.0002622074720860951,
+            -0.012983550838417323,
+            0.03266376564402191,
+            0.11449509387240629,
+            0.0017251916835047628,
+            0.007125215166916139,
+            -0.040229997815548584,
+            -0.020537987527148888,
+            -0.00245728735683473,
+            0.005779972828836593,
+            0.007004732428236457,
+            0.02225914982152608,
+            0.006505827715591914,
+            0.0006708045351142873,
+            0.031708522931495536,
+            -0.017334083511709092,
+            -0.002522861743921599,
+            -0.01745656559688129,
+            0.013890451996979048,
+            -0.014312605700272691,
+            -0.013784963572164409,
+            -0.01572417853955075,
+            0.004507425871748977,
+            -0.000601158156253668,
+            0.0016158265715634729,
+            0.01791303234734911,
+            0.0011926917387973374,
+            -0.026393351252156246,
+            0.0006694008039871878,
+            0.001026031515214626,
+            -0.004514419527575117,
+            -0.011320929719992686,
+            -0.02223001723942815,
+            0.016235841442653918,
+            0.014188311619681008,
+            -0.004895218832118094,
+            0.009244697376301004,
+            0.010558796153959677,
+            0.02707485418319703,
+            -0.028380342401606758,
+            0.0040204430418629405,
+            0.007023041225985294,
+            0.01606556581447933,
+            0.017617862031190843,
+            0.0034870847111552862,
+            -0.011253315705053224,
+            0.03238289399766116,
+            -0.010563282687352038,
+            0.0065093536270233385,
+            0.014966802208644462,
+            -0.03541611540945151,
+            -5.821682910348165e-05,
+            0.0061985444938402745,
+            -0.011716201661363997,
+            0.009580035961315467,
+            -0.03120692534097433,
+            -0.0039291243987978845,
+            0.009317468331974374,
+            0.017611312558635673,
+            0.008804996982574765,
+            -0.0034129169537151925,
+            0.010822226698822738,
+            0.012510661458054741,
+            -0.0027147975086409966,
+            -0.008855213644027594,
+            0.01238502686635956,
+            -0.0007687053446954701,
+            0.00450108863711973,
+            -0.00799170686059026,
+            0.0029755595812552293,
+            0.002290650917244691,
+            -0.01554856300008412,
+            -0.011779430686394945,
+            0.00438393126679289,
+            0.0036931891075561957,
+            0.001535662168147833,
+            0.012014408636737167,
+            -0.006087255045485002,
+            -0.0077816220325470226,
+            0.0040573724156794284,
+            0.003136943055301706,
+            -5.505014316363865e-05,
+            -0.030625245024561508,
+            0.01585923084309149,
+            0.0067595435845396405,
+            0.010013308598626031,
+            -0.005315626260228517,
+            -0.00903142414091241,
+            -0.007240920015018893,
+            -0.01228521172101564,
+            -0.010657894223173453,
+            0.007658605592936815,
+            -0.00724924993432376,
+            0.011124174211132038,
+            -0.03128027787357322,
+            -0.013988715435985316,
+            0.01231792993245938,
+            -0.01528161242159377,
+            0.0007675821323979293,
+            -0.008389914312201402,
+            0.0009417242487395995,
+            -0.005119304379761705,
+            0.007887436211143328,
+            -0.03731503169547854,
+            0.007878001990271541,
+            0.02456170736765724,
+            -0.00330924625934981,
+            -0.020259771366167557,
+            -0.009613930457931786,
+            0.0241314385136084,
+            -0.008267831641886344,
+            -0.01645888287387047,
+            0.008421304862524506,
+            0.010140813149093993,
+            -0.0006309118552875393,
+            0.0008237784055549769,
+            -0.003726984191590917,
+            0.01669125388798323,
+            0.02054777616420997,
+            0.041930770964107515,
+            -0.014789610432004559,
+            0.01092349703703152,
+            0.02866938974792608,
+            -0.03378576980722847,
+            0.014728113846952417,
+            0.02518409645155824,
+            0.014590531327650454,
+            -0.0004446956315190408,
+            0.001609777186372437,
+            0.0068939321280085475,
+            -0.014273492115311677,
+            0.009247676157551494,
+            0.013380104440744367,
+            -0.018407746260686045,
+            0.020444430509441124,
+            0.03140071554008003,
+            0.0025839998101985736,
+            0.01355205268903058,
+            -0.006268939830271642,
+            -0.013860692186849113,
+            0.014717364068116991,
+            0.031429975488060896,
+            0.0006919565473572909,
+            -0.018561500446336806,
+            0.019128244642842268,
+            0.006988095806264863,
+            -0.003678529269242034,
+            -0.0021993078588142664,
+            -0.021334919445200388,
+            -0.021003586791369228,
+            0.017304350704792804,
+            0.013508779408333964,
+            0.027794485858413365,
+            -0.010594702782952029,
+            -0.007818089330662161,
+            0.014293688127901867,
+            -0.022368568887004094,
+            0.015426482139881204,
+            -0.02160820059598751,
+            -0.00571499813703566,
+            -0.0077402210272634235,
+            0.03274360648629609,
+            0.001441304806200403,
+            0.02734331677589297,
+            0.016457027764064002,
+            -0.033609420134045206,
+            -0.029790535553657092,
+            -0.00712106584238484,
+            -0.02307933297713104,
+            0.010356561171963577,
+            -0.03901183838032092,
+            -0.00071735150189382,
+            0.008251646656432826,
+            -0.008079917288471352,
+            -0.0007087526632966007,
+            -0.006419191399519291,
+            0.013989344185652506,
+            0.013661633160336152,
+            -0.0228103143826663,
+            0.023644639659577426,
+            0.008343481940133684,
+            0.025998483906975782,
+            -0.02076055376006461,
+            0.019148568238437927,
+            -0.017705363646416224,
+            -0.02777313933338929,
+            0.014358130220942483,
+            0.023585731219788467,
+            0.01635364038245971,
+            -0.007122853965024528,
+            0.007110003583832299,
+            -0.03556761320536796,
+            0.027585457849964405,
+            -0.013064062623866305,
+            -0.012504161289952458,
+            -0.0032379276562094555,
+            -0.011931827532360686,
+            -0.002574161005800279,
+            0.0008533907465694845,
+            0.0034345232246102806,
+            -0.03131462673429805,
+            -0.0002450959360520896,
+            -0.03727416416766574,
+            -0.028944814604815072,
+            0.01360126708805471,
+            -0.00036361284498363273,
+            -0.0049097813612814616,
+            0.02933242705521636,
+            -0.026339136251046315,
+            -0.02693295650849956,
+            -0.009590247544093352,
+            0.02853096456887074,
+            -0.0220673798697511,
+            -0.004519050393279848,
+            0.0015034014871316069,
+            -0.01828226786700525,
+            0.004436383319946364,
+            -0.0012748481042235446,
+            -0.010416065158876917,
+            -0.01105146139270597,
+            -0.03229479099927737,
+            -0.005822133182096844,
+            0.02106215163620738,
+            -0.006957125617688484,
+            0.002746241228040847,
+            0.013312248331456934,
+            -0.010968629118224582,
+            -0.0029946900704643812,
+            0.0017875284441794235,
+            0.023304175718480834,
+            -0.021033941064173063,
+            0.007714334541806199,
+            0.005955442031111047,
+            0.0016405586860131207,
+            0.018406562522853148,
+            -0.0054922197063461825,
+            0.0037690571959124617,
+            0.0162126534419016,
+            -0.009969304596173302,
+            0.017959307638726878,
+            0.007691549802475525,
+            0.006681837561745192,
+            0.002775700618222622,
+            0.007712770283579111,
+            0.009752652973700739,
+            -0.013506020126078112,
+            0.016911141425375933,
+            0.022468715726349706,
+            0.00023233555209372766,
+            0.021066580799163234,
+            -0.02302516292580905,
+            -0.028618544628453015,
+            0.018297878810036087,
+            -0.0225078941299981,
+            -0.012735208680311652,
+            0.016459925496937065,
+            -0.0043032724578332855,
+            -0.00797609922711709,
+            -0.014986057073351055,
+            0.02260618561608301,
+            0.02132930921464508,
+            -0.020249788476562527,
+            -0.012385936327068884,
+            0.0072304431443959505,
+            -0.004373475447051833,
+            0.007086656081865733,
+            0.03195810137269375,
+            -0.015982210897392764,
+            0.006818890093901413,
+            -0.0038306138017521626,
+            0.0030753642449792215,
+            0.017830546304667783,
+            0.009470018737705222,
+            -0.0019815241970236146,
+            0.030100238416495066,
+            -0.001330614013444443,
+            0.016392226282459177,
+            0.0004888028477871524,
+            -0.010948833692356176,
+            0.006343890495626752,
+            -0.026652806306173643,
+            -0.008627570055007363,
+            -0.001203530223345183,
+            -0.009581494451831351,
+            0.07598616930784373,
+            0.003930153552069478,
+            -0.01295957554290434,
+            -0.001673007115291578,
+            -0.001368287102181597,
+            -0.006897179139643672,
+            -0.00960679451603541,
+            0.0287166325268306,
+            -0.016613613331623346,
+            -0.01894527592131826,
+            0.0008370014132002703,
+            -0.006785618888421608,
+            0.00919924251249592,
+            0.01816363936999796,
+            -0.010225961928154075,
+            -0.01604759761423727,
+            0.001577138242530474,
+            -0.003520564936698765,
+            -0.004395434431595494,
+            0.0052399043934902014,
+            -0.010127122101423486,
+            0.006456662777583096,
+            0.026560491731327886,
+            -0.013684536319659296,
+            -0.01425196172663986,
+            0.004084982328707292,
+            -0.0010682084059972982,
+            0.009758641913073978,
+            -0.00864879062601467,
+            -0.00481222050200678,
+            -0.014375330954464725,
+            -0.011378890549403524,
+            0.02505641780042516,
+            -0.02861164504722546,
+            0.0034060847953066354,
+            0.014937227531980433,
+            -0.00016904224334342915,
+            0.01541381851648916,
+            0.0033028855218525644,
+            -0.03615935765718852,
+            -0.013677774380513626,
+            -0.02073839691250734,
+            0.019821074228011413,
+            0.01688422012576042,
+            -0.0033731840788147442,
+            -0.013840993586441808,
+            -0.026149880178874887,
+            -0.007430563876740068,
+            -0.02466234818567362,
+            -0.020708282040996014,
+            -0.019276991448222817,
+            0.006137002799644838,
+            0.005162892426268572,
+            0.022881759407670583,
+            0.0025854880108157857,
+            -0.015058889448404231,
+            0.0029948608544278386,
+            -0.006722836436935071,
+            0.011845298212977357,
+            -0.02747944486818378,
+            0.015311272959213262,
+            -0.01610493464983465,
+            0.025239768880017402,
+            0.007425208902253236,
+            0.0010569008485112994,
+            -0.008822681490988575,
+            0.00010704167529046774,
+            0.014883787023445807,
+            -0.004186968199973372,
+            -0.0007781259002896265,
+            0.02787555889813697,
+            0.02087373813833492,
+            -0.0017783425135865238,
+            0.008743718448136432,
+            0.028026471206552342,
+            0.019081083283793984,
+            0.0325261942213039,
+            -0.013815526479760766,
+            0.005113676604826083,
+            -0.028873600114081546,
+            0.010510475843707614,
+            -0.010871039729515013,
+            -0.010201824683340651,
+            -0.0067946599549165735,
+            0.027756887504389347,
+            -0.007962601205446472,
+            -0.0006687976124826922,
+            0.03469894503563926,
+            -0.014185294847328063,
+            -0.00010543064621374546,
+            0.0007386418834840544,
+            -0.009759481309152999,
+            0.02742722730386853,
+            -0.0073590367291510046,
+            0.018456032721630843,
+            -0.019560882612141943,
+            -0.013705917411254969,
+            -0.011894560208199602,
+            -0.001142768182039817,
+            -0.02576754877141608,
+            -0.008755240066826174,
+            -0.02249287216087102,
+            0.00859172262856532,
+            -0.0046012388553094375,
+            0.018845920253446127,
+            0.0011617844393195566,
+            -0.005275799428018491,
+            0.029069367036565145,
+            0.0088080893110627,
+            0.018100411389202663,
+            -0.0004971948011851863,
+            0.01216566778850985,
+            0.02369448391321832,
+            0.002637001729688794,
+            0.005012948614448268,
+            -0.024213704079796478,
+            0.003420184574622472,
+            -0.0004359900462023086,
+            8.265488264269535e-05,
+            0.016267948861008616,
+            -0.019572032453100303,
+            -0.02216264534756073,
+            0.021871880626992544,
+            -0.012742926070754964,
+            0.011676769226089007,
+            0.0035362870996336194,
+            -0.010732229460038356,
+            0.0006838777488770268,
+            -0.03083438636386314,
+            0.008820746379709355,
+            -0.03216685858285627,
+            -0.032230808828686675,
+            -0.013547681710236494,
+            -0.009994106571421729,
+            0.0008758088338659712,
+            0.028447187234357855,
+            -0.00795317029726568,
+            -0.015797892954136448,
+            0.005363565557248085,
+            -0.011199252697624111,
+            -0.00817162221789202,
+            -0.0007867892785855576,
+            0.005856939999700809,
+            0.009461959355310273,
+            0.005997804620690314,
+            0.011648335019189696,
+            0.007491946658046298,
+            -0.008028800679169293,
+            -0.008837725220736191,
+            0.015889923814605387,
+            -0.0014409716087425222,
+            -0.01176424391171685,
+            0.030430130082762363,
+            -0.013300620185884817,
+            -0.0005592926632214984,
+            -0.0021975669766032472,
+            -0.0015687208316663597,
+            -0.0021652076181411244,
+            -0.014225289143358926,
+            0.01870568404189425,
+            8.334501002782455e-05,
+            -0.014288806218067678,
+            0.0032841743905975634,
+            0.003841780428153241,
+            -0.0036498339483411074,
+            0.008388922960269383,
+            -0.008147275785908849,
+            0.017348147498892,
+            -0.015113395971380741,
+            0.008740452410210708,
+            0.0038335161236289234,
+            0.0043482078469644385,
+            -0.0065755348640655865,
+            -0.01592809184384243,
+            0.009642593996701513,
+            0.012051872087663658,
+            0.02173264012652684,
+            -0.02525660485859878,
+            0.00876161653901566,
+            -0.005374533920781607,
+            -0.014534179652013885,
+            -0.016127866637369738,
+            0.00982303248417706,
+            0.022322910435388785,
+            0.0063086346898091296,
+            0.012139263756190026,
+            0.00013383660542132047,
+            0.005848259030010402,
+            -0.014767198851314148,
+            0.010570158368602513,
+            -0.011355774405066537,
+            -0.018439261453084426,
+            -0.005243587266408389,
+            0.028894120979627323,
+            -0.005492024421080143,
+            0.017938747701972086,
+            0.0021350711250093858,
+            0.029007356490801176,
+            -0.008686781541806358,
+            0.0014883465513564527,
+            0.0039866022469891155,
+            -0.01295271525889559,
+            -0.004272008629260297,
+            0.01976273843646234,
+            0.04882868638601926,
+            0.010204475902027572,
+            -0.0009155109545661691,
+            0.010069071220988797,
+            0.005626418787776309,
+            0.002148101019776153,
+            -0.00805781810841286,
+            -0.030431071067211577,
+            0.008883707378543252,
+            0.005628218215212441,
+            -0.01997825125961566,
+            0.019838234890736068,
+            0.026483365072033395,
+            0.023040623182975964,
+            -0.011585643598171132,
+            0.016539930468533123,
+            -0.010092529089936188,
+            0.0177008735686489,
+            0.028665424114962996,
+            0.013810061902648515,
+            -0.010487150104822674,
+            -0.0020847870330548534,
+            0.006329342599564054,
+            0.004186981876004016,
+            0.00026239408223817384,
+            0.015190614519766204,
+            -0.019155180136775043,
+            0.01005728016778505,
+            0.014725826697964637,
+            0.03322436221319799,
+            8.049930133627564e-05,
+            -0.01645191835557271,
+            0.0016249281996128196,
+            -0.009976155662939693,
+            0.00856399492362911,
+            0.000632427127231252,
+            0.0032242260692672124,
+            -0.009688119824276678,
+            -0.012682779503915694,
+            -0.002851918183031152,
+            0.009403205508474241,
+            0.0021188716361101698,
+            0.002759004011231831,
+            0.007641925150177999,
+            0.0022830062031146204,
+            0.01713024474384216,
+            -0.0011181980841546123,
+            -0.00029141889484139217,
+            0.004601090256674874,
+            0.010509199590505558,
+            -0.01649852835819883,
+            -0.017548369964263514,
+            0.004337237475053179,
+            0.012217494210546133,
+            -0.013159179948224596,
+            -0.013241981632059755,
+            0.015704543608959117,
+            -0.01913575130892795,
+            0.024934102868065366,
+            -0.00942381892138331,
+            -0.004543611336526587,
+            0.000343200128934031,
+            -0.00727538880611703,
+            -0.0038751118497522673,
+            -0.0030064279504724797,
+            0.003984715814505561,
+            0.0019261818110367147,
+            0.007472350699291607,
+            0.008472369788801792,
+            0.014568547873264247,
+            0.006043895352236181,
+            0.0032580830408778395,
+            0.009109104498825864,
+            -0.014523846294121576,
+            -0.014800649887278973,
+            -0.002704791598011488,
+            -0.008895394038700809,
+            0.00902016867266151,
+            -0.006607182154874112,
+            0.00845408382182461,
+            -0.015501867835442603,
+            0.015573642225745032,
+            -0.012200552219016517,
+            -0.0008900215712478553,
+            -0.009666233545144905,
+            -2.7470277092642562e-05,
+            0.008191697904514857,
+            0.014605287653054787,
+            0.0070552942843382215,
+            -0.031921405225964206,
+            0.0017403427534633655,
+            -0.008846568352320488,
+            0.03446319360336241,
+            0.002612606662691094,
+            -0.01060863123202008,
+            0.020352181356568008,
+            -0.01592124132155158,
+            -0.00027704668715660693,
+            -0.01602075523652062,
+            -0.004797557413047212,
+            0.00707939446796679,
+            0.016399017978990946,
+            0.02759049512664933,
+            -0.005697020356865688,
+            -0.007995305872076221,
+            0.006384047872862793,
+            0.029590696670334232,
+            0.012246955508890425,
+            0.024748351124415785,
+            -0.007633712004672087,
+            0.014348228716100414,
+            0.007773260552660029,
+            0.0076045488653348975,
+            0.004321689250440021,
+            -0.002591801797145718,
+            -0.011698574606499644,
+            -0.014974971814360001,
+            -0.0037563183330623384,
+            -0.016987915346053498,
+            0.00012247833530362867,
+            0.021344400276501117,
+            -0.00475351423959752,
+            -0.0033847161875730026,
+            -0.01874186437694691,
+            0.0033461670218611346,
+            -0.010029445302549983,
+            -0.00367421246775698,
+            -0.013133469163793961,
+            -0.0254122035745965,
+            -0.018556369080030412,
+            -0.002155244024999317,
+            -0.00929309186646207,
+            -0.015194269781263309,
+            0.0070354690219415535,
+            -0.016305117309271634,
+            0.0080837166473439,
+            -0.00448473417024141,
+            -0.016508775226236883,
+            -0.00958925666995396,
+            0.01100335153658195,
+            -0.0041819993634961685,
+            -0.01929493648733792,
+            -0.01282300765645703,
+            -0.011300260564838231,
+            0.0051733704121938884,
+            0.005996780699975213,
+            -0.0038111648304178544,
+            -0.0028279639348479987,
+            0.016640121845366295,
+            -0.010197365811240557,
+            0.009878730312873492,
+            -0.03335474019383245,
+            -0.03551826040406959,
+            0.004497847402931096,
+            0.01507007013703286,
+            0.0013085881889541252,
+            -0.02078680101065954,
+            0.010806315988966003,
+            -0.016499186313125556,
+            -0.00858013792516617,
+            -0.008522534127498645,
+            0.009649521088411411,
+            -0.007283003380127847,
+            0.01299542385123007,
+            -5.3226845309105124e-05,
+            0.030477683597373687,
+            0.0074153455615142755,
+            0.004161881195438522,
+            -0.00679081048132963,
+            0.008354912600159931,
+            -0.001621776416108105,
+            0.004080191054849395,
+            0.02120524874226696,
+            0.012440848502230418,
+            -0.00218041171518331,
+            0.006458415098595872,
+            0.005678055538564355,
+            -0.017186537377121235,
+            0.02507960402044456,
+            0.024678178106806115,
+            -0.008205288425409454,
+            0.004108402466376321,
+            0.005783335302169668,
+            -0.008846663924040241,
+            -0.006897707231869481,
+            0.006527795262026734,
+            -0.018474181903007085,
+            -0.0008721381678008483,
+            0.010598084358464908,
+            -0.02255441909004843,
+            -0.004840511019512733,
+            0.021745651362109403,
+            0.004631567631335321,
+            -0.018558972627958834,
+            0.02764788406651861,
+            -0.004942361238163609,
+            -0.0027290999934532494,
+            -0.010876135202650048,
+            0.01671497143541799,
+            -0.012173527207375091,
+            0.008443317346655462,
+            0.011021541861818043,
+            -0.006295639322287843,
+            0.003083685676540999,
+            0.008277206401503182,
+            -0.004504302913715969,
+            -0.0068075424927125676,
+            -0.0010508416962215848,
+            0.010167160518175114,
+            0.0036980815941666336,
+            -0.022704963865525812,
+            -0.006020073261002279,
+            0.02934227992246887,
+            -0.015082922782242507,
+            -0.0050916398557337705,
+            0.005376294716056329,
+            0.007709342013963906,
+            -0.003476236561444464,
+            -0.0023399991233097337,
+            -0.00787486645740744,
+            0.010356414979428607,
+            -0.015950083845400274,
+            0.02076360898078935,
+            0.025608263494550438,
+            0.024504236990840842,
+            -0.014814235990238182,
+            0.010828584133720797,
+            -0.011703992394890898,
+            -0.011047470905668532,
+            -0.015542835222173864,
+            -0.001658296303013357,
+            0.001970208852236451,
+            0.013107754130616248,
+            0.009666887546609378,
+            0.0235853683657408,
+            0.006473544089109194,
+            0.008973300312205858,
+            0.014118601565410305,
+            -0.003902202854013708,
+            -0.018303438244719996,
+            -0.014584226585460755,
+            -0.011977613285028946,
+            -0.01756345491020528,
+            0.0004563602312336091,
+            0.020461659996271687,
+            0.012417744207683839,
+            0.024136626592375227,
+            0.002119213701333967,
+            0.013759902247467922,
+            -2.9525392347047554e-05,
+            -0.000892319792935443,
+            0.00045783825646242434,
+            -0.005238483120312843,
+            0.019255756091120846,
+            -0.004599752813817219,
+            0.01656589522110913,
+            0.0006443773300684925,
+            0.007133940816771615,
+            0.013554761340256566,
+            -0.01704388669741371,
+            -0.003443100892108368,
+            -0.010029620309829285,
+            0.000941391966877521,
+            -0.019633111750884034,
+            0.01934571074127927,
+            0.010273330977233852,
+            0.0017121345993533963,
+            -0.018327189396014198,
+            -0.0029706385099949693,
+            0.03448635131998772,
+            0.019458999453011265,
+            0.0050734354363614525,
+            0.02632443373898792,
+            0.013018762352749236,
+            0.014443611411502168,
+            0.010153370950346075,
+            0.02356766253713842,
+            -0.023479811300626968,
+            0.02095255383200973,
+            -0.011590347850340786,
+            -0.007414136601134069,
+            -0.0035774888741646705,
+            0.0025141508183687115,
+            -0.013588691733097813,
+            0.00982877622088977,
+            -0.0017916539399655826,
+            -0.025386211797192578,
+            0.009164316443206016,
+            -0.008811052979979271,
+            0.010464442616928296,
+            -0.007774569252594923,
+            -0.0002806136360286729,
+            -0.0127084026190163,
+            -0.015184317235990752,
+            0.001259201880504347,
+            -0.009158429132281081,
+            0.023908150915412617,
+            -0.021210848953875233,
+            -0.01321384014436847,
+            0.013997502713857512,
+            -0.006869443694669701,
+            -0.011364239609894453,
+            -0.00374449980963666,
+            0.023673050288879224,
+            -0.009243341446512421,
+            -0.01359367519009039,
+            0.014464166953530989,
+            0.003834735113400469,
+            0.0059713317014342395,
+            -0.007508862694272452,
+            -0.008428705474135382,
+            -0.003280788036976103,
+            0.005998840043285335,
+            0.026615077617284676,
+            -0.014059930972065399,
+            -0.0032833264512892022,
+            -0.009189999799498163,
+            0.004118204537632102,
+            -0.02359889635281631,
+            0.02771075351384059,
+            -0.008871395303200375,
+            0.021705177758560575,
+            -0.0015924449308442096,
+            -0.006239188158271628,
+            -0.024231036074349822,
+            0.027522945599928003,
+            0.003683870966132199,
+            -0.010221180899173243,
+            0.0012768725460662862,
+            -0.052005292736885915,
+            0.016636109796947025,
+            0.008429895913163985,
+            -0.019507994429834353,
+            0.003697519981610208,
+            -0.004773584856005853,
+            0.018983847580558853,
+            -0.022966993228850697,
+            -0.014078986007109447,
+            0.008283766047449966,
+            -0.01153285124362746,
+            0.0031439180465649996,
+            -0.02527749501821793,
+            -0.005754537200570427,
+            0.0011760345167229269,
+            -0.005276936074280004,
+            -0.001705933750221904,
+            -0.020022761608132548,
+            -0.025972227850795115,
+            0.017666017379157135,
+            0.015820875579600418,
+            -0.002278262448951789,
+            0.01320818011561616,
+            0.008709899932124187,
+            0.00836573261091119,
+            0.016034263394940008,
+            0.007124509234103267,
+            0.0081473711608784,
+            0.045498533239850446,
+            -0.019846323044145485,
+            0.007371629414298193,
+            -0.0014854149174929534,
+            0.013951741898629716,
+            -0.030467924817555014,
+            -0.0055144480998344705,
+            0.0062802744565489945,
+            -0.0049373689340857915,
+            0.03510454786147091,
+            -0.017326094216003073,
+            0.01694160526735264,
+            -0.0026075809870445238,
+            0.0007420394265014909,
+            -0.02042003877666893,
+            -0.026156937726134134,
+            -0.014294938857971834,
+            0.0011857406889443987,
+            -0.0010047941261608975,
+            -0.011946583993069242,
+            0.006797001710574735,
+            0.004917161312925866,
+            -0.0020713379451245515,
+            0.015652523103715316,
+            -0.02952061566240878,
+            0.007356800041369564,
+            -0.010733578620243498,
+            0.020008504830359524,
+            -0.00990331048284697,
+            -0.0013319400098308093,
+            0.007236960885181047,
+            -0.004122868383988748,
+            0.013216043472071927,
+            -0.0038337251451538677,
+            0.004227115653354611,
+            -0.005221321656766545,
+            -0.01794343569722877,
+            -0.0016435261495179767,
+            0.021429008432348098,
+            -0.008349684096540855,
+            0.031864497493526205,
+            -0.01264743878037828,
+            0.009174982784686633,
+            0.010692990337462762,
+            0.0006628831036334152,
+            -0.005286285851702483,
+            0.009052971814211658,
+            0.0021431970274545893,
+            -0.008690520448278274,
+            0.006934310986093342,
+            -0.018349449937310143,
+            0.008494451785507706,
+            0.03394632739506833,
+            0.0018511328780472266,
+            -0.0007519661670070526,
+            -0.007266663179613392,
+            -0.01335844201823745,
+            0.018540713029383066,
+            0.005488273906733518,
+            -0.002546531152114063,
+            -0.027536660869590543,
+            0.02007746621157705,
+            0.0012265798861291223,
+            -0.004755325069578531,
+            0.012466557901411086,
+            0.0020495867632654885,
+            0.019989531151046736,
+            -0.010171678639113316,
+            0.030757957899447096,
+            0.010050036220124393,
+            0.002959833051227505,
+            0.0038556928367700113,
+            -0.036655749074851536,
+            -0.01652152778069314,
+            -0.00019357502472740266,
+            0.014271581054865031,
+            -0.012430647252826173,
+            -0.022432580814844155,
+            0.0010346169306977334,
+            0.01148004051449691,
+            -0.0033898642762881046,
+            -0.009650665305271706,
+            0.01560757541276278,
+            -0.020342213902670575,
+            0.006701302394408393,
+            0.007693672045981921,
+            0.0287100214318104,
+            -0.0020735384554022153,
+            -0.0012324838849990056,
+            0.000626416803904792,
+            -0.0055669860801192935,
+            -0.0008037366126870824,
+            0.017251369734470945,
+            -0.013661316947002466,
+            0.0175926211217534,
+            -0.012735241490550407,
+            0.0064713656749838585,
+            -0.0021092378432037563,
+            -0.010629579860050812,
+            -0.020114807971202787,
+            0.0029609495764307357,
+            0.009378003163375899,
+            0.006920144698428806,
+            -0.010438984881449953,
+            0.010715409812532968,
+            0.012197673689706352,
+            -0.00753962861484439,
+            0.014519905132928041,
+            0.022173989895384192,
+            -0.007687503607783572,
+            0.013633321414009471,
+            -0.0041446334728745095,
+            -0.014056264816568229,
+            -0.009985463824440833,
+            0.023929750617160402,
+            -0.0005563806942667597,
+            0.008756997373618809,
+            0.00388125051641897,
+            0.0007856447663040696,
+            -0.011226530517316183,
+            -0.001653447552601681,
+            0.019294448360333633,
+            0.011831934675073843,
+            -0.0013888534822008192,
+            -0.0010525764356699885,
+            -0.0014566936525653407,
+            -0.009427557739483988,
+            0.02049188741459728,
+            -0.022030783927622997,
+            -0.0019604940727076573,
+            0.00461018148439991,
+            -0.022589816803875618,
+            0.0001863117949313752,
+            -0.024570217962989905,
+            -0.01569952241187498,
+            0.011763033781122343,
+            -0.0022530631823737554,
+            -0.016028810867854907,
+            -0.0003242701040942633,
+            7.424127301806556e-05,
+            -0.007815789940508527,
+            -0.004135500768181258,
+            0.006115909992703774,
+            -0.00034193217510010117,
+            -0.0032262189374874842,
+            0.013430154310506884,
+            -0.009996622196870008,
+            0.015431528911575157,
+            0.0067029980830614145,
+            -0.015702167721800483,
+            0.0021406610839557003,
+            0.018500219544264753,
+            -0.0058910087864916085,
+            0.00787021425357441,
+            -0.014681907249972176,
+            -0.0037495551894426303,
+            -0.005787661351944086,
+            0.006051351977821761,
+            0.024849195780885833,
+            0.012969434450643372,
+            0.01139340445077682,
+            0.017405361843075933,
+            0.004325000715054734,
+            -0.01572297405297993,
+            0.0007416798945675923,
+            0.01682727189828363,
+            0.010464036837245312,
+            0.009804326468113703,
+            0.0008975556667684365,
+            0.0043704860361487635,
+            -0.005142176468800175,
+            0.009317064072855551,
+            0.02915482449687158,
+            0.0013434689612997423,
+            0.008696141010760192,
+            0.017276730345147134,
+            0.027038741396131265,
+            -0.0035088997194497725,
+            -0.007469614319788499,
+            0.008739942310300076,
+            -0.009657991604173173,
+            -0.001081629174976152,
+            -0.017000351183542026,
+            -0.012310275736750868,
+            0.016510505259358398,
+            -0.006084627062230492,
+            0.02848913486719956,
+            -0.0010655806791907225,
+            -0.02026498848361482,
+            -0.01565505608194376,
+            0.001404549479440637,
+            -0.01362132895080652,
+            0.005287566256051299,
+            0.014187265685427415,
+            0.01917046513020831,
+            0.004687191682875024,
+            -0.022258337721085656,
+            0.037364962197399806,
+            0.01757108963349427,
+            0.019377870795760123,
+            -0.019422840506328382,
+            -0.006159311237412833,
+            -0.009077095160676276,
+            0.02116697012944498,
+            -0.00880692095488465,
+            -0.009875674856043954,
+            0.024014332004451508,
+            0.017812606845656484,
+            0.005679806286464216,
+            0.006736609904079586,
+            0.0020048663439319585,
+            0.0025414630253661444,
+            -0.015831593851012047,
+            0.017957902498542212,
+            -0.005858125698347523,
+            -0.0058068554681761945,
+            0.017529514573571298,
+            -0.005989581792993925,
+            -0.0030306021203732347,
+            0.00015839997806822045,
+            -0.019697052145531482,
+            0.011618112886207201,
+            0.008256007601576028,
+            -0.017229774429089857,
+            -0.0014757560402567304,
+            -0.0002404480326945295,
+            -0.014118694740134205,
+            0.02004885146995733,
+            0.01694893356262835,
+            -0.0013248884139405398,
+            -0.0051752815372013315,
+            0.008223868147393114,
+            0.007000679132597393,
+            0.005256194890343006,
+            -0.019496098876977206,
+            0.005230597149342019,
+            0.005103909294824105,
+            -0.0014716649591412774,
+            0.010274346323051634,
+            -0.009030594825985526,
+            0.033740380586893826,
+            -0.006505788197604522,
+            0.013065658941136721,
+            -0.0043634731684193745,
+            -0.0055529072007985125,
+            0.001816020676232658,
+            0.013054468478598695,
+            0.009753501030700457,
+            -0.00970446155762939,
+            -0.004172354962129865,
+            0.018697982092775737,
+            -0.011983156146516074,
+            0.0013300791598046338,
+            0.01564072865804035,
+            -0.004858814041438713,
+            0.004708235083775013,
+            0.008635204790345423,
+            -0.004489505237783172,
+            0.01801512858970922,
+            -0.016482572999490366,
+            0.0023953436660054026,
+            0.010694869684962088,
+            0.02390864159830096,
+            -0.019152187046410703,
+            0.021786596486708207,
+            -0.017951050280323393,
+            0.026538582634436314,
+            0.014484894489694262,
+            -0.0035222259457311444,
+            -0.01946200885147006,
+            -0.00850690913394181,
+            0.011654565129063417,
+            -0.021423803876241532,
+            -0.0007985948078290641,
+            -0.0007290222944686444,
+            -0.021367628543092133,
+            -0.011189364017231132,
+            -0.009961779129778331,
+            0.010701637745810138,
+            0.001918633898887013,
+            0.010380983868185359,
+            0.008002182827196827,
+            0.010784760574518087,
+            0.014405640442760822,
+            0.0008958107769384412,
+            -0.010743001445440941,
+            -0.023795589003655273,
+            0.016987708698679597,
+            0.00438463274382093,
+            0.0007862212695139549,
+            -0.01627468994515667,
+            0.011936651269737248,
+            0.013250643815768723,
+            -0.009385605548926286,
+            0.009121882469425302,
+            -0.03716901103204463,
+            -0.004552808410398574,
+            -0.0035704811925430366,
+            -0.0020309197365173475,
+            -0.00908678158053104,
+            0.01559445415882485,
+            -0.013682507836005851,
+            -0.0009590922904844632,
+            0.00011696985057536548,
+            -0.0074756526903548915,
+            -1.5467890432254378e-05,
+            0.020649928837031005,
+            0.008438785939908428,
+            -0.0037382105257999757,
+            0.010376213224654074,
+            0.008826031303751312,
+            -0.013900359449287676,
+            -0.01527474712786084,
+            0.019035366035884273,
+            -0.018776918681908793,
+            0.0063098418544296165,
+            0.01677916431403883,
+            0.020778813096283988,
+            -0.02891862501830075,
+            -0.0007231469961304435,
+            0.022093939332021185,
+            -0.0012807635799950273,
+            0.006370841109790851,
+            0.008781853138377092,
+            0.006754082980602544,
+            0.0029102159062355374,
+            -0.003711173531879644,
+            -4.843770941622283e-06,
+            -0.003940474588075707,
+            0.007155539022862352,
+            0.004366433868349658,
+            0.036453728071874844,
+            -0.013661129348165794,
+            0.002541308805735001,
+            -0.0058838191638835155,
+            -0.00347401304572066,
+            -0.00593469688967835,
+            0.004407665856950772,
+            -0.010055727994798024,
+            0.004330827357536078,
+            -0.001570861535288093,
+            -0.0003731129422346693,
+            0.004883969598300302,
+            -0.001701673379646408,
+            0.007036892564844683,
+            -0.011863650867472625,
+            -0.0003979727750011258,
+            -0.020174851412609312,
+            0.021117284557875267,
+            0.00011666551282792189,
+            0.003662462447326642,
+            0.03723741246290771,
+            0.00476064510662738,
+            -0.00238355131307445,
+            -0.012880057949435798,
+            -0.013551338766321525,
+            -0.012986444917747435,
+            -0.020703821629987552,
+            -0.0036665673548525388,
+            -0.021006311026247164,
+            -0.0052061406234528915,
+            0.010362862594785442,
+            -0.011540184420787572,
+            -0.004023659825622843,
+            -0.015881589631889473,
+            -0.006531981517815174,
+            -0.03114211633687585,
+            -0.01124607488650473,
+            0.007493228366452009,
+            -0.019149285949049886,
+            -0.0039932870926865805,
+            0.026850121590606017,
+            0.001095360635760988,
+            0.01387696424607248,
+            0.0038058248676879625,
+            0.008117686785954085,
+            -0.0045937472017383955,
+            -0.0005669025036141417,
+            -0.01928244877716143,
+            0.003391980330529026,
+            0.008525354854576925,
+            -0.00361507909530282,
+            0.0070008404454106905,
+            -0.008835727374865623,
+            -0.016950017602980382,
+            0.0021654956715219517,
+            0.009334379529385142,
+            -0.006639140003365139,
+            -0.017587472740268683,
+            -0.011049929670847685,
+            0.010401189523318744,
+            -0.02283429470064187,
+            0.006257145380266984,
+            0.016439581169238607,
+            -0.009363030375624585,
+            0.004651575425738153,
+            0.019809183749755425,
+            -0.011810674576781702,
+            -0.005811328501528397,
+            0.008471992070998505,
+            0.013891259999721588,
+            0.0095606134736748,
+            0.02954498207274139,
+            0.0026596809804008996,
+            0.000561870602466584,
+            0.004889263753490895,
+            -0.004277061142504552,
+            -0.017423200644235665,
+            -0.006243787897080996,
+            0.0038507494787521856,
+            0.012873996671952184,
+            -0.009879417960879032,
+            -0.00048575277472139055,
+            0.01682604801673206,
+            0.004530939990187663,
+            -0.0033901248079550284,
+            -0.011868399508738274,
+            -0.020080331108085103,
+            0.0015580259874809898,
+            -0.01362200460317948,
+            0.015354347040143093,
+            0.003390601069331352,
+            0.0015284610287850575,
+            -0.004239880336806429,
+            0.01828645373058079,
+            -0.005013418527935815,
+            0.012749833447770729,
+            -0.006148890994661865,
+            -0.004587831231448011,
+            0.007986533783001116,
+            0.009166578151006778,
+            0.0004939857995863664,
+            0.05421019830078428,
+            0.002838056604877719,
+            0.01333205648597435,
+            0.013419127695838135,
+            -0.014523733369484934,
+            0.014086525147734823,
+            0.012688569853552613,
+            0.018497344547821693,
+            0.013037819487160016,
+            -0.012353776077273462,
+            0.017354801119939964,
+            0.0024084723134891487,
+            0.003909681065813095,
+            0.0022202438064449653,
+            -0.004528576079628071,
+            0.019594683950225115,
+            0.004070222328547378,
+            -0.0074362443729372105,
+            0.004453618452623641,
+            0.005007543168224267,
+            -0.013105524222311019,
+            -0.0112958915514782,
+            0.027105208848863218,
+            -0.01219417557128091,
+            -0.018749971743148304,
+            0.006706594269933445,
+            -0.02008458683853808,
+            -0.03763686535076864,
+            -0.0021332074289244973,
+            0.009992361098878196,
+            0.002795183524495193,
+            -0.00253695604043486,
+            0.01614298064601032,
+            0.016730443047719683,
+            -0.016985296937992726,
+            -0.018245902728910347,
+            0.0025257780765481825,
+            0.0016333325293570182,
+            0.01861807236979066,
+            -0.023168663546030196,
+            -0.006378864850313035,
+            -1.3795465857380962e-05,
+            -0.006829008132963306,
+            0.004850306884033312,
+            0.004703280112191218,
+            -0.0071714759384722965,
+            -0.0027773519096287217,
+            0.002256846645341137,
+            0.01975383140755998,
+            0.013198754584066554,
+            0.01572971337890945,
+            0.002600741959577911,
+            -0.005033508062449482,
+            -0.016817359058724685,
+            -0.005450011991586895,
+            -0.013478330855299777,
+            -0.009091854345715177,
+            -0.003042052092498012,
+            0.008217376329142566,
+            -0.025053469059115735,
+            -0.011965083373220982,
+            0.0119294363235832,
+            0.00391377331459168,
+            0.0003099203772539202,
+            0.007607672728136279,
+            -0.003830132495364688,
+            0.010331053246451333,
+            0.013823890473448383,
+            0.0076112887952959766,
+            -0.004723377809559611,
+            0.009600411526173948,
+            0.011460781436250457,
+            0.006015477068316837,
+            -0.005072711638895421,
+            0.023859899048048694,
+            0.01821920335673539,
+            -0.01680365156728093,
+            -0.002149107383573353,
+            -0.0027829676356423997,
+            -0.01884308738652242,
+            0.009665830801645276,
+            0.00880322398336952,
+            0.008308268605449346,
+            0.003906369351868086,
+            0.00974531700889854,
+            -0.013875904819275925,
+            -0.0050570811397453235,
+            0.002870120865445989,
+            -0.005729556771404781,
+            -0.01901337075686543,
+            0.000297242372701423,
+            -0.011987192261104132,
+            -0.011327157550660895,
+            -0.01409710922165158,
+            -0.003221240464411861,
+            0.005283472511900512,
+            0.021472187640986972,
+            -0.004410881709548286,
+            -0.001855852749039012,
+            0.01351697599517034,
+            0.010060125186506398,
+            0.0021658738823899905,
+            0.0002206474541239167,
+            0.005418828324605768,
+            -0.0027919130944783397,
+            0.0008094042430809068,
+            -0.0015826797905407809,
+            0.021089276765434876,
+            -0.02016611086095107,
+            -0.009413977569213423,
+            -0.009288078098692112,
+            0.005842197094339927,
+            0.028793032462603636,
+            -0.007165634502373454,
+            -0.010104002083017078,
+            -0.01994315293556678,
+            0.0003786527922888834,
+            -0.07810609175343784,
+            -0.005210636756258548,
+            0.001870513618910238,
+            0.03126380113105983,
+            -0.010817888476436834,
+            0.0033673768058629823,
+            -0.002204487854177595,
+            0.014542299655712928,
+            -0.001760107965206874,
+            -0.0013283746982194259,
+            0.0071954806737858056,
+            -0.005640400818010697,
+            -0.006210989097047217,
+            0.005904163661619133,
+            -9.945941028848348e-05,
+            -0.010266938705916303,
+            0.00057263080955709,
+            0.02209498395848068,
+            0.0159398346608872,
+            -0.006575439759804175,
+            0.01816748552021211,
+            -0.00719294553484007,
+            -0.009838738592324234,
+            0.005639288780502127,
+            -0.0019007683459126327,
+            -0.015369619176147573,
+            -0.011475977270803784,
+            0.009862808084641291,
+            -0.030069890644150284,
+            -0.00010725753918890802,
+            0.00943698424646706,
+            -0.013540245857270492,
+            -0.008293042205008774,
+            0.007212974960781723,
+            0.006708343474532583,
+            -0.0012017968151025369,
+            -0.0078026000239507145,
+            -0.02527526731750743,
+            0.0027578400032925625,
+            -0.002446811971857853,
+            -0.004899702349784173,
+            0.0019960465173086183,
+            0.001479503476855694,
+            -0.012645905858466798,
+            0.01216748847452763,
+            -0.013378877368040743,
+            -0.012053251571418766,
+            0.012508842412556306,
+            -0.00027167241745418584,
+            0.01682680930386839,
+            -0.020607299598265426,
+            0.0010702151063620052,
+            -0.028075439824932304,
+            -0.007671848153932823,
+            -0.015486630894165054,
+            -0.0009776136330459065,
+            -0.0012939921569540803,
+            -0.03481880531471923,
+            0.012889447665794033,
+            -0.005200149829198305,
+            -0.006094702500154477,
+            -0.00459778303079732,
+            -0.024583504817899775,
+            0.006072281238752663,
+            0.00607350167867964,
+            -0.006506508970816635,
+            -0.016816439635445445,
+            0.008428048142588516,
+            -0.0011040593013238872,
+            0.004873670535200424,
+            -0.007475800310154908,
+            -0.0005638920132076605,
+            -0.013047135093134946,
+            -0.005463454269764494,
+            -0.0008522562324122834,
+            0.01827123304871433,
+            0.0044055851550317835,
+            -0.014526381222031981,
+            0.016862445236481526,
+            0.013250692156820124,
+            0.01602714293656341,
+            -0.0071089293427753115,
+            -0.0004265739315250755,
+            -0.02750630108787212,
+            -0.00327225092954903,
+            -0.023687644154808972,
+            0.00041213204365507744,
+            0.009447660230507965,
+            0.0203905264889617,
+            0.012694421477479161,
+            0.004620617778996077,
+            -0.001557511764988156,
+            0.0066751090724502,
+            0.02696959560423252,
+            -0.01099183126742224,
+            -0.006969111598423376,
+            -0.01884504192789845,
+            0.01113023717919988,
+            0.025135748541199255,
+            0.003381303139273845,
+            -0.011317957771935042,
+            0.008112598754655146,
+            -0.0009247845526290428,
+            -0.010302980598863016,
+            0.02889587382004902,
+            -0.0391134481322538,
+            -0.015599591108049343,
+            -0.004605824192962023,
+            0.007406520608110251,
+            -0.015596917115435033,
+            0.00770468362805995,
+            -0.012614339656105264,
+            0.001699049329817941,
+            -0.005851477164035241,
+            0.03535087340659861,
+            -0.0031950960727572925,
+            0.0006909856233673865,
+            -0.003395233050301427,
+            0.003531135044131077,
+            0.012355948349937309,
+            -0.004444221317023453,
+            -0.030726300006909638,
+            -0.0022983766869702346,
+            -0.011198252640110788,
+            0.009274666307515955,
+            -0.016207714291658314,
+            -0.017342435976846468,
+            -0.009424737744066259,
+            -0.021827243671837953,
+            -0.012403084982789827,
+            0.029747453318486338,
+            -0.005824001042844527,
+            0.01747569398142269,
+            -0.005219060548639168,
+            -0.008365003057800934,
+            0.00021272411948388247,
+            0.01344335658390732,
+            -0.028498240040284714,
+            0.00041963499785238845,
+            -0.002322931702641349,
+            0.0126393550661767,
+            -0.001595134597082556,
+            -0.00983073906977773,
+            0.006903313245694627,
+            0.0010522208685011098,
+            -0.015648940549713764,
+            -0.0007573245570947632,
+            -0.008533504878052644,
+            0.020475329445149498,
+            0.00027791387213409814,
+            0.001074213629646005,
+            0.013864331671732542,
+            -0.02102352993295736,
+            0.01576938071930521,
+            -0.021722775924283996,
+            0.01893267442161547,
+            -0.006577103101038035,
+            0.005772016664149475,
+            0.008283223343661555,
+            0.014068789332250016,
+            -0.014030042849491258,
+            -0.00663752723080222,
+            -0.034436220972634726,
+            -0.0122621879114314,
+            -0.001946072976826902,
+            -0.005308941654794638,
+            -0.014773123927909993,
+            -0.0011054822266082056,
+            -0.026205461509573584,
+            -0.006460373065662098,
+            0.012822188046808264,
+            0.019900312098359022,
+            -0.008409070094265829,
+            0.014163132434889265,
+            -0.006743711872055091,
+            -0.0028230771810882492,
+            -0.012652021749331811,
+            -0.018061180380009395,
+            -0.01083672324723357,
+            -0.01890966636827258,
+            -0.0018635408070944168,
+            0.004709222372611061,
+            -0.015760618199235518,
+            0.011435131737642627,
+            0.03198874888333405,
+            -0.01567271472650171,
+            0.008243406188694909,
+            0.016411496328333205,
+            0.0034236971031973503,
+            0.011728079139322837,
+            0.0038311415233052795,
+            0.004621789116267879,
+            -0.011109450058055078,
+            -0.011473158133621661,
+            0.006524490579223986,
+            0.00954511918454185,
+            -0.004913142601134428,
+            0.024347072255007807,
+            0.0008745419206472714,
+            0.005045794664611216,
+            0.005304506902446091,
+            0.011488187043525824,
+            -0.013135271908307747,
+            0.019746484188715552,
+            -0.002055210086078267,
+            -0.015571665869889405,
+            -0.01535499169234956,
+            -0.01105419264826655,
+            -0.018626399757822328,
+            0.0017108083455869296,
+            0.0181439079855921,
+            -0.0012493542718277685,
+            0.012137753522284387,
+            -0.00924460959150499,
+            -0.005019437549649743,
+            -0.0027156377206998474,
+            -0.038211954658821295,
+            -0.0021878970332906457,
+            -0.018809487178512628,
+            -0.02498945889484935,
+            0.03505216568400238,
+            -4.26477081213836e-05,
+            0.00997643255528116,
+            -0.010310197159232386,
+            -0.0002454902098986129,
+            -0.011065403358802515,
+            0.014954998647970473,
+            0.004624123403100598,
+            0.008872923678983478,
+            0.008560280683876702,
+            -0.016419500515801093,
+            -0.002997557798375716,
+            0.007472390947008284,
+            -0.00374822278027057,
+            0.012680407710694952,
+            0.015353684671120731,
+            0.020886474047003733,
+            0.011161262397502876,
+            0.002045862158732102,
+            0.00887271411363814,
+            0.009563361150297699,
+            -0.0003513237086717377,
+            -0.010459621030424415,
+            0.015419039131459582,
+            -0.02204848739950703,
+            0.012172257374410352,
+            0.04346634265794738,
+            0.019197433209794286,
+            0.029488039318675732,
+            -0.02272810439246353,
+            0.01448092574459596,
+            -0.003801043409441206,
+            0.009476229966122152,
+            0.0049309968883087685,
+            0.02346191840355209,
+            0.003973840862678076,
+            0.0073546443568171405,
+            0.010647581332452525,
+            -0.00972051412996416,
+            0.025644022498404723,
+            -0.00846669916801108,
+            5.1518540791212095e-05,
+            -0.004407265804145216,
+            -0.026400029105519137,
+            0.01615193008014211,
+            0.0081722536401398,
+            0.0010702314763197463,
+            0.0010263623479474385,
+            -0.0019566604229607643,
+            0.007982949510899019,
+            0.007079322685771456,
+            -0.015274148711633536,
+            -0.003886053737590683,
+            -0.018527455865296014,
+            0.0012864569064749438,
+            -0.024337133883771334,
+            0.010466651250597847,
+            -0.017568706814132484,
+            -0.014822354159709854,
+            -0.021666681078831483,
+            -0.013232016867264097,
+            -0.02011653224168991,
+            0.0035858355517491515,
+            -0.008632755352062642,
+            0.013025728891203198,
+            -0.016962939287139683,
+            -0.021494971929779273,
+            0.008430887589398292,
+            0.00870802359711456,
+            0.007481282677955534,
+            0.009728368603458082,
+            0.014996157358749115,
+            0.01736292711410622,
+            -0.022712386143092265,
+            -0.023163001611599246,
+            -0.006682209717364464,
+            -0.003531495447167166,
+            0.006762379833125297,
+            0.00933681196321128,
+            0.012884185500343249,
+            -0.007077895849501418,
+            -0.007554762054641417,
+            -0.013283067570021503,
+            0.006197966297581871,
+            -0.003993067576677973,
+            -0.009464789349849612,
+            -0.0015662739825834614,
+            -0.0061389795593180686,
+            0.002753664732282926,
+            -0.0001791088888074743,
+            0.0007595261237141144,
+            0.017122834676182343,
+            -0.008176319370751018,
+            -0.009787747261447871,
+            -0.02027228453943018,
+            -0.010683273300243025,
+            -0.015604745521156111,
+            -0.0016279406232527138,
+            -0.006506713191861036,
+            -0.014420323628589382,
+            0.010888206569219648,
+            -0.011930708571093279,
+            -0.00887210117616784,
+            0.010807147705360856,
+            0.0073900910661282935,
+            -0.003431679006289002,
+            0.037804020313161775,
+            0.0011945597874820288,
+            0.00778778394048267,
+            -0.003366543306746973,
+            0.005296727102519836,
+            -0.017761251541598876,
+            -0.026382400396890007,
+            0.0010357716464853195,
+            -0.01775777668381489,
+            0.010807713750049491,
+            -0.008515368407459745,
+            0.0016051389544388124,
+            -0.00034958600424912995,
+            -0.009585854918521076,
+            0.0036549720706351473,
+            -0.008110927907950148,
+            0.004212219431447508,
+            -0.03507890639477636,
+            0.011409377585617926,
+            -0.01819120811524444,
+            -0.008962242411035718,
+            0.01710177594046791,
+            0.037560936862121835,
+            0.0028513388998398104,
+            0.00030058353811717895,
+            0.0014792184333546447,
+            0.004427921639326168,
+            0.021438127785303025,
+            0.006339483221420334,
+            -0.001187747087595264,
+            -0.02237982599829809,
+            0.0062435650294483695,
+            0.002200578793697352,
+            0.010092828474934744,
+            0.023105023225115266,
+            -0.016797326021070008,
+            -0.015378520287395565,
+            -0.012732527281000097,
+            -0.010797463778182385,
+            0.009881499418260473,
+            0.02801767062785128,
+            0.01811139137849365,
+            -0.017358098300028903,
+            -0.020207173948076473,
+            0.0016242901745373126,
+            -0.026331540602868762,
+            0.007269751661479859,
+            0.11940631114242901,
+            -0.0045382135734049305,
+            -0.005994407567723939,
+            -0.008679712231646732,
+            -0.004482553208064535,
+            0.016709571294385293,
+            -0.013387043582971358,
+            0.069179848974036,
+            -0.007221545718220735,
+            0.0037392229364642915,
+            0.0028916224964057385,
+            0.00462729158956839,
+            -0.009727339449635306,
+            0.011570145487622916,
+            -0.011287365791681587,
+            -0.012991994493845187,
+            -0.013076194601468383,
+            -0.0180830261130804,
+            -0.007272768410218907,
+            -0.004155564814758654,
+            0.004204993170487392,
+            -0.005525575983182816,
+            0.008051108079520902,
+            -0.017551645696802648,
+            -0.010800070994485792,
+            0.013007176804761548,
+            -0.0023627485852619817,
+            0.005390986239687555,
+            -0.029668390278675554,
+            -0.0007650391259819867,
+            -0.0002446394046947245,
+            0.021280292985495608,
+            0.003137771886236958,
+            -0.0019077298815862946,
+            -0.010810159675564049,
+            0.007040107942856957,
+            0.023835993694726953,
+            0.021875366566457236,
+            0.0180511566130312,
+            0.012135351541044354,
+            0.020483975362807574,
+            -0.008667079096167644,
+            0.026324312888512866,
+            -0.001552207146726937,
+            -0.00397810465151493,
+            0.0077577136352257004,
+            -0.009849600842081875,
+            -0.00816392268697345,
+            0.00677353736462267,
+            -0.017288900282504148,
+            -0.020418071729964713,
+            -0.020383762873109595,
+            0.025741408097358233,
+            -0.011755094311186617,
+            0.002737129201159793,
+            -0.008266210469668866,
+            0.027287436144516414,
+            -0.006283892307996516,
+            -0.0029440642954701632,
+            0.009428876742171362,
+            -0.03788574758692254,
+            0.008834605607057885,
+            0.006463342392514337,
+            0.026711196829464153,
+            -3.9827332346130654e-05,
+            0.005565561841469247,
+            -0.0006738085601962865,
+            -0.006104320995853506,
+            -0.010792830169647851,
+            2.4565453208679464e-05,
+            0.017983908748172493,
+            0.0035181716362611077,
+            0.016970206528324653,
+            -0.005449808667944701,
+            -0.023171388778833197,
+            0.007092509502738752,
+            -0.01555320330471131,
+            -0.023306285898839978,
+            -0.002028783456704567,
+            -0.03196436161255752,
+            0.0033412618908711163,
+            0.007527633574188688,
+            -0.01890570959143866,
+            -0.017859735533232564,
+            0.010329340599904344,
+            0.004382396539605985,
+            -0.0016746397858953155,
+            0.006643785064338525,
+            -0.03307812909454471,
+            -0.01927475592253288,
+            0.004686970010943116,
+            0.012733525168279355,
+            -0.012993441126970916,
+            0.015291832952199082,
+            -0.020491167895268936,
+            0.016451976170508816,
+            -0.012259387189535237,
+            0.018393274368854132,
+            -0.006997668664226754,
+            0.011843450716200625,
+            0.011457415642860215,
+            0.009560877634897626,
+            -0.01790001115302499,
+            0.0214879415944361,
+            0.0066829070506954115,
+            0.03900547021123392,
+            0.02512484499576633,
+            -0.012651294056575184,
+            -0.0018425680226742782,
+            -0.00011361458320146007,
+            0.014982647266052349,
+            0.0061874424115559,
+            -0.003689464024837091,
+            0.02705902205734107,
+            0.018239469373522537,
+            -0.021070704644859155,
+            0.020037508334843183,
+            -0.00973804771680803,
+            0.018876283979591423,
+            -0.013303080480396818,
+            0.010592424494628194,
+            0.004717870260467274,
+            0.01070131238869639,
+            0.009087656015197565,
+            0.006708673669681232,
+            0.025928065153829043,
+            0.03663367449576964,
+            -0.00511073108390551,
+            0.002161381031392467,
+            0.004864092591944181,
+            0.012814518637357098,
+            0.001556833001747549,
+            0.007915766323215207,
+            -0.007012785514562176,
+            0.005834786334750152,
+            -0.002311533032619821,
+            -0.00122162928193159,
+            0.019525999280692766,
+            0.000824163362963678,
+            0.012875575199417254,
+            0.013259376059185821,
+            -0.002390133687709065,
+            -0.005682644446302278,
+            0.0019262218417782657,
+            -0.018193478045923098,
+            -0.015131696847666434,
+            0.029463723011600933,
+            -0.011447039651934148,
+            -0.008083114389964213,
+            0.0020642857121204753,
+            0.012160202510990566,
+            -0.00866944494225225,
+            -0.005378300635997713,
+            0.0038280364584418335,
+            0.014427318067525318,
+            -0.01799358399064332,
+            0.003381165206832161,
+            0.005252552821854107,
+            0.019433554824232185,
+            -0.00028791985674147885,
+            -0.013487678738391956,
+            0.0005317922815671783,
+            0.014982152801567637,
+            0.011902348165369078,
+            0.0016056390299304837,
+            -0.006935777197137747,
+            -0.00446584737370504,
+            0.004586814866182051,
+            0.006269466956110759,
+            0.0027516492952993445,
+            0.008157662045793032,
+            -0.0025001885993292637,
+            -0.016648747436203294,
+            0.017735757469839637,
+            -0.0030669873770852725,
+            0.018058248627079793,
+            -0.0038238312452176745,
+            -0.0147172470474118,
+            0.0043328417868095355,
+            -0.001888507803054499,
+            -0.018556943283494755,
+            0.009407930200720534,
+            0.002560949424273885,
+            0.010411159836451958,
+            0.0052571230238294825,
+            -0.007499470181994448,
+            -6.145889479758644e-05,
+            -0.015163819825329975,
+            0.022836682800304944,
+            0.016390944816242627,
+            0.001713572907204073,
+            -0.0035687505973373878,
+            -0.031210299824387475,
+            0.0029076511444359886,
+            -0.007822231980296341,
+            0.017821806420465356,
+            0.008250605126658402,
+            0.007908773724009905,
+            0.008351355849444162,
+            -0.00743970642051451,
+            -0.002411981861500092,
+            0.012543476434788293,
+            0.0202407777715137,
+            -0.0024635818448627447,
+            0.02535771413579989,
+            -0.012533449594824386,
+            -0.0022489292647974992,
+            0.0012362917358624392,
+            0.01578939075186126,
+            -0.0008001783766371555,
+            0.006311837812042848
+          ],
+          "keyphrases": [
+            "Switch Transformers",
+            "Mixture of Experts",
+            "sparsely-activated model",
+            "training instability",
+            "multilingual settings"
+          ],
+          "title": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "1deb6775-c7ea-4a7a-8654-e306d12e688d",
+        "properties": {
+          "page_content": "9 Conclusion 27 ",
+          "keyphrases": [],
+          "embedding": [
+            -0.0026620388962328434,
+            0.009736468084156513,
+            -0.01410017628222704,
+            0.027085084468126297,
+            -0.02409333363175392,
+            -0.023739280179142952,
+            0.002801447408273816,
+            0.09375333786010742,
+            -0.007904241792857647,
+            0.016392672434449196,
+            -0.00255582295358181,
+            -0.03965397924184799,
+            -0.002425265731289983,
+            -0.03738803789019585,
+            -0.047655586153268814,
+            0.007165155373513699,
+            -0.002056828932836652,
+            -0.002341178013011813,
+            -0.018729424104094505,
+            -0.010240994393825531,
+            -0.027563055977225304,
+            -0.00033054203959181905,
+            0.017215846106410027,
+            0.05247071012854576,
+            -0.00470448425039649,
+            -0.0022095143795013428,
+            0.020588204264640808,
+            0.01972077414393425,
+            -0.032856155186891556,
+            0.04528342932462692,
+            -0.008333531208336353,
+            -0.01013477798551321,
+            0.036502905189991,
+            -0.024925358593463898,
+            -0.01793280430138111,
+            0.013728420250117779,
+            0.02297806553542614,
+            -0.014091324992477894,
+            0.013666460290551186,
+            -0.018764829263091087,
+            -0.01830456033349037,
+            0.01579963229596615,
+            -0.004806274548172951,
+            -0.004509754944592714,
+            -0.004859382752329111,
+            -0.028182649984955788,
+            -0.021101580932736397,
+            -0.06061394140124321,
+            -0.015905847772955894,
+            -0.003113457001745701,
+            -0.0059967790730297565,
+            -0.0376712791621685,
+            0.01110842451453209,
+            -0.003370145568624139,
+            0.009320455603301525,
+            0.03606033697724342,
+            0.028288865461945534,
+            0.08199876546859741,
+            0.0022404941264539957,
+            0.011338559910655022,
+            -0.021455634385347366,
+            -0.029917510226368904,
+            -0.009603697806596756,
+            -0.00012419528502505273,
+            -0.009072617627680302,
+            -0.022376174107193947,
+            0.011188087053596973,
+            -0.01899496465921402,
+            -0.006395089440047741,
+            -0.028430486097931862,
+            0.01365760900080204,
+            0.021136987954378128,
+            0.005231138784438372,
+            0.016764428466558456,
+            -0.007868836633861065,
+            0.0030692003201693296,
+            0.05133774131536484,
+            0.036078039556741714,
+            -0.009506333619356155,
+            0.04036208614706993,
+            -0.007550188340246677,
+            -0.010471128858625889,
+            0.008178633637726307,
+            0.028182649984955788,
+            0.04556667059659958,
+            0.010612750425934792,
+            -0.0727933794260025,
+            -0.09934738278388977,
+            -0.010108224116265774,
+            0.004580565728247166,
+            0.022712524980306625,
+            -0.03039548359811306,
+            -0.02081833966076374,
+            -0.026518598198890686,
+            -0.046416398137807846,
+            0.02150874398648739,
+            0.002179641043767333,
+            0.005049686413258314,
+            -0.03901668265461922,
+            0.008023735135793686,
+            0.018145235255360603,
+            0.017313210293650627,
+            0.009824981912970543,
+            0.010559642687439919,
+            0.02203982323408127,
+            -0.011250046081840992,
+            0.024358872324228287,
+            0.0323781818151474,
+            0.009718765504658222,
+            0.05640070512890816,
+            -0.020729824900627136,
+            -0.0049478961154818535,
+            -0.03560006618499756,
+            -0.029563458636403084,
+            0.011418221518397331,
+            -0.004403539001941681,
+            -0.021225500851869583,
+            -0.04762018099427223,
+            0.014162135310471058,
+            0.017667263746261597,
+            -0.009585995227098465,
+            0.004894787911325693,
+            0.02825346030294895,
+            -0.03774208948016167,
+            -0.02223455347120762,
+            0.019826989620923996,
+            0.03239588439464569,
+            0.008992956019937992,
+            -0.007616573479026556,
+            0.010957952588796616,
+            0.006669480819255114,
+            0.015967806801199913,
+            -0.024695223197340965,
+            -0.005629448685795069,
+            0.018410775810480118,
+            -0.024128738790750504,
+            0.010046265088021755,
+            0.0012579959584400058,
+            -0.001970528392121196,
+            0.030802644789218903,
+            -0.044398292899131775,
+            -0.008298126049339771,
+            -0.005554212257266045,
+            0.02388090081512928,
+            -0.006204785779118538,
+            0.012869840487837791,
+            -0.01720699481666088,
+            0.029811294749379158,
+            0.0013188488082960248,
+            0.01239186804741621,
+            0.0006068696384318173,
+            0.011639504693448544,
+            -0.02643008530139923,
+            0.0038591818884015083,
+            -0.002400924451649189,
+            -0.03961857408285141,
+            0.031369131058454514,
+            0.012728218920528889,
+            0.0037750941701233387,
+            0.03660912066698074,
+            0.0031532880384474993,
+            0.054524220526218414,
+            -0.020499691367149353,
+            -0.0051381997764110565,
+            0.009727616794407368,
+            0.00795734953135252,
+            0.027580758556723595,
+            -0.015976659953594208,
+            -0.04273424297571182,
+            0.022871848195791245,
+            0.03616655245423317,
+            0.0002338688645977527,
+            0.025244006887078285,
+            -0.03558236360549927,
+            0.007231540512293577,
+            -0.007213837932795286,
+            -0.008218464441597462,
+            0.055834218859672546,
+            -0.0100551163777709,
+            0.024164143949747086,
+            -0.009453224949538708,
+            0.019579151645302773,
+            -0.01991550251841545,
+            0.004472136963158846,
+            -0.0227656327188015,
+            -0.0074705262668430805,
+            0.06341096013784409,
+            -0.03887506201863289,
+            0.03274993970990181,
+            0.01571997068822384,
+            -0.002052403287962079,
+            -0.022482389584183693,
+            0.03370588272809982,
+            0.010046265088021755,
+            0.004113657865673304,
+            -0.06093258783221245,
+            -0.0016231135232374072,
+            0.035210609436035156,
+            -0.014498486183583736,
+            0.011781126260757446,
+            -0.014560445211827755,
+            -0.022287661209702492,
+            0.018481586128473282,
+            0.010754371993243694,
+            -0.010710114613175392,
+            -0.0028280015103518963,
+            0.005558638367801905,
+            0.010471128858625889,
+            0.014294905588030815,
+            -0.0005191860836930573,
+            0.02747454307973385,
+            0.0150118637830019,
+            -0.008116673678159714,
+            0.01593240164220333,
+            -0.013232745230197906,
+            0.010692412033677101,
+            -0.006930594798177481,
+            0.02209293097257614,
+            -0.013002610765397549,
+            -0.030625617131590843,
+            0.014409972354769707,
+            0.013551393523812294,
+            0.04145964980125427,
+            -0.0002031657932093367,
+            -0.008138801902532578,
+            0.04850531369447708,
+            -0.03760046884417534,
+            0.04698288440704346,
+            0.02729751542210579,
+            -0.04698288440704346,
+            0.009798427112400532,
+            0.018906449899077415,
+            -0.01830456033349037,
+            0.003505128435790539,
+            -0.0057710702531039715,
+            0.008882313966751099,
+            0.0009399011032655835,
+            0.014206391759216785,
+            0.003244014224037528,
+            0.02928021550178528,
+            0.04528342932462692,
+            -0.02372157759964466,
+            -0.028660621494054794,
+            0.0412118136882782,
+            -0.025615762919187546,
+            0.007258094381541014,
+            -0.0412118136882782,
+            0.06741176545619965,
+            0.017260102555155754,
+            0.014082473702728748,
+            0.015224295668303967,
+            -0.0005056324880570173,
+            0.0017757989699020982,
+            0.03958316892385483,
+            0.019490638747811317,
+            0.04740775004029274,
+            0.009957751259207726,
+            -0.0043836236000061035,
+            -0.053249627351760864,
+            0.032094940543174744,
+            -0.0034498076420277357,
+            0.0035715135745704174,
+            -0.030324673280119896,
+            0.01626875251531601,
+            0.02497846633195877,
+            0.002735062502324581,
+            0.02207522839307785,
+            -0.010356061160564423,
+            0.021809687837958336,
+            0.02007482759654522,
+            0.05640070512890816,
+            -0.014170986600220203,
+            -0.00489036226645112,
+            0.01175457239151001,
+            -0.018393073230981827,
+            -0.031953319907188416,
+            0.032271966338157654,
+            -0.018516991287469864,
+            -0.013347812928259373,
+            0.019065774977207184,
+            -0.0007838963065296412,
+            0.03476804122328758,
+            -0.005558638367801905,
+            -0.020712122321128845,
+            -0.017410574480891228,
+            -0.022110633552074432,
+            -0.0010062861256301403,
+            0.01040916983038187,
+            -0.030430888757109642,
+            0.00484168017283082,
+            0.013480582274496555,
+            -0.01138281635940075,
+            -0.009152280166745186,
+            0.01111727673560381,
+            -0.004633673466742039,
+            0.002014785073697567,
+            0.018853342160582542,
+            0.007492654956877232,
+            -0.008523834869265556,
+            -0.0014095749938860536,
+            0.003102392889559269,
+            0.0005645491764880717,
+            -0.03441398963332176,
+            -0.013595649972558022,
+            -0.008453024551272392,
+            0.01607402414083481,
+            0.016498887911438942,
+            0.03584790602326393,
+            -0.0714125707745552,
+            0.011789977550506592,
+            0.035901013761758804,
+            0.016339562833309174,
+            0.000724703015293926,
+            -0.04418586194515228,
+            -0.029775889590382576,
+            0.020375771448016167,
+            0.01302031334489584,
+            0.010046265088021755,
+            -0.013454028405249119,
+            -0.004485413897782564,
+            0.012763625010848045,
+            -0.011728018522262573,
+            -0.018605506047606468,
+            0.030625617131590843,
+            0.020499691367149353,
+            -0.03260831534862518,
+            -0.021243203431367874,
+            0.008430896326899529,
+            0.01988009735941887,
+            -0.012444976717233658,
+            0.025633465498685837,
+            -0.022482389584183693,
+            0.019490638747811317,
+            -0.001468215137720108,
+            -0.014799431897699833,
+            -0.02713819220662117,
+            -0.023438334465026855,
+            0.02025185339152813,
+            -0.03657371550798416,
+            -0.006908466573804617,
+            0.019472936168313026,
+            0.0032019703648984432,
+            -0.008599071763455868,
+            0.0008060246473178267,
+            -0.01640152372419834,
+            -0.038237765431404114,
+            0.029563458636403084,
+            -0.0561174601316452,
+            -0.003299335017800331,
+            0.01527740340679884,
+            0.0009227516129612923,
+            -0.009152280166745186,
+            0.008160930126905441,
+            0.01793280430138111,
+            -0.0023124110884964466,
+            -0.008006032556295395,
+            -0.028111839666962624,
+            -0.027580758556723595,
+            -0.009470928460359573,
+            -0.008935422636568546,
+            0.013073421083390713,
+            0.006381812505424023,
+            0.014896796084940434,
+            -0.006311001721769571,
+            -0.04907179996371269,
+            -0.025066979229450226,
+            0.016339562833309174,
+            0.01293179951608181,
+            0.020570501685142517,
+            0.0008663243497721851,
+            -0.02223455347120762,
+            -0.002160832053050399,
+            0.021225500851869583,
+            -0.017083076760172844,
+            -0.01575537584722042,
+            -0.006549987476319075,
+            -0.013648757711052895,
+            0.011612950824201107,
+            -0.026890354230999947,
+            0.011294303461909294,
+            -0.02892616204917431,
+            0.012276801280677319,
+            0.006704885978251696,
+            0.004992152564227581,
+            0.0151092279702425,
+            -0.0027306366246193647,
+            0.012498084455728531,
+            0.017180440947413445,
+            -0.0073997159488499165,
+            -0.04393802583217621,
+            -0.04854071885347366,
+            -0.012329909019172192,
+            -0.004717761185020208,
+            -0.02095996029675007,
+            0.00849728100001812,
+            0.011993558146059513,
+            0.0390520878136158,
+            0.0034984899684786797,
+            0.0045849913731217384,
+            -0.01707422360777855,
+            -0.016773279756307602,
+            0.009293901734054089,
+            -0.006572116166353226,
+            -0.009621400386095047,
+            0.021030770614743233,
+            0.031227508559823036,
+            0.0006782335112802684,
+            -0.023650765419006348,
+            -0.008315828628838062,
+            -0.008979679085314274,
+            0.0402204655110836,
+            -0.03604263439774513,
+            0.0002623590989969671,
+            -0.046239372342824936,
+            0.029970619827508926,
+            0.00285012973472476,
+            0.020481988787651062,
+            0.0024186272639781237,
+            -0.027279812842607498,
+            -0.0227656327188015,
+            -0.021632662042975426,
+            -0.0019583578687161207,
+            0.027616163715720177,
+            0.001016243826597929,
+            -0.021650364622473717,
+            0.04138883948326111,
+            -0.01610942929983139,
+            0.019101180136203766,
+            0.00011299282050458714,
+            0.011312006041407585,
+            -0.04167208448052406,
+            0.007231540512293577,
+            -0.02044658362865448,
+            0.01989779993891716,
+            0.04662882909178734,
+            -0.0034741489216685295,
+            -0.03331642225384712,
+            0.011533289216458797,
+            -0.011409370228648186,
+            0.021491041406989098,
+            0.01742827706038952,
+            -0.019118882715702057,
+            0.00047078036004677415,
+            0.04312370344996452,
+            0.019242800772190094,
+            -0.00578434718772769,
+            -0.030360078439116478,
+            -0.007620999123901129,
+            0.0010770967928692698,
+            0.010878290049731731,
+            -0.03202413022518158,
+            0.038981277495622635,
+            -0.037706684321165085,
+            -0.006470325402915478,
+            0.04407964646816254,
+            0.005939245689660311,
+            -0.010736668482422829,
+            0.026129139587283134,
+            -0.019207395613193512,
+            -0.04563748091459274,
+            0.0004276301187928766,
+            0.054347194731235504,
+            0.022907253354787827,
+            -0.025598060339689255,
+            -0.018481586128473282,
+            0.016746725887060165,
+            -0.007138601504266262,
+            0.013427474536001682,
+            -0.004531883168965578,
+            0.00865217950195074,
+            -0.014250649139285088,
+            -0.0037308374885469675,
+            -0.06309231370687485,
+            0.029563458636403084,
+            -0.02439427748322487,
+            -0.0195260439068079,
+            0.01194045040756464,
+            -0.009462077170610428,
+            -0.012533489614725113,
+            0.02533251978456974,
+            -0.02425265684723854,
+            0.03597182407975197,
+            0.015135781839489937,
+            0.0024186272639781237,
+            -0.023579955101013184,
+            0.05077125504612923,
+            0.04754937067627907,
+            -0.015879293903708458,
+            -0.03604263439774513,
+            0.0064039407297968864,
+            -0.0072979251854121685,
+            -0.031900208443403244,
+            0.00047575923963449895,
+            0.001995975850149989,
+            0.008559240959584713,
+            -0.07343067228794098,
+            -0.004664653446525335,
+            -0.07930795848369598,
+            0.007979477755725384,
+            0.031015075743198395,
+            0.003403338138014078,
+            0.0006732546607963741,
+            -0.007824580185115337,
+            0.015250849537551403,
+            0.02331441454589367,
+            -0.02333211898803711,
+            -0.014330310747027397,
+            -0.016906049102544785,
+            -0.010444574989378452,
+            0.030165348201990128,
+            0.0368923619389534,
+            -0.009276199154555798,
+            0.007027959916740656,
+            -0.00438583642244339,
+            -0.014170986600220203,
+            -0.01679983362555504,
+            0.01257774606347084,
+            0.01864091120660305,
+            -0.014224094338715076,
+            -0.02080063708126545,
+            -0.00154677068348974,
+            -0.0028523425571620464,
+            -0.00985153578221798,
+            0.03876884654164314,
+            -0.023650765419006348,
+            0.02133171632885933,
+            -0.006076441146433353,
+            -0.01828685775399208,
+            0.05501989647746086,
+            -0.022747930139303207,
+            0.02604062668979168,
+            0.00314443651586771,
+            -0.0062490422278642654,
+            0.0011235662968829274,
+            0.027085084468126297,
+            0.004456647206097841,
+            -0.013772676698863506,
+            0.03639668971300125,
+            0.0011141618015244603,
+            -0.002290282864123583,
+            0.014047068543732166,
+            0.010179034434258938,
+            -0.003905651392415166,
+            -0.020906852558255196,
+            -0.011453626677393913,
+            0.0139320008456707,
+            -0.005571915302425623,
+            -0.007160729728639126,
+            -0.0034055509604513645,
+            -0.005881711840629578,
+            0.013338961638510227,
+            -0.006983703002333641,
+            0.03436088189482689,
+            -0.0323781818151474,
+            -0.006727014202624559,
+            0.007067790720611811,
+            0.004978875629603863,
+            0.02368617057800293,
+            0.020694419741630554,
+            0.017437128350138664,
+            0.02005712501704693,
+            0.010400318540632725,
+            -0.010940250009298325,
+            -0.0029342174530029297,
+            -0.0001850482076406479,
+            -0.016357267275452614,
+            0.02752765081822872,
+            0.036644525825977325,
+            -0.020322663709521294,
+            -0.04291126877069473,
+            0.0014018301153555512,
+            0.01148903276771307,
+            -0.028979269787669182,
+            -0.03381209820508957,
+            0.01830456033349037,
+            0.034290071576833725,
+            0.019791584461927414,
+            -0.03400682657957077,
+            -0.02409333363175392,
+            0.008289274759590626,
+            0.01347173098474741,
+            0.008090119808912277,
+            0.02025185339152813,
+            -0.003168777795508504,
+            0.009435522370040417,
+            -0.0005703578935936093,
+            -0.004082677885890007,
+            -0.013418623246252537,
+            -0.03529912233352661,
+            0.014480783604085445,
+            -0.0067668454721570015,
+            -0.0022194720804691315,
+            0.009789575822651386,
+            -0.017755776643753052,
+            -0.027014274150133133,
+            0.005819752346724272,
+            0.026890354230999947,
+            0.010382615961134434,
+            -0.012781327590346336,
+            -0.02281874045729637,
+            -0.006811101920902729,
+            0.0010627133306115866,
+            0.011896193958818913,
+            9.674231841927394e-05,
+            0.00715187843888998,
+            0.02080063708126545,
+            0.0003053710388485342,
+            -0.016374969854950905,
+            -0.037175606936216354,
+            0.03324561193585396,
+            -0.01565801165997982,
+            -0.010639304295182228,
+            0.031245211139321327,
+            -0.010789777152240276,
+            -0.004762018099427223,
+            -0.03351115435361862,
+            -0.008253869600594044,
+            -0.02386319823563099,
+            0.01110842451453209,
+            0.013383218087255955,
+            0.025580357760190964,
+            -0.02625305764377117,
+            -0.03437858447432518,
+            0.03184710070490837,
+            0.015684565529227257,
+            0.0005410378216765821,
+            -0.012648557312786579,
+            -0.004916916601359844,
+            -0.03873344138264656,
+            -0.03653831034898758,
+            0.019083477556705475,
+            0.0051204971969127655,
+            0.002542545786127448,
+            -0.022730227559804916,
+            -0.0076608299277722836,
+            0.02223455347120762,
+            0.04252181202173233,
+            0.004611545242369175,
+            0.022871848195791245,
+            -0.012329909019172192,
+            0.02352684736251831,
+            -0.031085887923836708,
+            -0.019826989620923996,
+            0.024482792243361473,
+            -0.019437531009316444,
+            -0.00976302195340395,
+            0.01013477798551321,
+            0.025757383555173874,
+            0.020552799105644226,
+            0.017552196979522705,
+            0.032856155186891556,
+            0.029634268954396248,
+            0.01862320862710476,
+            0.025721978396177292,
+            0.009532887488603592,
+            -0.019083477556705475,
+            -0.016020916402339935,
+            -0.00664292648434639,
+            -0.027633866295218468,
+            -0.004558437503874302,
+            -0.024836845695972443,
+            -0.016065172851085663,
+            0.021066175773739815,
+            0.0036224087234586477,
+            0.0028147243428975344,
+            0.013436325825750828,
+            -0.03653831034898758,
+            -0.00505853770300746,
+            -0.0048107001930475235,
+            0.008001606911420822,
+            -0.0013974044704809785,
+            -0.0008524941513314843,
+            0.017100779339671135,
+            0.01384348701685667,
+            -0.018056722357869148,
+            -0.0036334728356450796,
+            -0.01778233051300049,
+            0.030501699075102806,
+            -0.018552398309111595,
+            -0.006532284896820784,
+            -0.03958316892385483,
+            0.019986312836408615,
+            0.023438334465026855,
+            0.027899406850337982,
+            0.004341579508036375,
+            0.001363105489872396,
+            0.006833230145275593,
+            0.005660428665578365,
+            -0.0009476460400037467,
+            -0.0063065760768949986,
+            -0.003874671645462513,
+            0.014631256461143494,
+            -0.012117477133870125,
+            0.009568292647600174,
+            -0.0123122064396739,
+            -0.02223455347120762,
+            -0.024907656013965607,
+            0.0015157910529524088,
+            0.02294265851378441,
+            0.03922911360859871,
+            0.031758587807416916,
+            -0.016507739201188087,
+            -0.03506898880004883,
+            0.02278333529829979,
+            0.02223455347120762,
+            -0.009771873243153095,
+            -0.001461576670408249,
+            -0.005439145024865866,
+            -0.014525040052831173,
+            -0.017295507714152336,
+            0.0014593638479709625,
+            -0.01665821112692356,
+            -0.013896595686674118,
+            0.0386626310646534,
+            0.013542542234063148,
+            0.012197138741612434,
+            -0.0007601083489134908,
+            -0.017569899559020996,
+            -0.01665821112692356,
+            0.0002082829741993919,
+            0.019207395613193512,
+            -0.01320619136095047,
+            -0.015348214656114578,
+            0.011196938343346119,
+            -0.034095339477062225,
+            0.011250046081840992,
+            0.007793600205332041,
+            0.0019760604482144117,
+            0.022128336131572723,
+            0.008253869600594044,
+            -0.014445378445088863,
+            -0.02025185339152813,
+            -0.005226713139563799,
+            0.0026310591492801905,
+            -0.0037551787681877613,
+            0.016693616285920143,
+            0.008218464441597462,
+            0.027917109429836273,
+            0.027226705104112625,
+            -0.009258495643734932,
+            0.0123122064396739,
+            0.022323066368699074,
+            -0.010754371993243694,
+            -0.004567288793623447,
+            0.00045031163608655334,
+            -0.013958554714918137,
+            0.03784830868244171,
+            0.043725594878196716,
+            -0.016330711543560028,
+            0.013179637491703033,
+            -0.01194045040756464,
+            -0.03167007490992546,
+            0.04737234115600586,
+            -0.018888747319579124,
+            -0.01880023442208767,
+            0.00986923836171627,
+            0.007687384262681007,
+            -0.02115469053387642,
+            -0.017198143526911736,
+            0.007758195046335459,
+            -0.013091123662889004,
+            0.010966803878545761,
+            0.003606918966397643,
+            0.040432896465063095,
+            0.01972077414393425,
+            -0.006864210125058889,
+            -0.024111036211252213,
+            -0.018570100888609886,
+            0.015144633129239082,
+            0.00215087435208261,
+            0.03976019471883774,
+            -0.0032395885791629553,
+            0.0244296845048666,
+            -0.029616566374897957,
+            0.008050289005041122,
+            -0.013108826242387295,
+            0.004257491789758205,
+            -0.0070722163654863834,
+            -0.016923751682043076,
+            0.012427274137735367,
+            -0.018322262912988663,
+            0.002898812061175704,
+            -0.03289156034588814,
+            -0.028642918914556503,
+            -0.018499288707971573,
+            0.03455561026930809,
+            0.05169179290533066,
+            -0.01748138666152954,
+            -0.009152280166745186,
+            0.02207522839307785,
+            -0.013073421083390713,
+            -0.0024650967679917812,
+            -0.008758395910263062,
+            -0.0002622207975946367,
+            0.03432547673583031,
+            -0.03368818014860153,
+            -0.028483595699071884,
+            -0.0011606313055381179,
+            -0.02494306117296219,
+            -0.03653831034898758,
+            0.000632870418485254,
+            -0.0047929976135492325,
+            -0.03618425503373146,
+            0.015622605569660664,
+            0.00015973893459886312,
+            0.0184461809694767,
+            0.004832828883081675,
+            0.011338559910655022,
+            -0.010798628441989422,
+            0.006417217664420605,
+            0.03239588439464569,
+            -0.04340694472193718,
+            0.017906250432133675,
+            0.007727215066552162,
+            -0.024695223197340965,
+            0.034449394792318344,
+            0.014710918068885803,
+            -0.015215444378554821,
+            -0.019809287041425705,
+            0.009050489403307438,
+            -0.002253771061077714,
+            -0.01918969303369522,
+            0.01691490039229393,
+            -0.01897726207971573,
+            0.010887141339480877,
+            0.03370588272809982,
+            0.013586798682808876,
+            0.005833029747009277,
+            0.04089316725730896,
+            0.05463043600320816,
+            -0.031209805980324745,
+            0.002666464541107416,
+            -0.01410017628222704,
+            0.03632587566971779,
+            -0.00863447692245245,
+            0.009807278402149677,
+            -0.00397867476567626,
+            -0.010426872409880161,
+            -0.014348013326525688,
+            0.02062360942363739,
+            -0.009621400386095047,
+            0.018322262912988663,
+            0.0006389557383954525,
+            -0.0035892161540687084,
+            -0.00022640054521616548,
+            0.0034918515011668205,
+            -0.021668067201972008,
+            -0.012622003443539143,
+            0.009444373659789562,
+            0.021809687837958336,
+            0.018393073230981827,
+            0.02674873359501362,
+            -0.021119285374879837,
+            0.007253668736666441,
+            -0.0009448799537494779,
+            -0.005656003020703793,
+            0.013781527988612652,
+            0.027439137920737267,
+            0.01675557717680931,
+            0.022181443870067596,
+            0.015144633129239082,
+            0.05463043600320816,
+            -0.016516590490937233,
+            -0.03926451876759529,
+            0.034838851541280746,
+            -0.0025159919168800116,
+            -0.041318029165267944,
+            -0.030165348201990128,
+            -0.0317939929664135,
+            -0.03958316892385483,
+            -0.02264171466231346,
+            -0.0018941856687888503,
+            0.018711721524596214,
+            0.004978875629603863,
+            -0.010993357747793198,
+            -0.02117239311337471,
+            0.0037264118436723948,
+            -0.011612950824201107,
+            0.03563547506928444,
+            -0.01830456033349037,
+            -0.010957952588796616,
+            -0.013985108584165573,
+            -0.03513979911804199,
+            -0.00742184417322278,
+            -0.014586999081075191,
+            0.011462477967143059,
+            -0.014728620648384094,
+            -0.030147645622491837,
+            -0.0024119887966662645,
+            -0.022393876686692238,
+            -0.008860185742378235,
+            -0.008599071763455868,
+            0.00868315901607275,
+            -0.030661022290587425,
+            0.022146038711071014,
+            0.030625617131590843,
+            0.00010808309161802754,
+            0.0036047061439603567,
+            0.016330711543560028,
+            0.0009996476583182812,
+            -0.011550991795957088,
+            0.008815929293632507,
+            0.026465490460395813,
+            -0.03894587233662605,
+            -0.01374612282961607,
+            -0.027580758556723595,
+            0.000777257839217782,
+            0.0016806471394374967,
+            -0.02281874045729637,
+            0.004899213556200266,
+            0.0005875073256902397,
+            -0.01707422360777855,
+            -0.04177829995751381,
+            -0.03887506201863289,
+            0.01320619136095047,
+            0.035741690546274185,
+            -0.002186279743909836,
+            0.005284246988594532,
+            -0.008970827795565128,
+            0.013515988364815712,
+            0.023031173273921013,
+            -0.02081833966076374,
+            -0.014702066779136658,
+            0.021791985258460045,
+            0.015321659855544567,
+            -0.018339965492486954,
+            -0.0155517952516675,
+            0.02156185172498226,
+            -0.016242198646068573,
+            0.03313939645886421,
+            -0.006815527565777302,
+            0.021278608590364456,
+            0.052895575761795044,
+            -0.01710963062942028,
+            0.04436288774013519,
+            -0.006651777774095535,
+            -0.01693260297179222,
+            0.015711119398474693,
+            0.018924154341220856,
+            -0.01473747193813324,
+            0.020747527480125427,
+            0.037706684321165085,
+            0.020765231922268867,
+            0.020517393946647644,
+            -0.028873054310679436,
+            0.013108826242387295,
+            0.019260503351688385,
+            0.0506296344101429,
+            0.009134577587246895,
+            -0.013790379278361797,
+            -0.005164753645658493,
+            0.010639304295182228,
+            0.00314443651586771,
+            -0.02945724129676819,
+            0.0012380804400891066,
+            0.00031864806078374386,
+            -0.04068073257803917,
+            0.006793399341404438,
+            0.010028562508523464,
+            0.03253750503063202,
+            0.0134894335642457,
+            0.0027129340451210737,
+            -0.03653831034898758,
+            0.01465781033039093,
+            -0.01356024481356144,
+            0.02660711109638214,
+            0.03057250939309597,
+            0.022323066368699074,
+            -0.02692575938999653,
+            -0.0022947085089981556,
+            -0.007687384262681007,
+            0.017861993983387947,
+            0.055975839495658875,
+            0.020765231922268867,
+            -0.020198745653033257,
+            -0.021278608590364456,
+            -0.01789739914238453,
+            -0.013648757711052895,
+            -0.002648761961609125,
+            0.022553199902176857,
+            0.02368617057800293,
+            -0.01240072026848793,
+            0.022146038711071014,
+            -0.027085084468126297,
+            -0.007713938131928444,
+            -0.0033458045218139887,
+            -0.033581964671611786,
+            0.0027439137920737267,
+            0.024659818038344383,
+            0.00932930689305067,
+            0.00922309048473835,
+            -0.007820154540240765,
+            0.0041933199390769005,
+            -0.025775086134672165,
+            0.0007910880376584828,
+            0.0015677926130592823,
+            0.02474833093583584,
+            -0.008811503648757935,
+            0.008793801069259644,
+            -0.012338760308921337,
+            -0.020216448232531548,
+            -0.01229450386017561,
+            -0.023845495656132698,
+            0.013321259059011936,
+            0.021437931805849075,
+            0.006408366374671459,
+            0.020198745653033257,
+            0.005226713139563799,
+            0.017419425770640373,
+            0.025013871490955353,
+            -0.007364310324192047,
+            -0.00212432025000453,
+            0.021738877519965172,
+            0.01140051893889904,
+            -0.00790866743773222,
+            0.019118882715702057,
+            -0.0010660325642675161,
+            0.011471329256892204,
+            0.01076322328299284,
+            0.014277203008532524,
+            -0.012365314178168774,
+            -0.03549385070800781,
+            -0.0027107212226837873,
+            0.006045461632311344,
+            0.016569698229432106,
+            0.01640152372419834,
+            0.017773479223251343,
+            0.015861591324210167,
+            0.014887944795191288,
+            -0.005062963347882032,
+            0.0036157702561467886,
+            -0.007342182099819183,
+            0.004450008273124695,
+            -0.04450451210141182,
+            0.012223693542182446,
+            -0.006855358835309744,
+            0.005594043526798487,
+            -0.013126528821885586,
+            0.002008146606385708,
+            -0.019083477556705475,
+            -0.012241396121680737,
+            -0.014091324992477894,
+            0.02713819220662117,
+            0.02444738708436489,
+            -0.0022626223508268595,
+            -0.0031400108709931374,
+            -0.011011060327291489,
+            0.022553199902176857,
+            0.0026067181024700403,
+            0.0017392872832715511,
+            -0.009789575822651386,
+            0.016419226303696632,
+            0.033174801617860794,
+            -0.0045849913731217384,
+            -0.011418221518397331,
+            0.013480582274496555,
+            -0.01057734526693821,
+            -0.006271170452237129,
+            0.022836443036794662,
+            -0.017808884382247925,
+            -0.015410173684358597,
+            0.01502956636250019,
+            -0.004149063024669886,
+            0.001848822575993836,
+            0.009984305128455162,
+            0.015604902990162373,
+            0.009833833202719688,
+            -0.007461674977093935,
+            0.006213637068867683,
+            0.01204666681587696,
+            -0.011143830604851246,
+            -0.01783544011414051,
+            0.007682958617806435,
+            -0.010090521536767483,
+            0.013498285785317421,
+            0.020694419741630554,
+            -0.031953319907188416,
+            -0.014418824575841427,
+            -0.012790178880095482,
+            -0.00841761939227581,
+            0.013498285785317421,
+            -0.008484004065394402,
+            0.022376174107193947,
+            0.021278608590364456,
+            -0.008253869600594044,
+            -0.016295306384563446,
+            -0.02547414042055607,
+            0.017224697396159172,
+            0.012701665051281452,
+            0.0007081067888066173,
+            -0.0437610000371933,
+            0.0021221074275672436,
+            0.011427072808146477,
+            0.012152882292866707,
+            0.001480385661125183,
+            -0.0010759903816506267,
+            -0.009692211635410786,
+            0.06925284117460251,
+            -0.01455159392207861,
+            -0.0004137998912483454,
+            -0.007187283597886562,
+            -0.02658940851688385,
+            0.005678131245076656,
+            -0.028465893119573593,
+            0.0076431273482739925,
+            0.002303559798747301,
+            0.014852539636194706,
+            -0.004815125837922096,
+            0.013250447809696198,
+            0.005589617881923914,
+            0.024872250854969025,
+            0.023296711966395378,
+            0.01991550251841545,
+            -0.02170347236096859,
+            -0.0010394785786047578,
+            0.0041601271368563175,
+            -0.025651168078184128,
+            -0.003427679417654872,
+            -0.003272780915722251,
+            -0.014507337473332882,
+            -0.00269080582074821,
+            -0.025615762919187546,
+            0.014622405171394348,
+            -0.00782900582998991,
+            0.005377185996621847,
+            -0.02589900605380535,
+            -0.0006267851567827165,
+            0.018534693866968155,
+            0.03349345177412033,
+            0.008103396743535995,
+            0.005151476711034775,
+            -0.01013477798551321,
+            0.01778233051300049,
+            -0.0017912888433784246,
+            0.04124721884727478,
+            0.01644578017294407,
+            -0.005549786612391472,
+            -0.006430494599044323,
+            0.040999382734298706,
+            0.02007482759654522,
+            -0.027226705104112625,
+            -0.0025690998882055283,
+            0.005748942028731108,
+            0.025279412046074867,
+            -0.028642918914556503,
+            -0.008550389669835567,
+            -0.019968610256910324,
+            -0.0044367313385009766,
+            0.008780524134635925,
+            0.016047470271587372,
+            -0.00874069333076477,
+            -0.012551192194223404,
+            0.0018886534962803125,
+            0.011710315942764282,
+            0.017490237951278687,
+            -0.009267346933484077,
+            0.006204785779118538,
+            0.016313008964061737,
+            -0.02804102748632431,
+            0.0006511263200081885,
+            -0.009585995227098465,
+            0.024323467165231705,
+            0.027191299945116043,
+            0.0027660420164465904,
+            -0.0030736259650439024,
+            0.00939126592129469,
+            0.007479377556592226,
+            -0.007143027149140835,
+            0.037919119000434875,
+            -0.010418021120131016,
+            -0.00046386526082642376,
+            -0.004290684591978788,
+            0.003370145568624139,
+            0.014808283187448978,
+            -0.014967607334256172,
+            0.0015711118467152119,
+            -0.008700861595571041,
+            -0.0021530871745198965,
+            0.006386238150298595,
+            -0.006341981235891581,
+            0.015188890509307384,
+            0.0034807873889803886,
+            -0.029705079272389412,
+            0.010019711218774319,
+            0.0017337552271783352,
+            0.005434719379991293,
+            -0.025704275816679,
+            -0.03037778101861477,
+            0.015126930549740791,
+            -0.01738402061164379,
+            -0.01883563958108425,
+            -0.017746925354003906,
+            -0.022924955934286118,
+            0.019844692200422287,
+            -0.0025027149822562933,
+            -0.02731521800160408,
+            0.03073183447122574,
+            0.031192103400826454,
+            -0.005098368972539902,
+            0.03197102248668671,
+            -0.010586196556687355,
+            0.012613152153789997,
+            0.0349273681640625,
+            -0.012630854733288288,
+            -0.0015025140019133687,
+            0.013852338306605816,
+            -0.008886739611625671,
+            0.005704685114324093,
+            -0.008284849114716053,
+            -0.022340768948197365,
+            -0.020552799105644226,
+            -0.018924154341220856,
+            -0.012223693542182446,
+            0.00976302195340395,
+            -0.0029076633509248495,
+            0.012807881459593773,
+            -0.010143629275262356,
+            -0.015348214656114578,
+            0.02333211898803711,
+            -0.0035670879296958447,
+            -0.01989779993891716,
+            -0.005930394399911165,
+            0.0222699586302042,
+            0.007483803667128086,
+            -0.01991550251841545,
+            0.032626017928123474,
+            0.027279812842607498,
+            -0.025739680975675583,
+            -0.014498486183583736,
+            -0.024358872324228287,
+            0.009966602548956871,
+            -0.0032418014016002417,
+            -0.025456437841057777,
+            0.006833230145275593,
+            -0.025084681808948517,
+            0.001977166859433055,
+            -0.003029369283467531,
+            -0.008961976505815983,
+            -0.0017160525312647223,
+            -0.016790982335805893,
+            0.006063164211809635,
+            -0.014179837889969349,
+            -0.017198143526911736,
+            -0.03113899566233158,
+            0.017693817615509033,
+            0.018587803468108177,
+            0.04705369472503662,
+            -0.005147051066160202,
+            -0.0034918515011668205,
+            0.0008707500528544188,
+            0.026182247325778008,
+            0.013622203841805458,
+            -0.012701665051281452,
+            0.0034874258562922478,
+            -0.007474951911717653,
+            0.014153284020721912,
+            -0.01638382114470005,
+            0.0076077221892774105,
+            0.022393876686692238,
+            -0.014427675865590572,
+            -0.0023677321150898933,
+            0.008537111803889275,
+            0.0026620388962328434,
+            -0.005629448685795069,
+            0.07944957911968231,
+            -0.00874069333076477,
+            0.008692010305821896,
+            -0.007174006663262844,
+            0.0024783737026154995,
+            -0.011019911617040634,
+            0.000400799501221627,
+            0.03558236360549927,
+            -0.007873262278735638,
+            0.01102876290678978,
+            0.024659818038344383,
+            0.008523834869265556,
+            0.004376985132694244,
+            0.007921944372355938,
+            0.02839508093893528,
+            -0.009647955186665058,
+            0.015613754279911518,
+            0.012064369395375252,
+            -0.011745721101760864,
+            -0.004244214855134487,
+            -0.012082071974873543,
+            0.0007075535831972957,
+            -0.0009885834297165275,
+            0.004545160569250584,
+            0.012595449574291706,
+            0.02554495260119438,
+            0.014109027571976185,
+            0.014817134477198124,
+            0.006434920243918896,
+            0.019632259383797646,
+            0.0021110433153808117,
+            -0.0111349793151021,
+            -0.009258495643734932,
+            0.00836893729865551,
+            0.005501104518771172,
+            0.0034697232767939568,
+            -0.006850933190435171,
+            0.009506333619356155,
+            -0.017861993983387947,
+            0.02062360942363739,
+            -0.02080063708126545,
+            -0.009524036198854446,
+            -0.003230737056583166,
+            -0.020729824900627136,
+            0.007342182099819183,
+            0.045141804963350296,
+            0.004700058605521917,
+            -0.005868434906005859,
+            0.04776180163025856,
+            -0.015162336640059948,
+            -0.03912289813160896,
+            0.034732636064291,
+            0.014932201243937016,
+            -0.029439538717269897,
+            -0.0017459257505834103,
+            0.00014438739162869751,
+            0.028837649151682854,
+            0.002750552259385586,
+            -0.00041960857925005257,
+            0.015153485350310802,
+            -0.02188050001859665,
+            0.016242198646068573,
+            -0.004478775430470705,
+            0.0007838963065296412,
+            -0.01915428787469864,
+            0.02602292411029339,
+            -0.006182657089084387,
+            -0.005855157971382141,
+            0.0013011462287977338,
+            -0.014728620648384094,
+            -0.004215448163449764,
+            -0.00939126592129469,
+            -0.00669603468850255,
+            -0.00978072453290224,
+            -0.019402125850319862,
+            0.015702268108725548,
+            0.0030758387874811888,
+            -0.02749224565923214,
+            -0.0004948449204675853,
+            0.003244014224037528,
+            -0.01620679348707199,
+            -0.009568292647600174,
+            -0.0009376882808282971,
+            0.014224094338715076,
+            -0.02658940851688385,
+            0.013126528821885586,
+            -0.002604505280032754,
+            -0.010887141339480877,
+            0.022199146449565887,
+            -0.007634276058524847,
+            0.016313008964061737,
+            0.009453224949538708,
+            0.013091123662889004,
+            -0.0008679840248078108,
+            0.0008591326768510044,
+            0.00792637001723051,
+            0.004801848903298378,
+            -0.03788371384143829,
+            0.0070899189449846745,
+            0.014507337473332882,
+            -0.01748138666152954,
+            -0.00872299075126648,
+            -0.006850933190435171,
+            -0.034467097371816635,
+            0.0058064754121005535,
+            -0.011913896538317204,
+            0.022022120654582977,
+            0.01438341848552227,
+            0.002887747948989272,
+            0.011373965069651604,
+            -0.014294905588030815,
+            0.01194045040756464,
+            0.0025270560290664434,
+            -0.00489036226645112,
+            0.05983502045273781,
+            -0.023579955101013184,
+            -0.02205752581357956,
+            -0.008315828628838062,
+            0.02929791808128357,
+            -0.0014814921887591481,
+            0.01646348275244236,
+            0.04195532575249672,
+            0.030448591336607933,
+            -0.004149063024669886,
+            0.0025093534495681524,
+            0.0067491428926587105,
+            -0.00010379572631791234,
+            -0.006961574777960777,
+            0.01328585296869278,
+            -0.027970217168331146,
+            0.01710963062942028,
+            -0.002752765081822872,
+            -0.02731521800160408,
+            -0.012321057729423046,
+            0.009081468917429447,
+            0.023916305974125862,
+            -0.005894988775253296,
+            0.014586999081075191,
+            -0.008559240959584713,
+            -0.018764829263091087,
+            -0.00912572629749775,
+            0.009506333619356155,
+            0.020694419741630554,
+            0.0006511263200081885,
+            0.02823575772345066,
+            -0.030625617131590843,
+            0.028483595699071884,
+            0.006815527565777302,
+            -0.003624621545895934,
+            0.015374768525362015,
+            -0.005231138784438372,
+            0.0009896898409351707,
+            0.009285050444304943,
+            0.015082674100995064,
+            -0.021951310336589813,
+            -0.046558018773794174,
+            0.013940852135419846,
+            0.02515549212694168,
+            0.024146441370248795,
+            0.0037573915906250477,
+            -0.012064369395375252,
+            -0.0193313155323267,
+            0.022163741290569305,
+            -0.016684764996170998,
+            -0.010152480565011501,
+            0.010541939176619053,
+            0.007337756454944611,
+            0.0044367313385009766,
+            0.0267841387540102,
+            0.019968610256910324,
+            0.003317037597298622,
+            -0.004443369805812836,
+            -0.0021044048480689526,
+            -0.0027239981573075056,
+            0.010825182311236858,
+            -0.030873455107212067,
+            0.0033944868482649326,
+            0.006125123705714941,
+            0.0008646647329442203,
+            -0.007116473279893398,
+            0.006833230145275593,
+            0.0144719323143363,
+            -0.02409333363175392,
+            -0.05679016187787056,
+            0.021136987954378128,
+            -0.01799476332962513,
+            0.015888145193457603,
+            -0.0005501657724380493,
+            0.0004071613948326558,
+            0.014985309913754463,
+            0.013914298266172409,
+            -0.003914502914994955,
+            -0.013569096103310585,
+            -0.0155517952516675,
+            0.014684364199638367,
+            -0.010710114613175392,
+            0.013338961638510227,
+            -0.011462477967143059,
+            -0.011657207272946835,
+            -0.006864210125058889,
+            0.01652544178068638,
+            0.024465089663863182,
+            0.012604300864040852,
+            0.01707422360777855,
+            0.0020557225216180086,
+            0.007820154540240765,
+            0.00505853770300746,
+            -0.0038414793089032173,
+            0.014507337473332882,
+            -0.034095339477062225,
+            0.013347812928259373,
+            -0.012453828006982803,
+            -0.0022637287620455027,
+            -0.004425667226314545,
+            -0.004049485549330711,
+            -0.02189820259809494,
+            -0.019862394779920578,
+            -0.0076608299277722836,
+            -0.018322262912988663,
+            0.00484168017283082,
+            0.02625305764377117,
+            -0.011630653403699398,
+            -0.021278608590364456,
+            -0.0042796204797923565,
+            -0.027102787047624588,
+            0.00949748232960701,
+            -0.02802332490682602,
+            0.041353434324264526,
+            0.008838057518005371,
+            0.00135757343377918,
+            0.006700460333377123,
+            -0.02189820259809494,
+            0.012170584872364998,
+            0.011073019355535507,
+            -0.019278205931186676,
+            -0.015135781839489937,
+            0.006656203418970108,
+            0.025084681808948517,
+            -0.0015257487539201975,
+            0.010913695208728313,
+            -0.0018875470850616693,
+            0.015534092672169209,
+            -0.03620195761322975,
+            -0.008191910572350025,
+            -0.00039858664968051016,
+            -0.02368617057800293,
+            -0.00914342887699604,
+            0.025952113792300224,
+            -0.018853342160582542,
+            0.015542943961918354,
+            -0.005470125004649162,
+            -0.006408366374671459,
+            -0.01365760900080204,
+            0.01701226457953453,
+            -2.6744170099846087e-05,
+            0.023066578432917595,
+            0.012453828006982803,
+            0.015224295668303967,
+            -0.015711119398474693,
+            -0.006961574777960777,
+            -0.0033922740258276463,
+            -0.0043327282182872295,
+            -0.015056120231747627,
+            -0.01130315475165844,
+            0.0134894335642457,
+            0.008461875841021538,
+            -0.008616774342954159,
+            0.015286254696547985,
+            0.0001645794982323423,
+            -0.0031046057119965553,
+            -0.023031173273921013,
+            0.0005709110992029309,
+            -0.004908064845949411,
+            -0.007567890919744968,
+            -0.013648757711052895,
+            -0.017808884382247925,
+            0.025598060339689255,
+            0.01862320862710476,
+            -0.04361937567591667,
+            0.01084288489073515,
+            -0.016065172851085663,
+            0.025226304307579994,
+            -0.010232143104076385,
+            0.010019711218774319,
+            -0.0024296913761645555,
+            -0.0216857697814703,
+            0.0028656194917857647,
+            -0.006514582317322493,
+            -0.019437531009316444,
+            -0.021827390417456627,
+            0.014790580607950687,
+            -0.007041236851364374,
+            -0.0037950098048895597,
+            -0.0030692003201693296,
+            -0.030537104234099388,
+            -0.009639102965593338,
+            0.01954374648630619,
+            0.004801848903298378,
+            0.007669681683182716,
+            0.016985710710287094,
+            -0.011506735347211361,
+            0.028129542246460915,
+            -0.011232343502342701,
+            -0.011250046081840992,
+            0.0025204175617545843,
+            -0.0056338743306696415,
+            0.016569698229432106,
+            -0.0116749107837677,
+            -0.022588606923818588,
+            -0.002199556678533554,
+            0.009754170663654804,
+            0.021738877519965172,
+            -0.005859583616256714,
+            0.01357794739305973,
+            0.024571305140852928,
+            0.013462879694998264,
+            -0.0015489835059270263,
+            0.007266945671290159,
+            0.06128663942217827,
+            0.019950907677412033,
+            0.013754974119365215,
+            0.0019052497809752822,
+            0.025704275816679,
+            -0.014277203008532524,
+            0.000535782368388027,
+            -0.013542542234063148,
+            -0.0025580357760190964,
+            -0.018083276227116585,
+            -0.011931599117815495,
+            -0.005682556889951229,
+            0.010364912450313568,
+            -0.006757994182407856,
+            0.011276599951088428,
+            -0.010471128858625889,
+            -0.005386037286370993,
+            -0.01610942929983139,
+            0.00011700358300004154,
+            0.008253869600594044,
+            0.015321659855544567,
+            -0.019649963825941086,
+            -0.026111437007784843,
+            0.002363306237384677,
+            -0.0060941437259316444,
+            0.011550991795957088,
+            0.0044632856734097,
+            0.0021652576979249716,
+            0.0145692965015769,
+            -0.022323066368699074,
+            0.032997775822877884,
+            -0.007156304083764553,
+            -0.018888747319579124,
+            0.017136184498667717,
+            -0.005403739865869284,
+            0.020552799105644226,
+            0.0182514525949955,
+            0.007846708409488201,
+            0.00985153578221798,
+            0.003244014224037528,
+            -0.0030669874977320433,
+            -0.02007482759654522,
+            0.012896394357085228,
+            -0.002684167120605707,
+            0.007948498241603374,
+            -0.008568092249333858,
+            -0.014082473702728748,
+            -0.004554011858999729,
+            -0.004970024339854717,
+            0.009134577587246895,
+            -0.0023987116292119026,
+            0.014374567195773125,
+            0.0036644525825977325,
+            -0.0019738476257771254,
+            -0.01076322328299284,
+            -0.0002287516836076975,
+            0.028111839666962624,
+            -0.0011849724687635899,
+            0.005496678873896599,
+            0.008895590901374817,
+            -0.011798828840255737,
+            -0.010435723699629307,
+            -0.004312812816351652,
+            -0.00028628535801544785,
+            0.003912290092557669,
+            0.015268552117049694,
+            0.0030138795264065266,
+            0.0018443968147039413,
+            -0.015339363366365433,
+            0.004952321760356426,
+            0.0015146846417337656,
+            0.009400117211043835,
+            -0.005005429964512587,
+            -0.0034498076420277357,
+            -0.018729424104094505,
+            -0.011258897371590137,
+            -0.01148903276771307,
+            -0.0014848114224150777,
+            -0.006244616582989693,
+            -0.022659417241811752,
+            0.011090721935033798,
+            0.018198344856500626,
+            -0.01546328142285347,
+            -0.0013863402418792248,
+            -0.0368923619389534,
+            -0.0007130856392905116,
+            0.008886739611625671,
+            0.013338961638510227,
+            0.00309354136697948,
+            0.03234277665615082,
+            0.003520618425682187,
+            -0.0068199532106518745,
+            0.014764025807380676,
+            -0.020481988787651062,
+            -0.013011462055146694,
+            -0.0201102327555418,
+            0.007682958617806435,
+            0.004135786090046167,
+            -0.00655883876606822,
+            -0.018216047435998917,
+            0.011373965069651604,
+            -0.009169982746243477,
+            -0.010196737945079803,
+            -0.0026310591492801905,
+            -0.013675311580300331,
+            0.00013850955292582512,
+            0.013693015091121197,
+            -0.012170584872364998,
+            -0.0195260439068079,
+            -0.016675913706421852,
+            -0.0061339749954640865,
+            -0.02189820259809494,
+            -0.02281874045729637,
+            -0.003064774675294757,
+            -0.0025381201412528753,
+            0.015374768525362015,
+            -0.012781327590346336,
+            0.004872659686952829,
+            -0.01665821112692356,
+            0.003991951700299978,
+            -0.004268555901944637,
+            -0.038627225905656815,
+            0.008758395910263062,
+            0.006063164211809635,
+            -0.00715187843888998,
+            -0.0037507531233131886,
+            0.019419828429818153,
+            -0.00709877023473382,
+            -0.008501706644892693,
+            -0.00043675804045051336,
+            0.004102593753486872,
+            0.014277203008532524,
+            0.0008995168609544635,
+            -0.013046867214143276,
+            -0.00026069945306517184,
+            -0.003642324125394225,
+            -0.008846908807754517,
+            -0.006271170452237129,
+            -0.0040738265961408615,
+            0.0008314722217619419,
+            -0.021225500851869583,
+            0.010825182311236858,
+            -0.0035737263970077038,
+            0.0032949093729257584,
+            0.03556466102600098,
+            -0.005332929082214832,
+            -0.026642518118023872,
+            -0.009240793064236641,
+            0.01664050854742527,
+            0.005133774131536484,
+            -0.012878691777586937,
+            -0.012471530586481094,
+            -0.0046425252221524715,
+            -0.014719769358634949,
+            -0.007629850413650274,
+            -0.01229450386017561,
+            0.0018001401331275702,
+            0.020889149978756905,
+            -0.025297114625573158,
+            -0.015613754279911518,
+            -0.007439546752721071,
+            -0.015923550352454185,
+            0.013781527988612652,
+            0.002002614550292492,
+            0.0018510353984311223,
+            -0.0074351211078464985,
+            0.02639468014240265,
+            0.004441156983375549,
+            -0.016233347356319427,
+            -0.01548098400235176,
+            0.0036047061439603567,
+            0.01311767753213644,
+            -0.0021641512867063284,
+            -0.026359274983406067,
+            0.030076835304498672,
+            0.0006124016945250332,
+            0.010648155584931374,
+            0.004379197955131531,
+            0.023208199068903923,
+            0.014047068543732166,
+            0.0007872155983932316,
+            0.0034675104543566704,
+            0.012613152153789997,
+            -1.3069548913335893e-05,
+            0.018216047435998917,
+            -0.0009874770184978843,
+            -0.03333412483334541,
+            0.012843286618590355,
+            0.00543029373511672,
+            -0.006058738566935062,
+            0.0047398898750543594,
+            0.016313008964061737,
+            -0.006757994182407856,
+            -0.011205789633095264,
+            0.016870643943548203,
+            0.0041601271368563175,
+            0.016215644776821136,
+            0.007085493300110102,
+            0.013710717670619488,
+            -0.01581733487546444,
+            -0.0057887728326022625,
+            -0.008922145701944828,
+            0.0016197942895814776,
+            -0.00228585721924901,
+            0.0025890152901411057,
+            0.024111036211252213,
+            0.003662239760160446,
+            0.01988009735941887,
+            0.002788170473650098,
+            0.0021210010163486004,
+            -0.02154414914548397,
+            0.006580967456102371,
+            0.013383218087255955,
+            0.00013837125152349472,
+            -0.028147244825959206,
+            0.0018001401331275702,
+            -0.017764627933502197,
+            0.014038216322660446,
+            0.017419425770640373,
+            0.00392999267205596,
+            0.002414201619103551,
+            0.007461674977093935,
+            0.004450008273124695,
+            -0.01579963229596615,
+            0.024465089663863182,
+            -0.028819944709539413,
+            0.010072818957269192,
+            -0.006563264410942793,
+            -0.023031173273921013,
+            -0.023916305974125862,
+            0.013834635727107525,
+            0.014648959040641785,
+            -0.00046690789167769253,
+            0.04252181202173233,
+            0.005217861849814653,
+            -0.00339006120339036,
+            0.014887944795191288,
+            0.013161934912204742,
+            0.013701866380870342,
+            -0.03781289979815483,
+            -0.015135781839489937,
+            0.007474951911717653,
+            0.010382615961134434,
+            -0.014994161203503609,
+            -0.0038857359904795885,
+            -0.00031975447200238705,
+            0.003423253772780299,
+            -0.00664292648434639,
+            -0.0017083075363188982,
+            -2.6139097826671787e-05,
+            0.004219873808324337,
+            -0.018924154341220856,
+            -0.009886940941214561,
+            -0.008501706644892693,
+            0.01202896423637867,
+            -0.0003352442872710526,
+            0.026642518118023872,
+            -0.005266543943434954,
+            0.003834840841591358,
+            -0.00877167284488678,
+            0.008178633637726307,
+            -0.014852539636194706,
+            0.0028523425571620464,
+            0.0024562454782426357,
+            0.009479779750108719,
+            0.02949264645576477,
+            -0.0012790178880095482,
+            0.010356061160564423,
+            -0.006164954509586096,
+            0.0178531426936388,
+            0.01709192804992199,
+            -0.007842282764613628,
+            -0.007952923886477947,
+            0.011922747828066349,
+            0.02715589478611946,
+            0.011719167232513428,
+            -0.01519774179905653,
+            0.009506333619356155,
+            0.036467500030994415,
+            -0.01266625989228487,
+            -0.028696026653051376,
+            0.008121099323034286,
+            0.008103396743535995,
+            0.012763625010848045,
+            -0.01579963229596615,
+            -0.0030913285445421934,
+            -0.0029209405183792114,
+            -0.012418422847986221,
+            0.03365277498960495,
+            -0.004669079091399908,
+            -0.012719367630779743,
+            0.002144235884770751,
+            0.007567890919744968,
+            -0.010957952588796616,
+            -0.00732005387544632,
+            -0.03365277498960495,
+            0.00931160431355238,
+            -0.024677520617842674,
+            -0.007651978638023138,
+            -0.02517319656908512,
+            -0.024535899981856346,
+            0.00801488384604454,
+            0.007842282764613628,
+            -0.02476603351533413,
+            0.002686380175873637,
+            -0.015764227136969566,
+            0.011179235763847828,
+            -0.015126930549740791,
+            0.0016795407282188535,
+            -0.0201102327555418,
+            0.014179837889969349,
+            0.007412992883473635,
+            0.0019318037666380405,
+            -0.006142826285213232,
+            -0.013321259059011936,
+            0.0005742303328588605,
+            -0.02481914311647415,
+            -0.016410375013947487,
+            -0.022482389584183693,
+            0.0047752950340509415,
+            -0.02225225605070591,
+            0.005394888576120138,
+            -0.0010643729474395514,
+            0.0004124168772250414,
+            -0.004903639201074839,
+            -0.03460871800780296,
+            -0.015286254696547985,
+            -0.02860751375555992,
+            -0.0018255878239870071,
+            -0.0033435916993767023,
+            0.030430888757109642,
+            -0.009754170663654804,
+            -0.007103195879608393,
+            0.00823174137622118,
+            0.018499288707971573,
+            -0.0009360286640003324,
+            -0.022588606923818588,
+            -0.0036024930886924267,
+            -0.00976302195340395,
+            0.014542742632329464,
+            0.0014471932081505656,
+            -0.02095996029675007,
+            0.0018598866881802678,
+            -0.008953125216066837,
+            -0.008298126049339771,
+            0.020021719858050346,
+            -0.014224094338715076,
+            0.0001842183992266655,
+            -0.0034121894277632236,
+            0.01683523878455162,
+            -0.0017957144882529974,
+            0.008395491167902946,
+            0.009798427112400532,
+            -0.00484168017283082,
+            -0.008067991584539413,
+            -0.011223492212593555,
+            0.029970619827508926,
+            0.016959156841039658,
+            -0.011365113779902458,
+            0.019030369818210602,
+            -0.005695833824574947,
+            0.010630453005433083,
+            -0.016135983169078827,
+            -0.012630854733288288,
+            -0.0019141010707244277,
+            0.0037618172354996204,
+            0.02264171466231346,
+            -0.012498084455728531,
+            0.00900623295456171,
+            -0.011019911617040634,
+            -0.010187885724008083,
+            0.006687183398753405,
+            -0.021119285374879837,
+            -0.0022947085089981556,
+            0.007266945671290159,
+            0.017322061583399773,
+            0.006793399341404438,
+            -0.0009188791736960411,
+            0.01436571590602398,
+            -0.03643209487199783,
+            0.023031173273921013,
+            0.009293901734054089,
+            0.01675557717680931,
+            -0.009647955186665058,
+            0.025084681808948517,
+            -0.00651015667244792,
+            -0.00836893729865551,
+            -0.0030758387874811888,
+            0.0278640016913414,
+            -0.028341973200440407,
+            -0.004257491789758205,
+            0.003027156461030245,
+            -0.01644578017294407,
+            0.015003012493252754,
+            -0.015056120231747627,
+            -0.019225098192691803,
+            -0.0015888145426288247,
+            0.027987919747829437,
+            0.020145637914538383,
+            -0.017047669738531113,
+            0.013905446976423264,
+            -0.0007213837816379964,
+            0.01865861378610134,
+            -0.0014914498897269368,
+            0.002449606778100133,
+            -0.0235445499420166,
+            -0.011126128025352955,
+            0.003907864447683096,
+            0.01022329181432724,
+            0.00497444998472929,
+            0.02368617057800293,
+            0.00957714393734932,
+            -0.010984506458044052,
+            0.003137798048555851,
+            0.01175457239151001,
+            0.0008934315992519259,
+            -0.012737070210278034,
+            -0.009462077170610428,
+            0.014808283187448978,
+            -0.01464010775089264,
+            -0.007041236851364374,
+            0.0065942443907260895,
+            0.00832910556346178,
+            0.03522831201553345,
+            0.002270367229357362,
+            -0.008138801902532578,
+            0.0001835268922150135,
+            0.037919119000434875,
+            0.0017968208994716406,
+            -0.023013470694422722,
+            -0.0005200158921070397,
+            -0.0014294905122369528,
+            0.00578434718772769,
+            -0.00012903586321044713,
+            0.001651880331337452,
+            0.015985511243343353,
+            -0.002372157759964466,
+            -0.0036091317888349295,
+            0.0005570809007622302,
+            0.003617983078584075,
+            0.0023854346945881844,
+            0.01575537584722042,
+            -0.008568092249333858,
+            0.006603095680475235,
+            -0.008191910572350025,
+            -0.009515184909105301,
+            -0.0005355057655833662,
+            0.0034011253155767918,
+            0.018534693866968155,
+            -0.01697685942053795,
+            0.007532485760748386,
+            0.0133920693770051,
+            0.012506935745477676,
+            -0.014489634893834591,
+            0.017083076760172844,
+            0.002892173593863845,
+            -8.899740350898355e-05,
+            0.020765231922268867,
+            0.006080866791307926,
+            -0.006585393100976944,
+            -0.033387232571840286,
+            0.01202011201530695,
+            0.012958354316651821,
+            0.02621765248477459,
+            -0.004536308813840151,
+            -0.001487024244852364,
+            0.0035604494623839855,
+            0.008501706644892693,
+            0.015392471104860306,
+            -0.006226914003491402,
+            -0.00489036226645112,
+            -0.008616774342954159,
+            -0.0019937630277127028,
+            0.001215952099300921,
+            0.009116875007748604,
+            0.013666460290551186,
+            -0.02081833966076374,
+            0.015666862949728966,
+            0.015985511243343353,
+            0.014710918068885803,
+            -0.020499691367149353,
+            0.011533289216458797,
+            -0.012887543067336082,
+            0.009382414631545544,
+            -0.020375771448016167,
+            -0.005213436204940081,
+            0.009453224949538708,
+            -0.004823977593332529,
+            -0.03703398257493973,
+            -0.0020446584094315767,
+            -0.0028656194917857647,
+            0.007258094381541014,
+            -0.0029209405183792114,
+            0.012347611598670483,
+            0.00865217950195074,
+            -0.008599071763455868,
+            -0.003348017344251275,
+            -0.0036688782274723053,
+            -0.008413193747401237,
+            0.01601206511259079,
+            -0.011267748661339283,
+            0.004961173050105572,
+            0.02101306803524494,
+            0.004100380931049585,
+            -0.018747126683592796,
+            0.00718285795301199,
+            -0.02336752414703369,
+            0.0014117879327386618,
+            0.024482792243361473,
+            -0.006368535105139017,
+            -0.006886338349431753,
+            -0.002708508400246501,
+            -0.020694419741630554,
+            -0.014356864616274834,
+            -0.004118083510547876,
+            0.02389860339462757,
+            -0.004474349785596132,
+            -0.02207522839307785,
+            0.015498686581850052,
+            0.023934008553624153,
+            -0.0089265713468194,
+            -0.00028213628684170544,
+            -0.0072979251854121685,
+            -0.014232946559786797,
+            -0.0003125627408735454,
+            0.0011772274738177657,
+            0.019048072397708893,
+            -0.0001921016228152439,
+            -0.0001586325088283047,
+            0.007116473279893398,
+            -0.005571915302425623,
+            -0.002701869932934642,
+            0.029899807646870613,
+            0.00985153578221798,
+            -0.004540734458714724,
+            -0.01648118533194065,
+            0.0193313155323267,
+            -0.007550188340246677,
+            0.014861390925943851,
+            -0.001994869438931346,
+            0.012161733582615852,
+            -0.009347009472548962,
+            -0.007603296544402838,
+            -0.0014095749938860536,
+            -0.0017647348577156663,
+            -0.024358872324228287,
+            0.015436727553606033,
+            0.00502755818888545,
+            -0.007647552993148565,
+            -0.04468153789639473,
+            -0.004531883168965578,
+            -0.01050653401762247,
+            0.0026797414757311344,
+            -0.008594646118581295,
+            -0.005846306681632996,
+            -0.006164954509586096,
+            0.0017127332976087928,
+            -0.042273975908756256,
+            0.010807479731738567,
+            0.021791985258460045,
+            0.0003241801168769598,
+            -0.002584589645266533,
+            -0.0076254247687757015,
+            0.006178231444209814,
+            0.016490036621689796,
+            -0.0029120889957994223,
+            0.004370346665382385,
+            0.008293700404465199,
+            -0.011312006041407585,
+            -0.010541939176619053,
+            0.02315509133040905,
+            0.018924154341220856,
+            0.009745319373905659,
+            0.014250649139285088,
+            0.00701468251645565,
+            -0.009417819790542126,
+            0.01634841598570347,
+            -0.024004818871617317,
+            -0.0122148422524333,
+            -0.005948096979409456,
+            -0.017888547852635384,
+            -0.005248841363936663,
+            -0.018694018945097923,
+            -0.015843888744711876,
+            0.01086058747023344,
+            0.03078494220972061,
+            0.00012578576570376754,
+            -0.007576742675155401,
+            -0.008377788588404655,
+            0.009488631039857864,
+            0.014312608167529106,
+            0.013259299099445343,
+            0.009515184909105301,
+            0.010834033600986004,
+            -0.020535096526145935,
+            -0.009302753023803234,
+            0.0018443968147039413,
+            -0.02876683697104454,
+            0.0026310591492801905,
+            -0.014746323227882385,
+            -0.0201102327555418,
+            -0.02278333529829979,
+            0.00823174137622118,
+            -0.007771471980959177,
+            0.0027549779042601585,
+            -0.005492253229022026,
+            -0.012896394357085228,
+            0.009285050444304943,
+            0.014330310747027397,
+            -0.004443369805812836,
+            -0.015118079259991646,
+            0.01104646548628807,
+            0.006589818745851517,
+            0.004363708198070526,
+            0.0003681601956486702,
+            -0.005200159270316362,
+            -0.0008176420233212411,
+            -0.0031599265057593584,
+            0.013569096103310585,
+            0.017322061583399773,
+            0.010125926695764065,
+            0.01864091120660305,
+            0.0027217853348702192,
+            0.006842081900686026,
+            -0.002210620790719986,
+            -0.003874671645462513,
+            -0.0010411381954327226,
+            -0.02517319656908512,
+            -0.0019849117379635572,
+            -0.024181846529245377,
+            0.012630854733288288,
+            0.001221484155394137,
+            -0.01878253184258938,
+            0.003179841907694936,
+            -0.051620982587337494,
+            -0.011471329256892204,
+            -0.011090721935033798,
+            0.005673705600202084,
+            -0.0055984691716730595,
+            -0.02005712501704693,
+            -0.018587803468108177,
+            0.012462679296731949,
+            -0.006001204717904329,
+            -0.00492576789110899,
+            -0.00031062652124091983,
+            0.004120296332985163,
+            0.009108023717999458,
+            -0.014445378445088863,
+            0.0068199532106518745,
+            -0.007355459034442902,
+            -0.03092656284570694,
+            0.0038813103456050158,
+            0.006337555591017008,
+            -0.013728420250117779,
+            -0.03940614312887192,
+            0.005014281254261732,
+            0.013754974119365215,
+            0.010267548263072968,
+            -0.013383218087255955,
+            -0.009771873243153095,
+            0.011896193958818913,
+            -0.0032086088322103024,
+            0.03041318617761135,
+            -0.008758395910263062,
+            -0.016242198646068573,
+            -0.01464010775089264,
+            0.006505731027573347,
+            0.02152644656598568,
+            0.013551393523812294,
+            -0.03476804122328758,
+            0.0009957751026377082,
+            -0.010612750425934792,
+            -0.0005637193680740893,
+            0.01220599003136158,
+            -0.0024783737026154995,
+            0.002996176714077592,
+            -0.001240293262526393,
+            0.021136987954378128,
+            0.010966803878545761,
+            0.01691490039229393,
+            0.02692575938999653,
+            0.00610742112621665,
+            0.0058241779915988445,
+            -0.01901266723871231,
+            -8.051775694184471e-06,
+            -0.019968610256910324,
+            0.009904643520712852,
+            -0.022836443036794662,
+            0.014834837056696415,
+            -0.004219873808324337,
+            -0.007461674977093935,
+            -2.411642981314799e-06,
+            0.006505731027573347,
+            -0.008612348698079586,
+            -0.016162537038326263,
+            -0.006085292436182499,
+            0.03404223173856735,
+            -0.015826186165213585,
+            0.01683523878455162,
+            -0.010984506458044052,
+            0.01465781033039093,
+            0.015330511145293713,
+            -0.0035958546213805676,
+            -0.018888747319579124,
+            -0.014055919833481312,
+            -0.014215243048965931,
+            -0.010143629275262356,
+            0.0222699586302042,
+            0.019101180136203766,
+            0.009931197389960289,
+            0.0010704583255574107,
+            0.005509955808520317,
+            -0.006036610342562199,
+            0.014666661620140076,
+            0.007138601504266262,
+            0.002217259258031845,
+            -0.01084288489073515,
+            -0.005545360967516899,
+            -0.009860387071967125,
+            0.011763423681259155,
+            0.005266543943434954,
+            -0.004100380931049585,
+            0.008377788588404655,
+            0.005642725620418787,
+            0.0016806471394374967,
+            -0.0015611541457474232,
+            -0.005262118298560381,
+            0.0008071310585364699,
+            0.0003344144788570702,
+            0.0026399106718599796,
+            -0.006961574777960777,
+            0.016197942197322845,
+            -0.012055518105626106,
+            -0.013454028405249119,
+            -0.024341169744729996,
+            -0.006430494599044323,
+            -0.005377185996621847,
+            0.001866525155492127,
+            -0.03643209487199783,
+            0.012807881459593773,
+            0.0003496277204249054,
+            0.0014671087265014648,
+            0.0013785953633487225,
+            -0.0025093534495681524,
+            0.02476603351533413,
+            -0.016729023307561874,
+            -0.010834033600986004,
+            -0.010497682727873325,
+            -0.0018952920800074935,
+            0.002427478553727269,
+            0.005872860550880432,
+            -0.002425265731289983,
+            0.014339162036776543,
+            -0.004372559487819672,
+            -0.009550590068101883,
+            0.006948297843337059,
+            -0.0010084989480674267,
+            -0.020375771448016167,
+            0.008621199987828732,
+            0.014578147791326046,
+            0.018003614619374275,
+            0.026536300778388977,
+            0.00432608975097537,
+            -0.009015084244310856,
+            -0.029262512922286987,
+            -0.004295110236853361,
+            0.03855641186237335,
+            -0.02552725002169609,
+            0.006740291137248278,
+            -0.006952723488211632,
+            0.026500895619392395,
+            0.015950104221701622,
+            0.005235564429312944,
+            0.012790178880095482,
+            -0.008275997824966908,
+            0.0050010038539767265,
+            -0.0009587102103978395,
+            0.008581369183957577,
+            -0.014073622412979603,
+            0.014480783604085445,
+            0.0013099975185468793,
+            -0.005076240282505751,
+            -0.004104806575924158,
+            -0.021986715495586395,
+            -0.024642115458846092,
+            0.019632259383797646,
+            -0.004611545242369175,
+            -0.012489233165979385,
+            0.013524839654564857,
+            -0.011444775387644768,
+            -0.01757875084877014,
+            0.03351115435361862,
+            -0.01258659828454256,
+            -0.015534092672169209,
+            0.0012945076450705528,
+            0.013808081857860088,
+            0.015056120231747627,
+            0.03547614812850952,
+            -0.008050289005041122,
+            0.02299576811492443,
+            0.0035781520418822765,
+            -0.010834033600986004,
+            -0.005563064012676477,
+            -0.02101306803524494,
+            -0.013454028405249119,
+            0.008793801069259644,
+            -0.017375169321894646,
+            0.00932930689305067,
+            0.017631858587265015,
+            0.002112149726599455,
+            0.0012934012338519096,
+            0.02115469053387642,
+            -0.02427035942673683,
+            0.015534092672169209,
+            -0.001430596923455596,
+            0.006129549350589514,
+            0.0012579959584400058,
+            -0.01807442493736744,
+            -0.02134941890835762,
+            -0.01428605429828167,
+            0.003914502914994955,
+            -0.030129943042993546,
+            0.005518807098269463,
+            -0.005386037286370993,
+            -0.020765231922268867,
+            -0.01589699648320675,
+            -0.008864611387252808,
+            -0.0072271148674190044,
+            0.009913494810461998,
+            -0.0016740086721256375,
+            -0.01772037148475647,
+            -0.01915428787469864,
+            0.0033878483809530735,
+            -0.00610299501568079,
+            -0.007253668736666441,
+            0.014914498664438725,
+            -0.010603899136185646,
+            0.011081870645284653,
+            0.01039146725088358,
+            -0.00519573362544179,
+            0.019278205931186676,
+            -0.00878937542438507,
+            0.03795452415943146,
+            -0.002117681782692671,
+            -0.007992755621671677,
+            -0.004801848903298378,
+            -0.0026686773635447025,
+            0.010603899136185646,
+            0.003272780915722251,
+            0.0038813103456050158,
+            0.007377587258815765,
+            0.013781527988612652,
+            -0.011099573224782944,
+            -0.0012248033890500665,
+            0.003967610653489828,
+            0.033369529992341995,
+            -0.014153284020721912,
+            -0.0128521379083395,
+            0.0032949093729257584,
+            -0.010161331854760647,
+            0.0002121554280165583,
+            -0.00024368830781895667,
+            -0.004615970887243748,
+            0.031900208443403244,
+            0.003887948812916875,
+            0.004503116477280855,
+            0.012515787035226822,
+            0.007085493300110102,
+            -0.026270762085914612,
+            0.0007169580785557628,
+            0.0035361081827431917,
+            0.002909876173362136,
+            0.018039019778370857,
+            -0.019844692200422287,
+            -0.019667666405439377,
+            0.008382214233279228,
+            -0.0033325275871902704,
+            -0.004204384051263332,
+            0.0020933407358825207,
+            -0.013418623246252537,
+            -0.00885133445262909,
+            -0.01897726207971573,
+            -0.0013011462287977338,
+            0.004664653446525335,
+            -0.005023132544010878,
+            0.019950907677412033,
+            -0.01793280430138111,
+            -0.04057451710104942,
+            -0.0018654187442734838,
+            0.01121464092284441,
+            0.016233347356319427,
+            0.015622605569660664,
+            0.020181043073534966,
+            -0.002648761961609125,
+            -0.013586798682808876,
+            0.015516390092670918,
+            0.0015854953089728951,
+            0.005341780371963978,
+            -0.0018997177248820662,
+            0.015826186165213585,
+            0.009940048679709435,
+            -0.0034785745665431023,
+            -0.004443369805812836,
+            0.03459101542830467,
+            -0.0046026939526200294,
+            -0.014374567195773125,
+            0.023066578432917595,
+            -0.012958354316651821,
+            0.004157914314419031,
+            0.005355057306587696,
+            -0.0008641115273348987,
+            0.012223693542182446,
+            0.021986715495586395,
+            0.018127532675862312,
+            -0.021809687837958336,
+            -0.00524441571906209,
+            -0.006430494599044323,
+            -0.023084281012415886,
+            -0.010789777152240276,
+            0.010028562508523464,
+            0.004916916601359844,
+            0.024854548275470734,
+            0.0007888752152211964,
+            0.005819752346724272,
+            -0.001608730060979724,
+            -0.008970827795565128,
+            0.0059259687550365925,
+            -0.0035737263970077038,
+            0.006806676276028156,
+            -0.009904643520712852,
+            0.019313612952828407,
+            -0.0012413996737450361,
+            -0.024164143949747086,
+            0.0014648959040641785,
+            0.004388049244880676,
+            -0.008992956019937992,
+            -0.0043504307977855206,
+            0.006926169153302908,
+            -0.0024960762821137905,
+            0.01303801592439413,
+            0.015312808565795422,
+            0.003624621545895934,
+            0.012604300864040852,
+            0.011630653403699398,
+            0.005492253229022026,
+            -0.006846507545560598,
+            0.0014018301153555512,
+            -0.0023478164803236723,
+            -0.004118083510547876,
+            0.01772037148475647,
+            -0.010267548263072968,
+            0.018552398309111595,
+            -0.011444775387644768,
+            0.007253668736666441,
+            0.01954374648630619,
+            -0.0020236363634467125,
+            -0.0012369740288704634,
+            -0.0024540324229747057,
+            0.017224697396159172,
+            0.01492334995418787,
+            -0.002376583404839039,
+            -0.0004226512392051518,
+            0.007868836633861065,
+            0.007426269818097353,
+            0.018959559500217438,
+            -0.0044876267202198505,
+            0.0015866017201915383,
+            -0.002130958717316389,
+            -0.0025934409350156784,
+            -0.021862797439098358,
+            0.0014814921887591481,
+            0.005660428665578365,
+            0.005191307980567217,
+            0.011170384474098682,
+            0.007076642010360956,
+            0.018747126683592796,
+            0.0018764829728752375,
+            -0.021437931805849075,
+            0.011648355983197689,
+            -0.01917199045419693,
+            -0.018083276227116585,
+            0.011993558146059513,
+            -0.023456037044525146,
+            -0.0032086088322103024,
+            -0.02586359903216362,
+            -0.0054834019392728806,
+            -0.00010566280980128795,
+            0.007138601504266262,
+            0.0133920693770051,
+            0.0029143018182367086,
+            0.007837857119739056,
+            -0.015534092672169209,
+            -0.010550790466368198,
+            0.009878089651465416,
+            0.008568092249333858,
+            -0.011223492212593555,
+            0.013967406004667282,
+            -0.007904241792857647,
+            -0.011869640089571476,
+            0.002974048489704728,
+            -0.010373763740062714,
+            -0.0032152472995221615,
+            -0.00157332478556782,
+            0.0063242786563932896,
+            0.0023367523681372404,
+            0.007802451495081186,
+            0.017808884382247925,
+            0.008780524134635925,
+            -0.001572218257933855,
+            -0.008324679918587208,
+            -0.025279412046074867,
+            -0.0014648959040641785,
+            -0.00465580215677619,
+            0.007010256871581078,
+            0.031085887923836708,
+            -0.001259102369658649,
+            0.012250247411429882,
+            0.002670890185981989,
+            0.007497080601751804,
+            -0.0001987401192309335,
+            -0.0004624822468031198,
+            -0.016082875430583954,
+            -0.006465899758040905,
+            -0.0037241990212351084,
+            -0.02097766287624836,
+            0.019756179302930832,
+            0.010559642687439919,
+            0.008050289005041122,
+            0.00912572629749775,
+            -0.015374768525362015,
+            -0.0030869028996676207,
+            0.01880023442208767,
+            -0.0028412784449756145,
+            3.170575655531138e-05,
+            0.017056521028280258,
+            -0.013454028405249119,
+            0.005864009261131287,
+            -0.0031333724036812782,
+            -0.015401322394609451,
+            0.006364109460264444,
+            -0.0036378984805196524,
+            -0.00865217950195074,
+            -0.002591228112578392,
+            0.024730628356337547,
+            0.02244698442518711,
+            0.012082071974873543,
+            -0.016605103388428688,
+            -0.011258897371590137,
+            -0.002659826073795557,
+            -0.001719371764920652,
+            0.011497884057462215,
+            -0.015516390092670918,
+            -0.024288062006235123,
+            0.008470727130770683,
+            -0.008581369183957577,
+            -0.019225098192691803,
+            -0.00020731484983116388,
+            0.02207522839307785,
+            -0.016162537038326263,
+            -0.004180043004453182,
+            0.013223893940448761,
+            -1.0778920113807544e-05,
+            -0.010975655168294907,
+            0.014055919833481312,
+            0.00489036226645112,
+            0.013480582274496555,
+            -0.011453626677393913,
+            -0.0065765418112277985,
+            0.007204986177384853,
+            -0.020588204264640808,
+            -0.00844859890639782,
+            -0.003230737056583166,
+            -0.018587803468108177,
+            0.015684565529227257,
+            -0.010462277568876743,
+            0.00914342887699604,
+            0.004204384051263332,
+            0.0035338953603059053,
+            0.010754371993243694,
+            0.0016286455793306231,
+            0.007603296544402838,
+            -0.004700058605521917,
+            -0.0269788671284914,
+            -0.030094537883996964,
+            0.0011893981136381626,
+            0.0024186272639781237,
+            0.013728420250117779,
+            -0.0026310591492801905,
+            -0.016056321561336517,
+            -0.00966565776616335,
+            0.017605304718017578,
+            -0.030342375859618187,
+            0.016790982335805893,
+            0.01842847838997841,
+            0.0024739480577409267,
+            0.001541238627396524,
+            0.006302150432020426,
+            0.0009874770184978843,
+            -0.016135983169078827,
+            -0.00489036226645112,
+            -0.0011407157871872187,
+            0.02170347236096859,
+            -0.005062963347882032,
+            0.015427876263856888,
+            -0.021774282678961754,
+            0.005518807098269463,
+            -0.012011260725557804,
+            0.0017912888433784246,
+            0.0004840573528781533,
+            0.005992353428155184,
+            0.003629047190770507,
+            -0.008992956019937992,
+            0.010090521536767483,
+            -0.005031983833760023,
+            0.00042514066444709897,
+            0.004275194834917784,
+            0.05392232909798622,
+            0.00392999267205596,
+            0.03795452415943146,
+            -0.015312808565795422,
+            -0.00885133445262909,
+            0.025421032682061195,
+            0.011480181477963924,
+            -0.007443972397595644,
+            0.01917199045419693,
+            0.010179034434258938,
+            0.018481586128473282,
+            0.003328101709485054,
+            -0.0008696436416357756,
+            0.02223455347120762,
+            0.018198344856500626,
+            0.006368535105139017,
+            0.008059140294790268,
+            0.008391065523028374,
+            -0.004016293212771416,
+            0.019118882715702057,
+            -0.0005769963609054685,
+            -0.0195260439068079,
+            0.007231540512293577,
+            -0.01277247630059719,
+            0.015516390092670918,
+            0.0006262319511733949,
+            0.0021331715397536755,
+            0.010533087886869907,
+            0.0034365307074040174,
+            0.0023190497886389494,
+            0.0027726804837584496,
+            0.0007540230872109532,
+            -0.0025890152901411057,
+            0.012506935745477676,
+            0.010913695208728313,
+            0.0031909062527120113,
+            -0.007005831226706505,
+            0.010913695208728313,
+            0.013445177115499973,
+            0.0033258888870477676,
+            -0.012524638324975967,
+            -0.011612950824201107,
+            -0.008700861595571041,
+            -0.015790781006217003,
+            0.010187885724008083,
+            0.003611344611272216,
+            -0.009993156418204308,
+            -0.025969816371798515,
+            0.00849728100001812,
+            0.00986923836171627,
+            -0.017746925354003906,
+            0.0030625618528574705,
+            0.00826272089034319,
+            0.000512824219185859,
+            0.006032184697687626,
+            0.020358068868517876,
+            0.0006129549001343548,
+            0.012179436162114143,
+            0.01130315475165844,
+            -0.0029452815651893616,
+            -0.004872659686952829,
+            0.015604902990162373,
+            0.01411787886172533,
+            -0.008992956019937992,
+            -0.016020916402339935,
+            -0.012347611598670483,
+            -0.035175204277038574,
+            -0.006381812505424023,
+            -0.00357593921944499,
+            -0.01237416546791792,
+            -0.002407562918961048,
+            -0.00023013469763100147,
+            0.012701665051281452,
+            -0.024730628356337547,
+            0.010311804711818695,
+            -0.021083878353238106,
+            -0.017454832792282104,
+            -0.00013359983859118074,
+            0.00674471678212285,
+            -0.01697685942053795,
+            0.00044173692003823817,
+            0.006850933190435171,
+            0.0009907962521538138,
+            0.0027992345858365297,
+            0.048399098217487335,
+            -0.02641238272190094,
+            -0.0008840270456857979,
+            -0.009293901734054089,
+            0.017587602138519287,
+            0.019295910373330116,
+            0.012356462888419628,
+            -0.004372559487819672,
+            0.008864611387252808,
+            -0.006164954509586096,
+            0.003783945459872484,
+            0.024465089663863182,
+            0.004157914314419031,
+            0.025279412046074867,
+            0.005213436204940081,
+            0.0017149461200460792,
+            -0.01475517451763153,
+            -0.005209010560065508,
+            -0.014445378445088863,
+            -0.0013365515042096376,
+            -0.021614959463477135,
+            -0.03675074130296707,
+            0.02062360942363739,
+            -0.017534494400024414,
+            0.015003012493252754,
+            -0.004934619180858135,
+            0.0008032586192712188,
+            0.005076240282505751,
+            0.022323066368699074,
+            0.015604902990162373,
+            -0.022199146449565887,
+            0.030678726732730865,
+            0.004290684591978788,
+            0.03328101709485054,
+            0.0026376976165920496,
+            -0.0014604702591896057,
+            0.0017868631985038519,
+            0.020535096526145935,
+            0.0003781179548241198,
+            -0.011639504693448544,
+            -0.0020601481664925814,
+            0.012285652570426464,
+            -0.017649561166763306,
+            0.003852543421089649,
+            -0.03018305078148842,
+            -0.012630854733288288,
+            0.0027439137920737267,
+            -0.0032749937381595373,
+            0.00048710001283325255,
+            -0.01436571590602398,
+            0.03271453455090523,
+            -0.0046203965321183205,
+            0.011807680130004883,
+            0.023385226726531982,
+            -0.0036378984805196524,
+            -0.01148903276771307,
+            -0.016870643943548203,
+            0.01883563958108425,
+            -0.013799230568110943,
+            0.01842847838997841,
+            0.009771873243153095,
+            0.0011595248943194747,
+            -0.0014859178336337209,
+            -0.0021951310336589813,
+            0.00015379194519482553,
+            0.0016441354528069496,
+            0.007289073895663023,
+            -0.004509754944592714,
+            -0.020535096526145935,
+            0.021314013749361038,
+            -0.011391667649149895,
+            -0.00618708273395896,
+            0.021986715495586395,
+            -0.013268150389194489,
+            0.006341981235891581,
+            0.015020715072751045,
+            -0.02788170427083969,
+            -0.03540533781051636,
+            0.019791584461927414,
+            -0.003113457001745701,
+            0.008156504482030869,
+            0.005594043526798487,
+            0.00841761939227581,
+            0.008935422636568546,
+            0.00994889996945858,
+            -0.008913294412195683,
+            0.05919772759079933,
+            -0.021190095692873,
+            0.0032108216546475887,
+            0.0007175113423727453,
+            -0.017295507714152336,
+            0.014082473702728748,
+            0.00939126592129469,
+            0.010612750425934792,
+            -0.013091123662889004,
+            -0.008174207992851734,
+            -0.017083076760172844,
+            3.744529385585338e-05,
+            0.0015666862018406391,
+            -0.0008696436416357756,
+            -0.006541136186569929,
+            -0.003989738877862692,
+            -0.0394415482878685,
+            -0.02765156887471676,
+            0.03813154995441437,
+            -0.005279821343719959,
+            0.016684764996170998,
+            -0.006443771533668041,
+            -0.008718564175069332,
+            0.0014471932081505656,
+            0.010364912450313568,
+            0.004135786090046167,
+            -0.012338760308921337,
+            -0.005828603636473417,
+            0.007412992883473635,
+            -0.0561174601316452,
+            -0.0022438133601099253,
+            0.0057710702531039715,
+            -0.007209412287920713,
+            0.0033192504197359085,
+            0.018322262912988663,
+            0.02460671029984951,
+            -0.011196938343346119,
+            0.011453626677393913,
+            -0.004427880048751831,
+            0.011825382709503174,
+            0.02315509133040905,
+            0.0027815320063382387,
+            0.005116071552038193,
+            0.013604501262307167,
+            0.004691207315772772,
+            -0.008762821555137634,
+            -0.009754170663654804,
+            -0.018127532675862312,
+            -0.011312006041407585,
+            0.006032184697687626,
+            0.0003062008472625166,
+            0.004396900534629822,
+            0.014560445211827755,
+            0.017959358170628548,
+            -0.015808483585715294,
+            0.02133171632885933,
+            0.013631055131554604,
+            -0.005859583616256714,
+            -1.9491951661620988e-06,
+            -0.014967607334256172,
+            -0.011922747828066349,
+            -0.004135786090046167,
+            -0.010772074572741985,
+            0.01212632842361927,
+            -0.04163667932152748,
+            0.019083477556705475,
+            -0.0032285242341458797,
+            -0.010081670247018337,
+            -0.011851937510073185,
+            0.006846507545560598,
+            0.032112643122673035,
+            0.0030470718629658222,
+            -0.014445378445088863,
+            0.0012358676176518202,
+            0.004531883168965578,
+            -0.023827793076634407,
+            0.01585274003446102,
+            -0.008333531208336353,
+            0.001153992721810937,
+            -0.009050489403307438,
+            -0.015773078426718712,
+            0.04443369805812836,
+            -0.014808283187448978,
+            0.02203982323408127,
+            0.034626420587301254,
+            0.029404133558273315,
+            -0.0010887141106650233,
+            -0.012551192194223404,
+            -0.0139320008456707,
+            0.017083076760172844,
+            0.0032019703648984432,
+            0.012179436162114143,
+            0.017516791820526123,
+            0.017410574480891228,
+            -0.013799230568110943,
+            0.014073622412979603,
+            -0.006346406880766153,
+            -0.006727014202624559,
+            0.00046995055163279176,
+            -0.0020490840543061495,
+            0.0073997159488499165,
+            0.013400920666754246,
+            0.02062360942363739,
+            -0.01086058747023344,
+            0.0017249038210138679,
+            0.02264171466231346,
+            0.017047669738531113,
+            0.011922747828066349,
+            -0.008072417229413986,
+            3.713411570060998e-05,
+            0.005164753645658493,
+            0.010356061160564423,
+            0.008382214233279228,
+            0.004580565728247166,
+            -0.01011707540601492,
+            0.024535899981856346,
+            0.004270769190043211,
+            0.0008060246473178267,
+            0.022588606923818588,
+            -0.01076322328299284,
+            -0.00701468251645565,
+            -0.0016242199344560504,
+            0.007882113568484783,
+            -0.006797824986279011,
+            -0.03558236360549927,
+            0.02025185339152813,
+            -0.015339363366365433,
+            -0.01707422360777855,
+            0.01022329181432724,
+            -0.010984506458044052,
+            0.007992755621671677,
+            -0.0025757383555173874,
+            0.013666460290551186,
+            -0.009435522370040417,
+            0.0015622605569660664,
+            0.007005831226706505,
+            -0.0029497072100639343,
+            0.017596453428268433,
+            -0.014941052533686161,
+            0.009320455603301525,
+            -0.008815929293632507,
+            0.04609775170683861,
+            0.0140293650329113,
+            0.0026376976165920496,
+            -0.0030382205732166767,
+            -0.008112248033285141,
+            0.0006129549001343548,
+            0.003963185008615255,
+            2.6778745450428687e-05,
+            -0.008205187506973743,
+            -0.02621765248477459,
+            -0.006757994182407856,
+            -0.002617782214656472,
+            0.01672017201781273,
+            -0.008395491167902946,
+            0.019402125850319862,
+            0.013701866380870342,
+            -0.019136585295200348,
+            -0.02281874045729637,
+            0.022287661209702492,
+            -0.00809454545378685,
+            -0.007054513785988092,
+            -0.0067491428926587105,
+            -0.01810983009636402,
+            -0.00024811396724544466,
+            0.025704275816679,
+            0.003976461943238974,
+            0.0015102589968591928,
+            0.0023920731619000435,
+            0.004770869389176369,
+            0.025704275816679,
+            0.001831119880080223,
+            0.023119686171412468,
+            0.004016293212771416,
+            0.009453224949538708,
+            -0.012064369395375252,
+            0.0011827596463263035,
+            0.0014770664274692535,
+            -0.019649963825941086,
+            -0.011896193958818913,
+            0.0036578141152858734,
+            0.0065765418112277985,
+            0.006850933190435171,
+            0.017145035788416862,
+            0.016923751682043076,
+            -0.014047068543732166,
+            0.009134577587246895,
+            -0.008581369183957577,
+            -0.004507542122155428,
+            0.00966565776616335,
+            0.012453828006982803,
+            -0.00846630148589611,
+            0.0016253263456746936,
+            0.009709914214909077,
+            -0.007174006663262844,
+            0.018198344856500626,
+            0.01583503745496273,
+            0.0001586325088283047,
+            -0.002617782214656472,
+            0.03094426542520523,
+            0.007492654956877232,
+            -0.015241998247802258,
+            0.009187685325741768,
+            -0.025456437841057777
+          ],
+          "title": "9 Conclusion"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "04b6f5c8-c693-46cc-a1e4-976e79aec442",
+      "type": "child",
+      "source": {
+        "id": "65742957-6bda-446d-a61a-0fe20313c760",
+        "properties": {
+          "page_content": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity\n\nWilliam Fedus∗ liamfedus@google.com Barret Zoph∗ barretzoph@google.com Noam Shazeer noam@google.com Google, Mountain View, CA 94043, USA Editor: Alexander Clark\n\nAbstract\n\nIn deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) models defy this and instead select different parameters for each incoming example. The result is a sparsely-activated model—with an outrageous number of parameters—but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs, and training instability. We address these with the introduction of the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques mitigate the instabilities, and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large (Raffel et al., 2019) to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the \"Colossal Clean Crawled Corpus\", and achieve a 4x speedup over the T5-XXL model.12 Keywords: mixture-of-experts, natural language processing, sparsity, large-scale machine learning, distributed computing arXiv:2101.03961v3 [cs.LG] 16 Jun 2022 Contents\n\n1 Introduction 3 2 Switch Transformer 4 2.1 Simplifying Sparse Routing 5 2.2 Efficient Sparse Routing 6 2.3 Putting It All Together: The Switch Transformer 8 2.4 Improved Training and Fine-Tuning Techniques 8 3 Scaling Properties 11 3.1 Scaling Results on a Step-Basis 12 3.2 Scaling Results on a Time-Basis 13 3.3 Scaling Versus a Larger Dense Model 13 4 Downstream Results 14 4.1 Fine-Tuning 14 4.2 Distillation 16 4.3 Multilingual Learning 17 5 Designing Models with Data, Model, and Expert-Parallelism 18 5.1 Data Parallelism 20 5.2 Model Parallelism 20 5.3 Model and Data Parallelism 21 5.4 Expert and Data Parallelism 22 5.5 Expert, Model and Data Parallelism 22 5.6 Towards Trillion Parameter Models 22 6 Related Work 24 7 Discussion 25 8 Future Work 26 9 Conclusion 27 A Switch for Attention 27 B Preventing Token Dropping with No-Token-Left-Behind 29\n\nC Encouraging Exploration Across Experts 29 D Switch Transformers in Lower Compute Regimes 29 E Relation of Upstream to Downstream Model Performance 32 F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/switch_transformers.md"
+          },
+          "summary": "The Switch Transformer is a sparsely-activated model that simplifies the Mixture of Experts (MoE) approach, allowing for efficient scaling to trillion-parameter models with reduced communication and computational costs. It achieves significant speedups in pre-training and fine-tuning by activating only a subset of neural network weights for each input, maintaining a constant computational cost. The model demonstrates superior scaling properties and improved performance across various natural language processing tasks, including multilingual settings. It also allows for effective distillation into smaller dense models, preserving a portion of the quality gains. The architecture is designed to maximize parameter count while keeping floating point operations per example constant, making it suitable for both large-scale and smaller-scale applications.",
+          "headlines": {
+            "1 Introduction": [],
+            "2 Switch Transformer": [
+              "2.1 Simplifying Sparse Routing",
+              "2.2 Efficient Sparse Routing",
+              "2.3 Putting It All Together: The Switch Transformer",
+              "2.4 Improved Training and Fine-Tuning Techniques"
+            ],
+            "3 Scaling Properties": [
+              "3.1 Scaling Results on a Step-Basis",
+              "3.2 Scaling Results on a Time-Basis",
+              "3.3 Scaling Versus a Larger Dense Model"
+            ],
+            "4 Downstream Results": [
+              "4.1 Fine-Tuning",
+              "4.2 Distillation",
+              "4.3 Multilingual Learning"
+            ],
+            "5 Designing Models with Data, Model, and Expert-Parallelism": [
+              "5.1 Data Parallelism",
+              "5.2 Model Parallelism",
+              "5.3 Model and Data Parallelism",
+              "5.4 Expert and Data Parallelism",
+              "5.5 Expert, Model and Data Parallelism",
+              "5.6 Towards Trillion Parameter Models"
+            ],
+            "6 Related Work": [],
+            "7 Discussion": [],
+            "8 Future Work": [],
+            "9 Conclusion": [],
+            "A Switch for Attention": [],
+            "B Preventing Token Dropping with No-Token-Left-Behind": [],
+            "C Encouraging Exploration Across Experts": [],
+            "D Switch Transformers in Lower Compute Regimes": [],
+            "E Relation of Upstream to Downstream Model Performance": [],
+            "F Pseudo Code for Switch Transformers": []
+          },
+          "summary_embedding": [
+            -0.0060447100549936295,
+            -0.00390331051312387,
+            -0.01784370467066765,
+            -0.0021820589900016785,
+            0.013227885589003563,
+            0.011338187381625175,
+            0.0165580902248621,
+            0.024240797385573387,
+            -0.005986624862998724,
+            0.03596621751785278,
+            -0.004852031823247671,
+            -0.0068811336532235146,
+            -0.003574162255972624,
+            0.014242436736822128,
+            0.011578272096812725,
+            -0.033302053809165955,
+            -0.024519605562090874,
+            0.01050176378339529,
+            0.016093410551548004,
+            -0.006795942317694426,
+            0.007473600562661886,
+            -0.030513975769281387,
+            -0.06124480068683624,
+            -0.006838538218289614,
+            -0.0026099516544491053,
+            0.01029265858232975,
+            0.0010329445358365774,
+            -0.010873507708311081,
+            -0.053469158709049225,
+            0.023311438038945198,
+            0.025294071063399315,
+            0.005092116538435221,
+            -0.029584616422653198,
+            0.0009361362317577004,
+            -0.05161044001579285,
+            -0.04448534920811653,
+            0.0379488542675972,
+            0.012561844661831856,
+            -0.04364892467856407,
+            0.039249956607818604,
+            -0.004875265993177891,
+            -0.0022440163884311914,
+            -0.021499186754226685,
+            -0.0034909071400761604,
+            -0.007039899472147226,
+            -0.009851212613284588,
+            -0.02874819189310074,
+            0.0014037203509360552,
+            0.021886419504880905,
+            -0.0056962003000080585,
+            0.023264970630407333,
+            -0.024271776899695396,
+            0.01936165988445282,
+            0.04240977764129639,
+            -0.06957805901765823,
+            0.024659009650349617,
+            0.017549408599734306,
+            0.05120771750807762,
+            0.06511713564395905,
+            0.018339363858103752,
+            0.021669568493962288,
+            -0.014141756109893322,
+            -0.017766257748007774,
+            -0.011872569099068642,
+            0.004673904739320278,
+            -0.014970434829592705,
+            -0.01891246810555458,
+            0.008449427783489227,
+            -0.020461400970816612,
+            0.008209343999624252,
+            -0.05576157942414284,
+            -0.014730350114405155,
+            -0.034200433641672134,
+            -0.01140014547854662,
+            0.017642343416810036,
+            -0.018215449526906013,
+            0.030126741155982018,
+            0.02997184917330742,
+            0.002638994250446558,
+            0.0007822110201232135,
+            0.021049996837973595,
+            0.006509389728307724,
+            -0.0053438181057572365,
+            0.012321759946644306,
+            0.028159597888588905,
+            -0.01450575515627861,
+            -0.010106786154210567,
+            -0.03454120084643364,
+            0.01861817203462124,
+            0.01589204929769039,
+            -0.04194509983062744,
+            -0.012569589540362358,
+            -0.013824224472045898,
+            0.032341714948415756,
+            -0.009549170732498169,
+            0.04098476096987724,
+            -0.010083552449941635,
+            0.001383390510454774,
+            -0.031149037182331085,
+            -0.009463978931307793,
+            0.017069239169359207,
+            0.006172496825456619,
+            -0.016589069738984108,
+            -0.0012846461031585932,
+            0.016170857474207878,
+            0.006617815233767033,
+            -0.030142230913043022,
+            0.04488807171583176,
+            0.023497309535741806,
+            0.055296897888183594,
+            0.05579255893826485,
+            -0.037391237914562225,
+            -0.002445377642288804,
+            0.0344172865152359,
+            -0.0228312686085701,
+            -0.015040136873722076,
+            -0.034014564007520676,
+            -0.01768881268799305,
+            0.035935238003730774,
+            -0.0318460576236248,
+            0.01556677371263504,
+            0.028066661208868027,
+            -0.025712283328175545,
+            0.003488970920443535,
+            -0.03943582624197006,
+            0.003622566582635045,
+            0.01220559049397707,
+            0.006095050368458033,
+            -0.0026293133851140738,
+            -0.013615119270980358,
+            0.008906363509595394,
+            0.003055269829928875,
+            0.00800798274576664,
+            -0.04615819454193115,
+            0.018478767946362495,
+            -0.00454224506393075,
+            0.04033420979976654,
+            0.042502716183662415,
+            -0.02730768360197544,
+            -0.016743963584303856,
+            0.008557853288948536,
+            0.012104909867048264,
+            0.04188314080238342,
+            -0.007287728600203991,
+            -0.00121784838847816,
+            -0.0007836631848476827,
+            0.007322579622268677,
+            0.047552235424518585,
+            0.004646798130124807,
+            0.0020697612781077623,
+            -0.027230238541960716,
+            0.014978179708123207,
+            -0.019733402878046036,
+            0.027648448944091797,
+            0.023373395204544067,
+            0.02942972257733345,
+            -0.04048910364508629,
+            0.017518429085612297,
+            -0.011818356812000275,
+            -0.005568413063883781,
+            -0.002726121572777629,
+            -0.02856232039630413,
+            -0.0074426219798624516,
+            0.033983584493398666,
+            0.015365412458777428,
+            0.012956822291016579,
+            0.03271345794200897,
+            0.0012052633101120591,
+            -0.042502716183662415,
+            0.03804178908467293,
+            0.024550583213567734,
+            0.04640602692961693,
+            0.023466331884264946,
+            0.010796061716973782,
+            0.006571347359567881,
+            0.013375034555792809,
+            -0.019377149641513824,
+            0.015907539054751396,
+            0.02560385875403881,
+            0.009750531986355782,
+            -0.03167567402124405,
+            -0.0009501734166406095,
+            0.01178737822920084,
+            -0.004546117503196001,
+            0.03701949119567871,
+            0.04104671627283096,
+            0.020569827407598495,
+            0.05030933395028114,
+            0.0020155487582087517,
+            0.017936641350388527,
+            0.005936284549534321,
+            0.02820606529712677,
+            0.030467506498098373,
+            -0.005285732913762331,
+            0.005076627247035503,
+            -0.01136916596442461,
+            -0.03385967016220093,
+            0.014110777527093887,
+            0.01561324205249548,
+            -0.008077683858573437,
+            -0.01697630248963833,
+            0.006033093202859163,
+            0.009169681929051876,
+            -0.01795213110744953,
+            -0.017456471920013428,
+            0.005971135571599007,
+            0.018866000697016716,
+            -0.013576395809650421,
+            0.024689987301826477,
+            -0.04442339017987251,
+            0.029119934886693954,
+            -0.033209118992090225,
+            -0.003384418087080121,
+            -0.010641167871654034,
+            -0.0030843124259263277,
+            0.0041666291654109955,
+            0.011314953677356243,
+            0.013406013138592243,
+            0.002042655134573579,
+            -0.013909416273236275,
+            0.024519605562090874,
+            0.018447788432240486,
+            0.012073930352926254,
+            -0.03148980438709259,
+            -0.014397329650819302,
+            -0.0015673263696953654,
+            -0.008743725717067719,
+            -0.034138478338718414,
+            0.026966920122504234,
+            0.014048820361495018,
+            -0.03844451159238815,
+            0.017797237262129784,
+            0.029538147151470184,
+            0.014420564286410809,
+            -0.004790074657648802,
+            -0.0463130883872509,
+            0.04045812413096428,
+            -0.030436528846621513,
+            -0.033333033323287964,
+            -0.010478530079126358,
+            -0.0003146269591525197,
+            0.02894955314695835,
+            -0.0064358157105743885,
+            -0.050835974514484406,
+            0.01184933539479971,
+            0.020631784573197365,
+            0.00010763872705865651,
+            0.005731051322072744,
+            -0.019593998789787292,
+            0.00028631053282879293,
+            -0.03395260497927666,
+            0.0090844901278615,
+            0.026750069111585617,
+            0.008867640048265457,
+            -0.016418687999248505,
+            0.03351890295743942,
+            -0.0004121613164898008,
+            -0.018695618957281113,
+            -0.003889757441356778,
+            0.006885006092488766,
+            0.034912943840026855,
+            0.004836542531847954,
+            -0.02946070022881031,
+            0.05665995925664902,
+            -0.02100352756679058,
+            0.021344292908906937,
+            0.05631919577717781,
+            -0.019439106807112694,
+            -0.008472662419080734,
+            -0.002065889071673155,
+            0.0026196325197815895,
+            -0.0260530486702919,
+            -0.008937342092394829,
+            0.032341714948415756,
+            0.041728246957063675,
+            -0.018742086365818977,
+            -0.001398879918269813,
+            0.006780453026294708,
+            0.009425255469977856,
+            0.004948840010911226,
+            0.02692045085132122,
+            -0.012128143571317196,
+            -0.014559967443346977,
+            0.05198218300938606,
+            0.028314489871263504,
+            -0.012081675231456757,
+            -0.0164031982421875,
+            0.019888296723365784,
+            -0.024844881147146225,
+            -0.020244551822543144,
+            -0.02431824430823326,
+            -0.020414933562278748,
+            -0.0042518205009400845,
+            0.009525936096906662,
+            0.037484172731637955,
+            -0.03968365862965584,
+            0.012747716158628464,
+            0.021700548008084297,
+            0.02981695532798767,
+            -0.019532041624188423,
+            -0.05910727381706238,
+            -0.0016128262504935265,
+            0.03296129032969475,
+            -0.015458349138498306,
+            -0.049379974603652954,
+            0.015164051204919815,
+            0.0260530486702919,
+            0.05957195535302162,
+            -0.001461805310100317,
+            -0.014954946003854275,
+            0.061926331371068954,
+            0.03026614524424076,
+            -0.004383479710668325,
+            -0.009494957514107227,
+            -0.0127864396199584,
+            0.003396035172045231,
+            0.00465454300865531,
+            -0.03364282101392746,
+            -0.041170634329319,
+            -0.011841590516269207,
+            -0.010284913703799248,
+            0.008108663372695446,
+            -0.040922801941633224,
+            0.03531566634774208,
+            -0.021917399019002914,
+            0.010950954630970955,
+            0.015473837964236736,
+            0.037391237914562225,
+            -0.021344292908906937,
+            -0.01284839678555727,
+            -0.01572941243648529,
+            -0.024968795478343964,
+            -0.014010096900165081,
+            0.021917399019002914,
+            0.007682706229388714,
+            3.4699722164077684e-05,
+            0.010075807571411133,
+            -0.024659009650349617,
+            -0.051052823662757874,
+            0.0291044469922781,
+            0.0015605497173964977,
+            -0.015938518568873405,
+            -0.004623564425855875,
+            -0.01977987214922905,
+            -0.01122201792895794,
+            -0.012608313001692295,
+            -0.039249956607818604,
+            0.021111954003572464,
+            -0.01135367713868618,
+            0.015876561403274536,
+            0.0009250032599084079,
+            0.003659353591501713,
+            0.028624277561903,
+            0.01116006076335907,
+            0.0051811798475682735,
+            -0.03255856782197952,
+            -0.018277406692504883,
+            0.010122275911271572,
+            -0.00561100896447897,
+            -0.018339363858103752,
+            -0.013351799920201302,
+            -0.05037129297852516,
+            -0.03305422514677048,
+            -0.008054450154304504,
+            -0.05415068939328194,
+            0.0020697612781077623,
+            -0.014296649023890495,
+            -0.02232012152671814,
+            -0.005990497302263975,
+            0.03107159025967121,
+            -0.003235333366319537,
+            0.025294071063399315,
+            0.0076594725251197815,
+            0.037360258400440216,
+            0.040117356926202774,
+            0.008426194079220295,
+            0.03065337985754013,
+            -0.0029991210903972387,
+            -0.016867877915501595,
+            0.05486319959163666,
+            -0.01881953328847885,
+            0.013529927469789982,
+            -0.008604321628808975,
+            0.008604321628808975,
+            0.03299226611852646,
+            -0.003794885240495205,
+            0.02007416822016239,
+            -0.0311955064535141,
+            -0.07998688519001007,
+            0.01637221872806549,
+            0.028159597888588905,
+            -0.06418777257204056,
+            0.014691627584397793,
+            -0.020275529474020004,
+            0.02052335813641548,
+            0.02411688305437565,
+            -0.0009114501299336553,
+            -0.014064309187233448,
+            0.005595519673079252,
+            0.0020833145827054977,
+            -0.01456771232187748,
+            -0.002691270550712943,
+            -0.009417511522769928,
+            0.008774704299867153,
+            -0.017007282003760338,
+            -0.013406013138592243,
+            -0.034231413155794144,
+            0.010873507708311081,
+            0.01233724970370531,
+            -0.003969140350818634,
+            0.0011500825639814138,
+            -0.03277541697025299,
+            -0.007876322604715824,
+            -0.02145271934568882,
+            0.03382869064807892,
+            0.012058441527187824,
+            -0.030699847266077995,
+            -0.007814365439116955,
+            0.015264731831848621,
+            -0.007876322604715824,
+            0.0008906363509595394,
+            0.024132372811436653,
+            0.02103450708091259,
+            0.0028364830650389194,
+            -0.0009700191440060735,
+            0.02438020147383213,
+            0.007427132688462734,
+            -0.01720864325761795,
+            0.028794659301638603,
+            0.014288905076682568,
+            0.060160547494888306,
+            -0.03271345794200897,
+            0.001297231181524694,
+            -0.04135650396347046,
+            -0.031118059530854225,
+            0.006408709101378918,
+            0.014350862242281437,
+            0.007597515359520912,
+            -0.028252532705664635,
+            0.005978880450129509,
+            0.02004319056868553,
+            0.01469937153160572,
+            0.009704063646495342,
+            0.0037580979987978935,
+            -0.036740683019161224,
+            -0.000691695255227387,
+            -0.011214273050427437,
+            -0.0038781403563916683,
+            0.035656433552503586,
+            -0.008186109364032745,
+            -0.05151750147342682,
+            -0.0006418389966711402,
+            -0.03255856782197952,
+            -0.0353776253759861,
+            0.021313315257430077,
+            -0.00839521549642086,
+            -0.0039207362569868565,
+            -0.013986863195896149,
+            -0.000537770101800561,
+            -0.010524998418986797,
+            0.027509044855833054,
+            0.028345469385385513,
+            0.022490503266453743,
+            -0.05384090170264244,
+            0.009479468688368797,
+            0.00023766436788719147,
+            -0.05817791447043419,
+            0.0009709871956147254,
+            -0.007117345929145813,
+            0.011191039346158504,
+            -0.020213572308421135,
+            -0.02414786070585251,
+            0.011500826105475426,
+            0.03392162546515465,
+            0.014304393902420998,
+            0.0321558453142643,
+            -0.013367289677262306,
+            -0.09250226616859436,
+            0.018292896449565887,
+            -0.008689512498676777,
+            0.04126356914639473,
+            0.00865078903734684,
+            -0.004278926644474268,
+            -0.015845581889152527,
+            -0.00865078903734684,
+            -0.006943091284483671,
+            0.01283290795981884,
+            0.004530628211796284,
+            -0.005626498255878687,
+            0.0016631665639579296,
+            0.013708055019378662,
+            -0.005967263597995043,
+            0.007481344975531101,
+            0.003651608945801854,
+            -0.013189162127673626,
+            -0.06059424951672554,
+            -0.04055105894804001,
+            -0.018540725111961365,
+            0.050835974514484406,
+            -0.0069159846752882,
+            -0.046684831380844116,
+            -0.010850274004042149,
+            0.05058814212679863,
+            0.012259802781045437,
+            -0.0334569476544857,
+            -0.0007841471815481782,
+            0.0050882440991699696,
+            0.015458349138498306,
+            0.004546117503196001,
+            -0.027292195707559586,
+            0.02820606529712677,
+            0.0008412640891037881,
+            0.02300165221095085,
+            -0.018339363858103752,
+            -0.005312839522957802,
+            0.035780347883701324,
+            -0.024844881147146225,
+            0.023791607469320297,
+            0.0055219451896846294,
+            0.031629208475351334,
+            0.002952652983367443,
+            0.010060317814350128,
+            0.004704883322119713,
+            0.0186491496860981,
+            0.004275054205209017,
+            0.0276639387011528,
+            0.011462102644145489,
+            0.006428070832043886,
+            0.021623101085424423,
+            -0.009378788061439991,
+            -0.014513500034809113,
+            0.01885051093995571,
+            -0.0014695499558001757,
+            0.025232113897800446,
+            -0.020538847893476486,
+            0.002726121572777629,
+            -0.0016883367206901312,
+            -0.017967619001865387,
+            -0.040922801941633224,
+            -0.0007405834621749818,
+            0.021747015416622162,
+            -0.011671207845211029,
+            -0.0228312686085701,
+            -0.013715799897909164,
+            -0.01585332676768303,
+            0.0038490979932248592,
+            0.034169457852840424,
+            -0.0398695282638073,
+            0.00963436160236597,
+            -0.000830615172162652,
+            -0.03125746175646782,
+            -0.0239619892090559,
+            -0.01093546487390995,
+            0.014614180661737919,
+            -0.01652711257338524,
+            -0.021173911169171333,
+            -0.028469383716583252,
+            -0.008410705253481865,
+            0.006048582494258881,
+            -0.05065010115504265,
+            -0.004054331686347723,
+            0.019965743646025658,
+            0.0276639387011528,
+            -0.003099801717326045,
+            -0.02177799493074417,
+            0.0012468908680602908,
+            0.022351099178195,
+            0.02936776541173458,
+            0.014226946979761124,
+            0.027586491778492928,
+            -0.002178186783567071,
+            -0.0011675080750137568,
+            -0.0073148347437381744,
+            0.01307299267500639,
+            0.0011287847300991416,
+            0.036616768687963486,
+            0.0122210793197155,
+            -0.020306508988142014,
+            0.0013959755888208747,
+            -0.006431943271309137,
+            0.029181893914937973,
+            -0.02270735427737236,
+            -9.832092473516241e-05,
+            0.03531566634774208,
+            0.027741385623812675,
+            0.004933350719511509,
+            -0.0143121387809515,
+            -0.05142456665635109,
+            0.021592123433947563,
+            -0.01765783317387104,
+            0.031025122851133347,
+            0.016666516661643982,
+            0.003397971158847213,
+            -0.0034212053287774324,
+            0.012368228286504745,
+            -0.022087780758738518,
+            -0.004406713880598545,
+            -0.021142931655049324,
+            0.04389675334095955,
+            -0.043184246867895126,
+            0.008472662419080734,
+            0.029445212334394455,
+            0.011926782317459583,
+            -0.019082851707935333,
+            -0.005425137002021074,
+            0.029212871566414833,
+            -0.00162637943867594,
+            0.01308848150074482,
+            0.0058665829710662365,
+            -0.026935940608382225,
+            0.01202746294438839,
+            -0.031660184264183044,
+            0.02177799493074417,
+            -0.013436991721391678,
+            0.018447788432240486,
+            0.03342596814036369,
+            -0.0030397805385291576,
+            -0.0038239278364926577,
+            -0.016263794153928757,
+            -0.00969631876796484,
+            0.0662323608994484,
+            0.01054823212325573,
+            0.03726732358336449,
+            0.0018974426202476025,
+            -0.009773765690624714,
+            0.007969259284436703,
+            0.0063041565008461475,
+            -0.012329504825174809,
+            0.010494019836187363,
+            0.024178840219974518,
+            0.0012730291346088052,
+            -0.002412462839856744,
+            -0.019408127292990685,
+            -0.03438630700111389,
+            0.003727119415998459,
+            -0.030932186171412468,
+            0.022691864520311356,
+            0.009208405390381813,
+            0.017456471920013428,
+            0.008240322582423687,
+            -0.015907539054751396,
+            0.03200095146894455,
+            0.05461536720395088,
+            0.0019768255297094584,
+            0.013375034555792809,
+            0.007938280701637268,
+            0.007036027032881975,
+            -0.011338187381625175,
+            0.04702559858560562,
+            0.000662652775645256,
+            0.005169562995433807,
+            0.022536970674991608,
+            0.02145271934568882,
+            0.022939693182706833,
+            0.002304037567228079,
+            -0.011694442480802536,
+            0.010478530079126358,
+            -0.0006302220281213522,
+            0.004445436876267195,
+            -0.010695381090044975,
+            -0.027865299955010414,
+            0.007492962293326855,
+            0.01302652433514595,
+            0.022598929703235626,
+            0.021592123433947563,
+            -0.03178409859538078,
+            -0.01855621486902237,
+            0.03364282101392746,
+            -0.02997184917330742,
+            0.03612111136317253,
+            -0.0014424435794353485,
+            -0.023807097226381302,
+            -0.0058356039226055145,
+            0.016929835081100464,
+            0.02656419761478901,
+            -0.029181893914937973,
+            0.01804506592452526,
+            0.029693040996789932,
+            -0.0074426219798624516,
+            0.0005939188995398581,
+            0.012112653814256191,
+            0.02447313815355301,
+            -0.009293596260249615,
+            -0.0024027819745242596,
+            -0.0013320822035893798,
+            -0.011926782317459583,
+            -0.018370341509580612,
+            -0.0044725434854626656,
+            -0.022629907354712486,
+            0.005564541090279818,
+            -0.007365175057202578,
+            -0.003481226274743676,
+            -0.008581086993217468,
+            0.01154729351401329,
+            0.001979729626327753,
+            -0.019857317209243774,
+            -0.022335609421133995,
+            0.01652711257338524,
+            0.03887821361422539,
+            -0.018215449526906013,
+            -0.02106548473238945,
+            0.024442158639431,
+            -0.01759587600827217,
+            -0.01601596549153328,
+            0.036678727716207504,
+            0.013739033602178097,
+            0.007736918982118368,
+            -0.03358086198568344,
+            -0.030049296095967293,
+            -0.03274443745613098,
+            0.006296411622315645,
+            0.03785591572523117,
+            -0.0028752065263688564,
+            0.02569679357111454,
+            0.03701949119567871,
+            0.0057891360484063625,
+            -0.028934063389897346,
+            -0.014823286794126034,
+            -0.045104920864105225,
+            0.060160547494888306,
+            0.0011481463443487883,
+            0.05161044001579285,
+            0.016294771805405617,
+            -0.01456771232187748,
+            0.00419760774821043,
+            -0.013251119293272495,
+            0.026192452758550644,
+            0.02280029095709324,
+            0.025929134339094162,
+            0.006582964211702347,
+            0.022505993023514748,
+            -0.0037522895727306604,
+            -0.012197845615446568,
+            0.007307090330868959,
+            0.014536733739078045,
+            0.0002497654058970511,
+            -0.011392400600016117,
+            0.015783624723553658,
+            -0.01307299267500639,
+            -0.011361422017216682,
+            0.023295948281884193,
+            0.01368482131510973,
+            -0.012344993650913239,
+            0.014064309187233448,
+            -0.008565598167479038,
+            0.01239920686930418,
+            -0.023915521800518036,
+            0.022691864520311356,
+            -0.03262052312493324,
+            0.009587894193828106,
+            0.0015053689712658525,
+            -0.0073148347437381744,
+            0.024550583213567734,
+            -0.0027454833034425974,
+            -0.05071205645799637,
+            0.03550153970718384,
+            -0.008015726692974567,
+            0.0037213109899312258,
+            0.006381602957844734,
+            0.011392400600016117,
+            -0.0012623801594600081,
+            0.03807276859879494,
+            0.01239920686930418,
+            0.015969496220350266,
+            0.02199484594166279,
+            0.011911292560398579,
+            -0.008433938957750797,
+            -0.004096927121281624,
+            -0.0016137943603098392,
+            -0.01784370467066765,
+            0.025944624096155167,
+            0.0018674320308491588,
+            0.03962169960141182,
+            -0.018168980255723,
+            -0.02300165221095085,
+            0.028066661208868027,
+            0.007597515359520912,
+            -0.02351279929280281,
+            -0.0017377089243382215,
+            0.01202746294438839,
+            -0.009185170754790306,
+            0.027385130524635315,
+            0.01196550577878952,
+            0.019113829359412193,
+            -0.018060555681586266,
+            -0.004340883810073137,
+            -0.005072754807770252,
+            -0.015938518568873405,
+            0.02238207869231701,
+            0.020771188661456108,
+            0.011152315884828568,
+            -0.011880313977599144,
+            -0.01307299267500639,
+            0.020600805059075356,
+            0.035036858171224594,
+            -0.0002744515077210963,
+            0.005672966130077839,
+            0.017487451434135437,
+            0.002373739378526807,
+            -0.04470219835639,
+            0.016232814639806747,
+            -0.014025585725903511,
+            0.011214273050427437,
+            0.019593998789787292,
+            -0.021917399019002914,
+            -0.03196997195482254,
+            -0.011539548635482788,
+            -0.0414804182946682,
+            0.00493722315877676,
+            -0.0069159846752882,
+            0.013576395809650421,
+            -0.010176488198339939,
+            -0.0014831031439825892,
+            0.016666516661643982,
+            0.0038103745318949223,
+            0.02177799493074417,
+            -0.011678952723741531,
+            -0.012073930352926254,
+            -0.020213572308421135,
+            -0.020786678418517113,
+            0.004278926644474268,
+            -0.030405549332499504,
+            -0.004488032776862383,
+            0.021375272423028946,
+            -0.04783104360103607,
+            -0.03689557686448097,
+            0.02898053266108036,
+            -0.008503641001880169,
+            0.020755698904395103,
+            0.022955182939767838,
+            -0.015094350092113018,
+            -0.0008717587334103882,
+            -0.018695618957281113,
+            0.02258343994617462,
+            -0.007837600074708462,
+            0.026300877332687378,
+            -0.0027551641687750816,
+            0.0006292539183050394,
+            -0.02659517526626587,
+            0.020306508988142014,
+            0.005227648187428713,
+            0.012275291606783867,
+            -0.02390003204345703,
+            0.006815304048359394,
+            -0.03133491054177284,
+            -0.01784370467066765,
+            0.0180140882730484,
+            -0.011152315884828568,
+            0.027989214286208153,
+            -0.006060199346393347,
+            0.028438406065106392,
+            0.022722844034433365,
+            0.024751944467425346,
+            0.02676555886864662,
+            -0.02363671362400055,
+            0.008495896123349667,
+            0.03726732358336449,
+            0.00865078903734684,
+            -0.015876561403274536,
+            0.008170620538294315,
+            -0.04863648861646652,
+            -0.008062195032835007,
+            0.05030933395028114,
+            0.002127846470102668,
+            0.023141054436564445,
+            0.016294771805405617,
+            -0.002873270306736231,
+            -0.005688455421477556,
+            0.010269423946738243,
+            -0.04296739399433136,
+            -0.023172033950686455,
+            0.022629907354712486,
+            -0.021561143919825554,
+            6.54212328754511e-07,
+            0.03122648410499096,
+            0.012120398692786694,
+            -0.0009109660750254989,
+            -0.05108380317687988,
+            0.02100352756679058,
+            0.01178737822920084,
+            0.014443797990679741,
+            -0.010300402529537678,
+            0.026068538427352905,
+            0.01733255758881569,
+            0.038413532078266144,
+            0.012755461037158966,
+            -0.002679653698578477,
+            -0.03782493621110916,
+            0.009177426807582378,
+            -0.021685058251023293,
+            0.0011471783509477973,
+            0.0074503663927316666,
+            -0.02273833192884922,
+            -0.024782923981547356,
+            0.0043912241235375404,
+            0.004588713403791189,
+            0.011880313977599144,
+            0.022087780758738518,
+            0.009417511522769928,
+            -0.021204888820648193,
+            0.015404135920107365,
+            -0.05446047708392143,
+            -0.015450604259967804,
+            -0.023884542286396027,
+            0.02566581591963768,
+            0.020012211054563522,
+            0.010184233076870441,
+            0.005545179359614849,
+            0.010904486291110516,
+            -0.020988039672374725,
+            -0.009239383973181248,
+            0.020631784573197365,
+            0.011074868962168694,
+            -0.012840652838349342,
+            0.04113965481519699,
+            -0.00011126903700642288,
+            0.031210994347929955,
+            -0.02280029095709324,
+            0.002941036131232977,
+            -0.001241082325577736,
+            -0.03649285435676575,
+            0.009130958467721939,
+            0.01981084980070591,
+            0.02151467651128769,
+            -0.01861817203462124,
+            0.0024143988266587257,
+            0.01048627495765686,
+            0.004766840487718582,
+            0.038351576775312424,
+            0.027121812105178833,
+            -0.044206541031599045,
+            0.011663463898003101,
+            0.012995545752346516,
+            -0.009363298304378986,
+            0.048419639468193054,
+            -0.017518429085612297,
+            0.017673322930932045,
+            0.008240322582423687,
+            -0.01798310875892639,
+            0.02318752370774746,
+            -0.01469937153160572,
+            -0.013282098807394505,
+            0.00650551775470376,
+            0.009192915633320808,
+            -0.0001770381786627695,
+            -0.037577107548713684,
+            -0.017859194427728653,
+            -0.005719434004276991,
+            0.0038297362625598907,
+            0.01562873087823391,
+            0.013498948886990547,
+            0.009967382065951824,
+            0.005161818582564592,
+            -0.00945623405277729,
+            -0.0016089539276435971,
+            0.008124152198433876,
+            -0.0005532593932002783,
+            0.0019216446671634912,
+            -0.01926872320473194,
+            0.02698240801692009,
+            0.02572777308523655,
+            -0.0018354853382334113,
+            -0.04652994126081467,
+            -0.0036709706764668226,
+            0.0037077576853334904,
+            -0.017735280096530914,
+            -0.0035315665882080793,
+            -0.04519785940647125,
+            -0.01810702309012413,
+            0.0143121387809515,
+            -0.014614180661737919,
+            -0.0460033044219017,
+            0.009076746180653572,
+            -0.013886182568967342,
+            0.0063970922492444515,
+            -0.00932457484304905,
+            0.005762029904872179,
+            0.003651608945801854,
+            0.005680711008608341,
+            -0.005909178406000137,
+            0.05108380317687988,
+            -0.03466511517763138,
+            0.028051171451807022,
+            0.008565598167479038,
+            -0.012437930330634117,
+            -0.02261441759765148,
+            0.00887538492679596,
+            -0.044981006532907486,
+            -0.033209118992090225,
+            0.016743963584303856,
+            -0.01781272701919079,
+            0.011074868962168694,
+            0.06161654368042946,
+            0.007330324035137892,
+            -0.023342415690422058,
+            -0.01233724970370531,
+            -0.021049996837973595,
+            -0.04191412031650543,
+            0.003442503046244383,
+            0.005463860463351011,
+            -0.004948840010911226,
+            -0.03807276859879494,
+            -0.021716037765145302,
+            0.04615819454193115,
+            -0.012004229240119457,
+            0.012422440573573112,
+            0.01080380566418171,
+            -0.0019410063978284597,
+            -0.04860550910234451,
+            -0.021019017323851585,
+            0.0395597442984581,
+            0.021886419504880905,
+            -0.008426194079220295,
+            0.0028364830650389194,
+            -0.018277406692504883,
+            -0.003289545886218548,
+            0.006296411622315645,
+            -0.009107724763453007,
+            -0.03562545403838158,
+            0.02067825198173523,
+            0.0054832217283546925,
+            -0.023373395204544067,
+            -0.025805220007896423,
+            -0.03296129032969475,
+            0.009944148361682892,
+            0.02453509531915188,
+            -0.01968693546950817,
+            0.005823987070471048,
+            0.00027203132049180567,
+            -0.000881923595443368,
+            0.00042740863864310086,
+            -0.009959637187421322,
+            -0.019764382392168045,
+            -0.020151615142822266,
+            0.03268248215317726,
+            -0.0060059865936636925,
+            0.0074387495405972,
+            -0.03630698472261429,
+            -0.010974188335239887,
+            0.011322698555886745,
+            -0.002395037328824401,
+            -0.013452480547130108,
+            -0.009092235006392002,
+            -0.01739451475441456,
+            -0.009882191196084023,
+            -0.007876322604715824,
+            0.022660886868834496,
+            0.006168624851852655,
+            0.005409647710621357,
+            0.0023795480374246836,
+            -0.008581086993217468,
+            0.02898053266108036,
+            0.017874684184789658,
+            -0.014188224449753761,
+            -0.00920066051185131,
+            0.03451022133231163,
+            0.02665713243186474,
+            -0.009115469641983509,
+            0.01798310875892639,
+            0.023264970630407333,
+            -0.0010484338272362947,
+            -0.01196550577878952,
+            0.001922612776979804,
+            -0.0009550138493068516,
+            -0.0032972905319184065,
+            0.00845717266201973,
+            -0.015365412458777428,
+            0.0315207801759243,
+            0.04287445917725563,
+            -0.002145271748304367,
+            0.009587894193828106,
+            -0.003727119415998459,
+            -0.008472662419080734,
+            0.017069239169359207,
+            0.009316830895841122,
+            0.04625113308429718,
+            0.01098193321377039,
+            0.005680711008608341,
+            -0.033983584493398666,
+            -0.001229465357027948,
+            -0.01184933539479971,
+            0.03308520466089249,
+            0.02061629481613636,
+            0.00889087375253439,
+            -0.008379725739359856,
+            0.01344473659992218,
+            0.008341003209352493,
+            0.00661007035523653,
+            0.020569827407598495,
+            -0.016542602330446243,
+            -0.017286090180277824,
+            0.030467506498098373,
+            -0.00672236829996109,
+            0.01289486512541771,
+            0.02029101923108101,
+            -0.002089123008772731,
+            0.010842529125511646,
+            0.07100307941436768,
+            -0.006184114143252373,
+            0.015713922679424286,
+            -0.019082851707935333,
+            0.0015789433382451534,
+            -0.007640110794454813,
+            -0.03370477631688118,
+            0.018246427178382874,
+            0.0039962464943528175,
+            0.00865078903734684,
+            0.01198099460452795,
+            0.009053511545062065,
+            -0.007946024648845196,
+            -0.00963436160236597,
+            0.015195030719041824,
+            0.0019080914789810777,
+            0.00033568276558071375,
+            0.001336922636255622,
+            -0.0065210070461034775,
+            -0.004011735785752535,
+            -0.021948376670479774,
+            0.02763296104967594,
+            0.007140580099076033,
+            0.030606910586357117,
+            -0.055296897888183594,
+            0.030095763504505157,
+            -0.00789568480104208,
+            0.02554190158843994,
+            -0.016201836988329887,
+            -0.011090358719229698,
+            0.001609921921044588,
+            0.018153492361307144,
+            0.004491904750466347,
+            0.006462921854108572,
+            0.011888058856129646,
+            0.023233991116285324,
+            -0.03444826602935791,
+            0.017053749412298203,
+            0.023110076785087585,
+            0.0048675211146473885,
+            0.0003535923024173826,
+            0.005165690556168556,
+            0.015458349138498306,
+            0.007198664825409651,
+            -0.023048119619488716,
+            -0.0025692922063171864,
+            0.017611365765333176,
+            -0.02306360937654972,
+            0.0036980770528316498,
+            -0.006923729553818703,
+            0.004623564425855875,
+            -0.0014705180656164885,
+            -0.03509881719946861,
+            -0.00100099784322083,
+            -0.025743262842297554,
+            0.01923774555325508,
+            0.009556914679706097,
+            -0.025867177173495293,
+            0.003932353109121323,
+            0.025650326162576675,
+            -0.0003695656778290868,
+            0.00821708794683218,
+            0.0004683101433329284,
+            0.00869725737720728,
+            -0.00974278710782528,
+            -0.01493945624679327,
+            0.004801691509783268,
+            0.01531120017170906,
+            -0.01493945624679327,
+            -0.007431004662066698,
+            -0.009510447271168232,
+            0.030560443177819252,
+            0.0006674932083114982,
+            0.004189862869679928,
+            -0.005045648664236069,
+            -0.01858719252049923,
+            -0.0032237162813544273,
+            -0.021142931655049324,
+            0.004367990419268608,
+            -0.0009046735358424485,
+            0.04197607934474945,
+            0.0038161829579621553,
+            -0.02029101923108101,
+            0.0032972905319184065,
+            -0.03136589005589485,
+            0.0005000148667022586,
+            -0.007276111748069525,
+            -0.02004319056868553,
+            0.0015005286550149322,
+            0.0016525175888091326,
+            0.012631546705961227,
+            -0.03255856782197952,
+            -0.009301341138780117,
+            0.018416810780763626,
+            -0.00020680672605521977,
+            -0.010687636211514473,
+            -0.017828216776251793,
+            0.00035189816844649613,
+            -0.011338187381625175,
+            -0.027973726391792297,
+            -0.02850036323070526,
+            -0.0026719090528786182,
+            0.013491204008460045,
+            -0.018277406692504883,
+            -0.014993669465184212,
+            0.0031249718740582466,
+            0.021592123433947563,
+            -0.009781510569155216,
+            0.008302279748022556,
+            0.02241305634379387,
+            -0.008255811408162117,
+            -0.005514200776815414,
+            -0.000941460661124438,
+            -0.009130958467721939,
+            0.004267309792339802,
+            0.006300284061580896,
+            0.043339136987924576,
+            -0.032403673976659775,
+            -0.0018238683696836233,
+            0.014815541915595531,
+            0.0034153966698795557,
+            0.0025905899237841368,
+            0.002503462601453066,
+            -0.0007841471815481782,
+            0.01344473659992218,
+            0.01178737822920084,
+            0.015466093085706234,
+            0.00100099784322083,
+            -0.0033011629711836576,
+            0.006408709101378918,
+            -0.021173911169171333,
+            0.015721667557954788,
+            0.027958236634731293,
+            -0.0033999073784798384,
+            0.005525817628949881,
+            -0.013731288723647594,
+            -0.008759214542806149,
+            0.013134949840605259,
+            0.0026641644071787596,
+            -0.005068882368505001,
+            -0.009394276887178421,
+            0.017967619001865387,
+            0.019082851707935333,
+            0.006668155547231436,
+            -0.037453193217515945,
+            -0.02898053266108036,
+            -0.009959637187421322,
+            0.020569827407598495,
+            0.014916222542524338,
+            0.0010339126456528902,
+            -0.010540487244725227,
+            -0.009595638141036034,
+            -0.011903548613190651,
+            0.010989678092300892,
+            0.02721474878489971,
+            -0.032372694462537766,
+            0.0008214184199459851,
+            -0.029383255168795586,
+            0.03068435750901699,
+            0.0020000594668090343,
+            0.011446612887084484,
+            -0.0008185141487047076,
+            -0.015140817500650883,
+            -0.029181893914937973,
+            0.013824224472045898,
+            -0.004395096562802792,
+            0.001024231780320406,
+            -0.0031753121875226498,
+            0.0201051477342844,
+            0.012887120246887207,
+            -0.0014472840121015906,
+            0.01968693546950817,
+            0.008178364485502243,
+            0.016480645164847374,
+            -0.010153254494071007,
+            -0.04135650396347046,
+            0.019965743646025658,
+            0.007562664337456226,
+            0.014219203032553196,
+            0.004739734344184399,
+            0.004588713403791189,
+            -0.020771188661456108,
+            0.002724185585975647,
+            0.004333139397203922,
+            0.014405074529349804,
+            0.011028401553630829,
+            0.0025615475606173277,
+            0.011678952723741531,
+            -0.02228914201259613,
+            0.01929970271885395,
+            -0.01631026156246662,
+            0.01006806269288063,
+            0.004445436876267195,
+            -0.026873983442783356,
+            0.007547175046056509,
+            0.007822110317647457,
+            -0.0019187404541298747,
+            -0.017549408599734306,
+            0.02160761132836342,
+            -0.0034831624943763018,
+            -0.03509881719946861,
+            0.0360281765460968,
+            -0.0043447562493383884,
+            0.019346170127391815,
+            0.03209388628602028,
+            -0.01771979033946991,
+            -0.01993476413190365,
+            0.00906900130212307,
+            0.010408828034996986,
+            -0.002873270306736231,
+            0.017859194427728653,
+            -0.002919738180935383,
+            -0.02232012152671814,
+            0.017766257748007774,
+            -0.028918573632836342,
+            -0.014234691858291626,
+            0.020337486639618874,
+            -0.03624502569437027,
+            -0.0021026763133704662,
+            0.03175312280654907,
+            -0.024070415645837784,
+            0.005734923295676708,
+            0.005196669604629278,
+            -0.0019351978553459048,
+            0.0012236569309607148,
+            -0.006451305001974106,
+            0.03801080957055092,
+            -0.010865762829780579,
+            0.0201051477342844,
+            0.01929970271885395,
+            -0.023388884961605072,
+            0.01977987214922905,
+            -0.001109423115849495,
+            0.01675945147871971,
+            -0.003940097521990538,
+            0.03466511517763138,
+            -0.006017603911459446,
+            -0.011330443434417248,
+            0.006431943271309137,
+            0.008488151244819164,
+            -0.009959637187421322,
+            0.008681767620146275,
+            -0.018122512847185135,
+            0.027152791619300842,
+            0.02091059274971485,
+            -0.0002691270783543587,
+            0.013800990767776966,
+            0.0004421719058882445,
+            -0.020833145827054977,
+            0.00789568480104208,
+            -0.027121812105178833,
+            -0.009967382065951824,
+            0.01646515540778637,
+            -0.005762029904872179,
+            0.007380664348602295,
+            -0.02064727433025837,
+            0.028004704043269157,
+            0.0077524082735180855,
+            -0.013591884635388851,
+            -0.0008824076503515244,
+            0.023760627955198288,
+            -0.005432881414890289,
+            0.009355553425848484,
+            0.02070923149585724,
+            0.015713922679424286,
+            0.013615119270980358,
+            0.016263794153928757,
+            0.03351890295743942,
+            0.005328328814357519,
+            0.005568413063883781,
+            0.003519949736073613,
+            0.016991792246699333,
+            -0.008557853288948536,
+            0.018029578030109406,
+            0.0008093173382803798,
+            -0.013576395809650421,
+            0.021499186754226685,
+            -0.0018684001406654716,
+            -0.013622863218188286,
+            -0.0040891822427511215,
+            -0.004337011836469173,
+            0.03271345794200897,
+            0.022041313350200653,
+            -0.004468671046197414,
+            -0.024442158639431,
+            -0.00394784240052104,
+            0.016682006418704987,
+            -0.003167567541822791,
+            0.011136826127767563,
+            0.0007952801533974707,
+            -0.01586107164621353,
+            -0.009595638141036034,
+            0.0063583687879145145,
+            0.010586955584585667,
+            -0.008921852335333824,
+            0.010346870869398117,
+            -0.0228312686085701,
+            -0.004011735785752535,
+            0.00046613195445388556,
+            -0.004844287410378456,
+            0.0053438181057572365,
+            -0.011314953677356243,
+            0.004948840010911226,
+            0.016062432900071144,
+            0.01026167906820774,
+            0.004832670092582703,
+            0.006989559158682823,
+            0.012066186405718327,
+            0.009161937050521374,
+            -0.01627928391098976,
+            0.00428667152300477,
+            -0.005072754807770252,
+            -0.02345084212720394,
+            0.005831731949001551,
+            0.00902253296226263,
+            -0.01122201792895794,
+            0.009386532939970493,
+            -0.010958699509501457,
+            0.03296129032969475,
+            0.01030814740806818,
+            -0.037391237914562225,
+            -0.0033379499800503254,
+            -0.0016244432190433145,
+            0.014885243959724903,
+            0.012298526242375374,
+            0.008550108410418034,
+            -0.017518429085612297,
+            -0.0047823297791182995,
+            -0.009835722856223583,
+            0.0012827098835259676,
+            -0.012306271120905876,
+            0.017456471920013428,
+            0.022567950189113617,
+            -0.036337960511446,
+            0.01968693546950817,
+            -0.010432061739265919,
+            -0.023652203381061554,
+            0.020476890727877617,
+            0.011500826105475426,
+            0.01942361705005169,
+            -0.009402021765708923,
+            0.018896980211138725,
+            -0.0001331114035565406,
+            0.025510922074317932,
+            -0.004228586331009865,
+            -0.022087780758738518,
+            -0.012499887496232986,
+            -0.03209388628602028,
+            -0.0051811798475682735,
+            0.006819176487624645,
+            -0.00045524101005867124,
+            0.013111716136336327,
+            0.014226946979761124,
+            -0.025712283328175545,
+            0.007295473478734493,
+            0.032589543610811234,
+            0.014637414366006851,
+            0.023311438038945198,
+            -0.011276230216026306,
+            -0.0010794125264510512,
+            -0.0026060794480144978,
+            -0.013770012184977531,
+            -0.03212486580014229,
+            -0.0017493258928880095,
+            -0.036585792899131775,
+            0.005680711008608341,
+            -0.016387708485126495,
+            -0.013096226379275322,
+            0.023264970630407333,
+            -0.034138478338718414,
+            0.0014608372002840042,
+            0.013708055019378662,
+            0.010540487244725227,
+            0.012391461990773678,
+            -0.00162637943867594,
+            0.015032391995191574,
+            -0.029243851080536842,
+            -0.010284913703799248,
+            -0.004491904750466347,
+            -0.01981084980070591,
+            -0.007338068913668394,
+            -0.018060555681586266,
+            -0.013127204962074757,
+            -0.003566417610272765,
+            -0.010060317814350128,
+            0.002247888594865799,
+            0.0032062907703220844,
+            -0.014714861288666725,
+            0.03206290677189827,
+            0.003157886676490307,
+            0.018958937376737595,
+            0.0033418224193155766,
+            0.004449309315532446,
+            0.018788553774356842,
+            -0.0026331855915486813,
+            -0.0058278595097362995,
+            0.030963165685534477,
+            -0.00032672801171429455,
+            0.008759214542806149,
+            0.019082851707935333,
+            0.004724245052784681,
+            0.0038529701996594667,
+            -0.0260530486702919,
+            0.012887120246887207,
+            -0.027447087690234184,
+            0.005080499220639467,
+            -0.0077175572514534,
+            -0.024302754551172256,
+            -0.028128618374466896,
+            -0.006571347359567881,
+            0.012058441527187824,
+            -0.02898053266108036,
+            -0.00613377382978797,
+            0.002315654419362545,
+            -0.00400399137288332,
+            -0.002673845272511244,
+            0.007241260726004839,
+            0.029150914400815964,
+            -0.033209118992090225,
+            0.008751469664275646,
+            0.005595519673079252,
+            -0.0005053392960689962,
+            0.013297587633132935,
+            0.008038961328566074,
+            0.003343758638948202,
+            0.020848635584115982,
+            0.006637176964432001,
+            0.002458930714055896,
+            -0.0025576751213520765,
+            -0.014242436736822128,
+            0.004000118933618069,
+            0.02154565416276455,
+            0.007562664337456226,
+            0.018385831266641617,
+            -0.008914108388125896,
+            0.002621568739414215,
+            -0.01455222349613905,
+            0.005552923772484064,
+            -0.002867461647838354,
+            0.004309905227273703,
+            -0.012236569076776505,
+            0.008093173615634441,
+            0.0028771425131708384,
+            -0.0043447562493383884,
+            0.018060555681586266,
+            -0.002319526858627796,
+            0.016480645164847374,
+            0.004015608225017786,
+            0.015620986931025982,
+            -0.0035412474535405636,
+            0.01178737822920084,
+            0.0005731051205657423,
+            -0.017905661836266518,
+            -0.017022771760821342,
+            -0.039249956607818604,
+            0.007702067960053682,
+            0.03531566634774208,
+            -0.0009564659558236599,
+            -0.03444826602935791,
+            0.010145509615540504,
+            -0.011942271143198013,
+            -0.009975126944482327,
+            -0.022862248122692108,
+            0.013862947933375835,
+            0.012430185452103615,
+            0.007415515370666981,
+            0.012252057902514935,
+            -0.006997303571552038,
+            0.01026167906820774,
+            -0.005216030869632959,
+            0.008356492035090923,
+            -0.0010823167394846678,
+            -0.037422213703393936,
+            0.012344993650913239,
+            0.02216522768139839,
+            -0.002635121811181307,
+            0.0016089539276435971,
+            -0.006145390681922436,
+            0.022103270515799522,
+            -0.013971373438835144,
+            -0.00650551775470376,
+            -0.00046443779137916863,
+            0.00797700323164463,
+            -0.01795213110744953,
+            0.004778457339853048,
+            0.0755879208445549,
+            -0.007775642443448305,
+            0.006122156511992216,
+            0.009270362555980682,
+            -0.015148562379181385,
+            -0.01637221872806549,
+            -0.004561606794595718,
+            -0.020260039716959,
+            0.008844406343996525,
+            -0.0018819533288478851,
+            -0.0110981035977602,
+            0.012468908913433552,
+            0.0053050946444272995,
+            0.002031038049608469,
+            -0.008379725739359856,
+            0.010811550542712212,
+            0.009525936096906662,
+            0.010432061739265919,
+            0.02360573597252369,
+            0.011477591469883919,
+            -0.00858883187174797,
+            -0.011795123107731342,
+            0.01129171997308731,
+            0.0028577810153365135,
+            0.016697494313120842,
+            0.007907302118837833,
+            0.0010648912284523249,
+            0.019501063972711563,
+            -0.002309845993295312,
+            0.02180897258222103,
+            -2.2689444449497387e-05,
+            -0.009301341138780117,
+            -0.004236331209540367,
+            -0.015349923633038998,
+            0.024271776899695396,
+            -0.0020155487582087517,
+            0.0015450604259967804,
+            -0.020213572308421135,
+            0.0013979118084535003,
+            0.013243375346064568,
+            0.011957760900259018,
+            -0.0020407189149409533,
+            -0.0051347119733691216,
+            0.0007154133054427803,
+            -0.016356728971004486,
+            0.032806396484375,
+            0.007578153628855944,
+            0.0007681738352403045,
+            0.0008838597568683326,
+            0.0009288755827583373,
+            -0.0032972905319184065,
+            0.0024608669336885214,
+            -0.004766840487718582,
+            0.009549170732498169,
+            -0.010943209752440453,
+            -0.010796061716973782,
+            0.010075807571411133,
+            -0.02402394637465477,
+            0.003566417610272765,
+            -0.00558777479454875,
+            -0.008062195032835007,
+            0.0076207490637898445,
+            0.00015537731815129519,
+            -0.007392281666398048,
+            -0.0030939930584281683,
+            0.0023408245760947466,
+            0.013429246842861176,
+            0.004406713880598545,
+            0.010850274004042149,
+            0.012956822291016579,
+            0.005359307397156954,
+            -0.006629432085901499,
+            0.015512561425566673,
+            -0.006946963258087635,
+            -0.004228586331009865,
+            -0.0012468908680602908,
+            -0.019996721297502518,
+            -0.0009419447160325944,
+            -0.004708755761384964,
+            0.0015092412941157818,
+            0.0001705036120256409,
+            -8.749050175538287e-05,
+            -0.0018490385264158249,
+            -0.011988739483058453,
+            -0.001831613015383482,
+            -0.011663463898003101,
+            -0.014498010277748108,
+            0.010253935120999813,
+            -0.0041201612912118435,
+            -0.022939693182706833,
+            -0.015349923633038998,
+            -0.020662762224674225,
+            0.02145271934568882,
+            -0.004387352149933577,
+            -0.005800753366202116,
+            0.006733985152095556,
+            -0.010215211659669876,
+            -0.015706177800893784,
+            -0.008069939911365509,
+            -0.0205853171646595,
+            0.010703125037252903,
+            0.003214035416021943,
+            0.0006791101768612862,
+            0.00882891658693552,
+            0.014397329650819302,
+            -0.01589204929769039,
+            0.0111755495890975,
+            0.012182355858385563,
+            0.011361422017216682,
+            -0.006854027509689331,
+            0.0008911203476600349,
+            -0.014103032648563385,
+            0.02596011385321617,
+            0.021623101085424423,
+            0.005169562995433807,
+            -0.04132552444934845,
+            -0.005862710531800985,
+            -0.013344055972993374,
+            -0.010524998418986797,
+            -0.012120398692786694,
+            0.011771888472139835,
+            -0.0038820127956569195,
+            0.0013117524795234203,
+            0.006288666743785143,
+            0.0015063370810821652,
+            0.004367990419268608,
+            -0.0022382077295333147,
+            -0.007709812838584185,
+            -0.03212486580014229,
+            -0.02817508764564991,
+            -0.00863530021160841,
+            0.020461400970816612,
+            0.002838419284671545,
+            -0.0009453330421820283,
+            -0.0036380558740347624,
+            0.0027048238553106785,
+            -0.005882072262465954,
+            -0.005018542055040598,
+            -0.014753584749996662,
+            -0.012739972211420536,
+            0.0101300198584795,
+            0.00802347157150507,
+            0.014195968396961689,
+            -0.011950016021728516,
+            0.0028190575540065765,
+            0.018788553774356842,
+            0.007690451107919216,
+            -0.023357905447483063,
+            0.01135367713868618,
+            -0.010912231169641018,
+            -0.009270362555980682,
+            -0.009433000348508358,
+            -0.009471723809838295,
+            0.0016980175860226154,
+            -0.008317768573760986,
+            0.003945906180888414,
+            -0.03615209087729454,
+            -0.011330443434417248,
+            -0.0013901671627536416,
+            -0.010346870869398117,
+            0.0009864765452221036,
+            0.02524760365486145,
+            0.006815304048359394,
+            0.01432762760668993,
+            -0.0006733016925863922,
+            0.017502939328551292,
+            -0.0018984107300639153,
+            0.018726596608757973,
+            0.012763205915689468,
+            -0.00797700323164463,
+            0.011717676185071468,
+            0.014738094992935658,
+            -0.0033863543067127466,
+            0.015620986931025982,
+            -0.010168743319809437,
+            0.005285732913762331,
+            -0.009556914679706097,
+            -0.003214035416021943,
+            0.002495717955753207,
+            0.030901208519935608,
+            -0.003618694143369794,
+            0.00858883187174797,
+            0.01098193321377039,
+            -0.015241498127579689,
+            0.0026951429899781942,
+            0.014962689951062202,
+            0.01141563430428505,
+            0.007760153152048588,
+            0.0024434414226561785,
+            -0.024225307628512383,
+            0.004193735308945179,
+            0.005037903785705566,
+            0.0031520782504230738,
+            -0.014366351068019867,
+            0.0021859314292669296,
+            0.015675200149416924,
+            0.008550108410418034,
+            -0.025325050577521324,
+            0.016418687999248505,
+            0.0009138703462667763,
+            -0.003909118939191103,
+            0.029119934886693954,
+            -0.011268485337495804,
+            -0.005723306443542242,
+            0.009463978931307793,
+            0.0009980935137718916,
+            0.0030029932968318462,
+            0.008937342092394829,
+            0.0228312686085701,
+            -0.003988501615822315,
+            -0.032341714948415756,
+            -0.022459525614976883,
+            0.015706177800893784,
+            -0.0047823297791182995,
+            -0.022939693182706833,
+            -0.026703599840402603,
+            -0.003113354789093137,
+            -0.02228914201259613,
+            -0.0022323993034660816,
+            -0.020058678463101387,
+            -0.014056564308702946,
+            0.001268188701942563,
+            0.029987338930368423,
+            0.016201836988329887,
+            0.008224832825362682,
+            -0.011051635257899761,
+            -0.029956359416246414,
+            -0.024271776899695396,
+            0.01810702309012413,
+            -0.007055388763546944,
+            -0.005541306920349598,
+            0.00591692328453064,
+            -0.009951893240213394,
+            -0.008480406366288662,
+            0.011802867986261845,
+            0.027137301862239838,
+            -0.002747419523075223,
+            -0.029987338930368423,
+            -0.02688947319984436,
+            -0.014335372485220432,
+            -0.01840132102370262,
+            -0.017704300582408905,
+            -0.008627555333077908,
+            0.0045035220682621,
+            0.00030300996149890125,
+            0.016186347231268883,
+            0.014962689951062202,
+            -0.000480653194244951,
+            0.0028597170021384954,
+            -2.7711375878425315e-05,
+            -0.001763847190886736,
+            0.006888878531754017,
+            -0.004933350719511509,
+            0.026997897773981094,
+            0.014614180661737919,
+            -0.006594581063836813,
+            0.018711106851696968,
+            0.028314489871263504,
+            0.020430423319339752,
+            0.003157886676490307,
+            -0.010695381090044975,
+            -0.014606435783207417,
+            0.009727297350764275,
+            -0.013421501964330673,
+            0.023264970630407333,
+            0.022552460432052612,
+            -0.01480779703706503,
+            -0.014885243959724903,
+            -0.017905661836266518,
+            0.015210519544780254,
+            -0.007562664337456226,
+            -0.008224832825362682,
+            0.012368228286504745,
+            0.022335609421133995,
+            -0.007024410180747509,
+            0.002371803391724825,
+            -0.0010232636705040932,
+            -0.021731525659561157,
+            -0.006122156511992216,
+            -0.004596457816660404,
+            -0.008666278794407845,
+            0.008681767620146275,
+            0.008131897076964378,
+            -0.001928421319462359,
+            -0.006319645792245865,
+            0.005471604876220226,
+            -0.013367289677262306,
+            0.01301103550940752,
+            -0.00789568480104208,
+            0.03379771113395691,
+            -0.009138703346252441,
+            -0.0006733016925863922,
+            -0.022598929703235626,
+            -0.011446612887084484,
+            0.00845717266201973,
+            0.0017967619933187962,
+            0.01682141050696373,
+            -0.0011626676423475146,
+            -0.005688455421477556,
+            0.009913169778883457,
+            -0.0010687635513022542,
+            -0.001450188341550529,
+            -0.006892750971019268,
+            -0.010401083156466484,
+            0.0002884887217078358,
+            0.013258864171802998,
+            -0.006025348324328661,
+            0.0018054747488349676,
+            0.006095050368458033,
+            -0.0175339188426733,
+            0.029027000069618225,
+            0.023884542286396027,
+            0.017255110666155815,
+            0.014660648070275784,
+            0.0006394187803380191,
+            0.011647974140942097,
+            -0.003572226269170642,
+            0.0038820127956569195,
+            -0.020337486639618874,
+            0.021080974489450455,
+            -0.010811550542712212,
+            0.01846327818930149,
+            0.011655719019472599,
+            -0.015233753249049187,
+            -0.005022414494305849,
+            0.021406250074505806,
+            0.009967382065951824,
+            0.00889087375253439,
+            0.0017541663255542517,
+            -0.02470547705888748,
+            0.017766257748007774,
+            0.010099041275680065,
+            -0.020399443805217743,
+            0.019547531381249428,
+            -0.009471723809838295,
+            0.01233724970370531,
+            -0.0005300253978930414,
+            0.004418330732733011,
+            0.0058743273839354515,
+            0.021499186754226685,
+            0.011570527218282223,
+            -0.03884723410010338,
+            -0.013615119270980358,
+            -0.0017357728211209178,
+            0.006857899948954582,
+            -0.0036303112283349037,
+            -0.015233753249049187,
+            -0.041635312139987946,
+            0.01604694314301014,
+            0.009998360648751259,
+            -0.01643417589366436,
+            0.004987563472241163,
+            -0.002427952131256461,
+            -0.0042053526267409325,
+            -0.008170620538294315,
+            0.003186929039657116,
+            -0.004170501604676247,
+            0.02324948087334633,
+            0.0018984107300639153,
+            -0.006946963258087635,
+            0.013258864171802998,
+            0.01344473659992218,
+            -0.021948376670479774,
+            0.0009874446550384164,
+            0.0021375271026045084,
+            0.000636998622212559,
+            0.023683181032538414,
+            -0.014707116410136223,
+            -0.0017657832941040397,
+            -0.01685238815844059,
+            -0.0013069120468571782,
+            -0.021592123433947563,
+            -0.001188805908896029,
+            -0.015187285840511322,
+            -0.008511385880410671,
+            0.025030752643942833,
+            -0.011005166918039322,
+            0.010997422970831394,
+            -0.006687517277896404,
+            0.006943091284483671,
+            -0.006904367823153734,
+            -0.01936165988445282,
+            -0.0008267428493127227,
+            0.009975126944482327,
+            -0.000512115890160203,
+            0.002941036131232977,
+            8.482827252009884e-05,
+            0.002464739140123129,
+            0.008867640048265457,
+            0.006234454456716776,
+            -0.0075278133153915405,
+            0.02489134855568409,
+            -0.012073930352926254,
+            0.0016399325104430318,
+            0.011911292560398579,
+            0.0004743606550619006,
+            -0.010424317792057991,
+            0.025588368996977806,
+            -0.02139076218008995,
+            -0.00800798274576664,
+            -0.00014799568452872336,
+            -0.0016544538084417582,
+            -0.006513262167572975,
+            0.006718495860695839,
+            0.02946070022881031,
+            0.0062654330395162106,
+            0.010447551496326923,
+            -0.0281441081315279,
+            -0.011493081226944923,
+            0.012538610957562923,
+            -0.022273652255535126,
+            -0.0007739823195151985,
+            -0.0004322490422055125,
+            -0.00619960343465209,
+            0.026378324255347252,
+            -0.02022906206548214,
+            -0.01093546487390995,
+            -0.018664639443159103,
+            0.005405775271356106,
+            -0.011121337302029133,
+            0.008294534869492054,
+            -0.0009777637897059321,
+            0.006540368776768446,
+            0.016805920749902725,
+            -0.019563021138310432,
+            0.016031453385949135,
+            0.005119222681969404,
+            -0.016217326745390892,
+            0.010222956538200378,
+            -0.017580386251211166,
+            -0.0008199662552215159,
+            0.020275529474020004,
+            -0.00400399137288332,
+            0.008751469664275646,
+            -0.003363120136782527,
+            0.005208286456763744,
+            -0.006513262167572975,
+            0.0001492057926952839,
+            -0.007249005138874054,
+            0.014428308233618736,
+            -0.016805920749902725,
+            -0.017828216776251793,
+            -0.008821171708405018,
+            0.0016709112096577883,
+            0.021297825500369072,
+            0.011136826127767563,
+            -0.010044828988611698,
+            -0.011059380136430264,
+            -0.011694442480802536,
+            0.041604332625865936,
+            0.005762029904872179,
+            0.027230238541960716,
+            -0.02261441759765148,
+            0.00545998802408576,
+            0.025170156732201576,
+            -0.014242436736822128,
+            0.0012788375606760383,
+            -0.0022788674104958773,
+            0.0053050946444272995,
+            0.011562783271074295,
+            -0.01819995976984501,
+            0.0049139889888465405,
+            -0.00019216447253711522,
+            -0.00782985519617796,
+            -0.000656844291370362,
+            0.003442503046244383,
+            0.00020196630794089288,
+            0.016325751319527626,
+            -0.004027225077152252,
+            -0.016511622816324234,
+            -0.004429947584867477,
+            -0.0003216455806978047,
+            -0.027896279469132423,
+            0.013514438644051552,
+            0.016589069738984108,
+            0.01566745527088642,
+            0.027989214286208153,
+            -0.014513500034809113,
+            0.016031453385949135,
+            0.009533680975437164,
+            0.013901671394705772,
+            -0.010610189288854599,
+            0.008952830918133259,
+            -0.01595400646328926,
+            0.007446494419127703,
+            -0.03454120084643364,
+            -0.007887939922511578,
+            -0.00802347157150507,
+            0.01984182931482792,
+            0.028763681650161743,
+            0.002923610620200634,
+            0.006493900436908007,
+            0.000196883876924403,
+            0.0024202074855566025,
+            -0.01714668609201908,
+            -0.020182594656944275,
+            -0.0003337466041557491,
+            0.0029332914855331182,
+            0.0006950835813768208,
+            -0.004495777189731598,
+            0.00472037261351943,
+            -0.012391461990773678,
+            0.0026835259050130844,
+            -0.013134949840605259,
+            0.005971135571599007,
+            -0.0059982421807944775,
+            0.009603383019566536,
+            0.0019429425010457635,
+            0.020864123478531837,
+            0.02300165221095085,
+            0.004100799560546875,
+            0.0046971384435892105,
+            -0.024333734065294266,
+            0.00920066051185131,
+            -0.013847459107637405,
+            0.012329504825174809,
+            0.019408127292990685,
+            0.008248066529631615,
+            0.006571347359567881,
+            0.01233724970370531,
+            0.012918098829686642,
+            -0.0031772484071552753,
+            0.024287264794111252,
+            0.005014669615775347,
+            -0.0009070937521755695,
+            0.002052335999906063,
+            -0.014954946003854275,
+            -0.0070979841984808445,
+            -0.0015286030247807503,
+            0.0018848575418815017,
+            -0.004731989465653896,
+            0.01906736195087433,
+            0.015256987884640694,
+            -0.01050176378339529,
+            0.017874684184789658,
+            -0.03937387093901634,
+            0.017379024997353554,
+            -0.0027377386577427387,
+            -0.0060911779291927814,
+            -0.00545998802408576,
+            0.00665653869509697,
+            0.004515138920396566,
+            0.003669034456834197,
+            0.03773200139403343,
+            -0.010408828034996986,
+            0.003963331691920757,
+            -0.005847221240401268,
+            0.007121218368411064,
+            -0.020864123478531837,
+            0.008302279748022556,
+            0.005336073227226734,
+            0.02160761132836342,
+            -0.0056497324258089066,
+            -0.0014482521219179034,
+            0.02328045852482319,
+            0.008859895169734955,
+            0.03469609469175339,
+            -0.01566745527088642,
+            -0.004418330732733011,
+            0.01048627495765686,
+            0.0207402091473341,
+            -0.00580462533980608,
+            -0.008976065553724766,
+            0.007903429679572582,
+            0.0035586729645729065,
+            0.021824462339282036,
+            -0.00515407370403409,
+            -0.0006650729919783771,
+            0.0054367538541555405,
+            -0.0060911779291927814,
+            0.009735042229294777,
+            0.011252996511757374,
+            -0.009076746180653572,
+            0.007539430167526007,
+            0.021530164405703545,
+            -0.009975126944482327,
+            -0.008658533915877342,
+            -0.009649851359426975,
+            0.0010658593382686377,
+            -0.03068435750901699,
+            0.001649613375775516,
+            0.005359307397156954,
+            0.015187285840511322,
+            -0.0004603234410751611,
+            -0.009014789015054703,
+            0.021019017323851585,
+            0.017255110666155815,
+            -0.005936284549534321,
+            -0.0028132491279393435,
+            0.0023795480374246836,
+            -0.0017154429806396365,
+            -0.006160879973322153,
+            -0.0006248975405469537,
+            -0.002656419761478901,
+            0.014730350114405155,
+            -0.0015469965292140841,
+            -0.006617815233767033,
+            -0.00756653631106019,
+            0.0033302053343504667,
+            0.010834784246981144,
+            -0.005661349277943373,
+            0.013057502917945385,
+            0.016805920749902725,
+            -0.00031704717548564076,
+            0.019671445712447166,
+            0.01037010457366705,
+            -0.010447551496326923,
+            0.015938518568873405,
+            -0.007984748110175133,
+            0.006563602481037378,
+            0.0164031982421875,
+            -0.01277095079421997,
+            -0.01246116403490305,
+            0.007210282143205404,
+            0.00665653869509697,
+            0.028918573632836342,
+            0.014792308211326599,
+            -0.02109646424651146,
+            0.014908477663993835,
+            0.015768134966492653,
+            -0.013018779456615448,
+            0.003537375247105956,
+            -0.01784370467066765,
+            0.020260039716959,
+            0.019051872193813324,
+            -0.015535795129835606,
+            -0.017224133014678955,
+            -0.014947201125323772,
+            -0.005092116538435221,
+            0.006331262644380331,
+            0.0033263331279158592,
+            -0.00617636926472187,
+            -0.027989214286208153,
+            -0.0049139889888465405,
+            -0.018633661791682243,
+            0.013870692811906338,
+            -0.008674023672938347,
+            0.0007715621031820774,
+            -0.004511266481131315,
+            0.007984748110175133,
+            -0.0033882902935147285,
+            0.032465629279613495,
+            -0.0051811798475682735,
+            -0.02832997962832451,
+            0.004398969002068043,
+            0.016356728971004486,
+            0.012623801827430725,
+            -0.029243851080536842,
+            0.010463040322065353,
+            0.004805563949048519,
+            -0.012530866079032421,
+            0.005990497302263975,
+            -0.02258343994617462,
+            -0.007632365915924311,
+            -0.0009395244996994734,
+            -0.016743963584303856,
+            -0.027121812105178833,
+            -0.008433938957750797,
+            -0.006993431597948074,
+            -0.005401902832090855,
+            -0.009525936096906662,
+            -0.008720491081476212,
+            -0.011477591469883919,
+            0.02303262986242771,
+            -0.007899557240307331,
+            -0.001866463921032846,
+            -0.0014637414133176208,
+            0.00930908601731062,
+            -0.023776117712259293,
+            -0.015605497173964977,
+            0.006389347370713949,
+            -0.002594462363049388,
+            0.022815778851509094,
+            -0.012747716158628464,
+            0.005940156988799572,
+            -0.009316830895841122,
+            0.009851212613284588,
+            0.015241498127579689,
+            0.004956584889441729,
+            0.017440982162952423,
+            -0.009889936074614525,
+            0.0021026763133704662,
+            -0.008627555333077908,
+            -0.00018986526993103325,
+            -1.027075541060185e-05,
+            0.014428308233618736,
+            0.007624621503055096,
+            0.014141756109893322,
+            0.017487451434135437,
+            -0.013553161174058914,
+            0.005003052763640881,
+            -0.0003274540649726987,
+            -0.017255110666155815,
+            0.0027725896798074245,
+            -0.007984748110175133,
+            -0.0008896682411432266,
+            0.001758038648404181,
+            6.395397940650582e-05,
+            -0.02112744189798832,
+            -0.0009124181815423071,
+            0.0018809852190315723,
+            0.0010222955606877804,
+            -0.025201136246323586,
+            0.00889087375253439,
+            -0.00889087375253439,
+            -0.0014918158994987607,
+            -0.019113829359412193,
+            0.019795360043644905,
+            0.026300877332687378,
+            -0.004619691986590624,
+            0.00036206303047947586,
+            -0.013537672348320484,
+            -0.02049238048493862,
+            0.0001489637652412057,
+            -0.011314953677356243,
+            -0.011345932260155678,
+            -0.0022169100120663643,
+            -0.003214035416021943,
+            0.008805682882666588,
+            0.0004189379105810076,
+            -0.009433000348508358,
+            -0.004956584889441729,
+            -0.011764144524931908,
+            -0.017549408599734306,
+            -0.010401083156466484,
+            0.01933068037033081,
+            -0.0018016024259850383,
+            0.0020794421434402466,
+            -0.0032508226577192545,
+            0.0005271211848594248,
+            0.011748654767870903,
+            0.013475715182721615,
+            0.013344055972993374,
+            -0.023915521800518036,
+            -0.017890173941850662,
+            -0.013320821337401867,
+            0.0029487807769328356,
+            -0.007187047973275185,
+            0.004844287410378456,
+            0.026362836360931396,
+            -0.0009695350890979171,
+            -0.01091997604817152,
+            0.002327271504327655,
+            0.019051872193813324,
+            -0.031272951513528824,
+            -0.013390523381531239,
+            -0.025077221915125847,
+            0.02225816436111927,
+            -0.013351799920201302,
+            0.011516314931213856,
+            0.010191977955400944,
+            -0.025944624096155167,
+            0.006768836174160242,
+            -0.007686578668653965,
+            0.026037558913230896,
+            0.01634124107658863,
+            0.0005934348446317017,
+            -0.00944074522703886,
+            0.013622863218188286,
+            -0.0008722427301108837,
+            -0.013576395809650421,
+            0.002201420720666647,
+            0.014645159244537354,
+            -0.014691627584397793,
+            -0.009510447271168232,
+            -0.004507394507527351,
+            0.004313777666538954,
+            0.012763205915689468,
+            -0.0030339721124619246,
+            -0.004031097516417503,
+            -0.011624740436673164,
+            0.013344055972993374,
+            -0.003930416889488697,
+            0.012383717112243176,
+            0.007380664348602295,
+            -0.014056564308702946,
+            -0.019532041624188423,
+            0.033333033323287964,
+            0.00400399137288332,
+            0.00338054564781487,
+            -0.010997422970831394,
+            0.02740062028169632,
+            -0.009897680021822453,
+            0.010571465827524662,
+            -0.006408709101378918,
+            0.0031985461246222258,
+            0.012321759946644306,
+            -0.012104909867048264,
+            -0.01981084980070591,
+            0.021468207240104675,
+            -0.00364192808046937,
+            -0.0010193913476541638,
+            0.008310024626553059,
+            -0.020182594656944275,
+            -0.00663330452516675,
+            0.002180122770369053,
+            0.009541425853967667,
+            -0.011818356812000275,
+            0.006369986105710268,
+            0.017905661836266518,
+            0.003912991378456354,
+            -0.0007672057254239917,
+            0.019996721297502518,
+            -0.01672847382724285,
+            0.011717676185071468,
+            -0.010850274004042149,
+            0.0016825281782075763,
+            0.0006065039779059589,
+            -0.0048597767017781734,
+            0.0058743273839354515,
+            -0.014536733739078045,
+            0.011988739483058453,
+            -6.244135147426277e-05,
+            -0.013979118317365646,
+            0.007601387333124876,
+            -0.013359544798731804,
+            -0.016836898401379585,
+            0.041542377322912216,
+            -0.005301222205162048,
+            -0.0038665232714265585,
+            -0.0056962003000080585,
+            0.01350669376552105,
+            0.02823704481124878,
+            -0.005336073227226734,
+            -0.01153180468827486,
+            -0.010780571959912777,
+            -0.016573579981923103,
+            0.015907539054751396,
+            -0.01810702309012413,
+            0.002495717955753207,
+            -0.010532742366194725,
+            -0.007903429679572582,
+            0.004302160814404488,
+            -0.003516077296808362,
+            0.014722606167197227,
+            0.00950270239263773,
+            0.023357905447483063,
+            0.00797700323164463,
+            -0.00027348342700861394,
+            0.010393339209258556,
+            0.011663463898003101,
+            0.003243078012019396,
+            -0.010509508661925793,
+            -0.00011949774489039555,
+            -0.02894955314695835,
+            -0.008449427783489227,
+            -0.004863648675382137,
+            -0.001485039247199893,
+            -0.00428667152300477,
+            -0.01239920686930418,
+            0.006617815233767033,
+            -0.0022052929271012545,
+            0.013909416273236275,
+            -0.009254872798919678,
+            -0.017766257748007774,
+            0.03389064967632294,
+            0.004495777189731598,
+            0.006466794293373823,
+            -0.008286789990961552,
+            -0.03705047070980072,
+            -0.0032566310837864876,
+            -0.008116407319903374,
+            -0.013529927469789982,
+            0.004538373090326786,
+            0.01283290795981884,
+            -0.014831030741333961,
+            -0.028097640722990036,
+            0.0019390701781958342,
+            -0.006683644838631153,
+            0.02978597767651081,
+            -0.007798876613378525,
+            0.00419760774821043,
+            0.0037251831963658333,
+            0.0011558911064639688,
+            0.01239920686930418,
+            -0.00845717266201973,
+            -0.0022536972537636757,
+            0.016387708485126495,
+            -0.010858018882572651,
+            0.011051635257899761,
+            -0.010966443456709385,
+            -0.02554190158843994,
+            -0.01456771232187748,
+            0.021344292908906937,
+            -0.0004823473282158375,
+            -0.00024903935263864696,
+            0.014823286794126034,
+            0.02354377694427967,
+            0.011260741390287876,
+            0.00969631876796484,
+            0.006408709101378918,
+            -0.001035848748870194,
+            0.005975008010864258,
+            0.006095050368458033,
+            -0.0024163350462913513,
+            -0.013398268260061741,
+            0.017564896494150162,
+            0.009773765690624714,
+            -0.01810702309012413,
+            0.0008514289511367679,
+            0.0036612898111343384,
+            0.0168833676725626,
+            0.006935346405953169,
+            0.009316830895841122,
+            -0.0143121387809515,
+            -0.023760627955198288,
+            -0.00014932679187040776,
+            0.002873270306736231,
+            0.015721667557954788,
+            0.010579210706055164,
+            -0.014319883659482002,
+            0.0005963391158729792,
+            0.008712747134268284,
+            0.020213572308421135,
+            -0.024209819734096527,
+            0.019702425226569176,
+            -0.004673904739320278,
+            -0.0021084847394376993,
+            0.007946024648845196,
+            0.011632485315203667,
+            -0.011872569099068642,
+            -0.00650551775470376,
+            0.006249943748116493,
+            0.003157886676490307,
+            0.03172214329242706,
+            -0.0017260919557884336,
+            0.009665340185165405,
+            -0.010997422970831394,
+            -0.0002405686245765537,
+            0.0006728176376782358,
+            -0.0006820144481025636,
+            -0.002884887158870697,
+            -0.011020656675100327,
+            0.0018974426202476025,
+            -0.020306508988142014,
+            0.0024473138619214296,
+            0.00463130883872509,
+            -0.005684582982212305,
+            -0.0032972905319184065,
+            -0.019222255796194077,
+            -0.0022672503255307674,
+            0.003293418325483799,
+            0.014335372485220432,
+            -0.031691163778305054,
+            0.004972074180841446,
+            0.010672146454453468,
+            -0.002309845993295312,
+            -0.010416572913527489,
+            0.011423379182815552,
+            -0.0026060794480144978,
+            0.011988739483058453,
+            -0.0038142469711601734,
+            -0.0004905760288238525,
+            0.025681305676698685,
+            -0.005308967083692551,
+            0.014707116410136223,
+            -0.01026167906820774,
+            -0.002993312431499362,
+            -0.028252532705664635,
+            0.005235392600297928,
+            -0.007403898518532515,
+            -0.0012081675231456757,
+            -0.004402841441333294,
+            -0.01130720879882574,
+            -0.0026990154292434454,
+            0.002122037811204791,
+            0.008000237867236137,
+            0.012910354882478714,
+            -0.0007725302129983902,
+            -0.010075807571411133,
+            0.004832670092582703,
+            0.0075355577282607555,
+            -0.015520306304097176,
+            -0.0070437719114124775,
+            -0.020306508988142014,
+            0.029662061482667923,
+            -0.013731288723647594,
+            -0.0011123273288831115,
+            -0.020864123478531837,
+            0.010160998441278934,
+            -0.0063119009137153625,
+            -0.002782270545139909,
+            0.004708755761384964,
+            -0.015109838917851448,
+            0.01344473659992218,
+            0.008751469664275646,
+            0.01450575515627861,
+            -0.02151467651128769,
+            -0.01762685552239418,
+            -0.0044725434854626656,
+            -0.002735802438110113,
+            -0.010648912750184536,
+            -0.0005334137240424752,
+            0.002013612538576126,
+            0.021251358091831207,
+            -0.00032503384863957763,
+            0.0042053526267409325,
+            -0.003214035416021943,
+            0.017224133014678955,
+            -0.010137764737010002,
+            -0.013057502917945385,
+            0.0024260159116238356,
+            0.005994369741529226,
+            0.016186347231268883,
+            0.03158273920416832,
+            -0.020693741738796234,
+            -0.00272999401204288,
+            0.0032740565948188305,
+            0.010656657628715038,
+            -0.004085310269147158,
+            6.304640555754304e-05,
+            -0.027803342789411545,
+            -0.04464024305343628,
+            -0.002422143705189228,
+            0.013057502917945385,
+            -0.015636475756764412,
+            0.012670270167291164,
+            0.008743725717067719,
+            0.010021595284342766,
+            -0.005196669604629278,
+            0.02620794251561165,
+            -0.004793947096914053,
+            0.014304393902420998,
+            0.011733165942132473,
+            0.011911292560398579,
+            0.0009874446550384164,
+            0.0007459079497493804,
+            -0.018354853615164757,
+            -0.008255811408162117,
+            -0.0015954007394611835,
+            0.01765783317387104,
+            -0.027478067204356194,
+            0.008526874706149101,
+            -0.011516314931213856,
+            0.013839714229106903,
+            0.0003666614356916398,
+            0.005661349277943373,
+            -0.01771979033946991,
+            0.012445674277842045,
+            -0.0009337160154245794,
+            -0.025325050577521324,
+            0.004805563949048519,
+            0.011276230216026306,
+            -0.010362359695136547,
+            0.006877261213958263,
+            0.019950253888964653,
+            0.0023079097736626863,
+            0.008495896123349667,
+            -0.014126266352832317,
+            0.010703125037252903,
+            0.009649851359426975,
+            -0.021080974489450455,
+            -0.01233724970370531,
+            -0.007059261202812195,
+            -0.006335135083645582,
+            -0.0035470561124384403,
+            -0.00013105422840453684,
+            0.010176488198339939,
+            -0.016000475734472275,
+            0.010099041275680065,
+            -0.006559730041772127,
+            0.006400964688509703,
+            0.00944074522703886,
+            0.001115231541916728,
+            -0.013576395809650421,
+            0.012716737575829029,
+            -0.0218709297478199,
+            0.008426194079220295,
+            -0.04287445917725563,
+            -0.005847221240401268,
+            -0.015078860335052013,
+            0.005204414017498493,
+            -0.003798757679760456,
+            -0.004453181754797697,
+            -0.006203475873917341,
+            -0.005061137955635786,
+            0.00920066051185131,
+            0.021747015416622162,
+            0.012135888449847698,
+            -0.016620049253106117,
+            -0.007551047019660473,
+            0.005425137002021074,
+            0.0004854935978073627,
+            -0.013336311094462872,
+            0.009316830895841122,
+            -0.01576039008796215,
+            0.0009259713697247207,
+            0.006339007057249546,
+            -0.01072635967284441,
+            0.013452480547130108,
+            0.004731989465653896,
+            -0.003401843598112464,
+            0.013390523381531239,
+            -0.0050417762249708176,
+            -0.0057891360484063625,
+            -0.007272239308804274,
+            -0.013739033602178097,
+            -0.003818119177594781,
+            -0.003072695340961218,
+            0.012747716158628464,
+            0.001529571134597063,
+            -0.0003243077953811735,
+            -0.00782985519617796,
+            0.009154192171990871,
+            0.002220782218500972,
+            -0.0013146566925570369,
+            0.005773646757006645,
+            -0.02300165221095085,
+            0.0001287550403503701,
+            0.004755223635584116,
+            -0.0009487213101238012,
+            -0.006246071308851242,
+            -0.012151377275586128,
+            -0.011903548613190651,
+            -0.038165703415870667,
+            0.009363298304378986,
+            0.0019197085639461875,
+            -0.0076207490637898445,
+            0.0064822835847735405,
+            0.017255110666155815,
+            0.01456771232187748,
+            0.016743963584303856,
+            -0.020693741738796234,
+            -0.02303262986242771,
+            0.005556796211749315,
+            -0.009402021765708923,
+            0.02261441759765148,
+            0.004046586807817221,
+            -0.005847221240401268,
+            0.0013243375578895211,
+            -0.013258864171802998,
+            0.008929597213864326,
+            0.005723306443542242,
+            -0.001592496526427567,
+            0.005425137002021074,
+            -0.013692565262317657,
+            -0.010904486291110516,
+            0.01048627495765686,
+            0.032496608793735504,
+            -0.0004261985304765403,
+            0.0024105266202241182,
+            0.023295948281884193,
+            0.015466093085706234,
+            0.016356728971004486,
+            -0.021111954003572464,
+            0.011167804710566998,
+            0.010060317814350128,
+            -0.0018645278178155422,
+            -0.014892988838255405,
+            0.00513083953410387,
+            -0.012716737575829029,
+            -0.010664401575922966,
+            0.014993669465184212,
+            0.007853088900446892,
+            0.017735280096530914,
+            -0.02369867078959942,
+            0.00789568480104208,
+            0.00042765066609717906,
+            0.00013710475468542427,
+            0.011524059809744358,
+            0.012956822291016579,
+            0.03370477631688118,
+            0.027152791619300842,
+            0.020662762224674225,
+            -0.009285852313041687,
+            0.014366351068019867,
+            -0.009192915633320808,
+            -0.011090358719229698,
+            -0.019470084458589554,
+            -0.015249243006110191,
+            -0.011136826127767563,
+            0.017998598515987396,
+            0.008790193125605583,
+            -0.009355553425848484,
+            -0.005947901867330074,
+            0.01586107164621353,
+            -0.003909118939191103,
+            -0.004635181277990341,
+            0.01646515540778637,
+            -0.026269899681210518,
+            0.016836898401379585,
+            -0.000745423894841224,
+            -0.014668392948806286,
+            -0.014691627584397793,
+            -0.004182118456810713,
+            -0.012670270167291164,
+            0.0003964299685321748,
+            -0.020817656069993973,
+            0.013181418180465698,
+            0.024597052484750748,
+            -0.0030920570716261864,
+            -0.004100799560546875,
+            -0.011299464851617813,
+            -0.0005111477803438902,
+            0.018385831266641617,
+            0.0034773540683090687,
+            0.009192915633320808,
+            -0.0023872926831245422,
+            0.013924905098974705,
+            0.00524313747882843,
+            0.0027958236169070005,
+            -0.0007565568666905165,
+            -0.016263794153928757,
+            0.012197845615446568,
+            0.0026835259050130844,
+            0.012135888449847698,
+            -0.022242674604058266,
+            0.007938280701637268,
+            -0.011764144524931908,
+            -0.0023698671720921993,
+            -0.02228914201259613,
+            0.0189744271337986,
+            -0.01610890030860901,
+            0.001046497724018991,
+            0.003841353114694357,
+            0.0006224773242138326,
+            0.009580149315297604,
+            0.01610890030860901,
+            -0.023388884961605072,
+            -0.0003460896550677717,
+            -0.014908477663993835,
+            -0.021917399019002914,
+            -0.01607792265713215,
+            0.004728117026388645,
+            -0.019501063972711563,
+            0.005444498732686043,
+            -0.0025557391345500946,
+            -0.015125328674912453,
+            0.018478767946362495,
+            0.014691627584397793,
+            -0.004379607271403074,
+            0.002412462839856744,
+            0.005425137002021074,
+            -0.0014405074762180448,
+            0.01971791312098503,
+            0.00013274837692733854,
+            -0.0010048701660707593,
+            -0.009425255469977856,
+            -0.0168833676725626,
+            -0.0065752193331718445,
+            -0.035005878657102585,
+            0.0260530486702919,
+            -0.004670032300055027,
+            -0.016511622816324234,
+            0.014319883659482002,
+            0.009115469641983509,
+            -0.014629669487476349,
+            -0.007059261202812195,
+            0.016294771805405617,
+            -0.024287264794111252,
+            0.02244403585791588,
+            -0.009611127898097038,
+            -0.008774704299867153,
+            0.00784921646118164,
+            0.05154848098754883,
+            0.009146448224782944,
+            0.003492843359708786,
+            0.011036145500838757,
+            0.005839476361870766,
+            -0.0013165927957743406,
+            -0.023683181032538414,
+            0.005568413063883781,
+            -0.007190920412540436,
+            -0.003601268632337451,
+            0.002536377403885126,
+            -0.004801691509783268,
+            0.031149037182331085,
+            -0.004379607271403074,
+            -0.005289605353027582,
+            0.004484160337597132,
+            -0.02196386642754078,
+            -0.0001332324172835797,
+            0.023203013464808464,
+            -2.3128106477088295e-05,
+            -0.023590246215462685,
+            -0.006571347359567881,
+            -0.015071115456521511,
+            -0.013770012184977531,
+            -0.018989915028214455,
+            0.005231520161032677,
+            -0.012755461037158966,
+            -0.0033940989524126053,
+            0.0016660707769915462,
+            -0.005940156988799572,
+            -0.002582845278084278,
+            0.009603383019566536,
+            0.05362405255436897,
+            0.014103032648563385,
+            -0.006335135083645582,
+            -0.005854965653270483,
+            0.009192915633320808,
+            -0.010524998418986797,
+            -0.019795360043644905,
+            -0.009820234030485153,
+            -0.029507169499993324,
+            -0.012747716158628464,
+            -0.0143121387809515,
+            -0.010191977955400944,
+            -0.010277168825268745,
+            0.00963436160236597,
+            -0.018122512847185135,
+            -0.007628493942320347,
+            -0.020337486639618874,
+            0.002941036131232977,
+            -0.011268485337495804,
+            -0.014714861288666725,
+            -0.008054450154304504,
+            -0.04807887226343155,
+            0.005796880926936865,
+            0.019593998789787292,
+            0.01263929158449173,
+            0.002515079453587532,
+            0.004383479710668325,
+            -0.01993476413190365,
+            -0.030792783945798874,
+            0.013839714229106903,
+            0.011283975094556808,
+            0.011733165942132473,
+            0.03283737599849701,
+            -0.011709931306540966,
+            -0.00813964195549488,
+            -0.004693266469985247,
+            -0.0017251238459721208,
+            0.012066186405718327,
+            0.00969631876796484,
+            -0.004917861428111792,
+            -0.012375972233712673,
+            0.016217326745390892,
+            -0.01981084980070591,
+            -0.023683181032538414,
+            -0.01672847382724285,
+            0.006157007534056902,
+            0.00813964195549488,
+            -0.008054450154304504,
+            0.002973950933665037,
+            0.0056962003000080585,
+            -0.007159941829741001,
+            -0.00022423222253564745,
+            0.004890755284577608,
+            -0.023048119619488716,
+            0.0008490087930113077,
+            -0.002257569460198283,
+            0.0061376458033919334,
+            0.021266845986247063,
+            -0.010168743319809437,
+            -0.0014075926737859845,
+            -0.0053902859799563885,
+            -0.0028984404634684324,
+            -0.006764963734894991,
+            0.02620794251561165,
+            0.009154192171990871,
+            -0.009882191196084023,
+            -0.01011453103274107,
+            -0.01129171997308731,
+            -0.0011278166202828288,
+            -0.017255110666155815,
+            -0.016186347231268883,
+            0.0015566773945465684,
+            -0.016712984070181847,
+            0.013971373438835144,
+            -0.00876695942133665,
+            0.00400399137288332,
+            -0.014110777527093887,
+            0.02019808255136013,
+            0.01929970271885395,
+            -0.009711808525025845,
+            0.023776117712259293,
+            -0.007709812838584185,
+            -0.038382552564144135,
+            -0.00800798274576664,
+            0.01843230053782463,
+            -0.006118284538388252,
+            0.010037084110081196,
+            -0.011314953677356243,
+            0.008209343999624252,
+            0.00015319912927225232,
+            0.0018122512847185135,
+            -0.0003112386621069163,
+            0.027555514127016068,
+            0.015535795129835606,
+            -0.005560668651014566,
+            -0.005494839046150446,
+            0.004255692940205336,
+            0.009541425853967667,
+            0.017580386251211166,
+            0.012151377275586128,
+            -0.021019017323851585,
+            -0.014257925562560558,
+            -0.016263794153928757,
+            0.011012911796569824,
+            0.015125328674912453,
+            -0.0040891822427511215,
+            0.01691434532403946,
+            0.014126266352832317,
+            -0.019098341464996338,
+            0.0019884423818439245,
+            -0.008193854242563248,
+            -0.00158184755127877,
+            -0.03029712475836277,
+            0.011043890379369259,
+            0.0074387495405972,
+            -0.0072218989953398705,
+            0.036616768687963486,
+            0.009858956560492516,
+            0.037577107548713684,
+            0.018680129200220108,
+            0.0008974128868430853,
+            0.009045767597854137,
+            -0.018370341509580612,
+            0.013065247796475887,
+            0.01500141341239214,
+            -0.012724482454359531,
+            -0.005734923295676708,
+            0.0013446672819554806,
+            0.0007551047019660473,
+            0.012631546705961227,
+            0.02004319056868553,
+            -0.01795213110744953,
+            0.009773765690624714,
+            -0.015032391995191574,
+            -0.005250881891697645,
+            0.006555858068168163,
+            -0.013932649977505207,
+            -0.008495896123349667,
+            -0.00963436160236597,
+            0.019408127292990685,
+            -0.012600568123161793,
+            0.016573579981923103,
+            -0.009595638141036034,
+            0.015520306304097176,
+            -0.029724018648266792,
+            -0.003669034456834197,
+            -0.006679772399365902,
+            -0.019593998789787292,
+            -0.012887120246887207,
+            0.009820234030485153,
+            0.010633422993123531,
+            -0.009301341138780117,
+            -0.011562783271074295,
+            0.020368466153740883,
+            -0.004185990896075964,
+            0.013529927469789982,
+            0.021189400926232338,
+            0.00737291993573308,
+            0.024132372811436653,
+            0.0005605200421996415,
+            0.014164989814162254,
+            0.005463860463351011,
+            0.010362359695136547,
+            -0.004677776712924242,
+            0.005161818582564592,
+            0.00932457484304905,
+            0.014203713275492191,
+            0.017766257748007774,
+            0.013196907006204128,
+            -0.00598275288939476,
+            0.007419387809932232,
+            0.02016710489988327,
+            0.0016254113288596272,
+            -0.030312614515423775,
+            -0.009719553403556347,
+            0.0001736498816171661,
+            0.00924712885171175,
+            0.009185170754790306,
+            3.817877222900279e-05,
+            -0.01093546487390995,
+            0.011144571006298065,
+            0.00969631876796484,
+            0.002286612056195736,
+            0.006280922330915928,
+            -0.01456771232187748,
+            -0.020957060158252716,
+            0.006106667220592499,
+            0.01798310875892639,
+            0.013917161151766777,
+            -0.014722606167197227,
+            0.022784801200032234,
+            0.006145390681922436,
+            -0.021406250074505806,
+            -0.01840132102370262,
+            -0.0009559819591231644,
+            0.0004542729293461889,
+            -0.002265314105898142,
+            0.03720536455512047,
+            -0.0011539548868313432,
+            -0.005955646280199289,
+            -0.0054754773154854774,
+            0.02775687538087368,
+            0.02049238048493862,
+            0.009177426807582378
+          ],
+          "embedding": [
+            0.0008215984936325879,
+            -0.019349418284688353,
+            -0.02322660633029795,
+            0.009747820845194852,
+            0.007353771220903641,
+            0.015252908783199839,
+            -0.012176175829980202,
+            0.051898051094966605,
+            -0.0068130353886146165,
+            0.008189212816890595,
+            0.008102287831931905,
+            0.01682801136011614,
+            0.02005557996049711,
+            -0.020779879580497183,
+            0.002332165268329781,
+            0.02337485065616377,
+            0.006074992895540845,
+            0.02576575146535689,
+            0.0007517286429767504,
+            -0.03763909942314134,
+            0.0011204729720314833,
+            0.00041212613589632806,
+            -0.06089176800314795,
+            0.012852035873395388,
+            -0.008901127092058543,
+            0.01702347055484216,
+            0.005803446395466281,
+            0.017889570275475296,
+            -0.05375047382413552,
+            0.021623778774252724,
+            0.014784317465883424,
+            -0.0018881880857885708,
+            0.0004735328554290247,
+            0.012393802850407656,
+            -0.019234442968055952,
+            -0.019566283611638122,
+            0.04407789373296535,
+            -0.0053565873209102995,
+            -0.026240866282480922,
+            0.048107902270426675,
+            -0.0054681314939387235,
+            -0.0009441397948041965,
+            -0.015205079626883548,
+            0.006885981841893229,
+            0.01403724256866746,
+            -0.026086548246792754,
+            0.0034778946051893067,
+            0.00044587533948775473,
+            0.032851328435891634,
+            -0.03882097333255334,
+            -0.011126344604254894,
+            -0.01648943336384904,
+            0.035810456555802166,
+            -0.019384589526165754,
+            -0.019430702768170285,
+            0.014297440272239852,
+            -0.007883453204663151,
+            0.050996900793086844,
+            0.06007149434153715,
+            0.03406832053483379,
+            0.025193439934671852,
+            0.00027785129663604897,
+            -0.008014391056168083,
+            -0.018806730908490114,
+            0.027193788903098558,
+            -0.006195006670328595,
+            -0.020466000344352826,
+            0.0068939884219589185,
+            0.017252276167086602,
+            -0.045073444648704136,
+            -0.049242269917908754,
+            0.023021140866008313,
+            -0.015676872472071106,
+            -0.024497517908898602,
+            0.005873633718973869,
+            -0.022828585938203683,
+            0.011748341004940436,
+            0.04624205727927451,
+            -0.015442165499674641,
+            0.018565551071491767,
+            0.015246505395507096,
+            0.004119269796224144,
+            0.004862735250819252,
+            -0.000145323321960046,
+            0.05155838952887077,
+            0.0027052423345323936,
+            -0.016218063670586664,
+            -0.07462470920944148,
+            0.043695974746295135,
+            -0.027608273720377708,
+            -0.03860912774695494,
+            0.002823110565150435,
+            0.026471469849508766,
+            -0.0026063577189074103,
+            -0.029230652014151735,
+            0.05684164470477843,
+            0.00022182868300572677,
+            0.015590845552956364,
+            -0.024939617549529242,
+            0.01035803286203917,
+            -0.010225298012384327,
+            0.019571847648759157,
+            0.01841613041441054,
+            0.0156981470538354,
+            0.021764297694793343,
+            -0.018831230007033416,
+            -0.014419529638789993,
+            -0.0029965475899652273,
+            0.028881185012095884,
+            0.09163526460163937,
+            0.022644302207811818,
+            -0.017933909936330133,
+            -0.04663928709016869,
+            0.004962951444105128,
+            -0.041333041174526026,
+            -0.027545628007617527,
+            -0.037142394579894954,
+            -0.013358143162788402,
+            0.02719686432396677,
+            -0.0034971800937504205,
+            0.04413635817371732,
+            0.04903857110328917,
+            0.03186067155630788,
+            0.007831051050269414,
+            -0.025257538623313574,
+            0.023517513773109414,
+            -0.0016527403209505964,
+            -0.0016101860707429347,
+            0.014448515185903624,
+            0.009905018312477684,
+            0.0024114582235163955,
+            0.00172043851384822,
+            -0.03818316221102173,
+            -0.02304537008110578,
+            0.01024969233570089,
+            -0.01901116356327339,
+            0.019791121460848986,
+            0.07985067340821365,
+            -0.051903191272236665,
+            -0.025924646976556942,
+            0.014766511532849252,
+            -0.002073086076894999,
+            0.030959269926800852,
+            0.006394417986617089,
+            -0.028177586693713442,
+            0.010725285336674526,
+            0.0009614052276924192,
+            0.07500253788978564,
+            -0.00036794182149960157,
+            0.004058458332826519,
+            -0.013218269166951234,
+            0.027613539268683245,
+            -0.012995170442480582,
+            0.023388707018488407,
+            0.026389423313105165,
+            -0.003068639956061266,
+            -0.02661967649632026,
+            0.026145274570635913,
+            -0.03300591322174331,
+            -0.004556072066733024,
+            0.020045722084112896,
+            0.0033437343914784865,
+            -0.010224220905483085,
+            -0.0025025498684079703,
+            0.03460103245395426,
+            0.04514362357278504,
+            0.03797031988058962,
+            -0.004586662475576956,
+            -0.06351723722596923,
+            0.06037983484369874,
+            0.019415753124363902,
+            0.028850618647994476,
+            -0.0034810237045849006,
+            0.02957938154808314,
+            0.02642652523231673,
+            0.013187653724133275,
+            -0.007856320804109569,
+            -0.007065882659482447,
+            -0.016672784625197085,
+            0.029020161154209707,
+            -0.013243058921970134,
+            -0.014900932834241732,
+            0.006001769901896033,
+            0.008648456186944271,
+            0.03484763413464497,
+            0.012279368984306564,
+            0.02379643508687133,
+            0.04933044959399941,
+            -0.021047723935647383,
+            -0.007264823701706426,
+            0.0007058774795716792,
+            -0.010512071832195085,
+            0.027489671668749157,
+            0.009731157570361245,
+            -0.03195919640758579,
+            -0.0033626031972691915,
+            -0.04282419484651207,
+            0.01873977910485268,
+            -0.014398285316695653,
+            0.012854631447494207,
+            -0.0017372959978092432,
+            -0.037331867825735776,
+            0.001088209292238142,
+            -0.012345684323522151,
+            0.037096063326414734,
+            0.005050219690024652,
+            0.0008906051515353004,
+            -0.003697709449630797,
+            0.006548479142737107,
+            -0.010727663712299397,
+            -0.007486226350610755,
+            -0.02704618481069485,
+            -0.019752552016312926,
+            -0.014632581296303486,
+            -0.0037501138119724717,
+            0.014588411795703245,
+            -0.007119137071719376,
+            0.016440258608998535,
+            -0.03686645352111013,
+            -0.023546402448713415,
+            0.005628703432152666,
+            0.024988145751166143,
+            0.011971199595808573,
+            -0.010868778329190568,
+            -0.03270046433624425,
+            -0.021950559285357267,
+            -0.011889914022657738,
+            -0.005123971523701009,
+            0.02619713302280345,
+            -0.021990810305904974,
+            0.01165257474485436,
+            -0.009117901804315863,
+            -0.007492082755249399,
+            0.010725902070291232,
+            0.01862811996181951,
+            -0.026694281868322685,
+            -0.01623776415010199,
+            -0.009403997538684658,
+            -0.02207329027097839,
+            0.025939540497196416,
+            0.014178803271471762,
+            -0.001141821318145744,
+            -0.006281187017652108,
+            -0.035993365198554964,
+            0.0026192776111449633,
+            -0.015254976771874396,
+            -0.0116872453009606,
+            -0.015780016491993135,
+            0.021421328283124968,
+            0.02889432250124136,
+            -0.01871009416296692,
+            -0.026540472640514536,
+            0.015703370682110186,
+            -0.0002849235245164438,
+            -0.02594930889995505,
+            0.04491828744809679,
+            0.044063522197410734,
+            -0.037441864491547405,
+            -0.007005339187980476,
+            -0.030381794393931814,
+            0.06901083845539985,
+            -0.017604082400603037,
+            -0.04060591381445593,
+            0.06044850935418443,
+            -0.01756490171206357,
+            0.025539717592705295,
+            0.03452018789549939,
+            0.0016217577522523927,
+            0.00018825054264059277,
+            -0.002563606092068712,
+            0.04770277039047993,
+            -0.0032010143255574255,
+            -0.03425328601475849,
+            -0.02105909767072799,
+            -0.00840881022563103,
+            -0.028699447677549002,
+            -0.015925160861589006,
+            -0.0055750742281166,
+            0.017513987082479483,
+            0.010399718005952185,
+            0.02267201252273823,
+            0.01258511878601025,
+            0.0015081978484714903,
+            0.01874981572091931,
+            0.011158889421057756,
+            -0.002607810686437211,
+            -0.003732023411827874,
+            -0.007787189972165309,
+            -0.05524012485122728,
+            -0.025110689689279583,
+            -0.03946383236703947,
+            0.004443572529325475,
+            -0.018745077598477586,
+            0.005926160888274727,
+            0.039430331582190524,
+            -0.0049500324516682,
+            0.004352795787899792,
+            0.015566139991780173,
+            0.013148156356408315,
+            -0.015279560337042242,
+            -0.03316846614871165,
+            0.0001592209771257332,
+            0.018362411161743347,
+            -0.012706582418607377,
+            -0.03424477956681259,
+            -0.001991022841226589,
+            0.0033467286047492175,
+            0.03357019284647071,
+            -0.024359110796680374,
+            0.00359117029951049,
+            0.005061818038334802,
+            0.011359582931760315,
+            -0.026793992746244076,
+            -0.018170136079817746,
+            -0.017421603453826397,
+            0.03996608926471779,
+            0.02587013058343799,
+            0.015763410145648683,
+            -0.04512536449983358,
+            -0.02293473890636517,
+            0.048575083783480195,
+            -0.009589860756342604,
+            -0.06486262416944208,
+            0.03447666009581266,
+            0.016105654870253654,
+            0.003526582643722421,
+            -0.01354754623151814,
+            0.05401575248587799,
+            -0.03563190295559527,
+            -0.017365166993914202,
+            -0.0046288709698227735,
+            -0.033657212157212645,
+            -0.007686252059880836,
+            -0.001385273642139782,
+            0.048229062789530236,
+            -0.029083855744728694,
+            -0.030465559177859865,
+            0.0024368661249980416,
+            -0.024144031154493057,
+            0.021693278579520123,
+            -0.016405088055634124,
+            -0.018439935250344958,
+            0.006948603359995389,
+            -0.033943037395199695,
+            -0.003007796423804453,
+            -0.016783195426536633,
+            -0.02270589491206155,
+            0.020874438753828814,
+            0.016724666903417947,
+            0.027033184180344852,
+            0.0019847647852695785,
+            0.017749524526371133,
+            0.0204661893602285,
+            -0.005604929968008821,
+            0.028462234633072944,
+            -0.020790459496727066,
+            -0.04183816355941877,
+            0.03946487892663695,
+            0.01192861758684541,
+            -0.0018024841275380242,
+            0.01077082591611702,
+            -0.01475239020295694,
+            -0.03952356245715099,
+            0.03683950018557526,
+            -0.056767672022551764,
+            -0.007507600090017693,
+            0.009530544352246083,
+            -0.00788856051420074,
+            -0.011204694027562521,
+            -0.010587701466318598,
+            -0.028604940377315504,
+            0.023749537635431462,
+            -0.013672419168235691,
+            0.01953449229707597,
+            -0.011449939491554354,
+            -0.012238180529977605,
+            0.05795536247491174,
+            -0.021518581774137322,
+            -0.040672955795925836,
+            0.026066659238628543,
+            -0.014061106927896723,
+            0.03328357230532424,
+            0.03957107406164427,
+            0.030670947051333917,
+            -0.004278577327978564,
+            0.025885946540364826,
+            0.0319106101668716,
+            -0.003462160036198642,
+            -0.04686868330851868,
+            0.00555303933159587,
+            0.02156143971867743,
+            -0.03739529489075782,
+            -0.01025626126223168,
+            -0.02141970191890338,
+            0.022134910980548407,
+            0.017807484142010496,
+            -0.007232643256393455,
+            0.011318041192005274,
+            -0.010489012862455913,
+            -0.013003217163997554,
+            -0.00847279880392288,
+            0.02357274819910594,
+            -0.0016485066139744577,
+            -0.005873113634992993,
+            0.006424249429163587,
+            0.004008703814301658,
+            0.0054639858216457724,
+            0.010198079500408441,
+            0.017694720520798957,
+            -0.017087021465208255,
+            -0.0025558869389290554,
+            -0.016503579212015177,
+            -0.0420082404922411,
+            -0.003563173424359332,
+            0.022924161700566468,
+            0.012345933179990547,
+            -0.00729908668365632,
+            -0.004234683369642856,
+            -0.0022962545569187934,
+            -0.020666205489143392,
+            -0.014796149043465308,
+            0.024562653223873056,
+            0.030217494971086438,
+            0.007810961433007407,
+            0.007781263640449456,
+            -0.0030348559388780655,
+            0.007310876971216526,
+            0.016099591640649783,
+            0.02311929597780174,
+            -0.04160162038353416,
+            0.027152100845897007,
+            -0.00025396315502635146,
+            0.023717123370023598,
+            -0.04186918759097667,
+            -0.017988531857836703,
+            -0.03106663560912813,
+            -0.023595079699181436,
+            0.02437134274097885,
+            -0.02317559303912261,
+            -0.012327263775257306,
+            0.000989566600374836,
+            0.00954921216572729,
+            -0.005155079395685017,
+            -0.017508404061379164,
+            -0.04582321461886958,
+            -0.0014367264231457263,
+            -0.030417229142190933,
+            -0.017829710557332023,
+            -0.001056472801554354,
+            0.005016912084754242,
+            -0.06276644565618253,
+            0.0109965681725148,
+            -0.07270669271977365,
+            -0.004542796125637244,
+            0.040139998232966566,
+            -0.017890441783630643,
+            0.055909102081208445,
+            -0.05029104308417623,
+            0.01673962668706085,
+            -0.04507293803055318,
+            0.015385506084493484,
+            0.025677372277539627,
+            0.030235171586316813,
+            -0.022189978375852563,
+            0.008461188938299745,
+            0.008033707483259569,
+            -0.045089505755429415,
+            -0.04144698743458133,
+            -0.013640256640978024,
+            0.01036782197518379,
+            0.0012696958402094232,
+            -0.016090728932978384,
+            0.023838437877040163,
+            0.011946482002767135,
+            0.03342604921859589,
+            0.04211801335157251,
+            0.002708015395637621,
+            -0.047156646610976634,
+            -0.013104464812562618,
+            -0.003933116891120681,
+            0.03155321708116329,
+            -0.018866109579446184,
+            0.015376812010630058,
+            -0.01955557376085726,
+            0.016255217383093355,
+            0.017868332102274922,
+            -0.020325179506828335,
+            0.013434364256590097,
+            0.004176904701252156,
+            -0.03877060944639854,
+            0.0006475542749369515,
+            0.00042468300294954125,
+            -0.02116992978038693,
+            0.0032654387271108293,
+            -0.00575003786159731,
+            -0.05149954071159492,
+            -0.031382926943128234,
+            -0.015730563640253065,
+            0.0382312502769352,
+            0.004297939848910064,
+            0.019724983545557365,
+            -0.017907869879607064,
+            -0.005165628493894463,
+            0.03698976707579373,
+            0.008501001373266799,
+            -0.01725527821298066,
+            0.006879620077191753,
+            0.02455792200929138,
+            -0.03224193972301503,
+            -0.039181328884413266,
+            -0.0008206986106407542,
+            -0.023306758893533952,
+            0.01672946960876128,
+            -0.02658587698869307,
+            0.005873620623736886,
+            0.0025331170979801232,
+            -0.03490809296591325,
+            0.02456764314036355,
+            0.003291113462353163,
+            -0.004303423672882112,
+            -0.020359544091059775,
+            -0.016892415165120203,
+            0.017174797487220006,
+            0.04391489511478065,
+            0.007327321684948843,
+            -0.003943713832544148,
+            0.00862684231313155,
+            -0.006361172785753631,
+            0.05826688726534012,
+            -0.02284956969636637,
+            -0.026680023999604643,
+            0.027527436347931256,
+            -0.022597669279069723,
+            -0.006277863878949772,
+            -0.017343446085471857,
+            -0.012065192152203348,
+            -0.0004183712490442143,
+            0.019339311388446943,
+            -0.019423562920897294,
+            -0.010212527447243049,
+            0.004783006556748757,
+            0.012824133870627983,
+            -0.004855945108275706,
+            0.006239949458562236,
+            0.006847581690471813,
+            0.03094005096378827,
+            0.026993131409761086,
+            -0.011331881651539968,
+            0.03297014291876675,
+            0.010265600643742144,
+            4.666113318251278e-05,
+            0.001970879281172325,
+            0.002523664335535017,
+            0.0034107811520368574,
+            -0.03143890864030425,
+            -0.01624584278940617,
+            0.013468729048885906,
+            0.005233309338223403,
+            -0.004550544967856448,
+            -0.023783653541634683,
+            -0.009402494239255264,
+            0.03334923712760375,
+            -0.003260424721959948,
+            0.02748395114281811,
+            -0.0058374897745283065,
+            -0.01985840930273332,
+            0.017595418271610163,
+            0.02881743246631328,
+            -0.05928552440605384,
+            0.017062540894682354,
+            -0.0011508547522925793,
+            0.008953761538519997,
+            0.01610561914042553,
+            -0.008894160513648953,
+            0.013301991014533684,
+            0.04080863883897929,
+            0.010738051773755976,
+            -0.023345002072886145,
+            0.004215726310838674,
+            0.01541749448570608,
+            0.01291813299324752,
+            -0.022747942296528025,
+            0.0008640791861357522,
+            0.011008522066277018,
+            0.005893793007012561,
+            0.0026106450193140156,
+            -0.006278394080583205,
+            -0.025659188287146268,
+            0.024180550333596625,
+            -0.0044491292740074925,
+            0.026967973346698617,
+            0.04390909273429254,
+            0.009596855239692042,
+            -0.03686020608700941,
+            -0.013109951664921729,
+            0.0002799680359670127,
+            -0.0038537577602593366,
+            -0.0037552094562213916,
+            0.011105295862811114,
+            -0.04185615640149602,
+            0.019950890919369135,
+            0.006453241355231516,
+            0.015314645427201533,
+            -0.020031193483734364,
+            -0.024530173488182337,
+            0.0031949828520724337,
+            -0.01492645564652531,
+            0.012875827757040941,
+            0.005057170413681063,
+            -0.004319834710426384,
+            0.005386862713494232,
+            0.00030848054013413157,
+            0.008384941028326983,
+            -0.010811526993301901,
+            0.03175068297908541,
+            0.004680987487229685,
+            0.009264547472281849,
+            -0.02856314241700808,
+            -0.021083721348429348,
+            0.020113396771629096,
+            0.03572366771051171,
+            -0.01708529918264109,
+            0.03346921724975398,
+            -0.009042046660366718,
+            0.0033219155387035227,
+            0.0151720168318272,
+            -0.015805318210244414,
+            -0.008057236707694303,
+            -0.0005483009439015931,
+            -0.002917527004423299,
+            -0.014538511124345527,
+            0.007302979537905532,
+            -0.0208529958115422,
+            -0.02965227242147972,
+            0.020064178921373875,
+            -0.00011892622932349096,
+            0.016455188521316146,
+            0.010335044657747514,
+            -0.007816155992852236,
+            -0.0052262330850905165,
+            0.006959344748883874,
+            0.04432147882994085,
+            0.026734484553994495,
+            0.05547661378166294,
+            -0.005787997952888261,
+            -0.004559444854138438,
+            -0.0082295581123363,
+            -0.006588172755707144,
+            0.04524009609473585,
+            0.015915003261638767,
+            -0.004910285095602864,
+            0.017495607570191165,
+            0.01557886571266685,
+            0.03858259260136037,
+            0.0027648525902558316,
+            -0.016912796336835408,
+            0.04014755669075221,
+            -0.004077405782937414,
+            0.008616827127381418,
+            0.011102692199778469,
+            0.0024185742218798562,
+            0.00807141794538637,
+            0.02996274720815659,
+            -0.012429186019907761,
+            0.01680942269506771,
+            -0.04571916592313758,
+            -0.02775530246832996,
+            0.014370280587419414,
+            -0.004931905540824582,
+            0.03419959665162407,
+            0.01668521021731861,
+            -0.023105305446763272,
+            -0.024761385385514877,
+            0.021529428247380984,
+            -0.002603042335561961,
+            -0.024135326707477325,
+            -0.015906833215844473,
+            0.003286605488908195,
+            -0.0023437997044735617,
+            0.0060251848371355166,
+            -0.013481652076194643,
+            0.0002908426939460218,
+            0.01953362396253743,
+            0.0038544889405965057,
+            0.020760460182873287,
+            -0.016224396121239458,
+            -0.00259233446682179,
+            -0.009282861237653619,
+            -0.02952130765234087,
+            0.017988370145301916,
+            0.006138236653154911,
+            -0.007607182955958238,
+            -0.009516767050028553,
+            0.0048347778733230365,
+            -0.005819209642901871,
+            -0.0025851701689263706,
+            -0.01073014664868326,
+            0.012433465684360061,
+            0.025784014942833097,
+            -0.016875624785552303,
+            0.007944349714123104,
+            8.651589872318479e-05,
+            -0.022306044741692753,
+            -0.022127016745066307,
+            0.020143472307319017,
+            0.002792646249762062,
+            0.008768813346304779,
+            -0.004074214276300631,
+            -0.00015024341785520542,
+            0.002990550758627699,
+            0.01951746553838439,
+            0.03896451381932467,
+            0.004232673242619471,
+            0.027876586543280867,
+            0.0145860303663199,
+            0.03994234200854897,
+            0.009657017763624678,
+            0.00696607158576712,
+            -0.026055387694656488,
+            0.02819890357011778,
+            0.00018297502510753014,
+            0.043190622821775236,
+            0.022488402221902314,
+            -0.0407730101161195,
+            -0.0021946518523759066,
+            0.01669520746276111,
+            0.013212623257794862,
+            0.017009844136059122,
+            -0.004537512994041156,
+            -0.014041213424740049,
+            0.028135404528975894,
+            -0.022859586430516003,
+            0.0008637006862481223,
+            -0.004486332934188017,
+            -0.025031795510001887,
+            0.007351370254175368,
+            -0.004567816916440959,
+            0.030025413528233364,
+            0.0014976740344892735,
+            -0.022285096307825804,
+            0.022837170656498543,
+            -0.009551269792616045,
+            0.011045530488684207,
+            0.008809047185288755,
+            0.020065001354321994,
+            0.031993362733068575,
+            0.009143210434415064,
+            0.02133640814803879,
+            0.012241777474889248,
+            0.014771538127546474,
+            -0.008804192497858033,
+            -0.02341427260922065,
+            0.023167424359504862,
+            -0.0011598649565273595,
+            -0.029627604902846688,
+            0.0021991474725787303,
+            0.003261878079344105,
+            -0.0005935713331682358,
+            -0.009144689470851402,
+            -0.005580557862988033,
+            -0.007240836451875213,
+            0.019631377011298905,
+            -0.007085213886352692,
+            0.007017573363709754,
+            -0.004074171791476129,
+            -0.00822871835665378,
+            -0.0421101573155962,
+            0.004210914075286222,
+            0.003408427700622687,
+            -0.01752179330665445,
+            0.006384440404174512,
+            0.011121481025573962,
+            0.043020575455262985,
+            -0.022809979913698254,
+            -0.0005737138409049003,
+            0.019206623212867476,
+            0.0009562551889820241,
+            -0.023427467810802533,
+            -0.038931688376706186,
+            0.007172345672967499,
+            -0.03467050750542982,
+            0.023185301620612612,
+            -0.0007075596339842658,
+            0.02387977204565927,
+            -0.03043468414449851,
+            0.013909304572370437,
+            0.004784668158421625,
+            0.012524879174804915,
+            0.021654512058639427,
+            0.03142146922870715,
+            -0.02053473475540882,
+            -0.050409862638621725,
+            -0.005406667316025549,
+            0.00035138762584085976,
+            0.022018487016014856,
+            -0.026014580970974616,
+            0.00025633004699302783,
+            -0.003349094027291921,
+            -0.02401366769212778,
+            -0.01735996745614562,
+            0.01962340791446025,
+            -0.050011480783403936,
+            0.015042913576492952,
+            0.0262737424555697,
+            -0.02507473901367252,
+            -0.05408889321034101,
+            -0.018988587082325706,
+            -0.028217893770659686,
+            -0.009285770094319318,
+            -0.00661159230987788,
+            0.028597405154740078,
+            0.003525576256054959,
+            -0.011462219358952466,
+            0.025401668390029348,
+            0.006155837251830549,
+            0.0026050693705924593,
+            0.015985722392086222,
+            0.007289910674212967,
+            -0.011452511453623235,
+            -0.014599577191293152,
+            -0.04558584556100621,
+            0.010197536827229795,
+            -0.0045431962637690155,
+            -0.019372598050508005,
+            -0.030147301333718016,
+            -0.0137828704002156,
+            -0.027060515748528666,
+            -0.0017291966950107505,
+            0.053783684195381654,
+            0.03292494290301388,
+            0.005560172296221419,
+            -0.0026085670565688026,
+            -0.010466890489356092,
+            0.036247483457995895,
+            -0.006789746401727701,
+            0.03662690472838588,
+            0.001784522073210422,
+            -0.0242673253380092,
+            0.01169382884548301,
+            0.02892627181542449,
+            0.009858451662166387,
+            -0.0005592283645355213,
+            -0.006160132931009508,
+            -0.008782830470079547,
+            0.0031207630972208394,
+            0.025613233596604418,
+            0.003268170392795548,
+            -0.010062991425127038,
+            0.018106967407336815,
+            0.004480291699976396,
+            0.02010374270557836,
+            0.03555357587371324,
+            0.023489837757983054,
+            -0.021782173652026497,
+            -0.026581762796336082,
+            -0.0013561486781077014,
+            0.0072266106510934505,
+            0.03551967295942857,
+            -0.01651315514034218,
+            0.0001274396319698452,
+            -0.008272735507120127,
+            -0.00894931703949637,
+            0.011149952898504355,
+            -0.0029671593121986116,
+            -0.0030752010285951576,
+            0.01662711184497578,
+            -0.0011672263615879709,
+            -0.033448222499147764,
+            0.0078288560210653,
+            -0.023496331346917084,
+            0.0051040606404415545,
+            -0.004113104164416575,
+            -0.007076525381595441,
+            -0.002157548492138289,
+            0.006302962169276006,
+            -0.008065841090139872,
+            -0.00035242603988955366,
+            -0.01586406637763306,
+            0.008271894439780969,
+            0.007438312999409211,
+            -0.0069617481460406095,
+            0.011402932583388205,
+            0.002666895909532698,
+            0.0005219881149353855,
+            -0.012159425670629751,
+            -0.012042145306711322,
+            -0.00957276588761916,
+            -0.03401863879157551,
+            -0.02604353712472079,
+            -0.024949533021906844,
+            -0.0014209149651479556,
+            -0.006566685925666972,
+            -0.03187796739029736,
+            -0.01209152137926458,
+            -0.011110763124407752,
+            0.007756961097537267,
+            -0.013260548545917543,
+            0.005433285079114047,
+            -6.399098520041351e-05,
+            -0.0044960488826674555,
+            0.016615636438440516,
+            -0.003643291599919672,
+            -0.019488010211707343,
+            -0.03927818878149417,
+            0.008209336177358258,
+            -0.02805593125673831,
+            -0.0038816771550190788,
+            0.007607119219721711,
+            -0.00528681677614909,
+            -0.029502897838743516,
+            -0.009149794737430706,
+            0.008767001199311985,
+            0.003507217008280066,
+            -0.01773349756494436,
+            0.03579670773605949,
+            0.026971617790626075,
+            0.011875179879209696,
+            -0.019891342676443626,
+            0.0508929813275565,
+            -0.0047010735337056275,
+            -0.012544935684845649,
+            0.018297503869683027,
+            0.02355022268131519,
+            0.03273167043015095,
+            -0.024830243320788693,
+            0.0009370814985214882,
+            -0.021937793653233454,
+            0.00535662615596573,
+            -0.035009915581788176,
+            0.0028595659314725012,
+            -0.014395027018699446,
+            0.015596459769560532,
+            0.03397372881171701,
+            -0.013254226072981457,
+            0.011195747181764677,
+            -0.016299384375589934,
+            0.00730244731485402,
+            0.00569944090461748,
+            -0.012916284280345044,
+            -0.005812311882399539,
+            -0.022902815152116493,
+            0.020140276669979015,
+            0.005809799950176385,
+            -0.0008130655714656685,
+            0.0104085826652653,
+            -0.044723432238900435,
+            0.03286429597337671,
+            0.008730225944501239,
+            0.017599189944921396,
+            0.0029762446259779998,
+            -0.0003068671951232415,
+            0.011611551606224214,
+            0.011228401175698422,
+            -0.016029727902740706,
+            0.004345913009102136,
+            -0.004613061137378433,
+            -0.03850533909474302,
+            0.009363351681220008,
+            -0.03631184057978666,
+            0.014427784677340159,
+            0.016026872002233575,
+            -0.004170582842643461,
+            -0.025166189571409366,
+            -0.005428471012333673,
+            0.010827910811085384,
+            -0.009783069664327864,
+            -0.009987805234276443,
+            -0.04163358183060102,
+            -0.025236440364963034,
+            0.034097508113604576,
+            -0.002119474613596975,
+            -0.025826403511707147,
+            -0.010830738112107807,
+            -0.011351132636491575,
+            0.020207687645133,
+            -0.006542026283946858,
+            -0.0006263975804936843,
+            -0.04150011545311013,
+            -0.0073396962917864515,
+            0.011192225907726928,
+            0.027746213794734322,
+            -0.035970017302005566,
+            0.02708126943501373,
+            0.002105633542220187,
+            0.014262674845755191,
+            -0.01626664983003942,
+            -0.005631805361903626,
+            -0.013990539604372344,
+            -0.035877093361375685,
+            0.016487723013906077,
+            -0.008607758248055054,
+            0.004714436224135091,
+            0.04458009880881818,
+            -0.0036872068506712727,
+            -0.009466125348499464,
+            0.007600705817195143,
+            -0.0034590883553048063,
+            -0.010351576085287273,
+            -0.0028564505793999295,
+            0.001959661298507,
+            -0.007080697863126256,
+            -0.04183476693971616,
+            -0.00499888976530217,
+            0.04684391782633148,
+            0.006361744381879857,
+            0.013276793765092465,
+            0.013516220521120684,
+            -0.018335746557932003,
+            -0.045577810515431215,
+            -0.0071616111758423025,
+            -0.0032763880176386808,
+            -0.022708028392758567,
+            -0.0240284945459861,
+            -0.008735040856657758,
+            0.005844145402902923,
+            0.006359368165686724,
+            -0.01120763599008222,
+            -0.022676131361171057,
+            -0.027085968700452136,
+            -0.010526103176253857,
+            -0.02123944716136792,
+            -0.010792145796733375,
+            -0.027142270190771887,
+            -0.028742234538796804,
+            0.02880426953386151,
+            0.007575182232859564,
+            -0.004084088301176117,
+            -0.010005765476121966,
+            -0.01255640546813973,
+            -0.0004774761617900028,
+            -0.0025307994917461232,
+            -0.005822065709999144,
+            0.0038614286313282164,
+            -0.011390482620811868,
+            0.023152086963976554,
+            0.0021220701260546895,
+            0.018329162172128138,
+            -0.05772324571231113,
+            -0.002709083851090071,
+            0.021497896395795905,
+            0.0033025410488410913,
+            -0.005412445141910314,
+            0.029352650459792132,
+            0.011223947499453582,
+            -0.006539460282439711,
+            -0.031355428613718234,
+            0.020942976176820262,
+            0.024122993515785947,
+            0.00817615634503138,
+            -0.012994160510030828,
+            0.01221095914384204,
+            0.01629778384790159,
+            -0.004494985018246821,
+            -0.011161098655072933,
+            0.005586857967039181,
+            0.020941039642372112,
+            0.013580622629911925,
+            0.0013398071867077221,
+            0.030905022925366163,
+            0.0006862084391508121,
+            -0.03521231612862609,
+            -0.01908191995763771,
+            -0.01275995164030872,
+            -0.003584331305986364,
+            -0.015264425022763126,
+            0.005294329654809353,
+            -0.002355465010876059,
+            0.006335052149475905,
+            0.022655379849785256,
+            0.03593231939948758,
+            0.0146937743203089,
+            0.00937330328448886,
+            0.003167462724350819,
+            -0.0013235176101076284,
+            0.012102872918129403,
+            0.008638011773075119,
+            0.0350701240268404,
+            -0.014653221697693864,
+            -0.011003994230902835,
+            -0.0037767288519369537,
+            0.010100131061186566,
+            -0.004113527518662366,
+            0.020414393256136372,
+            0.03274709615649469,
+            -0.0037222949335916876,
+            -0.005604552180246868,
+            -0.006160118231975786,
+            0.0041467454414452565,
+            0.01807879127455274,
+            -0.01252134965321004,
+            0.004364858987501461,
+            0.005549300665159658,
+            -0.0071897162833861,
+            0.018567059255813558,
+            0.0002622074720860951,
+            -0.012983550838417323,
+            0.03266376564402191,
+            0.11449509387240629,
+            0.0017251916835047628,
+            0.007125215166916139,
+            -0.040229997815548584,
+            -0.020537987527148888,
+            -0.00245728735683473,
+            0.005779972828836593,
+            0.007004732428236457,
+            0.02225914982152608,
+            0.006505827715591914,
+            0.0006708045351142873,
+            0.031708522931495536,
+            -0.017334083511709092,
+            -0.002522861743921599,
+            -0.01745656559688129,
+            0.013890451996979048,
+            -0.014312605700272691,
+            -0.013784963572164409,
+            -0.01572417853955075,
+            0.004507425871748977,
+            -0.000601158156253668,
+            0.0016158265715634729,
+            0.01791303234734911,
+            0.0011926917387973374,
+            -0.026393351252156246,
+            0.0006694008039871878,
+            0.001026031515214626,
+            -0.004514419527575117,
+            -0.011320929719992686,
+            -0.02223001723942815,
+            0.016235841442653918,
+            0.014188311619681008,
+            -0.004895218832118094,
+            0.009244697376301004,
+            0.010558796153959677,
+            0.02707485418319703,
+            -0.028380342401606758,
+            0.0040204430418629405,
+            0.007023041225985294,
+            0.01606556581447933,
+            0.017617862031190843,
+            0.0034870847111552862,
+            -0.011253315705053224,
+            0.03238289399766116,
+            -0.010563282687352038,
+            0.0065093536270233385,
+            0.014966802208644462,
+            -0.03541611540945151,
+            -5.821682910348165e-05,
+            0.0061985444938402745,
+            -0.011716201661363997,
+            0.009580035961315467,
+            -0.03120692534097433,
+            -0.0039291243987978845,
+            0.009317468331974374,
+            0.017611312558635673,
+            0.008804996982574765,
+            -0.0034129169537151925,
+            0.010822226698822738,
+            0.012510661458054741,
+            -0.0027147975086409966,
+            -0.008855213644027594,
+            0.01238502686635956,
+            -0.0007687053446954701,
+            0.00450108863711973,
+            -0.00799170686059026,
+            0.0029755595812552293,
+            0.002290650917244691,
+            -0.01554856300008412,
+            -0.011779430686394945,
+            0.00438393126679289,
+            0.0036931891075561957,
+            0.001535662168147833,
+            0.012014408636737167,
+            -0.006087255045485002,
+            -0.0077816220325470226,
+            0.0040573724156794284,
+            0.003136943055301706,
+            -5.505014316363865e-05,
+            -0.030625245024561508,
+            0.01585923084309149,
+            0.0067595435845396405,
+            0.010013308598626031,
+            -0.005315626260228517,
+            -0.00903142414091241,
+            -0.007240920015018893,
+            -0.01228521172101564,
+            -0.010657894223173453,
+            0.007658605592936815,
+            -0.00724924993432376,
+            0.011124174211132038,
+            -0.03128027787357322,
+            -0.013988715435985316,
+            0.01231792993245938,
+            -0.01528161242159377,
+            0.0007675821323979293,
+            -0.008389914312201402,
+            0.0009417242487395995,
+            -0.005119304379761705,
+            0.007887436211143328,
+            -0.03731503169547854,
+            0.007878001990271541,
+            0.02456170736765724,
+            -0.00330924625934981,
+            -0.020259771366167557,
+            -0.009613930457931786,
+            0.0241314385136084,
+            -0.008267831641886344,
+            -0.01645888287387047,
+            0.008421304862524506,
+            0.010140813149093993,
+            -0.0006309118552875393,
+            0.0008237784055549769,
+            -0.003726984191590917,
+            0.01669125388798323,
+            0.02054777616420997,
+            0.041930770964107515,
+            -0.014789610432004559,
+            0.01092349703703152,
+            0.02866938974792608,
+            -0.03378576980722847,
+            0.014728113846952417,
+            0.02518409645155824,
+            0.014590531327650454,
+            -0.0004446956315190408,
+            0.001609777186372437,
+            0.0068939321280085475,
+            -0.014273492115311677,
+            0.009247676157551494,
+            0.013380104440744367,
+            -0.018407746260686045,
+            0.020444430509441124,
+            0.03140071554008003,
+            0.0025839998101985736,
+            0.01355205268903058,
+            -0.006268939830271642,
+            -0.013860692186849113,
+            0.014717364068116991,
+            0.031429975488060896,
+            0.0006919565473572909,
+            -0.018561500446336806,
+            0.019128244642842268,
+            0.006988095806264863,
+            -0.003678529269242034,
+            -0.0021993078588142664,
+            -0.021334919445200388,
+            -0.021003586791369228,
+            0.017304350704792804,
+            0.013508779408333964,
+            0.027794485858413365,
+            -0.010594702782952029,
+            -0.007818089330662161,
+            0.014293688127901867,
+            -0.022368568887004094,
+            0.015426482139881204,
+            -0.02160820059598751,
+            -0.00571499813703566,
+            -0.0077402210272634235,
+            0.03274360648629609,
+            0.001441304806200403,
+            0.02734331677589297,
+            0.016457027764064002,
+            -0.033609420134045206,
+            -0.029790535553657092,
+            -0.00712106584238484,
+            -0.02307933297713104,
+            0.010356561171963577,
+            -0.03901183838032092,
+            -0.00071735150189382,
+            0.008251646656432826,
+            -0.008079917288471352,
+            -0.0007087526632966007,
+            -0.006419191399519291,
+            0.013989344185652506,
+            0.013661633160336152,
+            -0.0228103143826663,
+            0.023644639659577426,
+            0.008343481940133684,
+            0.025998483906975782,
+            -0.02076055376006461,
+            0.019148568238437927,
+            -0.017705363646416224,
+            -0.02777313933338929,
+            0.014358130220942483,
+            0.023585731219788467,
+            0.01635364038245971,
+            -0.007122853965024528,
+            0.007110003583832299,
+            -0.03556761320536796,
+            0.027585457849964405,
+            -0.013064062623866305,
+            -0.012504161289952458,
+            -0.0032379276562094555,
+            -0.011931827532360686,
+            -0.002574161005800279,
+            0.0008533907465694845,
+            0.0034345232246102806,
+            -0.03131462673429805,
+            -0.0002450959360520896,
+            -0.03727416416766574,
+            -0.028944814604815072,
+            0.01360126708805471,
+            -0.00036361284498363273,
+            -0.0049097813612814616,
+            0.02933242705521636,
+            -0.026339136251046315,
+            -0.02693295650849956,
+            -0.009590247544093352,
+            0.02853096456887074,
+            -0.0220673798697511,
+            -0.004519050393279848,
+            0.0015034014871316069,
+            -0.01828226786700525,
+            0.004436383319946364,
+            -0.0012748481042235446,
+            -0.010416065158876917,
+            -0.01105146139270597,
+            -0.03229479099927737,
+            -0.005822133182096844,
+            0.02106215163620738,
+            -0.006957125617688484,
+            0.002746241228040847,
+            0.013312248331456934,
+            -0.010968629118224582,
+            -0.0029946900704643812,
+            0.0017875284441794235,
+            0.023304175718480834,
+            -0.021033941064173063,
+            0.007714334541806199,
+            0.005955442031111047,
+            0.0016405586860131207,
+            0.018406562522853148,
+            -0.0054922197063461825,
+            0.0037690571959124617,
+            0.0162126534419016,
+            -0.009969304596173302,
+            0.017959307638726878,
+            0.007691549802475525,
+            0.006681837561745192,
+            0.002775700618222622,
+            0.007712770283579111,
+            0.009752652973700739,
+            -0.013506020126078112,
+            0.016911141425375933,
+            0.022468715726349706,
+            0.00023233555209372766,
+            0.021066580799163234,
+            -0.02302516292580905,
+            -0.028618544628453015,
+            0.018297878810036087,
+            -0.0225078941299981,
+            -0.012735208680311652,
+            0.016459925496937065,
+            -0.0043032724578332855,
+            -0.00797609922711709,
+            -0.014986057073351055,
+            0.02260618561608301,
+            0.02132930921464508,
+            -0.020249788476562527,
+            -0.012385936327068884,
+            0.0072304431443959505,
+            -0.004373475447051833,
+            0.007086656081865733,
+            0.03195810137269375,
+            -0.015982210897392764,
+            0.006818890093901413,
+            -0.0038306138017521626,
+            0.0030753642449792215,
+            0.017830546304667783,
+            0.009470018737705222,
+            -0.0019815241970236146,
+            0.030100238416495066,
+            -0.001330614013444443,
+            0.016392226282459177,
+            0.0004888028477871524,
+            -0.010948833692356176,
+            0.006343890495626752,
+            -0.026652806306173643,
+            -0.008627570055007363,
+            -0.001203530223345183,
+            -0.009581494451831351,
+            0.07598616930784373,
+            0.003930153552069478,
+            -0.01295957554290434,
+            -0.001673007115291578,
+            -0.001368287102181597,
+            -0.006897179139643672,
+            -0.00960679451603541,
+            0.0287166325268306,
+            -0.016613613331623346,
+            -0.01894527592131826,
+            0.0008370014132002703,
+            -0.006785618888421608,
+            0.00919924251249592,
+            0.01816363936999796,
+            -0.010225961928154075,
+            -0.01604759761423727,
+            0.001577138242530474,
+            -0.003520564936698765,
+            -0.004395434431595494,
+            0.0052399043934902014,
+            -0.010127122101423486,
+            0.006456662777583096,
+            0.026560491731327886,
+            -0.013684536319659296,
+            -0.01425196172663986,
+            0.004084982328707292,
+            -0.0010682084059972982,
+            0.009758641913073978,
+            -0.00864879062601467,
+            -0.00481222050200678,
+            -0.014375330954464725,
+            -0.011378890549403524,
+            0.02505641780042516,
+            -0.02861164504722546,
+            0.0034060847953066354,
+            0.014937227531980433,
+            -0.00016904224334342915,
+            0.01541381851648916,
+            0.0033028855218525644,
+            -0.03615935765718852,
+            -0.013677774380513626,
+            -0.02073839691250734,
+            0.019821074228011413,
+            0.01688422012576042,
+            -0.0033731840788147442,
+            -0.013840993586441808,
+            -0.026149880178874887,
+            -0.007430563876740068,
+            -0.02466234818567362,
+            -0.020708282040996014,
+            -0.019276991448222817,
+            0.006137002799644838,
+            0.005162892426268572,
+            0.022881759407670583,
+            0.0025854880108157857,
+            -0.015058889448404231,
+            0.0029948608544278386,
+            -0.006722836436935071,
+            0.011845298212977357,
+            -0.02747944486818378,
+            0.015311272959213262,
+            -0.01610493464983465,
+            0.025239768880017402,
+            0.007425208902253236,
+            0.0010569008485112994,
+            -0.008822681490988575,
+            0.00010704167529046774,
+            0.014883787023445807,
+            -0.004186968199973372,
+            -0.0007781259002896265,
+            0.02787555889813697,
+            0.02087373813833492,
+            -0.0017783425135865238,
+            0.008743718448136432,
+            0.028026471206552342,
+            0.019081083283793984,
+            0.0325261942213039,
+            -0.013815526479760766,
+            0.005113676604826083,
+            -0.028873600114081546,
+            0.010510475843707614,
+            -0.010871039729515013,
+            -0.010201824683340651,
+            -0.0067946599549165735,
+            0.027756887504389347,
+            -0.007962601205446472,
+            -0.0006687976124826922,
+            0.03469894503563926,
+            -0.014185294847328063,
+            -0.00010543064621374546,
+            0.0007386418834840544,
+            -0.009759481309152999,
+            0.02742722730386853,
+            -0.0073590367291510046,
+            0.018456032721630843,
+            -0.019560882612141943,
+            -0.013705917411254969,
+            -0.011894560208199602,
+            -0.001142768182039817,
+            -0.02576754877141608,
+            -0.008755240066826174,
+            -0.02249287216087102,
+            0.00859172262856532,
+            -0.0046012388553094375,
+            0.018845920253446127,
+            0.0011617844393195566,
+            -0.005275799428018491,
+            0.029069367036565145,
+            0.0088080893110627,
+            0.018100411389202663,
+            -0.0004971948011851863,
+            0.01216566778850985,
+            0.02369448391321832,
+            0.002637001729688794,
+            0.005012948614448268,
+            -0.024213704079796478,
+            0.003420184574622472,
+            -0.0004359900462023086,
+            8.265488264269535e-05,
+            0.016267948861008616,
+            -0.019572032453100303,
+            -0.02216264534756073,
+            0.021871880626992544,
+            -0.012742926070754964,
+            0.011676769226089007,
+            0.0035362870996336194,
+            -0.010732229460038356,
+            0.0006838777488770268,
+            -0.03083438636386314,
+            0.008820746379709355,
+            -0.03216685858285627,
+            -0.032230808828686675,
+            -0.013547681710236494,
+            -0.009994106571421729,
+            0.0008758088338659712,
+            0.028447187234357855,
+            -0.00795317029726568,
+            -0.015797892954136448,
+            0.005363565557248085,
+            -0.011199252697624111,
+            -0.00817162221789202,
+            -0.0007867892785855576,
+            0.005856939999700809,
+            0.009461959355310273,
+            0.005997804620690314,
+            0.011648335019189696,
+            0.007491946658046298,
+            -0.008028800679169293,
+            -0.008837725220736191,
+            0.015889923814605387,
+            -0.0014409716087425222,
+            -0.01176424391171685,
+            0.030430130082762363,
+            -0.013300620185884817,
+            -0.0005592926632214984,
+            -0.0021975669766032472,
+            -0.0015687208316663597,
+            -0.0021652076181411244,
+            -0.014225289143358926,
+            0.01870568404189425,
+            8.334501002782455e-05,
+            -0.014288806218067678,
+            0.0032841743905975634,
+            0.003841780428153241,
+            -0.0036498339483411074,
+            0.008388922960269383,
+            -0.008147275785908849,
+            0.017348147498892,
+            -0.015113395971380741,
+            0.008740452410210708,
+            0.0038335161236289234,
+            0.0043482078469644385,
+            -0.0065755348640655865,
+            -0.01592809184384243,
+            0.009642593996701513,
+            0.012051872087663658,
+            0.02173264012652684,
+            -0.02525660485859878,
+            0.00876161653901566,
+            -0.005374533920781607,
+            -0.014534179652013885,
+            -0.016127866637369738,
+            0.00982303248417706,
+            0.022322910435388785,
+            0.0063086346898091296,
+            0.012139263756190026,
+            0.00013383660542132047,
+            0.005848259030010402,
+            -0.014767198851314148,
+            0.010570158368602513,
+            -0.011355774405066537,
+            -0.018439261453084426,
+            -0.005243587266408389,
+            0.028894120979627323,
+            -0.005492024421080143,
+            0.017938747701972086,
+            0.0021350711250093858,
+            0.029007356490801176,
+            -0.008686781541806358,
+            0.0014883465513564527,
+            0.0039866022469891155,
+            -0.01295271525889559,
+            -0.004272008629260297,
+            0.01976273843646234,
+            0.04882868638601926,
+            0.010204475902027572,
+            -0.0009155109545661691,
+            0.010069071220988797,
+            0.005626418787776309,
+            0.002148101019776153,
+            -0.00805781810841286,
+            -0.030431071067211577,
+            0.008883707378543252,
+            0.005628218215212441,
+            -0.01997825125961566,
+            0.019838234890736068,
+            0.026483365072033395,
+            0.023040623182975964,
+            -0.011585643598171132,
+            0.016539930468533123,
+            -0.010092529089936188,
+            0.0177008735686489,
+            0.028665424114962996,
+            0.013810061902648515,
+            -0.010487150104822674,
+            -0.0020847870330548534,
+            0.006329342599564054,
+            0.004186981876004016,
+            0.00026239408223817384,
+            0.015190614519766204,
+            -0.019155180136775043,
+            0.01005728016778505,
+            0.014725826697964637,
+            0.03322436221319799,
+            8.049930133627564e-05,
+            -0.01645191835557271,
+            0.0016249281996128196,
+            -0.009976155662939693,
+            0.00856399492362911,
+            0.000632427127231252,
+            0.0032242260692672124,
+            -0.009688119824276678,
+            -0.012682779503915694,
+            -0.002851918183031152,
+            0.009403205508474241,
+            0.0021188716361101698,
+            0.002759004011231831,
+            0.007641925150177999,
+            0.0022830062031146204,
+            0.01713024474384216,
+            -0.0011181980841546123,
+            -0.00029141889484139217,
+            0.004601090256674874,
+            0.010509199590505558,
+            -0.01649852835819883,
+            -0.017548369964263514,
+            0.004337237475053179,
+            0.012217494210546133,
+            -0.013159179948224596,
+            -0.013241981632059755,
+            0.015704543608959117,
+            -0.01913575130892795,
+            0.024934102868065366,
+            -0.00942381892138331,
+            -0.004543611336526587,
+            0.000343200128934031,
+            -0.00727538880611703,
+            -0.0038751118497522673,
+            -0.0030064279504724797,
+            0.003984715814505561,
+            0.0019261818110367147,
+            0.007472350699291607,
+            0.008472369788801792,
+            0.014568547873264247,
+            0.006043895352236181,
+            0.0032580830408778395,
+            0.009109104498825864,
+            -0.014523846294121576,
+            -0.014800649887278973,
+            -0.002704791598011488,
+            -0.008895394038700809,
+            0.00902016867266151,
+            -0.006607182154874112,
+            0.00845408382182461,
+            -0.015501867835442603,
+            0.015573642225745032,
+            -0.012200552219016517,
+            -0.0008900215712478553,
+            -0.009666233545144905,
+            -2.7470277092642562e-05,
+            0.008191697904514857,
+            0.014605287653054787,
+            0.0070552942843382215,
+            -0.031921405225964206,
+            0.0017403427534633655,
+            -0.008846568352320488,
+            0.03446319360336241,
+            0.002612606662691094,
+            -0.01060863123202008,
+            0.020352181356568008,
+            -0.01592124132155158,
+            -0.00027704668715660693,
+            -0.01602075523652062,
+            -0.004797557413047212,
+            0.00707939446796679,
+            0.016399017978990946,
+            0.02759049512664933,
+            -0.005697020356865688,
+            -0.007995305872076221,
+            0.006384047872862793,
+            0.029590696670334232,
+            0.012246955508890425,
+            0.024748351124415785,
+            -0.007633712004672087,
+            0.014348228716100414,
+            0.007773260552660029,
+            0.0076045488653348975,
+            0.004321689250440021,
+            -0.002591801797145718,
+            -0.011698574606499644,
+            -0.014974971814360001,
+            -0.0037563183330623384,
+            -0.016987915346053498,
+            0.00012247833530362867,
+            0.021344400276501117,
+            -0.00475351423959752,
+            -0.0033847161875730026,
+            -0.01874186437694691,
+            0.0033461670218611346,
+            -0.010029445302549983,
+            -0.00367421246775698,
+            -0.013133469163793961,
+            -0.0254122035745965,
+            -0.018556369080030412,
+            -0.002155244024999317,
+            -0.00929309186646207,
+            -0.015194269781263309,
+            0.0070354690219415535,
+            -0.016305117309271634,
+            0.0080837166473439,
+            -0.00448473417024141,
+            -0.016508775226236883,
+            -0.00958925666995396,
+            0.01100335153658195,
+            -0.0041819993634961685,
+            -0.01929493648733792,
+            -0.01282300765645703,
+            -0.011300260564838231,
+            0.0051733704121938884,
+            0.005996780699975213,
+            -0.0038111648304178544,
+            -0.0028279639348479987,
+            0.016640121845366295,
+            -0.010197365811240557,
+            0.009878730312873492,
+            -0.03335474019383245,
+            -0.03551826040406959,
+            0.004497847402931096,
+            0.01507007013703286,
+            0.0013085881889541252,
+            -0.02078680101065954,
+            0.010806315988966003,
+            -0.016499186313125556,
+            -0.00858013792516617,
+            -0.008522534127498645,
+            0.009649521088411411,
+            -0.007283003380127847,
+            0.01299542385123007,
+            -5.3226845309105124e-05,
+            0.030477683597373687,
+            0.0074153455615142755,
+            0.004161881195438522,
+            -0.00679081048132963,
+            0.008354912600159931,
+            -0.001621776416108105,
+            0.004080191054849395,
+            0.02120524874226696,
+            0.012440848502230418,
+            -0.00218041171518331,
+            0.006458415098595872,
+            0.005678055538564355,
+            -0.017186537377121235,
+            0.02507960402044456,
+            0.024678178106806115,
+            -0.008205288425409454,
+            0.004108402466376321,
+            0.005783335302169668,
+            -0.008846663924040241,
+            -0.006897707231869481,
+            0.006527795262026734,
+            -0.018474181903007085,
+            -0.0008721381678008483,
+            0.010598084358464908,
+            -0.02255441909004843,
+            -0.004840511019512733,
+            0.021745651362109403,
+            0.004631567631335321,
+            -0.018558972627958834,
+            0.02764788406651861,
+            -0.004942361238163609,
+            -0.0027290999934532494,
+            -0.010876135202650048,
+            0.01671497143541799,
+            -0.012173527207375091,
+            0.008443317346655462,
+            0.011021541861818043,
+            -0.006295639322287843,
+            0.003083685676540999,
+            0.008277206401503182,
+            -0.004504302913715969,
+            -0.0068075424927125676,
+            -0.0010508416962215848,
+            0.010167160518175114,
+            0.0036980815941666336,
+            -0.022704963865525812,
+            -0.006020073261002279,
+            0.02934227992246887,
+            -0.015082922782242507,
+            -0.0050916398557337705,
+            0.005376294716056329,
+            0.007709342013963906,
+            -0.003476236561444464,
+            -0.0023399991233097337,
+            -0.00787486645740744,
+            0.010356414979428607,
+            -0.015950083845400274,
+            0.02076360898078935,
+            0.025608263494550438,
+            0.024504236990840842,
+            -0.014814235990238182,
+            0.010828584133720797,
+            -0.011703992394890898,
+            -0.011047470905668532,
+            -0.015542835222173864,
+            -0.001658296303013357,
+            0.001970208852236451,
+            0.013107754130616248,
+            0.009666887546609378,
+            0.0235853683657408,
+            0.006473544089109194,
+            0.008973300312205858,
+            0.014118601565410305,
+            -0.003902202854013708,
+            -0.018303438244719996,
+            -0.014584226585460755,
+            -0.011977613285028946,
+            -0.01756345491020528,
+            0.0004563602312336091,
+            0.020461659996271687,
+            0.012417744207683839,
+            0.024136626592375227,
+            0.002119213701333967,
+            0.013759902247467922,
+            -2.9525392347047554e-05,
+            -0.000892319792935443,
+            0.00045783825646242434,
+            -0.005238483120312843,
+            0.019255756091120846,
+            -0.004599752813817219,
+            0.01656589522110913,
+            0.0006443773300684925,
+            0.007133940816771615,
+            0.013554761340256566,
+            -0.01704388669741371,
+            -0.003443100892108368,
+            -0.010029620309829285,
+            0.000941391966877521,
+            -0.019633111750884034,
+            0.01934571074127927,
+            0.010273330977233852,
+            0.0017121345993533963,
+            -0.018327189396014198,
+            -0.0029706385099949693,
+            0.03448635131998772,
+            0.019458999453011265,
+            0.0050734354363614525,
+            0.02632443373898792,
+            0.013018762352749236,
+            0.014443611411502168,
+            0.010153370950346075,
+            0.02356766253713842,
+            -0.023479811300626968,
+            0.02095255383200973,
+            -0.011590347850340786,
+            -0.007414136601134069,
+            -0.0035774888741646705,
+            0.0025141508183687115,
+            -0.013588691733097813,
+            0.00982877622088977,
+            -0.0017916539399655826,
+            -0.025386211797192578,
+            0.009164316443206016,
+            -0.008811052979979271,
+            0.010464442616928296,
+            -0.007774569252594923,
+            -0.0002806136360286729,
+            -0.0127084026190163,
+            -0.015184317235990752,
+            0.001259201880504347,
+            -0.009158429132281081,
+            0.023908150915412617,
+            -0.021210848953875233,
+            -0.01321384014436847,
+            0.013997502713857512,
+            -0.006869443694669701,
+            -0.011364239609894453,
+            -0.00374449980963666,
+            0.023673050288879224,
+            -0.009243341446512421,
+            -0.01359367519009039,
+            0.014464166953530989,
+            0.003834735113400469,
+            0.0059713317014342395,
+            -0.007508862694272452,
+            -0.008428705474135382,
+            -0.003280788036976103,
+            0.005998840043285335,
+            0.026615077617284676,
+            -0.014059930972065399,
+            -0.0032833264512892022,
+            -0.009189999799498163,
+            0.004118204537632102,
+            -0.02359889635281631,
+            0.02771075351384059,
+            -0.008871395303200375,
+            0.021705177758560575,
+            -0.0015924449308442096,
+            -0.006239188158271628,
+            -0.024231036074349822,
+            0.027522945599928003,
+            0.003683870966132199,
+            -0.010221180899173243,
+            0.0012768725460662862,
+            -0.052005292736885915,
+            0.016636109796947025,
+            0.008429895913163985,
+            -0.019507994429834353,
+            0.003697519981610208,
+            -0.004773584856005853,
+            0.018983847580558853,
+            -0.022966993228850697,
+            -0.014078986007109447,
+            0.008283766047449966,
+            -0.01153285124362746,
+            0.0031439180465649996,
+            -0.02527749501821793,
+            -0.005754537200570427,
+            0.0011760345167229269,
+            -0.005276936074280004,
+            -0.001705933750221904,
+            -0.020022761608132548,
+            -0.025972227850795115,
+            0.017666017379157135,
+            0.015820875579600418,
+            -0.002278262448951789,
+            0.01320818011561616,
+            0.008709899932124187,
+            0.00836573261091119,
+            0.016034263394940008,
+            0.007124509234103267,
+            0.0081473711608784,
+            0.045498533239850446,
+            -0.019846323044145485,
+            0.007371629414298193,
+            -0.0014854149174929534,
+            0.013951741898629716,
+            -0.030467924817555014,
+            -0.0055144480998344705,
+            0.0062802744565489945,
+            -0.0049373689340857915,
+            0.03510454786147091,
+            -0.017326094216003073,
+            0.01694160526735264,
+            -0.0026075809870445238,
+            0.0007420394265014909,
+            -0.02042003877666893,
+            -0.026156937726134134,
+            -0.014294938857971834,
+            0.0011857406889443987,
+            -0.0010047941261608975,
+            -0.011946583993069242,
+            0.006797001710574735,
+            0.004917161312925866,
+            -0.0020713379451245515,
+            0.015652523103715316,
+            -0.02952061566240878,
+            0.007356800041369564,
+            -0.010733578620243498,
+            0.020008504830359524,
+            -0.00990331048284697,
+            -0.0013319400098308093,
+            0.007236960885181047,
+            -0.004122868383988748,
+            0.013216043472071927,
+            -0.0038337251451538677,
+            0.004227115653354611,
+            -0.005221321656766545,
+            -0.01794343569722877,
+            -0.0016435261495179767,
+            0.021429008432348098,
+            -0.008349684096540855,
+            0.031864497493526205,
+            -0.01264743878037828,
+            0.009174982784686633,
+            0.010692990337462762,
+            0.0006628831036334152,
+            -0.005286285851702483,
+            0.009052971814211658,
+            0.0021431970274545893,
+            -0.008690520448278274,
+            0.006934310986093342,
+            -0.018349449937310143,
+            0.008494451785507706,
+            0.03394632739506833,
+            0.0018511328780472266,
+            -0.0007519661670070526,
+            -0.007266663179613392,
+            -0.01335844201823745,
+            0.018540713029383066,
+            0.005488273906733518,
+            -0.002546531152114063,
+            -0.027536660869590543,
+            0.02007746621157705,
+            0.0012265798861291223,
+            -0.004755325069578531,
+            0.012466557901411086,
+            0.0020495867632654885,
+            0.019989531151046736,
+            -0.010171678639113316,
+            0.030757957899447096,
+            0.010050036220124393,
+            0.002959833051227505,
+            0.0038556928367700113,
+            -0.036655749074851536,
+            -0.01652152778069314,
+            -0.00019357502472740266,
+            0.014271581054865031,
+            -0.012430647252826173,
+            -0.022432580814844155,
+            0.0010346169306977334,
+            0.01148004051449691,
+            -0.0033898642762881046,
+            -0.009650665305271706,
+            0.01560757541276278,
+            -0.020342213902670575,
+            0.006701302394408393,
+            0.007693672045981921,
+            0.0287100214318104,
+            -0.0020735384554022153,
+            -0.0012324838849990056,
+            0.000626416803904792,
+            -0.0055669860801192935,
+            -0.0008037366126870824,
+            0.017251369734470945,
+            -0.013661316947002466,
+            0.0175926211217534,
+            -0.012735241490550407,
+            0.0064713656749838585,
+            -0.0021092378432037563,
+            -0.010629579860050812,
+            -0.020114807971202787,
+            0.0029609495764307357,
+            0.009378003163375899,
+            0.006920144698428806,
+            -0.010438984881449953,
+            0.010715409812532968,
+            0.012197673689706352,
+            -0.00753962861484439,
+            0.014519905132928041,
+            0.022173989895384192,
+            -0.007687503607783572,
+            0.013633321414009471,
+            -0.0041446334728745095,
+            -0.014056264816568229,
+            -0.009985463824440833,
+            0.023929750617160402,
+            -0.0005563806942667597,
+            0.008756997373618809,
+            0.00388125051641897,
+            0.0007856447663040696,
+            -0.011226530517316183,
+            -0.001653447552601681,
+            0.019294448360333633,
+            0.011831934675073843,
+            -0.0013888534822008192,
+            -0.0010525764356699885,
+            -0.0014566936525653407,
+            -0.009427557739483988,
+            0.02049188741459728,
+            -0.022030783927622997,
+            -0.0019604940727076573,
+            0.00461018148439991,
+            -0.022589816803875618,
+            0.0001863117949313752,
+            -0.024570217962989905,
+            -0.01569952241187498,
+            0.011763033781122343,
+            -0.0022530631823737554,
+            -0.016028810867854907,
+            -0.0003242701040942633,
+            7.424127301806556e-05,
+            -0.007815789940508527,
+            -0.004135500768181258,
+            0.006115909992703774,
+            -0.00034193217510010117,
+            -0.0032262189374874842,
+            0.013430154310506884,
+            -0.009996622196870008,
+            0.015431528911575157,
+            0.0067029980830614145,
+            -0.015702167721800483,
+            0.0021406610839557003,
+            0.018500219544264753,
+            -0.0058910087864916085,
+            0.00787021425357441,
+            -0.014681907249972176,
+            -0.0037495551894426303,
+            -0.005787661351944086,
+            0.006051351977821761,
+            0.024849195780885833,
+            0.012969434450643372,
+            0.01139340445077682,
+            0.017405361843075933,
+            0.004325000715054734,
+            -0.01572297405297993,
+            0.0007416798945675923,
+            0.01682727189828363,
+            0.010464036837245312,
+            0.009804326468113703,
+            0.0008975556667684365,
+            0.0043704860361487635,
+            -0.005142176468800175,
+            0.009317064072855551,
+            0.02915482449687158,
+            0.0013434689612997423,
+            0.008696141010760192,
+            0.017276730345147134,
+            0.027038741396131265,
+            -0.0035088997194497725,
+            -0.007469614319788499,
+            0.008739942310300076,
+            -0.009657991604173173,
+            -0.001081629174976152,
+            -0.017000351183542026,
+            -0.012310275736750868,
+            0.016510505259358398,
+            -0.006084627062230492,
+            0.02848913486719956,
+            -0.0010655806791907225,
+            -0.02026498848361482,
+            -0.01565505608194376,
+            0.001404549479440637,
+            -0.01362132895080652,
+            0.005287566256051299,
+            0.014187265685427415,
+            0.01917046513020831,
+            0.004687191682875024,
+            -0.022258337721085656,
+            0.037364962197399806,
+            0.01757108963349427,
+            0.019377870795760123,
+            -0.019422840506328382,
+            -0.006159311237412833,
+            -0.009077095160676276,
+            0.02116697012944498,
+            -0.00880692095488465,
+            -0.009875674856043954,
+            0.024014332004451508,
+            0.017812606845656484,
+            0.005679806286464216,
+            0.006736609904079586,
+            0.0020048663439319585,
+            0.0025414630253661444,
+            -0.015831593851012047,
+            0.017957902498542212,
+            -0.005858125698347523,
+            -0.0058068554681761945,
+            0.017529514573571298,
+            -0.005989581792993925,
+            -0.0030306021203732347,
+            0.00015839997806822045,
+            -0.019697052145531482,
+            0.011618112886207201,
+            0.008256007601576028,
+            -0.017229774429089857,
+            -0.0014757560402567304,
+            -0.0002404480326945295,
+            -0.014118694740134205,
+            0.02004885146995733,
+            0.01694893356262835,
+            -0.0013248884139405398,
+            -0.0051752815372013315,
+            0.008223868147393114,
+            0.007000679132597393,
+            0.005256194890343006,
+            -0.019496098876977206,
+            0.005230597149342019,
+            0.005103909294824105,
+            -0.0014716649591412774,
+            0.010274346323051634,
+            -0.009030594825985526,
+            0.033740380586893826,
+            -0.006505788197604522,
+            0.013065658941136721,
+            -0.0043634731684193745,
+            -0.0055529072007985125,
+            0.001816020676232658,
+            0.013054468478598695,
+            0.009753501030700457,
+            -0.00970446155762939,
+            -0.004172354962129865,
+            0.018697982092775737,
+            -0.011983156146516074,
+            0.0013300791598046338,
+            0.01564072865804035,
+            -0.004858814041438713,
+            0.004708235083775013,
+            0.008635204790345423,
+            -0.004489505237783172,
+            0.01801512858970922,
+            -0.016482572999490366,
+            0.0023953436660054026,
+            0.010694869684962088,
+            0.02390864159830096,
+            -0.019152187046410703,
+            0.021786596486708207,
+            -0.017951050280323393,
+            0.026538582634436314,
+            0.014484894489694262,
+            -0.0035222259457311444,
+            -0.01946200885147006,
+            -0.00850690913394181,
+            0.011654565129063417,
+            -0.021423803876241532,
+            -0.0007985948078290641,
+            -0.0007290222944686444,
+            -0.021367628543092133,
+            -0.011189364017231132,
+            -0.009961779129778331,
+            0.010701637745810138,
+            0.001918633898887013,
+            0.010380983868185359,
+            0.008002182827196827,
+            0.010784760574518087,
+            0.014405640442760822,
+            0.0008958107769384412,
+            -0.010743001445440941,
+            -0.023795589003655273,
+            0.016987708698679597,
+            0.00438463274382093,
+            0.0007862212695139549,
+            -0.01627468994515667,
+            0.011936651269737248,
+            0.013250643815768723,
+            -0.009385605548926286,
+            0.009121882469425302,
+            -0.03716901103204463,
+            -0.004552808410398574,
+            -0.0035704811925430366,
+            -0.0020309197365173475,
+            -0.00908678158053104,
+            0.01559445415882485,
+            -0.013682507836005851,
+            -0.0009590922904844632,
+            0.00011696985057536548,
+            -0.0074756526903548915,
+            -1.5467890432254378e-05,
+            0.020649928837031005,
+            0.008438785939908428,
+            -0.0037382105257999757,
+            0.010376213224654074,
+            0.008826031303751312,
+            -0.013900359449287676,
+            -0.01527474712786084,
+            0.019035366035884273,
+            -0.018776918681908793,
+            0.0063098418544296165,
+            0.01677916431403883,
+            0.020778813096283988,
+            -0.02891862501830075,
+            -0.0007231469961304435,
+            0.022093939332021185,
+            -0.0012807635799950273,
+            0.006370841109790851,
+            0.008781853138377092,
+            0.006754082980602544,
+            0.0029102159062355374,
+            -0.003711173531879644,
+            -4.843770941622283e-06,
+            -0.003940474588075707,
+            0.007155539022862352,
+            0.004366433868349658,
+            0.036453728071874844,
+            -0.013661129348165794,
+            0.002541308805735001,
+            -0.0058838191638835155,
+            -0.00347401304572066,
+            -0.00593469688967835,
+            0.004407665856950772,
+            -0.010055727994798024,
+            0.004330827357536078,
+            -0.001570861535288093,
+            -0.0003731129422346693,
+            0.004883969598300302,
+            -0.001701673379646408,
+            0.007036892564844683,
+            -0.011863650867472625,
+            -0.0003979727750011258,
+            -0.020174851412609312,
+            0.021117284557875267,
+            0.00011666551282792189,
+            0.003662462447326642,
+            0.03723741246290771,
+            0.00476064510662738,
+            -0.00238355131307445,
+            -0.012880057949435798,
+            -0.013551338766321525,
+            -0.012986444917747435,
+            -0.020703821629987552,
+            -0.0036665673548525388,
+            -0.021006311026247164,
+            -0.0052061406234528915,
+            0.010362862594785442,
+            -0.011540184420787572,
+            -0.004023659825622843,
+            -0.015881589631889473,
+            -0.006531981517815174,
+            -0.03114211633687585,
+            -0.01124607488650473,
+            0.007493228366452009,
+            -0.019149285949049886,
+            -0.0039932870926865805,
+            0.026850121590606017,
+            0.001095360635760988,
+            0.01387696424607248,
+            0.0038058248676879625,
+            0.008117686785954085,
+            -0.0045937472017383955,
+            -0.0005669025036141417,
+            -0.01928244877716143,
+            0.003391980330529026,
+            0.008525354854576925,
+            -0.00361507909530282,
+            0.0070008404454106905,
+            -0.008835727374865623,
+            -0.016950017602980382,
+            0.0021654956715219517,
+            0.009334379529385142,
+            -0.006639140003365139,
+            -0.017587472740268683,
+            -0.011049929670847685,
+            0.010401189523318744,
+            -0.02283429470064187,
+            0.006257145380266984,
+            0.016439581169238607,
+            -0.009363030375624585,
+            0.004651575425738153,
+            0.019809183749755425,
+            -0.011810674576781702,
+            -0.005811328501528397,
+            0.008471992070998505,
+            0.013891259999721588,
+            0.0095606134736748,
+            0.02954498207274139,
+            0.0026596809804008996,
+            0.000561870602466584,
+            0.004889263753490895,
+            -0.004277061142504552,
+            -0.017423200644235665,
+            -0.006243787897080996,
+            0.0038507494787521856,
+            0.012873996671952184,
+            -0.009879417960879032,
+            -0.00048575277472139055,
+            0.01682604801673206,
+            0.004530939990187663,
+            -0.0033901248079550284,
+            -0.011868399508738274,
+            -0.020080331108085103,
+            0.0015580259874809898,
+            -0.01362200460317948,
+            0.015354347040143093,
+            0.003390601069331352,
+            0.0015284610287850575,
+            -0.004239880336806429,
+            0.01828645373058079,
+            -0.005013418527935815,
+            0.012749833447770729,
+            -0.006148890994661865,
+            -0.004587831231448011,
+            0.007986533783001116,
+            0.009166578151006778,
+            0.0004939857995863664,
+            0.05421019830078428,
+            0.002838056604877719,
+            0.01333205648597435,
+            0.013419127695838135,
+            -0.014523733369484934,
+            0.014086525147734823,
+            0.012688569853552613,
+            0.018497344547821693,
+            0.013037819487160016,
+            -0.012353776077273462,
+            0.017354801119939964,
+            0.0024084723134891487,
+            0.003909681065813095,
+            0.0022202438064449653,
+            -0.004528576079628071,
+            0.019594683950225115,
+            0.004070222328547378,
+            -0.0074362443729372105,
+            0.004453618452623641,
+            0.005007543168224267,
+            -0.013105524222311019,
+            -0.0112958915514782,
+            0.027105208848863218,
+            -0.01219417557128091,
+            -0.018749971743148304,
+            0.006706594269933445,
+            -0.02008458683853808,
+            -0.03763686535076864,
+            -0.0021332074289244973,
+            0.009992361098878196,
+            0.002795183524495193,
+            -0.00253695604043486,
+            0.01614298064601032,
+            0.016730443047719683,
+            -0.016985296937992726,
+            -0.018245902728910347,
+            0.0025257780765481825,
+            0.0016333325293570182,
+            0.01861807236979066,
+            -0.023168663546030196,
+            -0.006378864850313035,
+            -1.3795465857380962e-05,
+            -0.006829008132963306,
+            0.004850306884033312,
+            0.004703280112191218,
+            -0.0071714759384722965,
+            -0.0027773519096287217,
+            0.002256846645341137,
+            0.01975383140755998,
+            0.013198754584066554,
+            0.01572971337890945,
+            0.002600741959577911,
+            -0.005033508062449482,
+            -0.016817359058724685,
+            -0.005450011991586895,
+            -0.013478330855299777,
+            -0.009091854345715177,
+            -0.003042052092498012,
+            0.008217376329142566,
+            -0.025053469059115735,
+            -0.011965083373220982,
+            0.0119294363235832,
+            0.00391377331459168,
+            0.0003099203772539202,
+            0.007607672728136279,
+            -0.003830132495364688,
+            0.010331053246451333,
+            0.013823890473448383,
+            0.0076112887952959766,
+            -0.004723377809559611,
+            0.009600411526173948,
+            0.011460781436250457,
+            0.006015477068316837,
+            -0.005072711638895421,
+            0.023859899048048694,
+            0.01821920335673539,
+            -0.01680365156728093,
+            -0.002149107383573353,
+            -0.0027829676356423997,
+            -0.01884308738652242,
+            0.009665830801645276,
+            0.00880322398336952,
+            0.008308268605449346,
+            0.003906369351868086,
+            0.00974531700889854,
+            -0.013875904819275925,
+            -0.0050570811397453235,
+            0.002870120865445989,
+            -0.005729556771404781,
+            -0.01901337075686543,
+            0.000297242372701423,
+            -0.011987192261104132,
+            -0.011327157550660895,
+            -0.01409710922165158,
+            -0.003221240464411861,
+            0.005283472511900512,
+            0.021472187640986972,
+            -0.004410881709548286,
+            -0.001855852749039012,
+            0.01351697599517034,
+            0.010060125186506398,
+            0.0021658738823899905,
+            0.0002206474541239167,
+            0.005418828324605768,
+            -0.0027919130944783397,
+            0.0008094042430809068,
+            -0.0015826797905407809,
+            0.021089276765434876,
+            -0.02016611086095107,
+            -0.009413977569213423,
+            -0.009288078098692112,
+            0.005842197094339927,
+            0.028793032462603636,
+            -0.007165634502373454,
+            -0.010104002083017078,
+            -0.01994315293556678,
+            0.0003786527922888834,
+            -0.07810609175343784,
+            -0.005210636756258548,
+            0.001870513618910238,
+            0.03126380113105983,
+            -0.010817888476436834,
+            0.0033673768058629823,
+            -0.002204487854177595,
+            0.014542299655712928,
+            -0.001760107965206874,
+            -0.0013283746982194259,
+            0.0071954806737858056,
+            -0.005640400818010697,
+            -0.006210989097047217,
+            0.005904163661619133,
+            -9.945941028848348e-05,
+            -0.010266938705916303,
+            0.00057263080955709,
+            0.02209498395848068,
+            0.0159398346608872,
+            -0.006575439759804175,
+            0.01816748552021211,
+            -0.00719294553484007,
+            -0.009838738592324234,
+            0.005639288780502127,
+            -0.0019007683459126327,
+            -0.015369619176147573,
+            -0.011475977270803784,
+            0.009862808084641291,
+            -0.030069890644150284,
+            -0.00010725753918890802,
+            0.00943698424646706,
+            -0.013540245857270492,
+            -0.008293042205008774,
+            0.007212974960781723,
+            0.006708343474532583,
+            -0.0012017968151025369,
+            -0.0078026000239507145,
+            -0.02527526731750743,
+            0.0027578400032925625,
+            -0.002446811971857853,
+            -0.004899702349784173,
+            0.0019960465173086183,
+            0.001479503476855694,
+            -0.012645905858466798,
+            0.01216748847452763,
+            -0.013378877368040743,
+            -0.012053251571418766,
+            0.012508842412556306,
+            -0.00027167241745418584,
+            0.01682680930386839,
+            -0.020607299598265426,
+            0.0010702151063620052,
+            -0.028075439824932304,
+            -0.007671848153932823,
+            -0.015486630894165054,
+            -0.0009776136330459065,
+            -0.0012939921569540803,
+            -0.03481880531471923,
+            0.012889447665794033,
+            -0.005200149829198305,
+            -0.006094702500154477,
+            -0.00459778303079732,
+            -0.024583504817899775,
+            0.006072281238752663,
+            0.00607350167867964,
+            -0.006506508970816635,
+            -0.016816439635445445,
+            0.008428048142588516,
+            -0.0011040593013238872,
+            0.004873670535200424,
+            -0.007475800310154908,
+            -0.0005638920132076605,
+            -0.013047135093134946,
+            -0.005463454269764494,
+            -0.0008522562324122834,
+            0.01827123304871433,
+            0.0044055851550317835,
+            -0.014526381222031981,
+            0.016862445236481526,
+            0.013250692156820124,
+            0.01602714293656341,
+            -0.0071089293427753115,
+            -0.0004265739315250755,
+            -0.02750630108787212,
+            -0.00327225092954903,
+            -0.023687644154808972,
+            0.00041213204365507744,
+            0.009447660230507965,
+            0.0203905264889617,
+            0.012694421477479161,
+            0.004620617778996077,
+            -0.001557511764988156,
+            0.0066751090724502,
+            0.02696959560423252,
+            -0.01099183126742224,
+            -0.006969111598423376,
+            -0.01884504192789845,
+            0.01113023717919988,
+            0.025135748541199255,
+            0.003381303139273845,
+            -0.011317957771935042,
+            0.008112598754655146,
+            -0.0009247845526290428,
+            -0.010302980598863016,
+            0.02889587382004902,
+            -0.0391134481322538,
+            -0.015599591108049343,
+            -0.004605824192962023,
+            0.007406520608110251,
+            -0.015596917115435033,
+            0.00770468362805995,
+            -0.012614339656105264,
+            0.001699049329817941,
+            -0.005851477164035241,
+            0.03535087340659861,
+            -0.0031950960727572925,
+            0.0006909856233673865,
+            -0.003395233050301427,
+            0.003531135044131077,
+            0.012355948349937309,
+            -0.004444221317023453,
+            -0.030726300006909638,
+            -0.0022983766869702346,
+            -0.011198252640110788,
+            0.009274666307515955,
+            -0.016207714291658314,
+            -0.017342435976846468,
+            -0.009424737744066259,
+            -0.021827243671837953,
+            -0.012403084982789827,
+            0.029747453318486338,
+            -0.005824001042844527,
+            0.01747569398142269,
+            -0.005219060548639168,
+            -0.008365003057800934,
+            0.00021272411948388247,
+            0.01344335658390732,
+            -0.028498240040284714,
+            0.00041963499785238845,
+            -0.002322931702641349,
+            0.0126393550661767,
+            -0.001595134597082556,
+            -0.00983073906977773,
+            0.006903313245694627,
+            0.0010522208685011098,
+            -0.015648940549713764,
+            -0.0007573245570947632,
+            -0.008533504878052644,
+            0.020475329445149498,
+            0.00027791387213409814,
+            0.001074213629646005,
+            0.013864331671732542,
+            -0.02102352993295736,
+            0.01576938071930521,
+            -0.021722775924283996,
+            0.01893267442161547,
+            -0.006577103101038035,
+            0.005772016664149475,
+            0.008283223343661555,
+            0.014068789332250016,
+            -0.014030042849491258,
+            -0.00663752723080222,
+            -0.034436220972634726,
+            -0.0122621879114314,
+            -0.001946072976826902,
+            -0.005308941654794638,
+            -0.014773123927909993,
+            -0.0011054822266082056,
+            -0.026205461509573584,
+            -0.006460373065662098,
+            0.012822188046808264,
+            0.019900312098359022,
+            -0.008409070094265829,
+            0.014163132434889265,
+            -0.006743711872055091,
+            -0.0028230771810882492,
+            -0.012652021749331811,
+            -0.018061180380009395,
+            -0.01083672324723357,
+            -0.01890966636827258,
+            -0.0018635408070944168,
+            0.004709222372611061,
+            -0.015760618199235518,
+            0.011435131737642627,
+            0.03198874888333405,
+            -0.01567271472650171,
+            0.008243406188694909,
+            0.016411496328333205,
+            0.0034236971031973503,
+            0.011728079139322837,
+            0.0038311415233052795,
+            0.004621789116267879,
+            -0.011109450058055078,
+            -0.011473158133621661,
+            0.006524490579223986,
+            0.00954511918454185,
+            -0.004913142601134428,
+            0.024347072255007807,
+            0.0008745419206472714,
+            0.005045794664611216,
+            0.005304506902446091,
+            0.011488187043525824,
+            -0.013135271908307747,
+            0.019746484188715552,
+            -0.002055210086078267,
+            -0.015571665869889405,
+            -0.01535499169234956,
+            -0.01105419264826655,
+            -0.018626399757822328,
+            0.0017108083455869296,
+            0.0181439079855921,
+            -0.0012493542718277685,
+            0.012137753522284387,
+            -0.00924460959150499,
+            -0.005019437549649743,
+            -0.0027156377206998474,
+            -0.038211954658821295,
+            -0.0021878970332906457,
+            -0.018809487178512628,
+            -0.02498945889484935,
+            0.03505216568400238,
+            -4.26477081213836e-05,
+            0.00997643255528116,
+            -0.010310197159232386,
+            -0.0002454902098986129,
+            -0.011065403358802515,
+            0.014954998647970473,
+            0.004624123403100598,
+            0.008872923678983478,
+            0.008560280683876702,
+            -0.016419500515801093,
+            -0.002997557798375716,
+            0.007472390947008284,
+            -0.00374822278027057,
+            0.012680407710694952,
+            0.015353684671120731,
+            0.020886474047003733,
+            0.011161262397502876,
+            0.002045862158732102,
+            0.00887271411363814,
+            0.009563361150297699,
+            -0.0003513237086717377,
+            -0.010459621030424415,
+            0.015419039131459582,
+            -0.02204848739950703,
+            0.012172257374410352,
+            0.04346634265794738,
+            0.019197433209794286,
+            0.029488039318675732,
+            -0.02272810439246353,
+            0.01448092574459596,
+            -0.003801043409441206,
+            0.009476229966122152,
+            0.0049309968883087685,
+            0.02346191840355209,
+            0.003973840862678076,
+            0.0073546443568171405,
+            0.010647581332452525,
+            -0.00972051412996416,
+            0.025644022498404723,
+            -0.00846669916801108,
+            5.1518540791212095e-05,
+            -0.004407265804145216,
+            -0.026400029105519137,
+            0.01615193008014211,
+            0.0081722536401398,
+            0.0010702314763197463,
+            0.0010263623479474385,
+            -0.0019566604229607643,
+            0.007982949510899019,
+            0.007079322685771456,
+            -0.015274148711633536,
+            -0.003886053737590683,
+            -0.018527455865296014,
+            0.0012864569064749438,
+            -0.024337133883771334,
+            0.010466651250597847,
+            -0.017568706814132484,
+            -0.014822354159709854,
+            -0.021666681078831483,
+            -0.013232016867264097,
+            -0.02011653224168991,
+            0.0035858355517491515,
+            -0.008632755352062642,
+            0.013025728891203198,
+            -0.016962939287139683,
+            -0.021494971929779273,
+            0.008430887589398292,
+            0.00870802359711456,
+            0.007481282677955534,
+            0.009728368603458082,
+            0.014996157358749115,
+            0.01736292711410622,
+            -0.022712386143092265,
+            -0.023163001611599246,
+            -0.006682209717364464,
+            -0.003531495447167166,
+            0.006762379833125297,
+            0.00933681196321128,
+            0.012884185500343249,
+            -0.007077895849501418,
+            -0.007554762054641417,
+            -0.013283067570021503,
+            0.006197966297581871,
+            -0.003993067576677973,
+            -0.009464789349849612,
+            -0.0015662739825834614,
+            -0.0061389795593180686,
+            0.002753664732282926,
+            -0.0001791088888074743,
+            0.0007595261237141144,
+            0.017122834676182343,
+            -0.008176319370751018,
+            -0.009787747261447871,
+            -0.02027228453943018,
+            -0.010683273300243025,
+            -0.015604745521156111,
+            -0.0016279406232527138,
+            -0.006506713191861036,
+            -0.014420323628589382,
+            0.010888206569219648,
+            -0.011930708571093279,
+            -0.00887210117616784,
+            0.010807147705360856,
+            0.0073900910661282935,
+            -0.003431679006289002,
+            0.037804020313161775,
+            0.0011945597874820288,
+            0.00778778394048267,
+            -0.003366543306746973,
+            0.005296727102519836,
+            -0.017761251541598876,
+            -0.026382400396890007,
+            0.0010357716464853195,
+            -0.01775777668381489,
+            0.010807713750049491,
+            -0.008515368407459745,
+            0.0016051389544388124,
+            -0.00034958600424912995,
+            -0.009585854918521076,
+            0.0036549720706351473,
+            -0.008110927907950148,
+            0.004212219431447508,
+            -0.03507890639477636,
+            0.011409377585617926,
+            -0.01819120811524444,
+            -0.008962242411035718,
+            0.01710177594046791,
+            0.037560936862121835,
+            0.0028513388998398104,
+            0.00030058353811717895,
+            0.0014792184333546447,
+            0.004427921639326168,
+            0.021438127785303025,
+            0.006339483221420334,
+            -0.001187747087595264,
+            -0.02237982599829809,
+            0.0062435650294483695,
+            0.002200578793697352,
+            0.010092828474934744,
+            0.023105023225115266,
+            -0.016797326021070008,
+            -0.015378520287395565,
+            -0.012732527281000097,
+            -0.010797463778182385,
+            0.009881499418260473,
+            0.02801767062785128,
+            0.01811139137849365,
+            -0.017358098300028903,
+            -0.020207173948076473,
+            0.0016242901745373126,
+            -0.026331540602868762,
+            0.007269751661479859,
+            0.11940631114242901,
+            -0.0045382135734049305,
+            -0.005994407567723939,
+            -0.008679712231646732,
+            -0.004482553208064535,
+            0.016709571294385293,
+            -0.013387043582971358,
+            0.069179848974036,
+            -0.007221545718220735,
+            0.0037392229364642915,
+            0.0028916224964057385,
+            0.00462729158956839,
+            -0.009727339449635306,
+            0.011570145487622916,
+            -0.011287365791681587,
+            -0.012991994493845187,
+            -0.013076194601468383,
+            -0.0180830261130804,
+            -0.007272768410218907,
+            -0.004155564814758654,
+            0.004204993170487392,
+            -0.005525575983182816,
+            0.008051108079520902,
+            -0.017551645696802648,
+            -0.010800070994485792,
+            0.013007176804761548,
+            -0.0023627485852619817,
+            0.005390986239687555,
+            -0.029668390278675554,
+            -0.0007650391259819867,
+            -0.0002446394046947245,
+            0.021280292985495608,
+            0.003137771886236958,
+            -0.0019077298815862946,
+            -0.010810159675564049,
+            0.007040107942856957,
+            0.023835993694726953,
+            0.021875366566457236,
+            0.0180511566130312,
+            0.012135351541044354,
+            0.020483975362807574,
+            -0.008667079096167644,
+            0.026324312888512866,
+            -0.001552207146726937,
+            -0.00397810465151493,
+            0.0077577136352257004,
+            -0.009849600842081875,
+            -0.00816392268697345,
+            0.00677353736462267,
+            -0.017288900282504148,
+            -0.020418071729964713,
+            -0.020383762873109595,
+            0.025741408097358233,
+            -0.011755094311186617,
+            0.002737129201159793,
+            -0.008266210469668866,
+            0.027287436144516414,
+            -0.006283892307996516,
+            -0.0029440642954701632,
+            0.009428876742171362,
+            -0.03788574758692254,
+            0.008834605607057885,
+            0.006463342392514337,
+            0.026711196829464153,
+            -3.9827332346130654e-05,
+            0.005565561841469247,
+            -0.0006738085601962865,
+            -0.006104320995853506,
+            -0.010792830169647851,
+            2.4565453208679464e-05,
+            0.017983908748172493,
+            0.0035181716362611077,
+            0.016970206528324653,
+            -0.005449808667944701,
+            -0.023171388778833197,
+            0.007092509502738752,
+            -0.01555320330471131,
+            -0.023306285898839978,
+            -0.002028783456704567,
+            -0.03196436161255752,
+            0.0033412618908711163,
+            0.007527633574188688,
+            -0.01890570959143866,
+            -0.017859735533232564,
+            0.010329340599904344,
+            0.004382396539605985,
+            -0.0016746397858953155,
+            0.006643785064338525,
+            -0.03307812909454471,
+            -0.01927475592253288,
+            0.004686970010943116,
+            0.012733525168279355,
+            -0.012993441126970916,
+            0.015291832952199082,
+            -0.020491167895268936,
+            0.016451976170508816,
+            -0.012259387189535237,
+            0.018393274368854132,
+            -0.006997668664226754,
+            0.011843450716200625,
+            0.011457415642860215,
+            0.009560877634897626,
+            -0.01790001115302499,
+            0.0214879415944361,
+            0.0066829070506954115,
+            0.03900547021123392,
+            0.02512484499576633,
+            -0.012651294056575184,
+            -0.0018425680226742782,
+            -0.00011361458320146007,
+            0.014982647266052349,
+            0.0061874424115559,
+            -0.003689464024837091,
+            0.02705902205734107,
+            0.018239469373522537,
+            -0.021070704644859155,
+            0.020037508334843183,
+            -0.00973804771680803,
+            0.018876283979591423,
+            -0.013303080480396818,
+            0.010592424494628194,
+            0.004717870260467274,
+            0.01070131238869639,
+            0.009087656015197565,
+            0.006708673669681232,
+            0.025928065153829043,
+            0.03663367449576964,
+            -0.00511073108390551,
+            0.002161381031392467,
+            0.004864092591944181,
+            0.012814518637357098,
+            0.001556833001747549,
+            0.007915766323215207,
+            -0.007012785514562176,
+            0.005834786334750152,
+            -0.002311533032619821,
+            -0.00122162928193159,
+            0.019525999280692766,
+            0.000824163362963678,
+            0.012875575199417254,
+            0.013259376059185821,
+            -0.002390133687709065,
+            -0.005682644446302278,
+            0.0019262218417782657,
+            -0.018193478045923098,
+            -0.015131696847666434,
+            0.029463723011600933,
+            -0.011447039651934148,
+            -0.008083114389964213,
+            0.0020642857121204753,
+            0.012160202510990566,
+            -0.00866944494225225,
+            -0.005378300635997713,
+            0.0038280364584418335,
+            0.014427318067525318,
+            -0.01799358399064332,
+            0.003381165206832161,
+            0.005252552821854107,
+            0.019433554824232185,
+            -0.00028791985674147885,
+            -0.013487678738391956,
+            0.0005317922815671783,
+            0.014982152801567637,
+            0.011902348165369078,
+            0.0016056390299304837,
+            -0.006935777197137747,
+            -0.00446584737370504,
+            0.004586814866182051,
+            0.006269466956110759,
+            0.0027516492952993445,
+            0.008157662045793032,
+            -0.0025001885993292637,
+            -0.016648747436203294,
+            0.017735757469839637,
+            -0.0030669873770852725,
+            0.018058248627079793,
+            -0.0038238312452176745,
+            -0.0147172470474118,
+            0.0043328417868095355,
+            -0.001888507803054499,
+            -0.018556943283494755,
+            0.009407930200720534,
+            0.002560949424273885,
+            0.010411159836451958,
+            0.0052571230238294825,
+            -0.007499470181994448,
+            -6.145889479758644e-05,
+            -0.015163819825329975,
+            0.022836682800304944,
+            0.016390944816242627,
+            0.001713572907204073,
+            -0.0035687505973373878,
+            -0.031210299824387475,
+            0.0029076511444359886,
+            -0.007822231980296341,
+            0.017821806420465356,
+            0.008250605126658402,
+            0.007908773724009905,
+            0.008351355849444162,
+            -0.00743970642051451,
+            -0.002411981861500092,
+            0.012543476434788293,
+            0.0202407777715137,
+            -0.0024635818448627447,
+            0.02535771413579989,
+            -0.012533449594824386,
+            -0.0022489292647974992,
+            0.0012362917358624392,
+            0.01578939075186126,
+            -0.0008001783766371555,
+            0.006311837812042848
+          ],
+          "keyphrases": [
+            "Switch Transformers",
+            "Mixture of Experts",
+            "sparsely-activated model",
+            "training instability",
+            "multilingual settings"
+          ],
+          "title": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "b8ffe9ad-9e6f-4478-8767-d2c5561f1266",
+        "properties": {
+          "page_content": "A Switch for Attention 27 ",
+          "keyphrases": [
+            "Switch",
+            "Attention"
+          ],
+          "embedding": [
+            -0.004847599659115076,
+            -0.013338678516447544,
+            -0.017561476677656174,
+            0.018164733424782753,
+            -0.036252863705158234,
+            0.0018600418698042631,
+            0.006209715269505978,
+            0.06599438190460205,
+            0.014593069441616535,
+            -0.005357495043426752,
+            -0.02489631250500679,
+            0.024474991485476494,
+            0.01628793403506279,
+            -0.022100266069173813,
+            -0.018451998010277748,
+            -0.02642839401960373,
+            -0.014775004237890244,
+            0.0039068059995770454,
+            -0.036712486296892166,
+            -0.008148754946887493,
+            0.020644789561629295,
+            -0.02066393941640854,
+            -0.034203704446554184,
+            0.016929492354393005,
+            -0.03311209753155708,
+            -0.00733004929497838,
+            0.03295888751745224,
+            0.011835324577987194,
+            -0.04140448570251465,
+            0.04393241927027702,
+            -0.03259501978754997,
+            0.010667112655937672,
+            -2.788491838146001e-05,
+            0.008067362941801548,
+            -0.04025542363524437,
+            -0.014545192010700703,
+            -0.0072869593277573586,
+            -0.023517441004514694,
+            0.001378872781060636,
+            0.02303866483271122,
+            -0.045158080756664276,
+            -0.006899151485413313,
+            -0.05542302504181862,
+            -0.02713698148727417,
+            0.010226639918982983,
+            -0.015732554718852043,
+            -0.02164064161479473,
+            -0.03544852137565613,
+            -0.0038780795875936747,
+            -0.018672235310077667,
+            0.024723954498767853,
+            -0.03631031513214111,
+            -0.0005529854097403586,
+            0.00697575556114316,
+            0.007416228763759136,
+            0.0014470982132479548,
+            -0.003313124878332019,
+            0.010542631149291992,
+            -0.011088434606790543,
+            0.037287015467882156,
+            0.003653055289760232,
+            -0.01817430928349495,
+            -0.0034160614013671875,
+            -0.024800557643175125,
+            -0.015694253146648407,
+            -0.011126737110316753,
+            -0.022119417786598206,
+            -0.0168050117790699,
+            -0.022866306826472282,
+            -0.0010132081806659698,
+            -0.042974866926670074,
+            0.012716270983219147,
+            -0.04653695598244667,
+            0.01056178193539381,
+            0.006458678282797337,
+            0.007713069207966328,
+            0.03776579350233078,
+            0.03698059916496277,
+            -0.05538472160696983,
+            -0.002126959152519703,
+            -0.010284092277288437,
+            -0.02424517832696438,
+            0.010657536797225475,
+            0.026409242302179337,
+            0.011777871288359165,
+            -0.02905208244919777,
+            -0.02650499902665615,
+            -0.09989167004823685,
+            0.05113319680094719,
+            0.0056447600945830345,
+            -0.02842009998857975,
+            -0.025164427235722542,
+            0.007488044910132885,
+            -0.006329408846795559,
+            0.01437283307313919,
+            -0.008871705271303654,
+            0.028688212856650352,
+            -0.014803730882704258,
+            -0.007205567788332701,
+            -0.0008031455217860639,
+            0.029109535738825798,
+            -0.0010479193879291415,
+            -0.025624051690101624,
+            -0.025164427235722542,
+            0.03203963860869408,
+            0.03052671067416668,
+            -0.02981812320649624,
+            0.024091970175504684,
+            0.0008827418787404895,
+            0.004270675126463175,
+            0.01378872711211443,
+            -0.01235240139067173,
+            -0.013941935263574123,
+            -0.0073731387965381145,
+            0.004337703809142113,
+            -0.002435769187286496,
+            0.002875045407563448,
+            -0.018384970724582672,
+            -0.014075992628932,
+            -0.009834043681621552,
+            -0.0008546138415113091,
+            0.030737372115254402,
+            0.004285038448870182,
+            -0.012371552176773548,
+            -0.020568184554576874,
+            -0.005850633606314659,
+            0.012591789476573467,
+            0.01174914464354515,
+            -0.014028115198016167,
+            0.010102157481014729,
+            0.01855732873082161,
+            0.019601058214902878,
+            -0.0066980659030377865,
+            0.0014459012309089303,
+            0.023517441004514694,
+            0.002047961112111807,
+            0.025815561413764954,
+            0.03431861102581024,
+            -0.03956598788499832,
+            -0.0008270842372439802,
+            -0.028305193409323692,
+            0.0037583857774734497,
+            0.015598498284816742,
+            0.02265564538538456,
+            -0.013032262213528156,
+            0.024015367031097412,
+            0.014822881668806076,
+            0.009977675974369049,
+            -0.009541991166770458,
+            0.029473405331373215,
+            0.018643507733941078,
+            -0.0008181072189472616,
+            -0.03347596526145935,
+            -0.001977341715246439,
+            0.054235659539699554,
+            0.021104414016008377,
+            -0.0035405431408435106,
+            -0.00020632220548577607,
+            -0.01043730042874813,
+            0.010274517349898815,
+            -0.01779128797352314,
+            -0.006635825149714947,
+            -0.021295923739671707,
+            -0.004670452792197466,
+            0.00989149697124958,
+            -0.005845845676958561,
+            0.033514268696308136,
+            0.00400495482608676,
+            -0.031580016016960144,
+            0.05132470652461052,
+            0.029224442318081856,
+            0.025183578953146935,
+            0.02834349498152733,
+            0.015943216159939766,
+            0.0025758109986782074,
+            0.00035818375181406736,
+            0.05224395543336868,
+            0.013530189171433449,
+            0.003772749099880457,
+            0.014803730882704258,
+            -0.035295311361551285,
+            0.006875212769955397,
+            -0.0050558666698634624,
+            -0.025834713131189346,
+            0.008967460133135319,
+            0.0135972173884511,
+            0.02169809490442276,
+            0.008057788014411926,
+            -0.0009449826902709901,
+            0.004304189700633287,
+            -0.025585750117897987,
+            -0.03711465746164322,
+            0.019725540652871132,
+            -0.02087460085749626,
+            -0.01820303499698639,
+            -0.01625920832157135,
+            -0.020625637844204903,
+            -0.0026284761261194944,
+            0.04366430267691612,
+            0.00017774531443137676,
+            -0.009977675974369049,
+            0.026332639157772064,
+            0.018672235310077667,
+            0.012074711732566357,
+            -0.05971284955739975,
+            0.003966652788221836,
+            -0.009355268441140652,
+            -0.020204314962029457,
+            -0.0038541408721357584,
+            0.00015784621064085513,
+            0.03276737779378891,
+            -0.004438246600329876,
+            0.01959148421883583,
+            0.01880629174411297,
+            0.005544217303395271,
+            0.006554433144629002,
+            0.008124816231429577,
+            0.015761282294988632,
+            0.006875212769955397,
+            -0.0067076412960886955,
+            0.039834100753068924,
+            0.02196620963513851,
+            -0.006295894738286734,
+            0.025241032242774963,
+            -0.06572626531124115,
+            0.040025610476732254,
+            -0.03397389128804207,
+            0.03052671067416668,
+            0.0010760474251583219,
+            -0.027749814093112946,
+            -0.07373139262199402,
+            0.0029277107678353786,
+            0.005917662288993597,
+            0.03077567368745804,
+            -0.008775950409471989,
+            -0.004912233911454678,
+            -0.009790954180061817,
+            0.01677628420293331,
+            0.01882544346153736,
+            -0.013999388553202152,
+            0.01795407198369503,
+            0.012045985087752342,
+            -0.015349534340202808,
+            0.0011514545185491443,
+            0.004505275283008814,
+            -0.009259513579308987,
+            -0.020127711817622185,
+            0.027481699362397194,
+            -0.02309611812233925,
+            -0.012620516121387482,
+            -0.0157134048640728,
+            0.0023316354490816593,
+            -0.019026529043912888,
+            0.012180042453110218,
+            -0.0004691997601184994,
+            0.05978945270180702,
+            -0.011270369403064251,
+            -0.02016601338982582,
+            0.0089483093470335,
+            0.014468587934970856,
+            0.009939374402165413,
+            0.02522188052535057,
+            -0.005553793162107468,
+            0.039834100753068924,
+            0.010312818922102451,
+            -0.027673209086060524,
+            0.03742107376456261,
+            -0.05982775613665581,
+            0.006616673897951841,
+            -0.019409548491239548,
+            0.002752957632765174,
+            -0.006286319345235825,
+            0.013549339957535267,
+            0.01759977824985981,
+            0.014391983859241009,
+            0.020912902429699898,
+            0.0032676411792635918,
+            6.702853715978563e-05,
+            -0.014190898276865482,
+            -0.018136005848646164,
+            -0.010466027073562145,
+            0.06400267779827118,
+            0.033629175275564194,
+            -0.005525066517293453,
+            -0.0273284912109375,
+            0.00811524037271738,
+            0.03194388374686241,
+            0.03274822607636452,
+            -0.028822271153330803,
+            -0.0005173765239305794,
+            0.004268281627446413,
+            -0.018193459138274193,
+            0.0030162842012941837,
+            -0.044315438717603683,
+            0.03487398847937584,
+            -0.0241877268999815,
+            0.037018902599811554,
+            0.04795413091778755,
+            0.005022352561354637,
+            -0.019917050376534462,
+            -0.03000963293015957,
+            -0.014746277593076229,
+            -0.005209074821323156,
+            -0.03272907808423042,
+            -0.0379764549434185,
+            0.0254133902490139,
+            0.01751359924674034,
+            0.021544886752963066,
+            -0.06415588408708572,
+            0.014707976020872593,
+            0.05676359310746193,
+            0.014075992628932,
+            -0.014794155023992062,
+            -0.017283786088228226,
+            -0.010169186629354954,
+            -0.0022598193027079105,
+            0.018566904589533806,
+            0.010485177859663963,
+            -0.014535617083311081,
+            -0.03090973012149334,
+            -0.017140153795480728,
+            -0.004510062746703625,
+            -0.025758108124136925,
+            0.03680824115872383,
+            -0.015071844682097435,
+            -0.010418149642646313,
+            0.008507836610078812,
+            0.037918999791145325,
+            -0.033629175275564194,
+            0.012409854680299759,
+            -0.055959250777959824,
+            -0.02093205414712429,
+            -0.008479109965264797,
+            -0.01735081523656845,
+            0.0037129020784050226,
+            -0.0025087823159992695,
+            0.014085568487644196,
+            0.02476225607097149,
+            -0.01907440647482872,
+            0.006564009003341198,
+            -0.011500181630253792,
+            0.015646375715732574,
+            -0.007665191777050495,
+            -0.020778845995664597,
+            -0.05235886201262474,
+            0.0008205011254176497,
+            0.03027774766087532,
+            -0.01257263869047165,
+            0.015234628692269325,
+            -0.0014913849299773574,
+            0.0011592345545068383,
+            -0.02137252688407898,
+            0.007320473901927471,
+            -0.006564009003341198,
+            0.036003898829221725,
+            -0.018605206161737442,
+            -0.023019514977931976,
+            0.011902352795004845,
+            -0.011605512350797653,
+            -0.01891162246465683,
+            0.025011219084262848,
+            -0.03263332322239876,
+            0.01043730042874813,
+            0.005817119497805834,
+            -0.06281531602144241,
+            -0.05163112282752991,
+            0.0039762286469340324,
+            -0.009786166250705719,
+            -0.009872346185147762,
+            -0.004897871054708958,
+            -0.08097047358751297,
+            -0.009671260602772236,
+            -0.04385581240057945,
+            0.02552829682826996,
+            0.0168050117790699,
+            0.017341239377856255,
+            -0.0038876549806445837,
+            0.005750090815126896,
+            -0.03761258348822594,
+            0.02330677956342697,
+            -0.014842033386230469,
+            0.017427420243620872,
+            -0.019189313054084778,
+            -0.029971331357955933,
+            0.04063844308257103,
+            0.02700292505323887,
+            0.02533678710460663,
+            0.006592735182493925,
+            -0.043626002967357635,
+            0.028017928823828697,
+            -0.021391678601503372,
+            -0.005075017921626568,
+            -0.0064491028897464275,
+            0.0048859016969799995,
+            0.021927906200289726,
+            -0.008129604160785675,
+            0.008268448524177074,
+            -0.0007175644277594984,
+            -0.001700849155895412,
+            -0.05339301750063896,
+            -0.013520613312721252,
+            0.02661990374326706,
+            -0.005697425454854965,
+            -0.030392654240131378,
+            0.010801170021295547,
+            0.010063855908811092,
+            -0.06534324586391449,
+            -0.010944802314043045,
+            0.0004428671090863645,
+            0.013156743720173836,
+            0.024398386478424072,
+            -0.06534324586391449,
+            0.013338678516447544,
+            -0.030584163963794708,
+            0.010628811083734035,
+            0.045541103929281235,
+            0.006157049909234047,
+            0.0551932118833065,
+            0.004232373088598251,
+            0.0008839388028718531,
+            0.014545192010700703,
+            0.0316949225962162,
+            0.013980237767100334,
+            0.02951170690357685,
+            -0.014238775707781315,
+            0.0012543911579996347,
+            0.03596559911966324,
+            0.005199499428272247,
+            0.04301317036151886,
+            0.0017367572290822864,
+            0.00875679962337017,
+            -0.04370260611176491,
+            -0.03975749760866165,
+            -0.008866917341947556,
+            -0.0007863883511163294,
+            -0.001040737726725638,
+            -0.03994900733232498,
+            -0.0033107309136539698,
+            -0.03968089446425438,
+            -0.04423883557319641,
+            0.02323017455637455,
+            0.03994900733232498,
+            -0.02426433004438877,
+            0.01786789298057556,
+            -0.007732220459729433,
+            0.019763842225074768,
+            0.002664384199306369,
+            -0.013204621151089668,
+            0.006477829068899155,
+            0.0224066823720932,
+            -0.028190286830067635,
+            0.03617625683546066,
+            -0.0029588311444967985,
+            0.002010856056585908,
+            -0.007052359636873007,
+            0.004778177011758089,
+            0.01732208952307701,
+            0.026102827861905098,
+            -0.031139543280005455,
+            -0.025432541966438293,
+            0.0445452481508255,
+            0.03265247121453285,
+            0.03768918663263321,
+            -0.04940960556268692,
+            0.005448462441563606,
+            -0.005620821379125118,
+            -0.06055549532175064,
+            0.04075334966182709,
+            -9.201461944030598e-05,
+            0.014688825234770775,
+            -0.03060331381857395,
+            -0.022559890523552895,
+            -0.04703488200902939,
+            0.007018845062702894,
+            0.0016769103240221739,
+            -0.015483591705560684,
+            -0.015502742491662502,
+            -0.04504317790269852,
+            -0.027098679915070534,
+            -0.0336100235581398,
+            -0.0188541691750288,
+            -0.01513887383043766,
+            -0.0037224777042865753,
+            0.003921169321984053,
+            0.01097352895885706,
+            0.016709256917238235,
+            0.01247688289731741,
+            -0.008210995234549046,
+            -0.05186093598604202,
+            -0.01637411303818226,
+            0.022349229082465172,
+            -0.008622742258012295,
+            0.035161253064870834,
+            -0.006678914651274681,
+            -0.013022687286138535,
+            -0.053278110921382904,
+            -0.01699652150273323,
+            -0.04243863746523857,
+            0.02393876202404499,
+            0.05389094352722168,
+            0.00978137832134962,
+            -0.012936507351696491,
+            0.007511983625590801,
+            -0.0024537232238799334,
+            -0.019007377326488495,
+            -0.017475297674536705,
+            0.02566235326230526,
+            -0.017044398933649063,
+            -0.0073635634034872055,
+            -0.027290189638733864,
+            0.019763842225074768,
+            0.027347642928361893,
+            0.01561764907091856,
+            0.0010646764421835542,
+            0.02003195695579052,
+            0.004567516036331654,
+            -0.003028253559023142,
+            0.027826417237520218,
+            0.01593364030122757,
+            -0.0025303272996097803,
+            0.01721675880253315,
+            -0.012314099818468094,
+            0.005477189086377621,
+            0.042093921452760696,
+            -0.01270669512450695,
+            0.018815867602825165,
+            0.03678908944129944,
+            -0.00822535902261734,
+            0.027998777106404305,
+            -0.04132787883281708,
+            0.021295923739671707,
+            0.046115633100271225,
+            -0.02368979901075363,
+            0.009206848219037056,
+            -0.001639805268496275,
+            0.002424996579065919,
+            -0.014583494514226913,
+            0.002250243676826358,
+            -0.02188960462808609,
+            -0.0060325684025883675,
+            0.00822535902261734,
+            0.016307085752487183,
+            -0.02778811566531658,
+            -0.01008300669491291,
+            0.010456451214849949,
+            0.02932019717991352,
+            0.025777259841561317,
+            -0.025451691821217537,
+            0.01462179608643055,
+            -0.009527627378702164,
+            0.03301634266972542,
+            -0.016910342499613762,
+            0.03188643231987953,
+            0.004318553023040295,
+            -0.007095449138432741,
+            -0.02956916019320488,
+            -0.012783299200236797,
+            0.04653695598244667,
+            0.005893723573535681,
+            0.010580932721495628,
+            0.003533361479640007,
+            -0.01718803122639656,
+            0.021046960726380348,
+            0.02573895826935768,
+            -0.010963953100144863,
+            -0.004222797695547342,
+            -0.0013704941375181079,
+            0.033188700675964355,
+            0.03148426115512848,
+            -0.014401559717953205,
+            -0.017724260687828064,
+            0.0016350174555554986,
+            0.01283117663115263,
+            -0.028764817863702774,
+            -0.004581879358738661,
+            0.058333978056907654,
+            0.004543577320873737,
+            -0.02713698148727417,
+            -0.015004816465079784,
+            -0.007162477821111679,
+            -0.011299096047878265,
+            -0.023900460451841354,
+            0.029722368344664574,
+            -0.009800529107451439,
+            -0.02137252688407898,
+            -0.02183215133845806,
+            0.00822535902261734,
+            -0.026658207178115845,
+            -0.03129275143146515,
+            0.03014368936419487,
+            0.017810439690947533,
+            0.04014051705598831,
+            0.006338984239846468,
+            -0.015837885439395905,
+            -0.024666501209139824,
+            4.4511136366054416e-05,
+            0.01978299394249916,
+            -0.01490906160324812,
+            -0.0013297982513904572,
+            -0.034012194722890854,
+            0.004964899271726608,
+            0.010983104817569256,
+            0.018068978562951088,
+            -0.01577085629105568,
+            0.0026715658605098724,
+            0.029281895607709885,
+            -0.013444009236991405,
+            0.012055560946464539,
+            -0.017695533111691475,
+            0.016623076051473618,
+            -0.01658477447926998,
+            -0.005161197390407324,
+            0.033629175275564194,
+            -0.03678908944129944,
+            0.008129604160785675,
+            0.003705720417201519,
+            0.009379207156598568,
+            0.025049520656466484,
+            -0.002621294464915991,
+            -0.005333556327968836,
+            0.024302631616592407,
+            -0.021334225311875343,
+            0.024934615939855576,
+            0.008234934881329536,
+            0.010188337415456772,
+            0.006640612613409758,
+            -0.017504023388028145,
+            -0.011710843071341515,
+            -0.02183215133845806,
+            -0.0018253306625410914,
+            0.01132782269269228,
+            -0.0047135422937572,
+            -0.008455171249806881,
+            -0.019285067915916443,
+            0.0004862561181653291,
+            0.009680835530161858,
+            0.04557940363883972,
+            -0.014870759099721909,
+            0.013846180401742458,
+            0.01295565813779831,
+            -0.010130884125828743,
+            0.007282171864062548,
+            0.03140765801072121,
+            0.03372493013739586,
+            0.0008701740298420191,
+            0.004385581240057945,
+            0.022310927510261536,
+            0.026773111894726753,
+            0.0032125820871442556,
+            0.0057213641703128815,
+            -0.007492832839488983,
+            0.00989149697124958,
+            0.0008456368232145905,
+            0.01951487921178341,
+            0.011959806084632874,
+            -0.03014368936419487,
+            -0.02497291751205921,
+            -0.011385275982320309,
+            -0.023823857307434082,
+            0.01948615349829197,
+            -0.031637467443943024,
+            -0.02060648798942566,
+            0.011844899505376816,
+            0.009455811232328415,
+            0.0021125958301126957,
+            -0.00465369550511241,
+            -0.02093205414712429,
+            0.03378238156437874,
+            -0.015129297971725464,
+            0.060708701610565186,
+            -0.009757439605891705,
+            -0.03713380917906761,
+            -0.023670649155974388,
+            -0.008163117803633213,
+            -0.017255060374736786,
+            -0.03768918663263321,
+            -0.020070258527994156,
+            0.022808853536844254,
+            -0.01091607566922903,
+            -0.017532749101519585,
+            -0.0008426444255746901,
+            -0.000540118315257132,
+            -0.011624663136899471,
+            0.0006493389373645186,
+            0.021908756345510483,
+            0.005046291276812553,
+            0.0008887265576049685,
+            0.008124816231429577,
+            -0.021851303055882454,
+            0.005754878744482994,
+            -0.004373611882328987,
+            0.02278970181941986,
+            -0.026390092447400093,
+            -0.0026308700907975435,
+            -0.029684066772460938,
+            -0.05986605957150459,
+            -0.030316049233078957,
+            -0.017618929967284203,
+            0.012926931492984295,
+            0.007468894124031067,
+            0.011509757488965988,
+            -0.025566598400473595,
+            -0.004541183356195688,
+            -0.011978956870734692,
+            0.011710843071341515,
+            0.022483285516500473,
+            -0.005496339872479439,
+            -0.020836299285292625,
+            -0.003054586239159107,
+            -0.022061964496970177,
+            0.006812972016632557,
+            0.006927878130227327,
+            -0.0273284912109375,
+            0.05484849214553833,
+            -0.0005957759567536414,
+            0.0325184166431427,
+            -0.013568490743637085,
+            -0.019581908360123634,
+            -0.040102217346429825,
+            -0.011069283820688725,
+            -0.007248657289892435,
+            0.022425832226872444,
+            -0.025126125663518906,
+            -0.009609019383788109,
+            -0.019342521205544472,
+            -0.008230146951973438,
+            0.023670649155974388,
+            0.04224712774157524,
+            0.0333419106900692,
+            -0.003293973859399557,
+            0.00793809350579977,
+            0.002160473261028528,
+            -0.016623076051473618,
+            -0.014363258145749569,
+            -0.014315379783511162,
+            -0.003641085932031274,
+            0.007224718574434519,
+            0.03537191450595856,
+            0.030124539509415627,
+            -0.04305147007107735,
+            0.018126431852579117,
+            0.03784239664673805,
+            0.010054280050098896,
+            0.014344106428325176,
+            0.019744692370295525,
+            0.011729993857443333,
+            0.02905208244919777,
+            0.015081420540809631,
+            -0.01306098885834217,
+            -0.005917662288993597,
+            -0.010025554336607456,
+            -0.03339936211705208,
+            0.0243217833340168,
+            -0.010446876287460327,
+            -0.010284092277288437,
+            0.016766710206866264,
+            0.019581908360123634,
+            -0.0005595685797743499,
+            0.019821295514702797,
+            0.0024776619393378496,
+            -0.032863132655620575,
+            -0.009230786934494972,
+            -0.0018791928887367249,
+            0.014775004237890244,
+            0.012926931492984295,
+            -0.015196327120065689,
+            0.025260182097554207,
+            -0.023440835997462273,
+            0.01603897102177143,
+            0.011902352795004845,
+            0.014822881668806076,
+            0.015426138415932655,
+            0.022061964496970177,
+            -0.008862130343914032,
+            -0.004617787431925535,
+            0.015090996399521828,
+            0.004258705768734217,
+            -0.006161837372928858,
+            -0.015081420540809631,
+            0.016565624624490738,
+            -0.007842338643968105,
+            -0.004950535949319601,
+            -0.02765405923128128,
+            -0.005285678897053003,
+            -0.010839471593499184,
+            -0.007291747257113457,
+            0.015512318350374699,
+            -0.005146834068000317,
+            0.0074401674792170525,
+            -0.0003764370339922607,
+            0.021736396476626396,
+            -0.00026108213933184743,
+            0.0004120459489058703,
+            0.016489019617438316,
+            0.050520364195108414,
+            -0.02079799771308899,
+            -0.01209386344999075,
+            -0.0022514406591653824,
+            -0.0008977036341093481,
+            -0.010169186629354954,
+            -0.004342491738498211,
+            -0.005199499428272247,
+            -0.006501767784357071,
+            0.01270669512450695,
+            0.02060648798942566,
+            -0.025394240394234657,
+            -0.010992679744958878,
+            -0.023191872984170914,
+            -0.0377083383500576,
+            0.010178761556744576,
+            -0.01490906160324812,
+            -0.04730299487709999,
+            0.03937447816133499,
+            -0.014152596704661846,
+            0.010734140872955322,
+            0.029205290600657463,
+            -0.0032197635155171156,
+            -0.04722639173269272,
+            -0.0060421437956392765,
+            0.005414948333054781,
+            -0.00098208780400455,
+            -0.0031216146890074015,
+            0.0401788204908371,
+            -0.017829591408371925,
+            0.003597995964810252,
+            -0.0185860563069582,
+            0.01997450366616249,
+            -0.01689119078218937,
+            0.036137957125902176,
+            0.03598474711179733,
+            -0.04680506885051727,
+            -0.0033059432171285152,
+            -0.03292058780789375,
+            0.020912902429699898,
+            0.007784885820001364,
+            0.011059708893299103,
+            -0.014765429310500622,
+            0.020434128120541573,
+            -0.023536590859293938,
+            0.028726516291499138,
+            0.011835324577987194,
+            0.033054642379283905,
+            -0.023440835997462273,
+            -0.01746572181582451,
+            -0.02713698148727417,
+            -0.003203006461262703,
+            0.01937124691903591,
+            -0.02361319586634636,
+            0.010130884125828743,
+            0.019083982333540916,
+            0.01414302084594965,
+            0.026964621618390083,
+            0.020702242851257324,
+            -0.0015284899855032563,
+            -0.015090996399521828,
+            -0.00834984052926302,
+            0.04511978104710579,
+            0.007382714655250311,
+            0.0004383785999380052,
+            -0.005467613227665424,
+            0.011318246833980083,
+            0.026524148881435394,
+            0.030833126977086067,
+            -0.015646375715732574,
+            0.01625920832157135,
+            -0.008167905732989311,
+            -0.011011830531060696,
+            -0.028783967718482018,
+            -0.0021628672257065773,
+            -0.03186728060245514,
+            -0.03893400356173515,
+            0.007957245223224163,
+            -0.03521870821714401,
+            -0.002626082394272089,
+            0.015359110198915005,
+            0.021870452910661697,
+            0.02464734949171543,
+            -0.028592457994818687,
+            0.028879722580313683,
+            0.019332945346832275,
+            -0.008354628458619118,
+            -0.001125121838413179,
+            -0.027175283059477806,
+            -0.003918775357306004,
+            0.0018947530770674348,
+            -0.014918637461960316,
+            -0.021276772022247314,
+            -0.007176841143518686,
+            -0.012084287591278553,
+            -0.0061187478713691235,
+            0.009446235373616219,
+            -0.007813611999154091,
+            0.002906165784224868,
+            -0.025375088676810265,
+            0.03349511697888374,
+            0.003466332796961069,
+            0.007861489430069923,
+            -0.01577085629105568,
+            0.0006451496738009155,
+            0.0021137928124517202,
+            -0.003650661325082183,
+            -0.009269089438021183,
+            0.03259501978754997,
+            -0.010398998856544495,
+            0.0054532503709197044,
+            -0.013233347795903683,
+            0.0019222826231271029,
+            0.01863393373787403,
+            0.016824161633849144,
+            -0.017590202391147614,
+            0.0043329158797860146,
+            -0.005060654599219561,
+            0.004687209613621235,
+            -0.005194711498916149,
+            0.027998777106404305,
+            -0.011931079439818859,
+            -9.934586705639958e-05,
+            -0.0038948366418480873,
+            -0.009316966868937016,
+            0.024111121892929077,
+            0.000726541446056217,
+            0.014899485744535923,
+            0.023459987714886665,
+            0.023000363260507584,
+            -0.013242923654615879,
+            -0.007023632992058992,
+            0.022674795240163803,
+            0.024226028472185135,
+            0.01209386344999075,
+            -0.009541991166770458,
+            0.002733806613832712,
+            0.004045650828629732,
+            0.029664915055036545,
+            0.0029444678220897913,
+            0.010178761556744576,
+            -0.02669650875031948,
+            -0.02688801847398281,
+            0.013846180401742458,
+            -0.012400278821587563,
+            0.0049888379871845245,
+            0.012132165022194386,
+            0.013568490743637085,
+            0.005003201309591532,
+            -0.001164620858617127,
+            0.009302603080868721,
+            0.012065136805176735,
+            0.006147474516183138,
+            0.02355574257671833,
+            0.012821601703763008,
+            -0.015119723044335842,
+            -0.006875212769955397,
+            0.002081475453451276,
+            0.00019390397937968373,
+            -0.008857342414557934,
+            -0.03096718341112137,
+            -0.06235568970441818,
+            -0.02183215133845806,
+            -0.014526041224598885,
+            0.004560334142297506,
+            0.02503037080168724,
+            0.025317635387182236,
+            -0.0049888379871845245,
+            -0.04653695598244667,
+            -0.0034783021546900272,
+            0.006980543024837971,
+            0.014727126806974411,
+            -0.004538789391517639,
+            0.02451329305768013,
+            -0.023574894294142723,
+            0.005223438143730164,
+            0.0072007798589766026,
+            -0.023862158879637718,
+            0.03904891014099121,
+            0.02368979901075363,
+            -0.01174914464354515,
+            -0.01735081523656845,
+            -0.018432848155498505,
+            0.012180042453110218,
+            -0.0034495757427066565,
+            0.022636493667960167,
+            0.03297803923487663,
+            -0.03979580104351044,
+            0.043357886373996735,
+            -0.006937453523278236,
+            -0.002997133182361722,
+            -0.01056178193539381,
+            -0.0028606820851564407,
+            0.0014662492321804166,
+            0.0029708005022257566,
+            -0.004320946522057056,
+            0.01779128797352314,
+            -0.018796715885400772,
+            0.023402534425258636,
+            0.008287599310278893,
+            0.022368380799889565,
+            -0.004325734451413155,
+            0.020146863535046577,
+            -0.07281213998794556,
+            0.007004482205957174,
+            -0.004548364784568548,
+            -0.010896924883127213,
+            -0.04967772215604782,
+            -0.02721358649432659,
+            0.06036398559808731,
+            -0.03437606245279312,
+            0.02175554819405079,
+            0.02688801847398281,
+            0.0251452773809433,
+            -0.032346054911613464,
+            0.00023968686582520604,
+            -0.003961865324527025,
+            0.024628199636936188,
+            -0.00918290950357914,
+            0.034471817314624786,
+            0.021161865442991257,
+            0.015129297971725464,
+            0.015004816465079784,
+            -0.002413027221336961,
+            -0.01574213057756424,
+            0.08441765606403351,
+            -0.011461880058050156,
+            -0.041902411729097366,
+            0.03000963293015957,
+            -0.0404469333589077,
+            0.00390441226772964,
+            0.01295565813779831,
+            -0.0061187478713691235,
+            0.014995240606367588,
+            -0.0024142242036759853,
+            -0.009585080668330193,
+            -0.021085262298583984,
+            0.0020240223966538906,
+            -0.03301634266972542,
+            -0.02252158895134926,
+            -0.004790146369487047,
+            0.003511816496029496,
+            0.0008617954445071518,
+            -0.0029875575564801693,
+            -0.004189283587038517,
+            0.006338984239846468,
+            0.04347279295325279,
+            -0.023364232853055,
+            0.011030982248485088,
+            -0.005654335953295231,
+            -0.026600753888487816,
+            0.033839836716651917,
+            0.013558914884924889,
+            0.005046291276812553,
+            -0.01497608982026577,
+            0.008359416387975216,
+            0.002378316130489111,
+            -0.031388506293296814,
+            0.00649698032066226,
+            -0.029741518199443817,
+            -0.004256311804056168,
+            0.005496339872479439,
+            -0.00047727907076478004,
+            -0.0066118864342570305,
+            0.030545860528945923,
+            -0.01645071804523468,
+            -0.021046960726380348,
+            -0.009800529107451439,
+            -0.004708754830062389,
+            -0.009503688663244247,
+            -0.007339624688029289,
+            -0.00037075160071253777,
+            0.0007103827665559947,
+            0.03320785239338875,
+            0.02661990374326706,
+            0.011174614541232586,
+            -0.010772443376481533,
+            -0.0252793338149786,
+            0.003966652788221836,
+            -0.0022693946957588196,
+            0.02164064161479473,
+            0.025202728807926178,
+            -0.0262368842959404,
+            -0.008546138182282448,
+            -0.036003898829221725,
+            -0.010226639918982983,
+            -0.009144607000052929,
+            0.011423577554523945,
+            -0.011346973478794098,
+            0.038819096982479095,
+            -0.010446876287460327,
+            0.013367405161261559,
+            -0.026390092447400093,
+            0.008531775325536728,
+            0.013453585095703602,
+            -0.003674600040540099,
+            -0.04557940363883972,
+            0.01490906160324812,
+            0.01587618701159954,
+            -0.0016505776438862085,
+            0.008689770475029945,
+            -0.02407282032072544,
+            -0.01533038355410099,
+            0.06955646723508835,
+            0.01037027221173048,
+            -0.01620175503194332,
+            -0.013252499513328075,
+            -0.024666501209139824,
+            -0.011902352795004845,
+            -0.013769576326012611,
+            0.01735081523656845,
+            -0.010858622379601002,
+            0.004232373088598251,
+            -0.007669979706406593,
+            0.024091970175504684,
+            0.007459318730980158,
+            0.0012986778747290373,
+            0.02650499902665615,
+            -0.003511816496029496,
+            -0.02495376579463482,
+            0.03916381672024727,
+            -0.017235908657312393,
+            -0.011366124264895916,
+            -0.019304217770695686,
+            -0.0021149897947907448,
+            -0.02125762216746807,
+            0.03571663424372673,
+            -0.021525735035538673,
+            -0.0011867642169818282,
+            -0.011797022074460983,
+            -0.019668087363243103,
+            -0.007574224378913641,
+            0.009939374402165413,
+            0.01904567889869213,
+            0.010351121425628662,
+            -0.0029253168031573296,
+            0.00827802438288927,
+            -0.008498260751366615,
+            0.031714074313640594,
+            0.0035668755881488323,
+            0.015148448757827282,
+            0.020395826548337936,
+            0.013175895437598228,
+            0.0045723035000264645,
+            0.018461573868989944,
+            -0.010762867517769337,
+            0.006812972016632557,
+            -0.026466695591807365,
+            -0.014382408931851387,
+            0.023919612169265747,
+            -0.013290801085531712,
+            -0.014334531500935555,
+            0.0010844259522855282,
+            -0.0039068059995770454,
+            0.0022454559803009033,
+            -0.012333250604569912,
+            -0.0054628257639706135,
+            -0.017006097361445427,
+            0.009834043681621552,
+            0.009776590391993523,
+            0.01863393373787403,
+            0.004569910001009703,
+            0.029645763337612152,
+            -0.007751371245831251,
+            -0.007780097890645266,
+            -0.0073635634034872055,
+            -0.02298121154308319,
+            0.005237801466137171,
+            0.021870452910661697,
+            0.0006702853715978563,
+            -0.005596882663667202,
+            -0.023459987714886665,
+            0.0061378986574709415,
+            -0.011959806084632874,
+            0.021027809008955956,
+            -0.0007313292007893324,
+            0.018672235310077667,
+            0.0028032290283590555,
+            -0.03288228437304497,
+            -0.0033298819325864315,
+            -0.008421656675636768,
+            -0.00590329896658659,
+            -0.014889910817146301,
+            -0.004088740795850754,
+            -0.0015464440220966935,
+            -0.0031886431388556957,
+            0.037401922047138214,
+            -0.007128963712602854,
+            -0.009000974707305431,
+            -0.0014949757605791092,
+            0.015722978860139847,
+            0.020012805238366127,
+            0.0009467780473642051,
+            -0.008474322035908699,
+            0.010427725501358509,
+            -8.408490248257294e-05,
+            0.010762867517769337,
+            -0.0011221294989809394,
+            -0.008503048680722713,
+            0.001982129644602537,
+            -0.0003803270810749382,
+            -0.027979625388979912,
+            -0.005884147714823484,
+            0.0011592345545068383,
+            -0.013376981019973755,
+            0.038302019238471985,
+            0.005328768398612738,
+            -0.007004482205957174,
+            0.0015428533079102635,
+            0.018346667289733887,
+            0.01132782269269228,
+            0.010226639918982983,
+            -0.003126402385532856,
+            0.004490911960601807,
+            -0.04511978104710579,
+            -0.027749814093112946,
+            -0.025949619710445404,
+            -0.03156086429953575,
+            0.015445290133357048,
+            0.018317941576242447,
+            -0.006875212769955397,
+            0.013673821464180946,
+            0.012812025845050812,
+            -0.001474627759307623,
+            0.009695199318230152,
+            -0.012802449986338615,
+            -0.0017247878713533282,
+            -0.00949890073388815,
+            -0.011710843071341515,
+            0.018001949414610863,
+            0.011797022074460983,
+            0.024206876754760742,
+            0.03929787129163742,
+            -0.011950230225920677,
+            0.0026859291829168797,
+            0.0011843702523037791,
+            0.006631037220358849,
+            -0.005711788777261972,
+            -0.013989812694489956,
+            0.010580932721495628,
+            -0.000983883161097765,
+            0.014707976020872593,
+            -0.01209386344999075,
+            0.0019665693398565054,
+            -0.00619535194709897,
+            -0.013961086049675941,
+            0.011366124264895916,
+            -0.01805940270423889,
+            0.012371552176773548,
+            -0.03941277787089348,
+            -0.007315685972571373,
+            -0.016489019617438316,
+            0.051554519683122635,
+            0.007933305576443672,
+            -0.01782001554965973,
+            -0.0017511205514892936,
+            0.025624051690101624,
+            0.004100710153579712,
+            -0.009771803393959999,
+            0.008641893044114113,
+            -0.00548676447942853,
+            0.011050133034586906,
+            0.0009144607465714216,
+            0.0029995269142091274,
+            -0.026026222854852676,
+            -0.005855421535670757,
+            -0.0054628257639706135,
+            0.01989790052175522,
+            0.000962338293902576,
+            0.016900766640901566,
+            0.08870747685432434,
+            -0.0035070287995040417,
+            -0.009451023302972317,
+            -0.0068608494475483894,
+            -0.019763842225074768,
+            -0.004562728106975555,
+            -0.008900431916117668,
+            -0.005639972630888224,
+            0.0030426166485995054,
+            -0.007186416536569595,
+            0.0054724011570215225,
+            0.014229200780391693,
+            -0.010925651527941227,
+            -0.008699346333742142,
+            -0.007195991929620504,
+            -0.01430580485612154,
+            0.016852889209985733,
+            0.031656619161367416,
+            0.0116821164265275,
+            0.04902658611536026,
+            -0.02533678710460663,
+            -0.014861184172332287,
+            0.004873932339251041,
+            0.004251524340361357,
+            0.00989149697124958,
+            0.007464106194674969,
+            -0.012409854680299759,
+            0.02330677956342697,
+            0.007756159175187349,
+            -0.0037990817800164223,
+            -0.012687544338405132,
+            0.015588922426104546,
+            0.015464440919458866,
+            -0.006061294581741095,
+            0.031522564589977264,
+            0.0025542660150676966,
+            0.010667112655937672,
+            0.010398998856544495,
+            0.0049026585184037685,
+            0.01557934656739235,
+            0.014095143415033817,
+            -0.0006271955789998174,
+            0.002482449635863304,
+            -0.002757745562121272,
+            0.013319527730345726,
+            0.018289214000105858,
+            0.012984384782612324,
+            -0.005831482820212841,
+            0.02560489997267723,
+            0.026581602171063423,
+            -0.00262368842959404,
+            -0.031120391562581062,
+            0.03864673897624016,
+            -0.020261768251657486,
+            0.0006313848425634205,
+            -0.008890856057405472,
+            0.018384970724582672,
+            -0.009403145872056484,
+            -0.004950535949319601,
+            -0.009163758717477322,
+            -0.028822271153330803,
+            -0.003729659365490079,
+            -0.003921169321984053,
+            0.012668393552303314,
+            0.01869138516485691,
+            -0.010628811083734035,
+            0.02112356387078762,
+            -0.012649241834878922,
+            0.020510731264948845,
+            -0.008570076897740364,
+            -0.022004511207342148,
+            0.008713710121810436,
+            -0.001651774626225233,
+            0.0019426306243985891,
+            -0.015177175402641296,
+            0.021219318732619286,
+            0.04684337228536606,
+            0.004869144409894943,
+            -0.00040815590182319283,
+            -0.007990758866071701,
+            -0.0033969103824347258,
+            -0.010734140872955322,
+            0.021295923739671707,
+            0.0058793602511286736,
+            -0.032863132655620575,
+            0.009752651676535606,
+            0.015885762870311737,
+            0.0014782185899093747,
+            0.017810439690947533,
+            -0.0013908421387895942,
+            0.03156086429953575,
+            0.006587947718799114,
+            0.004258705768734217,
+            0.0036554490216076374,
+            -0.014928212389349937,
+            0.0025279333349317312,
+            -0.0027745026163756847,
+            -0.02727103792130947,
+            0.00483084237203002,
+            -0.0012160891201347113,
+            -0.020912902429699898,
+            0.02233007736504078,
+            -0.03841692581772804,
+            -0.019083982333540916,
+            -0.003083312651142478,
+            -0.018844593316316605,
+            -0.013252499513328075,
+            0.01937124691903591,
+            -0.013999388553202152,
+            0.03636777028441429,
+            -0.007899791933596134,
+            0.007962032221257687,
+            -0.0036865693982690573,
+            -0.016948644071817398,
+            0.038876552134752274,
+            0.0045579406432807446,
+            -0.011078859679400921,
+            -0.008694558404386044,
+            0.022425832226872444,
+            -0.02213856764137745,
+            0.017168881371617317,
+            0.005673486739397049,
+            0.029856424778699875,
+            -0.02233007736504078,
+            0.01068626344203949,
+            0.025777259841561317,
+            0.009613807313144207,
+            -0.006003841757774353,
+            -0.010226639918982983,
+            -0.031139543280005455,
+            0.031771525740623474,
+            0.04151938855648041,
+            0.004936173092573881,
+            -0.005544217303395271,
+            0.0022514406591653824,
+            0.013511037454009056,
+            0.014583494514226913,
+            0.010552207008004189,
+            0.007119387853890657,
+            0.0020240223966538906,
+            -0.032097093760967255,
+            0.018739264458417892,
+            0.002458510920405388,
+            0.015177175402641296,
+            0.009063215926289558,
+            -0.026792263612151146,
+            -0.0009743076516315341,
+            -0.0021437162067741156,
+            0.007617314346134663,
+            0.02003195695579052,
+            0.013444009236991405,
+            0.028017928823828697,
+            0.00994895026087761,
+            0.0016924705123528838,
+            -0.012563062831759453,
+            -0.03315039724111557,
+            0.010178761556744576,
+            0.016737982630729675,
+            0.004593848716467619,
+            -0.01686246506869793,
+            0.010466027073562145,
+            0.007282171864062548,
+            -0.01145230419933796,
+            -0.014995240606367588,
+            -0.0064634657464921474,
+            -0.0038852612487971783,
+            -0.0302011426538229,
+            0.006094809155911207,
+            0.0020324010401964188,
+            -0.007081086281687021,
+            0.011337397620081902,
+            0.015598498284816742,
+            0.013137592934072018,
+            -0.022885456681251526,
+            0.013913208618760109,
+            -0.010762867517769337,
+            -0.015004816465079784,
+            -0.008986611850559711,
+            0.006970967631787062,
+            -0.0020946417935192585,
+            -0.006980543024837971,
+            0.01155763491988182,
+            0.0011813779128715396,
+            -0.030354350805282593,
+            0.032997190952301025,
+            -0.0037033266853541136,
+            -0.03060331381857395,
+            -0.024685652926564217,
+            0.008584440685808659,
+            0.02771151252090931,
+            0.0022203202825039625,
+            -0.004696785472333431,
+            0.03182898089289665,
+            -0.010580932721495628,
+            -0.0029899515211582184,
+            -0.03380153328180313,
+            -0.0034280307590961456,
+            -0.012342826463282108,
+            -0.020893752574920654,
+            -0.012294949032366276,
+            0.00763167766854167,
+            -0.007464106194674969,
+            -0.011634238995611668,
+            0.01805940270423889,
+            -0.01686246506869793,
+            0.002427390543743968,
+            0.003870897926390171,
+            -0.0024441475979983807,
+            0.021161865442991257,
+            -0.017427420243620872,
+            0.0018133613048121333,
+            -0.0035692695528268814,
+            -0.02188960462808609,
+            0.006755518727004528,
+            -0.001415977836586535,
+            -0.03192473575472832,
+            -0.01861478202044964,
+            -0.020146863535046577,
+            -0.0251452773809433,
+            0.006592735182493925,
+            0.009202060289680958,
+            0.015339959412813187,
+            0.005544217303395271,
+            0.01786789298057556,
+            -0.005458037834614515,
+            -0.005525066517293453,
+            -0.027883870527148247,
+            0.001639805268496275,
+            0.009570717811584473,
+            0.014219624921679497,
+            -0.024436689913272858,
+            -0.010542631149291992,
+            0.010868198238313198,
+            0.02162148989737034,
+            -0.004100710153579712,
+            0.014899485744535923,
+            0.014420710504055023,
+            -0.0241877268999815,
+            -4.5165711526351515e-06,
+            -0.013645094819366932,
+            -0.0254133902490139,
+            -0.002503994619473815,
+            -0.0002746972895693034,
+            -0.004031287506222725,
+            -0.015311232767999172,
+            0.015004816465079784,
+            0.010523480363190174,
+            0.01631666161119938,
+            -0.02464734949171543,
+            -0.004459791351109743,
+            -0.004115073475986719,
+            -0.013999388553202152,
+            -0.021678943186998367,
+            0.012543912045657635,
+            -0.026351790875196457,
+            0.014114294201135635,
+            -0.001409993157722056,
+            0.020319221541285515,
+            0.01978299394249916,
+            0.017561476677656174,
+            -0.0019031316041946411,
+            -0.0026955048087984324,
+            0.009024913422763348,
+            0.0071146003901958466,
+            -0.016192179173231125,
+            0.015406987629830837,
+            -0.008503048680722713,
+            -0.0061283232644200325,
+            0.012266222387552261,
+            -0.008823827840387821,
+            0.002645233180373907,
+            0.014047265984117985,
+            -0.0059655397199094296,
+            0.0009988448582589626,
+            -0.008388142101466656,
+            -0.004206040408462286,
+            -0.009479749947786331,
+            0.007468894124031067,
+            -0.03330360725522041,
+            0.013128018006682396,
+            -0.0036722063086926937,
+            0.0033418512903153896,
+            -0.000942588783800602,
+            -0.007559861522167921,
+            -0.02093205414712429,
+            -0.008273236453533173,
+            -0.014324955642223358,
+            -0.002482449635863304,
+            -0.012065136805176735,
+            0.01939997263252735,
+            0.009139820002019405,
+            -0.005496339872479439,
+            0.009364844299852848,
+            0.006918302271515131,
+            -0.019036104902625084,
+            -0.0004512456653174013,
+            0.02317272312939167,
+            0.018183885142207146,
+            -0.00739229004830122,
+            0.03326530382037163,
+            -0.012026834301650524,
+            0.015081420540809631,
+            0.014286654070019722,
+            0.01830836571753025,
+            -0.006353347562253475,
+            -0.030507558956742287,
+            0.017896618694067,
+            0.020070258527994156,
+            -0.020759694278240204,
+            -0.013242923654615879,
+            0.006578371860086918,
+            0.042974866926670074,
+            -0.007085873745381832,
+            0.016086848452687263,
+            0.0012603758368641138,
+            -0.006827335339039564,
+            -0.011145887896418571,
+            -0.011193765327334404,
+            0.06863722205162048,
+            0.023364232853055,
+            0.013424858450889587,
+            -0.007124175783246756,
+            0.0022011692635715008,
+            -0.02112356387078762,
+            -0.001406402327120304,
+            0.0026428394485265017,
+            0.002515963977202773,
+            -0.008752011694014072,
+            -0.025011219084262848,
+            0.01916058547794819,
+            0.030450105667114258,
+            0.009163758717477322,
+            -0.011270369403064251,
+            -0.013874907046556473,
+            0.02303866483271122,
+            0.002209547907114029,
+            0.017341239377856255,
+            0.011797022074460983,
+            0.001982129644602537,
+            -0.03765088692307472,
+            0.003921169321984053,
+            0.015033543109893799,
+            0.0038852612487971783,
+            0.0036434796638786793,
+            0.0078567024320364,
+            0.012400278821587563,
+            0.011212916113436222,
+            0.0034280307590961456,
+            0.027979625388979912,
+            -0.026677357032895088,
+            -0.0019486153032630682,
+            0.0007307307096198201,
+            -0.0005475992220453918,
+            0.009015338495373726,
+            0.013721698895096779,
+            -0.015196327120065689,
+            -0.011174614541232586,
+            0.0011562422150745988,
+            0.0065592210739851,
+            -0.02284715510904789,
+            -0.01091607566922903,
+            0.01414302084594965,
+            -0.01984044723212719,
+            0.011816173791885376,
+            -0.0015763675328344107,
+            0.0006122338818386197,
+            -0.02323017455637455,
+            -0.0008929158793762326,
+            0.006683702580630779,
+            0.010935226455330849,
+            -0.007397077511996031,
+            0.022368380799889565,
+            -0.024111121892929077,
+            -0.01942870020866394,
+            0.017638079822063446,
+            -0.009747864678502083,
+            0.013156743720173836,
+            0.014669673517346382,
+            -0.013989812694489956,
+            -0.003739234758540988,
+            0.0007684343145228922,
+            -0.004940960556268692,
+            -0.010466027073562145,
+            0.00400495482608676,
+            0.0021664579398930073,
+            -0.005735727492719889,
+            0.002329241717234254,
+            0.017293361946940422,
+            -0.011548059061169624,
+            -0.01628793403506279,
+            -0.00590329896658659,
+            0.012400278821587563,
+            -0.000718761351890862,
+            0.018049826845526695,
+            0.01587618701159954,
+            -0.005917662288993597,
+            0.009834043681621552,
+            -0.022808853536844254,
+            0.01078201923519373,
+            0.008167905732989311,
+            -0.008498260751366615,
+            0.006769882049411535,
+            -0.0022670007310807705,
+            -0.004732693545520306,
+            0.03311209753155708,
+            0.020070258527994156,
+            0.007928518578410149,
+            -0.01283117663115263,
+            -0.005888935644179583,
+            -0.026792263612151146,
+            0.0003776339872274548,
+            -0.0028056229930371046,
+            -0.011672540567815304,
+            -0.026524148881435394,
+            0.011528908275067806,
+            0.002091050846502185,
+            -0.004423883277922869,
+            -0.020702242851257324,
+            0.013740849681198597,
+            0.015627223998308182,
+            0.0005272512789815664,
+            0.002843925030902028,
+            0.015167600475251675,
+            0.010657536797225475,
+            0.018298789858818054,
+            -0.010504329577088356,
+            -0.0056447600945830345,
+            0.001198135083541274,
+            -0.026524148881435394,
+            0.0036722063086926937,
+            -0.006085233297199011,
+            0.003061767667531967,
+            0.01786789298057556,
+            -0.011481030844151974,
+            -0.01989790052175522,
+            0.003468726761639118,
+            0.0032987615559250116,
+            -0.01935209520161152,
+            0.014880334958434105,
+            -0.023249326273798943,
+            -0.0004925400717183948,
+            -0.030430955812335014,
+            -0.0060325684025883675,
+            -0.002427390543743968,
+            0.0017834377940744162,
+            -0.0049792625941336155,
+            0.0019569939468055964,
+            -0.015435714274644852,
+            0.010829896666109562,
+            0.03188643231987953,
+            0.010284092277288437,
+            -0.0012071121018379927,
+            0.001107766292989254,
+            -0.0043161590583622456,
+            -0.006841698195785284,
+            -0.01139485090970993,
+            -0.009786166250705719,
+            0.01299396064132452,
+            -0.0058793602511286736,
+            0.02495376579463482,
+            -0.009283452294766903,
+            0.0024513292592018843,
+            0.006846486125141382,
+            0.008708922192454338,
+            -0.011059708893299103,
+            -0.018298789858818054,
+            0.00947017502039671,
+            -0.011864051222801208,
+            0.00815833080559969,
+            -0.008929158560931683,
+            -0.011021406389772892,
+            0.0016050940612331033,
+            0.0032245514448732138,
+            -0.012649241834878922,
+            -0.014707976020872593,
+            -0.005333556327968836,
+            -0.0006906333146616817,
+            -0.005802756175398827,
+            0.00834984052926302,
+            -0.008163117803633213,
+            -0.008373779244720936,
+            -0.005391009617596865,
+            -0.0011041754623875022,
+            -0.0023172723595052958,
+            0.0032413084991276264,
+            0.007765734568238258,
+            0.020051108673214912,
+            0.00900576263666153,
+            0.012505609542131424,
+            0.023268477991223335,
+            0.01513887383043766,
+            0.010523480363190174,
+            -0.008301963098347187,
+            0.01948615349829197,
+            -0.006243229378014803,
+            0.004411913920193911,
+            0.022828003391623497,
+            0.010829896666109562,
+            -0.00691351480782032,
+            0.02757745422422886,
+            -0.010150035843253136,
+            -0.024915464222431183,
+            -0.01917973719537258,
+            0.010858622379601002,
+            -0.02413027361035347,
+            0.003813444869592786,
+            -0.004356855060905218,
+            -0.022253474220633507,
+            0.01174914464354515,
+            0.01596236787736416,
+            0.021085262298583984,
+            0.001044927048496902,
+            -0.004809297621250153,
+            0.0007546694832853973,
+            0.016115576028823853,
+            0.01276414841413498,
+            -0.005836270283907652,
+            0.0012879054993391037,
+            -0.003830202156677842,
+            0.011433153413236141,
+            0.010734140872955322,
+            -0.0018672235310077667,
+            0.007306110579520464,
+            -0.001930661266669631,
+            0.0016565624391660094,
+            0.0016936674946919084,
+            0.011825748719274998,
+            -0.005946388468146324,
+            -0.001229255460202694,
+            0.0012376339873299003,
+            0.011978956870734692,
+            0.0015727767022326589,
+            -0.00798118393868208,
+            0.015895338729023933,
+            0.011337397620081902,
+            -0.014717550948262215,
+            0.01916058547794819,
+            -0.010063855908811092,
+            -0.003739234758540988,
+            0.0006499374285340309,
+            0.015636799857020378,
+            0.004601030144840479,
+            0.01759977824985981,
+            -0.0032077941577881575,
+            0.008139179088175297,
+            -0.015158024616539478,
+            -0.020434128120541573,
+            -0.04672846570611,
+            0.008617954328656197,
+            -0.0020431734155863523,
+            -0.006884788163006306,
+            0.0019498122856020927,
+            0.01901695318520069,
+            0.013300376944243908,
+            0.009417509660124779,
+            -0.0034591511357575655,
+            0.014746277593076229,
+            -0.018021101132035255,
+            -0.019218038767576218,
+            -0.015943216159939766,
+            0.00912545621395111,
+            0.0018001949647441506,
+            -0.002961224876344204,
+            -0.005630397237837315,
+            0.020568184554576874,
+            -0.006420376244932413,
+            0.011184190399944782,
+            -3.63569961336907e-05,
+            -0.018442422151565552,
+            -0.007382714655250311,
+            -0.004452609922736883,
+            0.00857965275645256,
+            0.0029468617867678404,
+            0.022387530654668808,
+            0.0116821164265275,
+            0.00011879610974574462,
+            0.010475602932274342,
+            -0.010150035843253136,
+            -0.0066884905099868774,
+            0.016594350337982178,
+            0.013578066602349281,
+            0.0002670668181963265,
+            0.012361977249383926,
+            0.014353682287037373,
+            0.004675240255892277,
+            0.0019426306243985891,
+            0.01590491458773613,
+            0.05787435173988342,
+            0.018902046605944633,
+            0.003605177626013756,
+            -0.008440807461738586,
+            0.008120028302073479,
+            0.004897871054708958,
+            0.013740849681198597,
+            -0.00372726540081203,
+            -0.0234982892870903,
+            0.006832122802734375,
+            0.010236214846372604,
+            -0.01948615349829197,
+            -0.009958525188267231,
+            0.006022992543876171,
+            0.00836420338600874,
+            -0.015857037156820297,
+            -0.0004030689306091517,
+            -0.011844899505376816,
+            -0.016086848452687263,
+            0.0004171329492237419,
+            0.013884482905268669,
+            -0.023440835997462273,
+            -0.0011951427441090345,
+            -0.0009521642932668328,
+            -0.008215783163905144,
+            -0.009556354023516178,
+            0.0005706402589567006,
+            -0.0021832152269780636,
+            -0.018480725586414337,
+            -0.0022059569600969553,
+            -0.00936005637049675,
+            -0.007444955408573151,
+            0.022636493667960167,
+            -0.0102457907050848,
+            -0.006109172478318214,
+            -0.02213856764137745,
+            -0.023957913741469383,
+            0.01784874126315117,
+            -0.016249632462859154,
+            0.003097675973549485,
+            -0.018155157566070557,
+            0.002120974473655224,
+            -0.019036104902625084,
+            0.01677628420293331,
+            0.010389422997832298,
+            -0.01942870020866394,
+            0.02131507359445095,
+            0.006511343643069267,
+            0.004809297621250153,
+            -0.012649241834878922,
+            -0.002219123300164938,
+            -0.012314099818468094,
+            0.0088142529129982,
+            0.009795742109417915,
+            0.010647961869835854,
+            0.008680195547640324,
+            0.008282812312245369,
+            0.006865637376904488,
+            0.005788392852991819,
+            -0.012697120197117329,
+            -0.0401788204908371,
+            0.01882544346153736,
+            0.016077272593975067,
+            -0.01628793403506279,
+            0.016278358176350594,
+            -0.013865331187844276,
+            -0.020012805238366127,
+            -0.01378872711211443,
+            -0.008689770475029945,
+            -0.014583494514226913,
+            0.011825748719274998,
+            0.01601981930434704,
+            -0.011366124264895916,
+            0.00036416843067854643,
+            0.006372498814016581,
+            0.016173027455806732,
+            -0.009546778164803982,
+            0.005979903042316437,
+            -0.0015272931195795536,
+            -0.01049475371837616,
+            0.01593364030122757,
+            -0.016986945644021034,
+            -0.006248017307370901,
+            0.004277857020497322,
+            -0.022579040378332138,
+            -0.018672235310077667,
+            -0.011461880058050156,
+            0.014947363175451756,
+            -0.0052282256074249744,
+            -0.02194705791771412,
+            -0.0046608769334852695,
+            -0.03539106622338295,
+            0.008459959179162979,
+            0.00566391134634614,
+            -0.0017786500975489616,
+            -0.015541044995188713,
+            0.036712486296892166,
+            0.004751844331622124,
+            -0.02047242969274521,
+            0.005027140025049448,
+            -0.02041497640311718,
+            0.013807878829538822,
+            0.009518052451312542,
+            -0.017149729654192924,
+            0.002458510920405388,
+            0.0007905776146799326,
+            -0.0023807098623365164,
+            -0.00625759270042181,
+            0.004285038448870182,
+            0.004735087510198355,
+            -0.01896907575428486,
+            -0.004124648869037628,
+            0.015464440919458866,
+            -0.0033442452549934387,
+            -0.010513904504477978,
+            -0.011835324577987194,
+            -0.0037751428317278624,
+            0.0025662353727966547,
+            -0.0037631734739989042,
+            -0.0003393319493625313,
+            -0.01866265945136547,
+            -0.01650817133486271,
+            0.034529272466897964,
+            0.0020754907745867968,
+            0.017590202391147614,
+            -0.00494574848562479,
+            -0.0015979124000295997,
+            -0.008694558404386044,
+            0.015339959412813187,
+            0.013874907046556473,
+            0.011471454985439777,
+            0.0029708005022257566,
+            0.009436660446226597,
+            0.01773383468389511,
+            0.00483802380040288,
+            0.01869138516485691,
+            -0.009168545715510845,
+            -0.00888128113001585,
+            -0.011385275982320309,
+            0.029090384021401405,
+            0.009522839449346066,
+            -0.007023632992058992,
+            -0.025394240394234657,
+            -0.0019905080553144217,
+            0.009460599161684513,
+            0.024168575182557106,
+            -0.018767990171909332,
+            -0.028898874297738075,
+            -0.013204621151089668,
+            0.01418132334947586,
+            0.01549316756427288,
+            -0.014085568487644196,
+            -0.0022394713014364243,
+            0.011241642758250237,
+            0.023823857307434082,
+            -0.027826417237520218,
+            0.00936005637049675,
+            -0.0033298819325864315,
+            0.0037320530973374844,
+            -0.01939997263252735,
+            -0.02476225607097149,
+            0.00017475297499913722,
+            0.0034040920436382294,
+            0.009111093357205391,
+            -0.015301656909286976,
+            0.0017211970407515764,
+            0.020759694278240204,
+            -0.009628170169889927,
+            0.015406987629830837,
+            -0.0007564648985862732,
+            -0.010418149642646313,
+            0.0013369799125939608,
+            -0.005323980934917927,
+            -0.009814892895519733,
+            0.015560195781290531,
+            0.005414948333054781,
+            0.00428982637822628,
+            0.002503994619473815,
+            -0.00953720323741436,
+            -0.003291579894721508,
+            -0.008981823921203613,
+            -0.010284092277288437,
+            0.015110147185623646,
+            0.013769576326012611,
+            -0.020644789561629295,
+            0.024666501209139824,
+            -0.0029037718195468187,
+            -0.0049002645537257195,
+            -0.00507022999227047,
+            -0.008459959179162979,
+            0.0170827005058527,
+            0.0006918302387930453,
+            -0.00016442938067484647,
+            0.015282506123185158,
+            0.0038996245712041855,
+            0.001734363380819559,
+            -0.017178455367684364,
+            -0.007047571707516909,
+            -0.00023445025726687163,
+            0.005917662288993597,
+            0.0065257069654762745,
+            0.016661379486322403,
+            -0.006310258060693741,
+            -0.01014045998454094,
+            0.007023632992058992,
+            0.00955156609416008,
+            0.015924064442515373,
+            0.013022687286138535,
+            0.014449437148869038,
+            0.018767990171909332,
+            0.019189313054084778,
+            -0.008412080816924572,
+            -0.01727421209216118,
+            0.017235908657312393,
+            -0.008139179088175297,
+            -0.003289185930043459,
+            -0.04117467254400253,
+            0.01705397479236126,
+            0.020817147567868233,
+            -0.02137252688407898,
+            0.001020988216623664,
+            -0.004663270898163319,
+            -0.007784885820001364,
+            -0.021602340042591095,
+            0.018155157566070557,
+            -0.004852387122809887,
+            0.0019210857572034001,
+            0.004888295195996761,
+            -0.01820303499698639,
+            0.005855421535670757,
+            0.023574894294142723,
+            -0.009000974707305431,
+            0.004907446447759867,
+            -0.0021772305481135845,
+            -0.003554906230419874,
+            -0.012917356565594673,
+            0.006597523111850023,
+            -0.02457074634730816,
+            0.023670649155974388,
+            0.018193459138274193,
+            0.011002255603671074,
+            -0.007952457293868065,
+            0.007473681587725878,
+            -0.015349534340202808,
+            -0.0027266251854598522,
+            -0.00199888669885695,
+            0.014104719273746014,
+            0.002975588198751211,
+            0.00817269366234541,
+            0.0058697848580777645,
+            0.002906165784224868,
+            0.0032748228404670954,
+            -0.018241336569190025,
+            0.016144301742315292,
+            -0.006147474516183138,
+            0.007947669364511967,
+            0.013635518960654736,
+            0.0026931108441203833,
+            -0.009599443525075912,
+            -0.010312818922102451,
+            -0.004821266978979111,
+            0.006592735182493925,
+            0.004931385163217783,
+            0.011911928653717041,
+            0.009977675974369049,
+            0.009824467822909355,
+            -0.02503037080168724,
+            0.012879054062068462,
+            0.0032556718215346336,
+            -0.03282483294606209,
+            -0.002697898540645838,
+            0.0016972583252936602,
+            -0.0030761309899389744,
+            0.010255365632474422,
+            0.02317272312939167,
+            -0.017504023388028145,
+            -0.009206848219037056,
+            0.01801152527332306,
+            0.005199499428272247,
+            -0.016077272593975067,
+            -0.016852889209985733,
+            0.007377926725894213,
+            0.009901071898639202,
+            0.011098010465502739,
+            -0.014985665678977966,
+            0.009077578783035278,
+            0.007052359636873007,
+            0.0254133902490139,
+            -0.03085227683186531,
+            0.018902046605944633,
+            0.012266222387552261,
+            0.0169390682131052,
+            -0.00888128113001585,
+            -0.010896924883127213,
+            0.0008031455217860639,
+            0.012888629920780659,
+            -0.0017571052303537726,
+            0.015598498284816742,
+            -0.002170048886910081,
+            0.005256952252238989,
+            -0.02368979901075363,
+            -0.005041503347456455,
+            -0.0026404454838484526,
+            0.023057816550135612,
+            0.01485160831362009,
+            0.02661990374326706,
+            0.006233653984963894,
+            -0.012007683515548706,
+            0.010743716731667519,
+            -0.006429951637983322,
+            0.0003551913832779974,
+            -0.021219318732619286,
+            -0.00039409188320860267,
+            -0.019610634073615074,
+            -0.003442394081503153,
+            -0.016900766640901566,
+            0.03609965369105339,
+            0.02047242969274521,
+            0.014123870059847832,
+            0.025509145110845566,
+            -9.672756277723238e-05,
+            0.008670619688928127,
+            -0.004225191660225391,
+            -0.0018923592288047075,
+            -0.015253779478371143,
+            0.00774658378213644,
+            -0.006789033301174641,
+            0.001808573491871357,
+            0.008622742258012295,
+            -0.004129436798393726,
+            0.016632651910185814,
+            -0.022483285516500473,
+            0.014411135576665401,
+            0.01378872711211443,
+            0.014315379783511162,
+            0.016603926196694374,
+            -0.010284092277288437,
+            -0.0031862494070082903,
+            -0.0028223800472915173,
+            0.001198135083541274,
+            -0.009039277210831642,
+            -0.0023711344692856073,
+            -0.0028918024618178606,
+            -0.0088142529129982,
+            -0.005951176397502422,
+            0.022387530654668808,
+            0.0024968129582703114,
+            0.018298789858818054,
+            0.0024477385450154543,
+            0.015885762870311737,
+            0.005673486739397049,
+            0.005783604923635721,
+            -0.0157134048640728,
+            -0.0005550800706259906,
+            0.013482310809195042,
+            0.0021999722812324762,
+            -0.0031790677458047867,
+            -0.021813001483678818,
+            -0.021659793332219124,
+            0.0007851914269849658,
+            0.00673636794090271,
+            0.02162148989737034,
+            0.0017666807398200035,
+            -0.015272930264472961,
+            0.0008462352561764419,
+            -0.021295923739671707,
+            -0.0243217833340168,
+            -0.014564343728125095,
+            0.0011191371595486999,
+            0.029856424778699875,
+            0.0016350174555554986,
+            -0.009010550566017628,
+            0.009407933801412582,
+            -0.021985359489917755,
+            -0.0023615588434040546,
+            0.002669172128662467,
+            -0.0204915814101696,
+            0.009202060289680958,
+            -0.031522564589977264,
+            0.015474015846848488,
+            -0.013463160023093224,
+            0.00035459292121231556,
+            0.022310927510261536,
+            -0.005165984854102135,
+            -0.0008492275956086814,
+            -0.011385275982320309,
+            0.00044556023203767836,
+            0.0159910935908556,
+            -0.004129436798393726,
+            -0.004514850676059723,
+            -0.0014111900236457586,
+            0.003509422531351447,
+            -0.0016745164757594466,
+            -0.005941601004451513,
+            0.0019881143234670162,
+            0.0011125539895147085,
+            -0.019581908360123634,
+            0.0006631037103943527,
+            0.018404120579361916,
+            -0.012974809855222702,
+            -0.03246096149086952,
+            0.01653689704835415,
+            -0.02405366860330105,
+            -0.010801170021295547,
+            -0.0035261798184365034,
+            0.017006097361445427,
+            0.0022658039815723896,
+            -0.007492832839488983,
+            0.020721392706036568,
+            0.007933305576443672,
+            0.01008300669491291,
+            0.004426277242600918,
+            0.003159916726872325,
+            -0.0002832254976965487,
+            -0.0026428394485265017,
+            -5.97533971813391e-06,
+            0.0064634657464921474,
+            0.015971941873431206,
+            -0.018375394865870476,
+            0.027883870527148247,
+            -0.0070284209214150906,
+            -0.0034280307590961456,
+            0.0043233404867351055,
+            -0.006808184087276459,
+            -0.008368991315364838,
+            0.010877774097025394,
+            -0.02721358649432659,
+            0.017944496124982834,
+            -0.015090996399521828,
+            -0.011921503581106663,
+            -0.011883202008903027,
+            0.014229200780391693,
+            0.006664551794528961,
+            -0.01951487921178341,
+            0.005740515422075987,
+            -0.01490906160324812,
+            -0.01596236787736416,
+            0.002160473261028528,
+            -0.004100710153579712,
+            0.014928212389349937,
+            -0.0116821164265275,
+            -0.01164381392300129,
+            0.00536228297278285,
+            -0.019859597086906433,
+            0.008321113884449005,
+            -0.0077705224975943565,
+            0.009403145872056484,
+            -0.025892166420817375,
+            0.011576785705983639,
+            0.015110147185623646,
+            -0.004735087510198355,
+            -0.017197607085108757,
+            0.015857037156820297,
+            0.016469869762659073,
+            -0.004447821993380785,
+            -0.011481030844151974,
+            0.00822535902261734,
+            0.013147168792784214,
+            0.0025518720503896475,
+            -0.004984050523489714,
+            -0.017130577936768532,
+            -0.01126079447567463,
+            0.00923557486385107,
+            -0.0016074879094958305,
+            -0.036655034869909286,
+            -0.01672840677201748,
+            -0.02131507359445095,
+            -0.011433153413236141,
+            -0.009474962018430233,
+            0.006003841757774353,
+            0.006607098504900932,
+            -0.002515963977202773,
+            -0.02738594450056553,
+            -0.00035638833651319146,
+            0.004857175052165985,
+            -0.014813306741416454,
+            -0.001930661266669631,
+            -0.003621934913098812,
+            -0.0036985387559980154,
+            -0.0048164790496230125,
+            -0.005060654599219561,
+            0.013769576326012611,
+            -0.003739234758540988,
+            0.01239070389419794,
+            0.011011830531060696,
+            -0.00787106528878212,
+            -0.0026619904674589634,
+            -0.03211624547839165,
+            -0.014449437148869038,
+            0.008182269521057606,
+            0.0068800002336502075,
+            -0.002951649483293295,
+            -0.0005161795997992158,
+            0.008321113884449005,
+            0.00029130480834282935,
+            -0.005084593314677477,
+            0.005237801466137171,
+            0.013195046223700047,
+            -0.013501462526619434,
+            0.024858010932803154,
+            0.010456451214849949,
+            -0.00863710604608059,
+            -0.012534336186945438,
+            -0.01078201923519373,
+            0.018509451299905777,
+            0.009068002924323082,
+            0.017331663519144058,
+            -0.0023519834503531456,
+            0.01247688289731741,
+            0.0014231594977900386,
+            0.0262368842959404,
+            -0.006482616998255253,
+            -0.006808184087276459,
+            -0.010839471593499184,
+            0.0021568825468420982,
+            -0.02778811566531658,
+            0.00447415467351675,
+            0.01478458009660244,
+            -0.009987251833081245,
+            -0.002028810093179345,
+            0.006406012922525406,
+            -0.006468253675848246,
+            0.011548059061169624,
+            -0.009977675974369049,
+            0.0024800559040158987,
+            -0.017551900818943977,
+            -0.0020096590742468834,
+            0.016546472907066345,
+            0.0010138065554201603,
+            0.006640612613409758,
+            0.04749450460076332,
+            -0.009723925031721592,
+            -0.009393570944666862,
+            0.026102827861905098,
+            -1.738365608616732e-05,
+            0.019993655383586884,
+            -0.0006035560509189963,
+            0.0010419347090646625,
+            0.004675240255892277,
+            0.013645094819366932,
+            0.018538178876042366,
+            0.005304829683154821,
+            0.00196776632219553,
+            -0.01084904745221138,
+            -0.003356214612722397,
+            -0.011605512350797653,
+            -0.033514268696308136,
+            -0.005419735796749592,
+            0.003049798309803009,
+            -0.0069422414526343346,
+            0.007579012308269739,
+            -0.01847114972770214,
+            0.007138539105653763,
+            0.02483886107802391,
+            -0.017006097361445427,
+            -0.015158024616539478,
+            -0.008646680973470211,
+            -0.006487404927611351,
+            0.0004410716937854886,
+            -0.01696779578924179,
+            0.001875602058134973,
+            -0.007076298352330923,
+            0.0036817817017436028,
+            0.005103744100779295,
+            -0.0135972173884511,
+            0.022004511207342148,
+            0.0030210718978196383,
+            0.020721392706036568,
+            -0.006645400542765856,
+            -0.0001726583286654204,
+            0.005932025611400604,
+            -0.010935226455330849,
+            -0.021468281745910645,
+            0.004325734451413155,
+            0.01049475371837616,
+            -0.02162148989737034,
+            -0.022253474220633507,
+            -0.0029109534807503223,
+            0.0015655950410291553,
+            -0.01030324399471283,
+            0.006990118883550167,
+            -0.0016421991167590022,
+            0.0126396669074893,
+            -0.00013046625826973468,
+            -0.0069422414526343346,
+            0.016527321189641953,
+            0.02175554819405079,
+            -0.004735087510198355,
+            -0.0029444678220897913,
+            -0.0004012735153082758,
+            -0.015550619922578335,
+            0.011662965640425682,
+            0.008866917341947556,
+            -0.018681811168789864,
+            0.015272930264472961,
+            0.017915770411491394,
+            0.016785860061645508,
+            0.0015955185517668724,
+            -0.014516465365886688,
+            -0.0005520877311937511,
+            -0.0038948366418480873,
+            -0.00017789493722375482,
+            -0.01549316756427288,
+            -0.005098956637084484,
+            -0.004912233911454678,
+            -0.014803730882704258,
+            0.0020802784711122513,
+            -0.026581602171063423,
+            -0.00043299238313920796,
+            -0.011940655298531055,
+            -0.0003560890909284353,
+            0.020261768251657486,
+            -0.019218038767576218,
+            -0.0006852470687590539,
+            -0.011308671906590462,
+            -0.018356243148446083,
+            0.013070564717054367,
+            -0.005089380778372288,
+            -0.021927906200289726,
+            0.0066118864342570305,
+            0.007344412617385387,
+            -0.008910007774829865,
+            -0.014640947803854942,
+            0.007985970936715603,
+            -0.012553486973047256,
+            0.0009617398027330637,
+            0.020376674830913544,
+            0.016718832775950432,
+            0.009518052451312542,
+            -0.00857965275645256,
+            -0.004433458670973778,
+            0.011739569716155529,
+            0.01689119078218937,
+            0.004895477090030909,
+            0.02694547176361084,
+            -0.024743104353547096,
+            -0.029090384021401405,
+            0.04308977350592613,
+            0.007009269669651985,
+            -0.009546778164803982,
+            0.013415282592177391,
+            -0.024628199636936188,
+            0.009599443525075912,
+            0.033322758972644806,
+            0.008924370631575584,
+            -0.017312513664364815,
+            0.014832457527518272,
+            0.002272985642775893,
+            -0.021525735035538673,
+            0.0035237858537584543,
+            0.005410160403698683,
+            -0.015416563488543034,
+            0.005395797081291676,
+            0.016948644071817398,
+            0.006640612613409758,
+            0.01286947913467884,
+            -0.00235318043269217,
+            0.008067362941801548,
+            0.010255365632474422,
+            0.004277857020497322,
+            -0.007947669364511967,
+            -0.006951816845685244,
+            0.009316966868937016,
+            -0.039834100753068924,
+            -0.01485160831362009,
+            -0.013128018006682396,
+            -0.0052138627506792545,
+            -0.007847126573324203,
+            0.021659793332219124,
+            -0.013405706733465195,
+            -0.01874883845448494,
+            0.02164064161479473,
+            0.024398386478424072,
+            -0.006980543024837971,
+            -0.004665664862841368,
+            0.0037200837396085262,
+            -0.016517747193574905,
+            -0.0020060683600604534,
+            0.005491552408784628,
+            0.035046350210905075,
+            -0.0072773839347064495,
+            -0.0060325684025883675,
+            0.008699346333742142,
+            -0.014545192010700703,
+            0.02956916019320488,
+            0.005285678897053003,
+            -0.006329408846795559,
+            -0.00924993772059679,
+            -0.022119417786598206,
+            -0.0030019208788871765,
+            -0.017130577936768532,
+            -0.02183215133845806,
+            0.0038804735522717237,
+            -0.0010461239144206047,
+            -0.0008893250487744808,
+            0.004881113767623901,
+            0.0022394713014364243,
+            0.018241336569190025,
+            -0.009800529107451439,
+            0.011117161251604557,
+            -0.018432848155498505,
+            -0.0024501322768628597,
+            0.006812972016632557,
+            -0.0017630899092182517,
+            -0.010772443376481533,
+            -0.021027809008955956,
+            0.023364232853055,
+            -0.006894363556057215,
+            0.012496034614741802,
+            -0.01920846290886402,
+            0.013386555947363377,
+            -0.0166709553450346,
+            -0.002225107979029417,
+            0.0032293391413986683,
+            -0.005271315574645996,
+            -0.004964899271726608,
+            -0.003289185930043459,
+            0.01939997263252735,
+            -0.011567209847271442,
+            -0.012084287591278553,
+            0.0251452773809433,
+            0.0241877268999815,
+            0.004268281627446413,
+            -0.01257263869047165,
+            0.001366903306916356,
+            0.007263020612299442,
+            0.0028941964264959097,
+            0.0016015032306313515,
+            0.002992345253005624,
+            0.002399860881268978,
+            0.002276576356962323,
+            0.006712429225444794,
+            0.01978299394249916,
+            -0.016766710206866264,
+            0.0060517191886901855,
+            0.023440835997462273,
+            0.01874883845448494,
+            -0.006908726878464222,
+            -0.002071899827569723,
+            -0.012266222387552261,
+            0.005970327649265528,
+            0.012984384782612324,
+            0.0052138627506792545,
+            0.0015452471561729908,
+            0.0019450244726613164,
+            0.007162477821111679,
+            -0.024302631616592407,
+            0.030181992799043655,
+            0.0028271679766476154,
+            0.014516465365886688,
+            6.553236016770825e-05,
+            0.015177175402641296,
+            -0.013530189171433449,
+            -0.002506388584151864,
+            0.013798302970826626,
+            -0.006248017307370901,
+            0.008790313266217709,
+            0.007574224378913641,
+            -0.005898511037230492,
+            -0.012582213617861271,
+            -0.00044137093937024474,
+            -0.0036674183793365955,
+            -0.00715290242806077,
+            -0.0026165067683905363,
+            -0.002841531066223979,
+            0.0013214197242632508,
+            -0.008316325955092907,
+            0.006410800851881504,
+            -0.007344412617385387,
+            0.007306110579520464,
+            -0.002270591678097844,
+            -0.003770355135202408,
+            0.0008683786145411432,
+            0.025968769565224648,
+            0.00806257501244545,
+            0.03135020285844803,
+            0.003291579894721508,
+            0.0016589562874287367,
+            -0.0008611970115453005,
+            -0.014535617083311081,
+            -0.0262368842959404,
+            0.011346973478794098,
+            -0.02771151252090931,
+            0.00031449549715034664,
+            -0.0032676411792635918,
+            0.003902018303051591,
+            0.012534336186945438,
+            0.011098010465502739,
+            0.009096729569137096,
+            -0.018930774182081223,
+            -0.013568490743637085,
+            -0.002460904885083437,
+            -0.011921503581106663,
+            0.002963618841022253,
+            0.012218344956636429,
+            -0.005118107423186302,
+            0.012792875058948994,
+            -0.0004982255049981177,
+            0.002997133182361722,
+            -0.03699975088238716,
+            -0.012256646528840065,
+            0.00241183047182858,
+            -0.00994895026087761,
+            0.007708281744271517,
+            0.009039277210831642,
+            -0.001272345194593072,
+            0.012208769097924232,
+            0.01601981930434704,
+            0.0058697848580777645,
+            -0.0035668755881488323,
+            -0.0023519834503531456,
+            -0.0011442728573456407,
+            -0.017963647842407227,
+            0.012610940262675285,
+            -0.017446570098400116,
+            0.00590329896658659,
+            0.004404732491821051,
+            0.016575198620557785,
+            0.017398692667484283,
+            -0.0053000422194600105,
+            0.020510731264948845,
+            0.0064634657464921474,
+            -0.0066214618273079395,
+            0.027175283059477806,
+            -0.00019285666348878294,
+            -0.006540069822221994,
+            -0.021334225311875343,
+            -0.003413667669519782,
+            0.019725540652871132,
+            -0.010236214846372604,
+            -0.0010856229346245527,
+            0.010734140872955322,
+            0.008479109965264797,
+            -2.2947548131924123e-05,
+            0.01765723153948784,
+            -0.009441448375582695,
+            0.0009192485013045371,
+            0.007315685972571373,
+            0.004950535949319601,
+            0.011232067830860615,
+            0.01174914464354515,
+            0.01740826852619648,
+            0.003947502002120018,
+            0.0023304384667426348,
+            -0.0037583857774734497,
+            0.004866750445216894,
+            0.011672540567815304,
+            0.03675078973174095,
+            0.007861489430069923,
+            0.007124175783246756,
+            -0.007176841143518686,
+            -0.004009742755442858,
+            0.01020748820155859,
+            -0.013415282592177391,
+            -0.013242923654615879,
+            0.015119723044335842,
+            0.005170772783458233,
+            0.032288603484630585,
+            -0.0121608916670084,
+            0.009996827691793442,
+            0.025585750117897987,
+            -0.008368991315364838,
+            -0.004658483434468508,
+            -0.004842811729758978,
+            0.0017319695325568318,
+            0.0024896312970668077,
+            0.007827975787222385,
+            0.0018037857953459024,
+            0.020587336272001266,
+            -0.009666472673416138,
+            -0.031771525740623474,
+            0.028075380250811577,
+            0.013491886667907238,
+            0.017398692667484283,
+            0.02164064161479473,
+            0.0079955467954278,
+            0.012677968479692936,
+            0.00971435010433197,
+            0.005893723573535681,
+            0.00416773883625865,
+            0.01609642431139946,
+            0.0073539880104362965,
+            -0.01767638325691223,
+            0.02054903469979763,
+            0.015483591705560684,
+            0.010753292590379715,
+            -0.02951170690357685,
+            -0.01116503868252039,
+            0.012936507351696491,
+            -0.01203641016036272,
+            -0.01061923522502184,
+            0.009929798543453217,
+            -0.011892776936292648,
+            0.005170772783458233,
+            0.015158024616539478,
+            0.010025554336607456,
+            0.0014123870059847832,
+            -0.02309611812233925,
+            -0.016489019617438316,
+            0.005496339872479439,
+            0.0021341408137232065,
+            0.012563062831759453,
+            0.0126396669074893,
+            0.017226334661245346,
+            -0.004950535949319601,
+            -0.00040037580765783787,
+            -0.017752986401319504,
+            -0.009580292738974094,
+            -0.007784885820001364,
+            0.02752000093460083,
+            -0.02164064161479473,
+            -0.0019330551149323583,
+            0.004105497617274523,
+            -0.02169809490442276,
+            0.0006218093913048506,
+            0.00840250588953495,
+            0.0025949617847800255,
+            -0.005357495043426752,
+            -0.008919582702219486,
+            0.003964259289205074,
+            -0.010753292590379715,
+            0.01414302084594965,
+            0.0011592345545068383,
+            -0.018729688599705696,
+            0.008368991315364838,
+            0.0013776757987216115,
+            -0.01212258916348219,
+            0.019629785791039467,
+            0.01852860301733017,
+            -0.007823187857866287,
+            0.005118107423186302,
+            0.041634295135736465,
+            0.0024992069229483604,
+            -0.022770551964640617,
+            -0.007545498199760914,
+            0.005323980934917927,
+            0.026102827861905098,
+            0.006066082511097193,
+            0.00822535902261734,
+            -0.0002223312621936202,
+            0.0030402229167521,
+            0.0017738622846081853,
+            -0.007406653370708227,
+            0.016211330890655518,
+            -0.00697575556114316,
+            0.004349673166871071,
+            0.014545192010700703,
+            -0.00877116248011589,
+            0.002016840735450387,
+            -0.01866265945136547,
+            0.0021233682055026293,
+            -0.025183578953146935,
+            -0.009546778164803982,
+            -0.003923563286662102,
+            -0.010264941491186619,
+            -0.008340264670550823,
+            0.0025303272996097803,
+            -0.011375700123608112,
+            -0.017398692667484283,
+            -0.010571357794106007,
+            0.01945742592215538,
+            -0.02066393941640854,
+            -0.0008234934648498893,
+            0.0065592210739851,
+            0.019189313054084778,
+            0.012151315808296204,
+            0.0029037718195468187,
+            0.0073539880104362965,
+            0.01561764907091856,
+            0.0017020460218191147,
+            -0.02188960462808609,
+            -0.027692360803484917,
+            0.02424517832696438,
+            0.001624245080165565,
+            0.0013681002892553806,
+            0.001211899914778769,
+            -0.011825748719274998,
+            -0.00985319446772337,
+            0.009048852138221264,
+            0.0006984134088270366,
+            -0.018183885142207146,
+            0.015474015846848488,
+            -0.002918135141953826,
+            -0.0017726654186844826,
+            -0.0031431596726179123,
+            0.014066416770219803,
+            0.014631371945142746,
+            0.0044095199555158615,
+            0.0055490052327513695,
+            0.005683062132447958,
+            0.012065136805176735,
+            0.010580932721495628,
+            -0.019878748804330826,
+            -0.025183578953146935,
+            0.019198887050151825,
+            -0.008459959179162979,
+            0.016852889209985733,
+            0.01382702961564064,
+            0.013635518960654736,
+            -0.028324345126748085,
+            -0.0037320530973374844,
+            -0.024436689913272858,
+            0.004550758749246597,
+            0.0008234934648498893,
+            -0.0019940990023314953,
+            -0.002559053711593151,
+            1.956245796463918e-05,
+            -0.021774698048830032,
+            0.006602310575544834,
+            0.025164427235722542,
+            -0.003890048945322633,
+            -0.011308671906590462,
+            0.02874566614627838,
+            -0.028918026015162468,
+            0.002788865938782692,
+            0.023076968267560005,
+            -0.004907446447759867,
+            0.07740838080644608,
+            0.008728072978556156,
+            0.0031670983880758286,
+            -0.015110147185623646,
+            0.0053000422194600105,
+            0.014487739652395248,
+            0.00834984052926302,
+            0.013817453756928444,
+            -0.00955156609416008,
+            -0.015158024616539478,
+            -0.008282812312245369,
+            0.01187362615019083,
+            0.006554433144629002,
+            -0.0031216146890074015,
+            0.002810410689562559,
+            0.0005098956753499806,
+            -0.010963953100144863,
+            -0.011739569716155529,
+            0.024111121892929077,
+            -0.025777259841561317,
+            -0.00871849711984396,
+            -0.028132833540439606,
+            -0.01366424560546875,
+            -0.00023789145052433014,
+            0.012754572555422783,
+            0.011011830531060696,
+            -0.022119417786598206,
+            -0.013606793247163296,
+            -0.020510731264948845,
+            -0.0020635214168578386,
+            0.018576480448246002,
+            0.0015536256833001971,
+            -0.000921043916605413,
+            -0.002472874242812395,
+            -0.001449492061510682,
+            0.009130244143307209,
+            -0.01295565813779831,
+            0.014344106428325176,
+            -0.006200139410793781,
+            0.0027649272233247757,
+            0.0038014755118638277,
+            0.00733004929497838,
+            0.01343443337827921,
+            -0.0034783021546900272,
+            0.006324620917439461,
+            -0.010753292590379715,
+            0.02361319586634636,
+            -0.01545486506074667,
+            -0.0031240086536854506,
+            -0.0015739736845716834,
+            0.008440807461738586,
+            -0.007325261365622282,
+            0.006760306656360626,
+            -0.00854135025292635,
+            -0.009269089438021183,
+            0.014468587934970856,
+            0.025566598400473595,
+            -0.02399621531367302,
+            0.004878719802945852,
+            -0.022770551964640617,
+            -0.0006918302387930453,
+            -0.007325261365622282,
+            -0.006425164174288511,
+            -0.002341211074963212,
+            -0.0401788204908371,
+            -0.005319193005561829,
+            0.012314099818468094,
+            0.0026284761261194944,
+            -0.01634538732469082,
+            0.009111093357205391,
+            -0.002007265342399478,
+            0.003736840793862939,
+            -0.03994900733232498,
+            0.017130577936768532,
+            0.025068672373890877,
+            -0.015129297971725464,
+            0.007253445219248533,
+            0.005845845676958561,
+            -0.007698706351220608,
+            0.0005371259758248925,
+            0.004426277242600918,
+            0.002757745562121272,
+            -0.02880311943590641,
+            0.020434128120541573,
+            0.018375394865870476,
+            0.007579012308269739,
+            0.004888295195996761,
+            -0.008929158560931683,
+            0.002621294464915991,
+            0.00989149697124958,
+            0.005544217303395271,
+            0.02405366860330105,
+            0.0031431596726179123,
+            -0.0006415588431991637,
+            0.0016374114202335477,
+            -0.0037224777042865753,
+            0.009767015464603901,
+            -0.005970327649265528,
+            0.005740515422075987,
+            -0.017877468839287758,
+            -0.0043329158797860146,
+            -0.0018349061720073223,
+            -0.004172526299953461,
+            -0.007794461213052273,
+            -0.001229255460202694,
+            -0.011844899505376816,
+            0.021410830318927765,
+            0.007377926725894213,
+            0.014210049994289875,
+            -0.01603897102177143,
+            0.0032556718215346336,
+            -0.021334225311875343,
+            -0.0036099653225392103,
+            -0.008311538025736809,
+            0.005946388468146324,
+            0.04404732212424278,
+            0.010006402619183064,
+            0.016661379486322403,
+            0.00033514268579892814,
+            0.020070258527994156,
+            -0.0066884905099868774,
+            -0.0055585806258022785,
+            -0.00047518443898297846,
+            -0.003830202156677842,
+            -0.0032867921981960535,
+            -0.0016050940612331033,
+            -0.008570076897740364,
+            -0.008321113884449005,
+            0.006678914651274681,
+            0.007890216074883938,
+            0.00667412718757987,
+            -0.0214491318911314,
+            0.01830836571753025,
+            -0.0007079889182932675,
+            -0.006037355866283178,
+            0.019763842225074768,
+            -0.0015620043268427253,
+            -0.002276576356962323,
+            -0.010418149642646313,
+            0.008258873596787453,
+            -0.0024082395248115063,
+            0.0469965785741806,
+            -0.009034489281475544,
+            0.02188960462808609,
+            -0.006439527031034231,
+            0.0023771191481500864,
+            0.003736840793862939,
+            0.0009264301043003798,
+            -0.006745943333953619,
+            -0.0021125958301126957,
+            -0.015541044995188713,
+            0.0056016705930233,
+            -0.015215477906167507,
+            0.012342826463282108,
+            -0.016795435920357704,
+            0.00372726540081203,
+            0.02361319586634636,
+            -0.0050558666698634624,
+            -0.00902012549340725,
+            0.0036913573276251554,
+            0.006587947718799114,
+            -0.0028271679766476154,
+            -0.02106611058115959,
+            0.013635518960654736,
+            -0.00193664594553411,
+            0.017427420243620872,
+            -0.007516771554946899,
+            -0.010963953100144863,
+            0.021104414016008377,
+            0.010092582553625107,
+            0.025049520656466484,
+            0.017312513664364815,
+            0.018863745033740997,
+            -0.001241224817931652,
+            0.014966514892876148,
+            -0.020721392706036568,
+            0.002918135141953826,
+            -0.012668393552303314,
+            -0.01989790052175522,
+            -0.0028175923507660627,
+            -0.004445428028702736,
+            0.01116503868252039,
+            -0.008546138182282448,
+            -0.010954378172755241,
+            0.007899791933596134,
+            -0.023345081135630608,
+            0.00324849016033113,
+            -0.008301963098347187,
+            0.017839165404438972,
+            0.0035477245692163706,
+            -0.012658817693591118,
+            -0.008795101195573807,
+            -0.006841698195785284,
+            -0.01174914464354515,
+            -0.020568184554576874,
+            0.03008623793721199,
+            0.01636453904211521,
+            0.015435714274644852,
+            -0.005678274668753147,
+            -0.002378316130489111,
+            -0.008689770475029945,
+            -0.0036267226096242666,
+            0.0008923173882067204,
+            -0.02822858840227127
+          ],
+          "title": "A Switch for Attention"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "e453f0b2-d6f0-42f8-a7f8-2c465d813d77",
+      "type": "child",
+      "source": {
+        "id": "65742957-6bda-446d-a61a-0fe20313c760",
+        "properties": {
+          "page_content": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity\n\nWilliam Fedus∗ liamfedus@google.com Barret Zoph∗ barretzoph@google.com Noam Shazeer noam@google.com Google, Mountain View, CA 94043, USA Editor: Alexander Clark\n\nAbstract\n\nIn deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) models defy this and instead select different parameters for each incoming example. The result is a sparsely-activated model—with an outrageous number of parameters—but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs, and training instability. We address these with the introduction of the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques mitigate the instabilities, and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large (Raffel et al., 2019) to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the \"Colossal Clean Crawled Corpus\", and achieve a 4x speedup over the T5-XXL model.12 Keywords: mixture-of-experts, natural language processing, sparsity, large-scale machine learning, distributed computing arXiv:2101.03961v3 [cs.LG] 16 Jun 2022 Contents\n\n1 Introduction 3 2 Switch Transformer 4 2.1 Simplifying Sparse Routing 5 2.2 Efficient Sparse Routing 6 2.3 Putting It All Together: The Switch Transformer 8 2.4 Improved Training and Fine-Tuning Techniques 8 3 Scaling Properties 11 3.1 Scaling Results on a Step-Basis 12 3.2 Scaling Results on a Time-Basis 13 3.3 Scaling Versus a Larger Dense Model 13 4 Downstream Results 14 4.1 Fine-Tuning 14 4.2 Distillation 16 4.3 Multilingual Learning 17 5 Designing Models with Data, Model, and Expert-Parallelism 18 5.1 Data Parallelism 20 5.2 Model Parallelism 20 5.3 Model and Data Parallelism 21 5.4 Expert and Data Parallelism 22 5.5 Expert, Model and Data Parallelism 22 5.6 Towards Trillion Parameter Models 22 6 Related Work 24 7 Discussion 25 8 Future Work 26 9 Conclusion 27 A Switch for Attention 27 B Preventing Token Dropping with No-Token-Left-Behind 29\n\nC Encouraging Exploration Across Experts 29 D Switch Transformers in Lower Compute Regimes 29 E Relation of Upstream to Downstream Model Performance 32 F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/switch_transformers.md"
+          },
+          "summary": "The Switch Transformer is a sparsely-activated model that simplifies the Mixture of Experts (MoE) approach, allowing for efficient scaling to trillion-parameter models with reduced communication and computational costs. It achieves significant speedups in pre-training and fine-tuning by activating only a subset of neural network weights for each input, maintaining a constant computational cost. The model demonstrates superior scaling properties and improved performance across various natural language processing tasks, including multilingual settings. It also allows for effective distillation into smaller dense models, preserving a portion of the quality gains. The architecture is designed to maximize parameter count while keeping floating point operations per example constant, making it suitable for both large-scale and smaller-scale applications.",
+          "headlines": {
+            "1 Introduction": [],
+            "2 Switch Transformer": [
+              "2.1 Simplifying Sparse Routing",
+              "2.2 Efficient Sparse Routing",
+              "2.3 Putting It All Together: The Switch Transformer",
+              "2.4 Improved Training and Fine-Tuning Techniques"
+            ],
+            "3 Scaling Properties": [
+              "3.1 Scaling Results on a Step-Basis",
+              "3.2 Scaling Results on a Time-Basis",
+              "3.3 Scaling Versus a Larger Dense Model"
+            ],
+            "4 Downstream Results": [
+              "4.1 Fine-Tuning",
+              "4.2 Distillation",
+              "4.3 Multilingual Learning"
+            ],
+            "5 Designing Models with Data, Model, and Expert-Parallelism": [
+              "5.1 Data Parallelism",
+              "5.2 Model Parallelism",
+              "5.3 Model and Data Parallelism",
+              "5.4 Expert and Data Parallelism",
+              "5.5 Expert, Model and Data Parallelism",
+              "5.6 Towards Trillion Parameter Models"
+            ],
+            "6 Related Work": [],
+            "7 Discussion": [],
+            "8 Future Work": [],
+            "9 Conclusion": [],
+            "A Switch for Attention": [],
+            "B Preventing Token Dropping with No-Token-Left-Behind": [],
+            "C Encouraging Exploration Across Experts": [],
+            "D Switch Transformers in Lower Compute Regimes": [],
+            "E Relation of Upstream to Downstream Model Performance": [],
+            "F Pseudo Code for Switch Transformers": []
+          },
+          "summary_embedding": [
+            -0.0060447100549936295,
+            -0.00390331051312387,
+            -0.01784370467066765,
+            -0.0021820589900016785,
+            0.013227885589003563,
+            0.011338187381625175,
+            0.0165580902248621,
+            0.024240797385573387,
+            -0.005986624862998724,
+            0.03596621751785278,
+            -0.004852031823247671,
+            -0.0068811336532235146,
+            -0.003574162255972624,
+            0.014242436736822128,
+            0.011578272096812725,
+            -0.033302053809165955,
+            -0.024519605562090874,
+            0.01050176378339529,
+            0.016093410551548004,
+            -0.006795942317694426,
+            0.007473600562661886,
+            -0.030513975769281387,
+            -0.06124480068683624,
+            -0.006838538218289614,
+            -0.0026099516544491053,
+            0.01029265858232975,
+            0.0010329445358365774,
+            -0.010873507708311081,
+            -0.053469158709049225,
+            0.023311438038945198,
+            0.025294071063399315,
+            0.005092116538435221,
+            -0.029584616422653198,
+            0.0009361362317577004,
+            -0.05161044001579285,
+            -0.04448534920811653,
+            0.0379488542675972,
+            0.012561844661831856,
+            -0.04364892467856407,
+            0.039249956607818604,
+            -0.004875265993177891,
+            -0.0022440163884311914,
+            -0.021499186754226685,
+            -0.0034909071400761604,
+            -0.007039899472147226,
+            -0.009851212613284588,
+            -0.02874819189310074,
+            0.0014037203509360552,
+            0.021886419504880905,
+            -0.0056962003000080585,
+            0.023264970630407333,
+            -0.024271776899695396,
+            0.01936165988445282,
+            0.04240977764129639,
+            -0.06957805901765823,
+            0.024659009650349617,
+            0.017549408599734306,
+            0.05120771750807762,
+            0.06511713564395905,
+            0.018339363858103752,
+            0.021669568493962288,
+            -0.014141756109893322,
+            -0.017766257748007774,
+            -0.011872569099068642,
+            0.004673904739320278,
+            -0.014970434829592705,
+            -0.01891246810555458,
+            0.008449427783489227,
+            -0.020461400970816612,
+            0.008209343999624252,
+            -0.05576157942414284,
+            -0.014730350114405155,
+            -0.034200433641672134,
+            -0.01140014547854662,
+            0.017642343416810036,
+            -0.018215449526906013,
+            0.030126741155982018,
+            0.02997184917330742,
+            0.002638994250446558,
+            0.0007822110201232135,
+            0.021049996837973595,
+            0.006509389728307724,
+            -0.0053438181057572365,
+            0.012321759946644306,
+            0.028159597888588905,
+            -0.01450575515627861,
+            -0.010106786154210567,
+            -0.03454120084643364,
+            0.01861817203462124,
+            0.01589204929769039,
+            -0.04194509983062744,
+            -0.012569589540362358,
+            -0.013824224472045898,
+            0.032341714948415756,
+            -0.009549170732498169,
+            0.04098476096987724,
+            -0.010083552449941635,
+            0.001383390510454774,
+            -0.031149037182331085,
+            -0.009463978931307793,
+            0.017069239169359207,
+            0.006172496825456619,
+            -0.016589069738984108,
+            -0.0012846461031585932,
+            0.016170857474207878,
+            0.006617815233767033,
+            -0.030142230913043022,
+            0.04488807171583176,
+            0.023497309535741806,
+            0.055296897888183594,
+            0.05579255893826485,
+            -0.037391237914562225,
+            -0.002445377642288804,
+            0.0344172865152359,
+            -0.0228312686085701,
+            -0.015040136873722076,
+            -0.034014564007520676,
+            -0.01768881268799305,
+            0.035935238003730774,
+            -0.0318460576236248,
+            0.01556677371263504,
+            0.028066661208868027,
+            -0.025712283328175545,
+            0.003488970920443535,
+            -0.03943582624197006,
+            0.003622566582635045,
+            0.01220559049397707,
+            0.006095050368458033,
+            -0.0026293133851140738,
+            -0.013615119270980358,
+            0.008906363509595394,
+            0.003055269829928875,
+            0.00800798274576664,
+            -0.04615819454193115,
+            0.018478767946362495,
+            -0.00454224506393075,
+            0.04033420979976654,
+            0.042502716183662415,
+            -0.02730768360197544,
+            -0.016743963584303856,
+            0.008557853288948536,
+            0.012104909867048264,
+            0.04188314080238342,
+            -0.007287728600203991,
+            -0.00121784838847816,
+            -0.0007836631848476827,
+            0.007322579622268677,
+            0.047552235424518585,
+            0.004646798130124807,
+            0.0020697612781077623,
+            -0.027230238541960716,
+            0.014978179708123207,
+            -0.019733402878046036,
+            0.027648448944091797,
+            0.023373395204544067,
+            0.02942972257733345,
+            -0.04048910364508629,
+            0.017518429085612297,
+            -0.011818356812000275,
+            -0.005568413063883781,
+            -0.002726121572777629,
+            -0.02856232039630413,
+            -0.0074426219798624516,
+            0.033983584493398666,
+            0.015365412458777428,
+            0.012956822291016579,
+            0.03271345794200897,
+            0.0012052633101120591,
+            -0.042502716183662415,
+            0.03804178908467293,
+            0.024550583213567734,
+            0.04640602692961693,
+            0.023466331884264946,
+            0.010796061716973782,
+            0.006571347359567881,
+            0.013375034555792809,
+            -0.019377149641513824,
+            0.015907539054751396,
+            0.02560385875403881,
+            0.009750531986355782,
+            -0.03167567402124405,
+            -0.0009501734166406095,
+            0.01178737822920084,
+            -0.004546117503196001,
+            0.03701949119567871,
+            0.04104671627283096,
+            0.020569827407598495,
+            0.05030933395028114,
+            0.0020155487582087517,
+            0.017936641350388527,
+            0.005936284549534321,
+            0.02820606529712677,
+            0.030467506498098373,
+            -0.005285732913762331,
+            0.005076627247035503,
+            -0.01136916596442461,
+            -0.03385967016220093,
+            0.014110777527093887,
+            0.01561324205249548,
+            -0.008077683858573437,
+            -0.01697630248963833,
+            0.006033093202859163,
+            0.009169681929051876,
+            -0.01795213110744953,
+            -0.017456471920013428,
+            0.005971135571599007,
+            0.018866000697016716,
+            -0.013576395809650421,
+            0.024689987301826477,
+            -0.04442339017987251,
+            0.029119934886693954,
+            -0.033209118992090225,
+            -0.003384418087080121,
+            -0.010641167871654034,
+            -0.0030843124259263277,
+            0.0041666291654109955,
+            0.011314953677356243,
+            0.013406013138592243,
+            0.002042655134573579,
+            -0.013909416273236275,
+            0.024519605562090874,
+            0.018447788432240486,
+            0.012073930352926254,
+            -0.03148980438709259,
+            -0.014397329650819302,
+            -0.0015673263696953654,
+            -0.008743725717067719,
+            -0.034138478338718414,
+            0.026966920122504234,
+            0.014048820361495018,
+            -0.03844451159238815,
+            0.017797237262129784,
+            0.029538147151470184,
+            0.014420564286410809,
+            -0.004790074657648802,
+            -0.0463130883872509,
+            0.04045812413096428,
+            -0.030436528846621513,
+            -0.033333033323287964,
+            -0.010478530079126358,
+            -0.0003146269591525197,
+            0.02894955314695835,
+            -0.0064358157105743885,
+            -0.050835974514484406,
+            0.01184933539479971,
+            0.020631784573197365,
+            0.00010763872705865651,
+            0.005731051322072744,
+            -0.019593998789787292,
+            0.00028631053282879293,
+            -0.03395260497927666,
+            0.0090844901278615,
+            0.026750069111585617,
+            0.008867640048265457,
+            -0.016418687999248505,
+            0.03351890295743942,
+            -0.0004121613164898008,
+            -0.018695618957281113,
+            -0.003889757441356778,
+            0.006885006092488766,
+            0.034912943840026855,
+            0.004836542531847954,
+            -0.02946070022881031,
+            0.05665995925664902,
+            -0.02100352756679058,
+            0.021344292908906937,
+            0.05631919577717781,
+            -0.019439106807112694,
+            -0.008472662419080734,
+            -0.002065889071673155,
+            0.0026196325197815895,
+            -0.0260530486702919,
+            -0.008937342092394829,
+            0.032341714948415756,
+            0.041728246957063675,
+            -0.018742086365818977,
+            -0.001398879918269813,
+            0.006780453026294708,
+            0.009425255469977856,
+            0.004948840010911226,
+            0.02692045085132122,
+            -0.012128143571317196,
+            -0.014559967443346977,
+            0.05198218300938606,
+            0.028314489871263504,
+            -0.012081675231456757,
+            -0.0164031982421875,
+            0.019888296723365784,
+            -0.024844881147146225,
+            -0.020244551822543144,
+            -0.02431824430823326,
+            -0.020414933562278748,
+            -0.0042518205009400845,
+            0.009525936096906662,
+            0.037484172731637955,
+            -0.03968365862965584,
+            0.012747716158628464,
+            0.021700548008084297,
+            0.02981695532798767,
+            -0.019532041624188423,
+            -0.05910727381706238,
+            -0.0016128262504935265,
+            0.03296129032969475,
+            -0.015458349138498306,
+            -0.049379974603652954,
+            0.015164051204919815,
+            0.0260530486702919,
+            0.05957195535302162,
+            -0.001461805310100317,
+            -0.014954946003854275,
+            0.061926331371068954,
+            0.03026614524424076,
+            -0.004383479710668325,
+            -0.009494957514107227,
+            -0.0127864396199584,
+            0.003396035172045231,
+            0.00465454300865531,
+            -0.03364282101392746,
+            -0.041170634329319,
+            -0.011841590516269207,
+            -0.010284913703799248,
+            0.008108663372695446,
+            -0.040922801941633224,
+            0.03531566634774208,
+            -0.021917399019002914,
+            0.010950954630970955,
+            0.015473837964236736,
+            0.037391237914562225,
+            -0.021344292908906937,
+            -0.01284839678555727,
+            -0.01572941243648529,
+            -0.024968795478343964,
+            -0.014010096900165081,
+            0.021917399019002914,
+            0.007682706229388714,
+            3.4699722164077684e-05,
+            0.010075807571411133,
+            -0.024659009650349617,
+            -0.051052823662757874,
+            0.0291044469922781,
+            0.0015605497173964977,
+            -0.015938518568873405,
+            -0.004623564425855875,
+            -0.01977987214922905,
+            -0.01122201792895794,
+            -0.012608313001692295,
+            -0.039249956607818604,
+            0.021111954003572464,
+            -0.01135367713868618,
+            0.015876561403274536,
+            0.0009250032599084079,
+            0.003659353591501713,
+            0.028624277561903,
+            0.01116006076335907,
+            0.0051811798475682735,
+            -0.03255856782197952,
+            -0.018277406692504883,
+            0.010122275911271572,
+            -0.00561100896447897,
+            -0.018339363858103752,
+            -0.013351799920201302,
+            -0.05037129297852516,
+            -0.03305422514677048,
+            -0.008054450154304504,
+            -0.05415068939328194,
+            0.0020697612781077623,
+            -0.014296649023890495,
+            -0.02232012152671814,
+            -0.005990497302263975,
+            0.03107159025967121,
+            -0.003235333366319537,
+            0.025294071063399315,
+            0.0076594725251197815,
+            0.037360258400440216,
+            0.040117356926202774,
+            0.008426194079220295,
+            0.03065337985754013,
+            -0.0029991210903972387,
+            -0.016867877915501595,
+            0.05486319959163666,
+            -0.01881953328847885,
+            0.013529927469789982,
+            -0.008604321628808975,
+            0.008604321628808975,
+            0.03299226611852646,
+            -0.003794885240495205,
+            0.02007416822016239,
+            -0.0311955064535141,
+            -0.07998688519001007,
+            0.01637221872806549,
+            0.028159597888588905,
+            -0.06418777257204056,
+            0.014691627584397793,
+            -0.020275529474020004,
+            0.02052335813641548,
+            0.02411688305437565,
+            -0.0009114501299336553,
+            -0.014064309187233448,
+            0.005595519673079252,
+            0.0020833145827054977,
+            -0.01456771232187748,
+            -0.002691270550712943,
+            -0.009417511522769928,
+            0.008774704299867153,
+            -0.017007282003760338,
+            -0.013406013138592243,
+            -0.034231413155794144,
+            0.010873507708311081,
+            0.01233724970370531,
+            -0.003969140350818634,
+            0.0011500825639814138,
+            -0.03277541697025299,
+            -0.007876322604715824,
+            -0.02145271934568882,
+            0.03382869064807892,
+            0.012058441527187824,
+            -0.030699847266077995,
+            -0.007814365439116955,
+            0.015264731831848621,
+            -0.007876322604715824,
+            0.0008906363509595394,
+            0.024132372811436653,
+            0.02103450708091259,
+            0.0028364830650389194,
+            -0.0009700191440060735,
+            0.02438020147383213,
+            0.007427132688462734,
+            -0.01720864325761795,
+            0.028794659301638603,
+            0.014288905076682568,
+            0.060160547494888306,
+            -0.03271345794200897,
+            0.001297231181524694,
+            -0.04135650396347046,
+            -0.031118059530854225,
+            0.006408709101378918,
+            0.014350862242281437,
+            0.007597515359520912,
+            -0.028252532705664635,
+            0.005978880450129509,
+            0.02004319056868553,
+            0.01469937153160572,
+            0.009704063646495342,
+            0.0037580979987978935,
+            -0.036740683019161224,
+            -0.000691695255227387,
+            -0.011214273050427437,
+            -0.0038781403563916683,
+            0.035656433552503586,
+            -0.008186109364032745,
+            -0.05151750147342682,
+            -0.0006418389966711402,
+            -0.03255856782197952,
+            -0.0353776253759861,
+            0.021313315257430077,
+            -0.00839521549642086,
+            -0.0039207362569868565,
+            -0.013986863195896149,
+            -0.000537770101800561,
+            -0.010524998418986797,
+            0.027509044855833054,
+            0.028345469385385513,
+            0.022490503266453743,
+            -0.05384090170264244,
+            0.009479468688368797,
+            0.00023766436788719147,
+            -0.05817791447043419,
+            0.0009709871956147254,
+            -0.007117345929145813,
+            0.011191039346158504,
+            -0.020213572308421135,
+            -0.02414786070585251,
+            0.011500826105475426,
+            0.03392162546515465,
+            0.014304393902420998,
+            0.0321558453142643,
+            -0.013367289677262306,
+            -0.09250226616859436,
+            0.018292896449565887,
+            -0.008689512498676777,
+            0.04126356914639473,
+            0.00865078903734684,
+            -0.004278926644474268,
+            -0.015845581889152527,
+            -0.00865078903734684,
+            -0.006943091284483671,
+            0.01283290795981884,
+            0.004530628211796284,
+            -0.005626498255878687,
+            0.0016631665639579296,
+            0.013708055019378662,
+            -0.005967263597995043,
+            0.007481344975531101,
+            0.003651608945801854,
+            -0.013189162127673626,
+            -0.06059424951672554,
+            -0.04055105894804001,
+            -0.018540725111961365,
+            0.050835974514484406,
+            -0.0069159846752882,
+            -0.046684831380844116,
+            -0.010850274004042149,
+            0.05058814212679863,
+            0.012259802781045437,
+            -0.0334569476544857,
+            -0.0007841471815481782,
+            0.0050882440991699696,
+            0.015458349138498306,
+            0.004546117503196001,
+            -0.027292195707559586,
+            0.02820606529712677,
+            0.0008412640891037881,
+            0.02300165221095085,
+            -0.018339363858103752,
+            -0.005312839522957802,
+            0.035780347883701324,
+            -0.024844881147146225,
+            0.023791607469320297,
+            0.0055219451896846294,
+            0.031629208475351334,
+            0.002952652983367443,
+            0.010060317814350128,
+            0.004704883322119713,
+            0.0186491496860981,
+            0.004275054205209017,
+            0.0276639387011528,
+            0.011462102644145489,
+            0.006428070832043886,
+            0.021623101085424423,
+            -0.009378788061439991,
+            -0.014513500034809113,
+            0.01885051093995571,
+            -0.0014695499558001757,
+            0.025232113897800446,
+            -0.020538847893476486,
+            0.002726121572777629,
+            -0.0016883367206901312,
+            -0.017967619001865387,
+            -0.040922801941633224,
+            -0.0007405834621749818,
+            0.021747015416622162,
+            -0.011671207845211029,
+            -0.0228312686085701,
+            -0.013715799897909164,
+            -0.01585332676768303,
+            0.0038490979932248592,
+            0.034169457852840424,
+            -0.0398695282638073,
+            0.00963436160236597,
+            -0.000830615172162652,
+            -0.03125746175646782,
+            -0.0239619892090559,
+            -0.01093546487390995,
+            0.014614180661737919,
+            -0.01652711257338524,
+            -0.021173911169171333,
+            -0.028469383716583252,
+            -0.008410705253481865,
+            0.006048582494258881,
+            -0.05065010115504265,
+            -0.004054331686347723,
+            0.019965743646025658,
+            0.0276639387011528,
+            -0.003099801717326045,
+            -0.02177799493074417,
+            0.0012468908680602908,
+            0.022351099178195,
+            0.02936776541173458,
+            0.014226946979761124,
+            0.027586491778492928,
+            -0.002178186783567071,
+            -0.0011675080750137568,
+            -0.0073148347437381744,
+            0.01307299267500639,
+            0.0011287847300991416,
+            0.036616768687963486,
+            0.0122210793197155,
+            -0.020306508988142014,
+            0.0013959755888208747,
+            -0.006431943271309137,
+            0.029181893914937973,
+            -0.02270735427737236,
+            -9.832092473516241e-05,
+            0.03531566634774208,
+            0.027741385623812675,
+            0.004933350719511509,
+            -0.0143121387809515,
+            -0.05142456665635109,
+            0.021592123433947563,
+            -0.01765783317387104,
+            0.031025122851133347,
+            0.016666516661643982,
+            0.003397971158847213,
+            -0.0034212053287774324,
+            0.012368228286504745,
+            -0.022087780758738518,
+            -0.004406713880598545,
+            -0.021142931655049324,
+            0.04389675334095955,
+            -0.043184246867895126,
+            0.008472662419080734,
+            0.029445212334394455,
+            0.011926782317459583,
+            -0.019082851707935333,
+            -0.005425137002021074,
+            0.029212871566414833,
+            -0.00162637943867594,
+            0.01308848150074482,
+            0.0058665829710662365,
+            -0.026935940608382225,
+            0.01202746294438839,
+            -0.031660184264183044,
+            0.02177799493074417,
+            -0.013436991721391678,
+            0.018447788432240486,
+            0.03342596814036369,
+            -0.0030397805385291576,
+            -0.0038239278364926577,
+            -0.016263794153928757,
+            -0.00969631876796484,
+            0.0662323608994484,
+            0.01054823212325573,
+            0.03726732358336449,
+            0.0018974426202476025,
+            -0.009773765690624714,
+            0.007969259284436703,
+            0.0063041565008461475,
+            -0.012329504825174809,
+            0.010494019836187363,
+            0.024178840219974518,
+            0.0012730291346088052,
+            -0.002412462839856744,
+            -0.019408127292990685,
+            -0.03438630700111389,
+            0.003727119415998459,
+            -0.030932186171412468,
+            0.022691864520311356,
+            0.009208405390381813,
+            0.017456471920013428,
+            0.008240322582423687,
+            -0.015907539054751396,
+            0.03200095146894455,
+            0.05461536720395088,
+            0.0019768255297094584,
+            0.013375034555792809,
+            0.007938280701637268,
+            0.007036027032881975,
+            -0.011338187381625175,
+            0.04702559858560562,
+            0.000662652775645256,
+            0.005169562995433807,
+            0.022536970674991608,
+            0.02145271934568882,
+            0.022939693182706833,
+            0.002304037567228079,
+            -0.011694442480802536,
+            0.010478530079126358,
+            -0.0006302220281213522,
+            0.004445436876267195,
+            -0.010695381090044975,
+            -0.027865299955010414,
+            0.007492962293326855,
+            0.01302652433514595,
+            0.022598929703235626,
+            0.021592123433947563,
+            -0.03178409859538078,
+            -0.01855621486902237,
+            0.03364282101392746,
+            -0.02997184917330742,
+            0.03612111136317253,
+            -0.0014424435794353485,
+            -0.023807097226381302,
+            -0.0058356039226055145,
+            0.016929835081100464,
+            0.02656419761478901,
+            -0.029181893914937973,
+            0.01804506592452526,
+            0.029693040996789932,
+            -0.0074426219798624516,
+            0.0005939188995398581,
+            0.012112653814256191,
+            0.02447313815355301,
+            -0.009293596260249615,
+            -0.0024027819745242596,
+            -0.0013320822035893798,
+            -0.011926782317459583,
+            -0.018370341509580612,
+            -0.0044725434854626656,
+            -0.022629907354712486,
+            0.005564541090279818,
+            -0.007365175057202578,
+            -0.003481226274743676,
+            -0.008581086993217468,
+            0.01154729351401329,
+            0.001979729626327753,
+            -0.019857317209243774,
+            -0.022335609421133995,
+            0.01652711257338524,
+            0.03887821361422539,
+            -0.018215449526906013,
+            -0.02106548473238945,
+            0.024442158639431,
+            -0.01759587600827217,
+            -0.01601596549153328,
+            0.036678727716207504,
+            0.013739033602178097,
+            0.007736918982118368,
+            -0.03358086198568344,
+            -0.030049296095967293,
+            -0.03274443745613098,
+            0.006296411622315645,
+            0.03785591572523117,
+            -0.0028752065263688564,
+            0.02569679357111454,
+            0.03701949119567871,
+            0.0057891360484063625,
+            -0.028934063389897346,
+            -0.014823286794126034,
+            -0.045104920864105225,
+            0.060160547494888306,
+            0.0011481463443487883,
+            0.05161044001579285,
+            0.016294771805405617,
+            -0.01456771232187748,
+            0.00419760774821043,
+            -0.013251119293272495,
+            0.026192452758550644,
+            0.02280029095709324,
+            0.025929134339094162,
+            0.006582964211702347,
+            0.022505993023514748,
+            -0.0037522895727306604,
+            -0.012197845615446568,
+            0.007307090330868959,
+            0.014536733739078045,
+            0.0002497654058970511,
+            -0.011392400600016117,
+            0.015783624723553658,
+            -0.01307299267500639,
+            -0.011361422017216682,
+            0.023295948281884193,
+            0.01368482131510973,
+            -0.012344993650913239,
+            0.014064309187233448,
+            -0.008565598167479038,
+            0.01239920686930418,
+            -0.023915521800518036,
+            0.022691864520311356,
+            -0.03262052312493324,
+            0.009587894193828106,
+            0.0015053689712658525,
+            -0.0073148347437381744,
+            0.024550583213567734,
+            -0.0027454833034425974,
+            -0.05071205645799637,
+            0.03550153970718384,
+            -0.008015726692974567,
+            0.0037213109899312258,
+            0.006381602957844734,
+            0.011392400600016117,
+            -0.0012623801594600081,
+            0.03807276859879494,
+            0.01239920686930418,
+            0.015969496220350266,
+            0.02199484594166279,
+            0.011911292560398579,
+            -0.008433938957750797,
+            -0.004096927121281624,
+            -0.0016137943603098392,
+            -0.01784370467066765,
+            0.025944624096155167,
+            0.0018674320308491588,
+            0.03962169960141182,
+            -0.018168980255723,
+            -0.02300165221095085,
+            0.028066661208868027,
+            0.007597515359520912,
+            -0.02351279929280281,
+            -0.0017377089243382215,
+            0.01202746294438839,
+            -0.009185170754790306,
+            0.027385130524635315,
+            0.01196550577878952,
+            0.019113829359412193,
+            -0.018060555681586266,
+            -0.004340883810073137,
+            -0.005072754807770252,
+            -0.015938518568873405,
+            0.02238207869231701,
+            0.020771188661456108,
+            0.011152315884828568,
+            -0.011880313977599144,
+            -0.01307299267500639,
+            0.020600805059075356,
+            0.035036858171224594,
+            -0.0002744515077210963,
+            0.005672966130077839,
+            0.017487451434135437,
+            0.002373739378526807,
+            -0.04470219835639,
+            0.016232814639806747,
+            -0.014025585725903511,
+            0.011214273050427437,
+            0.019593998789787292,
+            -0.021917399019002914,
+            -0.03196997195482254,
+            -0.011539548635482788,
+            -0.0414804182946682,
+            0.00493722315877676,
+            -0.0069159846752882,
+            0.013576395809650421,
+            -0.010176488198339939,
+            -0.0014831031439825892,
+            0.016666516661643982,
+            0.0038103745318949223,
+            0.02177799493074417,
+            -0.011678952723741531,
+            -0.012073930352926254,
+            -0.020213572308421135,
+            -0.020786678418517113,
+            0.004278926644474268,
+            -0.030405549332499504,
+            -0.004488032776862383,
+            0.021375272423028946,
+            -0.04783104360103607,
+            -0.03689557686448097,
+            0.02898053266108036,
+            -0.008503641001880169,
+            0.020755698904395103,
+            0.022955182939767838,
+            -0.015094350092113018,
+            -0.0008717587334103882,
+            -0.018695618957281113,
+            0.02258343994617462,
+            -0.007837600074708462,
+            0.026300877332687378,
+            -0.0027551641687750816,
+            0.0006292539183050394,
+            -0.02659517526626587,
+            0.020306508988142014,
+            0.005227648187428713,
+            0.012275291606783867,
+            -0.02390003204345703,
+            0.006815304048359394,
+            -0.03133491054177284,
+            -0.01784370467066765,
+            0.0180140882730484,
+            -0.011152315884828568,
+            0.027989214286208153,
+            -0.006060199346393347,
+            0.028438406065106392,
+            0.022722844034433365,
+            0.024751944467425346,
+            0.02676555886864662,
+            -0.02363671362400055,
+            0.008495896123349667,
+            0.03726732358336449,
+            0.00865078903734684,
+            -0.015876561403274536,
+            0.008170620538294315,
+            -0.04863648861646652,
+            -0.008062195032835007,
+            0.05030933395028114,
+            0.002127846470102668,
+            0.023141054436564445,
+            0.016294771805405617,
+            -0.002873270306736231,
+            -0.005688455421477556,
+            0.010269423946738243,
+            -0.04296739399433136,
+            -0.023172033950686455,
+            0.022629907354712486,
+            -0.021561143919825554,
+            6.54212328754511e-07,
+            0.03122648410499096,
+            0.012120398692786694,
+            -0.0009109660750254989,
+            -0.05108380317687988,
+            0.02100352756679058,
+            0.01178737822920084,
+            0.014443797990679741,
+            -0.010300402529537678,
+            0.026068538427352905,
+            0.01733255758881569,
+            0.038413532078266144,
+            0.012755461037158966,
+            -0.002679653698578477,
+            -0.03782493621110916,
+            0.009177426807582378,
+            -0.021685058251023293,
+            0.0011471783509477973,
+            0.0074503663927316666,
+            -0.02273833192884922,
+            -0.024782923981547356,
+            0.0043912241235375404,
+            0.004588713403791189,
+            0.011880313977599144,
+            0.022087780758738518,
+            0.009417511522769928,
+            -0.021204888820648193,
+            0.015404135920107365,
+            -0.05446047708392143,
+            -0.015450604259967804,
+            -0.023884542286396027,
+            0.02566581591963768,
+            0.020012211054563522,
+            0.010184233076870441,
+            0.005545179359614849,
+            0.010904486291110516,
+            -0.020988039672374725,
+            -0.009239383973181248,
+            0.020631784573197365,
+            0.011074868962168694,
+            -0.012840652838349342,
+            0.04113965481519699,
+            -0.00011126903700642288,
+            0.031210994347929955,
+            -0.02280029095709324,
+            0.002941036131232977,
+            -0.001241082325577736,
+            -0.03649285435676575,
+            0.009130958467721939,
+            0.01981084980070591,
+            0.02151467651128769,
+            -0.01861817203462124,
+            0.0024143988266587257,
+            0.01048627495765686,
+            0.004766840487718582,
+            0.038351576775312424,
+            0.027121812105178833,
+            -0.044206541031599045,
+            0.011663463898003101,
+            0.012995545752346516,
+            -0.009363298304378986,
+            0.048419639468193054,
+            -0.017518429085612297,
+            0.017673322930932045,
+            0.008240322582423687,
+            -0.01798310875892639,
+            0.02318752370774746,
+            -0.01469937153160572,
+            -0.013282098807394505,
+            0.00650551775470376,
+            0.009192915633320808,
+            -0.0001770381786627695,
+            -0.037577107548713684,
+            -0.017859194427728653,
+            -0.005719434004276991,
+            0.0038297362625598907,
+            0.01562873087823391,
+            0.013498948886990547,
+            0.009967382065951824,
+            0.005161818582564592,
+            -0.00945623405277729,
+            -0.0016089539276435971,
+            0.008124152198433876,
+            -0.0005532593932002783,
+            0.0019216446671634912,
+            -0.01926872320473194,
+            0.02698240801692009,
+            0.02572777308523655,
+            -0.0018354853382334113,
+            -0.04652994126081467,
+            -0.0036709706764668226,
+            0.0037077576853334904,
+            -0.017735280096530914,
+            -0.0035315665882080793,
+            -0.04519785940647125,
+            -0.01810702309012413,
+            0.0143121387809515,
+            -0.014614180661737919,
+            -0.0460033044219017,
+            0.009076746180653572,
+            -0.013886182568967342,
+            0.0063970922492444515,
+            -0.00932457484304905,
+            0.005762029904872179,
+            0.003651608945801854,
+            0.005680711008608341,
+            -0.005909178406000137,
+            0.05108380317687988,
+            -0.03466511517763138,
+            0.028051171451807022,
+            0.008565598167479038,
+            -0.012437930330634117,
+            -0.02261441759765148,
+            0.00887538492679596,
+            -0.044981006532907486,
+            -0.033209118992090225,
+            0.016743963584303856,
+            -0.01781272701919079,
+            0.011074868962168694,
+            0.06161654368042946,
+            0.007330324035137892,
+            -0.023342415690422058,
+            -0.01233724970370531,
+            -0.021049996837973595,
+            -0.04191412031650543,
+            0.003442503046244383,
+            0.005463860463351011,
+            -0.004948840010911226,
+            -0.03807276859879494,
+            -0.021716037765145302,
+            0.04615819454193115,
+            -0.012004229240119457,
+            0.012422440573573112,
+            0.01080380566418171,
+            -0.0019410063978284597,
+            -0.04860550910234451,
+            -0.021019017323851585,
+            0.0395597442984581,
+            0.021886419504880905,
+            -0.008426194079220295,
+            0.0028364830650389194,
+            -0.018277406692504883,
+            -0.003289545886218548,
+            0.006296411622315645,
+            -0.009107724763453007,
+            -0.03562545403838158,
+            0.02067825198173523,
+            0.0054832217283546925,
+            -0.023373395204544067,
+            -0.025805220007896423,
+            -0.03296129032969475,
+            0.009944148361682892,
+            0.02453509531915188,
+            -0.01968693546950817,
+            0.005823987070471048,
+            0.00027203132049180567,
+            -0.000881923595443368,
+            0.00042740863864310086,
+            -0.009959637187421322,
+            -0.019764382392168045,
+            -0.020151615142822266,
+            0.03268248215317726,
+            -0.0060059865936636925,
+            0.0074387495405972,
+            -0.03630698472261429,
+            -0.010974188335239887,
+            0.011322698555886745,
+            -0.002395037328824401,
+            -0.013452480547130108,
+            -0.009092235006392002,
+            -0.01739451475441456,
+            -0.009882191196084023,
+            -0.007876322604715824,
+            0.022660886868834496,
+            0.006168624851852655,
+            0.005409647710621357,
+            0.0023795480374246836,
+            -0.008581086993217468,
+            0.02898053266108036,
+            0.017874684184789658,
+            -0.014188224449753761,
+            -0.00920066051185131,
+            0.03451022133231163,
+            0.02665713243186474,
+            -0.009115469641983509,
+            0.01798310875892639,
+            0.023264970630407333,
+            -0.0010484338272362947,
+            -0.01196550577878952,
+            0.001922612776979804,
+            -0.0009550138493068516,
+            -0.0032972905319184065,
+            0.00845717266201973,
+            -0.015365412458777428,
+            0.0315207801759243,
+            0.04287445917725563,
+            -0.002145271748304367,
+            0.009587894193828106,
+            -0.003727119415998459,
+            -0.008472662419080734,
+            0.017069239169359207,
+            0.009316830895841122,
+            0.04625113308429718,
+            0.01098193321377039,
+            0.005680711008608341,
+            -0.033983584493398666,
+            -0.001229465357027948,
+            -0.01184933539479971,
+            0.03308520466089249,
+            0.02061629481613636,
+            0.00889087375253439,
+            -0.008379725739359856,
+            0.01344473659992218,
+            0.008341003209352493,
+            0.00661007035523653,
+            0.020569827407598495,
+            -0.016542602330446243,
+            -0.017286090180277824,
+            0.030467506498098373,
+            -0.00672236829996109,
+            0.01289486512541771,
+            0.02029101923108101,
+            -0.002089123008772731,
+            0.010842529125511646,
+            0.07100307941436768,
+            -0.006184114143252373,
+            0.015713922679424286,
+            -0.019082851707935333,
+            0.0015789433382451534,
+            -0.007640110794454813,
+            -0.03370477631688118,
+            0.018246427178382874,
+            0.0039962464943528175,
+            0.00865078903734684,
+            0.01198099460452795,
+            0.009053511545062065,
+            -0.007946024648845196,
+            -0.00963436160236597,
+            0.015195030719041824,
+            0.0019080914789810777,
+            0.00033568276558071375,
+            0.001336922636255622,
+            -0.0065210070461034775,
+            -0.004011735785752535,
+            -0.021948376670479774,
+            0.02763296104967594,
+            0.007140580099076033,
+            0.030606910586357117,
+            -0.055296897888183594,
+            0.030095763504505157,
+            -0.00789568480104208,
+            0.02554190158843994,
+            -0.016201836988329887,
+            -0.011090358719229698,
+            0.001609921921044588,
+            0.018153492361307144,
+            0.004491904750466347,
+            0.006462921854108572,
+            0.011888058856129646,
+            0.023233991116285324,
+            -0.03444826602935791,
+            0.017053749412298203,
+            0.023110076785087585,
+            0.0048675211146473885,
+            0.0003535923024173826,
+            0.005165690556168556,
+            0.015458349138498306,
+            0.007198664825409651,
+            -0.023048119619488716,
+            -0.0025692922063171864,
+            0.017611365765333176,
+            -0.02306360937654972,
+            0.0036980770528316498,
+            -0.006923729553818703,
+            0.004623564425855875,
+            -0.0014705180656164885,
+            -0.03509881719946861,
+            -0.00100099784322083,
+            -0.025743262842297554,
+            0.01923774555325508,
+            0.009556914679706097,
+            -0.025867177173495293,
+            0.003932353109121323,
+            0.025650326162576675,
+            -0.0003695656778290868,
+            0.00821708794683218,
+            0.0004683101433329284,
+            0.00869725737720728,
+            -0.00974278710782528,
+            -0.01493945624679327,
+            0.004801691509783268,
+            0.01531120017170906,
+            -0.01493945624679327,
+            -0.007431004662066698,
+            -0.009510447271168232,
+            0.030560443177819252,
+            0.0006674932083114982,
+            0.004189862869679928,
+            -0.005045648664236069,
+            -0.01858719252049923,
+            -0.0032237162813544273,
+            -0.021142931655049324,
+            0.004367990419268608,
+            -0.0009046735358424485,
+            0.04197607934474945,
+            0.0038161829579621553,
+            -0.02029101923108101,
+            0.0032972905319184065,
+            -0.03136589005589485,
+            0.0005000148667022586,
+            -0.007276111748069525,
+            -0.02004319056868553,
+            0.0015005286550149322,
+            0.0016525175888091326,
+            0.012631546705961227,
+            -0.03255856782197952,
+            -0.009301341138780117,
+            0.018416810780763626,
+            -0.00020680672605521977,
+            -0.010687636211514473,
+            -0.017828216776251793,
+            0.00035189816844649613,
+            -0.011338187381625175,
+            -0.027973726391792297,
+            -0.02850036323070526,
+            -0.0026719090528786182,
+            0.013491204008460045,
+            -0.018277406692504883,
+            -0.014993669465184212,
+            0.0031249718740582466,
+            0.021592123433947563,
+            -0.009781510569155216,
+            0.008302279748022556,
+            0.02241305634379387,
+            -0.008255811408162117,
+            -0.005514200776815414,
+            -0.000941460661124438,
+            -0.009130958467721939,
+            0.004267309792339802,
+            0.006300284061580896,
+            0.043339136987924576,
+            -0.032403673976659775,
+            -0.0018238683696836233,
+            0.014815541915595531,
+            0.0034153966698795557,
+            0.0025905899237841368,
+            0.002503462601453066,
+            -0.0007841471815481782,
+            0.01344473659992218,
+            0.01178737822920084,
+            0.015466093085706234,
+            0.00100099784322083,
+            -0.0033011629711836576,
+            0.006408709101378918,
+            -0.021173911169171333,
+            0.015721667557954788,
+            0.027958236634731293,
+            -0.0033999073784798384,
+            0.005525817628949881,
+            -0.013731288723647594,
+            -0.008759214542806149,
+            0.013134949840605259,
+            0.0026641644071787596,
+            -0.005068882368505001,
+            -0.009394276887178421,
+            0.017967619001865387,
+            0.019082851707935333,
+            0.006668155547231436,
+            -0.037453193217515945,
+            -0.02898053266108036,
+            -0.009959637187421322,
+            0.020569827407598495,
+            0.014916222542524338,
+            0.0010339126456528902,
+            -0.010540487244725227,
+            -0.009595638141036034,
+            -0.011903548613190651,
+            0.010989678092300892,
+            0.02721474878489971,
+            -0.032372694462537766,
+            0.0008214184199459851,
+            -0.029383255168795586,
+            0.03068435750901699,
+            0.0020000594668090343,
+            0.011446612887084484,
+            -0.0008185141487047076,
+            -0.015140817500650883,
+            -0.029181893914937973,
+            0.013824224472045898,
+            -0.004395096562802792,
+            0.001024231780320406,
+            -0.0031753121875226498,
+            0.0201051477342844,
+            0.012887120246887207,
+            -0.0014472840121015906,
+            0.01968693546950817,
+            0.008178364485502243,
+            0.016480645164847374,
+            -0.010153254494071007,
+            -0.04135650396347046,
+            0.019965743646025658,
+            0.007562664337456226,
+            0.014219203032553196,
+            0.004739734344184399,
+            0.004588713403791189,
+            -0.020771188661456108,
+            0.002724185585975647,
+            0.004333139397203922,
+            0.014405074529349804,
+            0.011028401553630829,
+            0.0025615475606173277,
+            0.011678952723741531,
+            -0.02228914201259613,
+            0.01929970271885395,
+            -0.01631026156246662,
+            0.01006806269288063,
+            0.004445436876267195,
+            -0.026873983442783356,
+            0.007547175046056509,
+            0.007822110317647457,
+            -0.0019187404541298747,
+            -0.017549408599734306,
+            0.02160761132836342,
+            -0.0034831624943763018,
+            -0.03509881719946861,
+            0.0360281765460968,
+            -0.0043447562493383884,
+            0.019346170127391815,
+            0.03209388628602028,
+            -0.01771979033946991,
+            -0.01993476413190365,
+            0.00906900130212307,
+            0.010408828034996986,
+            -0.002873270306736231,
+            0.017859194427728653,
+            -0.002919738180935383,
+            -0.02232012152671814,
+            0.017766257748007774,
+            -0.028918573632836342,
+            -0.014234691858291626,
+            0.020337486639618874,
+            -0.03624502569437027,
+            -0.0021026763133704662,
+            0.03175312280654907,
+            -0.024070415645837784,
+            0.005734923295676708,
+            0.005196669604629278,
+            -0.0019351978553459048,
+            0.0012236569309607148,
+            -0.006451305001974106,
+            0.03801080957055092,
+            -0.010865762829780579,
+            0.0201051477342844,
+            0.01929970271885395,
+            -0.023388884961605072,
+            0.01977987214922905,
+            -0.001109423115849495,
+            0.01675945147871971,
+            -0.003940097521990538,
+            0.03466511517763138,
+            -0.006017603911459446,
+            -0.011330443434417248,
+            0.006431943271309137,
+            0.008488151244819164,
+            -0.009959637187421322,
+            0.008681767620146275,
+            -0.018122512847185135,
+            0.027152791619300842,
+            0.02091059274971485,
+            -0.0002691270783543587,
+            0.013800990767776966,
+            0.0004421719058882445,
+            -0.020833145827054977,
+            0.00789568480104208,
+            -0.027121812105178833,
+            -0.009967382065951824,
+            0.01646515540778637,
+            -0.005762029904872179,
+            0.007380664348602295,
+            -0.02064727433025837,
+            0.028004704043269157,
+            0.0077524082735180855,
+            -0.013591884635388851,
+            -0.0008824076503515244,
+            0.023760627955198288,
+            -0.005432881414890289,
+            0.009355553425848484,
+            0.02070923149585724,
+            0.015713922679424286,
+            0.013615119270980358,
+            0.016263794153928757,
+            0.03351890295743942,
+            0.005328328814357519,
+            0.005568413063883781,
+            0.003519949736073613,
+            0.016991792246699333,
+            -0.008557853288948536,
+            0.018029578030109406,
+            0.0008093173382803798,
+            -0.013576395809650421,
+            0.021499186754226685,
+            -0.0018684001406654716,
+            -0.013622863218188286,
+            -0.0040891822427511215,
+            -0.004337011836469173,
+            0.03271345794200897,
+            0.022041313350200653,
+            -0.004468671046197414,
+            -0.024442158639431,
+            -0.00394784240052104,
+            0.016682006418704987,
+            -0.003167567541822791,
+            0.011136826127767563,
+            0.0007952801533974707,
+            -0.01586107164621353,
+            -0.009595638141036034,
+            0.0063583687879145145,
+            0.010586955584585667,
+            -0.008921852335333824,
+            0.010346870869398117,
+            -0.0228312686085701,
+            -0.004011735785752535,
+            0.00046613195445388556,
+            -0.004844287410378456,
+            0.0053438181057572365,
+            -0.011314953677356243,
+            0.004948840010911226,
+            0.016062432900071144,
+            0.01026167906820774,
+            0.004832670092582703,
+            0.006989559158682823,
+            0.012066186405718327,
+            0.009161937050521374,
+            -0.01627928391098976,
+            0.00428667152300477,
+            -0.005072754807770252,
+            -0.02345084212720394,
+            0.005831731949001551,
+            0.00902253296226263,
+            -0.01122201792895794,
+            0.009386532939970493,
+            -0.010958699509501457,
+            0.03296129032969475,
+            0.01030814740806818,
+            -0.037391237914562225,
+            -0.0033379499800503254,
+            -0.0016244432190433145,
+            0.014885243959724903,
+            0.012298526242375374,
+            0.008550108410418034,
+            -0.017518429085612297,
+            -0.0047823297791182995,
+            -0.009835722856223583,
+            0.0012827098835259676,
+            -0.012306271120905876,
+            0.017456471920013428,
+            0.022567950189113617,
+            -0.036337960511446,
+            0.01968693546950817,
+            -0.010432061739265919,
+            -0.023652203381061554,
+            0.020476890727877617,
+            0.011500826105475426,
+            0.01942361705005169,
+            -0.009402021765708923,
+            0.018896980211138725,
+            -0.0001331114035565406,
+            0.025510922074317932,
+            -0.004228586331009865,
+            -0.022087780758738518,
+            -0.012499887496232986,
+            -0.03209388628602028,
+            -0.0051811798475682735,
+            0.006819176487624645,
+            -0.00045524101005867124,
+            0.013111716136336327,
+            0.014226946979761124,
+            -0.025712283328175545,
+            0.007295473478734493,
+            0.032589543610811234,
+            0.014637414366006851,
+            0.023311438038945198,
+            -0.011276230216026306,
+            -0.0010794125264510512,
+            -0.0026060794480144978,
+            -0.013770012184977531,
+            -0.03212486580014229,
+            -0.0017493258928880095,
+            -0.036585792899131775,
+            0.005680711008608341,
+            -0.016387708485126495,
+            -0.013096226379275322,
+            0.023264970630407333,
+            -0.034138478338718414,
+            0.0014608372002840042,
+            0.013708055019378662,
+            0.010540487244725227,
+            0.012391461990773678,
+            -0.00162637943867594,
+            0.015032391995191574,
+            -0.029243851080536842,
+            -0.010284913703799248,
+            -0.004491904750466347,
+            -0.01981084980070591,
+            -0.007338068913668394,
+            -0.018060555681586266,
+            -0.013127204962074757,
+            -0.003566417610272765,
+            -0.010060317814350128,
+            0.002247888594865799,
+            0.0032062907703220844,
+            -0.014714861288666725,
+            0.03206290677189827,
+            0.003157886676490307,
+            0.018958937376737595,
+            0.0033418224193155766,
+            0.004449309315532446,
+            0.018788553774356842,
+            -0.0026331855915486813,
+            -0.0058278595097362995,
+            0.030963165685534477,
+            -0.00032672801171429455,
+            0.008759214542806149,
+            0.019082851707935333,
+            0.004724245052784681,
+            0.0038529701996594667,
+            -0.0260530486702919,
+            0.012887120246887207,
+            -0.027447087690234184,
+            0.005080499220639467,
+            -0.0077175572514534,
+            -0.024302754551172256,
+            -0.028128618374466896,
+            -0.006571347359567881,
+            0.012058441527187824,
+            -0.02898053266108036,
+            -0.00613377382978797,
+            0.002315654419362545,
+            -0.00400399137288332,
+            -0.002673845272511244,
+            0.007241260726004839,
+            0.029150914400815964,
+            -0.033209118992090225,
+            0.008751469664275646,
+            0.005595519673079252,
+            -0.0005053392960689962,
+            0.013297587633132935,
+            0.008038961328566074,
+            0.003343758638948202,
+            0.020848635584115982,
+            0.006637176964432001,
+            0.002458930714055896,
+            -0.0025576751213520765,
+            -0.014242436736822128,
+            0.004000118933618069,
+            0.02154565416276455,
+            0.007562664337456226,
+            0.018385831266641617,
+            -0.008914108388125896,
+            0.002621568739414215,
+            -0.01455222349613905,
+            0.005552923772484064,
+            -0.002867461647838354,
+            0.004309905227273703,
+            -0.012236569076776505,
+            0.008093173615634441,
+            0.0028771425131708384,
+            -0.0043447562493383884,
+            0.018060555681586266,
+            -0.002319526858627796,
+            0.016480645164847374,
+            0.004015608225017786,
+            0.015620986931025982,
+            -0.0035412474535405636,
+            0.01178737822920084,
+            0.0005731051205657423,
+            -0.017905661836266518,
+            -0.017022771760821342,
+            -0.039249956607818604,
+            0.007702067960053682,
+            0.03531566634774208,
+            -0.0009564659558236599,
+            -0.03444826602935791,
+            0.010145509615540504,
+            -0.011942271143198013,
+            -0.009975126944482327,
+            -0.022862248122692108,
+            0.013862947933375835,
+            0.012430185452103615,
+            0.007415515370666981,
+            0.012252057902514935,
+            -0.006997303571552038,
+            0.01026167906820774,
+            -0.005216030869632959,
+            0.008356492035090923,
+            -0.0010823167394846678,
+            -0.037422213703393936,
+            0.012344993650913239,
+            0.02216522768139839,
+            -0.002635121811181307,
+            0.0016089539276435971,
+            -0.006145390681922436,
+            0.022103270515799522,
+            -0.013971373438835144,
+            -0.00650551775470376,
+            -0.00046443779137916863,
+            0.00797700323164463,
+            -0.01795213110744953,
+            0.004778457339853048,
+            0.0755879208445549,
+            -0.007775642443448305,
+            0.006122156511992216,
+            0.009270362555980682,
+            -0.015148562379181385,
+            -0.01637221872806549,
+            -0.004561606794595718,
+            -0.020260039716959,
+            0.008844406343996525,
+            -0.0018819533288478851,
+            -0.0110981035977602,
+            0.012468908913433552,
+            0.0053050946444272995,
+            0.002031038049608469,
+            -0.008379725739359856,
+            0.010811550542712212,
+            0.009525936096906662,
+            0.010432061739265919,
+            0.02360573597252369,
+            0.011477591469883919,
+            -0.00858883187174797,
+            -0.011795123107731342,
+            0.01129171997308731,
+            0.0028577810153365135,
+            0.016697494313120842,
+            0.007907302118837833,
+            0.0010648912284523249,
+            0.019501063972711563,
+            -0.002309845993295312,
+            0.02180897258222103,
+            -2.2689444449497387e-05,
+            -0.009301341138780117,
+            -0.004236331209540367,
+            -0.015349923633038998,
+            0.024271776899695396,
+            -0.0020155487582087517,
+            0.0015450604259967804,
+            -0.020213572308421135,
+            0.0013979118084535003,
+            0.013243375346064568,
+            0.011957760900259018,
+            -0.0020407189149409533,
+            -0.0051347119733691216,
+            0.0007154133054427803,
+            -0.016356728971004486,
+            0.032806396484375,
+            0.007578153628855944,
+            0.0007681738352403045,
+            0.0008838597568683326,
+            0.0009288755827583373,
+            -0.0032972905319184065,
+            0.0024608669336885214,
+            -0.004766840487718582,
+            0.009549170732498169,
+            -0.010943209752440453,
+            -0.010796061716973782,
+            0.010075807571411133,
+            -0.02402394637465477,
+            0.003566417610272765,
+            -0.00558777479454875,
+            -0.008062195032835007,
+            0.0076207490637898445,
+            0.00015537731815129519,
+            -0.007392281666398048,
+            -0.0030939930584281683,
+            0.0023408245760947466,
+            0.013429246842861176,
+            0.004406713880598545,
+            0.010850274004042149,
+            0.012956822291016579,
+            0.005359307397156954,
+            -0.006629432085901499,
+            0.015512561425566673,
+            -0.006946963258087635,
+            -0.004228586331009865,
+            -0.0012468908680602908,
+            -0.019996721297502518,
+            -0.0009419447160325944,
+            -0.004708755761384964,
+            0.0015092412941157818,
+            0.0001705036120256409,
+            -8.749050175538287e-05,
+            -0.0018490385264158249,
+            -0.011988739483058453,
+            -0.001831613015383482,
+            -0.011663463898003101,
+            -0.014498010277748108,
+            0.010253935120999813,
+            -0.0041201612912118435,
+            -0.022939693182706833,
+            -0.015349923633038998,
+            -0.020662762224674225,
+            0.02145271934568882,
+            -0.004387352149933577,
+            -0.005800753366202116,
+            0.006733985152095556,
+            -0.010215211659669876,
+            -0.015706177800893784,
+            -0.008069939911365509,
+            -0.0205853171646595,
+            0.010703125037252903,
+            0.003214035416021943,
+            0.0006791101768612862,
+            0.00882891658693552,
+            0.014397329650819302,
+            -0.01589204929769039,
+            0.0111755495890975,
+            0.012182355858385563,
+            0.011361422017216682,
+            -0.006854027509689331,
+            0.0008911203476600349,
+            -0.014103032648563385,
+            0.02596011385321617,
+            0.021623101085424423,
+            0.005169562995433807,
+            -0.04132552444934845,
+            -0.005862710531800985,
+            -0.013344055972993374,
+            -0.010524998418986797,
+            -0.012120398692786694,
+            0.011771888472139835,
+            -0.0038820127956569195,
+            0.0013117524795234203,
+            0.006288666743785143,
+            0.0015063370810821652,
+            0.004367990419268608,
+            -0.0022382077295333147,
+            -0.007709812838584185,
+            -0.03212486580014229,
+            -0.02817508764564991,
+            -0.00863530021160841,
+            0.020461400970816612,
+            0.002838419284671545,
+            -0.0009453330421820283,
+            -0.0036380558740347624,
+            0.0027048238553106785,
+            -0.005882072262465954,
+            -0.005018542055040598,
+            -0.014753584749996662,
+            -0.012739972211420536,
+            0.0101300198584795,
+            0.00802347157150507,
+            0.014195968396961689,
+            -0.011950016021728516,
+            0.0028190575540065765,
+            0.018788553774356842,
+            0.007690451107919216,
+            -0.023357905447483063,
+            0.01135367713868618,
+            -0.010912231169641018,
+            -0.009270362555980682,
+            -0.009433000348508358,
+            -0.009471723809838295,
+            0.0016980175860226154,
+            -0.008317768573760986,
+            0.003945906180888414,
+            -0.03615209087729454,
+            -0.011330443434417248,
+            -0.0013901671627536416,
+            -0.010346870869398117,
+            0.0009864765452221036,
+            0.02524760365486145,
+            0.006815304048359394,
+            0.01432762760668993,
+            -0.0006733016925863922,
+            0.017502939328551292,
+            -0.0018984107300639153,
+            0.018726596608757973,
+            0.012763205915689468,
+            -0.00797700323164463,
+            0.011717676185071468,
+            0.014738094992935658,
+            -0.0033863543067127466,
+            0.015620986931025982,
+            -0.010168743319809437,
+            0.005285732913762331,
+            -0.009556914679706097,
+            -0.003214035416021943,
+            0.002495717955753207,
+            0.030901208519935608,
+            -0.003618694143369794,
+            0.00858883187174797,
+            0.01098193321377039,
+            -0.015241498127579689,
+            0.0026951429899781942,
+            0.014962689951062202,
+            0.01141563430428505,
+            0.007760153152048588,
+            0.0024434414226561785,
+            -0.024225307628512383,
+            0.004193735308945179,
+            0.005037903785705566,
+            0.0031520782504230738,
+            -0.014366351068019867,
+            0.0021859314292669296,
+            0.015675200149416924,
+            0.008550108410418034,
+            -0.025325050577521324,
+            0.016418687999248505,
+            0.0009138703462667763,
+            -0.003909118939191103,
+            0.029119934886693954,
+            -0.011268485337495804,
+            -0.005723306443542242,
+            0.009463978931307793,
+            0.0009980935137718916,
+            0.0030029932968318462,
+            0.008937342092394829,
+            0.0228312686085701,
+            -0.003988501615822315,
+            -0.032341714948415756,
+            -0.022459525614976883,
+            0.015706177800893784,
+            -0.0047823297791182995,
+            -0.022939693182706833,
+            -0.026703599840402603,
+            -0.003113354789093137,
+            -0.02228914201259613,
+            -0.0022323993034660816,
+            -0.020058678463101387,
+            -0.014056564308702946,
+            0.001268188701942563,
+            0.029987338930368423,
+            0.016201836988329887,
+            0.008224832825362682,
+            -0.011051635257899761,
+            -0.029956359416246414,
+            -0.024271776899695396,
+            0.01810702309012413,
+            -0.007055388763546944,
+            -0.005541306920349598,
+            0.00591692328453064,
+            -0.009951893240213394,
+            -0.008480406366288662,
+            0.011802867986261845,
+            0.027137301862239838,
+            -0.002747419523075223,
+            -0.029987338930368423,
+            -0.02688947319984436,
+            -0.014335372485220432,
+            -0.01840132102370262,
+            -0.017704300582408905,
+            -0.008627555333077908,
+            0.0045035220682621,
+            0.00030300996149890125,
+            0.016186347231268883,
+            0.014962689951062202,
+            -0.000480653194244951,
+            0.0028597170021384954,
+            -2.7711375878425315e-05,
+            -0.001763847190886736,
+            0.006888878531754017,
+            -0.004933350719511509,
+            0.026997897773981094,
+            0.014614180661737919,
+            -0.006594581063836813,
+            0.018711106851696968,
+            0.028314489871263504,
+            0.020430423319339752,
+            0.003157886676490307,
+            -0.010695381090044975,
+            -0.014606435783207417,
+            0.009727297350764275,
+            -0.013421501964330673,
+            0.023264970630407333,
+            0.022552460432052612,
+            -0.01480779703706503,
+            -0.014885243959724903,
+            -0.017905661836266518,
+            0.015210519544780254,
+            -0.007562664337456226,
+            -0.008224832825362682,
+            0.012368228286504745,
+            0.022335609421133995,
+            -0.007024410180747509,
+            0.002371803391724825,
+            -0.0010232636705040932,
+            -0.021731525659561157,
+            -0.006122156511992216,
+            -0.004596457816660404,
+            -0.008666278794407845,
+            0.008681767620146275,
+            0.008131897076964378,
+            -0.001928421319462359,
+            -0.006319645792245865,
+            0.005471604876220226,
+            -0.013367289677262306,
+            0.01301103550940752,
+            -0.00789568480104208,
+            0.03379771113395691,
+            -0.009138703346252441,
+            -0.0006733016925863922,
+            -0.022598929703235626,
+            -0.011446612887084484,
+            0.00845717266201973,
+            0.0017967619933187962,
+            0.01682141050696373,
+            -0.0011626676423475146,
+            -0.005688455421477556,
+            0.009913169778883457,
+            -0.0010687635513022542,
+            -0.001450188341550529,
+            -0.006892750971019268,
+            -0.010401083156466484,
+            0.0002884887217078358,
+            0.013258864171802998,
+            -0.006025348324328661,
+            0.0018054747488349676,
+            0.006095050368458033,
+            -0.0175339188426733,
+            0.029027000069618225,
+            0.023884542286396027,
+            0.017255110666155815,
+            0.014660648070275784,
+            0.0006394187803380191,
+            0.011647974140942097,
+            -0.003572226269170642,
+            0.0038820127956569195,
+            -0.020337486639618874,
+            0.021080974489450455,
+            -0.010811550542712212,
+            0.01846327818930149,
+            0.011655719019472599,
+            -0.015233753249049187,
+            -0.005022414494305849,
+            0.021406250074505806,
+            0.009967382065951824,
+            0.00889087375253439,
+            0.0017541663255542517,
+            -0.02470547705888748,
+            0.017766257748007774,
+            0.010099041275680065,
+            -0.020399443805217743,
+            0.019547531381249428,
+            -0.009471723809838295,
+            0.01233724970370531,
+            -0.0005300253978930414,
+            0.004418330732733011,
+            0.0058743273839354515,
+            0.021499186754226685,
+            0.011570527218282223,
+            -0.03884723410010338,
+            -0.013615119270980358,
+            -0.0017357728211209178,
+            0.006857899948954582,
+            -0.0036303112283349037,
+            -0.015233753249049187,
+            -0.041635312139987946,
+            0.01604694314301014,
+            0.009998360648751259,
+            -0.01643417589366436,
+            0.004987563472241163,
+            -0.002427952131256461,
+            -0.0042053526267409325,
+            -0.008170620538294315,
+            0.003186929039657116,
+            -0.004170501604676247,
+            0.02324948087334633,
+            0.0018984107300639153,
+            -0.006946963258087635,
+            0.013258864171802998,
+            0.01344473659992218,
+            -0.021948376670479774,
+            0.0009874446550384164,
+            0.0021375271026045084,
+            0.000636998622212559,
+            0.023683181032538414,
+            -0.014707116410136223,
+            -0.0017657832941040397,
+            -0.01685238815844059,
+            -0.0013069120468571782,
+            -0.021592123433947563,
+            -0.001188805908896029,
+            -0.015187285840511322,
+            -0.008511385880410671,
+            0.025030752643942833,
+            -0.011005166918039322,
+            0.010997422970831394,
+            -0.006687517277896404,
+            0.006943091284483671,
+            -0.006904367823153734,
+            -0.01936165988445282,
+            -0.0008267428493127227,
+            0.009975126944482327,
+            -0.000512115890160203,
+            0.002941036131232977,
+            8.482827252009884e-05,
+            0.002464739140123129,
+            0.008867640048265457,
+            0.006234454456716776,
+            -0.0075278133153915405,
+            0.02489134855568409,
+            -0.012073930352926254,
+            0.0016399325104430318,
+            0.011911292560398579,
+            0.0004743606550619006,
+            -0.010424317792057991,
+            0.025588368996977806,
+            -0.02139076218008995,
+            -0.00800798274576664,
+            -0.00014799568452872336,
+            -0.0016544538084417582,
+            -0.006513262167572975,
+            0.006718495860695839,
+            0.02946070022881031,
+            0.0062654330395162106,
+            0.010447551496326923,
+            -0.0281441081315279,
+            -0.011493081226944923,
+            0.012538610957562923,
+            -0.022273652255535126,
+            -0.0007739823195151985,
+            -0.0004322490422055125,
+            -0.00619960343465209,
+            0.026378324255347252,
+            -0.02022906206548214,
+            -0.01093546487390995,
+            -0.018664639443159103,
+            0.005405775271356106,
+            -0.011121337302029133,
+            0.008294534869492054,
+            -0.0009777637897059321,
+            0.006540368776768446,
+            0.016805920749902725,
+            -0.019563021138310432,
+            0.016031453385949135,
+            0.005119222681969404,
+            -0.016217326745390892,
+            0.010222956538200378,
+            -0.017580386251211166,
+            -0.0008199662552215159,
+            0.020275529474020004,
+            -0.00400399137288332,
+            0.008751469664275646,
+            -0.003363120136782527,
+            0.005208286456763744,
+            -0.006513262167572975,
+            0.0001492057926952839,
+            -0.007249005138874054,
+            0.014428308233618736,
+            -0.016805920749902725,
+            -0.017828216776251793,
+            -0.008821171708405018,
+            0.0016709112096577883,
+            0.021297825500369072,
+            0.011136826127767563,
+            -0.010044828988611698,
+            -0.011059380136430264,
+            -0.011694442480802536,
+            0.041604332625865936,
+            0.005762029904872179,
+            0.027230238541960716,
+            -0.02261441759765148,
+            0.00545998802408576,
+            0.025170156732201576,
+            -0.014242436736822128,
+            0.0012788375606760383,
+            -0.0022788674104958773,
+            0.0053050946444272995,
+            0.011562783271074295,
+            -0.01819995976984501,
+            0.0049139889888465405,
+            -0.00019216447253711522,
+            -0.00782985519617796,
+            -0.000656844291370362,
+            0.003442503046244383,
+            0.00020196630794089288,
+            0.016325751319527626,
+            -0.004027225077152252,
+            -0.016511622816324234,
+            -0.004429947584867477,
+            -0.0003216455806978047,
+            -0.027896279469132423,
+            0.013514438644051552,
+            0.016589069738984108,
+            0.01566745527088642,
+            0.027989214286208153,
+            -0.014513500034809113,
+            0.016031453385949135,
+            0.009533680975437164,
+            0.013901671394705772,
+            -0.010610189288854599,
+            0.008952830918133259,
+            -0.01595400646328926,
+            0.007446494419127703,
+            -0.03454120084643364,
+            -0.007887939922511578,
+            -0.00802347157150507,
+            0.01984182931482792,
+            0.028763681650161743,
+            0.002923610620200634,
+            0.006493900436908007,
+            0.000196883876924403,
+            0.0024202074855566025,
+            -0.01714668609201908,
+            -0.020182594656944275,
+            -0.0003337466041557491,
+            0.0029332914855331182,
+            0.0006950835813768208,
+            -0.004495777189731598,
+            0.00472037261351943,
+            -0.012391461990773678,
+            0.0026835259050130844,
+            -0.013134949840605259,
+            0.005971135571599007,
+            -0.0059982421807944775,
+            0.009603383019566536,
+            0.0019429425010457635,
+            0.020864123478531837,
+            0.02300165221095085,
+            0.004100799560546875,
+            0.0046971384435892105,
+            -0.024333734065294266,
+            0.00920066051185131,
+            -0.013847459107637405,
+            0.012329504825174809,
+            0.019408127292990685,
+            0.008248066529631615,
+            0.006571347359567881,
+            0.01233724970370531,
+            0.012918098829686642,
+            -0.0031772484071552753,
+            0.024287264794111252,
+            0.005014669615775347,
+            -0.0009070937521755695,
+            0.002052335999906063,
+            -0.014954946003854275,
+            -0.0070979841984808445,
+            -0.0015286030247807503,
+            0.0018848575418815017,
+            -0.004731989465653896,
+            0.01906736195087433,
+            0.015256987884640694,
+            -0.01050176378339529,
+            0.017874684184789658,
+            -0.03937387093901634,
+            0.017379024997353554,
+            -0.0027377386577427387,
+            -0.0060911779291927814,
+            -0.00545998802408576,
+            0.00665653869509697,
+            0.004515138920396566,
+            0.003669034456834197,
+            0.03773200139403343,
+            -0.010408828034996986,
+            0.003963331691920757,
+            -0.005847221240401268,
+            0.007121218368411064,
+            -0.020864123478531837,
+            0.008302279748022556,
+            0.005336073227226734,
+            0.02160761132836342,
+            -0.0056497324258089066,
+            -0.0014482521219179034,
+            0.02328045852482319,
+            0.008859895169734955,
+            0.03469609469175339,
+            -0.01566745527088642,
+            -0.004418330732733011,
+            0.01048627495765686,
+            0.0207402091473341,
+            -0.00580462533980608,
+            -0.008976065553724766,
+            0.007903429679572582,
+            0.0035586729645729065,
+            0.021824462339282036,
+            -0.00515407370403409,
+            -0.0006650729919783771,
+            0.0054367538541555405,
+            -0.0060911779291927814,
+            0.009735042229294777,
+            0.011252996511757374,
+            -0.009076746180653572,
+            0.007539430167526007,
+            0.021530164405703545,
+            -0.009975126944482327,
+            -0.008658533915877342,
+            -0.009649851359426975,
+            0.0010658593382686377,
+            -0.03068435750901699,
+            0.001649613375775516,
+            0.005359307397156954,
+            0.015187285840511322,
+            -0.0004603234410751611,
+            -0.009014789015054703,
+            0.021019017323851585,
+            0.017255110666155815,
+            -0.005936284549534321,
+            -0.0028132491279393435,
+            0.0023795480374246836,
+            -0.0017154429806396365,
+            -0.006160879973322153,
+            -0.0006248975405469537,
+            -0.002656419761478901,
+            0.014730350114405155,
+            -0.0015469965292140841,
+            -0.006617815233767033,
+            -0.00756653631106019,
+            0.0033302053343504667,
+            0.010834784246981144,
+            -0.005661349277943373,
+            0.013057502917945385,
+            0.016805920749902725,
+            -0.00031704717548564076,
+            0.019671445712447166,
+            0.01037010457366705,
+            -0.010447551496326923,
+            0.015938518568873405,
+            -0.007984748110175133,
+            0.006563602481037378,
+            0.0164031982421875,
+            -0.01277095079421997,
+            -0.01246116403490305,
+            0.007210282143205404,
+            0.00665653869509697,
+            0.028918573632836342,
+            0.014792308211326599,
+            -0.02109646424651146,
+            0.014908477663993835,
+            0.015768134966492653,
+            -0.013018779456615448,
+            0.003537375247105956,
+            -0.01784370467066765,
+            0.020260039716959,
+            0.019051872193813324,
+            -0.015535795129835606,
+            -0.017224133014678955,
+            -0.014947201125323772,
+            -0.005092116538435221,
+            0.006331262644380331,
+            0.0033263331279158592,
+            -0.00617636926472187,
+            -0.027989214286208153,
+            -0.0049139889888465405,
+            -0.018633661791682243,
+            0.013870692811906338,
+            -0.008674023672938347,
+            0.0007715621031820774,
+            -0.004511266481131315,
+            0.007984748110175133,
+            -0.0033882902935147285,
+            0.032465629279613495,
+            -0.0051811798475682735,
+            -0.02832997962832451,
+            0.004398969002068043,
+            0.016356728971004486,
+            0.012623801827430725,
+            -0.029243851080536842,
+            0.010463040322065353,
+            0.004805563949048519,
+            -0.012530866079032421,
+            0.005990497302263975,
+            -0.02258343994617462,
+            -0.007632365915924311,
+            -0.0009395244996994734,
+            -0.016743963584303856,
+            -0.027121812105178833,
+            -0.008433938957750797,
+            -0.006993431597948074,
+            -0.005401902832090855,
+            -0.009525936096906662,
+            -0.008720491081476212,
+            -0.011477591469883919,
+            0.02303262986242771,
+            -0.007899557240307331,
+            -0.001866463921032846,
+            -0.0014637414133176208,
+            0.00930908601731062,
+            -0.023776117712259293,
+            -0.015605497173964977,
+            0.006389347370713949,
+            -0.002594462363049388,
+            0.022815778851509094,
+            -0.012747716158628464,
+            0.005940156988799572,
+            -0.009316830895841122,
+            0.009851212613284588,
+            0.015241498127579689,
+            0.004956584889441729,
+            0.017440982162952423,
+            -0.009889936074614525,
+            0.0021026763133704662,
+            -0.008627555333077908,
+            -0.00018986526993103325,
+            -1.027075541060185e-05,
+            0.014428308233618736,
+            0.007624621503055096,
+            0.014141756109893322,
+            0.017487451434135437,
+            -0.013553161174058914,
+            0.005003052763640881,
+            -0.0003274540649726987,
+            -0.017255110666155815,
+            0.0027725896798074245,
+            -0.007984748110175133,
+            -0.0008896682411432266,
+            0.001758038648404181,
+            6.395397940650582e-05,
+            -0.02112744189798832,
+            -0.0009124181815423071,
+            0.0018809852190315723,
+            0.0010222955606877804,
+            -0.025201136246323586,
+            0.00889087375253439,
+            -0.00889087375253439,
+            -0.0014918158994987607,
+            -0.019113829359412193,
+            0.019795360043644905,
+            0.026300877332687378,
+            -0.004619691986590624,
+            0.00036206303047947586,
+            -0.013537672348320484,
+            -0.02049238048493862,
+            0.0001489637652412057,
+            -0.011314953677356243,
+            -0.011345932260155678,
+            -0.0022169100120663643,
+            -0.003214035416021943,
+            0.008805682882666588,
+            0.0004189379105810076,
+            -0.009433000348508358,
+            -0.004956584889441729,
+            -0.011764144524931908,
+            -0.017549408599734306,
+            -0.010401083156466484,
+            0.01933068037033081,
+            -0.0018016024259850383,
+            0.0020794421434402466,
+            -0.0032508226577192545,
+            0.0005271211848594248,
+            0.011748654767870903,
+            0.013475715182721615,
+            0.013344055972993374,
+            -0.023915521800518036,
+            -0.017890173941850662,
+            -0.013320821337401867,
+            0.0029487807769328356,
+            -0.007187047973275185,
+            0.004844287410378456,
+            0.026362836360931396,
+            -0.0009695350890979171,
+            -0.01091997604817152,
+            0.002327271504327655,
+            0.019051872193813324,
+            -0.031272951513528824,
+            -0.013390523381531239,
+            -0.025077221915125847,
+            0.02225816436111927,
+            -0.013351799920201302,
+            0.011516314931213856,
+            0.010191977955400944,
+            -0.025944624096155167,
+            0.006768836174160242,
+            -0.007686578668653965,
+            0.026037558913230896,
+            0.01634124107658863,
+            0.0005934348446317017,
+            -0.00944074522703886,
+            0.013622863218188286,
+            -0.0008722427301108837,
+            -0.013576395809650421,
+            0.002201420720666647,
+            0.014645159244537354,
+            -0.014691627584397793,
+            -0.009510447271168232,
+            -0.004507394507527351,
+            0.004313777666538954,
+            0.012763205915689468,
+            -0.0030339721124619246,
+            -0.004031097516417503,
+            -0.011624740436673164,
+            0.013344055972993374,
+            -0.003930416889488697,
+            0.012383717112243176,
+            0.007380664348602295,
+            -0.014056564308702946,
+            -0.019532041624188423,
+            0.033333033323287964,
+            0.00400399137288332,
+            0.00338054564781487,
+            -0.010997422970831394,
+            0.02740062028169632,
+            -0.009897680021822453,
+            0.010571465827524662,
+            -0.006408709101378918,
+            0.0031985461246222258,
+            0.012321759946644306,
+            -0.012104909867048264,
+            -0.01981084980070591,
+            0.021468207240104675,
+            -0.00364192808046937,
+            -0.0010193913476541638,
+            0.008310024626553059,
+            -0.020182594656944275,
+            -0.00663330452516675,
+            0.002180122770369053,
+            0.009541425853967667,
+            -0.011818356812000275,
+            0.006369986105710268,
+            0.017905661836266518,
+            0.003912991378456354,
+            -0.0007672057254239917,
+            0.019996721297502518,
+            -0.01672847382724285,
+            0.011717676185071468,
+            -0.010850274004042149,
+            0.0016825281782075763,
+            0.0006065039779059589,
+            -0.0048597767017781734,
+            0.0058743273839354515,
+            -0.014536733739078045,
+            0.011988739483058453,
+            -6.244135147426277e-05,
+            -0.013979118317365646,
+            0.007601387333124876,
+            -0.013359544798731804,
+            -0.016836898401379585,
+            0.041542377322912216,
+            -0.005301222205162048,
+            -0.0038665232714265585,
+            -0.0056962003000080585,
+            0.01350669376552105,
+            0.02823704481124878,
+            -0.005336073227226734,
+            -0.01153180468827486,
+            -0.010780571959912777,
+            -0.016573579981923103,
+            0.015907539054751396,
+            -0.01810702309012413,
+            0.002495717955753207,
+            -0.010532742366194725,
+            -0.007903429679572582,
+            0.004302160814404488,
+            -0.003516077296808362,
+            0.014722606167197227,
+            0.00950270239263773,
+            0.023357905447483063,
+            0.00797700323164463,
+            -0.00027348342700861394,
+            0.010393339209258556,
+            0.011663463898003101,
+            0.003243078012019396,
+            -0.010509508661925793,
+            -0.00011949774489039555,
+            -0.02894955314695835,
+            -0.008449427783489227,
+            -0.004863648675382137,
+            -0.001485039247199893,
+            -0.00428667152300477,
+            -0.01239920686930418,
+            0.006617815233767033,
+            -0.0022052929271012545,
+            0.013909416273236275,
+            -0.009254872798919678,
+            -0.017766257748007774,
+            0.03389064967632294,
+            0.004495777189731598,
+            0.006466794293373823,
+            -0.008286789990961552,
+            -0.03705047070980072,
+            -0.0032566310837864876,
+            -0.008116407319903374,
+            -0.013529927469789982,
+            0.004538373090326786,
+            0.01283290795981884,
+            -0.014831030741333961,
+            -0.028097640722990036,
+            0.0019390701781958342,
+            -0.006683644838631153,
+            0.02978597767651081,
+            -0.007798876613378525,
+            0.00419760774821043,
+            0.0037251831963658333,
+            0.0011558911064639688,
+            0.01239920686930418,
+            -0.00845717266201973,
+            -0.0022536972537636757,
+            0.016387708485126495,
+            -0.010858018882572651,
+            0.011051635257899761,
+            -0.010966443456709385,
+            -0.02554190158843994,
+            -0.01456771232187748,
+            0.021344292908906937,
+            -0.0004823473282158375,
+            -0.00024903935263864696,
+            0.014823286794126034,
+            0.02354377694427967,
+            0.011260741390287876,
+            0.00969631876796484,
+            0.006408709101378918,
+            -0.001035848748870194,
+            0.005975008010864258,
+            0.006095050368458033,
+            -0.0024163350462913513,
+            -0.013398268260061741,
+            0.017564896494150162,
+            0.009773765690624714,
+            -0.01810702309012413,
+            0.0008514289511367679,
+            0.0036612898111343384,
+            0.0168833676725626,
+            0.006935346405953169,
+            0.009316830895841122,
+            -0.0143121387809515,
+            -0.023760627955198288,
+            -0.00014932679187040776,
+            0.002873270306736231,
+            0.015721667557954788,
+            0.010579210706055164,
+            -0.014319883659482002,
+            0.0005963391158729792,
+            0.008712747134268284,
+            0.020213572308421135,
+            -0.024209819734096527,
+            0.019702425226569176,
+            -0.004673904739320278,
+            -0.0021084847394376993,
+            0.007946024648845196,
+            0.011632485315203667,
+            -0.011872569099068642,
+            -0.00650551775470376,
+            0.006249943748116493,
+            0.003157886676490307,
+            0.03172214329242706,
+            -0.0017260919557884336,
+            0.009665340185165405,
+            -0.010997422970831394,
+            -0.0002405686245765537,
+            0.0006728176376782358,
+            -0.0006820144481025636,
+            -0.002884887158870697,
+            -0.011020656675100327,
+            0.0018974426202476025,
+            -0.020306508988142014,
+            0.0024473138619214296,
+            0.00463130883872509,
+            -0.005684582982212305,
+            -0.0032972905319184065,
+            -0.019222255796194077,
+            -0.0022672503255307674,
+            0.003293418325483799,
+            0.014335372485220432,
+            -0.031691163778305054,
+            0.004972074180841446,
+            0.010672146454453468,
+            -0.002309845993295312,
+            -0.010416572913527489,
+            0.011423379182815552,
+            -0.0026060794480144978,
+            0.011988739483058453,
+            -0.0038142469711601734,
+            -0.0004905760288238525,
+            0.025681305676698685,
+            -0.005308967083692551,
+            0.014707116410136223,
+            -0.01026167906820774,
+            -0.002993312431499362,
+            -0.028252532705664635,
+            0.005235392600297928,
+            -0.007403898518532515,
+            -0.0012081675231456757,
+            -0.004402841441333294,
+            -0.01130720879882574,
+            -0.0026990154292434454,
+            0.002122037811204791,
+            0.008000237867236137,
+            0.012910354882478714,
+            -0.0007725302129983902,
+            -0.010075807571411133,
+            0.004832670092582703,
+            0.0075355577282607555,
+            -0.015520306304097176,
+            -0.0070437719114124775,
+            -0.020306508988142014,
+            0.029662061482667923,
+            -0.013731288723647594,
+            -0.0011123273288831115,
+            -0.020864123478531837,
+            0.010160998441278934,
+            -0.0063119009137153625,
+            -0.002782270545139909,
+            0.004708755761384964,
+            -0.015109838917851448,
+            0.01344473659992218,
+            0.008751469664275646,
+            0.01450575515627861,
+            -0.02151467651128769,
+            -0.01762685552239418,
+            -0.0044725434854626656,
+            -0.002735802438110113,
+            -0.010648912750184536,
+            -0.0005334137240424752,
+            0.002013612538576126,
+            0.021251358091831207,
+            -0.00032503384863957763,
+            0.0042053526267409325,
+            -0.003214035416021943,
+            0.017224133014678955,
+            -0.010137764737010002,
+            -0.013057502917945385,
+            0.0024260159116238356,
+            0.005994369741529226,
+            0.016186347231268883,
+            0.03158273920416832,
+            -0.020693741738796234,
+            -0.00272999401204288,
+            0.0032740565948188305,
+            0.010656657628715038,
+            -0.004085310269147158,
+            6.304640555754304e-05,
+            -0.027803342789411545,
+            -0.04464024305343628,
+            -0.002422143705189228,
+            0.013057502917945385,
+            -0.015636475756764412,
+            0.012670270167291164,
+            0.008743725717067719,
+            0.010021595284342766,
+            -0.005196669604629278,
+            0.02620794251561165,
+            -0.004793947096914053,
+            0.014304393902420998,
+            0.011733165942132473,
+            0.011911292560398579,
+            0.0009874446550384164,
+            0.0007459079497493804,
+            -0.018354853615164757,
+            -0.008255811408162117,
+            -0.0015954007394611835,
+            0.01765783317387104,
+            -0.027478067204356194,
+            0.008526874706149101,
+            -0.011516314931213856,
+            0.013839714229106903,
+            0.0003666614356916398,
+            0.005661349277943373,
+            -0.01771979033946991,
+            0.012445674277842045,
+            -0.0009337160154245794,
+            -0.025325050577521324,
+            0.004805563949048519,
+            0.011276230216026306,
+            -0.010362359695136547,
+            0.006877261213958263,
+            0.019950253888964653,
+            0.0023079097736626863,
+            0.008495896123349667,
+            -0.014126266352832317,
+            0.010703125037252903,
+            0.009649851359426975,
+            -0.021080974489450455,
+            -0.01233724970370531,
+            -0.007059261202812195,
+            -0.006335135083645582,
+            -0.0035470561124384403,
+            -0.00013105422840453684,
+            0.010176488198339939,
+            -0.016000475734472275,
+            0.010099041275680065,
+            -0.006559730041772127,
+            0.006400964688509703,
+            0.00944074522703886,
+            0.001115231541916728,
+            -0.013576395809650421,
+            0.012716737575829029,
+            -0.0218709297478199,
+            0.008426194079220295,
+            -0.04287445917725563,
+            -0.005847221240401268,
+            -0.015078860335052013,
+            0.005204414017498493,
+            -0.003798757679760456,
+            -0.004453181754797697,
+            -0.006203475873917341,
+            -0.005061137955635786,
+            0.00920066051185131,
+            0.021747015416622162,
+            0.012135888449847698,
+            -0.016620049253106117,
+            -0.007551047019660473,
+            0.005425137002021074,
+            0.0004854935978073627,
+            -0.013336311094462872,
+            0.009316830895841122,
+            -0.01576039008796215,
+            0.0009259713697247207,
+            0.006339007057249546,
+            -0.01072635967284441,
+            0.013452480547130108,
+            0.004731989465653896,
+            -0.003401843598112464,
+            0.013390523381531239,
+            -0.0050417762249708176,
+            -0.0057891360484063625,
+            -0.007272239308804274,
+            -0.013739033602178097,
+            -0.003818119177594781,
+            -0.003072695340961218,
+            0.012747716158628464,
+            0.001529571134597063,
+            -0.0003243077953811735,
+            -0.00782985519617796,
+            0.009154192171990871,
+            0.002220782218500972,
+            -0.0013146566925570369,
+            0.005773646757006645,
+            -0.02300165221095085,
+            0.0001287550403503701,
+            0.004755223635584116,
+            -0.0009487213101238012,
+            -0.006246071308851242,
+            -0.012151377275586128,
+            -0.011903548613190651,
+            -0.038165703415870667,
+            0.009363298304378986,
+            0.0019197085639461875,
+            -0.0076207490637898445,
+            0.0064822835847735405,
+            0.017255110666155815,
+            0.01456771232187748,
+            0.016743963584303856,
+            -0.020693741738796234,
+            -0.02303262986242771,
+            0.005556796211749315,
+            -0.009402021765708923,
+            0.02261441759765148,
+            0.004046586807817221,
+            -0.005847221240401268,
+            0.0013243375578895211,
+            -0.013258864171802998,
+            0.008929597213864326,
+            0.005723306443542242,
+            -0.001592496526427567,
+            0.005425137002021074,
+            -0.013692565262317657,
+            -0.010904486291110516,
+            0.01048627495765686,
+            0.032496608793735504,
+            -0.0004261985304765403,
+            0.0024105266202241182,
+            0.023295948281884193,
+            0.015466093085706234,
+            0.016356728971004486,
+            -0.021111954003572464,
+            0.011167804710566998,
+            0.010060317814350128,
+            -0.0018645278178155422,
+            -0.014892988838255405,
+            0.00513083953410387,
+            -0.012716737575829029,
+            -0.010664401575922966,
+            0.014993669465184212,
+            0.007853088900446892,
+            0.017735280096530914,
+            -0.02369867078959942,
+            0.00789568480104208,
+            0.00042765066609717906,
+            0.00013710475468542427,
+            0.011524059809744358,
+            0.012956822291016579,
+            0.03370477631688118,
+            0.027152791619300842,
+            0.020662762224674225,
+            -0.009285852313041687,
+            0.014366351068019867,
+            -0.009192915633320808,
+            -0.011090358719229698,
+            -0.019470084458589554,
+            -0.015249243006110191,
+            -0.011136826127767563,
+            0.017998598515987396,
+            0.008790193125605583,
+            -0.009355553425848484,
+            -0.005947901867330074,
+            0.01586107164621353,
+            -0.003909118939191103,
+            -0.004635181277990341,
+            0.01646515540778637,
+            -0.026269899681210518,
+            0.016836898401379585,
+            -0.000745423894841224,
+            -0.014668392948806286,
+            -0.014691627584397793,
+            -0.004182118456810713,
+            -0.012670270167291164,
+            0.0003964299685321748,
+            -0.020817656069993973,
+            0.013181418180465698,
+            0.024597052484750748,
+            -0.0030920570716261864,
+            -0.004100799560546875,
+            -0.011299464851617813,
+            -0.0005111477803438902,
+            0.018385831266641617,
+            0.0034773540683090687,
+            0.009192915633320808,
+            -0.0023872926831245422,
+            0.013924905098974705,
+            0.00524313747882843,
+            0.0027958236169070005,
+            -0.0007565568666905165,
+            -0.016263794153928757,
+            0.012197845615446568,
+            0.0026835259050130844,
+            0.012135888449847698,
+            -0.022242674604058266,
+            0.007938280701637268,
+            -0.011764144524931908,
+            -0.0023698671720921993,
+            -0.02228914201259613,
+            0.0189744271337986,
+            -0.01610890030860901,
+            0.001046497724018991,
+            0.003841353114694357,
+            0.0006224773242138326,
+            0.009580149315297604,
+            0.01610890030860901,
+            -0.023388884961605072,
+            -0.0003460896550677717,
+            -0.014908477663993835,
+            -0.021917399019002914,
+            -0.01607792265713215,
+            0.004728117026388645,
+            -0.019501063972711563,
+            0.005444498732686043,
+            -0.0025557391345500946,
+            -0.015125328674912453,
+            0.018478767946362495,
+            0.014691627584397793,
+            -0.004379607271403074,
+            0.002412462839856744,
+            0.005425137002021074,
+            -0.0014405074762180448,
+            0.01971791312098503,
+            0.00013274837692733854,
+            -0.0010048701660707593,
+            -0.009425255469977856,
+            -0.0168833676725626,
+            -0.0065752193331718445,
+            -0.035005878657102585,
+            0.0260530486702919,
+            -0.004670032300055027,
+            -0.016511622816324234,
+            0.014319883659482002,
+            0.009115469641983509,
+            -0.014629669487476349,
+            -0.007059261202812195,
+            0.016294771805405617,
+            -0.024287264794111252,
+            0.02244403585791588,
+            -0.009611127898097038,
+            -0.008774704299867153,
+            0.00784921646118164,
+            0.05154848098754883,
+            0.009146448224782944,
+            0.003492843359708786,
+            0.011036145500838757,
+            0.005839476361870766,
+            -0.0013165927957743406,
+            -0.023683181032538414,
+            0.005568413063883781,
+            -0.007190920412540436,
+            -0.003601268632337451,
+            0.002536377403885126,
+            -0.004801691509783268,
+            0.031149037182331085,
+            -0.004379607271403074,
+            -0.005289605353027582,
+            0.004484160337597132,
+            -0.02196386642754078,
+            -0.0001332324172835797,
+            0.023203013464808464,
+            -2.3128106477088295e-05,
+            -0.023590246215462685,
+            -0.006571347359567881,
+            -0.015071115456521511,
+            -0.013770012184977531,
+            -0.018989915028214455,
+            0.005231520161032677,
+            -0.012755461037158966,
+            -0.0033940989524126053,
+            0.0016660707769915462,
+            -0.005940156988799572,
+            -0.002582845278084278,
+            0.009603383019566536,
+            0.05362405255436897,
+            0.014103032648563385,
+            -0.006335135083645582,
+            -0.005854965653270483,
+            0.009192915633320808,
+            -0.010524998418986797,
+            -0.019795360043644905,
+            -0.009820234030485153,
+            -0.029507169499993324,
+            -0.012747716158628464,
+            -0.0143121387809515,
+            -0.010191977955400944,
+            -0.010277168825268745,
+            0.00963436160236597,
+            -0.018122512847185135,
+            -0.007628493942320347,
+            -0.020337486639618874,
+            0.002941036131232977,
+            -0.011268485337495804,
+            -0.014714861288666725,
+            -0.008054450154304504,
+            -0.04807887226343155,
+            0.005796880926936865,
+            0.019593998789787292,
+            0.01263929158449173,
+            0.002515079453587532,
+            0.004383479710668325,
+            -0.01993476413190365,
+            -0.030792783945798874,
+            0.013839714229106903,
+            0.011283975094556808,
+            0.011733165942132473,
+            0.03283737599849701,
+            -0.011709931306540966,
+            -0.00813964195549488,
+            -0.004693266469985247,
+            -0.0017251238459721208,
+            0.012066186405718327,
+            0.00969631876796484,
+            -0.004917861428111792,
+            -0.012375972233712673,
+            0.016217326745390892,
+            -0.01981084980070591,
+            -0.023683181032538414,
+            -0.01672847382724285,
+            0.006157007534056902,
+            0.00813964195549488,
+            -0.008054450154304504,
+            0.002973950933665037,
+            0.0056962003000080585,
+            -0.007159941829741001,
+            -0.00022423222253564745,
+            0.004890755284577608,
+            -0.023048119619488716,
+            0.0008490087930113077,
+            -0.002257569460198283,
+            0.0061376458033919334,
+            0.021266845986247063,
+            -0.010168743319809437,
+            -0.0014075926737859845,
+            -0.0053902859799563885,
+            -0.0028984404634684324,
+            -0.006764963734894991,
+            0.02620794251561165,
+            0.009154192171990871,
+            -0.009882191196084023,
+            -0.01011453103274107,
+            -0.01129171997308731,
+            -0.0011278166202828288,
+            -0.017255110666155815,
+            -0.016186347231268883,
+            0.0015566773945465684,
+            -0.016712984070181847,
+            0.013971373438835144,
+            -0.00876695942133665,
+            0.00400399137288332,
+            -0.014110777527093887,
+            0.02019808255136013,
+            0.01929970271885395,
+            -0.009711808525025845,
+            0.023776117712259293,
+            -0.007709812838584185,
+            -0.038382552564144135,
+            -0.00800798274576664,
+            0.01843230053782463,
+            -0.006118284538388252,
+            0.010037084110081196,
+            -0.011314953677356243,
+            0.008209343999624252,
+            0.00015319912927225232,
+            0.0018122512847185135,
+            -0.0003112386621069163,
+            0.027555514127016068,
+            0.015535795129835606,
+            -0.005560668651014566,
+            -0.005494839046150446,
+            0.004255692940205336,
+            0.009541425853967667,
+            0.017580386251211166,
+            0.012151377275586128,
+            -0.021019017323851585,
+            -0.014257925562560558,
+            -0.016263794153928757,
+            0.011012911796569824,
+            0.015125328674912453,
+            -0.0040891822427511215,
+            0.01691434532403946,
+            0.014126266352832317,
+            -0.019098341464996338,
+            0.0019884423818439245,
+            -0.008193854242563248,
+            -0.00158184755127877,
+            -0.03029712475836277,
+            0.011043890379369259,
+            0.0074387495405972,
+            -0.0072218989953398705,
+            0.036616768687963486,
+            0.009858956560492516,
+            0.037577107548713684,
+            0.018680129200220108,
+            0.0008974128868430853,
+            0.009045767597854137,
+            -0.018370341509580612,
+            0.013065247796475887,
+            0.01500141341239214,
+            -0.012724482454359531,
+            -0.005734923295676708,
+            0.0013446672819554806,
+            0.0007551047019660473,
+            0.012631546705961227,
+            0.02004319056868553,
+            -0.01795213110744953,
+            0.009773765690624714,
+            -0.015032391995191574,
+            -0.005250881891697645,
+            0.006555858068168163,
+            -0.013932649977505207,
+            -0.008495896123349667,
+            -0.00963436160236597,
+            0.019408127292990685,
+            -0.012600568123161793,
+            0.016573579981923103,
+            -0.009595638141036034,
+            0.015520306304097176,
+            -0.029724018648266792,
+            -0.003669034456834197,
+            -0.006679772399365902,
+            -0.019593998789787292,
+            -0.012887120246887207,
+            0.009820234030485153,
+            0.010633422993123531,
+            -0.009301341138780117,
+            -0.011562783271074295,
+            0.020368466153740883,
+            -0.004185990896075964,
+            0.013529927469789982,
+            0.021189400926232338,
+            0.00737291993573308,
+            0.024132372811436653,
+            0.0005605200421996415,
+            0.014164989814162254,
+            0.005463860463351011,
+            0.010362359695136547,
+            -0.004677776712924242,
+            0.005161818582564592,
+            0.00932457484304905,
+            0.014203713275492191,
+            0.017766257748007774,
+            0.013196907006204128,
+            -0.00598275288939476,
+            0.007419387809932232,
+            0.02016710489988327,
+            0.0016254113288596272,
+            -0.030312614515423775,
+            -0.009719553403556347,
+            0.0001736498816171661,
+            0.00924712885171175,
+            0.009185170754790306,
+            3.817877222900279e-05,
+            -0.01093546487390995,
+            0.011144571006298065,
+            0.00969631876796484,
+            0.002286612056195736,
+            0.006280922330915928,
+            -0.01456771232187748,
+            -0.020957060158252716,
+            0.006106667220592499,
+            0.01798310875892639,
+            0.013917161151766777,
+            -0.014722606167197227,
+            0.022784801200032234,
+            0.006145390681922436,
+            -0.021406250074505806,
+            -0.01840132102370262,
+            -0.0009559819591231644,
+            0.0004542729293461889,
+            -0.002265314105898142,
+            0.03720536455512047,
+            -0.0011539548868313432,
+            -0.005955646280199289,
+            -0.0054754773154854774,
+            0.02775687538087368,
+            0.02049238048493862,
+            0.009177426807582378
+          ],
+          "embedding": [
+            0.0008215984936325879,
+            -0.019349418284688353,
+            -0.02322660633029795,
+            0.009747820845194852,
+            0.007353771220903641,
+            0.015252908783199839,
+            -0.012176175829980202,
+            0.051898051094966605,
+            -0.0068130353886146165,
+            0.008189212816890595,
+            0.008102287831931905,
+            0.01682801136011614,
+            0.02005557996049711,
+            -0.020779879580497183,
+            0.002332165268329781,
+            0.02337485065616377,
+            0.006074992895540845,
+            0.02576575146535689,
+            0.0007517286429767504,
+            -0.03763909942314134,
+            0.0011204729720314833,
+            0.00041212613589632806,
+            -0.06089176800314795,
+            0.012852035873395388,
+            -0.008901127092058543,
+            0.01702347055484216,
+            0.005803446395466281,
+            0.017889570275475296,
+            -0.05375047382413552,
+            0.021623778774252724,
+            0.014784317465883424,
+            -0.0018881880857885708,
+            0.0004735328554290247,
+            0.012393802850407656,
+            -0.019234442968055952,
+            -0.019566283611638122,
+            0.04407789373296535,
+            -0.0053565873209102995,
+            -0.026240866282480922,
+            0.048107902270426675,
+            -0.0054681314939387235,
+            -0.0009441397948041965,
+            -0.015205079626883548,
+            0.006885981841893229,
+            0.01403724256866746,
+            -0.026086548246792754,
+            0.0034778946051893067,
+            0.00044587533948775473,
+            0.032851328435891634,
+            -0.03882097333255334,
+            -0.011126344604254894,
+            -0.01648943336384904,
+            0.035810456555802166,
+            -0.019384589526165754,
+            -0.019430702768170285,
+            0.014297440272239852,
+            -0.007883453204663151,
+            0.050996900793086844,
+            0.06007149434153715,
+            0.03406832053483379,
+            0.025193439934671852,
+            0.00027785129663604897,
+            -0.008014391056168083,
+            -0.018806730908490114,
+            0.027193788903098558,
+            -0.006195006670328595,
+            -0.020466000344352826,
+            0.0068939884219589185,
+            0.017252276167086602,
+            -0.045073444648704136,
+            -0.049242269917908754,
+            0.023021140866008313,
+            -0.015676872472071106,
+            -0.024497517908898602,
+            0.005873633718973869,
+            -0.022828585938203683,
+            0.011748341004940436,
+            0.04624205727927451,
+            -0.015442165499674641,
+            0.018565551071491767,
+            0.015246505395507096,
+            0.004119269796224144,
+            0.004862735250819252,
+            -0.000145323321960046,
+            0.05155838952887077,
+            0.0027052423345323936,
+            -0.016218063670586664,
+            -0.07462470920944148,
+            0.043695974746295135,
+            -0.027608273720377708,
+            -0.03860912774695494,
+            0.002823110565150435,
+            0.026471469849508766,
+            -0.0026063577189074103,
+            -0.029230652014151735,
+            0.05684164470477843,
+            0.00022182868300572677,
+            0.015590845552956364,
+            -0.024939617549529242,
+            0.01035803286203917,
+            -0.010225298012384327,
+            0.019571847648759157,
+            0.01841613041441054,
+            0.0156981470538354,
+            0.021764297694793343,
+            -0.018831230007033416,
+            -0.014419529638789993,
+            -0.0029965475899652273,
+            0.028881185012095884,
+            0.09163526460163937,
+            0.022644302207811818,
+            -0.017933909936330133,
+            -0.04663928709016869,
+            0.004962951444105128,
+            -0.041333041174526026,
+            -0.027545628007617527,
+            -0.037142394579894954,
+            -0.013358143162788402,
+            0.02719686432396677,
+            -0.0034971800937504205,
+            0.04413635817371732,
+            0.04903857110328917,
+            0.03186067155630788,
+            0.007831051050269414,
+            -0.025257538623313574,
+            0.023517513773109414,
+            -0.0016527403209505964,
+            -0.0016101860707429347,
+            0.014448515185903624,
+            0.009905018312477684,
+            0.0024114582235163955,
+            0.00172043851384822,
+            -0.03818316221102173,
+            -0.02304537008110578,
+            0.01024969233570089,
+            -0.01901116356327339,
+            0.019791121460848986,
+            0.07985067340821365,
+            -0.051903191272236665,
+            -0.025924646976556942,
+            0.014766511532849252,
+            -0.002073086076894999,
+            0.030959269926800852,
+            0.006394417986617089,
+            -0.028177586693713442,
+            0.010725285336674526,
+            0.0009614052276924192,
+            0.07500253788978564,
+            -0.00036794182149960157,
+            0.004058458332826519,
+            -0.013218269166951234,
+            0.027613539268683245,
+            -0.012995170442480582,
+            0.023388707018488407,
+            0.026389423313105165,
+            -0.003068639956061266,
+            -0.02661967649632026,
+            0.026145274570635913,
+            -0.03300591322174331,
+            -0.004556072066733024,
+            0.020045722084112896,
+            0.0033437343914784865,
+            -0.010224220905483085,
+            -0.0025025498684079703,
+            0.03460103245395426,
+            0.04514362357278504,
+            0.03797031988058962,
+            -0.004586662475576956,
+            -0.06351723722596923,
+            0.06037983484369874,
+            0.019415753124363902,
+            0.028850618647994476,
+            -0.0034810237045849006,
+            0.02957938154808314,
+            0.02642652523231673,
+            0.013187653724133275,
+            -0.007856320804109569,
+            -0.007065882659482447,
+            -0.016672784625197085,
+            0.029020161154209707,
+            -0.013243058921970134,
+            -0.014900932834241732,
+            0.006001769901896033,
+            0.008648456186944271,
+            0.03484763413464497,
+            0.012279368984306564,
+            0.02379643508687133,
+            0.04933044959399941,
+            -0.021047723935647383,
+            -0.007264823701706426,
+            0.0007058774795716792,
+            -0.010512071832195085,
+            0.027489671668749157,
+            0.009731157570361245,
+            -0.03195919640758579,
+            -0.0033626031972691915,
+            -0.04282419484651207,
+            0.01873977910485268,
+            -0.014398285316695653,
+            0.012854631447494207,
+            -0.0017372959978092432,
+            -0.037331867825735776,
+            0.001088209292238142,
+            -0.012345684323522151,
+            0.037096063326414734,
+            0.005050219690024652,
+            0.0008906051515353004,
+            -0.003697709449630797,
+            0.006548479142737107,
+            -0.010727663712299397,
+            -0.007486226350610755,
+            -0.02704618481069485,
+            -0.019752552016312926,
+            -0.014632581296303486,
+            -0.0037501138119724717,
+            0.014588411795703245,
+            -0.007119137071719376,
+            0.016440258608998535,
+            -0.03686645352111013,
+            -0.023546402448713415,
+            0.005628703432152666,
+            0.024988145751166143,
+            0.011971199595808573,
+            -0.010868778329190568,
+            -0.03270046433624425,
+            -0.021950559285357267,
+            -0.011889914022657738,
+            -0.005123971523701009,
+            0.02619713302280345,
+            -0.021990810305904974,
+            0.01165257474485436,
+            -0.009117901804315863,
+            -0.007492082755249399,
+            0.010725902070291232,
+            0.01862811996181951,
+            -0.026694281868322685,
+            -0.01623776415010199,
+            -0.009403997538684658,
+            -0.02207329027097839,
+            0.025939540497196416,
+            0.014178803271471762,
+            -0.001141821318145744,
+            -0.006281187017652108,
+            -0.035993365198554964,
+            0.0026192776111449633,
+            -0.015254976771874396,
+            -0.0116872453009606,
+            -0.015780016491993135,
+            0.021421328283124968,
+            0.02889432250124136,
+            -0.01871009416296692,
+            -0.026540472640514536,
+            0.015703370682110186,
+            -0.0002849235245164438,
+            -0.02594930889995505,
+            0.04491828744809679,
+            0.044063522197410734,
+            -0.037441864491547405,
+            -0.007005339187980476,
+            -0.030381794393931814,
+            0.06901083845539985,
+            -0.017604082400603037,
+            -0.04060591381445593,
+            0.06044850935418443,
+            -0.01756490171206357,
+            0.025539717592705295,
+            0.03452018789549939,
+            0.0016217577522523927,
+            0.00018825054264059277,
+            -0.002563606092068712,
+            0.04770277039047993,
+            -0.0032010143255574255,
+            -0.03425328601475849,
+            -0.02105909767072799,
+            -0.00840881022563103,
+            -0.028699447677549002,
+            -0.015925160861589006,
+            -0.0055750742281166,
+            0.017513987082479483,
+            0.010399718005952185,
+            0.02267201252273823,
+            0.01258511878601025,
+            0.0015081978484714903,
+            0.01874981572091931,
+            0.011158889421057756,
+            -0.002607810686437211,
+            -0.003732023411827874,
+            -0.007787189972165309,
+            -0.05524012485122728,
+            -0.025110689689279583,
+            -0.03946383236703947,
+            0.004443572529325475,
+            -0.018745077598477586,
+            0.005926160888274727,
+            0.039430331582190524,
+            -0.0049500324516682,
+            0.004352795787899792,
+            0.015566139991780173,
+            0.013148156356408315,
+            -0.015279560337042242,
+            -0.03316846614871165,
+            0.0001592209771257332,
+            0.018362411161743347,
+            -0.012706582418607377,
+            -0.03424477956681259,
+            -0.001991022841226589,
+            0.0033467286047492175,
+            0.03357019284647071,
+            -0.024359110796680374,
+            0.00359117029951049,
+            0.005061818038334802,
+            0.011359582931760315,
+            -0.026793992746244076,
+            -0.018170136079817746,
+            -0.017421603453826397,
+            0.03996608926471779,
+            0.02587013058343799,
+            0.015763410145648683,
+            -0.04512536449983358,
+            -0.02293473890636517,
+            0.048575083783480195,
+            -0.009589860756342604,
+            -0.06486262416944208,
+            0.03447666009581266,
+            0.016105654870253654,
+            0.003526582643722421,
+            -0.01354754623151814,
+            0.05401575248587799,
+            -0.03563190295559527,
+            -0.017365166993914202,
+            -0.0046288709698227735,
+            -0.033657212157212645,
+            -0.007686252059880836,
+            -0.001385273642139782,
+            0.048229062789530236,
+            -0.029083855744728694,
+            -0.030465559177859865,
+            0.0024368661249980416,
+            -0.024144031154493057,
+            0.021693278579520123,
+            -0.016405088055634124,
+            -0.018439935250344958,
+            0.006948603359995389,
+            -0.033943037395199695,
+            -0.003007796423804453,
+            -0.016783195426536633,
+            -0.02270589491206155,
+            0.020874438753828814,
+            0.016724666903417947,
+            0.027033184180344852,
+            0.0019847647852695785,
+            0.017749524526371133,
+            0.0204661893602285,
+            -0.005604929968008821,
+            0.028462234633072944,
+            -0.020790459496727066,
+            -0.04183816355941877,
+            0.03946487892663695,
+            0.01192861758684541,
+            -0.0018024841275380242,
+            0.01077082591611702,
+            -0.01475239020295694,
+            -0.03952356245715099,
+            0.03683950018557526,
+            -0.056767672022551764,
+            -0.007507600090017693,
+            0.009530544352246083,
+            -0.00788856051420074,
+            -0.011204694027562521,
+            -0.010587701466318598,
+            -0.028604940377315504,
+            0.023749537635431462,
+            -0.013672419168235691,
+            0.01953449229707597,
+            -0.011449939491554354,
+            -0.012238180529977605,
+            0.05795536247491174,
+            -0.021518581774137322,
+            -0.040672955795925836,
+            0.026066659238628543,
+            -0.014061106927896723,
+            0.03328357230532424,
+            0.03957107406164427,
+            0.030670947051333917,
+            -0.004278577327978564,
+            0.025885946540364826,
+            0.0319106101668716,
+            -0.003462160036198642,
+            -0.04686868330851868,
+            0.00555303933159587,
+            0.02156143971867743,
+            -0.03739529489075782,
+            -0.01025626126223168,
+            -0.02141970191890338,
+            0.022134910980548407,
+            0.017807484142010496,
+            -0.007232643256393455,
+            0.011318041192005274,
+            -0.010489012862455913,
+            -0.013003217163997554,
+            -0.00847279880392288,
+            0.02357274819910594,
+            -0.0016485066139744577,
+            -0.005873113634992993,
+            0.006424249429163587,
+            0.004008703814301658,
+            0.0054639858216457724,
+            0.010198079500408441,
+            0.017694720520798957,
+            -0.017087021465208255,
+            -0.0025558869389290554,
+            -0.016503579212015177,
+            -0.0420082404922411,
+            -0.003563173424359332,
+            0.022924161700566468,
+            0.012345933179990547,
+            -0.00729908668365632,
+            -0.004234683369642856,
+            -0.0022962545569187934,
+            -0.020666205489143392,
+            -0.014796149043465308,
+            0.024562653223873056,
+            0.030217494971086438,
+            0.007810961433007407,
+            0.007781263640449456,
+            -0.0030348559388780655,
+            0.007310876971216526,
+            0.016099591640649783,
+            0.02311929597780174,
+            -0.04160162038353416,
+            0.027152100845897007,
+            -0.00025396315502635146,
+            0.023717123370023598,
+            -0.04186918759097667,
+            -0.017988531857836703,
+            -0.03106663560912813,
+            -0.023595079699181436,
+            0.02437134274097885,
+            -0.02317559303912261,
+            -0.012327263775257306,
+            0.000989566600374836,
+            0.00954921216572729,
+            -0.005155079395685017,
+            -0.017508404061379164,
+            -0.04582321461886958,
+            -0.0014367264231457263,
+            -0.030417229142190933,
+            -0.017829710557332023,
+            -0.001056472801554354,
+            0.005016912084754242,
+            -0.06276644565618253,
+            0.0109965681725148,
+            -0.07270669271977365,
+            -0.004542796125637244,
+            0.040139998232966566,
+            -0.017890441783630643,
+            0.055909102081208445,
+            -0.05029104308417623,
+            0.01673962668706085,
+            -0.04507293803055318,
+            0.015385506084493484,
+            0.025677372277539627,
+            0.030235171586316813,
+            -0.022189978375852563,
+            0.008461188938299745,
+            0.008033707483259569,
+            -0.045089505755429415,
+            -0.04144698743458133,
+            -0.013640256640978024,
+            0.01036782197518379,
+            0.0012696958402094232,
+            -0.016090728932978384,
+            0.023838437877040163,
+            0.011946482002767135,
+            0.03342604921859589,
+            0.04211801335157251,
+            0.002708015395637621,
+            -0.047156646610976634,
+            -0.013104464812562618,
+            -0.003933116891120681,
+            0.03155321708116329,
+            -0.018866109579446184,
+            0.015376812010630058,
+            -0.01955557376085726,
+            0.016255217383093355,
+            0.017868332102274922,
+            -0.020325179506828335,
+            0.013434364256590097,
+            0.004176904701252156,
+            -0.03877060944639854,
+            0.0006475542749369515,
+            0.00042468300294954125,
+            -0.02116992978038693,
+            0.0032654387271108293,
+            -0.00575003786159731,
+            -0.05149954071159492,
+            -0.031382926943128234,
+            -0.015730563640253065,
+            0.0382312502769352,
+            0.004297939848910064,
+            0.019724983545557365,
+            -0.017907869879607064,
+            -0.005165628493894463,
+            0.03698976707579373,
+            0.008501001373266799,
+            -0.01725527821298066,
+            0.006879620077191753,
+            0.02455792200929138,
+            -0.03224193972301503,
+            -0.039181328884413266,
+            -0.0008206986106407542,
+            -0.023306758893533952,
+            0.01672946960876128,
+            -0.02658587698869307,
+            0.005873620623736886,
+            0.0025331170979801232,
+            -0.03490809296591325,
+            0.02456764314036355,
+            0.003291113462353163,
+            -0.004303423672882112,
+            -0.020359544091059775,
+            -0.016892415165120203,
+            0.017174797487220006,
+            0.04391489511478065,
+            0.007327321684948843,
+            -0.003943713832544148,
+            0.00862684231313155,
+            -0.006361172785753631,
+            0.05826688726534012,
+            -0.02284956969636637,
+            -0.026680023999604643,
+            0.027527436347931256,
+            -0.022597669279069723,
+            -0.006277863878949772,
+            -0.017343446085471857,
+            -0.012065192152203348,
+            -0.0004183712490442143,
+            0.019339311388446943,
+            -0.019423562920897294,
+            -0.010212527447243049,
+            0.004783006556748757,
+            0.012824133870627983,
+            -0.004855945108275706,
+            0.006239949458562236,
+            0.006847581690471813,
+            0.03094005096378827,
+            0.026993131409761086,
+            -0.011331881651539968,
+            0.03297014291876675,
+            0.010265600643742144,
+            4.666113318251278e-05,
+            0.001970879281172325,
+            0.002523664335535017,
+            0.0034107811520368574,
+            -0.03143890864030425,
+            -0.01624584278940617,
+            0.013468729048885906,
+            0.005233309338223403,
+            -0.004550544967856448,
+            -0.023783653541634683,
+            -0.009402494239255264,
+            0.03334923712760375,
+            -0.003260424721959948,
+            0.02748395114281811,
+            -0.0058374897745283065,
+            -0.01985840930273332,
+            0.017595418271610163,
+            0.02881743246631328,
+            -0.05928552440605384,
+            0.017062540894682354,
+            -0.0011508547522925793,
+            0.008953761538519997,
+            0.01610561914042553,
+            -0.008894160513648953,
+            0.013301991014533684,
+            0.04080863883897929,
+            0.010738051773755976,
+            -0.023345002072886145,
+            0.004215726310838674,
+            0.01541749448570608,
+            0.01291813299324752,
+            -0.022747942296528025,
+            0.0008640791861357522,
+            0.011008522066277018,
+            0.005893793007012561,
+            0.0026106450193140156,
+            -0.006278394080583205,
+            -0.025659188287146268,
+            0.024180550333596625,
+            -0.0044491292740074925,
+            0.026967973346698617,
+            0.04390909273429254,
+            0.009596855239692042,
+            -0.03686020608700941,
+            -0.013109951664921729,
+            0.0002799680359670127,
+            -0.0038537577602593366,
+            -0.0037552094562213916,
+            0.011105295862811114,
+            -0.04185615640149602,
+            0.019950890919369135,
+            0.006453241355231516,
+            0.015314645427201533,
+            -0.020031193483734364,
+            -0.024530173488182337,
+            0.0031949828520724337,
+            -0.01492645564652531,
+            0.012875827757040941,
+            0.005057170413681063,
+            -0.004319834710426384,
+            0.005386862713494232,
+            0.00030848054013413157,
+            0.008384941028326983,
+            -0.010811526993301901,
+            0.03175068297908541,
+            0.004680987487229685,
+            0.009264547472281849,
+            -0.02856314241700808,
+            -0.021083721348429348,
+            0.020113396771629096,
+            0.03572366771051171,
+            -0.01708529918264109,
+            0.03346921724975398,
+            -0.009042046660366718,
+            0.0033219155387035227,
+            0.0151720168318272,
+            -0.015805318210244414,
+            -0.008057236707694303,
+            -0.0005483009439015931,
+            -0.002917527004423299,
+            -0.014538511124345527,
+            0.007302979537905532,
+            -0.0208529958115422,
+            -0.02965227242147972,
+            0.020064178921373875,
+            -0.00011892622932349096,
+            0.016455188521316146,
+            0.010335044657747514,
+            -0.007816155992852236,
+            -0.0052262330850905165,
+            0.006959344748883874,
+            0.04432147882994085,
+            0.026734484553994495,
+            0.05547661378166294,
+            -0.005787997952888261,
+            -0.004559444854138438,
+            -0.0082295581123363,
+            -0.006588172755707144,
+            0.04524009609473585,
+            0.015915003261638767,
+            -0.004910285095602864,
+            0.017495607570191165,
+            0.01557886571266685,
+            0.03858259260136037,
+            0.0027648525902558316,
+            -0.016912796336835408,
+            0.04014755669075221,
+            -0.004077405782937414,
+            0.008616827127381418,
+            0.011102692199778469,
+            0.0024185742218798562,
+            0.00807141794538637,
+            0.02996274720815659,
+            -0.012429186019907761,
+            0.01680942269506771,
+            -0.04571916592313758,
+            -0.02775530246832996,
+            0.014370280587419414,
+            -0.004931905540824582,
+            0.03419959665162407,
+            0.01668521021731861,
+            -0.023105305446763272,
+            -0.024761385385514877,
+            0.021529428247380984,
+            -0.002603042335561961,
+            -0.024135326707477325,
+            -0.015906833215844473,
+            0.003286605488908195,
+            -0.0023437997044735617,
+            0.0060251848371355166,
+            -0.013481652076194643,
+            0.0002908426939460218,
+            0.01953362396253743,
+            0.0038544889405965057,
+            0.020760460182873287,
+            -0.016224396121239458,
+            -0.00259233446682179,
+            -0.009282861237653619,
+            -0.02952130765234087,
+            0.017988370145301916,
+            0.006138236653154911,
+            -0.007607182955958238,
+            -0.009516767050028553,
+            0.0048347778733230365,
+            -0.005819209642901871,
+            -0.0025851701689263706,
+            -0.01073014664868326,
+            0.012433465684360061,
+            0.025784014942833097,
+            -0.016875624785552303,
+            0.007944349714123104,
+            8.651589872318479e-05,
+            -0.022306044741692753,
+            -0.022127016745066307,
+            0.020143472307319017,
+            0.002792646249762062,
+            0.008768813346304779,
+            -0.004074214276300631,
+            -0.00015024341785520542,
+            0.002990550758627699,
+            0.01951746553838439,
+            0.03896451381932467,
+            0.004232673242619471,
+            0.027876586543280867,
+            0.0145860303663199,
+            0.03994234200854897,
+            0.009657017763624678,
+            0.00696607158576712,
+            -0.026055387694656488,
+            0.02819890357011778,
+            0.00018297502510753014,
+            0.043190622821775236,
+            0.022488402221902314,
+            -0.0407730101161195,
+            -0.0021946518523759066,
+            0.01669520746276111,
+            0.013212623257794862,
+            0.017009844136059122,
+            -0.004537512994041156,
+            -0.014041213424740049,
+            0.028135404528975894,
+            -0.022859586430516003,
+            0.0008637006862481223,
+            -0.004486332934188017,
+            -0.025031795510001887,
+            0.007351370254175368,
+            -0.004567816916440959,
+            0.030025413528233364,
+            0.0014976740344892735,
+            -0.022285096307825804,
+            0.022837170656498543,
+            -0.009551269792616045,
+            0.011045530488684207,
+            0.008809047185288755,
+            0.020065001354321994,
+            0.031993362733068575,
+            0.009143210434415064,
+            0.02133640814803879,
+            0.012241777474889248,
+            0.014771538127546474,
+            -0.008804192497858033,
+            -0.02341427260922065,
+            0.023167424359504862,
+            -0.0011598649565273595,
+            -0.029627604902846688,
+            0.0021991474725787303,
+            0.003261878079344105,
+            -0.0005935713331682358,
+            -0.009144689470851402,
+            -0.005580557862988033,
+            -0.007240836451875213,
+            0.019631377011298905,
+            -0.007085213886352692,
+            0.007017573363709754,
+            -0.004074171791476129,
+            -0.00822871835665378,
+            -0.0421101573155962,
+            0.004210914075286222,
+            0.003408427700622687,
+            -0.01752179330665445,
+            0.006384440404174512,
+            0.011121481025573962,
+            0.043020575455262985,
+            -0.022809979913698254,
+            -0.0005737138409049003,
+            0.019206623212867476,
+            0.0009562551889820241,
+            -0.023427467810802533,
+            -0.038931688376706186,
+            0.007172345672967499,
+            -0.03467050750542982,
+            0.023185301620612612,
+            -0.0007075596339842658,
+            0.02387977204565927,
+            -0.03043468414449851,
+            0.013909304572370437,
+            0.004784668158421625,
+            0.012524879174804915,
+            0.021654512058639427,
+            0.03142146922870715,
+            -0.02053473475540882,
+            -0.050409862638621725,
+            -0.005406667316025549,
+            0.00035138762584085976,
+            0.022018487016014856,
+            -0.026014580970974616,
+            0.00025633004699302783,
+            -0.003349094027291921,
+            -0.02401366769212778,
+            -0.01735996745614562,
+            0.01962340791446025,
+            -0.050011480783403936,
+            0.015042913576492952,
+            0.0262737424555697,
+            -0.02507473901367252,
+            -0.05408889321034101,
+            -0.018988587082325706,
+            -0.028217893770659686,
+            -0.009285770094319318,
+            -0.00661159230987788,
+            0.028597405154740078,
+            0.003525576256054959,
+            -0.011462219358952466,
+            0.025401668390029348,
+            0.006155837251830549,
+            0.0026050693705924593,
+            0.015985722392086222,
+            0.007289910674212967,
+            -0.011452511453623235,
+            -0.014599577191293152,
+            -0.04558584556100621,
+            0.010197536827229795,
+            -0.0045431962637690155,
+            -0.019372598050508005,
+            -0.030147301333718016,
+            -0.0137828704002156,
+            -0.027060515748528666,
+            -0.0017291966950107505,
+            0.053783684195381654,
+            0.03292494290301388,
+            0.005560172296221419,
+            -0.0026085670565688026,
+            -0.010466890489356092,
+            0.036247483457995895,
+            -0.006789746401727701,
+            0.03662690472838588,
+            0.001784522073210422,
+            -0.0242673253380092,
+            0.01169382884548301,
+            0.02892627181542449,
+            0.009858451662166387,
+            -0.0005592283645355213,
+            -0.006160132931009508,
+            -0.008782830470079547,
+            0.0031207630972208394,
+            0.025613233596604418,
+            0.003268170392795548,
+            -0.010062991425127038,
+            0.018106967407336815,
+            0.004480291699976396,
+            0.02010374270557836,
+            0.03555357587371324,
+            0.023489837757983054,
+            -0.021782173652026497,
+            -0.026581762796336082,
+            -0.0013561486781077014,
+            0.0072266106510934505,
+            0.03551967295942857,
+            -0.01651315514034218,
+            0.0001274396319698452,
+            -0.008272735507120127,
+            -0.00894931703949637,
+            0.011149952898504355,
+            -0.0029671593121986116,
+            -0.0030752010285951576,
+            0.01662711184497578,
+            -0.0011672263615879709,
+            -0.033448222499147764,
+            0.0078288560210653,
+            -0.023496331346917084,
+            0.0051040606404415545,
+            -0.004113104164416575,
+            -0.007076525381595441,
+            -0.002157548492138289,
+            0.006302962169276006,
+            -0.008065841090139872,
+            -0.00035242603988955366,
+            -0.01586406637763306,
+            0.008271894439780969,
+            0.007438312999409211,
+            -0.0069617481460406095,
+            0.011402932583388205,
+            0.002666895909532698,
+            0.0005219881149353855,
+            -0.012159425670629751,
+            -0.012042145306711322,
+            -0.00957276588761916,
+            -0.03401863879157551,
+            -0.02604353712472079,
+            -0.024949533021906844,
+            -0.0014209149651479556,
+            -0.006566685925666972,
+            -0.03187796739029736,
+            -0.01209152137926458,
+            -0.011110763124407752,
+            0.007756961097537267,
+            -0.013260548545917543,
+            0.005433285079114047,
+            -6.399098520041351e-05,
+            -0.0044960488826674555,
+            0.016615636438440516,
+            -0.003643291599919672,
+            -0.019488010211707343,
+            -0.03927818878149417,
+            0.008209336177358258,
+            -0.02805593125673831,
+            -0.0038816771550190788,
+            0.007607119219721711,
+            -0.00528681677614909,
+            -0.029502897838743516,
+            -0.009149794737430706,
+            0.008767001199311985,
+            0.003507217008280066,
+            -0.01773349756494436,
+            0.03579670773605949,
+            0.026971617790626075,
+            0.011875179879209696,
+            -0.019891342676443626,
+            0.0508929813275565,
+            -0.0047010735337056275,
+            -0.012544935684845649,
+            0.018297503869683027,
+            0.02355022268131519,
+            0.03273167043015095,
+            -0.024830243320788693,
+            0.0009370814985214882,
+            -0.021937793653233454,
+            0.00535662615596573,
+            -0.035009915581788176,
+            0.0028595659314725012,
+            -0.014395027018699446,
+            0.015596459769560532,
+            0.03397372881171701,
+            -0.013254226072981457,
+            0.011195747181764677,
+            -0.016299384375589934,
+            0.00730244731485402,
+            0.00569944090461748,
+            -0.012916284280345044,
+            -0.005812311882399539,
+            -0.022902815152116493,
+            0.020140276669979015,
+            0.005809799950176385,
+            -0.0008130655714656685,
+            0.0104085826652653,
+            -0.044723432238900435,
+            0.03286429597337671,
+            0.008730225944501239,
+            0.017599189944921396,
+            0.0029762446259779998,
+            -0.0003068671951232415,
+            0.011611551606224214,
+            0.011228401175698422,
+            -0.016029727902740706,
+            0.004345913009102136,
+            -0.004613061137378433,
+            -0.03850533909474302,
+            0.009363351681220008,
+            -0.03631184057978666,
+            0.014427784677340159,
+            0.016026872002233575,
+            -0.004170582842643461,
+            -0.025166189571409366,
+            -0.005428471012333673,
+            0.010827910811085384,
+            -0.009783069664327864,
+            -0.009987805234276443,
+            -0.04163358183060102,
+            -0.025236440364963034,
+            0.034097508113604576,
+            -0.002119474613596975,
+            -0.025826403511707147,
+            -0.010830738112107807,
+            -0.011351132636491575,
+            0.020207687645133,
+            -0.006542026283946858,
+            -0.0006263975804936843,
+            -0.04150011545311013,
+            -0.0073396962917864515,
+            0.011192225907726928,
+            0.027746213794734322,
+            -0.035970017302005566,
+            0.02708126943501373,
+            0.002105633542220187,
+            0.014262674845755191,
+            -0.01626664983003942,
+            -0.005631805361903626,
+            -0.013990539604372344,
+            -0.035877093361375685,
+            0.016487723013906077,
+            -0.008607758248055054,
+            0.004714436224135091,
+            0.04458009880881818,
+            -0.0036872068506712727,
+            -0.009466125348499464,
+            0.007600705817195143,
+            -0.0034590883553048063,
+            -0.010351576085287273,
+            -0.0028564505793999295,
+            0.001959661298507,
+            -0.007080697863126256,
+            -0.04183476693971616,
+            -0.00499888976530217,
+            0.04684391782633148,
+            0.006361744381879857,
+            0.013276793765092465,
+            0.013516220521120684,
+            -0.018335746557932003,
+            -0.045577810515431215,
+            -0.0071616111758423025,
+            -0.0032763880176386808,
+            -0.022708028392758567,
+            -0.0240284945459861,
+            -0.008735040856657758,
+            0.005844145402902923,
+            0.006359368165686724,
+            -0.01120763599008222,
+            -0.022676131361171057,
+            -0.027085968700452136,
+            -0.010526103176253857,
+            -0.02123944716136792,
+            -0.010792145796733375,
+            -0.027142270190771887,
+            -0.028742234538796804,
+            0.02880426953386151,
+            0.007575182232859564,
+            -0.004084088301176117,
+            -0.010005765476121966,
+            -0.01255640546813973,
+            -0.0004774761617900028,
+            -0.0025307994917461232,
+            -0.005822065709999144,
+            0.0038614286313282164,
+            -0.011390482620811868,
+            0.023152086963976554,
+            0.0021220701260546895,
+            0.018329162172128138,
+            -0.05772324571231113,
+            -0.002709083851090071,
+            0.021497896395795905,
+            0.0033025410488410913,
+            -0.005412445141910314,
+            0.029352650459792132,
+            0.011223947499453582,
+            -0.006539460282439711,
+            -0.031355428613718234,
+            0.020942976176820262,
+            0.024122993515785947,
+            0.00817615634503138,
+            -0.012994160510030828,
+            0.01221095914384204,
+            0.01629778384790159,
+            -0.004494985018246821,
+            -0.011161098655072933,
+            0.005586857967039181,
+            0.020941039642372112,
+            0.013580622629911925,
+            0.0013398071867077221,
+            0.030905022925366163,
+            0.0006862084391508121,
+            -0.03521231612862609,
+            -0.01908191995763771,
+            -0.01275995164030872,
+            -0.003584331305986364,
+            -0.015264425022763126,
+            0.005294329654809353,
+            -0.002355465010876059,
+            0.006335052149475905,
+            0.022655379849785256,
+            0.03593231939948758,
+            0.0146937743203089,
+            0.00937330328448886,
+            0.003167462724350819,
+            -0.0013235176101076284,
+            0.012102872918129403,
+            0.008638011773075119,
+            0.0350701240268404,
+            -0.014653221697693864,
+            -0.011003994230902835,
+            -0.0037767288519369537,
+            0.010100131061186566,
+            -0.004113527518662366,
+            0.020414393256136372,
+            0.03274709615649469,
+            -0.0037222949335916876,
+            -0.005604552180246868,
+            -0.006160118231975786,
+            0.0041467454414452565,
+            0.01807879127455274,
+            -0.01252134965321004,
+            0.004364858987501461,
+            0.005549300665159658,
+            -0.0071897162833861,
+            0.018567059255813558,
+            0.0002622074720860951,
+            -0.012983550838417323,
+            0.03266376564402191,
+            0.11449509387240629,
+            0.0017251916835047628,
+            0.007125215166916139,
+            -0.040229997815548584,
+            -0.020537987527148888,
+            -0.00245728735683473,
+            0.005779972828836593,
+            0.007004732428236457,
+            0.02225914982152608,
+            0.006505827715591914,
+            0.0006708045351142873,
+            0.031708522931495536,
+            -0.017334083511709092,
+            -0.002522861743921599,
+            -0.01745656559688129,
+            0.013890451996979048,
+            -0.014312605700272691,
+            -0.013784963572164409,
+            -0.01572417853955075,
+            0.004507425871748977,
+            -0.000601158156253668,
+            0.0016158265715634729,
+            0.01791303234734911,
+            0.0011926917387973374,
+            -0.026393351252156246,
+            0.0006694008039871878,
+            0.001026031515214626,
+            -0.004514419527575117,
+            -0.011320929719992686,
+            -0.02223001723942815,
+            0.016235841442653918,
+            0.014188311619681008,
+            -0.004895218832118094,
+            0.009244697376301004,
+            0.010558796153959677,
+            0.02707485418319703,
+            -0.028380342401606758,
+            0.0040204430418629405,
+            0.007023041225985294,
+            0.01606556581447933,
+            0.017617862031190843,
+            0.0034870847111552862,
+            -0.011253315705053224,
+            0.03238289399766116,
+            -0.010563282687352038,
+            0.0065093536270233385,
+            0.014966802208644462,
+            -0.03541611540945151,
+            -5.821682910348165e-05,
+            0.0061985444938402745,
+            -0.011716201661363997,
+            0.009580035961315467,
+            -0.03120692534097433,
+            -0.0039291243987978845,
+            0.009317468331974374,
+            0.017611312558635673,
+            0.008804996982574765,
+            -0.0034129169537151925,
+            0.010822226698822738,
+            0.012510661458054741,
+            -0.0027147975086409966,
+            -0.008855213644027594,
+            0.01238502686635956,
+            -0.0007687053446954701,
+            0.00450108863711973,
+            -0.00799170686059026,
+            0.0029755595812552293,
+            0.002290650917244691,
+            -0.01554856300008412,
+            -0.011779430686394945,
+            0.00438393126679289,
+            0.0036931891075561957,
+            0.001535662168147833,
+            0.012014408636737167,
+            -0.006087255045485002,
+            -0.0077816220325470226,
+            0.0040573724156794284,
+            0.003136943055301706,
+            -5.505014316363865e-05,
+            -0.030625245024561508,
+            0.01585923084309149,
+            0.0067595435845396405,
+            0.010013308598626031,
+            -0.005315626260228517,
+            -0.00903142414091241,
+            -0.007240920015018893,
+            -0.01228521172101564,
+            -0.010657894223173453,
+            0.007658605592936815,
+            -0.00724924993432376,
+            0.011124174211132038,
+            -0.03128027787357322,
+            -0.013988715435985316,
+            0.01231792993245938,
+            -0.01528161242159377,
+            0.0007675821323979293,
+            -0.008389914312201402,
+            0.0009417242487395995,
+            -0.005119304379761705,
+            0.007887436211143328,
+            -0.03731503169547854,
+            0.007878001990271541,
+            0.02456170736765724,
+            -0.00330924625934981,
+            -0.020259771366167557,
+            -0.009613930457931786,
+            0.0241314385136084,
+            -0.008267831641886344,
+            -0.01645888287387047,
+            0.008421304862524506,
+            0.010140813149093993,
+            -0.0006309118552875393,
+            0.0008237784055549769,
+            -0.003726984191590917,
+            0.01669125388798323,
+            0.02054777616420997,
+            0.041930770964107515,
+            -0.014789610432004559,
+            0.01092349703703152,
+            0.02866938974792608,
+            -0.03378576980722847,
+            0.014728113846952417,
+            0.02518409645155824,
+            0.014590531327650454,
+            -0.0004446956315190408,
+            0.001609777186372437,
+            0.0068939321280085475,
+            -0.014273492115311677,
+            0.009247676157551494,
+            0.013380104440744367,
+            -0.018407746260686045,
+            0.020444430509441124,
+            0.03140071554008003,
+            0.0025839998101985736,
+            0.01355205268903058,
+            -0.006268939830271642,
+            -0.013860692186849113,
+            0.014717364068116991,
+            0.031429975488060896,
+            0.0006919565473572909,
+            -0.018561500446336806,
+            0.019128244642842268,
+            0.006988095806264863,
+            -0.003678529269242034,
+            -0.0021993078588142664,
+            -0.021334919445200388,
+            -0.021003586791369228,
+            0.017304350704792804,
+            0.013508779408333964,
+            0.027794485858413365,
+            -0.010594702782952029,
+            -0.007818089330662161,
+            0.014293688127901867,
+            -0.022368568887004094,
+            0.015426482139881204,
+            -0.02160820059598751,
+            -0.00571499813703566,
+            -0.0077402210272634235,
+            0.03274360648629609,
+            0.001441304806200403,
+            0.02734331677589297,
+            0.016457027764064002,
+            -0.033609420134045206,
+            -0.029790535553657092,
+            -0.00712106584238484,
+            -0.02307933297713104,
+            0.010356561171963577,
+            -0.03901183838032092,
+            -0.00071735150189382,
+            0.008251646656432826,
+            -0.008079917288471352,
+            -0.0007087526632966007,
+            -0.006419191399519291,
+            0.013989344185652506,
+            0.013661633160336152,
+            -0.0228103143826663,
+            0.023644639659577426,
+            0.008343481940133684,
+            0.025998483906975782,
+            -0.02076055376006461,
+            0.019148568238437927,
+            -0.017705363646416224,
+            -0.02777313933338929,
+            0.014358130220942483,
+            0.023585731219788467,
+            0.01635364038245971,
+            -0.007122853965024528,
+            0.007110003583832299,
+            -0.03556761320536796,
+            0.027585457849964405,
+            -0.013064062623866305,
+            -0.012504161289952458,
+            -0.0032379276562094555,
+            -0.011931827532360686,
+            -0.002574161005800279,
+            0.0008533907465694845,
+            0.0034345232246102806,
+            -0.03131462673429805,
+            -0.0002450959360520896,
+            -0.03727416416766574,
+            -0.028944814604815072,
+            0.01360126708805471,
+            -0.00036361284498363273,
+            -0.0049097813612814616,
+            0.02933242705521636,
+            -0.026339136251046315,
+            -0.02693295650849956,
+            -0.009590247544093352,
+            0.02853096456887074,
+            -0.0220673798697511,
+            -0.004519050393279848,
+            0.0015034014871316069,
+            -0.01828226786700525,
+            0.004436383319946364,
+            -0.0012748481042235446,
+            -0.010416065158876917,
+            -0.01105146139270597,
+            -0.03229479099927737,
+            -0.005822133182096844,
+            0.02106215163620738,
+            -0.006957125617688484,
+            0.002746241228040847,
+            0.013312248331456934,
+            -0.010968629118224582,
+            -0.0029946900704643812,
+            0.0017875284441794235,
+            0.023304175718480834,
+            -0.021033941064173063,
+            0.007714334541806199,
+            0.005955442031111047,
+            0.0016405586860131207,
+            0.018406562522853148,
+            -0.0054922197063461825,
+            0.0037690571959124617,
+            0.0162126534419016,
+            -0.009969304596173302,
+            0.017959307638726878,
+            0.007691549802475525,
+            0.006681837561745192,
+            0.002775700618222622,
+            0.007712770283579111,
+            0.009752652973700739,
+            -0.013506020126078112,
+            0.016911141425375933,
+            0.022468715726349706,
+            0.00023233555209372766,
+            0.021066580799163234,
+            -0.02302516292580905,
+            -0.028618544628453015,
+            0.018297878810036087,
+            -0.0225078941299981,
+            -0.012735208680311652,
+            0.016459925496937065,
+            -0.0043032724578332855,
+            -0.00797609922711709,
+            -0.014986057073351055,
+            0.02260618561608301,
+            0.02132930921464508,
+            -0.020249788476562527,
+            -0.012385936327068884,
+            0.0072304431443959505,
+            -0.004373475447051833,
+            0.007086656081865733,
+            0.03195810137269375,
+            -0.015982210897392764,
+            0.006818890093901413,
+            -0.0038306138017521626,
+            0.0030753642449792215,
+            0.017830546304667783,
+            0.009470018737705222,
+            -0.0019815241970236146,
+            0.030100238416495066,
+            -0.001330614013444443,
+            0.016392226282459177,
+            0.0004888028477871524,
+            -0.010948833692356176,
+            0.006343890495626752,
+            -0.026652806306173643,
+            -0.008627570055007363,
+            -0.001203530223345183,
+            -0.009581494451831351,
+            0.07598616930784373,
+            0.003930153552069478,
+            -0.01295957554290434,
+            -0.001673007115291578,
+            -0.001368287102181597,
+            -0.006897179139643672,
+            -0.00960679451603541,
+            0.0287166325268306,
+            -0.016613613331623346,
+            -0.01894527592131826,
+            0.0008370014132002703,
+            -0.006785618888421608,
+            0.00919924251249592,
+            0.01816363936999796,
+            -0.010225961928154075,
+            -0.01604759761423727,
+            0.001577138242530474,
+            -0.003520564936698765,
+            -0.004395434431595494,
+            0.0052399043934902014,
+            -0.010127122101423486,
+            0.006456662777583096,
+            0.026560491731327886,
+            -0.013684536319659296,
+            -0.01425196172663986,
+            0.004084982328707292,
+            -0.0010682084059972982,
+            0.009758641913073978,
+            -0.00864879062601467,
+            -0.00481222050200678,
+            -0.014375330954464725,
+            -0.011378890549403524,
+            0.02505641780042516,
+            -0.02861164504722546,
+            0.0034060847953066354,
+            0.014937227531980433,
+            -0.00016904224334342915,
+            0.01541381851648916,
+            0.0033028855218525644,
+            -0.03615935765718852,
+            -0.013677774380513626,
+            -0.02073839691250734,
+            0.019821074228011413,
+            0.01688422012576042,
+            -0.0033731840788147442,
+            -0.013840993586441808,
+            -0.026149880178874887,
+            -0.007430563876740068,
+            -0.02466234818567362,
+            -0.020708282040996014,
+            -0.019276991448222817,
+            0.006137002799644838,
+            0.005162892426268572,
+            0.022881759407670583,
+            0.0025854880108157857,
+            -0.015058889448404231,
+            0.0029948608544278386,
+            -0.006722836436935071,
+            0.011845298212977357,
+            -0.02747944486818378,
+            0.015311272959213262,
+            -0.01610493464983465,
+            0.025239768880017402,
+            0.007425208902253236,
+            0.0010569008485112994,
+            -0.008822681490988575,
+            0.00010704167529046774,
+            0.014883787023445807,
+            -0.004186968199973372,
+            -0.0007781259002896265,
+            0.02787555889813697,
+            0.02087373813833492,
+            -0.0017783425135865238,
+            0.008743718448136432,
+            0.028026471206552342,
+            0.019081083283793984,
+            0.0325261942213039,
+            -0.013815526479760766,
+            0.005113676604826083,
+            -0.028873600114081546,
+            0.010510475843707614,
+            -0.010871039729515013,
+            -0.010201824683340651,
+            -0.0067946599549165735,
+            0.027756887504389347,
+            -0.007962601205446472,
+            -0.0006687976124826922,
+            0.03469894503563926,
+            -0.014185294847328063,
+            -0.00010543064621374546,
+            0.0007386418834840544,
+            -0.009759481309152999,
+            0.02742722730386853,
+            -0.0073590367291510046,
+            0.018456032721630843,
+            -0.019560882612141943,
+            -0.013705917411254969,
+            -0.011894560208199602,
+            -0.001142768182039817,
+            -0.02576754877141608,
+            -0.008755240066826174,
+            -0.02249287216087102,
+            0.00859172262856532,
+            -0.0046012388553094375,
+            0.018845920253446127,
+            0.0011617844393195566,
+            -0.005275799428018491,
+            0.029069367036565145,
+            0.0088080893110627,
+            0.018100411389202663,
+            -0.0004971948011851863,
+            0.01216566778850985,
+            0.02369448391321832,
+            0.002637001729688794,
+            0.005012948614448268,
+            -0.024213704079796478,
+            0.003420184574622472,
+            -0.0004359900462023086,
+            8.265488264269535e-05,
+            0.016267948861008616,
+            -0.019572032453100303,
+            -0.02216264534756073,
+            0.021871880626992544,
+            -0.012742926070754964,
+            0.011676769226089007,
+            0.0035362870996336194,
+            -0.010732229460038356,
+            0.0006838777488770268,
+            -0.03083438636386314,
+            0.008820746379709355,
+            -0.03216685858285627,
+            -0.032230808828686675,
+            -0.013547681710236494,
+            -0.009994106571421729,
+            0.0008758088338659712,
+            0.028447187234357855,
+            -0.00795317029726568,
+            -0.015797892954136448,
+            0.005363565557248085,
+            -0.011199252697624111,
+            -0.00817162221789202,
+            -0.0007867892785855576,
+            0.005856939999700809,
+            0.009461959355310273,
+            0.005997804620690314,
+            0.011648335019189696,
+            0.007491946658046298,
+            -0.008028800679169293,
+            -0.008837725220736191,
+            0.015889923814605387,
+            -0.0014409716087425222,
+            -0.01176424391171685,
+            0.030430130082762363,
+            -0.013300620185884817,
+            -0.0005592926632214984,
+            -0.0021975669766032472,
+            -0.0015687208316663597,
+            -0.0021652076181411244,
+            -0.014225289143358926,
+            0.01870568404189425,
+            8.334501002782455e-05,
+            -0.014288806218067678,
+            0.0032841743905975634,
+            0.003841780428153241,
+            -0.0036498339483411074,
+            0.008388922960269383,
+            -0.008147275785908849,
+            0.017348147498892,
+            -0.015113395971380741,
+            0.008740452410210708,
+            0.0038335161236289234,
+            0.0043482078469644385,
+            -0.0065755348640655865,
+            -0.01592809184384243,
+            0.009642593996701513,
+            0.012051872087663658,
+            0.02173264012652684,
+            -0.02525660485859878,
+            0.00876161653901566,
+            -0.005374533920781607,
+            -0.014534179652013885,
+            -0.016127866637369738,
+            0.00982303248417706,
+            0.022322910435388785,
+            0.0063086346898091296,
+            0.012139263756190026,
+            0.00013383660542132047,
+            0.005848259030010402,
+            -0.014767198851314148,
+            0.010570158368602513,
+            -0.011355774405066537,
+            -0.018439261453084426,
+            -0.005243587266408389,
+            0.028894120979627323,
+            -0.005492024421080143,
+            0.017938747701972086,
+            0.0021350711250093858,
+            0.029007356490801176,
+            -0.008686781541806358,
+            0.0014883465513564527,
+            0.0039866022469891155,
+            -0.01295271525889559,
+            -0.004272008629260297,
+            0.01976273843646234,
+            0.04882868638601926,
+            0.010204475902027572,
+            -0.0009155109545661691,
+            0.010069071220988797,
+            0.005626418787776309,
+            0.002148101019776153,
+            -0.00805781810841286,
+            -0.030431071067211577,
+            0.008883707378543252,
+            0.005628218215212441,
+            -0.01997825125961566,
+            0.019838234890736068,
+            0.026483365072033395,
+            0.023040623182975964,
+            -0.011585643598171132,
+            0.016539930468533123,
+            -0.010092529089936188,
+            0.0177008735686489,
+            0.028665424114962996,
+            0.013810061902648515,
+            -0.010487150104822674,
+            -0.0020847870330548534,
+            0.006329342599564054,
+            0.004186981876004016,
+            0.00026239408223817384,
+            0.015190614519766204,
+            -0.019155180136775043,
+            0.01005728016778505,
+            0.014725826697964637,
+            0.03322436221319799,
+            8.049930133627564e-05,
+            -0.01645191835557271,
+            0.0016249281996128196,
+            -0.009976155662939693,
+            0.00856399492362911,
+            0.000632427127231252,
+            0.0032242260692672124,
+            -0.009688119824276678,
+            -0.012682779503915694,
+            -0.002851918183031152,
+            0.009403205508474241,
+            0.0021188716361101698,
+            0.002759004011231831,
+            0.007641925150177999,
+            0.0022830062031146204,
+            0.01713024474384216,
+            -0.0011181980841546123,
+            -0.00029141889484139217,
+            0.004601090256674874,
+            0.010509199590505558,
+            -0.01649852835819883,
+            -0.017548369964263514,
+            0.004337237475053179,
+            0.012217494210546133,
+            -0.013159179948224596,
+            -0.013241981632059755,
+            0.015704543608959117,
+            -0.01913575130892795,
+            0.024934102868065366,
+            -0.00942381892138331,
+            -0.004543611336526587,
+            0.000343200128934031,
+            -0.00727538880611703,
+            -0.0038751118497522673,
+            -0.0030064279504724797,
+            0.003984715814505561,
+            0.0019261818110367147,
+            0.007472350699291607,
+            0.008472369788801792,
+            0.014568547873264247,
+            0.006043895352236181,
+            0.0032580830408778395,
+            0.009109104498825864,
+            -0.014523846294121576,
+            -0.014800649887278973,
+            -0.002704791598011488,
+            -0.008895394038700809,
+            0.00902016867266151,
+            -0.006607182154874112,
+            0.00845408382182461,
+            -0.015501867835442603,
+            0.015573642225745032,
+            -0.012200552219016517,
+            -0.0008900215712478553,
+            -0.009666233545144905,
+            -2.7470277092642562e-05,
+            0.008191697904514857,
+            0.014605287653054787,
+            0.0070552942843382215,
+            -0.031921405225964206,
+            0.0017403427534633655,
+            -0.008846568352320488,
+            0.03446319360336241,
+            0.002612606662691094,
+            -0.01060863123202008,
+            0.020352181356568008,
+            -0.01592124132155158,
+            -0.00027704668715660693,
+            -0.01602075523652062,
+            -0.004797557413047212,
+            0.00707939446796679,
+            0.016399017978990946,
+            0.02759049512664933,
+            -0.005697020356865688,
+            -0.007995305872076221,
+            0.006384047872862793,
+            0.029590696670334232,
+            0.012246955508890425,
+            0.024748351124415785,
+            -0.007633712004672087,
+            0.014348228716100414,
+            0.007773260552660029,
+            0.0076045488653348975,
+            0.004321689250440021,
+            -0.002591801797145718,
+            -0.011698574606499644,
+            -0.014974971814360001,
+            -0.0037563183330623384,
+            -0.016987915346053498,
+            0.00012247833530362867,
+            0.021344400276501117,
+            -0.00475351423959752,
+            -0.0033847161875730026,
+            -0.01874186437694691,
+            0.0033461670218611346,
+            -0.010029445302549983,
+            -0.00367421246775698,
+            -0.013133469163793961,
+            -0.0254122035745965,
+            -0.018556369080030412,
+            -0.002155244024999317,
+            -0.00929309186646207,
+            -0.015194269781263309,
+            0.0070354690219415535,
+            -0.016305117309271634,
+            0.0080837166473439,
+            -0.00448473417024141,
+            -0.016508775226236883,
+            -0.00958925666995396,
+            0.01100335153658195,
+            -0.0041819993634961685,
+            -0.01929493648733792,
+            -0.01282300765645703,
+            -0.011300260564838231,
+            0.0051733704121938884,
+            0.005996780699975213,
+            -0.0038111648304178544,
+            -0.0028279639348479987,
+            0.016640121845366295,
+            -0.010197365811240557,
+            0.009878730312873492,
+            -0.03335474019383245,
+            -0.03551826040406959,
+            0.004497847402931096,
+            0.01507007013703286,
+            0.0013085881889541252,
+            -0.02078680101065954,
+            0.010806315988966003,
+            -0.016499186313125556,
+            -0.00858013792516617,
+            -0.008522534127498645,
+            0.009649521088411411,
+            -0.007283003380127847,
+            0.01299542385123007,
+            -5.3226845309105124e-05,
+            0.030477683597373687,
+            0.0074153455615142755,
+            0.004161881195438522,
+            -0.00679081048132963,
+            0.008354912600159931,
+            -0.001621776416108105,
+            0.004080191054849395,
+            0.02120524874226696,
+            0.012440848502230418,
+            -0.00218041171518331,
+            0.006458415098595872,
+            0.005678055538564355,
+            -0.017186537377121235,
+            0.02507960402044456,
+            0.024678178106806115,
+            -0.008205288425409454,
+            0.004108402466376321,
+            0.005783335302169668,
+            -0.008846663924040241,
+            -0.006897707231869481,
+            0.006527795262026734,
+            -0.018474181903007085,
+            -0.0008721381678008483,
+            0.010598084358464908,
+            -0.02255441909004843,
+            -0.004840511019512733,
+            0.021745651362109403,
+            0.004631567631335321,
+            -0.018558972627958834,
+            0.02764788406651861,
+            -0.004942361238163609,
+            -0.0027290999934532494,
+            -0.010876135202650048,
+            0.01671497143541799,
+            -0.012173527207375091,
+            0.008443317346655462,
+            0.011021541861818043,
+            -0.006295639322287843,
+            0.003083685676540999,
+            0.008277206401503182,
+            -0.004504302913715969,
+            -0.0068075424927125676,
+            -0.0010508416962215848,
+            0.010167160518175114,
+            0.0036980815941666336,
+            -0.022704963865525812,
+            -0.006020073261002279,
+            0.02934227992246887,
+            -0.015082922782242507,
+            -0.0050916398557337705,
+            0.005376294716056329,
+            0.007709342013963906,
+            -0.003476236561444464,
+            -0.0023399991233097337,
+            -0.00787486645740744,
+            0.010356414979428607,
+            -0.015950083845400274,
+            0.02076360898078935,
+            0.025608263494550438,
+            0.024504236990840842,
+            -0.014814235990238182,
+            0.010828584133720797,
+            -0.011703992394890898,
+            -0.011047470905668532,
+            -0.015542835222173864,
+            -0.001658296303013357,
+            0.001970208852236451,
+            0.013107754130616248,
+            0.009666887546609378,
+            0.0235853683657408,
+            0.006473544089109194,
+            0.008973300312205858,
+            0.014118601565410305,
+            -0.003902202854013708,
+            -0.018303438244719996,
+            -0.014584226585460755,
+            -0.011977613285028946,
+            -0.01756345491020528,
+            0.0004563602312336091,
+            0.020461659996271687,
+            0.012417744207683839,
+            0.024136626592375227,
+            0.002119213701333967,
+            0.013759902247467922,
+            -2.9525392347047554e-05,
+            -0.000892319792935443,
+            0.00045783825646242434,
+            -0.005238483120312843,
+            0.019255756091120846,
+            -0.004599752813817219,
+            0.01656589522110913,
+            0.0006443773300684925,
+            0.007133940816771615,
+            0.013554761340256566,
+            -0.01704388669741371,
+            -0.003443100892108368,
+            -0.010029620309829285,
+            0.000941391966877521,
+            -0.019633111750884034,
+            0.01934571074127927,
+            0.010273330977233852,
+            0.0017121345993533963,
+            -0.018327189396014198,
+            -0.0029706385099949693,
+            0.03448635131998772,
+            0.019458999453011265,
+            0.0050734354363614525,
+            0.02632443373898792,
+            0.013018762352749236,
+            0.014443611411502168,
+            0.010153370950346075,
+            0.02356766253713842,
+            -0.023479811300626968,
+            0.02095255383200973,
+            -0.011590347850340786,
+            -0.007414136601134069,
+            -0.0035774888741646705,
+            0.0025141508183687115,
+            -0.013588691733097813,
+            0.00982877622088977,
+            -0.0017916539399655826,
+            -0.025386211797192578,
+            0.009164316443206016,
+            -0.008811052979979271,
+            0.010464442616928296,
+            -0.007774569252594923,
+            -0.0002806136360286729,
+            -0.0127084026190163,
+            -0.015184317235990752,
+            0.001259201880504347,
+            -0.009158429132281081,
+            0.023908150915412617,
+            -0.021210848953875233,
+            -0.01321384014436847,
+            0.013997502713857512,
+            -0.006869443694669701,
+            -0.011364239609894453,
+            -0.00374449980963666,
+            0.023673050288879224,
+            -0.009243341446512421,
+            -0.01359367519009039,
+            0.014464166953530989,
+            0.003834735113400469,
+            0.0059713317014342395,
+            -0.007508862694272452,
+            -0.008428705474135382,
+            -0.003280788036976103,
+            0.005998840043285335,
+            0.026615077617284676,
+            -0.014059930972065399,
+            -0.0032833264512892022,
+            -0.009189999799498163,
+            0.004118204537632102,
+            -0.02359889635281631,
+            0.02771075351384059,
+            -0.008871395303200375,
+            0.021705177758560575,
+            -0.0015924449308442096,
+            -0.006239188158271628,
+            -0.024231036074349822,
+            0.027522945599928003,
+            0.003683870966132199,
+            -0.010221180899173243,
+            0.0012768725460662862,
+            -0.052005292736885915,
+            0.016636109796947025,
+            0.008429895913163985,
+            -0.019507994429834353,
+            0.003697519981610208,
+            -0.004773584856005853,
+            0.018983847580558853,
+            -0.022966993228850697,
+            -0.014078986007109447,
+            0.008283766047449966,
+            -0.01153285124362746,
+            0.0031439180465649996,
+            -0.02527749501821793,
+            -0.005754537200570427,
+            0.0011760345167229269,
+            -0.005276936074280004,
+            -0.001705933750221904,
+            -0.020022761608132548,
+            -0.025972227850795115,
+            0.017666017379157135,
+            0.015820875579600418,
+            -0.002278262448951789,
+            0.01320818011561616,
+            0.008709899932124187,
+            0.00836573261091119,
+            0.016034263394940008,
+            0.007124509234103267,
+            0.0081473711608784,
+            0.045498533239850446,
+            -0.019846323044145485,
+            0.007371629414298193,
+            -0.0014854149174929534,
+            0.013951741898629716,
+            -0.030467924817555014,
+            -0.0055144480998344705,
+            0.0062802744565489945,
+            -0.0049373689340857915,
+            0.03510454786147091,
+            -0.017326094216003073,
+            0.01694160526735264,
+            -0.0026075809870445238,
+            0.0007420394265014909,
+            -0.02042003877666893,
+            -0.026156937726134134,
+            -0.014294938857971834,
+            0.0011857406889443987,
+            -0.0010047941261608975,
+            -0.011946583993069242,
+            0.006797001710574735,
+            0.004917161312925866,
+            -0.0020713379451245515,
+            0.015652523103715316,
+            -0.02952061566240878,
+            0.007356800041369564,
+            -0.010733578620243498,
+            0.020008504830359524,
+            -0.00990331048284697,
+            -0.0013319400098308093,
+            0.007236960885181047,
+            -0.004122868383988748,
+            0.013216043472071927,
+            -0.0038337251451538677,
+            0.004227115653354611,
+            -0.005221321656766545,
+            -0.01794343569722877,
+            -0.0016435261495179767,
+            0.021429008432348098,
+            -0.008349684096540855,
+            0.031864497493526205,
+            -0.01264743878037828,
+            0.009174982784686633,
+            0.010692990337462762,
+            0.0006628831036334152,
+            -0.005286285851702483,
+            0.009052971814211658,
+            0.0021431970274545893,
+            -0.008690520448278274,
+            0.006934310986093342,
+            -0.018349449937310143,
+            0.008494451785507706,
+            0.03394632739506833,
+            0.0018511328780472266,
+            -0.0007519661670070526,
+            -0.007266663179613392,
+            -0.01335844201823745,
+            0.018540713029383066,
+            0.005488273906733518,
+            -0.002546531152114063,
+            -0.027536660869590543,
+            0.02007746621157705,
+            0.0012265798861291223,
+            -0.004755325069578531,
+            0.012466557901411086,
+            0.0020495867632654885,
+            0.019989531151046736,
+            -0.010171678639113316,
+            0.030757957899447096,
+            0.010050036220124393,
+            0.002959833051227505,
+            0.0038556928367700113,
+            -0.036655749074851536,
+            -0.01652152778069314,
+            -0.00019357502472740266,
+            0.014271581054865031,
+            -0.012430647252826173,
+            -0.022432580814844155,
+            0.0010346169306977334,
+            0.01148004051449691,
+            -0.0033898642762881046,
+            -0.009650665305271706,
+            0.01560757541276278,
+            -0.020342213902670575,
+            0.006701302394408393,
+            0.007693672045981921,
+            0.0287100214318104,
+            -0.0020735384554022153,
+            -0.0012324838849990056,
+            0.000626416803904792,
+            -0.0055669860801192935,
+            -0.0008037366126870824,
+            0.017251369734470945,
+            -0.013661316947002466,
+            0.0175926211217534,
+            -0.012735241490550407,
+            0.0064713656749838585,
+            -0.0021092378432037563,
+            -0.010629579860050812,
+            -0.020114807971202787,
+            0.0029609495764307357,
+            0.009378003163375899,
+            0.006920144698428806,
+            -0.010438984881449953,
+            0.010715409812532968,
+            0.012197673689706352,
+            -0.00753962861484439,
+            0.014519905132928041,
+            0.022173989895384192,
+            -0.007687503607783572,
+            0.013633321414009471,
+            -0.0041446334728745095,
+            -0.014056264816568229,
+            -0.009985463824440833,
+            0.023929750617160402,
+            -0.0005563806942667597,
+            0.008756997373618809,
+            0.00388125051641897,
+            0.0007856447663040696,
+            -0.011226530517316183,
+            -0.001653447552601681,
+            0.019294448360333633,
+            0.011831934675073843,
+            -0.0013888534822008192,
+            -0.0010525764356699885,
+            -0.0014566936525653407,
+            -0.009427557739483988,
+            0.02049188741459728,
+            -0.022030783927622997,
+            -0.0019604940727076573,
+            0.00461018148439991,
+            -0.022589816803875618,
+            0.0001863117949313752,
+            -0.024570217962989905,
+            -0.01569952241187498,
+            0.011763033781122343,
+            -0.0022530631823737554,
+            -0.016028810867854907,
+            -0.0003242701040942633,
+            7.424127301806556e-05,
+            -0.007815789940508527,
+            -0.004135500768181258,
+            0.006115909992703774,
+            -0.00034193217510010117,
+            -0.0032262189374874842,
+            0.013430154310506884,
+            -0.009996622196870008,
+            0.015431528911575157,
+            0.0067029980830614145,
+            -0.015702167721800483,
+            0.0021406610839557003,
+            0.018500219544264753,
+            -0.0058910087864916085,
+            0.00787021425357441,
+            -0.014681907249972176,
+            -0.0037495551894426303,
+            -0.005787661351944086,
+            0.006051351977821761,
+            0.024849195780885833,
+            0.012969434450643372,
+            0.01139340445077682,
+            0.017405361843075933,
+            0.004325000715054734,
+            -0.01572297405297993,
+            0.0007416798945675923,
+            0.01682727189828363,
+            0.010464036837245312,
+            0.009804326468113703,
+            0.0008975556667684365,
+            0.0043704860361487635,
+            -0.005142176468800175,
+            0.009317064072855551,
+            0.02915482449687158,
+            0.0013434689612997423,
+            0.008696141010760192,
+            0.017276730345147134,
+            0.027038741396131265,
+            -0.0035088997194497725,
+            -0.007469614319788499,
+            0.008739942310300076,
+            -0.009657991604173173,
+            -0.001081629174976152,
+            -0.017000351183542026,
+            -0.012310275736750868,
+            0.016510505259358398,
+            -0.006084627062230492,
+            0.02848913486719956,
+            -0.0010655806791907225,
+            -0.02026498848361482,
+            -0.01565505608194376,
+            0.001404549479440637,
+            -0.01362132895080652,
+            0.005287566256051299,
+            0.014187265685427415,
+            0.01917046513020831,
+            0.004687191682875024,
+            -0.022258337721085656,
+            0.037364962197399806,
+            0.01757108963349427,
+            0.019377870795760123,
+            -0.019422840506328382,
+            -0.006159311237412833,
+            -0.009077095160676276,
+            0.02116697012944498,
+            -0.00880692095488465,
+            -0.009875674856043954,
+            0.024014332004451508,
+            0.017812606845656484,
+            0.005679806286464216,
+            0.006736609904079586,
+            0.0020048663439319585,
+            0.0025414630253661444,
+            -0.015831593851012047,
+            0.017957902498542212,
+            -0.005858125698347523,
+            -0.0058068554681761945,
+            0.017529514573571298,
+            -0.005989581792993925,
+            -0.0030306021203732347,
+            0.00015839997806822045,
+            -0.019697052145531482,
+            0.011618112886207201,
+            0.008256007601576028,
+            -0.017229774429089857,
+            -0.0014757560402567304,
+            -0.0002404480326945295,
+            -0.014118694740134205,
+            0.02004885146995733,
+            0.01694893356262835,
+            -0.0013248884139405398,
+            -0.0051752815372013315,
+            0.008223868147393114,
+            0.007000679132597393,
+            0.005256194890343006,
+            -0.019496098876977206,
+            0.005230597149342019,
+            0.005103909294824105,
+            -0.0014716649591412774,
+            0.010274346323051634,
+            -0.009030594825985526,
+            0.033740380586893826,
+            -0.006505788197604522,
+            0.013065658941136721,
+            -0.0043634731684193745,
+            -0.0055529072007985125,
+            0.001816020676232658,
+            0.013054468478598695,
+            0.009753501030700457,
+            -0.00970446155762939,
+            -0.004172354962129865,
+            0.018697982092775737,
+            -0.011983156146516074,
+            0.0013300791598046338,
+            0.01564072865804035,
+            -0.004858814041438713,
+            0.004708235083775013,
+            0.008635204790345423,
+            -0.004489505237783172,
+            0.01801512858970922,
+            -0.016482572999490366,
+            0.0023953436660054026,
+            0.010694869684962088,
+            0.02390864159830096,
+            -0.019152187046410703,
+            0.021786596486708207,
+            -0.017951050280323393,
+            0.026538582634436314,
+            0.014484894489694262,
+            -0.0035222259457311444,
+            -0.01946200885147006,
+            -0.00850690913394181,
+            0.011654565129063417,
+            -0.021423803876241532,
+            -0.0007985948078290641,
+            -0.0007290222944686444,
+            -0.021367628543092133,
+            -0.011189364017231132,
+            -0.009961779129778331,
+            0.010701637745810138,
+            0.001918633898887013,
+            0.010380983868185359,
+            0.008002182827196827,
+            0.010784760574518087,
+            0.014405640442760822,
+            0.0008958107769384412,
+            -0.010743001445440941,
+            -0.023795589003655273,
+            0.016987708698679597,
+            0.00438463274382093,
+            0.0007862212695139549,
+            -0.01627468994515667,
+            0.011936651269737248,
+            0.013250643815768723,
+            -0.009385605548926286,
+            0.009121882469425302,
+            -0.03716901103204463,
+            -0.004552808410398574,
+            -0.0035704811925430366,
+            -0.0020309197365173475,
+            -0.00908678158053104,
+            0.01559445415882485,
+            -0.013682507836005851,
+            -0.0009590922904844632,
+            0.00011696985057536548,
+            -0.0074756526903548915,
+            -1.5467890432254378e-05,
+            0.020649928837031005,
+            0.008438785939908428,
+            -0.0037382105257999757,
+            0.010376213224654074,
+            0.008826031303751312,
+            -0.013900359449287676,
+            -0.01527474712786084,
+            0.019035366035884273,
+            -0.018776918681908793,
+            0.0063098418544296165,
+            0.01677916431403883,
+            0.020778813096283988,
+            -0.02891862501830075,
+            -0.0007231469961304435,
+            0.022093939332021185,
+            -0.0012807635799950273,
+            0.006370841109790851,
+            0.008781853138377092,
+            0.006754082980602544,
+            0.0029102159062355374,
+            -0.003711173531879644,
+            -4.843770941622283e-06,
+            -0.003940474588075707,
+            0.007155539022862352,
+            0.004366433868349658,
+            0.036453728071874844,
+            -0.013661129348165794,
+            0.002541308805735001,
+            -0.0058838191638835155,
+            -0.00347401304572066,
+            -0.00593469688967835,
+            0.004407665856950772,
+            -0.010055727994798024,
+            0.004330827357536078,
+            -0.001570861535288093,
+            -0.0003731129422346693,
+            0.004883969598300302,
+            -0.001701673379646408,
+            0.007036892564844683,
+            -0.011863650867472625,
+            -0.0003979727750011258,
+            -0.020174851412609312,
+            0.021117284557875267,
+            0.00011666551282792189,
+            0.003662462447326642,
+            0.03723741246290771,
+            0.00476064510662738,
+            -0.00238355131307445,
+            -0.012880057949435798,
+            -0.013551338766321525,
+            -0.012986444917747435,
+            -0.020703821629987552,
+            -0.0036665673548525388,
+            -0.021006311026247164,
+            -0.0052061406234528915,
+            0.010362862594785442,
+            -0.011540184420787572,
+            -0.004023659825622843,
+            -0.015881589631889473,
+            -0.006531981517815174,
+            -0.03114211633687585,
+            -0.01124607488650473,
+            0.007493228366452009,
+            -0.019149285949049886,
+            -0.0039932870926865805,
+            0.026850121590606017,
+            0.001095360635760988,
+            0.01387696424607248,
+            0.0038058248676879625,
+            0.008117686785954085,
+            -0.0045937472017383955,
+            -0.0005669025036141417,
+            -0.01928244877716143,
+            0.003391980330529026,
+            0.008525354854576925,
+            -0.00361507909530282,
+            0.0070008404454106905,
+            -0.008835727374865623,
+            -0.016950017602980382,
+            0.0021654956715219517,
+            0.009334379529385142,
+            -0.006639140003365139,
+            -0.017587472740268683,
+            -0.011049929670847685,
+            0.010401189523318744,
+            -0.02283429470064187,
+            0.006257145380266984,
+            0.016439581169238607,
+            -0.009363030375624585,
+            0.004651575425738153,
+            0.019809183749755425,
+            -0.011810674576781702,
+            -0.005811328501528397,
+            0.008471992070998505,
+            0.013891259999721588,
+            0.0095606134736748,
+            0.02954498207274139,
+            0.0026596809804008996,
+            0.000561870602466584,
+            0.004889263753490895,
+            -0.004277061142504552,
+            -0.017423200644235665,
+            -0.006243787897080996,
+            0.0038507494787521856,
+            0.012873996671952184,
+            -0.009879417960879032,
+            -0.00048575277472139055,
+            0.01682604801673206,
+            0.004530939990187663,
+            -0.0033901248079550284,
+            -0.011868399508738274,
+            -0.020080331108085103,
+            0.0015580259874809898,
+            -0.01362200460317948,
+            0.015354347040143093,
+            0.003390601069331352,
+            0.0015284610287850575,
+            -0.004239880336806429,
+            0.01828645373058079,
+            -0.005013418527935815,
+            0.012749833447770729,
+            -0.006148890994661865,
+            -0.004587831231448011,
+            0.007986533783001116,
+            0.009166578151006778,
+            0.0004939857995863664,
+            0.05421019830078428,
+            0.002838056604877719,
+            0.01333205648597435,
+            0.013419127695838135,
+            -0.014523733369484934,
+            0.014086525147734823,
+            0.012688569853552613,
+            0.018497344547821693,
+            0.013037819487160016,
+            -0.012353776077273462,
+            0.017354801119939964,
+            0.0024084723134891487,
+            0.003909681065813095,
+            0.0022202438064449653,
+            -0.004528576079628071,
+            0.019594683950225115,
+            0.004070222328547378,
+            -0.0074362443729372105,
+            0.004453618452623641,
+            0.005007543168224267,
+            -0.013105524222311019,
+            -0.0112958915514782,
+            0.027105208848863218,
+            -0.01219417557128091,
+            -0.018749971743148304,
+            0.006706594269933445,
+            -0.02008458683853808,
+            -0.03763686535076864,
+            -0.0021332074289244973,
+            0.009992361098878196,
+            0.002795183524495193,
+            -0.00253695604043486,
+            0.01614298064601032,
+            0.016730443047719683,
+            -0.016985296937992726,
+            -0.018245902728910347,
+            0.0025257780765481825,
+            0.0016333325293570182,
+            0.01861807236979066,
+            -0.023168663546030196,
+            -0.006378864850313035,
+            -1.3795465857380962e-05,
+            -0.006829008132963306,
+            0.004850306884033312,
+            0.004703280112191218,
+            -0.0071714759384722965,
+            -0.0027773519096287217,
+            0.002256846645341137,
+            0.01975383140755998,
+            0.013198754584066554,
+            0.01572971337890945,
+            0.002600741959577911,
+            -0.005033508062449482,
+            -0.016817359058724685,
+            -0.005450011991586895,
+            -0.013478330855299777,
+            -0.009091854345715177,
+            -0.003042052092498012,
+            0.008217376329142566,
+            -0.025053469059115735,
+            -0.011965083373220982,
+            0.0119294363235832,
+            0.00391377331459168,
+            0.0003099203772539202,
+            0.007607672728136279,
+            -0.003830132495364688,
+            0.010331053246451333,
+            0.013823890473448383,
+            0.0076112887952959766,
+            -0.004723377809559611,
+            0.009600411526173948,
+            0.011460781436250457,
+            0.006015477068316837,
+            -0.005072711638895421,
+            0.023859899048048694,
+            0.01821920335673539,
+            -0.01680365156728093,
+            -0.002149107383573353,
+            -0.0027829676356423997,
+            -0.01884308738652242,
+            0.009665830801645276,
+            0.00880322398336952,
+            0.008308268605449346,
+            0.003906369351868086,
+            0.00974531700889854,
+            -0.013875904819275925,
+            -0.0050570811397453235,
+            0.002870120865445989,
+            -0.005729556771404781,
+            -0.01901337075686543,
+            0.000297242372701423,
+            -0.011987192261104132,
+            -0.011327157550660895,
+            -0.01409710922165158,
+            -0.003221240464411861,
+            0.005283472511900512,
+            0.021472187640986972,
+            -0.004410881709548286,
+            -0.001855852749039012,
+            0.01351697599517034,
+            0.010060125186506398,
+            0.0021658738823899905,
+            0.0002206474541239167,
+            0.005418828324605768,
+            -0.0027919130944783397,
+            0.0008094042430809068,
+            -0.0015826797905407809,
+            0.021089276765434876,
+            -0.02016611086095107,
+            -0.009413977569213423,
+            -0.009288078098692112,
+            0.005842197094339927,
+            0.028793032462603636,
+            -0.007165634502373454,
+            -0.010104002083017078,
+            -0.01994315293556678,
+            0.0003786527922888834,
+            -0.07810609175343784,
+            -0.005210636756258548,
+            0.001870513618910238,
+            0.03126380113105983,
+            -0.010817888476436834,
+            0.0033673768058629823,
+            -0.002204487854177595,
+            0.014542299655712928,
+            -0.001760107965206874,
+            -0.0013283746982194259,
+            0.0071954806737858056,
+            -0.005640400818010697,
+            -0.006210989097047217,
+            0.005904163661619133,
+            -9.945941028848348e-05,
+            -0.010266938705916303,
+            0.00057263080955709,
+            0.02209498395848068,
+            0.0159398346608872,
+            -0.006575439759804175,
+            0.01816748552021211,
+            -0.00719294553484007,
+            -0.009838738592324234,
+            0.005639288780502127,
+            -0.0019007683459126327,
+            -0.015369619176147573,
+            -0.011475977270803784,
+            0.009862808084641291,
+            -0.030069890644150284,
+            -0.00010725753918890802,
+            0.00943698424646706,
+            -0.013540245857270492,
+            -0.008293042205008774,
+            0.007212974960781723,
+            0.006708343474532583,
+            -0.0012017968151025369,
+            -0.0078026000239507145,
+            -0.02527526731750743,
+            0.0027578400032925625,
+            -0.002446811971857853,
+            -0.004899702349784173,
+            0.0019960465173086183,
+            0.001479503476855694,
+            -0.012645905858466798,
+            0.01216748847452763,
+            -0.013378877368040743,
+            -0.012053251571418766,
+            0.012508842412556306,
+            -0.00027167241745418584,
+            0.01682680930386839,
+            -0.020607299598265426,
+            0.0010702151063620052,
+            -0.028075439824932304,
+            -0.007671848153932823,
+            -0.015486630894165054,
+            -0.0009776136330459065,
+            -0.0012939921569540803,
+            -0.03481880531471923,
+            0.012889447665794033,
+            -0.005200149829198305,
+            -0.006094702500154477,
+            -0.00459778303079732,
+            -0.024583504817899775,
+            0.006072281238752663,
+            0.00607350167867964,
+            -0.006506508970816635,
+            -0.016816439635445445,
+            0.008428048142588516,
+            -0.0011040593013238872,
+            0.004873670535200424,
+            -0.007475800310154908,
+            -0.0005638920132076605,
+            -0.013047135093134946,
+            -0.005463454269764494,
+            -0.0008522562324122834,
+            0.01827123304871433,
+            0.0044055851550317835,
+            -0.014526381222031981,
+            0.016862445236481526,
+            0.013250692156820124,
+            0.01602714293656341,
+            -0.0071089293427753115,
+            -0.0004265739315250755,
+            -0.02750630108787212,
+            -0.00327225092954903,
+            -0.023687644154808972,
+            0.00041213204365507744,
+            0.009447660230507965,
+            0.0203905264889617,
+            0.012694421477479161,
+            0.004620617778996077,
+            -0.001557511764988156,
+            0.0066751090724502,
+            0.02696959560423252,
+            -0.01099183126742224,
+            -0.006969111598423376,
+            -0.01884504192789845,
+            0.01113023717919988,
+            0.025135748541199255,
+            0.003381303139273845,
+            -0.011317957771935042,
+            0.008112598754655146,
+            -0.0009247845526290428,
+            -0.010302980598863016,
+            0.02889587382004902,
+            -0.0391134481322538,
+            -0.015599591108049343,
+            -0.004605824192962023,
+            0.007406520608110251,
+            -0.015596917115435033,
+            0.00770468362805995,
+            -0.012614339656105264,
+            0.001699049329817941,
+            -0.005851477164035241,
+            0.03535087340659861,
+            -0.0031950960727572925,
+            0.0006909856233673865,
+            -0.003395233050301427,
+            0.003531135044131077,
+            0.012355948349937309,
+            -0.004444221317023453,
+            -0.030726300006909638,
+            -0.0022983766869702346,
+            -0.011198252640110788,
+            0.009274666307515955,
+            -0.016207714291658314,
+            -0.017342435976846468,
+            -0.009424737744066259,
+            -0.021827243671837953,
+            -0.012403084982789827,
+            0.029747453318486338,
+            -0.005824001042844527,
+            0.01747569398142269,
+            -0.005219060548639168,
+            -0.008365003057800934,
+            0.00021272411948388247,
+            0.01344335658390732,
+            -0.028498240040284714,
+            0.00041963499785238845,
+            -0.002322931702641349,
+            0.0126393550661767,
+            -0.001595134597082556,
+            -0.00983073906977773,
+            0.006903313245694627,
+            0.0010522208685011098,
+            -0.015648940549713764,
+            -0.0007573245570947632,
+            -0.008533504878052644,
+            0.020475329445149498,
+            0.00027791387213409814,
+            0.001074213629646005,
+            0.013864331671732542,
+            -0.02102352993295736,
+            0.01576938071930521,
+            -0.021722775924283996,
+            0.01893267442161547,
+            -0.006577103101038035,
+            0.005772016664149475,
+            0.008283223343661555,
+            0.014068789332250016,
+            -0.014030042849491258,
+            -0.00663752723080222,
+            -0.034436220972634726,
+            -0.0122621879114314,
+            -0.001946072976826902,
+            -0.005308941654794638,
+            -0.014773123927909993,
+            -0.0011054822266082056,
+            -0.026205461509573584,
+            -0.006460373065662098,
+            0.012822188046808264,
+            0.019900312098359022,
+            -0.008409070094265829,
+            0.014163132434889265,
+            -0.006743711872055091,
+            -0.0028230771810882492,
+            -0.012652021749331811,
+            -0.018061180380009395,
+            -0.01083672324723357,
+            -0.01890966636827258,
+            -0.0018635408070944168,
+            0.004709222372611061,
+            -0.015760618199235518,
+            0.011435131737642627,
+            0.03198874888333405,
+            -0.01567271472650171,
+            0.008243406188694909,
+            0.016411496328333205,
+            0.0034236971031973503,
+            0.011728079139322837,
+            0.0038311415233052795,
+            0.004621789116267879,
+            -0.011109450058055078,
+            -0.011473158133621661,
+            0.006524490579223986,
+            0.00954511918454185,
+            -0.004913142601134428,
+            0.024347072255007807,
+            0.0008745419206472714,
+            0.005045794664611216,
+            0.005304506902446091,
+            0.011488187043525824,
+            -0.013135271908307747,
+            0.019746484188715552,
+            -0.002055210086078267,
+            -0.015571665869889405,
+            -0.01535499169234956,
+            -0.01105419264826655,
+            -0.018626399757822328,
+            0.0017108083455869296,
+            0.0181439079855921,
+            -0.0012493542718277685,
+            0.012137753522284387,
+            -0.00924460959150499,
+            -0.005019437549649743,
+            -0.0027156377206998474,
+            -0.038211954658821295,
+            -0.0021878970332906457,
+            -0.018809487178512628,
+            -0.02498945889484935,
+            0.03505216568400238,
+            -4.26477081213836e-05,
+            0.00997643255528116,
+            -0.010310197159232386,
+            -0.0002454902098986129,
+            -0.011065403358802515,
+            0.014954998647970473,
+            0.004624123403100598,
+            0.008872923678983478,
+            0.008560280683876702,
+            -0.016419500515801093,
+            -0.002997557798375716,
+            0.007472390947008284,
+            -0.00374822278027057,
+            0.012680407710694952,
+            0.015353684671120731,
+            0.020886474047003733,
+            0.011161262397502876,
+            0.002045862158732102,
+            0.00887271411363814,
+            0.009563361150297699,
+            -0.0003513237086717377,
+            -0.010459621030424415,
+            0.015419039131459582,
+            -0.02204848739950703,
+            0.012172257374410352,
+            0.04346634265794738,
+            0.019197433209794286,
+            0.029488039318675732,
+            -0.02272810439246353,
+            0.01448092574459596,
+            -0.003801043409441206,
+            0.009476229966122152,
+            0.0049309968883087685,
+            0.02346191840355209,
+            0.003973840862678076,
+            0.0073546443568171405,
+            0.010647581332452525,
+            -0.00972051412996416,
+            0.025644022498404723,
+            -0.00846669916801108,
+            5.1518540791212095e-05,
+            -0.004407265804145216,
+            -0.026400029105519137,
+            0.01615193008014211,
+            0.0081722536401398,
+            0.0010702314763197463,
+            0.0010263623479474385,
+            -0.0019566604229607643,
+            0.007982949510899019,
+            0.007079322685771456,
+            -0.015274148711633536,
+            -0.003886053737590683,
+            -0.018527455865296014,
+            0.0012864569064749438,
+            -0.024337133883771334,
+            0.010466651250597847,
+            -0.017568706814132484,
+            -0.014822354159709854,
+            -0.021666681078831483,
+            -0.013232016867264097,
+            -0.02011653224168991,
+            0.0035858355517491515,
+            -0.008632755352062642,
+            0.013025728891203198,
+            -0.016962939287139683,
+            -0.021494971929779273,
+            0.008430887589398292,
+            0.00870802359711456,
+            0.007481282677955534,
+            0.009728368603458082,
+            0.014996157358749115,
+            0.01736292711410622,
+            -0.022712386143092265,
+            -0.023163001611599246,
+            -0.006682209717364464,
+            -0.003531495447167166,
+            0.006762379833125297,
+            0.00933681196321128,
+            0.012884185500343249,
+            -0.007077895849501418,
+            -0.007554762054641417,
+            -0.013283067570021503,
+            0.006197966297581871,
+            -0.003993067576677973,
+            -0.009464789349849612,
+            -0.0015662739825834614,
+            -0.0061389795593180686,
+            0.002753664732282926,
+            -0.0001791088888074743,
+            0.0007595261237141144,
+            0.017122834676182343,
+            -0.008176319370751018,
+            -0.009787747261447871,
+            -0.02027228453943018,
+            -0.010683273300243025,
+            -0.015604745521156111,
+            -0.0016279406232527138,
+            -0.006506713191861036,
+            -0.014420323628589382,
+            0.010888206569219648,
+            -0.011930708571093279,
+            -0.00887210117616784,
+            0.010807147705360856,
+            0.0073900910661282935,
+            -0.003431679006289002,
+            0.037804020313161775,
+            0.0011945597874820288,
+            0.00778778394048267,
+            -0.003366543306746973,
+            0.005296727102519836,
+            -0.017761251541598876,
+            -0.026382400396890007,
+            0.0010357716464853195,
+            -0.01775777668381489,
+            0.010807713750049491,
+            -0.008515368407459745,
+            0.0016051389544388124,
+            -0.00034958600424912995,
+            -0.009585854918521076,
+            0.0036549720706351473,
+            -0.008110927907950148,
+            0.004212219431447508,
+            -0.03507890639477636,
+            0.011409377585617926,
+            -0.01819120811524444,
+            -0.008962242411035718,
+            0.01710177594046791,
+            0.037560936862121835,
+            0.0028513388998398104,
+            0.00030058353811717895,
+            0.0014792184333546447,
+            0.004427921639326168,
+            0.021438127785303025,
+            0.006339483221420334,
+            -0.001187747087595264,
+            -0.02237982599829809,
+            0.0062435650294483695,
+            0.002200578793697352,
+            0.010092828474934744,
+            0.023105023225115266,
+            -0.016797326021070008,
+            -0.015378520287395565,
+            -0.012732527281000097,
+            -0.010797463778182385,
+            0.009881499418260473,
+            0.02801767062785128,
+            0.01811139137849365,
+            -0.017358098300028903,
+            -0.020207173948076473,
+            0.0016242901745373126,
+            -0.026331540602868762,
+            0.007269751661479859,
+            0.11940631114242901,
+            -0.0045382135734049305,
+            -0.005994407567723939,
+            -0.008679712231646732,
+            -0.004482553208064535,
+            0.016709571294385293,
+            -0.013387043582971358,
+            0.069179848974036,
+            -0.007221545718220735,
+            0.0037392229364642915,
+            0.0028916224964057385,
+            0.00462729158956839,
+            -0.009727339449635306,
+            0.011570145487622916,
+            -0.011287365791681587,
+            -0.012991994493845187,
+            -0.013076194601468383,
+            -0.0180830261130804,
+            -0.007272768410218907,
+            -0.004155564814758654,
+            0.004204993170487392,
+            -0.005525575983182816,
+            0.008051108079520902,
+            -0.017551645696802648,
+            -0.010800070994485792,
+            0.013007176804761548,
+            -0.0023627485852619817,
+            0.005390986239687555,
+            -0.029668390278675554,
+            -0.0007650391259819867,
+            -0.0002446394046947245,
+            0.021280292985495608,
+            0.003137771886236958,
+            -0.0019077298815862946,
+            -0.010810159675564049,
+            0.007040107942856957,
+            0.023835993694726953,
+            0.021875366566457236,
+            0.0180511566130312,
+            0.012135351541044354,
+            0.020483975362807574,
+            -0.008667079096167644,
+            0.026324312888512866,
+            -0.001552207146726937,
+            -0.00397810465151493,
+            0.0077577136352257004,
+            -0.009849600842081875,
+            -0.00816392268697345,
+            0.00677353736462267,
+            -0.017288900282504148,
+            -0.020418071729964713,
+            -0.020383762873109595,
+            0.025741408097358233,
+            -0.011755094311186617,
+            0.002737129201159793,
+            -0.008266210469668866,
+            0.027287436144516414,
+            -0.006283892307996516,
+            -0.0029440642954701632,
+            0.009428876742171362,
+            -0.03788574758692254,
+            0.008834605607057885,
+            0.006463342392514337,
+            0.026711196829464153,
+            -3.9827332346130654e-05,
+            0.005565561841469247,
+            -0.0006738085601962865,
+            -0.006104320995853506,
+            -0.010792830169647851,
+            2.4565453208679464e-05,
+            0.017983908748172493,
+            0.0035181716362611077,
+            0.016970206528324653,
+            -0.005449808667944701,
+            -0.023171388778833197,
+            0.007092509502738752,
+            -0.01555320330471131,
+            -0.023306285898839978,
+            -0.002028783456704567,
+            -0.03196436161255752,
+            0.0033412618908711163,
+            0.007527633574188688,
+            -0.01890570959143866,
+            -0.017859735533232564,
+            0.010329340599904344,
+            0.004382396539605985,
+            -0.0016746397858953155,
+            0.006643785064338525,
+            -0.03307812909454471,
+            -0.01927475592253288,
+            0.004686970010943116,
+            0.012733525168279355,
+            -0.012993441126970916,
+            0.015291832952199082,
+            -0.020491167895268936,
+            0.016451976170508816,
+            -0.012259387189535237,
+            0.018393274368854132,
+            -0.006997668664226754,
+            0.011843450716200625,
+            0.011457415642860215,
+            0.009560877634897626,
+            -0.01790001115302499,
+            0.0214879415944361,
+            0.0066829070506954115,
+            0.03900547021123392,
+            0.02512484499576633,
+            -0.012651294056575184,
+            -0.0018425680226742782,
+            -0.00011361458320146007,
+            0.014982647266052349,
+            0.0061874424115559,
+            -0.003689464024837091,
+            0.02705902205734107,
+            0.018239469373522537,
+            -0.021070704644859155,
+            0.020037508334843183,
+            -0.00973804771680803,
+            0.018876283979591423,
+            -0.013303080480396818,
+            0.010592424494628194,
+            0.004717870260467274,
+            0.01070131238869639,
+            0.009087656015197565,
+            0.006708673669681232,
+            0.025928065153829043,
+            0.03663367449576964,
+            -0.00511073108390551,
+            0.002161381031392467,
+            0.004864092591944181,
+            0.012814518637357098,
+            0.001556833001747549,
+            0.007915766323215207,
+            -0.007012785514562176,
+            0.005834786334750152,
+            -0.002311533032619821,
+            -0.00122162928193159,
+            0.019525999280692766,
+            0.000824163362963678,
+            0.012875575199417254,
+            0.013259376059185821,
+            -0.002390133687709065,
+            -0.005682644446302278,
+            0.0019262218417782657,
+            -0.018193478045923098,
+            -0.015131696847666434,
+            0.029463723011600933,
+            -0.011447039651934148,
+            -0.008083114389964213,
+            0.0020642857121204753,
+            0.012160202510990566,
+            -0.00866944494225225,
+            -0.005378300635997713,
+            0.0038280364584418335,
+            0.014427318067525318,
+            -0.01799358399064332,
+            0.003381165206832161,
+            0.005252552821854107,
+            0.019433554824232185,
+            -0.00028791985674147885,
+            -0.013487678738391956,
+            0.0005317922815671783,
+            0.014982152801567637,
+            0.011902348165369078,
+            0.0016056390299304837,
+            -0.006935777197137747,
+            -0.00446584737370504,
+            0.004586814866182051,
+            0.006269466956110759,
+            0.0027516492952993445,
+            0.008157662045793032,
+            -0.0025001885993292637,
+            -0.016648747436203294,
+            0.017735757469839637,
+            -0.0030669873770852725,
+            0.018058248627079793,
+            -0.0038238312452176745,
+            -0.0147172470474118,
+            0.0043328417868095355,
+            -0.001888507803054499,
+            -0.018556943283494755,
+            0.009407930200720534,
+            0.002560949424273885,
+            0.010411159836451958,
+            0.0052571230238294825,
+            -0.007499470181994448,
+            -6.145889479758644e-05,
+            -0.015163819825329975,
+            0.022836682800304944,
+            0.016390944816242627,
+            0.001713572907204073,
+            -0.0035687505973373878,
+            -0.031210299824387475,
+            0.0029076511444359886,
+            -0.007822231980296341,
+            0.017821806420465356,
+            0.008250605126658402,
+            0.007908773724009905,
+            0.008351355849444162,
+            -0.00743970642051451,
+            -0.002411981861500092,
+            0.012543476434788293,
+            0.0202407777715137,
+            -0.0024635818448627447,
+            0.02535771413579989,
+            -0.012533449594824386,
+            -0.0022489292647974992,
+            0.0012362917358624392,
+            0.01578939075186126,
+            -0.0008001783766371555,
+            0.006311837812042848
+          ],
+          "keyphrases": [
+            "Switch Transformers",
+            "Mixture of Experts",
+            "sparsely-activated model",
+            "training instability",
+            "multilingual settings"
+          ],
+          "title": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "44ddb1e4-8508-4bcf-9d46-226ebb75db2f",
+        "properties": {
+          "page_content": "B Preventing Token Dropping with No-Token-Left-Behind 29\n\n",
+          "keyphrases": [
+            "Preventing Token Dropping",
+            "No-Token-Left-Behind"
+          ],
+          "embedding": [
+            0.025537114590406418,
+            -0.019454315304756165,
+            -0.016811933368444443,
+            0.022433646023273468,
+            -0.0004264498711563647,
+            -0.0028263733256608248,
+            -0.020252350717782974,
+            0.05167718976736069,
+            -0.02958049066364765,
+            0.00320987356826663,
+            -0.031212029978632927,
+            -0.00974489189684391,
+            0.01094194408506155,
+            -0.049123477190732956,
+            -0.010197111405432224,
+            0.038660354912281036,
+            -0.002062698360532522,
+            0.016102569177746773,
+            0.016013899818062782,
+            -0.003059133654460311,
+            0.010631597600877285,
+            0.013460187241435051,
+            -0.01815972663462162,
+            0.039192378520965576,
+            -0.005244862288236618,
+            0.027842549607157707,
+            -0.010347851552069187,
+            0.005479839164763689,
+            -0.04174608737230301,
+            0.018115390092134476,
+            0.008871487341821194,
+            0.009558684192597866,
+            -0.0016282127471640706,
+            -0.052067339420318604,
+            -0.011119285598397255,
+            0.012387273833155632,
+            0.04429979994893074,
+            0.015579413622617722,
+            -0.07313545793294907,
+            -0.028835657984018326,
+            -0.0168474018573761,
+            0.003110119141638279,
+            -0.05912551283836365,
+            0.0050763883627951145,
+            0.019170571118593216,
+            0.025129230692982674,
+            -0.004646336194127798,
+            -0.01648385263979435,
+            -0.0052049607038497925,
+            0.0036554429680109024,
+            0.017237553372979164,
+            -0.042207177728414536,
+            0.011943921446800232,
+            -0.02472134679555893,
+            0.015464141964912415,
+            0.015251332893967628,
+            0.0375608392059803,
+            0.018283864483237267,
+            0.007479359861463308,
+            -0.01447989884763956,
+            0.01440009567886591,
+            0.012804025784134865,
+            0.034670181572437286,
+            0.004735006485134363,
+            -0.0008046851144172251,
+            -0.008299562148749828,
+            -0.011722245253622532,
+            -0.023072073236107826,
+            0.017343956977128983,
+            -0.021405067294836044,
+            -0.050719548016786575,
+            0.02541297674179077,
+            -0.003491402370855212,
+            -0.006645856890827417,
+            0.023781437426805496,
+            0.010489724576473236,
+            0.07838475704193115,
+            0.05848708748817444,
+            0.0006278982618823647,
+            -0.000581900414545089,
+            -0.005510874092578888,
+            0.021937090903520584,
+            -0.001937451190315187,
+            0.014329158701002598,
+            0.041426874697208405,
+            -0.00011402476957300678,
+            -0.046995386481285095,
+            -0.04749194160103798,
+            0.04476088657975197,
+            -0.021334132179617882,
+            -0.0384475439786911,
+            -0.02028781920671463,
+            -0.02064250037074089,
+            -0.04121406748890877,
+            -0.00320987356826663,
+            0.0588417686522007,
+            0.017228685319423676,
+            -0.019383380189538002,
+            -0.022256305441260338,
+            0.005936492700129747,
+            -0.025022825226187706,
+            -0.0026623329613357782,
+            -0.05469198897480965,
+            0.01869175024330616,
+            0.03926331549882889,
+            0.012874961830675602,
+            -0.0023054340854287148,
+            0.009425678290426731,
+            -0.022185368463397026,
+            0.0798034816980362,
+            -0.018815888091921806,
+            0.009212869219481945,
+            -0.020411957055330276,
+            0.04468994960188866,
+            -0.031247496604919434,
+            -0.016785332933068275,
+            -0.00320987356826663,
+            -0.034067220985889435,
+            -0.036638665944337845,
+            0.011669042520225048,
+            0.0056571802124381065,
+            -0.015269066207110882,
+            -0.012617817148566246,
+            0.013229643926024437,
+            -0.027186386287212372,
+            0.008844885975122452,
+            0.027150917798280716,
+            0.019738061353564262,
+            -0.021405067294836044,
+            -0.01984446682035923,
+            -0.014072014018893242,
+            -0.013149840757250786,
+            -0.058628957718610764,
+            0.012724222615361214,
+            -0.04224264249205589,
+            0.019879935309290886,
+            0.019436581060290337,
+            0.011988257057964802,
+            -0.026211010292172432,
+            0.04163968563079834,
+            -0.022912466898560524,
+            0.007377388887107372,
+            0.028392305597662926,
+            0.04745647311210632,
+            -0.012431609444320202,
+            0.009913366287946701,
+            -0.008441435173153877,
+            0.015747888013720512,
+            -0.018620813265442848,
+            0.021511472761631012,
+            0.02498735673725605,
+            0.004351506475359201,
+            0.014630638994276524,
+            0.03674507141113281,
+            0.005346833262592554,
+            0.01222766749560833,
+            -0.007204481400549412,
+            -0.02152920700609684,
+            0.007794140372425318,
+            -0.007452758494764566,
+            0.016155771911144257,
+            0.007075909059494734,
+            0.03532634302973747,
+            -0.00984242931008339,
+            0.024029714986681938,
+            0.006654724013060331,
+            0.03472338244318962,
+            0.013265112414956093,
+            -0.08313749730587006,
+            0.02347995899617672,
+            -0.0027244023513048887,
+            0.031034687533974648,
+            -0.010223712772130966,
+            0.03021891973912716,
+            -0.007612365763634443,
+            -0.020341021940112114,
+            -0.0064995503053069115,
+            -0.00956755131483078,
+            0.016599124297499657,
+            0.011411897838115692,
+            -0.011713378131389618,
+            0.03709975257515907,
+            -0.009053261950612068,
+            -0.024650409817695618,
+            0.022522317245602608,
+            -0.003440416883677244,
+            0.019206037744879723,
+            -0.0175301656126976,
+            -0.008538972586393356,
+            0.009434545412659645,
+            -0.02516469918191433,
+            0.018461205065250397,
+            0.01779617741703987,
+            -0.00274656992405653,
+            -0.01745922863483429,
+            0.0034071654081344604,
+            -0.014905516989529133,
+            -0.025022825226187706,
+            0.01737942546606064,
+            -0.00849907100200653,
+            -0.0026157808024436235,
+            0.0168474018573761,
+            -0.029917439445853233,
+            -0.00881828460842371,
+            0.0021292013116180897,
+            0.021121321246027946,
+            0.0020128211472183466,
+            0.04458354786038399,
+            0.007740938104689121,
+            0.01790258102118969,
+            -0.0017235336126759648,
+            0.0003020340227521956,
+            -0.007173446472734213,
+            0.012502545490860939,
+            -0.019418848678469658,
+            -0.01103061530739069,
+            -0.03181498870253563,
+            0.018053321167826653,
+            -0.022185368463397026,
+            0.039192378520965576,
+            0.03630171716213226,
+            -0.014976453967392445,
+            -0.022717392072081566,
+            -0.0026645497418940067,
+            -0.003846084466204047,
+            -0.017388291656970978,
+            0.018408004194498062,
+            0.025217900052666664,
+            -0.0026157808024436235,
+            -0.00030425077420659363,
+            0.010826672427356243,
+            0.02401198074221611,
+            -0.05298951268196106,
+            0.0066192555241286755,
+            -0.013309448026120663,
+            -0.009700557217001915,
+            -0.034084953367710114,
+            0.010933076962828636,
+            0.019791264086961746,
+            0.01067593228071928,
+            0.013814869336783886,
+            0.044654481112957,
+            -0.07682415097951889,
+            -0.0029549456667155027,
+            -0.03552141785621643,
+            0.06685758382081985,
+            -0.026636630296707153,
+            0.03678053990006447,
+            -0.03527313843369484,
+            -0.0006434156093746424,
+            -0.018141992390155792,
+            0.017672037705779076,
+            -0.010968545451760292,
+            0.007763105444610119,
+            0.02481001615524292,
+            0.06412653625011444,
+            -0.006934036035090685,
+            -0.025093762204051018,
+            0.001193727133795619,
+            -0.02489868737757206,
+            0.020695703104138374,
+            0.03848301246762276,
+            0.025483911857008934,
+            -0.01815972663462162,
+            0.008995626121759415,
+            -0.009922233410179615,
+            0.016155771911144257,
+            -0.014178418554365635,
+            0.025661254301667213,
+            -0.021688813343644142,
+            0.027062248438596725,
+            0.016368580982089043,
+            -0.03748990222811699,
+            -0.03177952021360397,
+            0.0035667724441736937,
+            0.003726379247382283,
+            -0.00783404242247343,
+            0.0015772271435707808,
+            0.005289197433739901,
+            0.034067220985889435,
+            -0.014870049431920052,
+            0.00894242338836193,
+            0.003912587650120258,
+            -0.033535197377204895,
+            -0.007878377102315426,
+            -0.035237669944763184,
+            -0.0650487095117569,
+            0.03780911862850189,
+            -0.08483996987342834,
+            -0.012910430319607258,
+            -0.017831645905971527,
+            0.01191732008010149,
+            -0.011057215742766857,
+            0.009039961732923985,
+            0.06430387496948242,
+            0.01702474243938923,
+            0.010471990332007408,
+            0.0023896710481494665,
+            -0.04312935099005699,
+            -0.0010507458355277777,
+            -0.007581330835819244,
+            -0.01966712437570095,
+            0.001189293572679162,
+            -0.029988376423716545,
+            -0.03589383512735367,
+            -0.017831645905971527,
+            0.0019474267028272152,
+            -0.027682941406965256,
+            -0.05171265825629234,
+            -0.002861841581761837,
+            0.020802108570933342,
+            -0.02383464016020298,
+            -0.02755880355834961,
+            -0.0328080989420414,
+            -0.040185488760471344,
+            -0.006588221061974764,
+            0.005807920359075069,
+            0.04075298085808754,
+            -0.023072073236107826,
+            -0.02995290793478489,
+            0.02037649042904377,
+            -0.017964651808142662,
+            -0.04057563841342926,
+            0.011243424378335476,
+            0.006925168912857771,
+            -0.0393342487514019,
+            0.027186386287212372,
+            0.04330668970942497,
+            -0.025749923661351204,
+            -0.00035828439285978675,
+            -0.014533101581037045,
+            -0.01913510262966156,
+            0.000581900414545089,
+            -0.029740098863840103,
+            -0.02709771692752838,
+            -0.03126523271203041,
+            0.0014054280472919345,
+            -0.018674015998840332,
+            -0.0025248934980481863,
+            0.007563597057014704,
+            -0.020500628277659416,
+            -0.03731256350874901,
+            0.009815828874707222,
+            0.0009986519580706954,
+            0.0007620124379172921,
+            -0.016253309324383736,
+            0.01440009567886591,
+            -0.021068120375275612,
+            -0.005524174310266972,
+            -0.002821939764544368,
+            0.0011904019629582763,
+            0.01320304349064827,
+            0.05518854409456253,
+            -0.026636630296707153,
+            0.0665738433599472,
+            -0.009141932241618633,
+            -0.01913510262966156,
+            0.008920256048440933,
+            0.006428613793104887,
+            -0.03731256350874901,
+            0.006348810624331236,
+            -0.02605140395462513,
+            0.009532082825899124,
+            0.004101012367755175,
+            -0.0023497692309319973,
+            0.00025395795819349587,
+            -0.012662152759730816,
+            0.01067593228071928,
+            -0.0247036125510931,
+            -0.014905516989529133,
+            -0.018727216869592667,
+            -0.011819782666862011,
+            0.034404169768095016,
+            -0.012724222615361214,
+            0.02417158894240856,
+            -0.027789346873760223,
+            0.03741896525025368,
+            0.015623748302459717,
+            -0.018141992390155792,
+            0.01977352984249592,
+            -0.022557783871889114,
+            -0.021688813343644142,
+            0.024419866502285004,
+            -0.01594296284019947,
+            0.02170654758810997,
+            0.022256305441260338,
+            0.01904643140733242,
+            0.0366741344332695,
+            -0.012600083835422993,
+            0.032116468995809555,
+            -0.03559235483407974,
+            0.02234497480094433,
+            -0.07061721384525299,
+            0.006096099503338337,
+            0.017317356541752815,
+            0.036886945366859436,
+            -0.02835683710873127,
+            0.00717788003385067,
+            0.021227726712822914,
+            -0.013398118317127228,
+            -0.007408423349261284,
+            0.010746869258582592,
+            0.00956755131483078,
+            -0.034510571509599686,
+            -0.012289736419916153,
+            0.028764722868800163,
+            -0.050896890461444855,
+            -0.013637528754770756,
+            -0.0027753878384828568,
+            0.007483793422579765,
+            0.0027776046190410852,
+            -0.047704748809337616,
+            -0.004488945938646793,
+            -0.02878245711326599,
+            -0.007918278686702251,
+            0.010152776725590229,
+            0.02427799254655838,
+            0.021671079099178314,
+            -0.01966712437570095,
+            -0.03284356743097305,
+            0.006645856890827417,
+            0.029545024037361145,
+            0.009993169456720352,
+            0.026441553607583046,
+            0.07192953675985336,
+            -0.036195311695337296,
+            -0.01502078864723444,
+            0.009833562187850475,
+            -0.00788724422454834,
+            -0.016359714791178703,
+            0.03475885093212128,
+            0.03279036656022072,
+            -0.03692241013050079,
+            -0.011057215742766857,
+            -0.020075010135769844,
+            -0.029527289792895317,
+            -0.05774225294589996,
+            0.0027709542773663998,
+            -0.0008872595499269664,
+            -0.02081984281539917,
+            0.00427170330658555,
+            0.014187285676598549,
+            -0.055685099214315414,
+            0.005280330311506987,
+            -0.016528187319636345,
+            -0.021458270028233528,
+            -0.028587380424141884,
+            -0.03569876030087471,
+            -0.013176442123949528,
+            0.008973458781838417,
+            -0.037702713161706924,
+            0.007523695006966591,
+            -0.022575518116354942,
+            0.030502665787935257,
+            0.008574441075325012,
+            -0.008303996175527573,
+            0.025217900052666664,
+            -0.021156789734959602,
+            0.013362649828195572,
+            -0.006867533084005117,
+            -0.04146234318614006,
+            -0.003145587397739291,
+            0.0023608531337231398,
+            -0.011021748185157776,
+            0.0055463421158492565,
+            0.023515427485108376,
+            -0.02791348472237587,
+            0.029970642179250717,
+            0.0017833862220868468,
+            -0.0061404346488416195,
+            0.0061315675266087055,
+            -0.01710454747080803,
+            -0.011748846620321274,
+            -0.021227726712822914,
+            0.0023165177553892136,
+            -0.0034382001031190157,
+            -0.04011455178260803,
+            -0.04394511878490448,
+            -0.048520516604185104,
+            -0.007541429251432419,
+            -0.014001077972352505,
+            -0.00903552770614624,
+            -0.0083217304199934,
+            -0.005462104920297861,
+            -0.004034509416669607,
+            -0.0031123359221965075,
+            -0.007027139887213707,
+            -0.010294649749994278,
+            -0.009372475557029247,
+            -0.004280570428818464,
+            0.02294793538749218,
+            -0.019808998331427574,
+            -0.019188303500413895,
+            -0.016599124297499657,
+            0.00631334213539958,
+            -0.024508535861968994,
+            -0.03545048087835312,
+            0.003611107589676976,
+            0.038412075489759445,
+            0.03373027220368385,
+            0.011465100571513176,
+            -0.018026720732450485,
+            0.0026290814857929945,
+            0.019596189260482788,
+            0.022699657827615738,
+            -0.05909004807472229,
+            -0.01332718227058649,
+            -0.0008351656142622232,
+            -0.02410065196454525,
+            -0.0027798213995993137,
+            -0.015384337864816189,
+            0.0010551793966442347,
+            -0.006526151672005653,
+            -0.0005539137637242675,
+            -0.007155712228268385,
+            -0.020802108570933342,
+            -6.234647298697382e-05,
+            0.0027111016679555178,
+            -0.0008235276327468455,
+            -0.002099274890497327,
+            0.016616858541965485,
+            0.019649391993880272,
+            -0.0164040494710207,
+            0.00757246371358633,
+            -0.011216823011636734,
+            0.018567610532045364,
+            0.03137163445353508,
+            -0.0016437300946563482,
+            0.020713437348604202,
+            -0.025306571274995804,
+            0.003990174271166325,
+            0.022735126316547394,
+            -0.034670181572437286,
+            -0.027168652042746544,
+            -0.0014575219247490168,
+            0.014453297480940819,
+            -0.0006240188959054649,
+            0.01984446682035923,
+            -0.06370091438293457,
+            -0.00586998974904418,
+            -0.025058293715119362,
+            0.00200284575112164,
+            -0.016439517959952354,
+            0.035503681749105453,
+            -0.003141153836622834,
+            0.012245400808751583,
+            0.03259528800845146,
+            0.006415313575416803,
+            0.016714395955204964,
+            -0.01779617741703987,
+            0.014408962801098824,
+            0.0354682132601738,
+            -0.047101788222789764,
+            -0.011881851591169834,
+            -0.006645856890827417,
+            -0.044512610882520676,
+            0.013371516950428486,
+            0.011296626180410385,
+            0.019702592864632607,
+            0.027594272047281265,
+            -0.024384398013353348,
+            0.03165538236498833,
+            -0.007324186619371176,
+            -0.009363608434796333,
+            0.026246478781104088,
+            -0.004805942997336388,
+            0.019472049549221992,
+            0.006362111307680607,
+            -0.0072133480571210384,
+            0.010569527745246887,
+            0.012245400808751583,
+            -0.011216823011636734,
+            0.006876400206238031,
+            -0.029296746477484703,
+            -8.721578342374414e-05,
+            0.012316337786614895,
+            -0.010516325943171978,
+            0.002083757659420371,
+            -0.03390761464834213,
+            0.025448445230722427,
+            -0.013140973635017872,
+            -0.01267101988196373,
+            0.03461697697639465,
+            0.004225151147693396,
+            0.016599124297499657,
+            0.03447510674595833,
+            0.0026889340952038765,
+            -0.009000059217214584,
+            0.006597088184207678,
+            -0.001766760484315455,
+            0.007341920398175716,
+            0.018993228673934937,
+            0.00934587512165308,
+            -0.023550894111394882,
+            -0.02374597080051899,
+            -0.013699597679078579,
+            0.008836018852889538,
+            0.01001090370118618,
+            0.0020848659332841635,
+            0.003216523677110672,
+            0.008707446977496147,
+            0.009159666486084461,
+            0.015792222693562508,
+            -0.013353782705962658,
+            -0.0031256363727152348,
+            0.01763656921684742,
+            -0.03773818165063858,
+            0.0036465758457779884,
+            -0.010285782627761364,
+            0.0010391079122200608,
+            0.00934587512165308,
+            -0.00047992929467000067,
+            0.0473145991563797,
+            0.005643879529088736,
+            0.008405966684222221,
+            0.010436521843075752,
+            0.019365645945072174,
+            -0.006774429231882095,
+            -0.017964651808142662,
+            0.005967527162283659,
+            0.026477022096514702,
+            -0.029456352815032005,
+            0.03766724467277527,
+            -0.00445569446310401,
+            -0.001095081097446382,
+            0.002797555411234498,
+            -0.006539452355355024,
+            -0.018833622336387634,
+            -0.00405446020886302,
+            -0.03509579971432686,
+            -0.009833562187850475,
+            0.01098627969622612,
+            0.027665207162499428,
+            -0.023941045626997948,
+            -0.02622874453663826,
+            0.03138937056064606,
+            0.023107541725039482,
+            -0.0005497573292814195,
+            0.030963752418756485,
+            -0.028232699260115623,
+            0.020624767988920212,
+            0.026920374482870102,
+            0.018815888091921806,
+            0.03748990222811699,
+            -0.030981486663222313,
+            -0.011713378131389618,
+            0.022167634218931198,
+            0.038944099098443985,
+            0.038837697356939316,
+            0.0012380623957142234,
+            -0.03748990222811699,
+            0.02347995899617672,
+            0.03422682732343674,
+            0.03426229581236839,
+            0.00675226142629981,
+            -0.02339128777384758,
+            -0.009638487361371517,
+            0.013469054363667965,
+            0.0005458780215121806,
+            0.010090706869959831,
+            -0.010684799402952194,
+            0.008295129053294659,
+            -0.013593193143606186,
+            -0.016625726595520973,
+            -0.0005375651526264846,
+            -0.018815888091921806,
+            -0.033801209181547165,
+            -0.030857346951961517,
+            0.016944939270615578,
+            -0.014417828992009163,
+            -0.03470565006136894,
+            -0.019879935309290886,
+            -0.013185309246182442,
+            -0.005847821943461895,
+            -0.04834317788481712,
+            -0.02800215594470501,
+            -0.020429691299796104,
+            0.027310525998473167,
+            -0.01879815384745598,
+            -0.007865076884627342,
+            -0.02727505750954151,
+            -0.003212090115994215,
+            0.0007664459408260882,
+            0.0384475439786911,
+            -0.013894673436880112,
+            -0.0001251778594451025,
+            -0.02488095313310623,
+            0.013238511048257351,
+            -0.02984650246798992,
+            0.0065039838664233685,
+            -0.015827691182494164,
+            -0.02638835273683071,
+            0.011305493302643299,
+            -0.011491701938211918,
+            -0.002347552450373769,
+            -0.011172487400472164,
+            0.01377053465694189,
+            -0.005151758436113596,
+            -0.010658198967576027,
+            0.007723203860223293,
+            0.0014763644430786371,
+            -0.008552273735404015,
+            -0.02259325236082077,
+            0.04962003231048584,
+            0.0036598765291273594,
+            -0.0062645734287798405,
+            -0.0013056736206635833,
+            -0.012387273833155632,
+            -0.006291174795478582,
+            -0.008303996175527573,
+            -0.019099634140729904,
+            0.002118117408826947,
+            0.017778443172574043,
+            0.026175541803240776,
+            0.026884907856583595,
+            -0.001146066701039672,
+            0.015419806353747845,
+            -0.00015406505553983152,
+            -0.04600227624177933,
+            -0.022717392072081566,
+            -0.03169085085391998,
+            -0.03169085085391998,
+            0.03197459504008293,
+            -0.019649391993880272,
+            -0.0030480497516691685,
+            -0.011872985400259495,
+            0.02463267557322979,
+            0.02913713827729225,
+            0.0003538508608471602,
+            -0.0030347490683197975,
+            -0.0001627935707801953,
+            0.0027842549607157707,
+            -0.01045425608754158,
+            0.0033827810548245907,
+            -0.016058234497904778,
+            0.024845484644174576,
+            -5.153975143912248e-05,
+            0.01727301999926567,
+            0.032471150159835815,
+            -0.02913713827729225,
+            0.033801209181547165,
+            0.010188245214521885,
+            0.008729614317417145,
+            0.015925228595733643,
+            -0.008135521784424782,
+            -0.0011011771857738495,
+            0.035822898149490356,
+            0.02073117159307003,
+            -0.031673114746809006,
+            0.01457743626087904,
+            -0.02372823655605316,
+            -0.026884907856583595,
+            0.020890777930617332,
+            -0.009806961752474308,
+            -0.028924329206347466,
+            0.029704630374908447,
+            0.020181413739919662,
+            0.026104606688022614,
+            0.032985441386699677,
+            -0.015818823128938675,
+            -0.006654724013060331,
+            -0.002751003485172987,
+            0.0004120409139432013,
+            -0.006344377063214779,
+            0.012600083835422993,
+            0.004087711684405804,
+            -0.0014054280472919345,
+            -0.001630429527722299,
+            0.018478939309716225,
+            0.0006295607890933752,
+            0.011598106473684311,
+            0.006277874112129211,
+            -0.00751039432361722,
+            -0.029030734673142433,
+            -0.008051284588873386,
+            0.011633574962615967,
+            0.03731256350874901,
+            -0.034865256398916245,
+            0.00859217531979084,
+            -0.022699657827615738,
+            -0.019188303500413895,
+            0.01737942546606064,
+            -0.03271942958235741,
+            0.004770474974066019,
+            0.016776464879512787,
+            0.0006539452006109059,
+            0.013903540559113026,
+            0.009425678290426731,
+            0.011598106473684311,
+            0.012963633053004742,
+            0.0057724518701434135,
+            0.002511593047529459,
+            0.015357737429440022,
+            0.012910430319607258,
+            0.07249703258275986,
+            -0.025040559470653534,
+            0.00045665327343158424,
+            0.03802192583680153,
+            -0.027168652042746544,
+            -0.003965789917856455,
+            0.03396081551909447,
+            -0.006468515843153,
+            -0.02401198074221611,
+            0.029562756419181824,
+            -0.01860307902097702,
+            -0.02816176228225231,
+            -0.016794199123978615,
+            -0.030272122472524643,
+            -0.013983343727886677,
+            0.012165597639977932,
+            0.01762770302593708,
+            -0.011145886965095997,
+            0.025005090981721878,
+            0.0008401533705182374,
+            -0.019915401935577393,
+            -0.008787250146269798,
+            0.02259325236082077,
+            -0.012316337786614895,
+            -0.006694625597447157,
+            0.0025780959986150265,
+            0.012520279735326767,
+            0.00962075311690569,
+            0.015340003184974194,
+            -0.041604217141866684,
+            -0.010303516872227192,
+            0.011119285598397255,
+            -0.0031034687999635935,
+            0.0030037143733352423,
+            0.0276474729180336,
+            0.01717548258602619,
+            0.011828649789094925,
+            -0.03163764625787735,
+            0.027168652042746544,
+            0.01853214204311371,
+            -0.023409022018313408,
+            0.03827020525932312,
+            -0.03936971724033356,
+            -0.025005090981721878,
+            -0.006676891352981329,
+            0.006109400186687708,
+            0.0049566831439733505,
+            -0.00631334213539958,
+            -0.010064106434583664,
+            -0.024916421622037888,
+            -0.03279036656022072,
+            -0.019010962918400764,
+            -0.0005746959359385073,
+            0.015295667573809624,
+            0.011802048422396183,
+            0.005235995166003704,
+            0.008286261931061745,
+            0.04057563841342926,
+            -0.0033606134820729494,
+            -0.002994847483932972,
+            -0.031939126551151276,
+            -0.012289736419916153,
+            0.03165538236498833,
+            0.027044514194130898,
+            -0.005945359356701374,
+            0.007204481400549412,
+            -0.005351266823709011,
+            0.00907986331731081,
+            0.0013245161389932036,
+            -0.013486788608133793,
+            0.026264213025569916,
+            0.04632148891687393,
+            -0.0022533400915563107,
+            -0.03279036656022072,
+            -0.006167036015540361,
+            -0.026884907856583595,
+            -0.01566808484494686,
+            0.012733088806271553,
+            -0.026689831167459488,
+            0.0027088848873972893,
+            -0.011988257057964802,
+            -0.04366137459874153,
+            -0.008756215684115887,
+            -0.009443412534892559,
+            0.01753903180360794,
+            -0.00571481604129076,
+            -0.0062468391843140125,
+            -0.0061227004043757915,
+            -0.0053335330449044704,
+            0.00350913661532104,
+            -0.0031832722015678883,
+            -0.03341105952858925,
+            0.020677968859672546,
+            -0.01869175024330616,
+            -0.023089807480573654,
+            -0.02002180740237236,
+            -0.003923671320080757,
+            0.002082649152725935,
+            -0.005572943482547998,
+            -0.010179378092288971,
+            0.014870049431920052,
+            -0.019383380189538002,
+            -0.024526270106434822,
+            -0.03837660700082779,
+            0.012014857493340969,
+            0.010046372190117836,
+            -0.011553770862519741,
+            -0.01186411827802658,
+            -0.0016160205705091357,
+            -0.004899047315120697,
+            0.01568581722676754,
+            -0.02346222475171089,
+            0.0049522495828568935,
+            0.029367681592702866,
+            0.018008986487984657,
+            -0.026477022096514702,
+            0.012050325982272625,
+            -0.01897549442946911,
+            0.02913713827729225,
+            -0.027079982683062553,
+            -0.011819782666862011,
+            0.02443760074675083,
+            0.0022965669631958008,
+            -0.0027044513262808323,
+            -0.025803126394748688,
+            0.015393204987049103,
+            -0.002391887828707695,
+            0.019915401935577393,
+            -0.021511472761631012,
+            0.02773614414036274,
+            0.004276136867702007,
+            -0.030360791832208633,
+            0.015047390013933182,
+            0.026104606688022614,
+            -0.011367563158273697,
+            0.02365729957818985,
+            0.01957845501601696,
+            -0.004899047315120697,
+            -0.010729135014116764,
+            0.0175301656126976,
+            -0.008188724517822266,
+            0.023799171671271324,
+            -0.0407884456217289,
+            0.0064951167441904545,
+            -0.044867292046546936,
+            -0.036106642335653305,
+            0.003602240700274706,
+            -0.006237972527742386,
+            -0.020979449152946472,
+            -0.040220957249403,
+            0.003229824360460043,
+            0.009922233410179615,
+            0.01842573843896389,
+            0.027842549607157707,
+            -0.0218838881701231,
+            0.02878245711326599,
+            0.0034226826392114162,
+            -0.027487866580486298,
+            0.0070936428382992744,
+            -0.0022810494992882013,
+            -0.032205138355493546,
+            -0.009239469654858112,
+            -0.022025762125849724,
+            -0.005586243700236082,
+            -0.049832843244075775,
+            0.013052303344011307,
+            -0.011074949987232685,
+            0.015934094786643982,
+            -0.017521297559142113,
+            -0.02833910472691059,
+            -0.0022489065304398537,
+            0.037773650139570236,
+            0.003945839125663042,
+            -0.036816008388996124,
+            -0.02924354374408722,
+            0.019010962918400764,
+            -0.02984650246798992,
+            -0.007071475498378277,
+            0.005936492700129747,
+            0.020252350717782974,
+            -0.010773470625281334,
+            -0.020411957055330276,
+            -0.010037505067884922,
+            -0.006065064575523138,
+            -0.00823305919766426,
+            0.013070037588477135,
+            0.04163968563079834,
+            -0.004659636877477169,
+            0.010356718674302101,
+            0.0027155352290719748,
+            0.04603774473071098,
+            -0.001809987355954945,
+            -0.018656281754374504,
+            0.028587380424141884,
+            -0.0014364627422764897,
+            -0.0059143248945474625,
+            0.016164638102054596,
+            -0.013238511048257351,
+            0.002917260630056262,
+            -0.0300238449126482,
+            0.028906594961881638,
+            0.011970522813498974,
+            -0.026867173612117767,
+            -0.00939907692372799,
+            -0.02782481536269188,
+            -0.02791348472237587,
+            -0.05022299289703369,
+            -0.011092684231698513,
+            -0.01700701005756855,
+            0.04518650472164154,
+            -0.05593337491154671,
+            -0.01728188805282116,
+            0.006198070477694273,
+            -0.006920735351741314,
+            -0.024224791675806046,
+            0.006645856890827417,
+            -0.01320304349064827,
+            -0.022185368463397026,
+            0.011145886965095997,
+            -0.033269185572862625,
+            -0.003879335941746831,
+            -0.015907494351267815,
+            -0.007315319497138262,
+            0.0017301838379353285,
+            -0.0014353543519973755,
+            0.06015409156680107,
+            0.009957700967788696,
+            0.02862284891307354,
+            -0.009443412534892559,
+            0.009780360385775566,
+            0.026104606688022614,
+            0.020766640082001686,
+            0.023444490507245064,
+            0.009904499165713787,
+            -0.0047305733896791935,
+            -0.019702592864632607,
+            0.0024184889625757933,
+            0.010241447016596794,
+            0.010072972625494003,
+            0.005621712189167738,
+            -0.028410039842128754,
+            -0.02126319520175457,
+            -0.006916301790624857,
+            -0.008667545393109322,
+            -0.004380324389785528,
+            0.031158827245235443,
+            -0.016794199123978615,
+            -0.029367681592702866,
+            -0.023284882307052612,
+            -0.005036486312747002,
+            -0.04635695740580559,
+            0.0003114552528131753,
+            0.015827691182494164,
+            0.002108142012730241,
+            5.7462664699414745e-05,
+            -0.0026623329613357782,
+            0.004570966120809317,
+            -0.011119285598397255,
+            0.014072014018893242,
+            0.0034515007864683867,
+            0.0058034867979586124,
+            -0.00996656809002161,
+            -0.0005685998476110399,
+            0.0024916420225054026,
+            0.013921274803578854,
+            0.006597088184207678,
+            -0.02259325236082077,
+            -0.04802396148443222,
+            -0.027416929602622986,
+            -0.0029615960083901882,
+            -0.013273979537189007,
+            -0.000881163461599499,
+            -0.010853273794054985,
+            -0.0049655502662062645,
+            0.0034382001031190157,
+            -0.0008839344372972846,
+            -0.011855251155793667,
+            -0.008073452860116959,
+            -0.004245101939886808,
+            -0.008937990292906761,
+            -0.023072073236107826,
+            0.023355819284915924,
+            -0.009443412534892559,
+            -0.01621784083545208,
+            -0.020890777930617332,
+            -0.007501527667045593,
+            -0.023426756262779236,
+            -0.010587261989712715,
+            0.009673955850303173,
+            0.006557186134159565,
+            -0.016412915661931038,
+            -0.006716793403029442,
+            -0.004449044354259968,
+            0.008286261931061745,
+            -3.664586984086782e-05,
+            -0.006162602454423904,
+            -0.029083935543894768,
+            0.007523695006966591,
+            0.008711880072951317,
+            0.0083217304199934,
+            -0.0015317834913730621,
+            -0.002019471488893032,
+            0.01470157504081726,
+            0.08065471798181534,
+            -0.012697621248662472,
+            0.009363608434796333,
+            0.0008722963975742459,
+            -0.032400213181972504,
+            0.01244934368878603,
+            -0.03848301246762276,
+            -0.001967377495020628,
+            0.02230950817465782,
+            0.008574441075325012,
+            0.024260258302092552,
+            -0.011606973595917225,
+            0.010090706869959831,
+            0.04827224090695381,
+            -0.006348810624331236,
+            0.0029793300200253725,
+            0.0026800669729709625,
+            2.926820525317453e-05,
+            0.0056616137735545635,
+            0.007031573448330164,
+            -0.0015417590038850904,
+            -0.009682822972536087,
+            -0.019347911700606346,
+            -0.016191240400075912,
+            0.01780504360795021,
+            -0.013850337825715542,
+            -0.0012258702190592885,
+            -0.03177952021360397,
+            -0.004198549780994654,
+            -0.010002036578953266,
+            0.015632616356015205,
+            -0.03057360090315342,
+            0.00029787758830934763,
+            0.012298603542149067,
+            -0.035663291811943054,
+            0.004311604890972376,
+            0.001890899264253676,
+            0.0007936013280414045,
+            0.01835480146110058,
+            -0.003152237506583333,
+            -0.007723203860223293,
+            0.022132165729999542,
+            0.0022522315848618746,
+            -0.013628661632537842,
+            -0.022841529920697212,
+            0.002338685328140855,
+            -0.015987297520041466,
+            -0.03231154382228851,
+            -0.019720327109098434,
+            -0.00039347552228718996,
+            -0.009097597561776638,
+            0.023054338991642,
+            -0.007049307692795992,
+            -0.028729254379868507,
+            -0.01922377198934555,
+            0.007816308178007603,
+            0.004668503999710083,
+            -0.017140014097094536,
+            -0.005834521260112524,
+            0.021812953054904938,
+            -0.03738350048661232,
+            -0.021103588864207268,
+            -0.009443412534892559,
+            0.002622431144118309,
+            0.004641902633011341,
+            -0.008969024755060673,
+            -0.017255285754799843,
+            0.0024362229742109776,
+            0.004056677222251892,
+            -0.015304534696042538,
+            -0.0016359713627025485,
+            0.028285901993513107,
+            0.019418848678469658,
+            0.004522197414189577,
+            -0.00288179237395525,
+            0.004067760892212391,
+            -0.0051916600205004215,
+            0.006211371161043644,
+            0.0025936132296919823,
+            -0.026618896052241325,
+            -0.008290695026516914,
+            0.04848505184054375,
+            0.003999041393399239,
+            0.0057680183090269566,
+            0.004797075875103474,
+            -0.007084776181727648,
+            -0.014541967771947384,
+            0.0005589014617726207,
+            0.020057275891304016,
+            0.013415852561593056,
+            0.022752860561013222,
+            0.015428673475980759,
+            -0.007918278686702251,
+            -0.004167515318840742,
+            0.0002563132729846984,
+            -0.02285926416516304,
+            0.036816008388996124,
+            0.0016891737468540668,
+            -0.0025337606202811003,
+            0.005626145750284195,
+            -0.0063887122087180614,
+            0.00031976812169887125,
+            0.025501646101474762,
+            -0.004087711684405804,
+            -0.014666106551885605,
+            0.02321394719183445,
+            0.013477921485900879,
+            -0.007257683668285608,
+            -0.034510571509599686,
+            0.008840452879667282,
+            0.007044874131679535,
+            0.01585429161787033,
+            -0.01737942546606064,
+            -0.003799532540142536,
+            -0.009115330874919891,
+            -0.013318315148353577,
+            0.03660319745540619,
+            -0.00723994942381978,
+            -0.006597088184207678,
+            -0.0010075189638882875,
+            -0.030449463054537773,
+            0.00680103013291955,
+            0.02383464016020298,
+            0.00819759164005518,
+            -0.027345994487404823,
+            -0.0068542324006557465,
+            0.004083278123289347,
+            -0.020270084962248802,
+            -0.010853273794054985,
+            0.005107422824949026,
+            -0.023781437426805496,
+            0.011278891935944557,
+            -0.0028175064362585545,
+            -0.0030702173244208097,
+            0.0018432388314977288,
+            0.0020050625316798687,
+            0.04784662276506424,
+            -0.02940315008163452,
+            0.01904643140733242,
+            -0.003897070186212659,
+            0.009239469654858112,
+            0.00018468410416971892,
+            0.011190221644937992,
+            -0.007270983885973692,
+            -0.007869509980082512,
+            -0.00019646066357381642,
+            -0.03745443373918533,
+            -0.0007570247398689389,
+            0.010401054285466671,
+            0.038305673748254776,
+            -0.007754238322377205,
+            -0.007115810643881559,
+            0.023621831089258194,
+            0.011314360424876213,
+            0.009106464684009552,
+            -0.008410400710999966,
+            -0.005989694967865944,
+            -0.007563597057014704,
+            -0.0011288868263363838,
+            -0.012564615346491337,
+            0.029296746477484703,
+            -0.00907986331731081,
+            0.003713078796863556,
+            -0.030077045783400536,
+            0.02933221310377121,
+            0.0056571802124381065,
+            0.02401198074221611,
+            -0.004273919854313135,
+            0.0019607271533459425,
+            0.005821220576763153,
+            0.025998201221227646,
+            -0.009274938143789768,
+            -0.023089807480573654,
+            0.0012979150051251054,
+            0.01763656921684742,
+            0.04387418180704117,
+            0.030609069392085075,
+            -0.017476962879300117,
+            0.023444490507245064,
+            -0.03596476837992668,
+            0.0003726933500729501,
+            0.006198070477694273,
+            -0.006570486817508936,
+            -0.0053202323615550995,
+            -0.014027679339051247,
+            0.0004497258923947811,
+            0.004637469071894884,
+            -0.01798238418996334,
+            -0.008175423368811607,
+            -0.010223712772130966,
+            0.01518926303833723,
+            0.01799125224351883,
+            0.004903480876237154,
+            -0.0010851057013496757,
+            7.121352246031165e-05,
+            -0.021724281832575798,
+            0.010081839747726917,
+            -0.026547959074378014,
+            0.03438643366098404,
+            -0.009088730439543724,
+            -0.00010841358744073659,
+            0.007355221081525087,
+            -0.01279515866190195,
+            0.01762770302593708,
+            0.01541093923151493,
+            -0.0019751363433897495,
+            0.0059187584556639194,
+            -0.039653465151786804,
+            0.020890777930617332,
+            -0.009709424339234829,
+            -0.04571852833032608,
+            -0.0011194655671715736,
+            0.011332094669342041,
+            -0.01621784083545208,
+            0.020358756184577942,
+            0.03365933522582054,
+            -0.01603163406252861,
+            0.0136020602658391,
+            -0.015517343766987324,
+            -0.009195134975016117,
+            0.027878016233444214,
+            0.006100533064454794,
+            -0.015818823128938675,
+            0.007975914515554905,
+            0.0057680183090269566,
+            0.02046515978872776,
+            -0.008525672368705273,
+            -2.9960943720652722e-05,
+            0.004695104900747538,
+            -0.010170510970056057,
+            0.040220957249403,
+            5.850177331012674e-05,
+            0.019436581060290337,
+            -0.01895776018500328,
+            -0.010906476527452469,
+            -0.006472949404269457,
+            0.003294110530987382,
+            -0.0067300936207175255,
+            -0.010321250185370445,
+            -0.018727216869592667,
+            -0.009239469654858112,
+            -0.00939907692372799,
+            0.031495776027441025,
+            -0.0014785812236368656,
+            -0.011704511009156704,
+            -0.0002506328164599836,
+            -0.013619794510304928,
+            -0.004240668378770351,
+            -0.029314478859305382,
+            -0.0054177697747945786,
+            -0.007324186619371176,
+            -0.01966712437570095,
+            -0.006295608356595039,
+            -0.01328284665942192,
+            0.005861122626811266,
+            0.014790245331823826,
+            0.008565573953092098,
+            0.008379366248846054,
+            -0.0040079085156321526,
+            -0.0021713196765631437,
+            -0.006827631499618292,
+            -0.037241626530885696,
+            0.017388291656970978,
+            0.0016326463082805276,
+            -0.01586315967142582,
+            0.00886705331504345,
+            -0.03656772896647453,
+            0.03571649268269539,
+            0.02330261655151844,
+            -0.02888886071741581,
+            0.00575915165245533,
+            0.025909531861543655,
+            0.024863218888640404,
+            -0.0036554429680109024,
+            0.009984302334487438,
+            -0.015543945133686066,
+            -0.007412856910377741,
+            -0.0053290994837880135,
+            0.02629968151450157,
+            -0.009727157652378082,
+            0.002292133402079344,
+            0.04206530377268791,
+            -0.0011837516212835908,
+            0.004788209218531847,
+            -0.011376430280506611,
+            0.005360133945941925,
+            -0.008441435173153877,
+            0.0027266191318631172,
+            -0.036354921758174896,
+            -0.006929602473974228,
+            -0.03413815796375275,
+            0.0062601398676633835,
+            0.01191732008010149,
+            -0.005954226478934288,
+            0.02842777408659458,
+            0.002972679678350687,
+            0.002906176960095763,
+            0.022433646023273468,
+            -0.025944998487830162,
+            -0.026796236634254456,
+            0.012378406710922718,
+            0.006951770279556513,
+            0.031123358756303787,
+            0.013637528754770756,
+            0.007222215179353952,
+            0.003555688541382551,
+            -0.0056660473346710205,
+            -0.04518650472164154,
+            0.00819759164005518,
+            -0.013406985439360142,
+            0.024402132257819176,
+            0.0046020010486245155,
+            0.0080601517111063,
+            0.03605344146490097,
+            0.01332718227058649,
+            0.004615301266312599,
+            0.016359714791178703,
+            0.0007514827884733677,
+            0.00819759164005518,
+            0.03004157729446888,
+            0.028658317402005196,
+            -0.007155712228268385,
+            -0.04649883136153221,
+            -0.032027799636125565,
+            0.010383320041000843,
+            -0.0023586363531649113,
+            -0.0006478491122834384,
+            0.010019770823419094,
+            -0.023373553529381752,
+            -0.01033011730760336,
+            0.023515427485108376,
+            -4.7383316996274516e-05,
+            -0.00653058523312211,
+            -0.03156671300530434,
+            -0.012050325982272625,
+            0.01771637424826622,
+            -0.012928164564073086,
+            0.011021748185157776,
+            0.014772512018680573,
+            -0.010871008038520813,
+            -0.015419806353747845,
+            -0.014932118356227875,
+            -0.002702234545722604,
+            0.019986338913440704,
+            0.009860163554549217,
+            -0.020004073157906532,
+            0.022486848756670952,
+            0.016634592786431313,
+            0.0062734405510127544,
+            0.012768557295203209,
+            -0.020571565255522728,
+            0.02638835273683071,
+            -0.013983343727886677,
+            0.014692707918584347,
+            -0.012050325982272625,
+            -0.00686309952288866,
+            0.011890718713402748,
+            0.004101012367755175,
+            -0.0014353543519973755,
+            -0.003888203063979745,
+            0.001440896186977625,
+            -0.012334072031080723,
+            -0.0013622011756524444,
+            -0.020589299499988556,
+            0.009283805266022682,
+            -0.011225690133869648,
+            -0.00047217062092386186,
+            0.023196212947368622,
+            0.01413408387452364,
+            0.003201006446033716,
+            0.0028441075701266527,
+            -0.014045413583517075,
+            -0.01984446682035923,
+            0.0050675212405622005,
+            -0.006308908574283123,
+            -0.0028441075701266527,
+            -0.02073117159307003,
+            -0.0254307109862566,
+            0.017583368346095085,
+            0.03321598470211029,
+            0.016058234497904778,
+            -0.027629738673567772,
+            -0.013876939192414284,
+            -0.004205200355499983,
+            0.014009945094585419,
+            0.011881851591169834,
+            -0.006632556207478046,
+            0.006397579330950975,
+            -0.008193157613277435,
+            -0.0036000239197164774,
+            -0.013433586806058884,
+            0.015996165573596954,
+            0.01711341366171837,
+            0.004468995146453381,
+            0.018301598727703094,
+            0.015632616356015205,
+            0.015446407720446587,
+            -0.030431728810071945,
+            -0.0014043196570128202,
+            0.018283864483237267,
+            -0.010915343649685383,
+            -0.03585836663842201,
+            -0.014692707918584347,
+            0.00228326627984643,
+            0.018461205065250397,
+            -0.0015905278269201517,
+            -0.02135186456143856,
+            -0.014719309285283089,
+            0.0002265255170641467,
+            -0.03373027220368385,
+            0.017335090786218643,
+            0.00018468410416971892,
+            -0.00043060630559921265,
+            0.015526210889220238,
+            0.0074483249336481094,
+            -0.0007398448069579899,
+            0.005107422824949026,
+            -0.019188303500413895,
+            -0.006521718110889196,
+            -0.007545862812548876,
+            0.02959822490811348,
+            -0.00028429992380551994,
+            0.0023209513165056705,
+            -0.005705948919057846,
+            -0.009390209801495075,
+            0.007528128568083048,
+            -0.010383320041000843,
+            0.00022555567556992173,
+            0.012422742322087288,
+            0.025271102786064148,
+            -0.008405966684222221,
+            0.011314360424876213,
+            0.039830803871154785,
+            0.00038350006798282266,
+            -0.03436870127916336,
+            0.00023594676167704165,
+            -0.01435576006770134,
+            -0.005129590630531311,
+            0.009008926339447498,
+            0.007270983885973692,
+            -0.0007564705447293818,
+            -0.001497423741966486,
+            -0.006885267328470945,
+            -0.021919356659054756,
+            -0.012360673397779465,
+            -0.030147982761263847,
+            -0.00988676492124796,
+            -0.0047261398285627365,
+            0.007222215179353952,
+            -0.020482894033193588,
+            0.006477382965385914,
+            -0.01611143723130226,
+            -0.009718290530145168,
+            -0.015242465771734715,
+            -0.002522676717489958,
+            -0.013539991341531277,
+            -0.014391228556632996,
+            0.0070936428382992744,
+            0.006167036015540361,
+            0.00934587512165308,
+            -0.0033716971520334482,
+            -0.009057695046067238,
+            -0.0019197171786800027,
+            -0.012422742322087288,
+            0.014719309285283089,
+            0.00864537712186575,
+            0.01637744903564453,
+            -0.003936972003430128,
+            -0.009115330874919891,
+            0.00031588878482580185,
+            0.01683853566646576,
+            0.012786291539669037,
+            0.010268048383295536,
+            -0.012165597639977932,
+            0.014231621287763119,
+            -0.003688694443553686,
+            0.01700701005756855,
+            -0.004191899672150612,
+            0.010241447016596794,
+            -0.004792642779648304,
+            0.004938948899507523,
+            0.019613923504948616,
+            0.018993228673934937,
+            0.004912347998470068,
+            0.004903480876237154,
+            0.06632556021213531,
+            0.013406985439360142,
+            0.033517464995384216,
+            0.011287759058177471,
+            0.0065039838664233685,
+            -0.01510945986956358,
+            -0.008437002077698708,
+            -0.012422742322087288,
+            0.026743033900856972,
+            0.017246419563889503,
+            0.007270983885973692,
+            -0.018656281754374504,
+            -0.0030236653983592987,
+            0.028658317402005196,
+            0.014657239429652691,
+            0.008219758979976177,
+            0.006286741234362125,
+            0.019454315304756165,
+            0.006082798819988966,
+            0.008366065099835396,
+            0.003460367675870657,
+            0.0060029951855540276,
+            0.006078365258872509,
+            0.015313401818275452,
+            0.012617817148566246,
+            0.01208579447120428,
+            0.005018752533942461,
+            0.00560397794470191,
+            0.02949182130396366,
+            0.014426696114242077,
+            0.029615959152579308,
+            -0.0007819632883183658,
+            0.005710382480174303,
+            0.031939126551151276,
+            0.012147863395512104,
+            0.01421388704329729,
+            0.02614007517695427,
+            -0.0017612185329198837,
+            -0.022486848756670952,
+            0.0037507638335227966,
+            0.02214989997446537,
+            -0.014630638994276524,
+            0.023284882307052612,
+            -0.0033606134820729494,
+            -0.0008224192424677312,
+            -0.026725299656391144,
+            -0.019117368385195732,
+            0.0027776046190410852,
+            -0.027328260242938995,
+            0.009939967654645443,
+            -0.0029926307033747435,
+            -0.007523695006966591,
+            0.005847821943461895,
+            -0.007900545373558998,
+            -0.010516325943171978,
+            0.004562098998576403,
+            0.016874004155397415,
+            -0.005630579311400652,
+            -0.0067389607429504395,
+            0.013300580903887749,
+            -0.000335839664330706,
+            -0.0005372880259528756,
+            0.011819782666862011,
+            -0.00394362211227417,
+            -0.015632616356015205,
+            0.00935474131256342,
+            -0.0013743933523073792,
+            -0.012236534617841244,
+            -0.015606014989316463,
+            0.0013888023095205426,
+            -0.005976394284516573,
+            -0.01569468528032303,
+            0.0029704628977924585,
+            0.007590197958052158,
+            -0.002387454267591238,
+            -0.0031322867143899202,
+            0.007049307692795992,
+            0.012883828952908516,
+            0.011447366327047348,
+            -0.002442873315885663,
+            0.005488706286996603,
+            0.003828350454568863,
+            -0.014178418554365635,
+            -0.010862140916287899,
+            -0.038412075489759445,
+            -0.005816787481307983,
+            0.01922377198934555,
+            0.01860307902097702,
+            0.0006855340907350183,
+            0.003285243408754468,
+            0.009195134975016117,
+            0.006699059158563614,
+            -0.0007453867001459002,
+            0.004801509436219931,
+            -0.025483911857008934,
+            0.002972679678350687,
+            0.012165597639977932,
+            -0.008352764882147312,
+            -0.0007985890260897577,
+            -0.01826613023877144,
+            -0.021316397935152054,
+            0.0041120960377156734,
+            0.003169971751049161,
+            -0.025111496448516846,
+            -0.011793181300163269,
+            0.0029638127889484167,
+            0.008716314099729061,
+            -0.002309867413714528,
+            -0.027612006291747093,
+            0.009070996195077896,
+            -0.025324305519461632,
+            0.013167575001716614,
+            -3.078356940022786e-06,
+            -0.021032651886343956,
+            0.03624851629137993,
+            0.002489425241947174,
+            0.01603163406252861,
+            0.004083278123289347,
+            -0.007705469615757465,
+            -0.009461146779358387,
+            -0.02649475634098053,
+            0.01350452285259962,
+            -0.0066901920363307,
+            0.0014752560527995229,
+            0.019596189260482788,
+            -0.004171948879957199,
+            -0.004588700365275145,
+            -0.013539991341531277,
+            0.003693128004670143,
+            -0.010064106434583664,
+            0.000359392783138901,
+            0.013096638023853302,
+            -0.0006201395881362259,
+            0.00035828439285978675,
+            -0.0025514946319162846,
+            -0.001351117272861302,
+            -0.005475405603647232,
+            0.0001952137245098129,
+            -0.0015495176194235682,
+            -0.009824695996940136,
+            -0.002011712873354554,
+            0.005120723508298397,
+            -0.005032053217291832,
+            0.019170571118593216,
+            0.015703551471233368,
+            -0.016856269910931587,
+            -0.004788209218531847,
+            -0.013797136023640633,
+            0.008969024755060673,
+            -0.022806061431765556,
+            0.008756215684115887,
+            0.0060162958689033985,
+            0.0159606970846653,
+            0.008552273735404015,
+            -0.0020737822633236647,
+            0.004034509416669607,
+            0.017166616395115852,
+            0.017521297559142113,
+            -0.0012602299684658647,
+            -0.006663591135293245,
+            -0.00728871813043952,
+            -0.007160145789384842,
+            -0.011083817109465599,
+            -0.002338685328140855,
+            -0.009673955850303173,
+            0.023338085040450096,
+            0.015242465771734715,
+            0.022664189338684082,
+            0.008490203879773617,
+            -0.0014785812236368656,
+            -0.007248816546052694,
+            0.016182372346520424,
+            0.0029748964589089155,
+            0.005240428727120161,
+            0.006317775696516037,
+            -0.0034581508953124285,
+            0.004801509436219931,
+            -0.004562098998576403,
+            0.008295129053294659,
+            0.008734048344194889,
+            -0.009638487361371517,
+            0.01835480146110058,
+            0.009266071021556854,
+            -0.01808878965675831,
+            0.008632076904177666,
+            -0.003972440026700497,
+            -0.0028928762767463923,
+            0.006823197938501835,
+            0.0044822958298027515,
+            -0.0036066740285605192,
+            0.008561140857636929,
+            0.007430591154843569,
+            -0.007226648740470409,
+            0.007599065080285072,
+            0.007519261445850134,
+            -0.003460367675870657,
+            0.014648373238742352,
+            0.012130129151046276,
+            -0.0003546821535564959,
+            0.014559702016413212,
+            0.0028729254845529795,
+            -0.008663111366331577,
+            -0.011500569060444832,
+            0.016155771911144257,
+            -0.0035401713103055954,
+            0.007417290471494198,
+            0.0036975613329559565,
+            0.03470565006136894,
+            0.006339943502098322,
+            -0.015934094786643982,
+            -0.03392534703016281,
+            0.003786231856793165,
+            -0.0037064284551888704,
+            0.014533101581037045,
+            -0.013238511048257351,
+            0.006699059158563614,
+            0.01462177187204361,
+            -0.00494338246062398,
+            0.026512490585446358,
+            0.004426876548677683,
+            0.007656700909137726,
+            0.006277874112129211,
+            -0.010658198967576027,
+            0.0006594871520064771,
+            -0.0017290754476562142,
+            0.020145945250988007,
+            0.0016060450579971075,
+            -0.013637528754770756,
+            -0.014949852600693703,
+            -0.014178418554365635,
+            0.006344377063214779,
+            0.013797136023640633,
+            -0.010968545451760292,
+            0.020890777930617332,
+            -0.011544903740286827,
+            0.012342939153313637,
+            0.03342879191040993,
+            0.02454400435090065,
+            0.011748846620321274,
+            0.0008745131781324744,
+            0.0004896276514045894,
+            -0.018230661749839783,
+            -0.009718290530145168,
+            -0.00956755131483078,
+            -0.0016891737468540668,
+            0.0017656520940363407,
+            0.011606973595917225,
+            0.005728116724640131,
+            0.012307470664381981,
+            -3.5606761230155826e-05,
+            -0.019418848678469658,
+            0.016350846737623215,
+            -0.02817949652671814,
+            -0.010170510970056057,
+            -0.007426157593727112,
+            0.029704630374908447,
+            0.024207057431340218,
+            -0.01577448844909668,
+            0.011092684231698513,
+            0.010853273794054985,
+            -0.0056483130902051926,
+            0.013832603581249714,
+            -0.010064106434583664,
+            -0.0009210652206093073,
+            0.009372475557029247,
+            0.011305493302643299,
+            0.006118267308920622,
+            0.020571565255522728,
+            0.024136120453476906,
+            0.027434663847088814,
+            -0.006167036015540361,
+            -0.005994128528982401,
+            0.012112395837903023,
+            0.00996656809002161,
+            0.012679887004196644,
+            0.020323287695646286,
+            0.0006567161763086915,
+            -0.016545921564102173,
+            0.03509579971432686,
+            0.009665088728070259,
+            0.003959139343351126,
+            0.0003106239892076701,
+            0.00403007585555315,
+            0.023621831089258194,
+            -0.02161787636578083,
+            0.004287220537662506,
+            0.0048502786085009575,
+            -0.013841470703482628,
+            0.03759630769491196,
+            0.008481336757540703,
+            0.012520279735326767,
+            0.00751039432361722,
+            0.0023564195726066828,
+            0.00026518033700995147,
+            -0.006193636916577816,
+            -0.00881385151296854,
+            0.016785332933068275,
+            -0.009425678290426731,
+            -0.018443472683429718,
+            -0.00962075311690569,
+            -0.01425822265446186,
+            0.0019285841844975948,
+            -0.007975914515554905,
+            -0.020748905837535858,
+            0.0003084072086494416,
+            -0.012068060226738453,
+            0.019897669553756714,
+            -0.004143130965530872,
+            -0.0021247677505016327,
+            -0.010853273794054985,
+            -0.00016237792442552745,
+            0.005630579311400652,
+            0.013460187241435051,
+            -0.018133124336600304,
+            -0.0067433943040668964,
+            0.01904643140733242,
+            -0.014231621287763119,
+            -0.0127153554931283,
+            -0.026335150003433228,
+            -0.020075010135769844,
+            0.001347792218439281,
+            -0.008441435173153877,
+            0.005147324874997139,
+            0.008583308197557926,
+            0.0050808219239115715,
+            -0.0013644179562106729,
+            0.005018752533942461,
+            -0.021458270028233528,
+            0.00033473127405159175,
+            -0.008995626121759415,
+            0.018904559314250946,
+            0.004899047315120697,
+            0.0037928821984678507,
+            -0.014249355532228947,
+            -0.0029881971422582865,
+            -0.010126175358891487,
+            -0.016412915661931038,
+            0.002062698360532522,
+            -0.03197459504008293,
+            0.013220776803791523,
+            -0.005351266823709011,
+            -0.014001077972352505,
+            -0.002966029569506645,
+            -0.012564615346491337,
+            -0.009044394828379154,
+            0.010338984429836273,
+            -0.002567012095823884,
+            -0.025022825226187706,
+            -0.0003854397509712726,
+            0.0034891855902969837,
+            0.018904559314250946,
+            -0.0035734225530177355,
+            0.0030236653983592987,
+            -0.0004976633936166763,
+            -0.007408423349261284,
+            -0.00913306511938572,
+            0.020571565255522728,
+            0.007333053275942802,
+            -0.01431142445653677,
+            0.006162602454423904,
+            0.0053202323615550995,
+            -0.03137163445353508,
+            -0.0205538310110569,
+            -0.0018365884898230433,
+            0.01528680045157671,
+            -0.0006112725241109729,
+            -0.0010607213480398059,
+            0.003118986263871193,
+            -0.015348870307207108,
+            -0.011553770862519741,
+            -0.0138237364590168,
+            0.006942903157323599,
+            -0.013743933290243149,
+            0.007860642857849598,
+            -0.010002036578953266,
+            0.0067256600596010685,
+            0.0014475465286523104,
+            0.006517284549772739,
+            0.021050386130809784,
+            -0.0007021598285064101,
+            0.003285243408754468,
+            -0.005608411505818367,
+            0.00420963391661644,
+            -0.015073991380631924,
+            0.009700557217001915,
+            0.00876951590180397,
+            -0.0018210711423307657,
+            0.017609968781471252,
+            0.002079324098303914,
+            -0.011323227547109127,
+            0.000915523327421397,
+            0.0010956353507936,
+            0.004646336194127798,
+            -0.018017852678894997,
+            -0.019418848678469658,
+            0.015934094786643982,
+            -0.004134263843297958,
+            0.010569527745246887,
+            -0.00086398352868855,
+            0.008751781657338142,
+            -0.004360373597592115,
+            0.00642418023198843,
+            0.025732189416885376,
+            -0.011598106473684311,
+            0.0012956982245668769,
+            -0.0067389607429504395,
+            0.006636989768594503,
+            -0.007412856910377741,
+            0.02958049066364765,
+            0.011270025745034218,
+            0.008804984390735626,
+            0.004393625073134899,
+            0.024313461035490036,
+            0.0026667665224522352,
+            -0.01279515866190195,
+            0.0017346173990517855,
+            0.033623866736888885,
+            0.0026601161807775497,
+            -0.003872685832902789,
+            0.00988676492124796,
+            0.012245400808751583,
+            0.0009875681716948748,
+            0.006747827865183353,
+            0.00675226142629981,
+            0.019525252282619476,
+            -0.0026046971324831247,
+            0.019330177456140518,
+            -0.004154214635491371,
+            -0.006215804722160101,
+            -0.026991311460733414,
+            0.0056660473346710205,
+            -0.02126319520175457,
+            0.01604936644434929,
+            -0.0027753878384828568,
+            0.016359714791178703,
+            0.012502545490860939,
+            -0.001784494612365961,
+            -0.022912466898560524,
+            -0.004404708743095398,
+            0.016811933368444443,
+            -0.0041608652099967,
+            0.011802048422396183,
+            -0.007536995690315962,
+            0.0015794439241290092,
+            0.007568030618131161,
+            -0.017308488488197327,
+            -0.0037418967112898827,
+            0.0006234647007659078,
+            -0.010507458820939064,
+            0.006588221061974764,
+            0.005963093601167202,
+            -0.008875920437276363,
+            0.017317356541752815,
+            -0.013921274803578854,
+            -0.003786231856793165,
+            -0.002456173999235034,
+            0.019276974722743034,
+            0.0006001886795274913,
+            0.008556706830859184,
+            -0.011571505106985569,
+            -0.015366604551672935,
+            6.151518755359575e-05,
+            0.012688754126429558,
+            -0.005630579311400652,
+            -0.0009875681716948748,
+            0.013593193143606186,
+            -0.008210891857743263,
+            0.032985441386699677,
+            -0.017929183319211006,
+            -0.028285901993513107,
+            -0.005874423310160637,
+            -0.0032187404576689005,
+            -0.010569527745246887,
+            0.003150020958855748,
+            0.013469054363667965,
+            0.0161203034222126,
+            -0.005519740749150515,
+            0.01222766749560833,
+            0.01701587624847889,
+            -0.0019906535744667053,
+            -0.023621831089258194,
+            -0.0023542027920484543,
+            -0.001258013304322958,
+            -0.013371516950428486,
+            -0.0045975674875080585,
+            -0.016705529764294624,
+            0.002338685328140855,
+            0.006707926280796528,
+            -0.003854951588436961,
+            -0.004067760892212391,
+            -0.005235995166003704,
+            0.007000538986176252,
+            0.0021624527871608734,
+            -0.019117368385195732,
+            0.00560397794470191,
+            -0.017335090786218643,
+            0.016874004155397415,
+            -0.005156191997230053,
+            -0.009000059217214584,
+            0.021582409739494324,
+            -0.008494637906551361,
+            -0.0056660473346710205,
+            0.001044095610268414,
+            0.0255903173238039,
+            -0.011225690133869648,
+            0.0056882151402533054,
+            -0.008069018833339214,
+            0.017335090786218643,
+            5.936769593972713e-05,
+            0.005284763872623444,
+            -0.0027244023513048887,
+            0.024242525920271873,
+            -0.006601521745324135,
+            -0.02596273273229599,
+            -0.0030657837633043528,
+            0.02895979769527912,
+            -0.02021688222885132,
+            -0.022132165729999542,
+            0.005280330311506987,
+            -0.000599080347456038,
+            0.0055374749936163425,
+            -0.003584506455808878,
+            0.0056837815791368484,
+            0.009815828874707222,
+            0.000171106425113976,
+            -0.006965070962905884,
+            0.024863218888640404,
+            -0.02771840989589691,
+            0.010143909603357315,
+            0.0047305733896791935,
+            -0.002263315487653017,
+            -0.010933076962828636,
+            0.02135186456143856,
+            0.014444430358707905,
+            0.006450781598687172,
+            -0.0017390509601682425,
+            0.01692720502614975,
+            -0.024082917720079422,
+            0.0021569107193499804,
+            0.01452423445880413,
+            0.01718435063958168,
+            -0.0033051942009478807,
+            -0.008241926319897175,
+            0.002852974459528923,
+            -0.0056571802124381065,
+            -0.01510945986956358,
+            0.011802048422396183,
+            -0.0018022286240011454,
+            0.015419806353747845,
+            -0.030857346951961517,
+            0.013185309246182442,
+            0.0006018512649461627,
+            0.014240488409996033,
+            0.002691150875762105,
+            0.007661134470254183,
+            -0.01333604846149683,
+            0.012121262960135937,
+            -0.017955783754587173,
+            0.00302144861780107,
+            0.017166616395115852,
+            0.020961714908480644,
+            -0.0036177579313516617,
+            -0.0031212028115987778,
+            0.01165130827575922,
+            0.014160685241222382,
+            0.016067100688815117,
+            0.003670960199087858,
+            0.0036864776629954576,
+            -0.017689771950244904,
+            -0.0300238449126482,
+            -0.00582565413787961,
+            -0.0014142950531095266,
+            -0.012369539588689804,
+            -0.0009338115924037993,
+            0.019436581060290337,
+            -0.001129995216615498,
+            0.018762685358524323,
+            -0.03536181151866913,
+            0.008069018833339214,
+            -0.007545862812548876,
+            -0.012484811246395111,
+            -0.0032519919332116842,
+            -0.014914384111762047,
+            -0.015127194114029408,
+            0.0067256600596010685,
+            -0.011500569060444832,
+            0.009124197997152805,
+            -0.01666119322180748,
+            -0.025324305519461632,
+            0.008889221586287022,
+            -0.01125229150056839,
+            -0.013007967732846737,
+            0.007856209762394428,
+            -0.01772524043917656,
+            0.007355221081525087,
+            0.007971481420099735,
+            0.0016016116132959723,
+            -0.00631334213539958,
+            0.002755437046289444,
+            0.0076167993247509,
+            0.003537954529747367,
+            0.015809956938028336,
+            -0.0006428614142350852,
+            -0.00048824213445186615,
+            -0.022717392072081566,
+            -0.007803007494658232,
+            -0.018106523901224136,
+            -0.002391887828707695,
+            -0.007036007009446621,
+            -0.03055586665868759,
+            -0.0031877057626843452,
+            0.010746869258582592,
+            0.003923671320080757,
+            0.014063146896660328,
+            0.004553232342004776,
+            -0.009540949948132038,
+            0.011926187202334404,
+            0.005502006970345974,
+            -0.0005051449406892061,
+            -0.0033561799209564924,
+            -0.005927625577896833,
+            0.008264093659818172,
+            0.00026725855423137546,
+            0.008960157632827759,
+            0.0006661374354735017,
+            0.025874063372612,
+            -0.006894134450703859,
+            -0.00030729881837032735,
+            0.010489724576473236,
+            -0.009736024774610996,
+            -0.0011155862594023347,
+            0.0007852884591557086,
+            0.0057768854312598705,
+            0.012662152759730816,
+            -0.021937090903520584,
+            0.012555748224258423,
+            -0.023621831089258194,
+            -0.011216823011636734,
+            0.007705469615757465,
+            0.0020471811294555664,
+            -0.010995146818459034,
+            0.0026002635713666677,
+            -0.01342471968382597,
+            -0.017343956977128983,
+            -0.00894242338836193,
+            0.017148882150650024,
+            -0.018585344776511192,
+            -0.03048493154346943,
+            -0.004703972022980452,
+            -0.007621232885867357,
+            -0.02303660660982132,
+            0.008343897759914398,
+            -0.024136120453476906,
+            -0.0030635669827461243,
+            0.000919956830330193,
+            0.015561679378151894,
+            -0.0011737762251868844,
+            0.014604037627577782,
+            -0.03064453788101673,
+            -0.0005134578095749021,
+            0.022433646023273468,
+            -0.021334132179617882,
+            -0.015162661671638489,
+            0.0061537353321909904,
+            -0.011048348620533943,
+            -0.019117368385195732,
+            0.011997124180197716,
+            -0.0007004972430877388,
+            0.002314300974830985,
+            -0.019986338913440704,
+            0.034067220985889435,
+            -0.037880051881074905,
+            0.005253729410469532,
+            0.009904499165713787,
+            0.023143010213971138,
+            0.008796117268502712,
+            0.016430649906396866,
+            0.02995290793478489,
+            0.0036554429680109024,
+            0.014391228556632996,
+            0.007763105444610119,
+            -0.005138457752764225,
+            -0.004885746631771326,
+            -0.009221736341714859,
+            -6.1956807257956825e-06,
+            -0.002870708703994751,
+            0.0058877235278487206,
+            0.005395602434873581,
+            0.022575518116354942,
+            -0.022912466898560524,
+            0.027416929602622986,
+            0.02488095313310623,
+            0.006769995670765638,
+            -0.00984242931008339,
+            0.0009886764455586672,
+            -0.013451320119202137,
+            0.023710502311587334,
+            -0.00353352096863091,
+            -0.021334132179617882,
+            -0.0012868312187492847,
+            0.019791264086961746,
+            0.015091725625097752,
+            0.0028884427156299353,
+            -0.009869030676782131,
+            -0.009381342679262161,
+            -0.0002915044024121016,
+            -0.03057360090315342,
+            0.007674435153603554,
+            0.012112395837903023,
+            -0.012076927348971367,
+            -0.01699814200401306,
+            -0.011713378131389618,
+            -0.013460187241435051,
+            0.001321190968155861,
+            -0.007333053275942802,
+            0.015224731527268887,
+            -0.013584326021373272,
+            0.0038660354912281036,
+            0.018283864483237267,
+            -0.0007010514382272959,
+            -0.022380443289875984,
+            -0.021901622414588928,
+            -0.004355940036475658,
+            -0.00917740073055029,
+            0.011145886965095997,
+            -0.008649811148643494,
+            -0.017166616395115852,
+            -0.019365645945072174,
+            0.0011859684018418193,
+            -0.02259325236082077,
+            -0.0016160205705091357,
+            -0.015322268940508366,
+            0.0006932927644811571,
+            -0.013619794510304928,
+            0.0014054280472919345,
+            -0.00885818712413311,
+            0.014586303383111954,
+            -0.0033517463598400354,
+            -0.0021114670671522617,
+            0.022628720849752426,
+            -0.011979389935731888,
+            0.017432628199458122,
+            0.003635491942986846,
+            0.0010668174363672733,
+            -0.006903001572936773,
+            -0.017867112532258034,
+            0.009558684192597866,
+            0.0024761247914284468,
+            -0.007767539005726576,
+            0.018230661749839783,
+            -0.012254267930984497,
+            0.0041231801733374596,
+            -0.0023497692309319973,
+            -0.01103061530739069,
+            0.0055507756769657135,
+            0.007355221081525087,
+            0.0013821519678458571,
+            -0.0013655262300744653,
+            0.003917021211236715,
+            -0.010117308236658573,
+            0.012218800373375416,
+            0.01492325123399496,
+            0.00881828460842371,
+            0.004480078816413879,
+            -0.026991311460733414,
+            0.006393145769834518,
+            0.023001138120889664,
+            -0.004515547305345535,
+            -0.0018498890567570925,
+            -0.0002811133163049817,
+            0.014683840796351433,
+            0.022841529920697212,
+            -0.007723203860223293,
+            0.009939967654645443,
+            -0.031141093000769615,
+            0.03349972888827324,
+            0.01629764400422573,
+            -0.013965609483420849,
+            -0.023675033822655678,
+            0.02376370318233967,
+            -0.01125229150056839,
+            0.0029748964589089155,
+            -0.011606973595917225,
+            -0.015517343766987324,
+            -0.024614941328763962,
+            0.01737942546606064,
+            0.015162661671638489,
+            0.007426157593727112,
+            -0.014471031725406647,
+            0.03064453788101673,
+            0.0006894134567119181,
+            0.004650769755244255,
+            -0.007324186619371176,
+            0.005351266823709011,
+            0.021546941250562668,
+            -0.003529087407514453,
+            -0.0031810556538403034,
+            -0.005258162971585989,
+            -0.01541093923151493,
+            0.0070936428382992744,
+            -0.014586303383111954,
+            0.0031345034949481487,
+            -0.010738002136349678,
+            -0.00030480496934615076,
+            0.009062129072844982,
+            0.027434663847088814,
+            0.008951290510594845,
+            0.011926187202334404,
+            0.023710502311587334,
+            -0.014497633092105389,
+            -0.010268048383295536,
+            -0.009558684192597866,
+            0.014125216752290726,
+            0.016368580982089043,
+            0.01984446682035923,
+            -0.0009166316594928503,
+            0.01622670888900757,
+            -0.013442453928291798,
+            -0.009718290530145168,
+            -0.009931100532412529,
+            -0.01744149439036846,
+            0.0007132436148822308,
+            0.00978922750800848,
+            -0.0026445987168699503,
+            -0.005169492214918137,
+            -0.011722245253622532,
+            0.012830627150833607,
+            -0.0015118326991796494,
+            -0.007372955325990915,
+            -0.010897609405219555,
+            0.01510945986956358,
+            -0.005293630994856358,
+            -0.004686237778514624,
+            0.016075968742370605,
+            0.023533159866929054,
+            -0.01993313618004322,
+            0.0026645497418940067,
+            -0.013406985439360142,
+            0.004267269745469093,
+            -0.007971481420099735,
+            0.031212029978632927,
+            -0.017920315265655518,
+            0.0205538310110569,
+            0.0014575219247490168,
+            -0.001433137571439147,
+            -0.009762626141309738,
+            0.0561816543340683,
+            -0.0012702054809778929,
+            -0.028853392228484154,
+            0.024207057431340218,
+            0.007927145808935165,
+            -0.012724222615361214,
+            -0.002011712873354554,
+            0.01262668427079916,
+            0.015650350600481033,
+            0.004085495136678219,
+            -0.011935054324567318,
+            -0.007275417447090149,
+            -0.007004972547292709,
+            -0.017601102590560913,
+            0.030520398169755936,
+            -0.01895776018500328,
+            0.004868012387305498,
+            0.011358696036040783,
+            0.021777484565973282,
+            -0.004686237778514624,
+            -0.01510945986956358,
+            0.00225888192653656,
+            0.007319753058254719,
+            -0.024685878306627274,
+            -0.004198549780994654,
+            0.00016542596858926117,
+            -0.0025315438397228718,
+            -0.004218501038849354,
+            -0.020429691299796104,
+            0.02383464016020298,
+            0.00984242931008339,
+            -0.00750596122816205,
+            -0.017929183319211006,
+            -0.01700701005756855,
+            -0.010844406671822071,
+            -0.017831645905971527,
+            -0.002586962888017297,
+            0.017973517999053,
+            0.016803067177534103,
+            0.001548409229144454,
+            0.013930141925811768,
+            -0.0055463421158492565,
+            0.0049566831439733505,
+            -0.010427655652165413,
+            0.012307470664381981,
+            -0.01753903180360794,
+            0.0038704690523445606,
+            0.01975579559803009,
+            -0.007297585252672434,
+            0.01948978379368782,
+            0.020677968859672546,
+            0.0067433943040668964,
+            -0.008623209781944752,
+            0.00010411861148895696,
+            0.0024207057431340218,
+            0.024047449231147766,
+            -0.0016115870093926787,
+            -0.002083757659420371,
+            0.0024362229742109776,
+            0.007807441055774689,
+            0.028906594961881638,
+            0.009860163554549217,
+            0.008184290491044521,
+            0.025182433426380157,
+            -0.0031677549704909325,
+            -0.009159666486084461,
+            -0.0055419085547327995,
+            -0.014958719722926617,
+            0.0020216882694512606,
+            0.005958660040050745,
+            0.003815049771219492,
+            -0.0029881971422582865,
+            -0.0038616019301116467,
+            -0.001410969882272184,
+            -0.01886909082531929,
+            0.010693666525185108,
+            0.0016670060576871037,
+            0.0020649151410907507,
+            0.02835683710873127,
+            -0.012272002175450325,
+            0.008312863297760487,
+            0.01788484677672386,
+            -0.003032532287761569,
+            0.00930153951048851,
+            0.013043436221778393,
+            -0.008095620200037956,
+            0.008964591659605503,
+            -0.010640464723110199,
+            -0.01853214204311371,
+            -0.004074411001056433,
+            -0.0021790785249322653,
+            0.022185368463397026,
+            -0.010915343649685383,
+            -0.02356862835586071,
+            0.025271102786064148,
+            0.009860163554549217,
+            -0.022894732654094696,
+            -0.009603018872439861,
+            -0.00011146163888042793,
+            0.00801138300448656,
+            0.00962075311690569,
+            0.022274039685726166,
+            -0.01566808484494686,
+            -0.007670001592487097,
+            0.0003970777615904808,
+            0.008804984390735626,
+            0.019170571118593216,
+            0.0070892092771828175,
+            -0.005001018289476633,
+            -0.0011915103532373905,
+            0.01173997949808836,
+            -0.012662152759730816,
+            -0.023320350795984268,
+            0.007355221081525087,
+            0.00723994942381978,
+            -0.006787729449570179,
+            0.011881851591169834,
+            -0.013575458899140358,
+            -0.02399424836039543,
+            -0.012990233488380909,
+            0.004398058634251356,
+            -0.023621831089258194,
+            -0.01518926303833723,
+            -0.0027199687901884317,
+            -0.025874063372612,
+            -0.005151758436113596,
+            0.008335030637681484,
+            -0.007749804761260748,
+            0.009141932241618633,
+            0.004484512377530336,
+            0.006574920378625393,
+            -0.002575879218056798,
+            -0.0026290814857929945,
+            -9.774541103979573e-05,
+            0.0006384278531186283,
+            -0.002926127752289176,
+            0.0009011143120005727,
+            -0.021937090903520584,
+            0.009939967654645443,
+            -0.008707446977496147,
+            0.010480857454240322,
+            -0.009221736341714859,
+            -0.005377868190407753,
+            -0.015898628160357475,
+            0.016013899818062782,
+            -0.018762685358524323,
+            0.01995087042450905,
+            -1.6495836462127045e-05,
+            0.006406446453183889,
+            -0.0002840228262357414,
+            0.018408004194498062,
+            -0.014417828992009163,
+            -0.007607932202517986,
+            0.011562637984752655,
+            0.006428613793104887,
+            0.024473069235682487,
+            -0.01728188805282116,
+            0.0019540770445019007,
+            0.013655262999236584,
+            0.011837516911327839,
+            0.01218333188444376,
+            -0.00036299502244219184,
+            -0.019330177456140518,
+            0.011456233449280262,
+            0.0007354112458415329,
+            0.005240428727120161,
+            -0.004205200355499983,
+            0.014515367336571217,
+            0.020181413739919662,
+            -0.021387333050370216,
+            0.015357737429440022,
+            0.0009304864797741175,
+            0.00236972002312541,
+            0.012121262960135937,
+            0.005612845066934824,
+            -0.01824839599430561,
+            -0.014151818118989468,
+            0.012928164564073086,
+            0.0066901920363307,
+            -0.028197230771183968,
+            0.024331195279955864,
+            -0.014426696114242077,
+            -0.0004627493617590517,
+            0.010613863356411457,
+            -0.001321190968155861,
+            -0.024685878306627274,
+            -0.00837493222206831,
+            0.000817431544419378,
+            -0.004912347998470068,
+            -0.008286261931061745,
+            0.0030524833127856255,
+            0.018301598727703094,
+            0.009647354483604431,
+            0.0017789526609703898,
+            0.020571565255522728,
+            0.005732550285756588,
+            0.012582349590957165,
+            0.03873129189014435,
+            -0.001021927921101451,
+            0.004471211694180965,
+            -0.02019914798438549,
+            0.011580372229218483,
+            0.011385297402739525,
+            -0.006056197918951511,
+            -0.01528680045157671,
+            -0.025377508252859116,
+            -0.002099274890497327,
+            -0.009780360385775566,
+            0.010862140916287899,
+            0.0018709483556449413,
+            -0.011527169495821,
+            0.004300521221011877,
+            0.00701827323064208,
+            -0.010046372190117836,
+            -0.012759690172970295,
+            0.01904643140733242,
+            0.009266071021556854,
+            0.008778383024036884,
+            0.0008296237210743129,
+            -0.001856539398431778,
+            -0.01528680045157671,
+            0.014302557334303856,
+            -0.020979449152946472,
+            -0.002135851653292775,
+            0.00805571861565113,
+            0.004114313051104546,
+            -0.010959678329527378,
+            0.0057724518701434135,
+            -0.009629620239138603,
+            0.0011582588776946068,
+            -0.00793157983571291,
+            0.035397280007600784,
+            -0.004114313051104546,
+            0.0050896890461444855,
+            0.010401054285466671,
+            -0.00304583297111094,
+            0.006583787500858307,
+            -0.012467077933251858,
+            -0.008836018852889538,
+            0.013247378170490265,
+            0.02612234093248844,
+            -0.010392187163233757,
+            -0.019738061353564262,
+            0.0009947726503014565,
+            0.003639925504103303,
+            -0.00837049912661314,
+            -0.0006694625481031835,
+            0.010995146818459034,
+            0.014417828992009163,
+            -0.008587741293013096,
+            -0.006459648720920086,
+            0.024739079177379608,
+            0.013477921485900879,
+            -0.004351506475359201,
+            -0.016891736537218094,
+            0.009022227488458157,
+            0.0002779267088044435,
+            0.006942903157323599,
+            0.0063798450864851475,
+            -0.007151278667151928,
+            -0.004522197414189577,
+            0.023231681436300278,
+            -0.011004013940691948,
+            -0.002135851653292775,
+            -0.01094194408506155,
+            -0.016492720693349838,
+            -0.022717392072081566,
+            0.002875142265111208,
+            -0.024136120453476906,
+            -4.776432615472004e-05,
+            0.025501646101474762,
+            -0.0016969323623925447,
+            -0.009833562187850475,
+            0.0018576477887108922,
+            -0.016891736537218094,
+            0.012857227586209774,
+            6.400904385372996e-05,
+            -0.001757893362082541,
+            -0.0013788269134238362,
+            0.027133185416460037,
+            0.03816379979252815,
+            0.02481001615524292,
+            -0.002146935323253274,
+            0.01658139005303383,
+            -0.007953747175633907,
+            0.012068060226738453,
+            -0.0054488047026097775,
+            0.01391240768134594,
+            0.015428673475980759,
+            0.01116362027823925,
+            0.004335989244282246,
+            -0.004992151167243719,
+            -0.006433047354221344,
+            0.006526151672005653,
+            -0.008663111366331577,
+            0.007758671883493662,
+            0.0223272405564785,
+            0.0027776046190410852,
+            0.01537547167390585,
+            0.012059193104505539,
+            -0.011332094669342041,
+            0.016457252204418182,
+            -0.01728188805282116,
+            0.0027111016679555178,
+            -0.016058234497904778,
+            -0.0028640583623200655,
+            0.035822898149490356,
+            0.015703551471233368,
+            -0.0027088848873972893,
+            -0.0050852554850280285,
+            -0.007137978449463844,
+            0.008184290491044521,
+            -0.019525252282619476,
+            -0.0007836258737370372,
+            0.00467737065628171,
+            0.0001997858053073287,
+            -0.019330177456140518,
+            0.025022825226187706,
+            0.0056660473346710205,
+            -0.02092624641954899,
+            0.009328140877187252,
+            -0.015029655769467354,
+            0.012564615346491337,
+            0.020270084962248802,
+            0.0011560420971363783,
+            0.01045425608754158,
+            0.00598082784563303,
+            0.009425678290426731,
+            -0.0026445987168699503,
+            -0.00044169012107886374,
+            -0.01711341366171837,
+            -0.00751039432361722,
+            -0.0014708224916830659,
+            0.012812892906367779,
+            -0.003662093309685588,
+            0.00984242931008339,
+            -0.010064106434583664,
+            0.009266071021556854,
+            -0.0030125814955681562,
+            -0.005107422824949026,
+            -0.013433586806058884,
+            0.003036965848878026,
+            0.001497423741966486,
+            0.012919297441840172,
+            -0.007173446472734213,
+            -0.0029638127889484167,
+            -0.022256305441260338,
+            0.007120244204998016,
+            0.011677909642457962,
+            -0.01788484677672386,
+            0.017787309363484383,
+            0.004797075875103474,
+            -0.014107482507824898,
+            -0.01023257989436388,
+            0.011562637984752655,
+            -0.0019540770445019007,
+            -0.009381342679262161,
+            0.012546881102025509,
+            -0.018177460879087448,
+            -0.009736024774610996,
+            7.412302693410311e-06,
+            0.0205538310110569,
+            0.0022012460976839066,
+            0.010560660623013973,
+            -0.007696602493524551,
+            0.00886262021958828,
+            0.002527110278606415,
+            0.0010834431741386652,
+            0.010773470625281334,
+            -4.464700032258406e-05,
+            0.010365585796535015,
+            0.007625666446983814,
+            -0.014009945094585419,
+            -0.006707926280796528,
+            -0.021103588864207268,
+            -0.001193727133795619,
+            -0.001488556619733572,
+            -0.014027679339051247,
+            0.0068497988395392895,
+            -0.0012070277007296681,
+            0.006938469596207142,
+            0.016882870346307755,
+            -0.004743873607367277,
+            -0.011500569060444832,
+            0.0004724477184936404,
+            0.011935054324567318,
+            0.006836498621851206,
+            -0.004664070438593626,
+            -0.014497633092105389,
+            0.015747888013720512,
+            0.010613863356411457,
+            -0.0005594556569121778,
+            0.0050852554850280285,
+            -0.01879815384745598,
+            -0.01904643140733242,
+            -0.014568569138646126,
+            -0.0050675212405622005,
+            0.010046372190117836,
+            0.004205200355499983,
+            0.0033273620065301657,
+            -0.0014386795228347182,
+            0.0023076506331562996,
+            -0.00571481604129076,
+            -0.00014935444050934166,
+            0.03057360090315342,
+            -0.0015805523144081235,
+            -0.0003696453059092164,
+            0.012103528715670109,
+            0.014905516989529133,
+            0.012812892906367779,
+            -0.005151758436113596,
+            0.012653285637497902,
+            0.00907986331731081,
+            0.010480857454240322,
+            0.01709567941725254,
+            -0.02693810872733593,
+            0.02135186456143856,
+            -0.011261158622801304,
+            -0.010631597600877285,
+            0.018106523901224136,
+            0.003500269493088126,
+            -0.0022400394082069397,
+            -0.002502725925296545,
+            0.0059098913334310055,
+            0.01674986444413662,
+            -0.003901503747329116,
+            -0.017849380150437355,
+            0.0004815918509848416,
+            -0.013123239390552044,
+            -0.00751039432361722,
+            -0.017503563314676285,
+            -0.017476962879300117,
+            0.0054266368970274925,
+            -0.0028396740090101957,
+            0.008844885975122452,
+            0.003846084466204047,
+            -0.021050386130809784,
+            -0.010463123209774494,
+            -0.005741417407989502,
+            -0.001937451190315187,
+            0.06781522929668427,
+            -0.005169492214918137,
+            0.018008986487984657,
+            -0.0013566592242568731,
+            0.006220238283276558,
+            0.0070936428382992744,
+            -0.01851440779864788,
+            -0.006681324914097786,
+            -0.005626145750284195,
+            -0.03383667767047882,
+            0.006162602454423904,
+            0.013735066168010235,
+            -0.007749804761260748,
+            0.005267030093818903,
+            0.0003879335999954492,
+            -0.020890777930617332,
+            -0.020766640082001686,
+            -0.006996105425059795,
+            0.001219219877384603,
+            -0.008663111366331577,
+            0.021546941250562668,
+            -0.007944880053400993,
+            -0.0011349829146638513,
+            0.006167036015540361,
+            0.004531064536422491,
+            -0.0015739020891487598,
+            -0.017202084884047508,
+            0.005630579311400652,
+            -2.2150315999169834e-05,
+            -0.03630171716213226,
+            0.0036975613329559565,
+            -0.0022810494992882013,
+            -0.012334072031080723,
+            -0.006632556207478046,
+            0.011855251155793667,
+            0.016457252204418182,
+            -0.009611885994672775,
+            0.01975579559803009,
+            -0.020075010135769844,
+            0.0014741476625204086,
+            0.007594631519168615,
+            0.011083817109465599,
+            -0.004322688560932875,
+            -0.002511593047529459,
+            -0.020075010135769844,
+            -0.0007210022886283696,
+            0.029988376423716545,
+            -0.03591156750917435,
+            0.0038106164429336786,
+            -0.010445388965308666,
+            -0.004178598988801241,
+            -0.007816308178007603,
+            0.01842573843896389,
+            -0.002203462878242135,
+            -0.025625785812735558,
+            0.017521297559142113,
+            0.005484272725880146,
+            0.002493858803063631,
+            0.0016492720460519195,
+            -0.022486848756670952,
+            0.004699538461863995,
+            -0.022983403876423836,
+            -0.0008013600017875433,
+            -0.008237493224442005,
+            -0.018674015998840332,
+            0.011509436182677746,
+            -0.00984242931008339,
+            0.010525193065404892,
+            -0.024739079177379608,
+            0.006189203355461359,
+            0.0028241565451025963,
+            -0.012786291539669037,
+            -0.011713378131389618,
+            -0.006446348037570715,
+            0.0054399375803768635,
+            -0.024685878306627274,
+            0.0008146605687215924,
+            0.01657252386212349,
+            0.015570546500384808,
+            0.005958660040050745,
+            0.0057901861146092415,
+            -0.006916301790624857,
+            -0.01710454747080803,
+            0.003584506455808878,
+            -0.0038992869667708874,
+            -0.022894732654094696,
+            0.0013622011756524444,
+            -0.0022267389576882124,
+            0.006521718110889196,
+            0.023533159866929054,
+            0.012812892906367779,
+            0.021919356659054756,
+            0.008481336757540703,
+            0.014763644896447659,
+            0.005847821943461895,
+            0.024934155866503716,
+            0.0042428853921592236,
+            -0.023338085040450096,
+            -0.004136480391025543,
+            -0.000975375936832279,
+            0.0014176202239468694,
+            0.014072014018893242,
+            -0.0063887122087180614,
+            -0.024065183475613594,
+            -0.03890863060951233,
+            -0.0058832899667322636,
+            0.013664130121469498,
+            0.00757246371358633,
+            -0.008303996175527573,
+            0.006809897255152464,
+            -0.0019197171786800027,
+            -0.0035468214191496372,
+            0.0027842549607157707,
+            0.0026845005340874195,
+            -0.020979449152946472,
+            0.028924329206347466,
+            0.014728176407516003,
+            0.046818044036626816,
+            0.03204553201794624,
+            0.009904499165713787,
+            -0.003105685580521822,
+            0.01950751803815365,
+            0.001809987355954945,
+            -0.01186411827802658,
+            0.016528187319636345,
+            0.022965669631958008,
+            -0.02498735673725605,
+            0.02126319520175457,
+            -0.008703012950718403,
+            -0.00784290861338377,
+            -0.00407884456217289,
+            -0.011935054324567318,
+            0.0170868132263422,
+            0.031673114746809006,
+            -0.006583787500858307,
+            0.000389319087844342,
+            -0.02417158894240856,
+            0.01710454747080803,
+            -0.010604996234178543,
+            0.016856269910931587,
+            -0.007244382984936237,
+            0.045044634491205215,
+            0.009665088728070259,
+            0.006557186134159565,
+            0.0035623388830572367,
+            -0.011066082864999771,
+            0.0006689083529636264,
+            0.0055507756769657135,
+            -0.002966029569506645,
+            0.004420226439833641,
+            0.019826732575893402,
+            -0.0053246659226715565,
+            0.023001138120889664,
+            0.005107422824949026,
+            -0.009549817070364952,
+            0.004167515318840742,
+            0.014861182309687138,
+            0.002815289655700326,
+            0.020128212869167328,
+            -0.006220238283276558,
+            -0.013353782705962658,
+            -0.0026290814857929945,
+            0.002706668106839061,
+            0.003090168349444866,
+            -0.0019518602639436722,
+            -0.006685758475214243,
+            -0.001582769094966352,
+            -0.01647498644888401,
+            -0.005608411505818367,
+            -0.018017852678894997,
+            -0.01311437226831913,
+            0.002930561313405633,
+            -0.013814869336783886,
+            0.01824839599430561,
+            0.029349947348237038,
+            -0.004087711684405804,
+            -0.0026645497418940067,
+            0.018283864483237267,
+            -0.014825713820755482,
+            -0.013442453928291798,
+            0.004236234817653894,
+            0.010551794432103634,
+            0.005630579311400652,
+            0.003850518027320504,
+            0.03587609902024269,
+            0.003948055673390627,
+            0.010374452918767929,
+            -0.013273979537189007,
+            -0.02372823655605316,
+            -0.000808564480394125,
+            -0.0043670241720974445,
+            -0.008428134955465794,
+            0.00013272871728986502,
+            -0.009470012970268726,
+            -0.009363608434796333,
+            0.01646611839532852,
+            0.01568581722676754,
+            -0.016758732497692108,
+            -0.021582409739494324,
+            0.0001651488710194826,
+            0.0019540770445019007,
+            -0.0037795817479491234,
+            -0.0311942957341671,
+            -0.011722245253622532
+          ],
+          "title": "Preventing Token Dropping with No-Token-Left-Behind"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "d0563e52-5eb6-410a-b369-9c07d6eeb84a",
+      "type": "child",
+      "source": {
+        "id": "65742957-6bda-446d-a61a-0fe20313c760",
+        "properties": {
+          "page_content": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity\n\nWilliam Fedus∗ liamfedus@google.com Barret Zoph∗ barretzoph@google.com Noam Shazeer noam@google.com Google, Mountain View, CA 94043, USA Editor: Alexander Clark\n\nAbstract\n\nIn deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) models defy this and instead select different parameters for each incoming example. The result is a sparsely-activated model—with an outrageous number of parameters—but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs, and training instability. We address these with the introduction of the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques mitigate the instabilities, and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large (Raffel et al., 2019) to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the \"Colossal Clean Crawled Corpus\", and achieve a 4x speedup over the T5-XXL model.12 Keywords: mixture-of-experts, natural language processing, sparsity, large-scale machine learning, distributed computing arXiv:2101.03961v3 [cs.LG] 16 Jun 2022 Contents\n\n1 Introduction 3 2 Switch Transformer 4 2.1 Simplifying Sparse Routing 5 2.2 Efficient Sparse Routing 6 2.3 Putting It All Together: The Switch Transformer 8 2.4 Improved Training and Fine-Tuning Techniques 8 3 Scaling Properties 11 3.1 Scaling Results on a Step-Basis 12 3.2 Scaling Results on a Time-Basis 13 3.3 Scaling Versus a Larger Dense Model 13 4 Downstream Results 14 4.1 Fine-Tuning 14 4.2 Distillation 16 4.3 Multilingual Learning 17 5 Designing Models with Data, Model, and Expert-Parallelism 18 5.1 Data Parallelism 20 5.2 Model Parallelism 20 5.3 Model and Data Parallelism 21 5.4 Expert and Data Parallelism 22 5.5 Expert, Model and Data Parallelism 22 5.6 Towards Trillion Parameter Models 22 6 Related Work 24 7 Discussion 25 8 Future Work 26 9 Conclusion 27 A Switch for Attention 27 B Preventing Token Dropping with No-Token-Left-Behind 29\n\nC Encouraging Exploration Across Experts 29 D Switch Transformers in Lower Compute Regimes 29 E Relation of Upstream to Downstream Model Performance 32 F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/switch_transformers.md"
+          },
+          "summary": "The Switch Transformer is a sparsely-activated model that simplifies the Mixture of Experts (MoE) approach, allowing for efficient scaling to trillion-parameter models with reduced communication and computational costs. It achieves significant speedups in pre-training and fine-tuning by activating only a subset of neural network weights for each input, maintaining a constant computational cost. The model demonstrates superior scaling properties and improved performance across various natural language processing tasks, including multilingual settings. It also allows for effective distillation into smaller dense models, preserving a portion of the quality gains. The architecture is designed to maximize parameter count while keeping floating point operations per example constant, making it suitable for both large-scale and smaller-scale applications.",
+          "headlines": {
+            "1 Introduction": [],
+            "2 Switch Transformer": [
+              "2.1 Simplifying Sparse Routing",
+              "2.2 Efficient Sparse Routing",
+              "2.3 Putting It All Together: The Switch Transformer",
+              "2.4 Improved Training and Fine-Tuning Techniques"
+            ],
+            "3 Scaling Properties": [
+              "3.1 Scaling Results on a Step-Basis",
+              "3.2 Scaling Results on a Time-Basis",
+              "3.3 Scaling Versus a Larger Dense Model"
+            ],
+            "4 Downstream Results": [
+              "4.1 Fine-Tuning",
+              "4.2 Distillation",
+              "4.3 Multilingual Learning"
+            ],
+            "5 Designing Models with Data, Model, and Expert-Parallelism": [
+              "5.1 Data Parallelism",
+              "5.2 Model Parallelism",
+              "5.3 Model and Data Parallelism",
+              "5.4 Expert and Data Parallelism",
+              "5.5 Expert, Model and Data Parallelism",
+              "5.6 Towards Trillion Parameter Models"
+            ],
+            "6 Related Work": [],
+            "7 Discussion": [],
+            "8 Future Work": [],
+            "9 Conclusion": [],
+            "A Switch for Attention": [],
+            "B Preventing Token Dropping with No-Token-Left-Behind": [],
+            "C Encouraging Exploration Across Experts": [],
+            "D Switch Transformers in Lower Compute Regimes": [],
+            "E Relation of Upstream to Downstream Model Performance": [],
+            "F Pseudo Code for Switch Transformers": []
+          },
+          "summary_embedding": [
+            -0.0060447100549936295,
+            -0.00390331051312387,
+            -0.01784370467066765,
+            -0.0021820589900016785,
+            0.013227885589003563,
+            0.011338187381625175,
+            0.0165580902248621,
+            0.024240797385573387,
+            -0.005986624862998724,
+            0.03596621751785278,
+            -0.004852031823247671,
+            -0.0068811336532235146,
+            -0.003574162255972624,
+            0.014242436736822128,
+            0.011578272096812725,
+            -0.033302053809165955,
+            -0.024519605562090874,
+            0.01050176378339529,
+            0.016093410551548004,
+            -0.006795942317694426,
+            0.007473600562661886,
+            -0.030513975769281387,
+            -0.06124480068683624,
+            -0.006838538218289614,
+            -0.0026099516544491053,
+            0.01029265858232975,
+            0.0010329445358365774,
+            -0.010873507708311081,
+            -0.053469158709049225,
+            0.023311438038945198,
+            0.025294071063399315,
+            0.005092116538435221,
+            -0.029584616422653198,
+            0.0009361362317577004,
+            -0.05161044001579285,
+            -0.04448534920811653,
+            0.0379488542675972,
+            0.012561844661831856,
+            -0.04364892467856407,
+            0.039249956607818604,
+            -0.004875265993177891,
+            -0.0022440163884311914,
+            -0.021499186754226685,
+            -0.0034909071400761604,
+            -0.007039899472147226,
+            -0.009851212613284588,
+            -0.02874819189310074,
+            0.0014037203509360552,
+            0.021886419504880905,
+            -0.0056962003000080585,
+            0.023264970630407333,
+            -0.024271776899695396,
+            0.01936165988445282,
+            0.04240977764129639,
+            -0.06957805901765823,
+            0.024659009650349617,
+            0.017549408599734306,
+            0.05120771750807762,
+            0.06511713564395905,
+            0.018339363858103752,
+            0.021669568493962288,
+            -0.014141756109893322,
+            -0.017766257748007774,
+            -0.011872569099068642,
+            0.004673904739320278,
+            -0.014970434829592705,
+            -0.01891246810555458,
+            0.008449427783489227,
+            -0.020461400970816612,
+            0.008209343999624252,
+            -0.05576157942414284,
+            -0.014730350114405155,
+            -0.034200433641672134,
+            -0.01140014547854662,
+            0.017642343416810036,
+            -0.018215449526906013,
+            0.030126741155982018,
+            0.02997184917330742,
+            0.002638994250446558,
+            0.0007822110201232135,
+            0.021049996837973595,
+            0.006509389728307724,
+            -0.0053438181057572365,
+            0.012321759946644306,
+            0.028159597888588905,
+            -0.01450575515627861,
+            -0.010106786154210567,
+            -0.03454120084643364,
+            0.01861817203462124,
+            0.01589204929769039,
+            -0.04194509983062744,
+            -0.012569589540362358,
+            -0.013824224472045898,
+            0.032341714948415756,
+            -0.009549170732498169,
+            0.04098476096987724,
+            -0.010083552449941635,
+            0.001383390510454774,
+            -0.031149037182331085,
+            -0.009463978931307793,
+            0.017069239169359207,
+            0.006172496825456619,
+            -0.016589069738984108,
+            -0.0012846461031585932,
+            0.016170857474207878,
+            0.006617815233767033,
+            -0.030142230913043022,
+            0.04488807171583176,
+            0.023497309535741806,
+            0.055296897888183594,
+            0.05579255893826485,
+            -0.037391237914562225,
+            -0.002445377642288804,
+            0.0344172865152359,
+            -0.0228312686085701,
+            -0.015040136873722076,
+            -0.034014564007520676,
+            -0.01768881268799305,
+            0.035935238003730774,
+            -0.0318460576236248,
+            0.01556677371263504,
+            0.028066661208868027,
+            -0.025712283328175545,
+            0.003488970920443535,
+            -0.03943582624197006,
+            0.003622566582635045,
+            0.01220559049397707,
+            0.006095050368458033,
+            -0.0026293133851140738,
+            -0.013615119270980358,
+            0.008906363509595394,
+            0.003055269829928875,
+            0.00800798274576664,
+            -0.04615819454193115,
+            0.018478767946362495,
+            -0.00454224506393075,
+            0.04033420979976654,
+            0.042502716183662415,
+            -0.02730768360197544,
+            -0.016743963584303856,
+            0.008557853288948536,
+            0.012104909867048264,
+            0.04188314080238342,
+            -0.007287728600203991,
+            -0.00121784838847816,
+            -0.0007836631848476827,
+            0.007322579622268677,
+            0.047552235424518585,
+            0.004646798130124807,
+            0.0020697612781077623,
+            -0.027230238541960716,
+            0.014978179708123207,
+            -0.019733402878046036,
+            0.027648448944091797,
+            0.023373395204544067,
+            0.02942972257733345,
+            -0.04048910364508629,
+            0.017518429085612297,
+            -0.011818356812000275,
+            -0.005568413063883781,
+            -0.002726121572777629,
+            -0.02856232039630413,
+            -0.0074426219798624516,
+            0.033983584493398666,
+            0.015365412458777428,
+            0.012956822291016579,
+            0.03271345794200897,
+            0.0012052633101120591,
+            -0.042502716183662415,
+            0.03804178908467293,
+            0.024550583213567734,
+            0.04640602692961693,
+            0.023466331884264946,
+            0.010796061716973782,
+            0.006571347359567881,
+            0.013375034555792809,
+            -0.019377149641513824,
+            0.015907539054751396,
+            0.02560385875403881,
+            0.009750531986355782,
+            -0.03167567402124405,
+            -0.0009501734166406095,
+            0.01178737822920084,
+            -0.004546117503196001,
+            0.03701949119567871,
+            0.04104671627283096,
+            0.020569827407598495,
+            0.05030933395028114,
+            0.0020155487582087517,
+            0.017936641350388527,
+            0.005936284549534321,
+            0.02820606529712677,
+            0.030467506498098373,
+            -0.005285732913762331,
+            0.005076627247035503,
+            -0.01136916596442461,
+            -0.03385967016220093,
+            0.014110777527093887,
+            0.01561324205249548,
+            -0.008077683858573437,
+            -0.01697630248963833,
+            0.006033093202859163,
+            0.009169681929051876,
+            -0.01795213110744953,
+            -0.017456471920013428,
+            0.005971135571599007,
+            0.018866000697016716,
+            -0.013576395809650421,
+            0.024689987301826477,
+            -0.04442339017987251,
+            0.029119934886693954,
+            -0.033209118992090225,
+            -0.003384418087080121,
+            -0.010641167871654034,
+            -0.0030843124259263277,
+            0.0041666291654109955,
+            0.011314953677356243,
+            0.013406013138592243,
+            0.002042655134573579,
+            -0.013909416273236275,
+            0.024519605562090874,
+            0.018447788432240486,
+            0.012073930352926254,
+            -0.03148980438709259,
+            -0.014397329650819302,
+            -0.0015673263696953654,
+            -0.008743725717067719,
+            -0.034138478338718414,
+            0.026966920122504234,
+            0.014048820361495018,
+            -0.03844451159238815,
+            0.017797237262129784,
+            0.029538147151470184,
+            0.014420564286410809,
+            -0.004790074657648802,
+            -0.0463130883872509,
+            0.04045812413096428,
+            -0.030436528846621513,
+            -0.033333033323287964,
+            -0.010478530079126358,
+            -0.0003146269591525197,
+            0.02894955314695835,
+            -0.0064358157105743885,
+            -0.050835974514484406,
+            0.01184933539479971,
+            0.020631784573197365,
+            0.00010763872705865651,
+            0.005731051322072744,
+            -0.019593998789787292,
+            0.00028631053282879293,
+            -0.03395260497927666,
+            0.0090844901278615,
+            0.026750069111585617,
+            0.008867640048265457,
+            -0.016418687999248505,
+            0.03351890295743942,
+            -0.0004121613164898008,
+            -0.018695618957281113,
+            -0.003889757441356778,
+            0.006885006092488766,
+            0.034912943840026855,
+            0.004836542531847954,
+            -0.02946070022881031,
+            0.05665995925664902,
+            -0.02100352756679058,
+            0.021344292908906937,
+            0.05631919577717781,
+            -0.019439106807112694,
+            -0.008472662419080734,
+            -0.002065889071673155,
+            0.0026196325197815895,
+            -0.0260530486702919,
+            -0.008937342092394829,
+            0.032341714948415756,
+            0.041728246957063675,
+            -0.018742086365818977,
+            -0.001398879918269813,
+            0.006780453026294708,
+            0.009425255469977856,
+            0.004948840010911226,
+            0.02692045085132122,
+            -0.012128143571317196,
+            -0.014559967443346977,
+            0.05198218300938606,
+            0.028314489871263504,
+            -0.012081675231456757,
+            -0.0164031982421875,
+            0.019888296723365784,
+            -0.024844881147146225,
+            -0.020244551822543144,
+            -0.02431824430823326,
+            -0.020414933562278748,
+            -0.0042518205009400845,
+            0.009525936096906662,
+            0.037484172731637955,
+            -0.03968365862965584,
+            0.012747716158628464,
+            0.021700548008084297,
+            0.02981695532798767,
+            -0.019532041624188423,
+            -0.05910727381706238,
+            -0.0016128262504935265,
+            0.03296129032969475,
+            -0.015458349138498306,
+            -0.049379974603652954,
+            0.015164051204919815,
+            0.0260530486702919,
+            0.05957195535302162,
+            -0.001461805310100317,
+            -0.014954946003854275,
+            0.061926331371068954,
+            0.03026614524424076,
+            -0.004383479710668325,
+            -0.009494957514107227,
+            -0.0127864396199584,
+            0.003396035172045231,
+            0.00465454300865531,
+            -0.03364282101392746,
+            -0.041170634329319,
+            -0.011841590516269207,
+            -0.010284913703799248,
+            0.008108663372695446,
+            -0.040922801941633224,
+            0.03531566634774208,
+            -0.021917399019002914,
+            0.010950954630970955,
+            0.015473837964236736,
+            0.037391237914562225,
+            -0.021344292908906937,
+            -0.01284839678555727,
+            -0.01572941243648529,
+            -0.024968795478343964,
+            -0.014010096900165081,
+            0.021917399019002914,
+            0.007682706229388714,
+            3.4699722164077684e-05,
+            0.010075807571411133,
+            -0.024659009650349617,
+            -0.051052823662757874,
+            0.0291044469922781,
+            0.0015605497173964977,
+            -0.015938518568873405,
+            -0.004623564425855875,
+            -0.01977987214922905,
+            -0.01122201792895794,
+            -0.012608313001692295,
+            -0.039249956607818604,
+            0.021111954003572464,
+            -0.01135367713868618,
+            0.015876561403274536,
+            0.0009250032599084079,
+            0.003659353591501713,
+            0.028624277561903,
+            0.01116006076335907,
+            0.0051811798475682735,
+            -0.03255856782197952,
+            -0.018277406692504883,
+            0.010122275911271572,
+            -0.00561100896447897,
+            -0.018339363858103752,
+            -0.013351799920201302,
+            -0.05037129297852516,
+            -0.03305422514677048,
+            -0.008054450154304504,
+            -0.05415068939328194,
+            0.0020697612781077623,
+            -0.014296649023890495,
+            -0.02232012152671814,
+            -0.005990497302263975,
+            0.03107159025967121,
+            -0.003235333366319537,
+            0.025294071063399315,
+            0.0076594725251197815,
+            0.037360258400440216,
+            0.040117356926202774,
+            0.008426194079220295,
+            0.03065337985754013,
+            -0.0029991210903972387,
+            -0.016867877915501595,
+            0.05486319959163666,
+            -0.01881953328847885,
+            0.013529927469789982,
+            -0.008604321628808975,
+            0.008604321628808975,
+            0.03299226611852646,
+            -0.003794885240495205,
+            0.02007416822016239,
+            -0.0311955064535141,
+            -0.07998688519001007,
+            0.01637221872806549,
+            0.028159597888588905,
+            -0.06418777257204056,
+            0.014691627584397793,
+            -0.020275529474020004,
+            0.02052335813641548,
+            0.02411688305437565,
+            -0.0009114501299336553,
+            -0.014064309187233448,
+            0.005595519673079252,
+            0.0020833145827054977,
+            -0.01456771232187748,
+            -0.002691270550712943,
+            -0.009417511522769928,
+            0.008774704299867153,
+            -0.017007282003760338,
+            -0.013406013138592243,
+            -0.034231413155794144,
+            0.010873507708311081,
+            0.01233724970370531,
+            -0.003969140350818634,
+            0.0011500825639814138,
+            -0.03277541697025299,
+            -0.007876322604715824,
+            -0.02145271934568882,
+            0.03382869064807892,
+            0.012058441527187824,
+            -0.030699847266077995,
+            -0.007814365439116955,
+            0.015264731831848621,
+            -0.007876322604715824,
+            0.0008906363509595394,
+            0.024132372811436653,
+            0.02103450708091259,
+            0.0028364830650389194,
+            -0.0009700191440060735,
+            0.02438020147383213,
+            0.007427132688462734,
+            -0.01720864325761795,
+            0.028794659301638603,
+            0.014288905076682568,
+            0.060160547494888306,
+            -0.03271345794200897,
+            0.001297231181524694,
+            -0.04135650396347046,
+            -0.031118059530854225,
+            0.006408709101378918,
+            0.014350862242281437,
+            0.007597515359520912,
+            -0.028252532705664635,
+            0.005978880450129509,
+            0.02004319056868553,
+            0.01469937153160572,
+            0.009704063646495342,
+            0.0037580979987978935,
+            -0.036740683019161224,
+            -0.000691695255227387,
+            -0.011214273050427437,
+            -0.0038781403563916683,
+            0.035656433552503586,
+            -0.008186109364032745,
+            -0.05151750147342682,
+            -0.0006418389966711402,
+            -0.03255856782197952,
+            -0.0353776253759861,
+            0.021313315257430077,
+            -0.00839521549642086,
+            -0.0039207362569868565,
+            -0.013986863195896149,
+            -0.000537770101800561,
+            -0.010524998418986797,
+            0.027509044855833054,
+            0.028345469385385513,
+            0.022490503266453743,
+            -0.05384090170264244,
+            0.009479468688368797,
+            0.00023766436788719147,
+            -0.05817791447043419,
+            0.0009709871956147254,
+            -0.007117345929145813,
+            0.011191039346158504,
+            -0.020213572308421135,
+            -0.02414786070585251,
+            0.011500826105475426,
+            0.03392162546515465,
+            0.014304393902420998,
+            0.0321558453142643,
+            -0.013367289677262306,
+            -0.09250226616859436,
+            0.018292896449565887,
+            -0.008689512498676777,
+            0.04126356914639473,
+            0.00865078903734684,
+            -0.004278926644474268,
+            -0.015845581889152527,
+            -0.00865078903734684,
+            -0.006943091284483671,
+            0.01283290795981884,
+            0.004530628211796284,
+            -0.005626498255878687,
+            0.0016631665639579296,
+            0.013708055019378662,
+            -0.005967263597995043,
+            0.007481344975531101,
+            0.003651608945801854,
+            -0.013189162127673626,
+            -0.06059424951672554,
+            -0.04055105894804001,
+            -0.018540725111961365,
+            0.050835974514484406,
+            -0.0069159846752882,
+            -0.046684831380844116,
+            -0.010850274004042149,
+            0.05058814212679863,
+            0.012259802781045437,
+            -0.0334569476544857,
+            -0.0007841471815481782,
+            0.0050882440991699696,
+            0.015458349138498306,
+            0.004546117503196001,
+            -0.027292195707559586,
+            0.02820606529712677,
+            0.0008412640891037881,
+            0.02300165221095085,
+            -0.018339363858103752,
+            -0.005312839522957802,
+            0.035780347883701324,
+            -0.024844881147146225,
+            0.023791607469320297,
+            0.0055219451896846294,
+            0.031629208475351334,
+            0.002952652983367443,
+            0.010060317814350128,
+            0.004704883322119713,
+            0.0186491496860981,
+            0.004275054205209017,
+            0.0276639387011528,
+            0.011462102644145489,
+            0.006428070832043886,
+            0.021623101085424423,
+            -0.009378788061439991,
+            -0.014513500034809113,
+            0.01885051093995571,
+            -0.0014695499558001757,
+            0.025232113897800446,
+            -0.020538847893476486,
+            0.002726121572777629,
+            -0.0016883367206901312,
+            -0.017967619001865387,
+            -0.040922801941633224,
+            -0.0007405834621749818,
+            0.021747015416622162,
+            -0.011671207845211029,
+            -0.0228312686085701,
+            -0.013715799897909164,
+            -0.01585332676768303,
+            0.0038490979932248592,
+            0.034169457852840424,
+            -0.0398695282638073,
+            0.00963436160236597,
+            -0.000830615172162652,
+            -0.03125746175646782,
+            -0.0239619892090559,
+            -0.01093546487390995,
+            0.014614180661737919,
+            -0.01652711257338524,
+            -0.021173911169171333,
+            -0.028469383716583252,
+            -0.008410705253481865,
+            0.006048582494258881,
+            -0.05065010115504265,
+            -0.004054331686347723,
+            0.019965743646025658,
+            0.0276639387011528,
+            -0.003099801717326045,
+            -0.02177799493074417,
+            0.0012468908680602908,
+            0.022351099178195,
+            0.02936776541173458,
+            0.014226946979761124,
+            0.027586491778492928,
+            -0.002178186783567071,
+            -0.0011675080750137568,
+            -0.0073148347437381744,
+            0.01307299267500639,
+            0.0011287847300991416,
+            0.036616768687963486,
+            0.0122210793197155,
+            -0.020306508988142014,
+            0.0013959755888208747,
+            -0.006431943271309137,
+            0.029181893914937973,
+            -0.02270735427737236,
+            -9.832092473516241e-05,
+            0.03531566634774208,
+            0.027741385623812675,
+            0.004933350719511509,
+            -0.0143121387809515,
+            -0.05142456665635109,
+            0.021592123433947563,
+            -0.01765783317387104,
+            0.031025122851133347,
+            0.016666516661643982,
+            0.003397971158847213,
+            -0.0034212053287774324,
+            0.012368228286504745,
+            -0.022087780758738518,
+            -0.004406713880598545,
+            -0.021142931655049324,
+            0.04389675334095955,
+            -0.043184246867895126,
+            0.008472662419080734,
+            0.029445212334394455,
+            0.011926782317459583,
+            -0.019082851707935333,
+            -0.005425137002021074,
+            0.029212871566414833,
+            -0.00162637943867594,
+            0.01308848150074482,
+            0.0058665829710662365,
+            -0.026935940608382225,
+            0.01202746294438839,
+            -0.031660184264183044,
+            0.02177799493074417,
+            -0.013436991721391678,
+            0.018447788432240486,
+            0.03342596814036369,
+            -0.0030397805385291576,
+            -0.0038239278364926577,
+            -0.016263794153928757,
+            -0.00969631876796484,
+            0.0662323608994484,
+            0.01054823212325573,
+            0.03726732358336449,
+            0.0018974426202476025,
+            -0.009773765690624714,
+            0.007969259284436703,
+            0.0063041565008461475,
+            -0.012329504825174809,
+            0.010494019836187363,
+            0.024178840219974518,
+            0.0012730291346088052,
+            -0.002412462839856744,
+            -0.019408127292990685,
+            -0.03438630700111389,
+            0.003727119415998459,
+            -0.030932186171412468,
+            0.022691864520311356,
+            0.009208405390381813,
+            0.017456471920013428,
+            0.008240322582423687,
+            -0.015907539054751396,
+            0.03200095146894455,
+            0.05461536720395088,
+            0.0019768255297094584,
+            0.013375034555792809,
+            0.007938280701637268,
+            0.007036027032881975,
+            -0.011338187381625175,
+            0.04702559858560562,
+            0.000662652775645256,
+            0.005169562995433807,
+            0.022536970674991608,
+            0.02145271934568882,
+            0.022939693182706833,
+            0.002304037567228079,
+            -0.011694442480802536,
+            0.010478530079126358,
+            -0.0006302220281213522,
+            0.004445436876267195,
+            -0.010695381090044975,
+            -0.027865299955010414,
+            0.007492962293326855,
+            0.01302652433514595,
+            0.022598929703235626,
+            0.021592123433947563,
+            -0.03178409859538078,
+            -0.01855621486902237,
+            0.03364282101392746,
+            -0.02997184917330742,
+            0.03612111136317253,
+            -0.0014424435794353485,
+            -0.023807097226381302,
+            -0.0058356039226055145,
+            0.016929835081100464,
+            0.02656419761478901,
+            -0.029181893914937973,
+            0.01804506592452526,
+            0.029693040996789932,
+            -0.0074426219798624516,
+            0.0005939188995398581,
+            0.012112653814256191,
+            0.02447313815355301,
+            -0.009293596260249615,
+            -0.0024027819745242596,
+            -0.0013320822035893798,
+            -0.011926782317459583,
+            -0.018370341509580612,
+            -0.0044725434854626656,
+            -0.022629907354712486,
+            0.005564541090279818,
+            -0.007365175057202578,
+            -0.003481226274743676,
+            -0.008581086993217468,
+            0.01154729351401329,
+            0.001979729626327753,
+            -0.019857317209243774,
+            -0.022335609421133995,
+            0.01652711257338524,
+            0.03887821361422539,
+            -0.018215449526906013,
+            -0.02106548473238945,
+            0.024442158639431,
+            -0.01759587600827217,
+            -0.01601596549153328,
+            0.036678727716207504,
+            0.013739033602178097,
+            0.007736918982118368,
+            -0.03358086198568344,
+            -0.030049296095967293,
+            -0.03274443745613098,
+            0.006296411622315645,
+            0.03785591572523117,
+            -0.0028752065263688564,
+            0.02569679357111454,
+            0.03701949119567871,
+            0.0057891360484063625,
+            -0.028934063389897346,
+            -0.014823286794126034,
+            -0.045104920864105225,
+            0.060160547494888306,
+            0.0011481463443487883,
+            0.05161044001579285,
+            0.016294771805405617,
+            -0.01456771232187748,
+            0.00419760774821043,
+            -0.013251119293272495,
+            0.026192452758550644,
+            0.02280029095709324,
+            0.025929134339094162,
+            0.006582964211702347,
+            0.022505993023514748,
+            -0.0037522895727306604,
+            -0.012197845615446568,
+            0.007307090330868959,
+            0.014536733739078045,
+            0.0002497654058970511,
+            -0.011392400600016117,
+            0.015783624723553658,
+            -0.01307299267500639,
+            -0.011361422017216682,
+            0.023295948281884193,
+            0.01368482131510973,
+            -0.012344993650913239,
+            0.014064309187233448,
+            -0.008565598167479038,
+            0.01239920686930418,
+            -0.023915521800518036,
+            0.022691864520311356,
+            -0.03262052312493324,
+            0.009587894193828106,
+            0.0015053689712658525,
+            -0.0073148347437381744,
+            0.024550583213567734,
+            -0.0027454833034425974,
+            -0.05071205645799637,
+            0.03550153970718384,
+            -0.008015726692974567,
+            0.0037213109899312258,
+            0.006381602957844734,
+            0.011392400600016117,
+            -0.0012623801594600081,
+            0.03807276859879494,
+            0.01239920686930418,
+            0.015969496220350266,
+            0.02199484594166279,
+            0.011911292560398579,
+            -0.008433938957750797,
+            -0.004096927121281624,
+            -0.0016137943603098392,
+            -0.01784370467066765,
+            0.025944624096155167,
+            0.0018674320308491588,
+            0.03962169960141182,
+            -0.018168980255723,
+            -0.02300165221095085,
+            0.028066661208868027,
+            0.007597515359520912,
+            -0.02351279929280281,
+            -0.0017377089243382215,
+            0.01202746294438839,
+            -0.009185170754790306,
+            0.027385130524635315,
+            0.01196550577878952,
+            0.019113829359412193,
+            -0.018060555681586266,
+            -0.004340883810073137,
+            -0.005072754807770252,
+            -0.015938518568873405,
+            0.02238207869231701,
+            0.020771188661456108,
+            0.011152315884828568,
+            -0.011880313977599144,
+            -0.01307299267500639,
+            0.020600805059075356,
+            0.035036858171224594,
+            -0.0002744515077210963,
+            0.005672966130077839,
+            0.017487451434135437,
+            0.002373739378526807,
+            -0.04470219835639,
+            0.016232814639806747,
+            -0.014025585725903511,
+            0.011214273050427437,
+            0.019593998789787292,
+            -0.021917399019002914,
+            -0.03196997195482254,
+            -0.011539548635482788,
+            -0.0414804182946682,
+            0.00493722315877676,
+            -0.0069159846752882,
+            0.013576395809650421,
+            -0.010176488198339939,
+            -0.0014831031439825892,
+            0.016666516661643982,
+            0.0038103745318949223,
+            0.02177799493074417,
+            -0.011678952723741531,
+            -0.012073930352926254,
+            -0.020213572308421135,
+            -0.020786678418517113,
+            0.004278926644474268,
+            -0.030405549332499504,
+            -0.004488032776862383,
+            0.021375272423028946,
+            -0.04783104360103607,
+            -0.03689557686448097,
+            0.02898053266108036,
+            -0.008503641001880169,
+            0.020755698904395103,
+            0.022955182939767838,
+            -0.015094350092113018,
+            -0.0008717587334103882,
+            -0.018695618957281113,
+            0.02258343994617462,
+            -0.007837600074708462,
+            0.026300877332687378,
+            -0.0027551641687750816,
+            0.0006292539183050394,
+            -0.02659517526626587,
+            0.020306508988142014,
+            0.005227648187428713,
+            0.012275291606783867,
+            -0.02390003204345703,
+            0.006815304048359394,
+            -0.03133491054177284,
+            -0.01784370467066765,
+            0.0180140882730484,
+            -0.011152315884828568,
+            0.027989214286208153,
+            -0.006060199346393347,
+            0.028438406065106392,
+            0.022722844034433365,
+            0.024751944467425346,
+            0.02676555886864662,
+            -0.02363671362400055,
+            0.008495896123349667,
+            0.03726732358336449,
+            0.00865078903734684,
+            -0.015876561403274536,
+            0.008170620538294315,
+            -0.04863648861646652,
+            -0.008062195032835007,
+            0.05030933395028114,
+            0.002127846470102668,
+            0.023141054436564445,
+            0.016294771805405617,
+            -0.002873270306736231,
+            -0.005688455421477556,
+            0.010269423946738243,
+            -0.04296739399433136,
+            -0.023172033950686455,
+            0.022629907354712486,
+            -0.021561143919825554,
+            6.54212328754511e-07,
+            0.03122648410499096,
+            0.012120398692786694,
+            -0.0009109660750254989,
+            -0.05108380317687988,
+            0.02100352756679058,
+            0.01178737822920084,
+            0.014443797990679741,
+            -0.010300402529537678,
+            0.026068538427352905,
+            0.01733255758881569,
+            0.038413532078266144,
+            0.012755461037158966,
+            -0.002679653698578477,
+            -0.03782493621110916,
+            0.009177426807582378,
+            -0.021685058251023293,
+            0.0011471783509477973,
+            0.0074503663927316666,
+            -0.02273833192884922,
+            -0.024782923981547356,
+            0.0043912241235375404,
+            0.004588713403791189,
+            0.011880313977599144,
+            0.022087780758738518,
+            0.009417511522769928,
+            -0.021204888820648193,
+            0.015404135920107365,
+            -0.05446047708392143,
+            -0.015450604259967804,
+            -0.023884542286396027,
+            0.02566581591963768,
+            0.020012211054563522,
+            0.010184233076870441,
+            0.005545179359614849,
+            0.010904486291110516,
+            -0.020988039672374725,
+            -0.009239383973181248,
+            0.020631784573197365,
+            0.011074868962168694,
+            -0.012840652838349342,
+            0.04113965481519699,
+            -0.00011126903700642288,
+            0.031210994347929955,
+            -0.02280029095709324,
+            0.002941036131232977,
+            -0.001241082325577736,
+            -0.03649285435676575,
+            0.009130958467721939,
+            0.01981084980070591,
+            0.02151467651128769,
+            -0.01861817203462124,
+            0.0024143988266587257,
+            0.01048627495765686,
+            0.004766840487718582,
+            0.038351576775312424,
+            0.027121812105178833,
+            -0.044206541031599045,
+            0.011663463898003101,
+            0.012995545752346516,
+            -0.009363298304378986,
+            0.048419639468193054,
+            -0.017518429085612297,
+            0.017673322930932045,
+            0.008240322582423687,
+            -0.01798310875892639,
+            0.02318752370774746,
+            -0.01469937153160572,
+            -0.013282098807394505,
+            0.00650551775470376,
+            0.009192915633320808,
+            -0.0001770381786627695,
+            -0.037577107548713684,
+            -0.017859194427728653,
+            -0.005719434004276991,
+            0.0038297362625598907,
+            0.01562873087823391,
+            0.013498948886990547,
+            0.009967382065951824,
+            0.005161818582564592,
+            -0.00945623405277729,
+            -0.0016089539276435971,
+            0.008124152198433876,
+            -0.0005532593932002783,
+            0.0019216446671634912,
+            -0.01926872320473194,
+            0.02698240801692009,
+            0.02572777308523655,
+            -0.0018354853382334113,
+            -0.04652994126081467,
+            -0.0036709706764668226,
+            0.0037077576853334904,
+            -0.017735280096530914,
+            -0.0035315665882080793,
+            -0.04519785940647125,
+            -0.01810702309012413,
+            0.0143121387809515,
+            -0.014614180661737919,
+            -0.0460033044219017,
+            0.009076746180653572,
+            -0.013886182568967342,
+            0.0063970922492444515,
+            -0.00932457484304905,
+            0.005762029904872179,
+            0.003651608945801854,
+            0.005680711008608341,
+            -0.005909178406000137,
+            0.05108380317687988,
+            -0.03466511517763138,
+            0.028051171451807022,
+            0.008565598167479038,
+            -0.012437930330634117,
+            -0.02261441759765148,
+            0.00887538492679596,
+            -0.044981006532907486,
+            -0.033209118992090225,
+            0.016743963584303856,
+            -0.01781272701919079,
+            0.011074868962168694,
+            0.06161654368042946,
+            0.007330324035137892,
+            -0.023342415690422058,
+            -0.01233724970370531,
+            -0.021049996837973595,
+            -0.04191412031650543,
+            0.003442503046244383,
+            0.005463860463351011,
+            -0.004948840010911226,
+            -0.03807276859879494,
+            -0.021716037765145302,
+            0.04615819454193115,
+            -0.012004229240119457,
+            0.012422440573573112,
+            0.01080380566418171,
+            -0.0019410063978284597,
+            -0.04860550910234451,
+            -0.021019017323851585,
+            0.0395597442984581,
+            0.021886419504880905,
+            -0.008426194079220295,
+            0.0028364830650389194,
+            -0.018277406692504883,
+            -0.003289545886218548,
+            0.006296411622315645,
+            -0.009107724763453007,
+            -0.03562545403838158,
+            0.02067825198173523,
+            0.0054832217283546925,
+            -0.023373395204544067,
+            -0.025805220007896423,
+            -0.03296129032969475,
+            0.009944148361682892,
+            0.02453509531915188,
+            -0.01968693546950817,
+            0.005823987070471048,
+            0.00027203132049180567,
+            -0.000881923595443368,
+            0.00042740863864310086,
+            -0.009959637187421322,
+            -0.019764382392168045,
+            -0.020151615142822266,
+            0.03268248215317726,
+            -0.0060059865936636925,
+            0.0074387495405972,
+            -0.03630698472261429,
+            -0.010974188335239887,
+            0.011322698555886745,
+            -0.002395037328824401,
+            -0.013452480547130108,
+            -0.009092235006392002,
+            -0.01739451475441456,
+            -0.009882191196084023,
+            -0.007876322604715824,
+            0.022660886868834496,
+            0.006168624851852655,
+            0.005409647710621357,
+            0.0023795480374246836,
+            -0.008581086993217468,
+            0.02898053266108036,
+            0.017874684184789658,
+            -0.014188224449753761,
+            -0.00920066051185131,
+            0.03451022133231163,
+            0.02665713243186474,
+            -0.009115469641983509,
+            0.01798310875892639,
+            0.023264970630407333,
+            -0.0010484338272362947,
+            -0.01196550577878952,
+            0.001922612776979804,
+            -0.0009550138493068516,
+            -0.0032972905319184065,
+            0.00845717266201973,
+            -0.015365412458777428,
+            0.0315207801759243,
+            0.04287445917725563,
+            -0.002145271748304367,
+            0.009587894193828106,
+            -0.003727119415998459,
+            -0.008472662419080734,
+            0.017069239169359207,
+            0.009316830895841122,
+            0.04625113308429718,
+            0.01098193321377039,
+            0.005680711008608341,
+            -0.033983584493398666,
+            -0.001229465357027948,
+            -0.01184933539479971,
+            0.03308520466089249,
+            0.02061629481613636,
+            0.00889087375253439,
+            -0.008379725739359856,
+            0.01344473659992218,
+            0.008341003209352493,
+            0.00661007035523653,
+            0.020569827407598495,
+            -0.016542602330446243,
+            -0.017286090180277824,
+            0.030467506498098373,
+            -0.00672236829996109,
+            0.01289486512541771,
+            0.02029101923108101,
+            -0.002089123008772731,
+            0.010842529125511646,
+            0.07100307941436768,
+            -0.006184114143252373,
+            0.015713922679424286,
+            -0.019082851707935333,
+            0.0015789433382451534,
+            -0.007640110794454813,
+            -0.03370477631688118,
+            0.018246427178382874,
+            0.0039962464943528175,
+            0.00865078903734684,
+            0.01198099460452795,
+            0.009053511545062065,
+            -0.007946024648845196,
+            -0.00963436160236597,
+            0.015195030719041824,
+            0.0019080914789810777,
+            0.00033568276558071375,
+            0.001336922636255622,
+            -0.0065210070461034775,
+            -0.004011735785752535,
+            -0.021948376670479774,
+            0.02763296104967594,
+            0.007140580099076033,
+            0.030606910586357117,
+            -0.055296897888183594,
+            0.030095763504505157,
+            -0.00789568480104208,
+            0.02554190158843994,
+            -0.016201836988329887,
+            -0.011090358719229698,
+            0.001609921921044588,
+            0.018153492361307144,
+            0.004491904750466347,
+            0.006462921854108572,
+            0.011888058856129646,
+            0.023233991116285324,
+            -0.03444826602935791,
+            0.017053749412298203,
+            0.023110076785087585,
+            0.0048675211146473885,
+            0.0003535923024173826,
+            0.005165690556168556,
+            0.015458349138498306,
+            0.007198664825409651,
+            -0.023048119619488716,
+            -0.0025692922063171864,
+            0.017611365765333176,
+            -0.02306360937654972,
+            0.0036980770528316498,
+            -0.006923729553818703,
+            0.004623564425855875,
+            -0.0014705180656164885,
+            -0.03509881719946861,
+            -0.00100099784322083,
+            -0.025743262842297554,
+            0.01923774555325508,
+            0.009556914679706097,
+            -0.025867177173495293,
+            0.003932353109121323,
+            0.025650326162576675,
+            -0.0003695656778290868,
+            0.00821708794683218,
+            0.0004683101433329284,
+            0.00869725737720728,
+            -0.00974278710782528,
+            -0.01493945624679327,
+            0.004801691509783268,
+            0.01531120017170906,
+            -0.01493945624679327,
+            -0.007431004662066698,
+            -0.009510447271168232,
+            0.030560443177819252,
+            0.0006674932083114982,
+            0.004189862869679928,
+            -0.005045648664236069,
+            -0.01858719252049923,
+            -0.0032237162813544273,
+            -0.021142931655049324,
+            0.004367990419268608,
+            -0.0009046735358424485,
+            0.04197607934474945,
+            0.0038161829579621553,
+            -0.02029101923108101,
+            0.0032972905319184065,
+            -0.03136589005589485,
+            0.0005000148667022586,
+            -0.007276111748069525,
+            -0.02004319056868553,
+            0.0015005286550149322,
+            0.0016525175888091326,
+            0.012631546705961227,
+            -0.03255856782197952,
+            -0.009301341138780117,
+            0.018416810780763626,
+            -0.00020680672605521977,
+            -0.010687636211514473,
+            -0.017828216776251793,
+            0.00035189816844649613,
+            -0.011338187381625175,
+            -0.027973726391792297,
+            -0.02850036323070526,
+            -0.0026719090528786182,
+            0.013491204008460045,
+            -0.018277406692504883,
+            -0.014993669465184212,
+            0.0031249718740582466,
+            0.021592123433947563,
+            -0.009781510569155216,
+            0.008302279748022556,
+            0.02241305634379387,
+            -0.008255811408162117,
+            -0.005514200776815414,
+            -0.000941460661124438,
+            -0.009130958467721939,
+            0.004267309792339802,
+            0.006300284061580896,
+            0.043339136987924576,
+            -0.032403673976659775,
+            -0.0018238683696836233,
+            0.014815541915595531,
+            0.0034153966698795557,
+            0.0025905899237841368,
+            0.002503462601453066,
+            -0.0007841471815481782,
+            0.01344473659992218,
+            0.01178737822920084,
+            0.015466093085706234,
+            0.00100099784322083,
+            -0.0033011629711836576,
+            0.006408709101378918,
+            -0.021173911169171333,
+            0.015721667557954788,
+            0.027958236634731293,
+            -0.0033999073784798384,
+            0.005525817628949881,
+            -0.013731288723647594,
+            -0.008759214542806149,
+            0.013134949840605259,
+            0.0026641644071787596,
+            -0.005068882368505001,
+            -0.009394276887178421,
+            0.017967619001865387,
+            0.019082851707935333,
+            0.006668155547231436,
+            -0.037453193217515945,
+            -0.02898053266108036,
+            -0.009959637187421322,
+            0.020569827407598495,
+            0.014916222542524338,
+            0.0010339126456528902,
+            -0.010540487244725227,
+            -0.009595638141036034,
+            -0.011903548613190651,
+            0.010989678092300892,
+            0.02721474878489971,
+            -0.032372694462537766,
+            0.0008214184199459851,
+            -0.029383255168795586,
+            0.03068435750901699,
+            0.0020000594668090343,
+            0.011446612887084484,
+            -0.0008185141487047076,
+            -0.015140817500650883,
+            -0.029181893914937973,
+            0.013824224472045898,
+            -0.004395096562802792,
+            0.001024231780320406,
+            -0.0031753121875226498,
+            0.0201051477342844,
+            0.012887120246887207,
+            -0.0014472840121015906,
+            0.01968693546950817,
+            0.008178364485502243,
+            0.016480645164847374,
+            -0.010153254494071007,
+            -0.04135650396347046,
+            0.019965743646025658,
+            0.007562664337456226,
+            0.014219203032553196,
+            0.004739734344184399,
+            0.004588713403791189,
+            -0.020771188661456108,
+            0.002724185585975647,
+            0.004333139397203922,
+            0.014405074529349804,
+            0.011028401553630829,
+            0.0025615475606173277,
+            0.011678952723741531,
+            -0.02228914201259613,
+            0.01929970271885395,
+            -0.01631026156246662,
+            0.01006806269288063,
+            0.004445436876267195,
+            -0.026873983442783356,
+            0.007547175046056509,
+            0.007822110317647457,
+            -0.0019187404541298747,
+            -0.017549408599734306,
+            0.02160761132836342,
+            -0.0034831624943763018,
+            -0.03509881719946861,
+            0.0360281765460968,
+            -0.0043447562493383884,
+            0.019346170127391815,
+            0.03209388628602028,
+            -0.01771979033946991,
+            -0.01993476413190365,
+            0.00906900130212307,
+            0.010408828034996986,
+            -0.002873270306736231,
+            0.017859194427728653,
+            -0.002919738180935383,
+            -0.02232012152671814,
+            0.017766257748007774,
+            -0.028918573632836342,
+            -0.014234691858291626,
+            0.020337486639618874,
+            -0.03624502569437027,
+            -0.0021026763133704662,
+            0.03175312280654907,
+            -0.024070415645837784,
+            0.005734923295676708,
+            0.005196669604629278,
+            -0.0019351978553459048,
+            0.0012236569309607148,
+            -0.006451305001974106,
+            0.03801080957055092,
+            -0.010865762829780579,
+            0.0201051477342844,
+            0.01929970271885395,
+            -0.023388884961605072,
+            0.01977987214922905,
+            -0.001109423115849495,
+            0.01675945147871971,
+            -0.003940097521990538,
+            0.03466511517763138,
+            -0.006017603911459446,
+            -0.011330443434417248,
+            0.006431943271309137,
+            0.008488151244819164,
+            -0.009959637187421322,
+            0.008681767620146275,
+            -0.018122512847185135,
+            0.027152791619300842,
+            0.02091059274971485,
+            -0.0002691270783543587,
+            0.013800990767776966,
+            0.0004421719058882445,
+            -0.020833145827054977,
+            0.00789568480104208,
+            -0.027121812105178833,
+            -0.009967382065951824,
+            0.01646515540778637,
+            -0.005762029904872179,
+            0.007380664348602295,
+            -0.02064727433025837,
+            0.028004704043269157,
+            0.0077524082735180855,
+            -0.013591884635388851,
+            -0.0008824076503515244,
+            0.023760627955198288,
+            -0.005432881414890289,
+            0.009355553425848484,
+            0.02070923149585724,
+            0.015713922679424286,
+            0.013615119270980358,
+            0.016263794153928757,
+            0.03351890295743942,
+            0.005328328814357519,
+            0.005568413063883781,
+            0.003519949736073613,
+            0.016991792246699333,
+            -0.008557853288948536,
+            0.018029578030109406,
+            0.0008093173382803798,
+            -0.013576395809650421,
+            0.021499186754226685,
+            -0.0018684001406654716,
+            -0.013622863218188286,
+            -0.0040891822427511215,
+            -0.004337011836469173,
+            0.03271345794200897,
+            0.022041313350200653,
+            -0.004468671046197414,
+            -0.024442158639431,
+            -0.00394784240052104,
+            0.016682006418704987,
+            -0.003167567541822791,
+            0.011136826127767563,
+            0.0007952801533974707,
+            -0.01586107164621353,
+            -0.009595638141036034,
+            0.0063583687879145145,
+            0.010586955584585667,
+            -0.008921852335333824,
+            0.010346870869398117,
+            -0.0228312686085701,
+            -0.004011735785752535,
+            0.00046613195445388556,
+            -0.004844287410378456,
+            0.0053438181057572365,
+            -0.011314953677356243,
+            0.004948840010911226,
+            0.016062432900071144,
+            0.01026167906820774,
+            0.004832670092582703,
+            0.006989559158682823,
+            0.012066186405718327,
+            0.009161937050521374,
+            -0.01627928391098976,
+            0.00428667152300477,
+            -0.005072754807770252,
+            -0.02345084212720394,
+            0.005831731949001551,
+            0.00902253296226263,
+            -0.01122201792895794,
+            0.009386532939970493,
+            -0.010958699509501457,
+            0.03296129032969475,
+            0.01030814740806818,
+            -0.037391237914562225,
+            -0.0033379499800503254,
+            -0.0016244432190433145,
+            0.014885243959724903,
+            0.012298526242375374,
+            0.008550108410418034,
+            -0.017518429085612297,
+            -0.0047823297791182995,
+            -0.009835722856223583,
+            0.0012827098835259676,
+            -0.012306271120905876,
+            0.017456471920013428,
+            0.022567950189113617,
+            -0.036337960511446,
+            0.01968693546950817,
+            -0.010432061739265919,
+            -0.023652203381061554,
+            0.020476890727877617,
+            0.011500826105475426,
+            0.01942361705005169,
+            -0.009402021765708923,
+            0.018896980211138725,
+            -0.0001331114035565406,
+            0.025510922074317932,
+            -0.004228586331009865,
+            -0.022087780758738518,
+            -0.012499887496232986,
+            -0.03209388628602028,
+            -0.0051811798475682735,
+            0.006819176487624645,
+            -0.00045524101005867124,
+            0.013111716136336327,
+            0.014226946979761124,
+            -0.025712283328175545,
+            0.007295473478734493,
+            0.032589543610811234,
+            0.014637414366006851,
+            0.023311438038945198,
+            -0.011276230216026306,
+            -0.0010794125264510512,
+            -0.0026060794480144978,
+            -0.013770012184977531,
+            -0.03212486580014229,
+            -0.0017493258928880095,
+            -0.036585792899131775,
+            0.005680711008608341,
+            -0.016387708485126495,
+            -0.013096226379275322,
+            0.023264970630407333,
+            -0.034138478338718414,
+            0.0014608372002840042,
+            0.013708055019378662,
+            0.010540487244725227,
+            0.012391461990773678,
+            -0.00162637943867594,
+            0.015032391995191574,
+            -0.029243851080536842,
+            -0.010284913703799248,
+            -0.004491904750466347,
+            -0.01981084980070591,
+            -0.007338068913668394,
+            -0.018060555681586266,
+            -0.013127204962074757,
+            -0.003566417610272765,
+            -0.010060317814350128,
+            0.002247888594865799,
+            0.0032062907703220844,
+            -0.014714861288666725,
+            0.03206290677189827,
+            0.003157886676490307,
+            0.018958937376737595,
+            0.0033418224193155766,
+            0.004449309315532446,
+            0.018788553774356842,
+            -0.0026331855915486813,
+            -0.0058278595097362995,
+            0.030963165685534477,
+            -0.00032672801171429455,
+            0.008759214542806149,
+            0.019082851707935333,
+            0.004724245052784681,
+            0.0038529701996594667,
+            -0.0260530486702919,
+            0.012887120246887207,
+            -0.027447087690234184,
+            0.005080499220639467,
+            -0.0077175572514534,
+            -0.024302754551172256,
+            -0.028128618374466896,
+            -0.006571347359567881,
+            0.012058441527187824,
+            -0.02898053266108036,
+            -0.00613377382978797,
+            0.002315654419362545,
+            -0.00400399137288332,
+            -0.002673845272511244,
+            0.007241260726004839,
+            0.029150914400815964,
+            -0.033209118992090225,
+            0.008751469664275646,
+            0.005595519673079252,
+            -0.0005053392960689962,
+            0.013297587633132935,
+            0.008038961328566074,
+            0.003343758638948202,
+            0.020848635584115982,
+            0.006637176964432001,
+            0.002458930714055896,
+            -0.0025576751213520765,
+            -0.014242436736822128,
+            0.004000118933618069,
+            0.02154565416276455,
+            0.007562664337456226,
+            0.018385831266641617,
+            -0.008914108388125896,
+            0.002621568739414215,
+            -0.01455222349613905,
+            0.005552923772484064,
+            -0.002867461647838354,
+            0.004309905227273703,
+            -0.012236569076776505,
+            0.008093173615634441,
+            0.0028771425131708384,
+            -0.0043447562493383884,
+            0.018060555681586266,
+            -0.002319526858627796,
+            0.016480645164847374,
+            0.004015608225017786,
+            0.015620986931025982,
+            -0.0035412474535405636,
+            0.01178737822920084,
+            0.0005731051205657423,
+            -0.017905661836266518,
+            -0.017022771760821342,
+            -0.039249956607818604,
+            0.007702067960053682,
+            0.03531566634774208,
+            -0.0009564659558236599,
+            -0.03444826602935791,
+            0.010145509615540504,
+            -0.011942271143198013,
+            -0.009975126944482327,
+            -0.022862248122692108,
+            0.013862947933375835,
+            0.012430185452103615,
+            0.007415515370666981,
+            0.012252057902514935,
+            -0.006997303571552038,
+            0.01026167906820774,
+            -0.005216030869632959,
+            0.008356492035090923,
+            -0.0010823167394846678,
+            -0.037422213703393936,
+            0.012344993650913239,
+            0.02216522768139839,
+            -0.002635121811181307,
+            0.0016089539276435971,
+            -0.006145390681922436,
+            0.022103270515799522,
+            -0.013971373438835144,
+            -0.00650551775470376,
+            -0.00046443779137916863,
+            0.00797700323164463,
+            -0.01795213110744953,
+            0.004778457339853048,
+            0.0755879208445549,
+            -0.007775642443448305,
+            0.006122156511992216,
+            0.009270362555980682,
+            -0.015148562379181385,
+            -0.01637221872806549,
+            -0.004561606794595718,
+            -0.020260039716959,
+            0.008844406343996525,
+            -0.0018819533288478851,
+            -0.0110981035977602,
+            0.012468908913433552,
+            0.0053050946444272995,
+            0.002031038049608469,
+            -0.008379725739359856,
+            0.010811550542712212,
+            0.009525936096906662,
+            0.010432061739265919,
+            0.02360573597252369,
+            0.011477591469883919,
+            -0.00858883187174797,
+            -0.011795123107731342,
+            0.01129171997308731,
+            0.0028577810153365135,
+            0.016697494313120842,
+            0.007907302118837833,
+            0.0010648912284523249,
+            0.019501063972711563,
+            -0.002309845993295312,
+            0.02180897258222103,
+            -2.2689444449497387e-05,
+            -0.009301341138780117,
+            -0.004236331209540367,
+            -0.015349923633038998,
+            0.024271776899695396,
+            -0.0020155487582087517,
+            0.0015450604259967804,
+            -0.020213572308421135,
+            0.0013979118084535003,
+            0.013243375346064568,
+            0.011957760900259018,
+            -0.0020407189149409533,
+            -0.0051347119733691216,
+            0.0007154133054427803,
+            -0.016356728971004486,
+            0.032806396484375,
+            0.007578153628855944,
+            0.0007681738352403045,
+            0.0008838597568683326,
+            0.0009288755827583373,
+            -0.0032972905319184065,
+            0.0024608669336885214,
+            -0.004766840487718582,
+            0.009549170732498169,
+            -0.010943209752440453,
+            -0.010796061716973782,
+            0.010075807571411133,
+            -0.02402394637465477,
+            0.003566417610272765,
+            -0.00558777479454875,
+            -0.008062195032835007,
+            0.0076207490637898445,
+            0.00015537731815129519,
+            -0.007392281666398048,
+            -0.0030939930584281683,
+            0.0023408245760947466,
+            0.013429246842861176,
+            0.004406713880598545,
+            0.010850274004042149,
+            0.012956822291016579,
+            0.005359307397156954,
+            -0.006629432085901499,
+            0.015512561425566673,
+            -0.006946963258087635,
+            -0.004228586331009865,
+            -0.0012468908680602908,
+            -0.019996721297502518,
+            -0.0009419447160325944,
+            -0.004708755761384964,
+            0.0015092412941157818,
+            0.0001705036120256409,
+            -8.749050175538287e-05,
+            -0.0018490385264158249,
+            -0.011988739483058453,
+            -0.001831613015383482,
+            -0.011663463898003101,
+            -0.014498010277748108,
+            0.010253935120999813,
+            -0.0041201612912118435,
+            -0.022939693182706833,
+            -0.015349923633038998,
+            -0.020662762224674225,
+            0.02145271934568882,
+            -0.004387352149933577,
+            -0.005800753366202116,
+            0.006733985152095556,
+            -0.010215211659669876,
+            -0.015706177800893784,
+            -0.008069939911365509,
+            -0.0205853171646595,
+            0.010703125037252903,
+            0.003214035416021943,
+            0.0006791101768612862,
+            0.00882891658693552,
+            0.014397329650819302,
+            -0.01589204929769039,
+            0.0111755495890975,
+            0.012182355858385563,
+            0.011361422017216682,
+            -0.006854027509689331,
+            0.0008911203476600349,
+            -0.014103032648563385,
+            0.02596011385321617,
+            0.021623101085424423,
+            0.005169562995433807,
+            -0.04132552444934845,
+            -0.005862710531800985,
+            -0.013344055972993374,
+            -0.010524998418986797,
+            -0.012120398692786694,
+            0.011771888472139835,
+            -0.0038820127956569195,
+            0.0013117524795234203,
+            0.006288666743785143,
+            0.0015063370810821652,
+            0.004367990419268608,
+            -0.0022382077295333147,
+            -0.007709812838584185,
+            -0.03212486580014229,
+            -0.02817508764564991,
+            -0.00863530021160841,
+            0.020461400970816612,
+            0.002838419284671545,
+            -0.0009453330421820283,
+            -0.0036380558740347624,
+            0.0027048238553106785,
+            -0.005882072262465954,
+            -0.005018542055040598,
+            -0.014753584749996662,
+            -0.012739972211420536,
+            0.0101300198584795,
+            0.00802347157150507,
+            0.014195968396961689,
+            -0.011950016021728516,
+            0.0028190575540065765,
+            0.018788553774356842,
+            0.007690451107919216,
+            -0.023357905447483063,
+            0.01135367713868618,
+            -0.010912231169641018,
+            -0.009270362555980682,
+            -0.009433000348508358,
+            -0.009471723809838295,
+            0.0016980175860226154,
+            -0.008317768573760986,
+            0.003945906180888414,
+            -0.03615209087729454,
+            -0.011330443434417248,
+            -0.0013901671627536416,
+            -0.010346870869398117,
+            0.0009864765452221036,
+            0.02524760365486145,
+            0.006815304048359394,
+            0.01432762760668993,
+            -0.0006733016925863922,
+            0.017502939328551292,
+            -0.0018984107300639153,
+            0.018726596608757973,
+            0.012763205915689468,
+            -0.00797700323164463,
+            0.011717676185071468,
+            0.014738094992935658,
+            -0.0033863543067127466,
+            0.015620986931025982,
+            -0.010168743319809437,
+            0.005285732913762331,
+            -0.009556914679706097,
+            -0.003214035416021943,
+            0.002495717955753207,
+            0.030901208519935608,
+            -0.003618694143369794,
+            0.00858883187174797,
+            0.01098193321377039,
+            -0.015241498127579689,
+            0.0026951429899781942,
+            0.014962689951062202,
+            0.01141563430428505,
+            0.007760153152048588,
+            0.0024434414226561785,
+            -0.024225307628512383,
+            0.004193735308945179,
+            0.005037903785705566,
+            0.0031520782504230738,
+            -0.014366351068019867,
+            0.0021859314292669296,
+            0.015675200149416924,
+            0.008550108410418034,
+            -0.025325050577521324,
+            0.016418687999248505,
+            0.0009138703462667763,
+            -0.003909118939191103,
+            0.029119934886693954,
+            -0.011268485337495804,
+            -0.005723306443542242,
+            0.009463978931307793,
+            0.0009980935137718916,
+            0.0030029932968318462,
+            0.008937342092394829,
+            0.0228312686085701,
+            -0.003988501615822315,
+            -0.032341714948415756,
+            -0.022459525614976883,
+            0.015706177800893784,
+            -0.0047823297791182995,
+            -0.022939693182706833,
+            -0.026703599840402603,
+            -0.003113354789093137,
+            -0.02228914201259613,
+            -0.0022323993034660816,
+            -0.020058678463101387,
+            -0.014056564308702946,
+            0.001268188701942563,
+            0.029987338930368423,
+            0.016201836988329887,
+            0.008224832825362682,
+            -0.011051635257899761,
+            -0.029956359416246414,
+            -0.024271776899695396,
+            0.01810702309012413,
+            -0.007055388763546944,
+            -0.005541306920349598,
+            0.00591692328453064,
+            -0.009951893240213394,
+            -0.008480406366288662,
+            0.011802867986261845,
+            0.027137301862239838,
+            -0.002747419523075223,
+            -0.029987338930368423,
+            -0.02688947319984436,
+            -0.014335372485220432,
+            -0.01840132102370262,
+            -0.017704300582408905,
+            -0.008627555333077908,
+            0.0045035220682621,
+            0.00030300996149890125,
+            0.016186347231268883,
+            0.014962689951062202,
+            -0.000480653194244951,
+            0.0028597170021384954,
+            -2.7711375878425315e-05,
+            -0.001763847190886736,
+            0.006888878531754017,
+            -0.004933350719511509,
+            0.026997897773981094,
+            0.014614180661737919,
+            -0.006594581063836813,
+            0.018711106851696968,
+            0.028314489871263504,
+            0.020430423319339752,
+            0.003157886676490307,
+            -0.010695381090044975,
+            -0.014606435783207417,
+            0.009727297350764275,
+            -0.013421501964330673,
+            0.023264970630407333,
+            0.022552460432052612,
+            -0.01480779703706503,
+            -0.014885243959724903,
+            -0.017905661836266518,
+            0.015210519544780254,
+            -0.007562664337456226,
+            -0.008224832825362682,
+            0.012368228286504745,
+            0.022335609421133995,
+            -0.007024410180747509,
+            0.002371803391724825,
+            -0.0010232636705040932,
+            -0.021731525659561157,
+            -0.006122156511992216,
+            -0.004596457816660404,
+            -0.008666278794407845,
+            0.008681767620146275,
+            0.008131897076964378,
+            -0.001928421319462359,
+            -0.006319645792245865,
+            0.005471604876220226,
+            -0.013367289677262306,
+            0.01301103550940752,
+            -0.00789568480104208,
+            0.03379771113395691,
+            -0.009138703346252441,
+            -0.0006733016925863922,
+            -0.022598929703235626,
+            -0.011446612887084484,
+            0.00845717266201973,
+            0.0017967619933187962,
+            0.01682141050696373,
+            -0.0011626676423475146,
+            -0.005688455421477556,
+            0.009913169778883457,
+            -0.0010687635513022542,
+            -0.001450188341550529,
+            -0.006892750971019268,
+            -0.010401083156466484,
+            0.0002884887217078358,
+            0.013258864171802998,
+            -0.006025348324328661,
+            0.0018054747488349676,
+            0.006095050368458033,
+            -0.0175339188426733,
+            0.029027000069618225,
+            0.023884542286396027,
+            0.017255110666155815,
+            0.014660648070275784,
+            0.0006394187803380191,
+            0.011647974140942097,
+            -0.003572226269170642,
+            0.0038820127956569195,
+            -0.020337486639618874,
+            0.021080974489450455,
+            -0.010811550542712212,
+            0.01846327818930149,
+            0.011655719019472599,
+            -0.015233753249049187,
+            -0.005022414494305849,
+            0.021406250074505806,
+            0.009967382065951824,
+            0.00889087375253439,
+            0.0017541663255542517,
+            -0.02470547705888748,
+            0.017766257748007774,
+            0.010099041275680065,
+            -0.020399443805217743,
+            0.019547531381249428,
+            -0.009471723809838295,
+            0.01233724970370531,
+            -0.0005300253978930414,
+            0.004418330732733011,
+            0.0058743273839354515,
+            0.021499186754226685,
+            0.011570527218282223,
+            -0.03884723410010338,
+            -0.013615119270980358,
+            -0.0017357728211209178,
+            0.006857899948954582,
+            -0.0036303112283349037,
+            -0.015233753249049187,
+            -0.041635312139987946,
+            0.01604694314301014,
+            0.009998360648751259,
+            -0.01643417589366436,
+            0.004987563472241163,
+            -0.002427952131256461,
+            -0.0042053526267409325,
+            -0.008170620538294315,
+            0.003186929039657116,
+            -0.004170501604676247,
+            0.02324948087334633,
+            0.0018984107300639153,
+            -0.006946963258087635,
+            0.013258864171802998,
+            0.01344473659992218,
+            -0.021948376670479774,
+            0.0009874446550384164,
+            0.0021375271026045084,
+            0.000636998622212559,
+            0.023683181032538414,
+            -0.014707116410136223,
+            -0.0017657832941040397,
+            -0.01685238815844059,
+            -0.0013069120468571782,
+            -0.021592123433947563,
+            -0.001188805908896029,
+            -0.015187285840511322,
+            -0.008511385880410671,
+            0.025030752643942833,
+            -0.011005166918039322,
+            0.010997422970831394,
+            -0.006687517277896404,
+            0.006943091284483671,
+            -0.006904367823153734,
+            -0.01936165988445282,
+            -0.0008267428493127227,
+            0.009975126944482327,
+            -0.000512115890160203,
+            0.002941036131232977,
+            8.482827252009884e-05,
+            0.002464739140123129,
+            0.008867640048265457,
+            0.006234454456716776,
+            -0.0075278133153915405,
+            0.02489134855568409,
+            -0.012073930352926254,
+            0.0016399325104430318,
+            0.011911292560398579,
+            0.0004743606550619006,
+            -0.010424317792057991,
+            0.025588368996977806,
+            -0.02139076218008995,
+            -0.00800798274576664,
+            -0.00014799568452872336,
+            -0.0016544538084417582,
+            -0.006513262167572975,
+            0.006718495860695839,
+            0.02946070022881031,
+            0.0062654330395162106,
+            0.010447551496326923,
+            -0.0281441081315279,
+            -0.011493081226944923,
+            0.012538610957562923,
+            -0.022273652255535126,
+            -0.0007739823195151985,
+            -0.0004322490422055125,
+            -0.00619960343465209,
+            0.026378324255347252,
+            -0.02022906206548214,
+            -0.01093546487390995,
+            -0.018664639443159103,
+            0.005405775271356106,
+            -0.011121337302029133,
+            0.008294534869492054,
+            -0.0009777637897059321,
+            0.006540368776768446,
+            0.016805920749902725,
+            -0.019563021138310432,
+            0.016031453385949135,
+            0.005119222681969404,
+            -0.016217326745390892,
+            0.010222956538200378,
+            -0.017580386251211166,
+            -0.0008199662552215159,
+            0.020275529474020004,
+            -0.00400399137288332,
+            0.008751469664275646,
+            -0.003363120136782527,
+            0.005208286456763744,
+            -0.006513262167572975,
+            0.0001492057926952839,
+            -0.007249005138874054,
+            0.014428308233618736,
+            -0.016805920749902725,
+            -0.017828216776251793,
+            -0.008821171708405018,
+            0.0016709112096577883,
+            0.021297825500369072,
+            0.011136826127767563,
+            -0.010044828988611698,
+            -0.011059380136430264,
+            -0.011694442480802536,
+            0.041604332625865936,
+            0.005762029904872179,
+            0.027230238541960716,
+            -0.02261441759765148,
+            0.00545998802408576,
+            0.025170156732201576,
+            -0.014242436736822128,
+            0.0012788375606760383,
+            -0.0022788674104958773,
+            0.0053050946444272995,
+            0.011562783271074295,
+            -0.01819995976984501,
+            0.0049139889888465405,
+            -0.00019216447253711522,
+            -0.00782985519617796,
+            -0.000656844291370362,
+            0.003442503046244383,
+            0.00020196630794089288,
+            0.016325751319527626,
+            -0.004027225077152252,
+            -0.016511622816324234,
+            -0.004429947584867477,
+            -0.0003216455806978047,
+            -0.027896279469132423,
+            0.013514438644051552,
+            0.016589069738984108,
+            0.01566745527088642,
+            0.027989214286208153,
+            -0.014513500034809113,
+            0.016031453385949135,
+            0.009533680975437164,
+            0.013901671394705772,
+            -0.010610189288854599,
+            0.008952830918133259,
+            -0.01595400646328926,
+            0.007446494419127703,
+            -0.03454120084643364,
+            -0.007887939922511578,
+            -0.00802347157150507,
+            0.01984182931482792,
+            0.028763681650161743,
+            0.002923610620200634,
+            0.006493900436908007,
+            0.000196883876924403,
+            0.0024202074855566025,
+            -0.01714668609201908,
+            -0.020182594656944275,
+            -0.0003337466041557491,
+            0.0029332914855331182,
+            0.0006950835813768208,
+            -0.004495777189731598,
+            0.00472037261351943,
+            -0.012391461990773678,
+            0.0026835259050130844,
+            -0.013134949840605259,
+            0.005971135571599007,
+            -0.0059982421807944775,
+            0.009603383019566536,
+            0.0019429425010457635,
+            0.020864123478531837,
+            0.02300165221095085,
+            0.004100799560546875,
+            0.0046971384435892105,
+            -0.024333734065294266,
+            0.00920066051185131,
+            -0.013847459107637405,
+            0.012329504825174809,
+            0.019408127292990685,
+            0.008248066529631615,
+            0.006571347359567881,
+            0.01233724970370531,
+            0.012918098829686642,
+            -0.0031772484071552753,
+            0.024287264794111252,
+            0.005014669615775347,
+            -0.0009070937521755695,
+            0.002052335999906063,
+            -0.014954946003854275,
+            -0.0070979841984808445,
+            -0.0015286030247807503,
+            0.0018848575418815017,
+            -0.004731989465653896,
+            0.01906736195087433,
+            0.015256987884640694,
+            -0.01050176378339529,
+            0.017874684184789658,
+            -0.03937387093901634,
+            0.017379024997353554,
+            -0.0027377386577427387,
+            -0.0060911779291927814,
+            -0.00545998802408576,
+            0.00665653869509697,
+            0.004515138920396566,
+            0.003669034456834197,
+            0.03773200139403343,
+            -0.010408828034996986,
+            0.003963331691920757,
+            -0.005847221240401268,
+            0.007121218368411064,
+            -0.020864123478531837,
+            0.008302279748022556,
+            0.005336073227226734,
+            0.02160761132836342,
+            -0.0056497324258089066,
+            -0.0014482521219179034,
+            0.02328045852482319,
+            0.008859895169734955,
+            0.03469609469175339,
+            -0.01566745527088642,
+            -0.004418330732733011,
+            0.01048627495765686,
+            0.0207402091473341,
+            -0.00580462533980608,
+            -0.008976065553724766,
+            0.007903429679572582,
+            0.0035586729645729065,
+            0.021824462339282036,
+            -0.00515407370403409,
+            -0.0006650729919783771,
+            0.0054367538541555405,
+            -0.0060911779291927814,
+            0.009735042229294777,
+            0.011252996511757374,
+            -0.009076746180653572,
+            0.007539430167526007,
+            0.021530164405703545,
+            -0.009975126944482327,
+            -0.008658533915877342,
+            -0.009649851359426975,
+            0.0010658593382686377,
+            -0.03068435750901699,
+            0.001649613375775516,
+            0.005359307397156954,
+            0.015187285840511322,
+            -0.0004603234410751611,
+            -0.009014789015054703,
+            0.021019017323851585,
+            0.017255110666155815,
+            -0.005936284549534321,
+            -0.0028132491279393435,
+            0.0023795480374246836,
+            -0.0017154429806396365,
+            -0.006160879973322153,
+            -0.0006248975405469537,
+            -0.002656419761478901,
+            0.014730350114405155,
+            -0.0015469965292140841,
+            -0.006617815233767033,
+            -0.00756653631106019,
+            0.0033302053343504667,
+            0.010834784246981144,
+            -0.005661349277943373,
+            0.013057502917945385,
+            0.016805920749902725,
+            -0.00031704717548564076,
+            0.019671445712447166,
+            0.01037010457366705,
+            -0.010447551496326923,
+            0.015938518568873405,
+            -0.007984748110175133,
+            0.006563602481037378,
+            0.0164031982421875,
+            -0.01277095079421997,
+            -0.01246116403490305,
+            0.007210282143205404,
+            0.00665653869509697,
+            0.028918573632836342,
+            0.014792308211326599,
+            -0.02109646424651146,
+            0.014908477663993835,
+            0.015768134966492653,
+            -0.013018779456615448,
+            0.003537375247105956,
+            -0.01784370467066765,
+            0.020260039716959,
+            0.019051872193813324,
+            -0.015535795129835606,
+            -0.017224133014678955,
+            -0.014947201125323772,
+            -0.005092116538435221,
+            0.006331262644380331,
+            0.0033263331279158592,
+            -0.00617636926472187,
+            -0.027989214286208153,
+            -0.0049139889888465405,
+            -0.018633661791682243,
+            0.013870692811906338,
+            -0.008674023672938347,
+            0.0007715621031820774,
+            -0.004511266481131315,
+            0.007984748110175133,
+            -0.0033882902935147285,
+            0.032465629279613495,
+            -0.0051811798475682735,
+            -0.02832997962832451,
+            0.004398969002068043,
+            0.016356728971004486,
+            0.012623801827430725,
+            -0.029243851080536842,
+            0.010463040322065353,
+            0.004805563949048519,
+            -0.012530866079032421,
+            0.005990497302263975,
+            -0.02258343994617462,
+            -0.007632365915924311,
+            -0.0009395244996994734,
+            -0.016743963584303856,
+            -0.027121812105178833,
+            -0.008433938957750797,
+            -0.006993431597948074,
+            -0.005401902832090855,
+            -0.009525936096906662,
+            -0.008720491081476212,
+            -0.011477591469883919,
+            0.02303262986242771,
+            -0.007899557240307331,
+            -0.001866463921032846,
+            -0.0014637414133176208,
+            0.00930908601731062,
+            -0.023776117712259293,
+            -0.015605497173964977,
+            0.006389347370713949,
+            -0.002594462363049388,
+            0.022815778851509094,
+            -0.012747716158628464,
+            0.005940156988799572,
+            -0.009316830895841122,
+            0.009851212613284588,
+            0.015241498127579689,
+            0.004956584889441729,
+            0.017440982162952423,
+            -0.009889936074614525,
+            0.0021026763133704662,
+            -0.008627555333077908,
+            -0.00018986526993103325,
+            -1.027075541060185e-05,
+            0.014428308233618736,
+            0.007624621503055096,
+            0.014141756109893322,
+            0.017487451434135437,
+            -0.013553161174058914,
+            0.005003052763640881,
+            -0.0003274540649726987,
+            -0.017255110666155815,
+            0.0027725896798074245,
+            -0.007984748110175133,
+            -0.0008896682411432266,
+            0.001758038648404181,
+            6.395397940650582e-05,
+            -0.02112744189798832,
+            -0.0009124181815423071,
+            0.0018809852190315723,
+            0.0010222955606877804,
+            -0.025201136246323586,
+            0.00889087375253439,
+            -0.00889087375253439,
+            -0.0014918158994987607,
+            -0.019113829359412193,
+            0.019795360043644905,
+            0.026300877332687378,
+            -0.004619691986590624,
+            0.00036206303047947586,
+            -0.013537672348320484,
+            -0.02049238048493862,
+            0.0001489637652412057,
+            -0.011314953677356243,
+            -0.011345932260155678,
+            -0.0022169100120663643,
+            -0.003214035416021943,
+            0.008805682882666588,
+            0.0004189379105810076,
+            -0.009433000348508358,
+            -0.004956584889441729,
+            -0.011764144524931908,
+            -0.017549408599734306,
+            -0.010401083156466484,
+            0.01933068037033081,
+            -0.0018016024259850383,
+            0.0020794421434402466,
+            -0.0032508226577192545,
+            0.0005271211848594248,
+            0.011748654767870903,
+            0.013475715182721615,
+            0.013344055972993374,
+            -0.023915521800518036,
+            -0.017890173941850662,
+            -0.013320821337401867,
+            0.0029487807769328356,
+            -0.007187047973275185,
+            0.004844287410378456,
+            0.026362836360931396,
+            -0.0009695350890979171,
+            -0.01091997604817152,
+            0.002327271504327655,
+            0.019051872193813324,
+            -0.031272951513528824,
+            -0.013390523381531239,
+            -0.025077221915125847,
+            0.02225816436111927,
+            -0.013351799920201302,
+            0.011516314931213856,
+            0.010191977955400944,
+            -0.025944624096155167,
+            0.006768836174160242,
+            -0.007686578668653965,
+            0.026037558913230896,
+            0.01634124107658863,
+            0.0005934348446317017,
+            -0.00944074522703886,
+            0.013622863218188286,
+            -0.0008722427301108837,
+            -0.013576395809650421,
+            0.002201420720666647,
+            0.014645159244537354,
+            -0.014691627584397793,
+            -0.009510447271168232,
+            -0.004507394507527351,
+            0.004313777666538954,
+            0.012763205915689468,
+            -0.0030339721124619246,
+            -0.004031097516417503,
+            -0.011624740436673164,
+            0.013344055972993374,
+            -0.003930416889488697,
+            0.012383717112243176,
+            0.007380664348602295,
+            -0.014056564308702946,
+            -0.019532041624188423,
+            0.033333033323287964,
+            0.00400399137288332,
+            0.00338054564781487,
+            -0.010997422970831394,
+            0.02740062028169632,
+            -0.009897680021822453,
+            0.010571465827524662,
+            -0.006408709101378918,
+            0.0031985461246222258,
+            0.012321759946644306,
+            -0.012104909867048264,
+            -0.01981084980070591,
+            0.021468207240104675,
+            -0.00364192808046937,
+            -0.0010193913476541638,
+            0.008310024626553059,
+            -0.020182594656944275,
+            -0.00663330452516675,
+            0.002180122770369053,
+            0.009541425853967667,
+            -0.011818356812000275,
+            0.006369986105710268,
+            0.017905661836266518,
+            0.003912991378456354,
+            -0.0007672057254239917,
+            0.019996721297502518,
+            -0.01672847382724285,
+            0.011717676185071468,
+            -0.010850274004042149,
+            0.0016825281782075763,
+            0.0006065039779059589,
+            -0.0048597767017781734,
+            0.0058743273839354515,
+            -0.014536733739078045,
+            0.011988739483058453,
+            -6.244135147426277e-05,
+            -0.013979118317365646,
+            0.007601387333124876,
+            -0.013359544798731804,
+            -0.016836898401379585,
+            0.041542377322912216,
+            -0.005301222205162048,
+            -0.0038665232714265585,
+            -0.0056962003000080585,
+            0.01350669376552105,
+            0.02823704481124878,
+            -0.005336073227226734,
+            -0.01153180468827486,
+            -0.010780571959912777,
+            -0.016573579981923103,
+            0.015907539054751396,
+            -0.01810702309012413,
+            0.002495717955753207,
+            -0.010532742366194725,
+            -0.007903429679572582,
+            0.004302160814404488,
+            -0.003516077296808362,
+            0.014722606167197227,
+            0.00950270239263773,
+            0.023357905447483063,
+            0.00797700323164463,
+            -0.00027348342700861394,
+            0.010393339209258556,
+            0.011663463898003101,
+            0.003243078012019396,
+            -0.010509508661925793,
+            -0.00011949774489039555,
+            -0.02894955314695835,
+            -0.008449427783489227,
+            -0.004863648675382137,
+            -0.001485039247199893,
+            -0.00428667152300477,
+            -0.01239920686930418,
+            0.006617815233767033,
+            -0.0022052929271012545,
+            0.013909416273236275,
+            -0.009254872798919678,
+            -0.017766257748007774,
+            0.03389064967632294,
+            0.004495777189731598,
+            0.006466794293373823,
+            -0.008286789990961552,
+            -0.03705047070980072,
+            -0.0032566310837864876,
+            -0.008116407319903374,
+            -0.013529927469789982,
+            0.004538373090326786,
+            0.01283290795981884,
+            -0.014831030741333961,
+            -0.028097640722990036,
+            0.0019390701781958342,
+            -0.006683644838631153,
+            0.02978597767651081,
+            -0.007798876613378525,
+            0.00419760774821043,
+            0.0037251831963658333,
+            0.0011558911064639688,
+            0.01239920686930418,
+            -0.00845717266201973,
+            -0.0022536972537636757,
+            0.016387708485126495,
+            -0.010858018882572651,
+            0.011051635257899761,
+            -0.010966443456709385,
+            -0.02554190158843994,
+            -0.01456771232187748,
+            0.021344292908906937,
+            -0.0004823473282158375,
+            -0.00024903935263864696,
+            0.014823286794126034,
+            0.02354377694427967,
+            0.011260741390287876,
+            0.00969631876796484,
+            0.006408709101378918,
+            -0.001035848748870194,
+            0.005975008010864258,
+            0.006095050368458033,
+            -0.0024163350462913513,
+            -0.013398268260061741,
+            0.017564896494150162,
+            0.009773765690624714,
+            -0.01810702309012413,
+            0.0008514289511367679,
+            0.0036612898111343384,
+            0.0168833676725626,
+            0.006935346405953169,
+            0.009316830895841122,
+            -0.0143121387809515,
+            -0.023760627955198288,
+            -0.00014932679187040776,
+            0.002873270306736231,
+            0.015721667557954788,
+            0.010579210706055164,
+            -0.014319883659482002,
+            0.0005963391158729792,
+            0.008712747134268284,
+            0.020213572308421135,
+            -0.024209819734096527,
+            0.019702425226569176,
+            -0.004673904739320278,
+            -0.0021084847394376993,
+            0.007946024648845196,
+            0.011632485315203667,
+            -0.011872569099068642,
+            -0.00650551775470376,
+            0.006249943748116493,
+            0.003157886676490307,
+            0.03172214329242706,
+            -0.0017260919557884336,
+            0.009665340185165405,
+            -0.010997422970831394,
+            -0.0002405686245765537,
+            0.0006728176376782358,
+            -0.0006820144481025636,
+            -0.002884887158870697,
+            -0.011020656675100327,
+            0.0018974426202476025,
+            -0.020306508988142014,
+            0.0024473138619214296,
+            0.00463130883872509,
+            -0.005684582982212305,
+            -0.0032972905319184065,
+            -0.019222255796194077,
+            -0.0022672503255307674,
+            0.003293418325483799,
+            0.014335372485220432,
+            -0.031691163778305054,
+            0.004972074180841446,
+            0.010672146454453468,
+            -0.002309845993295312,
+            -0.010416572913527489,
+            0.011423379182815552,
+            -0.0026060794480144978,
+            0.011988739483058453,
+            -0.0038142469711601734,
+            -0.0004905760288238525,
+            0.025681305676698685,
+            -0.005308967083692551,
+            0.014707116410136223,
+            -0.01026167906820774,
+            -0.002993312431499362,
+            -0.028252532705664635,
+            0.005235392600297928,
+            -0.007403898518532515,
+            -0.0012081675231456757,
+            -0.004402841441333294,
+            -0.01130720879882574,
+            -0.0026990154292434454,
+            0.002122037811204791,
+            0.008000237867236137,
+            0.012910354882478714,
+            -0.0007725302129983902,
+            -0.010075807571411133,
+            0.004832670092582703,
+            0.0075355577282607555,
+            -0.015520306304097176,
+            -0.0070437719114124775,
+            -0.020306508988142014,
+            0.029662061482667923,
+            -0.013731288723647594,
+            -0.0011123273288831115,
+            -0.020864123478531837,
+            0.010160998441278934,
+            -0.0063119009137153625,
+            -0.002782270545139909,
+            0.004708755761384964,
+            -0.015109838917851448,
+            0.01344473659992218,
+            0.008751469664275646,
+            0.01450575515627861,
+            -0.02151467651128769,
+            -0.01762685552239418,
+            -0.0044725434854626656,
+            -0.002735802438110113,
+            -0.010648912750184536,
+            -0.0005334137240424752,
+            0.002013612538576126,
+            0.021251358091831207,
+            -0.00032503384863957763,
+            0.0042053526267409325,
+            -0.003214035416021943,
+            0.017224133014678955,
+            -0.010137764737010002,
+            -0.013057502917945385,
+            0.0024260159116238356,
+            0.005994369741529226,
+            0.016186347231268883,
+            0.03158273920416832,
+            -0.020693741738796234,
+            -0.00272999401204288,
+            0.0032740565948188305,
+            0.010656657628715038,
+            -0.004085310269147158,
+            6.304640555754304e-05,
+            -0.027803342789411545,
+            -0.04464024305343628,
+            -0.002422143705189228,
+            0.013057502917945385,
+            -0.015636475756764412,
+            0.012670270167291164,
+            0.008743725717067719,
+            0.010021595284342766,
+            -0.005196669604629278,
+            0.02620794251561165,
+            -0.004793947096914053,
+            0.014304393902420998,
+            0.011733165942132473,
+            0.011911292560398579,
+            0.0009874446550384164,
+            0.0007459079497493804,
+            -0.018354853615164757,
+            -0.008255811408162117,
+            -0.0015954007394611835,
+            0.01765783317387104,
+            -0.027478067204356194,
+            0.008526874706149101,
+            -0.011516314931213856,
+            0.013839714229106903,
+            0.0003666614356916398,
+            0.005661349277943373,
+            -0.01771979033946991,
+            0.012445674277842045,
+            -0.0009337160154245794,
+            -0.025325050577521324,
+            0.004805563949048519,
+            0.011276230216026306,
+            -0.010362359695136547,
+            0.006877261213958263,
+            0.019950253888964653,
+            0.0023079097736626863,
+            0.008495896123349667,
+            -0.014126266352832317,
+            0.010703125037252903,
+            0.009649851359426975,
+            -0.021080974489450455,
+            -0.01233724970370531,
+            -0.007059261202812195,
+            -0.006335135083645582,
+            -0.0035470561124384403,
+            -0.00013105422840453684,
+            0.010176488198339939,
+            -0.016000475734472275,
+            0.010099041275680065,
+            -0.006559730041772127,
+            0.006400964688509703,
+            0.00944074522703886,
+            0.001115231541916728,
+            -0.013576395809650421,
+            0.012716737575829029,
+            -0.0218709297478199,
+            0.008426194079220295,
+            -0.04287445917725563,
+            -0.005847221240401268,
+            -0.015078860335052013,
+            0.005204414017498493,
+            -0.003798757679760456,
+            -0.004453181754797697,
+            -0.006203475873917341,
+            -0.005061137955635786,
+            0.00920066051185131,
+            0.021747015416622162,
+            0.012135888449847698,
+            -0.016620049253106117,
+            -0.007551047019660473,
+            0.005425137002021074,
+            0.0004854935978073627,
+            -0.013336311094462872,
+            0.009316830895841122,
+            -0.01576039008796215,
+            0.0009259713697247207,
+            0.006339007057249546,
+            -0.01072635967284441,
+            0.013452480547130108,
+            0.004731989465653896,
+            -0.003401843598112464,
+            0.013390523381531239,
+            -0.0050417762249708176,
+            -0.0057891360484063625,
+            -0.007272239308804274,
+            -0.013739033602178097,
+            -0.003818119177594781,
+            -0.003072695340961218,
+            0.012747716158628464,
+            0.001529571134597063,
+            -0.0003243077953811735,
+            -0.00782985519617796,
+            0.009154192171990871,
+            0.002220782218500972,
+            -0.0013146566925570369,
+            0.005773646757006645,
+            -0.02300165221095085,
+            0.0001287550403503701,
+            0.004755223635584116,
+            -0.0009487213101238012,
+            -0.006246071308851242,
+            -0.012151377275586128,
+            -0.011903548613190651,
+            -0.038165703415870667,
+            0.009363298304378986,
+            0.0019197085639461875,
+            -0.0076207490637898445,
+            0.0064822835847735405,
+            0.017255110666155815,
+            0.01456771232187748,
+            0.016743963584303856,
+            -0.020693741738796234,
+            -0.02303262986242771,
+            0.005556796211749315,
+            -0.009402021765708923,
+            0.02261441759765148,
+            0.004046586807817221,
+            -0.005847221240401268,
+            0.0013243375578895211,
+            -0.013258864171802998,
+            0.008929597213864326,
+            0.005723306443542242,
+            -0.001592496526427567,
+            0.005425137002021074,
+            -0.013692565262317657,
+            -0.010904486291110516,
+            0.01048627495765686,
+            0.032496608793735504,
+            -0.0004261985304765403,
+            0.0024105266202241182,
+            0.023295948281884193,
+            0.015466093085706234,
+            0.016356728971004486,
+            -0.021111954003572464,
+            0.011167804710566998,
+            0.010060317814350128,
+            -0.0018645278178155422,
+            -0.014892988838255405,
+            0.00513083953410387,
+            -0.012716737575829029,
+            -0.010664401575922966,
+            0.014993669465184212,
+            0.007853088900446892,
+            0.017735280096530914,
+            -0.02369867078959942,
+            0.00789568480104208,
+            0.00042765066609717906,
+            0.00013710475468542427,
+            0.011524059809744358,
+            0.012956822291016579,
+            0.03370477631688118,
+            0.027152791619300842,
+            0.020662762224674225,
+            -0.009285852313041687,
+            0.014366351068019867,
+            -0.009192915633320808,
+            -0.011090358719229698,
+            -0.019470084458589554,
+            -0.015249243006110191,
+            -0.011136826127767563,
+            0.017998598515987396,
+            0.008790193125605583,
+            -0.009355553425848484,
+            -0.005947901867330074,
+            0.01586107164621353,
+            -0.003909118939191103,
+            -0.004635181277990341,
+            0.01646515540778637,
+            -0.026269899681210518,
+            0.016836898401379585,
+            -0.000745423894841224,
+            -0.014668392948806286,
+            -0.014691627584397793,
+            -0.004182118456810713,
+            -0.012670270167291164,
+            0.0003964299685321748,
+            -0.020817656069993973,
+            0.013181418180465698,
+            0.024597052484750748,
+            -0.0030920570716261864,
+            -0.004100799560546875,
+            -0.011299464851617813,
+            -0.0005111477803438902,
+            0.018385831266641617,
+            0.0034773540683090687,
+            0.009192915633320808,
+            -0.0023872926831245422,
+            0.013924905098974705,
+            0.00524313747882843,
+            0.0027958236169070005,
+            -0.0007565568666905165,
+            -0.016263794153928757,
+            0.012197845615446568,
+            0.0026835259050130844,
+            0.012135888449847698,
+            -0.022242674604058266,
+            0.007938280701637268,
+            -0.011764144524931908,
+            -0.0023698671720921993,
+            -0.02228914201259613,
+            0.0189744271337986,
+            -0.01610890030860901,
+            0.001046497724018991,
+            0.003841353114694357,
+            0.0006224773242138326,
+            0.009580149315297604,
+            0.01610890030860901,
+            -0.023388884961605072,
+            -0.0003460896550677717,
+            -0.014908477663993835,
+            -0.021917399019002914,
+            -0.01607792265713215,
+            0.004728117026388645,
+            -0.019501063972711563,
+            0.005444498732686043,
+            -0.0025557391345500946,
+            -0.015125328674912453,
+            0.018478767946362495,
+            0.014691627584397793,
+            -0.004379607271403074,
+            0.002412462839856744,
+            0.005425137002021074,
+            -0.0014405074762180448,
+            0.01971791312098503,
+            0.00013274837692733854,
+            -0.0010048701660707593,
+            -0.009425255469977856,
+            -0.0168833676725626,
+            -0.0065752193331718445,
+            -0.035005878657102585,
+            0.0260530486702919,
+            -0.004670032300055027,
+            -0.016511622816324234,
+            0.014319883659482002,
+            0.009115469641983509,
+            -0.014629669487476349,
+            -0.007059261202812195,
+            0.016294771805405617,
+            -0.024287264794111252,
+            0.02244403585791588,
+            -0.009611127898097038,
+            -0.008774704299867153,
+            0.00784921646118164,
+            0.05154848098754883,
+            0.009146448224782944,
+            0.003492843359708786,
+            0.011036145500838757,
+            0.005839476361870766,
+            -0.0013165927957743406,
+            -0.023683181032538414,
+            0.005568413063883781,
+            -0.007190920412540436,
+            -0.003601268632337451,
+            0.002536377403885126,
+            -0.004801691509783268,
+            0.031149037182331085,
+            -0.004379607271403074,
+            -0.005289605353027582,
+            0.004484160337597132,
+            -0.02196386642754078,
+            -0.0001332324172835797,
+            0.023203013464808464,
+            -2.3128106477088295e-05,
+            -0.023590246215462685,
+            -0.006571347359567881,
+            -0.015071115456521511,
+            -0.013770012184977531,
+            -0.018989915028214455,
+            0.005231520161032677,
+            -0.012755461037158966,
+            -0.0033940989524126053,
+            0.0016660707769915462,
+            -0.005940156988799572,
+            -0.002582845278084278,
+            0.009603383019566536,
+            0.05362405255436897,
+            0.014103032648563385,
+            -0.006335135083645582,
+            -0.005854965653270483,
+            0.009192915633320808,
+            -0.010524998418986797,
+            -0.019795360043644905,
+            -0.009820234030485153,
+            -0.029507169499993324,
+            -0.012747716158628464,
+            -0.0143121387809515,
+            -0.010191977955400944,
+            -0.010277168825268745,
+            0.00963436160236597,
+            -0.018122512847185135,
+            -0.007628493942320347,
+            -0.020337486639618874,
+            0.002941036131232977,
+            -0.011268485337495804,
+            -0.014714861288666725,
+            -0.008054450154304504,
+            -0.04807887226343155,
+            0.005796880926936865,
+            0.019593998789787292,
+            0.01263929158449173,
+            0.002515079453587532,
+            0.004383479710668325,
+            -0.01993476413190365,
+            -0.030792783945798874,
+            0.013839714229106903,
+            0.011283975094556808,
+            0.011733165942132473,
+            0.03283737599849701,
+            -0.011709931306540966,
+            -0.00813964195549488,
+            -0.004693266469985247,
+            -0.0017251238459721208,
+            0.012066186405718327,
+            0.00969631876796484,
+            -0.004917861428111792,
+            -0.012375972233712673,
+            0.016217326745390892,
+            -0.01981084980070591,
+            -0.023683181032538414,
+            -0.01672847382724285,
+            0.006157007534056902,
+            0.00813964195549488,
+            -0.008054450154304504,
+            0.002973950933665037,
+            0.0056962003000080585,
+            -0.007159941829741001,
+            -0.00022423222253564745,
+            0.004890755284577608,
+            -0.023048119619488716,
+            0.0008490087930113077,
+            -0.002257569460198283,
+            0.0061376458033919334,
+            0.021266845986247063,
+            -0.010168743319809437,
+            -0.0014075926737859845,
+            -0.0053902859799563885,
+            -0.0028984404634684324,
+            -0.006764963734894991,
+            0.02620794251561165,
+            0.009154192171990871,
+            -0.009882191196084023,
+            -0.01011453103274107,
+            -0.01129171997308731,
+            -0.0011278166202828288,
+            -0.017255110666155815,
+            -0.016186347231268883,
+            0.0015566773945465684,
+            -0.016712984070181847,
+            0.013971373438835144,
+            -0.00876695942133665,
+            0.00400399137288332,
+            -0.014110777527093887,
+            0.02019808255136013,
+            0.01929970271885395,
+            -0.009711808525025845,
+            0.023776117712259293,
+            -0.007709812838584185,
+            -0.038382552564144135,
+            -0.00800798274576664,
+            0.01843230053782463,
+            -0.006118284538388252,
+            0.010037084110081196,
+            -0.011314953677356243,
+            0.008209343999624252,
+            0.00015319912927225232,
+            0.0018122512847185135,
+            -0.0003112386621069163,
+            0.027555514127016068,
+            0.015535795129835606,
+            -0.005560668651014566,
+            -0.005494839046150446,
+            0.004255692940205336,
+            0.009541425853967667,
+            0.017580386251211166,
+            0.012151377275586128,
+            -0.021019017323851585,
+            -0.014257925562560558,
+            -0.016263794153928757,
+            0.011012911796569824,
+            0.015125328674912453,
+            -0.0040891822427511215,
+            0.01691434532403946,
+            0.014126266352832317,
+            -0.019098341464996338,
+            0.0019884423818439245,
+            -0.008193854242563248,
+            -0.00158184755127877,
+            -0.03029712475836277,
+            0.011043890379369259,
+            0.0074387495405972,
+            -0.0072218989953398705,
+            0.036616768687963486,
+            0.009858956560492516,
+            0.037577107548713684,
+            0.018680129200220108,
+            0.0008974128868430853,
+            0.009045767597854137,
+            -0.018370341509580612,
+            0.013065247796475887,
+            0.01500141341239214,
+            -0.012724482454359531,
+            -0.005734923295676708,
+            0.0013446672819554806,
+            0.0007551047019660473,
+            0.012631546705961227,
+            0.02004319056868553,
+            -0.01795213110744953,
+            0.009773765690624714,
+            -0.015032391995191574,
+            -0.005250881891697645,
+            0.006555858068168163,
+            -0.013932649977505207,
+            -0.008495896123349667,
+            -0.00963436160236597,
+            0.019408127292990685,
+            -0.012600568123161793,
+            0.016573579981923103,
+            -0.009595638141036034,
+            0.015520306304097176,
+            -0.029724018648266792,
+            -0.003669034456834197,
+            -0.006679772399365902,
+            -0.019593998789787292,
+            -0.012887120246887207,
+            0.009820234030485153,
+            0.010633422993123531,
+            -0.009301341138780117,
+            -0.011562783271074295,
+            0.020368466153740883,
+            -0.004185990896075964,
+            0.013529927469789982,
+            0.021189400926232338,
+            0.00737291993573308,
+            0.024132372811436653,
+            0.0005605200421996415,
+            0.014164989814162254,
+            0.005463860463351011,
+            0.010362359695136547,
+            -0.004677776712924242,
+            0.005161818582564592,
+            0.00932457484304905,
+            0.014203713275492191,
+            0.017766257748007774,
+            0.013196907006204128,
+            -0.00598275288939476,
+            0.007419387809932232,
+            0.02016710489988327,
+            0.0016254113288596272,
+            -0.030312614515423775,
+            -0.009719553403556347,
+            0.0001736498816171661,
+            0.00924712885171175,
+            0.009185170754790306,
+            3.817877222900279e-05,
+            -0.01093546487390995,
+            0.011144571006298065,
+            0.00969631876796484,
+            0.002286612056195736,
+            0.006280922330915928,
+            -0.01456771232187748,
+            -0.020957060158252716,
+            0.006106667220592499,
+            0.01798310875892639,
+            0.013917161151766777,
+            -0.014722606167197227,
+            0.022784801200032234,
+            0.006145390681922436,
+            -0.021406250074505806,
+            -0.01840132102370262,
+            -0.0009559819591231644,
+            0.0004542729293461889,
+            -0.002265314105898142,
+            0.03720536455512047,
+            -0.0011539548868313432,
+            -0.005955646280199289,
+            -0.0054754773154854774,
+            0.02775687538087368,
+            0.02049238048493862,
+            0.009177426807582378
+          ],
+          "embedding": [
+            0.0008215984936325879,
+            -0.019349418284688353,
+            -0.02322660633029795,
+            0.009747820845194852,
+            0.007353771220903641,
+            0.015252908783199839,
+            -0.012176175829980202,
+            0.051898051094966605,
+            -0.0068130353886146165,
+            0.008189212816890595,
+            0.008102287831931905,
+            0.01682801136011614,
+            0.02005557996049711,
+            -0.020779879580497183,
+            0.002332165268329781,
+            0.02337485065616377,
+            0.006074992895540845,
+            0.02576575146535689,
+            0.0007517286429767504,
+            -0.03763909942314134,
+            0.0011204729720314833,
+            0.00041212613589632806,
+            -0.06089176800314795,
+            0.012852035873395388,
+            -0.008901127092058543,
+            0.01702347055484216,
+            0.005803446395466281,
+            0.017889570275475296,
+            -0.05375047382413552,
+            0.021623778774252724,
+            0.014784317465883424,
+            -0.0018881880857885708,
+            0.0004735328554290247,
+            0.012393802850407656,
+            -0.019234442968055952,
+            -0.019566283611638122,
+            0.04407789373296535,
+            -0.0053565873209102995,
+            -0.026240866282480922,
+            0.048107902270426675,
+            -0.0054681314939387235,
+            -0.0009441397948041965,
+            -0.015205079626883548,
+            0.006885981841893229,
+            0.01403724256866746,
+            -0.026086548246792754,
+            0.0034778946051893067,
+            0.00044587533948775473,
+            0.032851328435891634,
+            -0.03882097333255334,
+            -0.011126344604254894,
+            -0.01648943336384904,
+            0.035810456555802166,
+            -0.019384589526165754,
+            -0.019430702768170285,
+            0.014297440272239852,
+            -0.007883453204663151,
+            0.050996900793086844,
+            0.06007149434153715,
+            0.03406832053483379,
+            0.025193439934671852,
+            0.00027785129663604897,
+            -0.008014391056168083,
+            -0.018806730908490114,
+            0.027193788903098558,
+            -0.006195006670328595,
+            -0.020466000344352826,
+            0.0068939884219589185,
+            0.017252276167086602,
+            -0.045073444648704136,
+            -0.049242269917908754,
+            0.023021140866008313,
+            -0.015676872472071106,
+            -0.024497517908898602,
+            0.005873633718973869,
+            -0.022828585938203683,
+            0.011748341004940436,
+            0.04624205727927451,
+            -0.015442165499674641,
+            0.018565551071491767,
+            0.015246505395507096,
+            0.004119269796224144,
+            0.004862735250819252,
+            -0.000145323321960046,
+            0.05155838952887077,
+            0.0027052423345323936,
+            -0.016218063670586664,
+            -0.07462470920944148,
+            0.043695974746295135,
+            -0.027608273720377708,
+            -0.03860912774695494,
+            0.002823110565150435,
+            0.026471469849508766,
+            -0.0026063577189074103,
+            -0.029230652014151735,
+            0.05684164470477843,
+            0.00022182868300572677,
+            0.015590845552956364,
+            -0.024939617549529242,
+            0.01035803286203917,
+            -0.010225298012384327,
+            0.019571847648759157,
+            0.01841613041441054,
+            0.0156981470538354,
+            0.021764297694793343,
+            -0.018831230007033416,
+            -0.014419529638789993,
+            -0.0029965475899652273,
+            0.028881185012095884,
+            0.09163526460163937,
+            0.022644302207811818,
+            -0.017933909936330133,
+            -0.04663928709016869,
+            0.004962951444105128,
+            -0.041333041174526026,
+            -0.027545628007617527,
+            -0.037142394579894954,
+            -0.013358143162788402,
+            0.02719686432396677,
+            -0.0034971800937504205,
+            0.04413635817371732,
+            0.04903857110328917,
+            0.03186067155630788,
+            0.007831051050269414,
+            -0.025257538623313574,
+            0.023517513773109414,
+            -0.0016527403209505964,
+            -0.0016101860707429347,
+            0.014448515185903624,
+            0.009905018312477684,
+            0.0024114582235163955,
+            0.00172043851384822,
+            -0.03818316221102173,
+            -0.02304537008110578,
+            0.01024969233570089,
+            -0.01901116356327339,
+            0.019791121460848986,
+            0.07985067340821365,
+            -0.051903191272236665,
+            -0.025924646976556942,
+            0.014766511532849252,
+            -0.002073086076894999,
+            0.030959269926800852,
+            0.006394417986617089,
+            -0.028177586693713442,
+            0.010725285336674526,
+            0.0009614052276924192,
+            0.07500253788978564,
+            -0.00036794182149960157,
+            0.004058458332826519,
+            -0.013218269166951234,
+            0.027613539268683245,
+            -0.012995170442480582,
+            0.023388707018488407,
+            0.026389423313105165,
+            -0.003068639956061266,
+            -0.02661967649632026,
+            0.026145274570635913,
+            -0.03300591322174331,
+            -0.004556072066733024,
+            0.020045722084112896,
+            0.0033437343914784865,
+            -0.010224220905483085,
+            -0.0025025498684079703,
+            0.03460103245395426,
+            0.04514362357278504,
+            0.03797031988058962,
+            -0.004586662475576956,
+            -0.06351723722596923,
+            0.06037983484369874,
+            0.019415753124363902,
+            0.028850618647994476,
+            -0.0034810237045849006,
+            0.02957938154808314,
+            0.02642652523231673,
+            0.013187653724133275,
+            -0.007856320804109569,
+            -0.007065882659482447,
+            -0.016672784625197085,
+            0.029020161154209707,
+            -0.013243058921970134,
+            -0.014900932834241732,
+            0.006001769901896033,
+            0.008648456186944271,
+            0.03484763413464497,
+            0.012279368984306564,
+            0.02379643508687133,
+            0.04933044959399941,
+            -0.021047723935647383,
+            -0.007264823701706426,
+            0.0007058774795716792,
+            -0.010512071832195085,
+            0.027489671668749157,
+            0.009731157570361245,
+            -0.03195919640758579,
+            -0.0033626031972691915,
+            -0.04282419484651207,
+            0.01873977910485268,
+            -0.014398285316695653,
+            0.012854631447494207,
+            -0.0017372959978092432,
+            -0.037331867825735776,
+            0.001088209292238142,
+            -0.012345684323522151,
+            0.037096063326414734,
+            0.005050219690024652,
+            0.0008906051515353004,
+            -0.003697709449630797,
+            0.006548479142737107,
+            -0.010727663712299397,
+            -0.007486226350610755,
+            -0.02704618481069485,
+            -0.019752552016312926,
+            -0.014632581296303486,
+            -0.0037501138119724717,
+            0.014588411795703245,
+            -0.007119137071719376,
+            0.016440258608998535,
+            -0.03686645352111013,
+            -0.023546402448713415,
+            0.005628703432152666,
+            0.024988145751166143,
+            0.011971199595808573,
+            -0.010868778329190568,
+            -0.03270046433624425,
+            -0.021950559285357267,
+            -0.011889914022657738,
+            -0.005123971523701009,
+            0.02619713302280345,
+            -0.021990810305904974,
+            0.01165257474485436,
+            -0.009117901804315863,
+            -0.007492082755249399,
+            0.010725902070291232,
+            0.01862811996181951,
+            -0.026694281868322685,
+            -0.01623776415010199,
+            -0.009403997538684658,
+            -0.02207329027097839,
+            0.025939540497196416,
+            0.014178803271471762,
+            -0.001141821318145744,
+            -0.006281187017652108,
+            -0.035993365198554964,
+            0.0026192776111449633,
+            -0.015254976771874396,
+            -0.0116872453009606,
+            -0.015780016491993135,
+            0.021421328283124968,
+            0.02889432250124136,
+            -0.01871009416296692,
+            -0.026540472640514536,
+            0.015703370682110186,
+            -0.0002849235245164438,
+            -0.02594930889995505,
+            0.04491828744809679,
+            0.044063522197410734,
+            -0.037441864491547405,
+            -0.007005339187980476,
+            -0.030381794393931814,
+            0.06901083845539985,
+            -0.017604082400603037,
+            -0.04060591381445593,
+            0.06044850935418443,
+            -0.01756490171206357,
+            0.025539717592705295,
+            0.03452018789549939,
+            0.0016217577522523927,
+            0.00018825054264059277,
+            -0.002563606092068712,
+            0.04770277039047993,
+            -0.0032010143255574255,
+            -0.03425328601475849,
+            -0.02105909767072799,
+            -0.00840881022563103,
+            -0.028699447677549002,
+            -0.015925160861589006,
+            -0.0055750742281166,
+            0.017513987082479483,
+            0.010399718005952185,
+            0.02267201252273823,
+            0.01258511878601025,
+            0.0015081978484714903,
+            0.01874981572091931,
+            0.011158889421057756,
+            -0.002607810686437211,
+            -0.003732023411827874,
+            -0.007787189972165309,
+            -0.05524012485122728,
+            -0.025110689689279583,
+            -0.03946383236703947,
+            0.004443572529325475,
+            -0.018745077598477586,
+            0.005926160888274727,
+            0.039430331582190524,
+            -0.0049500324516682,
+            0.004352795787899792,
+            0.015566139991780173,
+            0.013148156356408315,
+            -0.015279560337042242,
+            -0.03316846614871165,
+            0.0001592209771257332,
+            0.018362411161743347,
+            -0.012706582418607377,
+            -0.03424477956681259,
+            -0.001991022841226589,
+            0.0033467286047492175,
+            0.03357019284647071,
+            -0.024359110796680374,
+            0.00359117029951049,
+            0.005061818038334802,
+            0.011359582931760315,
+            -0.026793992746244076,
+            -0.018170136079817746,
+            -0.017421603453826397,
+            0.03996608926471779,
+            0.02587013058343799,
+            0.015763410145648683,
+            -0.04512536449983358,
+            -0.02293473890636517,
+            0.048575083783480195,
+            -0.009589860756342604,
+            -0.06486262416944208,
+            0.03447666009581266,
+            0.016105654870253654,
+            0.003526582643722421,
+            -0.01354754623151814,
+            0.05401575248587799,
+            -0.03563190295559527,
+            -0.017365166993914202,
+            -0.0046288709698227735,
+            -0.033657212157212645,
+            -0.007686252059880836,
+            -0.001385273642139782,
+            0.048229062789530236,
+            -0.029083855744728694,
+            -0.030465559177859865,
+            0.0024368661249980416,
+            -0.024144031154493057,
+            0.021693278579520123,
+            -0.016405088055634124,
+            -0.018439935250344958,
+            0.006948603359995389,
+            -0.033943037395199695,
+            -0.003007796423804453,
+            -0.016783195426536633,
+            -0.02270589491206155,
+            0.020874438753828814,
+            0.016724666903417947,
+            0.027033184180344852,
+            0.0019847647852695785,
+            0.017749524526371133,
+            0.0204661893602285,
+            -0.005604929968008821,
+            0.028462234633072944,
+            -0.020790459496727066,
+            -0.04183816355941877,
+            0.03946487892663695,
+            0.01192861758684541,
+            -0.0018024841275380242,
+            0.01077082591611702,
+            -0.01475239020295694,
+            -0.03952356245715099,
+            0.03683950018557526,
+            -0.056767672022551764,
+            -0.007507600090017693,
+            0.009530544352246083,
+            -0.00788856051420074,
+            -0.011204694027562521,
+            -0.010587701466318598,
+            -0.028604940377315504,
+            0.023749537635431462,
+            -0.013672419168235691,
+            0.01953449229707597,
+            -0.011449939491554354,
+            -0.012238180529977605,
+            0.05795536247491174,
+            -0.021518581774137322,
+            -0.040672955795925836,
+            0.026066659238628543,
+            -0.014061106927896723,
+            0.03328357230532424,
+            0.03957107406164427,
+            0.030670947051333917,
+            -0.004278577327978564,
+            0.025885946540364826,
+            0.0319106101668716,
+            -0.003462160036198642,
+            -0.04686868330851868,
+            0.00555303933159587,
+            0.02156143971867743,
+            -0.03739529489075782,
+            -0.01025626126223168,
+            -0.02141970191890338,
+            0.022134910980548407,
+            0.017807484142010496,
+            -0.007232643256393455,
+            0.011318041192005274,
+            -0.010489012862455913,
+            -0.013003217163997554,
+            -0.00847279880392288,
+            0.02357274819910594,
+            -0.0016485066139744577,
+            -0.005873113634992993,
+            0.006424249429163587,
+            0.004008703814301658,
+            0.0054639858216457724,
+            0.010198079500408441,
+            0.017694720520798957,
+            -0.017087021465208255,
+            -0.0025558869389290554,
+            -0.016503579212015177,
+            -0.0420082404922411,
+            -0.003563173424359332,
+            0.022924161700566468,
+            0.012345933179990547,
+            -0.00729908668365632,
+            -0.004234683369642856,
+            -0.0022962545569187934,
+            -0.020666205489143392,
+            -0.014796149043465308,
+            0.024562653223873056,
+            0.030217494971086438,
+            0.007810961433007407,
+            0.007781263640449456,
+            -0.0030348559388780655,
+            0.007310876971216526,
+            0.016099591640649783,
+            0.02311929597780174,
+            -0.04160162038353416,
+            0.027152100845897007,
+            -0.00025396315502635146,
+            0.023717123370023598,
+            -0.04186918759097667,
+            -0.017988531857836703,
+            -0.03106663560912813,
+            -0.023595079699181436,
+            0.02437134274097885,
+            -0.02317559303912261,
+            -0.012327263775257306,
+            0.000989566600374836,
+            0.00954921216572729,
+            -0.005155079395685017,
+            -0.017508404061379164,
+            -0.04582321461886958,
+            -0.0014367264231457263,
+            -0.030417229142190933,
+            -0.017829710557332023,
+            -0.001056472801554354,
+            0.005016912084754242,
+            -0.06276644565618253,
+            0.0109965681725148,
+            -0.07270669271977365,
+            -0.004542796125637244,
+            0.040139998232966566,
+            -0.017890441783630643,
+            0.055909102081208445,
+            -0.05029104308417623,
+            0.01673962668706085,
+            -0.04507293803055318,
+            0.015385506084493484,
+            0.025677372277539627,
+            0.030235171586316813,
+            -0.022189978375852563,
+            0.008461188938299745,
+            0.008033707483259569,
+            -0.045089505755429415,
+            -0.04144698743458133,
+            -0.013640256640978024,
+            0.01036782197518379,
+            0.0012696958402094232,
+            -0.016090728932978384,
+            0.023838437877040163,
+            0.011946482002767135,
+            0.03342604921859589,
+            0.04211801335157251,
+            0.002708015395637621,
+            -0.047156646610976634,
+            -0.013104464812562618,
+            -0.003933116891120681,
+            0.03155321708116329,
+            -0.018866109579446184,
+            0.015376812010630058,
+            -0.01955557376085726,
+            0.016255217383093355,
+            0.017868332102274922,
+            -0.020325179506828335,
+            0.013434364256590097,
+            0.004176904701252156,
+            -0.03877060944639854,
+            0.0006475542749369515,
+            0.00042468300294954125,
+            -0.02116992978038693,
+            0.0032654387271108293,
+            -0.00575003786159731,
+            -0.05149954071159492,
+            -0.031382926943128234,
+            -0.015730563640253065,
+            0.0382312502769352,
+            0.004297939848910064,
+            0.019724983545557365,
+            -0.017907869879607064,
+            -0.005165628493894463,
+            0.03698976707579373,
+            0.008501001373266799,
+            -0.01725527821298066,
+            0.006879620077191753,
+            0.02455792200929138,
+            -0.03224193972301503,
+            -0.039181328884413266,
+            -0.0008206986106407542,
+            -0.023306758893533952,
+            0.01672946960876128,
+            -0.02658587698869307,
+            0.005873620623736886,
+            0.0025331170979801232,
+            -0.03490809296591325,
+            0.02456764314036355,
+            0.003291113462353163,
+            -0.004303423672882112,
+            -0.020359544091059775,
+            -0.016892415165120203,
+            0.017174797487220006,
+            0.04391489511478065,
+            0.007327321684948843,
+            -0.003943713832544148,
+            0.00862684231313155,
+            -0.006361172785753631,
+            0.05826688726534012,
+            -0.02284956969636637,
+            -0.026680023999604643,
+            0.027527436347931256,
+            -0.022597669279069723,
+            -0.006277863878949772,
+            -0.017343446085471857,
+            -0.012065192152203348,
+            -0.0004183712490442143,
+            0.019339311388446943,
+            -0.019423562920897294,
+            -0.010212527447243049,
+            0.004783006556748757,
+            0.012824133870627983,
+            -0.004855945108275706,
+            0.006239949458562236,
+            0.006847581690471813,
+            0.03094005096378827,
+            0.026993131409761086,
+            -0.011331881651539968,
+            0.03297014291876675,
+            0.010265600643742144,
+            4.666113318251278e-05,
+            0.001970879281172325,
+            0.002523664335535017,
+            0.0034107811520368574,
+            -0.03143890864030425,
+            -0.01624584278940617,
+            0.013468729048885906,
+            0.005233309338223403,
+            -0.004550544967856448,
+            -0.023783653541634683,
+            -0.009402494239255264,
+            0.03334923712760375,
+            -0.003260424721959948,
+            0.02748395114281811,
+            -0.0058374897745283065,
+            -0.01985840930273332,
+            0.017595418271610163,
+            0.02881743246631328,
+            -0.05928552440605384,
+            0.017062540894682354,
+            -0.0011508547522925793,
+            0.008953761538519997,
+            0.01610561914042553,
+            -0.008894160513648953,
+            0.013301991014533684,
+            0.04080863883897929,
+            0.010738051773755976,
+            -0.023345002072886145,
+            0.004215726310838674,
+            0.01541749448570608,
+            0.01291813299324752,
+            -0.022747942296528025,
+            0.0008640791861357522,
+            0.011008522066277018,
+            0.005893793007012561,
+            0.0026106450193140156,
+            -0.006278394080583205,
+            -0.025659188287146268,
+            0.024180550333596625,
+            -0.0044491292740074925,
+            0.026967973346698617,
+            0.04390909273429254,
+            0.009596855239692042,
+            -0.03686020608700941,
+            -0.013109951664921729,
+            0.0002799680359670127,
+            -0.0038537577602593366,
+            -0.0037552094562213916,
+            0.011105295862811114,
+            -0.04185615640149602,
+            0.019950890919369135,
+            0.006453241355231516,
+            0.015314645427201533,
+            -0.020031193483734364,
+            -0.024530173488182337,
+            0.0031949828520724337,
+            -0.01492645564652531,
+            0.012875827757040941,
+            0.005057170413681063,
+            -0.004319834710426384,
+            0.005386862713494232,
+            0.00030848054013413157,
+            0.008384941028326983,
+            -0.010811526993301901,
+            0.03175068297908541,
+            0.004680987487229685,
+            0.009264547472281849,
+            -0.02856314241700808,
+            -0.021083721348429348,
+            0.020113396771629096,
+            0.03572366771051171,
+            -0.01708529918264109,
+            0.03346921724975398,
+            -0.009042046660366718,
+            0.0033219155387035227,
+            0.0151720168318272,
+            -0.015805318210244414,
+            -0.008057236707694303,
+            -0.0005483009439015931,
+            -0.002917527004423299,
+            -0.014538511124345527,
+            0.007302979537905532,
+            -0.0208529958115422,
+            -0.02965227242147972,
+            0.020064178921373875,
+            -0.00011892622932349096,
+            0.016455188521316146,
+            0.010335044657747514,
+            -0.007816155992852236,
+            -0.0052262330850905165,
+            0.006959344748883874,
+            0.04432147882994085,
+            0.026734484553994495,
+            0.05547661378166294,
+            -0.005787997952888261,
+            -0.004559444854138438,
+            -0.0082295581123363,
+            -0.006588172755707144,
+            0.04524009609473585,
+            0.015915003261638767,
+            -0.004910285095602864,
+            0.017495607570191165,
+            0.01557886571266685,
+            0.03858259260136037,
+            0.0027648525902558316,
+            -0.016912796336835408,
+            0.04014755669075221,
+            -0.004077405782937414,
+            0.008616827127381418,
+            0.011102692199778469,
+            0.0024185742218798562,
+            0.00807141794538637,
+            0.02996274720815659,
+            -0.012429186019907761,
+            0.01680942269506771,
+            -0.04571916592313758,
+            -0.02775530246832996,
+            0.014370280587419414,
+            -0.004931905540824582,
+            0.03419959665162407,
+            0.01668521021731861,
+            -0.023105305446763272,
+            -0.024761385385514877,
+            0.021529428247380984,
+            -0.002603042335561961,
+            -0.024135326707477325,
+            -0.015906833215844473,
+            0.003286605488908195,
+            -0.0023437997044735617,
+            0.0060251848371355166,
+            -0.013481652076194643,
+            0.0002908426939460218,
+            0.01953362396253743,
+            0.0038544889405965057,
+            0.020760460182873287,
+            -0.016224396121239458,
+            -0.00259233446682179,
+            -0.009282861237653619,
+            -0.02952130765234087,
+            0.017988370145301916,
+            0.006138236653154911,
+            -0.007607182955958238,
+            -0.009516767050028553,
+            0.0048347778733230365,
+            -0.005819209642901871,
+            -0.0025851701689263706,
+            -0.01073014664868326,
+            0.012433465684360061,
+            0.025784014942833097,
+            -0.016875624785552303,
+            0.007944349714123104,
+            8.651589872318479e-05,
+            -0.022306044741692753,
+            -0.022127016745066307,
+            0.020143472307319017,
+            0.002792646249762062,
+            0.008768813346304779,
+            -0.004074214276300631,
+            -0.00015024341785520542,
+            0.002990550758627699,
+            0.01951746553838439,
+            0.03896451381932467,
+            0.004232673242619471,
+            0.027876586543280867,
+            0.0145860303663199,
+            0.03994234200854897,
+            0.009657017763624678,
+            0.00696607158576712,
+            -0.026055387694656488,
+            0.02819890357011778,
+            0.00018297502510753014,
+            0.043190622821775236,
+            0.022488402221902314,
+            -0.0407730101161195,
+            -0.0021946518523759066,
+            0.01669520746276111,
+            0.013212623257794862,
+            0.017009844136059122,
+            -0.004537512994041156,
+            -0.014041213424740049,
+            0.028135404528975894,
+            -0.022859586430516003,
+            0.0008637006862481223,
+            -0.004486332934188017,
+            -0.025031795510001887,
+            0.007351370254175368,
+            -0.004567816916440959,
+            0.030025413528233364,
+            0.0014976740344892735,
+            -0.022285096307825804,
+            0.022837170656498543,
+            -0.009551269792616045,
+            0.011045530488684207,
+            0.008809047185288755,
+            0.020065001354321994,
+            0.031993362733068575,
+            0.009143210434415064,
+            0.02133640814803879,
+            0.012241777474889248,
+            0.014771538127546474,
+            -0.008804192497858033,
+            -0.02341427260922065,
+            0.023167424359504862,
+            -0.0011598649565273595,
+            -0.029627604902846688,
+            0.0021991474725787303,
+            0.003261878079344105,
+            -0.0005935713331682358,
+            -0.009144689470851402,
+            -0.005580557862988033,
+            -0.007240836451875213,
+            0.019631377011298905,
+            -0.007085213886352692,
+            0.007017573363709754,
+            -0.004074171791476129,
+            -0.00822871835665378,
+            -0.0421101573155962,
+            0.004210914075286222,
+            0.003408427700622687,
+            -0.01752179330665445,
+            0.006384440404174512,
+            0.011121481025573962,
+            0.043020575455262985,
+            -0.022809979913698254,
+            -0.0005737138409049003,
+            0.019206623212867476,
+            0.0009562551889820241,
+            -0.023427467810802533,
+            -0.038931688376706186,
+            0.007172345672967499,
+            -0.03467050750542982,
+            0.023185301620612612,
+            -0.0007075596339842658,
+            0.02387977204565927,
+            -0.03043468414449851,
+            0.013909304572370437,
+            0.004784668158421625,
+            0.012524879174804915,
+            0.021654512058639427,
+            0.03142146922870715,
+            -0.02053473475540882,
+            -0.050409862638621725,
+            -0.005406667316025549,
+            0.00035138762584085976,
+            0.022018487016014856,
+            -0.026014580970974616,
+            0.00025633004699302783,
+            -0.003349094027291921,
+            -0.02401366769212778,
+            -0.01735996745614562,
+            0.01962340791446025,
+            -0.050011480783403936,
+            0.015042913576492952,
+            0.0262737424555697,
+            -0.02507473901367252,
+            -0.05408889321034101,
+            -0.018988587082325706,
+            -0.028217893770659686,
+            -0.009285770094319318,
+            -0.00661159230987788,
+            0.028597405154740078,
+            0.003525576256054959,
+            -0.011462219358952466,
+            0.025401668390029348,
+            0.006155837251830549,
+            0.0026050693705924593,
+            0.015985722392086222,
+            0.007289910674212967,
+            -0.011452511453623235,
+            -0.014599577191293152,
+            -0.04558584556100621,
+            0.010197536827229795,
+            -0.0045431962637690155,
+            -0.019372598050508005,
+            -0.030147301333718016,
+            -0.0137828704002156,
+            -0.027060515748528666,
+            -0.0017291966950107505,
+            0.053783684195381654,
+            0.03292494290301388,
+            0.005560172296221419,
+            -0.0026085670565688026,
+            -0.010466890489356092,
+            0.036247483457995895,
+            -0.006789746401727701,
+            0.03662690472838588,
+            0.001784522073210422,
+            -0.0242673253380092,
+            0.01169382884548301,
+            0.02892627181542449,
+            0.009858451662166387,
+            -0.0005592283645355213,
+            -0.006160132931009508,
+            -0.008782830470079547,
+            0.0031207630972208394,
+            0.025613233596604418,
+            0.003268170392795548,
+            -0.010062991425127038,
+            0.018106967407336815,
+            0.004480291699976396,
+            0.02010374270557836,
+            0.03555357587371324,
+            0.023489837757983054,
+            -0.021782173652026497,
+            -0.026581762796336082,
+            -0.0013561486781077014,
+            0.0072266106510934505,
+            0.03551967295942857,
+            -0.01651315514034218,
+            0.0001274396319698452,
+            -0.008272735507120127,
+            -0.00894931703949637,
+            0.011149952898504355,
+            -0.0029671593121986116,
+            -0.0030752010285951576,
+            0.01662711184497578,
+            -0.0011672263615879709,
+            -0.033448222499147764,
+            0.0078288560210653,
+            -0.023496331346917084,
+            0.0051040606404415545,
+            -0.004113104164416575,
+            -0.007076525381595441,
+            -0.002157548492138289,
+            0.006302962169276006,
+            -0.008065841090139872,
+            -0.00035242603988955366,
+            -0.01586406637763306,
+            0.008271894439780969,
+            0.007438312999409211,
+            -0.0069617481460406095,
+            0.011402932583388205,
+            0.002666895909532698,
+            0.0005219881149353855,
+            -0.012159425670629751,
+            -0.012042145306711322,
+            -0.00957276588761916,
+            -0.03401863879157551,
+            -0.02604353712472079,
+            -0.024949533021906844,
+            -0.0014209149651479556,
+            -0.006566685925666972,
+            -0.03187796739029736,
+            -0.01209152137926458,
+            -0.011110763124407752,
+            0.007756961097537267,
+            -0.013260548545917543,
+            0.005433285079114047,
+            -6.399098520041351e-05,
+            -0.0044960488826674555,
+            0.016615636438440516,
+            -0.003643291599919672,
+            -0.019488010211707343,
+            -0.03927818878149417,
+            0.008209336177358258,
+            -0.02805593125673831,
+            -0.0038816771550190788,
+            0.007607119219721711,
+            -0.00528681677614909,
+            -0.029502897838743516,
+            -0.009149794737430706,
+            0.008767001199311985,
+            0.003507217008280066,
+            -0.01773349756494436,
+            0.03579670773605949,
+            0.026971617790626075,
+            0.011875179879209696,
+            -0.019891342676443626,
+            0.0508929813275565,
+            -0.0047010735337056275,
+            -0.012544935684845649,
+            0.018297503869683027,
+            0.02355022268131519,
+            0.03273167043015095,
+            -0.024830243320788693,
+            0.0009370814985214882,
+            -0.021937793653233454,
+            0.00535662615596573,
+            -0.035009915581788176,
+            0.0028595659314725012,
+            -0.014395027018699446,
+            0.015596459769560532,
+            0.03397372881171701,
+            -0.013254226072981457,
+            0.011195747181764677,
+            -0.016299384375589934,
+            0.00730244731485402,
+            0.00569944090461748,
+            -0.012916284280345044,
+            -0.005812311882399539,
+            -0.022902815152116493,
+            0.020140276669979015,
+            0.005809799950176385,
+            -0.0008130655714656685,
+            0.0104085826652653,
+            -0.044723432238900435,
+            0.03286429597337671,
+            0.008730225944501239,
+            0.017599189944921396,
+            0.0029762446259779998,
+            -0.0003068671951232415,
+            0.011611551606224214,
+            0.011228401175698422,
+            -0.016029727902740706,
+            0.004345913009102136,
+            -0.004613061137378433,
+            -0.03850533909474302,
+            0.009363351681220008,
+            -0.03631184057978666,
+            0.014427784677340159,
+            0.016026872002233575,
+            -0.004170582842643461,
+            -0.025166189571409366,
+            -0.005428471012333673,
+            0.010827910811085384,
+            -0.009783069664327864,
+            -0.009987805234276443,
+            -0.04163358183060102,
+            -0.025236440364963034,
+            0.034097508113604576,
+            -0.002119474613596975,
+            -0.025826403511707147,
+            -0.010830738112107807,
+            -0.011351132636491575,
+            0.020207687645133,
+            -0.006542026283946858,
+            -0.0006263975804936843,
+            -0.04150011545311013,
+            -0.0073396962917864515,
+            0.011192225907726928,
+            0.027746213794734322,
+            -0.035970017302005566,
+            0.02708126943501373,
+            0.002105633542220187,
+            0.014262674845755191,
+            -0.01626664983003942,
+            -0.005631805361903626,
+            -0.013990539604372344,
+            -0.035877093361375685,
+            0.016487723013906077,
+            -0.008607758248055054,
+            0.004714436224135091,
+            0.04458009880881818,
+            -0.0036872068506712727,
+            -0.009466125348499464,
+            0.007600705817195143,
+            -0.0034590883553048063,
+            -0.010351576085287273,
+            -0.0028564505793999295,
+            0.001959661298507,
+            -0.007080697863126256,
+            -0.04183476693971616,
+            -0.00499888976530217,
+            0.04684391782633148,
+            0.006361744381879857,
+            0.013276793765092465,
+            0.013516220521120684,
+            -0.018335746557932003,
+            -0.045577810515431215,
+            -0.0071616111758423025,
+            -0.0032763880176386808,
+            -0.022708028392758567,
+            -0.0240284945459861,
+            -0.008735040856657758,
+            0.005844145402902923,
+            0.006359368165686724,
+            -0.01120763599008222,
+            -0.022676131361171057,
+            -0.027085968700452136,
+            -0.010526103176253857,
+            -0.02123944716136792,
+            -0.010792145796733375,
+            -0.027142270190771887,
+            -0.028742234538796804,
+            0.02880426953386151,
+            0.007575182232859564,
+            -0.004084088301176117,
+            -0.010005765476121966,
+            -0.01255640546813973,
+            -0.0004774761617900028,
+            -0.0025307994917461232,
+            -0.005822065709999144,
+            0.0038614286313282164,
+            -0.011390482620811868,
+            0.023152086963976554,
+            0.0021220701260546895,
+            0.018329162172128138,
+            -0.05772324571231113,
+            -0.002709083851090071,
+            0.021497896395795905,
+            0.0033025410488410913,
+            -0.005412445141910314,
+            0.029352650459792132,
+            0.011223947499453582,
+            -0.006539460282439711,
+            -0.031355428613718234,
+            0.020942976176820262,
+            0.024122993515785947,
+            0.00817615634503138,
+            -0.012994160510030828,
+            0.01221095914384204,
+            0.01629778384790159,
+            -0.004494985018246821,
+            -0.011161098655072933,
+            0.005586857967039181,
+            0.020941039642372112,
+            0.013580622629911925,
+            0.0013398071867077221,
+            0.030905022925366163,
+            0.0006862084391508121,
+            -0.03521231612862609,
+            -0.01908191995763771,
+            -0.01275995164030872,
+            -0.003584331305986364,
+            -0.015264425022763126,
+            0.005294329654809353,
+            -0.002355465010876059,
+            0.006335052149475905,
+            0.022655379849785256,
+            0.03593231939948758,
+            0.0146937743203089,
+            0.00937330328448886,
+            0.003167462724350819,
+            -0.0013235176101076284,
+            0.012102872918129403,
+            0.008638011773075119,
+            0.0350701240268404,
+            -0.014653221697693864,
+            -0.011003994230902835,
+            -0.0037767288519369537,
+            0.010100131061186566,
+            -0.004113527518662366,
+            0.020414393256136372,
+            0.03274709615649469,
+            -0.0037222949335916876,
+            -0.005604552180246868,
+            -0.006160118231975786,
+            0.0041467454414452565,
+            0.01807879127455274,
+            -0.01252134965321004,
+            0.004364858987501461,
+            0.005549300665159658,
+            -0.0071897162833861,
+            0.018567059255813558,
+            0.0002622074720860951,
+            -0.012983550838417323,
+            0.03266376564402191,
+            0.11449509387240629,
+            0.0017251916835047628,
+            0.007125215166916139,
+            -0.040229997815548584,
+            -0.020537987527148888,
+            -0.00245728735683473,
+            0.005779972828836593,
+            0.007004732428236457,
+            0.02225914982152608,
+            0.006505827715591914,
+            0.0006708045351142873,
+            0.031708522931495536,
+            -0.017334083511709092,
+            -0.002522861743921599,
+            -0.01745656559688129,
+            0.013890451996979048,
+            -0.014312605700272691,
+            -0.013784963572164409,
+            -0.01572417853955075,
+            0.004507425871748977,
+            -0.000601158156253668,
+            0.0016158265715634729,
+            0.01791303234734911,
+            0.0011926917387973374,
+            -0.026393351252156246,
+            0.0006694008039871878,
+            0.001026031515214626,
+            -0.004514419527575117,
+            -0.011320929719992686,
+            -0.02223001723942815,
+            0.016235841442653918,
+            0.014188311619681008,
+            -0.004895218832118094,
+            0.009244697376301004,
+            0.010558796153959677,
+            0.02707485418319703,
+            -0.028380342401606758,
+            0.0040204430418629405,
+            0.007023041225985294,
+            0.01606556581447933,
+            0.017617862031190843,
+            0.0034870847111552862,
+            -0.011253315705053224,
+            0.03238289399766116,
+            -0.010563282687352038,
+            0.0065093536270233385,
+            0.014966802208644462,
+            -0.03541611540945151,
+            -5.821682910348165e-05,
+            0.0061985444938402745,
+            -0.011716201661363997,
+            0.009580035961315467,
+            -0.03120692534097433,
+            -0.0039291243987978845,
+            0.009317468331974374,
+            0.017611312558635673,
+            0.008804996982574765,
+            -0.0034129169537151925,
+            0.010822226698822738,
+            0.012510661458054741,
+            -0.0027147975086409966,
+            -0.008855213644027594,
+            0.01238502686635956,
+            -0.0007687053446954701,
+            0.00450108863711973,
+            -0.00799170686059026,
+            0.0029755595812552293,
+            0.002290650917244691,
+            -0.01554856300008412,
+            -0.011779430686394945,
+            0.00438393126679289,
+            0.0036931891075561957,
+            0.001535662168147833,
+            0.012014408636737167,
+            -0.006087255045485002,
+            -0.0077816220325470226,
+            0.0040573724156794284,
+            0.003136943055301706,
+            -5.505014316363865e-05,
+            -0.030625245024561508,
+            0.01585923084309149,
+            0.0067595435845396405,
+            0.010013308598626031,
+            -0.005315626260228517,
+            -0.00903142414091241,
+            -0.007240920015018893,
+            -0.01228521172101564,
+            -0.010657894223173453,
+            0.007658605592936815,
+            -0.00724924993432376,
+            0.011124174211132038,
+            -0.03128027787357322,
+            -0.013988715435985316,
+            0.01231792993245938,
+            -0.01528161242159377,
+            0.0007675821323979293,
+            -0.008389914312201402,
+            0.0009417242487395995,
+            -0.005119304379761705,
+            0.007887436211143328,
+            -0.03731503169547854,
+            0.007878001990271541,
+            0.02456170736765724,
+            -0.00330924625934981,
+            -0.020259771366167557,
+            -0.009613930457931786,
+            0.0241314385136084,
+            -0.008267831641886344,
+            -0.01645888287387047,
+            0.008421304862524506,
+            0.010140813149093993,
+            -0.0006309118552875393,
+            0.0008237784055549769,
+            -0.003726984191590917,
+            0.01669125388798323,
+            0.02054777616420997,
+            0.041930770964107515,
+            -0.014789610432004559,
+            0.01092349703703152,
+            0.02866938974792608,
+            -0.03378576980722847,
+            0.014728113846952417,
+            0.02518409645155824,
+            0.014590531327650454,
+            -0.0004446956315190408,
+            0.001609777186372437,
+            0.0068939321280085475,
+            -0.014273492115311677,
+            0.009247676157551494,
+            0.013380104440744367,
+            -0.018407746260686045,
+            0.020444430509441124,
+            0.03140071554008003,
+            0.0025839998101985736,
+            0.01355205268903058,
+            -0.006268939830271642,
+            -0.013860692186849113,
+            0.014717364068116991,
+            0.031429975488060896,
+            0.0006919565473572909,
+            -0.018561500446336806,
+            0.019128244642842268,
+            0.006988095806264863,
+            -0.003678529269242034,
+            -0.0021993078588142664,
+            -0.021334919445200388,
+            -0.021003586791369228,
+            0.017304350704792804,
+            0.013508779408333964,
+            0.027794485858413365,
+            -0.010594702782952029,
+            -0.007818089330662161,
+            0.014293688127901867,
+            -0.022368568887004094,
+            0.015426482139881204,
+            -0.02160820059598751,
+            -0.00571499813703566,
+            -0.0077402210272634235,
+            0.03274360648629609,
+            0.001441304806200403,
+            0.02734331677589297,
+            0.016457027764064002,
+            -0.033609420134045206,
+            -0.029790535553657092,
+            -0.00712106584238484,
+            -0.02307933297713104,
+            0.010356561171963577,
+            -0.03901183838032092,
+            -0.00071735150189382,
+            0.008251646656432826,
+            -0.008079917288471352,
+            -0.0007087526632966007,
+            -0.006419191399519291,
+            0.013989344185652506,
+            0.013661633160336152,
+            -0.0228103143826663,
+            0.023644639659577426,
+            0.008343481940133684,
+            0.025998483906975782,
+            -0.02076055376006461,
+            0.019148568238437927,
+            -0.017705363646416224,
+            -0.02777313933338929,
+            0.014358130220942483,
+            0.023585731219788467,
+            0.01635364038245971,
+            -0.007122853965024528,
+            0.007110003583832299,
+            -0.03556761320536796,
+            0.027585457849964405,
+            -0.013064062623866305,
+            -0.012504161289952458,
+            -0.0032379276562094555,
+            -0.011931827532360686,
+            -0.002574161005800279,
+            0.0008533907465694845,
+            0.0034345232246102806,
+            -0.03131462673429805,
+            -0.0002450959360520896,
+            -0.03727416416766574,
+            -0.028944814604815072,
+            0.01360126708805471,
+            -0.00036361284498363273,
+            -0.0049097813612814616,
+            0.02933242705521636,
+            -0.026339136251046315,
+            -0.02693295650849956,
+            -0.009590247544093352,
+            0.02853096456887074,
+            -0.0220673798697511,
+            -0.004519050393279848,
+            0.0015034014871316069,
+            -0.01828226786700525,
+            0.004436383319946364,
+            -0.0012748481042235446,
+            -0.010416065158876917,
+            -0.01105146139270597,
+            -0.03229479099927737,
+            -0.005822133182096844,
+            0.02106215163620738,
+            -0.006957125617688484,
+            0.002746241228040847,
+            0.013312248331456934,
+            -0.010968629118224582,
+            -0.0029946900704643812,
+            0.0017875284441794235,
+            0.023304175718480834,
+            -0.021033941064173063,
+            0.007714334541806199,
+            0.005955442031111047,
+            0.0016405586860131207,
+            0.018406562522853148,
+            -0.0054922197063461825,
+            0.0037690571959124617,
+            0.0162126534419016,
+            -0.009969304596173302,
+            0.017959307638726878,
+            0.007691549802475525,
+            0.006681837561745192,
+            0.002775700618222622,
+            0.007712770283579111,
+            0.009752652973700739,
+            -0.013506020126078112,
+            0.016911141425375933,
+            0.022468715726349706,
+            0.00023233555209372766,
+            0.021066580799163234,
+            -0.02302516292580905,
+            -0.028618544628453015,
+            0.018297878810036087,
+            -0.0225078941299981,
+            -0.012735208680311652,
+            0.016459925496937065,
+            -0.0043032724578332855,
+            -0.00797609922711709,
+            -0.014986057073351055,
+            0.02260618561608301,
+            0.02132930921464508,
+            -0.020249788476562527,
+            -0.012385936327068884,
+            0.0072304431443959505,
+            -0.004373475447051833,
+            0.007086656081865733,
+            0.03195810137269375,
+            -0.015982210897392764,
+            0.006818890093901413,
+            -0.0038306138017521626,
+            0.0030753642449792215,
+            0.017830546304667783,
+            0.009470018737705222,
+            -0.0019815241970236146,
+            0.030100238416495066,
+            -0.001330614013444443,
+            0.016392226282459177,
+            0.0004888028477871524,
+            -0.010948833692356176,
+            0.006343890495626752,
+            -0.026652806306173643,
+            -0.008627570055007363,
+            -0.001203530223345183,
+            -0.009581494451831351,
+            0.07598616930784373,
+            0.003930153552069478,
+            -0.01295957554290434,
+            -0.001673007115291578,
+            -0.001368287102181597,
+            -0.006897179139643672,
+            -0.00960679451603541,
+            0.0287166325268306,
+            -0.016613613331623346,
+            -0.01894527592131826,
+            0.0008370014132002703,
+            -0.006785618888421608,
+            0.00919924251249592,
+            0.01816363936999796,
+            -0.010225961928154075,
+            -0.01604759761423727,
+            0.001577138242530474,
+            -0.003520564936698765,
+            -0.004395434431595494,
+            0.0052399043934902014,
+            -0.010127122101423486,
+            0.006456662777583096,
+            0.026560491731327886,
+            -0.013684536319659296,
+            -0.01425196172663986,
+            0.004084982328707292,
+            -0.0010682084059972982,
+            0.009758641913073978,
+            -0.00864879062601467,
+            -0.00481222050200678,
+            -0.014375330954464725,
+            -0.011378890549403524,
+            0.02505641780042516,
+            -0.02861164504722546,
+            0.0034060847953066354,
+            0.014937227531980433,
+            -0.00016904224334342915,
+            0.01541381851648916,
+            0.0033028855218525644,
+            -0.03615935765718852,
+            -0.013677774380513626,
+            -0.02073839691250734,
+            0.019821074228011413,
+            0.01688422012576042,
+            -0.0033731840788147442,
+            -0.013840993586441808,
+            -0.026149880178874887,
+            -0.007430563876740068,
+            -0.02466234818567362,
+            -0.020708282040996014,
+            -0.019276991448222817,
+            0.006137002799644838,
+            0.005162892426268572,
+            0.022881759407670583,
+            0.0025854880108157857,
+            -0.015058889448404231,
+            0.0029948608544278386,
+            -0.006722836436935071,
+            0.011845298212977357,
+            -0.02747944486818378,
+            0.015311272959213262,
+            -0.01610493464983465,
+            0.025239768880017402,
+            0.007425208902253236,
+            0.0010569008485112994,
+            -0.008822681490988575,
+            0.00010704167529046774,
+            0.014883787023445807,
+            -0.004186968199973372,
+            -0.0007781259002896265,
+            0.02787555889813697,
+            0.02087373813833492,
+            -0.0017783425135865238,
+            0.008743718448136432,
+            0.028026471206552342,
+            0.019081083283793984,
+            0.0325261942213039,
+            -0.013815526479760766,
+            0.005113676604826083,
+            -0.028873600114081546,
+            0.010510475843707614,
+            -0.010871039729515013,
+            -0.010201824683340651,
+            -0.0067946599549165735,
+            0.027756887504389347,
+            -0.007962601205446472,
+            -0.0006687976124826922,
+            0.03469894503563926,
+            -0.014185294847328063,
+            -0.00010543064621374546,
+            0.0007386418834840544,
+            -0.009759481309152999,
+            0.02742722730386853,
+            -0.0073590367291510046,
+            0.018456032721630843,
+            -0.019560882612141943,
+            -0.013705917411254969,
+            -0.011894560208199602,
+            -0.001142768182039817,
+            -0.02576754877141608,
+            -0.008755240066826174,
+            -0.02249287216087102,
+            0.00859172262856532,
+            -0.0046012388553094375,
+            0.018845920253446127,
+            0.0011617844393195566,
+            -0.005275799428018491,
+            0.029069367036565145,
+            0.0088080893110627,
+            0.018100411389202663,
+            -0.0004971948011851863,
+            0.01216566778850985,
+            0.02369448391321832,
+            0.002637001729688794,
+            0.005012948614448268,
+            -0.024213704079796478,
+            0.003420184574622472,
+            -0.0004359900462023086,
+            8.265488264269535e-05,
+            0.016267948861008616,
+            -0.019572032453100303,
+            -0.02216264534756073,
+            0.021871880626992544,
+            -0.012742926070754964,
+            0.011676769226089007,
+            0.0035362870996336194,
+            -0.010732229460038356,
+            0.0006838777488770268,
+            -0.03083438636386314,
+            0.008820746379709355,
+            -0.03216685858285627,
+            -0.032230808828686675,
+            -0.013547681710236494,
+            -0.009994106571421729,
+            0.0008758088338659712,
+            0.028447187234357855,
+            -0.00795317029726568,
+            -0.015797892954136448,
+            0.005363565557248085,
+            -0.011199252697624111,
+            -0.00817162221789202,
+            -0.0007867892785855576,
+            0.005856939999700809,
+            0.009461959355310273,
+            0.005997804620690314,
+            0.011648335019189696,
+            0.007491946658046298,
+            -0.008028800679169293,
+            -0.008837725220736191,
+            0.015889923814605387,
+            -0.0014409716087425222,
+            -0.01176424391171685,
+            0.030430130082762363,
+            -0.013300620185884817,
+            -0.0005592926632214984,
+            -0.0021975669766032472,
+            -0.0015687208316663597,
+            -0.0021652076181411244,
+            -0.014225289143358926,
+            0.01870568404189425,
+            8.334501002782455e-05,
+            -0.014288806218067678,
+            0.0032841743905975634,
+            0.003841780428153241,
+            -0.0036498339483411074,
+            0.008388922960269383,
+            -0.008147275785908849,
+            0.017348147498892,
+            -0.015113395971380741,
+            0.008740452410210708,
+            0.0038335161236289234,
+            0.0043482078469644385,
+            -0.0065755348640655865,
+            -0.01592809184384243,
+            0.009642593996701513,
+            0.012051872087663658,
+            0.02173264012652684,
+            -0.02525660485859878,
+            0.00876161653901566,
+            -0.005374533920781607,
+            -0.014534179652013885,
+            -0.016127866637369738,
+            0.00982303248417706,
+            0.022322910435388785,
+            0.0063086346898091296,
+            0.012139263756190026,
+            0.00013383660542132047,
+            0.005848259030010402,
+            -0.014767198851314148,
+            0.010570158368602513,
+            -0.011355774405066537,
+            -0.018439261453084426,
+            -0.005243587266408389,
+            0.028894120979627323,
+            -0.005492024421080143,
+            0.017938747701972086,
+            0.0021350711250093858,
+            0.029007356490801176,
+            -0.008686781541806358,
+            0.0014883465513564527,
+            0.0039866022469891155,
+            -0.01295271525889559,
+            -0.004272008629260297,
+            0.01976273843646234,
+            0.04882868638601926,
+            0.010204475902027572,
+            -0.0009155109545661691,
+            0.010069071220988797,
+            0.005626418787776309,
+            0.002148101019776153,
+            -0.00805781810841286,
+            -0.030431071067211577,
+            0.008883707378543252,
+            0.005628218215212441,
+            -0.01997825125961566,
+            0.019838234890736068,
+            0.026483365072033395,
+            0.023040623182975964,
+            -0.011585643598171132,
+            0.016539930468533123,
+            -0.010092529089936188,
+            0.0177008735686489,
+            0.028665424114962996,
+            0.013810061902648515,
+            -0.010487150104822674,
+            -0.0020847870330548534,
+            0.006329342599564054,
+            0.004186981876004016,
+            0.00026239408223817384,
+            0.015190614519766204,
+            -0.019155180136775043,
+            0.01005728016778505,
+            0.014725826697964637,
+            0.03322436221319799,
+            8.049930133627564e-05,
+            -0.01645191835557271,
+            0.0016249281996128196,
+            -0.009976155662939693,
+            0.00856399492362911,
+            0.000632427127231252,
+            0.0032242260692672124,
+            -0.009688119824276678,
+            -0.012682779503915694,
+            -0.002851918183031152,
+            0.009403205508474241,
+            0.0021188716361101698,
+            0.002759004011231831,
+            0.007641925150177999,
+            0.0022830062031146204,
+            0.01713024474384216,
+            -0.0011181980841546123,
+            -0.00029141889484139217,
+            0.004601090256674874,
+            0.010509199590505558,
+            -0.01649852835819883,
+            -0.017548369964263514,
+            0.004337237475053179,
+            0.012217494210546133,
+            -0.013159179948224596,
+            -0.013241981632059755,
+            0.015704543608959117,
+            -0.01913575130892795,
+            0.024934102868065366,
+            -0.00942381892138331,
+            -0.004543611336526587,
+            0.000343200128934031,
+            -0.00727538880611703,
+            -0.0038751118497522673,
+            -0.0030064279504724797,
+            0.003984715814505561,
+            0.0019261818110367147,
+            0.007472350699291607,
+            0.008472369788801792,
+            0.014568547873264247,
+            0.006043895352236181,
+            0.0032580830408778395,
+            0.009109104498825864,
+            -0.014523846294121576,
+            -0.014800649887278973,
+            -0.002704791598011488,
+            -0.008895394038700809,
+            0.00902016867266151,
+            -0.006607182154874112,
+            0.00845408382182461,
+            -0.015501867835442603,
+            0.015573642225745032,
+            -0.012200552219016517,
+            -0.0008900215712478553,
+            -0.009666233545144905,
+            -2.7470277092642562e-05,
+            0.008191697904514857,
+            0.014605287653054787,
+            0.0070552942843382215,
+            -0.031921405225964206,
+            0.0017403427534633655,
+            -0.008846568352320488,
+            0.03446319360336241,
+            0.002612606662691094,
+            -0.01060863123202008,
+            0.020352181356568008,
+            -0.01592124132155158,
+            -0.00027704668715660693,
+            -0.01602075523652062,
+            -0.004797557413047212,
+            0.00707939446796679,
+            0.016399017978990946,
+            0.02759049512664933,
+            -0.005697020356865688,
+            -0.007995305872076221,
+            0.006384047872862793,
+            0.029590696670334232,
+            0.012246955508890425,
+            0.024748351124415785,
+            -0.007633712004672087,
+            0.014348228716100414,
+            0.007773260552660029,
+            0.0076045488653348975,
+            0.004321689250440021,
+            -0.002591801797145718,
+            -0.011698574606499644,
+            -0.014974971814360001,
+            -0.0037563183330623384,
+            -0.016987915346053498,
+            0.00012247833530362867,
+            0.021344400276501117,
+            -0.00475351423959752,
+            -0.0033847161875730026,
+            -0.01874186437694691,
+            0.0033461670218611346,
+            -0.010029445302549983,
+            -0.00367421246775698,
+            -0.013133469163793961,
+            -0.0254122035745965,
+            -0.018556369080030412,
+            -0.002155244024999317,
+            -0.00929309186646207,
+            -0.015194269781263309,
+            0.0070354690219415535,
+            -0.016305117309271634,
+            0.0080837166473439,
+            -0.00448473417024141,
+            -0.016508775226236883,
+            -0.00958925666995396,
+            0.01100335153658195,
+            -0.0041819993634961685,
+            -0.01929493648733792,
+            -0.01282300765645703,
+            -0.011300260564838231,
+            0.0051733704121938884,
+            0.005996780699975213,
+            -0.0038111648304178544,
+            -0.0028279639348479987,
+            0.016640121845366295,
+            -0.010197365811240557,
+            0.009878730312873492,
+            -0.03335474019383245,
+            -0.03551826040406959,
+            0.004497847402931096,
+            0.01507007013703286,
+            0.0013085881889541252,
+            -0.02078680101065954,
+            0.010806315988966003,
+            -0.016499186313125556,
+            -0.00858013792516617,
+            -0.008522534127498645,
+            0.009649521088411411,
+            -0.007283003380127847,
+            0.01299542385123007,
+            -5.3226845309105124e-05,
+            0.030477683597373687,
+            0.0074153455615142755,
+            0.004161881195438522,
+            -0.00679081048132963,
+            0.008354912600159931,
+            -0.001621776416108105,
+            0.004080191054849395,
+            0.02120524874226696,
+            0.012440848502230418,
+            -0.00218041171518331,
+            0.006458415098595872,
+            0.005678055538564355,
+            -0.017186537377121235,
+            0.02507960402044456,
+            0.024678178106806115,
+            -0.008205288425409454,
+            0.004108402466376321,
+            0.005783335302169668,
+            -0.008846663924040241,
+            -0.006897707231869481,
+            0.006527795262026734,
+            -0.018474181903007085,
+            -0.0008721381678008483,
+            0.010598084358464908,
+            -0.02255441909004843,
+            -0.004840511019512733,
+            0.021745651362109403,
+            0.004631567631335321,
+            -0.018558972627958834,
+            0.02764788406651861,
+            -0.004942361238163609,
+            -0.0027290999934532494,
+            -0.010876135202650048,
+            0.01671497143541799,
+            -0.012173527207375091,
+            0.008443317346655462,
+            0.011021541861818043,
+            -0.006295639322287843,
+            0.003083685676540999,
+            0.008277206401503182,
+            -0.004504302913715969,
+            -0.0068075424927125676,
+            -0.0010508416962215848,
+            0.010167160518175114,
+            0.0036980815941666336,
+            -0.022704963865525812,
+            -0.006020073261002279,
+            0.02934227992246887,
+            -0.015082922782242507,
+            -0.0050916398557337705,
+            0.005376294716056329,
+            0.007709342013963906,
+            -0.003476236561444464,
+            -0.0023399991233097337,
+            -0.00787486645740744,
+            0.010356414979428607,
+            -0.015950083845400274,
+            0.02076360898078935,
+            0.025608263494550438,
+            0.024504236990840842,
+            -0.014814235990238182,
+            0.010828584133720797,
+            -0.011703992394890898,
+            -0.011047470905668532,
+            -0.015542835222173864,
+            -0.001658296303013357,
+            0.001970208852236451,
+            0.013107754130616248,
+            0.009666887546609378,
+            0.0235853683657408,
+            0.006473544089109194,
+            0.008973300312205858,
+            0.014118601565410305,
+            -0.003902202854013708,
+            -0.018303438244719996,
+            -0.014584226585460755,
+            -0.011977613285028946,
+            -0.01756345491020528,
+            0.0004563602312336091,
+            0.020461659996271687,
+            0.012417744207683839,
+            0.024136626592375227,
+            0.002119213701333967,
+            0.013759902247467922,
+            -2.9525392347047554e-05,
+            -0.000892319792935443,
+            0.00045783825646242434,
+            -0.005238483120312843,
+            0.019255756091120846,
+            -0.004599752813817219,
+            0.01656589522110913,
+            0.0006443773300684925,
+            0.007133940816771615,
+            0.013554761340256566,
+            -0.01704388669741371,
+            -0.003443100892108368,
+            -0.010029620309829285,
+            0.000941391966877521,
+            -0.019633111750884034,
+            0.01934571074127927,
+            0.010273330977233852,
+            0.0017121345993533963,
+            -0.018327189396014198,
+            -0.0029706385099949693,
+            0.03448635131998772,
+            0.019458999453011265,
+            0.0050734354363614525,
+            0.02632443373898792,
+            0.013018762352749236,
+            0.014443611411502168,
+            0.010153370950346075,
+            0.02356766253713842,
+            -0.023479811300626968,
+            0.02095255383200973,
+            -0.011590347850340786,
+            -0.007414136601134069,
+            -0.0035774888741646705,
+            0.0025141508183687115,
+            -0.013588691733097813,
+            0.00982877622088977,
+            -0.0017916539399655826,
+            -0.025386211797192578,
+            0.009164316443206016,
+            -0.008811052979979271,
+            0.010464442616928296,
+            -0.007774569252594923,
+            -0.0002806136360286729,
+            -0.0127084026190163,
+            -0.015184317235990752,
+            0.001259201880504347,
+            -0.009158429132281081,
+            0.023908150915412617,
+            -0.021210848953875233,
+            -0.01321384014436847,
+            0.013997502713857512,
+            -0.006869443694669701,
+            -0.011364239609894453,
+            -0.00374449980963666,
+            0.023673050288879224,
+            -0.009243341446512421,
+            -0.01359367519009039,
+            0.014464166953530989,
+            0.003834735113400469,
+            0.0059713317014342395,
+            -0.007508862694272452,
+            -0.008428705474135382,
+            -0.003280788036976103,
+            0.005998840043285335,
+            0.026615077617284676,
+            -0.014059930972065399,
+            -0.0032833264512892022,
+            -0.009189999799498163,
+            0.004118204537632102,
+            -0.02359889635281631,
+            0.02771075351384059,
+            -0.008871395303200375,
+            0.021705177758560575,
+            -0.0015924449308442096,
+            -0.006239188158271628,
+            -0.024231036074349822,
+            0.027522945599928003,
+            0.003683870966132199,
+            -0.010221180899173243,
+            0.0012768725460662862,
+            -0.052005292736885915,
+            0.016636109796947025,
+            0.008429895913163985,
+            -0.019507994429834353,
+            0.003697519981610208,
+            -0.004773584856005853,
+            0.018983847580558853,
+            -0.022966993228850697,
+            -0.014078986007109447,
+            0.008283766047449966,
+            -0.01153285124362746,
+            0.0031439180465649996,
+            -0.02527749501821793,
+            -0.005754537200570427,
+            0.0011760345167229269,
+            -0.005276936074280004,
+            -0.001705933750221904,
+            -0.020022761608132548,
+            -0.025972227850795115,
+            0.017666017379157135,
+            0.015820875579600418,
+            -0.002278262448951789,
+            0.01320818011561616,
+            0.008709899932124187,
+            0.00836573261091119,
+            0.016034263394940008,
+            0.007124509234103267,
+            0.0081473711608784,
+            0.045498533239850446,
+            -0.019846323044145485,
+            0.007371629414298193,
+            -0.0014854149174929534,
+            0.013951741898629716,
+            -0.030467924817555014,
+            -0.0055144480998344705,
+            0.0062802744565489945,
+            -0.0049373689340857915,
+            0.03510454786147091,
+            -0.017326094216003073,
+            0.01694160526735264,
+            -0.0026075809870445238,
+            0.0007420394265014909,
+            -0.02042003877666893,
+            -0.026156937726134134,
+            -0.014294938857971834,
+            0.0011857406889443987,
+            -0.0010047941261608975,
+            -0.011946583993069242,
+            0.006797001710574735,
+            0.004917161312925866,
+            -0.0020713379451245515,
+            0.015652523103715316,
+            -0.02952061566240878,
+            0.007356800041369564,
+            -0.010733578620243498,
+            0.020008504830359524,
+            -0.00990331048284697,
+            -0.0013319400098308093,
+            0.007236960885181047,
+            -0.004122868383988748,
+            0.013216043472071927,
+            -0.0038337251451538677,
+            0.004227115653354611,
+            -0.005221321656766545,
+            -0.01794343569722877,
+            -0.0016435261495179767,
+            0.021429008432348098,
+            -0.008349684096540855,
+            0.031864497493526205,
+            -0.01264743878037828,
+            0.009174982784686633,
+            0.010692990337462762,
+            0.0006628831036334152,
+            -0.005286285851702483,
+            0.009052971814211658,
+            0.0021431970274545893,
+            -0.008690520448278274,
+            0.006934310986093342,
+            -0.018349449937310143,
+            0.008494451785507706,
+            0.03394632739506833,
+            0.0018511328780472266,
+            -0.0007519661670070526,
+            -0.007266663179613392,
+            -0.01335844201823745,
+            0.018540713029383066,
+            0.005488273906733518,
+            -0.002546531152114063,
+            -0.027536660869590543,
+            0.02007746621157705,
+            0.0012265798861291223,
+            -0.004755325069578531,
+            0.012466557901411086,
+            0.0020495867632654885,
+            0.019989531151046736,
+            -0.010171678639113316,
+            0.030757957899447096,
+            0.010050036220124393,
+            0.002959833051227505,
+            0.0038556928367700113,
+            -0.036655749074851536,
+            -0.01652152778069314,
+            -0.00019357502472740266,
+            0.014271581054865031,
+            -0.012430647252826173,
+            -0.022432580814844155,
+            0.0010346169306977334,
+            0.01148004051449691,
+            -0.0033898642762881046,
+            -0.009650665305271706,
+            0.01560757541276278,
+            -0.020342213902670575,
+            0.006701302394408393,
+            0.007693672045981921,
+            0.0287100214318104,
+            -0.0020735384554022153,
+            -0.0012324838849990056,
+            0.000626416803904792,
+            -0.0055669860801192935,
+            -0.0008037366126870824,
+            0.017251369734470945,
+            -0.013661316947002466,
+            0.0175926211217534,
+            -0.012735241490550407,
+            0.0064713656749838585,
+            -0.0021092378432037563,
+            -0.010629579860050812,
+            -0.020114807971202787,
+            0.0029609495764307357,
+            0.009378003163375899,
+            0.006920144698428806,
+            -0.010438984881449953,
+            0.010715409812532968,
+            0.012197673689706352,
+            -0.00753962861484439,
+            0.014519905132928041,
+            0.022173989895384192,
+            -0.007687503607783572,
+            0.013633321414009471,
+            -0.0041446334728745095,
+            -0.014056264816568229,
+            -0.009985463824440833,
+            0.023929750617160402,
+            -0.0005563806942667597,
+            0.008756997373618809,
+            0.00388125051641897,
+            0.0007856447663040696,
+            -0.011226530517316183,
+            -0.001653447552601681,
+            0.019294448360333633,
+            0.011831934675073843,
+            -0.0013888534822008192,
+            -0.0010525764356699885,
+            -0.0014566936525653407,
+            -0.009427557739483988,
+            0.02049188741459728,
+            -0.022030783927622997,
+            -0.0019604940727076573,
+            0.00461018148439991,
+            -0.022589816803875618,
+            0.0001863117949313752,
+            -0.024570217962989905,
+            -0.01569952241187498,
+            0.011763033781122343,
+            -0.0022530631823737554,
+            -0.016028810867854907,
+            -0.0003242701040942633,
+            7.424127301806556e-05,
+            -0.007815789940508527,
+            -0.004135500768181258,
+            0.006115909992703774,
+            -0.00034193217510010117,
+            -0.0032262189374874842,
+            0.013430154310506884,
+            -0.009996622196870008,
+            0.015431528911575157,
+            0.0067029980830614145,
+            -0.015702167721800483,
+            0.0021406610839557003,
+            0.018500219544264753,
+            -0.0058910087864916085,
+            0.00787021425357441,
+            -0.014681907249972176,
+            -0.0037495551894426303,
+            -0.005787661351944086,
+            0.006051351977821761,
+            0.024849195780885833,
+            0.012969434450643372,
+            0.01139340445077682,
+            0.017405361843075933,
+            0.004325000715054734,
+            -0.01572297405297993,
+            0.0007416798945675923,
+            0.01682727189828363,
+            0.010464036837245312,
+            0.009804326468113703,
+            0.0008975556667684365,
+            0.0043704860361487635,
+            -0.005142176468800175,
+            0.009317064072855551,
+            0.02915482449687158,
+            0.0013434689612997423,
+            0.008696141010760192,
+            0.017276730345147134,
+            0.027038741396131265,
+            -0.0035088997194497725,
+            -0.007469614319788499,
+            0.008739942310300076,
+            -0.009657991604173173,
+            -0.001081629174976152,
+            -0.017000351183542026,
+            -0.012310275736750868,
+            0.016510505259358398,
+            -0.006084627062230492,
+            0.02848913486719956,
+            -0.0010655806791907225,
+            -0.02026498848361482,
+            -0.01565505608194376,
+            0.001404549479440637,
+            -0.01362132895080652,
+            0.005287566256051299,
+            0.014187265685427415,
+            0.01917046513020831,
+            0.004687191682875024,
+            -0.022258337721085656,
+            0.037364962197399806,
+            0.01757108963349427,
+            0.019377870795760123,
+            -0.019422840506328382,
+            -0.006159311237412833,
+            -0.009077095160676276,
+            0.02116697012944498,
+            -0.00880692095488465,
+            -0.009875674856043954,
+            0.024014332004451508,
+            0.017812606845656484,
+            0.005679806286464216,
+            0.006736609904079586,
+            0.0020048663439319585,
+            0.0025414630253661444,
+            -0.015831593851012047,
+            0.017957902498542212,
+            -0.005858125698347523,
+            -0.0058068554681761945,
+            0.017529514573571298,
+            -0.005989581792993925,
+            -0.0030306021203732347,
+            0.00015839997806822045,
+            -0.019697052145531482,
+            0.011618112886207201,
+            0.008256007601576028,
+            -0.017229774429089857,
+            -0.0014757560402567304,
+            -0.0002404480326945295,
+            -0.014118694740134205,
+            0.02004885146995733,
+            0.01694893356262835,
+            -0.0013248884139405398,
+            -0.0051752815372013315,
+            0.008223868147393114,
+            0.007000679132597393,
+            0.005256194890343006,
+            -0.019496098876977206,
+            0.005230597149342019,
+            0.005103909294824105,
+            -0.0014716649591412774,
+            0.010274346323051634,
+            -0.009030594825985526,
+            0.033740380586893826,
+            -0.006505788197604522,
+            0.013065658941136721,
+            -0.0043634731684193745,
+            -0.0055529072007985125,
+            0.001816020676232658,
+            0.013054468478598695,
+            0.009753501030700457,
+            -0.00970446155762939,
+            -0.004172354962129865,
+            0.018697982092775737,
+            -0.011983156146516074,
+            0.0013300791598046338,
+            0.01564072865804035,
+            -0.004858814041438713,
+            0.004708235083775013,
+            0.008635204790345423,
+            -0.004489505237783172,
+            0.01801512858970922,
+            -0.016482572999490366,
+            0.0023953436660054026,
+            0.010694869684962088,
+            0.02390864159830096,
+            -0.019152187046410703,
+            0.021786596486708207,
+            -0.017951050280323393,
+            0.026538582634436314,
+            0.014484894489694262,
+            -0.0035222259457311444,
+            -0.01946200885147006,
+            -0.00850690913394181,
+            0.011654565129063417,
+            -0.021423803876241532,
+            -0.0007985948078290641,
+            -0.0007290222944686444,
+            -0.021367628543092133,
+            -0.011189364017231132,
+            -0.009961779129778331,
+            0.010701637745810138,
+            0.001918633898887013,
+            0.010380983868185359,
+            0.008002182827196827,
+            0.010784760574518087,
+            0.014405640442760822,
+            0.0008958107769384412,
+            -0.010743001445440941,
+            -0.023795589003655273,
+            0.016987708698679597,
+            0.00438463274382093,
+            0.0007862212695139549,
+            -0.01627468994515667,
+            0.011936651269737248,
+            0.013250643815768723,
+            -0.009385605548926286,
+            0.009121882469425302,
+            -0.03716901103204463,
+            -0.004552808410398574,
+            -0.0035704811925430366,
+            -0.0020309197365173475,
+            -0.00908678158053104,
+            0.01559445415882485,
+            -0.013682507836005851,
+            -0.0009590922904844632,
+            0.00011696985057536548,
+            -0.0074756526903548915,
+            -1.5467890432254378e-05,
+            0.020649928837031005,
+            0.008438785939908428,
+            -0.0037382105257999757,
+            0.010376213224654074,
+            0.008826031303751312,
+            -0.013900359449287676,
+            -0.01527474712786084,
+            0.019035366035884273,
+            -0.018776918681908793,
+            0.0063098418544296165,
+            0.01677916431403883,
+            0.020778813096283988,
+            -0.02891862501830075,
+            -0.0007231469961304435,
+            0.022093939332021185,
+            -0.0012807635799950273,
+            0.006370841109790851,
+            0.008781853138377092,
+            0.006754082980602544,
+            0.0029102159062355374,
+            -0.003711173531879644,
+            -4.843770941622283e-06,
+            -0.003940474588075707,
+            0.007155539022862352,
+            0.004366433868349658,
+            0.036453728071874844,
+            -0.013661129348165794,
+            0.002541308805735001,
+            -0.0058838191638835155,
+            -0.00347401304572066,
+            -0.00593469688967835,
+            0.004407665856950772,
+            -0.010055727994798024,
+            0.004330827357536078,
+            -0.001570861535288093,
+            -0.0003731129422346693,
+            0.004883969598300302,
+            -0.001701673379646408,
+            0.007036892564844683,
+            -0.011863650867472625,
+            -0.0003979727750011258,
+            -0.020174851412609312,
+            0.021117284557875267,
+            0.00011666551282792189,
+            0.003662462447326642,
+            0.03723741246290771,
+            0.00476064510662738,
+            -0.00238355131307445,
+            -0.012880057949435798,
+            -0.013551338766321525,
+            -0.012986444917747435,
+            -0.020703821629987552,
+            -0.0036665673548525388,
+            -0.021006311026247164,
+            -0.0052061406234528915,
+            0.010362862594785442,
+            -0.011540184420787572,
+            -0.004023659825622843,
+            -0.015881589631889473,
+            -0.006531981517815174,
+            -0.03114211633687585,
+            -0.01124607488650473,
+            0.007493228366452009,
+            -0.019149285949049886,
+            -0.0039932870926865805,
+            0.026850121590606017,
+            0.001095360635760988,
+            0.01387696424607248,
+            0.0038058248676879625,
+            0.008117686785954085,
+            -0.0045937472017383955,
+            -0.0005669025036141417,
+            -0.01928244877716143,
+            0.003391980330529026,
+            0.008525354854576925,
+            -0.00361507909530282,
+            0.0070008404454106905,
+            -0.008835727374865623,
+            -0.016950017602980382,
+            0.0021654956715219517,
+            0.009334379529385142,
+            -0.006639140003365139,
+            -0.017587472740268683,
+            -0.011049929670847685,
+            0.010401189523318744,
+            -0.02283429470064187,
+            0.006257145380266984,
+            0.016439581169238607,
+            -0.009363030375624585,
+            0.004651575425738153,
+            0.019809183749755425,
+            -0.011810674576781702,
+            -0.005811328501528397,
+            0.008471992070998505,
+            0.013891259999721588,
+            0.0095606134736748,
+            0.02954498207274139,
+            0.0026596809804008996,
+            0.000561870602466584,
+            0.004889263753490895,
+            -0.004277061142504552,
+            -0.017423200644235665,
+            -0.006243787897080996,
+            0.0038507494787521856,
+            0.012873996671952184,
+            -0.009879417960879032,
+            -0.00048575277472139055,
+            0.01682604801673206,
+            0.004530939990187663,
+            -0.0033901248079550284,
+            -0.011868399508738274,
+            -0.020080331108085103,
+            0.0015580259874809898,
+            -0.01362200460317948,
+            0.015354347040143093,
+            0.003390601069331352,
+            0.0015284610287850575,
+            -0.004239880336806429,
+            0.01828645373058079,
+            -0.005013418527935815,
+            0.012749833447770729,
+            -0.006148890994661865,
+            -0.004587831231448011,
+            0.007986533783001116,
+            0.009166578151006778,
+            0.0004939857995863664,
+            0.05421019830078428,
+            0.002838056604877719,
+            0.01333205648597435,
+            0.013419127695838135,
+            -0.014523733369484934,
+            0.014086525147734823,
+            0.012688569853552613,
+            0.018497344547821693,
+            0.013037819487160016,
+            -0.012353776077273462,
+            0.017354801119939964,
+            0.0024084723134891487,
+            0.003909681065813095,
+            0.0022202438064449653,
+            -0.004528576079628071,
+            0.019594683950225115,
+            0.004070222328547378,
+            -0.0074362443729372105,
+            0.004453618452623641,
+            0.005007543168224267,
+            -0.013105524222311019,
+            -0.0112958915514782,
+            0.027105208848863218,
+            -0.01219417557128091,
+            -0.018749971743148304,
+            0.006706594269933445,
+            -0.02008458683853808,
+            -0.03763686535076864,
+            -0.0021332074289244973,
+            0.009992361098878196,
+            0.002795183524495193,
+            -0.00253695604043486,
+            0.01614298064601032,
+            0.016730443047719683,
+            -0.016985296937992726,
+            -0.018245902728910347,
+            0.0025257780765481825,
+            0.0016333325293570182,
+            0.01861807236979066,
+            -0.023168663546030196,
+            -0.006378864850313035,
+            -1.3795465857380962e-05,
+            -0.006829008132963306,
+            0.004850306884033312,
+            0.004703280112191218,
+            -0.0071714759384722965,
+            -0.0027773519096287217,
+            0.002256846645341137,
+            0.01975383140755998,
+            0.013198754584066554,
+            0.01572971337890945,
+            0.002600741959577911,
+            -0.005033508062449482,
+            -0.016817359058724685,
+            -0.005450011991586895,
+            -0.013478330855299777,
+            -0.009091854345715177,
+            -0.003042052092498012,
+            0.008217376329142566,
+            -0.025053469059115735,
+            -0.011965083373220982,
+            0.0119294363235832,
+            0.00391377331459168,
+            0.0003099203772539202,
+            0.007607672728136279,
+            -0.003830132495364688,
+            0.010331053246451333,
+            0.013823890473448383,
+            0.0076112887952959766,
+            -0.004723377809559611,
+            0.009600411526173948,
+            0.011460781436250457,
+            0.006015477068316837,
+            -0.005072711638895421,
+            0.023859899048048694,
+            0.01821920335673539,
+            -0.01680365156728093,
+            -0.002149107383573353,
+            -0.0027829676356423997,
+            -0.01884308738652242,
+            0.009665830801645276,
+            0.00880322398336952,
+            0.008308268605449346,
+            0.003906369351868086,
+            0.00974531700889854,
+            -0.013875904819275925,
+            -0.0050570811397453235,
+            0.002870120865445989,
+            -0.005729556771404781,
+            -0.01901337075686543,
+            0.000297242372701423,
+            -0.011987192261104132,
+            -0.011327157550660895,
+            -0.01409710922165158,
+            -0.003221240464411861,
+            0.005283472511900512,
+            0.021472187640986972,
+            -0.004410881709548286,
+            -0.001855852749039012,
+            0.01351697599517034,
+            0.010060125186506398,
+            0.0021658738823899905,
+            0.0002206474541239167,
+            0.005418828324605768,
+            -0.0027919130944783397,
+            0.0008094042430809068,
+            -0.0015826797905407809,
+            0.021089276765434876,
+            -0.02016611086095107,
+            -0.009413977569213423,
+            -0.009288078098692112,
+            0.005842197094339927,
+            0.028793032462603636,
+            -0.007165634502373454,
+            -0.010104002083017078,
+            -0.01994315293556678,
+            0.0003786527922888834,
+            -0.07810609175343784,
+            -0.005210636756258548,
+            0.001870513618910238,
+            0.03126380113105983,
+            -0.010817888476436834,
+            0.0033673768058629823,
+            -0.002204487854177595,
+            0.014542299655712928,
+            -0.001760107965206874,
+            -0.0013283746982194259,
+            0.0071954806737858056,
+            -0.005640400818010697,
+            -0.006210989097047217,
+            0.005904163661619133,
+            -9.945941028848348e-05,
+            -0.010266938705916303,
+            0.00057263080955709,
+            0.02209498395848068,
+            0.0159398346608872,
+            -0.006575439759804175,
+            0.01816748552021211,
+            -0.00719294553484007,
+            -0.009838738592324234,
+            0.005639288780502127,
+            -0.0019007683459126327,
+            -0.015369619176147573,
+            -0.011475977270803784,
+            0.009862808084641291,
+            -0.030069890644150284,
+            -0.00010725753918890802,
+            0.00943698424646706,
+            -0.013540245857270492,
+            -0.008293042205008774,
+            0.007212974960781723,
+            0.006708343474532583,
+            -0.0012017968151025369,
+            -0.0078026000239507145,
+            -0.02527526731750743,
+            0.0027578400032925625,
+            -0.002446811971857853,
+            -0.004899702349784173,
+            0.0019960465173086183,
+            0.001479503476855694,
+            -0.012645905858466798,
+            0.01216748847452763,
+            -0.013378877368040743,
+            -0.012053251571418766,
+            0.012508842412556306,
+            -0.00027167241745418584,
+            0.01682680930386839,
+            -0.020607299598265426,
+            0.0010702151063620052,
+            -0.028075439824932304,
+            -0.007671848153932823,
+            -0.015486630894165054,
+            -0.0009776136330459065,
+            -0.0012939921569540803,
+            -0.03481880531471923,
+            0.012889447665794033,
+            -0.005200149829198305,
+            -0.006094702500154477,
+            -0.00459778303079732,
+            -0.024583504817899775,
+            0.006072281238752663,
+            0.00607350167867964,
+            -0.006506508970816635,
+            -0.016816439635445445,
+            0.008428048142588516,
+            -0.0011040593013238872,
+            0.004873670535200424,
+            -0.007475800310154908,
+            -0.0005638920132076605,
+            -0.013047135093134946,
+            -0.005463454269764494,
+            -0.0008522562324122834,
+            0.01827123304871433,
+            0.0044055851550317835,
+            -0.014526381222031981,
+            0.016862445236481526,
+            0.013250692156820124,
+            0.01602714293656341,
+            -0.0071089293427753115,
+            -0.0004265739315250755,
+            -0.02750630108787212,
+            -0.00327225092954903,
+            -0.023687644154808972,
+            0.00041213204365507744,
+            0.009447660230507965,
+            0.0203905264889617,
+            0.012694421477479161,
+            0.004620617778996077,
+            -0.001557511764988156,
+            0.0066751090724502,
+            0.02696959560423252,
+            -0.01099183126742224,
+            -0.006969111598423376,
+            -0.01884504192789845,
+            0.01113023717919988,
+            0.025135748541199255,
+            0.003381303139273845,
+            -0.011317957771935042,
+            0.008112598754655146,
+            -0.0009247845526290428,
+            -0.010302980598863016,
+            0.02889587382004902,
+            -0.0391134481322538,
+            -0.015599591108049343,
+            -0.004605824192962023,
+            0.007406520608110251,
+            -0.015596917115435033,
+            0.00770468362805995,
+            -0.012614339656105264,
+            0.001699049329817941,
+            -0.005851477164035241,
+            0.03535087340659861,
+            -0.0031950960727572925,
+            0.0006909856233673865,
+            -0.003395233050301427,
+            0.003531135044131077,
+            0.012355948349937309,
+            -0.004444221317023453,
+            -0.030726300006909638,
+            -0.0022983766869702346,
+            -0.011198252640110788,
+            0.009274666307515955,
+            -0.016207714291658314,
+            -0.017342435976846468,
+            -0.009424737744066259,
+            -0.021827243671837953,
+            -0.012403084982789827,
+            0.029747453318486338,
+            -0.005824001042844527,
+            0.01747569398142269,
+            -0.005219060548639168,
+            -0.008365003057800934,
+            0.00021272411948388247,
+            0.01344335658390732,
+            -0.028498240040284714,
+            0.00041963499785238845,
+            -0.002322931702641349,
+            0.0126393550661767,
+            -0.001595134597082556,
+            -0.00983073906977773,
+            0.006903313245694627,
+            0.0010522208685011098,
+            -0.015648940549713764,
+            -0.0007573245570947632,
+            -0.008533504878052644,
+            0.020475329445149498,
+            0.00027791387213409814,
+            0.001074213629646005,
+            0.013864331671732542,
+            -0.02102352993295736,
+            0.01576938071930521,
+            -0.021722775924283996,
+            0.01893267442161547,
+            -0.006577103101038035,
+            0.005772016664149475,
+            0.008283223343661555,
+            0.014068789332250016,
+            -0.014030042849491258,
+            -0.00663752723080222,
+            -0.034436220972634726,
+            -0.0122621879114314,
+            -0.001946072976826902,
+            -0.005308941654794638,
+            -0.014773123927909993,
+            -0.0011054822266082056,
+            -0.026205461509573584,
+            -0.006460373065662098,
+            0.012822188046808264,
+            0.019900312098359022,
+            -0.008409070094265829,
+            0.014163132434889265,
+            -0.006743711872055091,
+            -0.0028230771810882492,
+            -0.012652021749331811,
+            -0.018061180380009395,
+            -0.01083672324723357,
+            -0.01890966636827258,
+            -0.0018635408070944168,
+            0.004709222372611061,
+            -0.015760618199235518,
+            0.011435131737642627,
+            0.03198874888333405,
+            -0.01567271472650171,
+            0.008243406188694909,
+            0.016411496328333205,
+            0.0034236971031973503,
+            0.011728079139322837,
+            0.0038311415233052795,
+            0.004621789116267879,
+            -0.011109450058055078,
+            -0.011473158133621661,
+            0.006524490579223986,
+            0.00954511918454185,
+            -0.004913142601134428,
+            0.024347072255007807,
+            0.0008745419206472714,
+            0.005045794664611216,
+            0.005304506902446091,
+            0.011488187043525824,
+            -0.013135271908307747,
+            0.019746484188715552,
+            -0.002055210086078267,
+            -0.015571665869889405,
+            -0.01535499169234956,
+            -0.01105419264826655,
+            -0.018626399757822328,
+            0.0017108083455869296,
+            0.0181439079855921,
+            -0.0012493542718277685,
+            0.012137753522284387,
+            -0.00924460959150499,
+            -0.005019437549649743,
+            -0.0027156377206998474,
+            -0.038211954658821295,
+            -0.0021878970332906457,
+            -0.018809487178512628,
+            -0.02498945889484935,
+            0.03505216568400238,
+            -4.26477081213836e-05,
+            0.00997643255528116,
+            -0.010310197159232386,
+            -0.0002454902098986129,
+            -0.011065403358802515,
+            0.014954998647970473,
+            0.004624123403100598,
+            0.008872923678983478,
+            0.008560280683876702,
+            -0.016419500515801093,
+            -0.002997557798375716,
+            0.007472390947008284,
+            -0.00374822278027057,
+            0.012680407710694952,
+            0.015353684671120731,
+            0.020886474047003733,
+            0.011161262397502876,
+            0.002045862158732102,
+            0.00887271411363814,
+            0.009563361150297699,
+            -0.0003513237086717377,
+            -0.010459621030424415,
+            0.015419039131459582,
+            -0.02204848739950703,
+            0.012172257374410352,
+            0.04346634265794738,
+            0.019197433209794286,
+            0.029488039318675732,
+            -0.02272810439246353,
+            0.01448092574459596,
+            -0.003801043409441206,
+            0.009476229966122152,
+            0.0049309968883087685,
+            0.02346191840355209,
+            0.003973840862678076,
+            0.0073546443568171405,
+            0.010647581332452525,
+            -0.00972051412996416,
+            0.025644022498404723,
+            -0.00846669916801108,
+            5.1518540791212095e-05,
+            -0.004407265804145216,
+            -0.026400029105519137,
+            0.01615193008014211,
+            0.0081722536401398,
+            0.0010702314763197463,
+            0.0010263623479474385,
+            -0.0019566604229607643,
+            0.007982949510899019,
+            0.007079322685771456,
+            -0.015274148711633536,
+            -0.003886053737590683,
+            -0.018527455865296014,
+            0.0012864569064749438,
+            -0.024337133883771334,
+            0.010466651250597847,
+            -0.017568706814132484,
+            -0.014822354159709854,
+            -0.021666681078831483,
+            -0.013232016867264097,
+            -0.02011653224168991,
+            0.0035858355517491515,
+            -0.008632755352062642,
+            0.013025728891203198,
+            -0.016962939287139683,
+            -0.021494971929779273,
+            0.008430887589398292,
+            0.00870802359711456,
+            0.007481282677955534,
+            0.009728368603458082,
+            0.014996157358749115,
+            0.01736292711410622,
+            -0.022712386143092265,
+            -0.023163001611599246,
+            -0.006682209717364464,
+            -0.003531495447167166,
+            0.006762379833125297,
+            0.00933681196321128,
+            0.012884185500343249,
+            -0.007077895849501418,
+            -0.007554762054641417,
+            -0.013283067570021503,
+            0.006197966297581871,
+            -0.003993067576677973,
+            -0.009464789349849612,
+            -0.0015662739825834614,
+            -0.0061389795593180686,
+            0.002753664732282926,
+            -0.0001791088888074743,
+            0.0007595261237141144,
+            0.017122834676182343,
+            -0.008176319370751018,
+            -0.009787747261447871,
+            -0.02027228453943018,
+            -0.010683273300243025,
+            -0.015604745521156111,
+            -0.0016279406232527138,
+            -0.006506713191861036,
+            -0.014420323628589382,
+            0.010888206569219648,
+            -0.011930708571093279,
+            -0.00887210117616784,
+            0.010807147705360856,
+            0.0073900910661282935,
+            -0.003431679006289002,
+            0.037804020313161775,
+            0.0011945597874820288,
+            0.00778778394048267,
+            -0.003366543306746973,
+            0.005296727102519836,
+            -0.017761251541598876,
+            -0.026382400396890007,
+            0.0010357716464853195,
+            -0.01775777668381489,
+            0.010807713750049491,
+            -0.008515368407459745,
+            0.0016051389544388124,
+            -0.00034958600424912995,
+            -0.009585854918521076,
+            0.0036549720706351473,
+            -0.008110927907950148,
+            0.004212219431447508,
+            -0.03507890639477636,
+            0.011409377585617926,
+            -0.01819120811524444,
+            -0.008962242411035718,
+            0.01710177594046791,
+            0.037560936862121835,
+            0.0028513388998398104,
+            0.00030058353811717895,
+            0.0014792184333546447,
+            0.004427921639326168,
+            0.021438127785303025,
+            0.006339483221420334,
+            -0.001187747087595264,
+            -0.02237982599829809,
+            0.0062435650294483695,
+            0.002200578793697352,
+            0.010092828474934744,
+            0.023105023225115266,
+            -0.016797326021070008,
+            -0.015378520287395565,
+            -0.012732527281000097,
+            -0.010797463778182385,
+            0.009881499418260473,
+            0.02801767062785128,
+            0.01811139137849365,
+            -0.017358098300028903,
+            -0.020207173948076473,
+            0.0016242901745373126,
+            -0.026331540602868762,
+            0.007269751661479859,
+            0.11940631114242901,
+            -0.0045382135734049305,
+            -0.005994407567723939,
+            -0.008679712231646732,
+            -0.004482553208064535,
+            0.016709571294385293,
+            -0.013387043582971358,
+            0.069179848974036,
+            -0.007221545718220735,
+            0.0037392229364642915,
+            0.0028916224964057385,
+            0.00462729158956839,
+            -0.009727339449635306,
+            0.011570145487622916,
+            -0.011287365791681587,
+            -0.012991994493845187,
+            -0.013076194601468383,
+            -0.0180830261130804,
+            -0.007272768410218907,
+            -0.004155564814758654,
+            0.004204993170487392,
+            -0.005525575983182816,
+            0.008051108079520902,
+            -0.017551645696802648,
+            -0.010800070994485792,
+            0.013007176804761548,
+            -0.0023627485852619817,
+            0.005390986239687555,
+            -0.029668390278675554,
+            -0.0007650391259819867,
+            -0.0002446394046947245,
+            0.021280292985495608,
+            0.003137771886236958,
+            -0.0019077298815862946,
+            -0.010810159675564049,
+            0.007040107942856957,
+            0.023835993694726953,
+            0.021875366566457236,
+            0.0180511566130312,
+            0.012135351541044354,
+            0.020483975362807574,
+            -0.008667079096167644,
+            0.026324312888512866,
+            -0.001552207146726937,
+            -0.00397810465151493,
+            0.0077577136352257004,
+            -0.009849600842081875,
+            -0.00816392268697345,
+            0.00677353736462267,
+            -0.017288900282504148,
+            -0.020418071729964713,
+            -0.020383762873109595,
+            0.025741408097358233,
+            -0.011755094311186617,
+            0.002737129201159793,
+            -0.008266210469668866,
+            0.027287436144516414,
+            -0.006283892307996516,
+            -0.0029440642954701632,
+            0.009428876742171362,
+            -0.03788574758692254,
+            0.008834605607057885,
+            0.006463342392514337,
+            0.026711196829464153,
+            -3.9827332346130654e-05,
+            0.005565561841469247,
+            -0.0006738085601962865,
+            -0.006104320995853506,
+            -0.010792830169647851,
+            2.4565453208679464e-05,
+            0.017983908748172493,
+            0.0035181716362611077,
+            0.016970206528324653,
+            -0.005449808667944701,
+            -0.023171388778833197,
+            0.007092509502738752,
+            -0.01555320330471131,
+            -0.023306285898839978,
+            -0.002028783456704567,
+            -0.03196436161255752,
+            0.0033412618908711163,
+            0.007527633574188688,
+            -0.01890570959143866,
+            -0.017859735533232564,
+            0.010329340599904344,
+            0.004382396539605985,
+            -0.0016746397858953155,
+            0.006643785064338525,
+            -0.03307812909454471,
+            -0.01927475592253288,
+            0.004686970010943116,
+            0.012733525168279355,
+            -0.012993441126970916,
+            0.015291832952199082,
+            -0.020491167895268936,
+            0.016451976170508816,
+            -0.012259387189535237,
+            0.018393274368854132,
+            -0.006997668664226754,
+            0.011843450716200625,
+            0.011457415642860215,
+            0.009560877634897626,
+            -0.01790001115302499,
+            0.0214879415944361,
+            0.0066829070506954115,
+            0.03900547021123392,
+            0.02512484499576633,
+            -0.012651294056575184,
+            -0.0018425680226742782,
+            -0.00011361458320146007,
+            0.014982647266052349,
+            0.0061874424115559,
+            -0.003689464024837091,
+            0.02705902205734107,
+            0.018239469373522537,
+            -0.021070704644859155,
+            0.020037508334843183,
+            -0.00973804771680803,
+            0.018876283979591423,
+            -0.013303080480396818,
+            0.010592424494628194,
+            0.004717870260467274,
+            0.01070131238869639,
+            0.009087656015197565,
+            0.006708673669681232,
+            0.025928065153829043,
+            0.03663367449576964,
+            -0.00511073108390551,
+            0.002161381031392467,
+            0.004864092591944181,
+            0.012814518637357098,
+            0.001556833001747549,
+            0.007915766323215207,
+            -0.007012785514562176,
+            0.005834786334750152,
+            -0.002311533032619821,
+            -0.00122162928193159,
+            0.019525999280692766,
+            0.000824163362963678,
+            0.012875575199417254,
+            0.013259376059185821,
+            -0.002390133687709065,
+            -0.005682644446302278,
+            0.0019262218417782657,
+            -0.018193478045923098,
+            -0.015131696847666434,
+            0.029463723011600933,
+            -0.011447039651934148,
+            -0.008083114389964213,
+            0.0020642857121204753,
+            0.012160202510990566,
+            -0.00866944494225225,
+            -0.005378300635997713,
+            0.0038280364584418335,
+            0.014427318067525318,
+            -0.01799358399064332,
+            0.003381165206832161,
+            0.005252552821854107,
+            0.019433554824232185,
+            -0.00028791985674147885,
+            -0.013487678738391956,
+            0.0005317922815671783,
+            0.014982152801567637,
+            0.011902348165369078,
+            0.0016056390299304837,
+            -0.006935777197137747,
+            -0.00446584737370504,
+            0.004586814866182051,
+            0.006269466956110759,
+            0.0027516492952993445,
+            0.008157662045793032,
+            -0.0025001885993292637,
+            -0.016648747436203294,
+            0.017735757469839637,
+            -0.0030669873770852725,
+            0.018058248627079793,
+            -0.0038238312452176745,
+            -0.0147172470474118,
+            0.0043328417868095355,
+            -0.001888507803054499,
+            -0.018556943283494755,
+            0.009407930200720534,
+            0.002560949424273885,
+            0.010411159836451958,
+            0.0052571230238294825,
+            -0.007499470181994448,
+            -6.145889479758644e-05,
+            -0.015163819825329975,
+            0.022836682800304944,
+            0.016390944816242627,
+            0.001713572907204073,
+            -0.0035687505973373878,
+            -0.031210299824387475,
+            0.0029076511444359886,
+            -0.007822231980296341,
+            0.017821806420465356,
+            0.008250605126658402,
+            0.007908773724009905,
+            0.008351355849444162,
+            -0.00743970642051451,
+            -0.002411981861500092,
+            0.012543476434788293,
+            0.0202407777715137,
+            -0.0024635818448627447,
+            0.02535771413579989,
+            -0.012533449594824386,
+            -0.0022489292647974992,
+            0.0012362917358624392,
+            0.01578939075186126,
+            -0.0008001783766371555,
+            0.006311837812042848
+          ],
+          "keyphrases": [
+            "Switch Transformers",
+            "Mixture of Experts",
+            "sparsely-activated model",
+            "training instability",
+            "multilingual settings"
+          ],
+          "title": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "460c555c-6aea-4625-af79-5ad4ea6dd448",
+        "properties": {
+          "page_content": "C Encouraging Exploration Across Experts 29 ",
+          "keyphrases": [
+            "Encouraging Exploration",
+            "Experts"
+          ],
+          "embedding": [
+            0.029896028339862823,
+            0.007837229408323765,
+            -0.022426677867770195,
+            -0.02060125023126602,
+            -0.0056159826926887035,
+            0.03129303827881813,
+            -0.005033894907683134,
+            0.0729425847530365,
+            -0.001533219125121832,
+            -0.0002334171877009794,
+            -0.04105348512530327,
+            -0.01453822385519743,
+            0.016559232026338577,
+            -0.02227766253054142,
+            -0.02035910077393055,
+            0.05126097798347473,
+            0.010049163363873959,
+            0.008787197060883045,
+            -0.022743333131074905,
+            -0.01933462731540203,
+            0.0470513179898262,
+            -0.040829963982105255,
+            -0.06571538001298904,
+            0.061505720019340515,
+            0.04552391916513443,
+            0.010365818627178669,
+            0.024457000195980072,
+            0.007217888254672289,
+            -0.06351741403341293,
+            0.04153778403997421,
+            0.019949311390519142,
+            0.008465884253382683,
+            -0.008838420733809471,
+            -0.03708597645163536,
+            -0.02613341249525547,
+            -0.024401118978857994,
+            0.018803762272000313,
+            -0.012619663029909134,
+            -0.013113273307681084,
+            0.015842100605368614,
+            -0.007562484126538038,
+            -0.030492085963487625,
+            -0.016037682071328163,
+            0.019148359075188637,
+            0.00031636469066143036,
+            -0.002261993009597063,
+            -0.041463274508714676,
+            -0.036601677536964417,
+            -0.014938700012862682,
+            -0.023302137851715088,
+            -0.02192375436425209,
+            -0.0339939258992672,
+            0.00943913497030735,
+            -0.010915310122072697,
+            0.007962960749864578,
+            0.03365864232182503,
+            0.030436204746365547,
+            -0.008545048534870148,
+            0.03002641536295414,
+            0.04511412978172302,
+            0.022780586034059525,
+            -0.0023202018346637487,
+            -0.02482953481376171,
+            -0.000566662463825196,
+            0.009844268672168255,
+            -0.041463274508714676,
+            -0.01307601947337389,
+            0.011865276843309402,
+            -0.006072339601814747,
+            -0.04101623222231865,
+            -0.014473030343651772,
+            0.010552086867392063,
+            -0.02872253768146038,
+            0.029784265905618668,
+            0.021495336666703224,
+            0.015385744161903858,
+            0.06273508816957474,
+            0.03393804654479027,
+            -0.030249936506152153,
+            -0.0006822068826295435,
+            0.0351860411465168,
+            -0.03609875589609146,
+            0.03326747938990593,
+            0.001204921631142497,
+            0.03285769000649452,
+            -0.01790967583656311,
+            -0.03591248765587807,
+            -0.08165992796421051,
+            0.052415840327739716,
+            0.0031362888403236866,
+            -0.02225903607904911,
+            -0.0011286681983619928,
+            0.003944226540625095,
+            -0.010617280378937721,
+            -0.05465105548501015,
+            0.026040278375148773,
+            0.008135258220136166,
+            0.0008545048185624182,
+            0.0029989161994308233,
+            -0.013578942976891994,
+            0.02455013431608677,
+            0.009993283078074455,
+            -0.015963174402713776,
+            -0.022668825462460518,
+            0.030845994129776955,
+            0.0056159826926887035,
+            0.01548819150775671,
+            0.020508116111159325,
+            -0.011343725956976414,
+            0.08039330691099167,
+            -0.027902958914637566,
+            0.005285356659442186,
+            -0.017024902626872063,
+            0.021420828998088837,
+            -0.013923538848757744,
+            -0.02676672302186489,
+            0.000443550874479115,
+            -0.04485335573554039,
+            0.021718857809901237,
+            0.00806540809571743,
+            -0.01486419327557087,
+            0.01780722849071026,
+            0.01503183413296938,
+            -0.029467610642313957,
+            -0.031423427164554596,
+            0.023767806589603424,
+            -0.010598653927445412,
+            0.014119120314717293,
+            0.020526742562651634,
+            0.04522589221596718,
+            0.030752860009670258,
+            0.0222404096275568,
+            -0.004028047434985638,
+            0.024419745430350304,
+            0.041798558086156845,
+            0.027213767170906067,
+            0.046678781509399414,
+            0.02423347719013691,
+            -0.02546284720301628,
+            -0.002768409438431263,
+            -0.03900453820824623,
+            0.018803762272000313,
+            -0.0029407073743641376,
+            0.01745332032442093,
+            -0.010216804221272469,
+            -0.00982564128935337,
+            0.024997176602482796,
+            0.04455532506108284,
+            -0.04906301200389862,
+            0.01417500153183937,
+            0.008279616013169289,
+            0.04973357915878296,
+            0.014910760335624218,
+            -0.009583492763340473,
+            0.03220575302839279,
+            -0.001704353024251759,
+            -0.014454402960836887,
+            0.0264873206615448,
+            -0.020545369014143944,
+            -0.021886499598622322,
+            0.013467182405292988,
+            -0.008177168667316437,
+            0.005229476373642683,
+            0.012498588301241398,
+            0.011893217451870441,
+            0.02715788595378399,
+            0.0011496233055368066,
+            -0.016913142055273056,
+            -0.07972273975610733,
+            0.04734934866428375,
+            0.0036461977288126945,
+            -0.018338093534111977,
+            0.04943554848432541,
+            0.002908110385760665,
+            0.0007311022491194308,
+            -0.020210087299346924,
+            -0.0017765318043529987,
+            -0.0043516880832612514,
+            -0.00693382928147912,
+            0.015860727056860924,
+            -0.04433180391788483,
+            -0.0029942593537271023,
+            0.022426677867770195,
+            -0.013141212984919548,
+            0.010701101273298264,
+            0.005406430922448635,
+            0.08411866426467896,
+            -0.016335710883140564,
+            0.028163734823465347,
+            0.017742035910487175,
+            0.015246042981743813,
+            -0.031516559422016144,
+            0.003408706048503518,
+            0.018328780308365822,
+            0.013755897991359234,
+            -0.011129518039524555,
+            0.009816328063607216,
+            0.03481350466609001,
+            0.0048429700545966625,
+            -0.025369713082909584,
+            -0.01563720591366291,
+            -0.03447822108864784,
+            0.01683863438665867,
+            0.000613520503975451,
+            0.01678275503218174,
+            0.024307984858751297,
+            -0.009956029243767262,
+            0.02646869421005249,
+            -0.0024750372394919395,
+            -0.01634502410888672,
+            0.016717560589313507,
+            -0.01307601947337389,
+            0.009047972038388252,
+            -0.013914225623011589,
+            -0.005001298151910305,
+            -0.003718376625329256,
+            -0.018487107008695602,
+            0.02516481839120388,
+            -0.0036718095652759075,
+            -0.012265753000974655,
+            0.015702400356531143,
+            0.023972703143954277,
+            0.026226546615362167,
+            -0.014612731523811817,
+            -0.01563720591366291,
+            0.006700994446873665,
+            0.04414553567767143,
+            0.009392567910254002,
+            0.021178681403398514,
+            -0.01552544441074133,
+            -0.005643922835588455,
+            0.03689970821142197,
+            -0.017667528241872787,
+            0.016400905326008797,
+            -0.0005166028859093785,
+            -0.016242576763033867,
+            0.018291525542736053,
+            -0.003939569927752018,
+            0.006426248699426651,
+            0.03639678284525871,
+            0.0035018399357795715,
+            -0.024792281910777092,
+            -0.030417578294873238,
+            0.028312748298048973,
+            0.026617709547281265,
+            0.004137479700148106,
+            -0.027660811319947243,
+            0.04354947805404663,
+            0.026375560089945793,
+            0.014333329163491726,
+            -0.02065712958574295,
+            -0.02587263658642769,
+            -0.010803548619151115,
+            0.011874590069055557,
+            -0.010654534213244915,
+            0.040867216885089874,
+            0.03283906355500221,
+            -0.00010513960296520963,
+            -0.01047758013010025,
+            0.004083927720785141,
+            0.021551217883825302,
+            0.01868268847465515,
+            0.010281998664140701,
+            0.0032131243497133255,
+            -0.01935325376689434,
+            -0.04202207922935486,
+            0.024922668933868408,
+            -0.023767806589603424,
+            0.00702230678871274,
+            -0.03948883339762688,
+            0.006477472838014364,
+            -0.010216804221272469,
+            0.009434478357434273,
+            -0.034385088831186295,
+            0.045337650924921036,
+            0.016754813492298126,
+            -0.006887262221425772,
+            -0.0130946459248662,
+            -0.011800083331763744,
+            0.030175428837537766,
+            -0.002717185765504837,
+            0.02034047432243824,
+            0.027847077697515488,
+            -0.013411302119493484,
+            -0.01632639765739441,
+            0.0054855952039361,
+            -0.052490346133708954,
+            0.02069438435137272,
+            -0.033211600035429,
+            0.009224927052855492,
+            -0.00036293172161094844,
+            -0.04641800746321678,
+            -0.022147275507450104,
+            0.028461763635277748,
+            0.037272244691848755,
+            -0.012424081563949585,
+            -0.016894515603780746,
+            0.004402911756187677,
+            -0.008880331180989742,
+            0.003743988461792469,
+            -0.005853474605828524,
+            -0.02447562664747238,
+            -0.03188909590244293,
+            -0.0067801582626998425,
+            -0.026207920163869858,
+            0.02844313718378544,
+            -0.011902530677616596,
+            0.0119584109634161,
+            -0.03704872354865074,
+            -0.02712063305079937,
+            0.022445304319262505,
+            0.005662549752742052,
+            -0.020470863208174706,
+            -0.04869047552347183,
+            -0.057966627180576324,
+            0.04004763811826706,
+            0.06381544470787048,
+            0.025630488991737366,
+            -0.00016007413796614856,
+            -0.027660811319947243,
+            0.02641281485557556,
+            -0.014938700012862682,
+            -0.037980061024427414,
+            0.01676412671804428,
+            0.011828023009002209,
+            0.007939676754176617,
+            0.01905522495508194,
+            0.028536271303892136,
+            -0.001995396800339222,
+            -0.004321419633924961,
+            -0.031237157061696053,
+            -0.04362398386001587,
+            -0.029467610642313957,
+            -0.02093653194606304,
+            -0.01437058299779892,
+            -0.03574484586715698,
+            -0.01940913498401642,
+            0.024047208949923515,
+            0.005806907545775175,
+            0.005862787831574678,
+            -0.02546284720301628,
+            0.012032918632030487,
+            -0.010980503633618355,
+            0.0053784907795488834,
+            0.010896682739257812,
+            0.0053272671066224575,
+            0.017797915264964104,
+            -0.0036834513302892447,
+            -0.00501061137765646,
+            -0.023786434903740883,
+            0.0012631304562091827,
+            0.011082950979471207,
+            0.0033947357442229986,
+            -0.0089455246925354,
+            0.024270731955766678,
+            0.01614944264292717,
+            -0.03513016179203987,
+            -0.013644137419760227,
+            0.03809182345867157,
+            0.0029290656093508005,
+            0.030827367678284645,
+            -0.07957372814416885,
+            -0.016969023272395134,
+            0.004898850806057453,
+            -0.03652717173099518,
+            -0.0075019472278654575,
+            0.005606669466942549,
+            0.019800297915935516,
+            -0.010570713318884373,
+            -0.0037649436853826046,
+            -0.027902958914637566,
+            -0.009215613827109337,
+            0.02356291189789772,
+            0.011064324527978897,
+            -0.02166297845542431,
+            0.013811778277158737,
+            0.00014312082203105092,
+            -0.01903659850358963,
+            -0.02933722361922264,
+            0.014584790915250778,
+            -0.03540956228971481,
+            -0.002244530478492379,
+            0.009881521575152874,
+            0.000486043281853199,
+            -0.016205323860049248,
+            0.05748233199119568,
+            0.023767806589603424,
+            0.018496420234441757,
+            -2.3683694962528534e-05,
+            0.0024890073109418154,
+            -0.02097378671169281,
+            -0.011734888888895512,
+            -0.05547063425183296,
+            0.002099008532240987,
+            0.02326488308608532,
+            -0.010328564792871475,
+            -0.023395271971821785,
+            -0.018971404060721397,
+            0.01658717356622219,
+            -0.015124968253076077,
+            -0.024699147790670395,
+            0.03514878824353218,
+            -0.010123670101165771,
+            -0.01340198889374733,
+            0.05174527317285538,
+            0.01275005005300045,
+            -0.04347497224807739,
+            -0.00793502014130354,
+            -0.0371604822576046,
+            0.04068094864487648,
+            -0.024028582498431206,
+            -0.03680657222867012,
+            0.0004083345702383667,
+            0.0027055440004915,
+            0.054017744958400726,
+            -0.005108402110636234,
+            0.04623173922300339,
+            0.023991329595446587,
+            0.0029919310472905636,
+            0.004945417400449514,
+            0.05032963678240776,
+            0.011092264205217361,
+            0.051149215549230576,
+            0.01966990903019905,
+            -0.025630488991737366,
+            0.002516947453841567,
+            -0.03645266219973564,
+            0.0035693622194230556,
+            -0.021551217883825302,
+            -0.043288704007864,
+            -0.004991984460502863,
+            0.019204240292310715,
+            -0.01682000793516636,
+            -0.005117715802043676,
+            0.0472375862300396,
+            -0.046678781509399414,
+            -0.03624776750802994,
+            0.01909247785806656,
+            -0.013001511804759502,
+            -0.03159106522798538,
+            0.0020047102589160204,
+            -0.0069245160557329655,
+            -0.031795963644981384,
+            -0.032168496400117874,
+            -5.2824463637080044e-05,
+            0.014230881817638874,
+            -0.029858773574233055,
+            -0.0166058000177145,
+            -0.04515138268470764,
+            0.01451959740370512,
+            -0.036992840468883514,
+            0.03580072522163391,
+            -0.03526054695248604,
+            0.0003108348755631596,
+            0.027847077697515488,
+            -0.009145762771368027,
+            -0.03125578537583351,
+            0.027958840131759644,
+            0.0016065621748566628,
+            -0.01836603321135044,
+            0.010095730423927307,
+            -0.0018673375016078353,
+            0.03453410416841507,
+            -0.030138175934553146,
+            0.0003000662545673549,
+            0.034422341734170914,
+            -0.03295082226395607,
+            -0.02393544837832451,
+            0.01079423539340496,
+            0.05971754714846611,
+            0.008088691160082817,
+            -0.027008872479200363,
+            -0.012358887121081352,
+            0.014808312989771366,
+            -0.009499671868979931,
+            0.009304090403020382,
+            0.020843397825956345,
+            -0.03749576583504677,
+            -0.022426677867770195,
+            -0.007134067825973034,
+            0.00888498779386282,
+            -0.007981587201356888,
+            -0.0022526795510202646,
+            0.036266397684812546,
+            0.017173917964100838,
+            6.915930134709924e-05,
+            0.03369589522480965,
+            -0.020899279043078423,
+            -0.019297374412417412,
+            -0.004482075572013855,
+            0.00969525333493948,
+            -0.017304304987192154,
+            0.00484762666746974,
+            -0.006649770773947239,
+            -0.003913958091288805,
+            -0.04030841216444969,
+            -0.03494389355182648,
+            -0.03477625176310539,
+            0.007203917950391769,
+            0.07160145044326782,
+            -0.009779074229300022,
+            -0.0028918120078742504,
+            0.018058691173791885,
+            0.01743469201028347,
+            -0.001707845483906567,
+            -0.021532591432332993,
+            -0.03198223188519478,
+            -0.028331374749541283,
+            0.008670778945088387,
+            0.00515962578356266,
+            -0.007688215002417564,
+            0.04485335573554039,
+            0.021867873147130013,
+            0.004153778310865164,
+            -0.015395057387650013,
+            0.024736400693655014,
+            -0.024959923699498177,
+            0.03520466759800911,
+            -0.00905262865126133,
+            0.0023958731908351183,
+            -0.0019977251067757607,
+            0.000827728770673275,
+            -0.0044378372840583324,
+            0.03593111410737038,
+            -0.00347389979287982,
+            0.0261892918497324,
+            0.014603417366743088,
+            -0.005308640189468861,
+            0.03168420121073723,
+            -0.0626605823636055,
+            -0.0012316977372393012,
+            0.02004244551062584,
+            0.0047498359344899654,
+            0.001548353466205299,
+            -0.004866253584623337,
+            -0.004782433155924082,
+            0.01258240919560194,
+            0.02354428544640541,
+            -0.003101363545283675,
+            -0.034999772906303406,
+            -0.02067575789988041,
+            0.048876743763685226,
+            0.003289960091933608,
+            0.0009470568038523197,
+            0.02069438435137272,
+            0.009229583665728569,
+            0.03464586287736893,
+            -0.01519016269594431,
+            0.026040278375148773,
+            0.01708078384399414,
+            0.009383254684507847,
+            0.008572988212108612,
+            0.015721026808023453,
+            0.011399607174098492,
+            -0.000368752604117617,
+            -0.026096157729625702,
+            0.018990030512213707,
+            0.01870131492614746,
+            0.0015646519605070353,
+            -0.04675328731536865,
+            0.014975953847169876,
+            0.014510284177958965,
+            -0.0008236541762016714,
+            0.012386827729642391,
+            -0.01875719614326954,
+            0.003408706048503518,
+            -0.011725575663149357,
+            0.016121502965688705,
+            -0.00448673265054822,
+            -0.015003894455730915,
+            -0.00463574705645442,
+            -0.0009453105158172548,
+            -0.017034215852618217,
+            -0.010421698912978172,
+            -0.003874376183375716,
+            0.01773272082209587,
+            0.039190806448459625,
+            -0.045039623975753784,
+            -0.010291311889886856,
+            -0.015553385019302368,
+            -0.011986351571977139,
+            -0.0326155424118042,
+            -0.001544860890135169,
+            -0.00967662688344717,
+            0.006808098405599594,
+            -0.005546132102608681,
+            -0.013523062691092491,
+            -0.04172405228018761,
+            -0.011408920399844646,
+            -0.027344154193997383,
+            -0.013737271539866924,
+            0.02941173128783703,
+            0.031181277707219124,
+            0.0015390400076285005,
+            0.009373941458761692,
+            0.010309938341379166,
+            0.009639373049139977,
+            -0.004787089768797159,
+            -0.015711713582277298,
+            -0.028461763635277748,
+            0.019446387887001038,
+            0.015097028575837612,
+            0.032168496400117874,
+            -0.01337404828518629,
+            0.0002654320269357413,
+            0.019949311390519142,
+            -0.002558857901021838,
+            -0.010729041881859303,
+            -0.0014517268864437938,
+            0.01098981685936451,
+            -0.02028459496796131,
+            -0.004447150509804487,
+            0.03229888528585434,
+            0.0009418179979547858,
+            0.01273142360150814,
+            0.03349100053310394,
+            0.004018733743578196,
+            0.02415897138416767,
+            0.005392461083829403,
+            0.018533675000071526,
+            0.029635252431035042,
+            -0.017816541716456413,
+            0.009220270439982414,
+            0.007990900427103043,
+            0.030175428837537766,
+            0.009169046767055988,
+            -0.014575477689504623,
+            -0.0339939258992672,
+            -0.02004244551062584,
+            0.0001397010637447238,
+            0.03136754408478737,
+            0.0005989682977087796,
+            -0.0033248853869736195,
+            -0.011297158896923065,
+            -0.01771409437060356,
+            0.014612731523811817,
+            0.001167668029665947,
+            0.033360615372657776,
+            0.031162651255726814,
+            -0.01486419327557087,
+            -0.02060125023126602,
+            -0.028275495395064354,
+            0.008647495880723,
+            0.0612821988761425,
+            -0.014240195043385029,
+            -0.011949097737669945,
+            0.029430357739329338,
+            0.015069087967276573,
+            0.020806144922971725,
+            0.003799868980422616,
+            -0.0026450068689882755,
+            0.009909462183713913,
+            0.01965128257870674,
+            0.026617709547281265,
+            -0.0229296013712883,
+            -0.029150955379009247,
+            -0.013653450645506382,
+            -0.012619663029909134,
+            -0.006966426502913237,
+            0.009099195711314678,
+            0.004433180205523968,
+            -0.03811044991016388,
+            0.009117823094129562,
+            0.00342966103926301,
+            -0.012098112143576145,
+            -0.014761745929718018,
+            -0.013588257133960724,
+            0.015804847702383995,
+            -0.01342061534523964,
+            0.02937447652220726,
+            -0.013904912397265434,
+            -0.03028719127178192,
+            -0.013634824194014072,
+            -0.01907385140657425,
+            -0.019763043150305748,
+            -0.034105684608221054,
+            -0.019763043150305748,
+            0.015450937673449516,
+            0.020806144922971725,
+            -0.006165473721921444,
+            -0.017425378784537315,
+            0.0009272657916881144,
+            -0.003071095095947385,
+            0.0020908592268824577,
+            0.028890179470181465,
+            0.009741820394992828,
+            -0.025015803053975105,
+            0.03509290888905525,
+            -0.018198391422629356,
+            0.003418019274249673,
+            -0.011492740362882614,
+            0.022482557222247124,
+            -0.014510284177958965,
+            0.030268562957644463,
+            -0.008018841035664082,
+            -0.006808098405599594,
+            -0.016000429168343544,
+            -0.018179764971137047,
+            0.005909354891628027,
+            0.04008489102125168,
+            -0.009639373049139977,
+            0.008675435557961464,
+            -0.019893432036042213,
+            0.013290227390825748,
+            0.0213463231921196,
+            -0.0006047891802154481,
+            -0.003813839051872492,
+            -0.01159518864005804,
+            -0.024084463715553284,
+            -0.005946608725935221,
+            0.0006787143065594137,
+            0.008852390572428703,
+            -0.0050478652119636536,
+            0.026394188404083252,
+            0.01940913498401642,
+            -0.0019651283510029316,
+            0.01808663085103035,
+            -0.008000214584171772,
+            -0.004433180205523968,
+            0.028610777109861374,
+            -0.002068739850074053,
+            0.0066683972254395485,
+            0.01785379648208618,
+            -0.0118093965575099,
+            -0.022780586034059525,
+            0.0059559219516813755,
+            0.01998656615614891,
+            0.02060125023126602,
+            0.029486237093806267,
+            -0.010570713318884373,
+            0.011129518039524555,
+            -0.014622044749557972,
+            -0.009946716018021107,
+            0.0041235098615288734,
+            0.0004292897356208414,
+            0.01940913498401642,
+            0.010850115679204464,
+            0.022389423102140427,
+            0.0027590959798544645,
+            -0.03609875589609146,
+            -0.002391216577962041,
+            -0.029206834733486176,
+            -0.0017311290139332414,
+            0.0025518727488815784,
+            0.003222437808290124,
+            -0.004312105942517519,
+            0.03416156768798828,
+            -0.011343725956976414,
+            -0.02516481839120388,
+            0.022724706679582596,
+            -0.0038976597134023905,
+            -0.0015809503383934498,
+            0.04656702280044556,
+            -0.030901875346899033,
+            -0.013020139187574387,
+            0.012647602707147598,
+            -0.0008125944877974689,
+            0.012181933037936687,
+            -0.0027102006133645773,
+            0.006118906661868095,
+            0.0175930205732584,
+            0.008503138087689877,
+            0.005401774309575558,
+            0.026375560089945793,
+            0.028331374749541283,
+            0.0010035192826762795,
+            0.009751134552061558,
+            -0.00637968210503459,
+            0.015171535313129425,
+            0.01613081619143486,
+            0.0007421618793159723,
+            -0.0006734755588695407,
+            0.04492786154150963,
+            -0.04392201453447342,
+            0.025891263037919998,
+            -0.024084463715553284,
+            0.018515046685934067,
+            0.005369177553802729,
+            -0.015246042981743813,
+            0.010878056287765503,
+            -0.031814590096473694,
+            -0.011446173302829266,
+            -0.024140343070030212,
+            -0.0009394896333105862,
+            -0.018747882917523384,
+            -0.025313831865787506,
+            0.009993283078074455,
+            0.0009202807559631765,
+            -0.006468159146606922,
+            0.0012631304562091827,
+            -0.008763913065195084,
+            -0.01996793784201145,
+            -0.0021467397455126047,
+            0.010281998664140701,
+            0.029635252431035042,
+            0.01451959740370512,
+            0.007124754134565592,
+            0.008833764120936394,
+            -0.04481610283255577,
+            -0.0322243794798851,
+            0.026729470118880272,
+            0.0029220804572105408,
+            -0.039228059351444244,
+            0.010729041881859303,
+            -0.006836038548499346,
+            -0.01725773885846138,
+            0.03937707468867302,
+            -0.026245173066854477,
+            -0.005704460199922323,
+            0.01733224466443062,
+            0.02393544837832451,
+            -0.008596272207796574,
+            0.0024307984858751297,
+            0.01483625266700983,
+            0.009881521575152874,
+            -0.003434317884966731,
+            0.013271600939333439,
+            -0.013523062691092491,
+            -0.012861811555922031,
+            0.023730553686618805,
+            0.029523491859436035,
+            0.019483640789985657,
+            -0.0024820221588015556,
+            0.0067801582626998425,
+            0.0035903172101825476,
+            0.026394188404083252,
+            0.03619188815355301,
+            -0.02192375436425209,
+            0.03127441182732582,
+            -0.006063025910407305,
+            -0.0355772040784359,
+            0.029430357739329338,
+            -0.0018033079104498029,
+            0.02415897138416767,
+            0.021830620244145393,
+            0.023302137851715088,
+            -0.009061941877007484,
+            -0.015450937673449516,
+            -0.01047758013010025,
+            0.03812907636165619,
+            0.014705865643918514,
+            -0.009583492763340473,
+            -0.015646519139409065,
+            -0.020470863208174706,
+            -0.008512451313436031,
+            0.01909247785806656,
+            -0.015981800854206085,
+            0.026859857141971588,
+            0.02909507416188717,
+            0.016736187040805817,
+            -0.0013271600473672152,
+            0.015469564124941826,
+            -0.02190512605011463,
+            -0.013085332699120045,
+            -0.03423607349395752,
+            -0.006426248699426651,
+            0.0044704340398311615,
+            0.010552086867392063,
+            0.0005579311400651932,
+            -0.006700994446873665,
+            0.02970976009964943,
+            0.01320640742778778,
+            -0.008037467487156391,
+            -0.010831489227712154,
+            0.03168420121073723,
+            -0.01613081619143486,
+            0.00866612233221531,
+            -0.03544681519269943,
+            -0.0072644553147256374,
+            -0.01790967583656311,
+            -0.018012123182415962,
+            0.04753561317920685,
+            -0.0235256589949131,
+            -0.0005762669024989009,
+            0.007483320310711861,
+            -0.008060751482844353,
+            -0.01030062511563301,
+            -0.0185802411288023,
+            0.014901447109878063,
+            0.0013876971788704395,
+            0.0027730660513043404,
+            -0.02522069774568081,
+            -0.0026822604704648256,
+            0.0009488030336797237,
+            -0.049584563821554184,
+            0.004912820644676685,
+            -0.03561445698142052,
+            -0.027251020073890686,
+            -0.00028114838642068207,
+            -0.056327469646930695,
+            0.02132769487798214,
+            -0.019893432036042213,
+            -0.0009668477578088641,
+            -0.025239326059818268,
+            0.012880438007414341,
+            0.01836603321135044,
+            -0.04712582379579544,
+            -0.012275067158043385,
+            -0.011902530677616596,
+            0.011557934805750847,
+            -0.010775608941912651,
+            -0.02553735487163067,
+            0.02484816312789917,
+            -0.038855522871017456,
+            0.007175977807492018,
+            -0.00043103599455207586,
+            0.017211170867085457,
+            0.007613707799464464,
+            0.02259431779384613,
+            0.00892689824104309,
+            0.012275067158043385,
+            -0.011651068925857544,
+            -0.0036881081759929657,
+            -0.03513016179203987,
+            0.004586851689964533,
+            0.008982778526842594,
+            0.02032184787094593,
+            -0.021048294380307198,
+            0.02199826017022133,
+            0.03231751173734665,
+            0.022203154861927032,
+            0.02747454307973385,
+            0.0038347942754626274,
+            0.022519811987876892,
+            0.008107318542897701,
+            -0.03233613818883896,
+            0.017695467919111252,
+            -0.006333115044981241,
+            -0.005192223004996777,
+            -0.020135579630732536,
+            -0.018608180806040764,
+            0.04556117206811905,
+            0.025276578962802887,
+            0.020116953179240227,
+            0.03296945244073868,
+            -0.03226163238286972,
+            -0.014780372381210327,
+            -0.016056308522820473,
+            -0.007231858558952808,
+            -0.013560316525399685,
+            -0.02747454307973385,
+            0.0016461441991850734,
+            -0.011893217451870441,
+            0.007883796468377113,
+            0.005290013737976551,
+            0.02000519260764122,
+            0.012526528909802437,
+            0.042729899287223816,
+            0.013979420065879822,
+            0.0056765200570225716,
+            0.013178466819226742,
+            0.0038813611026853323,
+            -0.009429821744561195,
+            0.03349100053310394,
+            -0.004423866979777813,
+            0.013495123013854027,
+            -0.01972579024732113,
+            0.03585660457611084,
+            0.0030361698009073734,
+            0.01081286184489727,
+            0.01468723826110363,
+            0.00033499152050353587,
+            -0.0336027629673481,
+            -0.005057178437709808,
+            0.027008872479200363,
+            0.036601677536964417,
+            0.012293693609535694,
+            -0.00782791618257761,
+            -0.02455013431608677,
+            0.018077317625284195,
+            -0.006226010620594025,
+            -0.0016985321417450905,
+            0.034105684608221054,
+            0.025090310722589493,
+            0.03103226236999035,
+            -0.014184314757585526,
+            0.0037719286046922207,
+            -0.02000519260764122,
+            0.008018841035664082,
+            0.019781669601798058,
+            0.036955587565898895,
+            -0.01081286184489727,
+            -0.012489275075495243,
+            -0.0006239980575628579,
+            0.03391941636800766,
+            -0.039898622781038284,
+            0.01355100329965353,
+            0.026524575427174568,
+            -0.02553735487163067,
+            0.019856177270412445,
+            -0.007311022374778986,
+            -0.034068431705236435,
+            0.010393759235739708,
+            -0.013774524442851543,
+            0.005350550636649132,
+            0.03255965933203697,
+            0.0006839531124569476,
+            -0.032131243497133255,
+            0.00471025425940752,
+            -0.024661894887685776,
+            -0.019241493195295334,
+            -0.007227201480418444,
+            -0.0002667417284101248,
+            0.03233613818883896,
+            -0.02844313718378544,
+            0.028461763635277748,
+            0.029840147122740746,
+            0.027549048885703087,
+            -0.008805823512375355,
+            0.010356505401432514,
+            -0.0261892918497324,
+            0.037570271641016006,
+            0.011660382151603699,
+            -0.011651068925857544,
+            0.013942166231572628,
+            0.009993283078074455,
+            -0.013513749465346336,
+            0.00011066943261539564,
+            -0.01421225443482399,
+            -0.01320640742778778,
+            -0.02423347719013691,
+            0.002894140314310789,
+            -0.03065972588956356,
+            -0.026897111907601357,
+            0.014500970020890236,
+            0.020824771374464035,
+            -0.004544941242784262,
+            0.00868940632790327,
+            0.00935997162014246,
+            -0.006076996214687824,
+            0.021066920831799507,
+            -0.00177769607398659,
+            0.01084080245345831,
+            -0.012368200346827507,
+            -0.001345786964520812,
+            -0.003157244063913822,
+            0.01854298822581768,
+            -0.07778555154800415,
+            -0.02484816312789917,
+            0.05010611563920975,
+            0.032373394817113876,
+            -0.014789685606956482,
+            -0.002556529361754656,
+            0.010822176001966,
+            -0.0005678266170434654,
+            0.018533675000071526,
+            0.004435508977621794,
+            0.015218102373182774,
+            -0.010961877182126045,
+            -0.011353040114045143,
+            -0.01113883126527071,
+            0.028238240629434586,
+            -0.002343485364690423,
+            -0.01275936421006918,
+            -0.02417759783565998,
+            0.02006107196211815,
+            0.0112692192196846,
+            -0.00757179781794548,
+            -0.0006111921393312514,
+            0.0031619006767868996,
+            -0.01307601947337389,
+            0.00035681980079971254,
+            -0.009145762771368027,
+            -0.01965128257870674,
+            0.019064538180828094,
+            0.025276578962802887,
+            0.009844268672168255,
+            0.01725773885846138,
+            0.029281342402100563,
+            0.008372750133275986,
+            0.01486419327557087,
+            0.005746370181441307,
+            0.007315678987652063,
+            -0.009844268672168255,
+            -0.01047758013010025,
+            0.008274959400296211,
+            0.0032131243497133255,
+            -0.014780372381210327,
+            -0.004163091536611319,
+            0.019614029675722122,
+            0.011399607174098492,
+            0.00034576014149934053,
+            0.009215613827109337,
+            0.022110020741820335,
+            0.013234347105026245,
+            -0.01804937794804573,
+            0.011865276843309402,
+            0.010542773641645908,
+            0.03581935167312622,
+            -0.00903400219976902,
+            -0.02877841889858246,
+            -0.006938486360013485,
+            0.021551217883825302,
+            0.004018733743578196,
+            -0.007418126333504915,
+            0.015683772042393684,
+            -0.01616806909441948,
+            0.07208574563264847,
+            0.0012747722212225199,
+            0.015786219388246536,
+            -0.027027498930692673,
+            -0.014789685606956482,
+            0.007497290149331093,
+            -0.014780372381210327,
+            0.021458083763718605,
+            0.002114142756909132,
+            0.026356933638453484,
+            0.014053926803171635,
+            0.014016672968864441,
+            0.0042399270460009575,
+            0.010542773641645908,
+            -0.0030734234023839235,
+            0.014631357975304127,
+            0.0019709491170942783,
+            0.007078187074512243,
+            -0.00109956378582865,
+            0.012200559489428997,
+            0.004558911547064781,
+            -0.0010244745062664151,
+            -0.0025239326059818268,
+            0.010011909529566765,
+            0.00854039192199707,
+            -0.007143381051719189,
+            0.004463449120521545,
+            -0.003844107501208782,
+            -0.030734233558177948,
+            -0.007418126333504915,
+            0.016037682071328163,
+            0.008503138087689877,
+            0.032392021268606186,
+            -0.0036485260352492332,
+            -0.018570927903056145,
+            0.02840588241815567,
+            0.0003201482759322971,
+            0.010216804221272469,
+            0.0182822123169899,
+            0.005313297268003225,
+            -0.027306901291012764,
+            0.03164694830775261,
+            -0.01905522495508194,
+            0.016652366146445274,
+            -0.023413898423314095,
+            -0.014798998832702637,
+            0.004258553963154554,
+            0.0030478115659207106,
+            -0.029448984190821648,
+            -0.005266730207949877,
+            0.01634502410888672,
+            0.024307984858751297,
+            0.00815388560295105,
+            0.009490358643233776,
+            0.010226117447018623,
+            0.004009420517832041,
+            0.013765211217105389,
+            0.01577690616250038,
+            -0.012796617113053799,
+            -0.007921050302684307,
+            0.0012421752326190472,
+            -0.015646519139409065,
+            0.013904912397265434,
+            -0.00281730480492115,
+            -0.0008748779073357582,
+            0.013886285945773125,
+            -0.02265019901096821,
+            -0.010375131852924824,
+            0.010132983326911926,
+            -0.009834954515099525,
+            -0.004193360451608896,
+            0.038222212344408035,
+            -0.006510069593787193,
+            0.008801166899502277,
+            -0.01565583236515522,
+            0.00038184956065379083,
+            0.012954945676028728,
+            -0.004202673677355051,
+            -0.026580454781651497,
+            -0.04381025210022926,
+            0.004917477257549763,
+            0.02905782125890255,
+            0.013001511804759502,
+            0.04056918993592262,
+            0.0019290389027446508,
+            -0.01996793784201145,
+            -0.025704994797706604,
+            0.02838725596666336,
+            -0.00868940632790327,
+            0.0017846811097115278,
+            -0.025555981323122978,
+            0.013560316525399685,
+            0.00861955527216196,
+            0.012833870947360992,
+            -0.007506603840738535,
+            -0.01501320768147707,
+            0.014277448877692223,
+            0.014072554185986519,
+            -0.03220575302839279,
+            0.007539200596511364,
+            0.004200345370918512,
+            0.0038557492662221193,
+            0.011520680971443653,
+            -0.01844054087996483,
+            -0.009974655695259571,
+            0.0004563567927107215,
+            0.0036881081759929657,
+            -0.012787303887307644,
+            0.0006915202829986811,
+            0.015273982658982277,
+            -0.02130906842648983,
+            -0.016540605574846268,
+            -0.01785379648208618,
+            -0.007674245163798332,
+            -0.003962853457778692,
+            0.00456123985350132,
+            0.01455685030668974,
+            -0.013802465051412582,
+            -0.008475197479128838,
+            -0.005136342253535986,
+            -0.004558911547064781,
+            0.02747454307973385,
+            0.006808098405599594,
+            -0.02613341249525547,
+            0.0068127550184726715,
+            0.012200559489428997,
+            -0.0011408920399844646,
+            -0.011082950979471207,
+            0.024065837264060974,
+            0.0008603257010690868,
+            0.003939569927752018,
+            -0.000613520503975451,
+            0.007902423851191998,
+            0.02229628898203373,
+            -0.00866612233221531,
+            0.007851200178265572,
+            0.014435776509344578,
+            0.013737271539866924,
+            0.00621669739484787,
+            -0.014649984426796436,
+            -0.012293693609535694,
+            -0.011362353339791298,
+            0.017071470618247986,
+            0.013485808856785297,
+            -0.012768677435815334,
+            -0.022110020741820335,
+            -0.012628976255655289,
+            -0.0011630113003775477,
+            0.0070688738487660885,
+            0.00862886942923069,
+            -0.008754599839448929,
+            0.023153122514486313,
+            0.020731637254357338,
+            0.004405240062624216,
+            -0.0046147918328642845,
+            -0.013737271539866924,
+            0.030119549483060837,
+            0.0014261150499805808,
+            0.0351860411465168,
+            0.013485808856785297,
+            0.005709116812795401,
+            0.012722110375761986,
+            -0.005541475489735603,
+            -0.018971404060721397,
+            0.029523491859436035,
+            0.0035181385464966297,
+            -0.010067789815366268,
+            0.06199001893401146,
+            0.0020198444835841656,
+            -0.0015972488326951861,
+            -0.008386720903217793,
+            -0.008866360411047935,
+            0.014426463283598423,
+            -0.004414553754031658,
+            0.01834740675985813,
+            -0.0015169207472354174,
+            0.01563720591366291,
+            -0.00870803277939558,
+            -0.006044399458914995,
+            0.0037020782474428415,
+            0.004223628900945187,
+            -0.007054903544485569,
+            0.014808312989771366,
+            0.02201688662171364,
+            0.006901232525706291,
+            -0.036974214017391205,
+            -0.02741866186261177,
+            -0.014249508269131184,
+            -0.037868302315473557,
+            0.014733805321156979,
+            0.008423973806202412,
+            9.793626668397337e-05,
+            0.014277448877692223,
+            -0.035968367010354996,
+            0.012247126549482346,
+            0.0122564397752285,
+            -0.0027614242862910032,
+            -0.0062120407819747925,
+            -0.0004831328405998647,
+            -0.02028459496796131,
+            -0.0011024741688743234,
+            0.01875719614326954,
+            0.013616196811199188,
+            -0.0007695200038142502,
+            0.02132769487798214,
+            -0.00475449301302433,
+            0.014612731523811817,
+            -0.006654427386820316,
+            0.025630488991737366,
+            -0.013830405659973621,
+            -0.017043529078364372,
+            -0.009788387455046177,
+            0.01468723826110363,
+            0.021532591432332993,
+            0.009467075578868389,
+            0.03034307062625885,
+            -0.007981587201356888,
+            0.009411195293068886,
+            -0.013849032111465931,
+            -0.014994580298662186,
+            0.00266130524687469,
+            0.024103090167045593,
+            0.004801060073077679,
+            0.015422997064888477,
+            0.013830405659973621,
+            -0.011772142723202705,
+            0.007939676754176617,
+            -0.007371559273451567,
+            0.0031130053102970123,
+            -0.008731316775083542,
+            -0.005969892255961895,
+            0.0038487643469125032,
+            -0.0006664904649369419,
+            0.012470648624002934,
+            0.002131605288013816,
+            0.018393972888588905,
+            -0.007595081347972155,
+            -0.005578729324042797,
+            -0.021811991930007935,
+            -0.001249160384759307,
+            0.0257981289178133,
+            -0.028554897755384445,
+            -0.02425210364162922,
+            0.016037682071328163,
+            0.0023073959164321423,
+            -0.0003850510693155229,
+            0.01243339478969574,
+            -0.04101623222231865,
+            -0.033360615372657776,
+            -0.0235256589949131,
+            -0.014854880049824715,
+            -0.0001042664734995924,
+            0.015078401193022728,
+            -0.01611218973994255,
+            -0.010486893355846405,
+            0.0219423808157444,
+            -0.015199475921690464,
+            0.024047208949923515,
+            -0.014286762103438377,
+            -0.0028359314892441034,
+            -0.004363330081105232,
+            0.006030429154634476,
+            0.00011692687985487282,
+            0.019893432036042213,
+            -0.019129732623696327,
+            -0.0030757517088204622,
+            -0.02071301080286503,
+            -0.001526234089396894,
+            0.00614219019189477,
+            0.001925546326674521,
+            -0.024587387219071388,
+            0.0016915469896048307,
+            0.020824771374464035,
+            -0.016615113243460655,
+            0.02715788595378399,
+            -0.01909247785806656,
+            0.020862026140093803,
+            -0.009611433371901512,
+            0.011688321828842163,
+            -0.008228392340242863,
+            -0.024307984858751297,
+            0.06638594716787338,
+            -0.0069245160557329655,
+            -0.00215488881804049,
+            -0.016456784680485725,
+            0.0016473083524033427,
+            -0.021066920831799507,
+            0.02581675723195076,
+            0.006896575912833214,
+            0.019427761435508728,
+            -0.014091180637478828,
+            0.026356933638453484,
+            -0.002856886712834239,
+            0.0064402190037071705,
+            0.02741866186261177,
+            -0.0018719942308962345,
+            -0.016792068257927895,
+            0.005313297268003225,
+            0.02291097491979599,
+            -0.020172834396362305,
+            -0.01747194677591324,
+            -0.021197307854890823,
+            -0.003471571486443281,
+            -0.004633418750017881,
+            -0.011259905993938446,
+            -0.002584469737485051,
+            0.002989602740854025,
+            -0.03349100053310394,
+            0.024680521339178085,
+            0.0036275710444897413,
+            -0.005108402110636234,
+            0.004400583449751139,
+            -0.026710843667387962,
+            0.000846355629619211,
+            0.00281730480492115,
+            -0.012694169767200947,
+            0.011716262437403202,
+            0.01646609790623188,
+            0.008447257801890373,
+            0.002136262133717537,
+            0.011846650391817093,
+            0.017015589401125908,
+            -0.017760662361979485,
+            0.008177168667316437,
+            -0.011362353339791298,
+            0.018654748797416687,
+            -0.010701101273298264,
+            -0.01371864415705204,
+            -0.01569308526813984,
+            0.001958143198862672,
+            -0.02060125023126602,
+            -0.004074614495038986,
+            0.00590469827875495,
+            -0.025071684271097183,
+            -0.0032573631033301353,
+            0.019837550818920135,
+            0.005057178437709808,
+            0.020862026140093803,
+            0.00023560001864098012,
+            -0.00982564128935337,
+            -0.031497932970523834,
+            -0.0025448875967413187,
+            -0.021495336666703224,
+            -0.012265753000974655,
+            0.016298457980155945,
+            0.007604394573718309,
+            -0.015497504733502865,
+            -0.005723086651414633,
+            0.001080936985090375,
+            -0.007734782062470913,
+            -0.035968367010354996,
+            0.03721636161208153,
+            0.006267921067774296,
+            0.014296075329184532,
+            -0.03390078991651535,
+            0.005313297268003225,
+            -0.003185184206813574,
+            -0.0005832519382238388,
+            -0.011511367745697498,
+            0.0032177811954170465,
+            -0.015627892687916756,
+            -0.00793502014130354,
+            -0.01939050666987896,
+            -0.009443791583180428,
+            -0.035670336335897446,
+            -0.021402202546596527,
+            -0.00012332983897067606,
+            0.012666230089962482,
+            0.008228392340242863,
+            0.001559995231218636,
+            0.02030322141945362,
+            -0.025555981323122978,
+            0.019465014338493347,
+            0.0015646519605070353,
+            0.02546284720301628,
+            0.006961769424378872,
+            -0.0007718483684584498,
+            0.008582302369177341,
+            -0.029933281242847443,
+            -0.013727957382798195,
+            -0.01755576767027378,
+            -0.0169038288295269,
+            -0.021551217883825302,
+            0.0064728157594799995,
+            -0.004742851015180349,
+            -0.017695467919111252,
+            -0.012638289481401443,
+            -0.004682314116507769,
+            0.004735866095870733,
+            -0.02287372015416622,
+            0.02255706489086151,
+            0.011278532445430756,
+            0.037942808121442795,
+            -0.01587004028260708,
+            0.02032184787094593,
+            0.00797693058848381,
+            -0.006519382819533348,
+            -0.01325297448784113,
+            -0.027213767170906067,
+            0.002933722222223878,
+            -0.0015553385019302368,
+            -0.014333329163491726,
+            0.004628761671483517,
+            0.012116738595068455,
+            0.026207920163869858,
+            0.027604930102825165,
+            -0.0209551602602005,
+            -0.008717346005141735,
+            -0.011371666565537453,
+            -0.006375025026500225,
+            0.0041048829443752766,
+            -0.019856177270412445,
+            -0.0012002650182694197,
+            -0.036340903490781784,
+            -0.00281730480492115,
+            -0.006351741496473551,
+            0.0003335362998768687,
+            0.006719620898365974,
+            -0.004540284629911184,
+            -0.0007153858314268291,
+            -0.00262405164539814,
+            -0.024680521339178085,
+            0.024028582498431206,
+            -0.021066920831799507,
+            -0.0072644553147256374,
+            -0.018496420234441757,
+            0.028629405423998833,
+            0.0025006490759551525,
+            0.005997832398861647,
+            -0.029914654791355133,
+            -0.011399607174098492,
+            -0.012964258901774883,
+            0.0038301374297589064,
+            0.00043481955071911216,
+            -0.00815388560295105,
+            0.03198223188519478,
+            0.008917584083974361,
+            0.004824343603104353,
+            0.012656915932893753,
+            0.020265966653823853,
+            -0.02004244551062584,
+            -0.019539522007107735,
+            -0.034422341734170914,
+            0.02227766253054142,
+            -0.017779288813471794,
+            -0.006370368413627148,
+            -0.002556529361754656,
+            0.0033318703062832355,
+            -0.007348275743424892,
+            -0.013625510036945343,
+            0.010254058055579662,
+            0.007990900427103043,
+            -0.011259905993938446,
+            -0.002887155395001173,
+            -0.009900148957967758,
+            0.002933722222223878,
+            -0.00788845308125019,
+            0.006458845920860767,
+            -0.009476388804614544,
+            0.007092157378792763,
+            0.0063563985750079155,
+            0.003799868980422616,
+            -0.011474113911390305,
+            0.028908805921673775,
+            0.010859428904950619,
+            0.01871994324028492,
+            0.010011909529566765,
+            -0.017462633550167084,
+            0.013262287713587284,
+            -0.006719620898365974,
+            0.026282425969839096,
+            -0.028461763635277748,
+            -0.021048294380307198,
+            -0.012628976255655289,
+            0.008866360411047935,
+            -0.006314488127827644,
+            -0.0033435120712965727,
+            -0.0006222518277354538,
+            0.0019756059627979994,
+            -0.01737881265580654,
+            -0.0012736080680042505,
+            0.02166297845542431,
+            0.006766187958419323,
+            0.007045590318739414,
+            -0.00759042426943779,
+            0.052788376808166504,
+            0.005886071361601353,
+            0.0219423808157444,
+            0.0003335362998768687,
+            0.01998656615614891,
+            -0.015739653259515762,
+            0.009900148957967758,
+            -0.03231751173734665,
+            -0.00177420349791646,
+            0.007520573679357767,
+            -0.007786005735397339,
+            0.0009982804767787457,
+            -0.003443631110712886,
+            0.009769761003553867,
+            0.002465723780915141,
+            -0.020489489659667015,
+            -0.027027498930692673,
+            -0.012954945676028728,
+            0.01650335267186165,
+            0.009038658812642097,
+            -0.011576561257243156,
+            -0.00351115339435637,
+            -0.00550422165542841,
+            0.0026636335533112288,
+            0.003341183764860034,
+            0.028834300115704536,
+            -0.038892775774002075,
+            0.0011263397755101323,
+            -0.00023865597904659808,
+            0.030920501798391342,
+            0.0037416601553559303,
+            -0.016046995297074318,
+            -0.00021944708714727312,
+            0.012470648624002934,
+            0.008526421152055264,
+            0.007236515171825886,
+            -0.0005649161757901311,
+            0.004423866979777813,
+            0.007031620014458895,
+            -0.009499671868979931,
+            0.014808312989771366,
+            -0.015283295884728432,
+            -0.009318061172962189,
+            -0.004035032354295254,
+            -0.007967617362737656,
+            0.003930256702005863,
+            0.005448341369628906,
+            0.0031293039210140705,
+            0.0007840722100809216,
+            -0.0006030429503880441,
+            -0.0012573095737025142,
+            -0.006468159146606922,
+            0.018328780308365822,
+            0.01998656615614891,
+            -0.0185802411288023,
+            0.0016158756334334612,
+            0.025276578962802887,
+            -0.025053057819604874,
+            0.014147060923278332,
+            0.023041361942887306,
+            0.003681123023852706,
+            -0.002554201055318117,
+            0.005648579448461533,
+            0.0007252813666127622,
+            0.010552086867392063,
+            0.007837229408323765,
+            0.01785379648208618,
+            0.010505519807338715,
+            -0.014482343569397926,
+            0.01771409437060356,
+            -0.006198070477694273,
+            -0.0011414741165935993,
+            -0.009243553504347801,
+            0.010784922167658806,
+            -0.002402858342975378,
+            0.004978014621883631,
+            -0.00016021965711843222,
+            -0.00423527043312788,
+            -0.007217888254672289,
+            0.013197094202041626,
+            0.00355539214797318,
+            0.017173917964100838,
+            -0.008023497648537159,
+            -0.0038720478769391775,
+            -0.02199826017022133,
+            -0.020265966653823853,
+            0.0024447685573250055,
+            0.019800297915935516,
+            0.02326488308608532,
+            -0.01666167937219143,
+            0.0074926335364580154,
+            0.008647495880723,
+            0.02265019901096821,
+            -0.008801166899502277,
+            -0.014147060923278332,
+            0.0034133626613765955,
+            -0.011073637753725052,
+            0.004714910872280598,
+            -0.012694169767200947,
+            -0.015301923267543316,
+            0.016689620912075043,
+            0.0212531890720129,
+            0.002812647959217429,
+            0.013709330931305885,
+            0.00645418930798769,
+            -0.004260882269591093,
+            0.019912058487534523,
+            -0.01678275503218174,
+            -0.017993496730923653,
+            -0.005369177553802729,
+            -0.0030920500867068768,
+            -0.00935997162014246,
+            -0.002068739850074053,
+            0.00021711873705498874,
+            0.0261892918497324,
+            0.007450723554939032,
+            0.003045483259484172,
+            -0.008503138087689877,
+            -0.002073396695777774,
+            -0.009224927052855492,
+            -0.027195140719413757,
+            0.008647495880723,
+            -0.007869826629757881,
+            -0.006742904428392649,
+            0.005103745497763157,
+            -0.01706215739250183,
+            -0.019129732623696327,
+            -0.007725468836724758,
+            -0.0022352170199155807,
+            -0.005681176669895649,
+            0.0166058000177145,
+            0.01771409437060356,
+            -0.022184528410434723,
+            -0.006598547101020813,
+            -0.00918301660567522,
+            -0.0012142350897192955,
+            -0.014957327395677567,
+            0.006724277976900339,
+            0.006826725322753191,
+            0.02199826017022133,
+            -0.023656046018004417,
+            -0.008251676335930824,
+            0.0015576668083667755,
+            -0.0015145923243835568,
+            0.02482953481376171,
+            -0.003285303246229887,
+            -0.017229797318577766,
+            -0.02836862951517105,
+            0.011828023009002209,
+            -0.008111975155770779,
+            -0.00614219019189477,
+            -0.019912058487534523,
+            -0.007450723554939032,
+            0.0017648900393396616,
+            -0.013020139187574387,
+            0.0027777228970080614,
+            -0.02712063305079937,
+            0.012377514503896236,
+            -0.010421698912978172,
+            -0.006673054303973913,
+            -0.0030221997294574976,
+            -0.011446173302829266,
+            0.026561828330159187,
+            0.019818924367427826,
+            0.016866575926542282,
+            0.015003894455730915,
+            0.008638182654976845,
+            0.0006239980575628579,
+            -0.00817251205444336,
+            0.003315571928396821,
+            -0.023693300783634186,
+            0.02548147365450859,
+            -0.002238709479570389,
+            -0.02190512605011463,
+            0.00493610417470336,
+            -0.004659030586481094,
+            -0.01079423539340496,
+            -0.000244331342400983,
+            0.012805930338799953,
+            -0.0003213124582543969,
+            -0.0021642022766172886,
+            0.010747668333351612,
+            0.013336794450879097,
+            0.011362353339791298,
+            0.030510712414979935,
+            0.016084248200058937,
+            -0.013513749465346336,
+            -0.02065712958574295,
+            0.011697635985910892,
+            -0.009723193943500519,
+            0.0044448222033679485,
+            0.025015803053975105,
+            0.017704781144857407,
+            -0.0017800243804231286,
+            -0.009047972038388252,
+            -0.0012771005276590586,
+            0.008079377934336662,
+            0.00935065746307373,
+            0.010552086867392063,
+            -0.012004978023469448,
+            -0.009425165131688118,
+            -0.01049620658159256,
+            -0.0027963495813310146,
+            0.001626353245228529,
+            -0.010859428904950619,
+            -0.0028475732542574406,
+            -0.001629845704883337,
+            0.011110891588032246,
+            0.010254058055579662,
+            -0.006319144740700722,
+            -0.054129503667354584,
+            0.007040933705866337,
+            0.010710414499044418,
+            0.009751134552061558,
+            -0.010505519807338715,
+            0.01870131492614746,
+            0.0012503245379775763,
+            -0.0014796671457588673,
+            0.010775608941912651,
+            0.010766295716166496,
+            0.0024936639238148928,
+            0.013495123013854027,
+            0.01583278737962246,
+            0.011092264205217361,
+            0.006840695161372423,
+            0.029951907694339752,
+            -0.021234560757875443,
+            -0.003932585008442402,
+            -0.022668825462460518,
+            -0.03349100053310394,
+            -0.007064217235893011,
+            0.01422156859189272,
+            0.004244584124535322,
+            0.008982778526842594,
+            0.020433608442544937,
+            0.03494389355182648,
+            0.013672077096998692,
+            -0.002324858447536826,
+            -0.002270142314955592,
+            -0.02162572368979454,
+            0.0037207049317657948,
+            -0.0014249508967623115,
+            0.00477311946451664,
+            0.006971083115786314,
+            -0.004305121023207903,
+            0.002726498991250992,
+            0.0090945390984416,
+            -0.0015588310780003667,
+            -0.00982564128935337,
+            -3.0923412850825116e-05,
+            0.010226117447018623,
+            0.01547887735068798,
+            -0.00524810329079628,
+            -0.017481260001659393,
+            0.007129410747438669,
+            0.022668825462460518,
+            -0.004165419843047857,
+            -0.004298136103898287,
+            -0.006840695161372423,
+            -0.013904912397265434,
+            -0.0024191567208617926,
+            -0.025630488991737366,
+            0.021811991930007935,
+            -0.012982885353267193,
+            0.017648901790380478,
+            0.01664305292069912,
+            -0.009313404560089111,
+            0.00016051069542299956,
+            -0.0027078723069280386,
+            0.03613600879907608,
+            0.0050478652119636536,
+            0.027008872479200363,
+            0.01692245528101921,
+            0.025239326059818268,
+            0.008722002618014812,
+            0.015786219388246536,
+            0.011883903294801712,
+            -0.004025719128549099,
+            0.00954623892903328,
+            0.011101577430963516,
+            0.008479854092001915,
+            -0.0032759900204837322,
+            0.013085332699120045,
+            0.0017660543089732528,
+            -0.018887583166360855,
+            -0.013904912397265434,
+            0.002901125466451049,
+            0.010654534213244915,
+            0.006738247815519571,
+            0.01840328611433506,
+            -0.008260989561676979,
+            -0.01723911054432392,
+            -0.0035088250879198313,
+            0.011697635985910892,
+            0.024661894887685776,
+            -0.01422156859189272,
+            -0.009620746597647667,
+            -0.02747454307973385,
+            0.0028662001714110374,
+            -0.0023341719061136246,
+            0.008503138087689877,
+            -0.0019860833417624235,
+            0.012163305655121803,
+            -0.00026732380501925945,
+            -0.015152908861637115,
+            -0.008614898659288883,
+            0.020396355539560318,
+            -0.026971617713570595,
+            0.017518512904644012,
+            0.0008731316193006933,
+            -0.01483625266700983,
+            0.017406752333045006,
+            0.0021502322051674128,
+            0.014249508269131184,
+            0.007902423851191998,
+            0.0070688738487660885,
+            -0.01245202124118805,
+            0.010207490995526314,
+            -0.024345237761735916,
+            0.010281998664140701,
+            0.011073637753725052,
+            0.0017159946728497744,
+            0.018868956714868546,
+            0.01356962975114584,
+            0.004668343812227249,
+            -0.019427761435508728,
+            0.009066599421203136,
+            -0.02544422075152397,
+            0.013271600939333439,
+            -0.001437756814993918,
+            0.024792281910777092,
+            -0.008600928820669651,
+            -0.0006106100627221167,
+            0.006058369297534227,
+            0.014910760335624218,
+            -0.003485541557893157,
+            -0.0020338145550340414,
+            0.006999023258686066,
+            -0.0006164309452287853,
+            -0.0017707109218463302,
+            0.0005800504586659372,
+            -0.019483640789985657,
+            -0.02294822782278061,
+            -0.0005075805238448083,
+            -0.009089882485568523,
+            0.01613081619143486,
+            -0.02745591476559639,
+            0.001415637438185513,
+            -0.00733430590480566,
+            0.003969838377088308,
+            -0.00020416728511918336,
+            -0.017145976424217224,
+            0.012209872715175152,
+            -0.005266730207949877,
+            0.0016321741277351975,
+            0.023134496062994003,
+            0.03362138941884041,
+            0.00048517013783566654,
+            -0.016223950311541557,
+            0.0175930205732584,
+            0.0066963378340005875,
+            -0.013392675668001175,
+            -0.020768892019987106,
+            -0.01049620658159256,
+            -0.017294991761446,
+            -0.009010718204081059,
+            0.0025518727488815784,
+            0.003837122581899166,
+            -0.0069571128115057945,
+            -0.010803548619151115,
+            0.010682474821805954,
+            -0.013755897991359234,
+            -0.02294822782278061,
+            0.00965800043195486,
+            -0.0012247127015143633,
+            0.016140129417181015,
+            0.0010663848370313644,
+            -0.004260882269591093,
+            0.019632656127214432,
+            -0.01664305292069912,
+            0.009741820394992828,
+            0.017499886453151703,
+            0.02296685427427292,
+            -0.026654962450265884,
+            -0.00935997162014246,
+            -0.012610348872840405,
+            -0.013625510036945343,
+            -0.0008585794712416828,
+            0.02740003541111946,
+            0.012172618880867958,
+            0.004596164915710688,
+            0.015450937673449516,
+            -0.0038417791947722435,
+            0.009089882485568523,
+            -0.025425594300031662,
+            0.016754813492298126,
+            -0.0118093965575099,
+            0.02615203894674778,
+            0.016223950311541557,
+            0.010077103041112423,
+            0.020806144922971725,
+            -0.0028149764984846115,
+            0.0018452182412147522,
+            -0.007380872964859009,
+            0.01773272082209587,
+            0.014715178869664669,
+            0.028312748298048973,
+            -0.011446173302829266,
+            0.016773441806435585,
+            0.005648579448461533,
+            0.005331923719495535,
+            -0.002933722222223878,
+            0.018794449046254158,
+            -0.0038464360404759645,
+            0.002593782963231206,
+            0.011297158896923065,
+            0.00281730480492115,
+            -0.01998656615614891,
+            -0.0023155452217906713,
+            0.012852497398853302,
+            0.0004173569323029369,
+            0.0030315129552036524,
+            0.0005468714516609907,
+            0.010952563025057316,
+            -0.0073575894348323345,
+            -0.018766509369015694,
+            0.017229797318577766,
+            0.016438158228993416,
+            -0.014128434471786022,
+            -0.026524575427174568,
+            -0.007324992213398218,
+            -0.000489535799715668,
+            0.021439457312226295,
+            0.0012293693143874407,
+            0.0047335377894341946,
+            0.014910760335624218,
+            0.010244744829833508,
+            0.003844107501208782,
+            -0.003769600298255682,
+            0.0009255195618607104,
+            -0.018459167331457138,
+            -0.003918614704161882,
+            0.01163244154304266,
+            -0.008330839686095715,
+            -1.7971959096030332e-05,
+            -0.0029593342915177345,
+            -0.008978121913969517,
+            -0.009965342469513416,
+            0.016717560589313507,
+            0.002754439366981387,
+            0.007525230757892132,
+            0.01193047035485506,
+            0.040196653455495834,
+            -0.007157351355999708,
+            -0.004172405228018761,
+            0.024624640122056007,
+            0.004095569718629122,
+            -0.001629845704883337,
+            0.0219423808157444,
+            -0.005178252700716257,
+            -0.01711803674697876,
+            0.0021607098169624805,
+            0.01842191442847252,
+            0.007324992213398218,
+            -0.007082843687385321,
+            0.012181933037936687,
+            -0.014780372381210327,
+            0.020824771374464035,
+            0.004991984460502863,
+            -0.009974655695259571,
+            -0.00937859807163477,
+            0.005313297268003225,
+            0.007939676754176617,
+            0.0043866136111319065,
+            0.024065837264060974,
+            0.019064538180828094,
+            -0.02032184787094593,
+            0.007166664581745863,
+            0.009807014837861061,
+            0.017034215852618217,
+            -0.02097378671169281,
+            -0.001526234089396894,
+            0.009816328063607216,
+            0.01552544441074133,
+            0.0025704996660351753,
+            0.003152587218210101,
+            -0.02004244551062584,
+            -0.005313297268003225,
+            -0.01940913498401642,
+            -0.016810694709420204,
+            -0.016140129417181015,
+            -0.006337771657854319,
+            0.00517359608784318,
+            0.01585141383111477,
+            -0.01646609790623188,
+            0.0033877508249133825,
+            0.0028103196527808905,
+            -0.024047208949923515,
+            -0.01404461357742548,
+            0.0029360507614910603,
+            0.002756767673417926,
+            -0.00086672866018489,
+            -0.0002302157081430778,
+            -0.007529887370765209,
+            -0.018161138519644737,
+            0.016279829666018486,
+            -0.012992198579013348,
+            -0.01404461357742548,
+            0.004889537114650011,
+            0.01402598712593317,
+            0.005993175785988569,
+            -0.0066683972254395485,
+            0.0029057820793241262,
+            -0.015022520907223225,
+            -0.009983968921005726,
+            0.012563781812787056,
+            0.0004697448166552931,
+            -0.02553735487163067,
+            0.00868940632790327,
+            0.02231491729617119,
+            -0.00725979870185256,
+            0.002270142314955592,
+            -0.003681123023852706,
+            -0.01179077010601759,
+            -0.02287372015416622,
+            0.0077813491225242615,
+            -0.005653236526995897,
+            -0.013681390322744846,
+            -0.0002440402895445004,
+            0.03188909590244293,
+            -0.012163305655121803,
+            -0.002957005752250552,
+            0.009723193943500519,
+            -0.0010244745062664151,
+            -0.009862895123660564,
+            -0.0017241439782083035,
+            -0.01826358586549759,
+            0.004225957207381725,
+            0.0022573363967239857,
+            -0.004682314116507769,
+            0.008028154261410236,
+            -0.0109711904078722,
+            -0.018626809120178223,
+            0.01340198889374733,
+            -0.007674245163798332,
+            0.002428470179438591,
+            -0.007511260453611612,
+            -0.006575263570994139,
+            -0.012172618880867958,
+            -0.03466448932886124,
+            0.013299541547894478,
+            -0.012470648624002934,
+            -0.02127181552350521,
+            -0.014584790915250778,
+            -0.005695146508514881,
+            0.014463717117905617,
+            0.004328404553234577,
+            -0.03485075756907463,
+            -0.018077317625284195,
+            0.00970456749200821,
+            0.015441624447703362,
+            -0.009751134552061558,
+            -0.0132436603307724,
+            0.011343725956976414,
+            0.038930028676986694,
+            0.0026776036247611046,
+            -0.007436753250658512,
+            0.000422304670792073,
+            0.01051483303308487,
+            -0.011697635985910892,
+            -0.004032704047858715,
+            0.021849246695637703,
+            0.010086417198181152,
+            -0.0056672063656151295,
+            -0.006193413864821196,
+            0.027828451246023178,
+            0.00942050851881504,
+            0.018338093534111977,
+            -0.006579920183867216,
+            -0.007641647942364216,
+            -0.014761745929718018,
+            -0.004277180880308151,
+            -0.015087714418768883,
+            -0.02097378671169281,
+            0.009834954515099525,
+            0.009862895123660564,
+            -0.011064324527978897,
+            0.006682367529720068,
+            0.0029407073743641376,
+            0.0229296013712883,
+            -0.0072644553147256374,
+            0.003131632227450609,
+            -0.015981800854206085,
+            0.004833656828850508,
+            -0.0036322276573628187,
+            0.0046730004251003265,
+            0.006076996214687824,
+            -0.010309938341379166,
+            0.006426248699426651,
+            -0.017015589401125908,
+            -0.00982564128935337,
+            0.0054390281438827515,
+            -0.005578729324042797,
+            0.00025306266616098583,
+            -0.001689218683168292,
+            0.017015589401125908,
+            -0.002021008636802435,
+            -0.005606669466942549,
+            -0.0004228867474012077,
+            0.012973572127521038,
+            -0.01163244154304266,
+            -0.02158847078680992,
+            0.009266837500035763,
+            0.003974495455622673,
+            0.009881521575152874,
+            -0.005117715802043676,
+            -0.0005270804977044463,
+            -0.009448448196053505,
+            0.0010093401651829481,
+            -0.02713925950229168,
+            -0.01662442646920681,
+            0.003101363545283675,
+            0.0011211010860279202,
+            0.02484816312789917,
+            0.024680521339178085,
+            -0.0012736080680042505,
+            -0.007422782946377993,
+            -0.013765211217105389,
+            -0.0036531828809529543,
+            -0.008116631768643856,
+            0.011706949211657047,
+            -0.01113883126527071,
+            -0.013951479457318783,
+            -7.86727980539581e-07,
+            -0.042096588760614395,
+            0.0005590952932834625,
+            -0.014826939441263676,
+            0.014873506501317024,
+            0.004787089768797159,
+            0.020880652591586113,
+            -0.0036345559637993574,
+            -0.003210796043276787,
+            -0.0045030307956039906,
+            0.008912927471101284,
+            0.009457762353122234,
+            0.013681390322744846,
+            0.004633418750017881,
+            -0.01968853734433651,
+            0.025630488991737366,
+            -0.016261203214526176,
+            -0.03511153534054756,
+            0.016531292349100113,
+            -0.02067575789988041,
+            0.011408920399844646,
+            -0.007874483242630959,
+            0.0025332458317279816,
+            -0.0043586730025708675,
+            0.0209551602602005,
+            -0.01688520237803459,
+            -0.015916608273983,
+            0.00019732775399461389,
+            -0.012610348872840405,
+            0.0008271466940641403,
+            -0.01838465966284275,
+            -0.011539307422935963,
+            -0.006673054303973913,
+            0.02320900373160839,
+            0.006258607842028141,
+            -0.0031805275939404964,
+            -0.025965770706534386,
+            0.015069087967276573,
+            -0.02425210364162922,
+            0.013429928570985794,
+            -0.0004976850468665361,
+            -2.5630051823100075e-05,
+            -0.016931768506765366,
+            0.031069517135620117,
+            0.004633418750017881,
+            0.0031665575224906206,
+            0.010142297483980656,
+            0.023358017206192017,
+            0.005364520940929651,
+            0.005154969170689583,
+            0.0015972488326951861,
+            -1.1186999472556636e-05,
+            -0.0166058000177145,
+            0.01771409437060356,
+            -0.007972273975610733,
+            0.020582623779773712,
+            -0.0111202048137784,
+            0.010067789815366268,
+            0.0037463167682290077,
+            -0.0033202285412698984,
+            0.005825534462928772,
+            -0.005806907545775175,
+            -0.013318167999386787,
+            0.017462633550167084,
+            -0.01974441669881344,
+            0.010933936573565006,
+            -0.006165473721921444,
+            0.005550788715481758,
+            -0.004139808006584644,
+            0.01289906445890665,
+            -0.006901232525706291,
+            -0.02099241316318512,
+            0.008754599839448929,
+            -0.01641021855175495,
+            0.013765211217105389,
+            0.014240195043385029,
+            -0.005718430038541555,
+            -0.013802465051412582,
+            -0.007804632652550936,
+            -0.007324992213398218,
+            0.025127563625574112,
+            -0.018990030512213707,
+            -0.013327481225132942,
+            -0.01062659453600645,
+            0.0006071175448596478,
+            0.006263264454901218,
+            0.014482343569397926,
+            -0.0012922348687425256,
+            0.007059560157358646,
+            0.017248423770070076,
+            -0.012032918632030487,
+            0.0024890073109418154,
+            -0.018114570528268814,
+            0.005704460199922323,
+            -0.0029919310472905636,
+            0.001448234310373664,
+            -0.0013504435773938894,
+            0.026710843667387962,
+            0.014612731523811817,
+            -0.019008658826351166,
+            -0.018487107008695602,
+            -0.006449532229453325,
+            -0.030436204746365547,
+            0.001108295051380992,
+            -0.020135579630732536,
+            -0.02287372015416622,
+            0.03770066052675247,
+            -0.015702400356531143,
+            -0.018971404060721397,
+            0.004533299710601568,
+            -0.005187565926462412,
+            -0.0011059667449444532,
+            -0.004305121023207903,
+            -0.01569308526813984,
+            0.0006711471942253411,
+            -0.018095944076776505,
+            -0.0049547310918569565,
+            0.024643268436193466,
+            0.0021246203687042,
+            0.01417500153183937,
+            -0.002210769336670637,
+            -0.015860727056860924,
+            0.0008812808664515615,
+            -0.005783624015748501,
+            -0.005345894023776054,
+            -0.0016577859641984105,
+            -0.009145762771368027,
+            -0.0023306794464588165,
+            0.013327481225132942,
+            0.011967724189162254,
+            0.0033761090599000454,
+            -0.019166985526680946,
+            -0.005704460199922323,
+            0.021495336666703224,
+            -0.0032317512668669224,
+            -0.013951479457318783,
+            -0.011334412731230259,
+            -0.007050246931612492,
+            0.005611326079815626,
+            0.005201536230742931,
+            0.0028079913463443518,
+            0.001922053750604391,
+            0.011064324527978897,
+            0.013048078864812851,
+            0.00109956378582865,
+            -0.02743728831410408,
+            0.011110891588032246,
+            0.001075698179192841,
+            -0.00645418930798769,
+            -0.006230667233467102,
+            0.00574171356856823,
+            -0.016438158228993416,
+            -0.017565080896019936,
+            -0.01223781332373619,
+            -0.00574171356856823,
+            -0.005518191959708929,
+            0.015078401193022728,
+            -0.030734233558177948,
+            0.010403072461485863,
+            -0.01854298822581768,
+            -0.014053926803171635,
+            0.001118190586566925,
+            0.006896575912833214,
+            -0.0028033347334712744,
+            0.014426463283598423,
+            -0.010132983326911926,
+            0.016475411131978035,
+            -0.01051483303308487,
+            -0.003876704489812255,
+            0.005536818876862526,
+            -0.01688520237803459,
+            -0.0002786745026241988,
+            0.01148342713713646,
+            0.021160054951906204,
+            -0.026282425969839096,
+            0.01514359563589096,
+            -0.0035064967814832926,
+            -0.0053272671066224575,
+            0.025611860677599907,
+            0.00524810329079628,
+            -0.0027777228970080614,
+            -0.002700887154787779,
+            -0.003732346696779132,
+            -0.003078080015257001,
+            -0.012517215684056282,
+            -0.0054855952039361,
+            -0.029635252431035042,
+            -0.011874590069055557,
+            -0.027232393622398376,
+            -0.016056308522820473,
+            0.0017381140496581793,
+            -0.008847733959555626,
+            0.0004837149172089994,
+            -0.01778860203921795,
+            0.008717346005141735,
+            0.024978550150990486,
+            -0.012712797150015831,
+            0.0035530636087059975,
+            0.0028894837014377117,
+            0.008805823512375355,
+            0.005867444444447756,
+            0.007059560157358646,
+            0.012293693609535694,
+            0.0012817572569474578,
+            0.0036718095652759075,
+            0.0029407073743641376,
+            -0.01966990903019905,
+            -0.013606883585453033,
+            -0.010766295716166496,
+            0.018468480557203293,
+            -0.0008836092310957611,
+            -0.027064751833677292,
+            0.0029290656093508005,
+            0.024382492527365685,
+            0.017024902626872063,
+            -0.01713666319847107,
+            0.004731209482997656,
+            0.01648472622036934,
+            0.023488406091928482,
+            0.0036974214017391205,
+            0.041127994656562805,
+            -0.0001826300285756588,
+            0.028461763635277748,
+            0.005415744613856077,
+            0.009397224523127079,
+            0.0002062045969069004,
+            -0.03254103288054466,
+            -0.0020489490125328302,
+            -0.013318167999386787,
+            -0.012051545083522797,
+            0.033397868275642395,
+            0.016615113243460655,
+            -0.017816541716456413,
+            0.0054855952039361,
+            -0.00030937965493649244,
+            0.005588042549788952,
+            -0.02617066539824009,
+            -0.0012631304562091827,
+            0.0069571128115057945,
+            0.006179443560540676,
+            0.00517359608784318,
+            -0.028890179470181465,
+            0.003296945011243224,
+            -0.005210849456489086,
+            -0.0061841001734137535,
+            0.0024075149558484554,
+            -0.009252866730093956,
+            -0.022091394290328026,
+            0.020210087299346924,
+            0.009355314075946808,
+            0.016810694709420204,
+            0.012722110375761986,
+            -0.008940868079662323,
+            -0.008139914833009243,
+            -0.02328351140022278,
+            -0.01066384743899107,
+            0.001526234089396894,
+            0.0019057553727179766,
+            -0.002250351244583726,
+            0.017704781144857407,
+            0.0025379026774317026,
+            -0.00637968210503459,
+            -0.0008271466940641403,
+            -0.0012479961151257157,
+            -0.01579553261399269,
+            -0.003152587218210101,
+            -0.008917584083974361,
+            0.00839137751609087,
+            0.01404461357742548,
+            -0.012619663029909134,
+            -0.007138724438846111,
+            0.014435776509344578,
+            0.005681176669895649,
+            0.024289358407258987,
+            -0.010822176001966,
+            0.0014179657446220517,
+            0.02583538368344307,
+            -0.010878056287765503,
+            -0.010878056287765503,
+            0.012945631518959999,
+            0.015981800854206085,
+            0.014333329163491726,
+            0.0013876971788704395,
+            0.005373834166675806,
+            -0.012377514503896236,
+            -0.007855856791138649,
+            0.009751134552061558,
+            -0.028797045350074768,
+            8.30202698125504e-05,
+            0.00817251205444336,
+            -0.025332460179924965,
+            -0.0009947880171239376,
+            -0.013476495631039143,
+            0.0006007145857438445,
+            -0.010067789815366268,
+            -0.004526314325630665,
+            0.00039320028736256063,
+            0.012042231857776642,
+            -0.007720812223851681,
+            -0.019893432036042213,
+            0.0025774845853447914,
+            -0.020396355539560318,
+            0.021066920831799507,
+            -0.016065621748566628,
+            0.013290227390825748,
+            0.004768462851643562,
+            0.015981800854206085,
+            0.024680521339178085,
+            -0.006831381935626268,
+            -0.005713773425668478,
+            0.01519016269594431,
+            -0.013960792683064938,
+            -0.007557827513664961,
+            -0.0030478115659207106,
+            0.007930363528430462,
+            0.011734888888895512,
+            0.012200559489428997,
+            0.00201169541105628,
+            -0.01179077010601759,
+            0.01081286184489727,
+            -0.00044704339234158397,
+            0.002244530478492379,
+            0.014733805321156979,
+            -0.0072644553147256374,
+            -0.009648687206208706,
+            -0.0029034537728875875,
+            0.0034366461914032698,
+            -0.015199475921690464,
+            -0.00423527043312788,
+            0.0053272671066224575,
+            -0.02939310297369957,
+            -0.013178466819226742,
+            0.0018638450419530272,
+            -0.006896575912833214,
+            -0.03978686407208443,
+            -0.013541690073907375,
+            -0.009006061591207981,
+            -0.008950181305408478,
+            0.0041561066173017025,
+            -0.008475197479128838,
+            -0.012004978023469448,
+            0.021215934306383133,
+            0.027251020073890686,
+            0.024345237761735916,
+            -0.007851200178265572,
+            -0.01131578627973795,
+            0.021495336666703224,
+            -0.02192375436425209,
+            -0.005452997982501984,
+            -0.010980503633618355,
+            -0.0019139045616611838,
+            0.0034506162628531456,
+            0.007795319426804781,
+            0.015758279711008072,
+            -0.0018382332054898143,
+            0.013979420065879822,
+            -0.025053057819604874,
+            0.014202941209077835,
+            -0.004379628226161003,
+            0.0056672063656151295,
+            0.001721815555356443,
+            0.003895331406965852,
+            -0.006887262221425772,
+            0.019297374412417412,
+            -0.002672947011888027,
+            -0.013783838599920273,
+            0.023022735491394997,
+            -0.005839504301548004,
+            -0.002350470284000039,
+            0.0007084007957018912,
+            -0.00501061137765646,
+            0.025239326059818268,
+            0.0111202048137784,
+            -0.011017757467925549,
+            0.016512665897607803,
+            0.006910545751452446,
+            0.014938700012862682,
+            -0.024382492527365685,
+            0.005737056955695152,
+            0.010719728656113148,
+            -0.011706949211657047,
+            0.015786219388246536,
+            -0.02548147365450859,
+            0.018198391422629356,
+            0.0047335377894341946,
+            -0.011949097737669945,
+            -0.019204240292310715,
+            0.030845994129776955,
+            0.02069438435137272,
+            -0.0004354016564320773,
+            0.010803548619151115,
+            -0.017490573227405548,
+            -0.004342374857515097,
+            0.010114356875419617,
+            -0.0020605907775461674,
+            0.00767890177667141,
+            -0.013970105908811092,
+            0.0013131899759173393,
+            -0.009555553086102009,
+            0.007143381051719189,
+            -0.015711713582277298,
+            -0.0010431013070046902,
+            -0.03138617053627968,
+            0.0010221460834145546,
+            -0.01434264238923788,
+            0.0031432739924639463,
+            0.012973572127521038,
+            0.0029593342915177345,
+            0.010281998664140701,
+            0.008116631768643856,
+            0.021644352003932,
+            0.02060125023126602,
+            0.016521979123353958,
+            0.002498320769518614,
+            0.009495015256106853,
+            -0.006193413864821196,
+            0.01178145594894886,
+            0.0013294884702190757,
+            -0.007371559273451567,
+            0.010226117447018623,
+            -0.01852436177432537,
+            -0.013234347105026245,
+            0.023171748965978622,
+            -0.003755630226805806,
+            0.00841000396758318,
+            -0.01709941029548645,
+            -0.006738247815519571,
+            0.01113883126527071,
+            -0.009639373049139977,
+            0.014286762103438377,
+            -0.015897981822490692,
+            0.021122800186276436,
+            0.030529338866472244,
+            -0.016615113243460655,
+            -0.01891552470624447,
+            -0.02646869421005249,
+            0.0064728157594799995,
+            -0.01532986294478178,
+            -0.010095730423927307,
+            -0.002369097201153636,
+            0.0025402309838682413,
+            -3.296072100056335e-05,
+            0.024438371881842613,
+            0.005317953880876303,
+            -0.0013329809298738837,
+            -0.009788387455046177,
+            -0.011744203045964241,
+            0.019129732623696327,
+            0.002738140756264329,
+            -0.00970456749200821,
+            -0.00501061137765646,
+            -0.024121716618537903,
+            -0.0059233251959085464,
+            0.008512451313436031,
+            -0.0027590959798544645,
+            -0.022706080228090286,
+            0.014147060923278332,
+            0.006808098405599594,
+            -0.025555981323122978,
+            -0.004563568159937859,
+            -0.002268978161737323,
+            0.0012584737269207835,
+            -0.007948990911245346,
+            0.007362246047705412,
+            -0.003385422518476844,
+            0.007352932821959257,
+            0.006645113695412874,
+            -0.007511260453611612,
+            0.013998046517372131,
+            0.0005355207249522209,
+            0.01706215739250183,
+            -0.012312320061028004,
+            -0.01569308526813984,
+            -0.01143686007708311,
+            0.014202941209077835,
+            0.0011909515596926212,
+            0.009797701612114906,
+            -0.004184046760201454,
+            -0.0043982551433146,
+            0.0014074882492423058,
+            0.0008958330727182329,
+            0.003196825971826911,
+            -0.01907385140657425,
+            0.03414294123649597,
+            0.009983968921005726,
+            -0.00013373466208577156,
+            7.618073868798092e-05,
+            -0.020414981991052628,
+            0.007460036780685186,
+            0.0005011775647290051,
+            -0.0047521647065877914,
+            -0.004945417400449514,
+            0.0033691239077597857,
+            0.014230881817638874,
+            0.013783838599920273,
+            0.008414660580456257,
+            0.008195796050131321,
+            0.013113273307681084,
+            0.01434264238923788,
+            -0.0012805931037291884,
+            0.011222652159631252,
+            -0.014072554185986519,
+            -0.00235512712970376,
+            -0.016242576763033867,
+            -0.016726873815059662,
+            0.029281342402100563,
+            -0.019465014338493347,
+            0.022799214348196983,
+            0.008177168667316437,
+            0.0024191567208617926,
+            0.024661894887685776,
+            0.006221354007720947,
+            0.025313831865787506,
+            0.017509199678897858,
+            0.015721026808023453,
+            0.0023493061307817698,
+            0.009518299251794815,
+            0.014184314757585526,
+            0.005513535346835852,
+            0.008475197479128838,
+            0.00039640176692046225,
+            -0.005862787831574678,
+            -0.010747668333351612,
+            0.00355539214797318,
+            0.009741820394992828,
+            -0.011883903294801712,
+            -0.008260989561676979,
+            0.017285678535699844,
+            -0.0053272671066224575,
+            -0.011380979791283607,
+            0.00090747483773157,
+            0.005154969170689583,
+            0.012684856541454792,
+            -0.0027404692955315113,
+            -0.007776692509651184,
+            0.002151396358385682,
+            -0.023991329595446587,
+            0.010375131852924824,
+            0.011101577430963516,
+            0.0039046446327120066,
+            0.004917477257549763,
+            0.003913958091288805,
+            0.007720812223851681,
+            0.0004953566822223365,
+            -0.013085332699120045,
+            0.010719728656113148,
+            -0.025891263037919998,
+            -0.008107318542897701,
+            -0.021439457312226295,
+            -0.011585874482989311,
+            -0.007115440908819437,
+            -0.01820770464837551,
+            -0.014817626215517521,
+            0.00671030767261982,
+            -0.021458083763718605,
+            0.0028103196527808905,
+            -0.001522741629742086,
+            -0.01737881265580654,
+            -0.002635693410411477,
+            0.0118093965575099,
+            0.017267052084207535,
+            0.029504865407943726,
+            0.011548621580004692,
+            0.005983862094581127,
+            0.009024688974022865,
+            0.0062446375377476215,
+            -0.007548514287918806,
+            0.03930256515741348,
+            0.0011152802035212517,
+            -0.015758279711008072,
+            0.014016672968864441,
+            -0.014752432703971863,
+            -0.007040933705866337,
+            -0.007036277092993259,
+            0.0053784907795488834,
+            0.0084984814748168,
+            0.020731637254357338,
+            -0.01840328611433506,
+            0.0005104909650981426,
+            0.007562484126538038,
+            0.023153122514486313,
+            0.004745179321616888,
+            0.011716262437403202,
+            -0.0019022627966478467,
+            0.024270731955766678,
+            -0.003932585008442402,
+            -0.0010052655125036836,
+            -0.015376430004835129,
+            -0.008982778526842594,
+            -0.006840695161372423,
+            -0.004731209482997656,
+            -0.017015589401125908,
+            -0.01909247785806656,
+            -0.006095623131841421,
+            0.005303983576595783,
+            -0.007944333367049694,
+            0.010431013070046902,
+            0.007478663697838783,
+            0.014268135651946068,
+            0.018077317625284195,
+            0.025704994797706604,
+            0.010403072461485863,
+            0.005974548868834972,
+            0.010077103041112423,
+            -0.009862895123660564,
+            -0.002845244947820902,
+            -0.006836038548499346,
+            -0.04403377324342728,
+            0.005117715802043676,
+            0.0030245280358940363,
+            -0.01628914475440979,
+            0.0005954757798463106,
+            -0.009965342469513416,
+            0.015376430004835129,
+            0.00629120459780097,
+            -0.011427546851336956,
+            -0.015534758567810059,
+            0.015767592936754227,
+            -0.010440326295793056,
+            0.014873506501317024,
+            0.007422782946377993,
+            0.011604501865804195,
+            -0.006235324312001467,
+            0.019539522007107735,
+            -0.004875567276030779,
+            -0.01535780355334282,
+            0.017481260001659393,
+            0.008633526042103767,
+            -0.0006752217886969447,
+            0.010421698912978172,
+            0.0014191300142556429,
+            -0.02037772908806801,
+            -0.010188864544034004,
+            0.031423427164554596,
+            -0.001983755035325885,
+            -0.015804847702383995,
+            -0.002610081573948264,
+            -0.01062659453600645,
+            0.005769653711467981,
+            0.006645113695412874,
+            -0.004691627342253923,
+            -0.01163244154304266,
+            -0.020526742562651634,
+            -0.018785135820508003,
+            -0.028648031875491142,
+            0.02577950246632099,
+            -0.0017509199678897858,
+            -0.006901232525706291,
+            0.011911843903362751,
+            -5.009592860005796e-05,
+            0.0008358780178241432,
+            -0.0023702613543719053,
+            -0.010542773641645908,
+            0.07178772240877151,
+            -0.019837550818920135,
+            0.010570713318884373,
+            0.00016967857663985342,
+            -0.002754439366981387,
+            0.0017683826154097915,
+            -0.003969838377088308,
+            0.00954623892903328,
+            0.003795212134718895,
+            -0.011744203045964241,
+            -0.01370001770555973,
+            0.003641541115939617,
+            -0.014826939441263676,
+            -0.010524147190153599,
+            -0.0012503245379775763,
+            -0.006575263570994139,
+            -0.019008658826351166,
+            0.007534543983638287,
+            0.0007706842152401805,
+            -0.004507687408477068,
+            -0.008586958982050419,
+            -0.008079377934336662,
+            -0.01776997558772564,
+            0.010235431604087353,
+            -0.0021071576047688723,
+            -0.003415690967813134,
+            -0.03974960744380951,
+            -0.023749180138111115,
+            -0.008871017955243587,
+            -0.028834300115704536,
+            0.00815388560295105,
+            -0.015106341801583767,
+            -0.023786434903740883,
+            0.0068732923828065395,
+            0.007539200596511364,
+            0.006253950763493776,
+            -0.01761164702475071,
+            0.011241278611123562,
+            -0.014053926803171635,
+            -0.002642678329721093,
+            -0.0075531709007918835,
+            0.012889751233160496,
+            0.009145762771368027,
+            -8.149228960974142e-05,
+            -0.013774524442851543,
+            -0.009574179537594318,
+            0.01531123649328947,
+            -0.010086417198181152,
+            0.007893109694123268,
+            -0.004593836609274149,
+            -0.014771059155464172,
+            -0.020526742562651634,
+            0.01755576767027378,
+            0.0033132436219602823,
+            -0.006556636653840542,
+            0.01483625266700983,
+            0.0017450990853831172,
+            -0.00021871947683393955,
+            0.024643268436193466,
+            -0.011185398325324059,
+            -0.011772142723202705,
+            -0.008763913065195084,
+            -0.015544071793556213,
+            0.009630059823393822,
+            -0.019446387887001038,
+            0.011185398325324059,
+            -0.01289906445890665,
+            -0.022352170199155807,
+            -0.005015267990529537,
+            0.00177769607398659,
+            0.02453150600194931,
+            0.0018242630176246166,
+            -0.020563997328281403,
+            -0.012852497398853302,
+            0.006919859442859888,
+            -0.025276578962802887,
+            0.00023836494074203074,
+            0.0002408388099865988,
+            0.015450937673449516,
+            -0.016186697408556938,
+            0.016196010634303093,
+            -0.008600928820669651,
+            -0.0049873278476297855,
+            -0.0028545584063977003,
+            0.016037682071328163,
+            -0.0023621120490133762,
+            -0.024997176602482796,
+            -0.024103090167045593,
+            -0.01273142360150814,
+            -0.013578942976891994,
+            -0.0024075149558484554,
+            0.0226315725594759,
+            0.0024773655459284782,
+            0.026003023609519005,
+            0.007474007084965706,
+            -0.015152908861637115,
+            -0.015711713582277298,
+            -8.636726852273569e-05,
+            0.008265646174550056,
+            0.005411088000983,
+            0.001437756814993918,
+            -0.0017439349321648479,
+            -0.007422782946377993,
+            -0.007762722205370665,
+            -0.0033248853869736195,
+            0.03105088882148266,
+            0.0074414098635315895,
+            -0.0022596647031605244,
+            -0.005029238294810057,
+            -0.002712528919801116,
+            0.005611326079815626,
+            0.0027102006133645773,
+            -0.007162007968872786,
+            0.007767379283905029,
+            -0.004624105058610439,
+            0.0053365807980299,
+            0.025239326059818268,
+            -0.002100172685459256,
+            0.015572011470794678,
+            0.016894515603780746,
+            -0.01533917710185051,
+            -0.026394188404083252,
+            0.025611860677599907,
+            0.0019942326471209526,
+            -0.011520680971443653,
+            -0.011557934805750847,
+            -0.0226315725594759,
+            -0.006337771657854319,
+            -0.004884880501776934,
+            0.006752218119800091,
+            -0.009355314075946808,
+            -0.01695970818400383,
+            -0.006272577680647373,
+            -0.007376216351985931,
+            0.0015658161137253046,
+            -0.003650854341685772,
+            0.017704781144857407,
+            0.01840328611433506,
+            -0.01131578627973795,
+            0.024792281910777092,
+            -0.01532986294478178,
+            0.0567372590303421,
+            0.0015530101954936981,
+            -0.006770845036953688,
+            -0.010142297483980656,
+            -0.012116738595068455,
+            -0.012330947443842888,
+            0.0010780264856293797,
+            0.012507901526987553,
+            -0.0001615293585928157,
+            -0.016261203214526176,
+            -0.008028154261410236,
+            0.00652869651094079,
+            0.007474007084965706,
+            -0.0028010064270347357,
+            0.005308640189468861,
+            0.026952991262078285,
+            0.0002060590632027015,
+            -0.014491656795144081,
+            0.02974701300263405,
+            -0.003955868538469076,
+            0.003459929721429944,
+            0.0024890073109418154,
+            0.008824450895190239,
+            -0.005704460199922323,
+            0.00614219019189477,
+            0.013234347105026245,
+            0.003997778985649347,
+            0.012638289481401443,
+            0.0015611593844369054,
+            -0.0008975793025456369,
+            0.009252866730093956,
+            -0.009555553086102009,
+            0.00508511858060956,
+            -0.002896468620747328,
+            7.610797183588147e-05,
+            -0.003762615378946066,
+            -0.004076942801475525,
+            -0.014985267072916031,
+            0.014491656795144081,
+            -0.006035085767507553,
+            -0.004340046551078558,
+            -0.003755630226805806,
+            -0.011706949211657047,
+            -0.019614029675722122,
+            -0.014333329163491726,
+            0.017937617376446724,
+            -0.016019055619835854,
+            -0.012973572127521038,
+            0.026971617713570595,
+            0.015907295048236847,
+            -0.03125578537583351,
+            -0.013783838599920273,
+            -0.008330839686095715,
+            -0.014566164463758469,
+            0.018626809120178223,
+            -0.0074088131077587605,
+            0.007650961633771658,
+            -0.013280914165079594,
+            0.01453822385519743,
+            0.005876758135855198,
+            -0.021141426637768745,
+            0.010617280378937721,
+            -0.009075912646949291
+          ],
+          "title": "C Encouraging Exploration Across Experts"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "41248fbf-99a8-4cad-acc9-08e36bf14246",
+      "type": "child",
+      "source": {
+        "id": "65742957-6bda-446d-a61a-0fe20313c760",
+        "properties": {
+          "page_content": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity\n\nWilliam Fedus∗ liamfedus@google.com Barret Zoph∗ barretzoph@google.com Noam Shazeer noam@google.com Google, Mountain View, CA 94043, USA Editor: Alexander Clark\n\nAbstract\n\nIn deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) models defy this and instead select different parameters for each incoming example. The result is a sparsely-activated model—with an outrageous number of parameters—but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs, and training instability. We address these with the introduction of the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques mitigate the instabilities, and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large (Raffel et al., 2019) to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the \"Colossal Clean Crawled Corpus\", and achieve a 4x speedup over the T5-XXL model.12 Keywords: mixture-of-experts, natural language processing, sparsity, large-scale machine learning, distributed computing arXiv:2101.03961v3 [cs.LG] 16 Jun 2022 Contents\n\n1 Introduction 3 2 Switch Transformer 4 2.1 Simplifying Sparse Routing 5 2.2 Efficient Sparse Routing 6 2.3 Putting It All Together: The Switch Transformer 8 2.4 Improved Training and Fine-Tuning Techniques 8 3 Scaling Properties 11 3.1 Scaling Results on a Step-Basis 12 3.2 Scaling Results on a Time-Basis 13 3.3 Scaling Versus a Larger Dense Model 13 4 Downstream Results 14 4.1 Fine-Tuning 14 4.2 Distillation 16 4.3 Multilingual Learning 17 5 Designing Models with Data, Model, and Expert-Parallelism 18 5.1 Data Parallelism 20 5.2 Model Parallelism 20 5.3 Model and Data Parallelism 21 5.4 Expert and Data Parallelism 22 5.5 Expert, Model and Data Parallelism 22 5.6 Towards Trillion Parameter Models 22 6 Related Work 24 7 Discussion 25 8 Future Work 26 9 Conclusion 27 A Switch for Attention 27 B Preventing Token Dropping with No-Token-Left-Behind 29\n\nC Encouraging Exploration Across Experts 29 D Switch Transformers in Lower Compute Regimes 29 E Relation of Upstream to Downstream Model Performance 32 F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/switch_transformers.md"
+          },
+          "summary": "The Switch Transformer is a sparsely-activated model that simplifies the Mixture of Experts (MoE) approach, allowing for efficient scaling to trillion-parameter models with reduced communication and computational costs. It achieves significant speedups in pre-training and fine-tuning by activating only a subset of neural network weights for each input, maintaining a constant computational cost. The model demonstrates superior scaling properties and improved performance across various natural language processing tasks, including multilingual settings. It also allows for effective distillation into smaller dense models, preserving a portion of the quality gains. The architecture is designed to maximize parameter count while keeping floating point operations per example constant, making it suitable for both large-scale and smaller-scale applications.",
+          "headlines": {
+            "1 Introduction": [],
+            "2 Switch Transformer": [
+              "2.1 Simplifying Sparse Routing",
+              "2.2 Efficient Sparse Routing",
+              "2.3 Putting It All Together: The Switch Transformer",
+              "2.4 Improved Training and Fine-Tuning Techniques"
+            ],
+            "3 Scaling Properties": [
+              "3.1 Scaling Results on a Step-Basis",
+              "3.2 Scaling Results on a Time-Basis",
+              "3.3 Scaling Versus a Larger Dense Model"
+            ],
+            "4 Downstream Results": [
+              "4.1 Fine-Tuning",
+              "4.2 Distillation",
+              "4.3 Multilingual Learning"
+            ],
+            "5 Designing Models with Data, Model, and Expert-Parallelism": [
+              "5.1 Data Parallelism",
+              "5.2 Model Parallelism",
+              "5.3 Model and Data Parallelism",
+              "5.4 Expert and Data Parallelism",
+              "5.5 Expert, Model and Data Parallelism",
+              "5.6 Towards Trillion Parameter Models"
+            ],
+            "6 Related Work": [],
+            "7 Discussion": [],
+            "8 Future Work": [],
+            "9 Conclusion": [],
+            "A Switch for Attention": [],
+            "B Preventing Token Dropping with No-Token-Left-Behind": [],
+            "C Encouraging Exploration Across Experts": [],
+            "D Switch Transformers in Lower Compute Regimes": [],
+            "E Relation of Upstream to Downstream Model Performance": [],
+            "F Pseudo Code for Switch Transformers": []
+          },
+          "summary_embedding": [
+            -0.0060447100549936295,
+            -0.00390331051312387,
+            -0.01784370467066765,
+            -0.0021820589900016785,
+            0.013227885589003563,
+            0.011338187381625175,
+            0.0165580902248621,
+            0.024240797385573387,
+            -0.005986624862998724,
+            0.03596621751785278,
+            -0.004852031823247671,
+            -0.0068811336532235146,
+            -0.003574162255972624,
+            0.014242436736822128,
+            0.011578272096812725,
+            -0.033302053809165955,
+            -0.024519605562090874,
+            0.01050176378339529,
+            0.016093410551548004,
+            -0.006795942317694426,
+            0.007473600562661886,
+            -0.030513975769281387,
+            -0.06124480068683624,
+            -0.006838538218289614,
+            -0.0026099516544491053,
+            0.01029265858232975,
+            0.0010329445358365774,
+            -0.010873507708311081,
+            -0.053469158709049225,
+            0.023311438038945198,
+            0.025294071063399315,
+            0.005092116538435221,
+            -0.029584616422653198,
+            0.0009361362317577004,
+            -0.05161044001579285,
+            -0.04448534920811653,
+            0.0379488542675972,
+            0.012561844661831856,
+            -0.04364892467856407,
+            0.039249956607818604,
+            -0.004875265993177891,
+            -0.0022440163884311914,
+            -0.021499186754226685,
+            -0.0034909071400761604,
+            -0.007039899472147226,
+            -0.009851212613284588,
+            -0.02874819189310074,
+            0.0014037203509360552,
+            0.021886419504880905,
+            -0.0056962003000080585,
+            0.023264970630407333,
+            -0.024271776899695396,
+            0.01936165988445282,
+            0.04240977764129639,
+            -0.06957805901765823,
+            0.024659009650349617,
+            0.017549408599734306,
+            0.05120771750807762,
+            0.06511713564395905,
+            0.018339363858103752,
+            0.021669568493962288,
+            -0.014141756109893322,
+            -0.017766257748007774,
+            -0.011872569099068642,
+            0.004673904739320278,
+            -0.014970434829592705,
+            -0.01891246810555458,
+            0.008449427783489227,
+            -0.020461400970816612,
+            0.008209343999624252,
+            -0.05576157942414284,
+            -0.014730350114405155,
+            -0.034200433641672134,
+            -0.01140014547854662,
+            0.017642343416810036,
+            -0.018215449526906013,
+            0.030126741155982018,
+            0.02997184917330742,
+            0.002638994250446558,
+            0.0007822110201232135,
+            0.021049996837973595,
+            0.006509389728307724,
+            -0.0053438181057572365,
+            0.012321759946644306,
+            0.028159597888588905,
+            -0.01450575515627861,
+            -0.010106786154210567,
+            -0.03454120084643364,
+            0.01861817203462124,
+            0.01589204929769039,
+            -0.04194509983062744,
+            -0.012569589540362358,
+            -0.013824224472045898,
+            0.032341714948415756,
+            -0.009549170732498169,
+            0.04098476096987724,
+            -0.010083552449941635,
+            0.001383390510454774,
+            -0.031149037182331085,
+            -0.009463978931307793,
+            0.017069239169359207,
+            0.006172496825456619,
+            -0.016589069738984108,
+            -0.0012846461031585932,
+            0.016170857474207878,
+            0.006617815233767033,
+            -0.030142230913043022,
+            0.04488807171583176,
+            0.023497309535741806,
+            0.055296897888183594,
+            0.05579255893826485,
+            -0.037391237914562225,
+            -0.002445377642288804,
+            0.0344172865152359,
+            -0.0228312686085701,
+            -0.015040136873722076,
+            -0.034014564007520676,
+            -0.01768881268799305,
+            0.035935238003730774,
+            -0.0318460576236248,
+            0.01556677371263504,
+            0.028066661208868027,
+            -0.025712283328175545,
+            0.003488970920443535,
+            -0.03943582624197006,
+            0.003622566582635045,
+            0.01220559049397707,
+            0.006095050368458033,
+            -0.0026293133851140738,
+            -0.013615119270980358,
+            0.008906363509595394,
+            0.003055269829928875,
+            0.00800798274576664,
+            -0.04615819454193115,
+            0.018478767946362495,
+            -0.00454224506393075,
+            0.04033420979976654,
+            0.042502716183662415,
+            -0.02730768360197544,
+            -0.016743963584303856,
+            0.008557853288948536,
+            0.012104909867048264,
+            0.04188314080238342,
+            -0.007287728600203991,
+            -0.00121784838847816,
+            -0.0007836631848476827,
+            0.007322579622268677,
+            0.047552235424518585,
+            0.004646798130124807,
+            0.0020697612781077623,
+            -0.027230238541960716,
+            0.014978179708123207,
+            -0.019733402878046036,
+            0.027648448944091797,
+            0.023373395204544067,
+            0.02942972257733345,
+            -0.04048910364508629,
+            0.017518429085612297,
+            -0.011818356812000275,
+            -0.005568413063883781,
+            -0.002726121572777629,
+            -0.02856232039630413,
+            -0.0074426219798624516,
+            0.033983584493398666,
+            0.015365412458777428,
+            0.012956822291016579,
+            0.03271345794200897,
+            0.0012052633101120591,
+            -0.042502716183662415,
+            0.03804178908467293,
+            0.024550583213567734,
+            0.04640602692961693,
+            0.023466331884264946,
+            0.010796061716973782,
+            0.006571347359567881,
+            0.013375034555792809,
+            -0.019377149641513824,
+            0.015907539054751396,
+            0.02560385875403881,
+            0.009750531986355782,
+            -0.03167567402124405,
+            -0.0009501734166406095,
+            0.01178737822920084,
+            -0.004546117503196001,
+            0.03701949119567871,
+            0.04104671627283096,
+            0.020569827407598495,
+            0.05030933395028114,
+            0.0020155487582087517,
+            0.017936641350388527,
+            0.005936284549534321,
+            0.02820606529712677,
+            0.030467506498098373,
+            -0.005285732913762331,
+            0.005076627247035503,
+            -0.01136916596442461,
+            -0.03385967016220093,
+            0.014110777527093887,
+            0.01561324205249548,
+            -0.008077683858573437,
+            -0.01697630248963833,
+            0.006033093202859163,
+            0.009169681929051876,
+            -0.01795213110744953,
+            -0.017456471920013428,
+            0.005971135571599007,
+            0.018866000697016716,
+            -0.013576395809650421,
+            0.024689987301826477,
+            -0.04442339017987251,
+            0.029119934886693954,
+            -0.033209118992090225,
+            -0.003384418087080121,
+            -0.010641167871654034,
+            -0.0030843124259263277,
+            0.0041666291654109955,
+            0.011314953677356243,
+            0.013406013138592243,
+            0.002042655134573579,
+            -0.013909416273236275,
+            0.024519605562090874,
+            0.018447788432240486,
+            0.012073930352926254,
+            -0.03148980438709259,
+            -0.014397329650819302,
+            -0.0015673263696953654,
+            -0.008743725717067719,
+            -0.034138478338718414,
+            0.026966920122504234,
+            0.014048820361495018,
+            -0.03844451159238815,
+            0.017797237262129784,
+            0.029538147151470184,
+            0.014420564286410809,
+            -0.004790074657648802,
+            -0.0463130883872509,
+            0.04045812413096428,
+            -0.030436528846621513,
+            -0.033333033323287964,
+            -0.010478530079126358,
+            -0.0003146269591525197,
+            0.02894955314695835,
+            -0.0064358157105743885,
+            -0.050835974514484406,
+            0.01184933539479971,
+            0.020631784573197365,
+            0.00010763872705865651,
+            0.005731051322072744,
+            -0.019593998789787292,
+            0.00028631053282879293,
+            -0.03395260497927666,
+            0.0090844901278615,
+            0.026750069111585617,
+            0.008867640048265457,
+            -0.016418687999248505,
+            0.03351890295743942,
+            -0.0004121613164898008,
+            -0.018695618957281113,
+            -0.003889757441356778,
+            0.006885006092488766,
+            0.034912943840026855,
+            0.004836542531847954,
+            -0.02946070022881031,
+            0.05665995925664902,
+            -0.02100352756679058,
+            0.021344292908906937,
+            0.05631919577717781,
+            -0.019439106807112694,
+            -0.008472662419080734,
+            -0.002065889071673155,
+            0.0026196325197815895,
+            -0.0260530486702919,
+            -0.008937342092394829,
+            0.032341714948415756,
+            0.041728246957063675,
+            -0.018742086365818977,
+            -0.001398879918269813,
+            0.006780453026294708,
+            0.009425255469977856,
+            0.004948840010911226,
+            0.02692045085132122,
+            -0.012128143571317196,
+            -0.014559967443346977,
+            0.05198218300938606,
+            0.028314489871263504,
+            -0.012081675231456757,
+            -0.0164031982421875,
+            0.019888296723365784,
+            -0.024844881147146225,
+            -0.020244551822543144,
+            -0.02431824430823326,
+            -0.020414933562278748,
+            -0.0042518205009400845,
+            0.009525936096906662,
+            0.037484172731637955,
+            -0.03968365862965584,
+            0.012747716158628464,
+            0.021700548008084297,
+            0.02981695532798767,
+            -0.019532041624188423,
+            -0.05910727381706238,
+            -0.0016128262504935265,
+            0.03296129032969475,
+            -0.015458349138498306,
+            -0.049379974603652954,
+            0.015164051204919815,
+            0.0260530486702919,
+            0.05957195535302162,
+            -0.001461805310100317,
+            -0.014954946003854275,
+            0.061926331371068954,
+            0.03026614524424076,
+            -0.004383479710668325,
+            -0.009494957514107227,
+            -0.0127864396199584,
+            0.003396035172045231,
+            0.00465454300865531,
+            -0.03364282101392746,
+            -0.041170634329319,
+            -0.011841590516269207,
+            -0.010284913703799248,
+            0.008108663372695446,
+            -0.040922801941633224,
+            0.03531566634774208,
+            -0.021917399019002914,
+            0.010950954630970955,
+            0.015473837964236736,
+            0.037391237914562225,
+            -0.021344292908906937,
+            -0.01284839678555727,
+            -0.01572941243648529,
+            -0.024968795478343964,
+            -0.014010096900165081,
+            0.021917399019002914,
+            0.007682706229388714,
+            3.4699722164077684e-05,
+            0.010075807571411133,
+            -0.024659009650349617,
+            -0.051052823662757874,
+            0.0291044469922781,
+            0.0015605497173964977,
+            -0.015938518568873405,
+            -0.004623564425855875,
+            -0.01977987214922905,
+            -0.01122201792895794,
+            -0.012608313001692295,
+            -0.039249956607818604,
+            0.021111954003572464,
+            -0.01135367713868618,
+            0.015876561403274536,
+            0.0009250032599084079,
+            0.003659353591501713,
+            0.028624277561903,
+            0.01116006076335907,
+            0.0051811798475682735,
+            -0.03255856782197952,
+            -0.018277406692504883,
+            0.010122275911271572,
+            -0.00561100896447897,
+            -0.018339363858103752,
+            -0.013351799920201302,
+            -0.05037129297852516,
+            -0.03305422514677048,
+            -0.008054450154304504,
+            -0.05415068939328194,
+            0.0020697612781077623,
+            -0.014296649023890495,
+            -0.02232012152671814,
+            -0.005990497302263975,
+            0.03107159025967121,
+            -0.003235333366319537,
+            0.025294071063399315,
+            0.0076594725251197815,
+            0.037360258400440216,
+            0.040117356926202774,
+            0.008426194079220295,
+            0.03065337985754013,
+            -0.0029991210903972387,
+            -0.016867877915501595,
+            0.05486319959163666,
+            -0.01881953328847885,
+            0.013529927469789982,
+            -0.008604321628808975,
+            0.008604321628808975,
+            0.03299226611852646,
+            -0.003794885240495205,
+            0.02007416822016239,
+            -0.0311955064535141,
+            -0.07998688519001007,
+            0.01637221872806549,
+            0.028159597888588905,
+            -0.06418777257204056,
+            0.014691627584397793,
+            -0.020275529474020004,
+            0.02052335813641548,
+            0.02411688305437565,
+            -0.0009114501299336553,
+            -0.014064309187233448,
+            0.005595519673079252,
+            0.0020833145827054977,
+            -0.01456771232187748,
+            -0.002691270550712943,
+            -0.009417511522769928,
+            0.008774704299867153,
+            -0.017007282003760338,
+            -0.013406013138592243,
+            -0.034231413155794144,
+            0.010873507708311081,
+            0.01233724970370531,
+            -0.003969140350818634,
+            0.0011500825639814138,
+            -0.03277541697025299,
+            -0.007876322604715824,
+            -0.02145271934568882,
+            0.03382869064807892,
+            0.012058441527187824,
+            -0.030699847266077995,
+            -0.007814365439116955,
+            0.015264731831848621,
+            -0.007876322604715824,
+            0.0008906363509595394,
+            0.024132372811436653,
+            0.02103450708091259,
+            0.0028364830650389194,
+            -0.0009700191440060735,
+            0.02438020147383213,
+            0.007427132688462734,
+            -0.01720864325761795,
+            0.028794659301638603,
+            0.014288905076682568,
+            0.060160547494888306,
+            -0.03271345794200897,
+            0.001297231181524694,
+            -0.04135650396347046,
+            -0.031118059530854225,
+            0.006408709101378918,
+            0.014350862242281437,
+            0.007597515359520912,
+            -0.028252532705664635,
+            0.005978880450129509,
+            0.02004319056868553,
+            0.01469937153160572,
+            0.009704063646495342,
+            0.0037580979987978935,
+            -0.036740683019161224,
+            -0.000691695255227387,
+            -0.011214273050427437,
+            -0.0038781403563916683,
+            0.035656433552503586,
+            -0.008186109364032745,
+            -0.05151750147342682,
+            -0.0006418389966711402,
+            -0.03255856782197952,
+            -0.0353776253759861,
+            0.021313315257430077,
+            -0.00839521549642086,
+            -0.0039207362569868565,
+            -0.013986863195896149,
+            -0.000537770101800561,
+            -0.010524998418986797,
+            0.027509044855833054,
+            0.028345469385385513,
+            0.022490503266453743,
+            -0.05384090170264244,
+            0.009479468688368797,
+            0.00023766436788719147,
+            -0.05817791447043419,
+            0.0009709871956147254,
+            -0.007117345929145813,
+            0.011191039346158504,
+            -0.020213572308421135,
+            -0.02414786070585251,
+            0.011500826105475426,
+            0.03392162546515465,
+            0.014304393902420998,
+            0.0321558453142643,
+            -0.013367289677262306,
+            -0.09250226616859436,
+            0.018292896449565887,
+            -0.008689512498676777,
+            0.04126356914639473,
+            0.00865078903734684,
+            -0.004278926644474268,
+            -0.015845581889152527,
+            -0.00865078903734684,
+            -0.006943091284483671,
+            0.01283290795981884,
+            0.004530628211796284,
+            -0.005626498255878687,
+            0.0016631665639579296,
+            0.013708055019378662,
+            -0.005967263597995043,
+            0.007481344975531101,
+            0.003651608945801854,
+            -0.013189162127673626,
+            -0.06059424951672554,
+            -0.04055105894804001,
+            -0.018540725111961365,
+            0.050835974514484406,
+            -0.0069159846752882,
+            -0.046684831380844116,
+            -0.010850274004042149,
+            0.05058814212679863,
+            0.012259802781045437,
+            -0.0334569476544857,
+            -0.0007841471815481782,
+            0.0050882440991699696,
+            0.015458349138498306,
+            0.004546117503196001,
+            -0.027292195707559586,
+            0.02820606529712677,
+            0.0008412640891037881,
+            0.02300165221095085,
+            -0.018339363858103752,
+            -0.005312839522957802,
+            0.035780347883701324,
+            -0.024844881147146225,
+            0.023791607469320297,
+            0.0055219451896846294,
+            0.031629208475351334,
+            0.002952652983367443,
+            0.010060317814350128,
+            0.004704883322119713,
+            0.0186491496860981,
+            0.004275054205209017,
+            0.0276639387011528,
+            0.011462102644145489,
+            0.006428070832043886,
+            0.021623101085424423,
+            -0.009378788061439991,
+            -0.014513500034809113,
+            0.01885051093995571,
+            -0.0014695499558001757,
+            0.025232113897800446,
+            -0.020538847893476486,
+            0.002726121572777629,
+            -0.0016883367206901312,
+            -0.017967619001865387,
+            -0.040922801941633224,
+            -0.0007405834621749818,
+            0.021747015416622162,
+            -0.011671207845211029,
+            -0.0228312686085701,
+            -0.013715799897909164,
+            -0.01585332676768303,
+            0.0038490979932248592,
+            0.034169457852840424,
+            -0.0398695282638073,
+            0.00963436160236597,
+            -0.000830615172162652,
+            -0.03125746175646782,
+            -0.0239619892090559,
+            -0.01093546487390995,
+            0.014614180661737919,
+            -0.01652711257338524,
+            -0.021173911169171333,
+            -0.028469383716583252,
+            -0.008410705253481865,
+            0.006048582494258881,
+            -0.05065010115504265,
+            -0.004054331686347723,
+            0.019965743646025658,
+            0.0276639387011528,
+            -0.003099801717326045,
+            -0.02177799493074417,
+            0.0012468908680602908,
+            0.022351099178195,
+            0.02936776541173458,
+            0.014226946979761124,
+            0.027586491778492928,
+            -0.002178186783567071,
+            -0.0011675080750137568,
+            -0.0073148347437381744,
+            0.01307299267500639,
+            0.0011287847300991416,
+            0.036616768687963486,
+            0.0122210793197155,
+            -0.020306508988142014,
+            0.0013959755888208747,
+            -0.006431943271309137,
+            0.029181893914937973,
+            -0.02270735427737236,
+            -9.832092473516241e-05,
+            0.03531566634774208,
+            0.027741385623812675,
+            0.004933350719511509,
+            -0.0143121387809515,
+            -0.05142456665635109,
+            0.021592123433947563,
+            -0.01765783317387104,
+            0.031025122851133347,
+            0.016666516661643982,
+            0.003397971158847213,
+            -0.0034212053287774324,
+            0.012368228286504745,
+            -0.022087780758738518,
+            -0.004406713880598545,
+            -0.021142931655049324,
+            0.04389675334095955,
+            -0.043184246867895126,
+            0.008472662419080734,
+            0.029445212334394455,
+            0.011926782317459583,
+            -0.019082851707935333,
+            -0.005425137002021074,
+            0.029212871566414833,
+            -0.00162637943867594,
+            0.01308848150074482,
+            0.0058665829710662365,
+            -0.026935940608382225,
+            0.01202746294438839,
+            -0.031660184264183044,
+            0.02177799493074417,
+            -0.013436991721391678,
+            0.018447788432240486,
+            0.03342596814036369,
+            -0.0030397805385291576,
+            -0.0038239278364926577,
+            -0.016263794153928757,
+            -0.00969631876796484,
+            0.0662323608994484,
+            0.01054823212325573,
+            0.03726732358336449,
+            0.0018974426202476025,
+            -0.009773765690624714,
+            0.007969259284436703,
+            0.0063041565008461475,
+            -0.012329504825174809,
+            0.010494019836187363,
+            0.024178840219974518,
+            0.0012730291346088052,
+            -0.002412462839856744,
+            -0.019408127292990685,
+            -0.03438630700111389,
+            0.003727119415998459,
+            -0.030932186171412468,
+            0.022691864520311356,
+            0.009208405390381813,
+            0.017456471920013428,
+            0.008240322582423687,
+            -0.015907539054751396,
+            0.03200095146894455,
+            0.05461536720395088,
+            0.0019768255297094584,
+            0.013375034555792809,
+            0.007938280701637268,
+            0.007036027032881975,
+            -0.011338187381625175,
+            0.04702559858560562,
+            0.000662652775645256,
+            0.005169562995433807,
+            0.022536970674991608,
+            0.02145271934568882,
+            0.022939693182706833,
+            0.002304037567228079,
+            -0.011694442480802536,
+            0.010478530079126358,
+            -0.0006302220281213522,
+            0.004445436876267195,
+            -0.010695381090044975,
+            -0.027865299955010414,
+            0.007492962293326855,
+            0.01302652433514595,
+            0.022598929703235626,
+            0.021592123433947563,
+            -0.03178409859538078,
+            -0.01855621486902237,
+            0.03364282101392746,
+            -0.02997184917330742,
+            0.03612111136317253,
+            -0.0014424435794353485,
+            -0.023807097226381302,
+            -0.0058356039226055145,
+            0.016929835081100464,
+            0.02656419761478901,
+            -0.029181893914937973,
+            0.01804506592452526,
+            0.029693040996789932,
+            -0.0074426219798624516,
+            0.0005939188995398581,
+            0.012112653814256191,
+            0.02447313815355301,
+            -0.009293596260249615,
+            -0.0024027819745242596,
+            -0.0013320822035893798,
+            -0.011926782317459583,
+            -0.018370341509580612,
+            -0.0044725434854626656,
+            -0.022629907354712486,
+            0.005564541090279818,
+            -0.007365175057202578,
+            -0.003481226274743676,
+            -0.008581086993217468,
+            0.01154729351401329,
+            0.001979729626327753,
+            -0.019857317209243774,
+            -0.022335609421133995,
+            0.01652711257338524,
+            0.03887821361422539,
+            -0.018215449526906013,
+            -0.02106548473238945,
+            0.024442158639431,
+            -0.01759587600827217,
+            -0.01601596549153328,
+            0.036678727716207504,
+            0.013739033602178097,
+            0.007736918982118368,
+            -0.03358086198568344,
+            -0.030049296095967293,
+            -0.03274443745613098,
+            0.006296411622315645,
+            0.03785591572523117,
+            -0.0028752065263688564,
+            0.02569679357111454,
+            0.03701949119567871,
+            0.0057891360484063625,
+            -0.028934063389897346,
+            -0.014823286794126034,
+            -0.045104920864105225,
+            0.060160547494888306,
+            0.0011481463443487883,
+            0.05161044001579285,
+            0.016294771805405617,
+            -0.01456771232187748,
+            0.00419760774821043,
+            -0.013251119293272495,
+            0.026192452758550644,
+            0.02280029095709324,
+            0.025929134339094162,
+            0.006582964211702347,
+            0.022505993023514748,
+            -0.0037522895727306604,
+            -0.012197845615446568,
+            0.007307090330868959,
+            0.014536733739078045,
+            0.0002497654058970511,
+            -0.011392400600016117,
+            0.015783624723553658,
+            -0.01307299267500639,
+            -0.011361422017216682,
+            0.023295948281884193,
+            0.01368482131510973,
+            -0.012344993650913239,
+            0.014064309187233448,
+            -0.008565598167479038,
+            0.01239920686930418,
+            -0.023915521800518036,
+            0.022691864520311356,
+            -0.03262052312493324,
+            0.009587894193828106,
+            0.0015053689712658525,
+            -0.0073148347437381744,
+            0.024550583213567734,
+            -0.0027454833034425974,
+            -0.05071205645799637,
+            0.03550153970718384,
+            -0.008015726692974567,
+            0.0037213109899312258,
+            0.006381602957844734,
+            0.011392400600016117,
+            -0.0012623801594600081,
+            0.03807276859879494,
+            0.01239920686930418,
+            0.015969496220350266,
+            0.02199484594166279,
+            0.011911292560398579,
+            -0.008433938957750797,
+            -0.004096927121281624,
+            -0.0016137943603098392,
+            -0.01784370467066765,
+            0.025944624096155167,
+            0.0018674320308491588,
+            0.03962169960141182,
+            -0.018168980255723,
+            -0.02300165221095085,
+            0.028066661208868027,
+            0.007597515359520912,
+            -0.02351279929280281,
+            -0.0017377089243382215,
+            0.01202746294438839,
+            -0.009185170754790306,
+            0.027385130524635315,
+            0.01196550577878952,
+            0.019113829359412193,
+            -0.018060555681586266,
+            -0.004340883810073137,
+            -0.005072754807770252,
+            -0.015938518568873405,
+            0.02238207869231701,
+            0.020771188661456108,
+            0.011152315884828568,
+            -0.011880313977599144,
+            -0.01307299267500639,
+            0.020600805059075356,
+            0.035036858171224594,
+            -0.0002744515077210963,
+            0.005672966130077839,
+            0.017487451434135437,
+            0.002373739378526807,
+            -0.04470219835639,
+            0.016232814639806747,
+            -0.014025585725903511,
+            0.011214273050427437,
+            0.019593998789787292,
+            -0.021917399019002914,
+            -0.03196997195482254,
+            -0.011539548635482788,
+            -0.0414804182946682,
+            0.00493722315877676,
+            -0.0069159846752882,
+            0.013576395809650421,
+            -0.010176488198339939,
+            -0.0014831031439825892,
+            0.016666516661643982,
+            0.0038103745318949223,
+            0.02177799493074417,
+            -0.011678952723741531,
+            -0.012073930352926254,
+            -0.020213572308421135,
+            -0.020786678418517113,
+            0.004278926644474268,
+            -0.030405549332499504,
+            -0.004488032776862383,
+            0.021375272423028946,
+            -0.04783104360103607,
+            -0.03689557686448097,
+            0.02898053266108036,
+            -0.008503641001880169,
+            0.020755698904395103,
+            0.022955182939767838,
+            -0.015094350092113018,
+            -0.0008717587334103882,
+            -0.018695618957281113,
+            0.02258343994617462,
+            -0.007837600074708462,
+            0.026300877332687378,
+            -0.0027551641687750816,
+            0.0006292539183050394,
+            -0.02659517526626587,
+            0.020306508988142014,
+            0.005227648187428713,
+            0.012275291606783867,
+            -0.02390003204345703,
+            0.006815304048359394,
+            -0.03133491054177284,
+            -0.01784370467066765,
+            0.0180140882730484,
+            -0.011152315884828568,
+            0.027989214286208153,
+            -0.006060199346393347,
+            0.028438406065106392,
+            0.022722844034433365,
+            0.024751944467425346,
+            0.02676555886864662,
+            -0.02363671362400055,
+            0.008495896123349667,
+            0.03726732358336449,
+            0.00865078903734684,
+            -0.015876561403274536,
+            0.008170620538294315,
+            -0.04863648861646652,
+            -0.008062195032835007,
+            0.05030933395028114,
+            0.002127846470102668,
+            0.023141054436564445,
+            0.016294771805405617,
+            -0.002873270306736231,
+            -0.005688455421477556,
+            0.010269423946738243,
+            -0.04296739399433136,
+            -0.023172033950686455,
+            0.022629907354712486,
+            -0.021561143919825554,
+            6.54212328754511e-07,
+            0.03122648410499096,
+            0.012120398692786694,
+            -0.0009109660750254989,
+            -0.05108380317687988,
+            0.02100352756679058,
+            0.01178737822920084,
+            0.014443797990679741,
+            -0.010300402529537678,
+            0.026068538427352905,
+            0.01733255758881569,
+            0.038413532078266144,
+            0.012755461037158966,
+            -0.002679653698578477,
+            -0.03782493621110916,
+            0.009177426807582378,
+            -0.021685058251023293,
+            0.0011471783509477973,
+            0.0074503663927316666,
+            -0.02273833192884922,
+            -0.024782923981547356,
+            0.0043912241235375404,
+            0.004588713403791189,
+            0.011880313977599144,
+            0.022087780758738518,
+            0.009417511522769928,
+            -0.021204888820648193,
+            0.015404135920107365,
+            -0.05446047708392143,
+            -0.015450604259967804,
+            -0.023884542286396027,
+            0.02566581591963768,
+            0.020012211054563522,
+            0.010184233076870441,
+            0.005545179359614849,
+            0.010904486291110516,
+            -0.020988039672374725,
+            -0.009239383973181248,
+            0.020631784573197365,
+            0.011074868962168694,
+            -0.012840652838349342,
+            0.04113965481519699,
+            -0.00011126903700642288,
+            0.031210994347929955,
+            -0.02280029095709324,
+            0.002941036131232977,
+            -0.001241082325577736,
+            -0.03649285435676575,
+            0.009130958467721939,
+            0.01981084980070591,
+            0.02151467651128769,
+            -0.01861817203462124,
+            0.0024143988266587257,
+            0.01048627495765686,
+            0.004766840487718582,
+            0.038351576775312424,
+            0.027121812105178833,
+            -0.044206541031599045,
+            0.011663463898003101,
+            0.012995545752346516,
+            -0.009363298304378986,
+            0.048419639468193054,
+            -0.017518429085612297,
+            0.017673322930932045,
+            0.008240322582423687,
+            -0.01798310875892639,
+            0.02318752370774746,
+            -0.01469937153160572,
+            -0.013282098807394505,
+            0.00650551775470376,
+            0.009192915633320808,
+            -0.0001770381786627695,
+            -0.037577107548713684,
+            -0.017859194427728653,
+            -0.005719434004276991,
+            0.0038297362625598907,
+            0.01562873087823391,
+            0.013498948886990547,
+            0.009967382065951824,
+            0.005161818582564592,
+            -0.00945623405277729,
+            -0.0016089539276435971,
+            0.008124152198433876,
+            -0.0005532593932002783,
+            0.0019216446671634912,
+            -0.01926872320473194,
+            0.02698240801692009,
+            0.02572777308523655,
+            -0.0018354853382334113,
+            -0.04652994126081467,
+            -0.0036709706764668226,
+            0.0037077576853334904,
+            -0.017735280096530914,
+            -0.0035315665882080793,
+            -0.04519785940647125,
+            -0.01810702309012413,
+            0.0143121387809515,
+            -0.014614180661737919,
+            -0.0460033044219017,
+            0.009076746180653572,
+            -0.013886182568967342,
+            0.0063970922492444515,
+            -0.00932457484304905,
+            0.005762029904872179,
+            0.003651608945801854,
+            0.005680711008608341,
+            -0.005909178406000137,
+            0.05108380317687988,
+            -0.03466511517763138,
+            0.028051171451807022,
+            0.008565598167479038,
+            -0.012437930330634117,
+            -0.02261441759765148,
+            0.00887538492679596,
+            -0.044981006532907486,
+            -0.033209118992090225,
+            0.016743963584303856,
+            -0.01781272701919079,
+            0.011074868962168694,
+            0.06161654368042946,
+            0.007330324035137892,
+            -0.023342415690422058,
+            -0.01233724970370531,
+            -0.021049996837973595,
+            -0.04191412031650543,
+            0.003442503046244383,
+            0.005463860463351011,
+            -0.004948840010911226,
+            -0.03807276859879494,
+            -0.021716037765145302,
+            0.04615819454193115,
+            -0.012004229240119457,
+            0.012422440573573112,
+            0.01080380566418171,
+            -0.0019410063978284597,
+            -0.04860550910234451,
+            -0.021019017323851585,
+            0.0395597442984581,
+            0.021886419504880905,
+            -0.008426194079220295,
+            0.0028364830650389194,
+            -0.018277406692504883,
+            -0.003289545886218548,
+            0.006296411622315645,
+            -0.009107724763453007,
+            -0.03562545403838158,
+            0.02067825198173523,
+            0.0054832217283546925,
+            -0.023373395204544067,
+            -0.025805220007896423,
+            -0.03296129032969475,
+            0.009944148361682892,
+            0.02453509531915188,
+            -0.01968693546950817,
+            0.005823987070471048,
+            0.00027203132049180567,
+            -0.000881923595443368,
+            0.00042740863864310086,
+            -0.009959637187421322,
+            -0.019764382392168045,
+            -0.020151615142822266,
+            0.03268248215317726,
+            -0.0060059865936636925,
+            0.0074387495405972,
+            -0.03630698472261429,
+            -0.010974188335239887,
+            0.011322698555886745,
+            -0.002395037328824401,
+            -0.013452480547130108,
+            -0.009092235006392002,
+            -0.01739451475441456,
+            -0.009882191196084023,
+            -0.007876322604715824,
+            0.022660886868834496,
+            0.006168624851852655,
+            0.005409647710621357,
+            0.0023795480374246836,
+            -0.008581086993217468,
+            0.02898053266108036,
+            0.017874684184789658,
+            -0.014188224449753761,
+            -0.00920066051185131,
+            0.03451022133231163,
+            0.02665713243186474,
+            -0.009115469641983509,
+            0.01798310875892639,
+            0.023264970630407333,
+            -0.0010484338272362947,
+            -0.01196550577878952,
+            0.001922612776979804,
+            -0.0009550138493068516,
+            -0.0032972905319184065,
+            0.00845717266201973,
+            -0.015365412458777428,
+            0.0315207801759243,
+            0.04287445917725563,
+            -0.002145271748304367,
+            0.009587894193828106,
+            -0.003727119415998459,
+            -0.008472662419080734,
+            0.017069239169359207,
+            0.009316830895841122,
+            0.04625113308429718,
+            0.01098193321377039,
+            0.005680711008608341,
+            -0.033983584493398666,
+            -0.001229465357027948,
+            -0.01184933539479971,
+            0.03308520466089249,
+            0.02061629481613636,
+            0.00889087375253439,
+            -0.008379725739359856,
+            0.01344473659992218,
+            0.008341003209352493,
+            0.00661007035523653,
+            0.020569827407598495,
+            -0.016542602330446243,
+            -0.017286090180277824,
+            0.030467506498098373,
+            -0.00672236829996109,
+            0.01289486512541771,
+            0.02029101923108101,
+            -0.002089123008772731,
+            0.010842529125511646,
+            0.07100307941436768,
+            -0.006184114143252373,
+            0.015713922679424286,
+            -0.019082851707935333,
+            0.0015789433382451534,
+            -0.007640110794454813,
+            -0.03370477631688118,
+            0.018246427178382874,
+            0.0039962464943528175,
+            0.00865078903734684,
+            0.01198099460452795,
+            0.009053511545062065,
+            -0.007946024648845196,
+            -0.00963436160236597,
+            0.015195030719041824,
+            0.0019080914789810777,
+            0.00033568276558071375,
+            0.001336922636255622,
+            -0.0065210070461034775,
+            -0.004011735785752535,
+            -0.021948376670479774,
+            0.02763296104967594,
+            0.007140580099076033,
+            0.030606910586357117,
+            -0.055296897888183594,
+            0.030095763504505157,
+            -0.00789568480104208,
+            0.02554190158843994,
+            -0.016201836988329887,
+            -0.011090358719229698,
+            0.001609921921044588,
+            0.018153492361307144,
+            0.004491904750466347,
+            0.006462921854108572,
+            0.011888058856129646,
+            0.023233991116285324,
+            -0.03444826602935791,
+            0.017053749412298203,
+            0.023110076785087585,
+            0.0048675211146473885,
+            0.0003535923024173826,
+            0.005165690556168556,
+            0.015458349138498306,
+            0.007198664825409651,
+            -0.023048119619488716,
+            -0.0025692922063171864,
+            0.017611365765333176,
+            -0.02306360937654972,
+            0.0036980770528316498,
+            -0.006923729553818703,
+            0.004623564425855875,
+            -0.0014705180656164885,
+            -0.03509881719946861,
+            -0.00100099784322083,
+            -0.025743262842297554,
+            0.01923774555325508,
+            0.009556914679706097,
+            -0.025867177173495293,
+            0.003932353109121323,
+            0.025650326162576675,
+            -0.0003695656778290868,
+            0.00821708794683218,
+            0.0004683101433329284,
+            0.00869725737720728,
+            -0.00974278710782528,
+            -0.01493945624679327,
+            0.004801691509783268,
+            0.01531120017170906,
+            -0.01493945624679327,
+            -0.007431004662066698,
+            -0.009510447271168232,
+            0.030560443177819252,
+            0.0006674932083114982,
+            0.004189862869679928,
+            -0.005045648664236069,
+            -0.01858719252049923,
+            -0.0032237162813544273,
+            -0.021142931655049324,
+            0.004367990419268608,
+            -0.0009046735358424485,
+            0.04197607934474945,
+            0.0038161829579621553,
+            -0.02029101923108101,
+            0.0032972905319184065,
+            -0.03136589005589485,
+            0.0005000148667022586,
+            -0.007276111748069525,
+            -0.02004319056868553,
+            0.0015005286550149322,
+            0.0016525175888091326,
+            0.012631546705961227,
+            -0.03255856782197952,
+            -0.009301341138780117,
+            0.018416810780763626,
+            -0.00020680672605521977,
+            -0.010687636211514473,
+            -0.017828216776251793,
+            0.00035189816844649613,
+            -0.011338187381625175,
+            -0.027973726391792297,
+            -0.02850036323070526,
+            -0.0026719090528786182,
+            0.013491204008460045,
+            -0.018277406692504883,
+            -0.014993669465184212,
+            0.0031249718740582466,
+            0.021592123433947563,
+            -0.009781510569155216,
+            0.008302279748022556,
+            0.02241305634379387,
+            -0.008255811408162117,
+            -0.005514200776815414,
+            -0.000941460661124438,
+            -0.009130958467721939,
+            0.004267309792339802,
+            0.006300284061580896,
+            0.043339136987924576,
+            -0.032403673976659775,
+            -0.0018238683696836233,
+            0.014815541915595531,
+            0.0034153966698795557,
+            0.0025905899237841368,
+            0.002503462601453066,
+            -0.0007841471815481782,
+            0.01344473659992218,
+            0.01178737822920084,
+            0.015466093085706234,
+            0.00100099784322083,
+            -0.0033011629711836576,
+            0.006408709101378918,
+            -0.021173911169171333,
+            0.015721667557954788,
+            0.027958236634731293,
+            -0.0033999073784798384,
+            0.005525817628949881,
+            -0.013731288723647594,
+            -0.008759214542806149,
+            0.013134949840605259,
+            0.0026641644071787596,
+            -0.005068882368505001,
+            -0.009394276887178421,
+            0.017967619001865387,
+            0.019082851707935333,
+            0.006668155547231436,
+            -0.037453193217515945,
+            -0.02898053266108036,
+            -0.009959637187421322,
+            0.020569827407598495,
+            0.014916222542524338,
+            0.0010339126456528902,
+            -0.010540487244725227,
+            -0.009595638141036034,
+            -0.011903548613190651,
+            0.010989678092300892,
+            0.02721474878489971,
+            -0.032372694462537766,
+            0.0008214184199459851,
+            -0.029383255168795586,
+            0.03068435750901699,
+            0.0020000594668090343,
+            0.011446612887084484,
+            -0.0008185141487047076,
+            -0.015140817500650883,
+            -0.029181893914937973,
+            0.013824224472045898,
+            -0.004395096562802792,
+            0.001024231780320406,
+            -0.0031753121875226498,
+            0.0201051477342844,
+            0.012887120246887207,
+            -0.0014472840121015906,
+            0.01968693546950817,
+            0.008178364485502243,
+            0.016480645164847374,
+            -0.010153254494071007,
+            -0.04135650396347046,
+            0.019965743646025658,
+            0.007562664337456226,
+            0.014219203032553196,
+            0.004739734344184399,
+            0.004588713403791189,
+            -0.020771188661456108,
+            0.002724185585975647,
+            0.004333139397203922,
+            0.014405074529349804,
+            0.011028401553630829,
+            0.0025615475606173277,
+            0.011678952723741531,
+            -0.02228914201259613,
+            0.01929970271885395,
+            -0.01631026156246662,
+            0.01006806269288063,
+            0.004445436876267195,
+            -0.026873983442783356,
+            0.007547175046056509,
+            0.007822110317647457,
+            -0.0019187404541298747,
+            -0.017549408599734306,
+            0.02160761132836342,
+            -0.0034831624943763018,
+            -0.03509881719946861,
+            0.0360281765460968,
+            -0.0043447562493383884,
+            0.019346170127391815,
+            0.03209388628602028,
+            -0.01771979033946991,
+            -0.01993476413190365,
+            0.00906900130212307,
+            0.010408828034996986,
+            -0.002873270306736231,
+            0.017859194427728653,
+            -0.002919738180935383,
+            -0.02232012152671814,
+            0.017766257748007774,
+            -0.028918573632836342,
+            -0.014234691858291626,
+            0.020337486639618874,
+            -0.03624502569437027,
+            -0.0021026763133704662,
+            0.03175312280654907,
+            -0.024070415645837784,
+            0.005734923295676708,
+            0.005196669604629278,
+            -0.0019351978553459048,
+            0.0012236569309607148,
+            -0.006451305001974106,
+            0.03801080957055092,
+            -0.010865762829780579,
+            0.0201051477342844,
+            0.01929970271885395,
+            -0.023388884961605072,
+            0.01977987214922905,
+            -0.001109423115849495,
+            0.01675945147871971,
+            -0.003940097521990538,
+            0.03466511517763138,
+            -0.006017603911459446,
+            -0.011330443434417248,
+            0.006431943271309137,
+            0.008488151244819164,
+            -0.009959637187421322,
+            0.008681767620146275,
+            -0.018122512847185135,
+            0.027152791619300842,
+            0.02091059274971485,
+            -0.0002691270783543587,
+            0.013800990767776966,
+            0.0004421719058882445,
+            -0.020833145827054977,
+            0.00789568480104208,
+            -0.027121812105178833,
+            -0.009967382065951824,
+            0.01646515540778637,
+            -0.005762029904872179,
+            0.007380664348602295,
+            -0.02064727433025837,
+            0.028004704043269157,
+            0.0077524082735180855,
+            -0.013591884635388851,
+            -0.0008824076503515244,
+            0.023760627955198288,
+            -0.005432881414890289,
+            0.009355553425848484,
+            0.02070923149585724,
+            0.015713922679424286,
+            0.013615119270980358,
+            0.016263794153928757,
+            0.03351890295743942,
+            0.005328328814357519,
+            0.005568413063883781,
+            0.003519949736073613,
+            0.016991792246699333,
+            -0.008557853288948536,
+            0.018029578030109406,
+            0.0008093173382803798,
+            -0.013576395809650421,
+            0.021499186754226685,
+            -0.0018684001406654716,
+            -0.013622863218188286,
+            -0.0040891822427511215,
+            -0.004337011836469173,
+            0.03271345794200897,
+            0.022041313350200653,
+            -0.004468671046197414,
+            -0.024442158639431,
+            -0.00394784240052104,
+            0.016682006418704987,
+            -0.003167567541822791,
+            0.011136826127767563,
+            0.0007952801533974707,
+            -0.01586107164621353,
+            -0.009595638141036034,
+            0.0063583687879145145,
+            0.010586955584585667,
+            -0.008921852335333824,
+            0.010346870869398117,
+            -0.0228312686085701,
+            -0.004011735785752535,
+            0.00046613195445388556,
+            -0.004844287410378456,
+            0.0053438181057572365,
+            -0.011314953677356243,
+            0.004948840010911226,
+            0.016062432900071144,
+            0.01026167906820774,
+            0.004832670092582703,
+            0.006989559158682823,
+            0.012066186405718327,
+            0.009161937050521374,
+            -0.01627928391098976,
+            0.00428667152300477,
+            -0.005072754807770252,
+            -0.02345084212720394,
+            0.005831731949001551,
+            0.00902253296226263,
+            -0.01122201792895794,
+            0.009386532939970493,
+            -0.010958699509501457,
+            0.03296129032969475,
+            0.01030814740806818,
+            -0.037391237914562225,
+            -0.0033379499800503254,
+            -0.0016244432190433145,
+            0.014885243959724903,
+            0.012298526242375374,
+            0.008550108410418034,
+            -0.017518429085612297,
+            -0.0047823297791182995,
+            -0.009835722856223583,
+            0.0012827098835259676,
+            -0.012306271120905876,
+            0.017456471920013428,
+            0.022567950189113617,
+            -0.036337960511446,
+            0.01968693546950817,
+            -0.010432061739265919,
+            -0.023652203381061554,
+            0.020476890727877617,
+            0.011500826105475426,
+            0.01942361705005169,
+            -0.009402021765708923,
+            0.018896980211138725,
+            -0.0001331114035565406,
+            0.025510922074317932,
+            -0.004228586331009865,
+            -0.022087780758738518,
+            -0.012499887496232986,
+            -0.03209388628602028,
+            -0.0051811798475682735,
+            0.006819176487624645,
+            -0.00045524101005867124,
+            0.013111716136336327,
+            0.014226946979761124,
+            -0.025712283328175545,
+            0.007295473478734493,
+            0.032589543610811234,
+            0.014637414366006851,
+            0.023311438038945198,
+            -0.011276230216026306,
+            -0.0010794125264510512,
+            -0.0026060794480144978,
+            -0.013770012184977531,
+            -0.03212486580014229,
+            -0.0017493258928880095,
+            -0.036585792899131775,
+            0.005680711008608341,
+            -0.016387708485126495,
+            -0.013096226379275322,
+            0.023264970630407333,
+            -0.034138478338718414,
+            0.0014608372002840042,
+            0.013708055019378662,
+            0.010540487244725227,
+            0.012391461990773678,
+            -0.00162637943867594,
+            0.015032391995191574,
+            -0.029243851080536842,
+            -0.010284913703799248,
+            -0.004491904750466347,
+            -0.01981084980070591,
+            -0.007338068913668394,
+            -0.018060555681586266,
+            -0.013127204962074757,
+            -0.003566417610272765,
+            -0.010060317814350128,
+            0.002247888594865799,
+            0.0032062907703220844,
+            -0.014714861288666725,
+            0.03206290677189827,
+            0.003157886676490307,
+            0.018958937376737595,
+            0.0033418224193155766,
+            0.004449309315532446,
+            0.018788553774356842,
+            -0.0026331855915486813,
+            -0.0058278595097362995,
+            0.030963165685534477,
+            -0.00032672801171429455,
+            0.008759214542806149,
+            0.019082851707935333,
+            0.004724245052784681,
+            0.0038529701996594667,
+            -0.0260530486702919,
+            0.012887120246887207,
+            -0.027447087690234184,
+            0.005080499220639467,
+            -0.0077175572514534,
+            -0.024302754551172256,
+            -0.028128618374466896,
+            -0.006571347359567881,
+            0.012058441527187824,
+            -0.02898053266108036,
+            -0.00613377382978797,
+            0.002315654419362545,
+            -0.00400399137288332,
+            -0.002673845272511244,
+            0.007241260726004839,
+            0.029150914400815964,
+            -0.033209118992090225,
+            0.008751469664275646,
+            0.005595519673079252,
+            -0.0005053392960689962,
+            0.013297587633132935,
+            0.008038961328566074,
+            0.003343758638948202,
+            0.020848635584115982,
+            0.006637176964432001,
+            0.002458930714055896,
+            -0.0025576751213520765,
+            -0.014242436736822128,
+            0.004000118933618069,
+            0.02154565416276455,
+            0.007562664337456226,
+            0.018385831266641617,
+            -0.008914108388125896,
+            0.002621568739414215,
+            -0.01455222349613905,
+            0.005552923772484064,
+            -0.002867461647838354,
+            0.004309905227273703,
+            -0.012236569076776505,
+            0.008093173615634441,
+            0.0028771425131708384,
+            -0.0043447562493383884,
+            0.018060555681586266,
+            -0.002319526858627796,
+            0.016480645164847374,
+            0.004015608225017786,
+            0.015620986931025982,
+            -0.0035412474535405636,
+            0.01178737822920084,
+            0.0005731051205657423,
+            -0.017905661836266518,
+            -0.017022771760821342,
+            -0.039249956607818604,
+            0.007702067960053682,
+            0.03531566634774208,
+            -0.0009564659558236599,
+            -0.03444826602935791,
+            0.010145509615540504,
+            -0.011942271143198013,
+            -0.009975126944482327,
+            -0.022862248122692108,
+            0.013862947933375835,
+            0.012430185452103615,
+            0.007415515370666981,
+            0.012252057902514935,
+            -0.006997303571552038,
+            0.01026167906820774,
+            -0.005216030869632959,
+            0.008356492035090923,
+            -0.0010823167394846678,
+            -0.037422213703393936,
+            0.012344993650913239,
+            0.02216522768139839,
+            -0.002635121811181307,
+            0.0016089539276435971,
+            -0.006145390681922436,
+            0.022103270515799522,
+            -0.013971373438835144,
+            -0.00650551775470376,
+            -0.00046443779137916863,
+            0.00797700323164463,
+            -0.01795213110744953,
+            0.004778457339853048,
+            0.0755879208445549,
+            -0.007775642443448305,
+            0.006122156511992216,
+            0.009270362555980682,
+            -0.015148562379181385,
+            -0.01637221872806549,
+            -0.004561606794595718,
+            -0.020260039716959,
+            0.008844406343996525,
+            -0.0018819533288478851,
+            -0.0110981035977602,
+            0.012468908913433552,
+            0.0053050946444272995,
+            0.002031038049608469,
+            -0.008379725739359856,
+            0.010811550542712212,
+            0.009525936096906662,
+            0.010432061739265919,
+            0.02360573597252369,
+            0.011477591469883919,
+            -0.00858883187174797,
+            -0.011795123107731342,
+            0.01129171997308731,
+            0.0028577810153365135,
+            0.016697494313120842,
+            0.007907302118837833,
+            0.0010648912284523249,
+            0.019501063972711563,
+            -0.002309845993295312,
+            0.02180897258222103,
+            -2.2689444449497387e-05,
+            -0.009301341138780117,
+            -0.004236331209540367,
+            -0.015349923633038998,
+            0.024271776899695396,
+            -0.0020155487582087517,
+            0.0015450604259967804,
+            -0.020213572308421135,
+            0.0013979118084535003,
+            0.013243375346064568,
+            0.011957760900259018,
+            -0.0020407189149409533,
+            -0.0051347119733691216,
+            0.0007154133054427803,
+            -0.016356728971004486,
+            0.032806396484375,
+            0.007578153628855944,
+            0.0007681738352403045,
+            0.0008838597568683326,
+            0.0009288755827583373,
+            -0.0032972905319184065,
+            0.0024608669336885214,
+            -0.004766840487718582,
+            0.009549170732498169,
+            -0.010943209752440453,
+            -0.010796061716973782,
+            0.010075807571411133,
+            -0.02402394637465477,
+            0.003566417610272765,
+            -0.00558777479454875,
+            -0.008062195032835007,
+            0.0076207490637898445,
+            0.00015537731815129519,
+            -0.007392281666398048,
+            -0.0030939930584281683,
+            0.0023408245760947466,
+            0.013429246842861176,
+            0.004406713880598545,
+            0.010850274004042149,
+            0.012956822291016579,
+            0.005359307397156954,
+            -0.006629432085901499,
+            0.015512561425566673,
+            -0.006946963258087635,
+            -0.004228586331009865,
+            -0.0012468908680602908,
+            -0.019996721297502518,
+            -0.0009419447160325944,
+            -0.004708755761384964,
+            0.0015092412941157818,
+            0.0001705036120256409,
+            -8.749050175538287e-05,
+            -0.0018490385264158249,
+            -0.011988739483058453,
+            -0.001831613015383482,
+            -0.011663463898003101,
+            -0.014498010277748108,
+            0.010253935120999813,
+            -0.0041201612912118435,
+            -0.022939693182706833,
+            -0.015349923633038998,
+            -0.020662762224674225,
+            0.02145271934568882,
+            -0.004387352149933577,
+            -0.005800753366202116,
+            0.006733985152095556,
+            -0.010215211659669876,
+            -0.015706177800893784,
+            -0.008069939911365509,
+            -0.0205853171646595,
+            0.010703125037252903,
+            0.003214035416021943,
+            0.0006791101768612862,
+            0.00882891658693552,
+            0.014397329650819302,
+            -0.01589204929769039,
+            0.0111755495890975,
+            0.012182355858385563,
+            0.011361422017216682,
+            -0.006854027509689331,
+            0.0008911203476600349,
+            -0.014103032648563385,
+            0.02596011385321617,
+            0.021623101085424423,
+            0.005169562995433807,
+            -0.04132552444934845,
+            -0.005862710531800985,
+            -0.013344055972993374,
+            -0.010524998418986797,
+            -0.012120398692786694,
+            0.011771888472139835,
+            -0.0038820127956569195,
+            0.0013117524795234203,
+            0.006288666743785143,
+            0.0015063370810821652,
+            0.004367990419268608,
+            -0.0022382077295333147,
+            -0.007709812838584185,
+            -0.03212486580014229,
+            -0.02817508764564991,
+            -0.00863530021160841,
+            0.020461400970816612,
+            0.002838419284671545,
+            -0.0009453330421820283,
+            -0.0036380558740347624,
+            0.0027048238553106785,
+            -0.005882072262465954,
+            -0.005018542055040598,
+            -0.014753584749996662,
+            -0.012739972211420536,
+            0.0101300198584795,
+            0.00802347157150507,
+            0.014195968396961689,
+            -0.011950016021728516,
+            0.0028190575540065765,
+            0.018788553774356842,
+            0.007690451107919216,
+            -0.023357905447483063,
+            0.01135367713868618,
+            -0.010912231169641018,
+            -0.009270362555980682,
+            -0.009433000348508358,
+            -0.009471723809838295,
+            0.0016980175860226154,
+            -0.008317768573760986,
+            0.003945906180888414,
+            -0.03615209087729454,
+            -0.011330443434417248,
+            -0.0013901671627536416,
+            -0.010346870869398117,
+            0.0009864765452221036,
+            0.02524760365486145,
+            0.006815304048359394,
+            0.01432762760668993,
+            -0.0006733016925863922,
+            0.017502939328551292,
+            -0.0018984107300639153,
+            0.018726596608757973,
+            0.012763205915689468,
+            -0.00797700323164463,
+            0.011717676185071468,
+            0.014738094992935658,
+            -0.0033863543067127466,
+            0.015620986931025982,
+            -0.010168743319809437,
+            0.005285732913762331,
+            -0.009556914679706097,
+            -0.003214035416021943,
+            0.002495717955753207,
+            0.030901208519935608,
+            -0.003618694143369794,
+            0.00858883187174797,
+            0.01098193321377039,
+            -0.015241498127579689,
+            0.0026951429899781942,
+            0.014962689951062202,
+            0.01141563430428505,
+            0.007760153152048588,
+            0.0024434414226561785,
+            -0.024225307628512383,
+            0.004193735308945179,
+            0.005037903785705566,
+            0.0031520782504230738,
+            -0.014366351068019867,
+            0.0021859314292669296,
+            0.015675200149416924,
+            0.008550108410418034,
+            -0.025325050577521324,
+            0.016418687999248505,
+            0.0009138703462667763,
+            -0.003909118939191103,
+            0.029119934886693954,
+            -0.011268485337495804,
+            -0.005723306443542242,
+            0.009463978931307793,
+            0.0009980935137718916,
+            0.0030029932968318462,
+            0.008937342092394829,
+            0.0228312686085701,
+            -0.003988501615822315,
+            -0.032341714948415756,
+            -0.022459525614976883,
+            0.015706177800893784,
+            -0.0047823297791182995,
+            -0.022939693182706833,
+            -0.026703599840402603,
+            -0.003113354789093137,
+            -0.02228914201259613,
+            -0.0022323993034660816,
+            -0.020058678463101387,
+            -0.014056564308702946,
+            0.001268188701942563,
+            0.029987338930368423,
+            0.016201836988329887,
+            0.008224832825362682,
+            -0.011051635257899761,
+            -0.029956359416246414,
+            -0.024271776899695396,
+            0.01810702309012413,
+            -0.007055388763546944,
+            -0.005541306920349598,
+            0.00591692328453064,
+            -0.009951893240213394,
+            -0.008480406366288662,
+            0.011802867986261845,
+            0.027137301862239838,
+            -0.002747419523075223,
+            -0.029987338930368423,
+            -0.02688947319984436,
+            -0.014335372485220432,
+            -0.01840132102370262,
+            -0.017704300582408905,
+            -0.008627555333077908,
+            0.0045035220682621,
+            0.00030300996149890125,
+            0.016186347231268883,
+            0.014962689951062202,
+            -0.000480653194244951,
+            0.0028597170021384954,
+            -2.7711375878425315e-05,
+            -0.001763847190886736,
+            0.006888878531754017,
+            -0.004933350719511509,
+            0.026997897773981094,
+            0.014614180661737919,
+            -0.006594581063836813,
+            0.018711106851696968,
+            0.028314489871263504,
+            0.020430423319339752,
+            0.003157886676490307,
+            -0.010695381090044975,
+            -0.014606435783207417,
+            0.009727297350764275,
+            -0.013421501964330673,
+            0.023264970630407333,
+            0.022552460432052612,
+            -0.01480779703706503,
+            -0.014885243959724903,
+            -0.017905661836266518,
+            0.015210519544780254,
+            -0.007562664337456226,
+            -0.008224832825362682,
+            0.012368228286504745,
+            0.022335609421133995,
+            -0.007024410180747509,
+            0.002371803391724825,
+            -0.0010232636705040932,
+            -0.021731525659561157,
+            -0.006122156511992216,
+            -0.004596457816660404,
+            -0.008666278794407845,
+            0.008681767620146275,
+            0.008131897076964378,
+            -0.001928421319462359,
+            -0.006319645792245865,
+            0.005471604876220226,
+            -0.013367289677262306,
+            0.01301103550940752,
+            -0.00789568480104208,
+            0.03379771113395691,
+            -0.009138703346252441,
+            -0.0006733016925863922,
+            -0.022598929703235626,
+            -0.011446612887084484,
+            0.00845717266201973,
+            0.0017967619933187962,
+            0.01682141050696373,
+            -0.0011626676423475146,
+            -0.005688455421477556,
+            0.009913169778883457,
+            -0.0010687635513022542,
+            -0.001450188341550529,
+            -0.006892750971019268,
+            -0.010401083156466484,
+            0.0002884887217078358,
+            0.013258864171802998,
+            -0.006025348324328661,
+            0.0018054747488349676,
+            0.006095050368458033,
+            -0.0175339188426733,
+            0.029027000069618225,
+            0.023884542286396027,
+            0.017255110666155815,
+            0.014660648070275784,
+            0.0006394187803380191,
+            0.011647974140942097,
+            -0.003572226269170642,
+            0.0038820127956569195,
+            -0.020337486639618874,
+            0.021080974489450455,
+            -0.010811550542712212,
+            0.01846327818930149,
+            0.011655719019472599,
+            -0.015233753249049187,
+            -0.005022414494305849,
+            0.021406250074505806,
+            0.009967382065951824,
+            0.00889087375253439,
+            0.0017541663255542517,
+            -0.02470547705888748,
+            0.017766257748007774,
+            0.010099041275680065,
+            -0.020399443805217743,
+            0.019547531381249428,
+            -0.009471723809838295,
+            0.01233724970370531,
+            -0.0005300253978930414,
+            0.004418330732733011,
+            0.0058743273839354515,
+            0.021499186754226685,
+            0.011570527218282223,
+            -0.03884723410010338,
+            -0.013615119270980358,
+            -0.0017357728211209178,
+            0.006857899948954582,
+            -0.0036303112283349037,
+            -0.015233753249049187,
+            -0.041635312139987946,
+            0.01604694314301014,
+            0.009998360648751259,
+            -0.01643417589366436,
+            0.004987563472241163,
+            -0.002427952131256461,
+            -0.0042053526267409325,
+            -0.008170620538294315,
+            0.003186929039657116,
+            -0.004170501604676247,
+            0.02324948087334633,
+            0.0018984107300639153,
+            -0.006946963258087635,
+            0.013258864171802998,
+            0.01344473659992218,
+            -0.021948376670479774,
+            0.0009874446550384164,
+            0.0021375271026045084,
+            0.000636998622212559,
+            0.023683181032538414,
+            -0.014707116410136223,
+            -0.0017657832941040397,
+            -0.01685238815844059,
+            -0.0013069120468571782,
+            -0.021592123433947563,
+            -0.001188805908896029,
+            -0.015187285840511322,
+            -0.008511385880410671,
+            0.025030752643942833,
+            -0.011005166918039322,
+            0.010997422970831394,
+            -0.006687517277896404,
+            0.006943091284483671,
+            -0.006904367823153734,
+            -0.01936165988445282,
+            -0.0008267428493127227,
+            0.009975126944482327,
+            -0.000512115890160203,
+            0.002941036131232977,
+            8.482827252009884e-05,
+            0.002464739140123129,
+            0.008867640048265457,
+            0.006234454456716776,
+            -0.0075278133153915405,
+            0.02489134855568409,
+            -0.012073930352926254,
+            0.0016399325104430318,
+            0.011911292560398579,
+            0.0004743606550619006,
+            -0.010424317792057991,
+            0.025588368996977806,
+            -0.02139076218008995,
+            -0.00800798274576664,
+            -0.00014799568452872336,
+            -0.0016544538084417582,
+            -0.006513262167572975,
+            0.006718495860695839,
+            0.02946070022881031,
+            0.0062654330395162106,
+            0.010447551496326923,
+            -0.0281441081315279,
+            -0.011493081226944923,
+            0.012538610957562923,
+            -0.022273652255535126,
+            -0.0007739823195151985,
+            -0.0004322490422055125,
+            -0.00619960343465209,
+            0.026378324255347252,
+            -0.02022906206548214,
+            -0.01093546487390995,
+            -0.018664639443159103,
+            0.005405775271356106,
+            -0.011121337302029133,
+            0.008294534869492054,
+            -0.0009777637897059321,
+            0.006540368776768446,
+            0.016805920749902725,
+            -0.019563021138310432,
+            0.016031453385949135,
+            0.005119222681969404,
+            -0.016217326745390892,
+            0.010222956538200378,
+            -0.017580386251211166,
+            -0.0008199662552215159,
+            0.020275529474020004,
+            -0.00400399137288332,
+            0.008751469664275646,
+            -0.003363120136782527,
+            0.005208286456763744,
+            -0.006513262167572975,
+            0.0001492057926952839,
+            -0.007249005138874054,
+            0.014428308233618736,
+            -0.016805920749902725,
+            -0.017828216776251793,
+            -0.008821171708405018,
+            0.0016709112096577883,
+            0.021297825500369072,
+            0.011136826127767563,
+            -0.010044828988611698,
+            -0.011059380136430264,
+            -0.011694442480802536,
+            0.041604332625865936,
+            0.005762029904872179,
+            0.027230238541960716,
+            -0.02261441759765148,
+            0.00545998802408576,
+            0.025170156732201576,
+            -0.014242436736822128,
+            0.0012788375606760383,
+            -0.0022788674104958773,
+            0.0053050946444272995,
+            0.011562783271074295,
+            -0.01819995976984501,
+            0.0049139889888465405,
+            -0.00019216447253711522,
+            -0.00782985519617796,
+            -0.000656844291370362,
+            0.003442503046244383,
+            0.00020196630794089288,
+            0.016325751319527626,
+            -0.004027225077152252,
+            -0.016511622816324234,
+            -0.004429947584867477,
+            -0.0003216455806978047,
+            -0.027896279469132423,
+            0.013514438644051552,
+            0.016589069738984108,
+            0.01566745527088642,
+            0.027989214286208153,
+            -0.014513500034809113,
+            0.016031453385949135,
+            0.009533680975437164,
+            0.013901671394705772,
+            -0.010610189288854599,
+            0.008952830918133259,
+            -0.01595400646328926,
+            0.007446494419127703,
+            -0.03454120084643364,
+            -0.007887939922511578,
+            -0.00802347157150507,
+            0.01984182931482792,
+            0.028763681650161743,
+            0.002923610620200634,
+            0.006493900436908007,
+            0.000196883876924403,
+            0.0024202074855566025,
+            -0.01714668609201908,
+            -0.020182594656944275,
+            -0.0003337466041557491,
+            0.0029332914855331182,
+            0.0006950835813768208,
+            -0.004495777189731598,
+            0.00472037261351943,
+            -0.012391461990773678,
+            0.0026835259050130844,
+            -0.013134949840605259,
+            0.005971135571599007,
+            -0.0059982421807944775,
+            0.009603383019566536,
+            0.0019429425010457635,
+            0.020864123478531837,
+            0.02300165221095085,
+            0.004100799560546875,
+            0.0046971384435892105,
+            -0.024333734065294266,
+            0.00920066051185131,
+            -0.013847459107637405,
+            0.012329504825174809,
+            0.019408127292990685,
+            0.008248066529631615,
+            0.006571347359567881,
+            0.01233724970370531,
+            0.012918098829686642,
+            -0.0031772484071552753,
+            0.024287264794111252,
+            0.005014669615775347,
+            -0.0009070937521755695,
+            0.002052335999906063,
+            -0.014954946003854275,
+            -0.0070979841984808445,
+            -0.0015286030247807503,
+            0.0018848575418815017,
+            -0.004731989465653896,
+            0.01906736195087433,
+            0.015256987884640694,
+            -0.01050176378339529,
+            0.017874684184789658,
+            -0.03937387093901634,
+            0.017379024997353554,
+            -0.0027377386577427387,
+            -0.0060911779291927814,
+            -0.00545998802408576,
+            0.00665653869509697,
+            0.004515138920396566,
+            0.003669034456834197,
+            0.03773200139403343,
+            -0.010408828034996986,
+            0.003963331691920757,
+            -0.005847221240401268,
+            0.007121218368411064,
+            -0.020864123478531837,
+            0.008302279748022556,
+            0.005336073227226734,
+            0.02160761132836342,
+            -0.0056497324258089066,
+            -0.0014482521219179034,
+            0.02328045852482319,
+            0.008859895169734955,
+            0.03469609469175339,
+            -0.01566745527088642,
+            -0.004418330732733011,
+            0.01048627495765686,
+            0.0207402091473341,
+            -0.00580462533980608,
+            -0.008976065553724766,
+            0.007903429679572582,
+            0.0035586729645729065,
+            0.021824462339282036,
+            -0.00515407370403409,
+            -0.0006650729919783771,
+            0.0054367538541555405,
+            -0.0060911779291927814,
+            0.009735042229294777,
+            0.011252996511757374,
+            -0.009076746180653572,
+            0.007539430167526007,
+            0.021530164405703545,
+            -0.009975126944482327,
+            -0.008658533915877342,
+            -0.009649851359426975,
+            0.0010658593382686377,
+            -0.03068435750901699,
+            0.001649613375775516,
+            0.005359307397156954,
+            0.015187285840511322,
+            -0.0004603234410751611,
+            -0.009014789015054703,
+            0.021019017323851585,
+            0.017255110666155815,
+            -0.005936284549534321,
+            -0.0028132491279393435,
+            0.0023795480374246836,
+            -0.0017154429806396365,
+            -0.006160879973322153,
+            -0.0006248975405469537,
+            -0.002656419761478901,
+            0.014730350114405155,
+            -0.0015469965292140841,
+            -0.006617815233767033,
+            -0.00756653631106019,
+            0.0033302053343504667,
+            0.010834784246981144,
+            -0.005661349277943373,
+            0.013057502917945385,
+            0.016805920749902725,
+            -0.00031704717548564076,
+            0.019671445712447166,
+            0.01037010457366705,
+            -0.010447551496326923,
+            0.015938518568873405,
+            -0.007984748110175133,
+            0.006563602481037378,
+            0.0164031982421875,
+            -0.01277095079421997,
+            -0.01246116403490305,
+            0.007210282143205404,
+            0.00665653869509697,
+            0.028918573632836342,
+            0.014792308211326599,
+            -0.02109646424651146,
+            0.014908477663993835,
+            0.015768134966492653,
+            -0.013018779456615448,
+            0.003537375247105956,
+            -0.01784370467066765,
+            0.020260039716959,
+            0.019051872193813324,
+            -0.015535795129835606,
+            -0.017224133014678955,
+            -0.014947201125323772,
+            -0.005092116538435221,
+            0.006331262644380331,
+            0.0033263331279158592,
+            -0.00617636926472187,
+            -0.027989214286208153,
+            -0.0049139889888465405,
+            -0.018633661791682243,
+            0.013870692811906338,
+            -0.008674023672938347,
+            0.0007715621031820774,
+            -0.004511266481131315,
+            0.007984748110175133,
+            -0.0033882902935147285,
+            0.032465629279613495,
+            -0.0051811798475682735,
+            -0.02832997962832451,
+            0.004398969002068043,
+            0.016356728971004486,
+            0.012623801827430725,
+            -0.029243851080536842,
+            0.010463040322065353,
+            0.004805563949048519,
+            -0.012530866079032421,
+            0.005990497302263975,
+            -0.02258343994617462,
+            -0.007632365915924311,
+            -0.0009395244996994734,
+            -0.016743963584303856,
+            -0.027121812105178833,
+            -0.008433938957750797,
+            -0.006993431597948074,
+            -0.005401902832090855,
+            -0.009525936096906662,
+            -0.008720491081476212,
+            -0.011477591469883919,
+            0.02303262986242771,
+            -0.007899557240307331,
+            -0.001866463921032846,
+            -0.0014637414133176208,
+            0.00930908601731062,
+            -0.023776117712259293,
+            -0.015605497173964977,
+            0.006389347370713949,
+            -0.002594462363049388,
+            0.022815778851509094,
+            -0.012747716158628464,
+            0.005940156988799572,
+            -0.009316830895841122,
+            0.009851212613284588,
+            0.015241498127579689,
+            0.004956584889441729,
+            0.017440982162952423,
+            -0.009889936074614525,
+            0.0021026763133704662,
+            -0.008627555333077908,
+            -0.00018986526993103325,
+            -1.027075541060185e-05,
+            0.014428308233618736,
+            0.007624621503055096,
+            0.014141756109893322,
+            0.017487451434135437,
+            -0.013553161174058914,
+            0.005003052763640881,
+            -0.0003274540649726987,
+            -0.017255110666155815,
+            0.0027725896798074245,
+            -0.007984748110175133,
+            -0.0008896682411432266,
+            0.001758038648404181,
+            6.395397940650582e-05,
+            -0.02112744189798832,
+            -0.0009124181815423071,
+            0.0018809852190315723,
+            0.0010222955606877804,
+            -0.025201136246323586,
+            0.00889087375253439,
+            -0.00889087375253439,
+            -0.0014918158994987607,
+            -0.019113829359412193,
+            0.019795360043644905,
+            0.026300877332687378,
+            -0.004619691986590624,
+            0.00036206303047947586,
+            -0.013537672348320484,
+            -0.02049238048493862,
+            0.0001489637652412057,
+            -0.011314953677356243,
+            -0.011345932260155678,
+            -0.0022169100120663643,
+            -0.003214035416021943,
+            0.008805682882666588,
+            0.0004189379105810076,
+            -0.009433000348508358,
+            -0.004956584889441729,
+            -0.011764144524931908,
+            -0.017549408599734306,
+            -0.010401083156466484,
+            0.01933068037033081,
+            -0.0018016024259850383,
+            0.0020794421434402466,
+            -0.0032508226577192545,
+            0.0005271211848594248,
+            0.011748654767870903,
+            0.013475715182721615,
+            0.013344055972993374,
+            -0.023915521800518036,
+            -0.017890173941850662,
+            -0.013320821337401867,
+            0.0029487807769328356,
+            -0.007187047973275185,
+            0.004844287410378456,
+            0.026362836360931396,
+            -0.0009695350890979171,
+            -0.01091997604817152,
+            0.002327271504327655,
+            0.019051872193813324,
+            -0.031272951513528824,
+            -0.013390523381531239,
+            -0.025077221915125847,
+            0.02225816436111927,
+            -0.013351799920201302,
+            0.011516314931213856,
+            0.010191977955400944,
+            -0.025944624096155167,
+            0.006768836174160242,
+            -0.007686578668653965,
+            0.026037558913230896,
+            0.01634124107658863,
+            0.0005934348446317017,
+            -0.00944074522703886,
+            0.013622863218188286,
+            -0.0008722427301108837,
+            -0.013576395809650421,
+            0.002201420720666647,
+            0.014645159244537354,
+            -0.014691627584397793,
+            -0.009510447271168232,
+            -0.004507394507527351,
+            0.004313777666538954,
+            0.012763205915689468,
+            -0.0030339721124619246,
+            -0.004031097516417503,
+            -0.011624740436673164,
+            0.013344055972993374,
+            -0.003930416889488697,
+            0.012383717112243176,
+            0.007380664348602295,
+            -0.014056564308702946,
+            -0.019532041624188423,
+            0.033333033323287964,
+            0.00400399137288332,
+            0.00338054564781487,
+            -0.010997422970831394,
+            0.02740062028169632,
+            -0.009897680021822453,
+            0.010571465827524662,
+            -0.006408709101378918,
+            0.0031985461246222258,
+            0.012321759946644306,
+            -0.012104909867048264,
+            -0.01981084980070591,
+            0.021468207240104675,
+            -0.00364192808046937,
+            -0.0010193913476541638,
+            0.008310024626553059,
+            -0.020182594656944275,
+            -0.00663330452516675,
+            0.002180122770369053,
+            0.009541425853967667,
+            -0.011818356812000275,
+            0.006369986105710268,
+            0.017905661836266518,
+            0.003912991378456354,
+            -0.0007672057254239917,
+            0.019996721297502518,
+            -0.01672847382724285,
+            0.011717676185071468,
+            -0.010850274004042149,
+            0.0016825281782075763,
+            0.0006065039779059589,
+            -0.0048597767017781734,
+            0.0058743273839354515,
+            -0.014536733739078045,
+            0.011988739483058453,
+            -6.244135147426277e-05,
+            -0.013979118317365646,
+            0.007601387333124876,
+            -0.013359544798731804,
+            -0.016836898401379585,
+            0.041542377322912216,
+            -0.005301222205162048,
+            -0.0038665232714265585,
+            -0.0056962003000080585,
+            0.01350669376552105,
+            0.02823704481124878,
+            -0.005336073227226734,
+            -0.01153180468827486,
+            -0.010780571959912777,
+            -0.016573579981923103,
+            0.015907539054751396,
+            -0.01810702309012413,
+            0.002495717955753207,
+            -0.010532742366194725,
+            -0.007903429679572582,
+            0.004302160814404488,
+            -0.003516077296808362,
+            0.014722606167197227,
+            0.00950270239263773,
+            0.023357905447483063,
+            0.00797700323164463,
+            -0.00027348342700861394,
+            0.010393339209258556,
+            0.011663463898003101,
+            0.003243078012019396,
+            -0.010509508661925793,
+            -0.00011949774489039555,
+            -0.02894955314695835,
+            -0.008449427783489227,
+            -0.004863648675382137,
+            -0.001485039247199893,
+            -0.00428667152300477,
+            -0.01239920686930418,
+            0.006617815233767033,
+            -0.0022052929271012545,
+            0.013909416273236275,
+            -0.009254872798919678,
+            -0.017766257748007774,
+            0.03389064967632294,
+            0.004495777189731598,
+            0.006466794293373823,
+            -0.008286789990961552,
+            -0.03705047070980072,
+            -0.0032566310837864876,
+            -0.008116407319903374,
+            -0.013529927469789982,
+            0.004538373090326786,
+            0.01283290795981884,
+            -0.014831030741333961,
+            -0.028097640722990036,
+            0.0019390701781958342,
+            -0.006683644838631153,
+            0.02978597767651081,
+            -0.007798876613378525,
+            0.00419760774821043,
+            0.0037251831963658333,
+            0.0011558911064639688,
+            0.01239920686930418,
+            -0.00845717266201973,
+            -0.0022536972537636757,
+            0.016387708485126495,
+            -0.010858018882572651,
+            0.011051635257899761,
+            -0.010966443456709385,
+            -0.02554190158843994,
+            -0.01456771232187748,
+            0.021344292908906937,
+            -0.0004823473282158375,
+            -0.00024903935263864696,
+            0.014823286794126034,
+            0.02354377694427967,
+            0.011260741390287876,
+            0.00969631876796484,
+            0.006408709101378918,
+            -0.001035848748870194,
+            0.005975008010864258,
+            0.006095050368458033,
+            -0.0024163350462913513,
+            -0.013398268260061741,
+            0.017564896494150162,
+            0.009773765690624714,
+            -0.01810702309012413,
+            0.0008514289511367679,
+            0.0036612898111343384,
+            0.0168833676725626,
+            0.006935346405953169,
+            0.009316830895841122,
+            -0.0143121387809515,
+            -0.023760627955198288,
+            -0.00014932679187040776,
+            0.002873270306736231,
+            0.015721667557954788,
+            0.010579210706055164,
+            -0.014319883659482002,
+            0.0005963391158729792,
+            0.008712747134268284,
+            0.020213572308421135,
+            -0.024209819734096527,
+            0.019702425226569176,
+            -0.004673904739320278,
+            -0.0021084847394376993,
+            0.007946024648845196,
+            0.011632485315203667,
+            -0.011872569099068642,
+            -0.00650551775470376,
+            0.006249943748116493,
+            0.003157886676490307,
+            0.03172214329242706,
+            -0.0017260919557884336,
+            0.009665340185165405,
+            -0.010997422970831394,
+            -0.0002405686245765537,
+            0.0006728176376782358,
+            -0.0006820144481025636,
+            -0.002884887158870697,
+            -0.011020656675100327,
+            0.0018974426202476025,
+            -0.020306508988142014,
+            0.0024473138619214296,
+            0.00463130883872509,
+            -0.005684582982212305,
+            -0.0032972905319184065,
+            -0.019222255796194077,
+            -0.0022672503255307674,
+            0.003293418325483799,
+            0.014335372485220432,
+            -0.031691163778305054,
+            0.004972074180841446,
+            0.010672146454453468,
+            -0.002309845993295312,
+            -0.010416572913527489,
+            0.011423379182815552,
+            -0.0026060794480144978,
+            0.011988739483058453,
+            -0.0038142469711601734,
+            -0.0004905760288238525,
+            0.025681305676698685,
+            -0.005308967083692551,
+            0.014707116410136223,
+            -0.01026167906820774,
+            -0.002993312431499362,
+            -0.028252532705664635,
+            0.005235392600297928,
+            -0.007403898518532515,
+            -0.0012081675231456757,
+            -0.004402841441333294,
+            -0.01130720879882574,
+            -0.0026990154292434454,
+            0.002122037811204791,
+            0.008000237867236137,
+            0.012910354882478714,
+            -0.0007725302129983902,
+            -0.010075807571411133,
+            0.004832670092582703,
+            0.0075355577282607555,
+            -0.015520306304097176,
+            -0.0070437719114124775,
+            -0.020306508988142014,
+            0.029662061482667923,
+            -0.013731288723647594,
+            -0.0011123273288831115,
+            -0.020864123478531837,
+            0.010160998441278934,
+            -0.0063119009137153625,
+            -0.002782270545139909,
+            0.004708755761384964,
+            -0.015109838917851448,
+            0.01344473659992218,
+            0.008751469664275646,
+            0.01450575515627861,
+            -0.02151467651128769,
+            -0.01762685552239418,
+            -0.0044725434854626656,
+            -0.002735802438110113,
+            -0.010648912750184536,
+            -0.0005334137240424752,
+            0.002013612538576126,
+            0.021251358091831207,
+            -0.00032503384863957763,
+            0.0042053526267409325,
+            -0.003214035416021943,
+            0.017224133014678955,
+            -0.010137764737010002,
+            -0.013057502917945385,
+            0.0024260159116238356,
+            0.005994369741529226,
+            0.016186347231268883,
+            0.03158273920416832,
+            -0.020693741738796234,
+            -0.00272999401204288,
+            0.0032740565948188305,
+            0.010656657628715038,
+            -0.004085310269147158,
+            6.304640555754304e-05,
+            -0.027803342789411545,
+            -0.04464024305343628,
+            -0.002422143705189228,
+            0.013057502917945385,
+            -0.015636475756764412,
+            0.012670270167291164,
+            0.008743725717067719,
+            0.010021595284342766,
+            -0.005196669604629278,
+            0.02620794251561165,
+            -0.004793947096914053,
+            0.014304393902420998,
+            0.011733165942132473,
+            0.011911292560398579,
+            0.0009874446550384164,
+            0.0007459079497493804,
+            -0.018354853615164757,
+            -0.008255811408162117,
+            -0.0015954007394611835,
+            0.01765783317387104,
+            -0.027478067204356194,
+            0.008526874706149101,
+            -0.011516314931213856,
+            0.013839714229106903,
+            0.0003666614356916398,
+            0.005661349277943373,
+            -0.01771979033946991,
+            0.012445674277842045,
+            -0.0009337160154245794,
+            -0.025325050577521324,
+            0.004805563949048519,
+            0.011276230216026306,
+            -0.010362359695136547,
+            0.006877261213958263,
+            0.019950253888964653,
+            0.0023079097736626863,
+            0.008495896123349667,
+            -0.014126266352832317,
+            0.010703125037252903,
+            0.009649851359426975,
+            -0.021080974489450455,
+            -0.01233724970370531,
+            -0.007059261202812195,
+            -0.006335135083645582,
+            -0.0035470561124384403,
+            -0.00013105422840453684,
+            0.010176488198339939,
+            -0.016000475734472275,
+            0.010099041275680065,
+            -0.006559730041772127,
+            0.006400964688509703,
+            0.00944074522703886,
+            0.001115231541916728,
+            -0.013576395809650421,
+            0.012716737575829029,
+            -0.0218709297478199,
+            0.008426194079220295,
+            -0.04287445917725563,
+            -0.005847221240401268,
+            -0.015078860335052013,
+            0.005204414017498493,
+            -0.003798757679760456,
+            -0.004453181754797697,
+            -0.006203475873917341,
+            -0.005061137955635786,
+            0.00920066051185131,
+            0.021747015416622162,
+            0.012135888449847698,
+            -0.016620049253106117,
+            -0.007551047019660473,
+            0.005425137002021074,
+            0.0004854935978073627,
+            -0.013336311094462872,
+            0.009316830895841122,
+            -0.01576039008796215,
+            0.0009259713697247207,
+            0.006339007057249546,
+            -0.01072635967284441,
+            0.013452480547130108,
+            0.004731989465653896,
+            -0.003401843598112464,
+            0.013390523381531239,
+            -0.0050417762249708176,
+            -0.0057891360484063625,
+            -0.007272239308804274,
+            -0.013739033602178097,
+            -0.003818119177594781,
+            -0.003072695340961218,
+            0.012747716158628464,
+            0.001529571134597063,
+            -0.0003243077953811735,
+            -0.00782985519617796,
+            0.009154192171990871,
+            0.002220782218500972,
+            -0.0013146566925570369,
+            0.005773646757006645,
+            -0.02300165221095085,
+            0.0001287550403503701,
+            0.004755223635584116,
+            -0.0009487213101238012,
+            -0.006246071308851242,
+            -0.012151377275586128,
+            -0.011903548613190651,
+            -0.038165703415870667,
+            0.009363298304378986,
+            0.0019197085639461875,
+            -0.0076207490637898445,
+            0.0064822835847735405,
+            0.017255110666155815,
+            0.01456771232187748,
+            0.016743963584303856,
+            -0.020693741738796234,
+            -0.02303262986242771,
+            0.005556796211749315,
+            -0.009402021765708923,
+            0.02261441759765148,
+            0.004046586807817221,
+            -0.005847221240401268,
+            0.0013243375578895211,
+            -0.013258864171802998,
+            0.008929597213864326,
+            0.005723306443542242,
+            -0.001592496526427567,
+            0.005425137002021074,
+            -0.013692565262317657,
+            -0.010904486291110516,
+            0.01048627495765686,
+            0.032496608793735504,
+            -0.0004261985304765403,
+            0.0024105266202241182,
+            0.023295948281884193,
+            0.015466093085706234,
+            0.016356728971004486,
+            -0.021111954003572464,
+            0.011167804710566998,
+            0.010060317814350128,
+            -0.0018645278178155422,
+            -0.014892988838255405,
+            0.00513083953410387,
+            -0.012716737575829029,
+            -0.010664401575922966,
+            0.014993669465184212,
+            0.007853088900446892,
+            0.017735280096530914,
+            -0.02369867078959942,
+            0.00789568480104208,
+            0.00042765066609717906,
+            0.00013710475468542427,
+            0.011524059809744358,
+            0.012956822291016579,
+            0.03370477631688118,
+            0.027152791619300842,
+            0.020662762224674225,
+            -0.009285852313041687,
+            0.014366351068019867,
+            -0.009192915633320808,
+            -0.011090358719229698,
+            -0.019470084458589554,
+            -0.015249243006110191,
+            -0.011136826127767563,
+            0.017998598515987396,
+            0.008790193125605583,
+            -0.009355553425848484,
+            -0.005947901867330074,
+            0.01586107164621353,
+            -0.003909118939191103,
+            -0.004635181277990341,
+            0.01646515540778637,
+            -0.026269899681210518,
+            0.016836898401379585,
+            -0.000745423894841224,
+            -0.014668392948806286,
+            -0.014691627584397793,
+            -0.004182118456810713,
+            -0.012670270167291164,
+            0.0003964299685321748,
+            -0.020817656069993973,
+            0.013181418180465698,
+            0.024597052484750748,
+            -0.0030920570716261864,
+            -0.004100799560546875,
+            -0.011299464851617813,
+            -0.0005111477803438902,
+            0.018385831266641617,
+            0.0034773540683090687,
+            0.009192915633320808,
+            -0.0023872926831245422,
+            0.013924905098974705,
+            0.00524313747882843,
+            0.0027958236169070005,
+            -0.0007565568666905165,
+            -0.016263794153928757,
+            0.012197845615446568,
+            0.0026835259050130844,
+            0.012135888449847698,
+            -0.022242674604058266,
+            0.007938280701637268,
+            -0.011764144524931908,
+            -0.0023698671720921993,
+            -0.02228914201259613,
+            0.0189744271337986,
+            -0.01610890030860901,
+            0.001046497724018991,
+            0.003841353114694357,
+            0.0006224773242138326,
+            0.009580149315297604,
+            0.01610890030860901,
+            -0.023388884961605072,
+            -0.0003460896550677717,
+            -0.014908477663993835,
+            -0.021917399019002914,
+            -0.01607792265713215,
+            0.004728117026388645,
+            -0.019501063972711563,
+            0.005444498732686043,
+            -0.0025557391345500946,
+            -0.015125328674912453,
+            0.018478767946362495,
+            0.014691627584397793,
+            -0.004379607271403074,
+            0.002412462839856744,
+            0.005425137002021074,
+            -0.0014405074762180448,
+            0.01971791312098503,
+            0.00013274837692733854,
+            -0.0010048701660707593,
+            -0.009425255469977856,
+            -0.0168833676725626,
+            -0.0065752193331718445,
+            -0.035005878657102585,
+            0.0260530486702919,
+            -0.004670032300055027,
+            -0.016511622816324234,
+            0.014319883659482002,
+            0.009115469641983509,
+            -0.014629669487476349,
+            -0.007059261202812195,
+            0.016294771805405617,
+            -0.024287264794111252,
+            0.02244403585791588,
+            -0.009611127898097038,
+            -0.008774704299867153,
+            0.00784921646118164,
+            0.05154848098754883,
+            0.009146448224782944,
+            0.003492843359708786,
+            0.011036145500838757,
+            0.005839476361870766,
+            -0.0013165927957743406,
+            -0.023683181032538414,
+            0.005568413063883781,
+            -0.007190920412540436,
+            -0.003601268632337451,
+            0.002536377403885126,
+            -0.004801691509783268,
+            0.031149037182331085,
+            -0.004379607271403074,
+            -0.005289605353027582,
+            0.004484160337597132,
+            -0.02196386642754078,
+            -0.0001332324172835797,
+            0.023203013464808464,
+            -2.3128106477088295e-05,
+            -0.023590246215462685,
+            -0.006571347359567881,
+            -0.015071115456521511,
+            -0.013770012184977531,
+            -0.018989915028214455,
+            0.005231520161032677,
+            -0.012755461037158966,
+            -0.0033940989524126053,
+            0.0016660707769915462,
+            -0.005940156988799572,
+            -0.002582845278084278,
+            0.009603383019566536,
+            0.05362405255436897,
+            0.014103032648563385,
+            -0.006335135083645582,
+            -0.005854965653270483,
+            0.009192915633320808,
+            -0.010524998418986797,
+            -0.019795360043644905,
+            -0.009820234030485153,
+            -0.029507169499993324,
+            -0.012747716158628464,
+            -0.0143121387809515,
+            -0.010191977955400944,
+            -0.010277168825268745,
+            0.00963436160236597,
+            -0.018122512847185135,
+            -0.007628493942320347,
+            -0.020337486639618874,
+            0.002941036131232977,
+            -0.011268485337495804,
+            -0.014714861288666725,
+            -0.008054450154304504,
+            -0.04807887226343155,
+            0.005796880926936865,
+            0.019593998789787292,
+            0.01263929158449173,
+            0.002515079453587532,
+            0.004383479710668325,
+            -0.01993476413190365,
+            -0.030792783945798874,
+            0.013839714229106903,
+            0.011283975094556808,
+            0.011733165942132473,
+            0.03283737599849701,
+            -0.011709931306540966,
+            -0.00813964195549488,
+            -0.004693266469985247,
+            -0.0017251238459721208,
+            0.012066186405718327,
+            0.00969631876796484,
+            -0.004917861428111792,
+            -0.012375972233712673,
+            0.016217326745390892,
+            -0.01981084980070591,
+            -0.023683181032538414,
+            -0.01672847382724285,
+            0.006157007534056902,
+            0.00813964195549488,
+            -0.008054450154304504,
+            0.002973950933665037,
+            0.0056962003000080585,
+            -0.007159941829741001,
+            -0.00022423222253564745,
+            0.004890755284577608,
+            -0.023048119619488716,
+            0.0008490087930113077,
+            -0.002257569460198283,
+            0.0061376458033919334,
+            0.021266845986247063,
+            -0.010168743319809437,
+            -0.0014075926737859845,
+            -0.0053902859799563885,
+            -0.0028984404634684324,
+            -0.006764963734894991,
+            0.02620794251561165,
+            0.009154192171990871,
+            -0.009882191196084023,
+            -0.01011453103274107,
+            -0.01129171997308731,
+            -0.0011278166202828288,
+            -0.017255110666155815,
+            -0.016186347231268883,
+            0.0015566773945465684,
+            -0.016712984070181847,
+            0.013971373438835144,
+            -0.00876695942133665,
+            0.00400399137288332,
+            -0.014110777527093887,
+            0.02019808255136013,
+            0.01929970271885395,
+            -0.009711808525025845,
+            0.023776117712259293,
+            -0.007709812838584185,
+            -0.038382552564144135,
+            -0.00800798274576664,
+            0.01843230053782463,
+            -0.006118284538388252,
+            0.010037084110081196,
+            -0.011314953677356243,
+            0.008209343999624252,
+            0.00015319912927225232,
+            0.0018122512847185135,
+            -0.0003112386621069163,
+            0.027555514127016068,
+            0.015535795129835606,
+            -0.005560668651014566,
+            -0.005494839046150446,
+            0.004255692940205336,
+            0.009541425853967667,
+            0.017580386251211166,
+            0.012151377275586128,
+            -0.021019017323851585,
+            -0.014257925562560558,
+            -0.016263794153928757,
+            0.011012911796569824,
+            0.015125328674912453,
+            -0.0040891822427511215,
+            0.01691434532403946,
+            0.014126266352832317,
+            -0.019098341464996338,
+            0.0019884423818439245,
+            -0.008193854242563248,
+            -0.00158184755127877,
+            -0.03029712475836277,
+            0.011043890379369259,
+            0.0074387495405972,
+            -0.0072218989953398705,
+            0.036616768687963486,
+            0.009858956560492516,
+            0.037577107548713684,
+            0.018680129200220108,
+            0.0008974128868430853,
+            0.009045767597854137,
+            -0.018370341509580612,
+            0.013065247796475887,
+            0.01500141341239214,
+            -0.012724482454359531,
+            -0.005734923295676708,
+            0.0013446672819554806,
+            0.0007551047019660473,
+            0.012631546705961227,
+            0.02004319056868553,
+            -0.01795213110744953,
+            0.009773765690624714,
+            -0.015032391995191574,
+            -0.005250881891697645,
+            0.006555858068168163,
+            -0.013932649977505207,
+            -0.008495896123349667,
+            -0.00963436160236597,
+            0.019408127292990685,
+            -0.012600568123161793,
+            0.016573579981923103,
+            -0.009595638141036034,
+            0.015520306304097176,
+            -0.029724018648266792,
+            -0.003669034456834197,
+            -0.006679772399365902,
+            -0.019593998789787292,
+            -0.012887120246887207,
+            0.009820234030485153,
+            0.010633422993123531,
+            -0.009301341138780117,
+            -0.011562783271074295,
+            0.020368466153740883,
+            -0.004185990896075964,
+            0.013529927469789982,
+            0.021189400926232338,
+            0.00737291993573308,
+            0.024132372811436653,
+            0.0005605200421996415,
+            0.014164989814162254,
+            0.005463860463351011,
+            0.010362359695136547,
+            -0.004677776712924242,
+            0.005161818582564592,
+            0.00932457484304905,
+            0.014203713275492191,
+            0.017766257748007774,
+            0.013196907006204128,
+            -0.00598275288939476,
+            0.007419387809932232,
+            0.02016710489988327,
+            0.0016254113288596272,
+            -0.030312614515423775,
+            -0.009719553403556347,
+            0.0001736498816171661,
+            0.00924712885171175,
+            0.009185170754790306,
+            3.817877222900279e-05,
+            -0.01093546487390995,
+            0.011144571006298065,
+            0.00969631876796484,
+            0.002286612056195736,
+            0.006280922330915928,
+            -0.01456771232187748,
+            -0.020957060158252716,
+            0.006106667220592499,
+            0.01798310875892639,
+            0.013917161151766777,
+            -0.014722606167197227,
+            0.022784801200032234,
+            0.006145390681922436,
+            -0.021406250074505806,
+            -0.01840132102370262,
+            -0.0009559819591231644,
+            0.0004542729293461889,
+            -0.002265314105898142,
+            0.03720536455512047,
+            -0.0011539548868313432,
+            -0.005955646280199289,
+            -0.0054754773154854774,
+            0.02775687538087368,
+            0.02049238048493862,
+            0.009177426807582378
+          ],
+          "embedding": [
+            0.0008215984936325879,
+            -0.019349418284688353,
+            -0.02322660633029795,
+            0.009747820845194852,
+            0.007353771220903641,
+            0.015252908783199839,
+            -0.012176175829980202,
+            0.051898051094966605,
+            -0.0068130353886146165,
+            0.008189212816890595,
+            0.008102287831931905,
+            0.01682801136011614,
+            0.02005557996049711,
+            -0.020779879580497183,
+            0.002332165268329781,
+            0.02337485065616377,
+            0.006074992895540845,
+            0.02576575146535689,
+            0.0007517286429767504,
+            -0.03763909942314134,
+            0.0011204729720314833,
+            0.00041212613589632806,
+            -0.06089176800314795,
+            0.012852035873395388,
+            -0.008901127092058543,
+            0.01702347055484216,
+            0.005803446395466281,
+            0.017889570275475296,
+            -0.05375047382413552,
+            0.021623778774252724,
+            0.014784317465883424,
+            -0.0018881880857885708,
+            0.0004735328554290247,
+            0.012393802850407656,
+            -0.019234442968055952,
+            -0.019566283611638122,
+            0.04407789373296535,
+            -0.0053565873209102995,
+            -0.026240866282480922,
+            0.048107902270426675,
+            -0.0054681314939387235,
+            -0.0009441397948041965,
+            -0.015205079626883548,
+            0.006885981841893229,
+            0.01403724256866746,
+            -0.026086548246792754,
+            0.0034778946051893067,
+            0.00044587533948775473,
+            0.032851328435891634,
+            -0.03882097333255334,
+            -0.011126344604254894,
+            -0.01648943336384904,
+            0.035810456555802166,
+            -0.019384589526165754,
+            -0.019430702768170285,
+            0.014297440272239852,
+            -0.007883453204663151,
+            0.050996900793086844,
+            0.06007149434153715,
+            0.03406832053483379,
+            0.025193439934671852,
+            0.00027785129663604897,
+            -0.008014391056168083,
+            -0.018806730908490114,
+            0.027193788903098558,
+            -0.006195006670328595,
+            -0.020466000344352826,
+            0.0068939884219589185,
+            0.017252276167086602,
+            -0.045073444648704136,
+            -0.049242269917908754,
+            0.023021140866008313,
+            -0.015676872472071106,
+            -0.024497517908898602,
+            0.005873633718973869,
+            -0.022828585938203683,
+            0.011748341004940436,
+            0.04624205727927451,
+            -0.015442165499674641,
+            0.018565551071491767,
+            0.015246505395507096,
+            0.004119269796224144,
+            0.004862735250819252,
+            -0.000145323321960046,
+            0.05155838952887077,
+            0.0027052423345323936,
+            -0.016218063670586664,
+            -0.07462470920944148,
+            0.043695974746295135,
+            -0.027608273720377708,
+            -0.03860912774695494,
+            0.002823110565150435,
+            0.026471469849508766,
+            -0.0026063577189074103,
+            -0.029230652014151735,
+            0.05684164470477843,
+            0.00022182868300572677,
+            0.015590845552956364,
+            -0.024939617549529242,
+            0.01035803286203917,
+            -0.010225298012384327,
+            0.019571847648759157,
+            0.01841613041441054,
+            0.0156981470538354,
+            0.021764297694793343,
+            -0.018831230007033416,
+            -0.014419529638789993,
+            -0.0029965475899652273,
+            0.028881185012095884,
+            0.09163526460163937,
+            0.022644302207811818,
+            -0.017933909936330133,
+            -0.04663928709016869,
+            0.004962951444105128,
+            -0.041333041174526026,
+            -0.027545628007617527,
+            -0.037142394579894954,
+            -0.013358143162788402,
+            0.02719686432396677,
+            -0.0034971800937504205,
+            0.04413635817371732,
+            0.04903857110328917,
+            0.03186067155630788,
+            0.007831051050269414,
+            -0.025257538623313574,
+            0.023517513773109414,
+            -0.0016527403209505964,
+            -0.0016101860707429347,
+            0.014448515185903624,
+            0.009905018312477684,
+            0.0024114582235163955,
+            0.00172043851384822,
+            -0.03818316221102173,
+            -0.02304537008110578,
+            0.01024969233570089,
+            -0.01901116356327339,
+            0.019791121460848986,
+            0.07985067340821365,
+            -0.051903191272236665,
+            -0.025924646976556942,
+            0.014766511532849252,
+            -0.002073086076894999,
+            0.030959269926800852,
+            0.006394417986617089,
+            -0.028177586693713442,
+            0.010725285336674526,
+            0.0009614052276924192,
+            0.07500253788978564,
+            -0.00036794182149960157,
+            0.004058458332826519,
+            -0.013218269166951234,
+            0.027613539268683245,
+            -0.012995170442480582,
+            0.023388707018488407,
+            0.026389423313105165,
+            -0.003068639956061266,
+            -0.02661967649632026,
+            0.026145274570635913,
+            -0.03300591322174331,
+            -0.004556072066733024,
+            0.020045722084112896,
+            0.0033437343914784865,
+            -0.010224220905483085,
+            -0.0025025498684079703,
+            0.03460103245395426,
+            0.04514362357278504,
+            0.03797031988058962,
+            -0.004586662475576956,
+            -0.06351723722596923,
+            0.06037983484369874,
+            0.019415753124363902,
+            0.028850618647994476,
+            -0.0034810237045849006,
+            0.02957938154808314,
+            0.02642652523231673,
+            0.013187653724133275,
+            -0.007856320804109569,
+            -0.007065882659482447,
+            -0.016672784625197085,
+            0.029020161154209707,
+            -0.013243058921970134,
+            -0.014900932834241732,
+            0.006001769901896033,
+            0.008648456186944271,
+            0.03484763413464497,
+            0.012279368984306564,
+            0.02379643508687133,
+            0.04933044959399941,
+            -0.021047723935647383,
+            -0.007264823701706426,
+            0.0007058774795716792,
+            -0.010512071832195085,
+            0.027489671668749157,
+            0.009731157570361245,
+            -0.03195919640758579,
+            -0.0033626031972691915,
+            -0.04282419484651207,
+            0.01873977910485268,
+            -0.014398285316695653,
+            0.012854631447494207,
+            -0.0017372959978092432,
+            -0.037331867825735776,
+            0.001088209292238142,
+            -0.012345684323522151,
+            0.037096063326414734,
+            0.005050219690024652,
+            0.0008906051515353004,
+            -0.003697709449630797,
+            0.006548479142737107,
+            -0.010727663712299397,
+            -0.007486226350610755,
+            -0.02704618481069485,
+            -0.019752552016312926,
+            -0.014632581296303486,
+            -0.0037501138119724717,
+            0.014588411795703245,
+            -0.007119137071719376,
+            0.016440258608998535,
+            -0.03686645352111013,
+            -0.023546402448713415,
+            0.005628703432152666,
+            0.024988145751166143,
+            0.011971199595808573,
+            -0.010868778329190568,
+            -0.03270046433624425,
+            -0.021950559285357267,
+            -0.011889914022657738,
+            -0.005123971523701009,
+            0.02619713302280345,
+            -0.021990810305904974,
+            0.01165257474485436,
+            -0.009117901804315863,
+            -0.007492082755249399,
+            0.010725902070291232,
+            0.01862811996181951,
+            -0.026694281868322685,
+            -0.01623776415010199,
+            -0.009403997538684658,
+            -0.02207329027097839,
+            0.025939540497196416,
+            0.014178803271471762,
+            -0.001141821318145744,
+            -0.006281187017652108,
+            -0.035993365198554964,
+            0.0026192776111449633,
+            -0.015254976771874396,
+            -0.0116872453009606,
+            -0.015780016491993135,
+            0.021421328283124968,
+            0.02889432250124136,
+            -0.01871009416296692,
+            -0.026540472640514536,
+            0.015703370682110186,
+            -0.0002849235245164438,
+            -0.02594930889995505,
+            0.04491828744809679,
+            0.044063522197410734,
+            -0.037441864491547405,
+            -0.007005339187980476,
+            -0.030381794393931814,
+            0.06901083845539985,
+            -0.017604082400603037,
+            -0.04060591381445593,
+            0.06044850935418443,
+            -0.01756490171206357,
+            0.025539717592705295,
+            0.03452018789549939,
+            0.0016217577522523927,
+            0.00018825054264059277,
+            -0.002563606092068712,
+            0.04770277039047993,
+            -0.0032010143255574255,
+            -0.03425328601475849,
+            -0.02105909767072799,
+            -0.00840881022563103,
+            -0.028699447677549002,
+            -0.015925160861589006,
+            -0.0055750742281166,
+            0.017513987082479483,
+            0.010399718005952185,
+            0.02267201252273823,
+            0.01258511878601025,
+            0.0015081978484714903,
+            0.01874981572091931,
+            0.011158889421057756,
+            -0.002607810686437211,
+            -0.003732023411827874,
+            -0.007787189972165309,
+            -0.05524012485122728,
+            -0.025110689689279583,
+            -0.03946383236703947,
+            0.004443572529325475,
+            -0.018745077598477586,
+            0.005926160888274727,
+            0.039430331582190524,
+            -0.0049500324516682,
+            0.004352795787899792,
+            0.015566139991780173,
+            0.013148156356408315,
+            -0.015279560337042242,
+            -0.03316846614871165,
+            0.0001592209771257332,
+            0.018362411161743347,
+            -0.012706582418607377,
+            -0.03424477956681259,
+            -0.001991022841226589,
+            0.0033467286047492175,
+            0.03357019284647071,
+            -0.024359110796680374,
+            0.00359117029951049,
+            0.005061818038334802,
+            0.011359582931760315,
+            -0.026793992746244076,
+            -0.018170136079817746,
+            -0.017421603453826397,
+            0.03996608926471779,
+            0.02587013058343799,
+            0.015763410145648683,
+            -0.04512536449983358,
+            -0.02293473890636517,
+            0.048575083783480195,
+            -0.009589860756342604,
+            -0.06486262416944208,
+            0.03447666009581266,
+            0.016105654870253654,
+            0.003526582643722421,
+            -0.01354754623151814,
+            0.05401575248587799,
+            -0.03563190295559527,
+            -0.017365166993914202,
+            -0.0046288709698227735,
+            -0.033657212157212645,
+            -0.007686252059880836,
+            -0.001385273642139782,
+            0.048229062789530236,
+            -0.029083855744728694,
+            -0.030465559177859865,
+            0.0024368661249980416,
+            -0.024144031154493057,
+            0.021693278579520123,
+            -0.016405088055634124,
+            -0.018439935250344958,
+            0.006948603359995389,
+            -0.033943037395199695,
+            -0.003007796423804453,
+            -0.016783195426536633,
+            -0.02270589491206155,
+            0.020874438753828814,
+            0.016724666903417947,
+            0.027033184180344852,
+            0.0019847647852695785,
+            0.017749524526371133,
+            0.0204661893602285,
+            -0.005604929968008821,
+            0.028462234633072944,
+            -0.020790459496727066,
+            -0.04183816355941877,
+            0.03946487892663695,
+            0.01192861758684541,
+            -0.0018024841275380242,
+            0.01077082591611702,
+            -0.01475239020295694,
+            -0.03952356245715099,
+            0.03683950018557526,
+            -0.056767672022551764,
+            -0.007507600090017693,
+            0.009530544352246083,
+            -0.00788856051420074,
+            -0.011204694027562521,
+            -0.010587701466318598,
+            -0.028604940377315504,
+            0.023749537635431462,
+            -0.013672419168235691,
+            0.01953449229707597,
+            -0.011449939491554354,
+            -0.012238180529977605,
+            0.05795536247491174,
+            -0.021518581774137322,
+            -0.040672955795925836,
+            0.026066659238628543,
+            -0.014061106927896723,
+            0.03328357230532424,
+            0.03957107406164427,
+            0.030670947051333917,
+            -0.004278577327978564,
+            0.025885946540364826,
+            0.0319106101668716,
+            -0.003462160036198642,
+            -0.04686868330851868,
+            0.00555303933159587,
+            0.02156143971867743,
+            -0.03739529489075782,
+            -0.01025626126223168,
+            -0.02141970191890338,
+            0.022134910980548407,
+            0.017807484142010496,
+            -0.007232643256393455,
+            0.011318041192005274,
+            -0.010489012862455913,
+            -0.013003217163997554,
+            -0.00847279880392288,
+            0.02357274819910594,
+            -0.0016485066139744577,
+            -0.005873113634992993,
+            0.006424249429163587,
+            0.004008703814301658,
+            0.0054639858216457724,
+            0.010198079500408441,
+            0.017694720520798957,
+            -0.017087021465208255,
+            -0.0025558869389290554,
+            -0.016503579212015177,
+            -0.0420082404922411,
+            -0.003563173424359332,
+            0.022924161700566468,
+            0.012345933179990547,
+            -0.00729908668365632,
+            -0.004234683369642856,
+            -0.0022962545569187934,
+            -0.020666205489143392,
+            -0.014796149043465308,
+            0.024562653223873056,
+            0.030217494971086438,
+            0.007810961433007407,
+            0.007781263640449456,
+            -0.0030348559388780655,
+            0.007310876971216526,
+            0.016099591640649783,
+            0.02311929597780174,
+            -0.04160162038353416,
+            0.027152100845897007,
+            -0.00025396315502635146,
+            0.023717123370023598,
+            -0.04186918759097667,
+            -0.017988531857836703,
+            -0.03106663560912813,
+            -0.023595079699181436,
+            0.02437134274097885,
+            -0.02317559303912261,
+            -0.012327263775257306,
+            0.000989566600374836,
+            0.00954921216572729,
+            -0.005155079395685017,
+            -0.017508404061379164,
+            -0.04582321461886958,
+            -0.0014367264231457263,
+            -0.030417229142190933,
+            -0.017829710557332023,
+            -0.001056472801554354,
+            0.005016912084754242,
+            -0.06276644565618253,
+            0.0109965681725148,
+            -0.07270669271977365,
+            -0.004542796125637244,
+            0.040139998232966566,
+            -0.017890441783630643,
+            0.055909102081208445,
+            -0.05029104308417623,
+            0.01673962668706085,
+            -0.04507293803055318,
+            0.015385506084493484,
+            0.025677372277539627,
+            0.030235171586316813,
+            -0.022189978375852563,
+            0.008461188938299745,
+            0.008033707483259569,
+            -0.045089505755429415,
+            -0.04144698743458133,
+            -0.013640256640978024,
+            0.01036782197518379,
+            0.0012696958402094232,
+            -0.016090728932978384,
+            0.023838437877040163,
+            0.011946482002767135,
+            0.03342604921859589,
+            0.04211801335157251,
+            0.002708015395637621,
+            -0.047156646610976634,
+            -0.013104464812562618,
+            -0.003933116891120681,
+            0.03155321708116329,
+            -0.018866109579446184,
+            0.015376812010630058,
+            -0.01955557376085726,
+            0.016255217383093355,
+            0.017868332102274922,
+            -0.020325179506828335,
+            0.013434364256590097,
+            0.004176904701252156,
+            -0.03877060944639854,
+            0.0006475542749369515,
+            0.00042468300294954125,
+            -0.02116992978038693,
+            0.0032654387271108293,
+            -0.00575003786159731,
+            -0.05149954071159492,
+            -0.031382926943128234,
+            -0.015730563640253065,
+            0.0382312502769352,
+            0.004297939848910064,
+            0.019724983545557365,
+            -0.017907869879607064,
+            -0.005165628493894463,
+            0.03698976707579373,
+            0.008501001373266799,
+            -0.01725527821298066,
+            0.006879620077191753,
+            0.02455792200929138,
+            -0.03224193972301503,
+            -0.039181328884413266,
+            -0.0008206986106407542,
+            -0.023306758893533952,
+            0.01672946960876128,
+            -0.02658587698869307,
+            0.005873620623736886,
+            0.0025331170979801232,
+            -0.03490809296591325,
+            0.02456764314036355,
+            0.003291113462353163,
+            -0.004303423672882112,
+            -0.020359544091059775,
+            -0.016892415165120203,
+            0.017174797487220006,
+            0.04391489511478065,
+            0.007327321684948843,
+            -0.003943713832544148,
+            0.00862684231313155,
+            -0.006361172785753631,
+            0.05826688726534012,
+            -0.02284956969636637,
+            -0.026680023999604643,
+            0.027527436347931256,
+            -0.022597669279069723,
+            -0.006277863878949772,
+            -0.017343446085471857,
+            -0.012065192152203348,
+            -0.0004183712490442143,
+            0.019339311388446943,
+            -0.019423562920897294,
+            -0.010212527447243049,
+            0.004783006556748757,
+            0.012824133870627983,
+            -0.004855945108275706,
+            0.006239949458562236,
+            0.006847581690471813,
+            0.03094005096378827,
+            0.026993131409761086,
+            -0.011331881651539968,
+            0.03297014291876675,
+            0.010265600643742144,
+            4.666113318251278e-05,
+            0.001970879281172325,
+            0.002523664335535017,
+            0.0034107811520368574,
+            -0.03143890864030425,
+            -0.01624584278940617,
+            0.013468729048885906,
+            0.005233309338223403,
+            -0.004550544967856448,
+            -0.023783653541634683,
+            -0.009402494239255264,
+            0.03334923712760375,
+            -0.003260424721959948,
+            0.02748395114281811,
+            -0.0058374897745283065,
+            -0.01985840930273332,
+            0.017595418271610163,
+            0.02881743246631328,
+            -0.05928552440605384,
+            0.017062540894682354,
+            -0.0011508547522925793,
+            0.008953761538519997,
+            0.01610561914042553,
+            -0.008894160513648953,
+            0.013301991014533684,
+            0.04080863883897929,
+            0.010738051773755976,
+            -0.023345002072886145,
+            0.004215726310838674,
+            0.01541749448570608,
+            0.01291813299324752,
+            -0.022747942296528025,
+            0.0008640791861357522,
+            0.011008522066277018,
+            0.005893793007012561,
+            0.0026106450193140156,
+            -0.006278394080583205,
+            -0.025659188287146268,
+            0.024180550333596625,
+            -0.0044491292740074925,
+            0.026967973346698617,
+            0.04390909273429254,
+            0.009596855239692042,
+            -0.03686020608700941,
+            -0.013109951664921729,
+            0.0002799680359670127,
+            -0.0038537577602593366,
+            -0.0037552094562213916,
+            0.011105295862811114,
+            -0.04185615640149602,
+            0.019950890919369135,
+            0.006453241355231516,
+            0.015314645427201533,
+            -0.020031193483734364,
+            -0.024530173488182337,
+            0.0031949828520724337,
+            -0.01492645564652531,
+            0.012875827757040941,
+            0.005057170413681063,
+            -0.004319834710426384,
+            0.005386862713494232,
+            0.00030848054013413157,
+            0.008384941028326983,
+            -0.010811526993301901,
+            0.03175068297908541,
+            0.004680987487229685,
+            0.009264547472281849,
+            -0.02856314241700808,
+            -0.021083721348429348,
+            0.020113396771629096,
+            0.03572366771051171,
+            -0.01708529918264109,
+            0.03346921724975398,
+            -0.009042046660366718,
+            0.0033219155387035227,
+            0.0151720168318272,
+            -0.015805318210244414,
+            -0.008057236707694303,
+            -0.0005483009439015931,
+            -0.002917527004423299,
+            -0.014538511124345527,
+            0.007302979537905532,
+            -0.0208529958115422,
+            -0.02965227242147972,
+            0.020064178921373875,
+            -0.00011892622932349096,
+            0.016455188521316146,
+            0.010335044657747514,
+            -0.007816155992852236,
+            -0.0052262330850905165,
+            0.006959344748883874,
+            0.04432147882994085,
+            0.026734484553994495,
+            0.05547661378166294,
+            -0.005787997952888261,
+            -0.004559444854138438,
+            -0.0082295581123363,
+            -0.006588172755707144,
+            0.04524009609473585,
+            0.015915003261638767,
+            -0.004910285095602864,
+            0.017495607570191165,
+            0.01557886571266685,
+            0.03858259260136037,
+            0.0027648525902558316,
+            -0.016912796336835408,
+            0.04014755669075221,
+            -0.004077405782937414,
+            0.008616827127381418,
+            0.011102692199778469,
+            0.0024185742218798562,
+            0.00807141794538637,
+            0.02996274720815659,
+            -0.012429186019907761,
+            0.01680942269506771,
+            -0.04571916592313758,
+            -0.02775530246832996,
+            0.014370280587419414,
+            -0.004931905540824582,
+            0.03419959665162407,
+            0.01668521021731861,
+            -0.023105305446763272,
+            -0.024761385385514877,
+            0.021529428247380984,
+            -0.002603042335561961,
+            -0.024135326707477325,
+            -0.015906833215844473,
+            0.003286605488908195,
+            -0.0023437997044735617,
+            0.0060251848371355166,
+            -0.013481652076194643,
+            0.0002908426939460218,
+            0.01953362396253743,
+            0.0038544889405965057,
+            0.020760460182873287,
+            -0.016224396121239458,
+            -0.00259233446682179,
+            -0.009282861237653619,
+            -0.02952130765234087,
+            0.017988370145301916,
+            0.006138236653154911,
+            -0.007607182955958238,
+            -0.009516767050028553,
+            0.0048347778733230365,
+            -0.005819209642901871,
+            -0.0025851701689263706,
+            -0.01073014664868326,
+            0.012433465684360061,
+            0.025784014942833097,
+            -0.016875624785552303,
+            0.007944349714123104,
+            8.651589872318479e-05,
+            -0.022306044741692753,
+            -0.022127016745066307,
+            0.020143472307319017,
+            0.002792646249762062,
+            0.008768813346304779,
+            -0.004074214276300631,
+            -0.00015024341785520542,
+            0.002990550758627699,
+            0.01951746553838439,
+            0.03896451381932467,
+            0.004232673242619471,
+            0.027876586543280867,
+            0.0145860303663199,
+            0.03994234200854897,
+            0.009657017763624678,
+            0.00696607158576712,
+            -0.026055387694656488,
+            0.02819890357011778,
+            0.00018297502510753014,
+            0.043190622821775236,
+            0.022488402221902314,
+            -0.0407730101161195,
+            -0.0021946518523759066,
+            0.01669520746276111,
+            0.013212623257794862,
+            0.017009844136059122,
+            -0.004537512994041156,
+            -0.014041213424740049,
+            0.028135404528975894,
+            -0.022859586430516003,
+            0.0008637006862481223,
+            -0.004486332934188017,
+            -0.025031795510001887,
+            0.007351370254175368,
+            -0.004567816916440959,
+            0.030025413528233364,
+            0.0014976740344892735,
+            -0.022285096307825804,
+            0.022837170656498543,
+            -0.009551269792616045,
+            0.011045530488684207,
+            0.008809047185288755,
+            0.020065001354321994,
+            0.031993362733068575,
+            0.009143210434415064,
+            0.02133640814803879,
+            0.012241777474889248,
+            0.014771538127546474,
+            -0.008804192497858033,
+            -0.02341427260922065,
+            0.023167424359504862,
+            -0.0011598649565273595,
+            -0.029627604902846688,
+            0.0021991474725787303,
+            0.003261878079344105,
+            -0.0005935713331682358,
+            -0.009144689470851402,
+            -0.005580557862988033,
+            -0.007240836451875213,
+            0.019631377011298905,
+            -0.007085213886352692,
+            0.007017573363709754,
+            -0.004074171791476129,
+            -0.00822871835665378,
+            -0.0421101573155962,
+            0.004210914075286222,
+            0.003408427700622687,
+            -0.01752179330665445,
+            0.006384440404174512,
+            0.011121481025573962,
+            0.043020575455262985,
+            -0.022809979913698254,
+            -0.0005737138409049003,
+            0.019206623212867476,
+            0.0009562551889820241,
+            -0.023427467810802533,
+            -0.038931688376706186,
+            0.007172345672967499,
+            -0.03467050750542982,
+            0.023185301620612612,
+            -0.0007075596339842658,
+            0.02387977204565927,
+            -0.03043468414449851,
+            0.013909304572370437,
+            0.004784668158421625,
+            0.012524879174804915,
+            0.021654512058639427,
+            0.03142146922870715,
+            -0.02053473475540882,
+            -0.050409862638621725,
+            -0.005406667316025549,
+            0.00035138762584085976,
+            0.022018487016014856,
+            -0.026014580970974616,
+            0.00025633004699302783,
+            -0.003349094027291921,
+            -0.02401366769212778,
+            -0.01735996745614562,
+            0.01962340791446025,
+            -0.050011480783403936,
+            0.015042913576492952,
+            0.0262737424555697,
+            -0.02507473901367252,
+            -0.05408889321034101,
+            -0.018988587082325706,
+            -0.028217893770659686,
+            -0.009285770094319318,
+            -0.00661159230987788,
+            0.028597405154740078,
+            0.003525576256054959,
+            -0.011462219358952466,
+            0.025401668390029348,
+            0.006155837251830549,
+            0.0026050693705924593,
+            0.015985722392086222,
+            0.007289910674212967,
+            -0.011452511453623235,
+            -0.014599577191293152,
+            -0.04558584556100621,
+            0.010197536827229795,
+            -0.0045431962637690155,
+            -0.019372598050508005,
+            -0.030147301333718016,
+            -0.0137828704002156,
+            -0.027060515748528666,
+            -0.0017291966950107505,
+            0.053783684195381654,
+            0.03292494290301388,
+            0.005560172296221419,
+            -0.0026085670565688026,
+            -0.010466890489356092,
+            0.036247483457995895,
+            -0.006789746401727701,
+            0.03662690472838588,
+            0.001784522073210422,
+            -0.0242673253380092,
+            0.01169382884548301,
+            0.02892627181542449,
+            0.009858451662166387,
+            -0.0005592283645355213,
+            -0.006160132931009508,
+            -0.008782830470079547,
+            0.0031207630972208394,
+            0.025613233596604418,
+            0.003268170392795548,
+            -0.010062991425127038,
+            0.018106967407336815,
+            0.004480291699976396,
+            0.02010374270557836,
+            0.03555357587371324,
+            0.023489837757983054,
+            -0.021782173652026497,
+            -0.026581762796336082,
+            -0.0013561486781077014,
+            0.0072266106510934505,
+            0.03551967295942857,
+            -0.01651315514034218,
+            0.0001274396319698452,
+            -0.008272735507120127,
+            -0.00894931703949637,
+            0.011149952898504355,
+            -0.0029671593121986116,
+            -0.0030752010285951576,
+            0.01662711184497578,
+            -0.0011672263615879709,
+            -0.033448222499147764,
+            0.0078288560210653,
+            -0.023496331346917084,
+            0.0051040606404415545,
+            -0.004113104164416575,
+            -0.007076525381595441,
+            -0.002157548492138289,
+            0.006302962169276006,
+            -0.008065841090139872,
+            -0.00035242603988955366,
+            -0.01586406637763306,
+            0.008271894439780969,
+            0.007438312999409211,
+            -0.0069617481460406095,
+            0.011402932583388205,
+            0.002666895909532698,
+            0.0005219881149353855,
+            -0.012159425670629751,
+            -0.012042145306711322,
+            -0.00957276588761916,
+            -0.03401863879157551,
+            -0.02604353712472079,
+            -0.024949533021906844,
+            -0.0014209149651479556,
+            -0.006566685925666972,
+            -0.03187796739029736,
+            -0.01209152137926458,
+            -0.011110763124407752,
+            0.007756961097537267,
+            -0.013260548545917543,
+            0.005433285079114047,
+            -6.399098520041351e-05,
+            -0.0044960488826674555,
+            0.016615636438440516,
+            -0.003643291599919672,
+            -0.019488010211707343,
+            -0.03927818878149417,
+            0.008209336177358258,
+            -0.02805593125673831,
+            -0.0038816771550190788,
+            0.007607119219721711,
+            -0.00528681677614909,
+            -0.029502897838743516,
+            -0.009149794737430706,
+            0.008767001199311985,
+            0.003507217008280066,
+            -0.01773349756494436,
+            0.03579670773605949,
+            0.026971617790626075,
+            0.011875179879209696,
+            -0.019891342676443626,
+            0.0508929813275565,
+            -0.0047010735337056275,
+            -0.012544935684845649,
+            0.018297503869683027,
+            0.02355022268131519,
+            0.03273167043015095,
+            -0.024830243320788693,
+            0.0009370814985214882,
+            -0.021937793653233454,
+            0.00535662615596573,
+            -0.035009915581788176,
+            0.0028595659314725012,
+            -0.014395027018699446,
+            0.015596459769560532,
+            0.03397372881171701,
+            -0.013254226072981457,
+            0.011195747181764677,
+            -0.016299384375589934,
+            0.00730244731485402,
+            0.00569944090461748,
+            -0.012916284280345044,
+            -0.005812311882399539,
+            -0.022902815152116493,
+            0.020140276669979015,
+            0.005809799950176385,
+            -0.0008130655714656685,
+            0.0104085826652653,
+            -0.044723432238900435,
+            0.03286429597337671,
+            0.008730225944501239,
+            0.017599189944921396,
+            0.0029762446259779998,
+            -0.0003068671951232415,
+            0.011611551606224214,
+            0.011228401175698422,
+            -0.016029727902740706,
+            0.004345913009102136,
+            -0.004613061137378433,
+            -0.03850533909474302,
+            0.009363351681220008,
+            -0.03631184057978666,
+            0.014427784677340159,
+            0.016026872002233575,
+            -0.004170582842643461,
+            -0.025166189571409366,
+            -0.005428471012333673,
+            0.010827910811085384,
+            -0.009783069664327864,
+            -0.009987805234276443,
+            -0.04163358183060102,
+            -0.025236440364963034,
+            0.034097508113604576,
+            -0.002119474613596975,
+            -0.025826403511707147,
+            -0.010830738112107807,
+            -0.011351132636491575,
+            0.020207687645133,
+            -0.006542026283946858,
+            -0.0006263975804936843,
+            -0.04150011545311013,
+            -0.0073396962917864515,
+            0.011192225907726928,
+            0.027746213794734322,
+            -0.035970017302005566,
+            0.02708126943501373,
+            0.002105633542220187,
+            0.014262674845755191,
+            -0.01626664983003942,
+            -0.005631805361903626,
+            -0.013990539604372344,
+            -0.035877093361375685,
+            0.016487723013906077,
+            -0.008607758248055054,
+            0.004714436224135091,
+            0.04458009880881818,
+            -0.0036872068506712727,
+            -0.009466125348499464,
+            0.007600705817195143,
+            -0.0034590883553048063,
+            -0.010351576085287273,
+            -0.0028564505793999295,
+            0.001959661298507,
+            -0.007080697863126256,
+            -0.04183476693971616,
+            -0.00499888976530217,
+            0.04684391782633148,
+            0.006361744381879857,
+            0.013276793765092465,
+            0.013516220521120684,
+            -0.018335746557932003,
+            -0.045577810515431215,
+            -0.0071616111758423025,
+            -0.0032763880176386808,
+            -0.022708028392758567,
+            -0.0240284945459861,
+            -0.008735040856657758,
+            0.005844145402902923,
+            0.006359368165686724,
+            -0.01120763599008222,
+            -0.022676131361171057,
+            -0.027085968700452136,
+            -0.010526103176253857,
+            -0.02123944716136792,
+            -0.010792145796733375,
+            -0.027142270190771887,
+            -0.028742234538796804,
+            0.02880426953386151,
+            0.007575182232859564,
+            -0.004084088301176117,
+            -0.010005765476121966,
+            -0.01255640546813973,
+            -0.0004774761617900028,
+            -0.0025307994917461232,
+            -0.005822065709999144,
+            0.0038614286313282164,
+            -0.011390482620811868,
+            0.023152086963976554,
+            0.0021220701260546895,
+            0.018329162172128138,
+            -0.05772324571231113,
+            -0.002709083851090071,
+            0.021497896395795905,
+            0.0033025410488410913,
+            -0.005412445141910314,
+            0.029352650459792132,
+            0.011223947499453582,
+            -0.006539460282439711,
+            -0.031355428613718234,
+            0.020942976176820262,
+            0.024122993515785947,
+            0.00817615634503138,
+            -0.012994160510030828,
+            0.01221095914384204,
+            0.01629778384790159,
+            -0.004494985018246821,
+            -0.011161098655072933,
+            0.005586857967039181,
+            0.020941039642372112,
+            0.013580622629911925,
+            0.0013398071867077221,
+            0.030905022925366163,
+            0.0006862084391508121,
+            -0.03521231612862609,
+            -0.01908191995763771,
+            -0.01275995164030872,
+            -0.003584331305986364,
+            -0.015264425022763126,
+            0.005294329654809353,
+            -0.002355465010876059,
+            0.006335052149475905,
+            0.022655379849785256,
+            0.03593231939948758,
+            0.0146937743203089,
+            0.00937330328448886,
+            0.003167462724350819,
+            -0.0013235176101076284,
+            0.012102872918129403,
+            0.008638011773075119,
+            0.0350701240268404,
+            -0.014653221697693864,
+            -0.011003994230902835,
+            -0.0037767288519369537,
+            0.010100131061186566,
+            -0.004113527518662366,
+            0.020414393256136372,
+            0.03274709615649469,
+            -0.0037222949335916876,
+            -0.005604552180246868,
+            -0.006160118231975786,
+            0.0041467454414452565,
+            0.01807879127455274,
+            -0.01252134965321004,
+            0.004364858987501461,
+            0.005549300665159658,
+            -0.0071897162833861,
+            0.018567059255813558,
+            0.0002622074720860951,
+            -0.012983550838417323,
+            0.03266376564402191,
+            0.11449509387240629,
+            0.0017251916835047628,
+            0.007125215166916139,
+            -0.040229997815548584,
+            -0.020537987527148888,
+            -0.00245728735683473,
+            0.005779972828836593,
+            0.007004732428236457,
+            0.02225914982152608,
+            0.006505827715591914,
+            0.0006708045351142873,
+            0.031708522931495536,
+            -0.017334083511709092,
+            -0.002522861743921599,
+            -0.01745656559688129,
+            0.013890451996979048,
+            -0.014312605700272691,
+            -0.013784963572164409,
+            -0.01572417853955075,
+            0.004507425871748977,
+            -0.000601158156253668,
+            0.0016158265715634729,
+            0.01791303234734911,
+            0.0011926917387973374,
+            -0.026393351252156246,
+            0.0006694008039871878,
+            0.001026031515214626,
+            -0.004514419527575117,
+            -0.011320929719992686,
+            -0.02223001723942815,
+            0.016235841442653918,
+            0.014188311619681008,
+            -0.004895218832118094,
+            0.009244697376301004,
+            0.010558796153959677,
+            0.02707485418319703,
+            -0.028380342401606758,
+            0.0040204430418629405,
+            0.007023041225985294,
+            0.01606556581447933,
+            0.017617862031190843,
+            0.0034870847111552862,
+            -0.011253315705053224,
+            0.03238289399766116,
+            -0.010563282687352038,
+            0.0065093536270233385,
+            0.014966802208644462,
+            -0.03541611540945151,
+            -5.821682910348165e-05,
+            0.0061985444938402745,
+            -0.011716201661363997,
+            0.009580035961315467,
+            -0.03120692534097433,
+            -0.0039291243987978845,
+            0.009317468331974374,
+            0.017611312558635673,
+            0.008804996982574765,
+            -0.0034129169537151925,
+            0.010822226698822738,
+            0.012510661458054741,
+            -0.0027147975086409966,
+            -0.008855213644027594,
+            0.01238502686635956,
+            -0.0007687053446954701,
+            0.00450108863711973,
+            -0.00799170686059026,
+            0.0029755595812552293,
+            0.002290650917244691,
+            -0.01554856300008412,
+            -0.011779430686394945,
+            0.00438393126679289,
+            0.0036931891075561957,
+            0.001535662168147833,
+            0.012014408636737167,
+            -0.006087255045485002,
+            -0.0077816220325470226,
+            0.0040573724156794284,
+            0.003136943055301706,
+            -5.505014316363865e-05,
+            -0.030625245024561508,
+            0.01585923084309149,
+            0.0067595435845396405,
+            0.010013308598626031,
+            -0.005315626260228517,
+            -0.00903142414091241,
+            -0.007240920015018893,
+            -0.01228521172101564,
+            -0.010657894223173453,
+            0.007658605592936815,
+            -0.00724924993432376,
+            0.011124174211132038,
+            -0.03128027787357322,
+            -0.013988715435985316,
+            0.01231792993245938,
+            -0.01528161242159377,
+            0.0007675821323979293,
+            -0.008389914312201402,
+            0.0009417242487395995,
+            -0.005119304379761705,
+            0.007887436211143328,
+            -0.03731503169547854,
+            0.007878001990271541,
+            0.02456170736765724,
+            -0.00330924625934981,
+            -0.020259771366167557,
+            -0.009613930457931786,
+            0.0241314385136084,
+            -0.008267831641886344,
+            -0.01645888287387047,
+            0.008421304862524506,
+            0.010140813149093993,
+            -0.0006309118552875393,
+            0.0008237784055549769,
+            -0.003726984191590917,
+            0.01669125388798323,
+            0.02054777616420997,
+            0.041930770964107515,
+            -0.014789610432004559,
+            0.01092349703703152,
+            0.02866938974792608,
+            -0.03378576980722847,
+            0.014728113846952417,
+            0.02518409645155824,
+            0.014590531327650454,
+            -0.0004446956315190408,
+            0.001609777186372437,
+            0.0068939321280085475,
+            -0.014273492115311677,
+            0.009247676157551494,
+            0.013380104440744367,
+            -0.018407746260686045,
+            0.020444430509441124,
+            0.03140071554008003,
+            0.0025839998101985736,
+            0.01355205268903058,
+            -0.006268939830271642,
+            -0.013860692186849113,
+            0.014717364068116991,
+            0.031429975488060896,
+            0.0006919565473572909,
+            -0.018561500446336806,
+            0.019128244642842268,
+            0.006988095806264863,
+            -0.003678529269242034,
+            -0.0021993078588142664,
+            -0.021334919445200388,
+            -0.021003586791369228,
+            0.017304350704792804,
+            0.013508779408333964,
+            0.027794485858413365,
+            -0.010594702782952029,
+            -0.007818089330662161,
+            0.014293688127901867,
+            -0.022368568887004094,
+            0.015426482139881204,
+            -0.02160820059598751,
+            -0.00571499813703566,
+            -0.0077402210272634235,
+            0.03274360648629609,
+            0.001441304806200403,
+            0.02734331677589297,
+            0.016457027764064002,
+            -0.033609420134045206,
+            -0.029790535553657092,
+            -0.00712106584238484,
+            -0.02307933297713104,
+            0.010356561171963577,
+            -0.03901183838032092,
+            -0.00071735150189382,
+            0.008251646656432826,
+            -0.008079917288471352,
+            -0.0007087526632966007,
+            -0.006419191399519291,
+            0.013989344185652506,
+            0.013661633160336152,
+            -0.0228103143826663,
+            0.023644639659577426,
+            0.008343481940133684,
+            0.025998483906975782,
+            -0.02076055376006461,
+            0.019148568238437927,
+            -0.017705363646416224,
+            -0.02777313933338929,
+            0.014358130220942483,
+            0.023585731219788467,
+            0.01635364038245971,
+            -0.007122853965024528,
+            0.007110003583832299,
+            -0.03556761320536796,
+            0.027585457849964405,
+            -0.013064062623866305,
+            -0.012504161289952458,
+            -0.0032379276562094555,
+            -0.011931827532360686,
+            -0.002574161005800279,
+            0.0008533907465694845,
+            0.0034345232246102806,
+            -0.03131462673429805,
+            -0.0002450959360520896,
+            -0.03727416416766574,
+            -0.028944814604815072,
+            0.01360126708805471,
+            -0.00036361284498363273,
+            -0.0049097813612814616,
+            0.02933242705521636,
+            -0.026339136251046315,
+            -0.02693295650849956,
+            -0.009590247544093352,
+            0.02853096456887074,
+            -0.0220673798697511,
+            -0.004519050393279848,
+            0.0015034014871316069,
+            -0.01828226786700525,
+            0.004436383319946364,
+            -0.0012748481042235446,
+            -0.010416065158876917,
+            -0.01105146139270597,
+            -0.03229479099927737,
+            -0.005822133182096844,
+            0.02106215163620738,
+            -0.006957125617688484,
+            0.002746241228040847,
+            0.013312248331456934,
+            -0.010968629118224582,
+            -0.0029946900704643812,
+            0.0017875284441794235,
+            0.023304175718480834,
+            -0.021033941064173063,
+            0.007714334541806199,
+            0.005955442031111047,
+            0.0016405586860131207,
+            0.018406562522853148,
+            -0.0054922197063461825,
+            0.0037690571959124617,
+            0.0162126534419016,
+            -0.009969304596173302,
+            0.017959307638726878,
+            0.007691549802475525,
+            0.006681837561745192,
+            0.002775700618222622,
+            0.007712770283579111,
+            0.009752652973700739,
+            -0.013506020126078112,
+            0.016911141425375933,
+            0.022468715726349706,
+            0.00023233555209372766,
+            0.021066580799163234,
+            -0.02302516292580905,
+            -0.028618544628453015,
+            0.018297878810036087,
+            -0.0225078941299981,
+            -0.012735208680311652,
+            0.016459925496937065,
+            -0.0043032724578332855,
+            -0.00797609922711709,
+            -0.014986057073351055,
+            0.02260618561608301,
+            0.02132930921464508,
+            -0.020249788476562527,
+            -0.012385936327068884,
+            0.0072304431443959505,
+            -0.004373475447051833,
+            0.007086656081865733,
+            0.03195810137269375,
+            -0.015982210897392764,
+            0.006818890093901413,
+            -0.0038306138017521626,
+            0.0030753642449792215,
+            0.017830546304667783,
+            0.009470018737705222,
+            -0.0019815241970236146,
+            0.030100238416495066,
+            -0.001330614013444443,
+            0.016392226282459177,
+            0.0004888028477871524,
+            -0.010948833692356176,
+            0.006343890495626752,
+            -0.026652806306173643,
+            -0.008627570055007363,
+            -0.001203530223345183,
+            -0.009581494451831351,
+            0.07598616930784373,
+            0.003930153552069478,
+            -0.01295957554290434,
+            -0.001673007115291578,
+            -0.001368287102181597,
+            -0.006897179139643672,
+            -0.00960679451603541,
+            0.0287166325268306,
+            -0.016613613331623346,
+            -0.01894527592131826,
+            0.0008370014132002703,
+            -0.006785618888421608,
+            0.00919924251249592,
+            0.01816363936999796,
+            -0.010225961928154075,
+            -0.01604759761423727,
+            0.001577138242530474,
+            -0.003520564936698765,
+            -0.004395434431595494,
+            0.0052399043934902014,
+            -0.010127122101423486,
+            0.006456662777583096,
+            0.026560491731327886,
+            -0.013684536319659296,
+            -0.01425196172663986,
+            0.004084982328707292,
+            -0.0010682084059972982,
+            0.009758641913073978,
+            -0.00864879062601467,
+            -0.00481222050200678,
+            -0.014375330954464725,
+            -0.011378890549403524,
+            0.02505641780042516,
+            -0.02861164504722546,
+            0.0034060847953066354,
+            0.014937227531980433,
+            -0.00016904224334342915,
+            0.01541381851648916,
+            0.0033028855218525644,
+            -0.03615935765718852,
+            -0.013677774380513626,
+            -0.02073839691250734,
+            0.019821074228011413,
+            0.01688422012576042,
+            -0.0033731840788147442,
+            -0.013840993586441808,
+            -0.026149880178874887,
+            -0.007430563876740068,
+            -0.02466234818567362,
+            -0.020708282040996014,
+            -0.019276991448222817,
+            0.006137002799644838,
+            0.005162892426268572,
+            0.022881759407670583,
+            0.0025854880108157857,
+            -0.015058889448404231,
+            0.0029948608544278386,
+            -0.006722836436935071,
+            0.011845298212977357,
+            -0.02747944486818378,
+            0.015311272959213262,
+            -0.01610493464983465,
+            0.025239768880017402,
+            0.007425208902253236,
+            0.0010569008485112994,
+            -0.008822681490988575,
+            0.00010704167529046774,
+            0.014883787023445807,
+            -0.004186968199973372,
+            -0.0007781259002896265,
+            0.02787555889813697,
+            0.02087373813833492,
+            -0.0017783425135865238,
+            0.008743718448136432,
+            0.028026471206552342,
+            0.019081083283793984,
+            0.0325261942213039,
+            -0.013815526479760766,
+            0.005113676604826083,
+            -0.028873600114081546,
+            0.010510475843707614,
+            -0.010871039729515013,
+            -0.010201824683340651,
+            -0.0067946599549165735,
+            0.027756887504389347,
+            -0.007962601205446472,
+            -0.0006687976124826922,
+            0.03469894503563926,
+            -0.014185294847328063,
+            -0.00010543064621374546,
+            0.0007386418834840544,
+            -0.009759481309152999,
+            0.02742722730386853,
+            -0.0073590367291510046,
+            0.018456032721630843,
+            -0.019560882612141943,
+            -0.013705917411254969,
+            -0.011894560208199602,
+            -0.001142768182039817,
+            -0.02576754877141608,
+            -0.008755240066826174,
+            -0.02249287216087102,
+            0.00859172262856532,
+            -0.0046012388553094375,
+            0.018845920253446127,
+            0.0011617844393195566,
+            -0.005275799428018491,
+            0.029069367036565145,
+            0.0088080893110627,
+            0.018100411389202663,
+            -0.0004971948011851863,
+            0.01216566778850985,
+            0.02369448391321832,
+            0.002637001729688794,
+            0.005012948614448268,
+            -0.024213704079796478,
+            0.003420184574622472,
+            -0.0004359900462023086,
+            8.265488264269535e-05,
+            0.016267948861008616,
+            -0.019572032453100303,
+            -0.02216264534756073,
+            0.021871880626992544,
+            -0.012742926070754964,
+            0.011676769226089007,
+            0.0035362870996336194,
+            -0.010732229460038356,
+            0.0006838777488770268,
+            -0.03083438636386314,
+            0.008820746379709355,
+            -0.03216685858285627,
+            -0.032230808828686675,
+            -0.013547681710236494,
+            -0.009994106571421729,
+            0.0008758088338659712,
+            0.028447187234357855,
+            -0.00795317029726568,
+            -0.015797892954136448,
+            0.005363565557248085,
+            -0.011199252697624111,
+            -0.00817162221789202,
+            -0.0007867892785855576,
+            0.005856939999700809,
+            0.009461959355310273,
+            0.005997804620690314,
+            0.011648335019189696,
+            0.007491946658046298,
+            -0.008028800679169293,
+            -0.008837725220736191,
+            0.015889923814605387,
+            -0.0014409716087425222,
+            -0.01176424391171685,
+            0.030430130082762363,
+            -0.013300620185884817,
+            -0.0005592926632214984,
+            -0.0021975669766032472,
+            -0.0015687208316663597,
+            -0.0021652076181411244,
+            -0.014225289143358926,
+            0.01870568404189425,
+            8.334501002782455e-05,
+            -0.014288806218067678,
+            0.0032841743905975634,
+            0.003841780428153241,
+            -0.0036498339483411074,
+            0.008388922960269383,
+            -0.008147275785908849,
+            0.017348147498892,
+            -0.015113395971380741,
+            0.008740452410210708,
+            0.0038335161236289234,
+            0.0043482078469644385,
+            -0.0065755348640655865,
+            -0.01592809184384243,
+            0.009642593996701513,
+            0.012051872087663658,
+            0.02173264012652684,
+            -0.02525660485859878,
+            0.00876161653901566,
+            -0.005374533920781607,
+            -0.014534179652013885,
+            -0.016127866637369738,
+            0.00982303248417706,
+            0.022322910435388785,
+            0.0063086346898091296,
+            0.012139263756190026,
+            0.00013383660542132047,
+            0.005848259030010402,
+            -0.014767198851314148,
+            0.010570158368602513,
+            -0.011355774405066537,
+            -0.018439261453084426,
+            -0.005243587266408389,
+            0.028894120979627323,
+            -0.005492024421080143,
+            0.017938747701972086,
+            0.0021350711250093858,
+            0.029007356490801176,
+            -0.008686781541806358,
+            0.0014883465513564527,
+            0.0039866022469891155,
+            -0.01295271525889559,
+            -0.004272008629260297,
+            0.01976273843646234,
+            0.04882868638601926,
+            0.010204475902027572,
+            -0.0009155109545661691,
+            0.010069071220988797,
+            0.005626418787776309,
+            0.002148101019776153,
+            -0.00805781810841286,
+            -0.030431071067211577,
+            0.008883707378543252,
+            0.005628218215212441,
+            -0.01997825125961566,
+            0.019838234890736068,
+            0.026483365072033395,
+            0.023040623182975964,
+            -0.011585643598171132,
+            0.016539930468533123,
+            -0.010092529089936188,
+            0.0177008735686489,
+            0.028665424114962996,
+            0.013810061902648515,
+            -0.010487150104822674,
+            -0.0020847870330548534,
+            0.006329342599564054,
+            0.004186981876004016,
+            0.00026239408223817384,
+            0.015190614519766204,
+            -0.019155180136775043,
+            0.01005728016778505,
+            0.014725826697964637,
+            0.03322436221319799,
+            8.049930133627564e-05,
+            -0.01645191835557271,
+            0.0016249281996128196,
+            -0.009976155662939693,
+            0.00856399492362911,
+            0.000632427127231252,
+            0.0032242260692672124,
+            -0.009688119824276678,
+            -0.012682779503915694,
+            -0.002851918183031152,
+            0.009403205508474241,
+            0.0021188716361101698,
+            0.002759004011231831,
+            0.007641925150177999,
+            0.0022830062031146204,
+            0.01713024474384216,
+            -0.0011181980841546123,
+            -0.00029141889484139217,
+            0.004601090256674874,
+            0.010509199590505558,
+            -0.01649852835819883,
+            -0.017548369964263514,
+            0.004337237475053179,
+            0.012217494210546133,
+            -0.013159179948224596,
+            -0.013241981632059755,
+            0.015704543608959117,
+            -0.01913575130892795,
+            0.024934102868065366,
+            -0.00942381892138331,
+            -0.004543611336526587,
+            0.000343200128934031,
+            -0.00727538880611703,
+            -0.0038751118497522673,
+            -0.0030064279504724797,
+            0.003984715814505561,
+            0.0019261818110367147,
+            0.007472350699291607,
+            0.008472369788801792,
+            0.014568547873264247,
+            0.006043895352236181,
+            0.0032580830408778395,
+            0.009109104498825864,
+            -0.014523846294121576,
+            -0.014800649887278973,
+            -0.002704791598011488,
+            -0.008895394038700809,
+            0.00902016867266151,
+            -0.006607182154874112,
+            0.00845408382182461,
+            -0.015501867835442603,
+            0.015573642225745032,
+            -0.012200552219016517,
+            -0.0008900215712478553,
+            -0.009666233545144905,
+            -2.7470277092642562e-05,
+            0.008191697904514857,
+            0.014605287653054787,
+            0.0070552942843382215,
+            -0.031921405225964206,
+            0.0017403427534633655,
+            -0.008846568352320488,
+            0.03446319360336241,
+            0.002612606662691094,
+            -0.01060863123202008,
+            0.020352181356568008,
+            -0.01592124132155158,
+            -0.00027704668715660693,
+            -0.01602075523652062,
+            -0.004797557413047212,
+            0.00707939446796679,
+            0.016399017978990946,
+            0.02759049512664933,
+            -0.005697020356865688,
+            -0.007995305872076221,
+            0.006384047872862793,
+            0.029590696670334232,
+            0.012246955508890425,
+            0.024748351124415785,
+            -0.007633712004672087,
+            0.014348228716100414,
+            0.007773260552660029,
+            0.0076045488653348975,
+            0.004321689250440021,
+            -0.002591801797145718,
+            -0.011698574606499644,
+            -0.014974971814360001,
+            -0.0037563183330623384,
+            -0.016987915346053498,
+            0.00012247833530362867,
+            0.021344400276501117,
+            -0.00475351423959752,
+            -0.0033847161875730026,
+            -0.01874186437694691,
+            0.0033461670218611346,
+            -0.010029445302549983,
+            -0.00367421246775698,
+            -0.013133469163793961,
+            -0.0254122035745965,
+            -0.018556369080030412,
+            -0.002155244024999317,
+            -0.00929309186646207,
+            -0.015194269781263309,
+            0.0070354690219415535,
+            -0.016305117309271634,
+            0.0080837166473439,
+            -0.00448473417024141,
+            -0.016508775226236883,
+            -0.00958925666995396,
+            0.01100335153658195,
+            -0.0041819993634961685,
+            -0.01929493648733792,
+            -0.01282300765645703,
+            -0.011300260564838231,
+            0.0051733704121938884,
+            0.005996780699975213,
+            -0.0038111648304178544,
+            -0.0028279639348479987,
+            0.016640121845366295,
+            -0.010197365811240557,
+            0.009878730312873492,
+            -0.03335474019383245,
+            -0.03551826040406959,
+            0.004497847402931096,
+            0.01507007013703286,
+            0.0013085881889541252,
+            -0.02078680101065954,
+            0.010806315988966003,
+            -0.016499186313125556,
+            -0.00858013792516617,
+            -0.008522534127498645,
+            0.009649521088411411,
+            -0.007283003380127847,
+            0.01299542385123007,
+            -5.3226845309105124e-05,
+            0.030477683597373687,
+            0.0074153455615142755,
+            0.004161881195438522,
+            -0.00679081048132963,
+            0.008354912600159931,
+            -0.001621776416108105,
+            0.004080191054849395,
+            0.02120524874226696,
+            0.012440848502230418,
+            -0.00218041171518331,
+            0.006458415098595872,
+            0.005678055538564355,
+            -0.017186537377121235,
+            0.02507960402044456,
+            0.024678178106806115,
+            -0.008205288425409454,
+            0.004108402466376321,
+            0.005783335302169668,
+            -0.008846663924040241,
+            -0.006897707231869481,
+            0.006527795262026734,
+            -0.018474181903007085,
+            -0.0008721381678008483,
+            0.010598084358464908,
+            -0.02255441909004843,
+            -0.004840511019512733,
+            0.021745651362109403,
+            0.004631567631335321,
+            -0.018558972627958834,
+            0.02764788406651861,
+            -0.004942361238163609,
+            -0.0027290999934532494,
+            -0.010876135202650048,
+            0.01671497143541799,
+            -0.012173527207375091,
+            0.008443317346655462,
+            0.011021541861818043,
+            -0.006295639322287843,
+            0.003083685676540999,
+            0.008277206401503182,
+            -0.004504302913715969,
+            -0.0068075424927125676,
+            -0.0010508416962215848,
+            0.010167160518175114,
+            0.0036980815941666336,
+            -0.022704963865525812,
+            -0.006020073261002279,
+            0.02934227992246887,
+            -0.015082922782242507,
+            -0.0050916398557337705,
+            0.005376294716056329,
+            0.007709342013963906,
+            -0.003476236561444464,
+            -0.0023399991233097337,
+            -0.00787486645740744,
+            0.010356414979428607,
+            -0.015950083845400274,
+            0.02076360898078935,
+            0.025608263494550438,
+            0.024504236990840842,
+            -0.014814235990238182,
+            0.010828584133720797,
+            -0.011703992394890898,
+            -0.011047470905668532,
+            -0.015542835222173864,
+            -0.001658296303013357,
+            0.001970208852236451,
+            0.013107754130616248,
+            0.009666887546609378,
+            0.0235853683657408,
+            0.006473544089109194,
+            0.008973300312205858,
+            0.014118601565410305,
+            -0.003902202854013708,
+            -0.018303438244719996,
+            -0.014584226585460755,
+            -0.011977613285028946,
+            -0.01756345491020528,
+            0.0004563602312336091,
+            0.020461659996271687,
+            0.012417744207683839,
+            0.024136626592375227,
+            0.002119213701333967,
+            0.013759902247467922,
+            -2.9525392347047554e-05,
+            -0.000892319792935443,
+            0.00045783825646242434,
+            -0.005238483120312843,
+            0.019255756091120846,
+            -0.004599752813817219,
+            0.01656589522110913,
+            0.0006443773300684925,
+            0.007133940816771615,
+            0.013554761340256566,
+            -0.01704388669741371,
+            -0.003443100892108368,
+            -0.010029620309829285,
+            0.000941391966877521,
+            -0.019633111750884034,
+            0.01934571074127927,
+            0.010273330977233852,
+            0.0017121345993533963,
+            -0.018327189396014198,
+            -0.0029706385099949693,
+            0.03448635131998772,
+            0.019458999453011265,
+            0.0050734354363614525,
+            0.02632443373898792,
+            0.013018762352749236,
+            0.014443611411502168,
+            0.010153370950346075,
+            0.02356766253713842,
+            -0.023479811300626968,
+            0.02095255383200973,
+            -0.011590347850340786,
+            -0.007414136601134069,
+            -0.0035774888741646705,
+            0.0025141508183687115,
+            -0.013588691733097813,
+            0.00982877622088977,
+            -0.0017916539399655826,
+            -0.025386211797192578,
+            0.009164316443206016,
+            -0.008811052979979271,
+            0.010464442616928296,
+            -0.007774569252594923,
+            -0.0002806136360286729,
+            -0.0127084026190163,
+            -0.015184317235990752,
+            0.001259201880504347,
+            -0.009158429132281081,
+            0.023908150915412617,
+            -0.021210848953875233,
+            -0.01321384014436847,
+            0.013997502713857512,
+            -0.006869443694669701,
+            -0.011364239609894453,
+            -0.00374449980963666,
+            0.023673050288879224,
+            -0.009243341446512421,
+            -0.01359367519009039,
+            0.014464166953530989,
+            0.003834735113400469,
+            0.0059713317014342395,
+            -0.007508862694272452,
+            -0.008428705474135382,
+            -0.003280788036976103,
+            0.005998840043285335,
+            0.026615077617284676,
+            -0.014059930972065399,
+            -0.0032833264512892022,
+            -0.009189999799498163,
+            0.004118204537632102,
+            -0.02359889635281631,
+            0.02771075351384059,
+            -0.008871395303200375,
+            0.021705177758560575,
+            -0.0015924449308442096,
+            -0.006239188158271628,
+            -0.024231036074349822,
+            0.027522945599928003,
+            0.003683870966132199,
+            -0.010221180899173243,
+            0.0012768725460662862,
+            -0.052005292736885915,
+            0.016636109796947025,
+            0.008429895913163985,
+            -0.019507994429834353,
+            0.003697519981610208,
+            -0.004773584856005853,
+            0.018983847580558853,
+            -0.022966993228850697,
+            -0.014078986007109447,
+            0.008283766047449966,
+            -0.01153285124362746,
+            0.0031439180465649996,
+            -0.02527749501821793,
+            -0.005754537200570427,
+            0.0011760345167229269,
+            -0.005276936074280004,
+            -0.001705933750221904,
+            -0.020022761608132548,
+            -0.025972227850795115,
+            0.017666017379157135,
+            0.015820875579600418,
+            -0.002278262448951789,
+            0.01320818011561616,
+            0.008709899932124187,
+            0.00836573261091119,
+            0.016034263394940008,
+            0.007124509234103267,
+            0.0081473711608784,
+            0.045498533239850446,
+            -0.019846323044145485,
+            0.007371629414298193,
+            -0.0014854149174929534,
+            0.013951741898629716,
+            -0.030467924817555014,
+            -0.0055144480998344705,
+            0.0062802744565489945,
+            -0.0049373689340857915,
+            0.03510454786147091,
+            -0.017326094216003073,
+            0.01694160526735264,
+            -0.0026075809870445238,
+            0.0007420394265014909,
+            -0.02042003877666893,
+            -0.026156937726134134,
+            -0.014294938857971834,
+            0.0011857406889443987,
+            -0.0010047941261608975,
+            -0.011946583993069242,
+            0.006797001710574735,
+            0.004917161312925866,
+            -0.0020713379451245515,
+            0.015652523103715316,
+            -0.02952061566240878,
+            0.007356800041369564,
+            -0.010733578620243498,
+            0.020008504830359524,
+            -0.00990331048284697,
+            -0.0013319400098308093,
+            0.007236960885181047,
+            -0.004122868383988748,
+            0.013216043472071927,
+            -0.0038337251451538677,
+            0.004227115653354611,
+            -0.005221321656766545,
+            -0.01794343569722877,
+            -0.0016435261495179767,
+            0.021429008432348098,
+            -0.008349684096540855,
+            0.031864497493526205,
+            -0.01264743878037828,
+            0.009174982784686633,
+            0.010692990337462762,
+            0.0006628831036334152,
+            -0.005286285851702483,
+            0.009052971814211658,
+            0.0021431970274545893,
+            -0.008690520448278274,
+            0.006934310986093342,
+            -0.018349449937310143,
+            0.008494451785507706,
+            0.03394632739506833,
+            0.0018511328780472266,
+            -0.0007519661670070526,
+            -0.007266663179613392,
+            -0.01335844201823745,
+            0.018540713029383066,
+            0.005488273906733518,
+            -0.002546531152114063,
+            -0.027536660869590543,
+            0.02007746621157705,
+            0.0012265798861291223,
+            -0.004755325069578531,
+            0.012466557901411086,
+            0.0020495867632654885,
+            0.019989531151046736,
+            -0.010171678639113316,
+            0.030757957899447096,
+            0.010050036220124393,
+            0.002959833051227505,
+            0.0038556928367700113,
+            -0.036655749074851536,
+            -0.01652152778069314,
+            -0.00019357502472740266,
+            0.014271581054865031,
+            -0.012430647252826173,
+            -0.022432580814844155,
+            0.0010346169306977334,
+            0.01148004051449691,
+            -0.0033898642762881046,
+            -0.009650665305271706,
+            0.01560757541276278,
+            -0.020342213902670575,
+            0.006701302394408393,
+            0.007693672045981921,
+            0.0287100214318104,
+            -0.0020735384554022153,
+            -0.0012324838849990056,
+            0.000626416803904792,
+            -0.0055669860801192935,
+            -0.0008037366126870824,
+            0.017251369734470945,
+            -0.013661316947002466,
+            0.0175926211217534,
+            -0.012735241490550407,
+            0.0064713656749838585,
+            -0.0021092378432037563,
+            -0.010629579860050812,
+            -0.020114807971202787,
+            0.0029609495764307357,
+            0.009378003163375899,
+            0.006920144698428806,
+            -0.010438984881449953,
+            0.010715409812532968,
+            0.012197673689706352,
+            -0.00753962861484439,
+            0.014519905132928041,
+            0.022173989895384192,
+            -0.007687503607783572,
+            0.013633321414009471,
+            -0.0041446334728745095,
+            -0.014056264816568229,
+            -0.009985463824440833,
+            0.023929750617160402,
+            -0.0005563806942667597,
+            0.008756997373618809,
+            0.00388125051641897,
+            0.0007856447663040696,
+            -0.011226530517316183,
+            -0.001653447552601681,
+            0.019294448360333633,
+            0.011831934675073843,
+            -0.0013888534822008192,
+            -0.0010525764356699885,
+            -0.0014566936525653407,
+            -0.009427557739483988,
+            0.02049188741459728,
+            -0.022030783927622997,
+            -0.0019604940727076573,
+            0.00461018148439991,
+            -0.022589816803875618,
+            0.0001863117949313752,
+            -0.024570217962989905,
+            -0.01569952241187498,
+            0.011763033781122343,
+            -0.0022530631823737554,
+            -0.016028810867854907,
+            -0.0003242701040942633,
+            7.424127301806556e-05,
+            -0.007815789940508527,
+            -0.004135500768181258,
+            0.006115909992703774,
+            -0.00034193217510010117,
+            -0.0032262189374874842,
+            0.013430154310506884,
+            -0.009996622196870008,
+            0.015431528911575157,
+            0.0067029980830614145,
+            -0.015702167721800483,
+            0.0021406610839557003,
+            0.018500219544264753,
+            -0.0058910087864916085,
+            0.00787021425357441,
+            -0.014681907249972176,
+            -0.0037495551894426303,
+            -0.005787661351944086,
+            0.006051351977821761,
+            0.024849195780885833,
+            0.012969434450643372,
+            0.01139340445077682,
+            0.017405361843075933,
+            0.004325000715054734,
+            -0.01572297405297993,
+            0.0007416798945675923,
+            0.01682727189828363,
+            0.010464036837245312,
+            0.009804326468113703,
+            0.0008975556667684365,
+            0.0043704860361487635,
+            -0.005142176468800175,
+            0.009317064072855551,
+            0.02915482449687158,
+            0.0013434689612997423,
+            0.008696141010760192,
+            0.017276730345147134,
+            0.027038741396131265,
+            -0.0035088997194497725,
+            -0.007469614319788499,
+            0.008739942310300076,
+            -0.009657991604173173,
+            -0.001081629174976152,
+            -0.017000351183542026,
+            -0.012310275736750868,
+            0.016510505259358398,
+            -0.006084627062230492,
+            0.02848913486719956,
+            -0.0010655806791907225,
+            -0.02026498848361482,
+            -0.01565505608194376,
+            0.001404549479440637,
+            -0.01362132895080652,
+            0.005287566256051299,
+            0.014187265685427415,
+            0.01917046513020831,
+            0.004687191682875024,
+            -0.022258337721085656,
+            0.037364962197399806,
+            0.01757108963349427,
+            0.019377870795760123,
+            -0.019422840506328382,
+            -0.006159311237412833,
+            -0.009077095160676276,
+            0.02116697012944498,
+            -0.00880692095488465,
+            -0.009875674856043954,
+            0.024014332004451508,
+            0.017812606845656484,
+            0.005679806286464216,
+            0.006736609904079586,
+            0.0020048663439319585,
+            0.0025414630253661444,
+            -0.015831593851012047,
+            0.017957902498542212,
+            -0.005858125698347523,
+            -0.0058068554681761945,
+            0.017529514573571298,
+            -0.005989581792993925,
+            -0.0030306021203732347,
+            0.00015839997806822045,
+            -0.019697052145531482,
+            0.011618112886207201,
+            0.008256007601576028,
+            -0.017229774429089857,
+            -0.0014757560402567304,
+            -0.0002404480326945295,
+            -0.014118694740134205,
+            0.02004885146995733,
+            0.01694893356262835,
+            -0.0013248884139405398,
+            -0.0051752815372013315,
+            0.008223868147393114,
+            0.007000679132597393,
+            0.005256194890343006,
+            -0.019496098876977206,
+            0.005230597149342019,
+            0.005103909294824105,
+            -0.0014716649591412774,
+            0.010274346323051634,
+            -0.009030594825985526,
+            0.033740380586893826,
+            -0.006505788197604522,
+            0.013065658941136721,
+            -0.0043634731684193745,
+            -0.0055529072007985125,
+            0.001816020676232658,
+            0.013054468478598695,
+            0.009753501030700457,
+            -0.00970446155762939,
+            -0.004172354962129865,
+            0.018697982092775737,
+            -0.011983156146516074,
+            0.0013300791598046338,
+            0.01564072865804035,
+            -0.004858814041438713,
+            0.004708235083775013,
+            0.008635204790345423,
+            -0.004489505237783172,
+            0.01801512858970922,
+            -0.016482572999490366,
+            0.0023953436660054026,
+            0.010694869684962088,
+            0.02390864159830096,
+            -0.019152187046410703,
+            0.021786596486708207,
+            -0.017951050280323393,
+            0.026538582634436314,
+            0.014484894489694262,
+            -0.0035222259457311444,
+            -0.01946200885147006,
+            -0.00850690913394181,
+            0.011654565129063417,
+            -0.021423803876241532,
+            -0.0007985948078290641,
+            -0.0007290222944686444,
+            -0.021367628543092133,
+            -0.011189364017231132,
+            -0.009961779129778331,
+            0.010701637745810138,
+            0.001918633898887013,
+            0.010380983868185359,
+            0.008002182827196827,
+            0.010784760574518087,
+            0.014405640442760822,
+            0.0008958107769384412,
+            -0.010743001445440941,
+            -0.023795589003655273,
+            0.016987708698679597,
+            0.00438463274382093,
+            0.0007862212695139549,
+            -0.01627468994515667,
+            0.011936651269737248,
+            0.013250643815768723,
+            -0.009385605548926286,
+            0.009121882469425302,
+            -0.03716901103204463,
+            -0.004552808410398574,
+            -0.0035704811925430366,
+            -0.0020309197365173475,
+            -0.00908678158053104,
+            0.01559445415882485,
+            -0.013682507836005851,
+            -0.0009590922904844632,
+            0.00011696985057536548,
+            -0.0074756526903548915,
+            -1.5467890432254378e-05,
+            0.020649928837031005,
+            0.008438785939908428,
+            -0.0037382105257999757,
+            0.010376213224654074,
+            0.008826031303751312,
+            -0.013900359449287676,
+            -0.01527474712786084,
+            0.019035366035884273,
+            -0.018776918681908793,
+            0.0063098418544296165,
+            0.01677916431403883,
+            0.020778813096283988,
+            -0.02891862501830075,
+            -0.0007231469961304435,
+            0.022093939332021185,
+            -0.0012807635799950273,
+            0.006370841109790851,
+            0.008781853138377092,
+            0.006754082980602544,
+            0.0029102159062355374,
+            -0.003711173531879644,
+            -4.843770941622283e-06,
+            -0.003940474588075707,
+            0.007155539022862352,
+            0.004366433868349658,
+            0.036453728071874844,
+            -0.013661129348165794,
+            0.002541308805735001,
+            -0.0058838191638835155,
+            -0.00347401304572066,
+            -0.00593469688967835,
+            0.004407665856950772,
+            -0.010055727994798024,
+            0.004330827357536078,
+            -0.001570861535288093,
+            -0.0003731129422346693,
+            0.004883969598300302,
+            -0.001701673379646408,
+            0.007036892564844683,
+            -0.011863650867472625,
+            -0.0003979727750011258,
+            -0.020174851412609312,
+            0.021117284557875267,
+            0.00011666551282792189,
+            0.003662462447326642,
+            0.03723741246290771,
+            0.00476064510662738,
+            -0.00238355131307445,
+            -0.012880057949435798,
+            -0.013551338766321525,
+            -0.012986444917747435,
+            -0.020703821629987552,
+            -0.0036665673548525388,
+            -0.021006311026247164,
+            -0.0052061406234528915,
+            0.010362862594785442,
+            -0.011540184420787572,
+            -0.004023659825622843,
+            -0.015881589631889473,
+            -0.006531981517815174,
+            -0.03114211633687585,
+            -0.01124607488650473,
+            0.007493228366452009,
+            -0.019149285949049886,
+            -0.0039932870926865805,
+            0.026850121590606017,
+            0.001095360635760988,
+            0.01387696424607248,
+            0.0038058248676879625,
+            0.008117686785954085,
+            -0.0045937472017383955,
+            -0.0005669025036141417,
+            -0.01928244877716143,
+            0.003391980330529026,
+            0.008525354854576925,
+            -0.00361507909530282,
+            0.0070008404454106905,
+            -0.008835727374865623,
+            -0.016950017602980382,
+            0.0021654956715219517,
+            0.009334379529385142,
+            -0.006639140003365139,
+            -0.017587472740268683,
+            -0.011049929670847685,
+            0.010401189523318744,
+            -0.02283429470064187,
+            0.006257145380266984,
+            0.016439581169238607,
+            -0.009363030375624585,
+            0.004651575425738153,
+            0.019809183749755425,
+            -0.011810674576781702,
+            -0.005811328501528397,
+            0.008471992070998505,
+            0.013891259999721588,
+            0.0095606134736748,
+            0.02954498207274139,
+            0.0026596809804008996,
+            0.000561870602466584,
+            0.004889263753490895,
+            -0.004277061142504552,
+            -0.017423200644235665,
+            -0.006243787897080996,
+            0.0038507494787521856,
+            0.012873996671952184,
+            -0.009879417960879032,
+            -0.00048575277472139055,
+            0.01682604801673206,
+            0.004530939990187663,
+            -0.0033901248079550284,
+            -0.011868399508738274,
+            -0.020080331108085103,
+            0.0015580259874809898,
+            -0.01362200460317948,
+            0.015354347040143093,
+            0.003390601069331352,
+            0.0015284610287850575,
+            -0.004239880336806429,
+            0.01828645373058079,
+            -0.005013418527935815,
+            0.012749833447770729,
+            -0.006148890994661865,
+            -0.004587831231448011,
+            0.007986533783001116,
+            0.009166578151006778,
+            0.0004939857995863664,
+            0.05421019830078428,
+            0.002838056604877719,
+            0.01333205648597435,
+            0.013419127695838135,
+            -0.014523733369484934,
+            0.014086525147734823,
+            0.012688569853552613,
+            0.018497344547821693,
+            0.013037819487160016,
+            -0.012353776077273462,
+            0.017354801119939964,
+            0.0024084723134891487,
+            0.003909681065813095,
+            0.0022202438064449653,
+            -0.004528576079628071,
+            0.019594683950225115,
+            0.004070222328547378,
+            -0.0074362443729372105,
+            0.004453618452623641,
+            0.005007543168224267,
+            -0.013105524222311019,
+            -0.0112958915514782,
+            0.027105208848863218,
+            -0.01219417557128091,
+            -0.018749971743148304,
+            0.006706594269933445,
+            -0.02008458683853808,
+            -0.03763686535076864,
+            -0.0021332074289244973,
+            0.009992361098878196,
+            0.002795183524495193,
+            -0.00253695604043486,
+            0.01614298064601032,
+            0.016730443047719683,
+            -0.016985296937992726,
+            -0.018245902728910347,
+            0.0025257780765481825,
+            0.0016333325293570182,
+            0.01861807236979066,
+            -0.023168663546030196,
+            -0.006378864850313035,
+            -1.3795465857380962e-05,
+            -0.006829008132963306,
+            0.004850306884033312,
+            0.004703280112191218,
+            -0.0071714759384722965,
+            -0.0027773519096287217,
+            0.002256846645341137,
+            0.01975383140755998,
+            0.013198754584066554,
+            0.01572971337890945,
+            0.002600741959577911,
+            -0.005033508062449482,
+            -0.016817359058724685,
+            -0.005450011991586895,
+            -0.013478330855299777,
+            -0.009091854345715177,
+            -0.003042052092498012,
+            0.008217376329142566,
+            -0.025053469059115735,
+            -0.011965083373220982,
+            0.0119294363235832,
+            0.00391377331459168,
+            0.0003099203772539202,
+            0.007607672728136279,
+            -0.003830132495364688,
+            0.010331053246451333,
+            0.013823890473448383,
+            0.0076112887952959766,
+            -0.004723377809559611,
+            0.009600411526173948,
+            0.011460781436250457,
+            0.006015477068316837,
+            -0.005072711638895421,
+            0.023859899048048694,
+            0.01821920335673539,
+            -0.01680365156728093,
+            -0.002149107383573353,
+            -0.0027829676356423997,
+            -0.01884308738652242,
+            0.009665830801645276,
+            0.00880322398336952,
+            0.008308268605449346,
+            0.003906369351868086,
+            0.00974531700889854,
+            -0.013875904819275925,
+            -0.0050570811397453235,
+            0.002870120865445989,
+            -0.005729556771404781,
+            -0.01901337075686543,
+            0.000297242372701423,
+            -0.011987192261104132,
+            -0.011327157550660895,
+            -0.01409710922165158,
+            -0.003221240464411861,
+            0.005283472511900512,
+            0.021472187640986972,
+            -0.004410881709548286,
+            -0.001855852749039012,
+            0.01351697599517034,
+            0.010060125186506398,
+            0.0021658738823899905,
+            0.0002206474541239167,
+            0.005418828324605768,
+            -0.0027919130944783397,
+            0.0008094042430809068,
+            -0.0015826797905407809,
+            0.021089276765434876,
+            -0.02016611086095107,
+            -0.009413977569213423,
+            -0.009288078098692112,
+            0.005842197094339927,
+            0.028793032462603636,
+            -0.007165634502373454,
+            -0.010104002083017078,
+            -0.01994315293556678,
+            0.0003786527922888834,
+            -0.07810609175343784,
+            -0.005210636756258548,
+            0.001870513618910238,
+            0.03126380113105983,
+            -0.010817888476436834,
+            0.0033673768058629823,
+            -0.002204487854177595,
+            0.014542299655712928,
+            -0.001760107965206874,
+            -0.0013283746982194259,
+            0.0071954806737858056,
+            -0.005640400818010697,
+            -0.006210989097047217,
+            0.005904163661619133,
+            -9.945941028848348e-05,
+            -0.010266938705916303,
+            0.00057263080955709,
+            0.02209498395848068,
+            0.0159398346608872,
+            -0.006575439759804175,
+            0.01816748552021211,
+            -0.00719294553484007,
+            -0.009838738592324234,
+            0.005639288780502127,
+            -0.0019007683459126327,
+            -0.015369619176147573,
+            -0.011475977270803784,
+            0.009862808084641291,
+            -0.030069890644150284,
+            -0.00010725753918890802,
+            0.00943698424646706,
+            -0.013540245857270492,
+            -0.008293042205008774,
+            0.007212974960781723,
+            0.006708343474532583,
+            -0.0012017968151025369,
+            -0.0078026000239507145,
+            -0.02527526731750743,
+            0.0027578400032925625,
+            -0.002446811971857853,
+            -0.004899702349784173,
+            0.0019960465173086183,
+            0.001479503476855694,
+            -0.012645905858466798,
+            0.01216748847452763,
+            -0.013378877368040743,
+            -0.012053251571418766,
+            0.012508842412556306,
+            -0.00027167241745418584,
+            0.01682680930386839,
+            -0.020607299598265426,
+            0.0010702151063620052,
+            -0.028075439824932304,
+            -0.007671848153932823,
+            -0.015486630894165054,
+            -0.0009776136330459065,
+            -0.0012939921569540803,
+            -0.03481880531471923,
+            0.012889447665794033,
+            -0.005200149829198305,
+            -0.006094702500154477,
+            -0.00459778303079732,
+            -0.024583504817899775,
+            0.006072281238752663,
+            0.00607350167867964,
+            -0.006506508970816635,
+            -0.016816439635445445,
+            0.008428048142588516,
+            -0.0011040593013238872,
+            0.004873670535200424,
+            -0.007475800310154908,
+            -0.0005638920132076605,
+            -0.013047135093134946,
+            -0.005463454269764494,
+            -0.0008522562324122834,
+            0.01827123304871433,
+            0.0044055851550317835,
+            -0.014526381222031981,
+            0.016862445236481526,
+            0.013250692156820124,
+            0.01602714293656341,
+            -0.0071089293427753115,
+            -0.0004265739315250755,
+            -0.02750630108787212,
+            -0.00327225092954903,
+            -0.023687644154808972,
+            0.00041213204365507744,
+            0.009447660230507965,
+            0.0203905264889617,
+            0.012694421477479161,
+            0.004620617778996077,
+            -0.001557511764988156,
+            0.0066751090724502,
+            0.02696959560423252,
+            -0.01099183126742224,
+            -0.006969111598423376,
+            -0.01884504192789845,
+            0.01113023717919988,
+            0.025135748541199255,
+            0.003381303139273845,
+            -0.011317957771935042,
+            0.008112598754655146,
+            -0.0009247845526290428,
+            -0.010302980598863016,
+            0.02889587382004902,
+            -0.0391134481322538,
+            -0.015599591108049343,
+            -0.004605824192962023,
+            0.007406520608110251,
+            -0.015596917115435033,
+            0.00770468362805995,
+            -0.012614339656105264,
+            0.001699049329817941,
+            -0.005851477164035241,
+            0.03535087340659861,
+            -0.0031950960727572925,
+            0.0006909856233673865,
+            -0.003395233050301427,
+            0.003531135044131077,
+            0.012355948349937309,
+            -0.004444221317023453,
+            -0.030726300006909638,
+            -0.0022983766869702346,
+            -0.011198252640110788,
+            0.009274666307515955,
+            -0.016207714291658314,
+            -0.017342435976846468,
+            -0.009424737744066259,
+            -0.021827243671837953,
+            -0.012403084982789827,
+            0.029747453318486338,
+            -0.005824001042844527,
+            0.01747569398142269,
+            -0.005219060548639168,
+            -0.008365003057800934,
+            0.00021272411948388247,
+            0.01344335658390732,
+            -0.028498240040284714,
+            0.00041963499785238845,
+            -0.002322931702641349,
+            0.0126393550661767,
+            -0.001595134597082556,
+            -0.00983073906977773,
+            0.006903313245694627,
+            0.0010522208685011098,
+            -0.015648940549713764,
+            -0.0007573245570947632,
+            -0.008533504878052644,
+            0.020475329445149498,
+            0.00027791387213409814,
+            0.001074213629646005,
+            0.013864331671732542,
+            -0.02102352993295736,
+            0.01576938071930521,
+            -0.021722775924283996,
+            0.01893267442161547,
+            -0.006577103101038035,
+            0.005772016664149475,
+            0.008283223343661555,
+            0.014068789332250016,
+            -0.014030042849491258,
+            -0.00663752723080222,
+            -0.034436220972634726,
+            -0.0122621879114314,
+            -0.001946072976826902,
+            -0.005308941654794638,
+            -0.014773123927909993,
+            -0.0011054822266082056,
+            -0.026205461509573584,
+            -0.006460373065662098,
+            0.012822188046808264,
+            0.019900312098359022,
+            -0.008409070094265829,
+            0.014163132434889265,
+            -0.006743711872055091,
+            -0.0028230771810882492,
+            -0.012652021749331811,
+            -0.018061180380009395,
+            -0.01083672324723357,
+            -0.01890966636827258,
+            -0.0018635408070944168,
+            0.004709222372611061,
+            -0.015760618199235518,
+            0.011435131737642627,
+            0.03198874888333405,
+            -0.01567271472650171,
+            0.008243406188694909,
+            0.016411496328333205,
+            0.0034236971031973503,
+            0.011728079139322837,
+            0.0038311415233052795,
+            0.004621789116267879,
+            -0.011109450058055078,
+            -0.011473158133621661,
+            0.006524490579223986,
+            0.00954511918454185,
+            -0.004913142601134428,
+            0.024347072255007807,
+            0.0008745419206472714,
+            0.005045794664611216,
+            0.005304506902446091,
+            0.011488187043525824,
+            -0.013135271908307747,
+            0.019746484188715552,
+            -0.002055210086078267,
+            -0.015571665869889405,
+            -0.01535499169234956,
+            -0.01105419264826655,
+            -0.018626399757822328,
+            0.0017108083455869296,
+            0.0181439079855921,
+            -0.0012493542718277685,
+            0.012137753522284387,
+            -0.00924460959150499,
+            -0.005019437549649743,
+            -0.0027156377206998474,
+            -0.038211954658821295,
+            -0.0021878970332906457,
+            -0.018809487178512628,
+            -0.02498945889484935,
+            0.03505216568400238,
+            -4.26477081213836e-05,
+            0.00997643255528116,
+            -0.010310197159232386,
+            -0.0002454902098986129,
+            -0.011065403358802515,
+            0.014954998647970473,
+            0.004624123403100598,
+            0.008872923678983478,
+            0.008560280683876702,
+            -0.016419500515801093,
+            -0.002997557798375716,
+            0.007472390947008284,
+            -0.00374822278027057,
+            0.012680407710694952,
+            0.015353684671120731,
+            0.020886474047003733,
+            0.011161262397502876,
+            0.002045862158732102,
+            0.00887271411363814,
+            0.009563361150297699,
+            -0.0003513237086717377,
+            -0.010459621030424415,
+            0.015419039131459582,
+            -0.02204848739950703,
+            0.012172257374410352,
+            0.04346634265794738,
+            0.019197433209794286,
+            0.029488039318675732,
+            -0.02272810439246353,
+            0.01448092574459596,
+            -0.003801043409441206,
+            0.009476229966122152,
+            0.0049309968883087685,
+            0.02346191840355209,
+            0.003973840862678076,
+            0.0073546443568171405,
+            0.010647581332452525,
+            -0.00972051412996416,
+            0.025644022498404723,
+            -0.00846669916801108,
+            5.1518540791212095e-05,
+            -0.004407265804145216,
+            -0.026400029105519137,
+            0.01615193008014211,
+            0.0081722536401398,
+            0.0010702314763197463,
+            0.0010263623479474385,
+            -0.0019566604229607643,
+            0.007982949510899019,
+            0.007079322685771456,
+            -0.015274148711633536,
+            -0.003886053737590683,
+            -0.018527455865296014,
+            0.0012864569064749438,
+            -0.024337133883771334,
+            0.010466651250597847,
+            -0.017568706814132484,
+            -0.014822354159709854,
+            -0.021666681078831483,
+            -0.013232016867264097,
+            -0.02011653224168991,
+            0.0035858355517491515,
+            -0.008632755352062642,
+            0.013025728891203198,
+            -0.016962939287139683,
+            -0.021494971929779273,
+            0.008430887589398292,
+            0.00870802359711456,
+            0.007481282677955534,
+            0.009728368603458082,
+            0.014996157358749115,
+            0.01736292711410622,
+            -0.022712386143092265,
+            -0.023163001611599246,
+            -0.006682209717364464,
+            -0.003531495447167166,
+            0.006762379833125297,
+            0.00933681196321128,
+            0.012884185500343249,
+            -0.007077895849501418,
+            -0.007554762054641417,
+            -0.013283067570021503,
+            0.006197966297581871,
+            -0.003993067576677973,
+            -0.009464789349849612,
+            -0.0015662739825834614,
+            -0.0061389795593180686,
+            0.002753664732282926,
+            -0.0001791088888074743,
+            0.0007595261237141144,
+            0.017122834676182343,
+            -0.008176319370751018,
+            -0.009787747261447871,
+            -0.02027228453943018,
+            -0.010683273300243025,
+            -0.015604745521156111,
+            -0.0016279406232527138,
+            -0.006506713191861036,
+            -0.014420323628589382,
+            0.010888206569219648,
+            -0.011930708571093279,
+            -0.00887210117616784,
+            0.010807147705360856,
+            0.0073900910661282935,
+            -0.003431679006289002,
+            0.037804020313161775,
+            0.0011945597874820288,
+            0.00778778394048267,
+            -0.003366543306746973,
+            0.005296727102519836,
+            -0.017761251541598876,
+            -0.026382400396890007,
+            0.0010357716464853195,
+            -0.01775777668381489,
+            0.010807713750049491,
+            -0.008515368407459745,
+            0.0016051389544388124,
+            -0.00034958600424912995,
+            -0.009585854918521076,
+            0.0036549720706351473,
+            -0.008110927907950148,
+            0.004212219431447508,
+            -0.03507890639477636,
+            0.011409377585617926,
+            -0.01819120811524444,
+            -0.008962242411035718,
+            0.01710177594046791,
+            0.037560936862121835,
+            0.0028513388998398104,
+            0.00030058353811717895,
+            0.0014792184333546447,
+            0.004427921639326168,
+            0.021438127785303025,
+            0.006339483221420334,
+            -0.001187747087595264,
+            -0.02237982599829809,
+            0.0062435650294483695,
+            0.002200578793697352,
+            0.010092828474934744,
+            0.023105023225115266,
+            -0.016797326021070008,
+            -0.015378520287395565,
+            -0.012732527281000097,
+            -0.010797463778182385,
+            0.009881499418260473,
+            0.02801767062785128,
+            0.01811139137849365,
+            -0.017358098300028903,
+            -0.020207173948076473,
+            0.0016242901745373126,
+            -0.026331540602868762,
+            0.007269751661479859,
+            0.11940631114242901,
+            -0.0045382135734049305,
+            -0.005994407567723939,
+            -0.008679712231646732,
+            -0.004482553208064535,
+            0.016709571294385293,
+            -0.013387043582971358,
+            0.069179848974036,
+            -0.007221545718220735,
+            0.0037392229364642915,
+            0.0028916224964057385,
+            0.00462729158956839,
+            -0.009727339449635306,
+            0.011570145487622916,
+            -0.011287365791681587,
+            -0.012991994493845187,
+            -0.013076194601468383,
+            -0.0180830261130804,
+            -0.007272768410218907,
+            -0.004155564814758654,
+            0.004204993170487392,
+            -0.005525575983182816,
+            0.008051108079520902,
+            -0.017551645696802648,
+            -0.010800070994485792,
+            0.013007176804761548,
+            -0.0023627485852619817,
+            0.005390986239687555,
+            -0.029668390278675554,
+            -0.0007650391259819867,
+            -0.0002446394046947245,
+            0.021280292985495608,
+            0.003137771886236958,
+            -0.0019077298815862946,
+            -0.010810159675564049,
+            0.007040107942856957,
+            0.023835993694726953,
+            0.021875366566457236,
+            0.0180511566130312,
+            0.012135351541044354,
+            0.020483975362807574,
+            -0.008667079096167644,
+            0.026324312888512866,
+            -0.001552207146726937,
+            -0.00397810465151493,
+            0.0077577136352257004,
+            -0.009849600842081875,
+            -0.00816392268697345,
+            0.00677353736462267,
+            -0.017288900282504148,
+            -0.020418071729964713,
+            -0.020383762873109595,
+            0.025741408097358233,
+            -0.011755094311186617,
+            0.002737129201159793,
+            -0.008266210469668866,
+            0.027287436144516414,
+            -0.006283892307996516,
+            -0.0029440642954701632,
+            0.009428876742171362,
+            -0.03788574758692254,
+            0.008834605607057885,
+            0.006463342392514337,
+            0.026711196829464153,
+            -3.9827332346130654e-05,
+            0.005565561841469247,
+            -0.0006738085601962865,
+            -0.006104320995853506,
+            -0.010792830169647851,
+            2.4565453208679464e-05,
+            0.017983908748172493,
+            0.0035181716362611077,
+            0.016970206528324653,
+            -0.005449808667944701,
+            -0.023171388778833197,
+            0.007092509502738752,
+            -0.01555320330471131,
+            -0.023306285898839978,
+            -0.002028783456704567,
+            -0.03196436161255752,
+            0.0033412618908711163,
+            0.007527633574188688,
+            -0.01890570959143866,
+            -0.017859735533232564,
+            0.010329340599904344,
+            0.004382396539605985,
+            -0.0016746397858953155,
+            0.006643785064338525,
+            -0.03307812909454471,
+            -0.01927475592253288,
+            0.004686970010943116,
+            0.012733525168279355,
+            -0.012993441126970916,
+            0.015291832952199082,
+            -0.020491167895268936,
+            0.016451976170508816,
+            -0.012259387189535237,
+            0.018393274368854132,
+            -0.006997668664226754,
+            0.011843450716200625,
+            0.011457415642860215,
+            0.009560877634897626,
+            -0.01790001115302499,
+            0.0214879415944361,
+            0.0066829070506954115,
+            0.03900547021123392,
+            0.02512484499576633,
+            -0.012651294056575184,
+            -0.0018425680226742782,
+            -0.00011361458320146007,
+            0.014982647266052349,
+            0.0061874424115559,
+            -0.003689464024837091,
+            0.02705902205734107,
+            0.018239469373522537,
+            -0.021070704644859155,
+            0.020037508334843183,
+            -0.00973804771680803,
+            0.018876283979591423,
+            -0.013303080480396818,
+            0.010592424494628194,
+            0.004717870260467274,
+            0.01070131238869639,
+            0.009087656015197565,
+            0.006708673669681232,
+            0.025928065153829043,
+            0.03663367449576964,
+            -0.00511073108390551,
+            0.002161381031392467,
+            0.004864092591944181,
+            0.012814518637357098,
+            0.001556833001747549,
+            0.007915766323215207,
+            -0.007012785514562176,
+            0.005834786334750152,
+            -0.002311533032619821,
+            -0.00122162928193159,
+            0.019525999280692766,
+            0.000824163362963678,
+            0.012875575199417254,
+            0.013259376059185821,
+            -0.002390133687709065,
+            -0.005682644446302278,
+            0.0019262218417782657,
+            -0.018193478045923098,
+            -0.015131696847666434,
+            0.029463723011600933,
+            -0.011447039651934148,
+            -0.008083114389964213,
+            0.0020642857121204753,
+            0.012160202510990566,
+            -0.00866944494225225,
+            -0.005378300635997713,
+            0.0038280364584418335,
+            0.014427318067525318,
+            -0.01799358399064332,
+            0.003381165206832161,
+            0.005252552821854107,
+            0.019433554824232185,
+            -0.00028791985674147885,
+            -0.013487678738391956,
+            0.0005317922815671783,
+            0.014982152801567637,
+            0.011902348165369078,
+            0.0016056390299304837,
+            -0.006935777197137747,
+            -0.00446584737370504,
+            0.004586814866182051,
+            0.006269466956110759,
+            0.0027516492952993445,
+            0.008157662045793032,
+            -0.0025001885993292637,
+            -0.016648747436203294,
+            0.017735757469839637,
+            -0.0030669873770852725,
+            0.018058248627079793,
+            -0.0038238312452176745,
+            -0.0147172470474118,
+            0.0043328417868095355,
+            -0.001888507803054499,
+            -0.018556943283494755,
+            0.009407930200720534,
+            0.002560949424273885,
+            0.010411159836451958,
+            0.0052571230238294825,
+            -0.007499470181994448,
+            -6.145889479758644e-05,
+            -0.015163819825329975,
+            0.022836682800304944,
+            0.016390944816242627,
+            0.001713572907204073,
+            -0.0035687505973373878,
+            -0.031210299824387475,
+            0.0029076511444359886,
+            -0.007822231980296341,
+            0.017821806420465356,
+            0.008250605126658402,
+            0.007908773724009905,
+            0.008351355849444162,
+            -0.00743970642051451,
+            -0.002411981861500092,
+            0.012543476434788293,
+            0.0202407777715137,
+            -0.0024635818448627447,
+            0.02535771413579989,
+            -0.012533449594824386,
+            -0.0022489292647974992,
+            0.0012362917358624392,
+            0.01578939075186126,
+            -0.0008001783766371555,
+            0.006311837812042848
+          ],
+          "keyphrases": [
+            "Switch Transformers",
+            "Mixture of Experts",
+            "sparsely-activated model",
+            "training instability",
+            "multilingual settings"
+          ],
+          "title": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "12f04ba2-cf28-4e5a-aaa8-f47613191ec5",
+        "properties": {
+          "page_content": "D Switch Transformers in Lower Compute Regimes 29 ",
+          "embedding": [
+            0.023223955184221268,
+            0.008607305586338043,
+            -0.01453364733606577,
+            0.043360237032175064,
+            -0.007881038822233677,
+            0.03894453123211861,
+            0.03295178711414337,
+            0.07682663947343826,
+            0.0186422448605299,
+            0.02312435209751129,
+            -0.010591053403913975,
+            -0.009802534244954586,
+            0.008665407076478004,
+            0.01330521795898676,
+            -0.00808024313300848,
+            0.023273756727576256,
+            -0.016957305371761322,
+            0.01791182905435562,
+            0.009520327672362328,
+            -0.021547315642237663,
+            0.02274254336953163,
+            0.002058449201285839,
+            -0.04790874570608139,
+            0.03579045459628105,
+            -0.01648419350385666,
+            0.01867544651031494,
+            -0.016533995047211647,
+            -0.012840406969189644,
+            -0.023522762581706047,
+            0.02939930371940136,
+            -0.0005597446579486132,
+            0.009113618172705173,
+            -0.018758447840809822,
+            -0.025431808084249496,
+            -0.048473160713911057,
+            -0.027888666838407516,
+            -0.008001391775906086,
+            -0.017513418570160866,
+            -0.04266301915049553,
+            0.006719010882079601,
+            -0.04774274304509163,
+            -0.01855924353003502,
+            -0.026710039004683495,
+            0.03499363735318184,
+            0.02893449179828167,
+            0.0012471048394218087,
+            -0.011661779135465622,
+            0.01660039648413658,
+            0.01438424363732338,
+            -0.01910705678164959,
+            -0.009603329934179783,
+            -0.057968586683273315,
+            0.004353453870862722,
+            0.01889125257730484,
+            -0.026079224422574043,
+            0.02126510813832283,
+            0.025049999356269836,
+            0.11447633802890778,
+            -0.006486604921519756,
+            -0.0010707256151363254,
+            0.02028568461537361,
+            -0.02486739493906498,
+            -0.01766282320022583,
+            -0.03291858732700348,
+            -0.028585882857441902,
+            -0.01542176865041256,
+            -0.03071073442697525,
+            0.007947440259158611,
+            0.0005597446579486132,
+            -0.03542524576187134,
+            -0.04130178689956665,
+            -0.015604373067617416,
+            0.010881559923291206,
+            0.015371967107057571,
+            0.010873259976506233,
+            0.010449949651956558,
+            0.0406709723174572,
+            0.034628428518772125,
+            -0.02661043591797352,
+            0.01175308134406805,
+            0.01771262288093567,
+            -0.032221369445323944,
+            0.018625644966959953,
+            0.012201291508972645,
+            0.01673319935798645,
+            -0.001978559885174036,
+            -0.052291251718997955,
+            -0.04757673665881157,
+            0.03821411356329918,
+            0.00491371750831604,
+            -0.021846123039722443,
+            -0.02601282112300396,
+            0.003295178757980466,
+            -0.020401887595653534,
+            -0.008972514420747757,
+            0.04877196624875069,
+            -0.006113096140325069,
+            -0.01459174882620573,
+            0.025398606434464455,
+            0.008989115245640278,
+            0.0039674947038292885,
+            0.016807902604341507,
+            0.0043908050283789635,
+            -0.008997415192425251,
+            0.030146321281790733,
+            -0.011819482780992985,
+            -0.014367643743753433,
+            0.04452226310968399,
+            0.0186422448605299,
+            0.048738766461610794,
+            0.013047912158071995,
+            -0.024551987648010254,
+            -0.022344134747982025,
+            0.02325715683400631,
+            -0.003189351176843047,
+            -0.026710039004683495,
+            0.003627186641097069,
+            -0.012159790843725204,
+            -0.000483227166114375,
+            -0.00844960194081068,
+            0.026378029957413673,
+            0.013056212104856968,
+            -0.008881212212145329,
+            -0.01224279310554266,
+            -0.02101610228419304,
+            0.012201291508972645,
+            0.00771088432520628,
+            -0.02647763304412365,
+            0.006582057569175959,
+            0.01943906396627426,
+            0.0009275471675209701,
+            0.015172762796282768,
+            -0.00734982592985034,
+            -0.034163616597652435,
+            0.01629328913986683,
+            0.00948712695389986,
+            0.035724055022001266,
+            0.04681311920285225,
+            -0.001914233318530023,
+            -0.00859900563955307,
+            -0.0402393639087677,
+            -0.0012087164213880897,
+            0.0311423446983099,
+            0.00649075536057353,
+            -0.05415049567818642,
+            0.02388797141611576,
+            -0.005996893160045147,
+            0.03655407577753067,
+            0.004627360496670008,
+            0.005270625930279493,
+            -0.00020218764257151634,
+            0.03871212527155876,
+            -0.018775049597024918,
+            -0.0005281001213006675,
+            0.07596341520547867,
+            0.040139760822057724,
+            -0.02481759339570999,
+            -0.005204224493354559,
+            -0.0001957031199708581,
+            -0.01050805114209652,
+            0.015521370805799961,
+            0.022427136078476906,
+            -0.030810337513685226,
+            -0.024004174396395683,
+            0.005996893160045147,
+            0.004648111294955015,
+            0.04249701648950577,
+            0.007715034298598766,
+            -0.05577733367681503,
+            0.023622365668416023,
+            0.03731769323348999,
+            0.0220287274569273,
+            0.031374748796224594,
+            0.022559938952326775,
+            -0.004332703538239002,
+            0.01850944198668003,
+            -0.006075745448470116,
+            0.010151143185794353,
+            0.023921171203255653,
+            -0.021032702177762985,
+            -0.032885387539863586,
+            -0.016002783551812172,
+            -0.0109064606949687,
+            -0.017563220113515854,
+            0.003915618639439344,
+            0.011238468810915947,
+            0.02104930393397808,
+            -0.001071763108484447,
+            0.02762306109070778,
+            0.003759989980608225,
+            0.005399279296398163,
+            0.008134194649755955,
+            0.01791182905435562,
+            0.010383548215031624,
+            -0.019638270139694214,
+            -0.04359264299273491,
+            -0.00263116299174726,
+            0.035392045974731445,
+            -0.0014027334982529283,
+            -0.0031416250858455896,
+            -0.00401937123388052,
+            0.008715208619832993,
+            0.020385287702083588,
+            0.01194398570805788,
+            -0.016608696430921555,
+            -0.004839015658944845,
+            0.037550099194049835,
+            -0.016251789405941963,
+            0.009097017347812653,
+            -0.0258468184620142,
+            0.018227236345410347,
+            -0.005399279296398163,
+            0.008931013755500317,
+            0.013272017240524292,
+            -0.0014255590504035354,
+            0.00751997996121645,
+            0.025697413831949234,
+            0.009578429162502289,
+            -0.0377161018550396,
+            -0.01973787136375904,
+            0.021995525807142258,
+            0.00754903070628643,
+            0.023240555077791214,
+            -0.03778250515460968,
+            -0.02330695651471615,
+            0.011454273946583271,
+            0.015637574717402458,
+            -0.002007610397413373,
+            -0.002900919411331415,
+            -0.0402393639087677,
+            -0.038778528571128845,
+            0.006465854588896036,
+            0.014475545845925808,
+            -0.013819830492138863,
+            0.00028843191103078425,
+            -0.02727445214986801,
+            -0.00632060132920742,
+            0.05219164863228798,
+            -0.028287075459957123,
+            0.027888666838407516,
+            0.012334095314145088,
+            0.015313866548240185,
+            -0.01762962155044079,
+            -0.015513070859014988,
+            -0.0150399599224329,
+            0.008806510828435421,
+            -0.015919780358672142,
+            0.0648079514503479,
+            0.03321739286184311,
+            -0.013462921604514122,
+            -0.04116898402571678,
+            -0.01347122248262167,
+            0.002583436667919159,
+            -0.004127273801714182,
+            0.0014608348719775677,
+            0.02964830957353115,
+            -0.003932218998670578,
+            -0.015181062743067741,
+            -0.026129024103283882,
+            0.02016948163509369,
+            0.016102384775877,
+            -0.003091824008151889,
+            -0.030561331659555435,
+            0.029930515214800835,
+            -0.017048608511686325,
+            0.030826937407255173,
+            0.03612246364355087,
+            0.00788933876901865,
+            -0.0036126612685620785,
+            0.004760163836181164,
+            0.00200968561694026,
+            -0.0313415490090847,
+            -0.05806818976998329,
+            0.010325446724891663,
+            0.02085009776055813,
+            -0.032852184027433395,
+            -0.026278428733348846,
+            0.01188588421791792,
+            0.003847141982987523,
+            -0.010026639327406883,
+            0.001663152244873345,
+            0.0011174142127856612,
+            -0.00038388418033719063,
+            -0.007300024386495352,
+            -0.00700121745467186,
+            -0.04126858711242676,
+            -0.0063538020476698875,
+            0.04863916337490082,
+            -0.03834691643714905,
+            -0.0377161018550396,
+            0.005249875597655773,
+            -0.01296490989625454,
+            -0.006594507489353418,
+            0.01120526809245348,
+            0.034163616597652435,
+            -0.03489403426647186,
+            0.05401769280433655,
+            -0.027971668168902397,
+            0.02080029807984829,
+            -0.00859900563955307,
+            -0.015205963514745235,
+            -0.0016185387503355742,
+            0.02825387567281723,
+            -0.02443578466773033,
+            -0.028951091691851616,
+            -0.002340656006708741,
+            0.01588658057153225,
+            0.016044283285737038,
+            0.003556634997949004,
+            0.0351596400141716,
+            0.04485427215695381,
+            0.0056192344054579735,
+            -0.004855616018176079,
+            -0.06620237976312637,
+            -0.025979621335864067,
+            -0.010234144516289234,
+            0.020899899303913116,
+            -0.012400496751070023,
+            -0.011620277538895607,
+            -0.00472281314432621,
+            0.008706907741725445,
+            -0.036155663430690765,
+            -0.05584373697638512,
+            0.05268966034054756,
+            0.02085009776055813,
+            -0.03041192702949047,
+            -0.0025626863352954388,
+            0.026876043528318405,
+            -0.011429373174905777,
+            -0.028320277109742165,
+            -0.017264412716031075,
+            -0.019986877217888832,
+            0.012591401115059853,
+            0.007748235482722521,
+            0.025232603773474693,
+            0.026544034481048584,
+            0.0011475024512037635,
+            -0.020451689139008522,
+            -0.044289860874414444,
+            0.017978230491280556,
+            0.036321669816970825,
+            0.02926649898290634,
+            -0.04004015773534775,
+            -0.03456202521920204,
+            -0.03363240510225296,
+            -0.02186272293329239,
+            0.008798209950327873,
+            -0.007138170767575502,
+            -0.02923329919576645,
+            0.008237946778535843,
+            0.0010442687198519707,
+            -0.007573931012302637,
+            0.014774353243410587,
+            -0.03758329898118973,
+            0.062085483223199844,
+            -0.0478755459189415,
+            -0.02508319914340973,
+            -0.004056721925735474,
+            -0.025182802230119705,
+            -0.04824075475335121,
+            0.010815158486366272,
+            -0.021663518622517586,
+            0.0021912523079663515,
+            -0.024518785998225212,
+            -0.03894453123211861,
+            -0.03834691643714905,
+            0.03426321968436241,
+            -0.008773310109972954,
+            -0.021115705370903015,
+            0.021082503721117973,
+            -0.026129024103283882,
+            -0.0016403266927227378,
+            0.009072116576135159,
+            0.027772463858127594,
+            0.010806858539581299,
+            -0.018360039219260216,
+            0.03758329898118973,
+            -0.012491798959672451,
+            -0.02885149046778679,
+            0.008200596086680889,
+            -0.043526239693164825,
+            0.023605763912200928,
+            -0.04203220456838608,
+            -0.04837355762720108,
+            0.07735785096883774,
+            0.00859900563955307,
+            0.026793040335178375,
+            -0.014351042918860912,
+            -0.04143458977341652,
+            0.0336822047829628,
+            -9.992660488933325e-05,
+            -0.01703200675547123,
+            -0.02063429355621338,
+            -0.012566500343382359,
+            0.0347280316054821,
+            0.021281708031892776,
+            -0.0220287274569273,
+            0.014558548107743263,
+            0.007831237278878689,
+            -0.01910705678164959,
+            -0.005888990592211485,
+            0.013379920274019241,
+            -0.03084353730082512,
+            -0.018609045073390007,
+            0.008823110722005367,
+            0.01296490989625454,
+            -0.06839363276958466,
+            -0.013396520167589188,
+            -0.006573757156729698,
+            -0.00281169218942523,
+            -0.00595124252140522,
+            -0.05162723362445831,
+            0.011852683499455452,
+            -0.013562524691224098,
+            0.02765626087784767,
+            0.0010017302120104432,
+            0.0025128850247710943,
+            0.027805665507912636,
+            -0.022725943475961685,
+            -0.023057950660586357,
+            0.009395824745297432,
+            0.047510337084531784,
+            0.01880824938416481,
+            0.015596073120832443,
+            0.033781807869672775,
+            0.015562872402369976,
+            0.03396441042423248,
+            0.022510137408971786,
+            0.020816897973418236,
+            0.0020958001259714365,
+            0.02377176843583584,
+            -0.0013612326001748443,
+            -0.04163379594683647,
+            -0.009312822483479977,
+            -0.04608270153403282,
+            0.02511640079319477,
+            -0.014691350981593132,
+            0.036155663430690765,
+            -0.012973209843039513,
+            0.0017254037084057927,
+            0.028320277109742165,
+            -0.010383548215031624,
+            -0.02486739493906498,
+            0.020999502390623093,
+            -0.04963518679141998,
+            0.028187474235892296,
+            -0.008200596086680889,
+            0.016168786212801933,
+            -0.0037309392355382442,
+            0.013886231929063797,
+            -0.03469482809305191,
+            0.0258468184620142,
+            -0.013844731263816357,
+            -0.022759143263101578,
+            0.02126510813832283,
+            -0.03851291909813881,
+            0.027772463858127594,
+            0.01624348759651184,
+            -0.015322166495025158,
+            -0.03960854560136795,
+            0.03336679935455322,
+            0.033150993287563324,
+            0.032553378492593765,
+            -0.0347280316054821,
+            0.013338418677449226,
+            0.0423642136156559,
+            -0.05255685746669769,
+            -0.005967842880636454,
+            -0.02685944177210331,
+            0.02426978014409542,
+            -0.01311431359499693,
+            -0.00978593435138464,
+            0.008329248987138271,
+            0.05800178647041321,
+            -0.024087175726890564,
+            -0.0041438741609454155,
+            0.0018260437063872814,
+            -0.07616262137889862,
+            -0.014118637889623642,
+            -0.01370362751185894,
+            0.012425397522747517,
+            0.004959368612617254,
+            0.0026871892623603344,
+            -0.018758447840809822,
+            0.03122534602880478,
+            0.019721271470189095,
+            0.018907852470874786,
+            0.0330347903072834,
+            -0.05036560446023941,
+            0.018758447840809822,
+            -0.005880690645426512,
+            -0.026029422879219055,
+            -0.00212277565151453,
+            -0.0235061626881361,
+            -0.011595377698540688,
+            -0.06328071653842926,
+            -0.044123854488134384,
+            -0.04960198700428009,
+            0.025016797706484795,
+            0.03549164906144142,
+            -0.05109602212905884,
+            -0.021331509575247765,
+            -0.005839189514517784,
+            0.008457901887595654,
+            -0.02227773331105709,
+            -0.03409721702337265,
+            0.006283250171691179,
+            0.0009960237657651305,
+            -0.0025045848451554775,
+            0.0005753075238317251,
+            0.05939621850848198,
+            0.0022472788114100695,
+            0.028917891904711723,
+            -0.022941747680306435,
+            -0.027208050712943077,
+            0.006864264141768217,
+            -0.006191947963088751,
+            0.005486431065946817,
+            0.017463617026805878,
+            -0.007441128138452768,
+            0.006158747244626284,
+            -0.0047975145280361176,
+            0.007466028444468975,
+            0.04621550440788269,
+            -0.005984443239867687,
+            0.023057950660586357,
+            0.018077831715345383,
+            -0.009744432754814625,
+            0.04047176614403725,
+            0.009951937943696976,
+            0.007142320740967989,
+            0.024751191958785057,
+            -0.008947613649070263,
+            -0.021630316972732544,
+            -0.008486952632665634,
+            0.03535884618759155,
+            -0.012035287916660309,
+            -0.007835387252271175,
+            -0.026278428733348846,
+            0.020518090575933456,
+            0.01518936362117529,
+            0.04641471058130264,
+            -0.02219473011791706,
+            -0.009080417454242706,
+            -0.007573931012302637,
+            0.005258175544440746,
+            0.014458945952355862,
+            -0.011869283393025398,
+            0.013645526021718979,
+            0.003253677859902382,
+            -0.01791182905435562,
+            -0.0436258427798748,
+            -0.006768811959773302,
+            -0.009453926235437393,
+            -0.006042544264346361,
+            -0.0072585237212479115,
+            0.0011620278237387538,
+            0.023008150979876518,
+            -0.00014642068708781153,
+            -0.01654229499399662,
+            -0.007428677752614021,
+            0.020833497866988182,
+            0.01635139063000679,
+            0.011346370913088322,
+            0.013886231929063797,
+            -0.00018052931409329176,
+            0.01621858775615692,
+            0.04100298136472702,
+            -0.0070634689182043076,
+            -0.011196968145668507,
+            -0.022095128893852234,
+            0.0177956260740757,
+            -0.00961992982774973,
+            0.004436456132680178,
+            0.00422480097040534,
+            0.05521291866898537,
+            -0.026842841878533363,
+            -0.013944333419203758,
+            -0.015737175941467285,
+            0.022061927244067192,
+            0.00933772325515747,
+            -0.03144115209579468,
+            0.011130565777420998,
+            0.006582057569175959,
+            -0.017994830384850502,
+            -0.024452384561300278,
+            -0.006901614833623171,
+            -0.025747215375304222,
+            0.007187971845269203,
+            -0.0018654696177691221,
+            0.006345501635223627,
+            0.04262981936335564,
+            -0.017015406861901283,
+            -0.01546326931566,
+            0.006598657928407192,
+            0.010889860801398754,
+            0.02159711718559265,
+            -0.010806858539581299,
+            0.022808944806456566,
+            -0.04103618115186691,
+            0.015529671683907509,
+            0.02579701691865921,
+            0.013670426793396473,
+            -0.013230516575276852,
+            0.004448906518518925,
+            0.032984986901283264,
+            -0.02592981979250908,
+            -0.0005654510459862649,
+            -0.002218228066340089,
+            0.009819135069847107,
+            0.03592325747013092,
+            -0.00855750497430563,
+            0.020401887595653534,
+            -0.014168438501656055,
+            -0.017546620219945908,
+            0.024933796375989914,
+            -0.046149104833602905,
+            0.024966996163129807,
+            0.0035524850245565176,
+            -0.01711500994861126,
+            0.035392045974731445,
+            -0.017380615696310997,
+            0.011371271684765816,
+            0.024136977270245552,
+            -0.016177086159586906,
+            0.019040655344724655,
+            0.0055113318376243114,
+            -0.02707524783909321,
+            -0.013985834084451199,
+            -0.0018488692585378885,
+            0.021115705370903015,
+            -0.004357604309916496,
+            -0.013653826899826527,
+            -0.04146779328584671,
+            0.021680118516087532,
+            -0.022725943475961685,
+            0.009927037172019482,
+            -0.020186083391308784,
+            0.020601091906428337,
+            0.004635660909116268,
+            -0.007881038822233677,
+            0.017131609842181206,
+            0.032603178173303604,
+            -0.03213836997747421,
+            -0.019472265616059303,
+            0.013396520167589188,
+            0.012848706915974617,
+            0.02028568461537361,
+            0.007478478830307722,
+            0.020899899303913116,
+            0.03456202521920204,
+            0.009860635735094547,
+            0.0175798200070858,
+            0.017845427617430687,
+            0.013446321710944176,
+            -0.008130044676363468,
+            0.0037952656857669353,
+            -0.007619582116603851,
+            -0.012209592387080193,
+            0.0022970798891037703,
+            -0.0389777310192585,
+            -0.009968538768589497,
+            0.028735287487506866,
+            0.021846123039722443,
+            -0.012168090790510178,
+            -0.033067990094423294,
+            0.000248616881435737,
+            0.02151411399245262,
+            -0.01075705699622631,
+            0.024966996163129807,
+            -0.03502683714032173,
+            -0.0343296192586422,
+            0.006984617095440626,
+            -0.026145625859498978,
+            0.005565282888710499,
+            -0.026544034481048584,
+            -0.00019012641860172153,
+            0.026444431394338608,
+            -0.012217892333865166,
+            0.010350347496569157,
+            -0.01919005811214447,
+            0.020451689139008522,
+            0.008574104867875576,
+            -8.741146302781999e-05,
+            0.012309194542467594,
+            0.007374726235866547,
+            -0.041202183812856674,
+            -0.011429373174905777,
+            -0.02672663889825344,
+            0.03366560488939285,
+            -0.011811182834208012,
+            -0.006113096140325069,
+            -0.009669731371104717,
+            0.013346719555556774,
+            -0.019538667052984238,
+            -0.04189940169453621,
+            0.010823458433151245,
+            0.004980118945240974,
+            0.017231212928891182,
+            -0.021763119846582413,
+            0.008266997523605824,
+            -0.01512296125292778,
+            0.003008821979165077,
+            0.010292246006429195,
+            0.007312474772334099,
+            0.003932218998670578,
+            0.0065447064116597176,
+            0.00808024313300848,
+            -0.01419333927333355,
+            -0.012275993824005127,
+            0.011562176048755646,
+            -0.009570129215717316,
+            -0.00560263404622674,
+            0.06656759232282639,
+            0.03258657827973366,
+            0.0025398607831448317,
+            -0.02320735529065132,
+            -0.01512296125292778,
+            -0.03911053389310837,
+            -0.002927894936874509,
+            0.02661043591797352,
+            0.03057793155312538,
+            -0.01957186870276928,
+            -0.0046066101640462875,
+            -0.012441997416317463,
+            -0.015006759203970432,
+            0.03688608109951019,
+            0.004598309751600027,
+            0.024070575833320618,
+            -0.0029341201297938824,
+            0.0147245517000556,
+            0.009669731371104717,
+            -0.02699224464595318,
+            -0.0002613265533000231,
+            -0.0032702782191336155,
+            -0.006017643958330154,
+            0.012068488635122776,
+            0.03529244288802147,
+            0.0013384070480242372,
+            -0.024468984454870224,
+            0.01668339967727661,
+            0.0003351205086801201,
+            -0.016343090683221817,
+            0.02541520819067955,
+            0.001071763108484447,
+            -0.017181411385536194,
+            0.016500795260071754,
+            0.002089574933052063,
+            -0.017845427617430687,
+            0.039376143366098404,
+            -0.015928080305457115,
+            -0.05544532462954521,
+            0.00670241005718708,
+            -0.0008647768991068006,
+            -0.02694244496524334,
+            0.021464312449097633,
+            0.006914065219461918,
+            -0.00192772108130157,
+            0.01400243490934372,
+            0.014483846724033356,
+            -0.002081274753436446,
+            0.022393936291337013,
+            0.00719212181866169,
+            0.029465705156326294,
+            0.02554801106452942,
+            0.011802881956100464,
+            -0.013263717293739319,
+            -0.044920675456523895,
+            0.013902832753956318,
+            0.05833379551768303,
+            0.0381145104765892,
+            0.031059343367815018,
+            0.01434274297207594,
+            -0.004880516789853573,
+            0.0020750495605170727,
+            0.016284989193081856,
+            0.017148209735751152,
+            -0.03794850781559944,
+            0.0013653826899826527,
+            0.021962326020002365,
+            0.00855750497430563,
+            -0.007627882529050112,
+            -0.02792186848819256,
+            0.001390283228829503,
+            -0.017994830384850502,
+            0.008059492334723473,
+            0.02320735529065132,
+            -0.002274254336953163,
+            0.014624949544668198,
+            0.019173458218574524,
+            -0.00793083943426609,
+            0.016666797921061516,
+            -0.013446321710944176,
+            0.010474850423634052,
+            0.016550594940781593,
+            -0.004197825212031603,
+            -0.021613717079162598,
+            0.01158707682043314,
+            -0.045352283865213394,
+            -0.044289860874414444,
+            0.022642942145466805,
+            0.002205777680501342,
+            -0.020916501060128212,
+            0.004739413503557444,
+            0.00719212181866169,
+            -0.007129870355129242,
+            0.018144233152270317,
+            -0.04037216678261757,
+            0.013371619395911694,
+            -0.001614388544112444,
+            0.031241947785019875,
+            -0.0495687834918499,
+            0.019090456888079643,
+            0.036454472690820694,
+            -0.002919594757258892,
+            -0.01050805114209652,
+            -0.0003117762098554522,
+            -0.006308150943368673,
+            0.00015926006017252803,
+            -0.003587760729715228,
+            0.023024750873446465,
+            0.021215306594967842,
+            0.01321391575038433,
+            -0.023987572640180588,
+            -0.017928428947925568,
+            -0.018658846616744995,
+            0.035093240439891815,
+            -0.008798209950327873,
+            0.026212027296423912,
+            0.046912722289562225,
+            -0.01826043613255024,
+            0.00944562628865242,
+            -0.01071555633097887,
+            -0.01557117234915495,
+            -6.3183638303598855e-06,
+            0.02999691665172577,
+            -0.012026987969875336,
+            0.016152186319231987,
+            -0.03914373740553856,
+            0.010723856277763844,
+            0.018758447840809822,
+            0.013587424531579018,
+            -0.015330466441810131,
+            -0.0317731611430645,
+            -0.016774700954556465,
+            -0.015031659044325352,
+            -0.0003994470462203026,
+            0.006685809697955847,
+            0.03582365810871124,
+            0.05262325704097748,
+            0.04774274304509163,
+            0.024253180250525475,
+            0.008026291616261005,
+            -0.004270452074706554,
+            -0.05879860743880272,
+            0.008399800397455692,
+            0.030212722718715668,
+            0.00123361696023494,
+            -0.019306261092424393,
+            -9.441475413041189e-05,
+            -0.007810486946254969,
+            0.02528240531682968,
+            0.05023280158638954,
+            -0.027855465188622475,
+            0.025232603773474693,
+            -0.0026353129651397467,
+            0.0036126612685620785,
+            0.01194398570805788,
+            0.011852683499455452,
+            -0.04329383373260498,
+            -0.05398448929190636,
+            0.017397215589880943,
+            -0.01243369746953249,
+            -0.0034196816850453615,
+            0.013089412823319435,
+            -0.01782882586121559,
+            -0.024236580356955528,
+            -0.025730615481734276,
+            0.037550099194049835,
+            0.036188866943120956,
+            0.005191774107515812,
+            -0.004502857569605112,
+            -0.013462921604514122,
+            0.00810929387807846,
+            0.03665367513895035,
+            0.022891947999596596,
+            -0.0027535909321159124,
+            -0.0003333048371132463,
+            0.02385476976633072,
+            -0.017895227298140526,
+            -0.006536406464874744,
+            -0.013421420939266682,
+            0.026676837354898453,
+            -0.015537971630692482,
+            0.001002248958684504,
+            0.0018716948106884956,
+            0.007673533633351326,
+            0.01012624241411686,
+            0.009561828337609768,
+            -0.013761729001998901,
+            -0.0043078032322227955,
+            -0.039840951561927795,
+            0.020435089245438576,
+            -0.013180715031921864,
+            0.01867544651031494,
+            -0.010806858539581299,
+            0.0005216156132519245,
+            0.02863568440079689,
+            0.023456361144781113,
+            -0.015347067266702652,
+            -0.010109641589224339,
+            -0.015687374398112297,
+            0.02036868780851364,
+            -0.0073539759032428265,
+            0.028370078653097153,
+            0.003654162399470806,
+            0.0042829024605453014,
+            -0.03046172857284546,
+            0.026195425540208817,
+            0.0037662149406969547,
+            -0.021065903827548027,
+            0.0036375620402395725,
+            0.02481759339570999,
+            0.02765626087784767,
+            -0.019937077537178993,
+            -0.018409840762615204,
+            -0.01621858775615692,
+            0.02008648030459881,
+            -0.0020345861557871103,
+            0.03681968152523041,
+            -0.010425048880279064,
+            0.024170177057385445,
+            0.02194572426378727,
+            -0.025448407977819443,
+            0.0258468184620142,
+            -0.01226769294589758,
+            -0.004320253152400255,
+            0.018360039219260216,
+            -0.013886231929063797,
+            -0.0011091140331700444,
+            -0.027390655130147934,
+            0.016210287809371948,
+            0.00700121745467186,
+            -0.02533220499753952,
+            -0.007055168505758047,
+            0.024767791852355003,
+            -0.011346370913088322,
+            0.029515504837036133,
+            -0.007378876209259033,
+            -0.0032267021015286446,
+            0.0036292618606239557,
+            -0.004519457928836346,
+            -0.01012624241411686,
+            0.009478827007114887,
+            -0.006295700557529926,
+            -0.05833379551768303,
+            -0.021032702177762985,
+            0.024983597919344902,
+            -0.012109989300370216,
+            0.023223955184221268,
+            0.037981707602739334,
+            0.0021373010240495205,
+            -0.02617882564663887,
+            0.0007885188679210842,
+            0.022128328680992126,
+            -0.029034094884991646,
+            -0.01673319935798645,
+            -0.020833497866988182,
+            -0.03781570494174957,
+            -0.0019318711711093783,
+            -0.025299005210399628,
+            -0.006324751302599907,
+            0.04511987790465355,
+            0.007685983553528786,
+            -0.012035287916660309,
+            -0.007084219250828028,
+            0.014873955398797989,
+            -0.0004430231056176126,
+            0.006150446832180023,
+            0.008482802659273148,
+            0.029830913990736008,
+            -0.00192253349814564,
+            0.014774353243410587,
+            0.05484770983457565,
+            0.016467593610286713,
+            -0.012184691615402699,
+            -0.006387002766132355,
+            -0.01867544651031494,
+            -0.003214251948520541,
+            -0.004502857569605112,
+            0.0011827782727777958,
+            -0.02338995970785618,
+            0.04040536656975746,
+            -0.02888469025492668,
+            0.014525347389280796,
+            -0.02224453166127205,
+            0.0020491115283221006,
+            -0.05006679892539978,
+            0.010591053403913975,
+            -0.013803229667246342,
+            -0.02036868780851364,
+            -0.009918737225234509,
+            0.0002780566574074328,
+            0.05185963958501816,
+            0.0004969743895344436,
+            0.01867544651031494,
+            -0.004502857569605112,
+            -0.019206659868359566,
+            -0.015454969368875027,
+            -0.02046828903257847,
+            0.005270625930279493,
+            -6.296478386502713e-05,
+            0.018160834908485413,
+            0.0006251087179407477,
+            -0.0006152522400952876,
+            0.02418677881360054,
+            0.0288016889244318,
+            -0.019306261092424393,
+            -0.027506858110427856,
+            0.05899780988693237,
+            -0.013645526021718979,
+            -0.02579701691865921,
+            -0.02020268328487873,
+            0.005864090286195278,
+            -0.003253677859902382,
+            0.012251093052327633,
+            0.004847316071391106,
+            -0.002434033202007413,
+            -0.00297147105447948,
+            -0.02592981979250908,
+            -0.0004222725983709097,
+            -0.013064512051641941,
+            -0.015338766388595104,
+            -0.022676141932606697,
+            0.0008803398231975734,
+            0.0018696197075769305,
+            0.011670079082250595,
+            -0.008540904149413109,
+            -0.019173458218574524,
+            0.00047259253915399313,
+            -0.0006634971359744668,
+            -0.011138866655528545,
+            0.02792186848819256,
+            -0.012002087198197842,
+            -0.03562445193529129,
+            0.019854074344038963,
+            0.0024547837674617767,
+            0.017729224637150764,
+            0.0129234092310071,
+            0.014409144409000874,
+            0.009047216735780239,
+            0.0032184019219130278,
+            0.01812763325870037,
+            -0.02426978014409542,
+            0.002842817921191454,
+            -0.001082138391211629,
+            0.0030025967862457037,
+            -0.02868548594415188,
+            0.023904571309685707,
+            0.003488158341497183,
+            -0.012350695207715034,
+            -0.0005498881218954921,
+            0.002855268307030201,
+            -0.01867544651031494,
+            0.005129522643983364,
+            0.02524920366704464,
+            -0.0015614748699590564,
+            0.012358995154500008,
+            0.016193687915802002,
+            -0.01258310116827488,
+            0.0029133695643395185,
+            -0.0009591917041689157,
+            -0.005187624134123325,
+            -0.0020449613220989704,
+            0.016475893557071686,
+            0.020302286371588707,
+            0.010192643851041794,
+            -0.012026987969875336,
+            -0.03376520797610283,
+            0.00971953198313713,
+            -0.012566500343382359,
+            0.01673319935798645,
+            -0.007656933274120092,
+            0.018575843423604965,
+            -0.029432503506541252,
+            0.028270475566387177,
+            -0.013238816522061825,
+            -0.00683106342330575,
+            -0.015546271577477455,
+            -0.025016797706484795,
+            -0.023373357951641083,
+            0.011238468810915947,
+            0.025365406647324562,
+            -0.006938965991139412,
+            0.015977881848812103,
+            -0.012085088528692722,
+            -0.005245725624263287,
+            0.06291550397872925,
+            0.01586167886853218,
+            0.012649502605199814,
+            -0.023323558270931244,
+            -0.00913021806627512,
+            -0.01385303121060133,
+            -0.0298641137778759,
+            -0.0007330112857744098,
+            0.025066599249839783,
+            0.02754005789756775,
+            0.014857355505228043,
+            0.008615606464445591,
+            0.011761381290853024,
+            0.009578429162502289,
+            0.0034985337406396866,
+            -0.00613384647294879,
+            -0.009644830599427223,
+            2.2695854568155482e-05,
+            0.00281169218942523,
+            -0.008296048268675804,
+            -0.012184691615402699,
+            -0.0036583123728632927,
+            -0.009287921711802483,
+            0.015197663567960262,
+            -0.016633598133921623,
+            0.006611107848584652,
+            -0.011122265830636024,
+            0.0015365742146968842,
+            -0.019538667052984238,
+            -0.01336331944912672,
+            0.011827782727777958,
+            0.01396093424409628,
+            -0.0007003292557783425,
+            -0.00424555130302906,
+            -0.003886567894369364,
+            0.02219473011791706,
+            -0.023921171203255653,
+            0.013537623919546604,
+            0.01910705678164959,
+            0.017845427617430687,
+            -0.012226192280650139,
+            0.023622365668416023,
+            0.008781610056757927,
+            0.008781610056757927,
+            -0.03808131068944931,
+            -0.018277037888765335,
+            0.007225322537124157,
+            -0.0038222414441406727,
+            0.00012969059753231704,
+            0.02016948163509369,
+            0.003577385563403368,
+            0.011570476926863194,
+            -0.03374860808253288,
+            -0.0065322560258209705,
+            -0.011221867986023426,
+            0.006743911188095808,
+            0.006557156797498465,
+            -0.009503726847469807,
+            0.00756148062646389,
+            0.037848904728889465,
+            -0.002983921440318227,
+            0.0122842937707901,
+            0.0069555663503706455,
+            -0.002749440725892782,
+            -0.010084740817546844,
+            0.0147245517000556,
+            -0.016849402338266373,
+            0.013612325303256512,
+            -0.02546500973403454,
+            -0.002585511887446046,
+            -0.016301589086651802,
+            0.01973787136375904,
+            0.004345153924077749,
+            0.018609045073390007,
+            -0.0025917370803654194,
+            -0.020534690469503403,
+            -0.009188319556415081,
+            -0.017513418570160866,
+            0.002475534100085497,
+            -0.006407753098756075,
+            0.0018457566620782018,
+            -0.006357952021062374,
+            -0.00014655037375632674,
+            0.009196619503200054,
+            -0.019771073013544083,
+            -0.018791649490594864,
+            -0.019771073013544083,
+            -0.015488170087337494,
+            0.006250049453228712,
+            -0.014425745233893394,
+            -0.006785412319004536,
+            -0.002328205620869994,
+            0.005843339487910271,
+            0.041833002120256424,
+            -0.020833497866988182,
+            -0.028652284294366837,
+            0.02503339946269989,
+            -0.00022877422452438623,
+            -0.009171719662845135,
+            0.01370362751185894,
+            -0.0012211666908115149,
+            -0.005154423415660858,
+            0.018990853801369667,
+            -0.01889125257730484,
+            0.007229472976177931,
+            0.008673707023262978,
+            0.02109910361468792,
+            -0.008682007901370525,
+            -0.0017918053781613708,
+            0.00472281314432621,
+            0.0207504965364933,
+            -0.00898081436753273,
+            0.006478304974734783,
+            -0.006590357515960932,
+            0.006075745448470116,
+            0.002392532303929329,
+            0.043061431497335434,
+            -0.010109641589224339,
+            0.004353453870862722,
+            -0.006428503897041082,
+            -0.011221867986023426,
+            0.003284803591668606,
+            0.03212176635861397,
+            -0.012350695207715034,
+            -0.009453926235437393,
+            -0.0004974931362085044,
+            0.004258001688867807,
+            -0.0012782305711880326,
+            0.009669731371104717,
+            -0.002780566457659006,
+            -0.019422464072704315,
+            0.009370923973619938,
+            -0.0021808771416544914,
+            -0.010109641589224339,
+            -0.008839711546897888,
+            0.012417096644639969,
+            -0.014873955398797989,
+            0.003652087412774563,
+            -0.009022315964102745,
+            0.0038969432935118675,
+            0.017729224637150764,
+            0.02312435209751129,
+            -0.013545923866331577,
+            0.00015848191105760634,
+            -0.014019034802913666,
+            -0.02991391532123089,
+            -0.022344134747982025,
+            0.004980118945240974,
+            0.001614388544112444,
+            -0.007856138050556183,
+            0.007237772922962904,
+            0.0004977525095455348,
+            -0.011686678975820541,
+            0.007922539487481117,
+            0.001984784845262766,
+            -0.020003478974103928,
+            0.017646221444010735,
+            -0.005183474160730839,
+            0.014425745233893394,
+            0.004206125624477863,
+            0.012691003270447254,
+            0.004094073083251715,
+            -0.0008103068685159087,
+            -0.004677162040024996,
+            0.005909741390496492,
+            -0.01829363778233528,
+            0.02249353751540184,
+            0.04160059615969658,
+            -0.0004712956433650106,
+            0.012035287916660309,
+            0.013628926128149033,
+            0.01829363778233528,
+            -0.005058970768004656,
+            -0.010259045287966728,
+            0.002500434871762991,
+            -0.01744701713323593,
+            -0.005166873335838318,
+            -0.004938618279993534,
+            0.028204074129462242,
+            0.00562338437885046,
+            0.005818439181894064,
+            0.00597199285402894,
+            0.0006686847191303968,
+            0.013745129108428955,
+            0.007490929216146469,
+            0.026842841878533363,
+            0.020617693662643433,
+            -0.020783696323633194,
+            -0.011346370913088322,
+            -0.0039674947038292885,
+            0.018061231821775436,
+            0.018360039219260216,
+            -0.00022812576207797974,
+            -0.048307154327631,
+            0.026361430063843727,
+            -0.007295874413102865,
+            0.02486739493906498,
+            -0.00315615045838058,
+            0.021912524476647377,
+            0.01962166838347912,
+            -0.01523086428642273,
+            0.01834343932569027,
+            0.01923985965549946,
+            0.001934983767569065,
+            0.03384821116924286,
+            -0.028054671362042427,
+            -0.0063413516618311405,
+            -0.010416748933494091,
+            -0.005830889567732811,
+            0.01736401580274105,
+            -0.0006453404203057289,
+            -0.028585882857441902,
+            0.0021539016161113977,
+            0.023954372853040695,
+            -0.0046896119602024555,
+            0.022858746349811554,
+            -0.009495426900684834,
+            -0.028287075459957123,
+            -0.026461033150553703,
+            0.001020405674353242,
+            -0.023788368329405785,
+            -0.018924452364444733,
+            0.004780914168804884,
+            0.02850288152694702,
+            0.0039031682536005974,
+            -0.016334790736436844,
+            0.00545323034748435,
+            -0.02118210680782795,
+            0.013786629773676395,
+            0.005399279296398163,
+            -0.001994122751057148,
+            0.01665019802749157,
+            0.006893314886838198,
+            0.0053038266487419605,
+            0.0021497514098882675,
+            0.03335019573569298,
+            -0.01262460183352232,
+            -0.006266649812459946,
+            -0.008549204096198082,
+            0.003048247890546918,
+            0.00037999346386641264,
+            0.011072465218603611,
+            0.02516620233654976,
+            0.0008331324206665158,
+            0.005581883247941732,
+            -0.0003052916727028787,
+            0.017463617026805878,
+            -0.015388567931950092,
+            -0.03881172835826874,
+            0.00280131702311337,
+            -0.006171197630465031,
+            -0.021680118516087532,
+            0.015396867878735065,
+            -0.02885149046778679,
+            -0.007673533633351326,
+            -0.0008206821512430906,
+            0.0013581200037151575,
+            0.031756557524204254,
+            -0.005353628192096949,
+            -0.017347414046525955,
+            0.0014421595260500908,
+            -0.028702085837721825,
+            -0.0030129719525575638,
+            0.009927037172019482,
+            0.024385983124375343,
+            0.019306261092424393,
+            -0.01804463192820549,
+            0.018028030171990395,
+            0.027772463858127594,
+            -0.003533809445798397,
+            -0.023987572640180588,
+            -0.010707256384193897,
+            -0.009935337118804455,
+            0.011570476926863194,
+            0.0023116052616387606,
+            -0.013313517905771732,
+            0.010615954175591469,
+            -0.01075705699622631,
+            -0.0019412089604884386,
+            -0.0010655380319803953,
+            -0.013794929720461369,
+            0.052888862788677216,
+            0.006005193572491407,
+            0.0049220179207623005,
+            -0.03957534581422806,
+            0.005108772311359644,
+            -0.011180367320775986,
+            0.013911132700741291,
+            0.009736132808029652,
+            0.014873955398797989,
+            0.00668165972456336,
+            -0.001169290510006249,
+            0.01557117234915495,
+            -0.011147166602313519,
+            0.013728528283536434,
+            0.005652435123920441,
+            -0.0300799198448658,
+            0.0023302806075662374,
+            0.0006536406581290066,
+            0.0051253726705908775,
+            -0.004789214581251144,
+            -1.1988275218755007e-05,
+            0.007304174825549126,
+            0.025149600580334663,
+            0.014508746564388275,
+            0.008254547603428364,
+            -0.011578776873648167,
+            -0.023057950660586357,
+            0.01508146058768034,
+            0.009495426900684834,
+            -0.000578420062083751,
+            0.002919594757258892,
+            -0.030594531446695328,
+            0.005615084432065487,
+            0.004050496965646744,
+            0.010740457102656364,
+            0.01347122248262167,
+            -0.010815158486366272,
+            0.024385983124375343,
+            -0.015454969368875027,
+            -0.003757914761081338,
+            0.011163766495883465,
+            -0.05494731292128563,
+            0.00810929387807846,
+            0.03867892548441887,
+            0.004399104975163937,
+            -0.027307653799653053,
+            -0.01543836947530508,
+            -0.01910705678164959,
+            -0.000976829556748271,
+            0.0006375589873641729,
+            0.005527932196855545,
+            0.006988767068833113,
+            -0.016832802444696426,
+            0.011213568039238453,
+            -0.01880824938416481,
+            -0.034163616597652435,
+            -0.011869283393025398,
+            0.019605068489909172,
+            0.012940009124577045,
+            -0.023439761251211166,
+            0.018443040549755096,
+            -0.010425048880279064,
+            0.011279969476163387,
+            -0.01247519813477993,
+            -0.032885387539863586,
+            0.003793190699070692,
+            -0.015737175941467285,
+            0.007283424027264118,
+            0.0165173951536417,
+            -0.03159055486321449,
+            0.007839537225663662,
+            0.006611107848584652,
+            -0.013039612211287022,
+            -0.03029572404921055,
+            0.018160834908485413,
+            -0.0019806348718702793,
+            0.019040655344724655,
+            -0.01982087455689907,
+            0.019040655344724655,
+            -0.007590531371533871,
+            -0.00685181375592947,
+            -0.03313439339399338,
+            -0.007042718585580587,
+            -0.010682355612516403,
+            -0.0283866785466671,
+            -0.011977186426520348,
+            -0.009935337118804455,
+            0.02342315949499607,
+            -0.040438566356897354,
+            0.003598136128857732,
+            -0.012915108352899551,
+            0.017679423093795776,
+            0.014135237783193588,
+            -0.023871371522545815,
+            0.020684095099568367,
+            -0.018625644966959953,
+            -0.013628926128149033,
+            -0.006868414115160704,
+            -0.011304870247840881,
+            0.005648285150527954,
+            -0.000889158749487251,
+            -0.004494557622820139,
+            -0.005295526701956987,
+            -0.02270934358239174,
+            -0.03180636093020439,
+            -0.002793016843497753,
+            0.0006531218532472849,
+            -0.0025979620404541492,
+            0.012857007794082165,
+            0.02846967987716198,
+            -0.008453751914203167,
+            -0.0032163269352167845,
+            0.005855789873749018,
+            0.017679423093795776,
+            0.0008793022716417909,
+            0.023107752203941345,
+            -0.015347067266702652,
+            -0.02363896556198597,
+            0.0012605926021933556,
+            0.01574547588825226,
+            -0.010532951913774014,
+            -0.0017357789911329746,
+            0.01489055622369051,
+            -0.006270799785852432,
+            0.003816016251221299,
+            -0.02134811133146286,
+            -0.0288016889244318,
+            -0.0022701043635606766,
+            -0.002444408368319273,
+            -0.015703976154327393,
+            -0.0013435946311801672,
+            -0.012616301886737347,
+            -0.00893931370228529,
+            0.021447712555527687,
+            -0.01493205688893795,
+            -0.01139617245644331,
+            0.01037524826824665,
+            -0.01826043613255024,
+            0.008872912265360355,
+            0.0013321818551048636,
+            0.017696022987365723,
+            0.02066749334335327,
+            -0.002402907470241189,
+            0.011338070966303349,
+            0.013944333419203758,
+            0.0035794605500996113,
+            -0.0056316847912967205,
+            0.014052236452698708,
+            0.010225844569504261,
+            -0.011528975330293179,
+            0.001997235231101513,
+            0.011578776873648167,
+            0.01423483993858099,
+            -0.03095974028110504,
+            -0.022128328680992126,
+            -0.014575148932635784,
+            0.015181062743067741,
+            0.0192730613052845,
+            0.0013487822143360972,
+            -0.002954870695248246,
+            0.0049925693310797215,
+            -0.009113618172705173,
+            0.005171023774892092,
+            0.003899018280208111,
+            -0.0186422448605299,
+            0.004660561215132475,
+            0.0014815854374319315,
+            0.007685983553528786,
+            -0.006100645754486322,
+            0.010723856277763844,
+            -0.003911468666046858,
+            -0.011553876101970673,
+            -0.007163071073591709,
+            -0.01894105225801468,
+            -0.00889781303703785,
+            0.020020078867673874,
+            0.00022086309036239982,
+            -0.02211172878742218,
+            0.005656585097312927,
+            0.013413120992481709,
+            -0.015023359097540379,
+            -0.02334015816450119,
+            0.010026639327406883,
+            -0.0005804951069876552,
+            0.0008767084800638258,
+            0.010267345234751701,
+            -0.011761381290853024,
+            0.016907503828406334,
+            -0.014276341535151005,
+            0.010964562185108662,
+            0.0068227630108594894,
+            -0.04189940169453621,
+            0.010441649705171585,
+            0.019671469926834106,
+            -0.01224279310554266,
+            -0.01829363778233528,
+            0.0008450639434158802,
+            0.0277392640709877,
+            -0.0012159791076555848,
+            0.011935685761272907,
+            0.019986877217888832,
+            -0.0007024043006822467,
+            -0.012317494489252567,
+            -0.004851466044783592,
+            0.0918997973203659,
+            0.016309890896081924,
+            0.0136621268466115,
+            0.010200943797826767,
+            0.011836082674562931,
+            -0.030129719525575638,
+            -0.007225322537124157,
+            -0.008906112983822823,
+            -0.010823458433151245,
+            -0.017098408192396164,
+            -0.014434045180678368,
+            0.006777111906558275,
+            0.012790605425834656,
+            0.015919780358672142,
+            0.015952982008457184,
+            -0.00895591452717781,
+            0.007781436201184988,
+            0.008204746060073376,
+            0.0037662149406969547,
+            0.013031311333179474,
+            -0.0056316847912967205,
+            -0.0003042541502509266,
+            -0.011230168864130974,
+            0.010939661413431168,
+            0.014417444355785847,
+            0.011230168864130974,
+            -0.00262701278552413,
+            0.004590009804815054,
+            0.02500019781291485,
+            0.00793083943426609,
+            0.016060883179306984,
+            -0.016002783551812172,
+            0.0045941597782075405,
+            -0.013189014978706837,
+            0.040903378278017044,
+            0.015255765058100224,
+            0.0037765903398394585,
+            -0.005241575185209513,
+            0.0122842937707901,
+            0.004681312013417482,
+            0.018243836238980293,
+            -0.014940356835722923,
+            0.013545923866331577,
+            -0.009453926235437393,
+            -0.0038616673555225134,
+            0.03067753277719021,
+            -0.006677509751170874,
+            0.0029112945776432753,
+            -0.014093737117946148,
+            0.004560959059745073,
+            0.007980640977621078,
+            0.003305553924292326,
+            0.004830715712159872,
+            0.016633598133921623,
+            -0.01894105225801468,
+            -0.011138866655528545,
+            0.020551292225718498,
+            -0.013637226074934006,
+            -0.0017949179746210575,
+            0.017613021656870842,
+            0.006100645754486322,
+            -0.0013819830492138863,
+            0.010856659151613712,
+            -0.012334095314145088,
+            -0.007358125876635313,
+            0.008433002047240734,
+            0.006183648016303778,
+            4.756402631755918e-05,
+            0.005577733274549246,
+            0.024452384561300278,
+            0.005320427007973194,
+            -0.017231212928891182,
+            0.012699303217232227,
+            0.0008367637638002634,
+            0.010483150370419025,
+            0.017928428947925568,
+            0.011952285654842854,
+            0.01948886550962925,
+            -0.015977881848812103,
+            0.00719212181866169,
+            0.006100645754486322,
+            0.01572057604789734,
+            -0.02689264342188835,
+            -0.006992917042225599,
+            -0.0011672154068946838,
+            -0.0017814300954341888,
+            0.012715904042124748,
+            0.014840754680335522,
+            -0.00022125216491986066,
+            0.008972514420747757,
+            -0.0024278080090880394,
+            -0.010408448986709118,
+            0.02172992005944252,
+            -0.022725943475961685,
+            -0.02588001824915409,
+            -0.024634988978505135,
+            0.0073664262890815735,
+            -0.012558200396597385,
+            0.008304348215460777,
+            -0.005270625930279493,
+            0.009835734963417053,
+            0.004432306159287691,
+            0.015181062743067741,
+            0.002600037259981036,
+            -0.007586381398141384,
+            -0.014176739379763603,
+            0.021414512768387794,
+            -0.0033906311728060246,
+            0.009254720993340015,
+            -0.009960237890481949,
+            0.014301242306828499,
+            0.0038222414441406727,
+            0.00560263404622674,
+            -0.010242444463074207,
+            0.016641898080706596,
+            -0.014774353243410587,
+            0.002942420309409499,
+            0.016832802444696426,
+            -0.01736401580274105,
+            -0.012566500343382359,
+            -0.006719010882079601,
+            -0.011703279800713062,
+            0.006905765272676945,
+            -0.028071271255612373,
+            -0.011869283393025398,
+            0.007723334711045027,
+            -0.018077831715345383,
+            0.0071464707143604755,
+            -0.00963653065264225,
+            -0.016558896750211716,
+            0.0235061626881361,
+            0.027855465188622475,
+            -0.011379571631550789,
+            -0.004123123362660408,
+            -0.0025564611423760653,
+            0.003811866044998169,
+            -0.026660237461328506,
+            0.0006546781514771283,
+            -0.0056192344054579735,
+            -0.008773310109972954,
+            -0.003048247890546918,
+            0.016849402338266373,
+            0.002066749380901456,
+            0.0013435946311801672,
+            0.013462921604514122,
+            -0.006810312625020742,
+            -0.020534690469503403,
+            0.01109736505895853,
+            0.014458945952355862,
+            -0.01762962155044079,
+            0.003143700072541833,
+            0.0009285847190767527,
+            -0.019389264285564423,
+            0.01547987014055252,
+            0.00855750497430563,
+            0.016110684722661972,
+            -0.0004414668073877692,
+            0.013819830492138863,
+            -0.011819482780992985,
+            -0.005349477753043175,
+            -0.008590705692768097,
+            0.025780417025089264,
+            0.00685181375592947,
+            0.015073160640895367,
+            0.010873259976506233,
+            0.006270799785852432,
+            -0.00476431380957365,
+            0.027390655130147934,
+            0.023738566786050797,
+            -0.0007309362408705056,
+            -0.005361928138881922,
+            0.010076440870761871,
+            -0.005851639900356531,
+            0.0027100148145109415,
+            -0.017231212928891182,
+            0.0080511923879385,
+            0.0024713841266930103,
+            0.010400149039924145,
+            0.029465705156326294,
+            0.012151490896940231,
+            -0.014301242306828499,
+            0.02571401558816433,
+            -0.003058623056858778,
+            -0.023987572640180588,
+            -0.00014538316463585943,
+            -0.01425144076347351,
+            -0.009080417454242706,
+            0.010840059258043766,
+            -0.0014691351680085063,
+            -0.015106361359357834,
+            0.015297265723347664,
+            0.01453364733606577,
+            0.009927037172019482,
+            -0.008105143904685974,
+            0.004266302101314068,
+            0.00011510041076689959,
+            0.01315581426024437,
+            0.004901267122477293,
+            0.014326142147183418,
+            -0.004185375291854143,
+            0.0029237449634820223,
+            -5.317314571584575e-05,
+            -0.0014784728409722447,
+            -0.012201291508972645,
+            0.007881038822233677,
+            0.012358995154500008,
+            0.0033034789375960827,
+            0.013562524691224098,
+            0.00297147105447948,
+            -0.015820179134607315,
+            -0.03290198743343353,
+            -0.0009156156447716057,
+            0.010840059258043766,
+            -0.009794234298169613,
+            -0.01141277328133583,
+            0.010723856277763844,
+            0.007208722177892923,
+            0.009005715139210224,
+            0.013288618065416813,
+            0.003759989980608225,
+            -0.002585511887446046,
+            -0.010275646112859249,
+            0.021148905158042908,
+            0.007453578058630228,
+            -0.002770191291347146,
+            -0.0036686877720057964,
+            -0.010134542360901833,
+            -0.009586729109287262,
+            -0.0034196816850453615,
+            -0.014392544515430927,
+            0.013313517905771732,
+            0.007565631065517664,
+            0.007059318944811821,
+            0.011952285654842854,
+            0.016467593610286713,
+            0.0035919109359383583,
+            0.009022315964102745,
+            -0.010690655559301376,
+            -0.006175347603857517,
+            -0.020136281847953796,
+            -0.01850944198668003,
+            -0.014135237783193588,
+            0.0015510995872318745,
+            -0.006864264141768217,
+            -0.0004572890466079116,
+            -0.004614910576492548,
+            0.02131490968167782,
+            -0.011288270354270935,
+            -0.0012854932574555278,
+            -3.089878737227991e-05,
+            -0.008856311440467834,
+            -0.01298151072114706,
+            0.020916501060128212,
+            0.0016662648413330317,
+            0.0157786775380373,
+            0.007395477034151554,
+            0.026046022772789,
+            0.007026118226349354,
+            0.002187102334573865,
+            -0.015106361359357834,
+            -0.02431958168745041,
+            -0.011487474665045738,
+            0.021082503721117973,
+            -0.004193675238639116,
+            -0.0001918124035000801,
+            0.02334015816450119,
+            -0.011827782727777958,
+            0.007167221512645483,
+            -0.015313866548240185,
+            0.030312323942780495,
+            -0.008856311440467834,
+            0.012275993824005127,
+            0.012765705585479736,
+            0.01826043613255024,
+            -0.0018997079459950328,
+            0.01741381548345089,
+            0.0021829521283507347,
+            -0.008997415192425251,
+            -0.008383200503885746,
+            -0.0017679422162473202,
+            -0.021148905158042908,
+            -0.012782305479049683,
+            0.00771918473765254,
+            0.0050299204885959625,
+            -0.0054158796556293964,
+            0.013504423201084137,
+            0.0015355367213487625,
+            -0.0007049980922602117,
+            -0.00982743501663208,
+            0.00670241005718708,
+            -0.0013653826899826527,
+            -0.01523086428642273,
+            -0.0017181411385536194,
+            0.011454273946583271,
+            -0.027174849063158035,
+            -0.008856311440467834,
+            0.010010039433836937,
+            -0.00751997996121645,
+            -0.010433349758386612,
+            -0.00786028802394867,
+            0.0022763293236494064,
+            0.008154945448040962,
+            0.004158399533480406,
+            -0.009022315964102745,
+            0.0026456883642822504,
+            0.0015687375562265515,
+            -0.00035249904613010585,
+            -0.011968886479735374,
+            -0.005710536614060402,
+            -0.00898081436753273,
+            0.019937077537178993,
+            -0.007565631065517664,
+            -0.00580183882266283,
+            0.003598136128857732,
+            -0.0068227630108594894,
+            0.015056559816002846,
+            -0.013902832753956318,
+            -0.007885188795626163,
+            -0.01812763325870037,
+            0.021364711225032806,
+            -0.014683051034808159,
+            -0.00026443914975970984,
+            -0.008034592494368553,
+            -0.013280317187309265,
+            -0.0020646743942052126,
+            -0.0029693960677832365,
+            0.017214611172676086,
+            -0.0024278080090880394,
+            0.0003117762098554522,
+            -0.010516351088881493,
+            0.002209927886724472,
+            0.00788933876901865,
+            -0.013064512051641941,
+            0.0020677868742495775,
+            -0.0011402397649362683,
+            0.006241749040782452,
+            -0.007627882529050112,
+            -0.012051887810230255,
+            -0.007997240871191025,
+            0.019156858325004578,
+            0.012010387144982815,
+            -0.00773163465783,
+            -0.017081808298826218,
+            -0.007154771126806736,
+            0.013628926128149033,
+            0.014102037064731121,
+            0.005565282888710499,
+            -0.033649004995822906,
+            -0.003977870102971792,
+            -0.004859765991568565,
+            0.009121918119490147,
+            -0.022443735972046852,
+            -0.0074867792427539825,
+            -0.005270625930279493,
+            -0.02363896556198597,
+            -0.0010634629288688302,
+            0.010898160748183727,
+            0.006474155001342297,
+            -0.007470178417861462,
+            -0.009121918119490147,
+            -0.008291898295283318,
+            0.009951937943696976,
+            -0.01198548637330532,
+            0.0051959240809082985,
+            -0.0031416250858455896,
+            0.005805988796055317,
+            0.02126510813832283,
+            -0.018609045073390007,
+            0.0034093065187335014,
+            -0.005025770049542189,
+            -0.00947052612900734,
+            0.006710710469633341,
+            -0.007104970049113035,
+            -0.014649850316345692,
+            -0.006743911188095808,
+            -0.009495426900684834,
+            0.0004225319717079401,
+            0.007042718585580587,
+            0.0013684951700270176,
+            -0.028021469712257385,
+            0.004258001688867807,
+            -0.003836766816675663,
+            0.014832454733550549,
+            -0.00011587855260586366,
+            -0.008881212212145329,
+            0.011960585601627827,
+            -0.002284629736095667,
+            0.0050299204885959625,
+            0.012516699731349945,
+            -0.01071555633097887,
+            -0.023190755397081375,
+            -0.006868414115160704,
+            -0.010076440870761871,
+            0.005067271180450916,
+            -0.004361754283308983,
+            0.022941747680306435,
+            -0.03535884618759155,
+            0.014085437171161175,
+            -0.0074494280852377415,
+            -0.006528106052428484,
+            0.006972166709601879,
+            -0.005345327779650688,
+            0.027722662314772606,
+            0.00489296717569232,
+            0.008433002047240734,
+            -0.003654162399470806,
+            -0.008549204096198082,
+            -0.020385287702083588,
+            0.02443578466773033,
+            0.010574452579021454,
+            -0.0014286716468632221,
+            0.0023468811996281147,
+            0.006760511547327042,
+            0.011562176048755646,
+            0.024468984454870224,
+            -0.024717990309000015,
+            -0.007926689460873604,
+            -0.018111033365130424,
+            -0.013255417346954346,
+            -0.008416401222348213,
+            -0.011113965883851051,
+            -0.0006640158826485276,
+            -0.007237772922962904,
+            -0.0021310760639607906,
+            -0.0026228628121316433,
+            -0.0015479869907721877,
+            -0.004270452074706554,
+            -0.0014743227511644363,
+            -0.003079373622313142,
+            -0.015737175941467285,
+            -0.003106349380686879,
+            0.005224974825978279,
+            0.00948712695389986,
+            -0.004847316071391106,
+            0.0009778671665117145,
+            0.01056615263223648,
+            0.014467245899140835,
+            -0.007237772922962904,
+            -0.021464312449097633,
+            0.02385476976633072,
+            -0.010682355612516403,
+            0.000497233762871474,
+            -0.013238816522061825,
+            -0.009121918119490147,
+            -0.0034508074168115854,
+            0.0024402583949267864,
+            -0.0032267021015286446,
+            -0.015239164233207703,
+            -0.0031810509972274303,
+            -0.011421073228120804,
+            -0.016177086159586906,
+            0.01648419350385666,
+            0.03278578445315361,
+            -0.0015230864519253373,
+            0.016509095206856728,
+            -1.2742101716867182e-05,
+            0.010433349758386612,
+            -0.004805814940482378,
+            0.0005177248967811465,
+            0.003600211115553975,
+            -0.009578429162502289,
+            -0.004946918226778507,
+            0.007511679548770189,
+            -0.004021446220576763,
+            -0.006727310828864574,
+            -0.00961992982774973,
+            -0.017231212928891182,
+            -0.00893931370228529,
+            -0.005440779961645603,
+            -0.009263021871447563,
+            0.012350695207715034,
+            0.004345153924077749,
+            -0.008781610056757927,
+            0.014824154786765575,
+            0.00424555130302906,
+            -0.007034418173134327,
+            -0.011080765165388584,
+            -0.00013040390331298113,
+            0.0230911523103714,
+            0.023140953853726387,
+            -0.00809269305318594,
+            -0.026444431394338608,
+            0.01593638025224209,
+            -0.025597812607884407,
+            -0.004365904256701469,
+            -0.011371271684765816,
+            -0.01012624241411686,
+            -0.03838011622428894,
+            0.0087899100035429,
+            0.011180367320775986,
+            0.01635139063000679,
+            0.002259728964418173,
+            -0.0031167245469987392,
+            0.014608349651098251,
+            -0.008831411600112915,
+            -0.016235187649726868,
+            -0.004556809086352587,
+            0.003867892548441887,
+            -0.008706907741725445,
+            0.015089760534465313,
+            0.005436629988253117,
+            0.007793886587023735,
+            -0.0019754471722990274,
+            0.002940345322713256,
+            0.0036769879516214132,
+            0.021298309788107872,
+            0.004332703538239002,
+            0.004685461986809969,
+            0.0035836107563227415,
+            -0.0004712956433650106,
+            -0.010931361466646194,
+            0.007142320740967989,
+            -0.0013446321245282888,
+            -0.007055168505758047,
+            -0.00889781303703785,
+            0.030743934214115143,
+            0.02948230504989624,
+            0.010649154894053936,
+            -0.008266997523605824,
+            0.009495426900684834,
+            0.00908871740102768,
+            -0.015015059150755405,
+            0.033233996480703354,
+            -0.004955218639224768,
+            0.0014608348719775677,
+            0.012649502605199814,
+            0.016849402338266373,
+            -0.008287748321890831,
+            -0.006374552380293608,
+            0.007295874413102865,
+            -0.007395477034151554,
+            0.006988767068833113,
+            0.025946419686079025,
+            0.007706734351813793,
+            -0.004287052433937788,
+            -0.030528130009770393,
+            0.0038637423422187567,
+            -0.0031623756512999535,
+            -0.009478827007114887,
+            -0.005328727420419455,
+            0.01209338940680027,
+            -0.011670079082250595,
+            0.007818787358701229,
+            0.008466202765703201,
+            -0.0101677430793643,
+            0.0050880215130746365,
+            -0.00849110260605812,
+            0.0030752234160900116,
+            -0.01527236495167017,
+            -0.01654229499399662,
+            -0.005428329575806856,
+            0.01258310116827488,
+            -0.003857517149299383,
+            -0.008831411600112915,
+            0.012541599571704865,
+            0.01330521795898676,
+            0.01796162873506546,
+            -0.006864264141768217,
+            -0.0028677184600383043,
+            0.00908871740102768,
+            0.025863418355584145,
+            0.0016901278868317604,
+            -0.011852683499455452,
+            0.0013799079461023211,
+            0.008848011493682861,
+            0.017397215589880943,
+            0.007291724439710379,
+            -0.004818265326321125,
+            -0.01330521795898676,
+            -0.01741381548345089,
+            -0.001454609795473516,
+            0.003328379476442933,
+            -0.008565804921090603,
+            0.01103926356881857,
+            0.015056559816002846,
+            -0.009155118837952614,
+            0.007773135788738728,
+            0.0006463779718615115,
+            0.015961281955242157,
+            -0.022061927244067192,
+            -0.026212027296423912,
+            0.013197315856814384,
+            -0.003984095063060522,
+            -0.02028568461537361,
+            -0.00384506699629128,
+            0.001266817795112729,
+            0.022393936291337013,
+            -0.00034731143387034535,
+            -0.00038621859857812524,
+            0.00439495500177145,
+            -0.007138170767575502,
+            0.011006062850356102,
+            -0.02888469025492668,
+            -0.004490407183766365,
+            -3.7837231502635404e-05,
+            -0.0040878476575016975,
+            0.00018014023953583091,
+            -0.014898856170475483,
+            0.006727310828864574,
+            0.01733081415295601,
+            -0.008889512158930302,
+            0.01826043613255024,
+            -0.004697912372648716,
+            0.024070575833320618,
+            0.010607653297483921,
+            -0.0012657803017646074,
+            -0.0052332752384245396,
+            0.001739929080940783,
+            -0.028370078653097153,
+            0.004436456132680178,
+            -0.007358125876635313,
+            -0.010342047549784184,
+            -0.00489296717569232,
+            -0.024087175726890564,
+            0.013587424531579018,
+            0.006893314886838198,
+            0.018775049597024918,
+            0.004552658647298813,
+            0.008773310109972954,
+            0.020352086052298546,
+            -0.010856659151613712,
+            -0.0006178460316732526,
+            0.0036292618606239557,
+            0.012674403376877308,
+            0.0069182151928544044,
+            -0.004519457928836346,
+            -0.024800993502140045,
+            -0.030561331659555435,
+            0.0040359715931117535,
+            0.0016081634676083922,
+            -0.00978593435138464,
+            -0.0022763293236494064,
+            -0.011113965883851051,
+            0.0034508074168115854,
+            -0.02749025821685791,
+            -0.010250745341181755,
+            0.007312474772334099,
+            0.030146321281790733,
+            -0.006793712265789509,
+            -0.0005296564195305109,
+            -0.020518090575933456,
+            0.002039773855358362,
+            -0.02699224464595318,
+            -0.014268041588366032,
+            0.008320949040353298,
+            -0.00786028802394867,
+            0.021962326020002365,
+            -0.02999691665172577,
+            -0.001976484665647149,
+            -0.0046066101640462875,
+            -0.011852683499455452,
+            0.004560959059745073,
+            -0.013296918012201786,
+            -0.0063413516618311405,
+            0.0033470550552010536,
+            0.004174999892711639,
+            -0.00864050630480051,
+            0.016998806968331337,
+            -0.018841451033949852,
+            -0.012549900449812412,
+            -0.00456510903313756,
+            -0.01711500994861126,
+            -0.003201801562681794,
+            0.015712276101112366,
+            -0.007528279908001423,
+            0.005590183660387993,
+            0.017812225967645645,
+            0.0006666096742264926,
+            -0.007084219250828028,
+            -0.02672663889825344,
+            0.01593638025224209,
+            0.001115339226089418,
+            -0.004038046579807997,
+            -0.0005029401509091258,
+            0.02833687700331211,
+            0.00314577529206872,
+            0.007590531371533871,
+            0.014027335681021214,
+            -0.006528106052428484,
+            -0.017148209735751152,
+            0.008814810775220394,
+            -0.002479684306308627,
+            -0.01880824938416481,
+            0.010350347496569157,
+            -9.426885299035348e-06,
+            -0.004631510935723782,
+            -0.003811866044998169,
+            -0.016940705478191376,
+            -0.0064492542296648026,
+            -0.012466898187994957,
+            -0.0015282740350812674,
+            0.015330466441810131,
+            -0.005735436920076609,
+            -0.00933772325515747,
+            -0.014467245899140835,
+            -0.021829521283507347,
+            0.018459642305970192,
+            -0.011960585601627827,
+            8.036764484131709e-06,
+            -0.0022161530796438456,
+            -0.006457554176449776,
+            -0.0023323558270931244,
+            -0.012491798959672451,
+            0.000848176539875567,
+            -0.0028635684866458178,
+            -0.02164691686630249,
+            -0.0032059517689049244,
+            0.020318886265158653,
+            0.02850288152694702,
+            0.005743737332522869,
+            -2.742956166912336e-05,
+            -0.010765357874333858,
+            -0.02669343911111355,
+            0.01406053639948368,
+            0.009180019609630108,
+            -0.020435089245438576,
+            -0.01509806141257286,
+            0.01706520840525627,
+            0.0012720053782686591,
+            -0.005573583301156759,
+            -0.02558121085166931,
+            0.0032785783987492323,
+            0.003392706159502268,
+            -0.03125854581594467,
+            -0.0028635684866458178,
+            0.008823110722005367,
+            7.243220170494169e-05,
+            -0.01434274297207594,
+            0.016367990523576736,
+            -0.0017606796463951468,
+            0.011387872509658337,
+            0.007266823668032885,
+            0.0157786775380373,
+            -0.03678647801280022,
+            -0.033233996480703354,
+            0.013844731263816357,
+            0.010516351088881493,
+            -0.018526043742895126,
+            -0.011504075489938259,
+            0.013653826899826527,
+            -0.0002509512996766716,
+            -0.008125894702970982,
+            -0.0022618041839450598,
+            0.01266610249876976,
+            -0.03452882543206215,
+            -0.021447712555527687,
+            -0.009188319556415081,
+            0.011113965883851051,
+            0.002952795708552003,
+            0.011105665937066078,
+            0.014649850316345692,
+            -0.014110337011516094,
+            -0.010043240152299404,
+            -0.03894453123211861,
+            -0.004772614222019911,
+            0.009163418784737587,
+            -0.01733081415295601,
+            -0.0095286276191473,
+            0.003585685743018985,
+            -0.004751863423734903,
+            0.0084205511957407,
+            0.008881212212145329,
+            0.014525347389280796,
+            -0.002135226037353277,
+            0.015181062743067741,
+            0.0007112232269719243,
+            0.01943906396627426,
+            0.01283210702240467,
+            0.002712089801207185,
+            0.019854074344038963,
+            0.0007480554049834609,
+            -0.003438357263803482,
+            -0.004258001688867807,
+            0.002369706751778722,
+            0.009918737225234509,
+            -0.003120874520391226,
+            0.00016457737365271896,
+            0.020152881741523743,
+            0.007968190126121044,
+            0.009769333526492119,
+            -0.0021891773212701082,
+            0.03545844927430153,
+            -0.011769681237637997,
+            0.005814289208501577,
+            -0.005897291004657745,
+            -0.015513070859014988,
+            0.020833497866988182,
+            0.012765705585479736,
+            0.0016413643024861813,
+            0.018326837569475174,
+            0.00528307631611824,
+            0.0007226360030472279,
+            -0.00543248001486063,
+            -0.022227931767702103,
+            0.0013591574970632792,
+            0.017596419900655746,
+            -0.007698433939367533,
+            0.006096495781093836,
+            -0.004536058288067579,
+            0.023406559601426125,
+            0.0005312127177603543,
+            0.006897464860230684,
+            0.016417792066931725,
+            -0.006067445036023855,
+            0.003957119770348072,
+            -0.0006334088975563645,
+            0.0022784045431762934,
+            -0.011329771019518375,
+            -0.003199726575985551,
+            0.016318190842866898,
+            0.0010800632881000638,
+            0.0020034604240208864,
+            0.004614910576492548,
+            0.0002532857470214367,
+            0.0009825360029935837,
+            0.002994296606630087,
+            -0.0062085483223199844,
+            0.010101341642439365,
+            -0.0068601141683757305,
+            -0.006934816017746925,
+            -0.017762424424290657,
+            0.0063164508901536465,
+            0.029764512553811073,
+            -0.01050805114209652,
+            -0.02016948163509369,
+            0.0039052434731274843,
+            -0.005556982941925526,
+            0.003988245502114296,
+            -0.0008751521818339825,
+            -0.005025770049542189,
+            -0.01787862740457058,
+            -0.009121918119490147,
+            0.007914239540696144,
+            -0.0018135933205485344,
+            0.018575843423604965,
+            0.029681509360671043,
+            0.005673185456544161,
+            -7.55447763367556e-05,
+            0.009595029056072235,
+            0.004598309751600027,
+            0.001292755943723023,
+            0.0017752049025148153,
+            0.004760163836181164,
+            0.006922365631908178,
+            -0.020451689139008522,
+            -0.01001833938062191,
+            -0.019754473119974136,
+            -0.003938443958759308,
+            -0.009113618172705173,
+            -0.00839150045067072,
+            0.00650320528075099,
+            -0.002766041085124016,
+            -0.0017285163048654795,
+            0.0005358815542422235,
+            -0.0019080081256106496,
+            0.01054955180734396,
+            -0.019754473119974136,
+            -0.0054158796556293964,
+            -0.021779721602797508,
+            -0.004482107236981392,
+            0.003309704130515456,
+            -0.008354149758815765,
+            -0.016525695100426674,
+            -0.0038077160716056824,
+            0.03320079296827316,
+            -0.017613021656870842,
+            -0.00844960194081068,
+            -0.0062583498656749725,
+            0.0072585237212479115,
+            0.00927962176501751,
+            -0.02071729488670826,
+            0.0012356920633465052,
+            -0.00017806519463192672,
+            -0.01277400553226471,
+            -0.007851988077163696,
+            -0.017945028841495514,
+            -0.02634483017027378,
+            0.004818265326321125,
+            -0.0011962661519646645,
+            -0.01749681867659092,
+            -0.008947613649070263,
+            -0.015687374398112297,
+            -0.006494905333966017,
+            0.00843715202063322,
+            -0.024087175726890564,
+            0.010151143185794353,
+            0.012043587863445282,
+            -0.004193675238639116,
+            0.003143700072541833,
+            0.0063538020476698875,
+            0.016509095206856728,
+            0.005648285150527954,
+            0.005864090286195278,
+            -0.01425144076347351,
+            -0.017314214259386063,
+            0.0022244532592594624,
+            0.005710536614060402,
+            0.014367643743753433,
+            -0.004901267122477293,
+            0.0059180413372814655,
+            0.007980640977621078,
+            0.0020491115283221006,
+            0.01404393557459116,
+            0.025315605103969574,
+            -0.020302286371588707,
+            -0.01065745484083891,
+            0.0044987075962126255,
+            0.015396867878735065,
+            0.003959194757044315,
+            0.007138170767575502,
+            -0.014525347389280796,
+            -0.009935337118804455,
+            -0.001842644065618515,
+            0.007017817813903093,
+            -0.017596419900655746,
+            0.021829521283507347,
+            -0.00685181375592947,
+            0.02546500973403454,
+            0.014757752418518066,
+            0.02282554656267166,
+            0.017546620219945908,
+            -0.010914760641753674,
+            8.55309481266886e-05,
+            -0.0015376117080450058,
+            0.01542176865041256,
+            -0.008362449705600739,
+            -0.006768811959773302,
+            0.0028822438325732946,
+            0.009312822483479977,
+            -0.004901267122477293,
+            0.009379223920404911,
+            0.016807902604341507,
+            -0.005366078112274408,
+            0.0011962661519646645,
+            -0.014351042918860912,
+            -0.004197825212031603,
+            -0.010350347496569157,
+            0.0029361951164901257,
+            0.0033034789375960827,
+            -0.022045327350497246,
+            0.011910784989595413,
+            0.013545923866331577,
+            0.0024859094992280006,
+            -0.026909243315458298,
+            0.0063164508901536465,
+            0.0032578278332948685,
+            0.002357256365939975,
+            0.0122842937707901,
+            0.027224650606513023,
+            0.00406087189912796,
+            -0.006756361573934555,
+            -0.003811866044998169,
+            0.006868414115160704,
+            0.012126590125262737,
+            0.00263738795183599,
+            -0.006793712265789509,
+            -0.021298309788107872,
+            -0.009254720993340015,
+            -0.028917891904711723,
+            0.002919594757258892,
+            -0.026411231607198715,
+            0.0005919078830629587,
+            0.006432653870433569,
+            -0.009702932089567184,
+            -0.024419184774160385,
+            0.01842644065618515,
+            0.016152186319231987,
+            0.0017088033491745591,
+            0.009354324080049992,
+            -0.015703976154327393,
+            -0.00526232598349452,
+            0.015737175941467285,
+            -0.009362624026834965,
+            -0.011869283393025398,
+            -0.029714711010456085,
+            0.009503726847469807,
+            0.0017129534389823675,
+            0.0038741177413612604,
+            -0.010350347496569157,
+            0.0015791127225384116,
+            -0.014757752418518066,
+            0.006196097936481237,
+            -0.0008502515847794712,
+            -0.017247812822461128,
+            0.010200943797826767,
+            0.00368528813123703,
+            0.003071073442697525,
+            -0.003888642881065607,
+            -0.0014660225715488195,
+            0.010275646112859249,
+            0.016658497974276543,
+            -0.0032931037712842226,
+            -0.009960237890481949,
+            -0.0021165506914258003,
+            0.014409144409000874,
+            -0.005781088024377823,
+            0.016450993716716766,
+            -0.0011371271684765816,
+            0.0053162770345807076,
+            -0.003025422338396311,
+            0.002425733022391796,
+            0.0144008444622159,
+            -0.0005436629871837795,
+            0.007864437997341156,
+            0.019588468596339226,
+            -0.006474155001342297,
+            -0.01347122248262167,
+            0.00961992982774973,
+            -0.010649154894053936,
+            0.015977881848812103,
+            0.0031935013830661774,
+            -0.009387524798512459,
+            -0.01018434390425682,
+            0.014683051034808159,
+            0.023987572640180588,
+            -0.022061927244067192,
+            0.009221520274877548,
+            0.009412424638867378,
+            0.004104448016732931,
+            -0.01787862740457058,
+            0.007279274053871632,
+            -0.007075919304043055,
+            -0.0035939859226346016,
+            0.01018434390425682,
+            0.00889781303703785,
+            0.002546085976064205,
+            0.021547315642237663,
+            -0.011371271684765816,
+            -0.0012004162417724729,
+            -0.005573583301156759,
+            0.010840059258043766,
+            -0.017978230491280556,
+            0.0012720053782686591,
+            -0.00913021806627512,
+            -0.004436456132680178,
+            0.005212524440139532,
+            0.018227236345410347,
+            -0.011960585601627827,
+            -0.005980292800813913,
+            0.003371955594047904,
+            -0.014226539991796017,
+            -0.010068140923976898,
+            -0.00997683871537447,
+            -0.0044032554142177105,
+            0.008665407076478004,
+            0.0074369776993989944,
+            -0.018443040549755096,
+            -0.010624254122376442,
+            0.0008336511673405766,
+            -0.02159711718559265,
+            0.0036168114747852087,
+            -0.031109143048524857,
+            0.015322166495025158,
+            -0.0021705019753426313,
+            -0.006059145089238882,
+            -0.0023738567251712084,
+            0.018227236345410347,
+            -0.005540382582694292,
+            0.0023427309934049845,
+            0.0068725645542144775,
+            0.004569259472191334,
+            0.01659209653735161,
+            0.01523086428642273,
+            -0.010516351088881493,
+            -0.01207678858190775,
+            0.025348806753754616,
+            -0.007594681810587645,
+            0.01243369746953249,
+            -0.03698568418622017,
+            0.004237251356244087,
+            -0.014541947282850742,
+            -0.0031395500991493464,
+            -0.018144233152270317,
+            0.013936033472418785,
+            0.003759989980608225,
+            0.005171023774892092,
+            -0.0026436131447553635,
+            0.013338418677449226,
+            0.008715208619832993,
+            -0.008906112983822823,
+            0.008117593824863434,
+            -0.00849110260605812,
+            0.012840406969189644,
+            -0.024883994832634926,
+            0.005025770049542189,
+            0.012383895926177502,
+            -0.0011682529002428055,
+            0.007229472976177931,
+            -0.01662529818713665,
+            0.004868066404014826,
+            0.00908871740102768,
+            -0.021331509575247765,
+            0.02197892591357231,
+            -0.008354149758815765,
+            -0.00166522734798491,
+            -0.020783696323633194,
+            -0.009844035841524601,
+            0.014367643743753433,
+            -0.0006354839424602687,
+            -0.012508398853242397,
+            0.015579472295939922,
+            0.019771073013544083,
+            0.0005976142711006105,
+            0.023057950660586357,
+            -0.017297614365816116,
+            0.010408448986709118,
+            -0.0017492668703198433,
+            -0.00029595394153147936,
+            -0.0074867792427539825,
+            0.0122842937707901,
+            0.0015116736758500338,
+            -0.0006769849569536746,
+            -0.01419333927333355,
+            -0.005133672617375851,
+            -0.013786629773676395,
+            0.02885149046778679,
+            0.02312435209751129,
+            -0.0006583094946108758,
+            0.0031706758309155703,
+            -0.0038077160716056824,
+            -0.0034279818646609783,
+            0.004370054695755243,
+            -0.01914025843143463,
+            -0.01965487003326416,
+            -0.0036707627587020397,
+            -0.013819830492138863,
+            0.018526043742895126,
+            0.018775049597024918,
+            -0.006901614833623171,
+            0.019721271470189095,
+            -0.013745129108428955,
+            0.013778329826891422,
+            -0.0066443090327084064,
+            -0.010051540099084377,
+            -0.003716413863003254,
+            0.002954870695248246,
+            -0.0063164508901536465,
+            0.025149600580334663,
+            0.008233796805143356,
+            -0.0230911523103714,
+            0.013902832753956318,
+            -0.010732156224548817,
+            -0.015703976154327393,
+            0.0015178987523540854,
+            -0.0177956260740757,
+            0.025216002017259598,
+            0.012840406969189644,
+            0.0003945188072975725,
+            -0.01400243490934372,
+            0.016011083498597145,
+            0.008183995261788368,
+            0.002693414455279708,
+            0.011620277538895607,
+            0.02113230526447296,
+            -8.961620187619701e-05,
+            -0.0177956260740757,
+            -0.0012678552884608507,
+            0.019339462742209435,
+            -0.02224453166127205,
+            -0.004177074879407883,
+            0.010640854015946388,
+            0.017729224637150764,
+            0.009893836453557014,
+            -0.0005239500314928591,
+            -0.0070510185323655605,
+            0.0010364872869104147,
+            -0.010781957767903805,
+            -0.01995367743074894,
+            0.01855924353003502,
+            -0.011329771019518375,
+            0.020916501060128212,
+            0.016251789405941963,
+            0.02148091420531273,
+            -0.005843339487910271,
+            -0.021331509575247765,
+            0.005440779961645603,
+            -0.010541251860558987,
+            -0.010914760641753674,
+            0.026361430063843727,
+            -0.009072116576135159,
+            -0.004407405387610197,
+            0.006050844676792622,
+            -0.012275993824005127,
+            0.013537623919546604,
+            -0.014367643743753433,
+            0.0038844929076731205,
+            -0.013413120992481709,
+            -0.022676141932606697,
+            0.009213220328092575,
+            0.004768464248627424,
+            0.013205615803599358,
+            -0.01902405545115471,
+            -0.01662529818713665,
+            0.006976316682994366,
+            0.01812763325870037,
+            -0.00788933876901865,
+            0.008997415192425251,
+            0.013272017240524292,
+            -0.003542109625414014,
+            -0.010350347496569157,
+            0.037417296320199966,
+            0.004872216377407312,
+            -0.002952795708552003,
+            0.006988767068833113,
+            0.006594507489353418,
+            0.024070575833320618,
+            -0.005710536614060402,
+            0.02410377562046051,
+            -0.019970277324318886,
+            7.839797035558149e-05,
+            -0.00401937123388052,
+            -0.0002860974636860192,
+            0.017214611172676086,
+            -0.01736401580274105,
+            0.017812225967645645,
+            -0.0022618041839450598,
+            0.004469656851142645,
+            0.04777594283223152,
+            -0.012715904042124748,
+            -0.028585882857441902,
+            -0.018409840762615204,
+            -0.025481609627604485,
+            -0.0009156156447716057,
+            -0.001976484665647149,
+            -0.01990387588739395,
+            0.0017326663946732879,
+            -0.0029590206686407328,
+            -0.00722117256373167,
+            0.004938618279993534,
+            0.012334095314145088,
+            -0.009653130546212196,
+            0.021331509575247765,
+            0.020551292225718498,
+            0.013446321710944176,
+            0.006951416376978159,
+            0.005648285150527954,
+            -0.005561132915318012,
+            0.0008476577932015061,
+            0.00026612510555423796,
+            -0.0013684951700270176,
+            -0.03213836997747421,
+            0.02172992005944252,
+            -0.015869978815317154,
+            0.00472281314432621,
+            0.011802881956100464,
+            -0.03358260169625282,
+            -0.004668861627578735,
+            -0.006586207542568445,
+            0.009138518944382668,
+            -0.008167395368218422,
+            0.01817743480205536,
+            -0.016741501167416573,
+            -0.00506312120705843,
+            0.008806510828435421,
+            0.013014711439609528,
+            0.00927962176501751,
+            0.015264065004885197,
+            0.016035983338952065,
+            0.010707256384193897,
+            -0.013728528283536434,
+            0.02101610228419304,
+            -0.019007453694939613,
+            -0.0044032554142177105,
+            0.01957186870276928,
+            -0.012358995154500008,
+            0.016002783551812172,
+            0.022974949330091476,
+            -0.00579353841021657,
+            -0.024551987648010254,
+            0.01593638025224209,
+            0.008914412930607796,
+            0.01435934379696846,
+            0.044887471944093704,
+            -0.0012554050190374255,
+            -0.01689920388162136,
+            -0.027324253693223,
+            -0.02224453166127205,
+            -0.01105586439371109,
+            0.013911132700741291,
+            -0.003187276190146804,
+            0.0033657304011285305,
+            0.02194572426378727,
+            -0.0030316475313156843,
+            -0.014840754680335522,
+            0.011869283393025398,
+            0.001804255647584796,
+            0.0643763393163681,
+            0.004258001688867807,
+            -0.005681485868990421,
+            0.018775049597024918,
+            -0.0062583498656749725,
+            -0.012217892333865166,
+            0.006561306770890951,
+            -0.0063164508901536465,
+            -0.012906808406114578,
+            -0.01850944198668003,
+            0.010134542360901833,
+            -0.023140953853726387,
+            0.004278752487152815,
+            0.004756013862788677,
+            0.0025481609627604485,
+            -0.017513418570160866,
+            -0.010109641589224339,
+            -0.022178130224347115,
+            0.0015490244841203094,
+            -0.017812225967645645,
+            -0.018609045073390007,
+            -0.04027256369590759,
+            -0.016691699624061584,
+            0.016401192173361778,
+            -0.00908871740102768,
+            0.015031659044325352,
+            -0.008453751914203167,
+            -0.010350347496569157,
+            -0.009304522536695004,
+            -0.006308150943368673,
+            -0.005374378524720669,
+            0.014981858432292938,
+            0.025863418355584145,
+            -0.004681312013417482,
+            -0.004469656851142645,
+            0.004419855773448944,
+            -0.013753429055213928,
+            0.011487474665045738,
+            -0.010748757049441338,
+            0.011636878363788128,
+            -0.00386996753513813,
+            0.01889125257730484,
+            0.0003467926580924541,
+            -0.008723508566617966,
+            -0.02778906375169754,
+            -0.012740804813802242,
+            -0.0010925136739388108,
+            -0.00773163465783,
+            0.010300545953214169,
+            -0.005229125265032053,
+            0.003774515353143215,
+            0.0015510995872318745,
+            0.0034217569045722485,
+            -0.020534690469503403,
+            -0.007088369689881802,
+            -0.00034860832965932786,
+            0.0030025967862457037,
+            0.008183995261788368,
+            0.006880864500999451,
+            -0.005627534352242947,
+            0.0035732353571802378,
+            -0.017646221444010735,
+            -0.010649154894053936,
+            0.01580357737839222,
+            -0.013628926128149033,
+            -0.002390457084402442,
+            0.008673707023262978,
+            -0.013844731263816357,
+            -0.026577236130833626,
+            0.009578429162502289,
+            0.011421073228120804,
+            0.00367491296492517,
+            -0.021464312449097633,
+            -0.0003724714042618871,
+            0.012989810667932034,
+            0.0008263884810730815,
+            0.020103080198168755,
+            0.022393936291337013,
+            0.00212277565151453,
+            0.00032215146347880363,
+            0.003147850278764963,
+            0.005839189514517784,
+            -0.028270475566387177,
+            0.024850795045495033,
+            0.022775745019316673,
+            -0.011238468810915947,
+            0.015330466441810131,
+            0.00405464693903923,
+            0.0009856485994532704,
+            -0.01948886550962925,
+            0.0042829024605453014,
+            0.021530715748667717,
+            0.00335120502859354,
+            0.007291724439710379,
+            0.0015095985727384686,
+            0.0011630653170868754,
+            0.017397215589880943,
+            0.0006385965389199555,
+            0.004199900198727846,
+            0.0041832998394966125,
+            -0.020816897973418236,
+            0.00827114749699831,
+            -0.011080765165388584,
+            -0.0028594182804226875,
+            -0.007727484684437513,
+            0.0003626149264164269,
+            0.02699224464595318,
+            0.011213568039238453,
+            -0.014309542253613472,
+            0.014450645074248314,
+            0.011578776873648167,
+            -0.00803874246776104,
+            -0.012043587863445282,
+            -0.0007942252559587359,
+            -0.00632060132920742,
+            0.0041438741609454155,
+            0.017264412716031075,
+            -0.0013487822143360972,
+            0.009702932089567184,
+            0.014492146670818329,
+            -0.00175756704993546,
+            -0.012524999678134918,
+            0.01351272314786911,
+            0.011329771019518375,
+            0.013164115138351917,
+            -0.0073622758500278,
+            -0.0023655565455555916,
+            -0.011761381290853024,
+            -0.007424527313560247,
+            0.005627534352242947,
+            0.0022472788114100695,
+            -0.012466898187994957,
+            -0.007735785096883774,
+            -0.0016880529001355171,
+            0.0009555603610351682,
+            0.011728180572390556,
+            -0.007831237278878689,
+            0.0076652332209050655,
+            -0.003666612785309553,
+            0.012491798959672451,
+            0.000912503048311919,
+            0.04883836954832077,
+            0.005610933993011713,
+            0.003147850278764963,
+            0.0052083744667470455,
+            0.0030772986356168985,
+            0.01728101260960102,
+            -0.008026291616261005,
+            0.024120377376675606,
+            0.0076776836067438126,
+            -0.010599353350698948,
+            -0.026710039004683495,
+            -0.015762077644467354,
+            0.021364711225032806,
+            -0.004739413503557444,
+            -0.001546949497424066,
+            0.019389264285564423,
+            0.01574547588825226,
+            0.011761381290853024,
+            0.023705366998910904,
+            0.00614214688539505,
+            0.013039612211287022,
+            0.0011423148680478334,
+            0.02104930393397808,
+            0.004697912372648716,
+            -0.0027556659188121557,
+            0.016011083498597145,
+            0.009586729109287262,
+            0.019156858325004578,
+            -0.004531908314675093,
+            -0.00043420412112027407,
+            0.019920475780963898,
+            -0.0144008444622159,
+            -0.008665407076478004,
+            0.003533809445798397,
+            -0.009163418784737587,
+            -0.0018540568416938186,
+            0.0026124874129891396,
+            -0.022227931767702103,
+            -0.012018687091767788,
+            -0.007715034298598766,
+            0.00489296717569232,
+            0.008922713808715343,
+            0.0010593128390610218,
+            0.02672663889825344,
+            -0.011479174718260765,
+            0.011977186426520348,
+            0.00929622258991003,
+            0.010192643851041794,
+            -0.005370228551328182,
+            0.002778491470962763,
+            0.0021414512302726507,
+            -0.01762962155044079,
+            -0.00683106342330575,
+            -0.018227236345410347,
+            0.007407926954329014,
+            0.01169497985392809,
+            0.026876043528318405,
+            -0.01489055622369051,
+            -0.001508561079390347,
+            -0.011089065112173557,
+            0.010059840977191925,
+            0.014840754680335522,
+            0.005527932196855545
+          ],
+          "keyphrases": [
+            "Switch Transformers",
+            "Lower Compute Regimes"
+          ],
+          "title": "D Switch Transformers in Lower Compute Regimes"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "7de78a2d-a09d-472f-bb1d-34dcda9c4c28",
+      "type": "child",
+      "source": {
+        "id": "65742957-6bda-446d-a61a-0fe20313c760",
+        "properties": {
+          "page_content": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity\n\nWilliam Fedus∗ liamfedus@google.com Barret Zoph∗ barretzoph@google.com Noam Shazeer noam@google.com Google, Mountain View, CA 94043, USA Editor: Alexander Clark\n\nAbstract\n\nIn deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) models defy this and instead select different parameters for each incoming example. The result is a sparsely-activated model—with an outrageous number of parameters—but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs, and training instability. We address these with the introduction of the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques mitigate the instabilities, and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large (Raffel et al., 2019) to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the \"Colossal Clean Crawled Corpus\", and achieve a 4x speedup over the T5-XXL model.12 Keywords: mixture-of-experts, natural language processing, sparsity, large-scale machine learning, distributed computing arXiv:2101.03961v3 [cs.LG] 16 Jun 2022 Contents\n\n1 Introduction 3 2 Switch Transformer 4 2.1 Simplifying Sparse Routing 5 2.2 Efficient Sparse Routing 6 2.3 Putting It All Together: The Switch Transformer 8 2.4 Improved Training and Fine-Tuning Techniques 8 3 Scaling Properties 11 3.1 Scaling Results on a Step-Basis 12 3.2 Scaling Results on a Time-Basis 13 3.3 Scaling Versus a Larger Dense Model 13 4 Downstream Results 14 4.1 Fine-Tuning 14 4.2 Distillation 16 4.3 Multilingual Learning 17 5 Designing Models with Data, Model, and Expert-Parallelism 18 5.1 Data Parallelism 20 5.2 Model Parallelism 20 5.3 Model and Data Parallelism 21 5.4 Expert and Data Parallelism 22 5.5 Expert, Model and Data Parallelism 22 5.6 Towards Trillion Parameter Models 22 6 Related Work 24 7 Discussion 25 8 Future Work 26 9 Conclusion 27 A Switch for Attention 27 B Preventing Token Dropping with No-Token-Left-Behind 29\n\nC Encouraging Exploration Across Experts 29 D Switch Transformers in Lower Compute Regimes 29 E Relation of Upstream to Downstream Model Performance 32 F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/switch_transformers.md"
+          },
+          "summary": "The Switch Transformer is a sparsely-activated model that simplifies the Mixture of Experts (MoE) approach, allowing for efficient scaling to trillion-parameter models with reduced communication and computational costs. It achieves significant speedups in pre-training and fine-tuning by activating only a subset of neural network weights for each input, maintaining a constant computational cost. The model demonstrates superior scaling properties and improved performance across various natural language processing tasks, including multilingual settings. It also allows for effective distillation into smaller dense models, preserving a portion of the quality gains. The architecture is designed to maximize parameter count while keeping floating point operations per example constant, making it suitable for both large-scale and smaller-scale applications.",
+          "headlines": {
+            "1 Introduction": [],
+            "2 Switch Transformer": [
+              "2.1 Simplifying Sparse Routing",
+              "2.2 Efficient Sparse Routing",
+              "2.3 Putting It All Together: The Switch Transformer",
+              "2.4 Improved Training and Fine-Tuning Techniques"
+            ],
+            "3 Scaling Properties": [
+              "3.1 Scaling Results on a Step-Basis",
+              "3.2 Scaling Results on a Time-Basis",
+              "3.3 Scaling Versus a Larger Dense Model"
+            ],
+            "4 Downstream Results": [
+              "4.1 Fine-Tuning",
+              "4.2 Distillation",
+              "4.3 Multilingual Learning"
+            ],
+            "5 Designing Models with Data, Model, and Expert-Parallelism": [
+              "5.1 Data Parallelism",
+              "5.2 Model Parallelism",
+              "5.3 Model and Data Parallelism",
+              "5.4 Expert and Data Parallelism",
+              "5.5 Expert, Model and Data Parallelism",
+              "5.6 Towards Trillion Parameter Models"
+            ],
+            "6 Related Work": [],
+            "7 Discussion": [],
+            "8 Future Work": [],
+            "9 Conclusion": [],
+            "A Switch for Attention": [],
+            "B Preventing Token Dropping with No-Token-Left-Behind": [],
+            "C Encouraging Exploration Across Experts": [],
+            "D Switch Transformers in Lower Compute Regimes": [],
+            "E Relation of Upstream to Downstream Model Performance": [],
+            "F Pseudo Code for Switch Transformers": []
+          },
+          "summary_embedding": [
+            -0.0060447100549936295,
+            -0.00390331051312387,
+            -0.01784370467066765,
+            -0.0021820589900016785,
+            0.013227885589003563,
+            0.011338187381625175,
+            0.0165580902248621,
+            0.024240797385573387,
+            -0.005986624862998724,
+            0.03596621751785278,
+            -0.004852031823247671,
+            -0.0068811336532235146,
+            -0.003574162255972624,
+            0.014242436736822128,
+            0.011578272096812725,
+            -0.033302053809165955,
+            -0.024519605562090874,
+            0.01050176378339529,
+            0.016093410551548004,
+            -0.006795942317694426,
+            0.007473600562661886,
+            -0.030513975769281387,
+            -0.06124480068683624,
+            -0.006838538218289614,
+            -0.0026099516544491053,
+            0.01029265858232975,
+            0.0010329445358365774,
+            -0.010873507708311081,
+            -0.053469158709049225,
+            0.023311438038945198,
+            0.025294071063399315,
+            0.005092116538435221,
+            -0.029584616422653198,
+            0.0009361362317577004,
+            -0.05161044001579285,
+            -0.04448534920811653,
+            0.0379488542675972,
+            0.012561844661831856,
+            -0.04364892467856407,
+            0.039249956607818604,
+            -0.004875265993177891,
+            -0.0022440163884311914,
+            -0.021499186754226685,
+            -0.0034909071400761604,
+            -0.007039899472147226,
+            -0.009851212613284588,
+            -0.02874819189310074,
+            0.0014037203509360552,
+            0.021886419504880905,
+            -0.0056962003000080585,
+            0.023264970630407333,
+            -0.024271776899695396,
+            0.01936165988445282,
+            0.04240977764129639,
+            -0.06957805901765823,
+            0.024659009650349617,
+            0.017549408599734306,
+            0.05120771750807762,
+            0.06511713564395905,
+            0.018339363858103752,
+            0.021669568493962288,
+            -0.014141756109893322,
+            -0.017766257748007774,
+            -0.011872569099068642,
+            0.004673904739320278,
+            -0.014970434829592705,
+            -0.01891246810555458,
+            0.008449427783489227,
+            -0.020461400970816612,
+            0.008209343999624252,
+            -0.05576157942414284,
+            -0.014730350114405155,
+            -0.034200433641672134,
+            -0.01140014547854662,
+            0.017642343416810036,
+            -0.018215449526906013,
+            0.030126741155982018,
+            0.02997184917330742,
+            0.002638994250446558,
+            0.0007822110201232135,
+            0.021049996837973595,
+            0.006509389728307724,
+            -0.0053438181057572365,
+            0.012321759946644306,
+            0.028159597888588905,
+            -0.01450575515627861,
+            -0.010106786154210567,
+            -0.03454120084643364,
+            0.01861817203462124,
+            0.01589204929769039,
+            -0.04194509983062744,
+            -0.012569589540362358,
+            -0.013824224472045898,
+            0.032341714948415756,
+            -0.009549170732498169,
+            0.04098476096987724,
+            -0.010083552449941635,
+            0.001383390510454774,
+            -0.031149037182331085,
+            -0.009463978931307793,
+            0.017069239169359207,
+            0.006172496825456619,
+            -0.016589069738984108,
+            -0.0012846461031585932,
+            0.016170857474207878,
+            0.006617815233767033,
+            -0.030142230913043022,
+            0.04488807171583176,
+            0.023497309535741806,
+            0.055296897888183594,
+            0.05579255893826485,
+            -0.037391237914562225,
+            -0.002445377642288804,
+            0.0344172865152359,
+            -0.0228312686085701,
+            -0.015040136873722076,
+            -0.034014564007520676,
+            -0.01768881268799305,
+            0.035935238003730774,
+            -0.0318460576236248,
+            0.01556677371263504,
+            0.028066661208868027,
+            -0.025712283328175545,
+            0.003488970920443535,
+            -0.03943582624197006,
+            0.003622566582635045,
+            0.01220559049397707,
+            0.006095050368458033,
+            -0.0026293133851140738,
+            -0.013615119270980358,
+            0.008906363509595394,
+            0.003055269829928875,
+            0.00800798274576664,
+            -0.04615819454193115,
+            0.018478767946362495,
+            -0.00454224506393075,
+            0.04033420979976654,
+            0.042502716183662415,
+            -0.02730768360197544,
+            -0.016743963584303856,
+            0.008557853288948536,
+            0.012104909867048264,
+            0.04188314080238342,
+            -0.007287728600203991,
+            -0.00121784838847816,
+            -0.0007836631848476827,
+            0.007322579622268677,
+            0.047552235424518585,
+            0.004646798130124807,
+            0.0020697612781077623,
+            -0.027230238541960716,
+            0.014978179708123207,
+            -0.019733402878046036,
+            0.027648448944091797,
+            0.023373395204544067,
+            0.02942972257733345,
+            -0.04048910364508629,
+            0.017518429085612297,
+            -0.011818356812000275,
+            -0.005568413063883781,
+            -0.002726121572777629,
+            -0.02856232039630413,
+            -0.0074426219798624516,
+            0.033983584493398666,
+            0.015365412458777428,
+            0.012956822291016579,
+            0.03271345794200897,
+            0.0012052633101120591,
+            -0.042502716183662415,
+            0.03804178908467293,
+            0.024550583213567734,
+            0.04640602692961693,
+            0.023466331884264946,
+            0.010796061716973782,
+            0.006571347359567881,
+            0.013375034555792809,
+            -0.019377149641513824,
+            0.015907539054751396,
+            0.02560385875403881,
+            0.009750531986355782,
+            -0.03167567402124405,
+            -0.0009501734166406095,
+            0.01178737822920084,
+            -0.004546117503196001,
+            0.03701949119567871,
+            0.04104671627283096,
+            0.020569827407598495,
+            0.05030933395028114,
+            0.0020155487582087517,
+            0.017936641350388527,
+            0.005936284549534321,
+            0.02820606529712677,
+            0.030467506498098373,
+            -0.005285732913762331,
+            0.005076627247035503,
+            -0.01136916596442461,
+            -0.03385967016220093,
+            0.014110777527093887,
+            0.01561324205249548,
+            -0.008077683858573437,
+            -0.01697630248963833,
+            0.006033093202859163,
+            0.009169681929051876,
+            -0.01795213110744953,
+            -0.017456471920013428,
+            0.005971135571599007,
+            0.018866000697016716,
+            -0.013576395809650421,
+            0.024689987301826477,
+            -0.04442339017987251,
+            0.029119934886693954,
+            -0.033209118992090225,
+            -0.003384418087080121,
+            -0.010641167871654034,
+            -0.0030843124259263277,
+            0.0041666291654109955,
+            0.011314953677356243,
+            0.013406013138592243,
+            0.002042655134573579,
+            -0.013909416273236275,
+            0.024519605562090874,
+            0.018447788432240486,
+            0.012073930352926254,
+            -0.03148980438709259,
+            -0.014397329650819302,
+            -0.0015673263696953654,
+            -0.008743725717067719,
+            -0.034138478338718414,
+            0.026966920122504234,
+            0.014048820361495018,
+            -0.03844451159238815,
+            0.017797237262129784,
+            0.029538147151470184,
+            0.014420564286410809,
+            -0.004790074657648802,
+            -0.0463130883872509,
+            0.04045812413096428,
+            -0.030436528846621513,
+            -0.033333033323287964,
+            -0.010478530079126358,
+            -0.0003146269591525197,
+            0.02894955314695835,
+            -0.0064358157105743885,
+            -0.050835974514484406,
+            0.01184933539479971,
+            0.020631784573197365,
+            0.00010763872705865651,
+            0.005731051322072744,
+            -0.019593998789787292,
+            0.00028631053282879293,
+            -0.03395260497927666,
+            0.0090844901278615,
+            0.026750069111585617,
+            0.008867640048265457,
+            -0.016418687999248505,
+            0.03351890295743942,
+            -0.0004121613164898008,
+            -0.018695618957281113,
+            -0.003889757441356778,
+            0.006885006092488766,
+            0.034912943840026855,
+            0.004836542531847954,
+            -0.02946070022881031,
+            0.05665995925664902,
+            -0.02100352756679058,
+            0.021344292908906937,
+            0.05631919577717781,
+            -0.019439106807112694,
+            -0.008472662419080734,
+            -0.002065889071673155,
+            0.0026196325197815895,
+            -0.0260530486702919,
+            -0.008937342092394829,
+            0.032341714948415756,
+            0.041728246957063675,
+            -0.018742086365818977,
+            -0.001398879918269813,
+            0.006780453026294708,
+            0.009425255469977856,
+            0.004948840010911226,
+            0.02692045085132122,
+            -0.012128143571317196,
+            -0.014559967443346977,
+            0.05198218300938606,
+            0.028314489871263504,
+            -0.012081675231456757,
+            -0.0164031982421875,
+            0.019888296723365784,
+            -0.024844881147146225,
+            -0.020244551822543144,
+            -0.02431824430823326,
+            -0.020414933562278748,
+            -0.0042518205009400845,
+            0.009525936096906662,
+            0.037484172731637955,
+            -0.03968365862965584,
+            0.012747716158628464,
+            0.021700548008084297,
+            0.02981695532798767,
+            -0.019532041624188423,
+            -0.05910727381706238,
+            -0.0016128262504935265,
+            0.03296129032969475,
+            -0.015458349138498306,
+            -0.049379974603652954,
+            0.015164051204919815,
+            0.0260530486702919,
+            0.05957195535302162,
+            -0.001461805310100317,
+            -0.014954946003854275,
+            0.061926331371068954,
+            0.03026614524424076,
+            -0.004383479710668325,
+            -0.009494957514107227,
+            -0.0127864396199584,
+            0.003396035172045231,
+            0.00465454300865531,
+            -0.03364282101392746,
+            -0.041170634329319,
+            -0.011841590516269207,
+            -0.010284913703799248,
+            0.008108663372695446,
+            -0.040922801941633224,
+            0.03531566634774208,
+            -0.021917399019002914,
+            0.010950954630970955,
+            0.015473837964236736,
+            0.037391237914562225,
+            -0.021344292908906937,
+            -0.01284839678555727,
+            -0.01572941243648529,
+            -0.024968795478343964,
+            -0.014010096900165081,
+            0.021917399019002914,
+            0.007682706229388714,
+            3.4699722164077684e-05,
+            0.010075807571411133,
+            -0.024659009650349617,
+            -0.051052823662757874,
+            0.0291044469922781,
+            0.0015605497173964977,
+            -0.015938518568873405,
+            -0.004623564425855875,
+            -0.01977987214922905,
+            -0.01122201792895794,
+            -0.012608313001692295,
+            -0.039249956607818604,
+            0.021111954003572464,
+            -0.01135367713868618,
+            0.015876561403274536,
+            0.0009250032599084079,
+            0.003659353591501713,
+            0.028624277561903,
+            0.01116006076335907,
+            0.0051811798475682735,
+            -0.03255856782197952,
+            -0.018277406692504883,
+            0.010122275911271572,
+            -0.00561100896447897,
+            -0.018339363858103752,
+            -0.013351799920201302,
+            -0.05037129297852516,
+            -0.03305422514677048,
+            -0.008054450154304504,
+            -0.05415068939328194,
+            0.0020697612781077623,
+            -0.014296649023890495,
+            -0.02232012152671814,
+            -0.005990497302263975,
+            0.03107159025967121,
+            -0.003235333366319537,
+            0.025294071063399315,
+            0.0076594725251197815,
+            0.037360258400440216,
+            0.040117356926202774,
+            0.008426194079220295,
+            0.03065337985754013,
+            -0.0029991210903972387,
+            -0.016867877915501595,
+            0.05486319959163666,
+            -0.01881953328847885,
+            0.013529927469789982,
+            -0.008604321628808975,
+            0.008604321628808975,
+            0.03299226611852646,
+            -0.003794885240495205,
+            0.02007416822016239,
+            -0.0311955064535141,
+            -0.07998688519001007,
+            0.01637221872806549,
+            0.028159597888588905,
+            -0.06418777257204056,
+            0.014691627584397793,
+            -0.020275529474020004,
+            0.02052335813641548,
+            0.02411688305437565,
+            -0.0009114501299336553,
+            -0.014064309187233448,
+            0.005595519673079252,
+            0.0020833145827054977,
+            -0.01456771232187748,
+            -0.002691270550712943,
+            -0.009417511522769928,
+            0.008774704299867153,
+            -0.017007282003760338,
+            -0.013406013138592243,
+            -0.034231413155794144,
+            0.010873507708311081,
+            0.01233724970370531,
+            -0.003969140350818634,
+            0.0011500825639814138,
+            -0.03277541697025299,
+            -0.007876322604715824,
+            -0.02145271934568882,
+            0.03382869064807892,
+            0.012058441527187824,
+            -0.030699847266077995,
+            -0.007814365439116955,
+            0.015264731831848621,
+            -0.007876322604715824,
+            0.0008906363509595394,
+            0.024132372811436653,
+            0.02103450708091259,
+            0.0028364830650389194,
+            -0.0009700191440060735,
+            0.02438020147383213,
+            0.007427132688462734,
+            -0.01720864325761795,
+            0.028794659301638603,
+            0.014288905076682568,
+            0.060160547494888306,
+            -0.03271345794200897,
+            0.001297231181524694,
+            -0.04135650396347046,
+            -0.031118059530854225,
+            0.006408709101378918,
+            0.014350862242281437,
+            0.007597515359520912,
+            -0.028252532705664635,
+            0.005978880450129509,
+            0.02004319056868553,
+            0.01469937153160572,
+            0.009704063646495342,
+            0.0037580979987978935,
+            -0.036740683019161224,
+            -0.000691695255227387,
+            -0.011214273050427437,
+            -0.0038781403563916683,
+            0.035656433552503586,
+            -0.008186109364032745,
+            -0.05151750147342682,
+            -0.0006418389966711402,
+            -0.03255856782197952,
+            -0.0353776253759861,
+            0.021313315257430077,
+            -0.00839521549642086,
+            -0.0039207362569868565,
+            -0.013986863195896149,
+            -0.000537770101800561,
+            -0.010524998418986797,
+            0.027509044855833054,
+            0.028345469385385513,
+            0.022490503266453743,
+            -0.05384090170264244,
+            0.009479468688368797,
+            0.00023766436788719147,
+            -0.05817791447043419,
+            0.0009709871956147254,
+            -0.007117345929145813,
+            0.011191039346158504,
+            -0.020213572308421135,
+            -0.02414786070585251,
+            0.011500826105475426,
+            0.03392162546515465,
+            0.014304393902420998,
+            0.0321558453142643,
+            -0.013367289677262306,
+            -0.09250226616859436,
+            0.018292896449565887,
+            -0.008689512498676777,
+            0.04126356914639473,
+            0.00865078903734684,
+            -0.004278926644474268,
+            -0.015845581889152527,
+            -0.00865078903734684,
+            -0.006943091284483671,
+            0.01283290795981884,
+            0.004530628211796284,
+            -0.005626498255878687,
+            0.0016631665639579296,
+            0.013708055019378662,
+            -0.005967263597995043,
+            0.007481344975531101,
+            0.003651608945801854,
+            -0.013189162127673626,
+            -0.06059424951672554,
+            -0.04055105894804001,
+            -0.018540725111961365,
+            0.050835974514484406,
+            -0.0069159846752882,
+            -0.046684831380844116,
+            -0.010850274004042149,
+            0.05058814212679863,
+            0.012259802781045437,
+            -0.0334569476544857,
+            -0.0007841471815481782,
+            0.0050882440991699696,
+            0.015458349138498306,
+            0.004546117503196001,
+            -0.027292195707559586,
+            0.02820606529712677,
+            0.0008412640891037881,
+            0.02300165221095085,
+            -0.018339363858103752,
+            -0.005312839522957802,
+            0.035780347883701324,
+            -0.024844881147146225,
+            0.023791607469320297,
+            0.0055219451896846294,
+            0.031629208475351334,
+            0.002952652983367443,
+            0.010060317814350128,
+            0.004704883322119713,
+            0.0186491496860981,
+            0.004275054205209017,
+            0.0276639387011528,
+            0.011462102644145489,
+            0.006428070832043886,
+            0.021623101085424423,
+            -0.009378788061439991,
+            -0.014513500034809113,
+            0.01885051093995571,
+            -0.0014695499558001757,
+            0.025232113897800446,
+            -0.020538847893476486,
+            0.002726121572777629,
+            -0.0016883367206901312,
+            -0.017967619001865387,
+            -0.040922801941633224,
+            -0.0007405834621749818,
+            0.021747015416622162,
+            -0.011671207845211029,
+            -0.0228312686085701,
+            -0.013715799897909164,
+            -0.01585332676768303,
+            0.0038490979932248592,
+            0.034169457852840424,
+            -0.0398695282638073,
+            0.00963436160236597,
+            -0.000830615172162652,
+            -0.03125746175646782,
+            -0.0239619892090559,
+            -0.01093546487390995,
+            0.014614180661737919,
+            -0.01652711257338524,
+            -0.021173911169171333,
+            -0.028469383716583252,
+            -0.008410705253481865,
+            0.006048582494258881,
+            -0.05065010115504265,
+            -0.004054331686347723,
+            0.019965743646025658,
+            0.0276639387011528,
+            -0.003099801717326045,
+            -0.02177799493074417,
+            0.0012468908680602908,
+            0.022351099178195,
+            0.02936776541173458,
+            0.014226946979761124,
+            0.027586491778492928,
+            -0.002178186783567071,
+            -0.0011675080750137568,
+            -0.0073148347437381744,
+            0.01307299267500639,
+            0.0011287847300991416,
+            0.036616768687963486,
+            0.0122210793197155,
+            -0.020306508988142014,
+            0.0013959755888208747,
+            -0.006431943271309137,
+            0.029181893914937973,
+            -0.02270735427737236,
+            -9.832092473516241e-05,
+            0.03531566634774208,
+            0.027741385623812675,
+            0.004933350719511509,
+            -0.0143121387809515,
+            -0.05142456665635109,
+            0.021592123433947563,
+            -0.01765783317387104,
+            0.031025122851133347,
+            0.016666516661643982,
+            0.003397971158847213,
+            -0.0034212053287774324,
+            0.012368228286504745,
+            -0.022087780758738518,
+            -0.004406713880598545,
+            -0.021142931655049324,
+            0.04389675334095955,
+            -0.043184246867895126,
+            0.008472662419080734,
+            0.029445212334394455,
+            0.011926782317459583,
+            -0.019082851707935333,
+            -0.005425137002021074,
+            0.029212871566414833,
+            -0.00162637943867594,
+            0.01308848150074482,
+            0.0058665829710662365,
+            -0.026935940608382225,
+            0.01202746294438839,
+            -0.031660184264183044,
+            0.02177799493074417,
+            -0.013436991721391678,
+            0.018447788432240486,
+            0.03342596814036369,
+            -0.0030397805385291576,
+            -0.0038239278364926577,
+            -0.016263794153928757,
+            -0.00969631876796484,
+            0.0662323608994484,
+            0.01054823212325573,
+            0.03726732358336449,
+            0.0018974426202476025,
+            -0.009773765690624714,
+            0.007969259284436703,
+            0.0063041565008461475,
+            -0.012329504825174809,
+            0.010494019836187363,
+            0.024178840219974518,
+            0.0012730291346088052,
+            -0.002412462839856744,
+            -0.019408127292990685,
+            -0.03438630700111389,
+            0.003727119415998459,
+            -0.030932186171412468,
+            0.022691864520311356,
+            0.009208405390381813,
+            0.017456471920013428,
+            0.008240322582423687,
+            -0.015907539054751396,
+            0.03200095146894455,
+            0.05461536720395088,
+            0.0019768255297094584,
+            0.013375034555792809,
+            0.007938280701637268,
+            0.007036027032881975,
+            -0.011338187381625175,
+            0.04702559858560562,
+            0.000662652775645256,
+            0.005169562995433807,
+            0.022536970674991608,
+            0.02145271934568882,
+            0.022939693182706833,
+            0.002304037567228079,
+            -0.011694442480802536,
+            0.010478530079126358,
+            -0.0006302220281213522,
+            0.004445436876267195,
+            -0.010695381090044975,
+            -0.027865299955010414,
+            0.007492962293326855,
+            0.01302652433514595,
+            0.022598929703235626,
+            0.021592123433947563,
+            -0.03178409859538078,
+            -0.01855621486902237,
+            0.03364282101392746,
+            -0.02997184917330742,
+            0.03612111136317253,
+            -0.0014424435794353485,
+            -0.023807097226381302,
+            -0.0058356039226055145,
+            0.016929835081100464,
+            0.02656419761478901,
+            -0.029181893914937973,
+            0.01804506592452526,
+            0.029693040996789932,
+            -0.0074426219798624516,
+            0.0005939188995398581,
+            0.012112653814256191,
+            0.02447313815355301,
+            -0.009293596260249615,
+            -0.0024027819745242596,
+            -0.0013320822035893798,
+            -0.011926782317459583,
+            -0.018370341509580612,
+            -0.0044725434854626656,
+            -0.022629907354712486,
+            0.005564541090279818,
+            -0.007365175057202578,
+            -0.003481226274743676,
+            -0.008581086993217468,
+            0.01154729351401329,
+            0.001979729626327753,
+            -0.019857317209243774,
+            -0.022335609421133995,
+            0.01652711257338524,
+            0.03887821361422539,
+            -0.018215449526906013,
+            -0.02106548473238945,
+            0.024442158639431,
+            -0.01759587600827217,
+            -0.01601596549153328,
+            0.036678727716207504,
+            0.013739033602178097,
+            0.007736918982118368,
+            -0.03358086198568344,
+            -0.030049296095967293,
+            -0.03274443745613098,
+            0.006296411622315645,
+            0.03785591572523117,
+            -0.0028752065263688564,
+            0.02569679357111454,
+            0.03701949119567871,
+            0.0057891360484063625,
+            -0.028934063389897346,
+            -0.014823286794126034,
+            -0.045104920864105225,
+            0.060160547494888306,
+            0.0011481463443487883,
+            0.05161044001579285,
+            0.016294771805405617,
+            -0.01456771232187748,
+            0.00419760774821043,
+            -0.013251119293272495,
+            0.026192452758550644,
+            0.02280029095709324,
+            0.025929134339094162,
+            0.006582964211702347,
+            0.022505993023514748,
+            -0.0037522895727306604,
+            -0.012197845615446568,
+            0.007307090330868959,
+            0.014536733739078045,
+            0.0002497654058970511,
+            -0.011392400600016117,
+            0.015783624723553658,
+            -0.01307299267500639,
+            -0.011361422017216682,
+            0.023295948281884193,
+            0.01368482131510973,
+            -0.012344993650913239,
+            0.014064309187233448,
+            -0.008565598167479038,
+            0.01239920686930418,
+            -0.023915521800518036,
+            0.022691864520311356,
+            -0.03262052312493324,
+            0.009587894193828106,
+            0.0015053689712658525,
+            -0.0073148347437381744,
+            0.024550583213567734,
+            -0.0027454833034425974,
+            -0.05071205645799637,
+            0.03550153970718384,
+            -0.008015726692974567,
+            0.0037213109899312258,
+            0.006381602957844734,
+            0.011392400600016117,
+            -0.0012623801594600081,
+            0.03807276859879494,
+            0.01239920686930418,
+            0.015969496220350266,
+            0.02199484594166279,
+            0.011911292560398579,
+            -0.008433938957750797,
+            -0.004096927121281624,
+            -0.0016137943603098392,
+            -0.01784370467066765,
+            0.025944624096155167,
+            0.0018674320308491588,
+            0.03962169960141182,
+            -0.018168980255723,
+            -0.02300165221095085,
+            0.028066661208868027,
+            0.007597515359520912,
+            -0.02351279929280281,
+            -0.0017377089243382215,
+            0.01202746294438839,
+            -0.009185170754790306,
+            0.027385130524635315,
+            0.01196550577878952,
+            0.019113829359412193,
+            -0.018060555681586266,
+            -0.004340883810073137,
+            -0.005072754807770252,
+            -0.015938518568873405,
+            0.02238207869231701,
+            0.020771188661456108,
+            0.011152315884828568,
+            -0.011880313977599144,
+            -0.01307299267500639,
+            0.020600805059075356,
+            0.035036858171224594,
+            -0.0002744515077210963,
+            0.005672966130077839,
+            0.017487451434135437,
+            0.002373739378526807,
+            -0.04470219835639,
+            0.016232814639806747,
+            -0.014025585725903511,
+            0.011214273050427437,
+            0.019593998789787292,
+            -0.021917399019002914,
+            -0.03196997195482254,
+            -0.011539548635482788,
+            -0.0414804182946682,
+            0.00493722315877676,
+            -0.0069159846752882,
+            0.013576395809650421,
+            -0.010176488198339939,
+            -0.0014831031439825892,
+            0.016666516661643982,
+            0.0038103745318949223,
+            0.02177799493074417,
+            -0.011678952723741531,
+            -0.012073930352926254,
+            -0.020213572308421135,
+            -0.020786678418517113,
+            0.004278926644474268,
+            -0.030405549332499504,
+            -0.004488032776862383,
+            0.021375272423028946,
+            -0.04783104360103607,
+            -0.03689557686448097,
+            0.02898053266108036,
+            -0.008503641001880169,
+            0.020755698904395103,
+            0.022955182939767838,
+            -0.015094350092113018,
+            -0.0008717587334103882,
+            -0.018695618957281113,
+            0.02258343994617462,
+            -0.007837600074708462,
+            0.026300877332687378,
+            -0.0027551641687750816,
+            0.0006292539183050394,
+            -0.02659517526626587,
+            0.020306508988142014,
+            0.005227648187428713,
+            0.012275291606783867,
+            -0.02390003204345703,
+            0.006815304048359394,
+            -0.03133491054177284,
+            -0.01784370467066765,
+            0.0180140882730484,
+            -0.011152315884828568,
+            0.027989214286208153,
+            -0.006060199346393347,
+            0.028438406065106392,
+            0.022722844034433365,
+            0.024751944467425346,
+            0.02676555886864662,
+            -0.02363671362400055,
+            0.008495896123349667,
+            0.03726732358336449,
+            0.00865078903734684,
+            -0.015876561403274536,
+            0.008170620538294315,
+            -0.04863648861646652,
+            -0.008062195032835007,
+            0.05030933395028114,
+            0.002127846470102668,
+            0.023141054436564445,
+            0.016294771805405617,
+            -0.002873270306736231,
+            -0.005688455421477556,
+            0.010269423946738243,
+            -0.04296739399433136,
+            -0.023172033950686455,
+            0.022629907354712486,
+            -0.021561143919825554,
+            6.54212328754511e-07,
+            0.03122648410499096,
+            0.012120398692786694,
+            -0.0009109660750254989,
+            -0.05108380317687988,
+            0.02100352756679058,
+            0.01178737822920084,
+            0.014443797990679741,
+            -0.010300402529537678,
+            0.026068538427352905,
+            0.01733255758881569,
+            0.038413532078266144,
+            0.012755461037158966,
+            -0.002679653698578477,
+            -0.03782493621110916,
+            0.009177426807582378,
+            -0.021685058251023293,
+            0.0011471783509477973,
+            0.0074503663927316666,
+            -0.02273833192884922,
+            -0.024782923981547356,
+            0.0043912241235375404,
+            0.004588713403791189,
+            0.011880313977599144,
+            0.022087780758738518,
+            0.009417511522769928,
+            -0.021204888820648193,
+            0.015404135920107365,
+            -0.05446047708392143,
+            -0.015450604259967804,
+            -0.023884542286396027,
+            0.02566581591963768,
+            0.020012211054563522,
+            0.010184233076870441,
+            0.005545179359614849,
+            0.010904486291110516,
+            -0.020988039672374725,
+            -0.009239383973181248,
+            0.020631784573197365,
+            0.011074868962168694,
+            -0.012840652838349342,
+            0.04113965481519699,
+            -0.00011126903700642288,
+            0.031210994347929955,
+            -0.02280029095709324,
+            0.002941036131232977,
+            -0.001241082325577736,
+            -0.03649285435676575,
+            0.009130958467721939,
+            0.01981084980070591,
+            0.02151467651128769,
+            -0.01861817203462124,
+            0.0024143988266587257,
+            0.01048627495765686,
+            0.004766840487718582,
+            0.038351576775312424,
+            0.027121812105178833,
+            -0.044206541031599045,
+            0.011663463898003101,
+            0.012995545752346516,
+            -0.009363298304378986,
+            0.048419639468193054,
+            -0.017518429085612297,
+            0.017673322930932045,
+            0.008240322582423687,
+            -0.01798310875892639,
+            0.02318752370774746,
+            -0.01469937153160572,
+            -0.013282098807394505,
+            0.00650551775470376,
+            0.009192915633320808,
+            -0.0001770381786627695,
+            -0.037577107548713684,
+            -0.017859194427728653,
+            -0.005719434004276991,
+            0.0038297362625598907,
+            0.01562873087823391,
+            0.013498948886990547,
+            0.009967382065951824,
+            0.005161818582564592,
+            -0.00945623405277729,
+            -0.0016089539276435971,
+            0.008124152198433876,
+            -0.0005532593932002783,
+            0.0019216446671634912,
+            -0.01926872320473194,
+            0.02698240801692009,
+            0.02572777308523655,
+            -0.0018354853382334113,
+            -0.04652994126081467,
+            -0.0036709706764668226,
+            0.0037077576853334904,
+            -0.017735280096530914,
+            -0.0035315665882080793,
+            -0.04519785940647125,
+            -0.01810702309012413,
+            0.0143121387809515,
+            -0.014614180661737919,
+            -0.0460033044219017,
+            0.009076746180653572,
+            -0.013886182568967342,
+            0.0063970922492444515,
+            -0.00932457484304905,
+            0.005762029904872179,
+            0.003651608945801854,
+            0.005680711008608341,
+            -0.005909178406000137,
+            0.05108380317687988,
+            -0.03466511517763138,
+            0.028051171451807022,
+            0.008565598167479038,
+            -0.012437930330634117,
+            -0.02261441759765148,
+            0.00887538492679596,
+            -0.044981006532907486,
+            -0.033209118992090225,
+            0.016743963584303856,
+            -0.01781272701919079,
+            0.011074868962168694,
+            0.06161654368042946,
+            0.007330324035137892,
+            -0.023342415690422058,
+            -0.01233724970370531,
+            -0.021049996837973595,
+            -0.04191412031650543,
+            0.003442503046244383,
+            0.005463860463351011,
+            -0.004948840010911226,
+            -0.03807276859879494,
+            -0.021716037765145302,
+            0.04615819454193115,
+            -0.012004229240119457,
+            0.012422440573573112,
+            0.01080380566418171,
+            -0.0019410063978284597,
+            -0.04860550910234451,
+            -0.021019017323851585,
+            0.0395597442984581,
+            0.021886419504880905,
+            -0.008426194079220295,
+            0.0028364830650389194,
+            -0.018277406692504883,
+            -0.003289545886218548,
+            0.006296411622315645,
+            -0.009107724763453007,
+            -0.03562545403838158,
+            0.02067825198173523,
+            0.0054832217283546925,
+            -0.023373395204544067,
+            -0.025805220007896423,
+            -0.03296129032969475,
+            0.009944148361682892,
+            0.02453509531915188,
+            -0.01968693546950817,
+            0.005823987070471048,
+            0.00027203132049180567,
+            -0.000881923595443368,
+            0.00042740863864310086,
+            -0.009959637187421322,
+            -0.019764382392168045,
+            -0.020151615142822266,
+            0.03268248215317726,
+            -0.0060059865936636925,
+            0.0074387495405972,
+            -0.03630698472261429,
+            -0.010974188335239887,
+            0.011322698555886745,
+            -0.002395037328824401,
+            -0.013452480547130108,
+            -0.009092235006392002,
+            -0.01739451475441456,
+            -0.009882191196084023,
+            -0.007876322604715824,
+            0.022660886868834496,
+            0.006168624851852655,
+            0.005409647710621357,
+            0.0023795480374246836,
+            -0.008581086993217468,
+            0.02898053266108036,
+            0.017874684184789658,
+            -0.014188224449753761,
+            -0.00920066051185131,
+            0.03451022133231163,
+            0.02665713243186474,
+            -0.009115469641983509,
+            0.01798310875892639,
+            0.023264970630407333,
+            -0.0010484338272362947,
+            -0.01196550577878952,
+            0.001922612776979804,
+            -0.0009550138493068516,
+            -0.0032972905319184065,
+            0.00845717266201973,
+            -0.015365412458777428,
+            0.0315207801759243,
+            0.04287445917725563,
+            -0.002145271748304367,
+            0.009587894193828106,
+            -0.003727119415998459,
+            -0.008472662419080734,
+            0.017069239169359207,
+            0.009316830895841122,
+            0.04625113308429718,
+            0.01098193321377039,
+            0.005680711008608341,
+            -0.033983584493398666,
+            -0.001229465357027948,
+            -0.01184933539479971,
+            0.03308520466089249,
+            0.02061629481613636,
+            0.00889087375253439,
+            -0.008379725739359856,
+            0.01344473659992218,
+            0.008341003209352493,
+            0.00661007035523653,
+            0.020569827407598495,
+            -0.016542602330446243,
+            -0.017286090180277824,
+            0.030467506498098373,
+            -0.00672236829996109,
+            0.01289486512541771,
+            0.02029101923108101,
+            -0.002089123008772731,
+            0.010842529125511646,
+            0.07100307941436768,
+            -0.006184114143252373,
+            0.015713922679424286,
+            -0.019082851707935333,
+            0.0015789433382451534,
+            -0.007640110794454813,
+            -0.03370477631688118,
+            0.018246427178382874,
+            0.0039962464943528175,
+            0.00865078903734684,
+            0.01198099460452795,
+            0.009053511545062065,
+            -0.007946024648845196,
+            -0.00963436160236597,
+            0.015195030719041824,
+            0.0019080914789810777,
+            0.00033568276558071375,
+            0.001336922636255622,
+            -0.0065210070461034775,
+            -0.004011735785752535,
+            -0.021948376670479774,
+            0.02763296104967594,
+            0.007140580099076033,
+            0.030606910586357117,
+            -0.055296897888183594,
+            0.030095763504505157,
+            -0.00789568480104208,
+            0.02554190158843994,
+            -0.016201836988329887,
+            -0.011090358719229698,
+            0.001609921921044588,
+            0.018153492361307144,
+            0.004491904750466347,
+            0.006462921854108572,
+            0.011888058856129646,
+            0.023233991116285324,
+            -0.03444826602935791,
+            0.017053749412298203,
+            0.023110076785087585,
+            0.0048675211146473885,
+            0.0003535923024173826,
+            0.005165690556168556,
+            0.015458349138498306,
+            0.007198664825409651,
+            -0.023048119619488716,
+            -0.0025692922063171864,
+            0.017611365765333176,
+            -0.02306360937654972,
+            0.0036980770528316498,
+            -0.006923729553818703,
+            0.004623564425855875,
+            -0.0014705180656164885,
+            -0.03509881719946861,
+            -0.00100099784322083,
+            -0.025743262842297554,
+            0.01923774555325508,
+            0.009556914679706097,
+            -0.025867177173495293,
+            0.003932353109121323,
+            0.025650326162576675,
+            -0.0003695656778290868,
+            0.00821708794683218,
+            0.0004683101433329284,
+            0.00869725737720728,
+            -0.00974278710782528,
+            -0.01493945624679327,
+            0.004801691509783268,
+            0.01531120017170906,
+            -0.01493945624679327,
+            -0.007431004662066698,
+            -0.009510447271168232,
+            0.030560443177819252,
+            0.0006674932083114982,
+            0.004189862869679928,
+            -0.005045648664236069,
+            -0.01858719252049923,
+            -0.0032237162813544273,
+            -0.021142931655049324,
+            0.004367990419268608,
+            -0.0009046735358424485,
+            0.04197607934474945,
+            0.0038161829579621553,
+            -0.02029101923108101,
+            0.0032972905319184065,
+            -0.03136589005589485,
+            0.0005000148667022586,
+            -0.007276111748069525,
+            -0.02004319056868553,
+            0.0015005286550149322,
+            0.0016525175888091326,
+            0.012631546705961227,
+            -0.03255856782197952,
+            -0.009301341138780117,
+            0.018416810780763626,
+            -0.00020680672605521977,
+            -0.010687636211514473,
+            -0.017828216776251793,
+            0.00035189816844649613,
+            -0.011338187381625175,
+            -0.027973726391792297,
+            -0.02850036323070526,
+            -0.0026719090528786182,
+            0.013491204008460045,
+            -0.018277406692504883,
+            -0.014993669465184212,
+            0.0031249718740582466,
+            0.021592123433947563,
+            -0.009781510569155216,
+            0.008302279748022556,
+            0.02241305634379387,
+            -0.008255811408162117,
+            -0.005514200776815414,
+            -0.000941460661124438,
+            -0.009130958467721939,
+            0.004267309792339802,
+            0.006300284061580896,
+            0.043339136987924576,
+            -0.032403673976659775,
+            -0.0018238683696836233,
+            0.014815541915595531,
+            0.0034153966698795557,
+            0.0025905899237841368,
+            0.002503462601453066,
+            -0.0007841471815481782,
+            0.01344473659992218,
+            0.01178737822920084,
+            0.015466093085706234,
+            0.00100099784322083,
+            -0.0033011629711836576,
+            0.006408709101378918,
+            -0.021173911169171333,
+            0.015721667557954788,
+            0.027958236634731293,
+            -0.0033999073784798384,
+            0.005525817628949881,
+            -0.013731288723647594,
+            -0.008759214542806149,
+            0.013134949840605259,
+            0.0026641644071787596,
+            -0.005068882368505001,
+            -0.009394276887178421,
+            0.017967619001865387,
+            0.019082851707935333,
+            0.006668155547231436,
+            -0.037453193217515945,
+            -0.02898053266108036,
+            -0.009959637187421322,
+            0.020569827407598495,
+            0.014916222542524338,
+            0.0010339126456528902,
+            -0.010540487244725227,
+            -0.009595638141036034,
+            -0.011903548613190651,
+            0.010989678092300892,
+            0.02721474878489971,
+            -0.032372694462537766,
+            0.0008214184199459851,
+            -0.029383255168795586,
+            0.03068435750901699,
+            0.0020000594668090343,
+            0.011446612887084484,
+            -0.0008185141487047076,
+            -0.015140817500650883,
+            -0.029181893914937973,
+            0.013824224472045898,
+            -0.004395096562802792,
+            0.001024231780320406,
+            -0.0031753121875226498,
+            0.0201051477342844,
+            0.012887120246887207,
+            -0.0014472840121015906,
+            0.01968693546950817,
+            0.008178364485502243,
+            0.016480645164847374,
+            -0.010153254494071007,
+            -0.04135650396347046,
+            0.019965743646025658,
+            0.007562664337456226,
+            0.014219203032553196,
+            0.004739734344184399,
+            0.004588713403791189,
+            -0.020771188661456108,
+            0.002724185585975647,
+            0.004333139397203922,
+            0.014405074529349804,
+            0.011028401553630829,
+            0.0025615475606173277,
+            0.011678952723741531,
+            -0.02228914201259613,
+            0.01929970271885395,
+            -0.01631026156246662,
+            0.01006806269288063,
+            0.004445436876267195,
+            -0.026873983442783356,
+            0.007547175046056509,
+            0.007822110317647457,
+            -0.0019187404541298747,
+            -0.017549408599734306,
+            0.02160761132836342,
+            -0.0034831624943763018,
+            -0.03509881719946861,
+            0.0360281765460968,
+            -0.0043447562493383884,
+            0.019346170127391815,
+            0.03209388628602028,
+            -0.01771979033946991,
+            -0.01993476413190365,
+            0.00906900130212307,
+            0.010408828034996986,
+            -0.002873270306736231,
+            0.017859194427728653,
+            -0.002919738180935383,
+            -0.02232012152671814,
+            0.017766257748007774,
+            -0.028918573632836342,
+            -0.014234691858291626,
+            0.020337486639618874,
+            -0.03624502569437027,
+            -0.0021026763133704662,
+            0.03175312280654907,
+            -0.024070415645837784,
+            0.005734923295676708,
+            0.005196669604629278,
+            -0.0019351978553459048,
+            0.0012236569309607148,
+            -0.006451305001974106,
+            0.03801080957055092,
+            -0.010865762829780579,
+            0.0201051477342844,
+            0.01929970271885395,
+            -0.023388884961605072,
+            0.01977987214922905,
+            -0.001109423115849495,
+            0.01675945147871971,
+            -0.003940097521990538,
+            0.03466511517763138,
+            -0.006017603911459446,
+            -0.011330443434417248,
+            0.006431943271309137,
+            0.008488151244819164,
+            -0.009959637187421322,
+            0.008681767620146275,
+            -0.018122512847185135,
+            0.027152791619300842,
+            0.02091059274971485,
+            -0.0002691270783543587,
+            0.013800990767776966,
+            0.0004421719058882445,
+            -0.020833145827054977,
+            0.00789568480104208,
+            -0.027121812105178833,
+            -0.009967382065951824,
+            0.01646515540778637,
+            -0.005762029904872179,
+            0.007380664348602295,
+            -0.02064727433025837,
+            0.028004704043269157,
+            0.0077524082735180855,
+            -0.013591884635388851,
+            -0.0008824076503515244,
+            0.023760627955198288,
+            -0.005432881414890289,
+            0.009355553425848484,
+            0.02070923149585724,
+            0.015713922679424286,
+            0.013615119270980358,
+            0.016263794153928757,
+            0.03351890295743942,
+            0.005328328814357519,
+            0.005568413063883781,
+            0.003519949736073613,
+            0.016991792246699333,
+            -0.008557853288948536,
+            0.018029578030109406,
+            0.0008093173382803798,
+            -0.013576395809650421,
+            0.021499186754226685,
+            -0.0018684001406654716,
+            -0.013622863218188286,
+            -0.0040891822427511215,
+            -0.004337011836469173,
+            0.03271345794200897,
+            0.022041313350200653,
+            -0.004468671046197414,
+            -0.024442158639431,
+            -0.00394784240052104,
+            0.016682006418704987,
+            -0.003167567541822791,
+            0.011136826127767563,
+            0.0007952801533974707,
+            -0.01586107164621353,
+            -0.009595638141036034,
+            0.0063583687879145145,
+            0.010586955584585667,
+            -0.008921852335333824,
+            0.010346870869398117,
+            -0.0228312686085701,
+            -0.004011735785752535,
+            0.00046613195445388556,
+            -0.004844287410378456,
+            0.0053438181057572365,
+            -0.011314953677356243,
+            0.004948840010911226,
+            0.016062432900071144,
+            0.01026167906820774,
+            0.004832670092582703,
+            0.006989559158682823,
+            0.012066186405718327,
+            0.009161937050521374,
+            -0.01627928391098976,
+            0.00428667152300477,
+            -0.005072754807770252,
+            -0.02345084212720394,
+            0.005831731949001551,
+            0.00902253296226263,
+            -0.01122201792895794,
+            0.009386532939970493,
+            -0.010958699509501457,
+            0.03296129032969475,
+            0.01030814740806818,
+            -0.037391237914562225,
+            -0.0033379499800503254,
+            -0.0016244432190433145,
+            0.014885243959724903,
+            0.012298526242375374,
+            0.008550108410418034,
+            -0.017518429085612297,
+            -0.0047823297791182995,
+            -0.009835722856223583,
+            0.0012827098835259676,
+            -0.012306271120905876,
+            0.017456471920013428,
+            0.022567950189113617,
+            -0.036337960511446,
+            0.01968693546950817,
+            -0.010432061739265919,
+            -0.023652203381061554,
+            0.020476890727877617,
+            0.011500826105475426,
+            0.01942361705005169,
+            -0.009402021765708923,
+            0.018896980211138725,
+            -0.0001331114035565406,
+            0.025510922074317932,
+            -0.004228586331009865,
+            -0.022087780758738518,
+            -0.012499887496232986,
+            -0.03209388628602028,
+            -0.0051811798475682735,
+            0.006819176487624645,
+            -0.00045524101005867124,
+            0.013111716136336327,
+            0.014226946979761124,
+            -0.025712283328175545,
+            0.007295473478734493,
+            0.032589543610811234,
+            0.014637414366006851,
+            0.023311438038945198,
+            -0.011276230216026306,
+            -0.0010794125264510512,
+            -0.0026060794480144978,
+            -0.013770012184977531,
+            -0.03212486580014229,
+            -0.0017493258928880095,
+            -0.036585792899131775,
+            0.005680711008608341,
+            -0.016387708485126495,
+            -0.013096226379275322,
+            0.023264970630407333,
+            -0.034138478338718414,
+            0.0014608372002840042,
+            0.013708055019378662,
+            0.010540487244725227,
+            0.012391461990773678,
+            -0.00162637943867594,
+            0.015032391995191574,
+            -0.029243851080536842,
+            -0.010284913703799248,
+            -0.004491904750466347,
+            -0.01981084980070591,
+            -0.007338068913668394,
+            -0.018060555681586266,
+            -0.013127204962074757,
+            -0.003566417610272765,
+            -0.010060317814350128,
+            0.002247888594865799,
+            0.0032062907703220844,
+            -0.014714861288666725,
+            0.03206290677189827,
+            0.003157886676490307,
+            0.018958937376737595,
+            0.0033418224193155766,
+            0.004449309315532446,
+            0.018788553774356842,
+            -0.0026331855915486813,
+            -0.0058278595097362995,
+            0.030963165685534477,
+            -0.00032672801171429455,
+            0.008759214542806149,
+            0.019082851707935333,
+            0.004724245052784681,
+            0.0038529701996594667,
+            -0.0260530486702919,
+            0.012887120246887207,
+            -0.027447087690234184,
+            0.005080499220639467,
+            -0.0077175572514534,
+            -0.024302754551172256,
+            -0.028128618374466896,
+            -0.006571347359567881,
+            0.012058441527187824,
+            -0.02898053266108036,
+            -0.00613377382978797,
+            0.002315654419362545,
+            -0.00400399137288332,
+            -0.002673845272511244,
+            0.007241260726004839,
+            0.029150914400815964,
+            -0.033209118992090225,
+            0.008751469664275646,
+            0.005595519673079252,
+            -0.0005053392960689962,
+            0.013297587633132935,
+            0.008038961328566074,
+            0.003343758638948202,
+            0.020848635584115982,
+            0.006637176964432001,
+            0.002458930714055896,
+            -0.0025576751213520765,
+            -0.014242436736822128,
+            0.004000118933618069,
+            0.02154565416276455,
+            0.007562664337456226,
+            0.018385831266641617,
+            -0.008914108388125896,
+            0.002621568739414215,
+            -0.01455222349613905,
+            0.005552923772484064,
+            -0.002867461647838354,
+            0.004309905227273703,
+            -0.012236569076776505,
+            0.008093173615634441,
+            0.0028771425131708384,
+            -0.0043447562493383884,
+            0.018060555681586266,
+            -0.002319526858627796,
+            0.016480645164847374,
+            0.004015608225017786,
+            0.015620986931025982,
+            -0.0035412474535405636,
+            0.01178737822920084,
+            0.0005731051205657423,
+            -0.017905661836266518,
+            -0.017022771760821342,
+            -0.039249956607818604,
+            0.007702067960053682,
+            0.03531566634774208,
+            -0.0009564659558236599,
+            -0.03444826602935791,
+            0.010145509615540504,
+            -0.011942271143198013,
+            -0.009975126944482327,
+            -0.022862248122692108,
+            0.013862947933375835,
+            0.012430185452103615,
+            0.007415515370666981,
+            0.012252057902514935,
+            -0.006997303571552038,
+            0.01026167906820774,
+            -0.005216030869632959,
+            0.008356492035090923,
+            -0.0010823167394846678,
+            -0.037422213703393936,
+            0.012344993650913239,
+            0.02216522768139839,
+            -0.002635121811181307,
+            0.0016089539276435971,
+            -0.006145390681922436,
+            0.022103270515799522,
+            -0.013971373438835144,
+            -0.00650551775470376,
+            -0.00046443779137916863,
+            0.00797700323164463,
+            -0.01795213110744953,
+            0.004778457339853048,
+            0.0755879208445549,
+            -0.007775642443448305,
+            0.006122156511992216,
+            0.009270362555980682,
+            -0.015148562379181385,
+            -0.01637221872806549,
+            -0.004561606794595718,
+            -0.020260039716959,
+            0.008844406343996525,
+            -0.0018819533288478851,
+            -0.0110981035977602,
+            0.012468908913433552,
+            0.0053050946444272995,
+            0.002031038049608469,
+            -0.008379725739359856,
+            0.010811550542712212,
+            0.009525936096906662,
+            0.010432061739265919,
+            0.02360573597252369,
+            0.011477591469883919,
+            -0.00858883187174797,
+            -0.011795123107731342,
+            0.01129171997308731,
+            0.0028577810153365135,
+            0.016697494313120842,
+            0.007907302118837833,
+            0.0010648912284523249,
+            0.019501063972711563,
+            -0.002309845993295312,
+            0.02180897258222103,
+            -2.2689444449497387e-05,
+            -0.009301341138780117,
+            -0.004236331209540367,
+            -0.015349923633038998,
+            0.024271776899695396,
+            -0.0020155487582087517,
+            0.0015450604259967804,
+            -0.020213572308421135,
+            0.0013979118084535003,
+            0.013243375346064568,
+            0.011957760900259018,
+            -0.0020407189149409533,
+            -0.0051347119733691216,
+            0.0007154133054427803,
+            -0.016356728971004486,
+            0.032806396484375,
+            0.007578153628855944,
+            0.0007681738352403045,
+            0.0008838597568683326,
+            0.0009288755827583373,
+            -0.0032972905319184065,
+            0.0024608669336885214,
+            -0.004766840487718582,
+            0.009549170732498169,
+            -0.010943209752440453,
+            -0.010796061716973782,
+            0.010075807571411133,
+            -0.02402394637465477,
+            0.003566417610272765,
+            -0.00558777479454875,
+            -0.008062195032835007,
+            0.0076207490637898445,
+            0.00015537731815129519,
+            -0.007392281666398048,
+            -0.0030939930584281683,
+            0.0023408245760947466,
+            0.013429246842861176,
+            0.004406713880598545,
+            0.010850274004042149,
+            0.012956822291016579,
+            0.005359307397156954,
+            -0.006629432085901499,
+            0.015512561425566673,
+            -0.006946963258087635,
+            -0.004228586331009865,
+            -0.0012468908680602908,
+            -0.019996721297502518,
+            -0.0009419447160325944,
+            -0.004708755761384964,
+            0.0015092412941157818,
+            0.0001705036120256409,
+            -8.749050175538287e-05,
+            -0.0018490385264158249,
+            -0.011988739483058453,
+            -0.001831613015383482,
+            -0.011663463898003101,
+            -0.014498010277748108,
+            0.010253935120999813,
+            -0.0041201612912118435,
+            -0.022939693182706833,
+            -0.015349923633038998,
+            -0.020662762224674225,
+            0.02145271934568882,
+            -0.004387352149933577,
+            -0.005800753366202116,
+            0.006733985152095556,
+            -0.010215211659669876,
+            -0.015706177800893784,
+            -0.008069939911365509,
+            -0.0205853171646595,
+            0.010703125037252903,
+            0.003214035416021943,
+            0.0006791101768612862,
+            0.00882891658693552,
+            0.014397329650819302,
+            -0.01589204929769039,
+            0.0111755495890975,
+            0.012182355858385563,
+            0.011361422017216682,
+            -0.006854027509689331,
+            0.0008911203476600349,
+            -0.014103032648563385,
+            0.02596011385321617,
+            0.021623101085424423,
+            0.005169562995433807,
+            -0.04132552444934845,
+            -0.005862710531800985,
+            -0.013344055972993374,
+            -0.010524998418986797,
+            -0.012120398692786694,
+            0.011771888472139835,
+            -0.0038820127956569195,
+            0.0013117524795234203,
+            0.006288666743785143,
+            0.0015063370810821652,
+            0.004367990419268608,
+            -0.0022382077295333147,
+            -0.007709812838584185,
+            -0.03212486580014229,
+            -0.02817508764564991,
+            -0.00863530021160841,
+            0.020461400970816612,
+            0.002838419284671545,
+            -0.0009453330421820283,
+            -0.0036380558740347624,
+            0.0027048238553106785,
+            -0.005882072262465954,
+            -0.005018542055040598,
+            -0.014753584749996662,
+            -0.012739972211420536,
+            0.0101300198584795,
+            0.00802347157150507,
+            0.014195968396961689,
+            -0.011950016021728516,
+            0.0028190575540065765,
+            0.018788553774356842,
+            0.007690451107919216,
+            -0.023357905447483063,
+            0.01135367713868618,
+            -0.010912231169641018,
+            -0.009270362555980682,
+            -0.009433000348508358,
+            -0.009471723809838295,
+            0.0016980175860226154,
+            -0.008317768573760986,
+            0.003945906180888414,
+            -0.03615209087729454,
+            -0.011330443434417248,
+            -0.0013901671627536416,
+            -0.010346870869398117,
+            0.0009864765452221036,
+            0.02524760365486145,
+            0.006815304048359394,
+            0.01432762760668993,
+            -0.0006733016925863922,
+            0.017502939328551292,
+            -0.0018984107300639153,
+            0.018726596608757973,
+            0.012763205915689468,
+            -0.00797700323164463,
+            0.011717676185071468,
+            0.014738094992935658,
+            -0.0033863543067127466,
+            0.015620986931025982,
+            -0.010168743319809437,
+            0.005285732913762331,
+            -0.009556914679706097,
+            -0.003214035416021943,
+            0.002495717955753207,
+            0.030901208519935608,
+            -0.003618694143369794,
+            0.00858883187174797,
+            0.01098193321377039,
+            -0.015241498127579689,
+            0.0026951429899781942,
+            0.014962689951062202,
+            0.01141563430428505,
+            0.007760153152048588,
+            0.0024434414226561785,
+            -0.024225307628512383,
+            0.004193735308945179,
+            0.005037903785705566,
+            0.0031520782504230738,
+            -0.014366351068019867,
+            0.0021859314292669296,
+            0.015675200149416924,
+            0.008550108410418034,
+            -0.025325050577521324,
+            0.016418687999248505,
+            0.0009138703462667763,
+            -0.003909118939191103,
+            0.029119934886693954,
+            -0.011268485337495804,
+            -0.005723306443542242,
+            0.009463978931307793,
+            0.0009980935137718916,
+            0.0030029932968318462,
+            0.008937342092394829,
+            0.0228312686085701,
+            -0.003988501615822315,
+            -0.032341714948415756,
+            -0.022459525614976883,
+            0.015706177800893784,
+            -0.0047823297791182995,
+            -0.022939693182706833,
+            -0.026703599840402603,
+            -0.003113354789093137,
+            -0.02228914201259613,
+            -0.0022323993034660816,
+            -0.020058678463101387,
+            -0.014056564308702946,
+            0.001268188701942563,
+            0.029987338930368423,
+            0.016201836988329887,
+            0.008224832825362682,
+            -0.011051635257899761,
+            -0.029956359416246414,
+            -0.024271776899695396,
+            0.01810702309012413,
+            -0.007055388763546944,
+            -0.005541306920349598,
+            0.00591692328453064,
+            -0.009951893240213394,
+            -0.008480406366288662,
+            0.011802867986261845,
+            0.027137301862239838,
+            -0.002747419523075223,
+            -0.029987338930368423,
+            -0.02688947319984436,
+            -0.014335372485220432,
+            -0.01840132102370262,
+            -0.017704300582408905,
+            -0.008627555333077908,
+            0.0045035220682621,
+            0.00030300996149890125,
+            0.016186347231268883,
+            0.014962689951062202,
+            -0.000480653194244951,
+            0.0028597170021384954,
+            -2.7711375878425315e-05,
+            -0.001763847190886736,
+            0.006888878531754017,
+            -0.004933350719511509,
+            0.026997897773981094,
+            0.014614180661737919,
+            -0.006594581063836813,
+            0.018711106851696968,
+            0.028314489871263504,
+            0.020430423319339752,
+            0.003157886676490307,
+            -0.010695381090044975,
+            -0.014606435783207417,
+            0.009727297350764275,
+            -0.013421501964330673,
+            0.023264970630407333,
+            0.022552460432052612,
+            -0.01480779703706503,
+            -0.014885243959724903,
+            -0.017905661836266518,
+            0.015210519544780254,
+            -0.007562664337456226,
+            -0.008224832825362682,
+            0.012368228286504745,
+            0.022335609421133995,
+            -0.007024410180747509,
+            0.002371803391724825,
+            -0.0010232636705040932,
+            -0.021731525659561157,
+            -0.006122156511992216,
+            -0.004596457816660404,
+            -0.008666278794407845,
+            0.008681767620146275,
+            0.008131897076964378,
+            -0.001928421319462359,
+            -0.006319645792245865,
+            0.005471604876220226,
+            -0.013367289677262306,
+            0.01301103550940752,
+            -0.00789568480104208,
+            0.03379771113395691,
+            -0.009138703346252441,
+            -0.0006733016925863922,
+            -0.022598929703235626,
+            -0.011446612887084484,
+            0.00845717266201973,
+            0.0017967619933187962,
+            0.01682141050696373,
+            -0.0011626676423475146,
+            -0.005688455421477556,
+            0.009913169778883457,
+            -0.0010687635513022542,
+            -0.001450188341550529,
+            -0.006892750971019268,
+            -0.010401083156466484,
+            0.0002884887217078358,
+            0.013258864171802998,
+            -0.006025348324328661,
+            0.0018054747488349676,
+            0.006095050368458033,
+            -0.0175339188426733,
+            0.029027000069618225,
+            0.023884542286396027,
+            0.017255110666155815,
+            0.014660648070275784,
+            0.0006394187803380191,
+            0.011647974140942097,
+            -0.003572226269170642,
+            0.0038820127956569195,
+            -0.020337486639618874,
+            0.021080974489450455,
+            -0.010811550542712212,
+            0.01846327818930149,
+            0.011655719019472599,
+            -0.015233753249049187,
+            -0.005022414494305849,
+            0.021406250074505806,
+            0.009967382065951824,
+            0.00889087375253439,
+            0.0017541663255542517,
+            -0.02470547705888748,
+            0.017766257748007774,
+            0.010099041275680065,
+            -0.020399443805217743,
+            0.019547531381249428,
+            -0.009471723809838295,
+            0.01233724970370531,
+            -0.0005300253978930414,
+            0.004418330732733011,
+            0.0058743273839354515,
+            0.021499186754226685,
+            0.011570527218282223,
+            -0.03884723410010338,
+            -0.013615119270980358,
+            -0.0017357728211209178,
+            0.006857899948954582,
+            -0.0036303112283349037,
+            -0.015233753249049187,
+            -0.041635312139987946,
+            0.01604694314301014,
+            0.009998360648751259,
+            -0.01643417589366436,
+            0.004987563472241163,
+            -0.002427952131256461,
+            -0.0042053526267409325,
+            -0.008170620538294315,
+            0.003186929039657116,
+            -0.004170501604676247,
+            0.02324948087334633,
+            0.0018984107300639153,
+            -0.006946963258087635,
+            0.013258864171802998,
+            0.01344473659992218,
+            -0.021948376670479774,
+            0.0009874446550384164,
+            0.0021375271026045084,
+            0.000636998622212559,
+            0.023683181032538414,
+            -0.014707116410136223,
+            -0.0017657832941040397,
+            -0.01685238815844059,
+            -0.0013069120468571782,
+            -0.021592123433947563,
+            -0.001188805908896029,
+            -0.015187285840511322,
+            -0.008511385880410671,
+            0.025030752643942833,
+            -0.011005166918039322,
+            0.010997422970831394,
+            -0.006687517277896404,
+            0.006943091284483671,
+            -0.006904367823153734,
+            -0.01936165988445282,
+            -0.0008267428493127227,
+            0.009975126944482327,
+            -0.000512115890160203,
+            0.002941036131232977,
+            8.482827252009884e-05,
+            0.002464739140123129,
+            0.008867640048265457,
+            0.006234454456716776,
+            -0.0075278133153915405,
+            0.02489134855568409,
+            -0.012073930352926254,
+            0.0016399325104430318,
+            0.011911292560398579,
+            0.0004743606550619006,
+            -0.010424317792057991,
+            0.025588368996977806,
+            -0.02139076218008995,
+            -0.00800798274576664,
+            -0.00014799568452872336,
+            -0.0016544538084417582,
+            -0.006513262167572975,
+            0.006718495860695839,
+            0.02946070022881031,
+            0.0062654330395162106,
+            0.010447551496326923,
+            -0.0281441081315279,
+            -0.011493081226944923,
+            0.012538610957562923,
+            -0.022273652255535126,
+            -0.0007739823195151985,
+            -0.0004322490422055125,
+            -0.00619960343465209,
+            0.026378324255347252,
+            -0.02022906206548214,
+            -0.01093546487390995,
+            -0.018664639443159103,
+            0.005405775271356106,
+            -0.011121337302029133,
+            0.008294534869492054,
+            -0.0009777637897059321,
+            0.006540368776768446,
+            0.016805920749902725,
+            -0.019563021138310432,
+            0.016031453385949135,
+            0.005119222681969404,
+            -0.016217326745390892,
+            0.010222956538200378,
+            -0.017580386251211166,
+            -0.0008199662552215159,
+            0.020275529474020004,
+            -0.00400399137288332,
+            0.008751469664275646,
+            -0.003363120136782527,
+            0.005208286456763744,
+            -0.006513262167572975,
+            0.0001492057926952839,
+            -0.007249005138874054,
+            0.014428308233618736,
+            -0.016805920749902725,
+            -0.017828216776251793,
+            -0.008821171708405018,
+            0.0016709112096577883,
+            0.021297825500369072,
+            0.011136826127767563,
+            -0.010044828988611698,
+            -0.011059380136430264,
+            -0.011694442480802536,
+            0.041604332625865936,
+            0.005762029904872179,
+            0.027230238541960716,
+            -0.02261441759765148,
+            0.00545998802408576,
+            0.025170156732201576,
+            -0.014242436736822128,
+            0.0012788375606760383,
+            -0.0022788674104958773,
+            0.0053050946444272995,
+            0.011562783271074295,
+            -0.01819995976984501,
+            0.0049139889888465405,
+            -0.00019216447253711522,
+            -0.00782985519617796,
+            -0.000656844291370362,
+            0.003442503046244383,
+            0.00020196630794089288,
+            0.016325751319527626,
+            -0.004027225077152252,
+            -0.016511622816324234,
+            -0.004429947584867477,
+            -0.0003216455806978047,
+            -0.027896279469132423,
+            0.013514438644051552,
+            0.016589069738984108,
+            0.01566745527088642,
+            0.027989214286208153,
+            -0.014513500034809113,
+            0.016031453385949135,
+            0.009533680975437164,
+            0.013901671394705772,
+            -0.010610189288854599,
+            0.008952830918133259,
+            -0.01595400646328926,
+            0.007446494419127703,
+            -0.03454120084643364,
+            -0.007887939922511578,
+            -0.00802347157150507,
+            0.01984182931482792,
+            0.028763681650161743,
+            0.002923610620200634,
+            0.006493900436908007,
+            0.000196883876924403,
+            0.0024202074855566025,
+            -0.01714668609201908,
+            -0.020182594656944275,
+            -0.0003337466041557491,
+            0.0029332914855331182,
+            0.0006950835813768208,
+            -0.004495777189731598,
+            0.00472037261351943,
+            -0.012391461990773678,
+            0.0026835259050130844,
+            -0.013134949840605259,
+            0.005971135571599007,
+            -0.0059982421807944775,
+            0.009603383019566536,
+            0.0019429425010457635,
+            0.020864123478531837,
+            0.02300165221095085,
+            0.004100799560546875,
+            0.0046971384435892105,
+            -0.024333734065294266,
+            0.00920066051185131,
+            -0.013847459107637405,
+            0.012329504825174809,
+            0.019408127292990685,
+            0.008248066529631615,
+            0.006571347359567881,
+            0.01233724970370531,
+            0.012918098829686642,
+            -0.0031772484071552753,
+            0.024287264794111252,
+            0.005014669615775347,
+            -0.0009070937521755695,
+            0.002052335999906063,
+            -0.014954946003854275,
+            -0.0070979841984808445,
+            -0.0015286030247807503,
+            0.0018848575418815017,
+            -0.004731989465653896,
+            0.01906736195087433,
+            0.015256987884640694,
+            -0.01050176378339529,
+            0.017874684184789658,
+            -0.03937387093901634,
+            0.017379024997353554,
+            -0.0027377386577427387,
+            -0.0060911779291927814,
+            -0.00545998802408576,
+            0.00665653869509697,
+            0.004515138920396566,
+            0.003669034456834197,
+            0.03773200139403343,
+            -0.010408828034996986,
+            0.003963331691920757,
+            -0.005847221240401268,
+            0.007121218368411064,
+            -0.020864123478531837,
+            0.008302279748022556,
+            0.005336073227226734,
+            0.02160761132836342,
+            -0.0056497324258089066,
+            -0.0014482521219179034,
+            0.02328045852482319,
+            0.008859895169734955,
+            0.03469609469175339,
+            -0.01566745527088642,
+            -0.004418330732733011,
+            0.01048627495765686,
+            0.0207402091473341,
+            -0.00580462533980608,
+            -0.008976065553724766,
+            0.007903429679572582,
+            0.0035586729645729065,
+            0.021824462339282036,
+            -0.00515407370403409,
+            -0.0006650729919783771,
+            0.0054367538541555405,
+            -0.0060911779291927814,
+            0.009735042229294777,
+            0.011252996511757374,
+            -0.009076746180653572,
+            0.007539430167526007,
+            0.021530164405703545,
+            -0.009975126944482327,
+            -0.008658533915877342,
+            -0.009649851359426975,
+            0.0010658593382686377,
+            -0.03068435750901699,
+            0.001649613375775516,
+            0.005359307397156954,
+            0.015187285840511322,
+            -0.0004603234410751611,
+            -0.009014789015054703,
+            0.021019017323851585,
+            0.017255110666155815,
+            -0.005936284549534321,
+            -0.0028132491279393435,
+            0.0023795480374246836,
+            -0.0017154429806396365,
+            -0.006160879973322153,
+            -0.0006248975405469537,
+            -0.002656419761478901,
+            0.014730350114405155,
+            -0.0015469965292140841,
+            -0.006617815233767033,
+            -0.00756653631106019,
+            0.0033302053343504667,
+            0.010834784246981144,
+            -0.005661349277943373,
+            0.013057502917945385,
+            0.016805920749902725,
+            -0.00031704717548564076,
+            0.019671445712447166,
+            0.01037010457366705,
+            -0.010447551496326923,
+            0.015938518568873405,
+            -0.007984748110175133,
+            0.006563602481037378,
+            0.0164031982421875,
+            -0.01277095079421997,
+            -0.01246116403490305,
+            0.007210282143205404,
+            0.00665653869509697,
+            0.028918573632836342,
+            0.014792308211326599,
+            -0.02109646424651146,
+            0.014908477663993835,
+            0.015768134966492653,
+            -0.013018779456615448,
+            0.003537375247105956,
+            -0.01784370467066765,
+            0.020260039716959,
+            0.019051872193813324,
+            -0.015535795129835606,
+            -0.017224133014678955,
+            -0.014947201125323772,
+            -0.005092116538435221,
+            0.006331262644380331,
+            0.0033263331279158592,
+            -0.00617636926472187,
+            -0.027989214286208153,
+            -0.0049139889888465405,
+            -0.018633661791682243,
+            0.013870692811906338,
+            -0.008674023672938347,
+            0.0007715621031820774,
+            -0.004511266481131315,
+            0.007984748110175133,
+            -0.0033882902935147285,
+            0.032465629279613495,
+            -0.0051811798475682735,
+            -0.02832997962832451,
+            0.004398969002068043,
+            0.016356728971004486,
+            0.012623801827430725,
+            -0.029243851080536842,
+            0.010463040322065353,
+            0.004805563949048519,
+            -0.012530866079032421,
+            0.005990497302263975,
+            -0.02258343994617462,
+            -0.007632365915924311,
+            -0.0009395244996994734,
+            -0.016743963584303856,
+            -0.027121812105178833,
+            -0.008433938957750797,
+            -0.006993431597948074,
+            -0.005401902832090855,
+            -0.009525936096906662,
+            -0.008720491081476212,
+            -0.011477591469883919,
+            0.02303262986242771,
+            -0.007899557240307331,
+            -0.001866463921032846,
+            -0.0014637414133176208,
+            0.00930908601731062,
+            -0.023776117712259293,
+            -0.015605497173964977,
+            0.006389347370713949,
+            -0.002594462363049388,
+            0.022815778851509094,
+            -0.012747716158628464,
+            0.005940156988799572,
+            -0.009316830895841122,
+            0.009851212613284588,
+            0.015241498127579689,
+            0.004956584889441729,
+            0.017440982162952423,
+            -0.009889936074614525,
+            0.0021026763133704662,
+            -0.008627555333077908,
+            -0.00018986526993103325,
+            -1.027075541060185e-05,
+            0.014428308233618736,
+            0.007624621503055096,
+            0.014141756109893322,
+            0.017487451434135437,
+            -0.013553161174058914,
+            0.005003052763640881,
+            -0.0003274540649726987,
+            -0.017255110666155815,
+            0.0027725896798074245,
+            -0.007984748110175133,
+            -0.0008896682411432266,
+            0.001758038648404181,
+            6.395397940650582e-05,
+            -0.02112744189798832,
+            -0.0009124181815423071,
+            0.0018809852190315723,
+            0.0010222955606877804,
+            -0.025201136246323586,
+            0.00889087375253439,
+            -0.00889087375253439,
+            -0.0014918158994987607,
+            -0.019113829359412193,
+            0.019795360043644905,
+            0.026300877332687378,
+            -0.004619691986590624,
+            0.00036206303047947586,
+            -0.013537672348320484,
+            -0.02049238048493862,
+            0.0001489637652412057,
+            -0.011314953677356243,
+            -0.011345932260155678,
+            -0.0022169100120663643,
+            -0.003214035416021943,
+            0.008805682882666588,
+            0.0004189379105810076,
+            -0.009433000348508358,
+            -0.004956584889441729,
+            -0.011764144524931908,
+            -0.017549408599734306,
+            -0.010401083156466484,
+            0.01933068037033081,
+            -0.0018016024259850383,
+            0.0020794421434402466,
+            -0.0032508226577192545,
+            0.0005271211848594248,
+            0.011748654767870903,
+            0.013475715182721615,
+            0.013344055972993374,
+            -0.023915521800518036,
+            -0.017890173941850662,
+            -0.013320821337401867,
+            0.0029487807769328356,
+            -0.007187047973275185,
+            0.004844287410378456,
+            0.026362836360931396,
+            -0.0009695350890979171,
+            -0.01091997604817152,
+            0.002327271504327655,
+            0.019051872193813324,
+            -0.031272951513528824,
+            -0.013390523381531239,
+            -0.025077221915125847,
+            0.02225816436111927,
+            -0.013351799920201302,
+            0.011516314931213856,
+            0.010191977955400944,
+            -0.025944624096155167,
+            0.006768836174160242,
+            -0.007686578668653965,
+            0.026037558913230896,
+            0.01634124107658863,
+            0.0005934348446317017,
+            -0.00944074522703886,
+            0.013622863218188286,
+            -0.0008722427301108837,
+            -0.013576395809650421,
+            0.002201420720666647,
+            0.014645159244537354,
+            -0.014691627584397793,
+            -0.009510447271168232,
+            -0.004507394507527351,
+            0.004313777666538954,
+            0.012763205915689468,
+            -0.0030339721124619246,
+            -0.004031097516417503,
+            -0.011624740436673164,
+            0.013344055972993374,
+            -0.003930416889488697,
+            0.012383717112243176,
+            0.007380664348602295,
+            -0.014056564308702946,
+            -0.019532041624188423,
+            0.033333033323287964,
+            0.00400399137288332,
+            0.00338054564781487,
+            -0.010997422970831394,
+            0.02740062028169632,
+            -0.009897680021822453,
+            0.010571465827524662,
+            -0.006408709101378918,
+            0.0031985461246222258,
+            0.012321759946644306,
+            -0.012104909867048264,
+            -0.01981084980070591,
+            0.021468207240104675,
+            -0.00364192808046937,
+            -0.0010193913476541638,
+            0.008310024626553059,
+            -0.020182594656944275,
+            -0.00663330452516675,
+            0.002180122770369053,
+            0.009541425853967667,
+            -0.011818356812000275,
+            0.006369986105710268,
+            0.017905661836266518,
+            0.003912991378456354,
+            -0.0007672057254239917,
+            0.019996721297502518,
+            -0.01672847382724285,
+            0.011717676185071468,
+            -0.010850274004042149,
+            0.0016825281782075763,
+            0.0006065039779059589,
+            -0.0048597767017781734,
+            0.0058743273839354515,
+            -0.014536733739078045,
+            0.011988739483058453,
+            -6.244135147426277e-05,
+            -0.013979118317365646,
+            0.007601387333124876,
+            -0.013359544798731804,
+            -0.016836898401379585,
+            0.041542377322912216,
+            -0.005301222205162048,
+            -0.0038665232714265585,
+            -0.0056962003000080585,
+            0.01350669376552105,
+            0.02823704481124878,
+            -0.005336073227226734,
+            -0.01153180468827486,
+            -0.010780571959912777,
+            -0.016573579981923103,
+            0.015907539054751396,
+            -0.01810702309012413,
+            0.002495717955753207,
+            -0.010532742366194725,
+            -0.007903429679572582,
+            0.004302160814404488,
+            -0.003516077296808362,
+            0.014722606167197227,
+            0.00950270239263773,
+            0.023357905447483063,
+            0.00797700323164463,
+            -0.00027348342700861394,
+            0.010393339209258556,
+            0.011663463898003101,
+            0.003243078012019396,
+            -0.010509508661925793,
+            -0.00011949774489039555,
+            -0.02894955314695835,
+            -0.008449427783489227,
+            -0.004863648675382137,
+            -0.001485039247199893,
+            -0.00428667152300477,
+            -0.01239920686930418,
+            0.006617815233767033,
+            -0.0022052929271012545,
+            0.013909416273236275,
+            -0.009254872798919678,
+            -0.017766257748007774,
+            0.03389064967632294,
+            0.004495777189731598,
+            0.006466794293373823,
+            -0.008286789990961552,
+            -0.03705047070980072,
+            -0.0032566310837864876,
+            -0.008116407319903374,
+            -0.013529927469789982,
+            0.004538373090326786,
+            0.01283290795981884,
+            -0.014831030741333961,
+            -0.028097640722990036,
+            0.0019390701781958342,
+            -0.006683644838631153,
+            0.02978597767651081,
+            -0.007798876613378525,
+            0.00419760774821043,
+            0.0037251831963658333,
+            0.0011558911064639688,
+            0.01239920686930418,
+            -0.00845717266201973,
+            -0.0022536972537636757,
+            0.016387708485126495,
+            -0.010858018882572651,
+            0.011051635257899761,
+            -0.010966443456709385,
+            -0.02554190158843994,
+            -0.01456771232187748,
+            0.021344292908906937,
+            -0.0004823473282158375,
+            -0.00024903935263864696,
+            0.014823286794126034,
+            0.02354377694427967,
+            0.011260741390287876,
+            0.00969631876796484,
+            0.006408709101378918,
+            -0.001035848748870194,
+            0.005975008010864258,
+            0.006095050368458033,
+            -0.0024163350462913513,
+            -0.013398268260061741,
+            0.017564896494150162,
+            0.009773765690624714,
+            -0.01810702309012413,
+            0.0008514289511367679,
+            0.0036612898111343384,
+            0.0168833676725626,
+            0.006935346405953169,
+            0.009316830895841122,
+            -0.0143121387809515,
+            -0.023760627955198288,
+            -0.00014932679187040776,
+            0.002873270306736231,
+            0.015721667557954788,
+            0.010579210706055164,
+            -0.014319883659482002,
+            0.0005963391158729792,
+            0.008712747134268284,
+            0.020213572308421135,
+            -0.024209819734096527,
+            0.019702425226569176,
+            -0.004673904739320278,
+            -0.0021084847394376993,
+            0.007946024648845196,
+            0.011632485315203667,
+            -0.011872569099068642,
+            -0.00650551775470376,
+            0.006249943748116493,
+            0.003157886676490307,
+            0.03172214329242706,
+            -0.0017260919557884336,
+            0.009665340185165405,
+            -0.010997422970831394,
+            -0.0002405686245765537,
+            0.0006728176376782358,
+            -0.0006820144481025636,
+            -0.002884887158870697,
+            -0.011020656675100327,
+            0.0018974426202476025,
+            -0.020306508988142014,
+            0.0024473138619214296,
+            0.00463130883872509,
+            -0.005684582982212305,
+            -0.0032972905319184065,
+            -0.019222255796194077,
+            -0.0022672503255307674,
+            0.003293418325483799,
+            0.014335372485220432,
+            -0.031691163778305054,
+            0.004972074180841446,
+            0.010672146454453468,
+            -0.002309845993295312,
+            -0.010416572913527489,
+            0.011423379182815552,
+            -0.0026060794480144978,
+            0.011988739483058453,
+            -0.0038142469711601734,
+            -0.0004905760288238525,
+            0.025681305676698685,
+            -0.005308967083692551,
+            0.014707116410136223,
+            -0.01026167906820774,
+            -0.002993312431499362,
+            -0.028252532705664635,
+            0.005235392600297928,
+            -0.007403898518532515,
+            -0.0012081675231456757,
+            -0.004402841441333294,
+            -0.01130720879882574,
+            -0.0026990154292434454,
+            0.002122037811204791,
+            0.008000237867236137,
+            0.012910354882478714,
+            -0.0007725302129983902,
+            -0.010075807571411133,
+            0.004832670092582703,
+            0.0075355577282607555,
+            -0.015520306304097176,
+            -0.0070437719114124775,
+            -0.020306508988142014,
+            0.029662061482667923,
+            -0.013731288723647594,
+            -0.0011123273288831115,
+            -0.020864123478531837,
+            0.010160998441278934,
+            -0.0063119009137153625,
+            -0.002782270545139909,
+            0.004708755761384964,
+            -0.015109838917851448,
+            0.01344473659992218,
+            0.008751469664275646,
+            0.01450575515627861,
+            -0.02151467651128769,
+            -0.01762685552239418,
+            -0.0044725434854626656,
+            -0.002735802438110113,
+            -0.010648912750184536,
+            -0.0005334137240424752,
+            0.002013612538576126,
+            0.021251358091831207,
+            -0.00032503384863957763,
+            0.0042053526267409325,
+            -0.003214035416021943,
+            0.017224133014678955,
+            -0.010137764737010002,
+            -0.013057502917945385,
+            0.0024260159116238356,
+            0.005994369741529226,
+            0.016186347231268883,
+            0.03158273920416832,
+            -0.020693741738796234,
+            -0.00272999401204288,
+            0.0032740565948188305,
+            0.010656657628715038,
+            -0.004085310269147158,
+            6.304640555754304e-05,
+            -0.027803342789411545,
+            -0.04464024305343628,
+            -0.002422143705189228,
+            0.013057502917945385,
+            -0.015636475756764412,
+            0.012670270167291164,
+            0.008743725717067719,
+            0.010021595284342766,
+            -0.005196669604629278,
+            0.02620794251561165,
+            -0.004793947096914053,
+            0.014304393902420998,
+            0.011733165942132473,
+            0.011911292560398579,
+            0.0009874446550384164,
+            0.0007459079497493804,
+            -0.018354853615164757,
+            -0.008255811408162117,
+            -0.0015954007394611835,
+            0.01765783317387104,
+            -0.027478067204356194,
+            0.008526874706149101,
+            -0.011516314931213856,
+            0.013839714229106903,
+            0.0003666614356916398,
+            0.005661349277943373,
+            -0.01771979033946991,
+            0.012445674277842045,
+            -0.0009337160154245794,
+            -0.025325050577521324,
+            0.004805563949048519,
+            0.011276230216026306,
+            -0.010362359695136547,
+            0.006877261213958263,
+            0.019950253888964653,
+            0.0023079097736626863,
+            0.008495896123349667,
+            -0.014126266352832317,
+            0.010703125037252903,
+            0.009649851359426975,
+            -0.021080974489450455,
+            -0.01233724970370531,
+            -0.007059261202812195,
+            -0.006335135083645582,
+            -0.0035470561124384403,
+            -0.00013105422840453684,
+            0.010176488198339939,
+            -0.016000475734472275,
+            0.010099041275680065,
+            -0.006559730041772127,
+            0.006400964688509703,
+            0.00944074522703886,
+            0.001115231541916728,
+            -0.013576395809650421,
+            0.012716737575829029,
+            -0.0218709297478199,
+            0.008426194079220295,
+            -0.04287445917725563,
+            -0.005847221240401268,
+            -0.015078860335052013,
+            0.005204414017498493,
+            -0.003798757679760456,
+            -0.004453181754797697,
+            -0.006203475873917341,
+            -0.005061137955635786,
+            0.00920066051185131,
+            0.021747015416622162,
+            0.012135888449847698,
+            -0.016620049253106117,
+            -0.007551047019660473,
+            0.005425137002021074,
+            0.0004854935978073627,
+            -0.013336311094462872,
+            0.009316830895841122,
+            -0.01576039008796215,
+            0.0009259713697247207,
+            0.006339007057249546,
+            -0.01072635967284441,
+            0.013452480547130108,
+            0.004731989465653896,
+            -0.003401843598112464,
+            0.013390523381531239,
+            -0.0050417762249708176,
+            -0.0057891360484063625,
+            -0.007272239308804274,
+            -0.013739033602178097,
+            -0.003818119177594781,
+            -0.003072695340961218,
+            0.012747716158628464,
+            0.001529571134597063,
+            -0.0003243077953811735,
+            -0.00782985519617796,
+            0.009154192171990871,
+            0.002220782218500972,
+            -0.0013146566925570369,
+            0.005773646757006645,
+            -0.02300165221095085,
+            0.0001287550403503701,
+            0.004755223635584116,
+            -0.0009487213101238012,
+            -0.006246071308851242,
+            -0.012151377275586128,
+            -0.011903548613190651,
+            -0.038165703415870667,
+            0.009363298304378986,
+            0.0019197085639461875,
+            -0.0076207490637898445,
+            0.0064822835847735405,
+            0.017255110666155815,
+            0.01456771232187748,
+            0.016743963584303856,
+            -0.020693741738796234,
+            -0.02303262986242771,
+            0.005556796211749315,
+            -0.009402021765708923,
+            0.02261441759765148,
+            0.004046586807817221,
+            -0.005847221240401268,
+            0.0013243375578895211,
+            -0.013258864171802998,
+            0.008929597213864326,
+            0.005723306443542242,
+            -0.001592496526427567,
+            0.005425137002021074,
+            -0.013692565262317657,
+            -0.010904486291110516,
+            0.01048627495765686,
+            0.032496608793735504,
+            -0.0004261985304765403,
+            0.0024105266202241182,
+            0.023295948281884193,
+            0.015466093085706234,
+            0.016356728971004486,
+            -0.021111954003572464,
+            0.011167804710566998,
+            0.010060317814350128,
+            -0.0018645278178155422,
+            -0.014892988838255405,
+            0.00513083953410387,
+            -0.012716737575829029,
+            -0.010664401575922966,
+            0.014993669465184212,
+            0.007853088900446892,
+            0.017735280096530914,
+            -0.02369867078959942,
+            0.00789568480104208,
+            0.00042765066609717906,
+            0.00013710475468542427,
+            0.011524059809744358,
+            0.012956822291016579,
+            0.03370477631688118,
+            0.027152791619300842,
+            0.020662762224674225,
+            -0.009285852313041687,
+            0.014366351068019867,
+            -0.009192915633320808,
+            -0.011090358719229698,
+            -0.019470084458589554,
+            -0.015249243006110191,
+            -0.011136826127767563,
+            0.017998598515987396,
+            0.008790193125605583,
+            -0.009355553425848484,
+            -0.005947901867330074,
+            0.01586107164621353,
+            -0.003909118939191103,
+            -0.004635181277990341,
+            0.01646515540778637,
+            -0.026269899681210518,
+            0.016836898401379585,
+            -0.000745423894841224,
+            -0.014668392948806286,
+            -0.014691627584397793,
+            -0.004182118456810713,
+            -0.012670270167291164,
+            0.0003964299685321748,
+            -0.020817656069993973,
+            0.013181418180465698,
+            0.024597052484750748,
+            -0.0030920570716261864,
+            -0.004100799560546875,
+            -0.011299464851617813,
+            -0.0005111477803438902,
+            0.018385831266641617,
+            0.0034773540683090687,
+            0.009192915633320808,
+            -0.0023872926831245422,
+            0.013924905098974705,
+            0.00524313747882843,
+            0.0027958236169070005,
+            -0.0007565568666905165,
+            -0.016263794153928757,
+            0.012197845615446568,
+            0.0026835259050130844,
+            0.012135888449847698,
+            -0.022242674604058266,
+            0.007938280701637268,
+            -0.011764144524931908,
+            -0.0023698671720921993,
+            -0.02228914201259613,
+            0.0189744271337986,
+            -0.01610890030860901,
+            0.001046497724018991,
+            0.003841353114694357,
+            0.0006224773242138326,
+            0.009580149315297604,
+            0.01610890030860901,
+            -0.023388884961605072,
+            -0.0003460896550677717,
+            -0.014908477663993835,
+            -0.021917399019002914,
+            -0.01607792265713215,
+            0.004728117026388645,
+            -0.019501063972711563,
+            0.005444498732686043,
+            -0.0025557391345500946,
+            -0.015125328674912453,
+            0.018478767946362495,
+            0.014691627584397793,
+            -0.004379607271403074,
+            0.002412462839856744,
+            0.005425137002021074,
+            -0.0014405074762180448,
+            0.01971791312098503,
+            0.00013274837692733854,
+            -0.0010048701660707593,
+            -0.009425255469977856,
+            -0.0168833676725626,
+            -0.0065752193331718445,
+            -0.035005878657102585,
+            0.0260530486702919,
+            -0.004670032300055027,
+            -0.016511622816324234,
+            0.014319883659482002,
+            0.009115469641983509,
+            -0.014629669487476349,
+            -0.007059261202812195,
+            0.016294771805405617,
+            -0.024287264794111252,
+            0.02244403585791588,
+            -0.009611127898097038,
+            -0.008774704299867153,
+            0.00784921646118164,
+            0.05154848098754883,
+            0.009146448224782944,
+            0.003492843359708786,
+            0.011036145500838757,
+            0.005839476361870766,
+            -0.0013165927957743406,
+            -0.023683181032538414,
+            0.005568413063883781,
+            -0.007190920412540436,
+            -0.003601268632337451,
+            0.002536377403885126,
+            -0.004801691509783268,
+            0.031149037182331085,
+            -0.004379607271403074,
+            -0.005289605353027582,
+            0.004484160337597132,
+            -0.02196386642754078,
+            -0.0001332324172835797,
+            0.023203013464808464,
+            -2.3128106477088295e-05,
+            -0.023590246215462685,
+            -0.006571347359567881,
+            -0.015071115456521511,
+            -0.013770012184977531,
+            -0.018989915028214455,
+            0.005231520161032677,
+            -0.012755461037158966,
+            -0.0033940989524126053,
+            0.0016660707769915462,
+            -0.005940156988799572,
+            -0.002582845278084278,
+            0.009603383019566536,
+            0.05362405255436897,
+            0.014103032648563385,
+            -0.006335135083645582,
+            -0.005854965653270483,
+            0.009192915633320808,
+            -0.010524998418986797,
+            -0.019795360043644905,
+            -0.009820234030485153,
+            -0.029507169499993324,
+            -0.012747716158628464,
+            -0.0143121387809515,
+            -0.010191977955400944,
+            -0.010277168825268745,
+            0.00963436160236597,
+            -0.018122512847185135,
+            -0.007628493942320347,
+            -0.020337486639618874,
+            0.002941036131232977,
+            -0.011268485337495804,
+            -0.014714861288666725,
+            -0.008054450154304504,
+            -0.04807887226343155,
+            0.005796880926936865,
+            0.019593998789787292,
+            0.01263929158449173,
+            0.002515079453587532,
+            0.004383479710668325,
+            -0.01993476413190365,
+            -0.030792783945798874,
+            0.013839714229106903,
+            0.011283975094556808,
+            0.011733165942132473,
+            0.03283737599849701,
+            -0.011709931306540966,
+            -0.00813964195549488,
+            -0.004693266469985247,
+            -0.0017251238459721208,
+            0.012066186405718327,
+            0.00969631876796484,
+            -0.004917861428111792,
+            -0.012375972233712673,
+            0.016217326745390892,
+            -0.01981084980070591,
+            -0.023683181032538414,
+            -0.01672847382724285,
+            0.006157007534056902,
+            0.00813964195549488,
+            -0.008054450154304504,
+            0.002973950933665037,
+            0.0056962003000080585,
+            -0.007159941829741001,
+            -0.00022423222253564745,
+            0.004890755284577608,
+            -0.023048119619488716,
+            0.0008490087930113077,
+            -0.002257569460198283,
+            0.0061376458033919334,
+            0.021266845986247063,
+            -0.010168743319809437,
+            -0.0014075926737859845,
+            -0.0053902859799563885,
+            -0.0028984404634684324,
+            -0.006764963734894991,
+            0.02620794251561165,
+            0.009154192171990871,
+            -0.009882191196084023,
+            -0.01011453103274107,
+            -0.01129171997308731,
+            -0.0011278166202828288,
+            -0.017255110666155815,
+            -0.016186347231268883,
+            0.0015566773945465684,
+            -0.016712984070181847,
+            0.013971373438835144,
+            -0.00876695942133665,
+            0.00400399137288332,
+            -0.014110777527093887,
+            0.02019808255136013,
+            0.01929970271885395,
+            -0.009711808525025845,
+            0.023776117712259293,
+            -0.007709812838584185,
+            -0.038382552564144135,
+            -0.00800798274576664,
+            0.01843230053782463,
+            -0.006118284538388252,
+            0.010037084110081196,
+            -0.011314953677356243,
+            0.008209343999624252,
+            0.00015319912927225232,
+            0.0018122512847185135,
+            -0.0003112386621069163,
+            0.027555514127016068,
+            0.015535795129835606,
+            -0.005560668651014566,
+            -0.005494839046150446,
+            0.004255692940205336,
+            0.009541425853967667,
+            0.017580386251211166,
+            0.012151377275586128,
+            -0.021019017323851585,
+            -0.014257925562560558,
+            -0.016263794153928757,
+            0.011012911796569824,
+            0.015125328674912453,
+            -0.0040891822427511215,
+            0.01691434532403946,
+            0.014126266352832317,
+            -0.019098341464996338,
+            0.0019884423818439245,
+            -0.008193854242563248,
+            -0.00158184755127877,
+            -0.03029712475836277,
+            0.011043890379369259,
+            0.0074387495405972,
+            -0.0072218989953398705,
+            0.036616768687963486,
+            0.009858956560492516,
+            0.037577107548713684,
+            0.018680129200220108,
+            0.0008974128868430853,
+            0.009045767597854137,
+            -0.018370341509580612,
+            0.013065247796475887,
+            0.01500141341239214,
+            -0.012724482454359531,
+            -0.005734923295676708,
+            0.0013446672819554806,
+            0.0007551047019660473,
+            0.012631546705961227,
+            0.02004319056868553,
+            -0.01795213110744953,
+            0.009773765690624714,
+            -0.015032391995191574,
+            -0.005250881891697645,
+            0.006555858068168163,
+            -0.013932649977505207,
+            -0.008495896123349667,
+            -0.00963436160236597,
+            0.019408127292990685,
+            -0.012600568123161793,
+            0.016573579981923103,
+            -0.009595638141036034,
+            0.015520306304097176,
+            -0.029724018648266792,
+            -0.003669034456834197,
+            -0.006679772399365902,
+            -0.019593998789787292,
+            -0.012887120246887207,
+            0.009820234030485153,
+            0.010633422993123531,
+            -0.009301341138780117,
+            -0.011562783271074295,
+            0.020368466153740883,
+            -0.004185990896075964,
+            0.013529927469789982,
+            0.021189400926232338,
+            0.00737291993573308,
+            0.024132372811436653,
+            0.0005605200421996415,
+            0.014164989814162254,
+            0.005463860463351011,
+            0.010362359695136547,
+            -0.004677776712924242,
+            0.005161818582564592,
+            0.00932457484304905,
+            0.014203713275492191,
+            0.017766257748007774,
+            0.013196907006204128,
+            -0.00598275288939476,
+            0.007419387809932232,
+            0.02016710489988327,
+            0.0016254113288596272,
+            -0.030312614515423775,
+            -0.009719553403556347,
+            0.0001736498816171661,
+            0.00924712885171175,
+            0.009185170754790306,
+            3.817877222900279e-05,
+            -0.01093546487390995,
+            0.011144571006298065,
+            0.00969631876796484,
+            0.002286612056195736,
+            0.006280922330915928,
+            -0.01456771232187748,
+            -0.020957060158252716,
+            0.006106667220592499,
+            0.01798310875892639,
+            0.013917161151766777,
+            -0.014722606167197227,
+            0.022784801200032234,
+            0.006145390681922436,
+            -0.021406250074505806,
+            -0.01840132102370262,
+            -0.0009559819591231644,
+            0.0004542729293461889,
+            -0.002265314105898142,
+            0.03720536455512047,
+            -0.0011539548868313432,
+            -0.005955646280199289,
+            -0.0054754773154854774,
+            0.02775687538087368,
+            0.02049238048493862,
+            0.009177426807582378
+          ],
+          "embedding": [
+            0.0008215984936325879,
+            -0.019349418284688353,
+            -0.02322660633029795,
+            0.009747820845194852,
+            0.007353771220903641,
+            0.015252908783199839,
+            -0.012176175829980202,
+            0.051898051094966605,
+            -0.0068130353886146165,
+            0.008189212816890595,
+            0.008102287831931905,
+            0.01682801136011614,
+            0.02005557996049711,
+            -0.020779879580497183,
+            0.002332165268329781,
+            0.02337485065616377,
+            0.006074992895540845,
+            0.02576575146535689,
+            0.0007517286429767504,
+            -0.03763909942314134,
+            0.0011204729720314833,
+            0.00041212613589632806,
+            -0.06089176800314795,
+            0.012852035873395388,
+            -0.008901127092058543,
+            0.01702347055484216,
+            0.005803446395466281,
+            0.017889570275475296,
+            -0.05375047382413552,
+            0.021623778774252724,
+            0.014784317465883424,
+            -0.0018881880857885708,
+            0.0004735328554290247,
+            0.012393802850407656,
+            -0.019234442968055952,
+            -0.019566283611638122,
+            0.04407789373296535,
+            -0.0053565873209102995,
+            -0.026240866282480922,
+            0.048107902270426675,
+            -0.0054681314939387235,
+            -0.0009441397948041965,
+            -0.015205079626883548,
+            0.006885981841893229,
+            0.01403724256866746,
+            -0.026086548246792754,
+            0.0034778946051893067,
+            0.00044587533948775473,
+            0.032851328435891634,
+            -0.03882097333255334,
+            -0.011126344604254894,
+            -0.01648943336384904,
+            0.035810456555802166,
+            -0.019384589526165754,
+            -0.019430702768170285,
+            0.014297440272239852,
+            -0.007883453204663151,
+            0.050996900793086844,
+            0.06007149434153715,
+            0.03406832053483379,
+            0.025193439934671852,
+            0.00027785129663604897,
+            -0.008014391056168083,
+            -0.018806730908490114,
+            0.027193788903098558,
+            -0.006195006670328595,
+            -0.020466000344352826,
+            0.0068939884219589185,
+            0.017252276167086602,
+            -0.045073444648704136,
+            -0.049242269917908754,
+            0.023021140866008313,
+            -0.015676872472071106,
+            -0.024497517908898602,
+            0.005873633718973869,
+            -0.022828585938203683,
+            0.011748341004940436,
+            0.04624205727927451,
+            -0.015442165499674641,
+            0.018565551071491767,
+            0.015246505395507096,
+            0.004119269796224144,
+            0.004862735250819252,
+            -0.000145323321960046,
+            0.05155838952887077,
+            0.0027052423345323936,
+            -0.016218063670586664,
+            -0.07462470920944148,
+            0.043695974746295135,
+            -0.027608273720377708,
+            -0.03860912774695494,
+            0.002823110565150435,
+            0.026471469849508766,
+            -0.0026063577189074103,
+            -0.029230652014151735,
+            0.05684164470477843,
+            0.00022182868300572677,
+            0.015590845552956364,
+            -0.024939617549529242,
+            0.01035803286203917,
+            -0.010225298012384327,
+            0.019571847648759157,
+            0.01841613041441054,
+            0.0156981470538354,
+            0.021764297694793343,
+            -0.018831230007033416,
+            -0.014419529638789993,
+            -0.0029965475899652273,
+            0.028881185012095884,
+            0.09163526460163937,
+            0.022644302207811818,
+            -0.017933909936330133,
+            -0.04663928709016869,
+            0.004962951444105128,
+            -0.041333041174526026,
+            -0.027545628007617527,
+            -0.037142394579894954,
+            -0.013358143162788402,
+            0.02719686432396677,
+            -0.0034971800937504205,
+            0.04413635817371732,
+            0.04903857110328917,
+            0.03186067155630788,
+            0.007831051050269414,
+            -0.025257538623313574,
+            0.023517513773109414,
+            -0.0016527403209505964,
+            -0.0016101860707429347,
+            0.014448515185903624,
+            0.009905018312477684,
+            0.0024114582235163955,
+            0.00172043851384822,
+            -0.03818316221102173,
+            -0.02304537008110578,
+            0.01024969233570089,
+            -0.01901116356327339,
+            0.019791121460848986,
+            0.07985067340821365,
+            -0.051903191272236665,
+            -0.025924646976556942,
+            0.014766511532849252,
+            -0.002073086076894999,
+            0.030959269926800852,
+            0.006394417986617089,
+            -0.028177586693713442,
+            0.010725285336674526,
+            0.0009614052276924192,
+            0.07500253788978564,
+            -0.00036794182149960157,
+            0.004058458332826519,
+            -0.013218269166951234,
+            0.027613539268683245,
+            -0.012995170442480582,
+            0.023388707018488407,
+            0.026389423313105165,
+            -0.003068639956061266,
+            -0.02661967649632026,
+            0.026145274570635913,
+            -0.03300591322174331,
+            -0.004556072066733024,
+            0.020045722084112896,
+            0.0033437343914784865,
+            -0.010224220905483085,
+            -0.0025025498684079703,
+            0.03460103245395426,
+            0.04514362357278504,
+            0.03797031988058962,
+            -0.004586662475576956,
+            -0.06351723722596923,
+            0.06037983484369874,
+            0.019415753124363902,
+            0.028850618647994476,
+            -0.0034810237045849006,
+            0.02957938154808314,
+            0.02642652523231673,
+            0.013187653724133275,
+            -0.007856320804109569,
+            -0.007065882659482447,
+            -0.016672784625197085,
+            0.029020161154209707,
+            -0.013243058921970134,
+            -0.014900932834241732,
+            0.006001769901896033,
+            0.008648456186944271,
+            0.03484763413464497,
+            0.012279368984306564,
+            0.02379643508687133,
+            0.04933044959399941,
+            -0.021047723935647383,
+            -0.007264823701706426,
+            0.0007058774795716792,
+            -0.010512071832195085,
+            0.027489671668749157,
+            0.009731157570361245,
+            -0.03195919640758579,
+            -0.0033626031972691915,
+            -0.04282419484651207,
+            0.01873977910485268,
+            -0.014398285316695653,
+            0.012854631447494207,
+            -0.0017372959978092432,
+            -0.037331867825735776,
+            0.001088209292238142,
+            -0.012345684323522151,
+            0.037096063326414734,
+            0.005050219690024652,
+            0.0008906051515353004,
+            -0.003697709449630797,
+            0.006548479142737107,
+            -0.010727663712299397,
+            -0.007486226350610755,
+            -0.02704618481069485,
+            -0.019752552016312926,
+            -0.014632581296303486,
+            -0.0037501138119724717,
+            0.014588411795703245,
+            -0.007119137071719376,
+            0.016440258608998535,
+            -0.03686645352111013,
+            -0.023546402448713415,
+            0.005628703432152666,
+            0.024988145751166143,
+            0.011971199595808573,
+            -0.010868778329190568,
+            -0.03270046433624425,
+            -0.021950559285357267,
+            -0.011889914022657738,
+            -0.005123971523701009,
+            0.02619713302280345,
+            -0.021990810305904974,
+            0.01165257474485436,
+            -0.009117901804315863,
+            -0.007492082755249399,
+            0.010725902070291232,
+            0.01862811996181951,
+            -0.026694281868322685,
+            -0.01623776415010199,
+            -0.009403997538684658,
+            -0.02207329027097839,
+            0.025939540497196416,
+            0.014178803271471762,
+            -0.001141821318145744,
+            -0.006281187017652108,
+            -0.035993365198554964,
+            0.0026192776111449633,
+            -0.015254976771874396,
+            -0.0116872453009606,
+            -0.015780016491993135,
+            0.021421328283124968,
+            0.02889432250124136,
+            -0.01871009416296692,
+            -0.026540472640514536,
+            0.015703370682110186,
+            -0.0002849235245164438,
+            -0.02594930889995505,
+            0.04491828744809679,
+            0.044063522197410734,
+            -0.037441864491547405,
+            -0.007005339187980476,
+            -0.030381794393931814,
+            0.06901083845539985,
+            -0.017604082400603037,
+            -0.04060591381445593,
+            0.06044850935418443,
+            -0.01756490171206357,
+            0.025539717592705295,
+            0.03452018789549939,
+            0.0016217577522523927,
+            0.00018825054264059277,
+            -0.002563606092068712,
+            0.04770277039047993,
+            -0.0032010143255574255,
+            -0.03425328601475849,
+            -0.02105909767072799,
+            -0.00840881022563103,
+            -0.028699447677549002,
+            -0.015925160861589006,
+            -0.0055750742281166,
+            0.017513987082479483,
+            0.010399718005952185,
+            0.02267201252273823,
+            0.01258511878601025,
+            0.0015081978484714903,
+            0.01874981572091931,
+            0.011158889421057756,
+            -0.002607810686437211,
+            -0.003732023411827874,
+            -0.007787189972165309,
+            -0.05524012485122728,
+            -0.025110689689279583,
+            -0.03946383236703947,
+            0.004443572529325475,
+            -0.018745077598477586,
+            0.005926160888274727,
+            0.039430331582190524,
+            -0.0049500324516682,
+            0.004352795787899792,
+            0.015566139991780173,
+            0.013148156356408315,
+            -0.015279560337042242,
+            -0.03316846614871165,
+            0.0001592209771257332,
+            0.018362411161743347,
+            -0.012706582418607377,
+            -0.03424477956681259,
+            -0.001991022841226589,
+            0.0033467286047492175,
+            0.03357019284647071,
+            -0.024359110796680374,
+            0.00359117029951049,
+            0.005061818038334802,
+            0.011359582931760315,
+            -0.026793992746244076,
+            -0.018170136079817746,
+            -0.017421603453826397,
+            0.03996608926471779,
+            0.02587013058343799,
+            0.015763410145648683,
+            -0.04512536449983358,
+            -0.02293473890636517,
+            0.048575083783480195,
+            -0.009589860756342604,
+            -0.06486262416944208,
+            0.03447666009581266,
+            0.016105654870253654,
+            0.003526582643722421,
+            -0.01354754623151814,
+            0.05401575248587799,
+            -0.03563190295559527,
+            -0.017365166993914202,
+            -0.0046288709698227735,
+            -0.033657212157212645,
+            -0.007686252059880836,
+            -0.001385273642139782,
+            0.048229062789530236,
+            -0.029083855744728694,
+            -0.030465559177859865,
+            0.0024368661249980416,
+            -0.024144031154493057,
+            0.021693278579520123,
+            -0.016405088055634124,
+            -0.018439935250344958,
+            0.006948603359995389,
+            -0.033943037395199695,
+            -0.003007796423804453,
+            -0.016783195426536633,
+            -0.02270589491206155,
+            0.020874438753828814,
+            0.016724666903417947,
+            0.027033184180344852,
+            0.0019847647852695785,
+            0.017749524526371133,
+            0.0204661893602285,
+            -0.005604929968008821,
+            0.028462234633072944,
+            -0.020790459496727066,
+            -0.04183816355941877,
+            0.03946487892663695,
+            0.01192861758684541,
+            -0.0018024841275380242,
+            0.01077082591611702,
+            -0.01475239020295694,
+            -0.03952356245715099,
+            0.03683950018557526,
+            -0.056767672022551764,
+            -0.007507600090017693,
+            0.009530544352246083,
+            -0.00788856051420074,
+            -0.011204694027562521,
+            -0.010587701466318598,
+            -0.028604940377315504,
+            0.023749537635431462,
+            -0.013672419168235691,
+            0.01953449229707597,
+            -0.011449939491554354,
+            -0.012238180529977605,
+            0.05795536247491174,
+            -0.021518581774137322,
+            -0.040672955795925836,
+            0.026066659238628543,
+            -0.014061106927896723,
+            0.03328357230532424,
+            0.03957107406164427,
+            0.030670947051333917,
+            -0.004278577327978564,
+            0.025885946540364826,
+            0.0319106101668716,
+            -0.003462160036198642,
+            -0.04686868330851868,
+            0.00555303933159587,
+            0.02156143971867743,
+            -0.03739529489075782,
+            -0.01025626126223168,
+            -0.02141970191890338,
+            0.022134910980548407,
+            0.017807484142010496,
+            -0.007232643256393455,
+            0.011318041192005274,
+            -0.010489012862455913,
+            -0.013003217163997554,
+            -0.00847279880392288,
+            0.02357274819910594,
+            -0.0016485066139744577,
+            -0.005873113634992993,
+            0.006424249429163587,
+            0.004008703814301658,
+            0.0054639858216457724,
+            0.010198079500408441,
+            0.017694720520798957,
+            -0.017087021465208255,
+            -0.0025558869389290554,
+            -0.016503579212015177,
+            -0.0420082404922411,
+            -0.003563173424359332,
+            0.022924161700566468,
+            0.012345933179990547,
+            -0.00729908668365632,
+            -0.004234683369642856,
+            -0.0022962545569187934,
+            -0.020666205489143392,
+            -0.014796149043465308,
+            0.024562653223873056,
+            0.030217494971086438,
+            0.007810961433007407,
+            0.007781263640449456,
+            -0.0030348559388780655,
+            0.007310876971216526,
+            0.016099591640649783,
+            0.02311929597780174,
+            -0.04160162038353416,
+            0.027152100845897007,
+            -0.00025396315502635146,
+            0.023717123370023598,
+            -0.04186918759097667,
+            -0.017988531857836703,
+            -0.03106663560912813,
+            -0.023595079699181436,
+            0.02437134274097885,
+            -0.02317559303912261,
+            -0.012327263775257306,
+            0.000989566600374836,
+            0.00954921216572729,
+            -0.005155079395685017,
+            -0.017508404061379164,
+            -0.04582321461886958,
+            -0.0014367264231457263,
+            -0.030417229142190933,
+            -0.017829710557332023,
+            -0.001056472801554354,
+            0.005016912084754242,
+            -0.06276644565618253,
+            0.0109965681725148,
+            -0.07270669271977365,
+            -0.004542796125637244,
+            0.040139998232966566,
+            -0.017890441783630643,
+            0.055909102081208445,
+            -0.05029104308417623,
+            0.01673962668706085,
+            -0.04507293803055318,
+            0.015385506084493484,
+            0.025677372277539627,
+            0.030235171586316813,
+            -0.022189978375852563,
+            0.008461188938299745,
+            0.008033707483259569,
+            -0.045089505755429415,
+            -0.04144698743458133,
+            -0.013640256640978024,
+            0.01036782197518379,
+            0.0012696958402094232,
+            -0.016090728932978384,
+            0.023838437877040163,
+            0.011946482002767135,
+            0.03342604921859589,
+            0.04211801335157251,
+            0.002708015395637621,
+            -0.047156646610976634,
+            -0.013104464812562618,
+            -0.003933116891120681,
+            0.03155321708116329,
+            -0.018866109579446184,
+            0.015376812010630058,
+            -0.01955557376085726,
+            0.016255217383093355,
+            0.017868332102274922,
+            -0.020325179506828335,
+            0.013434364256590097,
+            0.004176904701252156,
+            -0.03877060944639854,
+            0.0006475542749369515,
+            0.00042468300294954125,
+            -0.02116992978038693,
+            0.0032654387271108293,
+            -0.00575003786159731,
+            -0.05149954071159492,
+            -0.031382926943128234,
+            -0.015730563640253065,
+            0.0382312502769352,
+            0.004297939848910064,
+            0.019724983545557365,
+            -0.017907869879607064,
+            -0.005165628493894463,
+            0.03698976707579373,
+            0.008501001373266799,
+            -0.01725527821298066,
+            0.006879620077191753,
+            0.02455792200929138,
+            -0.03224193972301503,
+            -0.039181328884413266,
+            -0.0008206986106407542,
+            -0.023306758893533952,
+            0.01672946960876128,
+            -0.02658587698869307,
+            0.005873620623736886,
+            0.0025331170979801232,
+            -0.03490809296591325,
+            0.02456764314036355,
+            0.003291113462353163,
+            -0.004303423672882112,
+            -0.020359544091059775,
+            -0.016892415165120203,
+            0.017174797487220006,
+            0.04391489511478065,
+            0.007327321684948843,
+            -0.003943713832544148,
+            0.00862684231313155,
+            -0.006361172785753631,
+            0.05826688726534012,
+            -0.02284956969636637,
+            -0.026680023999604643,
+            0.027527436347931256,
+            -0.022597669279069723,
+            -0.006277863878949772,
+            -0.017343446085471857,
+            -0.012065192152203348,
+            -0.0004183712490442143,
+            0.019339311388446943,
+            -0.019423562920897294,
+            -0.010212527447243049,
+            0.004783006556748757,
+            0.012824133870627983,
+            -0.004855945108275706,
+            0.006239949458562236,
+            0.006847581690471813,
+            0.03094005096378827,
+            0.026993131409761086,
+            -0.011331881651539968,
+            0.03297014291876675,
+            0.010265600643742144,
+            4.666113318251278e-05,
+            0.001970879281172325,
+            0.002523664335535017,
+            0.0034107811520368574,
+            -0.03143890864030425,
+            -0.01624584278940617,
+            0.013468729048885906,
+            0.005233309338223403,
+            -0.004550544967856448,
+            -0.023783653541634683,
+            -0.009402494239255264,
+            0.03334923712760375,
+            -0.003260424721959948,
+            0.02748395114281811,
+            -0.0058374897745283065,
+            -0.01985840930273332,
+            0.017595418271610163,
+            0.02881743246631328,
+            -0.05928552440605384,
+            0.017062540894682354,
+            -0.0011508547522925793,
+            0.008953761538519997,
+            0.01610561914042553,
+            -0.008894160513648953,
+            0.013301991014533684,
+            0.04080863883897929,
+            0.010738051773755976,
+            -0.023345002072886145,
+            0.004215726310838674,
+            0.01541749448570608,
+            0.01291813299324752,
+            -0.022747942296528025,
+            0.0008640791861357522,
+            0.011008522066277018,
+            0.005893793007012561,
+            0.0026106450193140156,
+            -0.006278394080583205,
+            -0.025659188287146268,
+            0.024180550333596625,
+            -0.0044491292740074925,
+            0.026967973346698617,
+            0.04390909273429254,
+            0.009596855239692042,
+            -0.03686020608700941,
+            -0.013109951664921729,
+            0.0002799680359670127,
+            -0.0038537577602593366,
+            -0.0037552094562213916,
+            0.011105295862811114,
+            -0.04185615640149602,
+            0.019950890919369135,
+            0.006453241355231516,
+            0.015314645427201533,
+            -0.020031193483734364,
+            -0.024530173488182337,
+            0.0031949828520724337,
+            -0.01492645564652531,
+            0.012875827757040941,
+            0.005057170413681063,
+            -0.004319834710426384,
+            0.005386862713494232,
+            0.00030848054013413157,
+            0.008384941028326983,
+            -0.010811526993301901,
+            0.03175068297908541,
+            0.004680987487229685,
+            0.009264547472281849,
+            -0.02856314241700808,
+            -0.021083721348429348,
+            0.020113396771629096,
+            0.03572366771051171,
+            -0.01708529918264109,
+            0.03346921724975398,
+            -0.009042046660366718,
+            0.0033219155387035227,
+            0.0151720168318272,
+            -0.015805318210244414,
+            -0.008057236707694303,
+            -0.0005483009439015931,
+            -0.002917527004423299,
+            -0.014538511124345527,
+            0.007302979537905532,
+            -0.0208529958115422,
+            -0.02965227242147972,
+            0.020064178921373875,
+            -0.00011892622932349096,
+            0.016455188521316146,
+            0.010335044657747514,
+            -0.007816155992852236,
+            -0.0052262330850905165,
+            0.006959344748883874,
+            0.04432147882994085,
+            0.026734484553994495,
+            0.05547661378166294,
+            -0.005787997952888261,
+            -0.004559444854138438,
+            -0.0082295581123363,
+            -0.006588172755707144,
+            0.04524009609473585,
+            0.015915003261638767,
+            -0.004910285095602864,
+            0.017495607570191165,
+            0.01557886571266685,
+            0.03858259260136037,
+            0.0027648525902558316,
+            -0.016912796336835408,
+            0.04014755669075221,
+            -0.004077405782937414,
+            0.008616827127381418,
+            0.011102692199778469,
+            0.0024185742218798562,
+            0.00807141794538637,
+            0.02996274720815659,
+            -0.012429186019907761,
+            0.01680942269506771,
+            -0.04571916592313758,
+            -0.02775530246832996,
+            0.014370280587419414,
+            -0.004931905540824582,
+            0.03419959665162407,
+            0.01668521021731861,
+            -0.023105305446763272,
+            -0.024761385385514877,
+            0.021529428247380984,
+            -0.002603042335561961,
+            -0.024135326707477325,
+            -0.015906833215844473,
+            0.003286605488908195,
+            -0.0023437997044735617,
+            0.0060251848371355166,
+            -0.013481652076194643,
+            0.0002908426939460218,
+            0.01953362396253743,
+            0.0038544889405965057,
+            0.020760460182873287,
+            -0.016224396121239458,
+            -0.00259233446682179,
+            -0.009282861237653619,
+            -0.02952130765234087,
+            0.017988370145301916,
+            0.006138236653154911,
+            -0.007607182955958238,
+            -0.009516767050028553,
+            0.0048347778733230365,
+            -0.005819209642901871,
+            -0.0025851701689263706,
+            -0.01073014664868326,
+            0.012433465684360061,
+            0.025784014942833097,
+            -0.016875624785552303,
+            0.007944349714123104,
+            8.651589872318479e-05,
+            -0.022306044741692753,
+            -0.022127016745066307,
+            0.020143472307319017,
+            0.002792646249762062,
+            0.008768813346304779,
+            -0.004074214276300631,
+            -0.00015024341785520542,
+            0.002990550758627699,
+            0.01951746553838439,
+            0.03896451381932467,
+            0.004232673242619471,
+            0.027876586543280867,
+            0.0145860303663199,
+            0.03994234200854897,
+            0.009657017763624678,
+            0.00696607158576712,
+            -0.026055387694656488,
+            0.02819890357011778,
+            0.00018297502510753014,
+            0.043190622821775236,
+            0.022488402221902314,
+            -0.0407730101161195,
+            -0.0021946518523759066,
+            0.01669520746276111,
+            0.013212623257794862,
+            0.017009844136059122,
+            -0.004537512994041156,
+            -0.014041213424740049,
+            0.028135404528975894,
+            -0.022859586430516003,
+            0.0008637006862481223,
+            -0.004486332934188017,
+            -0.025031795510001887,
+            0.007351370254175368,
+            -0.004567816916440959,
+            0.030025413528233364,
+            0.0014976740344892735,
+            -0.022285096307825804,
+            0.022837170656498543,
+            -0.009551269792616045,
+            0.011045530488684207,
+            0.008809047185288755,
+            0.020065001354321994,
+            0.031993362733068575,
+            0.009143210434415064,
+            0.02133640814803879,
+            0.012241777474889248,
+            0.014771538127546474,
+            -0.008804192497858033,
+            -0.02341427260922065,
+            0.023167424359504862,
+            -0.0011598649565273595,
+            -0.029627604902846688,
+            0.0021991474725787303,
+            0.003261878079344105,
+            -0.0005935713331682358,
+            -0.009144689470851402,
+            -0.005580557862988033,
+            -0.007240836451875213,
+            0.019631377011298905,
+            -0.007085213886352692,
+            0.007017573363709754,
+            -0.004074171791476129,
+            -0.00822871835665378,
+            -0.0421101573155962,
+            0.004210914075286222,
+            0.003408427700622687,
+            -0.01752179330665445,
+            0.006384440404174512,
+            0.011121481025573962,
+            0.043020575455262985,
+            -0.022809979913698254,
+            -0.0005737138409049003,
+            0.019206623212867476,
+            0.0009562551889820241,
+            -0.023427467810802533,
+            -0.038931688376706186,
+            0.007172345672967499,
+            -0.03467050750542982,
+            0.023185301620612612,
+            -0.0007075596339842658,
+            0.02387977204565927,
+            -0.03043468414449851,
+            0.013909304572370437,
+            0.004784668158421625,
+            0.012524879174804915,
+            0.021654512058639427,
+            0.03142146922870715,
+            -0.02053473475540882,
+            -0.050409862638621725,
+            -0.005406667316025549,
+            0.00035138762584085976,
+            0.022018487016014856,
+            -0.026014580970974616,
+            0.00025633004699302783,
+            -0.003349094027291921,
+            -0.02401366769212778,
+            -0.01735996745614562,
+            0.01962340791446025,
+            -0.050011480783403936,
+            0.015042913576492952,
+            0.0262737424555697,
+            -0.02507473901367252,
+            -0.05408889321034101,
+            -0.018988587082325706,
+            -0.028217893770659686,
+            -0.009285770094319318,
+            -0.00661159230987788,
+            0.028597405154740078,
+            0.003525576256054959,
+            -0.011462219358952466,
+            0.025401668390029348,
+            0.006155837251830549,
+            0.0026050693705924593,
+            0.015985722392086222,
+            0.007289910674212967,
+            -0.011452511453623235,
+            -0.014599577191293152,
+            -0.04558584556100621,
+            0.010197536827229795,
+            -0.0045431962637690155,
+            -0.019372598050508005,
+            -0.030147301333718016,
+            -0.0137828704002156,
+            -0.027060515748528666,
+            -0.0017291966950107505,
+            0.053783684195381654,
+            0.03292494290301388,
+            0.005560172296221419,
+            -0.0026085670565688026,
+            -0.010466890489356092,
+            0.036247483457995895,
+            -0.006789746401727701,
+            0.03662690472838588,
+            0.001784522073210422,
+            -0.0242673253380092,
+            0.01169382884548301,
+            0.02892627181542449,
+            0.009858451662166387,
+            -0.0005592283645355213,
+            -0.006160132931009508,
+            -0.008782830470079547,
+            0.0031207630972208394,
+            0.025613233596604418,
+            0.003268170392795548,
+            -0.010062991425127038,
+            0.018106967407336815,
+            0.004480291699976396,
+            0.02010374270557836,
+            0.03555357587371324,
+            0.023489837757983054,
+            -0.021782173652026497,
+            -0.026581762796336082,
+            -0.0013561486781077014,
+            0.0072266106510934505,
+            0.03551967295942857,
+            -0.01651315514034218,
+            0.0001274396319698452,
+            -0.008272735507120127,
+            -0.00894931703949637,
+            0.011149952898504355,
+            -0.0029671593121986116,
+            -0.0030752010285951576,
+            0.01662711184497578,
+            -0.0011672263615879709,
+            -0.033448222499147764,
+            0.0078288560210653,
+            -0.023496331346917084,
+            0.0051040606404415545,
+            -0.004113104164416575,
+            -0.007076525381595441,
+            -0.002157548492138289,
+            0.006302962169276006,
+            -0.008065841090139872,
+            -0.00035242603988955366,
+            -0.01586406637763306,
+            0.008271894439780969,
+            0.007438312999409211,
+            -0.0069617481460406095,
+            0.011402932583388205,
+            0.002666895909532698,
+            0.0005219881149353855,
+            -0.012159425670629751,
+            -0.012042145306711322,
+            -0.00957276588761916,
+            -0.03401863879157551,
+            -0.02604353712472079,
+            -0.024949533021906844,
+            -0.0014209149651479556,
+            -0.006566685925666972,
+            -0.03187796739029736,
+            -0.01209152137926458,
+            -0.011110763124407752,
+            0.007756961097537267,
+            -0.013260548545917543,
+            0.005433285079114047,
+            -6.399098520041351e-05,
+            -0.0044960488826674555,
+            0.016615636438440516,
+            -0.003643291599919672,
+            -0.019488010211707343,
+            -0.03927818878149417,
+            0.008209336177358258,
+            -0.02805593125673831,
+            -0.0038816771550190788,
+            0.007607119219721711,
+            -0.00528681677614909,
+            -0.029502897838743516,
+            -0.009149794737430706,
+            0.008767001199311985,
+            0.003507217008280066,
+            -0.01773349756494436,
+            0.03579670773605949,
+            0.026971617790626075,
+            0.011875179879209696,
+            -0.019891342676443626,
+            0.0508929813275565,
+            -0.0047010735337056275,
+            -0.012544935684845649,
+            0.018297503869683027,
+            0.02355022268131519,
+            0.03273167043015095,
+            -0.024830243320788693,
+            0.0009370814985214882,
+            -0.021937793653233454,
+            0.00535662615596573,
+            -0.035009915581788176,
+            0.0028595659314725012,
+            -0.014395027018699446,
+            0.015596459769560532,
+            0.03397372881171701,
+            -0.013254226072981457,
+            0.011195747181764677,
+            -0.016299384375589934,
+            0.00730244731485402,
+            0.00569944090461748,
+            -0.012916284280345044,
+            -0.005812311882399539,
+            -0.022902815152116493,
+            0.020140276669979015,
+            0.005809799950176385,
+            -0.0008130655714656685,
+            0.0104085826652653,
+            -0.044723432238900435,
+            0.03286429597337671,
+            0.008730225944501239,
+            0.017599189944921396,
+            0.0029762446259779998,
+            -0.0003068671951232415,
+            0.011611551606224214,
+            0.011228401175698422,
+            -0.016029727902740706,
+            0.004345913009102136,
+            -0.004613061137378433,
+            -0.03850533909474302,
+            0.009363351681220008,
+            -0.03631184057978666,
+            0.014427784677340159,
+            0.016026872002233575,
+            -0.004170582842643461,
+            -0.025166189571409366,
+            -0.005428471012333673,
+            0.010827910811085384,
+            -0.009783069664327864,
+            -0.009987805234276443,
+            -0.04163358183060102,
+            -0.025236440364963034,
+            0.034097508113604576,
+            -0.002119474613596975,
+            -0.025826403511707147,
+            -0.010830738112107807,
+            -0.011351132636491575,
+            0.020207687645133,
+            -0.006542026283946858,
+            -0.0006263975804936843,
+            -0.04150011545311013,
+            -0.0073396962917864515,
+            0.011192225907726928,
+            0.027746213794734322,
+            -0.035970017302005566,
+            0.02708126943501373,
+            0.002105633542220187,
+            0.014262674845755191,
+            -0.01626664983003942,
+            -0.005631805361903626,
+            -0.013990539604372344,
+            -0.035877093361375685,
+            0.016487723013906077,
+            -0.008607758248055054,
+            0.004714436224135091,
+            0.04458009880881818,
+            -0.0036872068506712727,
+            -0.009466125348499464,
+            0.007600705817195143,
+            -0.0034590883553048063,
+            -0.010351576085287273,
+            -0.0028564505793999295,
+            0.001959661298507,
+            -0.007080697863126256,
+            -0.04183476693971616,
+            -0.00499888976530217,
+            0.04684391782633148,
+            0.006361744381879857,
+            0.013276793765092465,
+            0.013516220521120684,
+            -0.018335746557932003,
+            -0.045577810515431215,
+            -0.0071616111758423025,
+            -0.0032763880176386808,
+            -0.022708028392758567,
+            -0.0240284945459861,
+            -0.008735040856657758,
+            0.005844145402902923,
+            0.006359368165686724,
+            -0.01120763599008222,
+            -0.022676131361171057,
+            -0.027085968700452136,
+            -0.010526103176253857,
+            -0.02123944716136792,
+            -0.010792145796733375,
+            -0.027142270190771887,
+            -0.028742234538796804,
+            0.02880426953386151,
+            0.007575182232859564,
+            -0.004084088301176117,
+            -0.010005765476121966,
+            -0.01255640546813973,
+            -0.0004774761617900028,
+            -0.0025307994917461232,
+            -0.005822065709999144,
+            0.0038614286313282164,
+            -0.011390482620811868,
+            0.023152086963976554,
+            0.0021220701260546895,
+            0.018329162172128138,
+            -0.05772324571231113,
+            -0.002709083851090071,
+            0.021497896395795905,
+            0.0033025410488410913,
+            -0.005412445141910314,
+            0.029352650459792132,
+            0.011223947499453582,
+            -0.006539460282439711,
+            -0.031355428613718234,
+            0.020942976176820262,
+            0.024122993515785947,
+            0.00817615634503138,
+            -0.012994160510030828,
+            0.01221095914384204,
+            0.01629778384790159,
+            -0.004494985018246821,
+            -0.011161098655072933,
+            0.005586857967039181,
+            0.020941039642372112,
+            0.013580622629911925,
+            0.0013398071867077221,
+            0.030905022925366163,
+            0.0006862084391508121,
+            -0.03521231612862609,
+            -0.01908191995763771,
+            -0.01275995164030872,
+            -0.003584331305986364,
+            -0.015264425022763126,
+            0.005294329654809353,
+            -0.002355465010876059,
+            0.006335052149475905,
+            0.022655379849785256,
+            0.03593231939948758,
+            0.0146937743203089,
+            0.00937330328448886,
+            0.003167462724350819,
+            -0.0013235176101076284,
+            0.012102872918129403,
+            0.008638011773075119,
+            0.0350701240268404,
+            -0.014653221697693864,
+            -0.011003994230902835,
+            -0.0037767288519369537,
+            0.010100131061186566,
+            -0.004113527518662366,
+            0.020414393256136372,
+            0.03274709615649469,
+            -0.0037222949335916876,
+            -0.005604552180246868,
+            -0.006160118231975786,
+            0.0041467454414452565,
+            0.01807879127455274,
+            -0.01252134965321004,
+            0.004364858987501461,
+            0.005549300665159658,
+            -0.0071897162833861,
+            0.018567059255813558,
+            0.0002622074720860951,
+            -0.012983550838417323,
+            0.03266376564402191,
+            0.11449509387240629,
+            0.0017251916835047628,
+            0.007125215166916139,
+            -0.040229997815548584,
+            -0.020537987527148888,
+            -0.00245728735683473,
+            0.005779972828836593,
+            0.007004732428236457,
+            0.02225914982152608,
+            0.006505827715591914,
+            0.0006708045351142873,
+            0.031708522931495536,
+            -0.017334083511709092,
+            -0.002522861743921599,
+            -0.01745656559688129,
+            0.013890451996979048,
+            -0.014312605700272691,
+            -0.013784963572164409,
+            -0.01572417853955075,
+            0.004507425871748977,
+            -0.000601158156253668,
+            0.0016158265715634729,
+            0.01791303234734911,
+            0.0011926917387973374,
+            -0.026393351252156246,
+            0.0006694008039871878,
+            0.001026031515214626,
+            -0.004514419527575117,
+            -0.011320929719992686,
+            -0.02223001723942815,
+            0.016235841442653918,
+            0.014188311619681008,
+            -0.004895218832118094,
+            0.009244697376301004,
+            0.010558796153959677,
+            0.02707485418319703,
+            -0.028380342401606758,
+            0.0040204430418629405,
+            0.007023041225985294,
+            0.01606556581447933,
+            0.017617862031190843,
+            0.0034870847111552862,
+            -0.011253315705053224,
+            0.03238289399766116,
+            -0.010563282687352038,
+            0.0065093536270233385,
+            0.014966802208644462,
+            -0.03541611540945151,
+            -5.821682910348165e-05,
+            0.0061985444938402745,
+            -0.011716201661363997,
+            0.009580035961315467,
+            -0.03120692534097433,
+            -0.0039291243987978845,
+            0.009317468331974374,
+            0.017611312558635673,
+            0.008804996982574765,
+            -0.0034129169537151925,
+            0.010822226698822738,
+            0.012510661458054741,
+            -0.0027147975086409966,
+            -0.008855213644027594,
+            0.01238502686635956,
+            -0.0007687053446954701,
+            0.00450108863711973,
+            -0.00799170686059026,
+            0.0029755595812552293,
+            0.002290650917244691,
+            -0.01554856300008412,
+            -0.011779430686394945,
+            0.00438393126679289,
+            0.0036931891075561957,
+            0.001535662168147833,
+            0.012014408636737167,
+            -0.006087255045485002,
+            -0.0077816220325470226,
+            0.0040573724156794284,
+            0.003136943055301706,
+            -5.505014316363865e-05,
+            -0.030625245024561508,
+            0.01585923084309149,
+            0.0067595435845396405,
+            0.010013308598626031,
+            -0.005315626260228517,
+            -0.00903142414091241,
+            -0.007240920015018893,
+            -0.01228521172101564,
+            -0.010657894223173453,
+            0.007658605592936815,
+            -0.00724924993432376,
+            0.011124174211132038,
+            -0.03128027787357322,
+            -0.013988715435985316,
+            0.01231792993245938,
+            -0.01528161242159377,
+            0.0007675821323979293,
+            -0.008389914312201402,
+            0.0009417242487395995,
+            -0.005119304379761705,
+            0.007887436211143328,
+            -0.03731503169547854,
+            0.007878001990271541,
+            0.02456170736765724,
+            -0.00330924625934981,
+            -0.020259771366167557,
+            -0.009613930457931786,
+            0.0241314385136084,
+            -0.008267831641886344,
+            -0.01645888287387047,
+            0.008421304862524506,
+            0.010140813149093993,
+            -0.0006309118552875393,
+            0.0008237784055549769,
+            -0.003726984191590917,
+            0.01669125388798323,
+            0.02054777616420997,
+            0.041930770964107515,
+            -0.014789610432004559,
+            0.01092349703703152,
+            0.02866938974792608,
+            -0.03378576980722847,
+            0.014728113846952417,
+            0.02518409645155824,
+            0.014590531327650454,
+            -0.0004446956315190408,
+            0.001609777186372437,
+            0.0068939321280085475,
+            -0.014273492115311677,
+            0.009247676157551494,
+            0.013380104440744367,
+            -0.018407746260686045,
+            0.020444430509441124,
+            0.03140071554008003,
+            0.0025839998101985736,
+            0.01355205268903058,
+            -0.006268939830271642,
+            -0.013860692186849113,
+            0.014717364068116991,
+            0.031429975488060896,
+            0.0006919565473572909,
+            -0.018561500446336806,
+            0.019128244642842268,
+            0.006988095806264863,
+            -0.003678529269242034,
+            -0.0021993078588142664,
+            -0.021334919445200388,
+            -0.021003586791369228,
+            0.017304350704792804,
+            0.013508779408333964,
+            0.027794485858413365,
+            -0.010594702782952029,
+            -0.007818089330662161,
+            0.014293688127901867,
+            -0.022368568887004094,
+            0.015426482139881204,
+            -0.02160820059598751,
+            -0.00571499813703566,
+            -0.0077402210272634235,
+            0.03274360648629609,
+            0.001441304806200403,
+            0.02734331677589297,
+            0.016457027764064002,
+            -0.033609420134045206,
+            -0.029790535553657092,
+            -0.00712106584238484,
+            -0.02307933297713104,
+            0.010356561171963577,
+            -0.03901183838032092,
+            -0.00071735150189382,
+            0.008251646656432826,
+            -0.008079917288471352,
+            -0.0007087526632966007,
+            -0.006419191399519291,
+            0.013989344185652506,
+            0.013661633160336152,
+            -0.0228103143826663,
+            0.023644639659577426,
+            0.008343481940133684,
+            0.025998483906975782,
+            -0.02076055376006461,
+            0.019148568238437927,
+            -0.017705363646416224,
+            -0.02777313933338929,
+            0.014358130220942483,
+            0.023585731219788467,
+            0.01635364038245971,
+            -0.007122853965024528,
+            0.007110003583832299,
+            -0.03556761320536796,
+            0.027585457849964405,
+            -0.013064062623866305,
+            -0.012504161289952458,
+            -0.0032379276562094555,
+            -0.011931827532360686,
+            -0.002574161005800279,
+            0.0008533907465694845,
+            0.0034345232246102806,
+            -0.03131462673429805,
+            -0.0002450959360520896,
+            -0.03727416416766574,
+            -0.028944814604815072,
+            0.01360126708805471,
+            -0.00036361284498363273,
+            -0.0049097813612814616,
+            0.02933242705521636,
+            -0.026339136251046315,
+            -0.02693295650849956,
+            -0.009590247544093352,
+            0.02853096456887074,
+            -0.0220673798697511,
+            -0.004519050393279848,
+            0.0015034014871316069,
+            -0.01828226786700525,
+            0.004436383319946364,
+            -0.0012748481042235446,
+            -0.010416065158876917,
+            -0.01105146139270597,
+            -0.03229479099927737,
+            -0.005822133182096844,
+            0.02106215163620738,
+            -0.006957125617688484,
+            0.002746241228040847,
+            0.013312248331456934,
+            -0.010968629118224582,
+            -0.0029946900704643812,
+            0.0017875284441794235,
+            0.023304175718480834,
+            -0.021033941064173063,
+            0.007714334541806199,
+            0.005955442031111047,
+            0.0016405586860131207,
+            0.018406562522853148,
+            -0.0054922197063461825,
+            0.0037690571959124617,
+            0.0162126534419016,
+            -0.009969304596173302,
+            0.017959307638726878,
+            0.007691549802475525,
+            0.006681837561745192,
+            0.002775700618222622,
+            0.007712770283579111,
+            0.009752652973700739,
+            -0.013506020126078112,
+            0.016911141425375933,
+            0.022468715726349706,
+            0.00023233555209372766,
+            0.021066580799163234,
+            -0.02302516292580905,
+            -0.028618544628453015,
+            0.018297878810036087,
+            -0.0225078941299981,
+            -0.012735208680311652,
+            0.016459925496937065,
+            -0.0043032724578332855,
+            -0.00797609922711709,
+            -0.014986057073351055,
+            0.02260618561608301,
+            0.02132930921464508,
+            -0.020249788476562527,
+            -0.012385936327068884,
+            0.0072304431443959505,
+            -0.004373475447051833,
+            0.007086656081865733,
+            0.03195810137269375,
+            -0.015982210897392764,
+            0.006818890093901413,
+            -0.0038306138017521626,
+            0.0030753642449792215,
+            0.017830546304667783,
+            0.009470018737705222,
+            -0.0019815241970236146,
+            0.030100238416495066,
+            -0.001330614013444443,
+            0.016392226282459177,
+            0.0004888028477871524,
+            -0.010948833692356176,
+            0.006343890495626752,
+            -0.026652806306173643,
+            -0.008627570055007363,
+            -0.001203530223345183,
+            -0.009581494451831351,
+            0.07598616930784373,
+            0.003930153552069478,
+            -0.01295957554290434,
+            -0.001673007115291578,
+            -0.001368287102181597,
+            -0.006897179139643672,
+            -0.00960679451603541,
+            0.0287166325268306,
+            -0.016613613331623346,
+            -0.01894527592131826,
+            0.0008370014132002703,
+            -0.006785618888421608,
+            0.00919924251249592,
+            0.01816363936999796,
+            -0.010225961928154075,
+            -0.01604759761423727,
+            0.001577138242530474,
+            -0.003520564936698765,
+            -0.004395434431595494,
+            0.0052399043934902014,
+            -0.010127122101423486,
+            0.006456662777583096,
+            0.026560491731327886,
+            -0.013684536319659296,
+            -0.01425196172663986,
+            0.004084982328707292,
+            -0.0010682084059972982,
+            0.009758641913073978,
+            -0.00864879062601467,
+            -0.00481222050200678,
+            -0.014375330954464725,
+            -0.011378890549403524,
+            0.02505641780042516,
+            -0.02861164504722546,
+            0.0034060847953066354,
+            0.014937227531980433,
+            -0.00016904224334342915,
+            0.01541381851648916,
+            0.0033028855218525644,
+            -0.03615935765718852,
+            -0.013677774380513626,
+            -0.02073839691250734,
+            0.019821074228011413,
+            0.01688422012576042,
+            -0.0033731840788147442,
+            -0.013840993586441808,
+            -0.026149880178874887,
+            -0.007430563876740068,
+            -0.02466234818567362,
+            -0.020708282040996014,
+            -0.019276991448222817,
+            0.006137002799644838,
+            0.005162892426268572,
+            0.022881759407670583,
+            0.0025854880108157857,
+            -0.015058889448404231,
+            0.0029948608544278386,
+            -0.006722836436935071,
+            0.011845298212977357,
+            -0.02747944486818378,
+            0.015311272959213262,
+            -0.01610493464983465,
+            0.025239768880017402,
+            0.007425208902253236,
+            0.0010569008485112994,
+            -0.008822681490988575,
+            0.00010704167529046774,
+            0.014883787023445807,
+            -0.004186968199973372,
+            -0.0007781259002896265,
+            0.02787555889813697,
+            0.02087373813833492,
+            -0.0017783425135865238,
+            0.008743718448136432,
+            0.028026471206552342,
+            0.019081083283793984,
+            0.0325261942213039,
+            -0.013815526479760766,
+            0.005113676604826083,
+            -0.028873600114081546,
+            0.010510475843707614,
+            -0.010871039729515013,
+            -0.010201824683340651,
+            -0.0067946599549165735,
+            0.027756887504389347,
+            -0.007962601205446472,
+            -0.0006687976124826922,
+            0.03469894503563926,
+            -0.014185294847328063,
+            -0.00010543064621374546,
+            0.0007386418834840544,
+            -0.009759481309152999,
+            0.02742722730386853,
+            -0.0073590367291510046,
+            0.018456032721630843,
+            -0.019560882612141943,
+            -0.013705917411254969,
+            -0.011894560208199602,
+            -0.001142768182039817,
+            -0.02576754877141608,
+            -0.008755240066826174,
+            -0.02249287216087102,
+            0.00859172262856532,
+            -0.0046012388553094375,
+            0.018845920253446127,
+            0.0011617844393195566,
+            -0.005275799428018491,
+            0.029069367036565145,
+            0.0088080893110627,
+            0.018100411389202663,
+            -0.0004971948011851863,
+            0.01216566778850985,
+            0.02369448391321832,
+            0.002637001729688794,
+            0.005012948614448268,
+            -0.024213704079796478,
+            0.003420184574622472,
+            -0.0004359900462023086,
+            8.265488264269535e-05,
+            0.016267948861008616,
+            -0.019572032453100303,
+            -0.02216264534756073,
+            0.021871880626992544,
+            -0.012742926070754964,
+            0.011676769226089007,
+            0.0035362870996336194,
+            -0.010732229460038356,
+            0.0006838777488770268,
+            -0.03083438636386314,
+            0.008820746379709355,
+            -0.03216685858285627,
+            -0.032230808828686675,
+            -0.013547681710236494,
+            -0.009994106571421729,
+            0.0008758088338659712,
+            0.028447187234357855,
+            -0.00795317029726568,
+            -0.015797892954136448,
+            0.005363565557248085,
+            -0.011199252697624111,
+            -0.00817162221789202,
+            -0.0007867892785855576,
+            0.005856939999700809,
+            0.009461959355310273,
+            0.005997804620690314,
+            0.011648335019189696,
+            0.007491946658046298,
+            -0.008028800679169293,
+            -0.008837725220736191,
+            0.015889923814605387,
+            -0.0014409716087425222,
+            -0.01176424391171685,
+            0.030430130082762363,
+            -0.013300620185884817,
+            -0.0005592926632214984,
+            -0.0021975669766032472,
+            -0.0015687208316663597,
+            -0.0021652076181411244,
+            -0.014225289143358926,
+            0.01870568404189425,
+            8.334501002782455e-05,
+            -0.014288806218067678,
+            0.0032841743905975634,
+            0.003841780428153241,
+            -0.0036498339483411074,
+            0.008388922960269383,
+            -0.008147275785908849,
+            0.017348147498892,
+            -0.015113395971380741,
+            0.008740452410210708,
+            0.0038335161236289234,
+            0.0043482078469644385,
+            -0.0065755348640655865,
+            -0.01592809184384243,
+            0.009642593996701513,
+            0.012051872087663658,
+            0.02173264012652684,
+            -0.02525660485859878,
+            0.00876161653901566,
+            -0.005374533920781607,
+            -0.014534179652013885,
+            -0.016127866637369738,
+            0.00982303248417706,
+            0.022322910435388785,
+            0.0063086346898091296,
+            0.012139263756190026,
+            0.00013383660542132047,
+            0.005848259030010402,
+            -0.014767198851314148,
+            0.010570158368602513,
+            -0.011355774405066537,
+            -0.018439261453084426,
+            -0.005243587266408389,
+            0.028894120979627323,
+            -0.005492024421080143,
+            0.017938747701972086,
+            0.0021350711250093858,
+            0.029007356490801176,
+            -0.008686781541806358,
+            0.0014883465513564527,
+            0.0039866022469891155,
+            -0.01295271525889559,
+            -0.004272008629260297,
+            0.01976273843646234,
+            0.04882868638601926,
+            0.010204475902027572,
+            -0.0009155109545661691,
+            0.010069071220988797,
+            0.005626418787776309,
+            0.002148101019776153,
+            -0.00805781810841286,
+            -0.030431071067211577,
+            0.008883707378543252,
+            0.005628218215212441,
+            -0.01997825125961566,
+            0.019838234890736068,
+            0.026483365072033395,
+            0.023040623182975964,
+            -0.011585643598171132,
+            0.016539930468533123,
+            -0.010092529089936188,
+            0.0177008735686489,
+            0.028665424114962996,
+            0.013810061902648515,
+            -0.010487150104822674,
+            -0.0020847870330548534,
+            0.006329342599564054,
+            0.004186981876004016,
+            0.00026239408223817384,
+            0.015190614519766204,
+            -0.019155180136775043,
+            0.01005728016778505,
+            0.014725826697964637,
+            0.03322436221319799,
+            8.049930133627564e-05,
+            -0.01645191835557271,
+            0.0016249281996128196,
+            -0.009976155662939693,
+            0.00856399492362911,
+            0.000632427127231252,
+            0.0032242260692672124,
+            -0.009688119824276678,
+            -0.012682779503915694,
+            -0.002851918183031152,
+            0.009403205508474241,
+            0.0021188716361101698,
+            0.002759004011231831,
+            0.007641925150177999,
+            0.0022830062031146204,
+            0.01713024474384216,
+            -0.0011181980841546123,
+            -0.00029141889484139217,
+            0.004601090256674874,
+            0.010509199590505558,
+            -0.01649852835819883,
+            -0.017548369964263514,
+            0.004337237475053179,
+            0.012217494210546133,
+            -0.013159179948224596,
+            -0.013241981632059755,
+            0.015704543608959117,
+            -0.01913575130892795,
+            0.024934102868065366,
+            -0.00942381892138331,
+            -0.004543611336526587,
+            0.000343200128934031,
+            -0.00727538880611703,
+            -0.0038751118497522673,
+            -0.0030064279504724797,
+            0.003984715814505561,
+            0.0019261818110367147,
+            0.007472350699291607,
+            0.008472369788801792,
+            0.014568547873264247,
+            0.006043895352236181,
+            0.0032580830408778395,
+            0.009109104498825864,
+            -0.014523846294121576,
+            -0.014800649887278973,
+            -0.002704791598011488,
+            -0.008895394038700809,
+            0.00902016867266151,
+            -0.006607182154874112,
+            0.00845408382182461,
+            -0.015501867835442603,
+            0.015573642225745032,
+            -0.012200552219016517,
+            -0.0008900215712478553,
+            -0.009666233545144905,
+            -2.7470277092642562e-05,
+            0.008191697904514857,
+            0.014605287653054787,
+            0.0070552942843382215,
+            -0.031921405225964206,
+            0.0017403427534633655,
+            -0.008846568352320488,
+            0.03446319360336241,
+            0.002612606662691094,
+            -0.01060863123202008,
+            0.020352181356568008,
+            -0.01592124132155158,
+            -0.00027704668715660693,
+            -0.01602075523652062,
+            -0.004797557413047212,
+            0.00707939446796679,
+            0.016399017978990946,
+            0.02759049512664933,
+            -0.005697020356865688,
+            -0.007995305872076221,
+            0.006384047872862793,
+            0.029590696670334232,
+            0.012246955508890425,
+            0.024748351124415785,
+            -0.007633712004672087,
+            0.014348228716100414,
+            0.007773260552660029,
+            0.0076045488653348975,
+            0.004321689250440021,
+            -0.002591801797145718,
+            -0.011698574606499644,
+            -0.014974971814360001,
+            -0.0037563183330623384,
+            -0.016987915346053498,
+            0.00012247833530362867,
+            0.021344400276501117,
+            -0.00475351423959752,
+            -0.0033847161875730026,
+            -0.01874186437694691,
+            0.0033461670218611346,
+            -0.010029445302549983,
+            -0.00367421246775698,
+            -0.013133469163793961,
+            -0.0254122035745965,
+            -0.018556369080030412,
+            -0.002155244024999317,
+            -0.00929309186646207,
+            -0.015194269781263309,
+            0.0070354690219415535,
+            -0.016305117309271634,
+            0.0080837166473439,
+            -0.00448473417024141,
+            -0.016508775226236883,
+            -0.00958925666995396,
+            0.01100335153658195,
+            -0.0041819993634961685,
+            -0.01929493648733792,
+            -0.01282300765645703,
+            -0.011300260564838231,
+            0.0051733704121938884,
+            0.005996780699975213,
+            -0.0038111648304178544,
+            -0.0028279639348479987,
+            0.016640121845366295,
+            -0.010197365811240557,
+            0.009878730312873492,
+            -0.03335474019383245,
+            -0.03551826040406959,
+            0.004497847402931096,
+            0.01507007013703286,
+            0.0013085881889541252,
+            -0.02078680101065954,
+            0.010806315988966003,
+            -0.016499186313125556,
+            -0.00858013792516617,
+            -0.008522534127498645,
+            0.009649521088411411,
+            -0.007283003380127847,
+            0.01299542385123007,
+            -5.3226845309105124e-05,
+            0.030477683597373687,
+            0.0074153455615142755,
+            0.004161881195438522,
+            -0.00679081048132963,
+            0.008354912600159931,
+            -0.001621776416108105,
+            0.004080191054849395,
+            0.02120524874226696,
+            0.012440848502230418,
+            -0.00218041171518331,
+            0.006458415098595872,
+            0.005678055538564355,
+            -0.017186537377121235,
+            0.02507960402044456,
+            0.024678178106806115,
+            -0.008205288425409454,
+            0.004108402466376321,
+            0.005783335302169668,
+            -0.008846663924040241,
+            -0.006897707231869481,
+            0.006527795262026734,
+            -0.018474181903007085,
+            -0.0008721381678008483,
+            0.010598084358464908,
+            -0.02255441909004843,
+            -0.004840511019512733,
+            0.021745651362109403,
+            0.004631567631335321,
+            -0.018558972627958834,
+            0.02764788406651861,
+            -0.004942361238163609,
+            -0.0027290999934532494,
+            -0.010876135202650048,
+            0.01671497143541799,
+            -0.012173527207375091,
+            0.008443317346655462,
+            0.011021541861818043,
+            -0.006295639322287843,
+            0.003083685676540999,
+            0.008277206401503182,
+            -0.004504302913715969,
+            -0.0068075424927125676,
+            -0.0010508416962215848,
+            0.010167160518175114,
+            0.0036980815941666336,
+            -0.022704963865525812,
+            -0.006020073261002279,
+            0.02934227992246887,
+            -0.015082922782242507,
+            -0.0050916398557337705,
+            0.005376294716056329,
+            0.007709342013963906,
+            -0.003476236561444464,
+            -0.0023399991233097337,
+            -0.00787486645740744,
+            0.010356414979428607,
+            -0.015950083845400274,
+            0.02076360898078935,
+            0.025608263494550438,
+            0.024504236990840842,
+            -0.014814235990238182,
+            0.010828584133720797,
+            -0.011703992394890898,
+            -0.011047470905668532,
+            -0.015542835222173864,
+            -0.001658296303013357,
+            0.001970208852236451,
+            0.013107754130616248,
+            0.009666887546609378,
+            0.0235853683657408,
+            0.006473544089109194,
+            0.008973300312205858,
+            0.014118601565410305,
+            -0.003902202854013708,
+            -0.018303438244719996,
+            -0.014584226585460755,
+            -0.011977613285028946,
+            -0.01756345491020528,
+            0.0004563602312336091,
+            0.020461659996271687,
+            0.012417744207683839,
+            0.024136626592375227,
+            0.002119213701333967,
+            0.013759902247467922,
+            -2.9525392347047554e-05,
+            -0.000892319792935443,
+            0.00045783825646242434,
+            -0.005238483120312843,
+            0.019255756091120846,
+            -0.004599752813817219,
+            0.01656589522110913,
+            0.0006443773300684925,
+            0.007133940816771615,
+            0.013554761340256566,
+            -0.01704388669741371,
+            -0.003443100892108368,
+            -0.010029620309829285,
+            0.000941391966877521,
+            -0.019633111750884034,
+            0.01934571074127927,
+            0.010273330977233852,
+            0.0017121345993533963,
+            -0.018327189396014198,
+            -0.0029706385099949693,
+            0.03448635131998772,
+            0.019458999453011265,
+            0.0050734354363614525,
+            0.02632443373898792,
+            0.013018762352749236,
+            0.014443611411502168,
+            0.010153370950346075,
+            0.02356766253713842,
+            -0.023479811300626968,
+            0.02095255383200973,
+            -0.011590347850340786,
+            -0.007414136601134069,
+            -0.0035774888741646705,
+            0.0025141508183687115,
+            -0.013588691733097813,
+            0.00982877622088977,
+            -0.0017916539399655826,
+            -0.025386211797192578,
+            0.009164316443206016,
+            -0.008811052979979271,
+            0.010464442616928296,
+            -0.007774569252594923,
+            -0.0002806136360286729,
+            -0.0127084026190163,
+            -0.015184317235990752,
+            0.001259201880504347,
+            -0.009158429132281081,
+            0.023908150915412617,
+            -0.021210848953875233,
+            -0.01321384014436847,
+            0.013997502713857512,
+            -0.006869443694669701,
+            -0.011364239609894453,
+            -0.00374449980963666,
+            0.023673050288879224,
+            -0.009243341446512421,
+            -0.01359367519009039,
+            0.014464166953530989,
+            0.003834735113400469,
+            0.0059713317014342395,
+            -0.007508862694272452,
+            -0.008428705474135382,
+            -0.003280788036976103,
+            0.005998840043285335,
+            0.026615077617284676,
+            -0.014059930972065399,
+            -0.0032833264512892022,
+            -0.009189999799498163,
+            0.004118204537632102,
+            -0.02359889635281631,
+            0.02771075351384059,
+            -0.008871395303200375,
+            0.021705177758560575,
+            -0.0015924449308442096,
+            -0.006239188158271628,
+            -0.024231036074349822,
+            0.027522945599928003,
+            0.003683870966132199,
+            -0.010221180899173243,
+            0.0012768725460662862,
+            -0.052005292736885915,
+            0.016636109796947025,
+            0.008429895913163985,
+            -0.019507994429834353,
+            0.003697519981610208,
+            -0.004773584856005853,
+            0.018983847580558853,
+            -0.022966993228850697,
+            -0.014078986007109447,
+            0.008283766047449966,
+            -0.01153285124362746,
+            0.0031439180465649996,
+            -0.02527749501821793,
+            -0.005754537200570427,
+            0.0011760345167229269,
+            -0.005276936074280004,
+            -0.001705933750221904,
+            -0.020022761608132548,
+            -0.025972227850795115,
+            0.017666017379157135,
+            0.015820875579600418,
+            -0.002278262448951789,
+            0.01320818011561616,
+            0.008709899932124187,
+            0.00836573261091119,
+            0.016034263394940008,
+            0.007124509234103267,
+            0.0081473711608784,
+            0.045498533239850446,
+            -0.019846323044145485,
+            0.007371629414298193,
+            -0.0014854149174929534,
+            0.013951741898629716,
+            -0.030467924817555014,
+            -0.0055144480998344705,
+            0.0062802744565489945,
+            -0.0049373689340857915,
+            0.03510454786147091,
+            -0.017326094216003073,
+            0.01694160526735264,
+            -0.0026075809870445238,
+            0.0007420394265014909,
+            -0.02042003877666893,
+            -0.026156937726134134,
+            -0.014294938857971834,
+            0.0011857406889443987,
+            -0.0010047941261608975,
+            -0.011946583993069242,
+            0.006797001710574735,
+            0.004917161312925866,
+            -0.0020713379451245515,
+            0.015652523103715316,
+            -0.02952061566240878,
+            0.007356800041369564,
+            -0.010733578620243498,
+            0.020008504830359524,
+            -0.00990331048284697,
+            -0.0013319400098308093,
+            0.007236960885181047,
+            -0.004122868383988748,
+            0.013216043472071927,
+            -0.0038337251451538677,
+            0.004227115653354611,
+            -0.005221321656766545,
+            -0.01794343569722877,
+            -0.0016435261495179767,
+            0.021429008432348098,
+            -0.008349684096540855,
+            0.031864497493526205,
+            -0.01264743878037828,
+            0.009174982784686633,
+            0.010692990337462762,
+            0.0006628831036334152,
+            -0.005286285851702483,
+            0.009052971814211658,
+            0.0021431970274545893,
+            -0.008690520448278274,
+            0.006934310986093342,
+            -0.018349449937310143,
+            0.008494451785507706,
+            0.03394632739506833,
+            0.0018511328780472266,
+            -0.0007519661670070526,
+            -0.007266663179613392,
+            -0.01335844201823745,
+            0.018540713029383066,
+            0.005488273906733518,
+            -0.002546531152114063,
+            -0.027536660869590543,
+            0.02007746621157705,
+            0.0012265798861291223,
+            -0.004755325069578531,
+            0.012466557901411086,
+            0.0020495867632654885,
+            0.019989531151046736,
+            -0.010171678639113316,
+            0.030757957899447096,
+            0.010050036220124393,
+            0.002959833051227505,
+            0.0038556928367700113,
+            -0.036655749074851536,
+            -0.01652152778069314,
+            -0.00019357502472740266,
+            0.014271581054865031,
+            -0.012430647252826173,
+            -0.022432580814844155,
+            0.0010346169306977334,
+            0.01148004051449691,
+            -0.0033898642762881046,
+            -0.009650665305271706,
+            0.01560757541276278,
+            -0.020342213902670575,
+            0.006701302394408393,
+            0.007693672045981921,
+            0.0287100214318104,
+            -0.0020735384554022153,
+            -0.0012324838849990056,
+            0.000626416803904792,
+            -0.0055669860801192935,
+            -0.0008037366126870824,
+            0.017251369734470945,
+            -0.013661316947002466,
+            0.0175926211217534,
+            -0.012735241490550407,
+            0.0064713656749838585,
+            -0.0021092378432037563,
+            -0.010629579860050812,
+            -0.020114807971202787,
+            0.0029609495764307357,
+            0.009378003163375899,
+            0.006920144698428806,
+            -0.010438984881449953,
+            0.010715409812532968,
+            0.012197673689706352,
+            -0.00753962861484439,
+            0.014519905132928041,
+            0.022173989895384192,
+            -0.007687503607783572,
+            0.013633321414009471,
+            -0.0041446334728745095,
+            -0.014056264816568229,
+            -0.009985463824440833,
+            0.023929750617160402,
+            -0.0005563806942667597,
+            0.008756997373618809,
+            0.00388125051641897,
+            0.0007856447663040696,
+            -0.011226530517316183,
+            -0.001653447552601681,
+            0.019294448360333633,
+            0.011831934675073843,
+            -0.0013888534822008192,
+            -0.0010525764356699885,
+            -0.0014566936525653407,
+            -0.009427557739483988,
+            0.02049188741459728,
+            -0.022030783927622997,
+            -0.0019604940727076573,
+            0.00461018148439991,
+            -0.022589816803875618,
+            0.0001863117949313752,
+            -0.024570217962989905,
+            -0.01569952241187498,
+            0.011763033781122343,
+            -0.0022530631823737554,
+            -0.016028810867854907,
+            -0.0003242701040942633,
+            7.424127301806556e-05,
+            -0.007815789940508527,
+            -0.004135500768181258,
+            0.006115909992703774,
+            -0.00034193217510010117,
+            -0.0032262189374874842,
+            0.013430154310506884,
+            -0.009996622196870008,
+            0.015431528911575157,
+            0.0067029980830614145,
+            -0.015702167721800483,
+            0.0021406610839557003,
+            0.018500219544264753,
+            -0.0058910087864916085,
+            0.00787021425357441,
+            -0.014681907249972176,
+            -0.0037495551894426303,
+            -0.005787661351944086,
+            0.006051351977821761,
+            0.024849195780885833,
+            0.012969434450643372,
+            0.01139340445077682,
+            0.017405361843075933,
+            0.004325000715054734,
+            -0.01572297405297993,
+            0.0007416798945675923,
+            0.01682727189828363,
+            0.010464036837245312,
+            0.009804326468113703,
+            0.0008975556667684365,
+            0.0043704860361487635,
+            -0.005142176468800175,
+            0.009317064072855551,
+            0.02915482449687158,
+            0.0013434689612997423,
+            0.008696141010760192,
+            0.017276730345147134,
+            0.027038741396131265,
+            -0.0035088997194497725,
+            -0.007469614319788499,
+            0.008739942310300076,
+            -0.009657991604173173,
+            -0.001081629174976152,
+            -0.017000351183542026,
+            -0.012310275736750868,
+            0.016510505259358398,
+            -0.006084627062230492,
+            0.02848913486719956,
+            -0.0010655806791907225,
+            -0.02026498848361482,
+            -0.01565505608194376,
+            0.001404549479440637,
+            -0.01362132895080652,
+            0.005287566256051299,
+            0.014187265685427415,
+            0.01917046513020831,
+            0.004687191682875024,
+            -0.022258337721085656,
+            0.037364962197399806,
+            0.01757108963349427,
+            0.019377870795760123,
+            -0.019422840506328382,
+            -0.006159311237412833,
+            -0.009077095160676276,
+            0.02116697012944498,
+            -0.00880692095488465,
+            -0.009875674856043954,
+            0.024014332004451508,
+            0.017812606845656484,
+            0.005679806286464216,
+            0.006736609904079586,
+            0.0020048663439319585,
+            0.0025414630253661444,
+            -0.015831593851012047,
+            0.017957902498542212,
+            -0.005858125698347523,
+            -0.0058068554681761945,
+            0.017529514573571298,
+            -0.005989581792993925,
+            -0.0030306021203732347,
+            0.00015839997806822045,
+            -0.019697052145531482,
+            0.011618112886207201,
+            0.008256007601576028,
+            -0.017229774429089857,
+            -0.0014757560402567304,
+            -0.0002404480326945295,
+            -0.014118694740134205,
+            0.02004885146995733,
+            0.01694893356262835,
+            -0.0013248884139405398,
+            -0.0051752815372013315,
+            0.008223868147393114,
+            0.007000679132597393,
+            0.005256194890343006,
+            -0.019496098876977206,
+            0.005230597149342019,
+            0.005103909294824105,
+            -0.0014716649591412774,
+            0.010274346323051634,
+            -0.009030594825985526,
+            0.033740380586893826,
+            -0.006505788197604522,
+            0.013065658941136721,
+            -0.0043634731684193745,
+            -0.0055529072007985125,
+            0.001816020676232658,
+            0.013054468478598695,
+            0.009753501030700457,
+            -0.00970446155762939,
+            -0.004172354962129865,
+            0.018697982092775737,
+            -0.011983156146516074,
+            0.0013300791598046338,
+            0.01564072865804035,
+            -0.004858814041438713,
+            0.004708235083775013,
+            0.008635204790345423,
+            -0.004489505237783172,
+            0.01801512858970922,
+            -0.016482572999490366,
+            0.0023953436660054026,
+            0.010694869684962088,
+            0.02390864159830096,
+            -0.019152187046410703,
+            0.021786596486708207,
+            -0.017951050280323393,
+            0.026538582634436314,
+            0.014484894489694262,
+            -0.0035222259457311444,
+            -0.01946200885147006,
+            -0.00850690913394181,
+            0.011654565129063417,
+            -0.021423803876241532,
+            -0.0007985948078290641,
+            -0.0007290222944686444,
+            -0.021367628543092133,
+            -0.011189364017231132,
+            -0.009961779129778331,
+            0.010701637745810138,
+            0.001918633898887013,
+            0.010380983868185359,
+            0.008002182827196827,
+            0.010784760574518087,
+            0.014405640442760822,
+            0.0008958107769384412,
+            -0.010743001445440941,
+            -0.023795589003655273,
+            0.016987708698679597,
+            0.00438463274382093,
+            0.0007862212695139549,
+            -0.01627468994515667,
+            0.011936651269737248,
+            0.013250643815768723,
+            -0.009385605548926286,
+            0.009121882469425302,
+            -0.03716901103204463,
+            -0.004552808410398574,
+            -0.0035704811925430366,
+            -0.0020309197365173475,
+            -0.00908678158053104,
+            0.01559445415882485,
+            -0.013682507836005851,
+            -0.0009590922904844632,
+            0.00011696985057536548,
+            -0.0074756526903548915,
+            -1.5467890432254378e-05,
+            0.020649928837031005,
+            0.008438785939908428,
+            -0.0037382105257999757,
+            0.010376213224654074,
+            0.008826031303751312,
+            -0.013900359449287676,
+            -0.01527474712786084,
+            0.019035366035884273,
+            -0.018776918681908793,
+            0.0063098418544296165,
+            0.01677916431403883,
+            0.020778813096283988,
+            -0.02891862501830075,
+            -0.0007231469961304435,
+            0.022093939332021185,
+            -0.0012807635799950273,
+            0.006370841109790851,
+            0.008781853138377092,
+            0.006754082980602544,
+            0.0029102159062355374,
+            -0.003711173531879644,
+            -4.843770941622283e-06,
+            -0.003940474588075707,
+            0.007155539022862352,
+            0.004366433868349658,
+            0.036453728071874844,
+            -0.013661129348165794,
+            0.002541308805735001,
+            -0.0058838191638835155,
+            -0.00347401304572066,
+            -0.00593469688967835,
+            0.004407665856950772,
+            -0.010055727994798024,
+            0.004330827357536078,
+            -0.001570861535288093,
+            -0.0003731129422346693,
+            0.004883969598300302,
+            -0.001701673379646408,
+            0.007036892564844683,
+            -0.011863650867472625,
+            -0.0003979727750011258,
+            -0.020174851412609312,
+            0.021117284557875267,
+            0.00011666551282792189,
+            0.003662462447326642,
+            0.03723741246290771,
+            0.00476064510662738,
+            -0.00238355131307445,
+            -0.012880057949435798,
+            -0.013551338766321525,
+            -0.012986444917747435,
+            -0.020703821629987552,
+            -0.0036665673548525388,
+            -0.021006311026247164,
+            -0.0052061406234528915,
+            0.010362862594785442,
+            -0.011540184420787572,
+            -0.004023659825622843,
+            -0.015881589631889473,
+            -0.006531981517815174,
+            -0.03114211633687585,
+            -0.01124607488650473,
+            0.007493228366452009,
+            -0.019149285949049886,
+            -0.0039932870926865805,
+            0.026850121590606017,
+            0.001095360635760988,
+            0.01387696424607248,
+            0.0038058248676879625,
+            0.008117686785954085,
+            -0.0045937472017383955,
+            -0.0005669025036141417,
+            -0.01928244877716143,
+            0.003391980330529026,
+            0.008525354854576925,
+            -0.00361507909530282,
+            0.0070008404454106905,
+            -0.008835727374865623,
+            -0.016950017602980382,
+            0.0021654956715219517,
+            0.009334379529385142,
+            -0.006639140003365139,
+            -0.017587472740268683,
+            -0.011049929670847685,
+            0.010401189523318744,
+            -0.02283429470064187,
+            0.006257145380266984,
+            0.016439581169238607,
+            -0.009363030375624585,
+            0.004651575425738153,
+            0.019809183749755425,
+            -0.011810674576781702,
+            -0.005811328501528397,
+            0.008471992070998505,
+            0.013891259999721588,
+            0.0095606134736748,
+            0.02954498207274139,
+            0.0026596809804008996,
+            0.000561870602466584,
+            0.004889263753490895,
+            -0.004277061142504552,
+            -0.017423200644235665,
+            -0.006243787897080996,
+            0.0038507494787521856,
+            0.012873996671952184,
+            -0.009879417960879032,
+            -0.00048575277472139055,
+            0.01682604801673206,
+            0.004530939990187663,
+            -0.0033901248079550284,
+            -0.011868399508738274,
+            -0.020080331108085103,
+            0.0015580259874809898,
+            -0.01362200460317948,
+            0.015354347040143093,
+            0.003390601069331352,
+            0.0015284610287850575,
+            -0.004239880336806429,
+            0.01828645373058079,
+            -0.005013418527935815,
+            0.012749833447770729,
+            -0.006148890994661865,
+            -0.004587831231448011,
+            0.007986533783001116,
+            0.009166578151006778,
+            0.0004939857995863664,
+            0.05421019830078428,
+            0.002838056604877719,
+            0.01333205648597435,
+            0.013419127695838135,
+            -0.014523733369484934,
+            0.014086525147734823,
+            0.012688569853552613,
+            0.018497344547821693,
+            0.013037819487160016,
+            -0.012353776077273462,
+            0.017354801119939964,
+            0.0024084723134891487,
+            0.003909681065813095,
+            0.0022202438064449653,
+            -0.004528576079628071,
+            0.019594683950225115,
+            0.004070222328547378,
+            -0.0074362443729372105,
+            0.004453618452623641,
+            0.005007543168224267,
+            -0.013105524222311019,
+            -0.0112958915514782,
+            0.027105208848863218,
+            -0.01219417557128091,
+            -0.018749971743148304,
+            0.006706594269933445,
+            -0.02008458683853808,
+            -0.03763686535076864,
+            -0.0021332074289244973,
+            0.009992361098878196,
+            0.002795183524495193,
+            -0.00253695604043486,
+            0.01614298064601032,
+            0.016730443047719683,
+            -0.016985296937992726,
+            -0.018245902728910347,
+            0.0025257780765481825,
+            0.0016333325293570182,
+            0.01861807236979066,
+            -0.023168663546030196,
+            -0.006378864850313035,
+            -1.3795465857380962e-05,
+            -0.006829008132963306,
+            0.004850306884033312,
+            0.004703280112191218,
+            -0.0071714759384722965,
+            -0.0027773519096287217,
+            0.002256846645341137,
+            0.01975383140755998,
+            0.013198754584066554,
+            0.01572971337890945,
+            0.002600741959577911,
+            -0.005033508062449482,
+            -0.016817359058724685,
+            -0.005450011991586895,
+            -0.013478330855299777,
+            -0.009091854345715177,
+            -0.003042052092498012,
+            0.008217376329142566,
+            -0.025053469059115735,
+            -0.011965083373220982,
+            0.0119294363235832,
+            0.00391377331459168,
+            0.0003099203772539202,
+            0.007607672728136279,
+            -0.003830132495364688,
+            0.010331053246451333,
+            0.013823890473448383,
+            0.0076112887952959766,
+            -0.004723377809559611,
+            0.009600411526173948,
+            0.011460781436250457,
+            0.006015477068316837,
+            -0.005072711638895421,
+            0.023859899048048694,
+            0.01821920335673539,
+            -0.01680365156728093,
+            -0.002149107383573353,
+            -0.0027829676356423997,
+            -0.01884308738652242,
+            0.009665830801645276,
+            0.00880322398336952,
+            0.008308268605449346,
+            0.003906369351868086,
+            0.00974531700889854,
+            -0.013875904819275925,
+            -0.0050570811397453235,
+            0.002870120865445989,
+            -0.005729556771404781,
+            -0.01901337075686543,
+            0.000297242372701423,
+            -0.011987192261104132,
+            -0.011327157550660895,
+            -0.01409710922165158,
+            -0.003221240464411861,
+            0.005283472511900512,
+            0.021472187640986972,
+            -0.004410881709548286,
+            -0.001855852749039012,
+            0.01351697599517034,
+            0.010060125186506398,
+            0.0021658738823899905,
+            0.0002206474541239167,
+            0.005418828324605768,
+            -0.0027919130944783397,
+            0.0008094042430809068,
+            -0.0015826797905407809,
+            0.021089276765434876,
+            -0.02016611086095107,
+            -0.009413977569213423,
+            -0.009288078098692112,
+            0.005842197094339927,
+            0.028793032462603636,
+            -0.007165634502373454,
+            -0.010104002083017078,
+            -0.01994315293556678,
+            0.0003786527922888834,
+            -0.07810609175343784,
+            -0.005210636756258548,
+            0.001870513618910238,
+            0.03126380113105983,
+            -0.010817888476436834,
+            0.0033673768058629823,
+            -0.002204487854177595,
+            0.014542299655712928,
+            -0.001760107965206874,
+            -0.0013283746982194259,
+            0.0071954806737858056,
+            -0.005640400818010697,
+            -0.006210989097047217,
+            0.005904163661619133,
+            -9.945941028848348e-05,
+            -0.010266938705916303,
+            0.00057263080955709,
+            0.02209498395848068,
+            0.0159398346608872,
+            -0.006575439759804175,
+            0.01816748552021211,
+            -0.00719294553484007,
+            -0.009838738592324234,
+            0.005639288780502127,
+            -0.0019007683459126327,
+            -0.015369619176147573,
+            -0.011475977270803784,
+            0.009862808084641291,
+            -0.030069890644150284,
+            -0.00010725753918890802,
+            0.00943698424646706,
+            -0.013540245857270492,
+            -0.008293042205008774,
+            0.007212974960781723,
+            0.006708343474532583,
+            -0.0012017968151025369,
+            -0.0078026000239507145,
+            -0.02527526731750743,
+            0.0027578400032925625,
+            -0.002446811971857853,
+            -0.004899702349784173,
+            0.0019960465173086183,
+            0.001479503476855694,
+            -0.012645905858466798,
+            0.01216748847452763,
+            -0.013378877368040743,
+            -0.012053251571418766,
+            0.012508842412556306,
+            -0.00027167241745418584,
+            0.01682680930386839,
+            -0.020607299598265426,
+            0.0010702151063620052,
+            -0.028075439824932304,
+            -0.007671848153932823,
+            -0.015486630894165054,
+            -0.0009776136330459065,
+            -0.0012939921569540803,
+            -0.03481880531471923,
+            0.012889447665794033,
+            -0.005200149829198305,
+            -0.006094702500154477,
+            -0.00459778303079732,
+            -0.024583504817899775,
+            0.006072281238752663,
+            0.00607350167867964,
+            -0.006506508970816635,
+            -0.016816439635445445,
+            0.008428048142588516,
+            -0.0011040593013238872,
+            0.004873670535200424,
+            -0.007475800310154908,
+            -0.0005638920132076605,
+            -0.013047135093134946,
+            -0.005463454269764494,
+            -0.0008522562324122834,
+            0.01827123304871433,
+            0.0044055851550317835,
+            -0.014526381222031981,
+            0.016862445236481526,
+            0.013250692156820124,
+            0.01602714293656341,
+            -0.0071089293427753115,
+            -0.0004265739315250755,
+            -0.02750630108787212,
+            -0.00327225092954903,
+            -0.023687644154808972,
+            0.00041213204365507744,
+            0.009447660230507965,
+            0.0203905264889617,
+            0.012694421477479161,
+            0.004620617778996077,
+            -0.001557511764988156,
+            0.0066751090724502,
+            0.02696959560423252,
+            -0.01099183126742224,
+            -0.006969111598423376,
+            -0.01884504192789845,
+            0.01113023717919988,
+            0.025135748541199255,
+            0.003381303139273845,
+            -0.011317957771935042,
+            0.008112598754655146,
+            -0.0009247845526290428,
+            -0.010302980598863016,
+            0.02889587382004902,
+            -0.0391134481322538,
+            -0.015599591108049343,
+            -0.004605824192962023,
+            0.007406520608110251,
+            -0.015596917115435033,
+            0.00770468362805995,
+            -0.012614339656105264,
+            0.001699049329817941,
+            -0.005851477164035241,
+            0.03535087340659861,
+            -0.0031950960727572925,
+            0.0006909856233673865,
+            -0.003395233050301427,
+            0.003531135044131077,
+            0.012355948349937309,
+            -0.004444221317023453,
+            -0.030726300006909638,
+            -0.0022983766869702346,
+            -0.011198252640110788,
+            0.009274666307515955,
+            -0.016207714291658314,
+            -0.017342435976846468,
+            -0.009424737744066259,
+            -0.021827243671837953,
+            -0.012403084982789827,
+            0.029747453318486338,
+            -0.005824001042844527,
+            0.01747569398142269,
+            -0.005219060548639168,
+            -0.008365003057800934,
+            0.00021272411948388247,
+            0.01344335658390732,
+            -0.028498240040284714,
+            0.00041963499785238845,
+            -0.002322931702641349,
+            0.0126393550661767,
+            -0.001595134597082556,
+            -0.00983073906977773,
+            0.006903313245694627,
+            0.0010522208685011098,
+            -0.015648940549713764,
+            -0.0007573245570947632,
+            -0.008533504878052644,
+            0.020475329445149498,
+            0.00027791387213409814,
+            0.001074213629646005,
+            0.013864331671732542,
+            -0.02102352993295736,
+            0.01576938071930521,
+            -0.021722775924283996,
+            0.01893267442161547,
+            -0.006577103101038035,
+            0.005772016664149475,
+            0.008283223343661555,
+            0.014068789332250016,
+            -0.014030042849491258,
+            -0.00663752723080222,
+            -0.034436220972634726,
+            -0.0122621879114314,
+            -0.001946072976826902,
+            -0.005308941654794638,
+            -0.014773123927909993,
+            -0.0011054822266082056,
+            -0.026205461509573584,
+            -0.006460373065662098,
+            0.012822188046808264,
+            0.019900312098359022,
+            -0.008409070094265829,
+            0.014163132434889265,
+            -0.006743711872055091,
+            -0.0028230771810882492,
+            -0.012652021749331811,
+            -0.018061180380009395,
+            -0.01083672324723357,
+            -0.01890966636827258,
+            -0.0018635408070944168,
+            0.004709222372611061,
+            -0.015760618199235518,
+            0.011435131737642627,
+            0.03198874888333405,
+            -0.01567271472650171,
+            0.008243406188694909,
+            0.016411496328333205,
+            0.0034236971031973503,
+            0.011728079139322837,
+            0.0038311415233052795,
+            0.004621789116267879,
+            -0.011109450058055078,
+            -0.011473158133621661,
+            0.006524490579223986,
+            0.00954511918454185,
+            -0.004913142601134428,
+            0.024347072255007807,
+            0.0008745419206472714,
+            0.005045794664611216,
+            0.005304506902446091,
+            0.011488187043525824,
+            -0.013135271908307747,
+            0.019746484188715552,
+            -0.002055210086078267,
+            -0.015571665869889405,
+            -0.01535499169234956,
+            -0.01105419264826655,
+            -0.018626399757822328,
+            0.0017108083455869296,
+            0.0181439079855921,
+            -0.0012493542718277685,
+            0.012137753522284387,
+            -0.00924460959150499,
+            -0.005019437549649743,
+            -0.0027156377206998474,
+            -0.038211954658821295,
+            -0.0021878970332906457,
+            -0.018809487178512628,
+            -0.02498945889484935,
+            0.03505216568400238,
+            -4.26477081213836e-05,
+            0.00997643255528116,
+            -0.010310197159232386,
+            -0.0002454902098986129,
+            -0.011065403358802515,
+            0.014954998647970473,
+            0.004624123403100598,
+            0.008872923678983478,
+            0.008560280683876702,
+            -0.016419500515801093,
+            -0.002997557798375716,
+            0.007472390947008284,
+            -0.00374822278027057,
+            0.012680407710694952,
+            0.015353684671120731,
+            0.020886474047003733,
+            0.011161262397502876,
+            0.002045862158732102,
+            0.00887271411363814,
+            0.009563361150297699,
+            -0.0003513237086717377,
+            -0.010459621030424415,
+            0.015419039131459582,
+            -0.02204848739950703,
+            0.012172257374410352,
+            0.04346634265794738,
+            0.019197433209794286,
+            0.029488039318675732,
+            -0.02272810439246353,
+            0.01448092574459596,
+            -0.003801043409441206,
+            0.009476229966122152,
+            0.0049309968883087685,
+            0.02346191840355209,
+            0.003973840862678076,
+            0.0073546443568171405,
+            0.010647581332452525,
+            -0.00972051412996416,
+            0.025644022498404723,
+            -0.00846669916801108,
+            5.1518540791212095e-05,
+            -0.004407265804145216,
+            -0.026400029105519137,
+            0.01615193008014211,
+            0.0081722536401398,
+            0.0010702314763197463,
+            0.0010263623479474385,
+            -0.0019566604229607643,
+            0.007982949510899019,
+            0.007079322685771456,
+            -0.015274148711633536,
+            -0.003886053737590683,
+            -0.018527455865296014,
+            0.0012864569064749438,
+            -0.024337133883771334,
+            0.010466651250597847,
+            -0.017568706814132484,
+            -0.014822354159709854,
+            -0.021666681078831483,
+            -0.013232016867264097,
+            -0.02011653224168991,
+            0.0035858355517491515,
+            -0.008632755352062642,
+            0.013025728891203198,
+            -0.016962939287139683,
+            -0.021494971929779273,
+            0.008430887589398292,
+            0.00870802359711456,
+            0.007481282677955534,
+            0.009728368603458082,
+            0.014996157358749115,
+            0.01736292711410622,
+            -0.022712386143092265,
+            -0.023163001611599246,
+            -0.006682209717364464,
+            -0.003531495447167166,
+            0.006762379833125297,
+            0.00933681196321128,
+            0.012884185500343249,
+            -0.007077895849501418,
+            -0.007554762054641417,
+            -0.013283067570021503,
+            0.006197966297581871,
+            -0.003993067576677973,
+            -0.009464789349849612,
+            -0.0015662739825834614,
+            -0.0061389795593180686,
+            0.002753664732282926,
+            -0.0001791088888074743,
+            0.0007595261237141144,
+            0.017122834676182343,
+            -0.008176319370751018,
+            -0.009787747261447871,
+            -0.02027228453943018,
+            -0.010683273300243025,
+            -0.015604745521156111,
+            -0.0016279406232527138,
+            -0.006506713191861036,
+            -0.014420323628589382,
+            0.010888206569219648,
+            -0.011930708571093279,
+            -0.00887210117616784,
+            0.010807147705360856,
+            0.0073900910661282935,
+            -0.003431679006289002,
+            0.037804020313161775,
+            0.0011945597874820288,
+            0.00778778394048267,
+            -0.003366543306746973,
+            0.005296727102519836,
+            -0.017761251541598876,
+            -0.026382400396890007,
+            0.0010357716464853195,
+            -0.01775777668381489,
+            0.010807713750049491,
+            -0.008515368407459745,
+            0.0016051389544388124,
+            -0.00034958600424912995,
+            -0.009585854918521076,
+            0.0036549720706351473,
+            -0.008110927907950148,
+            0.004212219431447508,
+            -0.03507890639477636,
+            0.011409377585617926,
+            -0.01819120811524444,
+            -0.008962242411035718,
+            0.01710177594046791,
+            0.037560936862121835,
+            0.0028513388998398104,
+            0.00030058353811717895,
+            0.0014792184333546447,
+            0.004427921639326168,
+            0.021438127785303025,
+            0.006339483221420334,
+            -0.001187747087595264,
+            -0.02237982599829809,
+            0.0062435650294483695,
+            0.002200578793697352,
+            0.010092828474934744,
+            0.023105023225115266,
+            -0.016797326021070008,
+            -0.015378520287395565,
+            -0.012732527281000097,
+            -0.010797463778182385,
+            0.009881499418260473,
+            0.02801767062785128,
+            0.01811139137849365,
+            -0.017358098300028903,
+            -0.020207173948076473,
+            0.0016242901745373126,
+            -0.026331540602868762,
+            0.007269751661479859,
+            0.11940631114242901,
+            -0.0045382135734049305,
+            -0.005994407567723939,
+            -0.008679712231646732,
+            -0.004482553208064535,
+            0.016709571294385293,
+            -0.013387043582971358,
+            0.069179848974036,
+            -0.007221545718220735,
+            0.0037392229364642915,
+            0.0028916224964057385,
+            0.00462729158956839,
+            -0.009727339449635306,
+            0.011570145487622916,
+            -0.011287365791681587,
+            -0.012991994493845187,
+            -0.013076194601468383,
+            -0.0180830261130804,
+            -0.007272768410218907,
+            -0.004155564814758654,
+            0.004204993170487392,
+            -0.005525575983182816,
+            0.008051108079520902,
+            -0.017551645696802648,
+            -0.010800070994485792,
+            0.013007176804761548,
+            -0.0023627485852619817,
+            0.005390986239687555,
+            -0.029668390278675554,
+            -0.0007650391259819867,
+            -0.0002446394046947245,
+            0.021280292985495608,
+            0.003137771886236958,
+            -0.0019077298815862946,
+            -0.010810159675564049,
+            0.007040107942856957,
+            0.023835993694726953,
+            0.021875366566457236,
+            0.0180511566130312,
+            0.012135351541044354,
+            0.020483975362807574,
+            -0.008667079096167644,
+            0.026324312888512866,
+            -0.001552207146726937,
+            -0.00397810465151493,
+            0.0077577136352257004,
+            -0.009849600842081875,
+            -0.00816392268697345,
+            0.00677353736462267,
+            -0.017288900282504148,
+            -0.020418071729964713,
+            -0.020383762873109595,
+            0.025741408097358233,
+            -0.011755094311186617,
+            0.002737129201159793,
+            -0.008266210469668866,
+            0.027287436144516414,
+            -0.006283892307996516,
+            -0.0029440642954701632,
+            0.009428876742171362,
+            -0.03788574758692254,
+            0.008834605607057885,
+            0.006463342392514337,
+            0.026711196829464153,
+            -3.9827332346130654e-05,
+            0.005565561841469247,
+            -0.0006738085601962865,
+            -0.006104320995853506,
+            -0.010792830169647851,
+            2.4565453208679464e-05,
+            0.017983908748172493,
+            0.0035181716362611077,
+            0.016970206528324653,
+            -0.005449808667944701,
+            -0.023171388778833197,
+            0.007092509502738752,
+            -0.01555320330471131,
+            -0.023306285898839978,
+            -0.002028783456704567,
+            -0.03196436161255752,
+            0.0033412618908711163,
+            0.007527633574188688,
+            -0.01890570959143866,
+            -0.017859735533232564,
+            0.010329340599904344,
+            0.004382396539605985,
+            -0.0016746397858953155,
+            0.006643785064338525,
+            -0.03307812909454471,
+            -0.01927475592253288,
+            0.004686970010943116,
+            0.012733525168279355,
+            -0.012993441126970916,
+            0.015291832952199082,
+            -0.020491167895268936,
+            0.016451976170508816,
+            -0.012259387189535237,
+            0.018393274368854132,
+            -0.006997668664226754,
+            0.011843450716200625,
+            0.011457415642860215,
+            0.009560877634897626,
+            -0.01790001115302499,
+            0.0214879415944361,
+            0.0066829070506954115,
+            0.03900547021123392,
+            0.02512484499576633,
+            -0.012651294056575184,
+            -0.0018425680226742782,
+            -0.00011361458320146007,
+            0.014982647266052349,
+            0.0061874424115559,
+            -0.003689464024837091,
+            0.02705902205734107,
+            0.018239469373522537,
+            -0.021070704644859155,
+            0.020037508334843183,
+            -0.00973804771680803,
+            0.018876283979591423,
+            -0.013303080480396818,
+            0.010592424494628194,
+            0.004717870260467274,
+            0.01070131238869639,
+            0.009087656015197565,
+            0.006708673669681232,
+            0.025928065153829043,
+            0.03663367449576964,
+            -0.00511073108390551,
+            0.002161381031392467,
+            0.004864092591944181,
+            0.012814518637357098,
+            0.001556833001747549,
+            0.007915766323215207,
+            -0.007012785514562176,
+            0.005834786334750152,
+            -0.002311533032619821,
+            -0.00122162928193159,
+            0.019525999280692766,
+            0.000824163362963678,
+            0.012875575199417254,
+            0.013259376059185821,
+            -0.002390133687709065,
+            -0.005682644446302278,
+            0.0019262218417782657,
+            -0.018193478045923098,
+            -0.015131696847666434,
+            0.029463723011600933,
+            -0.011447039651934148,
+            -0.008083114389964213,
+            0.0020642857121204753,
+            0.012160202510990566,
+            -0.00866944494225225,
+            -0.005378300635997713,
+            0.0038280364584418335,
+            0.014427318067525318,
+            -0.01799358399064332,
+            0.003381165206832161,
+            0.005252552821854107,
+            0.019433554824232185,
+            -0.00028791985674147885,
+            -0.013487678738391956,
+            0.0005317922815671783,
+            0.014982152801567637,
+            0.011902348165369078,
+            0.0016056390299304837,
+            -0.006935777197137747,
+            -0.00446584737370504,
+            0.004586814866182051,
+            0.006269466956110759,
+            0.0027516492952993445,
+            0.008157662045793032,
+            -0.0025001885993292637,
+            -0.016648747436203294,
+            0.017735757469839637,
+            -0.0030669873770852725,
+            0.018058248627079793,
+            -0.0038238312452176745,
+            -0.0147172470474118,
+            0.0043328417868095355,
+            -0.001888507803054499,
+            -0.018556943283494755,
+            0.009407930200720534,
+            0.002560949424273885,
+            0.010411159836451958,
+            0.0052571230238294825,
+            -0.007499470181994448,
+            -6.145889479758644e-05,
+            -0.015163819825329975,
+            0.022836682800304944,
+            0.016390944816242627,
+            0.001713572907204073,
+            -0.0035687505973373878,
+            -0.031210299824387475,
+            0.0029076511444359886,
+            -0.007822231980296341,
+            0.017821806420465356,
+            0.008250605126658402,
+            0.007908773724009905,
+            0.008351355849444162,
+            -0.00743970642051451,
+            -0.002411981861500092,
+            0.012543476434788293,
+            0.0202407777715137,
+            -0.0024635818448627447,
+            0.02535771413579989,
+            -0.012533449594824386,
+            -0.0022489292647974992,
+            0.0012362917358624392,
+            0.01578939075186126,
+            -0.0008001783766371555,
+            0.006311837812042848
+          ],
+          "keyphrases": [
+            "Switch Transformers",
+            "Mixture of Experts",
+            "sparsely-activated model",
+            "training instability",
+            "multilingual settings"
+          ],
+          "title": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "6df469f9-914b-4652-b189-540abdf1161a",
+        "properties": {
+          "page_content": "E Relation of Upstream to Downstream Model Performance 32 ",
+          "embedding": [
+            -0.018082372844219208,
+            0.023730995133519173,
+            -0.010728987865149975,
+            0.047936949878931046,
+            -0.03446846455335617,
+            -0.022373968735337257,
+            6.639356433879584e-05,
+            0.06883515417575836,
+            -0.040982190519571304,
+            -0.011789164505898952,
+            -0.011102170683443546,
+            -0.019473325461149216,
+            0.0032950290478765965,
+            -0.020541982725262642,
+            -0.018150225281715393,
+            0.06174468994140625,
+            -0.013655075803399086,
+            -0.0006122520426288247,
+            0.0013835305580869317,
+            0.010686581023037434,
+            -0.011840052902698517,
+            -0.03555408492684364,
+            -0.012162347324192524,
+            0.02306944504380226,
+            -0.017946671694517136,
+            0.0030490681529045105,
+            0.011161540634930134,
+            -0.005020997021347284,
+            -0.02865021489560604,
+            -0.0051015703938901424,
+            0.03867524489760399,
+            0.03236507251858711,
+            -0.004855609033256769,
+            -0.017844894900918007,
+            -0.03636830300092697,
+            0.01401977613568306,
+            0.01845555566251278,
+            -0.04328913614153862,
+            -0.03602904453873634,
+            0.020287541672587395,
+            -0.04037152975797653,
+            0.025851348415017128,
+            -0.025495128706097603,
+            0.022017749026417732,
+            0.03172048553824425,
+            -0.010008067823946476,
+            0.0009758926462382078,
+            0.005945470649749041,
+            0.01615709252655506,
+            -0.010118326172232628,
+            -0.01730208285152912,
+            -0.0348077192902565,
+            0.03915020450949669,
+            0.004249188117682934,
+            -0.02006702497601509,
+            0.01911710575222969,
+            0.04488363862037659,
+            0.1082228347659111,
+            -0.006721520330756903,
+            0.028073478490114212,
+            0.05166877061128616,
+            -0.012043607421219349,
+            0.002571988618001342,
+            -0.03602904453873634,
+            -0.02803955227136612,
+            -0.012798452749848366,
+            -0.06045551598072052,
+            0.016767755150794983,
+            0.011365094222128391,
+            0.017760079354047775,
+            -0.02951531857252121,
+            0.020253615453839302,
+            0.02417202852666378,
+            0.04423905164003372,
+            -0.014910325407981873,
+            0.030634865164756775,
+            0.02975279837846756,
+            0.021084794774651527,
+            -0.006059969775378704,
+            0.02388366125524044,
+            0.0036957759875804186,
+            0.010949504561722279,
+            0.013510892167687416,
+            0.015546430833637714,
+            0.022730188444256783,
+            -0.011585610918700695,
+            -0.027547631412744522,
+            -0.06934403628110886,
+            0.037114664912223816,
+            0.01806541159749031,
+            -0.023188184946775436,
+            -0.007624790538102388,
+            -0.004177096299827099,
+            0.005703750532120466,
+            -0.023001592606306076,
+            0.04657992348074913,
+            0.020525021478533745,
+            0.0024362860713154078,
+            0.011721313931047916,
+            0.04043937847018242,
+            -0.0042152623645961285,
+            0.03552015870809555,
+            0.005071885418146849,
+            0.030007241293787956,
+            0.01556339394301176,
+            -0.0288537684828043,
+            -0.007315219379961491,
+            0.0028221902903169394,
+            0.003935375716537237,
+            0.10937631130218506,
+            -0.014002813957631588,
+            0.006509484723210335,
+            -0.0004625020665116608,
+            0.015554912388324738,
+            -0.024850541725754738,
+            -0.01725119538605213,
+            0.00812519434839487,
+            -0.037352144718170166,
+            0.04885294288396835,
+            -0.022153452038764954,
+            0.013748371042311192,
+            0.01494425069540739,
+            0.005110051482915878,
+            0.016479386016726494,
+            -0.005339049734175205,
+            0.048513684421777725,
+            0.015936575829982758,
+            -0.001556339324451983,
+            0.0006970661925151944,
+            0.005203347187489271,
+            0.01067809946835041,
+            -0.008668004535138607,
+            -0.06221964955329895,
+            -0.0252746120095253,
+            0.0008497316157445312,
+            0.007446681149303913,
+            0.018438592553138733,
+            0.0020864277612417936,
+            0.011458389461040497,
+            -0.008205767720937729,
+            0.00984692107886076,
+            -0.05506133660674095,
+            0.020474132150411606,
+            -0.02403632551431656,
+            0.01446929108351469,
+            0.04261062294244766,
+            0.0181162990629673,
+            0.06048944219946861,
+            -0.0016040473710745573,
+            0.00034826804767362773,
+            0.0008396599441766739,
+            0.00017042340186890215,
+            0.026071865111589432,
+            0.004707184620201588,
+            0.03619867190718651,
+            -0.004170734900981188,
+            -0.013841667212545872,
+            0.004982830490916967,
+            -0.0059836371801793575,
+            0.018862662836909294,
+            0.0063313753344118595,
+            0.0233578123152256,
+            0.034587204456329346,
+            0.006967481225728989,
+            -0.010627211071550846,
+            -0.0009297749493271112,
+            0.022051675245165825,
+            -0.023815808817744255,
+            -0.023289961740374565,
+            0.02822614461183548,
+            0.028718065470457077,
+            -0.03390869125723839,
+            0.01740385964512825,
+            0.01787881925702095,
+            -0.010499989613890648,
+            -0.01446929108351469,
+            -0.013595705851912498,
+            0.018252002075314522,
+            -0.012722120620310307,
+            0.0033459176775068045,
+            -0.046885255724191666,
+            -0.06876730173826218,
+            -0.02227219194173813,
+            -0.048751164227724075,
+            0.007815622724592686,
+            -0.004187698010355234,
+            0.06045551598072052,
+            0.005635899491608143,
+            -0.013010487891733646,
+            0.013476965948939323,
+            0.02454521134495735,
+            0.018557332456111908,
+            0.026258455589413643,
+            0.008621356450021267,
+            -0.022187378257513046,
+            -0.0016793198883533478,
+            0.0016931021818891168,
+            0.03494342416524887,
+            0.016029872000217438,
+            -0.0022645373828709126,
+            -0.0023875178303569555,
+            -0.020626798272132874,
+            0.01029643602669239,
+            -0.002057802863419056,
+            0.024613061919808388,
+            0.007561180274933577,
+            0.04990463703870773,
+            0.038302063941955566,
+            -0.024731801822781563,
+            -0.04715665802359581,
+            -0.014774622395634651,
+            -0.041083965450525284,
+            -0.04542645066976547,
+            0.03548623248934746,
+            -0.012009681202471256,
+            0.03867524489760399,
+            -0.006895389407873154,
+            0.03418009728193283,
+            -0.03562193736433983,
+            -0.013264930807054043,
+            0.0021479178685694933,
+            -0.01122939120978117,
+            0.024765728041529655,
+            0.012425270862877369,
+            -0.036877185106277466,
+            0.011424464173614979,
+            0.011483834125101566,
+            -0.015376802533864975,
+            0.015453135594725609,
+            -0.009134482592344284,
+            -0.00803613942116499,
+            0.036266524344682693,
+            -0.0024256843607872725,
+            -0.017174862325191498,
+            -0.05119381099939346,
+            -0.00827361922711134,
+            -0.021814195439219475,
+            0.006585817784070969,
+            -0.022577522322535515,
+            0.020100949332118034,
+            -0.01032188069075346,
+            -0.01535135880112648,
+            -0.03378995135426521,
+            0.001905137556605041,
+            0.04837798327207565,
+            -0.003979903180152178,
+            -0.010118326172232628,
+            0.06822448968887329,
+            0.02451128512620926,
+            0.00183834636118263,
+            0.01592809520661831,
+            -0.022781075909733772,
+            0.010457582771778107,
+            -0.012611862272024155,
+            0.03602904453873634,
+            0.03643615171313286,
+            0.003526147687807679,
+            -0.008646801114082336,
+            -0.025732608512043953,
+            -0.00999110471457243,
+            0.002476572757586837,
+            -0.004346724599599838,
+            -0.04895471781492233,
+            0.018268965184688568,
+            -0.005572288762778044,
+            -0.006488281302154064,
+            0.03667363151907921,
+            0.022679299116134644,
+            -0.004376409575343132,
+            0.002985457656905055,
+            0.02875199168920517,
+            0.04389979690313339,
+            -0.0567236952483654,
+            -0.055807702243328094,
+            0.016377609223127365,
+            0.011186984367668629,
+            -0.022611448541283607,
+            -0.02164456807076931,
+            -0.001569061540067196,
+            0.01508843433111906,
+            0.004217382986098528,
+            0.022085601463913918,
+            -0.011314205825328827,
+            0.003244140651077032,
+            -0.010169214569032192,
+            -0.008358432911336422,
+            0.022611448541283607,
+            0.01256097387522459,
+            -0.04423905164003372,
+            -0.0008041440160013735,
+            0.01435055211186409,
+            0.009261703118681908,
+            -0.028429698199033737,
+            0.01769222877919674,
+            0.0023853974416851997,
+            0.011585610918700695,
+            0.022340042516589165,
+            -0.011882460676133633,
+            0.025851348415017128,
+            0.010499989613890648,
+            -0.0031190398149192333,
+            0.011670424602925777,
+            0.008214249275624752,
+            -0.022899815812706947,
+            -0.02259448543190956,
+            -0.010898616164922714,
+            -0.00786651112139225,
+            0.004537556320428848,
+            1.3956232578493655e-05,
+            -0.01950725167989731,
+            0.029549244791269302,
+            -0.01735297217965126,
+            -0.02132227271795273,
+            -0.037216443568468094,
+            -0.011899422854185104,
+            -0.010262510739266872,
+            0.02904035896062851,
+            0.03738607093691826,
+            -0.013799259439110756,
+            0.004073198884725571,
+            0.009482220746576786,
+            0.011068244464695454,
+            -0.0639159306883812,
+            0.053093649446964264,
+            0.022730188444256783,
+            0.012798452749848366,
+            -0.019422436133027077,
+            0.065815769135952,
+            0.0029282080940902233,
+            0.01191638596355915,
+            -0.007900436408817768,
+            -0.02832792140543461,
+            -0.007022610399872065,
+            -0.013188597746193409,
+            0.025003207847476006,
+            -0.05492563545703888,
+            -0.027954738587141037,
+            -0.030787531286478043,
+            -0.020558945834636688,
+            0.013604187406599522,
+            -0.016055315732955933,
+            -0.002962133614346385,
+            -0.03562193736433983,
+            -0.017344491556286812,
+            0.010584804229438305,
+            -0.04563000425696373,
+            -0.017285121604800224,
+            -0.0030617902521044016,
+            0.0011131855426356196,
+            0.04135537147521973,
+            0.0010888014221563935,
+            -0.017522599548101425,
+            0.00937196146696806,
+            0.026903044432401657,
+            0.05119381099939346,
+            -0.003013022243976593,
+            -0.006912352051585913,
+            0.001623130519874394,
+            0.006526447832584381,
+            -0.005326327867805958,
+            0.005203347187489271,
+            1.655697815294843e-05,
+            -0.038302063941955566,
+            0.027852961793541908,
+            -0.025444241240620613,
+            -0.02541031502187252,
+            -0.003863283898681402,
+            0.008252415806055069,
+            0.02179723232984543,
+            -0.00820152647793293,
+            -0.03684325888752937,
+            0.019625991582870483,
+            0.03070271760225296,
+            -0.004711425397545099,
+            -0.0031169194262474775,
+            -0.00047098350478336215,
+            -0.01473221555352211,
+            -0.031025011092424393,
+            -0.03928590565919876,
+            0.013646594248712063,
+            -0.0467156246304512,
+            0.011042800731956959,
+            -0.03514697775244713,
+            0.0077901785261929035,
+            -0.0003336906083859503,
+            -0.004987071268260479,
+            0.012688194401562214,
+            0.05207587778568268,
+            0.005958192981779575,
+            0.02770029567182064,
+            -0.008154879324138165,
+            -0.03314536437392235,
+            0.006424670573323965,
+            -0.03426491096615791,
+            0.07972528785467148,
+            -0.005631658714264631,
+            -2.9949991585453972e-05,
+            0.005415382329374552,
+            0.007684160489588976,
+            -0.03314536437392235,
+            -0.029108211398124695,
+            0.032924845814704895,
+            0.0081039909273386,
+            0.0018171428237110376,
+            0.026156678795814514,
+            0.03207670524716377,
+            -0.007463643793016672,
+            -0.03565586358308792,
+            0.002756459405645728,
+            -0.0015775428619235754,
+            0.0106102479621768,
+            -0.019727768376469612,
+            0.018235038965940475,
+            -0.04077863693237305,
+            0.033772990107536316,
+            0.011186984367668629,
+            -0.021763307973742485,
+            0.0152580626308918,
+            -0.025817422196269035,
+            -0.0146643640473485,
+            -0.02870110422372818,
+            0.03088930808007717,
+            0.02946443110704422,
+            0.028480585664510727,
+            0.0007718086126260459,
+            0.04030367732048035,
+            -0.018608221784234047,
+            0.024731801822781563,
+            0.013756852596998215,
+            -0.0024362860713154078,
+            0.028158292174339294,
+            0.04033760353922844,
+            -0.031109824776649475,
+            0.0006530688260681927,
+            -0.010737469419836998,
+            0.0035855176392942667,
+            -0.03450239077210426,
+            0.0014619836583733559,
+            -0.022255228832364082,
+            -0.022085601463913918,
+            0.00032229372300207615,
+            0.013587224297225475,
+            0.005992118734866381,
+            0.0025889514945447445,
+            0.004779276438057423,
+            0.0024914152454584837,
+            -0.01139901950955391,
+            -0.031245527788996696,
+            -0.05601125583052635,
+            -0.004643573891371489,
+            -0.015766948461532593,
+            0.03714859113097191,
+            -0.040507230907678604,
+            0.014961213804781437,
+            0.018557332456111908,
+            -0.016564199700951576,
+            0.010949504561722279,
+            0.005415382329374552,
+            0.02407025173306465,
+            -0.04963323101401329,
+            -0.028599325567483902,
+            0.015173248946666718,
+            0.041457150131464005,
+            -0.030770568177103996,
+            0.009931734763085842,
+            0.00635257875546813,
+            0.0024723319802433252,
+            0.01610620506107807,
+            -0.0006970661925151944,
+            0.02817525528371334,
+            0.014214849099516869,
+            -0.008795225992798805,
+            -0.0056740655563771725,
+            0.018472518771886826,
+            0.020999979227781296,
+            0.028904657810926437,
+            -0.015105397440493107,
+            -0.05760576203465462,
+            -0.016275832429528236,
+            0.02074553817510605,
+            -0.00039306050166487694,
+            0.02222130261361599,
+            0.004588444717228413,
+            0.03231418505311012,
+            0.037589624524116516,
+            0.0037572660949081182,
+            0.020202726125717163,
+            0.0066960761323571205,
+            -0.056045182049274445,
+            -0.01574150286614895,
+            0.011008874513208866,
+            -0.023951511830091476,
+            -0.015079952776432037,
+            -0.005763120483607054,
+            0.004503630567342043,
+            -0.0410161167383194,
+            0.009736662730574608,
+            -0.00865104142576456,
+            0.026258455589413643,
+            0.02751370519399643,
+            0.00678089028224349,
+            0.011195465922355652,
+            -0.015410728752613068,
+            -0.006013322155922651,
+            -0.02016880176961422,
+            -0.026309344917535782,
+            0.026529861614108086,
+            -0.02317122183740139,
+            -0.025681721046566963,
+            -0.0036258043255656958,
+            0.01223019789904356,
+            0.0015839040279388428,
+            0.02498624473810196,
+            -0.018574295565485954,
+            -0.007200720254331827,
+            0.01993132196366787,
+            -0.0116789061576128,
+            -0.009405887685716152,
+            -0.0017874578479677439,
+            0.006874185521155596,
+            -0.0054662711918354034,
+            -0.015207174234092236,
+            0.004766554571688175,
+            0.03504519909620285,
+            -0.0038590431213378906,
+            0.009109037928283215,
+            -0.01178068295121193,
+            0.0034116485621780157,
+            0.06299994140863419,
+            -0.027089634910225868,
+            -0.04437475651502609,
+            0.04600318521261215,
+            0.051227737218141556,
+            -0.0011746757663786411,
+            -0.01542769093066454,
+            -0.019337622448801994,
+            -0.03356943652033806,
+            0.03823421150445938,
+            -0.043933723121881485,
+            0.014206367544829845,
+            -0.012891748920083046,
+            0.024002401158213615,
+            -0.001378229702822864,
+            -0.001866971142590046,
+            -0.020525021478533745,
+            0.04257669672369957,
+            0.03589334338903427,
+            -0.011331168003380299,
+            0.005279679782688618,
+            -0.0004161193501204252,
+            0.015224137343466282,
+            -0.02245878241956234,
+            0.02803955227136612,
+            0.0029388098046183586,
+            -0.010304917581379414,
+            0.005962433759123087,
+            -0.01664053276181221,
+            -0.012476159259676933,
+            -0.0033607601653784513,
+            0.013748371042311192,
+            -0.007289774715900421,
+            0.035350531339645386,
+            -0.04118574410676956,
+            -0.0035473511088639498,
+            0.0041262079030275345,
+            0.005890341475605965,
+            0.009838439524173737,
+            -0.00034800299908965826,
+            -0.036266524344682693,
+            -0.024494322016835213,
+            -0.029260877519845963,
+            0.015605800785124302,
+            -0.04708880931138992,
+            -0.01988043263554573,
+            0.03504519909620285,
+            0.008663764223456383,
+            -0.01782793179154396,
+            -0.02069464884698391,
+            -0.007603587117046118,
+            -0.004728388041257858,
+            0.013799259439110756,
+            -0.009838439524173737,
+            0.0021447374019771814,
+            -0.013672038912773132,
+            0.023103371262550354,
+            -0.015105397440493107,
+            0.02454521134495735,
+            -0.04183033108711243,
+            0.011144577525556087,
+            -0.014316625893115997,
+            0.010796839371323586,
+            0.03999834507703781,
+            0.017675265669822693,
+            -0.018557332456111908,
+            -0.008455969393253326,
+            -0.028785917907953262,
+            -0.005207587964832783,
+            -0.005614695604890585,
+            0.016097722575068474,
+            -0.02164456807076931,
+            0.010355805978178978,
+            0.01518173050135374,
+            0.011170021258294582,
+            -0.014995139092206955,
+            -0.011025837622582912,
+            0.0367075577378273,
+            -0.002381156897172332,
+            0.021865084767341614,
+            0.014503217302262783,
+            -0.01115305908024311,
+            0.015419209375977516,
+            0.009880846366286278,
+            0.03952338546514511,
+            0.00752301374450326,
+            -0.02937961556017399,
+            -0.02846362441778183,
+            -0.005334808956831694,
+            0.030770568177103996,
+            -0.02917606197297573,
+            -0.0024680914357304573,
+            0.048038724809885025,
+            -0.019761692732572556,
+            0.015673652291297913,
+            -0.02464698813855648,
+            -0.020016135647892952,
+            0.0055468445643782616,
+            -0.022764114663004875,
+            -0.05774146318435669,
+            -0.018489481881260872,
+            -0.019965248182415962,
+            -0.026088828220963478,
+            0.03952338546514511,
+            -0.027157485485076904,
+            -0.048649389296770096,
+            0.014291182160377502,
+            0.012179309502243996,
+            0.009363480843603611,
+            -0.025342464447021484,
+            -0.00943981297314167,
+            -0.02230611816048622,
+            -0.01774311624467373,
+            -0.00170476408675313,
+            0.0032823069486767054,
+            0.004656296223402023,
+            -0.021576715633273125,
+            -0.03976086527109146,
+            0.004270391538739204,
+            0.028141330927610397,
+            0.00511429226025939,
+            0.02435861900448799,
+            0.0042152623645961285,
+            0.013595705851912498,
+            0.008769781328737736,
+            0.01161105465143919,
+            0.009117519482970238,
+            -0.04325520992279053,
+            -0.004329761490225792,
+            -0.006263523828238249,
+            -0.040066197514534,
+            -0.010313399136066437,
+            -0.012484640814363956,
+            0.004085920751094818,
+            0.01854036934673786,
+            0.007993732579052448,
+            0.0026313585694879293,
+            -0.014265737496316433,
+            0.005521400365978479,
+            -0.00511429226025939,
+            0.010389731265604496,
+            0.02188204601407051,
+            -0.0005422803806141019,
+            -0.0357576385140419,
+            0.019388511776924133,
+            0.031584784388542175,
+            0.01644546166062355,
+            -0.014749177731573582,
+            -0.019778655841946602,
+            0.02798866480588913,
+            0.022526634857058525,
+            -0.020558945834636688,
+            -0.04094826430082321,
+            0.02927783876657486,
+            -0.026953931897878647,
+            0.018268965184688568,
+            0.029023397713899612,
+            0.004105004016309977,
+            -0.02354440465569496,
+            0.008163360878825188,
+            7.255584205267951e-05,
+            -0.0025211002212017775,
+            -0.011288761161267757,
+            -0.014146997593343258,
+            0.0001689656637609005,
+            -0.008379636332392693,
+            0.003568554762750864,
+            0.02880288101732731,
+            -0.016716865822672844,
+            -0.014588030986487865,
+            0.02530853822827339,
+            0.01982954517006874,
+            0.0038060343358665705,
+            0.001432298682630062,
+            -0.034298837184906006,
+            -0.005754638928920031,
+            0.010864690877497196,
+            -0.024714838713407516,
+            0.01063569262623787,
+            0.009176889434456825,
+            -0.00511429226025939,
+            -0.03302662447094917,
+            -0.0002263477217638865,
+            -0.01921888254582882,
+            -0.016954345628619194,
+            0.030193831771612167,
+            0.023001592606306076,
+            0.0009414369123987854,
+            0.012806934304535389,
+            -0.027361039072275162,
+            -0.016954345628619194,
+            -0.016606608405709267,
+            0.006607021205127239,
+            -0.0021701816003769636,
+            -0.023714032024145126,
+            -0.00695051858201623,
+            -0.014978175982832909,
+            0.015699096024036407,
+            0.031008047983050346,
+            -0.005360253155231476,
+            -0.00413680961355567,
+            0.00024556342395953834,
+            0.0027331355959177017,
+            0.00522030983120203,
+            -0.026631638407707214,
+            0.0023853974416851997,
+            0.017607415094971657,
+            0.009354999288916588,
+            0.012315012514591217,
+            0.008684967644512653,
+            -0.0008836572524160147,
+            -0.04505326971411705,
+            0.020253615453839302,
+            -0.030380424112081528,
+            -0.008795225992798805,
+            0.03972693905234337,
+            3.036412272194866e-05,
+            -0.010542397387325764,
+            0.019625991582870483,
+            -0.0036872944328933954,
+            -0.04172855615615845,
+            0.016894975677132607,
+            -0.002285741036757827,
+            -0.030787531286478043,
+            0.018676072359085083,
+            -0.020253615453839302,
+            0.008099749684333801,
+            0.014910325407981873,
+            0.026156678795814514,
+            -0.03009205497801304,
+            -0.007018369622528553,
+            0.020609835162758827,
+            -0.028548438102006912,
+            0.0010400332976132631,
+            0.0032632239162921906,
+            0.018659109249711037,
+            0.0036046006716787815,
+            0.007366107776761055,
+            -0.051566995680332184,
+            -0.028768954798579216,
+            0.014893362298607826,
+            -0.0010236005764454603,
+            0.0018341057002544403,
+            0.05794501677155495,
+            0.02412113919854164,
+            -0.07450073957443237,
+            0.01711549237370491,
+            0.026394158601760864,
+            0.025020169094204903,
+            -0.027157485485076904,
+            -0.009142964147031307,
+            0.036300450563430786,
+            0.021050868555903435,
+            0.009126001037657261,
+            -0.005398419685661793,
+            0.0022242506965994835,
+            -0.023103371262550354,
+            0.021525828167796135,
+            0.013290374539792538,
+            -0.014392958953976631,
+            -0.036775410175323486,
+            -0.024138102307915688,
+            -0.025088021531701088,
+            -0.0032971494365483522,
+            -0.015139322727918625,
+            -0.018438592553138733,
+            0.03518090397119522,
+            -0.005597732961177826,
+            0.0011502917623147368,
+            0.029142137616872787,
+            -0.002459609881043434,
+            -0.003973542246967554,
+            0.01532591413706541,
+            0.019863469526171684,
+            -0.014783103950321674,
+            -0.002510498510673642,
+            -0.015453135594725609,
+            0.0020228170324116945,
+            0.026716452091932297,
+            -0.024002401158213615,
+            0.015656689181923866,
+            0.0029282080940902233,
+            0.028904657810926437,
+            -0.00970273744314909,
+            0.011042800731956959,
+            -0.01720878854393959,
+            0.00021826387092005461,
+            -0.014986657537519932,
+            0.034451503306627274,
+            -0.006848741322755814,
+            -0.016284313052892685,
+            -0.036504004150629044,
+            0.03847169131040573,
+            0.020779462531208992,
+            -0.014384477399289608,
+            -0.046037111431360245,
+            -0.01535135880112648,
+            -0.028904657810926437,
+            0.0007373528787866235,
+            0.008328747935593128,
+            0.010720506310462952,
+            0.014867917634546757,
+            0.017030678689479828,
+            -0.007794418837875128,
+            0.0002812118618749082,
+            -0.009066631086170673,
+            -0.03367121145129204,
+            0.010211621411144733,
+            -0.038878798484802246,
+            0.01736145280301571,
+            -0.008167601190507412,
+            0.007785937748849392,
+            0.01697978936135769,
+            -0.038030657917261124,
+            -0.010576322674751282,
+            -0.03156781941652298,
+            0.011195465922355652,
+            -0.011008874513208866,
+            0.015800872817635536,
+            -0.030109018087387085,
+            0.011568647809326649,
+            0.020558945834636688,
+            -0.0038081547245383263,
+            0.033874765038490295,
+            -0.006047247909009457,
+            -0.01859125867486,
+            -0.05183839797973633,
+            -0.008027657866477966,
+            0.028531474992632866,
+            0.03986264392733574,
+            -0.0026186364702880383,
+            0.027394965291023254,
+            -0.004440019838511944,
+            0.012238679453730583,
+            0.00418133707717061,
+            -0.021474938839673996,
+            0.027666371315717697,
+            -0.02354440465569496,
+            0.005682547111064196,
+            -0.049565382301807404,
+            -0.03891272470355034,
+            -0.057809315621852875,
+            -0.03155085816979408,
+            -0.0028836806304752827,
+            -0.010330361314117908,
+            0.01270515751093626,
+            0.015758465975522995,
+            0.02103390544652939,
+            -0.0023557126987725496,
+            0.007035332731902599,
+            -0.004804720636457205,
+            -0.028768954798579216,
+            0.03060094080865383,
+            -0.004745350684970617,
+            0.003000300144776702,
+            -0.013078339397907257,
+            0.04047330468893051,
+            0.011220909655094147,
+            0.0116789061576128,
+            0.00045879147364757955,
+            -0.025003207847476006,
+            -0.05160091817378998,
+            0.007472125347703695,
+            -0.008481413125991821,
+            0.006424670573323965,
+            -0.004749591462314129,
+            -0.027479778975248337,
+            0.005249994806945324,
+            0.01587720587849617,
+            -0.0055001964792609215,
+            -0.0029790964908897877,
+            -0.024239879101514816,
+            -0.008905484341084957,
+            -0.03915020450949669,
+            -0.006318653002381325,
+            -0.005004033911973238,
+            0.008778262883424759,
+            -0.027428891509771347,
+            0.011729794554412365,
+            0.011857016012072563,
+            0.008065824396908283,
+            -0.009948697872459888,
+            -0.025206761434674263,
+            -0.007484847214072943,
+            -0.005097329616546631,
+            -0.01518173050135374,
+            0.01363811269402504,
+            -0.026343271136283875,
+            -0.027479778975248337,
+            -0.02627541869878769,
+            0.014791585505008698,
+            -0.016428498551249504,
+            -0.01921888254582882,
+            -0.005020997021347284,
+            0.024901429191231728,
+            0.020728575065732002,
+            0.02764940820634365,
+            0.0014895482454448938,
+            0.024019362404942513,
+            0.027598518878221512,
+            -0.00041585430153645575,
+            -0.0012404067674651742,
+            0.007200720254331827,
+            -0.0023281481117010117,
+            0.0031529655680060387,
+            -0.01328189391642809,
+            0.010949504561722279,
+            0.003430731827393174,
+            0.009032705798745155,
+            -0.026088828220963478,
+            -0.005691028665751219,
+            -0.00013152816973160952,
+            -0.02016880176961422,
+            0.0123998261988163,
+            -0.04257669672369957,
+            -0.0047368695959448814,
+            0.024392545223236084,
+            -0.014189404435455799,
+            0.025902237743139267,
+            0.031686559319496155,
+            -0.012442233972251415,
+            0.009583997540175915,
+            0.0020567427854985,
+            -0.02751370519399643,
+            -0.010805320926010609,
+            -0.0046817404218018055,
+            0.021288348361849785,
+            9.351089829578996e-06,
+            -0.019049255177378654,
+            -0.008684967644512653,
+            -0.029430504888296127,
+            0.01494425069540739,
+            0.02312033250927925,
+            0.014409921132028103,
+            -0.04420512542128563,
+            0.0029918185900896788,
+            0.0025529053527861834,
+            -0.008337229490280151,
+            -0.01663205213844776,
+            -0.010601767338812351,
+            -0.020813388749957085,
+            0.01731056533753872,
+            -0.016182536259293556,
+            -0.00791315920650959,
+            0.015758465975522995,
+            0.01532591413706541,
+            0.01332430075854063,
+            0.01630127616226673,
+            -0.0074254777282476425,
+            -0.013417595997452736,
+            0.0008804767276160419,
+            0.005771602038294077,
+            0.039082352072000504,
+            0.023849735036492348,
+            0.041457150131464005,
+            -0.006560373120009899,
+            0.0028285514563322067,
+            0.0011555926175788045,
+            -0.019710805267095566,
+            0.0015934455441311002,
+            -0.010525434277951717,
+            -0.0007967227720655501,
+            -0.015648208558559418,
+            -0.00786651112139225,
+            0.005979396402835846,
+            0.024765728041529655,
+            0.006475559435784817,
+            0.01877784915268421,
+            0.007985251024365425,
+            -0.0190662182867527,
+            0.002330268267542124,
+            -0.0035240272991359234,
+            -0.009897809475660324,
+            -0.0021468577906489372,
+            -0.035350531339645386,
+            0.0176413394510746,
+            -0.01753956265747547,
+            0.009176889434456825,
+            0.00035197866964153945,
+            -0.01806541159749031,
+            -0.008210008032619953,
+            0.01077987626194954,
+            -0.030295608565211296,
+            0.013587224297225475,
+            -0.01044061966240406,
+            -0.025783497840166092,
+            -0.04081256315112114,
+            0.038505617529153824,
+            0.01735297217965126,
+            -0.007756252773106098,
+            0.02988850139081478,
+            0.01611468568444252,
+            -0.013315819203853607,
+            -0.014367514289915562,
+            -0.009668811224400997,
+            -0.010508471168577671,
+            0.01032188069075346,
+            0.012586417607963085,
+            0.01854036934673786,
+            0.005491715390235186,
+            0.0019242207054048777,
+            0.003341676900163293,
+            0.003956579603254795,
+            -0.01508843433111906,
+            0.0004513702297117561,
+            -0.007811381947249174,
+            0.0052542355842888355,
+            -0.021763307973742485,
+            0.005330568179488182,
+            -0.04115181788802147,
+            -0.0017514119390398264,
+            0.01387559249997139,
+            0.01921888254582882,
+            0.0008030838216654956,
+            0.00867648608982563,
+            0.014783103950321674,
+            -0.022170415148139,
+            0.007124387193471193,
+            -0.016462422907352448,
+            0.0058097681030631065,
+            -0.015376802533864975,
+            0.012645787559449673,
+            0.01418092381209135,
+            0.017056122422218323,
+            0.013553299009799957,
+            -0.0017132455250248313,
+            0.016886495053768158,
+            0.044510457664728165,
+            -0.01325644925236702,
+            -0.0012393465731292963,
+            0.024155065417289734,
+            0.019371548667550087,
+            -0.005678306333720684,
+            -0.009804514236748219,
+            0.01887962594628334,
+            0.011288761161267757,
+            -0.02417202852666378,
+            0.017844894900918007,
+            0.014749177731573582,
+            0.022204341366887093,
+            0.052890095859766006,
+            -0.005555325653403997,
+            -0.005652862135320902,
+            0.012510084547102451,
+            0.009685774333775043,
+            -0.009634885936975479,
+            -0.012501603923738003,
+            0.025613868609070778,
+            -0.02127138525247574,
+            -0.028633251786231995,
+            -0.010576322674751282,
+            -0.008718892931938171,
+            -0.01456258725374937,
+            -0.014418402686715126,
+            0.009779069572687149,
+            0.009100556373596191,
+            -0.005529881455004215,
+            0.004864090587943792,
+            0.016420016065239906,
+            -0.0031466044019907713,
+            0.031025011092424393,
+            -0.013307337649166584,
+            -0.03265344351530075,
+            0.004554518964141607,
+            0.009736662730574608,
+            -0.00592426722869277,
+            -0.0024532489478588104,
+            -0.01882873848080635,
+            0.0024256843607872725,
+            0.050651002675294876,
+            -0.0063865045085549355,
+            0.0011004634434357285,
+            -0.02398543804883957,
+            -0.004673258867114782,
+            0.007489087991416454,
+            0.006933555472642183,
+            0.005741917062550783,
+            0.022662336006760597,
+            0.021915972232818604,
+            0.00367669272236526,
+            -0.004149531479924917,
+            -0.0003323653945699334,
+            -0.028768954798579216,
+            -0.010550878010690212,
+            0.0036809334997087717,
+            0.016411535441875458,
+            0.005245754029601812,
+            0.013383670710027218,
+            0.007378829643130302,
+            0.007785937748849392,
+            -0.01187397912144661,
+            -0.027004821226000786,
+            -0.0057758428156375885,
+            -0.005724953953176737,
+            0.012806934304535389,
+            -0.01187397912144661,
+            0.0048174429684877396,
+            -0.012162347324192524,
+            -0.021050868555903435,
+            0.03185619041323662,
+            0.009244740940630436,
+            -0.005279679782688618,
+            0.002268778160214424,
+            -0.007065017241984606,
+            0.015105397440493107,
+            -0.010067437775433064,
+            0.0019496649038046598,
+            0.0029133656062185764,
+            0.0357576385140419,
+            -0.007344903890043497,
+            0.012213235720992088,
+            0.02103390544652939,
+            0.01835377886891365,
+            -0.04261062294244766,
+            -0.015410728752613068,
+            0.02985457517206669,
+            -0.024019362404942513,
+            0.01845555566251278,
+            0.0058394530788064,
+            0.007489087991416454,
+            -0.023137295618653297,
+            -0.00018818136595655233,
+            0.00791315920650959,
+            0.0047368695959448814,
+            0.026343271136283875,
+            0.02330692484974861,
+            0.04037152975797653,
+            0.009507664479315281,
+            0.00305754947476089,
+            -0.006420430261641741,
+            0.0064373929053545,
+            -0.0031699282117187977,
+            -0.0003058609727304429,
+            -0.014409921132028103,
+            0.008468691259622574,
+            0.000756436085794121,
+            -0.013103784061968327,
+            -0.007086221128702164,
+            0.00927018467336893,
+            -0.0110852075740695,
+            0.027683334425091743,
+            0.004630851559340954,
+            0.00511429226025939,
+            -0.005860656965523958,
+            -0.007476366125047207,
+            0.020016135647892952,
+            -0.02985457517206669,
+            -0.010313399136066437,
+            -0.017191825434565544,
+            0.0015054509276524186,
+            -0.004073198884725571,
+            -0.007031091954559088,
+            0.012747564353048801,
+            -0.006284727249294519,
+            -0.006072692107409239,
+            -0.004342483822256327,
+            -0.014011295512318611,
+            -0.006276246160268784,
+            0.006725761108100414,
+            -0.02627541869878769,
+            -0.0035473511088639498,
+            -0.01032188069075346,
+            -0.027717258781194687,
+            -0.03448542580008507,
+            -0.01721726916730404,
+            0.03151693195104599,
+            -0.01184853445738554,
+            -0.005033718887716532,
+            0.006310171447694302,
+            -0.0026143956929445267,
+            0.017149418592453003,
+            0.011170021258294582,
+            -0.016224944964051247,
+            0.011738276109099388,
+            -0.0012340457178652287,
+            0.006861463654786348,
+            0.0008582130540162325,
+            -0.00943981297314167,
+            -0.011373575776815414,
+            -0.019388511776924133,
+            -0.00015664110833313316,
+            -0.004550278186798096,
+            -0.012756045907735825,
+            0.004775035660713911,
+            0.007120146416127682,
+            0.012527047656476498,
+            0.008646801114082336,
+            -0.0012404067674651742,
+            0.00803189817816019,
+            -0.012730601243674755,
+            -0.013917999342083931,
+            0.004982830490916967,
+            0.00899029802531004,
+            -0.01821807585656643,
+            -0.00784530770033598,
+            0.04142322391271591,
+            -0.022051675245165825,
+            -0.0028200699016451836,
+            -0.007192238699644804,
+            0.009405887685716152,
+            -0.009075112640857697,
+            -0.0048725721426308155,
+            0.012408307753503323,
+            -0.005644380580633879,
+            -0.009931734763085842,
+            -0.036775410175323486,
+            -0.0013983730459585786,
+            0.043221283704042435,
+            -0.019015328958630562,
+            0.01759045198559761,
+            0.0007219803519546986,
+            0.02651289850473404,
+            -0.001526654465124011,
+            0.013756852596998215,
+            -0.03555408492684364,
+            -0.009762107394635677,
+            0.003136002691462636,
+            0.015487060882151127,
+            -0.006238079629838467,
+            -0.016699902713298798,
+            0.03438365086913109,
+            -0.017794005572795868,
+            -0.022187378257513046,
+            0.00362368393689394,
+            -0.03470594435930252,
+            0.007981009781360626,
+            0.0283618476241827,
+            0.011619536206126213,
+            -0.015156285837292671,
+            -0.013578742742538452,
+            0.008222730830311775,
+            0.00781986303627491,
+            -0.002936689415946603,
+            0.0009637005859985948,
+            0.01740385964512825,
+            0.023476552218198776,
+            0.036741483956575394,
+            -0.03312839940190315,
+            -0.021441012620925903,
+            0.0008216369315050542,
+            0.015190212056040764,
+            -0.004999793134629726,
+            0.029956351965665817,
+            0.030431311577558517,
+            0.012043607421219349,
+            0.010584804229438305,
+            -0.018031485378742218,
+            0.009405887685716152,
+            0.00692507391795516,
+            0.011543204076588154,
+            0.02079642564058304,
+            -0.0011524120345711708,
+            0.024613061919808388,
+            0.01582631841301918,
+            0.005890341475605965,
+            -0.0006186130922287703,
+            -0.02040628157556057,
+            -0.007667197845876217,
+            0.016988271847367287,
+            0.004847127944231033,
+            -9.746999421622604e-05,
+            0.005936989560723305,
+            0.0035240272991359234,
+            0.019337622448801994,
+            0.006004840601235628,
+            -0.006632465403527021,
+            0.008231211453676224,
+            0.0004375879361759871,
+            -0.009329554624855518,
+            -0.0025677478406578302,
+            0.014248774386942387,
+            0.023663144558668137,
+            -0.00216382066719234,
+            0.00714135030284524,
+            -0.021491901949048042,
+            0.01639457233250141,
+            -0.0011566528119146824,
+            -0.0040647173300385475,
+            -0.0034116485621780157,
+            -0.021966861560940742,
+            0.02664860151708126,
+            0.0004627671150956303,
+            0.012348937802016735,
+            -0.0005454609054140747,
+            0.0036194431595504284,
+            0.021661529317498207,
+            -0.02359529212117195,
+            -7.586889114463702e-05,
+            -0.009982624091207981,
+            -0.010788357816636562,
+            -0.028497548773884773,
+            0.0033586397767066956,
+            0.03402743116021156,
+            0.013392152264714241,
+            -0.008735856041312218,
+            -0.0009186430834233761,
+            -0.011746757663786411,
+            0.016767755150794983,
+            0.010279472917318344,
+            -0.002824310678988695,
+            0.0009409068152308464,
+            -0.017912745475769043,
+            -0.003307751379907131,
+            0.0028073478024452925,
+            -0.005623177159577608,
+            -0.007938602939248085,
+            -0.010898616164922714,
+            -0.0037297015078365803,
+            0.018608221784234047,
+            0.006602780427783728,
+            -0.007128627970814705,
+            0.005987877957522869,
+            -0.013002007268369198,
+            0.001110004959627986,
+            -0.000263983994955197,
+            0.013332782313227654,
+            -0.016191018745303154,
+            -0.02398543804883957,
+            0.016428498551249504,
+            -0.0005292932037264109,
+            -0.009176889434456825,
+            -0.019473325461149216,
+            -0.010754432529211044,
+            -0.022390931844711304,
+            0.010041994042694569,
+            -0.006963240448385477,
+            0.006013322155922651,
+            0.001960266614332795,
+            -0.023018555715680122,
+            0.009397406131029129,
+            0.007955566048622131,
+            -0.01122939120978117,
+            0.031160714104771614,
+            -0.024918392300605774,
+            0.018268965184688568,
+            -0.0009165227529592812,
+            0.014698289334774017,
+            0.02069464884698391,
+            -0.0013474845327436924,
+            -0.03255166485905647,
+            0.006670631933957338,
+            -0.009516146034002304,
+            0.01578390970826149,
+            -0.0068402597680687904,
+            -0.002148978179320693,
+            -0.006428911350667477,
+            -0.04077863693237305,
+            -0.017912745475769043,
+            -0.0069717220030725,
+            -0.0007262210128828883,
+            0.055468443781137466,
+            -0.009456776082515717,
+            -0.001471525290980935,
+            -0.022153452038764954,
+            -0.0021839640103280544,
+            -0.033874765038490295,
+            0.0011142457369714975,
+            0.017675265669822693,
+            0.005860656965523958,
+            -0.0045630005188286304,
+            0.010881653986871243,
+            0.010101363062858582,
+            0.0056740655563771725,
+            0.013841667212545872,
+            -0.00018341056420467794,
+            -0.04213566333055496,
+            0.005975155625492334,
+            -0.012951118871569633,
+            -0.006382263731211424,
+            0.021424051374197006,
+            0.011042800731956959,
+            0.0014047340955585241,
+            0.018082372844219208,
+            0.0029833372682332993,
+            0.002404480706900358,
+            -0.010864690877497196,
+            -0.017132455483078957,
+            0.017336009070277214,
+            0.0024150824174284935,
+            0.007662957068532705,
+            -0.004830164834856987,
+            -0.004520593211054802,
+            0.013146190904080868,
+            0.0011661943281069398,
+            0.012603380717337132,
+            0.015173248946666718,
+            -0.008044620975852013,
+            -0.002330268267542124,
+            0.024630025029182434,
+            -0.005330568179488182,
+            0.0022730189375579357,
+            -0.07456858456134796,
+            0.038166359066963196,
+            0.028141330927610397,
+            -0.009329554624855518,
+            -0.00543658621609211,
+            -0.015376802533864975,
+            -0.015512505546212196,
+            -0.0046817404218018055,
+            -0.024494322016835213,
+            -0.014130034483969212,
+            0.02583438530564308,
+            -0.014791585505008698,
+            0.00025165945407934487,
+            0.004363687243312597,
+            0.0048725721426308155,
+            0.005385697353631258,
+            0.006233838852494955,
+            -0.00752301374450326,
+            0.017675265669822693,
+            -0.021542789414525032,
+            0.002406601095572114,
+            -0.010712025687098503,
+            -0.01610620506107807,
+            -0.0016803800826892257,
+            0.007573902141302824,
+            -0.008701929822564125,
+            0.0065773362293839455,
+            0.030380424112081528,
+            -0.015062990598380566,
+            0.031873151659965515,
+            0.003867524676024914,
+            -0.016911938786506653,
+            -0.04420512542128563,
+            0.009999586269259453,
+            0.021474938839673996,
+            0.015266544185578823,
+            -0.029498355463147163,
+            0.0037106184754520655,
+            -0.03165263682603836,
+            -0.013239486142992973,
+            -0.01329885609447956,
+            -0.012815415859222412,
+            -0.024256842210888863,
+            0.00754421716555953,
+            0.010025030933320522,
+            -0.017047641798853874,
+            -0.010008067823946476,
+            -0.02717444859445095,
+            -0.01025402918457985,
+            -0.00856198649853468,
+            -0.023256035521626472,
+            -0.03270433098077774,
+            -0.008239693008363247,
+            0.0036957759875804186,
+            -0.01349392905831337,
+            -0.0123998261988163,
+            -0.019659915938973427,
+            -0.003948098048567772,
+            0.002957893069833517,
+            0.007824104279279709,
+            -0.006089654751121998,
+            0.02030450478196144,
+            -0.01587720587849617,
+            -0.008358432911336422,
+            0.007226164452731609,
+            0.012781490571796894,
+            0.003311991924419999,
+            0.022492708638310432,
+            0.014138516038656235,
+            0.02965102158486843,
+            0.02427380532026291,
+            0.00798949133604765,
+            0.0037805901374667883,
+            0.024307731539011,
+            0.007768974639475346,
+            -0.026241492480039597,
+            -0.019371548667550087,
+            -0.0016570561565458775,
+            0.022390931844711304,
+            -0.005249994806945324,
+            -0.010126807726919651,
+            0.017573488876223564,
+            -0.0007357626454904675,
+            0.015071472153067589,
+            -0.030109018087387085,
+            -0.013951925560832024,
+            0.0027055710088461637,
+            -0.030770568177103996,
+            -0.008595912717282772,
+            -0.0008905483991838992,
+            0.00686570443212986,
+            -0.0012181430356577039,
+            -0.006623983848839998,
+            -0.011144577525556087,
+            0.03155085816979408,
+            0.0074424403719604015,
+            0.0021267144475132227,
+            0.010423657484352589,
+            -0.004864090587943792,
+            0.00592426722869277,
+            0.004486667923629284,
+            -0.032907884567976,
+            0.00645859632641077,
+            -0.004978589713573456,
+            0.014537142589688301,
+            -0.0169289018958807,
+            -0.005589251406490803,
+            0.0024299249053001404,
+            -0.0051015703938901424,
+            -4.8271169362124056e-05,
+            0.02250967174768448,
+            -0.01602138951420784,
+            0.0030639106407761574,
+            0.0016517553012818098,
+            -0.0036618502344936132,
+            0.006967481225728989,
+            0.03619867190718651,
+            0.0044527421705424786,
+            -0.006085413973778486,
+            -0.021254422143101692,
+            -0.005712232086807489,
+            -0.0049446639604866505,
+            -0.007645994424819946,
+            -0.042508844286203384,
+            -0.005996359512209892,
+            -0.0010326120536774397,
+            -0.02588527463376522,
+            -0.024528248235583305,
+            -0.007896196097135544,
+            0.01187397912144661,
+            -0.01253552921116352,
+            -0.008455969393253326,
+            -0.014503217302262783,
+            0.007187997922301292,
+            0.0006684414111077785,
+            0.010991912335157394,
+            -0.015156285837292671,
+            0.008197286166250706,
+            9.607851097825915e-05,
+            -0.0023832772858440876,
+            -0.010008067823946476,
+            0.0110852075740695,
+            0.010728987865149975,
+            0.008061583153903484,
+            0.008040379732847214,
+            -0.008214249275624752,
+            0.015911132097244263,
+            -0.005512918811291456,
+            0.005275439005345106,
+            0.010822284035384655,
+            -0.01223019789904356,
+            -0.013375189155340195,
+            -0.0007044874364510179,
+            -0.009499182924628258,
+            -0.020050061866641045,
+            0.010423657484352589,
+            0.009880846366286278,
+            -0.009736662730574608,
+            0.011042800731956959,
+            0.0027607001829892397,
+            0.0019083180231973529,
+            -0.007484847214072943,
+            0.012968081049621105,
+            0.08813884854316711,
+            -0.0005459910025820136,
+            0.018387705087661743,
+            0.01601290889084339,
+            -0.0056571029126644135,
+            -0.030923234298825264,
+            -0.0014535022201016545,
+            -0.03418009728193283,
+            -0.008566227741539478,
+            -0.014384477399289608,
+            -0.026190605014562607,
+            -0.01898140273988247,
+            -0.014291182160377502,
+            0.01549554243683815,
+            0.006496762856841087,
+            0.0023048240691423416,
+            0.00865528266876936,
+            0.006280486937612295,
+            0.031008047983050346,
+            -0.010839246213436127,
+            -0.01044061966240406,
+            -0.02680126577615738,
+            0.011161540634930134,
+            0.000879946630448103,
+            0.019659915938973427,
+            -0.00891396589577198,
+            0.019439399242401123,
+            0.024528248235583305,
+            0.016199499368667603,
+            0.03136426582932472,
+            0.025393351912498474,
+            0.009388924576342106,
+            0.011806127615272999,
+            -0.005644380580633879,
+            -0.0026822469662874937,
+            0.014757659286260605,
+            -0.014520179480314255,
+            0.001473645563237369,
+            -0.002315425779670477,
+            -0.01387559249997139,
+            0.0033925652969628572,
+            -0.016555719077587128,
+            0.0036682114005088806,
+            0.012315012514591217,
+            -0.003678813111037016,
+            -0.0025762293953448534,
+            0.001580723444931209,
+            -6.443886377383024e-05,
+            -0.005360253155231476,
+            -0.000742653792258352,
+            -0.008540783077478409,
+            -0.008829151280224323,
+            0.00013795548875350505,
+            0.016080759465694427,
+            0.006547651253640652,
+            0.00362368393689394,
+            0.013858629390597343,
+            -0.008451728150248528,
+            -0.013214042410254478,
+            0.0054111420176923275,
+            -0.00035091847530566156,
+            0.003598239738494158,
+            0.014588030986487865,
+            -0.004520593211054802,
+            -0.007226164452731609,
+            0.002864597365260124,
+            0.00047389898099936545,
+            0.0006806334131397307,
+            -0.007768974639475346,
+            0.022950705140829086,
+            0.01935458555817604,
+            0.0010066378163173795,
+            0.012382864020764828,
+            0.01893051527440548,
+            0.0022624169941991568,
+            -0.03173745051026344,
+            -0.003996866289526224,
+            -0.007361866999417543,
+            0.013587224297225475,
+            -0.0021171728149056435,
+            -0.01682712510228157,
+            0.00807854626327753,
+            -0.009583997540175915,
+            -0.005576529540121555,
+            -0.009346517734229565,
+            -0.002582590328529477,
+            0.017912745475769043,
+            0.004461223725229502,
+            -0.024138102307915688,
+            0.01577542908489704,
+            0.011263317428529263,
+            -0.003971422091126442,
+            0.0013326420448720455,
+            -0.015724539756774902,
+            -0.004083800595253706,
+            -0.006602780427783728,
+            0.010813802480697632,
+            -0.0031529655680060387,
+            -0.013383670710027218,
+            -0.01774311624467373,
+            0.029532281681895256,
+            -0.005869138054549694,
+            0.005402660463005304,
+            -0.004821683745831251,
+            0.0011301483027637005,
+            0.02206863835453987,
+            0.014604994095861912,
+            -0.005606214515864849,
+            0.013502410613000393,
+            -0.013443040661513805,
+            -0.0063653006218373775,
+            0.028293995186686516,
+            -0.006051488686352968,
+            -0.007645994424819946,
+            0.007692642044275999,
+            -0.007302497047930956,
+            -0.024239879101514816,
+            0.0036872944328933954,
+            -0.02412113919854164,
+            0.004707184620201588,
+            -0.015122360549867153,
+            0.022238265722990036,
+            0.015164767391979694,
+            -0.008621356450021267,
+            -0.003013022243976593,
+            0.0026546823792159557,
+            -0.002449008170515299,
+            -0.00984692107886076,
+            0.0027925053145736456,
+            0.015945058315992355,
+            -0.004804720636457205,
+            0.015071472153067589,
+            -0.010092882439494133,
+            -0.024324694648385048,
+            -0.0005115352687425911,
+            -0.011068244464695454,
+            -0.01411307230591774,
+            0.0018902950687333941,
+            -0.027581557631492615,
+            0.007081980351358652,
+            -0.019948285073041916,
+            0.000553942343685776,
+            -0.015724539756774902,
+            -0.006992925424128771,
+            -0.007171035278588533,
+            0.020525021478533745,
+            -0.003299269825220108,
+            0.0041198465041816235,
+            0.0022730189375579357,
+            -0.007811381947249174,
+            0.01184853445738554,
+            -0.027021784335374832,
+            0.003909931518137455,
+            0.01677623577415943,
+            0.0033183530904352665,
+            -0.01835377886891365,
+            -0.0013994332402944565,
+            0.023900622501969337,
+            -0.008871558122336864,
+            -0.0029833372682332993,
+            -0.007400033064186573,
+            0.008812189102172852,
+            0.007192238699644804,
+            0.003044827375560999,
+            -0.007764733862131834,
+            0.023442627862095833,
+            0.01701371558010578,
+            0.008468691259622574,
+            0.019422436133027077,
+            -0.012925674207508564,
+            -0.00535177206620574,
+            0.022204341366887093,
+            -0.001378229702822864,
+            -0.014163960702717304,
+            0.0011460509849712253,
+            -0.005271198693662882,
+            -0.009354999288916588,
+            0.006242320407181978,
+            0.01518173050135374,
+            -0.010389731265604496,
+            -0.002245454117655754,
+            -0.014477772638201714,
+            0.005771602038294077,
+            -0.011729794554412365,
+            -0.0025741090066730976,
+            -0.01132268738001585,
+            -0.007730808574706316,
+            0.007349144667387009,
+            -0.005635899491608143,
+            -0.01669142208993435,
+            -0.009100556373596191,
+            -0.006204153876751661,
+            -0.003032105276361108,
+            0.010033512488007545,
+            0.025342464447021484,
+            -0.01387559249997139,
+            0.015716059133410454,
+            -0.0048895347863435745,
+            0.008803707547485828,
+            0.016046835109591484,
+            0.013358226045966148,
+            -0.013061377219855785,
+            -0.01630127616226673,
+            0.0011131855426356196,
+            -0.005805527325719595,
+            0.010762914083898067,
+            0.0016157092759385705,
+            0.010270991362631321,
+            0.0018309251172468066,
+            0.009219296276569366,
+            -0.01625886932015419,
+            0.002569868229329586,
+            0.0031423636246472597,
+            -0.008392359130084515,
+            -0.018760886043310165,
+            -0.010627211071550846,
+            -0.009066631086170673,
+            0.021763307973742485,
+            0.018285928294062614,
+            0.007879232987761497,
+            -0.016852568835020065,
+            0.02106783166527748,
+            -0.007450921926647425,
+            0.007289774715900421,
+            -0.014443847350776196,
+            0.00716255372390151,
+            0.02580045908689499,
+            -0.004596926271915436,
+            0.0035346290096640587,
+            -0.015758465975522995,
+            -0.005805527325719595,
+            -0.0007999032968655229,
+            0.00023284131020773202,
+            0.014647400937974453,
+            0.011186984367668629,
+            0.013773815706372261,
+            0.000118408483103849,
+            0.013197079300880432,
+            -0.020677685737609863,
+            -0.019422436133027077,
+            -0.021474938839673996,
+            -0.013129227794706821,
+            -0.004965867381542921,
+            0.0026292381808161736,
+            0.0047156657092273235,
+            0.002610154915601015,
+            0.009600959718227386,
+            -0.00987236574292183,
+            -0.005843693856149912,
+            -3.207034387742169e-05,
+            0.008379636332392693,
+            -0.0006069511291570961,
+            0.008328747935593128,
+            -0.013790778815746307,
+            0.010245547629892826,
+            0.007484847214072943,
+            -0.0016284313751384616,
+            0.010186177678406239,
+            0.003610961837694049,
+            -0.006878426298499107,
+            -0.008099749684333801,
+            0.00025099681806750596,
+            -0.010423657484352589,
+            0.000959989964030683,
+            -0.009812995791435242,
+            0.006123580504208803,
+            0.009397406131029129,
+            0.01921888254582882,
+            0.02150886505842209,
+            0.014070665463805199,
+            3.816635944531299e-05,
+            0.027717258781194687,
+            0.006450114771723747,
+            0.022051675245165825,
+            -0.008061583153903484,
+            0.030244721099734306,
+            0.0008868377772159874,
+            -0.022713225334882736,
+            0.011967274360358715,
+            0.0088206697255373,
+            0.0026568027678877115,
+            -0.012959599494934082,
+            -0.016267351806163788,
+            -0.006132062058895826,
+            -0.03412920981645584,
+            -0.02098301611840725,
+            0.005177902989089489,
+            0.006657909601926804,
+            0.021339235827326775,
+            0.008964854292571545,
+            -0.00597091531381011,
+            -0.01830288954079151,
+            0.005936989560723305,
+            -0.00392265385016799,
+            -0.010720506310462952,
+            -0.013180116191506386,
+            0.012289567850530148,
+            -0.0028306718450039625,
+            0.015393765643239021,
+            0.01005895622074604,
+            -0.00754421716555953,
+            -0.0031593265011906624,
+            -0.007739289663732052,
+            -0.010313399136066437,
+            0.02135619893670082,
+            -0.024256842210888863,
+            -0.00801069475710392,
+            -0.0072516086511313915,
+            -0.008222730830311775,
+            -0.009092075750231743,
+            0.003127521136775613,
+            0.013332782313227654,
+            -0.0032653443049639463,
+            0.0030787531286478043,
+            0.004486667923629284,
+            -0.0012563093332573771,
+            0.008210008032619953,
+            -0.01225564256310463,
+            -0.037793178111314774,
+            0.024918392300605774,
+            -0.007539976388216019,
+            0.020541982725262642,
+            0.014910325407981873,
+            0.0020323586650192738,
+            -0.0012139022583141923,
+            -0.018523406237363815,
+            0.011721313931047916,
+            -0.03183922544121742,
+            0.009075112640857697,
+            -0.016470905393362045,
+            0.034875571727752686,
+            -0.013366707600653172,
+            -0.020677685737609863,
+            0.01769222877919674,
+            0.0011725553777068853,
+            -0.011687387712299824,
+            -0.027157485485076904,
+            -0.005432345438748598,
+            0.005780083127319813,
+            0.008430524729192257,
+            0.015537949278950691,
+            -0.02746281772851944,
+            -0.010957986116409302,
+            -0.025851348415017128,
+            -0.004618129692971706,
+            0.015597319230437279,
+            -0.007552698720246553,
+            -5.648753813147778e-06,
+            0.0014651642413809896,
+            0.014435365796089172,
+            0.0029515319038182497,
+            0.012035125866532326,
+            0.0008990298374556005,
+            0.0031805301550775766,
+            -0.001866971142590046,
+            0.010220102965831757,
+            0.005729194730520248,
+            0.019965248182415962,
+            -0.01208601426333189,
+            -0.00860863458365202,
+            0.011644980870187283,
+            0.008880039677023888,
+            -0.03041434846818447,
+            -0.00784530770033598,
+            0.002970615169033408,
+            -0.015275025740265846,
+            0.02888769470155239,
+            0.005644380580633879,
+            -0.010194659233093262,
+            0.01263730600476265,
+            -0.023442627862095833,
+            -0.017624378204345703,
+            -0.009473739191889763,
+            -0.0030363460537046194,
+            -0.012968081049621105,
+            -0.0031020769383758307,
+            -0.0016676579834893346,
+            0.025189798325300217,
+            0.020423242822289467,
+            -0.011687387712299824,
+            0.016581162810325623,
+            -0.010330361314117908,
+            0.004957386292517185,
+            -0.005979396402835846,
+            0.019422436133027077,
+            0.024528248235583305,
+            -0.0010442740749567747,
+            0.006908111274242401,
+            -0.0042555490508675575,
+            0.0058097681030631065,
+            -0.010092882439494133,
+            0.011687387712299824,
+            -0.012102977372705936,
+            0.002580470172688365,
+            -0.00922777783125639,
+            -0.004172855522483587,
+            -0.012204754166305065,
+            0.046274591237306595,
+            0.0030617902521044016,
+            0.011695869266986847,
+            0.026920005679130554,
+            0.0027925053145736456,
+            0.009448294527828693,
+            0.030109018087387085,
+            0.020338429138064384,
+            -0.00865528266876936,
+            -0.026869118213653564,
+            0.002639839891344309,
+            -0.014952732250094414,
+            -0.01046606432646513,
+            0.0013220403343439102,
+            -0.009626404382288456,
+            0.013366707600653172,
+            0.0030066610779613256,
+            0.047461990267038345,
+            0.003341676900163293,
+            -0.009923254139721394,
+            -0.015800872817635536,
+            0.00781986303627491,
+            0.0014651642413809896,
+            0.006513725500553846,
+            -0.004999793134629726,
+            -0.01490184385329485,
+            0.0038738856092095375,
+            -0.013332782313227654,
+            -0.017200306057929993,
+            0.01782793179154396,
+            -0.004639333114027977,
+            -0.013807740993797779,
+            -0.006869944743812084,
+            0.01725967600941658,
+            -0.009719699621200562,
+            0.03009205497801304,
+            -0.003854802343994379,
+            -0.013629631139338017,
+            0.022577522322535515,
+            -0.0021309552248567343,
+            0.015800872817635536,
+            -0.0026016735937446356,
+            -0.005122773814946413,
+            -0.017607415094971657,
+            -0.013790778815746307,
+            0.01602138951420784,
+            0.004406094551086426,
+            -0.008332989178597927,
+            -0.0009811935015022755,
+            0.01229804940521717,
+            0.01153472252190113,
+            0.0058648972772061825,
+            0.015393765643239021,
+            -0.020999979227781296,
+            0.011246354319155216,
+            0.02093212865293026,
+            0.003155085723847151,
+            -0.007421236950904131,
+            0.0005931688356213272,
+            0.0012881146976724267,
+            0.0009610501583665609,
+            -0.0022772594820708036,
+            -0.015156285837292671,
+            -0.005402660463005304,
+            0.0017885180423036218,
+            0.013307337649166584,
+            0.007552698720246553,
+            0.005033718887716532,
+            -0.02188204601407051,
+            0.0017196065746247768,
+            -0.013672038912773132,
+            0.006327134557068348,
+            -0.001542557030916214,
+            0.0017259676242247224,
+            0.017624378204345703,
+            0.005593492183834314,
+            0.00413468899205327,
+            0.005551085341721773,
+            -0.02751370519399643,
+            -0.020541982725262642,
+            -0.004864090587943792,
+            0.00905814953148365,
+            -0.006797852925956249,
+            0.011458389461040497,
+            -0.01115305908024311,
+            0.012433752417564392,
+            0.009075112640857697,
+            0.007026851177215576,
+            -0.010126807726919651,
+            -0.006255042273551226,
+            -0.011008874513208866,
+            -0.010805320926010609,
+            4.200950206723064e-05,
+            -0.010974949225783348,
+            -0.011661943979561329,
+            0.015546430833637714,
+            0.00383783970028162,
+            0.004359446465969086,
+            -0.015173248946666718,
+            -0.0025529053527861834,
+            -0.011076726019382477,
+            -0.007018369622528553,
+            0.0062295980751514435,
+            0.018947478383779526,
+            0.0024574894923716784,
+            -0.0027734222821891308,
+            -0.008256656117737293,
+            -0.006445874460041523,
+            -0.0036300448700785637,
+            -0.022424858063459396,
+            -0.03643615171313286,
+            0.01325644925236702,
+            0.007641753647476435,
+            0.008973335847258568,
+            0.001959206536412239,
+            -0.012577936053276062,
+            -0.028735028579831123,
+            -0.005127014592289925,
+            0.0134684843942523,
+            0.02369706891477108,
+            -0.022475745528936386,
+            0.011907904408872128,
+            0.023849735036492348,
+            0.012951118871569633,
+            -0.0031444840133190155,
+            0.022662336006760597,
+            0.01256097387522459,
+            0.01911710575222969,
+            -0.0024532489478588104,
+            -0.006314412225037813,
+            0.0070607769303023815,
+            -0.012391345575451851,
+            -0.0013050774578005075,
+            -0.009354999288916588,
+            0.008829151280224323,
+            0.0027607001829892397,
+            -0.0023832772858440876,
+            0.007917399518191814,
+            -0.030482200905680656,
+            0.010661136358976364,
+            -0.022153452038764954,
+            0.013366707600653172,
+            0.014359032735228539,
+            -0.008557746186852455,
+            0.00946525763720274,
+            0.0007617369410581887,
+            0.0053093647584319115,
+            0.02317122183740139,
+            -0.002353592310100794,
+            0.009185370989143848,
+            -0.010576322674751282,
+            -0.0018553092377260327,
+            -0.018252002075314522,
+            0.003301390213891864,
+            0.003998986445367336,
+            -0.004596926271915436,
+            -0.012425270862877369,
+            0.02059287205338478,
+            -0.00473262881860137,
+            -0.0001673754013609141,
+            -0.024239879101514816,
+            -0.00029923487454652786,
+            -0.005843693856149912,
+            0.021525828167796135,
+            -0.017726154997944832,
+            -0.0005565927713178098,
+            0.009388924576342106,
+            0.010211621411144733,
+            0.0017916986253112555,
+            -0.0319579653441906,
+            0.023001592606306076,
+            -0.02993938885629177,
+            -0.025427278131246567,
+            -0.005025237333029509,
+            -0.004864090587943792,
+            -0.006564613897353411,
+            -0.022323081269860268,
+            0.012459196150302887,
+            0.012331975623965263,
+            -0.004196179565042257,
+            -0.014723733998835087,
+            -0.0018319853115826845,
+            -0.003093595616519451,
+            -0.00624656118452549,
+            -0.005093088839203119,
+            -0.0032377797178924084,
+            -0.005941230338066816,
+            0.0021161127369850874,
+            -0.010432139039039612,
+            -0.007739289663732052,
+            -0.00784106645733118,
+            0.007213442120701075,
+            0.0018574295099824667,
+            0.01206056959927082,
+            -0.002894282341003418,
+            -0.020677685737609863,
+            0.011950311250984669,
+            0.001987831201404333,
+            0.015724539756774902,
+            0.004719906486570835,
+            0.006738482974469662,
+            0.010941023007035255,
+            0.013790778815746307,
+            -0.011526240967214108,
+            -0.00982147641479969,
+            -0.01648786850273609,
+            -0.005907304584980011,
+            0.013739889487624168,
+            -0.012026644311845303,
+            -0.0040074680000543594,
+            -0.00012821512063965201,
+            -0.020083988085389137,
+            0.005440826993435621,
+            0.011102170683443546,
+            0.005152458790689707,
+            -0.011144577525556087,
+            -0.026292381808161736,
+            0.013790778815746307,
+            0.0058097681030631065,
+            -0.013392152264714241,
+            -0.003216576064005494,
+            0.01682712510228157,
+            0.026190605014562607,
+            0.02641112171113491,
+            0.011195465922355652,
+            0.015291988849639893,
+            -0.009007261134684086,
+            0.031143750995397568,
+            0.0076163094490766525,
+            0.007221923675388098,
+            -0.024765728041529655,
+            0.01574150286614895,
+            -0.0015467978082597256,
+            0.0022221303079277277,
+            -0.00020103600400034338,
+            -0.028582364320755005,
+            -0.009728181175887585,
+            -0.036741483956575394,
+            -0.015953538939356804,
+            -0.013205560855567455,
+            -0.004758073017001152,
+            0.0037212201859802008,
+            0.006416189484298229,
+            0.00021481829753611237,
+            0.011840052902698517,
+            -0.0017810967983677983,
+            0.009142964147031307,
+            0.006869944743812084,
+            -0.015444654040038586,
+            0.0025062577333301306,
+            -0.009354999288916588,
+            -0.0076884012669324875,
+            -0.0030087814666330814,
+            -0.03156781941652298,
+            0.0018690915312618017,
+            -0.02483357861638069,
+            -0.010627211071550846,
+            -0.008973335847258568,
+            -0.004953145515173674,
+            -0.0028985231183469296,
+            0.0023896382190287113,
+            -0.017794005572795868,
+            -0.018506444990634918,
+            -0.0110852075740695,
+            -0.020491095259785652,
+            -0.013460002839565277,
+            -0.0070607769303023815,
+            -0.003810275113210082,
+            -0.002542303642258048,
+            0.03251773864030838,
+            0.006717279553413391,
+            -0.003307751379907131,
+            -0.032348111271858215,
+            0.014223330654203892,
+            -0.005381457041949034,
+            -0.014842473901808262,
+            -0.00867648608982563,
+            0.014995139092206955,
+            -0.022662336006760597,
+            0.018319852650165558,
+            0.00943981297314167,
+            -0.021254422143101692,
+            0.013621150515973568,
+            0.027903851121664047,
+            0.025732608512043953,
+            0.010220102965831757,
+            -0.00850685779005289,
+            -6.033896170265507e-06,
+            0.002220009919255972,
+            0.001824564067646861,
+            0.015809355303645134,
+            0.011509277857840061,
+            0.015308951027691364,
+            -0.002446887781843543,
+            0.015436172485351562,
+            -0.02780207432806492,
+            -0.0015902650775387883,
+            -0.0076163094490766525,
+            -0.005894582252949476,
+            0.008405080996453762,
+            -0.008108231239020824,
+            -0.018913552165031433,
+            -0.025953125208616257,
+            -0.00010051800200017169,
+            0.024002401158213615,
+            -0.016886495053768158,
+            -0.014359032735228539,
+            -0.010881653986871243,
+            -0.0004110835143364966,
+            -0.007366107776761055,
+            0.010576322674751282,
+            0.022916778922080994,
+            -0.0005102100549265742,
+            0.0060218037106096745,
+            0.012289567850530148,
+            -0.02717444859445095,
+            0.01340063288807869,
+            -0.017895782366394997,
+            -0.0074254777282476425,
+            -0.00798949133604765,
+            -0.010881653986871243,
+            0.0033374361228197813,
+            -0.011526240967214108,
+            0.014800066128373146,
+            0.01606379635632038,
+            0.0034158893395215273,
+            -0.013137709349393845,
+            -0.0027522186283022165,
+            -0.020423242822289467,
+            0.00011118603288196027,
+            -0.007752011995762587,
+            0.003678813111037016,
+            0.009957179427146912,
+            0.004340363200753927,
+            0.021525828167796135,
+            0.008786744438111782,
+            -0.03680933639407158,
+            -0.0021691215224564075,
+            0.012925674207508564,
+            -0.00213307561352849,
+            -0.028378808870911598,
+            -0.012713639065623283,
+            0.009134482592344284,
+            0.00038484414108097553,
+            0.003937496338039637,
+            0.003950218204408884,
+            -0.004558759741485119,
+            0.017471712082624435,
+            -0.04043937847018242,
+            -0.01840466819703579,
+            -0.00040207200800068676,
+            -0.016470905393362045,
+            -0.007103183772414923,
+            0.008481413125991821,
+            0.0026037939824163914,
+            0.010499989613890648,
+            0.009668811224400997,
+            -0.018438592553138733,
+            0.01178068295121193,
+            -0.0009260643273591995,
+            -0.01373140886425972,
+            0.0013167394790798426,
+            0.0007988431607373059,
+            0.009448294527828693,
+            0.0021055107936263084,
+            -0.00862983800470829,
+            -0.00786651112139225,
+            -0.0100165493786335,
+            0.004842887166887522,
+            0.029684947803616524,
+            -0.00023959993268363178,
+            -0.005856416188180447,
+            0.005016756244003773,
+            -0.010381249710917473,
+            -0.013273412361741066,
+            -0.01432510744780302,
+            0.0006949458038434386,
+            0.013460002839565277,
+            0.0073067378252744675,
+            0.014749177731573582,
+            -0.011170021258294582,
+            -0.005071885418146849,
+            0.011390537954866886,
+            -0.01170435082167387,
+            0.013409114442765713,
+            -0.01625886932015419,
+            -0.015995945781469345,
+            -0.007993732579052448,
+            0.0025401832535862923,
+            0.015139322727918625,
+            0.00922777783125639,
+            -0.000635575910564512,
+            0.0333828441798687,
+            -0.008803707547485828,
+            0.020474132150411606,
+            -0.01425725594162941,
+            -0.01697130873799324,
+            0.005283920560032129,
+            -0.016564199700951576,
+            -0.007531495299190283,
+            -0.004859849810600281,
+            -0.020558945834636688,
+            0.007166794501245022,
+            0.018726961687207222,
+            -0.004189818166196346,
+            0.016148611903190613,
+            0.008167601190507412,
+            -0.0039523388259112835,
+            0.0030893548391759396,
+            -0.00927018467336893,
+            -0.012069051153957844,
+            0.004098643083125353,
+            0.0146643640473485,
+            -0.01759045198559761,
+            0.017522599548101425,
+            0.004516352899372578,
+            0.007285534404218197,
+            0.015512505546212196,
+            0.012985044158995152,
+            -0.02680126577615738,
+            -0.011865497566759586,
+            0.022441819310188293,
+            -0.01206056959927082,
+            0.005597732961177826,
+            0.011840052902698517,
+            0.0038993298076093197,
+            -0.009397406131029129,
+            -0.013510892167687416,
+            0.0011259076418355107,
+            -0.00991477258503437,
+            0.013544817455112934,
+            -0.00015120769967325032,
+            -0.041083965450525284,
+            0.007997972890734673,
+            -0.006441633682698011,
+            0.003487981390208006,
+            0.0027331355959177017,
+            -0.0058394530788064,
+            0.023256035521626472,
+            0.02259448543190956,
+            -0.015504023991525173,
+            -0.00588610116392374,
+            -0.013960406184196472,
+            0.012798452749848366,
+            0.008693449199199677,
+            0.007976769469678402,
+            -0.02493535540997982,
+            -0.003610961837694049,
+            0.007213442120701075,
+            -0.0064373929053545,
+            -0.00473262881860137,
+            0.014545624144375324,
+            -0.009346517734229565,
+            -0.0013951925793662667,
+            -0.003825117601081729,
+            0.0015043907333165407,
+            0.013044414110481739,
+            0.0036703317891806364,
+            0.015529467724263668,
+            -0.01950725167989731,
+            0.013833185657858849,
+            -0.0058648972772061825,
+            0.0010368528310209513,
+            0.009897809475660324,
+            -0.014197885990142822,
+            -0.026139715686440468,
+            0.01630127616226673,
+            0.00692507391795516,
+            0.00037477246951311827,
+            -0.023086408153176308,
+            -0.00645859632641077,
+            -0.01993132196366787,
+            0.0029409301932901144,
+            -0.00028465743525885046,
+            0.0038738856092095375,
+            0.025088021531701088,
+            -0.000675862655043602,
+            -0.015860242769122124,
+            -0.005797046236693859,
+            0.001527714659459889,
+            0.0015923853497952223,
+            0.007145591080188751,
+            -0.0193036962300539,
+            -0.025088021531701088,
+            -0.0050464412197470665,
+            -0.013858629390597343,
+            0.018319852650165558,
+            -0.017760079354047775,
+            0.015716059133410454,
+            -0.003555832663550973,
+            0.008922447450459003,
+            0.00562741793692112,
+            -0.0029812168795615435,
+            0.03294181078672409,
+            -0.015597319230437279,
+            0.010304917581379414,
+            0.007548457942903042,
+            -0.011382057331502438,
+            -0.011449907906353474,
+            0.01774311624467373,
+            -0.010508471168577671,
+            0.0032505018170922995,
+            0.0038463210221379995,
+            -0.00929562933743,
+            0.023476552218198776,
+            0.0027500982396304607,
+            0.014757659286260605,
+            -0.01070354413241148,
+            -0.019914358854293823,
+            0.0032907885033637285,
+            0.010686581023037434,
+            0.0013496049214154482,
+            0.00812519434839487,
+            -0.02888769470155239,
+            0.00483864638954401,
+            -0.009643367491662502,
+            0.02106783166527748,
+            0.009728181175887585,
+            0.0018097215797752142,
+            -0.02999027818441391,
+            0.0008857776410877705,
+            -0.014070665463805199,
+            0.02651289850473404,
+            0.005165180657058954,
+            0.003301390213891864,
+            -0.01139901950955391,
+            -0.003975662402808666,
+            0.012823897413909435,
+            -0.005754638928920031,
+            -0.009643367491662502,
+            -0.0038039139471948147,
+            0.011178502812981606,
+            0.004821683745831251,
+            -0.013426077552139759,
+            -0.013934962451457977,
+            -0.011882460676133633,
+            0.010160733014345169,
+            0.0012520686723291874,
+            0.0037424236070364714,
+            -0.004323400557041168,
+            0.007484847214072943,
+            -0.011763720773160458,
+            0.022950705140829086,
+            0.009668811224400997,
+            0.01592809520661831,
+            0.01893051527440548,
+            -0.008176082745194435,
+            0.009643367491662502,
+            0.008837632834911346,
+            0.01046606432646513,
+            0.016174055635929108,
+            0.022373968735337257,
+            0.021899009123444557,
+            0.019710805267095566,
+            -0.021949898451566696,
+            -0.0010585864074528217,
+            -0.006602780427783728,
+            -0.0073237004689872265,
+            -0.011958792805671692,
+            -0.018998365849256516,
+            0.001702643814496696,
+            -0.03372209891676903,
+            -0.019015328958630562,
+            -0.0017206667689606547,
+            0.007090461906045675,
+            0.013027451001107693,
+            -0.014333589002490044,
+            0.012985044158995152,
+            0.008583190850913525,
+            0.00036417070077732205,
+            -0.014486254192888737,
+            -0.010949504561722279,
+            -0.012544010765850544,
+            0.004367928020656109,
+            0.021610641852021217,
+            0.004153772257268429,
+            0.0049446639604866505,
+            -0.01301896944642067,
+            0.00937196146696806,
+            0.02822614461183548,
+            0.0015351359033957124,
+            -0.00953310914337635,
+            0.0058479346334934235,
+            -0.004073198884725571,
+            0.008752819150686264,
+            -0.012162347324192524,
+            -0.010669617913663387,
+            0.000553412246517837,
+            0.005754638928920031,
+            -0.013748371042311192,
+            0.02140708826482296,
+            -0.004196179565042257,
+            0.018285928294062614,
+            -2.4102455427055247e-06,
+            -0.0007198599632829428,
+            0.0024511285591870546,
+            0.003971422091126442,
+            0.02198382467031479,
+            0.005275439005345106,
+            -0.0008550324710085988,
+            -0.023730995133519173,
+            -0.0005356542533263564,
+            -0.0029939389787614346,
+            -0.020134875550866127,
+            0.004906497895717621,
+            0.0069589996710419655,
+            -0.004359446465969086,
+            0.02717444859445095,
+            0.013960406184196472,
+            0.0013156792847439647,
+            0.005669824779033661,
+            0.0014089748729020357,
+            0.007366107776761055,
+            -0.008498376235365868,
+            -0.00946525763720274,
+            0.0041262079030275345,
+            -0.0027755426708608866,
+            0.0051185330376029015,
+            -0.027089634910225868,
+            0.013570261187851429,
+            0.0050634038634598255,
+            0.0038166360463947058,
+            0.006208394654095173,
+            0.008693449199199677,
+            -0.005996359512209892,
+            0.0012128421803936362,
+            0.006975962780416012,
+            0.008260896429419518,
+            0.001960266614332795,
+            0.031075898557901382,
+            -0.027428891509771347,
+            -0.00036019503022544086,
+            -0.005945470649749041,
+            -0.005233032163232565,
+            -0.0032547423616051674,
+            -0.01816718839108944,
+            -0.013782297261059284,
+            -0.00392053322866559,
+            -0.001568001345731318,
+            0.023900622501969337,
+            0.013578742742538452,
+            0.01611468568444252,
+            0.01149231567978859,
+            -0.006344097200781107,
+            -0.018862662836909294,
+            0.009176889434456825,
+            -0.0035367493983358145,
+            -0.00619991309940815,
+            0.031686559319496155,
+            0.016199499368667603,
+            -0.0036830538883805275,
+            0.01887962594628334,
+            -0.0051354956813156605,
+            0.011178502812981606,
+            -0.016174055635929108,
+            0.001324160723015666,
+            -0.006869944743812084,
+            -0.0031741689890623093,
+            0.013426077552139759,
+            -0.002298463135957718,
+            0.007930121384561062,
+            -0.0024574894923716784,
+            -0.00526695791631937,
+            0.0076714386232197285,
+            -0.0031317619141191244,
+            -0.004162253811955452,
+            0.013977369293570518,
+            -0.003746664384379983,
+            -0.011161540634930134,
+            -0.015368320979177952,
+            0.011127614416182041,
+            -0.01328189391642809,
+            0.006216876208782196,
+            -0.0036745723336935043,
+            -0.007925881072878838,
+            -0.002419323194772005,
+            0.03060094080865383,
+            -0.0020779462065547705,
+            0.017794005572795868,
+            0.004851368721574545,
+            -0.02203471213579178,
+            -0.02083035185933113,
+            -0.008167601190507412,
+            0.03151693195104599,
+            -0.00507612619549036,
+            -0.009159926325082779,
+            -0.019795618951320648,
+            -0.01091557927429676,
+            0.004268271382898092,
+            -0.012823897413909435,
+            -0.0003002950397785753,
+            -5.681884431396611e-05,
+            0.003638526424765587,
+            -0.0007946024416014552,
+            -0.01408762764185667,
+            -0.004656296223402023,
+            0.00031142690568231046,
+            0.01253552921116352,
+            0.005059163086116314,
+            0.002565627684816718,
+            0.014308144338428974,
+            0.005470511503517628,
+            -0.02751370519399643,
+            -0.006551892030984163,
+            0.013315819203853607,
+            0.012527047656476498,
+            0.012348937802016735,
+            -0.002095969393849373,
+            -0.007900436408817768,
+            -0.02659771218895912,
+            -0.010381249710917473,
+            -0.0362325981259346,
+            -0.0013803500914946198,
+            -0.023052481934428215,
+            -0.018896589055657387,
+            -0.02103390544652939,
+            -0.021254422143101692,
+            -0.010330361314117908,
+            -0.01067809946835041,
+            0.025681721046566963,
+            0.010983430780470371,
+            0.0026716452557593584,
+            0.00791315920650959,
+            -0.023374775424599648,
+            0.0027119319420307875,
+            -0.009541589766740799,
+            -0.006963240448385477,
+            0.02359529212117195,
+            0.00011436655768193305,
+            0.016793198883533478,
+            -0.0014216969721019268,
+            -0.011907904408872128,
+            0.007395792752504349,
+            -0.007701123598963022,
+            -0.007662957068532705,
+            0.010932542383670807,
+            -0.012628824450075626,
+            -0.013485447503626347,
+            0.016428498551249504,
+            0.026156678795814514,
+            -0.002173362299799919,
+            0.0005157759878784418,
+            -0.024155065417289734,
+            -0.0011969394981861115,
+            -0.0020355391316115856,
+            0.0009202333749271929,
+            -0.0015828438336029649,
+            0.007421236950904131,
+            0.002241213573142886,
+            0.0034116485621780157,
+            0.02006702497601509,
+            -0.012603380717337132,
+            0.01816718839108944,
+            0.007506051100790501,
+            -0.001392011996358633,
+            0.021491901949048042,
+            0.010949504561722279,
+            0.0016284313751384616,
+            -0.023052481934428215,
+            -0.004363687243312597,
+            -0.010033512488007545,
+            -0.001446080976165831,
+            0.014435365796089172,
+            0.013154672458767891,
+            -0.020016135647892952,
+            -0.025257648900151253,
+            4.452741995919496e-05,
+            -0.001272212015464902,
+            0.009829957969486713,
+            -0.015444654040038586,
+            -0.0020620436407625675,
+            0.0006551892147399485,
+            -0.018557332456111908,
+            0.015571875497698784,
+            0.013417595997452736,
+            0.0033925652969628572,
+            0.01332430075854063,
+            -0.005283920560032129,
+            0.015275025740265846,
+            -0.007743530441075563,
+            0.010203140787780285,
+            -0.003099956549704075,
+            -0.013146190904080868,
+            0.006288968026638031,
+            -0.00784106645733118,
+            -0.01163649931550026,
+            -0.010118326172232628,
+            -0.017369935289025307,
+            -0.021542789414525032,
+            -0.020915165543556213,
+            -0.019235845655202866,
+            0.015291988849639893,
+            -0.011365094222128391,
+            0.009261703118681908,
+            0.004961627069860697,
+            -0.006081173662096262,
+            0.021101756021380424,
+            0.005890341475605965,
+            -0.0017641340382397175,
+            0.007209201343357563,
+            -0.003203853964805603,
+            -0.0008216369315050542,
+            -0.004393372219055891,
+            0.032348111271858215,
+            -0.0013315819669514894,
+            0.024494322016835213,
+            -0.0006445874460041523,
+            0.008205767720937729,
+            0.0016125286929309368,
+            0.0015892048832029104,
+            -0.023137295618653297,
+            0.012908711098134518,
+            -0.011314205825328827,
+            -0.014571068808436394,
+            -0.02412113919854164,
+            -0.00022078178881201893,
+            0.006178709678351879,
+            0.017047641798853874,
+            0.0041431705467402935,
+            -0.0009424970485270023,
+            0.008646801114082336,
+            -0.01950725167989731,
+            0.002160640200600028,
+            0.009007261134684086,
+            -0.00156694115139544,
+            -0.008290581405162811,
+            -0.02030450478196144,
+            -0.0019570861477404833,
+            -0.027394965291023254,
+            0.005568047985434532,
+            0.0007946024416014552,
+            -0.01504602748900652,
+            0.010949504561722279,
+            -0.007904677651822567,
+            0.021339235827326775,
+            0.01830288954079151,
+            0.0011916386429220438,
+            -0.006581577006727457,
+            -0.015300470404326916,
+            -0.015962019562721252,
+            0.013527854345738888,
+            0.015308951027691364,
+            0.011110651306807995,
+            0.006178709678351879,
+            0.005398419685661793,
+            -0.004329761490225792,
+            0.01446929108351469,
+            0.00937196146696806,
+            -0.028599325567483902,
+            1.502469149272656e-05,
+            -0.010983430780470371,
+            0.023680105805397034,
+            -0.02330692484974861,
+            0.014002813957631588,
+            -0.020999979227781296,
+            0.01935458555817604,
+            -0.0003612551954574883,
+            0.012722120620310307,
+            0.009278666228055954,
+            8.89223229023628e-05,
+            -0.013375189155340195,
+            0.005852175410836935,
+            0.001298716408200562,
+            -0.0011555926175788045,
+            0.0048174429684877396,
+            0.022492708638310432,
+            -0.02951531857252121,
+            -0.004893775563687086,
+            -0.005190624855458736,
+            -2.678602686501108e-05,
+            -0.010313399136066437,
+            -0.024188991636037827,
+            0.0033586397767066956,
+            -0.015012102201581001,
+            0.002998179756104946,
+            0.018862662836909294,
+            0.00516094034537673,
+            0.02111871913075447,
+            -0.02330692484974861,
+            -0.009829957969486713,
+            -0.013553299009799957,
+            0.017437785863876343,
+            -0.0029536522924900055,
+            0.0016825003549456596,
+            0.006517966277897358,
+            -0.013909517787396908,
+            0.0058097681030631065,
+            -0.0073237004689872265,
+            -0.00441033486276865,
+            0.06323742121458054,
+            -0.0005393648752942681,
+            -0.007014128845185041,
+            -0.01022858452051878,
+            -0.010092882439494133,
+            0.015317432582378387,
+            0.006929314695298672,
+            -0.003528268076479435,
+            -0.010695062577724457,
+            -0.004397612996399403,
+            -0.023256035521626472,
+            -0.01648786850273609,
+            -0.004217382986098528,
+            -0.0074254777282476425,
+            0.01178068295121193,
+            -0.000947267864830792,
+            -0.009999586269259453,
+            0.003445574315264821,
+            -0.0054111420176923275,
+            -0.00846020970493555,
+            0.006378022953867912,
+            -0.01615709252655506,
+            0.001985711045563221,
+            0.008939409628510475,
+            -0.015554912388324738,
+            0.018252002075314522,
+            -0.014528661035001278,
+            -0.0048174429684877396,
+            0.022170415148139,
+            -0.021288348361849785,
+            0.005394178908318281,
+            0.0011725553777068853,
+            0.01935458555817604,
+            -0.019625991582870483,
+            -0.009456776082515717,
+            0.03173745051026344,
+            -0.012645787559449673,
+            0.008400839753448963,
+            0.00853230245411396,
+            0.006759686395525932,
+            0.0018902950687333941,
+            0.0011736155720427632,
+            0.007412755396217108,
+            0.014859436079859734,
+            -0.01636064611375332,
+            -0.01950725167989731,
+            0.007832585833966732,
+            -0.004448501393198967,
+            -0.0036003601271659136,
+            0.010372769087553024,
+            -0.009278666228055954,
+            0.0025529053527861834,
+            -0.0053560128435492516,
+            0.01470677088946104,
+            -0.005682547111064196,
+            -0.006187191233038902,
+            0.018099335953593254,
+            -0.01039821282029152,
+            0.0053093647584319115,
+            0.02927783876657486,
+            -0.012544010765850544,
+            -0.016911938786506653,
+            -0.0009944457560777664,
+            -0.00522030983120203,
+            -0.013332782313227654,
+            0.01740385964512825,
+            -0.01702219620347023,
+            -0.005525640677660704,
+            -0.02393454872071743,
+            -0.004520593211054802,
+            0.005826731212437153,
+            0.0190662182867527,
+            0.013273412361741066,
+            -0.019337622448801994,
+            0.003812395269051194,
+            -0.011271798983216286,
+            0.023493515327572823,
+            0.0031953726429492235,
+            0.012196272611618042,
+            -0.0010442740749567747,
+            -0.03894665092229843,
+            -0.012764527462422848,
+            -0.03562193736433983,
+            0.014825510792434216,
+            -0.004813202191144228,
+            -0.024952318519353867,
+            -0.02627541869878769,
+            -0.0021161127369850874,
+            -0.014723733998835087,
+            0.007187997922301292,
+            0.012611862272024155,
+            0.01425725594162941,
+            0.012645787559449673,
+            0.03775925189256668,
+            0.0021277745254337788,
+            -0.0004309618379920721,
+            -0.0008476112852804363,
+            0.022289155051112175,
+            0.01149231567978859,
+            -0.01720878854393959,
+            0.009142964147031307,
+            0.0043721687979996204,
+            -0.012187791056931019,
+            -0.004596926271915436,
+            -0.017709191888570786,
+            -0.016479386016726494,
+            0.014842473901808262,
+            -0.01877784915268421,
+            0.007561180274933577,
+            0.01578390970826149,
+            -0.010169214569032192,
+            0.009185370989143848,
+            -0.003369241487234831,
+            -0.005173662211745977,
+            -0.01490184385329485,
+            0.0010071678552776575,
+            -0.008693449199199677,
+            0.01582631841301918,
+            -0.012739082798361778,
+            0.01864214614033699,
+            -0.014172442257404327,
+            -0.00826937798410654,
+            0.021339235827326775,
+            0.022475745528936386,
+            -0.009567034430801868,
+            0.008710411377251148,
+            -0.021525828167796135,
+            0.0008878979715518653,
+            0.014630438759922981,
+            -0.019490288570523262,
+            0.02856540121138096,
+            -0.012925674207508564,
+            0.014613475650548935,
+            0.0011322686914354563,
+            -0.009287147782742977,
+            0.002555025741457939,
+            -0.010550878010690212,
+            0.013672038912773132,
+            -0.0019411834655329585,
+            0.012094495818018913,
+            -0.01615709252655506,
+            0.015809355303645134,
+            -0.011653462424874306,
+            0.012510084547102451,
+            0.009244740940630436,
+            -0.005860656965523958,
+            -0.0005868077860213816,
+            -0.013069857843220234,
+            0.0052542355842888355,
+            -0.003812395269051194,
+            0.023849735036492348,
+            0.006399226374924183,
+            -0.015809355303645134,
+            0.0014821270015090704,
+            0.0028094681911170483,
+            0.015537949278950691,
+            0.011314205825328827,
+            0.007086221128702164,
+            -0.013095302507281303,
+            0.01935458555817604,
+            -0.00987236574292183,
+            0.008998779579997063,
+            -0.0011036439100280404,
+            0.011407501064240932,
+            -0.022577522322535515,
+            -0.00588610116392374,
+            0.007048054598271847,
+            -0.015291988849639893,
+            0.012857822701334953,
+            -0.0076163094490766525,
+            0.0006000599823892117,
+            0.007137109525501728,
+            0.006590058095753193,
+            0.019439399242401123,
+            0.004276752937585115,
+            0.002393878996372223,
+            0.003301390213891864,
+            -0.011619536206126213,
+            -0.0159789826720953,
+            -0.008668004535138607,
+            -0.020185764878988266,
+            0.005088848061859608,
+            0.010237066075205803,
+            0.02064376138150692,
+            0.0008592731901444495,
+            -0.008718892931938171,
+            -0.00801069475710392,
+            -0.02011791244149208,
+            0.00045190032687969506,
+            0.021135682240128517,
+            0.012416789308190346,
+            -0.011543204076588154,
+            -0.011415982618927956,
+            0.020491095259785652,
+            -0.00654341047629714,
+            -0.02103390544652939,
+            -0.012934155762195587,
+            -0.006836019456386566,
+            -0.0027267744299024343,
+            -0.007981009781360626,
+            0.016581162810325623,
+            0.004775035660713911,
+            0.0076332720927894115,
+            0.03613082319498062
+          ],
+          "title": "E Relation of Upstream to Downstream Model Performance",
+          "keyphrases": [
+            "Upstream to Downstream Model Performance"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "f58344aa-a620-4e81-9be1-0f149a64c192",
+      "type": "child",
+      "source": {
+        "id": "65742957-6bda-446d-a61a-0fe20313c760",
+        "properties": {
+          "page_content": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity\n\nWilliam Fedus∗ liamfedus@google.com Barret Zoph∗ barretzoph@google.com Noam Shazeer noam@google.com Google, Mountain View, CA 94043, USA Editor: Alexander Clark\n\nAbstract\n\nIn deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) models defy this and instead select different parameters for each incoming example. The result is a sparsely-activated model—with an outrageous number of parameters—but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs, and training instability. We address these with the introduction of the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques mitigate the instabilities, and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large (Raffel et al., 2019) to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the \"Colossal Clean Crawled Corpus\", and achieve a 4x speedup over the T5-XXL model.12 Keywords: mixture-of-experts, natural language processing, sparsity, large-scale machine learning, distributed computing arXiv:2101.03961v3 [cs.LG] 16 Jun 2022 Contents\n\n1 Introduction 3 2 Switch Transformer 4 2.1 Simplifying Sparse Routing 5 2.2 Efficient Sparse Routing 6 2.3 Putting It All Together: The Switch Transformer 8 2.4 Improved Training and Fine-Tuning Techniques 8 3 Scaling Properties 11 3.1 Scaling Results on a Step-Basis 12 3.2 Scaling Results on a Time-Basis 13 3.3 Scaling Versus a Larger Dense Model 13 4 Downstream Results 14 4.1 Fine-Tuning 14 4.2 Distillation 16 4.3 Multilingual Learning 17 5 Designing Models with Data, Model, and Expert-Parallelism 18 5.1 Data Parallelism 20 5.2 Model Parallelism 20 5.3 Model and Data Parallelism 21 5.4 Expert and Data Parallelism 22 5.5 Expert, Model and Data Parallelism 22 5.6 Towards Trillion Parameter Models 22 6 Related Work 24 7 Discussion 25 8 Future Work 26 9 Conclusion 27 A Switch for Attention 27 B Preventing Token Dropping with No-Token-Left-Behind 29\n\nC Encouraging Exploration Across Experts 29 D Switch Transformers in Lower Compute Regimes 29 E Relation of Upstream to Downstream Model Performance 32 F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/switch_transformers.md"
+          },
+          "summary": "The Switch Transformer is a sparsely-activated model that simplifies the Mixture of Experts (MoE) approach, allowing for efficient scaling to trillion-parameter models with reduced communication and computational costs. It achieves significant speedups in pre-training and fine-tuning by activating only a subset of neural network weights for each input, maintaining a constant computational cost. The model demonstrates superior scaling properties and improved performance across various natural language processing tasks, including multilingual settings. It also allows for effective distillation into smaller dense models, preserving a portion of the quality gains. The architecture is designed to maximize parameter count while keeping floating point operations per example constant, making it suitable for both large-scale and smaller-scale applications.",
+          "headlines": {
+            "1 Introduction": [],
+            "2 Switch Transformer": [
+              "2.1 Simplifying Sparse Routing",
+              "2.2 Efficient Sparse Routing",
+              "2.3 Putting It All Together: The Switch Transformer",
+              "2.4 Improved Training and Fine-Tuning Techniques"
+            ],
+            "3 Scaling Properties": [
+              "3.1 Scaling Results on a Step-Basis",
+              "3.2 Scaling Results on a Time-Basis",
+              "3.3 Scaling Versus a Larger Dense Model"
+            ],
+            "4 Downstream Results": [
+              "4.1 Fine-Tuning",
+              "4.2 Distillation",
+              "4.3 Multilingual Learning"
+            ],
+            "5 Designing Models with Data, Model, and Expert-Parallelism": [
+              "5.1 Data Parallelism",
+              "5.2 Model Parallelism",
+              "5.3 Model and Data Parallelism",
+              "5.4 Expert and Data Parallelism",
+              "5.5 Expert, Model and Data Parallelism",
+              "5.6 Towards Trillion Parameter Models"
+            ],
+            "6 Related Work": [],
+            "7 Discussion": [],
+            "8 Future Work": [],
+            "9 Conclusion": [],
+            "A Switch for Attention": [],
+            "B Preventing Token Dropping with No-Token-Left-Behind": [],
+            "C Encouraging Exploration Across Experts": [],
+            "D Switch Transformers in Lower Compute Regimes": [],
+            "E Relation of Upstream to Downstream Model Performance": [],
+            "F Pseudo Code for Switch Transformers": []
+          },
+          "summary_embedding": [
+            -0.0060447100549936295,
+            -0.00390331051312387,
+            -0.01784370467066765,
+            -0.0021820589900016785,
+            0.013227885589003563,
+            0.011338187381625175,
+            0.0165580902248621,
+            0.024240797385573387,
+            -0.005986624862998724,
+            0.03596621751785278,
+            -0.004852031823247671,
+            -0.0068811336532235146,
+            -0.003574162255972624,
+            0.014242436736822128,
+            0.011578272096812725,
+            -0.033302053809165955,
+            -0.024519605562090874,
+            0.01050176378339529,
+            0.016093410551548004,
+            -0.006795942317694426,
+            0.007473600562661886,
+            -0.030513975769281387,
+            -0.06124480068683624,
+            -0.006838538218289614,
+            -0.0026099516544491053,
+            0.01029265858232975,
+            0.0010329445358365774,
+            -0.010873507708311081,
+            -0.053469158709049225,
+            0.023311438038945198,
+            0.025294071063399315,
+            0.005092116538435221,
+            -0.029584616422653198,
+            0.0009361362317577004,
+            -0.05161044001579285,
+            -0.04448534920811653,
+            0.0379488542675972,
+            0.012561844661831856,
+            -0.04364892467856407,
+            0.039249956607818604,
+            -0.004875265993177891,
+            -0.0022440163884311914,
+            -0.021499186754226685,
+            -0.0034909071400761604,
+            -0.007039899472147226,
+            -0.009851212613284588,
+            -0.02874819189310074,
+            0.0014037203509360552,
+            0.021886419504880905,
+            -0.0056962003000080585,
+            0.023264970630407333,
+            -0.024271776899695396,
+            0.01936165988445282,
+            0.04240977764129639,
+            -0.06957805901765823,
+            0.024659009650349617,
+            0.017549408599734306,
+            0.05120771750807762,
+            0.06511713564395905,
+            0.018339363858103752,
+            0.021669568493962288,
+            -0.014141756109893322,
+            -0.017766257748007774,
+            -0.011872569099068642,
+            0.004673904739320278,
+            -0.014970434829592705,
+            -0.01891246810555458,
+            0.008449427783489227,
+            -0.020461400970816612,
+            0.008209343999624252,
+            -0.05576157942414284,
+            -0.014730350114405155,
+            -0.034200433641672134,
+            -0.01140014547854662,
+            0.017642343416810036,
+            -0.018215449526906013,
+            0.030126741155982018,
+            0.02997184917330742,
+            0.002638994250446558,
+            0.0007822110201232135,
+            0.021049996837973595,
+            0.006509389728307724,
+            -0.0053438181057572365,
+            0.012321759946644306,
+            0.028159597888588905,
+            -0.01450575515627861,
+            -0.010106786154210567,
+            -0.03454120084643364,
+            0.01861817203462124,
+            0.01589204929769039,
+            -0.04194509983062744,
+            -0.012569589540362358,
+            -0.013824224472045898,
+            0.032341714948415756,
+            -0.009549170732498169,
+            0.04098476096987724,
+            -0.010083552449941635,
+            0.001383390510454774,
+            -0.031149037182331085,
+            -0.009463978931307793,
+            0.017069239169359207,
+            0.006172496825456619,
+            -0.016589069738984108,
+            -0.0012846461031585932,
+            0.016170857474207878,
+            0.006617815233767033,
+            -0.030142230913043022,
+            0.04488807171583176,
+            0.023497309535741806,
+            0.055296897888183594,
+            0.05579255893826485,
+            -0.037391237914562225,
+            -0.002445377642288804,
+            0.0344172865152359,
+            -0.0228312686085701,
+            -0.015040136873722076,
+            -0.034014564007520676,
+            -0.01768881268799305,
+            0.035935238003730774,
+            -0.0318460576236248,
+            0.01556677371263504,
+            0.028066661208868027,
+            -0.025712283328175545,
+            0.003488970920443535,
+            -0.03943582624197006,
+            0.003622566582635045,
+            0.01220559049397707,
+            0.006095050368458033,
+            -0.0026293133851140738,
+            -0.013615119270980358,
+            0.008906363509595394,
+            0.003055269829928875,
+            0.00800798274576664,
+            -0.04615819454193115,
+            0.018478767946362495,
+            -0.00454224506393075,
+            0.04033420979976654,
+            0.042502716183662415,
+            -0.02730768360197544,
+            -0.016743963584303856,
+            0.008557853288948536,
+            0.012104909867048264,
+            0.04188314080238342,
+            -0.007287728600203991,
+            -0.00121784838847816,
+            -0.0007836631848476827,
+            0.007322579622268677,
+            0.047552235424518585,
+            0.004646798130124807,
+            0.0020697612781077623,
+            -0.027230238541960716,
+            0.014978179708123207,
+            -0.019733402878046036,
+            0.027648448944091797,
+            0.023373395204544067,
+            0.02942972257733345,
+            -0.04048910364508629,
+            0.017518429085612297,
+            -0.011818356812000275,
+            -0.005568413063883781,
+            -0.002726121572777629,
+            -0.02856232039630413,
+            -0.0074426219798624516,
+            0.033983584493398666,
+            0.015365412458777428,
+            0.012956822291016579,
+            0.03271345794200897,
+            0.0012052633101120591,
+            -0.042502716183662415,
+            0.03804178908467293,
+            0.024550583213567734,
+            0.04640602692961693,
+            0.023466331884264946,
+            0.010796061716973782,
+            0.006571347359567881,
+            0.013375034555792809,
+            -0.019377149641513824,
+            0.015907539054751396,
+            0.02560385875403881,
+            0.009750531986355782,
+            -0.03167567402124405,
+            -0.0009501734166406095,
+            0.01178737822920084,
+            -0.004546117503196001,
+            0.03701949119567871,
+            0.04104671627283096,
+            0.020569827407598495,
+            0.05030933395028114,
+            0.0020155487582087517,
+            0.017936641350388527,
+            0.005936284549534321,
+            0.02820606529712677,
+            0.030467506498098373,
+            -0.005285732913762331,
+            0.005076627247035503,
+            -0.01136916596442461,
+            -0.03385967016220093,
+            0.014110777527093887,
+            0.01561324205249548,
+            -0.008077683858573437,
+            -0.01697630248963833,
+            0.006033093202859163,
+            0.009169681929051876,
+            -0.01795213110744953,
+            -0.017456471920013428,
+            0.005971135571599007,
+            0.018866000697016716,
+            -0.013576395809650421,
+            0.024689987301826477,
+            -0.04442339017987251,
+            0.029119934886693954,
+            -0.033209118992090225,
+            -0.003384418087080121,
+            -0.010641167871654034,
+            -0.0030843124259263277,
+            0.0041666291654109955,
+            0.011314953677356243,
+            0.013406013138592243,
+            0.002042655134573579,
+            -0.013909416273236275,
+            0.024519605562090874,
+            0.018447788432240486,
+            0.012073930352926254,
+            -0.03148980438709259,
+            -0.014397329650819302,
+            -0.0015673263696953654,
+            -0.008743725717067719,
+            -0.034138478338718414,
+            0.026966920122504234,
+            0.014048820361495018,
+            -0.03844451159238815,
+            0.017797237262129784,
+            0.029538147151470184,
+            0.014420564286410809,
+            -0.004790074657648802,
+            -0.0463130883872509,
+            0.04045812413096428,
+            -0.030436528846621513,
+            -0.033333033323287964,
+            -0.010478530079126358,
+            -0.0003146269591525197,
+            0.02894955314695835,
+            -0.0064358157105743885,
+            -0.050835974514484406,
+            0.01184933539479971,
+            0.020631784573197365,
+            0.00010763872705865651,
+            0.005731051322072744,
+            -0.019593998789787292,
+            0.00028631053282879293,
+            -0.03395260497927666,
+            0.0090844901278615,
+            0.026750069111585617,
+            0.008867640048265457,
+            -0.016418687999248505,
+            0.03351890295743942,
+            -0.0004121613164898008,
+            -0.018695618957281113,
+            -0.003889757441356778,
+            0.006885006092488766,
+            0.034912943840026855,
+            0.004836542531847954,
+            -0.02946070022881031,
+            0.05665995925664902,
+            -0.02100352756679058,
+            0.021344292908906937,
+            0.05631919577717781,
+            -0.019439106807112694,
+            -0.008472662419080734,
+            -0.002065889071673155,
+            0.0026196325197815895,
+            -0.0260530486702919,
+            -0.008937342092394829,
+            0.032341714948415756,
+            0.041728246957063675,
+            -0.018742086365818977,
+            -0.001398879918269813,
+            0.006780453026294708,
+            0.009425255469977856,
+            0.004948840010911226,
+            0.02692045085132122,
+            -0.012128143571317196,
+            -0.014559967443346977,
+            0.05198218300938606,
+            0.028314489871263504,
+            -0.012081675231456757,
+            -0.0164031982421875,
+            0.019888296723365784,
+            -0.024844881147146225,
+            -0.020244551822543144,
+            -0.02431824430823326,
+            -0.020414933562278748,
+            -0.0042518205009400845,
+            0.009525936096906662,
+            0.037484172731637955,
+            -0.03968365862965584,
+            0.012747716158628464,
+            0.021700548008084297,
+            0.02981695532798767,
+            -0.019532041624188423,
+            -0.05910727381706238,
+            -0.0016128262504935265,
+            0.03296129032969475,
+            -0.015458349138498306,
+            -0.049379974603652954,
+            0.015164051204919815,
+            0.0260530486702919,
+            0.05957195535302162,
+            -0.001461805310100317,
+            -0.014954946003854275,
+            0.061926331371068954,
+            0.03026614524424076,
+            -0.004383479710668325,
+            -0.009494957514107227,
+            -0.0127864396199584,
+            0.003396035172045231,
+            0.00465454300865531,
+            -0.03364282101392746,
+            -0.041170634329319,
+            -0.011841590516269207,
+            -0.010284913703799248,
+            0.008108663372695446,
+            -0.040922801941633224,
+            0.03531566634774208,
+            -0.021917399019002914,
+            0.010950954630970955,
+            0.015473837964236736,
+            0.037391237914562225,
+            -0.021344292908906937,
+            -0.01284839678555727,
+            -0.01572941243648529,
+            -0.024968795478343964,
+            -0.014010096900165081,
+            0.021917399019002914,
+            0.007682706229388714,
+            3.4699722164077684e-05,
+            0.010075807571411133,
+            -0.024659009650349617,
+            -0.051052823662757874,
+            0.0291044469922781,
+            0.0015605497173964977,
+            -0.015938518568873405,
+            -0.004623564425855875,
+            -0.01977987214922905,
+            -0.01122201792895794,
+            -0.012608313001692295,
+            -0.039249956607818604,
+            0.021111954003572464,
+            -0.01135367713868618,
+            0.015876561403274536,
+            0.0009250032599084079,
+            0.003659353591501713,
+            0.028624277561903,
+            0.01116006076335907,
+            0.0051811798475682735,
+            -0.03255856782197952,
+            -0.018277406692504883,
+            0.010122275911271572,
+            -0.00561100896447897,
+            -0.018339363858103752,
+            -0.013351799920201302,
+            -0.05037129297852516,
+            -0.03305422514677048,
+            -0.008054450154304504,
+            -0.05415068939328194,
+            0.0020697612781077623,
+            -0.014296649023890495,
+            -0.02232012152671814,
+            -0.005990497302263975,
+            0.03107159025967121,
+            -0.003235333366319537,
+            0.025294071063399315,
+            0.0076594725251197815,
+            0.037360258400440216,
+            0.040117356926202774,
+            0.008426194079220295,
+            0.03065337985754013,
+            -0.0029991210903972387,
+            -0.016867877915501595,
+            0.05486319959163666,
+            -0.01881953328847885,
+            0.013529927469789982,
+            -0.008604321628808975,
+            0.008604321628808975,
+            0.03299226611852646,
+            -0.003794885240495205,
+            0.02007416822016239,
+            -0.0311955064535141,
+            -0.07998688519001007,
+            0.01637221872806549,
+            0.028159597888588905,
+            -0.06418777257204056,
+            0.014691627584397793,
+            -0.020275529474020004,
+            0.02052335813641548,
+            0.02411688305437565,
+            -0.0009114501299336553,
+            -0.014064309187233448,
+            0.005595519673079252,
+            0.0020833145827054977,
+            -0.01456771232187748,
+            -0.002691270550712943,
+            -0.009417511522769928,
+            0.008774704299867153,
+            -0.017007282003760338,
+            -0.013406013138592243,
+            -0.034231413155794144,
+            0.010873507708311081,
+            0.01233724970370531,
+            -0.003969140350818634,
+            0.0011500825639814138,
+            -0.03277541697025299,
+            -0.007876322604715824,
+            -0.02145271934568882,
+            0.03382869064807892,
+            0.012058441527187824,
+            -0.030699847266077995,
+            -0.007814365439116955,
+            0.015264731831848621,
+            -0.007876322604715824,
+            0.0008906363509595394,
+            0.024132372811436653,
+            0.02103450708091259,
+            0.0028364830650389194,
+            -0.0009700191440060735,
+            0.02438020147383213,
+            0.007427132688462734,
+            -0.01720864325761795,
+            0.028794659301638603,
+            0.014288905076682568,
+            0.060160547494888306,
+            -0.03271345794200897,
+            0.001297231181524694,
+            -0.04135650396347046,
+            -0.031118059530854225,
+            0.006408709101378918,
+            0.014350862242281437,
+            0.007597515359520912,
+            -0.028252532705664635,
+            0.005978880450129509,
+            0.02004319056868553,
+            0.01469937153160572,
+            0.009704063646495342,
+            0.0037580979987978935,
+            -0.036740683019161224,
+            -0.000691695255227387,
+            -0.011214273050427437,
+            -0.0038781403563916683,
+            0.035656433552503586,
+            -0.008186109364032745,
+            -0.05151750147342682,
+            -0.0006418389966711402,
+            -0.03255856782197952,
+            -0.0353776253759861,
+            0.021313315257430077,
+            -0.00839521549642086,
+            -0.0039207362569868565,
+            -0.013986863195896149,
+            -0.000537770101800561,
+            -0.010524998418986797,
+            0.027509044855833054,
+            0.028345469385385513,
+            0.022490503266453743,
+            -0.05384090170264244,
+            0.009479468688368797,
+            0.00023766436788719147,
+            -0.05817791447043419,
+            0.0009709871956147254,
+            -0.007117345929145813,
+            0.011191039346158504,
+            -0.020213572308421135,
+            -0.02414786070585251,
+            0.011500826105475426,
+            0.03392162546515465,
+            0.014304393902420998,
+            0.0321558453142643,
+            -0.013367289677262306,
+            -0.09250226616859436,
+            0.018292896449565887,
+            -0.008689512498676777,
+            0.04126356914639473,
+            0.00865078903734684,
+            -0.004278926644474268,
+            -0.015845581889152527,
+            -0.00865078903734684,
+            -0.006943091284483671,
+            0.01283290795981884,
+            0.004530628211796284,
+            -0.005626498255878687,
+            0.0016631665639579296,
+            0.013708055019378662,
+            -0.005967263597995043,
+            0.007481344975531101,
+            0.003651608945801854,
+            -0.013189162127673626,
+            -0.06059424951672554,
+            -0.04055105894804001,
+            -0.018540725111961365,
+            0.050835974514484406,
+            -0.0069159846752882,
+            -0.046684831380844116,
+            -0.010850274004042149,
+            0.05058814212679863,
+            0.012259802781045437,
+            -0.0334569476544857,
+            -0.0007841471815481782,
+            0.0050882440991699696,
+            0.015458349138498306,
+            0.004546117503196001,
+            -0.027292195707559586,
+            0.02820606529712677,
+            0.0008412640891037881,
+            0.02300165221095085,
+            -0.018339363858103752,
+            -0.005312839522957802,
+            0.035780347883701324,
+            -0.024844881147146225,
+            0.023791607469320297,
+            0.0055219451896846294,
+            0.031629208475351334,
+            0.002952652983367443,
+            0.010060317814350128,
+            0.004704883322119713,
+            0.0186491496860981,
+            0.004275054205209017,
+            0.0276639387011528,
+            0.011462102644145489,
+            0.006428070832043886,
+            0.021623101085424423,
+            -0.009378788061439991,
+            -0.014513500034809113,
+            0.01885051093995571,
+            -0.0014695499558001757,
+            0.025232113897800446,
+            -0.020538847893476486,
+            0.002726121572777629,
+            -0.0016883367206901312,
+            -0.017967619001865387,
+            -0.040922801941633224,
+            -0.0007405834621749818,
+            0.021747015416622162,
+            -0.011671207845211029,
+            -0.0228312686085701,
+            -0.013715799897909164,
+            -0.01585332676768303,
+            0.0038490979932248592,
+            0.034169457852840424,
+            -0.0398695282638073,
+            0.00963436160236597,
+            -0.000830615172162652,
+            -0.03125746175646782,
+            -0.0239619892090559,
+            -0.01093546487390995,
+            0.014614180661737919,
+            -0.01652711257338524,
+            -0.021173911169171333,
+            -0.028469383716583252,
+            -0.008410705253481865,
+            0.006048582494258881,
+            -0.05065010115504265,
+            -0.004054331686347723,
+            0.019965743646025658,
+            0.0276639387011528,
+            -0.003099801717326045,
+            -0.02177799493074417,
+            0.0012468908680602908,
+            0.022351099178195,
+            0.02936776541173458,
+            0.014226946979761124,
+            0.027586491778492928,
+            -0.002178186783567071,
+            -0.0011675080750137568,
+            -0.0073148347437381744,
+            0.01307299267500639,
+            0.0011287847300991416,
+            0.036616768687963486,
+            0.0122210793197155,
+            -0.020306508988142014,
+            0.0013959755888208747,
+            -0.006431943271309137,
+            0.029181893914937973,
+            -0.02270735427737236,
+            -9.832092473516241e-05,
+            0.03531566634774208,
+            0.027741385623812675,
+            0.004933350719511509,
+            -0.0143121387809515,
+            -0.05142456665635109,
+            0.021592123433947563,
+            -0.01765783317387104,
+            0.031025122851133347,
+            0.016666516661643982,
+            0.003397971158847213,
+            -0.0034212053287774324,
+            0.012368228286504745,
+            -0.022087780758738518,
+            -0.004406713880598545,
+            -0.021142931655049324,
+            0.04389675334095955,
+            -0.043184246867895126,
+            0.008472662419080734,
+            0.029445212334394455,
+            0.011926782317459583,
+            -0.019082851707935333,
+            -0.005425137002021074,
+            0.029212871566414833,
+            -0.00162637943867594,
+            0.01308848150074482,
+            0.0058665829710662365,
+            -0.026935940608382225,
+            0.01202746294438839,
+            -0.031660184264183044,
+            0.02177799493074417,
+            -0.013436991721391678,
+            0.018447788432240486,
+            0.03342596814036369,
+            -0.0030397805385291576,
+            -0.0038239278364926577,
+            -0.016263794153928757,
+            -0.00969631876796484,
+            0.0662323608994484,
+            0.01054823212325573,
+            0.03726732358336449,
+            0.0018974426202476025,
+            -0.009773765690624714,
+            0.007969259284436703,
+            0.0063041565008461475,
+            -0.012329504825174809,
+            0.010494019836187363,
+            0.024178840219974518,
+            0.0012730291346088052,
+            -0.002412462839856744,
+            -0.019408127292990685,
+            -0.03438630700111389,
+            0.003727119415998459,
+            -0.030932186171412468,
+            0.022691864520311356,
+            0.009208405390381813,
+            0.017456471920013428,
+            0.008240322582423687,
+            -0.015907539054751396,
+            0.03200095146894455,
+            0.05461536720395088,
+            0.0019768255297094584,
+            0.013375034555792809,
+            0.007938280701637268,
+            0.007036027032881975,
+            -0.011338187381625175,
+            0.04702559858560562,
+            0.000662652775645256,
+            0.005169562995433807,
+            0.022536970674991608,
+            0.02145271934568882,
+            0.022939693182706833,
+            0.002304037567228079,
+            -0.011694442480802536,
+            0.010478530079126358,
+            -0.0006302220281213522,
+            0.004445436876267195,
+            -0.010695381090044975,
+            -0.027865299955010414,
+            0.007492962293326855,
+            0.01302652433514595,
+            0.022598929703235626,
+            0.021592123433947563,
+            -0.03178409859538078,
+            -0.01855621486902237,
+            0.03364282101392746,
+            -0.02997184917330742,
+            0.03612111136317253,
+            -0.0014424435794353485,
+            -0.023807097226381302,
+            -0.0058356039226055145,
+            0.016929835081100464,
+            0.02656419761478901,
+            -0.029181893914937973,
+            0.01804506592452526,
+            0.029693040996789932,
+            -0.0074426219798624516,
+            0.0005939188995398581,
+            0.012112653814256191,
+            0.02447313815355301,
+            -0.009293596260249615,
+            -0.0024027819745242596,
+            -0.0013320822035893798,
+            -0.011926782317459583,
+            -0.018370341509580612,
+            -0.0044725434854626656,
+            -0.022629907354712486,
+            0.005564541090279818,
+            -0.007365175057202578,
+            -0.003481226274743676,
+            -0.008581086993217468,
+            0.01154729351401329,
+            0.001979729626327753,
+            -0.019857317209243774,
+            -0.022335609421133995,
+            0.01652711257338524,
+            0.03887821361422539,
+            -0.018215449526906013,
+            -0.02106548473238945,
+            0.024442158639431,
+            -0.01759587600827217,
+            -0.01601596549153328,
+            0.036678727716207504,
+            0.013739033602178097,
+            0.007736918982118368,
+            -0.03358086198568344,
+            -0.030049296095967293,
+            -0.03274443745613098,
+            0.006296411622315645,
+            0.03785591572523117,
+            -0.0028752065263688564,
+            0.02569679357111454,
+            0.03701949119567871,
+            0.0057891360484063625,
+            -0.028934063389897346,
+            -0.014823286794126034,
+            -0.045104920864105225,
+            0.060160547494888306,
+            0.0011481463443487883,
+            0.05161044001579285,
+            0.016294771805405617,
+            -0.01456771232187748,
+            0.00419760774821043,
+            -0.013251119293272495,
+            0.026192452758550644,
+            0.02280029095709324,
+            0.025929134339094162,
+            0.006582964211702347,
+            0.022505993023514748,
+            -0.0037522895727306604,
+            -0.012197845615446568,
+            0.007307090330868959,
+            0.014536733739078045,
+            0.0002497654058970511,
+            -0.011392400600016117,
+            0.015783624723553658,
+            -0.01307299267500639,
+            -0.011361422017216682,
+            0.023295948281884193,
+            0.01368482131510973,
+            -0.012344993650913239,
+            0.014064309187233448,
+            -0.008565598167479038,
+            0.01239920686930418,
+            -0.023915521800518036,
+            0.022691864520311356,
+            -0.03262052312493324,
+            0.009587894193828106,
+            0.0015053689712658525,
+            -0.0073148347437381744,
+            0.024550583213567734,
+            -0.0027454833034425974,
+            -0.05071205645799637,
+            0.03550153970718384,
+            -0.008015726692974567,
+            0.0037213109899312258,
+            0.006381602957844734,
+            0.011392400600016117,
+            -0.0012623801594600081,
+            0.03807276859879494,
+            0.01239920686930418,
+            0.015969496220350266,
+            0.02199484594166279,
+            0.011911292560398579,
+            -0.008433938957750797,
+            -0.004096927121281624,
+            -0.0016137943603098392,
+            -0.01784370467066765,
+            0.025944624096155167,
+            0.0018674320308491588,
+            0.03962169960141182,
+            -0.018168980255723,
+            -0.02300165221095085,
+            0.028066661208868027,
+            0.007597515359520912,
+            -0.02351279929280281,
+            -0.0017377089243382215,
+            0.01202746294438839,
+            -0.009185170754790306,
+            0.027385130524635315,
+            0.01196550577878952,
+            0.019113829359412193,
+            -0.018060555681586266,
+            -0.004340883810073137,
+            -0.005072754807770252,
+            -0.015938518568873405,
+            0.02238207869231701,
+            0.020771188661456108,
+            0.011152315884828568,
+            -0.011880313977599144,
+            -0.01307299267500639,
+            0.020600805059075356,
+            0.035036858171224594,
+            -0.0002744515077210963,
+            0.005672966130077839,
+            0.017487451434135437,
+            0.002373739378526807,
+            -0.04470219835639,
+            0.016232814639806747,
+            -0.014025585725903511,
+            0.011214273050427437,
+            0.019593998789787292,
+            -0.021917399019002914,
+            -0.03196997195482254,
+            -0.011539548635482788,
+            -0.0414804182946682,
+            0.00493722315877676,
+            -0.0069159846752882,
+            0.013576395809650421,
+            -0.010176488198339939,
+            -0.0014831031439825892,
+            0.016666516661643982,
+            0.0038103745318949223,
+            0.02177799493074417,
+            -0.011678952723741531,
+            -0.012073930352926254,
+            -0.020213572308421135,
+            -0.020786678418517113,
+            0.004278926644474268,
+            -0.030405549332499504,
+            -0.004488032776862383,
+            0.021375272423028946,
+            -0.04783104360103607,
+            -0.03689557686448097,
+            0.02898053266108036,
+            -0.008503641001880169,
+            0.020755698904395103,
+            0.022955182939767838,
+            -0.015094350092113018,
+            -0.0008717587334103882,
+            -0.018695618957281113,
+            0.02258343994617462,
+            -0.007837600074708462,
+            0.026300877332687378,
+            -0.0027551641687750816,
+            0.0006292539183050394,
+            -0.02659517526626587,
+            0.020306508988142014,
+            0.005227648187428713,
+            0.012275291606783867,
+            -0.02390003204345703,
+            0.006815304048359394,
+            -0.03133491054177284,
+            -0.01784370467066765,
+            0.0180140882730484,
+            -0.011152315884828568,
+            0.027989214286208153,
+            -0.006060199346393347,
+            0.028438406065106392,
+            0.022722844034433365,
+            0.024751944467425346,
+            0.02676555886864662,
+            -0.02363671362400055,
+            0.008495896123349667,
+            0.03726732358336449,
+            0.00865078903734684,
+            -0.015876561403274536,
+            0.008170620538294315,
+            -0.04863648861646652,
+            -0.008062195032835007,
+            0.05030933395028114,
+            0.002127846470102668,
+            0.023141054436564445,
+            0.016294771805405617,
+            -0.002873270306736231,
+            -0.005688455421477556,
+            0.010269423946738243,
+            -0.04296739399433136,
+            -0.023172033950686455,
+            0.022629907354712486,
+            -0.021561143919825554,
+            6.54212328754511e-07,
+            0.03122648410499096,
+            0.012120398692786694,
+            -0.0009109660750254989,
+            -0.05108380317687988,
+            0.02100352756679058,
+            0.01178737822920084,
+            0.014443797990679741,
+            -0.010300402529537678,
+            0.026068538427352905,
+            0.01733255758881569,
+            0.038413532078266144,
+            0.012755461037158966,
+            -0.002679653698578477,
+            -0.03782493621110916,
+            0.009177426807582378,
+            -0.021685058251023293,
+            0.0011471783509477973,
+            0.0074503663927316666,
+            -0.02273833192884922,
+            -0.024782923981547356,
+            0.0043912241235375404,
+            0.004588713403791189,
+            0.011880313977599144,
+            0.022087780758738518,
+            0.009417511522769928,
+            -0.021204888820648193,
+            0.015404135920107365,
+            -0.05446047708392143,
+            -0.015450604259967804,
+            -0.023884542286396027,
+            0.02566581591963768,
+            0.020012211054563522,
+            0.010184233076870441,
+            0.005545179359614849,
+            0.010904486291110516,
+            -0.020988039672374725,
+            -0.009239383973181248,
+            0.020631784573197365,
+            0.011074868962168694,
+            -0.012840652838349342,
+            0.04113965481519699,
+            -0.00011126903700642288,
+            0.031210994347929955,
+            -0.02280029095709324,
+            0.002941036131232977,
+            -0.001241082325577736,
+            -0.03649285435676575,
+            0.009130958467721939,
+            0.01981084980070591,
+            0.02151467651128769,
+            -0.01861817203462124,
+            0.0024143988266587257,
+            0.01048627495765686,
+            0.004766840487718582,
+            0.038351576775312424,
+            0.027121812105178833,
+            -0.044206541031599045,
+            0.011663463898003101,
+            0.012995545752346516,
+            -0.009363298304378986,
+            0.048419639468193054,
+            -0.017518429085612297,
+            0.017673322930932045,
+            0.008240322582423687,
+            -0.01798310875892639,
+            0.02318752370774746,
+            -0.01469937153160572,
+            -0.013282098807394505,
+            0.00650551775470376,
+            0.009192915633320808,
+            -0.0001770381786627695,
+            -0.037577107548713684,
+            -0.017859194427728653,
+            -0.005719434004276991,
+            0.0038297362625598907,
+            0.01562873087823391,
+            0.013498948886990547,
+            0.009967382065951824,
+            0.005161818582564592,
+            -0.00945623405277729,
+            -0.0016089539276435971,
+            0.008124152198433876,
+            -0.0005532593932002783,
+            0.0019216446671634912,
+            -0.01926872320473194,
+            0.02698240801692009,
+            0.02572777308523655,
+            -0.0018354853382334113,
+            -0.04652994126081467,
+            -0.0036709706764668226,
+            0.0037077576853334904,
+            -0.017735280096530914,
+            -0.0035315665882080793,
+            -0.04519785940647125,
+            -0.01810702309012413,
+            0.0143121387809515,
+            -0.014614180661737919,
+            -0.0460033044219017,
+            0.009076746180653572,
+            -0.013886182568967342,
+            0.0063970922492444515,
+            -0.00932457484304905,
+            0.005762029904872179,
+            0.003651608945801854,
+            0.005680711008608341,
+            -0.005909178406000137,
+            0.05108380317687988,
+            -0.03466511517763138,
+            0.028051171451807022,
+            0.008565598167479038,
+            -0.012437930330634117,
+            -0.02261441759765148,
+            0.00887538492679596,
+            -0.044981006532907486,
+            -0.033209118992090225,
+            0.016743963584303856,
+            -0.01781272701919079,
+            0.011074868962168694,
+            0.06161654368042946,
+            0.007330324035137892,
+            -0.023342415690422058,
+            -0.01233724970370531,
+            -0.021049996837973595,
+            -0.04191412031650543,
+            0.003442503046244383,
+            0.005463860463351011,
+            -0.004948840010911226,
+            -0.03807276859879494,
+            -0.021716037765145302,
+            0.04615819454193115,
+            -0.012004229240119457,
+            0.012422440573573112,
+            0.01080380566418171,
+            -0.0019410063978284597,
+            -0.04860550910234451,
+            -0.021019017323851585,
+            0.0395597442984581,
+            0.021886419504880905,
+            -0.008426194079220295,
+            0.0028364830650389194,
+            -0.018277406692504883,
+            -0.003289545886218548,
+            0.006296411622315645,
+            -0.009107724763453007,
+            -0.03562545403838158,
+            0.02067825198173523,
+            0.0054832217283546925,
+            -0.023373395204544067,
+            -0.025805220007896423,
+            -0.03296129032969475,
+            0.009944148361682892,
+            0.02453509531915188,
+            -0.01968693546950817,
+            0.005823987070471048,
+            0.00027203132049180567,
+            -0.000881923595443368,
+            0.00042740863864310086,
+            -0.009959637187421322,
+            -0.019764382392168045,
+            -0.020151615142822266,
+            0.03268248215317726,
+            -0.0060059865936636925,
+            0.0074387495405972,
+            -0.03630698472261429,
+            -0.010974188335239887,
+            0.011322698555886745,
+            -0.002395037328824401,
+            -0.013452480547130108,
+            -0.009092235006392002,
+            -0.01739451475441456,
+            -0.009882191196084023,
+            -0.007876322604715824,
+            0.022660886868834496,
+            0.006168624851852655,
+            0.005409647710621357,
+            0.0023795480374246836,
+            -0.008581086993217468,
+            0.02898053266108036,
+            0.017874684184789658,
+            -0.014188224449753761,
+            -0.00920066051185131,
+            0.03451022133231163,
+            0.02665713243186474,
+            -0.009115469641983509,
+            0.01798310875892639,
+            0.023264970630407333,
+            -0.0010484338272362947,
+            -0.01196550577878952,
+            0.001922612776979804,
+            -0.0009550138493068516,
+            -0.0032972905319184065,
+            0.00845717266201973,
+            -0.015365412458777428,
+            0.0315207801759243,
+            0.04287445917725563,
+            -0.002145271748304367,
+            0.009587894193828106,
+            -0.003727119415998459,
+            -0.008472662419080734,
+            0.017069239169359207,
+            0.009316830895841122,
+            0.04625113308429718,
+            0.01098193321377039,
+            0.005680711008608341,
+            -0.033983584493398666,
+            -0.001229465357027948,
+            -0.01184933539479971,
+            0.03308520466089249,
+            0.02061629481613636,
+            0.00889087375253439,
+            -0.008379725739359856,
+            0.01344473659992218,
+            0.008341003209352493,
+            0.00661007035523653,
+            0.020569827407598495,
+            -0.016542602330446243,
+            -0.017286090180277824,
+            0.030467506498098373,
+            -0.00672236829996109,
+            0.01289486512541771,
+            0.02029101923108101,
+            -0.002089123008772731,
+            0.010842529125511646,
+            0.07100307941436768,
+            -0.006184114143252373,
+            0.015713922679424286,
+            -0.019082851707935333,
+            0.0015789433382451534,
+            -0.007640110794454813,
+            -0.03370477631688118,
+            0.018246427178382874,
+            0.0039962464943528175,
+            0.00865078903734684,
+            0.01198099460452795,
+            0.009053511545062065,
+            -0.007946024648845196,
+            -0.00963436160236597,
+            0.015195030719041824,
+            0.0019080914789810777,
+            0.00033568276558071375,
+            0.001336922636255622,
+            -0.0065210070461034775,
+            -0.004011735785752535,
+            -0.021948376670479774,
+            0.02763296104967594,
+            0.007140580099076033,
+            0.030606910586357117,
+            -0.055296897888183594,
+            0.030095763504505157,
+            -0.00789568480104208,
+            0.02554190158843994,
+            -0.016201836988329887,
+            -0.011090358719229698,
+            0.001609921921044588,
+            0.018153492361307144,
+            0.004491904750466347,
+            0.006462921854108572,
+            0.011888058856129646,
+            0.023233991116285324,
+            -0.03444826602935791,
+            0.017053749412298203,
+            0.023110076785087585,
+            0.0048675211146473885,
+            0.0003535923024173826,
+            0.005165690556168556,
+            0.015458349138498306,
+            0.007198664825409651,
+            -0.023048119619488716,
+            -0.0025692922063171864,
+            0.017611365765333176,
+            -0.02306360937654972,
+            0.0036980770528316498,
+            -0.006923729553818703,
+            0.004623564425855875,
+            -0.0014705180656164885,
+            -0.03509881719946861,
+            -0.00100099784322083,
+            -0.025743262842297554,
+            0.01923774555325508,
+            0.009556914679706097,
+            -0.025867177173495293,
+            0.003932353109121323,
+            0.025650326162576675,
+            -0.0003695656778290868,
+            0.00821708794683218,
+            0.0004683101433329284,
+            0.00869725737720728,
+            -0.00974278710782528,
+            -0.01493945624679327,
+            0.004801691509783268,
+            0.01531120017170906,
+            -0.01493945624679327,
+            -0.007431004662066698,
+            -0.009510447271168232,
+            0.030560443177819252,
+            0.0006674932083114982,
+            0.004189862869679928,
+            -0.005045648664236069,
+            -0.01858719252049923,
+            -0.0032237162813544273,
+            -0.021142931655049324,
+            0.004367990419268608,
+            -0.0009046735358424485,
+            0.04197607934474945,
+            0.0038161829579621553,
+            -0.02029101923108101,
+            0.0032972905319184065,
+            -0.03136589005589485,
+            0.0005000148667022586,
+            -0.007276111748069525,
+            -0.02004319056868553,
+            0.0015005286550149322,
+            0.0016525175888091326,
+            0.012631546705961227,
+            -0.03255856782197952,
+            -0.009301341138780117,
+            0.018416810780763626,
+            -0.00020680672605521977,
+            -0.010687636211514473,
+            -0.017828216776251793,
+            0.00035189816844649613,
+            -0.011338187381625175,
+            -0.027973726391792297,
+            -0.02850036323070526,
+            -0.0026719090528786182,
+            0.013491204008460045,
+            -0.018277406692504883,
+            -0.014993669465184212,
+            0.0031249718740582466,
+            0.021592123433947563,
+            -0.009781510569155216,
+            0.008302279748022556,
+            0.02241305634379387,
+            -0.008255811408162117,
+            -0.005514200776815414,
+            -0.000941460661124438,
+            -0.009130958467721939,
+            0.004267309792339802,
+            0.006300284061580896,
+            0.043339136987924576,
+            -0.032403673976659775,
+            -0.0018238683696836233,
+            0.014815541915595531,
+            0.0034153966698795557,
+            0.0025905899237841368,
+            0.002503462601453066,
+            -0.0007841471815481782,
+            0.01344473659992218,
+            0.01178737822920084,
+            0.015466093085706234,
+            0.00100099784322083,
+            -0.0033011629711836576,
+            0.006408709101378918,
+            -0.021173911169171333,
+            0.015721667557954788,
+            0.027958236634731293,
+            -0.0033999073784798384,
+            0.005525817628949881,
+            -0.013731288723647594,
+            -0.008759214542806149,
+            0.013134949840605259,
+            0.0026641644071787596,
+            -0.005068882368505001,
+            -0.009394276887178421,
+            0.017967619001865387,
+            0.019082851707935333,
+            0.006668155547231436,
+            -0.037453193217515945,
+            -0.02898053266108036,
+            -0.009959637187421322,
+            0.020569827407598495,
+            0.014916222542524338,
+            0.0010339126456528902,
+            -0.010540487244725227,
+            -0.009595638141036034,
+            -0.011903548613190651,
+            0.010989678092300892,
+            0.02721474878489971,
+            -0.032372694462537766,
+            0.0008214184199459851,
+            -0.029383255168795586,
+            0.03068435750901699,
+            0.0020000594668090343,
+            0.011446612887084484,
+            -0.0008185141487047076,
+            -0.015140817500650883,
+            -0.029181893914937973,
+            0.013824224472045898,
+            -0.004395096562802792,
+            0.001024231780320406,
+            -0.0031753121875226498,
+            0.0201051477342844,
+            0.012887120246887207,
+            -0.0014472840121015906,
+            0.01968693546950817,
+            0.008178364485502243,
+            0.016480645164847374,
+            -0.010153254494071007,
+            -0.04135650396347046,
+            0.019965743646025658,
+            0.007562664337456226,
+            0.014219203032553196,
+            0.004739734344184399,
+            0.004588713403791189,
+            -0.020771188661456108,
+            0.002724185585975647,
+            0.004333139397203922,
+            0.014405074529349804,
+            0.011028401553630829,
+            0.0025615475606173277,
+            0.011678952723741531,
+            -0.02228914201259613,
+            0.01929970271885395,
+            -0.01631026156246662,
+            0.01006806269288063,
+            0.004445436876267195,
+            -0.026873983442783356,
+            0.007547175046056509,
+            0.007822110317647457,
+            -0.0019187404541298747,
+            -0.017549408599734306,
+            0.02160761132836342,
+            -0.0034831624943763018,
+            -0.03509881719946861,
+            0.0360281765460968,
+            -0.0043447562493383884,
+            0.019346170127391815,
+            0.03209388628602028,
+            -0.01771979033946991,
+            -0.01993476413190365,
+            0.00906900130212307,
+            0.010408828034996986,
+            -0.002873270306736231,
+            0.017859194427728653,
+            -0.002919738180935383,
+            -0.02232012152671814,
+            0.017766257748007774,
+            -0.028918573632836342,
+            -0.014234691858291626,
+            0.020337486639618874,
+            -0.03624502569437027,
+            -0.0021026763133704662,
+            0.03175312280654907,
+            -0.024070415645837784,
+            0.005734923295676708,
+            0.005196669604629278,
+            -0.0019351978553459048,
+            0.0012236569309607148,
+            -0.006451305001974106,
+            0.03801080957055092,
+            -0.010865762829780579,
+            0.0201051477342844,
+            0.01929970271885395,
+            -0.023388884961605072,
+            0.01977987214922905,
+            -0.001109423115849495,
+            0.01675945147871971,
+            -0.003940097521990538,
+            0.03466511517763138,
+            -0.006017603911459446,
+            -0.011330443434417248,
+            0.006431943271309137,
+            0.008488151244819164,
+            -0.009959637187421322,
+            0.008681767620146275,
+            -0.018122512847185135,
+            0.027152791619300842,
+            0.02091059274971485,
+            -0.0002691270783543587,
+            0.013800990767776966,
+            0.0004421719058882445,
+            -0.020833145827054977,
+            0.00789568480104208,
+            -0.027121812105178833,
+            -0.009967382065951824,
+            0.01646515540778637,
+            -0.005762029904872179,
+            0.007380664348602295,
+            -0.02064727433025837,
+            0.028004704043269157,
+            0.0077524082735180855,
+            -0.013591884635388851,
+            -0.0008824076503515244,
+            0.023760627955198288,
+            -0.005432881414890289,
+            0.009355553425848484,
+            0.02070923149585724,
+            0.015713922679424286,
+            0.013615119270980358,
+            0.016263794153928757,
+            0.03351890295743942,
+            0.005328328814357519,
+            0.005568413063883781,
+            0.003519949736073613,
+            0.016991792246699333,
+            -0.008557853288948536,
+            0.018029578030109406,
+            0.0008093173382803798,
+            -0.013576395809650421,
+            0.021499186754226685,
+            -0.0018684001406654716,
+            -0.013622863218188286,
+            -0.0040891822427511215,
+            -0.004337011836469173,
+            0.03271345794200897,
+            0.022041313350200653,
+            -0.004468671046197414,
+            -0.024442158639431,
+            -0.00394784240052104,
+            0.016682006418704987,
+            -0.003167567541822791,
+            0.011136826127767563,
+            0.0007952801533974707,
+            -0.01586107164621353,
+            -0.009595638141036034,
+            0.0063583687879145145,
+            0.010586955584585667,
+            -0.008921852335333824,
+            0.010346870869398117,
+            -0.0228312686085701,
+            -0.004011735785752535,
+            0.00046613195445388556,
+            -0.004844287410378456,
+            0.0053438181057572365,
+            -0.011314953677356243,
+            0.004948840010911226,
+            0.016062432900071144,
+            0.01026167906820774,
+            0.004832670092582703,
+            0.006989559158682823,
+            0.012066186405718327,
+            0.009161937050521374,
+            -0.01627928391098976,
+            0.00428667152300477,
+            -0.005072754807770252,
+            -0.02345084212720394,
+            0.005831731949001551,
+            0.00902253296226263,
+            -0.01122201792895794,
+            0.009386532939970493,
+            -0.010958699509501457,
+            0.03296129032969475,
+            0.01030814740806818,
+            -0.037391237914562225,
+            -0.0033379499800503254,
+            -0.0016244432190433145,
+            0.014885243959724903,
+            0.012298526242375374,
+            0.008550108410418034,
+            -0.017518429085612297,
+            -0.0047823297791182995,
+            -0.009835722856223583,
+            0.0012827098835259676,
+            -0.012306271120905876,
+            0.017456471920013428,
+            0.022567950189113617,
+            -0.036337960511446,
+            0.01968693546950817,
+            -0.010432061739265919,
+            -0.023652203381061554,
+            0.020476890727877617,
+            0.011500826105475426,
+            0.01942361705005169,
+            -0.009402021765708923,
+            0.018896980211138725,
+            -0.0001331114035565406,
+            0.025510922074317932,
+            -0.004228586331009865,
+            -0.022087780758738518,
+            -0.012499887496232986,
+            -0.03209388628602028,
+            -0.0051811798475682735,
+            0.006819176487624645,
+            -0.00045524101005867124,
+            0.013111716136336327,
+            0.014226946979761124,
+            -0.025712283328175545,
+            0.007295473478734493,
+            0.032589543610811234,
+            0.014637414366006851,
+            0.023311438038945198,
+            -0.011276230216026306,
+            -0.0010794125264510512,
+            -0.0026060794480144978,
+            -0.013770012184977531,
+            -0.03212486580014229,
+            -0.0017493258928880095,
+            -0.036585792899131775,
+            0.005680711008608341,
+            -0.016387708485126495,
+            -0.013096226379275322,
+            0.023264970630407333,
+            -0.034138478338718414,
+            0.0014608372002840042,
+            0.013708055019378662,
+            0.010540487244725227,
+            0.012391461990773678,
+            -0.00162637943867594,
+            0.015032391995191574,
+            -0.029243851080536842,
+            -0.010284913703799248,
+            -0.004491904750466347,
+            -0.01981084980070591,
+            -0.007338068913668394,
+            -0.018060555681586266,
+            -0.013127204962074757,
+            -0.003566417610272765,
+            -0.010060317814350128,
+            0.002247888594865799,
+            0.0032062907703220844,
+            -0.014714861288666725,
+            0.03206290677189827,
+            0.003157886676490307,
+            0.018958937376737595,
+            0.0033418224193155766,
+            0.004449309315532446,
+            0.018788553774356842,
+            -0.0026331855915486813,
+            -0.0058278595097362995,
+            0.030963165685534477,
+            -0.00032672801171429455,
+            0.008759214542806149,
+            0.019082851707935333,
+            0.004724245052784681,
+            0.0038529701996594667,
+            -0.0260530486702919,
+            0.012887120246887207,
+            -0.027447087690234184,
+            0.005080499220639467,
+            -0.0077175572514534,
+            -0.024302754551172256,
+            -0.028128618374466896,
+            -0.006571347359567881,
+            0.012058441527187824,
+            -0.02898053266108036,
+            -0.00613377382978797,
+            0.002315654419362545,
+            -0.00400399137288332,
+            -0.002673845272511244,
+            0.007241260726004839,
+            0.029150914400815964,
+            -0.033209118992090225,
+            0.008751469664275646,
+            0.005595519673079252,
+            -0.0005053392960689962,
+            0.013297587633132935,
+            0.008038961328566074,
+            0.003343758638948202,
+            0.020848635584115982,
+            0.006637176964432001,
+            0.002458930714055896,
+            -0.0025576751213520765,
+            -0.014242436736822128,
+            0.004000118933618069,
+            0.02154565416276455,
+            0.007562664337456226,
+            0.018385831266641617,
+            -0.008914108388125896,
+            0.002621568739414215,
+            -0.01455222349613905,
+            0.005552923772484064,
+            -0.002867461647838354,
+            0.004309905227273703,
+            -0.012236569076776505,
+            0.008093173615634441,
+            0.0028771425131708384,
+            -0.0043447562493383884,
+            0.018060555681586266,
+            -0.002319526858627796,
+            0.016480645164847374,
+            0.004015608225017786,
+            0.015620986931025982,
+            -0.0035412474535405636,
+            0.01178737822920084,
+            0.0005731051205657423,
+            -0.017905661836266518,
+            -0.017022771760821342,
+            -0.039249956607818604,
+            0.007702067960053682,
+            0.03531566634774208,
+            -0.0009564659558236599,
+            -0.03444826602935791,
+            0.010145509615540504,
+            -0.011942271143198013,
+            -0.009975126944482327,
+            -0.022862248122692108,
+            0.013862947933375835,
+            0.012430185452103615,
+            0.007415515370666981,
+            0.012252057902514935,
+            -0.006997303571552038,
+            0.01026167906820774,
+            -0.005216030869632959,
+            0.008356492035090923,
+            -0.0010823167394846678,
+            -0.037422213703393936,
+            0.012344993650913239,
+            0.02216522768139839,
+            -0.002635121811181307,
+            0.0016089539276435971,
+            -0.006145390681922436,
+            0.022103270515799522,
+            -0.013971373438835144,
+            -0.00650551775470376,
+            -0.00046443779137916863,
+            0.00797700323164463,
+            -0.01795213110744953,
+            0.004778457339853048,
+            0.0755879208445549,
+            -0.007775642443448305,
+            0.006122156511992216,
+            0.009270362555980682,
+            -0.015148562379181385,
+            -0.01637221872806549,
+            -0.004561606794595718,
+            -0.020260039716959,
+            0.008844406343996525,
+            -0.0018819533288478851,
+            -0.0110981035977602,
+            0.012468908913433552,
+            0.0053050946444272995,
+            0.002031038049608469,
+            -0.008379725739359856,
+            0.010811550542712212,
+            0.009525936096906662,
+            0.010432061739265919,
+            0.02360573597252369,
+            0.011477591469883919,
+            -0.00858883187174797,
+            -0.011795123107731342,
+            0.01129171997308731,
+            0.0028577810153365135,
+            0.016697494313120842,
+            0.007907302118837833,
+            0.0010648912284523249,
+            0.019501063972711563,
+            -0.002309845993295312,
+            0.02180897258222103,
+            -2.2689444449497387e-05,
+            -0.009301341138780117,
+            -0.004236331209540367,
+            -0.015349923633038998,
+            0.024271776899695396,
+            -0.0020155487582087517,
+            0.0015450604259967804,
+            -0.020213572308421135,
+            0.0013979118084535003,
+            0.013243375346064568,
+            0.011957760900259018,
+            -0.0020407189149409533,
+            -0.0051347119733691216,
+            0.0007154133054427803,
+            -0.016356728971004486,
+            0.032806396484375,
+            0.007578153628855944,
+            0.0007681738352403045,
+            0.0008838597568683326,
+            0.0009288755827583373,
+            -0.0032972905319184065,
+            0.0024608669336885214,
+            -0.004766840487718582,
+            0.009549170732498169,
+            -0.010943209752440453,
+            -0.010796061716973782,
+            0.010075807571411133,
+            -0.02402394637465477,
+            0.003566417610272765,
+            -0.00558777479454875,
+            -0.008062195032835007,
+            0.0076207490637898445,
+            0.00015537731815129519,
+            -0.007392281666398048,
+            -0.0030939930584281683,
+            0.0023408245760947466,
+            0.013429246842861176,
+            0.004406713880598545,
+            0.010850274004042149,
+            0.012956822291016579,
+            0.005359307397156954,
+            -0.006629432085901499,
+            0.015512561425566673,
+            -0.006946963258087635,
+            -0.004228586331009865,
+            -0.0012468908680602908,
+            -0.019996721297502518,
+            -0.0009419447160325944,
+            -0.004708755761384964,
+            0.0015092412941157818,
+            0.0001705036120256409,
+            -8.749050175538287e-05,
+            -0.0018490385264158249,
+            -0.011988739483058453,
+            -0.001831613015383482,
+            -0.011663463898003101,
+            -0.014498010277748108,
+            0.010253935120999813,
+            -0.0041201612912118435,
+            -0.022939693182706833,
+            -0.015349923633038998,
+            -0.020662762224674225,
+            0.02145271934568882,
+            -0.004387352149933577,
+            -0.005800753366202116,
+            0.006733985152095556,
+            -0.010215211659669876,
+            -0.015706177800893784,
+            -0.008069939911365509,
+            -0.0205853171646595,
+            0.010703125037252903,
+            0.003214035416021943,
+            0.0006791101768612862,
+            0.00882891658693552,
+            0.014397329650819302,
+            -0.01589204929769039,
+            0.0111755495890975,
+            0.012182355858385563,
+            0.011361422017216682,
+            -0.006854027509689331,
+            0.0008911203476600349,
+            -0.014103032648563385,
+            0.02596011385321617,
+            0.021623101085424423,
+            0.005169562995433807,
+            -0.04132552444934845,
+            -0.005862710531800985,
+            -0.013344055972993374,
+            -0.010524998418986797,
+            -0.012120398692786694,
+            0.011771888472139835,
+            -0.0038820127956569195,
+            0.0013117524795234203,
+            0.006288666743785143,
+            0.0015063370810821652,
+            0.004367990419268608,
+            -0.0022382077295333147,
+            -0.007709812838584185,
+            -0.03212486580014229,
+            -0.02817508764564991,
+            -0.00863530021160841,
+            0.020461400970816612,
+            0.002838419284671545,
+            -0.0009453330421820283,
+            -0.0036380558740347624,
+            0.0027048238553106785,
+            -0.005882072262465954,
+            -0.005018542055040598,
+            -0.014753584749996662,
+            -0.012739972211420536,
+            0.0101300198584795,
+            0.00802347157150507,
+            0.014195968396961689,
+            -0.011950016021728516,
+            0.0028190575540065765,
+            0.018788553774356842,
+            0.007690451107919216,
+            -0.023357905447483063,
+            0.01135367713868618,
+            -0.010912231169641018,
+            -0.009270362555980682,
+            -0.009433000348508358,
+            -0.009471723809838295,
+            0.0016980175860226154,
+            -0.008317768573760986,
+            0.003945906180888414,
+            -0.03615209087729454,
+            -0.011330443434417248,
+            -0.0013901671627536416,
+            -0.010346870869398117,
+            0.0009864765452221036,
+            0.02524760365486145,
+            0.006815304048359394,
+            0.01432762760668993,
+            -0.0006733016925863922,
+            0.017502939328551292,
+            -0.0018984107300639153,
+            0.018726596608757973,
+            0.012763205915689468,
+            -0.00797700323164463,
+            0.011717676185071468,
+            0.014738094992935658,
+            -0.0033863543067127466,
+            0.015620986931025982,
+            -0.010168743319809437,
+            0.005285732913762331,
+            -0.009556914679706097,
+            -0.003214035416021943,
+            0.002495717955753207,
+            0.030901208519935608,
+            -0.003618694143369794,
+            0.00858883187174797,
+            0.01098193321377039,
+            -0.015241498127579689,
+            0.0026951429899781942,
+            0.014962689951062202,
+            0.01141563430428505,
+            0.007760153152048588,
+            0.0024434414226561785,
+            -0.024225307628512383,
+            0.004193735308945179,
+            0.005037903785705566,
+            0.0031520782504230738,
+            -0.014366351068019867,
+            0.0021859314292669296,
+            0.015675200149416924,
+            0.008550108410418034,
+            -0.025325050577521324,
+            0.016418687999248505,
+            0.0009138703462667763,
+            -0.003909118939191103,
+            0.029119934886693954,
+            -0.011268485337495804,
+            -0.005723306443542242,
+            0.009463978931307793,
+            0.0009980935137718916,
+            0.0030029932968318462,
+            0.008937342092394829,
+            0.0228312686085701,
+            -0.003988501615822315,
+            -0.032341714948415756,
+            -0.022459525614976883,
+            0.015706177800893784,
+            -0.0047823297791182995,
+            -0.022939693182706833,
+            -0.026703599840402603,
+            -0.003113354789093137,
+            -0.02228914201259613,
+            -0.0022323993034660816,
+            -0.020058678463101387,
+            -0.014056564308702946,
+            0.001268188701942563,
+            0.029987338930368423,
+            0.016201836988329887,
+            0.008224832825362682,
+            -0.011051635257899761,
+            -0.029956359416246414,
+            -0.024271776899695396,
+            0.01810702309012413,
+            -0.007055388763546944,
+            -0.005541306920349598,
+            0.00591692328453064,
+            -0.009951893240213394,
+            -0.008480406366288662,
+            0.011802867986261845,
+            0.027137301862239838,
+            -0.002747419523075223,
+            -0.029987338930368423,
+            -0.02688947319984436,
+            -0.014335372485220432,
+            -0.01840132102370262,
+            -0.017704300582408905,
+            -0.008627555333077908,
+            0.0045035220682621,
+            0.00030300996149890125,
+            0.016186347231268883,
+            0.014962689951062202,
+            -0.000480653194244951,
+            0.0028597170021384954,
+            -2.7711375878425315e-05,
+            -0.001763847190886736,
+            0.006888878531754017,
+            -0.004933350719511509,
+            0.026997897773981094,
+            0.014614180661737919,
+            -0.006594581063836813,
+            0.018711106851696968,
+            0.028314489871263504,
+            0.020430423319339752,
+            0.003157886676490307,
+            -0.010695381090044975,
+            -0.014606435783207417,
+            0.009727297350764275,
+            -0.013421501964330673,
+            0.023264970630407333,
+            0.022552460432052612,
+            -0.01480779703706503,
+            -0.014885243959724903,
+            -0.017905661836266518,
+            0.015210519544780254,
+            -0.007562664337456226,
+            -0.008224832825362682,
+            0.012368228286504745,
+            0.022335609421133995,
+            -0.007024410180747509,
+            0.002371803391724825,
+            -0.0010232636705040932,
+            -0.021731525659561157,
+            -0.006122156511992216,
+            -0.004596457816660404,
+            -0.008666278794407845,
+            0.008681767620146275,
+            0.008131897076964378,
+            -0.001928421319462359,
+            -0.006319645792245865,
+            0.005471604876220226,
+            -0.013367289677262306,
+            0.01301103550940752,
+            -0.00789568480104208,
+            0.03379771113395691,
+            -0.009138703346252441,
+            -0.0006733016925863922,
+            -0.022598929703235626,
+            -0.011446612887084484,
+            0.00845717266201973,
+            0.0017967619933187962,
+            0.01682141050696373,
+            -0.0011626676423475146,
+            -0.005688455421477556,
+            0.009913169778883457,
+            -0.0010687635513022542,
+            -0.001450188341550529,
+            -0.006892750971019268,
+            -0.010401083156466484,
+            0.0002884887217078358,
+            0.013258864171802998,
+            -0.006025348324328661,
+            0.0018054747488349676,
+            0.006095050368458033,
+            -0.0175339188426733,
+            0.029027000069618225,
+            0.023884542286396027,
+            0.017255110666155815,
+            0.014660648070275784,
+            0.0006394187803380191,
+            0.011647974140942097,
+            -0.003572226269170642,
+            0.0038820127956569195,
+            -0.020337486639618874,
+            0.021080974489450455,
+            -0.010811550542712212,
+            0.01846327818930149,
+            0.011655719019472599,
+            -0.015233753249049187,
+            -0.005022414494305849,
+            0.021406250074505806,
+            0.009967382065951824,
+            0.00889087375253439,
+            0.0017541663255542517,
+            -0.02470547705888748,
+            0.017766257748007774,
+            0.010099041275680065,
+            -0.020399443805217743,
+            0.019547531381249428,
+            -0.009471723809838295,
+            0.01233724970370531,
+            -0.0005300253978930414,
+            0.004418330732733011,
+            0.0058743273839354515,
+            0.021499186754226685,
+            0.011570527218282223,
+            -0.03884723410010338,
+            -0.013615119270980358,
+            -0.0017357728211209178,
+            0.006857899948954582,
+            -0.0036303112283349037,
+            -0.015233753249049187,
+            -0.041635312139987946,
+            0.01604694314301014,
+            0.009998360648751259,
+            -0.01643417589366436,
+            0.004987563472241163,
+            -0.002427952131256461,
+            -0.0042053526267409325,
+            -0.008170620538294315,
+            0.003186929039657116,
+            -0.004170501604676247,
+            0.02324948087334633,
+            0.0018984107300639153,
+            -0.006946963258087635,
+            0.013258864171802998,
+            0.01344473659992218,
+            -0.021948376670479774,
+            0.0009874446550384164,
+            0.0021375271026045084,
+            0.000636998622212559,
+            0.023683181032538414,
+            -0.014707116410136223,
+            -0.0017657832941040397,
+            -0.01685238815844059,
+            -0.0013069120468571782,
+            -0.021592123433947563,
+            -0.001188805908896029,
+            -0.015187285840511322,
+            -0.008511385880410671,
+            0.025030752643942833,
+            -0.011005166918039322,
+            0.010997422970831394,
+            -0.006687517277896404,
+            0.006943091284483671,
+            -0.006904367823153734,
+            -0.01936165988445282,
+            -0.0008267428493127227,
+            0.009975126944482327,
+            -0.000512115890160203,
+            0.002941036131232977,
+            8.482827252009884e-05,
+            0.002464739140123129,
+            0.008867640048265457,
+            0.006234454456716776,
+            -0.0075278133153915405,
+            0.02489134855568409,
+            -0.012073930352926254,
+            0.0016399325104430318,
+            0.011911292560398579,
+            0.0004743606550619006,
+            -0.010424317792057991,
+            0.025588368996977806,
+            -0.02139076218008995,
+            -0.00800798274576664,
+            -0.00014799568452872336,
+            -0.0016544538084417582,
+            -0.006513262167572975,
+            0.006718495860695839,
+            0.02946070022881031,
+            0.0062654330395162106,
+            0.010447551496326923,
+            -0.0281441081315279,
+            -0.011493081226944923,
+            0.012538610957562923,
+            -0.022273652255535126,
+            -0.0007739823195151985,
+            -0.0004322490422055125,
+            -0.00619960343465209,
+            0.026378324255347252,
+            -0.02022906206548214,
+            -0.01093546487390995,
+            -0.018664639443159103,
+            0.005405775271356106,
+            -0.011121337302029133,
+            0.008294534869492054,
+            -0.0009777637897059321,
+            0.006540368776768446,
+            0.016805920749902725,
+            -0.019563021138310432,
+            0.016031453385949135,
+            0.005119222681969404,
+            -0.016217326745390892,
+            0.010222956538200378,
+            -0.017580386251211166,
+            -0.0008199662552215159,
+            0.020275529474020004,
+            -0.00400399137288332,
+            0.008751469664275646,
+            -0.003363120136782527,
+            0.005208286456763744,
+            -0.006513262167572975,
+            0.0001492057926952839,
+            -0.007249005138874054,
+            0.014428308233618736,
+            -0.016805920749902725,
+            -0.017828216776251793,
+            -0.008821171708405018,
+            0.0016709112096577883,
+            0.021297825500369072,
+            0.011136826127767563,
+            -0.010044828988611698,
+            -0.011059380136430264,
+            -0.011694442480802536,
+            0.041604332625865936,
+            0.005762029904872179,
+            0.027230238541960716,
+            -0.02261441759765148,
+            0.00545998802408576,
+            0.025170156732201576,
+            -0.014242436736822128,
+            0.0012788375606760383,
+            -0.0022788674104958773,
+            0.0053050946444272995,
+            0.011562783271074295,
+            -0.01819995976984501,
+            0.0049139889888465405,
+            -0.00019216447253711522,
+            -0.00782985519617796,
+            -0.000656844291370362,
+            0.003442503046244383,
+            0.00020196630794089288,
+            0.016325751319527626,
+            -0.004027225077152252,
+            -0.016511622816324234,
+            -0.004429947584867477,
+            -0.0003216455806978047,
+            -0.027896279469132423,
+            0.013514438644051552,
+            0.016589069738984108,
+            0.01566745527088642,
+            0.027989214286208153,
+            -0.014513500034809113,
+            0.016031453385949135,
+            0.009533680975437164,
+            0.013901671394705772,
+            -0.010610189288854599,
+            0.008952830918133259,
+            -0.01595400646328926,
+            0.007446494419127703,
+            -0.03454120084643364,
+            -0.007887939922511578,
+            -0.00802347157150507,
+            0.01984182931482792,
+            0.028763681650161743,
+            0.002923610620200634,
+            0.006493900436908007,
+            0.000196883876924403,
+            0.0024202074855566025,
+            -0.01714668609201908,
+            -0.020182594656944275,
+            -0.0003337466041557491,
+            0.0029332914855331182,
+            0.0006950835813768208,
+            -0.004495777189731598,
+            0.00472037261351943,
+            -0.012391461990773678,
+            0.0026835259050130844,
+            -0.013134949840605259,
+            0.005971135571599007,
+            -0.0059982421807944775,
+            0.009603383019566536,
+            0.0019429425010457635,
+            0.020864123478531837,
+            0.02300165221095085,
+            0.004100799560546875,
+            0.0046971384435892105,
+            -0.024333734065294266,
+            0.00920066051185131,
+            -0.013847459107637405,
+            0.012329504825174809,
+            0.019408127292990685,
+            0.008248066529631615,
+            0.006571347359567881,
+            0.01233724970370531,
+            0.012918098829686642,
+            -0.0031772484071552753,
+            0.024287264794111252,
+            0.005014669615775347,
+            -0.0009070937521755695,
+            0.002052335999906063,
+            -0.014954946003854275,
+            -0.0070979841984808445,
+            -0.0015286030247807503,
+            0.0018848575418815017,
+            -0.004731989465653896,
+            0.01906736195087433,
+            0.015256987884640694,
+            -0.01050176378339529,
+            0.017874684184789658,
+            -0.03937387093901634,
+            0.017379024997353554,
+            -0.0027377386577427387,
+            -0.0060911779291927814,
+            -0.00545998802408576,
+            0.00665653869509697,
+            0.004515138920396566,
+            0.003669034456834197,
+            0.03773200139403343,
+            -0.010408828034996986,
+            0.003963331691920757,
+            -0.005847221240401268,
+            0.007121218368411064,
+            -0.020864123478531837,
+            0.008302279748022556,
+            0.005336073227226734,
+            0.02160761132836342,
+            -0.0056497324258089066,
+            -0.0014482521219179034,
+            0.02328045852482319,
+            0.008859895169734955,
+            0.03469609469175339,
+            -0.01566745527088642,
+            -0.004418330732733011,
+            0.01048627495765686,
+            0.0207402091473341,
+            -0.00580462533980608,
+            -0.008976065553724766,
+            0.007903429679572582,
+            0.0035586729645729065,
+            0.021824462339282036,
+            -0.00515407370403409,
+            -0.0006650729919783771,
+            0.0054367538541555405,
+            -0.0060911779291927814,
+            0.009735042229294777,
+            0.011252996511757374,
+            -0.009076746180653572,
+            0.007539430167526007,
+            0.021530164405703545,
+            -0.009975126944482327,
+            -0.008658533915877342,
+            -0.009649851359426975,
+            0.0010658593382686377,
+            -0.03068435750901699,
+            0.001649613375775516,
+            0.005359307397156954,
+            0.015187285840511322,
+            -0.0004603234410751611,
+            -0.009014789015054703,
+            0.021019017323851585,
+            0.017255110666155815,
+            -0.005936284549534321,
+            -0.0028132491279393435,
+            0.0023795480374246836,
+            -0.0017154429806396365,
+            -0.006160879973322153,
+            -0.0006248975405469537,
+            -0.002656419761478901,
+            0.014730350114405155,
+            -0.0015469965292140841,
+            -0.006617815233767033,
+            -0.00756653631106019,
+            0.0033302053343504667,
+            0.010834784246981144,
+            -0.005661349277943373,
+            0.013057502917945385,
+            0.016805920749902725,
+            -0.00031704717548564076,
+            0.019671445712447166,
+            0.01037010457366705,
+            -0.010447551496326923,
+            0.015938518568873405,
+            -0.007984748110175133,
+            0.006563602481037378,
+            0.0164031982421875,
+            -0.01277095079421997,
+            -0.01246116403490305,
+            0.007210282143205404,
+            0.00665653869509697,
+            0.028918573632836342,
+            0.014792308211326599,
+            -0.02109646424651146,
+            0.014908477663993835,
+            0.015768134966492653,
+            -0.013018779456615448,
+            0.003537375247105956,
+            -0.01784370467066765,
+            0.020260039716959,
+            0.019051872193813324,
+            -0.015535795129835606,
+            -0.017224133014678955,
+            -0.014947201125323772,
+            -0.005092116538435221,
+            0.006331262644380331,
+            0.0033263331279158592,
+            -0.00617636926472187,
+            -0.027989214286208153,
+            -0.0049139889888465405,
+            -0.018633661791682243,
+            0.013870692811906338,
+            -0.008674023672938347,
+            0.0007715621031820774,
+            -0.004511266481131315,
+            0.007984748110175133,
+            -0.0033882902935147285,
+            0.032465629279613495,
+            -0.0051811798475682735,
+            -0.02832997962832451,
+            0.004398969002068043,
+            0.016356728971004486,
+            0.012623801827430725,
+            -0.029243851080536842,
+            0.010463040322065353,
+            0.004805563949048519,
+            -0.012530866079032421,
+            0.005990497302263975,
+            -0.02258343994617462,
+            -0.007632365915924311,
+            -0.0009395244996994734,
+            -0.016743963584303856,
+            -0.027121812105178833,
+            -0.008433938957750797,
+            -0.006993431597948074,
+            -0.005401902832090855,
+            -0.009525936096906662,
+            -0.008720491081476212,
+            -0.011477591469883919,
+            0.02303262986242771,
+            -0.007899557240307331,
+            -0.001866463921032846,
+            -0.0014637414133176208,
+            0.00930908601731062,
+            -0.023776117712259293,
+            -0.015605497173964977,
+            0.006389347370713949,
+            -0.002594462363049388,
+            0.022815778851509094,
+            -0.012747716158628464,
+            0.005940156988799572,
+            -0.009316830895841122,
+            0.009851212613284588,
+            0.015241498127579689,
+            0.004956584889441729,
+            0.017440982162952423,
+            -0.009889936074614525,
+            0.0021026763133704662,
+            -0.008627555333077908,
+            -0.00018986526993103325,
+            -1.027075541060185e-05,
+            0.014428308233618736,
+            0.007624621503055096,
+            0.014141756109893322,
+            0.017487451434135437,
+            -0.013553161174058914,
+            0.005003052763640881,
+            -0.0003274540649726987,
+            -0.017255110666155815,
+            0.0027725896798074245,
+            -0.007984748110175133,
+            -0.0008896682411432266,
+            0.001758038648404181,
+            6.395397940650582e-05,
+            -0.02112744189798832,
+            -0.0009124181815423071,
+            0.0018809852190315723,
+            0.0010222955606877804,
+            -0.025201136246323586,
+            0.00889087375253439,
+            -0.00889087375253439,
+            -0.0014918158994987607,
+            -0.019113829359412193,
+            0.019795360043644905,
+            0.026300877332687378,
+            -0.004619691986590624,
+            0.00036206303047947586,
+            -0.013537672348320484,
+            -0.02049238048493862,
+            0.0001489637652412057,
+            -0.011314953677356243,
+            -0.011345932260155678,
+            -0.0022169100120663643,
+            -0.003214035416021943,
+            0.008805682882666588,
+            0.0004189379105810076,
+            -0.009433000348508358,
+            -0.004956584889441729,
+            -0.011764144524931908,
+            -0.017549408599734306,
+            -0.010401083156466484,
+            0.01933068037033081,
+            -0.0018016024259850383,
+            0.0020794421434402466,
+            -0.0032508226577192545,
+            0.0005271211848594248,
+            0.011748654767870903,
+            0.013475715182721615,
+            0.013344055972993374,
+            -0.023915521800518036,
+            -0.017890173941850662,
+            -0.013320821337401867,
+            0.0029487807769328356,
+            -0.007187047973275185,
+            0.004844287410378456,
+            0.026362836360931396,
+            -0.0009695350890979171,
+            -0.01091997604817152,
+            0.002327271504327655,
+            0.019051872193813324,
+            -0.031272951513528824,
+            -0.013390523381531239,
+            -0.025077221915125847,
+            0.02225816436111927,
+            -0.013351799920201302,
+            0.011516314931213856,
+            0.010191977955400944,
+            -0.025944624096155167,
+            0.006768836174160242,
+            -0.007686578668653965,
+            0.026037558913230896,
+            0.01634124107658863,
+            0.0005934348446317017,
+            -0.00944074522703886,
+            0.013622863218188286,
+            -0.0008722427301108837,
+            -0.013576395809650421,
+            0.002201420720666647,
+            0.014645159244537354,
+            -0.014691627584397793,
+            -0.009510447271168232,
+            -0.004507394507527351,
+            0.004313777666538954,
+            0.012763205915689468,
+            -0.0030339721124619246,
+            -0.004031097516417503,
+            -0.011624740436673164,
+            0.013344055972993374,
+            -0.003930416889488697,
+            0.012383717112243176,
+            0.007380664348602295,
+            -0.014056564308702946,
+            -0.019532041624188423,
+            0.033333033323287964,
+            0.00400399137288332,
+            0.00338054564781487,
+            -0.010997422970831394,
+            0.02740062028169632,
+            -0.009897680021822453,
+            0.010571465827524662,
+            -0.006408709101378918,
+            0.0031985461246222258,
+            0.012321759946644306,
+            -0.012104909867048264,
+            -0.01981084980070591,
+            0.021468207240104675,
+            -0.00364192808046937,
+            -0.0010193913476541638,
+            0.008310024626553059,
+            -0.020182594656944275,
+            -0.00663330452516675,
+            0.002180122770369053,
+            0.009541425853967667,
+            -0.011818356812000275,
+            0.006369986105710268,
+            0.017905661836266518,
+            0.003912991378456354,
+            -0.0007672057254239917,
+            0.019996721297502518,
+            -0.01672847382724285,
+            0.011717676185071468,
+            -0.010850274004042149,
+            0.0016825281782075763,
+            0.0006065039779059589,
+            -0.0048597767017781734,
+            0.0058743273839354515,
+            -0.014536733739078045,
+            0.011988739483058453,
+            -6.244135147426277e-05,
+            -0.013979118317365646,
+            0.007601387333124876,
+            -0.013359544798731804,
+            -0.016836898401379585,
+            0.041542377322912216,
+            -0.005301222205162048,
+            -0.0038665232714265585,
+            -0.0056962003000080585,
+            0.01350669376552105,
+            0.02823704481124878,
+            -0.005336073227226734,
+            -0.01153180468827486,
+            -0.010780571959912777,
+            -0.016573579981923103,
+            0.015907539054751396,
+            -0.01810702309012413,
+            0.002495717955753207,
+            -0.010532742366194725,
+            -0.007903429679572582,
+            0.004302160814404488,
+            -0.003516077296808362,
+            0.014722606167197227,
+            0.00950270239263773,
+            0.023357905447483063,
+            0.00797700323164463,
+            -0.00027348342700861394,
+            0.010393339209258556,
+            0.011663463898003101,
+            0.003243078012019396,
+            -0.010509508661925793,
+            -0.00011949774489039555,
+            -0.02894955314695835,
+            -0.008449427783489227,
+            -0.004863648675382137,
+            -0.001485039247199893,
+            -0.00428667152300477,
+            -0.01239920686930418,
+            0.006617815233767033,
+            -0.0022052929271012545,
+            0.013909416273236275,
+            -0.009254872798919678,
+            -0.017766257748007774,
+            0.03389064967632294,
+            0.004495777189731598,
+            0.006466794293373823,
+            -0.008286789990961552,
+            -0.03705047070980072,
+            -0.0032566310837864876,
+            -0.008116407319903374,
+            -0.013529927469789982,
+            0.004538373090326786,
+            0.01283290795981884,
+            -0.014831030741333961,
+            -0.028097640722990036,
+            0.0019390701781958342,
+            -0.006683644838631153,
+            0.02978597767651081,
+            -0.007798876613378525,
+            0.00419760774821043,
+            0.0037251831963658333,
+            0.0011558911064639688,
+            0.01239920686930418,
+            -0.00845717266201973,
+            -0.0022536972537636757,
+            0.016387708485126495,
+            -0.010858018882572651,
+            0.011051635257899761,
+            -0.010966443456709385,
+            -0.02554190158843994,
+            -0.01456771232187748,
+            0.021344292908906937,
+            -0.0004823473282158375,
+            -0.00024903935263864696,
+            0.014823286794126034,
+            0.02354377694427967,
+            0.011260741390287876,
+            0.00969631876796484,
+            0.006408709101378918,
+            -0.001035848748870194,
+            0.005975008010864258,
+            0.006095050368458033,
+            -0.0024163350462913513,
+            -0.013398268260061741,
+            0.017564896494150162,
+            0.009773765690624714,
+            -0.01810702309012413,
+            0.0008514289511367679,
+            0.0036612898111343384,
+            0.0168833676725626,
+            0.006935346405953169,
+            0.009316830895841122,
+            -0.0143121387809515,
+            -0.023760627955198288,
+            -0.00014932679187040776,
+            0.002873270306736231,
+            0.015721667557954788,
+            0.010579210706055164,
+            -0.014319883659482002,
+            0.0005963391158729792,
+            0.008712747134268284,
+            0.020213572308421135,
+            -0.024209819734096527,
+            0.019702425226569176,
+            -0.004673904739320278,
+            -0.0021084847394376993,
+            0.007946024648845196,
+            0.011632485315203667,
+            -0.011872569099068642,
+            -0.00650551775470376,
+            0.006249943748116493,
+            0.003157886676490307,
+            0.03172214329242706,
+            -0.0017260919557884336,
+            0.009665340185165405,
+            -0.010997422970831394,
+            -0.0002405686245765537,
+            0.0006728176376782358,
+            -0.0006820144481025636,
+            -0.002884887158870697,
+            -0.011020656675100327,
+            0.0018974426202476025,
+            -0.020306508988142014,
+            0.0024473138619214296,
+            0.00463130883872509,
+            -0.005684582982212305,
+            -0.0032972905319184065,
+            -0.019222255796194077,
+            -0.0022672503255307674,
+            0.003293418325483799,
+            0.014335372485220432,
+            -0.031691163778305054,
+            0.004972074180841446,
+            0.010672146454453468,
+            -0.002309845993295312,
+            -0.010416572913527489,
+            0.011423379182815552,
+            -0.0026060794480144978,
+            0.011988739483058453,
+            -0.0038142469711601734,
+            -0.0004905760288238525,
+            0.025681305676698685,
+            -0.005308967083692551,
+            0.014707116410136223,
+            -0.01026167906820774,
+            -0.002993312431499362,
+            -0.028252532705664635,
+            0.005235392600297928,
+            -0.007403898518532515,
+            -0.0012081675231456757,
+            -0.004402841441333294,
+            -0.01130720879882574,
+            -0.0026990154292434454,
+            0.002122037811204791,
+            0.008000237867236137,
+            0.012910354882478714,
+            -0.0007725302129983902,
+            -0.010075807571411133,
+            0.004832670092582703,
+            0.0075355577282607555,
+            -0.015520306304097176,
+            -0.0070437719114124775,
+            -0.020306508988142014,
+            0.029662061482667923,
+            -0.013731288723647594,
+            -0.0011123273288831115,
+            -0.020864123478531837,
+            0.010160998441278934,
+            -0.0063119009137153625,
+            -0.002782270545139909,
+            0.004708755761384964,
+            -0.015109838917851448,
+            0.01344473659992218,
+            0.008751469664275646,
+            0.01450575515627861,
+            -0.02151467651128769,
+            -0.01762685552239418,
+            -0.0044725434854626656,
+            -0.002735802438110113,
+            -0.010648912750184536,
+            -0.0005334137240424752,
+            0.002013612538576126,
+            0.021251358091831207,
+            -0.00032503384863957763,
+            0.0042053526267409325,
+            -0.003214035416021943,
+            0.017224133014678955,
+            -0.010137764737010002,
+            -0.013057502917945385,
+            0.0024260159116238356,
+            0.005994369741529226,
+            0.016186347231268883,
+            0.03158273920416832,
+            -0.020693741738796234,
+            -0.00272999401204288,
+            0.0032740565948188305,
+            0.010656657628715038,
+            -0.004085310269147158,
+            6.304640555754304e-05,
+            -0.027803342789411545,
+            -0.04464024305343628,
+            -0.002422143705189228,
+            0.013057502917945385,
+            -0.015636475756764412,
+            0.012670270167291164,
+            0.008743725717067719,
+            0.010021595284342766,
+            -0.005196669604629278,
+            0.02620794251561165,
+            -0.004793947096914053,
+            0.014304393902420998,
+            0.011733165942132473,
+            0.011911292560398579,
+            0.0009874446550384164,
+            0.0007459079497493804,
+            -0.018354853615164757,
+            -0.008255811408162117,
+            -0.0015954007394611835,
+            0.01765783317387104,
+            -0.027478067204356194,
+            0.008526874706149101,
+            -0.011516314931213856,
+            0.013839714229106903,
+            0.0003666614356916398,
+            0.005661349277943373,
+            -0.01771979033946991,
+            0.012445674277842045,
+            -0.0009337160154245794,
+            -0.025325050577521324,
+            0.004805563949048519,
+            0.011276230216026306,
+            -0.010362359695136547,
+            0.006877261213958263,
+            0.019950253888964653,
+            0.0023079097736626863,
+            0.008495896123349667,
+            -0.014126266352832317,
+            0.010703125037252903,
+            0.009649851359426975,
+            -0.021080974489450455,
+            -0.01233724970370531,
+            -0.007059261202812195,
+            -0.006335135083645582,
+            -0.0035470561124384403,
+            -0.00013105422840453684,
+            0.010176488198339939,
+            -0.016000475734472275,
+            0.010099041275680065,
+            -0.006559730041772127,
+            0.006400964688509703,
+            0.00944074522703886,
+            0.001115231541916728,
+            -0.013576395809650421,
+            0.012716737575829029,
+            -0.0218709297478199,
+            0.008426194079220295,
+            -0.04287445917725563,
+            -0.005847221240401268,
+            -0.015078860335052013,
+            0.005204414017498493,
+            -0.003798757679760456,
+            -0.004453181754797697,
+            -0.006203475873917341,
+            -0.005061137955635786,
+            0.00920066051185131,
+            0.021747015416622162,
+            0.012135888449847698,
+            -0.016620049253106117,
+            -0.007551047019660473,
+            0.005425137002021074,
+            0.0004854935978073627,
+            -0.013336311094462872,
+            0.009316830895841122,
+            -0.01576039008796215,
+            0.0009259713697247207,
+            0.006339007057249546,
+            -0.01072635967284441,
+            0.013452480547130108,
+            0.004731989465653896,
+            -0.003401843598112464,
+            0.013390523381531239,
+            -0.0050417762249708176,
+            -0.0057891360484063625,
+            -0.007272239308804274,
+            -0.013739033602178097,
+            -0.003818119177594781,
+            -0.003072695340961218,
+            0.012747716158628464,
+            0.001529571134597063,
+            -0.0003243077953811735,
+            -0.00782985519617796,
+            0.009154192171990871,
+            0.002220782218500972,
+            -0.0013146566925570369,
+            0.005773646757006645,
+            -0.02300165221095085,
+            0.0001287550403503701,
+            0.004755223635584116,
+            -0.0009487213101238012,
+            -0.006246071308851242,
+            -0.012151377275586128,
+            -0.011903548613190651,
+            -0.038165703415870667,
+            0.009363298304378986,
+            0.0019197085639461875,
+            -0.0076207490637898445,
+            0.0064822835847735405,
+            0.017255110666155815,
+            0.01456771232187748,
+            0.016743963584303856,
+            -0.020693741738796234,
+            -0.02303262986242771,
+            0.005556796211749315,
+            -0.009402021765708923,
+            0.02261441759765148,
+            0.004046586807817221,
+            -0.005847221240401268,
+            0.0013243375578895211,
+            -0.013258864171802998,
+            0.008929597213864326,
+            0.005723306443542242,
+            -0.001592496526427567,
+            0.005425137002021074,
+            -0.013692565262317657,
+            -0.010904486291110516,
+            0.01048627495765686,
+            0.032496608793735504,
+            -0.0004261985304765403,
+            0.0024105266202241182,
+            0.023295948281884193,
+            0.015466093085706234,
+            0.016356728971004486,
+            -0.021111954003572464,
+            0.011167804710566998,
+            0.010060317814350128,
+            -0.0018645278178155422,
+            -0.014892988838255405,
+            0.00513083953410387,
+            -0.012716737575829029,
+            -0.010664401575922966,
+            0.014993669465184212,
+            0.007853088900446892,
+            0.017735280096530914,
+            -0.02369867078959942,
+            0.00789568480104208,
+            0.00042765066609717906,
+            0.00013710475468542427,
+            0.011524059809744358,
+            0.012956822291016579,
+            0.03370477631688118,
+            0.027152791619300842,
+            0.020662762224674225,
+            -0.009285852313041687,
+            0.014366351068019867,
+            -0.009192915633320808,
+            -0.011090358719229698,
+            -0.019470084458589554,
+            -0.015249243006110191,
+            -0.011136826127767563,
+            0.017998598515987396,
+            0.008790193125605583,
+            -0.009355553425848484,
+            -0.005947901867330074,
+            0.01586107164621353,
+            -0.003909118939191103,
+            -0.004635181277990341,
+            0.01646515540778637,
+            -0.026269899681210518,
+            0.016836898401379585,
+            -0.000745423894841224,
+            -0.014668392948806286,
+            -0.014691627584397793,
+            -0.004182118456810713,
+            -0.012670270167291164,
+            0.0003964299685321748,
+            -0.020817656069993973,
+            0.013181418180465698,
+            0.024597052484750748,
+            -0.0030920570716261864,
+            -0.004100799560546875,
+            -0.011299464851617813,
+            -0.0005111477803438902,
+            0.018385831266641617,
+            0.0034773540683090687,
+            0.009192915633320808,
+            -0.0023872926831245422,
+            0.013924905098974705,
+            0.00524313747882843,
+            0.0027958236169070005,
+            -0.0007565568666905165,
+            -0.016263794153928757,
+            0.012197845615446568,
+            0.0026835259050130844,
+            0.012135888449847698,
+            -0.022242674604058266,
+            0.007938280701637268,
+            -0.011764144524931908,
+            -0.0023698671720921993,
+            -0.02228914201259613,
+            0.0189744271337986,
+            -0.01610890030860901,
+            0.001046497724018991,
+            0.003841353114694357,
+            0.0006224773242138326,
+            0.009580149315297604,
+            0.01610890030860901,
+            -0.023388884961605072,
+            -0.0003460896550677717,
+            -0.014908477663993835,
+            -0.021917399019002914,
+            -0.01607792265713215,
+            0.004728117026388645,
+            -0.019501063972711563,
+            0.005444498732686043,
+            -0.0025557391345500946,
+            -0.015125328674912453,
+            0.018478767946362495,
+            0.014691627584397793,
+            -0.004379607271403074,
+            0.002412462839856744,
+            0.005425137002021074,
+            -0.0014405074762180448,
+            0.01971791312098503,
+            0.00013274837692733854,
+            -0.0010048701660707593,
+            -0.009425255469977856,
+            -0.0168833676725626,
+            -0.0065752193331718445,
+            -0.035005878657102585,
+            0.0260530486702919,
+            -0.004670032300055027,
+            -0.016511622816324234,
+            0.014319883659482002,
+            0.009115469641983509,
+            -0.014629669487476349,
+            -0.007059261202812195,
+            0.016294771805405617,
+            -0.024287264794111252,
+            0.02244403585791588,
+            -0.009611127898097038,
+            -0.008774704299867153,
+            0.00784921646118164,
+            0.05154848098754883,
+            0.009146448224782944,
+            0.003492843359708786,
+            0.011036145500838757,
+            0.005839476361870766,
+            -0.0013165927957743406,
+            -0.023683181032538414,
+            0.005568413063883781,
+            -0.007190920412540436,
+            -0.003601268632337451,
+            0.002536377403885126,
+            -0.004801691509783268,
+            0.031149037182331085,
+            -0.004379607271403074,
+            -0.005289605353027582,
+            0.004484160337597132,
+            -0.02196386642754078,
+            -0.0001332324172835797,
+            0.023203013464808464,
+            -2.3128106477088295e-05,
+            -0.023590246215462685,
+            -0.006571347359567881,
+            -0.015071115456521511,
+            -0.013770012184977531,
+            -0.018989915028214455,
+            0.005231520161032677,
+            -0.012755461037158966,
+            -0.0033940989524126053,
+            0.0016660707769915462,
+            -0.005940156988799572,
+            -0.002582845278084278,
+            0.009603383019566536,
+            0.05362405255436897,
+            0.014103032648563385,
+            -0.006335135083645582,
+            -0.005854965653270483,
+            0.009192915633320808,
+            -0.010524998418986797,
+            -0.019795360043644905,
+            -0.009820234030485153,
+            -0.029507169499993324,
+            -0.012747716158628464,
+            -0.0143121387809515,
+            -0.010191977955400944,
+            -0.010277168825268745,
+            0.00963436160236597,
+            -0.018122512847185135,
+            -0.007628493942320347,
+            -0.020337486639618874,
+            0.002941036131232977,
+            -0.011268485337495804,
+            -0.014714861288666725,
+            -0.008054450154304504,
+            -0.04807887226343155,
+            0.005796880926936865,
+            0.019593998789787292,
+            0.01263929158449173,
+            0.002515079453587532,
+            0.004383479710668325,
+            -0.01993476413190365,
+            -0.030792783945798874,
+            0.013839714229106903,
+            0.011283975094556808,
+            0.011733165942132473,
+            0.03283737599849701,
+            -0.011709931306540966,
+            -0.00813964195549488,
+            -0.004693266469985247,
+            -0.0017251238459721208,
+            0.012066186405718327,
+            0.00969631876796484,
+            -0.004917861428111792,
+            -0.012375972233712673,
+            0.016217326745390892,
+            -0.01981084980070591,
+            -0.023683181032538414,
+            -0.01672847382724285,
+            0.006157007534056902,
+            0.00813964195549488,
+            -0.008054450154304504,
+            0.002973950933665037,
+            0.0056962003000080585,
+            -0.007159941829741001,
+            -0.00022423222253564745,
+            0.004890755284577608,
+            -0.023048119619488716,
+            0.0008490087930113077,
+            -0.002257569460198283,
+            0.0061376458033919334,
+            0.021266845986247063,
+            -0.010168743319809437,
+            -0.0014075926737859845,
+            -0.0053902859799563885,
+            -0.0028984404634684324,
+            -0.006764963734894991,
+            0.02620794251561165,
+            0.009154192171990871,
+            -0.009882191196084023,
+            -0.01011453103274107,
+            -0.01129171997308731,
+            -0.0011278166202828288,
+            -0.017255110666155815,
+            -0.016186347231268883,
+            0.0015566773945465684,
+            -0.016712984070181847,
+            0.013971373438835144,
+            -0.00876695942133665,
+            0.00400399137288332,
+            -0.014110777527093887,
+            0.02019808255136013,
+            0.01929970271885395,
+            -0.009711808525025845,
+            0.023776117712259293,
+            -0.007709812838584185,
+            -0.038382552564144135,
+            -0.00800798274576664,
+            0.01843230053782463,
+            -0.006118284538388252,
+            0.010037084110081196,
+            -0.011314953677356243,
+            0.008209343999624252,
+            0.00015319912927225232,
+            0.0018122512847185135,
+            -0.0003112386621069163,
+            0.027555514127016068,
+            0.015535795129835606,
+            -0.005560668651014566,
+            -0.005494839046150446,
+            0.004255692940205336,
+            0.009541425853967667,
+            0.017580386251211166,
+            0.012151377275586128,
+            -0.021019017323851585,
+            -0.014257925562560558,
+            -0.016263794153928757,
+            0.011012911796569824,
+            0.015125328674912453,
+            -0.0040891822427511215,
+            0.01691434532403946,
+            0.014126266352832317,
+            -0.019098341464996338,
+            0.0019884423818439245,
+            -0.008193854242563248,
+            -0.00158184755127877,
+            -0.03029712475836277,
+            0.011043890379369259,
+            0.0074387495405972,
+            -0.0072218989953398705,
+            0.036616768687963486,
+            0.009858956560492516,
+            0.037577107548713684,
+            0.018680129200220108,
+            0.0008974128868430853,
+            0.009045767597854137,
+            -0.018370341509580612,
+            0.013065247796475887,
+            0.01500141341239214,
+            -0.012724482454359531,
+            -0.005734923295676708,
+            0.0013446672819554806,
+            0.0007551047019660473,
+            0.012631546705961227,
+            0.02004319056868553,
+            -0.01795213110744953,
+            0.009773765690624714,
+            -0.015032391995191574,
+            -0.005250881891697645,
+            0.006555858068168163,
+            -0.013932649977505207,
+            -0.008495896123349667,
+            -0.00963436160236597,
+            0.019408127292990685,
+            -0.012600568123161793,
+            0.016573579981923103,
+            -0.009595638141036034,
+            0.015520306304097176,
+            -0.029724018648266792,
+            -0.003669034456834197,
+            -0.006679772399365902,
+            -0.019593998789787292,
+            -0.012887120246887207,
+            0.009820234030485153,
+            0.010633422993123531,
+            -0.009301341138780117,
+            -0.011562783271074295,
+            0.020368466153740883,
+            -0.004185990896075964,
+            0.013529927469789982,
+            0.021189400926232338,
+            0.00737291993573308,
+            0.024132372811436653,
+            0.0005605200421996415,
+            0.014164989814162254,
+            0.005463860463351011,
+            0.010362359695136547,
+            -0.004677776712924242,
+            0.005161818582564592,
+            0.00932457484304905,
+            0.014203713275492191,
+            0.017766257748007774,
+            0.013196907006204128,
+            -0.00598275288939476,
+            0.007419387809932232,
+            0.02016710489988327,
+            0.0016254113288596272,
+            -0.030312614515423775,
+            -0.009719553403556347,
+            0.0001736498816171661,
+            0.00924712885171175,
+            0.009185170754790306,
+            3.817877222900279e-05,
+            -0.01093546487390995,
+            0.011144571006298065,
+            0.00969631876796484,
+            0.002286612056195736,
+            0.006280922330915928,
+            -0.01456771232187748,
+            -0.020957060158252716,
+            0.006106667220592499,
+            0.01798310875892639,
+            0.013917161151766777,
+            -0.014722606167197227,
+            0.022784801200032234,
+            0.006145390681922436,
+            -0.021406250074505806,
+            -0.01840132102370262,
+            -0.0009559819591231644,
+            0.0004542729293461889,
+            -0.002265314105898142,
+            0.03720536455512047,
+            -0.0011539548868313432,
+            -0.005955646280199289,
+            -0.0054754773154854774,
+            0.02775687538087368,
+            0.02049238048493862,
+            0.009177426807582378
+          ],
+          "embedding": [
+            0.0008215984936325879,
+            -0.019349418284688353,
+            -0.02322660633029795,
+            0.009747820845194852,
+            0.007353771220903641,
+            0.015252908783199839,
+            -0.012176175829980202,
+            0.051898051094966605,
+            -0.0068130353886146165,
+            0.008189212816890595,
+            0.008102287831931905,
+            0.01682801136011614,
+            0.02005557996049711,
+            -0.020779879580497183,
+            0.002332165268329781,
+            0.02337485065616377,
+            0.006074992895540845,
+            0.02576575146535689,
+            0.0007517286429767504,
+            -0.03763909942314134,
+            0.0011204729720314833,
+            0.00041212613589632806,
+            -0.06089176800314795,
+            0.012852035873395388,
+            -0.008901127092058543,
+            0.01702347055484216,
+            0.005803446395466281,
+            0.017889570275475296,
+            -0.05375047382413552,
+            0.021623778774252724,
+            0.014784317465883424,
+            -0.0018881880857885708,
+            0.0004735328554290247,
+            0.012393802850407656,
+            -0.019234442968055952,
+            -0.019566283611638122,
+            0.04407789373296535,
+            -0.0053565873209102995,
+            -0.026240866282480922,
+            0.048107902270426675,
+            -0.0054681314939387235,
+            -0.0009441397948041965,
+            -0.015205079626883548,
+            0.006885981841893229,
+            0.01403724256866746,
+            -0.026086548246792754,
+            0.0034778946051893067,
+            0.00044587533948775473,
+            0.032851328435891634,
+            -0.03882097333255334,
+            -0.011126344604254894,
+            -0.01648943336384904,
+            0.035810456555802166,
+            -0.019384589526165754,
+            -0.019430702768170285,
+            0.014297440272239852,
+            -0.007883453204663151,
+            0.050996900793086844,
+            0.06007149434153715,
+            0.03406832053483379,
+            0.025193439934671852,
+            0.00027785129663604897,
+            -0.008014391056168083,
+            -0.018806730908490114,
+            0.027193788903098558,
+            -0.006195006670328595,
+            -0.020466000344352826,
+            0.0068939884219589185,
+            0.017252276167086602,
+            -0.045073444648704136,
+            -0.049242269917908754,
+            0.023021140866008313,
+            -0.015676872472071106,
+            -0.024497517908898602,
+            0.005873633718973869,
+            -0.022828585938203683,
+            0.011748341004940436,
+            0.04624205727927451,
+            -0.015442165499674641,
+            0.018565551071491767,
+            0.015246505395507096,
+            0.004119269796224144,
+            0.004862735250819252,
+            -0.000145323321960046,
+            0.05155838952887077,
+            0.0027052423345323936,
+            -0.016218063670586664,
+            -0.07462470920944148,
+            0.043695974746295135,
+            -0.027608273720377708,
+            -0.03860912774695494,
+            0.002823110565150435,
+            0.026471469849508766,
+            -0.0026063577189074103,
+            -0.029230652014151735,
+            0.05684164470477843,
+            0.00022182868300572677,
+            0.015590845552956364,
+            -0.024939617549529242,
+            0.01035803286203917,
+            -0.010225298012384327,
+            0.019571847648759157,
+            0.01841613041441054,
+            0.0156981470538354,
+            0.021764297694793343,
+            -0.018831230007033416,
+            -0.014419529638789993,
+            -0.0029965475899652273,
+            0.028881185012095884,
+            0.09163526460163937,
+            0.022644302207811818,
+            -0.017933909936330133,
+            -0.04663928709016869,
+            0.004962951444105128,
+            -0.041333041174526026,
+            -0.027545628007617527,
+            -0.037142394579894954,
+            -0.013358143162788402,
+            0.02719686432396677,
+            -0.0034971800937504205,
+            0.04413635817371732,
+            0.04903857110328917,
+            0.03186067155630788,
+            0.007831051050269414,
+            -0.025257538623313574,
+            0.023517513773109414,
+            -0.0016527403209505964,
+            -0.0016101860707429347,
+            0.014448515185903624,
+            0.009905018312477684,
+            0.0024114582235163955,
+            0.00172043851384822,
+            -0.03818316221102173,
+            -0.02304537008110578,
+            0.01024969233570089,
+            -0.01901116356327339,
+            0.019791121460848986,
+            0.07985067340821365,
+            -0.051903191272236665,
+            -0.025924646976556942,
+            0.014766511532849252,
+            -0.002073086076894999,
+            0.030959269926800852,
+            0.006394417986617089,
+            -0.028177586693713442,
+            0.010725285336674526,
+            0.0009614052276924192,
+            0.07500253788978564,
+            -0.00036794182149960157,
+            0.004058458332826519,
+            -0.013218269166951234,
+            0.027613539268683245,
+            -0.012995170442480582,
+            0.023388707018488407,
+            0.026389423313105165,
+            -0.003068639956061266,
+            -0.02661967649632026,
+            0.026145274570635913,
+            -0.03300591322174331,
+            -0.004556072066733024,
+            0.020045722084112896,
+            0.0033437343914784865,
+            -0.010224220905483085,
+            -0.0025025498684079703,
+            0.03460103245395426,
+            0.04514362357278504,
+            0.03797031988058962,
+            -0.004586662475576956,
+            -0.06351723722596923,
+            0.06037983484369874,
+            0.019415753124363902,
+            0.028850618647994476,
+            -0.0034810237045849006,
+            0.02957938154808314,
+            0.02642652523231673,
+            0.013187653724133275,
+            -0.007856320804109569,
+            -0.007065882659482447,
+            -0.016672784625197085,
+            0.029020161154209707,
+            -0.013243058921970134,
+            -0.014900932834241732,
+            0.006001769901896033,
+            0.008648456186944271,
+            0.03484763413464497,
+            0.012279368984306564,
+            0.02379643508687133,
+            0.04933044959399941,
+            -0.021047723935647383,
+            -0.007264823701706426,
+            0.0007058774795716792,
+            -0.010512071832195085,
+            0.027489671668749157,
+            0.009731157570361245,
+            -0.03195919640758579,
+            -0.0033626031972691915,
+            -0.04282419484651207,
+            0.01873977910485268,
+            -0.014398285316695653,
+            0.012854631447494207,
+            -0.0017372959978092432,
+            -0.037331867825735776,
+            0.001088209292238142,
+            -0.012345684323522151,
+            0.037096063326414734,
+            0.005050219690024652,
+            0.0008906051515353004,
+            -0.003697709449630797,
+            0.006548479142737107,
+            -0.010727663712299397,
+            -0.007486226350610755,
+            -0.02704618481069485,
+            -0.019752552016312926,
+            -0.014632581296303486,
+            -0.0037501138119724717,
+            0.014588411795703245,
+            -0.007119137071719376,
+            0.016440258608998535,
+            -0.03686645352111013,
+            -0.023546402448713415,
+            0.005628703432152666,
+            0.024988145751166143,
+            0.011971199595808573,
+            -0.010868778329190568,
+            -0.03270046433624425,
+            -0.021950559285357267,
+            -0.011889914022657738,
+            -0.005123971523701009,
+            0.02619713302280345,
+            -0.021990810305904974,
+            0.01165257474485436,
+            -0.009117901804315863,
+            -0.007492082755249399,
+            0.010725902070291232,
+            0.01862811996181951,
+            -0.026694281868322685,
+            -0.01623776415010199,
+            -0.009403997538684658,
+            -0.02207329027097839,
+            0.025939540497196416,
+            0.014178803271471762,
+            -0.001141821318145744,
+            -0.006281187017652108,
+            -0.035993365198554964,
+            0.0026192776111449633,
+            -0.015254976771874396,
+            -0.0116872453009606,
+            -0.015780016491993135,
+            0.021421328283124968,
+            0.02889432250124136,
+            -0.01871009416296692,
+            -0.026540472640514536,
+            0.015703370682110186,
+            -0.0002849235245164438,
+            -0.02594930889995505,
+            0.04491828744809679,
+            0.044063522197410734,
+            -0.037441864491547405,
+            -0.007005339187980476,
+            -0.030381794393931814,
+            0.06901083845539985,
+            -0.017604082400603037,
+            -0.04060591381445593,
+            0.06044850935418443,
+            -0.01756490171206357,
+            0.025539717592705295,
+            0.03452018789549939,
+            0.0016217577522523927,
+            0.00018825054264059277,
+            -0.002563606092068712,
+            0.04770277039047993,
+            -0.0032010143255574255,
+            -0.03425328601475849,
+            -0.02105909767072799,
+            -0.00840881022563103,
+            -0.028699447677549002,
+            -0.015925160861589006,
+            -0.0055750742281166,
+            0.017513987082479483,
+            0.010399718005952185,
+            0.02267201252273823,
+            0.01258511878601025,
+            0.0015081978484714903,
+            0.01874981572091931,
+            0.011158889421057756,
+            -0.002607810686437211,
+            -0.003732023411827874,
+            -0.007787189972165309,
+            -0.05524012485122728,
+            -0.025110689689279583,
+            -0.03946383236703947,
+            0.004443572529325475,
+            -0.018745077598477586,
+            0.005926160888274727,
+            0.039430331582190524,
+            -0.0049500324516682,
+            0.004352795787899792,
+            0.015566139991780173,
+            0.013148156356408315,
+            -0.015279560337042242,
+            -0.03316846614871165,
+            0.0001592209771257332,
+            0.018362411161743347,
+            -0.012706582418607377,
+            -0.03424477956681259,
+            -0.001991022841226589,
+            0.0033467286047492175,
+            0.03357019284647071,
+            -0.024359110796680374,
+            0.00359117029951049,
+            0.005061818038334802,
+            0.011359582931760315,
+            -0.026793992746244076,
+            -0.018170136079817746,
+            -0.017421603453826397,
+            0.03996608926471779,
+            0.02587013058343799,
+            0.015763410145648683,
+            -0.04512536449983358,
+            -0.02293473890636517,
+            0.048575083783480195,
+            -0.009589860756342604,
+            -0.06486262416944208,
+            0.03447666009581266,
+            0.016105654870253654,
+            0.003526582643722421,
+            -0.01354754623151814,
+            0.05401575248587799,
+            -0.03563190295559527,
+            -0.017365166993914202,
+            -0.0046288709698227735,
+            -0.033657212157212645,
+            -0.007686252059880836,
+            -0.001385273642139782,
+            0.048229062789530236,
+            -0.029083855744728694,
+            -0.030465559177859865,
+            0.0024368661249980416,
+            -0.024144031154493057,
+            0.021693278579520123,
+            -0.016405088055634124,
+            -0.018439935250344958,
+            0.006948603359995389,
+            -0.033943037395199695,
+            -0.003007796423804453,
+            -0.016783195426536633,
+            -0.02270589491206155,
+            0.020874438753828814,
+            0.016724666903417947,
+            0.027033184180344852,
+            0.0019847647852695785,
+            0.017749524526371133,
+            0.0204661893602285,
+            -0.005604929968008821,
+            0.028462234633072944,
+            -0.020790459496727066,
+            -0.04183816355941877,
+            0.03946487892663695,
+            0.01192861758684541,
+            -0.0018024841275380242,
+            0.01077082591611702,
+            -0.01475239020295694,
+            -0.03952356245715099,
+            0.03683950018557526,
+            -0.056767672022551764,
+            -0.007507600090017693,
+            0.009530544352246083,
+            -0.00788856051420074,
+            -0.011204694027562521,
+            -0.010587701466318598,
+            -0.028604940377315504,
+            0.023749537635431462,
+            -0.013672419168235691,
+            0.01953449229707597,
+            -0.011449939491554354,
+            -0.012238180529977605,
+            0.05795536247491174,
+            -0.021518581774137322,
+            -0.040672955795925836,
+            0.026066659238628543,
+            -0.014061106927896723,
+            0.03328357230532424,
+            0.03957107406164427,
+            0.030670947051333917,
+            -0.004278577327978564,
+            0.025885946540364826,
+            0.0319106101668716,
+            -0.003462160036198642,
+            -0.04686868330851868,
+            0.00555303933159587,
+            0.02156143971867743,
+            -0.03739529489075782,
+            -0.01025626126223168,
+            -0.02141970191890338,
+            0.022134910980548407,
+            0.017807484142010496,
+            -0.007232643256393455,
+            0.011318041192005274,
+            -0.010489012862455913,
+            -0.013003217163997554,
+            -0.00847279880392288,
+            0.02357274819910594,
+            -0.0016485066139744577,
+            -0.005873113634992993,
+            0.006424249429163587,
+            0.004008703814301658,
+            0.0054639858216457724,
+            0.010198079500408441,
+            0.017694720520798957,
+            -0.017087021465208255,
+            -0.0025558869389290554,
+            -0.016503579212015177,
+            -0.0420082404922411,
+            -0.003563173424359332,
+            0.022924161700566468,
+            0.012345933179990547,
+            -0.00729908668365632,
+            -0.004234683369642856,
+            -0.0022962545569187934,
+            -0.020666205489143392,
+            -0.014796149043465308,
+            0.024562653223873056,
+            0.030217494971086438,
+            0.007810961433007407,
+            0.007781263640449456,
+            -0.0030348559388780655,
+            0.007310876971216526,
+            0.016099591640649783,
+            0.02311929597780174,
+            -0.04160162038353416,
+            0.027152100845897007,
+            -0.00025396315502635146,
+            0.023717123370023598,
+            -0.04186918759097667,
+            -0.017988531857836703,
+            -0.03106663560912813,
+            -0.023595079699181436,
+            0.02437134274097885,
+            -0.02317559303912261,
+            -0.012327263775257306,
+            0.000989566600374836,
+            0.00954921216572729,
+            -0.005155079395685017,
+            -0.017508404061379164,
+            -0.04582321461886958,
+            -0.0014367264231457263,
+            -0.030417229142190933,
+            -0.017829710557332023,
+            -0.001056472801554354,
+            0.005016912084754242,
+            -0.06276644565618253,
+            0.0109965681725148,
+            -0.07270669271977365,
+            -0.004542796125637244,
+            0.040139998232966566,
+            -0.017890441783630643,
+            0.055909102081208445,
+            -0.05029104308417623,
+            0.01673962668706085,
+            -0.04507293803055318,
+            0.015385506084493484,
+            0.025677372277539627,
+            0.030235171586316813,
+            -0.022189978375852563,
+            0.008461188938299745,
+            0.008033707483259569,
+            -0.045089505755429415,
+            -0.04144698743458133,
+            -0.013640256640978024,
+            0.01036782197518379,
+            0.0012696958402094232,
+            -0.016090728932978384,
+            0.023838437877040163,
+            0.011946482002767135,
+            0.03342604921859589,
+            0.04211801335157251,
+            0.002708015395637621,
+            -0.047156646610976634,
+            -0.013104464812562618,
+            -0.003933116891120681,
+            0.03155321708116329,
+            -0.018866109579446184,
+            0.015376812010630058,
+            -0.01955557376085726,
+            0.016255217383093355,
+            0.017868332102274922,
+            -0.020325179506828335,
+            0.013434364256590097,
+            0.004176904701252156,
+            -0.03877060944639854,
+            0.0006475542749369515,
+            0.00042468300294954125,
+            -0.02116992978038693,
+            0.0032654387271108293,
+            -0.00575003786159731,
+            -0.05149954071159492,
+            -0.031382926943128234,
+            -0.015730563640253065,
+            0.0382312502769352,
+            0.004297939848910064,
+            0.019724983545557365,
+            -0.017907869879607064,
+            -0.005165628493894463,
+            0.03698976707579373,
+            0.008501001373266799,
+            -0.01725527821298066,
+            0.006879620077191753,
+            0.02455792200929138,
+            -0.03224193972301503,
+            -0.039181328884413266,
+            -0.0008206986106407542,
+            -0.023306758893533952,
+            0.01672946960876128,
+            -0.02658587698869307,
+            0.005873620623736886,
+            0.0025331170979801232,
+            -0.03490809296591325,
+            0.02456764314036355,
+            0.003291113462353163,
+            -0.004303423672882112,
+            -0.020359544091059775,
+            -0.016892415165120203,
+            0.017174797487220006,
+            0.04391489511478065,
+            0.007327321684948843,
+            -0.003943713832544148,
+            0.00862684231313155,
+            -0.006361172785753631,
+            0.05826688726534012,
+            -0.02284956969636637,
+            -0.026680023999604643,
+            0.027527436347931256,
+            -0.022597669279069723,
+            -0.006277863878949772,
+            -0.017343446085471857,
+            -0.012065192152203348,
+            -0.0004183712490442143,
+            0.019339311388446943,
+            -0.019423562920897294,
+            -0.010212527447243049,
+            0.004783006556748757,
+            0.012824133870627983,
+            -0.004855945108275706,
+            0.006239949458562236,
+            0.006847581690471813,
+            0.03094005096378827,
+            0.026993131409761086,
+            -0.011331881651539968,
+            0.03297014291876675,
+            0.010265600643742144,
+            4.666113318251278e-05,
+            0.001970879281172325,
+            0.002523664335535017,
+            0.0034107811520368574,
+            -0.03143890864030425,
+            -0.01624584278940617,
+            0.013468729048885906,
+            0.005233309338223403,
+            -0.004550544967856448,
+            -0.023783653541634683,
+            -0.009402494239255264,
+            0.03334923712760375,
+            -0.003260424721959948,
+            0.02748395114281811,
+            -0.0058374897745283065,
+            -0.01985840930273332,
+            0.017595418271610163,
+            0.02881743246631328,
+            -0.05928552440605384,
+            0.017062540894682354,
+            -0.0011508547522925793,
+            0.008953761538519997,
+            0.01610561914042553,
+            -0.008894160513648953,
+            0.013301991014533684,
+            0.04080863883897929,
+            0.010738051773755976,
+            -0.023345002072886145,
+            0.004215726310838674,
+            0.01541749448570608,
+            0.01291813299324752,
+            -0.022747942296528025,
+            0.0008640791861357522,
+            0.011008522066277018,
+            0.005893793007012561,
+            0.0026106450193140156,
+            -0.006278394080583205,
+            -0.025659188287146268,
+            0.024180550333596625,
+            -0.0044491292740074925,
+            0.026967973346698617,
+            0.04390909273429254,
+            0.009596855239692042,
+            -0.03686020608700941,
+            -0.013109951664921729,
+            0.0002799680359670127,
+            -0.0038537577602593366,
+            -0.0037552094562213916,
+            0.011105295862811114,
+            -0.04185615640149602,
+            0.019950890919369135,
+            0.006453241355231516,
+            0.015314645427201533,
+            -0.020031193483734364,
+            -0.024530173488182337,
+            0.0031949828520724337,
+            -0.01492645564652531,
+            0.012875827757040941,
+            0.005057170413681063,
+            -0.004319834710426384,
+            0.005386862713494232,
+            0.00030848054013413157,
+            0.008384941028326983,
+            -0.010811526993301901,
+            0.03175068297908541,
+            0.004680987487229685,
+            0.009264547472281849,
+            -0.02856314241700808,
+            -0.021083721348429348,
+            0.020113396771629096,
+            0.03572366771051171,
+            -0.01708529918264109,
+            0.03346921724975398,
+            -0.009042046660366718,
+            0.0033219155387035227,
+            0.0151720168318272,
+            -0.015805318210244414,
+            -0.008057236707694303,
+            -0.0005483009439015931,
+            -0.002917527004423299,
+            -0.014538511124345527,
+            0.007302979537905532,
+            -0.0208529958115422,
+            -0.02965227242147972,
+            0.020064178921373875,
+            -0.00011892622932349096,
+            0.016455188521316146,
+            0.010335044657747514,
+            -0.007816155992852236,
+            -0.0052262330850905165,
+            0.006959344748883874,
+            0.04432147882994085,
+            0.026734484553994495,
+            0.05547661378166294,
+            -0.005787997952888261,
+            -0.004559444854138438,
+            -0.0082295581123363,
+            -0.006588172755707144,
+            0.04524009609473585,
+            0.015915003261638767,
+            -0.004910285095602864,
+            0.017495607570191165,
+            0.01557886571266685,
+            0.03858259260136037,
+            0.0027648525902558316,
+            -0.016912796336835408,
+            0.04014755669075221,
+            -0.004077405782937414,
+            0.008616827127381418,
+            0.011102692199778469,
+            0.0024185742218798562,
+            0.00807141794538637,
+            0.02996274720815659,
+            -0.012429186019907761,
+            0.01680942269506771,
+            -0.04571916592313758,
+            -0.02775530246832996,
+            0.014370280587419414,
+            -0.004931905540824582,
+            0.03419959665162407,
+            0.01668521021731861,
+            -0.023105305446763272,
+            -0.024761385385514877,
+            0.021529428247380984,
+            -0.002603042335561961,
+            -0.024135326707477325,
+            -0.015906833215844473,
+            0.003286605488908195,
+            -0.0023437997044735617,
+            0.0060251848371355166,
+            -0.013481652076194643,
+            0.0002908426939460218,
+            0.01953362396253743,
+            0.0038544889405965057,
+            0.020760460182873287,
+            -0.016224396121239458,
+            -0.00259233446682179,
+            -0.009282861237653619,
+            -0.02952130765234087,
+            0.017988370145301916,
+            0.006138236653154911,
+            -0.007607182955958238,
+            -0.009516767050028553,
+            0.0048347778733230365,
+            -0.005819209642901871,
+            -0.0025851701689263706,
+            -0.01073014664868326,
+            0.012433465684360061,
+            0.025784014942833097,
+            -0.016875624785552303,
+            0.007944349714123104,
+            8.651589872318479e-05,
+            -0.022306044741692753,
+            -0.022127016745066307,
+            0.020143472307319017,
+            0.002792646249762062,
+            0.008768813346304779,
+            -0.004074214276300631,
+            -0.00015024341785520542,
+            0.002990550758627699,
+            0.01951746553838439,
+            0.03896451381932467,
+            0.004232673242619471,
+            0.027876586543280867,
+            0.0145860303663199,
+            0.03994234200854897,
+            0.009657017763624678,
+            0.00696607158576712,
+            -0.026055387694656488,
+            0.02819890357011778,
+            0.00018297502510753014,
+            0.043190622821775236,
+            0.022488402221902314,
+            -0.0407730101161195,
+            -0.0021946518523759066,
+            0.01669520746276111,
+            0.013212623257794862,
+            0.017009844136059122,
+            -0.004537512994041156,
+            -0.014041213424740049,
+            0.028135404528975894,
+            -0.022859586430516003,
+            0.0008637006862481223,
+            -0.004486332934188017,
+            -0.025031795510001887,
+            0.007351370254175368,
+            -0.004567816916440959,
+            0.030025413528233364,
+            0.0014976740344892735,
+            -0.022285096307825804,
+            0.022837170656498543,
+            -0.009551269792616045,
+            0.011045530488684207,
+            0.008809047185288755,
+            0.020065001354321994,
+            0.031993362733068575,
+            0.009143210434415064,
+            0.02133640814803879,
+            0.012241777474889248,
+            0.014771538127546474,
+            -0.008804192497858033,
+            -0.02341427260922065,
+            0.023167424359504862,
+            -0.0011598649565273595,
+            -0.029627604902846688,
+            0.0021991474725787303,
+            0.003261878079344105,
+            -0.0005935713331682358,
+            -0.009144689470851402,
+            -0.005580557862988033,
+            -0.007240836451875213,
+            0.019631377011298905,
+            -0.007085213886352692,
+            0.007017573363709754,
+            -0.004074171791476129,
+            -0.00822871835665378,
+            -0.0421101573155962,
+            0.004210914075286222,
+            0.003408427700622687,
+            -0.01752179330665445,
+            0.006384440404174512,
+            0.011121481025573962,
+            0.043020575455262985,
+            -0.022809979913698254,
+            -0.0005737138409049003,
+            0.019206623212867476,
+            0.0009562551889820241,
+            -0.023427467810802533,
+            -0.038931688376706186,
+            0.007172345672967499,
+            -0.03467050750542982,
+            0.023185301620612612,
+            -0.0007075596339842658,
+            0.02387977204565927,
+            -0.03043468414449851,
+            0.013909304572370437,
+            0.004784668158421625,
+            0.012524879174804915,
+            0.021654512058639427,
+            0.03142146922870715,
+            -0.02053473475540882,
+            -0.050409862638621725,
+            -0.005406667316025549,
+            0.00035138762584085976,
+            0.022018487016014856,
+            -0.026014580970974616,
+            0.00025633004699302783,
+            -0.003349094027291921,
+            -0.02401366769212778,
+            -0.01735996745614562,
+            0.01962340791446025,
+            -0.050011480783403936,
+            0.015042913576492952,
+            0.0262737424555697,
+            -0.02507473901367252,
+            -0.05408889321034101,
+            -0.018988587082325706,
+            -0.028217893770659686,
+            -0.009285770094319318,
+            -0.00661159230987788,
+            0.028597405154740078,
+            0.003525576256054959,
+            -0.011462219358952466,
+            0.025401668390029348,
+            0.006155837251830549,
+            0.0026050693705924593,
+            0.015985722392086222,
+            0.007289910674212967,
+            -0.011452511453623235,
+            -0.014599577191293152,
+            -0.04558584556100621,
+            0.010197536827229795,
+            -0.0045431962637690155,
+            -0.019372598050508005,
+            -0.030147301333718016,
+            -0.0137828704002156,
+            -0.027060515748528666,
+            -0.0017291966950107505,
+            0.053783684195381654,
+            0.03292494290301388,
+            0.005560172296221419,
+            -0.0026085670565688026,
+            -0.010466890489356092,
+            0.036247483457995895,
+            -0.006789746401727701,
+            0.03662690472838588,
+            0.001784522073210422,
+            -0.0242673253380092,
+            0.01169382884548301,
+            0.02892627181542449,
+            0.009858451662166387,
+            -0.0005592283645355213,
+            -0.006160132931009508,
+            -0.008782830470079547,
+            0.0031207630972208394,
+            0.025613233596604418,
+            0.003268170392795548,
+            -0.010062991425127038,
+            0.018106967407336815,
+            0.004480291699976396,
+            0.02010374270557836,
+            0.03555357587371324,
+            0.023489837757983054,
+            -0.021782173652026497,
+            -0.026581762796336082,
+            -0.0013561486781077014,
+            0.0072266106510934505,
+            0.03551967295942857,
+            -0.01651315514034218,
+            0.0001274396319698452,
+            -0.008272735507120127,
+            -0.00894931703949637,
+            0.011149952898504355,
+            -0.0029671593121986116,
+            -0.0030752010285951576,
+            0.01662711184497578,
+            -0.0011672263615879709,
+            -0.033448222499147764,
+            0.0078288560210653,
+            -0.023496331346917084,
+            0.0051040606404415545,
+            -0.004113104164416575,
+            -0.007076525381595441,
+            -0.002157548492138289,
+            0.006302962169276006,
+            -0.008065841090139872,
+            -0.00035242603988955366,
+            -0.01586406637763306,
+            0.008271894439780969,
+            0.007438312999409211,
+            -0.0069617481460406095,
+            0.011402932583388205,
+            0.002666895909532698,
+            0.0005219881149353855,
+            -0.012159425670629751,
+            -0.012042145306711322,
+            -0.00957276588761916,
+            -0.03401863879157551,
+            -0.02604353712472079,
+            -0.024949533021906844,
+            -0.0014209149651479556,
+            -0.006566685925666972,
+            -0.03187796739029736,
+            -0.01209152137926458,
+            -0.011110763124407752,
+            0.007756961097537267,
+            -0.013260548545917543,
+            0.005433285079114047,
+            -6.399098520041351e-05,
+            -0.0044960488826674555,
+            0.016615636438440516,
+            -0.003643291599919672,
+            -0.019488010211707343,
+            -0.03927818878149417,
+            0.008209336177358258,
+            -0.02805593125673831,
+            -0.0038816771550190788,
+            0.007607119219721711,
+            -0.00528681677614909,
+            -0.029502897838743516,
+            -0.009149794737430706,
+            0.008767001199311985,
+            0.003507217008280066,
+            -0.01773349756494436,
+            0.03579670773605949,
+            0.026971617790626075,
+            0.011875179879209696,
+            -0.019891342676443626,
+            0.0508929813275565,
+            -0.0047010735337056275,
+            -0.012544935684845649,
+            0.018297503869683027,
+            0.02355022268131519,
+            0.03273167043015095,
+            -0.024830243320788693,
+            0.0009370814985214882,
+            -0.021937793653233454,
+            0.00535662615596573,
+            -0.035009915581788176,
+            0.0028595659314725012,
+            -0.014395027018699446,
+            0.015596459769560532,
+            0.03397372881171701,
+            -0.013254226072981457,
+            0.011195747181764677,
+            -0.016299384375589934,
+            0.00730244731485402,
+            0.00569944090461748,
+            -0.012916284280345044,
+            -0.005812311882399539,
+            -0.022902815152116493,
+            0.020140276669979015,
+            0.005809799950176385,
+            -0.0008130655714656685,
+            0.0104085826652653,
+            -0.044723432238900435,
+            0.03286429597337671,
+            0.008730225944501239,
+            0.017599189944921396,
+            0.0029762446259779998,
+            -0.0003068671951232415,
+            0.011611551606224214,
+            0.011228401175698422,
+            -0.016029727902740706,
+            0.004345913009102136,
+            -0.004613061137378433,
+            -0.03850533909474302,
+            0.009363351681220008,
+            -0.03631184057978666,
+            0.014427784677340159,
+            0.016026872002233575,
+            -0.004170582842643461,
+            -0.025166189571409366,
+            -0.005428471012333673,
+            0.010827910811085384,
+            -0.009783069664327864,
+            -0.009987805234276443,
+            -0.04163358183060102,
+            -0.025236440364963034,
+            0.034097508113604576,
+            -0.002119474613596975,
+            -0.025826403511707147,
+            -0.010830738112107807,
+            -0.011351132636491575,
+            0.020207687645133,
+            -0.006542026283946858,
+            -0.0006263975804936843,
+            -0.04150011545311013,
+            -0.0073396962917864515,
+            0.011192225907726928,
+            0.027746213794734322,
+            -0.035970017302005566,
+            0.02708126943501373,
+            0.002105633542220187,
+            0.014262674845755191,
+            -0.01626664983003942,
+            -0.005631805361903626,
+            -0.013990539604372344,
+            -0.035877093361375685,
+            0.016487723013906077,
+            -0.008607758248055054,
+            0.004714436224135091,
+            0.04458009880881818,
+            -0.0036872068506712727,
+            -0.009466125348499464,
+            0.007600705817195143,
+            -0.0034590883553048063,
+            -0.010351576085287273,
+            -0.0028564505793999295,
+            0.001959661298507,
+            -0.007080697863126256,
+            -0.04183476693971616,
+            -0.00499888976530217,
+            0.04684391782633148,
+            0.006361744381879857,
+            0.013276793765092465,
+            0.013516220521120684,
+            -0.018335746557932003,
+            -0.045577810515431215,
+            -0.0071616111758423025,
+            -0.0032763880176386808,
+            -0.022708028392758567,
+            -0.0240284945459861,
+            -0.008735040856657758,
+            0.005844145402902923,
+            0.006359368165686724,
+            -0.01120763599008222,
+            -0.022676131361171057,
+            -0.027085968700452136,
+            -0.010526103176253857,
+            -0.02123944716136792,
+            -0.010792145796733375,
+            -0.027142270190771887,
+            -0.028742234538796804,
+            0.02880426953386151,
+            0.007575182232859564,
+            -0.004084088301176117,
+            -0.010005765476121966,
+            -0.01255640546813973,
+            -0.0004774761617900028,
+            -0.0025307994917461232,
+            -0.005822065709999144,
+            0.0038614286313282164,
+            -0.011390482620811868,
+            0.023152086963976554,
+            0.0021220701260546895,
+            0.018329162172128138,
+            -0.05772324571231113,
+            -0.002709083851090071,
+            0.021497896395795905,
+            0.0033025410488410913,
+            -0.005412445141910314,
+            0.029352650459792132,
+            0.011223947499453582,
+            -0.006539460282439711,
+            -0.031355428613718234,
+            0.020942976176820262,
+            0.024122993515785947,
+            0.00817615634503138,
+            -0.012994160510030828,
+            0.01221095914384204,
+            0.01629778384790159,
+            -0.004494985018246821,
+            -0.011161098655072933,
+            0.005586857967039181,
+            0.020941039642372112,
+            0.013580622629911925,
+            0.0013398071867077221,
+            0.030905022925366163,
+            0.0006862084391508121,
+            -0.03521231612862609,
+            -0.01908191995763771,
+            -0.01275995164030872,
+            -0.003584331305986364,
+            -0.015264425022763126,
+            0.005294329654809353,
+            -0.002355465010876059,
+            0.006335052149475905,
+            0.022655379849785256,
+            0.03593231939948758,
+            0.0146937743203089,
+            0.00937330328448886,
+            0.003167462724350819,
+            -0.0013235176101076284,
+            0.012102872918129403,
+            0.008638011773075119,
+            0.0350701240268404,
+            -0.014653221697693864,
+            -0.011003994230902835,
+            -0.0037767288519369537,
+            0.010100131061186566,
+            -0.004113527518662366,
+            0.020414393256136372,
+            0.03274709615649469,
+            -0.0037222949335916876,
+            -0.005604552180246868,
+            -0.006160118231975786,
+            0.0041467454414452565,
+            0.01807879127455274,
+            -0.01252134965321004,
+            0.004364858987501461,
+            0.005549300665159658,
+            -0.0071897162833861,
+            0.018567059255813558,
+            0.0002622074720860951,
+            -0.012983550838417323,
+            0.03266376564402191,
+            0.11449509387240629,
+            0.0017251916835047628,
+            0.007125215166916139,
+            -0.040229997815548584,
+            -0.020537987527148888,
+            -0.00245728735683473,
+            0.005779972828836593,
+            0.007004732428236457,
+            0.02225914982152608,
+            0.006505827715591914,
+            0.0006708045351142873,
+            0.031708522931495536,
+            -0.017334083511709092,
+            -0.002522861743921599,
+            -0.01745656559688129,
+            0.013890451996979048,
+            -0.014312605700272691,
+            -0.013784963572164409,
+            -0.01572417853955075,
+            0.004507425871748977,
+            -0.000601158156253668,
+            0.0016158265715634729,
+            0.01791303234734911,
+            0.0011926917387973374,
+            -0.026393351252156246,
+            0.0006694008039871878,
+            0.001026031515214626,
+            -0.004514419527575117,
+            -0.011320929719992686,
+            -0.02223001723942815,
+            0.016235841442653918,
+            0.014188311619681008,
+            -0.004895218832118094,
+            0.009244697376301004,
+            0.010558796153959677,
+            0.02707485418319703,
+            -0.028380342401606758,
+            0.0040204430418629405,
+            0.007023041225985294,
+            0.01606556581447933,
+            0.017617862031190843,
+            0.0034870847111552862,
+            -0.011253315705053224,
+            0.03238289399766116,
+            -0.010563282687352038,
+            0.0065093536270233385,
+            0.014966802208644462,
+            -0.03541611540945151,
+            -5.821682910348165e-05,
+            0.0061985444938402745,
+            -0.011716201661363997,
+            0.009580035961315467,
+            -0.03120692534097433,
+            -0.0039291243987978845,
+            0.009317468331974374,
+            0.017611312558635673,
+            0.008804996982574765,
+            -0.0034129169537151925,
+            0.010822226698822738,
+            0.012510661458054741,
+            -0.0027147975086409966,
+            -0.008855213644027594,
+            0.01238502686635956,
+            -0.0007687053446954701,
+            0.00450108863711973,
+            -0.00799170686059026,
+            0.0029755595812552293,
+            0.002290650917244691,
+            -0.01554856300008412,
+            -0.011779430686394945,
+            0.00438393126679289,
+            0.0036931891075561957,
+            0.001535662168147833,
+            0.012014408636737167,
+            -0.006087255045485002,
+            -0.0077816220325470226,
+            0.0040573724156794284,
+            0.003136943055301706,
+            -5.505014316363865e-05,
+            -0.030625245024561508,
+            0.01585923084309149,
+            0.0067595435845396405,
+            0.010013308598626031,
+            -0.005315626260228517,
+            -0.00903142414091241,
+            -0.007240920015018893,
+            -0.01228521172101564,
+            -0.010657894223173453,
+            0.007658605592936815,
+            -0.00724924993432376,
+            0.011124174211132038,
+            -0.03128027787357322,
+            -0.013988715435985316,
+            0.01231792993245938,
+            -0.01528161242159377,
+            0.0007675821323979293,
+            -0.008389914312201402,
+            0.0009417242487395995,
+            -0.005119304379761705,
+            0.007887436211143328,
+            -0.03731503169547854,
+            0.007878001990271541,
+            0.02456170736765724,
+            -0.00330924625934981,
+            -0.020259771366167557,
+            -0.009613930457931786,
+            0.0241314385136084,
+            -0.008267831641886344,
+            -0.01645888287387047,
+            0.008421304862524506,
+            0.010140813149093993,
+            -0.0006309118552875393,
+            0.0008237784055549769,
+            -0.003726984191590917,
+            0.01669125388798323,
+            0.02054777616420997,
+            0.041930770964107515,
+            -0.014789610432004559,
+            0.01092349703703152,
+            0.02866938974792608,
+            -0.03378576980722847,
+            0.014728113846952417,
+            0.02518409645155824,
+            0.014590531327650454,
+            -0.0004446956315190408,
+            0.001609777186372437,
+            0.0068939321280085475,
+            -0.014273492115311677,
+            0.009247676157551494,
+            0.013380104440744367,
+            -0.018407746260686045,
+            0.020444430509441124,
+            0.03140071554008003,
+            0.0025839998101985736,
+            0.01355205268903058,
+            -0.006268939830271642,
+            -0.013860692186849113,
+            0.014717364068116991,
+            0.031429975488060896,
+            0.0006919565473572909,
+            -0.018561500446336806,
+            0.019128244642842268,
+            0.006988095806264863,
+            -0.003678529269242034,
+            -0.0021993078588142664,
+            -0.021334919445200388,
+            -0.021003586791369228,
+            0.017304350704792804,
+            0.013508779408333964,
+            0.027794485858413365,
+            -0.010594702782952029,
+            -0.007818089330662161,
+            0.014293688127901867,
+            -0.022368568887004094,
+            0.015426482139881204,
+            -0.02160820059598751,
+            -0.00571499813703566,
+            -0.0077402210272634235,
+            0.03274360648629609,
+            0.001441304806200403,
+            0.02734331677589297,
+            0.016457027764064002,
+            -0.033609420134045206,
+            -0.029790535553657092,
+            -0.00712106584238484,
+            -0.02307933297713104,
+            0.010356561171963577,
+            -0.03901183838032092,
+            -0.00071735150189382,
+            0.008251646656432826,
+            -0.008079917288471352,
+            -0.0007087526632966007,
+            -0.006419191399519291,
+            0.013989344185652506,
+            0.013661633160336152,
+            -0.0228103143826663,
+            0.023644639659577426,
+            0.008343481940133684,
+            0.025998483906975782,
+            -0.02076055376006461,
+            0.019148568238437927,
+            -0.017705363646416224,
+            -0.02777313933338929,
+            0.014358130220942483,
+            0.023585731219788467,
+            0.01635364038245971,
+            -0.007122853965024528,
+            0.007110003583832299,
+            -0.03556761320536796,
+            0.027585457849964405,
+            -0.013064062623866305,
+            -0.012504161289952458,
+            -0.0032379276562094555,
+            -0.011931827532360686,
+            -0.002574161005800279,
+            0.0008533907465694845,
+            0.0034345232246102806,
+            -0.03131462673429805,
+            -0.0002450959360520896,
+            -0.03727416416766574,
+            -0.028944814604815072,
+            0.01360126708805471,
+            -0.00036361284498363273,
+            -0.0049097813612814616,
+            0.02933242705521636,
+            -0.026339136251046315,
+            -0.02693295650849956,
+            -0.009590247544093352,
+            0.02853096456887074,
+            -0.0220673798697511,
+            -0.004519050393279848,
+            0.0015034014871316069,
+            -0.01828226786700525,
+            0.004436383319946364,
+            -0.0012748481042235446,
+            -0.010416065158876917,
+            -0.01105146139270597,
+            -0.03229479099927737,
+            -0.005822133182096844,
+            0.02106215163620738,
+            -0.006957125617688484,
+            0.002746241228040847,
+            0.013312248331456934,
+            -0.010968629118224582,
+            -0.0029946900704643812,
+            0.0017875284441794235,
+            0.023304175718480834,
+            -0.021033941064173063,
+            0.007714334541806199,
+            0.005955442031111047,
+            0.0016405586860131207,
+            0.018406562522853148,
+            -0.0054922197063461825,
+            0.0037690571959124617,
+            0.0162126534419016,
+            -0.009969304596173302,
+            0.017959307638726878,
+            0.007691549802475525,
+            0.006681837561745192,
+            0.002775700618222622,
+            0.007712770283579111,
+            0.009752652973700739,
+            -0.013506020126078112,
+            0.016911141425375933,
+            0.022468715726349706,
+            0.00023233555209372766,
+            0.021066580799163234,
+            -0.02302516292580905,
+            -0.028618544628453015,
+            0.018297878810036087,
+            -0.0225078941299981,
+            -0.012735208680311652,
+            0.016459925496937065,
+            -0.0043032724578332855,
+            -0.00797609922711709,
+            -0.014986057073351055,
+            0.02260618561608301,
+            0.02132930921464508,
+            -0.020249788476562527,
+            -0.012385936327068884,
+            0.0072304431443959505,
+            -0.004373475447051833,
+            0.007086656081865733,
+            0.03195810137269375,
+            -0.015982210897392764,
+            0.006818890093901413,
+            -0.0038306138017521626,
+            0.0030753642449792215,
+            0.017830546304667783,
+            0.009470018737705222,
+            -0.0019815241970236146,
+            0.030100238416495066,
+            -0.001330614013444443,
+            0.016392226282459177,
+            0.0004888028477871524,
+            -0.010948833692356176,
+            0.006343890495626752,
+            -0.026652806306173643,
+            -0.008627570055007363,
+            -0.001203530223345183,
+            -0.009581494451831351,
+            0.07598616930784373,
+            0.003930153552069478,
+            -0.01295957554290434,
+            -0.001673007115291578,
+            -0.001368287102181597,
+            -0.006897179139643672,
+            -0.00960679451603541,
+            0.0287166325268306,
+            -0.016613613331623346,
+            -0.01894527592131826,
+            0.0008370014132002703,
+            -0.006785618888421608,
+            0.00919924251249592,
+            0.01816363936999796,
+            -0.010225961928154075,
+            -0.01604759761423727,
+            0.001577138242530474,
+            -0.003520564936698765,
+            -0.004395434431595494,
+            0.0052399043934902014,
+            -0.010127122101423486,
+            0.006456662777583096,
+            0.026560491731327886,
+            -0.013684536319659296,
+            -0.01425196172663986,
+            0.004084982328707292,
+            -0.0010682084059972982,
+            0.009758641913073978,
+            -0.00864879062601467,
+            -0.00481222050200678,
+            -0.014375330954464725,
+            -0.011378890549403524,
+            0.02505641780042516,
+            -0.02861164504722546,
+            0.0034060847953066354,
+            0.014937227531980433,
+            -0.00016904224334342915,
+            0.01541381851648916,
+            0.0033028855218525644,
+            -0.03615935765718852,
+            -0.013677774380513626,
+            -0.02073839691250734,
+            0.019821074228011413,
+            0.01688422012576042,
+            -0.0033731840788147442,
+            -0.013840993586441808,
+            -0.026149880178874887,
+            -0.007430563876740068,
+            -0.02466234818567362,
+            -0.020708282040996014,
+            -0.019276991448222817,
+            0.006137002799644838,
+            0.005162892426268572,
+            0.022881759407670583,
+            0.0025854880108157857,
+            -0.015058889448404231,
+            0.0029948608544278386,
+            -0.006722836436935071,
+            0.011845298212977357,
+            -0.02747944486818378,
+            0.015311272959213262,
+            -0.01610493464983465,
+            0.025239768880017402,
+            0.007425208902253236,
+            0.0010569008485112994,
+            -0.008822681490988575,
+            0.00010704167529046774,
+            0.014883787023445807,
+            -0.004186968199973372,
+            -0.0007781259002896265,
+            0.02787555889813697,
+            0.02087373813833492,
+            -0.0017783425135865238,
+            0.008743718448136432,
+            0.028026471206552342,
+            0.019081083283793984,
+            0.0325261942213039,
+            -0.013815526479760766,
+            0.005113676604826083,
+            -0.028873600114081546,
+            0.010510475843707614,
+            -0.010871039729515013,
+            -0.010201824683340651,
+            -0.0067946599549165735,
+            0.027756887504389347,
+            -0.007962601205446472,
+            -0.0006687976124826922,
+            0.03469894503563926,
+            -0.014185294847328063,
+            -0.00010543064621374546,
+            0.0007386418834840544,
+            -0.009759481309152999,
+            0.02742722730386853,
+            -0.0073590367291510046,
+            0.018456032721630843,
+            -0.019560882612141943,
+            -0.013705917411254969,
+            -0.011894560208199602,
+            -0.001142768182039817,
+            -0.02576754877141608,
+            -0.008755240066826174,
+            -0.02249287216087102,
+            0.00859172262856532,
+            -0.0046012388553094375,
+            0.018845920253446127,
+            0.0011617844393195566,
+            -0.005275799428018491,
+            0.029069367036565145,
+            0.0088080893110627,
+            0.018100411389202663,
+            -0.0004971948011851863,
+            0.01216566778850985,
+            0.02369448391321832,
+            0.002637001729688794,
+            0.005012948614448268,
+            -0.024213704079796478,
+            0.003420184574622472,
+            -0.0004359900462023086,
+            8.265488264269535e-05,
+            0.016267948861008616,
+            -0.019572032453100303,
+            -0.02216264534756073,
+            0.021871880626992544,
+            -0.012742926070754964,
+            0.011676769226089007,
+            0.0035362870996336194,
+            -0.010732229460038356,
+            0.0006838777488770268,
+            -0.03083438636386314,
+            0.008820746379709355,
+            -0.03216685858285627,
+            -0.032230808828686675,
+            -0.013547681710236494,
+            -0.009994106571421729,
+            0.0008758088338659712,
+            0.028447187234357855,
+            -0.00795317029726568,
+            -0.015797892954136448,
+            0.005363565557248085,
+            -0.011199252697624111,
+            -0.00817162221789202,
+            -0.0007867892785855576,
+            0.005856939999700809,
+            0.009461959355310273,
+            0.005997804620690314,
+            0.011648335019189696,
+            0.007491946658046298,
+            -0.008028800679169293,
+            -0.008837725220736191,
+            0.015889923814605387,
+            -0.0014409716087425222,
+            -0.01176424391171685,
+            0.030430130082762363,
+            -0.013300620185884817,
+            -0.0005592926632214984,
+            -0.0021975669766032472,
+            -0.0015687208316663597,
+            -0.0021652076181411244,
+            -0.014225289143358926,
+            0.01870568404189425,
+            8.334501002782455e-05,
+            -0.014288806218067678,
+            0.0032841743905975634,
+            0.003841780428153241,
+            -0.0036498339483411074,
+            0.008388922960269383,
+            -0.008147275785908849,
+            0.017348147498892,
+            -0.015113395971380741,
+            0.008740452410210708,
+            0.0038335161236289234,
+            0.0043482078469644385,
+            -0.0065755348640655865,
+            -0.01592809184384243,
+            0.009642593996701513,
+            0.012051872087663658,
+            0.02173264012652684,
+            -0.02525660485859878,
+            0.00876161653901566,
+            -0.005374533920781607,
+            -0.014534179652013885,
+            -0.016127866637369738,
+            0.00982303248417706,
+            0.022322910435388785,
+            0.0063086346898091296,
+            0.012139263756190026,
+            0.00013383660542132047,
+            0.005848259030010402,
+            -0.014767198851314148,
+            0.010570158368602513,
+            -0.011355774405066537,
+            -0.018439261453084426,
+            -0.005243587266408389,
+            0.028894120979627323,
+            -0.005492024421080143,
+            0.017938747701972086,
+            0.0021350711250093858,
+            0.029007356490801176,
+            -0.008686781541806358,
+            0.0014883465513564527,
+            0.0039866022469891155,
+            -0.01295271525889559,
+            -0.004272008629260297,
+            0.01976273843646234,
+            0.04882868638601926,
+            0.010204475902027572,
+            -0.0009155109545661691,
+            0.010069071220988797,
+            0.005626418787776309,
+            0.002148101019776153,
+            -0.00805781810841286,
+            -0.030431071067211577,
+            0.008883707378543252,
+            0.005628218215212441,
+            -0.01997825125961566,
+            0.019838234890736068,
+            0.026483365072033395,
+            0.023040623182975964,
+            -0.011585643598171132,
+            0.016539930468533123,
+            -0.010092529089936188,
+            0.0177008735686489,
+            0.028665424114962996,
+            0.013810061902648515,
+            -0.010487150104822674,
+            -0.0020847870330548534,
+            0.006329342599564054,
+            0.004186981876004016,
+            0.00026239408223817384,
+            0.015190614519766204,
+            -0.019155180136775043,
+            0.01005728016778505,
+            0.014725826697964637,
+            0.03322436221319799,
+            8.049930133627564e-05,
+            -0.01645191835557271,
+            0.0016249281996128196,
+            -0.009976155662939693,
+            0.00856399492362911,
+            0.000632427127231252,
+            0.0032242260692672124,
+            -0.009688119824276678,
+            -0.012682779503915694,
+            -0.002851918183031152,
+            0.009403205508474241,
+            0.0021188716361101698,
+            0.002759004011231831,
+            0.007641925150177999,
+            0.0022830062031146204,
+            0.01713024474384216,
+            -0.0011181980841546123,
+            -0.00029141889484139217,
+            0.004601090256674874,
+            0.010509199590505558,
+            -0.01649852835819883,
+            -0.017548369964263514,
+            0.004337237475053179,
+            0.012217494210546133,
+            -0.013159179948224596,
+            -0.013241981632059755,
+            0.015704543608959117,
+            -0.01913575130892795,
+            0.024934102868065366,
+            -0.00942381892138331,
+            -0.004543611336526587,
+            0.000343200128934031,
+            -0.00727538880611703,
+            -0.0038751118497522673,
+            -0.0030064279504724797,
+            0.003984715814505561,
+            0.0019261818110367147,
+            0.007472350699291607,
+            0.008472369788801792,
+            0.014568547873264247,
+            0.006043895352236181,
+            0.0032580830408778395,
+            0.009109104498825864,
+            -0.014523846294121576,
+            -0.014800649887278973,
+            -0.002704791598011488,
+            -0.008895394038700809,
+            0.00902016867266151,
+            -0.006607182154874112,
+            0.00845408382182461,
+            -0.015501867835442603,
+            0.015573642225745032,
+            -0.012200552219016517,
+            -0.0008900215712478553,
+            -0.009666233545144905,
+            -2.7470277092642562e-05,
+            0.008191697904514857,
+            0.014605287653054787,
+            0.0070552942843382215,
+            -0.031921405225964206,
+            0.0017403427534633655,
+            -0.008846568352320488,
+            0.03446319360336241,
+            0.002612606662691094,
+            -0.01060863123202008,
+            0.020352181356568008,
+            -0.01592124132155158,
+            -0.00027704668715660693,
+            -0.01602075523652062,
+            -0.004797557413047212,
+            0.00707939446796679,
+            0.016399017978990946,
+            0.02759049512664933,
+            -0.005697020356865688,
+            -0.007995305872076221,
+            0.006384047872862793,
+            0.029590696670334232,
+            0.012246955508890425,
+            0.024748351124415785,
+            -0.007633712004672087,
+            0.014348228716100414,
+            0.007773260552660029,
+            0.0076045488653348975,
+            0.004321689250440021,
+            -0.002591801797145718,
+            -0.011698574606499644,
+            -0.014974971814360001,
+            -0.0037563183330623384,
+            -0.016987915346053498,
+            0.00012247833530362867,
+            0.021344400276501117,
+            -0.00475351423959752,
+            -0.0033847161875730026,
+            -0.01874186437694691,
+            0.0033461670218611346,
+            -0.010029445302549983,
+            -0.00367421246775698,
+            -0.013133469163793961,
+            -0.0254122035745965,
+            -0.018556369080030412,
+            -0.002155244024999317,
+            -0.00929309186646207,
+            -0.015194269781263309,
+            0.0070354690219415535,
+            -0.016305117309271634,
+            0.0080837166473439,
+            -0.00448473417024141,
+            -0.016508775226236883,
+            -0.00958925666995396,
+            0.01100335153658195,
+            -0.0041819993634961685,
+            -0.01929493648733792,
+            -0.01282300765645703,
+            -0.011300260564838231,
+            0.0051733704121938884,
+            0.005996780699975213,
+            -0.0038111648304178544,
+            -0.0028279639348479987,
+            0.016640121845366295,
+            -0.010197365811240557,
+            0.009878730312873492,
+            -0.03335474019383245,
+            -0.03551826040406959,
+            0.004497847402931096,
+            0.01507007013703286,
+            0.0013085881889541252,
+            -0.02078680101065954,
+            0.010806315988966003,
+            -0.016499186313125556,
+            -0.00858013792516617,
+            -0.008522534127498645,
+            0.009649521088411411,
+            -0.007283003380127847,
+            0.01299542385123007,
+            -5.3226845309105124e-05,
+            0.030477683597373687,
+            0.0074153455615142755,
+            0.004161881195438522,
+            -0.00679081048132963,
+            0.008354912600159931,
+            -0.001621776416108105,
+            0.004080191054849395,
+            0.02120524874226696,
+            0.012440848502230418,
+            -0.00218041171518331,
+            0.006458415098595872,
+            0.005678055538564355,
+            -0.017186537377121235,
+            0.02507960402044456,
+            0.024678178106806115,
+            -0.008205288425409454,
+            0.004108402466376321,
+            0.005783335302169668,
+            -0.008846663924040241,
+            -0.006897707231869481,
+            0.006527795262026734,
+            -0.018474181903007085,
+            -0.0008721381678008483,
+            0.010598084358464908,
+            -0.02255441909004843,
+            -0.004840511019512733,
+            0.021745651362109403,
+            0.004631567631335321,
+            -0.018558972627958834,
+            0.02764788406651861,
+            -0.004942361238163609,
+            -0.0027290999934532494,
+            -0.010876135202650048,
+            0.01671497143541799,
+            -0.012173527207375091,
+            0.008443317346655462,
+            0.011021541861818043,
+            -0.006295639322287843,
+            0.003083685676540999,
+            0.008277206401503182,
+            -0.004504302913715969,
+            -0.0068075424927125676,
+            -0.0010508416962215848,
+            0.010167160518175114,
+            0.0036980815941666336,
+            -0.022704963865525812,
+            -0.006020073261002279,
+            0.02934227992246887,
+            -0.015082922782242507,
+            -0.0050916398557337705,
+            0.005376294716056329,
+            0.007709342013963906,
+            -0.003476236561444464,
+            -0.0023399991233097337,
+            -0.00787486645740744,
+            0.010356414979428607,
+            -0.015950083845400274,
+            0.02076360898078935,
+            0.025608263494550438,
+            0.024504236990840842,
+            -0.014814235990238182,
+            0.010828584133720797,
+            -0.011703992394890898,
+            -0.011047470905668532,
+            -0.015542835222173864,
+            -0.001658296303013357,
+            0.001970208852236451,
+            0.013107754130616248,
+            0.009666887546609378,
+            0.0235853683657408,
+            0.006473544089109194,
+            0.008973300312205858,
+            0.014118601565410305,
+            -0.003902202854013708,
+            -0.018303438244719996,
+            -0.014584226585460755,
+            -0.011977613285028946,
+            -0.01756345491020528,
+            0.0004563602312336091,
+            0.020461659996271687,
+            0.012417744207683839,
+            0.024136626592375227,
+            0.002119213701333967,
+            0.013759902247467922,
+            -2.9525392347047554e-05,
+            -0.000892319792935443,
+            0.00045783825646242434,
+            -0.005238483120312843,
+            0.019255756091120846,
+            -0.004599752813817219,
+            0.01656589522110913,
+            0.0006443773300684925,
+            0.007133940816771615,
+            0.013554761340256566,
+            -0.01704388669741371,
+            -0.003443100892108368,
+            -0.010029620309829285,
+            0.000941391966877521,
+            -0.019633111750884034,
+            0.01934571074127927,
+            0.010273330977233852,
+            0.0017121345993533963,
+            -0.018327189396014198,
+            -0.0029706385099949693,
+            0.03448635131998772,
+            0.019458999453011265,
+            0.0050734354363614525,
+            0.02632443373898792,
+            0.013018762352749236,
+            0.014443611411502168,
+            0.010153370950346075,
+            0.02356766253713842,
+            -0.023479811300626968,
+            0.02095255383200973,
+            -0.011590347850340786,
+            -0.007414136601134069,
+            -0.0035774888741646705,
+            0.0025141508183687115,
+            -0.013588691733097813,
+            0.00982877622088977,
+            -0.0017916539399655826,
+            -0.025386211797192578,
+            0.009164316443206016,
+            -0.008811052979979271,
+            0.010464442616928296,
+            -0.007774569252594923,
+            -0.0002806136360286729,
+            -0.0127084026190163,
+            -0.015184317235990752,
+            0.001259201880504347,
+            -0.009158429132281081,
+            0.023908150915412617,
+            -0.021210848953875233,
+            -0.01321384014436847,
+            0.013997502713857512,
+            -0.006869443694669701,
+            -0.011364239609894453,
+            -0.00374449980963666,
+            0.023673050288879224,
+            -0.009243341446512421,
+            -0.01359367519009039,
+            0.014464166953530989,
+            0.003834735113400469,
+            0.0059713317014342395,
+            -0.007508862694272452,
+            -0.008428705474135382,
+            -0.003280788036976103,
+            0.005998840043285335,
+            0.026615077617284676,
+            -0.014059930972065399,
+            -0.0032833264512892022,
+            -0.009189999799498163,
+            0.004118204537632102,
+            -0.02359889635281631,
+            0.02771075351384059,
+            -0.008871395303200375,
+            0.021705177758560575,
+            -0.0015924449308442096,
+            -0.006239188158271628,
+            -0.024231036074349822,
+            0.027522945599928003,
+            0.003683870966132199,
+            -0.010221180899173243,
+            0.0012768725460662862,
+            -0.052005292736885915,
+            0.016636109796947025,
+            0.008429895913163985,
+            -0.019507994429834353,
+            0.003697519981610208,
+            -0.004773584856005853,
+            0.018983847580558853,
+            -0.022966993228850697,
+            -0.014078986007109447,
+            0.008283766047449966,
+            -0.01153285124362746,
+            0.0031439180465649996,
+            -0.02527749501821793,
+            -0.005754537200570427,
+            0.0011760345167229269,
+            -0.005276936074280004,
+            -0.001705933750221904,
+            -0.020022761608132548,
+            -0.025972227850795115,
+            0.017666017379157135,
+            0.015820875579600418,
+            -0.002278262448951789,
+            0.01320818011561616,
+            0.008709899932124187,
+            0.00836573261091119,
+            0.016034263394940008,
+            0.007124509234103267,
+            0.0081473711608784,
+            0.045498533239850446,
+            -0.019846323044145485,
+            0.007371629414298193,
+            -0.0014854149174929534,
+            0.013951741898629716,
+            -0.030467924817555014,
+            -0.0055144480998344705,
+            0.0062802744565489945,
+            -0.0049373689340857915,
+            0.03510454786147091,
+            -0.017326094216003073,
+            0.01694160526735264,
+            -0.0026075809870445238,
+            0.0007420394265014909,
+            -0.02042003877666893,
+            -0.026156937726134134,
+            -0.014294938857971834,
+            0.0011857406889443987,
+            -0.0010047941261608975,
+            -0.011946583993069242,
+            0.006797001710574735,
+            0.004917161312925866,
+            -0.0020713379451245515,
+            0.015652523103715316,
+            -0.02952061566240878,
+            0.007356800041369564,
+            -0.010733578620243498,
+            0.020008504830359524,
+            -0.00990331048284697,
+            -0.0013319400098308093,
+            0.007236960885181047,
+            -0.004122868383988748,
+            0.013216043472071927,
+            -0.0038337251451538677,
+            0.004227115653354611,
+            -0.005221321656766545,
+            -0.01794343569722877,
+            -0.0016435261495179767,
+            0.021429008432348098,
+            -0.008349684096540855,
+            0.031864497493526205,
+            -0.01264743878037828,
+            0.009174982784686633,
+            0.010692990337462762,
+            0.0006628831036334152,
+            -0.005286285851702483,
+            0.009052971814211658,
+            0.0021431970274545893,
+            -0.008690520448278274,
+            0.006934310986093342,
+            -0.018349449937310143,
+            0.008494451785507706,
+            0.03394632739506833,
+            0.0018511328780472266,
+            -0.0007519661670070526,
+            -0.007266663179613392,
+            -0.01335844201823745,
+            0.018540713029383066,
+            0.005488273906733518,
+            -0.002546531152114063,
+            -0.027536660869590543,
+            0.02007746621157705,
+            0.0012265798861291223,
+            -0.004755325069578531,
+            0.012466557901411086,
+            0.0020495867632654885,
+            0.019989531151046736,
+            -0.010171678639113316,
+            0.030757957899447096,
+            0.010050036220124393,
+            0.002959833051227505,
+            0.0038556928367700113,
+            -0.036655749074851536,
+            -0.01652152778069314,
+            -0.00019357502472740266,
+            0.014271581054865031,
+            -0.012430647252826173,
+            -0.022432580814844155,
+            0.0010346169306977334,
+            0.01148004051449691,
+            -0.0033898642762881046,
+            -0.009650665305271706,
+            0.01560757541276278,
+            -0.020342213902670575,
+            0.006701302394408393,
+            0.007693672045981921,
+            0.0287100214318104,
+            -0.0020735384554022153,
+            -0.0012324838849990056,
+            0.000626416803904792,
+            -0.0055669860801192935,
+            -0.0008037366126870824,
+            0.017251369734470945,
+            -0.013661316947002466,
+            0.0175926211217534,
+            -0.012735241490550407,
+            0.0064713656749838585,
+            -0.0021092378432037563,
+            -0.010629579860050812,
+            -0.020114807971202787,
+            0.0029609495764307357,
+            0.009378003163375899,
+            0.006920144698428806,
+            -0.010438984881449953,
+            0.010715409812532968,
+            0.012197673689706352,
+            -0.00753962861484439,
+            0.014519905132928041,
+            0.022173989895384192,
+            -0.007687503607783572,
+            0.013633321414009471,
+            -0.0041446334728745095,
+            -0.014056264816568229,
+            -0.009985463824440833,
+            0.023929750617160402,
+            -0.0005563806942667597,
+            0.008756997373618809,
+            0.00388125051641897,
+            0.0007856447663040696,
+            -0.011226530517316183,
+            -0.001653447552601681,
+            0.019294448360333633,
+            0.011831934675073843,
+            -0.0013888534822008192,
+            -0.0010525764356699885,
+            -0.0014566936525653407,
+            -0.009427557739483988,
+            0.02049188741459728,
+            -0.022030783927622997,
+            -0.0019604940727076573,
+            0.00461018148439991,
+            -0.022589816803875618,
+            0.0001863117949313752,
+            -0.024570217962989905,
+            -0.01569952241187498,
+            0.011763033781122343,
+            -0.0022530631823737554,
+            -0.016028810867854907,
+            -0.0003242701040942633,
+            7.424127301806556e-05,
+            -0.007815789940508527,
+            -0.004135500768181258,
+            0.006115909992703774,
+            -0.00034193217510010117,
+            -0.0032262189374874842,
+            0.013430154310506884,
+            -0.009996622196870008,
+            0.015431528911575157,
+            0.0067029980830614145,
+            -0.015702167721800483,
+            0.0021406610839557003,
+            0.018500219544264753,
+            -0.0058910087864916085,
+            0.00787021425357441,
+            -0.014681907249972176,
+            -0.0037495551894426303,
+            -0.005787661351944086,
+            0.006051351977821761,
+            0.024849195780885833,
+            0.012969434450643372,
+            0.01139340445077682,
+            0.017405361843075933,
+            0.004325000715054734,
+            -0.01572297405297993,
+            0.0007416798945675923,
+            0.01682727189828363,
+            0.010464036837245312,
+            0.009804326468113703,
+            0.0008975556667684365,
+            0.0043704860361487635,
+            -0.005142176468800175,
+            0.009317064072855551,
+            0.02915482449687158,
+            0.0013434689612997423,
+            0.008696141010760192,
+            0.017276730345147134,
+            0.027038741396131265,
+            -0.0035088997194497725,
+            -0.007469614319788499,
+            0.008739942310300076,
+            -0.009657991604173173,
+            -0.001081629174976152,
+            -0.017000351183542026,
+            -0.012310275736750868,
+            0.016510505259358398,
+            -0.006084627062230492,
+            0.02848913486719956,
+            -0.0010655806791907225,
+            -0.02026498848361482,
+            -0.01565505608194376,
+            0.001404549479440637,
+            -0.01362132895080652,
+            0.005287566256051299,
+            0.014187265685427415,
+            0.01917046513020831,
+            0.004687191682875024,
+            -0.022258337721085656,
+            0.037364962197399806,
+            0.01757108963349427,
+            0.019377870795760123,
+            -0.019422840506328382,
+            -0.006159311237412833,
+            -0.009077095160676276,
+            0.02116697012944498,
+            -0.00880692095488465,
+            -0.009875674856043954,
+            0.024014332004451508,
+            0.017812606845656484,
+            0.005679806286464216,
+            0.006736609904079586,
+            0.0020048663439319585,
+            0.0025414630253661444,
+            -0.015831593851012047,
+            0.017957902498542212,
+            -0.005858125698347523,
+            -0.0058068554681761945,
+            0.017529514573571298,
+            -0.005989581792993925,
+            -0.0030306021203732347,
+            0.00015839997806822045,
+            -0.019697052145531482,
+            0.011618112886207201,
+            0.008256007601576028,
+            -0.017229774429089857,
+            -0.0014757560402567304,
+            -0.0002404480326945295,
+            -0.014118694740134205,
+            0.02004885146995733,
+            0.01694893356262835,
+            -0.0013248884139405398,
+            -0.0051752815372013315,
+            0.008223868147393114,
+            0.007000679132597393,
+            0.005256194890343006,
+            -0.019496098876977206,
+            0.005230597149342019,
+            0.005103909294824105,
+            -0.0014716649591412774,
+            0.010274346323051634,
+            -0.009030594825985526,
+            0.033740380586893826,
+            -0.006505788197604522,
+            0.013065658941136721,
+            -0.0043634731684193745,
+            -0.0055529072007985125,
+            0.001816020676232658,
+            0.013054468478598695,
+            0.009753501030700457,
+            -0.00970446155762939,
+            -0.004172354962129865,
+            0.018697982092775737,
+            -0.011983156146516074,
+            0.0013300791598046338,
+            0.01564072865804035,
+            -0.004858814041438713,
+            0.004708235083775013,
+            0.008635204790345423,
+            -0.004489505237783172,
+            0.01801512858970922,
+            -0.016482572999490366,
+            0.0023953436660054026,
+            0.010694869684962088,
+            0.02390864159830096,
+            -0.019152187046410703,
+            0.021786596486708207,
+            -0.017951050280323393,
+            0.026538582634436314,
+            0.014484894489694262,
+            -0.0035222259457311444,
+            -0.01946200885147006,
+            -0.00850690913394181,
+            0.011654565129063417,
+            -0.021423803876241532,
+            -0.0007985948078290641,
+            -0.0007290222944686444,
+            -0.021367628543092133,
+            -0.011189364017231132,
+            -0.009961779129778331,
+            0.010701637745810138,
+            0.001918633898887013,
+            0.010380983868185359,
+            0.008002182827196827,
+            0.010784760574518087,
+            0.014405640442760822,
+            0.0008958107769384412,
+            -0.010743001445440941,
+            -0.023795589003655273,
+            0.016987708698679597,
+            0.00438463274382093,
+            0.0007862212695139549,
+            -0.01627468994515667,
+            0.011936651269737248,
+            0.013250643815768723,
+            -0.009385605548926286,
+            0.009121882469425302,
+            -0.03716901103204463,
+            -0.004552808410398574,
+            -0.0035704811925430366,
+            -0.0020309197365173475,
+            -0.00908678158053104,
+            0.01559445415882485,
+            -0.013682507836005851,
+            -0.0009590922904844632,
+            0.00011696985057536548,
+            -0.0074756526903548915,
+            -1.5467890432254378e-05,
+            0.020649928837031005,
+            0.008438785939908428,
+            -0.0037382105257999757,
+            0.010376213224654074,
+            0.008826031303751312,
+            -0.013900359449287676,
+            -0.01527474712786084,
+            0.019035366035884273,
+            -0.018776918681908793,
+            0.0063098418544296165,
+            0.01677916431403883,
+            0.020778813096283988,
+            -0.02891862501830075,
+            -0.0007231469961304435,
+            0.022093939332021185,
+            -0.0012807635799950273,
+            0.006370841109790851,
+            0.008781853138377092,
+            0.006754082980602544,
+            0.0029102159062355374,
+            -0.003711173531879644,
+            -4.843770941622283e-06,
+            -0.003940474588075707,
+            0.007155539022862352,
+            0.004366433868349658,
+            0.036453728071874844,
+            -0.013661129348165794,
+            0.002541308805735001,
+            -0.0058838191638835155,
+            -0.00347401304572066,
+            -0.00593469688967835,
+            0.004407665856950772,
+            -0.010055727994798024,
+            0.004330827357536078,
+            -0.001570861535288093,
+            -0.0003731129422346693,
+            0.004883969598300302,
+            -0.001701673379646408,
+            0.007036892564844683,
+            -0.011863650867472625,
+            -0.0003979727750011258,
+            -0.020174851412609312,
+            0.021117284557875267,
+            0.00011666551282792189,
+            0.003662462447326642,
+            0.03723741246290771,
+            0.00476064510662738,
+            -0.00238355131307445,
+            -0.012880057949435798,
+            -0.013551338766321525,
+            -0.012986444917747435,
+            -0.020703821629987552,
+            -0.0036665673548525388,
+            -0.021006311026247164,
+            -0.0052061406234528915,
+            0.010362862594785442,
+            -0.011540184420787572,
+            -0.004023659825622843,
+            -0.015881589631889473,
+            -0.006531981517815174,
+            -0.03114211633687585,
+            -0.01124607488650473,
+            0.007493228366452009,
+            -0.019149285949049886,
+            -0.0039932870926865805,
+            0.026850121590606017,
+            0.001095360635760988,
+            0.01387696424607248,
+            0.0038058248676879625,
+            0.008117686785954085,
+            -0.0045937472017383955,
+            -0.0005669025036141417,
+            -0.01928244877716143,
+            0.003391980330529026,
+            0.008525354854576925,
+            -0.00361507909530282,
+            0.0070008404454106905,
+            -0.008835727374865623,
+            -0.016950017602980382,
+            0.0021654956715219517,
+            0.009334379529385142,
+            -0.006639140003365139,
+            -0.017587472740268683,
+            -0.011049929670847685,
+            0.010401189523318744,
+            -0.02283429470064187,
+            0.006257145380266984,
+            0.016439581169238607,
+            -0.009363030375624585,
+            0.004651575425738153,
+            0.019809183749755425,
+            -0.011810674576781702,
+            -0.005811328501528397,
+            0.008471992070998505,
+            0.013891259999721588,
+            0.0095606134736748,
+            0.02954498207274139,
+            0.0026596809804008996,
+            0.000561870602466584,
+            0.004889263753490895,
+            -0.004277061142504552,
+            -0.017423200644235665,
+            -0.006243787897080996,
+            0.0038507494787521856,
+            0.012873996671952184,
+            -0.009879417960879032,
+            -0.00048575277472139055,
+            0.01682604801673206,
+            0.004530939990187663,
+            -0.0033901248079550284,
+            -0.011868399508738274,
+            -0.020080331108085103,
+            0.0015580259874809898,
+            -0.01362200460317948,
+            0.015354347040143093,
+            0.003390601069331352,
+            0.0015284610287850575,
+            -0.004239880336806429,
+            0.01828645373058079,
+            -0.005013418527935815,
+            0.012749833447770729,
+            -0.006148890994661865,
+            -0.004587831231448011,
+            0.007986533783001116,
+            0.009166578151006778,
+            0.0004939857995863664,
+            0.05421019830078428,
+            0.002838056604877719,
+            0.01333205648597435,
+            0.013419127695838135,
+            -0.014523733369484934,
+            0.014086525147734823,
+            0.012688569853552613,
+            0.018497344547821693,
+            0.013037819487160016,
+            -0.012353776077273462,
+            0.017354801119939964,
+            0.0024084723134891487,
+            0.003909681065813095,
+            0.0022202438064449653,
+            -0.004528576079628071,
+            0.019594683950225115,
+            0.004070222328547378,
+            -0.0074362443729372105,
+            0.004453618452623641,
+            0.005007543168224267,
+            -0.013105524222311019,
+            -0.0112958915514782,
+            0.027105208848863218,
+            -0.01219417557128091,
+            -0.018749971743148304,
+            0.006706594269933445,
+            -0.02008458683853808,
+            -0.03763686535076864,
+            -0.0021332074289244973,
+            0.009992361098878196,
+            0.002795183524495193,
+            -0.00253695604043486,
+            0.01614298064601032,
+            0.016730443047719683,
+            -0.016985296937992726,
+            -0.018245902728910347,
+            0.0025257780765481825,
+            0.0016333325293570182,
+            0.01861807236979066,
+            -0.023168663546030196,
+            -0.006378864850313035,
+            -1.3795465857380962e-05,
+            -0.006829008132963306,
+            0.004850306884033312,
+            0.004703280112191218,
+            -0.0071714759384722965,
+            -0.0027773519096287217,
+            0.002256846645341137,
+            0.01975383140755998,
+            0.013198754584066554,
+            0.01572971337890945,
+            0.002600741959577911,
+            -0.005033508062449482,
+            -0.016817359058724685,
+            -0.005450011991586895,
+            -0.013478330855299777,
+            -0.009091854345715177,
+            -0.003042052092498012,
+            0.008217376329142566,
+            -0.025053469059115735,
+            -0.011965083373220982,
+            0.0119294363235832,
+            0.00391377331459168,
+            0.0003099203772539202,
+            0.007607672728136279,
+            -0.003830132495364688,
+            0.010331053246451333,
+            0.013823890473448383,
+            0.0076112887952959766,
+            -0.004723377809559611,
+            0.009600411526173948,
+            0.011460781436250457,
+            0.006015477068316837,
+            -0.005072711638895421,
+            0.023859899048048694,
+            0.01821920335673539,
+            -0.01680365156728093,
+            -0.002149107383573353,
+            -0.0027829676356423997,
+            -0.01884308738652242,
+            0.009665830801645276,
+            0.00880322398336952,
+            0.008308268605449346,
+            0.003906369351868086,
+            0.00974531700889854,
+            -0.013875904819275925,
+            -0.0050570811397453235,
+            0.002870120865445989,
+            -0.005729556771404781,
+            -0.01901337075686543,
+            0.000297242372701423,
+            -0.011987192261104132,
+            -0.011327157550660895,
+            -0.01409710922165158,
+            -0.003221240464411861,
+            0.005283472511900512,
+            0.021472187640986972,
+            -0.004410881709548286,
+            -0.001855852749039012,
+            0.01351697599517034,
+            0.010060125186506398,
+            0.0021658738823899905,
+            0.0002206474541239167,
+            0.005418828324605768,
+            -0.0027919130944783397,
+            0.0008094042430809068,
+            -0.0015826797905407809,
+            0.021089276765434876,
+            -0.02016611086095107,
+            -0.009413977569213423,
+            -0.009288078098692112,
+            0.005842197094339927,
+            0.028793032462603636,
+            -0.007165634502373454,
+            -0.010104002083017078,
+            -0.01994315293556678,
+            0.0003786527922888834,
+            -0.07810609175343784,
+            -0.005210636756258548,
+            0.001870513618910238,
+            0.03126380113105983,
+            -0.010817888476436834,
+            0.0033673768058629823,
+            -0.002204487854177595,
+            0.014542299655712928,
+            -0.001760107965206874,
+            -0.0013283746982194259,
+            0.0071954806737858056,
+            -0.005640400818010697,
+            -0.006210989097047217,
+            0.005904163661619133,
+            -9.945941028848348e-05,
+            -0.010266938705916303,
+            0.00057263080955709,
+            0.02209498395848068,
+            0.0159398346608872,
+            -0.006575439759804175,
+            0.01816748552021211,
+            -0.00719294553484007,
+            -0.009838738592324234,
+            0.005639288780502127,
+            -0.0019007683459126327,
+            -0.015369619176147573,
+            -0.011475977270803784,
+            0.009862808084641291,
+            -0.030069890644150284,
+            -0.00010725753918890802,
+            0.00943698424646706,
+            -0.013540245857270492,
+            -0.008293042205008774,
+            0.007212974960781723,
+            0.006708343474532583,
+            -0.0012017968151025369,
+            -0.0078026000239507145,
+            -0.02527526731750743,
+            0.0027578400032925625,
+            -0.002446811971857853,
+            -0.004899702349784173,
+            0.0019960465173086183,
+            0.001479503476855694,
+            -0.012645905858466798,
+            0.01216748847452763,
+            -0.013378877368040743,
+            -0.012053251571418766,
+            0.012508842412556306,
+            -0.00027167241745418584,
+            0.01682680930386839,
+            -0.020607299598265426,
+            0.0010702151063620052,
+            -0.028075439824932304,
+            -0.007671848153932823,
+            -0.015486630894165054,
+            -0.0009776136330459065,
+            -0.0012939921569540803,
+            -0.03481880531471923,
+            0.012889447665794033,
+            -0.005200149829198305,
+            -0.006094702500154477,
+            -0.00459778303079732,
+            -0.024583504817899775,
+            0.006072281238752663,
+            0.00607350167867964,
+            -0.006506508970816635,
+            -0.016816439635445445,
+            0.008428048142588516,
+            -0.0011040593013238872,
+            0.004873670535200424,
+            -0.007475800310154908,
+            -0.0005638920132076605,
+            -0.013047135093134946,
+            -0.005463454269764494,
+            -0.0008522562324122834,
+            0.01827123304871433,
+            0.0044055851550317835,
+            -0.014526381222031981,
+            0.016862445236481526,
+            0.013250692156820124,
+            0.01602714293656341,
+            -0.0071089293427753115,
+            -0.0004265739315250755,
+            -0.02750630108787212,
+            -0.00327225092954903,
+            -0.023687644154808972,
+            0.00041213204365507744,
+            0.009447660230507965,
+            0.0203905264889617,
+            0.012694421477479161,
+            0.004620617778996077,
+            -0.001557511764988156,
+            0.0066751090724502,
+            0.02696959560423252,
+            -0.01099183126742224,
+            -0.006969111598423376,
+            -0.01884504192789845,
+            0.01113023717919988,
+            0.025135748541199255,
+            0.003381303139273845,
+            -0.011317957771935042,
+            0.008112598754655146,
+            -0.0009247845526290428,
+            -0.010302980598863016,
+            0.02889587382004902,
+            -0.0391134481322538,
+            -0.015599591108049343,
+            -0.004605824192962023,
+            0.007406520608110251,
+            -0.015596917115435033,
+            0.00770468362805995,
+            -0.012614339656105264,
+            0.001699049329817941,
+            -0.005851477164035241,
+            0.03535087340659861,
+            -0.0031950960727572925,
+            0.0006909856233673865,
+            -0.003395233050301427,
+            0.003531135044131077,
+            0.012355948349937309,
+            -0.004444221317023453,
+            -0.030726300006909638,
+            -0.0022983766869702346,
+            -0.011198252640110788,
+            0.009274666307515955,
+            -0.016207714291658314,
+            -0.017342435976846468,
+            -0.009424737744066259,
+            -0.021827243671837953,
+            -0.012403084982789827,
+            0.029747453318486338,
+            -0.005824001042844527,
+            0.01747569398142269,
+            -0.005219060548639168,
+            -0.008365003057800934,
+            0.00021272411948388247,
+            0.01344335658390732,
+            -0.028498240040284714,
+            0.00041963499785238845,
+            -0.002322931702641349,
+            0.0126393550661767,
+            -0.001595134597082556,
+            -0.00983073906977773,
+            0.006903313245694627,
+            0.0010522208685011098,
+            -0.015648940549713764,
+            -0.0007573245570947632,
+            -0.008533504878052644,
+            0.020475329445149498,
+            0.00027791387213409814,
+            0.001074213629646005,
+            0.013864331671732542,
+            -0.02102352993295736,
+            0.01576938071930521,
+            -0.021722775924283996,
+            0.01893267442161547,
+            -0.006577103101038035,
+            0.005772016664149475,
+            0.008283223343661555,
+            0.014068789332250016,
+            -0.014030042849491258,
+            -0.00663752723080222,
+            -0.034436220972634726,
+            -0.0122621879114314,
+            -0.001946072976826902,
+            -0.005308941654794638,
+            -0.014773123927909993,
+            -0.0011054822266082056,
+            -0.026205461509573584,
+            -0.006460373065662098,
+            0.012822188046808264,
+            0.019900312098359022,
+            -0.008409070094265829,
+            0.014163132434889265,
+            -0.006743711872055091,
+            -0.0028230771810882492,
+            -0.012652021749331811,
+            -0.018061180380009395,
+            -0.01083672324723357,
+            -0.01890966636827258,
+            -0.0018635408070944168,
+            0.004709222372611061,
+            -0.015760618199235518,
+            0.011435131737642627,
+            0.03198874888333405,
+            -0.01567271472650171,
+            0.008243406188694909,
+            0.016411496328333205,
+            0.0034236971031973503,
+            0.011728079139322837,
+            0.0038311415233052795,
+            0.004621789116267879,
+            -0.011109450058055078,
+            -0.011473158133621661,
+            0.006524490579223986,
+            0.00954511918454185,
+            -0.004913142601134428,
+            0.024347072255007807,
+            0.0008745419206472714,
+            0.005045794664611216,
+            0.005304506902446091,
+            0.011488187043525824,
+            -0.013135271908307747,
+            0.019746484188715552,
+            -0.002055210086078267,
+            -0.015571665869889405,
+            -0.01535499169234956,
+            -0.01105419264826655,
+            -0.018626399757822328,
+            0.0017108083455869296,
+            0.0181439079855921,
+            -0.0012493542718277685,
+            0.012137753522284387,
+            -0.00924460959150499,
+            -0.005019437549649743,
+            -0.0027156377206998474,
+            -0.038211954658821295,
+            -0.0021878970332906457,
+            -0.018809487178512628,
+            -0.02498945889484935,
+            0.03505216568400238,
+            -4.26477081213836e-05,
+            0.00997643255528116,
+            -0.010310197159232386,
+            -0.0002454902098986129,
+            -0.011065403358802515,
+            0.014954998647970473,
+            0.004624123403100598,
+            0.008872923678983478,
+            0.008560280683876702,
+            -0.016419500515801093,
+            -0.002997557798375716,
+            0.007472390947008284,
+            -0.00374822278027057,
+            0.012680407710694952,
+            0.015353684671120731,
+            0.020886474047003733,
+            0.011161262397502876,
+            0.002045862158732102,
+            0.00887271411363814,
+            0.009563361150297699,
+            -0.0003513237086717377,
+            -0.010459621030424415,
+            0.015419039131459582,
+            -0.02204848739950703,
+            0.012172257374410352,
+            0.04346634265794738,
+            0.019197433209794286,
+            0.029488039318675732,
+            -0.02272810439246353,
+            0.01448092574459596,
+            -0.003801043409441206,
+            0.009476229966122152,
+            0.0049309968883087685,
+            0.02346191840355209,
+            0.003973840862678076,
+            0.0073546443568171405,
+            0.010647581332452525,
+            -0.00972051412996416,
+            0.025644022498404723,
+            -0.00846669916801108,
+            5.1518540791212095e-05,
+            -0.004407265804145216,
+            -0.026400029105519137,
+            0.01615193008014211,
+            0.0081722536401398,
+            0.0010702314763197463,
+            0.0010263623479474385,
+            -0.0019566604229607643,
+            0.007982949510899019,
+            0.007079322685771456,
+            -0.015274148711633536,
+            -0.003886053737590683,
+            -0.018527455865296014,
+            0.0012864569064749438,
+            -0.024337133883771334,
+            0.010466651250597847,
+            -0.017568706814132484,
+            -0.014822354159709854,
+            -0.021666681078831483,
+            -0.013232016867264097,
+            -0.02011653224168991,
+            0.0035858355517491515,
+            -0.008632755352062642,
+            0.013025728891203198,
+            -0.016962939287139683,
+            -0.021494971929779273,
+            0.008430887589398292,
+            0.00870802359711456,
+            0.007481282677955534,
+            0.009728368603458082,
+            0.014996157358749115,
+            0.01736292711410622,
+            -0.022712386143092265,
+            -0.023163001611599246,
+            -0.006682209717364464,
+            -0.003531495447167166,
+            0.006762379833125297,
+            0.00933681196321128,
+            0.012884185500343249,
+            -0.007077895849501418,
+            -0.007554762054641417,
+            -0.013283067570021503,
+            0.006197966297581871,
+            -0.003993067576677973,
+            -0.009464789349849612,
+            -0.0015662739825834614,
+            -0.0061389795593180686,
+            0.002753664732282926,
+            -0.0001791088888074743,
+            0.0007595261237141144,
+            0.017122834676182343,
+            -0.008176319370751018,
+            -0.009787747261447871,
+            -0.02027228453943018,
+            -0.010683273300243025,
+            -0.015604745521156111,
+            -0.0016279406232527138,
+            -0.006506713191861036,
+            -0.014420323628589382,
+            0.010888206569219648,
+            -0.011930708571093279,
+            -0.00887210117616784,
+            0.010807147705360856,
+            0.0073900910661282935,
+            -0.003431679006289002,
+            0.037804020313161775,
+            0.0011945597874820288,
+            0.00778778394048267,
+            -0.003366543306746973,
+            0.005296727102519836,
+            -0.017761251541598876,
+            -0.026382400396890007,
+            0.0010357716464853195,
+            -0.01775777668381489,
+            0.010807713750049491,
+            -0.008515368407459745,
+            0.0016051389544388124,
+            -0.00034958600424912995,
+            -0.009585854918521076,
+            0.0036549720706351473,
+            -0.008110927907950148,
+            0.004212219431447508,
+            -0.03507890639477636,
+            0.011409377585617926,
+            -0.01819120811524444,
+            -0.008962242411035718,
+            0.01710177594046791,
+            0.037560936862121835,
+            0.0028513388998398104,
+            0.00030058353811717895,
+            0.0014792184333546447,
+            0.004427921639326168,
+            0.021438127785303025,
+            0.006339483221420334,
+            -0.001187747087595264,
+            -0.02237982599829809,
+            0.0062435650294483695,
+            0.002200578793697352,
+            0.010092828474934744,
+            0.023105023225115266,
+            -0.016797326021070008,
+            -0.015378520287395565,
+            -0.012732527281000097,
+            -0.010797463778182385,
+            0.009881499418260473,
+            0.02801767062785128,
+            0.01811139137849365,
+            -0.017358098300028903,
+            -0.020207173948076473,
+            0.0016242901745373126,
+            -0.026331540602868762,
+            0.007269751661479859,
+            0.11940631114242901,
+            -0.0045382135734049305,
+            -0.005994407567723939,
+            -0.008679712231646732,
+            -0.004482553208064535,
+            0.016709571294385293,
+            -0.013387043582971358,
+            0.069179848974036,
+            -0.007221545718220735,
+            0.0037392229364642915,
+            0.0028916224964057385,
+            0.00462729158956839,
+            -0.009727339449635306,
+            0.011570145487622916,
+            -0.011287365791681587,
+            -0.012991994493845187,
+            -0.013076194601468383,
+            -0.0180830261130804,
+            -0.007272768410218907,
+            -0.004155564814758654,
+            0.004204993170487392,
+            -0.005525575983182816,
+            0.008051108079520902,
+            -0.017551645696802648,
+            -0.010800070994485792,
+            0.013007176804761548,
+            -0.0023627485852619817,
+            0.005390986239687555,
+            -0.029668390278675554,
+            -0.0007650391259819867,
+            -0.0002446394046947245,
+            0.021280292985495608,
+            0.003137771886236958,
+            -0.0019077298815862946,
+            -0.010810159675564049,
+            0.007040107942856957,
+            0.023835993694726953,
+            0.021875366566457236,
+            0.0180511566130312,
+            0.012135351541044354,
+            0.020483975362807574,
+            -0.008667079096167644,
+            0.026324312888512866,
+            -0.001552207146726937,
+            -0.00397810465151493,
+            0.0077577136352257004,
+            -0.009849600842081875,
+            -0.00816392268697345,
+            0.00677353736462267,
+            -0.017288900282504148,
+            -0.020418071729964713,
+            -0.020383762873109595,
+            0.025741408097358233,
+            -0.011755094311186617,
+            0.002737129201159793,
+            -0.008266210469668866,
+            0.027287436144516414,
+            -0.006283892307996516,
+            -0.0029440642954701632,
+            0.009428876742171362,
+            -0.03788574758692254,
+            0.008834605607057885,
+            0.006463342392514337,
+            0.026711196829464153,
+            -3.9827332346130654e-05,
+            0.005565561841469247,
+            -0.0006738085601962865,
+            -0.006104320995853506,
+            -0.010792830169647851,
+            2.4565453208679464e-05,
+            0.017983908748172493,
+            0.0035181716362611077,
+            0.016970206528324653,
+            -0.005449808667944701,
+            -0.023171388778833197,
+            0.007092509502738752,
+            -0.01555320330471131,
+            -0.023306285898839978,
+            -0.002028783456704567,
+            -0.03196436161255752,
+            0.0033412618908711163,
+            0.007527633574188688,
+            -0.01890570959143866,
+            -0.017859735533232564,
+            0.010329340599904344,
+            0.004382396539605985,
+            -0.0016746397858953155,
+            0.006643785064338525,
+            -0.03307812909454471,
+            -0.01927475592253288,
+            0.004686970010943116,
+            0.012733525168279355,
+            -0.012993441126970916,
+            0.015291832952199082,
+            -0.020491167895268936,
+            0.016451976170508816,
+            -0.012259387189535237,
+            0.018393274368854132,
+            -0.006997668664226754,
+            0.011843450716200625,
+            0.011457415642860215,
+            0.009560877634897626,
+            -0.01790001115302499,
+            0.0214879415944361,
+            0.0066829070506954115,
+            0.03900547021123392,
+            0.02512484499576633,
+            -0.012651294056575184,
+            -0.0018425680226742782,
+            -0.00011361458320146007,
+            0.014982647266052349,
+            0.0061874424115559,
+            -0.003689464024837091,
+            0.02705902205734107,
+            0.018239469373522537,
+            -0.021070704644859155,
+            0.020037508334843183,
+            -0.00973804771680803,
+            0.018876283979591423,
+            -0.013303080480396818,
+            0.010592424494628194,
+            0.004717870260467274,
+            0.01070131238869639,
+            0.009087656015197565,
+            0.006708673669681232,
+            0.025928065153829043,
+            0.03663367449576964,
+            -0.00511073108390551,
+            0.002161381031392467,
+            0.004864092591944181,
+            0.012814518637357098,
+            0.001556833001747549,
+            0.007915766323215207,
+            -0.007012785514562176,
+            0.005834786334750152,
+            -0.002311533032619821,
+            -0.00122162928193159,
+            0.019525999280692766,
+            0.000824163362963678,
+            0.012875575199417254,
+            0.013259376059185821,
+            -0.002390133687709065,
+            -0.005682644446302278,
+            0.0019262218417782657,
+            -0.018193478045923098,
+            -0.015131696847666434,
+            0.029463723011600933,
+            -0.011447039651934148,
+            -0.008083114389964213,
+            0.0020642857121204753,
+            0.012160202510990566,
+            -0.00866944494225225,
+            -0.005378300635997713,
+            0.0038280364584418335,
+            0.014427318067525318,
+            -0.01799358399064332,
+            0.003381165206832161,
+            0.005252552821854107,
+            0.019433554824232185,
+            -0.00028791985674147885,
+            -0.013487678738391956,
+            0.0005317922815671783,
+            0.014982152801567637,
+            0.011902348165369078,
+            0.0016056390299304837,
+            -0.006935777197137747,
+            -0.00446584737370504,
+            0.004586814866182051,
+            0.006269466956110759,
+            0.0027516492952993445,
+            0.008157662045793032,
+            -0.0025001885993292637,
+            -0.016648747436203294,
+            0.017735757469839637,
+            -0.0030669873770852725,
+            0.018058248627079793,
+            -0.0038238312452176745,
+            -0.0147172470474118,
+            0.0043328417868095355,
+            -0.001888507803054499,
+            -0.018556943283494755,
+            0.009407930200720534,
+            0.002560949424273885,
+            0.010411159836451958,
+            0.0052571230238294825,
+            -0.007499470181994448,
+            -6.145889479758644e-05,
+            -0.015163819825329975,
+            0.022836682800304944,
+            0.016390944816242627,
+            0.001713572907204073,
+            -0.0035687505973373878,
+            -0.031210299824387475,
+            0.0029076511444359886,
+            -0.007822231980296341,
+            0.017821806420465356,
+            0.008250605126658402,
+            0.007908773724009905,
+            0.008351355849444162,
+            -0.00743970642051451,
+            -0.002411981861500092,
+            0.012543476434788293,
+            0.0202407777715137,
+            -0.0024635818448627447,
+            0.02535771413579989,
+            -0.012533449594824386,
+            -0.0022489292647974992,
+            0.0012362917358624392,
+            0.01578939075186126,
+            -0.0008001783766371555,
+            0.006311837812042848
+          ],
+          "keyphrases": [
+            "Switch Transformers",
+            "Mixture of Experts",
+            "sparsely-activated model",
+            "training instability",
+            "multilingual settings"
+          ],
+          "title": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "addd7c61-bdd3-470e-8c4f-439af824f89e",
+        "properties": {
+          "page_content": "F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+          "embedding": [
+            0.0002445570513886185,
+            -0.025035678422212698,
+            -0.026056660824720954,
+            0.0021234386221176942,
+            -0.007526403638487191,
+            0.018963965001103752,
+            -0.007728070932730228,
+            0.062213823247851435,
+            -0.00501600735659763,
+            0.031461940272958254,
+            0.014585783857839997,
+            0.010118033107401152,
+            0.009983318452435566,
+            -0.03282616973330728,
+            0.015192449242879527,
+            0.01480596701302375,
+            0.019405922018728892,
+            0.022192910412596127,
+            0.008048967251594976,
+            -0.0367043822293015,
+            0.017737679976556845,
+            -0.022757028540067756,
+            -0.06290354368907101,
+            0.016842087557557506,
+            -0.0050319451760917995,
+            0.019311667659179976,
+            0.008548884527537198,
+            0.02344214718766695,
+            -0.042417710048062586,
+            0.01950972309054391,
+            0.021637767293446648,
+            0.010581067128107188,
+            -0.014214247204390872,
+            0.01016365063627519,
+            -0.025782209417014867,
+            -0.02611397356963557,
+            0.034251067799009986,
+            0.0071402676991737055,
+            -0.020147891665364803,
+            0.056472902950962155,
+            -0.006027534100459245,
+            0.005473493744726429,
+            -0.03139313966858068,
+            0.00043634965151753214,
+            0.013171990459657583,
+            -0.016474119463665594,
+            0.0075792906007691105,
+            -0.0032442837467945356,
+            0.02492845005795393,
+            -0.04366675539501486,
+            -0.006033838756045725,
+            -0.018660330752604155,
+            0.02088611082654532,
+            -0.019387265112916365,
+            -0.01222746328548615,
+            0.005977009213829137,
+            0.0012690574665308455,
+            0.05430319716489232,
+            0.058075720286678886,
+            0.02554562736016709,
+            0.02997758641056458,
+            0.00014178556329408164,
+            0.004139467932725944,
+            -0.004454862143349584,
+            0.031094774362479788,
+            0.0017837745263941452,
+            -0.013542834385584962,
+            0.01983846902748998,
+            0.010136662804503968,
+            -0.0387179389182337,
+            -0.06156798393873734,
+            0.015844092498186847,
+            -0.020820695961906926,
+            -0.02678200120240037,
+            -4.570339833017677e-05,
+            -0.015544616932001905,
+            0.005434882475939711,
+            0.050835936350804696,
+            -0.004985250023269435,
+            0.0016509410141492726,
+            0.022801366331672433,
+            -0.004080282248725238,
+            0.0078367492683837,
+            -0.00034876063727630653,
+            0.039772801289415724,
+            0.0288510926430687,
+            -0.022609202992981695,
+            -0.07091181813750308,
+            0.045194222785472735,
+            -0.03225547853169315,
+            -0.032585463285183205,
+            0.01670145480899693,
+            0.03286138231600811,
+            -0.009733112712376757,
+            -0.030197857075781466,
+            0.0731500616714208,
+            -0.005545147701735842,
+            0.017072915518860705,
+            -0.021002529816498382,
+            0.010525361666209598,
+            -0.012745643075817463,
+            0.025352505498532282,
+            -0.003102906957075941,
+            0.016391641977220053,
+            0.02534354270624064,
+            -0.008108690979345334,
+            -0.006451703723989414,
+            0.0065097623653779324,
+            0.04453924610212246,
+            0.0889545538715994,
+            0.019774398083206815,
+            -0.003981259900744729,
+            -0.046615274629371053,
+            0.00732791197790897,
+            -0.02916202133853672,
+            -0.04303144138453202,
+            -0.04367640508681853,
+            -0.010910501188273782,
+            0.023755744360759886,
+            -0.003399898430475389,
+            0.03910227884474275,
+            0.04137187598162877,
+            0.03363334616934253,
+            0.00916121492790106,
+            -0.031686589382074935,
+            0.026907320660055866,
+            -0.01693827759642556,
+            0.002542422168436765,
+            0.00734783450287368,
+            0.008189080458799796,
+            -0.006352648760821389,
+            -0.002333605867785783,
+            -0.03831808898525347,
+            -0.036762698962852246,
+            0.012773386000855496,
+            -0.008349610778914014,
+            0.027936454763857345,
+            0.07015560257394027,
+            -0.03713244149832974,
+            -0.0066245405611366675,
+            0.02802618921440782,
+            0.0017169880395367697,
+            0.023346396749133396,
+            0.007605522879978981,
+            -0.023484167590494896,
+            0.008594274917753499,
+            -0.0071765874950466295,
+            0.0677537727828362,
+            -0.008126000179550069,
+            0.011046754551416857,
+            -0.019516727128214784,
+            0.031654663874166704,
+            0.0007057013357055938,
+            0.021535694192642306,
+            0.03313372731162823,
+            -0.005794425715079591,
+            -0.03865395674172617,
+            0.032271538528276474,
+            -0.029790294001121286,
+            0.0013647166052139265,
+            0.014194415049797637,
+            -0.004899479294176156,
+            -0.015920769574443834,
+            0.0013708196727143433,
+            0.029757298661028133,
+            0.045717552753231847,
+            0.042911032378569906,
+            0.0016019404554248433,
+            -0.06844613496660937,
+            0.06365820914938357,
+            0.012895947255192629,
+            0.03208498337942363,
+            0.009724409704032419,
+            0.042361427690196844,
+            0.008120380964731038,
+            -0.0004384537671641681,
+            -0.005540416382489673,
+            -0.005125920468958917,
+            -0.021186731704155592,
+            0.03258917022091572,
+            -0.01443327736487225,
+            -0.007107908179884635,
+            -0.00206346150761894,
+            0.014042865018508809,
+            0.028517557440008168,
+            0.017283745982594707,
+            0.0029564774390786045,
+            0.04378212980825259,
+            -0.019454732028746872,
+            -0.009437148551161395,
+            0.018820938021162286,
+            0.0008903198503990673,
+            0.025297040557527045,
+            -0.0011429799732923295,
+            -0.01670583390791835,
+            -0.009421860087826363,
+            -0.0328764813310764,
+            0.009563088287577815,
+            -0.0015347427243049344,
+            0.019228666906882602,
+            -0.007174098839673089,
+            -0.04068735475327182,
+            0.008903031576514094,
+            -0.011472954071988202,
+            0.015735403376626513,
+            -0.007084760348074423,
+            0.008729329609340102,
+            -0.00638454270428521,
+            0.016210007209816298,
+            -0.02460475066044075,
+            -0.009047036293783832,
+            -0.022598561172568143,
+            -0.020144302113939427,
+            -0.010886272544512816,
+            -0.010886770387412563,
+            0.008591903125597681,
+            -0.012216526799178626,
+            0.023956058440750026,
+            -0.017402060516065108,
+            -0.018328740623422317,
+            -0.00046940812181371226,
+            0.022921229116282273,
+            0.009026508358954174,
+            -0.00446208425714566,
+            -0.04421346673289319,
+            -0.01214251475197077,
+            -0.016860336137191526,
+            0.005985059770480569,
+            0.02804423359971541,
+            -0.03510963462663318,
+            0.012588036039507566,
+            -0.0018909139908325232,
+            0.0019847602212350957,
+            0.011374423039138836,
+            0.015486762163219968,
+            -0.027040748338117617,
+            -0.016329845569406535,
+            -0.012600824227928017,
+            -0.01765236203915568,
+            0.01673827406092187,
+            0.0016641690585652725,
+            0.003997011410859195,
+            -0.00499922800986537,
+            -0.02782499674212141,
+            -0.008774389172636544,
+            -0.011506812701874534,
+            -0.009783555199790996,
+            -0.015740884114025612,
+            0.013493654174963239,
+            0.014254624157312108,
+            -0.033270488877124596,
+            -0.018446763439664365,
+            0.02247061755496804,
+            -0.0035396859846893343,
+            -0.03759408156279411,
+            0.04544560120093284,
+            0.052893430652542234,
+            -0.02220682841881151,
+            0.002828355769475809,
+            -0.03469141962159207,
+            0.05950248906327849,
+            -0.03444371252359212,
+            -0.03378776392860449,
+            0.06521072243320349,
+            -0.009713026070923203,
+            0.015249223153809483,
+            0.03131206469454657,
+            0.0034865285730464384,
+            -0.0025997752599464445,
+            0.00912836988165194,
+            0.033031075123542926,
+            -0.014878258361522118,
+            -0.013101723481793925,
+            -0.01797681408884685,
+            -0.002397237505725697,
+            -0.03438738791694146,
+            -0.021713511748922477,
+            0.00836495266415799,
+            0.013064910474172053,
+            0.017110113503472666,
+            0.017514586931181946,
+            0.016195079102773242,
+            -0.0024109742901844617,
+            0.01545592172528996,
+            0.001046760697009891,
+            -8.863738247355372e-05,
+            -0.010979717992873124,
+            -0.017762086340346584,
+            -0.052667758929128256,
+            -0.018996338856872796,
+            -0.04421628171000429,
+            -0.005982494173649518,
+            0.0012420734363470585,
+            0.009886466834005627,
+            0.04949340056515024,
+            -0.01295176306623405,
+            0.003508142432105508,
+            0.010930632194984236,
+            0.01019261004578912,
+            -0.006556730312565122,
+            -0.032765993356945154,
+            -0.006485728779853869,
+            0.025709969056450885,
+            -0.022932446214570953,
+            -0.021792413124305354,
+            -0.007047777916806282,
+            0.0038659532636633486,
+            0.02321123092409853,
+            -0.026003302747277822,
+            0.017643847938685852,
+            0.010129447679208158,
+            0.009551458127273026,
+            -0.017747795509094742,
+            -0.02891499441428287,
+            -0.030321565228144824,
+            0.039350178167852384,
+            0.02899873253152933,
+            0.011338359735211436,
+            -0.033634061116901,
+            -0.013629902520109783,
+            0.0376352218933715,
+            -0.0019251015746424632,
+            -0.07267257827375968,
+            0.036685408572352814,
+            0.0050526642351482985,
+            -0.00019289056191322955,
+            -0.013581714265389013,
+            0.05300479823846402,
+            -0.04139717386779843,
+            -0.03335524761191713,
+            -0.00920987783097114,
+            -0.022616612319255443,
+            -0.005274093640024995,
+            -0.0012667939280239298,
+            0.032502321377997974,
+            -0.020089629343721754,
+            -0.024354491736044894,
+            -0.010206744478678104,
+            -0.02465036256948816,
+            0.017966455981144307,
+            -0.012995995315432267,
+            -0.02135780149071486,
+            0.010967719068923269,
+            -0.03402154331207373,
+            -2.0626689329697705e-05,
+            -0.01092309016851862,
+            -0.018683564965672157,
+            0.01571041043301007,
+            0.016259448102797114,
+            0.015403085986247105,
+            0.0020047641027155683,
+            0.015172380396260792,
+            0.008099191720720747,
+            0.004074377809743208,
+            0.01893667379047588,
+            -0.03384452762579117,
+            -0.038822883991989264,
+            0.050315952008617104,
+            0.0008458969404873289,
+            0.0012049589464131246,
+            0.006406225290437734,
+            -0.0042817559898658255,
+            -0.0411169553122232,
+            0.012933323148113837,
+            -0.05784927886369837,
+            0.006262727192870561,
+            0.012928199517490538,
+            -0.01786898028943474,
+            -0.02291183998443847,
+            0.00046953379111417065,
+            -0.024864892714864046,
+            0.02151263275001755,
+            -0.017247234755598105,
+            0.020405349906524557,
+            -0.023068655052252782,
+            -0.006872857342813033,
+            0.05842934123173711,
+            -0.02308642746390288,
+            -0.04925938088613823,
+            0.032829595473385986,
+            -0.007184802367095134,
+            0.03878513331161335,
+            0.055971017386540985,
+            0.016337316575743225,
+            0.006439501224944926,
+            0.026084366083114555,
+            0.013204436213767278,
+            -0.012028383848550473,
+            -0.0383295105300241,
+            0.015805232298071025,
+            0.030751727443425712,
+            -0.04685643586863969,
+            -0.010258856888449854,
+            -0.02541648783174773,
+            0.01599307149086869,
+            0.020046184897703115,
+            -0.0044598427609889935,
+            -0.0012270315730225135,
+            -0.009664326045898607,
+            -0.01520444928791987,
+            -0.02631191309701645,
+            0.02193635473056903,
+            -0.013740214173212537,
+            0.0007504924364311022,
+            -0.001111971939163715,
+            0.0012797636039507028,
+            0.008478066024469051,
+            0.01501219347281274,
+            0.027440349509054086,
+            -0.022728634568050663,
+            0.006916913421268421,
+            -0.008499438828073956,
+            -0.03788213665656746,
+            -0.006718262893819488,
+            0.029198375723488434,
+            0.006169843558767785,
+            -0.016427547826608675,
+            -0.0049009468913267205,
+            0.0022509753971959087,
+            -0.016307639896946134,
+            0.00372456949880952,
+            0.010347360137213142,
+            0.035374109073612686,
+            0.01856811791368892,
+            0.019247903053478954,
+            0.0035591273657627456,
+            0.010292179556783772,
+            0.004040044337368795,
+            0.025869815038962875,
+            -0.04526297785801751,
+            0.030173313035569235,
+            0.01521363974209499,
+            0.009059160353158267,
+            -0.03914219183565228,
+            -0.02367109515190293,
+            -0.025737979702884085,
+            -0.021654077593318197,
+            0.02346511753268162,
+            -0.02730833269330862,
+            -0.006289840626554075,
+            0.006531934540756005,
+            -0.0006804032161296996,
+            0.003928945341546481,
+            -0.0033962179480786624,
+            -0.037975452991752895,
+            -0.009840510213526213,
+            -0.04023861493405766,
+            -0.0241245373830213,
+            0.003822063151525241,
+            0.003249755356115233,
+            -0.06594891648887229,
+            -0.001768385222533634,
+            -0.06222620899240395,
+            -0.013594921028316139,
+            0.03441942883093182,
+            -0.024438842229617237,
+            0.04820486752403123,
+            -0.05486229161412509,
+            0.018219866569279347,
+            -0.046562594596398506,
+            0.011354841468658838,
+            0.021738373879790916,
+            0.04347115691988908,
+            -0.007572536447300832,
+            0.014626550834376365,
+            0.009906079088662038,
+            -0.04530647508884267,
+            -0.04520033458696902,
+            -0.011400230794947595,
+            0.011239630851440557,
+            0.003287355301854498,
+            -0.012304101523783504,
+            0.02623659451177029,
+            0.015903981896266153,
+            0.04300392046214319,
+            0.03414364576071245,
+            -0.0005691335103219632,
+            -0.049506948352089984,
+            -0.027336474868358992,
+            -0.014910043127825028,
+            0.02552137904202476,
+            -0.03055258097953423,
+            0.003401887357920115,
+            -0.03228200232009397,
+            0.013338682663233296,
+            0.018207379090305786,
+            -0.02224210534533673,
+            0.032569850137027664,
+            0.014446434357859929,
+            -0.040301668913492676,
+            0.0012399866793165108,
+            0.0002450513991178417,
+            -0.038300625347349834,
+            0.008119751344998158,
+            -0.013712452991626932,
+            -0.05381257202269606,
+            -0.022199278450525977,
+            -0.014717114803771204,
+            0.02206967996899982,
+            -0.0023470596291977116,
+            0.017222681546831476,
+            -0.029965631581610256,
+            0.0053103260868707786,
+            0.039279027237044066,
+            -0.001286692479487674,
+            -0.01148541970963179,
+            -0.0008706049086180766,
+            0.022795196067681996,
+            -0.01968201411368477,
+            -0.045534614872904305,
+            0.009877856016973188,
+            -0.020566606587457816,
+            0.016002665913994348,
+            -0.015445999456355206,
+            -0.0052940252527317235,
+            0.010360748741468617,
+            -0.03170100753615497,
+            0.02004511444257144,
+            0.009722541379665509,
+            0.006411072442536516,
+            -0.017460683536795665,
+            -0.0033423349847155888,
+            0.015743901543629382,
+            0.047005401368949463,
+            0.0007329338628268773,
+            0.006153613674047577,
+            0.02501606022399451,
+            -0.013054974002495664,
+            0.05036387123988784,
+            -0.02807526639433245,
+            -0.03099731580589117,
+            0.04344605263104773,
+            -0.019489740746149623,
+            -0.017636713524640496,
+            -0.017960525325193457,
+            -0.009792405274000215,
+            -0.0062801692104839674,
+            0.02801421653619339,
+            -0.00712206060099242,
+            -0.005749470909187103,
+            0.020504942834135256,
+            0.013170883165927672,
+            -0.011218763497707175,
+            0.008523945766715354,
+            0.01511514654097483,
+            0.027931571407773977,
+            0.03214582418497515,
+            -0.02345593941074465,
+            0.0333746849861944,
+            0.009324947617983663,
+            -0.004234113006464689,
+            0.005131158074842622,
+            0.0020437263732432953,
+            0.00176803337042029,
+            -0.029372739581870692,
+            -0.01539406767336474,
+            0.015958225909710817,
+            0.007396588215584237,
+            -0.003291618431912445,
+            -0.0316834931316845,
+            -0.0028368107695447525,
+            0.02614797456533501,
+            0.003832488818435695,
+            0.026426519154286857,
+            -0.015605455255333021,
+            -0.02018445776455915,
+            0.008876565653043164,
+            0.023669947107607083,
+            -0.055401884612295824,
+            0.01882053182190406,
+            0.011046506385076857,
+            -2.6003594081234644e-05,
+            0.01591257026228522,
+            -0.0046432737555961385,
+            0.005094314300671912,
+            0.0482159962414822,
+            0.015126643838581288,
+            -0.013318945197936635,
+            0.0020418779929576307,
+            0.015136735547940301,
+            0.001727083579122356,
+            -0.020116332736417462,
+            0.002896210161586164,
+            0.012858057228188658,
+            0.0092589465483404,
+            -0.012984552574495295,
+            -0.005651788746934606,
+            -0.016856929103895944,
+            0.03249171052801259,
+            -0.005843029913406538,
+            0.02336833175997056,
+            0.04800276014595125,
+            0.02261889070736121,
+            -0.03578655846417886,
+            -0.011451513080556048,
+            0.0077334037670644135,
+            0.009415939321950421,
+            -0.012399032514032437,
+            0.006084821107593102,
+            -0.034655634244304866,
+            0.013104268092544735,
+            -0.0016781301605750378,
+            0.012249966178417542,
+            -0.025567673456449492,
+            -0.023488408901029257,
+            0.007732021927617865,
+            -0.006986715221001125,
+            0.01461433573263053,
+            0.00011845838735106802,
+            -0.005506714145045683,
+            -0.006827219009979501,
+            0.0005242424465709722,
+            0.002885486381705293,
+            -0.0042669493792680474,
+            0.021618593454103052,
+            0.00010034640948597114,
+            0.011602625527297699,
+            -0.023477647311478233,
+            -0.0162254255990209,
+            0.031352657169434826,
+            0.03529846802317018,
+            -0.0007059281329032357,
+            0.02568807107901636,
+            -0.00277252926066966,
+            0.010828396988798257,
+            0.00810821945842862,
+            -0.010307381937016405,
+            0.0003407672690264756,
+            -0.003768030765932326,
+            -0.0027367413382233317,
+            -0.012887542072043335,
+            0.0031930180018598015,
+            -0.014086667286395425,
+            -0.01504545079945656,
+            0.02010182276528891,
+            -0.002519095978589373,
+            0.011477055159845283,
+            0.006165264376334778,
+            -0.01139265727997407,
+            -0.016539552549751473,
+            0.01574258576985793,
+            0.05233886430069725,
+            0.022428487725773775,
+            0.044334901638531425,
+            -0.00706941862193002,
+            0.011585028247041065,
+            -0.016760013805097753,
+            0.001211362507398395,
+            0.048283299647824234,
+            0.015403290676022106,
+            -0.013446978976128664,
+            0.024015090204073844,
+            0.013753743307879434,
+            0.0398341544586196,
+            0.0034302789804899806,
+            -0.020152697770903245,
+            0.041614050886963766,
+            0.00392650113407049,
+            0.013465009394885544,
+            0.020487015678247938,
+            0.002430045518615804,
+            0.0035516244267459565,
+            0.02938659684630997,
+            -0.010142701862398358,
+            0.010335745659719125,
+            -0.033187389861872627,
+            -0.021792906328508686,
+            0.02532137764121119,
+            0.0005114146307176893,
+            0.033795722059780535,
+            0.007069260093915293,
+            -0.017628598898476375,
+            -0.017395163521872672,
+            0.021985468266342154,
+            -0.007987351575783343,
+            -0.025037973555114328,
+            0.002627649768366858,
+            0.008700784263932844,
+            -0.004480915144455101,
+            0.006798551384053197,
+            -0.014580806303786968,
+            -0.009741248878656092,
+            0.025066050175629597,
+            0.007330922027951371,
+            0.010472464570407973,
+            -0.024286928122882124,
+            -0.011159235410800293,
+            0.006107141408585794,
+            -0.04181412282877586,
+            0.008647396691893692,
+            0.00528510107024571,
+            -0.006647986735406744,
+            -0.002548771993161688,
+            -0.0011322362644692566,
+            0.00513401828632921,
+            -0.0035963203271502263,
+            -0.014021183815351008,
+            0.022993860073007393,
+            0.02875036422479212,
+            -0.012996574913859303,
+            0.014373647963439562,
+            -0.0093541571449835,
+            -0.019474535598737786,
+            -0.0233122546137944,
+            0.014269723193443946,
+            0.008784376430480408,
+            0.004317715767550436,
+            -0.008273153355003107,
+            0.0010203136295909875,
+            0.001909823045834306,
+            0.02406426043795662,
+            0.049377235716785324,
+            0.005265955549193235,
+            0.029166436381966566,
+            0.017136545810044906,
+            0.03882940870384413,
+            0.0007533463090350569,
+            0.016233986042907378,
+            -0.03311355052072762,
+            0.03082356484543373,
+            -0.0009431717656252606,
+            0.04077734731808,
+            0.0369941736537998,
+            -0.05477277258937169,
+            -0.014658721668336077,
+            0.028327060073149418,
+            0.010896122046930724,
+            0.01436699901738847,
+            -0.003307011398652852,
+            -0.01163199508062231,
+            0.029659278016495517,
+            -0.023936970669211955,
+            -0.010748756889752312,
+            -0.00767811402724157,
+            -0.010785978668072403,
+            0.009408061520238972,
+            -0.0038320809092061126,
+            0.038383104318077226,
+            0.0070413219383093,
+            -0.009684310528333753,
+            0.019154521368080882,
+            -0.006292471963504016,
+            0.01596901434992771,
+            0.0037167338451907566,
+            0.0019879266696342887,
+            0.03104390318244226,
+            -0.003561483519923973,
+            0.028949607591091022,
+            0.015156617705082505,
+            0.017473301120296506,
+            -0.0029321911929339165,
+            -0.017573294769065502,
+            0.027284173131869906,
+            0.009113069584708108,
+            -0.01971507323375489,
+            0.003667023499937692,
+            -0.0025339283534597176,
+            -0.0077697455666997945,
+            -0.010491575421861018,
+            -9.407001004405342e-05,
+            -0.00211520268689493,
+            0.02087980536494742,
+            -0.005582301812064227,
+            -0.0016376727058019977,
+            -0.0064225805186745605,
+            0.0052835419390265485,
+            -0.019662445004865352,
+            0.015117937189193789,
+            0.007575238218238001,
+            -0.018537762525347407,
+            0.014673223139419801,
+            0.019165766004277593,
+            0.0343151018851732,
+            -0.023949650834192267,
+            -0.0009279394158895472,
+            0.01730410035639405,
+            0.005928527001253011,
+            -0.018975612465080345,
+            -0.033873819454780706,
+            0.00825263613547798,
+            -0.01745801647450136,
+            0.027401724356781083,
+            0.008830481774058282,
+            0.0347823637819511,
+            -0.02309575756624861,
+            0.020559439488958053,
+            -1.2525886083543457e-05,
+            0.01593239963830551,
+            0.01455178268663934,
+            0.027993448238508672,
+            -0.017573784980205873,
+            -0.0379374065982424,
+            -0.0030450459335320053,
+            -0.0029968228865394214,
+            0.016851016821808874,
+            -0.013002468483615099,
+            -0.003337198735398279,
+            -0.0019044899095842511,
+            -0.029686441224288696,
+            -0.021487593939979652,
+            0.027796807698461906,
+            -0.04637191752173605,
+            0.0030176823995715597,
+            0.03281049040516897,
+            -0.030047693785808505,
+            -0.04246000386774685,
+            -0.0014603539776430917,
+            -0.03560639883119047,
+            0.0010602583020630973,
+            -0.01222013383414755,
+            0.029128085483749913,
+            0.007388529661657186,
+            -0.020518571527741426,
+            0.018184513437823,
+            0.0076594003031722,
+            -0.004161472492195519,
+            0.006100954284405263,
+            -0.005239699097592758,
+            -0.028136388241670696,
+            -0.015254134018585082,
+            -0.03037392857155718,
+            -1.6317930920712043e-06,
+            -0.002478230556191563,
+            -0.012845180214134591,
+            -0.024383682688014996,
+            -0.000484419407302015,
+            -0.014810859291916117,
+            3.958274758340876e-06,
+            0.05136363130801806,
+            0.02120810926469503,
+            0.005060247818396135,
+            0.012869784532979883,
+            -0.019665446839172217,
+            0.04743686110578455,
+            0.003514534022065111,
+            0.03583012040901262,
+            0.002062991366066978,
+            -0.02173965243546218,
+            -0.0005002960506349255,
+            0.0319890423992129,
+            -0.0016894669649440992,
+            -0.017249094049398075,
+            0.006136324758346181,
+            -0.009220681391996718,
+            0.00837047468537671,
+            0.012871598576797918,
+            0.01579543926540845,
+            -0.0061999233512262265,
+            0.014725777187858035,
+            -0.003927118678273384,
+            0.009336164668870259,
+            0.03122006739418818,
+            0.019771411252821138,
+            -0.024716950288097537,
+            -0.037736399088695305,
+            -0.012523788001500705,
+            0.001234987976751515,
+            0.034477269953960235,
+            -0.013836458205716058,
+            0.003837278407556882,
+            -0.011027628278972294,
+            -0.0054689562005572675,
+            0.005739209890610896,
+            -0.006985434270987544,
+            0.0015806421600885957,
+            0.012196603223058041,
+            -0.006859901421089723,
+            -0.025670287745499374,
+            -0.004011122451643543,
+            -0.023258896941014143,
+            0.012997299613183496,
+            -0.010144866692641839,
+            -0.006816231069049567,
+            -0.005213359047878286,
+            -0.0014167278420750282,
+            -0.0062535968890290545,
+            0.0006662897611595949,
+            -0.014236807237438571,
+            0.01012846684096945,
+            -0.00899852930003816,
+            0.0016620397335915663,
+            0.007838787681725669,
+            0.007725632848979133,
+            0.0033110678307931205,
+            -0.008206916937522372,
+            -0.008777095103563134,
+            -0.01631820698659747,
+            -0.032412484976539334,
+            -0.03000207979123948,
+            -0.02276516767125653,
+            -0.003870969029785218,
+            0.0034978865240734053,
+            -0.032594545526009384,
+            -0.020635972136797354,
+            -0.002367523739258502,
+            0.0030103817989158062,
+            -0.014587341442812854,
+            0.0014779462725289245,
+            0.007969995140846214,
+            -0.0029319878689507335,
+            0.019355924479361485,
+            -0.005448177691636249,
+            -0.021203447333791942,
+            -0.028956667864566195,
+            0.017851274292380093,
+            -0.015891501231440428,
+            -0.010070179095955473,
+            0.0064388768290910715,
+            -0.005633920123057214,
+            -0.03957042012546069,
+            -0.008945181709739484,
+            0.010992989857329521,
+            0.0035378401849368002,
+            -0.025936727040303652,
+            0.031903561853958325,
+            0.030132441728296204,
+            0.01845519141895959,
+            -0.017902241158092712,
+            0.04710441323879975,
+            -0.006739916600200837,
+            -0.012103058303430145,
+            0.016276540022040643,
+            0.023963592487909288,
+            0.028774476787161263,
+            -0.02643829153200483,
+            -0.005349178006882811,
+            -0.010545703050010008,
+            0.005354643441490569,
+            -0.02271588009302058,
+            0.008468528451368553,
+            -0.013601178278798316,
+            0.011576067943282815,
+            0.019227032134009135,
+            -0.013197207810056759,
+            0.02629307658052319,
+            0.0045263515787155985,
+            -6.067227645639678e-05,
+            -0.0037770406201685248,
+            -0.016287414859552752,
+            -0.012319957965328086,
+            -0.034674880061192896,
+            0.020297690806102293,
+            0.01889209267823917,
+            0.003953684662429981,
+            0.021845046751532583,
+            -0.049434011185276405,
+            0.02806744160678342,
+            0.0017849117483741536,
+            0.021763434864703356,
+            0.013889518910180847,
+            -0.001843068102127134,
+            0.013886691089114936,
+            -0.0032117229736863266,
+            -0.019971320584891997,
+            0.006079730961157539,
+            -0.006352535944493006,
+            -0.03316544192496317,
+            0.014736816574462923,
+            -0.03268321622529288,
+            0.013639134163846678,
+            0.015454170145821088,
+            -0.010170396823729307,
+            -0.021708948733883182,
+            -0.008303375382721087,
+            0.0010570623547459114,
+            -0.0014149688171089135,
+            -0.006849845444850982,
+            -0.04346483266888068,
+            -0.03020319096411844,
+            0.03124837430439466,
+            -0.0016720020681567616,
+            -0.03404132565569305,
+            -0.007209824644083462,
+            -0.02711263625315296,
+            0.004780610743996515,
+            -0.004212647822173262,
+            0.0025676935176963222,
+            -0.048750272936498695,
+            -0.01018434342955251,
+            0.011179560412168381,
+            0.031648611061279724,
+            -0.029298960703078986,
+            0.021370266671953315,
+            3.770954396468699e-05,
+            0.006629905234129844,
+            -0.00943632381701963,
+            -0.00484754585233744,
+            -0.012115337995959813,
+            -0.02607251387584821,
+            0.021868809148577698,
+            -0.0020587964818437345,
+            -0.0013502026980372474,
+            0.0341522429571705,
+            -0.00669289309893371,
+            -0.010113192232031375,
+            0.0005934017402521538,
+            -0.011327797122243258,
+            -0.017269539623150844,
+            -0.010467173785198161,
+            0.0005779464354975677,
+            0.0030406502310922306,
+            -0.024755136158113783,
+            0.004943446717510779,
+            0.04566273647510376,
+            0.013607936503018211,
+            0.02534200532143803,
+            0.017104750405207106,
+            -0.03792550084708064,
+            -0.04394052120951292,
+            0.0012600504352102988,
+            -0.005807815271334456,
+            -0.01944685298932453,
+            -0.01732108197375621,
+            0.0011677414026425442,
+            0.00790730598557849,
+            0.0037240231183698074,
+            -0.011427532982681041,
+            -0.02086964367350831,
+            -0.032539146090370744,
+            -0.0136797717535217,
+            -0.009464326388205681,
+            -0.007302204556505735,
+            -0.032703238152349344,
+            -0.015929357859286307,
+            0.026848214129378756,
+            0.01012098852614113,
+            -0.005646819862636667,
+            -0.0107564788588665,
+            -0.024118155140225247,
+            0.007122339033521904,
+            -0.0027183981945039814,
+            -0.009219030314883272,
+            -0.004672057262936543,
+            -0.016450690196817726,
+            0.02348093022809639,
+            0.009139210149358477,
+            0.024997129282118148,
+            -0.058533911093319214,
+            -0.014411932474734742,
+            0.014256038602455438,
+            0.0020188651815054508,
+            0.001417136751974749,
+            0.029836680491427268,
+            0.0028786535004858044,
+            -0.006526872714072965,
+            -0.02967220833325222,
+            0.028817295504169067,
+            0.027449694157944637,
+            0.01800200378294509,
+            -0.010560094796033294,
+            0.0033996190760546383,
+            0.006004691351249025,
+            -0.005679195734863024,
+            -0.011927786207717698,
+            0.0021260244673648285,
+            0.018998574211603905,
+            0.017174632765532803,
+            0.014659925822127794,
+            0.02455156987384756,
+            -0.005882590537857664,
+            -0.03911277764764406,
+            -0.009781110378157883,
+            -0.0119630929326396,
+            -0.004815125964940769,
+            -0.0026631770014339242,
+            0.008903827530107938,
+            -0.011824767329332145,
+            0.013088406716464123,
+            0.030264448923574067,
+            0.042592268021664714,
+            0.021292401951863188,
+            0.006150997237661769,
+            -0.005463801502425317,
+            0.0007399518998122848,
+            0.0038649546282459707,
+            0.011355675550395135,
+            0.035762872771750435,
+            -0.018147218572765546,
+            -0.019704502923991365,
+            0.0015558311499702044,
+            0.023543487977413267,
+            -0.007150197116420024,
+            0.02608209797565159,
+            0.023985156585794257,
+            -0.0003434024317322991,
+            -0.01271338268931938,
+            0.000985282363050309,
+            0.006710364567868783,
+            0.014540943712513688,
+            -0.02154205198718209,
+            0.004039619025717627,
+            0.001865104196024055,
+            -0.009756600776161738,
+            0.029738871754072076,
+            0.00039154552800346406,
+            -0.012981811693538832,
+            0.03825344802886,
+            0.11025963334575106,
+            0.001476339401950677,
+            0.0031187927813794386,
+            -0.031522639386920355,
+            -0.015113380587160163,
+            0.0021786482865071505,
+            -0.005948756393737655,
+            0.020082591515778573,
+            0.012751062800862873,
+            0.0055847042526916265,
+            0.010100961720464159,
+            0.038084609873482715,
+            -0.009203643420546616,
+            0.009804311800033327,
+            -0.01878272400403416,
+            0.017185477997947226,
+            -0.00972716681599207,
+            -0.00857276597428136,
+            -0.007641511960016827,
+            0.004167416313632853,
+            0.00984005584819001,
+            -0.004596999098606572,
+            0.018810908413859465,
+            0.002496310438422421,
+            -0.01613133688770545,
+            0.011505030137914962,
+            0.007437924384075958,
+            -0.005386885126779478,
+            -0.031755982499042136,
+            -0.013811586180238287,
+            0.01190667400285204,
+            -0.0013786018401773159,
+            0.0007177632889203917,
+            0.012342640512536284,
+            0.01205074763420124,
+            0.031217897495485192,
+            -0.0244324065837541,
+            0.0025071378510626696,
+            0.0070486374045588465,
+            0.0159322020159265,
+            0.01098799814774876,
+            0.00029844505852147526,
+            -0.015456787478044924,
+            0.027139764983358832,
+            -0.011978684728746428,
+            0.010378658252224036,
+            0.01146607671795495,
+            -0.04131658683485702,
+            -0.0009967932006190922,
+            -0.005916133731248287,
+            -0.015698754902727707,
+            0.013905118384587169,
+            -0.040163447416240027,
+            -0.005682084326086952,
+            0.004297114532998809,
+            0.010724201884417054,
+            0.0058749102353125955,
+            -0.017715270733111323,
+            0.009962853888582856,
+            0.00955216765250444,
+            0.0025392532243413593,
+            -0.006356691980285544,
+            -0.006218169970979339,
+            -0.008606942887730427,
+            -0.004280171880086814,
+            -0.015669721147884084,
+            0.0057481312044666884,
+            0.0077227094511494055,
+            -0.018751123856436715,
+            -0.01585864749751703,
+            0.009183099781375234,
+            0.006837438250956817,
+            -0.001445622524955126,
+            0.004828557942941401,
+            -0.010034012200008746,
+            -0.004543834537450213,
+            0.01097540631691429,
+            -0.009334158551770033,
+            -0.004804430123454511,
+            -0.02609985520241612,
+            0.013342299390862539,
+            0.011103136832190223,
+            -0.008261332966534102,
+            -0.006944131083745896,
+            -0.016931502665633764,
+            -0.0031318730206304824,
+            -0.0026508016484082988,
+            0.0016110884325639078,
+            0.017071266355203386,
+            -0.009906320666640836,
+            0.010877416311741906,
+            -0.03496707406301918,
+            -0.015357787514985604,
+            0.01622830705930841,
+            -0.01290003882907745,
+            -0.005625420341281716,
+            -0.009838333350984609,
+            0.0006911383780218238,
+            -0.007545081680638308,
+            0.006074752194220675,
+            -0.041320607445282646,
+            0.0035102345636816475,
+            0.018825308640738208,
+            -0.009723969339534365,
+            -0.020829421583878845,
+            -0.009055117268112308,
+            0.010487235049013567,
+            -0.015831829161845813,
+            -0.01780922120240258,
+            0.0016219998217122911,
+            0.010559783293374071,
+            -0.0055135690971065965,
+            -0.005160232484458576,
+            0.006163424486940105,
+            0.020247627989655837,
+            0.015220964429515309,
+            0.04337432385613251,
+            -0.011915355138717245,
+            0.009617347907919507,
+            0.027812934121040484,
+            -0.030652481946156104,
+            0.01037470561447702,
+            0.02599512377987521,
+            0.0037340561802882417,
+            -0.0013375196064018082,
+            -0.010197691376747194,
+            0.004161692215514214,
+            -0.012254402844820636,
+            0.008715552098450461,
+            0.004697361473725416,
+            -0.027978648968732336,
+            0.017196771853396124,
+            0.019448218659009996,
+            0.0038720194629308794,
+            0.0006995062900842749,
+            -0.01527901684789497,
+            -0.011825577013200781,
+            0.019194236643713274,
+            0.027690065919038755,
+            0.005460028121722145,
+            -0.015512497062157167,
+            0.02370643287056251,
+            0.012916257803027072,
+            -0.004973882005506947,
+            0.0022332716531575122,
+            -0.016984605526823043,
+            -0.01443316577897661,
+            0.030942241604983316,
+            0.011563751997976407,
+            0.01908030186367601,
+            -0.021614212853976106,
+            -0.003982310717524619,
+            0.019931579069642925,
+            -0.021029366993395197,
+            0.013006879711144009,
+            -0.019497570222553013,
+            -0.014067925104548129,
+            -0.010687576618198498,
+            0.02884494104893037,
+            0.009229770955288097,
+            0.026701705044769222,
+            0.012715371448003833,
+            -0.021444875656527786,
+            -0.021050591467198086,
+            -0.0027703267406452336,
+            -0.02721606030674425,
+            0.01688256721881862,
+            -0.055990128879151335,
+            0.0016563858333845547,
+            0.010911810464298247,
+            -0.008604898990983188,
+            0.004039521277648332,
+            -0.007858378479197747,
+            0.010021929060776957,
+            0.020926274809879138,
+            -0.02956229623532218,
+            0.01256060127924585,
+            0.01156331569664224,
+            0.0218688589850221,
+            -0.018861065515394883,
+            0.015339087155577661,
+            -0.00841001512858443,
+            -0.027396412391138852,
+            0.008323898095545889,
+            0.01763697170382491,
+            0.014160184414353919,
+            -0.011992345081843988,
+            0.004485633614765869,
+            -0.04000932898876895,
+            0.027999519977069676,
+            -0.016823500188831896,
+            0.002010511519234648,
+            -0.004537991252901591,
+            -0.0067219583064939085,
+            -0.008523354091034399,
+            2.682477256914193e-05,
+            -0.0005371719103087346,
+            -0.030920653652724776,
+            0.0012939076000836422,
+            -0.04144943703666512,
+            -0.02260198516292792,
+            0.013327519374598822,
+            -0.0012897465780583367,
+            -0.005059170593504863,
+            0.02999548027478466,
+            -0.02863650640718448,
+            -0.027272295036911975,
+            -0.014376307736824656,
+            0.03534844385341626,
+            -0.015358480569131909,
+            -0.000540068394391221,
+            -0.009287193719601908,
+            -0.02236594406404711,
+            0.01091078016405303,
+            -0.006469935602381157,
+            -0.0019272207412132534,
+            -0.01387843655423569,
+            -0.03972775413987396,
+            -0.005948332199741248,
+            0.019500395452013777,
+            0.0006122786247500248,
+            0.003656719755271774,
+            0.014473976840529128,
+            -0.0072278373928449675,
+            -0.0036623750417377524,
+            -0.0038809301345520044,
+            0.026929584793529297,
+            -0.022225170775945557,
+            0.009744440210592443,
+            0.004704883568603746,
+            -0.0005991305039095727,
+            0.017317686588588826,
+            0.000351879037187266,
+            0.0024030356296394478,
+            0.0029060795825935717,
+            -0.0035301907818594274,
+            0.02922907976800865,
+            0.0064070048516525,
+            0.0016291092834589405,
+            0.0001988403049139903,
+            0.004746570638478093,
+            0.01395197992587789,
+            -0.01069245020264236,
+            0.0157940052093005,
+            0.02865126209030693,
+            0.00021287181382353683,
+            0.018345429962063935,
+            -0.015921361819192573,
+            -0.036059435316760584,
+            0.019807131219096722,
+            -0.029642240399677812,
+            -0.003504127677138975,
+            0.022473005924273325,
+            -0.012903852741467797,
+            -0.0031750649075827082,
+            -0.01107957248024579,
+            0.027243132530764567,
+            0.023512079029549708,
+            -0.014856101416774535,
+            -0.015142539116300868,
+            0.004337433267586975,
+            -0.010210607143984678,
+            0.010939153927790107,
+            0.029382995339774957,
+            -0.011230337698862454,
+            0.011717621080350227,
+            0.001161341258722717,
+            -0.003778338045364591,
+            0.019300005069271462,
+            0.0027681684372537606,
+            -0.007401677666787755,
+            0.025628069381915978,
+            -0.008115857591022265,
+            0.013043669211084909,
+            -0.004672358463640539,
+            -0.012902494254220482,
+            -0.003106794914564588,
+            -0.017558145644151636,
+            -0.01225024815282763,
+            0.0011522376202485873,
+            -0.005528597019035285,
+            0.06711658458916447,
+            0.010930891527455397,
+            -0.017888423424597618,
+            -0.0059190199704595775,
+            0.00955567741445378,
+            -0.000966061435587513,
+            -0.009466328431452424,
+            0.027063767833815658,
+            -0.016888493246695876,
+            -0.009681104485277698,
+            -3.814655691584766e-05,
+            -0.0233149747136092,
+            0.0030596902889784496,
+            0.010827987186229144,
+            0.0012052973350181351,
+            -0.013994504803884577,
+            0.005318095523120499,
+            -0.004843763992616558,
+            0.0034188401938723923,
+            0.003678626789714963,
+            -0.0025749484272345573,
+            -0.005333294277998365,
+            0.02307826358450567,
+            -0.012791914224263883,
+            -0.008519982879775168,
+            0.01173424692936401,
+            0.001777123104204978,
+            0.003219553788359312,
+            -0.004564745566711077,
+            -0.002015107048152335,
+            -0.017123545909339455,
+            -0.011278616226036531,
+            0.023713473402287875,
+            -0.02733523566514418,
+            -0.00150943777243135,
+            0.017328382962199716,
+            -0.005705544203971564,
+            0.016650187392502975,
+            9.750950361601131e-05,
+            -0.02100066506382883,
+            0.00010090567939930427,
+            -0.024778741347883073,
+            0.010484659495766551,
+            0.013963183459158656,
+            -0.009531952698151101,
+            -0.020878167882780043,
+            -0.027881362421494915,
+            -0.010190031350698889,
+            -0.027246821338460653,
+            -0.0280692647056191,
+            -0.02020582179742232,
+            0.009778340348478517,
+            0.005715364307462615,
+            0.021929057840860695,
+            -0.006380523858228711,
+            -0.027552858236741976,
+            0.015095163660340326,
+            -0.013269344817111162,
+            0.008282918921343354,
+            -0.019991891695114384,
+            0.01754833974582726,
+            -0.014720830704852575,
+            0.02112161088830394,
+            -0.00041409266970033855,
+            -0.004490472334793758,
+            -0.012076561942010787,
+            0.003976913901514849,
+            0.008101357042612863,
+            0.00176590819486174,
+            -0.008977295353115363,
+            0.026242190932756135,
+            0.01707634775369835,
+            -0.007385093173634561,
+            0.01604386075569055,
+            0.031855080733360075,
+            0.014762206220119051,
+            0.038860660002328094,
+            -0.007238177370595944,
+            -0.009237098188374397,
+            -0.02764495845127417,
+            -0.004407870047633689,
+            -0.006820430763682645,
+            -0.004516837004332235,
+            -0.015251921783911087,
+            0.03164122113689215,
+            -0.010371400055822811,
+            -0.01686972159902524,
+            0.03979035122593764,
+            -0.021846728842184655,
+            -0.007084905708666271,
+            0.008642771511425465,
+            -0.013996636040226265,
+            0.03176661229383477,
+            0.007275612529363987,
+            0.018452818352301472,
+            -0.015039890816023917,
+            -0.015988701055097997,
+            -0.0019911190487748244,
+            -0.002806138139429132,
+            -0.020828646560807865,
+            -0.0063794261154720665,
+            -0.01614570050728432,
+            -0.0026319562705484634,
+            -0.00570753504957681,
+            0.015917008930969468,
+            0.0013916917451795444,
+            -0.015420032290910069,
+            0.04488171544231248,
+            0.005033499525006087,
+            0.022768662319611596,
+            0.010777933460735837,
+            0.012372342667053587,
+            0.015819823672091563,
+            0.006759824420801962,
+            -0.0001668726099910387,
+            -0.02211654164646909,
+            0.004261169256891085,
+            -0.005641717035723983,
+            0.0041005490092982915,
+            0.02012094437568505,
+            -0.024436624754739972,
+            -0.029708548399946657,
+            0.024493078535528394,
+            -0.01833101179919427,
+            0.0030908990956393764,
+            -0.0031025537219982234,
+            -0.010059653447038065,
+            -0.011612233298907742,
+            -0.03293899346571458,
+            -0.0005929867206193162,
+            -0.027438115397399914,
+            -0.03635734152790657,
+            -0.006287687204976032,
+            -0.007850474352061472,
+            -0.003574235211373211,
+            0.02449669755605501,
+            0.004269460596739739,
+            -0.029959856511873897,
+            0.01782698044631415,
+            -0.013252733005984911,
+            -0.013342147906120882,
+            0.0029581361690999564,
+            0.004786570391076097,
+            -0.0027618259744494854,
+            0.010720003765859173,
+            0.007898965393358715,
+            0.007358694001293221,
+            -0.010710502803266644,
+            -0.004049671482838962,
+            0.0013263710826446039,
+            0.0030943253037901135,
+            -0.013325799375903166,
+            0.030553973934607993,
+            -0.018812559793353154,
+            -0.004819166852190956,
+            -0.0006499911473215254,
+            0.0034556893023524674,
+            -0.0110929170801761,
+            -0.014862886414702483,
+            0.021004665462647988,
+            0.0031088550726355926,
+            -0.012570481398485388,
+            0.0096952264731202,
+            -0.0018868240406799254,
+            -0.0024767709620225923,
+            0.01176108203579327,
+            -0.002080730326597438,
+            0.014924914157541781,
+            -0.011040818699356567,
+            0.01490073610670544,
+            0.0034649445207609085,
+            0.008645740731109337,
+            0.003603641937848282,
+            -0.01905231764119889,
+            0.009780124784430865,
+            0.016356631955067272,
+            0.013415470760506568,
+            -0.029496530287207723,
+            0.008881639987471632,
+            -0.005975399013716009,
+            -0.009870580036511721,
+            -0.024372445588075428,
+            0.002590423942920311,
+            0.028960961301291905,
+            0.012044179251559706,
+            0.007934516374398097,
+            -0.0077949340827692054,
+            -0.005480592942515702,
+            -0.019923500717513334,
+            0.009041031662948522,
+            -0.01095651774810384,
+            -0.025157908389819465,
+            -0.007578712371715301,
+            0.03430545904801765,
+            -0.010740874257824011,
+            0.01343875386664532,
+            -0.005526757815283282,
+            0.02436552949831661,
+            -0.0027538610388853284,
+            0.0013690344627799137,
+            0.008416180197734094,
+            -0.006091433734127276,
+            -0.001708607133233843,
+            0.013349212286052506,
+            0.041066250931277824,
+            0.0038375614886832564,
+            -0.0050950709988837216,
+            0.01149207527692021,
+            -0.00326389523728132,
+            -0.005712157453687712,
+            -0.00309579814793649,
+            -0.025491974096530076,
+            0.015133356727314608,
+            0.0010745378801251735,
+            -0.017162228172591873,
+            0.021785292616181687,
+            0.027891598422935242,
+            0.013262622112280296,
+            -0.0178471114368855,
+            0.014456217536546996,
+            -0.009278392107874806,
+            0.01629461595238019,
+            0.02423194076020779,
+            0.010067296472207578,
+            -0.011865648674106681,
+            -0.0070073298858753805,
+            0.008985354045706703,
+            0.008096016282907443,
+            0.0004236756235618935,
+            0.020370823466310733,
+            -0.028334250141960886,
+            0.01483270667238479,
+            0.007137078270349882,
+            0.03314918453195744,
+            -0.0006231629401653488,
+            -0.014397909717816884,
+            -0.008918359605259833,
+            -0.002915473990900902,
+            0.011759189078587877,
+            0.003523992368256188,
+            -0.0006853609664806012,
+            -0.004749148984168426,
+            -0.014181883848149463,
+            -0.006778789302277201,
+            0.012376777107762334,
+            -0.0031646329407784245,
+            0.0030514749941358095,
+            -0.0007326603315688041,
+            -0.007811941797535734,
+            0.015661953163847418,
+            0.001669500869963878,
+            -0.0007143628544145883,
+            0.0016729300148826723,
+            6.604560293656475e-06,
+            -0.00467413979302959,
+            -0.019114577369302277,
+            0.008922356418978664,
+            0.00867683974577237,
+            -0.006578536023513047,
+            -0.006467824108991175,
+            0.009324373505311057,
+            -0.01432602608910974,
+            0.01852478469282692,
+            -0.0061254590933158826,
+            -0.006603842196861297,
+            0.00014277641689784565,
+            -0.012328738339684425,
+            -0.003642202698645936,
+            0.0034895681343535914,
+            0.0051453312116433265,
+            -0.000908593523739809,
+            0.0036062764245630334,
+            0.003940522150829955,
+            0.01083964694261132,
+            0.002824079422783228,
+            0.0015266576108775834,
+            0.007280263953807289,
+            -0.01039293741283943,
+            -0.007403606843286745,
+            -0.004993622657038095,
+            -0.006341662331986892,
+            0.007288958567606623,
+            -0.004364975055335228,
+            0.010457360052582658,
+            -0.0062292104145735515,
+            0.020471836369529147,
+            -0.006133009423930811,
+            -0.0024055488697334888,
+            -0.017695183455939345,
+            -0.007234478249839231,
+            0.007156119488831356,
+            0.007878044165936736,
+            0.009359297231370758,
+            -0.04487830669749157,
+            0.005038301058510894,
+            -0.0054192943462008595,
+            0.02941616633783806,
+            -0.009828941844770915,
+            -0.004335130343692854,
+            0.016067824555917796,
+            -0.019100842323053376,
+            0.0037066021889614513,
+            -0.013965982913833292,
+            0.003695036285467973,
+            0.0034558765116311525,
+            0.014205693020177169,
+            0.031307058208919394,
+            -0.007482180869231769,
+            -0.009383819319265733,
+            -0.006384083847148052,
+            0.026967380724567013,
+            0.018643952476557566,
+            0.023548246269020395,
+            -0.011885146764471029,
+            0.00960518654541979,
+            0.001357650512536872,
+            0.018271473493526338,
+            -0.009663920325675169,
+            -0.011163336807225407,
+            -0.000516828412255004,
+            -0.010288703356943627,
+            -0.0032111641654394615,
+            -0.021092460896394025,
+            -0.0033456309643941965,
+            0.025876107752976232,
+            0.0038584058642743987,
+            -0.0036689325941524813,
+            -0.015624044740149702,
+            2.2644755460272697e-05,
+            -0.011931276845804481,
+            -0.0055870010742218364,
+            -0.015697330226950804,
+            -0.028927689083316704,
+            -0.016192017705678192,
+            -0.002914436546908901,
+            -0.017224654082473787,
+            -0.009460990414684924,
+            0.002071634636546831,
+            -0.015533259213802986,
+            0.004918377471991415,
+            -0.015669642231243106,
+            -0.010915412245035698,
+            -0.00046472075835247,
+            0.004672390433470177,
+            -0.004682173537415046,
+            -0.024109027987987815,
+            -0.013628794248618615,
+            -0.0022054473551170926,
+            0.006500170428351522,
+            0.010016298501187265,
+            0.008324628078950177,
+            -0.0069374343736907625,
+            0.011201167788973098,
+            -0.009746811993090597,
+            0.010155637867574787,
+            -0.02773759960959338,
+            -0.04317847702979315,
+            -0.000791583637601122,
+            0.009241551203900427,
+            0.001979688190287368,
+            -0.01884488538245213,
+            0.0038489955050695704,
+            -0.011280060242688173,
+            -0.00847814532468811,
+            -0.007522173019881079,
+            0.01573091287228822,
+            0.004138348969142638,
+            0.013463997539337106,
+            0.0042065270170965705,
+            0.0285467221116185,
+            0.012635383326761538,
+            0.01030078781909351,
+            -0.003092067773698843,
+            0.004763648809248413,
+            -0.0022835042548997662,
+            0.009077632103272405,
+            0.02331475506951857,
+            0.0076225652102231605,
+            -0.002326207128504976,
+            0.00409858249509217,
+            -0.0005698076375460636,
+            -0.005442088641415082,
+            0.018565262151290427,
+            0.02915691852605287,
+            -0.0069811288298562155,
+            0.008120076286611811,
+            -0.0009110258445641983,
+            -0.009900329477406562,
+            0.0038288023984988134,
+            0.005765264445222657,
+            -0.014230725839572923,
+            -0.0024175374911696653,
+            0.009367987848760041,
+            -0.024664375361024386,
+            -0.004113757123212052,
+            0.020371264792326226,
+            0.0024813232005061487,
+            -0.016599537494340785,
+            0.03488073638324099,
+            -0.003709931703179228,
+            0.003010315958426392,
+            -0.00944047118505417,
+            0.016362178907316286,
+            -0.017989464717541103,
+            0.0005390907615384354,
+            0.013131172782591922,
+            -0.005251614069791733,
+            0.005783823857695169,
+            0.003744510952053123,
+            -0.010417674399178015,
+            -0.010824994224858631,
+            -0.0033951133363820767,
+            0.01085129781083312,
+            0.002642227227069058,
+            -0.024446190958178605,
+            -0.017731767444262785,
+            0.019381370848622046,
+            -0.01001206050737818,
+            -0.008290790009659911,
+            0.0009331678418736739,
+            0.0029270629970877303,
+            -0.004637651319894003,
+            -0.004290917518172777,
+            -0.003574121539057569,
+            0.009121344354139133,
+            -0.009485343005168732,
+            0.01637203851901565,
+            0.004043708359084711,
+            0.01843096334879628,
+            -0.011588935298420727,
+            0.005808551475129831,
+            -0.001368952044937682,
+            -0.007242291047654279,
+            -0.019004856431577073,
+            0.0059537808323522675,
+            0.0073777483129045685,
+            0.016051495408653346,
+            0.011060627996179347,
+            0.018569088420830965,
+            0.011345955812555888,
+            0.0039209921807600205,
+            0.001180658155391396,
+            -0.011006781869601802,
+            -0.016371246428912994,
+            -0.014232536816028922,
+            -0.007416908142074357,
+            -0.019561375671062513,
+            -0.0013253236527051844,
+            0.017798265625542643,
+            0.014528038148955117,
+            0.017704367192118375,
+            -0.0048389172759564884,
+            0.016018290057587425,
+            -2.7789810195444745e-05,
+            0.0009215076153434622,
+            -0.005897137279173464,
+            0.001794778099552753,
+            0.016314258230521846,
+            -0.003313666704140227,
+            0.0063152813149371136,
+            0.0002935873451081568,
+            0.01645203621687452,
+            0.022650990579075773,
+            -0.018899997585540014,
+            -0.0014442758993479705,
+            -0.006467599303099793,
+            -0.0021225224281092863,
+            -0.019894752041664215,
+            0.020266472457121338,
+            0.014019662562577204,
+            -0.005002061958298289,
+            -0.00943116174529331,
+            -0.004354703314807638,
+            0.02678118874395904,
+            0.020206530983694557,
+            0.006044260997975214,
+            0.026222728100119382,
+            0.01976601108150248,
+            0.01984865298844998,
+            0.011918890024498122,
+            0.009052182076567042,
+            -0.013092243192366595,
+            0.018358346459726883,
+            -0.01681770553983246,
+            0.0008504895530001561,
+            -0.003223133403363917,
+            0.002743153478851782,
+            -0.007937179998086906,
+            0.013021430914653432,
+            -0.00030955362267617273,
+            -0.025935204965869492,
+            0.007680669831594167,
+            -0.013183173230049736,
+            0.001375652741144253,
+            6.07690288088255e-06,
+            -0.001857629339163609,
+            -0.017652365850892467,
+            -0.0082709435899296,
+            0.0008448530375262078,
+            -0.01435595518048752,
+            0.023866770609514233,
+            -0.026826645910954506,
+            -0.02140002089501948,
+            0.00988141127158139,
+            -0.004436844338114915,
+            -0.012632978351515892,
+            -0.006972217855086974,
+            0.021864041851582767,
+            -0.005139891275508354,
+            -0.016596866242234234,
+            0.014168148679282804,
+            0.0005869000756971533,
+            0.00412958763467087,
+            0.0012364712526909765,
+            -0.008001798306987228,
+            -0.0030416500521329983,
+            0.006860667487819997,
+            0.028693213258586023,
+            -0.011895509688811421,
+            0.0016632663352826491,
+            -0.013799461077520309,
+            -0.0037922015693749754,
+            -0.02974150200357857,
+            0.027752493864472517,
+            -0.011830981692069491,
+            0.01989293240244598,
+            -0.004540523332199857,
+            -0.007667738626842118,
+            -0.021173122202316107,
+            0.035703796333655495,
+            0.004594443089494466,
+            -0.0006664409717898689,
+            0.007256660335524136,
+            -0.047588498838074554,
+            0.01105469709139337,
+            0.006498789757276137,
+            -0.023261890747840144,
+            -0.000964111432535798,
+            -0.003995007813335293,
+            0.02143067593410374,
+            -0.019461700447829744,
+            -0.012777737727876326,
+            0.0021024453304811634,
+            -0.021925886491102078,
+            -0.007056889535073896,
+            -0.0340604475320119,
+            -0.006137802768771592,
+            0.006503135515471076,
+            0.0038385769709340417,
+            0.0020799029890387023,
+            -0.021308499975514292,
+            -0.0309544649831362,
+            0.013308972238482328,
+            0.01651668347077017,
+            0.010007831710070633,
+            0.01200701454477496,
+            0.0067050757442456005,
+            0.011425518484475204,
+            0.00865023114755134,
+            0.00974582806570851,
+            0.008884712478547614,
+            0.039109252942848706,
+            -0.016122498485808056,
+            0.008671479031657285,
+            0.0005699279397862279,
+            0.02156543966774511,
+            -0.02391743870774471,
+            -0.009877938786506814,
+            0.015475531666380014,
+            0.003045729682805746,
+            0.030465388674890067,
+            -0.01511217342930631,
+            0.005735960079359902,
+            0.0030955813611916066,
+            0.003343849304714193,
+            -0.017685941897160877,
+            -0.019795694199965495,
+            -0.015889472669342385,
+            0.0005064000414834533,
+            -0.0007719666713392906,
+            -0.012817517834744753,
+            0.01370266779873307,
+            0.0029457043321269045,
+            0.004380574233134056,
+            0.003997906783171028,
+            -0.02610528890228495,
+            0.0033844277218836307,
+            -0.008397899012048187,
+            0.013703609594527308,
+            -0.0012702392316682096,
+            2.036849934035612e-05,
+            0.007261982817787643,
+            -0.00406175072535136,
+            0.018353316401139956,
+            -0.007206833180919801,
+            -0.004174154024217347,
+            -0.004097578846173991,
+            -0.00573702992914836,
+            -0.0017184308802689716,
+            0.022375713882789285,
+            -0.008746034802967245,
+            0.032729322137817145,
+            -0.014835515993879445,
+            0.01049714579786462,
+            0.013344412726432523,
+            0.006468296817982878,
+            -0.0036641997047404426,
+            0.0003384623491490165,
+            -0.0035345323089769998,
+            -0.006579417470328358,
+            0.015682816949756224,
+            -0.018934655843627496,
+            0.006860259740057359,
+            0.03727162736659208,
+            -0.002536806673033066,
+            -0.00615871353663963,
+            -0.008853057427430264,
+            -0.012968501800661,
+            0.028268388127570685,
+            0.005935705363468247,
+            -0.0013092005585187737,
+            -0.03024974000885413,
+            0.015180604705329429,
+            -0.0037501687534873948,
+            0.0016387074669205704,
+            0.00860407823411825,
+            0.00044285387135976006,
+            0.020127437993717013,
+            -0.0011272951769363546,
+            0.021290480413673304,
+            0.010032359217391118,
+            -0.0010666180914844765,
+            0.011529468490024128,
+            -0.036558434912810016,
+            -0.008579049583792913,
+            0.0038048395503202733,
+            0.019444233855445248,
+            -0.0006623807457136332,
+            -0.015853103419331053,
+            -0.004242407392496502,
+            0.009029483146616182,
+            -0.01059695082306233,
+            -0.013492380798775181,
+            0.016259633969459952,
+            -0.019096544374819852,
+            0.011560512412214506,
+            0.008617334373339341,
+            0.02512750386847096,
+            0.0041202753709624245,
+            0.002613588241299414,
+            0.0035069606381226207,
+            0.006602772841623022,
+            -0.002625433099855357,
+            0.011789793979724633,
+            -0.011592113209370399,
+            0.02102479518464762,
+            -0.008294039267346376,
+            0.007068661518691129,
+            0.0032541671568032306,
+            -0.022166914870027622,
+            -0.018234159989683987,
+            -0.0023642465680105544,
+            0.021312775708470108,
+            0.006016177710817625,
+            -0.012618763344460982,
+            0.0055411096806088885,
+            0.004690341026678943,
+            -0.00547301082703577,
+            0.009118596747082881,
+            0.024398283179066196,
+            -0.0028098889343668772,
+            0.01068095200128235,
+            -0.007137857371825465,
+            -0.027832295014375272,
+            -0.007421428223393234,
+            0.021707007996690945,
+            0.00020946296332069377,
+            0.016793613517175475,
+            0.014396690762872469,
+            0.012889706504719996,
+            -0.00488698200749025,
+            -0.005354238088985329,
+            0.018317372626432005,
+            0.014340608961907827,
+            -0.0048105230366691145,
+            0.001495716612807014,
+            -0.0018529641162211795,
+            -0.017240719704808234,
+            0.026933447120216628,
+            -0.024232238634425658,
+            -0.003438477892192365,
+            0.00892792234782449,
+            -0.012586276728773793,
+            -0.004923824123059311,
+            -0.023468301644204742,
+            -0.02117100775850015,
+            0.014296432515879623,
+            -0.003314359969711106,
+            -0.01750062790930347,
+            -0.006856892040084997,
+            6.621834660421394e-05,
+            -0.003983545180909168,
+            -0.010826144842427663,
+            -0.0028995230412062357,
+            0.0018029406642546938,
+            -0.01286603500476852,
+            0.00937188453692113,
+            -0.00635685147895941,
+            0.010242542894487791,
+            0.0062482230941642404,
+            -0.022974002980633818,
+            0.005235079251883088,
+            0.020160136045340955,
+            -0.008953506137454955,
+            0.0007027144749750591,
+            -0.015318432934167311,
+            -0.006155069535076527,
+            -8.304773776972749e-05,
+            0.010665400866776814,
+            0.013561082526851403,
+            0.019453190615040503,
+            0.008724224096424808,
+            0.017716608184445207,
+            0.0035170519664773133,
+            -0.013169473712817174,
+            0.009862672596286218,
+            0.011645070824075182,
+            0.011745045118639314,
+            0.011761235635339118,
+            -0.0051506969158396755,
+            0.00628898514628358,
+            -0.0049857719357619505,
+            0.008525048536699957,
+            0.027507594807546505,
+            0.0013209667681023782,
+            0.017047984110417937,
+            0.016368796085341072,
+            0.020105811276523177,
+            -0.003759327296302425,
+            3.4626946806238876e-05,
+            0.00035688794161846735,
+            -0.001063984476108484,
+            0.003291497239446139,
+            -0.018279629509070614,
+            -0.01135428882896519,
+            0.017033095140582537,
+            -5.492833742539044e-05,
+            0.024685530931127835,
+            -0.0010090516352729089,
+            -0.0184210790691242,
+            -0.006072072044463014,
+            0.0019766255786695347,
+            -0.0143465314042477,
+            -0.00209268265697904,
+            0.008867797876246725,
+            0.02090777280180668,
+            0.007785784409274212,
+            -0.014694559759226521,
+            0.03266241399157856,
+            0.017596572028431588,
+            0.02301544904486291,
+            -0.02019106568899837,
+            -0.008116927224141509,
+            -0.007678137378694171,
+            0.020318874678520133,
+            -0.0068566512882342135,
+            -0.002510061555870801,
+            0.021473513752723834,
+            0.006645783892214728,
+            0.011211922100954015,
+            0.012286872605920514,
+            0.0003084092164612384,
+            0.0028492628853667647,
+            -0.015161921733689642,
+            0.011855756525919763,
+            -0.005289433905822296,
+            -0.006904878573181685,
+            0.021517401941638887,
+            -0.010444313612915583,
+            -0.009104739247715266,
+            0.005434542886627656,
+            -0.011460046711120444,
+            0.013324525803587403,
+            -0.0016717915349416276,
+            -0.013225789210352699,
+            -0.014868291792978007,
+            -0.004059908595020105,
+            -0.017086039690741954,
+            0.018935544362274584,
+            0.006810606440007176,
+            -0.0016747502037076753,
+            -0.005949836930379681,
+            0.004577809518378731,
+            -0.002387527459083855,
+            -0.0006415824062693734,
+            -0.015923707971787795,
+            0.0035047065077239213,
+            0.009700583796235219,
+            -0.0031324165789907116,
+            0.010643108490671052,
+            -0.013353561795075988,
+            0.033779753889958034,
+            -0.003236875179603599,
+            0.0033068852921624326,
+            0.005106828066670305,
+            -0.003034255184307467,
+            0.001481657575335246,
+            0.013519816427440722,
+            0.009066434673988428,
+            -0.008339461460990311,
+            -0.0009211583209818409,
+            0.013497039042975171,
+            -0.018668722901146233,
+            0.0013796912320793106,
+            0.013796986913155082,
+            -0.015849997808276967,
+            0.0049427770053857125,
+            0.011400118227502903,
+            -0.00639345486098034,
+            0.015340608222895147,
+            -0.01371015934577743,
+            0.007008681477695071,
+            0.009775206231977338,
+            0.028490233702896408,
+            -0.01966685875736091,
+            0.01928758485650401,
+            -0.025635008258084145,
+            0.031283028009357144,
+            0.012122242939206308,
+            -0.005796910058436286,
+            -0.025521126197222552,
+            -0.011593941553769327,
+            0.01283377684428954,
+            -0.021686338476615465,
+            0.0014883363235708173,
+            0.004930128822274494,
+            -0.013589466747020829,
+            -0.014766714233107183,
+            -0.01202283835955618,
+            0.011738578705966321,
+            0.0006399892787922201,
+            0.01589633486439762,
+            -0.001565909739326631,
+            0.006195453613357957,
+            0.00799709051068248,
+            -0.007262779704915372,
+            -0.006717513726969218,
+            -0.01982600498555257,
+            0.01452043414139846,
+            0.004256754663990435,
+            -0.005262312998149605,
+            -0.0224549938750886,
+            0.009397743096285409,
+            0.017707219262338467,
+            -0.005753255460862127,
+            0.013047488144162397,
+            -0.03383526087792666,
+            0.0009452367103428395,
+            -0.005822570832057675,
+            -0.0027621026437538738,
+            -0.011741384193082396,
+            0.005215022845011562,
+            -0.015301048137828595,
+            0.002594097201808009,
+            -0.006446736651739076,
+            -0.016963075831388622,
+            0.002297125878220545,
+            0.02099207084363147,
+            -0.0032941933998622765,
+            -0.0028818889706579017,
+            0.01784874229400347,
+            0.008499720521713671,
+            -0.01436426626087195,
+            -0.004000179664908013,
+            0.006397941984552902,
+            -0.018873859156915206,
+            0.0011599963694687036,
+            0.02198209878751397,
+            0.012652203334020664,
+            -0.026378312815268385,
+            -0.0016397712646032201,
+            0.013833745513458322,
+            0.008230921549345205,
+            0.008667892596847433,
+            0.0040942632709300665,
+            0.014593014365768302,
+            -0.006350891941908016,
+            -0.001374093276890415,
+            -6.3005496707736715e-06,
+            -0.006943874751919358,
+            0.011912879044321251,
+            0.003688355066059739,
+            0.040741622477833674,
+            -0.010639328654064204,
+            0.011006744363538067,
+            -0.01080331270092793,
+            -0.004477468739500037,
+            -0.004503151926350927,
+            0.0027311865257027917,
+            -0.013139063275489816,
+            -0.0001322676781706513,
+            0.0021051078165230533,
+            -0.0019788024041678294,
+            0.00649268452600941,
+            0.003305903531478554,
+            0.013176851471483939,
+            -0.017229299828998786,
+            -0.0024884474570636707,
+            -0.009910839696360717,
+            0.019347666346683084,
+            -0.010724923526200773,
+            0.01051871403061451,
+            0.03256893707453101,
+            0.013242686936629418,
+            0.00022445440868234016,
+            -0.014947304198470308,
+            -0.008836459838385262,
+            -0.008992977994567497,
+            -0.01739161854615299,
+            -0.012908166671408042,
+            -0.022654409684101404,
+            -0.00294050967213434,
+            0.006390611639719366,
+            -0.0010555036392130776,
+            -0.002785796833161689,
+            -0.027340941664129582,
+            -0.00792516100590229,
+            -0.03210910565574123,
+            -0.013679170743667392,
+            0.0056845041078574104,
+            -0.021558178294402857,
+            0.005195998493021035,
+            0.029445968279023734,
+            -0.008256723194649932,
+            0.018403154314213917,
+            0.006923789005793937,
+            0.011433558571124386,
+            -0.0022324062216246424,
+            -0.011870704203994723,
+            -0.011035736680889242,
+            0.00032924465177187114,
+            0.003455564411617082,
+            0.004938023824535148,
+            0.00916834019939669,
+            -0.01732811783799426,
+            -0.010720420451770168,
+            0.00025951769018448803,
+            0.01732173296763328,
+            -0.0036441011455210317,
+            -0.014654182169779525,
+            -0.009890543675136605,
+            0.00934755871928037,
+            -0.017560953324850772,
+            0.0028803276180087,
+            0.008144645056336303,
+            -0.006844751836577803,
+            0.009686357271083834,
+            0.00910269542748879,
+            -0.003634975911996508,
+            -0.01125116892179341,
+            0.015123292946443665,
+            0.011417498955214564,
+            -0.0005683452620824292,
+            0.009547451569266946,
+            0.0031907584166949686,
+            -0.004914343407341038,
+            0.0004946517266559977,
+            -0.013981094295278457,
+            -0.008438177812073363,
+            -0.009698303451884585,
+            -0.0004873949117926506,
+            0.008310275049300703,
+            -0.011244097579407173,
+            0.002589333020166005,
+            0.021677389796728952,
+            0.0016179416889548232,
+            -0.011231602157838225,
+            -0.0045296258349024725,
+            -0.03198651985714177,
+            -0.005752566699226788,
+            -0.017524199510403467,
+            0.014980719037744585,
+            -0.002529640898744345,
+            0.01028755110586937,
+            -0.0021848874276835985,
+            0.015817733088597956,
+            -0.005450667684071798,
+            0.012212046004369696,
+            -0.0008714643856749736,
+            -0.004560548606687623,
+            0.006746171236132563,
+            0.0034423786329218276,
+            -0.0009995673054631,
+            0.05643134636651925,
+            0.010296478797092773,
+            0.014377556149215558,
+            0.013825497434998718,
+            -0.017485595187900386,
+            0.013330532940316836,
+            0.013333976202885402,
+            0.012417102160127097,
+            0.004536479576179834,
+            -0.017256228281240734,
+            0.024337236913894427,
+            0.012755805964855683,
+            0.009622694631579495,
+            -0.0018516129725815202,
+            -0.0015877067248695194,
+            0.019827793457522933,
+            0.018266833138588977,
+            -0.009808363592736615,
+            0.011083663940891296,
+            0.004458465015627857,
+            -0.01068421391840819,
+            -0.0050807436529387556,
+            0.024563333063926475,
+            -0.01325209293057542,
+            -0.016539133573613984,
+            0.0059168302415277515,
+            -0.02488439856049784,
+            -0.029755917745609557,
+            0.001920092403971772,
+            0.006364691645397147,
+            0.002128416061225683,
+            -0.0021827593501562285,
+            0.00636889353893684,
+            0.01505350329221805,
+            -0.02223481012881443,
+            -0.01913380824474348,
+            0.004876437698740333,
+            -0.0031323018872010133,
+            0.015711985535422395,
+            -0.02142499388661488,
+            -0.011646214273628826,
+            0.003244037052013272,
+            -0.006889665577990765,
+            0.004421548112756594,
+            0.004704830748153097,
+            -0.004955793799612646,
+            -0.005239972361031197,
+            0.0007828188358756139,
+            0.019667585310240637,
+            0.017115475760556154,
+            0.02008255444898991,
+            0.012206187649127842,
+            -0.004915394518125827,
+            -0.00997481157928988,
+            0.002664275191354035,
+            -0.017453428141032505,
+            -0.005114569347643741,
+            0.006659824028894164,
+            0.013233844584497872,
+            -0.023794827807567007,
+            -0.007354308951883939,
+            0.012842257789962674,
+            0.011294252132532885,
+            -0.00824836528452984,
+            0.004256336058369505,
+            -0.011193345143979632,
+            0.009248664100690892,
+            0.010363170772580968,
+            0.010981447978991254,
+            -0.01250390246078834,
+            0.008163162620777558,
+            0.002528845973804195,
+            0.007003579605495149,
+            -0.005812703866552746,
+            0.008764871082692903,
+            0.013124123077340675,
+            -0.012212447674686537,
+            -0.0013940507123620205,
+            -0.001544873017903471,
+            -0.020493969663982807,
+            0.005782189204516662,
+            0.007033733269347455,
+            0.012878926833341835,
+            -0.008338295354334415,
+            0.015844568159504493,
+            -0.013423240193733334,
+            -0.00468920516522183,
+            0.0063319500880227525,
+            -0.004924778574104498,
+            -0.012053197667708071,
+            0.0017742020845283848,
+            -0.0035640186446405536,
+            -0.008666749197879684,
+            -0.0013894177719374007,
+            0.0016400693271668141,
+            0.004501038858023909,
+            0.02043221461176279,
+            0.001901506572332553,
+            -0.0056076433740056926,
+            0.019660750419391273,
+            0.003538773720733224,
+            -0.0018659146421562808,
+            0.005354994676719758,
+            0.0006789623595561136,
+            -0.0021516503031452773,
+            -0.0027566542393029298,
+            -0.010552067426241374,
+            0.01699044655141112,
+            -0.01883386136434379,
+            -0.006428123254736374,
+            -0.012514829515253224,
+            0.006923832110523631,
+            0.023065674483579466,
+            -0.0031719793455512423,
+            -0.006131653355136594,
+            -0.015640226612880566,
+            0.003589774262421098,
+            -0.07234225978543331,
+            -0.010636515019878534,
+            0.005436135654579842,
+            0.030362090167699224,
+            -0.008735845879763853,
+            0.0012226978854869956,
+            -0.00802187367908004,
+            0.010460901462092434,
+            -0.0024797678489701584,
+            0.005049311815982583,
+            0.0037643360764659196,
+            -0.003379308997651945,
+            -0.0029803517660911536,
+            0.0047612984744651015,
+            -0.010726918354479238,
+            -0.018870259480662628,
+            0.0028549213021684043,
+            0.025034678854420084,
+            0.009004274035878184,
+            0.004138885468102832,
+            0.009765532829693857,
+            -0.008537379113922283,
+            -0.0014902115840393893,
+            0.008462743183234337,
+            0.007328272099011593,
+            -0.009076904245691212,
+            -0.0008002047467975742,
+            0.008347212876126893,
+            -0.021799374855548816,
+            0.0020253943807330738,
+            0.012246389250647798,
+            -0.008912465890948629,
+            -0.006244789880293639,
+            0.002413038795382646,
+            0.004202974630810528,
+            -0.004339960146144246,
+            -0.009919301413657872,
+            -0.023862183602380987,
+            0.006400371798756263,
+            -0.007410957922309058,
+            -0.0001606112434129549,
+            0.0031662502228666104,
+            0.00012871401736865864,
+            -0.01687187870925491,
+            0.01374600947628664,
+            -0.008834671549515985,
+            -0.0033344289547729495,
+            0.012937173759658114,
+            -0.003267867578821276,
+            0.01864135643890992,
+            -0.014050228437162755,
+            0.007864998326137568,
+            -0.018988002335416778,
+            -0.003944450943320823,
+            -0.0032084001089808037,
+            -0.0023149926514803745,
+            -0.007750311931438483,
+            -0.045019596490520515,
+            0.013573650825957857,
+            -0.004374086208559833,
+            -0.005869005771503507,
+            -0.01250037134701792,
+            -0.016625210662158708,
+            0.011639132814162461,
+            0.004045997475412364,
+            -0.0014885281342871186,
+            -0.020321856013396592,
+            0.006222318025479057,
+            0.011121009394355636,
+            0.00904225037228108,
+            -0.011585148988670874,
+            -0.0018474735072673144,
+            -0.011325387916132704,
+            -0.0025890917877355725,
+            -0.009148717095934154,
+            0.018495348432506,
+            0.002404017871470538,
+            -0.010232069372010613,
+            0.022941087706292104,
+            -0.0024570420644715457,
+            0.020988341201251853,
+            -0.00442363926883869,
+            -0.004485898239018673,
+            -0.020447941631096866,
+            -0.004158168168969816,
+            -0.019358617104349624,
+            0.00036464612100595666,
+            0.0017749921060331022,
+            0.02078748568527368,
+            0.007256675717496499,
+            0.012612722550842472,
+            0.006032799184116686,
+            0.00631983142330159,
+            0.030151438614535002,
+            -0.010852408231267204,
+            0.0022451749744724233,
+            -0.017863828302314567,
+            0.010653806941450981,
+            0.028289026813870025,
+            0.009882042879445822,
+            -0.006920783006708277,
+            0.014053717954960316,
+            0.004932716665921658,
+            -0.00616100713292227,
+            0.025975267677767474,
+            -0.038836632659249616,
+            -0.021059874446788137,
+            -0.0061122622002863615,
+            0.001439949038430725,
+            -3.496872289060673e-05,
+            0.005030571386336924,
+            -0.011007922189367617,
+            0.009222155733548683,
+            0.0026655561130744807,
+            0.03101907459085608,
+            -0.002317540524509921,
+            -0.007348120500263025,
+            0.003987101796623003,
+            0.001511347369284844,
+            0.021229069065914295,
+            -0.01197457000398312,
+            -0.038429200853184124,
+            -0.0031917847963042626,
+            -0.0039488592790253305,
+            0.008371872140716302,
+            -0.012355524487988753,
+            -0.010839396359961,
+            -0.004574476927098761,
+            -0.01882180934875166,
+            -0.010292867074240718,
+            0.025595330740853118,
+            -0.007162452868646569,
+            0.01788200594893745,
+            -0.0076384034977797525,
+            -0.008112046063674713,
+            0.00683079906571602,
+            0.011572398977364099,
+            -0.01932794114157753,
+            0.0034055065356524052,
+            -0.0027824032743918262,
+            0.01627342501880651,
+            0.011718826886349412,
+            -0.013342112644751174,
+            0.006234666245077415,
+            0.01365078200475753,
+            -0.015476695744659024,
+            -0.0015378873781631448,
+            -0.0069121070205125585,
+            0.014866073481143516,
+            -0.003093423905641203,
+            -0.0033762696975023986,
+            0.011958295888528918,
+            -0.021982979614167224,
+            0.01697002117600754,
+            -0.02183323542526832,
+            0.022295922951583082,
+            -0.0006195453455557405,
+            -0.0006320640234089136,
+            0.008423668060539851,
+            0.008314680408345411,
+            -0.013378756052181496,
+            0.0016194432891316378,
+            -0.03416272433582183,
+            -0.009962172336151867,
+            -0.015051382650973002,
+            0.0038009542554177348,
+            -0.010106196779819915,
+            -0.01203081704994182,
+            -0.021869748479536547,
+            -0.013231917312580397,
+            0.009488867268928662,
+            0.02421155136770049,
+            -0.004349181861466944,
+            0.015665735941744613,
+            -0.008077879618601087,
+            -0.00346140349979159,
+            -0.019022501109520248,
+            -0.020158795455478176,
+            -0.011854460388465065,
+            -0.01570428154296937,
+            -0.007073290993912972,
+            0.004794071576762192,
+            -0.014492538032767992,
+            0.013391092874532436,
+            0.03615243857749714,
+            -0.017029836110560772,
+            0.008609740828901341,
+            0.019179133878629843,
+            0.0009923228421593012,
+            0.015393048602064972,
+            0.003790868078763006,
+            0.011710499244312703,
+            -0.014885526287290077,
+            -0.011829126494406653,
+            -0.0005627435441977781,
+            0.0020582596099707785,
+            0.003728052108395647,
+            0.027914733124690644,
+            0.0038604865752177283,
+            0.005302574635039923,
+            0.012293011832771443,
+            0.013320598642264739,
+            -0.010706185765391238,
+            0.015853648683536427,
+            0.004090821159159474,
+            -0.016980287717038522,
+            -0.0031197192883179663,
+            -0.006894138060845641,
+            -0.012739354473335233,
+            -0.002311031608856898,
+            0.0159372988694303,
+            -0.008091532959711314,
+            0.015169701976466646,
+            -0.004316674756425857,
+            -0.015090694840235596,
+            0.0007287957651103694,
+            -0.03810971752015664,
+            -0.0018729979445690102,
+            -0.01301867399306695,
+            -0.023136174288569147,
+            0.04241471786668668,
+            0.00415464531754789,
+            0.020836251111427635,
+            -0.006703690210920993,
+            0.006147438331654669,
+            -0.020673361951422737,
+            0.011783676068789637,
+            -0.0037516561524537765,
+            0.009702463136945385,
+            0.009366943837185086,
+            -0.018879598518030567,
+            -0.01132474738893142,
+            0.018484010977000243,
+            0.0008964474295952695,
+            0.001899853004269779,
+            0.014452586238316783,
+            0.019746940887134506,
+            0.01136527620507647,
+            -0.0020960934961233306,
+            0.02093114800466563,
+            0.018023169361765227,
+            -0.0002475129435970239,
+            -0.007118840223520255,
+            0.013527604253725194,
+            -0.02531530780276405,
+            0.014601163833111726,
+            0.03268673940622878,
+            0.013556815596615061,
+            0.020396267313433348,
+            -0.016161264086739215,
+            0.016259608464481445,
+            -0.006718110409512467,
+            0.017378727943854342,
+            0.001211924791251576,
+            0.01911265175070918,
+            0.014022657097776944,
+            0.01308266544853342,
+            0.006784756253744209,
+            -0.013367188766230505,
+            0.024435211011756285,
+            -0.008342072323415817,
+            -0.0004209982405190692,
+            -0.002823543536521797,
+            -0.023279946325307846,
+            0.01506543259962226,
+            0.0017095020186682082,
+            0.003411744637061027,
+            -0.0035317155649076414,
+            -0.001957702240702538,
+            0.008037874596341196,
+            0.008176809711854463,
+            -0.015780325305129808,
+            0.0031090160698441006,
+            -0.01924979849975192,
+            -0.008566500568783712,
+            -0.024111269290772643,
+            0.003704817310374187,
+            -0.015182531891766549,
+            -0.009971369164255839,
+            -0.019503345629597463,
+            -0.011275291123690537,
+            -0.010129027262987365,
+            0.00026549411615768147,
+            -0.0077553590287942975,
+            0.014297350661562049,
+            -0.017119422159171904,
+            -0.021341131823214652,
+            0.007945720599865192,
+            0.01439604029848557,
+            0.0037256533134793816,
+            0.005472351446140726,
+            0.008325407758477418,
+            0.009082034215555286,
+            -0.010830069811539129,
+            -0.010352768708989164,
+            0.001902868536460922,
+            0.0010307757364948558,
+            0.011572880679973516,
+            0.01480960939553474,
+            0.011328449881374384,
+            -0.013429607187664061,
+            0.0019326261581148884,
+            -0.014862631307250641,
+            0.004686987038977201,
+            -0.0011432874921856958,
+            -0.007026108142521568,
+            -0.0023122147402109927,
+            -0.01067305755615663,
+            0.008329587642130216,
+            0.0051774766682198365,
+            0.0014058609424645312,
+            0.0175095191841222,
+            -0.0016739751490535205,
+            -0.01295313808457633,
+            -0.012767766732175707,
+            -0.011653259281510757,
+            -0.015653450921388538,
+            0.006739771869002279,
+            -0.006908534840842188,
+            -0.006309735406143942,
+            0.00969748644524724,
+            -0.004300978285892996,
+            -0.0071082865141610775,
+            0.009870599453242536,
+            0.0003007194077865931,
+            -0.0011846789159672513,
+            0.03344372215649544,
+            -0.004482311849310301,
+            0.004153681423879411,
+            0.00022516282287549753,
+            -0.00096173441700326,
+            -0.01608715959652527,
+            -0.026963713856310283,
+            -0.0014323662880336336,
+            -0.02273696636049197,
+            0.017938234567205,
+            -0.0058065575851078514,
+            -0.001006759040376689,
+            0.002493842516193206,
+            -0.0042740143630253385,
+            -6.58295494211919e-05,
+            -0.011768915967276026,
+            0.0022431022211178383,
+            -0.029090994997079067,
+            0.009898341521414653,
+            -0.009544941198003441,
+            -0.02154260971685405,
+            0.01133866603878075,
+            0.040150135462722274,
+            -0.0034082812055093122,
+            0.0006477463440760081,
+            0.0010662318198694314,
+            0.010346430535548179,
+            0.014953186668393513,
+            -0.0019239187336010772,
+            -0.01603518108536549,
+            -0.010550241019652155,
+            0.007810013487665037,
+            0.0003366114893320438,
+            0.0030211673366186986,
+            0.025036691996012227,
+            -0.019097101590128545,
+            -0.016027314610013832,
+            -0.011102216330632247,
+            -0.011450582072890358,
+            0.009254748792585386,
+            0.019696746434919966,
+            0.013185751991632784,
+            -0.020709092762886617,
+            -0.017120964154502993,
+            0.00025737692261907426,
+            -0.02662183259737709,
+            -0.0015175233130328546,
+            0.12569047385271798,
+            -0.004250752280902458,
+            -0.009469510190778525,
+            -0.016304049953896647,
+            -0.00863767582150345,
+            0.02183050591695681,
+            -0.008697910752438377,
+            0.06936580824413162,
+            0.0006861269173917291,
+            -0.0029257096931296273,
+            -0.0020896063525880225,
+            -0.002993306834763656,
+            -0.013871927123029641,
+            0.001864949832879178,
+            -0.016847962023455045,
+            -0.01637518023758693,
+            -0.006332388047537449,
+            -0.013325712872379694,
+            -0.0017155737214029734,
+            -0.0061798136617018905,
+            0.004759205228314821,
+            -0.007830001529147273,
+            0.011495001353183024,
+            -0.026107152853762815,
+            -0.010130127689151806,
+            0.005962957535201191,
+            -0.011637257947784772,
+            0.0011307756612654312,
+            -0.030816554116329904,
+            -0.00441741097001083,
+            7.409036337997905e-05,
+            0.016501246370776126,
+            0.0015748302561177006,
+            -0.00940138978060357,
+            -0.011216292373500447,
+            0.006508185064247167,
+            0.021984022174755303,
+            0.02029850289834145,
+            0.020629050747183544,
+            0.009741513502990842,
+            0.017418812192334685,
+            -0.007031919478404256,
+            0.02625468226435525,
+            0.006196721544177288,
+            0.002206946290342702,
+            0.005596780938975919,
+            -0.01211153541496144,
+            -0.008534755574472074,
+            0.0074101096913398124,
+            -0.0219026467376419,
+            -0.0238583746438936,
+            -0.012260621283748488,
+            0.025739429552984337,
+            -0.009054977254128021,
+            -0.0019759724570709175,
+            -0.0030737356218219617,
+            0.024727831360321823,
+            -0.005149317224364814,
+            0.002332136352952536,
+            0.015587381251136284,
+            -0.028035348333895063,
+            0.008643405636450286,
+            0.007622365990511162,
+            0.02564089666732,
+            -0.0032690983288409517,
+            0.0030897564314706603,
+            -0.009711753474608342,
+            -0.009285462125807194,
+            -0.009613024258506932,
+            0.004240295765592783,
+            0.018961596475867332,
+            0.005797211858953558,
+            0.009922313409983964,
+            -0.006335322307636977,
+            -0.015248479856831972,
+            0.018517674622994848,
+            -0.018526204881631703,
+            -0.017805480495530986,
+            -0.0031014632675223346,
+            -0.023499333544416315,
+            0.007508363381003107,
+            0.006594465447638183,
+            -0.013304818586245387,
+            -0.0232989529893049,
+            0.020407788403667446,
+            0.003426820233878876,
+            -0.006520960224510217,
+            0.0028020004149410366,
+            -0.03282622678411901,
+            -0.018099884128291804,
+            0.004341732255376245,
+            0.012442902960644058,
+            -0.006914419410521592,
+            0.01684289728480369,
+            -0.0213314411022478,
+            0.021170682961259797,
+            -0.018444729425471068,
+            0.015400297173773751,
+            -0.005856807329212788,
+            0.01661347113847185,
+            0.011889835104373263,
+            0.0013585650928227913,
+            -0.019926344687386053,
+            0.02233564528528766,
+            0.0006518460244140545,
+            0.029977976063273147,
+            0.026669882018303386,
+            -0.005623183408671759,
+            -0.004497323576380724,
+            -0.017964594171161242,
+            0.011149712487916564,
+            0.0034380718823694635,
+            0.00295265329839474,
+            0.023835186931253683,
+            0.02803963164846445,
+            -0.028097235508129332,
+            0.017703040589712025,
+            -0.012893023698870347,
+            0.0089348477172039,
+            -0.011630331709306501,
+            0.0096768368423985,
+            0.00669273666692988,
+            0.003825890215203911,
+            0.008289398654583944,
+            0.011807275797649763,
+            0.024893949220958504,
+            0.02893733381281628,
+            0.004069052493622539,
+            0.012465630768347759,
+            -0.003600044227856378,
+            0.004954707230784128,
+            0.005985890100113585,
+            0.0014569711209781388,
+            -0.00648453455551435,
+            0.0031945554565833674,
+            -0.006090168119075504,
+            -0.003744222112779555,
+            0.015266602618825367,
+            -0.00028966361595811053,
+            0.015634624351841753,
+            0.012395295870300452,
+            -0.006577117978669073,
+            -0.003069372773552921,
+            -0.0029369830414480856,
+            -0.007598245840968154,
+            -0.017753359726409172,
+            0.025057809370078198,
+            -0.01606095813358513,
+            -0.019631204097884002,
+            0.0008782653868466668,
+            0.0031106969086903075,
+            -0.013675045629189149,
+            -0.008463196400259825,
+            0.008777824363912681,
+            0.014471929439779247,
+            -0.018478175346087953,
+            0.0017862044141924377,
+            -0.0004910793014860654,
+            0.013862986726603731,
+            -0.003641460118879092,
+            -0.00902817956483427,
+            -0.002076665354355939,
+            0.010393957588184604,
+            0.016413601414994997,
+            0.0038017609859922327,
+            -0.008644753777502864,
+            -0.0005484639867925445,
+            0.006452371249117759,
+            0.007391332924183688,
+            0.0036307379469783137,
+            -4.895894824382588e-05,
+            -0.008129811261078888,
+            -0.013655410823646597,
+            0.018829640339982075,
+            -0.007579460691893014,
+            0.016191139169828096,
+            -0.004984562798832502,
+            -0.011846989328030879,
+            0.004110855640416496,
+            0.0004775146965349116,
+            -0.014308885325290177,
+            0.002778951512640393,
+            -0.003583348080466357,
+            0.0017351729844533707,
+            0.006444402813802743,
+            -0.004945263366121632,
+            0.003803172980003619,
+            -0.008178062077842394,
+            0.02700102856098108,
+            0.015477045085093356,
+            0.005818362056942523,
+            -0.00876269298951217,
+            -0.024697543974427114,
+            0.008183329610246195,
+            0.00518473714350693,
+            0.018661120751406378,
+            0.014464207183253898,
+            0.0019868192486975638,
+            0.009827229842256728,
+            -0.013395371817208287,
+            -0.0022376892192351944,
+            0.009711801873474911,
+            0.017963581533215926,
+            -0.0059700736266576305,
+            0.019169874693002874,
+            -0.010316234316657931,
+            -0.0008125759237295765,
+            0.004003428870809722,
+            0.017109043212624506,
+            -0.004590883969173384,
+            0.013958072401225709
+          ],
+          "title": "F Pseudo Code for Switch Transformers 33",
+          "keyphrases": [
+            "Switch Transformers",
+            "Mixture-of-Experts",
+            "sparsely-activated expert model",
+            "natural language tasks",
+            "pre-training and fine-tuning"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "70180fa2-ac31-4356-94fc-047922ade533",
+      "type": "next",
+      "source": {
+        "id": "23ceada7-a437-4b2d-b989-f16e928f439a",
+        "properties": {
+          "page_content": "1 Introduction 3 ",
+          "embedding": [
+            0.009373236447572708,
+            -0.00019147605053149164,
+            -0.011275587603449821,
+            0.02236648090183735,
+            -0.016502436250448227,
+            0.024490466341376305,
+            -0.009955023415386677,
+            0.09330756217241287,
+            0.00554082915186882,
+            0.0107676787301898,
+            0.006459683645516634,
+            -0.0031074818689376116,
+            0.005069858860224485,
+            -0.029274048283696175,
+            -0.0028881572652608156,
+            0.012771611101925373,
+            0.004903634078800678,
+            0.0010071611031889915,
+            -0.00228790077380836,
+            -0.00550850760191679,
+            0.008833005093038082,
+            0.009594869799911976,
+            -0.036237023770809174,
+            0.07295425236225128,
+            0.023493116721510887,
+            -0.021886276081204414,
+            0.03270936384797096,
+            0.024564342573285103,
+            -0.023142196238040924,
+            -0.0063627189956605434,
+            0.006759811658412218,
+            0.008625224232673645,
+            0.0010533347958698869,
+            -0.021516887471079826,
+            -0.06157708168029785,
+            0.0061087640933692455,
+            0.01108165830373764,
+            0.020574945956468582,
+            -0.03252466768026352,
+            -0.003042838769033551,
+            -0.011358699761331081,
+            -0.008699102327227592,
+            -0.01935596391558647,
+            0.006593586876988411,
+            0.04015254229307175,
+            -0.0013990594306960702,
+            -0.043292347341775894,
+            -0.018090806901454926,
+            -0.023991791531443596,
+            -0.014581616036593914,
+            0.029440272599458694,
+            -0.0307700727134943,
+            0.045434799045324326,
+            0.010758443735539913,
+            0.027667207643389702,
+            -0.0015398887917399406,
+            0.01821085810661316,
+            0.09737084060907364,
+            0.003970927558839321,
+            -0.02190474607050419,
+            0.005037537310272455,
+            -0.05751380696892738,
+            -0.011543394066393375,
+            -0.022107908502221107,
+            -0.010001196525990963,
+            -0.0067828986793756485,
+            -0.016742538660764694,
+            0.04137152433395386,
+            -0.02829516865313053,
+            -0.02114749886095524,
+            0.02690996043384075,
+            -0.002694228198379278,
+            0.0043634031899273396,
+            0.03933988884091377,
+            0.002493373118340969,
+            -0.005203762091696262,
+            0.069408118724823,
+            0.05537135526537895,
+            -0.001163574168458581,
+            0.031915176659822464,
+            -0.00717537384480238,
+            -0.0074200937524437904,
+            0.0161607526242733,
+            0.005069858860224485,
+            0.022772807627916336,
+            -0.01561590377241373,
+            -0.07964018732309341,
+            -0.10867413133382797,
+            0.0017811458092182875,
+            -0.0003122488269582391,
+            0.006099529564380646,
+            -0.008445147424936295,
+            -0.008514408022165298,
+            -0.04657990485429764,
+            -0.01678871363401413,
+            -0.003474561730399728,
+            0.0430707149207592,
+            0.00898999534547329,
+            -0.008334331214427948,
+            -0.01108165830373764,
+            0.03139803186058998,
+            -0.027556391432881355,
+            -0.01748131588101387,
+            -0.007743308786302805,
+            -0.020870456472039223,
+            -0.015976058319211006,
+            0.043255407363176346,
+            0.020741170272231102,
+            -0.0013759726425632834,
+            0.022348010912537575,
+            -0.03315262869000435,
+            0.00933167990297079,
+            -0.013057887554168701,
+            -0.01091543398797512,
+            0.01863565668463707,
+            -0.006025651935487986,
+            -0.03577528893947601,
+            -0.014886361546814442,
+            0.015458913519978523,
+            0.02661444991827011,
+            -0.013741256669163704,
+            -0.010287472978234291,
+            -0.009424027055501938,
+            -0.014535441994667053,
+            -0.027593329548835754,
+            0.0079002995043993,
+            -0.004626592621207237,
+            0.003342967014759779,
+            -0.007669431157410145,
+            0.04181479290127754,
+            0.02417648583650589,
+            0.04436357319355011,
+            -0.02628200128674507,
+            0.003675416810438037,
+            0.017398204654455185,
+            0.019503720104694366,
+            0.021627703681588173,
+            -0.006436596624553204,
+            0.00369388610124588,
+            0.02476750686764717,
+            -0.07927079498767853,
+            -0.010028901509940624,
+            -0.017527490854263306,
+            0.025709448382258415,
+            -0.009433262050151825,
+            0.018728002905845642,
+            -0.04750337824225426,
+            0.019485250115394592,
+            0.007235399447381496,
+            0.010730739682912827,
+            -0.01968841440975666,
+            0.006709021050482988,
+            -0.02216331660747528,
+            -0.012660794891417027,
+            0.004765113350003958,
+            0.011423342861235142,
+            0.03219221904873848,
+            0.017065754160284996,
+            0.03143497183918953,
+            0.03298640251159668,
+            -0.004991363734006882,
+            0.034020692110061646,
+            0.026429755613207817,
+            -0.02631893940269947,
+            0.026134245097637177,
+            -0.019208207726478577,
+            0.012134416028857231,
+            -0.04089131951332092,
+            -0.03645865619182587,
+            0.025044549256563187,
+            0.01748131588101387,
+            -0.006067208014428616,
+            0.040780503302812576,
+            0.0038855066522955894,
+            -0.013131765648722649,
+            -0.00817734096199274,
+            0.017499785870313644,
+            0.01250380463898182,
+            -0.014997177757322788,
+            0.005356134846806526,
+            0.006769046653062105,
+            -0.02308678813278675,
+            0.0019115861505270004,
+            0.0045896535739302635,
+            -0.027002308517694473,
+            -0.010416759178042412,
+            0.020925864577293396,
+            -0.030751602724194527,
+            0.006810602732002735,
+            -0.0014486960135400295,
+            0.05489115044474602,
+            0.0032944846898317337,
+            0.028442922979593277,
+            0.034149978309869766,
+            -0.01456314604729414,
+            -0.04414194077253342,
+            -0.003213681047782302,
+            0.004421120043843985,
+            0.0161607526242733,
+            -0.010970842093229294,
+            -0.0005338820046745241,
+            -0.01942984201014042,
+            0.01731509156525135,
+            0.03333732485771179,
+            0.0022463444620370865,
+            -0.019023513421416283,
+            0.010860025882720947,
+            0.038083966821432114,
+            0.02090739645063877,
+            -0.002989739179611206,
+            0.027500983327627182,
+            0.008200427517294884,
+            -0.03113946132361889,
+            -0.009890380315482616,
+            0.01893116720020771,
+            0.009308593347668648,
+            -0.0269469004124403,
+            0.05725523456931114,
+            0.0020755024161189795,
+            -0.021129028871655464,
+            0.026743736118078232,
+            0.012180590070784092,
+            0.049461137503385544,
+            0.021812397986650467,
+            -0.022311072796583176,
+            0.026577511802315712,
+            0.008893030695617199,
+            0.032469261437654495,
+            0.005651645828038454,
+            -0.00040199869545176625,
+            0.002104360843077302,
+            0.03051150031387806,
+            -0.0017707566730678082,
+            0.007401624694466591,
+            -0.03723437339067459,
+            0.01070303563028574,
+            0.0019427533261477947,
+            0.00814501941204071,
+            0.021165968850255013,
+            0.022384950891137123,
+            -0.006778281182050705,
+            0.008223514072597027,
+            -0.009770329110324383,
+            -0.021221376955509186,
+            -0.018986575305461884,
+            0.005268405191600323,
+            -0.0123375803232193,
+            0.08414673060178757,
+            0.006907567381858826,
+            -0.05141889676451683,
+            0.06157708168029785,
+            0.010823086835443974,
+            0.014156819321215153,
+            -0.007627875078469515,
+            0.011164771392941475,
+            0.0864369347691536,
+            -0.0008519025286659598,
+            -0.00424566026777029,
+            -0.03786233440041542,
+            0.04454826936125755,
+            -0.020593415945768356,
+            0.037751518189907074,
+            -0.0031813594978302717,
+            -0.012032833881676197,
+            0.019780760630965233,
+            -0.060801368206739426,
+            0.043920308351516724,
+            -0.008976143784821033,
+            -0.0009165455121546984,
+            0.03198905289173126,
+            0.06349790096282959,
+            -0.009040786884725094,
+            0.019337493926286697,
+            0.018247798085212708,
+            0.015569730661809444,
+            -0.021258315071463585,
+            0.020796580240130424,
+            -0.0016149209113791585,
+            -0.0142491664737463,
+            -0.033004872500896454,
+            0.004381872713565826,
+            0.026706798002123833,
+            0.025173833593726158,
+            -0.027427105233073235,
+            0.00783565640449524,
+            -0.024194953963160515,
+            0.013482684269547462,
+            0.020593415945768356,
+            -0.03904437646269798,
+            -0.026411285623908043,
+            0.004485763143748045,
+            -0.003559982869774103,
+            -0.04905480891466141,
+            -0.011958956718444824,
+            0.01857101358473301,
+            0.0034791789948940277,
+            0.013381103053689003,
+            -0.0331895686686039,
+            -0.036828044801950455,
+            -0.006542795803397894,
+            0.004314920864999294,
+            -0.0001486211986048147,
+            -0.006436596624553204,
+            -0.035701408982276917,
+            0.0030012824572622776,
+            0.07155057787895203,
+            -0.01592988334596157,
+            0.0028350576758384705,
+            -0.03740059956908226,
+            0.0222926028072834,
+            0.03392834588885307,
+            -0.03908131644129753,
+            0.017324326559901237,
+            -0.06593587249517441,
+            -0.021572295576334,
+            0.023659341037273407,
+            -0.007928003557026386,
+            -0.011718854308128357,
+            -0.03937682881951332,
+            -0.008929969742894173,
+            -0.004912868607789278,
+            -0.003446857677772641,
+            -0.024619750678539276,
+            -0.016040701419115067,
+            0.009622573852539062,
+            -0.007928003557026386,
+            -0.00038872379809617996,
+            0.015246515162289143,
+            -0.01649320125579834,
+            -0.01435074768960476,
+            -0.0038624198641628027,
+            0.027057716622948647,
+            0.03344814106822014,
+            0.01913432963192463,
+            -0.016881059855222702,
+            -0.055629927664995193,
+            0.0015387344174087048,
+            -0.0015075673582032323,
+            -0.004356476943939924,
+            -0.02711312472820282,
+            0.01589294523000717,
+            0.010813851840794086,
+            -0.014443094842135906,
+            0.026392817497253418,
+            0.0029112440533936024,
+            -0.016253098845481873,
+            0.016465498134493828,
+            -0.039561521261930466,
+            -0.006492005195468664,
+            0.013011714443564415,
+            0.0008392047602683306,
+            -0.009548695757985115,
+            0.014212227426469326,
+            -0.02328995242714882,
+            0.01676100865006447,
+            -0.03703121095895767,
+            0.021756989881396294,
+            -0.02354852482676506,
+            0.025783326476812363,
+            0.004104831255972385,
+            0.004903634078800678,
+            0.0001385207287967205,
+            0.02236648090183735,
+            0.006662847474217415,
+            -0.0007001068443059921,
+            -0.06257443130016327,
+            0.01767524518072605,
+            -0.026669858023524284,
+            0.02090739645063877,
+            -0.006542795803397894,
+            -0.04247969016432762,
+            0.018792646005749702,
+            0.0180723387748003,
+            0.008717571385204792,
+            -0.04414194077253342,
+            0.0005454253987409174,
+            0.0029320220928639174,
+            -0.0011347156250849366,
+            -0.04229499772191048,
+            0.01797999069094658,
+            0.004199486691504717,
+            -0.023160666227340698,
+            -0.004656605422496796,
+            0.008435912430286407,
+            0.001627618563361466,
+            -0.030991705134510994,
+            0.005356134846806526,
+            0.004040188156068325,
+            -0.0046242838725447655,
+            -0.008985377848148346,
+            -0.026134245097637177,
+            -0.002957417629659176,
+            -0.020131679251790047,
+            -0.037382129579782486,
+            0.042073365300893784,
+            0.0010117784840986133,
+            -0.0060164169408380985,
+            -0.01735202968120575,
+            0.0010550662409514189,
+            0.01141410879790783,
+            -0.009733390063047409,
+            0.0033845233265310526,
+            -0.014904830604791641,
+            0.027057716622948647,
+            0.02354852482676506,
+            -0.00550850760191679,
+            -0.03512885794043541,
+            -0.02774108573794365,
+            0.0052360836416482925,
+            0.0026064985431730747,
+            -0.03546130657196045,
+            -0.05289645120501518,
+            0.023326890543103218,
+            -0.021923214197158813,
+            0.020131679251790047,
+            0.010481402277946472,
+            -0.030456092208623886,
+            0.028239760547876358,
+            -0.022772807627916336,
+            -0.02061188593506813,
+            -0.001733817858621478,
+            0.005840957630425692,
+            0.0088560925796628,
+            0.002608807059004903,
+            -0.04137152433395386,
+            0.024749036878347397,
+            -0.03134262561798096,
+            -0.007904916070401669,
+            0.024952201172709465,
+            -0.006348866969347,
+            -0.014258400537073612,
+            -0.007119965739548206,
+            0.003691577585414052,
+            0.030659254640340805,
+            0.0076140230521559715,
+            -0.03139803186058998,
+            -0.010213595815002918,
+            -0.002126293256878853,
+            0.023105258122086525,
+            0.005559298675507307,
+            0.021055150777101517,
+            -0.041482340544462204,
+            0.016437793150544167,
+            0.0362185537815094,
+            -0.027685677632689476,
+            0.0021874732337892056,
+            -0.03919213265180588,
+            -0.026429755613207817,
+            0.026429755613207817,
+            -0.010536810383200645,
+            0.0200208630412817,
+            0.0041694738902151585,
+            -0.024121077731251717,
+            0.005577768199145794,
+            0.026411285623908043,
+            -0.00010345767077524215,
+            -0.000978302676230669,
+            0.05452176183462143,
+            -0.008689867332577705,
+            0.021258315071463585,
+            -0.00016622488328721374,
+            0.014433860778808594,
+            0.011349465698003769,
+            -0.017204275354743004,
+            -0.01873723790049553,
+            0.004361094441264868,
+            -0.02282821759581566,
+            -0.013279520906507969,
+            -0.00630731089040637,
+            0.011931252665817738,
+            -0.016345446929335594,
+            -0.002054724143818021,
+            -0.014849422499537468,
+            0.028018126264214516,
+            -0.026429755613207817,
+            -0.029015475884079933,
+            -0.010878494940698147,
+            -0.010112013667821884,
+            -0.006672082003206015,
+            -0.02203403040766716,
+            -0.015218811109662056,
+            0.04092825949192047,
+            0.012097476981580257,
+            -0.0010181274265050888,
+            0.0024218042381107807,
+            0.014221461489796638,
+            0.044806838035583496,
+            -0.026374347507953644,
+            0.010758443735539913,
+            0.027168532833456993,
+            -0.012411457486450672,
+            0.004467293620109558,
+            0.004324155393987894,
+            0.019282085821032524,
+            -0.01784146949648857,
+            -0.07816263288259506,
+            -0.020002393051981926,
+            -0.0386011116206646,
+            -0.01360273640602827,
+            0.04558255523443222,
+            -0.002902009291574359,
+            0.016668662428855896,
+            -0.006007182411849499,
+            -0.011155536398291588,
+            0.010795382782816887,
+            0.03394681587815285,
+            0.0025418554432690144,
+            -0.0037054296117275953,
+            -0.004377255216240883,
+            0.01626233384013176,
+            -0.004307994619011879,
+            -0.02740863524377346,
+            0.008657545782625675,
+            0.004963659681379795,
+            0.008851475082337856,
+            -0.019577596336603165,
+            -0.005088328383862972,
+            0.005887131206691265,
+            0.017887644469738007,
+            -0.02005780301988125,
+            0.0027080802246928215,
+            0.03241385146975517,
+            -0.033004872500896454,
+            0.015477383509278297,
+            -0.022551175206899643,
+            0.026245061308145523,
+            0.022551175206899643,
+            -0.0063581014983356,
+            0.013528858311474323,
+            -0.018552543595433235,
+            -0.004201795440167189,
+            0.0053976913914084435,
+            0.018063103780150414,
+            0.009816502220928669,
+            0.013418042100965977,
+            -0.019078921526670456,
+            -0.024065667763352394,
+            0.03549824655056,
+            0.011469516903162003,
+            0.00992731936275959,
+            -0.050938691943883896,
+            0.008223514072597027,
+            0.010398290120065212,
+            -0.026559041813015938,
+            -0.025210773572325706,
+            0.03559059277176857,
+            0.010610688477754593,
+            -0.02295750193297863,
+            0.01391671597957611,
+            0.05444788187742233,
+            -0.008625224232673645,
+            0.02325301431119442,
+            0.01254997868090868,
+            0.010906198993325233,
+            -4.3900970922550187e-05,
+            0.022218724712729454,
+            -0.0075124409049749374,
+            0.0037746899761259556,
+            0.0031328771729022264,
+            -0.0238809734582901,
+            0.026928430423140526,
+            -0.008560581132769585,
+            -0.01689029484987259,
+            0.001642625080421567,
+            0.028018126264214516,
+            -0.0199839249253273,
+            0.00649662222713232,
+            0.015135698951780796,
+            -0.014997177757322788,
+            0.009622573852539062,
+            0.02437964826822281,
+            0.010970842093229294,
+            -0.04454826936125755,
+            -0.01976229064166546,
+            0.012429927475750446,
+            0.012799316085875034,
+            0.02661444991827011,
+            0.009110046550631523,
+            0.006196494214236736,
+            0.0003523621126078069,
+            -0.010047370567917824,
+            -0.01873723790049553,
+            0.014951004646718502,
+            -0.015375801362097263,
+            -0.029828131198883057,
+            -0.028646087273955345,
+            -0.005850192159414291,
+            -0.004857460502535105,
+            -0.013141000643372536,
+            -0.009733390063047409,
+            0.0158005990087986,
+            -0.015135698951780796,
+            0.01959606632590294,
+            0.006602821405977011,
+            -0.00042508551268838346,
+            0.033300384879112244,
+            0.03064078651368618,
+            0.02144300937652588,
+            -0.00038497219793498516,
+            -0.04281214252114296,
+            0.004885164555162191,
+            0.0005820756778120995,
+            0.007678666152060032,
+            0.0036130824591964483,
+            0.00820966251194477,
+            0.04809439927339554,
+            0.004931338131427765,
+            0.013390337117016315,
+            0.006921419408172369,
+            0.03732671961188316,
+            -0.029865069314837456,
+            0.0032044462859630585,
+            0.032007522881031036,
+            -0.017961522564291954,
+            -0.03165660426020622,
+            -0.014590850099921227,
+            0.006944505963474512,
+            0.007794099859893322,
+            -0.005222231615334749,
+            -0.00103255663998425,
+            0.02111055888235569,
+            -0.0013344164472073317,
+            0.008265070617198944,
+            0.020297905430197716,
+            0.0007843736675567925,
+            0.01606840454041958,
+            -0.016345446929335594,
+            0.0039432235062122345,
+            -0.009816502220928669,
+            -0.02988353930413723,
+            -0.00913775060325861,
+            0.002717314986512065,
+            0.004215647466480732,
+            0.004573492798954248,
+            -0.0077479262836277485,
+            0.013270285911858082,
+            0.03592304512858391,
+            -0.0024633603170514107,
+            0.010739973746240139,
+            -0.02018708735704422,
+            0.017398204654455185,
+            0.0026388198602944613,
+            0.006602821405977011,
+            0.01985463872551918,
+            -0.02476750686764717,
+            0.007443180773407221,
+            -0.011672680266201496,
+            0.017296621575951576,
+            -0.005600854754447937,
+            0.03952458128333092,
+            0.0018192389979958534,
+            0.014877126552164555,
+            0.016151517629623413,
+            0.029440272599458694,
+            0.01764754205942154,
+            -0.039302948862314224,
+            -0.014803249388933182,
+            0.008232749067246914,
+            -0.0063211629167199135,
+            -0.012513039633631706,
+            0.0017615220276638865,
+            -0.011783497408032417,
+            0.032044462859630585,
+            -0.010204360820353031,
+            0.012743907049298286,
+            -0.014840187504887581,
+            -0.01439692173153162,
+            -0.04824215546250343,
+            -0.0015548951923847198,
+            0.00037256305222399533,
+            0.0019970072899013758,
+            -0.036292433738708496,
+            0.002897392027080059,
+            0.024989139288663864,
+            -0.02870149537920952,
+            0.002140145283192396,
+            -0.04318153113126755,
+            0.01896810531616211,
+            0.005854809656739235,
+            -0.021165968850255013,
+            -0.01606840454041958,
+            -0.012679263949394226,
+            0.0038947411812841892,
+            0.04488071799278259,
+            0.00553159462288022,
+            0.008902265690267086,
+            -0.03291252627968788,
+            -0.00022769345378037542,
+            -0.0012697733473032713,
+            0.003028986742720008,
+            0.010315177030861378,
+            0.014424625784158707,
+            0.03206293284893036,
+            0.00712458323687315,
+            -0.024435056373476982,
+            -0.010130482725799084,
+            0.01385207287967205,
+            0.021646173670887947,
+            0.016945702955126762,
+            0.008006498217582703,
+            -0.007567849475890398,
+            -0.005633176304399967,
+            -0.030234457924962044,
+            0.01441539078950882,
+            0.016871824860572815,
+            0.007138435263186693,
+            0.006750577129423618,
+            -0.004326464142650366,
+            0.01546814851462841,
+            -0.03150884807109833,
+            -0.026411285623908043,
+            0.023659341037273407,
+            -0.01883881911635399,
+            0.010370585136115551,
+            -0.006986062508076429,
+            0.020371781662106514,
+            -0.02328995242714882,
+            -0.02279127761721611,
+            -0.004040188156068325,
+            0.012697733938694,
+            -0.0001104558541555889,
+            -0.007364685647189617,
+            -0.004379563964903355,
+            -0.006233432795852423,
+            -0.0046750749461352825,
+            -3.780461702262983e-05,
+            -0.0146277891471982,
+            0.03812090680003166,
+            0.004192560911178589,
+            -0.00451115844771266,
+            -0.018063103780150414,
+            -0.018820350989699364,
+            0.00834356527775526,
+            0.013168704695999622,
+            0.03126874566078186,
+            0.001201667357236147,
+            -0.010629157535731792,
+            0.015643607825040817,
+            0.0024148779921233654,
+            -0.005171440541744232,
+            0.03185976669192314,
+            -0.0030751603189855814,
+            -0.01968841440975666,
+            0.006764429155737162,
+            -0.025210773572325706,
+            0.007660196628421545,
+            0.02437964826822281,
+            -0.0008998076082207263,
+            -0.020168619230389595,
+            0.02262505330145359,
+            -0.02064882405102253,
+            -0.03014211170375347,
+            0.010416759178042412,
+            -0.034648653119802475,
+            -0.01867259480059147,
+            0.01001043152064085,
+            0.02321607433259487,
+            0.007623257581144571,
+            0.007350833620876074,
+            0.002272894373163581,
+            -0.011201709508895874,
+            0.01826626807451248,
+            -0.03856417164206505,
+            0.009982727468013763,
+            0.025321589782834053,
+            0.0061087640933692455,
+            -0.013113295659422874,
+            -0.014258400537073612,
+            -0.012226763181388378,
+            0.003841641591861844,
+            0.022440358996391296,
+            0.031675074249506,
+            0.027390167117118835,
+            -0.009834972210228443,
+            0.0014302266063168645,
+            -0.002456434303894639,
+            0.009493287652730942,
+            -0.040189482271671295,
+            -0.013870542868971825,
+            0.003945532254874706,
+            -0.04096519947052002,
+            -0.0082789221778512,
+            -0.0030543820466846228,
+            0.007558614481240511,
+            0.022348010912537575,
+            0.00971492100507021,
+            -0.018386319279670715,
+            -0.037788454443216324,
+            -0.0238440353423357,
+            0.0051437364891171455,
+            0.04303377494215965,
+            0.03021598979830742,
+            -0.016908764839172363,
+            0.011238648556172848,
+            0.045434799045324326,
+            -0.005799401551485062,
+            -0.015911415219306946,
+            0.021627703681588173,
+            -0.012346814386546612,
+            0.016086874529719353,
+            0.016530141234397888,
+            0.0024218042381107807,
+            -0.04875929653644562,
+            0.029329456388950348,
+            0.017952287569642067,
+            0.006935271434485912,
+            0.02628200128674507,
+            -0.023068320006132126,
+            -0.03512885794043541,
+            -0.002382556675001979,
+            -0.0009690679726190865,
+            0.014138349331915379,
+            -0.009770329110324383,
+            -0.0013436510926112533,
+            0.003975545056164265,
+            0.052231550216674805,
+            0.010305942967534065,
+            -0.002495681867003441,
+            -0.01265155989676714,
+            0.02338230051100254,
+            0.013805899769067764,
+            -0.010204360820353031,
+            0.015578964725136757,
+            -0.019337493926286697,
+            0.02413954585790634,
+            0.046653781086206436,
+            0.007493971846997738,
+            0.009465583600103855,
+            0.018257033079862595,
+            0.010749208740890026,
+            -0.014932534657418728,
+            0.00727233849465847,
+            -0.013547327369451523,
+            0.006949123460799456,
+            0.010416759178042412,
+            0.007184608839452267,
+            -0.015791364014148712,
+            0.016216160729527473,
+            -0.017952287569642067,
+            0.012162120081484318,
+            0.00358768692240119,
+            -0.014932534657418728,
+            0.009290123358368874,
+            -0.0123375803232193,
+            -0.025026079267263412,
+            -0.0044626761227846146,
+            -0.00965027790516615,
+            0.0034999572671949863,
+            0.024749036878347397,
+            -0.021775459870696068,
+            0.016206925734877586,
+            0.014295339584350586,
+            0.0030520735308527946,
+            0.017823001369833946,
+            -0.04137152433395386,
+            0.009871911257505417,
+            0.010970842093229294,
+            0.05167746916413307,
+            0.03322650492191315,
+            -0.015542026609182358,
+            0.014101410284638405,
+            0.04159316048026085,
+            -0.019078921526670456,
+            -0.016142282634973526,
+            0.012217528186738491,
+            -0.005614707246422768,
+            -0.040189482271671295,
+            -0.04838990792632103,
+            -0.03021598979830742,
+            -0.003042838769033551,
+            -0.015246515162289143,
+            -0.0037262076511979103,
+            -0.01376896072179079,
+            0.0005832299939356744,
+            0.008717571385204792,
+            0.00018700299551710486,
+            0.015117228962481022,
+            -0.009484052658081055,
+            -0.0018411714117974043,
+            -0.023807097226381302,
+            -0.019005045294761658,
+            0.0009771483018994331,
+            -0.0177121851593256,
+            0.009364001452922821,
+            -0.020297905430197716,
+            -0.011174005456268787,
+            0.0038855066522955894,
+            0.024435056373476982,
+            0.03437161073088646,
+            -0.01870029978454113,
+            0.011451046913862228,
+            0.010481402277946472,
+            -0.006958357989788055,
+            -0.04613664001226425,
+            0.01185737457126379,
+            0.008583667688071728,
+            -0.0037192816380411386,
+            0.008943822234869003,
+            -0.009622573852539062,
+            0.001551432185806334,
+            -0.006427362095564604,
+            0.029643436893820763,
+            0.018487900495529175,
+            -0.02391791343688965,
+            -0.004525010474026203,
+            0.015578964725136757,
+            0.01857101358473301,
+            0.005134501960128546,
+            -0.011663445271551609,
+            -0.010555279441177845,
+            -0.01708422414958477,
+            -0.027353227138519287,
+            -0.04074356332421303,
+            -0.049128685146570206,
+            0.006593586876988411,
+            -0.0061087640933692455,
+            0.0055546811781823635,
+            0.029015475884079933,
+            -0.0002024422719841823,
+            0.041777852922677994,
+            0.0063858055509626865,
+            -0.0017072680639103055,
+            -0.013150234706699848,
+            0.009705686010420322,
+            0.021572295576334,
+            -0.05267481878399849,
+            -0.02354852482676506,
+            0.0022070968989282846,
+            0.017536723986268044,
+            0.004187943413853645,
+            -0.0009448268101550639,
+            0.020556475967168808,
+            -0.00783565640449524,
+            -0.00719846086576581,
+            0.04779888689517975,
+            -0.010509106330573559,
+            -0.007526292931288481,
+            0.011321761645376682,
+            -0.010656861588358879,
+            0.020371781662106514,
+            0.023954851552844048,
+            0.03331885486841202,
+            0.016373150050640106,
+            -0.008833005093038082,
+            0.022107908502221107,
+            -0.0215353574603796,
+            0.004848225507885218,
+            0.025450875982642174,
+            0.001217828132212162,
+            0.0021066695917397738,
+            -0.025395467877388,
+            0.025284651666879654,
+            -0.004425737541168928,
+            0.014092176221311092,
+            -0.0017188114579766989,
+            0.013371868059039116,
+            -0.014738606289029121,
+            0.025450875982642174,
+            0.0010175502393394709,
+            0.03136109560728073,
+            0.017065754160284996,
+            -0.029902009293437004,
+            -0.015218811109662056,
+            -0.012032833881676197,
+            -0.04151928052306175,
+            0.007544762454926968,
+            0.006713638082146645,
+            0.021516887471079826,
+            -0.02245882898569107,
+            -0.004580419044941664,
+            -0.01896810531616211,
+            -0.015643607825040817,
+            0.0051437364891171455,
+            -0.028683027252554893,
+            0.000924048712477088,
+            -0.010112013667821884,
+            -0.0034076101146638393,
+            0.0009985036449506879,
+            -0.02962496690452099,
+            0.020390251651406288,
+            0.0013782812748104334,
+            -0.016336211934685707,
+            0.02681761421263218,
+            0.019614536315202713,
+            -0.00643197912722826,
+            -0.016317741945385933,
+            0.003005899954587221,
+            -0.00890688318759203,
+            -0.001478708814829588,
+            0.04802051931619644,
+            0.010084309615194798,
+            0.012300641275942326,
+            0.004732791800051928,
+            -0.014064472168684006,
+            -0.00013296547695063055,
+            0.0014475416392087936,
+            0.010102778673171997,
+            -0.010333647020161152,
+            -0.008057288825511932,
+            0.005947156809270382,
+            -0.0045896535739302635,
+            -0.012697733938694,
+            0.01454467698931694,
+            0.016604019328951836,
+            -0.0036061564460396767,
+            -0.0193005558103323,
+            0.023345360532402992,
+            0.010305942967534065,
+            0.0048666950315237045,
+            0.0042133391834795475,
+            -0.0018053868552669883,
+            -0.006926036439836025,
+            0.01750902086496353,
+            -0.008555963635444641,
+            0.013547327369451523,
+            0.019023513421416283,
+            -0.03664335235953331,
+            0.018127746880054474,
+            0.030696194618940353,
+            0.04310765117406845,
+            -0.0011774261947721243,
+            -0.01449850294739008,
+            -0.018090806901454926,
+            -0.01555126067250967,
+            0.004229499492794275,
+            0.015643607825040817,
+            0.02015014924108982,
+            0.005379221867769957,
+            0.0068244547583162785,
+            -0.011460281908512115,
+            -0.018386319279670715,
+            0.009502522647380829,
+            -0.0032991021871566772,
+            -0.0200208630412817,
+            0.032303035259246826,
+            0.0003789119073189795,
+            -0.008966908790171146,
+            -0.03051150031387806,
+            0.006141085643321276,
+            0.003566908882930875,
+            0.017370499670505524,
+            0.007558614481240511,
+            -0.001290551503188908,
+            -0.008158870972692966,
+            0.0035415133461356163,
+            0.008943822234869003,
+            -0.008999230340123177,
+            0.04181479290127754,
+            -0.0009627190884202719,
+            0.013464215211570263,
+            0.006699786055833101,
+            -0.021036682650446892,
+            0.0002056167140835896,
+            -0.005900983233004808,
+            -0.01689029484987259,
+            0.001274390728212893,
+            -0.02077811025083065,
+            -0.0027311670128256083,
+            0.025118425488471985,
+            -0.00929935835301876,
+            -0.029661906883120537,
+            0.030234457924962044,
+            -0.012513039633631706,
+            -0.02123984508216381,
+            0.0044395895674824715,
+            0.006879862863570452,
+            0.013944420032203197,
+            0.00624266779050231,
+            -0.004894399084150791,
+            0.01590218022465706,
+            0.008315861225128174,
+            0.001127789611928165,
+            0.02328995242714882,
+            -0.019633004441857338,
+            -0.00524531863629818,
+            -0.0022244120482355356,
+            -0.020888926461338997,
+            -0.01635468192398548,
+            0.006335014943033457,
+            -0.023696279153227806,
+            -0.018792646005749702,
+            -0.001254766946658492,
+            -0.0092347152531147,
+            0.016049936413764954,
+            0.03051150031387806,
+            0.0035368960816413164,
+            -0.01254997868090868,
+            0.007974176667630672,
+            -0.00046606454998254776,
+            -0.0038831979036331177,
+            -0.05042154714465141,
+            -0.021258315071463585,
+            0.011829670518636703,
+            0.007909533567726612,
+            -0.01681641675531864,
+            -0.001661094487644732,
+            0.0036061564460396767,
+            0.06763505935668945,
+            -0.02232954278588295,
+            -0.016576314345002174,
+            -0.030603846535086632,
+            0.002384865190833807,
+            0.003539204830303788,
+            0.003673108061775565,
+            0.003989397082477808,
+            -0.014830953441560268,
+            0.030252927914261818,
+            -0.0012963231420144439,
+            0.04613664001226425,
+            -0.009604103863239288,
+            0.008860709145665169,
+            0.028110474348068237,
+            0.0199839249253273,
+            0.015163403004407883,
+            -0.0238440353423357,
+            -0.000534170598257333,
+            -0.00905463844537735,
+            0.0022417271975427866,
+            -0.01439692173153162,
+            0.008200427517294884,
+            0.02774108573794365,
+            -0.016447028145194054,
+            0.00649662222713232,
+            0.017102692276239395,
+            -0.020427191630005836,
+            -0.022569645196199417,
+            -0.0011924325954169035,
+            -0.0008091919589787722,
+            0.02668832801282406,
+            0.01299324445426464,
+            -0.005600854754447937,
+            0.010619922541081905,
+            0.022089440375566483,
+            -0.007678666152060032,
+            0.01728738658130169,
+            0.014119880273938179,
+            -0.014489268884062767,
+            0.011811201460659504,
+            0.007295425049960613,
+            0.0029158613178879023,
+            -0.03147191181778908,
+            -0.022237194702029228,
+            -0.0018965797498822212,
+            0.007946472615003586,
+            -0.001829628017731011,
+            -0.02236648090183735,
+            -0.008389739319682121,
+            0.02777802385389805,
+            -0.00840820837765932,
+            0.011663445271551609,
+            0.0002845446579158306,
+            -0.009936554357409477,
+            0.029902009293437004,
+            0.03457477316260338,
+            -0.002913552802056074,
+            0.007041470613330603,
+            -0.029809661209583282,
+            0.019743822515010834,
+            -0.018100041896104813,
+            -0.0038139375392347574,
+            -0.03571987897157669,
+            0.03856417164206505,
+            0.016280803829431534,
+            -0.0028535271994769573,
+            -0.004751261323690414,
+            0.028516801074147224,
+            0.02299444191157818,
+            -0.0386011116206646,
+            0.03274630010128021,
+            0.008135784417390823,
+            0.0031651987228542566,
+            -0.0038624198641628027,
+            -0.020722702145576477,
+            0.009114664047956467,
+            -0.033725179731845856,
+            0.0020974348299205303,
+            -0.01533886231482029,
+            0.005619324278086424,
+            -0.0246751606464386,
+            -0.0146277891471982,
+            0.0036823428235948086,
+            -0.006699786055833101,
+            0.005259170662611723,
+            -0.0036615647841244936,
+            0.003359127789735794,
+            -0.002754253800958395,
+            -0.02203403040766716,
+            -0.033060282468795776,
+            0.026115775108337402,
+            -0.01250380463898182,
+            0.004513467196375132,
+            -0.009484052658081055,
+            -0.03106558322906494,
+            0.02005780301988125,
+            -0.013205642811954021,
+            -0.03248772770166397,
+            -0.009303975850343704,
+            0.026041897013783455,
+            0.000664899533148855,
+            -0.008980761282145977,
+            0.00920239370316267,
+            0.04377255216240883,
+            0.03634783998131752,
+            -0.017952287569642067,
+            0.014239931479096413,
+            0.019928516820073128,
+            0.0010221676202490926,
+            0.013242581859230995,
+            -0.019614536315202713,
+            -0.013362633064389229,
+            0.004407268017530441,
+            -0.019448310136795044,
+            -0.014757075347006321,
+            -0.040817443281412125,
+            -0.00986267626285553,
+            0.02602342888712883,
+            -0.01544967945665121,
+            -0.015357332304120064,
+            0.03919213265180588,
+            0.007036853116005659,
+            -0.04613664001226425,
+            -0.00831124372780323,
+            0.027851901948451996,
+            0.00270115421153605,
+            -0.012429927475750446,
+            0.004880547057837248,
+            0.013353399001061916,
+            -0.027131594717502594,
+            -0.001456776401028037,
+            0.023622402921319008,
+            0.03054843842983246,
+            -0.03440855070948601,
+            -0.01519110705703497,
+            0.014812483452260494,
+            -0.0010071611031889915,
+            -0.00793262105435133,
+            -0.014747840352356434,
+            -0.0068244547583162785,
+            0.022274134680628777,
+            0.006515091750770807,
+            -0.01286395825445652,
+            0.006196494214236736,
+            -0.011755792424082756,
+            -0.03472252935171127,
+            0.02266199141740799,
+            -0.01286395825445652,
+            -0.00313749467022717,
+            0.01735202968120575,
+            0.003996323328465223,
+            0.007023001089692116,
+            0.012245233170688152,
+            -0.012706968933343887,
+            -0.017102692276239395,
+            0.028849251568317413,
+            0.02460128255188465,
+            -0.022384950891137123,
+            0.015772894024848938,
+            -0.0045550232753157616,
+            0.01918973959982395,
+            -0.012808550149202347,
+            -0.007309277541935444,
+            -0.0019646857399493456,
+            0.0003399529668968171,
+            0.003322188975289464,
+            0.06257443130016327,
+            -0.03974621742963791,
+            -0.019466780126094818,
+            0.008597520180046558,
+            0.016973407939076424,
+            -0.02321607433259487,
+            -0.01456314604729414,
+            0.028719965368509293,
+            -0.025487814098596573,
+            0.02484138496220112,
+            0.014507737942039967,
+            -0.014535441994667053,
+            -0.0188295841217041,
+            0.015301923267543316,
+            0.02668832801282406,
+            -0.010444463230669498,
+            0.007309277541935444,
+            -0.01326105184853077,
+            0.004903634078800678,
+            0.006468918174505234,
+            0.010324412025511265,
+            -0.008597520180046558,
+            0.005199145060032606,
+            0.021516887471079826,
+            0.0002038852107943967,
+            -0.00269191968254745,
+            -0.00965027790516615,
+            0.014618555083870888,
+            0.007803334388881922,
+            0.000943095306865871,
+            -0.0005249358364380896,
+            -3.19607715937309e-05,
+            -0.006616673897951841,
+            0.021332193166017532,
+            -0.026559041813015938,
+            -0.002375630661845207,
+            -0.007669431157410145,
+            0.024102607741951942,
+            -0.01685335673391819,
+            0.02022402733564377,
+            -0.0070091490633785725,
+            0.026669858023524284,
+            0.00461735762655735,
+            0.00012438007979653776,
+            0.012300641275942326,
+            0.018395552411675453,
+            -0.02292056381702423,
+            0.002876613987609744,
+            -0.024989139288663864,
+            -0.0009125053184106946,
+            -0.0027265497483313084,
+            0.0269469004124403,
+            -0.00020720393513329327,
+            -0.02783343195915222,
+            0.03165660426020622,
+            -0.006048738490790129,
+            0.007323129568248987,
+            -0.012457631528377533,
+            -0.007757160812616348,
+            0.017407439649105072,
+            -0.03361436352133751,
+            0.015006412751972675,
+            0.020002393051981926,
+            0.012817785143852234,
+            -0.017527490854263306,
+            0.030197519809007645,
+            -0.001605686149559915,
+            -0.01935596391558647,
+            0.011534160003066063,
+            0.00199585291557014,
+            0.01070303563028574,
+            -0.0044557503424584866,
+            0.00849132053554058,
+            -0.02127678506076336,
+            -0.012577682733535767,
+            0.019743822515010834,
+            0.018681829795241356,
+            -0.004564258269965649,
+            -0.0005191641394048929,
+            -0.0017672936664894223,
+            -0.024435056373476982,
+            -0.018035398796200752,
+            0.001420991844497621,
+            -0.0008836468332447112,
+            0.0029851216822862625,
+            -0.0051437364891171455,
+            -0.03492569550871849,
+            0.004130226559937,
+            0.021646173670887947,
+            0.014600085094571114,
+            0.0067828986793756485,
+            -0.021313723176717758,
+            0.008320478722453117,
+            0.013990594074130058,
+            -0.023825565353035927,
+            0.03139803186058998,
+            -0.00910542905330658,
+            -0.03719743341207504,
+            0.004407268017530441,
+            0.0051206499338150024,
+            -0.016410090029239655,
+            -0.0004279713612049818,
+            0.01183890551328659,
+            -0.0061780246905982494,
+            0.016945702955126762,
+            0.021036682650446892,
+            -0.016908764839172363,
+            0.0088560925796628,
+            0.013381103053689003,
+            -0.012365284375846386,
+            -0.0014613937819376588,
+            -0.0006545104552060366,
+            -0.00712458323687315,
+            -0.0014867892023175955,
+            0.059323813766241074,
+            -0.012171355076134205,
+            -0.0038901239167898893,
+            -0.025968020781874657,
+            0.024952201172709465,
+            -0.01989157684147358,
+            -0.009668746963143349,
+            -0.001956605352461338,
+            -0.010887729935348034,
+            -0.009982727468013763,
+            0.005956391338258982,
+            -0.014323043636977673,
+            0.018293971195816994,
+            0.03054843842983246,
+            0.027334757149219513,
+            0.003559982869774103,
+            0.014193757437169552,
+            -0.00628422386944294,
+            -0.004693544004112482,
+            0.02434271015226841,
+            -0.019448310136795044,
+            0.0064781527034938335,
+            0.0007786019705235958,
+            -0.0007474347949028015,
+            -0.004432663321495056,
+            -0.010832320898771286,
+            -0.04100213572382927,
+            -0.0292555782943964,
+            0.02249576710164547,
+            0.006173407193273306,
+            0.020888926461338997,
+            0.002327148336917162,
+            0.009677981957793236,
+            -0.0028004273772239685,
+            0.012032833881676197,
+            0.011303291656076908,
+            -0.02391791343688965,
+            -0.014516972936689854,
+            -0.0033337322529405355,
+            -0.004559640772640705,
+            -0.012559212744235992,
+            -0.010555279441177845,
+            0.018949635326862335,
+            0.01800769567489624,
+            0.009548695757985115,
+            -0.015837537124753,
+            0.00036159681621938944,
+            -0.0007029927219264209,
+            -0.0013702008873224258,
+            0.0008253526757471263,
+            -0.007083026692271233,
+            -0.033466607332229614,
+            0.009354766458272934,
+            -0.016770243644714355,
+            0.006196494214236736,
+            0.015412740409374237,
+            -0.010943138040602207,
+            -0.0032575458753854036,
+            0.0046312096528708935,
+            -0.02183086797595024,
+            -0.007817186415195465,
+            -0.018358614295721054,
+            0.006704403553158045,
+            -0.027851901948451996,
+            0.0012097477447241545,
+            0.00803420227020979,
+            0.0010573748731985688,
+            0.026263531297445297,
+            0.00920239370316267,
+            -0.030880888924002647,
+            0.02321607433259487,
+            -0.003158272709697485,
+            0.008805301040410995,
+            -0.0037700727116316557,
+            -0.012402222491800785,
+            0.01636391505599022,
+            0.0074200937524437904,
+            0.00840820837765932,
+            -0.008126549422740936,
+            0.019651474431157112,
+            -0.0001968148717423901,
+            0.007138435263186693,
+            0.017195040360093117,
+            -0.022975971922278404,
+            -0.006085677538067102,
+            -0.0026896109338849783,
+            0.017933817580342293,
+            0.026540571823716164,
+            -0.01368584856390953,
+            -0.017407439649105072,
+            -0.009585634805262089,
+            0.023991791531443596,
+            0.014600085094571114,
+            0.015874475240707397,
+            0.002256733598187566,
+            -0.023142196238040924,
+            -0.0199839249253273,
+            0.02186780609190464,
+            -0.004266438540071249,
+            0.0017430525040253997,
+            0.026374347507953644,
+            -0.028405984863638878,
+            -0.014406155794858932,
+            -0.022089440375566483,
+            0.006034886464476585,
+            0.0012293715262785554,
+            0.009050020948052406,
+            -0.003119025146588683,
+            -0.015883710235357285,
+            -0.003446857677772641,
+            0.0013517314800992608,
+            -0.005240701138973236,
+            0.003998631611466408,
+            0.010130482725799084,
+            -0.003250619862228632,
+            0.005610089749097824,
+            0.011617272160947323,
+            -0.0238440353423357,
+            -0.004372637718915939,
+            0.010305942967534065,
+            -0.02262505330145359,
+            -0.02598648890852928,
+            -0.0027219324838370085,
+            0.014313808642327785,
+            0.016668662428855896,
+            0.006399658042937517,
+            0.006538178771734238,
+            -0.006399658042937517,
+            -0.027981188148260117,
+            0.01860795170068741,
+            -0.004351859912276268,
+            0.00405634893104434,
+            -0.013778195716440678,
+            0.020888926461338997,
+            0.017592133954167366,
+            0.007646344602108002,
+            0.009677981957793236,
+            -0.013519623316824436,
+            -0.03680957481265068,
+            -0.00034803335438482463,
+            0.00910542905330658,
+            -0.006335014943033457,
+            0.003012825967743993,
+            -0.008006498217582703,
+            -0.017832236364483833,
+            0.01883881911635399,
+            -0.007863360457122326,
+            0.0021932448726147413,
+            0.004642753396183252,
+            -0.032044462859630585,
+            0.008763745427131653,
+            0.019540658220648766,
+            0.01857101358473301,
+            -0.01446156483143568,
+            -0.015680547803640366,
+            -0.01276237703859806,
+            0.0107676787301898,
+            0.002585720270872116,
+            0.000738777220249176,
+            -0.0188295841217041,
+            -0.021720051765441895,
+            -0.028405984863638878,
+            0.006256519816815853,
+            0.0005335934110917151,
+            -0.022902093827724457,
+            -0.011238648556172848,
+            -0.005503890570253134,
+            0.01896810531616211,
+            0.01276237703859806,
+            0.007489354349672794,
+            0.02421342395246029,
+            -0.008306626230478287,
+            0.017130397260189056,
+            0.004940572660416365,
+            0.01368584856390953,
+            -0.0142491664737463,
+            -0.0002904606517404318,
+            -0.00988114532083273,
+            0.01926361583173275,
+            0.029477212578058243,
+            -0.008833005093038082,
+            -0.004679691977798939,
+            0.011330995708703995,
+            -0.004598888102918863,
+            0.022107908502221107,
+            0.02341923862695694,
+            -0.01085079088807106,
+            0.03198905289173126,
+            0.0008190038497559726,
+            0.015181872062385082,
+            -0.010527575388550758,
+            -0.013085591606795788,
+            -0.009585634805262089,
+            0.00042508551268838346,
+            -0.028812311589717865,
+            -0.027464043349027634,
+            0.009290123358368874,
+            0.0075216758996248245,
+            0.003876271890476346,
+            -0.015458913519978523,
+            0.009188542142510414,
+            0.0030913210939615965,
+            0.01613304764032364,
+            0.009377853944897652,
+            0.09301205724477768,
+            0.012106711976230145,
+            0.01566207781434059,
+            0.013667378574609756,
+            0.0011185549665242434,
+            -0.013150234706699848,
+            0.005042154807597399,
+            -0.0016368533251807094,
+            9.480012522544712e-05,
+            -0.005910217761993408,
+            -0.014166053384542465,
+            -0.01774912327528,
+            -0.008588285185396671,
+            0.025968020781874657,
+            0.015292689204216003,
+            -0.014877126552164555,
+            -0.004912868607789278,
+            -0.020833518356084824,
+            -0.003654638770967722,
+            -0.013177938759326935,
+            0.009073108434677124,
+            -0.0216092336922884,
+            -0.006501239724457264,
+            0.013565797358751297,
+            0.021184436976909637,
+            0.004109448287636042,
+            -0.015689780935645103,
+            -0.006191876716911793,
+            0.027223940938711166,
+            0.015588199719786644,
+            0.009322444908320904,
+            -0.006542795803397894,
+            -0.004328772891312838,
+            0.025376997888088226,
+            -0.006939888931810856,
+            0.018811115995049477,
+            0.01009354367852211,
+            -0.008523642085492611,
+            -0.0034607097040861845,
+            0.0010025438386946917,
+            0.006787515711039305,
+            0.003964001778513193,
+            0.01446156483143568,
+            0.018644889816641808,
+            0.012365284375846386,
+            -0.009613338857889175,
+            -0.008676014840602875,
+            0.004268747288733721,
+            -0.01913432963192463,
+            0.020205557346343994,
+            -0.019540658220648766,
+            -0.011635741218924522,
+            -0.02345617674291134,
+            0.0006851004436612129,
+            -0.0014579306589439511,
+            0.005637793801724911,
+            -0.006972210016101599,
+            -0.01859871670603752,
+            -0.029366394504904747,
+            0.01935596391558647,
+            0.0013690466294065118,
+            -0.01162650715559721,
+            -0.011700384318828583,
+            0.016807181760668755,
+            -0.005157588515430689,
+            0.01368584856390953,
+            0.0034768704790621996,
+            0.011469516903162003,
+            -0.008648310787975788,
+            0.024859854951500893,
+            0.0030913210939615965,
+            0.0005491769989021122,
+            0.007276955991983414,
+            -0.008648310787975788,
+            0.014064472168684006,
+            0.019466780126094818,
+            0.015819067135453224,
+            -0.0010123556712642312,
+            0.015717485919594765,
+            -0.00630731089040637,
+            -0.005817870609462261,
+            0.012457631528377533,
+            -0.003691577585414052,
+            0.0077987173572182655,
+            0.004176400136202574,
+            0.0026849934365600348,
+            0.020851988345384598,
+            0.005152971483767033,
+            0.020242497324943542,
+            0.02018708735704422,
+            -0.036754168570041656,
+            -0.003506883280351758,
+            -0.012235998176038265,
+            -0.016382385045289993,
+            0.034390080720186234,
+            0.006353484466671944,
+            -0.01009354367852211,
+            0.003511500544846058,
+            -0.01385207287967205,
+            -0.00631654541939497,
+            0.010989311151206493,
+            -0.005623941775411367,
+            0.006588969379663467,
+            0.006579734850674868,
+            -0.01242069248110056,
+            -0.004236425738781691,
+            0.018460195511579514,
+            -8.520467963535339e-05,
+            -0.007914151065051556,
+            -0.01242069248110056,
+            -0.0034837964922189713,
+            0.005222231615334749,
+            -0.04281214252114296,
+            -0.00917007215321064,
+            0.0011710773687809706,
+            0.0009078879957087338,
+            0.006866010837256908,
+            0.009146985597908497,
+            -0.02027943544089794,
+            0.0027773405890911818,
+            0.01893116720020771,
+            -0.03195211663842201,
+            -0.0032390763517469168,
+            -0.01206977292895317,
+            0.010509106330573559,
+            0.010795382782816887,
+            0.0011618426069617271,
+            -0.004658914171159267,
+            -0.0024310387670993805,
+            0.001940444577485323,
+            0.011201709508895874,
+            -0.0031282599084079266,
+            0.015505087561905384,
+            -0.0007918768678791821,
+            0.010758443735539913,
+            -0.00044788370723836124,
+            0.004721248522400856,
+            0.017730653285980225,
+            0.016280803829431534,
+            -0.002516459906473756,
+            -0.014378451742231846,
+            0.009548695757985115,
+            0.002673450158908963,
+            0.013020948506891727,
+            -0.0068244547583162785,
+            0.01939290203154087,
+            -0.02476750686764717,
+            0.0026665241457521915,
+            -0.001568747335113585,
+            0.003139803186058998,
+            -0.0066397604532539845,
+            -0.013981359079480171,
+            -0.001972766127437353,
+            -0.031878236681222916,
+            -0.03298640251159668,
+            -0.01649320125579834,
+            0.00920239370316267,
+            0.005102180410176516,
+            0.011109362356364727,
+            -0.005840957630425692,
+            -0.008740657940506935,
+            0.005166823510080576,
+            0.031157929450273514,
+            0.0008772979490458965,
+            -0.005060624331235886,
+            0.0028142796363681555,
+            0.013122530654072762,
+            0.02552475407719612,
+            -0.007410859223455191,
+            -0.028092004358768463,
+            -0.0006781744305044413,
+            -0.0001631947379792109,
+            -0.02375168912112713,
+            -0.02807353436946869,
+            -0.009013081900775433,
+            -0.0054207779467105865,
+            -0.0020778109319508076,
+            0.0020697305444628,
+            0.01099854614585638,
+            0.007397007197141647,
+            -0.004229499492794275,
+            0.005157588515430689,
+            -0.00984420720487833,
+            0.006898332387208939,
+            0.0036223172210156918,
+            0.01169115025550127,
+            -0.026466695591807365,
+            -0.007974176667630672,
+            -0.025469345971941948,
+            -0.007438563276082277,
+            0.006889097858220339,
+            -0.01446156483143568,
+            0.015301923267543316,
+            0.022052500396966934,
+            0.018589481711387634,
+            -0.010666096583008766,
+            0.029532620683312416,
+            0.016770243644714355,
+            -0.03461171314120293,
+            0.021221376955509186,
+            0.03014211170375347,
+            -0.01681641675531864,
+            -0.0005194527329877019,
+            -0.00033158401492983103,
+            0.019633004441857338,
+            0.012919367291033268,
+            0.010361351072788239,
+            -0.013399572111666203,
+            0.00460812309756875,
+            -0.01517263799905777,
+            -0.05197297781705856,
+            0.0039085932075977325,
+            0.024398118257522583,
+            0.004668148700147867,
+            -0.004483454395085573,
+            0.016253098845481873,
+            0.0005252244300208986,
+            -0.006787515711039305,
+            0.03169354423880577,
+            0.02744557522237301,
+            0.01922667771577835,
+            0.0016841812757775187,
+            -0.006884480360895395,
+            -0.029403334483504295,
+            -0.01676100865006447,
+            0.006404275074601173,
+            0.014110645279288292,
+            0.006990679539740086,
+            -0.01790611259639263,
+            -0.02430577203631401,
+            0.005356134846806526,
+            0.023530054837465286,
+            0.004211030434817076,
+            0.017859939485788345,
+            0.021018212661147118,
+            5.732016870751977e-05,
+            0.01582830213010311,
+            -0.008648310787975788,
+            -0.0006066053756512702,
+            -0.015754424035549164,
+            -0.00013974722241982818,
+            -0.014055237174034119,
+            0.0012859341222792864,
+            0.002643437357619405,
+            0.006173407193273306,
+            0.006621290929615498,
+            0.006672082003206015,
+            -0.009257802739739418,
+            -0.004968277178704739,
+            -3.21411389450077e-05,
+            0.016465498134493828,
+            -0.0068244547583162785,
+            0.004707396030426025,
+            -0.012143651023507118,
+            -0.004317229613661766,
+            0.008546729572117329,
+            0.0007001068443059921,
+            0.007840273901820183,
+            0.012152886018157005,
+            0.004116374533623457,
+            0.010112013667821884,
+            -0.0010764215840026736,
+            0.0024125694762915373,
+            -0.004737408831715584,
+            0.004280290566384792,
+            0.01198666077107191,
+            -0.004174091387540102,
+            -0.006187259219586849,
+            -0.02450893446803093,
+            0.037751518189907074,
+            -0.023179136216640472,
+            -0.002844292437657714,
+            -0.01057374943047762,
+            0.022514237090945244,
+            -0.0042548952624201775,
+            0.01877417601644993,
+            -0.011174005456268787,
+            -0.009368618950247765,
+            0.014784779399633408,
+            -0.006810602732002735,
+            -0.025709448382258415,
+            -0.01015818677842617,
+            -0.005707054398953915,
+            0.02421342395246029,
+            -0.007216930389404297,
+            -0.012688498944044113,
+            0.009142368100583553,
+            -0.0035761436447501183,
+            0.004940572660416365,
+            -0.020334843546152115,
+            -0.006588969379663467,
+            0.00010259191913064569,
+            -0.007553997449576855,
+            -0.009770329110324383,
+            -0.016216160729527473,
+            0.009327062405645847,
+            -0.003246002597734332,
+            0.0037423684261739254,
+            0.014156819321215153,
+            -0.008676014840602875,
+            -0.0014325352385640144,
+            0.021129028871655464,
+            0.0033129542134702206,
+            -0.004628901369869709,
+            0.009908849373459816,
+            -0.01169115025550127,
+            0.01952218823134899,
+            -0.015412740409374237,
+            -0.002085891319438815,
+            -0.007710987236350775,
+            -0.0077063702046871185,
+            -0.006584352347999811,
+            0.007004531566053629,
+            0.0031074818689376116,
+            0.021036682650446892,
+            -0.003495339769870043,
+            -0.009724155068397522,
+            -0.017832236364483833,
+            0.006805985234677792,
+            0.018848054111003876,
+            -0.000645852938760072,
+            -0.001607994781807065,
+            -0.0123375803232193,
+            -0.013141000643372536,
+            0.005268405191600323,
+            -0.0039940145798027515,
+            0.00269191968254745,
+            -0.026245061308145523,
+            -0.010028901509940624,
+            0.0045388625003397465,
+            0.0315273180603981,
+            -0.0075124409049749374,
+            -0.010527575388550758,
+            0.005106797907501459,
+            0.008075758814811707,
+            -0.01959606632590294,
+            -0.016945702955126762,
+            0.003282941412180662,
+            0.018183154985308647,
+            -0.0012697733473032713,
+            -0.0032021375373005867,
+            0.0013794356491416693,
+            -0.0016483967192471027,
+            -0.015200342051684856,
+            0.002309833187609911,
+            0.011090893298387527,
+            -0.030529970303177834,
+            0.003931680228561163,
+            -0.049498073756694794,
+            0.027464043349027634,
+            0.013713552616536617,
+            0.002952800365164876,
+            -0.008837622590363026,
+            -0.022551175206899643,
+            -0.023659341037273407,
+            -0.002936639590188861,
+            0.042073365300893784,
+            0.009096194989979267,
+            -0.010952373035252094,
+            0.022846685722470284,
+            -0.0017118854448199272,
+            -0.004234116990119219,
+            -0.01347345020622015,
+            0.0077479262836277485,
+            -0.0062011112459003925,
+            0.008117315359413624,
+            0.009373236447572708,
+            0.008232749067246914,
+            0.01800769567489624,
+            0.0010452544083818793,
+            0.010582984425127506,
+            -0.02870149537920952,
+            -0.0035899956710636616,
+            0.010906198993325233,
+            0.008528259582817554,
+            0.001420991844497621,
+            0.02031637355685234,
+            0.013584266416728497,
+            -0.00829277466982603,
+            -0.002938948106020689,
+            0.019152799621224403,
+            -0.014618555083870888,
+            0.014960238710045815,
+            0.004155621863901615,
+            0.009724155068397522,
+            0.0007053014123812318,
+            -0.001627618563361466,
+            0.025340059772133827,
+            0.003255237126722932,
+            -0.026392817497253418,
+            -0.021627703681588173,
+            0.010666096583008766,
+            -0.017869174480438232,
+            -0.00643197912722826,
+            -0.005836340133100748,
+            0.0038023940287530422,
+            0.010786147788167,
+            0.010462932288646698,
+            0.01926361583173275,
+            0.011044720187783241,
+            0.0016599401133134961,
+            -0.0030382215045392513,
+            0.013214877806603909,
+            -0.01804463379085064,
+            0.0013055579038336873,
+            -0.018654124811291695,
+            0.00895767379552126,
+            0.012125181034207344,
+            -0.00037631465238519013,
+            0.007295425049960613,
+            0.002249807585030794,
+            -0.0028373664245009422,
+            0.014877126552164555,
+            0.0079002995043993,
+            0.011257118545472622,
+            -0.027500983327627182,
+            0.0024541255552321672,
+            -0.02572791650891304,
+            0.0200208630412817,
+            -0.016530141234397888,
+            0.008463616482913494,
+            0.013094826601445675,
+            -0.0031467291992157698,
+            0.012956305406987667,
+            0.010980077087879181,
+            -0.002865070477128029,
+            0.004982129205018282,
+            0.023991791531443596,
+            -0.013861307874321938,
+            -0.05721829831600189,
+            0.0028927745297551155,
+            0.004825138952583075,
+            0.007854125462472439,
+            0.012439161539077759,
+            0.017231978476047516,
+            -0.0056978194043040276,
+            -0.014867891557514668,
+            -0.006898332387208939,
+            -0.002091662958264351,
+            -0.004252586513757706,
+            -0.0015918341232463717,
+            -0.0003933411499019712,
+            0.016253098845481873,
+            -0.0017038050573319197,
+            -0.017268918454647064,
+            0.001054489053785801,
+            -0.0022094056475907564,
+            0.004414193797856569,
+            0.01827550120651722,
+            -0.015782129019498825,
+            0.0057670800015330315,
+            0.01632697694003582,
+            0.0161976907402277,
+            0.006792133208364248,
+            0.008615989238023758,
+            -0.006191876716911793,
+            0.004991363734006882,
+            0.009456348605453968,
+            0.013944420032203197,
+            -0.006122616585344076,
+            -0.012079007923603058,
+            0.007881829515099525,
+            -0.01072150468826294,
+            0.02127678506076336,
+            -0.0030728515703231096,
+            -0.013676613569259644,
+            -0.01538503635674715,
+            -0.009761094115674496,
+            -0.009770329110324383,
+            0.005448481999337673,
+            0.001754595898091793,
+            0.003354510525241494,
+            -0.0028696877416223288,
+            0.01242069248110056,
+            0.0014094484504312277,
+            -0.016567079350352287,
+            0.0004233539802953601,
+            0.010416759178042412,
+            0.02905241586267948,
+            -0.0009754167986102402,
+            -0.018164684996008873,
+            -0.010204360820353031,
+            -0.01985463872551918,
+            0.02988353930413723,
+            0.0020085505675524473,
+            -0.0003895895497407764,
+            0.005157588515430689,
+            -0.006556647829711437,
+            -0.005517742596566677,
+            -0.007881829515099525,
+            0.01141410879790783,
+            0.0021389909088611603,
+            0.02220025658607483,
+            0.0008761436329223216,
+            0.0074708848260343075,
+            -0.006704403553158045,
+            0.004700470250099897,
+            -0.002366395900025964,
+            -0.004208721686154604,
+            0.003599230432882905,
+            0.0066397604532539845,
+            -0.0029597263783216476,
+            0.0034768704790621996,
+            0.005665497854351997,
+            -0.013547327369451523,
+            0.004811286926269531,
+            -0.02199709229171276,
+            0.030566908419132233,
+            0.006395040545612574,
+            0.015874475240707397,
+            0.004114065784960985,
+            -0.017952287569642067,
+            0.006727490108460188,
+            0.01183890551328659,
+            -0.0041209920309484005,
+            0.031213337555527687,
+            -0.002458743052557111,
+            -0.043550919741392136,
+            -0.00550850760191679,
+            0.02005780301988125,
+            0.03307875245809555,
+            0.0011791577562689781,
+            -0.01705651916563511,
+            0.0024471995420753956,
+            -0.0011197092244401574,
+            0.009142368100583553,
+            0.019577596336603165,
+            0.00720769539475441,
+            0.0040771267376840115,
+            -0.003915519453585148,
+            -0.014073706232011318,
+            -0.0031282599084079266,
+            0.020796580240130424,
+            -0.005845575127750635,
+            -0.008394356817007065,
+            0.009031551890075207,
+            0.01668713055551052,
+            -0.01863565668463707,
+            0.01070303563028574,
+            0.007443180773407221,
+            -0.02790731005370617,
+            -0.017102692276239395,
+            0.012365284375846386,
+            0.023437708616256714,
+            0.002456434303894639,
+            -0.014701667241752148,
+            -0.012079007923603058,
+            0.02245882898569107,
+            0.01642855815589428,
+            0.0061364686116576195,
+            -0.010610688477754593,
+            -0.0017430525040253997,
+            -0.011007781140506268,
+            -0.02114749886095524,
+            -0.0006423898739740252,
+            0.018349379301071167,
+            -0.014323043636977673,
+            0.008814536035060883,
+            -0.0005018490483053029,
+            0.0024310387670993805,
+            0.006339631974697113,
+            0.005799401551485062,
+            -0.011894313618540764,
+            0.005180675536394119,
+            0.0058317226357758045,
+            -0.005928687285631895,
+            0.004176400136202574,
+            0.0065058572217822075,
+            0.0012339889071881771,
+            -0.025081487372517586,
+            -0.003864728379994631,
+            -8.809052815195173e-05,
+            0.002162077696993947,
+            0.0010816160356625915,
+            -0.026965368539094925,
+            -0.012402222491800785,
+            -0.014840187504887581,
+            -0.025340059772133827,
+            -0.003996323328465223,
+            -0.01456314604729414,
+            -0.02031637355685234,
+            -0.001719965715892613,
+            -0.0035692176315933466,
+            0.0008086147718131542,
+            -0.0006602821522392333,
+            0.011072424240410328,
+            0.014036767184734344,
+            -0.0067321076057851315,
+            0.003070542821660638,
+            -0.0027911928482353687,
+            0.0323953814804554,
+            0.008583667688071728,
+            0.003964001778513193,
+            0.010832320898771286,
+            0.011183240450918674,
+            0.004116374533623457,
+            -0.006755194626748562,
+            -0.012910132296383381,
+            0.011487985961139202,
+            -0.00905463844537735,
+            -0.015985293313860893,
+            0.013464215211570263,
+            -0.010259768925607204,
+            0.0074708848260343075,
+            -0.0010060068452730775,
+            -0.015726720914244652,
+            -0.008315861225128174,
+            -0.0015952971298247576,
+            -0.01622539572417736,
+            -0.03507344797253609,
+            0.009631807915866375,
+            0.013972125016152859,
+            0.0013932876754552126,
+            -0.007397007197141647,
+            -0.028405984863638878,
+            -0.007424711249768734,
+            -0.006939888931810856,
+            0.004471911117434502,
+            0.01830320619046688,
+            0.00926703680306673,
+            -0.011469516903162003,
+            -0.008435912430286407,
+            -0.010361351072788239,
+            -0.003213681047782302,
+            0.005743992980569601,
+            0.0025372381787747145,
+            -0.000359576748451218,
+            -0.010361351072788239,
+            0.02173851989209652,
+            -0.01108165830373764,
+            0.0008034202619455755,
+            0.01448003388941288,
+            -0.026078836992383003,
+            0.0024841385893523693,
+            -0.02908935397863388,
+            -0.01681641675531864,
+            0.02648516371846199,
+            0.0025372381787747145,
+            0.013935185968875885,
+            -0.017333561554551125,
+            -0.004176400136202574,
+            0.018450962379574776,
+            -0.006921419408172369,
+            -0.025672508403658867,
+            -0.005748610477894545,
+            -0.010555279441177845,
+            -0.018432492390275,
+            0.012272937223315239,
+            0.0026064985431730747,
+            -0.014489268884062767,
+            -0.01542197447270155,
+            0.011801966466009617,
+            -0.005448481999337673,
+            -0.007807951886206865,
+            -0.02262505330145359,
+            0.026632919907569885,
+            -0.005268405191600323,
+            -0.019208207726478577,
+            -0.00136673788074404,
+            -0.014258400537073612,
+            0.0058825137093663216,
+            -0.00915160309523344,
+            0.005550064146518707,
+            0.010250533930957317,
+            -0.030326806008815765,
+            -0.009761094115674496,
+            -0.012836254201829433,
+            -0.01418452337384224,
+            -0.03911825641989708,
+            -0.005171440541744232,
+            0.011737323366105556,
+            -0.001036019646562636,
+            -0.01827550120651722,
+            -0.007609405554831028,
+            -0.013353399001061916,
+            -0.005009833257645369,
+            -0.0030543820466846228,
+            0.0073046600446105,
+            -0.007784865330904722,
+            -0.010619922541081905,
+            0.018524838611483574,
+            0.014433860778808594,
+            0.012125181034207344,
+            -0.019873108714818954,
+            -2.845266408257885e-06,
+            -0.00039622699841856956,
+            0.015135698951780796,
+            0.001699187676422298,
+            0.015034116804599762,
+            0.01057374943047762,
+            -0.020408721640706062,
+            -0.004926720634102821,
+            0.007632492575794458,
+            -0.013177938759326935,
+            -0.009419409558176994,
+            -0.0034791789948940277,
+            0.031915176659822464,
+            -0.00627498934045434,
+            0.027556391432881355,
+            -0.004074818454682827,
+            0.008994612842798233,
+            0.01527421921491623,
+            -0.0025187686551362276,
+            -0.026245061308145523,
+            -0.0038831979036331177,
+            -0.005799401551485062,
+            -0.010232064872980118,
+            -0.0006856776308268309,
+            -0.007821803912520409,
+            0.020574945956468582,
+            -0.005693201906979084,
+            0.011404873803257942,
+            0.0019889269024133682,
+            0.010813851840794086,
+            -0.016779478639364243,
+            -0.024749036878347397,
+            0.011912782676517963,
+            0.010102778673171997,
+            -0.0022255664225667715,
+            0.020925864577293396,
+            0.005448481999337673,
+            -0.0019150491571053863,
+            0.024121077731251717,
+            -0.004418811295181513,
+            -0.006667464505881071,
+            -0.01731509156525135,
+            0.0034214621409773827,
+            0.01599452644586563,
+            0.019208207726478577,
+            -0.019633004441857338,
+            0.011811201460659504,
+            0.038970500230789185,
+            0.00713381776586175,
+            -0.016945702955126762,
+            -0.026411285623908043,
+            -0.005730140954256058,
+            -0.008837622590363026,
+            0.005314578767865896,
+            -0.03274630010128021,
+            0.0002919035905506462,
+            -0.005157588515430689,
+            -0.005725523456931114,
+            0.016105344519019127,
+            0.011718854308128357,
+            -0.0038624198641628027,
+            -0.0014302266063168645,
+            0.012965540401637554,
+            -0.029957417398691177,
+            0.0003561137127690017,
+            0.008708336390554905,
+            -0.016142282634973526,
+            -0.007332364097237587,
+            0.007877212017774582,
+            -0.013353399001061916,
+            -0.004550406243652105,
+            -0.013399572111666203,
+            0.01265155989676714,
+            -0.00977956410497427,
+            -0.010398290120065212,
+            0.019651474431157112,
+            -0.004116374533623457,
+            -0.013307224959135056,
+            -0.0021828559692949057,
+            0.004026336129754782,
+            -0.03446395695209503,
+            0.020999742671847343,
+            0.005840957630425692,
+            0.00155258656013757,
+            -0.01164497621357441,
+            0.011995895765721798,
+            0.007766395807266235,
+            0.005323813296854496,
+            -0.009936554357409477,
+            -0.008126549422740936,
+            -0.008980761282145977,
+            0.003707738360390067,
+            0.011423342861235142,
+            -0.004164856858551502,
+            -0.009696451015770435,
+            0.0034122273791581392,
+            0.010582984425127506,
+            0.016964172944426537,
+            -0.0064781527034938335,
+            -0.013889011926949024,
+            0.0011878153309226036,
+            -0.002876613987609744,
+            -0.014637024141848087,
+            -0.0035345873329788446,
+            0.009123899042606354,
+            -0.016345446929335594,
+            -0.000193640444194898,
+            0.011580333113670349,
+            -0.010185890831053257,
+            -0.01918973959982395,
+            -0.025210773572325706,
+            -0.000573129509575665,
+            -0.006561265327036381,
+            0.000608914066106081,
+            -0.021553825587034225,
+            0.014720136299729347,
+            -0.0006037194980308414,
+            -0.022052500396966934,
+            0.017398204654455185,
+            -0.005069858860224485,
+            -0.00199585291557014,
+            0.030289866030216217,
+            0.00716613931581378,
+            -0.0015006412286311388,
+            -0.01972535252571106,
+            0.007821803912520409,
+            -0.005660880822688341,
+            -0.0067321076057851315,
+            -0.0007705215830355883,
+            0.02044565975666046,
+            -0.005106797907501459,
+            -0.005993330385535955,
+            0.025026079267263412,
+            0.02081504836678505,
+            0.013307224959135056,
+            -0.014904830604791641,
+            0.004610431846231222,
+            -0.026337409391999245,
+            0.022274134680628777,
+            0.009705686010420322,
+            -0.011358699761331081,
+            -0.007794099859893322,
+            0.0008917272207327187,
+            -0.014683197252452374,
+            0.025949550792574883,
+            0.011174005456268787,
+            -0.011958956718444824,
+            -0.008883796632289886,
+            -0.0023029071744531393,
+            0.0030797775834798813,
+            -0.008288157172501087,
+            -0.008551346138119698,
+            -0.004458059091120958,
+            0.018718767911195755,
+            0.005517742596566677,
+            -0.001165305613540113,
+            -0.012559212744235992,
+            -0.014646259136497974,
+            0.011958956718444824,
+            0.014341513626277447,
+            -0.004615048877894878,
+            -0.028092004358768463,
+            0.010564514435827732,
+            0.016530141234397888,
+            -0.00984420720487833,
+            0.004400341771543026,
+            -0.005730140954256058,
+            0.031675074249506,
+            -0.0019877725280821323,
+            -0.01754595898091793,
+            -0.008061906322836876,
+            0.026078836992383003,
+            0.0023410003632307053,
+            0.022237194702029228,
+            0.005891748704016209,
+            -0.0054900385439395905,
+            -0.014673963189125061,
+            0.0011179777793586254,
+            0.005864044185727835,
+            0.01198666077107191,
+            0.021479947492480278,
+            0.00395014975219965,
+            -0.006279606372117996,
+            0.00994578842073679,
+            -0.013399572111666203,
+            -0.012679263949394226,
+            0.00181231286842376,
+            -0.017435142770409584,
+            0.007600171025842428,
+            -0.023899443447589874,
+            -0.0014613937819376588,
+            0.013011714443564415,
+            0.009410175494849682,
+            0.0011231722310185432,
+            0.006099529564380646,
+            0.008385121822357178,
+            0.0008888413431122899,
+            -0.014655493199825287,
+            -0.005374604370445013,
+            -0.009225481189787388,
+            0.011377169750630856,
+            0.010435228236019611,
+            -0.010407524183392525,
+            -0.022255664691329002,
+            0.018284736201167107,
+            -0.016243863850831985,
+            0.007447798270732164,
+            -0.005032919812947512,
+            -0.0016022231429815292,
+            0.006127233617007732,
+            0.013205642811954021,
+            -0.02661444991827011,
+            0.012799316085875034,
+            -0.0021666951943188906,
+            -0.03017904981970787,
+            0.0025557074695825577,
+            -0.004317229613661766,
+            -0.0004957888158969581,
+            -0.014997177757322788,
+            0.01527421921491623,
+            -0.010619922541081905,
+            0.009350149892270565,
+            0.00536075234413147,
+            -0.0005206071073189378,
+            -0.003250619862228632,
+            0.004712013527750969,
+            -0.0014579306589439511,
+            -0.012790081091225147,
+            0.016613252460956573,
+            0.011765027418732643,
+            -0.005263787694275379,
+            0.007258486468344927,
+            -0.004656605422496796,
+            -0.014313808642327785,
+            -0.030234457924962044,
+            0.005979478359222412,
+            0.010213595815002918,
+            0.002398717449977994,
+            -0.00888841412961483,
+            -0.0008876870269887149,
+            0.0038531851023435593,
+            0.013759725727140903,
+            0.004485763143748045,
+            -0.010555279441177845,
+            -0.02476750686764717,
+            0.011866609565913677,
+            0.013002479448914528,
+            -0.012079007923603058,
+            -0.005887131206691265,
+            0.013787430711090565,
+            -0.006427362095564604,
+            0.001054489053785801,
+            0.0037746899761259556,
+            8.174165850505233e-05,
+            0.01162650715559721,
+            0.014710902236402035,
+            -0.012134416028857231,
+            0.022643523290753365,
+            -0.0015572039410471916,
+            -0.004610431846231222,
+            -0.006085677538067102,
+            0.0023525438737124205,
+            -0.011044720187783241,
+            0.0007659042021259665,
+            -0.0025764857418835163,
+            0.01698264107108116,
+            -0.0033291149884462357,
+            0.011875844560563564,
+            -0.008154253475368023,
+            0.00728157302364707,
+            -0.0005664920900017023,
+            -0.01106318924576044,
+            0.003303719451650977,
+            -0.01248533558100462,
+            0.011958956718444824,
+            -0.005600854754447937,
+            0.004275673069059849,
+            -0.022939033806324005,
+            0.017527490854263306,
+            0.009918084368109703,
+            -0.012374518439173698,
+            -0.017111927270889282,
+            0.013732021674513817,
+            -0.006048738490790129,
+            -0.047614194452762604,
+            -0.011894313618540764,
+            -0.02833210676908493,
+            0.00019970072025898844,
+            0.014720136299729347,
+            0.02816588245332241,
+            -0.012965540401637554,
+            -0.00265959813259542,
+            0.01748131588101387,
+            0.01072150468826294,
+            0.0009956177091225982,
+            0.01596682332456112,
+            -0.006939888931810856,
+            0.00965027790516615,
+            0.0034607097040861845,
+            0.0022982899099588394,
+            -0.001016973052173853,
+            -0.00042421973193995655,
+            -0.007406241726130247,
+            0.029606498777866364,
+            -0.008168105967342854,
+            0.004778965376317501,
+            -0.012162120081484318,
+            0.00811269786208868,
+            -0.0222926028072834,
+            0.0017257374711334705,
+            0.014710902236402035,
+            -0.021720051765441895,
+            0.006468918174505234,
+            0.003730825148522854,
+            -0.015680547803640366,
+            -0.0036500212736427784,
+            0.013962890021502972,
+            -0.01880188100039959,
+            -0.0025372381787747145,
+            0.004912868607789278,
+            0.012984010390937328,
+            0.008929969742894173,
+            0.004760495852679014,
+            -0.005762462504208088,
+            0.020999742671847343,
+            0.0011185549665242434,
+            0.008357417769730091,
+            0.004058657679706812,
+            0.00638118851929903,
+            0.0006348866736516356,
+            0.01599452644586563,
+            -0.009257802739739418,
+            0.003559982869774103,
+            0.019910046830773354,
+            -0.02522924356162548,
+            -0.01699187606573105,
+            0.03274630010128021,
+            0.0016726378817111254,
+            0.021978622302412987,
+            -0.0013690466294065118,
+            0.0031351859215646982,
+            0.0066397604532539845,
+            0.006214963737875223,
+            -0.0033129542134702206,
+            0.00624266779050231,
+            0.007276955991983414,
+            0.011238648556172848,
+            0.01741667278110981,
+            8.679189340909943e-05,
+            0.006057973485440016,
+            0.01347345020622015,
+            -0.02358546294271946,
+            0.006076443009078503,
+            0.005032919812947512,
+            0.007752543780952692,
+            -0.004238734487444162,
+            0.01926361583173275,
+            -0.003153655445203185,
+            -0.005254553165286779,
+            0.017527490854263306,
+            0.028646087273955345,
+            -0.0004600042593665421,
+            0.01566207781434059,
+            0.026503633707761765,
+            -0.004522701725363731,
+            -0.009936554357409477,
+            -0.006418127100914717,
+            -0.018284736201167107,
+            -0.0013078665360808372,
+            -0.0008201581658795476,
+            -0.0017569046467542648,
+            -0.01553279161453247,
+            0.0009119281312450767,
+            0.01622539572417736,
+            0.01353809330612421,
+            -0.025949550792574883,
+            0.01952218823134899,
+            0.0006902950117364526,
+            0.00324831111356616,
+            -0.00979803316295147,
+            0.014747840352356434,
+            -0.024250362068414688,
+            -0.002902009291574359,
+            0.017305856570601463,
+            -0.0161607526242733,
+            -0.012392988428473473,
+            0.0073138945735991,
+            -0.009211628697812557,
+            -0.011635741218924522,
+            0.006750577129423618,
+            0.0038624198641628027,
+            0.0108046168461442,
+            -0.00038843523361720145,
+            -0.00716613931581378,
+            -0.006588969379663467,
+            0.025173833593726158,
+            0.007383155170828104,
+            -0.005670115351676941,
+            -0.020297905430197716,
+            -0.009825737215578556,
+            -0.011977425776422024,
+            0.016650192439556122,
+            0.0017222744645550847,
+            -0.00720769539475441,
+            -0.0022648139856755733,
+            -0.013741256669163704,
+            -0.007831038907170296,
+            -0.02249576710164547,
+            0.009853441268205643,
+            0.014701667241752148,
+            0.009096194989979267,
+            0.010121248662471771,
+            0.0024218042381107807,
+            0.012300641275942326,
+            -0.000943095306865871,
+            -0.0007555151823908091,
+            0.009114664047956467,
+            0.014369217678904533,
+            -0.003474561730399728,
+            0.019946984946727753,
+            -0.012402222491800785,
+            0.009687216952443123,
+            -0.022089440375566483,
+            0.005517742596566677,
+            -0.00977956410497427,
+            0.005730140954256058,
+            0.005176058039069176,
+            -0.007503206375986338,
+            0.019873108714818954,
+            0.00545771699398756,
+            0.0006037194980308414,
+            -0.0003411073121242225,
+            0.014175288379192352,
+            0.0057901665568351746,
+            0.00893920473754406,
+            -0.01918973959982395,
+            0.0017661392921581864,
+            0.0014279178576543927,
+            0.010518341325223446,
+            -0.0028489097021520138,
+            0.010462932288646698,
+            -0.007567849475890398,
+            0.012365284375846386,
+            0.004995981231331825,
+            -0.00709226168692112,
+            0.026263531297445297,
+            0.0064781527034938335,
+            -0.0011947413440793753,
+            0.01022282987833023,
+            0.015440444462001324,
+            -0.015708250924944878,
+            -0.0012524583144113421,
+            -0.020427191630005836,
+            0.00012603943469002843,
+            0.0022255664225667715,
+            -0.02484138496220112,
+            0.0238809734582901,
+            0.023530054837465286,
+            0.0177121851593256,
+            -0.008948439732193947,
+            -0.004940572660416365,
+            -0.007928003557026386,
+            -0.0054623340256512165,
+            0.0018515604315325618,
+            -0.0005136810359545052,
+            0.023142196238040924,
+            0.009303975850343704,
+            0.035867635160684586,
+            0.01326105184853077,
+            -0.0020893544424325228,
+            0.0005399422952905297,
+            -0.004557332023978233,
+            -0.021258315071463585,
+            0.00941479206085205,
+            -0.01555126067250967,
+            -0.019041983410716057,
+            0.013205642811954021,
+            -0.00550850760191679,
+            -0.0007543608080595732,
+            -0.004416502546519041,
+            -0.008814536035060883,
+            0.015569730661809444,
+            -0.024102607741951942,
+            -0.014646259136497974,
+            0.016234630718827248,
+            -0.004765113350003958,
+            -0.017730653285980225,
+            -0.009276271797716618,
+            -0.02988353930413723,
+            -0.0038831979036331177,
+            0.02236648090183735,
+            0.002456434303894639,
+            0.0013182556722313166,
+            -0.001848097424954176,
+            -0.007678666152060032,
+            -0.006898332387208939,
+            0.002126293256878853,
+            0.021627703681588173,
+            -0.017989225685596466,
+            0.01101701520383358,
+            -0.01741667278110981,
+            -0.010001196525990963,
+            -0.0029089353047311306,
+            -0.019946984946727753,
+            0.009276271797716618,
+            0.004688926972448826,
+            -0.011127832345664501,
+            -0.0018746472196653485,
+            -0.017896879464387894,
+            -0.021165968850255013,
+            0.013667378574609756,
+            -0.003936297260224819,
+            0.012522274628281593,
+            0.008712953887879848,
+            0.006953740958124399,
+            0.002585720270872116,
+            0.058363400399684906,
+            -0.012291406281292439,
+            0.021018212661147118,
+            -0.013048653490841389,
+            0.03017904981970787,
+            -0.005365369841456413,
+            0.01781376637518406,
+            0.00918392464518547,
+            0.00460812309756875,
+            0.0018931166268885136,
+            0.0292555782943964,
+            0.022440358996391296,
+            -0.016465498134493828,
+            0.013233347795903683,
+            0.00042941427091136575,
+            0.009031551890075207,
+            -0.004982129205018282,
+            -0.011072424240410328,
+            -0.011487985961139202,
+            -0.020427191630005836,
+            -0.008509790524840355,
+            -0.00817734096199274,
+            -0.0032021375373005867,
+            -0.002143608406186104,
+            -0.003262163372710347,
+            -0.009918084368109703,
+            0.00938247051090002,
+            -0.02170158177614212,
+            0.028110474348068237,
+            0.01790611259639263,
+            0.003031295258551836,
+            0.0009298204095102847,
+            -0.003749294439330697,
+            0.007517058402299881,
+            0.0030405300203710794,
+            -0.0053699868731200695,
+            0.0003194634336978197,
+            0.01942984201014042,
+            -0.006935271434485912,
+            0.002050106879323721,
+            -0.009340914897620678,
+            0.024896793067455292,
+            -0.020168619230389595,
+            -0.0014579306589439511,
+            -0.002174775581806898,
+            -0.0004100790829397738,
+            -0.004116374533623457,
+            0.011783497408032417,
+            -0.0007058785413391888,
+            0.004146387334913015,
+            0.012263702228665352,
+            0.008482086472213268,
+            0.019005045294761658,
+            0.0216092336922884,
+            -0.005134501960128546,
+            0.0044049592688679695,
+            -0.007461650297045708,
+            -0.007226164918392897,
+            -0.012272937223315239,
+            -0.0008438221411779523,
+            -0.002327148336917162,
+            -0.022588113322854042,
+            -0.02820282056927681,
+            -0.0005673578125424683,
+            0.00369388610124588,
+            0.003262163372710347,
+            0.0034791789948940277,
+            -0.014304574579000473,
+            -0.01972535252571106,
+            0.023234544321894646,
+            -0.009834972210228443,
+            -0.02354852482676506,
+            0.013445746153593063,
+            -0.0011023941915482283,
+            -0.012900897301733494,
+            0.0027034629601985216,
+            -0.007590936031192541,
+            -0.032598547637462616,
+            0.004146387334913015,
+            -0.006150320637971163,
+            0.007447798270732164,
+            -0.033097218722105026,
+            -0.017222745344042778,
+            -0.0038347155787050724,
+            0.007475502323359251,
+            0.0015745189739391208,
+            0.062352798879146576,
+            -0.005651645828038454,
+            0.017231978476047516,
+            -0.015264985151588917,
+            -0.0012997862650081515,
+            -0.004875930026173592,
+            0.004940572660416365,
+            0.013104061596095562,
+            -0.01658554933965206,
+            -0.012134416028857231,
+            0.010961607098579407,
+            0.009017699398100376,
+            0.01276237703859806,
+            0.026208123192191124,
+            0.001071804203093052,
+            -0.014018298126757145,
+            -0.0033498930279165506,
+            -0.004520393442362547,
+            0.006621290929615498,
+            -0.006709021050482988,
+            0.0246751606464386,
+            -0.0005027148290537298,
+            -0.022846685722470284,
+            0.0003916096466127783,
+            -0.012725437991321087,
+            0.016899529844522476,
+            -0.013676613569259644,
+            -0.005217614118009806,
+            -0.017924582585692406,
+            -0.0254324059933424,
+            -0.018894227221608162,
+            0.007637109607458115,
+            0.0021701580844819546,
+            0.012291406281292439,
+            -0.0016680205008015037,
+            0.004857460502535105,
+            -0.007720222231000662,
+            0.0238809734582901,
+            -0.012291406281292439,
+            -0.010564514435827732,
+            0.002860453212633729,
+            -0.026559041813015938,
+            0.0019970072899013758,
+            0.01708422414958477,
+            -0.002918170066550374,
+            -0.008366651833057404,
+            -0.020870456472039223,
+            -0.03097323514521122,
+            0.016040701419115067,
+            -0.004843608476221561,
+            -0.003841641591861844,
+            -0.009567164815962315,
+            0.017102692276239395,
+            0.02035331353545189,
+            -0.002636511344462633,
+            0.03937682881951332,
+            0.02220025658607483,
+            -0.014747840352356434,
+            0.01658554933965206,
+            -0.016317741945385933,
+            -0.002957417629659176,
+            -0.011608037166297436,
+            0.014978708699345589,
+            -0.004079435486346483,
+            -0.030271397903561592,
+            0.025284651666879654,
+            0.0026526721194386482,
+            -0.0008380504441447556,
+            -0.006935271434485912,
+            0.0006423898739740252,
+            0.022218724712729454,
+            0.0013009405229240656,
+            -0.010379820130765438,
+            -0.010121248662471771,
+            0.009243950247764587,
+            0.009511756710708141,
+            0.0009482898749411106,
+            -0.03134262561798096,
+            0.013104061596095562,
+            0.005563916172832251,
+            -0.003269089385867119,
+            0.02123984508216381,
+            -0.012734672985970974,
+            0.021498417481780052,
+            0.020094741135835648,
+            -0.022237194702029228,
+            0.0007347370265051723,
+            -0.03387293592095375,
+            -0.014757075347006321,
+            0.00905463844537735,
+            -0.010213595815002918,
+            0.012476100586354733,
+            -0.0054023084230721,
+            -0.005291492212563753,
+            0.008925352245569229,
+            -0.01093390304595232,
+            -0.011266352608799934,
+            0.006958357989788055,
+            -0.0034122273791581392,
+            -0.007115348242223263,
+            0.003271397901698947,
+            0.012282171286642551,
+            0.020925864577293396,
+            -0.010730739682912827,
+            -0.0062934583984315395,
+            -0.009368618950247765,
+            0.028904659673571587,
+            0.00917007215321064,
+            -0.007623257581144571,
+            0.0039847795851528645,
+            0.005009833257645369,
+            0.006667464505881071,
+            0.00632577994838357,
+            0.004695852752774954,
+            0.013205642811954021,
+            0.007817186415195465,
+            0.023179136216640472,
+            -0.01446156483143568,
+            0.02345617674291134,
+            0.00627498934045434,
+            -0.010730739682912827,
+            -0.0012028217315673828,
+            0.0009032706147991121,
+            0.009567164815962315,
+            -0.022181786596775055,
+            -0.011358699761331081,
+            -0.005448481999337673,
+            -0.024859854951500893,
+            0.01926361583173275,
+            0.003287558676674962,
+            -0.0007843736675567925,
+            0.006528943777084351,
+            0.0025187686551362276,
+            0.0016934159211814404,
+            0.0018769559683278203,
+            -0.00024255557218566537,
+            -0.004224882461130619,
+            0.004252586513757706,
+            0.022902093827724457,
+            0.004321846645325422,
+            0.00014443672262132168,
+            0.0554082915186882,
+            -0.0029481828678399324,
+            -0.01141410879790783,
+            0.02245882898569107,
+            0.0018365540308877826,
+            0.011958956718444824,
+            -0.006298075895756483,
+            0.012429927475750446,
+            -0.026559041813015938,
+            -0.01099854614585638,
+            -0.03265395388007164,
+            0.017259683459997177,
+            0.002401025965809822,
+            -0.030419152230024338,
+            -0.007650961633771658,
+            0.0051852925680577755,
+            0.017158102244138718,
+            -0.011672680266201496,
+            0.026337409391999245,
+            0.007466267328709364,
+            -0.008994612842798233,
+            -0.00020330803818069398,
+            -0.01070303563028574,
+            0.006104147061705589,
+            -0.01689029484987259,
+            -0.01286395825445652,
+            -0.005356134846806526,
+            0.0019854637794196606,
+            0.00979803316295147,
+            0.02652210369706154,
+            0.022680461406707764,
+            -0.012208294123411179,
+            0.015652842819690704,
+            0.022865155711770058,
+            0.0018908079946413636,
+            0.012429927475750446,
+            -0.006376571021974087,
+            -0.024582812562584877,
+            -0.0378069244325161,
+            0.00639042304828763,
+            0.04159316048026085,
+            -0.005065241362899542,
+            0.004848225507885218,
+            0.027630269527435303,
+            0.00046029285294935107,
+            0.01714886724948883,
+            -0.005494655575603247,
+            0.005845575127750635,
+            0.01676100865006447,
+            0.0050513893365859985,
+            -0.04369867220520973,
+            0.011571098119020462,
+            0.0034976485185325146,
+            -0.005342282820492983,
+            0.014895596541464329,
+            -0.022311072796583176,
+            0.00012365861039143056,
+            0.0075632319785654545,
+            0.01334416400641203,
+            0.008200427517294884,
+            -0.004834373481571674,
+            0.004728174302726984,
+            -0.005868661683052778
+          ],
+          "title": "1 Introduction 3",
+          "keyphrases": [
+            "Introduction"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "47c02fce-72d4-4324-913d-f36ecb5160fd",
+        "properties": {
+          "page_content": "2 Switch Transformer 4 2.1 Simplifying Sparse Routing 5 2.2 Efficient Sparse Routing 6 2.3 Putting It All Together: The Switch Transformer 8 2.4 Improved Training and Fine-Tuning Techniques 8 ",
+          "embedding": [
+            0.03316810354590416,
+            0.0006708452710881829,
+            -0.017672521993517876,
+            0.002751410473138094,
+            -0.007615511305630207,
+            0.019335461780428886,
+            0.034437984228134155,
+            0.057870324701070786,
+            -0.03555668890476227,
+            0.048315975815057755,
+            -0.011119023896753788,
+            -0.004195145331323147,
+            0.03028063289821148,
+            0.0179899912327528,
+            -0.011209730058908463,
+            0.001036503235809505,
+            0.0006793489446863532,
+            0.0007010805420577526,
+            0.017612051218748093,
+            -0.04595762491226196,
+            0.018035344779491425,
+            0.0009675289620645344,
+            -0.059351854026317596,
+            0.008488554507493973,
+            -0.024520812556147575,
+            -0.01762716844677925,
+            0.012955817393958569,
+            0.012260406278073788,
+            -0.03809645399451256,
+            0.034891512244939804,
+            0.024278929457068443,
+            0.003327770624309778,
+            -0.005680453032255173,
+            -0.009335142560303211,
+            -0.03256339579820633,
+            -0.025382516905665398,
+            0.021603107452392578,
+            0.03576833754777908,
+            -0.049616094678640366,
+            0.04940444603562355,
+            0.019698284566402435,
+            -0.0011952384375035763,
+            -0.008284466341137886,
+            0.006708452943712473,
+            0.023900989443063736,
+            -0.014611199498176575,
+            -0.017581814900040627,
+            -0.004780953750014305,
+            0.02580581232905388,
+            -0.012744170613586903,
+            0.0013350766384974122,
+            -0.07056914269924164,
+            0.026440752670168877,
+            0.009962525218725204,
+            -0.013636111281812191,
+            0.03818715736269951,
+            0.009312466718256474,
+            0.05723538622260094,
+            0.035859040915966034,
+            0.03422633558511734,
+            0.017158521339297295,
+            0.002755190012976527,
+            -0.014467581175267696,
+            -0.03377280756831169,
+            0.01649334467947483,
+            -0.0113684656098485,
+            -0.03395421802997589,
+            0.013137228786945343,
+            -0.006243585143238306,
+            0.004047748167067766,
+            -0.05055338889360428,
+            -0.003365564625710249,
+            -0.05070456489920616,
+            -0.0017942748963832855,
+            -0.017218992114067078,
+            -0.003101005917415023,
+            0.03486127778887749,
+            0.03897327557206154,
+            -0.004614659585058689,
+            0.01395358145236969,
+            0.02783157490193844,
+            -0.01479261089116335,
+            0.005737144500017166,
+            0.00222796224988997,
+            0.030462045222520828,
+            -0.013583199121057987,
+            -0.01650846377015114,
+            -0.05324432998895645,
+            0.05019056424498558,
+            -0.004943468142300844,
+            -0.03510316088795662,
+            -0.027695516124367714,
+            -0.009131054393947124,
+            0.003350446932017803,
+            -0.011973170563578606,
+            0.05324432998895645,
+            0.029010750353336334,
+            -0.014240817166864872,
+            -0.0005191964446566999,
+            0.009592142887413502,
+            0.0290409866720438,
+            0.0069012027233839035,
+            -0.036191631108522415,
+            0.004875438753515482,
+            0.06062173470854759,
+            0.007555040530860424,
+            0.00041620753472670913,
+            0.0489509180188179,
+            0.025775576010346413,
+            0.0024244915693998337,
+            0.04616927355527878,
+            -0.026652399450540543,
+            -0.031898219138383865,
+            0.03431704267859459,
+            -0.004985041916370392,
+            -0.05448397621512413,
+            -0.03126328065991402,
+            -0.04299456998705864,
+            0.014127434231340885,
+            -0.022615989670157433,
+            0.039638452231884,
+            0.045413389801979065,
+            -0.037794098258018494,
+            -0.021119343116879463,
+            -0.07449973374605179,
+            0.006077291443943977,
+            0.033137865364551544,
+            0.005703129805624485,
+            -0.011277759447693825,
+            0.023326518014073372,
+            0.003941924776881933,
+            -0.03126328065991402,
+            0.00961481872946024,
+            -0.03731033578515053,
+            -0.0009084756602533162,
+            -0.022268284112215042,
+            -0.00158546247985214,
+            0.0030953367240726948,
+            -0.015162993222475052,
+            0.0004497497866395861,
+            0.0014427897986024618,
+            -0.017234109342098236,
+            0.02633492834866047,
+            0.012547641061246395,
+            -0.02922239899635315,
+            0.03516362980008125,
+            -0.013749493286013603,
+            0.0394570417702198,
+            0.021724049001932144,
+            0.028058340772986412,
+            -0.05230703577399254,
+            0.004981262609362602,
+            -0.021104225888848305,
+            0.037340570241212845,
+            0.032865747809410095,
+            0.016478227451443672,
+            -0.02264622412621975,
+            -0.0021051312796771526,
+            0.0319586917757988,
+            0.004334983415901661,
+            0.054090917110443115,
+            -0.02650122344493866,
+            -0.002630469389259815,
+            0.029116574674844742,
+            0.024263812229037285,
+            -0.0018538006115704775,
+            0.029796868562698364,
+            0.02043904922902584,
+            -0.042964331805706024,
+            0.01938081532716751,
+            0.025563929229974747,
+            0.05176280066370964,
+            0.04142233356833458,
+            0.02940380945801735,
+            -0.014550728723406792,
+            -0.004678909666836262,
+            -0.02931310422718525,
+            0.013280847109854221,
+            0.007120408583432436,
+            -0.016266580671072006,
+            -0.021391460672020912,
+            -0.023024166002869606,
+            -0.01171617116779089,
+            0.007422761060297489,
+            0.019773872569203377,
+            0.01912381500005722,
+            0.014641433954238892,
+            0.015427551232278347,
+            -0.011610347777605057,
+            0.0028893589042127132,
+            0.0160700511187315,
+            0.01514031644910574,
+            0.017234109342098236,
+            -0.0007648580940440297,
+            -0.011527200229465961,
+            -0.05650973692536354,
+            -0.01764228567481041,
+            0.004017512779682875,
+            0.005332747474312782,
+            -0.013855316676199436,
+            -0.013159905560314655,
+            0.024777812883257866,
+            0.02288810722529888,
+            -0.0030065206810832024,
+            -0.025065047666430473,
+            -0.03123304434120655,
+            -0.0004428996180649847,
+            -0.005578408949077129,
+            -0.009713083505630493,
+            -0.05061385780572891,
+            0.019531991332769394,
+            -0.020559990778565407,
+            0.015767699107527733,
+            0.0018566352082416415,
+            0.014951346442103386,
+            -0.010355583392083645,
+            -0.014656552113592625,
+            -0.004531512502580881,
+            0.009584583342075348,
+            -0.02343234233558178,
+            0.02879910357296467,
+            0.027861811220645905,
+            0.03906398266553879,
+            -0.02482316456735134,
+            -0.02157287299633026,
+            -0.015200787223875523,
+            -0.035919513553380966,
+            -0.02142169512808323,
+            -0.011232406832277775,
+            -0.01685616932809353,
+            -0.0354357473552227,
+            0.008140848949551582,
+            0.02254040166735649,
+            -0.007687320001423359,
+            -0.002987623680382967,
+            -0.05254891514778137,
+            0.0034184763208031654,
+            0.0072526875883340836,
+            -0.005121100693941116,
+            -0.02359863556921482,
+            -0.004996379837393761,
+            0.004758276976644993,
+            -0.004860321059823036,
+            -0.02025763690471649,
+            -0.004229160025715828,
+            0.014316405169665813,
+            -0.036100924015045166,
+            0.04299456998705864,
+            -0.016961991786956787,
+            -0.0025322046130895615,
+            -0.005873203277587891,
+            0.025548812001943588,
+            0.018035344779491425,
+            -0.0014890875900164247,
+            -0.004100659862160683,
+            0.03165633976459503,
+            -0.017869049683213234,
+            -0.04710656777024269,
+            -0.02063557878136635,
+            0.018730755895376205,
+            0.040908332914114,
+            -0.002668263390660286,
+            -0.0011149259516969323,
+            0.02306951954960823,
+            0.0037624025717377663,
+            -0.007793143391609192,
+            0.032775044441223145,
+            0.020454166457057,
+            -0.01676546223461628,
+            1.5036442164273467e-05,
+            0.04003151133656502,
+            -0.022872989997267723,
+            0.005820291582494974,
+            0.030114339664578438,
+            0.026984987780451775,
+            -0.037431277334690094,
+            0.002757079666480422,
+            0.026652399450540543,
+            -0.005812732502818108,
+            -0.014029170386493206,
+            0.02588140033185482,
+            0.006784040946513414,
+            -0.030507396906614304,
+            -0.00842052511870861,
+            0.016720110550522804,
+            -0.03002363257110119,
+            -0.033833280205726624,
+            0.057265620678663254,
+            -0.02560928277671337,
+            -0.03815692290663719,
+            -0.008435643278062344,
+            -0.014097199775278568,
+            -0.02069604955613613,
+            -0.010075907222926617,
+            0.05366762354969978,
+            -0.025639517232775688,
+            0.02359863556921482,
+            -0.04245033487677574,
+            0.021829871460795403,
+            -0.01729458011686802,
+            -0.008722878061234951,
+            0.010620142333209515,
+            0.005688012111932039,
+            -0.03712892532348633,
+            -0.05929138511419296,
+            0.033047161996364594,
+            0.029328221455216408,
+            0.045141272246837616,
+            -0.02385563589632511,
+            -0.001808447646908462,
+            0.0442039780318737,
+            0.02317534200847149,
+            0.010068348608911037,
+            -0.0412711575627327,
+            -0.004750718362629414,
+            -0.011360906064510345,
+            0.022570636123418808,
+            -0.013719257898628712,
+            -0.03220057487487793,
+            -0.008556583896279335,
+            -0.04934397712349892,
+            -0.011958053335547447,
+            -0.029267750680446625,
+            0.06845267117023468,
+            0.008866495452821255,
+            0.01175396516919136,
+            0.024006811901926994,
+            0.02783157490193844,
+            -0.03186798468232155,
+            0.0007024978403933346,
+            -0.008972319774329662,
+            -0.009440965950489044,
+            0.008186202496290207,
+            -0.00898743700236082,
+            -0.0029214839451014996,
+            -0.02589651755988598,
+            -0.016130521893501282,
+            0.003979718778282404,
+            -0.036463748663663864,
+            -0.018231874331831932,
+            -0.009365377947688103,
+            0.005846747197210789,
+            0.016387522220611572,
+            -0.016100287437438965,
+            -0.03537527844309807,
+            -0.03186798468232155,
+            -0.018821461126208305,
+            -0.012010964564979076,
+            -0.009289789944887161,
+            0.003407138166949153,
+            0.0008442256948910654,
+            0.023039283230900764,
+            0.023825401440262794,
+            0.002604013541713357,
+            0.028118809685111046,
+            -0.03867092356085777,
+            -0.01650846377015114,
+            0.01570722833275795,
+            0.015034493058919907,
+            -0.04284339025616646,
+            0.0247475765645504,
+            -0.03344022110104561,
+            -0.03939656913280487,
+            -0.021996166557073593,
+            -0.053123388439416885,
+            -0.04021292179822922,
+            0.0015656206523999572,
+            -0.04737868532538414,
+            -0.026607045903801918,
+            0.02940380945801735,
+            -0.0289502814412117,
+            0.005827850196510553,
+            0.010468965396285057,
+            -0.014150111004710197,
+            -0.010650377720594406,
+            -0.019017990678548813,
+            0.05212562158703804,
+            -0.011285318061709404,
+            -0.001033668639138341,
+            0.02763504534959793,
+            -0.017702756449580193,
+            0.005162674002349377,
+            -0.005812732502818108,
+            -0.004282071720808744,
+            0.022071754559874535,
+            0.024278929457068443,
+            0.04344809800386429,
+            -0.007097731810063124,
+            -0.08961737155914307,
+            0.016735227778553963,
+            0.040575746446847916,
+            -0.014807728119194508,
+            0.00541967386379838,
+            -0.023659106343984604,
+            0.05626785755157471,
+            0.011345788836479187,
+            -0.013537846505641937,
+            -0.0009372936910949647,
+            0.012668582610785961,
+            -0.005098423920571804,
+            -0.028315339237451553,
+            -0.016614286229014397,
+            -0.042329393327236176,
+            -0.006855849642306566,
+            -0.009055466391146183,
+            0.018095815554261208,
+            -0.02186010777950287,
+            0.022253166884183884,
+            0.007914084941148758,
+            0.008934524841606617,
+            0.013515169732272625,
+            -0.026108164340257645,
+            0.03247269243001938,
+            -0.04178515821695328,
+            0.00832981988787651,
+            0.012222612276673317,
+            -0.03924539312720299,
+            0.016432873904705048,
+            -0.02195081301033497,
+            -0.030144574120640755,
+            -0.011648141779005527,
+            0.030205044895410538,
+            0.024324283003807068,
+            -0.014452463947236538,
+            0.008148407563567162,
+            0.008284466341137886,
+            0.0229183416813612,
+            -0.03464962914586067,
+            0.003811534959822893,
+            0.030235279351472855,
+            0.03903374820947647,
+            0.0062398058362305164,
+            0.00339957932010293,
+            -0.004554189275950193,
+            -0.04450633376836777,
+            0.0027608589734882116,
+            -0.029978280887007713,
+            -0.008269349113106728,
+            -0.035224102437496185,
+            0.024157987907528877,
+            0.04580644890666008,
+            0.023281166329979897,
+            0.014974022284150124,
+            0.00031628936994820833,
+            -0.028738632798194885,
+            0.0008456429932266474,
+            -0.029857339337468147,
+            0.006198232527822256,
+            0.031112102791666985,
+            -0.0010667385067790747,
+            -0.028738632798194885,
+            -0.00025251181796193123,
+            -0.047167036682367325,
+            -0.03933610022068024,
+            0.017324814572930336,
+            -0.02780134044587612,
+            0.021527519449591637,
+            0.00541589455679059,
+            0.003002741141244769,
+            -0.022268284112215042,
+            0.005910997278988361,
+            0.01904822699725628,
+            0.02089257910847664,
+            -0.054000210016965866,
+            0.024067282676696777,
+            -0.024233577772974968,
+            -0.05309315025806427,
+            0.0020503299310803413,
+            -0.027332693338394165,
+            0.003017858834937215,
+            -0.0003706183924805373,
+            -0.026002340018749237,
+            0.00039353108149953187,
+            0.019169166684150696,
+            -0.014611199498176575,
+            0.033651866018772125,
+            -0.01649334467947483,
+            -0.09403172135353088,
+            0.0020408814307302237,
+            -0.025034811347723007,
+            -0.011338230222463608,
+            0.03885233402252197,
+            -0.0027627486269921064,
+            0.0054612476378679276,
+            -0.007226231973618269,
+            0.012781964614987373,
+            0.017612051218748093,
+            0.029615456238389015,
+            0.00965261273086071,
+            0.00511354161426425,
+            0.030068986117839813,
+            0.005665335804224014,
+            9.962288459064439e-05,
+            -0.0040061743929982185,
+            -0.020393695682287216,
+            -0.05152091756463051,
+            -0.022872989997267723,
+            -0.05396997556090355,
+            0.020030872896313667,
+            0.0071733202785253525,
+            -0.030129456892609596,
+            -0.0049888212233781815,
+            0.04870903491973877,
+            0.020484402775764465,
+            -0.02176940254867077,
+            -0.007056158501654863,
+            0.019108697772026062,
+            0.024157987907528877,
+            0.005593526642769575,
+            -0.001357753062620759,
+            0.010620142333209515,
+            0.0015826279995962977,
+            0.011897582560777664,
+            -0.02195081301033497,
+            -0.03090045601129532,
+            0.006889864336699247,
+            -0.028390927240252495,
+            0.022782282903790474,
+            0.006470350082963705,
+            -0.0012490949593484402,
+            0.00991717167198658,
+            0.006421217694878578,
+            -0.0027117265854030848,
+            0.023976577445864677,
+            -0.01087714172899723,
+            0.015004257671535015,
+            0.037854570895433426,
+            0.014467581175267696,
+            0.05294197425246239,
+            -0.0076684230007231236,
+            -0.010854465886950493,
+            0.008397849276661873,
+            -0.0028251090552657843,
+            -0.017778344452381134,
+            -0.00031487210071645677,
+            -0.007675981614738703,
+            0.0089949956163764,
+            0.013515169732272625,
+            -0.0398198626935482,
+            -0.011958053335547447,
+            -0.002184499055147171,
+            -0.01649334467947483,
+            -0.0004769143124576658,
+            -0.0031085647642612457,
+            8.887518924893811e-05,
+            0.03906398266553879,
+            -0.0022090652491897345,
+            -0.016977109014987946,
+            -0.00726402597501874,
+            0.02000063844025135,
+            -0.05245821177959442,
+            -0.00741520244628191,
+            0.003911689389497042,
+            0.03359139710664749,
+            -0.011217288672924042,
+            0.005767379887402058,
+            -0.028904927894473076,
+            0.01694687455892563,
+            -0.01394602283835411,
+            -0.026304693892598152,
+            -0.0039494833908975124,
+            0.0024887414183467627,
+            0.01416522916406393,
+            0.01518566906452179,
+            -0.01079399511218071,
+            0.009849142283201218,
+            0.017869049683213234,
+            0.03331927955150604,
+            -0.021300755441188812,
+            0.011255082674324512,
+            -0.004013733472675085,
+            0.03099116124212742,
+            -0.007093952503055334,
+            -0.00710907019674778,
+            0.017445756122469902,
+            0.015253698453307152,
+            -0.00209379312582314,
+            0.010998083278536797,
+            -0.013689023442566395,
+            -0.0068785264156758785,
+            0.01684105023741722,
+            -0.03168657422065735,
+            0.015321727842092514,
+            0.01139870099723339,
+            0.005332747474312782,
+            -0.009191525168716908,
+            -0.02297881245613098,
+            -0.020998401567339897,
+            0.003896571695804596,
+            -0.016357285901904106,
+            0.03368210047483444,
+            0.013809964060783386,
+            0.007354731671512127,
+            -0.020136697217822075,
+            -0.0016043595969676971,
+            0.011580112390220165,
+            0.009433407336473465,
+            0.005030394531786442,
+            0.03537527844309807,
+            -0.0637359693646431,
+            0.02684892900288105,
+            0.011232406832277775,
+            -0.004818747751414776,
+            -0.01417278777807951,
+            0.0011470509925857186,
+            -0.004542850889265537,
+            0.0102270832285285,
+            0.01056723017245531,
+            0.02368934266269207,
+            0.0020087563898414373,
+            0.014633875340223312,
+            -0.03168657422065735,
+            0.035586923360824585,
+            -0.009131054393947124,
+            0.0034241455141454935,
+            -0.0006117919692769647,
+            -0.012003405950963497,
+            0.00869264267385006,
+            -0.009645054116845131,
+            0.004576865583658218,
+            0.08411455154418945,
+            0.013568081893026829,
+            0.0101363779976964,
+            0.005291174165904522,
+            -0.027967633679509163,
+            0.03546598181128502,
+            -0.014860640279948711,
+            -0.021376343443989754,
+            0.0014824735699221492,
+            0.0035658732522279024,
+            0.009622378274798393,
+            -0.02595698833465576,
+            -0.007921643555164337,
+            -0.03897327557206154,
+            -0.010778876952826977,
+            -0.018307462334632874,
+            0.010559671558439732,
+            -0.0035450865980237722,
+            0.036826569586992264,
+            -0.01948663778603077,
+            -0.01913893222808838,
+            0.03437751531600952,
+            0.0582633838057518,
+            0.03386351466178894,
+            -0.00868508405983448,
+            0.008707760833203793,
+            0.004512615501880646,
+            -0.013106994330883026,
+            0.02124028466641903,
+            0.015813050791621208,
+            0.008155967108905315,
+            0.026879163458943367,
+            -0.010091024450957775,
+            0.03625210002064705,
+            0.023900989443063736,
+            -0.017082933336496353,
+            -0.02692451700568199,
+            -0.0052382624708116055,
+            -0.02604769356548786,
+            -0.0040666451677680016,
+            -0.00307266041636467,
+            0.006965452805161476,
+            0.005820291582494974,
+            1.740595507726539e-05,
+            0.01718875765800476,
+            -0.023326518014073372,
+            -0.018035344779491425,
+            0.007400084752589464,
+            -0.022752048447728157,
+            0.01746087335050106,
+            0.0035791012924164534,
+            -0.03138422220945358,
+            -0.009153731167316437,
+            0.028300222009420395,
+            0.000411955697927624,
+            -0.030386457219719887,
+            -0.006039496976882219,
+            0.005521717946976423,
+            -0.028103692457079887,
+            0.0013946022372692823,
+            -0.025911634787917137,
+            -0.0071959965862333775,
+            -0.004474821500480175,
+            0.008881613612174988,
+            0.009395613335072994,
+            -0.00038762573967687786,
+            -0.02394634112715721,
+            -0.015200787223875523,
+            -0.029721280559897423,
+            0.027075693011283875,
+            -0.02631981112062931,
+            0.015079845674335957,
+            -0.02553369477391243,
+            0.02246481366455555,
+            -0.01737016811966896,
+            -0.007449217140674591,
+            -0.004365218337625265,
+            0.030855102464556694,
+            0.061468325555324554,
+            -0.030598104000091553,
+            0.0010865803342312574,
+            0.010098583996295929,
+            0.003289976390078664,
+            -0.017869049683213234,
+            0.03649398311972618,
+            0.02589651755988598,
+            -3.9772385207470506e-05,
+            -0.0032918660435825586,
+            -0.013016288168728352,
+            -0.041906099766492844,
+            -0.0024263812229037285,
+            0.011648141779005527,
+            -0.0197285208851099,
+            0.02210198901593685,
+            0.031081868335604668,
+            0.01684105023741722,
+            -0.03044692613184452,
+            -0.023659106343984604,
+            -0.03625210002064705,
+            0.032593630254268646,
+            -0.013122111558914185,
+            0.043478332459926605,
+            0.014807728119194508,
+            -0.025140635669231415,
+            -0.010899818502366543,
+            -0.008722878061234951,
+            0.014399551786482334,
+            -0.02263110689818859,
+            0.027151281014084816,
+            -0.012177258729934692,
+            0.02192057855427265,
+            -0.007033482193946838,
+            -0.003972159698605537,
+            0.011617906391620636,
+            -0.007097731810063124,
+            -0.02043904922902584,
+            -0.0012557089794427156,
+            0.005975247360765934,
+            0.017581814900040627,
+            -0.0066064088605344296,
+            0.034619394689798355,
+            0.03062833845615387,
+            -0.0040326304733753204,
+            0.023356754332780838,
+            0.02774086967110634,
+            0.00572202680632472,
+            0.01966805011034012,
+            0.011557435616850853,
+            -0.001473969896323979,
+            -0.021542636677622795,
+            0.001912381500005722,
+            -0.032865747809410095,
+            0.010544554330408573,
+            0.005465026944875717,
+            -0.02279740199446678,
+            0.021542636677622795,
+            0.01702246256172657,
+            -0.019184285774827003,
+            -0.00036636655568145216,
+            -0.024898752570152283,
+            0.0007379298331215978,
+            0.02668263390660286,
+            -0.025669753551483154,
+            0.027362927794456482,
+            0.020650696009397507,
+            0.010363142006099224,
+            -0.024535929784178734,
+            -0.005619982723146677,
+            0.0035299689043313265,
+            -0.007698658388108015,
+            0.024157987907528877,
+            0.027952516451478004,
+            0.03174704313278198,
+            -0.016629403457045555,
+            0.0036811453755944967,
+            0.013061640784144402,
+            0.0036282336805015802,
+            -0.019320344552397728,
+            -0.0264861062169075,
+            0.013832640834152699,
+            0.005907217971980572,
+            0.0009306797292083502,
+            0.020862342789769173,
+            0.003520520403981209,
+            -0.004195145331323147,
+            -0.0012292531318962574,
+            -0.007127967197448015,
+            -0.029948044568300247,
+            -0.0075248051434755325,
+            0.02545810490846634,
+            0.012139464728534222,
+            0.03017481043934822,
+            -0.026380281895399094,
+            0.011239965446293354,
+            0.042510803788900375,
+            -0.018987756222486496,
+            -0.009380496107041836,
+            0.017869049683213234,
+            0.003953262697905302,
+            -0.015321727842092514,
+            -0.014308846555650234,
+            -0.009818906895816326,
+            -0.004414350725710392,
+            0.03310763090848923,
+            -0.00987937767058611,
+            -0.046108800917863846,
+            -0.004021292086690664,
+            -0.033289045095443726,
+            -0.006746246945112944,
+            7.907234248705208e-05,
+            0.023659106343984604,
+            -0.02052975445985794,
+            0.007929202169179916,
+            0.025473223999142647,
+            -0.005427232943475246,
+            0.0356171615421772,
+            0.012388906441628933,
+            -0.02580581232905388,
+            0.011821994557976723,
+            -0.025155752897262573,
+            0.01417278777807951,
+            -0.006825614254921675,
+            0.005313850473612547,
+            -0.013099434785544872,
+            -0.04411327466368675,
+            -0.05170232802629471,
+            0.0030896677635610104,
+            -0.02518598921597004,
+            0.04142233356833458,
+            0.01938081532716751,
+            -0.02105887234210968,
+            0.0025643296539783478,
+            -0.017355050891637802,
+            0.017324814572930336,
+            0.04680421203374863,
+            0.011678377166390419,
+            -0.009667730890214443,
+            0.01966805011034012,
+            -0.014059404842555523,
+            0.02376493066549301,
+            0.020121578127145767,
+            0.003779409918934107,
+            -0.03477057069540024,
+            -0.007793143391609192,
+            -0.06434067338705063,
+            0.0033806823194026947,
+            0.013076758943498135,
+            0.008246672339737415,
+            0.03809645399451256,
+            -0.0009150896221399307,
+            0.03099116124212742,
+            0.02692451700568199,
+            0.024324283003807068,
+            0.001319014118053019,
+            -0.03368210047483444,
+            -0.0024055945686995983,
+            0.04641115665435791,
+            0.038640689104795456,
+            -0.012842435389757156,
+            -0.014648993499577045,
+            -0.016735227778553963,
+            -0.005170233082026243,
+            0.026470987126231194,
+            -0.018972638994455338,
+            0.011648141779005527,
+            -0.008019908331334591,
+            -0.004353880416601896,
+            -0.010657936334609985,
+            0.007154423277825117,
+            -0.020862342789769173,
+            -0.03885233402252197,
+            -0.0290409866720438,
+            -0.03229127824306488,
+            0.02615351788699627,
+            0.02677334100008011,
+            -0.042601510882377625,
+            -0.008239113725721836,
+            -0.04226892068982124,
+            0.010771318338811398,
+            0.021451931446790695,
+            0.0014409000286832452,
+            0.01665963977575302,
+            -0.013167464174330235,
+            0.03791503980755806,
+            0.02352304756641388,
+            0.0016922308132052422,
+            -0.005986585281789303,
+            -0.007543702609837055,
+            -0.00833737850189209,
+            -0.007936760783195496,
+            0.00445592449977994,
+            0.03386351466178894,
+            -0.01888193190097809,
+            -0.008193761110305786,
+            0.004743159748613834,
+            0.0003840825520455837,
+            -0.0012216942850500345,
+            0.0161607563495636,
+            0.011965611949563026,
+            -0.004784733057022095,
+            -0.0023848076816648245,
+            -0.01364366989582777,
+            -0.004055306781083345,
+            -0.005533056333661079,
+            0.007071276195347309,
+            -0.009743318893015385,
+            -0.006666879169642925,
+            0.031172573566436768,
+            0.0013246831949800253,
+            -0.026289576664566994,
+            -0.014815287664532661,
+            0.0032351750414818525,
+            -0.0013624773127958179,
+            -0.009456084109842777,
+            0.03380304202437401,
+            0.052609387785196304,
+            -0.0049548065289855,
+            -0.023659106343984604,
+            0.010121259838342667,
+            -0.001774432952515781,
+            0.002562440000474453,
+            0.0005012442707084119,
+            0.01876099221408367,
+            0.02668263390660286,
+            -0.0038927923887968063,
+            -0.004395453725010157,
+            -0.01913893222808838,
+            0.008624613285064697,
+            0.022600872442126274,
+            0.025140635669231415,
+            -0.022041520103812218,
+            0.014921111054718494,
+            0.0019596240017563105,
+            -0.0197285208851099,
+            0.015767699107527733,
+            0.0160700511187315,
+            -0.007233790587633848,
+            0.012509847059845924,
+            -0.005174012389034033,
+            0.00825423188507557,
+            -0.010431171394884586,
+            0.010990524664521217,
+            0.01975875534117222,
+            0.008299584500491619,
+            -0.010242201387882233,
+            -0.017687639221549034,
+            0.007517246529459953,
+            0.005623762030154467,
+            -0.0018358484376221895,
+            0.017173638567328453,
+            -0.005706909112632275,
+            0.004890556447207928,
+            -0.005427232943475246,
+            -0.024551047012209892,
+            -0.004176247864961624,
+            -0.004229160025715828,
+            -0.010809112340211868,
+            0.009758437052369118,
+            -0.026365164667367935,
+            0.016901521012187004,
+            0.023235812783241272,
+            -0.0057787178084254265,
+            0.0007317882846109569,
+            -0.0089042903855443,
+            0.001393657410517335,
+            -0.017475992441177368,
+            0.017339933663606644,
+            -0.029539868235588074,
+            -0.02615351788699627,
+            0.04311550781130791,
+            -0.025684870779514313,
+            -0.026395399123430252,
+            0.0006264371913857758,
+            -0.01824699155986309,
+            -0.0022525284439325333,
+            0.009675289504230022,
+            0.006859628949314356,
+            -0.015419992618262768,
+            -0.017143404111266136,
+            0.015412434004247189,
+            0.013689023442566395,
+            -0.03238198533654213,
+            0.022963695228099823,
+            -0.014958905056118965,
+            -0.019169166684150696,
+            -0.00019806470663752407,
+            -0.0031936015002429485,
+            -0.03570786491036415,
+            -0.02562440000474453,
+            -0.006088629364967346,
+            -0.005041732918471098,
+            -0.0012708266731351614,
+            0.032865747809410095,
+            0.0005503766005858779,
+            -0.0008706816006451845,
+            -0.01605493389070034,
+            -0.009592142887413502,
+            -0.04411327466368675,
+            0.006436335388571024,
+            0.01650846377015114,
+            -0.021739166229963303,
+            -0.01334131695330143,
+            -0.0006311614997684956,
+            0.059442561119794846,
+            -0.02001575566828251,
+            0.005139997694641352,
+            0.01139870099723339,
+            -0.018715638667345047,
+            -0.030764397233724594,
+            0.014679228886961937,
+            0.03289598599076271,
+            0.0009817017707973719,
+            -0.02201128378510475,
+            -0.014323963783681393,
+            -0.011376024223864079,
+            -0.0024396092630922794,
+            0.017778344452381134,
+            0.0068180556409060955,
+            -0.022510165348649025,
+            0.025140635669231415,
+            -0.002588895848020911,
+            0.0007237570243887603,
+            -0.03192845731973648,
+            -4.650445771403611e-05,
+            0.010151495225727558,
+            0.02299392968416214,
+            -0.013099434785544872,
+            0.0008319426560774446,
+            -0.005676673725247383,
+            -0.002031432930380106,
+            -0.00829202588647604,
+            -0.0070901731960475445,
+            -0.026516340672969818,
+            -0.024263812229037285,
+            0.026108164340257645,
+            -0.005593526642769575,
+            0.012555199675261974,
+            -0.03271457180380821,
+            -0.01483040489256382,
+            0.032261043787002563,
+            0.0014616867993026972,
+            -0.025231340900063515,
+            0.019365696236491203,
+            -0.025503458455204964,
+            -0.00900255423039198,
+            0.006005482282489538,
+            0.0051551153883337975,
+            0.006859628949314356,
+            0.019259873777627945,
+            0.015949109569191933,
+            0.023190459236502647,
+            4.854179860558361e-05,
+            0.015933992341160774,
+            -0.022328754886984825,
+            -0.018201638013124466,
+            0.02798275090754032,
+            -0.0021278078202158213,
+            0.008564143441617489,
+            0.02808857522904873,
+            0.0037246085703372955,
+            -0.015737462788820267,
+            8.344228990608826e-05,
+            -0.004561747889965773,
+            0.010302671231329441,
+            0.0010998082580044866,
+            0.014958905056118965,
+            -0.005994144361466169,
+            -0.0002851092431228608,
+            0.015858404338359833,
+            0.01452805195003748,
+            0.007921643555164337,
+            -0.011572553776204586,
+            -0.01894240267574787,
+            0.0021693813614547253,
+            -0.0005267552915029228,
+            0.04281315580010414,
+            0.007222452666610479,
+            0.004985041916370392,
+            -0.05433279648423195,
+            0.004678909666836262,
+            -0.01842840388417244,
+            0.004788512364029884,
+            0.024460341781377792,
+            0.002974395640194416,
+            -0.00863973144441843,
+            0.011224847286939621,
+            -0.0015457787085324526,
+            -0.0029233735986053944,
+            0.006254923529922962,
+            -0.043024804443120956,
+            -0.02246481366455555,
+            0.01623634621500969,
+            0.008896730840206146,
+            0.022767165675759315,
+            0.016478227451443672,
+            -0.004969924222677946,
+            0.00691254111006856,
+            0.05956350266933441,
+            0.0008007625001482666,
+            -0.009425848722457886,
+            -0.027060575783252716,
+            0.00016924670489970595,
+            -0.006988129112869501,
+            -0.03422633558511734,
+            0.03906398266553879,
+            0.008435643278062344,
+            0.02001575566828251,
+            0.009690407663583755,
+            0.02439987100660801,
+            -0.011852229945361614,
+            0.011746406555175781,
+            0.014580964110791683,
+            -0.009856700897216797,
+            -0.006825614254921675,
+            -0.0015542823821306229,
+            -0.015004257671535015,
+            0.010990524664521217,
+            -0.0031180132646113634,
+            0.014399551786482334,
+            -0.0074454378336668015,
+            0.03703821823000908,
+            -0.0315958671271801,
+            0.024944106116890907,
+            0.02482316456735134,
+            0.007449217140674591,
+            -0.02421845868229866,
+            -0.007407643832266331,
+            0.012827317230403423,
+            0.017657402902841568,
+            0.021346107125282288,
+            0.025034811347723007,
+            0.015004257671535015,
+            0.03721962869167328,
+            -0.03344022110104561,
+            0.015676992014050484,
+            0.009191525168716908,
+            0.028920045122504234,
+            0.013106994330883026,
+            0.014301287941634655,
+            0.02052975445985794,
+            -0.006621526554226875,
+            -0.021618224680423737,
+            -0.0015608962858095765,
+            0.006595070473849773,
+            -0.00929734855890274,
+            -0.01545022800564766,
+            -0.01762716844677925,
+            -0.0020068667363375425,
+            0.00934270117431879,
+            -0.029721280559897423,
+            -0.020590225234627724,
+            -0.022842753678560257,
+            0.0077629080042243,
+            0.01675034500658512,
+            -0.026607045903801918,
+            0.010159053839743137,
+            0.011557435616850853,
+            0.016206109896302223,
+            0.009796231053769588,
+            0.009040349163115025,
+            0.004285851027816534,
+            0.01588863879442215,
+            -0.013673905283212662,
+            0.019637813791632652,
+            0.010476524941623211,
+            0.011625465005636215,
+            -0.012169700115919113,
+            -0.035405512899160385,
+            0.007464334834367037,
+            0.01703757978975773,
+            0.0003247930435463786,
+            -0.018549343571066856,
+            -0.0040061743929982185,
+            -0.0014862529933452606,
+            -0.0291316919028759,
+            -0.0013294074451550841,
+            -0.0020068667363375425,
+            0.01329596433788538,
+            -0.008760672062635422,
+            -0.0009996539447456598,
+            0.004656233359128237,
+            -0.020212285220623016,
+            -0.0032276161946356297,
+            -0.010559671558439732,
+            0.010385818779468536,
+            0.001661050715483725,
+            -0.005782497115433216,
+            0.010506759397685528,
+            -0.018927285447716713,
+            -0.01694687455892563,
+            0.023039283230900764,
+            0.006375864613801241,
+            -0.019320344552397728,
+            0.01167081855237484,
+            -0.016568932682275772,
+            -0.005563291721045971,
+            -0.005971467588096857,
+            -0.03168657422065735,
+            0.004690248053520918,
+            0.018398167565464973,
+            -0.005638879723846912,
+            0.011633023619651794,
+            0.009577024728059769,
+            0.018473755568265915,
+            -0.013507611118257046,
+            0.023039283230900764,
+            0.004236718639731407,
+            0.004391674418002367,
+            -0.005087085999548435,
+            -0.008662407286465168,
+            0.005162674002349377,
+            -0.0009509940282441676,
+            -1.5781988622620702e-05,
+            0.0223741065710783,
+            -0.0438411571085453,
+            0.005929894279688597,
+            0.006270041223615408,
+            0.009992759674787521,
+            0.022661343216896057,
+            0.010249760001897812,
+            -0.022600872442126274,
+            0.0049283504486083984,
+            0.005899658892303705,
+            0.006942776031792164,
+            -4.81284259876702e-05,
+            0.0014049956807866693,
+            -0.00842052511870861,
+            -0.011746406555175781,
+            0.016614286229014397,
+            0.023099754005670547,
+            -0.01148940622806549,
+            0.014853081665933132,
+            -0.024883635342121124,
+            -0.0031917118467390537,
+            0.013129670172929764,
+            0.03676610067486763,
+            -0.035859040915966034,
+            -0.010605024173855782,
+            0.020302990451455116,
+            0.005064409226179123,
+            0.011942935176193714,
+            -0.03694751113653183,
+            -0.02385563589632511,
+            -0.02193569578230381,
+            0.01788416877388954,
+            0.0003573904687073082,
+            -0.0022487489040941,
+            -0.01170861255377531,
+            -0.02010646089911461,
+            -0.010091024450957775,
+            0.012283082120120525,
+            0.00025794474640861154,
+            -0.011897582560777664,
+            -0.001873642555437982,
+            0.006069732364267111,
+            0.03555668890476227,
+            -0.011995847336947918,
+            0.02113446034491062,
+            -0.008322261273860931,
+            -0.019456403329968452,
+            -0.020514637231826782,
+            -0.010506759397685528,
+            0.0010327238123863935,
+            0.005037953611463308,
+            0.012388906441628933,
+            0.010234642773866653,
+            0.005480144638568163,
+            -0.007241349667310715,
+            0.004357659723609686,
+            -0.0060735116712749004,
+            0.0019426167709752917,
+            -0.008246672339737415,
+            -0.025911634787917137,
+            0.02492898888885975,
+            0.00829202588647604,
+            0.008586819283664227,
+            -0.006065953057259321,
+            0.0032295058481395245,
+            -0.022147342562675476,
+            0.004406792111694813,
+            0.018564462661743164,
+            -0.003560204291716218,
+            0.002891248557716608,
+            0.013424464501440525,
+            -0.02343234233558178,
+            -0.008042584173381329,
+            0.006504364777356386,
+            0.00034912300179712474,
+            0.010703288950026035,
+            0.014271052554249763,
+            -0.017959756776690483,
+            0.006077291443943977,
+            0.010552112944424152,
+            -0.009577024728059769,
+            -0.005714467726647854,
+            0.009176407940685749,
+            0.006844511721283197,
+            -0.03658469021320343,
+            0.023538164794445038,
+            -0.0037321674171835184,
+            -0.0027325134724378586,
+            0.02774086967110634,
+            -0.019879696890711784,
+            -0.019017990678548813,
+            -0.005166453309357166,
+            0.015918875113129616,
+            -0.0013719258131459355,
+            0.01048408355563879,
+            -0.0034865057095885277,
+            0.003429814474657178,
+            -0.0005040788091719151,
+            0.004818747751414776,
+            -0.01057478878647089,
+            0.00075399229535833,
+            -0.008140848949551582,
+            0.001432396355085075,
+            0.009289789944887161,
+            -0.007604172918945551,
+            -0.008186202496290207,
+            0.003972159698605537,
+            -0.0011952384375035763,
+            0.003414697013795376,
+            -0.007014585193246603,
+            0.018972638994455338,
+            -0.03126328065991402,
+            -0.0003994363942183554,
+            0.00568423280492425,
+            -0.009289789944887161,
+            0.023296283558011055,
+            0.011013200506567955,
+            0.010385818779468536,
+            0.005117321386933327,
+            0.012131906114518642,
+            -0.013258170336484909,
+            0.0008073764620348811,
+            -0.006304055918008089,
+            -0.009335142560303211,
+            0.004686468280851841,
+            0.026213988661766052,
+            0.016100287437438965,
+            0.026093047112226486,
+            0.018201638013124466,
+            -0.0002548739721532911,
+            0.004002395085990429,
+            0.004731821361929178,
+            -0.0054234531708061695,
+            0.01720387488603592,
+            -0.019078461453318596,
+            0.0028496752493083477,
+            0.007868731394410133,
+            -0.023825401440262794,
+            -0.003443042514845729,
+            -0.003862557001411915,
+            0.031535398215055466,
+            0.01851910911500454,
+            0.015102522447705269,
+            0.010763759724795818,
+            0.006738687865436077,
+            -0.0160700511187315,
+            0.03044692613184452,
+            -0.007861172780394554,
+            0.016463110223412514,
+            0.010884701274335384,
+            -0.007177099585533142,
+            0.0030839985702186823,
+            0.010703288950026035,
+            0.002562440000474453,
+            -0.022479930892586708,
+            0.020318107679486275,
+            -0.018624933436512947,
+            0.0076684230007231236,
+            0.009501436725258827,
+            -0.008866495452821255,
+            0.023296283558011055,
+            -0.01079399511218071,
+            -0.00863973144441843,
+            -0.005389438476413488,
+            -0.005272277165204287,
+            0.05578409135341644,
+            0.002758969319984317,
+            -0.019320344552397728,
+            -0.024611517786979675,
+            0.024551047012209892,
+            0.0039834980852901936,
+            -0.0061906734481453896,
+            0.021965930238366127,
+            0.012010964564979076,
+            -0.03480080887675285,
+            0.015586286783218384,
+            0.004225380253046751,
+            -0.008012348785996437,
+            0.020151814445853233,
+            0.0031028955709189177,
+            -0.006141541060060263,
+            0.030401574447751045,
+            -0.003520520403981209,
+            -0.001451293472200632,
+            0.008737996220588684,
+            -0.001432396355085075,
+            -0.017869049683213234,
+            0.05327456444501877,
+            0.001982300542294979,
+            0.025216223672032356,
+            -0.0056124236434698105,
+            -0.009100819006562233,
+            -0.017566697672009468,
+            -0.011028318665921688,
+            0.024233577772974968,
+            -0.012184818275272846,
+            -0.015110081061720848,
+            0.0011602789163589478,
+            0.013696582056581974,
+            0.0032654101960361004,
+            -0.0013568082358688116,
+            -0.006243585143238306,
+            0.03431704267859459,
+            -0.005136218387633562,
+            -0.018095815554261208,
+            -0.016085168346762657,
+            -0.0024887414183467627,
+            0.020998401567339897,
+            0.019017990678548813,
+            0.015737462788820267,
+            -0.02069604955613613,
+            -0.004890556447207928,
+            -0.013311082497239113,
+            -0.03362163156270981,
+            0.010620142333209515,
+            0.0004606155853252858,
+            0.023825401440262794,
+            -0.02579069323837757,
+            0.006825614254921675,
+            0.003658468835055828,
+            -0.04589715600013733,
+            0.019350579008460045,
+            0.014467581175267696,
+            0.00031416345154866576,
+            -0.014762375503778458,
+            0.0025529915001243353,
+            -0.0028950280975550413,
+            0.006266261916607618,
+            -0.012600553222000599,
+            -0.017869049683213234,
+            0.0024660651106387377,
+            0.01056723017245531,
+            0.023129988461732864,
+            0.001661050715483725,
+            -0.021557753905653954,
+            -0.0009731980971992016,
+            0.00042234905413351953,
+            -0.013159905560314655,
+            0.0009486319031566381,
+            0.02124028466641903,
+            0.02255551889538765,
+            0.02211710810661316,
+            -0.004274512641131878,
+            0.005703129805624485,
+            0.013144788332283497,
+            -0.007573937531560659,
+            -0.026531457901000977,
+            -0.008239113725721836,
+            -0.024112636223435402,
+            0.021633343771100044,
+            0.004414350725710392,
+            -0.00011361851647961885,
+            0.026440752670168877,
+            -0.023900989443063736,
+            0.012086553499102592,
+            0.009629936888813972,
+            -0.001953955041244626,
+            0.010053230449557304,
+            -0.00024920483701862395,
+            0.019244756549596786,
+            -0.025488341227173805,
+            0.012849994003772736,
+            -0.002855344209820032,
+            0.0067349085584282875,
+            0.015933992341160774,
+            -0.011942935176193714,
+            0.009486319497227669,
+            0.00047148141311481595,
+            -0.00418758625164628,
+            -0.011678377166390419,
+            -0.000992095097899437,
+            0.003769961418583989,
+            0.03079463355243206,
+            -0.011307994835078716,
+            0.004130895249545574,
+            0.015737462788820267,
+            0.009886936284601688,
+            0.014150111004710197,
+            0.01782369799911976,
+            0.0016667197924107313,
+            0.019002873450517654,
+            -0.014633875340223312,
+            0.006591291166841984,
+            0.009516553953289986,
+            -0.01522346306592226,
+            -0.006829394027590752,
+            -0.026289576664566994,
+            -0.0005437626386992633,
+            -0.003998615778982639,
+            -0.011610347777605057,
+            -0.009047907777130604,
+            -0.028496751561760902,
+            -0.007800702471286058,
+            -0.026213988661766052,
+            -0.007313158363103867,
+            -0.008019908331334591,
+            0.000944380066357553,
+            -0.000724229437764734,
+            0.01782369799911976,
+            -0.004678909666836262,
+            -0.004875438753515482,
+            0.021542636677622795,
+            -0.011421376839280128,
+            -0.00038644467713311315,
+            -0.012041199952363968,
+            0.0013596427161246538,
+            -0.0072526875883340836,
+            0.011549877002835274,
+            0.005752262193709612,
+            -0.00023007158597465605,
+            -0.002220403403043747,
+            0.0080803781747818,
+            -0.005465026944875717,
+            0.019002873450517654,
+            -0.03386351466178894,
+            0.01975875534117222,
+            0.010053230449557304,
+            -0.004678909666836262,
+            -0.020166931673884392,
+            -0.02052975445985794,
+            -0.021739166229963303,
+            0.005408335942775011,
+            0.026712870225310326,
+            -0.013386670500040054,
+            -0.006304055918008089,
+            -0.00480363005772233,
+            -0.01079399511218071,
+            0.0024944106116890907,
+            0.015231022611260414,
+            -0.027257105335593224,
+            0.02387075312435627,
+            -0.007914084941148758,
+            0.009902054443955421,
+            -0.008851378224790096,
+            0.003518630750477314,
+            0.00126704724971205,
+            -0.009803789667785168,
+            -0.0031917118467390537,
+            -0.018050462007522583,
+            0.010302671231329441,
+            0.025291811674833298,
+            -0.008934524841606617,
+            -0.025760458782315254,
+            0.007997231557965279,
+            -0.011005641892552376,
+            -0.012668582610785961,
+            -0.015556051395833492,
+            0.0027816458605229855,
+            0.015858404338359833,
+            0.002666373737156391,
+            0.022525284439325333,
+            0.0009061135351657867,
+            -0.0053176297806203365,
+            -0.006198232527822256,
+            0.0136058758944273,
+            0.010544554330408573,
+            -0.033289045095443726,
+            -0.008783348836004734,
+            0.0290409866720438,
+            -0.020061109215021133,
+            -0.004395453725010157,
+            0.003435483667999506,
+            0.026455869898200035,
+            -0.01171617116779089,
+            0.009629936888813972,
+            0.0003321156546007842,
+            0.001567510305903852,
+            -0.02009134367108345,
+            0.0010034333681687713,
+            0.07752326130867004,
+            0.0179899912327528,
+            -0.017128286883234978,
+            -0.00349784386344254,
+            -0.013658788055181503,
+            -0.012842435389757156,
+            0.00894964300096035,
+            -0.025382516905665398,
+            0.00860193744301796,
+            -0.00961481872946024,
+            -0.021890342235565186,
+            0.015752580016851425,
+            -0.00013605876301880926,
+            0.020408814772963524,
+            -0.03359139710664749,
+            0.012215052731335163,
+            0.015737462788820267,
+            0.01631193421781063,
+            0.01676546223461628,
+            0.0009358763927593827,
+            -0.013930905610322952,
+            -7.009624823695049e-05,
+            0.00649680569767952,
+            0.017566697672009468,
+            0.025397635996341705,
+            0.011534759774804115,
+            -0.012555199675261974,
+            0.03126328065991402,
+            0.005204247776418924,
+            0.0229183416813612,
+            0.013001170009374619,
+            -0.0021807195153087378,
+            -0.007808261085301638,
+            -0.007604172918945551,
+            0.014883317053318024,
+            0.01833769679069519,
+            -0.0036471306812018156,
+            -0.014490257948637009,
+            0.005960129667073488,
+            0.004985041916370392,
+            -0.006481688003987074,
+            -0.005438570864498615,
+            -0.0011205950286239386,
+            0.013840199448168278,
+            -0.009395613335072994,
+            0.026727987453341484,
+            0.00771755538880825,
+            0.0014087751042097807,
+            -0.015246139839291573,
+            -0.0015911315567791462,
+            -0.006617746781557798,
+            0.007728893309831619,
+            0.004796071443706751,
+            0.009350260719656944,
+            -0.0005513214273378253,
+            0.00126704724971205,
+            0.012925582006573677,
+            -0.03480080887675285,
+            -0.0007577717187814415,
+            0.023114871233701706,
+            -0.0026210208889096975,
+            -0.006889864336699247,
+            -0.009592142887413502,
+            -0.018352815881371498,
+            0.001622311770915985,
+            0.002823219168931246,
+            0.010582348331809044,
+            -0.0019917490426450968,
+            0.010703288950026035,
+            0.02885957434773445,
+            0.006727349944412708,
+            -0.016357285901904106,
+            0.0228276364505291,
+            0.006258702836930752,
+            0.008722878061234951,
+            0.014225699007511139,
+            -0.006236026529222727,
+            0.006118864752352238,
+            -0.010242201387882233,
+            -0.0066290851682424545,
+            0.02359863556921482,
+            0.01044628955423832,
+            -0.011549877002835274,
+            -0.004765836056321859,
+            0.014618758112192154,
+            -0.0265768114477396,
+            0.012744170613586903,
+            0.0019218300003558397,
+            0.013719257898628712,
+            1.0371232747274917e-05,
+            -0.016281697899103165,
+            -0.020076226443052292,
+            0.00599036505445838,
+            0.0032502925023436546,
+            -0.013054082170128822,
+            -0.018972638994455338,
+            0.0008801301009953022,
+            -0.014709463343024254,
+            -0.030159691348671913,
+            -0.021028637886047363,
+            0.014732140116393566,
+            0.0007105291006155312,
+            0.0076684230007231236,
+            0.010045671835541725,
+            -0.0025246459990739822,
+            0.007876290008425713,
+            0.02754434011876583,
+            0.014051846228539944,
+            0.018912168219685555,
+            -0.00829202588647604,
+            -0.002518976805731654,
+            0.011043435893952847,
+            0.011542318388819695,
+            0.01737016811966896,
+            -0.0018755322089418769,
+            -0.01983434334397316,
+            -0.013681463897228241,
+            0.01762716844677925,
+            -0.023371871560811996,
+            -0.02306951954960823,
+            0.013144788332283497,
+            -0.02341722510755062,
+            0.0002966836909763515,
+            -0.002492520958185196,
+            0.007573937531560659,
+            -0.003401468973606825,
+            -0.006001702975481749,
+            0.0018027785699814558,
+            -0.028451398015022278,
+            -0.03607068955898285,
+            -0.0024471678771078587,
+            0.009017672389745712,
+            -0.001181065570563078,
+            0.004221600946038961,
+            0.005933673586696386,
+            0.009070583619177341,
+            -0.01570722833275795,
+            -0.011610347777605057,
+            -0.007384967058897018,
+            0.010166613385081291,
+            0.008012348785996437,
+            0.027876928448677063,
+            -0.005449909251183271,
+            0.0035734320990741253,
+            0.01800510846078396,
+            0.009773554280400276,
+            -0.002562440000474453,
+            -0.004350101109594107,
+            0.012933140620589256,
+            -0.011345788836479187,
+            0.006292717531323433,
+            -0.019516874104738235,
+            0.0011782310903072357,
+            0.0018897049594670534,
+            0.009992759674787521,
+            0.019531991332769394,
+            -0.019199403002858162,
+            -0.0007086393889039755,
+            -0.008276907727122307,
+            -0.016795698553323746,
+            0.005669115111231804,
+            0.025216223672032356,
+            -0.004675130359828472,
+            0.018640050664544106,
+            -0.005994144361466169,
+            0.026365164667367935,
+            -0.01044628955423832,
+            0.013794846832752228,
+            0.0047167036682367325,
+            -0.004353880416601896,
+            0.008057702332735062,
+            0.01426349300891161,
+            -0.005204247776418924,
+            0.01570722833275795,
+            -0.012880229391157627,
+            0.016281697899103165,
+            0.0032011603470891714,
+            -0.0014805839164182544,
+            0.0018452969379723072,
+            0.0064627910032868385,
+            -0.02254040166735649,
+            0.011958053335547447,
+            0.01202608272433281,
+            -0.003779409918934107,
+            0.02195081301033497,
+            -0.011973170563578606,
+            -0.006973011419177055,
+            -0.0014730250695720315,
+            0.021905461326241493,
+            -0.02929798699915409,
+            0.01860981434583664,
+            0.0223741065710783,
+            -0.0005872258334420621,
+            -0.011935376562178135,
+            0.02124028466641903,
+            0.011345788836479187,
+            0.002981954487040639,
+            -0.009388054721057415,
+            0.020030872896313667,
+            -0.0006160438060760498,
+            -0.0027627486269921064,
+            0.009750877507030964,
+            -0.01087714172899723,
+            0.007577716838568449,
+            0.014769934117794037,
+            -0.00952411349862814,
+            0.0017215212574228644,
+            0.006220908835530281,
+            0.016251463443040848,
+            -0.015661874786019325,
+            -0.025820929557085037,
+            -0.018972638994455338,
+            0.024808047339320183,
+            -0.0022808739449828863,
+            -0.0064627910032868385,
+            -0.024883635342121124,
+            0.008186202496290207,
+            -0.026002340018749237,
+            0.013084317557513714,
+            -0.011519641615450382,
+            -0.002025763737037778,
+            -0.0007865896914154291,
+            0.005344085860997438,
+            0.00807281956076622,
+            0.018141167238354683,
+            0.0015618412289768457,
+            -0.014694346114993095,
+            -0.010914935730397701,
+            0.010302671231329441,
+            -0.001181065570563078,
+            -0.007048599887639284,
+            0.005657776724547148,
+            -0.005929894279688597,
+            0.0006406100001186132,
+            0.02378004789352417,
+            0.007403864059597254,
+            -0.0020975726656615734,
+            -0.005087085999548435,
+            -0.011436494998633862,
+            -0.006213350221514702,
+            -0.00021152885165065527,
+            -0.029615456238389015,
+            -0.0005579353892244399,
+            0.007815820164978504,
+            0.0039268070831894875,
+            0.0010818560840561986,
+            -0.007282922975718975,
+            -0.016916640102863312,
+            0.008473437279462814,
+            -2.8035545255988836e-05,
+            -0.02071116678416729,
+            -0.018624933436512947,
+            0.0003132185956928879,
+            0.010340466164052486,
+            0.0046335565857589245,
+            0.010506759397685528,
+            0.014543170109391212,
+            0.018035344779491425,
+            0.01746087335050106,
+            -0.002492520958185196,
+            -0.01895751990377903,
+            -0.006084850057959557,
+            0.013628552667796612,
+            -0.02833045832812786,
+            0.001375705236569047,
+            0.013636111281812191,
+            -0.0034997337497770786,
+            -0.005453688558191061,
+            -0.023674223572015762,
+            0.019093578681349754,
+            0.005933673586696386,
+            -0.004138453863561153,
+            0.00837517250329256,
+            0.010211966000497341,
+            -0.016523580998182297,
+            -0.00286101340316236,
+            0.003199270460754633,
+            -0.007294261362403631,
+            0.003316432237625122,
+            -0.01334131695330143,
+            -0.0058845411986112595,
+            -0.005079526919871569,
+            -0.002764638513326645,
+            0.003686814568936825,
+            0.012713935226202011,
+            -0.0014427897986024618,
+            -0.012245288118720055,
+            0.012910464778542519,
+            0.0003134548314847052,
+            0.019184285774827003,
+            -0.008186202496290207,
+            0.008768231607973576,
+            -0.020151814445853233,
+            -0.012751729227602482,
+            0.000203733827220276,
+            -0.012736611999571323,
+            0.015072287060320377,
+            -0.0049056741409003735,
+            0.006368305999785662,
+            0.0025511018466204405,
+            -0.01456584595143795,
+            -0.007426540832966566,
+            0.003635792527347803,
+            0.000584391294978559,
+            0.01622122712433338,
+            0.014898434281349182,
+            0.015367081388831139,
+            0.0012717714998871088,
+            0.004350101109594107,
+            -0.02938869222998619,
+            -0.0074454378336668015,
+            0.013711699284613132,
+            0.022404342889785767,
+            0.0019284439040347934,
+            0.008511231280863285,
+            0.013469817116856575,
+            -0.0009835914243012667,
+            -0.0113684656098485,
+            -0.019290108233690262,
+            0.016916640102863312,
+            -0.025911634787917137,
+            0.01417278777807951,
+            0.006640423554927111,
+            -0.003197380807250738,
+            0.0028458957094699144,
+            0.010204407386481762,
+            0.019940167665481567,
+            -0.01207143533974886,
+            -0.011806877329945564,
+            -0.012116788886487484,
+            0.019547108560800552,
+            -0.008458320051431656,
+            -0.009599701501429081,
+            0.000633523624856025,
+            -0.02080187201499939,
+            -0.003979718778282404,
+            0.01421814039349556,
+            -0.02264622412621975,
+            -0.006285158917307854,
+            0.006613967474550009,
+            -0.00873043667525053,
+            -0.02449057623744011,
+            0.00832981988787651,
+            0.006870967335999012,
+            -0.0026474767364561558,
+            0.007630628999322653,
+            -0.017702756449580193,
+            -0.04359927400946617,
+            0.010559671558439732,
+            0.0019426167709752917,
+            -0.013477375730872154,
+            0.0014342861250042915,
+            -0.003274858696386218,
+            -0.006474129389971495,
+            -0.004168689250946045,
+            -0.011648141779005527,
+            0.01965293288230896,
+            0.023009048774838448,
+            -0.007959437556564808,
+            0.0020427710842341185,
+            -0.027045458555221558,
+            0.01983434334397316,
+            -0.007709996309131384,
+            -0.0011999626876786351,
+            -0.010597465559840202,
+            0.0031784838065505028,
+            0.04311550781130791,
+            -0.015382198616862297,
+            -0.011300436221063137,
+            -0.014512934722006321,
+            0.0008928856113925576,
+            -0.02124028466641903,
+            -0.017687639221549034,
+            0.0012472053058445454,
+            -0.021210048347711563,
+            0.012600553222000599,
+            0.0022298519033938646,
+            0.013507611118257046,
+            0.00108563550747931,
+            -0.008397849276661873,
+            -0.005427232943475246,
+            -0.0223741065710783,
+            0.005831629503518343,
+            0.011814435943961143,
+            0.009221760556101799,
+            0.013855316676199436,
+            0.006198232527822256,
+            0.005616203416138887,
+            -0.0018538006115704775,
+            0.0033806823194026947,
+            -0.016266580671072006,
+            0.014588522724807262,
+            -0.018352815881371498,
+            -0.004516394808888435,
+            -0.004380336031317711,
+            -0.012003405950963497,
+            -0.0015542823821306229,
+            0.029086340218782425,
+            -0.004746939055621624,
+            -0.00023727607913315296,
+            -0.004350101109594107,
+            0.0006084850174374878,
+            0.017612051218748093,
+            0.00868508405983448,
+            0.02677334100008011,
+            -0.008889172226190567,
+            0.00495858583599329,
+            -0.022842753678560257,
+            0.004391674418002367,
+            0.006455232389271259,
+            -0.011360906064510345,
+            0.008065260946750641,
+            0.017052698880434036,
+            0.005790056195110083,
+            0.04459703713655472,
+            -0.01895751990377903,
+            0.0017479771049693227,
+            -0.03555668890476227,
+            0.023371871560811996,
+            -0.0007969830767251551,
+            0.004486159887164831,
+            -0.0028062118217349052,
+            0.011685935780405998,
+            -0.000542817753739655,
+            -0.008926966227591038,
+            0.010922495275735855,
+            0.013235493563115597,
+            -0.02140657790005207,
+            0.0011952384375035763,
+            -0.035314805805683136,
+            0.0029233735986053944,
+            0.0004955751355737448,
+            0.006326732225716114,
+            -0.0066442028619349,
+            -0.01800510846078396,
+            0.01021952461451292,
+            -0.00873043667525053,
+            0.00952411349862814,
+            0.0008295804727822542,
+            0.023825401440262794,
+            -0.020680932328104973,
+            0.03156563267111778,
+            -0.0004497497866395861,
+            -0.005117321386933327,
+            0.013106994330883026,
+            -0.011111465282738209,
+            -0.004599541891366243,
+            -0.01563164032995701,
+            -0.008216436952352524,
+            0.013122111558914185,
+            0.002330006333068013,
+            0.015321727842092514,
+            -0.011935376562178135,
+            0.009660172276198864,
+            0.017566697672009468,
+            -0.007838496007025242,
+            0.0007502128719352186,
+            0.004229160025715828,
+            0.011050994507968426,
+            0.005805173888802528,
+            -0.00930490717291832,
+            -0.006334291305392981,
+            -0.006750026252120733,
+            -0.022071754559874535,
+            -0.01667475700378418,
+            -0.012222612276673317,
+            -0.00526471808552742,
+            0.004391674418002367,
+            0.0061566587537527084,
+            -0.0018178962636739016,
+            0.008488554507493973,
+            -0.0006769868195988238,
+            -0.016553815454244614,
+            0.0006580897606909275,
+            0.013182582333683968,
+            0.015155433677136898,
+            0.007740231696516275,
+            0.0011309884721413255,
+            0.029086340218782425,
+            0.0013794846599921584,
+            -0.007861172780394554,
+            0.005060629919171333,
+            0.0003578628820832819,
+            -0.025034811347723007,
+            0.007543702609837055,
+            -0.019531991332769394,
+            0.005083306692540646,
+            -0.016372404992580414,
+            0.007929202169179916,
+            0.028632810339331627,
+            0.009115937165915966,
+            -0.0006354133365675807,
+            -0.012479611672461033,
+            0.007997231557965279,
+            -0.01596422865986824,
+            -0.01727946288883686,
+            0.011081229895353317,
+            5.4535707022296265e-05,
+            0.011194612830877304,
+            0.014694346114993095,
+            -0.008737996220588684,
+            -0.02043904922902584,
+            -0.00034061932819895446,
+            -0.015193228609859943,
+            0.00864729005843401,
+            0.01074864249676466,
+            0.0011461060494184494,
+            -0.014905992895364761,
+            0.014989140443503857,
+            0.0042782919481396675,
+            -0.00987181905657053,
+            -0.016735227778553963,
+            -0.03090045601129532,
+            0.01210922934114933,
+            0.011171936057507992,
+            0.025563929229974747,
+            -0.009645054116845131,
+            -0.005941232666373253,
+            -0.011648141779005527,
+            0.004323645029217005,
+            0.005948791280388832,
+            -0.019637813791632652,
+            0.014905992895364761,
+            -0.01382508222013712,
+            -0.0024963002651929855,
+            0.0027533001266419888,
+            -0.0024944106116890907,
+            -0.00895720161497593,
+            -0.005465026944875717,
+            0.010400936007499695,
+            -0.0032238366547971964,
+            0.0014853081665933132,
+            0.013212817721068859,
+            -0.0006325787398964167,
+            0.0199099313467741,
+            -0.02376493066549301,
+            0.03271457180380821,
+            0.006202011834830046,
+            3.097344597335905e-05,
+            0.00033376915962435305,
+            -0.008019908331334591,
+            -0.017491109669208527,
+            0.00215048436075449,
+            0.02299392968416214,
+            -0.004471042193472385,
+            -0.0113684656098485,
+            0.00046510365791618824,
+            0.0060735116712749004,
+            -0.013673905283212662,
+            -0.012970935553312302,
+            -0.01267614122480154,
+            0.01359831728041172,
+            -0.00378318945877254,
+            -0.005158894695341587,
+            0.029902692884206772,
+            0.0063002766110002995,
+            0.030945809558033943,
+            -0.013537846505641937,
+            -0.005412115249782801,
+            4.6858778659952804e-05,
+            0.013893111608922482,
+            0.004308527335524559,
+            -0.02122516743838787,
+            0.009403171949088573,
+            0.00011208312935195863,
+            0.017218992114067078,
+            -0.0033353292383253574,
+            -0.009697966277599335,
+            -0.005865644197911024,
+            0.015276375226676464,
+            -0.003953262697905302,
+            0.011436494998633862,
+            0.006149100139737129,
+            0.016568932682275772,
+            0.012623229064047337,
+            -0.01232843566685915,
+            0.005858085583895445,
+            -0.009939848445355892,
+            0.011897582560777664,
+            -0.02107398957014084,
+            -0.007539922837167978,
+            0.01175396516919136,
+            0.01390822883695364,
+            0.00245661661028862,
+            -0.0011621685698628426,
+            0.0009722532122395933,
+            0.0043387627229094505,
+            -0.005653997417539358,
+            0.012865111231803894,
+            -0.01176152378320694,
+            0.013938464224338531,
+            0.004731821361929178,
+            -0.007460555527359247,
+            -0.000914617208763957,
+            0.0007818654412403703,
+            -0.004463483113795519,
+            -0.006538379471749067,
+            -0.01176152378320694,
+            -0.016689874231815338,
+            -0.014490257948637009,
+            -0.01202608272433281,
+            0.014701904729008675,
+            0.011897582560777664,
+            0.00860193744301796,
+            0.008662407286465168,
+            0.011376024223864079,
+            -0.01885169744491577,
+            0.0071733202785253525,
+            -0.002624800195917487,
+            -0.003779409918934107,
+            0.02087746001780033,
+            -0.00232622679322958,
+            -0.014293728396296501,
+            0.0023791384883224964,
+            0.01718875765800476,
+            0.01513275783509016,
+            -0.00019511203572619706,
+            0.006663099862635136,
+            0.004387895110994577,
+            0.02281251922249794,
+            -0.019426167011260986,
+            -0.0032219470012933016,
+            -0.007815820164978504,
+            0.012192376889288425,
+            0.02263110689818859,
+            -0.004523953888565302,
+            -0.015276375226676464,
+            -0.03458916023373604,
+            0.013832640834152699,
+            -0.014671669341623783,
+            0.0014390103751793504,
+            0.013568081893026829,
+            -0.028315339237451553,
+            0.013749493286013603,
+            -0.022495048120617867,
+            0.007675981614738703,
+            0.017793461680412292,
+            -0.016977109014987946,
+            -0.016130521893501282,
+            -0.0036849246826022863,
+            0.004244277253746986,
+            0.025594163686037064,
+            -0.020590225234627724,
+            -0.029706163331866264,
+            -0.0022525284439325333,
+            -0.007808261085301638,
+            -0.011897582560777664,
+            -0.008004790171980858,
+            -0.009554348886013031,
+            0.02492898888885975,
+            -0.019093578681349754,
+            0.003085888223722577,
+            -0.02993292734026909,
+            -0.0066064088605344296,
+            -0.011315553449094296,
+            -0.007573937531560659,
+            -0.019199403002858162,
+            -0.005665335804224014,
+            -0.010725965723395348,
+            -0.0004521119117271155,
+            -0.008050143718719482,
+            0.0004632139462046325,
+            -0.019199403002858162,
+            0.012131906114518642,
+            0.008231555111706257,
+            0.011685935780405998,
+            0.013507611118257046,
+            -0.007951878942549229,
+            -0.01966805011034012,
+            -0.00599036505445838,
+            0.01114925928413868,
+            0.0017319145845249295,
+            0.009085701778531075,
+            -0.0057295854203403,
+            0.016901521012187004,
+            -0.025775576010346413,
+            0.004119556862860918,
+            0.018806343898177147,
+            0.007400084752589464,
+            0.01539731677621603,
+            -0.02054487355053425,
+            0.011648141779005527,
+            -0.00618311483412981,
+            -0.0016988448332995176,
+            -6.315010068647098e-06,
+            -0.003911689389497042,
+            -0.005676673725247383,
+            0.019093578681349754,
+            0.016190992668271065,
+            -0.010393377393484116,
+            -0.002549211960285902,
+            0.0016260911943390965,
+            -0.015556051395833492,
+            -0.004433247726410627,
+            -0.00013015343574807048,
+            -0.0034090278204530478,
+            0.005548174027353525,
+            0.0041497922502458096,
+            -0.012842435389757156,
+            -0.005053071305155754,
+            0.0019445064244791865,
+            -0.00023975632211659104,
+            -0.01711316779255867,
+            -0.007959437556564808,
+            -0.00476205674931407,
+            -0.009894495829939842,
+            -0.004637335892766714,
+            0.004183806944638491,
+            0.02210198901593685,
+            -0.0010005987714976072,
+            -0.005767379887402058,
+            -0.001888760132715106,
+            -0.0203634612262249,
+            0.0026153516955673695,
+            0.00038644467713311315,
+            -0.0020919034723192453,
+            -0.016100287437438965,
+            0.008715319447219372,
+            0.002876131096854806,
+            0.017581814900040627,
+            -0.026546575129032135,
+            -0.002184499055147171,
+            -0.014550728723406792,
+            -0.01426349300891161,
+            -0.02615351788699627,
+            0.01417278777807951,
+            -0.011685935780405998,
+            -0.0012207494582980871,
+            -0.01879122667014599,
+            -0.011481847614049911,
+            0.0073622907511889935,
+            0.003960821777582169,
+            0.022268284112215042,
+            -0.03582880645990372,
+            -0.028239751234650612,
+            0.00726402597501874,
+            -0.006761364638805389,
+            0.006304055918008089,
+            -0.019244756549596786,
+            0.010597465559840202,
+            -0.009697966277599335,
+            -0.021965930238366127,
+            -0.0006708452710881829,
+            0.02308463677763939,
+            -0.029086340218782425,
+            -0.04021292179822922,
+            -0.02569998800754547,
+            0.016266580671072006,
+            0.006836952641606331,
+            0.022389225661754608,
+            0.014082081615924835,
+            -0.0040666451677680016,
+            -0.007298040669411421,
+            -0.010544554330408573,
+            0.0036509099882096052,
+            0.019804108887910843,
+            -0.011890023946762085,
+            -0.0061075263656675816,
+            0.00015991629334166646,
+            0.021346107125282288,
+            -0.0009604425868019462,
+            0.013938464224338531,
+            0.006213350221514702,
+            -0.017400404438376427,
+            -0.004637335892766714,
+            0.0001971198507817462,
+            -0.005170233082026243,
+            0.0018972638063132763,
+            0.0011744516668841243,
+            -0.0025038591120392084,
+            0.010862024500966072,
+            0.007974554784595966,
+            -0.004622218664735556,
+            -0.013575640507042408,
+            -0.003023528028279543,
+            0.0017999439733102918,
+            -0.018730755895376205,
+            0.021799637004733086,
+            0.008889172226190567,
+            -0.0054839239455759525,
+            -0.004773394670337439,
+            0.0052382624708116055,
+            0.022147342562675476,
+            0.004263174720108509,
+            0.005298732779920101,
+            0.007638187613338232,
+            0.007071276195347309,
+            0.00863973144441843,
+            -0.0014796389732509851,
+            0.01052943617105484,
+            -0.003289976390078664,
+            0.017400404438376427,
+            -0.002144815167412162,
+            -0.015737462788820267,
+            -0.0009599701152183115,
+            -0.007475673221051693,
+            0.009622378274798393,
+            -0.00995496567338705,
+            0.011474289000034332,
+            0.015427551232278347,
+            0.008881613612174988,
+            0.0011961832642555237,
+            0.010189289227128029,
+            -0.014323963783681393,
+            0.027514105662703514,
+            -0.00039683806244283915,
+            -0.0010270546190440655,
+            0.009645054116845131,
+            0.005544394254684448,
+            -0.00041526267887093127,
+            -0.0011054774513468146,
+            0.018473755568265915,
+            -0.006663099862635136,
+            -0.02606281079351902,
+            0.022434577345848083,
+            -0.01974363811314106,
+            -0.018640050664544106,
+            0.010007877834141254,
+            -0.006867188028991222,
+            0.009577024728059769,
+            0.004459703806787729,
+            0.009206642396748066,
+            0.03168657422065735,
+            0.005438570864498615,
+            -0.01966805011034012,
+            -0.002471734071150422,
+            -0.010824230499565601,
+            0.008571702055633068,
+            -0.02176940254867077,
+            -0.0059223356656730175,
+            -0.009947407059371471,
+            0.004051527474075556,
+            -0.0021183593198657036,
+            -0.0018434072844684124,
+            0.017959756776690483,
+            0.0021674917079508305,
+            0.012131906114518642,
+            -0.001987969735637307,
+            -0.004769615363329649,
+            0.03186798468232155,
+            -0.0008933580247685313,
+            -0.014233258552849293,
+            -0.004951027221977711,
+            -0.0053176297806203365,
+            -0.039789628237485886,
+            -0.01083178911358118,
+            -0.0023696899879723787,
+            0.01703757978975773,
+            -0.0011120913550257683,
+            0.00833737850189209,
+            -0.007698658388108015,
+            -0.00695789372548461,
+            0.0004176248039584607,
+            0.003666027681902051,
+            -0.0009273727191612124,
+            0.02149728499352932,
+            -0.007793143391609192,
+            0.002653145929798484,
+            -0.002394256182014942,
+            -0.02817928045988083,
+            -0.02527669444680214,
+            -0.002486851764842868,
+            -0.022510165348649025,
+            0.0024981899186968803,
+            0.013205258175730705,
+            -0.012131906114518642,
+            -0.026017459109425545,
+            0.0031104544177651405,
+            -0.02379516512155533,
+            0.03613115847110748,
+            0.011655700393021107,
+            -0.0028591237496584654,
+            -0.014603639952838421,
+            -0.007581496611237526,
+            -0.0037718513049185276,
+            -0.020937930792570114,
+            -0.006791599560528994,
+            0.009289789944887161,
+            -0.0014210582012310624,
+            0.0016147529240697622,
+            0.0021618225146085024,
+            -0.04220845177769661,
+            -0.016251463443040848,
+            0.013961140997707844,
+            -0.0319586917757988,
+            0.018670285120606422,
+            0.03088533878326416,
+            -0.0011215399717912078,
+            0.0031028955709189177,
+            -0.0033258807379752398,
+            0.005257159471511841,
+            0.010295112617313862,
+            0.005646438337862492,
+            -0.00903278961777687,
+            -0.010967847891151905,
+            -0.015949109569191933,
+            0.025140635669231415,
+            0.009108378551900387,
+            -0.01263078860938549,
+            0.008768231607973576,
+            0.011353347450494766,
+            0.004002395085990429,
+            -0.004603321198374033,
+            0.0035715424455702305,
+            -0.02501969411969185,
+            -0.005895879585295916,
+            -0.0032219470012933016,
+            0.013099434785544872,
+            0.01267614122480154,
+            0.0037246085703372955,
+            0.00495858583599329,
+            0.007736452389508486,
+            -0.002653145929798484,
+            -0.0060735116712749004,
+            -0.02387075312435627,
+            0.020061109215021133,
+            -0.00042943545849993825,
+            0.0036981527227908373,
+            0.0080803781747818,
+            0.0025246459990739822,
+            0.002176940208300948,
+            -0.008012348785996437,
+            0.011519641615450382,
+            0.01903310790657997,
+            0.016115404665470123,
+            -0.006220908835530281,
+            0.007589055225253105,
+            -0.01259299460798502,
+            0.003051873529329896,
+            0.010635259561240673,
+            -0.002365910680964589,
+            -0.008307143114507198,
+            -0.025473223999142647,
+            0.00033754855394363403,
+            -0.01083934772759676,
+            -0.0007251743227243423,
+            -0.0002626689965836704,
+            0.0010629590833559632,
+            -0.00726402597501874,
+            -0.018141167238354683,
+            0.0011952384375035763,
+            0.0026588148903101683,
+            0.020302990451455116,
+            -0.027317576110363007,
+            -0.002471734071150422,
+            0.017763227224349976,
+            0.009886936284601688,
+            0.007778025697916746,
+            0.019290108233690262,
+            0.005982805974781513,
+            0.0013719258131459355,
+            -0.007033482193946838,
+            -0.002777866320684552,
+            0.0197285208851099,
+            -0.002549211960285902,
+            0.008276907727122307,
+            -0.0145204933360219,
+            -0.014550728723406792,
+            -0.016871286556124687,
+            0.010514318943023682,
+            -0.013507611118257046,
+            -0.008261790499091148,
+            0.016992228105664253,
+            -0.002679601777344942,
+            -0.011958053335547447,
+            0.01718875765800476,
+            0.0015892419032752514,
+            0.0070674968883395195,
+            -0.003622564487159252,
+            0.0023791384883224964,
+            0.006326732225716114,
+            -0.0019019880564883351,
+            0.0035016234032809734,
+            0.005381879862397909,
+            -0.026032576337456703,
+            0.019154049456119537,
+            0.004372777417302132,
+            0.020665813237428665,
+            -0.009478759951889515,
+            0.017687639221549034,
+            -0.002324337139725685,
+            0.00802746694535017,
+            0.020378578454256058,
+            -0.016977109014987946,
+            0.008397849276661873,
+            0.003934365697205067,
+            0.015556051395833492,
+            -0.010378260165452957,
+            -0.0022449695970863104,
+            -0.012691258452832699,
+            -0.00199552858248353,
+            -0.006780261639505625,
+            -0.0024755136109888554,
+            -0.0010185509454458952,
+            -0.0005735254962928593,
+            -0.0030613220296800137,
+            -0.006602629087865353,
+            0.005249600391834974,
+            0.021875225007534027,
+            -0.009320025332272053,
+            -0.016478227451443672,
+            0.004017512779682875,
+            0.0018452969379723072,
+            0.021708931773900986,
+            0.0401826873421669,
+            -0.011890023946762085,
+            0.02376493066549301,
+            -0.019063344225287437,
+            -0.011481847614049911,
+            0.008798466064035892,
+            0.00307266041636467,
+            -0.013054082170128822,
+            -0.035586923360824585,
+            0.00587698258459568,
+            0.012645905837416649,
+            -0.009660172276198864,
+            0.02113446034491062,
+            0.014384434558451176,
+            0.005449909251183271,
+            -0.017869049683213234,
+            0.020151814445853233,
+            -0.006674438249319792,
+            0.0055670710280537605,
+            -0.00898743700236082,
+            0.016175875440239906,
+            -0.00648924708366394,
+            0.013666346669197083,
+            -0.016342168673872948,
+            0.004618438892066479,
+            0.008964760228991508,
+            0.02843628078699112,
+            -0.02133098989725113,
+            0.014497816562652588,
+            -0.016432873904705048,
+            -0.004875438753515482,
+            -0.005442350637167692,
+            0.009697966277599335,
+            -0.0001299172145081684,
+            0.006228467915207148,
+            -0.00011285082291578874,
+            -0.016357285901904106,
+            -0.0007147809374146163,
+            -0.005257159471511841,
+            -0.010703288950026035,
+            -0.0003616423055063933,
+            0.0022865431383252144,
+            0.005918555893003941,
+            0.01298605278134346,
+            0.011421376839280128,
+            -0.01201852411031723,
+            0.012736611999571323,
+            -0.041633982211351395,
+            0.001619477174244821,
+            -0.004289630334824324,
+            -0.0005791945732198656,
+            0.0007459610351361334,
+            -0.015661874786019325,
+            -0.012124347500503063,
+            -0.0070674968883395195,
+            0.010196847841143608,
+            -0.00552927702665329,
+            -0.009765995666384697,
+            -0.011769082397222519,
+            0.007592834532260895,
+            0.001629870617762208,
+            0.006625305861234665,
+            0.010552112944424152,
+            0.012260406278073788,
+            -0.033047161996364594,
+            -0.01263834722340107,
+            -0.019607579335570335,
+            0.00012212219007778913,
+            -0.005623762030154467,
+            -0.0010818560840561986,
+            -0.014422228559851646,
+            -0.02553369477391243,
+            -0.0010998082580044866,
+            0.0154351107776165,
+            0.013175023719668388,
+            -0.004206483252346516,
+            0.009796231053769588,
+            -0.0034676087088882923,
+            0.004180027637630701,
+            -0.018141167238354683,
+            0.002394256182014942,
+            0.004100659862160683,
+            0.0036905938759446144,
+            0.020726284012198448,
+            -0.019788991659879684,
+            0.01921452023088932,
+            0.014278611168265343,
+            0.0019936386961489916,
+            0.011028318665921688,
+            -0.0011801207438111305,
+            0.0066442028619349,
+            -0.008465878665447235,
+            -0.012608111836016178,
+            0.005559511948376894,
+            -0.0010506759863346815,
+            0.008405407890677452,
+            0.006504364777356386,
+            0.005627541337162256,
+            0.008087937720119953,
+            0.020302990451455116,
+            -0.008163525722920895,
+            0.0028043221682310104,
+            -0.01596422865986824,
+            -0.0006836007814854383,
+            -0.0009590252884663641,
+            -0.004414350725710392,
+            0.013205258175730705,
+            0.0024736239574849606,
+            -0.013885552063584328,
+            -0.011391141451895237,
+            -0.02405216544866562,
+            0.011973170563578606,
+            0.014437345787882805,
+            0.003580990945920348,
+            0.010620142333209515,
+            -0.004951027221977711,
+            0.001301061944104731,
+            0.01545022800564766,
+            -0.007033482193946838,
+            -0.028904927894473076,
+            0.0055670710280537605,
+            0.010234642773866653,
+            0.023553283885121346,
+            -0.0020767857786267996,
+            -0.004845203831791878,
+            0.00418758625164628,
+            0.0017914402997121215,
+            0.014414669945836067,
+            0.005638879723846912,
+            0.012230170890688896,
+            0.019063344225287437,
+            -0.006013041362166405,
+            0.007010805420577526,
+            0.005752262193709612,
+            0.01694687455892563,
+            -0.0160700511187315,
+            -7.039151387289166e-05,
+            0.02122516743838787,
+            0.009486319497227669,
+            0.01206387672573328,
+            -0.010068348608911037,
+            0.014588522724807262,
+            0.0036093364469707012,
+            0.0019142711535096169,
+            -0.004860321059823036,
+            0.0018065579934045672,
+            0.00434632133692503,
+            0.0047167036682367325,
+            0.02842116355895996,
+            -0.0028723515570163727,
+            0.008050143718719482,
+            -0.040243156254291534,
+            0.016810815781354904,
+            0.00032148606260307133,
+            0.009131054393947124,
+            -0.0036527998745441437,
+            0.018367933109402657,
+            0.03088533878326416,
+            0.014104758389294147,
+            0.00511354161426425,
+            0.005771159194409847,
+            0.0034524910151958466,
+            -0.006413658615201712,
+            -0.011028318665921688,
+            0.004323645029217005,
+            0.004667571280151606,
+            -0.0074454378336668015,
+            0.014505375176668167,
+            0.0032994248904287815,
+            -0.014286169782280922,
+            -0.004992600530385971,
+            0.013273287564516068,
+            0.0006491136737167835,
+            -0.010680613107979298,
+            0.03483104333281517,
+            -0.012426700443029404,
+            -0.0016100286738947034,
+            -0.0001238937838934362,
+            -0.01563164032995701,
+            -0.0069843498058617115,
+            0.020333226770162582,
+            -0.014490257948637009,
+            -0.010884701274335384,
+            -0.037340570241212845,
+            0.012237729504704475,
+            0.0029082559049129486,
+            0.003282417543232441,
+            -0.014724581502377987,
+            -0.03942680358886719,
+            0.005022835917770863,
+            0.011194612830877304,
+            -0.007846055552363396,
+            0.009229319170117378,
+            0.00679915864020586,
+            -0.01198073010891676,
+            0.012789523229002953,
+            0.029691044241189957,
+            0.01817140355706215,
+            -0.01210922934114933,
+            0.004773394670337439,
+            0.024702223017811775,
+            0.008677525445818901,
+            -0.013522729277610779,
+            0.020756520330905914,
+            -0.02798275090754032,
+            0.007543702609837055,
+            0.0159793458878994,
+            -0.0014191684313118458,
+            0.0035847704857587814,
+            -8.899329259293154e-05,
+            0.015933992341160774,
+            0.004478600807487965,
+            0.008889172226190567,
+            0.03235175088047981,
+            -0.010673053562641144,
+            -0.00025322046712972224,
+            -0.013734376057982445,
+            -0.009463642723858356,
+            -0.0015391646884381771,
+            0.00645145308226347,
+            -0.02254040166735649,
+            -0.0007809206144884229,
+            0.002675822237506509,
+            -0.026274457573890686,
+            0.010642818175256252,
+            0.020680932328104973,
+            -0.016825933009386063,
+            -2.1598738385364413e-05,
+            0.014550728723406792,
+            0.017067816108465195,
+            0.010673053562641144,
+            -0.012509847059845924,
+            -0.0007492680451832712,
+            -0.009493878111243248,
+            -0.008722878061234951,
+            0.004229160025715828,
+            -0.02394634112715721,
+            0.03833833336830139,
+            -0.01229064166545868,
+            0.006024379748851061,
+            0.015601404942572117,
+            -0.004387895110994577,
+            -0.004973703529685736,
+            0.004667571280151606,
+            0.011557435616850853,
+            -0.023659106343984604,
+            0.03340998291969299,
+            -0.016024697571992874,
+            0.00491323322057724,
+            0.01703757978975773,
+            0.027075693011283875,
+            -0.007638187613338232,
+            0.0013416905421763659,
+            -0.00026030687149614096,
+            0.028889810666441917,
+            -4.538244684226811e-05,
+            -0.02325093001127243,
+            -0.01895751990377903,
+            -0.005136218387633562,
+            0.01938081532716751,
+            0.00757015822455287,
+            -0.015752580016851425,
+            0.028935162350535393,
+            -0.004939688835293055,
+            -0.01956222578883171,
+            -0.002256307750940323,
+            -0.0008097385871224105,
+            -0.0036206748336553574,
+            0.0317772775888443,
+            0.01017417199909687,
+            -0.01817140355706215,
+            -0.01965293288230896,
+            -0.008760672062635422,
+            -0.02686404623091221,
+            -0.0022789842914789915,
+            0.0004492773732636124,
+            -0.008874054998159409,
+            0.00301596918143332,
+            -0.012494729831814766,
+            -0.019622696563601494,
+            0.0012320876121520996,
+            0.014089640229940414,
+            0.04746938869357109,
+            0.001130043645389378,
+            0.007275364361703396,
+            -0.02624422311782837,
+            0.01272905245423317,
+            -0.011685935780405998,
+            -0.0033409984316676855,
+            -0.0027344031259417534,
+            -0.00925199594348669,
+            -0.00039022407145239413,
+            -0.006285158917307854,
+            -0.015646757557988167,
+            -0.029328221455216408,
+            0.01879122667014599,
+            0.002141035860404372,
+            -0.006311614532023668,
+            -0.013537846505641937,
+            -0.02387075312435627,
+            0.012759287841618061,
+            -0.01675034500658512,
+            -0.008624613285064697,
+            -0.0356171615421772,
+            -0.024989459663629532,
+            0.011285318061709404,
+            0.006882305722683668,
+            -0.0017583705484867096,
+            -0.0028855795972049236,
+            -0.013726817443966866,
+            -0.02124028466641903,
+            -0.03640327602624893,
+            0.009115937165915966,
+            -0.005355423782020807,
+            0.01622122712433338,
+            -0.011247524060308933,
+            -5.229168164078146e-05,
+            -0.006319173611700535,
+            0.008458320051431656,
+            0.020665813237428665,
+            0.01140625961124897,
+            0.003021638374775648,
+            0.002367800334468484,
+            0.01700734533369541,
+            -0.004463483113795519,
+            -0.02376493066549301,
+            -0.02518598921597004,
+            0.00286479271017015,
+            0.00599036505445838,
+            -0.01782369799911976,
+            0.009463642723858356,
+            0.0007790309027768672,
+            0.0019237196538597345,
+            0.009267113171517849,
+            0.0034317043609917164,
+            -0.0069843498058617115,
+            -0.010400936007499695,
+            -0.001878366805613041,
+            0.026969870552420616,
+            0.007815820164978504,
+            -0.010249760001897812,
+            -0.010718407109379768,
+            0.0023848076816648245,
+            -0.007203555665910244,
+            0.0011961832642555237,
+            0.012222612276673317,
+            -0.013794846832752228,
+            -0.001549558131955564,
+            -0.011307994835078716,
+            -0.010000319220125675,
+            -0.011988288722932339,
+            -0.017475992441177368,
+            0.0011961832642555237,
+            -0.000834304781164974,
+            -0.03854998201131821,
+            0.013855316676199436,
+            -0.0006779316463507712,
+            -0.007124187890440226,
+            -0.0015949109802022576,
+            0.007936760783195496,
+            0.0005286450032144785,
+            -0.003512961557134986,
+            -0.0002489686303306371,
+            0.018201638013124466,
+            -0.027181517332792282,
+            0.012970935553312302,
+            0.021618224680423737,
+            -0.017143404111266136,
+            -0.001795219723135233,
+            -0.009206642396748066,
+            0.010113701224327087,
+            0.009161289781332016,
+            0.012494729831814766,
+            -0.001033668639138341,
+            0.003919248003512621,
+            4.290220749680884e-05,
+            -0.0010431171394884586,
+            0.006999467499554157,
+            0.0028118810150772333,
+            0.010982965119183064,
+            0.00807281956076622,
+            0.006364526227116585,
+            -0.01026487722992897,
+            0.010453848168253899,
+            0.0025945650413632393,
+            -0.0006132092676125467,
+            0.0011253192787989974,
+            0.01390067022293806,
+            0.023659106343984604,
+            0.012351111508905888,
+            -0.015540934167802334,
+            0.019819226115942,
+            -0.013205258175730705,
+            0.006663099862635136,
+            0.00278731482103467,
+            0.0002310164418304339,
+            0.03477057069540024,
+            0.012555199675261974,
+            0.024021929129958153,
+            0.025639517232775688,
+            0.026652399450540543,
+            0.019970402121543884,
+            -0.0020030871964991093,
+            -0.015057169832289219,
+            -0.01623634621500969,
+            -0.002135366667062044,
+            0.019879696890711784,
+            0.0018481315346434712,
+            0.00027849528123624623,
+            -0.010461406782269478,
+            -0.002199616516008973,
+            -0.004304748028516769,
+            0.010642818175256252,
+            0.0053176297806203365,
+            -0.0034543806686997414,
+            -0.0037510644178837538,
+            -0.0007029702537693083,
+            0.0008220216841436923,
+            -0.008133290335536003,
+            -0.007400084752589464,
+            -0.004924571141600609,
+            0.0021334770135581493,
+            -0.03053763322532177,
+            0.05000915378332138,
+            -0.00803502555936575,
+            0.014354199171066284,
+            -0.009728201664984226,
+            0.003121792571619153,
+            0.004901894833892584,
+            -0.013507611118257046,
+            -0.002435829723253846,
+            -0.003199270460754633,
+            0.0026909399311989546,
+            -0.0147472582757473,
+            0.002702278085052967,
+            0.008571702055633068,
+            -0.014596081338822842,
+            0.0032011603470891714,
+            0.01149696484208107,
+            0.006753805559128523,
+            0.005933673586696386,
+            0.009025231003761292,
+            0.015193228609859943,
+            0.02464175410568714,
+            -0.00073462282307446,
+            -0.0005399832152761519,
+            0.0025322046130895615,
+            0.010695730336010456,
+            0.015828169882297516,
+            0.001142326625995338,
+            0.024883635342121124,
+            -0.0021126901265233755,
+            0.019063344225287437,
+            0.028194399550557137,
+            -0.010673053562641144,
+            -0.007490790449082851,
+            0.001940727001056075,
+            -0.01365122850984335,
+            0.008737996220588684,
+            -0.0019426167709752917,
+            -0.008019908331334591,
+            -0.013477375730872154,
+            0.001699789660051465,
+            0.02201128378510475,
+            0.00832981988787651,
+            -0.006511923391371965,
+            0.01263078860938549,
+            -0.002673932584002614,
+            0.0030197484884411097,
+            0.002269535791128874,
+            -0.0013917677570134401,
+            -0.0030367558356374502,
+            -0.010695730336010456,
+            0.01587352156639099,
+            -0.005230703391134739,
+            -0.021376343443989754,
+            0.009765995666384697,
+            -0.00434632133692503,
+            -0.008322261273860931,
+            0.03235175088047981,
+            -0.01622122712433338,
+            -0.006731129251420498,
+            -0.003365564625710249,
+            0.016296815127134323,
+            0.008019908331334591,
+            -0.0007880069897510111
+          ],
+          "title": "Switch Transformer",
+          "keyphrases": [
+            "Switch Transformer",
+            "Simplifying Sparse Routing",
+            "Efficient Sparse Routing",
+            "Improved Training",
+            "Fine-Tuning Techniques"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "6c808a6a-ee41-4851-9cee-c3902340c898",
+      "type": "next",
+      "source": {
+        "id": "47c02fce-72d4-4324-913d-f36ecb5160fd",
+        "properties": {
+          "page_content": "2 Switch Transformer 4 2.1 Simplifying Sparse Routing 5 2.2 Efficient Sparse Routing 6 2.3 Putting It All Together: The Switch Transformer 8 2.4 Improved Training and Fine-Tuning Techniques 8 ",
+          "embedding": [
+            0.03316810354590416,
+            0.0006708452710881829,
+            -0.017672521993517876,
+            0.002751410473138094,
+            -0.007615511305630207,
+            0.019335461780428886,
+            0.034437984228134155,
+            0.057870324701070786,
+            -0.03555668890476227,
+            0.048315975815057755,
+            -0.011119023896753788,
+            -0.004195145331323147,
+            0.03028063289821148,
+            0.0179899912327528,
+            -0.011209730058908463,
+            0.001036503235809505,
+            0.0006793489446863532,
+            0.0007010805420577526,
+            0.017612051218748093,
+            -0.04595762491226196,
+            0.018035344779491425,
+            0.0009675289620645344,
+            -0.059351854026317596,
+            0.008488554507493973,
+            -0.024520812556147575,
+            -0.01762716844677925,
+            0.012955817393958569,
+            0.012260406278073788,
+            -0.03809645399451256,
+            0.034891512244939804,
+            0.024278929457068443,
+            0.003327770624309778,
+            -0.005680453032255173,
+            -0.009335142560303211,
+            -0.03256339579820633,
+            -0.025382516905665398,
+            0.021603107452392578,
+            0.03576833754777908,
+            -0.049616094678640366,
+            0.04940444603562355,
+            0.019698284566402435,
+            -0.0011952384375035763,
+            -0.008284466341137886,
+            0.006708452943712473,
+            0.023900989443063736,
+            -0.014611199498176575,
+            -0.017581814900040627,
+            -0.004780953750014305,
+            0.02580581232905388,
+            -0.012744170613586903,
+            0.0013350766384974122,
+            -0.07056914269924164,
+            0.026440752670168877,
+            0.009962525218725204,
+            -0.013636111281812191,
+            0.03818715736269951,
+            0.009312466718256474,
+            0.05723538622260094,
+            0.035859040915966034,
+            0.03422633558511734,
+            0.017158521339297295,
+            0.002755190012976527,
+            -0.014467581175267696,
+            -0.03377280756831169,
+            0.01649334467947483,
+            -0.0113684656098485,
+            -0.03395421802997589,
+            0.013137228786945343,
+            -0.006243585143238306,
+            0.004047748167067766,
+            -0.05055338889360428,
+            -0.003365564625710249,
+            -0.05070456489920616,
+            -0.0017942748963832855,
+            -0.017218992114067078,
+            -0.003101005917415023,
+            0.03486127778887749,
+            0.03897327557206154,
+            -0.004614659585058689,
+            0.01395358145236969,
+            0.02783157490193844,
+            -0.01479261089116335,
+            0.005737144500017166,
+            0.00222796224988997,
+            0.030462045222520828,
+            -0.013583199121057987,
+            -0.01650846377015114,
+            -0.05324432998895645,
+            0.05019056424498558,
+            -0.004943468142300844,
+            -0.03510316088795662,
+            -0.027695516124367714,
+            -0.009131054393947124,
+            0.003350446932017803,
+            -0.011973170563578606,
+            0.05324432998895645,
+            0.029010750353336334,
+            -0.014240817166864872,
+            -0.0005191964446566999,
+            0.009592142887413502,
+            0.0290409866720438,
+            0.0069012027233839035,
+            -0.036191631108522415,
+            0.004875438753515482,
+            0.06062173470854759,
+            0.007555040530860424,
+            0.00041620753472670913,
+            0.0489509180188179,
+            0.025775576010346413,
+            0.0024244915693998337,
+            0.04616927355527878,
+            -0.026652399450540543,
+            -0.031898219138383865,
+            0.03431704267859459,
+            -0.004985041916370392,
+            -0.05448397621512413,
+            -0.03126328065991402,
+            -0.04299456998705864,
+            0.014127434231340885,
+            -0.022615989670157433,
+            0.039638452231884,
+            0.045413389801979065,
+            -0.037794098258018494,
+            -0.021119343116879463,
+            -0.07449973374605179,
+            0.006077291443943977,
+            0.033137865364551544,
+            0.005703129805624485,
+            -0.011277759447693825,
+            0.023326518014073372,
+            0.003941924776881933,
+            -0.03126328065991402,
+            0.00961481872946024,
+            -0.03731033578515053,
+            -0.0009084756602533162,
+            -0.022268284112215042,
+            -0.00158546247985214,
+            0.0030953367240726948,
+            -0.015162993222475052,
+            0.0004497497866395861,
+            0.0014427897986024618,
+            -0.017234109342098236,
+            0.02633492834866047,
+            0.012547641061246395,
+            -0.02922239899635315,
+            0.03516362980008125,
+            -0.013749493286013603,
+            0.0394570417702198,
+            0.021724049001932144,
+            0.028058340772986412,
+            -0.05230703577399254,
+            0.004981262609362602,
+            -0.021104225888848305,
+            0.037340570241212845,
+            0.032865747809410095,
+            0.016478227451443672,
+            -0.02264622412621975,
+            -0.0021051312796771526,
+            0.0319586917757988,
+            0.004334983415901661,
+            0.054090917110443115,
+            -0.02650122344493866,
+            -0.002630469389259815,
+            0.029116574674844742,
+            0.024263812229037285,
+            -0.0018538006115704775,
+            0.029796868562698364,
+            0.02043904922902584,
+            -0.042964331805706024,
+            0.01938081532716751,
+            0.025563929229974747,
+            0.05176280066370964,
+            0.04142233356833458,
+            0.02940380945801735,
+            -0.014550728723406792,
+            -0.004678909666836262,
+            -0.02931310422718525,
+            0.013280847109854221,
+            0.007120408583432436,
+            -0.016266580671072006,
+            -0.021391460672020912,
+            -0.023024166002869606,
+            -0.01171617116779089,
+            0.007422761060297489,
+            0.019773872569203377,
+            0.01912381500005722,
+            0.014641433954238892,
+            0.015427551232278347,
+            -0.011610347777605057,
+            0.0028893589042127132,
+            0.0160700511187315,
+            0.01514031644910574,
+            0.017234109342098236,
+            -0.0007648580940440297,
+            -0.011527200229465961,
+            -0.05650973692536354,
+            -0.01764228567481041,
+            0.004017512779682875,
+            0.005332747474312782,
+            -0.013855316676199436,
+            -0.013159905560314655,
+            0.024777812883257866,
+            0.02288810722529888,
+            -0.0030065206810832024,
+            -0.025065047666430473,
+            -0.03123304434120655,
+            -0.0004428996180649847,
+            -0.005578408949077129,
+            -0.009713083505630493,
+            -0.05061385780572891,
+            0.019531991332769394,
+            -0.020559990778565407,
+            0.015767699107527733,
+            0.0018566352082416415,
+            0.014951346442103386,
+            -0.010355583392083645,
+            -0.014656552113592625,
+            -0.004531512502580881,
+            0.009584583342075348,
+            -0.02343234233558178,
+            0.02879910357296467,
+            0.027861811220645905,
+            0.03906398266553879,
+            -0.02482316456735134,
+            -0.02157287299633026,
+            -0.015200787223875523,
+            -0.035919513553380966,
+            -0.02142169512808323,
+            -0.011232406832277775,
+            -0.01685616932809353,
+            -0.0354357473552227,
+            0.008140848949551582,
+            0.02254040166735649,
+            -0.007687320001423359,
+            -0.002987623680382967,
+            -0.05254891514778137,
+            0.0034184763208031654,
+            0.0072526875883340836,
+            -0.005121100693941116,
+            -0.02359863556921482,
+            -0.004996379837393761,
+            0.004758276976644993,
+            -0.004860321059823036,
+            -0.02025763690471649,
+            -0.004229160025715828,
+            0.014316405169665813,
+            -0.036100924015045166,
+            0.04299456998705864,
+            -0.016961991786956787,
+            -0.0025322046130895615,
+            -0.005873203277587891,
+            0.025548812001943588,
+            0.018035344779491425,
+            -0.0014890875900164247,
+            -0.004100659862160683,
+            0.03165633976459503,
+            -0.017869049683213234,
+            -0.04710656777024269,
+            -0.02063557878136635,
+            0.018730755895376205,
+            0.040908332914114,
+            -0.002668263390660286,
+            -0.0011149259516969323,
+            0.02306951954960823,
+            0.0037624025717377663,
+            -0.007793143391609192,
+            0.032775044441223145,
+            0.020454166457057,
+            -0.01676546223461628,
+            1.5036442164273467e-05,
+            0.04003151133656502,
+            -0.022872989997267723,
+            0.005820291582494974,
+            0.030114339664578438,
+            0.026984987780451775,
+            -0.037431277334690094,
+            0.002757079666480422,
+            0.026652399450540543,
+            -0.005812732502818108,
+            -0.014029170386493206,
+            0.02588140033185482,
+            0.006784040946513414,
+            -0.030507396906614304,
+            -0.00842052511870861,
+            0.016720110550522804,
+            -0.03002363257110119,
+            -0.033833280205726624,
+            0.057265620678663254,
+            -0.02560928277671337,
+            -0.03815692290663719,
+            -0.008435643278062344,
+            -0.014097199775278568,
+            -0.02069604955613613,
+            -0.010075907222926617,
+            0.05366762354969978,
+            -0.025639517232775688,
+            0.02359863556921482,
+            -0.04245033487677574,
+            0.021829871460795403,
+            -0.01729458011686802,
+            -0.008722878061234951,
+            0.010620142333209515,
+            0.005688012111932039,
+            -0.03712892532348633,
+            -0.05929138511419296,
+            0.033047161996364594,
+            0.029328221455216408,
+            0.045141272246837616,
+            -0.02385563589632511,
+            -0.001808447646908462,
+            0.0442039780318737,
+            0.02317534200847149,
+            0.010068348608911037,
+            -0.0412711575627327,
+            -0.004750718362629414,
+            -0.011360906064510345,
+            0.022570636123418808,
+            -0.013719257898628712,
+            -0.03220057487487793,
+            -0.008556583896279335,
+            -0.04934397712349892,
+            -0.011958053335547447,
+            -0.029267750680446625,
+            0.06845267117023468,
+            0.008866495452821255,
+            0.01175396516919136,
+            0.024006811901926994,
+            0.02783157490193844,
+            -0.03186798468232155,
+            0.0007024978403933346,
+            -0.008972319774329662,
+            -0.009440965950489044,
+            0.008186202496290207,
+            -0.00898743700236082,
+            -0.0029214839451014996,
+            -0.02589651755988598,
+            -0.016130521893501282,
+            0.003979718778282404,
+            -0.036463748663663864,
+            -0.018231874331831932,
+            -0.009365377947688103,
+            0.005846747197210789,
+            0.016387522220611572,
+            -0.016100287437438965,
+            -0.03537527844309807,
+            -0.03186798468232155,
+            -0.018821461126208305,
+            -0.012010964564979076,
+            -0.009289789944887161,
+            0.003407138166949153,
+            0.0008442256948910654,
+            0.023039283230900764,
+            0.023825401440262794,
+            0.002604013541713357,
+            0.028118809685111046,
+            -0.03867092356085777,
+            -0.01650846377015114,
+            0.01570722833275795,
+            0.015034493058919907,
+            -0.04284339025616646,
+            0.0247475765645504,
+            -0.03344022110104561,
+            -0.03939656913280487,
+            -0.021996166557073593,
+            -0.053123388439416885,
+            -0.04021292179822922,
+            0.0015656206523999572,
+            -0.04737868532538414,
+            -0.026607045903801918,
+            0.02940380945801735,
+            -0.0289502814412117,
+            0.005827850196510553,
+            0.010468965396285057,
+            -0.014150111004710197,
+            -0.010650377720594406,
+            -0.019017990678548813,
+            0.05212562158703804,
+            -0.011285318061709404,
+            -0.001033668639138341,
+            0.02763504534959793,
+            -0.017702756449580193,
+            0.005162674002349377,
+            -0.005812732502818108,
+            -0.004282071720808744,
+            0.022071754559874535,
+            0.024278929457068443,
+            0.04344809800386429,
+            -0.007097731810063124,
+            -0.08961737155914307,
+            0.016735227778553963,
+            0.040575746446847916,
+            -0.014807728119194508,
+            0.00541967386379838,
+            -0.023659106343984604,
+            0.05626785755157471,
+            0.011345788836479187,
+            -0.013537846505641937,
+            -0.0009372936910949647,
+            0.012668582610785961,
+            -0.005098423920571804,
+            -0.028315339237451553,
+            -0.016614286229014397,
+            -0.042329393327236176,
+            -0.006855849642306566,
+            -0.009055466391146183,
+            0.018095815554261208,
+            -0.02186010777950287,
+            0.022253166884183884,
+            0.007914084941148758,
+            0.008934524841606617,
+            0.013515169732272625,
+            -0.026108164340257645,
+            0.03247269243001938,
+            -0.04178515821695328,
+            0.00832981988787651,
+            0.012222612276673317,
+            -0.03924539312720299,
+            0.016432873904705048,
+            -0.02195081301033497,
+            -0.030144574120640755,
+            -0.011648141779005527,
+            0.030205044895410538,
+            0.024324283003807068,
+            -0.014452463947236538,
+            0.008148407563567162,
+            0.008284466341137886,
+            0.0229183416813612,
+            -0.03464962914586067,
+            0.003811534959822893,
+            0.030235279351472855,
+            0.03903374820947647,
+            0.0062398058362305164,
+            0.00339957932010293,
+            -0.004554189275950193,
+            -0.04450633376836777,
+            0.0027608589734882116,
+            -0.029978280887007713,
+            -0.008269349113106728,
+            -0.035224102437496185,
+            0.024157987907528877,
+            0.04580644890666008,
+            0.023281166329979897,
+            0.014974022284150124,
+            0.00031628936994820833,
+            -0.028738632798194885,
+            0.0008456429932266474,
+            -0.029857339337468147,
+            0.006198232527822256,
+            0.031112102791666985,
+            -0.0010667385067790747,
+            -0.028738632798194885,
+            -0.00025251181796193123,
+            -0.047167036682367325,
+            -0.03933610022068024,
+            0.017324814572930336,
+            -0.02780134044587612,
+            0.021527519449591637,
+            0.00541589455679059,
+            0.003002741141244769,
+            -0.022268284112215042,
+            0.005910997278988361,
+            0.01904822699725628,
+            0.02089257910847664,
+            -0.054000210016965866,
+            0.024067282676696777,
+            -0.024233577772974968,
+            -0.05309315025806427,
+            0.0020503299310803413,
+            -0.027332693338394165,
+            0.003017858834937215,
+            -0.0003706183924805373,
+            -0.026002340018749237,
+            0.00039353108149953187,
+            0.019169166684150696,
+            -0.014611199498176575,
+            0.033651866018772125,
+            -0.01649334467947483,
+            -0.09403172135353088,
+            0.0020408814307302237,
+            -0.025034811347723007,
+            -0.011338230222463608,
+            0.03885233402252197,
+            -0.0027627486269921064,
+            0.0054612476378679276,
+            -0.007226231973618269,
+            0.012781964614987373,
+            0.017612051218748093,
+            0.029615456238389015,
+            0.00965261273086071,
+            0.00511354161426425,
+            0.030068986117839813,
+            0.005665335804224014,
+            9.962288459064439e-05,
+            -0.0040061743929982185,
+            -0.020393695682287216,
+            -0.05152091756463051,
+            -0.022872989997267723,
+            -0.05396997556090355,
+            0.020030872896313667,
+            0.0071733202785253525,
+            -0.030129456892609596,
+            -0.0049888212233781815,
+            0.04870903491973877,
+            0.020484402775764465,
+            -0.02176940254867077,
+            -0.007056158501654863,
+            0.019108697772026062,
+            0.024157987907528877,
+            0.005593526642769575,
+            -0.001357753062620759,
+            0.010620142333209515,
+            0.0015826279995962977,
+            0.011897582560777664,
+            -0.02195081301033497,
+            -0.03090045601129532,
+            0.006889864336699247,
+            -0.028390927240252495,
+            0.022782282903790474,
+            0.006470350082963705,
+            -0.0012490949593484402,
+            0.00991717167198658,
+            0.006421217694878578,
+            -0.0027117265854030848,
+            0.023976577445864677,
+            -0.01087714172899723,
+            0.015004257671535015,
+            0.037854570895433426,
+            0.014467581175267696,
+            0.05294197425246239,
+            -0.0076684230007231236,
+            -0.010854465886950493,
+            0.008397849276661873,
+            -0.0028251090552657843,
+            -0.017778344452381134,
+            -0.00031487210071645677,
+            -0.007675981614738703,
+            0.0089949956163764,
+            0.013515169732272625,
+            -0.0398198626935482,
+            -0.011958053335547447,
+            -0.002184499055147171,
+            -0.01649334467947483,
+            -0.0004769143124576658,
+            -0.0031085647642612457,
+            8.887518924893811e-05,
+            0.03906398266553879,
+            -0.0022090652491897345,
+            -0.016977109014987946,
+            -0.00726402597501874,
+            0.02000063844025135,
+            -0.05245821177959442,
+            -0.00741520244628191,
+            0.003911689389497042,
+            0.03359139710664749,
+            -0.011217288672924042,
+            0.005767379887402058,
+            -0.028904927894473076,
+            0.01694687455892563,
+            -0.01394602283835411,
+            -0.026304693892598152,
+            -0.0039494833908975124,
+            0.0024887414183467627,
+            0.01416522916406393,
+            0.01518566906452179,
+            -0.01079399511218071,
+            0.009849142283201218,
+            0.017869049683213234,
+            0.03331927955150604,
+            -0.021300755441188812,
+            0.011255082674324512,
+            -0.004013733472675085,
+            0.03099116124212742,
+            -0.007093952503055334,
+            -0.00710907019674778,
+            0.017445756122469902,
+            0.015253698453307152,
+            -0.00209379312582314,
+            0.010998083278536797,
+            -0.013689023442566395,
+            -0.0068785264156758785,
+            0.01684105023741722,
+            -0.03168657422065735,
+            0.015321727842092514,
+            0.01139870099723339,
+            0.005332747474312782,
+            -0.009191525168716908,
+            -0.02297881245613098,
+            -0.020998401567339897,
+            0.003896571695804596,
+            -0.016357285901904106,
+            0.03368210047483444,
+            0.013809964060783386,
+            0.007354731671512127,
+            -0.020136697217822075,
+            -0.0016043595969676971,
+            0.011580112390220165,
+            0.009433407336473465,
+            0.005030394531786442,
+            0.03537527844309807,
+            -0.0637359693646431,
+            0.02684892900288105,
+            0.011232406832277775,
+            -0.004818747751414776,
+            -0.01417278777807951,
+            0.0011470509925857186,
+            -0.004542850889265537,
+            0.0102270832285285,
+            0.01056723017245531,
+            0.02368934266269207,
+            0.0020087563898414373,
+            0.014633875340223312,
+            -0.03168657422065735,
+            0.035586923360824585,
+            -0.009131054393947124,
+            0.0034241455141454935,
+            -0.0006117919692769647,
+            -0.012003405950963497,
+            0.00869264267385006,
+            -0.009645054116845131,
+            0.004576865583658218,
+            0.08411455154418945,
+            0.013568081893026829,
+            0.0101363779976964,
+            0.005291174165904522,
+            -0.027967633679509163,
+            0.03546598181128502,
+            -0.014860640279948711,
+            -0.021376343443989754,
+            0.0014824735699221492,
+            0.0035658732522279024,
+            0.009622378274798393,
+            -0.02595698833465576,
+            -0.007921643555164337,
+            -0.03897327557206154,
+            -0.010778876952826977,
+            -0.018307462334632874,
+            0.010559671558439732,
+            -0.0035450865980237722,
+            0.036826569586992264,
+            -0.01948663778603077,
+            -0.01913893222808838,
+            0.03437751531600952,
+            0.0582633838057518,
+            0.03386351466178894,
+            -0.00868508405983448,
+            0.008707760833203793,
+            0.004512615501880646,
+            -0.013106994330883026,
+            0.02124028466641903,
+            0.015813050791621208,
+            0.008155967108905315,
+            0.026879163458943367,
+            -0.010091024450957775,
+            0.03625210002064705,
+            0.023900989443063736,
+            -0.017082933336496353,
+            -0.02692451700568199,
+            -0.0052382624708116055,
+            -0.02604769356548786,
+            -0.0040666451677680016,
+            -0.00307266041636467,
+            0.006965452805161476,
+            0.005820291582494974,
+            1.740595507726539e-05,
+            0.01718875765800476,
+            -0.023326518014073372,
+            -0.018035344779491425,
+            0.007400084752589464,
+            -0.022752048447728157,
+            0.01746087335050106,
+            0.0035791012924164534,
+            -0.03138422220945358,
+            -0.009153731167316437,
+            0.028300222009420395,
+            0.000411955697927624,
+            -0.030386457219719887,
+            -0.006039496976882219,
+            0.005521717946976423,
+            -0.028103692457079887,
+            0.0013946022372692823,
+            -0.025911634787917137,
+            -0.0071959965862333775,
+            -0.004474821500480175,
+            0.008881613612174988,
+            0.009395613335072994,
+            -0.00038762573967687786,
+            -0.02394634112715721,
+            -0.015200787223875523,
+            -0.029721280559897423,
+            0.027075693011283875,
+            -0.02631981112062931,
+            0.015079845674335957,
+            -0.02553369477391243,
+            0.02246481366455555,
+            -0.01737016811966896,
+            -0.007449217140674591,
+            -0.004365218337625265,
+            0.030855102464556694,
+            0.061468325555324554,
+            -0.030598104000091553,
+            0.0010865803342312574,
+            0.010098583996295929,
+            0.003289976390078664,
+            -0.017869049683213234,
+            0.03649398311972618,
+            0.02589651755988598,
+            -3.9772385207470506e-05,
+            -0.0032918660435825586,
+            -0.013016288168728352,
+            -0.041906099766492844,
+            -0.0024263812229037285,
+            0.011648141779005527,
+            -0.0197285208851099,
+            0.02210198901593685,
+            0.031081868335604668,
+            0.01684105023741722,
+            -0.03044692613184452,
+            -0.023659106343984604,
+            -0.03625210002064705,
+            0.032593630254268646,
+            -0.013122111558914185,
+            0.043478332459926605,
+            0.014807728119194508,
+            -0.025140635669231415,
+            -0.010899818502366543,
+            -0.008722878061234951,
+            0.014399551786482334,
+            -0.02263110689818859,
+            0.027151281014084816,
+            -0.012177258729934692,
+            0.02192057855427265,
+            -0.007033482193946838,
+            -0.003972159698605537,
+            0.011617906391620636,
+            -0.007097731810063124,
+            -0.02043904922902584,
+            -0.0012557089794427156,
+            0.005975247360765934,
+            0.017581814900040627,
+            -0.0066064088605344296,
+            0.034619394689798355,
+            0.03062833845615387,
+            -0.0040326304733753204,
+            0.023356754332780838,
+            0.02774086967110634,
+            0.00572202680632472,
+            0.01966805011034012,
+            0.011557435616850853,
+            -0.001473969896323979,
+            -0.021542636677622795,
+            0.001912381500005722,
+            -0.032865747809410095,
+            0.010544554330408573,
+            0.005465026944875717,
+            -0.02279740199446678,
+            0.021542636677622795,
+            0.01702246256172657,
+            -0.019184285774827003,
+            -0.00036636655568145216,
+            -0.024898752570152283,
+            0.0007379298331215978,
+            0.02668263390660286,
+            -0.025669753551483154,
+            0.027362927794456482,
+            0.020650696009397507,
+            0.010363142006099224,
+            -0.024535929784178734,
+            -0.005619982723146677,
+            0.0035299689043313265,
+            -0.007698658388108015,
+            0.024157987907528877,
+            0.027952516451478004,
+            0.03174704313278198,
+            -0.016629403457045555,
+            0.0036811453755944967,
+            0.013061640784144402,
+            0.0036282336805015802,
+            -0.019320344552397728,
+            -0.0264861062169075,
+            0.013832640834152699,
+            0.005907217971980572,
+            0.0009306797292083502,
+            0.020862342789769173,
+            0.003520520403981209,
+            -0.004195145331323147,
+            -0.0012292531318962574,
+            -0.007127967197448015,
+            -0.029948044568300247,
+            -0.0075248051434755325,
+            0.02545810490846634,
+            0.012139464728534222,
+            0.03017481043934822,
+            -0.026380281895399094,
+            0.011239965446293354,
+            0.042510803788900375,
+            -0.018987756222486496,
+            -0.009380496107041836,
+            0.017869049683213234,
+            0.003953262697905302,
+            -0.015321727842092514,
+            -0.014308846555650234,
+            -0.009818906895816326,
+            -0.004414350725710392,
+            0.03310763090848923,
+            -0.00987937767058611,
+            -0.046108800917863846,
+            -0.004021292086690664,
+            -0.033289045095443726,
+            -0.006746246945112944,
+            7.907234248705208e-05,
+            0.023659106343984604,
+            -0.02052975445985794,
+            0.007929202169179916,
+            0.025473223999142647,
+            -0.005427232943475246,
+            0.0356171615421772,
+            0.012388906441628933,
+            -0.02580581232905388,
+            0.011821994557976723,
+            -0.025155752897262573,
+            0.01417278777807951,
+            -0.006825614254921675,
+            0.005313850473612547,
+            -0.013099434785544872,
+            -0.04411327466368675,
+            -0.05170232802629471,
+            0.0030896677635610104,
+            -0.02518598921597004,
+            0.04142233356833458,
+            0.01938081532716751,
+            -0.02105887234210968,
+            0.0025643296539783478,
+            -0.017355050891637802,
+            0.017324814572930336,
+            0.04680421203374863,
+            0.011678377166390419,
+            -0.009667730890214443,
+            0.01966805011034012,
+            -0.014059404842555523,
+            0.02376493066549301,
+            0.020121578127145767,
+            0.003779409918934107,
+            -0.03477057069540024,
+            -0.007793143391609192,
+            -0.06434067338705063,
+            0.0033806823194026947,
+            0.013076758943498135,
+            0.008246672339737415,
+            0.03809645399451256,
+            -0.0009150896221399307,
+            0.03099116124212742,
+            0.02692451700568199,
+            0.024324283003807068,
+            0.001319014118053019,
+            -0.03368210047483444,
+            -0.0024055945686995983,
+            0.04641115665435791,
+            0.038640689104795456,
+            -0.012842435389757156,
+            -0.014648993499577045,
+            -0.016735227778553963,
+            -0.005170233082026243,
+            0.026470987126231194,
+            -0.018972638994455338,
+            0.011648141779005527,
+            -0.008019908331334591,
+            -0.004353880416601896,
+            -0.010657936334609985,
+            0.007154423277825117,
+            -0.020862342789769173,
+            -0.03885233402252197,
+            -0.0290409866720438,
+            -0.03229127824306488,
+            0.02615351788699627,
+            0.02677334100008011,
+            -0.042601510882377625,
+            -0.008239113725721836,
+            -0.04226892068982124,
+            0.010771318338811398,
+            0.021451931446790695,
+            0.0014409000286832452,
+            0.01665963977575302,
+            -0.013167464174330235,
+            0.03791503980755806,
+            0.02352304756641388,
+            0.0016922308132052422,
+            -0.005986585281789303,
+            -0.007543702609837055,
+            -0.00833737850189209,
+            -0.007936760783195496,
+            0.00445592449977994,
+            0.03386351466178894,
+            -0.01888193190097809,
+            -0.008193761110305786,
+            0.004743159748613834,
+            0.0003840825520455837,
+            -0.0012216942850500345,
+            0.0161607563495636,
+            0.011965611949563026,
+            -0.004784733057022095,
+            -0.0023848076816648245,
+            -0.01364366989582777,
+            -0.004055306781083345,
+            -0.005533056333661079,
+            0.007071276195347309,
+            -0.009743318893015385,
+            -0.006666879169642925,
+            0.031172573566436768,
+            0.0013246831949800253,
+            -0.026289576664566994,
+            -0.014815287664532661,
+            0.0032351750414818525,
+            -0.0013624773127958179,
+            -0.009456084109842777,
+            0.03380304202437401,
+            0.052609387785196304,
+            -0.0049548065289855,
+            -0.023659106343984604,
+            0.010121259838342667,
+            -0.001774432952515781,
+            0.002562440000474453,
+            0.0005012442707084119,
+            0.01876099221408367,
+            0.02668263390660286,
+            -0.0038927923887968063,
+            -0.004395453725010157,
+            -0.01913893222808838,
+            0.008624613285064697,
+            0.022600872442126274,
+            0.025140635669231415,
+            -0.022041520103812218,
+            0.014921111054718494,
+            0.0019596240017563105,
+            -0.0197285208851099,
+            0.015767699107527733,
+            0.0160700511187315,
+            -0.007233790587633848,
+            0.012509847059845924,
+            -0.005174012389034033,
+            0.00825423188507557,
+            -0.010431171394884586,
+            0.010990524664521217,
+            0.01975875534117222,
+            0.008299584500491619,
+            -0.010242201387882233,
+            -0.017687639221549034,
+            0.007517246529459953,
+            0.005623762030154467,
+            -0.0018358484376221895,
+            0.017173638567328453,
+            -0.005706909112632275,
+            0.004890556447207928,
+            -0.005427232943475246,
+            -0.024551047012209892,
+            -0.004176247864961624,
+            -0.004229160025715828,
+            -0.010809112340211868,
+            0.009758437052369118,
+            -0.026365164667367935,
+            0.016901521012187004,
+            0.023235812783241272,
+            -0.0057787178084254265,
+            0.0007317882846109569,
+            -0.0089042903855443,
+            0.001393657410517335,
+            -0.017475992441177368,
+            0.017339933663606644,
+            -0.029539868235588074,
+            -0.02615351788699627,
+            0.04311550781130791,
+            -0.025684870779514313,
+            -0.026395399123430252,
+            0.0006264371913857758,
+            -0.01824699155986309,
+            -0.0022525284439325333,
+            0.009675289504230022,
+            0.006859628949314356,
+            -0.015419992618262768,
+            -0.017143404111266136,
+            0.015412434004247189,
+            0.013689023442566395,
+            -0.03238198533654213,
+            0.022963695228099823,
+            -0.014958905056118965,
+            -0.019169166684150696,
+            -0.00019806470663752407,
+            -0.0031936015002429485,
+            -0.03570786491036415,
+            -0.02562440000474453,
+            -0.006088629364967346,
+            -0.005041732918471098,
+            -0.0012708266731351614,
+            0.032865747809410095,
+            0.0005503766005858779,
+            -0.0008706816006451845,
+            -0.01605493389070034,
+            -0.009592142887413502,
+            -0.04411327466368675,
+            0.006436335388571024,
+            0.01650846377015114,
+            -0.021739166229963303,
+            -0.01334131695330143,
+            -0.0006311614997684956,
+            0.059442561119794846,
+            -0.02001575566828251,
+            0.005139997694641352,
+            0.01139870099723339,
+            -0.018715638667345047,
+            -0.030764397233724594,
+            0.014679228886961937,
+            0.03289598599076271,
+            0.0009817017707973719,
+            -0.02201128378510475,
+            -0.014323963783681393,
+            -0.011376024223864079,
+            -0.0024396092630922794,
+            0.017778344452381134,
+            0.0068180556409060955,
+            -0.022510165348649025,
+            0.025140635669231415,
+            -0.002588895848020911,
+            0.0007237570243887603,
+            -0.03192845731973648,
+            -4.650445771403611e-05,
+            0.010151495225727558,
+            0.02299392968416214,
+            -0.013099434785544872,
+            0.0008319426560774446,
+            -0.005676673725247383,
+            -0.002031432930380106,
+            -0.00829202588647604,
+            -0.0070901731960475445,
+            -0.026516340672969818,
+            -0.024263812229037285,
+            0.026108164340257645,
+            -0.005593526642769575,
+            0.012555199675261974,
+            -0.03271457180380821,
+            -0.01483040489256382,
+            0.032261043787002563,
+            0.0014616867993026972,
+            -0.025231340900063515,
+            0.019365696236491203,
+            -0.025503458455204964,
+            -0.00900255423039198,
+            0.006005482282489538,
+            0.0051551153883337975,
+            0.006859628949314356,
+            0.019259873777627945,
+            0.015949109569191933,
+            0.023190459236502647,
+            4.854179860558361e-05,
+            0.015933992341160774,
+            -0.022328754886984825,
+            -0.018201638013124466,
+            0.02798275090754032,
+            -0.0021278078202158213,
+            0.008564143441617489,
+            0.02808857522904873,
+            0.0037246085703372955,
+            -0.015737462788820267,
+            8.344228990608826e-05,
+            -0.004561747889965773,
+            0.010302671231329441,
+            0.0010998082580044866,
+            0.014958905056118965,
+            -0.005994144361466169,
+            -0.0002851092431228608,
+            0.015858404338359833,
+            0.01452805195003748,
+            0.007921643555164337,
+            -0.011572553776204586,
+            -0.01894240267574787,
+            0.0021693813614547253,
+            -0.0005267552915029228,
+            0.04281315580010414,
+            0.007222452666610479,
+            0.004985041916370392,
+            -0.05433279648423195,
+            0.004678909666836262,
+            -0.01842840388417244,
+            0.004788512364029884,
+            0.024460341781377792,
+            0.002974395640194416,
+            -0.00863973144441843,
+            0.011224847286939621,
+            -0.0015457787085324526,
+            -0.0029233735986053944,
+            0.006254923529922962,
+            -0.043024804443120956,
+            -0.02246481366455555,
+            0.01623634621500969,
+            0.008896730840206146,
+            0.022767165675759315,
+            0.016478227451443672,
+            -0.004969924222677946,
+            0.00691254111006856,
+            0.05956350266933441,
+            0.0008007625001482666,
+            -0.009425848722457886,
+            -0.027060575783252716,
+            0.00016924670489970595,
+            -0.006988129112869501,
+            -0.03422633558511734,
+            0.03906398266553879,
+            0.008435643278062344,
+            0.02001575566828251,
+            0.009690407663583755,
+            0.02439987100660801,
+            -0.011852229945361614,
+            0.011746406555175781,
+            0.014580964110791683,
+            -0.009856700897216797,
+            -0.006825614254921675,
+            -0.0015542823821306229,
+            -0.015004257671535015,
+            0.010990524664521217,
+            -0.0031180132646113634,
+            0.014399551786482334,
+            -0.0074454378336668015,
+            0.03703821823000908,
+            -0.0315958671271801,
+            0.024944106116890907,
+            0.02482316456735134,
+            0.007449217140674591,
+            -0.02421845868229866,
+            -0.007407643832266331,
+            0.012827317230403423,
+            0.017657402902841568,
+            0.021346107125282288,
+            0.025034811347723007,
+            0.015004257671535015,
+            0.03721962869167328,
+            -0.03344022110104561,
+            0.015676992014050484,
+            0.009191525168716908,
+            0.028920045122504234,
+            0.013106994330883026,
+            0.014301287941634655,
+            0.02052975445985794,
+            -0.006621526554226875,
+            -0.021618224680423737,
+            -0.0015608962858095765,
+            0.006595070473849773,
+            -0.00929734855890274,
+            -0.01545022800564766,
+            -0.01762716844677925,
+            -0.0020068667363375425,
+            0.00934270117431879,
+            -0.029721280559897423,
+            -0.020590225234627724,
+            -0.022842753678560257,
+            0.0077629080042243,
+            0.01675034500658512,
+            -0.026607045903801918,
+            0.010159053839743137,
+            0.011557435616850853,
+            0.016206109896302223,
+            0.009796231053769588,
+            0.009040349163115025,
+            0.004285851027816534,
+            0.01588863879442215,
+            -0.013673905283212662,
+            0.019637813791632652,
+            0.010476524941623211,
+            0.011625465005636215,
+            -0.012169700115919113,
+            -0.035405512899160385,
+            0.007464334834367037,
+            0.01703757978975773,
+            0.0003247930435463786,
+            -0.018549343571066856,
+            -0.0040061743929982185,
+            -0.0014862529933452606,
+            -0.0291316919028759,
+            -0.0013294074451550841,
+            -0.0020068667363375425,
+            0.01329596433788538,
+            -0.008760672062635422,
+            -0.0009996539447456598,
+            0.004656233359128237,
+            -0.020212285220623016,
+            -0.0032276161946356297,
+            -0.010559671558439732,
+            0.010385818779468536,
+            0.001661050715483725,
+            -0.005782497115433216,
+            0.010506759397685528,
+            -0.018927285447716713,
+            -0.01694687455892563,
+            0.023039283230900764,
+            0.006375864613801241,
+            -0.019320344552397728,
+            0.01167081855237484,
+            -0.016568932682275772,
+            -0.005563291721045971,
+            -0.005971467588096857,
+            -0.03168657422065735,
+            0.004690248053520918,
+            0.018398167565464973,
+            -0.005638879723846912,
+            0.011633023619651794,
+            0.009577024728059769,
+            0.018473755568265915,
+            -0.013507611118257046,
+            0.023039283230900764,
+            0.004236718639731407,
+            0.004391674418002367,
+            -0.005087085999548435,
+            -0.008662407286465168,
+            0.005162674002349377,
+            -0.0009509940282441676,
+            -1.5781988622620702e-05,
+            0.0223741065710783,
+            -0.0438411571085453,
+            0.005929894279688597,
+            0.006270041223615408,
+            0.009992759674787521,
+            0.022661343216896057,
+            0.010249760001897812,
+            -0.022600872442126274,
+            0.0049283504486083984,
+            0.005899658892303705,
+            0.006942776031792164,
+            -4.81284259876702e-05,
+            0.0014049956807866693,
+            -0.00842052511870861,
+            -0.011746406555175781,
+            0.016614286229014397,
+            0.023099754005670547,
+            -0.01148940622806549,
+            0.014853081665933132,
+            -0.024883635342121124,
+            -0.0031917118467390537,
+            0.013129670172929764,
+            0.03676610067486763,
+            -0.035859040915966034,
+            -0.010605024173855782,
+            0.020302990451455116,
+            0.005064409226179123,
+            0.011942935176193714,
+            -0.03694751113653183,
+            -0.02385563589632511,
+            -0.02193569578230381,
+            0.01788416877388954,
+            0.0003573904687073082,
+            -0.0022487489040941,
+            -0.01170861255377531,
+            -0.02010646089911461,
+            -0.010091024450957775,
+            0.012283082120120525,
+            0.00025794474640861154,
+            -0.011897582560777664,
+            -0.001873642555437982,
+            0.006069732364267111,
+            0.03555668890476227,
+            -0.011995847336947918,
+            0.02113446034491062,
+            -0.008322261273860931,
+            -0.019456403329968452,
+            -0.020514637231826782,
+            -0.010506759397685528,
+            0.0010327238123863935,
+            0.005037953611463308,
+            0.012388906441628933,
+            0.010234642773866653,
+            0.005480144638568163,
+            -0.007241349667310715,
+            0.004357659723609686,
+            -0.0060735116712749004,
+            0.0019426167709752917,
+            -0.008246672339737415,
+            -0.025911634787917137,
+            0.02492898888885975,
+            0.00829202588647604,
+            0.008586819283664227,
+            -0.006065953057259321,
+            0.0032295058481395245,
+            -0.022147342562675476,
+            0.004406792111694813,
+            0.018564462661743164,
+            -0.003560204291716218,
+            0.002891248557716608,
+            0.013424464501440525,
+            -0.02343234233558178,
+            -0.008042584173381329,
+            0.006504364777356386,
+            0.00034912300179712474,
+            0.010703288950026035,
+            0.014271052554249763,
+            -0.017959756776690483,
+            0.006077291443943977,
+            0.010552112944424152,
+            -0.009577024728059769,
+            -0.005714467726647854,
+            0.009176407940685749,
+            0.006844511721283197,
+            -0.03658469021320343,
+            0.023538164794445038,
+            -0.0037321674171835184,
+            -0.0027325134724378586,
+            0.02774086967110634,
+            -0.019879696890711784,
+            -0.019017990678548813,
+            -0.005166453309357166,
+            0.015918875113129616,
+            -0.0013719258131459355,
+            0.01048408355563879,
+            -0.0034865057095885277,
+            0.003429814474657178,
+            -0.0005040788091719151,
+            0.004818747751414776,
+            -0.01057478878647089,
+            0.00075399229535833,
+            -0.008140848949551582,
+            0.001432396355085075,
+            0.009289789944887161,
+            -0.007604172918945551,
+            -0.008186202496290207,
+            0.003972159698605537,
+            -0.0011952384375035763,
+            0.003414697013795376,
+            -0.007014585193246603,
+            0.018972638994455338,
+            -0.03126328065991402,
+            -0.0003994363942183554,
+            0.00568423280492425,
+            -0.009289789944887161,
+            0.023296283558011055,
+            0.011013200506567955,
+            0.010385818779468536,
+            0.005117321386933327,
+            0.012131906114518642,
+            -0.013258170336484909,
+            0.0008073764620348811,
+            -0.006304055918008089,
+            -0.009335142560303211,
+            0.004686468280851841,
+            0.026213988661766052,
+            0.016100287437438965,
+            0.026093047112226486,
+            0.018201638013124466,
+            -0.0002548739721532911,
+            0.004002395085990429,
+            0.004731821361929178,
+            -0.0054234531708061695,
+            0.01720387488603592,
+            -0.019078461453318596,
+            0.0028496752493083477,
+            0.007868731394410133,
+            -0.023825401440262794,
+            -0.003443042514845729,
+            -0.003862557001411915,
+            0.031535398215055466,
+            0.01851910911500454,
+            0.015102522447705269,
+            0.010763759724795818,
+            0.006738687865436077,
+            -0.0160700511187315,
+            0.03044692613184452,
+            -0.007861172780394554,
+            0.016463110223412514,
+            0.010884701274335384,
+            -0.007177099585533142,
+            0.0030839985702186823,
+            0.010703288950026035,
+            0.002562440000474453,
+            -0.022479930892586708,
+            0.020318107679486275,
+            -0.018624933436512947,
+            0.0076684230007231236,
+            0.009501436725258827,
+            -0.008866495452821255,
+            0.023296283558011055,
+            -0.01079399511218071,
+            -0.00863973144441843,
+            -0.005389438476413488,
+            -0.005272277165204287,
+            0.05578409135341644,
+            0.002758969319984317,
+            -0.019320344552397728,
+            -0.024611517786979675,
+            0.024551047012209892,
+            0.0039834980852901936,
+            -0.0061906734481453896,
+            0.021965930238366127,
+            0.012010964564979076,
+            -0.03480080887675285,
+            0.015586286783218384,
+            0.004225380253046751,
+            -0.008012348785996437,
+            0.020151814445853233,
+            0.0031028955709189177,
+            -0.006141541060060263,
+            0.030401574447751045,
+            -0.003520520403981209,
+            -0.001451293472200632,
+            0.008737996220588684,
+            -0.001432396355085075,
+            -0.017869049683213234,
+            0.05327456444501877,
+            0.001982300542294979,
+            0.025216223672032356,
+            -0.0056124236434698105,
+            -0.009100819006562233,
+            -0.017566697672009468,
+            -0.011028318665921688,
+            0.024233577772974968,
+            -0.012184818275272846,
+            -0.015110081061720848,
+            0.0011602789163589478,
+            0.013696582056581974,
+            0.0032654101960361004,
+            -0.0013568082358688116,
+            -0.006243585143238306,
+            0.03431704267859459,
+            -0.005136218387633562,
+            -0.018095815554261208,
+            -0.016085168346762657,
+            -0.0024887414183467627,
+            0.020998401567339897,
+            0.019017990678548813,
+            0.015737462788820267,
+            -0.02069604955613613,
+            -0.004890556447207928,
+            -0.013311082497239113,
+            -0.03362163156270981,
+            0.010620142333209515,
+            0.0004606155853252858,
+            0.023825401440262794,
+            -0.02579069323837757,
+            0.006825614254921675,
+            0.003658468835055828,
+            -0.04589715600013733,
+            0.019350579008460045,
+            0.014467581175267696,
+            0.00031416345154866576,
+            -0.014762375503778458,
+            0.0025529915001243353,
+            -0.0028950280975550413,
+            0.006266261916607618,
+            -0.012600553222000599,
+            -0.017869049683213234,
+            0.0024660651106387377,
+            0.01056723017245531,
+            0.023129988461732864,
+            0.001661050715483725,
+            -0.021557753905653954,
+            -0.0009731980971992016,
+            0.00042234905413351953,
+            -0.013159905560314655,
+            0.0009486319031566381,
+            0.02124028466641903,
+            0.02255551889538765,
+            0.02211710810661316,
+            -0.004274512641131878,
+            0.005703129805624485,
+            0.013144788332283497,
+            -0.007573937531560659,
+            -0.026531457901000977,
+            -0.008239113725721836,
+            -0.024112636223435402,
+            0.021633343771100044,
+            0.004414350725710392,
+            -0.00011361851647961885,
+            0.026440752670168877,
+            -0.023900989443063736,
+            0.012086553499102592,
+            0.009629936888813972,
+            -0.001953955041244626,
+            0.010053230449557304,
+            -0.00024920483701862395,
+            0.019244756549596786,
+            -0.025488341227173805,
+            0.012849994003772736,
+            -0.002855344209820032,
+            0.0067349085584282875,
+            0.015933992341160774,
+            -0.011942935176193714,
+            0.009486319497227669,
+            0.00047148141311481595,
+            -0.00418758625164628,
+            -0.011678377166390419,
+            -0.000992095097899437,
+            0.003769961418583989,
+            0.03079463355243206,
+            -0.011307994835078716,
+            0.004130895249545574,
+            0.015737462788820267,
+            0.009886936284601688,
+            0.014150111004710197,
+            0.01782369799911976,
+            0.0016667197924107313,
+            0.019002873450517654,
+            -0.014633875340223312,
+            0.006591291166841984,
+            0.009516553953289986,
+            -0.01522346306592226,
+            -0.006829394027590752,
+            -0.026289576664566994,
+            -0.0005437626386992633,
+            -0.003998615778982639,
+            -0.011610347777605057,
+            -0.009047907777130604,
+            -0.028496751561760902,
+            -0.007800702471286058,
+            -0.026213988661766052,
+            -0.007313158363103867,
+            -0.008019908331334591,
+            0.000944380066357553,
+            -0.000724229437764734,
+            0.01782369799911976,
+            -0.004678909666836262,
+            -0.004875438753515482,
+            0.021542636677622795,
+            -0.011421376839280128,
+            -0.00038644467713311315,
+            -0.012041199952363968,
+            0.0013596427161246538,
+            -0.0072526875883340836,
+            0.011549877002835274,
+            0.005752262193709612,
+            -0.00023007158597465605,
+            -0.002220403403043747,
+            0.0080803781747818,
+            -0.005465026944875717,
+            0.019002873450517654,
+            -0.03386351466178894,
+            0.01975875534117222,
+            0.010053230449557304,
+            -0.004678909666836262,
+            -0.020166931673884392,
+            -0.02052975445985794,
+            -0.021739166229963303,
+            0.005408335942775011,
+            0.026712870225310326,
+            -0.013386670500040054,
+            -0.006304055918008089,
+            -0.00480363005772233,
+            -0.01079399511218071,
+            0.0024944106116890907,
+            0.015231022611260414,
+            -0.027257105335593224,
+            0.02387075312435627,
+            -0.007914084941148758,
+            0.009902054443955421,
+            -0.008851378224790096,
+            0.003518630750477314,
+            0.00126704724971205,
+            -0.009803789667785168,
+            -0.0031917118467390537,
+            -0.018050462007522583,
+            0.010302671231329441,
+            0.025291811674833298,
+            -0.008934524841606617,
+            -0.025760458782315254,
+            0.007997231557965279,
+            -0.011005641892552376,
+            -0.012668582610785961,
+            -0.015556051395833492,
+            0.0027816458605229855,
+            0.015858404338359833,
+            0.002666373737156391,
+            0.022525284439325333,
+            0.0009061135351657867,
+            -0.0053176297806203365,
+            -0.006198232527822256,
+            0.0136058758944273,
+            0.010544554330408573,
+            -0.033289045095443726,
+            -0.008783348836004734,
+            0.0290409866720438,
+            -0.020061109215021133,
+            -0.004395453725010157,
+            0.003435483667999506,
+            0.026455869898200035,
+            -0.01171617116779089,
+            0.009629936888813972,
+            0.0003321156546007842,
+            0.001567510305903852,
+            -0.02009134367108345,
+            0.0010034333681687713,
+            0.07752326130867004,
+            0.0179899912327528,
+            -0.017128286883234978,
+            -0.00349784386344254,
+            -0.013658788055181503,
+            -0.012842435389757156,
+            0.00894964300096035,
+            -0.025382516905665398,
+            0.00860193744301796,
+            -0.00961481872946024,
+            -0.021890342235565186,
+            0.015752580016851425,
+            -0.00013605876301880926,
+            0.020408814772963524,
+            -0.03359139710664749,
+            0.012215052731335163,
+            0.015737462788820267,
+            0.01631193421781063,
+            0.01676546223461628,
+            0.0009358763927593827,
+            -0.013930905610322952,
+            -7.009624823695049e-05,
+            0.00649680569767952,
+            0.017566697672009468,
+            0.025397635996341705,
+            0.011534759774804115,
+            -0.012555199675261974,
+            0.03126328065991402,
+            0.005204247776418924,
+            0.0229183416813612,
+            0.013001170009374619,
+            -0.0021807195153087378,
+            -0.007808261085301638,
+            -0.007604172918945551,
+            0.014883317053318024,
+            0.01833769679069519,
+            -0.0036471306812018156,
+            -0.014490257948637009,
+            0.005960129667073488,
+            0.004985041916370392,
+            -0.006481688003987074,
+            -0.005438570864498615,
+            -0.0011205950286239386,
+            0.013840199448168278,
+            -0.009395613335072994,
+            0.026727987453341484,
+            0.00771755538880825,
+            0.0014087751042097807,
+            -0.015246139839291573,
+            -0.0015911315567791462,
+            -0.006617746781557798,
+            0.007728893309831619,
+            0.004796071443706751,
+            0.009350260719656944,
+            -0.0005513214273378253,
+            0.00126704724971205,
+            0.012925582006573677,
+            -0.03480080887675285,
+            -0.0007577717187814415,
+            0.023114871233701706,
+            -0.0026210208889096975,
+            -0.006889864336699247,
+            -0.009592142887413502,
+            -0.018352815881371498,
+            0.001622311770915985,
+            0.002823219168931246,
+            0.010582348331809044,
+            -0.0019917490426450968,
+            0.010703288950026035,
+            0.02885957434773445,
+            0.006727349944412708,
+            -0.016357285901904106,
+            0.0228276364505291,
+            0.006258702836930752,
+            0.008722878061234951,
+            0.014225699007511139,
+            -0.006236026529222727,
+            0.006118864752352238,
+            -0.010242201387882233,
+            -0.0066290851682424545,
+            0.02359863556921482,
+            0.01044628955423832,
+            -0.011549877002835274,
+            -0.004765836056321859,
+            0.014618758112192154,
+            -0.0265768114477396,
+            0.012744170613586903,
+            0.0019218300003558397,
+            0.013719257898628712,
+            1.0371232747274917e-05,
+            -0.016281697899103165,
+            -0.020076226443052292,
+            0.00599036505445838,
+            0.0032502925023436546,
+            -0.013054082170128822,
+            -0.018972638994455338,
+            0.0008801301009953022,
+            -0.014709463343024254,
+            -0.030159691348671913,
+            -0.021028637886047363,
+            0.014732140116393566,
+            0.0007105291006155312,
+            0.0076684230007231236,
+            0.010045671835541725,
+            -0.0025246459990739822,
+            0.007876290008425713,
+            0.02754434011876583,
+            0.014051846228539944,
+            0.018912168219685555,
+            -0.00829202588647604,
+            -0.002518976805731654,
+            0.011043435893952847,
+            0.011542318388819695,
+            0.01737016811966896,
+            -0.0018755322089418769,
+            -0.01983434334397316,
+            -0.013681463897228241,
+            0.01762716844677925,
+            -0.023371871560811996,
+            -0.02306951954960823,
+            0.013144788332283497,
+            -0.02341722510755062,
+            0.0002966836909763515,
+            -0.002492520958185196,
+            0.007573937531560659,
+            -0.003401468973606825,
+            -0.006001702975481749,
+            0.0018027785699814558,
+            -0.028451398015022278,
+            -0.03607068955898285,
+            -0.0024471678771078587,
+            0.009017672389745712,
+            -0.001181065570563078,
+            0.004221600946038961,
+            0.005933673586696386,
+            0.009070583619177341,
+            -0.01570722833275795,
+            -0.011610347777605057,
+            -0.007384967058897018,
+            0.010166613385081291,
+            0.008012348785996437,
+            0.027876928448677063,
+            -0.005449909251183271,
+            0.0035734320990741253,
+            0.01800510846078396,
+            0.009773554280400276,
+            -0.002562440000474453,
+            -0.004350101109594107,
+            0.012933140620589256,
+            -0.011345788836479187,
+            0.006292717531323433,
+            -0.019516874104738235,
+            0.0011782310903072357,
+            0.0018897049594670534,
+            0.009992759674787521,
+            0.019531991332769394,
+            -0.019199403002858162,
+            -0.0007086393889039755,
+            -0.008276907727122307,
+            -0.016795698553323746,
+            0.005669115111231804,
+            0.025216223672032356,
+            -0.004675130359828472,
+            0.018640050664544106,
+            -0.005994144361466169,
+            0.026365164667367935,
+            -0.01044628955423832,
+            0.013794846832752228,
+            0.0047167036682367325,
+            -0.004353880416601896,
+            0.008057702332735062,
+            0.01426349300891161,
+            -0.005204247776418924,
+            0.01570722833275795,
+            -0.012880229391157627,
+            0.016281697899103165,
+            0.0032011603470891714,
+            -0.0014805839164182544,
+            0.0018452969379723072,
+            0.0064627910032868385,
+            -0.02254040166735649,
+            0.011958053335547447,
+            0.01202608272433281,
+            -0.003779409918934107,
+            0.02195081301033497,
+            -0.011973170563578606,
+            -0.006973011419177055,
+            -0.0014730250695720315,
+            0.021905461326241493,
+            -0.02929798699915409,
+            0.01860981434583664,
+            0.0223741065710783,
+            -0.0005872258334420621,
+            -0.011935376562178135,
+            0.02124028466641903,
+            0.011345788836479187,
+            0.002981954487040639,
+            -0.009388054721057415,
+            0.020030872896313667,
+            -0.0006160438060760498,
+            -0.0027627486269921064,
+            0.009750877507030964,
+            -0.01087714172899723,
+            0.007577716838568449,
+            0.014769934117794037,
+            -0.00952411349862814,
+            0.0017215212574228644,
+            0.006220908835530281,
+            0.016251463443040848,
+            -0.015661874786019325,
+            -0.025820929557085037,
+            -0.018972638994455338,
+            0.024808047339320183,
+            -0.0022808739449828863,
+            -0.0064627910032868385,
+            -0.024883635342121124,
+            0.008186202496290207,
+            -0.026002340018749237,
+            0.013084317557513714,
+            -0.011519641615450382,
+            -0.002025763737037778,
+            -0.0007865896914154291,
+            0.005344085860997438,
+            0.00807281956076622,
+            0.018141167238354683,
+            0.0015618412289768457,
+            -0.014694346114993095,
+            -0.010914935730397701,
+            0.010302671231329441,
+            -0.001181065570563078,
+            -0.007048599887639284,
+            0.005657776724547148,
+            -0.005929894279688597,
+            0.0006406100001186132,
+            0.02378004789352417,
+            0.007403864059597254,
+            -0.0020975726656615734,
+            -0.005087085999548435,
+            -0.011436494998633862,
+            -0.006213350221514702,
+            -0.00021152885165065527,
+            -0.029615456238389015,
+            -0.0005579353892244399,
+            0.007815820164978504,
+            0.0039268070831894875,
+            0.0010818560840561986,
+            -0.007282922975718975,
+            -0.016916640102863312,
+            0.008473437279462814,
+            -2.8035545255988836e-05,
+            -0.02071116678416729,
+            -0.018624933436512947,
+            0.0003132185956928879,
+            0.010340466164052486,
+            0.0046335565857589245,
+            0.010506759397685528,
+            0.014543170109391212,
+            0.018035344779491425,
+            0.01746087335050106,
+            -0.002492520958185196,
+            -0.01895751990377903,
+            -0.006084850057959557,
+            0.013628552667796612,
+            -0.02833045832812786,
+            0.001375705236569047,
+            0.013636111281812191,
+            -0.0034997337497770786,
+            -0.005453688558191061,
+            -0.023674223572015762,
+            0.019093578681349754,
+            0.005933673586696386,
+            -0.004138453863561153,
+            0.00837517250329256,
+            0.010211966000497341,
+            -0.016523580998182297,
+            -0.00286101340316236,
+            0.003199270460754633,
+            -0.007294261362403631,
+            0.003316432237625122,
+            -0.01334131695330143,
+            -0.0058845411986112595,
+            -0.005079526919871569,
+            -0.002764638513326645,
+            0.003686814568936825,
+            0.012713935226202011,
+            -0.0014427897986024618,
+            -0.012245288118720055,
+            0.012910464778542519,
+            0.0003134548314847052,
+            0.019184285774827003,
+            -0.008186202496290207,
+            0.008768231607973576,
+            -0.020151814445853233,
+            -0.012751729227602482,
+            0.000203733827220276,
+            -0.012736611999571323,
+            0.015072287060320377,
+            -0.0049056741409003735,
+            0.006368305999785662,
+            0.0025511018466204405,
+            -0.01456584595143795,
+            -0.007426540832966566,
+            0.003635792527347803,
+            0.000584391294978559,
+            0.01622122712433338,
+            0.014898434281349182,
+            0.015367081388831139,
+            0.0012717714998871088,
+            0.004350101109594107,
+            -0.02938869222998619,
+            -0.0074454378336668015,
+            0.013711699284613132,
+            0.022404342889785767,
+            0.0019284439040347934,
+            0.008511231280863285,
+            0.013469817116856575,
+            -0.0009835914243012667,
+            -0.0113684656098485,
+            -0.019290108233690262,
+            0.016916640102863312,
+            -0.025911634787917137,
+            0.01417278777807951,
+            0.006640423554927111,
+            -0.003197380807250738,
+            0.0028458957094699144,
+            0.010204407386481762,
+            0.019940167665481567,
+            -0.01207143533974886,
+            -0.011806877329945564,
+            -0.012116788886487484,
+            0.019547108560800552,
+            -0.008458320051431656,
+            -0.009599701501429081,
+            0.000633523624856025,
+            -0.02080187201499939,
+            -0.003979718778282404,
+            0.01421814039349556,
+            -0.02264622412621975,
+            -0.006285158917307854,
+            0.006613967474550009,
+            -0.00873043667525053,
+            -0.02449057623744011,
+            0.00832981988787651,
+            0.006870967335999012,
+            -0.0026474767364561558,
+            0.007630628999322653,
+            -0.017702756449580193,
+            -0.04359927400946617,
+            0.010559671558439732,
+            0.0019426167709752917,
+            -0.013477375730872154,
+            0.0014342861250042915,
+            -0.003274858696386218,
+            -0.006474129389971495,
+            -0.004168689250946045,
+            -0.011648141779005527,
+            0.01965293288230896,
+            0.023009048774838448,
+            -0.007959437556564808,
+            0.0020427710842341185,
+            -0.027045458555221558,
+            0.01983434334397316,
+            -0.007709996309131384,
+            -0.0011999626876786351,
+            -0.010597465559840202,
+            0.0031784838065505028,
+            0.04311550781130791,
+            -0.015382198616862297,
+            -0.011300436221063137,
+            -0.014512934722006321,
+            0.0008928856113925576,
+            -0.02124028466641903,
+            -0.017687639221549034,
+            0.0012472053058445454,
+            -0.021210048347711563,
+            0.012600553222000599,
+            0.0022298519033938646,
+            0.013507611118257046,
+            0.00108563550747931,
+            -0.008397849276661873,
+            -0.005427232943475246,
+            -0.0223741065710783,
+            0.005831629503518343,
+            0.011814435943961143,
+            0.009221760556101799,
+            0.013855316676199436,
+            0.006198232527822256,
+            0.005616203416138887,
+            -0.0018538006115704775,
+            0.0033806823194026947,
+            -0.016266580671072006,
+            0.014588522724807262,
+            -0.018352815881371498,
+            -0.004516394808888435,
+            -0.004380336031317711,
+            -0.012003405950963497,
+            -0.0015542823821306229,
+            0.029086340218782425,
+            -0.004746939055621624,
+            -0.00023727607913315296,
+            -0.004350101109594107,
+            0.0006084850174374878,
+            0.017612051218748093,
+            0.00868508405983448,
+            0.02677334100008011,
+            -0.008889172226190567,
+            0.00495858583599329,
+            -0.022842753678560257,
+            0.004391674418002367,
+            0.006455232389271259,
+            -0.011360906064510345,
+            0.008065260946750641,
+            0.017052698880434036,
+            0.005790056195110083,
+            0.04459703713655472,
+            -0.01895751990377903,
+            0.0017479771049693227,
+            -0.03555668890476227,
+            0.023371871560811996,
+            -0.0007969830767251551,
+            0.004486159887164831,
+            -0.0028062118217349052,
+            0.011685935780405998,
+            -0.000542817753739655,
+            -0.008926966227591038,
+            0.010922495275735855,
+            0.013235493563115597,
+            -0.02140657790005207,
+            0.0011952384375035763,
+            -0.035314805805683136,
+            0.0029233735986053944,
+            0.0004955751355737448,
+            0.006326732225716114,
+            -0.0066442028619349,
+            -0.01800510846078396,
+            0.01021952461451292,
+            -0.00873043667525053,
+            0.00952411349862814,
+            0.0008295804727822542,
+            0.023825401440262794,
+            -0.020680932328104973,
+            0.03156563267111778,
+            -0.0004497497866395861,
+            -0.005117321386933327,
+            0.013106994330883026,
+            -0.011111465282738209,
+            -0.004599541891366243,
+            -0.01563164032995701,
+            -0.008216436952352524,
+            0.013122111558914185,
+            0.002330006333068013,
+            0.015321727842092514,
+            -0.011935376562178135,
+            0.009660172276198864,
+            0.017566697672009468,
+            -0.007838496007025242,
+            0.0007502128719352186,
+            0.004229160025715828,
+            0.011050994507968426,
+            0.005805173888802528,
+            -0.00930490717291832,
+            -0.006334291305392981,
+            -0.006750026252120733,
+            -0.022071754559874535,
+            -0.01667475700378418,
+            -0.012222612276673317,
+            -0.00526471808552742,
+            0.004391674418002367,
+            0.0061566587537527084,
+            -0.0018178962636739016,
+            0.008488554507493973,
+            -0.0006769868195988238,
+            -0.016553815454244614,
+            0.0006580897606909275,
+            0.013182582333683968,
+            0.015155433677136898,
+            0.007740231696516275,
+            0.0011309884721413255,
+            0.029086340218782425,
+            0.0013794846599921584,
+            -0.007861172780394554,
+            0.005060629919171333,
+            0.0003578628820832819,
+            -0.025034811347723007,
+            0.007543702609837055,
+            -0.019531991332769394,
+            0.005083306692540646,
+            -0.016372404992580414,
+            0.007929202169179916,
+            0.028632810339331627,
+            0.009115937165915966,
+            -0.0006354133365675807,
+            -0.012479611672461033,
+            0.007997231557965279,
+            -0.01596422865986824,
+            -0.01727946288883686,
+            0.011081229895353317,
+            5.4535707022296265e-05,
+            0.011194612830877304,
+            0.014694346114993095,
+            -0.008737996220588684,
+            -0.02043904922902584,
+            -0.00034061932819895446,
+            -0.015193228609859943,
+            0.00864729005843401,
+            0.01074864249676466,
+            0.0011461060494184494,
+            -0.014905992895364761,
+            0.014989140443503857,
+            0.0042782919481396675,
+            -0.00987181905657053,
+            -0.016735227778553963,
+            -0.03090045601129532,
+            0.01210922934114933,
+            0.011171936057507992,
+            0.025563929229974747,
+            -0.009645054116845131,
+            -0.005941232666373253,
+            -0.011648141779005527,
+            0.004323645029217005,
+            0.005948791280388832,
+            -0.019637813791632652,
+            0.014905992895364761,
+            -0.01382508222013712,
+            -0.0024963002651929855,
+            0.0027533001266419888,
+            -0.0024944106116890907,
+            -0.00895720161497593,
+            -0.005465026944875717,
+            0.010400936007499695,
+            -0.0032238366547971964,
+            0.0014853081665933132,
+            0.013212817721068859,
+            -0.0006325787398964167,
+            0.0199099313467741,
+            -0.02376493066549301,
+            0.03271457180380821,
+            0.006202011834830046,
+            3.097344597335905e-05,
+            0.00033376915962435305,
+            -0.008019908331334591,
+            -0.017491109669208527,
+            0.00215048436075449,
+            0.02299392968416214,
+            -0.004471042193472385,
+            -0.0113684656098485,
+            0.00046510365791618824,
+            0.0060735116712749004,
+            -0.013673905283212662,
+            -0.012970935553312302,
+            -0.01267614122480154,
+            0.01359831728041172,
+            -0.00378318945877254,
+            -0.005158894695341587,
+            0.029902692884206772,
+            0.0063002766110002995,
+            0.030945809558033943,
+            -0.013537846505641937,
+            -0.005412115249782801,
+            4.6858778659952804e-05,
+            0.013893111608922482,
+            0.004308527335524559,
+            -0.02122516743838787,
+            0.009403171949088573,
+            0.00011208312935195863,
+            0.017218992114067078,
+            -0.0033353292383253574,
+            -0.009697966277599335,
+            -0.005865644197911024,
+            0.015276375226676464,
+            -0.003953262697905302,
+            0.011436494998633862,
+            0.006149100139737129,
+            0.016568932682275772,
+            0.012623229064047337,
+            -0.01232843566685915,
+            0.005858085583895445,
+            -0.009939848445355892,
+            0.011897582560777664,
+            -0.02107398957014084,
+            -0.007539922837167978,
+            0.01175396516919136,
+            0.01390822883695364,
+            0.00245661661028862,
+            -0.0011621685698628426,
+            0.0009722532122395933,
+            0.0043387627229094505,
+            -0.005653997417539358,
+            0.012865111231803894,
+            -0.01176152378320694,
+            0.013938464224338531,
+            0.004731821361929178,
+            -0.007460555527359247,
+            -0.000914617208763957,
+            0.0007818654412403703,
+            -0.004463483113795519,
+            -0.006538379471749067,
+            -0.01176152378320694,
+            -0.016689874231815338,
+            -0.014490257948637009,
+            -0.01202608272433281,
+            0.014701904729008675,
+            0.011897582560777664,
+            0.00860193744301796,
+            0.008662407286465168,
+            0.011376024223864079,
+            -0.01885169744491577,
+            0.0071733202785253525,
+            -0.002624800195917487,
+            -0.003779409918934107,
+            0.02087746001780033,
+            -0.00232622679322958,
+            -0.014293728396296501,
+            0.0023791384883224964,
+            0.01718875765800476,
+            0.01513275783509016,
+            -0.00019511203572619706,
+            0.006663099862635136,
+            0.004387895110994577,
+            0.02281251922249794,
+            -0.019426167011260986,
+            -0.0032219470012933016,
+            -0.007815820164978504,
+            0.012192376889288425,
+            0.02263110689818859,
+            -0.004523953888565302,
+            -0.015276375226676464,
+            -0.03458916023373604,
+            0.013832640834152699,
+            -0.014671669341623783,
+            0.0014390103751793504,
+            0.013568081893026829,
+            -0.028315339237451553,
+            0.013749493286013603,
+            -0.022495048120617867,
+            0.007675981614738703,
+            0.017793461680412292,
+            -0.016977109014987946,
+            -0.016130521893501282,
+            -0.0036849246826022863,
+            0.004244277253746986,
+            0.025594163686037064,
+            -0.020590225234627724,
+            -0.029706163331866264,
+            -0.0022525284439325333,
+            -0.007808261085301638,
+            -0.011897582560777664,
+            -0.008004790171980858,
+            -0.009554348886013031,
+            0.02492898888885975,
+            -0.019093578681349754,
+            0.003085888223722577,
+            -0.02993292734026909,
+            -0.0066064088605344296,
+            -0.011315553449094296,
+            -0.007573937531560659,
+            -0.019199403002858162,
+            -0.005665335804224014,
+            -0.010725965723395348,
+            -0.0004521119117271155,
+            -0.008050143718719482,
+            0.0004632139462046325,
+            -0.019199403002858162,
+            0.012131906114518642,
+            0.008231555111706257,
+            0.011685935780405998,
+            0.013507611118257046,
+            -0.007951878942549229,
+            -0.01966805011034012,
+            -0.00599036505445838,
+            0.01114925928413868,
+            0.0017319145845249295,
+            0.009085701778531075,
+            -0.0057295854203403,
+            0.016901521012187004,
+            -0.025775576010346413,
+            0.004119556862860918,
+            0.018806343898177147,
+            0.007400084752589464,
+            0.01539731677621603,
+            -0.02054487355053425,
+            0.011648141779005527,
+            -0.00618311483412981,
+            -0.0016988448332995176,
+            -6.315010068647098e-06,
+            -0.003911689389497042,
+            -0.005676673725247383,
+            0.019093578681349754,
+            0.016190992668271065,
+            -0.010393377393484116,
+            -0.002549211960285902,
+            0.0016260911943390965,
+            -0.015556051395833492,
+            -0.004433247726410627,
+            -0.00013015343574807048,
+            -0.0034090278204530478,
+            0.005548174027353525,
+            0.0041497922502458096,
+            -0.012842435389757156,
+            -0.005053071305155754,
+            0.0019445064244791865,
+            -0.00023975632211659104,
+            -0.01711316779255867,
+            -0.007959437556564808,
+            -0.00476205674931407,
+            -0.009894495829939842,
+            -0.004637335892766714,
+            0.004183806944638491,
+            0.02210198901593685,
+            -0.0010005987714976072,
+            -0.005767379887402058,
+            -0.001888760132715106,
+            -0.0203634612262249,
+            0.0026153516955673695,
+            0.00038644467713311315,
+            -0.0020919034723192453,
+            -0.016100287437438965,
+            0.008715319447219372,
+            0.002876131096854806,
+            0.017581814900040627,
+            -0.026546575129032135,
+            -0.002184499055147171,
+            -0.014550728723406792,
+            -0.01426349300891161,
+            -0.02615351788699627,
+            0.01417278777807951,
+            -0.011685935780405998,
+            -0.0012207494582980871,
+            -0.01879122667014599,
+            -0.011481847614049911,
+            0.0073622907511889935,
+            0.003960821777582169,
+            0.022268284112215042,
+            -0.03582880645990372,
+            -0.028239751234650612,
+            0.00726402597501874,
+            -0.006761364638805389,
+            0.006304055918008089,
+            -0.019244756549596786,
+            0.010597465559840202,
+            -0.009697966277599335,
+            -0.021965930238366127,
+            -0.0006708452710881829,
+            0.02308463677763939,
+            -0.029086340218782425,
+            -0.04021292179822922,
+            -0.02569998800754547,
+            0.016266580671072006,
+            0.006836952641606331,
+            0.022389225661754608,
+            0.014082081615924835,
+            -0.0040666451677680016,
+            -0.007298040669411421,
+            -0.010544554330408573,
+            0.0036509099882096052,
+            0.019804108887910843,
+            -0.011890023946762085,
+            -0.0061075263656675816,
+            0.00015991629334166646,
+            0.021346107125282288,
+            -0.0009604425868019462,
+            0.013938464224338531,
+            0.006213350221514702,
+            -0.017400404438376427,
+            -0.004637335892766714,
+            0.0001971198507817462,
+            -0.005170233082026243,
+            0.0018972638063132763,
+            0.0011744516668841243,
+            -0.0025038591120392084,
+            0.010862024500966072,
+            0.007974554784595966,
+            -0.004622218664735556,
+            -0.013575640507042408,
+            -0.003023528028279543,
+            0.0017999439733102918,
+            -0.018730755895376205,
+            0.021799637004733086,
+            0.008889172226190567,
+            -0.0054839239455759525,
+            -0.004773394670337439,
+            0.0052382624708116055,
+            0.022147342562675476,
+            0.004263174720108509,
+            0.005298732779920101,
+            0.007638187613338232,
+            0.007071276195347309,
+            0.00863973144441843,
+            -0.0014796389732509851,
+            0.01052943617105484,
+            -0.003289976390078664,
+            0.017400404438376427,
+            -0.002144815167412162,
+            -0.015737462788820267,
+            -0.0009599701152183115,
+            -0.007475673221051693,
+            0.009622378274798393,
+            -0.00995496567338705,
+            0.011474289000034332,
+            0.015427551232278347,
+            0.008881613612174988,
+            0.0011961832642555237,
+            0.010189289227128029,
+            -0.014323963783681393,
+            0.027514105662703514,
+            -0.00039683806244283915,
+            -0.0010270546190440655,
+            0.009645054116845131,
+            0.005544394254684448,
+            -0.00041526267887093127,
+            -0.0011054774513468146,
+            0.018473755568265915,
+            -0.006663099862635136,
+            -0.02606281079351902,
+            0.022434577345848083,
+            -0.01974363811314106,
+            -0.018640050664544106,
+            0.010007877834141254,
+            -0.006867188028991222,
+            0.009577024728059769,
+            0.004459703806787729,
+            0.009206642396748066,
+            0.03168657422065735,
+            0.005438570864498615,
+            -0.01966805011034012,
+            -0.002471734071150422,
+            -0.010824230499565601,
+            0.008571702055633068,
+            -0.02176940254867077,
+            -0.0059223356656730175,
+            -0.009947407059371471,
+            0.004051527474075556,
+            -0.0021183593198657036,
+            -0.0018434072844684124,
+            0.017959756776690483,
+            0.0021674917079508305,
+            0.012131906114518642,
+            -0.001987969735637307,
+            -0.004769615363329649,
+            0.03186798468232155,
+            -0.0008933580247685313,
+            -0.014233258552849293,
+            -0.004951027221977711,
+            -0.0053176297806203365,
+            -0.039789628237485886,
+            -0.01083178911358118,
+            -0.0023696899879723787,
+            0.01703757978975773,
+            -0.0011120913550257683,
+            0.00833737850189209,
+            -0.007698658388108015,
+            -0.00695789372548461,
+            0.0004176248039584607,
+            0.003666027681902051,
+            -0.0009273727191612124,
+            0.02149728499352932,
+            -0.007793143391609192,
+            0.002653145929798484,
+            -0.002394256182014942,
+            -0.02817928045988083,
+            -0.02527669444680214,
+            -0.002486851764842868,
+            -0.022510165348649025,
+            0.0024981899186968803,
+            0.013205258175730705,
+            -0.012131906114518642,
+            -0.026017459109425545,
+            0.0031104544177651405,
+            -0.02379516512155533,
+            0.03613115847110748,
+            0.011655700393021107,
+            -0.0028591237496584654,
+            -0.014603639952838421,
+            -0.007581496611237526,
+            -0.0037718513049185276,
+            -0.020937930792570114,
+            -0.006791599560528994,
+            0.009289789944887161,
+            -0.0014210582012310624,
+            0.0016147529240697622,
+            0.0021618225146085024,
+            -0.04220845177769661,
+            -0.016251463443040848,
+            0.013961140997707844,
+            -0.0319586917757988,
+            0.018670285120606422,
+            0.03088533878326416,
+            -0.0011215399717912078,
+            0.0031028955709189177,
+            -0.0033258807379752398,
+            0.005257159471511841,
+            0.010295112617313862,
+            0.005646438337862492,
+            -0.00903278961777687,
+            -0.010967847891151905,
+            -0.015949109569191933,
+            0.025140635669231415,
+            0.009108378551900387,
+            -0.01263078860938549,
+            0.008768231607973576,
+            0.011353347450494766,
+            0.004002395085990429,
+            -0.004603321198374033,
+            0.0035715424455702305,
+            -0.02501969411969185,
+            -0.005895879585295916,
+            -0.0032219470012933016,
+            0.013099434785544872,
+            0.01267614122480154,
+            0.0037246085703372955,
+            0.00495858583599329,
+            0.007736452389508486,
+            -0.002653145929798484,
+            -0.0060735116712749004,
+            -0.02387075312435627,
+            0.020061109215021133,
+            -0.00042943545849993825,
+            0.0036981527227908373,
+            0.0080803781747818,
+            0.0025246459990739822,
+            0.002176940208300948,
+            -0.008012348785996437,
+            0.011519641615450382,
+            0.01903310790657997,
+            0.016115404665470123,
+            -0.006220908835530281,
+            0.007589055225253105,
+            -0.01259299460798502,
+            0.003051873529329896,
+            0.010635259561240673,
+            -0.002365910680964589,
+            -0.008307143114507198,
+            -0.025473223999142647,
+            0.00033754855394363403,
+            -0.01083934772759676,
+            -0.0007251743227243423,
+            -0.0002626689965836704,
+            0.0010629590833559632,
+            -0.00726402597501874,
+            -0.018141167238354683,
+            0.0011952384375035763,
+            0.0026588148903101683,
+            0.020302990451455116,
+            -0.027317576110363007,
+            -0.002471734071150422,
+            0.017763227224349976,
+            0.009886936284601688,
+            0.007778025697916746,
+            0.019290108233690262,
+            0.005982805974781513,
+            0.0013719258131459355,
+            -0.007033482193946838,
+            -0.002777866320684552,
+            0.0197285208851099,
+            -0.002549211960285902,
+            0.008276907727122307,
+            -0.0145204933360219,
+            -0.014550728723406792,
+            -0.016871286556124687,
+            0.010514318943023682,
+            -0.013507611118257046,
+            -0.008261790499091148,
+            0.016992228105664253,
+            -0.002679601777344942,
+            -0.011958053335547447,
+            0.01718875765800476,
+            0.0015892419032752514,
+            0.0070674968883395195,
+            -0.003622564487159252,
+            0.0023791384883224964,
+            0.006326732225716114,
+            -0.0019019880564883351,
+            0.0035016234032809734,
+            0.005381879862397909,
+            -0.026032576337456703,
+            0.019154049456119537,
+            0.004372777417302132,
+            0.020665813237428665,
+            -0.009478759951889515,
+            0.017687639221549034,
+            -0.002324337139725685,
+            0.00802746694535017,
+            0.020378578454256058,
+            -0.016977109014987946,
+            0.008397849276661873,
+            0.003934365697205067,
+            0.015556051395833492,
+            -0.010378260165452957,
+            -0.0022449695970863104,
+            -0.012691258452832699,
+            -0.00199552858248353,
+            -0.006780261639505625,
+            -0.0024755136109888554,
+            -0.0010185509454458952,
+            -0.0005735254962928593,
+            -0.0030613220296800137,
+            -0.006602629087865353,
+            0.005249600391834974,
+            0.021875225007534027,
+            -0.009320025332272053,
+            -0.016478227451443672,
+            0.004017512779682875,
+            0.0018452969379723072,
+            0.021708931773900986,
+            0.0401826873421669,
+            -0.011890023946762085,
+            0.02376493066549301,
+            -0.019063344225287437,
+            -0.011481847614049911,
+            0.008798466064035892,
+            0.00307266041636467,
+            -0.013054082170128822,
+            -0.035586923360824585,
+            0.00587698258459568,
+            0.012645905837416649,
+            -0.009660172276198864,
+            0.02113446034491062,
+            0.014384434558451176,
+            0.005449909251183271,
+            -0.017869049683213234,
+            0.020151814445853233,
+            -0.006674438249319792,
+            0.0055670710280537605,
+            -0.00898743700236082,
+            0.016175875440239906,
+            -0.00648924708366394,
+            0.013666346669197083,
+            -0.016342168673872948,
+            0.004618438892066479,
+            0.008964760228991508,
+            0.02843628078699112,
+            -0.02133098989725113,
+            0.014497816562652588,
+            -0.016432873904705048,
+            -0.004875438753515482,
+            -0.005442350637167692,
+            0.009697966277599335,
+            -0.0001299172145081684,
+            0.006228467915207148,
+            -0.00011285082291578874,
+            -0.016357285901904106,
+            -0.0007147809374146163,
+            -0.005257159471511841,
+            -0.010703288950026035,
+            -0.0003616423055063933,
+            0.0022865431383252144,
+            0.005918555893003941,
+            0.01298605278134346,
+            0.011421376839280128,
+            -0.01201852411031723,
+            0.012736611999571323,
+            -0.041633982211351395,
+            0.001619477174244821,
+            -0.004289630334824324,
+            -0.0005791945732198656,
+            0.0007459610351361334,
+            -0.015661874786019325,
+            -0.012124347500503063,
+            -0.0070674968883395195,
+            0.010196847841143608,
+            -0.00552927702665329,
+            -0.009765995666384697,
+            -0.011769082397222519,
+            0.007592834532260895,
+            0.001629870617762208,
+            0.006625305861234665,
+            0.010552112944424152,
+            0.012260406278073788,
+            -0.033047161996364594,
+            -0.01263834722340107,
+            -0.019607579335570335,
+            0.00012212219007778913,
+            -0.005623762030154467,
+            -0.0010818560840561986,
+            -0.014422228559851646,
+            -0.02553369477391243,
+            -0.0010998082580044866,
+            0.0154351107776165,
+            0.013175023719668388,
+            -0.004206483252346516,
+            0.009796231053769588,
+            -0.0034676087088882923,
+            0.004180027637630701,
+            -0.018141167238354683,
+            0.002394256182014942,
+            0.004100659862160683,
+            0.0036905938759446144,
+            0.020726284012198448,
+            -0.019788991659879684,
+            0.01921452023088932,
+            0.014278611168265343,
+            0.0019936386961489916,
+            0.011028318665921688,
+            -0.0011801207438111305,
+            0.0066442028619349,
+            -0.008465878665447235,
+            -0.012608111836016178,
+            0.005559511948376894,
+            -0.0010506759863346815,
+            0.008405407890677452,
+            0.006504364777356386,
+            0.005627541337162256,
+            0.008087937720119953,
+            0.020302990451455116,
+            -0.008163525722920895,
+            0.0028043221682310104,
+            -0.01596422865986824,
+            -0.0006836007814854383,
+            -0.0009590252884663641,
+            -0.004414350725710392,
+            0.013205258175730705,
+            0.0024736239574849606,
+            -0.013885552063584328,
+            -0.011391141451895237,
+            -0.02405216544866562,
+            0.011973170563578606,
+            0.014437345787882805,
+            0.003580990945920348,
+            0.010620142333209515,
+            -0.004951027221977711,
+            0.001301061944104731,
+            0.01545022800564766,
+            -0.007033482193946838,
+            -0.028904927894473076,
+            0.0055670710280537605,
+            0.010234642773866653,
+            0.023553283885121346,
+            -0.0020767857786267996,
+            -0.004845203831791878,
+            0.00418758625164628,
+            0.0017914402997121215,
+            0.014414669945836067,
+            0.005638879723846912,
+            0.012230170890688896,
+            0.019063344225287437,
+            -0.006013041362166405,
+            0.007010805420577526,
+            0.005752262193709612,
+            0.01694687455892563,
+            -0.0160700511187315,
+            -7.039151387289166e-05,
+            0.02122516743838787,
+            0.009486319497227669,
+            0.01206387672573328,
+            -0.010068348608911037,
+            0.014588522724807262,
+            0.0036093364469707012,
+            0.0019142711535096169,
+            -0.004860321059823036,
+            0.0018065579934045672,
+            0.00434632133692503,
+            0.0047167036682367325,
+            0.02842116355895996,
+            -0.0028723515570163727,
+            0.008050143718719482,
+            -0.040243156254291534,
+            0.016810815781354904,
+            0.00032148606260307133,
+            0.009131054393947124,
+            -0.0036527998745441437,
+            0.018367933109402657,
+            0.03088533878326416,
+            0.014104758389294147,
+            0.00511354161426425,
+            0.005771159194409847,
+            0.0034524910151958466,
+            -0.006413658615201712,
+            -0.011028318665921688,
+            0.004323645029217005,
+            0.004667571280151606,
+            -0.0074454378336668015,
+            0.014505375176668167,
+            0.0032994248904287815,
+            -0.014286169782280922,
+            -0.004992600530385971,
+            0.013273287564516068,
+            0.0006491136737167835,
+            -0.010680613107979298,
+            0.03483104333281517,
+            -0.012426700443029404,
+            -0.0016100286738947034,
+            -0.0001238937838934362,
+            -0.01563164032995701,
+            -0.0069843498058617115,
+            0.020333226770162582,
+            -0.014490257948637009,
+            -0.010884701274335384,
+            -0.037340570241212845,
+            0.012237729504704475,
+            0.0029082559049129486,
+            0.003282417543232441,
+            -0.014724581502377987,
+            -0.03942680358886719,
+            0.005022835917770863,
+            0.011194612830877304,
+            -0.007846055552363396,
+            0.009229319170117378,
+            0.00679915864020586,
+            -0.01198073010891676,
+            0.012789523229002953,
+            0.029691044241189957,
+            0.01817140355706215,
+            -0.01210922934114933,
+            0.004773394670337439,
+            0.024702223017811775,
+            0.008677525445818901,
+            -0.013522729277610779,
+            0.020756520330905914,
+            -0.02798275090754032,
+            0.007543702609837055,
+            0.0159793458878994,
+            -0.0014191684313118458,
+            0.0035847704857587814,
+            -8.899329259293154e-05,
+            0.015933992341160774,
+            0.004478600807487965,
+            0.008889172226190567,
+            0.03235175088047981,
+            -0.010673053562641144,
+            -0.00025322046712972224,
+            -0.013734376057982445,
+            -0.009463642723858356,
+            -0.0015391646884381771,
+            0.00645145308226347,
+            -0.02254040166735649,
+            -0.0007809206144884229,
+            0.002675822237506509,
+            -0.026274457573890686,
+            0.010642818175256252,
+            0.020680932328104973,
+            -0.016825933009386063,
+            -2.1598738385364413e-05,
+            0.014550728723406792,
+            0.017067816108465195,
+            0.010673053562641144,
+            -0.012509847059845924,
+            -0.0007492680451832712,
+            -0.009493878111243248,
+            -0.008722878061234951,
+            0.004229160025715828,
+            -0.02394634112715721,
+            0.03833833336830139,
+            -0.01229064166545868,
+            0.006024379748851061,
+            0.015601404942572117,
+            -0.004387895110994577,
+            -0.004973703529685736,
+            0.004667571280151606,
+            0.011557435616850853,
+            -0.023659106343984604,
+            0.03340998291969299,
+            -0.016024697571992874,
+            0.00491323322057724,
+            0.01703757978975773,
+            0.027075693011283875,
+            -0.007638187613338232,
+            0.0013416905421763659,
+            -0.00026030687149614096,
+            0.028889810666441917,
+            -4.538244684226811e-05,
+            -0.02325093001127243,
+            -0.01895751990377903,
+            -0.005136218387633562,
+            0.01938081532716751,
+            0.00757015822455287,
+            -0.015752580016851425,
+            0.028935162350535393,
+            -0.004939688835293055,
+            -0.01956222578883171,
+            -0.002256307750940323,
+            -0.0008097385871224105,
+            -0.0036206748336553574,
+            0.0317772775888443,
+            0.01017417199909687,
+            -0.01817140355706215,
+            -0.01965293288230896,
+            -0.008760672062635422,
+            -0.02686404623091221,
+            -0.0022789842914789915,
+            0.0004492773732636124,
+            -0.008874054998159409,
+            0.00301596918143332,
+            -0.012494729831814766,
+            -0.019622696563601494,
+            0.0012320876121520996,
+            0.014089640229940414,
+            0.04746938869357109,
+            0.001130043645389378,
+            0.007275364361703396,
+            -0.02624422311782837,
+            0.01272905245423317,
+            -0.011685935780405998,
+            -0.0033409984316676855,
+            -0.0027344031259417534,
+            -0.00925199594348669,
+            -0.00039022407145239413,
+            -0.006285158917307854,
+            -0.015646757557988167,
+            -0.029328221455216408,
+            0.01879122667014599,
+            0.002141035860404372,
+            -0.006311614532023668,
+            -0.013537846505641937,
+            -0.02387075312435627,
+            0.012759287841618061,
+            -0.01675034500658512,
+            -0.008624613285064697,
+            -0.0356171615421772,
+            -0.024989459663629532,
+            0.011285318061709404,
+            0.006882305722683668,
+            -0.0017583705484867096,
+            -0.0028855795972049236,
+            -0.013726817443966866,
+            -0.02124028466641903,
+            -0.03640327602624893,
+            0.009115937165915966,
+            -0.005355423782020807,
+            0.01622122712433338,
+            -0.011247524060308933,
+            -5.229168164078146e-05,
+            -0.006319173611700535,
+            0.008458320051431656,
+            0.020665813237428665,
+            0.01140625961124897,
+            0.003021638374775648,
+            0.002367800334468484,
+            0.01700734533369541,
+            -0.004463483113795519,
+            -0.02376493066549301,
+            -0.02518598921597004,
+            0.00286479271017015,
+            0.00599036505445838,
+            -0.01782369799911976,
+            0.009463642723858356,
+            0.0007790309027768672,
+            0.0019237196538597345,
+            0.009267113171517849,
+            0.0034317043609917164,
+            -0.0069843498058617115,
+            -0.010400936007499695,
+            -0.001878366805613041,
+            0.026969870552420616,
+            0.007815820164978504,
+            -0.010249760001897812,
+            -0.010718407109379768,
+            0.0023848076816648245,
+            -0.007203555665910244,
+            0.0011961832642555237,
+            0.012222612276673317,
+            -0.013794846832752228,
+            -0.001549558131955564,
+            -0.011307994835078716,
+            -0.010000319220125675,
+            -0.011988288722932339,
+            -0.017475992441177368,
+            0.0011961832642555237,
+            -0.000834304781164974,
+            -0.03854998201131821,
+            0.013855316676199436,
+            -0.0006779316463507712,
+            -0.007124187890440226,
+            -0.0015949109802022576,
+            0.007936760783195496,
+            0.0005286450032144785,
+            -0.003512961557134986,
+            -0.0002489686303306371,
+            0.018201638013124466,
+            -0.027181517332792282,
+            0.012970935553312302,
+            0.021618224680423737,
+            -0.017143404111266136,
+            -0.001795219723135233,
+            -0.009206642396748066,
+            0.010113701224327087,
+            0.009161289781332016,
+            0.012494729831814766,
+            -0.001033668639138341,
+            0.003919248003512621,
+            4.290220749680884e-05,
+            -0.0010431171394884586,
+            0.006999467499554157,
+            0.0028118810150772333,
+            0.010982965119183064,
+            0.00807281956076622,
+            0.006364526227116585,
+            -0.01026487722992897,
+            0.010453848168253899,
+            0.0025945650413632393,
+            -0.0006132092676125467,
+            0.0011253192787989974,
+            0.01390067022293806,
+            0.023659106343984604,
+            0.012351111508905888,
+            -0.015540934167802334,
+            0.019819226115942,
+            -0.013205258175730705,
+            0.006663099862635136,
+            0.00278731482103467,
+            0.0002310164418304339,
+            0.03477057069540024,
+            0.012555199675261974,
+            0.024021929129958153,
+            0.025639517232775688,
+            0.026652399450540543,
+            0.019970402121543884,
+            -0.0020030871964991093,
+            -0.015057169832289219,
+            -0.01623634621500969,
+            -0.002135366667062044,
+            0.019879696890711784,
+            0.0018481315346434712,
+            0.00027849528123624623,
+            -0.010461406782269478,
+            -0.002199616516008973,
+            -0.004304748028516769,
+            0.010642818175256252,
+            0.0053176297806203365,
+            -0.0034543806686997414,
+            -0.0037510644178837538,
+            -0.0007029702537693083,
+            0.0008220216841436923,
+            -0.008133290335536003,
+            -0.007400084752589464,
+            -0.004924571141600609,
+            0.0021334770135581493,
+            -0.03053763322532177,
+            0.05000915378332138,
+            -0.00803502555936575,
+            0.014354199171066284,
+            -0.009728201664984226,
+            0.003121792571619153,
+            0.004901894833892584,
+            -0.013507611118257046,
+            -0.002435829723253846,
+            -0.003199270460754633,
+            0.0026909399311989546,
+            -0.0147472582757473,
+            0.002702278085052967,
+            0.008571702055633068,
+            -0.014596081338822842,
+            0.0032011603470891714,
+            0.01149696484208107,
+            0.006753805559128523,
+            0.005933673586696386,
+            0.009025231003761292,
+            0.015193228609859943,
+            0.02464175410568714,
+            -0.00073462282307446,
+            -0.0005399832152761519,
+            0.0025322046130895615,
+            0.010695730336010456,
+            0.015828169882297516,
+            0.001142326625995338,
+            0.024883635342121124,
+            -0.0021126901265233755,
+            0.019063344225287437,
+            0.028194399550557137,
+            -0.010673053562641144,
+            -0.007490790449082851,
+            0.001940727001056075,
+            -0.01365122850984335,
+            0.008737996220588684,
+            -0.0019426167709752917,
+            -0.008019908331334591,
+            -0.013477375730872154,
+            0.001699789660051465,
+            0.02201128378510475,
+            0.00832981988787651,
+            -0.006511923391371965,
+            0.01263078860938549,
+            -0.002673932584002614,
+            0.0030197484884411097,
+            0.002269535791128874,
+            -0.0013917677570134401,
+            -0.0030367558356374502,
+            -0.010695730336010456,
+            0.01587352156639099,
+            -0.005230703391134739,
+            -0.021376343443989754,
+            0.009765995666384697,
+            -0.00434632133692503,
+            -0.008322261273860931,
+            0.03235175088047981,
+            -0.01622122712433338,
+            -0.006731129251420498,
+            -0.003365564625710249,
+            0.016296815127134323,
+            0.008019908331334591,
+            -0.0007880069897510111
+          ],
+          "title": "Switch Transformer",
+          "keyphrases": [
+            "Switch Transformer",
+            "Simplifying Sparse Routing",
+            "Efficient Sparse Routing",
+            "Improved Training",
+            "Fine-Tuning Techniques"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "d17e0745-ef0b-4687-8d99-5321eb6cab7e",
+        "properties": {
+          "page_content": "3 Scaling Properties 11 3.1 Scaling Results on a Step-Basis 12 3.2 Scaling Results on a Time-Basis 13 3.3 Scaling Versus a Larger Dense Model 13 ",
+          "keyphrases": [
+            "Scaling Properties",
+            "Scaling Results on a Step-Basis",
+            "Scaling Results on a Time-Basis",
+            "Scaling Versus a Larger Dense Model"
+          ],
+          "title": "Scaling Properties",
+          "embedding": [
+            -0.020178575068712234,
+            0.010542738251388073,
+            -0.025771139189600945,
+            0.015961479395627975,
+            0.009152155369520187,
+            -0.016702115535736084,
+            -0.001157245016656816,
+            0.07732853293418884,
+            -0.004825475625693798,
+            0.01630912534892559,
+            0.00782203208655119,
+            0.006609050091356039,
+            0.04265463352203369,
+            -0.01061831321567297,
+            0.03403906524181366,
+            0.045133501291275024,
+            0.008094102144241333,
+            0.006782873068004847,
+            -0.0005828736466355622,
+            -0.042836014181375504,
+            -0.026269935071468353,
+            -0.020103000104427338,
+            -0.027645403519272804,
+            0.029096446931362152,
+            0.025499068200588226,
+            0.0028454058337956667,
+            0.03932932764291763,
+            -0.0032780736219137907,
+            -0.010232879780232906,
+            0.0061329263262450695,
+            -0.011638578958809376,
+            0.002119883894920349,
+            -0.04081059992313385,
+            -0.002091543283313513,
+            -0.04446844011545181,
+            -0.02823488973081112,
+            0.06620386242866516,
+            -0.015024347230792046,
+            -0.0017486208816990256,
+            0.0305474903434515,
+            0.011358950287103653,
+            0.01002126932144165,
+            -0.01330123282968998,
+            -0.00594398844987154,
+            0.01584055833518505,
+            -0.04274532571434975,
+            -0.048398349434137344,
+            0.013391923159360886,
+            0.007130519021302462,
+            0.006314306985586882,
+            0.014230807311832905,
+            -0.0615786612033844,
+            0.040961749851703644,
+            -0.01794154942035675,
+            -0.02935340255498886,
+            0.01768459379673004,
+            -0.00886496901512146,
+            0.0859743282198906,
+            0.023745723068714142,
+            0.01330123282968998,
+            0.025272341445088387,
+            -0.028113968670368195,
+            -0.03201364725828171,
+            -0.006930244620889425,
+            0.0015370103064924479,
+            -0.006253846455365419,
+            -0.035550568252801895,
+            0.003850555745884776,
+            -0.03279963135719299,
+            -0.007655766326934099,
+            -0.033464692533016205,
+            -0.0024070697836577892,
+            0.0023749503307044506,
+            -0.01073167659342289,
+            0.006182050332427025,
+            -0.01117756962776184,
+            0.030834676697850227,
+            0.010504950769245625,
+            -0.014759833924472332,
+            0.029020871967077255,
+            0.052811939269304276,
+            0.039057254791259766,
+            -0.04205003380775452,
+            0.02893018163740635,
+            0.01701953262090683,
+            -0.03422044590115547,
+            -0.033011242747306824,
+            -0.055865176022052765,
+            0.009061465039849281,
+            -0.021266857162117958,
+            -0.0681990459561348,
+            -0.033736761659383774,
+            0.012983816675841808,
+            -0.03210433945059776,
+            -0.026919880881905556,
+            0.050967905670404434,
+            -0.01117001287639141,
+            0.010421818122267723,
+            -0.02122151292860508,
+            0.0011147339828312397,
+            -0.002512875013053417,
+            0.03597378730773926,
+            0.01862172596156597,
+            -0.01503946166485548,
+            0.02306554652750492,
+            -0.011600791476666927,
+            -0.0017051651375368237,
+            0.030199844390153885,
+            -0.006102696061134338,
+            0.05272125080227852,
+            0.0037315248046070337,
+            -0.01797178015112877,
+            -0.03198342025279999,
+            -0.03736437112092972,
+            -0.060036927461624146,
+            0.009363765828311443,
+            -0.00215578218922019,
+            -0.04594971239566803,
+            0.010527623817324638,
+            -0.02753959782421589,
+            0.02655711956322193,
+            0.02265744097530842,
+            -0.007481943350285292,
+            0.005112661514431238,
+            -0.032527558505535126,
+            0.04428705945611,
+            0.02823488973081112,
+            -0.02261209487915039,
+            -0.001841200515627861,
+            -0.02264232560992241,
+            0.004058387596160173,
+            0.019241442903876305,
+            -0.02985219843685627,
+            -0.007198536302894354,
+            -0.01142696850001812,
+            -0.016959071159362793,
+            0.050030771642923355,
+            0.025000272318720818,
+            -0.005399846937507391,
+            0.005161785054951906,
+            -0.011154897511005402,
+            -0.04071991145610809,
+            0.040689680725336075,
+            0.013452382758259773,
+            -0.03385768458247185,
+            0.0025582199450582266,
+            -0.028386039659380913,
+            0.062334414571523666,
+            -0.0035784849897027016,
+            -0.006261404138058424,
+            -0.02492469549179077,
+            -0.01747298240661621,
+            0.007746456656605005,
+            0.0467054657638073,
+            -0.0063445367850363255,
+            0.005543440114706755,
+            -0.018939141184091568,
+            0.0396013967692852,
+            -0.013928506523370743,
+            0.034432053565979004,
+            0.011661251075565815,
+            -0.007716226391494274,
+            0.014034312218427658,
+            0.01913563720881939,
+            0.028159314766526222,
+            0.00200652121566236,
+            -0.0012942251050844789,
+            0.022551635280251503,
+            -0.029217366129159927,
+            0.0015946364728733897,
+            0.02794770337641239,
+            0.011729269288480282,
+            0.012998932041227818,
+            0.01797178015112877,
+            -0.029459208250045776,
+            0.01653585024178028,
+            -0.02264232560992241,
+            0.026164129376411438,
+            -0.014880754053592682,
+            0.04005485028028488,
+            -0.003034343710169196,
+            -0.02941386215388775,
+            -0.011812401935458183,
+            0.021599387750029564,
+            0.030214959755539894,
+            -0.03056260570883751,
+            0.030456800013780594,
+            0.03518780693411827,
+            0.006627943832427263,
+            0.007213651668280363,
+            0.02353411354124546,
+            0.03476458415389061,
+            0.05646977946162224,
+            -0.00728922663256526,
+            -0.03286009281873703,
+            -0.027932588011026382,
+            0.010346243157982826,
+            -0.0043493518605828285,
+            0.01748809777200222,
+            0.026073439046740532,
+            -0.008320827968418598,
+            -0.02283882163465023,
+            0.0023995121009647846,
+            0.008373730815947056,
+            0.02237025462090969,
+            -0.011200242675840855,
+            -0.02078317664563656,
+            0.00010415205179015175,
+            0.017216026782989502,
+            -0.07104067504405975,
+            -0.010784579440951347,
+            -0.01410232950001955,
+            -0.024864235892891884,
+            -0.024380555376410484,
+            -0.008948101662099361,
+            0.013150081969797611,
+            0.01469937339425087,
+            0.02447124570608139,
+            0.0008743104408495128,
+            -0.02051110565662384,
+            -0.0035520337987691164,
+            -0.01797178015112877,
+            0.02985219843685627,
+            -0.04546603187918663,
+            -0.022294679656624794,
+            0.026405969634652138,
+            0.01073167659342289,
+            -0.0017287824302911758,
+            0.009650951251387596,
+            -0.039268866181373596,
+            0.037001609802246094,
+            -0.018304310739040375,
+            -0.001512448419816792,
+            0.009356208145618439,
+            0.009227730333805084,
+            -0.017548557370901108,
+            0.02332250215113163,
+            0.007973182015120983,
+            -0.021750537678599358,
+            0.03146950900554657,
+            0.022143529728055,
+            0.014631356112658978,
+            0.011661251075565815,
+            -0.03295078128576279,
+            0.015764983370900154,
+            0.013429710641503334,
+            -0.00931842066347599,
+            0.05486758425831795,
+            0.018833335489034653,
+            -0.004228431731462479,
+            0.022899281233549118,
+            -0.023216696456074715,
+            0.011381623335182667,
+            0.01455578114837408,
+            -0.0037031841930001974,
+            0.029020871967077255,
+            0.02543860860168934,
+            0.022521406412124634,
+            -0.04894249141216278,
+            0.01937747746706009,
+            0.03219502791762352,
+            -0.00419064424932003,
+            -0.0698314756155014,
+            0.025952519848942757,
+            0.0018770986935123801,
+            0.01748809777200222,
+            0.017352063208818436,
+            0.02144823782145977,
+            0.01674746163189411,
+            0.002259697997942567,
+            0.043924298137426376,
+            -0.039057254791259766,
+            0.019044946879148483,
+            0.0010344353504478931,
+            -0.026859421283006668,
+            -0.014638913795351982,
+            -0.007379916962236166,
+            -0.01726137287914753,
+            -0.010580525733530521,
+            0.026723386719822884,
+            0.02049599029123783,
+            0.033676303923130035,
+            -0.01191064901649952,
+            0.028809260576963425,
+            -0.0014926099684089422,
+            0.00100326060783118,
+            -0.0124850207939744,
+            -0.00945445615798235,
+            -0.017442753538489342,
+            -0.016732346266508102,
+            0.003291299333795905,
+            0.0034613434690982103,
+            -0.03875495493412018,
+            0.01202401239424944,
+            -0.007024713326245546,
+            -0.01751832850277424,
+            0.023458538576960564,
+            -0.033887915313243866,
+            0.0400850772857666,
+            -0.015780098736286163,
+            -0.05078652501106262,
+            0.006786651909351349,
+            0.01837988570332527,
+            -0.030381225049495697,
+            0.00044494890607893467,
+            0.004428706131875515,
+            0.030668411403894424,
+            -0.0060875811614096165,
+            -0.01933213323354721,
+            0.001051439787261188,
+            0.002350388327613473,
+            0.024622395634651184,
+            -0.0022294679656624794,
+            -0.016898611560463905,
+            -0.006306749302893877,
+            0.0045609623193740845,
+            0.020888982340693474,
+            0.0007798414444550872,
+            -0.025015385821461678,
+            0.02170519344508648,
+            0.025483952835202217,
+            -0.014412187971174717,
+            -0.05244917795062065,
+            0.03824104368686676,
+            0.033706534653902054,
+            -0.007478164508938789,
+            -0.04226164519786835,
+            0.016006823629140854,
+            -0.029081331565976143,
+            0.01798689365386963,
+            0.00021491693041753024,
+            -0.00454206857830286,
+            -0.016248665750026703,
+            0.019014716148376465,
+            0.021599387750029564,
+            -0.013271003030240536,
+            -0.005369617138057947,
+            -0.042291875928640366,
+            -0.036487698554992676,
+            0.017140451818704605,
+            -0.01304427720606327,
+            0.00782203208655119,
+            -0.012326313182711601,
+            -0.02421429008245468,
+            0.009605606086552143,
+            -0.05263055860996246,
+            -0.04640316590666771,
+            0.0015011121286079288,
+            0.024894466623663902,
+            -0.0001595344947418198,
+            0.0009215449099428952,
+            -0.010172420181334019,
+            0.01961931772530079,
+            0.020375071093440056,
+            0.024894466623663902,
+            -0.0358830988407135,
+            -0.002492091851308942,
+            0.028113968670368195,
+            0.0014963886933401227,
+            -0.018153158947825432,
+            0.014517993666231632,
+            -0.025499068200588226,
+            -0.006265182979404926,
+            0.0008100715349428356,
+            -0.0326484814286232,
+            0.004504281096160412,
+            -0.005407404620200396,
+            -0.0013424042845144868,
+            0.010247995145618916,
+            0.0316811166703701,
+            -0.021145937964320183,
+            0.004625201225280762,
+            0.040447838604450226,
+            -0.0043040066957473755,
+            -0.013134967535734177,
+            0.008260367438197136,
+            0.017911318689584732,
+            -0.017125336453318596,
+            -0.022083070129156113,
+            0.02329227328300476,
+            -0.02379106916487217,
+            0.014805179089307785,
+            0.045828793197870255,
+            -0.012696631252765656,
+            0.006484351120889187,
+            0.042291875928640366,
+            0.02540837787091732,
+            0.013679108582437038,
+            -0.01423836499452591,
+            -0.008676031604409218,
+            0.012054242193698883,
+            -0.059099797159433365,
+            9.500037413090467e-05,
+            -0.005857076961547136,
+            0.022899281233549118,
+            -0.002246472518891096,
+            0.014087215065956116,
+            -0.021372662857174873,
+            0.007104067597538233,
+            0.03009403869509697,
+            -0.02333761751651764,
+            0.014381958171725273,
+            -0.057678982615470886,
+            0.016369584947824478,
+            0.012394330464303493,
+            0.012160047888755798,
+            -0.0045118387788534164,
+            -0.0124850207939744,
+            0.027207067236304283,
+            -0.007920279167592525,
+            0.009280633181333542,
+            0.01745786890387535,
+            -0.022324910387396812,
+            -0.053023550659418106,
+            0.017805512994527817,
+            -0.010323570109903812,
+            -0.030653296038508415,
+            0.007485722191631794,
+            -0.02167496271431446,
+            -0.021312203258275986,
+            -0.044438209384679794,
+            0.012046684511005878,
+            0.037666670978069305,
+            -0.016732346266508102,
+            -0.04274532571434975,
+            -0.0004893493023701012,
+            -0.022249335423111916,
+            -0.013822701759636402,
+            0.010746791027486324,
+            -0.03225548937916756,
+            0.020178575068712234,
+            0.018138045445084572,
+            0.048882029950618744,
+            -0.010316013358533382,
+            -0.004345573019236326,
+            0.02823488973081112,
+            -0.029217366129159927,
+            0.005097546149045229,
+            -0.02471308596432209,
+            -0.03446228429675102,
+            0.02888483554124832,
+            0.0050597586669027805,
+            0.029020871967077255,
+            0.015235957689583302,
+            -0.007988297380506992,
+            0.016974186524748802,
+            -0.021281972527503967,
+            -0.027690747752785683,
+            -0.018062468618154526,
+            -0.026980342343449593,
+            -0.04504281282424927,
+            0.028355810791254044,
+            -0.05734645202755928,
+            -0.019966963678598404,
+            0.028612766414880753,
+            0.009552703239023685,
+            -0.0021274415776133537,
+            0.005853298120200634,
+            -0.019453052431344986,
+            -0.036517929285764694,
+            -0.011374065652489662,
+            0.028794145211577415,
+            0.009598048403859138,
+            -0.010187535546720028,
+            0.005762607790529728,
+            0.009598048403859138,
+            -0.04407544806599617,
+            -0.016898611560463905,
+            -0.00010893454600591213,
+            0.013694223947823048,
+            -0.02170519344508648,
+            -0.04425682872533798,
+            -0.04211049526929855,
+            0.009386437945067883,
+            -0.02729775756597519,
+            0.007912722416222095,
+            -0.030411455780267715,
+            -0.04845881089568138,
+            -0.004073502495884895,
+            -0.0071909790858626366,
+            0.02382129803299904,
+            0.02584671415388584,
+            -2.5447581720072776e-05,
+            0.03201364725828171,
+            -0.032527558505535126,
+            0.011389181017875671,
+            0.006571262609213591,
+            -0.021538928151130676,
+            -0.009008562192320824,
+            -0.0022294679656624794,
+            -0.029897544533014297,
+            0.029292942956089973,
+            0.01913563720881939,
+            0.00495017459616065,
+            -0.0025941182393580675,
+            -0.027146607637405396,
+            -0.02149358205497265,
+            -0.05789059400558472,
+            0.014752276241779327,
+            0.05380953103303909,
+            -0.030502144247293472,
+            -0.01051250845193863,
+            -0.017835743725299835,
+            0.019740238785743713,
+            0.04827743023633957,
+            0.007950509898364544,
+            0.03149973601102829,
+            -0.023050431162118912,
+            0.009514915756881237,
+            0.006975589785724878,
+            0.011018862016499043,
+            -0.044861432164907455,
+            -0.0014614351093769073,
+            -0.013376807793974876,
+            -0.0042586615309119225,
+            -0.003671064740046859,
+            -0.015598718076944351,
+            -0.025000272318720818,
+            -0.0045005022548139095,
+            0.006911350879818201,
+            -0.0050710951909422874,
+            0.002527989912778139,
+            -0.018017124384641647,
+            0.03479481488466263,
+            -0.04017576947808266,
+            0.03827127441763878,
+            -0.006427669432014227,
+            0.004568520002067089,
+            0.04981916397809982,
+            -0.018077583983540535,
+            -0.007648208644241095,
+            -0.008033642545342445,
+            -0.0038524451665580273,
+            -0.01141185313463211,
+            0.010096845217049122,
+            -0.014608683064579964,
+            0.0017864084802567959,
+            0.020465759560465813,
+            -0.02584671415388584,
+            -0.020344840362668037,
+            -0.005841962061822414,
+            0.0031590426806360483,
+            -0.0405687615275383,
+            0.006170713808387518,
+            0.0012951696990057826,
+            0.030169613659381866,
+            0.014155232347548008,
+            -0.03128812834620476,
+            0.000974919879809022,
+            0.01565917767584324,
+            -0.012492578476667404,
+            -0.02330738678574562,
+            0.03225548937916756,
+            0.003327197628095746,
+            -0.004972847178578377,
+            0.003485905472189188,
+            0.01769970916211605,
+            -0.013732011429965496,
+            0.01863684132695198,
+            -0.006522138603031635,
+            0.012265852652490139,
+            0.017397407442331314,
+            -0.03618539869785309,
+            0.016173088923096657,
+            0.007950509898364544,
+            -0.019241442903876305,
+            0.008139447309076786,
+            0.019044946879148483,
+            -0.025499068200588226,
+            0.021145937964320183,
+            -0.02097967080771923,
+            0.016626540571451187,
+            -0.04537534341216087,
+            -0.008192350156605244,
+            0.02896041050553322,
+            0.009802102111279964,
+            0.0013943621888756752,
+            -0.022234220057725906,
+            -0.0013839706080034375,
+            0.04809604957699776,
+            0.005229802802205086,
+            -0.011525215581059456,
+            -0.011850189417600632,
+            0.00513911247253418,
+            0.00043975311564281583,
+            0.0009342982666566968,
+            -0.02753959782421589,
+            -0.04588925465941429,
+            -0.006393660791218281,
+            -0.036941152065992355,
+            0.0326484814286232,
+            0.027343101799488068,
+            -0.0037126310635358095,
+            -0.016868380829691887,
+            0.0189089123159647,
+            -0.0006636445759795606,
+            0.03282986208796501,
+            -0.007478164508938789,
+            0.04150589182972908,
+            -0.052842170000076294,
+            0.0018478132551535964,
+            -0.011517658829689026,
+            -0.02193191833794117,
+            -0.007406368386000395,
+            -0.014638913795351982,
+            -0.018002009019255638,
+            0.01892402581870556,
+            0.01083748135715723,
+            0.0031533746514469385,
+            0.016974186524748802,
+            -0.0028246226720511913,
+            0.0035274717956781387,
+            0.012409445829689503,
+            -0.012084471993148327,
+            -0.003589821280911565,
+            0.04458935931324959,
+            0.009530031122267246,
+            -0.00979454442858696,
+            -0.017321832478046417,
+            -0.0024864235892891884,
+            0.052116647362709045,
+            -0.008494650945067406,
+            0.021569158881902695,
+            0.03606447950005531,
+            0.007958066649734974,
+            0.03074398636817932,
+            -0.004685661755502224,
+            0.026678040623664856,
+            0.003858113195747137,
+            -0.019180981442332268,
+            -0.006310528144240379,
+            -0.020405299961566925,
+            -0.01340703759342432,
+            -0.005849519278854132,
+            -0.023186467587947845,
+            0.026058323681354523,
+            -0.006525917444378138,
+            -0.00215578218922019,
+            0.0006490019150078297,
+            -0.030623065307736397,
+            0.021418007090687752,
+            0.018334539607167244,
+            0.014722046442329884,
+            0.04099198058247566,
+            0.007164527662098408,
+            0.02403290942311287,
+            -0.002614901401102543,
+            -0.004761236719787121,
+            0.014004082418978214,
+            0.024818891659379005,
+            0.024380555376410484,
+            0.03606447950005531,
+            -0.005350723396986723,
+            0.028567420318722725,
+            0.032527558505535126,
+            -0.027464022859930992,
+            0.014419745653867722,
+            0.016883496195077896,
+            -0.006008227355778217,
+            -0.021614503115415573,
+            -0.0022559193894267082,
+            0.02073783054947853,
+            0.002458082977682352,
+            -0.00838128849864006,
+            0.0363667793571949,
+            -0.02498515695333481,
+            -0.015946364030241966,
+            0.0031514852307736874,
+            -0.03267871215939522,
+            0.02495492622256279,
+            0.0028246226720511913,
+            -0.00792783685028553,
+            -0.02608855441212654,
+            -0.0076595451682806015,
+            0.0029757730662822723,
+            -0.011381623335182667,
+            -0.0038543345872312784,
+            0.04189888387918472,
+            -0.006654395256191492,
+            0.01318031270056963,
+            -0.0022389148361980915,
+            0.004957732278853655,
+            -0.0058306255377829075,
+            0.028854606673121452,
+            0.0006863171583972871,
+            0.0049199447967112064,
+            -0.022098183631896973,
+            -0.007580191362649202,
+            0.0049652899615466595,
+            0.015538258478045464,
+            0.027826784178614616,
+            0.019649548456072807,
+            0.010127075016498566,
+            0.027010571211576462,
+            -0.02172030881047249,
+            0.00020688706717919558,
+            -0.017397407442331314,
+            0.050726063549518585,
+            0.036034248769283295,
+            -0.023005086928606033,
+            -0.012787321582436562,
+            0.010323570109903812,
+            -0.03594356030225754,
+            0.012137374840676785,
+            0.02445613034069538,
+            -0.0005446136929094791,
+            -0.0065863775089383125,
+            0.0048632631078362465,
+            -0.008101659826934338,
+            -0.026028094813227654,
+            -0.00919750053435564,
+            0.02654200606048107,
+            0.0002616790879983455,
+            0.0062425103969872,
+            0.011827516369521618,
+            0.016173088923096657,
+            -0.017442753538489342,
+            -0.029338287189602852,
+            0.000420150812715292,
+            0.037424832582473755,
+            0.01109443698078394,
+            0.004372024443000555,
+            0.019286787137389183,
+            -0.009280633181333542,
+            -0.014140116982161999,
+            0.02474331669509411,
+            0.019906504079699516,
+            -0.0013216211227700114,
+            0.004938838537782431,
+            -0.007973182015120983,
+            0.00968118105083704,
+            -0.059553246945142746,
+            0.03258801996707916,
+            0.008570225909352303,
+            0.017079992219805717,
+            -0.0009522473555989563,
+            -0.007988297380506992,
+            0.028159314766526222,
+            -0.0003171795979142189,
+            0.008056314662098885,
+            0.016943957656621933,
+            -0.030895136296749115,
+            0.008366173133254051,
+            -0.018531035631895065,
+            -0.0010495503665879369,
+            0.019286787137389183,
+            0.01013463269919157,
+            -0.013021604157984257,
+            -0.012636170722544193,
+            -0.009371323511004448,
+            0.023443423211574554,
+            -0.007153191603720188,
+            0.015077250078320503,
+            0.0004227487079333514,
+            -0.009734083898365498,
+            -0.02241560071706772,
+            0.006733749061822891,
+            0.026647809892892838,
+            -0.025090962648391724,
+            0.01700441725552082,
+            -0.01529641728848219,
+            0.01435928512364626,
+            -0.029247596859931946,
+            0.005486758425831795,
+            0.04507303982973099,
+            0.011812401935458183,
+            -0.040659449994564056,
+            -0.010890384204685688,
+            -0.0003204860258847475,
+            -0.01154788862913847,
+            -0.007617978844791651,
+            0.012318755500018597,
+            0.05692322924733162,
+            -0.03051725961267948,
+            0.008532438427209854,
+            0.028975525870919228,
+            -0.014034312218427658,
+            0.016671886667609215,
+            -0.019966963678598404,
+            0.033222850412130356,
+            0.006850890815258026,
+            0.008539996109902859,
+            0.001844034530222416,
+            -0.0016796585405245423,
+            -0.0096585089340806,
+            0.0041037327609956264,
+            -0.018954256549477577,
+            -0.017064876854419708,
+            0.010172420181334019,
+            0.013429710641503334,
+            0.006714855320751667,
+            -0.00838128849864006,
+            -0.027222182601690292,
+            -0.001923388452269137,
+            0.019710008054971695,
+            -0.016671886667609215,
+            -0.032497331500053406,
+            0.014404630288481712,
+            0.012855338864028454,
+            -0.015538258478045464,
+            0.0391177162528038,
+            0.0041490779258310795,
+            0.01037647295743227,
+            0.015167939476668835,
+            -0.006321864202618599,
+            -0.008736491203308105,
+            0.024441014975309372,
+            0.0170497614890337,
+            -0.004334236960858107,
+            -0.016278894618153572,
+            0.015455125831067562,
+            0.030623065307736397,
+            0.013331462629139423,
+            0.005947767291218042,
+            -0.012401888146996498,
+            0.0378178246319294,
+            -0.004390918184071779,
+            -0.011555446311831474,
+            -0.03171134740114212,
+            -0.014034312218427658,
+            -0.02049599029123783,
+            -0.0033555382397025824,
+            -0.0039412458427250385,
+            0.012273410335183144,
+            -0.024531705304980278,
+            0.004594971425831318,
+            0.008509766310453415,
+            0.00636343052610755,
+            0.08131889998912811,
+            0.01798689365386963,
+            0.007633093744516373,
+            0.023413192480802536,
+            -0.01154788862913847,
+            0.029474321752786636,
+            -0.02708614617586136,
+            0.017427638173103333,
+            -0.013278559781610966,
+            0.011109552346169949,
+            -0.005350723396986723,
+            0.02590717375278473,
+            0.02332250215113163,
+            -0.0007576412172056735,
+            0.002580892527475953,
+            0.013225657865405083,
+            -0.0034084408544003963,
+            -0.019740238785743713,
+            0.012076914310455322,
+            0.030880020931363106,
+            0.03654816001653671,
+            0.012961144559085369,
+            0.02169007807970047,
+            0.008441748097538948,
+            -0.01049739308655262,
+            -0.02026926539838314,
+            -0.006866005714982748,
+            0.005524545907974243,
+            0.021614503115415573,
+            0.008358615450561047,
+            0.009582933969795704,
+            0.00642011221498251,
+            -0.008751606568694115,
+            -0.006091360002756119,
+            0.00636343052610755,
+            -0.014147674664855003,
+            0.03225548937916756,
+            0.005977997090667486,
+            -0.018092699348926544,
+            -0.028809260576963425,
+            -0.009137040004134178,
+            -0.01605216972529888,
+            -0.033041469752788544,
+            -0.01015730481594801,
+            0.008547553792595863,
+            -0.03482504561543465,
+            -0.008471978828310966,
+            0.01051250845193863,
+            -0.018077583983540535,
+            -0.03775736317038536,
+            0.014744718559086323,
+            -0.028129084035754204,
+            0.023911988362669945,
+            -0.024622395634651184,
+            0.010089287534356117,
+            0.002883193315938115,
+            -0.006457899697124958,
+            -0.011563003063201904,
+            -0.0022823705803602934,
+            -0.002686697756871581,
+            -0.0020103000570088625,
+            -0.03140904754400253,
+            -0.014321497641503811,
+            -0.017367178574204445,
+            -0.002590339398011565,
+            -0.033706534653902054,
+            0.025090962648391724,
+            0.030411455780267715,
+            0.012658843770623207,
+            0.007323235739022493,
+            -0.008751606568694115,
+            -0.01841011457145214,
+            -0.007874934002757072,
+            -0.033011242747306824,
+            -0.0064805722795426846,
+            -0.05928117781877518,
+            -0.010905499570071697,
+            -0.008932987228035927,
+            0.01772993803024292,
+            -0.013202984817326069,
+            -0.010814809240400791,
+            -0.02100990153849125,
+            -0.04099198058247566,
+            -0.019014716148376465,
+            0.007202315144240856,
+            -0.03379722312092781,
+            0.05925094708800316,
+            0.013210542500019073,
+            0.006968032103031874,
+            -0.024577051401138306,
+            0.010051500052213669,
+            -0.000378112104954198,
+            0.0096585089340806,
+            -0.005902422126382589,
+            -0.010913057252764702,
+            0.013339020311832428,
+            0.01698930189013481,
+            -0.008517323061823845,
+            0.0012687183916568756,
+            0.016173088923096657,
+            0.01682303659617901,
+            0.02614901401102543,
+            0.022249335423111916,
+            0.019014716148376465,
+            0.032769400626420975,
+            -0.029232481494545937,
+            0.06293901801109314,
+            -0.011479870416224003,
+            0.005369617138057947,
+            0.016928842291235924,
+            -0.019211212173104286,
+            0.029791738837957382,
+            -0.015251072123646736,
+            -0.02143312245607376,
+            0.00335175939835608,
+            0.005607679020613432,
+            0.013943621888756752,
+            -0.05217710882425308,
+            0.033434461802244186,
+            0.03963162750005722,
+            0.017352063208818436,
+            -0.00019826677453238517,
+            0.009174827486276627,
+            -0.0190751776099205,
+            0.038120124489068985,
+            -0.007342129480093718,
+            0.019997194409370422,
+            0.03727368265390396,
+            -0.0008039310341700912,
+            0.029292942956089973,
+            -0.012326313182711601,
+            0.02894529700279236,
+            -0.006605271250009537,
+            0.012666401453316212,
+            0.01198622491210699,
+            -0.009325978346168995,
+            -0.023262042552232742,
+            -0.015326648019254208,
+            0.0017731827683746815,
+            0.0012384883593767881,
+            -0.02843138575553894,
+            0.030245188623666763,
+            -0.019241442903876305,
+            -0.0007533901371061802,
+            0.006095138844102621,
+            -0.017412522807717323,
+            0.000406925129937008,
+            -0.0006593934958800673,
+            0.009507358074188232,
+            -0.005441413260996342,
+            0.010686331428587437,
+            0.01584055833518505,
+            -0.011842631734907627,
+            -0.02240048535168171,
+            0.01701953262090683,
+            -0.010066614486277103,
+            0.006968032103031874,
+            -0.03310193121433258,
+            -0.003370653372257948,
+            -0.03775736317038536,
+            -0.02847672998905182,
+            0.03948047757148743,
+            -0.0163242407143116,
+            -0.02265744097530842,
+            0.021599387750029564,
+            -0.03524826839566231,
+            -0.010111959651112556,
+            0.021508697420358658,
+            -0.010988632217049599,
+            -0.011736826039850712,
+            0.027554713189601898,
+            0.002395733492448926,
+            -0.009401553310453892,
+            -0.04990985244512558,
+            -0.026012979447841644,
+            0.004870820790529251,
+            0.011215358041226864,
+            -0.013308790512382984,
+            0.04546603187918663,
+            -0.01479762140661478,
+            -0.042352333664894104,
+            -0.015228400006890297,
+            0.01529641728848219,
+            -0.0099003491923213,
+            -0.019468167796730995,
+            -0.014230807311832905,
+            0.009726526215672493,
+            0.0024070697836577892,
+            -0.03149973601102829,
+            -0.01108688022941351,
+            0.02169007807970047,
+            -0.009847446344792843,
+            0.011744383722543716,
+            -0.03585286810994148,
+            -0.021856343373656273,
+            -0.01844034530222416,
+            0.020677370950579643,
+            0.029202252626419067,
+            -0.00115252158138901,
+            0.010368915274739265,
+            0.011895534582436085,
+            0.024652626365423203,
+            0.010452047921717167,
+            -0.0195588581264019,
+            0.015183054842054844,
+            -0.016037054359912872,
+            0.04032691940665245,
+            -0.0027830563485622406,
+            0.004417369607836008,
+            -0.03355538100004196,
+            0.0029323173221200705,
+            0.009303305298089981,
+            0.020223919302225113,
+            -0.009091694839298725,
+            0.03821081295609474,
+            -0.024335209280252457,
+            0.002350388327613473,
+            -0.009514915756881237,
+            0.01352040097117424,
+            0.03727368265390396,
+            0.002677250886335969,
+            -0.005146670155227184,
+            -0.007164527662098408,
+            0.019483283162117004,
+            0.024818891659379005,
+            -0.03760621324181557,
+            -0.013694223947823048,
+            0.039027027785778046,
+            0.00967362429946661,
+            -0.00875916425138712,
+            0.00459875026717782,
+            0.014737160876393318,
+            -0.023050431162118912,
+            0.017412522807717323,
+            0.015364435501396656,
+            0.009023677557706833,
+            -0.01889379695057869,
+            0.017352063208818436,
+            -0.016868380829691887,
+            -0.011419410817325115,
+            0.02753959782421589,
+            0.026285050436854362,
+            -0.025483952835202217,
+            0.016913726925849915,
+            -0.02401779405772686,
+            0.028068624436855316,
+            0.002153892768546939,
+            0.002055644989013672,
+            0.00874404888600111,
+            -0.004096175078302622,
+            -0.02867322601377964,
+            0.0012375436490401626,
+            0.006571262609213591,
+            0.006616607774049044,
+            0.0036238301545381546,
+            0.010028827004134655,
+            0.014926099218428135,
+            -0.008449305780231953,
+            0.008434190414845943,
+            0.00838884525001049,
+            0.02917202189564705,
+            -0.009356208145618439,
+            -0.027373332530260086,
+            -0.006102696061134338,
+            -0.0029512110631912947,
+            0.028506960719823837,
+            0.01292335707694292,
+            -0.00814700499176979,
+            0.02167496271431446,
+            0.09613163769245148,
+            0.006185829173773527,
+            -0.020163459703326225,
+            -0.02353411354124546,
+            -0.013437268324196339,
+            0.005891085602343082,
+            -0.006684625055640936,
+            0.009371323511004448,
+            0.004704555496573448,
+            -0.013815144076943398,
+            -0.009862561710178852,
+            0.03748529404401779,
+            -0.002057534409686923,
+            -0.008298155851662159,
+            -0.012885569594800472,
+            0.01841011457145214,
+            0.00780691672116518,
+            -0.008993446826934814,
+            0.004145299084484577,
+            -0.02519676648080349,
+            -0.03171134740114212,
+            -0.010656101629137993,
+            -0.005123997572809458,
+            0.02329227328300476,
+            -0.006423891056329012,
+            0.019014716148376465,
+            0.0054225195199251175,
+            0.012099587358534336,
+            -0.009991039521992207,
+            -0.0012252627639099956,
+            -0.018939141184091568,
+            0.019060062244534492,
+            -0.009371323511004448,
+            -0.0009862561710178852,
+            -0.01176705677062273,
+            0.003833551425486803,
+            -0.019634433090686798,
+            -0.0015766873257234693,
+            -0.029066216200590134,
+            0.016475390642881393,
+            -0.0054225195199251175,
+            0.0069491383619606495,
+            0.011842631734907627,
+            -0.012401888146996498,
+            0.0007992075989022851,
+            0.006166934967041016,
+            -0.0027981712482869625,
+            -0.03703184053301811,
+            0.011797286570072174,
+            0.007274111732840538,
+            0.02147846855223179,
+            -0.020405299961566925,
+            -0.034371595829725266,
+            -0.012076914310455322,
+            -0.0017467314610257745,
+            0.013369250111281872,
+            0.014109887182712555,
+            0.01059564109891653,
+            -0.025952519848942757,
+            -0.008426633663475513,
+            0.018153158947825432,
+            0.012144932523369789,
+            0.023141121491789818,
+            -0.006964253261685371,
+            0.041626811027526855,
+            -0.001176138874143362,
+            0.024335209280252457,
+            -0.006695961579680443,
+            -0.0018657624023035169,
+            -0.01934724859893322,
+            0.00885741226375103,
+            0.031348586082458496,
+            0.010860154405236244,
+            -0.015303974971175194,
+            -0.022944627329707146,
+            0.00953758880496025,
+            0.011245587840676308,
+            -0.006231173872947693,
+            0.009038791991770267,
+            -0.024365440011024475,
+            -0.00780691672116518,
+            0.0011780281784012914,
+            0.0012687183916568756,
+            0.021085476502776146,
+            -0.033918142318725586,
+            -0.01984604448080063,
+            -0.008373730815947056,
+            -0.014842966571450233,
+            -0.005003077443689108,
+            0.002580892527475953,
+            0.004020600114017725,
+            0.01200889702886343,
+            -0.012568153440952301,
+            -0.002246472518891096,
+            -0.024773545563220978,
+            0.00021161051699891686,
+            0.019271671772003174,
+            -0.007769129239022732,
+            -0.023201582953333855,
+            0.012658843770623207,
+            -0.016838151961565018,
+            0.010300897993147373,
+            0.026209475472569466,
+            -0.010399146005511284,
+            0.014412187971174717,
+            -0.015961479395627975,
+            -0.01936236210167408,
+            0.01328611746430397,
+            -0.005709705408662558,
+            -0.002926649060100317,
+            0.009975924156606197,
+            0.005796616896986961,
+            -0.00010568717698333785,
+            0.02982196770608425,
+            0.034432053565979004,
+            -0.023262042552232742,
+            0.013233215548098087,
+            -0.008124332875013351,
+            0.006427669432014227,
+            -0.0009295747731812298,
+            -0.02212841436266899,
+            0.014344170689582825,
+            0.014132560230791569,
+            -0.0022237999364733696,
+            -0.002837848151102662,
+            0.023624803870916367,
+            -0.0008270759717561305,
+            -0.01027822494506836,
+            0.031590428203344345,
+            0.002308822004124522,
+            0.0005960993003100157,
+            0.009016119875013828,
+            0.021508697420358658,
+            0.04171750321984291,
+            0.012658843770623207,
+            -0.00953758880496025,
+            0.0013452382991090417,
+            -0.005294041708111763,
+            0.009325978346168995,
+            -0.012069357559084892,
+            0.02072271518409252,
+            0.00896321702748537,
+            0.0244863610714674,
+            -0.003982812166213989,
+            -0.004791466984897852,
+            -0.02001230977475643,
+            -0.01537954993546009,
+            0.00814700499176979,
+            0.033192623406648636,
+            -0.000691512948833406,
+            -0.015674293041229248,
+            -0.0030362331308424473,
+            0.033041469752788544,
+            -0.0033668745309114456,
+            -0.030955595895648003,
+            -0.03122766688466072,
+            -0.00484059052541852,
+            -0.0019167757127434015,
+            0.012915799394249916,
+            -0.0017618464771658182,
+            0.007172085344791412,
+            0.015349320136010647,
+            -0.04576833173632622,
+            -0.046977534890174866,
+            0.015901019796729088,
+            -0.010777021758258343,
+            0.007247660309076309,
+            0.019407708197832108,
+            -0.033948373049497604,
+            -0.008789394050836563,
+            0.0060271210968494415,
+            0.012787321582436562,
+            0.008177234791219234,
+            0.006310528144240379,
+            0.015183054842054844,
+            -0.027872128412127495,
+            -0.019286787137389183,
+            0.020103000104427338,
+            0.0011317384196445346,
+            -0.004341794643551111,
+            0.0024788661394268274,
+            0.01132872048765421,
+            -0.009152155369520187,
+            0.01352040097117424,
+            0.0006357762613333762,
+            0.019301902502775192,
+            -0.011215358041226864,
+            -0.006333200726658106,
+            -0.010920614004135132,
+            0.0016258112154901028,
+            0.014147674664855003,
+            0.01724625751376152,
+            0.0027887243777513504,
+            0.006831996608525515,
+            0.001246045925654471,
+            0.00991546455770731,
+            -0.015016789548099041,
+            -0.0006310528260655701,
+            0.01071656122803688,
+            0.014019196853041649,
+            0.0035161355044692755,
+            0.009718968532979488,
+            0.006605271250009537,
+            0.0025393262039870024,
+            0.0024335209745913744,
+            -0.025967633351683617,
+            -0.009605606086552143,
+            -0.013905834406614304,
+            0.011804844252765179,
+            -0.004402254708111286,
+            0.0057852803729474545,
+            -0.010293340310454369,
+            -0.012968702241778374,
+            0.020118115469813347,
+            -0.0048292544670403,
+            -0.010520066134631634,
+            -0.026466431096196175,
+            -0.039571166038513184,
+            0.027630288153886795,
+            -0.009492243640124798,
+            0.0037579762283712626,
+            -0.01039158832281828,
+            0.022793475538492203,
+            0.0052713691256940365,
+            0.007814474403858185,
+            0.006166934967041016,
+            0.0067375279031693935,
+            -0.0031873835250735283,
+            -0.02097967080771923,
+            -0.0024316315539181232,
+            -0.01703464612364769,
+            0.017533443868160248,
+            -0.010210207663476467,
+            0.021735424175858498,
+            0.015719639137387276,
+            0.00023640862491447479,
+            0.006268961820751429,
+            0.02613389864563942,
+            0.012757091782987118,
+            0.0160824004560709,
+            -0.025816483423113823,
+            -0.0037749805487692356,
+            -0.01035380084067583,
+            0.024652626365423203,
+            -0.005505652166903019,
+            -0.00023735332069918513,
+            0.014865638688206673,
+            -0.022264450788497925,
+            -0.009870119392871857,
+            0.012878011912107468,
+            -0.006722413003444672,
+            -0.02234002575278282,
+            -0.001692884135991335,
+            -0.019664663821458817,
+            -0.007198536302894354,
+            -0.022143529728055,
+            0.005312935449182987,
+            0.018984487280249596,
+            -0.008721376769244671,
+            0.016399815678596497,
+            0.02915690653026104,
+            -0.006695961579680443,
+            0.024818891659379005,
+            0.0160824004560709,
+            0.0010372694814577699,
+            -0.00046148098772391677,
+            -0.005894864443689585,
+            0.004477829672396183,
+            -0.009333535097539425,
+            -0.012817551381886005,
+            -0.01748809777200222,
+            0.009061465039849281,
+            -0.018304310739040375,
+            0.015470240265130997,
+            0.03473435714840889,
+            -0.004020600114017725,
+            0.0042586615309119225,
+            -0.022959740832448006,
+            -0.016611425206065178,
+            -0.004447599872946739,
+            -0.014850524254143238,
+            0.06517603993415833,
+            -0.004795245360583067,
+            -0.014502878300845623,
+            -0.012666401453316212,
+            0.0036672858987003565,
+            -0.0028869721572846174,
+            -0.0036691753193736076,
+            -0.012182720005512238,
+            -0.015137709677219391,
+            -0.006964253261685371,
+            0.022445829585194588,
+            -0.032769400626420975,
+            0.005475422367453575,
+            0.030728871002793312,
+            -0.005641687661409378,
+            -0.0037579762283712626,
+            0.0023787289392203093,
+            -0.017397407442331314,
+            -0.012447233311831951,
+            -0.013565746136009693,
+            0.0008299099863506854,
+            0.0005710650002583861,
+            -0.0033215293660759926,
+            -0.020420415326952934,
+            0.01913563720881939,
+            -0.00850220862776041,
+            0.005309156607836485,
+            -0.029716163873672485,
+            -0.0055283247493207455,
+            0.012651286087930202,
+            0.0045005022548139095,
+            -0.019891388714313507,
+            0.013225657865405083,
+            -0.017790399491786957,
+            -0.0017023311229422688,
+            0.009356208145618439,
+            -0.01768459379673004,
+            -0.007391253020614386,
+            -0.024879351258277893,
+            -0.045798562467098236,
+            0.0036937373224645853,
+            -0.005743714049458504,
+            0.04011530801653862,
+            0.010452047921717167,
+            0.025816483423113823,
+            -0.011751941405236721,
+            -0.01798689365386963,
+            -0.008275482803583145,
+            -0.010897941887378693,
+            -0.015387107618153095,
+            -0.014193019829690456,
+            0.005683253984898329,
+            0.0029474322218447924,
+            -0.004885935690253973,
+            0.024380555376410484,
+            -0.006930244620889425,
+            -0.02188657410442829,
+            -0.003960139583796263,
+            -0.022037724032998085,
+            -0.01682303659617901,
+            0.006000669673085213,
+            -0.001827030093409121,
+            0.008630686439573765,
+            -0.005592563655227423,
+            -0.004398475866764784,
+            -0.016414931043982506,
+            0.00762931490316987,
+            0.020647140219807625,
+            0.010897941887378693,
+            -0.03691092133522034,
+            0.026511775329709053,
+            0.005849519278854132,
+            0.0007633093628101051,
+            0.017624134197831154,
+            0.004904829431325197,
+            0.007255217991769314,
+            0.01792643405497074,
+            -0.024773545563220978,
+            -0.004016821272671223,
+            -0.011880419217050076,
+            -0.025045616552233696,
+            -0.00943934079259634,
+            0.007799359038472176,
+            -0.020178575068712234,
+            -0.015689408406615257,
+            0.008366173133254051,
+            -0.02167496271431446,
+            0.01443486101925373,
+            -0.03503665700554848,
+            0.02026926539838314,
+            0.037878282368183136,
+            0.0072967843152582645,
+            0.0190751776099205,
+            0.002031083218753338,
+            0.00969629641622305,
+            -0.0014718268066644669,
+            -0.008328385651111603,
+            -0.02191680483520031,
+            0.01340703759342432,
+            -0.009257960133254528,
+            -0.003017339389771223,
+            -0.0030456800013780594,
+            0.00804120022803545,
+            -0.0009701964445412159,
+            -0.010127075016498566,
+            -0.0026111227925866842,
+            -0.015092364512383938,
+            0.008320827968418598,
+            -0.005660581402480602,
+            0.032315950840711594,
+            0.0008440803503617644,
+            0.018062468618154526,
+            0.0015332315815612674,
+            0.031620658934116364,
+            0.01863684132695198,
+            0.021841228008270264,
+            -0.0016891054110601544,
+            -0.008298155851662159,
+            0.00850220862776041,
+            -0.0019630654715001583,
+            -0.019906504079699516,
+            -0.01700441725552082,
+            -0.0028170649893581867,
+            0.0033196399454027414,
+            0.0006197165348567069,
+            0.007572633679956198,
+            0.006125368643552065,
+            -0.024093369022011757,
+            -0.01129093300551176,
+            0.003548254957422614,
+            0.0007727562915533781,
+            -0.005600121337920427,
+            0.009983481839299202,
+            -0.020647140219807625,
+            -0.0044060335494577885,
+            0.025317687541246414,
+            0.019755354151129723,
+            -0.025831598788499832,
+            -0.01549291331321001,
+            -0.005660581402480602,
+            0.0007500837091356516,
+            -0.018787991255521774,
+            -0.014389515854418278,
+            0.0037390822544693947,
+            -0.007504615932703018,
+            0.006779094226658344,
+            -0.01772993803024292,
+            -0.009514915756881237,
+            -0.0051202187314629555,
+            -0.030154500156641006,
+            0.012326313182711601,
+            0.013694223947823048,
+            0.030245188623666763,
+            -0.008676031604409218,
+            -0.01635446958243847,
+            -0.029323171824216843,
+            -0.006336979568004608,
+            -0.004765015561133623,
+            0.0033007462043315172,
+            0.004659210331737995,
+            -0.015115037560462952,
+            -0.025468837469816208,
+            -0.013376807793974876,
+            0.0063898819498717785,
+            -0.020571565255522728,
+            0.015251072123646736,
+            -0.008660916239023209,
+            0.03146950900554657,
+            -0.017805512994527817,
+            0.001601249212399125,
+            0.01222806517034769,
+            0.0005838182987645268,
+            0.007565075997263193,
+            0.0014954439830034971,
+            0.008071430027484894,
+            0.021130822598934174,
+            0.00826792512089014,
+            -0.01841011457145214,
+            0.007122961338609457,
+            0.006423891056329012,
+            -0.019422823563218117,
+            -0.0005328050465323031,
+            0.02052622102200985,
+            0.0030966931954026222,
+            0.00015670042193960398,
+            0.0021595610305666924,
+            0.004016821272671223,
+            0.01674746163189411,
+            -0.010436933487653732,
+            0.02495492622256279,
+            -0.00806387234479189,
+            -0.02732798643410206,
+            -0.01469181664288044,
+            -0.004077281337231398,
+            -0.03104628622531891,
+            -0.01108688022941351,
+            0.0009371322812512517,
+            -0.001392472768202424,
+            -0.006934023462235928,
+            0.0020443086978048086,
+            0.023866644129157066,
+            -0.003994148690253496,
+            -0.0021671184804290533,
+            0.01912052184343338,
+            0.09897326678037643,
+            0.002624348271638155,
+            -0.007746456656605005,
+            0.00437958212569356,
+            -0.011260703206062317,
+            -0.00647301459684968,
+            -0.018833335489034653,
+            -0.04924479126930237,
+            0.0016125855036079884,
+            1.3513492376659997e-05,
+            -0.004247325472533703,
+            -0.017125336453318596,
+            -0.013822701759636402,
+            0.013467498123645782,
+            -0.0060875811614096165,
+            0.01306694932281971,
+            -0.0052222455851733685,
+            0.0032686267513781786,
+            0.02329227328300476,
+            0.017881089821457863,
+            -0.006639279890805483,
+            0.009628279134631157,
+            -0.0029058658983558416,
+            0.008955659344792366,
+            0.018032239750027657,
+            0.014193019829690456,
+            -0.00769733265042305,
+            0.011661251075565815,
+            0.004946395754814148,
+            0.03981300815939903,
+            -0.007308120373636484,
+            -0.016928842291235924,
+            -0.00953758880496025,
+            -0.0009928690269589424,
+            -0.0053620594553649426,
+            0.011238030157983303,
+            -0.002766051795333624,
+            -0.016838151961565018,
+            -0.0017174461390823126,
+            -0.00780691672116518,
+            0.014306383207440376,
+            0.01117756962776184,
+            0.020163459703326225,
+            0.011457198299467564,
+            -0.002864299574866891,
+            0.028053509071469307,
+            -0.006129147484898567,
+            5.936785237281583e-05,
+            -0.013868046924471855,
+            0.004825475625693798,
+            -0.018017124384641647,
+            -0.002760383766144514,
+            0.005891085602343082,
+            -0.005078652407974005,
+            -0.0011563003063201904,
+            -0.0096585089340806,
+            0.005709705408662558,
+            -0.0027868349570780993,
+            0.010149747133255005,
+            -0.0036200513131916523,
+            -0.0018497026758268476,
+            0.009666066616773605,
+            0.00722120888531208,
+            -0.008056314662098885,
+            0.00922017265111208,
+            -0.021342432126402855,
+            -0.0007930670981295407,
+            -0.0074025895446538925,
+            0.003123144619166851,
+            -0.0019989637657999992,
+            -0.0030192285776138306,
+            -0.005569891072809696,
+            0.02031460963189602,
+            0.0017769616097211838,
+            -0.007980739697813988,
+            -0.022264450788497925,
+            0.022113298997282982,
+            -0.0031590426806360483,
+            0.014601126313209534,
+            -0.008827181532979012,
+            0.008925429545342922,
+            0.0035161355044692755,
+            -0.003438670886680484,
+            -0.002607343951240182,
+            -0.0018232513684779406,
+            -0.00361060444265604,
+            0.004307785537093878,
+            -0.001872375258244574,
+            0.0004959621583111584,
+            0.02311089262366295,
+            0.0009701964445412159,
+            -0.0027509366627782583,
+            0.012658843770623207,
+            0.006318085361272097,
+            0.0017004417022690177,
+            0.023881759494543076,
+            -0.008592898957431316,
+            -0.007727562915533781,
+            0.00012221925135236233,
+            -0.007258996833115816,
+            0.031590428203344345,
+            -0.009749199263751507,
+            0.00227103428915143,
+            0.020798292011022568,
+            -0.010285782627761364,
+            -0.003967697266489267,
+            0.003967697266489267,
+            -0.015553372912108898,
+            0.020344840362668037,
+            -0.023655032739043236,
+            0.022566750645637512,
+            0.012001339346170425,
+            -0.021372662857174873,
+            0.013913392089307308,
+            -0.00045439580571837723,
+            -0.013021604157984257,
+            0.006370988208800554,
+            0.007980739697813988,
+            -0.0420198030769825,
+            -0.008222579956054688,
+            -0.007882491685450077,
+            -0.03340423107147217,
+            0.009635835886001587,
+            -0.016430046409368515,
+            -0.0064956871792674065,
+            -0.0034084408544003963,
+            -0.006439005956053734,
+            -0.002450525527819991,
+            -0.02261209487915039,
+            -0.02327715791761875,
+            0.00642011221498251,
+            -0.02732798643410206,
+            -0.008184792473912239,
+            -0.0003235562762711197,
+            -0.002083985833451152,
+            -0.005668139085173607,
+            0.0052260239608585835,
+            0.016475390642881393,
+            0.0034821266308426857,
+            0.014049426652491093,
+            -0.005448970943689346,
+            0.011449640616774559,
+            -0.005626572761684656,
+            -0.01260594092309475,
+            -0.002288038842380047,
+            0.0027414897922426462,
+            -0.014011639170348644,
+            0.004251104313880205,
+            -0.008139447309076786,
+            -0.0060724662616848946,
+            0.012681515887379646,
+            -0.018939141184091568,
+            0.0018100256565958261,
+            0.008766721934080124,
+            7.917681796243414e-05,
+            -0.01258326880633831,
+            -0.028401155024766922,
+            0.00256955623626709,
+            -0.0054149623028934,
+            -0.002854852704331279,
+            0.00033772661117836833,
+            0.012417003512382507,
+            0.011540330946445465,
+            0.010701446793973446,
+            -0.002269144868478179,
+            0.007081395015120506,
+            0.028975525870919228,
+            0.013376807793974876,
+            -0.0074630496092140675,
+            0.0002914368233177811,
+            0.0022502511274069548,
+            0.0068886782974004745,
+            -0.020178575068712234,
+            -0.010323570109903812,
+            -0.0272372979670763,
+            -0.008766721934080124,
+            0.0014708820963278413,
+            0.0013594086049124599,
+            0.0244863610714674,
+            0.007493279874324799,
+            0.0018279748037457466,
+            -0.0028095075394958258,
+            0.010520066134631634,
+            0.007149412762373686,
+            0.0048028030432760715,
+            0.007009598426520824,
+            -0.018017124384641647,
+            -0.011593233793973923,
+            0.02146335318684578,
+            -0.0025884502101689577,
+            0.005543440114706755,
+            -0.0015200059860944748,
+            -0.0038203257136046886,
+            -0.004285112954676151,
+            0.01961931772530079,
+            -0.005671917926520109,
+            0.00012599800538737327,
+            0.029444092884659767,
+            -0.0035350292455404997,
+            0.0036238301545381546,
+            -0.006397439632564783,
+            -0.007224987726658583,
+            -0.04002461954951286,
+            0.01633935607969761,
+            -0.006983147002756596,
+            0.009431783109903336,
+            -0.010119517333805561,
+            0.009817216545343399,
+            0.010988632217049599,
+            0.0045609623193740845,
+            -0.010066614486277103,
+            0.0023995121009647846,
+            0.015047019347548485,
+            -0.011147339828312397,
+            -0.008313270285725594,
+            0.02100990153849125,
+            -0.006370988208800554,
+            -0.001745786750689149,
+            -0.0019177203066647053,
+            0.011948436498641968,
+            0.0027452686335891485,
+            0.018319424241781235,
+            0.005123997572809458,
+            -0.012076914310455322,
+            -0.0012970591196790338,
+            0.010308455675840378,
+            0.001875209272839129,
+            -0.006775315385311842,
+            0.005894864443689585,
+            -0.021554043516516685,
+            -0.0011270149843767285,
+            0.0018024682067334652,
+            0.023428307846188545,
+            0.005078652407974005,
+            0.026889652013778687,
+            -0.00035520337405614555,
+            0.0012167604872956872,
+            0.007727562915533781,
+            -0.007092731073498726,
+            -0.027146607637405396,
+            -0.01282510906457901,
+            -0.00828304048627615,
+            0.00018822944548446685,
+            0.0023749503307044506,
+            0.021735424175858498,
+            0.01841011457145214,
+            -0.02167496271431446,
+            0.007935394532978535,
+            -0.0047574578784406185,
+            -2.664320527401287e-05,
+            0.00888008438050747,
+            -0.00601200619712472,
+            0.003933688625693321,
+            0.00038921221857890487,
+            -0.0014944992726668715,
+            -0.007784244138747454,
+            -0.00886496901512146,
+            -0.0027112597599625587,
+            0.012084471993148327,
+            -0.003540697507560253,
+            0.008713819086551666,
+            -0.017654363065958023,
+            0.012386772781610489,
+            -0.008479535579681396,
+            0.03428090363740921,
+            0.010459605604410172,
+            -0.009061465039849281,
+            -0.013724453747272491,
+            -0.022083070129156113,
+            0.048851799219846725,
+            0.010346243157982826,
+            -0.0010835592402145267,
+            -0.00513911247253418,
+            -0.0077540138736367226,
+            -0.00036866520531475544,
+            -0.009582933969795704,
+            0.004315343219786882,
+            -0.004443821031600237,
+            0.006707297638058662,
+            0.0014160900609567761,
+            -0.0349157378077507,
+            0.0048179179430007935,
+            -0.014979002065956593,
+            -0.0012743865372613072,
+            -0.0008365228422917426,
+            -0.006892457138746977,
+            -0.00413396256044507,
+            -0.004625201225280762,
+            -0.008630686439573765,
+            0.01435928512364626,
+            0.0028850827366113663,
+            0.00026970895123668015,
+            -0.005585006438195705,
+            -0.013475055806338787,
+            0.008056314662098885,
+            -0.012386772781610489,
+            0.012462347745895386,
+            -0.006930244620889425,
+            -0.008668473921716213,
+            0.018304310739040375,
+            0.008252810686826706,
+            -0.0032421755604445934,
+            -0.02146335318684578,
+            -0.012333869934082031,
+            0.030895136296749115,
+            -0.010814809240400791,
+            0.005970439873635769,
+            -0.004451378248631954,
+            0.001610696199350059,
+            -0.006627943832427263,
+            0.008154562674462795,
+            0.014049426652491093,
+            0.01772993803024292,
+            0.017639247700572014,
+            -0.01771482452750206,
+            -0.0014689926756545901,
+            0.004799024201929569,
+            -0.02028438076376915,
+            -0.022959740832448006,
+            0.03682022914290428,
+            -0.002288038842380047,
+            0.014994116500020027,
+            0.0015162271447479725,
+            0.016369584947824478,
+            -0.011540330946445465,
+            0.017881089821457863,
+            0.004659210331737995,
+            -0.012280968017876148,
+            0.0019479504553601146,
+            -0.022007493302226067,
+            0.0396013967692852,
+            -0.010520066134631634,
+            -0.013210542500019073,
+            0.00734590832144022,
+            -0.02096455730497837,
+            -0.005093767773360014,
+            -0.024395670741796494,
+            -0.006080023944377899,
+            -0.00045415962813422084,
+            0.0004756513226311654,
+            0.007542403414845467,
+            -0.015523143112659454,
+            -0.028325580060482025,
+            -0.001077891094610095,
+            -0.011774614453315735,
+            -0.006148041225969791,
+            -0.01306694932281971,
+            -0.029761508107185364,
+            -0.011736826039850712,
+            0.039299096912145615,
+            -0.006979368627071381,
+            -0.0028964190278202295,
+            -0.0008029863820411265,
+            -0.0008398292702622712,
+            0.0038184362929314375,
+            0.019483283162117004,
+            -0.005887307226657867,
+            0.023609688505530357,
+            -0.005777723155915737,
+            0.010096845217049122,
+            0.004821696784347296,
+            -0.013822701759636402,
+            -0.013384365476667881,
+            0.014004082418978214,
+            0.003289409913122654,
+            -0.020390184596180916,
+            0.023670148104429245,
+            -0.008358615450561047,
+            0.01866707019507885,
+            -0.017216026782989502,
+            0.012099587358534336,
+            -0.02661758102476597,
+            -0.012757091782987118,
+            -0.014593568630516529,
+            -0.017805512994527817,
+            0.008819623850286007,
+            -0.011835074052214622,
+            0.01653585024178028,
+            -0.005743714049458504,
+            -0.0018837115494534373,
+            -0.0011723600327968597,
+            -0.037001609802246094,
+            0.0032591798808425665,
+            0.026693155989050865,
+            0.03675977140665054,
+            0.012311197817325592,
+            0.0030097817070782185,
+            -0.007950509898364544,
+            0.03654816001653671,
+            -0.010913057252764702,
+            -0.0122431805357337,
+            0.013187870383262634,
+            -0.006355873309075832,
+            -5.260741454549134e-05,
+            -0.01002126932144165,
+            0.02191680483520031,
+            0.013369250111281872,
+            0.04171750321984291,
+            -0.007319456897675991,
+            -0.000410467735491693,
+            -0.00897833239287138,
+            0.0007189089665189385,
+            0.007122961338609457,
+            0.0036616178695112467,
+            0.0017968000611290336,
+            -0.015704523772001266,
+            0.0067526428028941154,
+            -0.010867712087929249,
+            0.009046349674463272,
+            -0.0007529177819378674,
+            0.001029711915180087,
+            0.010043942369520664,
+            0.009348650462925434,
+            0.009076579473912716,
+            0.01958908885717392,
+            -0.012870454229414463,
+            -0.0059213158674538136,
+            -0.008895199745893478,
+            0.015024347230792046,
+            -0.0124850207939744,
+            0.0160824004560709,
+            0.006790430285036564,
+            -0.006091360002756119,
+            0.008411518298089504,
+            0.003138259518891573,
+            -0.0036559496074914932,
+            -0.0035180249251425266,
+            0.02422940544784069,
+            -0.007164527662098408,
+            -0.016399815678596497,
+            -0.00943934079259634,
+            0.018818221986293793,
+            -0.006114032585173845,
+            0.005214687902480364,
+            -0.005709705408662558,
+            -0.003502909792587161,
+            -0.013830259442329407,
+            -0.003361206501722336,
+            -0.010603198781609535,
+            0.0021274415776133537,
+            0.02028438076376915,
+            -0.0005294986767694354,
+            0.0034273345954716206,
+            0.0001297767594223842,
+            0.00804875697940588,
+            0.010913057252764702,
+            0.011472313664853573,
+            0.004387139342725277,
+            0.02333761751651764,
+            0.04126405343413353,
+            -0.00022826067288406193,
+            0.0005219411104917526,
+            -0.012537923641502857,
+            0.013928506523370743,
+            -0.0018374216742813587,
+            0.028340695425868034,
+            -0.01960420422255993,
+            0.0026848085690289736,
+            0.006125368643552065,
+            -0.01794154942035675,
+            -0.0003625247336458415,
+            0.001641870941966772,
+            0.00031694344943389297,
+            0.014147674664855003,
+            0.0051051038317382336,
+            -0.006627943832427263,
+            0.005199573002755642,
+            0.00804875697940588,
+            -0.0022237999364733696,
+            -0.015991710126399994,
+            0.012409445829689503,
+            0.0020461981184780598,
+            -0.0003955888678319752,
+            0.010127075016498566,
+            0.027887243777513504,
+            0.006185829173773527,
+            0.005566112697124481,
+            0.028053509071469307,
+            0.0027641623746603727,
+            0.010006154887378216,
+            0.00484059052541852,
+            0.007738898973912001,
+            0.007580191362649202,
+            -0.006321864202618599,
+            -0.014283710159361362,
+            -0.016974186524748802,
+            -0.01723114214837551,
+            -0.0030097817070782185,
+            -0.00016000683535821736,
+            0.03222525864839554,
+            0.001197866746224463,
+            0.01073167659342289,
+            0.01866707019507885,
+            -0.0017901872051879764,
+            -0.004885935690253973,
+            -0.025499068200588226,
+            -0.00489727221429348,
+            -0.0003348925383761525,
+            0.007073837332427502,
+            0.0005772055010311306,
+            0.004753679037094116,
+            -0.007799359038472176,
+            -0.00035071608726866543,
+            0.01655096560716629,
+            0.0014293156564235687,
+            0.013097180053591728,
+            -0.02351899817585945,
+            -0.023231811821460724,
+            0.010006154887378216,
+            0.020465759560465813,
+            0.005048422608524561,
+            -0.011661251075565815,
+            0.0046667675487697124,
+            0.01039158832281828,
+            0.0021425564773380756,
+            0.026753615587949753,
+            0.007323235739022493,
+            -0.018606610596179962,
+            0.008608013391494751,
+            0.010663658380508423,
+            0.01234898529946804,
+            -0.01176705677062273,
+            -0.0035010206047445536,
+            -0.007217430509626865,
+            -0.005690811667591333,
+            0.004107511602342129,
+            0.004353130701929331,
+            0.020934326574206352,
+            0.008509766310453415,
+            0.02008788473904133,
+            0.006151820067316294,
+            0.002475087298080325,
+            0.017382292076945305,
+            0.008441748097538948,
+            0.012069357559084892,
+            0.012757091782987118,
+            0.018712416291236877,
+            -0.02400267869234085,
+            0.017110222950577736,
+            -0.014963886700570583,
+            -0.007051164750009775,
+            0.0011336278403177857,
+            0.0048179179430007935,
+            0.004020600114017725,
+            -0.008237695321440697,
+            0.001068444224074483,
+            -0.010769464075565338,
+            0.00554721849039197,
+            -0.020919211208820343,
+            -0.00011064679711125791,
+            0.012160047888755798,
+            -0.003557701827958226,
+            -0.014767391607165337,
+            -0.010179977864027023,
+            0.02636062540113926,
+            0.011804844252765179,
+            0.012061799876391888,
+            -0.006544811185449362,
+            0.002210574224591255,
+            -0.015440010465681553,
+            0.013920948840677738,
+            0.018727531656622887,
+            -0.008872526697814465,
+            0.023246927186846733,
+            -0.014449975453317165,
+            0.006499466020613909,
+            -0.010149747133255005,
+            -0.006268961820751429,
+            -0.009212614968419075,
+            0.004292670637369156,
+            0.0041037327609956264,
+            0.016717230901122093,
+            0.008366173133254051,
+            -0.008940544910728931,
+            0.0086458008736372,
+            -0.0041982014663517475,
+            -0.008698703721165657,
+            -0.0172916017472744,
+            -0.014125002548098564,
+            -0.010754348710179329,
+            -0.0018138044979423285,
+            -0.0004062166262883693,
+            -0.022748131304979324,
+            0.00751973083242774,
+            0.0005649244994856417,
+            0.009114367887377739,
+            -0.001027822494506836,
+            -0.006873563397675753,
+            0.016974186524748802,
+            0.012076914310455322,
+            0.006896235514432192,
+            0.009552703239023685,
+            0.034643664956092834,
+            0.00945445615798235,
+            -0.007633093744516373,
+            0.013097180053591728,
+            -0.009446898475289345,
+            -0.027569828554987907,
+            0.0030910251662135124,
+            0.00241462723352015,
+            -0.000520996458362788,
+            0.0008346334798261523,
+            -0.005664360243827105,
+            0.0037220779340714216,
+            0.002367392648011446,
+            0.004062166437506676,
+            -0.0027981712482869625,
+            0.010671216063201427,
+            -0.026889652013778687,
+            0.014404630288481712,
+            0.014827851206064224,
+            0.011396737769246101,
+            -0.013482613489031792,
+            0.008260367438197136,
+            0.000955553783569485,
+            -0.017200913280248642,
+            -0.03243687003850937,
+            -0.01002126932144165,
+            0.01305939257144928,
+            0.011162455193698406,
+            0.004364467225968838,
+            0.0009560260805301368,
+            -0.0006178271723911166,
+            0.013792471028864384,
+            0.015213284641504288,
+            -0.004330458119511604,
+            -0.028310464695096016,
+            -0.023050431162118912,
+            0.032739169895648956,
+            0.007799359038472176,
+            0.006280297879129648,
+            0.019982079043984413,
+            0.006503244861960411,
+            0.0008549442864023149,
+            -0.02590717375278473,
+            0.014049426652491093,
+            -0.013505285605788231,
+            -0.004579856060445309,
+            0.006986925844103098,
+            -0.006499466020613909,
+            -0.016641655936837196,
+            0.013898276723921299,
+            0.001303671975620091,
+            -0.007243881467729807,
+            0.006677067838609219,
+            -0.008222579956054688,
+            -0.01001371257007122,
+            -0.01635446958243847,
+            -0.0008351058349944651,
+            -3.5425870009930804e-05,
+            -0.032074108719825745,
+            0.008721376769244671,
+            -0.012598383240401745,
+            0.007769129239022732,
+            0.014570895582437515,
+            0.004224652890115976,
+            -0.021750537678599358,
+            0.005562333855777979,
+            -0.012500136159360409,
+            -0.013497727923095226,
+            0.0069491383619606495,
+            -0.007334571797400713,
+            -0.00636343052610755,
+            -0.015024347230792046,
+            0.0051655638962984085,
+            0.006412554532289505,
+            0.019437937065958977,
+            -0.009552703239023685,
+            -0.0008813955937512219,
+            -0.014563338831067085,
+            0.014253480359911919,
+            -0.015455125831067562,
+            0.016278894618153572,
+            0.001953618600964546,
+            -0.004583634901791811,
+            -0.03237640857696533,
+            -0.003948803525418043,
+            0.0026281271129846573,
+            -0.006257625296711922,
+            -0.007723784074187279,
+            0.019241442903876305,
+            0.02005765400826931,
+            0.005101324990391731,
+            -0.01481273677200079,
+            -8.428404726146255e-06,
+            0.008192350156605244,
+            0.008018527179956436,
+            0.002970104804262519,
+            0.001875209272839129,
+            -0.016384700313210487,
+            -0.003960139583796263,
+            -0.022672556340694427,
+            -0.032557789236307144,
+            -0.006257625296711922,
+            0.0010920614004135132,
+            0.004084839019924402,
+            0.023216696456074715,
+            0.013905834406614304,
+            0.013527958653867245,
+            -0.006918908096849918,
+            0.021614503115415573,
+            0.009567818604409695,
+            -0.018092699348926544,
+            -0.009862561710178852,
+            -0.010043942369520664,
+            0.01629400998353958,
+            0.007213651668280363,
+            0.011147339828312397,
+            0.02823488973081112,
+            0.017352063208818436,
+            0.005558555014431477,
+            0.002766051795333624,
+            -0.007980739697813988,
+            -0.01364132110029459,
+            -0.01199378166347742,
+            -0.009257960133254528,
+            -0.0228690505027771,
+            0.009182385168969631,
+            -0.010520066134631634,
+            0.00665061641484499,
+            -0.008698703721165657,
+            -0.0022219105158001184,
+            0.0060271210968494415,
+            -0.01582544483244419,
+            0.0004508532292675227,
+            -0.01202401239424944,
+            -0.0008365228422917426,
+            0.00802608486264944,
+            0.010059057734906673,
+            -0.01889379695057869,
+            0.00792783685028553,
+            0.005150448996573687,
+            0.010920614004135132,
+            -0.02964058890938759,
+            -0.029323171824216843,
+            0.008320827968418598,
+            -0.01165369339287281,
+            -0.009824774228036404,
+            -0.026239704340696335,
+            -0.009250402450561523,
+            -0.013603533618152142,
+            -0.001517171855084598,
+            0.0019290565978735685,
+            0.021735424175858498,
+            -0.009159712120890617,
+            -0.03518780693411827,
+            -0.0252118818461895,
+            -0.015931248664855957,
+            -0.011215358041226864,
+            0.0019800697918981314,
+            -0.0023466094862669706,
+            -0.001889379695057869,
+            0.006896235514432192,
+            -0.02308066189289093,
+            -0.02049599029123783,
+            0.0006617552135139704,
+            0.010452047921717167,
+            -0.013339020311832428,
+            -0.010119517333805561,
+            0.026421085000038147,
+            -0.015553372912108898,
+            0.01001371257007122,
+            -0.013225657865405083,
+            -0.005925094708800316,
+            0.0010117627680301666,
+            -0.001555904163978994,
+            -0.004145299084484577,
+            0.00558878481388092,
+            -0.03403906524181366,
+            0.00513911247253418,
+            0.031832266598939896,
+            0.02382129803299904,
+            -0.004791466984897852,
+            -0.002518543042242527,
+            -0.03521803766489029,
+            -0.002465640427544713,
+            -0.012689073570072651,
+            0.02753959782421589,
+            -0.003945024684071541,
+            0.007474386133253574,
+            0.001541733741760254,
+            0.0036162727046757936,
+            -0.005600121337920427,
+            0.003094803774729371,
+            0.0005252475384622812,
+            -0.005857076961547136,
+            0.005671917926520109,
+            0.014963886700570583,
+            0.0009494132827967405,
+            0.012938471511006355,
+            -0.01635446958243847,
+            -0.008993446826934814,
+            -0.008872526697814465,
+            -0.024592164903879166,
+            -0.00558878481388092,
+            0.009839889593422413,
+            0.019226327538490295,
+            -0.01467670127749443,
+            -0.015946364030241966,
+            0.01527374517172575,
+            0.006624164991080761,
+            -0.02658735029399395,
+            -0.0007855095900595188,
+            -0.02403290942311287,
+            0.026436200365424156,
+            0.0163242407143116,
+            -0.0023371626157313585,
+            0.00244863610714674,
+            -0.006900014355778694,
+            -0.011691480875015259,
+            -0.013936064206063747,
+            0.00029309000819921494,
+            0.007867377251386642,
+            -0.032315950840711594,
+            0.0076142000034451485,
+            0.009159712120890617,
+            -0.0013546851696446538,
+            -0.02218887396156788,
+            0.017200913280248642,
+            0.00838884525001049,
+            0.0016569859581068158,
+            -0.019876273348927498,
+            0.008101659826934338,
+            0.0014264816418290138,
+            -0.004693218972533941,
+            -0.0049539534375071526,
+            -0.010406702756881714,
+            4.165491554886103e-05,
+            -0.023851528763771057,
+            -0.005974218249320984,
+            -0.007531067356467247,
+            -0.0029946668073534966,
+            0.017352063208818436,
+            -0.014510435983538628,
+            0.00020676899293903261,
+            0.009590490721166134,
+            0.018848450854420662,
+            0.010164862498641014,
+            -0.014593568630516529,
+            0.023216696456074715,
+            0.0019800697918981314,
+            -0.007364802062511444,
+            0.007047385908663273,
+            -0.005112661514431238,
+            -0.009303305298089981,
+            0.009431783109903336,
+            -0.0029927773866802454,
+            0.006563704926520586,
+            -0.008335943333804607,
+            -0.007281669415533543,
+            -0.013527958653867245,
+            -0.0013764130417257547,
+            0.014895869418978691,
+            0.003892122069373727,
+            -0.023896872997283936,
+            0.01528886053711176,
+            -0.002108547603711486,
+            -0.0024335209745913744,
+            -0.009998597204685211,
+            -0.019437937065958977,
+            -0.012976259924471378,
+            -0.009152155369520187,
+            -0.004776351619511843,
+            0.013210542500019073,
+            0.012968702241778374,
+            -0.011623463593423367,
+            -0.009567818604409695,
+            -0.004995519760996103,
+            -0.030487030744552612,
+            0.035792406648397446,
+            -0.019483283162117004,
+            0.02008788473904133,
+            -0.01910540647804737,
+            0.005192015320062637,
+            -0.001130793709307909,
+            0.0021614502184093,
+            -0.007799359038472176,
+            0.004995519760996103,
+            -0.019649548456072807,
+            -0.015961479395627975,
+            -0.006605271250009537,
+            -0.008033642545342445,
+            -0.021130822598934174,
+            -0.014608683064579964,
+            -0.007070058491080999,
+            0.012794879265129566,
+            -0.0011988113401457667,
+            -0.0014633245300501585,
+            0.020919211208820343,
+            0.007814474403858185,
+            0.011162455193698406,
+            0.026164129376411438,
+            0.022566750645637512,
+            0.03149973601102829,
+            -0.002148224739357829,
+            -0.01676257699728012,
+            0.006918908096849918,
+            0.0005550052737817168,
+            0.0007992075989022851,
+            0.002907755319029093,
+            4.006075323559344e-05,
+            0.011132225394248962,
+            -0.01469937339425087,
+            -0.003922352101653814,
+            -0.007640651427209377,
+            -0.010043942369520664,
+            0.006465457379817963,
+            0.007217430509626865,
+            0.0008539995760656893,
+            0.022476060315966606,
+            -0.017790399491786957,
+            0.007890049368143082,
+            -0.0018478132551535964,
+            0.01612774468958378,
+            0.01676257699728012,
+            0.011442082934081554,
+            -0.004477829672396183,
+            -0.005687032826244831,
+            -0.015916133299469948,
+            0.04226164519786835,
+            -0.005033307243138552,
+            0.00028010053210891783,
+            0.0086458008736372,
+            0.008351057767868042,
+            0.0008747827960178256,
+            -0.0095753762871027,
+            -0.010572968982160091,
+            -0.009038791991770267,
+            0.012326313182711601,
+            0.004734785296022892,
+            -0.011600791476666927,
+            -0.016414931043982506,
+            -0.006219837814569473,
+            0.021145937964320183,
+            0.009938136674463749,
+            -0.0077200052328407764,
+            0.001680603250861168,
+            -0.009076579473912716,
+            0.006268961820751429,
+            0.015266187489032745,
+            0.00838128849864006,
+            0.02821977436542511,
+            0.018818221986293793,
+            -0.030003348365426064,
+            0.02637574076652527,
+            0.009605606086552143,
+            0.005063537508249283,
+            0.0035236929543316364,
+            0.016868380829691887,
+            0.010777021758258343,
+            -0.009416667744517326,
+            -0.009666066616773605,
+            0.0013915281742811203,
+            0.0077200052328407764,
+            0.004564741160720587,
+            -0.016399815678596497,
+            0.0006296357605606318,
+            -0.0038108788430690765,
+            -0.014427303336560726,
+            -0.009152155369520187,
+            -0.010081729851663113,
+            -0.005350723396986723,
+            0.02237025462090969,
+            0.00523736048489809,
+            -0.009741641581058502,
+            -0.03213457018136978,
+            0.005101324990391731,
+            -0.016218435019254684,
+            -0.020858751609921455,
+            -0.014427303336560726,
+            -0.008766721934080124,
+            0.00010432918497826904,
+            -0.0113136051222682,
+            -0.022113298997282982,
+            -0.016218435019254684,
+            0.0007222153944894671,
+            -0.004621422849595547,
+            0.00804120022803545,
+            -0.005690811667591333,
+            0.031167207285761833,
+            -0.005932651925832033,
+            0.00530537823215127,
+            -0.006563704926520586,
+            0.004984183702617884,
+            0.004848148208111525,
+            -0.023186467587947845,
+            -0.005728599149733782,
+            0.006359652150422335,
+            -0.0013244551373645663,
+            0.00010574621410341933,
+            0.00646167853847146,
+            -0.012099587358534336,
+            0.005539661273360252,
+            0.003895900910720229,
+            -0.0006830107304267585,
+            -0.0014510435285046697,
+            0.005762607790529728,
+            0.005959103349596262,
+            0.015946364030241966,
+            -0.007890049368143082,
+            -0.017669478431344032,
+            -0.009144597686827183,
+            0.00047187256859615445,
+            -0.002110437024384737,
+            0.027675632387399673,
+            0.010565411299467087,
+            0.005516988690942526,
+            0.018138045445084572,
+            0.012477463111281395,
+            -0.0022200210951268673,
+            0.022989971563220024,
+            -0.0015974704874679446,
+            -0.015341762453317642,
+            0.009288189932703972,
+            -0.006609050091356039,
+            0.0014680479653179646,
+            -0.016717230901122093,
+            0.001082614529877901,
+            -0.007081395015120506,
+            0.0034159983042627573,
+            0.010240437462925911,
+            -0.001951729180291295,
+            0.013202984817326069,
+            -0.00954514555633068,
+            0.004889714531600475,
+            -0.0029852199368178844,
+            0.005789059214293957,
+            -0.023836413398385048,
+            0.006159377750009298,
+            -0.004821696784347296,
+            -0.0043002283200621605,
+            -0.002405180362984538,
+            0.006280297879129648,
+            -0.0032931887544691563,
+            0.013255887664854527,
+            -0.003586042672395706,
+            0.025529297068715096,
+            0.003841108875349164,
+            -0.0026999234687536955,
+            0.01410232950001955,
+            -0.0021274415776133537,
+            0.01036135759204626,
+            0.017200913280248642,
+            -0.02164473384618759,
+            -0.001862928387708962,
+            -0.01771482452750206,
+            0.00402437848970294,
+            0.00397147610783577,
+            0.006805545650422573,
+            -0.0016862713964655995,
+            0.0042435466311872005,
+            -0.02655711956322193,
+            -0.005411183461546898,
+            -0.006454120855778456,
+            0.014510435983538628,
+            -0.017064876854419708,
+            -0.018833335489034653,
+            -0.002526100492104888,
+            0.0037806485779583454,
+            0.0001020265044644475,
+            0.0008993447409011424,
+            0.0055736699141561985,
+            0.013490171171724796,
+            -0.001459545805118978,
+            -0.006302970461547375,
+            -0.011185127310454845,
+            -0.0017571230418980122,
+            0.01582544483244419,
+            -0.004194422625005245,
+            -0.015462683513760567,
+            -0.006794209126383066,
+            -0.013966294005513191,
+            -0.02702568657696247,
+            -0.00242029526270926,
+            -0.007281669415533543,
+            0.015228400006890297,
+            0.008970774710178375,
+            -0.0006919853040017188,
+            0.014261038042604923,
+            0.00027088981005363166,
+            0.010232879780232906,
+            -0.020148344337940216,
+            -0.024622395634651184,
+            -0.02311089262366295,
+            -0.010905499570071697,
+            0.003536918666213751,
+            -0.021176166832447052,
+            -0.01605216972529888,
+            0.02052622102200985,
+            -0.0048292544670403,
+            -0.0034802372101694345,
+            0.01073167659342289,
+            -0.004931280855089426,
+            -0.009114367887377739,
+            -0.007304341532289982,
+            -0.016248665750026703,
+            -0.0165660809725523,
+            0.01469181664288044,
+            0.003967697266489267,
+            -0.01549291331321001,
+            0.02658735029399395,
+            0.030487030744552612,
+            0.00922017265111208,
+            -0.008411518298089504,
+            -0.002813286380842328,
+            -0.009847446344792843,
+            -0.025136306881904602,
+            -0.0014633245300501585,
+            -0.007429040968418121,
+            -0.0040017059072852135,
+            -0.015764983370900154,
+            -0.005864634644240141,
+            -0.002218131674453616,
+            -0.005539661273360252,
+            -0.007995855063199997,
+            -0.002146335318684578,
+            0.03210433945059776,
+            0.009643393568694592,
+            0.010784579440951347,
+            0.0018393110949546099,
+            -0.010875268839299679,
+            0.014230807311832905,
+            -0.030623065307736397,
+            -0.016142860054969788,
+            0.013134967535734177,
+            0.001925277872942388,
+            0.010225323028862476,
+            -0.0023296051658689976,
+            0.0030437905807048082,
+            0.013051834888756275,
+            -0.004972847178578377,
+            0.015024347230792046,
+            0.00886496901512146,
+            -0.0017873531905934215,
+            -0.002206795383244753,
+            0.002907755319029093,
+            -0.002322047483175993,
+            0.008812067098915577,
+            0.01635446958243847,
+            -0.014971444383263588,
+            -0.00279439240694046,
+            0.007500837091356516,
+            0.0247282013297081,
+            -0.0026470208540558815,
+            -0.013021604157984257,
+            -0.0069907046854496,
+            0.021826114505529404,
+            0.007077616173774004,
+            -0.014638913795351982,
+            -0.0032610693015158176,
+            -0.006336979568004608,
+            0.031590428203344345,
+            0.02589205838739872,
+            0.02102501690387726,
+            -0.0013896387536078691,
+            -0.01865195669233799,
+            -0.004681882914155722,
+            -0.016974186524748802,
+            0.016248665750026703,
+            0.0027112597599625587,
+            -0.008910314179956913,
+            -0.0039412458427250385,
+            -0.015107479877769947,
+            -0.010043942369520664,
+            -0.0032535118516534567,
+            0.007421483285725117,
+            -0.017548557370901108,
+            -0.009242845699191093,
+            -0.003255401039496064,
+            -0.015417338348925114,
+            -0.002473197877407074,
+            0.01015730481594801,
+            -0.0065863775089383125,
+            -0.013679108582437038,
+            0.004976626019924879,
+            0.0007236324017867446,
+            0.012998932041227818,
+            -0.011925764381885529,
+            -0.026511775329709053,
+            -0.0021746759302914143,
+            0.011759499087929726,
+            0.0189089123159647,
+            -0.0008625017944723368,
+            -0.01862172596156597,
+            -0.03225548937916756,
+            -0.017427638173103333,
+            -0.012235622853040695,
+            -0.025529297068715096,
+            0.001894103130325675,
+            0.0020726495422422886,
+            0.0086458008736372,
+            -0.01505457703024149,
+            -0.043742917478084564,
+            0.005475422367453575,
+            -0.004326679278165102,
+            0.006336979568004608,
+            0.003963918425142765,
+            0.008018527179956436,
+            0.017397407442331314,
+            0.00670351879671216,
+            0.012273410335183144,
+            -0.00420198030769825,
+            -0.011819958686828613,
+            0.00852488074451685,
+            0.03470412641763687,
+            0.0057701654732227325,
+            -0.011880419217050076,
+            0.008955659344792366,
+            -0.014669143594801426,
+            -0.0009654730092734098,
+            0.003629498416557908,
+            -0.0012120370520278811,
+            -0.004043272230774164,
+            0.014654028229415417,
+            0.0009919243166223168,
+            -0.008494650945067406,
+            0.0008034586790017784,
+            0.02354922890663147,
+            -0.02007276937365532,
+            -0.001997074345126748,
+            -0.0067979879677295685,
+            -0.02631527930498123,
+            0.010709003545343876,
+            0.0014718268066644669,
+            -0.019543742761015892,
+            -0.01839500106871128,
+            0.0009612218709662557,
+            -0.014835408888757229,
+            0.03219502791762352,
+            0.01957397349178791,
+            -0.0038259937427937984,
+            -0.004296449478715658,
+            0.018969371914863586,
+            0.033192623406648636,
+            0.005093767773360014,
+            0.00524869654327631,
+            -0.011117110028862953,
+            0.009975924156606197,
+            0.01260594092309475,
+            0.0041528563015162945,
+            -0.000520996458362788,
+            0.017185797914862633,
+            -0.010293340310454369,
+            -0.015311532653868198,
+            -0.010875268839299679,
+            0.00218979106284678,
+            -0.014759833924472332,
+            -0.0045005022548139095,
+            -0.0030211179982870817,
+            -0.0009853114606812596,
+            0.02983708307147026,
+            -0.004999298602342606,
+            0.01798689365386963,
+            0.024320095777511597,
+            0.006571262609213591,
+            0.018757760524749756,
+            -0.016868380829691887,
+            -0.00021290946460794657,
+            0.0006858448032289743,
+            0.01909029111266136,
+            -0.0029058658983558416,
+            -0.00601200619712472,
+            -0.016460275277495384,
+            -0.005751271732151508,
+            -0.01176705677062273,
+            -0.03612493723630905,
+            0.02705591730773449,
+            -0.009061465039849281,
+            -0.024637511000037193,
+            -0.012658843770623207,
+            -0.028053509071469307,
+            -0.01366399321705103,
+            0.02353411354124546,
+            0.01844034530222416,
+            -0.0124850207939744,
+            -0.00780691672116518,
+            -0.02099478617310524,
+            -0.011404295451939106,
+            0.04873088002204895,
+            -0.0035255823750048876,
+            -0.02029949426651001,
+            0.009975924156606197,
+            -0.01889379695057869,
+            -0.015266187489032745,
+            0.012855338864028454,
+            0.006412554532289505,
+            0.04383360967040062,
+            0.011366507969796658,
+            0.00022991388686932623,
+            -0.01027822494506836,
+            0.000404563412303105,
+            -0.028809260576963425,
+            0.011638578958809376,
+            -0.014132560230791569,
+            -0.017382292076945305,
+            -0.0018648176919668913,
+            -0.011336278170347214,
+            -0.015311532653868198,
+            -0.0037107416428625584,
+            0.001894103130325675,
+            0.01236410066485405,
+            0.007217430509626865,
+            -0.008902756497263908,
+            0.01527374517172575,
+            0.018470576032996178,
+            -0.010127075016498566,
+            0.01933213323354721,
+            0.0025525519158691168,
+            0.006627943832427263,
+            -0.003502909792587161,
+            -0.01061831321567297,
+            -0.015508027747273445,
+            0.013852931559085846,
+            0.004523174837231636,
+            -0.016490506008267403,
+            0.0066014924086630344,
+            0.016384700313210487,
+            0.014858081936836243,
+            -0.008078987710177898,
+            -0.01198622491210699,
+            0.0007396921282634139,
+            0.005063537508249283,
+            -0.003856223775073886,
+            0.005033307243138552,
+            0.006378545891493559,
+            -0.006068687420338392,
+            -0.008517323061823845,
+            -0.0027377111837267876,
+            0.010965959168970585,
+            -0.0034802372101694345,
+            -0.018848450854420662,
+            0.003138259518891573,
+            -0.007651987485587597,
+            -0.008305712603032589,
+            0.02636062540113926,
+            0.0363667793571949,
+            -0.003128812648355961,
+            0.003111808327957988,
+            0.0034745691809803247,
+            0.0060422359965741634,
+            -0.010164862498641014,
+            0.012039126828312874,
+            0.00909925252199173,
+            -0.01609751395881176,
+            0.00792783685028553,
+            -0.015235957689583302,
+            -0.002306932583451271,
+            -0.012190277688205242,
+            0.008245253004133701,
+            -0.0004520340880844742,
+            0.004345573019236326,
+            0.014404630288481712,
+            -0.01505457703024149,
+            -0.01165369339287281,
+            -0.008585341274738312,
+            -0.009008562192320824,
+            0.027630288153886795,
+            0.01769970916211605,
+            -0.027615172788500786,
+            -0.000787398952525109,
+            -0.002512875013053417,
+            -0.023685263469815254,
+            0.0027641623746603727,
+            -0.0007160748937167227,
+            0.0017306717345491052,
+            0.0018251407891511917,
+            0.003895900910720229,
+            0.004644094966351986,
+            -0.00991546455770731,
+            0.013611091300845146,
+            0.013331462629139423,
+            -0.005611457396298647,
+            0.01445753313601017,
+            -0.01958908885717392,
+            0.007209872826933861,
+            -0.01585567370057106,
+            -0.008139447309076786,
+            0.006782873068004847,
+            0.024199174717068672,
+            0.0045458474196493626,
+            0.002652689116075635,
+            0.01517549715936184,
+            -1.436961792933289e-05,
+            0.0216598492115736,
+            0.010527623817324638,
+            0.0022237999364733696,
+            0.009650951251387596,
+            0.0025790033396333456,
+            0.005350723396986723,
+            -0.00977942906320095,
+            0.0009650006541050971,
+            -0.010542738251388073,
+            0.016490506008267403,
+            0.01457845326513052,
+            -0.0027112597599625587,
+            0.012507692910730839,
+            -0.006144262850284576,
+            -0.006488129962235689,
+            -0.01410232950001955,
+            0.005112661514431238,
+            0.009885234758257866,
+            -0.0038108788430690765,
+            0.002359835198149085,
+            0.005501873325556517,
+            -0.005135334096848965,
+            0.01339948084205389,
+            -0.005524545907974243,
+            0.003389547113329172,
+            0.02980685420334339,
+            -0.004345573019236326,
+            -0.0013556298799812794,
+            0.015568488277494907,
+            0.0029474322218447924,
+            -0.0036653964780271053,
+            0.003438670886680484,
+            -0.02608855441212654,
+            0.022022608667612076,
+            0.019725123420357704,
+            0.019468167796730995,
+            0.02421429008245468,
+            0.02335273288190365,
+            -0.007572633679956198,
+            -0.0016796585405245423,
+            0.009061465039849281,
+            0.0037220779340714216,
+            0.009688738733530045,
+            0.011381623335182667,
+            0.03594356030225754,
+            -0.002499649301171303,
+            0.005490537267178297,
+            -0.01961931772530079,
+            0.012961144559085369,
+            0.012681515887379646,
+            -0.004356909543275833,
+            0.0007760627195239067,
+            -0.022249335423111916,
+            0.013414595276117325,
+            -0.024395670741796494,
+            -0.0077540138736367226,
+            0.007791801821440458,
+            -0.01797178015112877,
+            0.031348586082458496,
+            -0.00478768814355135,
+            -0.012530365958809853,
+            0.009069022722542286,
+            0.00576638663187623,
+            0.005093767773360014,
+            -0.018561266362667084,
+            0.008728934451937675,
+            0.0011685813078656793,
+            -0.0035614806693047285,
+            -0.01039158832281828,
+            0.010467163287103176,
+            0.015976594761013985,
+            0.008827181532979012,
+            -0.00075150077464059,
+            -0.00804875697940588,
+            0.03621562942862511,
+            0.0008757274481467903,
+            0.0003894483670592308,
+            -0.0043946970254182816,
+            -0.0027282643131911755,
+            0.007882491685450077,
+            -0.00717964256182313,
+            -0.016188204288482666,
+            -0.014888311736285686,
+            -0.005267590284347534,
+            0.028083739802241325,
+            0.0036238301545381546,
+            0.013293675146996975,
+            -0.0075083947740495205,
+            -0.013293675146996975,
+            0.008351057767868042,
+            0.00518067879602313,
+            -0.0020972113125026226,
+            8.472686749882996e-05,
+            0.0027188172098249197,
+            -0.009378880262374878,
+            -0.034159984439611435,
+            -0.004235989414155483,
+            0.01676257699728012,
+            0.006733749061822891,
+            -0.006854669190943241,
+            0.01837988570332527,
+            -0.016702115535736084,
+            0.010671216063201427,
+            -0.0005975163076072931,
+            0.02283882163465023,
+            0.0163242407143116,
+            0.003030564868822694
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "153c961c-edfa-472c-922d-e04c3650a343",
+      "type": "next",
+      "source": {
+        "id": "d17e0745-ef0b-4687-8d99-5321eb6cab7e",
+        "properties": {
+          "page_content": "3 Scaling Properties 11 3.1 Scaling Results on a Step-Basis 12 3.2 Scaling Results on a Time-Basis 13 3.3 Scaling Versus a Larger Dense Model 13 ",
+          "keyphrases": [
+            "Scaling Properties",
+            "Scaling Results on a Step-Basis",
+            "Scaling Results on a Time-Basis",
+            "Scaling Versus a Larger Dense Model"
+          ],
+          "title": "Scaling Properties",
+          "embedding": [
+            -0.020178575068712234,
+            0.010542738251388073,
+            -0.025771139189600945,
+            0.015961479395627975,
+            0.009152155369520187,
+            -0.016702115535736084,
+            -0.001157245016656816,
+            0.07732853293418884,
+            -0.004825475625693798,
+            0.01630912534892559,
+            0.00782203208655119,
+            0.006609050091356039,
+            0.04265463352203369,
+            -0.01061831321567297,
+            0.03403906524181366,
+            0.045133501291275024,
+            0.008094102144241333,
+            0.006782873068004847,
+            -0.0005828736466355622,
+            -0.042836014181375504,
+            -0.026269935071468353,
+            -0.020103000104427338,
+            -0.027645403519272804,
+            0.029096446931362152,
+            0.025499068200588226,
+            0.0028454058337956667,
+            0.03932932764291763,
+            -0.0032780736219137907,
+            -0.010232879780232906,
+            0.0061329263262450695,
+            -0.011638578958809376,
+            0.002119883894920349,
+            -0.04081059992313385,
+            -0.002091543283313513,
+            -0.04446844011545181,
+            -0.02823488973081112,
+            0.06620386242866516,
+            -0.015024347230792046,
+            -0.0017486208816990256,
+            0.0305474903434515,
+            0.011358950287103653,
+            0.01002126932144165,
+            -0.01330123282968998,
+            -0.00594398844987154,
+            0.01584055833518505,
+            -0.04274532571434975,
+            -0.048398349434137344,
+            0.013391923159360886,
+            0.007130519021302462,
+            0.006314306985586882,
+            0.014230807311832905,
+            -0.0615786612033844,
+            0.040961749851703644,
+            -0.01794154942035675,
+            -0.02935340255498886,
+            0.01768459379673004,
+            -0.00886496901512146,
+            0.0859743282198906,
+            0.023745723068714142,
+            0.01330123282968998,
+            0.025272341445088387,
+            -0.028113968670368195,
+            -0.03201364725828171,
+            -0.006930244620889425,
+            0.0015370103064924479,
+            -0.006253846455365419,
+            -0.035550568252801895,
+            0.003850555745884776,
+            -0.03279963135719299,
+            -0.007655766326934099,
+            -0.033464692533016205,
+            -0.0024070697836577892,
+            0.0023749503307044506,
+            -0.01073167659342289,
+            0.006182050332427025,
+            -0.01117756962776184,
+            0.030834676697850227,
+            0.010504950769245625,
+            -0.014759833924472332,
+            0.029020871967077255,
+            0.052811939269304276,
+            0.039057254791259766,
+            -0.04205003380775452,
+            0.02893018163740635,
+            0.01701953262090683,
+            -0.03422044590115547,
+            -0.033011242747306824,
+            -0.055865176022052765,
+            0.009061465039849281,
+            -0.021266857162117958,
+            -0.0681990459561348,
+            -0.033736761659383774,
+            0.012983816675841808,
+            -0.03210433945059776,
+            -0.026919880881905556,
+            0.050967905670404434,
+            -0.01117001287639141,
+            0.010421818122267723,
+            -0.02122151292860508,
+            0.0011147339828312397,
+            -0.002512875013053417,
+            0.03597378730773926,
+            0.01862172596156597,
+            -0.01503946166485548,
+            0.02306554652750492,
+            -0.011600791476666927,
+            -0.0017051651375368237,
+            0.030199844390153885,
+            -0.006102696061134338,
+            0.05272125080227852,
+            0.0037315248046070337,
+            -0.01797178015112877,
+            -0.03198342025279999,
+            -0.03736437112092972,
+            -0.060036927461624146,
+            0.009363765828311443,
+            -0.00215578218922019,
+            -0.04594971239566803,
+            0.010527623817324638,
+            -0.02753959782421589,
+            0.02655711956322193,
+            0.02265744097530842,
+            -0.007481943350285292,
+            0.005112661514431238,
+            -0.032527558505535126,
+            0.04428705945611,
+            0.02823488973081112,
+            -0.02261209487915039,
+            -0.001841200515627861,
+            -0.02264232560992241,
+            0.004058387596160173,
+            0.019241442903876305,
+            -0.02985219843685627,
+            -0.007198536302894354,
+            -0.01142696850001812,
+            -0.016959071159362793,
+            0.050030771642923355,
+            0.025000272318720818,
+            -0.005399846937507391,
+            0.005161785054951906,
+            -0.011154897511005402,
+            -0.04071991145610809,
+            0.040689680725336075,
+            0.013452382758259773,
+            -0.03385768458247185,
+            0.0025582199450582266,
+            -0.028386039659380913,
+            0.062334414571523666,
+            -0.0035784849897027016,
+            -0.006261404138058424,
+            -0.02492469549179077,
+            -0.01747298240661621,
+            0.007746456656605005,
+            0.0467054657638073,
+            -0.0063445367850363255,
+            0.005543440114706755,
+            -0.018939141184091568,
+            0.0396013967692852,
+            -0.013928506523370743,
+            0.034432053565979004,
+            0.011661251075565815,
+            -0.007716226391494274,
+            0.014034312218427658,
+            0.01913563720881939,
+            0.028159314766526222,
+            0.00200652121566236,
+            -0.0012942251050844789,
+            0.022551635280251503,
+            -0.029217366129159927,
+            0.0015946364728733897,
+            0.02794770337641239,
+            0.011729269288480282,
+            0.012998932041227818,
+            0.01797178015112877,
+            -0.029459208250045776,
+            0.01653585024178028,
+            -0.02264232560992241,
+            0.026164129376411438,
+            -0.014880754053592682,
+            0.04005485028028488,
+            -0.003034343710169196,
+            -0.02941386215388775,
+            -0.011812401935458183,
+            0.021599387750029564,
+            0.030214959755539894,
+            -0.03056260570883751,
+            0.030456800013780594,
+            0.03518780693411827,
+            0.006627943832427263,
+            0.007213651668280363,
+            0.02353411354124546,
+            0.03476458415389061,
+            0.05646977946162224,
+            -0.00728922663256526,
+            -0.03286009281873703,
+            -0.027932588011026382,
+            0.010346243157982826,
+            -0.0043493518605828285,
+            0.01748809777200222,
+            0.026073439046740532,
+            -0.008320827968418598,
+            -0.02283882163465023,
+            0.0023995121009647846,
+            0.008373730815947056,
+            0.02237025462090969,
+            -0.011200242675840855,
+            -0.02078317664563656,
+            0.00010415205179015175,
+            0.017216026782989502,
+            -0.07104067504405975,
+            -0.010784579440951347,
+            -0.01410232950001955,
+            -0.024864235892891884,
+            -0.024380555376410484,
+            -0.008948101662099361,
+            0.013150081969797611,
+            0.01469937339425087,
+            0.02447124570608139,
+            0.0008743104408495128,
+            -0.02051110565662384,
+            -0.0035520337987691164,
+            -0.01797178015112877,
+            0.02985219843685627,
+            -0.04546603187918663,
+            -0.022294679656624794,
+            0.026405969634652138,
+            0.01073167659342289,
+            -0.0017287824302911758,
+            0.009650951251387596,
+            -0.039268866181373596,
+            0.037001609802246094,
+            -0.018304310739040375,
+            -0.001512448419816792,
+            0.009356208145618439,
+            0.009227730333805084,
+            -0.017548557370901108,
+            0.02332250215113163,
+            0.007973182015120983,
+            -0.021750537678599358,
+            0.03146950900554657,
+            0.022143529728055,
+            0.014631356112658978,
+            0.011661251075565815,
+            -0.03295078128576279,
+            0.015764983370900154,
+            0.013429710641503334,
+            -0.00931842066347599,
+            0.05486758425831795,
+            0.018833335489034653,
+            -0.004228431731462479,
+            0.022899281233549118,
+            -0.023216696456074715,
+            0.011381623335182667,
+            0.01455578114837408,
+            -0.0037031841930001974,
+            0.029020871967077255,
+            0.02543860860168934,
+            0.022521406412124634,
+            -0.04894249141216278,
+            0.01937747746706009,
+            0.03219502791762352,
+            -0.00419064424932003,
+            -0.0698314756155014,
+            0.025952519848942757,
+            0.0018770986935123801,
+            0.01748809777200222,
+            0.017352063208818436,
+            0.02144823782145977,
+            0.01674746163189411,
+            0.002259697997942567,
+            0.043924298137426376,
+            -0.039057254791259766,
+            0.019044946879148483,
+            0.0010344353504478931,
+            -0.026859421283006668,
+            -0.014638913795351982,
+            -0.007379916962236166,
+            -0.01726137287914753,
+            -0.010580525733530521,
+            0.026723386719822884,
+            0.02049599029123783,
+            0.033676303923130035,
+            -0.01191064901649952,
+            0.028809260576963425,
+            -0.0014926099684089422,
+            0.00100326060783118,
+            -0.0124850207939744,
+            -0.00945445615798235,
+            -0.017442753538489342,
+            -0.016732346266508102,
+            0.003291299333795905,
+            0.0034613434690982103,
+            -0.03875495493412018,
+            0.01202401239424944,
+            -0.007024713326245546,
+            -0.01751832850277424,
+            0.023458538576960564,
+            -0.033887915313243866,
+            0.0400850772857666,
+            -0.015780098736286163,
+            -0.05078652501106262,
+            0.006786651909351349,
+            0.01837988570332527,
+            -0.030381225049495697,
+            0.00044494890607893467,
+            0.004428706131875515,
+            0.030668411403894424,
+            -0.0060875811614096165,
+            -0.01933213323354721,
+            0.001051439787261188,
+            0.002350388327613473,
+            0.024622395634651184,
+            -0.0022294679656624794,
+            -0.016898611560463905,
+            -0.006306749302893877,
+            0.0045609623193740845,
+            0.020888982340693474,
+            0.0007798414444550872,
+            -0.025015385821461678,
+            0.02170519344508648,
+            0.025483952835202217,
+            -0.014412187971174717,
+            -0.05244917795062065,
+            0.03824104368686676,
+            0.033706534653902054,
+            -0.007478164508938789,
+            -0.04226164519786835,
+            0.016006823629140854,
+            -0.029081331565976143,
+            0.01798689365386963,
+            0.00021491693041753024,
+            -0.00454206857830286,
+            -0.016248665750026703,
+            0.019014716148376465,
+            0.021599387750029564,
+            -0.013271003030240536,
+            -0.005369617138057947,
+            -0.042291875928640366,
+            -0.036487698554992676,
+            0.017140451818704605,
+            -0.01304427720606327,
+            0.00782203208655119,
+            -0.012326313182711601,
+            -0.02421429008245468,
+            0.009605606086552143,
+            -0.05263055860996246,
+            -0.04640316590666771,
+            0.0015011121286079288,
+            0.024894466623663902,
+            -0.0001595344947418198,
+            0.0009215449099428952,
+            -0.010172420181334019,
+            0.01961931772530079,
+            0.020375071093440056,
+            0.024894466623663902,
+            -0.0358830988407135,
+            -0.002492091851308942,
+            0.028113968670368195,
+            0.0014963886933401227,
+            -0.018153158947825432,
+            0.014517993666231632,
+            -0.025499068200588226,
+            -0.006265182979404926,
+            0.0008100715349428356,
+            -0.0326484814286232,
+            0.004504281096160412,
+            -0.005407404620200396,
+            -0.0013424042845144868,
+            0.010247995145618916,
+            0.0316811166703701,
+            -0.021145937964320183,
+            0.004625201225280762,
+            0.040447838604450226,
+            -0.0043040066957473755,
+            -0.013134967535734177,
+            0.008260367438197136,
+            0.017911318689584732,
+            -0.017125336453318596,
+            -0.022083070129156113,
+            0.02329227328300476,
+            -0.02379106916487217,
+            0.014805179089307785,
+            0.045828793197870255,
+            -0.012696631252765656,
+            0.006484351120889187,
+            0.042291875928640366,
+            0.02540837787091732,
+            0.013679108582437038,
+            -0.01423836499452591,
+            -0.008676031604409218,
+            0.012054242193698883,
+            -0.059099797159433365,
+            9.500037413090467e-05,
+            -0.005857076961547136,
+            0.022899281233549118,
+            -0.002246472518891096,
+            0.014087215065956116,
+            -0.021372662857174873,
+            0.007104067597538233,
+            0.03009403869509697,
+            -0.02333761751651764,
+            0.014381958171725273,
+            -0.057678982615470886,
+            0.016369584947824478,
+            0.012394330464303493,
+            0.012160047888755798,
+            -0.0045118387788534164,
+            -0.0124850207939744,
+            0.027207067236304283,
+            -0.007920279167592525,
+            0.009280633181333542,
+            0.01745786890387535,
+            -0.022324910387396812,
+            -0.053023550659418106,
+            0.017805512994527817,
+            -0.010323570109903812,
+            -0.030653296038508415,
+            0.007485722191631794,
+            -0.02167496271431446,
+            -0.021312203258275986,
+            -0.044438209384679794,
+            0.012046684511005878,
+            0.037666670978069305,
+            -0.016732346266508102,
+            -0.04274532571434975,
+            -0.0004893493023701012,
+            -0.022249335423111916,
+            -0.013822701759636402,
+            0.010746791027486324,
+            -0.03225548937916756,
+            0.020178575068712234,
+            0.018138045445084572,
+            0.048882029950618744,
+            -0.010316013358533382,
+            -0.004345573019236326,
+            0.02823488973081112,
+            -0.029217366129159927,
+            0.005097546149045229,
+            -0.02471308596432209,
+            -0.03446228429675102,
+            0.02888483554124832,
+            0.0050597586669027805,
+            0.029020871967077255,
+            0.015235957689583302,
+            -0.007988297380506992,
+            0.016974186524748802,
+            -0.021281972527503967,
+            -0.027690747752785683,
+            -0.018062468618154526,
+            -0.026980342343449593,
+            -0.04504281282424927,
+            0.028355810791254044,
+            -0.05734645202755928,
+            -0.019966963678598404,
+            0.028612766414880753,
+            0.009552703239023685,
+            -0.0021274415776133537,
+            0.005853298120200634,
+            -0.019453052431344986,
+            -0.036517929285764694,
+            -0.011374065652489662,
+            0.028794145211577415,
+            0.009598048403859138,
+            -0.010187535546720028,
+            0.005762607790529728,
+            0.009598048403859138,
+            -0.04407544806599617,
+            -0.016898611560463905,
+            -0.00010893454600591213,
+            0.013694223947823048,
+            -0.02170519344508648,
+            -0.04425682872533798,
+            -0.04211049526929855,
+            0.009386437945067883,
+            -0.02729775756597519,
+            0.007912722416222095,
+            -0.030411455780267715,
+            -0.04845881089568138,
+            -0.004073502495884895,
+            -0.0071909790858626366,
+            0.02382129803299904,
+            0.02584671415388584,
+            -2.5447581720072776e-05,
+            0.03201364725828171,
+            -0.032527558505535126,
+            0.011389181017875671,
+            0.006571262609213591,
+            -0.021538928151130676,
+            -0.009008562192320824,
+            -0.0022294679656624794,
+            -0.029897544533014297,
+            0.029292942956089973,
+            0.01913563720881939,
+            0.00495017459616065,
+            -0.0025941182393580675,
+            -0.027146607637405396,
+            -0.02149358205497265,
+            -0.05789059400558472,
+            0.014752276241779327,
+            0.05380953103303909,
+            -0.030502144247293472,
+            -0.01051250845193863,
+            -0.017835743725299835,
+            0.019740238785743713,
+            0.04827743023633957,
+            0.007950509898364544,
+            0.03149973601102829,
+            -0.023050431162118912,
+            0.009514915756881237,
+            0.006975589785724878,
+            0.011018862016499043,
+            -0.044861432164907455,
+            -0.0014614351093769073,
+            -0.013376807793974876,
+            -0.0042586615309119225,
+            -0.003671064740046859,
+            -0.015598718076944351,
+            -0.025000272318720818,
+            -0.0045005022548139095,
+            0.006911350879818201,
+            -0.0050710951909422874,
+            0.002527989912778139,
+            -0.018017124384641647,
+            0.03479481488466263,
+            -0.04017576947808266,
+            0.03827127441763878,
+            -0.006427669432014227,
+            0.004568520002067089,
+            0.04981916397809982,
+            -0.018077583983540535,
+            -0.007648208644241095,
+            -0.008033642545342445,
+            -0.0038524451665580273,
+            -0.01141185313463211,
+            0.010096845217049122,
+            -0.014608683064579964,
+            0.0017864084802567959,
+            0.020465759560465813,
+            -0.02584671415388584,
+            -0.020344840362668037,
+            -0.005841962061822414,
+            0.0031590426806360483,
+            -0.0405687615275383,
+            0.006170713808387518,
+            0.0012951696990057826,
+            0.030169613659381866,
+            0.014155232347548008,
+            -0.03128812834620476,
+            0.000974919879809022,
+            0.01565917767584324,
+            -0.012492578476667404,
+            -0.02330738678574562,
+            0.03225548937916756,
+            0.003327197628095746,
+            -0.004972847178578377,
+            0.003485905472189188,
+            0.01769970916211605,
+            -0.013732011429965496,
+            0.01863684132695198,
+            -0.006522138603031635,
+            0.012265852652490139,
+            0.017397407442331314,
+            -0.03618539869785309,
+            0.016173088923096657,
+            0.007950509898364544,
+            -0.019241442903876305,
+            0.008139447309076786,
+            0.019044946879148483,
+            -0.025499068200588226,
+            0.021145937964320183,
+            -0.02097967080771923,
+            0.016626540571451187,
+            -0.04537534341216087,
+            -0.008192350156605244,
+            0.02896041050553322,
+            0.009802102111279964,
+            0.0013943621888756752,
+            -0.022234220057725906,
+            -0.0013839706080034375,
+            0.04809604957699776,
+            0.005229802802205086,
+            -0.011525215581059456,
+            -0.011850189417600632,
+            0.00513911247253418,
+            0.00043975311564281583,
+            0.0009342982666566968,
+            -0.02753959782421589,
+            -0.04588925465941429,
+            -0.006393660791218281,
+            -0.036941152065992355,
+            0.0326484814286232,
+            0.027343101799488068,
+            -0.0037126310635358095,
+            -0.016868380829691887,
+            0.0189089123159647,
+            -0.0006636445759795606,
+            0.03282986208796501,
+            -0.007478164508938789,
+            0.04150589182972908,
+            -0.052842170000076294,
+            0.0018478132551535964,
+            -0.011517658829689026,
+            -0.02193191833794117,
+            -0.007406368386000395,
+            -0.014638913795351982,
+            -0.018002009019255638,
+            0.01892402581870556,
+            0.01083748135715723,
+            0.0031533746514469385,
+            0.016974186524748802,
+            -0.0028246226720511913,
+            0.0035274717956781387,
+            0.012409445829689503,
+            -0.012084471993148327,
+            -0.003589821280911565,
+            0.04458935931324959,
+            0.009530031122267246,
+            -0.00979454442858696,
+            -0.017321832478046417,
+            -0.0024864235892891884,
+            0.052116647362709045,
+            -0.008494650945067406,
+            0.021569158881902695,
+            0.03606447950005531,
+            0.007958066649734974,
+            0.03074398636817932,
+            -0.004685661755502224,
+            0.026678040623664856,
+            0.003858113195747137,
+            -0.019180981442332268,
+            -0.006310528144240379,
+            -0.020405299961566925,
+            -0.01340703759342432,
+            -0.005849519278854132,
+            -0.023186467587947845,
+            0.026058323681354523,
+            -0.006525917444378138,
+            -0.00215578218922019,
+            0.0006490019150078297,
+            -0.030623065307736397,
+            0.021418007090687752,
+            0.018334539607167244,
+            0.014722046442329884,
+            0.04099198058247566,
+            0.007164527662098408,
+            0.02403290942311287,
+            -0.002614901401102543,
+            -0.004761236719787121,
+            0.014004082418978214,
+            0.024818891659379005,
+            0.024380555376410484,
+            0.03606447950005531,
+            -0.005350723396986723,
+            0.028567420318722725,
+            0.032527558505535126,
+            -0.027464022859930992,
+            0.014419745653867722,
+            0.016883496195077896,
+            -0.006008227355778217,
+            -0.021614503115415573,
+            -0.0022559193894267082,
+            0.02073783054947853,
+            0.002458082977682352,
+            -0.00838128849864006,
+            0.0363667793571949,
+            -0.02498515695333481,
+            -0.015946364030241966,
+            0.0031514852307736874,
+            -0.03267871215939522,
+            0.02495492622256279,
+            0.0028246226720511913,
+            -0.00792783685028553,
+            -0.02608855441212654,
+            -0.0076595451682806015,
+            0.0029757730662822723,
+            -0.011381623335182667,
+            -0.0038543345872312784,
+            0.04189888387918472,
+            -0.006654395256191492,
+            0.01318031270056963,
+            -0.0022389148361980915,
+            0.004957732278853655,
+            -0.0058306255377829075,
+            0.028854606673121452,
+            0.0006863171583972871,
+            0.0049199447967112064,
+            -0.022098183631896973,
+            -0.007580191362649202,
+            0.0049652899615466595,
+            0.015538258478045464,
+            0.027826784178614616,
+            0.019649548456072807,
+            0.010127075016498566,
+            0.027010571211576462,
+            -0.02172030881047249,
+            0.00020688706717919558,
+            -0.017397407442331314,
+            0.050726063549518585,
+            0.036034248769283295,
+            -0.023005086928606033,
+            -0.012787321582436562,
+            0.010323570109903812,
+            -0.03594356030225754,
+            0.012137374840676785,
+            0.02445613034069538,
+            -0.0005446136929094791,
+            -0.0065863775089383125,
+            0.0048632631078362465,
+            -0.008101659826934338,
+            -0.026028094813227654,
+            -0.00919750053435564,
+            0.02654200606048107,
+            0.0002616790879983455,
+            0.0062425103969872,
+            0.011827516369521618,
+            0.016173088923096657,
+            -0.017442753538489342,
+            -0.029338287189602852,
+            0.000420150812715292,
+            0.037424832582473755,
+            0.01109443698078394,
+            0.004372024443000555,
+            0.019286787137389183,
+            -0.009280633181333542,
+            -0.014140116982161999,
+            0.02474331669509411,
+            0.019906504079699516,
+            -0.0013216211227700114,
+            0.004938838537782431,
+            -0.007973182015120983,
+            0.00968118105083704,
+            -0.059553246945142746,
+            0.03258801996707916,
+            0.008570225909352303,
+            0.017079992219805717,
+            -0.0009522473555989563,
+            -0.007988297380506992,
+            0.028159314766526222,
+            -0.0003171795979142189,
+            0.008056314662098885,
+            0.016943957656621933,
+            -0.030895136296749115,
+            0.008366173133254051,
+            -0.018531035631895065,
+            -0.0010495503665879369,
+            0.019286787137389183,
+            0.01013463269919157,
+            -0.013021604157984257,
+            -0.012636170722544193,
+            -0.009371323511004448,
+            0.023443423211574554,
+            -0.007153191603720188,
+            0.015077250078320503,
+            0.0004227487079333514,
+            -0.009734083898365498,
+            -0.02241560071706772,
+            0.006733749061822891,
+            0.026647809892892838,
+            -0.025090962648391724,
+            0.01700441725552082,
+            -0.01529641728848219,
+            0.01435928512364626,
+            -0.029247596859931946,
+            0.005486758425831795,
+            0.04507303982973099,
+            0.011812401935458183,
+            -0.040659449994564056,
+            -0.010890384204685688,
+            -0.0003204860258847475,
+            -0.01154788862913847,
+            -0.007617978844791651,
+            0.012318755500018597,
+            0.05692322924733162,
+            -0.03051725961267948,
+            0.008532438427209854,
+            0.028975525870919228,
+            -0.014034312218427658,
+            0.016671886667609215,
+            -0.019966963678598404,
+            0.033222850412130356,
+            0.006850890815258026,
+            0.008539996109902859,
+            0.001844034530222416,
+            -0.0016796585405245423,
+            -0.0096585089340806,
+            0.0041037327609956264,
+            -0.018954256549477577,
+            -0.017064876854419708,
+            0.010172420181334019,
+            0.013429710641503334,
+            0.006714855320751667,
+            -0.00838128849864006,
+            -0.027222182601690292,
+            -0.001923388452269137,
+            0.019710008054971695,
+            -0.016671886667609215,
+            -0.032497331500053406,
+            0.014404630288481712,
+            0.012855338864028454,
+            -0.015538258478045464,
+            0.0391177162528038,
+            0.0041490779258310795,
+            0.01037647295743227,
+            0.015167939476668835,
+            -0.006321864202618599,
+            -0.008736491203308105,
+            0.024441014975309372,
+            0.0170497614890337,
+            -0.004334236960858107,
+            -0.016278894618153572,
+            0.015455125831067562,
+            0.030623065307736397,
+            0.013331462629139423,
+            0.005947767291218042,
+            -0.012401888146996498,
+            0.0378178246319294,
+            -0.004390918184071779,
+            -0.011555446311831474,
+            -0.03171134740114212,
+            -0.014034312218427658,
+            -0.02049599029123783,
+            -0.0033555382397025824,
+            -0.0039412458427250385,
+            0.012273410335183144,
+            -0.024531705304980278,
+            0.004594971425831318,
+            0.008509766310453415,
+            0.00636343052610755,
+            0.08131889998912811,
+            0.01798689365386963,
+            0.007633093744516373,
+            0.023413192480802536,
+            -0.01154788862913847,
+            0.029474321752786636,
+            -0.02708614617586136,
+            0.017427638173103333,
+            -0.013278559781610966,
+            0.011109552346169949,
+            -0.005350723396986723,
+            0.02590717375278473,
+            0.02332250215113163,
+            -0.0007576412172056735,
+            0.002580892527475953,
+            0.013225657865405083,
+            -0.0034084408544003963,
+            -0.019740238785743713,
+            0.012076914310455322,
+            0.030880020931363106,
+            0.03654816001653671,
+            0.012961144559085369,
+            0.02169007807970047,
+            0.008441748097538948,
+            -0.01049739308655262,
+            -0.02026926539838314,
+            -0.006866005714982748,
+            0.005524545907974243,
+            0.021614503115415573,
+            0.008358615450561047,
+            0.009582933969795704,
+            0.00642011221498251,
+            -0.008751606568694115,
+            -0.006091360002756119,
+            0.00636343052610755,
+            -0.014147674664855003,
+            0.03225548937916756,
+            0.005977997090667486,
+            -0.018092699348926544,
+            -0.028809260576963425,
+            -0.009137040004134178,
+            -0.01605216972529888,
+            -0.033041469752788544,
+            -0.01015730481594801,
+            0.008547553792595863,
+            -0.03482504561543465,
+            -0.008471978828310966,
+            0.01051250845193863,
+            -0.018077583983540535,
+            -0.03775736317038536,
+            0.014744718559086323,
+            -0.028129084035754204,
+            0.023911988362669945,
+            -0.024622395634651184,
+            0.010089287534356117,
+            0.002883193315938115,
+            -0.006457899697124958,
+            -0.011563003063201904,
+            -0.0022823705803602934,
+            -0.002686697756871581,
+            -0.0020103000570088625,
+            -0.03140904754400253,
+            -0.014321497641503811,
+            -0.017367178574204445,
+            -0.002590339398011565,
+            -0.033706534653902054,
+            0.025090962648391724,
+            0.030411455780267715,
+            0.012658843770623207,
+            0.007323235739022493,
+            -0.008751606568694115,
+            -0.01841011457145214,
+            -0.007874934002757072,
+            -0.033011242747306824,
+            -0.0064805722795426846,
+            -0.05928117781877518,
+            -0.010905499570071697,
+            -0.008932987228035927,
+            0.01772993803024292,
+            -0.013202984817326069,
+            -0.010814809240400791,
+            -0.02100990153849125,
+            -0.04099198058247566,
+            -0.019014716148376465,
+            0.007202315144240856,
+            -0.03379722312092781,
+            0.05925094708800316,
+            0.013210542500019073,
+            0.006968032103031874,
+            -0.024577051401138306,
+            0.010051500052213669,
+            -0.000378112104954198,
+            0.0096585089340806,
+            -0.005902422126382589,
+            -0.010913057252764702,
+            0.013339020311832428,
+            0.01698930189013481,
+            -0.008517323061823845,
+            0.0012687183916568756,
+            0.016173088923096657,
+            0.01682303659617901,
+            0.02614901401102543,
+            0.022249335423111916,
+            0.019014716148376465,
+            0.032769400626420975,
+            -0.029232481494545937,
+            0.06293901801109314,
+            -0.011479870416224003,
+            0.005369617138057947,
+            0.016928842291235924,
+            -0.019211212173104286,
+            0.029791738837957382,
+            -0.015251072123646736,
+            -0.02143312245607376,
+            0.00335175939835608,
+            0.005607679020613432,
+            0.013943621888756752,
+            -0.05217710882425308,
+            0.033434461802244186,
+            0.03963162750005722,
+            0.017352063208818436,
+            -0.00019826677453238517,
+            0.009174827486276627,
+            -0.0190751776099205,
+            0.038120124489068985,
+            -0.007342129480093718,
+            0.019997194409370422,
+            0.03727368265390396,
+            -0.0008039310341700912,
+            0.029292942956089973,
+            -0.012326313182711601,
+            0.02894529700279236,
+            -0.006605271250009537,
+            0.012666401453316212,
+            0.01198622491210699,
+            -0.009325978346168995,
+            -0.023262042552232742,
+            -0.015326648019254208,
+            0.0017731827683746815,
+            0.0012384883593767881,
+            -0.02843138575553894,
+            0.030245188623666763,
+            -0.019241442903876305,
+            -0.0007533901371061802,
+            0.006095138844102621,
+            -0.017412522807717323,
+            0.000406925129937008,
+            -0.0006593934958800673,
+            0.009507358074188232,
+            -0.005441413260996342,
+            0.010686331428587437,
+            0.01584055833518505,
+            -0.011842631734907627,
+            -0.02240048535168171,
+            0.01701953262090683,
+            -0.010066614486277103,
+            0.006968032103031874,
+            -0.03310193121433258,
+            -0.003370653372257948,
+            -0.03775736317038536,
+            -0.02847672998905182,
+            0.03948047757148743,
+            -0.0163242407143116,
+            -0.02265744097530842,
+            0.021599387750029564,
+            -0.03524826839566231,
+            -0.010111959651112556,
+            0.021508697420358658,
+            -0.010988632217049599,
+            -0.011736826039850712,
+            0.027554713189601898,
+            0.002395733492448926,
+            -0.009401553310453892,
+            -0.04990985244512558,
+            -0.026012979447841644,
+            0.004870820790529251,
+            0.011215358041226864,
+            -0.013308790512382984,
+            0.04546603187918663,
+            -0.01479762140661478,
+            -0.042352333664894104,
+            -0.015228400006890297,
+            0.01529641728848219,
+            -0.0099003491923213,
+            -0.019468167796730995,
+            -0.014230807311832905,
+            0.009726526215672493,
+            0.0024070697836577892,
+            -0.03149973601102829,
+            -0.01108688022941351,
+            0.02169007807970047,
+            -0.009847446344792843,
+            0.011744383722543716,
+            -0.03585286810994148,
+            -0.021856343373656273,
+            -0.01844034530222416,
+            0.020677370950579643,
+            0.029202252626419067,
+            -0.00115252158138901,
+            0.010368915274739265,
+            0.011895534582436085,
+            0.024652626365423203,
+            0.010452047921717167,
+            -0.0195588581264019,
+            0.015183054842054844,
+            -0.016037054359912872,
+            0.04032691940665245,
+            -0.0027830563485622406,
+            0.004417369607836008,
+            -0.03355538100004196,
+            0.0029323173221200705,
+            0.009303305298089981,
+            0.020223919302225113,
+            -0.009091694839298725,
+            0.03821081295609474,
+            -0.024335209280252457,
+            0.002350388327613473,
+            -0.009514915756881237,
+            0.01352040097117424,
+            0.03727368265390396,
+            0.002677250886335969,
+            -0.005146670155227184,
+            -0.007164527662098408,
+            0.019483283162117004,
+            0.024818891659379005,
+            -0.03760621324181557,
+            -0.013694223947823048,
+            0.039027027785778046,
+            0.00967362429946661,
+            -0.00875916425138712,
+            0.00459875026717782,
+            0.014737160876393318,
+            -0.023050431162118912,
+            0.017412522807717323,
+            0.015364435501396656,
+            0.009023677557706833,
+            -0.01889379695057869,
+            0.017352063208818436,
+            -0.016868380829691887,
+            -0.011419410817325115,
+            0.02753959782421589,
+            0.026285050436854362,
+            -0.025483952835202217,
+            0.016913726925849915,
+            -0.02401779405772686,
+            0.028068624436855316,
+            0.002153892768546939,
+            0.002055644989013672,
+            0.00874404888600111,
+            -0.004096175078302622,
+            -0.02867322601377964,
+            0.0012375436490401626,
+            0.006571262609213591,
+            0.006616607774049044,
+            0.0036238301545381546,
+            0.010028827004134655,
+            0.014926099218428135,
+            -0.008449305780231953,
+            0.008434190414845943,
+            0.00838884525001049,
+            0.02917202189564705,
+            -0.009356208145618439,
+            -0.027373332530260086,
+            -0.006102696061134338,
+            -0.0029512110631912947,
+            0.028506960719823837,
+            0.01292335707694292,
+            -0.00814700499176979,
+            0.02167496271431446,
+            0.09613163769245148,
+            0.006185829173773527,
+            -0.020163459703326225,
+            -0.02353411354124546,
+            -0.013437268324196339,
+            0.005891085602343082,
+            -0.006684625055640936,
+            0.009371323511004448,
+            0.004704555496573448,
+            -0.013815144076943398,
+            -0.009862561710178852,
+            0.03748529404401779,
+            -0.002057534409686923,
+            -0.008298155851662159,
+            -0.012885569594800472,
+            0.01841011457145214,
+            0.00780691672116518,
+            -0.008993446826934814,
+            0.004145299084484577,
+            -0.02519676648080349,
+            -0.03171134740114212,
+            -0.010656101629137993,
+            -0.005123997572809458,
+            0.02329227328300476,
+            -0.006423891056329012,
+            0.019014716148376465,
+            0.0054225195199251175,
+            0.012099587358534336,
+            -0.009991039521992207,
+            -0.0012252627639099956,
+            -0.018939141184091568,
+            0.019060062244534492,
+            -0.009371323511004448,
+            -0.0009862561710178852,
+            -0.01176705677062273,
+            0.003833551425486803,
+            -0.019634433090686798,
+            -0.0015766873257234693,
+            -0.029066216200590134,
+            0.016475390642881393,
+            -0.0054225195199251175,
+            0.0069491383619606495,
+            0.011842631734907627,
+            -0.012401888146996498,
+            0.0007992075989022851,
+            0.006166934967041016,
+            -0.0027981712482869625,
+            -0.03703184053301811,
+            0.011797286570072174,
+            0.007274111732840538,
+            0.02147846855223179,
+            -0.020405299961566925,
+            -0.034371595829725266,
+            -0.012076914310455322,
+            -0.0017467314610257745,
+            0.013369250111281872,
+            0.014109887182712555,
+            0.01059564109891653,
+            -0.025952519848942757,
+            -0.008426633663475513,
+            0.018153158947825432,
+            0.012144932523369789,
+            0.023141121491789818,
+            -0.006964253261685371,
+            0.041626811027526855,
+            -0.001176138874143362,
+            0.024335209280252457,
+            -0.006695961579680443,
+            -0.0018657624023035169,
+            -0.01934724859893322,
+            0.00885741226375103,
+            0.031348586082458496,
+            0.010860154405236244,
+            -0.015303974971175194,
+            -0.022944627329707146,
+            0.00953758880496025,
+            0.011245587840676308,
+            -0.006231173872947693,
+            0.009038791991770267,
+            -0.024365440011024475,
+            -0.00780691672116518,
+            0.0011780281784012914,
+            0.0012687183916568756,
+            0.021085476502776146,
+            -0.033918142318725586,
+            -0.01984604448080063,
+            -0.008373730815947056,
+            -0.014842966571450233,
+            -0.005003077443689108,
+            0.002580892527475953,
+            0.004020600114017725,
+            0.01200889702886343,
+            -0.012568153440952301,
+            -0.002246472518891096,
+            -0.024773545563220978,
+            0.00021161051699891686,
+            0.019271671772003174,
+            -0.007769129239022732,
+            -0.023201582953333855,
+            0.012658843770623207,
+            -0.016838151961565018,
+            0.010300897993147373,
+            0.026209475472569466,
+            -0.010399146005511284,
+            0.014412187971174717,
+            -0.015961479395627975,
+            -0.01936236210167408,
+            0.01328611746430397,
+            -0.005709705408662558,
+            -0.002926649060100317,
+            0.009975924156606197,
+            0.005796616896986961,
+            -0.00010568717698333785,
+            0.02982196770608425,
+            0.034432053565979004,
+            -0.023262042552232742,
+            0.013233215548098087,
+            -0.008124332875013351,
+            0.006427669432014227,
+            -0.0009295747731812298,
+            -0.02212841436266899,
+            0.014344170689582825,
+            0.014132560230791569,
+            -0.0022237999364733696,
+            -0.002837848151102662,
+            0.023624803870916367,
+            -0.0008270759717561305,
+            -0.01027822494506836,
+            0.031590428203344345,
+            0.002308822004124522,
+            0.0005960993003100157,
+            0.009016119875013828,
+            0.021508697420358658,
+            0.04171750321984291,
+            0.012658843770623207,
+            -0.00953758880496025,
+            0.0013452382991090417,
+            -0.005294041708111763,
+            0.009325978346168995,
+            -0.012069357559084892,
+            0.02072271518409252,
+            0.00896321702748537,
+            0.0244863610714674,
+            -0.003982812166213989,
+            -0.004791466984897852,
+            -0.02001230977475643,
+            -0.01537954993546009,
+            0.00814700499176979,
+            0.033192623406648636,
+            -0.000691512948833406,
+            -0.015674293041229248,
+            -0.0030362331308424473,
+            0.033041469752788544,
+            -0.0033668745309114456,
+            -0.030955595895648003,
+            -0.03122766688466072,
+            -0.00484059052541852,
+            -0.0019167757127434015,
+            0.012915799394249916,
+            -0.0017618464771658182,
+            0.007172085344791412,
+            0.015349320136010647,
+            -0.04576833173632622,
+            -0.046977534890174866,
+            0.015901019796729088,
+            -0.010777021758258343,
+            0.007247660309076309,
+            0.019407708197832108,
+            -0.033948373049497604,
+            -0.008789394050836563,
+            0.0060271210968494415,
+            0.012787321582436562,
+            0.008177234791219234,
+            0.006310528144240379,
+            0.015183054842054844,
+            -0.027872128412127495,
+            -0.019286787137389183,
+            0.020103000104427338,
+            0.0011317384196445346,
+            -0.004341794643551111,
+            0.0024788661394268274,
+            0.01132872048765421,
+            -0.009152155369520187,
+            0.01352040097117424,
+            0.0006357762613333762,
+            0.019301902502775192,
+            -0.011215358041226864,
+            -0.006333200726658106,
+            -0.010920614004135132,
+            0.0016258112154901028,
+            0.014147674664855003,
+            0.01724625751376152,
+            0.0027887243777513504,
+            0.006831996608525515,
+            0.001246045925654471,
+            0.00991546455770731,
+            -0.015016789548099041,
+            -0.0006310528260655701,
+            0.01071656122803688,
+            0.014019196853041649,
+            0.0035161355044692755,
+            0.009718968532979488,
+            0.006605271250009537,
+            0.0025393262039870024,
+            0.0024335209745913744,
+            -0.025967633351683617,
+            -0.009605606086552143,
+            -0.013905834406614304,
+            0.011804844252765179,
+            -0.004402254708111286,
+            0.0057852803729474545,
+            -0.010293340310454369,
+            -0.012968702241778374,
+            0.020118115469813347,
+            -0.0048292544670403,
+            -0.010520066134631634,
+            -0.026466431096196175,
+            -0.039571166038513184,
+            0.027630288153886795,
+            -0.009492243640124798,
+            0.0037579762283712626,
+            -0.01039158832281828,
+            0.022793475538492203,
+            0.0052713691256940365,
+            0.007814474403858185,
+            0.006166934967041016,
+            0.0067375279031693935,
+            -0.0031873835250735283,
+            -0.02097967080771923,
+            -0.0024316315539181232,
+            -0.01703464612364769,
+            0.017533443868160248,
+            -0.010210207663476467,
+            0.021735424175858498,
+            0.015719639137387276,
+            0.00023640862491447479,
+            0.006268961820751429,
+            0.02613389864563942,
+            0.012757091782987118,
+            0.0160824004560709,
+            -0.025816483423113823,
+            -0.0037749805487692356,
+            -0.01035380084067583,
+            0.024652626365423203,
+            -0.005505652166903019,
+            -0.00023735332069918513,
+            0.014865638688206673,
+            -0.022264450788497925,
+            -0.009870119392871857,
+            0.012878011912107468,
+            -0.006722413003444672,
+            -0.02234002575278282,
+            -0.001692884135991335,
+            -0.019664663821458817,
+            -0.007198536302894354,
+            -0.022143529728055,
+            0.005312935449182987,
+            0.018984487280249596,
+            -0.008721376769244671,
+            0.016399815678596497,
+            0.02915690653026104,
+            -0.006695961579680443,
+            0.024818891659379005,
+            0.0160824004560709,
+            0.0010372694814577699,
+            -0.00046148098772391677,
+            -0.005894864443689585,
+            0.004477829672396183,
+            -0.009333535097539425,
+            -0.012817551381886005,
+            -0.01748809777200222,
+            0.009061465039849281,
+            -0.018304310739040375,
+            0.015470240265130997,
+            0.03473435714840889,
+            -0.004020600114017725,
+            0.0042586615309119225,
+            -0.022959740832448006,
+            -0.016611425206065178,
+            -0.004447599872946739,
+            -0.014850524254143238,
+            0.06517603993415833,
+            -0.004795245360583067,
+            -0.014502878300845623,
+            -0.012666401453316212,
+            0.0036672858987003565,
+            -0.0028869721572846174,
+            -0.0036691753193736076,
+            -0.012182720005512238,
+            -0.015137709677219391,
+            -0.006964253261685371,
+            0.022445829585194588,
+            -0.032769400626420975,
+            0.005475422367453575,
+            0.030728871002793312,
+            -0.005641687661409378,
+            -0.0037579762283712626,
+            0.0023787289392203093,
+            -0.017397407442331314,
+            -0.012447233311831951,
+            -0.013565746136009693,
+            0.0008299099863506854,
+            0.0005710650002583861,
+            -0.0033215293660759926,
+            -0.020420415326952934,
+            0.01913563720881939,
+            -0.00850220862776041,
+            0.005309156607836485,
+            -0.029716163873672485,
+            -0.0055283247493207455,
+            0.012651286087930202,
+            0.0045005022548139095,
+            -0.019891388714313507,
+            0.013225657865405083,
+            -0.017790399491786957,
+            -0.0017023311229422688,
+            0.009356208145618439,
+            -0.01768459379673004,
+            -0.007391253020614386,
+            -0.024879351258277893,
+            -0.045798562467098236,
+            0.0036937373224645853,
+            -0.005743714049458504,
+            0.04011530801653862,
+            0.010452047921717167,
+            0.025816483423113823,
+            -0.011751941405236721,
+            -0.01798689365386963,
+            -0.008275482803583145,
+            -0.010897941887378693,
+            -0.015387107618153095,
+            -0.014193019829690456,
+            0.005683253984898329,
+            0.0029474322218447924,
+            -0.004885935690253973,
+            0.024380555376410484,
+            -0.006930244620889425,
+            -0.02188657410442829,
+            -0.003960139583796263,
+            -0.022037724032998085,
+            -0.01682303659617901,
+            0.006000669673085213,
+            -0.001827030093409121,
+            0.008630686439573765,
+            -0.005592563655227423,
+            -0.004398475866764784,
+            -0.016414931043982506,
+            0.00762931490316987,
+            0.020647140219807625,
+            0.010897941887378693,
+            -0.03691092133522034,
+            0.026511775329709053,
+            0.005849519278854132,
+            0.0007633093628101051,
+            0.017624134197831154,
+            0.004904829431325197,
+            0.007255217991769314,
+            0.01792643405497074,
+            -0.024773545563220978,
+            -0.004016821272671223,
+            -0.011880419217050076,
+            -0.025045616552233696,
+            -0.00943934079259634,
+            0.007799359038472176,
+            -0.020178575068712234,
+            -0.015689408406615257,
+            0.008366173133254051,
+            -0.02167496271431446,
+            0.01443486101925373,
+            -0.03503665700554848,
+            0.02026926539838314,
+            0.037878282368183136,
+            0.0072967843152582645,
+            0.0190751776099205,
+            0.002031083218753338,
+            0.00969629641622305,
+            -0.0014718268066644669,
+            -0.008328385651111603,
+            -0.02191680483520031,
+            0.01340703759342432,
+            -0.009257960133254528,
+            -0.003017339389771223,
+            -0.0030456800013780594,
+            0.00804120022803545,
+            -0.0009701964445412159,
+            -0.010127075016498566,
+            -0.0026111227925866842,
+            -0.015092364512383938,
+            0.008320827968418598,
+            -0.005660581402480602,
+            0.032315950840711594,
+            0.0008440803503617644,
+            0.018062468618154526,
+            0.0015332315815612674,
+            0.031620658934116364,
+            0.01863684132695198,
+            0.021841228008270264,
+            -0.0016891054110601544,
+            -0.008298155851662159,
+            0.00850220862776041,
+            -0.0019630654715001583,
+            -0.019906504079699516,
+            -0.01700441725552082,
+            -0.0028170649893581867,
+            0.0033196399454027414,
+            0.0006197165348567069,
+            0.007572633679956198,
+            0.006125368643552065,
+            -0.024093369022011757,
+            -0.01129093300551176,
+            0.003548254957422614,
+            0.0007727562915533781,
+            -0.005600121337920427,
+            0.009983481839299202,
+            -0.020647140219807625,
+            -0.0044060335494577885,
+            0.025317687541246414,
+            0.019755354151129723,
+            -0.025831598788499832,
+            -0.01549291331321001,
+            -0.005660581402480602,
+            0.0007500837091356516,
+            -0.018787991255521774,
+            -0.014389515854418278,
+            0.0037390822544693947,
+            -0.007504615932703018,
+            0.006779094226658344,
+            -0.01772993803024292,
+            -0.009514915756881237,
+            -0.0051202187314629555,
+            -0.030154500156641006,
+            0.012326313182711601,
+            0.013694223947823048,
+            0.030245188623666763,
+            -0.008676031604409218,
+            -0.01635446958243847,
+            -0.029323171824216843,
+            -0.006336979568004608,
+            -0.004765015561133623,
+            0.0033007462043315172,
+            0.004659210331737995,
+            -0.015115037560462952,
+            -0.025468837469816208,
+            -0.013376807793974876,
+            0.0063898819498717785,
+            -0.020571565255522728,
+            0.015251072123646736,
+            -0.008660916239023209,
+            0.03146950900554657,
+            -0.017805512994527817,
+            0.001601249212399125,
+            0.01222806517034769,
+            0.0005838182987645268,
+            0.007565075997263193,
+            0.0014954439830034971,
+            0.008071430027484894,
+            0.021130822598934174,
+            0.00826792512089014,
+            -0.01841011457145214,
+            0.007122961338609457,
+            0.006423891056329012,
+            -0.019422823563218117,
+            -0.0005328050465323031,
+            0.02052622102200985,
+            0.0030966931954026222,
+            0.00015670042193960398,
+            0.0021595610305666924,
+            0.004016821272671223,
+            0.01674746163189411,
+            -0.010436933487653732,
+            0.02495492622256279,
+            -0.00806387234479189,
+            -0.02732798643410206,
+            -0.01469181664288044,
+            -0.004077281337231398,
+            -0.03104628622531891,
+            -0.01108688022941351,
+            0.0009371322812512517,
+            -0.001392472768202424,
+            -0.006934023462235928,
+            0.0020443086978048086,
+            0.023866644129157066,
+            -0.003994148690253496,
+            -0.0021671184804290533,
+            0.01912052184343338,
+            0.09897326678037643,
+            0.002624348271638155,
+            -0.007746456656605005,
+            0.00437958212569356,
+            -0.011260703206062317,
+            -0.00647301459684968,
+            -0.018833335489034653,
+            -0.04924479126930237,
+            0.0016125855036079884,
+            1.3513492376659997e-05,
+            -0.004247325472533703,
+            -0.017125336453318596,
+            -0.013822701759636402,
+            0.013467498123645782,
+            -0.0060875811614096165,
+            0.01306694932281971,
+            -0.0052222455851733685,
+            0.0032686267513781786,
+            0.02329227328300476,
+            0.017881089821457863,
+            -0.006639279890805483,
+            0.009628279134631157,
+            -0.0029058658983558416,
+            0.008955659344792366,
+            0.018032239750027657,
+            0.014193019829690456,
+            -0.00769733265042305,
+            0.011661251075565815,
+            0.004946395754814148,
+            0.03981300815939903,
+            -0.007308120373636484,
+            -0.016928842291235924,
+            -0.00953758880496025,
+            -0.0009928690269589424,
+            -0.0053620594553649426,
+            0.011238030157983303,
+            -0.002766051795333624,
+            -0.016838151961565018,
+            -0.0017174461390823126,
+            -0.00780691672116518,
+            0.014306383207440376,
+            0.01117756962776184,
+            0.020163459703326225,
+            0.011457198299467564,
+            -0.002864299574866891,
+            0.028053509071469307,
+            -0.006129147484898567,
+            5.936785237281583e-05,
+            -0.013868046924471855,
+            0.004825475625693798,
+            -0.018017124384641647,
+            -0.002760383766144514,
+            0.005891085602343082,
+            -0.005078652407974005,
+            -0.0011563003063201904,
+            -0.0096585089340806,
+            0.005709705408662558,
+            -0.0027868349570780993,
+            0.010149747133255005,
+            -0.0036200513131916523,
+            -0.0018497026758268476,
+            0.009666066616773605,
+            0.00722120888531208,
+            -0.008056314662098885,
+            0.00922017265111208,
+            -0.021342432126402855,
+            -0.0007930670981295407,
+            -0.0074025895446538925,
+            0.003123144619166851,
+            -0.0019989637657999992,
+            -0.0030192285776138306,
+            -0.005569891072809696,
+            0.02031460963189602,
+            0.0017769616097211838,
+            -0.007980739697813988,
+            -0.022264450788497925,
+            0.022113298997282982,
+            -0.0031590426806360483,
+            0.014601126313209534,
+            -0.008827181532979012,
+            0.008925429545342922,
+            0.0035161355044692755,
+            -0.003438670886680484,
+            -0.002607343951240182,
+            -0.0018232513684779406,
+            -0.00361060444265604,
+            0.004307785537093878,
+            -0.001872375258244574,
+            0.0004959621583111584,
+            0.02311089262366295,
+            0.0009701964445412159,
+            -0.0027509366627782583,
+            0.012658843770623207,
+            0.006318085361272097,
+            0.0017004417022690177,
+            0.023881759494543076,
+            -0.008592898957431316,
+            -0.007727562915533781,
+            0.00012221925135236233,
+            -0.007258996833115816,
+            0.031590428203344345,
+            -0.009749199263751507,
+            0.00227103428915143,
+            0.020798292011022568,
+            -0.010285782627761364,
+            -0.003967697266489267,
+            0.003967697266489267,
+            -0.015553372912108898,
+            0.020344840362668037,
+            -0.023655032739043236,
+            0.022566750645637512,
+            0.012001339346170425,
+            -0.021372662857174873,
+            0.013913392089307308,
+            -0.00045439580571837723,
+            -0.013021604157984257,
+            0.006370988208800554,
+            0.007980739697813988,
+            -0.0420198030769825,
+            -0.008222579956054688,
+            -0.007882491685450077,
+            -0.03340423107147217,
+            0.009635835886001587,
+            -0.016430046409368515,
+            -0.0064956871792674065,
+            -0.0034084408544003963,
+            -0.006439005956053734,
+            -0.002450525527819991,
+            -0.02261209487915039,
+            -0.02327715791761875,
+            0.00642011221498251,
+            -0.02732798643410206,
+            -0.008184792473912239,
+            -0.0003235562762711197,
+            -0.002083985833451152,
+            -0.005668139085173607,
+            0.0052260239608585835,
+            0.016475390642881393,
+            0.0034821266308426857,
+            0.014049426652491093,
+            -0.005448970943689346,
+            0.011449640616774559,
+            -0.005626572761684656,
+            -0.01260594092309475,
+            -0.002288038842380047,
+            0.0027414897922426462,
+            -0.014011639170348644,
+            0.004251104313880205,
+            -0.008139447309076786,
+            -0.0060724662616848946,
+            0.012681515887379646,
+            -0.018939141184091568,
+            0.0018100256565958261,
+            0.008766721934080124,
+            7.917681796243414e-05,
+            -0.01258326880633831,
+            -0.028401155024766922,
+            0.00256955623626709,
+            -0.0054149623028934,
+            -0.002854852704331279,
+            0.00033772661117836833,
+            0.012417003512382507,
+            0.011540330946445465,
+            0.010701446793973446,
+            -0.002269144868478179,
+            0.007081395015120506,
+            0.028975525870919228,
+            0.013376807793974876,
+            -0.0074630496092140675,
+            0.0002914368233177811,
+            0.0022502511274069548,
+            0.0068886782974004745,
+            -0.020178575068712234,
+            -0.010323570109903812,
+            -0.0272372979670763,
+            -0.008766721934080124,
+            0.0014708820963278413,
+            0.0013594086049124599,
+            0.0244863610714674,
+            0.007493279874324799,
+            0.0018279748037457466,
+            -0.0028095075394958258,
+            0.010520066134631634,
+            0.007149412762373686,
+            0.0048028030432760715,
+            0.007009598426520824,
+            -0.018017124384641647,
+            -0.011593233793973923,
+            0.02146335318684578,
+            -0.0025884502101689577,
+            0.005543440114706755,
+            -0.0015200059860944748,
+            -0.0038203257136046886,
+            -0.004285112954676151,
+            0.01961931772530079,
+            -0.005671917926520109,
+            0.00012599800538737327,
+            0.029444092884659767,
+            -0.0035350292455404997,
+            0.0036238301545381546,
+            -0.006397439632564783,
+            -0.007224987726658583,
+            -0.04002461954951286,
+            0.01633935607969761,
+            -0.006983147002756596,
+            0.009431783109903336,
+            -0.010119517333805561,
+            0.009817216545343399,
+            0.010988632217049599,
+            0.0045609623193740845,
+            -0.010066614486277103,
+            0.0023995121009647846,
+            0.015047019347548485,
+            -0.011147339828312397,
+            -0.008313270285725594,
+            0.02100990153849125,
+            -0.006370988208800554,
+            -0.001745786750689149,
+            -0.0019177203066647053,
+            0.011948436498641968,
+            0.0027452686335891485,
+            0.018319424241781235,
+            0.005123997572809458,
+            -0.012076914310455322,
+            -0.0012970591196790338,
+            0.010308455675840378,
+            0.001875209272839129,
+            -0.006775315385311842,
+            0.005894864443689585,
+            -0.021554043516516685,
+            -0.0011270149843767285,
+            0.0018024682067334652,
+            0.023428307846188545,
+            0.005078652407974005,
+            0.026889652013778687,
+            -0.00035520337405614555,
+            0.0012167604872956872,
+            0.007727562915533781,
+            -0.007092731073498726,
+            -0.027146607637405396,
+            -0.01282510906457901,
+            -0.00828304048627615,
+            0.00018822944548446685,
+            0.0023749503307044506,
+            0.021735424175858498,
+            0.01841011457145214,
+            -0.02167496271431446,
+            0.007935394532978535,
+            -0.0047574578784406185,
+            -2.664320527401287e-05,
+            0.00888008438050747,
+            -0.00601200619712472,
+            0.003933688625693321,
+            0.00038921221857890487,
+            -0.0014944992726668715,
+            -0.007784244138747454,
+            -0.00886496901512146,
+            -0.0027112597599625587,
+            0.012084471993148327,
+            -0.003540697507560253,
+            0.008713819086551666,
+            -0.017654363065958023,
+            0.012386772781610489,
+            -0.008479535579681396,
+            0.03428090363740921,
+            0.010459605604410172,
+            -0.009061465039849281,
+            -0.013724453747272491,
+            -0.022083070129156113,
+            0.048851799219846725,
+            0.010346243157982826,
+            -0.0010835592402145267,
+            -0.00513911247253418,
+            -0.0077540138736367226,
+            -0.00036866520531475544,
+            -0.009582933969795704,
+            0.004315343219786882,
+            -0.004443821031600237,
+            0.006707297638058662,
+            0.0014160900609567761,
+            -0.0349157378077507,
+            0.0048179179430007935,
+            -0.014979002065956593,
+            -0.0012743865372613072,
+            -0.0008365228422917426,
+            -0.006892457138746977,
+            -0.00413396256044507,
+            -0.004625201225280762,
+            -0.008630686439573765,
+            0.01435928512364626,
+            0.0028850827366113663,
+            0.00026970895123668015,
+            -0.005585006438195705,
+            -0.013475055806338787,
+            0.008056314662098885,
+            -0.012386772781610489,
+            0.012462347745895386,
+            -0.006930244620889425,
+            -0.008668473921716213,
+            0.018304310739040375,
+            0.008252810686826706,
+            -0.0032421755604445934,
+            -0.02146335318684578,
+            -0.012333869934082031,
+            0.030895136296749115,
+            -0.010814809240400791,
+            0.005970439873635769,
+            -0.004451378248631954,
+            0.001610696199350059,
+            -0.006627943832427263,
+            0.008154562674462795,
+            0.014049426652491093,
+            0.01772993803024292,
+            0.017639247700572014,
+            -0.01771482452750206,
+            -0.0014689926756545901,
+            0.004799024201929569,
+            -0.02028438076376915,
+            -0.022959740832448006,
+            0.03682022914290428,
+            -0.002288038842380047,
+            0.014994116500020027,
+            0.0015162271447479725,
+            0.016369584947824478,
+            -0.011540330946445465,
+            0.017881089821457863,
+            0.004659210331737995,
+            -0.012280968017876148,
+            0.0019479504553601146,
+            -0.022007493302226067,
+            0.0396013967692852,
+            -0.010520066134631634,
+            -0.013210542500019073,
+            0.00734590832144022,
+            -0.02096455730497837,
+            -0.005093767773360014,
+            -0.024395670741796494,
+            -0.006080023944377899,
+            -0.00045415962813422084,
+            0.0004756513226311654,
+            0.007542403414845467,
+            -0.015523143112659454,
+            -0.028325580060482025,
+            -0.001077891094610095,
+            -0.011774614453315735,
+            -0.006148041225969791,
+            -0.01306694932281971,
+            -0.029761508107185364,
+            -0.011736826039850712,
+            0.039299096912145615,
+            -0.006979368627071381,
+            -0.0028964190278202295,
+            -0.0008029863820411265,
+            -0.0008398292702622712,
+            0.0038184362929314375,
+            0.019483283162117004,
+            -0.005887307226657867,
+            0.023609688505530357,
+            -0.005777723155915737,
+            0.010096845217049122,
+            0.004821696784347296,
+            -0.013822701759636402,
+            -0.013384365476667881,
+            0.014004082418978214,
+            0.003289409913122654,
+            -0.020390184596180916,
+            0.023670148104429245,
+            -0.008358615450561047,
+            0.01866707019507885,
+            -0.017216026782989502,
+            0.012099587358534336,
+            -0.02661758102476597,
+            -0.012757091782987118,
+            -0.014593568630516529,
+            -0.017805512994527817,
+            0.008819623850286007,
+            -0.011835074052214622,
+            0.01653585024178028,
+            -0.005743714049458504,
+            -0.0018837115494534373,
+            -0.0011723600327968597,
+            -0.037001609802246094,
+            0.0032591798808425665,
+            0.026693155989050865,
+            0.03675977140665054,
+            0.012311197817325592,
+            0.0030097817070782185,
+            -0.007950509898364544,
+            0.03654816001653671,
+            -0.010913057252764702,
+            -0.0122431805357337,
+            0.013187870383262634,
+            -0.006355873309075832,
+            -5.260741454549134e-05,
+            -0.01002126932144165,
+            0.02191680483520031,
+            0.013369250111281872,
+            0.04171750321984291,
+            -0.007319456897675991,
+            -0.000410467735491693,
+            -0.00897833239287138,
+            0.0007189089665189385,
+            0.007122961338609457,
+            0.0036616178695112467,
+            0.0017968000611290336,
+            -0.015704523772001266,
+            0.0067526428028941154,
+            -0.010867712087929249,
+            0.009046349674463272,
+            -0.0007529177819378674,
+            0.001029711915180087,
+            0.010043942369520664,
+            0.009348650462925434,
+            0.009076579473912716,
+            0.01958908885717392,
+            -0.012870454229414463,
+            -0.0059213158674538136,
+            -0.008895199745893478,
+            0.015024347230792046,
+            -0.0124850207939744,
+            0.0160824004560709,
+            0.006790430285036564,
+            -0.006091360002756119,
+            0.008411518298089504,
+            0.003138259518891573,
+            -0.0036559496074914932,
+            -0.0035180249251425266,
+            0.02422940544784069,
+            -0.007164527662098408,
+            -0.016399815678596497,
+            -0.00943934079259634,
+            0.018818221986293793,
+            -0.006114032585173845,
+            0.005214687902480364,
+            -0.005709705408662558,
+            -0.003502909792587161,
+            -0.013830259442329407,
+            -0.003361206501722336,
+            -0.010603198781609535,
+            0.0021274415776133537,
+            0.02028438076376915,
+            -0.0005294986767694354,
+            0.0034273345954716206,
+            0.0001297767594223842,
+            0.00804875697940588,
+            0.010913057252764702,
+            0.011472313664853573,
+            0.004387139342725277,
+            0.02333761751651764,
+            0.04126405343413353,
+            -0.00022826067288406193,
+            0.0005219411104917526,
+            -0.012537923641502857,
+            0.013928506523370743,
+            -0.0018374216742813587,
+            0.028340695425868034,
+            -0.01960420422255993,
+            0.0026848085690289736,
+            0.006125368643552065,
+            -0.01794154942035675,
+            -0.0003625247336458415,
+            0.001641870941966772,
+            0.00031694344943389297,
+            0.014147674664855003,
+            0.0051051038317382336,
+            -0.006627943832427263,
+            0.005199573002755642,
+            0.00804875697940588,
+            -0.0022237999364733696,
+            -0.015991710126399994,
+            0.012409445829689503,
+            0.0020461981184780598,
+            -0.0003955888678319752,
+            0.010127075016498566,
+            0.027887243777513504,
+            0.006185829173773527,
+            0.005566112697124481,
+            0.028053509071469307,
+            0.0027641623746603727,
+            0.010006154887378216,
+            0.00484059052541852,
+            0.007738898973912001,
+            0.007580191362649202,
+            -0.006321864202618599,
+            -0.014283710159361362,
+            -0.016974186524748802,
+            -0.01723114214837551,
+            -0.0030097817070782185,
+            -0.00016000683535821736,
+            0.03222525864839554,
+            0.001197866746224463,
+            0.01073167659342289,
+            0.01866707019507885,
+            -0.0017901872051879764,
+            -0.004885935690253973,
+            -0.025499068200588226,
+            -0.00489727221429348,
+            -0.0003348925383761525,
+            0.007073837332427502,
+            0.0005772055010311306,
+            0.004753679037094116,
+            -0.007799359038472176,
+            -0.00035071608726866543,
+            0.01655096560716629,
+            0.0014293156564235687,
+            0.013097180053591728,
+            -0.02351899817585945,
+            -0.023231811821460724,
+            0.010006154887378216,
+            0.020465759560465813,
+            0.005048422608524561,
+            -0.011661251075565815,
+            0.0046667675487697124,
+            0.01039158832281828,
+            0.0021425564773380756,
+            0.026753615587949753,
+            0.007323235739022493,
+            -0.018606610596179962,
+            0.008608013391494751,
+            0.010663658380508423,
+            0.01234898529946804,
+            -0.01176705677062273,
+            -0.0035010206047445536,
+            -0.007217430509626865,
+            -0.005690811667591333,
+            0.004107511602342129,
+            0.004353130701929331,
+            0.020934326574206352,
+            0.008509766310453415,
+            0.02008788473904133,
+            0.006151820067316294,
+            0.002475087298080325,
+            0.017382292076945305,
+            0.008441748097538948,
+            0.012069357559084892,
+            0.012757091782987118,
+            0.018712416291236877,
+            -0.02400267869234085,
+            0.017110222950577736,
+            -0.014963886700570583,
+            -0.007051164750009775,
+            0.0011336278403177857,
+            0.0048179179430007935,
+            0.004020600114017725,
+            -0.008237695321440697,
+            0.001068444224074483,
+            -0.010769464075565338,
+            0.00554721849039197,
+            -0.020919211208820343,
+            -0.00011064679711125791,
+            0.012160047888755798,
+            -0.003557701827958226,
+            -0.014767391607165337,
+            -0.010179977864027023,
+            0.02636062540113926,
+            0.011804844252765179,
+            0.012061799876391888,
+            -0.006544811185449362,
+            0.002210574224591255,
+            -0.015440010465681553,
+            0.013920948840677738,
+            0.018727531656622887,
+            -0.008872526697814465,
+            0.023246927186846733,
+            -0.014449975453317165,
+            0.006499466020613909,
+            -0.010149747133255005,
+            -0.006268961820751429,
+            -0.009212614968419075,
+            0.004292670637369156,
+            0.0041037327609956264,
+            0.016717230901122093,
+            0.008366173133254051,
+            -0.008940544910728931,
+            0.0086458008736372,
+            -0.0041982014663517475,
+            -0.008698703721165657,
+            -0.0172916017472744,
+            -0.014125002548098564,
+            -0.010754348710179329,
+            -0.0018138044979423285,
+            -0.0004062166262883693,
+            -0.022748131304979324,
+            0.00751973083242774,
+            0.0005649244994856417,
+            0.009114367887377739,
+            -0.001027822494506836,
+            -0.006873563397675753,
+            0.016974186524748802,
+            0.012076914310455322,
+            0.006896235514432192,
+            0.009552703239023685,
+            0.034643664956092834,
+            0.00945445615798235,
+            -0.007633093744516373,
+            0.013097180053591728,
+            -0.009446898475289345,
+            -0.027569828554987907,
+            0.0030910251662135124,
+            0.00241462723352015,
+            -0.000520996458362788,
+            0.0008346334798261523,
+            -0.005664360243827105,
+            0.0037220779340714216,
+            0.002367392648011446,
+            0.004062166437506676,
+            -0.0027981712482869625,
+            0.010671216063201427,
+            -0.026889652013778687,
+            0.014404630288481712,
+            0.014827851206064224,
+            0.011396737769246101,
+            -0.013482613489031792,
+            0.008260367438197136,
+            0.000955553783569485,
+            -0.017200913280248642,
+            -0.03243687003850937,
+            -0.01002126932144165,
+            0.01305939257144928,
+            0.011162455193698406,
+            0.004364467225968838,
+            0.0009560260805301368,
+            -0.0006178271723911166,
+            0.013792471028864384,
+            0.015213284641504288,
+            -0.004330458119511604,
+            -0.028310464695096016,
+            -0.023050431162118912,
+            0.032739169895648956,
+            0.007799359038472176,
+            0.006280297879129648,
+            0.019982079043984413,
+            0.006503244861960411,
+            0.0008549442864023149,
+            -0.02590717375278473,
+            0.014049426652491093,
+            -0.013505285605788231,
+            -0.004579856060445309,
+            0.006986925844103098,
+            -0.006499466020613909,
+            -0.016641655936837196,
+            0.013898276723921299,
+            0.001303671975620091,
+            -0.007243881467729807,
+            0.006677067838609219,
+            -0.008222579956054688,
+            -0.01001371257007122,
+            -0.01635446958243847,
+            -0.0008351058349944651,
+            -3.5425870009930804e-05,
+            -0.032074108719825745,
+            0.008721376769244671,
+            -0.012598383240401745,
+            0.007769129239022732,
+            0.014570895582437515,
+            0.004224652890115976,
+            -0.021750537678599358,
+            0.005562333855777979,
+            -0.012500136159360409,
+            -0.013497727923095226,
+            0.0069491383619606495,
+            -0.007334571797400713,
+            -0.00636343052610755,
+            -0.015024347230792046,
+            0.0051655638962984085,
+            0.006412554532289505,
+            0.019437937065958977,
+            -0.009552703239023685,
+            -0.0008813955937512219,
+            -0.014563338831067085,
+            0.014253480359911919,
+            -0.015455125831067562,
+            0.016278894618153572,
+            0.001953618600964546,
+            -0.004583634901791811,
+            -0.03237640857696533,
+            -0.003948803525418043,
+            0.0026281271129846573,
+            -0.006257625296711922,
+            -0.007723784074187279,
+            0.019241442903876305,
+            0.02005765400826931,
+            0.005101324990391731,
+            -0.01481273677200079,
+            -8.428404726146255e-06,
+            0.008192350156605244,
+            0.008018527179956436,
+            0.002970104804262519,
+            0.001875209272839129,
+            -0.016384700313210487,
+            -0.003960139583796263,
+            -0.022672556340694427,
+            -0.032557789236307144,
+            -0.006257625296711922,
+            0.0010920614004135132,
+            0.004084839019924402,
+            0.023216696456074715,
+            0.013905834406614304,
+            0.013527958653867245,
+            -0.006918908096849918,
+            0.021614503115415573,
+            0.009567818604409695,
+            -0.018092699348926544,
+            -0.009862561710178852,
+            -0.010043942369520664,
+            0.01629400998353958,
+            0.007213651668280363,
+            0.011147339828312397,
+            0.02823488973081112,
+            0.017352063208818436,
+            0.005558555014431477,
+            0.002766051795333624,
+            -0.007980739697813988,
+            -0.01364132110029459,
+            -0.01199378166347742,
+            -0.009257960133254528,
+            -0.0228690505027771,
+            0.009182385168969631,
+            -0.010520066134631634,
+            0.00665061641484499,
+            -0.008698703721165657,
+            -0.0022219105158001184,
+            0.0060271210968494415,
+            -0.01582544483244419,
+            0.0004508532292675227,
+            -0.01202401239424944,
+            -0.0008365228422917426,
+            0.00802608486264944,
+            0.010059057734906673,
+            -0.01889379695057869,
+            0.00792783685028553,
+            0.005150448996573687,
+            0.010920614004135132,
+            -0.02964058890938759,
+            -0.029323171824216843,
+            0.008320827968418598,
+            -0.01165369339287281,
+            -0.009824774228036404,
+            -0.026239704340696335,
+            -0.009250402450561523,
+            -0.013603533618152142,
+            -0.001517171855084598,
+            0.0019290565978735685,
+            0.021735424175858498,
+            -0.009159712120890617,
+            -0.03518780693411827,
+            -0.0252118818461895,
+            -0.015931248664855957,
+            -0.011215358041226864,
+            0.0019800697918981314,
+            -0.0023466094862669706,
+            -0.001889379695057869,
+            0.006896235514432192,
+            -0.02308066189289093,
+            -0.02049599029123783,
+            0.0006617552135139704,
+            0.010452047921717167,
+            -0.013339020311832428,
+            -0.010119517333805561,
+            0.026421085000038147,
+            -0.015553372912108898,
+            0.01001371257007122,
+            -0.013225657865405083,
+            -0.005925094708800316,
+            0.0010117627680301666,
+            -0.001555904163978994,
+            -0.004145299084484577,
+            0.00558878481388092,
+            -0.03403906524181366,
+            0.00513911247253418,
+            0.031832266598939896,
+            0.02382129803299904,
+            -0.004791466984897852,
+            -0.002518543042242527,
+            -0.03521803766489029,
+            -0.002465640427544713,
+            -0.012689073570072651,
+            0.02753959782421589,
+            -0.003945024684071541,
+            0.007474386133253574,
+            0.001541733741760254,
+            0.0036162727046757936,
+            -0.005600121337920427,
+            0.003094803774729371,
+            0.0005252475384622812,
+            -0.005857076961547136,
+            0.005671917926520109,
+            0.014963886700570583,
+            0.0009494132827967405,
+            0.012938471511006355,
+            -0.01635446958243847,
+            -0.008993446826934814,
+            -0.008872526697814465,
+            -0.024592164903879166,
+            -0.00558878481388092,
+            0.009839889593422413,
+            0.019226327538490295,
+            -0.01467670127749443,
+            -0.015946364030241966,
+            0.01527374517172575,
+            0.006624164991080761,
+            -0.02658735029399395,
+            -0.0007855095900595188,
+            -0.02403290942311287,
+            0.026436200365424156,
+            0.0163242407143116,
+            -0.0023371626157313585,
+            0.00244863610714674,
+            -0.006900014355778694,
+            -0.011691480875015259,
+            -0.013936064206063747,
+            0.00029309000819921494,
+            0.007867377251386642,
+            -0.032315950840711594,
+            0.0076142000034451485,
+            0.009159712120890617,
+            -0.0013546851696446538,
+            -0.02218887396156788,
+            0.017200913280248642,
+            0.00838884525001049,
+            0.0016569859581068158,
+            -0.019876273348927498,
+            0.008101659826934338,
+            0.0014264816418290138,
+            -0.004693218972533941,
+            -0.0049539534375071526,
+            -0.010406702756881714,
+            4.165491554886103e-05,
+            -0.023851528763771057,
+            -0.005974218249320984,
+            -0.007531067356467247,
+            -0.0029946668073534966,
+            0.017352063208818436,
+            -0.014510435983538628,
+            0.00020676899293903261,
+            0.009590490721166134,
+            0.018848450854420662,
+            0.010164862498641014,
+            -0.014593568630516529,
+            0.023216696456074715,
+            0.0019800697918981314,
+            -0.007364802062511444,
+            0.007047385908663273,
+            -0.005112661514431238,
+            -0.009303305298089981,
+            0.009431783109903336,
+            -0.0029927773866802454,
+            0.006563704926520586,
+            -0.008335943333804607,
+            -0.007281669415533543,
+            -0.013527958653867245,
+            -0.0013764130417257547,
+            0.014895869418978691,
+            0.003892122069373727,
+            -0.023896872997283936,
+            0.01528886053711176,
+            -0.002108547603711486,
+            -0.0024335209745913744,
+            -0.009998597204685211,
+            -0.019437937065958977,
+            -0.012976259924471378,
+            -0.009152155369520187,
+            -0.004776351619511843,
+            0.013210542500019073,
+            0.012968702241778374,
+            -0.011623463593423367,
+            -0.009567818604409695,
+            -0.004995519760996103,
+            -0.030487030744552612,
+            0.035792406648397446,
+            -0.019483283162117004,
+            0.02008788473904133,
+            -0.01910540647804737,
+            0.005192015320062637,
+            -0.001130793709307909,
+            0.0021614502184093,
+            -0.007799359038472176,
+            0.004995519760996103,
+            -0.019649548456072807,
+            -0.015961479395627975,
+            -0.006605271250009537,
+            -0.008033642545342445,
+            -0.021130822598934174,
+            -0.014608683064579964,
+            -0.007070058491080999,
+            0.012794879265129566,
+            -0.0011988113401457667,
+            -0.0014633245300501585,
+            0.020919211208820343,
+            0.007814474403858185,
+            0.011162455193698406,
+            0.026164129376411438,
+            0.022566750645637512,
+            0.03149973601102829,
+            -0.002148224739357829,
+            -0.01676257699728012,
+            0.006918908096849918,
+            0.0005550052737817168,
+            0.0007992075989022851,
+            0.002907755319029093,
+            4.006075323559344e-05,
+            0.011132225394248962,
+            -0.01469937339425087,
+            -0.003922352101653814,
+            -0.007640651427209377,
+            -0.010043942369520664,
+            0.006465457379817963,
+            0.007217430509626865,
+            0.0008539995760656893,
+            0.022476060315966606,
+            -0.017790399491786957,
+            0.007890049368143082,
+            -0.0018478132551535964,
+            0.01612774468958378,
+            0.01676257699728012,
+            0.011442082934081554,
+            -0.004477829672396183,
+            -0.005687032826244831,
+            -0.015916133299469948,
+            0.04226164519786835,
+            -0.005033307243138552,
+            0.00028010053210891783,
+            0.0086458008736372,
+            0.008351057767868042,
+            0.0008747827960178256,
+            -0.0095753762871027,
+            -0.010572968982160091,
+            -0.009038791991770267,
+            0.012326313182711601,
+            0.004734785296022892,
+            -0.011600791476666927,
+            -0.016414931043982506,
+            -0.006219837814569473,
+            0.021145937964320183,
+            0.009938136674463749,
+            -0.0077200052328407764,
+            0.001680603250861168,
+            -0.009076579473912716,
+            0.006268961820751429,
+            0.015266187489032745,
+            0.00838128849864006,
+            0.02821977436542511,
+            0.018818221986293793,
+            -0.030003348365426064,
+            0.02637574076652527,
+            0.009605606086552143,
+            0.005063537508249283,
+            0.0035236929543316364,
+            0.016868380829691887,
+            0.010777021758258343,
+            -0.009416667744517326,
+            -0.009666066616773605,
+            0.0013915281742811203,
+            0.0077200052328407764,
+            0.004564741160720587,
+            -0.016399815678596497,
+            0.0006296357605606318,
+            -0.0038108788430690765,
+            -0.014427303336560726,
+            -0.009152155369520187,
+            -0.010081729851663113,
+            -0.005350723396986723,
+            0.02237025462090969,
+            0.00523736048489809,
+            -0.009741641581058502,
+            -0.03213457018136978,
+            0.005101324990391731,
+            -0.016218435019254684,
+            -0.020858751609921455,
+            -0.014427303336560726,
+            -0.008766721934080124,
+            0.00010432918497826904,
+            -0.0113136051222682,
+            -0.022113298997282982,
+            -0.016218435019254684,
+            0.0007222153944894671,
+            -0.004621422849595547,
+            0.00804120022803545,
+            -0.005690811667591333,
+            0.031167207285761833,
+            -0.005932651925832033,
+            0.00530537823215127,
+            -0.006563704926520586,
+            0.004984183702617884,
+            0.004848148208111525,
+            -0.023186467587947845,
+            -0.005728599149733782,
+            0.006359652150422335,
+            -0.0013244551373645663,
+            0.00010574621410341933,
+            0.00646167853847146,
+            -0.012099587358534336,
+            0.005539661273360252,
+            0.003895900910720229,
+            -0.0006830107304267585,
+            -0.0014510435285046697,
+            0.005762607790529728,
+            0.005959103349596262,
+            0.015946364030241966,
+            -0.007890049368143082,
+            -0.017669478431344032,
+            -0.009144597686827183,
+            0.00047187256859615445,
+            -0.002110437024384737,
+            0.027675632387399673,
+            0.010565411299467087,
+            0.005516988690942526,
+            0.018138045445084572,
+            0.012477463111281395,
+            -0.0022200210951268673,
+            0.022989971563220024,
+            -0.0015974704874679446,
+            -0.015341762453317642,
+            0.009288189932703972,
+            -0.006609050091356039,
+            0.0014680479653179646,
+            -0.016717230901122093,
+            0.001082614529877901,
+            -0.007081395015120506,
+            0.0034159983042627573,
+            0.010240437462925911,
+            -0.001951729180291295,
+            0.013202984817326069,
+            -0.00954514555633068,
+            0.004889714531600475,
+            -0.0029852199368178844,
+            0.005789059214293957,
+            -0.023836413398385048,
+            0.006159377750009298,
+            -0.004821696784347296,
+            -0.0043002283200621605,
+            -0.002405180362984538,
+            0.006280297879129648,
+            -0.0032931887544691563,
+            0.013255887664854527,
+            -0.003586042672395706,
+            0.025529297068715096,
+            0.003841108875349164,
+            -0.0026999234687536955,
+            0.01410232950001955,
+            -0.0021274415776133537,
+            0.01036135759204626,
+            0.017200913280248642,
+            -0.02164473384618759,
+            -0.001862928387708962,
+            -0.01771482452750206,
+            0.00402437848970294,
+            0.00397147610783577,
+            0.006805545650422573,
+            -0.0016862713964655995,
+            0.0042435466311872005,
+            -0.02655711956322193,
+            -0.005411183461546898,
+            -0.006454120855778456,
+            0.014510435983538628,
+            -0.017064876854419708,
+            -0.018833335489034653,
+            -0.002526100492104888,
+            0.0037806485779583454,
+            0.0001020265044644475,
+            0.0008993447409011424,
+            0.0055736699141561985,
+            0.013490171171724796,
+            -0.001459545805118978,
+            -0.006302970461547375,
+            -0.011185127310454845,
+            -0.0017571230418980122,
+            0.01582544483244419,
+            -0.004194422625005245,
+            -0.015462683513760567,
+            -0.006794209126383066,
+            -0.013966294005513191,
+            -0.02702568657696247,
+            -0.00242029526270926,
+            -0.007281669415533543,
+            0.015228400006890297,
+            0.008970774710178375,
+            -0.0006919853040017188,
+            0.014261038042604923,
+            0.00027088981005363166,
+            0.010232879780232906,
+            -0.020148344337940216,
+            -0.024622395634651184,
+            -0.02311089262366295,
+            -0.010905499570071697,
+            0.003536918666213751,
+            -0.021176166832447052,
+            -0.01605216972529888,
+            0.02052622102200985,
+            -0.0048292544670403,
+            -0.0034802372101694345,
+            0.01073167659342289,
+            -0.004931280855089426,
+            -0.009114367887377739,
+            -0.007304341532289982,
+            -0.016248665750026703,
+            -0.0165660809725523,
+            0.01469181664288044,
+            0.003967697266489267,
+            -0.01549291331321001,
+            0.02658735029399395,
+            0.030487030744552612,
+            0.00922017265111208,
+            -0.008411518298089504,
+            -0.002813286380842328,
+            -0.009847446344792843,
+            -0.025136306881904602,
+            -0.0014633245300501585,
+            -0.007429040968418121,
+            -0.0040017059072852135,
+            -0.015764983370900154,
+            -0.005864634644240141,
+            -0.002218131674453616,
+            -0.005539661273360252,
+            -0.007995855063199997,
+            -0.002146335318684578,
+            0.03210433945059776,
+            0.009643393568694592,
+            0.010784579440951347,
+            0.0018393110949546099,
+            -0.010875268839299679,
+            0.014230807311832905,
+            -0.030623065307736397,
+            -0.016142860054969788,
+            0.013134967535734177,
+            0.001925277872942388,
+            0.010225323028862476,
+            -0.0023296051658689976,
+            0.0030437905807048082,
+            0.013051834888756275,
+            -0.004972847178578377,
+            0.015024347230792046,
+            0.00886496901512146,
+            -0.0017873531905934215,
+            -0.002206795383244753,
+            0.002907755319029093,
+            -0.002322047483175993,
+            0.008812067098915577,
+            0.01635446958243847,
+            -0.014971444383263588,
+            -0.00279439240694046,
+            0.007500837091356516,
+            0.0247282013297081,
+            -0.0026470208540558815,
+            -0.013021604157984257,
+            -0.0069907046854496,
+            0.021826114505529404,
+            0.007077616173774004,
+            -0.014638913795351982,
+            -0.0032610693015158176,
+            -0.006336979568004608,
+            0.031590428203344345,
+            0.02589205838739872,
+            0.02102501690387726,
+            -0.0013896387536078691,
+            -0.01865195669233799,
+            -0.004681882914155722,
+            -0.016974186524748802,
+            0.016248665750026703,
+            0.0027112597599625587,
+            -0.008910314179956913,
+            -0.0039412458427250385,
+            -0.015107479877769947,
+            -0.010043942369520664,
+            -0.0032535118516534567,
+            0.007421483285725117,
+            -0.017548557370901108,
+            -0.009242845699191093,
+            -0.003255401039496064,
+            -0.015417338348925114,
+            -0.002473197877407074,
+            0.01015730481594801,
+            -0.0065863775089383125,
+            -0.013679108582437038,
+            0.004976626019924879,
+            0.0007236324017867446,
+            0.012998932041227818,
+            -0.011925764381885529,
+            -0.026511775329709053,
+            -0.0021746759302914143,
+            0.011759499087929726,
+            0.0189089123159647,
+            -0.0008625017944723368,
+            -0.01862172596156597,
+            -0.03225548937916756,
+            -0.017427638173103333,
+            -0.012235622853040695,
+            -0.025529297068715096,
+            0.001894103130325675,
+            0.0020726495422422886,
+            0.0086458008736372,
+            -0.01505457703024149,
+            -0.043742917478084564,
+            0.005475422367453575,
+            -0.004326679278165102,
+            0.006336979568004608,
+            0.003963918425142765,
+            0.008018527179956436,
+            0.017397407442331314,
+            0.00670351879671216,
+            0.012273410335183144,
+            -0.00420198030769825,
+            -0.011819958686828613,
+            0.00852488074451685,
+            0.03470412641763687,
+            0.0057701654732227325,
+            -0.011880419217050076,
+            0.008955659344792366,
+            -0.014669143594801426,
+            -0.0009654730092734098,
+            0.003629498416557908,
+            -0.0012120370520278811,
+            -0.004043272230774164,
+            0.014654028229415417,
+            0.0009919243166223168,
+            -0.008494650945067406,
+            0.0008034586790017784,
+            0.02354922890663147,
+            -0.02007276937365532,
+            -0.001997074345126748,
+            -0.0067979879677295685,
+            -0.02631527930498123,
+            0.010709003545343876,
+            0.0014718268066644669,
+            -0.019543742761015892,
+            -0.01839500106871128,
+            0.0009612218709662557,
+            -0.014835408888757229,
+            0.03219502791762352,
+            0.01957397349178791,
+            -0.0038259937427937984,
+            -0.004296449478715658,
+            0.018969371914863586,
+            0.033192623406648636,
+            0.005093767773360014,
+            0.00524869654327631,
+            -0.011117110028862953,
+            0.009975924156606197,
+            0.01260594092309475,
+            0.0041528563015162945,
+            -0.000520996458362788,
+            0.017185797914862633,
+            -0.010293340310454369,
+            -0.015311532653868198,
+            -0.010875268839299679,
+            0.00218979106284678,
+            -0.014759833924472332,
+            -0.0045005022548139095,
+            -0.0030211179982870817,
+            -0.0009853114606812596,
+            0.02983708307147026,
+            -0.004999298602342606,
+            0.01798689365386963,
+            0.024320095777511597,
+            0.006571262609213591,
+            0.018757760524749756,
+            -0.016868380829691887,
+            -0.00021290946460794657,
+            0.0006858448032289743,
+            0.01909029111266136,
+            -0.0029058658983558416,
+            -0.00601200619712472,
+            -0.016460275277495384,
+            -0.005751271732151508,
+            -0.01176705677062273,
+            -0.03612493723630905,
+            0.02705591730773449,
+            -0.009061465039849281,
+            -0.024637511000037193,
+            -0.012658843770623207,
+            -0.028053509071469307,
+            -0.01366399321705103,
+            0.02353411354124546,
+            0.01844034530222416,
+            -0.0124850207939744,
+            -0.00780691672116518,
+            -0.02099478617310524,
+            -0.011404295451939106,
+            0.04873088002204895,
+            -0.0035255823750048876,
+            -0.02029949426651001,
+            0.009975924156606197,
+            -0.01889379695057869,
+            -0.015266187489032745,
+            0.012855338864028454,
+            0.006412554532289505,
+            0.04383360967040062,
+            0.011366507969796658,
+            0.00022991388686932623,
+            -0.01027822494506836,
+            0.000404563412303105,
+            -0.028809260576963425,
+            0.011638578958809376,
+            -0.014132560230791569,
+            -0.017382292076945305,
+            -0.0018648176919668913,
+            -0.011336278170347214,
+            -0.015311532653868198,
+            -0.0037107416428625584,
+            0.001894103130325675,
+            0.01236410066485405,
+            0.007217430509626865,
+            -0.008902756497263908,
+            0.01527374517172575,
+            0.018470576032996178,
+            -0.010127075016498566,
+            0.01933213323354721,
+            0.0025525519158691168,
+            0.006627943832427263,
+            -0.003502909792587161,
+            -0.01061831321567297,
+            -0.015508027747273445,
+            0.013852931559085846,
+            0.004523174837231636,
+            -0.016490506008267403,
+            0.0066014924086630344,
+            0.016384700313210487,
+            0.014858081936836243,
+            -0.008078987710177898,
+            -0.01198622491210699,
+            0.0007396921282634139,
+            0.005063537508249283,
+            -0.003856223775073886,
+            0.005033307243138552,
+            0.006378545891493559,
+            -0.006068687420338392,
+            -0.008517323061823845,
+            -0.0027377111837267876,
+            0.010965959168970585,
+            -0.0034802372101694345,
+            -0.018848450854420662,
+            0.003138259518891573,
+            -0.007651987485587597,
+            -0.008305712603032589,
+            0.02636062540113926,
+            0.0363667793571949,
+            -0.003128812648355961,
+            0.003111808327957988,
+            0.0034745691809803247,
+            0.0060422359965741634,
+            -0.010164862498641014,
+            0.012039126828312874,
+            0.00909925252199173,
+            -0.01609751395881176,
+            0.00792783685028553,
+            -0.015235957689583302,
+            -0.002306932583451271,
+            -0.012190277688205242,
+            0.008245253004133701,
+            -0.0004520340880844742,
+            0.004345573019236326,
+            0.014404630288481712,
+            -0.01505457703024149,
+            -0.01165369339287281,
+            -0.008585341274738312,
+            -0.009008562192320824,
+            0.027630288153886795,
+            0.01769970916211605,
+            -0.027615172788500786,
+            -0.000787398952525109,
+            -0.002512875013053417,
+            -0.023685263469815254,
+            0.0027641623746603727,
+            -0.0007160748937167227,
+            0.0017306717345491052,
+            0.0018251407891511917,
+            0.003895900910720229,
+            0.004644094966351986,
+            -0.00991546455770731,
+            0.013611091300845146,
+            0.013331462629139423,
+            -0.005611457396298647,
+            0.01445753313601017,
+            -0.01958908885717392,
+            0.007209872826933861,
+            -0.01585567370057106,
+            -0.008139447309076786,
+            0.006782873068004847,
+            0.024199174717068672,
+            0.0045458474196493626,
+            0.002652689116075635,
+            0.01517549715936184,
+            -1.436961792933289e-05,
+            0.0216598492115736,
+            0.010527623817324638,
+            0.0022237999364733696,
+            0.009650951251387596,
+            0.0025790033396333456,
+            0.005350723396986723,
+            -0.00977942906320095,
+            0.0009650006541050971,
+            -0.010542738251388073,
+            0.016490506008267403,
+            0.01457845326513052,
+            -0.0027112597599625587,
+            0.012507692910730839,
+            -0.006144262850284576,
+            -0.006488129962235689,
+            -0.01410232950001955,
+            0.005112661514431238,
+            0.009885234758257866,
+            -0.0038108788430690765,
+            0.002359835198149085,
+            0.005501873325556517,
+            -0.005135334096848965,
+            0.01339948084205389,
+            -0.005524545907974243,
+            0.003389547113329172,
+            0.02980685420334339,
+            -0.004345573019236326,
+            -0.0013556298799812794,
+            0.015568488277494907,
+            0.0029474322218447924,
+            -0.0036653964780271053,
+            0.003438670886680484,
+            -0.02608855441212654,
+            0.022022608667612076,
+            0.019725123420357704,
+            0.019468167796730995,
+            0.02421429008245468,
+            0.02335273288190365,
+            -0.007572633679956198,
+            -0.0016796585405245423,
+            0.009061465039849281,
+            0.0037220779340714216,
+            0.009688738733530045,
+            0.011381623335182667,
+            0.03594356030225754,
+            -0.002499649301171303,
+            0.005490537267178297,
+            -0.01961931772530079,
+            0.012961144559085369,
+            0.012681515887379646,
+            -0.004356909543275833,
+            0.0007760627195239067,
+            -0.022249335423111916,
+            0.013414595276117325,
+            -0.024395670741796494,
+            -0.0077540138736367226,
+            0.007791801821440458,
+            -0.01797178015112877,
+            0.031348586082458496,
+            -0.00478768814355135,
+            -0.012530365958809853,
+            0.009069022722542286,
+            0.00576638663187623,
+            0.005093767773360014,
+            -0.018561266362667084,
+            0.008728934451937675,
+            0.0011685813078656793,
+            -0.0035614806693047285,
+            -0.01039158832281828,
+            0.010467163287103176,
+            0.015976594761013985,
+            0.008827181532979012,
+            -0.00075150077464059,
+            -0.00804875697940588,
+            0.03621562942862511,
+            0.0008757274481467903,
+            0.0003894483670592308,
+            -0.0043946970254182816,
+            -0.0027282643131911755,
+            0.007882491685450077,
+            -0.00717964256182313,
+            -0.016188204288482666,
+            -0.014888311736285686,
+            -0.005267590284347534,
+            0.028083739802241325,
+            0.0036238301545381546,
+            0.013293675146996975,
+            -0.0075083947740495205,
+            -0.013293675146996975,
+            0.008351057767868042,
+            0.00518067879602313,
+            -0.0020972113125026226,
+            8.472686749882996e-05,
+            0.0027188172098249197,
+            -0.009378880262374878,
+            -0.034159984439611435,
+            -0.004235989414155483,
+            0.01676257699728012,
+            0.006733749061822891,
+            -0.006854669190943241,
+            0.01837988570332527,
+            -0.016702115535736084,
+            0.010671216063201427,
+            -0.0005975163076072931,
+            0.02283882163465023,
+            0.0163242407143116,
+            0.003030564868822694
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "8f4b7200-614f-45df-b481-ec48e0567e94",
+        "properties": {
+          "page_content": "4 Downstream Results 14 4.1 Fine-Tuning 14 4.2 Distillation 16 4.3 Multilingual Learning 17 ",
+          "keyphrases": [
+            "Downstream Results",
+            "Fine-Tuning",
+            "Distillation",
+            "Multilingual Learning"
+          ],
+          "embedding": [
+            0.007475246209651232,
+            0.018637971952557564,
+            -0.022418024018406868,
+            -0.003915054257959127,
+            -0.025920359417796135,
+            -0.01815968006849289,
+            -0.013469329103827477,
+            0.07850165665149689,
+            -0.036689650267362595,
+            0.014703632332384586,
+            0.017773959785699844,
+            -0.022819172590970993,
+            -0.010807863436639309,
+            -0.039034824818372726,
+            -0.02303517609834671,
+            0.02710837498307228,
+            0.017064236104488373,
+            -0.007691249251365662,
+            0.015552215278148651,
+            -0.02539578080177307,
+            -0.010514716617763042,
+            -0.0059323678724467754,
+            -0.04039255902171135,
+            0.03233873471617699,
+            -0.02058199979364872,
+            0.002258388325572014,
+            0.04107142612338066,
+            -0.006360516417771578,
+            -0.03289416804909706,
+            0.017465384677052498,
+            -0.020350567996501923,
+            -0.00360647845081985,
+            0.01323789730668068,
+            -0.029422692954540253,
+            -0.022634027525782585,
+            0.012574459426105022,
+            0.06967639178037643,
+            0.011378728784620762,
+            0.010777005925774574,
+            0.03610335662961006,
+            0.009380700998008251,
+            0.007938110269606113,
+            0.008539832197129726,
+            -0.007521532475948334,
+            -0.002765609649941325,
+            -0.047397226095199585,
+            -0.0303792767226696,
+            -0.04141085967421532,
+            0.011671875603497028,
+            -0.05992539972066879,
+            0.01718766614794731,
+            -0.060480836778879166,
+            0.016693945974111557,
+            -0.014703632332384586,
+            -0.002127243671566248,
+            0.057271648198366165,
+            0.014317912049591541,
+            0.05264301598072052,
+            0.05779622867703438,
+            0.020921431481838226,
+            0.030240418389439583,
+            -0.005635363515466452,
+            0.014194482006132603,
+            -0.025920359417796135,
+            0.002937254961580038,
+            -0.04468176141381264,
+            -0.009534989483654499,
+            -0.028512394055724144,
+            -0.029129546135663986,
+            0.025735212489962578,
+            -0.003972912207245827,
+            0.04144171625375748,
+            -0.007787679322063923,
+            -0.0030394706409424543,
+            -0.01823682337999344,
+            0.014634202234447002,
+            0.00830840039998293,
+            0.00699695385992527,
+            -0.010993009433150291,
+            0.03243130445480347,
+            0.0004541848611552268,
+            -0.0026498937513679266,
+            0.012991037219762802,
+            -0.016786517575383186,
+            0.03557877987623215,
+            -0.016925377771258354,
+            -0.04036170244216919,
+            -0.049773260951042175,
+            0.02891354262828827,
+            -0.026151791214942932,
+            -0.02970041148364544,
+            -0.02309689112007618,
+            0.0036836224608123302,
+            -0.013939906843006611,
+            -0.033449605107307434,
+            0.02380661480128765,
+            -0.007452103309333324,
+            0.001406912342645228,
+            -0.021631157025694847,
+            0.011209012009203434,
+            -0.003843696089461446,
+            0.005519647616893053,
+            0.004609349649399519,
+            0.0007053847657516599,
+            0.02632150799036026,
+            -0.017928248271346092,
+            0.00661894865334034,
+            -0.016678515821695328,
+            0.004844638518989086,
+            0.07368787378072739,
+            0.01564478874206543,
+            0.007540818769484758,
+            -0.02072085812687874,
+            0.025087203830480576,
+            -0.02229459397494793,
+            -0.018745973706245422,
+            -0.03391246870160103,
+            -0.04693436250090599,
+            0.0011436586501076818,
+            -0.027432380244135857,
+            0.04838467016816139,
+            0.025056347250938416,
+            0.006591948214918375,
+            -0.046378929167985916,
+            -0.053167592734098434,
+            0.034221045672893524,
+            0.025904929265379906,
+            0.031582724303007126,
+            -0.01393219269812107,
+            0.02979298308491707,
+            0.019393982365727425,
+            0.0023509610909968615,
+            -0.06424546241760254,
+            -0.0021214578300714493,
+            0.00862469058483839,
+            -0.0044126324355602264,
+            0.01812882162630558,
+            0.011733590625226498,
+            0.017912819981575012,
+            0.001125336973927915,
+            0.03468390926718712,
+            -0.03631936013698578,
+            0.02301974780857563,
+            -0.006086655426770449,
+            -0.022448882460594177,
+            0.0007304565515369177,
+            -0.009596704505383968,
+            0.09985509514808655,
+            0.018607115373015404,
+            -0.006129084620624781,
+            -0.015968792140483856,
+            0.003992198035120964,
+            -0.012682461179792881,
+            0.026861514896154404,
+            0.018406540155410767,
+            0.010159854777157307,
+            0.01717223785817623,
+            0.004825352691113949,
+            0.0030587567016482353,
+            0.0013577330391854048,
+            0.020767144858837128,
+            -0.007043240126222372,
+            0.019378554075956345,
+            0.006148370914161205,
+            0.07208327949047089,
+            0.03388161212205887,
+            -0.0052612158469855785,
+            0.026614654809236526,
+            -0.04964983090758324,
+            0.021044863387942314,
+            0.02553463913500309,
+            0.030857570469379425,
+            0.04526805505156517,
+            0.0030356133356690407,
+            -0.01653965748846531,
+            -0.016987092792987823,
+            -0.020828859880566597,
+            0.03162900730967522,
+            -0.06344316154718399,
+            0.01741909794509411,
+            -0.013153038918972015,
+            -0.014549343846738338,
+            0.020906003192067146,
+            -0.0063450876623392105,
+            0.01806710660457611,
+            -0.023621469736099243,
+            0.04292288050055504,
+            -0.00019466786761768162,
+            0.027324378490447998,
+            0.006726950407028198,
+            0.0067732366733253,
+            0.010908151045441628,
+            0.027386093512177467,
+            0.016940806061029434,
+            -0.0021947447676211596,
+            -0.007737535517662764,
+            -0.013716190122067928,
+            0.03545534610748291,
+            -0.0014773061266168952,
+            0.010576431639492512,
+            -0.019440269097685814,
+            -0.05048298463225365,
+            -0.0035293344408273697,
+            0.05329102277755737,
+            0.005943939555436373,
+            0.020412283018231392,
+            0.023528896272182465,
+            0.05344530940055847,
+            0.0020018848590552807,
+            -0.03142843395471573,
+            0.01661680079996586,
+            -0.01977970264852047,
+            -0.052735585719347,
+            0.026136361062526703,
+            -0.0027559667360037565,
+            -0.0024358194787055254,
+            -0.008269828744232655,
+            0.012867606244981289,
+            -0.04230572655797005,
+            -0.017650529742240906,
+            0.02061285637319088,
+            -0.0011561946012079716,
+            0.013461614958941936,
+            -0.01480391900986433,
+            -0.032554734498262405,
+            0.0017675601411610842,
+            -0.020427711308002472,
+            3.8240483263507485e-05,
+            0.009689277037978172,
+            -0.02221745066344738,
+            0.01800539158284664,
+            0.03056442365050316,
+            -0.022803744301199913,
+            -0.019270552322268486,
+            0.01731109619140625,
+            -0.018761401996016502,
+            0.014194482006132603,
+            0.00035775493597611785,
+            0.003957483451813459,
+            0.02880554087460041,
+            0.0005978654371574521,
+            -0.033418748527765274,
+            -0.02209402061998844,
+            -0.032585594803094864,
+            0.02317403443157673,
+            -0.03468390926718712,
+            -0.024207763373851776,
+            0.01882311701774597,
+            -0.008362401276826859,
+            0.012481886893510818,
+            0.006757807917892933,
+            0.005187929142266512,
+            0.016231082379817963,
+            0.01397847943007946,
+            -0.021584870293736458,
+            0.05446361005306244,
+            0.03206101432442665,
+            -0.024269478395581245,
+            -0.0248403437435627,
+            -0.04853895679116249,
+            0.018360253423452377,
+            -0.018483683466911316,
+            -0.051501285284757614,
+            0.0009676741319708526,
+            -0.003297902876511216,
+            -0.007864823564887047,
+            0.021260865032672882,
+            0.03378903865814209,
+            0.005824366118758917,
+            0.025704355910420418,
+            0.0031937584280967712,
+            0.004107913933694363,
+            -0.014927349053323269,
+            -0.021152863278985023,
+            -0.008015253581106663,
+            -0.03388161212205887,
+            0.019270552322268486,
+            -0.014024765230715275,
+            -0.01820596680045128,
+            0.022541454061865807,
+            0.008586118929088116,
+            0.04875496029853821,
+            0.01447991468012333,
+            0.0017743102507665753,
+            -0.022757457569241524,
+            -0.01633908413350582,
+            -0.014078766107559204,
+            0.02485577203333378,
+            -0.05665449798107147,
+            -0.0132224690169096,
+            0.011579303070902824,
+            -0.015544501133263111,
+            -0.05514247715473175,
+            0.005191786214709282,
+            -0.029808413237333298,
+            0.012782747857272625,
+            0.00558136310428381,
+            -0.005284358747303486,
+            0.010939008556306362,
+            0.01324561145156622,
+            -0.022464310750365257,
+            0.009635276161134243,
+            0.02153858356177807,
+            -0.008524403907358646,
+            -0.03366560861468315,
+            0.027648381888866425,
+            0.019424840807914734,
+            0.028651254251599312,
+            -0.058845385909080505,
+            0.00558907724916935,
+            0.028543252497911453,
+            -0.02470148541033268,
+            -0.013523329980671406,
+            -0.011533016338944435,
+            -0.03366560861468315,
+            0.007120384369045496,
+            0.02971583977341652,
+            0.007575533352792263,
+            -0.04468176141381264,
+            0.007679677568376064,
+            -0.011602446436882019,
+            0.009295842610299587,
+            -0.04212058335542679,
+            0.05168642848730087,
+            0.007108812686055899,
+            0.004316202364861965,
+            0.013052752241492271,
+            0.005288215819746256,
+            -0.011293870396912098,
+            0.03854110464453697,
+            -0.024207763373851776,
+            -0.010206141509115696,
+            -0.017465384677052498,
+            0.022526025772094727,
+            0.021662013605237007,
+            -0.04366346076130867,
+            -0.049032676964998245,
+            0.006044226698577404,
+            0.010661290027201176,
+            -0.015444213524460793,
+            -0.03527020290493965,
+            0.02459348365664482,
+            -0.007394245360046625,
+            -0.02064371481537819,
+            0.006414517294615507,
+            -0.029299262911081314,
+            -0.008964124135673046,
+            0.0019999563228338957,
+            0.0058937957510352135,
+            0.015536786057054996,
+            0.0009117447771131992,
+            0.010946722701191902,
+            -0.005990225821733475,
+            0.03780052438378334,
+            -0.012875321321189404,
+            -0.02951526641845703,
+            -0.020967718213796616,
+            0.019501984119415283,
+            0.024917487055063248,
+            0.03162900730967522,
+            -0.008022967725992203,
+            -0.04039255902171135,
+            -0.035825639963150024,
+            -0.003938197158277035,
+            -0.025843214243650436,
+            -0.035856496542692184,
+            -0.005191786214709282,
+            -0.003957483451813459,
+            -0.004836923908442259,
+            0.01324561145156622,
+            -0.02542663738131523,
+            -0.002941112034022808,
+            0.0429537370800972,
+            0.02146144025027752,
+            -0.0017280238680541515,
+            -0.033418748527765274,
+            -0.014271626248955727,
+            -0.03443704918026924,
+            -0.036597076803445816,
+            0.020319709554314613,
+            -0.0013972693122923374,
+            0.020273422822356224,
+            -0.021260865032672882,
+            -0.009249556809663773,
+            0.016061365604400635,
+            0.018283110111951828,
+            0.0148656340315938,
+            0.007347958628088236,
+            -0.039806265383958817,
+            -0.030641566962003708,
+            -0.01643165573477745,
+            -0.009851278737187386,
+            -0.021692872047424316,
+            -0.029283834621310234,
+            0.012682461179792881,
+            0.040917135775089264,
+            -0.02468605525791645,
+            0.005211072042584419,
+            0.015467356890439987,
+            -0.021708300337195396,
+            -0.057395078241825104,
+            0.03289416804909706,
+            -0.015220495872199535,
+            -0.02062828466296196,
+            0.0032169015612453222,
+            0.01279046293348074,
+            0.005014355294406414,
+            -0.022047733888030052,
+            0.015251354314386845,
+            0.01982598938047886,
+            0.009974709711968899,
+            -0.017743103206157684,
+            0.019285980612039566,
+            -0.03193758428096771,
+            0.02479405701160431,
+            0.0015737360808998346,
+            -0.022711170837283134,
+            -0.001737666898407042,
+            -0.06436888873577118,
+            -0.003911197185516357,
+            0.0006523483316414058,
+            0.020227137953042984,
+            0.03357303515076637,
+            -0.003241973463445902,
+            0.01581450365483761,
+            0.03369646519422531,
+            0.02561178244650364,
+            -0.0017135593807324767,
+            0.0020906003192067146,
+            -0.01655508577823639,
+            0.02309689112007618,
+            0.018622543662786484,
+            0.05415503680706024,
+            -0.01823682337999344,
+            -0.008370115421712399,
+            -0.017496241256594658,
+            -0.014634202234447002,
+            0.03755366429686546,
+            -0.0181133933365345,
+            -0.005079927388578653,
+            0.0010925508104264736,
+            -0.04853895679116249,
+            0.016771089285612106,
+            0.011239869520068169,
+            0.014209911227226257,
+            0.017588814720511436,
+            -0.00450520496815443,
+            0.013484758324921131,
+            -0.022587740793824196,
+            -0.017588814720511436,
+            -0.013253326527774334,
+            0.054031603038311005,
+            -0.07307072728872299,
+            -0.04116399958729744,
+            0.05032869800925255,
+            -0.02803410217165947,
+            0.02806496061384678,
+            0.003409761469811201,
+            0.024207763373851776,
+            -0.05208757892251015,
+            -0.013531044125556946,
+            0.02883639931678772,
+            0.014989065006375313,
+            -0.03465305268764496,
+            -0.0110624385997653,
+            -0.03378903865814209,
+            -0.051470424979925156,
+            -0.044002894312143326,
+            -0.022865459322929382,
+            0.009843564592301846,
+            0.006009511649608612,
+            -0.025102633982896805,
+            -0.012613032013177872,
+            -0.01561393029987812,
+            -0.021677441895008087,
+            0.00471735093742609,
+            -0.027401521801948547,
+            -0.03551706299185753,
+            -0.012512744404375553,
+            0.022896315902471542,
+            -0.0012661246582865715,
+            0.01993398927152157,
+            -0.017635101452469826,
+            -0.006931381765753031,
+            -0.024161476641893387,
+            -0.0010607289150357246,
+            0.03675136715173721,
+            0.009820421226322651,
+            0.04668750241398811,
+            -0.0010134782642126083,
+            0.009311271831393242,
+            -0.03354217857122421,
+            -0.000637883844319731,
+            0.008439545519649982,
+            0.005176357459276915,
+            -0.04934125393629074,
+            -0.003911197185516357,
+            -0.04764408618211746,
+            0.01885397545993328,
+            0.002808038843795657,
+            -0.01198045164346695,
+            -0.002281531458720565,
+            -0.012952464632689953,
+            -0.0004305595357436687,
+            0.026969516649842262,
+            -0.0102987140417099,
+            -0.0075948191806674,
+            -0.009357557632029057,
+            -0.034221045672893524,
+            0.03486905246973038,
+            -0.014348769560456276,
+            -0.017743103206157684,
+            -0.008732692338526249,
+            -0.019270552322268486,
+            -0.018375683575868607,
+            0.011425015516579151,
+            0.0012391243362799287,
+            -0.00016139955550897866,
+            -0.01235074270516634,
+            -0.005083784461021423,
+            0.02152315527200699,
+            0.01800539158284664,
+            -0.0023509610909968615,
+            0.03273988142609596,
+            -0.04175029322504997,
+            0.019100835546851158,
+            0.0014223410980775952,
+            -0.015637073665857315,
+            0.03709080070257187,
+            0.01435648463666439,
+            -0.02562721259891987,
+            0.001886168960481882,
+            0.03397418558597565,
+            -0.0020057419314980507,
+            0.010753863491117954,
+            -0.01074614841490984,
+            -0.011695018969476223,
+            0.025133490562438965,
+            -0.016678515821695328,
+            -0.0015814504586160183,
+            -0.021245436742901802,
+            0.014171338640153408,
+            -0.014873349107801914,
+            0.029052402824163437,
+            0.009396130219101906,
+            0.022803744301199913,
+            0.020998576655983925,
+            -0.013862763531506062,
+            0.006129084620624781,
+            0.019147122278809547,
+            -0.03456047922372818,
+            0.013099038042128086,
+            -0.009141555055975914,
+            0.004084771033376455,
+            -0.01147901639342308,
+            -0.007228385657072067,
+            0.012096167542040348,
+            -0.015073922462761402,
+            0.014564773067831993,
+            -0.013986193574965,
+            0.024207763373851776,
+            0.039219971746206284,
+            -0.026599224656820297,
+            0.007664248812943697,
+            0.001095443731173873,
+            0.0008090469054877758,
+            0.03480733931064606,
+            0.004443489946424961,
+            -0.03277073800563812,
+            -0.006946810521185398,
+            -0.0072823865339159966,
+            0.027756383642554283,
+            -0.026676369830965996,
+            -0.01720309443771839,
+            0.03807824105024338,
+            0.030795855447649956,
+            -0.002358675468713045,
+            -0.03227701783180237,
+            -0.019208837300539017,
+            0.003047185018658638,
+            0.01893111877143383,
+            -0.025735212489962578,
+            -0.029422692954540253,
+            -0.009619846940040588,
+            0.011841592378914356,
+            0.017511671409010887,
+            -0.00911841168999672,
+            -0.01280589122325182,
+            0.0030992571264505386,
+            -0.006811808329075575,
+            0.010784721001982689,
+            -0.004269916098564863,
+            0.0074598174542188644,
+            -0.05878366902470589,
+            -0.006645949091762304,
+            0.011232155375182629,
+            0.04067027568817139,
+            0.00450134789571166,
+            0.03452961891889572,
+            -0.06273344159126282,
+            -0.0036122642923146486,
+            -0.0038880538195371628,
+            -0.01735738292336464,
+            -0.015698788687586784,
+            -0.016832804307341576,
+            -0.004455061629414558,
+            -0.013847334310412407,
+            0.02058199979364872,
+            0.024500910192728043,
+            0.012520458549261093,
+            0.014742203988134861,
+            -0.0058976528234779835,
+            0.042490873485803604,
+            -0.009372986853122711,
+            -0.014186767861247063,
+            0.00041947010322473943,
+            -0.002705823164433241,
+            0.004146486055105925,
+            0.007818536832928658,
+            -0.006572662387043238,
+            0.035825639963150024,
+            0.020890574902296066,
+            0.008832979016005993,
+            0.00911841168999672,
+            0.005693221464753151,
+            0.0458543486893177,
+            0.027432380244135857,
+            -0.045761775225400925,
+            -0.013153038918972015,
+            -0.0098589938133955,
+            -0.017758531495928764,
+            -0.01808253675699234,
+            0.006534090265631676,
+            -0.029823841527104378,
+            -0.00904898252338171,
+            0.008262114599347115,
+            0.019301410764455795,
+            0.015876220539212227,
+            0.016801945865154266,
+            -0.01825225166976452,
+            -0.04684179276227951,
+            -0.01720309443771839,
+            -0.004208201076835394,
+            0.0246089119464159,
+            -0.008030682802200317,
+            -0.0034599050413817167,
+            -0.003587192390114069,
+            -4.5985012548044324e-05,
+            0.01820596680045128,
+            0.01360818836838007,
+            -0.0073248157277703285,
+            -0.010831006802618504,
+            0.012960179708898067,
+            0.037461090832948685,
+            0.028543252497911453,
+            -0.02294260263442993,
+            0.012620746158063412,
+            0.01393219269812107,
+            -0.022850031033158302,
+            0.02237173728644848,
+            0.04051598906517029,
+            0.020196279510855675,
+            0.0015100922901183367,
+            -0.02641407959163189,
+            0.014950492419302464,
+            -0.01068443339318037,
+            0.014773061498999596,
+            0.0029681124724447727,
+            0.009226413443684578,
+            0.025997502729296684,
+            0.02632150799036026,
+            -0.0033962612506002188,
+            -0.020288852974772453,
+            0.023220321163535118,
+            0.0009445309406146407,
+            -0.0037183372769504786,
+            0.012173310853540897,
+            0.030703281983733177,
+            -0.009897565469145775,
+            0.004424204118549824,
+            -0.03403589874505997,
+            0.006742379162460566,
+            0.011371014639735222,
+            0.02465519867837429,
+            -0.004300773609429598,
+            0.004293059464544058,
+            0.004624778404831886,
+            0.009743277914822102,
+            -0.02798781543970108,
+            -0.026815228164196014,
+            -0.007201385218650103,
+            0.016508799046278,
+            -0.016092222183942795,
+            0.02301974780857563,
+            -0.014695917256176472,
+            0.017851104959845543,
+            -0.03471476584672928,
+            0.04144171625375748,
+            0.024392908439040184,
+            -0.03372732549905777,
+            -0.020134564489126205,
+            -0.013376756571233273,
+            -0.05677792802453041,
+            0.011671875603497028,
+            0.03604163974523544,
+            -0.010074996389448643,
+            -0.0062910872511565685,
+            -0.017449956387281418,
+            0.007313244044780731,
+            -0.013299612328410149,
+            -0.014773061498999596,
+            0.007475246209651232,
+            0.011810734868049622,
+            0.018452826887369156,
+            -0.008755835704505444,
+            0.029006116092205048,
+            0.012775033712387085,
+            -0.03511591628193855,
+            0.016508799046278,
+            0.030101560056209564,
+            0.010098139755427837,
+            0.024917487055063248,
+            -0.001642201328650117,
+            -0.010599575005471706,
+            -0.007598676718771458,
+            0.03597992658615112,
+            0.022479739040136337,
+            0.01562164444476366,
+            0.010838720947504044,
+            -0.010591860860586166,
+            0.026043789461255074,
+            -0.012235026806592941,
+            -0.0222020223736763,
+            0.00449749082326889,
+            -0.018375683575868607,
+            -0.00942698772996664,
+            -0.00900269579142332,
+            0.018375683575868607,
+            -0.001383769093081355,
+            -0.010429858230054379,
+            0.010260142385959625,
+            0.0023914617486298084,
+            -0.003322974545881152,
+            -0.006456946488469839,
+            -0.004354774486273527,
+            0.004744351375848055,
+            0.03397418558597565,
+            0.009218699298799038,
+            -0.010476144962012768,
+            0.03687479719519615,
+            0.007386530749499798,
+            -0.027463236823678017,
+            0.01973341591656208,
+            -0.01902369223535061,
+            -0.019949419423937798,
+            -5.614028486888856e-05,
+            0.020844288170337677,
+            -0.012289026752114296,
+            -0.02465519867837429,
+            0.0018485613400116563,
+            -0.009797278791666031,
+            0.007309386972337961,
+            0.022726600989699364,
+            0.00014801989891566336,
+            0.028496965765953064,
+            -0.017048807814717293,
+            -0.016184795647859573,
+            0.024300336837768555,
+            0.011155011132359505,
+            -0.012150168418884277,
+            0.034221045672893524,
+            -0.006279515568166971,
+            0.05955510959029198,
+            -0.0363810732960701,
+            0.003913125488907099,
+            -0.024423766881227493,
+            -0.01806710660457611,
+            -0.014155910350382328,
+            -0.040886279195547104,
+            0.029345549643039703,
+            -0.05350702628493309,
+            0.019579127430915833,
+            0.0058089373633265495,
+            -0.016678515821695328,
+            -0.0376770943403244,
+            0.0015978434821590781,
+            0.017789389938116074,
+            -0.008053826168179512,
+            -0.02808038890361786,
+            0.004204344004392624,
+            -0.0008818514761514962,
+            0.006468518171459436,
+            -0.04273773357272148,
+            0.00906441081315279,
+            0.05292073264718056,
+            -0.03363475203514099,
+            0.010993009433150291,
+            0.009635276161134243,
+            0.0032496878411620855,
+            -0.016987092792987823,
+            0.03613421320915222,
+            0.003791623981669545,
+            -0.003820552956312895,
+            0.02295803278684616,
+            -0.03443704918026924,
+            -0.04474347457289696,
+            0.019316839054226875,
+            -0.0011455873027443886,
+            0.03320274502038956,
+            -0.003004755824804306,
+            0.026938658207654953,
+            0.0062949443235993385,
+            0.023420896381139755,
+            0.027740955352783203,
+            -0.0019594556652009487,
+            0.010954436846077442,
+            -0.00828525796532631,
+            -0.014402770437300205,
+            -0.010445287451148033,
+            -0.02954612299799919,
+            0.0102987140417099,
+            0.027000373229384422,
+            -0.025133490562438965,
+            0.02048942632973194,
+            -0.03613421320915222,
+            0.007749107200652361,
+            -0.017527099698781967,
+            0.011016152799129486,
+            0.051532141864299774,
+            0.047335512936115265,
+            0.0026981087867170572,
+            -0.025287779048085213,
+            0.009642990306019783,
+            0.01743452623486519,
+            -0.024932917207479477,
+            0.031814154237508774,
+            -0.0024628196842968464,
+            -0.007976681925356388,
+            -0.013099038042128086,
+            0.031845010817050934,
+            -0.0038282673340290785,
+            -0.0073711019940674305,
+            -0.000637883844319731,
+            -0.01242017187178135,
+            0.01324561145156622,
+            -0.006090512964874506,
+            -0.01657051593065262,
+            -0.017002521082758904,
+            0.012620746158063412,
+            -0.012582174502313137,
+            0.012065310031175613,
+            0.013014179654419422,
+            0.03388161212205887,
+            0.013407614082098007,
+            -0.019116263836622238,
+            0.023405466228723526,
+            0.02638322301208973,
+            0.0657266229391098,
+            -0.012952464632689953,
+            0.012435601092875004,
+            0.007614105474203825,
+            -0.021168293431401253,
+            -0.002835039282217622,
+            -0.02895982936024666,
+            0.0015014136442914605,
+            -0.028543252497911453,
+            -0.008917837403714657,
+            -0.034190189093351364,
+            -0.023606041446328163,
+            -0.03770795091986656,
+            0.01110101118683815,
+            -0.001489842077717185,
+            -0.026198076084256172,
+            0.01902369223535061,
+            -0.006495518609881401,
+            -0.010753863491117954,
+            -0.013477043248713017,
+            -0.026136361062526703,
+            0.010090425610542297,
+            -0.029345549643039703,
+            0.003562120720744133,
+            0.006237086374312639,
+            0.012119310908019543,
+            0.00829297211021185,
+            0.019548270851373672,
+            0.00910298340022564,
+            0.018468255177140236,
+            0.0013934121234342456,
+            -0.02726266346871853,
+            -0.026876943185925484,
+            0.010931294411420822,
+            0.011201297864317894,
+            0.014896491542458534,
+            -0.013947621919214725,
+            -0.006807951256632805,
+            0.02894440107047558,
+            0.0014638060238212347,
+            -0.010792435146868229,
+            -0.022016875445842743,
+            -0.014981349930167198,
+            -0.02153858356177807,
+            -0.025688927620649338,
+            -0.001375090447254479,
+            -0.02298888936638832,
+            -0.00899498164653778,
+            -0.007116526830941439,
+            -0.0008288150420412421,
+            -0.0005269894609227777,
+            0.0005901510012336075,
+            0.01817510835826397,
+            -0.03968283534049988,
+            -0.0009941923199221492,
+            0.0003018256102222949,
+            -0.04705779254436493,
+            0.024130620062351227,
+            -0.004995069000869989,
+            0.013014179654419422,
+            -0.027355235069990158,
+            0.020335137844085693,
+            0.007922681048512459,
+            0.011749019846320152,
+            -0.0015428784536197782,
+            -0.01718766614794731,
+            0.014742203988134861,
+            0.00575107941403985,
+            -0.018730545416474342,
+            0.0016566658159717917,
+            0.028203818947076797,
+            0.016693945974111557,
+            0.00786096602678299,
+            0.014240768738090992,
+            0.01198045164346695,
+            0.005241929553449154,
+            -0.00236060400493443,
+            0.047335512936115265,
+            -0.012967893853783607,
+            0.012204169295728207,
+            0.017527099698781967,
+            -0.005203357897698879,
+            -0.0003975322761107236,
+            0.006510947365313768,
+            -0.004740494303405285,
+            0.0062949443235993385,
+            0.003673979314044118,
+            0.029438121244311333,
+            -0.00906441081315279,
+            0.018452826887369156,
+            0.034961625933647156,
+            -0.014734489843249321,
+            0.0008148326887749135,
+            -0.005461789667606354,
+            0.007020097225904465,
+            0.009704705327749252,
+            -0.014256197027862072,
+            -0.014911920763552189,
+            0.028157532215118408,
+            -0.01199587993323803,
+            0.02953069470822811,
+            -0.004705779254436493,
+            0.014626488089561462,
+            0.03527020290493965,
+            0.00764882005751133,
+            0.004250630270689726,
+            -0.00861697643995285,
+            -0.0078146792948246,
+            0.013315041549503803,
+            -0.021260865032672882,
+            0.011679589748382568,
+            -0.021245436742901802,
+            0.027679240331053734,
+            -0.031690724194049835,
+            -0.005049069877713919,
+            -0.012636174447834492,
+            -0.01112415362149477,
+            0.02474777027964592,
+            0.0017627386841922998,
+            0.0034714764915406704,
+            -0.0057587940245866776,
+            -0.013369042426347733,
+            0.01110872533172369,
+            0.01965627260506153,
+            -0.007166670635342598,
+            0.012960179708898067,
+            0.004632492549717426,
+            -0.030286705121397972,
+            -0.014965921640396118,
+            0.030147844925522804,
+            0.0029777553863823414,
+            -0.021862588822841644,
+            0.014379628002643585,
+            -0.0062910872511565685,
+            -0.004401060752570629,
+            0.036535363644361496,
+            -0.009851278737187386,
+            -0.0074328170157969,
+            -0.0009223520755767822,
+            -0.001847596955485642,
+            -0.001201516599394381,
+            0.02394547499716282,
+            0.021785443648695946,
+            -0.02641407959163189,
+            -0.007999825291335583,
+            -0.014009336940944195,
+            0.007498389575630426,
+            -0.013816476799547672,
+            0.02371404320001602,
+            0.031783297657966614,
+            -0.00948870275169611,
+            0.00596708245575428,
+            0.004536062944680452,
+            0.013199325650930405,
+            -0.01517421007156372,
+            -0.023374609649181366,
+            -0.0017212738748639822,
+            0.005943939555436373,
+            -0.01569107361137867,
+            -0.006746236234903336,
+            -0.04384860768914223,
+            -0.011749019846320152,
+            -0.0026730368845164776,
+            -0.029268404468894005,
+            -0.04011484235525131,
+            -0.031845010817050934,
+            -0.005546648055315018,
+            0.02052028477191925,
+            0.016771089285612106,
+            0.0017193452222272754,
+            0.02386832982301712,
+            -0.005724078975617886,
+            0.005623792298138142,
+            0.021924303844571114,
+            0.00011264218483120203,
+            0.015073922462761402,
+            -0.0006962239276617765,
+            0.038170814514160156,
+            0.014665059745311737,
+            -0.0013162682298570871,
+            -0.05671621486544609,
+            -0.0188385471701622,
+            0.012937036342918873,
+            0.021168293431401253,
+            -0.0041271997615695,
+            0.02139972522854805,
+            -0.005245787091553211,
+            -0.020859716460108757,
+            -0.012543601915240288,
+            0.007035525981336832,
+            0.0037067655939608812,
+            0.007733678445219994,
+            0.0048176380805671215,
+            0.0036103357560932636,
+            0.029345549643039703,
+            0.026753513142466545,
+            0.00048239060561172664,
+            0.008940980769693851,
+            0.04224401339888573,
+            -0.006121370475739241,
+            0.0008119398262351751,
+            -0.00742510287091136,
+            0.022402595728635788,
+            -0.018653400242328644,
+            0.02377575822174549,
+            0.019995706155896187,
+            0.008771263994276524,
+            0.023467181250452995,
+            0.016169367358088493,
+            -0.00515707116574049,
+            -0.009226413443684578,
+            0.032647307962179184,
+            -0.010437573306262493,
+            -0.029098687693476677,
+            -0.007228385657072067,
+            -0.012975607998669147,
+            0.011826164089143276,
+            0.005608363542705774,
+            0.01191873662173748,
+            0.0033789039589464664,
+            0.0006263122195377946,
+            -0.02129172347486019,
+            -0.011949594132602215,
+            -0.00022540490317624062,
+            0.007760678883641958,
+            0.011193583719432354,
+            0.017465384677052498,
+            -0.0034406189806759357,
+            -0.018607115373015404,
+            0.033387891948223114,
+            -0.020782573148608208,
+            0.041565146297216415,
+            0.00011203949543414637,
+            -0.023189464583992958,
+            0.004038484301418066,
+            -0.00030399527167901397,
+            0.023405466228723526,
+            -0.009982423856854439,
+            -0.019069978967308998,
+            0.017943676561117172,
+            0.06066598370671272,
+            0.012273598462343216,
+            -0.016231082379817963,
+            -0.00787639431655407,
+            -0.02970041148364544,
+            0.01806710660457611,
+            0.006171513814479113,
+            0.026630083099007607,
+            0.02800324372947216,
+            0.021662013605237007,
+            -0.0042776307091116905,
+            0.02314317785203457,
+            -0.014626488089561462,
+            0.010530145838856697,
+            -0.004532205406576395,
+            0.02295803278684616,
+            0.0036450503394007683,
+            -0.013500186614692211,
+            0.034282758831977844,
+            0.0021793157793581486,
+            -0.002719323383644223,
+            -0.015590786933898926,
+            0.00575107941403985,
+            0.004875496029853821,
+            -0.00640294561162591,
+            0.02633693628013134,
+            0.03059528023004532,
+            0.003114685881882906,
+            -0.021631157025694847,
+            -0.010993009433150291,
+            0.016801945865154266,
+            0.020319709554314613,
+            0.0015920577570796013,
+            0.016925377771258354,
+            0.013816476799547672,
+            0.033295318484306335,
+            -0.0029989699833095074,
+            -0.0008153148810379207,
+            0.0041271997615695,
+            -0.0058937957510352135,
+            0.005461789667606354,
+            0.011417300440371037,
+            0.034190189093351364,
+            -0.00020033313194289804,
+            -0.018499113619327545,
+            0.009581275284290314,
+            0.011826164089143276,
+            -0.039034824818372726,
+            0.004393346607685089,
+            -0.022510597482323647,
+            0.0006914024124853313,
+            0.00992070883512497,
+            -0.022865459322929382,
+            -0.021615726873278618,
+            0.004046198911964893,
+            0.013029608875513077,
+            0.018360253423452377,
+            0.0156293585896492,
+            -0.019378554075956345,
+            -0.014194482006132603,
+            0.02718551829457283,
+            0.002271888544782996,
+            0.009781849570572376,
+            0.024346621707081795,
+            -0.006599662825465202,
+            0.0015824147267267108,
+            0.019116263836622238,
+            0.004138771444559097,
+            0.03150557726621628,
+            -0.002854325110092759,
+            -0.028311820700764656,
+            0.012466458603739738,
+            0.006210085935890675,
+            -0.009650705382227898,
+            -0.006896666716784239,
+            -0.027694668620824814,
+            -0.015035350807011127,
+            -0.030240418389439583,
+            0.003689408302307129,
+            -0.02305060438811779,
+            -0.0016373797552660108,
+            0.012235026806592941,
+            -0.011471301317214966,
+            0.006437660660594702,
+            -0.013777905143797398,
+            -0.02389918826520443,
+            -0.003689408302307129,
+            -0.0001803239283617586,
+            -0.012975607998669147,
+            -0.03033299185335636,
+            -0.02132258005440235,
+            -0.017835674807429314,
+            -0.014317912049591541,
+            -0.012073024176061153,
+            -0.019532842561602592,
+            -0.0008567796903662384,
+            0.02056656964123249,
+            0.007907251827418804,
+            -0.035085055977106094,
+            0.004208201076835394,
+            -0.004289202392101288,
+            -0.010128997266292572,
+            0.008871551603078842,
+            -0.02135343849658966,
+            -0.00038403208600357175,
+            0.02223287895321846,
+            0.011440443806350231,
+            -0.0165859442204237,
+            -0.004188915248960257,
+            -0.010954436846077442,
+            0.03059528023004532,
+            0.0094192735850811,
+            -0.008223542012274265,
+            -0.0058860816061496735,
+            0.024192335084080696,
+            -0.0023471040185540915,
+            -0.00023818186309654266,
+            -0.0024821057450026274,
+            -0.0009724955889396369,
+            0.0008399044745601714,
+            -0.024269478395581245,
+            0.030857570469379425,
+            0.00450520496815443,
+            0.002742466516792774,
+            -0.024979202076792717,
+            0.024315765127539635,
+            0.02314317785203457,
+            -0.030764997005462646,
+            0.0073749590665102005,
+            0.0012931249802932143,
+            0.010653575882315636,
+            0.00905669666826725,
+            0.0026267506182193756,
+            -0.008809836581349373,
+            0.03221530467271805,
+            -0.025256920605897903,
+            0.008123255334794521,
+            0.012929322198033333,
+            0.040053125470876694,
+            0.02226373739540577,
+            -0.007301672361791134,
+            0.009473273530602455,
+            0.020119136199355125,
+            0.013515615835785866,
+            0.017835674807429314,
+            -0.017665958032011986,
+            -0.01272874791175127,
+            0.0001393412094330415,
+            -0.0038109098095446825,
+            0.010468430817127228,
+            0.0013490543933585286,
+            -0.03155186399817467,
+            0.005631506443023682,
+            0.01067671924829483,
+            -0.007560104597359896,
+            -0.019285980612039566,
+            0.023667756468057632,
+            -0.004019198473542929,
+            0.007020097225904465,
+            -0.003754980629310012,
+            -0.006499375682324171,
+            0.011571588926017284,
+            -0.05936996266245842,
+            -0.020057421177625656,
+            0.008871551603078842,
+            0.0053036450408399105,
+            0.01447220053523779,
+            0.010020995512604713,
+            -0.02963869646191597,
+            -0.007197528146207333,
+            -0.02874382585287094,
+            0.018344825133681297,
+            -0.00987442210316658,
+            0.020057421177625656,
+            0.0221248771995306,
+            -0.008493546396493912,
+            0.0058436524122953415,
+            -0.003068399615585804,
+            0.018637971952557564,
+            -0.012451029382646084,
+            0.022603169083595276,
+            -0.016801945865154266,
+            -0.015004493296146393,
+            -0.00944241601973772,
+            0.011386442929506302,
+            -0.007938110269606113,
+            0.01449534296989441,
+            -0.024238621816039085,
+            -0.02141515351831913,
+            0.007922681048512459,
+            0.013531044125556946,
+            0.014603344723582268,
+            -0.01806710660457611,
+            0.007999825291335583,
+            -0.005616077687591314,
+            0.01105472445487976,
+            0.006152227986603975,
+            0.009712420403957367,
+            -0.0003524513158481568,
+            -0.022649455815553665,
+            -0.008354687131941319,
+            0.007563961669802666,
+            0.0008644941262900829,
+            0.004250630270689726,
+            -0.008794407360255718,
+            -0.029021544381976128,
+            0.007787679322063923,
+            -0.014896491542458534,
+            -0.019347695633769035,
+            -0.01653965748846531,
+            -0.009689277037978172,
+            -0.010838720947504044,
+            0.0018543470650911331,
+            0.02389918826520443,
+            -0.020072849467396736,
+            -0.010607290081679821,
+            0.01323789730668068,
+            -0.013083609752357006,
+            0.016153937205672264,
+            0.0035949070006608963,
+            -0.007629534229636192,
+            0.00552350515499711,
+            0.0003452190721873194,
+            0.03684393689036369,
+            0.0019903131760656834,
+            -0.02152315527200699,
+            0.01312218140810728,
+            -0.015498214401304722,
+            0.02382204309105873,
+            0.007976681925356388,
+            -0.0021349580492824316,
+            0.011571588926017284,
+            0.024886630475521088,
+            0.0027366806752979755,
+            0.020443139597773552,
+            -0.011008437722921371,
+            -0.029484407976269722,
+            0.007961252704262733,
+            -0.008817550726234913,
+            -0.01991856098175049,
+            -0.0198722742497921,
+            0.003754980629310012,
+            -0.010931294411420822,
+            0.023575183004140854,
+            -0.022649455815553665,
+            -0.0003042363387066871,
+            0.00038089812733232975,
+            0.0015814504586160183,
+            -0.014888777397572994,
+            0.013585045002400875,
+            0.007968967780470848,
+            -0.017542527988553047,
+            0.00779925100505352,
+            -0.01652422919869423,
+            -0.02130715176463127,
+            -2.7196247174288146e-05,
+            -0.008516688831150532,
+            -0.0022622456308454275,
+            -0.017064236104488373,
+            -0.0030201845802366734,
+            0.00033726359833963215,
+            -0.0006306516006588936,
+            0.016956234350800514,
+            0.019162550568580627,
+            -0.007745250128209591,
+            0.01894654706120491,
+            0.003976769279688597,
+            0.02209402061998844,
+            0.0031648294534534216,
+            -0.006985382176935673,
+            -0.022402595728635788,
+            0.02642950788140297,
+            -0.02567349746823311,
+            -0.0024338907096534967,
+            0.01235074270516634,
+            -0.014904206618666649,
+            -0.0012005523312836885,
+            -0.0297312680631876,
+            -0.004678778816014528,
+            -0.0014011265011504292,
+            -0.0018177037127315998,
+            0.04551491513848305,
+            -0.003259330755099654,
+            -0.012967893853783607,
+            -0.0131067531183362,
+            0.0006841701688244939,
+            0.0066112345084548,
+            0.005006640683859587,
+            0.01640079915523529,
+            -0.01154073141515255,
+            -0.02391461655497551,
+            0.016030507162213326,
+            -0.005855224095284939,
+            0.01647794246673584,
+            0.020196279510855675,
+            0.0026286791544407606,
+            -0.03307931497693062,
+            0.012505030259490013,
+            -0.01905454881489277,
+            -0.00944241601973772,
+            0.019270552322268486,
+            0.02801867388188839,
+            -0.00867869146168232,
+            -0.01072300598025322,
+            0.011633303947746754,
+            -0.000418505776906386,
+            -0.01066900510340929,
+            -0.02479405701160431,
+            -0.009403844363987446,
+            -0.013762475922703743,
+            0.013731618411839008,
+            0.006113655865192413,
+            -0.02065914310514927,
+            0.006483946926891804,
+            -0.006414517294615507,
+            0.01230445597320795,
+            -0.01154073141515255,
+            0.003432904602959752,
+            0.018514541909098625,
+            0.02150772698223591,
+            -0.0230814628303051,
+            0.025827785953879356,
+            -0.027648381888866425,
+            0.021075719967484474,
+            -0.0008799228817224503,
+            0.026707226410508156,
+            -0.029484407976269722,
+            -0.018313968554139137,
+            -0.013006465509533882,
+            -0.011378728784620762,
+            -0.025195205584168434,
+            -0.006078941281884909,
+            0.01441048551350832,
+            0.013970764353871346,
+            0.009311271831393242,
+            0.026799799874424934,
+            -0.012096167542040348,
+            0.018653400242328644,
+            0.02064371481537819,
+            -0.01193416491150856,
+            0.005681650247424841,
+            0.011733590625226498,
+            -0.009558131918311119,
+            0.012813606299459934,
+            0.001909312093630433,
+            0.010445287451148033,
+            -0.033264461904764175,
+            0.022078590467572212,
+            0.014819348230957985,
+            -0.0037299087271094322,
+            -0.03872625157237053,
+            -0.0016798089491203427,
+            -0.020057421177625656,
+            -0.02315860614180565,
+            -0.0014310197439044714,
+            0.011957308277487755,
+            0.028682110831141472,
+            0.018622543662786484,
+            -0.036689650267362595,
+            -0.000610401330050081,
+            -0.01579907536506653,
+            -0.011232155375182629,
+            -0.014024765230715275,
+            0.012921607121825218,
+            -0.009018125012516975,
+            0.014958206564188004,
+            -0.0004170593456365168,
+            0.002379890065640211,
+            -0.003363475203514099,
+            -0.01231217011809349,
+            0.016138508915901184,
+            0.010275570675730705,
+            -0.009457845240831375,
+            0.010429858230054379,
+            0.013816476799547672,
+            0.0026074645575135946,
+            -0.004644064232707024,
+            -0.012597602792084217,
+            -0.02485577203333378,
+            0.011617874726653099,
+            -0.0018205966334789991,
+            0.007498389575630426,
+            -0.009797278791666031,
+            0.0034676194190979004,
+            -0.047366369515657425,
+            0.003726051654666662,
+            0.0032265447080135345,
+            0.030641566962003708,
+            0.011749019846320152,
+            0.018622543662786484,
+            0.014919634908437729,
+            0.0005795437609776855,
+            0.010591860860586166,
+            0.02127629518508911,
+            0.03786223754286766,
+            0.00537307420745492,
+            0.014896491542458534,
+            -5.394048002926866e-06,
+            -0.007563961669802666,
+            0.0074559603817760944,
+            -0.006125227548182011,
+            -0.02062828466296196,
+            -0.006360516417771578,
+            -0.00020177957776468247,
+            0.0008881194517016411,
+            -0.00860926229506731,
+            0.004308488219976425,
+            -0.017465384677052498,
+            -0.010838720947504044,
+            -0.02794152870774269,
+            0.00216581579297781,
+            -0.025118062272667885,
+            -0.015891648828983307,
+            -0.016308225691318512,
+            -0.010869579389691353,
+            -0.003953625913709402,
+            0.022772885859012604,
+            -0.0049063535407185555,
+            -0.033418748527765274,
+            -0.0037029082886874676,
+            0.02727809175848961,
+            -0.008269828744232655,
+            -0.010530145838856697,
+            0.011725876480340958,
+            0.010059568099677563,
+            -0.008022967725992203,
+            0.01362361665815115,
+            0.010229283943772316,
+            -0.009357557632029057,
+            -0.0013104823883622885,
+            -0.01560621615499258,
+            0.02963869646191597,
+            0.017110522836446762,
+            0.002422319259494543,
+            -0.004528348334133625,
+            -0.017449956387281418,
+            -0.0016084507806226611,
+            -0.0013914835872128606,
+            0.02480948530137539,
+            -0.021569442003965378,
+            0.027756383642554283,
+            -0.0031474721617996693,
+            -0.02227916568517685,
+            0.013747047632932663,
+            0.02055114135146141,
+            -0.02721637673676014,
+            0.0010732648661360145,
+            -0.006588091142475605,
+            0.015505928546190262,
+            -0.008223542012274265,
+            -0.0057665081694722176,
+            -0.010622718371450901,
+            0.007517675403505564,
+            0.013214753940701485,
+            -0.01967170089483261,
+            0.017542527988553047,
+            0.0110624385997653,
+            0.01280589122325182,
+            -0.02547292411327362,
+            0.013307327404618263,
+            -0.006213943008333445,
+            -0.01275189034640789,
+            -0.0020751715637743473,
+            -0.004825352691113949,
+            0.017573386430740356,
+            0.006059655454009771,
+            0.002231388119980693,
+            -0.00758324796333909,
+            0.003984483890235424,
+            -0.006244800519198179,
+            0.010514716617763042,
+            0.011432729661464691,
+            -0.020273422822356224,
+            -0.004377917852252722,
+            -0.013006465509533882,
+            -0.009349843487143517,
+            0.003344189142808318,
+            -0.007178242318332195,
+            0.03650450333952904,
+            0.00986670795828104,
+            0.004296916536986828,
+            0.02231002226471901,
+            -0.011139582842588425,
+            -0.012913892976939678,
+            0.01660137251019478,
+            0.09436244517564774,
+            3.441342414589599e-05,
+            -0.003969055134803057,
+            0.002773324027657509,
+            -0.010476144962012768,
+            -0.0008866729913279414,
+            -0.01235074270516634,
+            -0.018468255177140236,
+            0.008123255334794521,
+            -0.00514549994841218,
+            -0.021739158779382706,
+            -0.009149269200861454,
+            -0.020011134445667267,
+            0.017141379415988922,
+            0.003322974545881152,
+            0.012242740951478481,
+            0.013939906843006611,
+            0.01905454881489277,
+            0.0031899013556540012,
+            0.014819348230957985,
+            -0.009519560262560844,
+            0.007243814412504435,
+            -0.011347871273756027,
+            0.007263100706040859,
+            0.008123255334794521,
+            0.009210984222590923,
+            0.006244800519198179,
+            0.015012207441031933,
+            0.012003595009446144,
+            0.029006116092205048,
+            0.010283284820616245,
+            -0.0012179097393527627,
+            -0.005049069877713919,
+            -0.016693945974111557,
+            -0.003693265374749899,
+            0.00904898252338171,
+            -0.01984141767024994,
+            -0.022634027525782585,
+            -0.013693046756088734,
+            -0.002790681552141905,
+            0.0019382410682737827,
+            -0.008940980769693851,
+            0.023405466228723526,
+            0.026954086497426033,
+            -0.006800237111747265,
+            0.004269916098564863,
+            -0.007479103282094002,
+            -0.0011108724866062403,
+            -0.00866326317191124,
+            -0.010514716617763042,
+            -0.0013066251995041966,
+            -0.0013596616918221116,
+            0.005295930430293083,
+            -0.021939732134342194,
+            -0.013677617534995079,
+            -0.004107913933694363,
+            0.009766421280801296,
+            -0.00040548774995841086,
+            0.0024956059642136097,
+            0.011363300494849682,
+            -0.01393219269812107,
+            -0.004944925662130117,
+            0.00558136310428381,
+            -0.009550417773425579,
+            -0.008385544642806053,
+            0.006275658495724201,
+            0.012134739197790623,
+            -0.0010173354530707002,
+            -0.011725876480340958,
+            -0.012613032013177872,
+            -0.010453001596033573,
+            0.006109798792749643,
+            0.0221248771995306,
+            -0.007548532914370298,
+            0.008501260541379452,
+            -0.008424116298556328,
+            0.003648907644674182,
+            -0.017727673053741455,
+            0.003157115075737238,
+            -0.0038880538195371628,
+            -0.0021542441099882126,
+            0.00992842298001051,
+            -0.006607376970350742,
+            -0.00817725621163845,
+            0.00011372702283551916,
+            0.0006518661393783987,
+            -0.007228385657072067,
+            0.02132258005440235,
+            0.028203818947076797,
+            0.029931843280792236,
+            0.002574678510427475,
+            -0.02629064954817295,
+            0.02312774769961834,
+            -0.009681562893092632,
+            -0.009573561139404774,
+            0.009156984277069569,
+            0.0005901510012336075,
+            -0.009411558508872986,
+            -0.01896197721362114,
+            -0.002981612691655755,
+            0.005230357870459557,
+            0.019162550568580627,
+            0.00224681687541306,
+            -0.0018312039319425821,
+            -0.022880887612700462,
+            0.003407832933589816,
+            0.02727809175848961,
+            0.0030780425295233727,
+            -0.009789563715457916,
+            -0.008339257910847664,
+            0.020813431590795517,
+            -0.003297902876511216,
+            0.004624778404831886,
+            0.003791623981669545,
+            -0.008339257910847664,
+            -0.003342260606586933,
+            -0.004941068589687347,
+            0.0026518225204199553,
+            -0.01805167831480503,
+            0.009758706204593182,
+            0.01116272620856762,
+            -0.010499288327991962,
+            0.004215915687382221,
+            -0.012705604545772076,
+            0.006210085935890675,
+            -0.0004380328464321792,
+            0.0077761076390743256,
+            -0.0015698788920417428,
+            -0.012289026752114296,
+            -0.02971583977341652,
+            -0.01185702160000801,
+            -0.01817510835826397,
+            -0.014634202234447002,
+            0.0007473317673429847,
+            0.02784895710647106,
+            -0.004266059026122093,
+            0.007563961669802666,
+            0.001351947314105928,
+            -0.009974709711968899,
+            0.012188740074634552,
+            -0.020026562735438347,
+            0.01480391900986433,
+            -0.013639045879244804,
+            0.013091323897242546,
+            0.015451927669346333,
+            0.01797453500330448,
+            -0.009396130219101906,
+            0.0004013894940726459,
+            -0.0020983146969228983,
+            -0.008100111968815327,
+            -0.00537307420745492,
+            -0.012813606299459934,
+            -0.008053826168179512,
+            -0.017635101452469826,
+            0.015328497625887394,
+            0.024099761620163918,
+            0.0009990137768909335,
+            0.01728023961186409,
+            0.0008259221212938428,
+            -1.7749129256117158e-05,
+            -0.01111643947660923,
+            -0.004092485178261995,
+            0.014348769560456276,
+            0.005068355705589056,
+            -0.025210633873939514,
+            -0.0036334788892418146,
+            0.01652422919869423,
+            0.006102084647864103,
+            -0.01393219269812107,
+            0.013469329103827477,
+            -0.003799338359385729,
+            -0.0021812445484101772,
+            -0.01355418749153614,
+            -0.0009030660730786622,
+            0.007675820495933294,
+            0.0052727870643138885,
+            -0.021122006699442863,
+            -0.008786693215370178,
+            0.0032496878411620855,
+            0.012435601092875004,
+            -0.004258344415575266,
+            0.00309347128495574,
+            0.01885397545993328,
+            -0.00536150299012661,
+            0.03465305268764496,
+            -0.012520458549261093,
+            -0.014503058046102524,
+            0.0020713144913315773,
+            0.020134564489126205,
+            -0.0062563722021877766,
+            -0.0007400995236821473,
+            0.011340157128870487,
+            0.0005506147863343358,
+            -0.008161826990544796,
+            0.02974669821560383,
+            0.006040369160473347,
+            0.01738824136555195,
+            -0.0010877293534576893,
+            0.014595630578696728,
+            -0.001114729675464332,
+            0.0002372175658820197,
+            -0.0009802100248634815,
+            -0.01035271491855383,
+            -0.005068355705589056,
+            -0.020026562735438347,
+            -0.006009511649608612,
+            -0.020273422822356224,
+            0.029160402715206146,
+            0.012389314360916615,
+            0.007494532503187656,
+            -0.02638322301208973,
+            0.002289246069267392,
+            0.006441517733037472,
+            -0.007170527707785368,
+            -0.006900524254888296,
+            -0.012204169295728207,
+            -0.022032305598258972,
+            0.009318985976278782,
+            0.007259243167936802,
+            0.010568717494606972,
+            0.004250630270689726,
+            -0.010182998143136501,
+            0.0032034015748649836,
+            -0.018529970198869705,
+            0.012705604545772076,
+            0.0024801772087812424,
+            0.003133971942588687,
+            0.0016942734364420176,
+            -0.0034483333583921194,
+            0.014665059745311737,
+            -3.730631942744367e-05,
+            0.011039295233786106,
+            0.014094195328652859,
+            0.01111643947660923,
+            0.007976681925356388,
+            0.004401060752570629,
+            0.010576431639492512,
+            -0.00816954206675291,
+            -0.030734140425920486,
+            -0.0264449380338192,
+            0.004080913495272398,
+            -0.01110872533172369,
+            -0.009627562016248703,
+            0.007984396070241928,
+            0.011139582842588425,
+            0.01069214753806591,
+            -0.016323653981089592,
+            -0.005666221491992474,
+            -0.0043200599029660225,
+            -2.2917169189895503e-05,
+            -0.011787591502070427,
+            -0.0007203313871286809,
+            0.012157882563769817,
+            0.002040456747636199,
+            0.006491661071777344,
+            0.01894654706120491,
+            -0.00906441081315279,
+            -0.007826250977814198,
+            0.01567564532160759,
+            -0.0074328170157969,
+            -0.019625414162874222,
+            -0.008910123258829117,
+            0.016215652227401733,
+            -0.008215827867388725,
+            0.011286156252026558,
+            0.009565846994519234,
+            0.010491574183106422,
+            -0.016632230952382088,
+            -0.007999825291335583,
+            0.02804953046143055,
+            0.03230787441134453,
+            -0.010807863436639309,
+            -0.002273817081004381,
+            0.002725109225139022,
+            -0.012127025052905083,
+            0.021122006699442863,
+            0.00860926229506731,
+            -0.010113568976521492,
+            0.016987092792987823,
+            0.0026903944090008736,
+            0.008138683624565601,
+            -0.0036238357424736023,
+            0.0012304455740377307,
+            -0.017604243010282516,
+            -0.018375683575868607,
+            -0.00948098860681057,
+            0.009164698421955109,
+            0.006742379162460566,
+            0.0073749590665102005,
+            8.919765969039872e-05,
+            0.011093296110630035,
+            -0.011332442983984947,
+            0.005604506004601717,
+            0.0008119398262351751,
+            0.027740955352783203,
+            -0.029885556548833847,
+            0.019903132691979408,
+            -0.003267045132815838,
+            -0.0018283110111951828,
+            -0.013585045002400875,
+            -0.012844463810324669,
+            -0.020828859880566597,
+            -0.011209012009203434,
+            0.014973635785281658,
+            0.016076793894171715,
+            -0.0005472397315315902,
+            -0.006908238399773836,
+            -0.002619036240503192,
+            0.0057973661459982395,
+            -0.006861952133476734,
+            0.002761752577498555,
+            -0.0017646672204136848,
+            0.0014956278027966619,
+            -0.0020751715637743473,
+            -0.015559929423034191,
+            0.007444388698786497,
+            -0.003652764717116952,
+            -0.0015669859712943435,
+            -0.018761401996016502,
+            0.03604163974523544,
+            -0.013060466386377811,
+            0.00904126837849617,
+            0.014155910350382328,
+            0.0018128822557628155,
+            -0.011471301317214966,
+            -0.013777905143797398,
+            0.011216727085411549,
+            -0.02149229682981968,
+            -0.021955160424113274,
+            -0.004096342250704765,
+            0.0223563089966774,
+            -0.015760503709316254,
+            -0.004046198911964893,
+            0.019532842561602592,
+            -0.006271800957620144,
+            -0.009658419527113438,
+            -0.022680314257740974,
+            -0.016231082379817963,
+            -1.1903066479135305e-05,
+            -0.016801945865154266,
+            -0.017912819981575012,
+            -0.017079664394259453,
+            -0.007864823564887047,
+            -0.01970255747437477,
+            -0.0006914024124853313,
+            0.0033924041781574488,
+            -0.01805167831480503,
+            -0.022695742547512054,
+            0.01275189034640789,
+            0.00818497035652399,
+            -0.0026306079234927893,
+            0.008524403907358646,
+            0.00596322538331151,
+            0.0027231804560869932,
+            -2.3640393919777125e-05,
+            0.007035525981336832,
+            0.02892897091805935,
+            0.017017949372529984,
+            0.006599662825465202,
+            0.017002521082758904,
+            -0.0062988013960421085,
+            -0.01029099989682436,
+            -0.010321857407689095,
+            -0.004169628955423832,
+            0.010445287451148033,
+            -0.019964847713708878,
+            0.033357031643390656,
+            -0.006202371325343847,
+            -0.018730545416474342,
+            -0.006410660222172737,
+            0.00987442210316658,
+            -0.010476144962012768,
+            -0.019532842561602592,
+            -0.007922681048512459,
+            -0.006885095499455929,
+            0.016092222183942795,
+            -0.0032901884987950325,
+            0.00898726750165224,
+            0.010113568976521492,
+            -0.015421070158481598,
+            0.010823292657732964,
+            -0.006240943446755409,
+            -0.014271626248955727,
+            0.025195205584168434,
+            0.022448882460594177,
+            -0.005064498633146286,
+            -0.007826250977814198,
+            -0.002401104662567377,
+            0.021245436742901802,
+            0.008940980769693851,
+            -0.0003459422732703388,
+            0.009172412566840649,
+            0.005608363542705774,
+            0.003905411111190915,
+            0.0024608911480754614,
+            0.001322053954936564,
+            0.011224441230297089,
+            0.03311017155647278,
+            0.030147844925522804,
+            -0.008046111091971397,
+            -0.002551535377278924,
+            0.01528992597013712,
+            0.00899498164653778,
+            0.028280962258577347,
+            -0.0031455436255782843,
+            -0.035023342818021774,
+            0.001998027553781867,
+            -0.008447259664535522,
+            0.014703632332384586,
+            0.0006648842245340347,
+            0.015876220539212227,
+            0.0017540600383654237,
+            -0.0014879134250804782,
+            0.018514541909098625,
+            0.019316839054226875,
+            0.00617922842502594,
+            -0.017635101452469826,
+            -0.01312989555299282,
+            0.008740406483411789,
+            0.006514804437756538,
+            -0.005685507319867611,
+            -0.012327599339187145,
+            -0.00015826558228582144,
+            0.029808413237333298,
+            -0.010198426432907581,
+            -0.017449956387281418,
+            0.01242017187178135,
+            0.011525302194058895,
+            0.007787679322063923,
+            -0.025951215997338295,
+            0.004169628955423832,
+            0.008354687131941319,
+            0.014819348230957985,
+            -0.00899498164653778,
+            -0.015004493296146393,
+            -0.000610401330050081,
+            0.011193583719432354,
+            -0.016261938959360123,
+            0.001589164836332202,
+            0.02218659222126007,
+            -0.02053571306169033,
+            0.03122786059975624,
+            -0.010059568099677563,
+            0.017804818227887154,
+            0.003477262333035469,
+            0.020088277757167816,
+            0.006495518609881401,
+            0.0012304455740377307,
+            0.004161914810538292,
+            0.013885905966162682,
+            -0.003245830535888672,
+            0.011795305646955967,
+            -0.01485020574182272,
+            -0.004536062944680452,
+            0.0003758355451282114,
+            -0.0021581011824309826,
+            -0.0024512482341378927,
+            0.008215827867388725,
+            0.02226373739540577,
+            -0.01581450365483761,
+            0.0015843433793634176,
+            -0.005284358747303486,
+            0.006800237111747265,
+            0.023312894627451897,
+            -0.011903307400643826,
+            -0.009519560262560844,
+            0.012890749610960484,
+            0.019085407257080078,
+            -0.0053537883795797825,
+            -0.027478665113449097,
+            -0.0015043065650388598,
+            0.014680488966405392,
+            0.017789389938116074,
+            -0.0031821869779378176,
+            0.036658793687820435,
+            -0.013060466386377811,
+            -0.024038046598434448,
+            -0.008856122381985188,
+            0.03789309412240982,
+            0.004605492111295462,
+            -0.005095356144011021,
+            0.0015264854300767183,
+            -0.010152140632271767,
+            0.008393258787691593,
+            0.015066208317875862,
+            -0.00662280572578311,
+            -0.01976427435874939,
+            0.01726480945944786,
+            -0.0153053542599082,
+            -0.009835850447416306,
+            -0.002852396573871374,
+            -0.005477218423038721,
+            -0.009450131095945835,
+            0.012481886893510818,
+            -0.01449534296989441,
+            0.0003293081244919449,
+            0.000391987559851259,
+            0.0029063974507153034,
+            0.004636350087821484,
+            0.013855048455297947,
+            -0.010831006802618504,
+            0.007679677568376064,
+            0.016061365604400635,
+            -0.0013828048249706626,
+            0.016724802553653717,
+            0.007131956052035093,
+            -0.007093383930623531,
+            -0.013446185737848282,
+            0.03212273120880127,
+            -0.0024531767703592777,
+            0.016076793894171715,
+            -0.011401872150599957,
+            -0.008215827867388725,
+            -0.02559635415673256,
+            -0.010576431639492512,
+            0.028358107432723045,
+            0.004952639807015657,
+            -0.024099761620163918,
+            0.003521620063111186,
+            0.0347764827311039,
+            0.00823125708848238,
+            0.006507089827209711,
+            0.008431831374764442,
+            0.009542703628540039,
+            0.005137785337865353,
+            -0.0022873173002153635,
+            -0.0016113437013700604,
+            -0.00553121929988265,
+            0.009581275284290314,
+            0.0021214578300714493,
+            -0.005130071192979813,
+            -0.013099038042128086,
+            -0.01360818836838007,
+            -0.028280962258577347,
+            0.005928510800004005,
+            -0.03517762944102287,
+            0.015876220539212227,
+            -0.02136886678636074,
+            0.02385290153324604,
+            -0.004871638957411051,
+            -0.005616077687591314,
+            -0.020211707800626755,
+            0.008539832197129726,
+            -0.002129172207787633,
+            0.006352802272886038,
+            0.002316246274858713,
+            -0.004316202364861965,
+            0.012204169295728207,
+            -0.012188740074634552,
+            -0.024254050105810165,
+            0.028527824208140373,
+            -0.006067369598895311,
+            -0.02480948530137539,
+            0.0025843214243650436,
+            0.003236187621951103,
+            0.006499375682324171,
+            0.005635363515466452,
+            -0.03227701783180237,
+            0.0025669641327112913,
+            -0.003905411111190915,
+            0.009234127588570118,
+            0.006171513814479113,
+            -0.028666682541370392,
+            0.023544326424598694,
+            0.008917837403714657,
+            -0.0013567687710747123,
+            -0.007891823537647724,
+            -0.018715115264058113,
+            0.00014283679774962366,
+            0.0042390585877001286,
+            0.011687304824590683,
+            0.004354774486273527,
+            0.0015592715935781598,
+            -0.011093296110630035,
+            0.0029642554000020027,
+            -0.011216727085411549,
+            0.016647659242153168,
+            -0.020134564489126205,
+            0.011633303947746754,
+            -0.005018212366849184,
+            -0.005110784899443388,
+            -0.0036373359616845846,
+            0.007899537682533264,
+            -0.01404019445180893,
+            0.02872839756309986,
+            0.005133928265422583,
+            -0.014533915556967258,
+            -0.012065310031175613,
+            0.012026737444102764,
+            0.0069930967874825,
+            0.011031581088900566,
+            0.0014358413172885776,
+            -0.005677792709320784,
+            0.0035794780123978853,
+            0.0027771813329309225,
+            0.01231217011809349,
+            -0.0030298277270048857,
+            0.0188385471701622,
+            0.011401872150599957,
+            -0.0030626137740910053,
+            -0.0058976528234779835,
+            -0.00829297211021185,
+            2.452935768815223e-05,
+            -0.01715680956840515,
+            0.011371014639735222,
+            0.0068349516950547695,
+            -0.010460715740919113,
+            0.002080957405269146,
+            0.005006640683859587,
+            -0.001408840878866613,
+            0.013361327350139618,
+            -0.001426198286935687,
+            0.0005809901631437242,
+            1.5458919733646326e-05,
+            0.007633391302078962,
+            -0.010576431639492512,
+            -0.024485481902956963,
+            -0.031042715534567833,
+            0.004034627228975296,
+            -0.0005313287838362157,
+            0.015081637538969517,
+            0.021260865032672882,
+            -0.015374784357845783,
+            -0.00021925750479567796,
+            0.005943939555436373,
+            -0.011332442983984947,
+            -0.015012207441031933,
+            -0.017727673053741455,
+            0.013214753940701485,
+            -0.016740230843424797,
+            0.002659536898136139,
+            0.019548270851373672,
+            -0.025226064026355743,
+            0.012975607998669147,
+            -0.034251902252435684,
+            -0.007228385657072067,
+            -0.011039295233786106,
+            -0.014209911227226257,
+            0.005045212805271149,
+            0.016894519329071045,
+            -0.014842490665614605,
+            -0.0005539897829294205,
+            -0.004528348334133625,
+            0.0012420171406120062,
+            -0.022896315902471542,
+            -0.02136886678636074,
+            0.016123080626130104,
+            -0.01888483203947544,
+            -0.02135343849658966,
+            -0.009558131918311119,
+            -0.0014609131030738354,
+            -0.001360625959932804,
+            -0.014711346477270126,
+            0.019594557583332062,
+            0.0005718293250538409,
+            0.012466458603739738,
+            6.394508091034368e-05,
+            0.015845362097024918,
+            -0.00558136310428381,
+            -0.006865809205919504,
+            -0.006703807041049004,
+            0.0036643364001065493,
+            -0.011201297864317894,
+            0.0029873985331505537,
+            -0.0005356681067496538,
+            0.0015168423997238278,
+            0.0024724628310650587,
+            -0.004358631558716297,
+            0.008061540313065052,
+            0.0003941572504118085,
+            0.016879091039299965,
+            -0.0017627386841922998,
+            0.008470403030514717,
+            0.014541629701852798,
+            0.0030433277133852243,
+            -0.013585045002400875,
+            -0.0038282673340290785,
+            0.011795305646955967,
+            0.002084814477711916,
+            0.006896666716784239,
+            0.022742029279470444,
+            -0.0031436148565262556,
+            0.014163624495267868,
+            0.0012092309771105647,
+            -8.505418918502983e-06,
+            0.005118499509990215,
+            -0.008979552425444126,
+            0.004246773198246956,
+            -0.0013500186614692211,
+            -0.0006889916840009391,
+            -0.002725109225139022,
+            0.014749918133020401,
+            0.001008656807243824,
+            -0.013392184861004353,
+            -0.010221569798886776,
+            -0.0018890618812292814,
+            0.03203015774488449,
+            0.00025168206775560975,
+            0.010807863436639309,
+            -0.02882097102701664,
+            -0.00742895994335413,
+            0.009380700998008251,
+            -0.025148918852210045,
+            -0.020921431481838226,
+            0.005650792270898819,
+            0.012682461179792881,
+            -0.003945911768823862,
+            -0.0004481579817365855,
+            0.04890924692153931,
+            0.020165421068668365,
+            -0.0035987640731036663,
+            -0.004802209325134754,
+            -0.0063026584684848785,
+            -0.0026884658727794886,
+            0.011617874726653099,
+            -0.012204169295728207,
+            -0.006148370914161205,
+            -0.0069275242276489735,
+            0.004474347457289696,
+            0.007830108515918255,
+            -0.013662189245223999,
+            -0.006044226698577404,
+            0.002401104662567377,
+            -0.011147296987473965,
+            -0.013723904266953468,
+            0.006588091142475605,
+            -0.0063412305898964405,
+            3.5859869967680424e-05,
+            0.006453089416027069,
+            -0.012250455096364021,
+            0.01155615970492363,
+            0.004983497317880392,
+            0.019100835546851158,
+            -0.01653965748846531,
+            -0.001184159191325307,
+            0.00904126837849617,
+            -0.015953363850712776,
+            0.0031841155141592026,
+            -0.013145324774086475,
+            0.006167656742036343,
+            -0.0012063381727784872,
+            0.0058513665571808815,
+            -0.002252602484077215,
+            -0.00450520496815443,
+            0.007189814001321793,
+            -0.03320274502038956,
+            -0.009812707081437111,
+            0.013685332611203194,
+            -0.025874072685837746,
+            0.010483859106898308,
+            0.004748208448290825,
+            0.009511846117675304,
+            -0.007961252704262733,
+            0.011949594132602215,
+            0.00112340843770653,
+            -0.015374784357845783,
+            0.020890574902296066,
+            0.007675820495933294,
+            0.013762475922703743,
+            0.005708650220185518,
+            0.005928510800004005,
+            0.009990138001739979,
+            0.013515615835785866,
+            -0.012119310908019543,
+            -0.01636994071304798,
+            0.011455873027443886,
+            0.003450262127444148,
+            -5.23433591297362e-05,
+            -0.01230445597320795,
+            -0.0148656340315938,
+            0.01066900510340929,
+            0.01279046293348074,
+            0.012921607121825218,
+            0.01275189034640789,
+            0.006981525104492903,
+            0.015343926846981049,
+            -0.02062828466296196,
+            -0.012767319567501545,
+            -0.025904929265379906,
+            0.014965921640396118,
+            -0.0026576081290841103,
+            -0.001909312093630433,
+            -0.017496241256594658,
+            -0.005072213243693113,
+            -0.02237173728644848,
+            0.007344101555645466,
+            0.016123080626130104,
+            0.0009990137768909335,
+            -0.0024551053065806627,
+            0.02389918826520443,
+            -0.013739332556724548,
+            0.04357088729739189,
+            -0.007521532475948334,
+            -0.019193409010767937,
+            -0.010831006802618504,
+            -0.004593920893967152,
+            0.013893621042370796,
+            0.005338359624147415,
+            -0.00617922842502594,
+            0.003953625913709402,
+            0.020149992778897285,
+            -0.001371233258396387,
+            -0.002293103141710162,
+            -0.011965022422373295,
+            0.01441048551350832,
+            -0.006534090265631676,
+            0.008539832197129726,
+            -0.011085581965744495,
+            0.019471125677227974,
+            0.00515707116574049,
+            0.010699862614274025,
+            0.009588989429175854,
+            -0.014348769560456276,
+            -0.014603344723582268,
+            -0.00595936831086874,
+            0.00558136310428381,
+            -0.019316839054226875,
+            -0.009095268324017525,
+            0.03465305268764496,
+            -7.075544272083789e-05,
+            -0.006676806602627039,
+            -0.005114641971886158,
+            0.011471301317214966,
+            -0.009288128465414047,
+            -0.001129194162786007,
+            -0.005716364830732346,
+            -0.010877293534576893,
+            -0.001112801139242947,
+            -0.005785794463008642,
+            -0.006526376120746136,
+            -0.013261040672659874,
+            -0.0058976528234779835,
+            0.0014483771519735456,
+            -0.007668105885386467,
+            -0.0006889916840009391,
+            0.004532205406576395,
+            0.012512744404375553,
+            0.0110624385997653,
+            -0.00276753818616271,
+            0.006865809205919504,
+            -0.0012603388167917728,
+            0.018637971952557564,
+            0.0135464733466506,
+            -0.013739332556724548,
+            -0.019378554075956345,
+            -0.0028793970122933388,
+            0.00617537135258317,
+            0.014611058868467808,
+            0.012890749610960484,
+            -0.006032655015587807,
+            -0.0003430493816267699,
+            -0.024485481902956963,
+            -0.005176357459276915,
+            0.008547547273337841,
+            -0.01273646205663681,
+            -0.007579390425235033,
+            -0.0029951129108667374,
+            -0.0010154069168493152,
+            -0.004485919140279293,
+            -0.016925377771258354,
+            -0.000276512757409364,
+            0.010877293534576893,
+            -0.006483946926891804,
+            0.017110522836446762,
+            0.018622543662786484,
+            -0.014973635785281658,
+            -0.007251529023051262,
+            0.0009271735325455666,
+            -0.03236959129571915,
+            0.0024146048817783594,
+            -0.000498060486279428,
+            0.02067457139492035,
+            0.0053152162581682205,
+            0.0005901510012336075,
+            0.002809967380017042,
+            -0.006460803560912609,
+            0.002123386599123478,
+            0.007309386972337961,
+            -0.008840694092214108,
+            -0.01073072012513876,
+            -0.010468430817127228,
+            -0.011039295233786106,
+            -0.009735562838613987,
+            -0.0025631068274378777,
+            -0.022572312504053116,
+            0.029931843280792236,
+            0.011795305646955967,
+            0.003629621583968401,
+            -0.01069214753806591,
+            -0.0010134782642126083,
+            0.018637971952557564,
+            -0.0035235488321632147,
+            0.023729471489787102,
+            0.013006465509533882,
+            -0.011826164089143276,
+            -0.017125951126217842,
+            0.023467181250452995,
+            -0.020983148366212845,
+            0.0062525151297450066,
+            -0.003961340524256229,
+            0.0019044906366616488,
+            0.003965197596698999,
+            -0.011417300440371037,
+            0.008022967725992203,
+            -0.016940806061029434,
+            -0.00298354122787714,
+            -0.00433934573084116,
+            0.006217800546437502,
+            -0.002638322301208973,
+            0.026768941432237625,
+            -0.0019739202689379454,
+            -0.004042341839522123,
+            0.013099038042128086,
+            0.039127398282289505,
+            0.029283834621310234,
+            0.018761401996016502,
+            -0.006665234919637442,
+            -0.006194657180458307,
+            -0.012705604545772076,
+            0.004678778816014528,
+            -0.017095094546675682,
+            -0.003265116596594453,
+            0.004663350060582161,
+            0.0010684434091672301,
+            -0.009704705327749252,
+            -0.0003309956518933177,
+            -0.019162550568580627,
+            0.0239609032869339,
+            -0.01236617099493742,
+            0.00011173815437359735,
+            -0.010962151922285557,
+            -0.01404790859669447,
+            -0.018468255177140236,
+            0.0023876044433563948,
+            -0.0072823865339159966,
+            -0.01891569048166275,
+            0.0024782486725598574,
+            -0.019579127430915833,
+            -0.03690565377473831,
+            0.021986018866300583,
+            0.0029970414470881224,
+            0.011664161458611488,
+            0.025025488808751106,
+            -0.011371014639735222,
+            0.01578364707529545,
+            0.02713923342525959,
+            0.017002521082758904,
+            0.004566920455545187,
+            0.011756733991205692,
+            0.01023699901998043,
+            0.012597602792084217,
+            -0.03227701783180237,
+            -0.00698923971503973,
+            0.015027636662125587,
+            0.009558131918311119,
+            -0.008323829621076584,
+            -0.010861864313483238,
+            0.0030548993963748217,
+            -0.02798781543970108,
+            0.006476232316344976,
+            -0.022495169192552567,
+            -0.007444388698786497,
+            0.012659317813813686,
+            -0.009781849570572376,
+            0.0008239935268647969,
+            0.004875496029853821,
+            0.005415503401309252,
+            -0.011702733114361763,
+            -0.01229674182832241,
+            -0.007714392617344856,
+            0.00742510287091136,
+            0.01316846814006567,
+            -0.004875496029853821,
+            -0.0001758640428306535,
+            -0.014950492419302464,
+            0.003351903520524502,
+            0.011625589802861214,
+            0.015482786111533642,
+            0.015729647129774094,
+            -0.00287361117079854,
+            -0.0024705342948436737,
+            0.0181133933365345,
+            0.00044237219844944775,
+            -0.007201385218650103,
+            0.008532118052244186,
+            0.010846436023712158,
+            0.000777707202360034,
+            0.017449956387281418,
+            0.009581275284290314,
+            -0.01657051593065262,
+            0.004809923470020294,
+            0.004995069000869989,
+            0.03292502835392952,
+            0.010105853900313377,
+            0.00742895994335413,
+            -0.0031243290286511183,
+            -0.0016412369441241026,
+            -0.009596704505383968,
+            0.012659317813813686,
+            0.017959104850888252,
+            -0.0068388087674975395,
+            0.004686493426561356,
+            0.02556549571454525,
+            0.004177343565970659,
+            0.044866908341646194,
+            0.004852352663874626,
+            0.0035274059046059847,
+            -0.006048083771020174,
+            -0.009018125012516975,
+            -0.0015804861905053258,
+            -0.0018958119908347726,
+            -0.004077056422829628,
+            -0.006206228863447905,
+            -0.0016711303032934666,
+            0.019316839054226875,
+            -0.004088628105819225,
+            0.00595936831086874,
+            0.009897565469145775,
+            -0.00216967286542058,
+            -0.006931381765753031,
+            0.014503058046102524,
+            -0.01155615970492363,
+            0.010522431693971157,
+            -0.021569442003965378,
+            -0.002233316656202078,
+            0.00411562854424119,
+            0.014148196205496788,
+            -0.010977580212056637,
+            0.00021202526113484055,
+            -0.003330688923597336,
+            0.015112495049834251,
+            0.004532205406576395,
+            -0.0032940455712378025,
+            0.0011947664897888899,
+            -0.01903912052512169,
+            -0.008084683679044247,
+            0.03230787441134453,
+            0.004007626790553331,
+            0.015390212647616863,
+            -0.005508076399564743,
+            0.008200399577617645,
+            0.005866795312613249,
+            0.013824190944433212,
+            -0.02794152870774269,
+            0.011664161458611488,
+            0.020196279510855675,
+            0.0013345899060368538,
+            0.00780696514993906,
+            -0.001996099017560482,
+            -0.01566021703183651,
+            0.01638536900281906,
+            -0.026799799874424934,
+            0.0049024964682757854,
+            -0.014287054538726807,
+            0.012011309154331684,
+            0.004212058149278164,
+            -0.004146486055105925,
+            0.007992110215127468,
+            0.006966096349060535,
+            0.017743103206157684,
+            0.025750642642378807,
+            0.026568368077278137,
+            0.0002040697872871533,
+            0.0148656340315938,
+            -0.007031668908894062,
+            -3.0887702450854704e-05,
+            -0.026059217751026154,
+            -0.01715680956840515,
+            0.003988340962678194,
+            -0.0363810732960701,
+            -7.202108827186748e-05,
+            -0.008030682802200317,
+            -0.018576256930828094,
+            0.041688576340675354,
+            -0.006117513403296471,
+            -0.013569616712629795,
+            0.007197528146207333,
+            -0.005724078975617886,
+            0.004836923908442259,
+            -0.009550417773425579,
+            0.030194131657481194,
+            -0.002079028869047761,
+            -0.017604243010282516,
+            -0.03314103186130524,
+            -0.00574336526915431,
+            -0.025766070932149887,
+            -0.00450520496815443,
+            -0.01633908413350582,
+            0.014109623618423939,
+            -0.020828859880566597,
+            0.002302746055647731,
+            0.0037974095903337,
+            0.0020539569668471813,
+            0.01965627260506153,
+            0.0014638060238212347,
+            -0.007386530749499798,
+            0.006267943885177374,
+            -0.005106927827000618,
+            -0.012651603668928146,
+            -0.025087203830480576,
+            0.0011369085405021906,
+            0.00449363375082612,
+            0.00041971117025241256,
+            0.013808762654662132,
+            -0.0037279801908880472,
+            0.022495169192552567,
+            -0.016323653981089592,
+            0.012775033712387085,
+            -0.01517421007156372,
+            0.0007073133601807058,
+            0.004875496029853821,
+            0.0006644020322710276,
+            -7.636043301317841e-05,
+            -0.00536150299012661,
+            -0.02558092586696148,
+            0.007725963834673166,
+            0.021708300337195396,
+            -0.0025708212051540613,
+            0.0063450876623392105,
+            -0.006213943008333445,
+            0.000337986828526482,
+            0.0035061913076788187,
+            0.008138683624565601,
+            -0.015382498502731323,
+            0.0012535888236016035,
+            0.03369646519422531,
+            0.013855048455297947,
+            -0.007984396070241928,
+            -0.01230445597320795,
+            0.0021388153545558453,
+            0.013515615835785866,
+            -0.016076793894171715,
+            0.004586206283420324,
+            0.01279817707836628,
+            0.024315765127539635,
+            0.0020944576244801283,
+            -0.00553121929988265,
+            -0.028219247236847878,
+            0.021230008453130722,
+            -0.011810734868049622,
+            0.015205067582428455,
+            0.0018620614428073168,
+            0.028358107432723045,
+            0.002680751495063305,
+            0.0014271625550463796,
+            0.0010935150785371661,
+            0.03351132199168205,
+            0.00942698772996664,
+            -0.023590611293911934,
+            -0.0024281051009893417,
+            -0.0009517631842754781,
+            0.003047185018658638,
+            0.029268404468894005,
+            0.03153643757104874,
+            -0.004755923058837652,
+            -0.039281684905290604,
+            0.008802121505141258,
+            -0.024115189909934998,
+            0.002595893107354641,
+            0.011000723578035831,
+            0.00617922842502594,
+            -0.01360047422349453,
+            -0.0012748033041134477,
+            0.0012034452520310879,
+            0.0038514104671776295,
+            0.014950492419302464,
+            -0.011448157951235771,
+            -0.002935326425358653,
+            0.009735562838613987,
+            0.0032130444888025522,
+            -0.012543601915240288,
+            0.017557958140969276,
+            0.0020867432467639446,
+            -0.003022113349288702,
+            -0.004208201076835394,
+            0.03357303515076637,
+            0.008532118052244186,
+            -0.016308225691318512,
+            -0.0054039317183196545,
+            -0.026028359308838844,
+            -0.009234127588570118,
+            -0.013531044125556946,
+            -0.006904381327331066,
+            -0.022772885859012604,
+            -0.007679677568376064,
+            -0.019347695633769035,
+            -0.007513818331062794,
+            -0.02895982936024666,
+            0.006117513403296471,
+            -0.0047752088867127895,
+            0.03375818207859993,
+            -0.0035505490377545357,
+            -0.031582724303007126,
+            0.01741909794509411,
+            -0.0035717636346817017,
+            0.019594557583332062,
+            -0.010777005925774574,
+            0.009156984277069569,
+            0.01069214753806591,
+            0.01312989555299282,
+            0.02152315527200699,
+            0.00026566439191810787,
+            0.006503232754766941,
+            0.0030857569072395563,
+            0.025010060518980026,
+            -0.0033576893620193005,
+            -0.007818536832928658,
+            0.002827324904501438,
+            -0.011571588926017284,
+            0.019255124032497406,
+            0.0030317562632262707,
+            0.014503058046102524,
+            -0.022526025772094727,
+            0.022418024018406868,
+            -0.004570777527987957,
+            0.013801048509776592,
+            0.0012468387139961123,
+            0.005330645013600588,
+            -0.012582174502313137,
+            -0.007351816166192293,
+            -0.004269916098564863,
+            -0.0035949070006608963,
+            0.015220495872199535,
+            -0.01361590251326561,
+            0.005130071192979813,
+            -0.03523934632539749,
+            0.027015801519155502,
+            0.008601547218859196,
+            0.003278616815805435,
+            -0.0017087379237636924,
+            0.010121283121407032,
+            0.005924653261899948,
+            0.01235074270516634,
+            0.005727936513721943,
+            -0.013099038042128086,
+            0.017465384677052498,
+            0.001782024628482759,
+            0.008370115421712399,
+            0.0036874795332551003,
+            -0.016771089285612106,
+            -0.010645861737430096,
+            0.01324561145156622,
+            0.01819053664803505,
+            0.003342260606586933,
+            0.0013596616918221116,
+            -0.03145929425954819,
+            0.030086129903793335,
+            0.0017289882525801659,
+            0.008562975563108921,
+            -0.007891823537647724,
+            0.02894440107047558,
+            -0.007444388698786497,
+            0.010622718371450901,
+            0.022603169083595276,
+            0.01567564532160759,
+            0.007128098513931036,
+            -0.008416402153670788,
+            0.007668105885386467,
+            0.027632953599095345,
+            0.007976681925356388,
+            0.005573648493736982,
+            0.018298538401722908,
+            0.009156984277069569,
+            0.012049880810081959,
+            -0.01576821878552437,
+            -0.04017655551433563,
+            0.001536128344014287,
+            -0.015444213524460793,
+            -0.018668830394744873,
+            0.011949594132602215,
+            0.011949594132602215,
+            0.004917925223708153,
+            0.02477862872183323,
+            0.015652501955628395,
+            -0.006858095061033964,
+            0.0018466326873749495,
+            -0.013006465509533882,
+            -0.019903132691979408,
+            0.005770365707576275,
+            -0.003265116596594453,
+            -0.023220321163535118,
+            -0.001962348585948348,
+            -0.011031581088900566,
+            1.3439917893265374e-05,
+            -0.006024940405040979,
+            -0.0069699534215033054,
+            0.04431147128343582,
+            0.008200399577617645,
+            -0.010321857407689095,
+            -0.014387342147529125,
+            0.013577330857515335,
+            -0.011841592378914356,
+            0.029175832867622375,
+            -0.0034155473113059998,
+            -0.001836989657022059,
+            -0.025982074439525604,
+            -0.0010607289150357246,
+            -0.009465559385716915,
+            -0.005496504716575146,
+            0.005230357870459557,
+            -0.01198045164346695,
+            -0.009789563715457916,
+            0.0028813255485147238,
+            -0.008709548972547054,
+            0.013307327404618263,
+            -0.009087554179131985,
+            0.006599662825465202,
+            -0.01272874791175127,
+            -0.0007323851459659636,
+            0.018684258684515953,
+            -4.550286030280404e-05,
+            0.016308225691318512,
+            -0.015104779973626137,
+            -0.0005366324330680072,
+            -0.00022648973390460014,
+            -0.03323360159993172,
+            0.016693945974111557,
+            0.012273598462343216,
+            0.0001559753727633506,
+            -0.02312774769961834,
+            0.002248745411634445,
+            0.024207763373851776,
+            0.0014117337996140122,
+            0.004354774486273527,
+            0.025040917098522186,
+            -0.005091499071568251,
+            0.0074598174542188644,
+            0.013446185737848282,
+            0.007922681048512459,
+            -0.020026562735438347,
+            -0.0020925288554280996,
+            0.009226413443684578,
+            0.007668105885386467,
+            0.007710535079240799,
+            -0.0008996910182759166,
+            0.0165859442204237,
+            0.0067848083563148975,
+            -0.019285980612039566,
+            -0.017696816474199295,
+            0.015120209194719791,
+            0.010098139755427837,
+            -0.0009392272913828492,
+            0.004393346607685089,
+            0.005731793586164713,
+            0.011394158005714417,
+            -0.011695018969476223,
+            -0.0013037322787567973,
+            -0.02298888936638832,
+            0.021662013605237007,
+            0.016663087531924248,
+            -0.016045935451984406,
+            0.034961625933647156,
+            -0.004250630270689726,
+            -0.007363387383520603,
+            -0.019239693880081177,
+            -0.02541120909154415,
+            -0.0021966733038425446,
+            0.015359355136752129,
+            -0.004458918701857328,
+            0.002956541022285819,
+            0.007961252704262733,
+            -0.032523877918720245,
+            -0.020365996286273003,
+            -0.011671875603497028,
+            0.0005154178361408412,
+            0.0052612158469855785,
+            0.0027347521390765905,
+            0.009704705327749252,
+            0.005018212366849184,
+            0.01116272620856762,
+            0.005862938240170479,
+            0.007706678006798029,
+            -0.017959104850888252,
+            -0.02135343849658966,
+            0.008161826990544796,
+            0.00027048587799072266,
+            -0.016107652336359024,
+            -0.021631157025694847,
+            0.007023954298347235,
+            0.004077056422829628,
+            -0.001974884420633316,
+            0.013492472469806671,
+            -0.020458567887544632,
+            0.017665958032011986,
+            0.009596704505383968,
+            -0.003650836180895567,
+            0.022556884214282036,
+            0.013137610629200935,
+            0.01318389642983675,
+            0.00037053192500025034,
+            -0.016925377771258354,
+            -0.0017810603603720665,
+            0.02649122290313244,
+            0.0009932280518114567,
+            0.0014628416392952204,
+            0.012528173625469208,
+            0.004543777089565992,
+            0.020288852974772453,
+            0.0034849767107516527,
+            -0.018715115264058113,
+            0.029484407976269722,
+            0.00019563216483220458,
+            0.012119310908019543,
+            0.018730545416474342,
+            0.021862588822841644,
+            0.0062525151297450066,
+            0.004632492549717426,
+            0.007097241003066301,
+            0.01903912052512169,
+            0.006545661948621273,
+            -0.013075895607471466,
+            -0.004185057710856199,
+            0.0069699534215033054,
+            0.0020057419314980507,
+            -0.004755923058837652,
+            -0.012196454219520092,
+            0.018298538401722908,
+            -0.007043240126222372,
+            0.0018292752793058753,
+            -0.022664885967969894,
+            0.0069930967874825,
+            -0.008848408237099648,
+            -0.004038484301418066,
+            -0.00823897123336792,
+            0.004786780569702387,
+            0.015876220539212227,
+            -0.004678778816014528,
+            0.03400504216551781,
+            -0.010098139755427837,
+            0.014595630578696728,
+            -0.009342129342257977,
+            -0.02227916568517685,
+            0.018483683466911316,
+            -0.01116272620856762,
+            -0.0014011265011504292,
+            0.00012547941878437996,
+            0.0022391024976968765,
+            -0.009596704505383968,
+            -0.014842490665614605,
+            0.004667207598686218,
+            -0.010869579389691353,
+            0.0303792767226696,
+            -0.0004961318918503821,
+            0.02315860614180565,
+            0.003916982561349869,
+            0.0021735301706939936,
+            0.01731109619140625,
+            -0.005558219738304615,
+            -0.003257402218878269,
+            0.011664161458611488,
+            -0.0072129569016397,
+            0.020011134445667267,
+            -0.00020913235493935645,
+            -0.001489842077717185,
+            0.016940806061029434,
+            -0.011772163212299347,
+            -0.012481886893510818,
+            0.018545400351285934,
+            0.012659317813813686,
+            0.0014657345600426197,
+            0.006144513376057148,
+            0.002813824685290456,
+            -0.0002004536654567346,
+            -0.003560192184522748,
+            -0.020921431481838226,
+            -0.005704793147742748,
+            0.002511034719645977,
+            0.002727037761360407,
+            -0.00455920584499836,
+            0.004401060752570629,
+            0.0012593745486810803,
+            -0.0004539437941275537,
+            -0.002684608567506075,
+            0.0025418922305107117,
+            -0.0031841155141592026,
+            0.004690350499004126,
+            0.005878366995602846,
+            -0.014395056292414665,
+            -0.02391461655497551,
+            0.0065533765591681,
+            0.013631331734359264,
+            -0.010020995512604713,
+            0.0019806702621281147,
+            0.03150557726621628,
+            -0.03786223754286766,
+            0.01738824136555195,
+            0.005434789694845676,
+            -0.007336387410759926,
+            0.005704793147742748,
+            -0.014287054538726807
+          ],
+          "title": "4 Downstream Results"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "a6a275c8-8419-4686-8775-f4096e1f57c2",
+      "type": "next",
+      "source": {
+        "id": "8f4b7200-614f-45df-b481-ec48e0567e94",
+        "properties": {
+          "page_content": "4 Downstream Results 14 4.1 Fine-Tuning 14 4.2 Distillation 16 4.3 Multilingual Learning 17 ",
+          "keyphrases": [
+            "Downstream Results",
+            "Fine-Tuning",
+            "Distillation",
+            "Multilingual Learning"
+          ],
+          "embedding": [
+            0.007475246209651232,
+            0.018637971952557564,
+            -0.022418024018406868,
+            -0.003915054257959127,
+            -0.025920359417796135,
+            -0.01815968006849289,
+            -0.013469329103827477,
+            0.07850165665149689,
+            -0.036689650267362595,
+            0.014703632332384586,
+            0.017773959785699844,
+            -0.022819172590970993,
+            -0.010807863436639309,
+            -0.039034824818372726,
+            -0.02303517609834671,
+            0.02710837498307228,
+            0.017064236104488373,
+            -0.007691249251365662,
+            0.015552215278148651,
+            -0.02539578080177307,
+            -0.010514716617763042,
+            -0.0059323678724467754,
+            -0.04039255902171135,
+            0.03233873471617699,
+            -0.02058199979364872,
+            0.002258388325572014,
+            0.04107142612338066,
+            -0.006360516417771578,
+            -0.03289416804909706,
+            0.017465384677052498,
+            -0.020350567996501923,
+            -0.00360647845081985,
+            0.01323789730668068,
+            -0.029422692954540253,
+            -0.022634027525782585,
+            0.012574459426105022,
+            0.06967639178037643,
+            0.011378728784620762,
+            0.010777005925774574,
+            0.03610335662961006,
+            0.009380700998008251,
+            0.007938110269606113,
+            0.008539832197129726,
+            -0.007521532475948334,
+            -0.002765609649941325,
+            -0.047397226095199585,
+            -0.0303792767226696,
+            -0.04141085967421532,
+            0.011671875603497028,
+            -0.05992539972066879,
+            0.01718766614794731,
+            -0.060480836778879166,
+            0.016693945974111557,
+            -0.014703632332384586,
+            -0.002127243671566248,
+            0.057271648198366165,
+            0.014317912049591541,
+            0.05264301598072052,
+            0.05779622867703438,
+            0.020921431481838226,
+            0.030240418389439583,
+            -0.005635363515466452,
+            0.014194482006132603,
+            -0.025920359417796135,
+            0.002937254961580038,
+            -0.04468176141381264,
+            -0.009534989483654499,
+            -0.028512394055724144,
+            -0.029129546135663986,
+            0.025735212489962578,
+            -0.003972912207245827,
+            0.04144171625375748,
+            -0.007787679322063923,
+            -0.0030394706409424543,
+            -0.01823682337999344,
+            0.014634202234447002,
+            0.00830840039998293,
+            0.00699695385992527,
+            -0.010993009433150291,
+            0.03243130445480347,
+            0.0004541848611552268,
+            -0.0026498937513679266,
+            0.012991037219762802,
+            -0.016786517575383186,
+            0.03557877987623215,
+            -0.016925377771258354,
+            -0.04036170244216919,
+            -0.049773260951042175,
+            0.02891354262828827,
+            -0.026151791214942932,
+            -0.02970041148364544,
+            -0.02309689112007618,
+            0.0036836224608123302,
+            -0.013939906843006611,
+            -0.033449605107307434,
+            0.02380661480128765,
+            -0.007452103309333324,
+            0.001406912342645228,
+            -0.021631157025694847,
+            0.011209012009203434,
+            -0.003843696089461446,
+            0.005519647616893053,
+            0.004609349649399519,
+            0.0007053847657516599,
+            0.02632150799036026,
+            -0.017928248271346092,
+            0.00661894865334034,
+            -0.016678515821695328,
+            0.004844638518989086,
+            0.07368787378072739,
+            0.01564478874206543,
+            0.007540818769484758,
+            -0.02072085812687874,
+            0.025087203830480576,
+            -0.02229459397494793,
+            -0.018745973706245422,
+            -0.03391246870160103,
+            -0.04693436250090599,
+            0.0011436586501076818,
+            -0.027432380244135857,
+            0.04838467016816139,
+            0.025056347250938416,
+            0.006591948214918375,
+            -0.046378929167985916,
+            -0.053167592734098434,
+            0.034221045672893524,
+            0.025904929265379906,
+            0.031582724303007126,
+            -0.01393219269812107,
+            0.02979298308491707,
+            0.019393982365727425,
+            0.0023509610909968615,
+            -0.06424546241760254,
+            -0.0021214578300714493,
+            0.00862469058483839,
+            -0.0044126324355602264,
+            0.01812882162630558,
+            0.011733590625226498,
+            0.017912819981575012,
+            0.001125336973927915,
+            0.03468390926718712,
+            -0.03631936013698578,
+            0.02301974780857563,
+            -0.006086655426770449,
+            -0.022448882460594177,
+            0.0007304565515369177,
+            -0.009596704505383968,
+            0.09985509514808655,
+            0.018607115373015404,
+            -0.006129084620624781,
+            -0.015968792140483856,
+            0.003992198035120964,
+            -0.012682461179792881,
+            0.026861514896154404,
+            0.018406540155410767,
+            0.010159854777157307,
+            0.01717223785817623,
+            0.004825352691113949,
+            0.0030587567016482353,
+            0.0013577330391854048,
+            0.020767144858837128,
+            -0.007043240126222372,
+            0.019378554075956345,
+            0.006148370914161205,
+            0.07208327949047089,
+            0.03388161212205887,
+            -0.0052612158469855785,
+            0.026614654809236526,
+            -0.04964983090758324,
+            0.021044863387942314,
+            0.02553463913500309,
+            0.030857570469379425,
+            0.04526805505156517,
+            0.0030356133356690407,
+            -0.01653965748846531,
+            -0.016987092792987823,
+            -0.020828859880566597,
+            0.03162900730967522,
+            -0.06344316154718399,
+            0.01741909794509411,
+            -0.013153038918972015,
+            -0.014549343846738338,
+            0.020906003192067146,
+            -0.0063450876623392105,
+            0.01806710660457611,
+            -0.023621469736099243,
+            0.04292288050055504,
+            -0.00019466786761768162,
+            0.027324378490447998,
+            0.006726950407028198,
+            0.0067732366733253,
+            0.010908151045441628,
+            0.027386093512177467,
+            0.016940806061029434,
+            -0.0021947447676211596,
+            -0.007737535517662764,
+            -0.013716190122067928,
+            0.03545534610748291,
+            -0.0014773061266168952,
+            0.010576431639492512,
+            -0.019440269097685814,
+            -0.05048298463225365,
+            -0.0035293344408273697,
+            0.05329102277755737,
+            0.005943939555436373,
+            0.020412283018231392,
+            0.023528896272182465,
+            0.05344530940055847,
+            0.0020018848590552807,
+            -0.03142843395471573,
+            0.01661680079996586,
+            -0.01977970264852047,
+            -0.052735585719347,
+            0.026136361062526703,
+            -0.0027559667360037565,
+            -0.0024358194787055254,
+            -0.008269828744232655,
+            0.012867606244981289,
+            -0.04230572655797005,
+            -0.017650529742240906,
+            0.02061285637319088,
+            -0.0011561946012079716,
+            0.013461614958941936,
+            -0.01480391900986433,
+            -0.032554734498262405,
+            0.0017675601411610842,
+            -0.020427711308002472,
+            3.8240483263507485e-05,
+            0.009689277037978172,
+            -0.02221745066344738,
+            0.01800539158284664,
+            0.03056442365050316,
+            -0.022803744301199913,
+            -0.019270552322268486,
+            0.01731109619140625,
+            -0.018761401996016502,
+            0.014194482006132603,
+            0.00035775493597611785,
+            0.003957483451813459,
+            0.02880554087460041,
+            0.0005978654371574521,
+            -0.033418748527765274,
+            -0.02209402061998844,
+            -0.032585594803094864,
+            0.02317403443157673,
+            -0.03468390926718712,
+            -0.024207763373851776,
+            0.01882311701774597,
+            -0.008362401276826859,
+            0.012481886893510818,
+            0.006757807917892933,
+            0.005187929142266512,
+            0.016231082379817963,
+            0.01397847943007946,
+            -0.021584870293736458,
+            0.05446361005306244,
+            0.03206101432442665,
+            -0.024269478395581245,
+            -0.0248403437435627,
+            -0.04853895679116249,
+            0.018360253423452377,
+            -0.018483683466911316,
+            -0.051501285284757614,
+            0.0009676741319708526,
+            -0.003297902876511216,
+            -0.007864823564887047,
+            0.021260865032672882,
+            0.03378903865814209,
+            0.005824366118758917,
+            0.025704355910420418,
+            0.0031937584280967712,
+            0.004107913933694363,
+            -0.014927349053323269,
+            -0.021152863278985023,
+            -0.008015253581106663,
+            -0.03388161212205887,
+            0.019270552322268486,
+            -0.014024765230715275,
+            -0.01820596680045128,
+            0.022541454061865807,
+            0.008586118929088116,
+            0.04875496029853821,
+            0.01447991468012333,
+            0.0017743102507665753,
+            -0.022757457569241524,
+            -0.01633908413350582,
+            -0.014078766107559204,
+            0.02485577203333378,
+            -0.05665449798107147,
+            -0.0132224690169096,
+            0.011579303070902824,
+            -0.015544501133263111,
+            -0.05514247715473175,
+            0.005191786214709282,
+            -0.029808413237333298,
+            0.012782747857272625,
+            0.00558136310428381,
+            -0.005284358747303486,
+            0.010939008556306362,
+            0.01324561145156622,
+            -0.022464310750365257,
+            0.009635276161134243,
+            0.02153858356177807,
+            -0.008524403907358646,
+            -0.03366560861468315,
+            0.027648381888866425,
+            0.019424840807914734,
+            0.028651254251599312,
+            -0.058845385909080505,
+            0.00558907724916935,
+            0.028543252497911453,
+            -0.02470148541033268,
+            -0.013523329980671406,
+            -0.011533016338944435,
+            -0.03366560861468315,
+            0.007120384369045496,
+            0.02971583977341652,
+            0.007575533352792263,
+            -0.04468176141381264,
+            0.007679677568376064,
+            -0.011602446436882019,
+            0.009295842610299587,
+            -0.04212058335542679,
+            0.05168642848730087,
+            0.007108812686055899,
+            0.004316202364861965,
+            0.013052752241492271,
+            0.005288215819746256,
+            -0.011293870396912098,
+            0.03854110464453697,
+            -0.024207763373851776,
+            -0.010206141509115696,
+            -0.017465384677052498,
+            0.022526025772094727,
+            0.021662013605237007,
+            -0.04366346076130867,
+            -0.049032676964998245,
+            0.006044226698577404,
+            0.010661290027201176,
+            -0.015444213524460793,
+            -0.03527020290493965,
+            0.02459348365664482,
+            -0.007394245360046625,
+            -0.02064371481537819,
+            0.006414517294615507,
+            -0.029299262911081314,
+            -0.008964124135673046,
+            0.0019999563228338957,
+            0.0058937957510352135,
+            0.015536786057054996,
+            0.0009117447771131992,
+            0.010946722701191902,
+            -0.005990225821733475,
+            0.03780052438378334,
+            -0.012875321321189404,
+            -0.02951526641845703,
+            -0.020967718213796616,
+            0.019501984119415283,
+            0.024917487055063248,
+            0.03162900730967522,
+            -0.008022967725992203,
+            -0.04039255902171135,
+            -0.035825639963150024,
+            -0.003938197158277035,
+            -0.025843214243650436,
+            -0.035856496542692184,
+            -0.005191786214709282,
+            -0.003957483451813459,
+            -0.004836923908442259,
+            0.01324561145156622,
+            -0.02542663738131523,
+            -0.002941112034022808,
+            0.0429537370800972,
+            0.02146144025027752,
+            -0.0017280238680541515,
+            -0.033418748527765274,
+            -0.014271626248955727,
+            -0.03443704918026924,
+            -0.036597076803445816,
+            0.020319709554314613,
+            -0.0013972693122923374,
+            0.020273422822356224,
+            -0.021260865032672882,
+            -0.009249556809663773,
+            0.016061365604400635,
+            0.018283110111951828,
+            0.0148656340315938,
+            0.007347958628088236,
+            -0.039806265383958817,
+            -0.030641566962003708,
+            -0.01643165573477745,
+            -0.009851278737187386,
+            -0.021692872047424316,
+            -0.029283834621310234,
+            0.012682461179792881,
+            0.040917135775089264,
+            -0.02468605525791645,
+            0.005211072042584419,
+            0.015467356890439987,
+            -0.021708300337195396,
+            -0.057395078241825104,
+            0.03289416804909706,
+            -0.015220495872199535,
+            -0.02062828466296196,
+            0.0032169015612453222,
+            0.01279046293348074,
+            0.005014355294406414,
+            -0.022047733888030052,
+            0.015251354314386845,
+            0.01982598938047886,
+            0.009974709711968899,
+            -0.017743103206157684,
+            0.019285980612039566,
+            -0.03193758428096771,
+            0.02479405701160431,
+            0.0015737360808998346,
+            -0.022711170837283134,
+            -0.001737666898407042,
+            -0.06436888873577118,
+            -0.003911197185516357,
+            0.0006523483316414058,
+            0.020227137953042984,
+            0.03357303515076637,
+            -0.003241973463445902,
+            0.01581450365483761,
+            0.03369646519422531,
+            0.02561178244650364,
+            -0.0017135593807324767,
+            0.0020906003192067146,
+            -0.01655508577823639,
+            0.02309689112007618,
+            0.018622543662786484,
+            0.05415503680706024,
+            -0.01823682337999344,
+            -0.008370115421712399,
+            -0.017496241256594658,
+            -0.014634202234447002,
+            0.03755366429686546,
+            -0.0181133933365345,
+            -0.005079927388578653,
+            0.0010925508104264736,
+            -0.04853895679116249,
+            0.016771089285612106,
+            0.011239869520068169,
+            0.014209911227226257,
+            0.017588814720511436,
+            -0.00450520496815443,
+            0.013484758324921131,
+            -0.022587740793824196,
+            -0.017588814720511436,
+            -0.013253326527774334,
+            0.054031603038311005,
+            -0.07307072728872299,
+            -0.04116399958729744,
+            0.05032869800925255,
+            -0.02803410217165947,
+            0.02806496061384678,
+            0.003409761469811201,
+            0.024207763373851776,
+            -0.05208757892251015,
+            -0.013531044125556946,
+            0.02883639931678772,
+            0.014989065006375313,
+            -0.03465305268764496,
+            -0.0110624385997653,
+            -0.03378903865814209,
+            -0.051470424979925156,
+            -0.044002894312143326,
+            -0.022865459322929382,
+            0.009843564592301846,
+            0.006009511649608612,
+            -0.025102633982896805,
+            -0.012613032013177872,
+            -0.01561393029987812,
+            -0.021677441895008087,
+            0.00471735093742609,
+            -0.027401521801948547,
+            -0.03551706299185753,
+            -0.012512744404375553,
+            0.022896315902471542,
+            -0.0012661246582865715,
+            0.01993398927152157,
+            -0.017635101452469826,
+            -0.006931381765753031,
+            -0.024161476641893387,
+            -0.0010607289150357246,
+            0.03675136715173721,
+            0.009820421226322651,
+            0.04668750241398811,
+            -0.0010134782642126083,
+            0.009311271831393242,
+            -0.03354217857122421,
+            -0.000637883844319731,
+            0.008439545519649982,
+            0.005176357459276915,
+            -0.04934125393629074,
+            -0.003911197185516357,
+            -0.04764408618211746,
+            0.01885397545993328,
+            0.002808038843795657,
+            -0.01198045164346695,
+            -0.002281531458720565,
+            -0.012952464632689953,
+            -0.0004305595357436687,
+            0.026969516649842262,
+            -0.0102987140417099,
+            -0.0075948191806674,
+            -0.009357557632029057,
+            -0.034221045672893524,
+            0.03486905246973038,
+            -0.014348769560456276,
+            -0.017743103206157684,
+            -0.008732692338526249,
+            -0.019270552322268486,
+            -0.018375683575868607,
+            0.011425015516579151,
+            0.0012391243362799287,
+            -0.00016139955550897866,
+            -0.01235074270516634,
+            -0.005083784461021423,
+            0.02152315527200699,
+            0.01800539158284664,
+            -0.0023509610909968615,
+            0.03273988142609596,
+            -0.04175029322504997,
+            0.019100835546851158,
+            0.0014223410980775952,
+            -0.015637073665857315,
+            0.03709080070257187,
+            0.01435648463666439,
+            -0.02562721259891987,
+            0.001886168960481882,
+            0.03397418558597565,
+            -0.0020057419314980507,
+            0.010753863491117954,
+            -0.01074614841490984,
+            -0.011695018969476223,
+            0.025133490562438965,
+            -0.016678515821695328,
+            -0.0015814504586160183,
+            -0.021245436742901802,
+            0.014171338640153408,
+            -0.014873349107801914,
+            0.029052402824163437,
+            0.009396130219101906,
+            0.022803744301199913,
+            0.020998576655983925,
+            -0.013862763531506062,
+            0.006129084620624781,
+            0.019147122278809547,
+            -0.03456047922372818,
+            0.013099038042128086,
+            -0.009141555055975914,
+            0.004084771033376455,
+            -0.01147901639342308,
+            -0.007228385657072067,
+            0.012096167542040348,
+            -0.015073922462761402,
+            0.014564773067831993,
+            -0.013986193574965,
+            0.024207763373851776,
+            0.039219971746206284,
+            -0.026599224656820297,
+            0.007664248812943697,
+            0.001095443731173873,
+            0.0008090469054877758,
+            0.03480733931064606,
+            0.004443489946424961,
+            -0.03277073800563812,
+            -0.006946810521185398,
+            -0.0072823865339159966,
+            0.027756383642554283,
+            -0.026676369830965996,
+            -0.01720309443771839,
+            0.03807824105024338,
+            0.030795855447649956,
+            -0.002358675468713045,
+            -0.03227701783180237,
+            -0.019208837300539017,
+            0.003047185018658638,
+            0.01893111877143383,
+            -0.025735212489962578,
+            -0.029422692954540253,
+            -0.009619846940040588,
+            0.011841592378914356,
+            0.017511671409010887,
+            -0.00911841168999672,
+            -0.01280589122325182,
+            0.0030992571264505386,
+            -0.006811808329075575,
+            0.010784721001982689,
+            -0.004269916098564863,
+            0.0074598174542188644,
+            -0.05878366902470589,
+            -0.006645949091762304,
+            0.011232155375182629,
+            0.04067027568817139,
+            0.00450134789571166,
+            0.03452961891889572,
+            -0.06273344159126282,
+            -0.0036122642923146486,
+            -0.0038880538195371628,
+            -0.01735738292336464,
+            -0.015698788687586784,
+            -0.016832804307341576,
+            -0.004455061629414558,
+            -0.013847334310412407,
+            0.02058199979364872,
+            0.024500910192728043,
+            0.012520458549261093,
+            0.014742203988134861,
+            -0.0058976528234779835,
+            0.042490873485803604,
+            -0.009372986853122711,
+            -0.014186767861247063,
+            0.00041947010322473943,
+            -0.002705823164433241,
+            0.004146486055105925,
+            0.007818536832928658,
+            -0.006572662387043238,
+            0.035825639963150024,
+            0.020890574902296066,
+            0.008832979016005993,
+            0.00911841168999672,
+            0.005693221464753151,
+            0.0458543486893177,
+            0.027432380244135857,
+            -0.045761775225400925,
+            -0.013153038918972015,
+            -0.0098589938133955,
+            -0.017758531495928764,
+            -0.01808253675699234,
+            0.006534090265631676,
+            -0.029823841527104378,
+            -0.00904898252338171,
+            0.008262114599347115,
+            0.019301410764455795,
+            0.015876220539212227,
+            0.016801945865154266,
+            -0.01825225166976452,
+            -0.04684179276227951,
+            -0.01720309443771839,
+            -0.004208201076835394,
+            0.0246089119464159,
+            -0.008030682802200317,
+            -0.0034599050413817167,
+            -0.003587192390114069,
+            -4.5985012548044324e-05,
+            0.01820596680045128,
+            0.01360818836838007,
+            -0.0073248157277703285,
+            -0.010831006802618504,
+            0.012960179708898067,
+            0.037461090832948685,
+            0.028543252497911453,
+            -0.02294260263442993,
+            0.012620746158063412,
+            0.01393219269812107,
+            -0.022850031033158302,
+            0.02237173728644848,
+            0.04051598906517029,
+            0.020196279510855675,
+            0.0015100922901183367,
+            -0.02641407959163189,
+            0.014950492419302464,
+            -0.01068443339318037,
+            0.014773061498999596,
+            0.0029681124724447727,
+            0.009226413443684578,
+            0.025997502729296684,
+            0.02632150799036026,
+            -0.0033962612506002188,
+            -0.020288852974772453,
+            0.023220321163535118,
+            0.0009445309406146407,
+            -0.0037183372769504786,
+            0.012173310853540897,
+            0.030703281983733177,
+            -0.009897565469145775,
+            0.004424204118549824,
+            -0.03403589874505997,
+            0.006742379162460566,
+            0.011371014639735222,
+            0.02465519867837429,
+            -0.004300773609429598,
+            0.004293059464544058,
+            0.004624778404831886,
+            0.009743277914822102,
+            -0.02798781543970108,
+            -0.026815228164196014,
+            -0.007201385218650103,
+            0.016508799046278,
+            -0.016092222183942795,
+            0.02301974780857563,
+            -0.014695917256176472,
+            0.017851104959845543,
+            -0.03471476584672928,
+            0.04144171625375748,
+            0.024392908439040184,
+            -0.03372732549905777,
+            -0.020134564489126205,
+            -0.013376756571233273,
+            -0.05677792802453041,
+            0.011671875603497028,
+            0.03604163974523544,
+            -0.010074996389448643,
+            -0.0062910872511565685,
+            -0.017449956387281418,
+            0.007313244044780731,
+            -0.013299612328410149,
+            -0.014773061498999596,
+            0.007475246209651232,
+            0.011810734868049622,
+            0.018452826887369156,
+            -0.008755835704505444,
+            0.029006116092205048,
+            0.012775033712387085,
+            -0.03511591628193855,
+            0.016508799046278,
+            0.030101560056209564,
+            0.010098139755427837,
+            0.024917487055063248,
+            -0.001642201328650117,
+            -0.010599575005471706,
+            -0.007598676718771458,
+            0.03597992658615112,
+            0.022479739040136337,
+            0.01562164444476366,
+            0.010838720947504044,
+            -0.010591860860586166,
+            0.026043789461255074,
+            -0.012235026806592941,
+            -0.0222020223736763,
+            0.00449749082326889,
+            -0.018375683575868607,
+            -0.00942698772996664,
+            -0.00900269579142332,
+            0.018375683575868607,
+            -0.001383769093081355,
+            -0.010429858230054379,
+            0.010260142385959625,
+            0.0023914617486298084,
+            -0.003322974545881152,
+            -0.006456946488469839,
+            -0.004354774486273527,
+            0.004744351375848055,
+            0.03397418558597565,
+            0.009218699298799038,
+            -0.010476144962012768,
+            0.03687479719519615,
+            0.007386530749499798,
+            -0.027463236823678017,
+            0.01973341591656208,
+            -0.01902369223535061,
+            -0.019949419423937798,
+            -5.614028486888856e-05,
+            0.020844288170337677,
+            -0.012289026752114296,
+            -0.02465519867837429,
+            0.0018485613400116563,
+            -0.009797278791666031,
+            0.007309386972337961,
+            0.022726600989699364,
+            0.00014801989891566336,
+            0.028496965765953064,
+            -0.017048807814717293,
+            -0.016184795647859573,
+            0.024300336837768555,
+            0.011155011132359505,
+            -0.012150168418884277,
+            0.034221045672893524,
+            -0.006279515568166971,
+            0.05955510959029198,
+            -0.0363810732960701,
+            0.003913125488907099,
+            -0.024423766881227493,
+            -0.01806710660457611,
+            -0.014155910350382328,
+            -0.040886279195547104,
+            0.029345549643039703,
+            -0.05350702628493309,
+            0.019579127430915833,
+            0.0058089373633265495,
+            -0.016678515821695328,
+            -0.0376770943403244,
+            0.0015978434821590781,
+            0.017789389938116074,
+            -0.008053826168179512,
+            -0.02808038890361786,
+            0.004204344004392624,
+            -0.0008818514761514962,
+            0.006468518171459436,
+            -0.04273773357272148,
+            0.00906441081315279,
+            0.05292073264718056,
+            -0.03363475203514099,
+            0.010993009433150291,
+            0.009635276161134243,
+            0.0032496878411620855,
+            -0.016987092792987823,
+            0.03613421320915222,
+            0.003791623981669545,
+            -0.003820552956312895,
+            0.02295803278684616,
+            -0.03443704918026924,
+            -0.04474347457289696,
+            0.019316839054226875,
+            -0.0011455873027443886,
+            0.03320274502038956,
+            -0.003004755824804306,
+            0.026938658207654953,
+            0.0062949443235993385,
+            0.023420896381139755,
+            0.027740955352783203,
+            -0.0019594556652009487,
+            0.010954436846077442,
+            -0.00828525796532631,
+            -0.014402770437300205,
+            -0.010445287451148033,
+            -0.02954612299799919,
+            0.0102987140417099,
+            0.027000373229384422,
+            -0.025133490562438965,
+            0.02048942632973194,
+            -0.03613421320915222,
+            0.007749107200652361,
+            -0.017527099698781967,
+            0.011016152799129486,
+            0.051532141864299774,
+            0.047335512936115265,
+            0.0026981087867170572,
+            -0.025287779048085213,
+            0.009642990306019783,
+            0.01743452623486519,
+            -0.024932917207479477,
+            0.031814154237508774,
+            -0.0024628196842968464,
+            -0.007976681925356388,
+            -0.013099038042128086,
+            0.031845010817050934,
+            -0.0038282673340290785,
+            -0.0073711019940674305,
+            -0.000637883844319731,
+            -0.01242017187178135,
+            0.01324561145156622,
+            -0.006090512964874506,
+            -0.01657051593065262,
+            -0.017002521082758904,
+            0.012620746158063412,
+            -0.012582174502313137,
+            0.012065310031175613,
+            0.013014179654419422,
+            0.03388161212205887,
+            0.013407614082098007,
+            -0.019116263836622238,
+            0.023405466228723526,
+            0.02638322301208973,
+            0.0657266229391098,
+            -0.012952464632689953,
+            0.012435601092875004,
+            0.007614105474203825,
+            -0.021168293431401253,
+            -0.002835039282217622,
+            -0.02895982936024666,
+            0.0015014136442914605,
+            -0.028543252497911453,
+            -0.008917837403714657,
+            -0.034190189093351364,
+            -0.023606041446328163,
+            -0.03770795091986656,
+            0.01110101118683815,
+            -0.001489842077717185,
+            -0.026198076084256172,
+            0.01902369223535061,
+            -0.006495518609881401,
+            -0.010753863491117954,
+            -0.013477043248713017,
+            -0.026136361062526703,
+            0.010090425610542297,
+            -0.029345549643039703,
+            0.003562120720744133,
+            0.006237086374312639,
+            0.012119310908019543,
+            0.00829297211021185,
+            0.019548270851373672,
+            0.00910298340022564,
+            0.018468255177140236,
+            0.0013934121234342456,
+            -0.02726266346871853,
+            -0.026876943185925484,
+            0.010931294411420822,
+            0.011201297864317894,
+            0.014896491542458534,
+            -0.013947621919214725,
+            -0.006807951256632805,
+            0.02894440107047558,
+            0.0014638060238212347,
+            -0.010792435146868229,
+            -0.022016875445842743,
+            -0.014981349930167198,
+            -0.02153858356177807,
+            -0.025688927620649338,
+            -0.001375090447254479,
+            -0.02298888936638832,
+            -0.00899498164653778,
+            -0.007116526830941439,
+            -0.0008288150420412421,
+            -0.0005269894609227777,
+            0.0005901510012336075,
+            0.01817510835826397,
+            -0.03968283534049988,
+            -0.0009941923199221492,
+            0.0003018256102222949,
+            -0.04705779254436493,
+            0.024130620062351227,
+            -0.004995069000869989,
+            0.013014179654419422,
+            -0.027355235069990158,
+            0.020335137844085693,
+            0.007922681048512459,
+            0.011749019846320152,
+            -0.0015428784536197782,
+            -0.01718766614794731,
+            0.014742203988134861,
+            0.00575107941403985,
+            -0.018730545416474342,
+            0.0016566658159717917,
+            0.028203818947076797,
+            0.016693945974111557,
+            0.00786096602678299,
+            0.014240768738090992,
+            0.01198045164346695,
+            0.005241929553449154,
+            -0.00236060400493443,
+            0.047335512936115265,
+            -0.012967893853783607,
+            0.012204169295728207,
+            0.017527099698781967,
+            -0.005203357897698879,
+            -0.0003975322761107236,
+            0.006510947365313768,
+            -0.004740494303405285,
+            0.0062949443235993385,
+            0.003673979314044118,
+            0.029438121244311333,
+            -0.00906441081315279,
+            0.018452826887369156,
+            0.034961625933647156,
+            -0.014734489843249321,
+            0.0008148326887749135,
+            -0.005461789667606354,
+            0.007020097225904465,
+            0.009704705327749252,
+            -0.014256197027862072,
+            -0.014911920763552189,
+            0.028157532215118408,
+            -0.01199587993323803,
+            0.02953069470822811,
+            -0.004705779254436493,
+            0.014626488089561462,
+            0.03527020290493965,
+            0.00764882005751133,
+            0.004250630270689726,
+            -0.00861697643995285,
+            -0.0078146792948246,
+            0.013315041549503803,
+            -0.021260865032672882,
+            0.011679589748382568,
+            -0.021245436742901802,
+            0.027679240331053734,
+            -0.031690724194049835,
+            -0.005049069877713919,
+            -0.012636174447834492,
+            -0.01112415362149477,
+            0.02474777027964592,
+            0.0017627386841922998,
+            0.0034714764915406704,
+            -0.0057587940245866776,
+            -0.013369042426347733,
+            0.01110872533172369,
+            0.01965627260506153,
+            -0.007166670635342598,
+            0.012960179708898067,
+            0.004632492549717426,
+            -0.030286705121397972,
+            -0.014965921640396118,
+            0.030147844925522804,
+            0.0029777553863823414,
+            -0.021862588822841644,
+            0.014379628002643585,
+            -0.0062910872511565685,
+            -0.004401060752570629,
+            0.036535363644361496,
+            -0.009851278737187386,
+            -0.0074328170157969,
+            -0.0009223520755767822,
+            -0.001847596955485642,
+            -0.001201516599394381,
+            0.02394547499716282,
+            0.021785443648695946,
+            -0.02641407959163189,
+            -0.007999825291335583,
+            -0.014009336940944195,
+            0.007498389575630426,
+            -0.013816476799547672,
+            0.02371404320001602,
+            0.031783297657966614,
+            -0.00948870275169611,
+            0.00596708245575428,
+            0.004536062944680452,
+            0.013199325650930405,
+            -0.01517421007156372,
+            -0.023374609649181366,
+            -0.0017212738748639822,
+            0.005943939555436373,
+            -0.01569107361137867,
+            -0.006746236234903336,
+            -0.04384860768914223,
+            -0.011749019846320152,
+            -0.0026730368845164776,
+            -0.029268404468894005,
+            -0.04011484235525131,
+            -0.031845010817050934,
+            -0.005546648055315018,
+            0.02052028477191925,
+            0.016771089285612106,
+            0.0017193452222272754,
+            0.02386832982301712,
+            -0.005724078975617886,
+            0.005623792298138142,
+            0.021924303844571114,
+            0.00011264218483120203,
+            0.015073922462761402,
+            -0.0006962239276617765,
+            0.038170814514160156,
+            0.014665059745311737,
+            -0.0013162682298570871,
+            -0.05671621486544609,
+            -0.0188385471701622,
+            0.012937036342918873,
+            0.021168293431401253,
+            -0.0041271997615695,
+            0.02139972522854805,
+            -0.005245787091553211,
+            -0.020859716460108757,
+            -0.012543601915240288,
+            0.007035525981336832,
+            0.0037067655939608812,
+            0.007733678445219994,
+            0.0048176380805671215,
+            0.0036103357560932636,
+            0.029345549643039703,
+            0.026753513142466545,
+            0.00048239060561172664,
+            0.008940980769693851,
+            0.04224401339888573,
+            -0.006121370475739241,
+            0.0008119398262351751,
+            -0.00742510287091136,
+            0.022402595728635788,
+            -0.018653400242328644,
+            0.02377575822174549,
+            0.019995706155896187,
+            0.008771263994276524,
+            0.023467181250452995,
+            0.016169367358088493,
+            -0.00515707116574049,
+            -0.009226413443684578,
+            0.032647307962179184,
+            -0.010437573306262493,
+            -0.029098687693476677,
+            -0.007228385657072067,
+            -0.012975607998669147,
+            0.011826164089143276,
+            0.005608363542705774,
+            0.01191873662173748,
+            0.0033789039589464664,
+            0.0006263122195377946,
+            -0.02129172347486019,
+            -0.011949594132602215,
+            -0.00022540490317624062,
+            0.007760678883641958,
+            0.011193583719432354,
+            0.017465384677052498,
+            -0.0034406189806759357,
+            -0.018607115373015404,
+            0.033387891948223114,
+            -0.020782573148608208,
+            0.041565146297216415,
+            0.00011203949543414637,
+            -0.023189464583992958,
+            0.004038484301418066,
+            -0.00030399527167901397,
+            0.023405466228723526,
+            -0.009982423856854439,
+            -0.019069978967308998,
+            0.017943676561117172,
+            0.06066598370671272,
+            0.012273598462343216,
+            -0.016231082379817963,
+            -0.00787639431655407,
+            -0.02970041148364544,
+            0.01806710660457611,
+            0.006171513814479113,
+            0.026630083099007607,
+            0.02800324372947216,
+            0.021662013605237007,
+            -0.0042776307091116905,
+            0.02314317785203457,
+            -0.014626488089561462,
+            0.010530145838856697,
+            -0.004532205406576395,
+            0.02295803278684616,
+            0.0036450503394007683,
+            -0.013500186614692211,
+            0.034282758831977844,
+            0.0021793157793581486,
+            -0.002719323383644223,
+            -0.015590786933898926,
+            0.00575107941403985,
+            0.004875496029853821,
+            -0.00640294561162591,
+            0.02633693628013134,
+            0.03059528023004532,
+            0.003114685881882906,
+            -0.021631157025694847,
+            -0.010993009433150291,
+            0.016801945865154266,
+            0.020319709554314613,
+            0.0015920577570796013,
+            0.016925377771258354,
+            0.013816476799547672,
+            0.033295318484306335,
+            -0.0029989699833095074,
+            -0.0008153148810379207,
+            0.0041271997615695,
+            -0.0058937957510352135,
+            0.005461789667606354,
+            0.011417300440371037,
+            0.034190189093351364,
+            -0.00020033313194289804,
+            -0.018499113619327545,
+            0.009581275284290314,
+            0.011826164089143276,
+            -0.039034824818372726,
+            0.004393346607685089,
+            -0.022510597482323647,
+            0.0006914024124853313,
+            0.00992070883512497,
+            -0.022865459322929382,
+            -0.021615726873278618,
+            0.004046198911964893,
+            0.013029608875513077,
+            0.018360253423452377,
+            0.0156293585896492,
+            -0.019378554075956345,
+            -0.014194482006132603,
+            0.02718551829457283,
+            0.002271888544782996,
+            0.009781849570572376,
+            0.024346621707081795,
+            -0.006599662825465202,
+            0.0015824147267267108,
+            0.019116263836622238,
+            0.004138771444559097,
+            0.03150557726621628,
+            -0.002854325110092759,
+            -0.028311820700764656,
+            0.012466458603739738,
+            0.006210085935890675,
+            -0.009650705382227898,
+            -0.006896666716784239,
+            -0.027694668620824814,
+            -0.015035350807011127,
+            -0.030240418389439583,
+            0.003689408302307129,
+            -0.02305060438811779,
+            -0.0016373797552660108,
+            0.012235026806592941,
+            -0.011471301317214966,
+            0.006437660660594702,
+            -0.013777905143797398,
+            -0.02389918826520443,
+            -0.003689408302307129,
+            -0.0001803239283617586,
+            -0.012975607998669147,
+            -0.03033299185335636,
+            -0.02132258005440235,
+            -0.017835674807429314,
+            -0.014317912049591541,
+            -0.012073024176061153,
+            -0.019532842561602592,
+            -0.0008567796903662384,
+            0.02056656964123249,
+            0.007907251827418804,
+            -0.035085055977106094,
+            0.004208201076835394,
+            -0.004289202392101288,
+            -0.010128997266292572,
+            0.008871551603078842,
+            -0.02135343849658966,
+            -0.00038403208600357175,
+            0.02223287895321846,
+            0.011440443806350231,
+            -0.0165859442204237,
+            -0.004188915248960257,
+            -0.010954436846077442,
+            0.03059528023004532,
+            0.0094192735850811,
+            -0.008223542012274265,
+            -0.0058860816061496735,
+            0.024192335084080696,
+            -0.0023471040185540915,
+            -0.00023818186309654266,
+            -0.0024821057450026274,
+            -0.0009724955889396369,
+            0.0008399044745601714,
+            -0.024269478395581245,
+            0.030857570469379425,
+            0.00450520496815443,
+            0.002742466516792774,
+            -0.024979202076792717,
+            0.024315765127539635,
+            0.02314317785203457,
+            -0.030764997005462646,
+            0.0073749590665102005,
+            0.0012931249802932143,
+            0.010653575882315636,
+            0.00905669666826725,
+            0.0026267506182193756,
+            -0.008809836581349373,
+            0.03221530467271805,
+            -0.025256920605897903,
+            0.008123255334794521,
+            0.012929322198033333,
+            0.040053125470876694,
+            0.02226373739540577,
+            -0.007301672361791134,
+            0.009473273530602455,
+            0.020119136199355125,
+            0.013515615835785866,
+            0.017835674807429314,
+            -0.017665958032011986,
+            -0.01272874791175127,
+            0.0001393412094330415,
+            -0.0038109098095446825,
+            0.010468430817127228,
+            0.0013490543933585286,
+            -0.03155186399817467,
+            0.005631506443023682,
+            0.01067671924829483,
+            -0.007560104597359896,
+            -0.019285980612039566,
+            0.023667756468057632,
+            -0.004019198473542929,
+            0.007020097225904465,
+            -0.003754980629310012,
+            -0.006499375682324171,
+            0.011571588926017284,
+            -0.05936996266245842,
+            -0.020057421177625656,
+            0.008871551603078842,
+            0.0053036450408399105,
+            0.01447220053523779,
+            0.010020995512604713,
+            -0.02963869646191597,
+            -0.007197528146207333,
+            -0.02874382585287094,
+            0.018344825133681297,
+            -0.00987442210316658,
+            0.020057421177625656,
+            0.0221248771995306,
+            -0.008493546396493912,
+            0.0058436524122953415,
+            -0.003068399615585804,
+            0.018637971952557564,
+            -0.012451029382646084,
+            0.022603169083595276,
+            -0.016801945865154266,
+            -0.015004493296146393,
+            -0.00944241601973772,
+            0.011386442929506302,
+            -0.007938110269606113,
+            0.01449534296989441,
+            -0.024238621816039085,
+            -0.02141515351831913,
+            0.007922681048512459,
+            0.013531044125556946,
+            0.014603344723582268,
+            -0.01806710660457611,
+            0.007999825291335583,
+            -0.005616077687591314,
+            0.01105472445487976,
+            0.006152227986603975,
+            0.009712420403957367,
+            -0.0003524513158481568,
+            -0.022649455815553665,
+            -0.008354687131941319,
+            0.007563961669802666,
+            0.0008644941262900829,
+            0.004250630270689726,
+            -0.008794407360255718,
+            -0.029021544381976128,
+            0.007787679322063923,
+            -0.014896491542458534,
+            -0.019347695633769035,
+            -0.01653965748846531,
+            -0.009689277037978172,
+            -0.010838720947504044,
+            0.0018543470650911331,
+            0.02389918826520443,
+            -0.020072849467396736,
+            -0.010607290081679821,
+            0.01323789730668068,
+            -0.013083609752357006,
+            0.016153937205672264,
+            0.0035949070006608963,
+            -0.007629534229636192,
+            0.00552350515499711,
+            0.0003452190721873194,
+            0.03684393689036369,
+            0.0019903131760656834,
+            -0.02152315527200699,
+            0.01312218140810728,
+            -0.015498214401304722,
+            0.02382204309105873,
+            0.007976681925356388,
+            -0.0021349580492824316,
+            0.011571588926017284,
+            0.024886630475521088,
+            0.0027366806752979755,
+            0.020443139597773552,
+            -0.011008437722921371,
+            -0.029484407976269722,
+            0.007961252704262733,
+            -0.008817550726234913,
+            -0.01991856098175049,
+            -0.0198722742497921,
+            0.003754980629310012,
+            -0.010931294411420822,
+            0.023575183004140854,
+            -0.022649455815553665,
+            -0.0003042363387066871,
+            0.00038089812733232975,
+            0.0015814504586160183,
+            -0.014888777397572994,
+            0.013585045002400875,
+            0.007968967780470848,
+            -0.017542527988553047,
+            0.00779925100505352,
+            -0.01652422919869423,
+            -0.02130715176463127,
+            -2.7196247174288146e-05,
+            -0.008516688831150532,
+            -0.0022622456308454275,
+            -0.017064236104488373,
+            -0.0030201845802366734,
+            0.00033726359833963215,
+            -0.0006306516006588936,
+            0.016956234350800514,
+            0.019162550568580627,
+            -0.007745250128209591,
+            0.01894654706120491,
+            0.003976769279688597,
+            0.02209402061998844,
+            0.0031648294534534216,
+            -0.006985382176935673,
+            -0.022402595728635788,
+            0.02642950788140297,
+            -0.02567349746823311,
+            -0.0024338907096534967,
+            0.01235074270516634,
+            -0.014904206618666649,
+            -0.0012005523312836885,
+            -0.0297312680631876,
+            -0.004678778816014528,
+            -0.0014011265011504292,
+            -0.0018177037127315998,
+            0.04551491513848305,
+            -0.003259330755099654,
+            -0.012967893853783607,
+            -0.0131067531183362,
+            0.0006841701688244939,
+            0.0066112345084548,
+            0.005006640683859587,
+            0.01640079915523529,
+            -0.01154073141515255,
+            -0.02391461655497551,
+            0.016030507162213326,
+            -0.005855224095284939,
+            0.01647794246673584,
+            0.020196279510855675,
+            0.0026286791544407606,
+            -0.03307931497693062,
+            0.012505030259490013,
+            -0.01905454881489277,
+            -0.00944241601973772,
+            0.019270552322268486,
+            0.02801867388188839,
+            -0.00867869146168232,
+            -0.01072300598025322,
+            0.011633303947746754,
+            -0.000418505776906386,
+            -0.01066900510340929,
+            -0.02479405701160431,
+            -0.009403844363987446,
+            -0.013762475922703743,
+            0.013731618411839008,
+            0.006113655865192413,
+            -0.02065914310514927,
+            0.006483946926891804,
+            -0.006414517294615507,
+            0.01230445597320795,
+            -0.01154073141515255,
+            0.003432904602959752,
+            0.018514541909098625,
+            0.02150772698223591,
+            -0.0230814628303051,
+            0.025827785953879356,
+            -0.027648381888866425,
+            0.021075719967484474,
+            -0.0008799228817224503,
+            0.026707226410508156,
+            -0.029484407976269722,
+            -0.018313968554139137,
+            -0.013006465509533882,
+            -0.011378728784620762,
+            -0.025195205584168434,
+            -0.006078941281884909,
+            0.01441048551350832,
+            0.013970764353871346,
+            0.009311271831393242,
+            0.026799799874424934,
+            -0.012096167542040348,
+            0.018653400242328644,
+            0.02064371481537819,
+            -0.01193416491150856,
+            0.005681650247424841,
+            0.011733590625226498,
+            -0.009558131918311119,
+            0.012813606299459934,
+            0.001909312093630433,
+            0.010445287451148033,
+            -0.033264461904764175,
+            0.022078590467572212,
+            0.014819348230957985,
+            -0.0037299087271094322,
+            -0.03872625157237053,
+            -0.0016798089491203427,
+            -0.020057421177625656,
+            -0.02315860614180565,
+            -0.0014310197439044714,
+            0.011957308277487755,
+            0.028682110831141472,
+            0.018622543662786484,
+            -0.036689650267362595,
+            -0.000610401330050081,
+            -0.01579907536506653,
+            -0.011232155375182629,
+            -0.014024765230715275,
+            0.012921607121825218,
+            -0.009018125012516975,
+            0.014958206564188004,
+            -0.0004170593456365168,
+            0.002379890065640211,
+            -0.003363475203514099,
+            -0.01231217011809349,
+            0.016138508915901184,
+            0.010275570675730705,
+            -0.009457845240831375,
+            0.010429858230054379,
+            0.013816476799547672,
+            0.0026074645575135946,
+            -0.004644064232707024,
+            -0.012597602792084217,
+            -0.02485577203333378,
+            0.011617874726653099,
+            -0.0018205966334789991,
+            0.007498389575630426,
+            -0.009797278791666031,
+            0.0034676194190979004,
+            -0.047366369515657425,
+            0.003726051654666662,
+            0.0032265447080135345,
+            0.030641566962003708,
+            0.011749019846320152,
+            0.018622543662786484,
+            0.014919634908437729,
+            0.0005795437609776855,
+            0.010591860860586166,
+            0.02127629518508911,
+            0.03786223754286766,
+            0.00537307420745492,
+            0.014896491542458534,
+            -5.394048002926866e-06,
+            -0.007563961669802666,
+            0.0074559603817760944,
+            -0.006125227548182011,
+            -0.02062828466296196,
+            -0.006360516417771578,
+            -0.00020177957776468247,
+            0.0008881194517016411,
+            -0.00860926229506731,
+            0.004308488219976425,
+            -0.017465384677052498,
+            -0.010838720947504044,
+            -0.02794152870774269,
+            0.00216581579297781,
+            -0.025118062272667885,
+            -0.015891648828983307,
+            -0.016308225691318512,
+            -0.010869579389691353,
+            -0.003953625913709402,
+            0.022772885859012604,
+            -0.0049063535407185555,
+            -0.033418748527765274,
+            -0.0037029082886874676,
+            0.02727809175848961,
+            -0.008269828744232655,
+            -0.010530145838856697,
+            0.011725876480340958,
+            0.010059568099677563,
+            -0.008022967725992203,
+            0.01362361665815115,
+            0.010229283943772316,
+            -0.009357557632029057,
+            -0.0013104823883622885,
+            -0.01560621615499258,
+            0.02963869646191597,
+            0.017110522836446762,
+            0.002422319259494543,
+            -0.004528348334133625,
+            -0.017449956387281418,
+            -0.0016084507806226611,
+            -0.0013914835872128606,
+            0.02480948530137539,
+            -0.021569442003965378,
+            0.027756383642554283,
+            -0.0031474721617996693,
+            -0.02227916568517685,
+            0.013747047632932663,
+            0.02055114135146141,
+            -0.02721637673676014,
+            0.0010732648661360145,
+            -0.006588091142475605,
+            0.015505928546190262,
+            -0.008223542012274265,
+            -0.0057665081694722176,
+            -0.010622718371450901,
+            0.007517675403505564,
+            0.013214753940701485,
+            -0.01967170089483261,
+            0.017542527988553047,
+            0.0110624385997653,
+            0.01280589122325182,
+            -0.02547292411327362,
+            0.013307327404618263,
+            -0.006213943008333445,
+            -0.01275189034640789,
+            -0.0020751715637743473,
+            -0.004825352691113949,
+            0.017573386430740356,
+            0.006059655454009771,
+            0.002231388119980693,
+            -0.00758324796333909,
+            0.003984483890235424,
+            -0.006244800519198179,
+            0.010514716617763042,
+            0.011432729661464691,
+            -0.020273422822356224,
+            -0.004377917852252722,
+            -0.013006465509533882,
+            -0.009349843487143517,
+            0.003344189142808318,
+            -0.007178242318332195,
+            0.03650450333952904,
+            0.00986670795828104,
+            0.004296916536986828,
+            0.02231002226471901,
+            -0.011139582842588425,
+            -0.012913892976939678,
+            0.01660137251019478,
+            0.09436244517564774,
+            3.441342414589599e-05,
+            -0.003969055134803057,
+            0.002773324027657509,
+            -0.010476144962012768,
+            -0.0008866729913279414,
+            -0.01235074270516634,
+            -0.018468255177140236,
+            0.008123255334794521,
+            -0.00514549994841218,
+            -0.021739158779382706,
+            -0.009149269200861454,
+            -0.020011134445667267,
+            0.017141379415988922,
+            0.003322974545881152,
+            0.012242740951478481,
+            0.013939906843006611,
+            0.01905454881489277,
+            0.0031899013556540012,
+            0.014819348230957985,
+            -0.009519560262560844,
+            0.007243814412504435,
+            -0.011347871273756027,
+            0.007263100706040859,
+            0.008123255334794521,
+            0.009210984222590923,
+            0.006244800519198179,
+            0.015012207441031933,
+            0.012003595009446144,
+            0.029006116092205048,
+            0.010283284820616245,
+            -0.0012179097393527627,
+            -0.005049069877713919,
+            -0.016693945974111557,
+            -0.003693265374749899,
+            0.00904898252338171,
+            -0.01984141767024994,
+            -0.022634027525782585,
+            -0.013693046756088734,
+            -0.002790681552141905,
+            0.0019382410682737827,
+            -0.008940980769693851,
+            0.023405466228723526,
+            0.026954086497426033,
+            -0.006800237111747265,
+            0.004269916098564863,
+            -0.007479103282094002,
+            -0.0011108724866062403,
+            -0.00866326317191124,
+            -0.010514716617763042,
+            -0.0013066251995041966,
+            -0.0013596616918221116,
+            0.005295930430293083,
+            -0.021939732134342194,
+            -0.013677617534995079,
+            -0.004107913933694363,
+            0.009766421280801296,
+            -0.00040548774995841086,
+            0.0024956059642136097,
+            0.011363300494849682,
+            -0.01393219269812107,
+            -0.004944925662130117,
+            0.00558136310428381,
+            -0.009550417773425579,
+            -0.008385544642806053,
+            0.006275658495724201,
+            0.012134739197790623,
+            -0.0010173354530707002,
+            -0.011725876480340958,
+            -0.012613032013177872,
+            -0.010453001596033573,
+            0.006109798792749643,
+            0.0221248771995306,
+            -0.007548532914370298,
+            0.008501260541379452,
+            -0.008424116298556328,
+            0.003648907644674182,
+            -0.017727673053741455,
+            0.003157115075737238,
+            -0.0038880538195371628,
+            -0.0021542441099882126,
+            0.00992842298001051,
+            -0.006607376970350742,
+            -0.00817725621163845,
+            0.00011372702283551916,
+            0.0006518661393783987,
+            -0.007228385657072067,
+            0.02132258005440235,
+            0.028203818947076797,
+            0.029931843280792236,
+            0.002574678510427475,
+            -0.02629064954817295,
+            0.02312774769961834,
+            -0.009681562893092632,
+            -0.009573561139404774,
+            0.009156984277069569,
+            0.0005901510012336075,
+            -0.009411558508872986,
+            -0.01896197721362114,
+            -0.002981612691655755,
+            0.005230357870459557,
+            0.019162550568580627,
+            0.00224681687541306,
+            -0.0018312039319425821,
+            -0.022880887612700462,
+            0.003407832933589816,
+            0.02727809175848961,
+            0.0030780425295233727,
+            -0.009789563715457916,
+            -0.008339257910847664,
+            0.020813431590795517,
+            -0.003297902876511216,
+            0.004624778404831886,
+            0.003791623981669545,
+            -0.008339257910847664,
+            -0.003342260606586933,
+            -0.004941068589687347,
+            0.0026518225204199553,
+            -0.01805167831480503,
+            0.009758706204593182,
+            0.01116272620856762,
+            -0.010499288327991962,
+            0.004215915687382221,
+            -0.012705604545772076,
+            0.006210085935890675,
+            -0.0004380328464321792,
+            0.0077761076390743256,
+            -0.0015698788920417428,
+            -0.012289026752114296,
+            -0.02971583977341652,
+            -0.01185702160000801,
+            -0.01817510835826397,
+            -0.014634202234447002,
+            0.0007473317673429847,
+            0.02784895710647106,
+            -0.004266059026122093,
+            0.007563961669802666,
+            0.001351947314105928,
+            -0.009974709711968899,
+            0.012188740074634552,
+            -0.020026562735438347,
+            0.01480391900986433,
+            -0.013639045879244804,
+            0.013091323897242546,
+            0.015451927669346333,
+            0.01797453500330448,
+            -0.009396130219101906,
+            0.0004013894940726459,
+            -0.0020983146969228983,
+            -0.008100111968815327,
+            -0.00537307420745492,
+            -0.012813606299459934,
+            -0.008053826168179512,
+            -0.017635101452469826,
+            0.015328497625887394,
+            0.024099761620163918,
+            0.0009990137768909335,
+            0.01728023961186409,
+            0.0008259221212938428,
+            -1.7749129256117158e-05,
+            -0.01111643947660923,
+            -0.004092485178261995,
+            0.014348769560456276,
+            0.005068355705589056,
+            -0.025210633873939514,
+            -0.0036334788892418146,
+            0.01652422919869423,
+            0.006102084647864103,
+            -0.01393219269812107,
+            0.013469329103827477,
+            -0.003799338359385729,
+            -0.0021812445484101772,
+            -0.01355418749153614,
+            -0.0009030660730786622,
+            0.007675820495933294,
+            0.0052727870643138885,
+            -0.021122006699442863,
+            -0.008786693215370178,
+            0.0032496878411620855,
+            0.012435601092875004,
+            -0.004258344415575266,
+            0.00309347128495574,
+            0.01885397545993328,
+            -0.00536150299012661,
+            0.03465305268764496,
+            -0.012520458549261093,
+            -0.014503058046102524,
+            0.0020713144913315773,
+            0.020134564489126205,
+            -0.0062563722021877766,
+            -0.0007400995236821473,
+            0.011340157128870487,
+            0.0005506147863343358,
+            -0.008161826990544796,
+            0.02974669821560383,
+            0.006040369160473347,
+            0.01738824136555195,
+            -0.0010877293534576893,
+            0.014595630578696728,
+            -0.001114729675464332,
+            0.0002372175658820197,
+            -0.0009802100248634815,
+            -0.01035271491855383,
+            -0.005068355705589056,
+            -0.020026562735438347,
+            -0.006009511649608612,
+            -0.020273422822356224,
+            0.029160402715206146,
+            0.012389314360916615,
+            0.007494532503187656,
+            -0.02638322301208973,
+            0.002289246069267392,
+            0.006441517733037472,
+            -0.007170527707785368,
+            -0.006900524254888296,
+            -0.012204169295728207,
+            -0.022032305598258972,
+            0.009318985976278782,
+            0.007259243167936802,
+            0.010568717494606972,
+            0.004250630270689726,
+            -0.010182998143136501,
+            0.0032034015748649836,
+            -0.018529970198869705,
+            0.012705604545772076,
+            0.0024801772087812424,
+            0.003133971942588687,
+            0.0016942734364420176,
+            -0.0034483333583921194,
+            0.014665059745311737,
+            -3.730631942744367e-05,
+            0.011039295233786106,
+            0.014094195328652859,
+            0.01111643947660923,
+            0.007976681925356388,
+            0.004401060752570629,
+            0.010576431639492512,
+            -0.00816954206675291,
+            -0.030734140425920486,
+            -0.0264449380338192,
+            0.004080913495272398,
+            -0.01110872533172369,
+            -0.009627562016248703,
+            0.007984396070241928,
+            0.011139582842588425,
+            0.01069214753806591,
+            -0.016323653981089592,
+            -0.005666221491992474,
+            -0.0043200599029660225,
+            -2.2917169189895503e-05,
+            -0.011787591502070427,
+            -0.0007203313871286809,
+            0.012157882563769817,
+            0.002040456747636199,
+            0.006491661071777344,
+            0.01894654706120491,
+            -0.00906441081315279,
+            -0.007826250977814198,
+            0.01567564532160759,
+            -0.0074328170157969,
+            -0.019625414162874222,
+            -0.008910123258829117,
+            0.016215652227401733,
+            -0.008215827867388725,
+            0.011286156252026558,
+            0.009565846994519234,
+            0.010491574183106422,
+            -0.016632230952382088,
+            -0.007999825291335583,
+            0.02804953046143055,
+            0.03230787441134453,
+            -0.010807863436639309,
+            -0.002273817081004381,
+            0.002725109225139022,
+            -0.012127025052905083,
+            0.021122006699442863,
+            0.00860926229506731,
+            -0.010113568976521492,
+            0.016987092792987823,
+            0.0026903944090008736,
+            0.008138683624565601,
+            -0.0036238357424736023,
+            0.0012304455740377307,
+            -0.017604243010282516,
+            -0.018375683575868607,
+            -0.00948098860681057,
+            0.009164698421955109,
+            0.006742379162460566,
+            0.0073749590665102005,
+            8.919765969039872e-05,
+            0.011093296110630035,
+            -0.011332442983984947,
+            0.005604506004601717,
+            0.0008119398262351751,
+            0.027740955352783203,
+            -0.029885556548833847,
+            0.019903132691979408,
+            -0.003267045132815838,
+            -0.0018283110111951828,
+            -0.013585045002400875,
+            -0.012844463810324669,
+            -0.020828859880566597,
+            -0.011209012009203434,
+            0.014973635785281658,
+            0.016076793894171715,
+            -0.0005472397315315902,
+            -0.006908238399773836,
+            -0.002619036240503192,
+            0.0057973661459982395,
+            -0.006861952133476734,
+            0.002761752577498555,
+            -0.0017646672204136848,
+            0.0014956278027966619,
+            -0.0020751715637743473,
+            -0.015559929423034191,
+            0.007444388698786497,
+            -0.003652764717116952,
+            -0.0015669859712943435,
+            -0.018761401996016502,
+            0.03604163974523544,
+            -0.013060466386377811,
+            0.00904126837849617,
+            0.014155910350382328,
+            0.0018128822557628155,
+            -0.011471301317214966,
+            -0.013777905143797398,
+            0.011216727085411549,
+            -0.02149229682981968,
+            -0.021955160424113274,
+            -0.004096342250704765,
+            0.0223563089966774,
+            -0.015760503709316254,
+            -0.004046198911964893,
+            0.019532842561602592,
+            -0.006271800957620144,
+            -0.009658419527113438,
+            -0.022680314257740974,
+            -0.016231082379817963,
+            -1.1903066479135305e-05,
+            -0.016801945865154266,
+            -0.017912819981575012,
+            -0.017079664394259453,
+            -0.007864823564887047,
+            -0.01970255747437477,
+            -0.0006914024124853313,
+            0.0033924041781574488,
+            -0.01805167831480503,
+            -0.022695742547512054,
+            0.01275189034640789,
+            0.00818497035652399,
+            -0.0026306079234927893,
+            0.008524403907358646,
+            0.00596322538331151,
+            0.0027231804560869932,
+            -2.3640393919777125e-05,
+            0.007035525981336832,
+            0.02892897091805935,
+            0.017017949372529984,
+            0.006599662825465202,
+            0.017002521082758904,
+            -0.0062988013960421085,
+            -0.01029099989682436,
+            -0.010321857407689095,
+            -0.004169628955423832,
+            0.010445287451148033,
+            -0.019964847713708878,
+            0.033357031643390656,
+            -0.006202371325343847,
+            -0.018730545416474342,
+            -0.006410660222172737,
+            0.00987442210316658,
+            -0.010476144962012768,
+            -0.019532842561602592,
+            -0.007922681048512459,
+            -0.006885095499455929,
+            0.016092222183942795,
+            -0.0032901884987950325,
+            0.00898726750165224,
+            0.010113568976521492,
+            -0.015421070158481598,
+            0.010823292657732964,
+            -0.006240943446755409,
+            -0.014271626248955727,
+            0.025195205584168434,
+            0.022448882460594177,
+            -0.005064498633146286,
+            -0.007826250977814198,
+            -0.002401104662567377,
+            0.021245436742901802,
+            0.008940980769693851,
+            -0.0003459422732703388,
+            0.009172412566840649,
+            0.005608363542705774,
+            0.003905411111190915,
+            0.0024608911480754614,
+            0.001322053954936564,
+            0.011224441230297089,
+            0.03311017155647278,
+            0.030147844925522804,
+            -0.008046111091971397,
+            -0.002551535377278924,
+            0.01528992597013712,
+            0.00899498164653778,
+            0.028280962258577347,
+            -0.0031455436255782843,
+            -0.035023342818021774,
+            0.001998027553781867,
+            -0.008447259664535522,
+            0.014703632332384586,
+            0.0006648842245340347,
+            0.015876220539212227,
+            0.0017540600383654237,
+            -0.0014879134250804782,
+            0.018514541909098625,
+            0.019316839054226875,
+            0.00617922842502594,
+            -0.017635101452469826,
+            -0.01312989555299282,
+            0.008740406483411789,
+            0.006514804437756538,
+            -0.005685507319867611,
+            -0.012327599339187145,
+            -0.00015826558228582144,
+            0.029808413237333298,
+            -0.010198426432907581,
+            -0.017449956387281418,
+            0.01242017187178135,
+            0.011525302194058895,
+            0.007787679322063923,
+            -0.025951215997338295,
+            0.004169628955423832,
+            0.008354687131941319,
+            0.014819348230957985,
+            -0.00899498164653778,
+            -0.015004493296146393,
+            -0.000610401330050081,
+            0.011193583719432354,
+            -0.016261938959360123,
+            0.001589164836332202,
+            0.02218659222126007,
+            -0.02053571306169033,
+            0.03122786059975624,
+            -0.010059568099677563,
+            0.017804818227887154,
+            0.003477262333035469,
+            0.020088277757167816,
+            0.006495518609881401,
+            0.0012304455740377307,
+            0.004161914810538292,
+            0.013885905966162682,
+            -0.003245830535888672,
+            0.011795305646955967,
+            -0.01485020574182272,
+            -0.004536062944680452,
+            0.0003758355451282114,
+            -0.0021581011824309826,
+            -0.0024512482341378927,
+            0.008215827867388725,
+            0.02226373739540577,
+            -0.01581450365483761,
+            0.0015843433793634176,
+            -0.005284358747303486,
+            0.006800237111747265,
+            0.023312894627451897,
+            -0.011903307400643826,
+            -0.009519560262560844,
+            0.012890749610960484,
+            0.019085407257080078,
+            -0.0053537883795797825,
+            -0.027478665113449097,
+            -0.0015043065650388598,
+            0.014680488966405392,
+            0.017789389938116074,
+            -0.0031821869779378176,
+            0.036658793687820435,
+            -0.013060466386377811,
+            -0.024038046598434448,
+            -0.008856122381985188,
+            0.03789309412240982,
+            0.004605492111295462,
+            -0.005095356144011021,
+            0.0015264854300767183,
+            -0.010152140632271767,
+            0.008393258787691593,
+            0.015066208317875862,
+            -0.00662280572578311,
+            -0.01976427435874939,
+            0.01726480945944786,
+            -0.0153053542599082,
+            -0.009835850447416306,
+            -0.002852396573871374,
+            -0.005477218423038721,
+            -0.009450131095945835,
+            0.012481886893510818,
+            -0.01449534296989441,
+            0.0003293081244919449,
+            0.000391987559851259,
+            0.0029063974507153034,
+            0.004636350087821484,
+            0.013855048455297947,
+            -0.010831006802618504,
+            0.007679677568376064,
+            0.016061365604400635,
+            -0.0013828048249706626,
+            0.016724802553653717,
+            0.007131956052035093,
+            -0.007093383930623531,
+            -0.013446185737848282,
+            0.03212273120880127,
+            -0.0024531767703592777,
+            0.016076793894171715,
+            -0.011401872150599957,
+            -0.008215827867388725,
+            -0.02559635415673256,
+            -0.010576431639492512,
+            0.028358107432723045,
+            0.004952639807015657,
+            -0.024099761620163918,
+            0.003521620063111186,
+            0.0347764827311039,
+            0.00823125708848238,
+            0.006507089827209711,
+            0.008431831374764442,
+            0.009542703628540039,
+            0.005137785337865353,
+            -0.0022873173002153635,
+            -0.0016113437013700604,
+            -0.00553121929988265,
+            0.009581275284290314,
+            0.0021214578300714493,
+            -0.005130071192979813,
+            -0.013099038042128086,
+            -0.01360818836838007,
+            -0.028280962258577347,
+            0.005928510800004005,
+            -0.03517762944102287,
+            0.015876220539212227,
+            -0.02136886678636074,
+            0.02385290153324604,
+            -0.004871638957411051,
+            -0.005616077687591314,
+            -0.020211707800626755,
+            0.008539832197129726,
+            -0.002129172207787633,
+            0.006352802272886038,
+            0.002316246274858713,
+            -0.004316202364861965,
+            0.012204169295728207,
+            -0.012188740074634552,
+            -0.024254050105810165,
+            0.028527824208140373,
+            -0.006067369598895311,
+            -0.02480948530137539,
+            0.0025843214243650436,
+            0.003236187621951103,
+            0.006499375682324171,
+            0.005635363515466452,
+            -0.03227701783180237,
+            0.0025669641327112913,
+            -0.003905411111190915,
+            0.009234127588570118,
+            0.006171513814479113,
+            -0.028666682541370392,
+            0.023544326424598694,
+            0.008917837403714657,
+            -0.0013567687710747123,
+            -0.007891823537647724,
+            -0.018715115264058113,
+            0.00014283679774962366,
+            0.0042390585877001286,
+            0.011687304824590683,
+            0.004354774486273527,
+            0.0015592715935781598,
+            -0.011093296110630035,
+            0.0029642554000020027,
+            -0.011216727085411549,
+            0.016647659242153168,
+            -0.020134564489126205,
+            0.011633303947746754,
+            -0.005018212366849184,
+            -0.005110784899443388,
+            -0.0036373359616845846,
+            0.007899537682533264,
+            -0.01404019445180893,
+            0.02872839756309986,
+            0.005133928265422583,
+            -0.014533915556967258,
+            -0.012065310031175613,
+            0.012026737444102764,
+            0.0069930967874825,
+            0.011031581088900566,
+            0.0014358413172885776,
+            -0.005677792709320784,
+            0.0035794780123978853,
+            0.0027771813329309225,
+            0.01231217011809349,
+            -0.0030298277270048857,
+            0.0188385471701622,
+            0.011401872150599957,
+            -0.0030626137740910053,
+            -0.0058976528234779835,
+            -0.00829297211021185,
+            2.452935768815223e-05,
+            -0.01715680956840515,
+            0.011371014639735222,
+            0.0068349516950547695,
+            -0.010460715740919113,
+            0.002080957405269146,
+            0.005006640683859587,
+            -0.001408840878866613,
+            0.013361327350139618,
+            -0.001426198286935687,
+            0.0005809901631437242,
+            1.5458919733646326e-05,
+            0.007633391302078962,
+            -0.010576431639492512,
+            -0.024485481902956963,
+            -0.031042715534567833,
+            0.004034627228975296,
+            -0.0005313287838362157,
+            0.015081637538969517,
+            0.021260865032672882,
+            -0.015374784357845783,
+            -0.00021925750479567796,
+            0.005943939555436373,
+            -0.011332442983984947,
+            -0.015012207441031933,
+            -0.017727673053741455,
+            0.013214753940701485,
+            -0.016740230843424797,
+            0.002659536898136139,
+            0.019548270851373672,
+            -0.025226064026355743,
+            0.012975607998669147,
+            -0.034251902252435684,
+            -0.007228385657072067,
+            -0.011039295233786106,
+            -0.014209911227226257,
+            0.005045212805271149,
+            0.016894519329071045,
+            -0.014842490665614605,
+            -0.0005539897829294205,
+            -0.004528348334133625,
+            0.0012420171406120062,
+            -0.022896315902471542,
+            -0.02136886678636074,
+            0.016123080626130104,
+            -0.01888483203947544,
+            -0.02135343849658966,
+            -0.009558131918311119,
+            -0.0014609131030738354,
+            -0.001360625959932804,
+            -0.014711346477270126,
+            0.019594557583332062,
+            0.0005718293250538409,
+            0.012466458603739738,
+            6.394508091034368e-05,
+            0.015845362097024918,
+            -0.00558136310428381,
+            -0.006865809205919504,
+            -0.006703807041049004,
+            0.0036643364001065493,
+            -0.011201297864317894,
+            0.0029873985331505537,
+            -0.0005356681067496538,
+            0.0015168423997238278,
+            0.0024724628310650587,
+            -0.004358631558716297,
+            0.008061540313065052,
+            0.0003941572504118085,
+            0.016879091039299965,
+            -0.0017627386841922998,
+            0.008470403030514717,
+            0.014541629701852798,
+            0.0030433277133852243,
+            -0.013585045002400875,
+            -0.0038282673340290785,
+            0.011795305646955967,
+            0.002084814477711916,
+            0.006896666716784239,
+            0.022742029279470444,
+            -0.0031436148565262556,
+            0.014163624495267868,
+            0.0012092309771105647,
+            -8.505418918502983e-06,
+            0.005118499509990215,
+            -0.008979552425444126,
+            0.004246773198246956,
+            -0.0013500186614692211,
+            -0.0006889916840009391,
+            -0.002725109225139022,
+            0.014749918133020401,
+            0.001008656807243824,
+            -0.013392184861004353,
+            -0.010221569798886776,
+            -0.0018890618812292814,
+            0.03203015774488449,
+            0.00025168206775560975,
+            0.010807863436639309,
+            -0.02882097102701664,
+            -0.00742895994335413,
+            0.009380700998008251,
+            -0.025148918852210045,
+            -0.020921431481838226,
+            0.005650792270898819,
+            0.012682461179792881,
+            -0.003945911768823862,
+            -0.0004481579817365855,
+            0.04890924692153931,
+            0.020165421068668365,
+            -0.0035987640731036663,
+            -0.004802209325134754,
+            -0.0063026584684848785,
+            -0.0026884658727794886,
+            0.011617874726653099,
+            -0.012204169295728207,
+            -0.006148370914161205,
+            -0.0069275242276489735,
+            0.004474347457289696,
+            0.007830108515918255,
+            -0.013662189245223999,
+            -0.006044226698577404,
+            0.002401104662567377,
+            -0.011147296987473965,
+            -0.013723904266953468,
+            0.006588091142475605,
+            -0.0063412305898964405,
+            3.5859869967680424e-05,
+            0.006453089416027069,
+            -0.012250455096364021,
+            0.01155615970492363,
+            0.004983497317880392,
+            0.019100835546851158,
+            -0.01653965748846531,
+            -0.001184159191325307,
+            0.00904126837849617,
+            -0.015953363850712776,
+            0.0031841155141592026,
+            -0.013145324774086475,
+            0.006167656742036343,
+            -0.0012063381727784872,
+            0.0058513665571808815,
+            -0.002252602484077215,
+            -0.00450520496815443,
+            0.007189814001321793,
+            -0.03320274502038956,
+            -0.009812707081437111,
+            0.013685332611203194,
+            -0.025874072685837746,
+            0.010483859106898308,
+            0.004748208448290825,
+            0.009511846117675304,
+            -0.007961252704262733,
+            0.011949594132602215,
+            0.00112340843770653,
+            -0.015374784357845783,
+            0.020890574902296066,
+            0.007675820495933294,
+            0.013762475922703743,
+            0.005708650220185518,
+            0.005928510800004005,
+            0.009990138001739979,
+            0.013515615835785866,
+            -0.012119310908019543,
+            -0.01636994071304798,
+            0.011455873027443886,
+            0.003450262127444148,
+            -5.23433591297362e-05,
+            -0.01230445597320795,
+            -0.0148656340315938,
+            0.01066900510340929,
+            0.01279046293348074,
+            0.012921607121825218,
+            0.01275189034640789,
+            0.006981525104492903,
+            0.015343926846981049,
+            -0.02062828466296196,
+            -0.012767319567501545,
+            -0.025904929265379906,
+            0.014965921640396118,
+            -0.0026576081290841103,
+            -0.001909312093630433,
+            -0.017496241256594658,
+            -0.005072213243693113,
+            -0.02237173728644848,
+            0.007344101555645466,
+            0.016123080626130104,
+            0.0009990137768909335,
+            -0.0024551053065806627,
+            0.02389918826520443,
+            -0.013739332556724548,
+            0.04357088729739189,
+            -0.007521532475948334,
+            -0.019193409010767937,
+            -0.010831006802618504,
+            -0.004593920893967152,
+            0.013893621042370796,
+            0.005338359624147415,
+            -0.00617922842502594,
+            0.003953625913709402,
+            0.020149992778897285,
+            -0.001371233258396387,
+            -0.002293103141710162,
+            -0.011965022422373295,
+            0.01441048551350832,
+            -0.006534090265631676,
+            0.008539832197129726,
+            -0.011085581965744495,
+            0.019471125677227974,
+            0.00515707116574049,
+            0.010699862614274025,
+            0.009588989429175854,
+            -0.014348769560456276,
+            -0.014603344723582268,
+            -0.00595936831086874,
+            0.00558136310428381,
+            -0.019316839054226875,
+            -0.009095268324017525,
+            0.03465305268764496,
+            -7.075544272083789e-05,
+            -0.006676806602627039,
+            -0.005114641971886158,
+            0.011471301317214966,
+            -0.009288128465414047,
+            -0.001129194162786007,
+            -0.005716364830732346,
+            -0.010877293534576893,
+            -0.001112801139242947,
+            -0.005785794463008642,
+            -0.006526376120746136,
+            -0.013261040672659874,
+            -0.0058976528234779835,
+            0.0014483771519735456,
+            -0.007668105885386467,
+            -0.0006889916840009391,
+            0.004532205406576395,
+            0.012512744404375553,
+            0.0110624385997653,
+            -0.00276753818616271,
+            0.006865809205919504,
+            -0.0012603388167917728,
+            0.018637971952557564,
+            0.0135464733466506,
+            -0.013739332556724548,
+            -0.019378554075956345,
+            -0.0028793970122933388,
+            0.00617537135258317,
+            0.014611058868467808,
+            0.012890749610960484,
+            -0.006032655015587807,
+            -0.0003430493816267699,
+            -0.024485481902956963,
+            -0.005176357459276915,
+            0.008547547273337841,
+            -0.01273646205663681,
+            -0.007579390425235033,
+            -0.0029951129108667374,
+            -0.0010154069168493152,
+            -0.004485919140279293,
+            -0.016925377771258354,
+            -0.000276512757409364,
+            0.010877293534576893,
+            -0.006483946926891804,
+            0.017110522836446762,
+            0.018622543662786484,
+            -0.014973635785281658,
+            -0.007251529023051262,
+            0.0009271735325455666,
+            -0.03236959129571915,
+            0.0024146048817783594,
+            -0.000498060486279428,
+            0.02067457139492035,
+            0.0053152162581682205,
+            0.0005901510012336075,
+            0.002809967380017042,
+            -0.006460803560912609,
+            0.002123386599123478,
+            0.007309386972337961,
+            -0.008840694092214108,
+            -0.01073072012513876,
+            -0.010468430817127228,
+            -0.011039295233786106,
+            -0.009735562838613987,
+            -0.0025631068274378777,
+            -0.022572312504053116,
+            0.029931843280792236,
+            0.011795305646955967,
+            0.003629621583968401,
+            -0.01069214753806591,
+            -0.0010134782642126083,
+            0.018637971952557564,
+            -0.0035235488321632147,
+            0.023729471489787102,
+            0.013006465509533882,
+            -0.011826164089143276,
+            -0.017125951126217842,
+            0.023467181250452995,
+            -0.020983148366212845,
+            0.0062525151297450066,
+            -0.003961340524256229,
+            0.0019044906366616488,
+            0.003965197596698999,
+            -0.011417300440371037,
+            0.008022967725992203,
+            -0.016940806061029434,
+            -0.00298354122787714,
+            -0.00433934573084116,
+            0.006217800546437502,
+            -0.002638322301208973,
+            0.026768941432237625,
+            -0.0019739202689379454,
+            -0.004042341839522123,
+            0.013099038042128086,
+            0.039127398282289505,
+            0.029283834621310234,
+            0.018761401996016502,
+            -0.006665234919637442,
+            -0.006194657180458307,
+            -0.012705604545772076,
+            0.004678778816014528,
+            -0.017095094546675682,
+            -0.003265116596594453,
+            0.004663350060582161,
+            0.0010684434091672301,
+            -0.009704705327749252,
+            -0.0003309956518933177,
+            -0.019162550568580627,
+            0.0239609032869339,
+            -0.01236617099493742,
+            0.00011173815437359735,
+            -0.010962151922285557,
+            -0.01404790859669447,
+            -0.018468255177140236,
+            0.0023876044433563948,
+            -0.0072823865339159966,
+            -0.01891569048166275,
+            0.0024782486725598574,
+            -0.019579127430915833,
+            -0.03690565377473831,
+            0.021986018866300583,
+            0.0029970414470881224,
+            0.011664161458611488,
+            0.025025488808751106,
+            -0.011371014639735222,
+            0.01578364707529545,
+            0.02713923342525959,
+            0.017002521082758904,
+            0.004566920455545187,
+            0.011756733991205692,
+            0.01023699901998043,
+            0.012597602792084217,
+            -0.03227701783180237,
+            -0.00698923971503973,
+            0.015027636662125587,
+            0.009558131918311119,
+            -0.008323829621076584,
+            -0.010861864313483238,
+            0.0030548993963748217,
+            -0.02798781543970108,
+            0.006476232316344976,
+            -0.022495169192552567,
+            -0.007444388698786497,
+            0.012659317813813686,
+            -0.009781849570572376,
+            0.0008239935268647969,
+            0.004875496029853821,
+            0.005415503401309252,
+            -0.011702733114361763,
+            -0.01229674182832241,
+            -0.007714392617344856,
+            0.00742510287091136,
+            0.01316846814006567,
+            -0.004875496029853821,
+            -0.0001758640428306535,
+            -0.014950492419302464,
+            0.003351903520524502,
+            0.011625589802861214,
+            0.015482786111533642,
+            0.015729647129774094,
+            -0.00287361117079854,
+            -0.0024705342948436737,
+            0.0181133933365345,
+            0.00044237219844944775,
+            -0.007201385218650103,
+            0.008532118052244186,
+            0.010846436023712158,
+            0.000777707202360034,
+            0.017449956387281418,
+            0.009581275284290314,
+            -0.01657051593065262,
+            0.004809923470020294,
+            0.004995069000869989,
+            0.03292502835392952,
+            0.010105853900313377,
+            0.00742895994335413,
+            -0.0031243290286511183,
+            -0.0016412369441241026,
+            -0.009596704505383968,
+            0.012659317813813686,
+            0.017959104850888252,
+            -0.0068388087674975395,
+            0.004686493426561356,
+            0.02556549571454525,
+            0.004177343565970659,
+            0.044866908341646194,
+            0.004852352663874626,
+            0.0035274059046059847,
+            -0.006048083771020174,
+            -0.009018125012516975,
+            -0.0015804861905053258,
+            -0.0018958119908347726,
+            -0.004077056422829628,
+            -0.006206228863447905,
+            -0.0016711303032934666,
+            0.019316839054226875,
+            -0.004088628105819225,
+            0.00595936831086874,
+            0.009897565469145775,
+            -0.00216967286542058,
+            -0.006931381765753031,
+            0.014503058046102524,
+            -0.01155615970492363,
+            0.010522431693971157,
+            -0.021569442003965378,
+            -0.002233316656202078,
+            0.00411562854424119,
+            0.014148196205496788,
+            -0.010977580212056637,
+            0.00021202526113484055,
+            -0.003330688923597336,
+            0.015112495049834251,
+            0.004532205406576395,
+            -0.0032940455712378025,
+            0.0011947664897888899,
+            -0.01903912052512169,
+            -0.008084683679044247,
+            0.03230787441134453,
+            0.004007626790553331,
+            0.015390212647616863,
+            -0.005508076399564743,
+            0.008200399577617645,
+            0.005866795312613249,
+            0.013824190944433212,
+            -0.02794152870774269,
+            0.011664161458611488,
+            0.020196279510855675,
+            0.0013345899060368538,
+            0.00780696514993906,
+            -0.001996099017560482,
+            -0.01566021703183651,
+            0.01638536900281906,
+            -0.026799799874424934,
+            0.0049024964682757854,
+            -0.014287054538726807,
+            0.012011309154331684,
+            0.004212058149278164,
+            -0.004146486055105925,
+            0.007992110215127468,
+            0.006966096349060535,
+            0.017743103206157684,
+            0.025750642642378807,
+            0.026568368077278137,
+            0.0002040697872871533,
+            0.0148656340315938,
+            -0.007031668908894062,
+            -3.0887702450854704e-05,
+            -0.026059217751026154,
+            -0.01715680956840515,
+            0.003988340962678194,
+            -0.0363810732960701,
+            -7.202108827186748e-05,
+            -0.008030682802200317,
+            -0.018576256930828094,
+            0.041688576340675354,
+            -0.006117513403296471,
+            -0.013569616712629795,
+            0.007197528146207333,
+            -0.005724078975617886,
+            0.004836923908442259,
+            -0.009550417773425579,
+            0.030194131657481194,
+            -0.002079028869047761,
+            -0.017604243010282516,
+            -0.03314103186130524,
+            -0.00574336526915431,
+            -0.025766070932149887,
+            -0.00450520496815443,
+            -0.01633908413350582,
+            0.014109623618423939,
+            -0.020828859880566597,
+            0.002302746055647731,
+            0.0037974095903337,
+            0.0020539569668471813,
+            0.01965627260506153,
+            0.0014638060238212347,
+            -0.007386530749499798,
+            0.006267943885177374,
+            -0.005106927827000618,
+            -0.012651603668928146,
+            -0.025087203830480576,
+            0.0011369085405021906,
+            0.00449363375082612,
+            0.00041971117025241256,
+            0.013808762654662132,
+            -0.0037279801908880472,
+            0.022495169192552567,
+            -0.016323653981089592,
+            0.012775033712387085,
+            -0.01517421007156372,
+            0.0007073133601807058,
+            0.004875496029853821,
+            0.0006644020322710276,
+            -7.636043301317841e-05,
+            -0.00536150299012661,
+            -0.02558092586696148,
+            0.007725963834673166,
+            0.021708300337195396,
+            -0.0025708212051540613,
+            0.0063450876623392105,
+            -0.006213943008333445,
+            0.000337986828526482,
+            0.0035061913076788187,
+            0.008138683624565601,
+            -0.015382498502731323,
+            0.0012535888236016035,
+            0.03369646519422531,
+            0.013855048455297947,
+            -0.007984396070241928,
+            -0.01230445597320795,
+            0.0021388153545558453,
+            0.013515615835785866,
+            -0.016076793894171715,
+            0.004586206283420324,
+            0.01279817707836628,
+            0.024315765127539635,
+            0.0020944576244801283,
+            -0.00553121929988265,
+            -0.028219247236847878,
+            0.021230008453130722,
+            -0.011810734868049622,
+            0.015205067582428455,
+            0.0018620614428073168,
+            0.028358107432723045,
+            0.002680751495063305,
+            0.0014271625550463796,
+            0.0010935150785371661,
+            0.03351132199168205,
+            0.00942698772996664,
+            -0.023590611293911934,
+            -0.0024281051009893417,
+            -0.0009517631842754781,
+            0.003047185018658638,
+            0.029268404468894005,
+            0.03153643757104874,
+            -0.004755923058837652,
+            -0.039281684905290604,
+            0.008802121505141258,
+            -0.024115189909934998,
+            0.002595893107354641,
+            0.011000723578035831,
+            0.00617922842502594,
+            -0.01360047422349453,
+            -0.0012748033041134477,
+            0.0012034452520310879,
+            0.0038514104671776295,
+            0.014950492419302464,
+            -0.011448157951235771,
+            -0.002935326425358653,
+            0.009735562838613987,
+            0.0032130444888025522,
+            -0.012543601915240288,
+            0.017557958140969276,
+            0.0020867432467639446,
+            -0.003022113349288702,
+            -0.004208201076835394,
+            0.03357303515076637,
+            0.008532118052244186,
+            -0.016308225691318512,
+            -0.0054039317183196545,
+            -0.026028359308838844,
+            -0.009234127588570118,
+            -0.013531044125556946,
+            -0.006904381327331066,
+            -0.022772885859012604,
+            -0.007679677568376064,
+            -0.019347695633769035,
+            -0.007513818331062794,
+            -0.02895982936024666,
+            0.006117513403296471,
+            -0.0047752088867127895,
+            0.03375818207859993,
+            -0.0035505490377545357,
+            -0.031582724303007126,
+            0.01741909794509411,
+            -0.0035717636346817017,
+            0.019594557583332062,
+            -0.010777005925774574,
+            0.009156984277069569,
+            0.01069214753806591,
+            0.01312989555299282,
+            0.02152315527200699,
+            0.00026566439191810787,
+            0.006503232754766941,
+            0.0030857569072395563,
+            0.025010060518980026,
+            -0.0033576893620193005,
+            -0.007818536832928658,
+            0.002827324904501438,
+            -0.011571588926017284,
+            0.019255124032497406,
+            0.0030317562632262707,
+            0.014503058046102524,
+            -0.022526025772094727,
+            0.022418024018406868,
+            -0.004570777527987957,
+            0.013801048509776592,
+            0.0012468387139961123,
+            0.005330645013600588,
+            -0.012582174502313137,
+            -0.007351816166192293,
+            -0.004269916098564863,
+            -0.0035949070006608963,
+            0.015220495872199535,
+            -0.01361590251326561,
+            0.005130071192979813,
+            -0.03523934632539749,
+            0.027015801519155502,
+            0.008601547218859196,
+            0.003278616815805435,
+            -0.0017087379237636924,
+            0.010121283121407032,
+            0.005924653261899948,
+            0.01235074270516634,
+            0.005727936513721943,
+            -0.013099038042128086,
+            0.017465384677052498,
+            0.001782024628482759,
+            0.008370115421712399,
+            0.0036874795332551003,
+            -0.016771089285612106,
+            -0.010645861737430096,
+            0.01324561145156622,
+            0.01819053664803505,
+            0.003342260606586933,
+            0.0013596616918221116,
+            -0.03145929425954819,
+            0.030086129903793335,
+            0.0017289882525801659,
+            0.008562975563108921,
+            -0.007891823537647724,
+            0.02894440107047558,
+            -0.007444388698786497,
+            0.010622718371450901,
+            0.022603169083595276,
+            0.01567564532160759,
+            0.007128098513931036,
+            -0.008416402153670788,
+            0.007668105885386467,
+            0.027632953599095345,
+            0.007976681925356388,
+            0.005573648493736982,
+            0.018298538401722908,
+            0.009156984277069569,
+            0.012049880810081959,
+            -0.01576821878552437,
+            -0.04017655551433563,
+            0.001536128344014287,
+            -0.015444213524460793,
+            -0.018668830394744873,
+            0.011949594132602215,
+            0.011949594132602215,
+            0.004917925223708153,
+            0.02477862872183323,
+            0.015652501955628395,
+            -0.006858095061033964,
+            0.0018466326873749495,
+            -0.013006465509533882,
+            -0.019903132691979408,
+            0.005770365707576275,
+            -0.003265116596594453,
+            -0.023220321163535118,
+            -0.001962348585948348,
+            -0.011031581088900566,
+            1.3439917893265374e-05,
+            -0.006024940405040979,
+            -0.0069699534215033054,
+            0.04431147128343582,
+            0.008200399577617645,
+            -0.010321857407689095,
+            -0.014387342147529125,
+            0.013577330857515335,
+            -0.011841592378914356,
+            0.029175832867622375,
+            -0.0034155473113059998,
+            -0.001836989657022059,
+            -0.025982074439525604,
+            -0.0010607289150357246,
+            -0.009465559385716915,
+            -0.005496504716575146,
+            0.005230357870459557,
+            -0.01198045164346695,
+            -0.009789563715457916,
+            0.0028813255485147238,
+            -0.008709548972547054,
+            0.013307327404618263,
+            -0.009087554179131985,
+            0.006599662825465202,
+            -0.01272874791175127,
+            -0.0007323851459659636,
+            0.018684258684515953,
+            -4.550286030280404e-05,
+            0.016308225691318512,
+            -0.015104779973626137,
+            -0.0005366324330680072,
+            -0.00022648973390460014,
+            -0.03323360159993172,
+            0.016693945974111557,
+            0.012273598462343216,
+            0.0001559753727633506,
+            -0.02312774769961834,
+            0.002248745411634445,
+            0.024207763373851776,
+            0.0014117337996140122,
+            0.004354774486273527,
+            0.025040917098522186,
+            -0.005091499071568251,
+            0.0074598174542188644,
+            0.013446185737848282,
+            0.007922681048512459,
+            -0.020026562735438347,
+            -0.0020925288554280996,
+            0.009226413443684578,
+            0.007668105885386467,
+            0.007710535079240799,
+            -0.0008996910182759166,
+            0.0165859442204237,
+            0.0067848083563148975,
+            -0.019285980612039566,
+            -0.017696816474199295,
+            0.015120209194719791,
+            0.010098139755427837,
+            -0.0009392272913828492,
+            0.004393346607685089,
+            0.005731793586164713,
+            0.011394158005714417,
+            -0.011695018969476223,
+            -0.0013037322787567973,
+            -0.02298888936638832,
+            0.021662013605237007,
+            0.016663087531924248,
+            -0.016045935451984406,
+            0.034961625933647156,
+            -0.004250630270689726,
+            -0.007363387383520603,
+            -0.019239693880081177,
+            -0.02541120909154415,
+            -0.0021966733038425446,
+            0.015359355136752129,
+            -0.004458918701857328,
+            0.002956541022285819,
+            0.007961252704262733,
+            -0.032523877918720245,
+            -0.020365996286273003,
+            -0.011671875603497028,
+            0.0005154178361408412,
+            0.0052612158469855785,
+            0.0027347521390765905,
+            0.009704705327749252,
+            0.005018212366849184,
+            0.01116272620856762,
+            0.005862938240170479,
+            0.007706678006798029,
+            -0.017959104850888252,
+            -0.02135343849658966,
+            0.008161826990544796,
+            0.00027048587799072266,
+            -0.016107652336359024,
+            -0.021631157025694847,
+            0.007023954298347235,
+            0.004077056422829628,
+            -0.001974884420633316,
+            0.013492472469806671,
+            -0.020458567887544632,
+            0.017665958032011986,
+            0.009596704505383968,
+            -0.003650836180895567,
+            0.022556884214282036,
+            0.013137610629200935,
+            0.01318389642983675,
+            0.00037053192500025034,
+            -0.016925377771258354,
+            -0.0017810603603720665,
+            0.02649122290313244,
+            0.0009932280518114567,
+            0.0014628416392952204,
+            0.012528173625469208,
+            0.004543777089565992,
+            0.020288852974772453,
+            0.0034849767107516527,
+            -0.018715115264058113,
+            0.029484407976269722,
+            0.00019563216483220458,
+            0.012119310908019543,
+            0.018730545416474342,
+            0.021862588822841644,
+            0.0062525151297450066,
+            0.004632492549717426,
+            0.007097241003066301,
+            0.01903912052512169,
+            0.006545661948621273,
+            -0.013075895607471466,
+            -0.004185057710856199,
+            0.0069699534215033054,
+            0.0020057419314980507,
+            -0.004755923058837652,
+            -0.012196454219520092,
+            0.018298538401722908,
+            -0.007043240126222372,
+            0.0018292752793058753,
+            -0.022664885967969894,
+            0.0069930967874825,
+            -0.008848408237099648,
+            -0.004038484301418066,
+            -0.00823897123336792,
+            0.004786780569702387,
+            0.015876220539212227,
+            -0.004678778816014528,
+            0.03400504216551781,
+            -0.010098139755427837,
+            0.014595630578696728,
+            -0.009342129342257977,
+            -0.02227916568517685,
+            0.018483683466911316,
+            -0.01116272620856762,
+            -0.0014011265011504292,
+            0.00012547941878437996,
+            0.0022391024976968765,
+            -0.009596704505383968,
+            -0.014842490665614605,
+            0.004667207598686218,
+            -0.010869579389691353,
+            0.0303792767226696,
+            -0.0004961318918503821,
+            0.02315860614180565,
+            0.003916982561349869,
+            0.0021735301706939936,
+            0.01731109619140625,
+            -0.005558219738304615,
+            -0.003257402218878269,
+            0.011664161458611488,
+            -0.0072129569016397,
+            0.020011134445667267,
+            -0.00020913235493935645,
+            -0.001489842077717185,
+            0.016940806061029434,
+            -0.011772163212299347,
+            -0.012481886893510818,
+            0.018545400351285934,
+            0.012659317813813686,
+            0.0014657345600426197,
+            0.006144513376057148,
+            0.002813824685290456,
+            -0.0002004536654567346,
+            -0.003560192184522748,
+            -0.020921431481838226,
+            -0.005704793147742748,
+            0.002511034719645977,
+            0.002727037761360407,
+            -0.00455920584499836,
+            0.004401060752570629,
+            0.0012593745486810803,
+            -0.0004539437941275537,
+            -0.002684608567506075,
+            0.0025418922305107117,
+            -0.0031841155141592026,
+            0.004690350499004126,
+            0.005878366995602846,
+            -0.014395056292414665,
+            -0.02391461655497551,
+            0.0065533765591681,
+            0.013631331734359264,
+            -0.010020995512604713,
+            0.0019806702621281147,
+            0.03150557726621628,
+            -0.03786223754286766,
+            0.01738824136555195,
+            0.005434789694845676,
+            -0.007336387410759926,
+            0.005704793147742748,
+            -0.014287054538726807
+          ],
+          "title": "4 Downstream Results"
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "9cc3633d-79f3-4045-b6a2-d3df9e569d10",
+        "properties": {
+          "page_content": "5 Designing Models with Data, Model, and Expert-Parallelism 18 5.1 Data Parallelism 20 5.2 Model Parallelism 20 5.3 Model and Data Parallelism 21 5.4 Expert and Data Parallelism 22 5.5 Expert, Model and Data Parallelism 22 5.6 Towards Trillion Parameter Models 22 ",
+          "keyphrases": [
+            "Designing Models",
+            "Data Parallelism",
+            "Model Parallelism",
+            "Expert Parallelism",
+            "Trillion Parameter Models"
+          ],
+          "embedding": [
+            -0.004642441403120756,
+            0.005233492702245712,
+            -0.022710707038640976,
+            0.004320049658417702,
+            0.027123892679810524,
+            -0.0023946543224155903,
+            -0.029717355966567993,
+            0.04969131574034691,
+            -0.019472461193799973,
+            0.01448613591492176,
+            0.021908311173319817,
+            -0.0279119610786438,
+            0.043014224618673325,
+            -0.02931615710258484,
+            -0.037196844816207886,
+            0.025519099086523056,
+            0.024114903062582016,
+            0.01901394873857498,
+            -0.010932662524282932,
+            -0.05880425497889519,
+            0.020575758069753647,
+            -0.04052106291055679,
+            -0.04877429082989693,
+            0.019085591658949852,
+            0.03307022899389267,
+            0.024630729109048843,
+            0.0260635819286108,
+            0.01029504369944334,
+            -0.05450569838285446,
+            0.01792498119175434,
+            0.012745220214128494,
+            -0.014686735346913338,
+            0.02420087344944477,
+            -0.021263526752591133,
+            -0.03960403427481651,
+            -0.017409155145287514,
+            0.05966396629810333,
+            -0.004216167610138655,
+            -0.01887066476047039,
+            0.022108910605311394,
+            0.006763062439858913,
+            0.0028657044749706984,
+            -0.030462438240647316,
+            0.002539730630815029,
+            0.006548134610056877,
+            -0.02255309373140335,
+            -0.047570694237947464,
+            0.003571384120732546,
+            0.01928619109094143,
+            -0.024874314665794373,
+            0.008267557248473167,
+            -0.046309784054756165,
+            0.025963282212615013,
+            0.00751530984416604,
+            -0.022108910605311394,
+            4.404901119414717e-05,
+            0.028943615034222603,
+            0.037196844816207886,
+            0.007013811729848385,
+            0.017695724964141846,
+            0.00950697436928749,
+            -0.033901285380125046,
+            -0.05513615533709526,
+            0.018369166180491447,
+            -0.012265214696526527,
+            -0.035477422177791595,
+            -0.017967967316508293,
+            0.017595425248146057,
+            -0.026450451463460922,
+            -0.011283711530268192,
+            -0.0028764507733285427,
+            0.014801363460719585,
+            0.003943925723433495,
+            -0.006902765482664108,
+            0.009463989175856113,
+            -0.014521957375109196,
+            0.029975268989801407,
+            0.040062546730041504,
+            0.027654048055410385,
+            -0.001376433647237718,
+            -0.02430117316544056,
+            0.004194675013422966,
+            -0.010445492342114449,
+            0.02546178363263607,
+            0.02044680155813694,
+            -0.00542692793533206,
+            -0.005899769254028797,
+            -0.06487955152988434,
+            -0.017265869304537773,
+            0.010130265727639198,
+            -0.04218316823244095,
+            0.005924844183027744,
+            0.025934625416994095,
+            -0.009299211204051971,
+            -0.0171512421220541,
+            0.027553748339414597,
+            0.028069574385881424,
+            0.007192918099462986,
+            -0.01659242808818817,
+            -0.017409155145287514,
+            0.0003219439822714776,
+            0.047398753464221954,
+            0.0451061874628067,
+            -0.01139833964407444,
+            0.04106554388999939,
+            -0.009227568283677101,
+            0.005846037063747644,
+            0.03155140578746796,
+            -0.013518961146473885,
+            0.03499025106430054,
+            -0.025791339576244354,
+            -0.009772052057087421,
+            -0.016950642690062523,
+            0.008518306538462639,
+            -0.038772981613874435,
+            -0.007185753900557756,
+            -0.017867667600512505,
+            -0.04335810989141464,
+            0.020719043910503387,
+            -0.04120882973074913,
+            0.013017462566494942,
+            0.017982294782996178,
+            -0.02114889957010746,
+            0.03243977576494217,
+            -0.04295691102743149,
+            0.029574070125818253,
+            -0.013239555060863495,
+            -0.021335169672966003,
+            -0.015431818552315235,
+            -0.02100561372935772,
+            0.02230951003730297,
+            0.018039610236883163,
+            -0.03513353690505028,
+            -0.027768677100539207,
+            0.008819205686450005,
+            -0.032353803515434265,
+            0.025691039860248566,
+            0.0003951537946704775,
+            -0.006573209539055824,
+            0.002344504464417696,
+            -0.0020274859853088856,
+            -0.018612749874591827,
+            0.0076227737590670586,
+            -0.053502701222896576,
+            -0.020246202126145363,
+            -0.026536423712968826,
+            0.01940082013607025,
+            0.042068541049957275,
+            -0.0010889676632359624,
+            0.026822993531823158,
+            -0.013318361714482307,
+            0.0033672028221189976,
+            0.010115936398506165,
+            0.018999621272087097,
+            -0.00021448006737045944,
+            -0.006286639254540205,
+            -0.007938001304864883,
+            0.022982949391007423,
+            -0.004549305886030197,
+            0.001577928545884788,
+            0.009292046539485455,
+            0.009901008568704128,
+            0.00019175592751707882,
+            0.05324478819966316,
+            0.0067773908376693726,
+            0.000919712008908391,
+            0.061039503663778305,
+            0.0037110873963683844,
+            -0.04668232426047325,
+            0.024043260142207146,
+            0.0024286846164613962,
+            -0.00724306795746088,
+            0.03946075215935707,
+            0.012451485730707645,
+            -0.0010235938243567944,
+            0.017208555713295937,
+            0.013339853845536709,
+            0.01571838930249214,
+            -0.004660351667553186,
+            0.03467502444982529,
+            -0.0015474804677069187,
+            -0.017796024680137634,
+            -0.0029695862904191017,
+            0.0010003099450841546,
+            0.04794323444366455,
+            0.0009313539485447109,
+            0.029545413330197334,
+            0.009693245403468609,
+            0.01960141770541668,
+            -0.017380498349666595,
+            0.02044680155813694,
+            0.02574835531413555,
+            0.015016291290521622,
+            0.012465814128518105,
+            -0.029330484569072723,
+            -0.03258306160569191,
+            0.01591898873448372,
+            -0.010946990922093391,
+            -0.01806826703250408,
+            0.010861019603908062,
+            0.006372610107064247,
+            -0.0444757342338562,
+            -0.0026239105500280857,
+            -0.01911424845457077,
+            -0.000753590720705688,
+            0.0021922639571130276,
+            -0.006401267368346453,
+            0.011090275831520557,
+            0.03828581050038338,
+            -0.0343024842441082,
+            -0.006039472296833992,
+            0.016004959121346474,
+            -0.024917300790548325,
+            -0.006129025481641293,
+            -0.003829297609627247,
+            0.0020471876487135887,
+            -0.03355740010738373,
+            0.0253901407122612,
+            0.013332690112292767,
+            -0.04043509066104889,
+            -0.008969655260443687,
+            -0.008145765401422977,
+            0.006798883900046349,
+            -0.02054710127413273,
+            0.003039437811821699,
+            -0.02093397080898285,
+            0.006709330715239048,
+            -0.015517789870500565,
+            -0.02230951003730297,
+            -0.02633582428097725,
+            0.017280198633670807,
+            0.003111080499365926,
+            -0.00244122208096087,
+            -0.010932662524282932,
+            0.030978266149759293,
+            -0.013848517090082169,
+            -0.0075368029065430164,
+            -0.020805014297366142,
+            -0.03857238218188286,
+            0.03261171653866768,
+            0.01520256232470274,
+            -0.022395480424165726,
+            -0.04120882973074913,
+            -0.01995963230729103,
+            0.008718905970454216,
+            0.009026968851685524,
+            -0.015804359689354897,
+            0.052557021379470825,
+            -0.014758378267288208,
+            -0.001351358718238771,
+            0.0011588193010538816,
+            -0.0005270209512673318,
+            0.015575103461742401,
+            0.024731028825044632,
+            -0.010846691206097603,
+            0.006917094346135855,
+            0.013934488408267498,
+            0.0006443357560783625,
+            -0.005061550531536341,
+            -0.0090484619140625,
+            0.05284358933568001,
+            -0.030491095036268234,
+            -0.047742635011672974,
+            0.04545007273554802,
+            -0.032840974628925323,
+            0.009772052057087421,
+            0.016778700053691864,
+            0.001771363546140492,
+            0.03524816408753395,
+            0.007343367673456669,
+            -0.00307705020532012,
+            -0.015317190438508987,
+            0.0018662899965420365,
+            -0.012673578225076199,
+            0.02412923239171505,
+            -0.03587862104177475,
+            0.0015277786878868937,
+            0.007644266821444035,
+            0.02437281608581543,
+            0.0067021665163338184,
+            -0.021822338923811913,
+            0.02655075117945671,
+            -0.008102779276669025,
+            0.002068680478259921,
+            -0.005036475602537394,
+            -0.0005041848635300994,
+            -0.01732318289577961,
+            -0.005151103716343641,
+            -0.015876002609729767,
+            -0.02357041835784912,
+            -0.024487445130944252,
+            -0.01950111798942089,
+            -0.01676437072455883,
+            0.01785333827137947,
+            0.00706754345446825,
+            0.005398270674049854,
+            0.007572623901069164,
+            0.012995969504117966,
+            0.007196500431746244,
+            0.005982158239930868,
+            -0.03272634372115135,
+            0.013712395913898945,
+            0.007823373191058636,
+            -0.004277063999325037,
+            -0.023341162130236626,
+            -0.0018430062336847186,
+            0.008704577572643757,
+            0.012687906622886658,
+            0.0016218096716329455,
+            -0.009549960494041443,
+            0.012129094451665878,
+            0.0023803256917744875,
+            -0.00688127288594842,
+            0.0008588157943449914,
+            -0.06619777530431747,
+            0.05814514309167862,
+            -0.0012635965831577778,
+            0.009528467431664467,
+            -0.057887230068445206,
+            -0.018913649022579193,
+            0.05909082666039467,
+            0.008109943941235542,
+            -0.048487719148397446,
+            -0.011899838224053383,
+            0.016563771292567253,
+            -0.02391430363059044,
+            -0.00041127335862256587,
+            0.029129885137081146,
+            -0.03143677860498428,
+            -0.04163868725299835,
+            -0.050837598741054535,
+            -0.040033891797065735,
+            -0.035477422177791595,
+            0.012172079645097256,
+            -0.011584610678255558,
+            -0.005043639801442623,
+            -0.026751350611448288,
+            0.006845451425760984,
+            -0.00787352304905653,
+            -0.03037646785378456,
+            0.0027152549009770155,
+            -0.02784031815826893,
+            0.0009698618669062853,
+            0.010158922523260117,
+            -0.005982158239930868,
+            -0.004366617184132338,
+            -0.03209589049220085,
+            -0.012917162850499153,
+            0.006064547225832939,
+            0.0251179002225399,
+            0.0007222470594570041,
+            0.02100561372935772,
+            0.021349499002099037,
+            0.010352357290685177,
+            -0.007099783048033714,
+            -0.0015698687639087439,
+            0.0029731683898717165,
+            0.004764233715832233,
+            0.003082423470914364,
+            -0.009951158426702023,
+            0.035162195563316345,
+            -0.05676960572600365,
+            -0.029860639944672585,
+            0.006770226638764143,
+            -0.01111893355846405,
+            -0.024071916937828064,
+            0.020174559205770493,
+            -0.0023713705595582724,
+            -0.04275631159543991,
+            0.01176371704787016,
+            -0.025590740144252777,
+            0.033299487084150314,
+            0.02742479182779789,
+            0.02524685673415661,
+            -0.010130265727639198,
+            -0.024745358154177666,
+            -0.004506320226937532,
+            -0.02216622419655323,
+            -0.015976302325725555,
+            0.020088588818907738,
+            -0.010173250921070576,
+            -0.005824544467031956,
+            0.011656252667307854,
+            -0.002566596493124962,
+            -0.0036824301350861788,
+            0.014385836198925972,
+            0.02441580221056938,
+            0.009986979886889458,
+            -0.04665366932749748,
+            0.00562036270275712,
+            -0.01613391563296318,
+            -0.029516756534576416,
+            -0.0067845555022358894,
+            0.011727895587682724,
+            0.0015833016950637102,
+            -0.010338028892874718,
+            -0.011835359036922455,
+            -0.025819996371865273,
+            0.014729720540344715,
+            -0.0009125477517955005,
+            -0.017265869304537773,
+            0.024143559858202934,
+            0.00978638045489788,
+            0.02381400391459465,
+            0.05378927290439606,
+            -0.004954086616635323,
+            0.009170254692435265,
+            0.021335169672966003,
+            0.021765025332570076,
+            0.0260635819286108,
+            -0.009893844835460186,
+            -0.04014851897954941,
+            -0.01577570289373398,
+            0.003718251595273614,
+            0.03711087256669998,
+            -0.018598422408103943,
+            -0.0006416491232812405,
+            0.027310162782669067,
+            -0.04023449122905731,
+            -0.001844797283411026,
+            0.007672923617064953,
+            0.011849688366055489,
+            0.05548003688454628,
+            -0.020790686830878258,
+            -0.013526124879717827,
+            0.016105258837342262,
+            -0.01596197299659252,
+            -0.018956635147333145,
+            -0.007844866253435612,
+            -0.020174559205770493,
+            0.012172079645097256,
+            -0.007350531872361898,
+            0.034245166927576065,
+            -0.00932786799967289,
+            0.0020794267766177654,
+            -0.04765666648745537,
+            0.022223537787795067,
+            0.05049371346831322,
+            -0.026149554178118706,
+            -0.020532771944999695,
+            -0.029330484569072723,
+            -0.013361346907913685,
+            0.011362518183887005,
+            -0.004449006170034409,
+            -0.058030515909194946,
+            -0.015245547518134117,
+            -0.02136382646858692,
+            -0.030032582581043243,
+            -0.04481961950659752,
+            -0.03413053974509239,
+            -0.0667136013507843,
+            0.021062927320599556,
+            -0.07714476436376572,
+            -0.032955601811409,
+            0.031236179172992706,
+            -0.016033615916967392,
+            -0.029574070125818253,
+            0.013318361714482307,
+            -0.008776219561696053,
+            -0.056339748203754425,
+            -0.03885895386338234,
+            0.05249970406293869,
+            0.016921984031796455,
+            -0.0033313813619315624,
+            -0.029803326353430748,
+            0.0007710536010563374,
+            -0.03166603296995163,
+            -0.028212860226631165,
+            0.004731994587928057,
+            0.012609099969267845,
+            0.023341162130236626,
+            -0.013261047191917896,
+            0.009972651489078999,
+            -0.002568387659266591,
+            0.013970308937132359,
+            0.02332683466374874,
+            0.02507491409778595,
+            -0.014357179403305054,
+            -0.004914683289825916,
+            -0.020260531455278397,
+            0.04398856312036514,
+            0.0203035157173872,
+            0.014149416238069534,
+            -0.006816794630140066,
+            -0.01901394873857498,
+            0.025275513529777527,
+            0.010775049217045307,
+            -0.005233492702245712,
+            1.3845774446963333e-05,
+            -0.006748734042048454,
+            0.03771267086267471,
+            -0.015603761188685894,
+            0.0014167326735332608,
+            0.002708090702071786,
+            -0.005896186921745539,
+            -0.02536148391664028,
+            -0.053502701222896576,
+            -0.016606757417321205,
+            0.0203035157173872,
+            -0.0028997347690165043,
+            -0.010108772665262222,
+            0.0023373402655124664,
+            -0.0171512421220541,
+            -0.02497461438179016,
+            0.025017600506544113,
+            -0.003610787680372596,
+            -0.01527420524507761,
+            -0.04633844271302223,
+            -0.0219226386398077,
+            -0.015446146950125694,
+            -0.015259875915944576,
+            0.025203870609402657,
+            0.028356146067380905,
+            0.009542795829474926,
+            0.01283835619688034,
+            0.017065269872546196,
+            -0.03272634372115135,
+            0.0004092584131285548,
+            -0.02563372626900673,
+            -0.0067845555022358894,
+            -0.005785140674561262,
+            -0.009822201915085316,
+            -0.01732318289577961,
+            0.02276802249252796,
+            -0.015302862040698528,
+            0.035964589565992355,
+            0.0021367408335208893,
+            -0.009263389743864536,
+            0.04851637780666351,
+            -0.016406157985329628,
+            0.002040023449808359,
+            0.008539799600839615,
+            -0.003204215783625841,
+            -2.553667400206905e-05,
+            -0.0020991284400224686,
+            -0.035018909722566605,
+            0.018383493646979332,
+            0.04843040555715561,
+            -0.0009501601452939212,
+            -0.030290495604276657,
+            -0.022524436935782433,
+            0.02563372626900673,
+            0.004309303127229214,
+            -0.006000068504363298,
+            0.01952977664768696,
+            -0.013963145203888416,
+            0.028069574385881424,
+            -0.048802945762872696,
+            0.0008507560123689473,
+            0.02461640164256096,
+            0.013347018510103226,
+            -0.04169600084424019,
+            0.048344433307647705,
+            -0.00925622507929802,
+            -0.038916267454624176,
+            -0.02240980975329876,
+            0.03762669861316681,
+            0.014371507801115513,
+            0.02100561372935772,
+            -0.018111253157258034,
+            0.012186408042907715,
+            0.009206075221300125,
+            -0.03708221763372421,
+            0.017523782327771187,
+            -0.029660040512681007,
+            0.004001239780336618,
+            0.004058553837239742,
+            0.02868570201098919,
+            -0.01623421534895897,
+            0.01816856674849987,
+            0.02556208334863186,
+            -0.006229325197637081,
+            -0.005118864588439465,
+            0.006662762723863125,
+            0.021091584116220474,
+            0.03742609918117523,
+            0.034846965223550797,
+            -0.02391430363059044,
+            0.012881341390311718,
+            0.013024627231061459,
+            0.03132215142250061,
+            -0.0009779216488823295,
+            0.0019397237338125706,
+            0.02020321600139141,
+            0.025590740144252777,
+            -0.003840043907985091,
+            -0.007307546213269234,
+            -0.008812041021883488,
+            0.0013379257870838046,
+            -0.00915592536330223,
+            0.0036860124673694372,
+            0.03289828822016716,
+            -0.010001308284699917,
+            -0.03928880766034126,
+            -0.0025271931663155556,
+            0.001104191760532558,
+            0.010273550637066364,
+            0.0023319669999182224,
+            0.05705617740750313,
+            -0.016606757417321205,
+            -0.006437088828533888,
+            0.013755381107330322,
+            -0.015732716768980026,
+            -0.024788344278931618,
+            -0.013769710436463356,
+            0.005043639801442623,
+            -0.023126235231757164,
+            -0.032525744289159775,
+            0.00379705848172307,
+            0.04387393593788147,
+            -0.016921984031796455,
+            0.013253883458673954,
+            0.03341411426663399,
+            -0.014385836198925972,
+            0.006494402885437012,
+            -0.0009349361062049866,
+            0.003689594566822052,
+            -0.00751530984416604,
+            -0.04871697723865509,
+            0.0206187441945076,
+            0.078864187002182,
+            0.005863947793841362,
+            0.02963138371706009,
+            -0.011964316479861736,
+            0.012702235020697117,
+            -0.013841352425515652,
+            0.024774014949798584,
+            9.951606625691056e-05,
+            0.014858677983283997,
+            -0.008045465685427189,
+            -0.019902316853404045,
+            -0.003714669495820999,
+            -0.004825130105018616,
+            -0.01394165214151144,
+            0.009356524795293808,
+            -0.014873006381094456,
+            -0.023226534947752953,
+            0.006652016658335924,
+            0.0009465779876336455,
+            -0.03441711142659187,
+            -0.0020203215535730124,
+            -0.02153576910495758,
+            0.023197878152132034,
+            0.026221195235848427,
+            -0.0019253952195867896,
+            0.0018125580390915275,
+            0.005254985764622688,
+            -0.010409671813249588,
+            0.01952977664768696,
+            -0.003539144992828369,
+            0.03759804368019104,
+            0.01328970491886139,
+            0.006046636495739222,
+            0.041466742753982544,
+            0.01816856674849987,
+            -0.03473233804106712,
+            -0.014930319972336292,
+            0.012422828935086727,
+            -0.005072297062724829,
+            0.0164778009057045,
+            -0.001104191760532558,
+            0.007866358384490013,
+            -0.0036519821733236313,
+            0.02384266071021557,
+            0.012931491248309612,
+            -0.028943615034222603,
+            -0.004463334567844868,
+            0.006261564325541258,
+            -0.018741706386208534,
+            0.01409926638007164,
+            -0.000238435561186634,
+            -0.04983460158109665,
+            0.0016110632568597794,
+            0.034560397267341614,
+            0.004470498766750097,
+            -0.014844349585473537,
+            0.006394103169441223,
+            0.04467633366584778,
+            0.012387007474899292,
+            0.010008472949266434,
+            0.0066126128658652306,
+            0.014686735346913338,
+            -0.010982812382280827,
+            -0.015317190438508987,
+            -0.028399132192134857,
+            -0.007422174792736769,
+            -0.003703922964632511,
+            -0.02430117316544056,
+            -0.010015636682510376,
+            -0.03292694315314293,
+            -0.0036681017372757196,
+            0.014973306097090244,
+            -0.006917094346135855,
+            -0.005183342844247818,
+            -0.015560775063931942,
+            -0.010230564512312412,
+            -0.004183928482234478,
+            0.039403434842824936,
+            0.025060584768652916,
+            0.01366224605590105,
+            -0.004932594019919634,
+            -0.00029597352840937674,
+            -0.035792648792266846,
+            -0.0056955874897539616,
+            0.021550098434090614,
+            0.026149554178118706,
+            0.006852615624666214,
+            -0.00939951092004776,
+            -0.05292956158518791,
+            -0.015618089586496353,
+            0.0007746357587166131,
+            -0.00939951092004776,
+            0.02296862192451954,
+            0.015861675143241882,
+            0.012730891816318035,
+            0.03384396806359291,
+            -0.0060323080979287624,
+            -0.01746646873652935,
+            -0.010818034410476685,
+            0.03665236011147499,
+            -0.004388109780848026,
+            0.004237660672515631,
+            0.016262872144579887,
+            -0.005849619396030903,
+            -0.035792648792266846,
+            0.023541761562228203,
+            0.0064478348940610886,
+            0.0024143559858202934,
+            0.016090931370854378,
+            -0.0026418212801218033,
+            -0.001410463941283524,
+            -0.019200220704078674,
+            0.012702235020697117,
+            0.023197878152132034,
+            0.035334136337041855,
+            -0.0018859917763620615,
+            0.013898666948080063,
+            0.043186165392398834,
+            0.004656769800931215,
+            -0.008790548890829086,
+            0.02676567994058132,
+            -0.019099920988082886,
+            -0.003883029567077756,
+            0.017480798065662384,
+            -0.01564674638211727,
+            0.04671098291873932,
+            0.011670581065118313,
+            0.005208417773246765,
+            -0.040492404252290726,
+            0.003263321006670594,
+            -0.003961836453527212,
+            -0.014142251573503017,
+            0.03160871937870979,
+            0.015689732506871223,
+            -0.04381662234663963,
+            0.014034787192940712,
+            0.016893327236175537,
+            0.006902765482664108,
+            0.012086108326911926,
+            0.022882649675011635,
+            -0.0038113868795335293,
+            0.03665236011147499,
+            0.017380498349666595,
+            -0.006086039822548628,
+            0.034560397267341614,
+            -0.01265924982726574,
+            0.003514070063829422,
+            0.01422105822712183,
+            -0.007758894935250282,
+            -0.001464195898734033,
+            -0.006175593007355928,
+            0.015517789870500565,
+            0.03619384765625,
+            -0.007680087815970182,
+            -0.03811386972665787,
+            0.03983329236507416,
+            -0.012193572707474232,
+            0.005885440390557051,
+            -0.04986325651407242,
+            0.010051458142697811,
+            -0.01858409307897091,
+            -0.007415010128170252,
+            9.391898493049666e-05,
+            0.008260393515229225,
+            -0.010359521955251694,
+            -0.0012224020902067423,
+            0.015661075711250305,
+            0.00903413351625204,
+            0.014958977699279785,
+            0.024286845698952675,
+            0.02994661219418049,
+            0.008704577572643757,
+            -0.015460475347936153,
+            -0.006064547225832939,
+            0.03903089463710785,
+            -0.01222222950309515,
+            -0.013225225731730461,
+            0.04530678689479828,
+            -0.025447456166148186,
+            -0.027697034180164337,
+            0.03665236011147499,
+            -0.03539144992828369,
+            0.021306512877345085,
+            0.04608052968978882,
+            -0.004029897041618824,
+            -0.02974601276218891,
+            0.01527420524507761,
+            -0.030920950695872307,
+            0.010911169461905956,
+            -0.0026633141096681356,
+            0.0012098646257072687,
+            0.005294389091432095,
+            0.016320187598466873,
+            0.005358867347240448,
+            0.007332621142268181,
+            0.011197740212082863,
+            -0.01792498119175434,
+            0.02107725664973259,
+            -0.03146543353796005,
+            -0.013769710436463356,
+            0.01057444978505373,
+            -0.027510762214660645,
+            -0.0005879171658307314,
+            0.013053284026682377,
+            -0.016563771292567253,
+            -0.030433781445026398,
+            0.018111253157258034,
+            0.009184583090245724,
+            0.06522343307733536,
+            0.01995963230729103,
+            0.016750043258070946,
+            0.009277718141674995,
+            -0.01577570289373398,
+            0.05049371346831322,
+            -0.000637171498965472,
+            0.03215320408344269,
+            0.020117245614528656,
+            -0.016248544678092003,
+            -0.016406157985329628,
+            0.014744049869477749,
+            0.015345847234129906,
+            0.008281885646283627,
+            -0.008016807958483696,
+            0.003514070063829422,
+            0.008955326862633228,
+            0.0017525573493912816,
+            0.02689463645219803,
+            0.006408431567251682,
+            0.02321220561861992,
+            0.03229648992419243,
+            0.0035570557229220867,
+            0.025232527405023575,
+            0.001274342997930944,
+            0.015790032222867012,
+            -0.01914290525019169,
+            0.014958977699279785,
+            0.01901394873857498,
+            0.00896249059587717,
+            0.0193578340113163,
+            -0.006103950552642345,
+            -0.013719560578465462,
+            0.008253228850662708,
+            0.005799469538033009,
+            0.01058877818286419,
+            0.010166086256504059,
+            -0.013726724311709404,
+            0.014887334778904915,
+            -0.032038576900959015,
+            0.014006130397319794,
+            -0.050522368401288986,
+            -0.005785140674561262,
+            0.030204525217413902,
+            0.017108255997300148,
+            -0.025304170325398445,
+            0.018053937703371048,
+            -0.015345847234129906,
+            -0.007852029986679554,
+            -0.038601040840148926,
+            -0.014421657659113407,
+            -0.003650191007182002,
+            0.022710707038640976,
+            0.011226397007703781,
+            0.015603761188685894,
+            0.017308855429291725,
+            -0.011247890070080757,
+            -0.007715909276157618,
+            -0.00841800682246685,
+            -0.03911686688661575,
+            -0.04120882973074913,
+            -0.012193572707474232,
+            0.0006353803910315037,
+            0.00048045325092971325,
+            -0.010875348001718521,
+            -0.02226652391254902,
+            0.009499810636043549,
+            0.022839665412902832,
+            0.0010853855637833476,
+            -0.020074259489774704,
+            0.007393517531454563,
+            -0.022710707038640976,
+            -0.006143353879451752,
+            -0.0424983985722065,
+            -0.020045602694153786,
+            -0.02819853276014328,
+            0.013504632748663425,
+            -0.0126950703561306,
+            0.026665380224585533,
+            -0.003232872812077403,
+            0.0014373299200087786,
+            -0.006444253027439117,
+            0.009707573801279068,
+            -0.008826369419693947,
+            0.006100368220359087,
+            -0.03656638786196709,
+            0.05568063631653786,
+            -0.006125443149358034,
+            0.029831983149051666,
+            -0.016004959121346474,
+            0.05158267915248871,
+            0.00843233522027731,
+            -0.008919505402445793,
+            0.03762669861316681,
+            -0.003850790439173579,
+            0.010882512666285038,
+            0.008797712624073029,
+            -0.03246843069791794,
+            -0.005441256333142519,
+            -0.01834050938487053,
+            0.0008409051806665957,
+            0.010345193557441235,
+            -0.017552439123392105,
+            0.031150206923484802,
+            0.030175868421792984,
+            -0.004663933999836445,
+            0.058976199477910995,
+            0.003345709992572665,
+            -0.018927978351712227,
+            0.008511141873896122,
+            -0.059606652706861496,
+            -0.009772052057087421,
+            -0.028571072965860367,
+            0.011391174979507923,
+            0.015546446666121483,
+            0.026364481076598167,
+            0.0051905070431530476,
+            -0.014185236766934395,
+            0.027611061930656433,
+            0.04281362518668175,
+            0.0002937347162514925,
+            0.020031273365020752,
+            0.012351186014711857,
+            0.00960727408528328,
+            0.009091447107493877,
+            0.0021080838050693274,
+            0.017122583463788033,
+            0.01869872212409973,
+            -0.004921847488731146,
+            -0.0012752384645864367,
+            -0.01581868901848793,
+            -0.006078875623643398,
+            0.0036663105711340904,
+            -0.01732318289577961,
+            -0.008704577572643757,
+            -0.003494368400424719,
+            0.014958977699279785,
+            -0.012723728083074093,
+            0.017738711088895798,
+            -0.02384266071021557,
+            -0.0539325587451458,
+            0.00913443323224783,
+            -0.010789377614855766,
+            -0.02490297146141529,
+            0.009005476720631123,
+            -0.014973306097090244,
+            0.01401329506188631,
+            0.004506320226937532,
+            -0.006172011140733957,
+            -0.014127923175692558,
+            0.010373850353062153,
+            0.003519443329423666,
+            0.040807630866765976,
+            0.006315296050161123,
+            0.006766644772142172,
+            -0.01124072540551424,
+            0.013332690112292767,
+            -0.025304170325398445,
+            0.02342713437974453,
+            -0.010732063092291355,
+            -0.06098219007253647,
+            0.03086363710463047,
+            -0.012623428367078304,
+            -0.010674749501049519,
+            0.03413053974509239,
+            -0.012100436724722385,
+            -0.0006783659919165075,
+            -0.002858540276065469,
+            0.006641270127147436,
+            -0.02286832220852375,
+            0.008396513760089874,
+            -0.04226914048194885,
+            0.010954155586659908,
+            -0.04522081837058067,
+            -0.019945302978157997,
+            0.045020218938589096,
+            0.02149278298020363,
+            0.00312182679772377,
+            0.05103819817304611,
+            0.017652738839387894,
+            0.013776874169707298,
+            0.0035212342627346516,
+            0.014285536482930183,
+            -0.016076602041721344,
+            -0.014242551289498806,
+            -0.006856197956949472,
+            0.019372161477804184,
+            -0.005785140674561262,
+            -0.020776357501745224,
+            0.015016291290521622,
+            0.0035176521632820368,
+            0.0029534667264670134,
+            0.010087279602885246,
+            0.0010773257818073034,
+            -0.013139255344867706,
+            -0.013017462566494942,
+            0.013798367232084274,
+            0.032067231833934784,
+            -0.03395859897136688,
+            -0.0020077840890735388,
+            0.010889677330851555,
+            0.00779471592977643,
+            -0.003854372538626194,
+            -0.03117886371910572,
+            0.01304611936211586,
+            -0.02209458127617836,
+            0.006884855218231678,
+            -0.009026968851685524,
+            -0.013698067516088486,
+            -0.056483034044504166,
+            0.001492852927185595,
+            0.019902316853404045,
+            0.0005453793564811349,
+            -0.009865188039839268,
+            0.0406930036842823,
+            0.016463471576571465,
+            0.021378155797719955,
+            0.0018483793828636408,
+            0.013805530965328217,
+            0.014887334778904915,
+            -0.0006734405760653317,
+            3.0084299851296237e-06,
+            0.005118864588439465,
+            -0.0004316467384342104,
+            -0.0038615367375314236,
+            -0.0015510625671595335,
+            -0.0039725829847157,
+            0.03054841049015522,
+            0.02739613503217697,
+            0.006859780289232731,
+            0.003499741666018963,
+            0.020260531455278397,
+            -0.008998312056064606,
+            -0.005029311403632164,
+            0.010037129744887352,
+            0.013719560578465462,
+            0.006931422743946314,
+            0.007823373191058636,
+            0.01627720147371292,
+            0.002285399241372943,
+            0.003814968978986144,
+            0.013490303419530392,
+            0.012630592100322247,
+            0.0073720249347388744,
+            0.011541624553501606,
+            -0.008797712624073029,
+            0.019759032875299454,
+            -0.022639065980911255,
+            -0.0016880790935829282,
+            0.0008315020822919905,
+            -0.018842006102204323,
+            -0.0004016463935840875,
+            0.00967891700565815,
+            0.012816863134503365,
+            0.011462817899882793,
+            -0.002675851574167609,
+            0.02497461438179016,
+            -0.03630847483873367,
+            -0.011899838224053383,
+            0.004452588502317667,
+            0.028628388419747353,
+            -0.01360493153333664,
+            -0.01957276090979576,
+            0.013368511572480202,
+            -0.008231735788285732,
+            0.02700926549732685,
+            -0.01050280686467886,
+            0.0017534529324620962,
+            -0.013640752993524075,
+            0.05722811818122864,
+            -0.006132607348263264,
+            0.01367657445371151,
+            -0.0012071779929101467,
+            -0.00041284054168500006,
+            0.011011469177901745,
+            0.0015295697376132011,
+            0.04493424668908119,
+            0.026522094383835793,
+            0.009091447107493877,
+            0.018670065328478813,
+            0.02276802249252796,
+            0.016921984031796455,
+            0.005520063452422619,
+            -0.005788723006844521,
+            0.006401267368346453,
+            0.0006600075867027044,
+            -0.011305204592645168,
+            0.02132084034383297,
+            -0.007296800147742033,
+            -0.016721386462450027,
+            0.004011986311525106,
+            -0.010524299927055836,
+            -0.006343953311443329,
+            -0.013647917658090591,
+            0.007758894935250282,
+            0.03679564595222473,
+            -0.001645989017561078,
+            -0.009284882806241512,
+            -0.03928880766034126,
+            0.008281885646283627,
+            0.01693631336092949,
+            0.013733888976275921,
+            -0.020647400990128517,
+            0.005036475602537394,
+            0.020217545330524445,
+            -0.035792648792266846,
+            0.005409017205238342,
+            -0.021578755229711533,
+            0.011169083416461945,
+            -0.010424000211060047,
+            0.027367478236556053,
+            0.0006586642703041434,
+            0.003519443329423666,
+            0.0067881373688578606,
+            -0.009772052057087421,
+            0.02381400391459465,
+            -0.02167905494570732,
+            -0.0003752281772904098,
+            0.005434092134237289,
+            0.008718905970454216,
+            -0.029373470693826675,
+            0.006354699842631817,
+            0.018354836851358414,
+            -0.01591898873448372,
+            0.024817001074552536,
+            0.0285854022949934,
+            -0.008009644225239754,
+            -0.009105775505304337,
+            -0.010911169461905956,
+            0.005828126333653927,
+            0.002023903885856271,
+            0.029187200590968132,
+            -0.003188096219673753,
+            -0.0006156786694191396,
+            -0.007895016111433506,
+            0.010667584836483002,
+            0.0025522680953145027,
+            0.015001962892711163,
+            -0.008561291731894016,
+            -0.03143677860498428,
+            0.039403434842824936,
+            -0.0009385182056576014,
+            -0.008561291731894016,
+            -0.01690765656530857,
+            -0.007415010128170252,
+            0.0015958391595631838,
+            -0.008059794083237648,
+            0.00271346396766603,
+            -0.03476099669933319,
+            0.02146412618458271,
+            0.0020937554072588682,
+            -0.0005014982889406383,
+            0.0013191195903345942,
+            -0.01437867246568203,
+            -0.006849033758044243,
+            -0.0024089827202260494,
+            -0.013225225731730461,
+            0.012480142526328564,
+            0.004574380815029144,
+            -0.018526779487729073,
+            0.013991801999509335,
+            0.009428167715668678,
+            0.016821684315800667,
+            -0.0032973513007164,
+            0.005570212844759226,
+            0.020948300138115883,
+            -0.011348189786076546,
+            -0.03590727597475052,
+            -0.0036197430454194546,
+            -0.00271346396766603,
+            0.030118554830551147,
+            0.006268728524446487,
+            -0.023484447970986366,
+            0.01981634646654129,
+            0.004388109780848026,
+            0.007128439843654633,
+            0.0009922501631081104,
+            -0.008661591447889805,
+            -0.004739158786833286,
+            0.002597044687718153,
+            -0.002835256280377507,
+            -0.00011233337863814086,
+            -0.01022340077906847,
+            0.009478317573666573,
+            -0.00980070885270834,
+            0.005871111992746592,
+            -0.00884069874882698,
+            0.013597767800092697,
+            -0.006791719701141119,
+            -0.026665380224585533,
+            0.004051389638334513,
+            -0.002688389038667083,
+            -0.05235642194747925,
+            -0.0036448179744184017,
+            0.02013157308101654,
+            -0.0074579957872629166,
+            0.005681259091943502,
+            -0.0013862844789400697,
+            0.0028889882378280163,
+            0.017409155145287514,
+            0.008804877288639545,
+            0.026178210973739624,
+            0.016033615916967392,
+            0.01921454817056656,
+            -0.01623421534895897,
+            0.011054455302655697,
+            -0.009564288891851902,
+            0.039260152727365494,
+            0.002247786847874522,
+            -0.0068669444881379604,
+            -0.01736616902053356,
+            0.04418916255235672,
+            -0.00967891700565815,
+            -0.0015304653206840158,
+            -0.01455777883529663,
+            -0.014142251573503017,
+            0.03269768878817558,
+            -0.002256742212921381,
+            0.011512967757880688,
+            -0.004760651383548975,
+            0.008031136356294155,
+            0.0206617284566164,
+            -0.02556208334863186,
+            -0.02766837738454342,
+            -0.017767367884516716,
+            0.007214411161839962,
+            -0.004112286027520895,
+            0.04106554388999939,
+            -0.017910653725266457,
+            -0.0016836014110594988,
+            0.02718120627105236,
+            -0.0269662793725729,
+            -0.03418785333633423,
+            -0.010158922523260117,
+            0.014615092426538467,
+            -0.0010486687533557415,
+            -0.005057968199253082,
+            -0.01942947693169117,
+            -0.020647400990128517,
+            -0.000571797601878643,
+            0.0021868906915187836,
+            0.015603761188685894,
+            0.008862190879881382,
+            0.015388833358883858,
+            0.0019827093929052353,
+            0.0031576480250805616,
+            -0.012236557900905609,
+            0.013683739118278027,
+            0.010810869745910168,
+            0.002262115478515625,
+            -0.014521957375109196,
+            -0.011011469177901745,
+            -0.022366823628544807,
+            -0.008847862482070923,
+            -0.00808128621429205,
+            -0.00023933110060170293,
+            -0.0034531739074736834,
+            -0.015317190438508987,
+            0.02805524691939354,
+            0.00480363704264164,
+            0.00878338422626257,
+            0.015804359689354897,
+            -0.023584747686982155,
+            0.019099920988082886,
+            0.013920159079134464,
+            -0.010739227756857872,
+            0.003189887385815382,
+            -0.006770226638764143,
+            -0.012229394167661667,
+            -0.014858677983283997,
+            0.0038078047800809145,
+            0.015087934210896492,
+            0.011921330355107784,
+            -0.0003962731861975044,
+            -0.015345847234129906,
+            -0.01268074195832014,
+            -0.011713567189872265,
+            0.02563372626900673,
+            -0.030634380877017975,
+            -0.002115248003974557,
+            0.023942960426211357,
+            -0.01995963230729103,
+            -0.011899838224053383,
+            -0.03143677860498428,
+            0.0027994350530207157,
+            -0.02960272692143917,
+            -0.029402127489447594,
+            0.01897096447646618,
+            0.049146831035614014,
+            -0.013261047191917896,
+            -0.010345193557441235,
+            0.01627720147371292,
+            -0.016463471576571465,
+            -0.0021671890281140804,
+            -0.033614713698625565,
+            0.02616388164460659,
+            -0.021664725616574287,
+            0.0038472083397209644,
+            -0.008740399032831192,
+            -0.03499025106430054,
+            0.012687906622886658,
+            0.023541761562228203,
+            0.03338545560836792,
+            -0.001543898251838982,
+            0.00379705848172307,
+            -0.007637102622538805,
+            0.017839010804891586,
+            0.018111253157258034,
+            0.002573760924860835,
+            0.03301291540265083,
+            0.0011032961774617434,
+            -0.04642441123723984,
+            -0.00022880859614815563,
+            0.005151103716343641,
+            -0.0003194812743458897,
+            0.002509282436221838,
+            -0.01366224605590105,
+            -0.026808666065335274,
+            0.015059277415275574,
+            -0.026536423712968826,
+            -0.00040813899249769747,
+            -0.016434814780950546,
+            -0.010818034410476685,
+            -0.007780387531965971,
+            0.005509316921234131,
+            0.004495573695749044,
+            0.024000274017453194,
+            0.009557124227285385,
+            -0.0006985154468566179,
+            0.015016291290521622,
+            0.014529122039675713,
+            0.03069169446825981,
+            -0.0006680673686787486,
+            0.02219488099217415,
+            0.02686597965657711,
+            -0.013110597617924213,
+            0.007343367673456669,
+            0.0028083904180675745,
+            0.026350151747465134,
+            -0.03163737803697586,
+            0.02517521381378174,
+            0.01250163558870554,
+            0.003983329050242901,
+            0.02657940983772278,
+            -0.011985808610916138,
+            -0.00912010483443737,
+            -0.00014709123934153467,
+            0.004001239780336618,
+            -0.003247201442718506,
+            -0.0046925912611186504,
+            0.040635690093040466,
+            0.011491474695503712,
+            -0.024186545982956886,
+            -0.012401335872709751,
+            -0.0158903319388628,
+            0.028040917590260506,
+            -0.019773360341787338,
+            0.00017183032468892634,
+            -0.016492128372192383,
+            0.004918265156447887,
+            0.014314194209873676,
+            0.0014597182162106037,
+            0.01879902184009552,
+            0.016019288450479507,
+            -0.0006586642703041434,
+            0.018354836851358414,
+            0.007751730736345053,
+            -0.026393137872219086,
+            0.004839458502829075,
+            0.007436503190547228,
+            -0.014020458795130253,
+            0.007529638707637787,
+            0.015503461472690105,
+            -0.00407646456733346,
+            -0.013683739118278027,
+            -0.01230820082128048,
+            -0.00862576998770237,
+            0.004595873411744833,
+            -0.010638928040862083,
+            0.019300520420074463,
+            0.0018806185107678175,
+            -0.0038113868795335293,
+            0.019859332591295242,
+            -0.006734405644237995,
+            -0.023025935515761375,
+            0.01897096447646618,
+            -0.01437867246568203,
+            0.02441580221056938,
+            0.012530292384326458,
+            -0.022123238071799278,
+            0.00579588720574975,
+            0.009929666295647621,
+            0.04226914048194885,
+            -0.005878276191651821,
+            0.00615410041064024,
+            -0.02703792229294777,
+            -0.005584541708230972,
+            -0.02609223872423172,
+            0.0067845555022358894,
+            0.003377949120476842,
+            -0.011276546865701675,
+            -0.00021190541156101972,
+            -0.0012447903864085674,
+            0.014142251573503017,
+            0.012809699401259422,
+            -0.014873006381094456,
+            -0.0011149381753057241,
+            0.0036860124673694372,
+            -0.01528853364288807,
+            -0.0024698791094124317,
+            0.0011955361114814878,
+            -0.008317707106471062,
+            0.02293996512889862,
+            0.0393461212515831,
+            -0.021693382412195206,
+            -0.01760975457727909,
+            0.016878999769687653,
+            -0.003571384120732546,
+            -0.022438466548919678,
+            -0.007508145645260811,
+            0.01113326195627451,
+            0.0054376740008592606,
+            -0.0025558501947671175,
+            -0.03372934088110924,
+            0.021507112309336662,
+            0.006472909823060036,
+            0.014958977699279785,
+            0.00788785144686699,
+            0.0026955532375723124,
+            0.030347811058163643,
+            -0.020002616569399834,
+            -0.018770365044474602,
+            -0.00569916982203722,
+            -0.020188888534903526,
+            0.019099920988082886,
+            0.008790548890829086,
+            -0.011770880781114101,
+            0.02829883247613907,
+            -0.026493437588214874,
+            0.015560775063931942,
+            0.009299211204051971,
+            -0.0051976717077195644,
+            0.010603106580674648,
+            0.0028227188158780336,
+            -0.004914683289825916,
+            -0.04854503273963928,
+            -0.005100953858345747,
+            0.008074122481048107,
+            0.02104859985411167,
+            -0.0060251434333622456,
+            -0.031408119946718216,
+            0.02974601276218891,
+            0.0008785175159573555,
+            -0.002616746351122856,
+            -0.010058622807264328,
+            0.00706754345446825,
+            -0.020561428740620613,
+            -0.006863362155854702,
+            0.007053215056657791,
+            0.020461129024624825,
+            0.0059427544474601746,
+            0.028485102578997612,
+            -0.006028725765645504,
+            0.05232776328921318,
+            -0.0006617986364290118,
+            0.009435332380235195,
+            0.0026418212801218033,
+            -0.002097337506711483,
+            -0.0017946474254131317,
+            -0.013504632748663425,
+            -0.026292838156223297,
+            -0.021120242774486542,
+            -0.006010815035551786,
+            -0.00208300887607038,
+            0.010316535830497742,
+            0.008704577572643757,
+            -0.02034650184214115,
+            0.0009967278456315398,
+            -0.039546720683574677,
+            0.009170254692435265,
+            -0.027023592963814735,
+            -0.009843694977462292,
+            -0.0035122791305184364,
+            -0.004481245297938585,
+            -0.022567423060536385,
+            0.022366823628544807,
+            0.019930975511670113,
+            -0.032181862741708755,
+            0.002450177213177085,
+            0.008847862482070923,
+            0.0079451659694314,
+            0.01339716836810112,
+            2.5690593247418292e-05,
+            -0.02412923239171505,
+            -0.014744049869477749,
+            -0.0006953810807317495,
+            0.01029504369944334,
+            -0.008740399032831192,
+            -0.026421794667840004,
+            0.0019253952195867896,
+            0.03788461163640022,
+            -0.007017393596470356,
+            0.013461646623909473,
+            -0.00047776667634025216,
+            -0.00869741290807724,
+            -0.0222378671169281,
+            0.004080046899616718,
+            0.019701717421412468,
+            0.025576412677764893,
+            -0.009378017857670784,
+            0.004954086616635323,
+            -0.0029928700532764196,
+            0.006343953311443329,
+            -0.00632246071472764,
+            0.0006474701222032309,
+            0.01826886646449566,
+            -0.029545413330197334,
+            0.004739158786833286,
+            -0.008031136356294155,
+            -0.011978644877672195,
+            0.010832362808287144,
+            -0.0004674680531024933,
+            0.014256879687309265,
+            -0.016506457701325417,
+            -0.0018573347479104996,
+            0.030003925785422325,
+            0.00407646456733346,
+            -0.025017600506544113,
+            0.012752384878695011,
+            -0.02556208334863186,
+            -0.007192918099462986,
+            -0.0103093720972538,
+            0.023685047402977943,
+            0.01331119704991579,
+            0.00925622507929802,
+            0.027152549475431442,
+            -0.0006125443032942712,
+            0.01792498119175434,
+            -0.004878861829638481,
+            -0.0033277992624789476,
+            0.0016853924607858062,
+            -0.03209589049220085,
+            0.0047785621136426926,
+            0.029889296740293503,
+            -0.03395859897136688,
+            -0.004402438644319773,
+            0.010524299927055836,
+            0.02359907701611519,
+            -0.016993626952171326,
+            0.008281885646283627,
+            0.014092101715505123,
+            -0.006326042581349611,
+            0.00897681899368763,
+            0.025089243426918983,
+            0.049462057650089264,
+            -0.00248599867336452,
+            0.006175593007355928,
+            0.009578617289662361,
+            0.013626424595713615,
+            -0.0007370233652181923,
+            0.008812041021883488,
+            -0.05089491233229637,
+            0.01258760690689087,
+            -0.006394103169441223,
+            -0.022682050243020058,
+            -0.0053624496795237064,
+            -0.016248544678092003,
+            0.017093926668167114,
+            0.026135224848985672,
+            0.02352743409574032,
+            0.01518823392689228,
+            0.013339853845536709,
+            0.02841345965862274,
+            0.012007301673293114,
+            0.0052728960290551186,
+            0.004689008928835392,
+            -0.003426307812333107,
+            0.0022065923549234867,
+            -0.004126614425331354,
+            0.009557124227285385,
+            -0.005287224892526865,
+            0.02819853276014328,
+            0.007723073475062847,
+            0.043472737073898315,
+            -0.03401591256260872,
+            -0.006397685036063194,
+            0.015790032222867012,
+            -0.016735713928937912,
+            -0.0017821099609136581,
+            0.007687252014875412,
+            0.02027485892176628,
+            -0.014543450437486172,
+            0.011985808610916138,
+            -0.0033994419500231743,
+            0.010323700495064259,
+            -0.0042698998004198074,
+            0.01339716836810112,
+            -0.002792270854115486,
+            0.016004959121346474,
+            0.01824020966887474,
+            0.006576791871339083,
+            0.005738573148846626,
+            0.006017979234457016,
+            0.00019522612274158746,
+            -0.022366823628544807,
+            -0.011892673559486866,
+            0.00208300887607038,
+            0.004230496007949114,
+            -0.00625081779435277,
+            0.004456170368939638,
+            0.014622257091104984,
+            -0.02034650184214115,
+            0.016578100621700287,
+            0.019515447318553925,
+            -0.004753487184643745,
+            -0.00034903385676443577,
+            0.0015304653206840158,
+            -0.011641924269497395,
+            -0.015245547518134117,
+            -0.022180551663041115,
+            -0.011161918751895428,
+            -0.012859849259257317,
+            -0.005681259091943502,
+            0.01654944382607937,
+            0.00994399469345808,
+            -0.014729720540344715,
+            0.019271861761808395,
+            -0.027768677100539207,
+            0.019071262329816818,
+            0.004742741119116545,
+            -0.0055308095179498196,
+            -0.014593600295484066,
+            -0.00994399469345808,
+            0.04416050761938095,
+            0.00787352304905653,
+            0.002009575255215168,
+            -0.010882512666285038,
+            -0.004201839212328196,
+            -0.018741706386208534,
+            -0.002480625407770276,
+            -0.0090484619140625,
+            0.024774014949798584,
+            0.01630585826933384,
+            0.008074122481048107,
+            -0.020160231739282608,
+            0.008281885646283627,
+            0.005462748929858208,
+            0.016176901757717133,
+            0.012394172139465809,
+            0.01819722354412079,
+            -0.010402507148683071,
+            -0.021765025332570076,
+            0.0010424000211060047,
+            -0.007880687713623047,
+            0.017767367884516716,
+            0.008983983658254147,
+            0.0038221334107220173,
+            0.006182757206261158,
+            -0.014299864880740643,
+            0.012773877941071987,
+            0.027969276532530785,
+            -0.0004370199458207935,
+            0.021750696003437042,
+            -0.00995832309126854,
+            -0.00834636390209198,
+            0.006852615624666214,
+            0.017409155145287514,
+            0.018354836851358414,
+            -0.03137946501374245,
+            -0.009471152909100056,
+            -0.008862190879881382,
+            -0.011075947433710098,
+            -0.011706402525305748,
+            0.0003340336843393743,
+            -0.019973959773778915,
+            -0.0126950703561306,
+            -0.0104884784668684,
+            -0.01749512553215027,
+            0.005401853006333113,
+            -0.009041297249495983,
+            0.004320049658417702,
+            -0.013089105486869812,
+            -0.034388452768325806,
+            -0.01504494808614254,
+            0.020074259489774704,
+            -0.002005993155762553,
+            -0.009793545119464397,
+            0.008145765401422977,
+            -0.0016997209750115871,
+            0.010373850353062153,
+            0.007629937957972288,
+            -0.013812695629894733,
+            -0.014350014738738537,
+            0.00013623290578834713,
+            -0.004474081099033356,
+            0.010538628324866295,
+            -0.010595941916108131,
+            -0.004581545013934374,
+            -0.012974477373063564,
+            -0.004277063999325037,
+            -0.017452139407396317,
+            -0.015790032222867012,
+            0.006730823311954737,
+            -0.018956635147333145,
+            0.01360493153333664,
+            -0.023728033527731895,
+            -0.012387007474899292,
+            0.020074259489774704,
+            0.002625701716169715,
+            -0.016850342974066734,
+            -0.01869872212409973,
+            -0.008045465685427189,
+            -0.019615747034549713,
+            -0.009456824511289597,
+            0.003207797883078456,
+            0.02686597965657711,
+            0.005838872864842415,
+            0.014314194209873676,
+            -0.0017068851739168167,
+            0.032038576900959015,
+            0.006286639254540205,
+            0.029889296740293503,
+            -0.005863947793841362,
+            0.009363689459860325,
+            -0.02458774484694004,
+            -0.003499741666018963,
+            -0.009112940169870853,
+            0.0036340714432299137,
+            0.00044642301509156823,
+            -0.006852615624666214,
+            -0.0012734474148601294,
+            -0.0023283849004656076,
+            -0.007400681730359793,
+            -0.009585781022906303,
+            0.015575103461742401,
+            0.006591120269149542,
+            0.005129611119627953,
+            0.004183928482234478,
+            0.03332814201712608,
+            -0.006684255786240101,
+            -0.013655081391334534,
+            0.01366224605590105,
+            0.023470120504498482,
+            -0.016893327236175537,
+            -0.000970757391769439,
+            0.009084283374249935,
+            0.011290875263512135,
+            -0.023871317505836487,
+            0.01940082013607025,
+            -0.004696173127740622,
+            -0.004871697630733252,
+            0.008074122481048107,
+            0.002426893450319767,
+            0.01705094240605831,
+            -0.0029212275985628366,
+            -0.0019988289568573236,
+            -0.009936830028891563,
+            0.003943925723433495,
+            -0.015102262608706951,
+            -0.0007929941639304161,
+            0.01255895011126995,
+            -0.011405503377318382,
+            0.008088450878858566,
+            -0.027825990691781044,
+            -0.029831983149051666,
+            -0.01705094240605831,
+            0.01050280686467886,
+            0.002369579393416643,
+            -0.022395480424165726,
+            0.018956635147333145,
+            -0.0038113868795335293,
+            -0.017280198633670807,
+            0.018598422408103943,
+            -0.006501567084342241,
+            -0.003922433126717806,
+            -0.0005767230177298188,
+            0.020633071660995483,
+            0.005842455197125673,
+            -0.005623945035040379,
+            -0.002880033105611801,
+            -0.0069529153406620026,
+            -0.00317376758903265,
+            -0.002838838379830122,
+            0.001708676340058446,
+            -0.0206187441945076,
+            0.00434154225513339,
+            0.01268074195832014,
+            0.01513091940432787,
+            -0.0007625460275448859,
+            0.02829883247613907,
+            -0.005086625460535288,
+            0.00397974718362093,
+            0.006852615624666214,
+            -0.0393461212515831,
+            5.3284191380953416e-05,
+            -0.018569765612483025,
+            -0.008582784794270992,
+            0.010044294409453869,
+            0.009478317573666573,
+            0.001631660503335297,
+            -0.008016807958483696,
+            0.0028065992519259453,
+            -0.009213239885866642,
+            -2.209446938650217e-05,
+            0.007286053616553545,
+            0.022080253809690475,
+            0.013196568936109543,
+            -0.004434677772223949,
+            0.02991795539855957,
+            -0.011434161104261875,
+            0.007443667389452457,
+            0.012322529219090939,
+            0.008898012340068817,
+            -0.0026382391806691885,
+            0.02076202817261219,
+            -0.009198911488056183,
+            -0.009772052057087421,
+            -0.01241566427052021,
+            0.014163744635879993,
+            0.01571838930249214,
+            -0.025261184200644493,
+            -0.0024770433083176613,
+            -0.003428098978474736,
+            0.005262149963527918,
+            -0.002656149910762906,
+            0.012974477373063564,
+            -0.01882767863571644,
+            0.025160884484648705,
+            0.031150206923484802,
+            -0.008704577572643757,
+            -0.011584610678255558,
+            0.0019110667053610086,
+            0.00267406040802598,
+            0.01268074195832014,
+            0.0033385457936674356,
+            0.006519477814435959,
+            -0.003358247457072139,
+            -0.02626418136060238,
+            -0.021836668252944946,
+            0.008403678424656391,
+            -0.01436434406787157,
+            -0.004072882700711489,
+            0.00931353960186243,
+            0.016076602041721344,
+            0.0010352357057854533,
+            0.0060358899645507336,
+            0.022452794015407562,
+            -0.02514655701816082,
+            0.0036824301350861788,
+            -0.01083952747285366,
+            -0.0023892810568213463,
+            -0.006017979234457016,
+            -0.034531738609075546,
+            0.011713567189872265,
+            0.006168428808450699,
+            -0.017667068168520927,
+            0.0023767435923218727,
+            -0.014063444919884205,
+            0.016463471576571465,
+            -0.0007370233652181923,
+            0.03160871937870979,
+            0.008195914328098297,
+            -0.006813212297856808,
+            0.00489319022744894,
+            0.02332683466374874,
+            0.04083628952503204,
+            0.010037129744887352,
+            0.004835876170545816,
+            -0.019300520420074463,
+            -0.00960727408528328,
+            -0.010746391490101814,
+            -0.007708745077252388,
+            -0.01942947693169117,
+            0.009528467431664467,
+            -0.012816863134503365,
+            -0.002960630925372243,
+            0.01213625818490982,
+            -0.0010952363954856992,
+            -0.020991284400224686,
+            0.014729720540344715,
+            0.014271208085119724,
+            -0.0316946916282177,
+            -0.013948816806077957,
+            -0.031236179172992706,
+            0.03054841049015522,
+            -0.020919643342494965,
+            -0.0257053691893816,
+            0.024501772597432137,
+            -0.007572623901069164,
+            0.018498122692108154,
+            0.010595941916108131,
+            0.005842455197125673,
+            -0.0002697792078834027,
+            0.004731994587928057,
+            0.007175007369369268,
+            -0.025576412677764893,
+            0.028786001726984978,
+            0.006759480573236942,
+            -0.016420487314462662,
+            0.008554127998650074,
+            -0.025347156450152397,
+            -0.024114903062582016,
+            0.02170771174132824,
+            0.019773360341787338,
+            0.00048358761705458164,
+            0.01078221295028925,
+            -0.0007495608297176659,
+            -0.0009940412128344178,
+            -0.013970308937132359,
+            -0.008940997533500195,
+            -0.000573140918277204,
+            0.02132084034383297,
+            -0.017036613076925278,
+            0.008446663618087769,
+            0.01255895011126995,
+            8.630695811007172e-05,
+            -0.02805524691939354,
+            -0.00924189668148756,
+            -0.004491991829127073,
+            -0.028800329193472862,
+            0.0076227737590670586,
+            -0.00469975546002388,
+            0.003743326524272561,
+            -0.02381400391459465,
+            0.0019468879327178001,
+            -0.04201122745871544,
+            -0.03089229390025139,
+            0.00140509067568928,
+            -0.029101228341460228,
+            -0.012250886298716068,
+            -0.02371370419859886,
+            0.01131236832588911,
+            -0.01503061968833208,
+            -0.0006752316257916391,
+            0.007665759418159723,
+            -0.01725153997540474,
+            0.006583956070244312,
+            -0.019859332591295242,
+            0.004363034851849079,
+            -0.011441324837505817,
+            0.012673578225076199,
+            0.017595425248146057,
+            0.020174559205770493,
+            0.005333792418241501,
+            -0.002863913308829069,
+            0.03103557974100113,
+            -0.008403678424656391,
+            -0.016363171860575676,
+            -0.006476492155343294,
+            0.017910653725266457,
+            -0.00014597181871067733,
+            0.03576399013400078,
+            0.0016558398492634296,
+            -0.0008068748866207898,
+            0.0036000411491841078,
+            0.007175007369369268,
+            0.0047785621136426926,
+            -0.006831123027950525,
+            0.0269662793725729,
+            0.010352357290685177,
+            0.012752384878695011,
+            -0.006150518078356981,
+            -0.009657423943281174,
+            -0.006759480573236942,
+            0.014077773317694664,
+            0.016463471576571465,
+            0.012107601389288902,
+            0.010409671813249588,
+            0.00489319022744894,
+            -0.017065269872546196,
+            0.01894230581820011,
+            -0.02422953210771084,
+            0.011369682848453522,
+            -0.020890984684228897,
+            0.007343367673456669,
+            -0.0007303068996407092,
+            0.00834636390209198,
+            -0.0020167394541203976,
+            -0.005745737347751856,
+            0.00823890045285225,
+            0.012236557900905609,
+            -0.01230820082128048,
+            -0.0014015085762366652,
+            -0.0253901407122612,
+            -0.01201446633785963,
+            0.0067057483829557896,
+            0.018297523260116577,
+            0.024243859574198723,
+            -0.03089229390025139,
+            0.0038078047800809145,
+            -0.000754934037104249,
+            -0.020719043910503387,
+            0.0035212342627346516,
+            0.013282540254294872,
+            -0.015546446666121483,
+            -0.0009528467198833823,
+            0.023054592311382294,
+            -0.002648985479027033,
+            0.009198911488056183,
+            0.013182240538299084,
+            -0.009299211204051971,
+            0.015317190438508987,
+            0.013741052709519863,
+            0.01729452610015869,
+            -0.009277718141674995,
+            0.016176901757717133,
+            -0.018541108816862106,
+            0.008647263050079346,
+            0.02059008739888668,
+            0.020461129024624825,
+            -0.02069038711488247,
+            0.026636723428964615,
+            0.018254537135362625,
+            -0.020174559205770493,
+            -0.0016925566596910357,
+            -0.007773223333060741,
+            0.011842523701488972,
+            0.014235386624932289,
+            -0.004312884993851185,
+            -0.025089243426918983,
+            -0.01346881128847599,
+            0.010280714370310307,
+            -0.010509971529245377,
+            -0.004739158786833286,
+            -0.015259875915944576,
+            0.010366685688495636,
+            0.0007634415524080396,
+            0.020676057785749435,
+            0.03258306160569191,
+            0.014335686340928078,
+            -0.006662762723863125,
+            0.02599193900823593,
+            0.03215320408344269,
+            0.00541976373642683,
+            -0.005946336779743433,
+            0.010022801347076893,
+            0.0010567285353317857,
+            -0.009213239885866642,
+            -0.003437054343521595,
+            -0.001859125797636807,
+            -0.0142927011474967,
+            -0.0064263422973454,
+            0.014930319972336292,
+            0.03100692294538021,
+            -0.005222746171057224,
+            -0.020575758069753647,
+            0.00858994945883751,
+            0.00912010483443737,
+            0.014564942568540573,
+            0.004997072275727987,
+            0.018999621272087097,
+            0.015489133074879646,
+            0.016219887882471085,
+            0.005835290532559156,
+            -0.023685047402977943,
+            -0.008031136356294155,
+            0.009750558994710445,
+            -0.02114889957010746,
+            -0.006451417226344347,
+            -0.004843040369451046,
+            -0.039747320115566254,
+            -0.021550098434090614,
+            0.014550614170730114,
+            0.026436123996973038,
+            0.01167774572968483,
+            -0.0009832947980612516,
+            -0.015073605813086033,
+            0.0029140631668269634,
+            0.0038113868795335293,
+            0.01346881128847599,
+            0.012601935304701328,
+            0.0013746425975114107,
+            0.00011899389937752858,
+            0.02301160618662834,
+            0.0028674956411123276,
+            -0.012580442242324352,
+            0.005785140674561262,
+            -0.008181585930287838,
+            -0.007923672907054424,
+            0.003062721574679017,
+            -0.005161850247532129,
+            -0.004452588502317667,
+            0.010775049217045307,
+            0.006096786353737116,
+            -0.003505114698782563,
+            -0.010194743983447552,
+            0.0011641924502328038,
+            0.010638928040862083,
+            0.002199428156018257,
+            0.0020453964825719595,
+            0.008847862482070923,
+            -0.011534460820257664,
+            -0.01043116394430399,
+            -0.015503461472690105,
+            0.007694416679441929,
+            -0.015016291290521622,
+            0.006902765482664108,
+            0.022108910605311394,
+            -0.004488409496843815,
+            -0.026708366349339485,
+            -0.008253228850662708,
+            0.008203078992664814,
+            -0.014507628977298737,
+            0.005745737347751856,
+            0.018096923828125,
+            -0.0009582199272699654,
+            0.009850858710706234,
+            -0.01346881128847599,
+            0.015489133074879646,
+            0.013253883458673954,
+            0.02139248326420784,
+            -0.0004813488048966974,
+            -0.007758894935250282,
+            0.011706402525305748,
+            0.0036681017372757196,
+            -0.017122583463788033,
+            0.009714738465845585,
+            0.03427382558584213,
+            -0.001524196588434279,
+            0.005964247509837151,
+            0.0016943477094173431,
+            0.0043558706529438496,
+            -0.001223297556862235,
+            -0.0010379223385825753,
+            0.004724830389022827,
+            0.012816863134503365,
+            0.005677676759660244,
+            -0.0038758653681725264,
+            0.009721902199089527,
+            0.008804877288639545,
+            -0.004735576454550028,
+            -0.02703792229294777,
+            -0.004499156028032303,
+            -0.0008422484388574958,
+            -0.0067773908376693726,
+            0.007386353332549334,
+            -0.016721386462450027,
+            0.007099783048033714,
+            -0.006043054163455963,
+            0.014350014738738537,
+            -0.0033134708646684885,
+            0.009736230596899986,
+            0.019128577783703804,
+            0.01422105822712183,
+            0.01237984374165535,
+            -0.007286053616553545,
+            0.010080115869641304,
+            0.007952329702675343,
+            0.009793545119464397,
+            0.0027403298299759626,
+            -0.006071711424738169,
+            0.0021421140991151333,
+            -0.004835876170545816,
+            0.0064263422973454,
+            0.0034191436134278774,
+            0.015603761188685894,
+            -0.01558943185955286,
+            -0.017308855429291725,
+            -0.000402318051783368,
+            0.032840974628925323,
+            0.001868081046268344,
+            0.0019182309042662382,
+            -0.009234732948243618,
+            0.010266385972499847,
+            0.000753590720705688,
+            -0.012859849259257317,
+            -0.007257396355271339,
+            0.026350151747465134,
+            -0.0021295766346156597,
+            0.0017749457620084286,
+            -0.0034728755708783865,
+            -0.0016692728968337178,
+            0.012745220214128494,
+            0.012365514412522316,
+            -0.011684910394251347,
+            0.018053937703371048,
+            -0.004585127346217632,
+            0.02293996512889862,
+            0.013225225731730461,
+            0.0018376329680904746,
+            -0.0005776185425929725,
+            -0.008038301020860672,
+            0.0074579957872629166,
+            -0.02339847758412361,
+            0.002009575255215168,
+            -0.0090484619140625,
+            -0.010445492342114449,
+            -0.0055308095179498196,
+            -0.025791339576244354,
+            0.00451706675812602,
+            0.002878241939470172,
+            -0.011684910394251347,
+            0.0016979299252852798,
+            0.005645437631756067,
+            0.007637102622538805,
+            0.013425825163722038,
+            -0.01420672982931137,
+            -0.005749319680035114,
+            0.010724899359047413,
+            0.0009832947980612516,
+            0.005520063452422619,
+            -0.020776357501745224,
+            0.01676437072455883,
+            -0.0065947026014328,
+            -0.02689463645219803,
+            0.007063961587846279,
+            -0.00406930036842823,
+            0.01662108674645424,
+            0.00806695781648159,
+            0.002824509982019663,
+            -0.020532771944999695,
+            0.018713049590587616,
+            -0.004284228198230267,
+            -0.012494471855461597,
+            -0.019343504682183266,
+            -0.0030179449822753668,
+            -0.007973822765052319,
+            0.018713049590587616,
+            -0.007651431020349264,
+            -0.0336720272898674,
+            0.004660351667553186,
+            -0.022223537787795067,
+            -0.017480798065662384,
+            -0.022223537787795067,
+            0.010323700495064259,
+            -0.005627527367323637,
+            0.011004305444657803,
+            -0.009714738465845585,
+            0.007551131304353476,
+            -0.029402127489447594,
+            0.019372161477804184,
+            0.007020975928753614,
+            0.017896324396133423,
+            0.019687389954924583,
+            0.0273244921118021,
+            -0.011649088934063911,
+            0.0048609510995447636,
+            -0.022710707038640976,
+            -1.0858333553187549e-05,
+            -0.019372161477804184,
+            -0.007472324185073376,
+            0.003976164851337671,
+            0.017738711088895798,
+            0.00680246576666832,
+            0.003290186868980527,
+            -0.016076602041721344,
+            -0.022037267684936523,
+            -0.03215320408344269,
+            -0.027123892679810524,
+            -0.004230496007949114,
+            0.006139772012829781,
+            0.013741052709519863,
+            -0.002645403379574418,
+            -0.015904659405350685,
+            3.201529034413397e-05,
+            0.012759549543261528,
+            -0.01960141770541668,
+            -0.002000619890168309,
+            -0.03278365731239319,
+            -0.010072951205074787,
+            -0.029545413330197334,
+            0.018211551010608673,
+            0.021936967968940735,
+            0.004760651383548975,
+            -0.011534460820257664,
+            -0.012967312708497047,
+            -0.018383493646979332,
+            0.0016504667000845075,
+            -0.006361864041537046,
+            -0.013003134168684483,
+            -0.03490427881479263,
+            -0.012143422849476337,
+            0.011935658752918243,
+            0.010087279602885246,
+            -0.006394103169441223,
+            -0.0018331552855670452,
+            0.011462817899882793,
+            -0.005681259091943502,
+            -0.005237075034528971,
+            -0.0036967587657272816,
+            -0.02146412618458271,
+            -0.015302862040698528,
+            0.006644852459430695,
+            -0.006390520837157965,
+            0.01257327850908041,
+            -0.013898666948080063,
+            0.0016003168420866132,
+            -0.008675919845700264,
+            0.006755898240953684,
+            0.00275824056006968,
+            -0.010072951205074787,
+            -0.010617434978485107,
+            0.016707057133316994,
+            0.010825199075043201,
+            -0.016878999769687653,
+            -0.003650191007182002,
+            0.01023772917687893,
+            0.00939951092004776,
+            -0.009700410068035126,
+            -0.028313159942626953,
+            -0.013432989828288555,
+            0.014242551289498806,
+            -0.01103296224027872,
+            0.01887066476047039,
+            -0.012093272991478443,
+            -0.00831054337322712,
+            0.020432472229003906,
+            0.008246064186096191,
+            -0.003408397315070033,
+            0.0007831432740204036,
+            -0.004123032093048096,
+            -0.003528398694470525,
+            -0.006956497672945261,
+            0.015016291290521622,
+            -0.015102262608706951,
+            0.008940997533500195,
+            -0.007096200715750456,
+            0.014314194209873676,
+            -0.00787352304905653,
+            0.0034782488364726305,
+            0.023341162130236626,
+            -0.006723659113049507,
+            -0.013411496765911579,
+            0.008389350026845932,
+            0.01448613591492176,
+            0.02122054249048233,
+            0.00913443323224783,
+            -0.003068094840273261,
+            -0.01296014804393053,
+            -0.014271208085119724,
+            -0.00896249059587717,
+            0.025203870609402657,
+            -0.01518823392689228,
+            -0.006652016658335924,
+            0.009456824511289597,
+            -0.0005064237047918141,
+            -0.011971480213105679,
+            -0.002588089322671294,
+            -0.025719698518514633,
+            -0.005351703148335218,
+            0.02286832220852375,
+            -0.03258306160569191,
+            -0.005910515319555998,
+            0.017065269872546196,
+            -0.01402762345969677,
+            -0.017423482611775398,
+            0.010982812382280827,
+            -0.015159576199948788,
+            0.009220404550433159,
+            -0.008038301020860672,
+            0.01430702954530716,
+            0.000650156696792692,
+            -0.02367071807384491,
+            0.005143939517438412,
+            0.021206213161349297,
+            -0.02437281608581543,
+            0.004577962681651115,
+            -0.0443897619843483,
+            0.002100919606164098,
+            0.003943925723433495,
+            0.014607928693294525,
+            -0.00925622507929802,
+            0.0002228756929980591,
+            -0.004044225439429283,
+            -0.012279544025659561,
+            0.017552439123392105,
+            -0.008948162198066711,
+            -0.02230951003730297,
+            0.002145696198567748,
+            0.005176178645342588,
+            0.007038886658847332,
+            -0.009743395261466503,
+            0.013239555060863495,
+            -0.0019146488048136234,
+            -0.010037129744887352,
+            0.016262872144579887,
+            -0.0007576206116937101,
+            -0.020604414865374565,
+            -0.02013157308101654,
+            -0.00244122208096087,
+            -0.024358488619327545,
+            0.012071779929101467,
+            -0.004760651383548975,
+            -0.0015743464464321733,
+            -0.005480659659951925,
+            0.016535114496946335,
+            0.00027179415337741375,
+            0.017867667600512505,
+            -0.0027295835316181183,
+            0.012200736440718174,
+            0.00022567423002328724,
+            0.00352481659501791,
+            -0.02325519174337387,
+            0.0005046326550655067,
+            -0.003368993755429983,
+            -0.012472978793084621,
+            -0.016334515064954758,
+            -0.01957276090979576,
+            -0.018010953441262245,
+            -0.010058622807264328,
+            0.016606757417321205,
+            0.025619398802518845,
+            0.006863362155854702,
+            -0.020432472229003906,
+            0.013755381107330322,
+            -0.004667516332119703,
+            0.00959294568747282,
+            0.0013459855690598488,
+            -0.009478317573666573,
+            0.029688699170947075,
+            0.013110597617924213,
+            0.018741706386208534,
+            -0.003703922964632511,
+            -0.028169874101877213,
+            -0.01819722354412079,
+            -0.0177816953510046,
+            0.010810869745910168,
+            -0.010825199075043201,
+            0.010338028892874718,
+            -0.011727895587682724,
+            -0.00787352304905653,
+            -0.001703303074464202,
+            -0.021234869956970215,
+            0.032181862741708755,
+            -0.024988943710923195,
+            0.0046102022752165794,
+            0.017409155145287514,
+            -0.0015886749606579542,
+            -0.010323700495064259,
+            0.023556090891361237,
+            0.002005993155762553,
+            -0.022925635799765587,
+            -0.00966458860784769,
+            0.010037129744887352,
+            -0.012988805770874023,
+            0.005176178645342588,
+            -1.1285110304015689e-05,
+            0.007171425502747297,
+            -0.028327489271759987,
+            0.012387007474899292,
+            0.018053937703371048,
+            0.013490303419530392,
+            0.02276802249252796,
+            0.0023122653365135193,
+            0.013003134168684483,
+            -0.012286707758903503,
+            0.006386938970535994,
+            0.01988798938691616,
+            0.0079451659694314,
+            -0.028040917590260506,
+            0.020676057785749435,
+            -0.021693382412195206,
+            0.018598422408103943,
+            -0.0042698998004198074,
+            -0.0031003339681774378,
+            0.03387262672185898,
+            -0.010667584836483002,
+            -0.004456170368939638,
+            -0.01069624163210392,
+            -0.004989908076822758,
+            -0.003675265936180949,
+            -0.011634760536253452,
+            0.02633582428097725,
+            0.024716701358556747,
+            -0.021019943058490753,
+            0.01211476605385542,
+            0.002511073602363467,
+            0.012243722565472126,
+            -0.00697082607075572,
+            0.023441461846232414,
+            0.014507628977298737,
+            -0.007314710877835751,
+            0.009019805118441582,
+            0.004051389638334513,
+            -0.003911686595529318,
+            0.0006891123484820127,
+            -0.013447318226099014,
+            -0.007386353332549334,
+            -0.006053800694644451,
+            0.011555952951312065,
+            0.008554127998650074,
+            -0.008016807958483696,
+            0.003718251595273614,
+            0.0003978403692599386,
+            0.0012931491946801543,
+            -0.01002996601164341,
+            0.0074579957872629166,
+            0.0035122791305184364,
+            0.00869741290807724,
+            0.011212068609893322,
+            0.0033672028221189976,
+            -0.009213239885866642,
+            -0.00451706675812602,
+            0.01824020966887474,
+            0.016191231086850166,
+            0.00976488832384348,
+            0.014958977699279785,
+            -0.01652078703045845,
+            0.0253901407122612,
+            0.024530429393053055,
+            0.0007558295619674027,
+            -0.01176371704787016,
+            0.0053552850149571896,
+            0.0028137636836618185,
+            -0.008632934652268887,
+            0.011054455302655697,
+            -0.006017979234457016,
+            0.015216890722513199,
+            0.003354665357619524,
+            -0.00806695781648159,
+            -0.006161264609545469,
+            -0.027095235884189606,
+            -0.037540730088949203,
+            0.002767195925116539,
+            -0.004599455744028091,
+            -0.006723659113049507,
+            -0.0026579408440738916,
+            -0.009986979886889458,
+            0.0014865841949358582,
+            0.006261564325541258,
+            -0.006010815035551786,
+            0.01202879473567009,
+            -0.023226534947752953,
+            -0.005423345603048801,
+            0.000444184202933684,
+            -0.010459821671247482,
+            -2.4809052774799056e-05,
+            -0.007046050857752562,
+            -0.010051458142697811,
+            0.005906933452934027,
+            -0.022782349959015846,
+            -0.000383287959266454,
+            -0.01763841137290001,
+            0.01889932155609131,
+            -0.008632934652268887,
+            0.0045206486247479916,
+            0.0016504667000845075,
+            -0.013074777089059353,
+            0.01504494808614254,
+            0.011491474695503712,
+            -0.010509971529245377,
+            0.002151069464161992,
+            0.008876519277691841,
+            0.0025558501947671175,
+            0.0011767299147322774,
+            -0.021793682128190994,
+            -0.01139833964407444,
+            0.02289697900414467,
+            0.002111665904521942,
+            -0.017423482611775398,
+            -9.363912977278233e-05,
+            0.009686080738902092,
+            0.003426307812333107,
+            0.016019288450479507,
+            0.00607529329136014,
+            0.013741052709519863,
+            0.014256879687309265,
+            -0.0008301587658934295,
+            0.020160231739282608,
+            -0.0009206075337715447,
+            -0.002340922364965081,
+            0.0024251025170087814,
+            0.006770226638764143,
+            0.0007200082764029503,
+            0.004807219374924898,
+            -0.010015636682510376,
+            -0.010739227756857872,
+            -0.0019021113403141499,
+            0.0014695690479129553,
+            0.00425198907032609,
+            0.016076602041721344,
+            0.0008444872801192105,
+            -0.01383418869227171,
+            0.014077773317694664,
+            0.04267033934593201,
+            0.0034191436134278774,
+            0.021062927320599556,
+            0.0010970274452120066,
+            -0.0005614989786408842,
+            0.009077118709683418,
+            0.00858994945883751,
+            -0.026636723428964615,
+            0.021621739491820335,
+            -0.006809630431234837,
+            0.00551648112013936,
+            -0.030175868421792984,
+            0.006286639254540205,
+            -0.017208555713295937,
+            0.002500327071174979,
+            -0.01166341733187437,
+            0.006394103169441223,
+            0.012315364554524422,
+            -0.006616195198148489,
+            0.017523782327771187,
+            -0.0008525470620952547,
+            0.016047945246100426,
+            0.0012421037536114454,
+            -0.0021188303362578154,
+            -0.00931353960186243,
+            0.012480142526328564,
+            0.018913649022579193,
+            0.018440809100866318,
+            0.0005261254264041781,
+            0.01222222950309515,
+            -0.005262149963527918,
+            -0.010015636682510376,
+            0.009284882806241512,
+            -0.004047807771712542,
+            -0.0005579168209806085,
+            0.0037934763822704554,
+            -0.011197740212082863,
+            -0.0033564562909305096,
+            0.01676437072455883,
+            0.019515447318553925,
+            0.0028657044749706984,
+            -0.011577446013689041,
+            -0.02037515863776207,
+            0.0006282161339186132,
+            0.014350014738738537,
+            -0.003111080499365926,
+            0.013683739118278027,
+            -0.0036716838367283344,
+            -0.0029695862904191017,
+            -0.008647263050079346,
+            0.0004466469108592719,
+            0.020074259489774704,
+            0.0051546860486269,
+            -0.014034787192940712,
+            -0.004534977488219738,
+            0.021936967968940735,
+            0.010610271245241165,
+            0.008145765401422977,
+            0.006576791871339083,
+            -0.007529638707637787,
+            0.01659242808818817,
+            -0.011169083416461945,
+            -0.00037948196404613554,
+            -0.012666413560509682,
+            -0.003236454911530018,
+            -0.005856783594936132,
+            -0.008496813476085663,
+            -0.007350531872361898,
+            -0.008568456396460533,
+            0.005487823858857155,
+            0.011161918751895428,
+            -0.01596197299659252,
+            -0.012236557900905609,
+            0.012250886298716068,
+            -0.002503909170627594,
+            -0.0004728412313852459,
+            -0.01620555855333805,
+            0.014156579971313477,
+            0.008059794083237648,
+            0.0033851133193820715,
+            -0.009779216721653938,
+            0.01558943185955286,
+            -0.005817380268126726,
+            0.0002970929490402341,
+            -0.0060323080979287624,
+            -0.0004099300713278353,
+            0.0014982260763645172,
+            -0.004090792965143919,
+            -0.005548720248043537,
+            -0.0018967381911352277,
+            0.007701580878347158,
+            -0.01952977664768696,
+            0.009686080738902092,
+            -0.010832362808287144,
+            -0.026565080508589745,
+            -0.013827024027705193,
+            0.022223537787795067,
+            0.025805668905377388,
+            0.027410462498664856,
+            0.007701580878347158,
+            0.002815554616972804,
+            0.00562036270275712,
+            0.0034155615139752626,
+            -0.0025845072232186794,
+            0.005376778077334166,
+            0.009026968851685524,
+            -0.0038078047800809145,
+            0.03057706728577614,
+            0.005641855765134096,
+            -0.004545723553746939,
+            0.014328522607684135,
+            0.018369166180491447,
+            0.01831185072660446,
+            -0.007687252014875412,
+            0.020174559205770493,
+            -0.016148244962096214,
+            -0.0008395618642680347,
+            -0.002708090702071786,
+            0.03507622331380844,
+            -0.029373470693826675,
+            -0.0012644920498132706,
+            -0.0027510763611644506,
+            0.02041814476251602,
+            0.006028725765645504,
+            -0.02304026484489441,
+            0.010531463660299778,
+            0.028112560510635376,
+            0.0025791339576244354,
+            0.012895669788122177,
+            -0.02051844447851181,
+            -0.009700410068035126,
+            0.00017686770297586918,
+            0.005308717489242554,
+            0.0029964521527290344,
+            0.005505734588950872,
+            -0.022323837503790855,
+            0.01623421534895897,
+            -0.01457210723310709,
+            0.019873660057783127,
+            0.0037218336947262287,
+            0.026292838156223297,
+            0.005366031546145678,
+            0.006064547225832939,
+            0.006960079539567232,
+            -0.013877173885703087,
+            0.014987634494900703,
+            0.0007853821152821183,
+            -0.013862845487892628,
+            0.002903316868469119,
+            -0.016721386462450027,
+            0.01167774572968483,
+            -0.0168360136449337,
+            0.015947645530104637,
+            -0.004359452985227108,
+            0.007565459702163935,
+            -0.0036089965142309666,
+            -0.0009138910681940615,
+            -0.013755381107330322,
+            0.012666413560509682,
+            -0.030032582581043243,
+            0.010287879034876823,
+            -0.010954155586659908,
+            -0.031092893332242966,
+            0.0037504907231777906,
+            0.002140322932973504,
+            0.01131236832588911,
+            -0.013633589260280132,
+            -0.017982294782996178,
+            0.01630585826933384,
+            -0.0019880824256688356,
+            0.004481245297938585,
+            -0.020819343626499176,
+            -0.007085454184561968,
+            0.003800640581175685,
+            0.024945957586169243,
+            -0.0017928563756868243,
+            -0.014536285772919655,
+            0.007325456943362951,
+            0.014371507801115513,
+            -0.009614438749849796,
+            0.020461129024624825,
+            0.005634691566228867,
+            -0.007701580878347158,
+            0.007253814488649368,
+            -0.0025468948297202587,
+            0.022954292595386505,
+            -0.00470691965892911,
+            -0.008496813476085663,
+            0.013905830681324005,
+            0.008375021629035473,
+            -0.02328384853899479,
+            0.005186925176531076,
+            -0.008726069703698158,
+            -0.0013871800620108843,
+            -0.0007809044909663498,
+            -0.001941514783538878,
+            0.010037129744887352,
+            0.028485102578997612,
+            -0.00834636390209198,
+            0.008948162198066711,
+            0.003689594566822052,
+            -0.00317376758903265,
+            5.6950279940792825e-06,
+            0.0006322460249066353,
+            -0.019830675795674324,
+            -0.00903413351625204,
+            -0.004749905318021774,
+            -0.010395343415439129,
+            0.0007531429291702807,
+            0.016463471576571465,
+            0.027582405135035515,
+            -0.0073720249347388744,
+            0.007167843170464039,
+            -0.0007070230203680694,
+            0.020461129024624825,
+            0.031752005219459534,
+            -0.014328522607684135,
+            0.004588709212839603,
+            -0.009370853193104267,
+            -0.012867012992501259,
+            -0.020561428740620613,
+            0.004259153269231319,
+            -0.0014668825315311551,
+            -0.02304026484489441,
+            -0.0009725484414957464,
+            0.00046657249913550913,
+            -0.005176178645342588,
+            -0.009549960494041443,
+            0.00488244416192174,
+            -0.011885508894920349,
+            0.0045278132893145084,
+            -0.002625701716169715,
+            0.004889608360826969,
+            0.010911169461905956,
+            0.034875623881816864,
+            -0.005946336779743433,
+            -0.002511073602363467,
+            -0.011498639360070229,
+            0.0033224262297153473,
+            0.020160231739282608,
+            0.000652843271382153,
+            0.008195914328098297,
+            -0.0066197775304317474,
+            0.00834636390209198,
+            -0.006841869559139013,
+            -0.012401335872709751,
+            0.01520256232470274,
+            0.008948162198066711,
+            0.003125408897176385,
+            -0.01076788455247879,
+            -0.0501498281955719,
+            -0.0015143457567319274,
+            0.032525744289159775,
+            0.010531463660299778,
+            -0.005720662418752909,
+            -0.012652085162699223,
+            -0.018541108816862106,
+            -0.008754727430641651,
+            -0.0020704714115709066,
+            0.0013638961827382445,
+            -0.023455791175365448,
+            0.0024645058438181877,
+            -0.005305135622620583,
+            -0.003789894049987197,
+            -0.0037289978936314583,
+            0.011441324837505817,
+            0.04390259087085724,
+            -0.015431818552315235,
+            0.00813860073685646,
+            -0.009657423943281174,
+            -0.01527420524507761,
+            -0.0006761271506547928,
+            0.018598422408103943,
+            -0.010216236114501953,
+            -0.012494471855461597,
+            -0.016964970156550407,
+            -0.010674749501049519,
+            -0.006472909823060036,
+            0.009979816153645515,
+            0.009628767147660255,
+            -0.011649088934063911,
+            -0.007751730736345053,
+            -0.004491991829127073,
+            -0.007053215056657791,
+            0.013511796481907368,
+            0.006193503737449646,
+            0.021091584116220474,
+            -0.0161625724285841,
+            -0.01185685209929943,
+            0.012021630071103573,
+            0.008754727430641651,
+            0.0030698860064148903,
+            -0.015360175631940365,
+            -0.005541556049138308,
+            -0.007465159986168146,
+            -0.0023122653365135193,
+            0.025662383064627647,
+            -0.004864533431828022,
+            0.016463471576571465,
+            -0.0003806013846769929,
+            -0.005584541708230972,
+            0.004574380815029144,
+            0.005545137915760279,
+            0.025232527405023575,
+            0.011806702241301537,
+            -0.0026042088866233826,
+            -0.01338283997029066,
+            0.0023283849004656076,
+            -0.010624599643051624,
+            -0.0022191298194229603,
+            -0.009356524795293808,
+            0.006086039822548628,
+            0.01574704609811306,
+            0.0034334722440689802,
+            0.021979954093694687,
+            -0.00030358557705767453,
+            -0.020848000422120094,
+            -0.004330795723944902,
+            0.00831054337322712,
+            0.0021779353264719248,
+            0.017437811940908432,
+            0.009707573801279068,
+            0.013432989828288555,
+            0.0058675301261246204,
+            0.012472978793084621,
+            -0.011298039928078651,
+            -0.004316467326134443,
+            -0.006086039822548628,
+            -0.002656149910762906,
+            0.014163744635879993,
+            6.481417221948504e-05,
+            0.03819984197616577,
+            -0.015073605813086033,
+            -0.019687389954924583,
+            0.002557641128078103,
+            -0.011520132422447205,
+            -0.012422828935086727,
+            0.00462811253964901,
+            -0.012809699401259422,
+            -0.0017874831100925803,
+            0.013167912140488625,
+            -0.020676057785749435,
+            -0.022452794015407562,
+            -0.013175075873732567,
+            0.011262218467891216,
+            -0.005183342844247818,
+            -0.005577377509325743,
+            -0.00020630833751056343,
+            0.010946990922093391,
+            0.0024698791094124317,
+            -0.007303964346647263,
+            0.018397822976112366,
+            -0.007680087815970182,
+            -0.007809044793248177,
+            0.013705231249332428,
+            -0.003943925723433495,
+            0.007114111445844173,
+            0.002369579393416643,
+            0.015933316200971603,
+            0.022925635799765587,
+            -0.004782144445925951,
+            0.0034746667370200157,
+            0.007880687713623047,
+            0.014701063744723797,
+            0.0020597251132130623,
+            0.008453828282654285,
+            -0.023785347118973732,
+            0.01360493153333664,
+            0.007164261303842068,
+            -0.007930837571620941,
+            0.013755381107330322,
+            0.01518823392689228,
+            0.0019182309042662382,
+            0.005459167063236237,
+            -0.005050804000347853,
+            0.01964440383017063,
+            -0.0009895635303109884,
+            0.002396445255726576,
+            -0.01967306062579155,
+            0.012738056480884552,
+            -0.004391692113131285,
+            0.011190575547516346,
+            0.009886680170893669,
+            0.01627720147371292,
+            0.029086900874972343,
+            -0.0010692659998312593,
+            0.00294630229473114,
+            -0.008955326862633228,
+            0.011498639360070229,
+            0.0038615367375314236,
+            -0.009901008568704128,
+            -0.0024286846164613962,
+            0.00641917809844017,
+            0.01792498119175434,
+            0.022753693163394928,
+            0.019314847886562347,
+            0.004255570936948061,
+            -0.0029588397592306137,
+            0.026923293247818947,
+            0.0013603140832856297,
+            -0.00029351082048378885,
+            -0.0027546584606170654,
+            -0.009177418425679207,
+            -0.002872868673875928,
+            -0.014744049869477749,
+            0.011985808610916138,
+            -0.0020848000422120094,
+            0.019730376079678535,
+            -0.000688216823618859,
+            0.02342713437974453,
+            0.010244893841445446,
+            -0.015431818552315235,
+            -0.0068669444881379604,
+            0.0017149450723081827,
+            0.002801226219162345,
+            -0.0171512421220541,
+            -0.0021725622937083244,
+            -0.003159439191222191,
+            0.0017015120247378945,
+            0.010825199075043201,
+            8.04300289019011e-05,
+            9.75011134869419e-05,
+            0.001118520274758339,
+            -0.0015600178157910705,
+            -0.002203010255470872,
+            0.0011821030639111996,
+            -0.0007885164814069867,
+            -0.005598870106041431,
+            0.008790548890829086,
+            0.013139255344867706,
+            0.01436434406787157,
+            0.015059277415275574,
+            0.02391430363059044,
+            0.01644914411008358,
+            0.013905830681324005,
+            -0.0027600317262113094,
+            -0.0025074915029108524,
+            0.024774014949798584,
+            0.003986911382526159,
+            -0.004710501525551081,
+            0.004581545013934374,
+            0.0026668962091207504,
+            0.0019343504682183266,
+            -0.006086039822548628,
+            0.0055308095179498196,
+            -0.016090931370854378,
+            5.056960799265653e-05,
+            -0.0036143697798252106,
+            -0.002208383521065116,
+            -0.010617434978485107,
+            0.0009152343845926225,
+            -0.017208555713295937,
+            0.009127268567681313,
+            0.01571838930249214,
+            0.003442427609115839,
+            0.01058877818286419,
+            0.007708745077252388,
+            -0.030949607491493225,
+            -0.009299211204051971,
+            0.011362518183887005,
+            0.003689594566822052,
+            0.011226397007703781,
+            0.012358350679278374,
+            0.008181585930287838,
+            -0.017910653725266457,
+            0.008246064186096191,
+            -0.002319429535418749,
+            0.011806702241301537,
+            0.010424000211060047,
+            -0.008948162198066711
+          ],
+          "title": "Designing Models with Data, Model, and Expert-Parallelism"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "0a3b6fbc-348e-45a1-bf4b-8a37ddfd208f",
+      "type": "next",
+      "source": {
+        "id": "9cc3633d-79f3-4045-b6a2-d3df9e569d10",
+        "properties": {
+          "page_content": "5 Designing Models with Data, Model, and Expert-Parallelism 18 5.1 Data Parallelism 20 5.2 Model Parallelism 20 5.3 Model and Data Parallelism 21 5.4 Expert and Data Parallelism 22 5.5 Expert, Model and Data Parallelism 22 5.6 Towards Trillion Parameter Models 22 ",
+          "keyphrases": [
+            "Designing Models",
+            "Data Parallelism",
+            "Model Parallelism",
+            "Expert Parallelism",
+            "Trillion Parameter Models"
+          ],
+          "embedding": [
+            -0.004642441403120756,
+            0.005233492702245712,
+            -0.022710707038640976,
+            0.004320049658417702,
+            0.027123892679810524,
+            -0.0023946543224155903,
+            -0.029717355966567993,
+            0.04969131574034691,
+            -0.019472461193799973,
+            0.01448613591492176,
+            0.021908311173319817,
+            -0.0279119610786438,
+            0.043014224618673325,
+            -0.02931615710258484,
+            -0.037196844816207886,
+            0.025519099086523056,
+            0.024114903062582016,
+            0.01901394873857498,
+            -0.010932662524282932,
+            -0.05880425497889519,
+            0.020575758069753647,
+            -0.04052106291055679,
+            -0.04877429082989693,
+            0.019085591658949852,
+            0.03307022899389267,
+            0.024630729109048843,
+            0.0260635819286108,
+            0.01029504369944334,
+            -0.05450569838285446,
+            0.01792498119175434,
+            0.012745220214128494,
+            -0.014686735346913338,
+            0.02420087344944477,
+            -0.021263526752591133,
+            -0.03960403427481651,
+            -0.017409155145287514,
+            0.05966396629810333,
+            -0.004216167610138655,
+            -0.01887066476047039,
+            0.022108910605311394,
+            0.006763062439858913,
+            0.0028657044749706984,
+            -0.030462438240647316,
+            0.002539730630815029,
+            0.006548134610056877,
+            -0.02255309373140335,
+            -0.047570694237947464,
+            0.003571384120732546,
+            0.01928619109094143,
+            -0.024874314665794373,
+            0.008267557248473167,
+            -0.046309784054756165,
+            0.025963282212615013,
+            0.00751530984416604,
+            -0.022108910605311394,
+            4.404901119414717e-05,
+            0.028943615034222603,
+            0.037196844816207886,
+            0.007013811729848385,
+            0.017695724964141846,
+            0.00950697436928749,
+            -0.033901285380125046,
+            -0.05513615533709526,
+            0.018369166180491447,
+            -0.012265214696526527,
+            -0.035477422177791595,
+            -0.017967967316508293,
+            0.017595425248146057,
+            -0.026450451463460922,
+            -0.011283711530268192,
+            -0.0028764507733285427,
+            0.014801363460719585,
+            0.003943925723433495,
+            -0.006902765482664108,
+            0.009463989175856113,
+            -0.014521957375109196,
+            0.029975268989801407,
+            0.040062546730041504,
+            0.027654048055410385,
+            -0.001376433647237718,
+            -0.02430117316544056,
+            0.004194675013422966,
+            -0.010445492342114449,
+            0.02546178363263607,
+            0.02044680155813694,
+            -0.00542692793533206,
+            -0.005899769254028797,
+            -0.06487955152988434,
+            -0.017265869304537773,
+            0.010130265727639198,
+            -0.04218316823244095,
+            0.005924844183027744,
+            0.025934625416994095,
+            -0.009299211204051971,
+            -0.0171512421220541,
+            0.027553748339414597,
+            0.028069574385881424,
+            0.007192918099462986,
+            -0.01659242808818817,
+            -0.017409155145287514,
+            0.0003219439822714776,
+            0.047398753464221954,
+            0.0451061874628067,
+            -0.01139833964407444,
+            0.04106554388999939,
+            -0.009227568283677101,
+            0.005846037063747644,
+            0.03155140578746796,
+            -0.013518961146473885,
+            0.03499025106430054,
+            -0.025791339576244354,
+            -0.009772052057087421,
+            -0.016950642690062523,
+            0.008518306538462639,
+            -0.038772981613874435,
+            -0.007185753900557756,
+            -0.017867667600512505,
+            -0.04335810989141464,
+            0.020719043910503387,
+            -0.04120882973074913,
+            0.013017462566494942,
+            0.017982294782996178,
+            -0.02114889957010746,
+            0.03243977576494217,
+            -0.04295691102743149,
+            0.029574070125818253,
+            -0.013239555060863495,
+            -0.021335169672966003,
+            -0.015431818552315235,
+            -0.02100561372935772,
+            0.02230951003730297,
+            0.018039610236883163,
+            -0.03513353690505028,
+            -0.027768677100539207,
+            0.008819205686450005,
+            -0.032353803515434265,
+            0.025691039860248566,
+            0.0003951537946704775,
+            -0.006573209539055824,
+            0.002344504464417696,
+            -0.0020274859853088856,
+            -0.018612749874591827,
+            0.0076227737590670586,
+            -0.053502701222896576,
+            -0.020246202126145363,
+            -0.026536423712968826,
+            0.01940082013607025,
+            0.042068541049957275,
+            -0.0010889676632359624,
+            0.026822993531823158,
+            -0.013318361714482307,
+            0.0033672028221189976,
+            0.010115936398506165,
+            0.018999621272087097,
+            -0.00021448006737045944,
+            -0.006286639254540205,
+            -0.007938001304864883,
+            0.022982949391007423,
+            -0.004549305886030197,
+            0.001577928545884788,
+            0.009292046539485455,
+            0.009901008568704128,
+            0.00019175592751707882,
+            0.05324478819966316,
+            0.0067773908376693726,
+            0.000919712008908391,
+            0.061039503663778305,
+            0.0037110873963683844,
+            -0.04668232426047325,
+            0.024043260142207146,
+            0.0024286846164613962,
+            -0.00724306795746088,
+            0.03946075215935707,
+            0.012451485730707645,
+            -0.0010235938243567944,
+            0.017208555713295937,
+            0.013339853845536709,
+            0.01571838930249214,
+            -0.004660351667553186,
+            0.03467502444982529,
+            -0.0015474804677069187,
+            -0.017796024680137634,
+            -0.0029695862904191017,
+            0.0010003099450841546,
+            0.04794323444366455,
+            0.0009313539485447109,
+            0.029545413330197334,
+            0.009693245403468609,
+            0.01960141770541668,
+            -0.017380498349666595,
+            0.02044680155813694,
+            0.02574835531413555,
+            0.015016291290521622,
+            0.012465814128518105,
+            -0.029330484569072723,
+            -0.03258306160569191,
+            0.01591898873448372,
+            -0.010946990922093391,
+            -0.01806826703250408,
+            0.010861019603908062,
+            0.006372610107064247,
+            -0.0444757342338562,
+            -0.0026239105500280857,
+            -0.01911424845457077,
+            -0.000753590720705688,
+            0.0021922639571130276,
+            -0.006401267368346453,
+            0.011090275831520557,
+            0.03828581050038338,
+            -0.0343024842441082,
+            -0.006039472296833992,
+            0.016004959121346474,
+            -0.024917300790548325,
+            -0.006129025481641293,
+            -0.003829297609627247,
+            0.0020471876487135887,
+            -0.03355740010738373,
+            0.0253901407122612,
+            0.013332690112292767,
+            -0.04043509066104889,
+            -0.008969655260443687,
+            -0.008145765401422977,
+            0.006798883900046349,
+            -0.02054710127413273,
+            0.003039437811821699,
+            -0.02093397080898285,
+            0.006709330715239048,
+            -0.015517789870500565,
+            -0.02230951003730297,
+            -0.02633582428097725,
+            0.017280198633670807,
+            0.003111080499365926,
+            -0.00244122208096087,
+            -0.010932662524282932,
+            0.030978266149759293,
+            -0.013848517090082169,
+            -0.0075368029065430164,
+            -0.020805014297366142,
+            -0.03857238218188286,
+            0.03261171653866768,
+            0.01520256232470274,
+            -0.022395480424165726,
+            -0.04120882973074913,
+            -0.01995963230729103,
+            0.008718905970454216,
+            0.009026968851685524,
+            -0.015804359689354897,
+            0.052557021379470825,
+            -0.014758378267288208,
+            -0.001351358718238771,
+            0.0011588193010538816,
+            -0.0005270209512673318,
+            0.015575103461742401,
+            0.024731028825044632,
+            -0.010846691206097603,
+            0.006917094346135855,
+            0.013934488408267498,
+            0.0006443357560783625,
+            -0.005061550531536341,
+            -0.0090484619140625,
+            0.05284358933568001,
+            -0.030491095036268234,
+            -0.047742635011672974,
+            0.04545007273554802,
+            -0.032840974628925323,
+            0.009772052057087421,
+            0.016778700053691864,
+            0.001771363546140492,
+            0.03524816408753395,
+            0.007343367673456669,
+            -0.00307705020532012,
+            -0.015317190438508987,
+            0.0018662899965420365,
+            -0.012673578225076199,
+            0.02412923239171505,
+            -0.03587862104177475,
+            0.0015277786878868937,
+            0.007644266821444035,
+            0.02437281608581543,
+            0.0067021665163338184,
+            -0.021822338923811913,
+            0.02655075117945671,
+            -0.008102779276669025,
+            0.002068680478259921,
+            -0.005036475602537394,
+            -0.0005041848635300994,
+            -0.01732318289577961,
+            -0.005151103716343641,
+            -0.015876002609729767,
+            -0.02357041835784912,
+            -0.024487445130944252,
+            -0.01950111798942089,
+            -0.01676437072455883,
+            0.01785333827137947,
+            0.00706754345446825,
+            0.005398270674049854,
+            0.007572623901069164,
+            0.012995969504117966,
+            0.007196500431746244,
+            0.005982158239930868,
+            -0.03272634372115135,
+            0.013712395913898945,
+            0.007823373191058636,
+            -0.004277063999325037,
+            -0.023341162130236626,
+            -0.0018430062336847186,
+            0.008704577572643757,
+            0.012687906622886658,
+            0.0016218096716329455,
+            -0.009549960494041443,
+            0.012129094451665878,
+            0.0023803256917744875,
+            -0.00688127288594842,
+            0.0008588157943449914,
+            -0.06619777530431747,
+            0.05814514309167862,
+            -0.0012635965831577778,
+            0.009528467431664467,
+            -0.057887230068445206,
+            -0.018913649022579193,
+            0.05909082666039467,
+            0.008109943941235542,
+            -0.048487719148397446,
+            -0.011899838224053383,
+            0.016563771292567253,
+            -0.02391430363059044,
+            -0.00041127335862256587,
+            0.029129885137081146,
+            -0.03143677860498428,
+            -0.04163868725299835,
+            -0.050837598741054535,
+            -0.040033891797065735,
+            -0.035477422177791595,
+            0.012172079645097256,
+            -0.011584610678255558,
+            -0.005043639801442623,
+            -0.026751350611448288,
+            0.006845451425760984,
+            -0.00787352304905653,
+            -0.03037646785378456,
+            0.0027152549009770155,
+            -0.02784031815826893,
+            0.0009698618669062853,
+            0.010158922523260117,
+            -0.005982158239930868,
+            -0.004366617184132338,
+            -0.03209589049220085,
+            -0.012917162850499153,
+            0.006064547225832939,
+            0.0251179002225399,
+            0.0007222470594570041,
+            0.02100561372935772,
+            0.021349499002099037,
+            0.010352357290685177,
+            -0.007099783048033714,
+            -0.0015698687639087439,
+            0.0029731683898717165,
+            0.004764233715832233,
+            0.003082423470914364,
+            -0.009951158426702023,
+            0.035162195563316345,
+            -0.05676960572600365,
+            -0.029860639944672585,
+            0.006770226638764143,
+            -0.01111893355846405,
+            -0.024071916937828064,
+            0.020174559205770493,
+            -0.0023713705595582724,
+            -0.04275631159543991,
+            0.01176371704787016,
+            -0.025590740144252777,
+            0.033299487084150314,
+            0.02742479182779789,
+            0.02524685673415661,
+            -0.010130265727639198,
+            -0.024745358154177666,
+            -0.004506320226937532,
+            -0.02216622419655323,
+            -0.015976302325725555,
+            0.020088588818907738,
+            -0.010173250921070576,
+            -0.005824544467031956,
+            0.011656252667307854,
+            -0.002566596493124962,
+            -0.0036824301350861788,
+            0.014385836198925972,
+            0.02441580221056938,
+            0.009986979886889458,
+            -0.04665366932749748,
+            0.00562036270275712,
+            -0.01613391563296318,
+            -0.029516756534576416,
+            -0.0067845555022358894,
+            0.011727895587682724,
+            0.0015833016950637102,
+            -0.010338028892874718,
+            -0.011835359036922455,
+            -0.025819996371865273,
+            0.014729720540344715,
+            -0.0009125477517955005,
+            -0.017265869304537773,
+            0.024143559858202934,
+            0.00978638045489788,
+            0.02381400391459465,
+            0.05378927290439606,
+            -0.004954086616635323,
+            0.009170254692435265,
+            0.021335169672966003,
+            0.021765025332570076,
+            0.0260635819286108,
+            -0.009893844835460186,
+            -0.04014851897954941,
+            -0.01577570289373398,
+            0.003718251595273614,
+            0.03711087256669998,
+            -0.018598422408103943,
+            -0.0006416491232812405,
+            0.027310162782669067,
+            -0.04023449122905731,
+            -0.001844797283411026,
+            0.007672923617064953,
+            0.011849688366055489,
+            0.05548003688454628,
+            -0.020790686830878258,
+            -0.013526124879717827,
+            0.016105258837342262,
+            -0.01596197299659252,
+            -0.018956635147333145,
+            -0.007844866253435612,
+            -0.020174559205770493,
+            0.012172079645097256,
+            -0.007350531872361898,
+            0.034245166927576065,
+            -0.00932786799967289,
+            0.0020794267766177654,
+            -0.04765666648745537,
+            0.022223537787795067,
+            0.05049371346831322,
+            -0.026149554178118706,
+            -0.020532771944999695,
+            -0.029330484569072723,
+            -0.013361346907913685,
+            0.011362518183887005,
+            -0.004449006170034409,
+            -0.058030515909194946,
+            -0.015245547518134117,
+            -0.02136382646858692,
+            -0.030032582581043243,
+            -0.04481961950659752,
+            -0.03413053974509239,
+            -0.0667136013507843,
+            0.021062927320599556,
+            -0.07714476436376572,
+            -0.032955601811409,
+            0.031236179172992706,
+            -0.016033615916967392,
+            -0.029574070125818253,
+            0.013318361714482307,
+            -0.008776219561696053,
+            -0.056339748203754425,
+            -0.03885895386338234,
+            0.05249970406293869,
+            0.016921984031796455,
+            -0.0033313813619315624,
+            -0.029803326353430748,
+            0.0007710536010563374,
+            -0.03166603296995163,
+            -0.028212860226631165,
+            0.004731994587928057,
+            0.012609099969267845,
+            0.023341162130236626,
+            -0.013261047191917896,
+            0.009972651489078999,
+            -0.002568387659266591,
+            0.013970308937132359,
+            0.02332683466374874,
+            0.02507491409778595,
+            -0.014357179403305054,
+            -0.004914683289825916,
+            -0.020260531455278397,
+            0.04398856312036514,
+            0.0203035157173872,
+            0.014149416238069534,
+            -0.006816794630140066,
+            -0.01901394873857498,
+            0.025275513529777527,
+            0.010775049217045307,
+            -0.005233492702245712,
+            1.3845774446963333e-05,
+            -0.006748734042048454,
+            0.03771267086267471,
+            -0.015603761188685894,
+            0.0014167326735332608,
+            0.002708090702071786,
+            -0.005896186921745539,
+            -0.02536148391664028,
+            -0.053502701222896576,
+            -0.016606757417321205,
+            0.0203035157173872,
+            -0.0028997347690165043,
+            -0.010108772665262222,
+            0.0023373402655124664,
+            -0.0171512421220541,
+            -0.02497461438179016,
+            0.025017600506544113,
+            -0.003610787680372596,
+            -0.01527420524507761,
+            -0.04633844271302223,
+            -0.0219226386398077,
+            -0.015446146950125694,
+            -0.015259875915944576,
+            0.025203870609402657,
+            0.028356146067380905,
+            0.009542795829474926,
+            0.01283835619688034,
+            0.017065269872546196,
+            -0.03272634372115135,
+            0.0004092584131285548,
+            -0.02563372626900673,
+            -0.0067845555022358894,
+            -0.005785140674561262,
+            -0.009822201915085316,
+            -0.01732318289577961,
+            0.02276802249252796,
+            -0.015302862040698528,
+            0.035964589565992355,
+            0.0021367408335208893,
+            -0.009263389743864536,
+            0.04851637780666351,
+            -0.016406157985329628,
+            0.002040023449808359,
+            0.008539799600839615,
+            -0.003204215783625841,
+            -2.553667400206905e-05,
+            -0.0020991284400224686,
+            -0.035018909722566605,
+            0.018383493646979332,
+            0.04843040555715561,
+            -0.0009501601452939212,
+            -0.030290495604276657,
+            -0.022524436935782433,
+            0.02563372626900673,
+            0.004309303127229214,
+            -0.006000068504363298,
+            0.01952977664768696,
+            -0.013963145203888416,
+            0.028069574385881424,
+            -0.048802945762872696,
+            0.0008507560123689473,
+            0.02461640164256096,
+            0.013347018510103226,
+            -0.04169600084424019,
+            0.048344433307647705,
+            -0.00925622507929802,
+            -0.038916267454624176,
+            -0.02240980975329876,
+            0.03762669861316681,
+            0.014371507801115513,
+            0.02100561372935772,
+            -0.018111253157258034,
+            0.012186408042907715,
+            0.009206075221300125,
+            -0.03708221763372421,
+            0.017523782327771187,
+            -0.029660040512681007,
+            0.004001239780336618,
+            0.004058553837239742,
+            0.02868570201098919,
+            -0.01623421534895897,
+            0.01816856674849987,
+            0.02556208334863186,
+            -0.006229325197637081,
+            -0.005118864588439465,
+            0.006662762723863125,
+            0.021091584116220474,
+            0.03742609918117523,
+            0.034846965223550797,
+            -0.02391430363059044,
+            0.012881341390311718,
+            0.013024627231061459,
+            0.03132215142250061,
+            -0.0009779216488823295,
+            0.0019397237338125706,
+            0.02020321600139141,
+            0.025590740144252777,
+            -0.003840043907985091,
+            -0.007307546213269234,
+            -0.008812041021883488,
+            0.0013379257870838046,
+            -0.00915592536330223,
+            0.0036860124673694372,
+            0.03289828822016716,
+            -0.010001308284699917,
+            -0.03928880766034126,
+            -0.0025271931663155556,
+            0.001104191760532558,
+            0.010273550637066364,
+            0.0023319669999182224,
+            0.05705617740750313,
+            -0.016606757417321205,
+            -0.006437088828533888,
+            0.013755381107330322,
+            -0.015732716768980026,
+            -0.024788344278931618,
+            -0.013769710436463356,
+            0.005043639801442623,
+            -0.023126235231757164,
+            -0.032525744289159775,
+            0.00379705848172307,
+            0.04387393593788147,
+            -0.016921984031796455,
+            0.013253883458673954,
+            0.03341411426663399,
+            -0.014385836198925972,
+            0.006494402885437012,
+            -0.0009349361062049866,
+            0.003689594566822052,
+            -0.00751530984416604,
+            -0.04871697723865509,
+            0.0206187441945076,
+            0.078864187002182,
+            0.005863947793841362,
+            0.02963138371706009,
+            -0.011964316479861736,
+            0.012702235020697117,
+            -0.013841352425515652,
+            0.024774014949798584,
+            9.951606625691056e-05,
+            0.014858677983283997,
+            -0.008045465685427189,
+            -0.019902316853404045,
+            -0.003714669495820999,
+            -0.004825130105018616,
+            -0.01394165214151144,
+            0.009356524795293808,
+            -0.014873006381094456,
+            -0.023226534947752953,
+            0.006652016658335924,
+            0.0009465779876336455,
+            -0.03441711142659187,
+            -0.0020203215535730124,
+            -0.02153576910495758,
+            0.023197878152132034,
+            0.026221195235848427,
+            -0.0019253952195867896,
+            0.0018125580390915275,
+            0.005254985764622688,
+            -0.010409671813249588,
+            0.01952977664768696,
+            -0.003539144992828369,
+            0.03759804368019104,
+            0.01328970491886139,
+            0.006046636495739222,
+            0.041466742753982544,
+            0.01816856674849987,
+            -0.03473233804106712,
+            -0.014930319972336292,
+            0.012422828935086727,
+            -0.005072297062724829,
+            0.0164778009057045,
+            -0.001104191760532558,
+            0.007866358384490013,
+            -0.0036519821733236313,
+            0.02384266071021557,
+            0.012931491248309612,
+            -0.028943615034222603,
+            -0.004463334567844868,
+            0.006261564325541258,
+            -0.018741706386208534,
+            0.01409926638007164,
+            -0.000238435561186634,
+            -0.04983460158109665,
+            0.0016110632568597794,
+            0.034560397267341614,
+            0.004470498766750097,
+            -0.014844349585473537,
+            0.006394103169441223,
+            0.04467633366584778,
+            0.012387007474899292,
+            0.010008472949266434,
+            0.0066126128658652306,
+            0.014686735346913338,
+            -0.010982812382280827,
+            -0.015317190438508987,
+            -0.028399132192134857,
+            -0.007422174792736769,
+            -0.003703922964632511,
+            -0.02430117316544056,
+            -0.010015636682510376,
+            -0.03292694315314293,
+            -0.0036681017372757196,
+            0.014973306097090244,
+            -0.006917094346135855,
+            -0.005183342844247818,
+            -0.015560775063931942,
+            -0.010230564512312412,
+            -0.004183928482234478,
+            0.039403434842824936,
+            0.025060584768652916,
+            0.01366224605590105,
+            -0.004932594019919634,
+            -0.00029597352840937674,
+            -0.035792648792266846,
+            -0.0056955874897539616,
+            0.021550098434090614,
+            0.026149554178118706,
+            0.006852615624666214,
+            -0.00939951092004776,
+            -0.05292956158518791,
+            -0.015618089586496353,
+            0.0007746357587166131,
+            -0.00939951092004776,
+            0.02296862192451954,
+            0.015861675143241882,
+            0.012730891816318035,
+            0.03384396806359291,
+            -0.0060323080979287624,
+            -0.01746646873652935,
+            -0.010818034410476685,
+            0.03665236011147499,
+            -0.004388109780848026,
+            0.004237660672515631,
+            0.016262872144579887,
+            -0.005849619396030903,
+            -0.035792648792266846,
+            0.023541761562228203,
+            0.0064478348940610886,
+            0.0024143559858202934,
+            0.016090931370854378,
+            -0.0026418212801218033,
+            -0.001410463941283524,
+            -0.019200220704078674,
+            0.012702235020697117,
+            0.023197878152132034,
+            0.035334136337041855,
+            -0.0018859917763620615,
+            0.013898666948080063,
+            0.043186165392398834,
+            0.004656769800931215,
+            -0.008790548890829086,
+            0.02676567994058132,
+            -0.019099920988082886,
+            -0.003883029567077756,
+            0.017480798065662384,
+            -0.01564674638211727,
+            0.04671098291873932,
+            0.011670581065118313,
+            0.005208417773246765,
+            -0.040492404252290726,
+            0.003263321006670594,
+            -0.003961836453527212,
+            -0.014142251573503017,
+            0.03160871937870979,
+            0.015689732506871223,
+            -0.04381662234663963,
+            0.014034787192940712,
+            0.016893327236175537,
+            0.006902765482664108,
+            0.012086108326911926,
+            0.022882649675011635,
+            -0.0038113868795335293,
+            0.03665236011147499,
+            0.017380498349666595,
+            -0.006086039822548628,
+            0.034560397267341614,
+            -0.01265924982726574,
+            0.003514070063829422,
+            0.01422105822712183,
+            -0.007758894935250282,
+            -0.001464195898734033,
+            -0.006175593007355928,
+            0.015517789870500565,
+            0.03619384765625,
+            -0.007680087815970182,
+            -0.03811386972665787,
+            0.03983329236507416,
+            -0.012193572707474232,
+            0.005885440390557051,
+            -0.04986325651407242,
+            0.010051458142697811,
+            -0.01858409307897091,
+            -0.007415010128170252,
+            9.391898493049666e-05,
+            0.008260393515229225,
+            -0.010359521955251694,
+            -0.0012224020902067423,
+            0.015661075711250305,
+            0.00903413351625204,
+            0.014958977699279785,
+            0.024286845698952675,
+            0.02994661219418049,
+            0.008704577572643757,
+            -0.015460475347936153,
+            -0.006064547225832939,
+            0.03903089463710785,
+            -0.01222222950309515,
+            -0.013225225731730461,
+            0.04530678689479828,
+            -0.025447456166148186,
+            -0.027697034180164337,
+            0.03665236011147499,
+            -0.03539144992828369,
+            0.021306512877345085,
+            0.04608052968978882,
+            -0.004029897041618824,
+            -0.02974601276218891,
+            0.01527420524507761,
+            -0.030920950695872307,
+            0.010911169461905956,
+            -0.0026633141096681356,
+            0.0012098646257072687,
+            0.005294389091432095,
+            0.016320187598466873,
+            0.005358867347240448,
+            0.007332621142268181,
+            0.011197740212082863,
+            -0.01792498119175434,
+            0.02107725664973259,
+            -0.03146543353796005,
+            -0.013769710436463356,
+            0.01057444978505373,
+            -0.027510762214660645,
+            -0.0005879171658307314,
+            0.013053284026682377,
+            -0.016563771292567253,
+            -0.030433781445026398,
+            0.018111253157258034,
+            0.009184583090245724,
+            0.06522343307733536,
+            0.01995963230729103,
+            0.016750043258070946,
+            0.009277718141674995,
+            -0.01577570289373398,
+            0.05049371346831322,
+            -0.000637171498965472,
+            0.03215320408344269,
+            0.020117245614528656,
+            -0.016248544678092003,
+            -0.016406157985329628,
+            0.014744049869477749,
+            0.015345847234129906,
+            0.008281885646283627,
+            -0.008016807958483696,
+            0.003514070063829422,
+            0.008955326862633228,
+            0.0017525573493912816,
+            0.02689463645219803,
+            0.006408431567251682,
+            0.02321220561861992,
+            0.03229648992419243,
+            0.0035570557229220867,
+            0.025232527405023575,
+            0.001274342997930944,
+            0.015790032222867012,
+            -0.01914290525019169,
+            0.014958977699279785,
+            0.01901394873857498,
+            0.00896249059587717,
+            0.0193578340113163,
+            -0.006103950552642345,
+            -0.013719560578465462,
+            0.008253228850662708,
+            0.005799469538033009,
+            0.01058877818286419,
+            0.010166086256504059,
+            -0.013726724311709404,
+            0.014887334778904915,
+            -0.032038576900959015,
+            0.014006130397319794,
+            -0.050522368401288986,
+            -0.005785140674561262,
+            0.030204525217413902,
+            0.017108255997300148,
+            -0.025304170325398445,
+            0.018053937703371048,
+            -0.015345847234129906,
+            -0.007852029986679554,
+            -0.038601040840148926,
+            -0.014421657659113407,
+            -0.003650191007182002,
+            0.022710707038640976,
+            0.011226397007703781,
+            0.015603761188685894,
+            0.017308855429291725,
+            -0.011247890070080757,
+            -0.007715909276157618,
+            -0.00841800682246685,
+            -0.03911686688661575,
+            -0.04120882973074913,
+            -0.012193572707474232,
+            0.0006353803910315037,
+            0.00048045325092971325,
+            -0.010875348001718521,
+            -0.02226652391254902,
+            0.009499810636043549,
+            0.022839665412902832,
+            0.0010853855637833476,
+            -0.020074259489774704,
+            0.007393517531454563,
+            -0.022710707038640976,
+            -0.006143353879451752,
+            -0.0424983985722065,
+            -0.020045602694153786,
+            -0.02819853276014328,
+            0.013504632748663425,
+            -0.0126950703561306,
+            0.026665380224585533,
+            -0.003232872812077403,
+            0.0014373299200087786,
+            -0.006444253027439117,
+            0.009707573801279068,
+            -0.008826369419693947,
+            0.006100368220359087,
+            -0.03656638786196709,
+            0.05568063631653786,
+            -0.006125443149358034,
+            0.029831983149051666,
+            -0.016004959121346474,
+            0.05158267915248871,
+            0.00843233522027731,
+            -0.008919505402445793,
+            0.03762669861316681,
+            -0.003850790439173579,
+            0.010882512666285038,
+            0.008797712624073029,
+            -0.03246843069791794,
+            -0.005441256333142519,
+            -0.01834050938487053,
+            0.0008409051806665957,
+            0.010345193557441235,
+            -0.017552439123392105,
+            0.031150206923484802,
+            0.030175868421792984,
+            -0.004663933999836445,
+            0.058976199477910995,
+            0.003345709992572665,
+            -0.018927978351712227,
+            0.008511141873896122,
+            -0.059606652706861496,
+            -0.009772052057087421,
+            -0.028571072965860367,
+            0.011391174979507923,
+            0.015546446666121483,
+            0.026364481076598167,
+            0.0051905070431530476,
+            -0.014185236766934395,
+            0.027611061930656433,
+            0.04281362518668175,
+            0.0002937347162514925,
+            0.020031273365020752,
+            0.012351186014711857,
+            0.00960727408528328,
+            0.009091447107493877,
+            0.0021080838050693274,
+            0.017122583463788033,
+            0.01869872212409973,
+            -0.004921847488731146,
+            -0.0012752384645864367,
+            -0.01581868901848793,
+            -0.006078875623643398,
+            0.0036663105711340904,
+            -0.01732318289577961,
+            -0.008704577572643757,
+            -0.003494368400424719,
+            0.014958977699279785,
+            -0.012723728083074093,
+            0.017738711088895798,
+            -0.02384266071021557,
+            -0.0539325587451458,
+            0.00913443323224783,
+            -0.010789377614855766,
+            -0.02490297146141529,
+            0.009005476720631123,
+            -0.014973306097090244,
+            0.01401329506188631,
+            0.004506320226937532,
+            -0.006172011140733957,
+            -0.014127923175692558,
+            0.010373850353062153,
+            0.003519443329423666,
+            0.040807630866765976,
+            0.006315296050161123,
+            0.006766644772142172,
+            -0.01124072540551424,
+            0.013332690112292767,
+            -0.025304170325398445,
+            0.02342713437974453,
+            -0.010732063092291355,
+            -0.06098219007253647,
+            0.03086363710463047,
+            -0.012623428367078304,
+            -0.010674749501049519,
+            0.03413053974509239,
+            -0.012100436724722385,
+            -0.0006783659919165075,
+            -0.002858540276065469,
+            0.006641270127147436,
+            -0.02286832220852375,
+            0.008396513760089874,
+            -0.04226914048194885,
+            0.010954155586659908,
+            -0.04522081837058067,
+            -0.019945302978157997,
+            0.045020218938589096,
+            0.02149278298020363,
+            0.00312182679772377,
+            0.05103819817304611,
+            0.017652738839387894,
+            0.013776874169707298,
+            0.0035212342627346516,
+            0.014285536482930183,
+            -0.016076602041721344,
+            -0.014242551289498806,
+            -0.006856197956949472,
+            0.019372161477804184,
+            -0.005785140674561262,
+            -0.020776357501745224,
+            0.015016291290521622,
+            0.0035176521632820368,
+            0.0029534667264670134,
+            0.010087279602885246,
+            0.0010773257818073034,
+            -0.013139255344867706,
+            -0.013017462566494942,
+            0.013798367232084274,
+            0.032067231833934784,
+            -0.03395859897136688,
+            -0.0020077840890735388,
+            0.010889677330851555,
+            0.00779471592977643,
+            -0.003854372538626194,
+            -0.03117886371910572,
+            0.01304611936211586,
+            -0.02209458127617836,
+            0.006884855218231678,
+            -0.009026968851685524,
+            -0.013698067516088486,
+            -0.056483034044504166,
+            0.001492852927185595,
+            0.019902316853404045,
+            0.0005453793564811349,
+            -0.009865188039839268,
+            0.0406930036842823,
+            0.016463471576571465,
+            0.021378155797719955,
+            0.0018483793828636408,
+            0.013805530965328217,
+            0.014887334778904915,
+            -0.0006734405760653317,
+            3.0084299851296237e-06,
+            0.005118864588439465,
+            -0.0004316467384342104,
+            -0.0038615367375314236,
+            -0.0015510625671595335,
+            -0.0039725829847157,
+            0.03054841049015522,
+            0.02739613503217697,
+            0.006859780289232731,
+            0.003499741666018963,
+            0.020260531455278397,
+            -0.008998312056064606,
+            -0.005029311403632164,
+            0.010037129744887352,
+            0.013719560578465462,
+            0.006931422743946314,
+            0.007823373191058636,
+            0.01627720147371292,
+            0.002285399241372943,
+            0.003814968978986144,
+            0.013490303419530392,
+            0.012630592100322247,
+            0.0073720249347388744,
+            0.011541624553501606,
+            -0.008797712624073029,
+            0.019759032875299454,
+            -0.022639065980911255,
+            -0.0016880790935829282,
+            0.0008315020822919905,
+            -0.018842006102204323,
+            -0.0004016463935840875,
+            0.00967891700565815,
+            0.012816863134503365,
+            0.011462817899882793,
+            -0.002675851574167609,
+            0.02497461438179016,
+            -0.03630847483873367,
+            -0.011899838224053383,
+            0.004452588502317667,
+            0.028628388419747353,
+            -0.01360493153333664,
+            -0.01957276090979576,
+            0.013368511572480202,
+            -0.008231735788285732,
+            0.02700926549732685,
+            -0.01050280686467886,
+            0.0017534529324620962,
+            -0.013640752993524075,
+            0.05722811818122864,
+            -0.006132607348263264,
+            0.01367657445371151,
+            -0.0012071779929101467,
+            -0.00041284054168500006,
+            0.011011469177901745,
+            0.0015295697376132011,
+            0.04493424668908119,
+            0.026522094383835793,
+            0.009091447107493877,
+            0.018670065328478813,
+            0.02276802249252796,
+            0.016921984031796455,
+            0.005520063452422619,
+            -0.005788723006844521,
+            0.006401267368346453,
+            0.0006600075867027044,
+            -0.011305204592645168,
+            0.02132084034383297,
+            -0.007296800147742033,
+            -0.016721386462450027,
+            0.004011986311525106,
+            -0.010524299927055836,
+            -0.006343953311443329,
+            -0.013647917658090591,
+            0.007758894935250282,
+            0.03679564595222473,
+            -0.001645989017561078,
+            -0.009284882806241512,
+            -0.03928880766034126,
+            0.008281885646283627,
+            0.01693631336092949,
+            0.013733888976275921,
+            -0.020647400990128517,
+            0.005036475602537394,
+            0.020217545330524445,
+            -0.035792648792266846,
+            0.005409017205238342,
+            -0.021578755229711533,
+            0.011169083416461945,
+            -0.010424000211060047,
+            0.027367478236556053,
+            0.0006586642703041434,
+            0.003519443329423666,
+            0.0067881373688578606,
+            -0.009772052057087421,
+            0.02381400391459465,
+            -0.02167905494570732,
+            -0.0003752281772904098,
+            0.005434092134237289,
+            0.008718905970454216,
+            -0.029373470693826675,
+            0.006354699842631817,
+            0.018354836851358414,
+            -0.01591898873448372,
+            0.024817001074552536,
+            0.0285854022949934,
+            -0.008009644225239754,
+            -0.009105775505304337,
+            -0.010911169461905956,
+            0.005828126333653927,
+            0.002023903885856271,
+            0.029187200590968132,
+            -0.003188096219673753,
+            -0.0006156786694191396,
+            -0.007895016111433506,
+            0.010667584836483002,
+            0.0025522680953145027,
+            0.015001962892711163,
+            -0.008561291731894016,
+            -0.03143677860498428,
+            0.039403434842824936,
+            -0.0009385182056576014,
+            -0.008561291731894016,
+            -0.01690765656530857,
+            -0.007415010128170252,
+            0.0015958391595631838,
+            -0.008059794083237648,
+            0.00271346396766603,
+            -0.03476099669933319,
+            0.02146412618458271,
+            0.0020937554072588682,
+            -0.0005014982889406383,
+            0.0013191195903345942,
+            -0.01437867246568203,
+            -0.006849033758044243,
+            -0.0024089827202260494,
+            -0.013225225731730461,
+            0.012480142526328564,
+            0.004574380815029144,
+            -0.018526779487729073,
+            0.013991801999509335,
+            0.009428167715668678,
+            0.016821684315800667,
+            -0.0032973513007164,
+            0.005570212844759226,
+            0.020948300138115883,
+            -0.011348189786076546,
+            -0.03590727597475052,
+            -0.0036197430454194546,
+            -0.00271346396766603,
+            0.030118554830551147,
+            0.006268728524446487,
+            -0.023484447970986366,
+            0.01981634646654129,
+            0.004388109780848026,
+            0.007128439843654633,
+            0.0009922501631081104,
+            -0.008661591447889805,
+            -0.004739158786833286,
+            0.002597044687718153,
+            -0.002835256280377507,
+            -0.00011233337863814086,
+            -0.01022340077906847,
+            0.009478317573666573,
+            -0.00980070885270834,
+            0.005871111992746592,
+            -0.00884069874882698,
+            0.013597767800092697,
+            -0.006791719701141119,
+            -0.026665380224585533,
+            0.004051389638334513,
+            -0.002688389038667083,
+            -0.05235642194747925,
+            -0.0036448179744184017,
+            0.02013157308101654,
+            -0.0074579957872629166,
+            0.005681259091943502,
+            -0.0013862844789400697,
+            0.0028889882378280163,
+            0.017409155145287514,
+            0.008804877288639545,
+            0.026178210973739624,
+            0.016033615916967392,
+            0.01921454817056656,
+            -0.01623421534895897,
+            0.011054455302655697,
+            -0.009564288891851902,
+            0.039260152727365494,
+            0.002247786847874522,
+            -0.0068669444881379604,
+            -0.01736616902053356,
+            0.04418916255235672,
+            -0.00967891700565815,
+            -0.0015304653206840158,
+            -0.01455777883529663,
+            -0.014142251573503017,
+            0.03269768878817558,
+            -0.002256742212921381,
+            0.011512967757880688,
+            -0.004760651383548975,
+            0.008031136356294155,
+            0.0206617284566164,
+            -0.02556208334863186,
+            -0.02766837738454342,
+            -0.017767367884516716,
+            0.007214411161839962,
+            -0.004112286027520895,
+            0.04106554388999939,
+            -0.017910653725266457,
+            -0.0016836014110594988,
+            0.02718120627105236,
+            -0.0269662793725729,
+            -0.03418785333633423,
+            -0.010158922523260117,
+            0.014615092426538467,
+            -0.0010486687533557415,
+            -0.005057968199253082,
+            -0.01942947693169117,
+            -0.020647400990128517,
+            -0.000571797601878643,
+            0.0021868906915187836,
+            0.015603761188685894,
+            0.008862190879881382,
+            0.015388833358883858,
+            0.0019827093929052353,
+            0.0031576480250805616,
+            -0.012236557900905609,
+            0.013683739118278027,
+            0.010810869745910168,
+            0.002262115478515625,
+            -0.014521957375109196,
+            -0.011011469177901745,
+            -0.022366823628544807,
+            -0.008847862482070923,
+            -0.00808128621429205,
+            -0.00023933110060170293,
+            -0.0034531739074736834,
+            -0.015317190438508987,
+            0.02805524691939354,
+            0.00480363704264164,
+            0.00878338422626257,
+            0.015804359689354897,
+            -0.023584747686982155,
+            0.019099920988082886,
+            0.013920159079134464,
+            -0.010739227756857872,
+            0.003189887385815382,
+            -0.006770226638764143,
+            -0.012229394167661667,
+            -0.014858677983283997,
+            0.0038078047800809145,
+            0.015087934210896492,
+            0.011921330355107784,
+            -0.0003962731861975044,
+            -0.015345847234129906,
+            -0.01268074195832014,
+            -0.011713567189872265,
+            0.02563372626900673,
+            -0.030634380877017975,
+            -0.002115248003974557,
+            0.023942960426211357,
+            -0.01995963230729103,
+            -0.011899838224053383,
+            -0.03143677860498428,
+            0.0027994350530207157,
+            -0.02960272692143917,
+            -0.029402127489447594,
+            0.01897096447646618,
+            0.049146831035614014,
+            -0.013261047191917896,
+            -0.010345193557441235,
+            0.01627720147371292,
+            -0.016463471576571465,
+            -0.0021671890281140804,
+            -0.033614713698625565,
+            0.02616388164460659,
+            -0.021664725616574287,
+            0.0038472083397209644,
+            -0.008740399032831192,
+            -0.03499025106430054,
+            0.012687906622886658,
+            0.023541761562228203,
+            0.03338545560836792,
+            -0.001543898251838982,
+            0.00379705848172307,
+            -0.007637102622538805,
+            0.017839010804891586,
+            0.018111253157258034,
+            0.002573760924860835,
+            0.03301291540265083,
+            0.0011032961774617434,
+            -0.04642441123723984,
+            -0.00022880859614815563,
+            0.005151103716343641,
+            -0.0003194812743458897,
+            0.002509282436221838,
+            -0.01366224605590105,
+            -0.026808666065335274,
+            0.015059277415275574,
+            -0.026536423712968826,
+            -0.00040813899249769747,
+            -0.016434814780950546,
+            -0.010818034410476685,
+            -0.007780387531965971,
+            0.005509316921234131,
+            0.004495573695749044,
+            0.024000274017453194,
+            0.009557124227285385,
+            -0.0006985154468566179,
+            0.015016291290521622,
+            0.014529122039675713,
+            0.03069169446825981,
+            -0.0006680673686787486,
+            0.02219488099217415,
+            0.02686597965657711,
+            -0.013110597617924213,
+            0.007343367673456669,
+            0.0028083904180675745,
+            0.026350151747465134,
+            -0.03163737803697586,
+            0.02517521381378174,
+            0.01250163558870554,
+            0.003983329050242901,
+            0.02657940983772278,
+            -0.011985808610916138,
+            -0.00912010483443737,
+            -0.00014709123934153467,
+            0.004001239780336618,
+            -0.003247201442718506,
+            -0.0046925912611186504,
+            0.040635690093040466,
+            0.011491474695503712,
+            -0.024186545982956886,
+            -0.012401335872709751,
+            -0.0158903319388628,
+            0.028040917590260506,
+            -0.019773360341787338,
+            0.00017183032468892634,
+            -0.016492128372192383,
+            0.004918265156447887,
+            0.014314194209873676,
+            0.0014597182162106037,
+            0.01879902184009552,
+            0.016019288450479507,
+            -0.0006586642703041434,
+            0.018354836851358414,
+            0.007751730736345053,
+            -0.026393137872219086,
+            0.004839458502829075,
+            0.007436503190547228,
+            -0.014020458795130253,
+            0.007529638707637787,
+            0.015503461472690105,
+            -0.00407646456733346,
+            -0.013683739118278027,
+            -0.01230820082128048,
+            -0.00862576998770237,
+            0.004595873411744833,
+            -0.010638928040862083,
+            0.019300520420074463,
+            0.0018806185107678175,
+            -0.0038113868795335293,
+            0.019859332591295242,
+            -0.006734405644237995,
+            -0.023025935515761375,
+            0.01897096447646618,
+            -0.01437867246568203,
+            0.02441580221056938,
+            0.012530292384326458,
+            -0.022123238071799278,
+            0.00579588720574975,
+            0.009929666295647621,
+            0.04226914048194885,
+            -0.005878276191651821,
+            0.00615410041064024,
+            -0.02703792229294777,
+            -0.005584541708230972,
+            -0.02609223872423172,
+            0.0067845555022358894,
+            0.003377949120476842,
+            -0.011276546865701675,
+            -0.00021190541156101972,
+            -0.0012447903864085674,
+            0.014142251573503017,
+            0.012809699401259422,
+            -0.014873006381094456,
+            -0.0011149381753057241,
+            0.0036860124673694372,
+            -0.01528853364288807,
+            -0.0024698791094124317,
+            0.0011955361114814878,
+            -0.008317707106471062,
+            0.02293996512889862,
+            0.0393461212515831,
+            -0.021693382412195206,
+            -0.01760975457727909,
+            0.016878999769687653,
+            -0.003571384120732546,
+            -0.022438466548919678,
+            -0.007508145645260811,
+            0.01113326195627451,
+            0.0054376740008592606,
+            -0.0025558501947671175,
+            -0.03372934088110924,
+            0.021507112309336662,
+            0.006472909823060036,
+            0.014958977699279785,
+            0.00788785144686699,
+            0.0026955532375723124,
+            0.030347811058163643,
+            -0.020002616569399834,
+            -0.018770365044474602,
+            -0.00569916982203722,
+            -0.020188888534903526,
+            0.019099920988082886,
+            0.008790548890829086,
+            -0.011770880781114101,
+            0.02829883247613907,
+            -0.026493437588214874,
+            0.015560775063931942,
+            0.009299211204051971,
+            -0.0051976717077195644,
+            0.010603106580674648,
+            0.0028227188158780336,
+            -0.004914683289825916,
+            -0.04854503273963928,
+            -0.005100953858345747,
+            0.008074122481048107,
+            0.02104859985411167,
+            -0.0060251434333622456,
+            -0.031408119946718216,
+            0.02974601276218891,
+            0.0008785175159573555,
+            -0.002616746351122856,
+            -0.010058622807264328,
+            0.00706754345446825,
+            -0.020561428740620613,
+            -0.006863362155854702,
+            0.007053215056657791,
+            0.020461129024624825,
+            0.0059427544474601746,
+            0.028485102578997612,
+            -0.006028725765645504,
+            0.05232776328921318,
+            -0.0006617986364290118,
+            0.009435332380235195,
+            0.0026418212801218033,
+            -0.002097337506711483,
+            -0.0017946474254131317,
+            -0.013504632748663425,
+            -0.026292838156223297,
+            -0.021120242774486542,
+            -0.006010815035551786,
+            -0.00208300887607038,
+            0.010316535830497742,
+            0.008704577572643757,
+            -0.02034650184214115,
+            0.0009967278456315398,
+            -0.039546720683574677,
+            0.009170254692435265,
+            -0.027023592963814735,
+            -0.009843694977462292,
+            -0.0035122791305184364,
+            -0.004481245297938585,
+            -0.022567423060536385,
+            0.022366823628544807,
+            0.019930975511670113,
+            -0.032181862741708755,
+            0.002450177213177085,
+            0.008847862482070923,
+            0.0079451659694314,
+            0.01339716836810112,
+            2.5690593247418292e-05,
+            -0.02412923239171505,
+            -0.014744049869477749,
+            -0.0006953810807317495,
+            0.01029504369944334,
+            -0.008740399032831192,
+            -0.026421794667840004,
+            0.0019253952195867896,
+            0.03788461163640022,
+            -0.007017393596470356,
+            0.013461646623909473,
+            -0.00047776667634025216,
+            -0.00869741290807724,
+            -0.0222378671169281,
+            0.004080046899616718,
+            0.019701717421412468,
+            0.025576412677764893,
+            -0.009378017857670784,
+            0.004954086616635323,
+            -0.0029928700532764196,
+            0.006343953311443329,
+            -0.00632246071472764,
+            0.0006474701222032309,
+            0.01826886646449566,
+            -0.029545413330197334,
+            0.004739158786833286,
+            -0.008031136356294155,
+            -0.011978644877672195,
+            0.010832362808287144,
+            -0.0004674680531024933,
+            0.014256879687309265,
+            -0.016506457701325417,
+            -0.0018573347479104996,
+            0.030003925785422325,
+            0.00407646456733346,
+            -0.025017600506544113,
+            0.012752384878695011,
+            -0.02556208334863186,
+            -0.007192918099462986,
+            -0.0103093720972538,
+            0.023685047402977943,
+            0.01331119704991579,
+            0.00925622507929802,
+            0.027152549475431442,
+            -0.0006125443032942712,
+            0.01792498119175434,
+            -0.004878861829638481,
+            -0.0033277992624789476,
+            0.0016853924607858062,
+            -0.03209589049220085,
+            0.0047785621136426926,
+            0.029889296740293503,
+            -0.03395859897136688,
+            -0.004402438644319773,
+            0.010524299927055836,
+            0.02359907701611519,
+            -0.016993626952171326,
+            0.008281885646283627,
+            0.014092101715505123,
+            -0.006326042581349611,
+            0.00897681899368763,
+            0.025089243426918983,
+            0.049462057650089264,
+            -0.00248599867336452,
+            0.006175593007355928,
+            0.009578617289662361,
+            0.013626424595713615,
+            -0.0007370233652181923,
+            0.008812041021883488,
+            -0.05089491233229637,
+            0.01258760690689087,
+            -0.006394103169441223,
+            -0.022682050243020058,
+            -0.0053624496795237064,
+            -0.016248544678092003,
+            0.017093926668167114,
+            0.026135224848985672,
+            0.02352743409574032,
+            0.01518823392689228,
+            0.013339853845536709,
+            0.02841345965862274,
+            0.012007301673293114,
+            0.0052728960290551186,
+            0.004689008928835392,
+            -0.003426307812333107,
+            0.0022065923549234867,
+            -0.004126614425331354,
+            0.009557124227285385,
+            -0.005287224892526865,
+            0.02819853276014328,
+            0.007723073475062847,
+            0.043472737073898315,
+            -0.03401591256260872,
+            -0.006397685036063194,
+            0.015790032222867012,
+            -0.016735713928937912,
+            -0.0017821099609136581,
+            0.007687252014875412,
+            0.02027485892176628,
+            -0.014543450437486172,
+            0.011985808610916138,
+            -0.0033994419500231743,
+            0.010323700495064259,
+            -0.0042698998004198074,
+            0.01339716836810112,
+            -0.002792270854115486,
+            0.016004959121346474,
+            0.01824020966887474,
+            0.006576791871339083,
+            0.005738573148846626,
+            0.006017979234457016,
+            0.00019522612274158746,
+            -0.022366823628544807,
+            -0.011892673559486866,
+            0.00208300887607038,
+            0.004230496007949114,
+            -0.00625081779435277,
+            0.004456170368939638,
+            0.014622257091104984,
+            -0.02034650184214115,
+            0.016578100621700287,
+            0.019515447318553925,
+            -0.004753487184643745,
+            -0.00034903385676443577,
+            0.0015304653206840158,
+            -0.011641924269497395,
+            -0.015245547518134117,
+            -0.022180551663041115,
+            -0.011161918751895428,
+            -0.012859849259257317,
+            -0.005681259091943502,
+            0.01654944382607937,
+            0.00994399469345808,
+            -0.014729720540344715,
+            0.019271861761808395,
+            -0.027768677100539207,
+            0.019071262329816818,
+            0.004742741119116545,
+            -0.0055308095179498196,
+            -0.014593600295484066,
+            -0.00994399469345808,
+            0.04416050761938095,
+            0.00787352304905653,
+            0.002009575255215168,
+            -0.010882512666285038,
+            -0.004201839212328196,
+            -0.018741706386208534,
+            -0.002480625407770276,
+            -0.0090484619140625,
+            0.024774014949798584,
+            0.01630585826933384,
+            0.008074122481048107,
+            -0.020160231739282608,
+            0.008281885646283627,
+            0.005462748929858208,
+            0.016176901757717133,
+            0.012394172139465809,
+            0.01819722354412079,
+            -0.010402507148683071,
+            -0.021765025332570076,
+            0.0010424000211060047,
+            -0.007880687713623047,
+            0.017767367884516716,
+            0.008983983658254147,
+            0.0038221334107220173,
+            0.006182757206261158,
+            -0.014299864880740643,
+            0.012773877941071987,
+            0.027969276532530785,
+            -0.0004370199458207935,
+            0.021750696003437042,
+            -0.00995832309126854,
+            -0.00834636390209198,
+            0.006852615624666214,
+            0.017409155145287514,
+            0.018354836851358414,
+            -0.03137946501374245,
+            -0.009471152909100056,
+            -0.008862190879881382,
+            -0.011075947433710098,
+            -0.011706402525305748,
+            0.0003340336843393743,
+            -0.019973959773778915,
+            -0.0126950703561306,
+            -0.0104884784668684,
+            -0.01749512553215027,
+            0.005401853006333113,
+            -0.009041297249495983,
+            0.004320049658417702,
+            -0.013089105486869812,
+            -0.034388452768325806,
+            -0.01504494808614254,
+            0.020074259489774704,
+            -0.002005993155762553,
+            -0.009793545119464397,
+            0.008145765401422977,
+            -0.0016997209750115871,
+            0.010373850353062153,
+            0.007629937957972288,
+            -0.013812695629894733,
+            -0.014350014738738537,
+            0.00013623290578834713,
+            -0.004474081099033356,
+            0.010538628324866295,
+            -0.010595941916108131,
+            -0.004581545013934374,
+            -0.012974477373063564,
+            -0.004277063999325037,
+            -0.017452139407396317,
+            -0.015790032222867012,
+            0.006730823311954737,
+            -0.018956635147333145,
+            0.01360493153333664,
+            -0.023728033527731895,
+            -0.012387007474899292,
+            0.020074259489774704,
+            0.002625701716169715,
+            -0.016850342974066734,
+            -0.01869872212409973,
+            -0.008045465685427189,
+            -0.019615747034549713,
+            -0.009456824511289597,
+            0.003207797883078456,
+            0.02686597965657711,
+            0.005838872864842415,
+            0.014314194209873676,
+            -0.0017068851739168167,
+            0.032038576900959015,
+            0.006286639254540205,
+            0.029889296740293503,
+            -0.005863947793841362,
+            0.009363689459860325,
+            -0.02458774484694004,
+            -0.003499741666018963,
+            -0.009112940169870853,
+            0.0036340714432299137,
+            0.00044642301509156823,
+            -0.006852615624666214,
+            -0.0012734474148601294,
+            -0.0023283849004656076,
+            -0.007400681730359793,
+            -0.009585781022906303,
+            0.015575103461742401,
+            0.006591120269149542,
+            0.005129611119627953,
+            0.004183928482234478,
+            0.03332814201712608,
+            -0.006684255786240101,
+            -0.013655081391334534,
+            0.01366224605590105,
+            0.023470120504498482,
+            -0.016893327236175537,
+            -0.000970757391769439,
+            0.009084283374249935,
+            0.011290875263512135,
+            -0.023871317505836487,
+            0.01940082013607025,
+            -0.004696173127740622,
+            -0.004871697630733252,
+            0.008074122481048107,
+            0.002426893450319767,
+            0.01705094240605831,
+            -0.0029212275985628366,
+            -0.0019988289568573236,
+            -0.009936830028891563,
+            0.003943925723433495,
+            -0.015102262608706951,
+            -0.0007929941639304161,
+            0.01255895011126995,
+            -0.011405503377318382,
+            0.008088450878858566,
+            -0.027825990691781044,
+            -0.029831983149051666,
+            -0.01705094240605831,
+            0.01050280686467886,
+            0.002369579393416643,
+            -0.022395480424165726,
+            0.018956635147333145,
+            -0.0038113868795335293,
+            -0.017280198633670807,
+            0.018598422408103943,
+            -0.006501567084342241,
+            -0.003922433126717806,
+            -0.0005767230177298188,
+            0.020633071660995483,
+            0.005842455197125673,
+            -0.005623945035040379,
+            -0.002880033105611801,
+            -0.0069529153406620026,
+            -0.00317376758903265,
+            -0.002838838379830122,
+            0.001708676340058446,
+            -0.0206187441945076,
+            0.00434154225513339,
+            0.01268074195832014,
+            0.01513091940432787,
+            -0.0007625460275448859,
+            0.02829883247613907,
+            -0.005086625460535288,
+            0.00397974718362093,
+            0.006852615624666214,
+            -0.0393461212515831,
+            5.3284191380953416e-05,
+            -0.018569765612483025,
+            -0.008582784794270992,
+            0.010044294409453869,
+            0.009478317573666573,
+            0.001631660503335297,
+            -0.008016807958483696,
+            0.0028065992519259453,
+            -0.009213239885866642,
+            -2.209446938650217e-05,
+            0.007286053616553545,
+            0.022080253809690475,
+            0.013196568936109543,
+            -0.004434677772223949,
+            0.02991795539855957,
+            -0.011434161104261875,
+            0.007443667389452457,
+            0.012322529219090939,
+            0.008898012340068817,
+            -0.0026382391806691885,
+            0.02076202817261219,
+            -0.009198911488056183,
+            -0.009772052057087421,
+            -0.01241566427052021,
+            0.014163744635879993,
+            0.01571838930249214,
+            -0.025261184200644493,
+            -0.0024770433083176613,
+            -0.003428098978474736,
+            0.005262149963527918,
+            -0.002656149910762906,
+            0.012974477373063564,
+            -0.01882767863571644,
+            0.025160884484648705,
+            0.031150206923484802,
+            -0.008704577572643757,
+            -0.011584610678255558,
+            0.0019110667053610086,
+            0.00267406040802598,
+            0.01268074195832014,
+            0.0033385457936674356,
+            0.006519477814435959,
+            -0.003358247457072139,
+            -0.02626418136060238,
+            -0.021836668252944946,
+            0.008403678424656391,
+            -0.01436434406787157,
+            -0.004072882700711489,
+            0.00931353960186243,
+            0.016076602041721344,
+            0.0010352357057854533,
+            0.0060358899645507336,
+            0.022452794015407562,
+            -0.02514655701816082,
+            0.0036824301350861788,
+            -0.01083952747285366,
+            -0.0023892810568213463,
+            -0.006017979234457016,
+            -0.034531738609075546,
+            0.011713567189872265,
+            0.006168428808450699,
+            -0.017667068168520927,
+            0.0023767435923218727,
+            -0.014063444919884205,
+            0.016463471576571465,
+            -0.0007370233652181923,
+            0.03160871937870979,
+            0.008195914328098297,
+            -0.006813212297856808,
+            0.00489319022744894,
+            0.02332683466374874,
+            0.04083628952503204,
+            0.010037129744887352,
+            0.004835876170545816,
+            -0.019300520420074463,
+            -0.00960727408528328,
+            -0.010746391490101814,
+            -0.007708745077252388,
+            -0.01942947693169117,
+            0.009528467431664467,
+            -0.012816863134503365,
+            -0.002960630925372243,
+            0.01213625818490982,
+            -0.0010952363954856992,
+            -0.020991284400224686,
+            0.014729720540344715,
+            0.014271208085119724,
+            -0.0316946916282177,
+            -0.013948816806077957,
+            -0.031236179172992706,
+            0.03054841049015522,
+            -0.020919643342494965,
+            -0.0257053691893816,
+            0.024501772597432137,
+            -0.007572623901069164,
+            0.018498122692108154,
+            0.010595941916108131,
+            0.005842455197125673,
+            -0.0002697792078834027,
+            0.004731994587928057,
+            0.007175007369369268,
+            -0.025576412677764893,
+            0.028786001726984978,
+            0.006759480573236942,
+            -0.016420487314462662,
+            0.008554127998650074,
+            -0.025347156450152397,
+            -0.024114903062582016,
+            0.02170771174132824,
+            0.019773360341787338,
+            0.00048358761705458164,
+            0.01078221295028925,
+            -0.0007495608297176659,
+            -0.0009940412128344178,
+            -0.013970308937132359,
+            -0.008940997533500195,
+            -0.000573140918277204,
+            0.02132084034383297,
+            -0.017036613076925278,
+            0.008446663618087769,
+            0.01255895011126995,
+            8.630695811007172e-05,
+            -0.02805524691939354,
+            -0.00924189668148756,
+            -0.004491991829127073,
+            -0.028800329193472862,
+            0.0076227737590670586,
+            -0.00469975546002388,
+            0.003743326524272561,
+            -0.02381400391459465,
+            0.0019468879327178001,
+            -0.04201122745871544,
+            -0.03089229390025139,
+            0.00140509067568928,
+            -0.029101228341460228,
+            -0.012250886298716068,
+            -0.02371370419859886,
+            0.01131236832588911,
+            -0.01503061968833208,
+            -0.0006752316257916391,
+            0.007665759418159723,
+            -0.01725153997540474,
+            0.006583956070244312,
+            -0.019859332591295242,
+            0.004363034851849079,
+            -0.011441324837505817,
+            0.012673578225076199,
+            0.017595425248146057,
+            0.020174559205770493,
+            0.005333792418241501,
+            -0.002863913308829069,
+            0.03103557974100113,
+            -0.008403678424656391,
+            -0.016363171860575676,
+            -0.006476492155343294,
+            0.017910653725266457,
+            -0.00014597181871067733,
+            0.03576399013400078,
+            0.0016558398492634296,
+            -0.0008068748866207898,
+            0.0036000411491841078,
+            0.007175007369369268,
+            0.0047785621136426926,
+            -0.006831123027950525,
+            0.0269662793725729,
+            0.010352357290685177,
+            0.012752384878695011,
+            -0.006150518078356981,
+            -0.009657423943281174,
+            -0.006759480573236942,
+            0.014077773317694664,
+            0.016463471576571465,
+            0.012107601389288902,
+            0.010409671813249588,
+            0.00489319022744894,
+            -0.017065269872546196,
+            0.01894230581820011,
+            -0.02422953210771084,
+            0.011369682848453522,
+            -0.020890984684228897,
+            0.007343367673456669,
+            -0.0007303068996407092,
+            0.00834636390209198,
+            -0.0020167394541203976,
+            -0.005745737347751856,
+            0.00823890045285225,
+            0.012236557900905609,
+            -0.01230820082128048,
+            -0.0014015085762366652,
+            -0.0253901407122612,
+            -0.01201446633785963,
+            0.0067057483829557896,
+            0.018297523260116577,
+            0.024243859574198723,
+            -0.03089229390025139,
+            0.0038078047800809145,
+            -0.000754934037104249,
+            -0.020719043910503387,
+            0.0035212342627346516,
+            0.013282540254294872,
+            -0.015546446666121483,
+            -0.0009528467198833823,
+            0.023054592311382294,
+            -0.002648985479027033,
+            0.009198911488056183,
+            0.013182240538299084,
+            -0.009299211204051971,
+            0.015317190438508987,
+            0.013741052709519863,
+            0.01729452610015869,
+            -0.009277718141674995,
+            0.016176901757717133,
+            -0.018541108816862106,
+            0.008647263050079346,
+            0.02059008739888668,
+            0.020461129024624825,
+            -0.02069038711488247,
+            0.026636723428964615,
+            0.018254537135362625,
+            -0.020174559205770493,
+            -0.0016925566596910357,
+            -0.007773223333060741,
+            0.011842523701488972,
+            0.014235386624932289,
+            -0.004312884993851185,
+            -0.025089243426918983,
+            -0.01346881128847599,
+            0.010280714370310307,
+            -0.010509971529245377,
+            -0.004739158786833286,
+            -0.015259875915944576,
+            0.010366685688495636,
+            0.0007634415524080396,
+            0.020676057785749435,
+            0.03258306160569191,
+            0.014335686340928078,
+            -0.006662762723863125,
+            0.02599193900823593,
+            0.03215320408344269,
+            0.00541976373642683,
+            -0.005946336779743433,
+            0.010022801347076893,
+            0.0010567285353317857,
+            -0.009213239885866642,
+            -0.003437054343521595,
+            -0.001859125797636807,
+            -0.0142927011474967,
+            -0.0064263422973454,
+            0.014930319972336292,
+            0.03100692294538021,
+            -0.005222746171057224,
+            -0.020575758069753647,
+            0.00858994945883751,
+            0.00912010483443737,
+            0.014564942568540573,
+            0.004997072275727987,
+            0.018999621272087097,
+            0.015489133074879646,
+            0.016219887882471085,
+            0.005835290532559156,
+            -0.023685047402977943,
+            -0.008031136356294155,
+            0.009750558994710445,
+            -0.02114889957010746,
+            -0.006451417226344347,
+            -0.004843040369451046,
+            -0.039747320115566254,
+            -0.021550098434090614,
+            0.014550614170730114,
+            0.026436123996973038,
+            0.01167774572968483,
+            -0.0009832947980612516,
+            -0.015073605813086033,
+            0.0029140631668269634,
+            0.0038113868795335293,
+            0.01346881128847599,
+            0.012601935304701328,
+            0.0013746425975114107,
+            0.00011899389937752858,
+            0.02301160618662834,
+            0.0028674956411123276,
+            -0.012580442242324352,
+            0.005785140674561262,
+            -0.008181585930287838,
+            -0.007923672907054424,
+            0.003062721574679017,
+            -0.005161850247532129,
+            -0.004452588502317667,
+            0.010775049217045307,
+            0.006096786353737116,
+            -0.003505114698782563,
+            -0.010194743983447552,
+            0.0011641924502328038,
+            0.010638928040862083,
+            0.002199428156018257,
+            0.0020453964825719595,
+            0.008847862482070923,
+            -0.011534460820257664,
+            -0.01043116394430399,
+            -0.015503461472690105,
+            0.007694416679441929,
+            -0.015016291290521622,
+            0.006902765482664108,
+            0.022108910605311394,
+            -0.004488409496843815,
+            -0.026708366349339485,
+            -0.008253228850662708,
+            0.008203078992664814,
+            -0.014507628977298737,
+            0.005745737347751856,
+            0.018096923828125,
+            -0.0009582199272699654,
+            0.009850858710706234,
+            -0.01346881128847599,
+            0.015489133074879646,
+            0.013253883458673954,
+            0.02139248326420784,
+            -0.0004813488048966974,
+            -0.007758894935250282,
+            0.011706402525305748,
+            0.0036681017372757196,
+            -0.017122583463788033,
+            0.009714738465845585,
+            0.03427382558584213,
+            -0.001524196588434279,
+            0.005964247509837151,
+            0.0016943477094173431,
+            0.0043558706529438496,
+            -0.001223297556862235,
+            -0.0010379223385825753,
+            0.004724830389022827,
+            0.012816863134503365,
+            0.005677676759660244,
+            -0.0038758653681725264,
+            0.009721902199089527,
+            0.008804877288639545,
+            -0.004735576454550028,
+            -0.02703792229294777,
+            -0.004499156028032303,
+            -0.0008422484388574958,
+            -0.0067773908376693726,
+            0.007386353332549334,
+            -0.016721386462450027,
+            0.007099783048033714,
+            -0.006043054163455963,
+            0.014350014738738537,
+            -0.0033134708646684885,
+            0.009736230596899986,
+            0.019128577783703804,
+            0.01422105822712183,
+            0.01237984374165535,
+            -0.007286053616553545,
+            0.010080115869641304,
+            0.007952329702675343,
+            0.009793545119464397,
+            0.0027403298299759626,
+            -0.006071711424738169,
+            0.0021421140991151333,
+            -0.004835876170545816,
+            0.0064263422973454,
+            0.0034191436134278774,
+            0.015603761188685894,
+            -0.01558943185955286,
+            -0.017308855429291725,
+            -0.000402318051783368,
+            0.032840974628925323,
+            0.001868081046268344,
+            0.0019182309042662382,
+            -0.009234732948243618,
+            0.010266385972499847,
+            0.000753590720705688,
+            -0.012859849259257317,
+            -0.007257396355271339,
+            0.026350151747465134,
+            -0.0021295766346156597,
+            0.0017749457620084286,
+            -0.0034728755708783865,
+            -0.0016692728968337178,
+            0.012745220214128494,
+            0.012365514412522316,
+            -0.011684910394251347,
+            0.018053937703371048,
+            -0.004585127346217632,
+            0.02293996512889862,
+            0.013225225731730461,
+            0.0018376329680904746,
+            -0.0005776185425929725,
+            -0.008038301020860672,
+            0.0074579957872629166,
+            -0.02339847758412361,
+            0.002009575255215168,
+            -0.0090484619140625,
+            -0.010445492342114449,
+            -0.0055308095179498196,
+            -0.025791339576244354,
+            0.00451706675812602,
+            0.002878241939470172,
+            -0.011684910394251347,
+            0.0016979299252852798,
+            0.005645437631756067,
+            0.007637102622538805,
+            0.013425825163722038,
+            -0.01420672982931137,
+            -0.005749319680035114,
+            0.010724899359047413,
+            0.0009832947980612516,
+            0.005520063452422619,
+            -0.020776357501745224,
+            0.01676437072455883,
+            -0.0065947026014328,
+            -0.02689463645219803,
+            0.007063961587846279,
+            -0.00406930036842823,
+            0.01662108674645424,
+            0.00806695781648159,
+            0.002824509982019663,
+            -0.020532771944999695,
+            0.018713049590587616,
+            -0.004284228198230267,
+            -0.012494471855461597,
+            -0.019343504682183266,
+            -0.0030179449822753668,
+            -0.007973822765052319,
+            0.018713049590587616,
+            -0.007651431020349264,
+            -0.0336720272898674,
+            0.004660351667553186,
+            -0.022223537787795067,
+            -0.017480798065662384,
+            -0.022223537787795067,
+            0.010323700495064259,
+            -0.005627527367323637,
+            0.011004305444657803,
+            -0.009714738465845585,
+            0.007551131304353476,
+            -0.029402127489447594,
+            0.019372161477804184,
+            0.007020975928753614,
+            0.017896324396133423,
+            0.019687389954924583,
+            0.0273244921118021,
+            -0.011649088934063911,
+            0.0048609510995447636,
+            -0.022710707038640976,
+            -1.0858333553187549e-05,
+            -0.019372161477804184,
+            -0.007472324185073376,
+            0.003976164851337671,
+            0.017738711088895798,
+            0.00680246576666832,
+            0.003290186868980527,
+            -0.016076602041721344,
+            -0.022037267684936523,
+            -0.03215320408344269,
+            -0.027123892679810524,
+            -0.004230496007949114,
+            0.006139772012829781,
+            0.013741052709519863,
+            -0.002645403379574418,
+            -0.015904659405350685,
+            3.201529034413397e-05,
+            0.012759549543261528,
+            -0.01960141770541668,
+            -0.002000619890168309,
+            -0.03278365731239319,
+            -0.010072951205074787,
+            -0.029545413330197334,
+            0.018211551010608673,
+            0.021936967968940735,
+            0.004760651383548975,
+            -0.011534460820257664,
+            -0.012967312708497047,
+            -0.018383493646979332,
+            0.0016504667000845075,
+            -0.006361864041537046,
+            -0.013003134168684483,
+            -0.03490427881479263,
+            -0.012143422849476337,
+            0.011935658752918243,
+            0.010087279602885246,
+            -0.006394103169441223,
+            -0.0018331552855670452,
+            0.011462817899882793,
+            -0.005681259091943502,
+            -0.005237075034528971,
+            -0.0036967587657272816,
+            -0.02146412618458271,
+            -0.015302862040698528,
+            0.006644852459430695,
+            -0.006390520837157965,
+            0.01257327850908041,
+            -0.013898666948080063,
+            0.0016003168420866132,
+            -0.008675919845700264,
+            0.006755898240953684,
+            0.00275824056006968,
+            -0.010072951205074787,
+            -0.010617434978485107,
+            0.016707057133316994,
+            0.010825199075043201,
+            -0.016878999769687653,
+            -0.003650191007182002,
+            0.01023772917687893,
+            0.00939951092004776,
+            -0.009700410068035126,
+            -0.028313159942626953,
+            -0.013432989828288555,
+            0.014242551289498806,
+            -0.01103296224027872,
+            0.01887066476047039,
+            -0.012093272991478443,
+            -0.00831054337322712,
+            0.020432472229003906,
+            0.008246064186096191,
+            -0.003408397315070033,
+            0.0007831432740204036,
+            -0.004123032093048096,
+            -0.003528398694470525,
+            -0.006956497672945261,
+            0.015016291290521622,
+            -0.015102262608706951,
+            0.008940997533500195,
+            -0.007096200715750456,
+            0.014314194209873676,
+            -0.00787352304905653,
+            0.0034782488364726305,
+            0.023341162130236626,
+            -0.006723659113049507,
+            -0.013411496765911579,
+            0.008389350026845932,
+            0.01448613591492176,
+            0.02122054249048233,
+            0.00913443323224783,
+            -0.003068094840273261,
+            -0.01296014804393053,
+            -0.014271208085119724,
+            -0.00896249059587717,
+            0.025203870609402657,
+            -0.01518823392689228,
+            -0.006652016658335924,
+            0.009456824511289597,
+            -0.0005064237047918141,
+            -0.011971480213105679,
+            -0.002588089322671294,
+            -0.025719698518514633,
+            -0.005351703148335218,
+            0.02286832220852375,
+            -0.03258306160569191,
+            -0.005910515319555998,
+            0.017065269872546196,
+            -0.01402762345969677,
+            -0.017423482611775398,
+            0.010982812382280827,
+            -0.015159576199948788,
+            0.009220404550433159,
+            -0.008038301020860672,
+            0.01430702954530716,
+            0.000650156696792692,
+            -0.02367071807384491,
+            0.005143939517438412,
+            0.021206213161349297,
+            -0.02437281608581543,
+            0.004577962681651115,
+            -0.0443897619843483,
+            0.002100919606164098,
+            0.003943925723433495,
+            0.014607928693294525,
+            -0.00925622507929802,
+            0.0002228756929980591,
+            -0.004044225439429283,
+            -0.012279544025659561,
+            0.017552439123392105,
+            -0.008948162198066711,
+            -0.02230951003730297,
+            0.002145696198567748,
+            0.005176178645342588,
+            0.007038886658847332,
+            -0.009743395261466503,
+            0.013239555060863495,
+            -0.0019146488048136234,
+            -0.010037129744887352,
+            0.016262872144579887,
+            -0.0007576206116937101,
+            -0.020604414865374565,
+            -0.02013157308101654,
+            -0.00244122208096087,
+            -0.024358488619327545,
+            0.012071779929101467,
+            -0.004760651383548975,
+            -0.0015743464464321733,
+            -0.005480659659951925,
+            0.016535114496946335,
+            0.00027179415337741375,
+            0.017867667600512505,
+            -0.0027295835316181183,
+            0.012200736440718174,
+            0.00022567423002328724,
+            0.00352481659501791,
+            -0.02325519174337387,
+            0.0005046326550655067,
+            -0.003368993755429983,
+            -0.012472978793084621,
+            -0.016334515064954758,
+            -0.01957276090979576,
+            -0.018010953441262245,
+            -0.010058622807264328,
+            0.016606757417321205,
+            0.025619398802518845,
+            0.006863362155854702,
+            -0.020432472229003906,
+            0.013755381107330322,
+            -0.004667516332119703,
+            0.00959294568747282,
+            0.0013459855690598488,
+            -0.009478317573666573,
+            0.029688699170947075,
+            0.013110597617924213,
+            0.018741706386208534,
+            -0.003703922964632511,
+            -0.028169874101877213,
+            -0.01819722354412079,
+            -0.0177816953510046,
+            0.010810869745910168,
+            -0.010825199075043201,
+            0.010338028892874718,
+            -0.011727895587682724,
+            -0.00787352304905653,
+            -0.001703303074464202,
+            -0.021234869956970215,
+            0.032181862741708755,
+            -0.024988943710923195,
+            0.0046102022752165794,
+            0.017409155145287514,
+            -0.0015886749606579542,
+            -0.010323700495064259,
+            0.023556090891361237,
+            0.002005993155762553,
+            -0.022925635799765587,
+            -0.00966458860784769,
+            0.010037129744887352,
+            -0.012988805770874023,
+            0.005176178645342588,
+            -1.1285110304015689e-05,
+            0.007171425502747297,
+            -0.028327489271759987,
+            0.012387007474899292,
+            0.018053937703371048,
+            0.013490303419530392,
+            0.02276802249252796,
+            0.0023122653365135193,
+            0.013003134168684483,
+            -0.012286707758903503,
+            0.006386938970535994,
+            0.01988798938691616,
+            0.0079451659694314,
+            -0.028040917590260506,
+            0.020676057785749435,
+            -0.021693382412195206,
+            0.018598422408103943,
+            -0.0042698998004198074,
+            -0.0031003339681774378,
+            0.03387262672185898,
+            -0.010667584836483002,
+            -0.004456170368939638,
+            -0.01069624163210392,
+            -0.004989908076822758,
+            -0.003675265936180949,
+            -0.011634760536253452,
+            0.02633582428097725,
+            0.024716701358556747,
+            -0.021019943058490753,
+            0.01211476605385542,
+            0.002511073602363467,
+            0.012243722565472126,
+            -0.00697082607075572,
+            0.023441461846232414,
+            0.014507628977298737,
+            -0.007314710877835751,
+            0.009019805118441582,
+            0.004051389638334513,
+            -0.003911686595529318,
+            0.0006891123484820127,
+            -0.013447318226099014,
+            -0.007386353332549334,
+            -0.006053800694644451,
+            0.011555952951312065,
+            0.008554127998650074,
+            -0.008016807958483696,
+            0.003718251595273614,
+            0.0003978403692599386,
+            0.0012931491946801543,
+            -0.01002996601164341,
+            0.0074579957872629166,
+            0.0035122791305184364,
+            0.00869741290807724,
+            0.011212068609893322,
+            0.0033672028221189976,
+            -0.009213239885866642,
+            -0.00451706675812602,
+            0.01824020966887474,
+            0.016191231086850166,
+            0.00976488832384348,
+            0.014958977699279785,
+            -0.01652078703045845,
+            0.0253901407122612,
+            0.024530429393053055,
+            0.0007558295619674027,
+            -0.01176371704787016,
+            0.0053552850149571896,
+            0.0028137636836618185,
+            -0.008632934652268887,
+            0.011054455302655697,
+            -0.006017979234457016,
+            0.015216890722513199,
+            0.003354665357619524,
+            -0.00806695781648159,
+            -0.006161264609545469,
+            -0.027095235884189606,
+            -0.037540730088949203,
+            0.002767195925116539,
+            -0.004599455744028091,
+            -0.006723659113049507,
+            -0.0026579408440738916,
+            -0.009986979886889458,
+            0.0014865841949358582,
+            0.006261564325541258,
+            -0.006010815035551786,
+            0.01202879473567009,
+            -0.023226534947752953,
+            -0.005423345603048801,
+            0.000444184202933684,
+            -0.010459821671247482,
+            -2.4809052774799056e-05,
+            -0.007046050857752562,
+            -0.010051458142697811,
+            0.005906933452934027,
+            -0.022782349959015846,
+            -0.000383287959266454,
+            -0.01763841137290001,
+            0.01889932155609131,
+            -0.008632934652268887,
+            0.0045206486247479916,
+            0.0016504667000845075,
+            -0.013074777089059353,
+            0.01504494808614254,
+            0.011491474695503712,
+            -0.010509971529245377,
+            0.002151069464161992,
+            0.008876519277691841,
+            0.0025558501947671175,
+            0.0011767299147322774,
+            -0.021793682128190994,
+            -0.01139833964407444,
+            0.02289697900414467,
+            0.002111665904521942,
+            -0.017423482611775398,
+            -9.363912977278233e-05,
+            0.009686080738902092,
+            0.003426307812333107,
+            0.016019288450479507,
+            0.00607529329136014,
+            0.013741052709519863,
+            0.014256879687309265,
+            -0.0008301587658934295,
+            0.020160231739282608,
+            -0.0009206075337715447,
+            -0.002340922364965081,
+            0.0024251025170087814,
+            0.006770226638764143,
+            0.0007200082764029503,
+            0.004807219374924898,
+            -0.010015636682510376,
+            -0.010739227756857872,
+            -0.0019021113403141499,
+            0.0014695690479129553,
+            0.00425198907032609,
+            0.016076602041721344,
+            0.0008444872801192105,
+            -0.01383418869227171,
+            0.014077773317694664,
+            0.04267033934593201,
+            0.0034191436134278774,
+            0.021062927320599556,
+            0.0010970274452120066,
+            -0.0005614989786408842,
+            0.009077118709683418,
+            0.00858994945883751,
+            -0.026636723428964615,
+            0.021621739491820335,
+            -0.006809630431234837,
+            0.00551648112013936,
+            -0.030175868421792984,
+            0.006286639254540205,
+            -0.017208555713295937,
+            0.002500327071174979,
+            -0.01166341733187437,
+            0.006394103169441223,
+            0.012315364554524422,
+            -0.006616195198148489,
+            0.017523782327771187,
+            -0.0008525470620952547,
+            0.016047945246100426,
+            0.0012421037536114454,
+            -0.0021188303362578154,
+            -0.00931353960186243,
+            0.012480142526328564,
+            0.018913649022579193,
+            0.018440809100866318,
+            0.0005261254264041781,
+            0.01222222950309515,
+            -0.005262149963527918,
+            -0.010015636682510376,
+            0.009284882806241512,
+            -0.004047807771712542,
+            -0.0005579168209806085,
+            0.0037934763822704554,
+            -0.011197740212082863,
+            -0.0033564562909305096,
+            0.01676437072455883,
+            0.019515447318553925,
+            0.0028657044749706984,
+            -0.011577446013689041,
+            -0.02037515863776207,
+            0.0006282161339186132,
+            0.014350014738738537,
+            -0.003111080499365926,
+            0.013683739118278027,
+            -0.0036716838367283344,
+            -0.0029695862904191017,
+            -0.008647263050079346,
+            0.0004466469108592719,
+            0.020074259489774704,
+            0.0051546860486269,
+            -0.014034787192940712,
+            -0.004534977488219738,
+            0.021936967968940735,
+            0.010610271245241165,
+            0.008145765401422977,
+            0.006576791871339083,
+            -0.007529638707637787,
+            0.01659242808818817,
+            -0.011169083416461945,
+            -0.00037948196404613554,
+            -0.012666413560509682,
+            -0.003236454911530018,
+            -0.005856783594936132,
+            -0.008496813476085663,
+            -0.007350531872361898,
+            -0.008568456396460533,
+            0.005487823858857155,
+            0.011161918751895428,
+            -0.01596197299659252,
+            -0.012236557900905609,
+            0.012250886298716068,
+            -0.002503909170627594,
+            -0.0004728412313852459,
+            -0.01620555855333805,
+            0.014156579971313477,
+            0.008059794083237648,
+            0.0033851133193820715,
+            -0.009779216721653938,
+            0.01558943185955286,
+            -0.005817380268126726,
+            0.0002970929490402341,
+            -0.0060323080979287624,
+            -0.0004099300713278353,
+            0.0014982260763645172,
+            -0.004090792965143919,
+            -0.005548720248043537,
+            -0.0018967381911352277,
+            0.007701580878347158,
+            -0.01952977664768696,
+            0.009686080738902092,
+            -0.010832362808287144,
+            -0.026565080508589745,
+            -0.013827024027705193,
+            0.022223537787795067,
+            0.025805668905377388,
+            0.027410462498664856,
+            0.007701580878347158,
+            0.002815554616972804,
+            0.00562036270275712,
+            0.0034155615139752626,
+            -0.0025845072232186794,
+            0.005376778077334166,
+            0.009026968851685524,
+            -0.0038078047800809145,
+            0.03057706728577614,
+            0.005641855765134096,
+            -0.004545723553746939,
+            0.014328522607684135,
+            0.018369166180491447,
+            0.01831185072660446,
+            -0.007687252014875412,
+            0.020174559205770493,
+            -0.016148244962096214,
+            -0.0008395618642680347,
+            -0.002708090702071786,
+            0.03507622331380844,
+            -0.029373470693826675,
+            -0.0012644920498132706,
+            -0.0027510763611644506,
+            0.02041814476251602,
+            0.006028725765645504,
+            -0.02304026484489441,
+            0.010531463660299778,
+            0.028112560510635376,
+            0.0025791339576244354,
+            0.012895669788122177,
+            -0.02051844447851181,
+            -0.009700410068035126,
+            0.00017686770297586918,
+            0.005308717489242554,
+            0.0029964521527290344,
+            0.005505734588950872,
+            -0.022323837503790855,
+            0.01623421534895897,
+            -0.01457210723310709,
+            0.019873660057783127,
+            0.0037218336947262287,
+            0.026292838156223297,
+            0.005366031546145678,
+            0.006064547225832939,
+            0.006960079539567232,
+            -0.013877173885703087,
+            0.014987634494900703,
+            0.0007853821152821183,
+            -0.013862845487892628,
+            0.002903316868469119,
+            -0.016721386462450027,
+            0.01167774572968483,
+            -0.0168360136449337,
+            0.015947645530104637,
+            -0.004359452985227108,
+            0.007565459702163935,
+            -0.0036089965142309666,
+            -0.0009138910681940615,
+            -0.013755381107330322,
+            0.012666413560509682,
+            -0.030032582581043243,
+            0.010287879034876823,
+            -0.010954155586659908,
+            -0.031092893332242966,
+            0.0037504907231777906,
+            0.002140322932973504,
+            0.01131236832588911,
+            -0.013633589260280132,
+            -0.017982294782996178,
+            0.01630585826933384,
+            -0.0019880824256688356,
+            0.004481245297938585,
+            -0.020819343626499176,
+            -0.007085454184561968,
+            0.003800640581175685,
+            0.024945957586169243,
+            -0.0017928563756868243,
+            -0.014536285772919655,
+            0.007325456943362951,
+            0.014371507801115513,
+            -0.009614438749849796,
+            0.020461129024624825,
+            0.005634691566228867,
+            -0.007701580878347158,
+            0.007253814488649368,
+            -0.0025468948297202587,
+            0.022954292595386505,
+            -0.00470691965892911,
+            -0.008496813476085663,
+            0.013905830681324005,
+            0.008375021629035473,
+            -0.02328384853899479,
+            0.005186925176531076,
+            -0.008726069703698158,
+            -0.0013871800620108843,
+            -0.0007809044909663498,
+            -0.001941514783538878,
+            0.010037129744887352,
+            0.028485102578997612,
+            -0.00834636390209198,
+            0.008948162198066711,
+            0.003689594566822052,
+            -0.00317376758903265,
+            5.6950279940792825e-06,
+            0.0006322460249066353,
+            -0.019830675795674324,
+            -0.00903413351625204,
+            -0.004749905318021774,
+            -0.010395343415439129,
+            0.0007531429291702807,
+            0.016463471576571465,
+            0.027582405135035515,
+            -0.0073720249347388744,
+            0.007167843170464039,
+            -0.0007070230203680694,
+            0.020461129024624825,
+            0.031752005219459534,
+            -0.014328522607684135,
+            0.004588709212839603,
+            -0.009370853193104267,
+            -0.012867012992501259,
+            -0.020561428740620613,
+            0.004259153269231319,
+            -0.0014668825315311551,
+            -0.02304026484489441,
+            -0.0009725484414957464,
+            0.00046657249913550913,
+            -0.005176178645342588,
+            -0.009549960494041443,
+            0.00488244416192174,
+            -0.011885508894920349,
+            0.0045278132893145084,
+            -0.002625701716169715,
+            0.004889608360826969,
+            0.010911169461905956,
+            0.034875623881816864,
+            -0.005946336779743433,
+            -0.002511073602363467,
+            -0.011498639360070229,
+            0.0033224262297153473,
+            0.020160231739282608,
+            0.000652843271382153,
+            0.008195914328098297,
+            -0.0066197775304317474,
+            0.00834636390209198,
+            -0.006841869559139013,
+            -0.012401335872709751,
+            0.01520256232470274,
+            0.008948162198066711,
+            0.003125408897176385,
+            -0.01076788455247879,
+            -0.0501498281955719,
+            -0.0015143457567319274,
+            0.032525744289159775,
+            0.010531463660299778,
+            -0.005720662418752909,
+            -0.012652085162699223,
+            -0.018541108816862106,
+            -0.008754727430641651,
+            -0.0020704714115709066,
+            0.0013638961827382445,
+            -0.023455791175365448,
+            0.0024645058438181877,
+            -0.005305135622620583,
+            -0.003789894049987197,
+            -0.0037289978936314583,
+            0.011441324837505817,
+            0.04390259087085724,
+            -0.015431818552315235,
+            0.00813860073685646,
+            -0.009657423943281174,
+            -0.01527420524507761,
+            -0.0006761271506547928,
+            0.018598422408103943,
+            -0.010216236114501953,
+            -0.012494471855461597,
+            -0.016964970156550407,
+            -0.010674749501049519,
+            -0.006472909823060036,
+            0.009979816153645515,
+            0.009628767147660255,
+            -0.011649088934063911,
+            -0.007751730736345053,
+            -0.004491991829127073,
+            -0.007053215056657791,
+            0.013511796481907368,
+            0.006193503737449646,
+            0.021091584116220474,
+            -0.0161625724285841,
+            -0.01185685209929943,
+            0.012021630071103573,
+            0.008754727430641651,
+            0.0030698860064148903,
+            -0.015360175631940365,
+            -0.005541556049138308,
+            -0.007465159986168146,
+            -0.0023122653365135193,
+            0.025662383064627647,
+            -0.004864533431828022,
+            0.016463471576571465,
+            -0.0003806013846769929,
+            -0.005584541708230972,
+            0.004574380815029144,
+            0.005545137915760279,
+            0.025232527405023575,
+            0.011806702241301537,
+            -0.0026042088866233826,
+            -0.01338283997029066,
+            0.0023283849004656076,
+            -0.010624599643051624,
+            -0.0022191298194229603,
+            -0.009356524795293808,
+            0.006086039822548628,
+            0.01574704609811306,
+            0.0034334722440689802,
+            0.021979954093694687,
+            -0.00030358557705767453,
+            -0.020848000422120094,
+            -0.004330795723944902,
+            0.00831054337322712,
+            0.0021779353264719248,
+            0.017437811940908432,
+            0.009707573801279068,
+            0.013432989828288555,
+            0.0058675301261246204,
+            0.012472978793084621,
+            -0.011298039928078651,
+            -0.004316467326134443,
+            -0.006086039822548628,
+            -0.002656149910762906,
+            0.014163744635879993,
+            6.481417221948504e-05,
+            0.03819984197616577,
+            -0.015073605813086033,
+            -0.019687389954924583,
+            0.002557641128078103,
+            -0.011520132422447205,
+            -0.012422828935086727,
+            0.00462811253964901,
+            -0.012809699401259422,
+            -0.0017874831100925803,
+            0.013167912140488625,
+            -0.020676057785749435,
+            -0.022452794015407562,
+            -0.013175075873732567,
+            0.011262218467891216,
+            -0.005183342844247818,
+            -0.005577377509325743,
+            -0.00020630833751056343,
+            0.010946990922093391,
+            0.0024698791094124317,
+            -0.007303964346647263,
+            0.018397822976112366,
+            -0.007680087815970182,
+            -0.007809044793248177,
+            0.013705231249332428,
+            -0.003943925723433495,
+            0.007114111445844173,
+            0.002369579393416643,
+            0.015933316200971603,
+            0.022925635799765587,
+            -0.004782144445925951,
+            0.0034746667370200157,
+            0.007880687713623047,
+            0.014701063744723797,
+            0.0020597251132130623,
+            0.008453828282654285,
+            -0.023785347118973732,
+            0.01360493153333664,
+            0.007164261303842068,
+            -0.007930837571620941,
+            0.013755381107330322,
+            0.01518823392689228,
+            0.0019182309042662382,
+            0.005459167063236237,
+            -0.005050804000347853,
+            0.01964440383017063,
+            -0.0009895635303109884,
+            0.002396445255726576,
+            -0.01967306062579155,
+            0.012738056480884552,
+            -0.004391692113131285,
+            0.011190575547516346,
+            0.009886680170893669,
+            0.01627720147371292,
+            0.029086900874972343,
+            -0.0010692659998312593,
+            0.00294630229473114,
+            -0.008955326862633228,
+            0.011498639360070229,
+            0.0038615367375314236,
+            -0.009901008568704128,
+            -0.0024286846164613962,
+            0.00641917809844017,
+            0.01792498119175434,
+            0.022753693163394928,
+            0.019314847886562347,
+            0.004255570936948061,
+            -0.0029588397592306137,
+            0.026923293247818947,
+            0.0013603140832856297,
+            -0.00029351082048378885,
+            -0.0027546584606170654,
+            -0.009177418425679207,
+            -0.002872868673875928,
+            -0.014744049869477749,
+            0.011985808610916138,
+            -0.0020848000422120094,
+            0.019730376079678535,
+            -0.000688216823618859,
+            0.02342713437974453,
+            0.010244893841445446,
+            -0.015431818552315235,
+            -0.0068669444881379604,
+            0.0017149450723081827,
+            0.002801226219162345,
+            -0.0171512421220541,
+            -0.0021725622937083244,
+            -0.003159439191222191,
+            0.0017015120247378945,
+            0.010825199075043201,
+            8.04300289019011e-05,
+            9.75011134869419e-05,
+            0.001118520274758339,
+            -0.0015600178157910705,
+            -0.002203010255470872,
+            0.0011821030639111996,
+            -0.0007885164814069867,
+            -0.005598870106041431,
+            0.008790548890829086,
+            0.013139255344867706,
+            0.01436434406787157,
+            0.015059277415275574,
+            0.02391430363059044,
+            0.01644914411008358,
+            0.013905830681324005,
+            -0.0027600317262113094,
+            -0.0025074915029108524,
+            0.024774014949798584,
+            0.003986911382526159,
+            -0.004710501525551081,
+            0.004581545013934374,
+            0.0026668962091207504,
+            0.0019343504682183266,
+            -0.006086039822548628,
+            0.0055308095179498196,
+            -0.016090931370854378,
+            5.056960799265653e-05,
+            -0.0036143697798252106,
+            -0.002208383521065116,
+            -0.010617434978485107,
+            0.0009152343845926225,
+            -0.017208555713295937,
+            0.009127268567681313,
+            0.01571838930249214,
+            0.003442427609115839,
+            0.01058877818286419,
+            0.007708745077252388,
+            -0.030949607491493225,
+            -0.009299211204051971,
+            0.011362518183887005,
+            0.003689594566822052,
+            0.011226397007703781,
+            0.012358350679278374,
+            0.008181585930287838,
+            -0.017910653725266457,
+            0.008246064186096191,
+            -0.002319429535418749,
+            0.011806702241301537,
+            0.010424000211060047,
+            -0.008948162198066711
+          ],
+          "title": "Designing Models with Data, Model, and Expert-Parallelism"
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "9204ecdb-832f-4b40-b0fe-5cc34ec75d3b",
+        "properties": {
+          "page_content": "6 Related Work 24 ",
+          "keyphrases": [
+            "Related Work"
+          ],
+          "embedding": [
+            0.002109470311552286,
+            0.024481380358338356,
+            -0.01675625890493393,
+            0.01014935877174139,
+            0.03144679218530655,
+            0.0017690955428406596,
+            0.0074690403416752815,
+            0.09587687253952026,
+            0.01939389668405056,
+            0.014093012548983097,
+            0.001160902320407331,
+            -0.013068686239421368,
+            -0.015330738388001919,
+            -0.026103228330612183,
+            0.03235161304473877,
+            0.036431845277547836,
+            -0.005620986223220825,
+            -0.00097150873625651,
+            -0.033068642020225525,
+            0.002234310144558549,
+            0.0004380058671813458,
+            0.024430163204669952,
+            -0.015834365040063858,
+            0.04742627218365669,
+            0.02104988880455494,
+            0.008480561897158623,
+            0.051182132214307785,
+            0.01811349019408226,
+            -0.03865121677517891,
+            0.0019728937186300755,
+            -0.008903096430003643,
+            0.010678594000637531,
+            8.36265753605403e-05,
+            -0.0034208204597234726,
+            -0.028698187321424484,
+            -0.018420787528157234,
+            0.014630783349275589,
+            0.01811349019408226,
+            -0.0005980567075312138,
+            0.012718708254396915,
+            -0.036909863352775574,
+            -0.027776293456554413,
+            -0.030541973188519478,
+            0.0029363997746258974,
+            0.03361494839191437,
+            -0.029739584773778915,
+            -0.01758425496518612,
+            -0.044626448303461075,
+            0.005258204415440559,
+            0.03020053170621395,
+            0.029500575736165047,
+            -0.023098541423678398,
+            0.01559535600244999,
+            -0.0012782729463651776,
+            -0.01341866422444582,
+            0.003841220634058118,
+            -0.0015738335205242038,
+            0.07300026714801788,
+            0.023798497393727303,
+            0.019069526344537735,
+            0.001734951394610107,
+            -0.016662361100316048,
+            -0.0010467326501384377,
+            -0.03286377713084221,
+            0.009978638030588627,
+            -0.01825006678700447,
+            0.018045201897621155,
+            -0.005023463163524866,
+            0.020264573395252228,
+            -0.008489097468554974,
+            -0.0041101062670350075,
+            -0.007831822149455547,
+            -0.026905616745352745,
+            0.05551844462752342,
+            -0.0061459532007575035,
+            0.00418479647487402,
+            0.042850952595472336,
+            0.027008049190044403,
+            -0.015689251944422722,
+            0.04995294287800789,
+            -0.005655130371451378,
+            -0.024703318253159523,
+            -0.012863821350038052,
+            -0.01522830594331026,
+            0.019701194018125534,
+            -0.005778903141617775,
+            -0.056850068271160126,
+            -0.13206970691680908,
+            0.004724701400846243,
+            -0.0011918453965336084,
+            0.011318797245621681,
+            0.014229589141905308,
+            -0.002349546644836664,
+            -0.03841220587491989,
+            -0.051182132214307785,
+            0.04595807194709778,
+            -0.008531778119504452,
+            -0.021391332149505615,
+            0.009551835246384144,
+            -0.01522830594331026,
+            0.008096439763903618,
+            0.016414817422628403,
+            0.0179598405957222,
+            -0.03494657203555107,
+            0.02656417526304722,
+            -0.03824148699641228,
+            -0.005292348563671112,
+            -0.00846775807440281,
+            0.026291022077202797,
+            0.03970968723297119,
+            -0.02081087976694107,
+            -0.025249624624848366,
+            -0.033444229513406754,
+            0.004285095259547234,
+            -0.0017040082020685077,
+            -0.029995666816830635,
+            0.011079788208007812,
+            -0.046162936836481094,
+            -0.021203538402915,
+            0.012360194697976112,
+            0.020486511290073395,
+            -0.0011032839538529515,
+            0.004673485178500414,
+            -0.02465210109949112,
+            -0.02274002693593502,
+            0.021664485335350037,
+            0.003265037667006254,
+            -0.01618434302508831,
+            -0.01535634696483612,
+            0.0282543133944273,
+            0.03605625778436661,
+            -0.00591121194884181,
+            0.0010622042464092374,
+            -0.03387103229761124,
+            0.01354670524597168,
+            -0.018574437126517296,
+            0.011933392845094204,
+            0.03660256415605545,
+            -0.022022999823093414,
+            -0.004733237437903881,
+            0.005288080777972937,
+            -0.019359752535820007,
+            -0.0030473684892058372,
+            0.022415656596422195,
+            -0.007490380667150021,
+            0.014912472106516361,
+            -0.03055904433131218,
+            0.012991862371563911,
+            -0.017789119854569435,
+            0.04548005387187004,
+            -0.03424661606550217,
+            0.017319638282060623,
+            -0.018284210935235023,
+            0.032027244567871094,
+            -0.019735338166356087,
+            -0.0157319325953722,
+            0.009287218563258648,
+            0.0017146782483905554,
+            0.016355063766241074,
+            -0.02403750643134117,
+            0.04097302258014679,
+            0.023320477455854416,
+            0.024447236210107803,
+            -0.01644042506814003,
+            0.021630341187119484,
+            -0.03378567099571228,
+            0.015211233869194984,
+            -0.014681999571621418,
+            -0.017908625304698944,
+            0.026512959972023964,
+            0.02734949253499508,
+            0.0033183880150318146,
+            0.018420787528157234,
+            0.0003475771227385849,
+            -0.026683680713176727,
+            -0.007976935245096684,
+            -0.007575741037726402,
+            0.03168580308556557,
+            -0.013956435024738312,
+            0.003331192070618272,
+            -0.021288899704813957,
+            0.03762689232826233,
+            -0.0013903084909543395,
+            -0.00194301747251302,
+            0.011071251705288887,
+            -0.013393056578934193,
+            0.029381070286035538,
+            -0.030132243409752846,
+            0.041314464062452316,
+            0.024378947913646698,
+            -0.0024221031926572323,
+            -0.019052455201745033,
+            0.01622702367603779,
+            0.012752852402627468,
+            0.005198452156037092,
+            -0.057737816125154495,
+            0.011199292726814747,
+            -0.006252653896808624,
+            -0.03568067401647568,
+            -0.021220611408352852,
+            -0.00606912886723876,
+            -0.022057143971323967,
+            0.007443432230502367,
+            0.004575320519506931,
+            -0.025847148150205612,
+            -0.009201857261359692,
+            -0.0024583812337368727,
+            0.002289794385433197,
+            -0.008659819141030312,
+            -0.03516850993037224,
+            0.00754159688949585,
+            0.0003875898546539247,
+            0.0006898192223161459,
+            -0.016406280919909477,
+            -0.0011064850259572268,
+            -0.023559486493468285,
+            -0.0224839448928833,
+            0.06524953991174698,
+            -0.0035787373781204224,
+            -0.0295347198843956,
+            0.03892436996102333,
+            -0.0030708424746990204,
+            0.007925719022750854,
+            0.008809199556708336,
+            -0.02647881582379341,
+            -0.023439982905983925,
+            -0.0022663201671093702,
+            0.05275276303291321,
+            0.0034955109003931284,
+            -0.01458810269832611,
+            0.0006594095611944795,
+            0.008292769081890583,
+            -0.004041817970573902,
+            -0.052616186439991,
+            -0.0314980112016201,
+            -0.01964997872710228,
+            -0.00800254289060831,
+            -0.030354179441928864,
+            -0.012300441972911358,
+            0.02801530249416828,
+            0.030763909220695496,
+            -0.0016613280167803168,
+            -0.042953383177518845,
+            0.017123308032751083,
+            -0.02466917410492897,
+            -0.020349934697151184,
+            -0.03163458779454231,
+            0.05312835052609444,
+            0.019871914759278297,
+            -0.02620566077530384,
+            0.0058045112527906895,
+            0.012616275809705257,
+            -0.009048208594322205,
+            0.02079380862414837,
+            0.019342679530382156,
+            0.09280389547348022,
+            0.026666607707738876,
+            -0.0018149768002331257,
+            -0.0026653804816305637,
+            -0.015211233869194984,
+            -0.015194161795079708,
+            -0.01675625890493393,
+            -0.04561663046479225,
+            -0.021783988922834396,
+            0.005377708934247494,
+            -0.0013038810575380921,
+            0.039368245750665665,
+            0.0044302078895270824,
+            0.022415656596422195,
+            -0.01001278217881918,
+            0.024686245247721672,
+            -0.013145511038601398,
+            -0.002503195544704795,
+            -0.02622273378074169,
+            -0.009603051468729973,
+            -0.030115170404314995,
+            -0.003077244618907571,
+            -0.015142945572733879,
+            0.016372136771678925,
+            -0.004588124807924032,
+            -0.009893277660012245,
+            0.04466059431433678,
+            0.008937240578234196,
+            -0.018984166905283928,
+            -0.021527908742427826,
+            -0.03557823970913887,
+            0.009449402801692486,
+            -0.0026995246298611164,
+            -0.0031199248041957617,
+            -0.019564617425203323,
+            0.01213825773447752,
+            -0.027912870049476624,
+            -0.040529146790504456,
+            0.044899601489305496,
+            0.006828837096691132,
+            -0.025198407471179962,
+            -0.01381132286041975,
+            0.006905661430209875,
+            -0.0014351228019222617,
+            -0.026410527527332306,
+            -0.001253731781616807,
+            0.012838213704526424,
+            -0.00121211854275316,
+            0.007302587386220694,
+            -0.019718267023563385,
+            0.008331180550158024,
+            0.02106696180999279,
+            0.03177116438746452,
+            -0.029978593811392784,
+            -0.026871472597122192,
+            0.01149805448949337,
+            -0.04868960753083229,
+            -0.0007074248278513551,
+            -0.04063158109784126,
+            0.012069969438016415,
+            0.024430163204669952,
+            0.005902675911784172,
+            0.021732773631811142,
+            -0.02874940261244774,
+            -0.04612879455089569,
+            0.0013113500317558646,
+            0.01078102644532919,
+            -0.04353383556008339,
+            0.024481380358338356,
+            -0.014724679291248322,
+            0.022296153008937836,
+            -0.003619283437728882,
+            0.04059743508696556,
+            0.01580875739455223,
+            -0.010763954371213913,
+            0.03006395511329174,
+            -0.00937257893383503,
+            -6.443714482884388e-06,
+            0.018967093899846077,
+            -0.011941928416490555,
+            -0.03156629949808121,
+            -0.03557823970913887,
+            0.015475851483643055,
+            0.007567205000668764,
+            -0.017823264002799988,
+            0.009662804193794727,
+            0.01554413978010416,
+            0.007511720526963472,
+            0.02451552450656891,
+            0.019086599349975586,
+            -0.05367465689778328,
+            0.035373374819755554,
+            -0.04029013589024544,
+            -0.012923574075102806,
+            0.007780605927109718,
+            0.0009704417316243052,
+            -0.03554409742355347,
+            -0.0024370411410927773,
+            0.012104113586246967,
+            -0.014238124713301659,
+            -0.047221407294273376,
+            -0.017703760415315628,
+            -0.029244493693113327,
+            0.020076779648661613,
+            -0.021801061928272247,
+            0.00022700548288412392,
+            -0.04022184759378433,
+            -0.009125033393502235,
+            -0.016022158786654472,
+            -0.05278690904378891,
+            -0.036670852452516556,
+            0.024464307352900505,
+            -0.005795975215733051,
+            -0.02145962044596672,
+            -0.0009016198455356061,
+            -0.059410881251096725,
+            0.007989739067852497,
+            0.0016175807686522603,
+            -0.006777620408684015,
+            -0.025095975026488304,
+            -0.011148076504468918,
+            0.014571030624210835,
+            0.017635472118854523,
+            -0.0397779755294323,
+            0.005680738482624292,
+            -0.01373449806123972,
+            0.01014935877174139,
+            0.005706346593797207,
+            -0.017635472118854523,
+            -0.024088721722364426,
+            0.0036897058598697186,
+            -0.0037665304262191057,
+            0.01168584730476141,
+            0.0030644405633211136,
+            -0.010678594000637531,
+            0.02480575069785118,
+            -0.04520690068602562,
+            0.003469902789220214,
+            -0.03632941097021103,
+            0.039880406111478806,
+            -0.006495931185781956,
+            0.02992737665772438,
+            0.015100265853106976,
+            -0.0030793785117566586,
+            -0.005181380081921816,
+            -0.0501236617565155,
+            0.005595378112047911,
+            0.005138699896633625,
+            0.006700796075165272,
+            0.04435329511761665,
+            -0.003719581989571452,
+            -0.017925696447491646,
+            0.01232605054974556,
+            0.016628216952085495,
+            -0.011643166653811932,
+            0.03168580308556557,
+            0.015023441053926945,
+            -0.000559644540771842,
+            -0.011216364800930023,
+            0.03574896231293678,
+            -0.002466917270794511,
+            0.009150641039013863,
+            0.061220522969961166,
+            -0.023542415350675583,
+            -0.010311543941497803,
+            -0.004592392593622208,
+            -0.008732374757528305,
+            0.07368315011262894,
+            -0.009611587971448898,
+            -0.008472025394439697,
+            0.0179598405957222,
+            -0.015663644298911095,
+            -0.011916320770978928,
+            0.01600508578121662,
+            -0.02569349855184555,
+            -0.02169862948358059,
+            0.012505307793617249,
+            0.026273949071764946,
+            -0.021101105958223343,
+            0.028032375499606133,
+            -0.015689251944422722,
+            -0.011139540001749992,
+            0.01201021671295166,
+            -0.016329456120729446,
+            -0.012300441972911358,
+            0.035407520830631256,
+            -0.004592392593622208,
+            0.03366616740822792,
+            0.03660256415605545,
+            0.006192901637405157,
+            -0.03131021559238434,
+            -0.04548005387187004,
+            -0.010670057497918606,
+            -0.03480999544262886,
+            0.017413534224033356,
+            -0.0003048969083465636,
+            0.013691818341612816,
+            -0.04469473659992218,
+            -0.00844641774892807,
+            0.05510871484875679,
+            -0.02209128811955452,
+            0.029090845957398415,
+            0.04343140125274658,
+            -0.03185652568936348,
+            -0.02412286587059498,
+            -0.021510835736989975,
+            0.08549703657627106,
+            0.040665723383426666,
+            -0.010055461898446083,
+            -0.0359879694879055,
+            0.01592826284468174,
+            -0.026666607707738876,
+            -0.025078903883695602,
+            2.39409419009462e-05,
+            0.04339725896716118,
+            0.00754159688949585,
+            -0.015527067705988884,
+            -0.017908625304698944,
+            0.01937682554125786,
+            -0.033853959292173386,
+            0.001105417963117361,
+            0.019991420209407806,
+            0.008087903261184692,
+            -0.04510446637868881,
+            0.012069969438016415,
+            0.017891552299261093,
+            0.02031579054892063,
+            -0.02504475973546505,
+            0.015322202816605568,
+            0.026512959972023964,
+            0.027451924979686737,
+            0.027947014197707176,
+            -0.005676470696926117,
+            0.015339274890720844,
+            -0.006965413689613342,
+            0.03865121677517891,
+            -0.0018619250040501356,
+            -0.007840358652174473,
+            0.01695258729159832,
+            -0.029483502730727196,
+            -0.04005112871527672,
+            -0.017080627381801605,
+            -0.07129305601119995,
+            0.015834365040063858,
+            0.027178769931197166,
+            -0.014272268861532211,
+            -0.00818180013448,
+            -0.008058027364313602,
+            0.0450020357966423,
+            0.010251791216433048,
+            -0.021869350224733353,
+            0.01799398474395275,
+            -0.018950022757053375,
+            -0.01213825773447752,
+            0.013008934445679188,
+            0.028220169246196747,
+            0.016090447083115578,
+            0.019479257985949516,
+            -0.0237472802400589,
+            0.003971395548433065,
+            0.006991021800786257,
+            0.012658956460654736,
+            -0.0007340999436564744,
+            -0.01065298542380333,
+            0.030115170404314995,
+            0.0077379257418215275,
+            0.0027976890560239553,
+            0.001323087140917778,
+            0.04855303093791008,
+            -0.03187359496951103,
+            0.008971384726464748,
+            -0.010584697127342224,
+            -0.02236444130539894,
+            0.05346979200839996,
+            -0.003363202093169093,
+            -0.012744316831231117,
+            0.031259000301361084,
+            0.027588501572608948,
+            -0.013316231779754162,
+            -0.0118821756914258,
+            -0.0011043510166928172,
+            -0.027008049190044403,
+            0.020725520327687263,
+            -0.02067430317401886,
+            -0.0068587129935622215,
+            0.01007253397256136,
+            0.016824547201395035,
+            0.011660238727927208,
+            -0.017754975706338882,
+            -0.011848031543195248,
+            0.05125042051076889,
+            0.041280318051576614,
+            -0.010883458890020847,
+            0.007140402682125568,
+            -0.0028446372598409653,
+            0.014161300845444202,
+            0.028442105278372765,
+            0.006534343119710684,
+            -0.003913776949048042,
+            -0.002244980074465275,
+            -0.010951747186481953,
+            -0.03236868605017662,
+            0.019957276061177254,
+            0.025864219292998314,
+            0.012223618105053902,
+            0.028100663796067238,
+            -0.03779761120676994,
+            -0.00943233072757721,
+            -0.0053819771856069565,
+            0.006555683445185423,
+            -0.04670924320816994,
+            0.014101548120379448,
+            0.021544979885220528,
+            -0.03520265221595764,
+            -0.021408403292298317,
+            -0.0025330716744065285,
+            0.011259044520556927,
+            -0.03096877597272396,
+            -0.035509951412677765,
+            -0.032163821160793304,
+            0.04742627218365669,
+            -0.0021820268593728542,
+            -0.0063508180901408195,
+            -0.04059743508696556,
+            -0.01738792657852173,
+            0.015646573156118393,
+            -0.02429358661174774,
+            -0.006760548334568739,
+            0.025744715705513954,
+            -0.006133148912340403,
+            0.006167293526232243,
+            -0.024447236210107803,
+            -0.015262450091540813,
+            -0.03622698038816452,
+            0.02878354676067829,
+            0.012846749275922775,
+            0.0025800198782235384,
+            -0.01966704986989498,
+            -0.024327730759978294,
+            -0.025539850816130638,
+            0.004728969652205706,
+            0.010891994461417198,
+            0.0009005528409034014,
+            0.0289030522108078,
+            -0.05428925156593323,
+            0.030729765072464943,
+            0.006235581822693348,
+            -0.026239804923534393,
+            0.006389230489730835,
+            0.010644449852406979,
+            -0.023030253127217293,
+            0.003160471096634865,
+            0.01677333004772663,
+            -0.009346970357000828,
+            0.011446838267147541,
+            0.02555692195892334,
+            -0.014093012548983097,
+            0.048484742641448975,
+            -0.004524104297161102,
+            0.009918885305523872,
+            -0.025915436446666718,
+            0.041690047830343246,
+            0.008587262593209743,
+            -0.0009405656019225717,
+            0.0015044781612232327,
+            0.03892436996102333,
+            -0.01886466145515442,
+            -0.0224839448928833,
+            -0.0112334368750453,
+            0.013828394934535027,
+            -0.033973462879657745,
+            -0.008886024355888367,
+            -0.007848894223570824,
+            -0.017439141869544983,
+            -0.011933392845094204,
+            0.005783171392977238,
+            0.010746882297098637,
+            0.005492945667356253,
+            0.010635913349688053,
+            0.018011057749390602,
+            0.008143387734889984,
+            0.038104910403490067,
+            -0.009346970357000828,
+            0.03687571734189987,
+            -0.012360194697976112,
+            -0.01600508578121662,
+            -0.016858691349625587,
+            0.019223175942897797,
+            0.041177887469530106,
+            -0.040255993604660034,
+            0.029568864032626152,
+            -0.014442989602684975,
+            0.0147673599421978,
+            0.005253936164081097,
+            0.01838664337992668,
+            0.011165148578584194,
+            -0.003881766926497221,
+            -0.0050405352376401424,
+            0.022893676534295082,
+            0.011020035482943058,
+            -0.00015231507131829858,
+            -0.0012836079113185406,
+            -0.02425944246351719,
+            -0.0474945604801178,
+            0.020127996802330017,
+            -0.012940646149218082,
+            -0.022808315232396126,
+            -0.005467337556183338,
+            0.010490800254046917,
+            0.005224060267210007,
+            0.029722511768341064,
+            0.015211233869194984,
+            0.005224060267210007,
+            -0.014442989602684975,
+            0.006577023304998875,
+            -0.004024745896458626,
+            -0.035885538905858994,
+            0.0050917514599859715,
+            0.020503582432866096,
+            -0.03131021559238434,
+            -0.018403714522719383,
+            -0.004159188363701105,
+            0.03337594121694565,
+            -0.0031113887671381235,
+            0.004270156845450401,
+            -0.016568465158343315,
+            -0.004618000704795122,
+            -0.021118177101016045,
+            0.017217205837368965,
+            0.002994018141180277,
+            0.01503197755664587,
+            -0.010542016476392746,
+            0.0003016958653461188,
+            0.006623971741646528,
+            -0.00018725951667875051,
+            -0.0041762604378163815,
+            0.005744758993387222,
+            -0.006700796075165272,
+            -0.0016666629817336798,
+            0.0013561643427237868,
+            0.010695666074752808,
+            -0.014272268861532211,
+            0.00950915552675724,
+            0.007405019830912352,
+            0.013512561097741127,
+            -0.02799823135137558,
+            -0.025761786848306656,
+            -0.055689163506031036,
+            0.01675625890493393,
+            -0.009355506859719753,
+            -0.011284653097391129,
+            -0.006585559342056513,
+            -0.006081932689994574,
+            -0.025130119174718857,
+            -0.014391773380339146,
+            -0.014622246846556664,
+            -0.011122467927634716,
+            0.010328616015613079,
+            -0.0007314324611797929,
+            -0.0015524934278801084,
+            0.01744767837226391,
+            0.002784885000437498,
+            -0.022569306194782257,
+            -0.01746474951505661,
+            0.013589384965598583,
+            -0.01876222901046276,
+            -0.0006722136167809367,
+            -0.034588057547807693,
+            -0.04923591390252113,
+            -0.019308535382151604,
+            0.006141684949398041,
+            0.026530031114816666,
+            -0.0002678184537217021,
+            -0.004404599778354168,
+            -0.003990601748228073,
+            0.005714882630854845,
+            -0.01118222065269947,
+            -0.000805055839009583,
+            0.005859995726495981,
+            0.0036342216189950705,
+            0.020264573395252228,
+            0.039368245750665665,
+            0.018523219972848892,
+            0.0006119278259575367,
+            -0.016337992623448372,
+            0.012547987513244152,
+            0.016269704326987267,
+            -0.006116076838225126,
+            0.015919726341962814,
+            0.00982498936355114,
+            -0.004507032223045826,
+            0.046060506254434586,
+            0.029227422550320625,
+            -0.04349968954920769,
+            0.009517691098153591,
+            0.014622246846556664,
+            -0.0250276867300272,
+            0.03404175117611885,
+            -0.017618399113416672,
+            -0.02028164640069008,
+            0.0032500994857400656,
+            0.021101105958223343,
+            -0.0002234043349744752,
+            -0.015783149749040604,
+            0.022022999823093414,
+            -0.01002131775021553,
+            -0.0014329887926578522,
+            0.007601349148899317,
+            0.026530031114816666,
+            -0.0011822424130514264,
+            -0.03356373310089111,
+            -0.002336742589250207,
+            -0.009475011378526688,
+            0.004985050763934851,
+            0.011224900372326374,
+            0.01811349019408226,
+            0.002705926541239023,
+            0.009398186579346657,
+            -0.0231156125664711,
+            0.012573596090078354,
+            -0.02337169460952282,
+            -0.008019614964723587,
+            -0.02400336228311062,
+            -0.011626094579696655,
+            0.01450274232774973,
+            0.006141684949398041,
+            0.01657700166106224,
+            -0.02053772658109665,
+            -0.009483546949923038,
+            -0.014280805364251137,
+            0.024447236210107803,
+            0.012419947423040867,
+            -0.02760557271540165,
+            -0.016585538163781166,
+            -0.012437019497156143,
+            -0.015211233869194984,
+            0.012437019497156143,
+            -0.01966704986989498,
+            0.02053772658109665,
+            0.04817744344472885,
+            -0.03568067401647568,
+            -0.011702919378876686,
+            0.03387103229761124,
+            -0.004997855052351952,
+            -0.013136974535882473,
+            0.02991030551493168,
+            0.01027739979326725,
+            -0.01014935877174139,
+            0.005172844044864178,
+            0.012633347883820534,
+            0.01400765124708414,
+            -0.020759664475917816,
+            -0.020998673513531685,
+            -0.019735338166356087,
+            -0.0031498009338974953,
+            -0.007951326668262482,
+            -0.02965422347187996,
+            0.017498893663287163,
+            0.0166794341057539,
+            -0.004447279963642359,
+            0.01650017686188221,
+            -0.028066519647836685,
+            -0.01457956712692976,
+            -0.01142976526170969,
+            -0.01387107465416193,
+            0.0314980112016201,
+            -0.0068587129935622215,
+            0.05749880522489548,
+            -0.006939805578440428,
+            0.00023194037203211337,
+            0.061083946377038956,
+            0.011404157616198063,
+            -0.0015354212373495102,
+            0.037456169724464417,
+            0.027434851974248886,
+            0.0093213627114892,
+            0.018215922638773918,
+            -0.01773790456354618,
+            0.013538168743252754,
+            -0.02081087976694107,
+            0.00026995246298611164,
+            -0.0068074967712163925,
+            0.016133127734065056,
+            -0.0385146401822567,
+            0.0359538272023201,
+            0.017200132831931114,
+            -0.002206567907705903,
+            0.01142976526170969,
+            -0.023303406313061714,
+            -0.010303007438778877,
+            0.016022158786654472,
+            -0.02197178266942501,
+            0.015424635261297226,
+            -0.004131446126848459,
+            -0.004946638830006123,
+            0.024720389395952225,
+            0.033819813281297684,
+            -0.016978194937109947,
+            -0.009893277660012245,
+            -0.042338788509368896,
+            0.0023068664595484734,
+            -0.018335426226258278,
+            0.02258637733757496,
+            0.010354223661124706,
+            -0.013854002580046654,
+            0.0009037538548000157,
+            -0.009227465838193893,
+            -0.016431888565421104,
+            0.004865546245127916,
+            0.008408005349338055,
+            -0.02492525428533554,
+            -0.039641398936510086,
+            -0.011404157616198063,
+            -0.012932109646499157,
+            -0.013136974535882473,
+            -0.016978194937109947,
+            0.0186427254229784,
+            0.0073836799710989,
+            0.011796815320849419,
+            0.009893277660012245,
+            -0.010610305704176426,
+            -0.019479257985949516,
+            0.011515126563608646,
+            0.005919747985899448,
+            0.0023900929372757673,
+            -0.026291022077202797,
+            0.003915911074727774,
+            -0.029585935175418854,
+            0.003687571967020631,
+            -0.010874922387301922,
+            0.03375152498483658,
+            -0.0027507408522069454,
+            -0.029363999143242836,
+            -0.026154445484280586,
+            -0.02492525428533554,
+            0.016884298995137215,
+            -0.010243255645036697,
+            0.00846775807440281,
+            -0.029961522668600082,
+            0.019103670492768288,
+            -0.008937240578234196,
+            -0.027025122195482254,
+            0.00028835830744355917,
+            0.004193332511931658,
+            -0.010874922387301922,
+            -0.021493764594197273,
+            -0.011327332817018032,
+            -0.011941928416490555,
+            -0.052343033254146576,
+            0.0030366983264684677,
+            -0.009389651007950306,
+            0.008766518905758858,
+            -0.0109176030382514,
+            0.00855311844497919,
+            -0.008681158535182476,
+            -0.004142116289585829,
+            -0.021510835736989975,
+            -0.014485670253634453,
+            -0.022296153008937836,
+            0.033068642020225525,
+            0.017353782430291176,
+            -0.023593630641698837,
+            -0.006713600363582373,
+            -0.008689695037901402,
+            -0.004310703370720148,
+            0.0166965052485466,
+            0.012923574075102806,
+            -0.02593250758945942,
+            0.0029022556263953447,
+            0.023474127054214478,
+            -0.003399480367079377,
+            -0.008548850193619728,
+            0.02903962880373001,
+            0.005518553778529167,
+            0.009142105467617512,
+            0.004101570229977369,
+            0.025232551619410515,
+            -0.005347833037376404,
+            0.004575320519506931,
+            0.054903849959373474,
+            0.0026355041190981865,
+            -0.005667934659868479,
+            0.03786589950323105,
+            0.0004075962060596794,
+            -0.01548438798636198,
+            0.024361874908208847,
+            0.012249225750565529,
+            0.025352057069540024,
+            0.013444272801280022,
+            -0.001758425496518612,
+            0.002910791663452983,
+            -0.00014244526391848922,
+            0.03416125476360321,
+            0.0029684100300073624,
+            0.01600508578121662,
+            -0.03947067633271217,
+            0.02195470966398716,
+            0.015689251944422722,
+            -0.0014649989316239953,
+            0.016944050788879395,
+            -0.004024745896458626,
+            -0.029620079323649406,
+            0.0031391310039907694,
+            0.0011331600835546851,
+            0.02390092983841896,
+            0.024617956951260567,
+            0.008343985304236412,
+            -0.019445113837718964,
+            -0.007823286578059196,
+            -0.002910791663452983,
+            0.05971817672252655,
+            0.011856568045914173,
+            0.03136143460869789,
+            -0.018693940714001656,
+            -0.00846775807440281,
+            -0.02106696180999279,
+            0.0021809597965329885,
+            0.025659354403614998,
+            -0.0029769460670650005,
+            -0.01540756318718195,
+            -0.010798098519444466,
+            -0.001411648583598435,
+            -0.004767381586134434,
+            -0.001800038735382259,
+            0.009142105467617512,
+            0.02079380862414837,
+            -0.013563777320086956,
+            0.01059323363006115,
+            0.028851835057139397,
+            -0.007784874178469181,
+            0.01915488764643669,
+            -0.027264131233096123,
+            -0.02746899612247944,
+            -0.010046926327049732,
+            0.012505307793617249,
+            0.009082352742552757,
+            0.013119902461767197,
+            0.020059708505868912,
+            0.008642747066915035,
+            0.008809199556708336,
+            0.014946616254746914,
+            0.015697788447141647,
+            -0.014084476046264172,
+            0.04698239639401436,
+            -0.00393298314884305,
+            -0.002027310896664858,
+            -0.010584697127342224,
+            -0.01065298542380333,
+            0.012266297824680805,
+            -0.0007298319251276553,
+            0.013478416949510574,
+            0.013085758313536644,
+            -0.02326926216483116,
+            0.01937682554125786,
+            0.00982498936355114,
+            0.0006257988861761987,
+            -0.0007986538112163544,
+            0.0166794341057539,
+            -0.013290624134242535,
+            0.026120301336050034,
+            -0.005949624348431826,
+            -0.01912074349820614,
+            0.010303007438778877,
+            0.01710623688995838,
+            0.0028659775853157043,
+            -0.02579593099653721,
+            -0.029995666816830635,
+            -0.0031049868557602167,
+            -0.03240283206105232,
+            0.02144254744052887,
+            0.007456236053258181,
+            0.014332021586596966,
+            0.014878327958285809,
+            -0.01336744800209999,
+            -0.000287024537101388,
+            0.003220223356038332,
+            0.016056302934885025,
+            0.0052454001270234585,
+            -0.05865970999002457,
+            0.004421671852469444,
+            -0.007823286578059196,
+            -0.005031999200582504,
+            -0.06091322377324104,
+            0.009500619024038315,
+            -0.006598363630473614,
+            0.022825386375188828,
+            -0.006692260038107634,
+            0.03803662210702896,
+            0.00777206989005208,
+            -0.0019302134169265628,
+            0.007819018326699734,
+            0.008220212534070015,
+            0.022944891825318336,
+            -0.007183082867413759,
+            0.00777206989005208,
+            0.0030751104932278395,
+            0.01720866933465004,
+            -0.0033525321632623672,
+            -0.012547987513244152,
+            0.0011875773780047894,
+            0.020981600508093834,
+            -0.006222777534276247,
+            0.009065280668437481,
+            0.03574896231293678,
+            0.002298330422490835,
+            -0.029773728922009468,
+            0.013666209764778614,
+            0.0020614550448954105,
+            -0.017891552299261093,
+            0.01964997872710228,
+            -0.005680738482624292,
+            0.011250508949160576,
+            -0.002041182015091181,
+            0.002332474570721388,
+            0.023525342345237732,
+            0.001920610317029059,
+            0.018540292978286743,
+            -0.0014820710057392716,
+            -0.010755417868494987,
+            0.027315348386764526,
+            -0.008766518905758858,
+            -0.022244935855269432,
+            -0.015381954610347748,
+            -0.03817319869995117,
+            -0.010268863290548325,
+            -0.0056508625857532024,
+            -0.03042246773838997,
+            0.005078947637230158,
+            0.027793366461992264,
+            0.019103670492768288,
+            -0.023730207234621048,
+            -0.023542415350675583,
+            -0.018198849633336067,
+            -0.004167724400758743,
+            0.007144670467823744,
+            -0.03322229161858559,
+            -0.012300441972911358,
+            0.012923574075102806,
+            -0.006056324578821659,
+            0.004511300474405289,
+            -0.009927421808242798,
+            -0.03096877597272396,
+            0.06944926828145981,
+            -0.015620964579284191,
+            -0.013896683230996132,
+            -0.02567642740905285,
+            -0.003478438826277852,
+            -0.011668775230646133,
+            0.023474127054214478,
+            -0.0034250884782522917,
+            0.005403317045420408,
+            0.015774613246321678,
+            0.005155771970748901,
+            0.011207828298211098,
+            -0.00033610680839046836,
+            0.009551835246384144,
+            0.014656390994787216,
+            0.005360636860132217,
+            -0.003478438826277852,
+            0.00591121194884181,
+            0.015262450091540813,
+            -0.027947014197707176,
+            0.008685426786541939,
+            -0.019428040832281113,
+            0.008681158535182476,
+            0.000938965065870434,
+            0.011617559008300304,
+            0.0031988832633942366,
+            -0.01290650200098753,
+            -0.005437461193650961,
+            -0.04073401167988777,
+            -0.009594515897333622,
+            0.023542415350675583,
+            0.017942769452929497,
+            0.03544166311621666,
+            0.0022321760188788176,
+            0.009253074415028095,
+            0.022671738639473915,
+            0.03841220587491989,
+            -0.018198849633336067,
+            0.0015813024947419763,
+            0.010550552979111671,
+            -0.0015770344762131572,
+            0.04916762560606003,
+            0.02362777478992939,
+            0.0012644018279388547,
+            -0.01149805448949337,
+            0.0006620771018788218,
+            0.00023580827109981328,
+            0.006154489237815142,
+            0.006611167918890715,
+            -0.0018437859835103154,
+            -0.002035846933722496,
+            -0.017669616267085075,
+            0.005539894104003906,
+            -0.006542879156768322,
+            -0.019735338166356087,
+            -0.0068587129935622215,
+            0.004835669882595539,
+            0.0289030522108078,
+            0.0002462115662638098,
+            0.004950906615704298,
+            -0.0005164307658560574,
+            0.0007132933242246509,
+            0.026786113157868385,
+            -0.009210393764078617,
+            0.01588558219373226,
+            0.002345278626307845,
+            -0.013597921468317509,
+            0.010174967348575592,
+            -0.008459221571683884,
+            0.0021574855782091618,
+            -0.012974790297448635,
+            0.014878327958285809,
+            0.015569748356938362,
+            -0.010192039422690868,
+            -0.025095975026488304,
+            -0.011762671172618866,
+            0.026769040152430534,
+            -0.015552676282823086,
+            0.011327332817018032,
+            0.003011090215295553,
+            0.0007778471917845309,
+            -0.00181390973739326,
+            -0.0009202924557030201,
+            0.03827562928199768,
+            -0.00943233072757721,
+            -0.016158735379576683,
+            -0.005749027244746685,
+            0.012479699216783047,
+            0.008343985304236412,
+            -0.014972224831581116,
+            -0.01925732009112835,
+            0.018796373158693314,
+            -0.017908625304698944,
+            0.015697788447141647,
+            -0.04182662442326546,
+            -0.023303406313061714,
+            0.026615392416715622,
+            -9.683077223598957e-05,
+            -0.03503193333745003,
+            0.01168584730476141,
+            0.016329456120729446,
+            0.0028617095667868853,
+            0.008271428756415844,
+            -0.008702498860657215,
+            0.0028190291486680508,
+            -0.007754997815936804,
+            -0.010849314741790295,
+            0.0013636333169415593,
+            0.007114794570952654,
+            -0.007955594919621944,
+            0.02082795277237892,
+            -0.014195444993674755,
+            -0.015211233869194984,
+            -0.005680738482624292,
+            -0.0128552857786417,
+            0.01553560420870781,
+            -0.00581731554120779,
+            -0.013256479986011982,
+            0.017473286017775536,
+            0.014485670253634453,
+            0.015433171764016151,
+            0.02478867769241333,
+            0.018301282078027725,
+            -0.02250101789832115,
+            -0.002558679785579443,
+            3.8512240280397236e-05,
+            -0.0024733194150030613,
+            0.01227483432739973,
+            0.035236798226833344,
+            0.0016421218169853091,
+            -0.009816452860832214,
+            -0.02364484779536724,
+            0.01277846097946167,
+            0.01309429481625557,
+            -0.014041795395314693,
+            -0.018711013719439507,
+            -0.004114374052733183,
+            -0.013964971527457237,
+            0.00652153929695487,
+            -0.0023111344780772924,
+            0.000925094005651772,
+            -0.0009939158335328102,
+            -0.003484840737655759,
+            -0.02091331221163273,
+            0.005356369074434042,
+            -0.01953047327697277,
+            -0.031907740980386734,
+            -0.012530915439128876,
+            0.008036687038838863,
+            0.024754533544182777,
+            0.002035846933722496,
+            -0.0224839448928833,
+            -0.0014415248297154903,
+            -0.00016365200281143188,
+            -0.0093213627114892,
+            -0.01495515275746584,
+            0.0016197147779166698,
+            0.0005118960398249328,
+            0.010038389824330807,
+            -0.007682441268116236,
+            -0.01722574047744274,
+            0.005471605341881514,
+            0.01734524592757225,
+            -0.022381512448191643,
+            -0.0025202676188200712,
+            -0.017174525186419487,
+            -0.01559535600244999,
+            0.0873408243060112,
+            -0.010891994461417198,
+            -0.012539451941847801,
+            -0.01418690849095583,
+            0.0038092106115072966,
+            -0.006333746016025543,
+            0.010678594000637531,
+            0.03264183923602104,
+            -0.006026448681950569,
+            -0.006338014267385006,
+            -0.0026717823930084705,
+            -0.0008578726556152105,
+            -0.019359752535820007,
+            0.001124624046497047,
+            0.007921450771391392,
+            -0.004566784482449293,
+            -0.003316253889352083,
+            0.01450274232774973,
+            -0.04435329511761665,
+            0.0021916297264397144,
+            0.005096019711345434,
+            -0.02260345034301281,
+            0.02002556435763836,
+            0.015919726341962814,
+            -0.022057143971323967,
+            0.003002554178237915,
+            0.00606912886723876,
+            0.03311985731124878,
+            0.022296153008937836,
+            0.009218929335474968,
+            0.015723396092653275,
+            -0.01836957037448883,
+            0.012863821350038052,
+            -0.002146815648302436,
+            -0.0002396761701675132,
+            0.008954312652349472,
+            0.004472888074815273,
+            0.013145511038601398,
+            0.0016581269446760416,
+            0.0009016198455356061,
+            -0.022688809782266617,
+            -0.006850176956504583,
+            -0.026820257306098938,
+            -0.027895798906683922,
+            -0.010379832237958908,
+            0.03182237967848778,
+            -0.008988456800580025,
+            -0.014332021586596966,
+            0.006628239993005991,
+            -0.008292769081890583,
+            -0.002063589170575142,
+            -0.024617956951260567,
+            0.016337992623448372,
+            -0.007272711489349604,
+            0.0024413091596215963,
+            0.0005537759861908853,
+            0.0112334368750453,
+            -0.0064660548232495785,
+            -0.022808315232396126,
+            0.010584697127342224,
+            -0.0024583812337368727,
+            0.03426368907094002,
+            -0.00924453791230917,
+            -0.01836957037448883,
+            0.004780185874551535,
+            0.00041613224311731756,
+            -0.01941096968948841,
+            -0.005962428171187639,
+            -0.007793410215526819,
+            -0.021425476297736168,
+            -0.014613711275160313,
+            -0.013597921468317509,
+            -0.012027288787066936,
+            -0.01655992865562439,
+            0.006278262007981539,
+            0.00014591304352506995,
+            0.008574457839131355,
+            -0.01027739979326725,
+            -0.000391057605156675,
+            0.012539451941847801,
+            -0.015433171764016151,
+            -0.019308535382151604,
+            -0.007784874178469181,
+            -0.0041976007632911205,
+            -0.0013294891687110066,
+            0.005915479734539986,
+            -0.0218522772192955,
+            -0.037319593131542206,
+            0.011335869319736958,
+            -0.011002963408827782,
+            -0.004342713393270969,
+            0.022791242226958275,
+            0.000883480766788125,
+            -0.022705882787704468,
+            0.009415258653461933,
+            0.012599203735589981,
+            0.0045027644373476505,
+            -0.015612428076565266,
+            0.005510017741471529,
+            0.0034677686635404825,
+            -0.01110539585351944,
+            -0.01860858127474785,
+            0.0034528307151049376,
+            -0.008685426786541939,
+            0.005731955170631409,
+            -0.02786165475845337,
+            0.013657674193382263,
+            0.014784432016313076,
+            0.01084077823907137,
+            0.00021033351367805153,
+            -0.00475457776337862,
+            -0.01719159632921219,
+            -0.002552277874201536,
+            -0.00642764288932085,
+            0.07013215124607086,
+            -0.012752852402627468,
+            -0.017635472118854523,
+            0.016056302934885025,
+            0.0011384951649233699,
+            -0.011617559008300304,
+            0.009662804193794727,
+            -0.01848907582461834,
+            0.020623087882995605,
+            -0.03868535906076431,
+            0.012291906401515007,
+            -0.006060592830181122,
+            0.0075842770747840405,
+            -0.008655550889670849,
+            0.0006172628491185606,
+            -0.021408403292298317,
+            0.023918000981211662,
+            0.027656789869070053,
+            0.010431048460304737,
+            -0.01847200281918049,
+            -0.027895798906683922,
+            0.016824547201395035,
+            0.017003804445266724,
+            0.002428505104035139,
+            -0.009637195616960526,
+            0.011754135601222515,
+            -0.023952145129442215,
+            0.008945776149630547,
+            -0.006346550304442644,
+            -0.0032372954301536083,
+            0.009218929335474968,
+            0.007814750075340271,
+            0.007951326668262482,
+            0.009364042431116104,
+            -0.03235161304473877,
+            0.016798939555883408,
+            0.028800619766116142,
+            0.004592392593622208,
+            0.028800619766116142,
+            -0.018796373158693314,
+            0.01823299378156662,
+            -0.006752012297511101,
+            0.009022600948810577,
+            0.021254755556583405,
+            0.011463910341262817,
+            -0.030081026256084442,
+            0.0027678129263222218,
+            -0.03380274400115013,
+            0.0017712295521050692,
+            0.02673489600419998,
+            -0.0044302078895270824,
+            0.001891801250167191,
+            -0.02284245938062668,
+            0.008860415779054165,
+            0.004443012177944183,
+            -0.020059708505868912,
+            -0.014357629232108593,
+            0.00600510835647583,
+            0.0025800198782235384,
+            -0.026547104120254517,
+            0.022518089041113853,
+            -0.023815568536520004,
+            -0.0010488666594028473,
+            0.017618399113416672,
+            0.0006231313454918563,
+            -0.01579168438911438,
+            0.008420809172093868,
+            0.02159619703888893,
+            -0.02579593099653721,
+            -0.032027244567871094,
+            0.006431910675019026,
+            0.0075842770747840405,
+            0.021630341187119484,
+            0.01978655532002449,
+            -0.008723839186131954,
+            0.02287660352885723,
+            0.007840358652174473,
+            0.0038177466485649347,
+            -0.0372513048350811,
+            -0.017635472118854523,
+            0.019957276061177254,
+            -0.014536886475980282,
+            0.018454931676387787,
+            -0.011412693187594414,
+            -0.001605843659490347,
+            0.009355506859719753,
+            -0.008139119483530521,
+            0.027793366461992264,
+            -0.01405886746942997,
+            -0.0036576958373188972,
+            0.004425940103828907,
+            -0.03341008350253105,
+            -0.0024178349412977695,
+            0.025727642700076103,
+            0.001300679985433817,
+            -0.02454966865479946,
+            -0.0030132243409752846,
+            -0.0023580826818943024,
+            0.003779334481805563,
+            -0.005475873593240976,
+            -0.007733657956123352,
+            -0.010644449852406979,
+            0.003572335233911872,
+            -0.006453251000493765,
+            -0.011352941393852234,
+            0.00214574858546257,
+            0.04985050857067108,
+            0.015817293897271156,
+            -0.007123330608010292,
+            0.0005857861833646894,
+            0.01728549413383007,
+            0.042953383177518845,
+            -0.022825386375188828,
+            0.03122485615313053,
+            0.0009693747269921005,
+            0.0038540249224752188,
+            -0.02478867769241333,
+            -0.026052013039588928,
+            0.02632516622543335,
+            0.010721273720264435,
+            -0.02542034536600113,
+            -0.002863843459635973,
+            0.00415492057800293,
+            0.012223618105053902,
+            0.014929544180631638,
+            0.014690535143017769,
+            -0.031241929158568382,
+            -0.015475851483643055,
+            -0.003343996126204729,
+            -0.008954312652349472,
+            -0.00606912886723876,
+            -0.005322224926203489,
+            0.013478416949510574,
+            0.008143387734889984,
+            -0.025778859853744507,
+            -0.0013454942964017391,
+            -0.005390513222664595,
+            -0.005343564786016941,
+            -0.01033715158700943,
+            0.009278682060539722,
+            -0.009227465838193893,
+            0.0040610237047076225,
+            -0.010448120534420013,
+            0.010832242667675018,
+            -0.015074657276272774,
+            0.010516408830881119,
+            0.004186930600553751,
+            -0.02094745635986328,
+            -0.012496771290898323,
+            -0.01838664337992668,
+            0.007183082867413759,
+            0.01937682554125786,
+            0.01835249923169613,
+            0.0062996018677949905,
+            -0.014084476046264172,
+            -0.016039229929447174,
+            -0.015151482075452805,
+            0.017908625304698944,
+            0.01014935877174139,
+            -0.005501481704413891,
+            -0.018028128892183304,
+            -0.025659354403614998,
+            0.022518089041113853,
+            0.005988036282360554,
+            -0.03612454608082771,
+            0.02250101789832115,
+            -0.008326913230121136,
+            -6.268659035413293e-06,
+            0.012761388905346394,
+            -0.010900530964136124,
+            -0.004400331526994705,
+            -0.0023474127519875765,
+            0.019206102937459946,
+            -0.0014916741056367755,
+            -0.0008450685418210924,
+            0.005706346593797207,
+            -0.00011263579654041678,
+            -0.008817736059427261,
+            -0.013213799335062504,
+            -0.0037537263706326485,
+            0.001452194876037538,
+            -0.0033226560335606337,
+            -0.0024498451966792345,
+            0.004545444622635841,
+            0.014681999571621418,
+            0.013802786357700825,
+            0.005578306037932634,
+            0.004746041726320982,
+            0.0007917182520031929,
+            -0.007242835126817226,
+            0.01059323363006115,
+            -0.021408403292298317,
+            0.006816032808274031,
+            -0.005569770000874996,
+            -0.020725520327687263,
+            -0.005902675911784172,
+            0.00018845988961402327,
+            0.02157912403345108,
+            -0.03431490436196327,
+            0.0038198805414140224,
+            0.017294028773903847,
+            0.009346970357000828,
+            -0.01836957037448883,
+            0.010883458890020847,
+            0.06518124788999557,
+            -0.005271008238196373,
+            0.008284232579171658,
+            -0.0010888794204220176,
+            0.011779743246734142,
+            -0.006184365600347519,
+            0.005322224926203489,
+            -0.0017360183410346508,
+            0.0346904918551445,
+            -0.007823286578059196,
+            -0.012940646149218082,
+            -0.023183900862932205,
+            -0.012496771290898323,
+            -0.016397744417190552,
+            0.02786165475845337,
+            -0.014477133750915527,
+            0.002011305885389447,
+            -0.012078505009412766,
+            0.0065386113710701466,
+            0.002077460289001465,
+            0.0014596638502553105,
+            -0.018215922638773918,
+            -0.009927421808242798,
+            0.016747722402215004,
+            0.0034442946780472994,
+            0.0023666187189519405,
+            0.00530088460072875,
+            -0.014528350904583931,
+            0.030524900183081627,
+            -0.0074477000162005424,
+            0.020059708505868912,
+            0.004391795489937067,
+            -0.005232596304267645,
+            0.015185626223683357,
+            -0.009466474875807762,
+            0.015859974548220634,
+            -0.0118821756914258,
+            -0.003529655048623681,
+            -0.030149314552545547,
+            -0.008565922267735004,
+            0.020623087882995605,
+            -0.014519814401865005,
+            0.0005495079676620662,
+            0.016525784507393837,
+            0.005014927126467228,
+            -0.007959863170981407,
+            0.011224900372326374,
+            0.003382408292964101,
+            -0.01758425496518612,
+            -0.010832242667675018,
+            -0.004647877067327499,
+            -0.01072981022298336,
+            0.011301725171506405,
+            0.013589384965598583,
+            -0.022398585453629494,
+            0.0015290192095562816,
+            -0.006914197467267513,
+            -0.008207408711314201,
+            0.00716174254193902,
+            0.013162583112716675,
+            -0.027571428567171097,
+            0.017370853573083878,
+            0.0021254755556583405,
+            -0.0002107336331391707,
+            0.004250951111316681,
+            -0.0004980249213986099,
+            0.014605174772441387,
+            -0.007575741037726402,
+            -0.002740070689469576,
+            -0.006845909170806408,
+            0.010934675112366676,
+            -0.0004094634496141225,
+            0.0157319325953722,
+            -0.00683737313374877,
+            -0.002915059681981802,
+            0.007234299089759588,
+            0.016286775469779968,
+            -0.02557399496436119,
+            0.015424635261297226,
+            -0.00810070801526308,
+            0.0002974278468172997,
+            0.00031343294540420175,
+            0.009253074415028095,
+            0.005591110326349735,
+            -0.015902653336524963,
+            -0.0072300308384001255,
+            0.015569748356938362,
+            0.005117359571158886,
+            0.018950022757053375,
+            0.021476691588759422,
+            0.007033701986074448,
+            0.023866785690188408,
+            -0.005744758993387222,
+            -0.013085758313536644,
+            -0.0020561201963573694,
+            -0.009338434785604477,
+            -0.002368752844631672,
+            0.006286798045039177,
+            -0.004054621793329716,
+            -0.025915436446666718,
+            -0.012821141630411148,
+            -0.007276979275047779,
+            -0.00038012079312466085,
+            0.009475011378526688,
+            -0.010311543941497803,
+            0.01707209274172783,
+            0.02236444130539894,
+            7.760800144751556e-06,
+            -0.006175829563289881,
+            -0.02094745635986328,
+            0.007251371163874865,
+            -0.007763533852994442,
+            -0.011737063527107239,
+            -0.010098142549395561,
+            -0.0028702456038445234,
+            0.0018395179649814963,
+            0.005407585296779871,
+            0.004775917623192072,
+            -0.017772048711776733,
+            -0.006862981244921684,
+            0.009423795156180859,
+            -0.01799398474395275,
+            -0.0023239385336637497,
+            0.00031343294540420175,
+            0.006666651926934719,
+            0.012872357852756977,
+            0.01758425496518612,
+            -0.013614993542432785,
+            -0.022449800744652748,
+            0.003002554178237915,
+            -0.009739628992974758,
+            0.03274427354335785,
+            -0.008168996311724186,
+            -0.005514285992830992,
+            -0.018284210935235023,
+            -0.006175829563289881,
+            -0.009346970357000828,
+            -0.00956890732049942,
+            -0.0004897556500509381,
+            0.014861255884170532,
+            -0.009304290637373924,
+            0.01482711173593998,
+            -0.016491640359163284,
+            -0.003205285407602787,
+            0.00409090006724,
+            0.00969694834202528,
+            -0.030012737959623337,
+            -0.006278262007981539,
+            -0.00799400731921196,
+            -0.02145962044596672,
+            -0.013708890415728092,
+            -0.012471163645386696,
+            0.007588544860482216,
+            0.0014073805650696158,
+            0.01642335206270218,
+            -0.007187350653111935,
+            -0.014195444993674755,
+            0.009978638030588627,
+            -0.02031579054892063,
+            -0.012983325868844986,
+            -0.008292769081890583,
+            -0.04073401167988777,
+            0.020469438284635544,
+            0.01354670524597168,
+            0.023337550461292267,
+            -0.0024434432853013277,
+            0.020964529365301132,
+            0.008723839186131954,
+            -0.01742207072675228,
+            0.006372158415615559,
+            -0.012505307793617249,
+            0.005744758993387222,
+            -0.02506183087825775,
+            -0.0066026318818330765,
+            0.005505749955773354,
+            0.01246262714266777,
+            -0.0037665304262191057,
+            2.7758822398027405e-05,
+            0.014280805364251137,
+            -0.01041397638618946,
+            -0.01787448115646839,
+            0.004067426081746817,
+            0.0067562805488705635,
+            -0.01713184453547001,
+            0.03670499846339226,
+            -0.00796413142234087,
+            -0.00017445544654037803,
+            0.013179655186831951,
+            0.02104988880455494,
+            -0.020264573395252228,
+            -0.01450274232774973,
+            0.01937682554125786,
+            -0.0010200574761256576,
+            -0.0004940236685797572,
+            0.017183061689138412,
+            -0.006828837096691132,
+            0.008032419718801975,
+            -0.005766099318861961,
+            -0.00484847417101264,
+            -0.00503626698628068,
+            0.0011683712946251035,
+            -0.016901372000575066,
+            0.0042637549340724945,
+            -0.000693553767632693,
+            -0.0006658115889877081,
+            0.019581690430641174,
+            -0.006453251000493765,
+            0.0020102388225495815,
+            0.011976072564721107,
+            0.00038332180702127516,
+            -0.05049924924969673,
+            -0.0047204336151480675,
+            0.0034869748633354902,
+            -0.003279975615441799,
+            -0.003469902789220214,
+            0.009807917289435863,
+            0.01195046491920948,
+            0.01110539585351944,
+            -0.005130163859575987,
+            0.013452808372676373,
+            0.00938111450523138,
+            0.006632507778704166,
+            0.0007287649204954505,
+            -0.004329909570515156,
+            0.011813887394964695,
+            -0.004588124807924032,
+            0.020127996802330017,
+            -0.004421671852469444,
+            -0.01889880560338497,
+            -0.016414817422628403,
+            0.0010739413555711508,
+            0.027025122195482254,
+            -0.0037665304262191057,
+            0.026103228330612183,
+            -0.0013839064631611109,
+            0.0033205219078809023,
+            -0.0018032396910712123,
+            -0.0032223574817180634,
+            -0.004827133845537901,
+            -0.035134363919496536,
+            0.009039673022925854,
+            -0.007084918208420277,
+            0.020008491352200508,
+            -0.02943228743970394,
+            0.003026028396561742,
+            -0.006175829563289881,
+            -0.005787439178675413,
+            -0.0025608139112591743,
+            0.005031999200582504,
+            -2.869245145120658e-05,
+            0.03803662210702896,
+            -0.014844183810055256,
+            -0.021271826699376106,
+            -0.010396904312074184,
+            0.006995289586484432,
+            0.02798115834593773,
+            -0.0004838871245738119,
+            0.01913781464099884,
+            0.020896241068840027,
+            -0.013068686239421368,
+            0.004844206385314465,
+            -0.0031156567856669426,
+            -0.002942801918834448,
+            0.00011677043949021026,
+            -0.009440867230296135,
+            0.005198452156037092,
+            -0.03530508652329445,
+            0.007008093874901533,
+            -0.0005225660861469805,
+            0.02427651546895504,
+            5.1282964705023915e-05,
+            0.004814330022782087,
+            -0.003311985870823264,
+            0.02274002693593502,
+            0.003945787437260151,
+            0.01719159632921219,
+            -0.0013017470482736826,
+            0.004767381586134434,
+            0.010396904312074184,
+            -0.019086599349975586,
+            -0.019820699468255043,
+            -0.015142945572733879,
+            0.001055268687196076,
+            0.0011448971927165985,
+            -0.015270986594259739,
+            -0.018830517306923866,
+            -0.0059923045337200165,
+            0.017515966668725014,
+            -0.011702919378876686,
+            0.01264188438653946,
+            0.013538168743252754,
+            0.009159177541732788,
+            -0.001976094674319029,
+            0.01227483432739973,
+            0.01592826284468174,
+            -0.018215922638773918,
+            0.009022600948810577,
+            -0.009577443823218346,
+            -0.008937240578234196,
+            -0.010960282757878304,
+            -0.01811349019408226,
+            0.03438319265842438,
+            0.014750287868082523,
+            -0.011677310802042484,
+            -0.008587262593209743,
+            -0.0010019184555858374,
+            0.004758845549076796,
+            0.00044013987644575536,
+            0.005663666408509016,
+            -0.004647877067327499,
+            -0.026137372478842735,
+            -0.014673463068902493,
+            -0.016466032713651657,
+            -0.005719150882214308,
+            0.002590690040960908,
+            0.014801504090428352,
+            -0.021732773631811142,
+            -0.016397744417190552,
+            -0.0263080932199955,
+            0.01925732009112835,
+            -0.017533037811517715,
+            0.01651724986732006,
+            0.012249225750565529,
+            0.002076393226161599,
+            -0.01265041995793581,
+            -0.02801530249416828,
+            0.014741751365363598,
+            -0.024361874908208847,
+            0.00670933211222291,
+            -0.0016261168057098985,
+            0.0500553734600544,
+            -0.00277634896337986,
+            -0.006192901637405157,
+            0.009500619024038315,
+            0.003561665304005146,
+            0.013845467008650303,
+            -0.02197178266942501,
+            -0.005403317045420408,
+            -0.003525387030094862,
+            0.011566342785954475,
+            -0.004571052733808756,
+            -0.0056892745196819305,
+            -0.002590690040960908,
+            -0.007375143934041262,
+            0.001402045600116253,
+            0.019291464239358902,
+            -0.006880053319036961,
+            -0.007008093874901533,
+            -0.0030921828001737595,
+            0.029842017218470573,
+            0.01963290572166443,
+            -0.008548850193619728,
+            -0.006444714963436127,
+            0.008365325629711151,
+            -0.01787448115646839,
+            -0.0014916741056367755,
+            0.017370853573083878,
+            0.021220611408352852,
+            -0.011224900372326374,
+            0.006999557837843895,
+            0.002942801918834448,
+            -0.0033269240520894527,
+            -0.021510835736989975,
+            -0.01137001346796751,
+            -0.0075202565640211105,
+            -0.0018896672409027815,
+            0.01632091961801052,
+            -0.011310260742902756,
+            0.02325218915939331,
+            0.01567218080163002,
+            0.009517691098153591,
+            0.0010883458890020847,
+            -0.041280318051576614,
+            -0.01548438798636198,
+            0.01687576249241829,
+            -0.0034613667521625757,
+            0.017362317070364952,
+            0.013649137690663338,
+            0.0031007188372313976,
+            -0.019479257985949516,
+            0.023952145129442215,
+            0.0008594731334596872,
+            0.0014233856927603483,
+            -0.001049933722242713,
+            0.015330738388001919,
+            0.0019632906187325716,
+            0.00962012354284525,
+            0.02106696180999279,
+            0.020640159025788307,
+            -0.01797691360116005,
+            -0.009987173601984978,
+            0.011301725171506405,
+            -0.017037948593497276,
+            0.019684122875332832,
+            0.001101149944588542,
+            0.003862560959532857,
+            0.015501460060477257,
+            0.027025122195482254,
+            0.03277841582894325,
+            0.011062716133892536,
+            -0.004933834541589022,
+            -0.007959863170981407,
+            -0.001679467037320137,
+            0.004545444622635841,
+            0.014041795395314693,
+            -0.02337169460952282,
+            -0.031054135411977768,
+            0.003213821444660425,
+            0.013529633171856403,
+            -0.02506183087825775,
+            0.007725121919065714,
+            0.01913781464099884,
+            0.02181813307106495,
+            0.0050917514599859715,
+            0.015552676282823086,
+            0.014519814401865005,
+            0.0034528307151049376,
+            -0.002049718052148819,
+            0.010046926327049732,
+            -0.013623529113829136,
+            0.005155771970748901,
+            0.0022833922412246466,
+            -0.015509995631873608,
+            0.02053772658109665,
+            -0.00037345202872529626,
+            0.00026034939219243824,
+            0.003395212348550558,
+            0.001364700379781425,
+            -0.00950915552675724,
+            -0.03426368907094002,
+            0.016551394015550613,
+            -0.004159188363701105,
+            0.010439584031701088,
+            -0.017020875588059425,
+            -0.0075714727863669395,
+            -0.007511720526963472,
+            0.03421247377991676,
+            0.009304290637373924,
+            0.007631225045770407,
+            -0.004293631296604872,
+            0.0008301304769702256,
+            0.003205285407602787,
+            0.007699513342231512,
+            -0.01799398474395275,
+            -0.018045201897621155,
+            0.009662804193794727,
+            0.004903958644717932,
+            8.744445949560031e-06,
+            0.01642335206270218,
+            0.009073817171156406,
+            0.012078505009412766,
+            0.02299610897898674,
+            0.008040955290198326,
+            -0.01728549413383007,
+            0.0033013159409165382,
+            0.019957276061177254,
+            -0.0008957513491623104,
+            0.01707209274172783,
+            0.007486112415790558,
+            -0.007008093874901533,
+            -0.026103228330612183,
+            0.00020659899746533483,
+            0.018164705485105515,
+            0.023952145129442215,
+            -0.023969218134880066,
+            -0.019052455201745033,
+            0.01850614883005619,
+            -0.0121980095282197,
+            0.010627377778291702,
+            -0.002761411014944315,
+            0.000420933763962239,
+            0.0031754090450704098,
+            -0.017217205837368965,
+            0.004562516696751118,
+            0.017037948593497276,
+            0.01014935877174139,
+            0.00015778347733430564,
+            0.005386244971305132,
+            0.01809641718864441,
+            0.022449800744652748,
+            -0.030661476776003838,
+            -0.010968819260597229,
+            -0.0029769460670650005,
+            0.004579588770866394,
+            -0.014613711275160313,
+            -0.0028531732968986034,
+            0.005108823534101248,
+            0.00937257893383503,
+            -0.022057143971323967,
+            -0.0109176030382514,
+            -0.0023025984410196543,
+            -0.0024626492522656918,
+            0.008903096430003643,
+            -0.004733237437903881,
+            0.0030559045262634754,
+            -0.016798939555883408,
+            0.005492945667356253,
+            -0.019615834578871727,
+            0.017183061689138412,
+            -0.0049252985045313835,
+            -0.0008722771890461445,
+            -0.010217647068202496,
+            0.0021062693558633327,
+            -0.0028702456038445234,
+            -0.02390092983841896,
+            0.022893676534295082,
+            -0.0157148614525795,
+            -0.0036897058598697186,
+            0.043738700449466705,
+            0.002827565185725689,
+            0.006828837096691132,
+            0.02364484779536724,
+            0.002174557652324438,
+            -0.004647877067327499,
+            0.026103228330612183,
+            0.03175409138202667,
+            -0.010550552979111671,
+            -0.009748164564371109,
+            0.017618399113416672,
+            0.009278682060539722,
+            0.006935537327080965,
+            0.011489517986774445,
+            -0.0027784830890595913,
+            -0.017686687409877777,
+            -0.00911649689078331,
+            0.0011790413409471512,
+            0.015492923557758331,
+            -0.0032308935187757015,
+            -0.0020059708040207624,
+            0.011924856342375278,
+            7.60908515076153e-05,
+            -0.007524524815380573,
+            0.02837381698191166,
+            0.002735802670940757,
+            -0.01784033700823784,
+            -0.029210349544882774,
+            0.0118821756914258,
+            -0.0125138433650136,
+            -0.018796373158693314,
+            -0.017294028773903847,
+            -0.0021030684001743793,
+            -0.011301725171506405,
+            -0.002526669530197978,
+            0.012829677201807499,
+            0.01163463108241558,
+            -0.005211255978792906,
+            0.01620141603052616,
+            0.006329478230327368,
+            -0.015902653336524963,
+            -0.01612459123134613,
+            -0.004276559222489595,
+            -0.003909509163349867,
+            -0.0035040469374507666,
+            -0.003911643289029598,
+            0.018198849633336067,
+            -0.009090889245271683,
+            -0.0044302078895270824,
+            -0.018557364121079445,
+            0.01760132610797882,
+            0.013700353913009167,
+            -0.009398186579346657,
+            -0.001582369557581842,
+            -0.001988898729905486,
+            0.028203096240758896,
+            0.0062996018677949905,
+            0.0017947036540135741,
+            0.004297899082303047,
+            -0.00968841277062893,
+            -0.0010846114018931985,
+            -0.02463502809405327,
+            0.020093852654099464,
+            -0.018437858670949936,
+            -0.026939760893583298,
+            -0.007170278578996658,
+            0.024959398433566093,
+            -0.0018395179649814963,
+            0.0015076791169121861,
+            -0.00831837672740221,
+            0.018932949751615524,
+            0.0029812140855938196,
+            -0.02823724038898945,
+            -0.0125138433650136,
+            -0.010183502919971943,
+            -0.015552676282823086,
+            -0.0013518963241949677,
+            0.020332861691713333,
+            -0.01360645703971386,
+            0.00035237864358350635,
+            0.01671357825398445,
+            -0.01149805448949337,
+            0.013469880446791649,
+            -0.003260769648477435,
+            -0.009125033393502235,
+            0.013845467008650303,
+            -0.008536046370863914,
+            -0.00408022990450263,
+            -0.003094316693022847,
+            0.005275276489555836,
+            -0.0012387937167659402,
+            -0.019086599349975586,
+            -0.009440867230296135,
+            0.01084077823907137,
+            0.002966275904327631,
+            0.008728107437491417,
+            -0.014050331898033619,
+            -0.005629522260278463,
+            0.001096881926059723,
+            -7.188950985437259e-05,
+            -0.0018960692686960101,
+            -0.012582131661474705,
+            -0.011327332817018032,
+            -0.020879168063402176,
+            -0.007665369194000959,
+            -0.01206143293529749,
+            0.011173684149980545,
+            0.015245378017425537,
+            -0.010738345794379711,
+            -0.005642326548695564,
+            0.008305572904646397,
+            0.04100716486573219,
+            -0.009782308712601662,
+            0.004383259452879429,
+            0.021613268181681633,
+            0.0032372954301536083,
+            0.007592813111841679,
+            0.02874940261244774,
+            -0.0017701626056805253,
+            -0.010704201646149158,
+            0.005117359571158886,
+            -0.01825006678700447,
+            -0.002957739867269993,
+            0.027554357424378395,
+            -0.021271826699376106,
+            0.005787439178675413,
+            -0.023730207234621048,
+            -0.008079367689788342,
+            -0.0033888102043420076,
+            -0.0121980095282197,
+            -0.014238124713301659,
+            -0.002049718052148819,
+            0.0006028582574799657,
+            0.023713136091828346,
+            -0.007439163979142904,
+            0.0050405352376401424,
+            0.009159177541732788,
+            -0.025130119174718857,
+            0.007426360156387091,
+            -0.010994426906108856,
+            -0.010166430845856667,
+            -0.004916762467473745,
+            -0.026803184300661087,
+            0.015501460060477257,
+            -0.011540734209120274,
+            -0.006717868149280548,
+            -0.018557364121079445,
+            0.033205218613147736,
+            0.017635472118854523,
+            0.011276116594672203,
+            -0.02313268557190895,
+            -0.015014905482530594,
+            -0.020110923796892166,
+            -0.037046439945697784,
+            -0.00030409664032049477,
+            0.0008386665140278637,
+            -0.002919327700510621,
+            0.007716585882008076,
+            0.025864219292998314,
+            0.0017701626056805253,
+            -0.006444714963436127,
+            -0.004575320519506931,
+            0.007375143934041262,
+            0.009005528874695301,
+            -0.014417381957173347,
+            0.00911649689078331,
+            0.018301282078027725,
+            0.0002280724875163287,
+            0.007725121919065714,
+            0.00813058391213417,
+            0.0020305118523538113,
+            0.00548867741599679,
+            0.028407961130142212,
+            0.0034122844226658344,
+            0.003109254874289036,
+            -0.0001357764849672094,
+            -7.043905043246923e-06,
+            -0.011148076504468918,
+            0.0010947479167953134,
+            -0.018591508269309998,
+            0.00600510835647583,
+            -0.01860858127474785,
+            0.0199231319129467,
+            -0.004127178341150284,
+            0.008092171512544155,
+            -0.02207421511411667,
+            -0.0243960190564394,
+            0.004268023185431957,
+            0.027178769931197166,
+            0.01259066816419363,
+            -0.007878770120441914,
+            -0.015859974548220634,
+            0.01888173446059227,
+            0.01104564405977726,
+            -0.011267581023275852,
+            -0.012505307793617249,
+            -0.017789119854569435,
+            -9.489682270213962e-05,
+            -0.02144254744052887,
+            0.02528376877307892,
+            0.015902653336524963,
+            0.0038881690707057714,
+            -0.013171118684113026,
+            0.02425944246351719,
+            -0.011549270711839199,
+            0.0070465062744915485,
+            -0.01503197755664587,
+            -0.015561211854219437,
+            -0.0186427254229784,
+            -0.005945356097072363,
+            -0.001342293224297464,
+            0.011651703156530857,
+            -0.027571428567171097,
+            0.013768642209470272,
+            0.014391773380339146,
+            -0.002705926541239023,
+            0.0022215060889720917,
+            -0.0055356258526444435,
+            0.014571030624210835,
+            -0.006875785067677498,
+            0.007336731534451246,
+            -0.006991021800786257,
+            -0.0021873617079108953,
+            -0.0065172710455954075,
+            -0.01418690849095583,
+            -0.014656390994787216,
+            -0.0053307609632611275,
+            -0.014110084623098373,
+            0.0035125829745084047,
+            0.019342679530382156,
+            -0.026052013039588928,
+            -0.00018205786182079464,
+            -0.001230257679708302,
+            -6.235315231606364e-05,
+            -0.0032735737040638924,
+            0.012795533053576946,
+            0.016918443143367767,
+            -0.0067861564457416534,
+            -0.005117359571158886,
+            -0.0007324994658119977,
+            -0.017498893663287163,
+            -0.0004924231325276196,
+            0.021203538402915,
+            -0.0031433990225195885,
+            -0.005962428171187639,
+            -0.010934675112366676,
+            -0.038856081664562225,
+            -0.01078956201672554,
+            -0.01862565241754055,
+            -0.021544979885220528,
+            -0.006939805578440428,
+            0.0007244969019666314,
+            0.008028151467442513,
+            0.01381132286041975,
+            -0.010712738148868084,
+            -0.011096860282123089,
+            0.0033290579449385405,
+            -0.01622702367603779,
+            0.0016463898355141282,
+            -0.019342679530382156,
+            -0.0013614993076771498,
+            0.008783590979874134,
+            0.014212517067790031,
+            0.0044515482150018215,
+            0.00632094219326973,
+            0.0019995688926428556,
+            -0.011301725171506405,
+            -0.009978638030588627,
+            0.019189031794667244,
+            -0.022552233189344406,
+            0.003687571967020631,
+            -0.004865546245127916,
+            0.0065386113710701466,
+            -0.0034400266595184803,
+            -0.02159619703888893,
+            0.006367890164256096,
+            -0.02069137617945671,
+            0.010362760163843632,
+            -0.0010099209612235427,
+            -0.004950906615704298,
+            0.006658115889877081,
+            -0.0021521507296711206,
+            -0.011719991452991962,
+            0.01772083155810833,
+            -0.006777620408684015,
+            -0.004511300474405289,
+            -0.007191618904471397,
+            0.006244117859750986,
+            0.01588558219373226,
+            0.011037107557058334,
+            -0.0053819771856069565,
+            0.0062483856454491615,
+            0.011455373838543892,
+            -0.0026995246298611164,
+            -0.00767817348241806,
+            -0.01760132610797882,
+            0.00048788837739266455,
+            0.013956435024738312,
+            -0.020657232031226158,
+            0.0059282840229570866,
+            0.011967536993324757,
+            0.024310659617185593,
+            -0.007494648452848196,
+            0.006991021800786257,
+            -0.01913781464099884,
+            0.015561211854219437,
+            -0.00414425041526556,
+            -0.02250101789832115,
+            -0.0015524934278801084,
+            -0.013802786357700825,
+            0.0009058878640644252,
+            -0.005305152852088213,
+            0.00010189905151491985,
+            -0.0060435207560658455,
+            -0.0010563357500359416,
+            -0.004724701400846243,
+            -0.015262450091540813,
+            0.0013454942964017391,
+            -0.01610751822590828,
+            0.006790424697101116,
+            0.015288058668375015,
+            0.0144515261054039,
+            0.0028489052783697844,
+            0.00439606374129653,
+            -0.0030196262523531914,
+            0.002827565185725689,
+            -0.011916320770978928,
+            -0.004750309512019157,
+            0.007481844164431095,
+            -0.018284210935235023,
+            0.0005361703806556761,
+            -0.009338434785604477,
+            0.009099424816668034,
+            -0.022927820682525635,
+            0.003696108004078269,
+            -0.00207212520763278,
+            -0.014545422978699207,
+            -0.007072114385664463,
+            -0.007908646948635578,
+            0.023013180121779442,
+            0.010285935364663601,
+            0.01458810269832611,
+            0.005049071274697781,
+            -0.018130561336874962,
+            -0.01567218080163002,
+            0.0008263959898613393,
+            -0.001281473902054131,
+            0.02209128811955452,
+            -0.018932949751615524,
+            0.004464352037757635,
+            0.00844641774892807,
+            -0.02878354676067829,
+            -0.012283369898796082,
+            0.001568498439155519,
+            -0.01335891243070364,
+            0.009483546949923038,
+            0.009850597009062767,
+            -0.00800254289060831,
+            -0.0037729323375970125,
+            -0.004417404066771269,
+            -0.01630384847521782,
+            -0.006854445207864046,
+            -0.002769947052001953,
+            0.01169438287615776,
+            -0.009731092490255833,
+            0.0021916297264397144,
+            0.005445997230708599,
+            0.0121980095282197,
+            -0.007853162474930286,
+            -0.011207828298211098,
+            -0.010687129572033882,
+            0.002074259100481868,
+            0.015416099689900875,
+            -0.01823299378156662,
+            -0.010260327719151974,
+            0.015347810462117195,
+            -0.017515966668725014,
+            0.009893277660012245,
+            -0.00962012354284525,
+            -0.021732773631811142,
+            -0.006730672437697649,
+            0.016978194937109947,
+            0.000151514817844145,
+            0.02453259564936161,
+            0.01124197244644165,
+            -0.00622704578563571,
+            -0.006397766526788473,
+            -0.02349119819700718,
+            0.01201021671295166,
+            -0.006901393178850412,
+            0.0069739497266709805,
+            -0.000578317150939256,
+            -6.068595394026488e-05,
+            0.011711454950273037,
+            -0.001123557100072503,
+            -0.004107972141355276,
+            -0.011907784268260002,
+            -0.008087903261184692,
+            0.02951764687895775,
+            -0.007554400712251663,
+            0.017686687409877777,
+            0.01259066816419363,
+            -0.006884321104735136,
+            0.008361057378351688,
+            0.009048208594322205,
+            0.00388603494502604,
+            -0.010823706164956093,
+            0.01458810269832611,
+            -0.010806634090840816,
+            -0.016337992623448372,
+            0.008373861201107502,
+            0.007106258533895016,
+            -0.012027288787066936,
+            -0.008971384726464748,
+            0.0018619250040501356,
+            -0.0011854433687403798,
+            0.0003201017389073968,
+            -0.011011499911546707,
+            -0.009150641039013863,
+            -0.002554411767050624,
+            -0.005416121333837509,
+            0.0021830936893820763,
+            0.0027976890560239553,
+            0.008480561897158623,
+            0.002078527119010687,
+            -0.01645749621093273,
+            -0.01976948231458664,
+            -0.022279080003499985,
+            -0.015629500150680542,
+            0.00491249468177557,
+            0.0001392442500218749,
+            0.010268863290548325,
+            -0.007716585882008076,
+            0.014758823439478874,
+            -0.02079380862414837,
+            0.011412693187594414,
+            0.013324768282473087,
+            0.006508735008537769,
+            -0.013760106638073921,
+            -0.023098541423678398,
+            -0.012573596090078354,
+            0.0030004202853888273,
+            0.006952609401196241,
+            0.015125873498618603,
+            -0.00032917127828113735,
+            -0.008757983334362507,
+            0.00399700365960598,
+            -0.012573596090078354,
+            0.006769084371626377,
+            -0.00616302527487278,
+            0.0069227335043251514,
+            0.0011267580557614565,
+            -0.006901393178850412,
+            0.007985470816493034,
+            -0.005740491207689047,
+            0.0033717381302267313,
+            -0.008655550889670849,
+            -0.01663675345480442,
+            0.016431888565421104,
+            -0.0013518963241949677,
+            -0.007434896193444729,
+            -0.00860433466732502,
+            -0.004257353022694588,
+            -0.0015055451076477766,
+            -0.0036790359299629927,
+            0.009526227600872517,
+            0.011335869319736958,
+            0.027042193338274956,
+            0.00886895228177309,
+            0.008164728060364723,
+            0.004287228919565678,
+            -0.0005313688889145851,
+            0.013162583112716675,
+            0.018830517306923866,
+            0.00020873300672974437,
+            0.002744338707998395,
+            -0.0045539806596934795,
+            -0.02168155647814274,
+            -0.004562516696751118,
+            0.017686687409877777,
+            0.00937257893383503,
+            0.002082795137539506,
+            -0.007302587386220694,
+            0.0013999115908518434,
+            0.0019451514817774296,
+            -0.02299610897898674,
+            0.015919726341962814,
+            -0.005919747985899448,
+            -0.0041613224893808365,
+            0.015014905482530594,
+            -0.016594072803854942,
+            0.0052667404524981976,
+            -0.009176249615848064,
+            0.0031988832633942366,
+            0.0023346086964011192,
+            0.008267160505056381,
+            0.03762689232826233,
+            -0.01496368832886219,
+            0.006534343119710684,
+            0.021135250106453896,
+            -0.008629942312836647,
+            -0.017772048711776733,
+            -0.003045234363526106,
+            -0.006000840570777655,
+            0.004191198851913214,
+            0.012223618105053902,
+            0.0030921828001737595,
+            -0.0007869167602621019,
+            0.017515966668725014,
+            0.001877930131740868,
+            0.01206143293529749,
+            -0.014852720312774181,
+            -1.1286920198472217e-05,
+            -0.008937240578234196,
+            0.013410128653049469,
+            0.0019728937186300755,
+            -0.015902653336524963,
+            -0.002094532363116741,
+            0.02043529413640499,
+            -0.018215922638773918,
+            0.013717425987124443,
+            -0.006090468727052212,
+            0.02673489600419998,
+            0.004711897578090429,
+            0.0073153916746377945,
+            -0.004818597808480263,
+            0.009705484844744205,
+            0.013614993542432785,
+            0.024361874908208847,
+            -0.0015610294649377465,
+            0.002044382970780134,
+            0.0058557274751365185,
+            0.013691818341612816,
+            0.01610751822590828,
+            0.006478859111666679,
+            -0.0016549259889870882,
+            0.029978593811392784,
+            0.01720866933465004,
+            0.007306855637580156,
+            0.004438743926584721,
+            0.0001512480666860938,
+            -0.017200132831931114,
+            -0.005889871623367071,
+            -0.03168580308556557,
+            0.0036342216189950705,
+            0.013947899453341961,
+            0.010371295735239983,
+            -0.015108801424503326,
+            0.0083482526242733,
+            0.007050774060189724,
+            -0.018335426226258278,
+            0.013196727260947227,
+            0.0012217216426506639,
+            0.008826271630823612,
+            0.004387527704238892,
+            0.008006811141967773,
+            0.008510437794029713,
+            0.022159576416015625,
+            0.008783590979874134,
+            0.0063508180901408195,
+            -0.008006811141967773,
+            -0.01912074349820614,
+            0.0026995246298611164,
+            -0.016926979646086693,
+            0.001235592644661665,
+            0.010388367809355259,
+            -0.006141684949398041,
+            0.002078527119010687,
+            0.014357629232108593,
+            0.011113932356238365,
+            0.00018219123012386262,
+            -0.009807917289435863,
+            0.021374259144067764,
+            -0.0026162981521338224,
+            0.002697390504181385,
+            -0.02134011499583721,
+            0.0044515482150018215,
+            0.013649137690663338,
+            -0.00956037174910307,
+            0.012018753215670586,
+            -0.01951340213418007,
+            0.013469880446791649,
+            0.001887533231638372,
+            -0.01213825773447752,
+            -0.001184376422315836,
+            -0.010772489942610264,
+            0.023218045011162758,
+            -0.01574900560081005,
+            0.014084476046264172,
+            0.02490818314254284,
+            -0.002483989344909787,
+            -0.005569770000874996,
+            0.0022684542927891016,
+            0.012727244757115841,
+            -0.014605174772441387,
+            -0.006329478230327368,
+            0.0015738335205242038,
+            0.030354179441928864,
+            -0.009765236638486385,
+            0.002919327700510621,
+            -0.00886895228177309,
+            0.0199231319129467,
+            -0.01315404660999775,
+            -0.006760548334568739,
+            0.005612450186163187,
+            0.02673489600419998,
+            -0.00268245255574584,
+            0.027554357424378395,
+            0.016722114756703377,
+            0.0010526011465117335,
+            0.006171561311930418,
+            -0.01521977037191391,
+            0.006905661430209875,
+            0.017686687409877777,
+            0.01246262714266777,
+            0.017396461218595505,
+            -0.009048208594322205,
+            0.009543299674987793,
+            0.027537284418940544,
+            -0.00713186664506793,
+            0.0038177466485649347,
+            0.012308978475630283,
+            0.011156612075865269,
+            0.003747324226424098,
+            -0.0017541574779897928,
+            0.004259487148374319,
+            -0.015236842446029186,
+            0.013136974535882473,
+            -0.00943233072757721,
+            0.006393498741090298,
+            0.021510835736989975,
+            0.007925719022750854,
+            9.296288044424728e-05,
+            0.0006012577214278281,
+            0.012445555068552494,
+            0.03315400332212448,
+            -0.003975663334131241,
+            0.013768642209470272,
+            -0.0067861564457416534,
+            0.004250951111316681,
+            -0.01007253397256136,
+            -0.012872357852756977,
+            -0.0001271737419301644,
+            -0.014630783349275589,
+            -0.0010915468446910381,
+            7.54239663365297e-05,
+            0.008783590979874134,
+            -0.017618399113416672,
+            0.0015279522631317377,
+            -0.006833104882389307,
+            -0.007187350653111935,
+            -0.011865103617310524,
+            0.01137001346796751,
+            -0.010968819260597229,
+            -0.007332463748753071,
+            0.007281247526407242,
+            0.0076739052310585976,
+            -6.271993333939463e-05,
+            -0.02156205289065838,
+            -0.01059323363006115,
+            0.002249248092994094,
+            0.0036064793821424246,
+            0.014750287868082523,
+            -0.02067430317401886,
+            -0.004289363045245409,
+            0.01002131775021553,
+            -0.009133568964898586,
+            -0.008224480785429478,
+            0.0033717381302267313,
+            -0.01052494440227747,
+            -0.017703760415315628,
+            -0.012078505009412766,
+            0.00831837672740221,
+            -0.019803626462817192,
+            -0.029876161366701126,
+            -0.017635472118854523,
+            0.009329898282885551,
+            -0.016448961570858955,
+            -0.004519836511462927,
+            -0.025352057069540024,
+            0.02181813307106495,
+            0.008489097468554974,
+            0.011173684149980545,
+            0.006811764556914568,
+            -0.006547147408127785,
+            0.012949181720614433,
+            0.0010008513927459717,
+            -0.004289363045245409,
+            0.014050331898033619,
+            0.015424635261297226,
+            -0.010900530964136124,
+            -0.010866386815905571,
+            0.048757895827293396,
+            -0.004455816000699997,
+            0.008809199556708336,
+            -0.0011715723667293787,
+            0.02555692195892334,
+            0.022381512448191643,
+            0.01118222065269947,
+            0.009944493882358074,
+            0.026171516627073288,
+            -0.005813047289848328,
+            0.0008845477714203298,
+            0.018711013719439507,
+            0.002041182015091181,
+            0.028425034135580063,
+            0.016209950670599937,
+            -0.003617149544879794,
+            -0.008412273600697517,
+            -0.0032436975743621588,
+            0.0024818554520606995,
+            0.017353782430291176,
+            -0.003011090215295553,
+            -0.00116517033893615,
+            0.011976072564721107,
+            -0.018796373158693314,
+            -0.02606908418238163,
+            -0.028544537723064423,
+            -0.008096439763903618,
+            -0.003841220634058118,
+            0.0006578090251423419,
+            0.021135250106453896,
+            0.0029641420114785433,
+            0.0074690403416752815,
+            0.01937682554125786,
+            0.005283812526613474,
+            0.0044302078895270824,
+            0.004472888074815273,
+            -0.0020977333188056946,
+            -0.01002131775021553,
+            0.004835669882595539,
+            0.004472888074815273,
+            -0.0012729378649964929,
+            0.01953047327697277,
+            0.008126315660774708,
+            0.008736643008887768,
+            -0.003049502382054925,
+            0.002296196296811104,
+            0.006598363630473614,
+            0.018454931676387787,
+            0.0014372568111866713,
+            -0.026171516627073288,
+            0.008011079393327236,
+            0.009867669083178043,
+            -0.004639341030269861,
+            -0.004054621793329716,
+            0.003964993637055159,
+            -0.003877498907968402,
+            -0.004150652326643467,
+            -0.003719581989571452,
+            0.021203538402915,
+            0.022313224151730537,
+            0.02017921209335327,
+            0.003395212348550558,
+            0.012377266772091389,
+            0.0010584697593003511,
+            -0.02750314027070999,
+            0.007648297119885683,
+            -0.007055042311549187,
+            0.01418690849095583,
+            -0.0359196811914444,
+            -0.00670933211222291,
+            -0.004733237437903881,
+            0.001975027611479163,
+            0.03120778501033783,
+            0.0021478827111423016,
+            0.006564219482243061,
+            0.00552282202988863,
+            0.011959000490605831,
+            -0.004310703370720148,
+            0.024378947913646698,
+            -0.002678184537217021,
+            0.004737505689263344,
+            -0.002415701048448682,
+            -0.01084077823907137,
+            -0.020110923796892166,
+            0.01110539585351944,
+            0.008689695037901402,
+            0.0616643987596035,
+            -0.006513003259897232,
+            0.023047324270009995,
+            -0.01373449806123972,
+            0.007234299089759588,
+            0.0218522772192955,
+            0.024447236210107803,
+            0.02748606912791729,
+            -0.0010691398056223989,
+            0.004707629326730967,
+            -0.014647855423390865,
+            -0.018523219972848892,
+            0.009585979394614697,
+            -0.007242835126817226,
+            0.00597523245960474,
+            -0.022415656596422195,
+            -0.02016214095056057,
+            -0.008988456800580025,
+            -0.0057532950304448605,
+            -0.001239860663190484,
+            0.004618000704795122,
+            0.005309420637786388,
+            -0.0034549646079540253,
+            0.025864219292998314,
+            -0.0005332361324690282,
+            -0.0033141199965029955,
+            -0.028595754876732826,
+            -0.009184785187244415,
+            0.011890712194144726,
+            -0.05049924924969673,
+            0.018779302015900612,
+            -0.00272940075956285,
+            -0.02748606912791729,
+            -0.01744767837226391,
+            -0.0035595311783254147,
+            0.015629500150680542,
+            -0.012334587052464485,
+            0.010610305704176426,
+            0.0026205661706626415,
+            -0.0006882187444716692,
+            0.0037772003561258316,
+            0.014545422978699207,
+            0.0224839448928833,
+            0.004370455630123615,
+            0.009329898282885551,
+            0.005556966178119183,
+            0.009978638030588627,
+            -0.007580008823424578,
+            0.019974347203969955,
+            0.001809641718864441,
+            0.0045326403342187405,
+            0.0020337130408734083,
+            -0.016961123794317245,
+            -0.004536908585578203,
+            -0.006017912644892931,
+            0.028083590790629387,
+            0.015339274890720844,
+            -0.01041397638618946,
+            -0.005347833037376404,
+            -0.005569770000874996,
+            -0.011924856342375278,
+            0.0013028139946982265,
+            0.013512561097741127,
+            0.010448120534420013,
+            -0.038856081664562225,
+            0.03317107632756233,
+            0.01728549413383007,
+            -0.01848907582461834,
+            -0.024208227172493935,
+            -0.002460515359416604,
+            0.020708447322249413,
+            0.023832641541957855,
+            -0.03271012753248215,
+            -0.013427200727164745,
+            0.014468598179519176,
+            -0.014383237808942795,
+            6.825502350693569e-05,
+            -0.0005495079676620662,
+            -0.0205547995865345,
+            0.011250508949160576,
+            -0.014810039661824703,
+            0.019291464239358902,
+            -0.0025928239338099957,
+            0.022518089041113853,
+            0.025727642700076103,
+            -0.004528372548520565,
+            0.0018352499464526772,
+            -0.01355524081736803,
+            -0.008207408711314201,
+            0.03277841582894325,
+            0.0018053737003356218,
+            -0.005428925156593323,
+            -0.007917182520031929,
+            -0.0023986289743334055,
+            -0.012676028534770012,
+            0.01490393653512001,
+            0.007349535822868347,
+            0.0012035825056955218,
+            0.005241132341325283,
+            -0.0010878123575821519,
+            0.025215480476617813,
+            0.004763113800436258,
+            -0.004763113800436258,
+            0.008834808133542538,
+            -0.015108801424503326,
+            0.01937682554125786,
+            -0.0029022556263953447,
+            -0.008361057378351688,
+            0.0053179566748440266,
+            0.014656390994787216,
+            0.004792989697307348,
+            -0.0038518907967954874,
+            -0.010482264682650566,
+            -0.004857010208070278,
+            0.007669637445360422,
+            0.027059266343712807,
+            0.014519814401865005,
+            -0.007597080897539854,
+            -0.011412693187594414,
+            0.01451127789914608,
+            0.007883038371801376,
+            0.0006924867630004883,
+            -0.0071788146160542965,
+            0.012095577083528042,
+            -0.01554413978010416,
+            0.014681999571621418,
+            -0.009346970357000828,
+            -0.020862096920609474,
+            0.008988456800580025,
+            -0.02506183087825775,
+            0.0017125442391261458,
+            -0.01835249923169613,
+            0.002155351685360074,
+            -0.007968398742377758,
+            -0.011395621113479137,
+            0.009500619024038315,
+            -0.000326237000990659,
+            0.03878779336810112,
+            0.011190756224095821,
+            0.009679876267910004,
+            -0.0014489938039332628,
+            0.07416117191314697,
+            -0.023439982905983925,
+            -0.01033715158700943,
+            -0.028954267501831055,
+            -0.012949181720614433,
+            -0.0063636223785579205,
+            0.01002131775021553,
+            0.008104975335299969,
+            -0.0004057289334014058,
+            -0.011301725171506405,
+            -0.007072114385664463,
+            0.015783149749040604,
+            -0.026615392416715622,
+            -0.0007479710620827973,
+            0.02159619703888893,
+            0.0030729766003787518,
+            0.006564219482243061,
+            0.015006368979811668,
+            -0.007268443237990141,
+            0.01966704986989498,
+            -0.016346529126167297,
+            -0.012744316831231117,
+            0.005894139874726534,
+            0.016329456120729446,
+            0.005339297000318766,
+            0.0008904163260012865,
+            0.011848031543195248,
+            -0.0006097938166931272,
+            0.005108823534101248,
+            0.02323511801660061,
+            -0.02014506794512272,
+            0.014417381957173347,
+            -0.008412273600697517,
+            0.01825006678700447,
+            0.007644029334187508,
+            0.018028128892183304,
+            -0.005134431645274162,
+            -0.008877487853169441,
+            -0.008442149497568607,
+            0.010943210683763027,
+            0.015962406992912292,
+            -0.012215081602334976,
+            -0.0019686257001012564,
+            0.004048219881951809,
+            -0.018028128892183304,
+            -0.006897125393152237,
+            -0.012180937454104424,
+            0.008049491792917252,
+            0.021510835736989975,
+            -0.012471163645386696,
+            -0.01675625890493393,
+            -0.022176647558808327,
+            0.012317514047026634,
+            -0.008190336637198925,
+            0.02772507816553116,
+            -0.013623529113829136,
+            -0.00523686408996582,
+            0.0211864672601223,
+            0.0004182662523817271,
+            0.0012174536241218448,
+            -0.027776293456554413,
+            -0.010166430845856667,
+            -0.014997833408415318
+          ],
+          "title": "6 Related Work"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "353d94c1-6a72-444c-9154-fb488da0e893",
+      "type": "next",
+      "source": {
+        "id": "9204ecdb-832f-4b40-b0fe-5cc34ec75d3b",
+        "properties": {
+          "page_content": "6 Related Work 24 ",
+          "keyphrases": [
+            "Related Work"
+          ],
+          "embedding": [
+            0.002109470311552286,
+            0.024481380358338356,
+            -0.01675625890493393,
+            0.01014935877174139,
+            0.03144679218530655,
+            0.0017690955428406596,
+            0.0074690403416752815,
+            0.09587687253952026,
+            0.01939389668405056,
+            0.014093012548983097,
+            0.001160902320407331,
+            -0.013068686239421368,
+            -0.015330738388001919,
+            -0.026103228330612183,
+            0.03235161304473877,
+            0.036431845277547836,
+            -0.005620986223220825,
+            -0.00097150873625651,
+            -0.033068642020225525,
+            0.002234310144558549,
+            0.0004380058671813458,
+            0.024430163204669952,
+            -0.015834365040063858,
+            0.04742627218365669,
+            0.02104988880455494,
+            0.008480561897158623,
+            0.051182132214307785,
+            0.01811349019408226,
+            -0.03865121677517891,
+            0.0019728937186300755,
+            -0.008903096430003643,
+            0.010678594000637531,
+            8.36265753605403e-05,
+            -0.0034208204597234726,
+            -0.028698187321424484,
+            -0.018420787528157234,
+            0.014630783349275589,
+            0.01811349019408226,
+            -0.0005980567075312138,
+            0.012718708254396915,
+            -0.036909863352775574,
+            -0.027776293456554413,
+            -0.030541973188519478,
+            0.0029363997746258974,
+            0.03361494839191437,
+            -0.029739584773778915,
+            -0.01758425496518612,
+            -0.044626448303461075,
+            0.005258204415440559,
+            0.03020053170621395,
+            0.029500575736165047,
+            -0.023098541423678398,
+            0.01559535600244999,
+            -0.0012782729463651776,
+            -0.01341866422444582,
+            0.003841220634058118,
+            -0.0015738335205242038,
+            0.07300026714801788,
+            0.023798497393727303,
+            0.019069526344537735,
+            0.001734951394610107,
+            -0.016662361100316048,
+            -0.0010467326501384377,
+            -0.03286377713084221,
+            0.009978638030588627,
+            -0.01825006678700447,
+            0.018045201897621155,
+            -0.005023463163524866,
+            0.020264573395252228,
+            -0.008489097468554974,
+            -0.0041101062670350075,
+            -0.007831822149455547,
+            -0.026905616745352745,
+            0.05551844462752342,
+            -0.0061459532007575035,
+            0.00418479647487402,
+            0.042850952595472336,
+            0.027008049190044403,
+            -0.015689251944422722,
+            0.04995294287800789,
+            -0.005655130371451378,
+            -0.024703318253159523,
+            -0.012863821350038052,
+            -0.01522830594331026,
+            0.019701194018125534,
+            -0.005778903141617775,
+            -0.056850068271160126,
+            -0.13206970691680908,
+            0.004724701400846243,
+            -0.0011918453965336084,
+            0.011318797245621681,
+            0.014229589141905308,
+            -0.002349546644836664,
+            -0.03841220587491989,
+            -0.051182132214307785,
+            0.04595807194709778,
+            -0.008531778119504452,
+            -0.021391332149505615,
+            0.009551835246384144,
+            -0.01522830594331026,
+            0.008096439763903618,
+            0.016414817422628403,
+            0.0179598405957222,
+            -0.03494657203555107,
+            0.02656417526304722,
+            -0.03824148699641228,
+            -0.005292348563671112,
+            -0.00846775807440281,
+            0.026291022077202797,
+            0.03970968723297119,
+            -0.02081087976694107,
+            -0.025249624624848366,
+            -0.033444229513406754,
+            0.004285095259547234,
+            -0.0017040082020685077,
+            -0.029995666816830635,
+            0.011079788208007812,
+            -0.046162936836481094,
+            -0.021203538402915,
+            0.012360194697976112,
+            0.020486511290073395,
+            -0.0011032839538529515,
+            0.004673485178500414,
+            -0.02465210109949112,
+            -0.02274002693593502,
+            0.021664485335350037,
+            0.003265037667006254,
+            -0.01618434302508831,
+            -0.01535634696483612,
+            0.0282543133944273,
+            0.03605625778436661,
+            -0.00591121194884181,
+            0.0010622042464092374,
+            -0.03387103229761124,
+            0.01354670524597168,
+            -0.018574437126517296,
+            0.011933392845094204,
+            0.03660256415605545,
+            -0.022022999823093414,
+            -0.004733237437903881,
+            0.005288080777972937,
+            -0.019359752535820007,
+            -0.0030473684892058372,
+            0.022415656596422195,
+            -0.007490380667150021,
+            0.014912472106516361,
+            -0.03055904433131218,
+            0.012991862371563911,
+            -0.017789119854569435,
+            0.04548005387187004,
+            -0.03424661606550217,
+            0.017319638282060623,
+            -0.018284210935235023,
+            0.032027244567871094,
+            -0.019735338166356087,
+            -0.0157319325953722,
+            0.009287218563258648,
+            0.0017146782483905554,
+            0.016355063766241074,
+            -0.02403750643134117,
+            0.04097302258014679,
+            0.023320477455854416,
+            0.024447236210107803,
+            -0.01644042506814003,
+            0.021630341187119484,
+            -0.03378567099571228,
+            0.015211233869194984,
+            -0.014681999571621418,
+            -0.017908625304698944,
+            0.026512959972023964,
+            0.02734949253499508,
+            0.0033183880150318146,
+            0.018420787528157234,
+            0.0003475771227385849,
+            -0.026683680713176727,
+            -0.007976935245096684,
+            -0.007575741037726402,
+            0.03168580308556557,
+            -0.013956435024738312,
+            0.003331192070618272,
+            -0.021288899704813957,
+            0.03762689232826233,
+            -0.0013903084909543395,
+            -0.00194301747251302,
+            0.011071251705288887,
+            -0.013393056578934193,
+            0.029381070286035538,
+            -0.030132243409752846,
+            0.041314464062452316,
+            0.024378947913646698,
+            -0.0024221031926572323,
+            -0.019052455201745033,
+            0.01622702367603779,
+            0.012752852402627468,
+            0.005198452156037092,
+            -0.057737816125154495,
+            0.011199292726814747,
+            -0.006252653896808624,
+            -0.03568067401647568,
+            -0.021220611408352852,
+            -0.00606912886723876,
+            -0.022057143971323967,
+            0.007443432230502367,
+            0.004575320519506931,
+            -0.025847148150205612,
+            -0.009201857261359692,
+            -0.0024583812337368727,
+            0.002289794385433197,
+            -0.008659819141030312,
+            -0.03516850993037224,
+            0.00754159688949585,
+            0.0003875898546539247,
+            0.0006898192223161459,
+            -0.016406280919909477,
+            -0.0011064850259572268,
+            -0.023559486493468285,
+            -0.0224839448928833,
+            0.06524953991174698,
+            -0.0035787373781204224,
+            -0.0295347198843956,
+            0.03892436996102333,
+            -0.0030708424746990204,
+            0.007925719022750854,
+            0.008809199556708336,
+            -0.02647881582379341,
+            -0.023439982905983925,
+            -0.0022663201671093702,
+            0.05275276303291321,
+            0.0034955109003931284,
+            -0.01458810269832611,
+            0.0006594095611944795,
+            0.008292769081890583,
+            -0.004041817970573902,
+            -0.052616186439991,
+            -0.0314980112016201,
+            -0.01964997872710228,
+            -0.00800254289060831,
+            -0.030354179441928864,
+            -0.012300441972911358,
+            0.02801530249416828,
+            0.030763909220695496,
+            -0.0016613280167803168,
+            -0.042953383177518845,
+            0.017123308032751083,
+            -0.02466917410492897,
+            -0.020349934697151184,
+            -0.03163458779454231,
+            0.05312835052609444,
+            0.019871914759278297,
+            -0.02620566077530384,
+            0.0058045112527906895,
+            0.012616275809705257,
+            -0.009048208594322205,
+            0.02079380862414837,
+            0.019342679530382156,
+            0.09280389547348022,
+            0.026666607707738876,
+            -0.0018149768002331257,
+            -0.0026653804816305637,
+            -0.015211233869194984,
+            -0.015194161795079708,
+            -0.01675625890493393,
+            -0.04561663046479225,
+            -0.021783988922834396,
+            0.005377708934247494,
+            -0.0013038810575380921,
+            0.039368245750665665,
+            0.0044302078895270824,
+            0.022415656596422195,
+            -0.01001278217881918,
+            0.024686245247721672,
+            -0.013145511038601398,
+            -0.002503195544704795,
+            -0.02622273378074169,
+            -0.009603051468729973,
+            -0.030115170404314995,
+            -0.003077244618907571,
+            -0.015142945572733879,
+            0.016372136771678925,
+            -0.004588124807924032,
+            -0.009893277660012245,
+            0.04466059431433678,
+            0.008937240578234196,
+            -0.018984166905283928,
+            -0.021527908742427826,
+            -0.03557823970913887,
+            0.009449402801692486,
+            -0.0026995246298611164,
+            -0.0031199248041957617,
+            -0.019564617425203323,
+            0.01213825773447752,
+            -0.027912870049476624,
+            -0.040529146790504456,
+            0.044899601489305496,
+            0.006828837096691132,
+            -0.025198407471179962,
+            -0.01381132286041975,
+            0.006905661430209875,
+            -0.0014351228019222617,
+            -0.026410527527332306,
+            -0.001253731781616807,
+            0.012838213704526424,
+            -0.00121211854275316,
+            0.007302587386220694,
+            -0.019718267023563385,
+            0.008331180550158024,
+            0.02106696180999279,
+            0.03177116438746452,
+            -0.029978593811392784,
+            -0.026871472597122192,
+            0.01149805448949337,
+            -0.04868960753083229,
+            -0.0007074248278513551,
+            -0.04063158109784126,
+            0.012069969438016415,
+            0.024430163204669952,
+            0.005902675911784172,
+            0.021732773631811142,
+            -0.02874940261244774,
+            -0.04612879455089569,
+            0.0013113500317558646,
+            0.01078102644532919,
+            -0.04353383556008339,
+            0.024481380358338356,
+            -0.014724679291248322,
+            0.022296153008937836,
+            -0.003619283437728882,
+            0.04059743508696556,
+            0.01580875739455223,
+            -0.010763954371213913,
+            0.03006395511329174,
+            -0.00937257893383503,
+            -6.443714482884388e-06,
+            0.018967093899846077,
+            -0.011941928416490555,
+            -0.03156629949808121,
+            -0.03557823970913887,
+            0.015475851483643055,
+            0.007567205000668764,
+            -0.017823264002799988,
+            0.009662804193794727,
+            0.01554413978010416,
+            0.007511720526963472,
+            0.02451552450656891,
+            0.019086599349975586,
+            -0.05367465689778328,
+            0.035373374819755554,
+            -0.04029013589024544,
+            -0.012923574075102806,
+            0.007780605927109718,
+            0.0009704417316243052,
+            -0.03554409742355347,
+            -0.0024370411410927773,
+            0.012104113586246967,
+            -0.014238124713301659,
+            -0.047221407294273376,
+            -0.017703760415315628,
+            -0.029244493693113327,
+            0.020076779648661613,
+            -0.021801061928272247,
+            0.00022700548288412392,
+            -0.04022184759378433,
+            -0.009125033393502235,
+            -0.016022158786654472,
+            -0.05278690904378891,
+            -0.036670852452516556,
+            0.024464307352900505,
+            -0.005795975215733051,
+            -0.02145962044596672,
+            -0.0009016198455356061,
+            -0.059410881251096725,
+            0.007989739067852497,
+            0.0016175807686522603,
+            -0.006777620408684015,
+            -0.025095975026488304,
+            -0.011148076504468918,
+            0.014571030624210835,
+            0.017635472118854523,
+            -0.0397779755294323,
+            0.005680738482624292,
+            -0.01373449806123972,
+            0.01014935877174139,
+            0.005706346593797207,
+            -0.017635472118854523,
+            -0.024088721722364426,
+            0.0036897058598697186,
+            -0.0037665304262191057,
+            0.01168584730476141,
+            0.0030644405633211136,
+            -0.010678594000637531,
+            0.02480575069785118,
+            -0.04520690068602562,
+            0.003469902789220214,
+            -0.03632941097021103,
+            0.039880406111478806,
+            -0.006495931185781956,
+            0.02992737665772438,
+            0.015100265853106976,
+            -0.0030793785117566586,
+            -0.005181380081921816,
+            -0.0501236617565155,
+            0.005595378112047911,
+            0.005138699896633625,
+            0.006700796075165272,
+            0.04435329511761665,
+            -0.003719581989571452,
+            -0.017925696447491646,
+            0.01232605054974556,
+            0.016628216952085495,
+            -0.011643166653811932,
+            0.03168580308556557,
+            0.015023441053926945,
+            -0.000559644540771842,
+            -0.011216364800930023,
+            0.03574896231293678,
+            -0.002466917270794511,
+            0.009150641039013863,
+            0.061220522969961166,
+            -0.023542415350675583,
+            -0.010311543941497803,
+            -0.004592392593622208,
+            -0.008732374757528305,
+            0.07368315011262894,
+            -0.009611587971448898,
+            -0.008472025394439697,
+            0.0179598405957222,
+            -0.015663644298911095,
+            -0.011916320770978928,
+            0.01600508578121662,
+            -0.02569349855184555,
+            -0.02169862948358059,
+            0.012505307793617249,
+            0.026273949071764946,
+            -0.021101105958223343,
+            0.028032375499606133,
+            -0.015689251944422722,
+            -0.011139540001749992,
+            0.01201021671295166,
+            -0.016329456120729446,
+            -0.012300441972911358,
+            0.035407520830631256,
+            -0.004592392593622208,
+            0.03366616740822792,
+            0.03660256415605545,
+            0.006192901637405157,
+            -0.03131021559238434,
+            -0.04548005387187004,
+            -0.010670057497918606,
+            -0.03480999544262886,
+            0.017413534224033356,
+            -0.0003048969083465636,
+            0.013691818341612816,
+            -0.04469473659992218,
+            -0.00844641774892807,
+            0.05510871484875679,
+            -0.02209128811955452,
+            0.029090845957398415,
+            0.04343140125274658,
+            -0.03185652568936348,
+            -0.02412286587059498,
+            -0.021510835736989975,
+            0.08549703657627106,
+            0.040665723383426666,
+            -0.010055461898446083,
+            -0.0359879694879055,
+            0.01592826284468174,
+            -0.026666607707738876,
+            -0.025078903883695602,
+            2.39409419009462e-05,
+            0.04339725896716118,
+            0.00754159688949585,
+            -0.015527067705988884,
+            -0.017908625304698944,
+            0.01937682554125786,
+            -0.033853959292173386,
+            0.001105417963117361,
+            0.019991420209407806,
+            0.008087903261184692,
+            -0.04510446637868881,
+            0.012069969438016415,
+            0.017891552299261093,
+            0.02031579054892063,
+            -0.02504475973546505,
+            0.015322202816605568,
+            0.026512959972023964,
+            0.027451924979686737,
+            0.027947014197707176,
+            -0.005676470696926117,
+            0.015339274890720844,
+            -0.006965413689613342,
+            0.03865121677517891,
+            -0.0018619250040501356,
+            -0.007840358652174473,
+            0.01695258729159832,
+            -0.029483502730727196,
+            -0.04005112871527672,
+            -0.017080627381801605,
+            -0.07129305601119995,
+            0.015834365040063858,
+            0.027178769931197166,
+            -0.014272268861532211,
+            -0.00818180013448,
+            -0.008058027364313602,
+            0.0450020357966423,
+            0.010251791216433048,
+            -0.021869350224733353,
+            0.01799398474395275,
+            -0.018950022757053375,
+            -0.01213825773447752,
+            0.013008934445679188,
+            0.028220169246196747,
+            0.016090447083115578,
+            0.019479257985949516,
+            -0.0237472802400589,
+            0.003971395548433065,
+            0.006991021800786257,
+            0.012658956460654736,
+            -0.0007340999436564744,
+            -0.01065298542380333,
+            0.030115170404314995,
+            0.0077379257418215275,
+            0.0027976890560239553,
+            0.001323087140917778,
+            0.04855303093791008,
+            -0.03187359496951103,
+            0.008971384726464748,
+            -0.010584697127342224,
+            -0.02236444130539894,
+            0.05346979200839996,
+            -0.003363202093169093,
+            -0.012744316831231117,
+            0.031259000301361084,
+            0.027588501572608948,
+            -0.013316231779754162,
+            -0.0118821756914258,
+            -0.0011043510166928172,
+            -0.027008049190044403,
+            0.020725520327687263,
+            -0.02067430317401886,
+            -0.0068587129935622215,
+            0.01007253397256136,
+            0.016824547201395035,
+            0.011660238727927208,
+            -0.017754975706338882,
+            -0.011848031543195248,
+            0.05125042051076889,
+            0.041280318051576614,
+            -0.010883458890020847,
+            0.007140402682125568,
+            -0.0028446372598409653,
+            0.014161300845444202,
+            0.028442105278372765,
+            0.006534343119710684,
+            -0.003913776949048042,
+            -0.002244980074465275,
+            -0.010951747186481953,
+            -0.03236868605017662,
+            0.019957276061177254,
+            0.025864219292998314,
+            0.012223618105053902,
+            0.028100663796067238,
+            -0.03779761120676994,
+            -0.00943233072757721,
+            -0.0053819771856069565,
+            0.006555683445185423,
+            -0.04670924320816994,
+            0.014101548120379448,
+            0.021544979885220528,
+            -0.03520265221595764,
+            -0.021408403292298317,
+            -0.0025330716744065285,
+            0.011259044520556927,
+            -0.03096877597272396,
+            -0.035509951412677765,
+            -0.032163821160793304,
+            0.04742627218365669,
+            -0.0021820268593728542,
+            -0.0063508180901408195,
+            -0.04059743508696556,
+            -0.01738792657852173,
+            0.015646573156118393,
+            -0.02429358661174774,
+            -0.006760548334568739,
+            0.025744715705513954,
+            -0.006133148912340403,
+            0.006167293526232243,
+            -0.024447236210107803,
+            -0.015262450091540813,
+            -0.03622698038816452,
+            0.02878354676067829,
+            0.012846749275922775,
+            0.0025800198782235384,
+            -0.01966704986989498,
+            -0.024327730759978294,
+            -0.025539850816130638,
+            0.004728969652205706,
+            0.010891994461417198,
+            0.0009005528409034014,
+            0.0289030522108078,
+            -0.05428925156593323,
+            0.030729765072464943,
+            0.006235581822693348,
+            -0.026239804923534393,
+            0.006389230489730835,
+            0.010644449852406979,
+            -0.023030253127217293,
+            0.003160471096634865,
+            0.01677333004772663,
+            -0.009346970357000828,
+            0.011446838267147541,
+            0.02555692195892334,
+            -0.014093012548983097,
+            0.048484742641448975,
+            -0.004524104297161102,
+            0.009918885305523872,
+            -0.025915436446666718,
+            0.041690047830343246,
+            0.008587262593209743,
+            -0.0009405656019225717,
+            0.0015044781612232327,
+            0.03892436996102333,
+            -0.01886466145515442,
+            -0.0224839448928833,
+            -0.0112334368750453,
+            0.013828394934535027,
+            -0.033973462879657745,
+            -0.008886024355888367,
+            -0.007848894223570824,
+            -0.017439141869544983,
+            -0.011933392845094204,
+            0.005783171392977238,
+            0.010746882297098637,
+            0.005492945667356253,
+            0.010635913349688053,
+            0.018011057749390602,
+            0.008143387734889984,
+            0.038104910403490067,
+            -0.009346970357000828,
+            0.03687571734189987,
+            -0.012360194697976112,
+            -0.01600508578121662,
+            -0.016858691349625587,
+            0.019223175942897797,
+            0.041177887469530106,
+            -0.040255993604660034,
+            0.029568864032626152,
+            -0.014442989602684975,
+            0.0147673599421978,
+            0.005253936164081097,
+            0.01838664337992668,
+            0.011165148578584194,
+            -0.003881766926497221,
+            -0.0050405352376401424,
+            0.022893676534295082,
+            0.011020035482943058,
+            -0.00015231507131829858,
+            -0.0012836079113185406,
+            -0.02425944246351719,
+            -0.0474945604801178,
+            0.020127996802330017,
+            -0.012940646149218082,
+            -0.022808315232396126,
+            -0.005467337556183338,
+            0.010490800254046917,
+            0.005224060267210007,
+            0.029722511768341064,
+            0.015211233869194984,
+            0.005224060267210007,
+            -0.014442989602684975,
+            0.006577023304998875,
+            -0.004024745896458626,
+            -0.035885538905858994,
+            0.0050917514599859715,
+            0.020503582432866096,
+            -0.03131021559238434,
+            -0.018403714522719383,
+            -0.004159188363701105,
+            0.03337594121694565,
+            -0.0031113887671381235,
+            0.004270156845450401,
+            -0.016568465158343315,
+            -0.004618000704795122,
+            -0.021118177101016045,
+            0.017217205837368965,
+            0.002994018141180277,
+            0.01503197755664587,
+            -0.010542016476392746,
+            0.0003016958653461188,
+            0.006623971741646528,
+            -0.00018725951667875051,
+            -0.0041762604378163815,
+            0.005744758993387222,
+            -0.006700796075165272,
+            -0.0016666629817336798,
+            0.0013561643427237868,
+            0.010695666074752808,
+            -0.014272268861532211,
+            0.00950915552675724,
+            0.007405019830912352,
+            0.013512561097741127,
+            -0.02799823135137558,
+            -0.025761786848306656,
+            -0.055689163506031036,
+            0.01675625890493393,
+            -0.009355506859719753,
+            -0.011284653097391129,
+            -0.006585559342056513,
+            -0.006081932689994574,
+            -0.025130119174718857,
+            -0.014391773380339146,
+            -0.014622246846556664,
+            -0.011122467927634716,
+            0.010328616015613079,
+            -0.0007314324611797929,
+            -0.0015524934278801084,
+            0.01744767837226391,
+            0.002784885000437498,
+            -0.022569306194782257,
+            -0.01746474951505661,
+            0.013589384965598583,
+            -0.01876222901046276,
+            -0.0006722136167809367,
+            -0.034588057547807693,
+            -0.04923591390252113,
+            -0.019308535382151604,
+            0.006141684949398041,
+            0.026530031114816666,
+            -0.0002678184537217021,
+            -0.004404599778354168,
+            -0.003990601748228073,
+            0.005714882630854845,
+            -0.01118222065269947,
+            -0.000805055839009583,
+            0.005859995726495981,
+            0.0036342216189950705,
+            0.020264573395252228,
+            0.039368245750665665,
+            0.018523219972848892,
+            0.0006119278259575367,
+            -0.016337992623448372,
+            0.012547987513244152,
+            0.016269704326987267,
+            -0.006116076838225126,
+            0.015919726341962814,
+            0.00982498936355114,
+            -0.004507032223045826,
+            0.046060506254434586,
+            0.029227422550320625,
+            -0.04349968954920769,
+            0.009517691098153591,
+            0.014622246846556664,
+            -0.0250276867300272,
+            0.03404175117611885,
+            -0.017618399113416672,
+            -0.02028164640069008,
+            0.0032500994857400656,
+            0.021101105958223343,
+            -0.0002234043349744752,
+            -0.015783149749040604,
+            0.022022999823093414,
+            -0.01002131775021553,
+            -0.0014329887926578522,
+            0.007601349148899317,
+            0.026530031114816666,
+            -0.0011822424130514264,
+            -0.03356373310089111,
+            -0.002336742589250207,
+            -0.009475011378526688,
+            0.004985050763934851,
+            0.011224900372326374,
+            0.01811349019408226,
+            0.002705926541239023,
+            0.009398186579346657,
+            -0.0231156125664711,
+            0.012573596090078354,
+            -0.02337169460952282,
+            -0.008019614964723587,
+            -0.02400336228311062,
+            -0.011626094579696655,
+            0.01450274232774973,
+            0.006141684949398041,
+            0.01657700166106224,
+            -0.02053772658109665,
+            -0.009483546949923038,
+            -0.014280805364251137,
+            0.024447236210107803,
+            0.012419947423040867,
+            -0.02760557271540165,
+            -0.016585538163781166,
+            -0.012437019497156143,
+            -0.015211233869194984,
+            0.012437019497156143,
+            -0.01966704986989498,
+            0.02053772658109665,
+            0.04817744344472885,
+            -0.03568067401647568,
+            -0.011702919378876686,
+            0.03387103229761124,
+            -0.004997855052351952,
+            -0.013136974535882473,
+            0.02991030551493168,
+            0.01027739979326725,
+            -0.01014935877174139,
+            0.005172844044864178,
+            0.012633347883820534,
+            0.01400765124708414,
+            -0.020759664475917816,
+            -0.020998673513531685,
+            -0.019735338166356087,
+            -0.0031498009338974953,
+            -0.007951326668262482,
+            -0.02965422347187996,
+            0.017498893663287163,
+            0.0166794341057539,
+            -0.004447279963642359,
+            0.01650017686188221,
+            -0.028066519647836685,
+            -0.01457956712692976,
+            -0.01142976526170969,
+            -0.01387107465416193,
+            0.0314980112016201,
+            -0.0068587129935622215,
+            0.05749880522489548,
+            -0.006939805578440428,
+            0.00023194037203211337,
+            0.061083946377038956,
+            0.011404157616198063,
+            -0.0015354212373495102,
+            0.037456169724464417,
+            0.027434851974248886,
+            0.0093213627114892,
+            0.018215922638773918,
+            -0.01773790456354618,
+            0.013538168743252754,
+            -0.02081087976694107,
+            0.00026995246298611164,
+            -0.0068074967712163925,
+            0.016133127734065056,
+            -0.0385146401822567,
+            0.0359538272023201,
+            0.017200132831931114,
+            -0.002206567907705903,
+            0.01142976526170969,
+            -0.023303406313061714,
+            -0.010303007438778877,
+            0.016022158786654472,
+            -0.02197178266942501,
+            0.015424635261297226,
+            -0.004131446126848459,
+            -0.004946638830006123,
+            0.024720389395952225,
+            0.033819813281297684,
+            -0.016978194937109947,
+            -0.009893277660012245,
+            -0.042338788509368896,
+            0.0023068664595484734,
+            -0.018335426226258278,
+            0.02258637733757496,
+            0.010354223661124706,
+            -0.013854002580046654,
+            0.0009037538548000157,
+            -0.009227465838193893,
+            -0.016431888565421104,
+            0.004865546245127916,
+            0.008408005349338055,
+            -0.02492525428533554,
+            -0.039641398936510086,
+            -0.011404157616198063,
+            -0.012932109646499157,
+            -0.013136974535882473,
+            -0.016978194937109947,
+            0.0186427254229784,
+            0.0073836799710989,
+            0.011796815320849419,
+            0.009893277660012245,
+            -0.010610305704176426,
+            -0.019479257985949516,
+            0.011515126563608646,
+            0.005919747985899448,
+            0.0023900929372757673,
+            -0.026291022077202797,
+            0.003915911074727774,
+            -0.029585935175418854,
+            0.003687571967020631,
+            -0.010874922387301922,
+            0.03375152498483658,
+            -0.0027507408522069454,
+            -0.029363999143242836,
+            -0.026154445484280586,
+            -0.02492525428533554,
+            0.016884298995137215,
+            -0.010243255645036697,
+            0.00846775807440281,
+            -0.029961522668600082,
+            0.019103670492768288,
+            -0.008937240578234196,
+            -0.027025122195482254,
+            0.00028835830744355917,
+            0.004193332511931658,
+            -0.010874922387301922,
+            -0.021493764594197273,
+            -0.011327332817018032,
+            -0.011941928416490555,
+            -0.052343033254146576,
+            0.0030366983264684677,
+            -0.009389651007950306,
+            0.008766518905758858,
+            -0.0109176030382514,
+            0.00855311844497919,
+            -0.008681158535182476,
+            -0.004142116289585829,
+            -0.021510835736989975,
+            -0.014485670253634453,
+            -0.022296153008937836,
+            0.033068642020225525,
+            0.017353782430291176,
+            -0.023593630641698837,
+            -0.006713600363582373,
+            -0.008689695037901402,
+            -0.004310703370720148,
+            0.0166965052485466,
+            0.012923574075102806,
+            -0.02593250758945942,
+            0.0029022556263953447,
+            0.023474127054214478,
+            -0.003399480367079377,
+            -0.008548850193619728,
+            0.02903962880373001,
+            0.005518553778529167,
+            0.009142105467617512,
+            0.004101570229977369,
+            0.025232551619410515,
+            -0.005347833037376404,
+            0.004575320519506931,
+            0.054903849959373474,
+            0.0026355041190981865,
+            -0.005667934659868479,
+            0.03786589950323105,
+            0.0004075962060596794,
+            -0.01548438798636198,
+            0.024361874908208847,
+            0.012249225750565529,
+            0.025352057069540024,
+            0.013444272801280022,
+            -0.001758425496518612,
+            0.002910791663452983,
+            -0.00014244526391848922,
+            0.03416125476360321,
+            0.0029684100300073624,
+            0.01600508578121662,
+            -0.03947067633271217,
+            0.02195470966398716,
+            0.015689251944422722,
+            -0.0014649989316239953,
+            0.016944050788879395,
+            -0.004024745896458626,
+            -0.029620079323649406,
+            0.0031391310039907694,
+            0.0011331600835546851,
+            0.02390092983841896,
+            0.024617956951260567,
+            0.008343985304236412,
+            -0.019445113837718964,
+            -0.007823286578059196,
+            -0.002910791663452983,
+            0.05971817672252655,
+            0.011856568045914173,
+            0.03136143460869789,
+            -0.018693940714001656,
+            -0.00846775807440281,
+            -0.02106696180999279,
+            0.0021809597965329885,
+            0.025659354403614998,
+            -0.0029769460670650005,
+            -0.01540756318718195,
+            -0.010798098519444466,
+            -0.001411648583598435,
+            -0.004767381586134434,
+            -0.001800038735382259,
+            0.009142105467617512,
+            0.02079380862414837,
+            -0.013563777320086956,
+            0.01059323363006115,
+            0.028851835057139397,
+            -0.007784874178469181,
+            0.01915488764643669,
+            -0.027264131233096123,
+            -0.02746899612247944,
+            -0.010046926327049732,
+            0.012505307793617249,
+            0.009082352742552757,
+            0.013119902461767197,
+            0.020059708505868912,
+            0.008642747066915035,
+            0.008809199556708336,
+            0.014946616254746914,
+            0.015697788447141647,
+            -0.014084476046264172,
+            0.04698239639401436,
+            -0.00393298314884305,
+            -0.002027310896664858,
+            -0.010584697127342224,
+            -0.01065298542380333,
+            0.012266297824680805,
+            -0.0007298319251276553,
+            0.013478416949510574,
+            0.013085758313536644,
+            -0.02326926216483116,
+            0.01937682554125786,
+            0.00982498936355114,
+            0.0006257988861761987,
+            -0.0007986538112163544,
+            0.0166794341057539,
+            -0.013290624134242535,
+            0.026120301336050034,
+            -0.005949624348431826,
+            -0.01912074349820614,
+            0.010303007438778877,
+            0.01710623688995838,
+            0.0028659775853157043,
+            -0.02579593099653721,
+            -0.029995666816830635,
+            -0.0031049868557602167,
+            -0.03240283206105232,
+            0.02144254744052887,
+            0.007456236053258181,
+            0.014332021586596966,
+            0.014878327958285809,
+            -0.01336744800209999,
+            -0.000287024537101388,
+            0.003220223356038332,
+            0.016056302934885025,
+            0.0052454001270234585,
+            -0.05865970999002457,
+            0.004421671852469444,
+            -0.007823286578059196,
+            -0.005031999200582504,
+            -0.06091322377324104,
+            0.009500619024038315,
+            -0.006598363630473614,
+            0.022825386375188828,
+            -0.006692260038107634,
+            0.03803662210702896,
+            0.00777206989005208,
+            -0.0019302134169265628,
+            0.007819018326699734,
+            0.008220212534070015,
+            0.022944891825318336,
+            -0.007183082867413759,
+            0.00777206989005208,
+            0.0030751104932278395,
+            0.01720866933465004,
+            -0.0033525321632623672,
+            -0.012547987513244152,
+            0.0011875773780047894,
+            0.020981600508093834,
+            -0.006222777534276247,
+            0.009065280668437481,
+            0.03574896231293678,
+            0.002298330422490835,
+            -0.029773728922009468,
+            0.013666209764778614,
+            0.0020614550448954105,
+            -0.017891552299261093,
+            0.01964997872710228,
+            -0.005680738482624292,
+            0.011250508949160576,
+            -0.002041182015091181,
+            0.002332474570721388,
+            0.023525342345237732,
+            0.001920610317029059,
+            0.018540292978286743,
+            -0.0014820710057392716,
+            -0.010755417868494987,
+            0.027315348386764526,
+            -0.008766518905758858,
+            -0.022244935855269432,
+            -0.015381954610347748,
+            -0.03817319869995117,
+            -0.010268863290548325,
+            -0.0056508625857532024,
+            -0.03042246773838997,
+            0.005078947637230158,
+            0.027793366461992264,
+            0.019103670492768288,
+            -0.023730207234621048,
+            -0.023542415350675583,
+            -0.018198849633336067,
+            -0.004167724400758743,
+            0.007144670467823744,
+            -0.03322229161858559,
+            -0.012300441972911358,
+            0.012923574075102806,
+            -0.006056324578821659,
+            0.004511300474405289,
+            -0.009927421808242798,
+            -0.03096877597272396,
+            0.06944926828145981,
+            -0.015620964579284191,
+            -0.013896683230996132,
+            -0.02567642740905285,
+            -0.003478438826277852,
+            -0.011668775230646133,
+            0.023474127054214478,
+            -0.0034250884782522917,
+            0.005403317045420408,
+            0.015774613246321678,
+            0.005155771970748901,
+            0.011207828298211098,
+            -0.00033610680839046836,
+            0.009551835246384144,
+            0.014656390994787216,
+            0.005360636860132217,
+            -0.003478438826277852,
+            0.00591121194884181,
+            0.015262450091540813,
+            -0.027947014197707176,
+            0.008685426786541939,
+            -0.019428040832281113,
+            0.008681158535182476,
+            0.000938965065870434,
+            0.011617559008300304,
+            0.0031988832633942366,
+            -0.01290650200098753,
+            -0.005437461193650961,
+            -0.04073401167988777,
+            -0.009594515897333622,
+            0.023542415350675583,
+            0.017942769452929497,
+            0.03544166311621666,
+            0.0022321760188788176,
+            0.009253074415028095,
+            0.022671738639473915,
+            0.03841220587491989,
+            -0.018198849633336067,
+            0.0015813024947419763,
+            0.010550552979111671,
+            -0.0015770344762131572,
+            0.04916762560606003,
+            0.02362777478992939,
+            0.0012644018279388547,
+            -0.01149805448949337,
+            0.0006620771018788218,
+            0.00023580827109981328,
+            0.006154489237815142,
+            0.006611167918890715,
+            -0.0018437859835103154,
+            -0.002035846933722496,
+            -0.017669616267085075,
+            0.005539894104003906,
+            -0.006542879156768322,
+            -0.019735338166356087,
+            -0.0068587129935622215,
+            0.004835669882595539,
+            0.0289030522108078,
+            0.0002462115662638098,
+            0.004950906615704298,
+            -0.0005164307658560574,
+            0.0007132933242246509,
+            0.026786113157868385,
+            -0.009210393764078617,
+            0.01588558219373226,
+            0.002345278626307845,
+            -0.013597921468317509,
+            0.010174967348575592,
+            -0.008459221571683884,
+            0.0021574855782091618,
+            -0.012974790297448635,
+            0.014878327958285809,
+            0.015569748356938362,
+            -0.010192039422690868,
+            -0.025095975026488304,
+            -0.011762671172618866,
+            0.026769040152430534,
+            -0.015552676282823086,
+            0.011327332817018032,
+            0.003011090215295553,
+            0.0007778471917845309,
+            -0.00181390973739326,
+            -0.0009202924557030201,
+            0.03827562928199768,
+            -0.00943233072757721,
+            -0.016158735379576683,
+            -0.005749027244746685,
+            0.012479699216783047,
+            0.008343985304236412,
+            -0.014972224831581116,
+            -0.01925732009112835,
+            0.018796373158693314,
+            -0.017908625304698944,
+            0.015697788447141647,
+            -0.04182662442326546,
+            -0.023303406313061714,
+            0.026615392416715622,
+            -9.683077223598957e-05,
+            -0.03503193333745003,
+            0.01168584730476141,
+            0.016329456120729446,
+            0.0028617095667868853,
+            0.008271428756415844,
+            -0.008702498860657215,
+            0.0028190291486680508,
+            -0.007754997815936804,
+            -0.010849314741790295,
+            0.0013636333169415593,
+            0.007114794570952654,
+            -0.007955594919621944,
+            0.02082795277237892,
+            -0.014195444993674755,
+            -0.015211233869194984,
+            -0.005680738482624292,
+            -0.0128552857786417,
+            0.01553560420870781,
+            -0.00581731554120779,
+            -0.013256479986011982,
+            0.017473286017775536,
+            0.014485670253634453,
+            0.015433171764016151,
+            0.02478867769241333,
+            0.018301282078027725,
+            -0.02250101789832115,
+            -0.002558679785579443,
+            3.8512240280397236e-05,
+            -0.0024733194150030613,
+            0.01227483432739973,
+            0.035236798226833344,
+            0.0016421218169853091,
+            -0.009816452860832214,
+            -0.02364484779536724,
+            0.01277846097946167,
+            0.01309429481625557,
+            -0.014041795395314693,
+            -0.018711013719439507,
+            -0.004114374052733183,
+            -0.013964971527457237,
+            0.00652153929695487,
+            -0.0023111344780772924,
+            0.000925094005651772,
+            -0.0009939158335328102,
+            -0.003484840737655759,
+            -0.02091331221163273,
+            0.005356369074434042,
+            -0.01953047327697277,
+            -0.031907740980386734,
+            -0.012530915439128876,
+            0.008036687038838863,
+            0.024754533544182777,
+            0.002035846933722496,
+            -0.0224839448928833,
+            -0.0014415248297154903,
+            -0.00016365200281143188,
+            -0.0093213627114892,
+            -0.01495515275746584,
+            0.0016197147779166698,
+            0.0005118960398249328,
+            0.010038389824330807,
+            -0.007682441268116236,
+            -0.01722574047744274,
+            0.005471605341881514,
+            0.01734524592757225,
+            -0.022381512448191643,
+            -0.0025202676188200712,
+            -0.017174525186419487,
+            -0.01559535600244999,
+            0.0873408243060112,
+            -0.010891994461417198,
+            -0.012539451941847801,
+            -0.01418690849095583,
+            0.0038092106115072966,
+            -0.006333746016025543,
+            0.010678594000637531,
+            0.03264183923602104,
+            -0.006026448681950569,
+            -0.006338014267385006,
+            -0.0026717823930084705,
+            -0.0008578726556152105,
+            -0.019359752535820007,
+            0.001124624046497047,
+            0.007921450771391392,
+            -0.004566784482449293,
+            -0.003316253889352083,
+            0.01450274232774973,
+            -0.04435329511761665,
+            0.0021916297264397144,
+            0.005096019711345434,
+            -0.02260345034301281,
+            0.02002556435763836,
+            0.015919726341962814,
+            -0.022057143971323967,
+            0.003002554178237915,
+            0.00606912886723876,
+            0.03311985731124878,
+            0.022296153008937836,
+            0.009218929335474968,
+            0.015723396092653275,
+            -0.01836957037448883,
+            0.012863821350038052,
+            -0.002146815648302436,
+            -0.0002396761701675132,
+            0.008954312652349472,
+            0.004472888074815273,
+            0.013145511038601398,
+            0.0016581269446760416,
+            0.0009016198455356061,
+            -0.022688809782266617,
+            -0.006850176956504583,
+            -0.026820257306098938,
+            -0.027895798906683922,
+            -0.010379832237958908,
+            0.03182237967848778,
+            -0.008988456800580025,
+            -0.014332021586596966,
+            0.006628239993005991,
+            -0.008292769081890583,
+            -0.002063589170575142,
+            -0.024617956951260567,
+            0.016337992623448372,
+            -0.007272711489349604,
+            0.0024413091596215963,
+            0.0005537759861908853,
+            0.0112334368750453,
+            -0.0064660548232495785,
+            -0.022808315232396126,
+            0.010584697127342224,
+            -0.0024583812337368727,
+            0.03426368907094002,
+            -0.00924453791230917,
+            -0.01836957037448883,
+            0.004780185874551535,
+            0.00041613224311731756,
+            -0.01941096968948841,
+            -0.005962428171187639,
+            -0.007793410215526819,
+            -0.021425476297736168,
+            -0.014613711275160313,
+            -0.013597921468317509,
+            -0.012027288787066936,
+            -0.01655992865562439,
+            0.006278262007981539,
+            0.00014591304352506995,
+            0.008574457839131355,
+            -0.01027739979326725,
+            -0.000391057605156675,
+            0.012539451941847801,
+            -0.015433171764016151,
+            -0.019308535382151604,
+            -0.007784874178469181,
+            -0.0041976007632911205,
+            -0.0013294891687110066,
+            0.005915479734539986,
+            -0.0218522772192955,
+            -0.037319593131542206,
+            0.011335869319736958,
+            -0.011002963408827782,
+            -0.004342713393270969,
+            0.022791242226958275,
+            0.000883480766788125,
+            -0.022705882787704468,
+            0.009415258653461933,
+            0.012599203735589981,
+            0.0045027644373476505,
+            -0.015612428076565266,
+            0.005510017741471529,
+            0.0034677686635404825,
+            -0.01110539585351944,
+            -0.01860858127474785,
+            0.0034528307151049376,
+            -0.008685426786541939,
+            0.005731955170631409,
+            -0.02786165475845337,
+            0.013657674193382263,
+            0.014784432016313076,
+            0.01084077823907137,
+            0.00021033351367805153,
+            -0.00475457776337862,
+            -0.01719159632921219,
+            -0.002552277874201536,
+            -0.00642764288932085,
+            0.07013215124607086,
+            -0.012752852402627468,
+            -0.017635472118854523,
+            0.016056302934885025,
+            0.0011384951649233699,
+            -0.011617559008300304,
+            0.009662804193794727,
+            -0.01848907582461834,
+            0.020623087882995605,
+            -0.03868535906076431,
+            0.012291906401515007,
+            -0.006060592830181122,
+            0.0075842770747840405,
+            -0.008655550889670849,
+            0.0006172628491185606,
+            -0.021408403292298317,
+            0.023918000981211662,
+            0.027656789869070053,
+            0.010431048460304737,
+            -0.01847200281918049,
+            -0.027895798906683922,
+            0.016824547201395035,
+            0.017003804445266724,
+            0.002428505104035139,
+            -0.009637195616960526,
+            0.011754135601222515,
+            -0.023952145129442215,
+            0.008945776149630547,
+            -0.006346550304442644,
+            -0.0032372954301536083,
+            0.009218929335474968,
+            0.007814750075340271,
+            0.007951326668262482,
+            0.009364042431116104,
+            -0.03235161304473877,
+            0.016798939555883408,
+            0.028800619766116142,
+            0.004592392593622208,
+            0.028800619766116142,
+            -0.018796373158693314,
+            0.01823299378156662,
+            -0.006752012297511101,
+            0.009022600948810577,
+            0.021254755556583405,
+            0.011463910341262817,
+            -0.030081026256084442,
+            0.0027678129263222218,
+            -0.03380274400115013,
+            0.0017712295521050692,
+            0.02673489600419998,
+            -0.0044302078895270824,
+            0.001891801250167191,
+            -0.02284245938062668,
+            0.008860415779054165,
+            0.004443012177944183,
+            -0.020059708505868912,
+            -0.014357629232108593,
+            0.00600510835647583,
+            0.0025800198782235384,
+            -0.026547104120254517,
+            0.022518089041113853,
+            -0.023815568536520004,
+            -0.0010488666594028473,
+            0.017618399113416672,
+            0.0006231313454918563,
+            -0.01579168438911438,
+            0.008420809172093868,
+            0.02159619703888893,
+            -0.02579593099653721,
+            -0.032027244567871094,
+            0.006431910675019026,
+            0.0075842770747840405,
+            0.021630341187119484,
+            0.01978655532002449,
+            -0.008723839186131954,
+            0.02287660352885723,
+            0.007840358652174473,
+            0.0038177466485649347,
+            -0.0372513048350811,
+            -0.017635472118854523,
+            0.019957276061177254,
+            -0.014536886475980282,
+            0.018454931676387787,
+            -0.011412693187594414,
+            -0.001605843659490347,
+            0.009355506859719753,
+            -0.008139119483530521,
+            0.027793366461992264,
+            -0.01405886746942997,
+            -0.0036576958373188972,
+            0.004425940103828907,
+            -0.03341008350253105,
+            -0.0024178349412977695,
+            0.025727642700076103,
+            0.001300679985433817,
+            -0.02454966865479946,
+            -0.0030132243409752846,
+            -0.0023580826818943024,
+            0.003779334481805563,
+            -0.005475873593240976,
+            -0.007733657956123352,
+            -0.010644449852406979,
+            0.003572335233911872,
+            -0.006453251000493765,
+            -0.011352941393852234,
+            0.00214574858546257,
+            0.04985050857067108,
+            0.015817293897271156,
+            -0.007123330608010292,
+            0.0005857861833646894,
+            0.01728549413383007,
+            0.042953383177518845,
+            -0.022825386375188828,
+            0.03122485615313053,
+            0.0009693747269921005,
+            0.0038540249224752188,
+            -0.02478867769241333,
+            -0.026052013039588928,
+            0.02632516622543335,
+            0.010721273720264435,
+            -0.02542034536600113,
+            -0.002863843459635973,
+            0.00415492057800293,
+            0.012223618105053902,
+            0.014929544180631638,
+            0.014690535143017769,
+            -0.031241929158568382,
+            -0.015475851483643055,
+            -0.003343996126204729,
+            -0.008954312652349472,
+            -0.00606912886723876,
+            -0.005322224926203489,
+            0.013478416949510574,
+            0.008143387734889984,
+            -0.025778859853744507,
+            -0.0013454942964017391,
+            -0.005390513222664595,
+            -0.005343564786016941,
+            -0.01033715158700943,
+            0.009278682060539722,
+            -0.009227465838193893,
+            0.0040610237047076225,
+            -0.010448120534420013,
+            0.010832242667675018,
+            -0.015074657276272774,
+            0.010516408830881119,
+            0.004186930600553751,
+            -0.02094745635986328,
+            -0.012496771290898323,
+            -0.01838664337992668,
+            0.007183082867413759,
+            0.01937682554125786,
+            0.01835249923169613,
+            0.0062996018677949905,
+            -0.014084476046264172,
+            -0.016039229929447174,
+            -0.015151482075452805,
+            0.017908625304698944,
+            0.01014935877174139,
+            -0.005501481704413891,
+            -0.018028128892183304,
+            -0.025659354403614998,
+            0.022518089041113853,
+            0.005988036282360554,
+            -0.03612454608082771,
+            0.02250101789832115,
+            -0.008326913230121136,
+            -6.268659035413293e-06,
+            0.012761388905346394,
+            -0.010900530964136124,
+            -0.004400331526994705,
+            -0.0023474127519875765,
+            0.019206102937459946,
+            -0.0014916741056367755,
+            -0.0008450685418210924,
+            0.005706346593797207,
+            -0.00011263579654041678,
+            -0.008817736059427261,
+            -0.013213799335062504,
+            -0.0037537263706326485,
+            0.001452194876037538,
+            -0.0033226560335606337,
+            -0.0024498451966792345,
+            0.004545444622635841,
+            0.014681999571621418,
+            0.013802786357700825,
+            0.005578306037932634,
+            0.004746041726320982,
+            0.0007917182520031929,
+            -0.007242835126817226,
+            0.01059323363006115,
+            -0.021408403292298317,
+            0.006816032808274031,
+            -0.005569770000874996,
+            -0.020725520327687263,
+            -0.005902675911784172,
+            0.00018845988961402327,
+            0.02157912403345108,
+            -0.03431490436196327,
+            0.0038198805414140224,
+            0.017294028773903847,
+            0.009346970357000828,
+            -0.01836957037448883,
+            0.010883458890020847,
+            0.06518124788999557,
+            -0.005271008238196373,
+            0.008284232579171658,
+            -0.0010888794204220176,
+            0.011779743246734142,
+            -0.006184365600347519,
+            0.005322224926203489,
+            -0.0017360183410346508,
+            0.0346904918551445,
+            -0.007823286578059196,
+            -0.012940646149218082,
+            -0.023183900862932205,
+            -0.012496771290898323,
+            -0.016397744417190552,
+            0.02786165475845337,
+            -0.014477133750915527,
+            0.002011305885389447,
+            -0.012078505009412766,
+            0.0065386113710701466,
+            0.002077460289001465,
+            0.0014596638502553105,
+            -0.018215922638773918,
+            -0.009927421808242798,
+            0.016747722402215004,
+            0.0034442946780472994,
+            0.0023666187189519405,
+            0.00530088460072875,
+            -0.014528350904583931,
+            0.030524900183081627,
+            -0.0074477000162005424,
+            0.020059708505868912,
+            0.004391795489937067,
+            -0.005232596304267645,
+            0.015185626223683357,
+            -0.009466474875807762,
+            0.015859974548220634,
+            -0.0118821756914258,
+            -0.003529655048623681,
+            -0.030149314552545547,
+            -0.008565922267735004,
+            0.020623087882995605,
+            -0.014519814401865005,
+            0.0005495079676620662,
+            0.016525784507393837,
+            0.005014927126467228,
+            -0.007959863170981407,
+            0.011224900372326374,
+            0.003382408292964101,
+            -0.01758425496518612,
+            -0.010832242667675018,
+            -0.004647877067327499,
+            -0.01072981022298336,
+            0.011301725171506405,
+            0.013589384965598583,
+            -0.022398585453629494,
+            0.0015290192095562816,
+            -0.006914197467267513,
+            -0.008207408711314201,
+            0.00716174254193902,
+            0.013162583112716675,
+            -0.027571428567171097,
+            0.017370853573083878,
+            0.0021254755556583405,
+            -0.0002107336331391707,
+            0.004250951111316681,
+            -0.0004980249213986099,
+            0.014605174772441387,
+            -0.007575741037726402,
+            -0.002740070689469576,
+            -0.006845909170806408,
+            0.010934675112366676,
+            -0.0004094634496141225,
+            0.0157319325953722,
+            -0.00683737313374877,
+            -0.002915059681981802,
+            0.007234299089759588,
+            0.016286775469779968,
+            -0.02557399496436119,
+            0.015424635261297226,
+            -0.00810070801526308,
+            0.0002974278468172997,
+            0.00031343294540420175,
+            0.009253074415028095,
+            0.005591110326349735,
+            -0.015902653336524963,
+            -0.0072300308384001255,
+            0.015569748356938362,
+            0.005117359571158886,
+            0.018950022757053375,
+            0.021476691588759422,
+            0.007033701986074448,
+            0.023866785690188408,
+            -0.005744758993387222,
+            -0.013085758313536644,
+            -0.0020561201963573694,
+            -0.009338434785604477,
+            -0.002368752844631672,
+            0.006286798045039177,
+            -0.004054621793329716,
+            -0.025915436446666718,
+            -0.012821141630411148,
+            -0.007276979275047779,
+            -0.00038012079312466085,
+            0.009475011378526688,
+            -0.010311543941497803,
+            0.01707209274172783,
+            0.02236444130539894,
+            7.760800144751556e-06,
+            -0.006175829563289881,
+            -0.02094745635986328,
+            0.007251371163874865,
+            -0.007763533852994442,
+            -0.011737063527107239,
+            -0.010098142549395561,
+            -0.0028702456038445234,
+            0.0018395179649814963,
+            0.005407585296779871,
+            0.004775917623192072,
+            -0.017772048711776733,
+            -0.006862981244921684,
+            0.009423795156180859,
+            -0.01799398474395275,
+            -0.0023239385336637497,
+            0.00031343294540420175,
+            0.006666651926934719,
+            0.012872357852756977,
+            0.01758425496518612,
+            -0.013614993542432785,
+            -0.022449800744652748,
+            0.003002554178237915,
+            -0.009739628992974758,
+            0.03274427354335785,
+            -0.008168996311724186,
+            -0.005514285992830992,
+            -0.018284210935235023,
+            -0.006175829563289881,
+            -0.009346970357000828,
+            -0.00956890732049942,
+            -0.0004897556500509381,
+            0.014861255884170532,
+            -0.009304290637373924,
+            0.01482711173593998,
+            -0.016491640359163284,
+            -0.003205285407602787,
+            0.00409090006724,
+            0.00969694834202528,
+            -0.030012737959623337,
+            -0.006278262007981539,
+            -0.00799400731921196,
+            -0.02145962044596672,
+            -0.013708890415728092,
+            -0.012471163645386696,
+            0.007588544860482216,
+            0.0014073805650696158,
+            0.01642335206270218,
+            -0.007187350653111935,
+            -0.014195444993674755,
+            0.009978638030588627,
+            -0.02031579054892063,
+            -0.012983325868844986,
+            -0.008292769081890583,
+            -0.04073401167988777,
+            0.020469438284635544,
+            0.01354670524597168,
+            0.023337550461292267,
+            -0.0024434432853013277,
+            0.020964529365301132,
+            0.008723839186131954,
+            -0.01742207072675228,
+            0.006372158415615559,
+            -0.012505307793617249,
+            0.005744758993387222,
+            -0.02506183087825775,
+            -0.0066026318818330765,
+            0.005505749955773354,
+            0.01246262714266777,
+            -0.0037665304262191057,
+            2.7758822398027405e-05,
+            0.014280805364251137,
+            -0.01041397638618946,
+            -0.01787448115646839,
+            0.004067426081746817,
+            0.0067562805488705635,
+            -0.01713184453547001,
+            0.03670499846339226,
+            -0.00796413142234087,
+            -0.00017445544654037803,
+            0.013179655186831951,
+            0.02104988880455494,
+            -0.020264573395252228,
+            -0.01450274232774973,
+            0.01937682554125786,
+            -0.0010200574761256576,
+            -0.0004940236685797572,
+            0.017183061689138412,
+            -0.006828837096691132,
+            0.008032419718801975,
+            -0.005766099318861961,
+            -0.00484847417101264,
+            -0.00503626698628068,
+            0.0011683712946251035,
+            -0.016901372000575066,
+            0.0042637549340724945,
+            -0.000693553767632693,
+            -0.0006658115889877081,
+            0.019581690430641174,
+            -0.006453251000493765,
+            0.0020102388225495815,
+            0.011976072564721107,
+            0.00038332180702127516,
+            -0.05049924924969673,
+            -0.0047204336151480675,
+            0.0034869748633354902,
+            -0.003279975615441799,
+            -0.003469902789220214,
+            0.009807917289435863,
+            0.01195046491920948,
+            0.01110539585351944,
+            -0.005130163859575987,
+            0.013452808372676373,
+            0.00938111450523138,
+            0.006632507778704166,
+            0.0007287649204954505,
+            -0.004329909570515156,
+            0.011813887394964695,
+            -0.004588124807924032,
+            0.020127996802330017,
+            -0.004421671852469444,
+            -0.01889880560338497,
+            -0.016414817422628403,
+            0.0010739413555711508,
+            0.027025122195482254,
+            -0.0037665304262191057,
+            0.026103228330612183,
+            -0.0013839064631611109,
+            0.0033205219078809023,
+            -0.0018032396910712123,
+            -0.0032223574817180634,
+            -0.004827133845537901,
+            -0.035134363919496536,
+            0.009039673022925854,
+            -0.007084918208420277,
+            0.020008491352200508,
+            -0.02943228743970394,
+            0.003026028396561742,
+            -0.006175829563289881,
+            -0.005787439178675413,
+            -0.0025608139112591743,
+            0.005031999200582504,
+            -2.869245145120658e-05,
+            0.03803662210702896,
+            -0.014844183810055256,
+            -0.021271826699376106,
+            -0.010396904312074184,
+            0.006995289586484432,
+            0.02798115834593773,
+            -0.0004838871245738119,
+            0.01913781464099884,
+            0.020896241068840027,
+            -0.013068686239421368,
+            0.004844206385314465,
+            -0.0031156567856669426,
+            -0.002942801918834448,
+            0.00011677043949021026,
+            -0.009440867230296135,
+            0.005198452156037092,
+            -0.03530508652329445,
+            0.007008093874901533,
+            -0.0005225660861469805,
+            0.02427651546895504,
+            5.1282964705023915e-05,
+            0.004814330022782087,
+            -0.003311985870823264,
+            0.02274002693593502,
+            0.003945787437260151,
+            0.01719159632921219,
+            -0.0013017470482736826,
+            0.004767381586134434,
+            0.010396904312074184,
+            -0.019086599349975586,
+            -0.019820699468255043,
+            -0.015142945572733879,
+            0.001055268687196076,
+            0.0011448971927165985,
+            -0.015270986594259739,
+            -0.018830517306923866,
+            -0.0059923045337200165,
+            0.017515966668725014,
+            -0.011702919378876686,
+            0.01264188438653946,
+            0.013538168743252754,
+            0.009159177541732788,
+            -0.001976094674319029,
+            0.01227483432739973,
+            0.01592826284468174,
+            -0.018215922638773918,
+            0.009022600948810577,
+            -0.009577443823218346,
+            -0.008937240578234196,
+            -0.010960282757878304,
+            -0.01811349019408226,
+            0.03438319265842438,
+            0.014750287868082523,
+            -0.011677310802042484,
+            -0.008587262593209743,
+            -0.0010019184555858374,
+            0.004758845549076796,
+            0.00044013987644575536,
+            0.005663666408509016,
+            -0.004647877067327499,
+            -0.026137372478842735,
+            -0.014673463068902493,
+            -0.016466032713651657,
+            -0.005719150882214308,
+            0.002590690040960908,
+            0.014801504090428352,
+            -0.021732773631811142,
+            -0.016397744417190552,
+            -0.0263080932199955,
+            0.01925732009112835,
+            -0.017533037811517715,
+            0.01651724986732006,
+            0.012249225750565529,
+            0.002076393226161599,
+            -0.01265041995793581,
+            -0.02801530249416828,
+            0.014741751365363598,
+            -0.024361874908208847,
+            0.00670933211222291,
+            -0.0016261168057098985,
+            0.0500553734600544,
+            -0.00277634896337986,
+            -0.006192901637405157,
+            0.009500619024038315,
+            0.003561665304005146,
+            0.013845467008650303,
+            -0.02197178266942501,
+            -0.005403317045420408,
+            -0.003525387030094862,
+            0.011566342785954475,
+            -0.004571052733808756,
+            -0.0056892745196819305,
+            -0.002590690040960908,
+            -0.007375143934041262,
+            0.001402045600116253,
+            0.019291464239358902,
+            -0.006880053319036961,
+            -0.007008093874901533,
+            -0.0030921828001737595,
+            0.029842017218470573,
+            0.01963290572166443,
+            -0.008548850193619728,
+            -0.006444714963436127,
+            0.008365325629711151,
+            -0.01787448115646839,
+            -0.0014916741056367755,
+            0.017370853573083878,
+            0.021220611408352852,
+            -0.011224900372326374,
+            0.006999557837843895,
+            0.002942801918834448,
+            -0.0033269240520894527,
+            -0.021510835736989975,
+            -0.01137001346796751,
+            -0.0075202565640211105,
+            -0.0018896672409027815,
+            0.01632091961801052,
+            -0.011310260742902756,
+            0.02325218915939331,
+            0.01567218080163002,
+            0.009517691098153591,
+            0.0010883458890020847,
+            -0.041280318051576614,
+            -0.01548438798636198,
+            0.01687576249241829,
+            -0.0034613667521625757,
+            0.017362317070364952,
+            0.013649137690663338,
+            0.0031007188372313976,
+            -0.019479257985949516,
+            0.023952145129442215,
+            0.0008594731334596872,
+            0.0014233856927603483,
+            -0.001049933722242713,
+            0.015330738388001919,
+            0.0019632906187325716,
+            0.00962012354284525,
+            0.02106696180999279,
+            0.020640159025788307,
+            -0.01797691360116005,
+            -0.009987173601984978,
+            0.011301725171506405,
+            -0.017037948593497276,
+            0.019684122875332832,
+            0.001101149944588542,
+            0.003862560959532857,
+            0.015501460060477257,
+            0.027025122195482254,
+            0.03277841582894325,
+            0.011062716133892536,
+            -0.004933834541589022,
+            -0.007959863170981407,
+            -0.001679467037320137,
+            0.004545444622635841,
+            0.014041795395314693,
+            -0.02337169460952282,
+            -0.031054135411977768,
+            0.003213821444660425,
+            0.013529633171856403,
+            -0.02506183087825775,
+            0.007725121919065714,
+            0.01913781464099884,
+            0.02181813307106495,
+            0.0050917514599859715,
+            0.015552676282823086,
+            0.014519814401865005,
+            0.0034528307151049376,
+            -0.002049718052148819,
+            0.010046926327049732,
+            -0.013623529113829136,
+            0.005155771970748901,
+            0.0022833922412246466,
+            -0.015509995631873608,
+            0.02053772658109665,
+            -0.00037345202872529626,
+            0.00026034939219243824,
+            0.003395212348550558,
+            0.001364700379781425,
+            -0.00950915552675724,
+            -0.03426368907094002,
+            0.016551394015550613,
+            -0.004159188363701105,
+            0.010439584031701088,
+            -0.017020875588059425,
+            -0.0075714727863669395,
+            -0.007511720526963472,
+            0.03421247377991676,
+            0.009304290637373924,
+            0.007631225045770407,
+            -0.004293631296604872,
+            0.0008301304769702256,
+            0.003205285407602787,
+            0.007699513342231512,
+            -0.01799398474395275,
+            -0.018045201897621155,
+            0.009662804193794727,
+            0.004903958644717932,
+            8.744445949560031e-06,
+            0.01642335206270218,
+            0.009073817171156406,
+            0.012078505009412766,
+            0.02299610897898674,
+            0.008040955290198326,
+            -0.01728549413383007,
+            0.0033013159409165382,
+            0.019957276061177254,
+            -0.0008957513491623104,
+            0.01707209274172783,
+            0.007486112415790558,
+            -0.007008093874901533,
+            -0.026103228330612183,
+            0.00020659899746533483,
+            0.018164705485105515,
+            0.023952145129442215,
+            -0.023969218134880066,
+            -0.019052455201745033,
+            0.01850614883005619,
+            -0.0121980095282197,
+            0.010627377778291702,
+            -0.002761411014944315,
+            0.000420933763962239,
+            0.0031754090450704098,
+            -0.017217205837368965,
+            0.004562516696751118,
+            0.017037948593497276,
+            0.01014935877174139,
+            0.00015778347733430564,
+            0.005386244971305132,
+            0.01809641718864441,
+            0.022449800744652748,
+            -0.030661476776003838,
+            -0.010968819260597229,
+            -0.0029769460670650005,
+            0.004579588770866394,
+            -0.014613711275160313,
+            -0.0028531732968986034,
+            0.005108823534101248,
+            0.00937257893383503,
+            -0.022057143971323967,
+            -0.0109176030382514,
+            -0.0023025984410196543,
+            -0.0024626492522656918,
+            0.008903096430003643,
+            -0.004733237437903881,
+            0.0030559045262634754,
+            -0.016798939555883408,
+            0.005492945667356253,
+            -0.019615834578871727,
+            0.017183061689138412,
+            -0.0049252985045313835,
+            -0.0008722771890461445,
+            -0.010217647068202496,
+            0.0021062693558633327,
+            -0.0028702456038445234,
+            -0.02390092983841896,
+            0.022893676534295082,
+            -0.0157148614525795,
+            -0.0036897058598697186,
+            0.043738700449466705,
+            0.002827565185725689,
+            0.006828837096691132,
+            0.02364484779536724,
+            0.002174557652324438,
+            -0.004647877067327499,
+            0.026103228330612183,
+            0.03175409138202667,
+            -0.010550552979111671,
+            -0.009748164564371109,
+            0.017618399113416672,
+            0.009278682060539722,
+            0.006935537327080965,
+            0.011489517986774445,
+            -0.0027784830890595913,
+            -0.017686687409877777,
+            -0.00911649689078331,
+            0.0011790413409471512,
+            0.015492923557758331,
+            -0.0032308935187757015,
+            -0.0020059708040207624,
+            0.011924856342375278,
+            7.60908515076153e-05,
+            -0.007524524815380573,
+            0.02837381698191166,
+            0.002735802670940757,
+            -0.01784033700823784,
+            -0.029210349544882774,
+            0.0118821756914258,
+            -0.0125138433650136,
+            -0.018796373158693314,
+            -0.017294028773903847,
+            -0.0021030684001743793,
+            -0.011301725171506405,
+            -0.002526669530197978,
+            0.012829677201807499,
+            0.01163463108241558,
+            -0.005211255978792906,
+            0.01620141603052616,
+            0.006329478230327368,
+            -0.015902653336524963,
+            -0.01612459123134613,
+            -0.004276559222489595,
+            -0.003909509163349867,
+            -0.0035040469374507666,
+            -0.003911643289029598,
+            0.018198849633336067,
+            -0.009090889245271683,
+            -0.0044302078895270824,
+            -0.018557364121079445,
+            0.01760132610797882,
+            0.013700353913009167,
+            -0.009398186579346657,
+            -0.001582369557581842,
+            -0.001988898729905486,
+            0.028203096240758896,
+            0.0062996018677949905,
+            0.0017947036540135741,
+            0.004297899082303047,
+            -0.00968841277062893,
+            -0.0010846114018931985,
+            -0.02463502809405327,
+            0.020093852654099464,
+            -0.018437858670949936,
+            -0.026939760893583298,
+            -0.007170278578996658,
+            0.024959398433566093,
+            -0.0018395179649814963,
+            0.0015076791169121861,
+            -0.00831837672740221,
+            0.018932949751615524,
+            0.0029812140855938196,
+            -0.02823724038898945,
+            -0.0125138433650136,
+            -0.010183502919971943,
+            -0.015552676282823086,
+            -0.0013518963241949677,
+            0.020332861691713333,
+            -0.01360645703971386,
+            0.00035237864358350635,
+            0.01671357825398445,
+            -0.01149805448949337,
+            0.013469880446791649,
+            -0.003260769648477435,
+            -0.009125033393502235,
+            0.013845467008650303,
+            -0.008536046370863914,
+            -0.00408022990450263,
+            -0.003094316693022847,
+            0.005275276489555836,
+            -0.0012387937167659402,
+            -0.019086599349975586,
+            -0.009440867230296135,
+            0.01084077823907137,
+            0.002966275904327631,
+            0.008728107437491417,
+            -0.014050331898033619,
+            -0.005629522260278463,
+            0.001096881926059723,
+            -7.188950985437259e-05,
+            -0.0018960692686960101,
+            -0.012582131661474705,
+            -0.011327332817018032,
+            -0.020879168063402176,
+            -0.007665369194000959,
+            -0.01206143293529749,
+            0.011173684149980545,
+            0.015245378017425537,
+            -0.010738345794379711,
+            -0.005642326548695564,
+            0.008305572904646397,
+            0.04100716486573219,
+            -0.009782308712601662,
+            0.004383259452879429,
+            0.021613268181681633,
+            0.0032372954301536083,
+            0.007592813111841679,
+            0.02874940261244774,
+            -0.0017701626056805253,
+            -0.010704201646149158,
+            0.005117359571158886,
+            -0.01825006678700447,
+            -0.002957739867269993,
+            0.027554357424378395,
+            -0.021271826699376106,
+            0.005787439178675413,
+            -0.023730207234621048,
+            -0.008079367689788342,
+            -0.0033888102043420076,
+            -0.0121980095282197,
+            -0.014238124713301659,
+            -0.002049718052148819,
+            0.0006028582574799657,
+            0.023713136091828346,
+            -0.007439163979142904,
+            0.0050405352376401424,
+            0.009159177541732788,
+            -0.025130119174718857,
+            0.007426360156387091,
+            -0.010994426906108856,
+            -0.010166430845856667,
+            -0.004916762467473745,
+            -0.026803184300661087,
+            0.015501460060477257,
+            -0.011540734209120274,
+            -0.006717868149280548,
+            -0.018557364121079445,
+            0.033205218613147736,
+            0.017635472118854523,
+            0.011276116594672203,
+            -0.02313268557190895,
+            -0.015014905482530594,
+            -0.020110923796892166,
+            -0.037046439945697784,
+            -0.00030409664032049477,
+            0.0008386665140278637,
+            -0.002919327700510621,
+            0.007716585882008076,
+            0.025864219292998314,
+            0.0017701626056805253,
+            -0.006444714963436127,
+            -0.004575320519506931,
+            0.007375143934041262,
+            0.009005528874695301,
+            -0.014417381957173347,
+            0.00911649689078331,
+            0.018301282078027725,
+            0.0002280724875163287,
+            0.007725121919065714,
+            0.00813058391213417,
+            0.0020305118523538113,
+            0.00548867741599679,
+            0.028407961130142212,
+            0.0034122844226658344,
+            0.003109254874289036,
+            -0.0001357764849672094,
+            -7.043905043246923e-06,
+            -0.011148076504468918,
+            0.0010947479167953134,
+            -0.018591508269309998,
+            0.00600510835647583,
+            -0.01860858127474785,
+            0.0199231319129467,
+            -0.004127178341150284,
+            0.008092171512544155,
+            -0.02207421511411667,
+            -0.0243960190564394,
+            0.004268023185431957,
+            0.027178769931197166,
+            0.01259066816419363,
+            -0.007878770120441914,
+            -0.015859974548220634,
+            0.01888173446059227,
+            0.01104564405977726,
+            -0.011267581023275852,
+            -0.012505307793617249,
+            -0.017789119854569435,
+            -9.489682270213962e-05,
+            -0.02144254744052887,
+            0.02528376877307892,
+            0.015902653336524963,
+            0.0038881690707057714,
+            -0.013171118684113026,
+            0.02425944246351719,
+            -0.011549270711839199,
+            0.0070465062744915485,
+            -0.01503197755664587,
+            -0.015561211854219437,
+            -0.0186427254229784,
+            -0.005945356097072363,
+            -0.001342293224297464,
+            0.011651703156530857,
+            -0.027571428567171097,
+            0.013768642209470272,
+            0.014391773380339146,
+            -0.002705926541239023,
+            0.0022215060889720917,
+            -0.0055356258526444435,
+            0.014571030624210835,
+            -0.006875785067677498,
+            0.007336731534451246,
+            -0.006991021800786257,
+            -0.0021873617079108953,
+            -0.0065172710455954075,
+            -0.01418690849095583,
+            -0.014656390994787216,
+            -0.0053307609632611275,
+            -0.014110084623098373,
+            0.0035125829745084047,
+            0.019342679530382156,
+            -0.026052013039588928,
+            -0.00018205786182079464,
+            -0.001230257679708302,
+            -6.235315231606364e-05,
+            -0.0032735737040638924,
+            0.012795533053576946,
+            0.016918443143367767,
+            -0.0067861564457416534,
+            -0.005117359571158886,
+            -0.0007324994658119977,
+            -0.017498893663287163,
+            -0.0004924231325276196,
+            0.021203538402915,
+            -0.0031433990225195885,
+            -0.005962428171187639,
+            -0.010934675112366676,
+            -0.038856081664562225,
+            -0.01078956201672554,
+            -0.01862565241754055,
+            -0.021544979885220528,
+            -0.006939805578440428,
+            0.0007244969019666314,
+            0.008028151467442513,
+            0.01381132286041975,
+            -0.010712738148868084,
+            -0.011096860282123089,
+            0.0033290579449385405,
+            -0.01622702367603779,
+            0.0016463898355141282,
+            -0.019342679530382156,
+            -0.0013614993076771498,
+            0.008783590979874134,
+            0.014212517067790031,
+            0.0044515482150018215,
+            0.00632094219326973,
+            0.0019995688926428556,
+            -0.011301725171506405,
+            -0.009978638030588627,
+            0.019189031794667244,
+            -0.022552233189344406,
+            0.003687571967020631,
+            -0.004865546245127916,
+            0.0065386113710701466,
+            -0.0034400266595184803,
+            -0.02159619703888893,
+            0.006367890164256096,
+            -0.02069137617945671,
+            0.010362760163843632,
+            -0.0010099209612235427,
+            -0.004950906615704298,
+            0.006658115889877081,
+            -0.0021521507296711206,
+            -0.011719991452991962,
+            0.01772083155810833,
+            -0.006777620408684015,
+            -0.004511300474405289,
+            -0.007191618904471397,
+            0.006244117859750986,
+            0.01588558219373226,
+            0.011037107557058334,
+            -0.0053819771856069565,
+            0.0062483856454491615,
+            0.011455373838543892,
+            -0.0026995246298611164,
+            -0.00767817348241806,
+            -0.01760132610797882,
+            0.00048788837739266455,
+            0.013956435024738312,
+            -0.020657232031226158,
+            0.0059282840229570866,
+            0.011967536993324757,
+            0.024310659617185593,
+            -0.007494648452848196,
+            0.006991021800786257,
+            -0.01913781464099884,
+            0.015561211854219437,
+            -0.00414425041526556,
+            -0.02250101789832115,
+            -0.0015524934278801084,
+            -0.013802786357700825,
+            0.0009058878640644252,
+            -0.005305152852088213,
+            0.00010189905151491985,
+            -0.0060435207560658455,
+            -0.0010563357500359416,
+            -0.004724701400846243,
+            -0.015262450091540813,
+            0.0013454942964017391,
+            -0.01610751822590828,
+            0.006790424697101116,
+            0.015288058668375015,
+            0.0144515261054039,
+            0.0028489052783697844,
+            0.00439606374129653,
+            -0.0030196262523531914,
+            0.002827565185725689,
+            -0.011916320770978928,
+            -0.004750309512019157,
+            0.007481844164431095,
+            -0.018284210935235023,
+            0.0005361703806556761,
+            -0.009338434785604477,
+            0.009099424816668034,
+            -0.022927820682525635,
+            0.003696108004078269,
+            -0.00207212520763278,
+            -0.014545422978699207,
+            -0.007072114385664463,
+            -0.007908646948635578,
+            0.023013180121779442,
+            0.010285935364663601,
+            0.01458810269832611,
+            0.005049071274697781,
+            -0.018130561336874962,
+            -0.01567218080163002,
+            0.0008263959898613393,
+            -0.001281473902054131,
+            0.02209128811955452,
+            -0.018932949751615524,
+            0.004464352037757635,
+            0.00844641774892807,
+            -0.02878354676067829,
+            -0.012283369898796082,
+            0.001568498439155519,
+            -0.01335891243070364,
+            0.009483546949923038,
+            0.009850597009062767,
+            -0.00800254289060831,
+            -0.0037729323375970125,
+            -0.004417404066771269,
+            -0.01630384847521782,
+            -0.006854445207864046,
+            -0.002769947052001953,
+            0.01169438287615776,
+            -0.009731092490255833,
+            0.0021916297264397144,
+            0.005445997230708599,
+            0.0121980095282197,
+            -0.007853162474930286,
+            -0.011207828298211098,
+            -0.010687129572033882,
+            0.002074259100481868,
+            0.015416099689900875,
+            -0.01823299378156662,
+            -0.010260327719151974,
+            0.015347810462117195,
+            -0.017515966668725014,
+            0.009893277660012245,
+            -0.00962012354284525,
+            -0.021732773631811142,
+            -0.006730672437697649,
+            0.016978194937109947,
+            0.000151514817844145,
+            0.02453259564936161,
+            0.01124197244644165,
+            -0.00622704578563571,
+            -0.006397766526788473,
+            -0.02349119819700718,
+            0.01201021671295166,
+            -0.006901393178850412,
+            0.0069739497266709805,
+            -0.000578317150939256,
+            -6.068595394026488e-05,
+            0.011711454950273037,
+            -0.001123557100072503,
+            -0.004107972141355276,
+            -0.011907784268260002,
+            -0.008087903261184692,
+            0.02951764687895775,
+            -0.007554400712251663,
+            0.017686687409877777,
+            0.01259066816419363,
+            -0.006884321104735136,
+            0.008361057378351688,
+            0.009048208594322205,
+            0.00388603494502604,
+            -0.010823706164956093,
+            0.01458810269832611,
+            -0.010806634090840816,
+            -0.016337992623448372,
+            0.008373861201107502,
+            0.007106258533895016,
+            -0.012027288787066936,
+            -0.008971384726464748,
+            0.0018619250040501356,
+            -0.0011854433687403798,
+            0.0003201017389073968,
+            -0.011011499911546707,
+            -0.009150641039013863,
+            -0.002554411767050624,
+            -0.005416121333837509,
+            0.0021830936893820763,
+            0.0027976890560239553,
+            0.008480561897158623,
+            0.002078527119010687,
+            -0.01645749621093273,
+            -0.01976948231458664,
+            -0.022279080003499985,
+            -0.015629500150680542,
+            0.00491249468177557,
+            0.0001392442500218749,
+            0.010268863290548325,
+            -0.007716585882008076,
+            0.014758823439478874,
+            -0.02079380862414837,
+            0.011412693187594414,
+            0.013324768282473087,
+            0.006508735008537769,
+            -0.013760106638073921,
+            -0.023098541423678398,
+            -0.012573596090078354,
+            0.0030004202853888273,
+            0.006952609401196241,
+            0.015125873498618603,
+            -0.00032917127828113735,
+            -0.008757983334362507,
+            0.00399700365960598,
+            -0.012573596090078354,
+            0.006769084371626377,
+            -0.00616302527487278,
+            0.0069227335043251514,
+            0.0011267580557614565,
+            -0.006901393178850412,
+            0.007985470816493034,
+            -0.005740491207689047,
+            0.0033717381302267313,
+            -0.008655550889670849,
+            -0.01663675345480442,
+            0.016431888565421104,
+            -0.0013518963241949677,
+            -0.007434896193444729,
+            -0.00860433466732502,
+            -0.004257353022694588,
+            -0.0015055451076477766,
+            -0.0036790359299629927,
+            0.009526227600872517,
+            0.011335869319736958,
+            0.027042193338274956,
+            0.00886895228177309,
+            0.008164728060364723,
+            0.004287228919565678,
+            -0.0005313688889145851,
+            0.013162583112716675,
+            0.018830517306923866,
+            0.00020873300672974437,
+            0.002744338707998395,
+            -0.0045539806596934795,
+            -0.02168155647814274,
+            -0.004562516696751118,
+            0.017686687409877777,
+            0.00937257893383503,
+            0.002082795137539506,
+            -0.007302587386220694,
+            0.0013999115908518434,
+            0.0019451514817774296,
+            -0.02299610897898674,
+            0.015919726341962814,
+            -0.005919747985899448,
+            -0.0041613224893808365,
+            0.015014905482530594,
+            -0.016594072803854942,
+            0.0052667404524981976,
+            -0.009176249615848064,
+            0.0031988832633942366,
+            0.0023346086964011192,
+            0.008267160505056381,
+            0.03762689232826233,
+            -0.01496368832886219,
+            0.006534343119710684,
+            0.021135250106453896,
+            -0.008629942312836647,
+            -0.017772048711776733,
+            -0.003045234363526106,
+            -0.006000840570777655,
+            0.004191198851913214,
+            0.012223618105053902,
+            0.0030921828001737595,
+            -0.0007869167602621019,
+            0.017515966668725014,
+            0.001877930131740868,
+            0.01206143293529749,
+            -0.014852720312774181,
+            -1.1286920198472217e-05,
+            -0.008937240578234196,
+            0.013410128653049469,
+            0.0019728937186300755,
+            -0.015902653336524963,
+            -0.002094532363116741,
+            0.02043529413640499,
+            -0.018215922638773918,
+            0.013717425987124443,
+            -0.006090468727052212,
+            0.02673489600419998,
+            0.004711897578090429,
+            0.0073153916746377945,
+            -0.004818597808480263,
+            0.009705484844744205,
+            0.013614993542432785,
+            0.024361874908208847,
+            -0.0015610294649377465,
+            0.002044382970780134,
+            0.0058557274751365185,
+            0.013691818341612816,
+            0.01610751822590828,
+            0.006478859111666679,
+            -0.0016549259889870882,
+            0.029978593811392784,
+            0.01720866933465004,
+            0.007306855637580156,
+            0.004438743926584721,
+            0.0001512480666860938,
+            -0.017200132831931114,
+            -0.005889871623367071,
+            -0.03168580308556557,
+            0.0036342216189950705,
+            0.013947899453341961,
+            0.010371295735239983,
+            -0.015108801424503326,
+            0.0083482526242733,
+            0.007050774060189724,
+            -0.018335426226258278,
+            0.013196727260947227,
+            0.0012217216426506639,
+            0.008826271630823612,
+            0.004387527704238892,
+            0.008006811141967773,
+            0.008510437794029713,
+            0.022159576416015625,
+            0.008783590979874134,
+            0.0063508180901408195,
+            -0.008006811141967773,
+            -0.01912074349820614,
+            0.0026995246298611164,
+            -0.016926979646086693,
+            0.001235592644661665,
+            0.010388367809355259,
+            -0.006141684949398041,
+            0.002078527119010687,
+            0.014357629232108593,
+            0.011113932356238365,
+            0.00018219123012386262,
+            -0.009807917289435863,
+            0.021374259144067764,
+            -0.0026162981521338224,
+            0.002697390504181385,
+            -0.02134011499583721,
+            0.0044515482150018215,
+            0.013649137690663338,
+            -0.00956037174910307,
+            0.012018753215670586,
+            -0.01951340213418007,
+            0.013469880446791649,
+            0.001887533231638372,
+            -0.01213825773447752,
+            -0.001184376422315836,
+            -0.010772489942610264,
+            0.023218045011162758,
+            -0.01574900560081005,
+            0.014084476046264172,
+            0.02490818314254284,
+            -0.002483989344909787,
+            -0.005569770000874996,
+            0.0022684542927891016,
+            0.012727244757115841,
+            -0.014605174772441387,
+            -0.006329478230327368,
+            0.0015738335205242038,
+            0.030354179441928864,
+            -0.009765236638486385,
+            0.002919327700510621,
+            -0.00886895228177309,
+            0.0199231319129467,
+            -0.01315404660999775,
+            -0.006760548334568739,
+            0.005612450186163187,
+            0.02673489600419998,
+            -0.00268245255574584,
+            0.027554357424378395,
+            0.016722114756703377,
+            0.0010526011465117335,
+            0.006171561311930418,
+            -0.01521977037191391,
+            0.006905661430209875,
+            0.017686687409877777,
+            0.01246262714266777,
+            0.017396461218595505,
+            -0.009048208594322205,
+            0.009543299674987793,
+            0.027537284418940544,
+            -0.00713186664506793,
+            0.0038177466485649347,
+            0.012308978475630283,
+            0.011156612075865269,
+            0.003747324226424098,
+            -0.0017541574779897928,
+            0.004259487148374319,
+            -0.015236842446029186,
+            0.013136974535882473,
+            -0.00943233072757721,
+            0.006393498741090298,
+            0.021510835736989975,
+            0.007925719022750854,
+            9.296288044424728e-05,
+            0.0006012577214278281,
+            0.012445555068552494,
+            0.03315400332212448,
+            -0.003975663334131241,
+            0.013768642209470272,
+            -0.0067861564457416534,
+            0.004250951111316681,
+            -0.01007253397256136,
+            -0.012872357852756977,
+            -0.0001271737419301644,
+            -0.014630783349275589,
+            -0.0010915468446910381,
+            7.54239663365297e-05,
+            0.008783590979874134,
+            -0.017618399113416672,
+            0.0015279522631317377,
+            -0.006833104882389307,
+            -0.007187350653111935,
+            -0.011865103617310524,
+            0.01137001346796751,
+            -0.010968819260597229,
+            -0.007332463748753071,
+            0.007281247526407242,
+            0.0076739052310585976,
+            -6.271993333939463e-05,
+            -0.02156205289065838,
+            -0.01059323363006115,
+            0.002249248092994094,
+            0.0036064793821424246,
+            0.014750287868082523,
+            -0.02067430317401886,
+            -0.004289363045245409,
+            0.01002131775021553,
+            -0.009133568964898586,
+            -0.008224480785429478,
+            0.0033717381302267313,
+            -0.01052494440227747,
+            -0.017703760415315628,
+            -0.012078505009412766,
+            0.00831837672740221,
+            -0.019803626462817192,
+            -0.029876161366701126,
+            -0.017635472118854523,
+            0.009329898282885551,
+            -0.016448961570858955,
+            -0.004519836511462927,
+            -0.025352057069540024,
+            0.02181813307106495,
+            0.008489097468554974,
+            0.011173684149980545,
+            0.006811764556914568,
+            -0.006547147408127785,
+            0.012949181720614433,
+            0.0010008513927459717,
+            -0.004289363045245409,
+            0.014050331898033619,
+            0.015424635261297226,
+            -0.010900530964136124,
+            -0.010866386815905571,
+            0.048757895827293396,
+            -0.004455816000699997,
+            0.008809199556708336,
+            -0.0011715723667293787,
+            0.02555692195892334,
+            0.022381512448191643,
+            0.01118222065269947,
+            0.009944493882358074,
+            0.026171516627073288,
+            -0.005813047289848328,
+            0.0008845477714203298,
+            0.018711013719439507,
+            0.002041182015091181,
+            0.028425034135580063,
+            0.016209950670599937,
+            -0.003617149544879794,
+            -0.008412273600697517,
+            -0.0032436975743621588,
+            0.0024818554520606995,
+            0.017353782430291176,
+            -0.003011090215295553,
+            -0.00116517033893615,
+            0.011976072564721107,
+            -0.018796373158693314,
+            -0.02606908418238163,
+            -0.028544537723064423,
+            -0.008096439763903618,
+            -0.003841220634058118,
+            0.0006578090251423419,
+            0.021135250106453896,
+            0.0029641420114785433,
+            0.0074690403416752815,
+            0.01937682554125786,
+            0.005283812526613474,
+            0.0044302078895270824,
+            0.004472888074815273,
+            -0.0020977333188056946,
+            -0.01002131775021553,
+            0.004835669882595539,
+            0.004472888074815273,
+            -0.0012729378649964929,
+            0.01953047327697277,
+            0.008126315660774708,
+            0.008736643008887768,
+            -0.003049502382054925,
+            0.002296196296811104,
+            0.006598363630473614,
+            0.018454931676387787,
+            0.0014372568111866713,
+            -0.026171516627073288,
+            0.008011079393327236,
+            0.009867669083178043,
+            -0.004639341030269861,
+            -0.004054621793329716,
+            0.003964993637055159,
+            -0.003877498907968402,
+            -0.004150652326643467,
+            -0.003719581989571452,
+            0.021203538402915,
+            0.022313224151730537,
+            0.02017921209335327,
+            0.003395212348550558,
+            0.012377266772091389,
+            0.0010584697593003511,
+            -0.02750314027070999,
+            0.007648297119885683,
+            -0.007055042311549187,
+            0.01418690849095583,
+            -0.0359196811914444,
+            -0.00670933211222291,
+            -0.004733237437903881,
+            0.001975027611479163,
+            0.03120778501033783,
+            0.0021478827111423016,
+            0.006564219482243061,
+            0.00552282202988863,
+            0.011959000490605831,
+            -0.004310703370720148,
+            0.024378947913646698,
+            -0.002678184537217021,
+            0.004737505689263344,
+            -0.002415701048448682,
+            -0.01084077823907137,
+            -0.020110923796892166,
+            0.01110539585351944,
+            0.008689695037901402,
+            0.0616643987596035,
+            -0.006513003259897232,
+            0.023047324270009995,
+            -0.01373449806123972,
+            0.007234299089759588,
+            0.0218522772192955,
+            0.024447236210107803,
+            0.02748606912791729,
+            -0.0010691398056223989,
+            0.004707629326730967,
+            -0.014647855423390865,
+            -0.018523219972848892,
+            0.009585979394614697,
+            -0.007242835126817226,
+            0.00597523245960474,
+            -0.022415656596422195,
+            -0.02016214095056057,
+            -0.008988456800580025,
+            -0.0057532950304448605,
+            -0.001239860663190484,
+            0.004618000704795122,
+            0.005309420637786388,
+            -0.0034549646079540253,
+            0.025864219292998314,
+            -0.0005332361324690282,
+            -0.0033141199965029955,
+            -0.028595754876732826,
+            -0.009184785187244415,
+            0.011890712194144726,
+            -0.05049924924969673,
+            0.018779302015900612,
+            -0.00272940075956285,
+            -0.02748606912791729,
+            -0.01744767837226391,
+            -0.0035595311783254147,
+            0.015629500150680542,
+            -0.012334587052464485,
+            0.010610305704176426,
+            0.0026205661706626415,
+            -0.0006882187444716692,
+            0.0037772003561258316,
+            0.014545422978699207,
+            0.0224839448928833,
+            0.004370455630123615,
+            0.009329898282885551,
+            0.005556966178119183,
+            0.009978638030588627,
+            -0.007580008823424578,
+            0.019974347203969955,
+            0.001809641718864441,
+            0.0045326403342187405,
+            0.0020337130408734083,
+            -0.016961123794317245,
+            -0.004536908585578203,
+            -0.006017912644892931,
+            0.028083590790629387,
+            0.015339274890720844,
+            -0.01041397638618946,
+            -0.005347833037376404,
+            -0.005569770000874996,
+            -0.011924856342375278,
+            0.0013028139946982265,
+            0.013512561097741127,
+            0.010448120534420013,
+            -0.038856081664562225,
+            0.03317107632756233,
+            0.01728549413383007,
+            -0.01848907582461834,
+            -0.024208227172493935,
+            -0.002460515359416604,
+            0.020708447322249413,
+            0.023832641541957855,
+            -0.03271012753248215,
+            -0.013427200727164745,
+            0.014468598179519176,
+            -0.014383237808942795,
+            6.825502350693569e-05,
+            -0.0005495079676620662,
+            -0.0205547995865345,
+            0.011250508949160576,
+            -0.014810039661824703,
+            0.019291464239358902,
+            -0.0025928239338099957,
+            0.022518089041113853,
+            0.025727642700076103,
+            -0.004528372548520565,
+            0.0018352499464526772,
+            -0.01355524081736803,
+            -0.008207408711314201,
+            0.03277841582894325,
+            0.0018053737003356218,
+            -0.005428925156593323,
+            -0.007917182520031929,
+            -0.0023986289743334055,
+            -0.012676028534770012,
+            0.01490393653512001,
+            0.007349535822868347,
+            0.0012035825056955218,
+            0.005241132341325283,
+            -0.0010878123575821519,
+            0.025215480476617813,
+            0.004763113800436258,
+            -0.004763113800436258,
+            0.008834808133542538,
+            -0.015108801424503326,
+            0.01937682554125786,
+            -0.0029022556263953447,
+            -0.008361057378351688,
+            0.0053179566748440266,
+            0.014656390994787216,
+            0.004792989697307348,
+            -0.0038518907967954874,
+            -0.010482264682650566,
+            -0.004857010208070278,
+            0.007669637445360422,
+            0.027059266343712807,
+            0.014519814401865005,
+            -0.007597080897539854,
+            -0.011412693187594414,
+            0.01451127789914608,
+            0.007883038371801376,
+            0.0006924867630004883,
+            -0.0071788146160542965,
+            0.012095577083528042,
+            -0.01554413978010416,
+            0.014681999571621418,
+            -0.009346970357000828,
+            -0.020862096920609474,
+            0.008988456800580025,
+            -0.02506183087825775,
+            0.0017125442391261458,
+            -0.01835249923169613,
+            0.002155351685360074,
+            -0.007968398742377758,
+            -0.011395621113479137,
+            0.009500619024038315,
+            -0.000326237000990659,
+            0.03878779336810112,
+            0.011190756224095821,
+            0.009679876267910004,
+            -0.0014489938039332628,
+            0.07416117191314697,
+            -0.023439982905983925,
+            -0.01033715158700943,
+            -0.028954267501831055,
+            -0.012949181720614433,
+            -0.0063636223785579205,
+            0.01002131775021553,
+            0.008104975335299969,
+            -0.0004057289334014058,
+            -0.011301725171506405,
+            -0.007072114385664463,
+            0.015783149749040604,
+            -0.026615392416715622,
+            -0.0007479710620827973,
+            0.02159619703888893,
+            0.0030729766003787518,
+            0.006564219482243061,
+            0.015006368979811668,
+            -0.007268443237990141,
+            0.01966704986989498,
+            -0.016346529126167297,
+            -0.012744316831231117,
+            0.005894139874726534,
+            0.016329456120729446,
+            0.005339297000318766,
+            0.0008904163260012865,
+            0.011848031543195248,
+            -0.0006097938166931272,
+            0.005108823534101248,
+            0.02323511801660061,
+            -0.02014506794512272,
+            0.014417381957173347,
+            -0.008412273600697517,
+            0.01825006678700447,
+            0.007644029334187508,
+            0.018028128892183304,
+            -0.005134431645274162,
+            -0.008877487853169441,
+            -0.008442149497568607,
+            0.010943210683763027,
+            0.015962406992912292,
+            -0.012215081602334976,
+            -0.0019686257001012564,
+            0.004048219881951809,
+            -0.018028128892183304,
+            -0.006897125393152237,
+            -0.012180937454104424,
+            0.008049491792917252,
+            0.021510835736989975,
+            -0.012471163645386696,
+            -0.01675625890493393,
+            -0.022176647558808327,
+            0.012317514047026634,
+            -0.008190336637198925,
+            0.02772507816553116,
+            -0.013623529113829136,
+            -0.00523686408996582,
+            0.0211864672601223,
+            0.0004182662523817271,
+            0.0012174536241218448,
+            -0.027776293456554413,
+            -0.010166430845856667,
+            -0.014997833408415318
+          ],
+          "title": "6 Related Work"
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "be3079d3-622e-4848-bff6-15e3ac00f52b",
+        "properties": {
+          "page_content": "7 Discussion 25 ",
+          "keyphrases": [
+            "Discussion"
+          ],
+          "title": "7 Discussion",
+          "embedding": [
+            -0.015314480289816856,
+            0.01669096015393734,
+            -0.018997248262166977,
+            0.02089332602918148,
+            -0.021677281707525253,
+            -0.005907013546675444,
+            -0.010774830356240273,
+            0.0937100350856781,
+            -0.00431403424590826,
+            0.05294434353709221,
+            0.0064402855932712555,
+            -0.027529601007699966,
+            -0.014475829899311066,
+            -0.01842295378446579,
+            -0.024448473006486893,
+            0.03755693882703781,
+            -0.012051037512719631,
+            0.033710088580846786,
+            -0.006408380344510078,
+            -0.030100245028734207,
+            0.01741110533475876,
+            -0.012816760689020157,
+            0.011850490234792233,
+            0.07365535199642181,
+            -0.013801263645291328,
+            0.025287123396992683,
+            0.04072922468185425,
+            0.02200544998049736,
+            -0.03755693882703781,
+            0.04565173387527466,
+            -0.010009106248617172,
+            0.004895164165645838,
+            -0.0015485400799661875,
+            -0.04878755658864975,
+            -0.024029148742556572,
+            0.00932086631655693,
+            0.03347307816147804,
+            -0.010191421955823898,
+            0.033710088580846786,
+            0.02512303926050663,
+            -0.030592497438192368,
+            -0.039453018456697464,
+            -0.0070100207813084126,
+            0.0217137448489666,
+            0.010382852517068386,
+            -0.006080213002860546,
+            -0.025505902245640755,
+            -0.018559690564870834,
+            0.0007360977469943464,
+            0.012707372196018696,
+            -0.016827696934342384,
+            -0.04269822686910629,
+            0.01159524917602539,
+            0.0005062096752226353,
+            0.014184125699102879,
+            -0.00020211352966725826,
+            0.011440280824899673,
+            0.034111179411411285,
+            0.00420920317992568,
+            -0.02865995466709137,
+            0.003798993770033121,
+            -0.044375527650117874,
+            -0.004564717877656221,
+            -0.004081582184880972,
+            -0.016408370807766914,
+            -0.02123972587287426,
+            -0.013500443659722805,
+            0.01520509086549282,
+            -0.02880580723285675,
+            -0.021130336448550224,
+            -0.014421135187149048,
+            -0.008003639057278633,
+            -0.0213126502931118,
+            0.04120324179530144,
+            -0.017447568476200104,
+            -0.012944381684064865,
+            0.03330899402499199,
+            0.01408385206013918,
+            -0.019799435511231422,
+            0.044448453933000565,
+            0.02428438887000084,
+            -0.019070174545049667,
+            -0.025378281250596046,
+            0.030829505994915962,
+            0.029790310189127922,
+            -0.01874200627207756,
+            -0.06716493517160416,
+            -0.11777564138174057,
+            0.026891497895121574,
+            0.022114837542176247,
+            0.006271644029766321,
+            -0.012962613254785538,
+            -0.012224236503243446,
+            -0.0341658741235733,
+            -0.009398350492119789,
+            -0.00598905561491847,
+            -0.01561530027538538,
+            -0.0026207815390080214,
+            0.023099340498447418,
+            -0.012780298478901386,
+            0.03454873710870743,
+            0.0032497690990567207,
+            0.0071695465594530106,
+            0.024685483425855637,
+            0.022461237385869026,
+            -0.00037574031739495695,
+            0.02880580723285675,
+            0.016700075939297676,
+            -0.0015508190263062716,
+            0.007880575954914093,
+            -0.024612557142972946,
+            0.022953487932682037,
+            -0.030009087175130844,
+            -0.010428431443870068,
+            0.00737465126439929,
+            -0.015505910851061344,
+            0.006800358183681965,
+            -0.04754781350493431,
+            -0.017155863344669342,
+            0.016453949734568596,
+            -0.0017046474386006594,
+            -0.005405646748840809,
+            -0.03053780272603035,
+            -0.02102094702422619,
+            -0.027183201164007187,
+            0.006253412459045649,
+            4.162627010373399e-05,
+            -0.011121229268610477,
+            -0.005282584112137556,
+            0.016435718163847923,
+            0.006668179761618376,
+            0.04557880759239197,
+            -0.02499541826546192,
+            0.029188670217990875,
+            0.002563808113336563,
+            -0.005733814090490341,
+            0.002456697868183255,
+            0.02665448747575283,
+            -0.008040102198719978,
+            -0.01068367250263691,
+            -0.02235184796154499,
+            -0.003769367467612028,
+            0.006385591346770525,
+            0.02900635451078415,
+            -0.01597081497311592,
+            0.023901527747511864,
+            -0.0013240644475445151,
+            0.021640818566083908,
+            0.006157697178423405,
+            0.021549660712480545,
+            0.0255423653870821,
+            -0.012470361776649952,
+            -0.0430263951420784,
+            0.03489513695240021,
+            0.006426611915230751,
+            -0.010510473512113094,
+            -0.009380118921399117,
+            0.014931618236005306,
+            -0.012133078649640083,
+            0.010865988209843636,
+            0.015414752997457981,
+            0.06450313329696655,
+            -0.022096605971455574,
+            -0.0035095682833343744,
+            0.04204189404845238,
+            -0.01353690680116415,
+            0.02561529166996479,
+            -0.008409290574491024,
+            -0.04204189404845238,
+            0.019544193521142006,
+            0.04448491707444191,
+            0.008309016935527325,
+            0.004799448419362307,
+            -0.012333625927567482,
+            0.008514121174812317,
+            0.02242477424442768,
+            -0.003452594857662916,
+            -0.0010278021218255162,
+            -0.014074736274778843,
+            0.023281656205654144,
+            -0.04393797367811203,
+            0.003309021471068263,
+            0.012762066908180714,
+            -0.0032725585624575615,
+            -0.034311726689338684,
+            -0.01408385206013918,
+            0.04587051272392273,
+            -0.02650863490998745,
+            0.04572466015815735,
+            0.010182306170463562,
+            -0.0075751980766654015,
+            -0.002725612837821245,
+            -0.007570640183985233,
+            -0.01061986293643713,
+            -0.014995427802205086,
+            -0.06351862847805023,
+            0.01214219443500042,
+            0.011686407029628754,
+            0.0002243331982754171,
+            0.01436644047498703,
+            -0.021677281707525253,
+            -0.013646295294165611,
+            -0.00037232189788483083,
+            0.007383767049759626,
+            -0.011987227015197277,
+            -0.0016602081013843417,
+            0.010364620946347713,
+            0.022734710946679115,
+            0.03617134317755699,
+            -0.024867799133062363,
+            0.010701904073357582,
+            0.02435731515288353,
+            -0.022151300683617592,
+            0.04054690897464752,
+            0.01696443371474743,
+            0.008126701228320599,
+            -0.019197793677449226,
+            0.06319046020507812,
+            0.01896078512072563,
+            -0.040218740701675415,
+            0.025432975962758064,
+            -0.000753759581129998,
+            0.01708293706178665,
+            0.026745645329356194,
+            -0.012634445913136005,
+            0.026071079075336456,
+            -0.02359159104526043,
+            0.021549660712480545,
+            -0.01819506101310253,
+            -0.052907880395650864,
+            0.02603461593389511,
+            0.014566987752914429,
+            -0.01453964039683342,
+            -0.010100264102220535,
+            -0.041677262634038925,
+            -0.026198700070381165,
+            -0.025761142373085022,
+            0.0063126650638878345,
+            0.024047380313277245,
+            0.03068365529179573,
+            0.029735615476965904,
+            -0.010109379887580872,
+            -0.040984466671943665,
+            0.010364620946347713,
+            -0.054803960025310516,
+            0.020783936604857445,
+            -0.025141270831227303,
+            0.029553299769759178,
+            0.0002243331982754171,
+            -0.010920682922005653,
+            0.020072907209396362,
+            -0.007365535479038954,
+            0.012406552210450172,
+            0.029808541759848595,
+            -0.008204185403883457,
+            0.045979902148246765,
+            0.04481308534741402,
+            -0.0034844998735934496,
+            -0.0023837718181312084,
+            0.004040561616420746,
+            -0.018204176798462868,
+            -0.010437547229230404,
+            -0.032379187643527985,
+            -0.014074736274778843,
+            0.02138557657599449,
+            -0.01175933238118887,
+            0.021458502858877182,
+            0.004427981097251177,
+            0.026472171768546104,
+            -0.009024604223668575,
+            0.02568821609020233,
+            -0.01901547983288765,
+            0.010391968302428722,
+            0.005587961990386248,
+            -0.027092043310403824,
+            -0.014402903616428375,
+            0.01896078512072563,
+            -0.005309931468218565,
+            0.004931627307087183,
+            0.032999057322740555,
+            0.00916134100407362,
+            0.03061072900891304,
+            -0.0073883249424397945,
+            -0.025943458080291748,
+            -0.010556052438914776,
+            -0.035223305225372314,
+            -0.016052857041358948,
+            0.024813104420900345,
+            -0.012716487981379032,
+            -0.0217137448489666,
+            0.009999990463256836,
+            -0.005027342587709427,
+            -0.0037306256126612425,
+            0.017228789627552032,
+            0.011431165039539337,
+            -0.009234266355633736,
+            0.021330881863832474,
+            0.005095710977911949,
+            -0.0022117116022855043,
+            0.005628983024507761,
+            -0.009735633619129658,
+            0.021257957443594933,
+            -0.024667251855134964,
+            0.001667044940404594,
+            -0.009516855701804161,
+            0.01627163402736187,
+            0.005911571439355612,
+            0.009343655779957771,
+            -0.0852506011724472,
+            -0.020236991345882416,
+            0.06257058680057526,
+            -0.03839558735489845,
+            -0.009361887350678444,
+            0.009972643107175827,
+            -0.017000894993543625,
+            0.019252488389611244,
+            0.02497718669474125,
+            0.013710105791687965,
+            -0.01644483394920826,
+            0.0024361873511224985,
+            0.02082039974629879,
+            0.005670004058629274,
+            -0.029735615476965904,
+            -0.004749312065541744,
+            0.0068185897544026375,
+            0.002308566588908434,
+            -0.024047380313277245,
+            0.027092043310403824,
+            0.006882400251924992,
+            0.0013126697158440948,
+            0.023792138323187828,
+            -0.03234272450208664,
+            0.005856877192854881,
+            -0.004876932594925165,
+            -0.00793527066707611,
+            0.004968089982867241,
+            -0.041677262634038925,
+            0.03624426946043968,
+            0.002105741063132882,
+            -0.030993590131402016,
+            0.01408385206013918,
+            0.009817675687372684,
+            0.011777563951909542,
+            0.02220599539577961,
+            -0.012579751200973988,
+            -0.006855052895843983,
+            0.01461256667971611,
+            -0.023281656205654144,
+            4.2480874981265515e-05,
+            0.010364620946347713,
+            0.000994757516309619,
+            0.006595253478735685,
+            -0.013582484796643257,
+            0.014949849806725979,
+            -0.0014881481183692813,
+            -0.035843174904584885,
+            -0.05560614541172981,
+            -0.014448482543230057,
+            0.0003068023652303964,
+            9.008936467580497e-05,
+            0.04477662220597267,
+            -0.013381938450038433,
+            -0.00829534325748682,
+            0.0055834040977060795,
+            -0.029024586081504822,
+            -0.07099355012178421,
+            0.035369157791137695,
+            0.022661784663796425,
+            -0.024940723553299904,
+            0.014758418314158916,
+            -0.03228802978992462,
+            -0.02789423055946827,
+            0.01589788869023323,
+            -0.020492233335971832,
+            -0.017119400203227997,
+            0.0018675917526707053,
+            -0.007629892788827419,
+            -0.005888781975954771,
+            -0.03201455622911453,
+            0.026417477056384087,
+            -0.02269824780523777,
+            0.01121238712221384,
+            -0.040911540389060974,
+            -0.013281664811074734,
+            0.005510477814823389,
+            -2.746301333900192e-06,
+            0.01567910984158516,
+            0.026289857923984528,
+            -0.02116679958999157,
+            0.00796261802315712,
+            -0.04754781350493431,
+            -0.012379204854369164,
+            -0.006859610788524151,
+            -0.017265252768993378,
+            0.03108474798500538,
+            0.006203276105225086,
+            0.01651776023209095,
+            -0.00870555266737938,
+            0.036262501031160355,
+            -0.030446644872426987,
+            -0.00527346832677722,
+            0.018577922135591507,
+            -0.006581580266356468,
+            -0.017401989549398422,
+            0.04196896776556969,
+            0.00760254543274641,
+            -0.019598888233304024,
+            -0.0020943463314324617,
+            -0.005838645622134209,
+            0.0072287991642951965,
+            0.009726517833769321,
+            -0.009917949326336384,
+            0.036116648465394974,
+            -0.011185039766132832,
+            0.06468544900417328,
+            0.0025205081328749657,
+            0.04565173387527466,
+            0.05367360636591911,
+            -0.02971738390624523,
+            -0.01061986293643713,
+            0.007739281747490168,
+            0.0238103698939085,
+            0.026818571612238884,
+            -0.017219673842191696,
+            -0.025232428684830666,
+            0.03615311160683632,
+            -0.015788499265909195,
+            -0.0039106616750359535,
+            0.010319042019546032,
+            -0.030556034296751022,
+            -0.028714649379253387,
+            0.04200543090701103,
+            -0.020528696477413177,
+            -0.007775744888931513,
+            0.018632616847753525,
+            -0.01936187781393528,
+            0.005574288312345743,
+            0.008422964252531528,
+            -0.03174108266830444,
+            0.024029148742556572,
+            0.010154958814382553,
+            -0.005246120970696211,
+            -0.015569721348583698,
+            0.046052828431129456,
+            -0.017593421041965485,
+            0.0016066529788076878,
+            -0.031412914395332336,
+            -0.021203262731432915,
+            -0.04477662220597267,
+            0.035843174904584885,
+            -0.03547854349017143,
+            0.029899699613451958,
+            -0.038213275372982025,
+            0.010574284009635448,
+            0.036061953753232956,
+            -0.00958066526800394,
+            0.0019553308375179768,
+            0.018924321979284286,
+            -0.020510464906692505,
+            -0.06891515851020813,
+            0.0106563251465559,
+            0.022734710946679115,
+            0.021002715453505516,
+            -0.015405637212097645,
+            -0.0364995114505291,
+            0.03330899402499199,
+            -0.0038628040347248316,
+            -0.03163169324398041,
+            0.005783950909972191,
+            0.018796700984239578,
+            -0.015834078192710876,
+            -0.010382852517068386,
+            -0.05425701290369034,
+            -0.005382857285439968,
+            -0.0376298651099205,
+            -0.024047380313277245,
+            0.0347675159573555,
+            -0.013117581605911255,
+            -0.04510479047894478,
+            0.03901546075940132,
+            -0.033837709575891495,
+            0.011996342800557613,
+            -0.0033477635588496923,
+            0.022224226966500282,
+            -0.02242477424442768,
+            0.015587952919304371,
+            0.042370062321424484,
+            -0.041458483785390854,
+            0.004712848924100399,
+            0.039380092173814774,
+            0.03296259418129921,
+            -0.01585230976343155,
+            0.06618043035268784,
+            -0.005820414051413536,
+            -0.013728337362408638,
+            -0.06771188229322433,
+            -0.00018388200260233134,
+            -0.07817677408456802,
+            -0.001375340623781085,
+            0.03365539386868477,
+            -0.008901541121304035,
+            -0.019325414672493935,
+            -0.033071983605623245,
+            0.03312667831778526,
+            0.009206918999552727,
+            -0.025177733972668648,
+            0.011713754385709763,
+            -0.04579758644104004,
+            -0.010802177712321281,
+            0.0192160252481699,
+            0.027985388413071632,
+            0.005551498848944902,
+            0.00091556430561468,
+            -0.012105731293559074,
+            -0.019197793677449226,
+            -0.007543292827904224,
+            -0.016645381227135658,
+            0.03132175654172897,
+            -0.004416586831212044,
+            0.012652677483856678,
+            0.0019040547776967287,
+            -0.007648124359548092,
+            -0.03387417271733284,
+            0.04324517399072647,
+            -0.01773015595972538,
+            0.012570635415613651,
+            0.013181391172111034,
+            -0.008810384199023247,
+            -0.0016602081013843417,
+            0.011257966049015522,
+            0.0009468997595831752,
+            0.004343660548329353,
+            0.006030076649039984,
+            0.022187763825058937,
+            0.031212368980050087,
+            -0.0015850031049922109,
+            0.0024908818304538727,
+            0.01916133053600788,
+            0.007939828559756279,
+            0.026490403339266777,
+            -0.0023814928717911243,
+            0.024193231016397476,
+            -0.030628960579633713,
+            -0.028295325115323067,
+            0.007643566466867924,
+            0.01916133053600788,
+            -0.00929807685315609,
+            -0.00896535161882639,
+            -0.015861425548791885,
+            -0.02915220707654953,
+            0.02386506460607052,
+            -0.0023792139254510403,
+            0.008290785364806652,
+            -0.01992705464363098,
+            0.0021137173753231764,
+            -0.0013696432579308748,
+            0.0315040722489357,
+            0.019416572526097298,
+            0.013728337362408638,
+            -0.004619412124156952,
+            0.011084766127169132,
+            0.0028988122940063477,
+            -0.027164969593286514,
+            0.0028646280989050865,
+            -0.011048302985727787,
+            -0.007023694459348917,
+            -0.014348208904266357,
+            0.043682731688022614,
+            0.004453049506992102,
+            -0.025031881406903267,
+            0.009571549482643604,
+            0.00199065450578928,
+            -0.01679123379290104,
+            -0.009097530506551266,
+            -0.0017673183465376496,
+            0.033965326845645905,
+            0.027875998988747597,
+            -0.028204167261719704,
+            -0.02499541826546192,
+            0.006207833997905254,
+            0.013783032074570656,
+            0.002743844408541918,
+            0.009872370399534702,
+            -0.004425702150911093,
+            -5.647499347105622e-05,
+            -0.01207838486880064,
+            -0.009389234706759453,
+            -0.02025522291660309,
+            -0.009617128409445286,
+            0.020802168175578117,
+            0.025998152792453766,
+            0.020364612340927124,
+            0.031066516414284706,
+            -0.04634453356266022,
+            -0.010957146063446999,
+            0.016453949734568596,
+            0.04856877774000168,
+            0.025159502401947975,
+            -0.006408380344510078,
+            -0.03290789946913719,
+            -0.008787594735622406,
+            0.010893335565924644,
+            -0.010200537741184235,
+            0.012461245991289616,
+            -0.01731083169579506,
+            0.004026887938380241,
+            -0.03194162994623184,
+            0.013427517376840115,
+            -0.0015154953580349684,
+            0.02700088731944561,
+            -0.02020052820444107,
+            0.012169541791081429,
+            0.01894255355000496,
+            -0.02957153134047985,
+            -0.011987227015197277,
+            -0.0315040722489357,
+            -0.003691883524879813,
+            0.001616908237338066,
+            -0.01589788869023323,
+            0.0061440235003829,
+            0.014402903616428375,
+            -0.010492241941392422,
+            0.0011018677614629269,
+            0.024466704577207565,
+            0.019106637686491013,
+            0.0077939764596521854,
+            -0.00453053368255496,
+            0.0063400124199688435,
+            -0.009361887350678444,
+            -0.028623491525650024,
+            -0.03219687193632126,
+            0.018331797793507576,
+            0.02180490270256996,
+            0.003411573823541403,
+            -0.006175928749144077,
+            0.005095710977911949,
+            0.057283446192741394,
+            -0.026891497895121574,
+            0.018833164125680923,
+            0.0016043740324676037,
+            -0.03901546075940132,
+            -0.031832240521907806,
+            -0.003343205666169524,
+            0.06089328974485397,
+            -0.028222398832440376,
+            0.01121238712221384,
+            0.015606184490025043,
+            0.01817682944238186,
+            0.002261848421767354,
+            0.0003395621315576136,
+            -0.034184105694293976,
+            0.014329977333545685,
+            -0.01207838486880064,
+            0.01894255355000496,
+            0.031394682824611664,
+            -0.04072922468185425,
+            -0.004804006312042475,
+            -0.021950755268335342,
+            -0.0012830435298383236,
+            -0.024029148742556572,
+            -0.024175001308321953,
+            0.00619416031986475,
+            0.00550136249512434,
+            0.0071695465594530106,
+            -0.0047265226021409035,
+            -0.006444843485951424,
+            -0.0015838636318221688,
+            -0.00231198500841856,
+            0.00924794003367424,
+            -0.007014578673988581,
+            0.0008050356991589069,
+            -0.01159524917602539,
+            0.01718321070075035,
+            0.0007492016884498298,
+            -0.03832266107201576,
+            -0.013491327874362469,
+            -0.020036444067955017,
+            0.011950763873755932,
+            -0.0008283948409371078,
+            -0.01908840611577034,
+            -0.020036444067955017,
+            -0.011340008117258549,
+            0.010948030278086662,
+            0.01696443371474743,
+            -0.02123972587287426,
+            0.0044781179167330265,
+            0.008669089525938034,
+            -0.0035095682833343744,
+            -0.007680029142647982,
+            -0.0018493601819500327,
+            0.0215131975710392,
+            0.04940742999315262,
+            0.004020051099359989,
+            0.006016402970999479,
+            -0.007771186996251345,
+            -0.017338179051876068,
+            -0.04102092981338501,
+            0.03923423960804939,
+            0.01339105423539877,
+            0.029735615476965904,
+            -0.00248176627792418,
+            -0.04058337211608887,
+            -0.0016624870477244258,
+            -0.01783042959868908,
+            0.004172740038484335,
+            -0.0034298053942620754,
+            -0.008637184277176857,
+            0.004475838970392942,
+            -0.017319947481155396,
+            -0.0326344259083271,
+            -0.016873275861144066,
+            -0.029334520921111107,
+            0.0009634220623411238,
+            0.022661784663796425,
+            -0.021604355424642563,
+            0.009361887350678444,
+            -0.0025979920756071806,
+            -0.039963498711586,
+            -0.007433903869241476,
+            -0.003502731444314122,
+            -0.01398357842117548,
+            0.013719221577048302,
+            -0.014220588840544224,
+            0.0023245192132890224,
+            -0.005054689943790436,
+            0.01467637624591589,
+            0.025232428684830666,
+            -0.02138557657599449,
+            0.005373741500079632,
+            -0.0001051873477990739,
+            -0.00041220334242098033,
+            -0.013609832152724266,
+            0.0005908153252676129,
+            -0.0012819040566682816,
+            -0.020109370350837708,
+            0.018131250515580177,
+            0.004927069414407015,
+            0.02776661142706871,
+            0.016636265441775322,
+            -0.03312667831778526,
+            0.010072916746139526,
+            -0.002771191531792283,
+            -0.01696443371474743,
+            0.015496795065701008,
+            -0.012898802757263184,
+            0.0008266856311820447,
+            0.02422969415783882,
+            0.017620768398046494,
+            -0.018377376720309258,
+            0.003115311497822404,
+            0.00978121254593134,
+            -0.04247945174574852,
+            0.062351811677217484,
+            -0.00193937832955271,
+            -0.03347307816147804,
+            0.016499528661370277,
+            0.018304450437426567,
+            0.01673653908073902,
+            -0.0045669968239963055,
+            0.00888786744326353,
+            -0.002016862388700247,
+            0.03555146977305412,
+            0.012944381684064865,
+            0.04776659235358238,
+            0.029407447203993797,
+            -0.014566987752914429,
+            -0.006458517163991928,
+            -0.01819506101310253,
+            -0.00801275484263897,
+            0.01683681271970272,
+            0.008960793726146221,
+            0.01742933690547943,
+            0.021458502858877182,
+            -0.013546021655201912,
+            0.015861425548791885,
+            0.005382857285439968,
+            0.00898814108222723,
+            -0.05330897495150566,
+            -0.0430263951420784,
+            0.007543292827904224,
+            -0.01565176248550415,
+            -0.033071983605623245,
+            -0.005929803010076284,
+            -0.004872374702244997,
+            -0.022607089951634407,
+            0.0025205081328749657,
+            0.041057392954826355,
+            -0.024175001308321953,
+            0.027711916714906693,
+            0.007306283339858055,
+            0.006303549278527498,
+            0.0255423653870821,
+            -0.02200544998049736,
+            -0.00387875665910542,
+            0.043974436819553375,
+            -0.04933450371026993,
+            0.009516855701804161,
+            -0.0050455741584300995,
+            -0.009197804145514965,
+            -0.015050122514367104,
+            0.0315040722489357,
+            0.009653591550886631,
+            -0.0064721908420324326,
+            0.015761151909828186,
+            0.01950773037970066,
+            0.0006551953847520053,
+            0.016326328739523888,
+            -0.020528696477413177,
+            -0.034238800406455994,
+            0.030027318745851517,
+            0.0009104367345571518,
+            -0.008144932799041271,
+            0.0414949469268322,
+            0.009945295751094818,
+            0.010975377634167671,
+            0.0034252475015819073,
+            -0.007392882835119963,
+            -0.026125773787498474,
+            0.00699178921058774,
+            0.023226961493492126,
+            0.025925226509571075,
+            0.011768448166549206,
+            0.020419307053089142,
+            -0.006025518756359816,
+            0.008286227472126484,
+            0.04732903465628624,
+            0.008760247379541397,
+            0.006900631822645664,
+            0.042588841170072556,
+            0.035369157791137695,
+            -0.02484956756234169,
+            -0.020984483882784843,
+            -0.00975386518985033,
+            0.022807635366916656,
+            0.0006956465658731759,
+            0.02054692804813385,
+            0.011221502907574177,
+            -0.0035665417090058327,
+            -0.005460341461002827,
+            0.030501339584589005,
+            -0.014393787831068039,
+            0.002228803699836135,
+            -0.0006791242631152272,
+            -0.017392873764038086,
+            -0.003037827555090189,
+            0.004402913153171539,
+            -0.020437538623809814,
+            0.009279845282435417,
+            0.02422969415783882,
+            0.00045806702109985054,
+            0.018979016691446304,
+            0.019124869257211685,
+            0.006358243990689516,
+            -0.0029717383440583944,
+            -0.024302620440721512,
+            -0.05061070993542671,
+            0.021257957443594933,
+            0.019343646243214607,
+            0.0026891496963799,
+            0.0029238807037472725,
+            0.012944381684064865,
+            0.024685483425855637,
+            -0.00985413882881403,
+            -0.02032814919948578,
+            0.03773925453424454,
+            -0.02319049835205078,
+            -0.008824057877063751,
+            -0.03159523010253906,
+            0.006791242398321629,
+            -0.023500433191657066,
+            -0.01207838486880064,
+            -0.020164065062999725,
+            -0.0066317166201770306,
+            0.0058340877294540405,
+            0.002695986535400152,
+            -0.0036235153675079346,
+            -0.012552403844892979,
+            0.001857336494140327,
+            0.004516860004514456,
+            -0.031759314239025116,
+            -0.02123972587287426,
+            -0.027456674724817276,
+            -0.0253418181091547,
+            -0.006207833997905254,
+            -0.02242477424442768,
+            0.003976751118898392,
+            -0.01965358294546604,
+            -0.046818554401397705,
+            -0.008546026423573494,
+            -0.010583399794995785,
+            0.0036736519541591406,
+            -0.006367359776049852,
+            0.014958965592086315,
+            -0.03157699853181839,
+            0.0030606170184910297,
+            0.003632631152868271,
+            -0.032105714082717896,
+            0.011896069161593914,
+            -0.011750216595828533,
+            -0.005565172526985407,
+            -0.00603919243440032,
+            0.0063400124199688435,
+            0.019471267238259315,
+            -0.032105714082717896,
+            -0.010391968302428722,
+            -0.028550565242767334,
+            0.024594325572252274,
+            0.004010935313999653,
+            0.014557871967554092,
+            0.006125791929662228,
+            -0.027365516871213913,
+            -0.02275294065475464,
+            -0.03713761270046234,
+            -0.02816770412027836,
+            0.010975377634167671,
+            0.02282586693763733,
+            -0.0024133978877216578,
+            0.016390139237046242,
+            -0.0016271634958684444,
+            0.024138538166880608,
+            0.010273464024066925,
+            -0.0008152909576892853,
+            -0.029024586081504822,
+            0.009644475765526295,
+            0.00855514220893383,
+            -0.04350041598081589,
+            -0.030009087175130844,
+            0.023026414215564728,
+            -0.013400170020759106,
+            0.009945295751094818,
+            0.021494966000318527,
+            0.017839545384049416,
+            0.026526866480708122,
+            0.006941652856767178,
+            0.027748379856348038,
+            -0.0115223228931427,
+            -0.016973547637462616,
+            0.031194137409329414,
+            0.015551489777863026,
+            -0.012725603766739368,
+            0.02712850645184517,
+            0.01391065213829279,
+            0.0023792139254510403,
+            0.008318132720887661,
+            0.021494966000318527,
+            0.005305373575538397,
+            0.0013343195896595716,
+            0.04977206140756607,
+            0.01817682944238186,
+            -0.008842289447784424,
+            -0.019525961950421333,
+            0.007429345976561308,
+            0.00982679147273302,
+            -0.007338188122957945,
+            0.021987218409776688,
+            0.00015268901188392192,
+            -0.04503186419606209,
+            0.01824975572526455,
+            0.019124869257211685,
+            0.024065611883997917,
+            0.0017057869117707014,
+            0.008277111686766148,
+            -0.027930693700909615,
+            0.00732451444491744,
+            -0.01023700088262558,
+            0.01985413022339344,
+            0.008381943218410015,
+            0.03226979821920395,
+            -0.03886960819363594,
+            0.020929789170622826,
+            -0.001477892859838903,
+            0.002021420281380415,
+            0.02601638436317444,
+            -0.020109370350837708,
+            -0.029735615476965904,
+            0.0013912931317463517,
+            -0.028678186237812042,
+            -0.011896069161593914,
+            0.004143113736063242,
+            0.015688225626945496,
+            0.004525975789874792,
+            -0.013719221577048302,
+            0.01617136225104332,
+            -0.008117586374282837,
+            -0.01589788869023323,
+            -0.004799448419362307,
+            -0.0009816535748541355,
+            0.043208710849285126,
+            0.016882391646504402,
+            0.013327243737876415,
+            -0.024466704577207565,
+            -0.03828619793057442,
+            -0.0034753840882331133,
+            -0.00029070733580738306,
+            0.03573378548026085,
+            0.007005462888628244,
+            0.007055599708110094,
+            -0.012233352288603783,
+            -0.011804911307990551,
+            0.0008768223342485726,
+            -0.016262518242001534,
+            0.0013081118231639266,
+            -0.011431165039539337,
+            -0.011494975537061691,
+            -0.004047398455440998,
+            0.004104371648281813,
+            0.010255232453346252,
+            0.013974462635815144,
+            0.012935265898704529,
+            0.006540559232234955,
+            -0.0032953477930277586,
+            -0.012689140625298023,
+            -0.0116225965321064,
+            -0.007488598581403494,
+            0.006271644029766321,
+            0.031066516414284706,
+            0.0032657217234373093,
+            -0.007023694459348917,
+            0.0047265226021409035,
+            0.032999057322740555,
+            -0.011504091322422028,
+            0.001646534539759159,
+            0.017474915832281113,
+            -0.02497718669474125,
+            0.016973547637462616,
+            -0.0008699854952283204,
+            0.015150396153330803,
+            0.006139465607702732,
+            0.015013659372925758,
+            -0.013518675230443478,
+            0.0009047393687069416,
+            0.011886953376233578,
+            0.008076565340161324,
+            -0.04886048287153244,
+            -0.00996352732181549,
+            0.009170456789433956,
+            -0.02173197641968727,
+            -0.01505923829972744,
+            -0.002342750784009695,
+            -0.0019188678124919534,
+            0.013345475308597088,
+            -0.017292600125074387,
+            0.014111199416220188,
+            0.03704645484685898,
+            -0.015086585655808449,
+            -0.02178667113184929,
+            0.008983583189547062,
+            0.02658156119287014,
+            0.013655411079525948,
+            -0.004981763660907745,
+            -0.008842289447784424,
+            0.014721955172717571,
+            0.009325424209237099,
+            -0.02116679958999157,
+            -0.013454864732921124,
+            0.040838614106178284,
+            -0.013044655323028564,
+            -0.016253402456641197,
+            0.023336350917816162,
+            -0.013309012167155743,
+            -0.02317226678133011,
+            0.017520494759082794,
+            0.014284398406744003,
+            -0.0061531392857432365,
+            0.020437538623809814,
+            0.0220783744007349,
+            0.003309021471068263,
+            -0.004156787414103746,
+            0.019945286214351654,
+            -0.006431169807910919,
+            -0.015350942499935627,
+            -0.007319956552237272,
+            -0.011303544975817204,
+            -0.012479477562010288,
+            0.014512293040752411,
+            0.008008196949958801,
+            -0.01928895153105259,
+            -0.005893339868634939,
+            -0.020638084039092064,
+            -0.004149950575083494,
+            0.04627160727977753,
+            -0.028222398832440376,
+            -0.0023336349986493587,
+            0.04051044583320618,
+            -0.003101638052612543,
+            -0.04470369592308998,
+            -0.01864173263311386,
+            0.007762071210891008,
+            -0.01752961054444313,
+            -0.009024604223668575,
+            -0.0422242097556591,
+            -0.005820414051413536,
+            0.020492233335971832,
+            0.015250669792294502,
+            -0.015697341412305832,
+            0.010082032531499863,
+            -0.020455770194530487,
+            0.06388325989246368,
+            0.0004905419191345572,
+            -0.009047393687069416,
+            -0.01092979870736599,
+            -0.02650863490998745,
+            0.011968995444476604,
+            0.02650863490998745,
+            0.014503177255392075,
+            0.012543288059532642,
+            0.022041911259293556,
+            0.02047400176525116,
+            0.012069269083440304,
+            0.025852300226688385,
+            0.02123972587287426,
+            0.030902432277798653,
+            0.024266157299280167,
+            -0.0003660548245534301,
+            0.01807655580341816,
+            -0.008509563282132149,
+            -0.04656331241130829,
+            -0.008573373779654503,
+            -0.003154053585603833,
+            0.00038371660048142076,
+            -0.011704638600349426,
+            -0.008855962194502354,
+            -0.00014143674343358725,
+            -0.020528696477413177,
+            -0.019416572526097298,
+            -0.041130319237709045,
+            -0.006490422412753105,
+            0.0028919754549860954,
+            0.03447581082582474,
+            0.011677291244268417,
+            -0.0007338188006542623,
+            -0.01835002936422825,
+            0.018577922135591507,
+            0.017228789627552032,
+            0.010145843029022217,
+            0.010692788287997246,
+            -0.00012954352132510394,
+            -0.014904270879924297,
+            0.050428394228219986,
+            0.021476734429597855,
+            0.0075159454718232155,
+            -0.027219664305448532,
+            -0.01492250245064497,
+            0.004343660548329353,
+            -0.005255236756056547,
+            -0.008477658964693546,
+            -0.0130537711083889,
+            -0.019270719960331917,
+            0.0033272530417889357,
+            0.014949849806725979,
+            -0.018304450437426567,
+            -0.015961699187755585,
+            0.003577936440706253,
+            0.003413852769881487,
+            0.024794872850179672,
+            -0.006148581393063068,
+            0.015907004475593567,
+            -0.012762066908180714,
+            -0.02025522291660309,
+            0.028422946110367775,
+            0.02061985246837139,
+            0.007857786491513252,
+            0.007392882835119963,
+            -0.006554232910275459,
+            0.006385591346770525,
+            -0.0011622596066445112,
+            0.012707372196018696,
+            -0.03810388594865799,
+            0.029370984062552452,
+            -0.004181855823844671,
+            5.131177022121847e-05,
+            -0.012278931215405464,
+            0.0220783744007349,
+            0.006517769768834114,
+            -0.0004099244251847267,
+            0.007739281747490168,
+            0.0025227870792150497,
+            -0.010419315658509731,
+            -0.01582496240735054,
+            0.002310845535248518,
+            0.036535974591970444,
+            -0.0031654483173042536,
+            0.016973547637462616,
+            -0.0038559671957045794,
+            0.02255239523947239,
+            0.007926154881715775,
+            -0.022807635366916656,
+            -0.023281656205654144,
+            -0.006025518756359816,
+            0.0015929794171825051,
+            0.03197809308767319,
+            -0.016134899109601974,
+            -0.03325429931282997,
+            0.022789403796195984,
+            0.01711028441786766,
+            -0.020145833492279053,
+            0.03578848019242287,
+            0.013454864732921124,
+            -0.0035095682833343744,
+            0.032999057322740555,
+            -0.0033454846125096083,
+            0.0018664522795006633,
+            0.009863254614174366,
+            0.014630797319114208,
+            0.0032360954210162163,
+            -0.012561519630253315,
+            -0.00414767162874341,
+            -0.009047393687069416,
+            -0.01595258340239525,
+            -0.006390149239450693,
+            0.004345939494669437,
+            -0.025943458080291748,
+            -0.0025729236658662558,
+            -0.0008483355632051826,
+            -0.001078508561477065,
+            0.015396521426737309,
+            0.01283499225974083,
+            -0.017119400203227997,
+            0.020729241892695427,
+            -0.007784860674291849,
+            -0.025906994938850403,
+            0.02393799088895321,
+            0.015405637212097645,
+            -0.00026165085728280246,
+            0.005268910434097052,
+            0.036955296993255615,
+            0.02395622245967388,
+            -0.009845023043453693,
+            -2.443629455228802e-06,
+            -0.001749086775816977,
+            0.009407466277480125,
+            -0.005979939829558134,
+            0.0009018906857818365,
+            0.00822241697460413,
+            0.0027005444280803204,
+            0.024375546723604202,
+            -0.008017312735319138,
+            -0.01575203612446785,
+            0.005838645622134209,
+            -0.017292600125074387,
+            0.01137647032737732,
+            -0.0018527786014601588,
+            -0.006567906588315964,
+            -0.01357336901128292,
+            0.008336364291608334,
+            0.014484945684671402,
+            0.047292571514844894,
+            0.014092967845499516,
+            0.0056153093464672565,
+            -0.017247021198272705,
+            0.01283499225974083,
+            0.02783953584730625,
+            0.001371922204270959,
+            0.012069269083440304,
+            -0.02811300940811634,
+            0.021968986839056015,
+            -0.002764354692772031,
+            0.003744299290701747,
+            0.0065998113714158535,
+            -0.0012978565646335483,
+            -0.0001713478413876146,
+            0.03440288454294205,
+            0.020145833492279053,
+            0.021950755268335342,
+            0.08576108515262604,
+            0.009644475765526295,
+            -0.016927970573306084,
+            -0.010774830356240273,
+            -0.0032429322600364685,
+            -0.028678186237812042,
+            -0.013454864732921124,
+            0.044521380215883255,
+            -0.004936185199767351,
+            0.004553323145955801,
+            -0.00706471549347043,
+            -0.0006426612380892038,
+            -0.013655411079525948,
+            0.009462160989642143,
+            0.010391968302428722,
+            0.0036508627235889435,
+            -0.0020054676569998264,
+            -0.0018641733331605792,
+            -0.023208729922771454,
+            -0.003976751118898392,
+            -0.025177733972668648,
+            -0.0072287991642951965,
+            -0.0035938890650868416,
+            0.011431165039539337,
+            0.0018687312258407474,
+            0.023482201620936394,
+            -1.8819062461261638e-05,
+            -0.007885133847594261,
+            0.00307201175019145,
+            0.017985397949814796,
+            -0.020382843911647797,
+            0.010018222033977509,
+            0.019179562106728554,
+            -0.0031358220148831606,
+            -0.0007053320878185332,
+            0.0021444829180836678,
+            -0.008500447496771812,
+            -0.009626244194805622,
+            -0.012306278571486473,
+            0.02415676973760128,
+            -0.006517769768834114,
+            -0.01810390315949917,
+            -0.010884219780564308,
+            0.00918413046747446,
+            0.000970258901361376,
+            0.01864173263311386,
+            0.00678668450564146,
+            -0.005460341461002827,
+            0.014886039309203625,
+            -0.01183225866407156,
+            -0.006927979178726673,
+            -0.00954420305788517,
+            0.010774830356240273,
+            -0.009726517833769321,
+            0.02242477424442768,
+            -0.013473096303641796,
+            0.009115762077271938,
+            -0.01881493255496025,
+            0.0065587908029556274,
+            0.00035124170244671404,
+            -0.024448473006486893,
+            0.011613480746746063,
+            -0.0036690940614789724,
+            -0.004936185199767351,
+            -0.008322690613567829,
+            0.005806740373373032,
+            0.015578837133944035,
+            -0.016982663422822952,
+            0.00034810815122909844,
+            -0.014138546772301197,
+            -0.02776661142706871,
+            -0.006134907715022564,
+            0.006955326069146395,
+            -0.008017312735319138,
+            -0.02464902028441429,
+            0.02054692804813385,
+            0.014594335108995438,
+            -0.02499541826546192,
+            -0.0005406786222010851,
+            -0.0014676376013085246,
+            -0.01723790541291237,
+            -0.025560596957802773,
+            0.0240109171718359,
+            -0.003445758018642664,
+            -0.007525061257183552,
+            0.020729241892695427,
+            -0.0003711824247147888,
+            -0.03551500663161278,
+            0.02929805777966976,
+            -0.004589786287397146,
+            0.00822241697460413,
+            0.022114837542176247,
+            0.003903825068846345,
+            -0.0007594568887725472,
+            0.006800358183681965,
+            0.02206014283001423,
+            0.008436637930572033,
+            -0.02756606414914131,
+            0.005164078902453184,
+            -0.014557871967554092,
+            -0.0027552389074116945,
+            -0.0035391945857554674,
+            -0.015907004475593567,
+            -0.03402002155780792,
+            0.0066590639762580395,
+            0.011531438678503036,
+            0.011358239687979221,
+            0.004120324272662401,
+            0.0054557835683226585,
+            0.018687311559915543,
+            0.0003073721018154174,
+            0.02082039974629879,
+            -0.0021365066058933735,
+            0.016581570729613304,
+            0.047365497797727585,
+            0.00021920558356214315,
+            -0.03394709527492523,
+            0.0007190057076513767,
+            0.00652688555419445,
+            -0.005200542043894529,
+            0.0028691859915852547,
+            0.03680944815278053,
+            0.01708293706178665,
+            -0.021968986839056015,
+            0.014010925777256489,
+            0.0013833168195560575,
+            -0.013208738528192043,
+            -0.019525961950421333,
+            0.0011759332846850157,
+            -0.02670918218791485,
+            0.007584313862025738,
+            -0.0025888762902468443,
+            -0.009229708462953568,
+            -0.020948020741343498,
+            -0.004721964709460735,
+            0.043062858283519745,
+            0.019307183101773262,
+            0.009001814760267735,
+            -0.001074520405381918,
+            -0.011048302985727787,
+            -0.03817681223154068,
+            0.010355505160987377,
+            0.029535068199038506,
+            -0.0015553769189864397,
+            0.006080213002860546,
+            -0.017000894993543625,
+            0.009589781053364277,
+            -0.0077347238548099995,
+            0.006335454527288675,
+            0.011467628180980682,
+            0.013409285806119442,
+            -0.0021102989558130503,
+            0.011823142878711224,
+            0.009762980975210667,
+            0.01006380096077919,
+            -0.0406927615404129,
+            0.007338188122957945,
+            0.0057429298758506775,
+            0.008568815886974335,
+            -0.00844575371593237,
+            0.016061972826719284,
+            -0.01970827765762806,
+            0.016326328739523888,
+            -0.03290789946913719,
+            -0.003418410662561655,
+            0.015733804553747177,
+            -0.01829533465206623,
+            -0.02443024143576622,
+            0.007702818606048822,
+            -0.0030287117697298527,
+            0.005984497722238302,
+            -0.006476748734712601,
+            0.00960801262408495,
+            -0.04557880759239197,
+            0.011075650341808796,
+            -0.02317226678133011,
+            -0.005578846205025911,
+            0.03617134317755699,
+            -0.004854143131524324,
+            -0.020291686058044434,
+            -0.00022817892022430897,
+            0.019489498808979988,
+            -0.020729241892695427,
+            -0.04350041598081589,
+            0.012725603766739368,
+            -0.014065620489418507,
+            0.013272549025714397,
+            -0.015843193978071213,
+            -0.007885133847594261,
+            0.015606184490025043,
+            0.013418401591479778,
+            0.009434813633561134,
+            -0.0028144915122538805,
+            -0.000781106820795685,
+            0.016034625470638275,
+            -0.03374655172228813,
+            0.006166812963783741,
+            -0.018723774701356888,
+            -0.03941655531525612,
+            -0.0012499988079071045,
+            0.023755675181746483,
+            -0.0042205979116261005,
+            0.02408384345471859,
+            0.01467637624591589,
+            -0.030063781887292862,
+            -0.02109387330710888,
+            -0.014484945684671402,
+            0.039671797305345535,
+            0.007971733808517456,
+            -0.023281656205654144,
+            -0.0027871441561728716,
+            -0.0007218543905764818,
+            0.014092967845499516,
+            -0.017985397949814796,
+            -0.0030879643745720387,
+            0.004181855823844671,
+            -0.004703733138740063,
+            -0.005975381936877966,
+            -0.03927070274949074,
+            0.012570635415613651,
+            0.019179562106728554,
+            -0.012935265898704529,
+            0.006271644029766321,
+            0.012844108045101166,
+            -0.01859615370631218,
+            0.012816760689020157,
+            -0.004300360567867756,
+            0.02796715684235096,
+            0.01680946536362171,
+            0.014594335108995438,
+            -0.009498624131083488,
+            -0.010993609204888344,
+            0.0011394702596589923,
+            -0.0007719910936430097,
+            -0.011285313405096531,
+            0.004767543636262417,
+            -0.003935730084776878,
+            0.020018212497234344,
+            0.017839545384049416,
+            0.029954394325613976,
+            -0.019179562106728554,
+            0.012798530049622059,
+            -0.03664536401629448,
+            0.00568823516368866,
+            0.0010044430382549763,
+            -0.00441430788487196,
+            -0.007739281747490168,
+            0.020236991345882416,
+            -0.01965358294546604,
+            0.0046672699972987175,
+            -0.011577017605304718,
+            -0.026599792763590813,
+            0.0009919088333845139,
+            0.007438461761921644,
+            -0.015961699187755585,
+            0.0018926600459963083,
+            -0.013546021655201912,
+            -0.01985413022339344,
+            -0.007497713901102543,
+            0.026563329622149467,
+            -0.008413848467171192,
+            -0.0005198832950554788,
+            -0.0005409634904935956,
+            -0.01391065213829279,
+            0.007333630230277777,
+            -0.003530078800395131,
+            -0.006567906588315964,
+            0.0017080658581107855,
+            -0.02428438887000084,
+            -0.007443019654601812,
+            0.005041016265749931,
+            0.02339104562997818,
+            0.015551489777863026,
+            -0.018441185355186462,
+            -0.024029148742556572,
+            -0.002853233367204666,
+            0.026526866480708122,
+            0.014247936196625233,
+            -0.026891497895121574,
+            0.01846853271126747,
+            -0.003641746938228607,
+            0.020638084039092064,
+            0.006216949783265591,
+            0.006618042942136526,
+            -0.0032657217234373093,
+            -0.008199627511203289,
+            0.007119410205632448,
+            -0.009525971487164497,
+            -0.02138557657599449,
+            -0.009735633619129658,
+            0.015496795065701008,
+            -0.010647210292518139,
+            0.003821783233433962,
+            0.000417046103393659,
+            -0.015916120260953903,
+            0.006289875600486994,
+            -0.009790328331291676,
+            0.015432984568178654,
+            0.011996342800557613,
+            0.00036747913691215217,
+            0.00036235153675079346,
+            0.030483108013868332,
+            -0.012297162786126137,
+            -0.007917039096355438,
+            0.00522333150729537,
+            -0.009047393687069416,
+            0.004010935313999653,
+            -0.007834997028112411,
+            -0.03241565078496933,
+            -0.0031859588343650103,
+            0.014311745762825012,
+            0.018605269491672516,
+            -0.01044666301459074,
+            -0.007958060130476952,
+            0.028058314695954323,
+            0.006285317707806826,
+            -0.007306283339858055,
+            -0.005018226802349091,
+            0.04718318209052086,
+            -0.0003959659079555422,
+            0.020984483882784843,
+            0.009535087272524834,
+            0.003142658853903413,
+            -0.004181855823844671,
+            -0.0003748857125174254,
+            0.008846846409142017,
+            -0.003760251682251692,
+            0.0021718302741646767,
+            -0.012415667995810509,
+            0.013254317454993725,
+            -0.0037078361492604017,
+            -0.028404714539647102,
+            0.008327248506247997,
+            -0.014603450894355774,
+            0.0001921431685332209,
+            -0.018377376720309258,
+            0.0027005444280803204,
+            0.012798530049622059,
+            -0.012561519630253315,
+            -0.0017525051953271031,
+            -0.007101178634911776,
+            -0.00947127677500248,
+            -0.0021923407912254333,
+            -0.0010414757998660207,
+            -0.02027345448732376,
+            -0.010893335565924644,
+            0.0008016172796487808,
+            -0.012762066908180714,
+            0.011886953376233578,
+            0.0002697695745155215,
+            -0.0006933676195330918,
+            0.00975386518985033,
+            -0.0009805141016840935,
+            0.019544193521142006,
+            0.00699178921058774,
+            0.0047265226021409035,
+            -0.006089328788220882,
+            0.004712848924100399,
+            0.020091138780117035,
+            -0.035770248621702194,
+            0.004095255862921476,
+            0.0005959429545328021,
+            0.009120319969952106,
+            -0.006604369264096022,
+            -0.016216941177845,
+            -0.002313124481588602,
+            -0.019598888233304024,
+            0.009899717755615711,
+            -0.01595258340239525,
+            -0.012680024839937687,
+            -0.000525010924320668,
+            0.0003065174969378859,
+            -0.004571554716676474,
+            0.006157697178423405,
+            0.011850490234792233,
+            0.004327707923948765,
+            0.0032133059576153755,
+            0.013937999494373798,
+            -0.020018212497234344,
+            0.004459886346012354,
+            -0.007944386452436447,
+            -0.0019747018814086914,
+            0.015195975080132484,
+            -0.0020134439691901207,
+            0.0006033494719304144,
+            -0.014895155094563961,
+            -0.002074975287541747,
+            0.005765719339251518,
+            -0.0012830435298383236,
+            -0.0036987203639000654,
+            0.013381938450038433,
+            -0.004958974663168192,
+            0.00013353166286833584,
+            -0.008587047457695007,
+            -0.00039767511771060526,
+            -0.004949858877807856,
+            0.006467632949352264,
+            -0.016982663422822952,
+            0.009735633619129658,
+            0.01721055805683136,
+            0.0061440235003829,
+            -0.002125111874192953,
+            -0.003600725904107094,
+            -0.00301959621720016,
+            0.03309021517634392,
+            0.006704642903059721,
+            0.02060162089765072,
+            0.029334520921111107,
+            0.008896983228623867,
+            -0.0037853200919926167,
+            0.0013092512963339686,
+            -0.008331806398928165,
+            -0.007456693332642317,
+            -0.0230446457862854,
+            -0.013299896381795406,
+            0.01464902888983488,
+            -0.006084770895540714,
+            0.012470361776649952,
+            -5.740081178373657e-05,
+            0.007383767049759626,
+            -0.00607109721750021,
+            -0.01896078512072563,
+            -0.030902432277798653,
+            -0.002962622558698058,
+            0.022114837542176247,
+            -0.005405646748840809,
+            -0.026964424178004265,
+            0.0004817110311705619,
+            0.010474010370671749,
+            -0.018094787374138832,
+            -0.015669994056224823,
+            0.010273464024066925,
+            -0.007429345976561308,
+            0.01886962726712227,
+            -0.008641742169857025,
+            0.017739271745085716,
+            -0.0009041696321219206,
+            -0.008641742169857025,
+            0.022114837542176247,
+            -0.024339083582162857,
+            0.013299896381795406,
+            -0.0004179007082711905,
+            -0.0031677272636443377,
+            -0.0003310161118861288,
+            0.010528705082833767,
+            -0.010765714570879936,
+            -0.0184503011405468,
+            0.005478573031723499,
+            -0.01346398051828146,
+            0.009352771565318108,
+            -0.008896983228623867,
+            0.011148576624691486,
+            0.02470371499657631,
+            -0.014475829899311066,
+            0.004977205768227577,
+            0.007206009700894356,
+            0.0012534172274172306,
+            0.0014949849573895335,
+            -0.005793066695332527,
+            0.013199622742831707,
+            0.0069781155325472355,
+            -0.010647210292518139,
+            0.02692796103656292,
+            0.010993609204888344,
+            -0.012752951122820377,
+            -0.0035847732797265053,
+            -0.007301725447177887,
+            -0.018796700984239578,
+            -0.000998745672404766,
+            -0.018997248262166977,
+            -0.011048302985727787,
+            -0.008587047457695007,
+            -0.011112113483250141,
+            0.0019507730612531304,
+            -0.005528709385544062,
+            -0.01259798277169466,
+            -0.0038377356249839067,
+            0.011804911307990551,
+            -0.006704642903059721,
+            -0.03318137302994728,
+            0.01812213473021984,
+            0.019762972369790077,
+            0.008140374906361103,
+            -0.03792157024145126,
+            0.019872359931468964,
+            0.00757975596934557,
+            -0.002256151055917144,
+            -0.012853223830461502,
+            0.0012841830030083656,
+            -0.01896078512072563,
+            0.013263433240354061,
+            -0.019872359931468964,
+            -0.001451685093343258,
+            0.0016818580916151404,
+            0.007958060130476952,
+            0.005100268870592117,
+            0.022880561649799347,
+            -0.005195984151214361,
+            -0.004268455319106579,
+            0.005478573031723499,
+            0.0038103885017335415,
+            -0.0025524133816361427,
+            0.030829505994915962,
+            -0.01051958929747343,
+            -0.023154035210609436,
+            -0.002796259941533208,
+            0.014521408826112747,
+            -0.009735633619129658,
+            -0.015387405641376972,
+            0.020437538623809814,
+            -0.01797628216445446,
+            0.0022401984315365553,
+            0.014731070958077908,
+            0.004284407943487167,
+            0.030720116570591927,
+            0.01290791854262352,
+            -0.005346394143998623,
+            -0.000578565988689661,
+            0.00010262354044243693,
+            0.0010055825114250183,
+            0.011613480746746063,
+            -0.004895164165645838,
+            -0.004762985743582249,
+            -0.005815856158733368,
+            0.012725603766739368,
+            -9.649888670537621e-05,
+            -0.0031996325124055147,
+            -0.00015781662659719586,
+            -0.015414752997457981,
+            0.014876923523843288,
+            0.0026731970719993114,
+            0.0036280732601881027,
+            0.007420230191200972,
+            -6.366789602907375e-05,
+            -0.0059617082588374615,
+            -0.01384684257209301,
+            0.006567906588315964,
+            -0.011340008117258549,
+            -0.0115223228931427,
+            -0.028696417808532715,
+            0.014949849806725979,
+            -0.02040107548236847,
+            0.00958066526800394,
+            -0.002073835814371705,
+            0.009863254614174366,
+            -0.02054692804813385,
+            -0.015788499265909195,
+            -0.009325424209237099,
+            0.012607098557054996,
+            -0.009662707336246967,
+            0.02144027128815651,
+            0.007174104452133179,
+            0.015770267695188522,
+            0.021257957443594933,
+            0.024247925728559494,
+            0.004867816809564829,
+            0.01629898138344288,
+            -0.02818593569099903,
+            -0.0007919318159110844,
+            -0.007748397532850504,
+            0.012014574371278286,
+            -0.0036166785284876823,
+            0.006394707132130861,
+            0.013108465820550919,
+            0.01723790541291237,
+            -0.010938914492726326,
+            0.002903370186686516,
+            -2.533540828153491e-05,
+            -0.00958066526800394,
+            -0.00711485231295228,
+            -0.017784850671887398,
+            -0.01859615370631218,
+            0.01631721295416355,
+            0.007511387579143047,
+            0.012780298478901386,
+            0.016052857041358948,
+            0.016289865598082542,
+            -0.0023199613206088543,
+            0.0021091594826430082,
+            -0.01943480409681797,
+            -0.0025934341829270124,
+            -0.012917034327983856,
+            -0.0013593879994004965,
+            0.025560596957802773,
+            -0.0017069263849407434,
+            0.017939819023013115,
+            0.007433903869241476,
+            0.034731052815914154,
+            0.0010015943553298712,
+            0.006927979178726673,
+            0.005278026219457388,
+            0.015633530914783478,
+            0.0027119391597807407,
+            0.02386506460607052,
+            -0.02040107548236847,
+            0.0013001355109736323,
+            0.03691883385181427,
+            0.022862330079078674,
+            -0.008195069618523121,
+            -0.01790335588157177,
+            -0.01943480409681797,
+            0.016681844368577003,
+            -0.03019140288233757,
+            -0.00757975596934557,
+            0.02386506460607052,
+            0.00624885456636548,
+            0.01319050695747137,
+            -0.0006648809066973627,
+            -0.018650848418474197,
+            0.012889686971902847,
+            0.008195069618523121,
+            -0.009498624131083488,
+            0.004423423204571009,
+            -0.006381033454090357,
+            -0.009489508345723152,
+            -0.0027825862634927034,
+            -0.0027871441561728716,
+            -0.017474915832281113,
+            -0.00865541584789753,
+            0.0004555032355710864,
+            -0.004541928414255381,
+            0.0021102989558130503,
+            -0.003470826428383589,
+            -0.0217137448489666,
+            0.0058340877294540405,
+            -0.012087500654160976,
+            -0.0025683660060167313,
+            0.0018128971569240093,
+            -0.009498624131083488,
+            -0.024667251855134964,
+            -0.00650409609079361,
+            -0.007142199203372002,
+            -0.005268910434097052,
+            0.01277118269354105,
+            -0.015460331924259663,
+            0.009257055819034576,
+            -0.014876923523843288,
+            0.023354582488536835,
+            -0.0005976521642878652,
+            0.008609836921095848,
+            0.013965346850454807,
+            -0.013454864732921124,
+            -0.011494975537061691,
+            -0.03442111611366272,
+            -0.0044781179167330265,
+            -0.020455770194530487,
+            -0.00565633038058877,
+            -0.00660892715677619,
+            0.01689150743186474,
+            0.0018197339959442616,
+            -0.017301715910434723,
+            0.01696443371474743,
+            0.004854143131524324,
+            -0.0013138091890141368,
+            -0.01561530027538538,
+            -0.0044484916143119335,
+            -9.236830373993143e-05,
+            -0.007452135439962149,
+            0.01956242509186268,
+            0.00161348981782794,
+            -0.002983133075758815,
+            0.0025136712938547134,
+            0.007803092245012522,
+            -0.007092062849551439,
+            -0.03389240428805351,
+            -0.012579751200973988,
+            -0.002392887370660901,
+            6.17094337940216e-05,
+            -0.007493156008422375,
+            0.003685046685859561,
+            0.014339093118906021,
+            -0.011185039766132832,
+            -0.007921596989035606,
+            -0.00013324679457582533,
+            0.009808559902012348,
+            0.015451216138899326,
+            -0.009334539994597435,
+            0.018094787374138832,
+            -0.01567910984158516,
+            0.01874200627207756,
+            0.0007093202439136803,
+            -0.002285777358338237,
+            -0.0022162694949656725,
+            0.010282579809427261,
+            -0.008473101072013378,
+            -0.006918863393366337,
+            -0.009211476892232895,
+            -0.008814942091703415,
+            -0.0073290723375976086,
+            0.0013046934036538005,
+            -0.020291686058044434,
+            -0.020528696477413177,
+            -0.006230622995644808,
+            0.0030013646464794874,
+            -0.006125791929662228,
+            0.011567901819944382,
+            -0.01211484707891941,
+            -0.014348208904266357,
+            0.014147662557661533,
+            0.007392882835119963,
+            0.0029216017574071884,
+            -0.017092052847146988,
+            -0.007274378091096878,
+            -0.007661798037588596,
+            0.008299901150166988,
+            -0.0031700062099844217,
+            0.034803979098796844,
+            -0.003343205666169524,
+            -0.010455778799951077,
+            0.003015038324519992,
+            -0.001371922204270959,
+            0.0058112982660532,
+            -0.016499528661370277,
+            0.005701908841729164,
+            0.0011172505328431726,
+            0.006814031861722469,
+            0.020528696477413177,
+            -0.0016294424422085285,
+            -0.012871455401182175,
+            0.008796710520982742,
+            0.010191421955823898,
+            0.002445303136482835,
+            -0.0016009557293727994,
+            -0.0016898344038054347,
+            -0.00801275484263897,
+            0.019945286214351654,
+            0.041604336351156235,
+            0.0035164051223546267,
+            0.01037373673170805,
+            0.009389234706759453,
+            -0.01641748659312725,
+            -0.009489508345723152,
+            0.014685492031276226,
+            -0.008436637930572033,
+            0.005683677736669779,
+            -0.006253412459045649,
+            0.019981749355793,
+            -0.011458512395620346,
+            -0.005232447292655706,
+            0.009981758892536163,
+            -0.003575657494366169,
+            0.014904270879924297,
+            0.006107560358941555,
+            -0.019817667081952095,
+            0.008600721135735512,
+            0.0014482666738331318,
+            -0.0035072893369942904,
+            -0.026818571612238884,
+            0.0024703715462237597,
+            0.002686870750039816,
+            0.009945295751094818,
+            -0.0232451930642128,
+            0.01252505648881197,
+            0.010145843029022217,
+            0.004908837843686342,
+            0.004651317372918129,
+            0.0037511358968913555,
+            -0.009329982101917267,
+            0.005369183607399464,
+            0.01923425681889057,
+            -0.008154048584401608,
+            -0.025852300226688385,
+            -0.0004278710694052279,
+            0.001534866401925683,
+            -0.010811293497681618,
+            -0.0058112982660532,
+            0.010528705082833767,
+            -0.009480392560362816,
+            -0.005337278358638287,
+            0.015569721348583698,
+            0.014138546772301197,
+            -0.012570635415613651,
+            0.02816770412027836,
+            -0.006203276105225086,
+            0.0061804866418242455,
+            0.025724679231643677,
+            -0.007365535479038954,
+            -0.012178657576441765,
+            -0.025141270831227303,
+            0.013245201669633389,
+            0.0010956006590276957,
+            0.019398340955376625,
+            -0.02747490629553795,
+            -0.021640818566083908,
+            -0.018359145149588585,
+            0.008359153755009174,
+            0.008208743296563625,
+            0.0027301707305014133,
+            0.02277117222547531,
+            -0.00683682132512331,
+            -0.0010357784340158105,
+            -0.0013331801164895296,
+            0.019270719960331917,
+            0.00732451444491744,
+            -0.0015758873196318746,
+            0.011686407029628754,
+            0.012789414264261723,
+            0.01567910984158516,
+            -0.011558786034584045,
+            -0.014594335108995438,
+            -0.014968081377446651,
+            0.00016992349992506206,
+            -0.01044666301459074,
+            -0.004863258916884661,
+            0.006513211876153946,
+            0.026271626353263855,
+            -0.021203262731432915,
+            0.010018222033977509,
+            0.006376475561410189,
+            0.002236780012026429,
+            -0.0016647659940645099,
+            -0.004175018984824419,
+            0.018113018944859505,
+            -0.017721040174365044,
+            0.00532816257327795,
+            -0.0010152680333703756,
+            0.0027073812671005726,
+            0.012470361776649952,
+            -0.023135803639888763,
+            -0.005565172526985407,
+            0.015341827645897865,
+            -0.004122603219002485,
+            -0.00947127677500248,
+            0.026052847504615784,
+            -0.03700999170541763,
+            0.021276189014315605,
+            0.015423868782818317,
+            -0.0024065610487014055,
+            -0.011896069161593914,
+            -0.01881493255496025,
+            0.0034024580381810665,
+            -0.027019118890166283,
+            -0.0018687312258407474,
+            0.005405646748840809,
+            -0.01267090905457735,
+            -0.010173190385103226,
+            0.009589781053364277,
+            0.011786679737269878,
+            -0.005515035707503557,
+            -0.009936179965734482,
+            -0.008003639057278633,
+            -0.009507739916443825,
+            -0.011996342800557613,
+            0.0057429298758506775,
+            -0.00898814108222723,
+            -0.003976751118898392,
+            0.0007685726741328835,
+            0.02678210847079754,
+            -4.931769581162371e-06,
+            -0.012661793269217014,
+            0.013892421498894691,
+            0.02541474439203739,
+            -0.009726517833769321,
+            -0.02235184796154499,
+            -0.008983583189547062,
+            -0.0009565852815285325,
+            -0.0037055572029203176,
+            -0.008263438008725643,
+            0.0009070183150470257,
+            -0.011804911307990551,
+            0.0040701874531805515,
+            0.003933451138436794,
+            -0.0009845022577792406,
+            -0.022151300683617592,
+            0.008331806398928165,
+            0.007999081164598465,
+            0.011440280824899673,
+            -0.0414949469268322,
+            -0.002910207025706768,
+            -0.012880571186542511,
+            -0.0042205979116261005,
+            -0.01429351419210434,
+            0.025232428684830666,
+            0.005847761407494545,
+            0.0065770223736763,
+            -0.02470371499657631,
+            0.008103912696242332,
+            0.022242458537220955,
+            -0.0014699165476486087,
+            0.0018698706990107894,
+            -0.01159524917602539,
+            -0.00257520261220634,
+            0.028751112520694733,
+            -0.007803092245012522,
+            0.00506836362183094,
+            -0.003101638052612543,
+            0.00027831559418700635,
+            0.004441654775291681,
+            0.013372822664678097,
+            0.0007588871521875262,
+            -0.00982679147273302,
+            0.014494061470031738,
+            0.00021607204689644277,
+            0.005756603553891182,
+            -0.0031107538379728794,
+            -0.01329078059643507,
+            0.0025432975962758064,
+            -0.009899717755615711,
+            -0.03144937753677368,
+            -0.01579761505126953,
+            -0.020510464906692505,
+            -0.013427517376840115,
+            -0.002880580723285675,
+            0.0334366150200367,
+            0.014776649884879589,
+            -0.014010925777256489,
+            -0.0012955776182934642,
+            -0.0034047369845211506,
+            0.027784842997789383,
+            0.011485859751701355,
+            -0.008030986413359642,
+            0.02020052820444107,
+            -0.0017593420343473554,
+            -0.00285551231354475,
+            0.018277103081345558,
+            -0.004343660548329353,
+            0.010802177712321281,
+            0.0024726504925638437,
+            -0.02421146258711815,
+            0.012743835337460041,
+            -0.002525066025555134,
+            0.001482450752519071,
+            0.010957146063446999,
+            -0.00960801262408495,
+            -0.01290791854262352,
+            0.0015417032409459352,
+            0.008523236960172653,
+            -0.012196889147162437,
+            -0.024375546723604202,
+            -0.0033477635588496923,
+            -0.008550584316253662,
+            -0.007948944345116615,
+            0.0016191871836781502,
+            0.014703723601996899,
+            -0.0004247375181876123,
+            0.006522327661514282,
+            -0.004936185199767351,
+            -0.014165894128382206,
+            -0.004694617353379726,
+            0.02144027128815651,
+            0.01866907998919487,
+            0.00954420305788517,
+            -4.436812378116883e-05,
+            0.016882391646504402,
+            -0.010255232453346252,
+            -0.005875108297914267,
+            -0.016927970573306084,
+            -0.02484956756234169,
+            0.0009457602864131331,
+            0.0106563251465559,
+            -0.024120306596159935,
+            0.0034844998735934496,
+            -0.048678167164325714,
+            -0.008700994774699211,
+            0.017174094915390015,
+            -0.01908840611577034,
+            0.005059247836470604,
+            0.00242023472674191,
+            -0.03041018173098564,
+            0.015268901363015175,
+            -0.019762972369790077,
+            0.01061986293643713,
+            -0.009311750531196594,
+            -0.02588876336812973,
+            0.016818581148982048,
+            -0.006677295546978712,
+            0.016335444524884224,
+            -0.009079298935830593,
+            -0.008304459042847157,
+            -0.0056426567025482655,
+            -0.006221507675945759,
+            0.00036690940032713115,
+            0.0006888097268529236,
+            0.001456242986023426,
+            0.0036736519541591406,
+            0.004899722058326006,
+            -0.01750226318836212,
+            -0.009945295751094818,
+            -0.01551502663642168,
+            -0.014895155094563961,
+            -0.01810390315949917,
+            0.009083856828510761,
+            -0.01520509086549282,
+            0.010027337819337845,
+            -0.0034252475015819073,
+            -0.017438452690839767,
+            0.007092062849551439,
+            -0.006809473969042301,
+            0.005232447292655706,
+            -0.005970824044197798,
+            -0.006595253478735685,
+            0.009899717755615711,
+            0.006905189715325832,
+            0.007871460169553757,
+            0.00706471549347043,
+            -0.012844108045101166,
+            -0.004193250555545092,
+            -0.003876477712765336,
+            0.03318137302994728,
+            0.0005891061155125499,
+            -0.01023700088262558,
+            -0.005788508802652359,
+            -1.0931792530755047e-05,
+            7.641572301508859e-05,
+            0.005086595192551613,
+            0.0007127386052161455,
+            0.016499528661370277,
+            -0.019197793677449226,
+            -0.0051367320120334625,
+            -0.0060209608636796474,
+            -0.005697350949048996,
+            -0.012534172274172306,
+            -0.016918854787945747,
+            0.002527344971895218,
+            0.01916133053600788,
+            0.009261613711714745,
+            0.01259798277169466,
+            -0.03110297955572605,
+            0.0029671804513782263,
+            0.021494966000318527,
+            -0.025706447660923004,
+            -0.010474010370671749,
+            -0.00793527066707611,
+            -0.028277093544602394,
+            -0.015551489777863026,
+            0.013855958357453346,
+            0.014311745762825012,
+            0.023208729922771454,
+            -0.017319947481155396,
+            -0.0007452135323546827,
+            -0.03274381533265114,
+            -0.0012978565646335483,
+            -0.0030811275355517864,
+            -0.008381943218410015,
+            -0.024375546723604202,
+            0.025834068655967712,
+            0.015979930758476257,
+            0.008559700101613998,
+            0.004355055280029774,
+            -0.000295692530926317,
+            0.024831335991621017,
+            -0.004131719004362822,
+            0.007629892788827419,
+            -0.001804920844733715,
+            0.017347294837236404,
+            -0.009553318843245506,
+            -0.012889686971902847,
+            0.005109384655952454,
+            -0.01068367250263691,
+            -7.24987912690267e-05,
+            -0.00688695814460516,
+            -0.027456674724817276,
+            -0.029443910345435143,
+            0.0053965309634804726,
+            0.0013400169555097818,
+            0.009872370399534702,
+            -0.022935256361961365,
+            0.018240639939904213,
+            -0.0066590639762580395,
+            -0.004744754172861576,
+            -0.0006774150533601642,
+            -0.0023906086571514606,
+            0.02720143273472786,
+            -0.02220599539577961,
+            -0.0026230604853481054,
+            -0.0015439821872860193,
+            -0.010027337819337845,
+            0.006595253478735685,
+            0.0008027567528188229,
+            0.013372822664678097,
+            0.016818581148982048,
+            -0.015004543587565422,
+            -0.04123970493674278,
+            -0.002021420281380415,
+            -0.035095684230327606,
+            -0.0077347238548099995,
+            -0.002716497052460909,
+            0.021768439561128616,
+            -0.006435727700591087,
+            0.037082917988300323,
+            -0.0014414298348128796,
+            -0.02464902028441429,
+            -0.0270738136023283,
+            0.01599816232919693,
+            0.023135803639888763,
+            -0.014457598328590393,
+            -0.0026549657341092825,
+            0.019525961950421333,
+            -0.004120324272662401,
+            0.01121238712221384,
+            0.012160426005721092,
+            0.015323596075177193,
+            -0.004138555843383074,
+            0.0018687312258407474,
+            0.008267995901405811,
+            -0.019690046086907387,
+            -0.028751112520694733,
+            -0.0036189574748277664,
+            0.00030025039450265467,
+            -0.015323596075177193,
+            0.0074475775472819805,
+            -0.01106653455644846,
+            -0.007060157600790262,
+            0.006016402970999479,
+            0.007876018062233925,
+            -0.007944386452436447,
+            0.025925226509571075,
+            -0.014193241484463215,
+            -0.002069277921691537,
+            0.023901527747511864,
+            0.0006523467018269002,
+            -0.016873275861144066,
+            -0.0047265226021409035,
+            0.0032452112063765526,
+            0.027948925271630287,
+            0.01679123379290104,
+            -0.007201451808214188,
+            -0.017848661169409752,
+            0.02443024143576622,
+            -0.011230618692934513,
+            -0.007629892788827419,
+            0.0001251992944162339,
+            0.01200545858591795,
+            0.01842295378446579,
+            -0.004234271589666605,
+            0.004813122097402811,
+            -0.0009480392327532172,
+            0.033491309732198715,
+            0.002347308676689863,
+            0.009453045204281807,
+            -0.01644483394920826,
+            0.009307192638516426,
+            -0.0023974452633410692,
+            -0.00021179902250878513,
+            -0.0021410644985735416,
+            -0.034731052815914154,
+            -0.01530536450445652,
+            -0.0042205979116261005,
+            0.003985866904258728,
+            -0.021914292126893997,
+            0.0041294400580227375,
+            0.001722879009321332,
+            -0.02665448747575283,
+            0.016216941177845,
+            -0.016918854787945747,
+            -0.00944392941892147,
+            0.017821313813328743,
+            -0.0070692733861505985,
+            -0.0073883249424397945,
+            -0.012206004932522774,
+            0.022716479375958443,
+            0.00975386518985033,
+            -0.008846846409142017,
+            0.009890601970255375,
+            -0.01482222881168127,
+            0.008158606477081776,
+            -0.0015451216604560614,
+            0.007834997028112411,
+            0.021185031160712242,
+            -0.017638999968767166,
+            0.017438452690839767,
+            0.00011950194311793894,
+            -0.009744749404489994,
+            -0.008523236960172653,
+            0.00436417106539011,
+            0.027037350460886955,
+            0.00035893311724066734,
+            0.017693692818284035,
+            0.007429345976561308,
+            0.03301728889346123,
+            -0.0034867788199335337,
+            0.004471281077712774,
+            0.012133078649640083,
+            0.016089320182800293,
+            -0.017174094915390015,
+            -0.013026423752307892,
+            0.0020795331802219152,
+            -0.009790328331291676,
+            0.0013650853652507067,
+            -0.009762980975210667,
+            -0.02165905013680458,
+            0.02623516321182251,
+            0.021531429141759872,
+            -0.017019126564264297,
+            0.0008016172796487808,
+            -0.008810384199023247,
+            -0.0351686105132103,
+            -0.018404724076390266,
+            0.0108568724244833,
+            0.015241554006934166,
+            0.0010813572444021702,
+            0.010309926234185696,
+            -0.01016407459974289,
+            -0.00021920558356214315,
+            -0.0036189574748277664,
+            -0.01723790541291237,
+            -0.01793070323765278,
+            -0.015405637212097645,
+            -0.009671823121607304,
+            -0.009029162116348743,
+            0.0039699142798781395,
+            -0.011303544975817204,
+            -0.006536001339554787,
+            0.004514581058174372,
+            0.004731080494821072,
+            -0.009735633619129658,
+            -0.011850490234792233,
+            0.013126697391271591,
+            -0.001399269443936646,
+            0.00046974659198895097,
+            0.016581570729613304,
+            -0.0035961680114269257,
+            -0.0213126502931118,
+            0.00387875665910542,
+            0.006025518756359816,
+            0.012515940703451633,
+            0.00429124478250742,
+            -0.010483126156032085,
+            0.01599816232919693,
+            0.00568823516368866,
+            -0.012333625927567482,
+            0.017146747559309006,
+            -0.022023679688572884,
+            -0.007538734935224056,
+            0.03296259418129921,
+            -0.013637179508805275,
+            0.009316308423876762,
+            0.0003042385505978018,
+            -0.006431169807910919,
+            0.02206014283001423,
+            0.026545098051428795,
+            0.01627163402736187,
+            0.00515040522441268,
+            0.012798530049622059,
+            0.00031363917514681816,
+            -0.0011656780261546373,
+            0.015843193978071213,
+            -0.0030765696428716183,
+            -0.026198700070381165,
+            -0.0025501344352960587,
+            0.0009018906857818365,
+            0.002495439723134041,
+            0.0029808541294187307,
+            -0.010628978721797466,
+            -0.005715582519769669,
+            -0.0010101404041051865,
+            -0.006226065568625927,
+            -0.0034092948772013187,
+            0.001910891616716981,
+            -0.011850490234792233,
+            0.0011998621048405766,
+            -0.02650863490998745,
+            0.00031335430685430765,
+            -0.012570635415613651,
+            -0.027912462130188942,
+            0.021075641736388206,
+            -0.005984497722238302,
+            0.011148576624691486,
+            0.022917024791240692,
+            0.019252488389611244,
+            -0.008792152628302574,
+            -0.004981763660907745,
+            0.0012978565646335483,
+            0.008673647418618202,
+            -0.014202357269823551,
+            -0.014594335108995438,
+            -0.009808559902012348,
+            -0.0019439362222328782,
+            -0.020218759775161743,
+            -0.021257957443594933,
+            -0.018231524154543877,
+            0.0010454639559611678,
+            0.01513216458261013,
+            0.0006540559115819633,
+            0.00459206523373723,
+            -0.016399255022406578,
+            0.00898814108222723,
+            -0.011030071415007114,
+            -0.011987227015197277,
+            0.003685046685859561,
+            0.004494070541113615,
+            -0.010793061926960945,
+            0.004712848924100399,
+            -0.005715582519769669,
+            -0.00301959621720016,
+            0.002584318397566676,
+            -0.014211473055183887,
+            -0.002613944699987769,
+            0.0006301270332187414,
+            0.025451207533478737,
+            0.0005329871783033013,
+            0.009772096760571003,
+            0.004045119509100914,
+            0.009188688360154629,
+            0.008267995901405811,
+            0.0003355739754624665,
+            0.0071695465594530106,
+            0.0014049668097868562,
+            -0.005401088856160641,
+            0.017620768398046494,
+            0.0011964436853304505,
+            -0.006586138159036636,
+            -0.007926154881715775,
+            -0.021403808146715164,
+            -0.003926614299416542,
+            0.0011098439572378993,
+            0.02180490270256996,
+            0.002579760504886508,
+            0.004731080494821072,
+            -0.013318127952516079,
+            -0.004443933721631765,
+            -0.004279850050806999,
+            -0.0008688460220582783,
+            0.0037830411456525326,
+            0.025651752948760986,
+            -0.004168182145804167,
+            -0.008819499984383583,
+            -0.01894255355000496,
+            -0.017711924389004707,
+            0.020145833492279053,
+            -0.006007287185639143,
+            0.006394707132130861,
+            -0.017420221120119095,
+            0.007789418566972017,
+            0.0064995381981134415,
+            0.018340913578867912,
+            -0.009717402048408985,
+            0.011704638600349426,
+            0.005246120970696211,
+            -0.0020077466033399105,
+            0.011959879659116268,
+            -0.0064995381981134415,
+            0.034038253128528595,
+            -0.014876923523843288,
+            0.009708286263048649,
+            0.0022048749960958958,
+            0.004213761072605848,
+            0.007844112813472748,
+            0.019070174545049667,
+            -0.004863258916884661,
+            -0.012762066908180714,
+            -0.005195984151214361,
+            -0.004863258916884661,
+            0.016800349578261375,
+            0.021841365844011307,
+            0.0046353647485375404,
+            -0.0008283948409371078,
+            0.0027415654622018337,
+            0.006171370856463909,
+            0.009726517833769321,
+            0.019070174545049667,
+            -0.01669096015393734,
+            0.005350952036678791,
+            -0.016353676095604897,
+            -0.009662707336246967,
+            0.00836826954036951,
+            -0.005843203514814377,
+            0.014147662557661533,
+            0.012616214342415333,
+            -0.000646079599391669,
+            -0.0004167612351011485,
+            0.015834078192710876,
+            -0.0021091594826430082,
+            0.006157697178423405,
+            0.012178657576441765,
+            -0.01100272499024868,
+            -0.001858475967310369,
+            -0.02680034004151821,
+            0.008053775876760483,
+            -0.023500433191657066,
+            -0.019471267238259315,
+            0.003286232240498066,
+            -0.025177733972668648,
+            0.005191426258534193,
+            0.018632616847753525,
+            0.01758430525660515,
+            -0.009143109433352947,
+            -0.010191421955823898,
+            -0.0009360748226754367,
+            0.02027345448732376,
+            -0.013555137440562248,
+            0.02032814919948578,
+            -0.002256151055917144,
+            0.004931627307087183,
+            0.007648124359548092,
+            -0.004412028938531876,
+            0.016216941177845,
+            -0.012734719552099705,
+            0.0024430241901427507,
+            -0.012588866986334324,
+            -0.0032702796161174774,
+            0.022151300683617592,
+            -0.02102094702422619,
+            0.027237895876169205,
+            0.009257055819034576,
+            -0.010109379887580872,
+            -0.0008483355632051826,
+            0.004177297931164503,
+            0.0215131975710392,
+            -0.03287143632769585,
+            -0.0010705322492867708,
+            -0.03239741921424866,
+            -0.001720600062981248,
+            -0.00011544257722562179,
+            0.03288966789841652,
+            -0.012251583859324455,
+            -0.019981749355793,
+            -0.002964901505038142,
+            0.006782126612961292,
+            -0.03305375203490257,
+            -0.001533726928755641,
+            0.007133083418011665,
+            -0.0037260677199810743,
+            0.01741110533475876,
+            0.0037807621993124485,
+            0.0022983115632086992,
+            -0.015186859294772148,
+            -0.012452131137251854,
+            0.0192160252481699,
+            0.016007278114557266,
+            -0.007351861800998449,
+            0.013509559445083141,
+            -0.022041911259293556,
+            -0.00624885456636548,
+            0.004491791594773531,
+            0.0030082014854997396,
+            -0.0008990420028567314,
+            0.034603431820869446,
+            0.018532343208789825,
+            -0.005829529836773872,
+            0.007456693332642317,
+            -0.0009639917989261448,
+            0.022096605971455574,
+            0.011841374449431896,
+            0.032452113926410675,
+            -0.00053782993927598,
+            0.03004555031657219,
+            -0.02858702838420868,
+            -0.00673199025914073,
+            -0.005259794648736715,
+            0.01412943098694086,
+            -0.025068344548344612,
+            0.015287132933735847,
+            -3.9169288356788456e-05,
+            0.028915196657180786,
+            0.009562434628605843,
+            0.0014608008787035942,
+            0.004250223748385906,
+            0.006344570312649012,
+            -0.0017376920441165566,
+            0.005241563078016043,
+            -0.0016909738769754767,
+            -0.026399245485663414,
+            -0.023445740342140198,
+            -0.0028122125659137964,
+            -0.018650848418474197,
+            0.014521408826112747,
+            -0.005770277231931686,
+            -0.006773010827600956,
+            -0.011795795522630215,
+            0.015688225626945496,
+            0.010638094507157803,
+            0.00916134100407362,
+            0.001238604192622006,
+            0.021185031160712242,
+            0.01832268200814724,
+            0.01577938348054886,
+            -0.003901546122506261,
+            0.0099908746778965,
+            -0.000999315408989787,
+            0.011485859751701355,
+            0.009699170477688313,
+            -0.011367355473339558,
+            0.020510464906692505,
+            -0.0032748375087976456,
+            -0.01415677834302187,
+            0.0075159454718232155,
+            -0.001373061677441001,
+            0.010638094507157803,
+            -0.02818593569099903,
+            0.0026754760183393955,
+            0.008591605350375175,
+            0.011412933468818665,
+            0.009917949326336384,
+            -0.01565176248550415,
+            -0.014895155094563961,
+            0.02450316771864891,
+            -0.005282584112137556,
+            0.01819506101310253,
+            0.0029762962367385626,
+            0.017912471666932106,
+            0.011914300732314587,
+            0.002582039451226592,
+            -0.021458502858877182,
+            -0.0015006823232397437,
+            0.0022686852607876062,
+            -0.0005760021740570664,
+            -0.008021870627999306,
+            -0.01054693665355444,
+            0.005597077775746584,
+            -0.036535974591970444,
+            0.004371007904410362,
+            -0.015031890943646431,
+            -0.02588876336812973,
+            0.004560159984976053,
+            -0.012935265898704529,
+            0.002186643425375223,
+            0.0032429322600364685,
+            0.02895165979862213,
+            -0.009999990463256836,
+            -0.02824063040316105,
+            1.6219646568060853e-05,
+            0.0030856854282319546,
+            -0.004113487433642149,
+            0.013309012167155743,
+            0.015788499265909195,
+            -0.0020054676569998264,
+            0.0026230604853481054,
+            0.042370062321424484,
+            -0.001320646028034389,
+            -0.006804916076362133,
+            0.00527346832677722,
+            -0.003721509827300906,
+            -0.012297162786126137,
+            -0.0003475384146440774,
+            0.008518679067492485,
+            0.011549670249223709,
+            -0.018587037920951843,
+            -0.00565633038058877,
+            0.010082032531499863,
+            0.030227866023778915,
+            0.017301715910434723,
+            0.013236085884273052,
+            -0.0024863239377737045,
+            0.005054689943790436,
+            -0.0018402443965896964,
+            -0.00023558546672575176,
+            -0.015578837133944035,
+            -0.029334520921111107,
+            -0.01970827765762806,
+            -0.0046991752460598946,
+            -0.01214219443500042,
+            -0.009908833540976048,
+            -0.011631712317466736,
+            0.0077347238548099995,
+            -0.0014608008787035942,
+            0.007014578673988581,
+            0.004357334226369858,
+            -0.010738367214798927,
+            0.010765714570879936,
+            0.022716479375958443,
+            0.017292600125074387,
+            -0.0018470812356099486,
+            0.01269825641065836,
+            -0.003842293517664075,
+            0.004008656367659569,
+            0.002835002029314637,
+            -0.010665440931916237,
+            -0.00978121254593134,
+            -0.0006865308387205005,
+            0.012415667995810509,
+            0.011349123902618885,
+            -0.01513216458261013,
+            0.002525066025555134,
+            0.011896069161593914,
+            0.010720135644078255,
+            0.0032953477930277586,
+            -0.020309917628765106,
+            0.015123048797249794,
+            0.0021205542143434286,
+            0.004703733138740063,
+            -0.0017638999270275235,
+            0.0008722644415684044,
+            -0.004024608992040157,
+            0.014211473055183887,
+            -0.01211484707891941,
+            0.014329977333545685,
+            0.024302620440721512,
+            0.01725613698363304,
+            0.005469457246363163,
+            0.007142199203372002,
+            0.008354595862329006,
+            0.0027278917841613293,
+            -0.009220592677593231,
+            -0.003871919820085168,
+            -0.0018413838697597384,
+            -0.00596626615151763,
+            0.023099340498447418,
+            -0.0092251505702734,
+            0.005414762534201145,
+            0.018906090408563614,
+            -0.008997256867587566,
+            -0.006467632949352264,
+            0.009279845282435417,
+            -0.002962622558698058,
+            -0.012187773361802101,
+            0.013263433240354061,
+            -0.002878301776945591,
+            0.01030081044882536,
+            0.014530524611473083,
+            -0.0013844562927260995,
+            -0.0022504536900669336,
+            -0.00414767162874341,
+            -0.0074475775472819805,
+            0.06745664030313492,
+            -0.015123048797249794,
+            0.004466723185032606,
+            -0.010984493419528008,
+            -0.017155863344669342,
+            0.014530524611473083,
+            0.02027345448732376,
+            0.02942567877471447,
+            -0.010701904073357582,
+            0.003972193226218224,
+            0.000644940126221627,
+            0.0017958050593733788,
+            0.018085671588778496,
+            0.006800358183681965,
+            -0.0009081577882170677,
+            -0.017265252768993378,
+            -0.01842295378446579,
+            -0.015241554006934166,
+            0.00473563838750124,
+            -0.01881493255496025,
+            0.0038149463944137096,
+            0.0046991752460598946,
+            -0.001347993267700076,
+            -0.013783032074570656,
+            0.005879666190594435,
+            0.003363716183230281,
+            -0.020510464906692505,
+            -0.0018744284752756357,
+            -0.008049217984080315,
+            -0.04740196093916893,
+            0.017721040174365044,
+            -0.005820414051413536,
+            0.011722870171070099,
+            -0.0032429322600364685,
+            0.016800349578261375,
+            0.007292609661817551,
+            0.004040561616420746,
+            0.005241563078016043,
+            0.009635359980165958,
+            0.009562434628605843,
+            0.013090234249830246,
+            -0.00982679147273302,
+            0.013072002679109573,
+            0.013637179508805275,
+            -0.007939828559756279,
+            -0.004498628433793783,
+            0.013454864732921124,
+            -0.006125791929662228,
+            -0.0011434584157541394,
+            0.0047584278509020805,
+            0.009744749404489994,
+            0.0062807598151266575,
+            -0.0013685037847608328,
+            0.004858701024204493,
+            -0.0004552183672785759,
+            0.02824063040316105,
+            0.0019040547776967287,
+            -0.00737465126439929,
+            -0.015496795065701008,
+            -0.01800362952053547,
+            -0.022096605971455574,
+            -0.010802177712321281,
+            -0.014074736274778843,
+            0.002639013109728694,
+            -0.05677296593785286,
+            0.022716479375958443,
+            0.0028144915122538805,
+            0.0037465780042111874,
+            -0.0024339084047824144,
+            -0.013810379430651665,
+            0.030665423721075058,
+            -0.004749312065541744,
+            -0.022096605971455574,
+            -0.01602550968527794,
+            0.011941648088395596,
+            0.002258430002257228,
+            0.0018539180746302009,
+            -0.009863254614174366,
+            -0.0026731970719993114,
+            -0.0013673643115907907,
+            0.006736548151820898,
+            0.014038273133337498,
+            0.005478573031723499,
+            0.007164988666772842,
+            0.02393799088895321,
+            0.007201451808214188,
+            -0.01711028441786766,
+            0.007538734935224056,
+            -0.008281669579446316,
+            0.0039995405822992325,
+            -0.005765719339251518,
+            -0.002180946059525013,
+            0.003687325632199645,
+            -0.00339334225282073,
+            -0.0075478507205843925,
+            -0.011349123902618885,
+            0.002987690968438983,
+            0.004881490487605333,
+            -0.003343205666169524,
+            -0.002072696341201663,
+            0.01467637624591589,
+            0.0217137448489666,
+            0.015423868782818317,
+            -0.0008209883235394955,
+            -0.005628983024507761,
+            0.014320861548185349,
+            0.0107565987855196,
+            0.005319047253578901,
+            -0.007898807525634766,
+            -0.006513211876153946,
+            0.007092062849551439,
+            -0.01738375797867775,
+            0.0050774794071912766,
+            -0.01950773037970066,
+            0.0003783041029237211,
+            0.005852319300174713,
+            0.016681844368577003,
+            -0.0005318477051332593,
+            -0.006162255071103573,
+            0.0037374624516814947,
+            0.0077666291035711765,
+            0.003691883524879813,
+            0.01515951193869114,
+            0.014357324689626694,
+            -0.010273464024066925,
+            0.022114837542176247,
+            0.000282588618574664,
+            -0.007493156008422375,
+            -0.0023814928717911243,
+            -0.0027051023207604885,
+            0.01985413022339344,
+            -0.0009167037787847221,
+            0.015123048797249794,
+            -0.006435727700591087,
+            0.020802168175578117,
+            0.007206009700894356,
+            0.004767543636262417,
+            0.025797605514526367,
+            0.01651776023209095,
+            0.002832723082974553,
+            0.012360973283648491,
+            0.08350037783384323,
+            0.018240639939904213,
+            -0.010975377634167671,
+            -0.017547842115163803,
+            -0.0047265226021409035,
+            -0.0071695465594530106,
+            0.003413852769881487,
+            0.017493147403001785,
+            -0.017292600125074387,
+            -0.023354582488536835,
+            0.0006494980189017951,
+            0.000889356539119035,
+            0.0037875990383327007,
+            -0.01554237399250269,
+            0.01780308224260807,
+            0.015296248719096184,
+            -0.0013821773463860154,
+            0.0046376436948776245,
+            0.005765719339251518,
+            -0.007593429647386074,
+            -0.007620777003467083,
+            0.010401084087789059,
+            -0.015387405641376972,
+            -0.014102083630859852,
+            0.01190518494695425,
+            -0.004876932594925165,
+            -0.00021450527128763497,
+            0.004020051099359989,
+            0.010082032531499863,
+            -0.00183112861122936,
+            0.015287132933735847,
+            0.02353689633309841,
+            0.004532812628895044,
+            0.003794435877352953,
+            0.015268901363015175,
+            0.01530536450445652,
+            -0.00046746767475269735,
+            -0.0036166785284876823,
+            -0.01671830751001835,
+            0.009626244194805622,
+            0.006845937110483646,
+            -0.015323596075177193,
+            0.016216941177845,
+            0.02193252369761467,
+            0.006257970351725817,
+            0.006485864520072937,
+            0.00468550156801939,
+            -0.0004227434692438692,
+            0.01916133053600788,
+            0.000195134271052666,
+            -0.0034571527503430843,
+            0.006230622995644808,
+            0.007520503364503384,
+            -0.012898802757263184,
+            0.017839545384049416,
+            0.013664526864886284,
+            0.012379204854369164,
+            0.007319956552237272,
+            0.0037853200919926167,
+            0.010720135644078255,
+            -0.022443005815148354,
+            0.016508644446730614,
+            -0.018514111638069153
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "3ab75d4b-415b-4dc3-9990-7f4556a5dec5",
+      "type": "next",
+      "source": {
+        "id": "be3079d3-622e-4848-bff6-15e3ac00f52b",
+        "properties": {
+          "page_content": "7 Discussion 25 ",
+          "keyphrases": [
+            "Discussion"
+          ],
+          "title": "7 Discussion",
+          "embedding": [
+            -0.015314480289816856,
+            0.01669096015393734,
+            -0.018997248262166977,
+            0.02089332602918148,
+            -0.021677281707525253,
+            -0.005907013546675444,
+            -0.010774830356240273,
+            0.0937100350856781,
+            -0.00431403424590826,
+            0.05294434353709221,
+            0.0064402855932712555,
+            -0.027529601007699966,
+            -0.014475829899311066,
+            -0.01842295378446579,
+            -0.024448473006486893,
+            0.03755693882703781,
+            -0.012051037512719631,
+            0.033710088580846786,
+            -0.006408380344510078,
+            -0.030100245028734207,
+            0.01741110533475876,
+            -0.012816760689020157,
+            0.011850490234792233,
+            0.07365535199642181,
+            -0.013801263645291328,
+            0.025287123396992683,
+            0.04072922468185425,
+            0.02200544998049736,
+            -0.03755693882703781,
+            0.04565173387527466,
+            -0.010009106248617172,
+            0.004895164165645838,
+            -0.0015485400799661875,
+            -0.04878755658864975,
+            -0.024029148742556572,
+            0.00932086631655693,
+            0.03347307816147804,
+            -0.010191421955823898,
+            0.033710088580846786,
+            0.02512303926050663,
+            -0.030592497438192368,
+            -0.039453018456697464,
+            -0.0070100207813084126,
+            0.0217137448489666,
+            0.010382852517068386,
+            -0.006080213002860546,
+            -0.025505902245640755,
+            -0.018559690564870834,
+            0.0007360977469943464,
+            0.012707372196018696,
+            -0.016827696934342384,
+            -0.04269822686910629,
+            0.01159524917602539,
+            0.0005062096752226353,
+            0.014184125699102879,
+            -0.00020211352966725826,
+            0.011440280824899673,
+            0.034111179411411285,
+            0.00420920317992568,
+            -0.02865995466709137,
+            0.003798993770033121,
+            -0.044375527650117874,
+            -0.004564717877656221,
+            -0.004081582184880972,
+            -0.016408370807766914,
+            -0.02123972587287426,
+            -0.013500443659722805,
+            0.01520509086549282,
+            -0.02880580723285675,
+            -0.021130336448550224,
+            -0.014421135187149048,
+            -0.008003639057278633,
+            -0.0213126502931118,
+            0.04120324179530144,
+            -0.017447568476200104,
+            -0.012944381684064865,
+            0.03330899402499199,
+            0.01408385206013918,
+            -0.019799435511231422,
+            0.044448453933000565,
+            0.02428438887000084,
+            -0.019070174545049667,
+            -0.025378281250596046,
+            0.030829505994915962,
+            0.029790310189127922,
+            -0.01874200627207756,
+            -0.06716493517160416,
+            -0.11777564138174057,
+            0.026891497895121574,
+            0.022114837542176247,
+            0.006271644029766321,
+            -0.012962613254785538,
+            -0.012224236503243446,
+            -0.0341658741235733,
+            -0.009398350492119789,
+            -0.00598905561491847,
+            -0.01561530027538538,
+            -0.0026207815390080214,
+            0.023099340498447418,
+            -0.012780298478901386,
+            0.03454873710870743,
+            0.0032497690990567207,
+            0.0071695465594530106,
+            0.024685483425855637,
+            0.022461237385869026,
+            -0.00037574031739495695,
+            0.02880580723285675,
+            0.016700075939297676,
+            -0.0015508190263062716,
+            0.007880575954914093,
+            -0.024612557142972946,
+            0.022953487932682037,
+            -0.030009087175130844,
+            -0.010428431443870068,
+            0.00737465126439929,
+            -0.015505910851061344,
+            0.006800358183681965,
+            -0.04754781350493431,
+            -0.017155863344669342,
+            0.016453949734568596,
+            -0.0017046474386006594,
+            -0.005405646748840809,
+            -0.03053780272603035,
+            -0.02102094702422619,
+            -0.027183201164007187,
+            0.006253412459045649,
+            4.162627010373399e-05,
+            -0.011121229268610477,
+            -0.005282584112137556,
+            0.016435718163847923,
+            0.006668179761618376,
+            0.04557880759239197,
+            -0.02499541826546192,
+            0.029188670217990875,
+            0.002563808113336563,
+            -0.005733814090490341,
+            0.002456697868183255,
+            0.02665448747575283,
+            -0.008040102198719978,
+            -0.01068367250263691,
+            -0.02235184796154499,
+            -0.003769367467612028,
+            0.006385591346770525,
+            0.02900635451078415,
+            -0.01597081497311592,
+            0.023901527747511864,
+            -0.0013240644475445151,
+            0.021640818566083908,
+            0.006157697178423405,
+            0.021549660712480545,
+            0.0255423653870821,
+            -0.012470361776649952,
+            -0.0430263951420784,
+            0.03489513695240021,
+            0.006426611915230751,
+            -0.010510473512113094,
+            -0.009380118921399117,
+            0.014931618236005306,
+            -0.012133078649640083,
+            0.010865988209843636,
+            0.015414752997457981,
+            0.06450313329696655,
+            -0.022096605971455574,
+            -0.0035095682833343744,
+            0.04204189404845238,
+            -0.01353690680116415,
+            0.02561529166996479,
+            -0.008409290574491024,
+            -0.04204189404845238,
+            0.019544193521142006,
+            0.04448491707444191,
+            0.008309016935527325,
+            0.004799448419362307,
+            -0.012333625927567482,
+            0.008514121174812317,
+            0.02242477424442768,
+            -0.003452594857662916,
+            -0.0010278021218255162,
+            -0.014074736274778843,
+            0.023281656205654144,
+            -0.04393797367811203,
+            0.003309021471068263,
+            0.012762066908180714,
+            -0.0032725585624575615,
+            -0.034311726689338684,
+            -0.01408385206013918,
+            0.04587051272392273,
+            -0.02650863490998745,
+            0.04572466015815735,
+            0.010182306170463562,
+            -0.0075751980766654015,
+            -0.002725612837821245,
+            -0.007570640183985233,
+            -0.01061986293643713,
+            -0.014995427802205086,
+            -0.06351862847805023,
+            0.01214219443500042,
+            0.011686407029628754,
+            0.0002243331982754171,
+            0.01436644047498703,
+            -0.021677281707525253,
+            -0.013646295294165611,
+            -0.00037232189788483083,
+            0.007383767049759626,
+            -0.011987227015197277,
+            -0.0016602081013843417,
+            0.010364620946347713,
+            0.022734710946679115,
+            0.03617134317755699,
+            -0.024867799133062363,
+            0.010701904073357582,
+            0.02435731515288353,
+            -0.022151300683617592,
+            0.04054690897464752,
+            0.01696443371474743,
+            0.008126701228320599,
+            -0.019197793677449226,
+            0.06319046020507812,
+            0.01896078512072563,
+            -0.040218740701675415,
+            0.025432975962758064,
+            -0.000753759581129998,
+            0.01708293706178665,
+            0.026745645329356194,
+            -0.012634445913136005,
+            0.026071079075336456,
+            -0.02359159104526043,
+            0.021549660712480545,
+            -0.01819506101310253,
+            -0.052907880395650864,
+            0.02603461593389511,
+            0.014566987752914429,
+            -0.01453964039683342,
+            -0.010100264102220535,
+            -0.041677262634038925,
+            -0.026198700070381165,
+            -0.025761142373085022,
+            0.0063126650638878345,
+            0.024047380313277245,
+            0.03068365529179573,
+            0.029735615476965904,
+            -0.010109379887580872,
+            -0.040984466671943665,
+            0.010364620946347713,
+            -0.054803960025310516,
+            0.020783936604857445,
+            -0.025141270831227303,
+            0.029553299769759178,
+            0.0002243331982754171,
+            -0.010920682922005653,
+            0.020072907209396362,
+            -0.007365535479038954,
+            0.012406552210450172,
+            0.029808541759848595,
+            -0.008204185403883457,
+            0.045979902148246765,
+            0.04481308534741402,
+            -0.0034844998735934496,
+            -0.0023837718181312084,
+            0.004040561616420746,
+            -0.018204176798462868,
+            -0.010437547229230404,
+            -0.032379187643527985,
+            -0.014074736274778843,
+            0.02138557657599449,
+            -0.01175933238118887,
+            0.021458502858877182,
+            0.004427981097251177,
+            0.026472171768546104,
+            -0.009024604223668575,
+            0.02568821609020233,
+            -0.01901547983288765,
+            0.010391968302428722,
+            0.005587961990386248,
+            -0.027092043310403824,
+            -0.014402903616428375,
+            0.01896078512072563,
+            -0.005309931468218565,
+            0.004931627307087183,
+            0.032999057322740555,
+            0.00916134100407362,
+            0.03061072900891304,
+            -0.0073883249424397945,
+            -0.025943458080291748,
+            -0.010556052438914776,
+            -0.035223305225372314,
+            -0.016052857041358948,
+            0.024813104420900345,
+            -0.012716487981379032,
+            -0.0217137448489666,
+            0.009999990463256836,
+            -0.005027342587709427,
+            -0.0037306256126612425,
+            0.017228789627552032,
+            0.011431165039539337,
+            -0.009234266355633736,
+            0.021330881863832474,
+            0.005095710977911949,
+            -0.0022117116022855043,
+            0.005628983024507761,
+            -0.009735633619129658,
+            0.021257957443594933,
+            -0.024667251855134964,
+            0.001667044940404594,
+            -0.009516855701804161,
+            0.01627163402736187,
+            0.005911571439355612,
+            0.009343655779957771,
+            -0.0852506011724472,
+            -0.020236991345882416,
+            0.06257058680057526,
+            -0.03839558735489845,
+            -0.009361887350678444,
+            0.009972643107175827,
+            -0.017000894993543625,
+            0.019252488389611244,
+            0.02497718669474125,
+            0.013710105791687965,
+            -0.01644483394920826,
+            0.0024361873511224985,
+            0.02082039974629879,
+            0.005670004058629274,
+            -0.029735615476965904,
+            -0.004749312065541744,
+            0.0068185897544026375,
+            0.002308566588908434,
+            -0.024047380313277245,
+            0.027092043310403824,
+            0.006882400251924992,
+            0.0013126697158440948,
+            0.023792138323187828,
+            -0.03234272450208664,
+            0.005856877192854881,
+            -0.004876932594925165,
+            -0.00793527066707611,
+            0.004968089982867241,
+            -0.041677262634038925,
+            0.03624426946043968,
+            0.002105741063132882,
+            -0.030993590131402016,
+            0.01408385206013918,
+            0.009817675687372684,
+            0.011777563951909542,
+            0.02220599539577961,
+            -0.012579751200973988,
+            -0.006855052895843983,
+            0.01461256667971611,
+            -0.023281656205654144,
+            4.2480874981265515e-05,
+            0.010364620946347713,
+            0.000994757516309619,
+            0.006595253478735685,
+            -0.013582484796643257,
+            0.014949849806725979,
+            -0.0014881481183692813,
+            -0.035843174904584885,
+            -0.05560614541172981,
+            -0.014448482543230057,
+            0.0003068023652303964,
+            9.008936467580497e-05,
+            0.04477662220597267,
+            -0.013381938450038433,
+            -0.00829534325748682,
+            0.0055834040977060795,
+            -0.029024586081504822,
+            -0.07099355012178421,
+            0.035369157791137695,
+            0.022661784663796425,
+            -0.024940723553299904,
+            0.014758418314158916,
+            -0.03228802978992462,
+            -0.02789423055946827,
+            0.01589788869023323,
+            -0.020492233335971832,
+            -0.017119400203227997,
+            0.0018675917526707053,
+            -0.007629892788827419,
+            -0.005888781975954771,
+            -0.03201455622911453,
+            0.026417477056384087,
+            -0.02269824780523777,
+            0.01121238712221384,
+            -0.040911540389060974,
+            -0.013281664811074734,
+            0.005510477814823389,
+            -2.746301333900192e-06,
+            0.01567910984158516,
+            0.026289857923984528,
+            -0.02116679958999157,
+            0.00796261802315712,
+            -0.04754781350493431,
+            -0.012379204854369164,
+            -0.006859610788524151,
+            -0.017265252768993378,
+            0.03108474798500538,
+            0.006203276105225086,
+            0.01651776023209095,
+            -0.00870555266737938,
+            0.036262501031160355,
+            -0.030446644872426987,
+            -0.00527346832677722,
+            0.018577922135591507,
+            -0.006581580266356468,
+            -0.017401989549398422,
+            0.04196896776556969,
+            0.00760254543274641,
+            -0.019598888233304024,
+            -0.0020943463314324617,
+            -0.005838645622134209,
+            0.0072287991642951965,
+            0.009726517833769321,
+            -0.009917949326336384,
+            0.036116648465394974,
+            -0.011185039766132832,
+            0.06468544900417328,
+            0.0025205081328749657,
+            0.04565173387527466,
+            0.05367360636591911,
+            -0.02971738390624523,
+            -0.01061986293643713,
+            0.007739281747490168,
+            0.0238103698939085,
+            0.026818571612238884,
+            -0.017219673842191696,
+            -0.025232428684830666,
+            0.03615311160683632,
+            -0.015788499265909195,
+            -0.0039106616750359535,
+            0.010319042019546032,
+            -0.030556034296751022,
+            -0.028714649379253387,
+            0.04200543090701103,
+            -0.020528696477413177,
+            -0.007775744888931513,
+            0.018632616847753525,
+            -0.01936187781393528,
+            0.005574288312345743,
+            0.008422964252531528,
+            -0.03174108266830444,
+            0.024029148742556572,
+            0.010154958814382553,
+            -0.005246120970696211,
+            -0.015569721348583698,
+            0.046052828431129456,
+            -0.017593421041965485,
+            0.0016066529788076878,
+            -0.031412914395332336,
+            -0.021203262731432915,
+            -0.04477662220597267,
+            0.035843174904584885,
+            -0.03547854349017143,
+            0.029899699613451958,
+            -0.038213275372982025,
+            0.010574284009635448,
+            0.036061953753232956,
+            -0.00958066526800394,
+            0.0019553308375179768,
+            0.018924321979284286,
+            -0.020510464906692505,
+            -0.06891515851020813,
+            0.0106563251465559,
+            0.022734710946679115,
+            0.021002715453505516,
+            -0.015405637212097645,
+            -0.0364995114505291,
+            0.03330899402499199,
+            -0.0038628040347248316,
+            -0.03163169324398041,
+            0.005783950909972191,
+            0.018796700984239578,
+            -0.015834078192710876,
+            -0.010382852517068386,
+            -0.05425701290369034,
+            -0.005382857285439968,
+            -0.0376298651099205,
+            -0.024047380313277245,
+            0.0347675159573555,
+            -0.013117581605911255,
+            -0.04510479047894478,
+            0.03901546075940132,
+            -0.033837709575891495,
+            0.011996342800557613,
+            -0.0033477635588496923,
+            0.022224226966500282,
+            -0.02242477424442768,
+            0.015587952919304371,
+            0.042370062321424484,
+            -0.041458483785390854,
+            0.004712848924100399,
+            0.039380092173814774,
+            0.03296259418129921,
+            -0.01585230976343155,
+            0.06618043035268784,
+            -0.005820414051413536,
+            -0.013728337362408638,
+            -0.06771188229322433,
+            -0.00018388200260233134,
+            -0.07817677408456802,
+            -0.001375340623781085,
+            0.03365539386868477,
+            -0.008901541121304035,
+            -0.019325414672493935,
+            -0.033071983605623245,
+            0.03312667831778526,
+            0.009206918999552727,
+            -0.025177733972668648,
+            0.011713754385709763,
+            -0.04579758644104004,
+            -0.010802177712321281,
+            0.0192160252481699,
+            0.027985388413071632,
+            0.005551498848944902,
+            0.00091556430561468,
+            -0.012105731293559074,
+            -0.019197793677449226,
+            -0.007543292827904224,
+            -0.016645381227135658,
+            0.03132175654172897,
+            -0.004416586831212044,
+            0.012652677483856678,
+            0.0019040547776967287,
+            -0.007648124359548092,
+            -0.03387417271733284,
+            0.04324517399072647,
+            -0.01773015595972538,
+            0.012570635415613651,
+            0.013181391172111034,
+            -0.008810384199023247,
+            -0.0016602081013843417,
+            0.011257966049015522,
+            0.0009468997595831752,
+            0.004343660548329353,
+            0.006030076649039984,
+            0.022187763825058937,
+            0.031212368980050087,
+            -0.0015850031049922109,
+            0.0024908818304538727,
+            0.01916133053600788,
+            0.007939828559756279,
+            0.026490403339266777,
+            -0.0023814928717911243,
+            0.024193231016397476,
+            -0.030628960579633713,
+            -0.028295325115323067,
+            0.007643566466867924,
+            0.01916133053600788,
+            -0.00929807685315609,
+            -0.00896535161882639,
+            -0.015861425548791885,
+            -0.02915220707654953,
+            0.02386506460607052,
+            -0.0023792139254510403,
+            0.008290785364806652,
+            -0.01992705464363098,
+            0.0021137173753231764,
+            -0.0013696432579308748,
+            0.0315040722489357,
+            0.019416572526097298,
+            0.013728337362408638,
+            -0.004619412124156952,
+            0.011084766127169132,
+            0.0028988122940063477,
+            -0.027164969593286514,
+            0.0028646280989050865,
+            -0.011048302985727787,
+            -0.007023694459348917,
+            -0.014348208904266357,
+            0.043682731688022614,
+            0.004453049506992102,
+            -0.025031881406903267,
+            0.009571549482643604,
+            0.00199065450578928,
+            -0.01679123379290104,
+            -0.009097530506551266,
+            -0.0017673183465376496,
+            0.033965326845645905,
+            0.027875998988747597,
+            -0.028204167261719704,
+            -0.02499541826546192,
+            0.006207833997905254,
+            0.013783032074570656,
+            0.002743844408541918,
+            0.009872370399534702,
+            -0.004425702150911093,
+            -5.647499347105622e-05,
+            -0.01207838486880064,
+            -0.009389234706759453,
+            -0.02025522291660309,
+            -0.009617128409445286,
+            0.020802168175578117,
+            0.025998152792453766,
+            0.020364612340927124,
+            0.031066516414284706,
+            -0.04634453356266022,
+            -0.010957146063446999,
+            0.016453949734568596,
+            0.04856877774000168,
+            0.025159502401947975,
+            -0.006408380344510078,
+            -0.03290789946913719,
+            -0.008787594735622406,
+            0.010893335565924644,
+            -0.010200537741184235,
+            0.012461245991289616,
+            -0.01731083169579506,
+            0.004026887938380241,
+            -0.03194162994623184,
+            0.013427517376840115,
+            -0.0015154953580349684,
+            0.02700088731944561,
+            -0.02020052820444107,
+            0.012169541791081429,
+            0.01894255355000496,
+            -0.02957153134047985,
+            -0.011987227015197277,
+            -0.0315040722489357,
+            -0.003691883524879813,
+            0.001616908237338066,
+            -0.01589788869023323,
+            0.0061440235003829,
+            0.014402903616428375,
+            -0.010492241941392422,
+            0.0011018677614629269,
+            0.024466704577207565,
+            0.019106637686491013,
+            0.0077939764596521854,
+            -0.00453053368255496,
+            0.0063400124199688435,
+            -0.009361887350678444,
+            -0.028623491525650024,
+            -0.03219687193632126,
+            0.018331797793507576,
+            0.02180490270256996,
+            0.003411573823541403,
+            -0.006175928749144077,
+            0.005095710977911949,
+            0.057283446192741394,
+            -0.026891497895121574,
+            0.018833164125680923,
+            0.0016043740324676037,
+            -0.03901546075940132,
+            -0.031832240521907806,
+            -0.003343205666169524,
+            0.06089328974485397,
+            -0.028222398832440376,
+            0.01121238712221384,
+            0.015606184490025043,
+            0.01817682944238186,
+            0.002261848421767354,
+            0.0003395621315576136,
+            -0.034184105694293976,
+            0.014329977333545685,
+            -0.01207838486880064,
+            0.01894255355000496,
+            0.031394682824611664,
+            -0.04072922468185425,
+            -0.004804006312042475,
+            -0.021950755268335342,
+            -0.0012830435298383236,
+            -0.024029148742556572,
+            -0.024175001308321953,
+            0.00619416031986475,
+            0.00550136249512434,
+            0.0071695465594530106,
+            -0.0047265226021409035,
+            -0.006444843485951424,
+            -0.0015838636318221688,
+            -0.00231198500841856,
+            0.00924794003367424,
+            -0.007014578673988581,
+            0.0008050356991589069,
+            -0.01159524917602539,
+            0.01718321070075035,
+            0.0007492016884498298,
+            -0.03832266107201576,
+            -0.013491327874362469,
+            -0.020036444067955017,
+            0.011950763873755932,
+            -0.0008283948409371078,
+            -0.01908840611577034,
+            -0.020036444067955017,
+            -0.011340008117258549,
+            0.010948030278086662,
+            0.01696443371474743,
+            -0.02123972587287426,
+            0.0044781179167330265,
+            0.008669089525938034,
+            -0.0035095682833343744,
+            -0.007680029142647982,
+            -0.0018493601819500327,
+            0.0215131975710392,
+            0.04940742999315262,
+            0.004020051099359989,
+            0.006016402970999479,
+            -0.007771186996251345,
+            -0.017338179051876068,
+            -0.04102092981338501,
+            0.03923423960804939,
+            0.01339105423539877,
+            0.029735615476965904,
+            -0.00248176627792418,
+            -0.04058337211608887,
+            -0.0016624870477244258,
+            -0.01783042959868908,
+            0.004172740038484335,
+            -0.0034298053942620754,
+            -0.008637184277176857,
+            0.004475838970392942,
+            -0.017319947481155396,
+            -0.0326344259083271,
+            -0.016873275861144066,
+            -0.029334520921111107,
+            0.0009634220623411238,
+            0.022661784663796425,
+            -0.021604355424642563,
+            0.009361887350678444,
+            -0.0025979920756071806,
+            -0.039963498711586,
+            -0.007433903869241476,
+            -0.003502731444314122,
+            -0.01398357842117548,
+            0.013719221577048302,
+            -0.014220588840544224,
+            0.0023245192132890224,
+            -0.005054689943790436,
+            0.01467637624591589,
+            0.025232428684830666,
+            -0.02138557657599449,
+            0.005373741500079632,
+            -0.0001051873477990739,
+            -0.00041220334242098033,
+            -0.013609832152724266,
+            0.0005908153252676129,
+            -0.0012819040566682816,
+            -0.020109370350837708,
+            0.018131250515580177,
+            0.004927069414407015,
+            0.02776661142706871,
+            0.016636265441775322,
+            -0.03312667831778526,
+            0.010072916746139526,
+            -0.002771191531792283,
+            -0.01696443371474743,
+            0.015496795065701008,
+            -0.012898802757263184,
+            0.0008266856311820447,
+            0.02422969415783882,
+            0.017620768398046494,
+            -0.018377376720309258,
+            0.003115311497822404,
+            0.00978121254593134,
+            -0.04247945174574852,
+            0.062351811677217484,
+            -0.00193937832955271,
+            -0.03347307816147804,
+            0.016499528661370277,
+            0.018304450437426567,
+            0.01673653908073902,
+            -0.0045669968239963055,
+            0.00888786744326353,
+            -0.002016862388700247,
+            0.03555146977305412,
+            0.012944381684064865,
+            0.04776659235358238,
+            0.029407447203993797,
+            -0.014566987752914429,
+            -0.006458517163991928,
+            -0.01819506101310253,
+            -0.00801275484263897,
+            0.01683681271970272,
+            0.008960793726146221,
+            0.01742933690547943,
+            0.021458502858877182,
+            -0.013546021655201912,
+            0.015861425548791885,
+            0.005382857285439968,
+            0.00898814108222723,
+            -0.05330897495150566,
+            -0.0430263951420784,
+            0.007543292827904224,
+            -0.01565176248550415,
+            -0.033071983605623245,
+            -0.005929803010076284,
+            -0.004872374702244997,
+            -0.022607089951634407,
+            0.0025205081328749657,
+            0.041057392954826355,
+            -0.024175001308321953,
+            0.027711916714906693,
+            0.007306283339858055,
+            0.006303549278527498,
+            0.0255423653870821,
+            -0.02200544998049736,
+            -0.00387875665910542,
+            0.043974436819553375,
+            -0.04933450371026993,
+            0.009516855701804161,
+            -0.0050455741584300995,
+            -0.009197804145514965,
+            -0.015050122514367104,
+            0.0315040722489357,
+            0.009653591550886631,
+            -0.0064721908420324326,
+            0.015761151909828186,
+            0.01950773037970066,
+            0.0006551953847520053,
+            0.016326328739523888,
+            -0.020528696477413177,
+            -0.034238800406455994,
+            0.030027318745851517,
+            0.0009104367345571518,
+            -0.008144932799041271,
+            0.0414949469268322,
+            0.009945295751094818,
+            0.010975377634167671,
+            0.0034252475015819073,
+            -0.007392882835119963,
+            -0.026125773787498474,
+            0.00699178921058774,
+            0.023226961493492126,
+            0.025925226509571075,
+            0.011768448166549206,
+            0.020419307053089142,
+            -0.006025518756359816,
+            0.008286227472126484,
+            0.04732903465628624,
+            0.008760247379541397,
+            0.006900631822645664,
+            0.042588841170072556,
+            0.035369157791137695,
+            -0.02484956756234169,
+            -0.020984483882784843,
+            -0.00975386518985033,
+            0.022807635366916656,
+            0.0006956465658731759,
+            0.02054692804813385,
+            0.011221502907574177,
+            -0.0035665417090058327,
+            -0.005460341461002827,
+            0.030501339584589005,
+            -0.014393787831068039,
+            0.002228803699836135,
+            -0.0006791242631152272,
+            -0.017392873764038086,
+            -0.003037827555090189,
+            0.004402913153171539,
+            -0.020437538623809814,
+            0.009279845282435417,
+            0.02422969415783882,
+            0.00045806702109985054,
+            0.018979016691446304,
+            0.019124869257211685,
+            0.006358243990689516,
+            -0.0029717383440583944,
+            -0.024302620440721512,
+            -0.05061070993542671,
+            0.021257957443594933,
+            0.019343646243214607,
+            0.0026891496963799,
+            0.0029238807037472725,
+            0.012944381684064865,
+            0.024685483425855637,
+            -0.00985413882881403,
+            -0.02032814919948578,
+            0.03773925453424454,
+            -0.02319049835205078,
+            -0.008824057877063751,
+            -0.03159523010253906,
+            0.006791242398321629,
+            -0.023500433191657066,
+            -0.01207838486880064,
+            -0.020164065062999725,
+            -0.0066317166201770306,
+            0.0058340877294540405,
+            0.002695986535400152,
+            -0.0036235153675079346,
+            -0.012552403844892979,
+            0.001857336494140327,
+            0.004516860004514456,
+            -0.031759314239025116,
+            -0.02123972587287426,
+            -0.027456674724817276,
+            -0.0253418181091547,
+            -0.006207833997905254,
+            -0.02242477424442768,
+            0.003976751118898392,
+            -0.01965358294546604,
+            -0.046818554401397705,
+            -0.008546026423573494,
+            -0.010583399794995785,
+            0.0036736519541591406,
+            -0.006367359776049852,
+            0.014958965592086315,
+            -0.03157699853181839,
+            0.0030606170184910297,
+            0.003632631152868271,
+            -0.032105714082717896,
+            0.011896069161593914,
+            -0.011750216595828533,
+            -0.005565172526985407,
+            -0.00603919243440032,
+            0.0063400124199688435,
+            0.019471267238259315,
+            -0.032105714082717896,
+            -0.010391968302428722,
+            -0.028550565242767334,
+            0.024594325572252274,
+            0.004010935313999653,
+            0.014557871967554092,
+            0.006125791929662228,
+            -0.027365516871213913,
+            -0.02275294065475464,
+            -0.03713761270046234,
+            -0.02816770412027836,
+            0.010975377634167671,
+            0.02282586693763733,
+            -0.0024133978877216578,
+            0.016390139237046242,
+            -0.0016271634958684444,
+            0.024138538166880608,
+            0.010273464024066925,
+            -0.0008152909576892853,
+            -0.029024586081504822,
+            0.009644475765526295,
+            0.00855514220893383,
+            -0.04350041598081589,
+            -0.030009087175130844,
+            0.023026414215564728,
+            -0.013400170020759106,
+            0.009945295751094818,
+            0.021494966000318527,
+            0.017839545384049416,
+            0.026526866480708122,
+            0.006941652856767178,
+            0.027748379856348038,
+            -0.0115223228931427,
+            -0.016973547637462616,
+            0.031194137409329414,
+            0.015551489777863026,
+            -0.012725603766739368,
+            0.02712850645184517,
+            0.01391065213829279,
+            0.0023792139254510403,
+            0.008318132720887661,
+            0.021494966000318527,
+            0.005305373575538397,
+            0.0013343195896595716,
+            0.04977206140756607,
+            0.01817682944238186,
+            -0.008842289447784424,
+            -0.019525961950421333,
+            0.007429345976561308,
+            0.00982679147273302,
+            -0.007338188122957945,
+            0.021987218409776688,
+            0.00015268901188392192,
+            -0.04503186419606209,
+            0.01824975572526455,
+            0.019124869257211685,
+            0.024065611883997917,
+            0.0017057869117707014,
+            0.008277111686766148,
+            -0.027930693700909615,
+            0.00732451444491744,
+            -0.01023700088262558,
+            0.01985413022339344,
+            0.008381943218410015,
+            0.03226979821920395,
+            -0.03886960819363594,
+            0.020929789170622826,
+            -0.001477892859838903,
+            0.002021420281380415,
+            0.02601638436317444,
+            -0.020109370350837708,
+            -0.029735615476965904,
+            0.0013912931317463517,
+            -0.028678186237812042,
+            -0.011896069161593914,
+            0.004143113736063242,
+            0.015688225626945496,
+            0.004525975789874792,
+            -0.013719221577048302,
+            0.01617136225104332,
+            -0.008117586374282837,
+            -0.01589788869023323,
+            -0.004799448419362307,
+            -0.0009816535748541355,
+            0.043208710849285126,
+            0.016882391646504402,
+            0.013327243737876415,
+            -0.024466704577207565,
+            -0.03828619793057442,
+            -0.0034753840882331133,
+            -0.00029070733580738306,
+            0.03573378548026085,
+            0.007005462888628244,
+            0.007055599708110094,
+            -0.012233352288603783,
+            -0.011804911307990551,
+            0.0008768223342485726,
+            -0.016262518242001534,
+            0.0013081118231639266,
+            -0.011431165039539337,
+            -0.011494975537061691,
+            -0.004047398455440998,
+            0.004104371648281813,
+            0.010255232453346252,
+            0.013974462635815144,
+            0.012935265898704529,
+            0.006540559232234955,
+            -0.0032953477930277586,
+            -0.012689140625298023,
+            -0.0116225965321064,
+            -0.007488598581403494,
+            0.006271644029766321,
+            0.031066516414284706,
+            0.0032657217234373093,
+            -0.007023694459348917,
+            0.0047265226021409035,
+            0.032999057322740555,
+            -0.011504091322422028,
+            0.001646534539759159,
+            0.017474915832281113,
+            -0.02497718669474125,
+            0.016973547637462616,
+            -0.0008699854952283204,
+            0.015150396153330803,
+            0.006139465607702732,
+            0.015013659372925758,
+            -0.013518675230443478,
+            0.0009047393687069416,
+            0.011886953376233578,
+            0.008076565340161324,
+            -0.04886048287153244,
+            -0.00996352732181549,
+            0.009170456789433956,
+            -0.02173197641968727,
+            -0.01505923829972744,
+            -0.002342750784009695,
+            -0.0019188678124919534,
+            0.013345475308597088,
+            -0.017292600125074387,
+            0.014111199416220188,
+            0.03704645484685898,
+            -0.015086585655808449,
+            -0.02178667113184929,
+            0.008983583189547062,
+            0.02658156119287014,
+            0.013655411079525948,
+            -0.004981763660907745,
+            -0.008842289447784424,
+            0.014721955172717571,
+            0.009325424209237099,
+            -0.02116679958999157,
+            -0.013454864732921124,
+            0.040838614106178284,
+            -0.013044655323028564,
+            -0.016253402456641197,
+            0.023336350917816162,
+            -0.013309012167155743,
+            -0.02317226678133011,
+            0.017520494759082794,
+            0.014284398406744003,
+            -0.0061531392857432365,
+            0.020437538623809814,
+            0.0220783744007349,
+            0.003309021471068263,
+            -0.004156787414103746,
+            0.019945286214351654,
+            -0.006431169807910919,
+            -0.015350942499935627,
+            -0.007319956552237272,
+            -0.011303544975817204,
+            -0.012479477562010288,
+            0.014512293040752411,
+            0.008008196949958801,
+            -0.01928895153105259,
+            -0.005893339868634939,
+            -0.020638084039092064,
+            -0.004149950575083494,
+            0.04627160727977753,
+            -0.028222398832440376,
+            -0.0023336349986493587,
+            0.04051044583320618,
+            -0.003101638052612543,
+            -0.04470369592308998,
+            -0.01864173263311386,
+            0.007762071210891008,
+            -0.01752961054444313,
+            -0.009024604223668575,
+            -0.0422242097556591,
+            -0.005820414051413536,
+            0.020492233335971832,
+            0.015250669792294502,
+            -0.015697341412305832,
+            0.010082032531499863,
+            -0.020455770194530487,
+            0.06388325989246368,
+            0.0004905419191345572,
+            -0.009047393687069416,
+            -0.01092979870736599,
+            -0.02650863490998745,
+            0.011968995444476604,
+            0.02650863490998745,
+            0.014503177255392075,
+            0.012543288059532642,
+            0.022041911259293556,
+            0.02047400176525116,
+            0.012069269083440304,
+            0.025852300226688385,
+            0.02123972587287426,
+            0.030902432277798653,
+            0.024266157299280167,
+            -0.0003660548245534301,
+            0.01807655580341816,
+            -0.008509563282132149,
+            -0.04656331241130829,
+            -0.008573373779654503,
+            -0.003154053585603833,
+            0.00038371660048142076,
+            -0.011704638600349426,
+            -0.008855962194502354,
+            -0.00014143674343358725,
+            -0.020528696477413177,
+            -0.019416572526097298,
+            -0.041130319237709045,
+            -0.006490422412753105,
+            0.0028919754549860954,
+            0.03447581082582474,
+            0.011677291244268417,
+            -0.0007338188006542623,
+            -0.01835002936422825,
+            0.018577922135591507,
+            0.017228789627552032,
+            0.010145843029022217,
+            0.010692788287997246,
+            -0.00012954352132510394,
+            -0.014904270879924297,
+            0.050428394228219986,
+            0.021476734429597855,
+            0.0075159454718232155,
+            -0.027219664305448532,
+            -0.01492250245064497,
+            0.004343660548329353,
+            -0.005255236756056547,
+            -0.008477658964693546,
+            -0.0130537711083889,
+            -0.019270719960331917,
+            0.0033272530417889357,
+            0.014949849806725979,
+            -0.018304450437426567,
+            -0.015961699187755585,
+            0.003577936440706253,
+            0.003413852769881487,
+            0.024794872850179672,
+            -0.006148581393063068,
+            0.015907004475593567,
+            -0.012762066908180714,
+            -0.02025522291660309,
+            0.028422946110367775,
+            0.02061985246837139,
+            0.007857786491513252,
+            0.007392882835119963,
+            -0.006554232910275459,
+            0.006385591346770525,
+            -0.0011622596066445112,
+            0.012707372196018696,
+            -0.03810388594865799,
+            0.029370984062552452,
+            -0.004181855823844671,
+            5.131177022121847e-05,
+            -0.012278931215405464,
+            0.0220783744007349,
+            0.006517769768834114,
+            -0.0004099244251847267,
+            0.007739281747490168,
+            0.0025227870792150497,
+            -0.010419315658509731,
+            -0.01582496240735054,
+            0.002310845535248518,
+            0.036535974591970444,
+            -0.0031654483173042536,
+            0.016973547637462616,
+            -0.0038559671957045794,
+            0.02255239523947239,
+            0.007926154881715775,
+            -0.022807635366916656,
+            -0.023281656205654144,
+            -0.006025518756359816,
+            0.0015929794171825051,
+            0.03197809308767319,
+            -0.016134899109601974,
+            -0.03325429931282997,
+            0.022789403796195984,
+            0.01711028441786766,
+            -0.020145833492279053,
+            0.03578848019242287,
+            0.013454864732921124,
+            -0.0035095682833343744,
+            0.032999057322740555,
+            -0.0033454846125096083,
+            0.0018664522795006633,
+            0.009863254614174366,
+            0.014630797319114208,
+            0.0032360954210162163,
+            -0.012561519630253315,
+            -0.00414767162874341,
+            -0.009047393687069416,
+            -0.01595258340239525,
+            -0.006390149239450693,
+            0.004345939494669437,
+            -0.025943458080291748,
+            -0.0025729236658662558,
+            -0.0008483355632051826,
+            -0.001078508561477065,
+            0.015396521426737309,
+            0.01283499225974083,
+            -0.017119400203227997,
+            0.020729241892695427,
+            -0.007784860674291849,
+            -0.025906994938850403,
+            0.02393799088895321,
+            0.015405637212097645,
+            -0.00026165085728280246,
+            0.005268910434097052,
+            0.036955296993255615,
+            0.02395622245967388,
+            -0.009845023043453693,
+            -2.443629455228802e-06,
+            -0.001749086775816977,
+            0.009407466277480125,
+            -0.005979939829558134,
+            0.0009018906857818365,
+            0.00822241697460413,
+            0.0027005444280803204,
+            0.024375546723604202,
+            -0.008017312735319138,
+            -0.01575203612446785,
+            0.005838645622134209,
+            -0.017292600125074387,
+            0.01137647032737732,
+            -0.0018527786014601588,
+            -0.006567906588315964,
+            -0.01357336901128292,
+            0.008336364291608334,
+            0.014484945684671402,
+            0.047292571514844894,
+            0.014092967845499516,
+            0.0056153093464672565,
+            -0.017247021198272705,
+            0.01283499225974083,
+            0.02783953584730625,
+            0.001371922204270959,
+            0.012069269083440304,
+            -0.02811300940811634,
+            0.021968986839056015,
+            -0.002764354692772031,
+            0.003744299290701747,
+            0.0065998113714158535,
+            -0.0012978565646335483,
+            -0.0001713478413876146,
+            0.03440288454294205,
+            0.020145833492279053,
+            0.021950755268335342,
+            0.08576108515262604,
+            0.009644475765526295,
+            -0.016927970573306084,
+            -0.010774830356240273,
+            -0.0032429322600364685,
+            -0.028678186237812042,
+            -0.013454864732921124,
+            0.044521380215883255,
+            -0.004936185199767351,
+            0.004553323145955801,
+            -0.00706471549347043,
+            -0.0006426612380892038,
+            -0.013655411079525948,
+            0.009462160989642143,
+            0.010391968302428722,
+            0.0036508627235889435,
+            -0.0020054676569998264,
+            -0.0018641733331605792,
+            -0.023208729922771454,
+            -0.003976751118898392,
+            -0.025177733972668648,
+            -0.0072287991642951965,
+            -0.0035938890650868416,
+            0.011431165039539337,
+            0.0018687312258407474,
+            0.023482201620936394,
+            -1.8819062461261638e-05,
+            -0.007885133847594261,
+            0.00307201175019145,
+            0.017985397949814796,
+            -0.020382843911647797,
+            0.010018222033977509,
+            0.019179562106728554,
+            -0.0031358220148831606,
+            -0.0007053320878185332,
+            0.0021444829180836678,
+            -0.008500447496771812,
+            -0.009626244194805622,
+            -0.012306278571486473,
+            0.02415676973760128,
+            -0.006517769768834114,
+            -0.01810390315949917,
+            -0.010884219780564308,
+            0.00918413046747446,
+            0.000970258901361376,
+            0.01864173263311386,
+            0.00678668450564146,
+            -0.005460341461002827,
+            0.014886039309203625,
+            -0.01183225866407156,
+            -0.006927979178726673,
+            -0.00954420305788517,
+            0.010774830356240273,
+            -0.009726517833769321,
+            0.02242477424442768,
+            -0.013473096303641796,
+            0.009115762077271938,
+            -0.01881493255496025,
+            0.0065587908029556274,
+            0.00035124170244671404,
+            -0.024448473006486893,
+            0.011613480746746063,
+            -0.0036690940614789724,
+            -0.004936185199767351,
+            -0.008322690613567829,
+            0.005806740373373032,
+            0.015578837133944035,
+            -0.016982663422822952,
+            0.00034810815122909844,
+            -0.014138546772301197,
+            -0.02776661142706871,
+            -0.006134907715022564,
+            0.006955326069146395,
+            -0.008017312735319138,
+            -0.02464902028441429,
+            0.02054692804813385,
+            0.014594335108995438,
+            -0.02499541826546192,
+            -0.0005406786222010851,
+            -0.0014676376013085246,
+            -0.01723790541291237,
+            -0.025560596957802773,
+            0.0240109171718359,
+            -0.003445758018642664,
+            -0.007525061257183552,
+            0.020729241892695427,
+            -0.0003711824247147888,
+            -0.03551500663161278,
+            0.02929805777966976,
+            -0.004589786287397146,
+            0.00822241697460413,
+            0.022114837542176247,
+            0.003903825068846345,
+            -0.0007594568887725472,
+            0.006800358183681965,
+            0.02206014283001423,
+            0.008436637930572033,
+            -0.02756606414914131,
+            0.005164078902453184,
+            -0.014557871967554092,
+            -0.0027552389074116945,
+            -0.0035391945857554674,
+            -0.015907004475593567,
+            -0.03402002155780792,
+            0.0066590639762580395,
+            0.011531438678503036,
+            0.011358239687979221,
+            0.004120324272662401,
+            0.0054557835683226585,
+            0.018687311559915543,
+            0.0003073721018154174,
+            0.02082039974629879,
+            -0.0021365066058933735,
+            0.016581570729613304,
+            0.047365497797727585,
+            0.00021920558356214315,
+            -0.03394709527492523,
+            0.0007190057076513767,
+            0.00652688555419445,
+            -0.005200542043894529,
+            0.0028691859915852547,
+            0.03680944815278053,
+            0.01708293706178665,
+            -0.021968986839056015,
+            0.014010925777256489,
+            0.0013833168195560575,
+            -0.013208738528192043,
+            -0.019525961950421333,
+            0.0011759332846850157,
+            -0.02670918218791485,
+            0.007584313862025738,
+            -0.0025888762902468443,
+            -0.009229708462953568,
+            -0.020948020741343498,
+            -0.004721964709460735,
+            0.043062858283519745,
+            0.019307183101773262,
+            0.009001814760267735,
+            -0.001074520405381918,
+            -0.011048302985727787,
+            -0.03817681223154068,
+            0.010355505160987377,
+            0.029535068199038506,
+            -0.0015553769189864397,
+            0.006080213002860546,
+            -0.017000894993543625,
+            0.009589781053364277,
+            -0.0077347238548099995,
+            0.006335454527288675,
+            0.011467628180980682,
+            0.013409285806119442,
+            -0.0021102989558130503,
+            0.011823142878711224,
+            0.009762980975210667,
+            0.01006380096077919,
+            -0.0406927615404129,
+            0.007338188122957945,
+            0.0057429298758506775,
+            0.008568815886974335,
+            -0.00844575371593237,
+            0.016061972826719284,
+            -0.01970827765762806,
+            0.016326328739523888,
+            -0.03290789946913719,
+            -0.003418410662561655,
+            0.015733804553747177,
+            -0.01829533465206623,
+            -0.02443024143576622,
+            0.007702818606048822,
+            -0.0030287117697298527,
+            0.005984497722238302,
+            -0.006476748734712601,
+            0.00960801262408495,
+            -0.04557880759239197,
+            0.011075650341808796,
+            -0.02317226678133011,
+            -0.005578846205025911,
+            0.03617134317755699,
+            -0.004854143131524324,
+            -0.020291686058044434,
+            -0.00022817892022430897,
+            0.019489498808979988,
+            -0.020729241892695427,
+            -0.04350041598081589,
+            0.012725603766739368,
+            -0.014065620489418507,
+            0.013272549025714397,
+            -0.015843193978071213,
+            -0.007885133847594261,
+            0.015606184490025043,
+            0.013418401591479778,
+            0.009434813633561134,
+            -0.0028144915122538805,
+            -0.000781106820795685,
+            0.016034625470638275,
+            -0.03374655172228813,
+            0.006166812963783741,
+            -0.018723774701356888,
+            -0.03941655531525612,
+            -0.0012499988079071045,
+            0.023755675181746483,
+            -0.0042205979116261005,
+            0.02408384345471859,
+            0.01467637624591589,
+            -0.030063781887292862,
+            -0.02109387330710888,
+            -0.014484945684671402,
+            0.039671797305345535,
+            0.007971733808517456,
+            -0.023281656205654144,
+            -0.0027871441561728716,
+            -0.0007218543905764818,
+            0.014092967845499516,
+            -0.017985397949814796,
+            -0.0030879643745720387,
+            0.004181855823844671,
+            -0.004703733138740063,
+            -0.005975381936877966,
+            -0.03927070274949074,
+            0.012570635415613651,
+            0.019179562106728554,
+            -0.012935265898704529,
+            0.006271644029766321,
+            0.012844108045101166,
+            -0.01859615370631218,
+            0.012816760689020157,
+            -0.004300360567867756,
+            0.02796715684235096,
+            0.01680946536362171,
+            0.014594335108995438,
+            -0.009498624131083488,
+            -0.010993609204888344,
+            0.0011394702596589923,
+            -0.0007719910936430097,
+            -0.011285313405096531,
+            0.004767543636262417,
+            -0.003935730084776878,
+            0.020018212497234344,
+            0.017839545384049416,
+            0.029954394325613976,
+            -0.019179562106728554,
+            0.012798530049622059,
+            -0.03664536401629448,
+            0.00568823516368866,
+            0.0010044430382549763,
+            -0.00441430788487196,
+            -0.007739281747490168,
+            0.020236991345882416,
+            -0.01965358294546604,
+            0.0046672699972987175,
+            -0.011577017605304718,
+            -0.026599792763590813,
+            0.0009919088333845139,
+            0.007438461761921644,
+            -0.015961699187755585,
+            0.0018926600459963083,
+            -0.013546021655201912,
+            -0.01985413022339344,
+            -0.007497713901102543,
+            0.026563329622149467,
+            -0.008413848467171192,
+            -0.0005198832950554788,
+            -0.0005409634904935956,
+            -0.01391065213829279,
+            0.007333630230277777,
+            -0.003530078800395131,
+            -0.006567906588315964,
+            0.0017080658581107855,
+            -0.02428438887000084,
+            -0.007443019654601812,
+            0.005041016265749931,
+            0.02339104562997818,
+            0.015551489777863026,
+            -0.018441185355186462,
+            -0.024029148742556572,
+            -0.002853233367204666,
+            0.026526866480708122,
+            0.014247936196625233,
+            -0.026891497895121574,
+            0.01846853271126747,
+            -0.003641746938228607,
+            0.020638084039092064,
+            0.006216949783265591,
+            0.006618042942136526,
+            -0.0032657217234373093,
+            -0.008199627511203289,
+            0.007119410205632448,
+            -0.009525971487164497,
+            -0.02138557657599449,
+            -0.009735633619129658,
+            0.015496795065701008,
+            -0.010647210292518139,
+            0.003821783233433962,
+            0.000417046103393659,
+            -0.015916120260953903,
+            0.006289875600486994,
+            -0.009790328331291676,
+            0.015432984568178654,
+            0.011996342800557613,
+            0.00036747913691215217,
+            0.00036235153675079346,
+            0.030483108013868332,
+            -0.012297162786126137,
+            -0.007917039096355438,
+            0.00522333150729537,
+            -0.009047393687069416,
+            0.004010935313999653,
+            -0.007834997028112411,
+            -0.03241565078496933,
+            -0.0031859588343650103,
+            0.014311745762825012,
+            0.018605269491672516,
+            -0.01044666301459074,
+            -0.007958060130476952,
+            0.028058314695954323,
+            0.006285317707806826,
+            -0.007306283339858055,
+            -0.005018226802349091,
+            0.04718318209052086,
+            -0.0003959659079555422,
+            0.020984483882784843,
+            0.009535087272524834,
+            0.003142658853903413,
+            -0.004181855823844671,
+            -0.0003748857125174254,
+            0.008846846409142017,
+            -0.003760251682251692,
+            0.0021718302741646767,
+            -0.012415667995810509,
+            0.013254317454993725,
+            -0.0037078361492604017,
+            -0.028404714539647102,
+            0.008327248506247997,
+            -0.014603450894355774,
+            0.0001921431685332209,
+            -0.018377376720309258,
+            0.0027005444280803204,
+            0.012798530049622059,
+            -0.012561519630253315,
+            -0.0017525051953271031,
+            -0.007101178634911776,
+            -0.00947127677500248,
+            -0.0021923407912254333,
+            -0.0010414757998660207,
+            -0.02027345448732376,
+            -0.010893335565924644,
+            0.0008016172796487808,
+            -0.012762066908180714,
+            0.011886953376233578,
+            0.0002697695745155215,
+            -0.0006933676195330918,
+            0.00975386518985033,
+            -0.0009805141016840935,
+            0.019544193521142006,
+            0.00699178921058774,
+            0.0047265226021409035,
+            -0.006089328788220882,
+            0.004712848924100399,
+            0.020091138780117035,
+            -0.035770248621702194,
+            0.004095255862921476,
+            0.0005959429545328021,
+            0.009120319969952106,
+            -0.006604369264096022,
+            -0.016216941177845,
+            -0.002313124481588602,
+            -0.019598888233304024,
+            0.009899717755615711,
+            -0.01595258340239525,
+            -0.012680024839937687,
+            -0.000525010924320668,
+            0.0003065174969378859,
+            -0.004571554716676474,
+            0.006157697178423405,
+            0.011850490234792233,
+            0.004327707923948765,
+            0.0032133059576153755,
+            0.013937999494373798,
+            -0.020018212497234344,
+            0.004459886346012354,
+            -0.007944386452436447,
+            -0.0019747018814086914,
+            0.015195975080132484,
+            -0.0020134439691901207,
+            0.0006033494719304144,
+            -0.014895155094563961,
+            -0.002074975287541747,
+            0.005765719339251518,
+            -0.0012830435298383236,
+            -0.0036987203639000654,
+            0.013381938450038433,
+            -0.004958974663168192,
+            0.00013353166286833584,
+            -0.008587047457695007,
+            -0.00039767511771060526,
+            -0.004949858877807856,
+            0.006467632949352264,
+            -0.016982663422822952,
+            0.009735633619129658,
+            0.01721055805683136,
+            0.0061440235003829,
+            -0.002125111874192953,
+            -0.003600725904107094,
+            -0.00301959621720016,
+            0.03309021517634392,
+            0.006704642903059721,
+            0.02060162089765072,
+            0.029334520921111107,
+            0.008896983228623867,
+            -0.0037853200919926167,
+            0.0013092512963339686,
+            -0.008331806398928165,
+            -0.007456693332642317,
+            -0.0230446457862854,
+            -0.013299896381795406,
+            0.01464902888983488,
+            -0.006084770895540714,
+            0.012470361776649952,
+            -5.740081178373657e-05,
+            0.007383767049759626,
+            -0.00607109721750021,
+            -0.01896078512072563,
+            -0.030902432277798653,
+            -0.002962622558698058,
+            0.022114837542176247,
+            -0.005405646748840809,
+            -0.026964424178004265,
+            0.0004817110311705619,
+            0.010474010370671749,
+            -0.018094787374138832,
+            -0.015669994056224823,
+            0.010273464024066925,
+            -0.007429345976561308,
+            0.01886962726712227,
+            -0.008641742169857025,
+            0.017739271745085716,
+            -0.0009041696321219206,
+            -0.008641742169857025,
+            0.022114837542176247,
+            -0.024339083582162857,
+            0.013299896381795406,
+            -0.0004179007082711905,
+            -0.0031677272636443377,
+            -0.0003310161118861288,
+            0.010528705082833767,
+            -0.010765714570879936,
+            -0.0184503011405468,
+            0.005478573031723499,
+            -0.01346398051828146,
+            0.009352771565318108,
+            -0.008896983228623867,
+            0.011148576624691486,
+            0.02470371499657631,
+            -0.014475829899311066,
+            0.004977205768227577,
+            0.007206009700894356,
+            0.0012534172274172306,
+            0.0014949849573895335,
+            -0.005793066695332527,
+            0.013199622742831707,
+            0.0069781155325472355,
+            -0.010647210292518139,
+            0.02692796103656292,
+            0.010993609204888344,
+            -0.012752951122820377,
+            -0.0035847732797265053,
+            -0.007301725447177887,
+            -0.018796700984239578,
+            -0.000998745672404766,
+            -0.018997248262166977,
+            -0.011048302985727787,
+            -0.008587047457695007,
+            -0.011112113483250141,
+            0.0019507730612531304,
+            -0.005528709385544062,
+            -0.01259798277169466,
+            -0.0038377356249839067,
+            0.011804911307990551,
+            -0.006704642903059721,
+            -0.03318137302994728,
+            0.01812213473021984,
+            0.019762972369790077,
+            0.008140374906361103,
+            -0.03792157024145126,
+            0.019872359931468964,
+            0.00757975596934557,
+            -0.002256151055917144,
+            -0.012853223830461502,
+            0.0012841830030083656,
+            -0.01896078512072563,
+            0.013263433240354061,
+            -0.019872359931468964,
+            -0.001451685093343258,
+            0.0016818580916151404,
+            0.007958060130476952,
+            0.005100268870592117,
+            0.022880561649799347,
+            -0.005195984151214361,
+            -0.004268455319106579,
+            0.005478573031723499,
+            0.0038103885017335415,
+            -0.0025524133816361427,
+            0.030829505994915962,
+            -0.01051958929747343,
+            -0.023154035210609436,
+            -0.002796259941533208,
+            0.014521408826112747,
+            -0.009735633619129658,
+            -0.015387405641376972,
+            0.020437538623809814,
+            -0.01797628216445446,
+            0.0022401984315365553,
+            0.014731070958077908,
+            0.004284407943487167,
+            0.030720116570591927,
+            0.01290791854262352,
+            -0.005346394143998623,
+            -0.000578565988689661,
+            0.00010262354044243693,
+            0.0010055825114250183,
+            0.011613480746746063,
+            -0.004895164165645838,
+            -0.004762985743582249,
+            -0.005815856158733368,
+            0.012725603766739368,
+            -9.649888670537621e-05,
+            -0.0031996325124055147,
+            -0.00015781662659719586,
+            -0.015414752997457981,
+            0.014876923523843288,
+            0.0026731970719993114,
+            0.0036280732601881027,
+            0.007420230191200972,
+            -6.366789602907375e-05,
+            -0.0059617082588374615,
+            -0.01384684257209301,
+            0.006567906588315964,
+            -0.011340008117258549,
+            -0.0115223228931427,
+            -0.028696417808532715,
+            0.014949849806725979,
+            -0.02040107548236847,
+            0.00958066526800394,
+            -0.002073835814371705,
+            0.009863254614174366,
+            -0.02054692804813385,
+            -0.015788499265909195,
+            -0.009325424209237099,
+            0.012607098557054996,
+            -0.009662707336246967,
+            0.02144027128815651,
+            0.007174104452133179,
+            0.015770267695188522,
+            0.021257957443594933,
+            0.024247925728559494,
+            0.004867816809564829,
+            0.01629898138344288,
+            -0.02818593569099903,
+            -0.0007919318159110844,
+            -0.007748397532850504,
+            0.012014574371278286,
+            -0.0036166785284876823,
+            0.006394707132130861,
+            0.013108465820550919,
+            0.01723790541291237,
+            -0.010938914492726326,
+            0.002903370186686516,
+            -2.533540828153491e-05,
+            -0.00958066526800394,
+            -0.00711485231295228,
+            -0.017784850671887398,
+            -0.01859615370631218,
+            0.01631721295416355,
+            0.007511387579143047,
+            0.012780298478901386,
+            0.016052857041358948,
+            0.016289865598082542,
+            -0.0023199613206088543,
+            0.0021091594826430082,
+            -0.01943480409681797,
+            -0.0025934341829270124,
+            -0.012917034327983856,
+            -0.0013593879994004965,
+            0.025560596957802773,
+            -0.0017069263849407434,
+            0.017939819023013115,
+            0.007433903869241476,
+            0.034731052815914154,
+            0.0010015943553298712,
+            0.006927979178726673,
+            0.005278026219457388,
+            0.015633530914783478,
+            0.0027119391597807407,
+            0.02386506460607052,
+            -0.02040107548236847,
+            0.0013001355109736323,
+            0.03691883385181427,
+            0.022862330079078674,
+            -0.008195069618523121,
+            -0.01790335588157177,
+            -0.01943480409681797,
+            0.016681844368577003,
+            -0.03019140288233757,
+            -0.00757975596934557,
+            0.02386506460607052,
+            0.00624885456636548,
+            0.01319050695747137,
+            -0.0006648809066973627,
+            -0.018650848418474197,
+            0.012889686971902847,
+            0.008195069618523121,
+            -0.009498624131083488,
+            0.004423423204571009,
+            -0.006381033454090357,
+            -0.009489508345723152,
+            -0.0027825862634927034,
+            -0.0027871441561728716,
+            -0.017474915832281113,
+            -0.00865541584789753,
+            0.0004555032355710864,
+            -0.004541928414255381,
+            0.0021102989558130503,
+            -0.003470826428383589,
+            -0.0217137448489666,
+            0.0058340877294540405,
+            -0.012087500654160976,
+            -0.0025683660060167313,
+            0.0018128971569240093,
+            -0.009498624131083488,
+            -0.024667251855134964,
+            -0.00650409609079361,
+            -0.007142199203372002,
+            -0.005268910434097052,
+            0.01277118269354105,
+            -0.015460331924259663,
+            0.009257055819034576,
+            -0.014876923523843288,
+            0.023354582488536835,
+            -0.0005976521642878652,
+            0.008609836921095848,
+            0.013965346850454807,
+            -0.013454864732921124,
+            -0.011494975537061691,
+            -0.03442111611366272,
+            -0.0044781179167330265,
+            -0.020455770194530487,
+            -0.00565633038058877,
+            -0.00660892715677619,
+            0.01689150743186474,
+            0.0018197339959442616,
+            -0.017301715910434723,
+            0.01696443371474743,
+            0.004854143131524324,
+            -0.0013138091890141368,
+            -0.01561530027538538,
+            -0.0044484916143119335,
+            -9.236830373993143e-05,
+            -0.007452135439962149,
+            0.01956242509186268,
+            0.00161348981782794,
+            -0.002983133075758815,
+            0.0025136712938547134,
+            0.007803092245012522,
+            -0.007092062849551439,
+            -0.03389240428805351,
+            -0.012579751200973988,
+            -0.002392887370660901,
+            6.17094337940216e-05,
+            -0.007493156008422375,
+            0.003685046685859561,
+            0.014339093118906021,
+            -0.011185039766132832,
+            -0.007921596989035606,
+            -0.00013324679457582533,
+            0.009808559902012348,
+            0.015451216138899326,
+            -0.009334539994597435,
+            0.018094787374138832,
+            -0.01567910984158516,
+            0.01874200627207756,
+            0.0007093202439136803,
+            -0.002285777358338237,
+            -0.0022162694949656725,
+            0.010282579809427261,
+            -0.008473101072013378,
+            -0.006918863393366337,
+            -0.009211476892232895,
+            -0.008814942091703415,
+            -0.0073290723375976086,
+            0.0013046934036538005,
+            -0.020291686058044434,
+            -0.020528696477413177,
+            -0.006230622995644808,
+            0.0030013646464794874,
+            -0.006125791929662228,
+            0.011567901819944382,
+            -0.01211484707891941,
+            -0.014348208904266357,
+            0.014147662557661533,
+            0.007392882835119963,
+            0.0029216017574071884,
+            -0.017092052847146988,
+            -0.007274378091096878,
+            -0.007661798037588596,
+            0.008299901150166988,
+            -0.0031700062099844217,
+            0.034803979098796844,
+            -0.003343205666169524,
+            -0.010455778799951077,
+            0.003015038324519992,
+            -0.001371922204270959,
+            0.0058112982660532,
+            -0.016499528661370277,
+            0.005701908841729164,
+            0.0011172505328431726,
+            0.006814031861722469,
+            0.020528696477413177,
+            -0.0016294424422085285,
+            -0.012871455401182175,
+            0.008796710520982742,
+            0.010191421955823898,
+            0.002445303136482835,
+            -0.0016009557293727994,
+            -0.0016898344038054347,
+            -0.00801275484263897,
+            0.019945286214351654,
+            0.041604336351156235,
+            0.0035164051223546267,
+            0.01037373673170805,
+            0.009389234706759453,
+            -0.01641748659312725,
+            -0.009489508345723152,
+            0.014685492031276226,
+            -0.008436637930572033,
+            0.005683677736669779,
+            -0.006253412459045649,
+            0.019981749355793,
+            -0.011458512395620346,
+            -0.005232447292655706,
+            0.009981758892536163,
+            -0.003575657494366169,
+            0.014904270879924297,
+            0.006107560358941555,
+            -0.019817667081952095,
+            0.008600721135735512,
+            0.0014482666738331318,
+            -0.0035072893369942904,
+            -0.026818571612238884,
+            0.0024703715462237597,
+            0.002686870750039816,
+            0.009945295751094818,
+            -0.0232451930642128,
+            0.01252505648881197,
+            0.010145843029022217,
+            0.004908837843686342,
+            0.004651317372918129,
+            0.0037511358968913555,
+            -0.009329982101917267,
+            0.005369183607399464,
+            0.01923425681889057,
+            -0.008154048584401608,
+            -0.025852300226688385,
+            -0.0004278710694052279,
+            0.001534866401925683,
+            -0.010811293497681618,
+            -0.0058112982660532,
+            0.010528705082833767,
+            -0.009480392560362816,
+            -0.005337278358638287,
+            0.015569721348583698,
+            0.014138546772301197,
+            -0.012570635415613651,
+            0.02816770412027836,
+            -0.006203276105225086,
+            0.0061804866418242455,
+            0.025724679231643677,
+            -0.007365535479038954,
+            -0.012178657576441765,
+            -0.025141270831227303,
+            0.013245201669633389,
+            0.0010956006590276957,
+            0.019398340955376625,
+            -0.02747490629553795,
+            -0.021640818566083908,
+            -0.018359145149588585,
+            0.008359153755009174,
+            0.008208743296563625,
+            0.0027301707305014133,
+            0.02277117222547531,
+            -0.00683682132512331,
+            -0.0010357784340158105,
+            -0.0013331801164895296,
+            0.019270719960331917,
+            0.00732451444491744,
+            -0.0015758873196318746,
+            0.011686407029628754,
+            0.012789414264261723,
+            0.01567910984158516,
+            -0.011558786034584045,
+            -0.014594335108995438,
+            -0.014968081377446651,
+            0.00016992349992506206,
+            -0.01044666301459074,
+            -0.004863258916884661,
+            0.006513211876153946,
+            0.026271626353263855,
+            -0.021203262731432915,
+            0.010018222033977509,
+            0.006376475561410189,
+            0.002236780012026429,
+            -0.0016647659940645099,
+            -0.004175018984824419,
+            0.018113018944859505,
+            -0.017721040174365044,
+            0.00532816257327795,
+            -0.0010152680333703756,
+            0.0027073812671005726,
+            0.012470361776649952,
+            -0.023135803639888763,
+            -0.005565172526985407,
+            0.015341827645897865,
+            -0.004122603219002485,
+            -0.00947127677500248,
+            0.026052847504615784,
+            -0.03700999170541763,
+            0.021276189014315605,
+            0.015423868782818317,
+            -0.0024065610487014055,
+            -0.011896069161593914,
+            -0.01881493255496025,
+            0.0034024580381810665,
+            -0.027019118890166283,
+            -0.0018687312258407474,
+            0.005405646748840809,
+            -0.01267090905457735,
+            -0.010173190385103226,
+            0.009589781053364277,
+            0.011786679737269878,
+            -0.005515035707503557,
+            -0.009936179965734482,
+            -0.008003639057278633,
+            -0.009507739916443825,
+            -0.011996342800557613,
+            0.0057429298758506775,
+            -0.00898814108222723,
+            -0.003976751118898392,
+            0.0007685726741328835,
+            0.02678210847079754,
+            -4.931769581162371e-06,
+            -0.012661793269217014,
+            0.013892421498894691,
+            0.02541474439203739,
+            -0.009726517833769321,
+            -0.02235184796154499,
+            -0.008983583189547062,
+            -0.0009565852815285325,
+            -0.0037055572029203176,
+            -0.008263438008725643,
+            0.0009070183150470257,
+            -0.011804911307990551,
+            0.0040701874531805515,
+            0.003933451138436794,
+            -0.0009845022577792406,
+            -0.022151300683617592,
+            0.008331806398928165,
+            0.007999081164598465,
+            0.011440280824899673,
+            -0.0414949469268322,
+            -0.002910207025706768,
+            -0.012880571186542511,
+            -0.0042205979116261005,
+            -0.01429351419210434,
+            0.025232428684830666,
+            0.005847761407494545,
+            0.0065770223736763,
+            -0.02470371499657631,
+            0.008103912696242332,
+            0.022242458537220955,
+            -0.0014699165476486087,
+            0.0018698706990107894,
+            -0.01159524917602539,
+            -0.00257520261220634,
+            0.028751112520694733,
+            -0.007803092245012522,
+            0.00506836362183094,
+            -0.003101638052612543,
+            0.00027831559418700635,
+            0.004441654775291681,
+            0.013372822664678097,
+            0.0007588871521875262,
+            -0.00982679147273302,
+            0.014494061470031738,
+            0.00021607204689644277,
+            0.005756603553891182,
+            -0.0031107538379728794,
+            -0.01329078059643507,
+            0.0025432975962758064,
+            -0.009899717755615711,
+            -0.03144937753677368,
+            -0.01579761505126953,
+            -0.020510464906692505,
+            -0.013427517376840115,
+            -0.002880580723285675,
+            0.0334366150200367,
+            0.014776649884879589,
+            -0.014010925777256489,
+            -0.0012955776182934642,
+            -0.0034047369845211506,
+            0.027784842997789383,
+            0.011485859751701355,
+            -0.008030986413359642,
+            0.02020052820444107,
+            -0.0017593420343473554,
+            -0.00285551231354475,
+            0.018277103081345558,
+            -0.004343660548329353,
+            0.010802177712321281,
+            0.0024726504925638437,
+            -0.02421146258711815,
+            0.012743835337460041,
+            -0.002525066025555134,
+            0.001482450752519071,
+            0.010957146063446999,
+            -0.00960801262408495,
+            -0.01290791854262352,
+            0.0015417032409459352,
+            0.008523236960172653,
+            -0.012196889147162437,
+            -0.024375546723604202,
+            -0.0033477635588496923,
+            -0.008550584316253662,
+            -0.007948944345116615,
+            0.0016191871836781502,
+            0.014703723601996899,
+            -0.0004247375181876123,
+            0.006522327661514282,
+            -0.004936185199767351,
+            -0.014165894128382206,
+            -0.004694617353379726,
+            0.02144027128815651,
+            0.01866907998919487,
+            0.00954420305788517,
+            -4.436812378116883e-05,
+            0.016882391646504402,
+            -0.010255232453346252,
+            -0.005875108297914267,
+            -0.016927970573306084,
+            -0.02484956756234169,
+            0.0009457602864131331,
+            0.0106563251465559,
+            -0.024120306596159935,
+            0.0034844998735934496,
+            -0.048678167164325714,
+            -0.008700994774699211,
+            0.017174094915390015,
+            -0.01908840611577034,
+            0.005059247836470604,
+            0.00242023472674191,
+            -0.03041018173098564,
+            0.015268901363015175,
+            -0.019762972369790077,
+            0.01061986293643713,
+            -0.009311750531196594,
+            -0.02588876336812973,
+            0.016818581148982048,
+            -0.006677295546978712,
+            0.016335444524884224,
+            -0.009079298935830593,
+            -0.008304459042847157,
+            -0.0056426567025482655,
+            -0.006221507675945759,
+            0.00036690940032713115,
+            0.0006888097268529236,
+            0.001456242986023426,
+            0.0036736519541591406,
+            0.004899722058326006,
+            -0.01750226318836212,
+            -0.009945295751094818,
+            -0.01551502663642168,
+            -0.014895155094563961,
+            -0.01810390315949917,
+            0.009083856828510761,
+            -0.01520509086549282,
+            0.010027337819337845,
+            -0.0034252475015819073,
+            -0.017438452690839767,
+            0.007092062849551439,
+            -0.006809473969042301,
+            0.005232447292655706,
+            -0.005970824044197798,
+            -0.006595253478735685,
+            0.009899717755615711,
+            0.006905189715325832,
+            0.007871460169553757,
+            0.00706471549347043,
+            -0.012844108045101166,
+            -0.004193250555545092,
+            -0.003876477712765336,
+            0.03318137302994728,
+            0.0005891061155125499,
+            -0.01023700088262558,
+            -0.005788508802652359,
+            -1.0931792530755047e-05,
+            7.641572301508859e-05,
+            0.005086595192551613,
+            0.0007127386052161455,
+            0.016499528661370277,
+            -0.019197793677449226,
+            -0.0051367320120334625,
+            -0.0060209608636796474,
+            -0.005697350949048996,
+            -0.012534172274172306,
+            -0.016918854787945747,
+            0.002527344971895218,
+            0.01916133053600788,
+            0.009261613711714745,
+            0.01259798277169466,
+            -0.03110297955572605,
+            0.0029671804513782263,
+            0.021494966000318527,
+            -0.025706447660923004,
+            -0.010474010370671749,
+            -0.00793527066707611,
+            -0.028277093544602394,
+            -0.015551489777863026,
+            0.013855958357453346,
+            0.014311745762825012,
+            0.023208729922771454,
+            -0.017319947481155396,
+            -0.0007452135323546827,
+            -0.03274381533265114,
+            -0.0012978565646335483,
+            -0.0030811275355517864,
+            -0.008381943218410015,
+            -0.024375546723604202,
+            0.025834068655967712,
+            0.015979930758476257,
+            0.008559700101613998,
+            0.004355055280029774,
+            -0.000295692530926317,
+            0.024831335991621017,
+            -0.004131719004362822,
+            0.007629892788827419,
+            -0.001804920844733715,
+            0.017347294837236404,
+            -0.009553318843245506,
+            -0.012889686971902847,
+            0.005109384655952454,
+            -0.01068367250263691,
+            -7.24987912690267e-05,
+            -0.00688695814460516,
+            -0.027456674724817276,
+            -0.029443910345435143,
+            0.0053965309634804726,
+            0.0013400169555097818,
+            0.009872370399534702,
+            -0.022935256361961365,
+            0.018240639939904213,
+            -0.0066590639762580395,
+            -0.004744754172861576,
+            -0.0006774150533601642,
+            -0.0023906086571514606,
+            0.02720143273472786,
+            -0.02220599539577961,
+            -0.0026230604853481054,
+            -0.0015439821872860193,
+            -0.010027337819337845,
+            0.006595253478735685,
+            0.0008027567528188229,
+            0.013372822664678097,
+            0.016818581148982048,
+            -0.015004543587565422,
+            -0.04123970493674278,
+            -0.002021420281380415,
+            -0.035095684230327606,
+            -0.0077347238548099995,
+            -0.002716497052460909,
+            0.021768439561128616,
+            -0.006435727700591087,
+            0.037082917988300323,
+            -0.0014414298348128796,
+            -0.02464902028441429,
+            -0.0270738136023283,
+            0.01599816232919693,
+            0.023135803639888763,
+            -0.014457598328590393,
+            -0.0026549657341092825,
+            0.019525961950421333,
+            -0.004120324272662401,
+            0.01121238712221384,
+            0.012160426005721092,
+            0.015323596075177193,
+            -0.004138555843383074,
+            0.0018687312258407474,
+            0.008267995901405811,
+            -0.019690046086907387,
+            -0.028751112520694733,
+            -0.0036189574748277664,
+            0.00030025039450265467,
+            -0.015323596075177193,
+            0.0074475775472819805,
+            -0.01106653455644846,
+            -0.007060157600790262,
+            0.006016402970999479,
+            0.007876018062233925,
+            -0.007944386452436447,
+            0.025925226509571075,
+            -0.014193241484463215,
+            -0.002069277921691537,
+            0.023901527747511864,
+            0.0006523467018269002,
+            -0.016873275861144066,
+            -0.0047265226021409035,
+            0.0032452112063765526,
+            0.027948925271630287,
+            0.01679123379290104,
+            -0.007201451808214188,
+            -0.017848661169409752,
+            0.02443024143576622,
+            -0.011230618692934513,
+            -0.007629892788827419,
+            0.0001251992944162339,
+            0.01200545858591795,
+            0.01842295378446579,
+            -0.004234271589666605,
+            0.004813122097402811,
+            -0.0009480392327532172,
+            0.033491309732198715,
+            0.002347308676689863,
+            0.009453045204281807,
+            -0.01644483394920826,
+            0.009307192638516426,
+            -0.0023974452633410692,
+            -0.00021179902250878513,
+            -0.0021410644985735416,
+            -0.034731052815914154,
+            -0.01530536450445652,
+            -0.0042205979116261005,
+            0.003985866904258728,
+            -0.021914292126893997,
+            0.0041294400580227375,
+            0.001722879009321332,
+            -0.02665448747575283,
+            0.016216941177845,
+            -0.016918854787945747,
+            -0.00944392941892147,
+            0.017821313813328743,
+            -0.0070692733861505985,
+            -0.0073883249424397945,
+            -0.012206004932522774,
+            0.022716479375958443,
+            0.00975386518985033,
+            -0.008846846409142017,
+            0.009890601970255375,
+            -0.01482222881168127,
+            0.008158606477081776,
+            -0.0015451216604560614,
+            0.007834997028112411,
+            0.021185031160712242,
+            -0.017638999968767166,
+            0.017438452690839767,
+            0.00011950194311793894,
+            -0.009744749404489994,
+            -0.008523236960172653,
+            0.00436417106539011,
+            0.027037350460886955,
+            0.00035893311724066734,
+            0.017693692818284035,
+            0.007429345976561308,
+            0.03301728889346123,
+            -0.0034867788199335337,
+            0.004471281077712774,
+            0.012133078649640083,
+            0.016089320182800293,
+            -0.017174094915390015,
+            -0.013026423752307892,
+            0.0020795331802219152,
+            -0.009790328331291676,
+            0.0013650853652507067,
+            -0.009762980975210667,
+            -0.02165905013680458,
+            0.02623516321182251,
+            0.021531429141759872,
+            -0.017019126564264297,
+            0.0008016172796487808,
+            -0.008810384199023247,
+            -0.0351686105132103,
+            -0.018404724076390266,
+            0.0108568724244833,
+            0.015241554006934166,
+            0.0010813572444021702,
+            0.010309926234185696,
+            -0.01016407459974289,
+            -0.00021920558356214315,
+            -0.0036189574748277664,
+            -0.01723790541291237,
+            -0.01793070323765278,
+            -0.015405637212097645,
+            -0.009671823121607304,
+            -0.009029162116348743,
+            0.0039699142798781395,
+            -0.011303544975817204,
+            -0.006536001339554787,
+            0.004514581058174372,
+            0.004731080494821072,
+            -0.009735633619129658,
+            -0.011850490234792233,
+            0.013126697391271591,
+            -0.001399269443936646,
+            0.00046974659198895097,
+            0.016581570729613304,
+            -0.0035961680114269257,
+            -0.0213126502931118,
+            0.00387875665910542,
+            0.006025518756359816,
+            0.012515940703451633,
+            0.00429124478250742,
+            -0.010483126156032085,
+            0.01599816232919693,
+            0.00568823516368866,
+            -0.012333625927567482,
+            0.017146747559309006,
+            -0.022023679688572884,
+            -0.007538734935224056,
+            0.03296259418129921,
+            -0.013637179508805275,
+            0.009316308423876762,
+            0.0003042385505978018,
+            -0.006431169807910919,
+            0.02206014283001423,
+            0.026545098051428795,
+            0.01627163402736187,
+            0.00515040522441268,
+            0.012798530049622059,
+            0.00031363917514681816,
+            -0.0011656780261546373,
+            0.015843193978071213,
+            -0.0030765696428716183,
+            -0.026198700070381165,
+            -0.0025501344352960587,
+            0.0009018906857818365,
+            0.002495439723134041,
+            0.0029808541294187307,
+            -0.010628978721797466,
+            -0.005715582519769669,
+            -0.0010101404041051865,
+            -0.006226065568625927,
+            -0.0034092948772013187,
+            0.001910891616716981,
+            -0.011850490234792233,
+            0.0011998621048405766,
+            -0.02650863490998745,
+            0.00031335430685430765,
+            -0.012570635415613651,
+            -0.027912462130188942,
+            0.021075641736388206,
+            -0.005984497722238302,
+            0.011148576624691486,
+            0.022917024791240692,
+            0.019252488389611244,
+            -0.008792152628302574,
+            -0.004981763660907745,
+            0.0012978565646335483,
+            0.008673647418618202,
+            -0.014202357269823551,
+            -0.014594335108995438,
+            -0.009808559902012348,
+            -0.0019439362222328782,
+            -0.020218759775161743,
+            -0.021257957443594933,
+            -0.018231524154543877,
+            0.0010454639559611678,
+            0.01513216458261013,
+            0.0006540559115819633,
+            0.00459206523373723,
+            -0.016399255022406578,
+            0.00898814108222723,
+            -0.011030071415007114,
+            -0.011987227015197277,
+            0.003685046685859561,
+            0.004494070541113615,
+            -0.010793061926960945,
+            0.004712848924100399,
+            -0.005715582519769669,
+            -0.00301959621720016,
+            0.002584318397566676,
+            -0.014211473055183887,
+            -0.002613944699987769,
+            0.0006301270332187414,
+            0.025451207533478737,
+            0.0005329871783033013,
+            0.009772096760571003,
+            0.004045119509100914,
+            0.009188688360154629,
+            0.008267995901405811,
+            0.0003355739754624665,
+            0.0071695465594530106,
+            0.0014049668097868562,
+            -0.005401088856160641,
+            0.017620768398046494,
+            0.0011964436853304505,
+            -0.006586138159036636,
+            -0.007926154881715775,
+            -0.021403808146715164,
+            -0.003926614299416542,
+            0.0011098439572378993,
+            0.02180490270256996,
+            0.002579760504886508,
+            0.004731080494821072,
+            -0.013318127952516079,
+            -0.004443933721631765,
+            -0.004279850050806999,
+            -0.0008688460220582783,
+            0.0037830411456525326,
+            0.025651752948760986,
+            -0.004168182145804167,
+            -0.008819499984383583,
+            -0.01894255355000496,
+            -0.017711924389004707,
+            0.020145833492279053,
+            -0.006007287185639143,
+            0.006394707132130861,
+            -0.017420221120119095,
+            0.007789418566972017,
+            0.0064995381981134415,
+            0.018340913578867912,
+            -0.009717402048408985,
+            0.011704638600349426,
+            0.005246120970696211,
+            -0.0020077466033399105,
+            0.011959879659116268,
+            -0.0064995381981134415,
+            0.034038253128528595,
+            -0.014876923523843288,
+            0.009708286263048649,
+            0.0022048749960958958,
+            0.004213761072605848,
+            0.007844112813472748,
+            0.019070174545049667,
+            -0.004863258916884661,
+            -0.012762066908180714,
+            -0.005195984151214361,
+            -0.004863258916884661,
+            0.016800349578261375,
+            0.021841365844011307,
+            0.0046353647485375404,
+            -0.0008283948409371078,
+            0.0027415654622018337,
+            0.006171370856463909,
+            0.009726517833769321,
+            0.019070174545049667,
+            -0.01669096015393734,
+            0.005350952036678791,
+            -0.016353676095604897,
+            -0.009662707336246967,
+            0.00836826954036951,
+            -0.005843203514814377,
+            0.014147662557661533,
+            0.012616214342415333,
+            -0.000646079599391669,
+            -0.0004167612351011485,
+            0.015834078192710876,
+            -0.0021091594826430082,
+            0.006157697178423405,
+            0.012178657576441765,
+            -0.01100272499024868,
+            -0.001858475967310369,
+            -0.02680034004151821,
+            0.008053775876760483,
+            -0.023500433191657066,
+            -0.019471267238259315,
+            0.003286232240498066,
+            -0.025177733972668648,
+            0.005191426258534193,
+            0.018632616847753525,
+            0.01758430525660515,
+            -0.009143109433352947,
+            -0.010191421955823898,
+            -0.0009360748226754367,
+            0.02027345448732376,
+            -0.013555137440562248,
+            0.02032814919948578,
+            -0.002256151055917144,
+            0.004931627307087183,
+            0.007648124359548092,
+            -0.004412028938531876,
+            0.016216941177845,
+            -0.012734719552099705,
+            0.0024430241901427507,
+            -0.012588866986334324,
+            -0.0032702796161174774,
+            0.022151300683617592,
+            -0.02102094702422619,
+            0.027237895876169205,
+            0.009257055819034576,
+            -0.010109379887580872,
+            -0.0008483355632051826,
+            0.004177297931164503,
+            0.0215131975710392,
+            -0.03287143632769585,
+            -0.0010705322492867708,
+            -0.03239741921424866,
+            -0.001720600062981248,
+            -0.00011544257722562179,
+            0.03288966789841652,
+            -0.012251583859324455,
+            -0.019981749355793,
+            -0.002964901505038142,
+            0.006782126612961292,
+            -0.03305375203490257,
+            -0.001533726928755641,
+            0.007133083418011665,
+            -0.0037260677199810743,
+            0.01741110533475876,
+            0.0037807621993124485,
+            0.0022983115632086992,
+            -0.015186859294772148,
+            -0.012452131137251854,
+            0.0192160252481699,
+            0.016007278114557266,
+            -0.007351861800998449,
+            0.013509559445083141,
+            -0.022041911259293556,
+            -0.00624885456636548,
+            0.004491791594773531,
+            0.0030082014854997396,
+            -0.0008990420028567314,
+            0.034603431820869446,
+            0.018532343208789825,
+            -0.005829529836773872,
+            0.007456693332642317,
+            -0.0009639917989261448,
+            0.022096605971455574,
+            0.011841374449431896,
+            0.032452113926410675,
+            -0.00053782993927598,
+            0.03004555031657219,
+            -0.02858702838420868,
+            -0.00673199025914073,
+            -0.005259794648736715,
+            0.01412943098694086,
+            -0.025068344548344612,
+            0.015287132933735847,
+            -3.9169288356788456e-05,
+            0.028915196657180786,
+            0.009562434628605843,
+            0.0014608008787035942,
+            0.004250223748385906,
+            0.006344570312649012,
+            -0.0017376920441165566,
+            0.005241563078016043,
+            -0.0016909738769754767,
+            -0.026399245485663414,
+            -0.023445740342140198,
+            -0.0028122125659137964,
+            -0.018650848418474197,
+            0.014521408826112747,
+            -0.005770277231931686,
+            -0.006773010827600956,
+            -0.011795795522630215,
+            0.015688225626945496,
+            0.010638094507157803,
+            0.00916134100407362,
+            0.001238604192622006,
+            0.021185031160712242,
+            0.01832268200814724,
+            0.01577938348054886,
+            -0.003901546122506261,
+            0.0099908746778965,
+            -0.000999315408989787,
+            0.011485859751701355,
+            0.009699170477688313,
+            -0.011367355473339558,
+            0.020510464906692505,
+            -0.0032748375087976456,
+            -0.01415677834302187,
+            0.0075159454718232155,
+            -0.001373061677441001,
+            0.010638094507157803,
+            -0.02818593569099903,
+            0.0026754760183393955,
+            0.008591605350375175,
+            0.011412933468818665,
+            0.009917949326336384,
+            -0.01565176248550415,
+            -0.014895155094563961,
+            0.02450316771864891,
+            -0.005282584112137556,
+            0.01819506101310253,
+            0.0029762962367385626,
+            0.017912471666932106,
+            0.011914300732314587,
+            0.002582039451226592,
+            -0.021458502858877182,
+            -0.0015006823232397437,
+            0.0022686852607876062,
+            -0.0005760021740570664,
+            -0.008021870627999306,
+            -0.01054693665355444,
+            0.005597077775746584,
+            -0.036535974591970444,
+            0.004371007904410362,
+            -0.015031890943646431,
+            -0.02588876336812973,
+            0.004560159984976053,
+            -0.012935265898704529,
+            0.002186643425375223,
+            0.0032429322600364685,
+            0.02895165979862213,
+            -0.009999990463256836,
+            -0.02824063040316105,
+            1.6219646568060853e-05,
+            0.0030856854282319546,
+            -0.004113487433642149,
+            0.013309012167155743,
+            0.015788499265909195,
+            -0.0020054676569998264,
+            0.0026230604853481054,
+            0.042370062321424484,
+            -0.001320646028034389,
+            -0.006804916076362133,
+            0.00527346832677722,
+            -0.003721509827300906,
+            -0.012297162786126137,
+            -0.0003475384146440774,
+            0.008518679067492485,
+            0.011549670249223709,
+            -0.018587037920951843,
+            -0.00565633038058877,
+            0.010082032531499863,
+            0.030227866023778915,
+            0.017301715910434723,
+            0.013236085884273052,
+            -0.0024863239377737045,
+            0.005054689943790436,
+            -0.0018402443965896964,
+            -0.00023558546672575176,
+            -0.015578837133944035,
+            -0.029334520921111107,
+            -0.01970827765762806,
+            -0.0046991752460598946,
+            -0.01214219443500042,
+            -0.009908833540976048,
+            -0.011631712317466736,
+            0.0077347238548099995,
+            -0.0014608008787035942,
+            0.007014578673988581,
+            0.004357334226369858,
+            -0.010738367214798927,
+            0.010765714570879936,
+            0.022716479375958443,
+            0.017292600125074387,
+            -0.0018470812356099486,
+            0.01269825641065836,
+            -0.003842293517664075,
+            0.004008656367659569,
+            0.002835002029314637,
+            -0.010665440931916237,
+            -0.00978121254593134,
+            -0.0006865308387205005,
+            0.012415667995810509,
+            0.011349123902618885,
+            -0.01513216458261013,
+            0.002525066025555134,
+            0.011896069161593914,
+            0.010720135644078255,
+            0.0032953477930277586,
+            -0.020309917628765106,
+            0.015123048797249794,
+            0.0021205542143434286,
+            0.004703733138740063,
+            -0.0017638999270275235,
+            0.0008722644415684044,
+            -0.004024608992040157,
+            0.014211473055183887,
+            -0.01211484707891941,
+            0.014329977333545685,
+            0.024302620440721512,
+            0.01725613698363304,
+            0.005469457246363163,
+            0.007142199203372002,
+            0.008354595862329006,
+            0.0027278917841613293,
+            -0.009220592677593231,
+            -0.003871919820085168,
+            -0.0018413838697597384,
+            -0.00596626615151763,
+            0.023099340498447418,
+            -0.0092251505702734,
+            0.005414762534201145,
+            0.018906090408563614,
+            -0.008997256867587566,
+            -0.006467632949352264,
+            0.009279845282435417,
+            -0.002962622558698058,
+            -0.012187773361802101,
+            0.013263433240354061,
+            -0.002878301776945591,
+            0.01030081044882536,
+            0.014530524611473083,
+            -0.0013844562927260995,
+            -0.0022504536900669336,
+            -0.00414767162874341,
+            -0.0074475775472819805,
+            0.06745664030313492,
+            -0.015123048797249794,
+            0.004466723185032606,
+            -0.010984493419528008,
+            -0.017155863344669342,
+            0.014530524611473083,
+            0.02027345448732376,
+            0.02942567877471447,
+            -0.010701904073357582,
+            0.003972193226218224,
+            0.000644940126221627,
+            0.0017958050593733788,
+            0.018085671588778496,
+            0.006800358183681965,
+            -0.0009081577882170677,
+            -0.017265252768993378,
+            -0.01842295378446579,
+            -0.015241554006934166,
+            0.00473563838750124,
+            -0.01881493255496025,
+            0.0038149463944137096,
+            0.0046991752460598946,
+            -0.001347993267700076,
+            -0.013783032074570656,
+            0.005879666190594435,
+            0.003363716183230281,
+            -0.020510464906692505,
+            -0.0018744284752756357,
+            -0.008049217984080315,
+            -0.04740196093916893,
+            0.017721040174365044,
+            -0.005820414051413536,
+            0.011722870171070099,
+            -0.0032429322600364685,
+            0.016800349578261375,
+            0.007292609661817551,
+            0.004040561616420746,
+            0.005241563078016043,
+            0.009635359980165958,
+            0.009562434628605843,
+            0.013090234249830246,
+            -0.00982679147273302,
+            0.013072002679109573,
+            0.013637179508805275,
+            -0.007939828559756279,
+            -0.004498628433793783,
+            0.013454864732921124,
+            -0.006125791929662228,
+            -0.0011434584157541394,
+            0.0047584278509020805,
+            0.009744749404489994,
+            0.0062807598151266575,
+            -0.0013685037847608328,
+            0.004858701024204493,
+            -0.0004552183672785759,
+            0.02824063040316105,
+            0.0019040547776967287,
+            -0.00737465126439929,
+            -0.015496795065701008,
+            -0.01800362952053547,
+            -0.022096605971455574,
+            -0.010802177712321281,
+            -0.014074736274778843,
+            0.002639013109728694,
+            -0.05677296593785286,
+            0.022716479375958443,
+            0.0028144915122538805,
+            0.0037465780042111874,
+            -0.0024339084047824144,
+            -0.013810379430651665,
+            0.030665423721075058,
+            -0.004749312065541744,
+            -0.022096605971455574,
+            -0.01602550968527794,
+            0.011941648088395596,
+            0.002258430002257228,
+            0.0018539180746302009,
+            -0.009863254614174366,
+            -0.0026731970719993114,
+            -0.0013673643115907907,
+            0.006736548151820898,
+            0.014038273133337498,
+            0.005478573031723499,
+            0.007164988666772842,
+            0.02393799088895321,
+            0.007201451808214188,
+            -0.01711028441786766,
+            0.007538734935224056,
+            -0.008281669579446316,
+            0.0039995405822992325,
+            -0.005765719339251518,
+            -0.002180946059525013,
+            0.003687325632199645,
+            -0.00339334225282073,
+            -0.0075478507205843925,
+            -0.011349123902618885,
+            0.002987690968438983,
+            0.004881490487605333,
+            -0.003343205666169524,
+            -0.002072696341201663,
+            0.01467637624591589,
+            0.0217137448489666,
+            0.015423868782818317,
+            -0.0008209883235394955,
+            -0.005628983024507761,
+            0.014320861548185349,
+            0.0107565987855196,
+            0.005319047253578901,
+            -0.007898807525634766,
+            -0.006513211876153946,
+            0.007092062849551439,
+            -0.01738375797867775,
+            0.0050774794071912766,
+            -0.01950773037970066,
+            0.0003783041029237211,
+            0.005852319300174713,
+            0.016681844368577003,
+            -0.0005318477051332593,
+            -0.006162255071103573,
+            0.0037374624516814947,
+            0.0077666291035711765,
+            0.003691883524879813,
+            0.01515951193869114,
+            0.014357324689626694,
+            -0.010273464024066925,
+            0.022114837542176247,
+            0.000282588618574664,
+            -0.007493156008422375,
+            -0.0023814928717911243,
+            -0.0027051023207604885,
+            0.01985413022339344,
+            -0.0009167037787847221,
+            0.015123048797249794,
+            -0.006435727700591087,
+            0.020802168175578117,
+            0.007206009700894356,
+            0.004767543636262417,
+            0.025797605514526367,
+            0.01651776023209095,
+            0.002832723082974553,
+            0.012360973283648491,
+            0.08350037783384323,
+            0.018240639939904213,
+            -0.010975377634167671,
+            -0.017547842115163803,
+            -0.0047265226021409035,
+            -0.0071695465594530106,
+            0.003413852769881487,
+            0.017493147403001785,
+            -0.017292600125074387,
+            -0.023354582488536835,
+            0.0006494980189017951,
+            0.000889356539119035,
+            0.0037875990383327007,
+            -0.01554237399250269,
+            0.01780308224260807,
+            0.015296248719096184,
+            -0.0013821773463860154,
+            0.0046376436948776245,
+            0.005765719339251518,
+            -0.007593429647386074,
+            -0.007620777003467083,
+            0.010401084087789059,
+            -0.015387405641376972,
+            -0.014102083630859852,
+            0.01190518494695425,
+            -0.004876932594925165,
+            -0.00021450527128763497,
+            0.004020051099359989,
+            0.010082032531499863,
+            -0.00183112861122936,
+            0.015287132933735847,
+            0.02353689633309841,
+            0.004532812628895044,
+            0.003794435877352953,
+            0.015268901363015175,
+            0.01530536450445652,
+            -0.00046746767475269735,
+            -0.0036166785284876823,
+            -0.01671830751001835,
+            0.009626244194805622,
+            0.006845937110483646,
+            -0.015323596075177193,
+            0.016216941177845,
+            0.02193252369761467,
+            0.006257970351725817,
+            0.006485864520072937,
+            0.00468550156801939,
+            -0.0004227434692438692,
+            0.01916133053600788,
+            0.000195134271052666,
+            -0.0034571527503430843,
+            0.006230622995644808,
+            0.007520503364503384,
+            -0.012898802757263184,
+            0.017839545384049416,
+            0.013664526864886284,
+            0.012379204854369164,
+            0.007319956552237272,
+            0.0037853200919926167,
+            0.010720135644078255,
+            -0.022443005815148354,
+            0.016508644446730614,
+            -0.018514111638069153
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "8f4dd4ba-f551-4c79-af63-003b9ccf3976",
+        "properties": {
+          "page_content": "8 Future Work 26 ",
+          "keyphrases": [],
+          "embedding": [
+            -0.024294501170516014,
+            0.01452074758708477,
+            -0.01423195656388998,
+            0.00711597828194499,
+            -0.015378094278275967,
+            0.004546194337308407,
+            -0.010901841334998608,
+            0.11226730793714523,
+            0.006362415850162506,
+            0.04544840380549431,
+            0.02425840124487877,
+            5.9330373915145174e-05,
+            -0.00413782661780715,
+            -0.018969925120472908,
+            -0.018825531005859375,
+            0.03312066197395325,
+            -0.018969925120472908,
+            0.014204883016645908,
+            -0.029601026326417923,
+            -0.0012081821914762259,
+            -0.01162381749600172,
+            -0.0036031126510351896,
+            -0.03490755334496498,
+            0.01664155349135399,
+            -0.001300685340538621,
+            0.011109409853816032,
+            0.04822801426053047,
+            0.030413249507546425,
+            -0.05245157331228256,
+            0.018013307824730873,
+            -0.007968813180923462,
+            0.023933513090014458,
+            0.019258717074990273,
+            -0.005049321334809065,
+            0.012463115155696869,
+            0.006443637888878584,
+            0.027326801791787148,
+            -0.008451634086668491,
+            0.0013762671733275056,
+            0.012914350256323814,
+            -0.0050673708319664,
+            -0.00494553754106164,
+            -0.009110437706112862,
+            0.026081392541527748,
+            0.035539280623197556,
+            -0.01277897972613573,
+            -0.012950449250638485,
+            -0.04808361828327179,
+            0.0016458802856504917,
+            0.007197200786322355,
+            0.0050809080712497234,
+            -0.022200770676136017,
+            0.001969641540199518,
+            0.0006689561414532363,
+            0.02433059923350811,
+            -0.00736415758728981,
+            0.012986548244953156,
+            0.07804563641548157,
+            0.020558273419737816,
+            0.007833442650735378,
+            -0.022200770676136017,
+            -0.02465548925101757,
+            -0.0032579179387539625,
+            -0.03779545798897743,
+            0.016109095886349678,
+            -0.02696581371128559,
+            0.029564928263425827,
+            -0.011903583072125912,
+            0.009882049635052681,
+            0.0025224045384675264,
+            -0.0248179342597723,
+            -0.005893131252378225,
+            -0.007359645329415798,
+            0.04508741572499275,
+            0.00730549730360508,
+            -0.017210109159350395,
+            0.026370182633399963,
+            0.018753333017230034,
+            -0.02730875089764595,
+            0.046711862087249756,
+            0.01647910848259926,
+            -0.03963649645447731,
+            -0.005058346316218376,
+            0.021496843546628952,
+            0.024529142305254936,
+            -0.004372468683868647,
+            -0.04992465674877167,
+            -0.11732114106416702,
+            0.016452033072710037,
+            0.0011562900617718697,
+            -0.005071883089840412,
+            0.015332970768213272,
+            0.0010463015642017126,
+            -0.028680507093667984,
+            -0.04238000512123108,
+            0.04259659722447395,
+            -0.00870432611554861,
+            0.010405482724308968,
+            -0.0004030094132758677,
+            -0.020215334370732307,
+            0.01059500128030777,
+            0.005374210886657238,
+            0.005992402788251638,
+            -0.02342812903225422,
+            0.0422356091439724,
+            -0.02474573627114296,
+            0.019872397184371948,
+            0.015026130713522434,
+            0.0034542051143944263,
+            -0.002572040306404233,
+            0.005545679945498705,
+            -0.033644095063209534,
+            -0.022327115759253502,
+            0.018148677423596382,
+            -0.006168384570628405,
+            -0.01757109723985195,
+            0.013952190987765789,
+            -0.03044934757053852,
+            -0.029474681243300438,
+            0.04494302347302437,
+            0.04559279978275299,
+            0.019511409103870392,
+            0.027687789872288704,
+            -0.03633345663547516,
+            -0.032073795795440674,
+            0.02905954420566559,
+            0.012327744625508785,
+            0.00182637432590127,
+            -0.003952819854021072,
+            0.03254307806491852,
+            0.02994396537542343,
+            0.004620647989213467,
+            -0.02211052179336548,
+            -0.03440216928720474,
+            0.021478792652487755,
+            -0.02243541181087494,
+            -0.009548136033117771,
+            0.024186205118894577,
+            -0.0227603018283844,
+            0.01911432109773159,
+            0.004505583085119724,
+            0.004850777797400951,
+            -0.011632842011749744,
+            0.0054193343967199326,
+            -0.006348878610879183,
+            0.03633345663547516,
+            -0.014078537002205849,
+            0.023067140951752663,
+            0.003984406590461731,
+            0.050466138869524,
+            -0.01692131906747818,
+            0.00340231298469007,
+            -0.01360925193876028,
+            0.015712007880210876,
+            0.0010214835638180375,
+            0.0005175103433430195,
+            0.000724796496797353,
+            0.024872081354260445,
+            0.014502697624266148,
+            0.012291645631194115,
+            0.02424035221338272,
+            0.03005226142704487,
+            -0.013672425411641598,
+            0.0012894044630229473,
+            0.005112494342029095,
+            -0.020504126325249672,
+            0.0355934277176857,
+            -0.01125380489975214,
+            -0.04353516921401024,
+            0.03377043828368187,
+            0.05858837440609932,
+            -0.01819380186498165,
+            0.01096501387655735,
+            -0.01952945813536644,
+            0.000879344530403614,
+            -0.0008167356136254966,
+            -0.014917834661900997,
+            0.03696518391370773,
+            -0.028102925047278404,
+            -0.019998742267489433,
+            -0.010802569799125195,
+            0.016677651554346085,
+            -0.007828930392861366,
+            0.003255661576986313,
+            0.01038743369281292,
+            -0.006457175128161907,
+            0.027417048811912537,
+            -0.037614963948726654,
+            0.052595969289541245,
+            0.013573153875768185,
+            0.01576615683734417,
+            -0.030503496527671814,
+            0.026009194552898407,
+            0.013419733382761478,
+            0.00960228405892849,
+            -0.06082649901509285,
+            -0.0021546478383243084,
+            0.026514578610658646,
+            0.004909438546746969,
+            -0.015937626361846924,
+            -0.014782464131712914,
+            -0.01398828998208046,
+            0.012273596599698067,
+            -0.007657460402697325,
+            -0.000787969387602061,
+            0.02068462036550045,
+            0.018500640988349915,
+            0.023969611153006554,
+            0.00773868290707469,
+            -0.024619389325380325,
+            0.023049091920256615,
+            -0.0033887759782373905,
+            -0.010910865850746632,
+            0.005360673647373915,
+            -0.008514807559549809,
+            -0.005405797157436609,
+            -0.02044997736811638,
+            0.04905828461050987,
+            -0.00019558223721105605,
+            -0.04313807934522629,
+            -0.007720633409917355,
+            0.0005521990242414176,
+            0.009854976087808609,
+            -0.004986148327589035,
+            -0.004424360580742359,
+            -0.03593636676669121,
+            -0.009854976087808609,
+            0.06310072541236877,
+            -0.02582870051264763,
+            -0.02730875089764595,
+            -0.01687619462609291,
+            0.004620647989213467,
+            0.0031721831765025854,
+            -0.028157074004411697,
+            -0.018807481974363327,
+            -0.0032285875640809536,
+            -0.012057003565132618,
+            -0.011524545960128307,
+            -0.025341367349028587,
+            0.022724201902747154,
+            0.05815518647432327,
+            0.013879992999136448,
+            -0.038842324167490005,
+            0.015278822742402554,
+            -0.013022646307945251,
+            -0.02059437334537506,
+            -0.04526790976524353,
+            0.09941612929105759,
+            0.023807166144251823,
+            0.003623418277129531,
+            0.0062270453199744225,
+            -0.00327822333201766,
+            0.0001518688368378207,
+            0.018717234954237938,
+            0.04006968066096306,
+            0.07746805250644684,
+            0.04956366866827011,
+            -0.007409281097352505,
+            -0.0322723388671875,
+            0.003966357093304396,
+            -0.021226102486252785,
+            -0.019006025046110153,
+            -0.045556701719760895,
+            -0.0063308291137218475,
+            -0.00036803868715651333,
+            -0.016605453565716743,
+            0.01588347740471363,
+            0.033553846180438995,
+            0.01112745888531208,
+            -0.005292988382279873,
+            0.04429324343800545,
+            0.005144080612808466,
+            0.014250006526708603,
+            -0.0033210907131433487,
+            -0.011831386014819145,
+            -0.02012508735060692,
+            -0.014376352541148663,
+            -0.0018714978359639645,
+            -0.008961530402302742,
+            0.015296871773898602,
+            -0.013419733382761478,
+            0.05848007649183273,
+            0.006195458583533764,
+            -0.006764015182852745,
+            -0.05191009119153023,
+            -0.03360799327492714,
+            -0.010540853254497051,
+            0.021749533712863922,
+            -0.0235725250095129,
+            -0.012381892651319504,
+            0.02202027477324009,
+            -0.0120028555393219,
+            -0.020486075431108475,
+            0.03530463948845863,
+            0.007066342514008284,
+            -0.028915148228406906,
+            0.005202741362154484,
+            0.020215334370732307,
+            -0.011867485009133816,
+            -0.0019312864169478416,
+            0.02341008000075817,
+            0.003925745841115713,
+            -0.025233069434762,
+            -0.004307039547711611,
+            0.005947279278188944,
+            0.021984176710247993,
+            0.015405167825520039,
+            0.025178922340273857,
+            -0.05801079049706459,
+            -0.014430500566959381,
+            0.043823957443237305,
+            -0.04173022881150246,
+            -0.020702669396996498,
+            -0.045881591737270355,
+            -0.016127144917845726,
+            0.03407727926969528,
+            0.028915148228406906,
+            0.02738094888627529,
+            -0.04660356789827347,
+            -0.03783155605196953,
+            -0.031063027679920197,
+            -0.00024550012312829494,
+            -0.05306525528430939,
+            0.030178606510162354,
+            0.0006040910957381129,
+            -0.0036865912843495607,
+            -0.023049091920256615,
+            0.049527570605278015,
+            0.01749889925122261,
+            -0.005482506938278675,
+            0.004674796015024185,
+            -0.007652948144823313,
+            0.018058430403470993,
+            -0.006592545658349991,
+            0.010640124790370464,
+            -0.034384120255708694,
+            -0.02878880314528942,
+            0.03631540387868881,
+            0.00046590028796344995,
+            -0.018112579360604286,
+            0.040502868592739105,
+            0.005446408409625292,
+            0.015179550275206566,
+            0.02218271978199482,
+            -0.018347222357988358,
+            -0.03573782369494438,
+            0.041405338793992996,
+            -0.05086322873830795,
+            0.004361187573522329,
+            0.0067188916727900505,
+            0.000985948834568262,
+            -0.04006968066096306,
+            -0.00813576951622963,
+            0.019655803218483925,
+            0.010179865173995495,
+            -0.044401541352272034,
+            -0.015305896289646626,
+            -0.004252891521900892,
+            0.021785633638501167,
+            -0.014340253546833992,
+            0.01498100720345974,
+            -0.04772263020277023,
+            -0.030070310458540916,
+            0.0056900754570961,
+            -0.05819128453731537,
+            -0.021406596526503563,
+            0.029691273346543312,
+            0.008726888336241245,
+            -0.013961215503513813,
+            -0.005094444844871759,
+            -0.03469095751643181,
+            0.013816820457577705,
+            0.02664092369377613,
+            -0.02804877795279026,
+            -0.00885323341935873,
+            -0.010739396326243877,
+            0.01009864266961813,
+            0.017950134351849556,
+            -0.04003358259797096,
+            0.022146621719002724,
+            -0.021659286692738533,
+            0.01274288073182106,
+            -0.014502697624266148,
+            -0.02994396537542343,
+            -0.01675887405872345,
+            -0.01637081243097782,
+            -0.01224652212113142,
+            0.02010703831911087,
+            -0.006646693684160709,
+            0.0017428958090022206,
+            0.002045223256573081,
+            -0.0347631573677063,
+            -0.0073867193423211575,
+            -0.027435097843408585,
+            0.03667639195919037,
+            0.027001911774277687,
+            0.02954687736928463,
+            -0.0042754532769322395,
+            0.018157701939344406,
+            -0.02979956939816475,
+            -0.028500013053417206,
+            -0.008420048281550407,
+            -0.0028269882313907146,
+            0.020052891224622726,
+            0.0463508740067482,
+            0.021875880658626556,
+            -0.0029646148905158043,
+            0.011632842011749744,
+            0.006800113711506128,
+            0.004968099296092987,
+            0.025251120328903198,
+            -0.03467290848493576,
+            0.011867485009133816,
+            -0.0421995110809803,
+            0.04981636255979538,
+            0.037542764097452164,
+            0.04310198128223419,
+            0.04104435071349144,
+            -0.03171280771493912,
+            0.0012860201532021165,
+            -0.0017541766865178943,
+            -0.01046865526586771,
+            0.03180305287241936,
+            -0.008650178089737892,
+            0.008907382376492023,
+            0.034546565264463425,
+            0.00457101222127676,
+            -0.013663400895893574,
+            -0.011578693985939026,
+            -0.023969611153006554,
+            -0.021550990641117096,
+            -0.000988204963505268,
+            0.025846749544143677,
+            -0.018347222357988358,
+            0.012327744625508785,
+            -0.02416815422475338,
+            0.004546194337308407,
+            -0.0022155647166073322,
+            -0.005121518857777119,
+            -0.025467712432146072,
+            0.014376352541148663,
+            -0.02631603367626667,
+            -0.002851806115359068,
+            0.03534073755145073,
+            0.006633156910538673,
+            -0.0163437370210886,
+            -0.04526790976524353,
+            -0.005414821673184633,
+            0.0007496143807657063,
+            0.00883067212998867,
+            -0.008478708565235138,
+            0.03772325813770294,
+            -0.040827758610248566,
+            -0.023319832980632782,
+            0.039744794368743896,
+            -0.015468341298401356,
+            -0.004737969022244215,
+            0.007720633409917355,
+            -0.02211052179336548,
+            -0.038842324167490005,
+            -0.0004580036911647767,
+            0.05786639824509621,
+            0.034709006547927856,
+            -0.002942053135484457,
+            -0.022363213822245598,
+            0.0013954447349533439,
+            -0.03815644606947899,
+            -0.014944908209145069,
+            -0.0017113093053922057,
+            0.05107982084155083,
+            0.003921233583241701,
+            -0.009575209580361843,
+            0.0023464227560907602,
+            -0.003269198816269636,
+            -0.021966127678751945,
+            0.016397885978221893,
+            0.01187650952488184,
+            -0.005450920667499304,
+            -0.018347222357988358,
+            0.005008710082620382,
+            -0.008911894634366035,
+            0.009412765502929688,
+            -0.027579491958022118,
+            -0.016072995960712433,
+            0.017354503273963928,
+            0.025052575394511223,
+            0.03180305287241936,
+            0.02017923630774021,
+            0.018590888008475304,
+            0.011118434369564056,
+            0.045809391885995865,
+            -0.013419733382761478,
+            0.001081836293451488,
+            0.006033014040440321,
+            -0.01724620722234249,
+            -0.042740993201732635,
+            0.006412051618099213,
+            -0.07840662449598312,
+            -0.004950049798935652,
+            0.03180305287241936,
+            -0.022236868739128113,
+            0.017607195302844048,
+            -0.0008167356136254966,
+            0.02880685217678547,
+            0.016632527112960815,
+            -0.027778036892414093,
+            -0.004611623473465443,
+            -0.015134426765143871,
+            0.011010137386620045,
+            0.012174324132502079,
+            0.0008443737751804292,
+            0.021984176710247993,
+            0.00013847279478795826,
+            0.00972863007336855,
+            -0.009295444004237652,
+            0.012625560164451599,
+            0.005144080612808466,
+            0.011037211865186691,
+            -0.021226102486252785,
+            0.006109723821282387,
+            -0.00761233689263463,
+            -0.000742281845305115,
+            0.008451634086668491,
+            0.03624320775270462,
+            -0.03862572833895683,
+            0.0006170640699565411,
+            -0.03873402625322342,
+            -0.008189918473362923,
+            0.06941801309585571,
+            -0.03167670965194702,
+            -0.01505320519208908,
+            0.006610595155507326,
+            -0.00080432667164132,
+            0.003090960904955864,
+            0.002966871252283454,
+            0.027850233018398285,
+            -0.043499067425727844,
+            0.019060172140598297,
+            -0.015332970768213272,
+            -0.026279935613274574,
+            -0.02433059923350811,
+            0.021370496600866318,
+            0.03275967389345169,
+            0.00581190874800086,
+            0.00792820192873478,
+            0.0388784222304821,
+            0.01985434629023075,
+            -0.031135225668549538,
+            0.01411463599652052,
+            -0.014222932048141956,
+            -0.0028269882313907146,
+            0.014268055558204651,
+            0.015802254900336266,
+            -0.014674167148768902,
+            0.012047979049384594,
+            0.0005457125371322036,
+            0.0025878336746245623,
+            0.015035155229270458,
+            0.021695386618375778,
+            0.0051892041228711605,
+            0.012842152267694473,
+            -0.020251434296369553,
+            -0.006633156910538673,
+            0.019150419160723686,
+            -0.010360359214246273,
+            -0.03923941031098366,
+            -0.0016537768533453345,
+            0.048877790570259094,
+            -0.03481730446219444,
+            -0.019511409103870392,
+            0.004349906928837299,
+            0.032055746763944626,
+            -0.009647407568991184,
+            -0.03481730446219444,
+            -0.012201398611068726,
+            0.022200770676136017,
+            0.014439525082707405,
+            -0.016722774133086205,
+            -0.03552123159170151,
+            0.0044401539489626884,
+            -0.004094959236681461,
+            -0.007440867833793163,
+            -0.004056604113429785,
+            0.016776923090219498,
+            0.005617877934128046,
+            -0.0017801227513700724,
+            -0.011768212541937828,
+            -0.02564820647239685,
+            -0.04945537447929382,
+            0.02308518998324871,
+            -0.0063082673586905,
+            0.019475309178233147,
+            0.01059500128030777,
+            -0.003131571924313903,
+            -0.03840913623571396,
+            0.010053519159555435,
+            -0.002939797006547451,
+            0.004234842024743557,
+            0.042488303035497665,
+            -0.037037380039691925,
+            0.00377683830447495,
+            0.023049091920256615,
+            -0.016154218465089798,
+            -0.0248179342597723,
+            0.00680913869291544,
+            -0.01299557276070118,
+            0.002973639639094472,
+            0.03451046347618103,
+            -0.01604592241346836,
+            0.0260633435100317,
+            0.02035973034799099,
+            0.0007941738585941494,
+            0.05006905272603035,
+            0.00674596568569541,
+            0.006118748802691698,
+            -0.006290218327194452,
+            0.014719290658831596,
+            -0.0037226900458335876,
+            -0.014349278062582016,
+            0.0008415535558015108,
+            0.0310269296169281,
+            -0.013067769818007946,
+            -0.01514345221221447,
+            -0.006204483564943075,
+            0.03461876139044762,
+            -0.005960816517472267,
+            0.007720633409917355,
+            -0.03797595202922821,
+            -0.006064600311219692,
+            0.00780185591429472,
+            0.020756816491484642,
+            -0.00798686221241951,
+            0.012463115155696869,
+            0.0011811080621555448,
+            -0.010161816142499447,
+            0.010441581718623638,
+            0.033553846180438995,
+            -0.00027962480089627206,
+            0.012806054204702377,
+            -0.011641867458820343,
+            -0.017137911170721054,
+            -0.011046236380934715,
+            0.018518690019845963,
+            0.0612235851585865,
+            -0.03321090713143349,
+            0.020305581390857697,
+            0.010640124790370464,
+            0.0044446662068367004,
+            0.015179550275206566,
+            0.016975466161966324,
+            0.020666569471359253,
+            0.0019245179137215018,
+            -0.007337083574384451,
+            0.03400508314371109,
+            0.018735283985733986,
+            -0.014583920128643513,
+            -0.0007501784712076187,
+            0.0025946020614355803,
+            -0.05219888314604759,
+            0.0003074039414059371,
+            -0.02216467075049877,
+            0.0019211337203159928,
+            -0.000984820770099759,
+            -0.011632842011749744,
+            0.002883392618969083,
+            0.017688417807221413,
+            0.004706382751464844,
+            0.01162381749600172,
+            -0.029853718355298042,
+            -0.009322518482804298,
+            0.0017553047509863973,
+            -0.021460743620991707,
+            0.04313807934522629,
+            0.037290073931217194,
+            -0.04653136804699898,
+            -0.01795915886759758,
+            0.004638697486370802,
+            0.049274880439043045,
+            0.0011342923389747739,
+            0.002632957184687257,
+            -0.029492730274796486,
+            -0.010567927733063698,
+            0.015278822742402554,
+            0.006705354433506727,
+            0.0028630869928747416,
+            0.017950134351849556,
+            -0.013455832377076149,
+            0.018040381371974945,
+            -0.005893131252378225,
+            0.008808109909296036,
+            -0.015666885301470757,
+            0.029727371409535408,
+            -0.011118434369564056,
+            0.000694338115863502,
+            0.01662350259721279,
+            0.018572838976979256,
+            -0.031171325594186783,
+            0.013356560841202736,
+            0.03176695480942726,
+            0.013870968483388424,
+            -0.018184777349233627,
+            -0.015594687312841415,
+            -0.044473737478256226,
+            0.026334084570407867,
+            0.007648435886949301,
+            0.0016470083501189947,
+            -0.00259685842320323,
+            -0.020161187276244164,
+            -0.018275024369359016,
+            -0.027038009837269783,
+            -0.01567590981721878,
+            -0.010694272816181183,
+            0.005139568354934454,
+            0.00446045957505703,
+            0.003928001970052719,
+            0.017814764752984047,
+            0.007034756243228912,
+            -0.02059437334537506,
+            -0.016100069507956505,
+            -0.014710266143083572,
+            0.0045304009690880775,
+            -0.024384748190641403,
+            -0.048047520220279694,
+            -0.03452851250767708,
+            -0.015450291335582733,
+            0.020504126325249672,
+            0.030918633565306664,
+            -0.006353390868753195,
+            0.011262829415500164,
+            -0.010730371810495853,
+            0.01472831517457962,
+            -0.0003917285066563636,
+            -0.0055772666819393635,
+            0.003946051467210054,
+            -0.012896301224827766,
+            0.03270552307367325,
+            0.01022498868405819,
+            0.021893929690122604,
+            0.010071569122374058,
+            -0.004316064063459635,
+            0.029402483254671097,
+            0.013636326417326927,
+            -0.013248263858258724,
+            0.0024005710147321224,
+            0.013645350933074951,
+            -0.006533884909003973,
+            0.048191916197538376,
+            -0.0014304154319688678,
+            -0.019078223034739494,
+            0.009313493967056274,
+            0.007562701124697924,
+            -0.0363876037299633,
+            0.05389552563428879,
+            -0.01178626250475645,
+            -0.0281209759414196,
+            0.010802569799125195,
+            0.01763427071273327,
+            -0.02581065148115158,
+            -0.006023989524692297,
+            0.015170525759458542,
+            -0.021388545632362366,
+            -0.0033662142232060432,
+            0.019258717074990273,
+            0.03617101162672043,
+            0.014944908209145069,
+            0.0013785234186798334,
+            -0.026677021756768227,
+            -0.017020590603351593,
+            0.023861315101385117,
+            -0.022200770676136017,
+            0.02516087330877781,
+            0.0013401684118434787,
+            -0.002479537157341838,
+            -0.02664092369377613,
+            0.009845951572060585,
+            -0.014565871097147465,
+            0.015666885301470757,
+            -0.025142822414636612,
+            0.015044179745018482,
+            0.008988603949546814,
+            0.00914202444255352,
+            0.015360044315457344,
+            -0.05277646332979202,
+            -0.03174890577793121,
+            -0.030900582671165466,
+            0.014746365137398243,
+            0.028355617076158524,
+            -0.03093668259680271,
+            -0.016569355502724648,
+            0.012887275777757168,
+            0.006452662870287895,
+            0.015278822742402554,
+            -0.012273596599698067,
+            4.801424074685201e-05,
+            0.02342812903225422,
+            -0.03270552307367325,
+            -0.013762672431766987,
+            0.0297815203666687,
+            0.006890360731631517,
+            -0.012562386691570282,
+            0.040936052799224854,
+            0.007571726106107235,
+            -0.03467290848493576,
+            -0.0036008565220981836,
+            -0.01303167175501585,
+            -0.0023396543692797422,
+            -0.005942767020314932,
+            0.005256889387965202,
+            0.008528344333171844,
+            0.012174324132502079,
+            0.0060736252926290035,
+            -0.023482277989387512,
+            0.02781413495540619,
+            0.005974353291094303,
+            0.014629043638706207,
+            0.01812160387635231,
+            -0.014638068154454231,
+            0.007513065356761217,
+            0.005653976462781429,
+            0.0037881191819906235,
+            0.013248263858258724,
+            0.0013954447349533439,
+            0.05891326069831848,
+            -0.014015363529324532,
+            0.005536655429750681,
+            0.0513325110077858,
+            0.011795287020504475,
+            -0.002567528048530221,
+            0.020468026399612427,
+            0.023789117112755775,
+            -0.013167042285203934,
+            -0.01766134425997734,
+            -0.011686990968883038,
+            -0.0031744393054395914,
+            -0.011389175429940224,
+            0.03146011382341385,
+            0.002407339634373784,
+            0.02158709056675434,
+            -0.03153231367468834,
+            0.03568367660045624,
+            0.006425588857382536,
+            -0.011307952925562859,
+            0.0132392393425107,
+            -0.0244208462536335,
+            -0.019493358209729195,
+            -0.006646693684160709,
+            -0.02472768723964691,
+            -0.006159360054880381,
+            -0.007106953766196966,
+            0.0310269296169281,
+            0.025918947532773018,
+            0.008537368848919868,
+            0.005667513702064753,
+            0.003379751229658723,
+            -0.0239876601845026,
+            -0.013230214826762676,
+            0.005568241700530052,
+            0.011064286343753338,
+            0.03451046347618103,
+            0.021334398537874222,
+            -0.0032669424545019865,
+            -0.006136798299849033,
+            -0.007806368172168732,
+            -0.011524545960128307,
+            0.017002541571855545,
+            -0.02887905016541481,
+            -0.03261527791619301,
+            -0.014078537002205849,
+            -0.003988918848335743,
+            -0.0025788089260458946,
+            0.00024634620058350265,
+            0.021226102486252785,
+            0.004410823807120323,
+            0.021532941609621048,
+            0.0016210622852668166,
+            0.012093102559447289,
+            -0.02664092369377613,
+            -0.007355133071541786,
+            0.031893301755189896,
+            0.0021501355804502964,
+            -0.014313179068267345,
+            0.013879992999136448,
+            -0.012860202230513096,
+            -0.012842152267694473,
+            -0.016749849542975426,
+            0.029925916343927383,
+            0.0033910321071743965,
+            -0.03423972427845001,
+            -0.010712322778999805,
+            -0.023698870092630386,
+            0.0020170211791992188,
+            -0.004401798825711012,
+            0.02615359053015709,
+            -0.03855353221297264,
+            0.0010874767322093248,
+            0.014881735667586327,
+            -0.005667513702064753,
+            -0.010306211188435555,
+            0.00447174021974206,
+            0.004006968345493078,
+            -0.017923060804605484,
+            -0.010495729744434357,
+            -0.012950449250638485,
+            -0.020468026399612427,
+            -0.009132999926805496,
+            -0.010243037715554237,
+            0.013690474443137646,
+            0.010784519836306572,
+            -0.008266627788543701,
+            0.008893844671547413,
+            -0.002168184844776988,
+            -0.02622578665614128,
+            -0.020088989287614822,
+            -0.022886646911501884,
+            0.03552123159170151,
+            -0.0030413249041885138,
+            -0.03837303817272186,
+            -0.0033323715906590223,
+            -0.0018940595909953117,
+            0.0011529058683663607,
+            0.0067279161885380745,
+            -0.01352803036570549,
+            -0.0182118508964777,
+            0.013753647916018963,
+            0.007237812038511038,
+            -0.013546079397201538,
+            0.011650891974568367,
+            0.026514578610658646,
+            0.017787689343094826,
+            0.02341008000075817,
+            0.011226730421185493,
+            0.037614963948726654,
+            0.017300356179475784,
+            -0.002073425566777587,
+            0.043065883219242096,
+            -0.001112294616177678,
+            -0.01844649389386177,
+            0.046314775943756104,
+            0.00023196307301986963,
+            -0.0017214621184393764,
+            0.027579491958022118,
+            -0.00013339638826437294,
+            0.0037046405486762524,
+            0.011668941006064415,
+            -0.027272652834653854,
+            0.005117006599903107,
+            0.004706382751464844,
+            0.03655004873871803,
+            0.005717149470001459,
+            -0.017760615795850754,
+            -0.022381264716386795,
+            0.012643609195947647,
+            0.018663085997104645,
+            -0.0011258317390456796,
+            0.014006339013576508,
+            0.006786576937884092,
+            -0.03920331224799156,
+            0.015784205868840218,
+            -0.0069760954938828945,
+            0.010622075758874416,
+            0.005762272980064154,
+            0.005153105594217777,
+            -0.016172267496585846,
+            0.0007783806649968028,
+            0.009575209580361843,
+            0.03815644606947899,
+            -0.013094844296574593,
+            0.03331920504570007,
+            -0.032308436930179596,
+            -0.009818877093493938,
+            -0.025846749544143677,
+            0.00782441720366478,
+            0.011542594991624355,
+            0.007355133071541786,
+            -0.042416103184223175,
+            0.005103469826281071,
+            -0.011479422450065613,
+            0.0010835283901542425,
+            -0.002407339634373784,
+            0.008203455246984959,
+            0.04097215086221695,
+            -0.02299494296312332,
+            0.025955045595765114,
+            -0.007350620813667774,
+            0.01837429590523243,
+            -0.0002003766130656004,
+            -0.009191660210490227,
+            -0.019728001207113266,
+            -0.004864315036684275,
+            0.003530915128067136,
+            -0.000880472594872117,
+            -0.007567213382571936,
+            -3.902479147654958e-05,
+            -0.005035784561187029,
+            0.0045394254848361015,
+            -0.006520348135381937,
+            0.01597372442483902,
+            -0.01729133166372776,
+            0.04227171093225479,
+            0.001362730166874826,
+            -0.011190632358193398,
+            -0.0151073532178998,
+            -0.03402313217520714,
+            0.0027705838438123465,
+            -0.006655718665570021,
+            0.02696581371128559,
+            0.014674167148768902,
+            0.020919261500239372,
+            -0.0008658074657432735,
+            0.008785548619925976,
+            0.020648520439863205,
+            -0.017850862815976143,
+            0.024944279342889786,
+            -0.01807648129761219,
+            0.002287762239575386,
+            -0.0040430668741464615,
+            0.016533255577087402,
+            0.01869918406009674,
+            0.01398828998208046,
+            0.0020847064442932606,
+            -0.008591517806053162,
+            -0.032001595944166183,
+            -0.008095158264040947,
+            -0.005893131252378225,
+            0.014583920128643513,
+            0.010279136709868908,
+            0.019637754186987877,
+            0.0015691702719777822,
+            0.010360359214246273,
+            0.00032376122544519603,
+            0.004873339552432299,
+            0.02456524223089218,
+            0.006777551956474781,
+            -0.03978089243173599,
+            0.0005428923177532852,
+            0.001857960713095963,
+            0.009182635694742203,
+            -0.021623188629746437,
+            0.014908809214830399,
+            -0.00537872314453125,
+            0.028752705082297325,
+            -0.007007681764662266,
+            0.02084706351161003,
+            -0.006136798299849033,
+            -0.004661259241402149,
+            -0.003783606691285968,
+            0.006524860393255949,
+            0.030846435576677322,
+            -0.00328047969378531,
+            0.024854032322764397,
+            0.023103240877389908,
+            0.0074724541045725346,
+            0.007851491682231426,
+            -0.017923060804605484,
+            0.0030774236656725407,
+            0.022904695942997932,
+            -0.0017146936152130365,
+            0.007887590676546097,
+            -0.004877852275967598,
+            0.006303755100816488,
+            -0.016677651554346085,
+            0.01627153903245926,
+            0.01452074758708477,
+            -0.0058209337294101715,
+            0.02781413495540619,
+            0.015567612834274769,
+            0.009737654589116573,
+            -0.002993945265188813,
+            0.004408567678183317,
+            0.014538796618580818,
+            0.00432283291593194,
+            0.004140082746744156,
+            -0.015657860785722733,
+            0.013573153875768185,
+            0.010802569799125195,
+            0.013130943290889263,
+            -0.03687493875622749,
+            -0.016036897897720337,
+            -0.02326568402349949,
+            -0.0019121089717373252,
+            0.0194572601467371,
+            0.00011612254456849769,
+            0.001237512449733913,
+            0.025612108409404755,
+            -0.0003043017059098929,
+            -0.008162843994796276,
+            -0.007075367495417595,
+            -0.020973410457372665,
+            0.0017135655507445335,
+            -0.0001287430350203067,
+            -0.04270489513874054,
+            0.007296472322195768,
+            0.007025731261819601,
+            0.001146137248724699,
+            -0.007910151965916157,
+            -0.02572040446102619,
+            -0.01134405191987753,
+            0.08793670684099197,
+            -0.012345793657004833,
+            -0.02283249981701374,
+            -0.008591517806053162,
+            -0.004950049798935652,
+            -0.008659202605485916,
+            -0.003582807257771492,
+            0.021370496600866318,
+            0.01860893703997135,
+            0.0015624017687514424,
+            0.015179550275206566,
+            -0.008108695968985558,
+            0.02936638332903385,
+            0.01208407711237669,
+            0.019998742267489433,
+            -0.0028044264763593674,
+            0.007856003940105438,
+            -0.0044401539489626884,
+            0.013546079397201538,
+            -0.022128572687506676,
+            -0.0011641867458820343,
+            -0.0031383405439555645,
+            0.00407465361058712,
+            -0.02202027477324009,
+            0.007883078418672085,
+            -0.011515521444380283,
+            -0.013302412815392017,
+            -0.0032015135511755943,
+            -0.030467398464679718,
+            -0.013320461846888065,
+            -0.002380265388637781,
+            0.039095014333724976,
+            0.009620334021747112,
+            0.01804940588772297,
+            0.0012442809529602528,
+            0.02267005480825901,
+            0.0072197625413537025,
+            0.0017428958090022206,
+            0.014096586033701897,
+            0.009137512184679508,
+            -0.00923227146267891,
+            0.04624257981777191,
+            0.015405167825520039,
+            -0.0012262315722182393,
+            -0.03277772292494774,
+            0.003679822664707899,
+            0.016424959525465965,
+            0.009295444004237652,
+            -0.0028021703474223614,
+            -0.0017451519379392266,
+            0.00537872314453125,
+            -0.0046793087385594845,
+            0.017399627715349197,
+            0.015008081682026386,
+            -0.02227296680212021,
+            0.017562072724103928,
+            -0.0018331428291276097,
+            0.02259785681962967,
+            0.0006396258249878883,
+            -0.00021673388255294412,
+            0.02837366610765457,
+            -0.0018444237066432834,
+            0.024150105193257332,
+            0.00885323341935873,
+            -0.002257303800433874,
+            0.0033684703521430492,
+            0.017300356179475784,
+            0.004404054954648018,
+            0.006655718665570021,
+            -0.004176181275397539,
+            -0.017968183383345604,
+            0.023067140951752663,
+            0.01650618202984333,
+            0.002729972591623664,
+            -0.01763427071273327,
+            -0.006786576937884092,
+            0.009182635694742203,
+            0.0013807795476168394,
+            0.006651206407696009,
+            -0.005459945183247328,
+            0.01026108767837286,
+            0.014908809214830399,
+            -0.006939996499568224,
+            0.03241673484444618,
+            -0.010992088355123997,
+            -0.02299494296312332,
+            -0.011885534040629864,
+            0.023121289908885956,
+            0.01943921111524105,
+            -0.004207768011838198,
+            -0.015134426765143871,
+            0.031315721571445465,
+            -0.011533570475876331,
+            -0.0036143935285508633,
+            -0.030900582671165466,
+            -0.036189060658216476,
+            0.04501521959900856,
+            -0.0004785912751685828,
+            -0.0446903295814991,
+            0.027453146874904633,
+            0.01692131906747818,
+            0.010694272816181183,
+            0.010243037715554237,
+            -0.022128572687506676,
+            0.015657860785722733,
+            -0.0049274880439043045,
+            0.0045349132269620895,
+            -0.008244066499173641,
+            0.006664743181318045,
+            -0.011010137386620045,
+            0.027669738978147507,
+            -0.01221042312681675,
+            -0.002286634175106883,
+            -0.0010615306673571467,
+            0.00028300905250944197,
+            0.005870569497346878,
+            -0.005062858574092388,
+            -0.014656118117272854,
+            0.011930657550692558,
+            0.01389804296195507,
+            -0.0018703697714954615,
+            0.0179862342774868,
+            0.020648520439863205,
+            -0.03581002354621887,
+            -0.0014416963094845414,
+            0.0012826359597966075,
+            -0.0005392259918153286,
+            0.008826159872114658,
+            0.012075052596628666,
+            0.023211536929011345,
+            -0.0138619439676404,
+            -0.0368388369679451,
+            0.008762986399233341,
+            0.023644722998142242,
+            -0.0022990431170910597,
+            -0.023446178063750267,
+            -0.0020858345087617636,
+            0.00829821452498436,
+            0.020540224388241768,
+            -0.0012905325274914503,
+            0.008059060201048851,
+            0.0077702696435153484,
+            -0.017210109159350395,
+            -0.009999371133744717,
+            -0.005766785237938166,
+            -0.015332970768213272,
+            -0.027778036892414093,
+            0.010775495320558548,
+            0.005978866014629602,
+            0.01911432109773159,
+            -0.022868597880005836,
+            -0.021370496600866318,
+            -0.021911978721618652,
+            0.0009171354467980564,
+            0.006926459725946188,
+            -0.020973410457372665,
+            0.0014687704388052225,
+            -0.010766470804810524,
+            0.025792602449655533,
+            -0.013636326417326927,
+            0.0014394401805475354,
+            0.0020486076828092337,
+            0.028319519013166428,
+            0.002041839063167572,
+            0.008401998318731785,
+            -0.020504126325249672,
+            -0.0035106095019727945,
+            0.07266691327095032,
+            -0.013943166472017765,
+            -0.0045258887112140656,
+            -0.01175016351044178,
+            0.017020590603351593,
+            -0.01009864266961813,
+            0.0036707979161292315,
+            0.04772263020277023,
+            -0.014403426088392735,
+            0.004473996814340353,
+            0.021911978721618652,
+            0.008415536023676395,
+            0.000976360053755343,
+            -0.00967448204755783,
+            -0.012932399287819862,
+            -0.0023464227560907602,
+            -0.017697442322969437,
+            0.02301299385726452,
+            -0.021081706508994102,
+            0.027074109762907028,
+            -0.015784205868840218,
+            -0.034131426364183426,
+            0.01712888665497303,
+            0.029240038245916367,
+            -0.015594687312841415,
+            -0.009909124113619328,
+            -0.000990461208857596,
+            0.010983063839375973,
+            0.035629529505968094,
+            0.014448549598455429,
+            0.01887967810034752,
+            0.005320062395185232,
+            -0.003212794428691268,
+            0.0018410393968224525,
+            0.02012508735060692,
+            0.015892501920461655,
+            -0.0013345279730856419,
+            0.0064707123674452305,
+            -0.003801656188443303,
+            0.02548576146364212,
+            -0.014845636673271656,
+            -0.02846391312777996,
+            -0.00235995976254344,
+            -0.026370182633399963,
+            -0.007576238363981247,
+            0.029438581317663193,
+            -0.009448864497244358,
+            0.012445065192878246,
+            0.025431614369153976,
+            -0.019475309178233147,
+            -0.027326801791787148,
+            -0.02117195352911949,
+            0.01729133166372776,
+            -0.005915693007409573,
+            0.005288476124405861,
+            0.0017293586861342192,
+            0.015892501920461655,
+            -0.016118120402097702,
+            -0.01514345221221447,
+            0.0020294301211833954,
+            -0.020829014480113983,
+            0.024150105193257332,
+            0.018365271389484406,
+            -0.004354419186711311,
+            0.008492245338857174,
+            0.019728001207113266,
+            -0.010549877770245075,
+            0.0012713550822809339,
+            -0.017905011773109436,
+            -0.004877852275967598,
+            -0.0004585677233990282,
+            0.009936198592185974,
+            -0.02458329126238823,
+            -0.007707096636295319,
+            -0.004516863729804754,
+            -0.00182637432590127,
+            0.01522467378526926,
+            -0.015621760860085487,
+            -0.0004402362974360585,
+            0.008550906553864479,
+            0.0022144364193081856,
+            -0.017065713182091713,
+            -0.0015274309553205967,
+            0.00039313864544965327,
+            -0.011154533363878727,
+            0.0017169497441500425,
+            -0.010279136709868908,
+            -0.023879364132881165,
+            -0.004006968345493078,
+            -0.01141624990850687,
+            -0.0045326570980250835,
+            0.005789346992969513,
+            -0.00901567842811346,
+            0.0007039268384687603,
+            0.020666569471359253,
+            0.016316663473844528,
+            0.009304468519985676,
+            -0.029149791225790977,
+            0.003840011078864336,
+            -0.0037001282908022404,
+            -0.0027412534691393375,
+            -0.01692131906747818,
+            0.0030210192780941725,
+            -0.023536425083875656,
+            0.005089932586997747,
+            -0.018275024369359016,
+            0.014746365137398243,
+            0.005802884232252836,
+            -0.0032579179387539625,
+            -0.0006774167995899916,
+            -0.0002590371877886355,
+            -0.0010288162156939507,
+            0.0007428458775393665,
+            0.01675887405872345,
+            0.050430040806531906,
+            0.0032082819379866123,
+            -0.034456316381692886,
+            -0.004128801636397839,
+            0.006813650950789452,
+            0.0019380550365895033,
+            0.0089931171387434,
+            0.012634584680199623,
+            0.04021407663822174,
+            -0.03732617199420929,
+            0.01564883440732956,
+            -0.021298298612236977,
+            0.005396772641688585,
+            -0.014177808538079262,
+            0.0049229757860302925,
+            -0.025088675320148468,
+            0.01009864266961813,
+            0.012237497605383396,
+            -0.0019910752307623625,
+            -0.005247864872217178,
+            -0.015576637350022793,
+            0.016975466161966324,
+            0.010937940329313278,
+            -0.019998742267489433,
+            -0.009218734689056873,
+            0.001989946933463216,
+            -0.03416752442717552,
+            -0.01618129201233387,
+            -0.005198229104280472,
+            -0.009710581041872501,
+            0.0042867339216172695,
+            -0.00875847414135933,
+            0.012327744625508785,
+            0.013852919451892376,
+            -0.008031985722482204,
+            -0.002881136490032077,
+            0.014683191664516926,
+            -0.003145109163597226,
+            0.0037204339168965816,
+            0.00328047969378531,
+            0.021803682669997215,
+            -0.03490755334496498,
+            0.019132370129227638,
+            -0.003659517038613558,
+            -0.0015939881559461355,
+            -0.008677251636981964,
+            -0.0006633157026953995,
+            -0.022796399891376495,
+            -0.007711608894169331,
+            0.004473996814340353,
+            -0.005234327632933855,
+            0.0025539910420775414,
+            -0.0190240740776062,
+            0.021893929690122604,
+            0.010567927733063698,
+            0.008388461545109749,
+            -0.021081706508994102,
+            -0.005302012898027897,
+            -0.014096586033701897,
+            -0.01786891184747219,
+            0.007549164351075888,
+            -0.013744622468948364,
+            0.0009086747886613011,
+            0.004415336064994335,
+            0.016406910493969917,
+            0.0015285591362044215,
+            0.003025531768798828,
+            0.009069826453924179,
+            -0.015757132321596146,
+            -0.03317480906844139,
+            0.008478708565235138,
+            -0.0101347416639328,
+            0.026695072650909424,
+            0.0256662555038929,
+            0.015576637350022793,
+            0.009493988007307053,
+            0.006430101115256548,
+            -0.00327822333201766,
+            -0.03535878658294678,
+            -0.023283734917640686,
+            0.011921633034944534,
+            -0.020468026399612427,
+            0.017598170787096024,
+            0.004169412888586521,
+            -0.003429387230426073,
+            0.012318720109760761,
+            0.01671374961733818,
+            0.028193172067403793,
+            0.025359416380524635,
+            -0.01657838001847267,
+            0.01687619462609291,
+            -0.019565556198358536,
+            -0.006366928108036518,
+            0.02862635813653469,
+            -0.0040362984873354435,
+            -0.011172582395374775,
+            0.007643923629075289,
+            -0.0095120370388031,
+            0.01918651908636093,
+            0.000988204963505268,
+            -0.010495729744434357,
+            -0.01038743369281292,
+            0.0036346991546452045,
+            -0.016397885978221893,
+            -0.01911432109773159,
+            0.01554956380277872,
+            0.027525344863533974,
+            0.01369949895888567,
+            -0.020305581390857697,
+            0.008663714863359928,
+            0.002093731192871928,
+            0.024962328374385834,
+            0.005667513702064753,
+            0.03718177601695061,
+            -0.0056765382178127766,
+            0.00643461337313056,
+            -0.016452033072710037,
+            -0.005392259918153286,
+            0.01766134425997734,
+            -0.003770069684833288,
+            -0.007467941846698523,
+            -0.007851491682231426,
+            0.0030165070202201605,
+            0.016289589926600456,
+            0.01224652212113142,
+            0.0024321575183421373,
+            -0.01729133166372776,
+            -0.0031044979114085436,
+            -0.010035470128059387,
+            -0.0036031126510351896,
+            -0.0010271240025758743,
+            -0.018681135028600693,
+            -0.0007941738585941494,
+            0.012165299616754055,
+            -0.04537620767951012,
+            -0.001970769604668021,
+            -0.015910550951957703,
+            -0.022056374698877335,
+            -0.024132056161761284,
+            0.023771068081259727,
+            0.009620334021747112,
+            0.013645350933074951,
+            0.007319034077227116,
+            0.01604592241346836,
+            -0.041008252650499344,
+            0.01960165612399578,
+            0.01017084065824747,
+            -0.005644951947033405,
+            0.0022730971686542034,
+            -0.02705606073141098,
+            0.006254119332879782,
+            0.008216992020606995,
+            -0.003864829195663333,
+            0.01551346480846405,
+            -0.026911664754152298,
+            -0.002569784177467227,
+            -0.011885534040629864,
+            0.011831386014819145,
+            0.027344850823283195,
+            -0.006145822815597057,
+            3.934206688427366e-05,
+            -0.015161501243710518,
+            0.005207253620028496,
+            0.002380265388637781,
+            -0.01837429590523243,
+            0.006637669168412685,
+            -0.014204883016645908,
+            -0.0036031126510351896,
+            0.00458229286596179,
+            -0.0012916605919599533,
+            -0.01683107204735279,
+            0.0019301583524793386,
+            -0.0011935170041397214,
+            -0.004171669017523527,
+            -0.005103469826281071,
+            -0.0022674566134810448,
+            0.012643609195947647,
+            -0.008925431407988071,
+            -0.008623103611171246,
+            -0.0035444521345198154,
+            -0.022363213822245598,
+            -0.0011991574428975582,
+            -0.00114275305531919,
+            -0.007070854771882296,
+            0.00494553754106164,
+            0.002865343354642391,
+            -0.004661259241402149,
+            0.01597372442483902,
+            -0.008203455246984959,
+            -0.00552763044834137,
+            0.008230529725551605,
+            -0.02259785681962967,
+            0.005685563199222088,
+            0.012390917167067528,
+            -0.04089995473623276,
+            0.0030480935238301754,
+            -0.004550706595182419,
+            0.028427815064787865,
+            -0.011425274424254894,
+            0.0042799655348062515,
+            0.021460743620991707,
+            -0.009011166170239449,
+            -0.026171639561653137,
+            0.00989107508212328,
+            0.053570639342069626,
+            0.004886876791715622,
+            -0.0008945736917667091,
+            -0.003650492522865534,
+            0.00023027094721328467,
+            -0.011975781060755253,
+            -0.0064797368831932545,
+            -0.014159759506583214,
+            0.016984490677714348,
+            0.005112494342029095,
+            -0.015784205868840218,
+            -0.020052891224622726,
+            -0.015558588318526745,
+            -0.013148992322385311,
+            0.023608623072504997,
+            -0.010585976764559746,
+            -0.01579323038458824,
+            0.004232585895806551,
+            0.02259785681962967,
+            0.010306211188435555,
+            -0.00575324846431613,
+            -0.02109975554049015,
+            -0.019096272066235542,
+            0.018067454919219017,
+            0.006303755100816488,
+            0.014331229031085968,
+            0.0024908180348575115,
+            -0.013708524405956268,
+            0.014177808538079262,
+            0.003025531768798828,
+            0.021081706508994102,
+            -0.0005378159112296999,
+            -0.007052805740386248,
+            0.005013222806155682,
+            -0.02456524223089218,
+            0.020666569471359253,
+            -0.008695301599800587,
+            0.013148992322385311,
+            -0.020648520439863205,
+            -0.008623103611171246,
+            0.004480765201151371,
+            -0.0030864484142512083,
+            -0.012634584680199623,
+            0.021496843546628952,
+            0.0024028271436691284,
+            -0.008271140046417713,
+            -0.0032895042095333338,
+            0.0008240682072937489,
+            -0.027778036892414093,
+            -0.001954976236447692,
+            0.006394002120941877,
+            -0.02465548925101757,
+            0.015405167825520039,
+            0.0015533770201727748,
+            -0.014240982010960579,
+            0.004015992861241102,
+            -0.00792820192873478,
+            -0.005274938885122538,
+            0.0003812937065958977,
+            0.013654375448822975,
+            -0.029907865449786186,
+            0.015982748940587044,
+            -0.0031766954343765974,
+            -0.0071295155212283134,
+            0.006935484241694212,
+            0.0018365271389484406,
+            0.00864566583186388,
+            -0.008027473464608192,
+            1.942426388268359e-05,
+            0.017778664827346802,
+            0.0063263168558478355,
+            0.008568955585360527,
+            0.00513054383918643,
+            -0.0031766954343765974,
+            -0.003219562815502286,
+            0.0006644437671639025,
+            0.002129829954355955,
+            -0.020810965448617935,
+            -0.005911180749535561,
+            -0.003986662719398737,
+            0.0014089817414060235,
+            0.01026108767837286,
+            -0.003939283080399036,
+            0.0013593458570539951,
+            -0.02573845349252224,
+            -0.004268684424459934,
+            0.005911180749535561,
+            0.0030187631491571665,
+            0.007580750621855259,
+            0.03028690442442894,
+            0.015396143309772015,
+            0.019258717074990273,
+            0.0020215334370732307,
+            -0.0015240467619150877,
+            -0.011948706582188606,
+            -0.0011399327777326107,
+            0.00792820192873478,
+            -0.00844712182879448,
+            0.009827901609241962,
+            -0.022814448922872543,
+            0.01072134729474783,
+            0.005261402111500502,
+            0.003183464054018259,
+            -0.000386370113119483,
+            -0.014096586033701897,
+            0.012129200622439384,
+            0.021821731701493263,
+            -0.021496843546628952,
+            0.001207054010592401,
+            -0.013780721463263035,
+            -0.00023619340208824724,
+            -0.011569669470191002,
+            -0.03656809777021408,
+            -0.008144794963300228,
+            -0.030665941536426544,
+            -0.00015750927559565753,
+            -0.014033413492143154,
+            -0.0030119947623461485,
+            -0.01079354528337717,
+            -0.010694272816181183,
+            0.001760945189744234,
+            -0.009791802614927292,
+            0.009692531079053879,
+            0.010414507240056992,
+            0.004737969022244215,
+            -0.0007857132004573941,
+            0.011235755868256092,
+            -0.009367641992866993,
+            -0.014872711151838303,
+            -0.010414507240056992,
+            -0.002763815224170685,
+            0.024547193199396133,
+            -0.010766470804810524,
+            -0.014845636673271656,
+            -0.00612326106056571,
+            -0.006430101115256548,
+            -0.011605768464505672,
+            -0.01837429590523243,
+            0.014195858500897884,
+            0.02754339389503002,
+            -0.013690474443137646,
+            0.0023667283821851015,
+            -0.010847693309187889,
+            0.013266313821077347,
+            0.023969611153006554,
+            -0.001973025733605027,
+            -0.0289692971855402,
+            -0.018717234954237938,
+            0.0012521775206550956,
+            -0.011425274424254894,
+            -0.0072287870571017265,
+            -0.01432220358401537,
+            -0.004706382751464844,
+            -0.015206624753773212,
+            0.017769640311598778,
+            -0.00926837045699358,
+            -0.02028753235936165,
+            -0.008456147275865078,
+            -0.0030165070202201605,
+            -0.00600142776966095,
+            -0.008690789341926575,
+            -0.022525658831000328,
+            0.011434298940002918,
+            0.008253091014921665,
+            0.009593259543180466,
+            -0.01567590981721878,
+            0.019673852249979973,
+            0.013383634388446808,
+            -0.024799883365631104,
+            -0.003431643359363079,
+            -0.01464709360152483,
+            -0.0020666569471359253,
+            -0.013843894936144352,
+            -0.011271853931248188,
+            0.0006982863997109234,
+            0.0014394401805475354,
+            0.010802569799125195,
+            0.0024231327697634697,
+            0.022633954882621765,
+            -4.344900844444055e-06,
+            -0.013627301901578903,
+            0.00643461337313056,
+            0.01522467378526926,
+            -0.014538796618580818,
+            0.02806682698428631,
+            0.012914350256323814,
+            -0.017670368775725365,
+            0.015937626361846924,
+            0.013049720786511898,
+            -0.015964699909090996,
+            0.005762272980064154,
+            0.004977123811841011,
+            -0.00854188110679388,
+            -0.01361827738583088,
+            0.018320146948099136,
+            0.01175016351044178,
+            0.021532941609621048,
+            0.001636855537071824,
+            0.012508238665759563,
+            -0.00686779897660017,
+            0.007409281097352505,
+            -0.022200770676136017,
+            0.01985434629023075,
+            -0.00963838305324316,
+            -0.02010703831911087,
+            0.014926859177649021,
+            -0.0026103954296559095,
+            0.012896301224827766,
+            0.011262829415500164,
+            -0.004832728765904903,
+            -0.05097152292728424,
+            -0.01038743369281292,
+            0.011470397934317589,
+            0.003955075982958078,
+            -0.014990031719207764,
+            0.019565556198358536,
+            -0.0036911035422235727,
+            0.030268853530287743,
+            -0.005049321334809065,
+            0.02631603367626667,
+            -0.005256889387965202,
+            0.018735283985733986,
+            -0.015125402249395847,
+            -0.016993515193462372,
+            0.0071295155212283134,
+            0.00026777986204251647,
+            -0.004674796015024185,
+            0.008307239040732384,
+            -0.030918633565306664,
+            -0.010531828738749027,
+            -0.007147565018385649,
+            0.01886162906885147,
+            -0.006177409086376429,
+            0.026604825630784035,
+            0.0019233898492529988,
+            0.009466913528740406,
+            0.017580121755599976,
+            -0.01352803036570549,
+            0.019565556198358536,
+            -0.05479799583554268,
+            -0.0020316862501204014,
+            -0.0044333855621516705,
+            0.027669738978147507,
+            -0.004205511882901192,
+            -0.0012668427079916,
+            0.004088190384209156,
+            3.040549358956923e-07,
+            -0.005017735064029694,
+            0.007661973126232624,
+            -3.456531703704968e-05,
+            0.01361827738583088,
+            -0.027272652834653854,
+            -0.02739899791777134,
+            0.00661961967125535,
+            0.004828216042369604,
+            0.016975466161966324,
+            -0.010306211188435555,
+            -0.001425903057679534,
+            0.017850862815976143,
+            -0.01439440157264471,
+            0.00471991952508688,
+            -0.010495729744434357,
+            -0.016316663473844528,
+            -0.018455518409609795,
+            0.0018771382747218013,
+            0.019493358209729195,
+            -0.006412051618099213,
+            0.005536655429750681,
+            0.001989946933463216,
+            0.013627301901578903,
+            0.0012352562043815851,
+            0.011425274424254894,
+            -0.011777237989008427,
+            0.027705838903784752,
+            -0.009710581041872501,
+            0.015982748940587044,
+            0.004088190384209156,
+            -0.0017146936152130365,
+            0.006168384570628405,
+            -0.007251348812133074,
+            -0.029655175283551216,
+            -0.004467227961868048,
+            -0.010243037715554237,
+            -0.0018342708935961127,
+            -0.02433059923350811,
+            -0.024799883365631104,
+            0.0040430668741464615,
+            0.0030390687752515078,
+            0.00963838305324316,
+            0.016100069507956505,
+            0.022236868739128113,
+            0.007589775137603283,
+            0.008731400594115257,
+            -0.0021726973354816437,
+            0.004106239881366491,
+            -0.020901212468743324,
+            0.008302726782858372,
+            -0.016614478081464767,
+            -0.010576952248811722,
+            -0.0019211337203159928,
+            -0.02597309648990631,
+            0.002477281028404832,
+            -0.008131257258355618,
+            0.007319034077227116,
+            -0.00026369054103270173,
+            0.0021140368189662695,
+            0.016749849542975426,
+            -0.00894348043948412,
+            0.010613051243126392,
+            -0.005089932586997747,
+            0.00327822333201766,
+            -0.004904926288872957,
+            -0.014583920128643513,
+            -0.0011207553325220942,
+            -0.014421476051211357,
+            0.012102127075195312,
+            -0.023139338940382004,
+            -0.0045213764533400536,
+            -0.05512288585305214,
+            -0.006096187047660351,
+            -0.021911978721618652,
+            0.005978866014629602,
+            0.013519004918634892,
+            -0.012273596599698067,
+            -0.01590152643620968,
+            -0.02043192833662033,
+            0.022561758756637573,
+            -0.012038953602313995,
+            -0.0012228472623974085,
+            -0.01059500128030777,
+            0.035467084497213364,
+            -0.030431298539042473,
+            -0.001330015598796308,
+            -0.0032263314351439476,
+            0.008916406892240047,
+            -0.006615107413381338,
+            -0.024294501170516014,
+            -0.005324574653059244,
+            -0.012806054204702377,
+            0.004453691188246012,
+            -0.007661973126232624,
+            -0.017814764752984047,
+            0.005324574653059244,
+            -0.022778350859880447,
+            0.0031947449315339327,
+            0.012562386691570282,
+            -0.006800113711506128,
+            -0.00527945114299655,
+            0.004900414030998945,
+            0.007711608894169331,
+            0.009773753583431244,
+            0.0076303863897919655,
+            -0.0014890760648995638,
+            0.006344366353005171,
+            -0.009385691024363041,
+            -0.009277394972741604,
+            0.012824103236198425,
+            0.02102755755186081,
+            -0.002097115386277437,
+            0.006046551279723644,
+            -0.019385062158107758,
+            -0.011957732029259205,
+            -0.02747119590640068,
+            -0.013546079397201538,
+            -0.02100950852036476,
+            0.0009092388208955526,
+            0.01125380489975214,
+            0.00643461337313056,
+            0.01432220358401537,
+            -0.00019713335495907813,
+            -0.0030526057817041874,
+            0.0016278307884931564,
+            -0.046964555978775024,
+            -0.010215964168310165,
+            0.006380465347319841,
+            -0.005306525621563196,
+            0.009038240648806095,
+            0.020829014480113983,
+            -0.007684534881263971,
+            -0.021984176710247993,
+            0.0028540624771267176,
+            -0.0006937740836292505,
+            -0.0010694273514673114,
+            0.014078537002205849,
+            0.014069512486457825,
+            -0.006678280420601368,
+            0.013654375448822975,
+            0.004363444168120623,
+            0.022399313747882843,
+            -0.029330285266041756,
+            0.008826159872114658,
+            0.011235755868256092,
+            -0.024781834334135056,
+            0.018347222357988358,
+            0.004832728765904903,
+            -0.018094530329108238,
+            0.004155875649303198,
+            0.015712007880210876,
+            0.019709952175617218,
+            -0.002291146432980895,
+            0.012354818172752857,
+            -0.005103469826281071,
+            0.003623418277129531,
+            0.019312864169478416,
+            0.004977123811841011,
+            -0.021081706508994102,
+            -0.017589146271348,
+            -0.011641867458820343,
+            0.018500640988349915,
+            -0.005969841033220291,
+            0.012390917167067528,
+            0.018365271389484406,
+            -0.0007405896903946996,
+            -0.009909124113619328,
+            0.022706152871251106,
+            -0.0006666999543085694,
+            0.006831700447946787,
+            0.0014180064899846911,
+            0.0004551834717858583,
+            -0.019006025046110153,
+            0.000448414939455688,
+            0.008153819479048252,
+            0.012815078720450401,
+            0.015919575467705727,
+            0.01398828998208046,
+            -0.01597372442483902,
+            -0.003149621421471238,
+            -0.001759817125275731,
+            -0.012463115155696869,
+            -0.023373981937766075,
+            0.014953932724893093,
+            -0.006588033400475979,
+            0.018085505813360214,
+            -0.005906668026000261,
+            -0.001572554581798613,
+            -0.01303167175501585,
+            0.018265999853610992,
+            0.001610909472219646,
+            0.01109135989099741,
+            -0.00012359612446743995,
+            -0.002317092614248395,
+            0.00717915128916502,
+            0.0005073575302958488,
+            -0.017228158190846443,
+            -0.013248263858258724,
+            0.013455832377076149,
+            0.005780322477221489,
+            -0.0029127229936420918,
+            0.02216467075049877,
+            -0.0010936812032014132,
+            0.024366699159145355,
+            0.019836297258734703,
+            0.01277897972613573,
+            -0.006520348135381937,
+            0.008456147275865078,
+            0.026081392541527748,
+            -0.021659286692738533,
+            0.013943166472017765,
+            0.004891389049589634,
+            -0.02061242237687111,
+            -0.03183915093541145,
+            0.0007620233809575438,
+            -0.0006949021480977535,
+            0.010495729744434357,
+            -0.02167733758687973,
+            -0.00934959203004837,
+            -0.015964699909090996,
+            -0.009909124113619328,
+            0.008388461545109749,
+            0.00376330129802227,
+            0.008587004616856575,
+            0.008546394295990467,
+            -0.01327533833682537,
+            0.0032105380669236183,
+            0.0165513064712286,
+            0.0033210907131433487,
+            -0.012255546636879444,
+            0.014006339013576508,
+            0.03220013901591301,
+            0.005450920667499304,
+            -0.031063027679920197,
+            0.016948392614722252,
+            0.0012792516499757767,
+            0.017697442322969437,
+            -0.004329601302742958,
+            -0.007968813180923462,
+            -0.008442609570920467,
+            0.01472831517457962,
+            -0.03218208998441696,
+            -0.0022753532975912094,
+            0.018933827057480812,
+            0.003181207925081253,
+            0.008559931069612503,
+            0.009430814534425735,
+            0.001052506035193801,
+            -0.017905011773109436,
+            0.004864315036684275,
+            -0.014403426088392735,
+            -0.0007287447806447744,
+            0.011795287020504475,
+            -0.01360925193876028,
+            -0.009191660210490227,
+            0.014331229031085968,
+            -0.00767550989985466,
+            -0.02821122296154499,
+            0.03292211890220642,
+            -0.013347536325454712,
+            -0.008013936690986156,
+            0.0372539758682251,
+            0.004769555758684874,
+            -0.005789346992969513,
+            0.011073310859501362,
+            0.015712007880210876,
+            -0.007314521819353104,
+            0.004113008268177509,
+            0.018320146948099136,
+            -0.019980693235993385,
+            -0.011280879378318787,
+            0.018482591956853867,
+            0.03019665740430355,
+            -0.0013221190311014652,
+            0.0007479222840629518,
+            -0.017589146271348,
+            -0.015423217788338661,
+            -0.005455432925373316,
+            0.008361387066543102,
+            0.01804940588772297,
+            -0.012544337660074234,
+            -0.01733645424246788,
+            0.02373497001826763,
+            0.0045236325822770596,
+            0.005062858574092388,
+            0.0071340277791023254,
+            0.022525658831000328,
+            -0.013049720786511898,
+            -0.025594057515263557,
+            0.014656118117272854,
+            -0.010360359214246273,
+            -0.01782378926873207,
+            -0.017760615795850754,
+            0.014087561517953873,
+            -0.012634584680199623,
+            -0.015531513839960098,
+            0.021641237661242485,
+            0.0013367841020226479,
+            -0.0032060258090496063,
+            0.02225491777062416,
+            0.010748421773314476,
+            -0.0167408250272274,
+            -0.03783155605196953,
+            -0.021370496600866318,
+            -0.00038298583240248263,
+            0.009584235027432442,
+            0.006538397632539272,
+            0.009029215201735497,
+            -0.025449663400650024,
+            -0.0029826643876731396,
+            -0.025052575394511223,
+            -0.0005820933729410172,
+            -0.006849749479442835,
+            -0.010694272816181183,
+            -0.004122033249586821,
+            -0.00376330129802227,
+            0.021695386618375778,
+            0.01647910848259926,
+            -0.0009487218922004104,
+            0.006344366353005171,
+            -0.002160288393497467,
+            -0.008623103611171246,
+            -0.011578693985939026,
+            0.01646105945110321,
+            0.016903268173336983,
+            -0.02017923630774021,
+            -0.0028224759735167027,
+            0.011190632358193398,
+            0.0026690559461712837,
+            -0.013509980402886868,
+            -0.019096272066235542,
+            -0.009493988007307053,
+            -0.009173611178994179,
+            -0.024547193199396133,
+            -0.002287762239575386,
+            -0.006023989524692297,
+            -0.025449663400650024,
+            0.012842152267694473,
+            0.04230780899524689,
+            -0.01606397144496441,
+            -0.0032285875640809536,
+            0.006664743181318045,
+            -0.0028292443603277206,
+            0.0072107380256056786,
+            0.005311037879437208,
+            -0.008975067175924778,
+            0.02052217535674572,
+            -0.004376980941742659,
+            -0.004986148327589035,
+            -0.002004612237215042,
+            0.005148593336343765,
+            -0.011389175429940224,
+            -0.02458329126238823,
+            -0.025016477331519127,
+            0.006935484241694212,
+            -0.0006949021480977535,
+            0.0011822361266240478,
+            -0.009087876416742802,
+            0.0012871483340859413,
+            -0.01749889925122261,
+            0.009137512184679508,
+            0.012941424734890461,
+            -0.0024592315312474966,
+            -0.0045258887112140656,
+            -0.013519004918634892,
+            -0.011136483401060104,
+            0.005771297495812178,
+            0.017047664150595665,
+            0.006127773318439722,
+            -0.01800428330898285,
+            0.0007157717482186854,
+            0.012661658227443695,
+            0.02680336870253086,
+            -0.023482277989387512,
+            0.013717548921704292,
+            0.012733856216073036,
+            0.001949335914105177,
+            0.0020666569471359253,
+            0.019728001207113266,
+            -0.007007681764662266,
+            -0.004864315036684275,
+            0.00606008805334568,
+            -0.015242723748087883,
+            0.0029578465037047863,
+            0.026099441573023796,
+            -0.008117720484733582,
+            0.014286105521023273,
+            -0.05353453755378723,
+            -0.010405482724308968,
+            0.0009977937443181872,
+            -0.016352761536836624,
+            -0.0055817789398133755,
+            0.024095958098769188,
+            -0.00013755621330346912,
+            -0.003215050557628274,
+            -0.0055772666819393635,
+            0.012237497605383396,
+            -0.02176758460700512,
+            -0.04353516921401024,
+            0.0018748820293694735,
+            -0.01590152643620968,
+            -0.016361786052584648,
+            0.00014249159721657634,
+            -0.018933827057480812,
+            0.007454404607415199,
+            -0.00963838305324316,
+            -0.008496757596731186,
+            -0.014665142633020878,
+            0.027290701866149902,
+            -0.000774996355175972,
+            -0.0012668427079916,
+            -0.01812160387635231,
+            -0.01184041053056717,
+            -0.015784205868840218,
+            -0.011849435046315193,
+            -0.008749449625611305,
+            -0.0028247321024537086,
+            -0.008226016536355019,
+            -0.0034451803658157587,
+            0.0045258887112140656,
+            0.017444750294089317,
+            -0.020413879305124283,
+            -0.02035973034799099,
+            -0.0028946734964847565,
+            -0.01009864266961813,
+            0.004832728765904903,
+            0.0138619439676404,
+            0.015125402249395847,
+            -0.0054328711703419685,
+            0.007359645329415798,
+            -0.007846979424357414,
+            0.0016571611631661654,
+            -0.010243037715554237,
+            0.03862572833895683,
+            0.0027322289533913136,
+            0.0008325288654305041,
+            -0.0019042124040424824,
+            -8.156604053510819e-06,
+            0.005446408409625292,
+            -0.004977123811841011,
+            -0.017507923766970634,
+            0.010549877770245075,
+            -0.02326568402349949,
+            0.021406596526503563,
+            -0.010279136709868908,
+            0.01017084065824747,
+            -0.005762272980064154,
+            -0.026586774736642838,
+            -0.002233613980934024,
+            0.024781834334135056,
+            0.0010919891064986587,
+            0.009227759204804897,
+            -0.01782378926873207,
+            -0.008636641316115856,
+            -0.0030751675367355347,
+            -0.022291017696261406,
+            -0.009584235027432442,
+            -0.00901567842811346,
+            0.002319348743185401,
+            -0.010576952248811722,
+            0.014935883693397045,
+            0.017580121755599976,
+            0.005053833592683077,
+            -0.006831700447946787,
+            0.009710581041872501,
+            -0.02979956939816475,
+            0.008781036362051964,
+            -0.003436155617237091,
+            -0.005473482422530651,
+            -0.019583605229854584,
+            0.019421162083745003,
+            0.0001576502836542204,
+            -0.0027389973402023315,
+            -0.002817963482812047,
+            0.02100950852036476,
+            0.008361387066543102,
+            0.009683506563305855,
+            -0.005468970164656639,
+            -0.004638697486370802,
+            -0.01221042312681675,
+            0.003219562815502286,
+            -0.00019558223721105605,
+            -0.0002899185928981751,
+            -0.0016176780918613076,
+            0.005320062395185232,
+            -0.010486705228686333,
+            -0.025142822414636612,
+            0.017192060127854347,
+            -0.00894348043948412,
+            -0.006664743181318045,
+            0.017155960202217102,
+            -0.01514345221221447,
+            0.0011213193647563457,
+            0.004088190384209156,
+            0.0004159824165981263,
+            0.004162644501775503,
+            0.004760530777275562,
+            0.008884820155799389,
+            -0.0063082673586905,
+            -0.020558273419737816,
+            0.011235755868256092,
+            -0.013482906855642796,
+            -0.005139568354934454,
+            0.021911978721618652,
+            -0.006529372651129961,
+            -0.002199771348387003,
+            -0.010486705228686333,
+            -0.02183978073298931,
+            0.01481856219470501,
+            -0.02689361572265625,
+            0.0016481364145874977,
+            -0.0029804082587361336,
+            0.01588347740471363,
+            -0.011939682066440582,
+            0.02368082106113434,
+            -0.006655718665570021,
+            -0.006263143848627806,
+            0.0021208052057772875,
+            -0.0260633435100317,
+            0.008198942989110947,
+            -0.035449035465717316,
+            0.004338625818490982,
+            -0.01125380489975214,
+            0.029258087277412415,
+            0.018645036965608597,
+            -0.006425588857382536,
+            0.011398199945688248,
+            -0.02904149517416954,
+            0.004214536398649216,
+            0.01411463599652052,
+            0.0038783662021160126,
+            -0.0040295301005244255,
+            0.007449892349541187,
+            -0.010865742340683937,
+            -0.023536425083875656,
+            -0.02954687736928463,
+            0.001267970772460103,
+            -0.03481730446219444,
+            0.0013785234186798334,
+            0.0012916605919599533,
+            -0.016912294551730156,
+            0.010324260219931602,
+            -0.006759502459317446,
+            -0.020666569471359253,
+            0.017228158190846443,
+            0.0031157787889242172,
+            -0.012878251262009144,
+            -0.0042777094058692455,
+            0.006750477943569422,
+            0.02160513959825039,
+            0.014638068154454231,
+            -0.009737654589116573,
+            0.0008404254913330078,
+            0.010585976764559746,
+            -0.006389489863067865,
+            -0.010197914205491543,
+            -0.022796399891376495,
+            -0.00883067212998867,
+            0.004004712216556072,
+            0.006443637888878584,
+            -0.006664743181318045,
+            0.02001679129898548,
+            0.011226730421185493,
+            0.0012205911334604025,
+            0.01352803036570549,
+            -0.01389804296195507,
+            0.018338195979595184,
+            -0.0031090101692825556,
+            -0.014629043638706207,
+            0.0019290302880108356,
+            -0.03593636676669121,
+            0.005965328775346279,
+            -0.006213508080691099,
+            0.025756502524018288,
+            -0.01439440157264471,
+            -0.0017530486220493913,
+            -0.018455518409609795,
+            -0.0025336854159832,
+            0.015684934332966805,
+            -0.012598485685884953,
+            -0.011280879378318787,
+            0.013537054881453514,
+            -0.013176066800951958,
+            -0.0020305581856518984,
+            0.016930343583226204,
+            -0.010089618153870106,
+            0.01233676914125681,
+            -0.007188176270574331,
+            -0.00027215119916945696,
+            -0.00182637432590127,
+            -0.0012566898949444294,
+            -0.004566499963402748,
+            -0.0013412964763119817,
+            0.0028337568510323763,
+            -0.020143138244748116,
+            0.029582977294921875,
+            -0.013509980402886868,
+            -0.010766470804810524,
+            0.004250635392963886,
+            -0.003032300155609846,
+            0.021875880658626556,
+            -0.011650891974568367,
+            0.00391220860183239,
+            0.000977488118223846,
+            -0.010342310182750225,
+            -0.008167356252670288,
+            -0.004559731110930443,
+            0.013013621792197227,
+            0.026279935613274574,
+            -0.019980693235993385,
+            0.009394715540111065,
+            0.00997229665517807,
+            -0.02350032702088356,
+            -0.01869918406009674,
+            -0.004543938208371401,
+            -0.013446807861328125,
+            0.01788696087896824,
+            -0.00039454875513911247,
+            -0.005658488720655441,
+            0.0036843351554125547,
+            0.005035784561187029,
+            -0.014340253546833992,
+            -0.000682493147905916,
+            -0.003898671828210354,
+            0.004345394670963287,
+            -0.006281193345785141,
+            0.011768212541937828,
+            0.0006520348251797259,
+            0.02059437334537506,
+            -0.00926837045699358,
+            -0.014132685028016567,
+            0.0064842491410672665,
+            -0.00997229665517807,
+            0.009882049635052681,
+            -0.021208051592111588,
+            0.0010152790928259492,
+            0.0074724541045725346,
+            -0.005098957102745771,
+            0.004841753281652927,
+            0.009484962560236454,
+            -0.02739899791777134,
+            0.005969841033220291,
+            0.005482506938278675,
+            -0.0015691702719777822,
+            0.016497157514095306,
+            0.009503012523055077,
+            -0.0007095672772265971,
+            0.0006988504319451749,
+            -0.014719290658831596,
+            0.019385062158107758,
+            0.006921947468072176,
+            0.019096272066235542,
+            0.006773039698600769,
+            0.015522489324212074,
+            -0.00013586408749688417,
+            -0.005622390192002058,
+            0.006782064214348793,
+            -0.006064600311219692,
+            -0.004595830105245113,
+            0.03557537868618965,
+            -0.0006255247280932963,
+            0.012300670146942139,
+            0.006696329917758703,
+            -0.00030599383171647787,
+            0.0056765382178127766,
+            0.0120028555393219,
+            0.0012454090174287558,
+            -0.017697442322969437,
+            0.011235755868256092,
+            -0.015991773456335068,
+            -0.019637754186987877,
+            0.0019380550365895033,
+            0.009394715540111065,
+            -0.00780185591429472,
+            -0.0026081393007189035,
+            0.026099441573023796,
+            0.008550906553864479,
+            0.009358617477118969,
+            0.003530915128067136,
+            -0.007901127450168133,
+            -0.002162544522434473,
+            0.005956304259598255,
+            0.000220682195504196,
+            0.001759817125275731,
+            -0.004805654287338257,
+            -0.02581065148115158,
+            0.003303041448816657,
+            -0.005446408409625292,
+            -0.00340456934645772,
+            -0.0015026130713522434,
+            0.018383320420980453,
+            -0.01352803036570549,
+            0.01544126681983471,
+            0.006033014040440321,
+            -0.004113008268177509,
+            -0.014304154552519321,
+            0.0062315575778484344,
+            0.004081421997398138,
+            0.020883163437247276,
+            -0.001704540802165866,
+            -0.019746050238609314,
+            -0.0227603018283844,
+            0.003797143930569291,
+            -5.8026020269608125e-05,
+            -0.0034880477469414473,
+            0.0022945308592170477,
+            -0.002249407349154353,
+            0.008216992020606995,
+            -0.0034857916180044413,
+            -0.006673768162727356,
+            0.002287762239575386,
+            -0.006154847331345081,
+            0.02017923630774021,
+            -0.002326117129996419,
+            -0.0006571111734956503,
+            0.0023667283821851015,
+            -0.004013736732304096,
+            0.001205925946123898,
+            -0.002102755941450596,
+            0.016794972121715546,
+            -0.017192060127854347,
+            -0.00393702695146203,
+            -0.009078850969672203,
+            -0.017480850219726562,
+            0.0014721547486260533,
+            -0.009259345009922981,
+            0.006664743181318045,
+            0.013943166472017765,
+            0.014764414168894291,
+            -0.004990661051124334,
+            -0.0017846350092440844,
+            0.01918651908636093,
+            0.011677965521812439,
+            0.0010474296286702156,
+            0.011353076435625553,
+            0.0009002141305245459,
+            -0.010992088355123997,
+            -0.0006266528507694602,
+            -0.010910865850746632,
+            -0.007607824634760618,
+            0.023554475978016853,
+            0.02274225279688835,
+            0.012318720109760761,
+            0.0025742966681718826,
+            -0.0023486788850277662,
+            -0.0029285161290317774,
+            0.006294730585068464,
+            0.000976360053755343,
+            -0.011885534040629864,
+            0.0021456230897456408,
+            0.0006243966636247933,
+            -0.022615905851125717,
+            0.013925116509199142,
+            -0.02929418720304966,
+            0.0240057110786438,
+            -0.0018365271389484406,
+            0.009187147952616215,
+            0.019547507166862488,
+            -0.0011280878679826856,
+            0.013464856892824173,
+            0.01096501387655735,
+            0.0019414392299950123,
+            -0.02465548925101757,
+            -0.00811320822685957,
+            -0.010378408245742321,
+            -0.005256889387965202,
+            0.006989632733166218,
+            0.009114949963986874,
+            0.000154830064275302,
+            0.0030841922853142023,
+            -0.0021241893991827965,
+            0.001891803345642984,
+            -0.01969190314412117,
+            0.011903583072125912,
+            0.0022178208455443382,
+            -0.010215964168310165,
+            -0.023933513090014458,
+            -0.005703612230718136,
+            0.009864000603556633,
+            0.0022099241614341736,
+            -0.003519634250551462,
+            0.008365900255739689,
+            -0.0063308291137218475,
+            -0.0008658074657432735,
+            0.00024973045219667256,
+            -0.003395544597879052,
+            0.0016526487888768315,
+            -0.005866057239472866,
+            0.005117006599903107,
+            0.005766785237938166,
+            -0.0008415535558015108,
+            0.008230529725551605,
+            -0.01046865526586771,
+            0.013952190987765789,
+            -0.011407224461436272,
+            0.005613365210592747,
+            -0.008537368848919868,
+            0.006831700447946787,
+            0.008893844671547413,
+            0.005861544515937567,
+            0.013148992322385311,
+            -0.0003739611420314759,
+            -0.0013164785923436284,
+            -0.016217391937971115,
+            -0.03485340252518654,
+            0.006366928108036518,
+            -0.0009064186015166342,
+            0.006588033400475979,
+            -0.02160513959825039,
+            0.0050809080712497234,
+            0.012580436654388905,
+            -0.017255231738090515,
+            0.0010169711895287037,
+            0.005784834735095501,
+            0.015910550951957703,
+            -0.011154533363878727,
+            0.000819555833004415,
+            0.007043780758976936,
+            0.018825531005859375,
+            0.0031202910467982292,
+            0.004189718514680862,
+            -0.02225491777062416,
+            -0.012670683674514294,
+            0.015341995283961296,
+            -0.030738139525055885,
+            0.008641153573989868,
+            0.020702669396996498,
+            -0.009286419488489628,
+            0.03109912760555744,
+            0.01788696087896824,
+            -0.013573153875768185,
+            -0.0073731825686991215,
+            -0.008573467843234539,
+            0.022453460842370987,
+            -0.0067324284464120865,
+            0.009683506563305855,
+            -0.014620019122958183,
+            0.011037211865186691,
+            0.0046838209964334965,
+            0.005202741362154484,
+            0.010983063839375973,
+            -0.027128256857395172,
+            0.007702583912760019,
+            4.857828389503993e-05,
+            -0.026586774736642838,
+            0.00804552249610424,
+            -0.005771297495812178,
+            0.020919261500239372,
+            -0.001577066839672625,
+            0.012607510201632977,
+            0.024962328374385834,
+            0.00027088209753856063,
+            -0.012381892651319504,
+            -0.0073686703108251095,
+            0.009900099597871304,
+            -0.012860202230513096,
+            0.028355617076158524,
+            -0.00042472509085200727,
+            0.0009227758855558932,
+            -0.005049321334809065,
+            0.008352362550795078,
+            -0.017598170787096024,
+            0.004701870493590832,
+            -0.013970240019261837,
+            -0.016731800511479378,
+            0.0011574181262403727,
+            0.012147250585258007,
+            -0.0006035270052962005,
+            0.013257289305329323,
+            0.011073310859501362,
+            0.0040385546162724495,
+            0.014195858500897884,
+            0.003397800726816058,
+            0.0023283734917640686,
+            0.022309066727757454,
+            0.010450606234371662,
+            0.001882778713479638,
+            -0.0018940595909953117,
+            -0.0022798655554652214,
+            0.03169475868344307,
+            -6.616940663661808e-05,
+            -0.004652234259992838,
+            0.010874766856431961,
+            0.01432220358401537,
+            0.006384977605193853,
+            0.01088379230350256,
+            0.015712007880210876,
+            0.0012713550822809339,
+            0.003219562815502286,
+            0.01046865526586771,
+            0.002475024899467826,
+            0.01240896712988615,
+            0.003131571924313903,
+            -0.006493274122476578,
+            -0.0038084248080849648,
+            0.006218020338565111,
+            0.013284362852573395,
+            -0.01000839564949274,
+            0.017489874735474586,
+            0.009493988007307053,
+            0.00943983905017376,
+            -0.005044809076935053,
+            0.007070854771882296,
+            0.012950449250638485,
+            0.0034384119790047407,
+            -0.007291960064321756,
+            -0.011389175429940224,
+            0.002443438395857811,
+            -0.027759985998272896,
+            0.012363843619823456,
+            0.0014349278062582016,
+            -0.016822047531604767,
+            -0.007314521819353104,
+            -0.003946051467210054,
+            -0.015919575467705727,
+            0.0032872480805963278,
+            -0.002628444693982601,
+            0.026117490604519844,
+            -0.0007608952582813799,
+            -0.02532331645488739,
+            -0.008411023765802383,
+            0.016993515193462372,
+            -0.007341595832258463,
+            0.005153105594217777,
+            0.012255546636879444,
+            0.028084876015782356,
+            0.017020590603351593,
+            -0.0006706482381559908,
+            -0.003896415699273348,
+            -0.004904926288872957,
+            -0.003957332111895084,
+            0.010775495320558548,
+            -0.0159286018460989,
+            -0.008591517806053162,
+            -0.01647910848259926,
+            -0.016587404534220695,
+            0.0023712406400591135,
+            0.009575209580361843,
+            -0.004505583085119724,
+            -0.007201713044196367,
+            -0.0020192773081362247,
+            0.001828630454838276,
+            -0.003657260909676552,
+            0.02043192833662033,
+            -0.01860893703997135,
+            -0.02781413495540619,
+            0.0031586461700499058,
+            -0.0038851348217576742,
+            0.0073776948265731335,
+            0.006660230923444033,
+            0.014656118117272854,
+            -0.002908210502937436,
+            -0.012616534717381,
+            0.04371566325426102,
+            -0.009277394972741604,
+            0.017074737697839737,
+            -0.006750477943569422,
+            0.016325687989592552,
+            0.015062229707837105,
+            0.0060871620662510395,
+            0.0054238466545939445,
+            0.019258717074990273,
+            -0.01647910848259926,
+            -0.002655518939718604,
+            0.007639411371201277,
+            0.009864000603556633,
+            0.02498037740588188,
+            0.008474196307361126,
+            0.0010474296286702156,
+            -0.02044997736811638,
+            -0.003066142788156867,
+            -0.019258717074990273,
+            -0.0009453376405872405,
+            -0.015260772779583931,
+            -0.003970869351178408,
+            0.01233676914125681,
+            -0.02218271978199482,
+            -0.013248263858258724,
+            -0.02160513959825039,
+            -0.006375952623784542,
+            0.009818877093493938,
+            0.0006221404764801264,
+            0.018437469378113747,
+            0.00897957943379879,
+            0.00789661519229412,
+            0.007869540713727474,
+            0.015161501243710518,
+            -0.006014964543282986,
+            0.002693873830139637,
+            0.00550055643543601,
+            0.0009013422532007098,
+            0.011578693985939026,
+            -0.00045151717495173216,
+            -0.013762672431766987,
+            0.012580436654388905,
+            0.0045281448401510715,
+            0.009236783720552921,
+            -0.02582870051264763,
+            -0.023789117112755775,
+            0.01112745888531208,
+            0.0067369407042860985,
+            0.004092703107744455,
+            -0.013356560841202736,
+            0.010658174753189087,
+            -0.003269198816269636,
+            -0.006466199643909931,
+            -0.005762272980064154,
+            0.0036346991546452045,
+            -0.016993515193462372,
+            -0.007783806417137384,
+            0.00637144036591053,
+            0.01664155349135399,
+            0.000557275430765003,
+            0.014881735667586327,
+            0.010288161225616932,
+            0.02283249981701374,
+            0.012733856216073036,
+            -0.026099441573023796,
+            -0.002752534346655011,
+            0.007143052760511637,
+            0.002515635918825865,
+            -0.023608623072504997,
+            -0.009340567514300346,
+            -0.002231357851997018,
+            -0.0010158431250602007,
+            0.026947762817144394,
+            -0.0025449662934988737,
+            0.011362100951373577,
+            -0.009295444004237652,
+            -0.004399542696774006,
+            -0.018843580037355423,
+            0.031008880585432053,
+            -0.003934770356863737,
+            0.005793859250843525,
+            -0.007607824634760618,
+            -0.013762672431766987,
+            -0.012165299616754055,
+            -0.004742481280118227,
+            0.00123638438526541,
+            0.06407539546489716,
+            -0.017562072724103928,
+            0.010477680712938309,
+            -0.008456147275865078,
+            0.006046551279723644,
+            0.022236868739128113,
+            0.010279136709868908,
+            0.0060826498083770275,
+            -0.011163557879626751,
+            -0.01472831517457962,
+            -0.004823703784495592,
+            -0.002286634175106883,
+            0.008451634086668491,
+            0.005157617852091789,
+            -0.0058344705030322075,
+            -0.02144269458949566,
+            -0.025016477331519127,
+            -0.023554475978016853,
+            -0.0012172068236395717,
+            -0.01022498868405819,
+            -0.003677566535770893,
+            0.00041711051017045975,
+            0.0013548335991799831,
+            0.008794573135674,
+            0.01960165612399578,
+            -0.007643923629075289,
+            -0.02391546405851841,
+            -0.027850233018398285,
+            0.014574895612895489,
+            -0.04227171093225479,
+            0.01860893703997135,
+            0.0004881800268776715,
+            -0.027453146874904633,
+            -0.012643609195947647,
+            0.013049720786511898,
+            0.017092788591980934,
+            -0.010901841334998608,
+            0.021731484681367874,
+            0.010685248300433159,
+            -0.011470397934317589,
+            0.0029307722579687834,
+            0.022633954882621765,
+            0.026586774736642838,
+            0.012634584680199623,
+            0.0032060258090496063,
+            0.01853674091398716,
+            0.013221190311014652,
+            -0.018013307824730873,
+            0.007698071654886007,
+            0.01567590981721878,
+            -0.012463115155696869,
+            0.008122232742607594,
+            -0.010432557202875614,
+            0.012986548244953156,
+            -0.005351649131625891,
+            -0.001114550861530006,
+            0.030665941536426544,
+            -0.028608309105038643,
+            0.021208051592111588,
+            -0.031171325594186783,
+            0.008000398986041546,
+            -0.004990661051124334,
+            -0.0004560295201372355,
+            0.010712322778999805,
+            -0.051513005048036575,
+            0.03245283290743828,
+            0.018717234954237938,
+            -5.53468125872314e-05,
+            -0.026297984644770622,
+            -0.01312191877514124,
+            0.021460743620991707,
+            0.007792831398546696,
+            -0.029907865449786186,
+            -0.019475309178233147,
+            0.009548136033117771,
+            -0.0010711194481700659,
+            0.0011500855907797813,
+            -0.007779294159263372,
+            -0.009173611178994179,
+            0.003406825475394726,
+            -0.02615359053015709,
+            0.02781413495540619,
+            -0.018455518409609795,
+            0.02285054884850979,
+            0.007670997641980648,
+            0.006384977605193853,
+            -0.0005039732786826789,
+            -0.008510295301675797,
+            -0.009620334021747112,
+            0.01680399663746357,
+            -0.0053290873765945435,
+            -0.002637469442561269,
+            -0.0173635296523571,
+            -0.0045349132269620895,
+            0.0067188916727900505,
+            0.0027886333409696817,
+            0.0022166927810758352,
+            0.011118434369564056,
+            -0.001003434183076024,
+            0.014051462523639202,
+            0.017372554168105125,
+            0.02622578665614128,
+            0.002996201394125819,
+            0.001920005539432168,
+            0.004189718514680862,
+            0.015991773456335068,
+            -0.0003677566419355571,
+            -0.00531555013731122,
+            0.0037858630530536175,
+            0.01088379230350256,
+            0.013879992999136448,
+            0.01175016351044178,
+            0.008284677751362324,
+            -0.00035986004513688385,
+            0.0044424100778996944,
+            0.016948392614722252,
+            0.020233385264873505,
+            -0.01026108767837286,
+            0.00556372944265604,
+            0.009069826453924179,
+            -0.00795076321810484,
+            0.0038512919563800097,
+            -0.007287447806447744,
+            0.007061830256134272,
+            -0.03245283290743828,
+            0.02936638332903385,
+            -0.0020666569471359253,
+            -0.008550906553864479,
+            -0.006168384570628405,
+            -0.013672425411641598,
+            0.013194115832448006,
+            -0.019637754186987877,
+            0.004367956425994635,
+            -0.02507062442600727,
+            0.008365900255739689,
+            0.002070041373372078,
+            -0.00022604061814490706,
+            0.030575694516301155,
+            0.012688732706010342,
+            0.01795915886759758,
+            -0.00787405390292406,
+            0.06663841009140015,
+            -0.004361187573522329,
+            -0.0046928455121815205,
+            -0.029276136308908463,
+            -0.018518690019845963,
+            -0.009412765502929688,
+            0.0005434563499875367,
+            0.0006892617093399167,
+            0.007413793820887804,
+            -0.0017293586861342192,
+            -0.00839297380298376,
+            0.012417991645634174,
+            -0.01141624990850687,
+            -0.013004597276449203,
+            -0.0007862772908993065,
+            0.015802254900336266,
+            0.005022247321903706,
+            -0.008383949287235737,
+            0.016054946929216385,
+            0.0030187631491571665,
+            0.0011709552491083741,
+            -0.006949021480977535,
+            0.012201398611068726,
+            -0.005107982084155083,
+            0.009078850969672203,
+            0.02573845349252224,
+            -0.003400056855753064,
+            -0.0020170211791992188,
+            0.004103983752429485,
+            -0.01187650952488184,
+            0.0018004282610490918,
+            0.006660230923444033,
+            -0.011037211865186691,
+            0.002903698245063424,
+            0.002323861001059413,
+            0.005536655429750681,
+            -0.010243037715554237,
+            -0.011407224461436272,
+            -0.0008404254913330078,
+            0.005649464204907417,
+            0.01554956380277872,
+            0.0001692131772870198,
+            0.0042777094058692455,
+            0.022652005776762962,
+            -0.01680399663746357,
+            -0.012075052596628666,
+            -0.02283249981701374,
+            0.007147565018385649,
+            0.01423195656388998,
+            0.002506611403077841,
+            -0.018987976014614105,
+            -0.012896301224827766,
+            0.02234516479074955,
+            0.009530086070299149,
+            0.034456316381692886,
+            0.006375952623784542,
+            0.0017146936152130365,
+            0.011154533363878727,
+            0.02756144292652607,
+            0.010423531755805016,
+            -0.019764099270105362,
+            0.0020621446892619133,
+            -0.012156275101006031
+          ],
+          "title": "8 Future Work"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "889920b5-afe5-4d91-8663-feb6e19ee898",
+      "type": "next",
+      "source": {
+        "id": "8f4dd4ba-f551-4c79-af63-003b9ccf3976",
+        "properties": {
+          "page_content": "8 Future Work 26 ",
+          "keyphrases": [],
+          "embedding": [
+            -0.024294501170516014,
+            0.01452074758708477,
+            -0.01423195656388998,
+            0.00711597828194499,
+            -0.015378094278275967,
+            0.004546194337308407,
+            -0.010901841334998608,
+            0.11226730793714523,
+            0.006362415850162506,
+            0.04544840380549431,
+            0.02425840124487877,
+            5.9330373915145174e-05,
+            -0.00413782661780715,
+            -0.018969925120472908,
+            -0.018825531005859375,
+            0.03312066197395325,
+            -0.018969925120472908,
+            0.014204883016645908,
+            -0.029601026326417923,
+            -0.0012081821914762259,
+            -0.01162381749600172,
+            -0.0036031126510351896,
+            -0.03490755334496498,
+            0.01664155349135399,
+            -0.001300685340538621,
+            0.011109409853816032,
+            0.04822801426053047,
+            0.030413249507546425,
+            -0.05245157331228256,
+            0.018013307824730873,
+            -0.007968813180923462,
+            0.023933513090014458,
+            0.019258717074990273,
+            -0.005049321334809065,
+            0.012463115155696869,
+            0.006443637888878584,
+            0.027326801791787148,
+            -0.008451634086668491,
+            0.0013762671733275056,
+            0.012914350256323814,
+            -0.0050673708319664,
+            -0.00494553754106164,
+            -0.009110437706112862,
+            0.026081392541527748,
+            0.035539280623197556,
+            -0.01277897972613573,
+            -0.012950449250638485,
+            -0.04808361828327179,
+            0.0016458802856504917,
+            0.007197200786322355,
+            0.0050809080712497234,
+            -0.022200770676136017,
+            0.001969641540199518,
+            0.0006689561414532363,
+            0.02433059923350811,
+            -0.00736415758728981,
+            0.012986548244953156,
+            0.07804563641548157,
+            0.020558273419737816,
+            0.007833442650735378,
+            -0.022200770676136017,
+            -0.02465548925101757,
+            -0.0032579179387539625,
+            -0.03779545798897743,
+            0.016109095886349678,
+            -0.02696581371128559,
+            0.029564928263425827,
+            -0.011903583072125912,
+            0.009882049635052681,
+            0.0025224045384675264,
+            -0.0248179342597723,
+            -0.005893131252378225,
+            -0.007359645329415798,
+            0.04508741572499275,
+            0.00730549730360508,
+            -0.017210109159350395,
+            0.026370182633399963,
+            0.018753333017230034,
+            -0.02730875089764595,
+            0.046711862087249756,
+            0.01647910848259926,
+            -0.03963649645447731,
+            -0.005058346316218376,
+            0.021496843546628952,
+            0.024529142305254936,
+            -0.004372468683868647,
+            -0.04992465674877167,
+            -0.11732114106416702,
+            0.016452033072710037,
+            0.0011562900617718697,
+            -0.005071883089840412,
+            0.015332970768213272,
+            0.0010463015642017126,
+            -0.028680507093667984,
+            -0.04238000512123108,
+            0.04259659722447395,
+            -0.00870432611554861,
+            0.010405482724308968,
+            -0.0004030094132758677,
+            -0.020215334370732307,
+            0.01059500128030777,
+            0.005374210886657238,
+            0.005992402788251638,
+            -0.02342812903225422,
+            0.0422356091439724,
+            -0.02474573627114296,
+            0.019872397184371948,
+            0.015026130713522434,
+            0.0034542051143944263,
+            -0.002572040306404233,
+            0.005545679945498705,
+            -0.033644095063209534,
+            -0.022327115759253502,
+            0.018148677423596382,
+            -0.006168384570628405,
+            -0.01757109723985195,
+            0.013952190987765789,
+            -0.03044934757053852,
+            -0.029474681243300438,
+            0.04494302347302437,
+            0.04559279978275299,
+            0.019511409103870392,
+            0.027687789872288704,
+            -0.03633345663547516,
+            -0.032073795795440674,
+            0.02905954420566559,
+            0.012327744625508785,
+            0.00182637432590127,
+            -0.003952819854021072,
+            0.03254307806491852,
+            0.02994396537542343,
+            0.004620647989213467,
+            -0.02211052179336548,
+            -0.03440216928720474,
+            0.021478792652487755,
+            -0.02243541181087494,
+            -0.009548136033117771,
+            0.024186205118894577,
+            -0.0227603018283844,
+            0.01911432109773159,
+            0.004505583085119724,
+            0.004850777797400951,
+            -0.011632842011749744,
+            0.0054193343967199326,
+            -0.006348878610879183,
+            0.03633345663547516,
+            -0.014078537002205849,
+            0.023067140951752663,
+            0.003984406590461731,
+            0.050466138869524,
+            -0.01692131906747818,
+            0.00340231298469007,
+            -0.01360925193876028,
+            0.015712007880210876,
+            0.0010214835638180375,
+            0.0005175103433430195,
+            0.000724796496797353,
+            0.024872081354260445,
+            0.014502697624266148,
+            0.012291645631194115,
+            0.02424035221338272,
+            0.03005226142704487,
+            -0.013672425411641598,
+            0.0012894044630229473,
+            0.005112494342029095,
+            -0.020504126325249672,
+            0.0355934277176857,
+            -0.01125380489975214,
+            -0.04353516921401024,
+            0.03377043828368187,
+            0.05858837440609932,
+            -0.01819380186498165,
+            0.01096501387655735,
+            -0.01952945813536644,
+            0.000879344530403614,
+            -0.0008167356136254966,
+            -0.014917834661900997,
+            0.03696518391370773,
+            -0.028102925047278404,
+            -0.019998742267489433,
+            -0.010802569799125195,
+            0.016677651554346085,
+            -0.007828930392861366,
+            0.003255661576986313,
+            0.01038743369281292,
+            -0.006457175128161907,
+            0.027417048811912537,
+            -0.037614963948726654,
+            0.052595969289541245,
+            0.013573153875768185,
+            0.01576615683734417,
+            -0.030503496527671814,
+            0.026009194552898407,
+            0.013419733382761478,
+            0.00960228405892849,
+            -0.06082649901509285,
+            -0.0021546478383243084,
+            0.026514578610658646,
+            0.004909438546746969,
+            -0.015937626361846924,
+            -0.014782464131712914,
+            -0.01398828998208046,
+            0.012273596599698067,
+            -0.007657460402697325,
+            -0.000787969387602061,
+            0.02068462036550045,
+            0.018500640988349915,
+            0.023969611153006554,
+            0.00773868290707469,
+            -0.024619389325380325,
+            0.023049091920256615,
+            -0.0033887759782373905,
+            -0.010910865850746632,
+            0.005360673647373915,
+            -0.008514807559549809,
+            -0.005405797157436609,
+            -0.02044997736811638,
+            0.04905828461050987,
+            -0.00019558223721105605,
+            -0.04313807934522629,
+            -0.007720633409917355,
+            0.0005521990242414176,
+            0.009854976087808609,
+            -0.004986148327589035,
+            -0.004424360580742359,
+            -0.03593636676669121,
+            -0.009854976087808609,
+            0.06310072541236877,
+            -0.02582870051264763,
+            -0.02730875089764595,
+            -0.01687619462609291,
+            0.004620647989213467,
+            0.0031721831765025854,
+            -0.028157074004411697,
+            -0.018807481974363327,
+            -0.0032285875640809536,
+            -0.012057003565132618,
+            -0.011524545960128307,
+            -0.025341367349028587,
+            0.022724201902747154,
+            0.05815518647432327,
+            0.013879992999136448,
+            -0.038842324167490005,
+            0.015278822742402554,
+            -0.013022646307945251,
+            -0.02059437334537506,
+            -0.04526790976524353,
+            0.09941612929105759,
+            0.023807166144251823,
+            0.003623418277129531,
+            0.0062270453199744225,
+            -0.00327822333201766,
+            0.0001518688368378207,
+            0.018717234954237938,
+            0.04006968066096306,
+            0.07746805250644684,
+            0.04956366866827011,
+            -0.007409281097352505,
+            -0.0322723388671875,
+            0.003966357093304396,
+            -0.021226102486252785,
+            -0.019006025046110153,
+            -0.045556701719760895,
+            -0.0063308291137218475,
+            -0.00036803868715651333,
+            -0.016605453565716743,
+            0.01588347740471363,
+            0.033553846180438995,
+            0.01112745888531208,
+            -0.005292988382279873,
+            0.04429324343800545,
+            0.005144080612808466,
+            0.014250006526708603,
+            -0.0033210907131433487,
+            -0.011831386014819145,
+            -0.02012508735060692,
+            -0.014376352541148663,
+            -0.0018714978359639645,
+            -0.008961530402302742,
+            0.015296871773898602,
+            -0.013419733382761478,
+            0.05848007649183273,
+            0.006195458583533764,
+            -0.006764015182852745,
+            -0.05191009119153023,
+            -0.03360799327492714,
+            -0.010540853254497051,
+            0.021749533712863922,
+            -0.0235725250095129,
+            -0.012381892651319504,
+            0.02202027477324009,
+            -0.0120028555393219,
+            -0.020486075431108475,
+            0.03530463948845863,
+            0.007066342514008284,
+            -0.028915148228406906,
+            0.005202741362154484,
+            0.020215334370732307,
+            -0.011867485009133816,
+            -0.0019312864169478416,
+            0.02341008000075817,
+            0.003925745841115713,
+            -0.025233069434762,
+            -0.004307039547711611,
+            0.005947279278188944,
+            0.021984176710247993,
+            0.015405167825520039,
+            0.025178922340273857,
+            -0.05801079049706459,
+            -0.014430500566959381,
+            0.043823957443237305,
+            -0.04173022881150246,
+            -0.020702669396996498,
+            -0.045881591737270355,
+            -0.016127144917845726,
+            0.03407727926969528,
+            0.028915148228406906,
+            0.02738094888627529,
+            -0.04660356789827347,
+            -0.03783155605196953,
+            -0.031063027679920197,
+            -0.00024550012312829494,
+            -0.05306525528430939,
+            0.030178606510162354,
+            0.0006040910957381129,
+            -0.0036865912843495607,
+            -0.023049091920256615,
+            0.049527570605278015,
+            0.01749889925122261,
+            -0.005482506938278675,
+            0.004674796015024185,
+            -0.007652948144823313,
+            0.018058430403470993,
+            -0.006592545658349991,
+            0.010640124790370464,
+            -0.034384120255708694,
+            -0.02878880314528942,
+            0.03631540387868881,
+            0.00046590028796344995,
+            -0.018112579360604286,
+            0.040502868592739105,
+            0.005446408409625292,
+            0.015179550275206566,
+            0.02218271978199482,
+            -0.018347222357988358,
+            -0.03573782369494438,
+            0.041405338793992996,
+            -0.05086322873830795,
+            0.004361187573522329,
+            0.0067188916727900505,
+            0.000985948834568262,
+            -0.04006968066096306,
+            -0.00813576951622963,
+            0.019655803218483925,
+            0.010179865173995495,
+            -0.044401541352272034,
+            -0.015305896289646626,
+            -0.004252891521900892,
+            0.021785633638501167,
+            -0.014340253546833992,
+            0.01498100720345974,
+            -0.04772263020277023,
+            -0.030070310458540916,
+            0.0056900754570961,
+            -0.05819128453731537,
+            -0.021406596526503563,
+            0.029691273346543312,
+            0.008726888336241245,
+            -0.013961215503513813,
+            -0.005094444844871759,
+            -0.03469095751643181,
+            0.013816820457577705,
+            0.02664092369377613,
+            -0.02804877795279026,
+            -0.00885323341935873,
+            -0.010739396326243877,
+            0.01009864266961813,
+            0.017950134351849556,
+            -0.04003358259797096,
+            0.022146621719002724,
+            -0.021659286692738533,
+            0.01274288073182106,
+            -0.014502697624266148,
+            -0.02994396537542343,
+            -0.01675887405872345,
+            -0.01637081243097782,
+            -0.01224652212113142,
+            0.02010703831911087,
+            -0.006646693684160709,
+            0.0017428958090022206,
+            0.002045223256573081,
+            -0.0347631573677063,
+            -0.0073867193423211575,
+            -0.027435097843408585,
+            0.03667639195919037,
+            0.027001911774277687,
+            0.02954687736928463,
+            -0.0042754532769322395,
+            0.018157701939344406,
+            -0.02979956939816475,
+            -0.028500013053417206,
+            -0.008420048281550407,
+            -0.0028269882313907146,
+            0.020052891224622726,
+            0.0463508740067482,
+            0.021875880658626556,
+            -0.0029646148905158043,
+            0.011632842011749744,
+            0.006800113711506128,
+            0.004968099296092987,
+            0.025251120328903198,
+            -0.03467290848493576,
+            0.011867485009133816,
+            -0.0421995110809803,
+            0.04981636255979538,
+            0.037542764097452164,
+            0.04310198128223419,
+            0.04104435071349144,
+            -0.03171280771493912,
+            0.0012860201532021165,
+            -0.0017541766865178943,
+            -0.01046865526586771,
+            0.03180305287241936,
+            -0.008650178089737892,
+            0.008907382376492023,
+            0.034546565264463425,
+            0.00457101222127676,
+            -0.013663400895893574,
+            -0.011578693985939026,
+            -0.023969611153006554,
+            -0.021550990641117096,
+            -0.000988204963505268,
+            0.025846749544143677,
+            -0.018347222357988358,
+            0.012327744625508785,
+            -0.02416815422475338,
+            0.004546194337308407,
+            -0.0022155647166073322,
+            -0.005121518857777119,
+            -0.025467712432146072,
+            0.014376352541148663,
+            -0.02631603367626667,
+            -0.002851806115359068,
+            0.03534073755145073,
+            0.006633156910538673,
+            -0.0163437370210886,
+            -0.04526790976524353,
+            -0.005414821673184633,
+            0.0007496143807657063,
+            0.00883067212998867,
+            -0.008478708565235138,
+            0.03772325813770294,
+            -0.040827758610248566,
+            -0.023319832980632782,
+            0.039744794368743896,
+            -0.015468341298401356,
+            -0.004737969022244215,
+            0.007720633409917355,
+            -0.02211052179336548,
+            -0.038842324167490005,
+            -0.0004580036911647767,
+            0.05786639824509621,
+            0.034709006547927856,
+            -0.002942053135484457,
+            -0.022363213822245598,
+            0.0013954447349533439,
+            -0.03815644606947899,
+            -0.014944908209145069,
+            -0.0017113093053922057,
+            0.05107982084155083,
+            0.003921233583241701,
+            -0.009575209580361843,
+            0.0023464227560907602,
+            -0.003269198816269636,
+            -0.021966127678751945,
+            0.016397885978221893,
+            0.01187650952488184,
+            -0.005450920667499304,
+            -0.018347222357988358,
+            0.005008710082620382,
+            -0.008911894634366035,
+            0.009412765502929688,
+            -0.027579491958022118,
+            -0.016072995960712433,
+            0.017354503273963928,
+            0.025052575394511223,
+            0.03180305287241936,
+            0.02017923630774021,
+            0.018590888008475304,
+            0.011118434369564056,
+            0.045809391885995865,
+            -0.013419733382761478,
+            0.001081836293451488,
+            0.006033014040440321,
+            -0.01724620722234249,
+            -0.042740993201732635,
+            0.006412051618099213,
+            -0.07840662449598312,
+            -0.004950049798935652,
+            0.03180305287241936,
+            -0.022236868739128113,
+            0.017607195302844048,
+            -0.0008167356136254966,
+            0.02880685217678547,
+            0.016632527112960815,
+            -0.027778036892414093,
+            -0.004611623473465443,
+            -0.015134426765143871,
+            0.011010137386620045,
+            0.012174324132502079,
+            0.0008443737751804292,
+            0.021984176710247993,
+            0.00013847279478795826,
+            0.00972863007336855,
+            -0.009295444004237652,
+            0.012625560164451599,
+            0.005144080612808466,
+            0.011037211865186691,
+            -0.021226102486252785,
+            0.006109723821282387,
+            -0.00761233689263463,
+            -0.000742281845305115,
+            0.008451634086668491,
+            0.03624320775270462,
+            -0.03862572833895683,
+            0.0006170640699565411,
+            -0.03873402625322342,
+            -0.008189918473362923,
+            0.06941801309585571,
+            -0.03167670965194702,
+            -0.01505320519208908,
+            0.006610595155507326,
+            -0.00080432667164132,
+            0.003090960904955864,
+            0.002966871252283454,
+            0.027850233018398285,
+            -0.043499067425727844,
+            0.019060172140598297,
+            -0.015332970768213272,
+            -0.026279935613274574,
+            -0.02433059923350811,
+            0.021370496600866318,
+            0.03275967389345169,
+            0.00581190874800086,
+            0.00792820192873478,
+            0.0388784222304821,
+            0.01985434629023075,
+            -0.031135225668549538,
+            0.01411463599652052,
+            -0.014222932048141956,
+            -0.0028269882313907146,
+            0.014268055558204651,
+            0.015802254900336266,
+            -0.014674167148768902,
+            0.012047979049384594,
+            0.0005457125371322036,
+            0.0025878336746245623,
+            0.015035155229270458,
+            0.021695386618375778,
+            0.0051892041228711605,
+            0.012842152267694473,
+            -0.020251434296369553,
+            -0.006633156910538673,
+            0.019150419160723686,
+            -0.010360359214246273,
+            -0.03923941031098366,
+            -0.0016537768533453345,
+            0.048877790570259094,
+            -0.03481730446219444,
+            -0.019511409103870392,
+            0.004349906928837299,
+            0.032055746763944626,
+            -0.009647407568991184,
+            -0.03481730446219444,
+            -0.012201398611068726,
+            0.022200770676136017,
+            0.014439525082707405,
+            -0.016722774133086205,
+            -0.03552123159170151,
+            0.0044401539489626884,
+            -0.004094959236681461,
+            -0.007440867833793163,
+            -0.004056604113429785,
+            0.016776923090219498,
+            0.005617877934128046,
+            -0.0017801227513700724,
+            -0.011768212541937828,
+            -0.02564820647239685,
+            -0.04945537447929382,
+            0.02308518998324871,
+            -0.0063082673586905,
+            0.019475309178233147,
+            0.01059500128030777,
+            -0.003131571924313903,
+            -0.03840913623571396,
+            0.010053519159555435,
+            -0.002939797006547451,
+            0.004234842024743557,
+            0.042488303035497665,
+            -0.037037380039691925,
+            0.00377683830447495,
+            0.023049091920256615,
+            -0.016154218465089798,
+            -0.0248179342597723,
+            0.00680913869291544,
+            -0.01299557276070118,
+            0.002973639639094472,
+            0.03451046347618103,
+            -0.01604592241346836,
+            0.0260633435100317,
+            0.02035973034799099,
+            0.0007941738585941494,
+            0.05006905272603035,
+            0.00674596568569541,
+            0.006118748802691698,
+            -0.006290218327194452,
+            0.014719290658831596,
+            -0.0037226900458335876,
+            -0.014349278062582016,
+            0.0008415535558015108,
+            0.0310269296169281,
+            -0.013067769818007946,
+            -0.01514345221221447,
+            -0.006204483564943075,
+            0.03461876139044762,
+            -0.005960816517472267,
+            0.007720633409917355,
+            -0.03797595202922821,
+            -0.006064600311219692,
+            0.00780185591429472,
+            0.020756816491484642,
+            -0.00798686221241951,
+            0.012463115155696869,
+            0.0011811080621555448,
+            -0.010161816142499447,
+            0.010441581718623638,
+            0.033553846180438995,
+            -0.00027962480089627206,
+            0.012806054204702377,
+            -0.011641867458820343,
+            -0.017137911170721054,
+            -0.011046236380934715,
+            0.018518690019845963,
+            0.0612235851585865,
+            -0.03321090713143349,
+            0.020305581390857697,
+            0.010640124790370464,
+            0.0044446662068367004,
+            0.015179550275206566,
+            0.016975466161966324,
+            0.020666569471359253,
+            0.0019245179137215018,
+            -0.007337083574384451,
+            0.03400508314371109,
+            0.018735283985733986,
+            -0.014583920128643513,
+            -0.0007501784712076187,
+            0.0025946020614355803,
+            -0.05219888314604759,
+            0.0003074039414059371,
+            -0.02216467075049877,
+            0.0019211337203159928,
+            -0.000984820770099759,
+            -0.011632842011749744,
+            0.002883392618969083,
+            0.017688417807221413,
+            0.004706382751464844,
+            0.01162381749600172,
+            -0.029853718355298042,
+            -0.009322518482804298,
+            0.0017553047509863973,
+            -0.021460743620991707,
+            0.04313807934522629,
+            0.037290073931217194,
+            -0.04653136804699898,
+            -0.01795915886759758,
+            0.004638697486370802,
+            0.049274880439043045,
+            0.0011342923389747739,
+            0.002632957184687257,
+            -0.029492730274796486,
+            -0.010567927733063698,
+            0.015278822742402554,
+            0.006705354433506727,
+            0.0028630869928747416,
+            0.017950134351849556,
+            -0.013455832377076149,
+            0.018040381371974945,
+            -0.005893131252378225,
+            0.008808109909296036,
+            -0.015666885301470757,
+            0.029727371409535408,
+            -0.011118434369564056,
+            0.000694338115863502,
+            0.01662350259721279,
+            0.018572838976979256,
+            -0.031171325594186783,
+            0.013356560841202736,
+            0.03176695480942726,
+            0.013870968483388424,
+            -0.018184777349233627,
+            -0.015594687312841415,
+            -0.044473737478256226,
+            0.026334084570407867,
+            0.007648435886949301,
+            0.0016470083501189947,
+            -0.00259685842320323,
+            -0.020161187276244164,
+            -0.018275024369359016,
+            -0.027038009837269783,
+            -0.01567590981721878,
+            -0.010694272816181183,
+            0.005139568354934454,
+            0.00446045957505703,
+            0.003928001970052719,
+            0.017814764752984047,
+            0.007034756243228912,
+            -0.02059437334537506,
+            -0.016100069507956505,
+            -0.014710266143083572,
+            0.0045304009690880775,
+            -0.024384748190641403,
+            -0.048047520220279694,
+            -0.03452851250767708,
+            -0.015450291335582733,
+            0.020504126325249672,
+            0.030918633565306664,
+            -0.006353390868753195,
+            0.011262829415500164,
+            -0.010730371810495853,
+            0.01472831517457962,
+            -0.0003917285066563636,
+            -0.0055772666819393635,
+            0.003946051467210054,
+            -0.012896301224827766,
+            0.03270552307367325,
+            0.01022498868405819,
+            0.021893929690122604,
+            0.010071569122374058,
+            -0.004316064063459635,
+            0.029402483254671097,
+            0.013636326417326927,
+            -0.013248263858258724,
+            0.0024005710147321224,
+            0.013645350933074951,
+            -0.006533884909003973,
+            0.048191916197538376,
+            -0.0014304154319688678,
+            -0.019078223034739494,
+            0.009313493967056274,
+            0.007562701124697924,
+            -0.0363876037299633,
+            0.05389552563428879,
+            -0.01178626250475645,
+            -0.0281209759414196,
+            0.010802569799125195,
+            0.01763427071273327,
+            -0.02581065148115158,
+            -0.006023989524692297,
+            0.015170525759458542,
+            -0.021388545632362366,
+            -0.0033662142232060432,
+            0.019258717074990273,
+            0.03617101162672043,
+            0.014944908209145069,
+            0.0013785234186798334,
+            -0.026677021756768227,
+            -0.017020590603351593,
+            0.023861315101385117,
+            -0.022200770676136017,
+            0.02516087330877781,
+            0.0013401684118434787,
+            -0.002479537157341838,
+            -0.02664092369377613,
+            0.009845951572060585,
+            -0.014565871097147465,
+            0.015666885301470757,
+            -0.025142822414636612,
+            0.015044179745018482,
+            0.008988603949546814,
+            0.00914202444255352,
+            0.015360044315457344,
+            -0.05277646332979202,
+            -0.03174890577793121,
+            -0.030900582671165466,
+            0.014746365137398243,
+            0.028355617076158524,
+            -0.03093668259680271,
+            -0.016569355502724648,
+            0.012887275777757168,
+            0.006452662870287895,
+            0.015278822742402554,
+            -0.012273596599698067,
+            4.801424074685201e-05,
+            0.02342812903225422,
+            -0.03270552307367325,
+            -0.013762672431766987,
+            0.0297815203666687,
+            0.006890360731631517,
+            -0.012562386691570282,
+            0.040936052799224854,
+            0.007571726106107235,
+            -0.03467290848493576,
+            -0.0036008565220981836,
+            -0.01303167175501585,
+            -0.0023396543692797422,
+            -0.005942767020314932,
+            0.005256889387965202,
+            0.008528344333171844,
+            0.012174324132502079,
+            0.0060736252926290035,
+            -0.023482277989387512,
+            0.02781413495540619,
+            0.005974353291094303,
+            0.014629043638706207,
+            0.01812160387635231,
+            -0.014638068154454231,
+            0.007513065356761217,
+            0.005653976462781429,
+            0.0037881191819906235,
+            0.013248263858258724,
+            0.0013954447349533439,
+            0.05891326069831848,
+            -0.014015363529324532,
+            0.005536655429750681,
+            0.0513325110077858,
+            0.011795287020504475,
+            -0.002567528048530221,
+            0.020468026399612427,
+            0.023789117112755775,
+            -0.013167042285203934,
+            -0.01766134425997734,
+            -0.011686990968883038,
+            -0.0031744393054395914,
+            -0.011389175429940224,
+            0.03146011382341385,
+            0.002407339634373784,
+            0.02158709056675434,
+            -0.03153231367468834,
+            0.03568367660045624,
+            0.006425588857382536,
+            -0.011307952925562859,
+            0.0132392393425107,
+            -0.0244208462536335,
+            -0.019493358209729195,
+            -0.006646693684160709,
+            -0.02472768723964691,
+            -0.006159360054880381,
+            -0.007106953766196966,
+            0.0310269296169281,
+            0.025918947532773018,
+            0.008537368848919868,
+            0.005667513702064753,
+            0.003379751229658723,
+            -0.0239876601845026,
+            -0.013230214826762676,
+            0.005568241700530052,
+            0.011064286343753338,
+            0.03451046347618103,
+            0.021334398537874222,
+            -0.0032669424545019865,
+            -0.006136798299849033,
+            -0.007806368172168732,
+            -0.011524545960128307,
+            0.017002541571855545,
+            -0.02887905016541481,
+            -0.03261527791619301,
+            -0.014078537002205849,
+            -0.003988918848335743,
+            -0.0025788089260458946,
+            0.00024634620058350265,
+            0.021226102486252785,
+            0.004410823807120323,
+            0.021532941609621048,
+            0.0016210622852668166,
+            0.012093102559447289,
+            -0.02664092369377613,
+            -0.007355133071541786,
+            0.031893301755189896,
+            0.0021501355804502964,
+            -0.014313179068267345,
+            0.013879992999136448,
+            -0.012860202230513096,
+            -0.012842152267694473,
+            -0.016749849542975426,
+            0.029925916343927383,
+            0.0033910321071743965,
+            -0.03423972427845001,
+            -0.010712322778999805,
+            -0.023698870092630386,
+            0.0020170211791992188,
+            -0.004401798825711012,
+            0.02615359053015709,
+            -0.03855353221297264,
+            0.0010874767322093248,
+            0.014881735667586327,
+            -0.005667513702064753,
+            -0.010306211188435555,
+            0.00447174021974206,
+            0.004006968345493078,
+            -0.017923060804605484,
+            -0.010495729744434357,
+            -0.012950449250638485,
+            -0.020468026399612427,
+            -0.009132999926805496,
+            -0.010243037715554237,
+            0.013690474443137646,
+            0.010784519836306572,
+            -0.008266627788543701,
+            0.008893844671547413,
+            -0.002168184844776988,
+            -0.02622578665614128,
+            -0.020088989287614822,
+            -0.022886646911501884,
+            0.03552123159170151,
+            -0.0030413249041885138,
+            -0.03837303817272186,
+            -0.0033323715906590223,
+            -0.0018940595909953117,
+            0.0011529058683663607,
+            0.0067279161885380745,
+            -0.01352803036570549,
+            -0.0182118508964777,
+            0.013753647916018963,
+            0.007237812038511038,
+            -0.013546079397201538,
+            0.011650891974568367,
+            0.026514578610658646,
+            0.017787689343094826,
+            0.02341008000075817,
+            0.011226730421185493,
+            0.037614963948726654,
+            0.017300356179475784,
+            -0.002073425566777587,
+            0.043065883219242096,
+            -0.001112294616177678,
+            -0.01844649389386177,
+            0.046314775943756104,
+            0.00023196307301986963,
+            -0.0017214621184393764,
+            0.027579491958022118,
+            -0.00013339638826437294,
+            0.0037046405486762524,
+            0.011668941006064415,
+            -0.027272652834653854,
+            0.005117006599903107,
+            0.004706382751464844,
+            0.03655004873871803,
+            0.005717149470001459,
+            -0.017760615795850754,
+            -0.022381264716386795,
+            0.012643609195947647,
+            0.018663085997104645,
+            -0.0011258317390456796,
+            0.014006339013576508,
+            0.006786576937884092,
+            -0.03920331224799156,
+            0.015784205868840218,
+            -0.0069760954938828945,
+            0.010622075758874416,
+            0.005762272980064154,
+            0.005153105594217777,
+            -0.016172267496585846,
+            0.0007783806649968028,
+            0.009575209580361843,
+            0.03815644606947899,
+            -0.013094844296574593,
+            0.03331920504570007,
+            -0.032308436930179596,
+            -0.009818877093493938,
+            -0.025846749544143677,
+            0.00782441720366478,
+            0.011542594991624355,
+            0.007355133071541786,
+            -0.042416103184223175,
+            0.005103469826281071,
+            -0.011479422450065613,
+            0.0010835283901542425,
+            -0.002407339634373784,
+            0.008203455246984959,
+            0.04097215086221695,
+            -0.02299494296312332,
+            0.025955045595765114,
+            -0.007350620813667774,
+            0.01837429590523243,
+            -0.0002003766130656004,
+            -0.009191660210490227,
+            -0.019728001207113266,
+            -0.004864315036684275,
+            0.003530915128067136,
+            -0.000880472594872117,
+            -0.007567213382571936,
+            -3.902479147654958e-05,
+            -0.005035784561187029,
+            0.0045394254848361015,
+            -0.006520348135381937,
+            0.01597372442483902,
+            -0.01729133166372776,
+            0.04227171093225479,
+            0.001362730166874826,
+            -0.011190632358193398,
+            -0.0151073532178998,
+            -0.03402313217520714,
+            0.0027705838438123465,
+            -0.006655718665570021,
+            0.02696581371128559,
+            0.014674167148768902,
+            0.020919261500239372,
+            -0.0008658074657432735,
+            0.008785548619925976,
+            0.020648520439863205,
+            -0.017850862815976143,
+            0.024944279342889786,
+            -0.01807648129761219,
+            0.002287762239575386,
+            -0.0040430668741464615,
+            0.016533255577087402,
+            0.01869918406009674,
+            0.01398828998208046,
+            0.0020847064442932606,
+            -0.008591517806053162,
+            -0.032001595944166183,
+            -0.008095158264040947,
+            -0.005893131252378225,
+            0.014583920128643513,
+            0.010279136709868908,
+            0.019637754186987877,
+            0.0015691702719777822,
+            0.010360359214246273,
+            0.00032376122544519603,
+            0.004873339552432299,
+            0.02456524223089218,
+            0.006777551956474781,
+            -0.03978089243173599,
+            0.0005428923177532852,
+            0.001857960713095963,
+            0.009182635694742203,
+            -0.021623188629746437,
+            0.014908809214830399,
+            -0.00537872314453125,
+            0.028752705082297325,
+            -0.007007681764662266,
+            0.02084706351161003,
+            -0.006136798299849033,
+            -0.004661259241402149,
+            -0.003783606691285968,
+            0.006524860393255949,
+            0.030846435576677322,
+            -0.00328047969378531,
+            0.024854032322764397,
+            0.023103240877389908,
+            0.0074724541045725346,
+            0.007851491682231426,
+            -0.017923060804605484,
+            0.0030774236656725407,
+            0.022904695942997932,
+            -0.0017146936152130365,
+            0.007887590676546097,
+            -0.004877852275967598,
+            0.006303755100816488,
+            -0.016677651554346085,
+            0.01627153903245926,
+            0.01452074758708477,
+            -0.0058209337294101715,
+            0.02781413495540619,
+            0.015567612834274769,
+            0.009737654589116573,
+            -0.002993945265188813,
+            0.004408567678183317,
+            0.014538796618580818,
+            0.00432283291593194,
+            0.004140082746744156,
+            -0.015657860785722733,
+            0.013573153875768185,
+            0.010802569799125195,
+            0.013130943290889263,
+            -0.03687493875622749,
+            -0.016036897897720337,
+            -0.02326568402349949,
+            -0.0019121089717373252,
+            0.0194572601467371,
+            0.00011612254456849769,
+            0.001237512449733913,
+            0.025612108409404755,
+            -0.0003043017059098929,
+            -0.008162843994796276,
+            -0.007075367495417595,
+            -0.020973410457372665,
+            0.0017135655507445335,
+            -0.0001287430350203067,
+            -0.04270489513874054,
+            0.007296472322195768,
+            0.007025731261819601,
+            0.001146137248724699,
+            -0.007910151965916157,
+            -0.02572040446102619,
+            -0.01134405191987753,
+            0.08793670684099197,
+            -0.012345793657004833,
+            -0.02283249981701374,
+            -0.008591517806053162,
+            -0.004950049798935652,
+            -0.008659202605485916,
+            -0.003582807257771492,
+            0.021370496600866318,
+            0.01860893703997135,
+            0.0015624017687514424,
+            0.015179550275206566,
+            -0.008108695968985558,
+            0.02936638332903385,
+            0.01208407711237669,
+            0.019998742267489433,
+            -0.0028044264763593674,
+            0.007856003940105438,
+            -0.0044401539489626884,
+            0.013546079397201538,
+            -0.022128572687506676,
+            -0.0011641867458820343,
+            -0.0031383405439555645,
+            0.00407465361058712,
+            -0.02202027477324009,
+            0.007883078418672085,
+            -0.011515521444380283,
+            -0.013302412815392017,
+            -0.0032015135511755943,
+            -0.030467398464679718,
+            -0.013320461846888065,
+            -0.002380265388637781,
+            0.039095014333724976,
+            0.009620334021747112,
+            0.01804940588772297,
+            0.0012442809529602528,
+            0.02267005480825901,
+            0.0072197625413537025,
+            0.0017428958090022206,
+            0.014096586033701897,
+            0.009137512184679508,
+            -0.00923227146267891,
+            0.04624257981777191,
+            0.015405167825520039,
+            -0.0012262315722182393,
+            -0.03277772292494774,
+            0.003679822664707899,
+            0.016424959525465965,
+            0.009295444004237652,
+            -0.0028021703474223614,
+            -0.0017451519379392266,
+            0.00537872314453125,
+            -0.0046793087385594845,
+            0.017399627715349197,
+            0.015008081682026386,
+            -0.02227296680212021,
+            0.017562072724103928,
+            -0.0018331428291276097,
+            0.02259785681962967,
+            0.0006396258249878883,
+            -0.00021673388255294412,
+            0.02837366610765457,
+            -0.0018444237066432834,
+            0.024150105193257332,
+            0.00885323341935873,
+            -0.002257303800433874,
+            0.0033684703521430492,
+            0.017300356179475784,
+            0.004404054954648018,
+            0.006655718665570021,
+            -0.004176181275397539,
+            -0.017968183383345604,
+            0.023067140951752663,
+            0.01650618202984333,
+            0.002729972591623664,
+            -0.01763427071273327,
+            -0.006786576937884092,
+            0.009182635694742203,
+            0.0013807795476168394,
+            0.006651206407696009,
+            -0.005459945183247328,
+            0.01026108767837286,
+            0.014908809214830399,
+            -0.006939996499568224,
+            0.03241673484444618,
+            -0.010992088355123997,
+            -0.02299494296312332,
+            -0.011885534040629864,
+            0.023121289908885956,
+            0.01943921111524105,
+            -0.004207768011838198,
+            -0.015134426765143871,
+            0.031315721571445465,
+            -0.011533570475876331,
+            -0.0036143935285508633,
+            -0.030900582671165466,
+            -0.036189060658216476,
+            0.04501521959900856,
+            -0.0004785912751685828,
+            -0.0446903295814991,
+            0.027453146874904633,
+            0.01692131906747818,
+            0.010694272816181183,
+            0.010243037715554237,
+            -0.022128572687506676,
+            0.015657860785722733,
+            -0.0049274880439043045,
+            0.0045349132269620895,
+            -0.008244066499173641,
+            0.006664743181318045,
+            -0.011010137386620045,
+            0.027669738978147507,
+            -0.01221042312681675,
+            -0.002286634175106883,
+            -0.0010615306673571467,
+            0.00028300905250944197,
+            0.005870569497346878,
+            -0.005062858574092388,
+            -0.014656118117272854,
+            0.011930657550692558,
+            0.01389804296195507,
+            -0.0018703697714954615,
+            0.0179862342774868,
+            0.020648520439863205,
+            -0.03581002354621887,
+            -0.0014416963094845414,
+            0.0012826359597966075,
+            -0.0005392259918153286,
+            0.008826159872114658,
+            0.012075052596628666,
+            0.023211536929011345,
+            -0.0138619439676404,
+            -0.0368388369679451,
+            0.008762986399233341,
+            0.023644722998142242,
+            -0.0022990431170910597,
+            -0.023446178063750267,
+            -0.0020858345087617636,
+            0.00829821452498436,
+            0.020540224388241768,
+            -0.0012905325274914503,
+            0.008059060201048851,
+            0.0077702696435153484,
+            -0.017210109159350395,
+            -0.009999371133744717,
+            -0.005766785237938166,
+            -0.015332970768213272,
+            -0.027778036892414093,
+            0.010775495320558548,
+            0.005978866014629602,
+            0.01911432109773159,
+            -0.022868597880005836,
+            -0.021370496600866318,
+            -0.021911978721618652,
+            0.0009171354467980564,
+            0.006926459725946188,
+            -0.020973410457372665,
+            0.0014687704388052225,
+            -0.010766470804810524,
+            0.025792602449655533,
+            -0.013636326417326927,
+            0.0014394401805475354,
+            0.0020486076828092337,
+            0.028319519013166428,
+            0.002041839063167572,
+            0.008401998318731785,
+            -0.020504126325249672,
+            -0.0035106095019727945,
+            0.07266691327095032,
+            -0.013943166472017765,
+            -0.0045258887112140656,
+            -0.01175016351044178,
+            0.017020590603351593,
+            -0.01009864266961813,
+            0.0036707979161292315,
+            0.04772263020277023,
+            -0.014403426088392735,
+            0.004473996814340353,
+            0.021911978721618652,
+            0.008415536023676395,
+            0.000976360053755343,
+            -0.00967448204755783,
+            -0.012932399287819862,
+            -0.0023464227560907602,
+            -0.017697442322969437,
+            0.02301299385726452,
+            -0.021081706508994102,
+            0.027074109762907028,
+            -0.015784205868840218,
+            -0.034131426364183426,
+            0.01712888665497303,
+            0.029240038245916367,
+            -0.015594687312841415,
+            -0.009909124113619328,
+            -0.000990461208857596,
+            0.010983063839375973,
+            0.035629529505968094,
+            0.014448549598455429,
+            0.01887967810034752,
+            0.005320062395185232,
+            -0.003212794428691268,
+            0.0018410393968224525,
+            0.02012508735060692,
+            0.015892501920461655,
+            -0.0013345279730856419,
+            0.0064707123674452305,
+            -0.003801656188443303,
+            0.02548576146364212,
+            -0.014845636673271656,
+            -0.02846391312777996,
+            -0.00235995976254344,
+            -0.026370182633399963,
+            -0.007576238363981247,
+            0.029438581317663193,
+            -0.009448864497244358,
+            0.012445065192878246,
+            0.025431614369153976,
+            -0.019475309178233147,
+            -0.027326801791787148,
+            -0.02117195352911949,
+            0.01729133166372776,
+            -0.005915693007409573,
+            0.005288476124405861,
+            0.0017293586861342192,
+            0.015892501920461655,
+            -0.016118120402097702,
+            -0.01514345221221447,
+            0.0020294301211833954,
+            -0.020829014480113983,
+            0.024150105193257332,
+            0.018365271389484406,
+            -0.004354419186711311,
+            0.008492245338857174,
+            0.019728001207113266,
+            -0.010549877770245075,
+            0.0012713550822809339,
+            -0.017905011773109436,
+            -0.004877852275967598,
+            -0.0004585677233990282,
+            0.009936198592185974,
+            -0.02458329126238823,
+            -0.007707096636295319,
+            -0.004516863729804754,
+            -0.00182637432590127,
+            0.01522467378526926,
+            -0.015621760860085487,
+            -0.0004402362974360585,
+            0.008550906553864479,
+            0.0022144364193081856,
+            -0.017065713182091713,
+            -0.0015274309553205967,
+            0.00039313864544965327,
+            -0.011154533363878727,
+            0.0017169497441500425,
+            -0.010279136709868908,
+            -0.023879364132881165,
+            -0.004006968345493078,
+            -0.01141624990850687,
+            -0.0045326570980250835,
+            0.005789346992969513,
+            -0.00901567842811346,
+            0.0007039268384687603,
+            0.020666569471359253,
+            0.016316663473844528,
+            0.009304468519985676,
+            -0.029149791225790977,
+            0.003840011078864336,
+            -0.0037001282908022404,
+            -0.0027412534691393375,
+            -0.01692131906747818,
+            0.0030210192780941725,
+            -0.023536425083875656,
+            0.005089932586997747,
+            -0.018275024369359016,
+            0.014746365137398243,
+            0.005802884232252836,
+            -0.0032579179387539625,
+            -0.0006774167995899916,
+            -0.0002590371877886355,
+            -0.0010288162156939507,
+            0.0007428458775393665,
+            0.01675887405872345,
+            0.050430040806531906,
+            0.0032082819379866123,
+            -0.034456316381692886,
+            -0.004128801636397839,
+            0.006813650950789452,
+            0.0019380550365895033,
+            0.0089931171387434,
+            0.012634584680199623,
+            0.04021407663822174,
+            -0.03732617199420929,
+            0.01564883440732956,
+            -0.021298298612236977,
+            0.005396772641688585,
+            -0.014177808538079262,
+            0.0049229757860302925,
+            -0.025088675320148468,
+            0.01009864266961813,
+            0.012237497605383396,
+            -0.0019910752307623625,
+            -0.005247864872217178,
+            -0.015576637350022793,
+            0.016975466161966324,
+            0.010937940329313278,
+            -0.019998742267489433,
+            -0.009218734689056873,
+            0.001989946933463216,
+            -0.03416752442717552,
+            -0.01618129201233387,
+            -0.005198229104280472,
+            -0.009710581041872501,
+            0.0042867339216172695,
+            -0.00875847414135933,
+            0.012327744625508785,
+            0.013852919451892376,
+            -0.008031985722482204,
+            -0.002881136490032077,
+            0.014683191664516926,
+            -0.003145109163597226,
+            0.0037204339168965816,
+            0.00328047969378531,
+            0.021803682669997215,
+            -0.03490755334496498,
+            0.019132370129227638,
+            -0.003659517038613558,
+            -0.0015939881559461355,
+            -0.008677251636981964,
+            -0.0006633157026953995,
+            -0.022796399891376495,
+            -0.007711608894169331,
+            0.004473996814340353,
+            -0.005234327632933855,
+            0.0025539910420775414,
+            -0.0190240740776062,
+            0.021893929690122604,
+            0.010567927733063698,
+            0.008388461545109749,
+            -0.021081706508994102,
+            -0.005302012898027897,
+            -0.014096586033701897,
+            -0.01786891184747219,
+            0.007549164351075888,
+            -0.013744622468948364,
+            0.0009086747886613011,
+            0.004415336064994335,
+            0.016406910493969917,
+            0.0015285591362044215,
+            0.003025531768798828,
+            0.009069826453924179,
+            -0.015757132321596146,
+            -0.03317480906844139,
+            0.008478708565235138,
+            -0.0101347416639328,
+            0.026695072650909424,
+            0.0256662555038929,
+            0.015576637350022793,
+            0.009493988007307053,
+            0.006430101115256548,
+            -0.00327822333201766,
+            -0.03535878658294678,
+            -0.023283734917640686,
+            0.011921633034944534,
+            -0.020468026399612427,
+            0.017598170787096024,
+            0.004169412888586521,
+            -0.003429387230426073,
+            0.012318720109760761,
+            0.01671374961733818,
+            0.028193172067403793,
+            0.025359416380524635,
+            -0.01657838001847267,
+            0.01687619462609291,
+            -0.019565556198358536,
+            -0.006366928108036518,
+            0.02862635813653469,
+            -0.0040362984873354435,
+            -0.011172582395374775,
+            0.007643923629075289,
+            -0.0095120370388031,
+            0.01918651908636093,
+            0.000988204963505268,
+            -0.010495729744434357,
+            -0.01038743369281292,
+            0.0036346991546452045,
+            -0.016397885978221893,
+            -0.01911432109773159,
+            0.01554956380277872,
+            0.027525344863533974,
+            0.01369949895888567,
+            -0.020305581390857697,
+            0.008663714863359928,
+            0.002093731192871928,
+            0.024962328374385834,
+            0.005667513702064753,
+            0.03718177601695061,
+            -0.0056765382178127766,
+            0.00643461337313056,
+            -0.016452033072710037,
+            -0.005392259918153286,
+            0.01766134425997734,
+            -0.003770069684833288,
+            -0.007467941846698523,
+            -0.007851491682231426,
+            0.0030165070202201605,
+            0.016289589926600456,
+            0.01224652212113142,
+            0.0024321575183421373,
+            -0.01729133166372776,
+            -0.0031044979114085436,
+            -0.010035470128059387,
+            -0.0036031126510351896,
+            -0.0010271240025758743,
+            -0.018681135028600693,
+            -0.0007941738585941494,
+            0.012165299616754055,
+            -0.04537620767951012,
+            -0.001970769604668021,
+            -0.015910550951957703,
+            -0.022056374698877335,
+            -0.024132056161761284,
+            0.023771068081259727,
+            0.009620334021747112,
+            0.013645350933074951,
+            0.007319034077227116,
+            0.01604592241346836,
+            -0.041008252650499344,
+            0.01960165612399578,
+            0.01017084065824747,
+            -0.005644951947033405,
+            0.0022730971686542034,
+            -0.02705606073141098,
+            0.006254119332879782,
+            0.008216992020606995,
+            -0.003864829195663333,
+            0.01551346480846405,
+            -0.026911664754152298,
+            -0.002569784177467227,
+            -0.011885534040629864,
+            0.011831386014819145,
+            0.027344850823283195,
+            -0.006145822815597057,
+            3.934206688427366e-05,
+            -0.015161501243710518,
+            0.005207253620028496,
+            0.002380265388637781,
+            -0.01837429590523243,
+            0.006637669168412685,
+            -0.014204883016645908,
+            -0.0036031126510351896,
+            0.00458229286596179,
+            -0.0012916605919599533,
+            -0.01683107204735279,
+            0.0019301583524793386,
+            -0.0011935170041397214,
+            -0.004171669017523527,
+            -0.005103469826281071,
+            -0.0022674566134810448,
+            0.012643609195947647,
+            -0.008925431407988071,
+            -0.008623103611171246,
+            -0.0035444521345198154,
+            -0.022363213822245598,
+            -0.0011991574428975582,
+            -0.00114275305531919,
+            -0.007070854771882296,
+            0.00494553754106164,
+            0.002865343354642391,
+            -0.004661259241402149,
+            0.01597372442483902,
+            -0.008203455246984959,
+            -0.00552763044834137,
+            0.008230529725551605,
+            -0.02259785681962967,
+            0.005685563199222088,
+            0.012390917167067528,
+            -0.04089995473623276,
+            0.0030480935238301754,
+            -0.004550706595182419,
+            0.028427815064787865,
+            -0.011425274424254894,
+            0.0042799655348062515,
+            0.021460743620991707,
+            -0.009011166170239449,
+            -0.026171639561653137,
+            0.00989107508212328,
+            0.053570639342069626,
+            0.004886876791715622,
+            -0.0008945736917667091,
+            -0.003650492522865534,
+            0.00023027094721328467,
+            -0.011975781060755253,
+            -0.0064797368831932545,
+            -0.014159759506583214,
+            0.016984490677714348,
+            0.005112494342029095,
+            -0.015784205868840218,
+            -0.020052891224622726,
+            -0.015558588318526745,
+            -0.013148992322385311,
+            0.023608623072504997,
+            -0.010585976764559746,
+            -0.01579323038458824,
+            0.004232585895806551,
+            0.02259785681962967,
+            0.010306211188435555,
+            -0.00575324846431613,
+            -0.02109975554049015,
+            -0.019096272066235542,
+            0.018067454919219017,
+            0.006303755100816488,
+            0.014331229031085968,
+            0.0024908180348575115,
+            -0.013708524405956268,
+            0.014177808538079262,
+            0.003025531768798828,
+            0.021081706508994102,
+            -0.0005378159112296999,
+            -0.007052805740386248,
+            0.005013222806155682,
+            -0.02456524223089218,
+            0.020666569471359253,
+            -0.008695301599800587,
+            0.013148992322385311,
+            -0.020648520439863205,
+            -0.008623103611171246,
+            0.004480765201151371,
+            -0.0030864484142512083,
+            -0.012634584680199623,
+            0.021496843546628952,
+            0.0024028271436691284,
+            -0.008271140046417713,
+            -0.0032895042095333338,
+            0.0008240682072937489,
+            -0.027778036892414093,
+            -0.001954976236447692,
+            0.006394002120941877,
+            -0.02465548925101757,
+            0.015405167825520039,
+            0.0015533770201727748,
+            -0.014240982010960579,
+            0.004015992861241102,
+            -0.00792820192873478,
+            -0.005274938885122538,
+            0.0003812937065958977,
+            0.013654375448822975,
+            -0.029907865449786186,
+            0.015982748940587044,
+            -0.0031766954343765974,
+            -0.0071295155212283134,
+            0.006935484241694212,
+            0.0018365271389484406,
+            0.00864566583186388,
+            -0.008027473464608192,
+            1.942426388268359e-05,
+            0.017778664827346802,
+            0.0063263168558478355,
+            0.008568955585360527,
+            0.00513054383918643,
+            -0.0031766954343765974,
+            -0.003219562815502286,
+            0.0006644437671639025,
+            0.002129829954355955,
+            -0.020810965448617935,
+            -0.005911180749535561,
+            -0.003986662719398737,
+            0.0014089817414060235,
+            0.01026108767837286,
+            -0.003939283080399036,
+            0.0013593458570539951,
+            -0.02573845349252224,
+            -0.004268684424459934,
+            0.005911180749535561,
+            0.0030187631491571665,
+            0.007580750621855259,
+            0.03028690442442894,
+            0.015396143309772015,
+            0.019258717074990273,
+            0.0020215334370732307,
+            -0.0015240467619150877,
+            -0.011948706582188606,
+            -0.0011399327777326107,
+            0.00792820192873478,
+            -0.00844712182879448,
+            0.009827901609241962,
+            -0.022814448922872543,
+            0.01072134729474783,
+            0.005261402111500502,
+            0.003183464054018259,
+            -0.000386370113119483,
+            -0.014096586033701897,
+            0.012129200622439384,
+            0.021821731701493263,
+            -0.021496843546628952,
+            0.001207054010592401,
+            -0.013780721463263035,
+            -0.00023619340208824724,
+            -0.011569669470191002,
+            -0.03656809777021408,
+            -0.008144794963300228,
+            -0.030665941536426544,
+            -0.00015750927559565753,
+            -0.014033413492143154,
+            -0.0030119947623461485,
+            -0.01079354528337717,
+            -0.010694272816181183,
+            0.001760945189744234,
+            -0.009791802614927292,
+            0.009692531079053879,
+            0.010414507240056992,
+            0.004737969022244215,
+            -0.0007857132004573941,
+            0.011235755868256092,
+            -0.009367641992866993,
+            -0.014872711151838303,
+            -0.010414507240056992,
+            -0.002763815224170685,
+            0.024547193199396133,
+            -0.010766470804810524,
+            -0.014845636673271656,
+            -0.00612326106056571,
+            -0.006430101115256548,
+            -0.011605768464505672,
+            -0.01837429590523243,
+            0.014195858500897884,
+            0.02754339389503002,
+            -0.013690474443137646,
+            0.0023667283821851015,
+            -0.010847693309187889,
+            0.013266313821077347,
+            0.023969611153006554,
+            -0.001973025733605027,
+            -0.0289692971855402,
+            -0.018717234954237938,
+            0.0012521775206550956,
+            -0.011425274424254894,
+            -0.0072287870571017265,
+            -0.01432220358401537,
+            -0.004706382751464844,
+            -0.015206624753773212,
+            0.017769640311598778,
+            -0.00926837045699358,
+            -0.02028753235936165,
+            -0.008456147275865078,
+            -0.0030165070202201605,
+            -0.00600142776966095,
+            -0.008690789341926575,
+            -0.022525658831000328,
+            0.011434298940002918,
+            0.008253091014921665,
+            0.009593259543180466,
+            -0.01567590981721878,
+            0.019673852249979973,
+            0.013383634388446808,
+            -0.024799883365631104,
+            -0.003431643359363079,
+            -0.01464709360152483,
+            -0.0020666569471359253,
+            -0.013843894936144352,
+            -0.011271853931248188,
+            0.0006982863997109234,
+            0.0014394401805475354,
+            0.010802569799125195,
+            0.0024231327697634697,
+            0.022633954882621765,
+            -4.344900844444055e-06,
+            -0.013627301901578903,
+            0.00643461337313056,
+            0.01522467378526926,
+            -0.014538796618580818,
+            0.02806682698428631,
+            0.012914350256323814,
+            -0.017670368775725365,
+            0.015937626361846924,
+            0.013049720786511898,
+            -0.015964699909090996,
+            0.005762272980064154,
+            0.004977123811841011,
+            -0.00854188110679388,
+            -0.01361827738583088,
+            0.018320146948099136,
+            0.01175016351044178,
+            0.021532941609621048,
+            0.001636855537071824,
+            0.012508238665759563,
+            -0.00686779897660017,
+            0.007409281097352505,
+            -0.022200770676136017,
+            0.01985434629023075,
+            -0.00963838305324316,
+            -0.02010703831911087,
+            0.014926859177649021,
+            -0.0026103954296559095,
+            0.012896301224827766,
+            0.011262829415500164,
+            -0.004832728765904903,
+            -0.05097152292728424,
+            -0.01038743369281292,
+            0.011470397934317589,
+            0.003955075982958078,
+            -0.014990031719207764,
+            0.019565556198358536,
+            -0.0036911035422235727,
+            0.030268853530287743,
+            -0.005049321334809065,
+            0.02631603367626667,
+            -0.005256889387965202,
+            0.018735283985733986,
+            -0.015125402249395847,
+            -0.016993515193462372,
+            0.0071295155212283134,
+            0.00026777986204251647,
+            -0.004674796015024185,
+            0.008307239040732384,
+            -0.030918633565306664,
+            -0.010531828738749027,
+            -0.007147565018385649,
+            0.01886162906885147,
+            -0.006177409086376429,
+            0.026604825630784035,
+            0.0019233898492529988,
+            0.009466913528740406,
+            0.017580121755599976,
+            -0.01352803036570549,
+            0.019565556198358536,
+            -0.05479799583554268,
+            -0.0020316862501204014,
+            -0.0044333855621516705,
+            0.027669738978147507,
+            -0.004205511882901192,
+            -0.0012668427079916,
+            0.004088190384209156,
+            3.040549358956923e-07,
+            -0.005017735064029694,
+            0.007661973126232624,
+            -3.456531703704968e-05,
+            0.01361827738583088,
+            -0.027272652834653854,
+            -0.02739899791777134,
+            0.00661961967125535,
+            0.004828216042369604,
+            0.016975466161966324,
+            -0.010306211188435555,
+            -0.001425903057679534,
+            0.017850862815976143,
+            -0.01439440157264471,
+            0.00471991952508688,
+            -0.010495729744434357,
+            -0.016316663473844528,
+            -0.018455518409609795,
+            0.0018771382747218013,
+            0.019493358209729195,
+            -0.006412051618099213,
+            0.005536655429750681,
+            0.001989946933463216,
+            0.013627301901578903,
+            0.0012352562043815851,
+            0.011425274424254894,
+            -0.011777237989008427,
+            0.027705838903784752,
+            -0.009710581041872501,
+            0.015982748940587044,
+            0.004088190384209156,
+            -0.0017146936152130365,
+            0.006168384570628405,
+            -0.007251348812133074,
+            -0.029655175283551216,
+            -0.004467227961868048,
+            -0.010243037715554237,
+            -0.0018342708935961127,
+            -0.02433059923350811,
+            -0.024799883365631104,
+            0.0040430668741464615,
+            0.0030390687752515078,
+            0.00963838305324316,
+            0.016100069507956505,
+            0.022236868739128113,
+            0.007589775137603283,
+            0.008731400594115257,
+            -0.0021726973354816437,
+            0.004106239881366491,
+            -0.020901212468743324,
+            0.008302726782858372,
+            -0.016614478081464767,
+            -0.010576952248811722,
+            -0.0019211337203159928,
+            -0.02597309648990631,
+            0.002477281028404832,
+            -0.008131257258355618,
+            0.007319034077227116,
+            -0.00026369054103270173,
+            0.0021140368189662695,
+            0.016749849542975426,
+            -0.00894348043948412,
+            0.010613051243126392,
+            -0.005089932586997747,
+            0.00327822333201766,
+            -0.004904926288872957,
+            -0.014583920128643513,
+            -0.0011207553325220942,
+            -0.014421476051211357,
+            0.012102127075195312,
+            -0.023139338940382004,
+            -0.0045213764533400536,
+            -0.05512288585305214,
+            -0.006096187047660351,
+            -0.021911978721618652,
+            0.005978866014629602,
+            0.013519004918634892,
+            -0.012273596599698067,
+            -0.01590152643620968,
+            -0.02043192833662033,
+            0.022561758756637573,
+            -0.012038953602313995,
+            -0.0012228472623974085,
+            -0.01059500128030777,
+            0.035467084497213364,
+            -0.030431298539042473,
+            -0.001330015598796308,
+            -0.0032263314351439476,
+            0.008916406892240047,
+            -0.006615107413381338,
+            -0.024294501170516014,
+            -0.005324574653059244,
+            -0.012806054204702377,
+            0.004453691188246012,
+            -0.007661973126232624,
+            -0.017814764752984047,
+            0.005324574653059244,
+            -0.022778350859880447,
+            0.0031947449315339327,
+            0.012562386691570282,
+            -0.006800113711506128,
+            -0.00527945114299655,
+            0.004900414030998945,
+            0.007711608894169331,
+            0.009773753583431244,
+            0.0076303863897919655,
+            -0.0014890760648995638,
+            0.006344366353005171,
+            -0.009385691024363041,
+            -0.009277394972741604,
+            0.012824103236198425,
+            0.02102755755186081,
+            -0.002097115386277437,
+            0.006046551279723644,
+            -0.019385062158107758,
+            -0.011957732029259205,
+            -0.02747119590640068,
+            -0.013546079397201538,
+            -0.02100950852036476,
+            0.0009092388208955526,
+            0.01125380489975214,
+            0.00643461337313056,
+            0.01432220358401537,
+            -0.00019713335495907813,
+            -0.0030526057817041874,
+            0.0016278307884931564,
+            -0.046964555978775024,
+            -0.010215964168310165,
+            0.006380465347319841,
+            -0.005306525621563196,
+            0.009038240648806095,
+            0.020829014480113983,
+            -0.007684534881263971,
+            -0.021984176710247993,
+            0.0028540624771267176,
+            -0.0006937740836292505,
+            -0.0010694273514673114,
+            0.014078537002205849,
+            0.014069512486457825,
+            -0.006678280420601368,
+            0.013654375448822975,
+            0.004363444168120623,
+            0.022399313747882843,
+            -0.029330285266041756,
+            0.008826159872114658,
+            0.011235755868256092,
+            -0.024781834334135056,
+            0.018347222357988358,
+            0.004832728765904903,
+            -0.018094530329108238,
+            0.004155875649303198,
+            0.015712007880210876,
+            0.019709952175617218,
+            -0.002291146432980895,
+            0.012354818172752857,
+            -0.005103469826281071,
+            0.003623418277129531,
+            0.019312864169478416,
+            0.004977123811841011,
+            -0.021081706508994102,
+            -0.017589146271348,
+            -0.011641867458820343,
+            0.018500640988349915,
+            -0.005969841033220291,
+            0.012390917167067528,
+            0.018365271389484406,
+            -0.0007405896903946996,
+            -0.009909124113619328,
+            0.022706152871251106,
+            -0.0006666999543085694,
+            0.006831700447946787,
+            0.0014180064899846911,
+            0.0004551834717858583,
+            -0.019006025046110153,
+            0.000448414939455688,
+            0.008153819479048252,
+            0.012815078720450401,
+            0.015919575467705727,
+            0.01398828998208046,
+            -0.01597372442483902,
+            -0.003149621421471238,
+            -0.001759817125275731,
+            -0.012463115155696869,
+            -0.023373981937766075,
+            0.014953932724893093,
+            -0.006588033400475979,
+            0.018085505813360214,
+            -0.005906668026000261,
+            -0.001572554581798613,
+            -0.01303167175501585,
+            0.018265999853610992,
+            0.001610909472219646,
+            0.01109135989099741,
+            -0.00012359612446743995,
+            -0.002317092614248395,
+            0.00717915128916502,
+            0.0005073575302958488,
+            -0.017228158190846443,
+            -0.013248263858258724,
+            0.013455832377076149,
+            0.005780322477221489,
+            -0.0029127229936420918,
+            0.02216467075049877,
+            -0.0010936812032014132,
+            0.024366699159145355,
+            0.019836297258734703,
+            0.01277897972613573,
+            -0.006520348135381937,
+            0.008456147275865078,
+            0.026081392541527748,
+            -0.021659286692738533,
+            0.013943166472017765,
+            0.004891389049589634,
+            -0.02061242237687111,
+            -0.03183915093541145,
+            0.0007620233809575438,
+            -0.0006949021480977535,
+            0.010495729744434357,
+            -0.02167733758687973,
+            -0.00934959203004837,
+            -0.015964699909090996,
+            -0.009909124113619328,
+            0.008388461545109749,
+            0.00376330129802227,
+            0.008587004616856575,
+            0.008546394295990467,
+            -0.01327533833682537,
+            0.0032105380669236183,
+            0.0165513064712286,
+            0.0033210907131433487,
+            -0.012255546636879444,
+            0.014006339013576508,
+            0.03220013901591301,
+            0.005450920667499304,
+            -0.031063027679920197,
+            0.016948392614722252,
+            0.0012792516499757767,
+            0.017697442322969437,
+            -0.004329601302742958,
+            -0.007968813180923462,
+            -0.008442609570920467,
+            0.01472831517457962,
+            -0.03218208998441696,
+            -0.0022753532975912094,
+            0.018933827057480812,
+            0.003181207925081253,
+            0.008559931069612503,
+            0.009430814534425735,
+            0.001052506035193801,
+            -0.017905011773109436,
+            0.004864315036684275,
+            -0.014403426088392735,
+            -0.0007287447806447744,
+            0.011795287020504475,
+            -0.01360925193876028,
+            -0.009191660210490227,
+            0.014331229031085968,
+            -0.00767550989985466,
+            -0.02821122296154499,
+            0.03292211890220642,
+            -0.013347536325454712,
+            -0.008013936690986156,
+            0.0372539758682251,
+            0.004769555758684874,
+            -0.005789346992969513,
+            0.011073310859501362,
+            0.015712007880210876,
+            -0.007314521819353104,
+            0.004113008268177509,
+            0.018320146948099136,
+            -0.019980693235993385,
+            -0.011280879378318787,
+            0.018482591956853867,
+            0.03019665740430355,
+            -0.0013221190311014652,
+            0.0007479222840629518,
+            -0.017589146271348,
+            -0.015423217788338661,
+            -0.005455432925373316,
+            0.008361387066543102,
+            0.01804940588772297,
+            -0.012544337660074234,
+            -0.01733645424246788,
+            0.02373497001826763,
+            0.0045236325822770596,
+            0.005062858574092388,
+            0.0071340277791023254,
+            0.022525658831000328,
+            -0.013049720786511898,
+            -0.025594057515263557,
+            0.014656118117272854,
+            -0.010360359214246273,
+            -0.01782378926873207,
+            -0.017760615795850754,
+            0.014087561517953873,
+            -0.012634584680199623,
+            -0.015531513839960098,
+            0.021641237661242485,
+            0.0013367841020226479,
+            -0.0032060258090496063,
+            0.02225491777062416,
+            0.010748421773314476,
+            -0.0167408250272274,
+            -0.03783155605196953,
+            -0.021370496600866318,
+            -0.00038298583240248263,
+            0.009584235027432442,
+            0.006538397632539272,
+            0.009029215201735497,
+            -0.025449663400650024,
+            -0.0029826643876731396,
+            -0.025052575394511223,
+            -0.0005820933729410172,
+            -0.006849749479442835,
+            -0.010694272816181183,
+            -0.004122033249586821,
+            -0.00376330129802227,
+            0.021695386618375778,
+            0.01647910848259926,
+            -0.0009487218922004104,
+            0.006344366353005171,
+            -0.002160288393497467,
+            -0.008623103611171246,
+            -0.011578693985939026,
+            0.01646105945110321,
+            0.016903268173336983,
+            -0.02017923630774021,
+            -0.0028224759735167027,
+            0.011190632358193398,
+            0.0026690559461712837,
+            -0.013509980402886868,
+            -0.019096272066235542,
+            -0.009493988007307053,
+            -0.009173611178994179,
+            -0.024547193199396133,
+            -0.002287762239575386,
+            -0.006023989524692297,
+            -0.025449663400650024,
+            0.012842152267694473,
+            0.04230780899524689,
+            -0.01606397144496441,
+            -0.0032285875640809536,
+            0.006664743181318045,
+            -0.0028292443603277206,
+            0.0072107380256056786,
+            0.005311037879437208,
+            -0.008975067175924778,
+            0.02052217535674572,
+            -0.004376980941742659,
+            -0.004986148327589035,
+            -0.002004612237215042,
+            0.005148593336343765,
+            -0.011389175429940224,
+            -0.02458329126238823,
+            -0.025016477331519127,
+            0.006935484241694212,
+            -0.0006949021480977535,
+            0.0011822361266240478,
+            -0.009087876416742802,
+            0.0012871483340859413,
+            -0.01749889925122261,
+            0.009137512184679508,
+            0.012941424734890461,
+            -0.0024592315312474966,
+            -0.0045258887112140656,
+            -0.013519004918634892,
+            -0.011136483401060104,
+            0.005771297495812178,
+            0.017047664150595665,
+            0.006127773318439722,
+            -0.01800428330898285,
+            0.0007157717482186854,
+            0.012661658227443695,
+            0.02680336870253086,
+            -0.023482277989387512,
+            0.013717548921704292,
+            0.012733856216073036,
+            0.001949335914105177,
+            0.0020666569471359253,
+            0.019728001207113266,
+            -0.007007681764662266,
+            -0.004864315036684275,
+            0.00606008805334568,
+            -0.015242723748087883,
+            0.0029578465037047863,
+            0.026099441573023796,
+            -0.008117720484733582,
+            0.014286105521023273,
+            -0.05353453755378723,
+            -0.010405482724308968,
+            0.0009977937443181872,
+            -0.016352761536836624,
+            -0.0055817789398133755,
+            0.024095958098769188,
+            -0.00013755621330346912,
+            -0.003215050557628274,
+            -0.0055772666819393635,
+            0.012237497605383396,
+            -0.02176758460700512,
+            -0.04353516921401024,
+            0.0018748820293694735,
+            -0.01590152643620968,
+            -0.016361786052584648,
+            0.00014249159721657634,
+            -0.018933827057480812,
+            0.007454404607415199,
+            -0.00963838305324316,
+            -0.008496757596731186,
+            -0.014665142633020878,
+            0.027290701866149902,
+            -0.000774996355175972,
+            -0.0012668427079916,
+            -0.01812160387635231,
+            -0.01184041053056717,
+            -0.015784205868840218,
+            -0.011849435046315193,
+            -0.008749449625611305,
+            -0.0028247321024537086,
+            -0.008226016536355019,
+            -0.0034451803658157587,
+            0.0045258887112140656,
+            0.017444750294089317,
+            -0.020413879305124283,
+            -0.02035973034799099,
+            -0.0028946734964847565,
+            -0.01009864266961813,
+            0.004832728765904903,
+            0.0138619439676404,
+            0.015125402249395847,
+            -0.0054328711703419685,
+            0.007359645329415798,
+            -0.007846979424357414,
+            0.0016571611631661654,
+            -0.010243037715554237,
+            0.03862572833895683,
+            0.0027322289533913136,
+            0.0008325288654305041,
+            -0.0019042124040424824,
+            -8.156604053510819e-06,
+            0.005446408409625292,
+            -0.004977123811841011,
+            -0.017507923766970634,
+            0.010549877770245075,
+            -0.02326568402349949,
+            0.021406596526503563,
+            -0.010279136709868908,
+            0.01017084065824747,
+            -0.005762272980064154,
+            -0.026586774736642838,
+            -0.002233613980934024,
+            0.024781834334135056,
+            0.0010919891064986587,
+            0.009227759204804897,
+            -0.01782378926873207,
+            -0.008636641316115856,
+            -0.0030751675367355347,
+            -0.022291017696261406,
+            -0.009584235027432442,
+            -0.00901567842811346,
+            0.002319348743185401,
+            -0.010576952248811722,
+            0.014935883693397045,
+            0.017580121755599976,
+            0.005053833592683077,
+            -0.006831700447946787,
+            0.009710581041872501,
+            -0.02979956939816475,
+            0.008781036362051964,
+            -0.003436155617237091,
+            -0.005473482422530651,
+            -0.019583605229854584,
+            0.019421162083745003,
+            0.0001576502836542204,
+            -0.0027389973402023315,
+            -0.002817963482812047,
+            0.02100950852036476,
+            0.008361387066543102,
+            0.009683506563305855,
+            -0.005468970164656639,
+            -0.004638697486370802,
+            -0.01221042312681675,
+            0.003219562815502286,
+            -0.00019558223721105605,
+            -0.0002899185928981751,
+            -0.0016176780918613076,
+            0.005320062395185232,
+            -0.010486705228686333,
+            -0.025142822414636612,
+            0.017192060127854347,
+            -0.00894348043948412,
+            -0.006664743181318045,
+            0.017155960202217102,
+            -0.01514345221221447,
+            0.0011213193647563457,
+            0.004088190384209156,
+            0.0004159824165981263,
+            0.004162644501775503,
+            0.004760530777275562,
+            0.008884820155799389,
+            -0.0063082673586905,
+            -0.020558273419737816,
+            0.011235755868256092,
+            -0.013482906855642796,
+            -0.005139568354934454,
+            0.021911978721618652,
+            -0.006529372651129961,
+            -0.002199771348387003,
+            -0.010486705228686333,
+            -0.02183978073298931,
+            0.01481856219470501,
+            -0.02689361572265625,
+            0.0016481364145874977,
+            -0.0029804082587361336,
+            0.01588347740471363,
+            -0.011939682066440582,
+            0.02368082106113434,
+            -0.006655718665570021,
+            -0.006263143848627806,
+            0.0021208052057772875,
+            -0.0260633435100317,
+            0.008198942989110947,
+            -0.035449035465717316,
+            0.004338625818490982,
+            -0.01125380489975214,
+            0.029258087277412415,
+            0.018645036965608597,
+            -0.006425588857382536,
+            0.011398199945688248,
+            -0.02904149517416954,
+            0.004214536398649216,
+            0.01411463599652052,
+            0.0038783662021160126,
+            -0.0040295301005244255,
+            0.007449892349541187,
+            -0.010865742340683937,
+            -0.023536425083875656,
+            -0.02954687736928463,
+            0.001267970772460103,
+            -0.03481730446219444,
+            0.0013785234186798334,
+            0.0012916605919599533,
+            -0.016912294551730156,
+            0.010324260219931602,
+            -0.006759502459317446,
+            -0.020666569471359253,
+            0.017228158190846443,
+            0.0031157787889242172,
+            -0.012878251262009144,
+            -0.0042777094058692455,
+            0.006750477943569422,
+            0.02160513959825039,
+            0.014638068154454231,
+            -0.009737654589116573,
+            0.0008404254913330078,
+            0.010585976764559746,
+            -0.006389489863067865,
+            -0.010197914205491543,
+            -0.022796399891376495,
+            -0.00883067212998867,
+            0.004004712216556072,
+            0.006443637888878584,
+            -0.006664743181318045,
+            0.02001679129898548,
+            0.011226730421185493,
+            0.0012205911334604025,
+            0.01352803036570549,
+            -0.01389804296195507,
+            0.018338195979595184,
+            -0.0031090101692825556,
+            -0.014629043638706207,
+            0.0019290302880108356,
+            -0.03593636676669121,
+            0.005965328775346279,
+            -0.006213508080691099,
+            0.025756502524018288,
+            -0.01439440157264471,
+            -0.0017530486220493913,
+            -0.018455518409609795,
+            -0.0025336854159832,
+            0.015684934332966805,
+            -0.012598485685884953,
+            -0.011280879378318787,
+            0.013537054881453514,
+            -0.013176066800951958,
+            -0.0020305581856518984,
+            0.016930343583226204,
+            -0.010089618153870106,
+            0.01233676914125681,
+            -0.007188176270574331,
+            -0.00027215119916945696,
+            -0.00182637432590127,
+            -0.0012566898949444294,
+            -0.004566499963402748,
+            -0.0013412964763119817,
+            0.0028337568510323763,
+            -0.020143138244748116,
+            0.029582977294921875,
+            -0.013509980402886868,
+            -0.010766470804810524,
+            0.004250635392963886,
+            -0.003032300155609846,
+            0.021875880658626556,
+            -0.011650891974568367,
+            0.00391220860183239,
+            0.000977488118223846,
+            -0.010342310182750225,
+            -0.008167356252670288,
+            -0.004559731110930443,
+            0.013013621792197227,
+            0.026279935613274574,
+            -0.019980693235993385,
+            0.009394715540111065,
+            0.00997229665517807,
+            -0.02350032702088356,
+            -0.01869918406009674,
+            -0.004543938208371401,
+            -0.013446807861328125,
+            0.01788696087896824,
+            -0.00039454875513911247,
+            -0.005658488720655441,
+            0.0036843351554125547,
+            0.005035784561187029,
+            -0.014340253546833992,
+            -0.000682493147905916,
+            -0.003898671828210354,
+            0.004345394670963287,
+            -0.006281193345785141,
+            0.011768212541937828,
+            0.0006520348251797259,
+            0.02059437334537506,
+            -0.00926837045699358,
+            -0.014132685028016567,
+            0.0064842491410672665,
+            -0.00997229665517807,
+            0.009882049635052681,
+            -0.021208051592111588,
+            0.0010152790928259492,
+            0.0074724541045725346,
+            -0.005098957102745771,
+            0.004841753281652927,
+            0.009484962560236454,
+            -0.02739899791777134,
+            0.005969841033220291,
+            0.005482506938278675,
+            -0.0015691702719777822,
+            0.016497157514095306,
+            0.009503012523055077,
+            -0.0007095672772265971,
+            0.0006988504319451749,
+            -0.014719290658831596,
+            0.019385062158107758,
+            0.006921947468072176,
+            0.019096272066235542,
+            0.006773039698600769,
+            0.015522489324212074,
+            -0.00013586408749688417,
+            -0.005622390192002058,
+            0.006782064214348793,
+            -0.006064600311219692,
+            -0.004595830105245113,
+            0.03557537868618965,
+            -0.0006255247280932963,
+            0.012300670146942139,
+            0.006696329917758703,
+            -0.00030599383171647787,
+            0.0056765382178127766,
+            0.0120028555393219,
+            0.0012454090174287558,
+            -0.017697442322969437,
+            0.011235755868256092,
+            -0.015991773456335068,
+            -0.019637754186987877,
+            0.0019380550365895033,
+            0.009394715540111065,
+            -0.00780185591429472,
+            -0.0026081393007189035,
+            0.026099441573023796,
+            0.008550906553864479,
+            0.009358617477118969,
+            0.003530915128067136,
+            -0.007901127450168133,
+            -0.002162544522434473,
+            0.005956304259598255,
+            0.000220682195504196,
+            0.001759817125275731,
+            -0.004805654287338257,
+            -0.02581065148115158,
+            0.003303041448816657,
+            -0.005446408409625292,
+            -0.00340456934645772,
+            -0.0015026130713522434,
+            0.018383320420980453,
+            -0.01352803036570549,
+            0.01544126681983471,
+            0.006033014040440321,
+            -0.004113008268177509,
+            -0.014304154552519321,
+            0.0062315575778484344,
+            0.004081421997398138,
+            0.020883163437247276,
+            -0.001704540802165866,
+            -0.019746050238609314,
+            -0.0227603018283844,
+            0.003797143930569291,
+            -5.8026020269608125e-05,
+            -0.0034880477469414473,
+            0.0022945308592170477,
+            -0.002249407349154353,
+            0.008216992020606995,
+            -0.0034857916180044413,
+            -0.006673768162727356,
+            0.002287762239575386,
+            -0.006154847331345081,
+            0.02017923630774021,
+            -0.002326117129996419,
+            -0.0006571111734956503,
+            0.0023667283821851015,
+            -0.004013736732304096,
+            0.001205925946123898,
+            -0.002102755941450596,
+            0.016794972121715546,
+            -0.017192060127854347,
+            -0.00393702695146203,
+            -0.009078850969672203,
+            -0.017480850219726562,
+            0.0014721547486260533,
+            -0.009259345009922981,
+            0.006664743181318045,
+            0.013943166472017765,
+            0.014764414168894291,
+            -0.004990661051124334,
+            -0.0017846350092440844,
+            0.01918651908636093,
+            0.011677965521812439,
+            0.0010474296286702156,
+            0.011353076435625553,
+            0.0009002141305245459,
+            -0.010992088355123997,
+            -0.0006266528507694602,
+            -0.010910865850746632,
+            -0.007607824634760618,
+            0.023554475978016853,
+            0.02274225279688835,
+            0.012318720109760761,
+            0.0025742966681718826,
+            -0.0023486788850277662,
+            -0.0029285161290317774,
+            0.006294730585068464,
+            0.000976360053755343,
+            -0.011885534040629864,
+            0.0021456230897456408,
+            0.0006243966636247933,
+            -0.022615905851125717,
+            0.013925116509199142,
+            -0.02929418720304966,
+            0.0240057110786438,
+            -0.0018365271389484406,
+            0.009187147952616215,
+            0.019547507166862488,
+            -0.0011280878679826856,
+            0.013464856892824173,
+            0.01096501387655735,
+            0.0019414392299950123,
+            -0.02465548925101757,
+            -0.00811320822685957,
+            -0.010378408245742321,
+            -0.005256889387965202,
+            0.006989632733166218,
+            0.009114949963986874,
+            0.000154830064275302,
+            0.0030841922853142023,
+            -0.0021241893991827965,
+            0.001891803345642984,
+            -0.01969190314412117,
+            0.011903583072125912,
+            0.0022178208455443382,
+            -0.010215964168310165,
+            -0.023933513090014458,
+            -0.005703612230718136,
+            0.009864000603556633,
+            0.0022099241614341736,
+            -0.003519634250551462,
+            0.008365900255739689,
+            -0.0063308291137218475,
+            -0.0008658074657432735,
+            0.00024973045219667256,
+            -0.003395544597879052,
+            0.0016526487888768315,
+            -0.005866057239472866,
+            0.005117006599903107,
+            0.005766785237938166,
+            -0.0008415535558015108,
+            0.008230529725551605,
+            -0.01046865526586771,
+            0.013952190987765789,
+            -0.011407224461436272,
+            0.005613365210592747,
+            -0.008537368848919868,
+            0.006831700447946787,
+            0.008893844671547413,
+            0.005861544515937567,
+            0.013148992322385311,
+            -0.0003739611420314759,
+            -0.0013164785923436284,
+            -0.016217391937971115,
+            -0.03485340252518654,
+            0.006366928108036518,
+            -0.0009064186015166342,
+            0.006588033400475979,
+            -0.02160513959825039,
+            0.0050809080712497234,
+            0.012580436654388905,
+            -0.017255231738090515,
+            0.0010169711895287037,
+            0.005784834735095501,
+            0.015910550951957703,
+            -0.011154533363878727,
+            0.000819555833004415,
+            0.007043780758976936,
+            0.018825531005859375,
+            0.0031202910467982292,
+            0.004189718514680862,
+            -0.02225491777062416,
+            -0.012670683674514294,
+            0.015341995283961296,
+            -0.030738139525055885,
+            0.008641153573989868,
+            0.020702669396996498,
+            -0.009286419488489628,
+            0.03109912760555744,
+            0.01788696087896824,
+            -0.013573153875768185,
+            -0.0073731825686991215,
+            -0.008573467843234539,
+            0.022453460842370987,
+            -0.0067324284464120865,
+            0.009683506563305855,
+            -0.014620019122958183,
+            0.011037211865186691,
+            0.0046838209964334965,
+            0.005202741362154484,
+            0.010983063839375973,
+            -0.027128256857395172,
+            0.007702583912760019,
+            4.857828389503993e-05,
+            -0.026586774736642838,
+            0.00804552249610424,
+            -0.005771297495812178,
+            0.020919261500239372,
+            -0.001577066839672625,
+            0.012607510201632977,
+            0.024962328374385834,
+            0.00027088209753856063,
+            -0.012381892651319504,
+            -0.0073686703108251095,
+            0.009900099597871304,
+            -0.012860202230513096,
+            0.028355617076158524,
+            -0.00042472509085200727,
+            0.0009227758855558932,
+            -0.005049321334809065,
+            0.008352362550795078,
+            -0.017598170787096024,
+            0.004701870493590832,
+            -0.013970240019261837,
+            -0.016731800511479378,
+            0.0011574181262403727,
+            0.012147250585258007,
+            -0.0006035270052962005,
+            0.013257289305329323,
+            0.011073310859501362,
+            0.0040385546162724495,
+            0.014195858500897884,
+            0.003397800726816058,
+            0.0023283734917640686,
+            0.022309066727757454,
+            0.010450606234371662,
+            0.001882778713479638,
+            -0.0018940595909953117,
+            -0.0022798655554652214,
+            0.03169475868344307,
+            -6.616940663661808e-05,
+            -0.004652234259992838,
+            0.010874766856431961,
+            0.01432220358401537,
+            0.006384977605193853,
+            0.01088379230350256,
+            0.015712007880210876,
+            0.0012713550822809339,
+            0.003219562815502286,
+            0.01046865526586771,
+            0.002475024899467826,
+            0.01240896712988615,
+            0.003131571924313903,
+            -0.006493274122476578,
+            -0.0038084248080849648,
+            0.006218020338565111,
+            0.013284362852573395,
+            -0.01000839564949274,
+            0.017489874735474586,
+            0.009493988007307053,
+            0.00943983905017376,
+            -0.005044809076935053,
+            0.007070854771882296,
+            0.012950449250638485,
+            0.0034384119790047407,
+            -0.007291960064321756,
+            -0.011389175429940224,
+            0.002443438395857811,
+            -0.027759985998272896,
+            0.012363843619823456,
+            0.0014349278062582016,
+            -0.016822047531604767,
+            -0.007314521819353104,
+            -0.003946051467210054,
+            -0.015919575467705727,
+            0.0032872480805963278,
+            -0.002628444693982601,
+            0.026117490604519844,
+            -0.0007608952582813799,
+            -0.02532331645488739,
+            -0.008411023765802383,
+            0.016993515193462372,
+            -0.007341595832258463,
+            0.005153105594217777,
+            0.012255546636879444,
+            0.028084876015782356,
+            0.017020590603351593,
+            -0.0006706482381559908,
+            -0.003896415699273348,
+            -0.004904926288872957,
+            -0.003957332111895084,
+            0.010775495320558548,
+            -0.0159286018460989,
+            -0.008591517806053162,
+            -0.01647910848259926,
+            -0.016587404534220695,
+            0.0023712406400591135,
+            0.009575209580361843,
+            -0.004505583085119724,
+            -0.007201713044196367,
+            -0.0020192773081362247,
+            0.001828630454838276,
+            -0.003657260909676552,
+            0.02043192833662033,
+            -0.01860893703997135,
+            -0.02781413495540619,
+            0.0031586461700499058,
+            -0.0038851348217576742,
+            0.0073776948265731335,
+            0.006660230923444033,
+            0.014656118117272854,
+            -0.002908210502937436,
+            -0.012616534717381,
+            0.04371566325426102,
+            -0.009277394972741604,
+            0.017074737697839737,
+            -0.006750477943569422,
+            0.016325687989592552,
+            0.015062229707837105,
+            0.0060871620662510395,
+            0.0054238466545939445,
+            0.019258717074990273,
+            -0.01647910848259926,
+            -0.002655518939718604,
+            0.007639411371201277,
+            0.009864000603556633,
+            0.02498037740588188,
+            0.008474196307361126,
+            0.0010474296286702156,
+            -0.02044997736811638,
+            -0.003066142788156867,
+            -0.019258717074990273,
+            -0.0009453376405872405,
+            -0.015260772779583931,
+            -0.003970869351178408,
+            0.01233676914125681,
+            -0.02218271978199482,
+            -0.013248263858258724,
+            -0.02160513959825039,
+            -0.006375952623784542,
+            0.009818877093493938,
+            0.0006221404764801264,
+            0.018437469378113747,
+            0.00897957943379879,
+            0.00789661519229412,
+            0.007869540713727474,
+            0.015161501243710518,
+            -0.006014964543282986,
+            0.002693873830139637,
+            0.00550055643543601,
+            0.0009013422532007098,
+            0.011578693985939026,
+            -0.00045151717495173216,
+            -0.013762672431766987,
+            0.012580436654388905,
+            0.0045281448401510715,
+            0.009236783720552921,
+            -0.02582870051264763,
+            -0.023789117112755775,
+            0.01112745888531208,
+            0.0067369407042860985,
+            0.004092703107744455,
+            -0.013356560841202736,
+            0.010658174753189087,
+            -0.003269198816269636,
+            -0.006466199643909931,
+            -0.005762272980064154,
+            0.0036346991546452045,
+            -0.016993515193462372,
+            -0.007783806417137384,
+            0.00637144036591053,
+            0.01664155349135399,
+            0.000557275430765003,
+            0.014881735667586327,
+            0.010288161225616932,
+            0.02283249981701374,
+            0.012733856216073036,
+            -0.026099441573023796,
+            -0.002752534346655011,
+            0.007143052760511637,
+            0.002515635918825865,
+            -0.023608623072504997,
+            -0.009340567514300346,
+            -0.002231357851997018,
+            -0.0010158431250602007,
+            0.026947762817144394,
+            -0.0025449662934988737,
+            0.011362100951373577,
+            -0.009295444004237652,
+            -0.004399542696774006,
+            -0.018843580037355423,
+            0.031008880585432053,
+            -0.003934770356863737,
+            0.005793859250843525,
+            -0.007607824634760618,
+            -0.013762672431766987,
+            -0.012165299616754055,
+            -0.004742481280118227,
+            0.00123638438526541,
+            0.06407539546489716,
+            -0.017562072724103928,
+            0.010477680712938309,
+            -0.008456147275865078,
+            0.006046551279723644,
+            0.022236868739128113,
+            0.010279136709868908,
+            0.0060826498083770275,
+            -0.011163557879626751,
+            -0.01472831517457962,
+            -0.004823703784495592,
+            -0.002286634175106883,
+            0.008451634086668491,
+            0.005157617852091789,
+            -0.0058344705030322075,
+            -0.02144269458949566,
+            -0.025016477331519127,
+            -0.023554475978016853,
+            -0.0012172068236395717,
+            -0.01022498868405819,
+            -0.003677566535770893,
+            0.00041711051017045975,
+            0.0013548335991799831,
+            0.008794573135674,
+            0.01960165612399578,
+            -0.007643923629075289,
+            -0.02391546405851841,
+            -0.027850233018398285,
+            0.014574895612895489,
+            -0.04227171093225479,
+            0.01860893703997135,
+            0.0004881800268776715,
+            -0.027453146874904633,
+            -0.012643609195947647,
+            0.013049720786511898,
+            0.017092788591980934,
+            -0.010901841334998608,
+            0.021731484681367874,
+            0.010685248300433159,
+            -0.011470397934317589,
+            0.0029307722579687834,
+            0.022633954882621765,
+            0.026586774736642838,
+            0.012634584680199623,
+            0.0032060258090496063,
+            0.01853674091398716,
+            0.013221190311014652,
+            -0.018013307824730873,
+            0.007698071654886007,
+            0.01567590981721878,
+            -0.012463115155696869,
+            0.008122232742607594,
+            -0.010432557202875614,
+            0.012986548244953156,
+            -0.005351649131625891,
+            -0.001114550861530006,
+            0.030665941536426544,
+            -0.028608309105038643,
+            0.021208051592111588,
+            -0.031171325594186783,
+            0.008000398986041546,
+            -0.004990661051124334,
+            -0.0004560295201372355,
+            0.010712322778999805,
+            -0.051513005048036575,
+            0.03245283290743828,
+            0.018717234954237938,
+            -5.53468125872314e-05,
+            -0.026297984644770622,
+            -0.01312191877514124,
+            0.021460743620991707,
+            0.007792831398546696,
+            -0.029907865449786186,
+            -0.019475309178233147,
+            0.009548136033117771,
+            -0.0010711194481700659,
+            0.0011500855907797813,
+            -0.007779294159263372,
+            -0.009173611178994179,
+            0.003406825475394726,
+            -0.02615359053015709,
+            0.02781413495540619,
+            -0.018455518409609795,
+            0.02285054884850979,
+            0.007670997641980648,
+            0.006384977605193853,
+            -0.0005039732786826789,
+            -0.008510295301675797,
+            -0.009620334021747112,
+            0.01680399663746357,
+            -0.0053290873765945435,
+            -0.002637469442561269,
+            -0.0173635296523571,
+            -0.0045349132269620895,
+            0.0067188916727900505,
+            0.0027886333409696817,
+            0.0022166927810758352,
+            0.011118434369564056,
+            -0.001003434183076024,
+            0.014051462523639202,
+            0.017372554168105125,
+            0.02622578665614128,
+            0.002996201394125819,
+            0.001920005539432168,
+            0.004189718514680862,
+            0.015991773456335068,
+            -0.0003677566419355571,
+            -0.00531555013731122,
+            0.0037858630530536175,
+            0.01088379230350256,
+            0.013879992999136448,
+            0.01175016351044178,
+            0.008284677751362324,
+            -0.00035986004513688385,
+            0.0044424100778996944,
+            0.016948392614722252,
+            0.020233385264873505,
+            -0.01026108767837286,
+            0.00556372944265604,
+            0.009069826453924179,
+            -0.00795076321810484,
+            0.0038512919563800097,
+            -0.007287447806447744,
+            0.007061830256134272,
+            -0.03245283290743828,
+            0.02936638332903385,
+            -0.0020666569471359253,
+            -0.008550906553864479,
+            -0.006168384570628405,
+            -0.013672425411641598,
+            0.013194115832448006,
+            -0.019637754186987877,
+            0.004367956425994635,
+            -0.02507062442600727,
+            0.008365900255739689,
+            0.002070041373372078,
+            -0.00022604061814490706,
+            0.030575694516301155,
+            0.012688732706010342,
+            0.01795915886759758,
+            -0.00787405390292406,
+            0.06663841009140015,
+            -0.004361187573522329,
+            -0.0046928455121815205,
+            -0.029276136308908463,
+            -0.018518690019845963,
+            -0.009412765502929688,
+            0.0005434563499875367,
+            0.0006892617093399167,
+            0.007413793820887804,
+            -0.0017293586861342192,
+            -0.00839297380298376,
+            0.012417991645634174,
+            -0.01141624990850687,
+            -0.013004597276449203,
+            -0.0007862772908993065,
+            0.015802254900336266,
+            0.005022247321903706,
+            -0.008383949287235737,
+            0.016054946929216385,
+            0.0030187631491571665,
+            0.0011709552491083741,
+            -0.006949021480977535,
+            0.012201398611068726,
+            -0.005107982084155083,
+            0.009078850969672203,
+            0.02573845349252224,
+            -0.003400056855753064,
+            -0.0020170211791992188,
+            0.004103983752429485,
+            -0.01187650952488184,
+            0.0018004282610490918,
+            0.006660230923444033,
+            -0.011037211865186691,
+            0.002903698245063424,
+            0.002323861001059413,
+            0.005536655429750681,
+            -0.010243037715554237,
+            -0.011407224461436272,
+            -0.0008404254913330078,
+            0.005649464204907417,
+            0.01554956380277872,
+            0.0001692131772870198,
+            0.0042777094058692455,
+            0.022652005776762962,
+            -0.01680399663746357,
+            -0.012075052596628666,
+            -0.02283249981701374,
+            0.007147565018385649,
+            0.01423195656388998,
+            0.002506611403077841,
+            -0.018987976014614105,
+            -0.012896301224827766,
+            0.02234516479074955,
+            0.009530086070299149,
+            0.034456316381692886,
+            0.006375952623784542,
+            0.0017146936152130365,
+            0.011154533363878727,
+            0.02756144292652607,
+            0.010423531755805016,
+            -0.019764099270105362,
+            0.0020621446892619133,
+            -0.012156275101006031
+          ],
+          "title": "8 Future Work"
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "1deb6775-c7ea-4a7a-8654-e306d12e688d",
+        "properties": {
+          "page_content": "9 Conclusion 27 ",
+          "keyphrases": [],
+          "embedding": [
+            -0.0026620388962328434,
+            0.009736468084156513,
+            -0.01410017628222704,
+            0.027085084468126297,
+            -0.02409333363175392,
+            -0.023739280179142952,
+            0.002801447408273816,
+            0.09375333786010742,
+            -0.007904241792857647,
+            0.016392672434449196,
+            -0.00255582295358181,
+            -0.03965397924184799,
+            -0.002425265731289983,
+            -0.03738803789019585,
+            -0.047655586153268814,
+            0.007165155373513699,
+            -0.002056828932836652,
+            -0.002341178013011813,
+            -0.018729424104094505,
+            -0.010240994393825531,
+            -0.027563055977225304,
+            -0.00033054203959181905,
+            0.017215846106410027,
+            0.05247071012854576,
+            -0.00470448425039649,
+            -0.0022095143795013428,
+            0.020588204264640808,
+            0.01972077414393425,
+            -0.032856155186891556,
+            0.04528342932462692,
+            -0.008333531208336353,
+            -0.01013477798551321,
+            0.036502905189991,
+            -0.024925358593463898,
+            -0.01793280430138111,
+            0.013728420250117779,
+            0.02297806553542614,
+            -0.014091324992477894,
+            0.013666460290551186,
+            -0.018764829263091087,
+            -0.01830456033349037,
+            0.01579963229596615,
+            -0.004806274548172951,
+            -0.004509754944592714,
+            -0.004859382752329111,
+            -0.028182649984955788,
+            -0.021101580932736397,
+            -0.06061394140124321,
+            -0.015905847772955894,
+            -0.003113457001745701,
+            -0.0059967790730297565,
+            -0.0376712791621685,
+            0.01110842451453209,
+            -0.003370145568624139,
+            0.009320455603301525,
+            0.03606033697724342,
+            0.028288865461945534,
+            0.08199876546859741,
+            0.0022404941264539957,
+            0.011338559910655022,
+            -0.021455634385347366,
+            -0.029917510226368904,
+            -0.009603697806596756,
+            -0.00012419528502505273,
+            -0.009072617627680302,
+            -0.022376174107193947,
+            0.011188087053596973,
+            -0.01899496465921402,
+            -0.006395089440047741,
+            -0.028430486097931862,
+            0.01365760900080204,
+            0.021136987954378128,
+            0.005231138784438372,
+            0.016764428466558456,
+            -0.007868836633861065,
+            0.0030692003201693296,
+            0.05133774131536484,
+            0.036078039556741714,
+            -0.009506333619356155,
+            0.04036208614706993,
+            -0.007550188340246677,
+            -0.010471128858625889,
+            0.008178633637726307,
+            0.028182649984955788,
+            0.04556667059659958,
+            0.010612750425934792,
+            -0.0727933794260025,
+            -0.09934738278388977,
+            -0.010108224116265774,
+            0.004580565728247166,
+            0.022712524980306625,
+            -0.03039548359811306,
+            -0.02081833966076374,
+            -0.026518598198890686,
+            -0.046416398137807846,
+            0.02150874398648739,
+            0.002179641043767333,
+            0.005049686413258314,
+            -0.03901668265461922,
+            0.008023735135793686,
+            0.018145235255360603,
+            0.017313210293650627,
+            0.009824981912970543,
+            0.010559642687439919,
+            0.02203982323408127,
+            -0.011250046081840992,
+            0.024358872324228287,
+            0.0323781818151474,
+            0.009718765504658222,
+            0.05640070512890816,
+            -0.020729824900627136,
+            -0.0049478961154818535,
+            -0.03560006618499756,
+            -0.029563458636403084,
+            0.011418221518397331,
+            -0.004403539001941681,
+            -0.021225500851869583,
+            -0.04762018099427223,
+            0.014162135310471058,
+            0.017667263746261597,
+            -0.009585995227098465,
+            0.004894787911325693,
+            0.02825346030294895,
+            -0.03774208948016167,
+            -0.02223455347120762,
+            0.019826989620923996,
+            0.03239588439464569,
+            0.008992956019937992,
+            -0.007616573479026556,
+            0.010957952588796616,
+            0.006669480819255114,
+            0.015967806801199913,
+            -0.024695223197340965,
+            -0.005629448685795069,
+            0.018410775810480118,
+            -0.024128738790750504,
+            0.010046265088021755,
+            0.0012579959584400058,
+            -0.001970528392121196,
+            0.030802644789218903,
+            -0.044398292899131775,
+            -0.008298126049339771,
+            -0.005554212257266045,
+            0.02388090081512928,
+            -0.006204785779118538,
+            0.012869840487837791,
+            -0.01720699481666088,
+            0.029811294749379158,
+            0.0013188488082960248,
+            0.01239186804741621,
+            0.0006068696384318173,
+            0.011639504693448544,
+            -0.02643008530139923,
+            0.0038591818884015083,
+            -0.002400924451649189,
+            -0.03961857408285141,
+            0.031369131058454514,
+            0.012728218920528889,
+            0.0037750941701233387,
+            0.03660912066698074,
+            0.0031532880384474993,
+            0.054524220526218414,
+            -0.020499691367149353,
+            -0.0051381997764110565,
+            0.009727616794407368,
+            0.00795734953135252,
+            0.027580758556723595,
+            -0.015976659953594208,
+            -0.04273424297571182,
+            0.022871848195791245,
+            0.03616655245423317,
+            0.0002338688645977527,
+            0.025244006887078285,
+            -0.03558236360549927,
+            0.007231540512293577,
+            -0.007213837932795286,
+            -0.008218464441597462,
+            0.055834218859672546,
+            -0.0100551163777709,
+            0.024164143949747086,
+            -0.009453224949538708,
+            0.019579151645302773,
+            -0.01991550251841545,
+            0.004472136963158846,
+            -0.0227656327188015,
+            -0.0074705262668430805,
+            0.06341096013784409,
+            -0.03887506201863289,
+            0.03274993970990181,
+            0.01571997068822384,
+            -0.002052403287962079,
+            -0.022482389584183693,
+            0.03370588272809982,
+            0.010046265088021755,
+            0.004113657865673304,
+            -0.06093258783221245,
+            -0.0016231135232374072,
+            0.035210609436035156,
+            -0.014498486183583736,
+            0.011781126260757446,
+            -0.014560445211827755,
+            -0.022287661209702492,
+            0.018481586128473282,
+            0.010754371993243694,
+            -0.010710114613175392,
+            -0.0028280015103518963,
+            0.005558638367801905,
+            0.010471128858625889,
+            0.014294905588030815,
+            -0.0005191860836930573,
+            0.02747454307973385,
+            0.0150118637830019,
+            -0.008116673678159714,
+            0.01593240164220333,
+            -0.013232745230197906,
+            0.010692412033677101,
+            -0.006930594798177481,
+            0.02209293097257614,
+            -0.013002610765397549,
+            -0.030625617131590843,
+            0.014409972354769707,
+            0.013551393523812294,
+            0.04145964980125427,
+            -0.0002031657932093367,
+            -0.008138801902532578,
+            0.04850531369447708,
+            -0.03760046884417534,
+            0.04698288440704346,
+            0.02729751542210579,
+            -0.04698288440704346,
+            0.009798427112400532,
+            0.018906449899077415,
+            -0.01830456033349037,
+            0.003505128435790539,
+            -0.0057710702531039715,
+            0.008882313966751099,
+            0.0009399011032655835,
+            0.014206391759216785,
+            0.003244014224037528,
+            0.02928021550178528,
+            0.04528342932462692,
+            -0.02372157759964466,
+            -0.028660621494054794,
+            0.0412118136882782,
+            -0.025615762919187546,
+            0.007258094381541014,
+            -0.0412118136882782,
+            0.06741176545619965,
+            0.017260102555155754,
+            0.014082473702728748,
+            0.015224295668303967,
+            -0.0005056324880570173,
+            0.0017757989699020982,
+            0.03958316892385483,
+            0.019490638747811317,
+            0.04740775004029274,
+            0.009957751259207726,
+            -0.0043836236000061035,
+            -0.053249627351760864,
+            0.032094940543174744,
+            -0.0034498076420277357,
+            0.0035715135745704174,
+            -0.030324673280119896,
+            0.01626875251531601,
+            0.02497846633195877,
+            0.002735062502324581,
+            0.02207522839307785,
+            -0.010356061160564423,
+            0.021809687837958336,
+            0.02007482759654522,
+            0.05640070512890816,
+            -0.014170986600220203,
+            -0.00489036226645112,
+            0.01175457239151001,
+            -0.018393073230981827,
+            -0.031953319907188416,
+            0.032271966338157654,
+            -0.018516991287469864,
+            -0.013347812928259373,
+            0.019065774977207184,
+            -0.0007838963065296412,
+            0.03476804122328758,
+            -0.005558638367801905,
+            -0.020712122321128845,
+            -0.017410574480891228,
+            -0.022110633552074432,
+            -0.0010062861256301403,
+            0.01040916983038187,
+            -0.030430888757109642,
+            0.00484168017283082,
+            0.013480582274496555,
+            -0.01138281635940075,
+            -0.009152280166745186,
+            0.01111727673560381,
+            -0.004633673466742039,
+            0.002014785073697567,
+            0.018853342160582542,
+            0.007492654956877232,
+            -0.008523834869265556,
+            -0.0014095749938860536,
+            0.003102392889559269,
+            0.0005645491764880717,
+            -0.03441398963332176,
+            -0.013595649972558022,
+            -0.008453024551272392,
+            0.01607402414083481,
+            0.016498887911438942,
+            0.03584790602326393,
+            -0.0714125707745552,
+            0.011789977550506592,
+            0.035901013761758804,
+            0.016339562833309174,
+            0.000724703015293926,
+            -0.04418586194515228,
+            -0.029775889590382576,
+            0.020375771448016167,
+            0.01302031334489584,
+            0.010046265088021755,
+            -0.013454028405249119,
+            -0.004485413897782564,
+            0.012763625010848045,
+            -0.011728018522262573,
+            -0.018605506047606468,
+            0.030625617131590843,
+            0.020499691367149353,
+            -0.03260831534862518,
+            -0.021243203431367874,
+            0.008430896326899529,
+            0.01988009735941887,
+            -0.012444976717233658,
+            0.025633465498685837,
+            -0.022482389584183693,
+            0.019490638747811317,
+            -0.001468215137720108,
+            -0.014799431897699833,
+            -0.02713819220662117,
+            -0.023438334465026855,
+            0.02025185339152813,
+            -0.03657371550798416,
+            -0.006908466573804617,
+            0.019472936168313026,
+            0.0032019703648984432,
+            -0.008599071763455868,
+            0.0008060246473178267,
+            -0.01640152372419834,
+            -0.038237765431404114,
+            0.029563458636403084,
+            -0.0561174601316452,
+            -0.003299335017800331,
+            0.01527740340679884,
+            0.0009227516129612923,
+            -0.009152280166745186,
+            0.008160930126905441,
+            0.01793280430138111,
+            -0.0023124110884964466,
+            -0.008006032556295395,
+            -0.028111839666962624,
+            -0.027580758556723595,
+            -0.009470928460359573,
+            -0.008935422636568546,
+            0.013073421083390713,
+            0.006381812505424023,
+            0.014896796084940434,
+            -0.006311001721769571,
+            -0.04907179996371269,
+            -0.025066979229450226,
+            0.016339562833309174,
+            0.01293179951608181,
+            0.020570501685142517,
+            0.0008663243497721851,
+            -0.02223455347120762,
+            -0.002160832053050399,
+            0.021225500851869583,
+            -0.017083076760172844,
+            -0.01575537584722042,
+            -0.006549987476319075,
+            -0.013648757711052895,
+            0.011612950824201107,
+            -0.026890354230999947,
+            0.011294303461909294,
+            -0.02892616204917431,
+            0.012276801280677319,
+            0.006704885978251696,
+            0.004992152564227581,
+            0.0151092279702425,
+            -0.0027306366246193647,
+            0.012498084455728531,
+            0.017180440947413445,
+            -0.0073997159488499165,
+            -0.04393802583217621,
+            -0.04854071885347366,
+            -0.012329909019172192,
+            -0.004717761185020208,
+            -0.02095996029675007,
+            0.00849728100001812,
+            0.011993558146059513,
+            0.0390520878136158,
+            0.0034984899684786797,
+            0.0045849913731217384,
+            -0.01707422360777855,
+            -0.016773279756307602,
+            0.009293901734054089,
+            -0.006572116166353226,
+            -0.009621400386095047,
+            0.021030770614743233,
+            0.031227508559823036,
+            0.0006782335112802684,
+            -0.023650765419006348,
+            -0.008315828628838062,
+            -0.008979679085314274,
+            0.0402204655110836,
+            -0.03604263439774513,
+            0.0002623590989969671,
+            -0.046239372342824936,
+            0.029970619827508926,
+            0.00285012973472476,
+            0.020481988787651062,
+            0.0024186272639781237,
+            -0.027279812842607498,
+            -0.0227656327188015,
+            -0.021632662042975426,
+            -0.0019583578687161207,
+            0.027616163715720177,
+            0.001016243826597929,
+            -0.021650364622473717,
+            0.04138883948326111,
+            -0.01610942929983139,
+            0.019101180136203766,
+            0.00011299282050458714,
+            0.011312006041407585,
+            -0.04167208448052406,
+            0.007231540512293577,
+            -0.02044658362865448,
+            0.01989779993891716,
+            0.04662882909178734,
+            -0.0034741489216685295,
+            -0.03331642225384712,
+            0.011533289216458797,
+            -0.011409370228648186,
+            0.021491041406989098,
+            0.01742827706038952,
+            -0.019118882715702057,
+            0.00047078036004677415,
+            0.04312370344996452,
+            0.019242800772190094,
+            -0.00578434718772769,
+            -0.030360078439116478,
+            -0.007620999123901129,
+            0.0010770967928692698,
+            0.010878290049731731,
+            -0.03202413022518158,
+            0.038981277495622635,
+            -0.037706684321165085,
+            -0.006470325402915478,
+            0.04407964646816254,
+            0.005939245689660311,
+            -0.010736668482422829,
+            0.026129139587283134,
+            -0.019207395613193512,
+            -0.04563748091459274,
+            0.0004276301187928766,
+            0.054347194731235504,
+            0.022907253354787827,
+            -0.025598060339689255,
+            -0.018481586128473282,
+            0.016746725887060165,
+            -0.007138601504266262,
+            0.013427474536001682,
+            -0.004531883168965578,
+            0.00865217950195074,
+            -0.014250649139285088,
+            -0.0037308374885469675,
+            -0.06309231370687485,
+            0.029563458636403084,
+            -0.02439427748322487,
+            -0.0195260439068079,
+            0.01194045040756464,
+            -0.009462077170610428,
+            -0.012533489614725113,
+            0.02533251978456974,
+            -0.02425265684723854,
+            0.03597182407975197,
+            0.015135781839489937,
+            0.0024186272639781237,
+            -0.023579955101013184,
+            0.05077125504612923,
+            0.04754937067627907,
+            -0.015879293903708458,
+            -0.03604263439774513,
+            0.0064039407297968864,
+            -0.0072979251854121685,
+            -0.031900208443403244,
+            0.00047575923963449895,
+            0.001995975850149989,
+            0.008559240959584713,
+            -0.07343067228794098,
+            -0.004664653446525335,
+            -0.07930795848369598,
+            0.007979477755725384,
+            0.031015075743198395,
+            0.003403338138014078,
+            0.0006732546607963741,
+            -0.007824580185115337,
+            0.015250849537551403,
+            0.02331441454589367,
+            -0.02333211898803711,
+            -0.014330310747027397,
+            -0.016906049102544785,
+            -0.010444574989378452,
+            0.030165348201990128,
+            0.0368923619389534,
+            -0.009276199154555798,
+            0.007027959916740656,
+            -0.00438583642244339,
+            -0.014170986600220203,
+            -0.01679983362555504,
+            0.01257774606347084,
+            0.01864091120660305,
+            -0.014224094338715076,
+            -0.02080063708126545,
+            -0.00154677068348974,
+            -0.0028523425571620464,
+            -0.00985153578221798,
+            0.03876884654164314,
+            -0.023650765419006348,
+            0.02133171632885933,
+            -0.006076441146433353,
+            -0.01828685775399208,
+            0.05501989647746086,
+            -0.022747930139303207,
+            0.02604062668979168,
+            0.00314443651586771,
+            -0.0062490422278642654,
+            0.0011235662968829274,
+            0.027085084468126297,
+            0.004456647206097841,
+            -0.013772676698863506,
+            0.03639668971300125,
+            0.0011141618015244603,
+            -0.002290282864123583,
+            0.014047068543732166,
+            0.010179034434258938,
+            -0.003905651392415166,
+            -0.020906852558255196,
+            -0.011453626677393913,
+            0.0139320008456707,
+            -0.005571915302425623,
+            -0.007160729728639126,
+            -0.0034055509604513645,
+            -0.005881711840629578,
+            0.013338961638510227,
+            -0.006983703002333641,
+            0.03436088189482689,
+            -0.0323781818151474,
+            -0.006727014202624559,
+            0.007067790720611811,
+            0.004978875629603863,
+            0.02368617057800293,
+            0.020694419741630554,
+            0.017437128350138664,
+            0.02005712501704693,
+            0.010400318540632725,
+            -0.010940250009298325,
+            -0.0029342174530029297,
+            -0.0001850482076406479,
+            -0.016357267275452614,
+            0.02752765081822872,
+            0.036644525825977325,
+            -0.020322663709521294,
+            -0.04291126877069473,
+            0.0014018301153555512,
+            0.01148903276771307,
+            -0.028979269787669182,
+            -0.03381209820508957,
+            0.01830456033349037,
+            0.034290071576833725,
+            0.019791584461927414,
+            -0.03400682657957077,
+            -0.02409333363175392,
+            0.008289274759590626,
+            0.01347173098474741,
+            0.008090119808912277,
+            0.02025185339152813,
+            -0.003168777795508504,
+            0.009435522370040417,
+            -0.0005703578935936093,
+            -0.004082677885890007,
+            -0.013418623246252537,
+            -0.03529912233352661,
+            0.014480783604085445,
+            -0.0067668454721570015,
+            -0.0022194720804691315,
+            0.009789575822651386,
+            -0.017755776643753052,
+            -0.027014274150133133,
+            0.005819752346724272,
+            0.026890354230999947,
+            0.010382615961134434,
+            -0.012781327590346336,
+            -0.02281874045729637,
+            -0.006811101920902729,
+            0.0010627133306115866,
+            0.011896193958818913,
+            9.674231841927394e-05,
+            0.00715187843888998,
+            0.02080063708126545,
+            0.0003053710388485342,
+            -0.016374969854950905,
+            -0.037175606936216354,
+            0.03324561193585396,
+            -0.01565801165997982,
+            -0.010639304295182228,
+            0.031245211139321327,
+            -0.010789777152240276,
+            -0.004762018099427223,
+            -0.03351115435361862,
+            -0.008253869600594044,
+            -0.02386319823563099,
+            0.01110842451453209,
+            0.013383218087255955,
+            0.025580357760190964,
+            -0.02625305764377117,
+            -0.03437858447432518,
+            0.03184710070490837,
+            0.015684565529227257,
+            0.0005410378216765821,
+            -0.012648557312786579,
+            -0.004916916601359844,
+            -0.03873344138264656,
+            -0.03653831034898758,
+            0.019083477556705475,
+            0.0051204971969127655,
+            0.002542545786127448,
+            -0.022730227559804916,
+            -0.0076608299277722836,
+            0.02223455347120762,
+            0.04252181202173233,
+            0.004611545242369175,
+            0.022871848195791245,
+            -0.012329909019172192,
+            0.02352684736251831,
+            -0.031085887923836708,
+            -0.019826989620923996,
+            0.024482792243361473,
+            -0.019437531009316444,
+            -0.00976302195340395,
+            0.01013477798551321,
+            0.025757383555173874,
+            0.020552799105644226,
+            0.017552196979522705,
+            0.032856155186891556,
+            0.029634268954396248,
+            0.01862320862710476,
+            0.025721978396177292,
+            0.009532887488603592,
+            -0.019083477556705475,
+            -0.016020916402339935,
+            -0.00664292648434639,
+            -0.027633866295218468,
+            -0.004558437503874302,
+            -0.024836845695972443,
+            -0.016065172851085663,
+            0.021066175773739815,
+            0.0036224087234586477,
+            0.0028147243428975344,
+            0.013436325825750828,
+            -0.03653831034898758,
+            -0.00505853770300746,
+            -0.0048107001930475235,
+            0.008001606911420822,
+            -0.0013974044704809785,
+            -0.0008524941513314843,
+            0.017100779339671135,
+            0.01384348701685667,
+            -0.018056722357869148,
+            -0.0036334728356450796,
+            -0.01778233051300049,
+            0.030501699075102806,
+            -0.018552398309111595,
+            -0.006532284896820784,
+            -0.03958316892385483,
+            0.019986312836408615,
+            0.023438334465026855,
+            0.027899406850337982,
+            0.004341579508036375,
+            0.001363105489872396,
+            0.006833230145275593,
+            0.005660428665578365,
+            -0.0009476460400037467,
+            -0.0063065760768949986,
+            -0.003874671645462513,
+            0.014631256461143494,
+            -0.012117477133870125,
+            0.009568292647600174,
+            -0.0123122064396739,
+            -0.02223455347120762,
+            -0.024907656013965607,
+            0.0015157910529524088,
+            0.02294265851378441,
+            0.03922911360859871,
+            0.031758587807416916,
+            -0.016507739201188087,
+            -0.03506898880004883,
+            0.02278333529829979,
+            0.02223455347120762,
+            -0.009771873243153095,
+            -0.001461576670408249,
+            -0.005439145024865866,
+            -0.014525040052831173,
+            -0.017295507714152336,
+            0.0014593638479709625,
+            -0.01665821112692356,
+            -0.013896595686674118,
+            0.0386626310646534,
+            0.013542542234063148,
+            0.012197138741612434,
+            -0.0007601083489134908,
+            -0.017569899559020996,
+            -0.01665821112692356,
+            0.0002082829741993919,
+            0.019207395613193512,
+            -0.01320619136095047,
+            -0.015348214656114578,
+            0.011196938343346119,
+            -0.034095339477062225,
+            0.011250046081840992,
+            0.007793600205332041,
+            0.0019760604482144117,
+            0.022128336131572723,
+            0.008253869600594044,
+            -0.014445378445088863,
+            -0.02025185339152813,
+            -0.005226713139563799,
+            0.0026310591492801905,
+            -0.0037551787681877613,
+            0.016693616285920143,
+            0.008218464441597462,
+            0.027917109429836273,
+            0.027226705104112625,
+            -0.009258495643734932,
+            0.0123122064396739,
+            0.022323066368699074,
+            -0.010754371993243694,
+            -0.004567288793623447,
+            0.00045031163608655334,
+            -0.013958554714918137,
+            0.03784830868244171,
+            0.043725594878196716,
+            -0.016330711543560028,
+            0.013179637491703033,
+            -0.01194045040756464,
+            -0.03167007490992546,
+            0.04737234115600586,
+            -0.018888747319579124,
+            -0.01880023442208767,
+            0.00986923836171627,
+            0.007687384262681007,
+            -0.02115469053387642,
+            -0.017198143526911736,
+            0.007758195046335459,
+            -0.013091123662889004,
+            0.010966803878545761,
+            0.003606918966397643,
+            0.040432896465063095,
+            0.01972077414393425,
+            -0.006864210125058889,
+            -0.024111036211252213,
+            -0.018570100888609886,
+            0.015144633129239082,
+            0.00215087435208261,
+            0.03976019471883774,
+            -0.0032395885791629553,
+            0.0244296845048666,
+            -0.029616566374897957,
+            0.008050289005041122,
+            -0.013108826242387295,
+            0.004257491789758205,
+            -0.0070722163654863834,
+            -0.016923751682043076,
+            0.012427274137735367,
+            -0.018322262912988663,
+            0.002898812061175704,
+            -0.03289156034588814,
+            -0.028642918914556503,
+            -0.018499288707971573,
+            0.03455561026930809,
+            0.05169179290533066,
+            -0.01748138666152954,
+            -0.009152280166745186,
+            0.02207522839307785,
+            -0.013073421083390713,
+            -0.0024650967679917812,
+            -0.008758395910263062,
+            -0.0002622207975946367,
+            0.03432547673583031,
+            -0.03368818014860153,
+            -0.028483595699071884,
+            -0.0011606313055381179,
+            -0.02494306117296219,
+            -0.03653831034898758,
+            0.000632870418485254,
+            -0.0047929976135492325,
+            -0.03618425503373146,
+            0.015622605569660664,
+            0.00015973893459886312,
+            0.0184461809694767,
+            0.004832828883081675,
+            0.011338559910655022,
+            -0.010798628441989422,
+            0.006417217664420605,
+            0.03239588439464569,
+            -0.04340694472193718,
+            0.017906250432133675,
+            0.007727215066552162,
+            -0.024695223197340965,
+            0.034449394792318344,
+            0.014710918068885803,
+            -0.015215444378554821,
+            -0.019809287041425705,
+            0.009050489403307438,
+            -0.002253771061077714,
+            -0.01918969303369522,
+            0.01691490039229393,
+            -0.01897726207971573,
+            0.010887141339480877,
+            0.03370588272809982,
+            0.013586798682808876,
+            0.005833029747009277,
+            0.04089316725730896,
+            0.05463043600320816,
+            -0.031209805980324745,
+            0.002666464541107416,
+            -0.01410017628222704,
+            0.03632587566971779,
+            -0.00863447692245245,
+            0.009807278402149677,
+            -0.00397867476567626,
+            -0.010426872409880161,
+            -0.014348013326525688,
+            0.02062360942363739,
+            -0.009621400386095047,
+            0.018322262912988663,
+            0.0006389557383954525,
+            -0.0035892161540687084,
+            -0.00022640054521616548,
+            0.0034918515011668205,
+            -0.021668067201972008,
+            -0.012622003443539143,
+            0.009444373659789562,
+            0.021809687837958336,
+            0.018393073230981827,
+            0.02674873359501362,
+            -0.021119285374879837,
+            0.007253668736666441,
+            -0.0009448799537494779,
+            -0.005656003020703793,
+            0.013781527988612652,
+            0.027439137920737267,
+            0.01675557717680931,
+            0.022181443870067596,
+            0.015144633129239082,
+            0.05463043600320816,
+            -0.016516590490937233,
+            -0.03926451876759529,
+            0.034838851541280746,
+            -0.0025159919168800116,
+            -0.041318029165267944,
+            -0.030165348201990128,
+            -0.0317939929664135,
+            -0.03958316892385483,
+            -0.02264171466231346,
+            -0.0018941856687888503,
+            0.018711721524596214,
+            0.004978875629603863,
+            -0.010993357747793198,
+            -0.02117239311337471,
+            0.0037264118436723948,
+            -0.011612950824201107,
+            0.03563547506928444,
+            -0.01830456033349037,
+            -0.010957952588796616,
+            -0.013985108584165573,
+            -0.03513979911804199,
+            -0.00742184417322278,
+            -0.014586999081075191,
+            0.011462477967143059,
+            -0.014728620648384094,
+            -0.030147645622491837,
+            -0.0024119887966662645,
+            -0.022393876686692238,
+            -0.008860185742378235,
+            -0.008599071763455868,
+            0.00868315901607275,
+            -0.030661022290587425,
+            0.022146038711071014,
+            0.030625617131590843,
+            0.00010808309161802754,
+            0.0036047061439603567,
+            0.016330711543560028,
+            0.0009996476583182812,
+            -0.011550991795957088,
+            0.008815929293632507,
+            0.026465490460395813,
+            -0.03894587233662605,
+            -0.01374612282961607,
+            -0.027580758556723595,
+            0.000777257839217782,
+            0.0016806471394374967,
+            -0.02281874045729637,
+            0.004899213556200266,
+            0.0005875073256902397,
+            -0.01707422360777855,
+            -0.04177829995751381,
+            -0.03887506201863289,
+            0.01320619136095047,
+            0.035741690546274185,
+            -0.002186279743909836,
+            0.005284246988594532,
+            -0.008970827795565128,
+            0.013515988364815712,
+            0.023031173273921013,
+            -0.02081833966076374,
+            -0.014702066779136658,
+            0.021791985258460045,
+            0.015321659855544567,
+            -0.018339965492486954,
+            -0.0155517952516675,
+            0.02156185172498226,
+            -0.016242198646068573,
+            0.03313939645886421,
+            -0.006815527565777302,
+            0.021278608590364456,
+            0.052895575761795044,
+            -0.01710963062942028,
+            0.04436288774013519,
+            -0.006651777774095535,
+            -0.01693260297179222,
+            0.015711119398474693,
+            0.018924154341220856,
+            -0.01473747193813324,
+            0.020747527480125427,
+            0.037706684321165085,
+            0.020765231922268867,
+            0.020517393946647644,
+            -0.028873054310679436,
+            0.013108826242387295,
+            0.019260503351688385,
+            0.0506296344101429,
+            0.009134577587246895,
+            -0.013790379278361797,
+            -0.005164753645658493,
+            0.010639304295182228,
+            0.00314443651586771,
+            -0.02945724129676819,
+            0.0012380804400891066,
+            0.00031864806078374386,
+            -0.04068073257803917,
+            0.006793399341404438,
+            0.010028562508523464,
+            0.03253750503063202,
+            0.0134894335642457,
+            0.0027129340451210737,
+            -0.03653831034898758,
+            0.01465781033039093,
+            -0.01356024481356144,
+            0.02660711109638214,
+            0.03057250939309597,
+            0.022323066368699074,
+            -0.02692575938999653,
+            -0.0022947085089981556,
+            -0.007687384262681007,
+            0.017861993983387947,
+            0.055975839495658875,
+            0.020765231922268867,
+            -0.020198745653033257,
+            -0.021278608590364456,
+            -0.01789739914238453,
+            -0.013648757711052895,
+            -0.002648761961609125,
+            0.022553199902176857,
+            0.02368617057800293,
+            -0.01240072026848793,
+            0.022146038711071014,
+            -0.027085084468126297,
+            -0.007713938131928444,
+            -0.0033458045218139887,
+            -0.033581964671611786,
+            0.0027439137920737267,
+            0.024659818038344383,
+            0.00932930689305067,
+            0.00922309048473835,
+            -0.007820154540240765,
+            0.0041933199390769005,
+            -0.025775086134672165,
+            0.0007910880376584828,
+            0.0015677926130592823,
+            0.02474833093583584,
+            -0.008811503648757935,
+            0.008793801069259644,
+            -0.012338760308921337,
+            -0.020216448232531548,
+            -0.01229450386017561,
+            -0.023845495656132698,
+            0.013321259059011936,
+            0.021437931805849075,
+            0.006408366374671459,
+            0.020198745653033257,
+            0.005226713139563799,
+            0.017419425770640373,
+            0.025013871490955353,
+            -0.007364310324192047,
+            -0.00212432025000453,
+            0.021738877519965172,
+            0.01140051893889904,
+            -0.00790866743773222,
+            0.019118882715702057,
+            -0.0010660325642675161,
+            0.011471329256892204,
+            0.01076322328299284,
+            0.014277203008532524,
+            -0.012365314178168774,
+            -0.03549385070800781,
+            -0.0027107212226837873,
+            0.006045461632311344,
+            0.016569698229432106,
+            0.01640152372419834,
+            0.017773479223251343,
+            0.015861591324210167,
+            0.014887944795191288,
+            -0.005062963347882032,
+            0.0036157702561467886,
+            -0.007342182099819183,
+            0.004450008273124695,
+            -0.04450451210141182,
+            0.012223693542182446,
+            -0.006855358835309744,
+            0.005594043526798487,
+            -0.013126528821885586,
+            0.002008146606385708,
+            -0.019083477556705475,
+            -0.012241396121680737,
+            -0.014091324992477894,
+            0.02713819220662117,
+            0.02444738708436489,
+            -0.0022626223508268595,
+            -0.0031400108709931374,
+            -0.011011060327291489,
+            0.022553199902176857,
+            0.0026067181024700403,
+            0.0017392872832715511,
+            -0.009789575822651386,
+            0.016419226303696632,
+            0.033174801617860794,
+            -0.0045849913731217384,
+            -0.011418221518397331,
+            0.013480582274496555,
+            -0.01057734526693821,
+            -0.006271170452237129,
+            0.022836443036794662,
+            -0.017808884382247925,
+            -0.015410173684358597,
+            0.01502956636250019,
+            -0.004149063024669886,
+            0.001848822575993836,
+            0.009984305128455162,
+            0.015604902990162373,
+            0.009833833202719688,
+            -0.007461674977093935,
+            0.006213637068867683,
+            0.01204666681587696,
+            -0.011143830604851246,
+            -0.01783544011414051,
+            0.007682958617806435,
+            -0.010090521536767483,
+            0.013498285785317421,
+            0.020694419741630554,
+            -0.031953319907188416,
+            -0.014418824575841427,
+            -0.012790178880095482,
+            -0.00841761939227581,
+            0.013498285785317421,
+            -0.008484004065394402,
+            0.022376174107193947,
+            0.021278608590364456,
+            -0.008253869600594044,
+            -0.016295306384563446,
+            -0.02547414042055607,
+            0.017224697396159172,
+            0.012701665051281452,
+            0.0007081067888066173,
+            -0.0437610000371933,
+            0.0021221074275672436,
+            0.011427072808146477,
+            0.012152882292866707,
+            0.001480385661125183,
+            -0.0010759903816506267,
+            -0.009692211635410786,
+            0.06925284117460251,
+            -0.01455159392207861,
+            -0.0004137998912483454,
+            -0.007187283597886562,
+            -0.02658940851688385,
+            0.005678131245076656,
+            -0.028465893119573593,
+            0.0076431273482739925,
+            0.002303559798747301,
+            0.014852539636194706,
+            -0.004815125837922096,
+            0.013250447809696198,
+            0.005589617881923914,
+            0.024872250854969025,
+            0.023296711966395378,
+            0.01991550251841545,
+            -0.02170347236096859,
+            -0.0010394785786047578,
+            0.0041601271368563175,
+            -0.025651168078184128,
+            -0.003427679417654872,
+            -0.003272780915722251,
+            -0.014507337473332882,
+            -0.00269080582074821,
+            -0.025615762919187546,
+            0.014622405171394348,
+            -0.00782900582998991,
+            0.005377185996621847,
+            -0.02589900605380535,
+            -0.0006267851567827165,
+            0.018534693866968155,
+            0.03349345177412033,
+            0.008103396743535995,
+            0.005151476711034775,
+            -0.01013477798551321,
+            0.01778233051300049,
+            -0.0017912888433784246,
+            0.04124721884727478,
+            0.01644578017294407,
+            -0.005549786612391472,
+            -0.006430494599044323,
+            0.040999382734298706,
+            0.02007482759654522,
+            -0.027226705104112625,
+            -0.0025690998882055283,
+            0.005748942028731108,
+            0.025279412046074867,
+            -0.028642918914556503,
+            -0.008550389669835567,
+            -0.019968610256910324,
+            -0.0044367313385009766,
+            0.008780524134635925,
+            0.016047470271587372,
+            -0.00874069333076477,
+            -0.012551192194223404,
+            0.0018886534962803125,
+            0.011710315942764282,
+            0.017490237951278687,
+            -0.009267346933484077,
+            0.006204785779118538,
+            0.016313008964061737,
+            -0.02804102748632431,
+            0.0006511263200081885,
+            -0.009585995227098465,
+            0.024323467165231705,
+            0.027191299945116043,
+            0.0027660420164465904,
+            -0.0030736259650439024,
+            0.00939126592129469,
+            0.007479377556592226,
+            -0.007143027149140835,
+            0.037919119000434875,
+            -0.010418021120131016,
+            -0.00046386526082642376,
+            -0.004290684591978788,
+            0.003370145568624139,
+            0.014808283187448978,
+            -0.014967607334256172,
+            0.0015711118467152119,
+            -0.008700861595571041,
+            -0.0021530871745198965,
+            0.006386238150298595,
+            -0.006341981235891581,
+            0.015188890509307384,
+            0.0034807873889803886,
+            -0.029705079272389412,
+            0.010019711218774319,
+            0.0017337552271783352,
+            0.005434719379991293,
+            -0.025704275816679,
+            -0.03037778101861477,
+            0.015126930549740791,
+            -0.01738402061164379,
+            -0.01883563958108425,
+            -0.017746925354003906,
+            -0.022924955934286118,
+            0.019844692200422287,
+            -0.0025027149822562933,
+            -0.02731521800160408,
+            0.03073183447122574,
+            0.031192103400826454,
+            -0.005098368972539902,
+            0.03197102248668671,
+            -0.010586196556687355,
+            0.012613152153789997,
+            0.0349273681640625,
+            -0.012630854733288288,
+            -0.0015025140019133687,
+            0.013852338306605816,
+            -0.008886739611625671,
+            0.005704685114324093,
+            -0.008284849114716053,
+            -0.022340768948197365,
+            -0.020552799105644226,
+            -0.018924154341220856,
+            -0.012223693542182446,
+            0.00976302195340395,
+            -0.0029076633509248495,
+            0.012807881459593773,
+            -0.010143629275262356,
+            -0.015348214656114578,
+            0.02333211898803711,
+            -0.0035670879296958447,
+            -0.01989779993891716,
+            -0.005930394399911165,
+            0.0222699586302042,
+            0.007483803667128086,
+            -0.01991550251841545,
+            0.032626017928123474,
+            0.027279812842607498,
+            -0.025739680975675583,
+            -0.014498486183583736,
+            -0.024358872324228287,
+            0.009966602548956871,
+            -0.0032418014016002417,
+            -0.025456437841057777,
+            0.006833230145275593,
+            -0.025084681808948517,
+            0.001977166859433055,
+            -0.003029369283467531,
+            -0.008961976505815983,
+            -0.0017160525312647223,
+            -0.016790982335805893,
+            0.006063164211809635,
+            -0.014179837889969349,
+            -0.017198143526911736,
+            -0.03113899566233158,
+            0.017693817615509033,
+            0.018587803468108177,
+            0.04705369472503662,
+            -0.005147051066160202,
+            -0.0034918515011668205,
+            0.0008707500528544188,
+            0.026182247325778008,
+            0.013622203841805458,
+            -0.012701665051281452,
+            0.0034874258562922478,
+            -0.007474951911717653,
+            0.014153284020721912,
+            -0.01638382114470005,
+            0.0076077221892774105,
+            0.022393876686692238,
+            -0.014427675865590572,
+            -0.0023677321150898933,
+            0.008537111803889275,
+            0.0026620388962328434,
+            -0.005629448685795069,
+            0.07944957911968231,
+            -0.00874069333076477,
+            0.008692010305821896,
+            -0.007174006663262844,
+            0.0024783737026154995,
+            -0.011019911617040634,
+            0.000400799501221627,
+            0.03558236360549927,
+            -0.007873262278735638,
+            0.01102876290678978,
+            0.024659818038344383,
+            0.008523834869265556,
+            0.004376985132694244,
+            0.007921944372355938,
+            0.02839508093893528,
+            -0.009647955186665058,
+            0.015613754279911518,
+            0.012064369395375252,
+            -0.011745721101760864,
+            -0.004244214855134487,
+            -0.012082071974873543,
+            0.0007075535831972957,
+            -0.0009885834297165275,
+            0.004545160569250584,
+            0.012595449574291706,
+            0.02554495260119438,
+            0.014109027571976185,
+            0.014817134477198124,
+            0.006434920243918896,
+            0.019632259383797646,
+            0.0021110433153808117,
+            -0.0111349793151021,
+            -0.009258495643734932,
+            0.00836893729865551,
+            0.005501104518771172,
+            0.0034697232767939568,
+            -0.006850933190435171,
+            0.009506333619356155,
+            -0.017861993983387947,
+            0.02062360942363739,
+            -0.02080063708126545,
+            -0.009524036198854446,
+            -0.003230737056583166,
+            -0.020729824900627136,
+            0.007342182099819183,
+            0.045141804963350296,
+            0.004700058605521917,
+            -0.005868434906005859,
+            0.04776180163025856,
+            -0.015162336640059948,
+            -0.03912289813160896,
+            0.034732636064291,
+            0.014932201243937016,
+            -0.029439538717269897,
+            -0.0017459257505834103,
+            0.00014438739162869751,
+            0.028837649151682854,
+            0.002750552259385586,
+            -0.00041960857925005257,
+            0.015153485350310802,
+            -0.02188050001859665,
+            0.016242198646068573,
+            -0.004478775430470705,
+            0.0007838963065296412,
+            -0.01915428787469864,
+            0.02602292411029339,
+            -0.006182657089084387,
+            -0.005855157971382141,
+            0.0013011462287977338,
+            -0.014728620648384094,
+            -0.004215448163449764,
+            -0.00939126592129469,
+            -0.00669603468850255,
+            -0.00978072453290224,
+            -0.019402125850319862,
+            0.015702268108725548,
+            0.0030758387874811888,
+            -0.02749224565923214,
+            -0.0004948449204675853,
+            0.003244014224037528,
+            -0.01620679348707199,
+            -0.009568292647600174,
+            -0.0009376882808282971,
+            0.014224094338715076,
+            -0.02658940851688385,
+            0.013126528821885586,
+            -0.002604505280032754,
+            -0.010887141339480877,
+            0.022199146449565887,
+            -0.007634276058524847,
+            0.016313008964061737,
+            0.009453224949538708,
+            0.013091123662889004,
+            -0.0008679840248078108,
+            0.0008591326768510044,
+            0.00792637001723051,
+            0.004801848903298378,
+            -0.03788371384143829,
+            0.0070899189449846745,
+            0.014507337473332882,
+            -0.01748138666152954,
+            -0.00872299075126648,
+            -0.006850933190435171,
+            -0.034467097371816635,
+            0.0058064754121005535,
+            -0.011913896538317204,
+            0.022022120654582977,
+            0.01438341848552227,
+            0.002887747948989272,
+            0.011373965069651604,
+            -0.014294905588030815,
+            0.01194045040756464,
+            0.0025270560290664434,
+            -0.00489036226645112,
+            0.05983502045273781,
+            -0.023579955101013184,
+            -0.02205752581357956,
+            -0.008315828628838062,
+            0.02929791808128357,
+            -0.0014814921887591481,
+            0.01646348275244236,
+            0.04195532575249672,
+            0.030448591336607933,
+            -0.004149063024669886,
+            0.0025093534495681524,
+            0.0067491428926587105,
+            -0.00010379572631791234,
+            -0.006961574777960777,
+            0.01328585296869278,
+            -0.027970217168331146,
+            0.01710963062942028,
+            -0.002752765081822872,
+            -0.02731521800160408,
+            -0.012321057729423046,
+            0.009081468917429447,
+            0.023916305974125862,
+            -0.005894988775253296,
+            0.014586999081075191,
+            -0.008559240959584713,
+            -0.018764829263091087,
+            -0.00912572629749775,
+            0.009506333619356155,
+            0.020694419741630554,
+            0.0006511263200081885,
+            0.02823575772345066,
+            -0.030625617131590843,
+            0.028483595699071884,
+            0.006815527565777302,
+            -0.003624621545895934,
+            0.015374768525362015,
+            -0.005231138784438372,
+            0.0009896898409351707,
+            0.009285050444304943,
+            0.015082674100995064,
+            -0.021951310336589813,
+            -0.046558018773794174,
+            0.013940852135419846,
+            0.02515549212694168,
+            0.024146441370248795,
+            0.0037573915906250477,
+            -0.012064369395375252,
+            -0.0193313155323267,
+            0.022163741290569305,
+            -0.016684764996170998,
+            -0.010152480565011501,
+            0.010541939176619053,
+            0.007337756454944611,
+            0.0044367313385009766,
+            0.0267841387540102,
+            0.019968610256910324,
+            0.003317037597298622,
+            -0.004443369805812836,
+            -0.0021044048480689526,
+            -0.0027239981573075056,
+            0.010825182311236858,
+            -0.030873455107212067,
+            0.0033944868482649326,
+            0.006125123705714941,
+            0.0008646647329442203,
+            -0.007116473279893398,
+            0.006833230145275593,
+            0.0144719323143363,
+            -0.02409333363175392,
+            -0.05679016187787056,
+            0.021136987954378128,
+            -0.01799476332962513,
+            0.015888145193457603,
+            -0.0005501657724380493,
+            0.0004071613948326558,
+            0.014985309913754463,
+            0.013914298266172409,
+            -0.003914502914994955,
+            -0.013569096103310585,
+            -0.0155517952516675,
+            0.014684364199638367,
+            -0.010710114613175392,
+            0.013338961638510227,
+            -0.011462477967143059,
+            -0.011657207272946835,
+            -0.006864210125058889,
+            0.01652544178068638,
+            0.024465089663863182,
+            0.012604300864040852,
+            0.01707422360777855,
+            0.0020557225216180086,
+            0.007820154540240765,
+            0.00505853770300746,
+            -0.0038414793089032173,
+            0.014507337473332882,
+            -0.034095339477062225,
+            0.013347812928259373,
+            -0.012453828006982803,
+            -0.0022637287620455027,
+            -0.004425667226314545,
+            -0.004049485549330711,
+            -0.02189820259809494,
+            -0.019862394779920578,
+            -0.0076608299277722836,
+            -0.018322262912988663,
+            0.00484168017283082,
+            0.02625305764377117,
+            -0.011630653403699398,
+            -0.021278608590364456,
+            -0.0042796204797923565,
+            -0.027102787047624588,
+            0.00949748232960701,
+            -0.02802332490682602,
+            0.041353434324264526,
+            0.008838057518005371,
+            0.00135757343377918,
+            0.006700460333377123,
+            -0.02189820259809494,
+            0.012170584872364998,
+            0.011073019355535507,
+            -0.019278205931186676,
+            -0.015135781839489937,
+            0.006656203418970108,
+            0.025084681808948517,
+            -0.0015257487539201975,
+            0.010913695208728313,
+            -0.0018875470850616693,
+            0.015534092672169209,
+            -0.03620195761322975,
+            -0.008191910572350025,
+            -0.00039858664968051016,
+            -0.02368617057800293,
+            -0.00914342887699604,
+            0.025952113792300224,
+            -0.018853342160582542,
+            0.015542943961918354,
+            -0.005470125004649162,
+            -0.006408366374671459,
+            -0.01365760900080204,
+            0.01701226457953453,
+            -2.6744170099846087e-05,
+            0.023066578432917595,
+            0.012453828006982803,
+            0.015224295668303967,
+            -0.015711119398474693,
+            -0.006961574777960777,
+            -0.0033922740258276463,
+            -0.0043327282182872295,
+            -0.015056120231747627,
+            -0.01130315475165844,
+            0.0134894335642457,
+            0.008461875841021538,
+            -0.008616774342954159,
+            0.015286254696547985,
+            0.0001645794982323423,
+            -0.0031046057119965553,
+            -0.023031173273921013,
+            0.0005709110992029309,
+            -0.004908064845949411,
+            -0.007567890919744968,
+            -0.013648757711052895,
+            -0.017808884382247925,
+            0.025598060339689255,
+            0.01862320862710476,
+            -0.04361937567591667,
+            0.01084288489073515,
+            -0.016065172851085663,
+            0.025226304307579994,
+            -0.010232143104076385,
+            0.010019711218774319,
+            -0.0024296913761645555,
+            -0.0216857697814703,
+            0.0028656194917857647,
+            -0.006514582317322493,
+            -0.019437531009316444,
+            -0.021827390417456627,
+            0.014790580607950687,
+            -0.007041236851364374,
+            -0.0037950098048895597,
+            -0.0030692003201693296,
+            -0.030537104234099388,
+            -0.009639102965593338,
+            0.01954374648630619,
+            0.004801848903298378,
+            0.007669681683182716,
+            0.016985710710287094,
+            -0.011506735347211361,
+            0.028129542246460915,
+            -0.011232343502342701,
+            -0.011250046081840992,
+            0.0025204175617545843,
+            -0.0056338743306696415,
+            0.016569698229432106,
+            -0.0116749107837677,
+            -0.022588606923818588,
+            -0.002199556678533554,
+            0.009754170663654804,
+            0.021738877519965172,
+            -0.005859583616256714,
+            0.01357794739305973,
+            0.024571305140852928,
+            0.013462879694998264,
+            -0.0015489835059270263,
+            0.007266945671290159,
+            0.06128663942217827,
+            0.019950907677412033,
+            0.013754974119365215,
+            0.0019052497809752822,
+            0.025704275816679,
+            -0.014277203008532524,
+            0.000535782368388027,
+            -0.013542542234063148,
+            -0.0025580357760190964,
+            -0.018083276227116585,
+            -0.011931599117815495,
+            -0.005682556889951229,
+            0.010364912450313568,
+            -0.006757994182407856,
+            0.011276599951088428,
+            -0.010471128858625889,
+            -0.005386037286370993,
+            -0.01610942929983139,
+            0.00011700358300004154,
+            0.008253869600594044,
+            0.015321659855544567,
+            -0.019649963825941086,
+            -0.026111437007784843,
+            0.002363306237384677,
+            -0.0060941437259316444,
+            0.011550991795957088,
+            0.0044632856734097,
+            0.0021652576979249716,
+            0.0145692965015769,
+            -0.022323066368699074,
+            0.032997775822877884,
+            -0.007156304083764553,
+            -0.018888747319579124,
+            0.017136184498667717,
+            -0.005403739865869284,
+            0.020552799105644226,
+            0.0182514525949955,
+            0.007846708409488201,
+            0.00985153578221798,
+            0.003244014224037528,
+            -0.0030669874977320433,
+            -0.02007482759654522,
+            0.012896394357085228,
+            -0.002684167120605707,
+            0.007948498241603374,
+            -0.008568092249333858,
+            -0.014082473702728748,
+            -0.004554011858999729,
+            -0.004970024339854717,
+            0.009134577587246895,
+            -0.0023987116292119026,
+            0.014374567195773125,
+            0.0036644525825977325,
+            -0.0019738476257771254,
+            -0.01076322328299284,
+            -0.0002287516836076975,
+            0.028111839666962624,
+            -0.0011849724687635899,
+            0.005496678873896599,
+            0.008895590901374817,
+            -0.011798828840255737,
+            -0.010435723699629307,
+            -0.004312812816351652,
+            -0.00028628535801544785,
+            0.003912290092557669,
+            0.015268552117049694,
+            0.0030138795264065266,
+            0.0018443968147039413,
+            -0.015339363366365433,
+            0.004952321760356426,
+            0.0015146846417337656,
+            0.009400117211043835,
+            -0.005005429964512587,
+            -0.0034498076420277357,
+            -0.018729424104094505,
+            -0.011258897371590137,
+            -0.01148903276771307,
+            -0.0014848114224150777,
+            -0.006244616582989693,
+            -0.022659417241811752,
+            0.011090721935033798,
+            0.018198344856500626,
+            -0.01546328142285347,
+            -0.0013863402418792248,
+            -0.0368923619389534,
+            -0.0007130856392905116,
+            0.008886739611625671,
+            0.013338961638510227,
+            0.00309354136697948,
+            0.03234277665615082,
+            0.003520618425682187,
+            -0.0068199532106518745,
+            0.014764025807380676,
+            -0.020481988787651062,
+            -0.013011462055146694,
+            -0.0201102327555418,
+            0.007682958617806435,
+            0.004135786090046167,
+            -0.00655883876606822,
+            -0.018216047435998917,
+            0.011373965069651604,
+            -0.009169982746243477,
+            -0.010196737945079803,
+            -0.0026310591492801905,
+            -0.013675311580300331,
+            0.00013850955292582512,
+            0.013693015091121197,
+            -0.012170584872364998,
+            -0.0195260439068079,
+            -0.016675913706421852,
+            -0.0061339749954640865,
+            -0.02189820259809494,
+            -0.02281874045729637,
+            -0.003064774675294757,
+            -0.0025381201412528753,
+            0.015374768525362015,
+            -0.012781327590346336,
+            0.004872659686952829,
+            -0.01665821112692356,
+            0.003991951700299978,
+            -0.004268555901944637,
+            -0.038627225905656815,
+            0.008758395910263062,
+            0.006063164211809635,
+            -0.00715187843888998,
+            -0.0037507531233131886,
+            0.019419828429818153,
+            -0.00709877023473382,
+            -0.008501706644892693,
+            -0.00043675804045051336,
+            0.004102593753486872,
+            0.014277203008532524,
+            0.0008995168609544635,
+            -0.013046867214143276,
+            -0.00026069945306517184,
+            -0.003642324125394225,
+            -0.008846908807754517,
+            -0.006271170452237129,
+            -0.0040738265961408615,
+            0.0008314722217619419,
+            -0.021225500851869583,
+            0.010825182311236858,
+            -0.0035737263970077038,
+            0.0032949093729257584,
+            0.03556466102600098,
+            -0.005332929082214832,
+            -0.026642518118023872,
+            -0.009240793064236641,
+            0.01664050854742527,
+            0.005133774131536484,
+            -0.012878691777586937,
+            -0.012471530586481094,
+            -0.0046425252221524715,
+            -0.014719769358634949,
+            -0.007629850413650274,
+            -0.01229450386017561,
+            0.0018001401331275702,
+            0.020889149978756905,
+            -0.025297114625573158,
+            -0.015613754279911518,
+            -0.007439546752721071,
+            -0.015923550352454185,
+            0.013781527988612652,
+            0.002002614550292492,
+            0.0018510353984311223,
+            -0.0074351211078464985,
+            0.02639468014240265,
+            0.004441156983375549,
+            -0.016233347356319427,
+            -0.01548098400235176,
+            0.0036047061439603567,
+            0.01311767753213644,
+            -0.0021641512867063284,
+            -0.026359274983406067,
+            0.030076835304498672,
+            0.0006124016945250332,
+            0.010648155584931374,
+            0.004379197955131531,
+            0.023208199068903923,
+            0.014047068543732166,
+            0.0007872155983932316,
+            0.0034675104543566704,
+            0.012613152153789997,
+            -1.3069548913335893e-05,
+            0.018216047435998917,
+            -0.0009874770184978843,
+            -0.03333412483334541,
+            0.012843286618590355,
+            0.00543029373511672,
+            -0.006058738566935062,
+            0.0047398898750543594,
+            0.016313008964061737,
+            -0.006757994182407856,
+            -0.011205789633095264,
+            0.016870643943548203,
+            0.0041601271368563175,
+            0.016215644776821136,
+            0.007085493300110102,
+            0.013710717670619488,
+            -0.01581733487546444,
+            -0.0057887728326022625,
+            -0.008922145701944828,
+            0.0016197942895814776,
+            -0.00228585721924901,
+            0.0025890152901411057,
+            0.024111036211252213,
+            0.003662239760160446,
+            0.01988009735941887,
+            0.002788170473650098,
+            0.0021210010163486004,
+            -0.02154414914548397,
+            0.006580967456102371,
+            0.013383218087255955,
+            0.00013837125152349472,
+            -0.028147244825959206,
+            0.0018001401331275702,
+            -0.017764627933502197,
+            0.014038216322660446,
+            0.017419425770640373,
+            0.00392999267205596,
+            0.002414201619103551,
+            0.007461674977093935,
+            0.004450008273124695,
+            -0.01579963229596615,
+            0.024465089663863182,
+            -0.028819944709539413,
+            0.010072818957269192,
+            -0.006563264410942793,
+            -0.023031173273921013,
+            -0.023916305974125862,
+            0.013834635727107525,
+            0.014648959040641785,
+            -0.00046690789167769253,
+            0.04252181202173233,
+            0.005217861849814653,
+            -0.00339006120339036,
+            0.014887944795191288,
+            0.013161934912204742,
+            0.013701866380870342,
+            -0.03781289979815483,
+            -0.015135781839489937,
+            0.007474951911717653,
+            0.010382615961134434,
+            -0.014994161203503609,
+            -0.0038857359904795885,
+            -0.00031975447200238705,
+            0.003423253772780299,
+            -0.00664292648434639,
+            -0.0017083075363188982,
+            -2.6139097826671787e-05,
+            0.004219873808324337,
+            -0.018924154341220856,
+            -0.009886940941214561,
+            -0.008501706644892693,
+            0.01202896423637867,
+            -0.0003352442872710526,
+            0.026642518118023872,
+            -0.005266543943434954,
+            0.003834840841591358,
+            -0.00877167284488678,
+            0.008178633637726307,
+            -0.014852539636194706,
+            0.0028523425571620464,
+            0.0024562454782426357,
+            0.009479779750108719,
+            0.02949264645576477,
+            -0.0012790178880095482,
+            0.010356061160564423,
+            -0.006164954509586096,
+            0.0178531426936388,
+            0.01709192804992199,
+            -0.007842282764613628,
+            -0.007952923886477947,
+            0.011922747828066349,
+            0.02715589478611946,
+            0.011719167232513428,
+            -0.01519774179905653,
+            0.009506333619356155,
+            0.036467500030994415,
+            -0.01266625989228487,
+            -0.028696026653051376,
+            0.008121099323034286,
+            0.008103396743535995,
+            0.012763625010848045,
+            -0.01579963229596615,
+            -0.0030913285445421934,
+            -0.0029209405183792114,
+            -0.012418422847986221,
+            0.03365277498960495,
+            -0.004669079091399908,
+            -0.012719367630779743,
+            0.002144235884770751,
+            0.007567890919744968,
+            -0.010957952588796616,
+            -0.00732005387544632,
+            -0.03365277498960495,
+            0.00931160431355238,
+            -0.024677520617842674,
+            -0.007651978638023138,
+            -0.02517319656908512,
+            -0.024535899981856346,
+            0.00801488384604454,
+            0.007842282764613628,
+            -0.02476603351533413,
+            0.002686380175873637,
+            -0.015764227136969566,
+            0.011179235763847828,
+            -0.015126930549740791,
+            0.0016795407282188535,
+            -0.0201102327555418,
+            0.014179837889969349,
+            0.007412992883473635,
+            0.0019318037666380405,
+            -0.006142826285213232,
+            -0.013321259059011936,
+            0.0005742303328588605,
+            -0.02481914311647415,
+            -0.016410375013947487,
+            -0.022482389584183693,
+            0.0047752950340509415,
+            -0.02225225605070591,
+            0.005394888576120138,
+            -0.0010643729474395514,
+            0.0004124168772250414,
+            -0.004903639201074839,
+            -0.03460871800780296,
+            -0.015286254696547985,
+            -0.02860751375555992,
+            -0.0018255878239870071,
+            -0.0033435916993767023,
+            0.030430888757109642,
+            -0.009754170663654804,
+            -0.007103195879608393,
+            0.00823174137622118,
+            0.018499288707971573,
+            -0.0009360286640003324,
+            -0.022588606923818588,
+            -0.0036024930886924267,
+            -0.00976302195340395,
+            0.014542742632329464,
+            0.0014471932081505656,
+            -0.02095996029675007,
+            0.0018598866881802678,
+            -0.008953125216066837,
+            -0.008298126049339771,
+            0.020021719858050346,
+            -0.014224094338715076,
+            0.0001842183992266655,
+            -0.0034121894277632236,
+            0.01683523878455162,
+            -0.0017957144882529974,
+            0.008395491167902946,
+            0.009798427112400532,
+            -0.00484168017283082,
+            -0.008067991584539413,
+            -0.011223492212593555,
+            0.029970619827508926,
+            0.016959156841039658,
+            -0.011365113779902458,
+            0.019030369818210602,
+            -0.005695833824574947,
+            0.010630453005433083,
+            -0.016135983169078827,
+            -0.012630854733288288,
+            -0.0019141010707244277,
+            0.0037618172354996204,
+            0.02264171466231346,
+            -0.012498084455728531,
+            0.00900623295456171,
+            -0.011019911617040634,
+            -0.010187885724008083,
+            0.006687183398753405,
+            -0.021119285374879837,
+            -0.0022947085089981556,
+            0.007266945671290159,
+            0.017322061583399773,
+            0.006793399341404438,
+            -0.0009188791736960411,
+            0.01436571590602398,
+            -0.03643209487199783,
+            0.023031173273921013,
+            0.009293901734054089,
+            0.01675557717680931,
+            -0.009647955186665058,
+            0.025084681808948517,
+            -0.00651015667244792,
+            -0.00836893729865551,
+            -0.0030758387874811888,
+            0.0278640016913414,
+            -0.028341973200440407,
+            -0.004257491789758205,
+            0.003027156461030245,
+            -0.01644578017294407,
+            0.015003012493252754,
+            -0.015056120231747627,
+            -0.019225098192691803,
+            -0.0015888145426288247,
+            0.027987919747829437,
+            0.020145637914538383,
+            -0.017047669738531113,
+            0.013905446976423264,
+            -0.0007213837816379964,
+            0.01865861378610134,
+            -0.0014914498897269368,
+            0.002449606778100133,
+            -0.0235445499420166,
+            -0.011126128025352955,
+            0.003907864447683096,
+            0.01022329181432724,
+            0.00497444998472929,
+            0.02368617057800293,
+            0.00957714393734932,
+            -0.010984506458044052,
+            0.003137798048555851,
+            0.01175457239151001,
+            0.0008934315992519259,
+            -0.012737070210278034,
+            -0.009462077170610428,
+            0.014808283187448978,
+            -0.01464010775089264,
+            -0.007041236851364374,
+            0.0065942443907260895,
+            0.00832910556346178,
+            0.03522831201553345,
+            0.002270367229357362,
+            -0.008138801902532578,
+            0.0001835268922150135,
+            0.037919119000434875,
+            0.0017968208994716406,
+            -0.023013470694422722,
+            -0.0005200158921070397,
+            -0.0014294905122369528,
+            0.00578434718772769,
+            -0.00012903586321044713,
+            0.001651880331337452,
+            0.015985511243343353,
+            -0.002372157759964466,
+            -0.0036091317888349295,
+            0.0005570809007622302,
+            0.003617983078584075,
+            0.0023854346945881844,
+            0.01575537584722042,
+            -0.008568092249333858,
+            0.006603095680475235,
+            -0.008191910572350025,
+            -0.009515184909105301,
+            -0.0005355057655833662,
+            0.0034011253155767918,
+            0.018534693866968155,
+            -0.01697685942053795,
+            0.007532485760748386,
+            0.0133920693770051,
+            0.012506935745477676,
+            -0.014489634893834591,
+            0.017083076760172844,
+            0.002892173593863845,
+            -8.899740350898355e-05,
+            0.020765231922268867,
+            0.006080866791307926,
+            -0.006585393100976944,
+            -0.033387232571840286,
+            0.01202011201530695,
+            0.012958354316651821,
+            0.02621765248477459,
+            -0.004536308813840151,
+            -0.001487024244852364,
+            0.0035604494623839855,
+            0.008501706644892693,
+            0.015392471104860306,
+            -0.006226914003491402,
+            -0.00489036226645112,
+            -0.008616774342954159,
+            -0.0019937630277127028,
+            0.001215952099300921,
+            0.009116875007748604,
+            0.013666460290551186,
+            -0.02081833966076374,
+            0.015666862949728966,
+            0.015985511243343353,
+            0.014710918068885803,
+            -0.020499691367149353,
+            0.011533289216458797,
+            -0.012887543067336082,
+            0.009382414631545544,
+            -0.020375771448016167,
+            -0.005213436204940081,
+            0.009453224949538708,
+            -0.004823977593332529,
+            -0.03703398257493973,
+            -0.0020446584094315767,
+            -0.0028656194917857647,
+            0.007258094381541014,
+            -0.0029209405183792114,
+            0.012347611598670483,
+            0.00865217950195074,
+            -0.008599071763455868,
+            -0.003348017344251275,
+            -0.0036688782274723053,
+            -0.008413193747401237,
+            0.01601206511259079,
+            -0.011267748661339283,
+            0.004961173050105572,
+            0.02101306803524494,
+            0.004100380931049585,
+            -0.018747126683592796,
+            0.00718285795301199,
+            -0.02336752414703369,
+            0.0014117879327386618,
+            0.024482792243361473,
+            -0.006368535105139017,
+            -0.006886338349431753,
+            -0.002708508400246501,
+            -0.020694419741630554,
+            -0.014356864616274834,
+            -0.004118083510547876,
+            0.02389860339462757,
+            -0.004474349785596132,
+            -0.02207522839307785,
+            0.015498686581850052,
+            0.023934008553624153,
+            -0.0089265713468194,
+            -0.00028213628684170544,
+            -0.0072979251854121685,
+            -0.014232946559786797,
+            -0.0003125627408735454,
+            0.0011772274738177657,
+            0.019048072397708893,
+            -0.0001921016228152439,
+            -0.0001586325088283047,
+            0.007116473279893398,
+            -0.005571915302425623,
+            -0.002701869932934642,
+            0.029899807646870613,
+            0.00985153578221798,
+            -0.004540734458714724,
+            -0.01648118533194065,
+            0.0193313155323267,
+            -0.007550188340246677,
+            0.014861390925943851,
+            -0.001994869438931346,
+            0.012161733582615852,
+            -0.009347009472548962,
+            -0.007603296544402838,
+            -0.0014095749938860536,
+            -0.0017647348577156663,
+            -0.024358872324228287,
+            0.015436727553606033,
+            0.00502755818888545,
+            -0.007647552993148565,
+            -0.04468153789639473,
+            -0.004531883168965578,
+            -0.01050653401762247,
+            0.0026797414757311344,
+            -0.008594646118581295,
+            -0.005846306681632996,
+            -0.006164954509586096,
+            0.0017127332976087928,
+            -0.042273975908756256,
+            0.010807479731738567,
+            0.021791985258460045,
+            0.0003241801168769598,
+            -0.002584589645266533,
+            -0.0076254247687757015,
+            0.006178231444209814,
+            0.016490036621689796,
+            -0.0029120889957994223,
+            0.004370346665382385,
+            0.008293700404465199,
+            -0.011312006041407585,
+            -0.010541939176619053,
+            0.02315509133040905,
+            0.018924154341220856,
+            0.009745319373905659,
+            0.014250649139285088,
+            0.00701468251645565,
+            -0.009417819790542126,
+            0.01634841598570347,
+            -0.024004818871617317,
+            -0.0122148422524333,
+            -0.005948096979409456,
+            -0.017888547852635384,
+            -0.005248841363936663,
+            -0.018694018945097923,
+            -0.015843888744711876,
+            0.01086058747023344,
+            0.03078494220972061,
+            0.00012578576570376754,
+            -0.007576742675155401,
+            -0.008377788588404655,
+            0.009488631039857864,
+            0.014312608167529106,
+            0.013259299099445343,
+            0.009515184909105301,
+            0.010834033600986004,
+            -0.020535096526145935,
+            -0.009302753023803234,
+            0.0018443968147039413,
+            -0.02876683697104454,
+            0.0026310591492801905,
+            -0.014746323227882385,
+            -0.0201102327555418,
+            -0.02278333529829979,
+            0.00823174137622118,
+            -0.007771471980959177,
+            0.0027549779042601585,
+            -0.005492253229022026,
+            -0.012896394357085228,
+            0.009285050444304943,
+            0.014330310747027397,
+            -0.004443369805812836,
+            -0.015118079259991646,
+            0.01104646548628807,
+            0.006589818745851517,
+            0.004363708198070526,
+            0.0003681601956486702,
+            -0.005200159270316362,
+            -0.0008176420233212411,
+            -0.0031599265057593584,
+            0.013569096103310585,
+            0.017322061583399773,
+            0.010125926695764065,
+            0.01864091120660305,
+            0.0027217853348702192,
+            0.006842081900686026,
+            -0.002210620790719986,
+            -0.003874671645462513,
+            -0.0010411381954327226,
+            -0.02517319656908512,
+            -0.0019849117379635572,
+            -0.024181846529245377,
+            0.012630854733288288,
+            0.001221484155394137,
+            -0.01878253184258938,
+            0.003179841907694936,
+            -0.051620982587337494,
+            -0.011471329256892204,
+            -0.011090721935033798,
+            0.005673705600202084,
+            -0.0055984691716730595,
+            -0.02005712501704693,
+            -0.018587803468108177,
+            0.012462679296731949,
+            -0.006001204717904329,
+            -0.00492576789110899,
+            -0.00031062652124091983,
+            0.004120296332985163,
+            0.009108023717999458,
+            -0.014445378445088863,
+            0.0068199532106518745,
+            -0.007355459034442902,
+            -0.03092656284570694,
+            0.0038813103456050158,
+            0.006337555591017008,
+            -0.013728420250117779,
+            -0.03940614312887192,
+            0.005014281254261732,
+            0.013754974119365215,
+            0.010267548263072968,
+            -0.013383218087255955,
+            -0.009771873243153095,
+            0.011896193958818913,
+            -0.0032086088322103024,
+            0.03041318617761135,
+            -0.008758395910263062,
+            -0.016242198646068573,
+            -0.01464010775089264,
+            0.006505731027573347,
+            0.02152644656598568,
+            0.013551393523812294,
+            -0.03476804122328758,
+            0.0009957751026377082,
+            -0.010612750425934792,
+            -0.0005637193680740893,
+            0.01220599003136158,
+            -0.0024783737026154995,
+            0.002996176714077592,
+            -0.001240293262526393,
+            0.021136987954378128,
+            0.010966803878545761,
+            0.01691490039229393,
+            0.02692575938999653,
+            0.00610742112621665,
+            0.0058241779915988445,
+            -0.01901266723871231,
+            -8.051775694184471e-06,
+            -0.019968610256910324,
+            0.009904643520712852,
+            -0.022836443036794662,
+            0.014834837056696415,
+            -0.004219873808324337,
+            -0.007461674977093935,
+            -2.411642981314799e-06,
+            0.006505731027573347,
+            -0.008612348698079586,
+            -0.016162537038326263,
+            -0.006085292436182499,
+            0.03404223173856735,
+            -0.015826186165213585,
+            0.01683523878455162,
+            -0.010984506458044052,
+            0.01465781033039093,
+            0.015330511145293713,
+            -0.0035958546213805676,
+            -0.018888747319579124,
+            -0.014055919833481312,
+            -0.014215243048965931,
+            -0.010143629275262356,
+            0.0222699586302042,
+            0.019101180136203766,
+            0.009931197389960289,
+            0.0010704583255574107,
+            0.005509955808520317,
+            -0.006036610342562199,
+            0.014666661620140076,
+            0.007138601504266262,
+            0.002217259258031845,
+            -0.01084288489073515,
+            -0.005545360967516899,
+            -0.009860387071967125,
+            0.011763423681259155,
+            0.005266543943434954,
+            -0.004100380931049585,
+            0.008377788588404655,
+            0.005642725620418787,
+            0.0016806471394374967,
+            -0.0015611541457474232,
+            -0.005262118298560381,
+            0.0008071310585364699,
+            0.0003344144788570702,
+            0.0026399106718599796,
+            -0.006961574777960777,
+            0.016197942197322845,
+            -0.012055518105626106,
+            -0.013454028405249119,
+            -0.024341169744729996,
+            -0.006430494599044323,
+            -0.005377185996621847,
+            0.001866525155492127,
+            -0.03643209487199783,
+            0.012807881459593773,
+            0.0003496277204249054,
+            0.0014671087265014648,
+            0.0013785953633487225,
+            -0.0025093534495681524,
+            0.02476603351533413,
+            -0.016729023307561874,
+            -0.010834033600986004,
+            -0.010497682727873325,
+            -0.0018952920800074935,
+            0.002427478553727269,
+            0.005872860550880432,
+            -0.002425265731289983,
+            0.014339162036776543,
+            -0.004372559487819672,
+            -0.009550590068101883,
+            0.006948297843337059,
+            -0.0010084989480674267,
+            -0.020375771448016167,
+            0.008621199987828732,
+            0.014578147791326046,
+            0.018003614619374275,
+            0.026536300778388977,
+            0.00432608975097537,
+            -0.009015084244310856,
+            -0.029262512922286987,
+            -0.004295110236853361,
+            0.03855641186237335,
+            -0.02552725002169609,
+            0.006740291137248278,
+            -0.006952723488211632,
+            0.026500895619392395,
+            0.015950104221701622,
+            0.005235564429312944,
+            0.012790178880095482,
+            -0.008275997824966908,
+            0.0050010038539767265,
+            -0.0009587102103978395,
+            0.008581369183957577,
+            -0.014073622412979603,
+            0.014480783604085445,
+            0.0013099975185468793,
+            -0.005076240282505751,
+            -0.004104806575924158,
+            -0.021986715495586395,
+            -0.024642115458846092,
+            0.019632259383797646,
+            -0.004611545242369175,
+            -0.012489233165979385,
+            0.013524839654564857,
+            -0.011444775387644768,
+            -0.01757875084877014,
+            0.03351115435361862,
+            -0.01258659828454256,
+            -0.015534092672169209,
+            0.0012945076450705528,
+            0.013808081857860088,
+            0.015056120231747627,
+            0.03547614812850952,
+            -0.008050289005041122,
+            0.02299576811492443,
+            0.0035781520418822765,
+            -0.010834033600986004,
+            -0.005563064012676477,
+            -0.02101306803524494,
+            -0.013454028405249119,
+            0.008793801069259644,
+            -0.017375169321894646,
+            0.00932930689305067,
+            0.017631858587265015,
+            0.002112149726599455,
+            0.0012934012338519096,
+            0.02115469053387642,
+            -0.02427035942673683,
+            0.015534092672169209,
+            -0.001430596923455596,
+            0.006129549350589514,
+            0.0012579959584400058,
+            -0.01807442493736744,
+            -0.02134941890835762,
+            -0.01428605429828167,
+            0.003914502914994955,
+            -0.030129943042993546,
+            0.005518807098269463,
+            -0.005386037286370993,
+            -0.020765231922268867,
+            -0.01589699648320675,
+            -0.008864611387252808,
+            -0.0072271148674190044,
+            0.009913494810461998,
+            -0.0016740086721256375,
+            -0.01772037148475647,
+            -0.01915428787469864,
+            0.0033878483809530735,
+            -0.00610299501568079,
+            -0.007253668736666441,
+            0.014914498664438725,
+            -0.010603899136185646,
+            0.011081870645284653,
+            0.01039146725088358,
+            -0.00519573362544179,
+            0.019278205931186676,
+            -0.00878937542438507,
+            0.03795452415943146,
+            -0.002117681782692671,
+            -0.007992755621671677,
+            -0.004801848903298378,
+            -0.0026686773635447025,
+            0.010603899136185646,
+            0.003272780915722251,
+            0.0038813103456050158,
+            0.007377587258815765,
+            0.013781527988612652,
+            -0.011099573224782944,
+            -0.0012248033890500665,
+            0.003967610653489828,
+            0.033369529992341995,
+            -0.014153284020721912,
+            -0.0128521379083395,
+            0.0032949093729257584,
+            -0.010161331854760647,
+            0.0002121554280165583,
+            -0.00024368830781895667,
+            -0.004615970887243748,
+            0.031900208443403244,
+            0.003887948812916875,
+            0.004503116477280855,
+            0.012515787035226822,
+            0.007085493300110102,
+            -0.026270762085914612,
+            0.0007169580785557628,
+            0.0035361081827431917,
+            0.002909876173362136,
+            0.018039019778370857,
+            -0.019844692200422287,
+            -0.019667666405439377,
+            0.008382214233279228,
+            -0.0033325275871902704,
+            -0.004204384051263332,
+            0.0020933407358825207,
+            -0.013418623246252537,
+            -0.00885133445262909,
+            -0.01897726207971573,
+            -0.0013011462287977338,
+            0.004664653446525335,
+            -0.005023132544010878,
+            0.019950907677412033,
+            -0.01793280430138111,
+            -0.04057451710104942,
+            -0.0018654187442734838,
+            0.01121464092284441,
+            0.016233347356319427,
+            0.015622605569660664,
+            0.020181043073534966,
+            -0.002648761961609125,
+            -0.013586798682808876,
+            0.015516390092670918,
+            0.0015854953089728951,
+            0.005341780371963978,
+            -0.0018997177248820662,
+            0.015826186165213585,
+            0.009940048679709435,
+            -0.0034785745665431023,
+            -0.004443369805812836,
+            0.03459101542830467,
+            -0.0046026939526200294,
+            -0.014374567195773125,
+            0.023066578432917595,
+            -0.012958354316651821,
+            0.004157914314419031,
+            0.005355057306587696,
+            -0.0008641115273348987,
+            0.012223693542182446,
+            0.021986715495586395,
+            0.018127532675862312,
+            -0.021809687837958336,
+            -0.00524441571906209,
+            -0.006430494599044323,
+            -0.023084281012415886,
+            -0.010789777152240276,
+            0.010028562508523464,
+            0.004916916601359844,
+            0.024854548275470734,
+            0.0007888752152211964,
+            0.005819752346724272,
+            -0.001608730060979724,
+            -0.008970827795565128,
+            0.0059259687550365925,
+            -0.0035737263970077038,
+            0.006806676276028156,
+            -0.009904643520712852,
+            0.019313612952828407,
+            -0.0012413996737450361,
+            -0.024164143949747086,
+            0.0014648959040641785,
+            0.004388049244880676,
+            -0.008992956019937992,
+            -0.0043504307977855206,
+            0.006926169153302908,
+            -0.0024960762821137905,
+            0.01303801592439413,
+            0.015312808565795422,
+            0.003624621545895934,
+            0.012604300864040852,
+            0.011630653403699398,
+            0.005492253229022026,
+            -0.006846507545560598,
+            0.0014018301153555512,
+            -0.0023478164803236723,
+            -0.004118083510547876,
+            0.01772037148475647,
+            -0.010267548263072968,
+            0.018552398309111595,
+            -0.011444775387644768,
+            0.007253668736666441,
+            0.01954374648630619,
+            -0.0020236363634467125,
+            -0.0012369740288704634,
+            -0.0024540324229747057,
+            0.017224697396159172,
+            0.01492334995418787,
+            -0.002376583404839039,
+            -0.0004226512392051518,
+            0.007868836633861065,
+            0.007426269818097353,
+            0.018959559500217438,
+            -0.0044876267202198505,
+            0.0015866017201915383,
+            -0.002130958717316389,
+            -0.0025934409350156784,
+            -0.021862797439098358,
+            0.0014814921887591481,
+            0.005660428665578365,
+            0.005191307980567217,
+            0.011170384474098682,
+            0.007076642010360956,
+            0.018747126683592796,
+            0.0018764829728752375,
+            -0.021437931805849075,
+            0.011648355983197689,
+            -0.01917199045419693,
+            -0.018083276227116585,
+            0.011993558146059513,
+            -0.023456037044525146,
+            -0.0032086088322103024,
+            -0.02586359903216362,
+            -0.0054834019392728806,
+            -0.00010566280980128795,
+            0.007138601504266262,
+            0.0133920693770051,
+            0.0029143018182367086,
+            0.007837857119739056,
+            -0.015534092672169209,
+            -0.010550790466368198,
+            0.009878089651465416,
+            0.008568092249333858,
+            -0.011223492212593555,
+            0.013967406004667282,
+            -0.007904241792857647,
+            -0.011869640089571476,
+            0.002974048489704728,
+            -0.010373763740062714,
+            -0.0032152472995221615,
+            -0.00157332478556782,
+            0.0063242786563932896,
+            0.0023367523681372404,
+            0.007802451495081186,
+            0.017808884382247925,
+            0.008780524134635925,
+            -0.001572218257933855,
+            -0.008324679918587208,
+            -0.025279412046074867,
+            -0.0014648959040641785,
+            -0.00465580215677619,
+            0.007010256871581078,
+            0.031085887923836708,
+            -0.001259102369658649,
+            0.012250247411429882,
+            0.002670890185981989,
+            0.007497080601751804,
+            -0.0001987401192309335,
+            -0.0004624822468031198,
+            -0.016082875430583954,
+            -0.006465899758040905,
+            -0.0037241990212351084,
+            -0.02097766287624836,
+            0.019756179302930832,
+            0.010559642687439919,
+            0.008050289005041122,
+            0.00912572629749775,
+            -0.015374768525362015,
+            -0.0030869028996676207,
+            0.01880023442208767,
+            -0.0028412784449756145,
+            3.170575655531138e-05,
+            0.017056521028280258,
+            -0.013454028405249119,
+            0.005864009261131287,
+            -0.0031333724036812782,
+            -0.015401322394609451,
+            0.006364109460264444,
+            -0.0036378984805196524,
+            -0.00865217950195074,
+            -0.002591228112578392,
+            0.024730628356337547,
+            0.02244698442518711,
+            0.012082071974873543,
+            -0.016605103388428688,
+            -0.011258897371590137,
+            -0.002659826073795557,
+            -0.001719371764920652,
+            0.011497884057462215,
+            -0.015516390092670918,
+            -0.024288062006235123,
+            0.008470727130770683,
+            -0.008581369183957577,
+            -0.019225098192691803,
+            -0.00020731484983116388,
+            0.02207522839307785,
+            -0.016162537038326263,
+            -0.004180043004453182,
+            0.013223893940448761,
+            -1.0778920113807544e-05,
+            -0.010975655168294907,
+            0.014055919833481312,
+            0.00489036226645112,
+            0.013480582274496555,
+            -0.011453626677393913,
+            -0.0065765418112277985,
+            0.007204986177384853,
+            -0.020588204264640808,
+            -0.00844859890639782,
+            -0.003230737056583166,
+            -0.018587803468108177,
+            0.015684565529227257,
+            -0.010462277568876743,
+            0.00914342887699604,
+            0.004204384051263332,
+            0.0035338953603059053,
+            0.010754371993243694,
+            0.0016286455793306231,
+            0.007603296544402838,
+            -0.004700058605521917,
+            -0.0269788671284914,
+            -0.030094537883996964,
+            0.0011893981136381626,
+            0.0024186272639781237,
+            0.013728420250117779,
+            -0.0026310591492801905,
+            -0.016056321561336517,
+            -0.00966565776616335,
+            0.017605304718017578,
+            -0.030342375859618187,
+            0.016790982335805893,
+            0.01842847838997841,
+            0.0024739480577409267,
+            0.001541238627396524,
+            0.006302150432020426,
+            0.0009874770184978843,
+            -0.016135983169078827,
+            -0.00489036226645112,
+            -0.0011407157871872187,
+            0.02170347236096859,
+            -0.005062963347882032,
+            0.015427876263856888,
+            -0.021774282678961754,
+            0.005518807098269463,
+            -0.012011260725557804,
+            0.0017912888433784246,
+            0.0004840573528781533,
+            0.005992353428155184,
+            0.003629047190770507,
+            -0.008992956019937992,
+            0.010090521536767483,
+            -0.005031983833760023,
+            0.00042514066444709897,
+            0.004275194834917784,
+            0.05392232909798622,
+            0.00392999267205596,
+            0.03795452415943146,
+            -0.015312808565795422,
+            -0.00885133445262909,
+            0.025421032682061195,
+            0.011480181477963924,
+            -0.007443972397595644,
+            0.01917199045419693,
+            0.010179034434258938,
+            0.018481586128473282,
+            0.003328101709485054,
+            -0.0008696436416357756,
+            0.02223455347120762,
+            0.018198344856500626,
+            0.006368535105139017,
+            0.008059140294790268,
+            0.008391065523028374,
+            -0.004016293212771416,
+            0.019118882715702057,
+            -0.0005769963609054685,
+            -0.0195260439068079,
+            0.007231540512293577,
+            -0.01277247630059719,
+            0.015516390092670918,
+            0.0006262319511733949,
+            0.0021331715397536755,
+            0.010533087886869907,
+            0.0034365307074040174,
+            0.0023190497886389494,
+            0.0027726804837584496,
+            0.0007540230872109532,
+            -0.0025890152901411057,
+            0.012506935745477676,
+            0.010913695208728313,
+            0.0031909062527120113,
+            -0.007005831226706505,
+            0.010913695208728313,
+            0.013445177115499973,
+            0.0033258888870477676,
+            -0.012524638324975967,
+            -0.011612950824201107,
+            -0.008700861595571041,
+            -0.015790781006217003,
+            0.010187885724008083,
+            0.003611344611272216,
+            -0.009993156418204308,
+            -0.025969816371798515,
+            0.00849728100001812,
+            0.00986923836171627,
+            -0.017746925354003906,
+            0.0030625618528574705,
+            0.00826272089034319,
+            0.000512824219185859,
+            0.006032184697687626,
+            0.020358068868517876,
+            0.0006129549001343548,
+            0.012179436162114143,
+            0.01130315475165844,
+            -0.0029452815651893616,
+            -0.004872659686952829,
+            0.015604902990162373,
+            0.01411787886172533,
+            -0.008992956019937992,
+            -0.016020916402339935,
+            -0.012347611598670483,
+            -0.035175204277038574,
+            -0.006381812505424023,
+            -0.00357593921944499,
+            -0.01237416546791792,
+            -0.002407562918961048,
+            -0.00023013469763100147,
+            0.012701665051281452,
+            -0.024730628356337547,
+            0.010311804711818695,
+            -0.021083878353238106,
+            -0.017454832792282104,
+            -0.00013359983859118074,
+            0.00674471678212285,
+            -0.01697685942053795,
+            0.00044173692003823817,
+            0.006850933190435171,
+            0.0009907962521538138,
+            0.0027992345858365297,
+            0.048399098217487335,
+            -0.02641238272190094,
+            -0.0008840270456857979,
+            -0.009293901734054089,
+            0.017587602138519287,
+            0.019295910373330116,
+            0.012356462888419628,
+            -0.004372559487819672,
+            0.008864611387252808,
+            -0.006164954509586096,
+            0.003783945459872484,
+            0.024465089663863182,
+            0.004157914314419031,
+            0.025279412046074867,
+            0.005213436204940081,
+            0.0017149461200460792,
+            -0.01475517451763153,
+            -0.005209010560065508,
+            -0.014445378445088863,
+            -0.0013365515042096376,
+            -0.021614959463477135,
+            -0.03675074130296707,
+            0.02062360942363739,
+            -0.017534494400024414,
+            0.015003012493252754,
+            -0.004934619180858135,
+            0.0008032586192712188,
+            0.005076240282505751,
+            0.022323066368699074,
+            0.015604902990162373,
+            -0.022199146449565887,
+            0.030678726732730865,
+            0.004290684591978788,
+            0.03328101709485054,
+            0.0026376976165920496,
+            -0.0014604702591896057,
+            0.0017868631985038519,
+            0.020535096526145935,
+            0.0003781179548241198,
+            -0.011639504693448544,
+            -0.0020601481664925814,
+            0.012285652570426464,
+            -0.017649561166763306,
+            0.003852543421089649,
+            -0.03018305078148842,
+            -0.012630854733288288,
+            0.0027439137920737267,
+            -0.0032749937381595373,
+            0.00048710001283325255,
+            -0.01436571590602398,
+            0.03271453455090523,
+            -0.0046203965321183205,
+            0.011807680130004883,
+            0.023385226726531982,
+            -0.0036378984805196524,
+            -0.01148903276771307,
+            -0.016870643943548203,
+            0.01883563958108425,
+            -0.013799230568110943,
+            0.01842847838997841,
+            0.009771873243153095,
+            0.0011595248943194747,
+            -0.0014859178336337209,
+            -0.0021951310336589813,
+            0.00015379194519482553,
+            0.0016441354528069496,
+            0.007289073895663023,
+            -0.004509754944592714,
+            -0.020535096526145935,
+            0.021314013749361038,
+            -0.011391667649149895,
+            -0.00618708273395896,
+            0.021986715495586395,
+            -0.013268150389194489,
+            0.006341981235891581,
+            0.015020715072751045,
+            -0.02788170427083969,
+            -0.03540533781051636,
+            0.019791584461927414,
+            -0.003113457001745701,
+            0.008156504482030869,
+            0.005594043526798487,
+            0.00841761939227581,
+            0.008935422636568546,
+            0.00994889996945858,
+            -0.008913294412195683,
+            0.05919772759079933,
+            -0.021190095692873,
+            0.0032108216546475887,
+            0.0007175113423727453,
+            -0.017295507714152336,
+            0.014082473702728748,
+            0.00939126592129469,
+            0.010612750425934792,
+            -0.013091123662889004,
+            -0.008174207992851734,
+            -0.017083076760172844,
+            3.744529385585338e-05,
+            0.0015666862018406391,
+            -0.0008696436416357756,
+            -0.006541136186569929,
+            -0.003989738877862692,
+            -0.0394415482878685,
+            -0.02765156887471676,
+            0.03813154995441437,
+            -0.005279821343719959,
+            0.016684764996170998,
+            -0.006443771533668041,
+            -0.008718564175069332,
+            0.0014471932081505656,
+            0.010364912450313568,
+            0.004135786090046167,
+            -0.012338760308921337,
+            -0.005828603636473417,
+            0.007412992883473635,
+            -0.0561174601316452,
+            -0.0022438133601099253,
+            0.0057710702531039715,
+            -0.007209412287920713,
+            0.0033192504197359085,
+            0.018322262912988663,
+            0.02460671029984951,
+            -0.011196938343346119,
+            0.011453626677393913,
+            -0.004427880048751831,
+            0.011825382709503174,
+            0.02315509133040905,
+            0.0027815320063382387,
+            0.005116071552038193,
+            0.013604501262307167,
+            0.004691207315772772,
+            -0.008762821555137634,
+            -0.009754170663654804,
+            -0.018127532675862312,
+            -0.011312006041407585,
+            0.006032184697687626,
+            0.0003062008472625166,
+            0.004396900534629822,
+            0.014560445211827755,
+            0.017959358170628548,
+            -0.015808483585715294,
+            0.02133171632885933,
+            0.013631055131554604,
+            -0.005859583616256714,
+            -1.9491951661620988e-06,
+            -0.014967607334256172,
+            -0.011922747828066349,
+            -0.004135786090046167,
+            -0.010772074572741985,
+            0.01212632842361927,
+            -0.04163667932152748,
+            0.019083477556705475,
+            -0.0032285242341458797,
+            -0.010081670247018337,
+            -0.011851937510073185,
+            0.006846507545560598,
+            0.032112643122673035,
+            0.0030470718629658222,
+            -0.014445378445088863,
+            0.0012358676176518202,
+            0.004531883168965578,
+            -0.023827793076634407,
+            0.01585274003446102,
+            -0.008333531208336353,
+            0.001153992721810937,
+            -0.009050489403307438,
+            -0.015773078426718712,
+            0.04443369805812836,
+            -0.014808283187448978,
+            0.02203982323408127,
+            0.034626420587301254,
+            0.029404133558273315,
+            -0.0010887141106650233,
+            -0.012551192194223404,
+            -0.0139320008456707,
+            0.017083076760172844,
+            0.0032019703648984432,
+            0.012179436162114143,
+            0.017516791820526123,
+            0.017410574480891228,
+            -0.013799230568110943,
+            0.014073622412979603,
+            -0.006346406880766153,
+            -0.006727014202624559,
+            0.00046995055163279176,
+            -0.0020490840543061495,
+            0.0073997159488499165,
+            0.013400920666754246,
+            0.02062360942363739,
+            -0.01086058747023344,
+            0.0017249038210138679,
+            0.02264171466231346,
+            0.017047669738531113,
+            0.011922747828066349,
+            -0.008072417229413986,
+            3.713411570060998e-05,
+            0.005164753645658493,
+            0.010356061160564423,
+            0.008382214233279228,
+            0.004580565728247166,
+            -0.01011707540601492,
+            0.024535899981856346,
+            0.004270769190043211,
+            0.0008060246473178267,
+            0.022588606923818588,
+            -0.01076322328299284,
+            -0.00701468251645565,
+            -0.0016242199344560504,
+            0.007882113568484783,
+            -0.006797824986279011,
+            -0.03558236360549927,
+            0.02025185339152813,
+            -0.015339363366365433,
+            -0.01707422360777855,
+            0.01022329181432724,
+            -0.010984506458044052,
+            0.007992755621671677,
+            -0.0025757383555173874,
+            0.013666460290551186,
+            -0.009435522370040417,
+            0.0015622605569660664,
+            0.007005831226706505,
+            -0.0029497072100639343,
+            0.017596453428268433,
+            -0.014941052533686161,
+            0.009320455603301525,
+            -0.008815929293632507,
+            0.04609775170683861,
+            0.0140293650329113,
+            0.0026376976165920496,
+            -0.0030382205732166767,
+            -0.008112248033285141,
+            0.0006129549001343548,
+            0.003963185008615255,
+            2.6778745450428687e-05,
+            -0.008205187506973743,
+            -0.02621765248477459,
+            -0.006757994182407856,
+            -0.002617782214656472,
+            0.01672017201781273,
+            -0.008395491167902946,
+            0.019402125850319862,
+            0.013701866380870342,
+            -0.019136585295200348,
+            -0.02281874045729637,
+            0.022287661209702492,
+            -0.00809454545378685,
+            -0.007054513785988092,
+            -0.0067491428926587105,
+            -0.01810983009636402,
+            -0.00024811396724544466,
+            0.025704275816679,
+            0.003976461943238974,
+            0.0015102589968591928,
+            0.0023920731619000435,
+            0.004770869389176369,
+            0.025704275816679,
+            0.001831119880080223,
+            0.023119686171412468,
+            0.004016293212771416,
+            0.009453224949538708,
+            -0.012064369395375252,
+            0.0011827596463263035,
+            0.0014770664274692535,
+            -0.019649963825941086,
+            -0.011896193958818913,
+            0.0036578141152858734,
+            0.0065765418112277985,
+            0.006850933190435171,
+            0.017145035788416862,
+            0.016923751682043076,
+            -0.014047068543732166,
+            0.009134577587246895,
+            -0.008581369183957577,
+            -0.004507542122155428,
+            0.00966565776616335,
+            0.012453828006982803,
+            -0.00846630148589611,
+            0.0016253263456746936,
+            0.009709914214909077,
+            -0.007174006663262844,
+            0.018198344856500626,
+            0.01583503745496273,
+            0.0001586325088283047,
+            -0.002617782214656472,
+            0.03094426542520523,
+            0.007492654956877232,
+            -0.015241998247802258,
+            0.009187685325741768,
+            -0.025456437841057777
+          ],
+          "title": "9 Conclusion"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "fb59d64d-47b5-4383-8bb2-bb0ec8c972ea",
+      "type": "next",
+      "source": {
+        "id": "1deb6775-c7ea-4a7a-8654-e306d12e688d",
+        "properties": {
+          "page_content": "9 Conclusion 27 ",
+          "keyphrases": [],
+          "embedding": [
+            -0.0026620388962328434,
+            0.009736468084156513,
+            -0.01410017628222704,
+            0.027085084468126297,
+            -0.02409333363175392,
+            -0.023739280179142952,
+            0.002801447408273816,
+            0.09375333786010742,
+            -0.007904241792857647,
+            0.016392672434449196,
+            -0.00255582295358181,
+            -0.03965397924184799,
+            -0.002425265731289983,
+            -0.03738803789019585,
+            -0.047655586153268814,
+            0.007165155373513699,
+            -0.002056828932836652,
+            -0.002341178013011813,
+            -0.018729424104094505,
+            -0.010240994393825531,
+            -0.027563055977225304,
+            -0.00033054203959181905,
+            0.017215846106410027,
+            0.05247071012854576,
+            -0.00470448425039649,
+            -0.0022095143795013428,
+            0.020588204264640808,
+            0.01972077414393425,
+            -0.032856155186891556,
+            0.04528342932462692,
+            -0.008333531208336353,
+            -0.01013477798551321,
+            0.036502905189991,
+            -0.024925358593463898,
+            -0.01793280430138111,
+            0.013728420250117779,
+            0.02297806553542614,
+            -0.014091324992477894,
+            0.013666460290551186,
+            -0.018764829263091087,
+            -0.01830456033349037,
+            0.01579963229596615,
+            -0.004806274548172951,
+            -0.004509754944592714,
+            -0.004859382752329111,
+            -0.028182649984955788,
+            -0.021101580932736397,
+            -0.06061394140124321,
+            -0.015905847772955894,
+            -0.003113457001745701,
+            -0.0059967790730297565,
+            -0.0376712791621685,
+            0.01110842451453209,
+            -0.003370145568624139,
+            0.009320455603301525,
+            0.03606033697724342,
+            0.028288865461945534,
+            0.08199876546859741,
+            0.0022404941264539957,
+            0.011338559910655022,
+            -0.021455634385347366,
+            -0.029917510226368904,
+            -0.009603697806596756,
+            -0.00012419528502505273,
+            -0.009072617627680302,
+            -0.022376174107193947,
+            0.011188087053596973,
+            -0.01899496465921402,
+            -0.006395089440047741,
+            -0.028430486097931862,
+            0.01365760900080204,
+            0.021136987954378128,
+            0.005231138784438372,
+            0.016764428466558456,
+            -0.007868836633861065,
+            0.0030692003201693296,
+            0.05133774131536484,
+            0.036078039556741714,
+            -0.009506333619356155,
+            0.04036208614706993,
+            -0.007550188340246677,
+            -0.010471128858625889,
+            0.008178633637726307,
+            0.028182649984955788,
+            0.04556667059659958,
+            0.010612750425934792,
+            -0.0727933794260025,
+            -0.09934738278388977,
+            -0.010108224116265774,
+            0.004580565728247166,
+            0.022712524980306625,
+            -0.03039548359811306,
+            -0.02081833966076374,
+            -0.026518598198890686,
+            -0.046416398137807846,
+            0.02150874398648739,
+            0.002179641043767333,
+            0.005049686413258314,
+            -0.03901668265461922,
+            0.008023735135793686,
+            0.018145235255360603,
+            0.017313210293650627,
+            0.009824981912970543,
+            0.010559642687439919,
+            0.02203982323408127,
+            -0.011250046081840992,
+            0.024358872324228287,
+            0.0323781818151474,
+            0.009718765504658222,
+            0.05640070512890816,
+            -0.020729824900627136,
+            -0.0049478961154818535,
+            -0.03560006618499756,
+            -0.029563458636403084,
+            0.011418221518397331,
+            -0.004403539001941681,
+            -0.021225500851869583,
+            -0.04762018099427223,
+            0.014162135310471058,
+            0.017667263746261597,
+            -0.009585995227098465,
+            0.004894787911325693,
+            0.02825346030294895,
+            -0.03774208948016167,
+            -0.02223455347120762,
+            0.019826989620923996,
+            0.03239588439464569,
+            0.008992956019937992,
+            -0.007616573479026556,
+            0.010957952588796616,
+            0.006669480819255114,
+            0.015967806801199913,
+            -0.024695223197340965,
+            -0.005629448685795069,
+            0.018410775810480118,
+            -0.024128738790750504,
+            0.010046265088021755,
+            0.0012579959584400058,
+            -0.001970528392121196,
+            0.030802644789218903,
+            -0.044398292899131775,
+            -0.008298126049339771,
+            -0.005554212257266045,
+            0.02388090081512928,
+            -0.006204785779118538,
+            0.012869840487837791,
+            -0.01720699481666088,
+            0.029811294749379158,
+            0.0013188488082960248,
+            0.01239186804741621,
+            0.0006068696384318173,
+            0.011639504693448544,
+            -0.02643008530139923,
+            0.0038591818884015083,
+            -0.002400924451649189,
+            -0.03961857408285141,
+            0.031369131058454514,
+            0.012728218920528889,
+            0.0037750941701233387,
+            0.03660912066698074,
+            0.0031532880384474993,
+            0.054524220526218414,
+            -0.020499691367149353,
+            -0.0051381997764110565,
+            0.009727616794407368,
+            0.00795734953135252,
+            0.027580758556723595,
+            -0.015976659953594208,
+            -0.04273424297571182,
+            0.022871848195791245,
+            0.03616655245423317,
+            0.0002338688645977527,
+            0.025244006887078285,
+            -0.03558236360549927,
+            0.007231540512293577,
+            -0.007213837932795286,
+            -0.008218464441597462,
+            0.055834218859672546,
+            -0.0100551163777709,
+            0.024164143949747086,
+            -0.009453224949538708,
+            0.019579151645302773,
+            -0.01991550251841545,
+            0.004472136963158846,
+            -0.0227656327188015,
+            -0.0074705262668430805,
+            0.06341096013784409,
+            -0.03887506201863289,
+            0.03274993970990181,
+            0.01571997068822384,
+            -0.002052403287962079,
+            -0.022482389584183693,
+            0.03370588272809982,
+            0.010046265088021755,
+            0.004113657865673304,
+            -0.06093258783221245,
+            -0.0016231135232374072,
+            0.035210609436035156,
+            -0.014498486183583736,
+            0.011781126260757446,
+            -0.014560445211827755,
+            -0.022287661209702492,
+            0.018481586128473282,
+            0.010754371993243694,
+            -0.010710114613175392,
+            -0.0028280015103518963,
+            0.005558638367801905,
+            0.010471128858625889,
+            0.014294905588030815,
+            -0.0005191860836930573,
+            0.02747454307973385,
+            0.0150118637830019,
+            -0.008116673678159714,
+            0.01593240164220333,
+            -0.013232745230197906,
+            0.010692412033677101,
+            -0.006930594798177481,
+            0.02209293097257614,
+            -0.013002610765397549,
+            -0.030625617131590843,
+            0.014409972354769707,
+            0.013551393523812294,
+            0.04145964980125427,
+            -0.0002031657932093367,
+            -0.008138801902532578,
+            0.04850531369447708,
+            -0.03760046884417534,
+            0.04698288440704346,
+            0.02729751542210579,
+            -0.04698288440704346,
+            0.009798427112400532,
+            0.018906449899077415,
+            -0.01830456033349037,
+            0.003505128435790539,
+            -0.0057710702531039715,
+            0.008882313966751099,
+            0.0009399011032655835,
+            0.014206391759216785,
+            0.003244014224037528,
+            0.02928021550178528,
+            0.04528342932462692,
+            -0.02372157759964466,
+            -0.028660621494054794,
+            0.0412118136882782,
+            -0.025615762919187546,
+            0.007258094381541014,
+            -0.0412118136882782,
+            0.06741176545619965,
+            0.017260102555155754,
+            0.014082473702728748,
+            0.015224295668303967,
+            -0.0005056324880570173,
+            0.0017757989699020982,
+            0.03958316892385483,
+            0.019490638747811317,
+            0.04740775004029274,
+            0.009957751259207726,
+            -0.0043836236000061035,
+            -0.053249627351760864,
+            0.032094940543174744,
+            -0.0034498076420277357,
+            0.0035715135745704174,
+            -0.030324673280119896,
+            0.01626875251531601,
+            0.02497846633195877,
+            0.002735062502324581,
+            0.02207522839307785,
+            -0.010356061160564423,
+            0.021809687837958336,
+            0.02007482759654522,
+            0.05640070512890816,
+            -0.014170986600220203,
+            -0.00489036226645112,
+            0.01175457239151001,
+            -0.018393073230981827,
+            -0.031953319907188416,
+            0.032271966338157654,
+            -0.018516991287469864,
+            -0.013347812928259373,
+            0.019065774977207184,
+            -0.0007838963065296412,
+            0.03476804122328758,
+            -0.005558638367801905,
+            -0.020712122321128845,
+            -0.017410574480891228,
+            -0.022110633552074432,
+            -0.0010062861256301403,
+            0.01040916983038187,
+            -0.030430888757109642,
+            0.00484168017283082,
+            0.013480582274496555,
+            -0.01138281635940075,
+            -0.009152280166745186,
+            0.01111727673560381,
+            -0.004633673466742039,
+            0.002014785073697567,
+            0.018853342160582542,
+            0.007492654956877232,
+            -0.008523834869265556,
+            -0.0014095749938860536,
+            0.003102392889559269,
+            0.0005645491764880717,
+            -0.03441398963332176,
+            -0.013595649972558022,
+            -0.008453024551272392,
+            0.01607402414083481,
+            0.016498887911438942,
+            0.03584790602326393,
+            -0.0714125707745552,
+            0.011789977550506592,
+            0.035901013761758804,
+            0.016339562833309174,
+            0.000724703015293926,
+            -0.04418586194515228,
+            -0.029775889590382576,
+            0.020375771448016167,
+            0.01302031334489584,
+            0.010046265088021755,
+            -0.013454028405249119,
+            -0.004485413897782564,
+            0.012763625010848045,
+            -0.011728018522262573,
+            -0.018605506047606468,
+            0.030625617131590843,
+            0.020499691367149353,
+            -0.03260831534862518,
+            -0.021243203431367874,
+            0.008430896326899529,
+            0.01988009735941887,
+            -0.012444976717233658,
+            0.025633465498685837,
+            -0.022482389584183693,
+            0.019490638747811317,
+            -0.001468215137720108,
+            -0.014799431897699833,
+            -0.02713819220662117,
+            -0.023438334465026855,
+            0.02025185339152813,
+            -0.03657371550798416,
+            -0.006908466573804617,
+            0.019472936168313026,
+            0.0032019703648984432,
+            -0.008599071763455868,
+            0.0008060246473178267,
+            -0.01640152372419834,
+            -0.038237765431404114,
+            0.029563458636403084,
+            -0.0561174601316452,
+            -0.003299335017800331,
+            0.01527740340679884,
+            0.0009227516129612923,
+            -0.009152280166745186,
+            0.008160930126905441,
+            0.01793280430138111,
+            -0.0023124110884964466,
+            -0.008006032556295395,
+            -0.028111839666962624,
+            -0.027580758556723595,
+            -0.009470928460359573,
+            -0.008935422636568546,
+            0.013073421083390713,
+            0.006381812505424023,
+            0.014896796084940434,
+            -0.006311001721769571,
+            -0.04907179996371269,
+            -0.025066979229450226,
+            0.016339562833309174,
+            0.01293179951608181,
+            0.020570501685142517,
+            0.0008663243497721851,
+            -0.02223455347120762,
+            -0.002160832053050399,
+            0.021225500851869583,
+            -0.017083076760172844,
+            -0.01575537584722042,
+            -0.006549987476319075,
+            -0.013648757711052895,
+            0.011612950824201107,
+            -0.026890354230999947,
+            0.011294303461909294,
+            -0.02892616204917431,
+            0.012276801280677319,
+            0.006704885978251696,
+            0.004992152564227581,
+            0.0151092279702425,
+            -0.0027306366246193647,
+            0.012498084455728531,
+            0.017180440947413445,
+            -0.0073997159488499165,
+            -0.04393802583217621,
+            -0.04854071885347366,
+            -0.012329909019172192,
+            -0.004717761185020208,
+            -0.02095996029675007,
+            0.00849728100001812,
+            0.011993558146059513,
+            0.0390520878136158,
+            0.0034984899684786797,
+            0.0045849913731217384,
+            -0.01707422360777855,
+            -0.016773279756307602,
+            0.009293901734054089,
+            -0.006572116166353226,
+            -0.009621400386095047,
+            0.021030770614743233,
+            0.031227508559823036,
+            0.0006782335112802684,
+            -0.023650765419006348,
+            -0.008315828628838062,
+            -0.008979679085314274,
+            0.0402204655110836,
+            -0.03604263439774513,
+            0.0002623590989969671,
+            -0.046239372342824936,
+            0.029970619827508926,
+            0.00285012973472476,
+            0.020481988787651062,
+            0.0024186272639781237,
+            -0.027279812842607498,
+            -0.0227656327188015,
+            -0.021632662042975426,
+            -0.0019583578687161207,
+            0.027616163715720177,
+            0.001016243826597929,
+            -0.021650364622473717,
+            0.04138883948326111,
+            -0.01610942929983139,
+            0.019101180136203766,
+            0.00011299282050458714,
+            0.011312006041407585,
+            -0.04167208448052406,
+            0.007231540512293577,
+            -0.02044658362865448,
+            0.01989779993891716,
+            0.04662882909178734,
+            -0.0034741489216685295,
+            -0.03331642225384712,
+            0.011533289216458797,
+            -0.011409370228648186,
+            0.021491041406989098,
+            0.01742827706038952,
+            -0.019118882715702057,
+            0.00047078036004677415,
+            0.04312370344996452,
+            0.019242800772190094,
+            -0.00578434718772769,
+            -0.030360078439116478,
+            -0.007620999123901129,
+            0.0010770967928692698,
+            0.010878290049731731,
+            -0.03202413022518158,
+            0.038981277495622635,
+            -0.037706684321165085,
+            -0.006470325402915478,
+            0.04407964646816254,
+            0.005939245689660311,
+            -0.010736668482422829,
+            0.026129139587283134,
+            -0.019207395613193512,
+            -0.04563748091459274,
+            0.0004276301187928766,
+            0.054347194731235504,
+            0.022907253354787827,
+            -0.025598060339689255,
+            -0.018481586128473282,
+            0.016746725887060165,
+            -0.007138601504266262,
+            0.013427474536001682,
+            -0.004531883168965578,
+            0.00865217950195074,
+            -0.014250649139285088,
+            -0.0037308374885469675,
+            -0.06309231370687485,
+            0.029563458636403084,
+            -0.02439427748322487,
+            -0.0195260439068079,
+            0.01194045040756464,
+            -0.009462077170610428,
+            -0.012533489614725113,
+            0.02533251978456974,
+            -0.02425265684723854,
+            0.03597182407975197,
+            0.015135781839489937,
+            0.0024186272639781237,
+            -0.023579955101013184,
+            0.05077125504612923,
+            0.04754937067627907,
+            -0.015879293903708458,
+            -0.03604263439774513,
+            0.0064039407297968864,
+            -0.0072979251854121685,
+            -0.031900208443403244,
+            0.00047575923963449895,
+            0.001995975850149989,
+            0.008559240959584713,
+            -0.07343067228794098,
+            -0.004664653446525335,
+            -0.07930795848369598,
+            0.007979477755725384,
+            0.031015075743198395,
+            0.003403338138014078,
+            0.0006732546607963741,
+            -0.007824580185115337,
+            0.015250849537551403,
+            0.02331441454589367,
+            -0.02333211898803711,
+            -0.014330310747027397,
+            -0.016906049102544785,
+            -0.010444574989378452,
+            0.030165348201990128,
+            0.0368923619389534,
+            -0.009276199154555798,
+            0.007027959916740656,
+            -0.00438583642244339,
+            -0.014170986600220203,
+            -0.01679983362555504,
+            0.01257774606347084,
+            0.01864091120660305,
+            -0.014224094338715076,
+            -0.02080063708126545,
+            -0.00154677068348974,
+            -0.0028523425571620464,
+            -0.00985153578221798,
+            0.03876884654164314,
+            -0.023650765419006348,
+            0.02133171632885933,
+            -0.006076441146433353,
+            -0.01828685775399208,
+            0.05501989647746086,
+            -0.022747930139303207,
+            0.02604062668979168,
+            0.00314443651586771,
+            -0.0062490422278642654,
+            0.0011235662968829274,
+            0.027085084468126297,
+            0.004456647206097841,
+            -0.013772676698863506,
+            0.03639668971300125,
+            0.0011141618015244603,
+            -0.002290282864123583,
+            0.014047068543732166,
+            0.010179034434258938,
+            -0.003905651392415166,
+            -0.020906852558255196,
+            -0.011453626677393913,
+            0.0139320008456707,
+            -0.005571915302425623,
+            -0.007160729728639126,
+            -0.0034055509604513645,
+            -0.005881711840629578,
+            0.013338961638510227,
+            -0.006983703002333641,
+            0.03436088189482689,
+            -0.0323781818151474,
+            -0.006727014202624559,
+            0.007067790720611811,
+            0.004978875629603863,
+            0.02368617057800293,
+            0.020694419741630554,
+            0.017437128350138664,
+            0.02005712501704693,
+            0.010400318540632725,
+            -0.010940250009298325,
+            -0.0029342174530029297,
+            -0.0001850482076406479,
+            -0.016357267275452614,
+            0.02752765081822872,
+            0.036644525825977325,
+            -0.020322663709521294,
+            -0.04291126877069473,
+            0.0014018301153555512,
+            0.01148903276771307,
+            -0.028979269787669182,
+            -0.03381209820508957,
+            0.01830456033349037,
+            0.034290071576833725,
+            0.019791584461927414,
+            -0.03400682657957077,
+            -0.02409333363175392,
+            0.008289274759590626,
+            0.01347173098474741,
+            0.008090119808912277,
+            0.02025185339152813,
+            -0.003168777795508504,
+            0.009435522370040417,
+            -0.0005703578935936093,
+            -0.004082677885890007,
+            -0.013418623246252537,
+            -0.03529912233352661,
+            0.014480783604085445,
+            -0.0067668454721570015,
+            -0.0022194720804691315,
+            0.009789575822651386,
+            -0.017755776643753052,
+            -0.027014274150133133,
+            0.005819752346724272,
+            0.026890354230999947,
+            0.010382615961134434,
+            -0.012781327590346336,
+            -0.02281874045729637,
+            -0.006811101920902729,
+            0.0010627133306115866,
+            0.011896193958818913,
+            9.674231841927394e-05,
+            0.00715187843888998,
+            0.02080063708126545,
+            0.0003053710388485342,
+            -0.016374969854950905,
+            -0.037175606936216354,
+            0.03324561193585396,
+            -0.01565801165997982,
+            -0.010639304295182228,
+            0.031245211139321327,
+            -0.010789777152240276,
+            -0.004762018099427223,
+            -0.03351115435361862,
+            -0.008253869600594044,
+            -0.02386319823563099,
+            0.01110842451453209,
+            0.013383218087255955,
+            0.025580357760190964,
+            -0.02625305764377117,
+            -0.03437858447432518,
+            0.03184710070490837,
+            0.015684565529227257,
+            0.0005410378216765821,
+            -0.012648557312786579,
+            -0.004916916601359844,
+            -0.03873344138264656,
+            -0.03653831034898758,
+            0.019083477556705475,
+            0.0051204971969127655,
+            0.002542545786127448,
+            -0.022730227559804916,
+            -0.0076608299277722836,
+            0.02223455347120762,
+            0.04252181202173233,
+            0.004611545242369175,
+            0.022871848195791245,
+            -0.012329909019172192,
+            0.02352684736251831,
+            -0.031085887923836708,
+            -0.019826989620923996,
+            0.024482792243361473,
+            -0.019437531009316444,
+            -0.00976302195340395,
+            0.01013477798551321,
+            0.025757383555173874,
+            0.020552799105644226,
+            0.017552196979522705,
+            0.032856155186891556,
+            0.029634268954396248,
+            0.01862320862710476,
+            0.025721978396177292,
+            0.009532887488603592,
+            -0.019083477556705475,
+            -0.016020916402339935,
+            -0.00664292648434639,
+            -0.027633866295218468,
+            -0.004558437503874302,
+            -0.024836845695972443,
+            -0.016065172851085663,
+            0.021066175773739815,
+            0.0036224087234586477,
+            0.0028147243428975344,
+            0.013436325825750828,
+            -0.03653831034898758,
+            -0.00505853770300746,
+            -0.0048107001930475235,
+            0.008001606911420822,
+            -0.0013974044704809785,
+            -0.0008524941513314843,
+            0.017100779339671135,
+            0.01384348701685667,
+            -0.018056722357869148,
+            -0.0036334728356450796,
+            -0.01778233051300049,
+            0.030501699075102806,
+            -0.018552398309111595,
+            -0.006532284896820784,
+            -0.03958316892385483,
+            0.019986312836408615,
+            0.023438334465026855,
+            0.027899406850337982,
+            0.004341579508036375,
+            0.001363105489872396,
+            0.006833230145275593,
+            0.005660428665578365,
+            -0.0009476460400037467,
+            -0.0063065760768949986,
+            -0.003874671645462513,
+            0.014631256461143494,
+            -0.012117477133870125,
+            0.009568292647600174,
+            -0.0123122064396739,
+            -0.02223455347120762,
+            -0.024907656013965607,
+            0.0015157910529524088,
+            0.02294265851378441,
+            0.03922911360859871,
+            0.031758587807416916,
+            -0.016507739201188087,
+            -0.03506898880004883,
+            0.02278333529829979,
+            0.02223455347120762,
+            -0.009771873243153095,
+            -0.001461576670408249,
+            -0.005439145024865866,
+            -0.014525040052831173,
+            -0.017295507714152336,
+            0.0014593638479709625,
+            -0.01665821112692356,
+            -0.013896595686674118,
+            0.0386626310646534,
+            0.013542542234063148,
+            0.012197138741612434,
+            -0.0007601083489134908,
+            -0.017569899559020996,
+            -0.01665821112692356,
+            0.0002082829741993919,
+            0.019207395613193512,
+            -0.01320619136095047,
+            -0.015348214656114578,
+            0.011196938343346119,
+            -0.034095339477062225,
+            0.011250046081840992,
+            0.007793600205332041,
+            0.0019760604482144117,
+            0.022128336131572723,
+            0.008253869600594044,
+            -0.014445378445088863,
+            -0.02025185339152813,
+            -0.005226713139563799,
+            0.0026310591492801905,
+            -0.0037551787681877613,
+            0.016693616285920143,
+            0.008218464441597462,
+            0.027917109429836273,
+            0.027226705104112625,
+            -0.009258495643734932,
+            0.0123122064396739,
+            0.022323066368699074,
+            -0.010754371993243694,
+            -0.004567288793623447,
+            0.00045031163608655334,
+            -0.013958554714918137,
+            0.03784830868244171,
+            0.043725594878196716,
+            -0.016330711543560028,
+            0.013179637491703033,
+            -0.01194045040756464,
+            -0.03167007490992546,
+            0.04737234115600586,
+            -0.018888747319579124,
+            -0.01880023442208767,
+            0.00986923836171627,
+            0.007687384262681007,
+            -0.02115469053387642,
+            -0.017198143526911736,
+            0.007758195046335459,
+            -0.013091123662889004,
+            0.010966803878545761,
+            0.003606918966397643,
+            0.040432896465063095,
+            0.01972077414393425,
+            -0.006864210125058889,
+            -0.024111036211252213,
+            -0.018570100888609886,
+            0.015144633129239082,
+            0.00215087435208261,
+            0.03976019471883774,
+            -0.0032395885791629553,
+            0.0244296845048666,
+            -0.029616566374897957,
+            0.008050289005041122,
+            -0.013108826242387295,
+            0.004257491789758205,
+            -0.0070722163654863834,
+            -0.016923751682043076,
+            0.012427274137735367,
+            -0.018322262912988663,
+            0.002898812061175704,
+            -0.03289156034588814,
+            -0.028642918914556503,
+            -0.018499288707971573,
+            0.03455561026930809,
+            0.05169179290533066,
+            -0.01748138666152954,
+            -0.009152280166745186,
+            0.02207522839307785,
+            -0.013073421083390713,
+            -0.0024650967679917812,
+            -0.008758395910263062,
+            -0.0002622207975946367,
+            0.03432547673583031,
+            -0.03368818014860153,
+            -0.028483595699071884,
+            -0.0011606313055381179,
+            -0.02494306117296219,
+            -0.03653831034898758,
+            0.000632870418485254,
+            -0.0047929976135492325,
+            -0.03618425503373146,
+            0.015622605569660664,
+            0.00015973893459886312,
+            0.0184461809694767,
+            0.004832828883081675,
+            0.011338559910655022,
+            -0.010798628441989422,
+            0.006417217664420605,
+            0.03239588439464569,
+            -0.04340694472193718,
+            0.017906250432133675,
+            0.007727215066552162,
+            -0.024695223197340965,
+            0.034449394792318344,
+            0.014710918068885803,
+            -0.015215444378554821,
+            -0.019809287041425705,
+            0.009050489403307438,
+            -0.002253771061077714,
+            -0.01918969303369522,
+            0.01691490039229393,
+            -0.01897726207971573,
+            0.010887141339480877,
+            0.03370588272809982,
+            0.013586798682808876,
+            0.005833029747009277,
+            0.04089316725730896,
+            0.05463043600320816,
+            -0.031209805980324745,
+            0.002666464541107416,
+            -0.01410017628222704,
+            0.03632587566971779,
+            -0.00863447692245245,
+            0.009807278402149677,
+            -0.00397867476567626,
+            -0.010426872409880161,
+            -0.014348013326525688,
+            0.02062360942363739,
+            -0.009621400386095047,
+            0.018322262912988663,
+            0.0006389557383954525,
+            -0.0035892161540687084,
+            -0.00022640054521616548,
+            0.0034918515011668205,
+            -0.021668067201972008,
+            -0.012622003443539143,
+            0.009444373659789562,
+            0.021809687837958336,
+            0.018393073230981827,
+            0.02674873359501362,
+            -0.021119285374879837,
+            0.007253668736666441,
+            -0.0009448799537494779,
+            -0.005656003020703793,
+            0.013781527988612652,
+            0.027439137920737267,
+            0.01675557717680931,
+            0.022181443870067596,
+            0.015144633129239082,
+            0.05463043600320816,
+            -0.016516590490937233,
+            -0.03926451876759529,
+            0.034838851541280746,
+            -0.0025159919168800116,
+            -0.041318029165267944,
+            -0.030165348201990128,
+            -0.0317939929664135,
+            -0.03958316892385483,
+            -0.02264171466231346,
+            -0.0018941856687888503,
+            0.018711721524596214,
+            0.004978875629603863,
+            -0.010993357747793198,
+            -0.02117239311337471,
+            0.0037264118436723948,
+            -0.011612950824201107,
+            0.03563547506928444,
+            -0.01830456033349037,
+            -0.010957952588796616,
+            -0.013985108584165573,
+            -0.03513979911804199,
+            -0.00742184417322278,
+            -0.014586999081075191,
+            0.011462477967143059,
+            -0.014728620648384094,
+            -0.030147645622491837,
+            -0.0024119887966662645,
+            -0.022393876686692238,
+            -0.008860185742378235,
+            -0.008599071763455868,
+            0.00868315901607275,
+            -0.030661022290587425,
+            0.022146038711071014,
+            0.030625617131590843,
+            0.00010808309161802754,
+            0.0036047061439603567,
+            0.016330711543560028,
+            0.0009996476583182812,
+            -0.011550991795957088,
+            0.008815929293632507,
+            0.026465490460395813,
+            -0.03894587233662605,
+            -0.01374612282961607,
+            -0.027580758556723595,
+            0.000777257839217782,
+            0.0016806471394374967,
+            -0.02281874045729637,
+            0.004899213556200266,
+            0.0005875073256902397,
+            -0.01707422360777855,
+            -0.04177829995751381,
+            -0.03887506201863289,
+            0.01320619136095047,
+            0.035741690546274185,
+            -0.002186279743909836,
+            0.005284246988594532,
+            -0.008970827795565128,
+            0.013515988364815712,
+            0.023031173273921013,
+            -0.02081833966076374,
+            -0.014702066779136658,
+            0.021791985258460045,
+            0.015321659855544567,
+            -0.018339965492486954,
+            -0.0155517952516675,
+            0.02156185172498226,
+            -0.016242198646068573,
+            0.03313939645886421,
+            -0.006815527565777302,
+            0.021278608590364456,
+            0.052895575761795044,
+            -0.01710963062942028,
+            0.04436288774013519,
+            -0.006651777774095535,
+            -0.01693260297179222,
+            0.015711119398474693,
+            0.018924154341220856,
+            -0.01473747193813324,
+            0.020747527480125427,
+            0.037706684321165085,
+            0.020765231922268867,
+            0.020517393946647644,
+            -0.028873054310679436,
+            0.013108826242387295,
+            0.019260503351688385,
+            0.0506296344101429,
+            0.009134577587246895,
+            -0.013790379278361797,
+            -0.005164753645658493,
+            0.010639304295182228,
+            0.00314443651586771,
+            -0.02945724129676819,
+            0.0012380804400891066,
+            0.00031864806078374386,
+            -0.04068073257803917,
+            0.006793399341404438,
+            0.010028562508523464,
+            0.03253750503063202,
+            0.0134894335642457,
+            0.0027129340451210737,
+            -0.03653831034898758,
+            0.01465781033039093,
+            -0.01356024481356144,
+            0.02660711109638214,
+            0.03057250939309597,
+            0.022323066368699074,
+            -0.02692575938999653,
+            -0.0022947085089981556,
+            -0.007687384262681007,
+            0.017861993983387947,
+            0.055975839495658875,
+            0.020765231922268867,
+            -0.020198745653033257,
+            -0.021278608590364456,
+            -0.01789739914238453,
+            -0.013648757711052895,
+            -0.002648761961609125,
+            0.022553199902176857,
+            0.02368617057800293,
+            -0.01240072026848793,
+            0.022146038711071014,
+            -0.027085084468126297,
+            -0.007713938131928444,
+            -0.0033458045218139887,
+            -0.033581964671611786,
+            0.0027439137920737267,
+            0.024659818038344383,
+            0.00932930689305067,
+            0.00922309048473835,
+            -0.007820154540240765,
+            0.0041933199390769005,
+            -0.025775086134672165,
+            0.0007910880376584828,
+            0.0015677926130592823,
+            0.02474833093583584,
+            -0.008811503648757935,
+            0.008793801069259644,
+            -0.012338760308921337,
+            -0.020216448232531548,
+            -0.01229450386017561,
+            -0.023845495656132698,
+            0.013321259059011936,
+            0.021437931805849075,
+            0.006408366374671459,
+            0.020198745653033257,
+            0.005226713139563799,
+            0.017419425770640373,
+            0.025013871490955353,
+            -0.007364310324192047,
+            -0.00212432025000453,
+            0.021738877519965172,
+            0.01140051893889904,
+            -0.00790866743773222,
+            0.019118882715702057,
+            -0.0010660325642675161,
+            0.011471329256892204,
+            0.01076322328299284,
+            0.014277203008532524,
+            -0.012365314178168774,
+            -0.03549385070800781,
+            -0.0027107212226837873,
+            0.006045461632311344,
+            0.016569698229432106,
+            0.01640152372419834,
+            0.017773479223251343,
+            0.015861591324210167,
+            0.014887944795191288,
+            -0.005062963347882032,
+            0.0036157702561467886,
+            -0.007342182099819183,
+            0.004450008273124695,
+            -0.04450451210141182,
+            0.012223693542182446,
+            -0.006855358835309744,
+            0.005594043526798487,
+            -0.013126528821885586,
+            0.002008146606385708,
+            -0.019083477556705475,
+            -0.012241396121680737,
+            -0.014091324992477894,
+            0.02713819220662117,
+            0.02444738708436489,
+            -0.0022626223508268595,
+            -0.0031400108709931374,
+            -0.011011060327291489,
+            0.022553199902176857,
+            0.0026067181024700403,
+            0.0017392872832715511,
+            -0.009789575822651386,
+            0.016419226303696632,
+            0.033174801617860794,
+            -0.0045849913731217384,
+            -0.011418221518397331,
+            0.013480582274496555,
+            -0.01057734526693821,
+            -0.006271170452237129,
+            0.022836443036794662,
+            -0.017808884382247925,
+            -0.015410173684358597,
+            0.01502956636250019,
+            -0.004149063024669886,
+            0.001848822575993836,
+            0.009984305128455162,
+            0.015604902990162373,
+            0.009833833202719688,
+            -0.007461674977093935,
+            0.006213637068867683,
+            0.01204666681587696,
+            -0.011143830604851246,
+            -0.01783544011414051,
+            0.007682958617806435,
+            -0.010090521536767483,
+            0.013498285785317421,
+            0.020694419741630554,
+            -0.031953319907188416,
+            -0.014418824575841427,
+            -0.012790178880095482,
+            -0.00841761939227581,
+            0.013498285785317421,
+            -0.008484004065394402,
+            0.022376174107193947,
+            0.021278608590364456,
+            -0.008253869600594044,
+            -0.016295306384563446,
+            -0.02547414042055607,
+            0.017224697396159172,
+            0.012701665051281452,
+            0.0007081067888066173,
+            -0.0437610000371933,
+            0.0021221074275672436,
+            0.011427072808146477,
+            0.012152882292866707,
+            0.001480385661125183,
+            -0.0010759903816506267,
+            -0.009692211635410786,
+            0.06925284117460251,
+            -0.01455159392207861,
+            -0.0004137998912483454,
+            -0.007187283597886562,
+            -0.02658940851688385,
+            0.005678131245076656,
+            -0.028465893119573593,
+            0.0076431273482739925,
+            0.002303559798747301,
+            0.014852539636194706,
+            -0.004815125837922096,
+            0.013250447809696198,
+            0.005589617881923914,
+            0.024872250854969025,
+            0.023296711966395378,
+            0.01991550251841545,
+            -0.02170347236096859,
+            -0.0010394785786047578,
+            0.0041601271368563175,
+            -0.025651168078184128,
+            -0.003427679417654872,
+            -0.003272780915722251,
+            -0.014507337473332882,
+            -0.00269080582074821,
+            -0.025615762919187546,
+            0.014622405171394348,
+            -0.00782900582998991,
+            0.005377185996621847,
+            -0.02589900605380535,
+            -0.0006267851567827165,
+            0.018534693866968155,
+            0.03349345177412033,
+            0.008103396743535995,
+            0.005151476711034775,
+            -0.01013477798551321,
+            0.01778233051300049,
+            -0.0017912888433784246,
+            0.04124721884727478,
+            0.01644578017294407,
+            -0.005549786612391472,
+            -0.006430494599044323,
+            0.040999382734298706,
+            0.02007482759654522,
+            -0.027226705104112625,
+            -0.0025690998882055283,
+            0.005748942028731108,
+            0.025279412046074867,
+            -0.028642918914556503,
+            -0.008550389669835567,
+            -0.019968610256910324,
+            -0.0044367313385009766,
+            0.008780524134635925,
+            0.016047470271587372,
+            -0.00874069333076477,
+            -0.012551192194223404,
+            0.0018886534962803125,
+            0.011710315942764282,
+            0.017490237951278687,
+            -0.009267346933484077,
+            0.006204785779118538,
+            0.016313008964061737,
+            -0.02804102748632431,
+            0.0006511263200081885,
+            -0.009585995227098465,
+            0.024323467165231705,
+            0.027191299945116043,
+            0.0027660420164465904,
+            -0.0030736259650439024,
+            0.00939126592129469,
+            0.007479377556592226,
+            -0.007143027149140835,
+            0.037919119000434875,
+            -0.010418021120131016,
+            -0.00046386526082642376,
+            -0.004290684591978788,
+            0.003370145568624139,
+            0.014808283187448978,
+            -0.014967607334256172,
+            0.0015711118467152119,
+            -0.008700861595571041,
+            -0.0021530871745198965,
+            0.006386238150298595,
+            -0.006341981235891581,
+            0.015188890509307384,
+            0.0034807873889803886,
+            -0.029705079272389412,
+            0.010019711218774319,
+            0.0017337552271783352,
+            0.005434719379991293,
+            -0.025704275816679,
+            -0.03037778101861477,
+            0.015126930549740791,
+            -0.01738402061164379,
+            -0.01883563958108425,
+            -0.017746925354003906,
+            -0.022924955934286118,
+            0.019844692200422287,
+            -0.0025027149822562933,
+            -0.02731521800160408,
+            0.03073183447122574,
+            0.031192103400826454,
+            -0.005098368972539902,
+            0.03197102248668671,
+            -0.010586196556687355,
+            0.012613152153789997,
+            0.0349273681640625,
+            -0.012630854733288288,
+            -0.0015025140019133687,
+            0.013852338306605816,
+            -0.008886739611625671,
+            0.005704685114324093,
+            -0.008284849114716053,
+            -0.022340768948197365,
+            -0.020552799105644226,
+            -0.018924154341220856,
+            -0.012223693542182446,
+            0.00976302195340395,
+            -0.0029076633509248495,
+            0.012807881459593773,
+            -0.010143629275262356,
+            -0.015348214656114578,
+            0.02333211898803711,
+            -0.0035670879296958447,
+            -0.01989779993891716,
+            -0.005930394399911165,
+            0.0222699586302042,
+            0.007483803667128086,
+            -0.01991550251841545,
+            0.032626017928123474,
+            0.027279812842607498,
+            -0.025739680975675583,
+            -0.014498486183583736,
+            -0.024358872324228287,
+            0.009966602548956871,
+            -0.0032418014016002417,
+            -0.025456437841057777,
+            0.006833230145275593,
+            -0.025084681808948517,
+            0.001977166859433055,
+            -0.003029369283467531,
+            -0.008961976505815983,
+            -0.0017160525312647223,
+            -0.016790982335805893,
+            0.006063164211809635,
+            -0.014179837889969349,
+            -0.017198143526911736,
+            -0.03113899566233158,
+            0.017693817615509033,
+            0.018587803468108177,
+            0.04705369472503662,
+            -0.005147051066160202,
+            -0.0034918515011668205,
+            0.0008707500528544188,
+            0.026182247325778008,
+            0.013622203841805458,
+            -0.012701665051281452,
+            0.0034874258562922478,
+            -0.007474951911717653,
+            0.014153284020721912,
+            -0.01638382114470005,
+            0.0076077221892774105,
+            0.022393876686692238,
+            -0.014427675865590572,
+            -0.0023677321150898933,
+            0.008537111803889275,
+            0.0026620388962328434,
+            -0.005629448685795069,
+            0.07944957911968231,
+            -0.00874069333076477,
+            0.008692010305821896,
+            -0.007174006663262844,
+            0.0024783737026154995,
+            -0.011019911617040634,
+            0.000400799501221627,
+            0.03558236360549927,
+            -0.007873262278735638,
+            0.01102876290678978,
+            0.024659818038344383,
+            0.008523834869265556,
+            0.004376985132694244,
+            0.007921944372355938,
+            0.02839508093893528,
+            -0.009647955186665058,
+            0.015613754279911518,
+            0.012064369395375252,
+            -0.011745721101760864,
+            -0.004244214855134487,
+            -0.012082071974873543,
+            0.0007075535831972957,
+            -0.0009885834297165275,
+            0.004545160569250584,
+            0.012595449574291706,
+            0.02554495260119438,
+            0.014109027571976185,
+            0.014817134477198124,
+            0.006434920243918896,
+            0.019632259383797646,
+            0.0021110433153808117,
+            -0.0111349793151021,
+            -0.009258495643734932,
+            0.00836893729865551,
+            0.005501104518771172,
+            0.0034697232767939568,
+            -0.006850933190435171,
+            0.009506333619356155,
+            -0.017861993983387947,
+            0.02062360942363739,
+            -0.02080063708126545,
+            -0.009524036198854446,
+            -0.003230737056583166,
+            -0.020729824900627136,
+            0.007342182099819183,
+            0.045141804963350296,
+            0.004700058605521917,
+            -0.005868434906005859,
+            0.04776180163025856,
+            -0.015162336640059948,
+            -0.03912289813160896,
+            0.034732636064291,
+            0.014932201243937016,
+            -0.029439538717269897,
+            -0.0017459257505834103,
+            0.00014438739162869751,
+            0.028837649151682854,
+            0.002750552259385586,
+            -0.00041960857925005257,
+            0.015153485350310802,
+            -0.02188050001859665,
+            0.016242198646068573,
+            -0.004478775430470705,
+            0.0007838963065296412,
+            -0.01915428787469864,
+            0.02602292411029339,
+            -0.006182657089084387,
+            -0.005855157971382141,
+            0.0013011462287977338,
+            -0.014728620648384094,
+            -0.004215448163449764,
+            -0.00939126592129469,
+            -0.00669603468850255,
+            -0.00978072453290224,
+            -0.019402125850319862,
+            0.015702268108725548,
+            0.0030758387874811888,
+            -0.02749224565923214,
+            -0.0004948449204675853,
+            0.003244014224037528,
+            -0.01620679348707199,
+            -0.009568292647600174,
+            -0.0009376882808282971,
+            0.014224094338715076,
+            -0.02658940851688385,
+            0.013126528821885586,
+            -0.002604505280032754,
+            -0.010887141339480877,
+            0.022199146449565887,
+            -0.007634276058524847,
+            0.016313008964061737,
+            0.009453224949538708,
+            0.013091123662889004,
+            -0.0008679840248078108,
+            0.0008591326768510044,
+            0.00792637001723051,
+            0.004801848903298378,
+            -0.03788371384143829,
+            0.0070899189449846745,
+            0.014507337473332882,
+            -0.01748138666152954,
+            -0.00872299075126648,
+            -0.006850933190435171,
+            -0.034467097371816635,
+            0.0058064754121005535,
+            -0.011913896538317204,
+            0.022022120654582977,
+            0.01438341848552227,
+            0.002887747948989272,
+            0.011373965069651604,
+            -0.014294905588030815,
+            0.01194045040756464,
+            0.0025270560290664434,
+            -0.00489036226645112,
+            0.05983502045273781,
+            -0.023579955101013184,
+            -0.02205752581357956,
+            -0.008315828628838062,
+            0.02929791808128357,
+            -0.0014814921887591481,
+            0.01646348275244236,
+            0.04195532575249672,
+            0.030448591336607933,
+            -0.004149063024669886,
+            0.0025093534495681524,
+            0.0067491428926587105,
+            -0.00010379572631791234,
+            -0.006961574777960777,
+            0.01328585296869278,
+            -0.027970217168331146,
+            0.01710963062942028,
+            -0.002752765081822872,
+            -0.02731521800160408,
+            -0.012321057729423046,
+            0.009081468917429447,
+            0.023916305974125862,
+            -0.005894988775253296,
+            0.014586999081075191,
+            -0.008559240959584713,
+            -0.018764829263091087,
+            -0.00912572629749775,
+            0.009506333619356155,
+            0.020694419741630554,
+            0.0006511263200081885,
+            0.02823575772345066,
+            -0.030625617131590843,
+            0.028483595699071884,
+            0.006815527565777302,
+            -0.003624621545895934,
+            0.015374768525362015,
+            -0.005231138784438372,
+            0.0009896898409351707,
+            0.009285050444304943,
+            0.015082674100995064,
+            -0.021951310336589813,
+            -0.046558018773794174,
+            0.013940852135419846,
+            0.02515549212694168,
+            0.024146441370248795,
+            0.0037573915906250477,
+            -0.012064369395375252,
+            -0.0193313155323267,
+            0.022163741290569305,
+            -0.016684764996170998,
+            -0.010152480565011501,
+            0.010541939176619053,
+            0.007337756454944611,
+            0.0044367313385009766,
+            0.0267841387540102,
+            0.019968610256910324,
+            0.003317037597298622,
+            -0.004443369805812836,
+            -0.0021044048480689526,
+            -0.0027239981573075056,
+            0.010825182311236858,
+            -0.030873455107212067,
+            0.0033944868482649326,
+            0.006125123705714941,
+            0.0008646647329442203,
+            -0.007116473279893398,
+            0.006833230145275593,
+            0.0144719323143363,
+            -0.02409333363175392,
+            -0.05679016187787056,
+            0.021136987954378128,
+            -0.01799476332962513,
+            0.015888145193457603,
+            -0.0005501657724380493,
+            0.0004071613948326558,
+            0.014985309913754463,
+            0.013914298266172409,
+            -0.003914502914994955,
+            -0.013569096103310585,
+            -0.0155517952516675,
+            0.014684364199638367,
+            -0.010710114613175392,
+            0.013338961638510227,
+            -0.011462477967143059,
+            -0.011657207272946835,
+            -0.006864210125058889,
+            0.01652544178068638,
+            0.024465089663863182,
+            0.012604300864040852,
+            0.01707422360777855,
+            0.0020557225216180086,
+            0.007820154540240765,
+            0.00505853770300746,
+            -0.0038414793089032173,
+            0.014507337473332882,
+            -0.034095339477062225,
+            0.013347812928259373,
+            -0.012453828006982803,
+            -0.0022637287620455027,
+            -0.004425667226314545,
+            -0.004049485549330711,
+            -0.02189820259809494,
+            -0.019862394779920578,
+            -0.0076608299277722836,
+            -0.018322262912988663,
+            0.00484168017283082,
+            0.02625305764377117,
+            -0.011630653403699398,
+            -0.021278608590364456,
+            -0.0042796204797923565,
+            -0.027102787047624588,
+            0.00949748232960701,
+            -0.02802332490682602,
+            0.041353434324264526,
+            0.008838057518005371,
+            0.00135757343377918,
+            0.006700460333377123,
+            -0.02189820259809494,
+            0.012170584872364998,
+            0.011073019355535507,
+            -0.019278205931186676,
+            -0.015135781839489937,
+            0.006656203418970108,
+            0.025084681808948517,
+            -0.0015257487539201975,
+            0.010913695208728313,
+            -0.0018875470850616693,
+            0.015534092672169209,
+            -0.03620195761322975,
+            -0.008191910572350025,
+            -0.00039858664968051016,
+            -0.02368617057800293,
+            -0.00914342887699604,
+            0.025952113792300224,
+            -0.018853342160582542,
+            0.015542943961918354,
+            -0.005470125004649162,
+            -0.006408366374671459,
+            -0.01365760900080204,
+            0.01701226457953453,
+            -2.6744170099846087e-05,
+            0.023066578432917595,
+            0.012453828006982803,
+            0.015224295668303967,
+            -0.015711119398474693,
+            -0.006961574777960777,
+            -0.0033922740258276463,
+            -0.0043327282182872295,
+            -0.015056120231747627,
+            -0.01130315475165844,
+            0.0134894335642457,
+            0.008461875841021538,
+            -0.008616774342954159,
+            0.015286254696547985,
+            0.0001645794982323423,
+            -0.0031046057119965553,
+            -0.023031173273921013,
+            0.0005709110992029309,
+            -0.004908064845949411,
+            -0.007567890919744968,
+            -0.013648757711052895,
+            -0.017808884382247925,
+            0.025598060339689255,
+            0.01862320862710476,
+            -0.04361937567591667,
+            0.01084288489073515,
+            -0.016065172851085663,
+            0.025226304307579994,
+            -0.010232143104076385,
+            0.010019711218774319,
+            -0.0024296913761645555,
+            -0.0216857697814703,
+            0.0028656194917857647,
+            -0.006514582317322493,
+            -0.019437531009316444,
+            -0.021827390417456627,
+            0.014790580607950687,
+            -0.007041236851364374,
+            -0.0037950098048895597,
+            -0.0030692003201693296,
+            -0.030537104234099388,
+            -0.009639102965593338,
+            0.01954374648630619,
+            0.004801848903298378,
+            0.007669681683182716,
+            0.016985710710287094,
+            -0.011506735347211361,
+            0.028129542246460915,
+            -0.011232343502342701,
+            -0.011250046081840992,
+            0.0025204175617545843,
+            -0.0056338743306696415,
+            0.016569698229432106,
+            -0.0116749107837677,
+            -0.022588606923818588,
+            -0.002199556678533554,
+            0.009754170663654804,
+            0.021738877519965172,
+            -0.005859583616256714,
+            0.01357794739305973,
+            0.024571305140852928,
+            0.013462879694998264,
+            -0.0015489835059270263,
+            0.007266945671290159,
+            0.06128663942217827,
+            0.019950907677412033,
+            0.013754974119365215,
+            0.0019052497809752822,
+            0.025704275816679,
+            -0.014277203008532524,
+            0.000535782368388027,
+            -0.013542542234063148,
+            -0.0025580357760190964,
+            -0.018083276227116585,
+            -0.011931599117815495,
+            -0.005682556889951229,
+            0.010364912450313568,
+            -0.006757994182407856,
+            0.011276599951088428,
+            -0.010471128858625889,
+            -0.005386037286370993,
+            -0.01610942929983139,
+            0.00011700358300004154,
+            0.008253869600594044,
+            0.015321659855544567,
+            -0.019649963825941086,
+            -0.026111437007784843,
+            0.002363306237384677,
+            -0.0060941437259316444,
+            0.011550991795957088,
+            0.0044632856734097,
+            0.0021652576979249716,
+            0.0145692965015769,
+            -0.022323066368699074,
+            0.032997775822877884,
+            -0.007156304083764553,
+            -0.018888747319579124,
+            0.017136184498667717,
+            -0.005403739865869284,
+            0.020552799105644226,
+            0.0182514525949955,
+            0.007846708409488201,
+            0.00985153578221798,
+            0.003244014224037528,
+            -0.0030669874977320433,
+            -0.02007482759654522,
+            0.012896394357085228,
+            -0.002684167120605707,
+            0.007948498241603374,
+            -0.008568092249333858,
+            -0.014082473702728748,
+            -0.004554011858999729,
+            -0.004970024339854717,
+            0.009134577587246895,
+            -0.0023987116292119026,
+            0.014374567195773125,
+            0.0036644525825977325,
+            -0.0019738476257771254,
+            -0.01076322328299284,
+            -0.0002287516836076975,
+            0.028111839666962624,
+            -0.0011849724687635899,
+            0.005496678873896599,
+            0.008895590901374817,
+            -0.011798828840255737,
+            -0.010435723699629307,
+            -0.004312812816351652,
+            -0.00028628535801544785,
+            0.003912290092557669,
+            0.015268552117049694,
+            0.0030138795264065266,
+            0.0018443968147039413,
+            -0.015339363366365433,
+            0.004952321760356426,
+            0.0015146846417337656,
+            0.009400117211043835,
+            -0.005005429964512587,
+            -0.0034498076420277357,
+            -0.018729424104094505,
+            -0.011258897371590137,
+            -0.01148903276771307,
+            -0.0014848114224150777,
+            -0.006244616582989693,
+            -0.022659417241811752,
+            0.011090721935033798,
+            0.018198344856500626,
+            -0.01546328142285347,
+            -0.0013863402418792248,
+            -0.0368923619389534,
+            -0.0007130856392905116,
+            0.008886739611625671,
+            0.013338961638510227,
+            0.00309354136697948,
+            0.03234277665615082,
+            0.003520618425682187,
+            -0.0068199532106518745,
+            0.014764025807380676,
+            -0.020481988787651062,
+            -0.013011462055146694,
+            -0.0201102327555418,
+            0.007682958617806435,
+            0.004135786090046167,
+            -0.00655883876606822,
+            -0.018216047435998917,
+            0.011373965069651604,
+            -0.009169982746243477,
+            -0.010196737945079803,
+            -0.0026310591492801905,
+            -0.013675311580300331,
+            0.00013850955292582512,
+            0.013693015091121197,
+            -0.012170584872364998,
+            -0.0195260439068079,
+            -0.016675913706421852,
+            -0.0061339749954640865,
+            -0.02189820259809494,
+            -0.02281874045729637,
+            -0.003064774675294757,
+            -0.0025381201412528753,
+            0.015374768525362015,
+            -0.012781327590346336,
+            0.004872659686952829,
+            -0.01665821112692356,
+            0.003991951700299978,
+            -0.004268555901944637,
+            -0.038627225905656815,
+            0.008758395910263062,
+            0.006063164211809635,
+            -0.00715187843888998,
+            -0.0037507531233131886,
+            0.019419828429818153,
+            -0.00709877023473382,
+            -0.008501706644892693,
+            -0.00043675804045051336,
+            0.004102593753486872,
+            0.014277203008532524,
+            0.0008995168609544635,
+            -0.013046867214143276,
+            -0.00026069945306517184,
+            -0.003642324125394225,
+            -0.008846908807754517,
+            -0.006271170452237129,
+            -0.0040738265961408615,
+            0.0008314722217619419,
+            -0.021225500851869583,
+            0.010825182311236858,
+            -0.0035737263970077038,
+            0.0032949093729257584,
+            0.03556466102600098,
+            -0.005332929082214832,
+            -0.026642518118023872,
+            -0.009240793064236641,
+            0.01664050854742527,
+            0.005133774131536484,
+            -0.012878691777586937,
+            -0.012471530586481094,
+            -0.0046425252221524715,
+            -0.014719769358634949,
+            -0.007629850413650274,
+            -0.01229450386017561,
+            0.0018001401331275702,
+            0.020889149978756905,
+            -0.025297114625573158,
+            -0.015613754279911518,
+            -0.007439546752721071,
+            -0.015923550352454185,
+            0.013781527988612652,
+            0.002002614550292492,
+            0.0018510353984311223,
+            -0.0074351211078464985,
+            0.02639468014240265,
+            0.004441156983375549,
+            -0.016233347356319427,
+            -0.01548098400235176,
+            0.0036047061439603567,
+            0.01311767753213644,
+            -0.0021641512867063284,
+            -0.026359274983406067,
+            0.030076835304498672,
+            0.0006124016945250332,
+            0.010648155584931374,
+            0.004379197955131531,
+            0.023208199068903923,
+            0.014047068543732166,
+            0.0007872155983932316,
+            0.0034675104543566704,
+            0.012613152153789997,
+            -1.3069548913335893e-05,
+            0.018216047435998917,
+            -0.0009874770184978843,
+            -0.03333412483334541,
+            0.012843286618590355,
+            0.00543029373511672,
+            -0.006058738566935062,
+            0.0047398898750543594,
+            0.016313008964061737,
+            -0.006757994182407856,
+            -0.011205789633095264,
+            0.016870643943548203,
+            0.0041601271368563175,
+            0.016215644776821136,
+            0.007085493300110102,
+            0.013710717670619488,
+            -0.01581733487546444,
+            -0.0057887728326022625,
+            -0.008922145701944828,
+            0.0016197942895814776,
+            -0.00228585721924901,
+            0.0025890152901411057,
+            0.024111036211252213,
+            0.003662239760160446,
+            0.01988009735941887,
+            0.002788170473650098,
+            0.0021210010163486004,
+            -0.02154414914548397,
+            0.006580967456102371,
+            0.013383218087255955,
+            0.00013837125152349472,
+            -0.028147244825959206,
+            0.0018001401331275702,
+            -0.017764627933502197,
+            0.014038216322660446,
+            0.017419425770640373,
+            0.00392999267205596,
+            0.002414201619103551,
+            0.007461674977093935,
+            0.004450008273124695,
+            -0.01579963229596615,
+            0.024465089663863182,
+            -0.028819944709539413,
+            0.010072818957269192,
+            -0.006563264410942793,
+            -0.023031173273921013,
+            -0.023916305974125862,
+            0.013834635727107525,
+            0.014648959040641785,
+            -0.00046690789167769253,
+            0.04252181202173233,
+            0.005217861849814653,
+            -0.00339006120339036,
+            0.014887944795191288,
+            0.013161934912204742,
+            0.013701866380870342,
+            -0.03781289979815483,
+            -0.015135781839489937,
+            0.007474951911717653,
+            0.010382615961134434,
+            -0.014994161203503609,
+            -0.0038857359904795885,
+            -0.00031975447200238705,
+            0.003423253772780299,
+            -0.00664292648434639,
+            -0.0017083075363188982,
+            -2.6139097826671787e-05,
+            0.004219873808324337,
+            -0.018924154341220856,
+            -0.009886940941214561,
+            -0.008501706644892693,
+            0.01202896423637867,
+            -0.0003352442872710526,
+            0.026642518118023872,
+            -0.005266543943434954,
+            0.003834840841591358,
+            -0.00877167284488678,
+            0.008178633637726307,
+            -0.014852539636194706,
+            0.0028523425571620464,
+            0.0024562454782426357,
+            0.009479779750108719,
+            0.02949264645576477,
+            -0.0012790178880095482,
+            0.010356061160564423,
+            -0.006164954509586096,
+            0.0178531426936388,
+            0.01709192804992199,
+            -0.007842282764613628,
+            -0.007952923886477947,
+            0.011922747828066349,
+            0.02715589478611946,
+            0.011719167232513428,
+            -0.01519774179905653,
+            0.009506333619356155,
+            0.036467500030994415,
+            -0.01266625989228487,
+            -0.028696026653051376,
+            0.008121099323034286,
+            0.008103396743535995,
+            0.012763625010848045,
+            -0.01579963229596615,
+            -0.0030913285445421934,
+            -0.0029209405183792114,
+            -0.012418422847986221,
+            0.03365277498960495,
+            -0.004669079091399908,
+            -0.012719367630779743,
+            0.002144235884770751,
+            0.007567890919744968,
+            -0.010957952588796616,
+            -0.00732005387544632,
+            -0.03365277498960495,
+            0.00931160431355238,
+            -0.024677520617842674,
+            -0.007651978638023138,
+            -0.02517319656908512,
+            -0.024535899981856346,
+            0.00801488384604454,
+            0.007842282764613628,
+            -0.02476603351533413,
+            0.002686380175873637,
+            -0.015764227136969566,
+            0.011179235763847828,
+            -0.015126930549740791,
+            0.0016795407282188535,
+            -0.0201102327555418,
+            0.014179837889969349,
+            0.007412992883473635,
+            0.0019318037666380405,
+            -0.006142826285213232,
+            -0.013321259059011936,
+            0.0005742303328588605,
+            -0.02481914311647415,
+            -0.016410375013947487,
+            -0.022482389584183693,
+            0.0047752950340509415,
+            -0.02225225605070591,
+            0.005394888576120138,
+            -0.0010643729474395514,
+            0.0004124168772250414,
+            -0.004903639201074839,
+            -0.03460871800780296,
+            -0.015286254696547985,
+            -0.02860751375555992,
+            -0.0018255878239870071,
+            -0.0033435916993767023,
+            0.030430888757109642,
+            -0.009754170663654804,
+            -0.007103195879608393,
+            0.00823174137622118,
+            0.018499288707971573,
+            -0.0009360286640003324,
+            -0.022588606923818588,
+            -0.0036024930886924267,
+            -0.00976302195340395,
+            0.014542742632329464,
+            0.0014471932081505656,
+            -0.02095996029675007,
+            0.0018598866881802678,
+            -0.008953125216066837,
+            -0.008298126049339771,
+            0.020021719858050346,
+            -0.014224094338715076,
+            0.0001842183992266655,
+            -0.0034121894277632236,
+            0.01683523878455162,
+            -0.0017957144882529974,
+            0.008395491167902946,
+            0.009798427112400532,
+            -0.00484168017283082,
+            -0.008067991584539413,
+            -0.011223492212593555,
+            0.029970619827508926,
+            0.016959156841039658,
+            -0.011365113779902458,
+            0.019030369818210602,
+            -0.005695833824574947,
+            0.010630453005433083,
+            -0.016135983169078827,
+            -0.012630854733288288,
+            -0.0019141010707244277,
+            0.0037618172354996204,
+            0.02264171466231346,
+            -0.012498084455728531,
+            0.00900623295456171,
+            -0.011019911617040634,
+            -0.010187885724008083,
+            0.006687183398753405,
+            -0.021119285374879837,
+            -0.0022947085089981556,
+            0.007266945671290159,
+            0.017322061583399773,
+            0.006793399341404438,
+            -0.0009188791736960411,
+            0.01436571590602398,
+            -0.03643209487199783,
+            0.023031173273921013,
+            0.009293901734054089,
+            0.01675557717680931,
+            -0.009647955186665058,
+            0.025084681808948517,
+            -0.00651015667244792,
+            -0.00836893729865551,
+            -0.0030758387874811888,
+            0.0278640016913414,
+            -0.028341973200440407,
+            -0.004257491789758205,
+            0.003027156461030245,
+            -0.01644578017294407,
+            0.015003012493252754,
+            -0.015056120231747627,
+            -0.019225098192691803,
+            -0.0015888145426288247,
+            0.027987919747829437,
+            0.020145637914538383,
+            -0.017047669738531113,
+            0.013905446976423264,
+            -0.0007213837816379964,
+            0.01865861378610134,
+            -0.0014914498897269368,
+            0.002449606778100133,
+            -0.0235445499420166,
+            -0.011126128025352955,
+            0.003907864447683096,
+            0.01022329181432724,
+            0.00497444998472929,
+            0.02368617057800293,
+            0.00957714393734932,
+            -0.010984506458044052,
+            0.003137798048555851,
+            0.01175457239151001,
+            0.0008934315992519259,
+            -0.012737070210278034,
+            -0.009462077170610428,
+            0.014808283187448978,
+            -0.01464010775089264,
+            -0.007041236851364374,
+            0.0065942443907260895,
+            0.00832910556346178,
+            0.03522831201553345,
+            0.002270367229357362,
+            -0.008138801902532578,
+            0.0001835268922150135,
+            0.037919119000434875,
+            0.0017968208994716406,
+            -0.023013470694422722,
+            -0.0005200158921070397,
+            -0.0014294905122369528,
+            0.00578434718772769,
+            -0.00012903586321044713,
+            0.001651880331337452,
+            0.015985511243343353,
+            -0.002372157759964466,
+            -0.0036091317888349295,
+            0.0005570809007622302,
+            0.003617983078584075,
+            0.0023854346945881844,
+            0.01575537584722042,
+            -0.008568092249333858,
+            0.006603095680475235,
+            -0.008191910572350025,
+            -0.009515184909105301,
+            -0.0005355057655833662,
+            0.0034011253155767918,
+            0.018534693866968155,
+            -0.01697685942053795,
+            0.007532485760748386,
+            0.0133920693770051,
+            0.012506935745477676,
+            -0.014489634893834591,
+            0.017083076760172844,
+            0.002892173593863845,
+            -8.899740350898355e-05,
+            0.020765231922268867,
+            0.006080866791307926,
+            -0.006585393100976944,
+            -0.033387232571840286,
+            0.01202011201530695,
+            0.012958354316651821,
+            0.02621765248477459,
+            -0.004536308813840151,
+            -0.001487024244852364,
+            0.0035604494623839855,
+            0.008501706644892693,
+            0.015392471104860306,
+            -0.006226914003491402,
+            -0.00489036226645112,
+            -0.008616774342954159,
+            -0.0019937630277127028,
+            0.001215952099300921,
+            0.009116875007748604,
+            0.013666460290551186,
+            -0.02081833966076374,
+            0.015666862949728966,
+            0.015985511243343353,
+            0.014710918068885803,
+            -0.020499691367149353,
+            0.011533289216458797,
+            -0.012887543067336082,
+            0.009382414631545544,
+            -0.020375771448016167,
+            -0.005213436204940081,
+            0.009453224949538708,
+            -0.004823977593332529,
+            -0.03703398257493973,
+            -0.0020446584094315767,
+            -0.0028656194917857647,
+            0.007258094381541014,
+            -0.0029209405183792114,
+            0.012347611598670483,
+            0.00865217950195074,
+            -0.008599071763455868,
+            -0.003348017344251275,
+            -0.0036688782274723053,
+            -0.008413193747401237,
+            0.01601206511259079,
+            -0.011267748661339283,
+            0.004961173050105572,
+            0.02101306803524494,
+            0.004100380931049585,
+            -0.018747126683592796,
+            0.00718285795301199,
+            -0.02336752414703369,
+            0.0014117879327386618,
+            0.024482792243361473,
+            -0.006368535105139017,
+            -0.006886338349431753,
+            -0.002708508400246501,
+            -0.020694419741630554,
+            -0.014356864616274834,
+            -0.004118083510547876,
+            0.02389860339462757,
+            -0.004474349785596132,
+            -0.02207522839307785,
+            0.015498686581850052,
+            0.023934008553624153,
+            -0.0089265713468194,
+            -0.00028213628684170544,
+            -0.0072979251854121685,
+            -0.014232946559786797,
+            -0.0003125627408735454,
+            0.0011772274738177657,
+            0.019048072397708893,
+            -0.0001921016228152439,
+            -0.0001586325088283047,
+            0.007116473279893398,
+            -0.005571915302425623,
+            -0.002701869932934642,
+            0.029899807646870613,
+            0.00985153578221798,
+            -0.004540734458714724,
+            -0.01648118533194065,
+            0.0193313155323267,
+            -0.007550188340246677,
+            0.014861390925943851,
+            -0.001994869438931346,
+            0.012161733582615852,
+            -0.009347009472548962,
+            -0.007603296544402838,
+            -0.0014095749938860536,
+            -0.0017647348577156663,
+            -0.024358872324228287,
+            0.015436727553606033,
+            0.00502755818888545,
+            -0.007647552993148565,
+            -0.04468153789639473,
+            -0.004531883168965578,
+            -0.01050653401762247,
+            0.0026797414757311344,
+            -0.008594646118581295,
+            -0.005846306681632996,
+            -0.006164954509586096,
+            0.0017127332976087928,
+            -0.042273975908756256,
+            0.010807479731738567,
+            0.021791985258460045,
+            0.0003241801168769598,
+            -0.002584589645266533,
+            -0.0076254247687757015,
+            0.006178231444209814,
+            0.016490036621689796,
+            -0.0029120889957994223,
+            0.004370346665382385,
+            0.008293700404465199,
+            -0.011312006041407585,
+            -0.010541939176619053,
+            0.02315509133040905,
+            0.018924154341220856,
+            0.009745319373905659,
+            0.014250649139285088,
+            0.00701468251645565,
+            -0.009417819790542126,
+            0.01634841598570347,
+            -0.024004818871617317,
+            -0.0122148422524333,
+            -0.005948096979409456,
+            -0.017888547852635384,
+            -0.005248841363936663,
+            -0.018694018945097923,
+            -0.015843888744711876,
+            0.01086058747023344,
+            0.03078494220972061,
+            0.00012578576570376754,
+            -0.007576742675155401,
+            -0.008377788588404655,
+            0.009488631039857864,
+            0.014312608167529106,
+            0.013259299099445343,
+            0.009515184909105301,
+            0.010834033600986004,
+            -0.020535096526145935,
+            -0.009302753023803234,
+            0.0018443968147039413,
+            -0.02876683697104454,
+            0.0026310591492801905,
+            -0.014746323227882385,
+            -0.0201102327555418,
+            -0.02278333529829979,
+            0.00823174137622118,
+            -0.007771471980959177,
+            0.0027549779042601585,
+            -0.005492253229022026,
+            -0.012896394357085228,
+            0.009285050444304943,
+            0.014330310747027397,
+            -0.004443369805812836,
+            -0.015118079259991646,
+            0.01104646548628807,
+            0.006589818745851517,
+            0.004363708198070526,
+            0.0003681601956486702,
+            -0.005200159270316362,
+            -0.0008176420233212411,
+            -0.0031599265057593584,
+            0.013569096103310585,
+            0.017322061583399773,
+            0.010125926695764065,
+            0.01864091120660305,
+            0.0027217853348702192,
+            0.006842081900686026,
+            -0.002210620790719986,
+            -0.003874671645462513,
+            -0.0010411381954327226,
+            -0.02517319656908512,
+            -0.0019849117379635572,
+            -0.024181846529245377,
+            0.012630854733288288,
+            0.001221484155394137,
+            -0.01878253184258938,
+            0.003179841907694936,
+            -0.051620982587337494,
+            -0.011471329256892204,
+            -0.011090721935033798,
+            0.005673705600202084,
+            -0.0055984691716730595,
+            -0.02005712501704693,
+            -0.018587803468108177,
+            0.012462679296731949,
+            -0.006001204717904329,
+            -0.00492576789110899,
+            -0.00031062652124091983,
+            0.004120296332985163,
+            0.009108023717999458,
+            -0.014445378445088863,
+            0.0068199532106518745,
+            -0.007355459034442902,
+            -0.03092656284570694,
+            0.0038813103456050158,
+            0.006337555591017008,
+            -0.013728420250117779,
+            -0.03940614312887192,
+            0.005014281254261732,
+            0.013754974119365215,
+            0.010267548263072968,
+            -0.013383218087255955,
+            -0.009771873243153095,
+            0.011896193958818913,
+            -0.0032086088322103024,
+            0.03041318617761135,
+            -0.008758395910263062,
+            -0.016242198646068573,
+            -0.01464010775089264,
+            0.006505731027573347,
+            0.02152644656598568,
+            0.013551393523812294,
+            -0.03476804122328758,
+            0.0009957751026377082,
+            -0.010612750425934792,
+            -0.0005637193680740893,
+            0.01220599003136158,
+            -0.0024783737026154995,
+            0.002996176714077592,
+            -0.001240293262526393,
+            0.021136987954378128,
+            0.010966803878545761,
+            0.01691490039229393,
+            0.02692575938999653,
+            0.00610742112621665,
+            0.0058241779915988445,
+            -0.01901266723871231,
+            -8.051775694184471e-06,
+            -0.019968610256910324,
+            0.009904643520712852,
+            -0.022836443036794662,
+            0.014834837056696415,
+            -0.004219873808324337,
+            -0.007461674977093935,
+            -2.411642981314799e-06,
+            0.006505731027573347,
+            -0.008612348698079586,
+            -0.016162537038326263,
+            -0.006085292436182499,
+            0.03404223173856735,
+            -0.015826186165213585,
+            0.01683523878455162,
+            -0.010984506458044052,
+            0.01465781033039093,
+            0.015330511145293713,
+            -0.0035958546213805676,
+            -0.018888747319579124,
+            -0.014055919833481312,
+            -0.014215243048965931,
+            -0.010143629275262356,
+            0.0222699586302042,
+            0.019101180136203766,
+            0.009931197389960289,
+            0.0010704583255574107,
+            0.005509955808520317,
+            -0.006036610342562199,
+            0.014666661620140076,
+            0.007138601504266262,
+            0.002217259258031845,
+            -0.01084288489073515,
+            -0.005545360967516899,
+            -0.009860387071967125,
+            0.011763423681259155,
+            0.005266543943434954,
+            -0.004100380931049585,
+            0.008377788588404655,
+            0.005642725620418787,
+            0.0016806471394374967,
+            -0.0015611541457474232,
+            -0.005262118298560381,
+            0.0008071310585364699,
+            0.0003344144788570702,
+            0.0026399106718599796,
+            -0.006961574777960777,
+            0.016197942197322845,
+            -0.012055518105626106,
+            -0.013454028405249119,
+            -0.024341169744729996,
+            -0.006430494599044323,
+            -0.005377185996621847,
+            0.001866525155492127,
+            -0.03643209487199783,
+            0.012807881459593773,
+            0.0003496277204249054,
+            0.0014671087265014648,
+            0.0013785953633487225,
+            -0.0025093534495681524,
+            0.02476603351533413,
+            -0.016729023307561874,
+            -0.010834033600986004,
+            -0.010497682727873325,
+            -0.0018952920800074935,
+            0.002427478553727269,
+            0.005872860550880432,
+            -0.002425265731289983,
+            0.014339162036776543,
+            -0.004372559487819672,
+            -0.009550590068101883,
+            0.006948297843337059,
+            -0.0010084989480674267,
+            -0.020375771448016167,
+            0.008621199987828732,
+            0.014578147791326046,
+            0.018003614619374275,
+            0.026536300778388977,
+            0.00432608975097537,
+            -0.009015084244310856,
+            -0.029262512922286987,
+            -0.004295110236853361,
+            0.03855641186237335,
+            -0.02552725002169609,
+            0.006740291137248278,
+            -0.006952723488211632,
+            0.026500895619392395,
+            0.015950104221701622,
+            0.005235564429312944,
+            0.012790178880095482,
+            -0.008275997824966908,
+            0.0050010038539767265,
+            -0.0009587102103978395,
+            0.008581369183957577,
+            -0.014073622412979603,
+            0.014480783604085445,
+            0.0013099975185468793,
+            -0.005076240282505751,
+            -0.004104806575924158,
+            -0.021986715495586395,
+            -0.024642115458846092,
+            0.019632259383797646,
+            -0.004611545242369175,
+            -0.012489233165979385,
+            0.013524839654564857,
+            -0.011444775387644768,
+            -0.01757875084877014,
+            0.03351115435361862,
+            -0.01258659828454256,
+            -0.015534092672169209,
+            0.0012945076450705528,
+            0.013808081857860088,
+            0.015056120231747627,
+            0.03547614812850952,
+            -0.008050289005041122,
+            0.02299576811492443,
+            0.0035781520418822765,
+            -0.010834033600986004,
+            -0.005563064012676477,
+            -0.02101306803524494,
+            -0.013454028405249119,
+            0.008793801069259644,
+            -0.017375169321894646,
+            0.00932930689305067,
+            0.017631858587265015,
+            0.002112149726599455,
+            0.0012934012338519096,
+            0.02115469053387642,
+            -0.02427035942673683,
+            0.015534092672169209,
+            -0.001430596923455596,
+            0.006129549350589514,
+            0.0012579959584400058,
+            -0.01807442493736744,
+            -0.02134941890835762,
+            -0.01428605429828167,
+            0.003914502914994955,
+            -0.030129943042993546,
+            0.005518807098269463,
+            -0.005386037286370993,
+            -0.020765231922268867,
+            -0.01589699648320675,
+            -0.008864611387252808,
+            -0.0072271148674190044,
+            0.009913494810461998,
+            -0.0016740086721256375,
+            -0.01772037148475647,
+            -0.01915428787469864,
+            0.0033878483809530735,
+            -0.00610299501568079,
+            -0.007253668736666441,
+            0.014914498664438725,
+            -0.010603899136185646,
+            0.011081870645284653,
+            0.01039146725088358,
+            -0.00519573362544179,
+            0.019278205931186676,
+            -0.00878937542438507,
+            0.03795452415943146,
+            -0.002117681782692671,
+            -0.007992755621671677,
+            -0.004801848903298378,
+            -0.0026686773635447025,
+            0.010603899136185646,
+            0.003272780915722251,
+            0.0038813103456050158,
+            0.007377587258815765,
+            0.013781527988612652,
+            -0.011099573224782944,
+            -0.0012248033890500665,
+            0.003967610653489828,
+            0.033369529992341995,
+            -0.014153284020721912,
+            -0.0128521379083395,
+            0.0032949093729257584,
+            -0.010161331854760647,
+            0.0002121554280165583,
+            -0.00024368830781895667,
+            -0.004615970887243748,
+            0.031900208443403244,
+            0.003887948812916875,
+            0.004503116477280855,
+            0.012515787035226822,
+            0.007085493300110102,
+            -0.026270762085914612,
+            0.0007169580785557628,
+            0.0035361081827431917,
+            0.002909876173362136,
+            0.018039019778370857,
+            -0.019844692200422287,
+            -0.019667666405439377,
+            0.008382214233279228,
+            -0.0033325275871902704,
+            -0.004204384051263332,
+            0.0020933407358825207,
+            -0.013418623246252537,
+            -0.00885133445262909,
+            -0.01897726207971573,
+            -0.0013011462287977338,
+            0.004664653446525335,
+            -0.005023132544010878,
+            0.019950907677412033,
+            -0.01793280430138111,
+            -0.04057451710104942,
+            -0.0018654187442734838,
+            0.01121464092284441,
+            0.016233347356319427,
+            0.015622605569660664,
+            0.020181043073534966,
+            -0.002648761961609125,
+            -0.013586798682808876,
+            0.015516390092670918,
+            0.0015854953089728951,
+            0.005341780371963978,
+            -0.0018997177248820662,
+            0.015826186165213585,
+            0.009940048679709435,
+            -0.0034785745665431023,
+            -0.004443369805812836,
+            0.03459101542830467,
+            -0.0046026939526200294,
+            -0.014374567195773125,
+            0.023066578432917595,
+            -0.012958354316651821,
+            0.004157914314419031,
+            0.005355057306587696,
+            -0.0008641115273348987,
+            0.012223693542182446,
+            0.021986715495586395,
+            0.018127532675862312,
+            -0.021809687837958336,
+            -0.00524441571906209,
+            -0.006430494599044323,
+            -0.023084281012415886,
+            -0.010789777152240276,
+            0.010028562508523464,
+            0.004916916601359844,
+            0.024854548275470734,
+            0.0007888752152211964,
+            0.005819752346724272,
+            -0.001608730060979724,
+            -0.008970827795565128,
+            0.0059259687550365925,
+            -0.0035737263970077038,
+            0.006806676276028156,
+            -0.009904643520712852,
+            0.019313612952828407,
+            -0.0012413996737450361,
+            -0.024164143949747086,
+            0.0014648959040641785,
+            0.004388049244880676,
+            -0.008992956019937992,
+            -0.0043504307977855206,
+            0.006926169153302908,
+            -0.0024960762821137905,
+            0.01303801592439413,
+            0.015312808565795422,
+            0.003624621545895934,
+            0.012604300864040852,
+            0.011630653403699398,
+            0.005492253229022026,
+            -0.006846507545560598,
+            0.0014018301153555512,
+            -0.0023478164803236723,
+            -0.004118083510547876,
+            0.01772037148475647,
+            -0.010267548263072968,
+            0.018552398309111595,
+            -0.011444775387644768,
+            0.007253668736666441,
+            0.01954374648630619,
+            -0.0020236363634467125,
+            -0.0012369740288704634,
+            -0.0024540324229747057,
+            0.017224697396159172,
+            0.01492334995418787,
+            -0.002376583404839039,
+            -0.0004226512392051518,
+            0.007868836633861065,
+            0.007426269818097353,
+            0.018959559500217438,
+            -0.0044876267202198505,
+            0.0015866017201915383,
+            -0.002130958717316389,
+            -0.0025934409350156784,
+            -0.021862797439098358,
+            0.0014814921887591481,
+            0.005660428665578365,
+            0.005191307980567217,
+            0.011170384474098682,
+            0.007076642010360956,
+            0.018747126683592796,
+            0.0018764829728752375,
+            -0.021437931805849075,
+            0.011648355983197689,
+            -0.01917199045419693,
+            -0.018083276227116585,
+            0.011993558146059513,
+            -0.023456037044525146,
+            -0.0032086088322103024,
+            -0.02586359903216362,
+            -0.0054834019392728806,
+            -0.00010566280980128795,
+            0.007138601504266262,
+            0.0133920693770051,
+            0.0029143018182367086,
+            0.007837857119739056,
+            -0.015534092672169209,
+            -0.010550790466368198,
+            0.009878089651465416,
+            0.008568092249333858,
+            -0.011223492212593555,
+            0.013967406004667282,
+            -0.007904241792857647,
+            -0.011869640089571476,
+            0.002974048489704728,
+            -0.010373763740062714,
+            -0.0032152472995221615,
+            -0.00157332478556782,
+            0.0063242786563932896,
+            0.0023367523681372404,
+            0.007802451495081186,
+            0.017808884382247925,
+            0.008780524134635925,
+            -0.001572218257933855,
+            -0.008324679918587208,
+            -0.025279412046074867,
+            -0.0014648959040641785,
+            -0.00465580215677619,
+            0.007010256871581078,
+            0.031085887923836708,
+            -0.001259102369658649,
+            0.012250247411429882,
+            0.002670890185981989,
+            0.007497080601751804,
+            -0.0001987401192309335,
+            -0.0004624822468031198,
+            -0.016082875430583954,
+            -0.006465899758040905,
+            -0.0037241990212351084,
+            -0.02097766287624836,
+            0.019756179302930832,
+            0.010559642687439919,
+            0.008050289005041122,
+            0.00912572629749775,
+            -0.015374768525362015,
+            -0.0030869028996676207,
+            0.01880023442208767,
+            -0.0028412784449756145,
+            3.170575655531138e-05,
+            0.017056521028280258,
+            -0.013454028405249119,
+            0.005864009261131287,
+            -0.0031333724036812782,
+            -0.015401322394609451,
+            0.006364109460264444,
+            -0.0036378984805196524,
+            -0.00865217950195074,
+            -0.002591228112578392,
+            0.024730628356337547,
+            0.02244698442518711,
+            0.012082071974873543,
+            -0.016605103388428688,
+            -0.011258897371590137,
+            -0.002659826073795557,
+            -0.001719371764920652,
+            0.011497884057462215,
+            -0.015516390092670918,
+            -0.024288062006235123,
+            0.008470727130770683,
+            -0.008581369183957577,
+            -0.019225098192691803,
+            -0.00020731484983116388,
+            0.02207522839307785,
+            -0.016162537038326263,
+            -0.004180043004453182,
+            0.013223893940448761,
+            -1.0778920113807544e-05,
+            -0.010975655168294907,
+            0.014055919833481312,
+            0.00489036226645112,
+            0.013480582274496555,
+            -0.011453626677393913,
+            -0.0065765418112277985,
+            0.007204986177384853,
+            -0.020588204264640808,
+            -0.00844859890639782,
+            -0.003230737056583166,
+            -0.018587803468108177,
+            0.015684565529227257,
+            -0.010462277568876743,
+            0.00914342887699604,
+            0.004204384051263332,
+            0.0035338953603059053,
+            0.010754371993243694,
+            0.0016286455793306231,
+            0.007603296544402838,
+            -0.004700058605521917,
+            -0.0269788671284914,
+            -0.030094537883996964,
+            0.0011893981136381626,
+            0.0024186272639781237,
+            0.013728420250117779,
+            -0.0026310591492801905,
+            -0.016056321561336517,
+            -0.00966565776616335,
+            0.017605304718017578,
+            -0.030342375859618187,
+            0.016790982335805893,
+            0.01842847838997841,
+            0.0024739480577409267,
+            0.001541238627396524,
+            0.006302150432020426,
+            0.0009874770184978843,
+            -0.016135983169078827,
+            -0.00489036226645112,
+            -0.0011407157871872187,
+            0.02170347236096859,
+            -0.005062963347882032,
+            0.015427876263856888,
+            -0.021774282678961754,
+            0.005518807098269463,
+            -0.012011260725557804,
+            0.0017912888433784246,
+            0.0004840573528781533,
+            0.005992353428155184,
+            0.003629047190770507,
+            -0.008992956019937992,
+            0.010090521536767483,
+            -0.005031983833760023,
+            0.00042514066444709897,
+            0.004275194834917784,
+            0.05392232909798622,
+            0.00392999267205596,
+            0.03795452415943146,
+            -0.015312808565795422,
+            -0.00885133445262909,
+            0.025421032682061195,
+            0.011480181477963924,
+            -0.007443972397595644,
+            0.01917199045419693,
+            0.010179034434258938,
+            0.018481586128473282,
+            0.003328101709485054,
+            -0.0008696436416357756,
+            0.02223455347120762,
+            0.018198344856500626,
+            0.006368535105139017,
+            0.008059140294790268,
+            0.008391065523028374,
+            -0.004016293212771416,
+            0.019118882715702057,
+            -0.0005769963609054685,
+            -0.0195260439068079,
+            0.007231540512293577,
+            -0.01277247630059719,
+            0.015516390092670918,
+            0.0006262319511733949,
+            0.0021331715397536755,
+            0.010533087886869907,
+            0.0034365307074040174,
+            0.0023190497886389494,
+            0.0027726804837584496,
+            0.0007540230872109532,
+            -0.0025890152901411057,
+            0.012506935745477676,
+            0.010913695208728313,
+            0.0031909062527120113,
+            -0.007005831226706505,
+            0.010913695208728313,
+            0.013445177115499973,
+            0.0033258888870477676,
+            -0.012524638324975967,
+            -0.011612950824201107,
+            -0.008700861595571041,
+            -0.015790781006217003,
+            0.010187885724008083,
+            0.003611344611272216,
+            -0.009993156418204308,
+            -0.025969816371798515,
+            0.00849728100001812,
+            0.00986923836171627,
+            -0.017746925354003906,
+            0.0030625618528574705,
+            0.00826272089034319,
+            0.000512824219185859,
+            0.006032184697687626,
+            0.020358068868517876,
+            0.0006129549001343548,
+            0.012179436162114143,
+            0.01130315475165844,
+            -0.0029452815651893616,
+            -0.004872659686952829,
+            0.015604902990162373,
+            0.01411787886172533,
+            -0.008992956019937992,
+            -0.016020916402339935,
+            -0.012347611598670483,
+            -0.035175204277038574,
+            -0.006381812505424023,
+            -0.00357593921944499,
+            -0.01237416546791792,
+            -0.002407562918961048,
+            -0.00023013469763100147,
+            0.012701665051281452,
+            -0.024730628356337547,
+            0.010311804711818695,
+            -0.021083878353238106,
+            -0.017454832792282104,
+            -0.00013359983859118074,
+            0.00674471678212285,
+            -0.01697685942053795,
+            0.00044173692003823817,
+            0.006850933190435171,
+            0.0009907962521538138,
+            0.0027992345858365297,
+            0.048399098217487335,
+            -0.02641238272190094,
+            -0.0008840270456857979,
+            -0.009293901734054089,
+            0.017587602138519287,
+            0.019295910373330116,
+            0.012356462888419628,
+            -0.004372559487819672,
+            0.008864611387252808,
+            -0.006164954509586096,
+            0.003783945459872484,
+            0.024465089663863182,
+            0.004157914314419031,
+            0.025279412046074867,
+            0.005213436204940081,
+            0.0017149461200460792,
+            -0.01475517451763153,
+            -0.005209010560065508,
+            -0.014445378445088863,
+            -0.0013365515042096376,
+            -0.021614959463477135,
+            -0.03675074130296707,
+            0.02062360942363739,
+            -0.017534494400024414,
+            0.015003012493252754,
+            -0.004934619180858135,
+            0.0008032586192712188,
+            0.005076240282505751,
+            0.022323066368699074,
+            0.015604902990162373,
+            -0.022199146449565887,
+            0.030678726732730865,
+            0.004290684591978788,
+            0.03328101709485054,
+            0.0026376976165920496,
+            -0.0014604702591896057,
+            0.0017868631985038519,
+            0.020535096526145935,
+            0.0003781179548241198,
+            -0.011639504693448544,
+            -0.0020601481664925814,
+            0.012285652570426464,
+            -0.017649561166763306,
+            0.003852543421089649,
+            -0.03018305078148842,
+            -0.012630854733288288,
+            0.0027439137920737267,
+            -0.0032749937381595373,
+            0.00048710001283325255,
+            -0.01436571590602398,
+            0.03271453455090523,
+            -0.0046203965321183205,
+            0.011807680130004883,
+            0.023385226726531982,
+            -0.0036378984805196524,
+            -0.01148903276771307,
+            -0.016870643943548203,
+            0.01883563958108425,
+            -0.013799230568110943,
+            0.01842847838997841,
+            0.009771873243153095,
+            0.0011595248943194747,
+            -0.0014859178336337209,
+            -0.0021951310336589813,
+            0.00015379194519482553,
+            0.0016441354528069496,
+            0.007289073895663023,
+            -0.004509754944592714,
+            -0.020535096526145935,
+            0.021314013749361038,
+            -0.011391667649149895,
+            -0.00618708273395896,
+            0.021986715495586395,
+            -0.013268150389194489,
+            0.006341981235891581,
+            0.015020715072751045,
+            -0.02788170427083969,
+            -0.03540533781051636,
+            0.019791584461927414,
+            -0.003113457001745701,
+            0.008156504482030869,
+            0.005594043526798487,
+            0.00841761939227581,
+            0.008935422636568546,
+            0.00994889996945858,
+            -0.008913294412195683,
+            0.05919772759079933,
+            -0.021190095692873,
+            0.0032108216546475887,
+            0.0007175113423727453,
+            -0.017295507714152336,
+            0.014082473702728748,
+            0.00939126592129469,
+            0.010612750425934792,
+            -0.013091123662889004,
+            -0.008174207992851734,
+            -0.017083076760172844,
+            3.744529385585338e-05,
+            0.0015666862018406391,
+            -0.0008696436416357756,
+            -0.006541136186569929,
+            -0.003989738877862692,
+            -0.0394415482878685,
+            -0.02765156887471676,
+            0.03813154995441437,
+            -0.005279821343719959,
+            0.016684764996170998,
+            -0.006443771533668041,
+            -0.008718564175069332,
+            0.0014471932081505656,
+            0.010364912450313568,
+            0.004135786090046167,
+            -0.012338760308921337,
+            -0.005828603636473417,
+            0.007412992883473635,
+            -0.0561174601316452,
+            -0.0022438133601099253,
+            0.0057710702531039715,
+            -0.007209412287920713,
+            0.0033192504197359085,
+            0.018322262912988663,
+            0.02460671029984951,
+            -0.011196938343346119,
+            0.011453626677393913,
+            -0.004427880048751831,
+            0.011825382709503174,
+            0.02315509133040905,
+            0.0027815320063382387,
+            0.005116071552038193,
+            0.013604501262307167,
+            0.004691207315772772,
+            -0.008762821555137634,
+            -0.009754170663654804,
+            -0.018127532675862312,
+            -0.011312006041407585,
+            0.006032184697687626,
+            0.0003062008472625166,
+            0.004396900534629822,
+            0.014560445211827755,
+            0.017959358170628548,
+            -0.015808483585715294,
+            0.02133171632885933,
+            0.013631055131554604,
+            -0.005859583616256714,
+            -1.9491951661620988e-06,
+            -0.014967607334256172,
+            -0.011922747828066349,
+            -0.004135786090046167,
+            -0.010772074572741985,
+            0.01212632842361927,
+            -0.04163667932152748,
+            0.019083477556705475,
+            -0.0032285242341458797,
+            -0.010081670247018337,
+            -0.011851937510073185,
+            0.006846507545560598,
+            0.032112643122673035,
+            0.0030470718629658222,
+            -0.014445378445088863,
+            0.0012358676176518202,
+            0.004531883168965578,
+            -0.023827793076634407,
+            0.01585274003446102,
+            -0.008333531208336353,
+            0.001153992721810937,
+            -0.009050489403307438,
+            -0.015773078426718712,
+            0.04443369805812836,
+            -0.014808283187448978,
+            0.02203982323408127,
+            0.034626420587301254,
+            0.029404133558273315,
+            -0.0010887141106650233,
+            -0.012551192194223404,
+            -0.0139320008456707,
+            0.017083076760172844,
+            0.0032019703648984432,
+            0.012179436162114143,
+            0.017516791820526123,
+            0.017410574480891228,
+            -0.013799230568110943,
+            0.014073622412979603,
+            -0.006346406880766153,
+            -0.006727014202624559,
+            0.00046995055163279176,
+            -0.0020490840543061495,
+            0.0073997159488499165,
+            0.013400920666754246,
+            0.02062360942363739,
+            -0.01086058747023344,
+            0.0017249038210138679,
+            0.02264171466231346,
+            0.017047669738531113,
+            0.011922747828066349,
+            -0.008072417229413986,
+            3.713411570060998e-05,
+            0.005164753645658493,
+            0.010356061160564423,
+            0.008382214233279228,
+            0.004580565728247166,
+            -0.01011707540601492,
+            0.024535899981856346,
+            0.004270769190043211,
+            0.0008060246473178267,
+            0.022588606923818588,
+            -0.01076322328299284,
+            -0.00701468251645565,
+            -0.0016242199344560504,
+            0.007882113568484783,
+            -0.006797824986279011,
+            -0.03558236360549927,
+            0.02025185339152813,
+            -0.015339363366365433,
+            -0.01707422360777855,
+            0.01022329181432724,
+            -0.010984506458044052,
+            0.007992755621671677,
+            -0.0025757383555173874,
+            0.013666460290551186,
+            -0.009435522370040417,
+            0.0015622605569660664,
+            0.007005831226706505,
+            -0.0029497072100639343,
+            0.017596453428268433,
+            -0.014941052533686161,
+            0.009320455603301525,
+            -0.008815929293632507,
+            0.04609775170683861,
+            0.0140293650329113,
+            0.0026376976165920496,
+            -0.0030382205732166767,
+            -0.008112248033285141,
+            0.0006129549001343548,
+            0.003963185008615255,
+            2.6778745450428687e-05,
+            -0.008205187506973743,
+            -0.02621765248477459,
+            -0.006757994182407856,
+            -0.002617782214656472,
+            0.01672017201781273,
+            -0.008395491167902946,
+            0.019402125850319862,
+            0.013701866380870342,
+            -0.019136585295200348,
+            -0.02281874045729637,
+            0.022287661209702492,
+            -0.00809454545378685,
+            -0.007054513785988092,
+            -0.0067491428926587105,
+            -0.01810983009636402,
+            -0.00024811396724544466,
+            0.025704275816679,
+            0.003976461943238974,
+            0.0015102589968591928,
+            0.0023920731619000435,
+            0.004770869389176369,
+            0.025704275816679,
+            0.001831119880080223,
+            0.023119686171412468,
+            0.004016293212771416,
+            0.009453224949538708,
+            -0.012064369395375252,
+            0.0011827596463263035,
+            0.0014770664274692535,
+            -0.019649963825941086,
+            -0.011896193958818913,
+            0.0036578141152858734,
+            0.0065765418112277985,
+            0.006850933190435171,
+            0.017145035788416862,
+            0.016923751682043076,
+            -0.014047068543732166,
+            0.009134577587246895,
+            -0.008581369183957577,
+            -0.004507542122155428,
+            0.00966565776616335,
+            0.012453828006982803,
+            -0.00846630148589611,
+            0.0016253263456746936,
+            0.009709914214909077,
+            -0.007174006663262844,
+            0.018198344856500626,
+            0.01583503745496273,
+            0.0001586325088283047,
+            -0.002617782214656472,
+            0.03094426542520523,
+            0.007492654956877232,
+            -0.015241998247802258,
+            0.009187685325741768,
+            -0.025456437841057777
+          ],
+          "title": "9 Conclusion"
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "b8ffe9ad-9e6f-4478-8767-d2c5561f1266",
+        "properties": {
+          "page_content": "A Switch for Attention 27 ",
+          "keyphrases": [
+            "Switch",
+            "Attention"
+          ],
+          "embedding": [
+            -0.004847599659115076,
+            -0.013338678516447544,
+            -0.017561476677656174,
+            0.018164733424782753,
+            -0.036252863705158234,
+            0.0018600418698042631,
+            0.006209715269505978,
+            0.06599438190460205,
+            0.014593069441616535,
+            -0.005357495043426752,
+            -0.02489631250500679,
+            0.024474991485476494,
+            0.01628793403506279,
+            -0.022100266069173813,
+            -0.018451998010277748,
+            -0.02642839401960373,
+            -0.014775004237890244,
+            0.0039068059995770454,
+            -0.036712486296892166,
+            -0.008148754946887493,
+            0.020644789561629295,
+            -0.02066393941640854,
+            -0.034203704446554184,
+            0.016929492354393005,
+            -0.03311209753155708,
+            -0.00733004929497838,
+            0.03295888751745224,
+            0.011835324577987194,
+            -0.04140448570251465,
+            0.04393241927027702,
+            -0.03259501978754997,
+            0.010667112655937672,
+            -2.788491838146001e-05,
+            0.008067362941801548,
+            -0.04025542363524437,
+            -0.014545192010700703,
+            -0.0072869593277573586,
+            -0.023517441004514694,
+            0.001378872781060636,
+            0.02303866483271122,
+            -0.045158080756664276,
+            -0.006899151485413313,
+            -0.05542302504181862,
+            -0.02713698148727417,
+            0.010226639918982983,
+            -0.015732554718852043,
+            -0.02164064161479473,
+            -0.03544852137565613,
+            -0.0038780795875936747,
+            -0.018672235310077667,
+            0.024723954498767853,
+            -0.03631031513214111,
+            -0.0005529854097403586,
+            0.00697575556114316,
+            0.007416228763759136,
+            0.0014470982132479548,
+            -0.003313124878332019,
+            0.010542631149291992,
+            -0.011088434606790543,
+            0.037287015467882156,
+            0.003653055289760232,
+            -0.01817430928349495,
+            -0.0034160614013671875,
+            -0.024800557643175125,
+            -0.015694253146648407,
+            -0.011126737110316753,
+            -0.022119417786598206,
+            -0.0168050117790699,
+            -0.022866306826472282,
+            -0.0010132081806659698,
+            -0.042974866926670074,
+            0.012716270983219147,
+            -0.04653695598244667,
+            0.01056178193539381,
+            0.006458678282797337,
+            0.007713069207966328,
+            0.03776579350233078,
+            0.03698059916496277,
+            -0.05538472160696983,
+            -0.002126959152519703,
+            -0.010284092277288437,
+            -0.02424517832696438,
+            0.010657536797225475,
+            0.026409242302179337,
+            0.011777871288359165,
+            -0.02905208244919777,
+            -0.02650499902665615,
+            -0.09989167004823685,
+            0.05113319680094719,
+            0.0056447600945830345,
+            -0.02842009998857975,
+            -0.025164427235722542,
+            0.007488044910132885,
+            -0.006329408846795559,
+            0.01437283307313919,
+            -0.008871705271303654,
+            0.028688212856650352,
+            -0.014803730882704258,
+            -0.007205567788332701,
+            -0.0008031455217860639,
+            0.029109535738825798,
+            -0.0010479193879291415,
+            -0.025624051690101624,
+            -0.025164427235722542,
+            0.03203963860869408,
+            0.03052671067416668,
+            -0.02981812320649624,
+            0.024091970175504684,
+            0.0008827418787404895,
+            0.004270675126463175,
+            0.01378872711211443,
+            -0.01235240139067173,
+            -0.013941935263574123,
+            -0.0073731387965381145,
+            0.004337703809142113,
+            -0.002435769187286496,
+            0.002875045407563448,
+            -0.018384970724582672,
+            -0.014075992628932,
+            -0.009834043681621552,
+            -0.0008546138415113091,
+            0.030737372115254402,
+            0.004285038448870182,
+            -0.012371552176773548,
+            -0.020568184554576874,
+            -0.005850633606314659,
+            0.012591789476573467,
+            0.01174914464354515,
+            -0.014028115198016167,
+            0.010102157481014729,
+            0.01855732873082161,
+            0.019601058214902878,
+            -0.0066980659030377865,
+            0.0014459012309089303,
+            0.023517441004514694,
+            0.002047961112111807,
+            0.025815561413764954,
+            0.03431861102581024,
+            -0.03956598788499832,
+            -0.0008270842372439802,
+            -0.028305193409323692,
+            0.0037583857774734497,
+            0.015598498284816742,
+            0.02265564538538456,
+            -0.013032262213528156,
+            0.024015367031097412,
+            0.014822881668806076,
+            0.009977675974369049,
+            -0.009541991166770458,
+            0.029473405331373215,
+            0.018643507733941078,
+            -0.0008181072189472616,
+            -0.03347596526145935,
+            -0.001977341715246439,
+            0.054235659539699554,
+            0.021104414016008377,
+            -0.0035405431408435106,
+            -0.00020632220548577607,
+            -0.01043730042874813,
+            0.010274517349898815,
+            -0.01779128797352314,
+            -0.006635825149714947,
+            -0.021295923739671707,
+            -0.004670452792197466,
+            0.00989149697124958,
+            -0.005845845676958561,
+            0.033514268696308136,
+            0.00400495482608676,
+            -0.031580016016960144,
+            0.05132470652461052,
+            0.029224442318081856,
+            0.025183578953146935,
+            0.02834349498152733,
+            0.015943216159939766,
+            0.0025758109986782074,
+            0.00035818375181406736,
+            0.05224395543336868,
+            0.013530189171433449,
+            0.003772749099880457,
+            0.014803730882704258,
+            -0.035295311361551285,
+            0.006875212769955397,
+            -0.0050558666698634624,
+            -0.025834713131189346,
+            0.008967460133135319,
+            0.0135972173884511,
+            0.02169809490442276,
+            0.008057788014411926,
+            -0.0009449826902709901,
+            0.004304189700633287,
+            -0.025585750117897987,
+            -0.03711465746164322,
+            0.019725540652871132,
+            -0.02087460085749626,
+            -0.01820303499698639,
+            -0.01625920832157135,
+            -0.020625637844204903,
+            -0.0026284761261194944,
+            0.04366430267691612,
+            0.00017774531443137676,
+            -0.009977675974369049,
+            0.026332639157772064,
+            0.018672235310077667,
+            0.012074711732566357,
+            -0.05971284955739975,
+            0.003966652788221836,
+            -0.009355268441140652,
+            -0.020204314962029457,
+            -0.0038541408721357584,
+            0.00015784621064085513,
+            0.03276737779378891,
+            -0.004438246600329876,
+            0.01959148421883583,
+            0.01880629174411297,
+            0.005544217303395271,
+            0.006554433144629002,
+            0.008124816231429577,
+            0.015761282294988632,
+            0.006875212769955397,
+            -0.0067076412960886955,
+            0.039834100753068924,
+            0.02196620963513851,
+            -0.006295894738286734,
+            0.025241032242774963,
+            -0.06572626531124115,
+            0.040025610476732254,
+            -0.03397389128804207,
+            0.03052671067416668,
+            0.0010760474251583219,
+            -0.027749814093112946,
+            -0.07373139262199402,
+            0.0029277107678353786,
+            0.005917662288993597,
+            0.03077567368745804,
+            -0.008775950409471989,
+            -0.004912233911454678,
+            -0.009790954180061817,
+            0.01677628420293331,
+            0.01882544346153736,
+            -0.013999388553202152,
+            0.01795407198369503,
+            0.012045985087752342,
+            -0.015349534340202808,
+            0.0011514545185491443,
+            0.004505275283008814,
+            -0.009259513579308987,
+            -0.020127711817622185,
+            0.027481699362397194,
+            -0.02309611812233925,
+            -0.012620516121387482,
+            -0.0157134048640728,
+            0.0023316354490816593,
+            -0.019026529043912888,
+            0.012180042453110218,
+            -0.0004691997601184994,
+            0.05978945270180702,
+            -0.011270369403064251,
+            -0.02016601338982582,
+            0.0089483093470335,
+            0.014468587934970856,
+            0.009939374402165413,
+            0.02522188052535057,
+            -0.005553793162107468,
+            0.039834100753068924,
+            0.010312818922102451,
+            -0.027673209086060524,
+            0.03742107376456261,
+            -0.05982775613665581,
+            0.006616673897951841,
+            -0.019409548491239548,
+            0.002752957632765174,
+            -0.006286319345235825,
+            0.013549339957535267,
+            0.01759977824985981,
+            0.014391983859241009,
+            0.020912902429699898,
+            0.0032676411792635918,
+            6.702853715978563e-05,
+            -0.014190898276865482,
+            -0.018136005848646164,
+            -0.010466027073562145,
+            0.06400267779827118,
+            0.033629175275564194,
+            -0.005525066517293453,
+            -0.0273284912109375,
+            0.00811524037271738,
+            0.03194388374686241,
+            0.03274822607636452,
+            -0.028822271153330803,
+            -0.0005173765239305794,
+            0.004268281627446413,
+            -0.018193459138274193,
+            0.0030162842012941837,
+            -0.044315438717603683,
+            0.03487398847937584,
+            -0.0241877268999815,
+            0.037018902599811554,
+            0.04795413091778755,
+            0.005022352561354637,
+            -0.019917050376534462,
+            -0.03000963293015957,
+            -0.014746277593076229,
+            -0.005209074821323156,
+            -0.03272907808423042,
+            -0.0379764549434185,
+            0.0254133902490139,
+            0.01751359924674034,
+            0.021544886752963066,
+            -0.06415588408708572,
+            0.014707976020872593,
+            0.05676359310746193,
+            0.014075992628932,
+            -0.014794155023992062,
+            -0.017283786088228226,
+            -0.010169186629354954,
+            -0.0022598193027079105,
+            0.018566904589533806,
+            0.010485177859663963,
+            -0.014535617083311081,
+            -0.03090973012149334,
+            -0.017140153795480728,
+            -0.004510062746703625,
+            -0.025758108124136925,
+            0.03680824115872383,
+            -0.015071844682097435,
+            -0.010418149642646313,
+            0.008507836610078812,
+            0.037918999791145325,
+            -0.033629175275564194,
+            0.012409854680299759,
+            -0.055959250777959824,
+            -0.02093205414712429,
+            -0.008479109965264797,
+            -0.01735081523656845,
+            0.0037129020784050226,
+            -0.0025087823159992695,
+            0.014085568487644196,
+            0.02476225607097149,
+            -0.01907440647482872,
+            0.006564009003341198,
+            -0.011500181630253792,
+            0.015646375715732574,
+            -0.007665191777050495,
+            -0.020778845995664597,
+            -0.05235886201262474,
+            0.0008205011254176497,
+            0.03027774766087532,
+            -0.01257263869047165,
+            0.015234628692269325,
+            -0.0014913849299773574,
+            0.0011592345545068383,
+            -0.02137252688407898,
+            0.007320473901927471,
+            -0.006564009003341198,
+            0.036003898829221725,
+            -0.018605206161737442,
+            -0.023019514977931976,
+            0.011902352795004845,
+            -0.011605512350797653,
+            -0.01891162246465683,
+            0.025011219084262848,
+            -0.03263332322239876,
+            0.01043730042874813,
+            0.005817119497805834,
+            -0.06281531602144241,
+            -0.05163112282752991,
+            0.0039762286469340324,
+            -0.009786166250705719,
+            -0.009872346185147762,
+            -0.004897871054708958,
+            -0.08097047358751297,
+            -0.009671260602772236,
+            -0.04385581240057945,
+            0.02552829682826996,
+            0.0168050117790699,
+            0.017341239377856255,
+            -0.0038876549806445837,
+            0.005750090815126896,
+            -0.03761258348822594,
+            0.02330677956342697,
+            -0.014842033386230469,
+            0.017427420243620872,
+            -0.019189313054084778,
+            -0.029971331357955933,
+            0.04063844308257103,
+            0.02700292505323887,
+            0.02533678710460663,
+            0.006592735182493925,
+            -0.043626002967357635,
+            0.028017928823828697,
+            -0.021391678601503372,
+            -0.005075017921626568,
+            -0.0064491028897464275,
+            0.0048859016969799995,
+            0.021927906200289726,
+            -0.008129604160785675,
+            0.008268448524177074,
+            -0.0007175644277594984,
+            -0.001700849155895412,
+            -0.05339301750063896,
+            -0.013520613312721252,
+            0.02661990374326706,
+            -0.005697425454854965,
+            -0.030392654240131378,
+            0.010801170021295547,
+            0.010063855908811092,
+            -0.06534324586391449,
+            -0.010944802314043045,
+            0.0004428671090863645,
+            0.013156743720173836,
+            0.024398386478424072,
+            -0.06534324586391449,
+            0.013338678516447544,
+            -0.030584163963794708,
+            0.010628811083734035,
+            0.045541103929281235,
+            0.006157049909234047,
+            0.0551932118833065,
+            0.004232373088598251,
+            0.0008839388028718531,
+            0.014545192010700703,
+            0.0316949225962162,
+            0.013980237767100334,
+            0.02951170690357685,
+            -0.014238775707781315,
+            0.0012543911579996347,
+            0.03596559911966324,
+            0.005199499428272247,
+            0.04301317036151886,
+            0.0017367572290822864,
+            0.00875679962337017,
+            -0.04370260611176491,
+            -0.03975749760866165,
+            -0.008866917341947556,
+            -0.0007863883511163294,
+            -0.001040737726725638,
+            -0.03994900733232498,
+            -0.0033107309136539698,
+            -0.03968089446425438,
+            -0.04423883557319641,
+            0.02323017455637455,
+            0.03994900733232498,
+            -0.02426433004438877,
+            0.01786789298057556,
+            -0.007732220459729433,
+            0.019763842225074768,
+            0.002664384199306369,
+            -0.013204621151089668,
+            0.006477829068899155,
+            0.0224066823720932,
+            -0.028190286830067635,
+            0.03617625683546066,
+            -0.0029588311444967985,
+            0.002010856056585908,
+            -0.007052359636873007,
+            0.004778177011758089,
+            0.01732208952307701,
+            0.026102827861905098,
+            -0.031139543280005455,
+            -0.025432541966438293,
+            0.0445452481508255,
+            0.03265247121453285,
+            0.03768918663263321,
+            -0.04940960556268692,
+            0.005448462441563606,
+            -0.005620821379125118,
+            -0.06055549532175064,
+            0.04075334966182709,
+            -9.201461944030598e-05,
+            0.014688825234770775,
+            -0.03060331381857395,
+            -0.022559890523552895,
+            -0.04703488200902939,
+            0.007018845062702894,
+            0.0016769103240221739,
+            -0.015483591705560684,
+            -0.015502742491662502,
+            -0.04504317790269852,
+            -0.027098679915070534,
+            -0.0336100235581398,
+            -0.0188541691750288,
+            -0.01513887383043766,
+            -0.0037224777042865753,
+            0.003921169321984053,
+            0.01097352895885706,
+            0.016709256917238235,
+            0.01247688289731741,
+            -0.008210995234549046,
+            -0.05186093598604202,
+            -0.01637411303818226,
+            0.022349229082465172,
+            -0.008622742258012295,
+            0.035161253064870834,
+            -0.006678914651274681,
+            -0.013022687286138535,
+            -0.053278110921382904,
+            -0.01699652150273323,
+            -0.04243863746523857,
+            0.02393876202404499,
+            0.05389094352722168,
+            0.00978137832134962,
+            -0.012936507351696491,
+            0.007511983625590801,
+            -0.0024537232238799334,
+            -0.019007377326488495,
+            -0.017475297674536705,
+            0.02566235326230526,
+            -0.017044398933649063,
+            -0.0073635634034872055,
+            -0.027290189638733864,
+            0.019763842225074768,
+            0.027347642928361893,
+            0.01561764907091856,
+            0.0010646764421835542,
+            0.02003195695579052,
+            0.004567516036331654,
+            -0.003028253559023142,
+            0.027826417237520218,
+            0.01593364030122757,
+            -0.0025303272996097803,
+            0.01721675880253315,
+            -0.012314099818468094,
+            0.005477189086377621,
+            0.042093921452760696,
+            -0.01270669512450695,
+            0.018815867602825165,
+            0.03678908944129944,
+            -0.00822535902261734,
+            0.027998777106404305,
+            -0.04132787883281708,
+            0.021295923739671707,
+            0.046115633100271225,
+            -0.02368979901075363,
+            0.009206848219037056,
+            -0.001639805268496275,
+            0.002424996579065919,
+            -0.014583494514226913,
+            0.002250243676826358,
+            -0.02188960462808609,
+            -0.0060325684025883675,
+            0.00822535902261734,
+            0.016307085752487183,
+            -0.02778811566531658,
+            -0.01008300669491291,
+            0.010456451214849949,
+            0.02932019717991352,
+            0.025777259841561317,
+            -0.025451691821217537,
+            0.01462179608643055,
+            -0.009527627378702164,
+            0.03301634266972542,
+            -0.016910342499613762,
+            0.03188643231987953,
+            0.004318553023040295,
+            -0.007095449138432741,
+            -0.02956916019320488,
+            -0.012783299200236797,
+            0.04653695598244667,
+            0.005893723573535681,
+            0.010580932721495628,
+            0.003533361479640007,
+            -0.01718803122639656,
+            0.021046960726380348,
+            0.02573895826935768,
+            -0.010963953100144863,
+            -0.004222797695547342,
+            -0.0013704941375181079,
+            0.033188700675964355,
+            0.03148426115512848,
+            -0.014401559717953205,
+            -0.017724260687828064,
+            0.0016350174555554986,
+            0.01283117663115263,
+            -0.028764817863702774,
+            -0.004581879358738661,
+            0.058333978056907654,
+            0.004543577320873737,
+            -0.02713698148727417,
+            -0.015004816465079784,
+            -0.007162477821111679,
+            -0.011299096047878265,
+            -0.023900460451841354,
+            0.029722368344664574,
+            -0.009800529107451439,
+            -0.02137252688407898,
+            -0.02183215133845806,
+            0.00822535902261734,
+            -0.026658207178115845,
+            -0.03129275143146515,
+            0.03014368936419487,
+            0.017810439690947533,
+            0.04014051705598831,
+            0.006338984239846468,
+            -0.015837885439395905,
+            -0.024666501209139824,
+            4.4511136366054416e-05,
+            0.01978299394249916,
+            -0.01490906160324812,
+            -0.0013297982513904572,
+            -0.034012194722890854,
+            0.004964899271726608,
+            0.010983104817569256,
+            0.018068978562951088,
+            -0.01577085629105568,
+            0.0026715658605098724,
+            0.029281895607709885,
+            -0.013444009236991405,
+            0.012055560946464539,
+            -0.017695533111691475,
+            0.016623076051473618,
+            -0.01658477447926998,
+            -0.005161197390407324,
+            0.033629175275564194,
+            -0.03678908944129944,
+            0.008129604160785675,
+            0.003705720417201519,
+            0.009379207156598568,
+            0.025049520656466484,
+            -0.002621294464915991,
+            -0.005333556327968836,
+            0.024302631616592407,
+            -0.021334225311875343,
+            0.024934615939855576,
+            0.008234934881329536,
+            0.010188337415456772,
+            0.006640612613409758,
+            -0.017504023388028145,
+            -0.011710843071341515,
+            -0.02183215133845806,
+            -0.0018253306625410914,
+            0.01132782269269228,
+            -0.0047135422937572,
+            -0.008455171249806881,
+            -0.019285067915916443,
+            0.0004862561181653291,
+            0.009680835530161858,
+            0.04557940363883972,
+            -0.014870759099721909,
+            0.013846180401742458,
+            0.01295565813779831,
+            -0.010130884125828743,
+            0.007282171864062548,
+            0.03140765801072121,
+            0.03372493013739586,
+            0.0008701740298420191,
+            0.004385581240057945,
+            0.022310927510261536,
+            0.026773111894726753,
+            0.0032125820871442556,
+            0.0057213641703128815,
+            -0.007492832839488983,
+            0.00989149697124958,
+            0.0008456368232145905,
+            0.01951487921178341,
+            0.011959806084632874,
+            -0.03014368936419487,
+            -0.02497291751205921,
+            -0.011385275982320309,
+            -0.023823857307434082,
+            0.01948615349829197,
+            -0.031637467443943024,
+            -0.02060648798942566,
+            0.011844899505376816,
+            0.009455811232328415,
+            0.0021125958301126957,
+            -0.00465369550511241,
+            -0.02093205414712429,
+            0.03378238156437874,
+            -0.015129297971725464,
+            0.060708701610565186,
+            -0.009757439605891705,
+            -0.03713380917906761,
+            -0.023670649155974388,
+            -0.008163117803633213,
+            -0.017255060374736786,
+            -0.03768918663263321,
+            -0.020070258527994156,
+            0.022808853536844254,
+            -0.01091607566922903,
+            -0.017532749101519585,
+            -0.0008426444255746901,
+            -0.000540118315257132,
+            -0.011624663136899471,
+            0.0006493389373645186,
+            0.021908756345510483,
+            0.005046291276812553,
+            0.0008887265576049685,
+            0.008124816231429577,
+            -0.021851303055882454,
+            0.005754878744482994,
+            -0.004373611882328987,
+            0.02278970181941986,
+            -0.026390092447400093,
+            -0.0026308700907975435,
+            -0.029684066772460938,
+            -0.05986605957150459,
+            -0.030316049233078957,
+            -0.017618929967284203,
+            0.012926931492984295,
+            0.007468894124031067,
+            0.011509757488965988,
+            -0.025566598400473595,
+            -0.004541183356195688,
+            -0.011978956870734692,
+            0.011710843071341515,
+            0.022483285516500473,
+            -0.005496339872479439,
+            -0.020836299285292625,
+            -0.003054586239159107,
+            -0.022061964496970177,
+            0.006812972016632557,
+            0.006927878130227327,
+            -0.0273284912109375,
+            0.05484849214553833,
+            -0.0005957759567536414,
+            0.0325184166431427,
+            -0.013568490743637085,
+            -0.019581908360123634,
+            -0.040102217346429825,
+            -0.011069283820688725,
+            -0.007248657289892435,
+            0.022425832226872444,
+            -0.025126125663518906,
+            -0.009609019383788109,
+            -0.019342521205544472,
+            -0.008230146951973438,
+            0.023670649155974388,
+            0.04224712774157524,
+            0.0333419106900692,
+            -0.003293973859399557,
+            0.00793809350579977,
+            0.002160473261028528,
+            -0.016623076051473618,
+            -0.014363258145749569,
+            -0.014315379783511162,
+            -0.003641085932031274,
+            0.007224718574434519,
+            0.03537191450595856,
+            0.030124539509415627,
+            -0.04305147007107735,
+            0.018126431852579117,
+            0.03784239664673805,
+            0.010054280050098896,
+            0.014344106428325176,
+            0.019744692370295525,
+            0.011729993857443333,
+            0.02905208244919777,
+            0.015081420540809631,
+            -0.01306098885834217,
+            -0.005917662288993597,
+            -0.010025554336607456,
+            -0.03339936211705208,
+            0.0243217833340168,
+            -0.010446876287460327,
+            -0.010284092277288437,
+            0.016766710206866264,
+            0.019581908360123634,
+            -0.0005595685797743499,
+            0.019821295514702797,
+            0.0024776619393378496,
+            -0.032863132655620575,
+            -0.009230786934494972,
+            -0.0018791928887367249,
+            0.014775004237890244,
+            0.012926931492984295,
+            -0.015196327120065689,
+            0.025260182097554207,
+            -0.023440835997462273,
+            0.01603897102177143,
+            0.011902352795004845,
+            0.014822881668806076,
+            0.015426138415932655,
+            0.022061964496970177,
+            -0.008862130343914032,
+            -0.004617787431925535,
+            0.015090996399521828,
+            0.004258705768734217,
+            -0.006161837372928858,
+            -0.015081420540809631,
+            0.016565624624490738,
+            -0.007842338643968105,
+            -0.004950535949319601,
+            -0.02765405923128128,
+            -0.005285678897053003,
+            -0.010839471593499184,
+            -0.007291747257113457,
+            0.015512318350374699,
+            -0.005146834068000317,
+            0.0074401674792170525,
+            -0.0003764370339922607,
+            0.021736396476626396,
+            -0.00026108213933184743,
+            0.0004120459489058703,
+            0.016489019617438316,
+            0.050520364195108414,
+            -0.02079799771308899,
+            -0.01209386344999075,
+            -0.0022514406591653824,
+            -0.0008977036341093481,
+            -0.010169186629354954,
+            -0.004342491738498211,
+            -0.005199499428272247,
+            -0.006501767784357071,
+            0.01270669512450695,
+            0.02060648798942566,
+            -0.025394240394234657,
+            -0.010992679744958878,
+            -0.023191872984170914,
+            -0.0377083383500576,
+            0.010178761556744576,
+            -0.01490906160324812,
+            -0.04730299487709999,
+            0.03937447816133499,
+            -0.014152596704661846,
+            0.010734140872955322,
+            0.029205290600657463,
+            -0.0032197635155171156,
+            -0.04722639173269272,
+            -0.0060421437956392765,
+            0.005414948333054781,
+            -0.00098208780400455,
+            -0.0031216146890074015,
+            0.0401788204908371,
+            -0.017829591408371925,
+            0.003597995964810252,
+            -0.0185860563069582,
+            0.01997450366616249,
+            -0.01689119078218937,
+            0.036137957125902176,
+            0.03598474711179733,
+            -0.04680506885051727,
+            -0.0033059432171285152,
+            -0.03292058780789375,
+            0.020912902429699898,
+            0.007784885820001364,
+            0.011059708893299103,
+            -0.014765429310500622,
+            0.020434128120541573,
+            -0.023536590859293938,
+            0.028726516291499138,
+            0.011835324577987194,
+            0.033054642379283905,
+            -0.023440835997462273,
+            -0.01746572181582451,
+            -0.02713698148727417,
+            -0.003203006461262703,
+            0.01937124691903591,
+            -0.02361319586634636,
+            0.010130884125828743,
+            0.019083982333540916,
+            0.01414302084594965,
+            0.026964621618390083,
+            0.020702242851257324,
+            -0.0015284899855032563,
+            -0.015090996399521828,
+            -0.00834984052926302,
+            0.04511978104710579,
+            0.007382714655250311,
+            0.0004383785999380052,
+            -0.005467613227665424,
+            0.011318246833980083,
+            0.026524148881435394,
+            0.030833126977086067,
+            -0.015646375715732574,
+            0.01625920832157135,
+            -0.008167905732989311,
+            -0.011011830531060696,
+            -0.028783967718482018,
+            -0.0021628672257065773,
+            -0.03186728060245514,
+            -0.03893400356173515,
+            0.007957245223224163,
+            -0.03521870821714401,
+            -0.002626082394272089,
+            0.015359110198915005,
+            0.021870452910661697,
+            0.02464734949171543,
+            -0.028592457994818687,
+            0.028879722580313683,
+            0.019332945346832275,
+            -0.008354628458619118,
+            -0.001125121838413179,
+            -0.027175283059477806,
+            -0.003918775357306004,
+            0.0018947530770674348,
+            -0.014918637461960316,
+            -0.021276772022247314,
+            -0.007176841143518686,
+            -0.012084287591278553,
+            -0.0061187478713691235,
+            0.009446235373616219,
+            -0.007813611999154091,
+            0.002906165784224868,
+            -0.025375088676810265,
+            0.03349511697888374,
+            0.003466332796961069,
+            0.007861489430069923,
+            -0.01577085629105568,
+            0.0006451496738009155,
+            0.0021137928124517202,
+            -0.003650661325082183,
+            -0.009269089438021183,
+            0.03259501978754997,
+            -0.010398998856544495,
+            0.0054532503709197044,
+            -0.013233347795903683,
+            0.0019222826231271029,
+            0.01863393373787403,
+            0.016824161633849144,
+            -0.017590202391147614,
+            0.0043329158797860146,
+            -0.005060654599219561,
+            0.004687209613621235,
+            -0.005194711498916149,
+            0.027998777106404305,
+            -0.011931079439818859,
+            -9.934586705639958e-05,
+            -0.0038948366418480873,
+            -0.009316966868937016,
+            0.024111121892929077,
+            0.000726541446056217,
+            0.014899485744535923,
+            0.023459987714886665,
+            0.023000363260507584,
+            -0.013242923654615879,
+            -0.007023632992058992,
+            0.022674795240163803,
+            0.024226028472185135,
+            0.01209386344999075,
+            -0.009541991166770458,
+            0.002733806613832712,
+            0.004045650828629732,
+            0.029664915055036545,
+            0.0029444678220897913,
+            0.010178761556744576,
+            -0.02669650875031948,
+            -0.02688801847398281,
+            0.013846180401742458,
+            -0.012400278821587563,
+            0.0049888379871845245,
+            0.012132165022194386,
+            0.013568490743637085,
+            0.005003201309591532,
+            -0.001164620858617127,
+            0.009302603080868721,
+            0.012065136805176735,
+            0.006147474516183138,
+            0.02355574257671833,
+            0.012821601703763008,
+            -0.015119723044335842,
+            -0.006875212769955397,
+            0.002081475453451276,
+            0.00019390397937968373,
+            -0.008857342414557934,
+            -0.03096718341112137,
+            -0.06235568970441818,
+            -0.02183215133845806,
+            -0.014526041224598885,
+            0.004560334142297506,
+            0.02503037080168724,
+            0.025317635387182236,
+            -0.0049888379871845245,
+            -0.04653695598244667,
+            -0.0034783021546900272,
+            0.006980543024837971,
+            0.014727126806974411,
+            -0.004538789391517639,
+            0.02451329305768013,
+            -0.023574894294142723,
+            0.005223438143730164,
+            0.0072007798589766026,
+            -0.023862158879637718,
+            0.03904891014099121,
+            0.02368979901075363,
+            -0.01174914464354515,
+            -0.01735081523656845,
+            -0.018432848155498505,
+            0.012180042453110218,
+            -0.0034495757427066565,
+            0.022636493667960167,
+            0.03297803923487663,
+            -0.03979580104351044,
+            0.043357886373996735,
+            -0.006937453523278236,
+            -0.002997133182361722,
+            -0.01056178193539381,
+            -0.0028606820851564407,
+            0.0014662492321804166,
+            0.0029708005022257566,
+            -0.004320946522057056,
+            0.01779128797352314,
+            -0.018796715885400772,
+            0.023402534425258636,
+            0.008287599310278893,
+            0.022368380799889565,
+            -0.004325734451413155,
+            0.020146863535046577,
+            -0.07281213998794556,
+            0.007004482205957174,
+            -0.004548364784568548,
+            -0.010896924883127213,
+            -0.04967772215604782,
+            -0.02721358649432659,
+            0.06036398559808731,
+            -0.03437606245279312,
+            0.02175554819405079,
+            0.02688801847398281,
+            0.0251452773809433,
+            -0.032346054911613464,
+            0.00023968686582520604,
+            -0.003961865324527025,
+            0.024628199636936188,
+            -0.00918290950357914,
+            0.034471817314624786,
+            0.021161865442991257,
+            0.015129297971725464,
+            0.015004816465079784,
+            -0.002413027221336961,
+            -0.01574213057756424,
+            0.08441765606403351,
+            -0.011461880058050156,
+            -0.041902411729097366,
+            0.03000963293015957,
+            -0.0404469333589077,
+            0.00390441226772964,
+            0.01295565813779831,
+            -0.0061187478713691235,
+            0.014995240606367588,
+            -0.0024142242036759853,
+            -0.009585080668330193,
+            -0.021085262298583984,
+            0.0020240223966538906,
+            -0.03301634266972542,
+            -0.02252158895134926,
+            -0.004790146369487047,
+            0.003511816496029496,
+            0.0008617954445071518,
+            -0.0029875575564801693,
+            -0.004189283587038517,
+            0.006338984239846468,
+            0.04347279295325279,
+            -0.023364232853055,
+            0.011030982248485088,
+            -0.005654335953295231,
+            -0.026600753888487816,
+            0.033839836716651917,
+            0.013558914884924889,
+            0.005046291276812553,
+            -0.01497608982026577,
+            0.008359416387975216,
+            0.002378316130489111,
+            -0.031388506293296814,
+            0.00649698032066226,
+            -0.029741518199443817,
+            -0.004256311804056168,
+            0.005496339872479439,
+            -0.00047727907076478004,
+            -0.0066118864342570305,
+            0.030545860528945923,
+            -0.01645071804523468,
+            -0.021046960726380348,
+            -0.009800529107451439,
+            -0.004708754830062389,
+            -0.009503688663244247,
+            -0.007339624688029289,
+            -0.00037075160071253777,
+            0.0007103827665559947,
+            0.03320785239338875,
+            0.02661990374326706,
+            0.011174614541232586,
+            -0.010772443376481533,
+            -0.0252793338149786,
+            0.003966652788221836,
+            -0.0022693946957588196,
+            0.02164064161479473,
+            0.025202728807926178,
+            -0.0262368842959404,
+            -0.008546138182282448,
+            -0.036003898829221725,
+            -0.010226639918982983,
+            -0.009144607000052929,
+            0.011423577554523945,
+            -0.011346973478794098,
+            0.038819096982479095,
+            -0.010446876287460327,
+            0.013367405161261559,
+            -0.026390092447400093,
+            0.008531775325536728,
+            0.013453585095703602,
+            -0.003674600040540099,
+            -0.04557940363883972,
+            0.01490906160324812,
+            0.01587618701159954,
+            -0.0016505776438862085,
+            0.008689770475029945,
+            -0.02407282032072544,
+            -0.01533038355410099,
+            0.06955646723508835,
+            0.01037027221173048,
+            -0.01620175503194332,
+            -0.013252499513328075,
+            -0.024666501209139824,
+            -0.011902352795004845,
+            -0.013769576326012611,
+            0.01735081523656845,
+            -0.010858622379601002,
+            0.004232373088598251,
+            -0.007669979706406593,
+            0.024091970175504684,
+            0.007459318730980158,
+            0.0012986778747290373,
+            0.02650499902665615,
+            -0.003511816496029496,
+            -0.02495376579463482,
+            0.03916381672024727,
+            -0.017235908657312393,
+            -0.011366124264895916,
+            -0.019304217770695686,
+            -0.0021149897947907448,
+            -0.02125762216746807,
+            0.03571663424372673,
+            -0.021525735035538673,
+            -0.0011867642169818282,
+            -0.011797022074460983,
+            -0.019668087363243103,
+            -0.007574224378913641,
+            0.009939374402165413,
+            0.01904567889869213,
+            0.010351121425628662,
+            -0.0029253168031573296,
+            0.00827802438288927,
+            -0.008498260751366615,
+            0.031714074313640594,
+            0.0035668755881488323,
+            0.015148448757827282,
+            0.020395826548337936,
+            0.013175895437598228,
+            0.0045723035000264645,
+            0.018461573868989944,
+            -0.010762867517769337,
+            0.006812972016632557,
+            -0.026466695591807365,
+            -0.014382408931851387,
+            0.023919612169265747,
+            -0.013290801085531712,
+            -0.014334531500935555,
+            0.0010844259522855282,
+            -0.0039068059995770454,
+            0.0022454559803009033,
+            -0.012333250604569912,
+            -0.0054628257639706135,
+            -0.017006097361445427,
+            0.009834043681621552,
+            0.009776590391993523,
+            0.01863393373787403,
+            0.004569910001009703,
+            0.029645763337612152,
+            -0.007751371245831251,
+            -0.007780097890645266,
+            -0.0073635634034872055,
+            -0.02298121154308319,
+            0.005237801466137171,
+            0.021870452910661697,
+            0.0006702853715978563,
+            -0.005596882663667202,
+            -0.023459987714886665,
+            0.0061378986574709415,
+            -0.011959806084632874,
+            0.021027809008955956,
+            -0.0007313292007893324,
+            0.018672235310077667,
+            0.0028032290283590555,
+            -0.03288228437304497,
+            -0.0033298819325864315,
+            -0.008421656675636768,
+            -0.00590329896658659,
+            -0.014889910817146301,
+            -0.004088740795850754,
+            -0.0015464440220966935,
+            -0.0031886431388556957,
+            0.037401922047138214,
+            -0.007128963712602854,
+            -0.009000974707305431,
+            -0.0014949757605791092,
+            0.015722978860139847,
+            0.020012805238366127,
+            0.0009467780473642051,
+            -0.008474322035908699,
+            0.010427725501358509,
+            -8.408490248257294e-05,
+            0.010762867517769337,
+            -0.0011221294989809394,
+            -0.008503048680722713,
+            0.001982129644602537,
+            -0.0003803270810749382,
+            -0.027979625388979912,
+            -0.005884147714823484,
+            0.0011592345545068383,
+            -0.013376981019973755,
+            0.038302019238471985,
+            0.005328768398612738,
+            -0.007004482205957174,
+            0.0015428533079102635,
+            0.018346667289733887,
+            0.01132782269269228,
+            0.010226639918982983,
+            -0.003126402385532856,
+            0.004490911960601807,
+            -0.04511978104710579,
+            -0.027749814093112946,
+            -0.025949619710445404,
+            -0.03156086429953575,
+            0.015445290133357048,
+            0.018317941576242447,
+            -0.006875212769955397,
+            0.013673821464180946,
+            0.012812025845050812,
+            -0.001474627759307623,
+            0.009695199318230152,
+            -0.012802449986338615,
+            -0.0017247878713533282,
+            -0.00949890073388815,
+            -0.011710843071341515,
+            0.018001949414610863,
+            0.011797022074460983,
+            0.024206876754760742,
+            0.03929787129163742,
+            -0.011950230225920677,
+            0.0026859291829168797,
+            0.0011843702523037791,
+            0.006631037220358849,
+            -0.005711788777261972,
+            -0.013989812694489956,
+            0.010580932721495628,
+            -0.000983883161097765,
+            0.014707976020872593,
+            -0.01209386344999075,
+            0.0019665693398565054,
+            -0.00619535194709897,
+            -0.013961086049675941,
+            0.011366124264895916,
+            -0.01805940270423889,
+            0.012371552176773548,
+            -0.03941277787089348,
+            -0.007315685972571373,
+            -0.016489019617438316,
+            0.051554519683122635,
+            0.007933305576443672,
+            -0.01782001554965973,
+            -0.0017511205514892936,
+            0.025624051690101624,
+            0.004100710153579712,
+            -0.009771803393959999,
+            0.008641893044114113,
+            -0.00548676447942853,
+            0.011050133034586906,
+            0.0009144607465714216,
+            0.0029995269142091274,
+            -0.026026222854852676,
+            -0.005855421535670757,
+            -0.0054628257639706135,
+            0.01989790052175522,
+            0.000962338293902576,
+            0.016900766640901566,
+            0.08870747685432434,
+            -0.0035070287995040417,
+            -0.009451023302972317,
+            -0.0068608494475483894,
+            -0.019763842225074768,
+            -0.004562728106975555,
+            -0.008900431916117668,
+            -0.005639972630888224,
+            0.0030426166485995054,
+            -0.007186416536569595,
+            0.0054724011570215225,
+            0.014229200780391693,
+            -0.010925651527941227,
+            -0.008699346333742142,
+            -0.007195991929620504,
+            -0.01430580485612154,
+            0.016852889209985733,
+            0.031656619161367416,
+            0.0116821164265275,
+            0.04902658611536026,
+            -0.02533678710460663,
+            -0.014861184172332287,
+            0.004873932339251041,
+            0.004251524340361357,
+            0.00989149697124958,
+            0.007464106194674969,
+            -0.012409854680299759,
+            0.02330677956342697,
+            0.007756159175187349,
+            -0.0037990817800164223,
+            -0.012687544338405132,
+            0.015588922426104546,
+            0.015464440919458866,
+            -0.006061294581741095,
+            0.031522564589977264,
+            0.0025542660150676966,
+            0.010667112655937672,
+            0.010398998856544495,
+            0.0049026585184037685,
+            0.01557934656739235,
+            0.014095143415033817,
+            -0.0006271955789998174,
+            0.002482449635863304,
+            -0.002757745562121272,
+            0.013319527730345726,
+            0.018289214000105858,
+            0.012984384782612324,
+            -0.005831482820212841,
+            0.02560489997267723,
+            0.026581602171063423,
+            -0.00262368842959404,
+            -0.031120391562581062,
+            0.03864673897624016,
+            -0.020261768251657486,
+            0.0006313848425634205,
+            -0.008890856057405472,
+            0.018384970724582672,
+            -0.009403145872056484,
+            -0.004950535949319601,
+            -0.009163758717477322,
+            -0.028822271153330803,
+            -0.003729659365490079,
+            -0.003921169321984053,
+            0.012668393552303314,
+            0.01869138516485691,
+            -0.010628811083734035,
+            0.02112356387078762,
+            -0.012649241834878922,
+            0.020510731264948845,
+            -0.008570076897740364,
+            -0.022004511207342148,
+            0.008713710121810436,
+            -0.001651774626225233,
+            0.0019426306243985891,
+            -0.015177175402641296,
+            0.021219318732619286,
+            0.04684337228536606,
+            0.004869144409894943,
+            -0.00040815590182319283,
+            -0.007990758866071701,
+            -0.0033969103824347258,
+            -0.010734140872955322,
+            0.021295923739671707,
+            0.0058793602511286736,
+            -0.032863132655620575,
+            0.009752651676535606,
+            0.015885762870311737,
+            0.0014782185899093747,
+            0.017810439690947533,
+            -0.0013908421387895942,
+            0.03156086429953575,
+            0.006587947718799114,
+            0.004258705768734217,
+            0.0036554490216076374,
+            -0.014928212389349937,
+            0.0025279333349317312,
+            -0.0027745026163756847,
+            -0.02727103792130947,
+            0.00483084237203002,
+            -0.0012160891201347113,
+            -0.020912902429699898,
+            0.02233007736504078,
+            -0.03841692581772804,
+            -0.019083982333540916,
+            -0.003083312651142478,
+            -0.018844593316316605,
+            -0.013252499513328075,
+            0.01937124691903591,
+            -0.013999388553202152,
+            0.03636777028441429,
+            -0.007899791933596134,
+            0.007962032221257687,
+            -0.0036865693982690573,
+            -0.016948644071817398,
+            0.038876552134752274,
+            0.0045579406432807446,
+            -0.011078859679400921,
+            -0.008694558404386044,
+            0.022425832226872444,
+            -0.02213856764137745,
+            0.017168881371617317,
+            0.005673486739397049,
+            0.029856424778699875,
+            -0.02233007736504078,
+            0.01068626344203949,
+            0.025777259841561317,
+            0.009613807313144207,
+            -0.006003841757774353,
+            -0.010226639918982983,
+            -0.031139543280005455,
+            0.031771525740623474,
+            0.04151938855648041,
+            0.004936173092573881,
+            -0.005544217303395271,
+            0.0022514406591653824,
+            0.013511037454009056,
+            0.014583494514226913,
+            0.010552207008004189,
+            0.007119387853890657,
+            0.0020240223966538906,
+            -0.032097093760967255,
+            0.018739264458417892,
+            0.002458510920405388,
+            0.015177175402641296,
+            0.009063215926289558,
+            -0.026792263612151146,
+            -0.0009743076516315341,
+            -0.0021437162067741156,
+            0.007617314346134663,
+            0.02003195695579052,
+            0.013444009236991405,
+            0.028017928823828697,
+            0.00994895026087761,
+            0.0016924705123528838,
+            -0.012563062831759453,
+            -0.03315039724111557,
+            0.010178761556744576,
+            0.016737982630729675,
+            0.004593848716467619,
+            -0.01686246506869793,
+            0.010466027073562145,
+            0.007282171864062548,
+            -0.01145230419933796,
+            -0.014995240606367588,
+            -0.0064634657464921474,
+            -0.0038852612487971783,
+            -0.0302011426538229,
+            0.006094809155911207,
+            0.0020324010401964188,
+            -0.007081086281687021,
+            0.011337397620081902,
+            0.015598498284816742,
+            0.013137592934072018,
+            -0.022885456681251526,
+            0.013913208618760109,
+            -0.010762867517769337,
+            -0.015004816465079784,
+            -0.008986611850559711,
+            0.006970967631787062,
+            -0.0020946417935192585,
+            -0.006980543024837971,
+            0.01155763491988182,
+            0.0011813779128715396,
+            -0.030354350805282593,
+            0.032997190952301025,
+            -0.0037033266853541136,
+            -0.03060331381857395,
+            -0.024685652926564217,
+            0.008584440685808659,
+            0.02771151252090931,
+            0.0022203202825039625,
+            -0.004696785472333431,
+            0.03182898089289665,
+            -0.010580932721495628,
+            -0.0029899515211582184,
+            -0.03380153328180313,
+            -0.0034280307590961456,
+            -0.012342826463282108,
+            -0.020893752574920654,
+            -0.012294949032366276,
+            0.00763167766854167,
+            -0.007464106194674969,
+            -0.011634238995611668,
+            0.01805940270423889,
+            -0.01686246506869793,
+            0.002427390543743968,
+            0.003870897926390171,
+            -0.0024441475979983807,
+            0.021161865442991257,
+            -0.017427420243620872,
+            0.0018133613048121333,
+            -0.0035692695528268814,
+            -0.02188960462808609,
+            0.006755518727004528,
+            -0.001415977836586535,
+            -0.03192473575472832,
+            -0.01861478202044964,
+            -0.020146863535046577,
+            -0.0251452773809433,
+            0.006592735182493925,
+            0.009202060289680958,
+            0.015339959412813187,
+            0.005544217303395271,
+            0.01786789298057556,
+            -0.005458037834614515,
+            -0.005525066517293453,
+            -0.027883870527148247,
+            0.001639805268496275,
+            0.009570717811584473,
+            0.014219624921679497,
+            -0.024436689913272858,
+            -0.010542631149291992,
+            0.010868198238313198,
+            0.02162148989737034,
+            -0.004100710153579712,
+            0.014899485744535923,
+            0.014420710504055023,
+            -0.0241877268999815,
+            -4.5165711526351515e-06,
+            -0.013645094819366932,
+            -0.0254133902490139,
+            -0.002503994619473815,
+            -0.0002746972895693034,
+            -0.004031287506222725,
+            -0.015311232767999172,
+            0.015004816465079784,
+            0.010523480363190174,
+            0.01631666161119938,
+            -0.02464734949171543,
+            -0.004459791351109743,
+            -0.004115073475986719,
+            -0.013999388553202152,
+            -0.021678943186998367,
+            0.012543912045657635,
+            -0.026351790875196457,
+            0.014114294201135635,
+            -0.001409993157722056,
+            0.020319221541285515,
+            0.01978299394249916,
+            0.017561476677656174,
+            -0.0019031316041946411,
+            -0.0026955048087984324,
+            0.009024913422763348,
+            0.0071146003901958466,
+            -0.016192179173231125,
+            0.015406987629830837,
+            -0.008503048680722713,
+            -0.0061283232644200325,
+            0.012266222387552261,
+            -0.008823827840387821,
+            0.002645233180373907,
+            0.014047265984117985,
+            -0.0059655397199094296,
+            0.0009988448582589626,
+            -0.008388142101466656,
+            -0.004206040408462286,
+            -0.009479749947786331,
+            0.007468894124031067,
+            -0.03330360725522041,
+            0.013128018006682396,
+            -0.0036722063086926937,
+            0.0033418512903153896,
+            -0.000942588783800602,
+            -0.007559861522167921,
+            -0.02093205414712429,
+            -0.008273236453533173,
+            -0.014324955642223358,
+            -0.002482449635863304,
+            -0.012065136805176735,
+            0.01939997263252735,
+            0.009139820002019405,
+            -0.005496339872479439,
+            0.009364844299852848,
+            0.006918302271515131,
+            -0.019036104902625084,
+            -0.0004512456653174013,
+            0.02317272312939167,
+            0.018183885142207146,
+            -0.00739229004830122,
+            0.03326530382037163,
+            -0.012026834301650524,
+            0.015081420540809631,
+            0.014286654070019722,
+            0.01830836571753025,
+            -0.006353347562253475,
+            -0.030507558956742287,
+            0.017896618694067,
+            0.020070258527994156,
+            -0.020759694278240204,
+            -0.013242923654615879,
+            0.006578371860086918,
+            0.042974866926670074,
+            -0.007085873745381832,
+            0.016086848452687263,
+            0.0012603758368641138,
+            -0.006827335339039564,
+            -0.011145887896418571,
+            -0.011193765327334404,
+            0.06863722205162048,
+            0.023364232853055,
+            0.013424858450889587,
+            -0.007124175783246756,
+            0.0022011692635715008,
+            -0.02112356387078762,
+            -0.001406402327120304,
+            0.0026428394485265017,
+            0.002515963977202773,
+            -0.008752011694014072,
+            -0.025011219084262848,
+            0.01916058547794819,
+            0.030450105667114258,
+            0.009163758717477322,
+            -0.011270369403064251,
+            -0.013874907046556473,
+            0.02303866483271122,
+            0.002209547907114029,
+            0.017341239377856255,
+            0.011797022074460983,
+            0.001982129644602537,
+            -0.03765088692307472,
+            0.003921169321984053,
+            0.015033543109893799,
+            0.0038852612487971783,
+            0.0036434796638786793,
+            0.0078567024320364,
+            0.012400278821587563,
+            0.011212916113436222,
+            0.0034280307590961456,
+            0.027979625388979912,
+            -0.026677357032895088,
+            -0.0019486153032630682,
+            0.0007307307096198201,
+            -0.0005475992220453918,
+            0.009015338495373726,
+            0.013721698895096779,
+            -0.015196327120065689,
+            -0.011174614541232586,
+            0.0011562422150745988,
+            0.0065592210739851,
+            -0.02284715510904789,
+            -0.01091607566922903,
+            0.01414302084594965,
+            -0.01984044723212719,
+            0.011816173791885376,
+            -0.0015763675328344107,
+            0.0006122338818386197,
+            -0.02323017455637455,
+            -0.0008929158793762326,
+            0.006683702580630779,
+            0.010935226455330849,
+            -0.007397077511996031,
+            0.022368380799889565,
+            -0.024111121892929077,
+            -0.01942870020866394,
+            0.017638079822063446,
+            -0.009747864678502083,
+            0.013156743720173836,
+            0.014669673517346382,
+            -0.013989812694489956,
+            -0.003739234758540988,
+            0.0007684343145228922,
+            -0.004940960556268692,
+            -0.010466027073562145,
+            0.00400495482608676,
+            0.0021664579398930073,
+            -0.005735727492719889,
+            0.002329241717234254,
+            0.017293361946940422,
+            -0.011548059061169624,
+            -0.01628793403506279,
+            -0.00590329896658659,
+            0.012400278821587563,
+            -0.000718761351890862,
+            0.018049826845526695,
+            0.01587618701159954,
+            -0.005917662288993597,
+            0.009834043681621552,
+            -0.022808853536844254,
+            0.01078201923519373,
+            0.008167905732989311,
+            -0.008498260751366615,
+            0.006769882049411535,
+            -0.0022670007310807705,
+            -0.004732693545520306,
+            0.03311209753155708,
+            0.020070258527994156,
+            0.007928518578410149,
+            -0.01283117663115263,
+            -0.005888935644179583,
+            -0.026792263612151146,
+            0.0003776339872274548,
+            -0.0028056229930371046,
+            -0.011672540567815304,
+            -0.026524148881435394,
+            0.011528908275067806,
+            0.002091050846502185,
+            -0.004423883277922869,
+            -0.020702242851257324,
+            0.013740849681198597,
+            0.015627223998308182,
+            0.0005272512789815664,
+            0.002843925030902028,
+            0.015167600475251675,
+            0.010657536797225475,
+            0.018298789858818054,
+            -0.010504329577088356,
+            -0.0056447600945830345,
+            0.001198135083541274,
+            -0.026524148881435394,
+            0.0036722063086926937,
+            -0.006085233297199011,
+            0.003061767667531967,
+            0.01786789298057556,
+            -0.011481030844151974,
+            -0.01989790052175522,
+            0.003468726761639118,
+            0.0032987615559250116,
+            -0.01935209520161152,
+            0.014880334958434105,
+            -0.023249326273798943,
+            -0.0004925400717183948,
+            -0.030430955812335014,
+            -0.0060325684025883675,
+            -0.002427390543743968,
+            0.0017834377940744162,
+            -0.0049792625941336155,
+            0.0019569939468055964,
+            -0.015435714274644852,
+            0.010829896666109562,
+            0.03188643231987953,
+            0.010284092277288437,
+            -0.0012071121018379927,
+            0.001107766292989254,
+            -0.0043161590583622456,
+            -0.006841698195785284,
+            -0.01139485090970993,
+            -0.009786166250705719,
+            0.01299396064132452,
+            -0.0058793602511286736,
+            0.02495376579463482,
+            -0.009283452294766903,
+            0.0024513292592018843,
+            0.006846486125141382,
+            0.008708922192454338,
+            -0.011059708893299103,
+            -0.018298789858818054,
+            0.00947017502039671,
+            -0.011864051222801208,
+            0.00815833080559969,
+            -0.008929158560931683,
+            -0.011021406389772892,
+            0.0016050940612331033,
+            0.0032245514448732138,
+            -0.012649241834878922,
+            -0.014707976020872593,
+            -0.005333556327968836,
+            -0.0006906333146616817,
+            -0.005802756175398827,
+            0.00834984052926302,
+            -0.008163117803633213,
+            -0.008373779244720936,
+            -0.005391009617596865,
+            -0.0011041754623875022,
+            -0.0023172723595052958,
+            0.0032413084991276264,
+            0.007765734568238258,
+            0.020051108673214912,
+            0.00900576263666153,
+            0.012505609542131424,
+            0.023268477991223335,
+            0.01513887383043766,
+            0.010523480363190174,
+            -0.008301963098347187,
+            0.01948615349829197,
+            -0.006243229378014803,
+            0.004411913920193911,
+            0.022828003391623497,
+            0.010829896666109562,
+            -0.00691351480782032,
+            0.02757745422422886,
+            -0.010150035843253136,
+            -0.024915464222431183,
+            -0.01917973719537258,
+            0.010858622379601002,
+            -0.02413027361035347,
+            0.003813444869592786,
+            -0.004356855060905218,
+            -0.022253474220633507,
+            0.01174914464354515,
+            0.01596236787736416,
+            0.021085262298583984,
+            0.001044927048496902,
+            -0.004809297621250153,
+            0.0007546694832853973,
+            0.016115576028823853,
+            0.01276414841413498,
+            -0.005836270283907652,
+            0.0012879054993391037,
+            -0.003830202156677842,
+            0.011433153413236141,
+            0.010734140872955322,
+            -0.0018672235310077667,
+            0.007306110579520464,
+            -0.001930661266669631,
+            0.0016565624391660094,
+            0.0016936674946919084,
+            0.011825748719274998,
+            -0.005946388468146324,
+            -0.001229255460202694,
+            0.0012376339873299003,
+            0.011978956870734692,
+            0.0015727767022326589,
+            -0.00798118393868208,
+            0.015895338729023933,
+            0.011337397620081902,
+            -0.014717550948262215,
+            0.01916058547794819,
+            -0.010063855908811092,
+            -0.003739234758540988,
+            0.0006499374285340309,
+            0.015636799857020378,
+            0.004601030144840479,
+            0.01759977824985981,
+            -0.0032077941577881575,
+            0.008139179088175297,
+            -0.015158024616539478,
+            -0.020434128120541573,
+            -0.04672846570611,
+            0.008617954328656197,
+            -0.0020431734155863523,
+            -0.006884788163006306,
+            0.0019498122856020927,
+            0.01901695318520069,
+            0.013300376944243908,
+            0.009417509660124779,
+            -0.0034591511357575655,
+            0.014746277593076229,
+            -0.018021101132035255,
+            -0.019218038767576218,
+            -0.015943216159939766,
+            0.00912545621395111,
+            0.0018001949647441506,
+            -0.002961224876344204,
+            -0.005630397237837315,
+            0.020568184554576874,
+            -0.006420376244932413,
+            0.011184190399944782,
+            -3.63569961336907e-05,
+            -0.018442422151565552,
+            -0.007382714655250311,
+            -0.004452609922736883,
+            0.00857965275645256,
+            0.0029468617867678404,
+            0.022387530654668808,
+            0.0116821164265275,
+            0.00011879610974574462,
+            0.010475602932274342,
+            -0.010150035843253136,
+            -0.0066884905099868774,
+            0.016594350337982178,
+            0.013578066602349281,
+            0.0002670668181963265,
+            0.012361977249383926,
+            0.014353682287037373,
+            0.004675240255892277,
+            0.0019426306243985891,
+            0.01590491458773613,
+            0.05787435173988342,
+            0.018902046605944633,
+            0.003605177626013756,
+            -0.008440807461738586,
+            0.008120028302073479,
+            0.004897871054708958,
+            0.013740849681198597,
+            -0.00372726540081203,
+            -0.0234982892870903,
+            0.006832122802734375,
+            0.010236214846372604,
+            -0.01948615349829197,
+            -0.009958525188267231,
+            0.006022992543876171,
+            0.00836420338600874,
+            -0.015857037156820297,
+            -0.0004030689306091517,
+            -0.011844899505376816,
+            -0.016086848452687263,
+            0.0004171329492237419,
+            0.013884482905268669,
+            -0.023440835997462273,
+            -0.0011951427441090345,
+            -0.0009521642932668328,
+            -0.008215783163905144,
+            -0.009556354023516178,
+            0.0005706402589567006,
+            -0.0021832152269780636,
+            -0.018480725586414337,
+            -0.0022059569600969553,
+            -0.00936005637049675,
+            -0.007444955408573151,
+            0.022636493667960167,
+            -0.0102457907050848,
+            -0.006109172478318214,
+            -0.02213856764137745,
+            -0.023957913741469383,
+            0.01784874126315117,
+            -0.016249632462859154,
+            0.003097675973549485,
+            -0.018155157566070557,
+            0.002120974473655224,
+            -0.019036104902625084,
+            0.01677628420293331,
+            0.010389422997832298,
+            -0.01942870020866394,
+            0.02131507359445095,
+            0.006511343643069267,
+            0.004809297621250153,
+            -0.012649241834878922,
+            -0.002219123300164938,
+            -0.012314099818468094,
+            0.0088142529129982,
+            0.009795742109417915,
+            0.010647961869835854,
+            0.008680195547640324,
+            0.008282812312245369,
+            0.006865637376904488,
+            0.005788392852991819,
+            -0.012697120197117329,
+            -0.0401788204908371,
+            0.01882544346153736,
+            0.016077272593975067,
+            -0.01628793403506279,
+            0.016278358176350594,
+            -0.013865331187844276,
+            -0.020012805238366127,
+            -0.01378872711211443,
+            -0.008689770475029945,
+            -0.014583494514226913,
+            0.011825748719274998,
+            0.01601981930434704,
+            -0.011366124264895916,
+            0.00036416843067854643,
+            0.006372498814016581,
+            0.016173027455806732,
+            -0.009546778164803982,
+            0.005979903042316437,
+            -0.0015272931195795536,
+            -0.01049475371837616,
+            0.01593364030122757,
+            -0.016986945644021034,
+            -0.006248017307370901,
+            0.004277857020497322,
+            -0.022579040378332138,
+            -0.018672235310077667,
+            -0.011461880058050156,
+            0.014947363175451756,
+            -0.0052282256074249744,
+            -0.02194705791771412,
+            -0.0046608769334852695,
+            -0.03539106622338295,
+            0.008459959179162979,
+            0.00566391134634614,
+            -0.0017786500975489616,
+            -0.015541044995188713,
+            0.036712486296892166,
+            0.004751844331622124,
+            -0.02047242969274521,
+            0.005027140025049448,
+            -0.02041497640311718,
+            0.013807878829538822,
+            0.009518052451312542,
+            -0.017149729654192924,
+            0.002458510920405388,
+            0.0007905776146799326,
+            -0.0023807098623365164,
+            -0.00625759270042181,
+            0.004285038448870182,
+            0.004735087510198355,
+            -0.01896907575428486,
+            -0.004124648869037628,
+            0.015464440919458866,
+            -0.0033442452549934387,
+            -0.010513904504477978,
+            -0.011835324577987194,
+            -0.0037751428317278624,
+            0.0025662353727966547,
+            -0.0037631734739989042,
+            -0.0003393319493625313,
+            -0.01866265945136547,
+            -0.01650817133486271,
+            0.034529272466897964,
+            0.0020754907745867968,
+            0.017590202391147614,
+            -0.00494574848562479,
+            -0.0015979124000295997,
+            -0.008694558404386044,
+            0.015339959412813187,
+            0.013874907046556473,
+            0.011471454985439777,
+            0.0029708005022257566,
+            0.009436660446226597,
+            0.01773383468389511,
+            0.00483802380040288,
+            0.01869138516485691,
+            -0.009168545715510845,
+            -0.00888128113001585,
+            -0.011385275982320309,
+            0.029090384021401405,
+            0.009522839449346066,
+            -0.007023632992058992,
+            -0.025394240394234657,
+            -0.0019905080553144217,
+            0.009460599161684513,
+            0.024168575182557106,
+            -0.018767990171909332,
+            -0.028898874297738075,
+            -0.013204621151089668,
+            0.01418132334947586,
+            0.01549316756427288,
+            -0.014085568487644196,
+            -0.0022394713014364243,
+            0.011241642758250237,
+            0.023823857307434082,
+            -0.027826417237520218,
+            0.00936005637049675,
+            -0.0033298819325864315,
+            0.0037320530973374844,
+            -0.01939997263252735,
+            -0.02476225607097149,
+            0.00017475297499913722,
+            0.0034040920436382294,
+            0.009111093357205391,
+            -0.015301656909286976,
+            0.0017211970407515764,
+            0.020759694278240204,
+            -0.009628170169889927,
+            0.015406987629830837,
+            -0.0007564648985862732,
+            -0.010418149642646313,
+            0.0013369799125939608,
+            -0.005323980934917927,
+            -0.009814892895519733,
+            0.015560195781290531,
+            0.005414948333054781,
+            0.00428982637822628,
+            0.002503994619473815,
+            -0.00953720323741436,
+            -0.003291579894721508,
+            -0.008981823921203613,
+            -0.010284092277288437,
+            0.015110147185623646,
+            0.013769576326012611,
+            -0.020644789561629295,
+            0.024666501209139824,
+            -0.0029037718195468187,
+            -0.0049002645537257195,
+            -0.00507022999227047,
+            -0.008459959179162979,
+            0.0170827005058527,
+            0.0006918302387930453,
+            -0.00016442938067484647,
+            0.015282506123185158,
+            0.0038996245712041855,
+            0.001734363380819559,
+            -0.017178455367684364,
+            -0.007047571707516909,
+            -0.00023445025726687163,
+            0.005917662288993597,
+            0.0065257069654762745,
+            0.016661379486322403,
+            -0.006310258060693741,
+            -0.01014045998454094,
+            0.007023632992058992,
+            0.00955156609416008,
+            0.015924064442515373,
+            0.013022687286138535,
+            0.014449437148869038,
+            0.018767990171909332,
+            0.019189313054084778,
+            -0.008412080816924572,
+            -0.01727421209216118,
+            0.017235908657312393,
+            -0.008139179088175297,
+            -0.003289185930043459,
+            -0.04117467254400253,
+            0.01705397479236126,
+            0.020817147567868233,
+            -0.02137252688407898,
+            0.001020988216623664,
+            -0.004663270898163319,
+            -0.007784885820001364,
+            -0.021602340042591095,
+            0.018155157566070557,
+            -0.004852387122809887,
+            0.0019210857572034001,
+            0.004888295195996761,
+            -0.01820303499698639,
+            0.005855421535670757,
+            0.023574894294142723,
+            -0.009000974707305431,
+            0.004907446447759867,
+            -0.0021772305481135845,
+            -0.003554906230419874,
+            -0.012917356565594673,
+            0.006597523111850023,
+            -0.02457074634730816,
+            0.023670649155974388,
+            0.018193459138274193,
+            0.011002255603671074,
+            -0.007952457293868065,
+            0.007473681587725878,
+            -0.015349534340202808,
+            -0.0027266251854598522,
+            -0.00199888669885695,
+            0.014104719273746014,
+            0.002975588198751211,
+            0.00817269366234541,
+            0.0058697848580777645,
+            0.002906165784224868,
+            0.0032748228404670954,
+            -0.018241336569190025,
+            0.016144301742315292,
+            -0.006147474516183138,
+            0.007947669364511967,
+            0.013635518960654736,
+            0.0026931108441203833,
+            -0.009599443525075912,
+            -0.010312818922102451,
+            -0.004821266978979111,
+            0.006592735182493925,
+            0.004931385163217783,
+            0.011911928653717041,
+            0.009977675974369049,
+            0.009824467822909355,
+            -0.02503037080168724,
+            0.012879054062068462,
+            0.0032556718215346336,
+            -0.03282483294606209,
+            -0.002697898540645838,
+            0.0016972583252936602,
+            -0.0030761309899389744,
+            0.010255365632474422,
+            0.02317272312939167,
+            -0.017504023388028145,
+            -0.009206848219037056,
+            0.01801152527332306,
+            0.005199499428272247,
+            -0.016077272593975067,
+            -0.016852889209985733,
+            0.007377926725894213,
+            0.009901071898639202,
+            0.011098010465502739,
+            -0.014985665678977966,
+            0.009077578783035278,
+            0.007052359636873007,
+            0.0254133902490139,
+            -0.03085227683186531,
+            0.018902046605944633,
+            0.012266222387552261,
+            0.0169390682131052,
+            -0.00888128113001585,
+            -0.010896924883127213,
+            0.0008031455217860639,
+            0.012888629920780659,
+            -0.0017571052303537726,
+            0.015598498284816742,
+            -0.002170048886910081,
+            0.005256952252238989,
+            -0.02368979901075363,
+            -0.005041503347456455,
+            -0.0026404454838484526,
+            0.023057816550135612,
+            0.01485160831362009,
+            0.02661990374326706,
+            0.006233653984963894,
+            -0.012007683515548706,
+            0.010743716731667519,
+            -0.006429951637983322,
+            0.0003551913832779974,
+            -0.021219318732619286,
+            -0.00039409188320860267,
+            -0.019610634073615074,
+            -0.003442394081503153,
+            -0.016900766640901566,
+            0.03609965369105339,
+            0.02047242969274521,
+            0.014123870059847832,
+            0.025509145110845566,
+            -9.672756277723238e-05,
+            0.008670619688928127,
+            -0.004225191660225391,
+            -0.0018923592288047075,
+            -0.015253779478371143,
+            0.00774658378213644,
+            -0.006789033301174641,
+            0.001808573491871357,
+            0.008622742258012295,
+            -0.004129436798393726,
+            0.016632651910185814,
+            -0.022483285516500473,
+            0.014411135576665401,
+            0.01378872711211443,
+            0.014315379783511162,
+            0.016603926196694374,
+            -0.010284092277288437,
+            -0.0031862494070082903,
+            -0.0028223800472915173,
+            0.001198135083541274,
+            -0.009039277210831642,
+            -0.0023711344692856073,
+            -0.0028918024618178606,
+            -0.0088142529129982,
+            -0.005951176397502422,
+            0.022387530654668808,
+            0.0024968129582703114,
+            0.018298789858818054,
+            0.0024477385450154543,
+            0.015885762870311737,
+            0.005673486739397049,
+            0.005783604923635721,
+            -0.0157134048640728,
+            -0.0005550800706259906,
+            0.013482310809195042,
+            0.0021999722812324762,
+            -0.0031790677458047867,
+            -0.021813001483678818,
+            -0.021659793332219124,
+            0.0007851914269849658,
+            0.00673636794090271,
+            0.02162148989737034,
+            0.0017666807398200035,
+            -0.015272930264472961,
+            0.0008462352561764419,
+            -0.021295923739671707,
+            -0.0243217833340168,
+            -0.014564343728125095,
+            0.0011191371595486999,
+            0.029856424778699875,
+            0.0016350174555554986,
+            -0.009010550566017628,
+            0.009407933801412582,
+            -0.021985359489917755,
+            -0.0023615588434040546,
+            0.002669172128662467,
+            -0.0204915814101696,
+            0.009202060289680958,
+            -0.031522564589977264,
+            0.015474015846848488,
+            -0.013463160023093224,
+            0.00035459292121231556,
+            0.022310927510261536,
+            -0.005165984854102135,
+            -0.0008492275956086814,
+            -0.011385275982320309,
+            0.00044556023203767836,
+            0.0159910935908556,
+            -0.004129436798393726,
+            -0.004514850676059723,
+            -0.0014111900236457586,
+            0.003509422531351447,
+            -0.0016745164757594466,
+            -0.005941601004451513,
+            0.0019881143234670162,
+            0.0011125539895147085,
+            -0.019581908360123634,
+            0.0006631037103943527,
+            0.018404120579361916,
+            -0.012974809855222702,
+            -0.03246096149086952,
+            0.01653689704835415,
+            -0.02405366860330105,
+            -0.010801170021295547,
+            -0.0035261798184365034,
+            0.017006097361445427,
+            0.0022658039815723896,
+            -0.007492832839488983,
+            0.020721392706036568,
+            0.007933305576443672,
+            0.01008300669491291,
+            0.004426277242600918,
+            0.003159916726872325,
+            -0.0002832254976965487,
+            -0.0026428394485265017,
+            -5.97533971813391e-06,
+            0.0064634657464921474,
+            0.015971941873431206,
+            -0.018375394865870476,
+            0.027883870527148247,
+            -0.0070284209214150906,
+            -0.0034280307590961456,
+            0.0043233404867351055,
+            -0.006808184087276459,
+            -0.008368991315364838,
+            0.010877774097025394,
+            -0.02721358649432659,
+            0.017944496124982834,
+            -0.015090996399521828,
+            -0.011921503581106663,
+            -0.011883202008903027,
+            0.014229200780391693,
+            0.006664551794528961,
+            -0.01951487921178341,
+            0.005740515422075987,
+            -0.01490906160324812,
+            -0.01596236787736416,
+            0.002160473261028528,
+            -0.004100710153579712,
+            0.014928212389349937,
+            -0.0116821164265275,
+            -0.01164381392300129,
+            0.00536228297278285,
+            -0.019859597086906433,
+            0.008321113884449005,
+            -0.0077705224975943565,
+            0.009403145872056484,
+            -0.025892166420817375,
+            0.011576785705983639,
+            0.015110147185623646,
+            -0.004735087510198355,
+            -0.017197607085108757,
+            0.015857037156820297,
+            0.016469869762659073,
+            -0.004447821993380785,
+            -0.011481030844151974,
+            0.00822535902261734,
+            0.013147168792784214,
+            0.0025518720503896475,
+            -0.004984050523489714,
+            -0.017130577936768532,
+            -0.01126079447567463,
+            0.00923557486385107,
+            -0.0016074879094958305,
+            -0.036655034869909286,
+            -0.01672840677201748,
+            -0.02131507359445095,
+            -0.011433153413236141,
+            -0.009474962018430233,
+            0.006003841757774353,
+            0.006607098504900932,
+            -0.002515963977202773,
+            -0.02738594450056553,
+            -0.00035638833651319146,
+            0.004857175052165985,
+            -0.014813306741416454,
+            -0.001930661266669631,
+            -0.003621934913098812,
+            -0.0036985387559980154,
+            -0.0048164790496230125,
+            -0.005060654599219561,
+            0.013769576326012611,
+            -0.003739234758540988,
+            0.01239070389419794,
+            0.011011830531060696,
+            -0.00787106528878212,
+            -0.0026619904674589634,
+            -0.03211624547839165,
+            -0.014449437148869038,
+            0.008182269521057606,
+            0.0068800002336502075,
+            -0.002951649483293295,
+            -0.0005161795997992158,
+            0.008321113884449005,
+            0.00029130480834282935,
+            -0.005084593314677477,
+            0.005237801466137171,
+            0.013195046223700047,
+            -0.013501462526619434,
+            0.024858010932803154,
+            0.010456451214849949,
+            -0.00863710604608059,
+            -0.012534336186945438,
+            -0.01078201923519373,
+            0.018509451299905777,
+            0.009068002924323082,
+            0.017331663519144058,
+            -0.0023519834503531456,
+            0.01247688289731741,
+            0.0014231594977900386,
+            0.0262368842959404,
+            -0.006482616998255253,
+            -0.006808184087276459,
+            -0.010839471593499184,
+            0.0021568825468420982,
+            -0.02778811566531658,
+            0.00447415467351675,
+            0.01478458009660244,
+            -0.009987251833081245,
+            -0.002028810093179345,
+            0.006406012922525406,
+            -0.006468253675848246,
+            0.011548059061169624,
+            -0.009977675974369049,
+            0.0024800559040158987,
+            -0.017551900818943977,
+            -0.0020096590742468834,
+            0.016546472907066345,
+            0.0010138065554201603,
+            0.006640612613409758,
+            0.04749450460076332,
+            -0.009723925031721592,
+            -0.009393570944666862,
+            0.026102827861905098,
+            -1.738365608616732e-05,
+            0.019993655383586884,
+            -0.0006035560509189963,
+            0.0010419347090646625,
+            0.004675240255892277,
+            0.013645094819366932,
+            0.018538178876042366,
+            0.005304829683154821,
+            0.00196776632219553,
+            -0.01084904745221138,
+            -0.003356214612722397,
+            -0.011605512350797653,
+            -0.033514268696308136,
+            -0.005419735796749592,
+            0.003049798309803009,
+            -0.0069422414526343346,
+            0.007579012308269739,
+            -0.01847114972770214,
+            0.007138539105653763,
+            0.02483886107802391,
+            -0.017006097361445427,
+            -0.015158024616539478,
+            -0.008646680973470211,
+            -0.006487404927611351,
+            0.0004410716937854886,
+            -0.01696779578924179,
+            0.001875602058134973,
+            -0.007076298352330923,
+            0.0036817817017436028,
+            0.005103744100779295,
+            -0.0135972173884511,
+            0.022004511207342148,
+            0.0030210718978196383,
+            0.020721392706036568,
+            -0.006645400542765856,
+            -0.0001726583286654204,
+            0.005932025611400604,
+            -0.010935226455330849,
+            -0.021468281745910645,
+            0.004325734451413155,
+            0.01049475371837616,
+            -0.02162148989737034,
+            -0.022253474220633507,
+            -0.0029109534807503223,
+            0.0015655950410291553,
+            -0.01030324399471283,
+            0.006990118883550167,
+            -0.0016421991167590022,
+            0.0126396669074893,
+            -0.00013046625826973468,
+            -0.0069422414526343346,
+            0.016527321189641953,
+            0.02175554819405079,
+            -0.004735087510198355,
+            -0.0029444678220897913,
+            -0.0004012735153082758,
+            -0.015550619922578335,
+            0.011662965640425682,
+            0.008866917341947556,
+            -0.018681811168789864,
+            0.015272930264472961,
+            0.017915770411491394,
+            0.016785860061645508,
+            0.0015955185517668724,
+            -0.014516465365886688,
+            -0.0005520877311937511,
+            -0.0038948366418480873,
+            -0.00017789493722375482,
+            -0.01549316756427288,
+            -0.005098956637084484,
+            -0.004912233911454678,
+            -0.014803730882704258,
+            0.0020802784711122513,
+            -0.026581602171063423,
+            -0.00043299238313920796,
+            -0.011940655298531055,
+            -0.0003560890909284353,
+            0.020261768251657486,
+            -0.019218038767576218,
+            -0.0006852470687590539,
+            -0.011308671906590462,
+            -0.018356243148446083,
+            0.013070564717054367,
+            -0.005089380778372288,
+            -0.021927906200289726,
+            0.0066118864342570305,
+            0.007344412617385387,
+            -0.008910007774829865,
+            -0.014640947803854942,
+            0.007985970936715603,
+            -0.012553486973047256,
+            0.0009617398027330637,
+            0.020376674830913544,
+            0.016718832775950432,
+            0.009518052451312542,
+            -0.00857965275645256,
+            -0.004433458670973778,
+            0.011739569716155529,
+            0.01689119078218937,
+            0.004895477090030909,
+            0.02694547176361084,
+            -0.024743104353547096,
+            -0.029090384021401405,
+            0.04308977350592613,
+            0.007009269669651985,
+            -0.009546778164803982,
+            0.013415282592177391,
+            -0.024628199636936188,
+            0.009599443525075912,
+            0.033322758972644806,
+            0.008924370631575584,
+            -0.017312513664364815,
+            0.014832457527518272,
+            0.002272985642775893,
+            -0.021525735035538673,
+            0.0035237858537584543,
+            0.005410160403698683,
+            -0.015416563488543034,
+            0.005395797081291676,
+            0.016948644071817398,
+            0.006640612613409758,
+            0.01286947913467884,
+            -0.00235318043269217,
+            0.008067362941801548,
+            0.010255365632474422,
+            0.004277857020497322,
+            -0.007947669364511967,
+            -0.006951816845685244,
+            0.009316966868937016,
+            -0.039834100753068924,
+            -0.01485160831362009,
+            -0.013128018006682396,
+            -0.0052138627506792545,
+            -0.007847126573324203,
+            0.021659793332219124,
+            -0.013405706733465195,
+            -0.01874883845448494,
+            0.02164064161479473,
+            0.024398386478424072,
+            -0.006980543024837971,
+            -0.004665664862841368,
+            0.0037200837396085262,
+            -0.016517747193574905,
+            -0.0020060683600604534,
+            0.005491552408784628,
+            0.035046350210905075,
+            -0.0072773839347064495,
+            -0.0060325684025883675,
+            0.008699346333742142,
+            -0.014545192010700703,
+            0.02956916019320488,
+            0.005285678897053003,
+            -0.006329408846795559,
+            -0.00924993772059679,
+            -0.022119417786598206,
+            -0.0030019208788871765,
+            -0.017130577936768532,
+            -0.02183215133845806,
+            0.0038804735522717237,
+            -0.0010461239144206047,
+            -0.0008893250487744808,
+            0.004881113767623901,
+            0.0022394713014364243,
+            0.018241336569190025,
+            -0.009800529107451439,
+            0.011117161251604557,
+            -0.018432848155498505,
+            -0.0024501322768628597,
+            0.006812972016632557,
+            -0.0017630899092182517,
+            -0.010772443376481533,
+            -0.021027809008955956,
+            0.023364232853055,
+            -0.006894363556057215,
+            0.012496034614741802,
+            -0.01920846290886402,
+            0.013386555947363377,
+            -0.0166709553450346,
+            -0.002225107979029417,
+            0.0032293391413986683,
+            -0.005271315574645996,
+            -0.004964899271726608,
+            -0.003289185930043459,
+            0.01939997263252735,
+            -0.011567209847271442,
+            -0.012084287591278553,
+            0.0251452773809433,
+            0.0241877268999815,
+            0.004268281627446413,
+            -0.01257263869047165,
+            0.001366903306916356,
+            0.007263020612299442,
+            0.0028941964264959097,
+            0.0016015032306313515,
+            0.002992345253005624,
+            0.002399860881268978,
+            0.002276576356962323,
+            0.006712429225444794,
+            0.01978299394249916,
+            -0.016766710206866264,
+            0.0060517191886901855,
+            0.023440835997462273,
+            0.01874883845448494,
+            -0.006908726878464222,
+            -0.002071899827569723,
+            -0.012266222387552261,
+            0.005970327649265528,
+            0.012984384782612324,
+            0.0052138627506792545,
+            0.0015452471561729908,
+            0.0019450244726613164,
+            0.007162477821111679,
+            -0.024302631616592407,
+            0.030181992799043655,
+            0.0028271679766476154,
+            0.014516465365886688,
+            6.553236016770825e-05,
+            0.015177175402641296,
+            -0.013530189171433449,
+            -0.002506388584151864,
+            0.013798302970826626,
+            -0.006248017307370901,
+            0.008790313266217709,
+            0.007574224378913641,
+            -0.005898511037230492,
+            -0.012582213617861271,
+            -0.00044137093937024474,
+            -0.0036674183793365955,
+            -0.00715290242806077,
+            -0.0026165067683905363,
+            -0.002841531066223979,
+            0.0013214197242632508,
+            -0.008316325955092907,
+            0.006410800851881504,
+            -0.007344412617385387,
+            0.007306110579520464,
+            -0.002270591678097844,
+            -0.003770355135202408,
+            0.0008683786145411432,
+            0.025968769565224648,
+            0.00806257501244545,
+            0.03135020285844803,
+            0.003291579894721508,
+            0.0016589562874287367,
+            -0.0008611970115453005,
+            -0.014535617083311081,
+            -0.0262368842959404,
+            0.011346973478794098,
+            -0.02771151252090931,
+            0.00031449549715034664,
+            -0.0032676411792635918,
+            0.003902018303051591,
+            0.012534336186945438,
+            0.011098010465502739,
+            0.009096729569137096,
+            -0.018930774182081223,
+            -0.013568490743637085,
+            -0.002460904885083437,
+            -0.011921503581106663,
+            0.002963618841022253,
+            0.012218344956636429,
+            -0.005118107423186302,
+            0.012792875058948994,
+            -0.0004982255049981177,
+            0.002997133182361722,
+            -0.03699975088238716,
+            -0.012256646528840065,
+            0.00241183047182858,
+            -0.00994895026087761,
+            0.007708281744271517,
+            0.009039277210831642,
+            -0.001272345194593072,
+            0.012208769097924232,
+            0.01601981930434704,
+            0.0058697848580777645,
+            -0.0035668755881488323,
+            -0.0023519834503531456,
+            -0.0011442728573456407,
+            -0.017963647842407227,
+            0.012610940262675285,
+            -0.017446570098400116,
+            0.00590329896658659,
+            0.004404732491821051,
+            0.016575198620557785,
+            0.017398692667484283,
+            -0.0053000422194600105,
+            0.020510731264948845,
+            0.0064634657464921474,
+            -0.0066214618273079395,
+            0.027175283059477806,
+            -0.00019285666348878294,
+            -0.006540069822221994,
+            -0.021334225311875343,
+            -0.003413667669519782,
+            0.019725540652871132,
+            -0.010236214846372604,
+            -0.0010856229346245527,
+            0.010734140872955322,
+            0.008479109965264797,
+            -2.2947548131924123e-05,
+            0.01765723153948784,
+            -0.009441448375582695,
+            0.0009192485013045371,
+            0.007315685972571373,
+            0.004950535949319601,
+            0.011232067830860615,
+            0.01174914464354515,
+            0.01740826852619648,
+            0.003947502002120018,
+            0.0023304384667426348,
+            -0.0037583857774734497,
+            0.004866750445216894,
+            0.011672540567815304,
+            0.03675078973174095,
+            0.007861489430069923,
+            0.007124175783246756,
+            -0.007176841143518686,
+            -0.004009742755442858,
+            0.01020748820155859,
+            -0.013415282592177391,
+            -0.013242923654615879,
+            0.015119723044335842,
+            0.005170772783458233,
+            0.032288603484630585,
+            -0.0121608916670084,
+            0.009996827691793442,
+            0.025585750117897987,
+            -0.008368991315364838,
+            -0.004658483434468508,
+            -0.004842811729758978,
+            0.0017319695325568318,
+            0.0024896312970668077,
+            0.007827975787222385,
+            0.0018037857953459024,
+            0.020587336272001266,
+            -0.009666472673416138,
+            -0.031771525740623474,
+            0.028075380250811577,
+            0.013491886667907238,
+            0.017398692667484283,
+            0.02164064161479473,
+            0.0079955467954278,
+            0.012677968479692936,
+            0.00971435010433197,
+            0.005893723573535681,
+            0.00416773883625865,
+            0.01609642431139946,
+            0.0073539880104362965,
+            -0.01767638325691223,
+            0.02054903469979763,
+            0.015483591705560684,
+            0.010753292590379715,
+            -0.02951170690357685,
+            -0.01116503868252039,
+            0.012936507351696491,
+            -0.01203641016036272,
+            -0.01061923522502184,
+            0.009929798543453217,
+            -0.011892776936292648,
+            0.005170772783458233,
+            0.015158024616539478,
+            0.010025554336607456,
+            0.0014123870059847832,
+            -0.02309611812233925,
+            -0.016489019617438316,
+            0.005496339872479439,
+            0.0021341408137232065,
+            0.012563062831759453,
+            0.0126396669074893,
+            0.017226334661245346,
+            -0.004950535949319601,
+            -0.00040037580765783787,
+            -0.017752986401319504,
+            -0.009580292738974094,
+            -0.007784885820001364,
+            0.02752000093460083,
+            -0.02164064161479473,
+            -0.0019330551149323583,
+            0.004105497617274523,
+            -0.02169809490442276,
+            0.0006218093913048506,
+            0.00840250588953495,
+            0.0025949617847800255,
+            -0.005357495043426752,
+            -0.008919582702219486,
+            0.003964259289205074,
+            -0.010753292590379715,
+            0.01414302084594965,
+            0.0011592345545068383,
+            -0.018729688599705696,
+            0.008368991315364838,
+            0.0013776757987216115,
+            -0.01212258916348219,
+            0.019629785791039467,
+            0.01852860301733017,
+            -0.007823187857866287,
+            0.005118107423186302,
+            0.041634295135736465,
+            0.0024992069229483604,
+            -0.022770551964640617,
+            -0.007545498199760914,
+            0.005323980934917927,
+            0.026102827861905098,
+            0.006066082511097193,
+            0.00822535902261734,
+            -0.0002223312621936202,
+            0.0030402229167521,
+            0.0017738622846081853,
+            -0.007406653370708227,
+            0.016211330890655518,
+            -0.00697575556114316,
+            0.004349673166871071,
+            0.014545192010700703,
+            -0.00877116248011589,
+            0.002016840735450387,
+            -0.01866265945136547,
+            0.0021233682055026293,
+            -0.025183578953146935,
+            -0.009546778164803982,
+            -0.003923563286662102,
+            -0.010264941491186619,
+            -0.008340264670550823,
+            0.0025303272996097803,
+            -0.011375700123608112,
+            -0.017398692667484283,
+            -0.010571357794106007,
+            0.01945742592215538,
+            -0.02066393941640854,
+            -0.0008234934648498893,
+            0.0065592210739851,
+            0.019189313054084778,
+            0.012151315808296204,
+            0.0029037718195468187,
+            0.0073539880104362965,
+            0.01561764907091856,
+            0.0017020460218191147,
+            -0.02188960462808609,
+            -0.027692360803484917,
+            0.02424517832696438,
+            0.001624245080165565,
+            0.0013681002892553806,
+            0.001211899914778769,
+            -0.011825748719274998,
+            -0.00985319446772337,
+            0.009048852138221264,
+            0.0006984134088270366,
+            -0.018183885142207146,
+            0.015474015846848488,
+            -0.002918135141953826,
+            -0.0017726654186844826,
+            -0.0031431596726179123,
+            0.014066416770219803,
+            0.014631371945142746,
+            0.0044095199555158615,
+            0.0055490052327513695,
+            0.005683062132447958,
+            0.012065136805176735,
+            0.010580932721495628,
+            -0.019878748804330826,
+            -0.025183578953146935,
+            0.019198887050151825,
+            -0.008459959179162979,
+            0.016852889209985733,
+            0.01382702961564064,
+            0.013635518960654736,
+            -0.028324345126748085,
+            -0.0037320530973374844,
+            -0.024436689913272858,
+            0.004550758749246597,
+            0.0008234934648498893,
+            -0.0019940990023314953,
+            -0.002559053711593151,
+            1.956245796463918e-05,
+            -0.021774698048830032,
+            0.006602310575544834,
+            0.025164427235722542,
+            -0.003890048945322633,
+            -0.011308671906590462,
+            0.02874566614627838,
+            -0.028918026015162468,
+            0.002788865938782692,
+            0.023076968267560005,
+            -0.004907446447759867,
+            0.07740838080644608,
+            0.008728072978556156,
+            0.0031670983880758286,
+            -0.015110147185623646,
+            0.0053000422194600105,
+            0.014487739652395248,
+            0.00834984052926302,
+            0.013817453756928444,
+            -0.00955156609416008,
+            -0.015158024616539478,
+            -0.008282812312245369,
+            0.01187362615019083,
+            0.006554433144629002,
+            -0.0031216146890074015,
+            0.002810410689562559,
+            0.0005098956753499806,
+            -0.010963953100144863,
+            -0.011739569716155529,
+            0.024111121892929077,
+            -0.025777259841561317,
+            -0.00871849711984396,
+            -0.028132833540439606,
+            -0.01366424560546875,
+            -0.00023789145052433014,
+            0.012754572555422783,
+            0.011011830531060696,
+            -0.022119417786598206,
+            -0.013606793247163296,
+            -0.020510731264948845,
+            -0.0020635214168578386,
+            0.018576480448246002,
+            0.0015536256833001971,
+            -0.000921043916605413,
+            -0.002472874242812395,
+            -0.001449492061510682,
+            0.009130244143307209,
+            -0.01295565813779831,
+            0.014344106428325176,
+            -0.006200139410793781,
+            0.0027649272233247757,
+            0.0038014755118638277,
+            0.00733004929497838,
+            0.01343443337827921,
+            -0.0034783021546900272,
+            0.006324620917439461,
+            -0.010753292590379715,
+            0.02361319586634636,
+            -0.01545486506074667,
+            -0.0031240086536854506,
+            -0.0015739736845716834,
+            0.008440807461738586,
+            -0.007325261365622282,
+            0.006760306656360626,
+            -0.00854135025292635,
+            -0.009269089438021183,
+            0.014468587934970856,
+            0.025566598400473595,
+            -0.02399621531367302,
+            0.004878719802945852,
+            -0.022770551964640617,
+            -0.0006918302387930453,
+            -0.007325261365622282,
+            -0.006425164174288511,
+            -0.002341211074963212,
+            -0.0401788204908371,
+            -0.005319193005561829,
+            0.012314099818468094,
+            0.0026284761261194944,
+            -0.01634538732469082,
+            0.009111093357205391,
+            -0.002007265342399478,
+            0.003736840793862939,
+            -0.03994900733232498,
+            0.017130577936768532,
+            0.025068672373890877,
+            -0.015129297971725464,
+            0.007253445219248533,
+            0.005845845676958561,
+            -0.007698706351220608,
+            0.0005371259758248925,
+            0.004426277242600918,
+            0.002757745562121272,
+            -0.02880311943590641,
+            0.020434128120541573,
+            0.018375394865870476,
+            0.007579012308269739,
+            0.004888295195996761,
+            -0.008929158560931683,
+            0.002621294464915991,
+            0.00989149697124958,
+            0.005544217303395271,
+            0.02405366860330105,
+            0.0031431596726179123,
+            -0.0006415588431991637,
+            0.0016374114202335477,
+            -0.0037224777042865753,
+            0.009767015464603901,
+            -0.005970327649265528,
+            0.005740515422075987,
+            -0.017877468839287758,
+            -0.0043329158797860146,
+            -0.0018349061720073223,
+            -0.004172526299953461,
+            -0.007794461213052273,
+            -0.001229255460202694,
+            -0.011844899505376816,
+            0.021410830318927765,
+            0.007377926725894213,
+            0.014210049994289875,
+            -0.01603897102177143,
+            0.0032556718215346336,
+            -0.021334225311875343,
+            -0.0036099653225392103,
+            -0.008311538025736809,
+            0.005946388468146324,
+            0.04404732212424278,
+            0.010006402619183064,
+            0.016661379486322403,
+            0.00033514268579892814,
+            0.020070258527994156,
+            -0.0066884905099868774,
+            -0.0055585806258022785,
+            -0.00047518443898297846,
+            -0.003830202156677842,
+            -0.0032867921981960535,
+            -0.0016050940612331033,
+            -0.008570076897740364,
+            -0.008321113884449005,
+            0.006678914651274681,
+            0.007890216074883938,
+            0.00667412718757987,
+            -0.0214491318911314,
+            0.01830836571753025,
+            -0.0007079889182932675,
+            -0.006037355866283178,
+            0.019763842225074768,
+            -0.0015620043268427253,
+            -0.002276576356962323,
+            -0.010418149642646313,
+            0.008258873596787453,
+            -0.0024082395248115063,
+            0.0469965785741806,
+            -0.009034489281475544,
+            0.02188960462808609,
+            -0.006439527031034231,
+            0.0023771191481500864,
+            0.003736840793862939,
+            0.0009264301043003798,
+            -0.006745943333953619,
+            -0.0021125958301126957,
+            -0.015541044995188713,
+            0.0056016705930233,
+            -0.015215477906167507,
+            0.012342826463282108,
+            -0.016795435920357704,
+            0.00372726540081203,
+            0.02361319586634636,
+            -0.0050558666698634624,
+            -0.00902012549340725,
+            0.0036913573276251554,
+            0.006587947718799114,
+            -0.0028271679766476154,
+            -0.02106611058115959,
+            0.013635518960654736,
+            -0.00193664594553411,
+            0.017427420243620872,
+            -0.007516771554946899,
+            -0.010963953100144863,
+            0.021104414016008377,
+            0.010092582553625107,
+            0.025049520656466484,
+            0.017312513664364815,
+            0.018863745033740997,
+            -0.001241224817931652,
+            0.014966514892876148,
+            -0.020721392706036568,
+            0.002918135141953826,
+            -0.012668393552303314,
+            -0.01989790052175522,
+            -0.0028175923507660627,
+            -0.004445428028702736,
+            0.01116503868252039,
+            -0.008546138182282448,
+            -0.010954378172755241,
+            0.007899791933596134,
+            -0.023345081135630608,
+            0.00324849016033113,
+            -0.008301963098347187,
+            0.017839165404438972,
+            0.0035477245692163706,
+            -0.012658817693591118,
+            -0.008795101195573807,
+            -0.006841698195785284,
+            -0.01174914464354515,
+            -0.020568184554576874,
+            0.03008623793721199,
+            0.01636453904211521,
+            0.015435714274644852,
+            -0.005678274668753147,
+            -0.002378316130489111,
+            -0.008689770475029945,
+            -0.0036267226096242666,
+            0.0008923173882067204,
+            -0.02822858840227127
+          ],
+          "title": "A Switch for Attention"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "3f90238f-c0a2-4a9a-8d7b-c937c99746b3",
+      "type": "next",
+      "source": {
+        "id": "b8ffe9ad-9e6f-4478-8767-d2c5561f1266",
+        "properties": {
+          "page_content": "A Switch for Attention 27 ",
+          "keyphrases": [
+            "Switch",
+            "Attention"
+          ],
+          "embedding": [
+            -0.004847599659115076,
+            -0.013338678516447544,
+            -0.017561476677656174,
+            0.018164733424782753,
+            -0.036252863705158234,
+            0.0018600418698042631,
+            0.006209715269505978,
+            0.06599438190460205,
+            0.014593069441616535,
+            -0.005357495043426752,
+            -0.02489631250500679,
+            0.024474991485476494,
+            0.01628793403506279,
+            -0.022100266069173813,
+            -0.018451998010277748,
+            -0.02642839401960373,
+            -0.014775004237890244,
+            0.0039068059995770454,
+            -0.036712486296892166,
+            -0.008148754946887493,
+            0.020644789561629295,
+            -0.02066393941640854,
+            -0.034203704446554184,
+            0.016929492354393005,
+            -0.03311209753155708,
+            -0.00733004929497838,
+            0.03295888751745224,
+            0.011835324577987194,
+            -0.04140448570251465,
+            0.04393241927027702,
+            -0.03259501978754997,
+            0.010667112655937672,
+            -2.788491838146001e-05,
+            0.008067362941801548,
+            -0.04025542363524437,
+            -0.014545192010700703,
+            -0.0072869593277573586,
+            -0.023517441004514694,
+            0.001378872781060636,
+            0.02303866483271122,
+            -0.045158080756664276,
+            -0.006899151485413313,
+            -0.05542302504181862,
+            -0.02713698148727417,
+            0.010226639918982983,
+            -0.015732554718852043,
+            -0.02164064161479473,
+            -0.03544852137565613,
+            -0.0038780795875936747,
+            -0.018672235310077667,
+            0.024723954498767853,
+            -0.03631031513214111,
+            -0.0005529854097403586,
+            0.00697575556114316,
+            0.007416228763759136,
+            0.0014470982132479548,
+            -0.003313124878332019,
+            0.010542631149291992,
+            -0.011088434606790543,
+            0.037287015467882156,
+            0.003653055289760232,
+            -0.01817430928349495,
+            -0.0034160614013671875,
+            -0.024800557643175125,
+            -0.015694253146648407,
+            -0.011126737110316753,
+            -0.022119417786598206,
+            -0.0168050117790699,
+            -0.022866306826472282,
+            -0.0010132081806659698,
+            -0.042974866926670074,
+            0.012716270983219147,
+            -0.04653695598244667,
+            0.01056178193539381,
+            0.006458678282797337,
+            0.007713069207966328,
+            0.03776579350233078,
+            0.03698059916496277,
+            -0.05538472160696983,
+            -0.002126959152519703,
+            -0.010284092277288437,
+            -0.02424517832696438,
+            0.010657536797225475,
+            0.026409242302179337,
+            0.011777871288359165,
+            -0.02905208244919777,
+            -0.02650499902665615,
+            -0.09989167004823685,
+            0.05113319680094719,
+            0.0056447600945830345,
+            -0.02842009998857975,
+            -0.025164427235722542,
+            0.007488044910132885,
+            -0.006329408846795559,
+            0.01437283307313919,
+            -0.008871705271303654,
+            0.028688212856650352,
+            -0.014803730882704258,
+            -0.007205567788332701,
+            -0.0008031455217860639,
+            0.029109535738825798,
+            -0.0010479193879291415,
+            -0.025624051690101624,
+            -0.025164427235722542,
+            0.03203963860869408,
+            0.03052671067416668,
+            -0.02981812320649624,
+            0.024091970175504684,
+            0.0008827418787404895,
+            0.004270675126463175,
+            0.01378872711211443,
+            -0.01235240139067173,
+            -0.013941935263574123,
+            -0.0073731387965381145,
+            0.004337703809142113,
+            -0.002435769187286496,
+            0.002875045407563448,
+            -0.018384970724582672,
+            -0.014075992628932,
+            -0.009834043681621552,
+            -0.0008546138415113091,
+            0.030737372115254402,
+            0.004285038448870182,
+            -0.012371552176773548,
+            -0.020568184554576874,
+            -0.005850633606314659,
+            0.012591789476573467,
+            0.01174914464354515,
+            -0.014028115198016167,
+            0.010102157481014729,
+            0.01855732873082161,
+            0.019601058214902878,
+            -0.0066980659030377865,
+            0.0014459012309089303,
+            0.023517441004514694,
+            0.002047961112111807,
+            0.025815561413764954,
+            0.03431861102581024,
+            -0.03956598788499832,
+            -0.0008270842372439802,
+            -0.028305193409323692,
+            0.0037583857774734497,
+            0.015598498284816742,
+            0.02265564538538456,
+            -0.013032262213528156,
+            0.024015367031097412,
+            0.014822881668806076,
+            0.009977675974369049,
+            -0.009541991166770458,
+            0.029473405331373215,
+            0.018643507733941078,
+            -0.0008181072189472616,
+            -0.03347596526145935,
+            -0.001977341715246439,
+            0.054235659539699554,
+            0.021104414016008377,
+            -0.0035405431408435106,
+            -0.00020632220548577607,
+            -0.01043730042874813,
+            0.010274517349898815,
+            -0.01779128797352314,
+            -0.006635825149714947,
+            -0.021295923739671707,
+            -0.004670452792197466,
+            0.00989149697124958,
+            -0.005845845676958561,
+            0.033514268696308136,
+            0.00400495482608676,
+            -0.031580016016960144,
+            0.05132470652461052,
+            0.029224442318081856,
+            0.025183578953146935,
+            0.02834349498152733,
+            0.015943216159939766,
+            0.0025758109986782074,
+            0.00035818375181406736,
+            0.05224395543336868,
+            0.013530189171433449,
+            0.003772749099880457,
+            0.014803730882704258,
+            -0.035295311361551285,
+            0.006875212769955397,
+            -0.0050558666698634624,
+            -0.025834713131189346,
+            0.008967460133135319,
+            0.0135972173884511,
+            0.02169809490442276,
+            0.008057788014411926,
+            -0.0009449826902709901,
+            0.004304189700633287,
+            -0.025585750117897987,
+            -0.03711465746164322,
+            0.019725540652871132,
+            -0.02087460085749626,
+            -0.01820303499698639,
+            -0.01625920832157135,
+            -0.020625637844204903,
+            -0.0026284761261194944,
+            0.04366430267691612,
+            0.00017774531443137676,
+            -0.009977675974369049,
+            0.026332639157772064,
+            0.018672235310077667,
+            0.012074711732566357,
+            -0.05971284955739975,
+            0.003966652788221836,
+            -0.009355268441140652,
+            -0.020204314962029457,
+            -0.0038541408721357584,
+            0.00015784621064085513,
+            0.03276737779378891,
+            -0.004438246600329876,
+            0.01959148421883583,
+            0.01880629174411297,
+            0.005544217303395271,
+            0.006554433144629002,
+            0.008124816231429577,
+            0.015761282294988632,
+            0.006875212769955397,
+            -0.0067076412960886955,
+            0.039834100753068924,
+            0.02196620963513851,
+            -0.006295894738286734,
+            0.025241032242774963,
+            -0.06572626531124115,
+            0.040025610476732254,
+            -0.03397389128804207,
+            0.03052671067416668,
+            0.0010760474251583219,
+            -0.027749814093112946,
+            -0.07373139262199402,
+            0.0029277107678353786,
+            0.005917662288993597,
+            0.03077567368745804,
+            -0.008775950409471989,
+            -0.004912233911454678,
+            -0.009790954180061817,
+            0.01677628420293331,
+            0.01882544346153736,
+            -0.013999388553202152,
+            0.01795407198369503,
+            0.012045985087752342,
+            -0.015349534340202808,
+            0.0011514545185491443,
+            0.004505275283008814,
+            -0.009259513579308987,
+            -0.020127711817622185,
+            0.027481699362397194,
+            -0.02309611812233925,
+            -0.012620516121387482,
+            -0.0157134048640728,
+            0.0023316354490816593,
+            -0.019026529043912888,
+            0.012180042453110218,
+            -0.0004691997601184994,
+            0.05978945270180702,
+            -0.011270369403064251,
+            -0.02016601338982582,
+            0.0089483093470335,
+            0.014468587934970856,
+            0.009939374402165413,
+            0.02522188052535057,
+            -0.005553793162107468,
+            0.039834100753068924,
+            0.010312818922102451,
+            -0.027673209086060524,
+            0.03742107376456261,
+            -0.05982775613665581,
+            0.006616673897951841,
+            -0.019409548491239548,
+            0.002752957632765174,
+            -0.006286319345235825,
+            0.013549339957535267,
+            0.01759977824985981,
+            0.014391983859241009,
+            0.020912902429699898,
+            0.0032676411792635918,
+            6.702853715978563e-05,
+            -0.014190898276865482,
+            -0.018136005848646164,
+            -0.010466027073562145,
+            0.06400267779827118,
+            0.033629175275564194,
+            -0.005525066517293453,
+            -0.0273284912109375,
+            0.00811524037271738,
+            0.03194388374686241,
+            0.03274822607636452,
+            -0.028822271153330803,
+            -0.0005173765239305794,
+            0.004268281627446413,
+            -0.018193459138274193,
+            0.0030162842012941837,
+            -0.044315438717603683,
+            0.03487398847937584,
+            -0.0241877268999815,
+            0.037018902599811554,
+            0.04795413091778755,
+            0.005022352561354637,
+            -0.019917050376534462,
+            -0.03000963293015957,
+            -0.014746277593076229,
+            -0.005209074821323156,
+            -0.03272907808423042,
+            -0.0379764549434185,
+            0.0254133902490139,
+            0.01751359924674034,
+            0.021544886752963066,
+            -0.06415588408708572,
+            0.014707976020872593,
+            0.05676359310746193,
+            0.014075992628932,
+            -0.014794155023992062,
+            -0.017283786088228226,
+            -0.010169186629354954,
+            -0.0022598193027079105,
+            0.018566904589533806,
+            0.010485177859663963,
+            -0.014535617083311081,
+            -0.03090973012149334,
+            -0.017140153795480728,
+            -0.004510062746703625,
+            -0.025758108124136925,
+            0.03680824115872383,
+            -0.015071844682097435,
+            -0.010418149642646313,
+            0.008507836610078812,
+            0.037918999791145325,
+            -0.033629175275564194,
+            0.012409854680299759,
+            -0.055959250777959824,
+            -0.02093205414712429,
+            -0.008479109965264797,
+            -0.01735081523656845,
+            0.0037129020784050226,
+            -0.0025087823159992695,
+            0.014085568487644196,
+            0.02476225607097149,
+            -0.01907440647482872,
+            0.006564009003341198,
+            -0.011500181630253792,
+            0.015646375715732574,
+            -0.007665191777050495,
+            -0.020778845995664597,
+            -0.05235886201262474,
+            0.0008205011254176497,
+            0.03027774766087532,
+            -0.01257263869047165,
+            0.015234628692269325,
+            -0.0014913849299773574,
+            0.0011592345545068383,
+            -0.02137252688407898,
+            0.007320473901927471,
+            -0.006564009003341198,
+            0.036003898829221725,
+            -0.018605206161737442,
+            -0.023019514977931976,
+            0.011902352795004845,
+            -0.011605512350797653,
+            -0.01891162246465683,
+            0.025011219084262848,
+            -0.03263332322239876,
+            0.01043730042874813,
+            0.005817119497805834,
+            -0.06281531602144241,
+            -0.05163112282752991,
+            0.0039762286469340324,
+            -0.009786166250705719,
+            -0.009872346185147762,
+            -0.004897871054708958,
+            -0.08097047358751297,
+            -0.009671260602772236,
+            -0.04385581240057945,
+            0.02552829682826996,
+            0.0168050117790699,
+            0.017341239377856255,
+            -0.0038876549806445837,
+            0.005750090815126896,
+            -0.03761258348822594,
+            0.02330677956342697,
+            -0.014842033386230469,
+            0.017427420243620872,
+            -0.019189313054084778,
+            -0.029971331357955933,
+            0.04063844308257103,
+            0.02700292505323887,
+            0.02533678710460663,
+            0.006592735182493925,
+            -0.043626002967357635,
+            0.028017928823828697,
+            -0.021391678601503372,
+            -0.005075017921626568,
+            -0.0064491028897464275,
+            0.0048859016969799995,
+            0.021927906200289726,
+            -0.008129604160785675,
+            0.008268448524177074,
+            -0.0007175644277594984,
+            -0.001700849155895412,
+            -0.05339301750063896,
+            -0.013520613312721252,
+            0.02661990374326706,
+            -0.005697425454854965,
+            -0.030392654240131378,
+            0.010801170021295547,
+            0.010063855908811092,
+            -0.06534324586391449,
+            -0.010944802314043045,
+            0.0004428671090863645,
+            0.013156743720173836,
+            0.024398386478424072,
+            -0.06534324586391449,
+            0.013338678516447544,
+            -0.030584163963794708,
+            0.010628811083734035,
+            0.045541103929281235,
+            0.006157049909234047,
+            0.0551932118833065,
+            0.004232373088598251,
+            0.0008839388028718531,
+            0.014545192010700703,
+            0.0316949225962162,
+            0.013980237767100334,
+            0.02951170690357685,
+            -0.014238775707781315,
+            0.0012543911579996347,
+            0.03596559911966324,
+            0.005199499428272247,
+            0.04301317036151886,
+            0.0017367572290822864,
+            0.00875679962337017,
+            -0.04370260611176491,
+            -0.03975749760866165,
+            -0.008866917341947556,
+            -0.0007863883511163294,
+            -0.001040737726725638,
+            -0.03994900733232498,
+            -0.0033107309136539698,
+            -0.03968089446425438,
+            -0.04423883557319641,
+            0.02323017455637455,
+            0.03994900733232498,
+            -0.02426433004438877,
+            0.01786789298057556,
+            -0.007732220459729433,
+            0.019763842225074768,
+            0.002664384199306369,
+            -0.013204621151089668,
+            0.006477829068899155,
+            0.0224066823720932,
+            -0.028190286830067635,
+            0.03617625683546066,
+            -0.0029588311444967985,
+            0.002010856056585908,
+            -0.007052359636873007,
+            0.004778177011758089,
+            0.01732208952307701,
+            0.026102827861905098,
+            -0.031139543280005455,
+            -0.025432541966438293,
+            0.0445452481508255,
+            0.03265247121453285,
+            0.03768918663263321,
+            -0.04940960556268692,
+            0.005448462441563606,
+            -0.005620821379125118,
+            -0.06055549532175064,
+            0.04075334966182709,
+            -9.201461944030598e-05,
+            0.014688825234770775,
+            -0.03060331381857395,
+            -0.022559890523552895,
+            -0.04703488200902939,
+            0.007018845062702894,
+            0.0016769103240221739,
+            -0.015483591705560684,
+            -0.015502742491662502,
+            -0.04504317790269852,
+            -0.027098679915070534,
+            -0.0336100235581398,
+            -0.0188541691750288,
+            -0.01513887383043766,
+            -0.0037224777042865753,
+            0.003921169321984053,
+            0.01097352895885706,
+            0.016709256917238235,
+            0.01247688289731741,
+            -0.008210995234549046,
+            -0.05186093598604202,
+            -0.01637411303818226,
+            0.022349229082465172,
+            -0.008622742258012295,
+            0.035161253064870834,
+            -0.006678914651274681,
+            -0.013022687286138535,
+            -0.053278110921382904,
+            -0.01699652150273323,
+            -0.04243863746523857,
+            0.02393876202404499,
+            0.05389094352722168,
+            0.00978137832134962,
+            -0.012936507351696491,
+            0.007511983625590801,
+            -0.0024537232238799334,
+            -0.019007377326488495,
+            -0.017475297674536705,
+            0.02566235326230526,
+            -0.017044398933649063,
+            -0.0073635634034872055,
+            -0.027290189638733864,
+            0.019763842225074768,
+            0.027347642928361893,
+            0.01561764907091856,
+            0.0010646764421835542,
+            0.02003195695579052,
+            0.004567516036331654,
+            -0.003028253559023142,
+            0.027826417237520218,
+            0.01593364030122757,
+            -0.0025303272996097803,
+            0.01721675880253315,
+            -0.012314099818468094,
+            0.005477189086377621,
+            0.042093921452760696,
+            -0.01270669512450695,
+            0.018815867602825165,
+            0.03678908944129944,
+            -0.00822535902261734,
+            0.027998777106404305,
+            -0.04132787883281708,
+            0.021295923739671707,
+            0.046115633100271225,
+            -0.02368979901075363,
+            0.009206848219037056,
+            -0.001639805268496275,
+            0.002424996579065919,
+            -0.014583494514226913,
+            0.002250243676826358,
+            -0.02188960462808609,
+            -0.0060325684025883675,
+            0.00822535902261734,
+            0.016307085752487183,
+            -0.02778811566531658,
+            -0.01008300669491291,
+            0.010456451214849949,
+            0.02932019717991352,
+            0.025777259841561317,
+            -0.025451691821217537,
+            0.01462179608643055,
+            -0.009527627378702164,
+            0.03301634266972542,
+            -0.016910342499613762,
+            0.03188643231987953,
+            0.004318553023040295,
+            -0.007095449138432741,
+            -0.02956916019320488,
+            -0.012783299200236797,
+            0.04653695598244667,
+            0.005893723573535681,
+            0.010580932721495628,
+            0.003533361479640007,
+            -0.01718803122639656,
+            0.021046960726380348,
+            0.02573895826935768,
+            -0.010963953100144863,
+            -0.004222797695547342,
+            -0.0013704941375181079,
+            0.033188700675964355,
+            0.03148426115512848,
+            -0.014401559717953205,
+            -0.017724260687828064,
+            0.0016350174555554986,
+            0.01283117663115263,
+            -0.028764817863702774,
+            -0.004581879358738661,
+            0.058333978056907654,
+            0.004543577320873737,
+            -0.02713698148727417,
+            -0.015004816465079784,
+            -0.007162477821111679,
+            -0.011299096047878265,
+            -0.023900460451841354,
+            0.029722368344664574,
+            -0.009800529107451439,
+            -0.02137252688407898,
+            -0.02183215133845806,
+            0.00822535902261734,
+            -0.026658207178115845,
+            -0.03129275143146515,
+            0.03014368936419487,
+            0.017810439690947533,
+            0.04014051705598831,
+            0.006338984239846468,
+            -0.015837885439395905,
+            -0.024666501209139824,
+            4.4511136366054416e-05,
+            0.01978299394249916,
+            -0.01490906160324812,
+            -0.0013297982513904572,
+            -0.034012194722890854,
+            0.004964899271726608,
+            0.010983104817569256,
+            0.018068978562951088,
+            -0.01577085629105568,
+            0.0026715658605098724,
+            0.029281895607709885,
+            -0.013444009236991405,
+            0.012055560946464539,
+            -0.017695533111691475,
+            0.016623076051473618,
+            -0.01658477447926998,
+            -0.005161197390407324,
+            0.033629175275564194,
+            -0.03678908944129944,
+            0.008129604160785675,
+            0.003705720417201519,
+            0.009379207156598568,
+            0.025049520656466484,
+            -0.002621294464915991,
+            -0.005333556327968836,
+            0.024302631616592407,
+            -0.021334225311875343,
+            0.024934615939855576,
+            0.008234934881329536,
+            0.010188337415456772,
+            0.006640612613409758,
+            -0.017504023388028145,
+            -0.011710843071341515,
+            -0.02183215133845806,
+            -0.0018253306625410914,
+            0.01132782269269228,
+            -0.0047135422937572,
+            -0.008455171249806881,
+            -0.019285067915916443,
+            0.0004862561181653291,
+            0.009680835530161858,
+            0.04557940363883972,
+            -0.014870759099721909,
+            0.013846180401742458,
+            0.01295565813779831,
+            -0.010130884125828743,
+            0.007282171864062548,
+            0.03140765801072121,
+            0.03372493013739586,
+            0.0008701740298420191,
+            0.004385581240057945,
+            0.022310927510261536,
+            0.026773111894726753,
+            0.0032125820871442556,
+            0.0057213641703128815,
+            -0.007492832839488983,
+            0.00989149697124958,
+            0.0008456368232145905,
+            0.01951487921178341,
+            0.011959806084632874,
+            -0.03014368936419487,
+            -0.02497291751205921,
+            -0.011385275982320309,
+            -0.023823857307434082,
+            0.01948615349829197,
+            -0.031637467443943024,
+            -0.02060648798942566,
+            0.011844899505376816,
+            0.009455811232328415,
+            0.0021125958301126957,
+            -0.00465369550511241,
+            -0.02093205414712429,
+            0.03378238156437874,
+            -0.015129297971725464,
+            0.060708701610565186,
+            -0.009757439605891705,
+            -0.03713380917906761,
+            -0.023670649155974388,
+            -0.008163117803633213,
+            -0.017255060374736786,
+            -0.03768918663263321,
+            -0.020070258527994156,
+            0.022808853536844254,
+            -0.01091607566922903,
+            -0.017532749101519585,
+            -0.0008426444255746901,
+            -0.000540118315257132,
+            -0.011624663136899471,
+            0.0006493389373645186,
+            0.021908756345510483,
+            0.005046291276812553,
+            0.0008887265576049685,
+            0.008124816231429577,
+            -0.021851303055882454,
+            0.005754878744482994,
+            -0.004373611882328987,
+            0.02278970181941986,
+            -0.026390092447400093,
+            -0.0026308700907975435,
+            -0.029684066772460938,
+            -0.05986605957150459,
+            -0.030316049233078957,
+            -0.017618929967284203,
+            0.012926931492984295,
+            0.007468894124031067,
+            0.011509757488965988,
+            -0.025566598400473595,
+            -0.004541183356195688,
+            -0.011978956870734692,
+            0.011710843071341515,
+            0.022483285516500473,
+            -0.005496339872479439,
+            -0.020836299285292625,
+            -0.003054586239159107,
+            -0.022061964496970177,
+            0.006812972016632557,
+            0.006927878130227327,
+            -0.0273284912109375,
+            0.05484849214553833,
+            -0.0005957759567536414,
+            0.0325184166431427,
+            -0.013568490743637085,
+            -0.019581908360123634,
+            -0.040102217346429825,
+            -0.011069283820688725,
+            -0.007248657289892435,
+            0.022425832226872444,
+            -0.025126125663518906,
+            -0.009609019383788109,
+            -0.019342521205544472,
+            -0.008230146951973438,
+            0.023670649155974388,
+            0.04224712774157524,
+            0.0333419106900692,
+            -0.003293973859399557,
+            0.00793809350579977,
+            0.002160473261028528,
+            -0.016623076051473618,
+            -0.014363258145749569,
+            -0.014315379783511162,
+            -0.003641085932031274,
+            0.007224718574434519,
+            0.03537191450595856,
+            0.030124539509415627,
+            -0.04305147007107735,
+            0.018126431852579117,
+            0.03784239664673805,
+            0.010054280050098896,
+            0.014344106428325176,
+            0.019744692370295525,
+            0.011729993857443333,
+            0.02905208244919777,
+            0.015081420540809631,
+            -0.01306098885834217,
+            -0.005917662288993597,
+            -0.010025554336607456,
+            -0.03339936211705208,
+            0.0243217833340168,
+            -0.010446876287460327,
+            -0.010284092277288437,
+            0.016766710206866264,
+            0.019581908360123634,
+            -0.0005595685797743499,
+            0.019821295514702797,
+            0.0024776619393378496,
+            -0.032863132655620575,
+            -0.009230786934494972,
+            -0.0018791928887367249,
+            0.014775004237890244,
+            0.012926931492984295,
+            -0.015196327120065689,
+            0.025260182097554207,
+            -0.023440835997462273,
+            0.01603897102177143,
+            0.011902352795004845,
+            0.014822881668806076,
+            0.015426138415932655,
+            0.022061964496970177,
+            -0.008862130343914032,
+            -0.004617787431925535,
+            0.015090996399521828,
+            0.004258705768734217,
+            -0.006161837372928858,
+            -0.015081420540809631,
+            0.016565624624490738,
+            -0.007842338643968105,
+            -0.004950535949319601,
+            -0.02765405923128128,
+            -0.005285678897053003,
+            -0.010839471593499184,
+            -0.007291747257113457,
+            0.015512318350374699,
+            -0.005146834068000317,
+            0.0074401674792170525,
+            -0.0003764370339922607,
+            0.021736396476626396,
+            -0.00026108213933184743,
+            0.0004120459489058703,
+            0.016489019617438316,
+            0.050520364195108414,
+            -0.02079799771308899,
+            -0.01209386344999075,
+            -0.0022514406591653824,
+            -0.0008977036341093481,
+            -0.010169186629354954,
+            -0.004342491738498211,
+            -0.005199499428272247,
+            -0.006501767784357071,
+            0.01270669512450695,
+            0.02060648798942566,
+            -0.025394240394234657,
+            -0.010992679744958878,
+            -0.023191872984170914,
+            -0.0377083383500576,
+            0.010178761556744576,
+            -0.01490906160324812,
+            -0.04730299487709999,
+            0.03937447816133499,
+            -0.014152596704661846,
+            0.010734140872955322,
+            0.029205290600657463,
+            -0.0032197635155171156,
+            -0.04722639173269272,
+            -0.0060421437956392765,
+            0.005414948333054781,
+            -0.00098208780400455,
+            -0.0031216146890074015,
+            0.0401788204908371,
+            -0.017829591408371925,
+            0.003597995964810252,
+            -0.0185860563069582,
+            0.01997450366616249,
+            -0.01689119078218937,
+            0.036137957125902176,
+            0.03598474711179733,
+            -0.04680506885051727,
+            -0.0033059432171285152,
+            -0.03292058780789375,
+            0.020912902429699898,
+            0.007784885820001364,
+            0.011059708893299103,
+            -0.014765429310500622,
+            0.020434128120541573,
+            -0.023536590859293938,
+            0.028726516291499138,
+            0.011835324577987194,
+            0.033054642379283905,
+            -0.023440835997462273,
+            -0.01746572181582451,
+            -0.02713698148727417,
+            -0.003203006461262703,
+            0.01937124691903591,
+            -0.02361319586634636,
+            0.010130884125828743,
+            0.019083982333540916,
+            0.01414302084594965,
+            0.026964621618390083,
+            0.020702242851257324,
+            -0.0015284899855032563,
+            -0.015090996399521828,
+            -0.00834984052926302,
+            0.04511978104710579,
+            0.007382714655250311,
+            0.0004383785999380052,
+            -0.005467613227665424,
+            0.011318246833980083,
+            0.026524148881435394,
+            0.030833126977086067,
+            -0.015646375715732574,
+            0.01625920832157135,
+            -0.008167905732989311,
+            -0.011011830531060696,
+            -0.028783967718482018,
+            -0.0021628672257065773,
+            -0.03186728060245514,
+            -0.03893400356173515,
+            0.007957245223224163,
+            -0.03521870821714401,
+            -0.002626082394272089,
+            0.015359110198915005,
+            0.021870452910661697,
+            0.02464734949171543,
+            -0.028592457994818687,
+            0.028879722580313683,
+            0.019332945346832275,
+            -0.008354628458619118,
+            -0.001125121838413179,
+            -0.027175283059477806,
+            -0.003918775357306004,
+            0.0018947530770674348,
+            -0.014918637461960316,
+            -0.021276772022247314,
+            -0.007176841143518686,
+            -0.012084287591278553,
+            -0.0061187478713691235,
+            0.009446235373616219,
+            -0.007813611999154091,
+            0.002906165784224868,
+            -0.025375088676810265,
+            0.03349511697888374,
+            0.003466332796961069,
+            0.007861489430069923,
+            -0.01577085629105568,
+            0.0006451496738009155,
+            0.0021137928124517202,
+            -0.003650661325082183,
+            -0.009269089438021183,
+            0.03259501978754997,
+            -0.010398998856544495,
+            0.0054532503709197044,
+            -0.013233347795903683,
+            0.0019222826231271029,
+            0.01863393373787403,
+            0.016824161633849144,
+            -0.017590202391147614,
+            0.0043329158797860146,
+            -0.005060654599219561,
+            0.004687209613621235,
+            -0.005194711498916149,
+            0.027998777106404305,
+            -0.011931079439818859,
+            -9.934586705639958e-05,
+            -0.0038948366418480873,
+            -0.009316966868937016,
+            0.024111121892929077,
+            0.000726541446056217,
+            0.014899485744535923,
+            0.023459987714886665,
+            0.023000363260507584,
+            -0.013242923654615879,
+            -0.007023632992058992,
+            0.022674795240163803,
+            0.024226028472185135,
+            0.01209386344999075,
+            -0.009541991166770458,
+            0.002733806613832712,
+            0.004045650828629732,
+            0.029664915055036545,
+            0.0029444678220897913,
+            0.010178761556744576,
+            -0.02669650875031948,
+            -0.02688801847398281,
+            0.013846180401742458,
+            -0.012400278821587563,
+            0.0049888379871845245,
+            0.012132165022194386,
+            0.013568490743637085,
+            0.005003201309591532,
+            -0.001164620858617127,
+            0.009302603080868721,
+            0.012065136805176735,
+            0.006147474516183138,
+            0.02355574257671833,
+            0.012821601703763008,
+            -0.015119723044335842,
+            -0.006875212769955397,
+            0.002081475453451276,
+            0.00019390397937968373,
+            -0.008857342414557934,
+            -0.03096718341112137,
+            -0.06235568970441818,
+            -0.02183215133845806,
+            -0.014526041224598885,
+            0.004560334142297506,
+            0.02503037080168724,
+            0.025317635387182236,
+            -0.0049888379871845245,
+            -0.04653695598244667,
+            -0.0034783021546900272,
+            0.006980543024837971,
+            0.014727126806974411,
+            -0.004538789391517639,
+            0.02451329305768013,
+            -0.023574894294142723,
+            0.005223438143730164,
+            0.0072007798589766026,
+            -0.023862158879637718,
+            0.03904891014099121,
+            0.02368979901075363,
+            -0.01174914464354515,
+            -0.01735081523656845,
+            -0.018432848155498505,
+            0.012180042453110218,
+            -0.0034495757427066565,
+            0.022636493667960167,
+            0.03297803923487663,
+            -0.03979580104351044,
+            0.043357886373996735,
+            -0.006937453523278236,
+            -0.002997133182361722,
+            -0.01056178193539381,
+            -0.0028606820851564407,
+            0.0014662492321804166,
+            0.0029708005022257566,
+            -0.004320946522057056,
+            0.01779128797352314,
+            -0.018796715885400772,
+            0.023402534425258636,
+            0.008287599310278893,
+            0.022368380799889565,
+            -0.004325734451413155,
+            0.020146863535046577,
+            -0.07281213998794556,
+            0.007004482205957174,
+            -0.004548364784568548,
+            -0.010896924883127213,
+            -0.04967772215604782,
+            -0.02721358649432659,
+            0.06036398559808731,
+            -0.03437606245279312,
+            0.02175554819405079,
+            0.02688801847398281,
+            0.0251452773809433,
+            -0.032346054911613464,
+            0.00023968686582520604,
+            -0.003961865324527025,
+            0.024628199636936188,
+            -0.00918290950357914,
+            0.034471817314624786,
+            0.021161865442991257,
+            0.015129297971725464,
+            0.015004816465079784,
+            -0.002413027221336961,
+            -0.01574213057756424,
+            0.08441765606403351,
+            -0.011461880058050156,
+            -0.041902411729097366,
+            0.03000963293015957,
+            -0.0404469333589077,
+            0.00390441226772964,
+            0.01295565813779831,
+            -0.0061187478713691235,
+            0.014995240606367588,
+            -0.0024142242036759853,
+            -0.009585080668330193,
+            -0.021085262298583984,
+            0.0020240223966538906,
+            -0.03301634266972542,
+            -0.02252158895134926,
+            -0.004790146369487047,
+            0.003511816496029496,
+            0.0008617954445071518,
+            -0.0029875575564801693,
+            -0.004189283587038517,
+            0.006338984239846468,
+            0.04347279295325279,
+            -0.023364232853055,
+            0.011030982248485088,
+            -0.005654335953295231,
+            -0.026600753888487816,
+            0.033839836716651917,
+            0.013558914884924889,
+            0.005046291276812553,
+            -0.01497608982026577,
+            0.008359416387975216,
+            0.002378316130489111,
+            -0.031388506293296814,
+            0.00649698032066226,
+            -0.029741518199443817,
+            -0.004256311804056168,
+            0.005496339872479439,
+            -0.00047727907076478004,
+            -0.0066118864342570305,
+            0.030545860528945923,
+            -0.01645071804523468,
+            -0.021046960726380348,
+            -0.009800529107451439,
+            -0.004708754830062389,
+            -0.009503688663244247,
+            -0.007339624688029289,
+            -0.00037075160071253777,
+            0.0007103827665559947,
+            0.03320785239338875,
+            0.02661990374326706,
+            0.011174614541232586,
+            -0.010772443376481533,
+            -0.0252793338149786,
+            0.003966652788221836,
+            -0.0022693946957588196,
+            0.02164064161479473,
+            0.025202728807926178,
+            -0.0262368842959404,
+            -0.008546138182282448,
+            -0.036003898829221725,
+            -0.010226639918982983,
+            -0.009144607000052929,
+            0.011423577554523945,
+            -0.011346973478794098,
+            0.038819096982479095,
+            -0.010446876287460327,
+            0.013367405161261559,
+            -0.026390092447400093,
+            0.008531775325536728,
+            0.013453585095703602,
+            -0.003674600040540099,
+            -0.04557940363883972,
+            0.01490906160324812,
+            0.01587618701159954,
+            -0.0016505776438862085,
+            0.008689770475029945,
+            -0.02407282032072544,
+            -0.01533038355410099,
+            0.06955646723508835,
+            0.01037027221173048,
+            -0.01620175503194332,
+            -0.013252499513328075,
+            -0.024666501209139824,
+            -0.011902352795004845,
+            -0.013769576326012611,
+            0.01735081523656845,
+            -0.010858622379601002,
+            0.004232373088598251,
+            -0.007669979706406593,
+            0.024091970175504684,
+            0.007459318730980158,
+            0.0012986778747290373,
+            0.02650499902665615,
+            -0.003511816496029496,
+            -0.02495376579463482,
+            0.03916381672024727,
+            -0.017235908657312393,
+            -0.011366124264895916,
+            -0.019304217770695686,
+            -0.0021149897947907448,
+            -0.02125762216746807,
+            0.03571663424372673,
+            -0.021525735035538673,
+            -0.0011867642169818282,
+            -0.011797022074460983,
+            -0.019668087363243103,
+            -0.007574224378913641,
+            0.009939374402165413,
+            0.01904567889869213,
+            0.010351121425628662,
+            -0.0029253168031573296,
+            0.00827802438288927,
+            -0.008498260751366615,
+            0.031714074313640594,
+            0.0035668755881488323,
+            0.015148448757827282,
+            0.020395826548337936,
+            0.013175895437598228,
+            0.0045723035000264645,
+            0.018461573868989944,
+            -0.010762867517769337,
+            0.006812972016632557,
+            -0.026466695591807365,
+            -0.014382408931851387,
+            0.023919612169265747,
+            -0.013290801085531712,
+            -0.014334531500935555,
+            0.0010844259522855282,
+            -0.0039068059995770454,
+            0.0022454559803009033,
+            -0.012333250604569912,
+            -0.0054628257639706135,
+            -0.017006097361445427,
+            0.009834043681621552,
+            0.009776590391993523,
+            0.01863393373787403,
+            0.004569910001009703,
+            0.029645763337612152,
+            -0.007751371245831251,
+            -0.007780097890645266,
+            -0.0073635634034872055,
+            -0.02298121154308319,
+            0.005237801466137171,
+            0.021870452910661697,
+            0.0006702853715978563,
+            -0.005596882663667202,
+            -0.023459987714886665,
+            0.0061378986574709415,
+            -0.011959806084632874,
+            0.021027809008955956,
+            -0.0007313292007893324,
+            0.018672235310077667,
+            0.0028032290283590555,
+            -0.03288228437304497,
+            -0.0033298819325864315,
+            -0.008421656675636768,
+            -0.00590329896658659,
+            -0.014889910817146301,
+            -0.004088740795850754,
+            -0.0015464440220966935,
+            -0.0031886431388556957,
+            0.037401922047138214,
+            -0.007128963712602854,
+            -0.009000974707305431,
+            -0.0014949757605791092,
+            0.015722978860139847,
+            0.020012805238366127,
+            0.0009467780473642051,
+            -0.008474322035908699,
+            0.010427725501358509,
+            -8.408490248257294e-05,
+            0.010762867517769337,
+            -0.0011221294989809394,
+            -0.008503048680722713,
+            0.001982129644602537,
+            -0.0003803270810749382,
+            -0.027979625388979912,
+            -0.005884147714823484,
+            0.0011592345545068383,
+            -0.013376981019973755,
+            0.038302019238471985,
+            0.005328768398612738,
+            -0.007004482205957174,
+            0.0015428533079102635,
+            0.018346667289733887,
+            0.01132782269269228,
+            0.010226639918982983,
+            -0.003126402385532856,
+            0.004490911960601807,
+            -0.04511978104710579,
+            -0.027749814093112946,
+            -0.025949619710445404,
+            -0.03156086429953575,
+            0.015445290133357048,
+            0.018317941576242447,
+            -0.006875212769955397,
+            0.013673821464180946,
+            0.012812025845050812,
+            -0.001474627759307623,
+            0.009695199318230152,
+            -0.012802449986338615,
+            -0.0017247878713533282,
+            -0.00949890073388815,
+            -0.011710843071341515,
+            0.018001949414610863,
+            0.011797022074460983,
+            0.024206876754760742,
+            0.03929787129163742,
+            -0.011950230225920677,
+            0.0026859291829168797,
+            0.0011843702523037791,
+            0.006631037220358849,
+            -0.005711788777261972,
+            -0.013989812694489956,
+            0.010580932721495628,
+            -0.000983883161097765,
+            0.014707976020872593,
+            -0.01209386344999075,
+            0.0019665693398565054,
+            -0.00619535194709897,
+            -0.013961086049675941,
+            0.011366124264895916,
+            -0.01805940270423889,
+            0.012371552176773548,
+            -0.03941277787089348,
+            -0.007315685972571373,
+            -0.016489019617438316,
+            0.051554519683122635,
+            0.007933305576443672,
+            -0.01782001554965973,
+            -0.0017511205514892936,
+            0.025624051690101624,
+            0.004100710153579712,
+            -0.009771803393959999,
+            0.008641893044114113,
+            -0.00548676447942853,
+            0.011050133034586906,
+            0.0009144607465714216,
+            0.0029995269142091274,
+            -0.026026222854852676,
+            -0.005855421535670757,
+            -0.0054628257639706135,
+            0.01989790052175522,
+            0.000962338293902576,
+            0.016900766640901566,
+            0.08870747685432434,
+            -0.0035070287995040417,
+            -0.009451023302972317,
+            -0.0068608494475483894,
+            -0.019763842225074768,
+            -0.004562728106975555,
+            -0.008900431916117668,
+            -0.005639972630888224,
+            0.0030426166485995054,
+            -0.007186416536569595,
+            0.0054724011570215225,
+            0.014229200780391693,
+            -0.010925651527941227,
+            -0.008699346333742142,
+            -0.007195991929620504,
+            -0.01430580485612154,
+            0.016852889209985733,
+            0.031656619161367416,
+            0.0116821164265275,
+            0.04902658611536026,
+            -0.02533678710460663,
+            -0.014861184172332287,
+            0.004873932339251041,
+            0.004251524340361357,
+            0.00989149697124958,
+            0.007464106194674969,
+            -0.012409854680299759,
+            0.02330677956342697,
+            0.007756159175187349,
+            -0.0037990817800164223,
+            -0.012687544338405132,
+            0.015588922426104546,
+            0.015464440919458866,
+            -0.006061294581741095,
+            0.031522564589977264,
+            0.0025542660150676966,
+            0.010667112655937672,
+            0.010398998856544495,
+            0.0049026585184037685,
+            0.01557934656739235,
+            0.014095143415033817,
+            -0.0006271955789998174,
+            0.002482449635863304,
+            -0.002757745562121272,
+            0.013319527730345726,
+            0.018289214000105858,
+            0.012984384782612324,
+            -0.005831482820212841,
+            0.02560489997267723,
+            0.026581602171063423,
+            -0.00262368842959404,
+            -0.031120391562581062,
+            0.03864673897624016,
+            -0.020261768251657486,
+            0.0006313848425634205,
+            -0.008890856057405472,
+            0.018384970724582672,
+            -0.009403145872056484,
+            -0.004950535949319601,
+            -0.009163758717477322,
+            -0.028822271153330803,
+            -0.003729659365490079,
+            -0.003921169321984053,
+            0.012668393552303314,
+            0.01869138516485691,
+            -0.010628811083734035,
+            0.02112356387078762,
+            -0.012649241834878922,
+            0.020510731264948845,
+            -0.008570076897740364,
+            -0.022004511207342148,
+            0.008713710121810436,
+            -0.001651774626225233,
+            0.0019426306243985891,
+            -0.015177175402641296,
+            0.021219318732619286,
+            0.04684337228536606,
+            0.004869144409894943,
+            -0.00040815590182319283,
+            -0.007990758866071701,
+            -0.0033969103824347258,
+            -0.010734140872955322,
+            0.021295923739671707,
+            0.0058793602511286736,
+            -0.032863132655620575,
+            0.009752651676535606,
+            0.015885762870311737,
+            0.0014782185899093747,
+            0.017810439690947533,
+            -0.0013908421387895942,
+            0.03156086429953575,
+            0.006587947718799114,
+            0.004258705768734217,
+            0.0036554490216076374,
+            -0.014928212389349937,
+            0.0025279333349317312,
+            -0.0027745026163756847,
+            -0.02727103792130947,
+            0.00483084237203002,
+            -0.0012160891201347113,
+            -0.020912902429699898,
+            0.02233007736504078,
+            -0.03841692581772804,
+            -0.019083982333540916,
+            -0.003083312651142478,
+            -0.018844593316316605,
+            -0.013252499513328075,
+            0.01937124691903591,
+            -0.013999388553202152,
+            0.03636777028441429,
+            -0.007899791933596134,
+            0.007962032221257687,
+            -0.0036865693982690573,
+            -0.016948644071817398,
+            0.038876552134752274,
+            0.0045579406432807446,
+            -0.011078859679400921,
+            -0.008694558404386044,
+            0.022425832226872444,
+            -0.02213856764137745,
+            0.017168881371617317,
+            0.005673486739397049,
+            0.029856424778699875,
+            -0.02233007736504078,
+            0.01068626344203949,
+            0.025777259841561317,
+            0.009613807313144207,
+            -0.006003841757774353,
+            -0.010226639918982983,
+            -0.031139543280005455,
+            0.031771525740623474,
+            0.04151938855648041,
+            0.004936173092573881,
+            -0.005544217303395271,
+            0.0022514406591653824,
+            0.013511037454009056,
+            0.014583494514226913,
+            0.010552207008004189,
+            0.007119387853890657,
+            0.0020240223966538906,
+            -0.032097093760967255,
+            0.018739264458417892,
+            0.002458510920405388,
+            0.015177175402641296,
+            0.009063215926289558,
+            -0.026792263612151146,
+            -0.0009743076516315341,
+            -0.0021437162067741156,
+            0.007617314346134663,
+            0.02003195695579052,
+            0.013444009236991405,
+            0.028017928823828697,
+            0.00994895026087761,
+            0.0016924705123528838,
+            -0.012563062831759453,
+            -0.03315039724111557,
+            0.010178761556744576,
+            0.016737982630729675,
+            0.004593848716467619,
+            -0.01686246506869793,
+            0.010466027073562145,
+            0.007282171864062548,
+            -0.01145230419933796,
+            -0.014995240606367588,
+            -0.0064634657464921474,
+            -0.0038852612487971783,
+            -0.0302011426538229,
+            0.006094809155911207,
+            0.0020324010401964188,
+            -0.007081086281687021,
+            0.011337397620081902,
+            0.015598498284816742,
+            0.013137592934072018,
+            -0.022885456681251526,
+            0.013913208618760109,
+            -0.010762867517769337,
+            -0.015004816465079784,
+            -0.008986611850559711,
+            0.006970967631787062,
+            -0.0020946417935192585,
+            -0.006980543024837971,
+            0.01155763491988182,
+            0.0011813779128715396,
+            -0.030354350805282593,
+            0.032997190952301025,
+            -0.0037033266853541136,
+            -0.03060331381857395,
+            -0.024685652926564217,
+            0.008584440685808659,
+            0.02771151252090931,
+            0.0022203202825039625,
+            -0.004696785472333431,
+            0.03182898089289665,
+            -0.010580932721495628,
+            -0.0029899515211582184,
+            -0.03380153328180313,
+            -0.0034280307590961456,
+            -0.012342826463282108,
+            -0.020893752574920654,
+            -0.012294949032366276,
+            0.00763167766854167,
+            -0.007464106194674969,
+            -0.011634238995611668,
+            0.01805940270423889,
+            -0.01686246506869793,
+            0.002427390543743968,
+            0.003870897926390171,
+            -0.0024441475979983807,
+            0.021161865442991257,
+            -0.017427420243620872,
+            0.0018133613048121333,
+            -0.0035692695528268814,
+            -0.02188960462808609,
+            0.006755518727004528,
+            -0.001415977836586535,
+            -0.03192473575472832,
+            -0.01861478202044964,
+            -0.020146863535046577,
+            -0.0251452773809433,
+            0.006592735182493925,
+            0.009202060289680958,
+            0.015339959412813187,
+            0.005544217303395271,
+            0.01786789298057556,
+            -0.005458037834614515,
+            -0.005525066517293453,
+            -0.027883870527148247,
+            0.001639805268496275,
+            0.009570717811584473,
+            0.014219624921679497,
+            -0.024436689913272858,
+            -0.010542631149291992,
+            0.010868198238313198,
+            0.02162148989737034,
+            -0.004100710153579712,
+            0.014899485744535923,
+            0.014420710504055023,
+            -0.0241877268999815,
+            -4.5165711526351515e-06,
+            -0.013645094819366932,
+            -0.0254133902490139,
+            -0.002503994619473815,
+            -0.0002746972895693034,
+            -0.004031287506222725,
+            -0.015311232767999172,
+            0.015004816465079784,
+            0.010523480363190174,
+            0.01631666161119938,
+            -0.02464734949171543,
+            -0.004459791351109743,
+            -0.004115073475986719,
+            -0.013999388553202152,
+            -0.021678943186998367,
+            0.012543912045657635,
+            -0.026351790875196457,
+            0.014114294201135635,
+            -0.001409993157722056,
+            0.020319221541285515,
+            0.01978299394249916,
+            0.017561476677656174,
+            -0.0019031316041946411,
+            -0.0026955048087984324,
+            0.009024913422763348,
+            0.0071146003901958466,
+            -0.016192179173231125,
+            0.015406987629830837,
+            -0.008503048680722713,
+            -0.0061283232644200325,
+            0.012266222387552261,
+            -0.008823827840387821,
+            0.002645233180373907,
+            0.014047265984117985,
+            -0.0059655397199094296,
+            0.0009988448582589626,
+            -0.008388142101466656,
+            -0.004206040408462286,
+            -0.009479749947786331,
+            0.007468894124031067,
+            -0.03330360725522041,
+            0.013128018006682396,
+            -0.0036722063086926937,
+            0.0033418512903153896,
+            -0.000942588783800602,
+            -0.007559861522167921,
+            -0.02093205414712429,
+            -0.008273236453533173,
+            -0.014324955642223358,
+            -0.002482449635863304,
+            -0.012065136805176735,
+            0.01939997263252735,
+            0.009139820002019405,
+            -0.005496339872479439,
+            0.009364844299852848,
+            0.006918302271515131,
+            -0.019036104902625084,
+            -0.0004512456653174013,
+            0.02317272312939167,
+            0.018183885142207146,
+            -0.00739229004830122,
+            0.03326530382037163,
+            -0.012026834301650524,
+            0.015081420540809631,
+            0.014286654070019722,
+            0.01830836571753025,
+            -0.006353347562253475,
+            -0.030507558956742287,
+            0.017896618694067,
+            0.020070258527994156,
+            -0.020759694278240204,
+            -0.013242923654615879,
+            0.006578371860086918,
+            0.042974866926670074,
+            -0.007085873745381832,
+            0.016086848452687263,
+            0.0012603758368641138,
+            -0.006827335339039564,
+            -0.011145887896418571,
+            -0.011193765327334404,
+            0.06863722205162048,
+            0.023364232853055,
+            0.013424858450889587,
+            -0.007124175783246756,
+            0.0022011692635715008,
+            -0.02112356387078762,
+            -0.001406402327120304,
+            0.0026428394485265017,
+            0.002515963977202773,
+            -0.008752011694014072,
+            -0.025011219084262848,
+            0.01916058547794819,
+            0.030450105667114258,
+            0.009163758717477322,
+            -0.011270369403064251,
+            -0.013874907046556473,
+            0.02303866483271122,
+            0.002209547907114029,
+            0.017341239377856255,
+            0.011797022074460983,
+            0.001982129644602537,
+            -0.03765088692307472,
+            0.003921169321984053,
+            0.015033543109893799,
+            0.0038852612487971783,
+            0.0036434796638786793,
+            0.0078567024320364,
+            0.012400278821587563,
+            0.011212916113436222,
+            0.0034280307590961456,
+            0.027979625388979912,
+            -0.026677357032895088,
+            -0.0019486153032630682,
+            0.0007307307096198201,
+            -0.0005475992220453918,
+            0.009015338495373726,
+            0.013721698895096779,
+            -0.015196327120065689,
+            -0.011174614541232586,
+            0.0011562422150745988,
+            0.0065592210739851,
+            -0.02284715510904789,
+            -0.01091607566922903,
+            0.01414302084594965,
+            -0.01984044723212719,
+            0.011816173791885376,
+            -0.0015763675328344107,
+            0.0006122338818386197,
+            -0.02323017455637455,
+            -0.0008929158793762326,
+            0.006683702580630779,
+            0.010935226455330849,
+            -0.007397077511996031,
+            0.022368380799889565,
+            -0.024111121892929077,
+            -0.01942870020866394,
+            0.017638079822063446,
+            -0.009747864678502083,
+            0.013156743720173836,
+            0.014669673517346382,
+            -0.013989812694489956,
+            -0.003739234758540988,
+            0.0007684343145228922,
+            -0.004940960556268692,
+            -0.010466027073562145,
+            0.00400495482608676,
+            0.0021664579398930073,
+            -0.005735727492719889,
+            0.002329241717234254,
+            0.017293361946940422,
+            -0.011548059061169624,
+            -0.01628793403506279,
+            -0.00590329896658659,
+            0.012400278821587563,
+            -0.000718761351890862,
+            0.018049826845526695,
+            0.01587618701159954,
+            -0.005917662288993597,
+            0.009834043681621552,
+            -0.022808853536844254,
+            0.01078201923519373,
+            0.008167905732989311,
+            -0.008498260751366615,
+            0.006769882049411535,
+            -0.0022670007310807705,
+            -0.004732693545520306,
+            0.03311209753155708,
+            0.020070258527994156,
+            0.007928518578410149,
+            -0.01283117663115263,
+            -0.005888935644179583,
+            -0.026792263612151146,
+            0.0003776339872274548,
+            -0.0028056229930371046,
+            -0.011672540567815304,
+            -0.026524148881435394,
+            0.011528908275067806,
+            0.002091050846502185,
+            -0.004423883277922869,
+            -0.020702242851257324,
+            0.013740849681198597,
+            0.015627223998308182,
+            0.0005272512789815664,
+            0.002843925030902028,
+            0.015167600475251675,
+            0.010657536797225475,
+            0.018298789858818054,
+            -0.010504329577088356,
+            -0.0056447600945830345,
+            0.001198135083541274,
+            -0.026524148881435394,
+            0.0036722063086926937,
+            -0.006085233297199011,
+            0.003061767667531967,
+            0.01786789298057556,
+            -0.011481030844151974,
+            -0.01989790052175522,
+            0.003468726761639118,
+            0.0032987615559250116,
+            -0.01935209520161152,
+            0.014880334958434105,
+            -0.023249326273798943,
+            -0.0004925400717183948,
+            -0.030430955812335014,
+            -0.0060325684025883675,
+            -0.002427390543743968,
+            0.0017834377940744162,
+            -0.0049792625941336155,
+            0.0019569939468055964,
+            -0.015435714274644852,
+            0.010829896666109562,
+            0.03188643231987953,
+            0.010284092277288437,
+            -0.0012071121018379927,
+            0.001107766292989254,
+            -0.0043161590583622456,
+            -0.006841698195785284,
+            -0.01139485090970993,
+            -0.009786166250705719,
+            0.01299396064132452,
+            -0.0058793602511286736,
+            0.02495376579463482,
+            -0.009283452294766903,
+            0.0024513292592018843,
+            0.006846486125141382,
+            0.008708922192454338,
+            -0.011059708893299103,
+            -0.018298789858818054,
+            0.00947017502039671,
+            -0.011864051222801208,
+            0.00815833080559969,
+            -0.008929158560931683,
+            -0.011021406389772892,
+            0.0016050940612331033,
+            0.0032245514448732138,
+            -0.012649241834878922,
+            -0.014707976020872593,
+            -0.005333556327968836,
+            -0.0006906333146616817,
+            -0.005802756175398827,
+            0.00834984052926302,
+            -0.008163117803633213,
+            -0.008373779244720936,
+            -0.005391009617596865,
+            -0.0011041754623875022,
+            -0.0023172723595052958,
+            0.0032413084991276264,
+            0.007765734568238258,
+            0.020051108673214912,
+            0.00900576263666153,
+            0.012505609542131424,
+            0.023268477991223335,
+            0.01513887383043766,
+            0.010523480363190174,
+            -0.008301963098347187,
+            0.01948615349829197,
+            -0.006243229378014803,
+            0.004411913920193911,
+            0.022828003391623497,
+            0.010829896666109562,
+            -0.00691351480782032,
+            0.02757745422422886,
+            -0.010150035843253136,
+            -0.024915464222431183,
+            -0.01917973719537258,
+            0.010858622379601002,
+            -0.02413027361035347,
+            0.003813444869592786,
+            -0.004356855060905218,
+            -0.022253474220633507,
+            0.01174914464354515,
+            0.01596236787736416,
+            0.021085262298583984,
+            0.001044927048496902,
+            -0.004809297621250153,
+            0.0007546694832853973,
+            0.016115576028823853,
+            0.01276414841413498,
+            -0.005836270283907652,
+            0.0012879054993391037,
+            -0.003830202156677842,
+            0.011433153413236141,
+            0.010734140872955322,
+            -0.0018672235310077667,
+            0.007306110579520464,
+            -0.001930661266669631,
+            0.0016565624391660094,
+            0.0016936674946919084,
+            0.011825748719274998,
+            -0.005946388468146324,
+            -0.001229255460202694,
+            0.0012376339873299003,
+            0.011978956870734692,
+            0.0015727767022326589,
+            -0.00798118393868208,
+            0.015895338729023933,
+            0.011337397620081902,
+            -0.014717550948262215,
+            0.01916058547794819,
+            -0.010063855908811092,
+            -0.003739234758540988,
+            0.0006499374285340309,
+            0.015636799857020378,
+            0.004601030144840479,
+            0.01759977824985981,
+            -0.0032077941577881575,
+            0.008139179088175297,
+            -0.015158024616539478,
+            -0.020434128120541573,
+            -0.04672846570611,
+            0.008617954328656197,
+            -0.0020431734155863523,
+            -0.006884788163006306,
+            0.0019498122856020927,
+            0.01901695318520069,
+            0.013300376944243908,
+            0.009417509660124779,
+            -0.0034591511357575655,
+            0.014746277593076229,
+            -0.018021101132035255,
+            -0.019218038767576218,
+            -0.015943216159939766,
+            0.00912545621395111,
+            0.0018001949647441506,
+            -0.002961224876344204,
+            -0.005630397237837315,
+            0.020568184554576874,
+            -0.006420376244932413,
+            0.011184190399944782,
+            -3.63569961336907e-05,
+            -0.018442422151565552,
+            -0.007382714655250311,
+            -0.004452609922736883,
+            0.00857965275645256,
+            0.0029468617867678404,
+            0.022387530654668808,
+            0.0116821164265275,
+            0.00011879610974574462,
+            0.010475602932274342,
+            -0.010150035843253136,
+            -0.0066884905099868774,
+            0.016594350337982178,
+            0.013578066602349281,
+            0.0002670668181963265,
+            0.012361977249383926,
+            0.014353682287037373,
+            0.004675240255892277,
+            0.0019426306243985891,
+            0.01590491458773613,
+            0.05787435173988342,
+            0.018902046605944633,
+            0.003605177626013756,
+            -0.008440807461738586,
+            0.008120028302073479,
+            0.004897871054708958,
+            0.013740849681198597,
+            -0.00372726540081203,
+            -0.0234982892870903,
+            0.006832122802734375,
+            0.010236214846372604,
+            -0.01948615349829197,
+            -0.009958525188267231,
+            0.006022992543876171,
+            0.00836420338600874,
+            -0.015857037156820297,
+            -0.0004030689306091517,
+            -0.011844899505376816,
+            -0.016086848452687263,
+            0.0004171329492237419,
+            0.013884482905268669,
+            -0.023440835997462273,
+            -0.0011951427441090345,
+            -0.0009521642932668328,
+            -0.008215783163905144,
+            -0.009556354023516178,
+            0.0005706402589567006,
+            -0.0021832152269780636,
+            -0.018480725586414337,
+            -0.0022059569600969553,
+            -0.00936005637049675,
+            -0.007444955408573151,
+            0.022636493667960167,
+            -0.0102457907050848,
+            -0.006109172478318214,
+            -0.02213856764137745,
+            -0.023957913741469383,
+            0.01784874126315117,
+            -0.016249632462859154,
+            0.003097675973549485,
+            -0.018155157566070557,
+            0.002120974473655224,
+            -0.019036104902625084,
+            0.01677628420293331,
+            0.010389422997832298,
+            -0.01942870020866394,
+            0.02131507359445095,
+            0.006511343643069267,
+            0.004809297621250153,
+            -0.012649241834878922,
+            -0.002219123300164938,
+            -0.012314099818468094,
+            0.0088142529129982,
+            0.009795742109417915,
+            0.010647961869835854,
+            0.008680195547640324,
+            0.008282812312245369,
+            0.006865637376904488,
+            0.005788392852991819,
+            -0.012697120197117329,
+            -0.0401788204908371,
+            0.01882544346153736,
+            0.016077272593975067,
+            -0.01628793403506279,
+            0.016278358176350594,
+            -0.013865331187844276,
+            -0.020012805238366127,
+            -0.01378872711211443,
+            -0.008689770475029945,
+            -0.014583494514226913,
+            0.011825748719274998,
+            0.01601981930434704,
+            -0.011366124264895916,
+            0.00036416843067854643,
+            0.006372498814016581,
+            0.016173027455806732,
+            -0.009546778164803982,
+            0.005979903042316437,
+            -0.0015272931195795536,
+            -0.01049475371837616,
+            0.01593364030122757,
+            -0.016986945644021034,
+            -0.006248017307370901,
+            0.004277857020497322,
+            -0.022579040378332138,
+            -0.018672235310077667,
+            -0.011461880058050156,
+            0.014947363175451756,
+            -0.0052282256074249744,
+            -0.02194705791771412,
+            -0.0046608769334852695,
+            -0.03539106622338295,
+            0.008459959179162979,
+            0.00566391134634614,
+            -0.0017786500975489616,
+            -0.015541044995188713,
+            0.036712486296892166,
+            0.004751844331622124,
+            -0.02047242969274521,
+            0.005027140025049448,
+            -0.02041497640311718,
+            0.013807878829538822,
+            0.009518052451312542,
+            -0.017149729654192924,
+            0.002458510920405388,
+            0.0007905776146799326,
+            -0.0023807098623365164,
+            -0.00625759270042181,
+            0.004285038448870182,
+            0.004735087510198355,
+            -0.01896907575428486,
+            -0.004124648869037628,
+            0.015464440919458866,
+            -0.0033442452549934387,
+            -0.010513904504477978,
+            -0.011835324577987194,
+            -0.0037751428317278624,
+            0.0025662353727966547,
+            -0.0037631734739989042,
+            -0.0003393319493625313,
+            -0.01866265945136547,
+            -0.01650817133486271,
+            0.034529272466897964,
+            0.0020754907745867968,
+            0.017590202391147614,
+            -0.00494574848562479,
+            -0.0015979124000295997,
+            -0.008694558404386044,
+            0.015339959412813187,
+            0.013874907046556473,
+            0.011471454985439777,
+            0.0029708005022257566,
+            0.009436660446226597,
+            0.01773383468389511,
+            0.00483802380040288,
+            0.01869138516485691,
+            -0.009168545715510845,
+            -0.00888128113001585,
+            -0.011385275982320309,
+            0.029090384021401405,
+            0.009522839449346066,
+            -0.007023632992058992,
+            -0.025394240394234657,
+            -0.0019905080553144217,
+            0.009460599161684513,
+            0.024168575182557106,
+            -0.018767990171909332,
+            -0.028898874297738075,
+            -0.013204621151089668,
+            0.01418132334947586,
+            0.01549316756427288,
+            -0.014085568487644196,
+            -0.0022394713014364243,
+            0.011241642758250237,
+            0.023823857307434082,
+            -0.027826417237520218,
+            0.00936005637049675,
+            -0.0033298819325864315,
+            0.0037320530973374844,
+            -0.01939997263252735,
+            -0.02476225607097149,
+            0.00017475297499913722,
+            0.0034040920436382294,
+            0.009111093357205391,
+            -0.015301656909286976,
+            0.0017211970407515764,
+            0.020759694278240204,
+            -0.009628170169889927,
+            0.015406987629830837,
+            -0.0007564648985862732,
+            -0.010418149642646313,
+            0.0013369799125939608,
+            -0.005323980934917927,
+            -0.009814892895519733,
+            0.015560195781290531,
+            0.005414948333054781,
+            0.00428982637822628,
+            0.002503994619473815,
+            -0.00953720323741436,
+            -0.003291579894721508,
+            -0.008981823921203613,
+            -0.010284092277288437,
+            0.015110147185623646,
+            0.013769576326012611,
+            -0.020644789561629295,
+            0.024666501209139824,
+            -0.0029037718195468187,
+            -0.0049002645537257195,
+            -0.00507022999227047,
+            -0.008459959179162979,
+            0.0170827005058527,
+            0.0006918302387930453,
+            -0.00016442938067484647,
+            0.015282506123185158,
+            0.0038996245712041855,
+            0.001734363380819559,
+            -0.017178455367684364,
+            -0.007047571707516909,
+            -0.00023445025726687163,
+            0.005917662288993597,
+            0.0065257069654762745,
+            0.016661379486322403,
+            -0.006310258060693741,
+            -0.01014045998454094,
+            0.007023632992058992,
+            0.00955156609416008,
+            0.015924064442515373,
+            0.013022687286138535,
+            0.014449437148869038,
+            0.018767990171909332,
+            0.019189313054084778,
+            -0.008412080816924572,
+            -0.01727421209216118,
+            0.017235908657312393,
+            -0.008139179088175297,
+            -0.003289185930043459,
+            -0.04117467254400253,
+            0.01705397479236126,
+            0.020817147567868233,
+            -0.02137252688407898,
+            0.001020988216623664,
+            -0.004663270898163319,
+            -0.007784885820001364,
+            -0.021602340042591095,
+            0.018155157566070557,
+            -0.004852387122809887,
+            0.0019210857572034001,
+            0.004888295195996761,
+            -0.01820303499698639,
+            0.005855421535670757,
+            0.023574894294142723,
+            -0.009000974707305431,
+            0.004907446447759867,
+            -0.0021772305481135845,
+            -0.003554906230419874,
+            -0.012917356565594673,
+            0.006597523111850023,
+            -0.02457074634730816,
+            0.023670649155974388,
+            0.018193459138274193,
+            0.011002255603671074,
+            -0.007952457293868065,
+            0.007473681587725878,
+            -0.015349534340202808,
+            -0.0027266251854598522,
+            -0.00199888669885695,
+            0.014104719273746014,
+            0.002975588198751211,
+            0.00817269366234541,
+            0.0058697848580777645,
+            0.002906165784224868,
+            0.0032748228404670954,
+            -0.018241336569190025,
+            0.016144301742315292,
+            -0.006147474516183138,
+            0.007947669364511967,
+            0.013635518960654736,
+            0.0026931108441203833,
+            -0.009599443525075912,
+            -0.010312818922102451,
+            -0.004821266978979111,
+            0.006592735182493925,
+            0.004931385163217783,
+            0.011911928653717041,
+            0.009977675974369049,
+            0.009824467822909355,
+            -0.02503037080168724,
+            0.012879054062068462,
+            0.0032556718215346336,
+            -0.03282483294606209,
+            -0.002697898540645838,
+            0.0016972583252936602,
+            -0.0030761309899389744,
+            0.010255365632474422,
+            0.02317272312939167,
+            -0.017504023388028145,
+            -0.009206848219037056,
+            0.01801152527332306,
+            0.005199499428272247,
+            -0.016077272593975067,
+            -0.016852889209985733,
+            0.007377926725894213,
+            0.009901071898639202,
+            0.011098010465502739,
+            -0.014985665678977966,
+            0.009077578783035278,
+            0.007052359636873007,
+            0.0254133902490139,
+            -0.03085227683186531,
+            0.018902046605944633,
+            0.012266222387552261,
+            0.0169390682131052,
+            -0.00888128113001585,
+            -0.010896924883127213,
+            0.0008031455217860639,
+            0.012888629920780659,
+            -0.0017571052303537726,
+            0.015598498284816742,
+            -0.002170048886910081,
+            0.005256952252238989,
+            -0.02368979901075363,
+            -0.005041503347456455,
+            -0.0026404454838484526,
+            0.023057816550135612,
+            0.01485160831362009,
+            0.02661990374326706,
+            0.006233653984963894,
+            -0.012007683515548706,
+            0.010743716731667519,
+            -0.006429951637983322,
+            0.0003551913832779974,
+            -0.021219318732619286,
+            -0.00039409188320860267,
+            -0.019610634073615074,
+            -0.003442394081503153,
+            -0.016900766640901566,
+            0.03609965369105339,
+            0.02047242969274521,
+            0.014123870059847832,
+            0.025509145110845566,
+            -9.672756277723238e-05,
+            0.008670619688928127,
+            -0.004225191660225391,
+            -0.0018923592288047075,
+            -0.015253779478371143,
+            0.00774658378213644,
+            -0.006789033301174641,
+            0.001808573491871357,
+            0.008622742258012295,
+            -0.004129436798393726,
+            0.016632651910185814,
+            -0.022483285516500473,
+            0.014411135576665401,
+            0.01378872711211443,
+            0.014315379783511162,
+            0.016603926196694374,
+            -0.010284092277288437,
+            -0.0031862494070082903,
+            -0.0028223800472915173,
+            0.001198135083541274,
+            -0.009039277210831642,
+            -0.0023711344692856073,
+            -0.0028918024618178606,
+            -0.0088142529129982,
+            -0.005951176397502422,
+            0.022387530654668808,
+            0.0024968129582703114,
+            0.018298789858818054,
+            0.0024477385450154543,
+            0.015885762870311737,
+            0.005673486739397049,
+            0.005783604923635721,
+            -0.0157134048640728,
+            -0.0005550800706259906,
+            0.013482310809195042,
+            0.0021999722812324762,
+            -0.0031790677458047867,
+            -0.021813001483678818,
+            -0.021659793332219124,
+            0.0007851914269849658,
+            0.00673636794090271,
+            0.02162148989737034,
+            0.0017666807398200035,
+            -0.015272930264472961,
+            0.0008462352561764419,
+            -0.021295923739671707,
+            -0.0243217833340168,
+            -0.014564343728125095,
+            0.0011191371595486999,
+            0.029856424778699875,
+            0.0016350174555554986,
+            -0.009010550566017628,
+            0.009407933801412582,
+            -0.021985359489917755,
+            -0.0023615588434040546,
+            0.002669172128662467,
+            -0.0204915814101696,
+            0.009202060289680958,
+            -0.031522564589977264,
+            0.015474015846848488,
+            -0.013463160023093224,
+            0.00035459292121231556,
+            0.022310927510261536,
+            -0.005165984854102135,
+            -0.0008492275956086814,
+            -0.011385275982320309,
+            0.00044556023203767836,
+            0.0159910935908556,
+            -0.004129436798393726,
+            -0.004514850676059723,
+            -0.0014111900236457586,
+            0.003509422531351447,
+            -0.0016745164757594466,
+            -0.005941601004451513,
+            0.0019881143234670162,
+            0.0011125539895147085,
+            -0.019581908360123634,
+            0.0006631037103943527,
+            0.018404120579361916,
+            -0.012974809855222702,
+            -0.03246096149086952,
+            0.01653689704835415,
+            -0.02405366860330105,
+            -0.010801170021295547,
+            -0.0035261798184365034,
+            0.017006097361445427,
+            0.0022658039815723896,
+            -0.007492832839488983,
+            0.020721392706036568,
+            0.007933305576443672,
+            0.01008300669491291,
+            0.004426277242600918,
+            0.003159916726872325,
+            -0.0002832254976965487,
+            -0.0026428394485265017,
+            -5.97533971813391e-06,
+            0.0064634657464921474,
+            0.015971941873431206,
+            -0.018375394865870476,
+            0.027883870527148247,
+            -0.0070284209214150906,
+            -0.0034280307590961456,
+            0.0043233404867351055,
+            -0.006808184087276459,
+            -0.008368991315364838,
+            0.010877774097025394,
+            -0.02721358649432659,
+            0.017944496124982834,
+            -0.015090996399521828,
+            -0.011921503581106663,
+            -0.011883202008903027,
+            0.014229200780391693,
+            0.006664551794528961,
+            -0.01951487921178341,
+            0.005740515422075987,
+            -0.01490906160324812,
+            -0.01596236787736416,
+            0.002160473261028528,
+            -0.004100710153579712,
+            0.014928212389349937,
+            -0.0116821164265275,
+            -0.01164381392300129,
+            0.00536228297278285,
+            -0.019859597086906433,
+            0.008321113884449005,
+            -0.0077705224975943565,
+            0.009403145872056484,
+            -0.025892166420817375,
+            0.011576785705983639,
+            0.015110147185623646,
+            -0.004735087510198355,
+            -0.017197607085108757,
+            0.015857037156820297,
+            0.016469869762659073,
+            -0.004447821993380785,
+            -0.011481030844151974,
+            0.00822535902261734,
+            0.013147168792784214,
+            0.0025518720503896475,
+            -0.004984050523489714,
+            -0.017130577936768532,
+            -0.01126079447567463,
+            0.00923557486385107,
+            -0.0016074879094958305,
+            -0.036655034869909286,
+            -0.01672840677201748,
+            -0.02131507359445095,
+            -0.011433153413236141,
+            -0.009474962018430233,
+            0.006003841757774353,
+            0.006607098504900932,
+            -0.002515963977202773,
+            -0.02738594450056553,
+            -0.00035638833651319146,
+            0.004857175052165985,
+            -0.014813306741416454,
+            -0.001930661266669631,
+            -0.003621934913098812,
+            -0.0036985387559980154,
+            -0.0048164790496230125,
+            -0.005060654599219561,
+            0.013769576326012611,
+            -0.003739234758540988,
+            0.01239070389419794,
+            0.011011830531060696,
+            -0.00787106528878212,
+            -0.0026619904674589634,
+            -0.03211624547839165,
+            -0.014449437148869038,
+            0.008182269521057606,
+            0.0068800002336502075,
+            -0.002951649483293295,
+            -0.0005161795997992158,
+            0.008321113884449005,
+            0.00029130480834282935,
+            -0.005084593314677477,
+            0.005237801466137171,
+            0.013195046223700047,
+            -0.013501462526619434,
+            0.024858010932803154,
+            0.010456451214849949,
+            -0.00863710604608059,
+            -0.012534336186945438,
+            -0.01078201923519373,
+            0.018509451299905777,
+            0.009068002924323082,
+            0.017331663519144058,
+            -0.0023519834503531456,
+            0.01247688289731741,
+            0.0014231594977900386,
+            0.0262368842959404,
+            -0.006482616998255253,
+            -0.006808184087276459,
+            -0.010839471593499184,
+            0.0021568825468420982,
+            -0.02778811566531658,
+            0.00447415467351675,
+            0.01478458009660244,
+            -0.009987251833081245,
+            -0.002028810093179345,
+            0.006406012922525406,
+            -0.006468253675848246,
+            0.011548059061169624,
+            -0.009977675974369049,
+            0.0024800559040158987,
+            -0.017551900818943977,
+            -0.0020096590742468834,
+            0.016546472907066345,
+            0.0010138065554201603,
+            0.006640612613409758,
+            0.04749450460076332,
+            -0.009723925031721592,
+            -0.009393570944666862,
+            0.026102827861905098,
+            -1.738365608616732e-05,
+            0.019993655383586884,
+            -0.0006035560509189963,
+            0.0010419347090646625,
+            0.004675240255892277,
+            0.013645094819366932,
+            0.018538178876042366,
+            0.005304829683154821,
+            0.00196776632219553,
+            -0.01084904745221138,
+            -0.003356214612722397,
+            -0.011605512350797653,
+            -0.033514268696308136,
+            -0.005419735796749592,
+            0.003049798309803009,
+            -0.0069422414526343346,
+            0.007579012308269739,
+            -0.01847114972770214,
+            0.007138539105653763,
+            0.02483886107802391,
+            -0.017006097361445427,
+            -0.015158024616539478,
+            -0.008646680973470211,
+            -0.006487404927611351,
+            0.0004410716937854886,
+            -0.01696779578924179,
+            0.001875602058134973,
+            -0.007076298352330923,
+            0.0036817817017436028,
+            0.005103744100779295,
+            -0.0135972173884511,
+            0.022004511207342148,
+            0.0030210718978196383,
+            0.020721392706036568,
+            -0.006645400542765856,
+            -0.0001726583286654204,
+            0.005932025611400604,
+            -0.010935226455330849,
+            -0.021468281745910645,
+            0.004325734451413155,
+            0.01049475371837616,
+            -0.02162148989737034,
+            -0.022253474220633507,
+            -0.0029109534807503223,
+            0.0015655950410291553,
+            -0.01030324399471283,
+            0.006990118883550167,
+            -0.0016421991167590022,
+            0.0126396669074893,
+            -0.00013046625826973468,
+            -0.0069422414526343346,
+            0.016527321189641953,
+            0.02175554819405079,
+            -0.004735087510198355,
+            -0.0029444678220897913,
+            -0.0004012735153082758,
+            -0.015550619922578335,
+            0.011662965640425682,
+            0.008866917341947556,
+            -0.018681811168789864,
+            0.015272930264472961,
+            0.017915770411491394,
+            0.016785860061645508,
+            0.0015955185517668724,
+            -0.014516465365886688,
+            -0.0005520877311937511,
+            -0.0038948366418480873,
+            -0.00017789493722375482,
+            -0.01549316756427288,
+            -0.005098956637084484,
+            -0.004912233911454678,
+            -0.014803730882704258,
+            0.0020802784711122513,
+            -0.026581602171063423,
+            -0.00043299238313920796,
+            -0.011940655298531055,
+            -0.0003560890909284353,
+            0.020261768251657486,
+            -0.019218038767576218,
+            -0.0006852470687590539,
+            -0.011308671906590462,
+            -0.018356243148446083,
+            0.013070564717054367,
+            -0.005089380778372288,
+            -0.021927906200289726,
+            0.0066118864342570305,
+            0.007344412617385387,
+            -0.008910007774829865,
+            -0.014640947803854942,
+            0.007985970936715603,
+            -0.012553486973047256,
+            0.0009617398027330637,
+            0.020376674830913544,
+            0.016718832775950432,
+            0.009518052451312542,
+            -0.00857965275645256,
+            -0.004433458670973778,
+            0.011739569716155529,
+            0.01689119078218937,
+            0.004895477090030909,
+            0.02694547176361084,
+            -0.024743104353547096,
+            -0.029090384021401405,
+            0.04308977350592613,
+            0.007009269669651985,
+            -0.009546778164803982,
+            0.013415282592177391,
+            -0.024628199636936188,
+            0.009599443525075912,
+            0.033322758972644806,
+            0.008924370631575584,
+            -0.017312513664364815,
+            0.014832457527518272,
+            0.002272985642775893,
+            -0.021525735035538673,
+            0.0035237858537584543,
+            0.005410160403698683,
+            -0.015416563488543034,
+            0.005395797081291676,
+            0.016948644071817398,
+            0.006640612613409758,
+            0.01286947913467884,
+            -0.00235318043269217,
+            0.008067362941801548,
+            0.010255365632474422,
+            0.004277857020497322,
+            -0.007947669364511967,
+            -0.006951816845685244,
+            0.009316966868937016,
+            -0.039834100753068924,
+            -0.01485160831362009,
+            -0.013128018006682396,
+            -0.0052138627506792545,
+            -0.007847126573324203,
+            0.021659793332219124,
+            -0.013405706733465195,
+            -0.01874883845448494,
+            0.02164064161479473,
+            0.024398386478424072,
+            -0.006980543024837971,
+            -0.004665664862841368,
+            0.0037200837396085262,
+            -0.016517747193574905,
+            -0.0020060683600604534,
+            0.005491552408784628,
+            0.035046350210905075,
+            -0.0072773839347064495,
+            -0.0060325684025883675,
+            0.008699346333742142,
+            -0.014545192010700703,
+            0.02956916019320488,
+            0.005285678897053003,
+            -0.006329408846795559,
+            -0.00924993772059679,
+            -0.022119417786598206,
+            -0.0030019208788871765,
+            -0.017130577936768532,
+            -0.02183215133845806,
+            0.0038804735522717237,
+            -0.0010461239144206047,
+            -0.0008893250487744808,
+            0.004881113767623901,
+            0.0022394713014364243,
+            0.018241336569190025,
+            -0.009800529107451439,
+            0.011117161251604557,
+            -0.018432848155498505,
+            -0.0024501322768628597,
+            0.006812972016632557,
+            -0.0017630899092182517,
+            -0.010772443376481533,
+            -0.021027809008955956,
+            0.023364232853055,
+            -0.006894363556057215,
+            0.012496034614741802,
+            -0.01920846290886402,
+            0.013386555947363377,
+            -0.0166709553450346,
+            -0.002225107979029417,
+            0.0032293391413986683,
+            -0.005271315574645996,
+            -0.004964899271726608,
+            -0.003289185930043459,
+            0.01939997263252735,
+            -0.011567209847271442,
+            -0.012084287591278553,
+            0.0251452773809433,
+            0.0241877268999815,
+            0.004268281627446413,
+            -0.01257263869047165,
+            0.001366903306916356,
+            0.007263020612299442,
+            0.0028941964264959097,
+            0.0016015032306313515,
+            0.002992345253005624,
+            0.002399860881268978,
+            0.002276576356962323,
+            0.006712429225444794,
+            0.01978299394249916,
+            -0.016766710206866264,
+            0.0060517191886901855,
+            0.023440835997462273,
+            0.01874883845448494,
+            -0.006908726878464222,
+            -0.002071899827569723,
+            -0.012266222387552261,
+            0.005970327649265528,
+            0.012984384782612324,
+            0.0052138627506792545,
+            0.0015452471561729908,
+            0.0019450244726613164,
+            0.007162477821111679,
+            -0.024302631616592407,
+            0.030181992799043655,
+            0.0028271679766476154,
+            0.014516465365886688,
+            6.553236016770825e-05,
+            0.015177175402641296,
+            -0.013530189171433449,
+            -0.002506388584151864,
+            0.013798302970826626,
+            -0.006248017307370901,
+            0.008790313266217709,
+            0.007574224378913641,
+            -0.005898511037230492,
+            -0.012582213617861271,
+            -0.00044137093937024474,
+            -0.0036674183793365955,
+            -0.00715290242806077,
+            -0.0026165067683905363,
+            -0.002841531066223979,
+            0.0013214197242632508,
+            -0.008316325955092907,
+            0.006410800851881504,
+            -0.007344412617385387,
+            0.007306110579520464,
+            -0.002270591678097844,
+            -0.003770355135202408,
+            0.0008683786145411432,
+            0.025968769565224648,
+            0.00806257501244545,
+            0.03135020285844803,
+            0.003291579894721508,
+            0.0016589562874287367,
+            -0.0008611970115453005,
+            -0.014535617083311081,
+            -0.0262368842959404,
+            0.011346973478794098,
+            -0.02771151252090931,
+            0.00031449549715034664,
+            -0.0032676411792635918,
+            0.003902018303051591,
+            0.012534336186945438,
+            0.011098010465502739,
+            0.009096729569137096,
+            -0.018930774182081223,
+            -0.013568490743637085,
+            -0.002460904885083437,
+            -0.011921503581106663,
+            0.002963618841022253,
+            0.012218344956636429,
+            -0.005118107423186302,
+            0.012792875058948994,
+            -0.0004982255049981177,
+            0.002997133182361722,
+            -0.03699975088238716,
+            -0.012256646528840065,
+            0.00241183047182858,
+            -0.00994895026087761,
+            0.007708281744271517,
+            0.009039277210831642,
+            -0.001272345194593072,
+            0.012208769097924232,
+            0.01601981930434704,
+            0.0058697848580777645,
+            -0.0035668755881488323,
+            -0.0023519834503531456,
+            -0.0011442728573456407,
+            -0.017963647842407227,
+            0.012610940262675285,
+            -0.017446570098400116,
+            0.00590329896658659,
+            0.004404732491821051,
+            0.016575198620557785,
+            0.017398692667484283,
+            -0.0053000422194600105,
+            0.020510731264948845,
+            0.0064634657464921474,
+            -0.0066214618273079395,
+            0.027175283059477806,
+            -0.00019285666348878294,
+            -0.006540069822221994,
+            -0.021334225311875343,
+            -0.003413667669519782,
+            0.019725540652871132,
+            -0.010236214846372604,
+            -0.0010856229346245527,
+            0.010734140872955322,
+            0.008479109965264797,
+            -2.2947548131924123e-05,
+            0.01765723153948784,
+            -0.009441448375582695,
+            0.0009192485013045371,
+            0.007315685972571373,
+            0.004950535949319601,
+            0.011232067830860615,
+            0.01174914464354515,
+            0.01740826852619648,
+            0.003947502002120018,
+            0.0023304384667426348,
+            -0.0037583857774734497,
+            0.004866750445216894,
+            0.011672540567815304,
+            0.03675078973174095,
+            0.007861489430069923,
+            0.007124175783246756,
+            -0.007176841143518686,
+            -0.004009742755442858,
+            0.01020748820155859,
+            -0.013415282592177391,
+            -0.013242923654615879,
+            0.015119723044335842,
+            0.005170772783458233,
+            0.032288603484630585,
+            -0.0121608916670084,
+            0.009996827691793442,
+            0.025585750117897987,
+            -0.008368991315364838,
+            -0.004658483434468508,
+            -0.004842811729758978,
+            0.0017319695325568318,
+            0.0024896312970668077,
+            0.007827975787222385,
+            0.0018037857953459024,
+            0.020587336272001266,
+            -0.009666472673416138,
+            -0.031771525740623474,
+            0.028075380250811577,
+            0.013491886667907238,
+            0.017398692667484283,
+            0.02164064161479473,
+            0.0079955467954278,
+            0.012677968479692936,
+            0.00971435010433197,
+            0.005893723573535681,
+            0.00416773883625865,
+            0.01609642431139946,
+            0.0073539880104362965,
+            -0.01767638325691223,
+            0.02054903469979763,
+            0.015483591705560684,
+            0.010753292590379715,
+            -0.02951170690357685,
+            -0.01116503868252039,
+            0.012936507351696491,
+            -0.01203641016036272,
+            -0.01061923522502184,
+            0.009929798543453217,
+            -0.011892776936292648,
+            0.005170772783458233,
+            0.015158024616539478,
+            0.010025554336607456,
+            0.0014123870059847832,
+            -0.02309611812233925,
+            -0.016489019617438316,
+            0.005496339872479439,
+            0.0021341408137232065,
+            0.012563062831759453,
+            0.0126396669074893,
+            0.017226334661245346,
+            -0.004950535949319601,
+            -0.00040037580765783787,
+            -0.017752986401319504,
+            -0.009580292738974094,
+            -0.007784885820001364,
+            0.02752000093460083,
+            -0.02164064161479473,
+            -0.0019330551149323583,
+            0.004105497617274523,
+            -0.02169809490442276,
+            0.0006218093913048506,
+            0.00840250588953495,
+            0.0025949617847800255,
+            -0.005357495043426752,
+            -0.008919582702219486,
+            0.003964259289205074,
+            -0.010753292590379715,
+            0.01414302084594965,
+            0.0011592345545068383,
+            -0.018729688599705696,
+            0.008368991315364838,
+            0.0013776757987216115,
+            -0.01212258916348219,
+            0.019629785791039467,
+            0.01852860301733017,
+            -0.007823187857866287,
+            0.005118107423186302,
+            0.041634295135736465,
+            0.0024992069229483604,
+            -0.022770551964640617,
+            -0.007545498199760914,
+            0.005323980934917927,
+            0.026102827861905098,
+            0.006066082511097193,
+            0.00822535902261734,
+            -0.0002223312621936202,
+            0.0030402229167521,
+            0.0017738622846081853,
+            -0.007406653370708227,
+            0.016211330890655518,
+            -0.00697575556114316,
+            0.004349673166871071,
+            0.014545192010700703,
+            -0.00877116248011589,
+            0.002016840735450387,
+            -0.01866265945136547,
+            0.0021233682055026293,
+            -0.025183578953146935,
+            -0.009546778164803982,
+            -0.003923563286662102,
+            -0.010264941491186619,
+            -0.008340264670550823,
+            0.0025303272996097803,
+            -0.011375700123608112,
+            -0.017398692667484283,
+            -0.010571357794106007,
+            0.01945742592215538,
+            -0.02066393941640854,
+            -0.0008234934648498893,
+            0.0065592210739851,
+            0.019189313054084778,
+            0.012151315808296204,
+            0.0029037718195468187,
+            0.0073539880104362965,
+            0.01561764907091856,
+            0.0017020460218191147,
+            -0.02188960462808609,
+            -0.027692360803484917,
+            0.02424517832696438,
+            0.001624245080165565,
+            0.0013681002892553806,
+            0.001211899914778769,
+            -0.011825748719274998,
+            -0.00985319446772337,
+            0.009048852138221264,
+            0.0006984134088270366,
+            -0.018183885142207146,
+            0.015474015846848488,
+            -0.002918135141953826,
+            -0.0017726654186844826,
+            -0.0031431596726179123,
+            0.014066416770219803,
+            0.014631371945142746,
+            0.0044095199555158615,
+            0.0055490052327513695,
+            0.005683062132447958,
+            0.012065136805176735,
+            0.010580932721495628,
+            -0.019878748804330826,
+            -0.025183578953146935,
+            0.019198887050151825,
+            -0.008459959179162979,
+            0.016852889209985733,
+            0.01382702961564064,
+            0.013635518960654736,
+            -0.028324345126748085,
+            -0.0037320530973374844,
+            -0.024436689913272858,
+            0.004550758749246597,
+            0.0008234934648498893,
+            -0.0019940990023314953,
+            -0.002559053711593151,
+            1.956245796463918e-05,
+            -0.021774698048830032,
+            0.006602310575544834,
+            0.025164427235722542,
+            -0.003890048945322633,
+            -0.011308671906590462,
+            0.02874566614627838,
+            -0.028918026015162468,
+            0.002788865938782692,
+            0.023076968267560005,
+            -0.004907446447759867,
+            0.07740838080644608,
+            0.008728072978556156,
+            0.0031670983880758286,
+            -0.015110147185623646,
+            0.0053000422194600105,
+            0.014487739652395248,
+            0.00834984052926302,
+            0.013817453756928444,
+            -0.00955156609416008,
+            -0.015158024616539478,
+            -0.008282812312245369,
+            0.01187362615019083,
+            0.006554433144629002,
+            -0.0031216146890074015,
+            0.002810410689562559,
+            0.0005098956753499806,
+            -0.010963953100144863,
+            -0.011739569716155529,
+            0.024111121892929077,
+            -0.025777259841561317,
+            -0.00871849711984396,
+            -0.028132833540439606,
+            -0.01366424560546875,
+            -0.00023789145052433014,
+            0.012754572555422783,
+            0.011011830531060696,
+            -0.022119417786598206,
+            -0.013606793247163296,
+            -0.020510731264948845,
+            -0.0020635214168578386,
+            0.018576480448246002,
+            0.0015536256833001971,
+            -0.000921043916605413,
+            -0.002472874242812395,
+            -0.001449492061510682,
+            0.009130244143307209,
+            -0.01295565813779831,
+            0.014344106428325176,
+            -0.006200139410793781,
+            0.0027649272233247757,
+            0.0038014755118638277,
+            0.00733004929497838,
+            0.01343443337827921,
+            -0.0034783021546900272,
+            0.006324620917439461,
+            -0.010753292590379715,
+            0.02361319586634636,
+            -0.01545486506074667,
+            -0.0031240086536854506,
+            -0.0015739736845716834,
+            0.008440807461738586,
+            -0.007325261365622282,
+            0.006760306656360626,
+            -0.00854135025292635,
+            -0.009269089438021183,
+            0.014468587934970856,
+            0.025566598400473595,
+            -0.02399621531367302,
+            0.004878719802945852,
+            -0.022770551964640617,
+            -0.0006918302387930453,
+            -0.007325261365622282,
+            -0.006425164174288511,
+            -0.002341211074963212,
+            -0.0401788204908371,
+            -0.005319193005561829,
+            0.012314099818468094,
+            0.0026284761261194944,
+            -0.01634538732469082,
+            0.009111093357205391,
+            -0.002007265342399478,
+            0.003736840793862939,
+            -0.03994900733232498,
+            0.017130577936768532,
+            0.025068672373890877,
+            -0.015129297971725464,
+            0.007253445219248533,
+            0.005845845676958561,
+            -0.007698706351220608,
+            0.0005371259758248925,
+            0.004426277242600918,
+            0.002757745562121272,
+            -0.02880311943590641,
+            0.020434128120541573,
+            0.018375394865870476,
+            0.007579012308269739,
+            0.004888295195996761,
+            -0.008929158560931683,
+            0.002621294464915991,
+            0.00989149697124958,
+            0.005544217303395271,
+            0.02405366860330105,
+            0.0031431596726179123,
+            -0.0006415588431991637,
+            0.0016374114202335477,
+            -0.0037224777042865753,
+            0.009767015464603901,
+            -0.005970327649265528,
+            0.005740515422075987,
+            -0.017877468839287758,
+            -0.0043329158797860146,
+            -0.0018349061720073223,
+            -0.004172526299953461,
+            -0.007794461213052273,
+            -0.001229255460202694,
+            -0.011844899505376816,
+            0.021410830318927765,
+            0.007377926725894213,
+            0.014210049994289875,
+            -0.01603897102177143,
+            0.0032556718215346336,
+            -0.021334225311875343,
+            -0.0036099653225392103,
+            -0.008311538025736809,
+            0.005946388468146324,
+            0.04404732212424278,
+            0.010006402619183064,
+            0.016661379486322403,
+            0.00033514268579892814,
+            0.020070258527994156,
+            -0.0066884905099868774,
+            -0.0055585806258022785,
+            -0.00047518443898297846,
+            -0.003830202156677842,
+            -0.0032867921981960535,
+            -0.0016050940612331033,
+            -0.008570076897740364,
+            -0.008321113884449005,
+            0.006678914651274681,
+            0.007890216074883938,
+            0.00667412718757987,
+            -0.0214491318911314,
+            0.01830836571753025,
+            -0.0007079889182932675,
+            -0.006037355866283178,
+            0.019763842225074768,
+            -0.0015620043268427253,
+            -0.002276576356962323,
+            -0.010418149642646313,
+            0.008258873596787453,
+            -0.0024082395248115063,
+            0.0469965785741806,
+            -0.009034489281475544,
+            0.02188960462808609,
+            -0.006439527031034231,
+            0.0023771191481500864,
+            0.003736840793862939,
+            0.0009264301043003798,
+            -0.006745943333953619,
+            -0.0021125958301126957,
+            -0.015541044995188713,
+            0.0056016705930233,
+            -0.015215477906167507,
+            0.012342826463282108,
+            -0.016795435920357704,
+            0.00372726540081203,
+            0.02361319586634636,
+            -0.0050558666698634624,
+            -0.00902012549340725,
+            0.0036913573276251554,
+            0.006587947718799114,
+            -0.0028271679766476154,
+            -0.02106611058115959,
+            0.013635518960654736,
+            -0.00193664594553411,
+            0.017427420243620872,
+            -0.007516771554946899,
+            -0.010963953100144863,
+            0.021104414016008377,
+            0.010092582553625107,
+            0.025049520656466484,
+            0.017312513664364815,
+            0.018863745033740997,
+            -0.001241224817931652,
+            0.014966514892876148,
+            -0.020721392706036568,
+            0.002918135141953826,
+            -0.012668393552303314,
+            -0.01989790052175522,
+            -0.0028175923507660627,
+            -0.004445428028702736,
+            0.01116503868252039,
+            -0.008546138182282448,
+            -0.010954378172755241,
+            0.007899791933596134,
+            -0.023345081135630608,
+            0.00324849016033113,
+            -0.008301963098347187,
+            0.017839165404438972,
+            0.0035477245692163706,
+            -0.012658817693591118,
+            -0.008795101195573807,
+            -0.006841698195785284,
+            -0.01174914464354515,
+            -0.020568184554576874,
+            0.03008623793721199,
+            0.01636453904211521,
+            0.015435714274644852,
+            -0.005678274668753147,
+            -0.002378316130489111,
+            -0.008689770475029945,
+            -0.0036267226096242666,
+            0.0008923173882067204,
+            -0.02822858840227127
+          ],
+          "title": "A Switch for Attention"
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "44ddb1e4-8508-4bcf-9d46-226ebb75db2f",
+        "properties": {
+          "page_content": "B Preventing Token Dropping with No-Token-Left-Behind 29\n\n",
+          "keyphrases": [
+            "Preventing Token Dropping",
+            "No-Token-Left-Behind"
+          ],
+          "embedding": [
+            0.025537114590406418,
+            -0.019454315304756165,
+            -0.016811933368444443,
+            0.022433646023273468,
+            -0.0004264498711563647,
+            -0.0028263733256608248,
+            -0.020252350717782974,
+            0.05167718976736069,
+            -0.02958049066364765,
+            0.00320987356826663,
+            -0.031212029978632927,
+            -0.00974489189684391,
+            0.01094194408506155,
+            -0.049123477190732956,
+            -0.010197111405432224,
+            0.038660354912281036,
+            -0.002062698360532522,
+            0.016102569177746773,
+            0.016013899818062782,
+            -0.003059133654460311,
+            0.010631597600877285,
+            0.013460187241435051,
+            -0.01815972663462162,
+            0.039192378520965576,
+            -0.005244862288236618,
+            0.027842549607157707,
+            -0.010347851552069187,
+            0.005479839164763689,
+            -0.04174608737230301,
+            0.018115390092134476,
+            0.008871487341821194,
+            0.009558684192597866,
+            -0.0016282127471640706,
+            -0.052067339420318604,
+            -0.011119285598397255,
+            0.012387273833155632,
+            0.04429979994893074,
+            0.015579413622617722,
+            -0.07313545793294907,
+            -0.028835657984018326,
+            -0.0168474018573761,
+            0.003110119141638279,
+            -0.05912551283836365,
+            0.0050763883627951145,
+            0.019170571118593216,
+            0.025129230692982674,
+            -0.004646336194127798,
+            -0.01648385263979435,
+            -0.0052049607038497925,
+            0.0036554429680109024,
+            0.017237553372979164,
+            -0.042207177728414536,
+            0.011943921446800232,
+            -0.02472134679555893,
+            0.015464141964912415,
+            0.015251332893967628,
+            0.0375608392059803,
+            0.018283864483237267,
+            0.007479359861463308,
+            -0.01447989884763956,
+            0.01440009567886591,
+            0.012804025784134865,
+            0.034670181572437286,
+            0.004735006485134363,
+            -0.0008046851144172251,
+            -0.008299562148749828,
+            -0.011722245253622532,
+            -0.023072073236107826,
+            0.017343956977128983,
+            -0.021405067294836044,
+            -0.050719548016786575,
+            0.02541297674179077,
+            -0.003491402370855212,
+            -0.006645856890827417,
+            0.023781437426805496,
+            0.010489724576473236,
+            0.07838475704193115,
+            0.05848708748817444,
+            0.0006278982618823647,
+            -0.000581900414545089,
+            -0.005510874092578888,
+            0.021937090903520584,
+            -0.001937451190315187,
+            0.014329158701002598,
+            0.041426874697208405,
+            -0.00011402476957300678,
+            -0.046995386481285095,
+            -0.04749194160103798,
+            0.04476088657975197,
+            -0.021334132179617882,
+            -0.0384475439786911,
+            -0.02028781920671463,
+            -0.02064250037074089,
+            -0.04121406748890877,
+            -0.00320987356826663,
+            0.0588417686522007,
+            0.017228685319423676,
+            -0.019383380189538002,
+            -0.022256305441260338,
+            0.005936492700129747,
+            -0.025022825226187706,
+            -0.0026623329613357782,
+            -0.05469198897480965,
+            0.01869175024330616,
+            0.03926331549882889,
+            0.012874961830675602,
+            -0.0023054340854287148,
+            0.009425678290426731,
+            -0.022185368463397026,
+            0.0798034816980362,
+            -0.018815888091921806,
+            0.009212869219481945,
+            -0.020411957055330276,
+            0.04468994960188866,
+            -0.031247496604919434,
+            -0.016785332933068275,
+            -0.00320987356826663,
+            -0.034067220985889435,
+            -0.036638665944337845,
+            0.011669042520225048,
+            0.0056571802124381065,
+            -0.015269066207110882,
+            -0.012617817148566246,
+            0.013229643926024437,
+            -0.027186386287212372,
+            0.008844885975122452,
+            0.027150917798280716,
+            0.019738061353564262,
+            -0.021405067294836044,
+            -0.01984446682035923,
+            -0.014072014018893242,
+            -0.013149840757250786,
+            -0.058628957718610764,
+            0.012724222615361214,
+            -0.04224264249205589,
+            0.019879935309290886,
+            0.019436581060290337,
+            0.011988257057964802,
+            -0.026211010292172432,
+            0.04163968563079834,
+            -0.022912466898560524,
+            0.007377388887107372,
+            0.028392305597662926,
+            0.04745647311210632,
+            -0.012431609444320202,
+            0.009913366287946701,
+            -0.008441435173153877,
+            0.015747888013720512,
+            -0.018620813265442848,
+            0.021511472761631012,
+            0.02498735673725605,
+            0.004351506475359201,
+            0.014630638994276524,
+            0.03674507141113281,
+            0.005346833262592554,
+            0.01222766749560833,
+            -0.007204481400549412,
+            -0.02152920700609684,
+            0.007794140372425318,
+            -0.007452758494764566,
+            0.016155771911144257,
+            0.007075909059494734,
+            0.03532634302973747,
+            -0.00984242931008339,
+            0.024029714986681938,
+            0.006654724013060331,
+            0.03472338244318962,
+            0.013265112414956093,
+            -0.08313749730587006,
+            0.02347995899617672,
+            -0.0027244023513048887,
+            0.031034687533974648,
+            -0.010223712772130966,
+            0.03021891973912716,
+            -0.007612365763634443,
+            -0.020341021940112114,
+            -0.0064995503053069115,
+            -0.00956755131483078,
+            0.016599124297499657,
+            0.011411897838115692,
+            -0.011713378131389618,
+            0.03709975257515907,
+            -0.009053261950612068,
+            -0.024650409817695618,
+            0.022522317245602608,
+            -0.003440416883677244,
+            0.019206037744879723,
+            -0.0175301656126976,
+            -0.008538972586393356,
+            0.009434545412659645,
+            -0.02516469918191433,
+            0.018461205065250397,
+            0.01779617741703987,
+            -0.00274656992405653,
+            -0.01745922863483429,
+            0.0034071654081344604,
+            -0.014905516989529133,
+            -0.025022825226187706,
+            0.01737942546606064,
+            -0.00849907100200653,
+            -0.0026157808024436235,
+            0.0168474018573761,
+            -0.029917439445853233,
+            -0.00881828460842371,
+            0.0021292013116180897,
+            0.021121321246027946,
+            0.0020128211472183466,
+            0.04458354786038399,
+            0.007740938104689121,
+            0.01790258102118969,
+            -0.0017235336126759648,
+            0.0003020340227521956,
+            -0.007173446472734213,
+            0.012502545490860939,
+            -0.019418848678469658,
+            -0.01103061530739069,
+            -0.03181498870253563,
+            0.018053321167826653,
+            -0.022185368463397026,
+            0.039192378520965576,
+            0.03630171716213226,
+            -0.014976453967392445,
+            -0.022717392072081566,
+            -0.0026645497418940067,
+            -0.003846084466204047,
+            -0.017388291656970978,
+            0.018408004194498062,
+            0.025217900052666664,
+            -0.0026157808024436235,
+            -0.00030425077420659363,
+            0.010826672427356243,
+            0.02401198074221611,
+            -0.05298951268196106,
+            0.0066192555241286755,
+            -0.013309448026120663,
+            -0.009700557217001915,
+            -0.034084953367710114,
+            0.010933076962828636,
+            0.019791264086961746,
+            0.01067593228071928,
+            0.013814869336783886,
+            0.044654481112957,
+            -0.07682415097951889,
+            -0.0029549456667155027,
+            -0.03552141785621643,
+            0.06685758382081985,
+            -0.026636630296707153,
+            0.03678053990006447,
+            -0.03527313843369484,
+            -0.0006434156093746424,
+            -0.018141992390155792,
+            0.017672037705779076,
+            -0.010968545451760292,
+            0.007763105444610119,
+            0.02481001615524292,
+            0.06412653625011444,
+            -0.006934036035090685,
+            -0.025093762204051018,
+            0.001193727133795619,
+            -0.02489868737757206,
+            0.020695703104138374,
+            0.03848301246762276,
+            0.025483911857008934,
+            -0.01815972663462162,
+            0.008995626121759415,
+            -0.009922233410179615,
+            0.016155771911144257,
+            -0.014178418554365635,
+            0.025661254301667213,
+            -0.021688813343644142,
+            0.027062248438596725,
+            0.016368580982089043,
+            -0.03748990222811699,
+            -0.03177952021360397,
+            0.0035667724441736937,
+            0.003726379247382283,
+            -0.00783404242247343,
+            0.0015772271435707808,
+            0.005289197433739901,
+            0.034067220985889435,
+            -0.014870049431920052,
+            0.00894242338836193,
+            0.003912587650120258,
+            -0.033535197377204895,
+            -0.007878377102315426,
+            -0.035237669944763184,
+            -0.0650487095117569,
+            0.03780911862850189,
+            -0.08483996987342834,
+            -0.012910430319607258,
+            -0.017831645905971527,
+            0.01191732008010149,
+            -0.011057215742766857,
+            0.009039961732923985,
+            0.06430387496948242,
+            0.01702474243938923,
+            0.010471990332007408,
+            0.0023896710481494665,
+            -0.04312935099005699,
+            -0.0010507458355277777,
+            -0.007581330835819244,
+            -0.01966712437570095,
+            0.001189293572679162,
+            -0.029988376423716545,
+            -0.03589383512735367,
+            -0.017831645905971527,
+            0.0019474267028272152,
+            -0.027682941406965256,
+            -0.05171265825629234,
+            -0.002861841581761837,
+            0.020802108570933342,
+            -0.02383464016020298,
+            -0.02755880355834961,
+            -0.0328080989420414,
+            -0.040185488760471344,
+            -0.006588221061974764,
+            0.005807920359075069,
+            0.04075298085808754,
+            -0.023072073236107826,
+            -0.02995290793478489,
+            0.02037649042904377,
+            -0.017964651808142662,
+            -0.04057563841342926,
+            0.011243424378335476,
+            0.006925168912857771,
+            -0.0393342487514019,
+            0.027186386287212372,
+            0.04330668970942497,
+            -0.025749923661351204,
+            -0.00035828439285978675,
+            -0.014533101581037045,
+            -0.01913510262966156,
+            0.000581900414545089,
+            -0.029740098863840103,
+            -0.02709771692752838,
+            -0.03126523271203041,
+            0.0014054280472919345,
+            -0.018674015998840332,
+            -0.0025248934980481863,
+            0.007563597057014704,
+            -0.020500628277659416,
+            -0.03731256350874901,
+            0.009815828874707222,
+            0.0009986519580706954,
+            0.0007620124379172921,
+            -0.016253309324383736,
+            0.01440009567886591,
+            -0.021068120375275612,
+            -0.005524174310266972,
+            -0.002821939764544368,
+            0.0011904019629582763,
+            0.01320304349064827,
+            0.05518854409456253,
+            -0.026636630296707153,
+            0.0665738433599472,
+            -0.009141932241618633,
+            -0.01913510262966156,
+            0.008920256048440933,
+            0.006428613793104887,
+            -0.03731256350874901,
+            0.006348810624331236,
+            -0.02605140395462513,
+            0.009532082825899124,
+            0.004101012367755175,
+            -0.0023497692309319973,
+            0.00025395795819349587,
+            -0.012662152759730816,
+            0.01067593228071928,
+            -0.0247036125510931,
+            -0.014905516989529133,
+            -0.018727216869592667,
+            -0.011819782666862011,
+            0.034404169768095016,
+            -0.012724222615361214,
+            0.02417158894240856,
+            -0.027789346873760223,
+            0.03741896525025368,
+            0.015623748302459717,
+            -0.018141992390155792,
+            0.01977352984249592,
+            -0.022557783871889114,
+            -0.021688813343644142,
+            0.024419866502285004,
+            -0.01594296284019947,
+            0.02170654758810997,
+            0.022256305441260338,
+            0.01904643140733242,
+            0.0366741344332695,
+            -0.012600083835422993,
+            0.032116468995809555,
+            -0.03559235483407974,
+            0.02234497480094433,
+            -0.07061721384525299,
+            0.006096099503338337,
+            0.017317356541752815,
+            0.036886945366859436,
+            -0.02835683710873127,
+            0.00717788003385067,
+            0.021227726712822914,
+            -0.013398118317127228,
+            -0.007408423349261284,
+            0.010746869258582592,
+            0.00956755131483078,
+            -0.034510571509599686,
+            -0.012289736419916153,
+            0.028764722868800163,
+            -0.050896890461444855,
+            -0.013637528754770756,
+            -0.0027753878384828568,
+            0.007483793422579765,
+            0.0027776046190410852,
+            -0.047704748809337616,
+            -0.004488945938646793,
+            -0.02878245711326599,
+            -0.007918278686702251,
+            0.010152776725590229,
+            0.02427799254655838,
+            0.021671079099178314,
+            -0.01966712437570095,
+            -0.03284356743097305,
+            0.006645856890827417,
+            0.029545024037361145,
+            0.009993169456720352,
+            0.026441553607583046,
+            0.07192953675985336,
+            -0.036195311695337296,
+            -0.01502078864723444,
+            0.009833562187850475,
+            -0.00788724422454834,
+            -0.016359714791178703,
+            0.03475885093212128,
+            0.03279036656022072,
+            -0.03692241013050079,
+            -0.011057215742766857,
+            -0.020075010135769844,
+            -0.029527289792895317,
+            -0.05774225294589996,
+            0.0027709542773663998,
+            -0.0008872595499269664,
+            -0.02081984281539917,
+            0.00427170330658555,
+            0.014187285676598549,
+            -0.055685099214315414,
+            0.005280330311506987,
+            -0.016528187319636345,
+            -0.021458270028233528,
+            -0.028587380424141884,
+            -0.03569876030087471,
+            -0.013176442123949528,
+            0.008973458781838417,
+            -0.037702713161706924,
+            0.007523695006966591,
+            -0.022575518116354942,
+            0.030502665787935257,
+            0.008574441075325012,
+            -0.008303996175527573,
+            0.025217900052666664,
+            -0.021156789734959602,
+            0.013362649828195572,
+            -0.006867533084005117,
+            -0.04146234318614006,
+            -0.003145587397739291,
+            0.0023608531337231398,
+            -0.011021748185157776,
+            0.0055463421158492565,
+            0.023515427485108376,
+            -0.02791348472237587,
+            0.029970642179250717,
+            0.0017833862220868468,
+            -0.0061404346488416195,
+            0.0061315675266087055,
+            -0.01710454747080803,
+            -0.011748846620321274,
+            -0.021227726712822914,
+            0.0023165177553892136,
+            -0.0034382001031190157,
+            -0.04011455178260803,
+            -0.04394511878490448,
+            -0.048520516604185104,
+            -0.007541429251432419,
+            -0.014001077972352505,
+            -0.00903552770614624,
+            -0.0083217304199934,
+            -0.005462104920297861,
+            -0.004034509416669607,
+            -0.0031123359221965075,
+            -0.007027139887213707,
+            -0.010294649749994278,
+            -0.009372475557029247,
+            -0.004280570428818464,
+            0.02294793538749218,
+            -0.019808998331427574,
+            -0.019188303500413895,
+            -0.016599124297499657,
+            0.00631334213539958,
+            -0.024508535861968994,
+            -0.03545048087835312,
+            0.003611107589676976,
+            0.038412075489759445,
+            0.03373027220368385,
+            0.011465100571513176,
+            -0.018026720732450485,
+            0.0026290814857929945,
+            0.019596189260482788,
+            0.022699657827615738,
+            -0.05909004807472229,
+            -0.01332718227058649,
+            -0.0008351656142622232,
+            -0.02410065196454525,
+            -0.0027798213995993137,
+            -0.015384337864816189,
+            0.0010551793966442347,
+            -0.006526151672005653,
+            -0.0005539137637242675,
+            -0.007155712228268385,
+            -0.020802108570933342,
+            -6.234647298697382e-05,
+            0.0027111016679555178,
+            -0.0008235276327468455,
+            -0.002099274890497327,
+            0.016616858541965485,
+            0.019649391993880272,
+            -0.0164040494710207,
+            0.00757246371358633,
+            -0.011216823011636734,
+            0.018567610532045364,
+            0.03137163445353508,
+            -0.0016437300946563482,
+            0.020713437348604202,
+            -0.025306571274995804,
+            0.003990174271166325,
+            0.022735126316547394,
+            -0.034670181572437286,
+            -0.027168652042746544,
+            -0.0014575219247490168,
+            0.014453297480940819,
+            -0.0006240188959054649,
+            0.01984446682035923,
+            -0.06370091438293457,
+            -0.00586998974904418,
+            -0.025058293715119362,
+            0.00200284575112164,
+            -0.016439517959952354,
+            0.035503681749105453,
+            -0.003141153836622834,
+            0.012245400808751583,
+            0.03259528800845146,
+            0.006415313575416803,
+            0.016714395955204964,
+            -0.01779617741703987,
+            0.014408962801098824,
+            0.0354682132601738,
+            -0.047101788222789764,
+            -0.011881851591169834,
+            -0.006645856890827417,
+            -0.044512610882520676,
+            0.013371516950428486,
+            0.011296626180410385,
+            0.019702592864632607,
+            0.027594272047281265,
+            -0.024384398013353348,
+            0.03165538236498833,
+            -0.007324186619371176,
+            -0.009363608434796333,
+            0.026246478781104088,
+            -0.004805942997336388,
+            0.019472049549221992,
+            0.006362111307680607,
+            -0.0072133480571210384,
+            0.010569527745246887,
+            0.012245400808751583,
+            -0.011216823011636734,
+            0.006876400206238031,
+            -0.029296746477484703,
+            -8.721578342374414e-05,
+            0.012316337786614895,
+            -0.010516325943171978,
+            0.002083757659420371,
+            -0.03390761464834213,
+            0.025448445230722427,
+            -0.013140973635017872,
+            -0.01267101988196373,
+            0.03461697697639465,
+            0.004225151147693396,
+            0.016599124297499657,
+            0.03447510674595833,
+            0.0026889340952038765,
+            -0.009000059217214584,
+            0.006597088184207678,
+            -0.001766760484315455,
+            0.007341920398175716,
+            0.018993228673934937,
+            0.00934587512165308,
+            -0.023550894111394882,
+            -0.02374597080051899,
+            -0.013699597679078579,
+            0.008836018852889538,
+            0.01001090370118618,
+            0.0020848659332841635,
+            0.003216523677110672,
+            0.008707446977496147,
+            0.009159666486084461,
+            0.015792222693562508,
+            -0.013353782705962658,
+            -0.0031256363727152348,
+            0.01763656921684742,
+            -0.03773818165063858,
+            0.0036465758457779884,
+            -0.010285782627761364,
+            0.0010391079122200608,
+            0.00934587512165308,
+            -0.00047992929467000067,
+            0.0473145991563797,
+            0.005643879529088736,
+            0.008405966684222221,
+            0.010436521843075752,
+            0.019365645945072174,
+            -0.006774429231882095,
+            -0.017964651808142662,
+            0.005967527162283659,
+            0.026477022096514702,
+            -0.029456352815032005,
+            0.03766724467277527,
+            -0.00445569446310401,
+            -0.001095081097446382,
+            0.002797555411234498,
+            -0.006539452355355024,
+            -0.018833622336387634,
+            -0.00405446020886302,
+            -0.03509579971432686,
+            -0.009833562187850475,
+            0.01098627969622612,
+            0.027665207162499428,
+            -0.023941045626997948,
+            -0.02622874453663826,
+            0.03138937056064606,
+            0.023107541725039482,
+            -0.0005497573292814195,
+            0.030963752418756485,
+            -0.028232699260115623,
+            0.020624767988920212,
+            0.026920374482870102,
+            0.018815888091921806,
+            0.03748990222811699,
+            -0.030981486663222313,
+            -0.011713378131389618,
+            0.022167634218931198,
+            0.038944099098443985,
+            0.038837697356939316,
+            0.0012380623957142234,
+            -0.03748990222811699,
+            0.02347995899617672,
+            0.03422682732343674,
+            0.03426229581236839,
+            0.00675226142629981,
+            -0.02339128777384758,
+            -0.009638487361371517,
+            0.013469054363667965,
+            0.0005458780215121806,
+            0.010090706869959831,
+            -0.010684799402952194,
+            0.008295129053294659,
+            -0.013593193143606186,
+            -0.016625726595520973,
+            -0.0005375651526264846,
+            -0.018815888091921806,
+            -0.033801209181547165,
+            -0.030857346951961517,
+            0.016944939270615578,
+            -0.014417828992009163,
+            -0.03470565006136894,
+            -0.019879935309290886,
+            -0.013185309246182442,
+            -0.005847821943461895,
+            -0.04834317788481712,
+            -0.02800215594470501,
+            -0.020429691299796104,
+            0.027310525998473167,
+            -0.01879815384745598,
+            -0.007865076884627342,
+            -0.02727505750954151,
+            -0.003212090115994215,
+            0.0007664459408260882,
+            0.0384475439786911,
+            -0.013894673436880112,
+            -0.0001251778594451025,
+            -0.02488095313310623,
+            0.013238511048257351,
+            -0.02984650246798992,
+            0.0065039838664233685,
+            -0.015827691182494164,
+            -0.02638835273683071,
+            0.011305493302643299,
+            -0.011491701938211918,
+            -0.002347552450373769,
+            -0.011172487400472164,
+            0.01377053465694189,
+            -0.005151758436113596,
+            -0.010658198967576027,
+            0.007723203860223293,
+            0.0014763644430786371,
+            -0.008552273735404015,
+            -0.02259325236082077,
+            0.04962003231048584,
+            0.0036598765291273594,
+            -0.0062645734287798405,
+            -0.0013056736206635833,
+            -0.012387273833155632,
+            -0.006291174795478582,
+            -0.008303996175527573,
+            -0.019099634140729904,
+            0.002118117408826947,
+            0.017778443172574043,
+            0.026175541803240776,
+            0.026884907856583595,
+            -0.001146066701039672,
+            0.015419806353747845,
+            -0.00015406505553983152,
+            -0.04600227624177933,
+            -0.022717392072081566,
+            -0.03169085085391998,
+            -0.03169085085391998,
+            0.03197459504008293,
+            -0.019649391993880272,
+            -0.0030480497516691685,
+            -0.011872985400259495,
+            0.02463267557322979,
+            0.02913713827729225,
+            0.0003538508608471602,
+            -0.0030347490683197975,
+            -0.0001627935707801953,
+            0.0027842549607157707,
+            -0.01045425608754158,
+            0.0033827810548245907,
+            -0.016058234497904778,
+            0.024845484644174576,
+            -5.153975143912248e-05,
+            0.01727301999926567,
+            0.032471150159835815,
+            -0.02913713827729225,
+            0.033801209181547165,
+            0.010188245214521885,
+            0.008729614317417145,
+            0.015925228595733643,
+            -0.008135521784424782,
+            -0.0011011771857738495,
+            0.035822898149490356,
+            0.02073117159307003,
+            -0.031673114746809006,
+            0.01457743626087904,
+            -0.02372823655605316,
+            -0.026884907856583595,
+            0.020890777930617332,
+            -0.009806961752474308,
+            -0.028924329206347466,
+            0.029704630374908447,
+            0.020181413739919662,
+            0.026104606688022614,
+            0.032985441386699677,
+            -0.015818823128938675,
+            -0.006654724013060331,
+            -0.002751003485172987,
+            0.0004120409139432013,
+            -0.006344377063214779,
+            0.012600083835422993,
+            0.004087711684405804,
+            -0.0014054280472919345,
+            -0.001630429527722299,
+            0.018478939309716225,
+            0.0006295607890933752,
+            0.011598106473684311,
+            0.006277874112129211,
+            -0.00751039432361722,
+            -0.029030734673142433,
+            -0.008051284588873386,
+            0.011633574962615967,
+            0.03731256350874901,
+            -0.034865256398916245,
+            0.00859217531979084,
+            -0.022699657827615738,
+            -0.019188303500413895,
+            0.01737942546606064,
+            -0.03271942958235741,
+            0.004770474974066019,
+            0.016776464879512787,
+            0.0006539452006109059,
+            0.013903540559113026,
+            0.009425678290426731,
+            0.011598106473684311,
+            0.012963633053004742,
+            0.0057724518701434135,
+            0.002511593047529459,
+            0.015357737429440022,
+            0.012910430319607258,
+            0.07249703258275986,
+            -0.025040559470653534,
+            0.00045665327343158424,
+            0.03802192583680153,
+            -0.027168652042746544,
+            -0.003965789917856455,
+            0.03396081551909447,
+            -0.006468515843153,
+            -0.02401198074221611,
+            0.029562756419181824,
+            -0.01860307902097702,
+            -0.02816176228225231,
+            -0.016794199123978615,
+            -0.030272122472524643,
+            -0.013983343727886677,
+            0.012165597639977932,
+            0.01762770302593708,
+            -0.011145886965095997,
+            0.025005090981721878,
+            0.0008401533705182374,
+            -0.019915401935577393,
+            -0.008787250146269798,
+            0.02259325236082077,
+            -0.012316337786614895,
+            -0.006694625597447157,
+            0.0025780959986150265,
+            0.012520279735326767,
+            0.00962075311690569,
+            0.015340003184974194,
+            -0.041604217141866684,
+            -0.010303516872227192,
+            0.011119285598397255,
+            -0.0031034687999635935,
+            0.0030037143733352423,
+            0.0276474729180336,
+            0.01717548258602619,
+            0.011828649789094925,
+            -0.03163764625787735,
+            0.027168652042746544,
+            0.01853214204311371,
+            -0.023409022018313408,
+            0.03827020525932312,
+            -0.03936971724033356,
+            -0.025005090981721878,
+            -0.006676891352981329,
+            0.006109400186687708,
+            0.0049566831439733505,
+            -0.00631334213539958,
+            -0.010064106434583664,
+            -0.024916421622037888,
+            -0.03279036656022072,
+            -0.019010962918400764,
+            -0.0005746959359385073,
+            0.015295667573809624,
+            0.011802048422396183,
+            0.005235995166003704,
+            0.008286261931061745,
+            0.04057563841342926,
+            -0.0033606134820729494,
+            -0.002994847483932972,
+            -0.031939126551151276,
+            -0.012289736419916153,
+            0.03165538236498833,
+            0.027044514194130898,
+            -0.005945359356701374,
+            0.007204481400549412,
+            -0.005351266823709011,
+            0.00907986331731081,
+            0.0013245161389932036,
+            -0.013486788608133793,
+            0.026264213025569916,
+            0.04632148891687393,
+            -0.0022533400915563107,
+            -0.03279036656022072,
+            -0.006167036015540361,
+            -0.026884907856583595,
+            -0.01566808484494686,
+            0.012733088806271553,
+            -0.026689831167459488,
+            0.0027088848873972893,
+            -0.011988257057964802,
+            -0.04366137459874153,
+            -0.008756215684115887,
+            -0.009443412534892559,
+            0.01753903180360794,
+            -0.00571481604129076,
+            -0.0062468391843140125,
+            -0.0061227004043757915,
+            -0.0053335330449044704,
+            0.00350913661532104,
+            -0.0031832722015678883,
+            -0.03341105952858925,
+            0.020677968859672546,
+            -0.01869175024330616,
+            -0.023089807480573654,
+            -0.02002180740237236,
+            -0.003923671320080757,
+            0.002082649152725935,
+            -0.005572943482547998,
+            -0.010179378092288971,
+            0.014870049431920052,
+            -0.019383380189538002,
+            -0.024526270106434822,
+            -0.03837660700082779,
+            0.012014857493340969,
+            0.010046372190117836,
+            -0.011553770862519741,
+            -0.01186411827802658,
+            -0.0016160205705091357,
+            -0.004899047315120697,
+            0.01568581722676754,
+            -0.02346222475171089,
+            0.0049522495828568935,
+            0.029367681592702866,
+            0.018008986487984657,
+            -0.026477022096514702,
+            0.012050325982272625,
+            -0.01897549442946911,
+            0.02913713827729225,
+            -0.027079982683062553,
+            -0.011819782666862011,
+            0.02443760074675083,
+            0.0022965669631958008,
+            -0.0027044513262808323,
+            -0.025803126394748688,
+            0.015393204987049103,
+            -0.002391887828707695,
+            0.019915401935577393,
+            -0.021511472761631012,
+            0.02773614414036274,
+            0.004276136867702007,
+            -0.030360791832208633,
+            0.015047390013933182,
+            0.026104606688022614,
+            -0.011367563158273697,
+            0.02365729957818985,
+            0.01957845501601696,
+            -0.004899047315120697,
+            -0.010729135014116764,
+            0.0175301656126976,
+            -0.008188724517822266,
+            0.023799171671271324,
+            -0.0407884456217289,
+            0.0064951167441904545,
+            -0.044867292046546936,
+            -0.036106642335653305,
+            0.003602240700274706,
+            -0.006237972527742386,
+            -0.020979449152946472,
+            -0.040220957249403,
+            0.003229824360460043,
+            0.009922233410179615,
+            0.01842573843896389,
+            0.027842549607157707,
+            -0.0218838881701231,
+            0.02878245711326599,
+            0.0034226826392114162,
+            -0.027487866580486298,
+            0.0070936428382992744,
+            -0.0022810494992882013,
+            -0.032205138355493546,
+            -0.009239469654858112,
+            -0.022025762125849724,
+            -0.005586243700236082,
+            -0.049832843244075775,
+            0.013052303344011307,
+            -0.011074949987232685,
+            0.015934094786643982,
+            -0.017521297559142113,
+            -0.02833910472691059,
+            -0.0022489065304398537,
+            0.037773650139570236,
+            0.003945839125663042,
+            -0.036816008388996124,
+            -0.02924354374408722,
+            0.019010962918400764,
+            -0.02984650246798992,
+            -0.007071475498378277,
+            0.005936492700129747,
+            0.020252350717782974,
+            -0.010773470625281334,
+            -0.020411957055330276,
+            -0.010037505067884922,
+            -0.006065064575523138,
+            -0.00823305919766426,
+            0.013070037588477135,
+            0.04163968563079834,
+            -0.004659636877477169,
+            0.010356718674302101,
+            0.0027155352290719748,
+            0.04603774473071098,
+            -0.001809987355954945,
+            -0.018656281754374504,
+            0.028587380424141884,
+            -0.0014364627422764897,
+            -0.0059143248945474625,
+            0.016164638102054596,
+            -0.013238511048257351,
+            0.002917260630056262,
+            -0.0300238449126482,
+            0.028906594961881638,
+            0.011970522813498974,
+            -0.026867173612117767,
+            -0.00939907692372799,
+            -0.02782481536269188,
+            -0.02791348472237587,
+            -0.05022299289703369,
+            -0.011092684231698513,
+            -0.01700701005756855,
+            0.04518650472164154,
+            -0.05593337491154671,
+            -0.01728188805282116,
+            0.006198070477694273,
+            -0.006920735351741314,
+            -0.024224791675806046,
+            0.006645856890827417,
+            -0.01320304349064827,
+            -0.022185368463397026,
+            0.011145886965095997,
+            -0.033269185572862625,
+            -0.003879335941746831,
+            -0.015907494351267815,
+            -0.007315319497138262,
+            0.0017301838379353285,
+            -0.0014353543519973755,
+            0.06015409156680107,
+            0.009957700967788696,
+            0.02862284891307354,
+            -0.009443412534892559,
+            0.009780360385775566,
+            0.026104606688022614,
+            0.020766640082001686,
+            0.023444490507245064,
+            0.009904499165713787,
+            -0.0047305733896791935,
+            -0.019702592864632607,
+            0.0024184889625757933,
+            0.010241447016596794,
+            0.010072972625494003,
+            0.005621712189167738,
+            -0.028410039842128754,
+            -0.02126319520175457,
+            -0.006916301790624857,
+            -0.008667545393109322,
+            -0.004380324389785528,
+            0.031158827245235443,
+            -0.016794199123978615,
+            -0.029367681592702866,
+            -0.023284882307052612,
+            -0.005036486312747002,
+            -0.04635695740580559,
+            0.0003114552528131753,
+            0.015827691182494164,
+            0.002108142012730241,
+            5.7462664699414745e-05,
+            -0.0026623329613357782,
+            0.004570966120809317,
+            -0.011119285598397255,
+            0.014072014018893242,
+            0.0034515007864683867,
+            0.0058034867979586124,
+            -0.00996656809002161,
+            -0.0005685998476110399,
+            0.0024916420225054026,
+            0.013921274803578854,
+            0.006597088184207678,
+            -0.02259325236082077,
+            -0.04802396148443222,
+            -0.027416929602622986,
+            -0.0029615960083901882,
+            -0.013273979537189007,
+            -0.000881163461599499,
+            -0.010853273794054985,
+            -0.0049655502662062645,
+            0.0034382001031190157,
+            -0.0008839344372972846,
+            -0.011855251155793667,
+            -0.008073452860116959,
+            -0.004245101939886808,
+            -0.008937990292906761,
+            -0.023072073236107826,
+            0.023355819284915924,
+            -0.009443412534892559,
+            -0.01621784083545208,
+            -0.020890777930617332,
+            -0.007501527667045593,
+            -0.023426756262779236,
+            -0.010587261989712715,
+            0.009673955850303173,
+            0.006557186134159565,
+            -0.016412915661931038,
+            -0.006716793403029442,
+            -0.004449044354259968,
+            0.008286261931061745,
+            -3.664586984086782e-05,
+            -0.006162602454423904,
+            -0.029083935543894768,
+            0.007523695006966591,
+            0.008711880072951317,
+            0.0083217304199934,
+            -0.0015317834913730621,
+            -0.002019471488893032,
+            0.01470157504081726,
+            0.08065471798181534,
+            -0.012697621248662472,
+            0.009363608434796333,
+            0.0008722963975742459,
+            -0.032400213181972504,
+            0.01244934368878603,
+            -0.03848301246762276,
+            -0.001967377495020628,
+            0.02230950817465782,
+            0.008574441075325012,
+            0.024260258302092552,
+            -0.011606973595917225,
+            0.010090706869959831,
+            0.04827224090695381,
+            -0.006348810624331236,
+            0.0029793300200253725,
+            0.0026800669729709625,
+            2.926820525317453e-05,
+            0.0056616137735545635,
+            0.007031573448330164,
+            -0.0015417590038850904,
+            -0.009682822972536087,
+            -0.019347911700606346,
+            -0.016191240400075912,
+            0.01780504360795021,
+            -0.013850337825715542,
+            -0.0012258702190592885,
+            -0.03177952021360397,
+            -0.004198549780994654,
+            -0.010002036578953266,
+            0.015632616356015205,
+            -0.03057360090315342,
+            0.00029787758830934763,
+            0.012298603542149067,
+            -0.035663291811943054,
+            0.004311604890972376,
+            0.001890899264253676,
+            0.0007936013280414045,
+            0.01835480146110058,
+            -0.003152237506583333,
+            -0.007723203860223293,
+            0.022132165729999542,
+            0.0022522315848618746,
+            -0.013628661632537842,
+            -0.022841529920697212,
+            0.002338685328140855,
+            -0.015987297520041466,
+            -0.03231154382228851,
+            -0.019720327109098434,
+            -0.00039347552228718996,
+            -0.009097597561776638,
+            0.023054338991642,
+            -0.007049307692795992,
+            -0.028729254379868507,
+            -0.01922377198934555,
+            0.007816308178007603,
+            0.004668503999710083,
+            -0.017140014097094536,
+            -0.005834521260112524,
+            0.021812953054904938,
+            -0.03738350048661232,
+            -0.021103588864207268,
+            -0.009443412534892559,
+            0.002622431144118309,
+            0.004641902633011341,
+            -0.008969024755060673,
+            -0.017255285754799843,
+            0.0024362229742109776,
+            0.004056677222251892,
+            -0.015304534696042538,
+            -0.0016359713627025485,
+            0.028285901993513107,
+            0.019418848678469658,
+            0.004522197414189577,
+            -0.00288179237395525,
+            0.004067760892212391,
+            -0.0051916600205004215,
+            0.006211371161043644,
+            0.0025936132296919823,
+            -0.026618896052241325,
+            -0.008290695026516914,
+            0.04848505184054375,
+            0.003999041393399239,
+            0.0057680183090269566,
+            0.004797075875103474,
+            -0.007084776181727648,
+            -0.014541967771947384,
+            0.0005589014617726207,
+            0.020057275891304016,
+            0.013415852561593056,
+            0.022752860561013222,
+            0.015428673475980759,
+            -0.007918278686702251,
+            -0.004167515318840742,
+            0.0002563132729846984,
+            -0.02285926416516304,
+            0.036816008388996124,
+            0.0016891737468540668,
+            -0.0025337606202811003,
+            0.005626145750284195,
+            -0.0063887122087180614,
+            0.00031976812169887125,
+            0.025501646101474762,
+            -0.004087711684405804,
+            -0.014666106551885605,
+            0.02321394719183445,
+            0.013477921485900879,
+            -0.007257683668285608,
+            -0.034510571509599686,
+            0.008840452879667282,
+            0.007044874131679535,
+            0.01585429161787033,
+            -0.01737942546606064,
+            -0.003799532540142536,
+            -0.009115330874919891,
+            -0.013318315148353577,
+            0.03660319745540619,
+            -0.00723994942381978,
+            -0.006597088184207678,
+            -0.0010075189638882875,
+            -0.030449463054537773,
+            0.00680103013291955,
+            0.02383464016020298,
+            0.00819759164005518,
+            -0.027345994487404823,
+            -0.0068542324006557465,
+            0.004083278123289347,
+            -0.020270084962248802,
+            -0.010853273794054985,
+            0.005107422824949026,
+            -0.023781437426805496,
+            0.011278891935944557,
+            -0.0028175064362585545,
+            -0.0030702173244208097,
+            0.0018432388314977288,
+            0.0020050625316798687,
+            0.04784662276506424,
+            -0.02940315008163452,
+            0.01904643140733242,
+            -0.003897070186212659,
+            0.009239469654858112,
+            0.00018468410416971892,
+            0.011190221644937992,
+            -0.007270983885973692,
+            -0.007869509980082512,
+            -0.00019646066357381642,
+            -0.03745443373918533,
+            -0.0007570247398689389,
+            0.010401054285466671,
+            0.038305673748254776,
+            -0.007754238322377205,
+            -0.007115810643881559,
+            0.023621831089258194,
+            0.011314360424876213,
+            0.009106464684009552,
+            -0.008410400710999966,
+            -0.005989694967865944,
+            -0.007563597057014704,
+            -0.0011288868263363838,
+            -0.012564615346491337,
+            0.029296746477484703,
+            -0.00907986331731081,
+            0.003713078796863556,
+            -0.030077045783400536,
+            0.02933221310377121,
+            0.0056571802124381065,
+            0.02401198074221611,
+            -0.004273919854313135,
+            0.0019607271533459425,
+            0.005821220576763153,
+            0.025998201221227646,
+            -0.009274938143789768,
+            -0.023089807480573654,
+            0.0012979150051251054,
+            0.01763656921684742,
+            0.04387418180704117,
+            0.030609069392085075,
+            -0.017476962879300117,
+            0.023444490507245064,
+            -0.03596476837992668,
+            0.0003726933500729501,
+            0.006198070477694273,
+            -0.006570486817508936,
+            -0.0053202323615550995,
+            -0.014027679339051247,
+            0.0004497258923947811,
+            0.004637469071894884,
+            -0.01798238418996334,
+            -0.008175423368811607,
+            -0.010223712772130966,
+            0.01518926303833723,
+            0.01799125224351883,
+            0.004903480876237154,
+            -0.0010851057013496757,
+            7.121352246031165e-05,
+            -0.021724281832575798,
+            0.010081839747726917,
+            -0.026547959074378014,
+            0.03438643366098404,
+            -0.009088730439543724,
+            -0.00010841358744073659,
+            0.007355221081525087,
+            -0.01279515866190195,
+            0.01762770302593708,
+            0.01541093923151493,
+            -0.0019751363433897495,
+            0.0059187584556639194,
+            -0.039653465151786804,
+            0.020890777930617332,
+            -0.009709424339234829,
+            -0.04571852833032608,
+            -0.0011194655671715736,
+            0.011332094669342041,
+            -0.01621784083545208,
+            0.020358756184577942,
+            0.03365933522582054,
+            -0.01603163406252861,
+            0.0136020602658391,
+            -0.015517343766987324,
+            -0.009195134975016117,
+            0.027878016233444214,
+            0.006100533064454794,
+            -0.015818823128938675,
+            0.007975914515554905,
+            0.0057680183090269566,
+            0.02046515978872776,
+            -0.008525672368705273,
+            -2.9960943720652722e-05,
+            0.004695104900747538,
+            -0.010170510970056057,
+            0.040220957249403,
+            5.850177331012674e-05,
+            0.019436581060290337,
+            -0.01895776018500328,
+            -0.010906476527452469,
+            -0.006472949404269457,
+            0.003294110530987382,
+            -0.0067300936207175255,
+            -0.010321250185370445,
+            -0.018727216869592667,
+            -0.009239469654858112,
+            -0.00939907692372799,
+            0.031495776027441025,
+            -0.0014785812236368656,
+            -0.011704511009156704,
+            -0.0002506328164599836,
+            -0.013619794510304928,
+            -0.004240668378770351,
+            -0.029314478859305382,
+            -0.0054177697747945786,
+            -0.007324186619371176,
+            -0.01966712437570095,
+            -0.006295608356595039,
+            -0.01328284665942192,
+            0.005861122626811266,
+            0.014790245331823826,
+            0.008565573953092098,
+            0.008379366248846054,
+            -0.0040079085156321526,
+            -0.0021713196765631437,
+            -0.006827631499618292,
+            -0.037241626530885696,
+            0.017388291656970978,
+            0.0016326463082805276,
+            -0.01586315967142582,
+            0.00886705331504345,
+            -0.03656772896647453,
+            0.03571649268269539,
+            0.02330261655151844,
+            -0.02888886071741581,
+            0.00575915165245533,
+            0.025909531861543655,
+            0.024863218888640404,
+            -0.0036554429680109024,
+            0.009984302334487438,
+            -0.015543945133686066,
+            -0.007412856910377741,
+            -0.0053290994837880135,
+            0.02629968151450157,
+            -0.009727157652378082,
+            0.002292133402079344,
+            0.04206530377268791,
+            -0.0011837516212835908,
+            0.004788209218531847,
+            -0.011376430280506611,
+            0.005360133945941925,
+            -0.008441435173153877,
+            0.0027266191318631172,
+            -0.036354921758174896,
+            -0.006929602473974228,
+            -0.03413815796375275,
+            0.0062601398676633835,
+            0.01191732008010149,
+            -0.005954226478934288,
+            0.02842777408659458,
+            0.002972679678350687,
+            0.002906176960095763,
+            0.022433646023273468,
+            -0.025944998487830162,
+            -0.026796236634254456,
+            0.012378406710922718,
+            0.006951770279556513,
+            0.031123358756303787,
+            0.013637528754770756,
+            0.007222215179353952,
+            0.003555688541382551,
+            -0.0056660473346710205,
+            -0.04518650472164154,
+            0.00819759164005518,
+            -0.013406985439360142,
+            0.024402132257819176,
+            0.0046020010486245155,
+            0.0080601517111063,
+            0.03605344146490097,
+            0.01332718227058649,
+            0.004615301266312599,
+            0.016359714791178703,
+            0.0007514827884733677,
+            0.00819759164005518,
+            0.03004157729446888,
+            0.028658317402005196,
+            -0.007155712228268385,
+            -0.04649883136153221,
+            -0.032027799636125565,
+            0.010383320041000843,
+            -0.0023586363531649113,
+            -0.0006478491122834384,
+            0.010019770823419094,
+            -0.023373553529381752,
+            -0.01033011730760336,
+            0.023515427485108376,
+            -4.7383316996274516e-05,
+            -0.00653058523312211,
+            -0.03156671300530434,
+            -0.012050325982272625,
+            0.01771637424826622,
+            -0.012928164564073086,
+            0.011021748185157776,
+            0.014772512018680573,
+            -0.010871008038520813,
+            -0.015419806353747845,
+            -0.014932118356227875,
+            -0.002702234545722604,
+            0.019986338913440704,
+            0.009860163554549217,
+            -0.020004073157906532,
+            0.022486848756670952,
+            0.016634592786431313,
+            0.0062734405510127544,
+            0.012768557295203209,
+            -0.020571565255522728,
+            0.02638835273683071,
+            -0.013983343727886677,
+            0.014692707918584347,
+            -0.012050325982272625,
+            -0.00686309952288866,
+            0.011890718713402748,
+            0.004101012367755175,
+            -0.0014353543519973755,
+            -0.003888203063979745,
+            0.001440896186977625,
+            -0.012334072031080723,
+            -0.0013622011756524444,
+            -0.020589299499988556,
+            0.009283805266022682,
+            -0.011225690133869648,
+            -0.00047217062092386186,
+            0.023196212947368622,
+            0.01413408387452364,
+            0.003201006446033716,
+            0.0028441075701266527,
+            -0.014045413583517075,
+            -0.01984446682035923,
+            0.0050675212405622005,
+            -0.006308908574283123,
+            -0.0028441075701266527,
+            -0.02073117159307003,
+            -0.0254307109862566,
+            0.017583368346095085,
+            0.03321598470211029,
+            0.016058234497904778,
+            -0.027629738673567772,
+            -0.013876939192414284,
+            -0.004205200355499983,
+            0.014009945094585419,
+            0.011881851591169834,
+            -0.006632556207478046,
+            0.006397579330950975,
+            -0.008193157613277435,
+            -0.0036000239197164774,
+            -0.013433586806058884,
+            0.015996165573596954,
+            0.01711341366171837,
+            0.004468995146453381,
+            0.018301598727703094,
+            0.015632616356015205,
+            0.015446407720446587,
+            -0.030431728810071945,
+            -0.0014043196570128202,
+            0.018283864483237267,
+            -0.010915343649685383,
+            -0.03585836663842201,
+            -0.014692707918584347,
+            0.00228326627984643,
+            0.018461205065250397,
+            -0.0015905278269201517,
+            -0.02135186456143856,
+            -0.014719309285283089,
+            0.0002265255170641467,
+            -0.03373027220368385,
+            0.017335090786218643,
+            0.00018468410416971892,
+            -0.00043060630559921265,
+            0.015526210889220238,
+            0.0074483249336481094,
+            -0.0007398448069579899,
+            0.005107422824949026,
+            -0.019188303500413895,
+            -0.006521718110889196,
+            -0.007545862812548876,
+            0.02959822490811348,
+            -0.00028429992380551994,
+            0.0023209513165056705,
+            -0.005705948919057846,
+            -0.009390209801495075,
+            0.007528128568083048,
+            -0.010383320041000843,
+            0.00022555567556992173,
+            0.012422742322087288,
+            0.025271102786064148,
+            -0.008405966684222221,
+            0.011314360424876213,
+            0.039830803871154785,
+            0.00038350006798282266,
+            -0.03436870127916336,
+            0.00023594676167704165,
+            -0.01435576006770134,
+            -0.005129590630531311,
+            0.009008926339447498,
+            0.007270983885973692,
+            -0.0007564705447293818,
+            -0.001497423741966486,
+            -0.006885267328470945,
+            -0.021919356659054756,
+            -0.012360673397779465,
+            -0.030147982761263847,
+            -0.00988676492124796,
+            -0.0047261398285627365,
+            0.007222215179353952,
+            -0.020482894033193588,
+            0.006477382965385914,
+            -0.01611143723130226,
+            -0.009718290530145168,
+            -0.015242465771734715,
+            -0.002522676717489958,
+            -0.013539991341531277,
+            -0.014391228556632996,
+            0.0070936428382992744,
+            0.006167036015540361,
+            0.00934587512165308,
+            -0.0033716971520334482,
+            -0.009057695046067238,
+            -0.0019197171786800027,
+            -0.012422742322087288,
+            0.014719309285283089,
+            0.00864537712186575,
+            0.01637744903564453,
+            -0.003936972003430128,
+            -0.009115330874919891,
+            0.00031588878482580185,
+            0.01683853566646576,
+            0.012786291539669037,
+            0.010268048383295536,
+            -0.012165597639977932,
+            0.014231621287763119,
+            -0.003688694443553686,
+            0.01700701005756855,
+            -0.004191899672150612,
+            0.010241447016596794,
+            -0.004792642779648304,
+            0.004938948899507523,
+            0.019613923504948616,
+            0.018993228673934937,
+            0.004912347998470068,
+            0.004903480876237154,
+            0.06632556021213531,
+            0.013406985439360142,
+            0.033517464995384216,
+            0.011287759058177471,
+            0.0065039838664233685,
+            -0.01510945986956358,
+            -0.008437002077698708,
+            -0.012422742322087288,
+            0.026743033900856972,
+            0.017246419563889503,
+            0.007270983885973692,
+            -0.018656281754374504,
+            -0.0030236653983592987,
+            0.028658317402005196,
+            0.014657239429652691,
+            0.008219758979976177,
+            0.006286741234362125,
+            0.019454315304756165,
+            0.006082798819988966,
+            0.008366065099835396,
+            0.003460367675870657,
+            0.0060029951855540276,
+            0.006078365258872509,
+            0.015313401818275452,
+            0.012617817148566246,
+            0.01208579447120428,
+            0.005018752533942461,
+            0.00560397794470191,
+            0.02949182130396366,
+            0.014426696114242077,
+            0.029615959152579308,
+            -0.0007819632883183658,
+            0.005710382480174303,
+            0.031939126551151276,
+            0.012147863395512104,
+            0.01421388704329729,
+            0.02614007517695427,
+            -0.0017612185329198837,
+            -0.022486848756670952,
+            0.0037507638335227966,
+            0.02214989997446537,
+            -0.014630638994276524,
+            0.023284882307052612,
+            -0.0033606134820729494,
+            -0.0008224192424677312,
+            -0.026725299656391144,
+            -0.019117368385195732,
+            0.0027776046190410852,
+            -0.027328260242938995,
+            0.009939967654645443,
+            -0.0029926307033747435,
+            -0.007523695006966591,
+            0.005847821943461895,
+            -0.007900545373558998,
+            -0.010516325943171978,
+            0.004562098998576403,
+            0.016874004155397415,
+            -0.005630579311400652,
+            -0.0067389607429504395,
+            0.013300580903887749,
+            -0.000335839664330706,
+            -0.0005372880259528756,
+            0.011819782666862011,
+            -0.00394362211227417,
+            -0.015632616356015205,
+            0.00935474131256342,
+            -0.0013743933523073792,
+            -0.012236534617841244,
+            -0.015606014989316463,
+            0.0013888023095205426,
+            -0.005976394284516573,
+            -0.01569468528032303,
+            0.0029704628977924585,
+            0.007590197958052158,
+            -0.002387454267591238,
+            -0.0031322867143899202,
+            0.007049307692795992,
+            0.012883828952908516,
+            0.011447366327047348,
+            -0.002442873315885663,
+            0.005488706286996603,
+            0.003828350454568863,
+            -0.014178418554365635,
+            -0.010862140916287899,
+            -0.038412075489759445,
+            -0.005816787481307983,
+            0.01922377198934555,
+            0.01860307902097702,
+            0.0006855340907350183,
+            0.003285243408754468,
+            0.009195134975016117,
+            0.006699059158563614,
+            -0.0007453867001459002,
+            0.004801509436219931,
+            -0.025483911857008934,
+            0.002972679678350687,
+            0.012165597639977932,
+            -0.008352764882147312,
+            -0.0007985890260897577,
+            -0.01826613023877144,
+            -0.021316397935152054,
+            0.0041120960377156734,
+            0.003169971751049161,
+            -0.025111496448516846,
+            -0.011793181300163269,
+            0.0029638127889484167,
+            0.008716314099729061,
+            -0.002309867413714528,
+            -0.027612006291747093,
+            0.009070996195077896,
+            -0.025324305519461632,
+            0.013167575001716614,
+            -3.078356940022786e-06,
+            -0.021032651886343956,
+            0.03624851629137993,
+            0.002489425241947174,
+            0.01603163406252861,
+            0.004083278123289347,
+            -0.007705469615757465,
+            -0.009461146779358387,
+            -0.02649475634098053,
+            0.01350452285259962,
+            -0.0066901920363307,
+            0.0014752560527995229,
+            0.019596189260482788,
+            -0.004171948879957199,
+            -0.004588700365275145,
+            -0.013539991341531277,
+            0.003693128004670143,
+            -0.010064106434583664,
+            0.000359392783138901,
+            0.013096638023853302,
+            -0.0006201395881362259,
+            0.00035828439285978675,
+            -0.0025514946319162846,
+            -0.001351117272861302,
+            -0.005475405603647232,
+            0.0001952137245098129,
+            -0.0015495176194235682,
+            -0.009824695996940136,
+            -0.002011712873354554,
+            0.005120723508298397,
+            -0.005032053217291832,
+            0.019170571118593216,
+            0.015703551471233368,
+            -0.016856269910931587,
+            -0.004788209218531847,
+            -0.013797136023640633,
+            0.008969024755060673,
+            -0.022806061431765556,
+            0.008756215684115887,
+            0.0060162958689033985,
+            0.0159606970846653,
+            0.008552273735404015,
+            -0.0020737822633236647,
+            0.004034509416669607,
+            0.017166616395115852,
+            0.017521297559142113,
+            -0.0012602299684658647,
+            -0.006663591135293245,
+            -0.00728871813043952,
+            -0.007160145789384842,
+            -0.011083817109465599,
+            -0.002338685328140855,
+            -0.009673955850303173,
+            0.023338085040450096,
+            0.015242465771734715,
+            0.022664189338684082,
+            0.008490203879773617,
+            -0.0014785812236368656,
+            -0.007248816546052694,
+            0.016182372346520424,
+            0.0029748964589089155,
+            0.005240428727120161,
+            0.006317775696516037,
+            -0.0034581508953124285,
+            0.004801509436219931,
+            -0.004562098998576403,
+            0.008295129053294659,
+            0.008734048344194889,
+            -0.009638487361371517,
+            0.01835480146110058,
+            0.009266071021556854,
+            -0.01808878965675831,
+            0.008632076904177666,
+            -0.003972440026700497,
+            -0.0028928762767463923,
+            0.006823197938501835,
+            0.0044822958298027515,
+            -0.0036066740285605192,
+            0.008561140857636929,
+            0.007430591154843569,
+            -0.007226648740470409,
+            0.007599065080285072,
+            0.007519261445850134,
+            -0.003460367675870657,
+            0.014648373238742352,
+            0.012130129151046276,
+            -0.0003546821535564959,
+            0.014559702016413212,
+            0.0028729254845529795,
+            -0.008663111366331577,
+            -0.011500569060444832,
+            0.016155771911144257,
+            -0.0035401713103055954,
+            0.007417290471494198,
+            0.0036975613329559565,
+            0.03470565006136894,
+            0.006339943502098322,
+            -0.015934094786643982,
+            -0.03392534703016281,
+            0.003786231856793165,
+            -0.0037064284551888704,
+            0.014533101581037045,
+            -0.013238511048257351,
+            0.006699059158563614,
+            0.01462177187204361,
+            -0.00494338246062398,
+            0.026512490585446358,
+            0.004426876548677683,
+            0.007656700909137726,
+            0.006277874112129211,
+            -0.010658198967576027,
+            0.0006594871520064771,
+            -0.0017290754476562142,
+            0.020145945250988007,
+            0.0016060450579971075,
+            -0.013637528754770756,
+            -0.014949852600693703,
+            -0.014178418554365635,
+            0.006344377063214779,
+            0.013797136023640633,
+            -0.010968545451760292,
+            0.020890777930617332,
+            -0.011544903740286827,
+            0.012342939153313637,
+            0.03342879191040993,
+            0.02454400435090065,
+            0.011748846620321274,
+            0.0008745131781324744,
+            0.0004896276514045894,
+            -0.018230661749839783,
+            -0.009718290530145168,
+            -0.00956755131483078,
+            -0.0016891737468540668,
+            0.0017656520940363407,
+            0.011606973595917225,
+            0.005728116724640131,
+            0.012307470664381981,
+            -3.5606761230155826e-05,
+            -0.019418848678469658,
+            0.016350846737623215,
+            -0.02817949652671814,
+            -0.010170510970056057,
+            -0.007426157593727112,
+            0.029704630374908447,
+            0.024207057431340218,
+            -0.01577448844909668,
+            0.011092684231698513,
+            0.010853273794054985,
+            -0.0056483130902051926,
+            0.013832603581249714,
+            -0.010064106434583664,
+            -0.0009210652206093073,
+            0.009372475557029247,
+            0.011305493302643299,
+            0.006118267308920622,
+            0.020571565255522728,
+            0.024136120453476906,
+            0.027434663847088814,
+            -0.006167036015540361,
+            -0.005994128528982401,
+            0.012112395837903023,
+            0.00996656809002161,
+            0.012679887004196644,
+            0.020323287695646286,
+            0.0006567161763086915,
+            -0.016545921564102173,
+            0.03509579971432686,
+            0.009665088728070259,
+            0.003959139343351126,
+            0.0003106239892076701,
+            0.00403007585555315,
+            0.023621831089258194,
+            -0.02161787636578083,
+            0.004287220537662506,
+            0.0048502786085009575,
+            -0.013841470703482628,
+            0.03759630769491196,
+            0.008481336757540703,
+            0.012520279735326767,
+            0.00751039432361722,
+            0.0023564195726066828,
+            0.00026518033700995147,
+            -0.006193636916577816,
+            -0.00881385151296854,
+            0.016785332933068275,
+            -0.009425678290426731,
+            -0.018443472683429718,
+            -0.00962075311690569,
+            -0.01425822265446186,
+            0.0019285841844975948,
+            -0.007975914515554905,
+            -0.020748905837535858,
+            0.0003084072086494416,
+            -0.012068060226738453,
+            0.019897669553756714,
+            -0.004143130965530872,
+            -0.0021247677505016327,
+            -0.010853273794054985,
+            -0.00016237792442552745,
+            0.005630579311400652,
+            0.013460187241435051,
+            -0.018133124336600304,
+            -0.0067433943040668964,
+            0.01904643140733242,
+            -0.014231621287763119,
+            -0.0127153554931283,
+            -0.026335150003433228,
+            -0.020075010135769844,
+            0.001347792218439281,
+            -0.008441435173153877,
+            0.005147324874997139,
+            0.008583308197557926,
+            0.0050808219239115715,
+            -0.0013644179562106729,
+            0.005018752533942461,
+            -0.021458270028233528,
+            0.00033473127405159175,
+            -0.008995626121759415,
+            0.018904559314250946,
+            0.004899047315120697,
+            0.0037928821984678507,
+            -0.014249355532228947,
+            -0.0029881971422582865,
+            -0.010126175358891487,
+            -0.016412915661931038,
+            0.002062698360532522,
+            -0.03197459504008293,
+            0.013220776803791523,
+            -0.005351266823709011,
+            -0.014001077972352505,
+            -0.002966029569506645,
+            -0.012564615346491337,
+            -0.009044394828379154,
+            0.010338984429836273,
+            -0.002567012095823884,
+            -0.025022825226187706,
+            -0.0003854397509712726,
+            0.0034891855902969837,
+            0.018904559314250946,
+            -0.0035734225530177355,
+            0.0030236653983592987,
+            -0.0004976633936166763,
+            -0.007408423349261284,
+            -0.00913306511938572,
+            0.020571565255522728,
+            0.007333053275942802,
+            -0.01431142445653677,
+            0.006162602454423904,
+            0.0053202323615550995,
+            -0.03137163445353508,
+            -0.0205538310110569,
+            -0.0018365884898230433,
+            0.01528680045157671,
+            -0.0006112725241109729,
+            -0.0010607213480398059,
+            0.003118986263871193,
+            -0.015348870307207108,
+            -0.011553770862519741,
+            -0.0138237364590168,
+            0.006942903157323599,
+            -0.013743933290243149,
+            0.007860642857849598,
+            -0.010002036578953266,
+            0.0067256600596010685,
+            0.0014475465286523104,
+            0.006517284549772739,
+            0.021050386130809784,
+            -0.0007021598285064101,
+            0.003285243408754468,
+            -0.005608411505818367,
+            0.00420963391661644,
+            -0.015073991380631924,
+            0.009700557217001915,
+            0.00876951590180397,
+            -0.0018210711423307657,
+            0.017609968781471252,
+            0.002079324098303914,
+            -0.011323227547109127,
+            0.000915523327421397,
+            0.0010956353507936,
+            0.004646336194127798,
+            -0.018017852678894997,
+            -0.019418848678469658,
+            0.015934094786643982,
+            -0.004134263843297958,
+            0.010569527745246887,
+            -0.00086398352868855,
+            0.008751781657338142,
+            -0.004360373597592115,
+            0.00642418023198843,
+            0.025732189416885376,
+            -0.011598106473684311,
+            0.0012956982245668769,
+            -0.0067389607429504395,
+            0.006636989768594503,
+            -0.007412856910377741,
+            0.02958049066364765,
+            0.011270025745034218,
+            0.008804984390735626,
+            0.004393625073134899,
+            0.024313461035490036,
+            0.0026667665224522352,
+            -0.01279515866190195,
+            0.0017346173990517855,
+            0.033623866736888885,
+            0.0026601161807775497,
+            -0.003872685832902789,
+            0.00988676492124796,
+            0.012245400808751583,
+            0.0009875681716948748,
+            0.006747827865183353,
+            0.00675226142629981,
+            0.019525252282619476,
+            -0.0026046971324831247,
+            0.019330177456140518,
+            -0.004154214635491371,
+            -0.006215804722160101,
+            -0.026991311460733414,
+            0.0056660473346710205,
+            -0.02126319520175457,
+            0.01604936644434929,
+            -0.0027753878384828568,
+            0.016359714791178703,
+            0.012502545490860939,
+            -0.001784494612365961,
+            -0.022912466898560524,
+            -0.004404708743095398,
+            0.016811933368444443,
+            -0.0041608652099967,
+            0.011802048422396183,
+            -0.007536995690315962,
+            0.0015794439241290092,
+            0.007568030618131161,
+            -0.017308488488197327,
+            -0.0037418967112898827,
+            0.0006234647007659078,
+            -0.010507458820939064,
+            0.006588221061974764,
+            0.005963093601167202,
+            -0.008875920437276363,
+            0.017317356541752815,
+            -0.013921274803578854,
+            -0.003786231856793165,
+            -0.002456173999235034,
+            0.019276974722743034,
+            0.0006001886795274913,
+            0.008556706830859184,
+            -0.011571505106985569,
+            -0.015366604551672935,
+            6.151518755359575e-05,
+            0.012688754126429558,
+            -0.005630579311400652,
+            -0.0009875681716948748,
+            0.013593193143606186,
+            -0.008210891857743263,
+            0.032985441386699677,
+            -0.017929183319211006,
+            -0.028285901993513107,
+            -0.005874423310160637,
+            -0.0032187404576689005,
+            -0.010569527745246887,
+            0.003150020958855748,
+            0.013469054363667965,
+            0.0161203034222126,
+            -0.005519740749150515,
+            0.01222766749560833,
+            0.01701587624847889,
+            -0.0019906535744667053,
+            -0.023621831089258194,
+            -0.0023542027920484543,
+            -0.001258013304322958,
+            -0.013371516950428486,
+            -0.0045975674875080585,
+            -0.016705529764294624,
+            0.002338685328140855,
+            0.006707926280796528,
+            -0.003854951588436961,
+            -0.004067760892212391,
+            -0.005235995166003704,
+            0.007000538986176252,
+            0.0021624527871608734,
+            -0.019117368385195732,
+            0.00560397794470191,
+            -0.017335090786218643,
+            0.016874004155397415,
+            -0.005156191997230053,
+            -0.009000059217214584,
+            0.021582409739494324,
+            -0.008494637906551361,
+            -0.0056660473346710205,
+            0.001044095610268414,
+            0.0255903173238039,
+            -0.011225690133869648,
+            0.0056882151402533054,
+            -0.008069018833339214,
+            0.017335090786218643,
+            5.936769593972713e-05,
+            0.005284763872623444,
+            -0.0027244023513048887,
+            0.024242525920271873,
+            -0.006601521745324135,
+            -0.02596273273229599,
+            -0.0030657837633043528,
+            0.02895979769527912,
+            -0.02021688222885132,
+            -0.022132165729999542,
+            0.005280330311506987,
+            -0.000599080347456038,
+            0.0055374749936163425,
+            -0.003584506455808878,
+            0.0056837815791368484,
+            0.009815828874707222,
+            0.000171106425113976,
+            -0.006965070962905884,
+            0.024863218888640404,
+            -0.02771840989589691,
+            0.010143909603357315,
+            0.0047305733896791935,
+            -0.002263315487653017,
+            -0.010933076962828636,
+            0.02135186456143856,
+            0.014444430358707905,
+            0.006450781598687172,
+            -0.0017390509601682425,
+            0.01692720502614975,
+            -0.024082917720079422,
+            0.0021569107193499804,
+            0.01452423445880413,
+            0.01718435063958168,
+            -0.0033051942009478807,
+            -0.008241926319897175,
+            0.002852974459528923,
+            -0.0056571802124381065,
+            -0.01510945986956358,
+            0.011802048422396183,
+            -0.0018022286240011454,
+            0.015419806353747845,
+            -0.030857346951961517,
+            0.013185309246182442,
+            0.0006018512649461627,
+            0.014240488409996033,
+            0.002691150875762105,
+            0.007661134470254183,
+            -0.01333604846149683,
+            0.012121262960135937,
+            -0.017955783754587173,
+            0.00302144861780107,
+            0.017166616395115852,
+            0.020961714908480644,
+            -0.0036177579313516617,
+            -0.0031212028115987778,
+            0.01165130827575922,
+            0.014160685241222382,
+            0.016067100688815117,
+            0.003670960199087858,
+            0.0036864776629954576,
+            -0.017689771950244904,
+            -0.0300238449126482,
+            -0.00582565413787961,
+            -0.0014142950531095266,
+            -0.012369539588689804,
+            -0.0009338115924037993,
+            0.019436581060290337,
+            -0.001129995216615498,
+            0.018762685358524323,
+            -0.03536181151866913,
+            0.008069018833339214,
+            -0.007545862812548876,
+            -0.012484811246395111,
+            -0.0032519919332116842,
+            -0.014914384111762047,
+            -0.015127194114029408,
+            0.0067256600596010685,
+            -0.011500569060444832,
+            0.009124197997152805,
+            -0.01666119322180748,
+            -0.025324305519461632,
+            0.008889221586287022,
+            -0.01125229150056839,
+            -0.013007967732846737,
+            0.007856209762394428,
+            -0.01772524043917656,
+            0.007355221081525087,
+            0.007971481420099735,
+            0.0016016116132959723,
+            -0.00631334213539958,
+            0.002755437046289444,
+            0.0076167993247509,
+            0.003537954529747367,
+            0.015809956938028336,
+            -0.0006428614142350852,
+            -0.00048824213445186615,
+            -0.022717392072081566,
+            -0.007803007494658232,
+            -0.018106523901224136,
+            -0.002391887828707695,
+            -0.007036007009446621,
+            -0.03055586665868759,
+            -0.0031877057626843452,
+            0.010746869258582592,
+            0.003923671320080757,
+            0.014063146896660328,
+            0.004553232342004776,
+            -0.009540949948132038,
+            0.011926187202334404,
+            0.005502006970345974,
+            -0.0005051449406892061,
+            -0.0033561799209564924,
+            -0.005927625577896833,
+            0.008264093659818172,
+            0.00026725855423137546,
+            0.008960157632827759,
+            0.0006661374354735017,
+            0.025874063372612,
+            -0.006894134450703859,
+            -0.00030729881837032735,
+            0.010489724576473236,
+            -0.009736024774610996,
+            -0.0011155862594023347,
+            0.0007852884591557086,
+            0.0057768854312598705,
+            0.012662152759730816,
+            -0.021937090903520584,
+            0.012555748224258423,
+            -0.023621831089258194,
+            -0.011216823011636734,
+            0.007705469615757465,
+            0.0020471811294555664,
+            -0.010995146818459034,
+            0.0026002635713666677,
+            -0.01342471968382597,
+            -0.017343956977128983,
+            -0.00894242338836193,
+            0.017148882150650024,
+            -0.018585344776511192,
+            -0.03048493154346943,
+            -0.004703972022980452,
+            -0.007621232885867357,
+            -0.02303660660982132,
+            0.008343897759914398,
+            -0.024136120453476906,
+            -0.0030635669827461243,
+            0.000919956830330193,
+            0.015561679378151894,
+            -0.0011737762251868844,
+            0.014604037627577782,
+            -0.03064453788101673,
+            -0.0005134578095749021,
+            0.022433646023273468,
+            -0.021334132179617882,
+            -0.015162661671638489,
+            0.0061537353321909904,
+            -0.011048348620533943,
+            -0.019117368385195732,
+            0.011997124180197716,
+            -0.0007004972430877388,
+            0.002314300974830985,
+            -0.019986338913440704,
+            0.034067220985889435,
+            -0.037880051881074905,
+            0.005253729410469532,
+            0.009904499165713787,
+            0.023143010213971138,
+            0.008796117268502712,
+            0.016430649906396866,
+            0.02995290793478489,
+            0.0036554429680109024,
+            0.014391228556632996,
+            0.007763105444610119,
+            -0.005138457752764225,
+            -0.004885746631771326,
+            -0.009221736341714859,
+            -6.1956807257956825e-06,
+            -0.002870708703994751,
+            0.0058877235278487206,
+            0.005395602434873581,
+            0.022575518116354942,
+            -0.022912466898560524,
+            0.027416929602622986,
+            0.02488095313310623,
+            0.006769995670765638,
+            -0.00984242931008339,
+            0.0009886764455586672,
+            -0.013451320119202137,
+            0.023710502311587334,
+            -0.00353352096863091,
+            -0.021334132179617882,
+            -0.0012868312187492847,
+            0.019791264086961746,
+            0.015091725625097752,
+            0.0028884427156299353,
+            -0.009869030676782131,
+            -0.009381342679262161,
+            -0.0002915044024121016,
+            -0.03057360090315342,
+            0.007674435153603554,
+            0.012112395837903023,
+            -0.012076927348971367,
+            -0.01699814200401306,
+            -0.011713378131389618,
+            -0.013460187241435051,
+            0.001321190968155861,
+            -0.007333053275942802,
+            0.015224731527268887,
+            -0.013584326021373272,
+            0.0038660354912281036,
+            0.018283864483237267,
+            -0.0007010514382272959,
+            -0.022380443289875984,
+            -0.021901622414588928,
+            -0.004355940036475658,
+            -0.00917740073055029,
+            0.011145886965095997,
+            -0.008649811148643494,
+            -0.017166616395115852,
+            -0.019365645945072174,
+            0.0011859684018418193,
+            -0.02259325236082077,
+            -0.0016160205705091357,
+            -0.015322268940508366,
+            0.0006932927644811571,
+            -0.013619794510304928,
+            0.0014054280472919345,
+            -0.00885818712413311,
+            0.014586303383111954,
+            -0.0033517463598400354,
+            -0.0021114670671522617,
+            0.022628720849752426,
+            -0.011979389935731888,
+            0.017432628199458122,
+            0.003635491942986846,
+            0.0010668174363672733,
+            -0.006903001572936773,
+            -0.017867112532258034,
+            0.009558684192597866,
+            0.0024761247914284468,
+            -0.007767539005726576,
+            0.018230661749839783,
+            -0.012254267930984497,
+            0.0041231801733374596,
+            -0.0023497692309319973,
+            -0.01103061530739069,
+            0.0055507756769657135,
+            0.007355221081525087,
+            0.0013821519678458571,
+            -0.0013655262300744653,
+            0.003917021211236715,
+            -0.010117308236658573,
+            0.012218800373375416,
+            0.01492325123399496,
+            0.00881828460842371,
+            0.004480078816413879,
+            -0.026991311460733414,
+            0.006393145769834518,
+            0.023001138120889664,
+            -0.004515547305345535,
+            -0.0018498890567570925,
+            -0.0002811133163049817,
+            0.014683840796351433,
+            0.022841529920697212,
+            -0.007723203860223293,
+            0.009939967654645443,
+            -0.031141093000769615,
+            0.03349972888827324,
+            0.01629764400422573,
+            -0.013965609483420849,
+            -0.023675033822655678,
+            0.02376370318233967,
+            -0.01125229150056839,
+            0.0029748964589089155,
+            -0.011606973595917225,
+            -0.015517343766987324,
+            -0.024614941328763962,
+            0.01737942546606064,
+            0.015162661671638489,
+            0.007426157593727112,
+            -0.014471031725406647,
+            0.03064453788101673,
+            0.0006894134567119181,
+            0.004650769755244255,
+            -0.007324186619371176,
+            0.005351266823709011,
+            0.021546941250562668,
+            -0.003529087407514453,
+            -0.0031810556538403034,
+            -0.005258162971585989,
+            -0.01541093923151493,
+            0.0070936428382992744,
+            -0.014586303383111954,
+            0.0031345034949481487,
+            -0.010738002136349678,
+            -0.00030480496934615076,
+            0.009062129072844982,
+            0.027434663847088814,
+            0.008951290510594845,
+            0.011926187202334404,
+            0.023710502311587334,
+            -0.014497633092105389,
+            -0.010268048383295536,
+            -0.009558684192597866,
+            0.014125216752290726,
+            0.016368580982089043,
+            0.01984446682035923,
+            -0.0009166316594928503,
+            0.01622670888900757,
+            -0.013442453928291798,
+            -0.009718290530145168,
+            -0.009931100532412529,
+            -0.01744149439036846,
+            0.0007132436148822308,
+            0.00978922750800848,
+            -0.0026445987168699503,
+            -0.005169492214918137,
+            -0.011722245253622532,
+            0.012830627150833607,
+            -0.0015118326991796494,
+            -0.007372955325990915,
+            -0.010897609405219555,
+            0.01510945986956358,
+            -0.005293630994856358,
+            -0.004686237778514624,
+            0.016075968742370605,
+            0.023533159866929054,
+            -0.01993313618004322,
+            0.0026645497418940067,
+            -0.013406985439360142,
+            0.004267269745469093,
+            -0.007971481420099735,
+            0.031212029978632927,
+            -0.017920315265655518,
+            0.0205538310110569,
+            0.0014575219247490168,
+            -0.001433137571439147,
+            -0.009762626141309738,
+            0.0561816543340683,
+            -0.0012702054809778929,
+            -0.028853392228484154,
+            0.024207057431340218,
+            0.007927145808935165,
+            -0.012724222615361214,
+            -0.002011712873354554,
+            0.01262668427079916,
+            0.015650350600481033,
+            0.004085495136678219,
+            -0.011935054324567318,
+            -0.007275417447090149,
+            -0.007004972547292709,
+            -0.017601102590560913,
+            0.030520398169755936,
+            -0.01895776018500328,
+            0.004868012387305498,
+            0.011358696036040783,
+            0.021777484565973282,
+            -0.004686237778514624,
+            -0.01510945986956358,
+            0.00225888192653656,
+            0.007319753058254719,
+            -0.024685878306627274,
+            -0.004198549780994654,
+            0.00016542596858926117,
+            -0.0025315438397228718,
+            -0.004218501038849354,
+            -0.020429691299796104,
+            0.02383464016020298,
+            0.00984242931008339,
+            -0.00750596122816205,
+            -0.017929183319211006,
+            -0.01700701005756855,
+            -0.010844406671822071,
+            -0.017831645905971527,
+            -0.002586962888017297,
+            0.017973517999053,
+            0.016803067177534103,
+            0.001548409229144454,
+            0.013930141925811768,
+            -0.0055463421158492565,
+            0.0049566831439733505,
+            -0.010427655652165413,
+            0.012307470664381981,
+            -0.01753903180360794,
+            0.0038704690523445606,
+            0.01975579559803009,
+            -0.007297585252672434,
+            0.01948978379368782,
+            0.020677968859672546,
+            0.0067433943040668964,
+            -0.008623209781944752,
+            0.00010411861148895696,
+            0.0024207057431340218,
+            0.024047449231147766,
+            -0.0016115870093926787,
+            -0.002083757659420371,
+            0.0024362229742109776,
+            0.007807441055774689,
+            0.028906594961881638,
+            0.009860163554549217,
+            0.008184290491044521,
+            0.025182433426380157,
+            -0.0031677549704909325,
+            -0.009159666486084461,
+            -0.0055419085547327995,
+            -0.014958719722926617,
+            0.0020216882694512606,
+            0.005958660040050745,
+            0.003815049771219492,
+            -0.0029881971422582865,
+            -0.0038616019301116467,
+            -0.001410969882272184,
+            -0.01886909082531929,
+            0.010693666525185108,
+            0.0016670060576871037,
+            0.0020649151410907507,
+            0.02835683710873127,
+            -0.012272002175450325,
+            0.008312863297760487,
+            0.01788484677672386,
+            -0.003032532287761569,
+            0.00930153951048851,
+            0.013043436221778393,
+            -0.008095620200037956,
+            0.008964591659605503,
+            -0.010640464723110199,
+            -0.01853214204311371,
+            -0.004074411001056433,
+            -0.0021790785249322653,
+            0.022185368463397026,
+            -0.010915343649685383,
+            -0.02356862835586071,
+            0.025271102786064148,
+            0.009860163554549217,
+            -0.022894732654094696,
+            -0.009603018872439861,
+            -0.00011146163888042793,
+            0.00801138300448656,
+            0.00962075311690569,
+            0.022274039685726166,
+            -0.01566808484494686,
+            -0.007670001592487097,
+            0.0003970777615904808,
+            0.008804984390735626,
+            0.019170571118593216,
+            0.0070892092771828175,
+            -0.005001018289476633,
+            -0.0011915103532373905,
+            0.01173997949808836,
+            -0.012662152759730816,
+            -0.023320350795984268,
+            0.007355221081525087,
+            0.00723994942381978,
+            -0.006787729449570179,
+            0.011881851591169834,
+            -0.013575458899140358,
+            -0.02399424836039543,
+            -0.012990233488380909,
+            0.004398058634251356,
+            -0.023621831089258194,
+            -0.01518926303833723,
+            -0.0027199687901884317,
+            -0.025874063372612,
+            -0.005151758436113596,
+            0.008335030637681484,
+            -0.007749804761260748,
+            0.009141932241618633,
+            0.004484512377530336,
+            0.006574920378625393,
+            -0.002575879218056798,
+            -0.0026290814857929945,
+            -9.774541103979573e-05,
+            0.0006384278531186283,
+            -0.002926127752289176,
+            0.0009011143120005727,
+            -0.021937090903520584,
+            0.009939967654645443,
+            -0.008707446977496147,
+            0.010480857454240322,
+            -0.009221736341714859,
+            -0.005377868190407753,
+            -0.015898628160357475,
+            0.016013899818062782,
+            -0.018762685358524323,
+            0.01995087042450905,
+            -1.6495836462127045e-05,
+            0.006406446453183889,
+            -0.0002840228262357414,
+            0.018408004194498062,
+            -0.014417828992009163,
+            -0.007607932202517986,
+            0.011562637984752655,
+            0.006428613793104887,
+            0.024473069235682487,
+            -0.01728188805282116,
+            0.0019540770445019007,
+            0.013655262999236584,
+            0.011837516911327839,
+            0.01218333188444376,
+            -0.00036299502244219184,
+            -0.019330177456140518,
+            0.011456233449280262,
+            0.0007354112458415329,
+            0.005240428727120161,
+            -0.004205200355499983,
+            0.014515367336571217,
+            0.020181413739919662,
+            -0.021387333050370216,
+            0.015357737429440022,
+            0.0009304864797741175,
+            0.00236972002312541,
+            0.012121262960135937,
+            0.005612845066934824,
+            -0.01824839599430561,
+            -0.014151818118989468,
+            0.012928164564073086,
+            0.0066901920363307,
+            -0.028197230771183968,
+            0.024331195279955864,
+            -0.014426696114242077,
+            -0.0004627493617590517,
+            0.010613863356411457,
+            -0.001321190968155861,
+            -0.024685878306627274,
+            -0.00837493222206831,
+            0.000817431544419378,
+            -0.004912347998470068,
+            -0.008286261931061745,
+            0.0030524833127856255,
+            0.018301598727703094,
+            0.009647354483604431,
+            0.0017789526609703898,
+            0.020571565255522728,
+            0.005732550285756588,
+            0.012582349590957165,
+            0.03873129189014435,
+            -0.001021927921101451,
+            0.004471211694180965,
+            -0.02019914798438549,
+            0.011580372229218483,
+            0.011385297402739525,
+            -0.006056197918951511,
+            -0.01528680045157671,
+            -0.025377508252859116,
+            -0.002099274890497327,
+            -0.009780360385775566,
+            0.010862140916287899,
+            0.0018709483556449413,
+            -0.011527169495821,
+            0.004300521221011877,
+            0.00701827323064208,
+            -0.010046372190117836,
+            -0.012759690172970295,
+            0.01904643140733242,
+            0.009266071021556854,
+            0.008778383024036884,
+            0.0008296237210743129,
+            -0.001856539398431778,
+            -0.01528680045157671,
+            0.014302557334303856,
+            -0.020979449152946472,
+            -0.002135851653292775,
+            0.00805571861565113,
+            0.004114313051104546,
+            -0.010959678329527378,
+            0.0057724518701434135,
+            -0.009629620239138603,
+            0.0011582588776946068,
+            -0.00793157983571291,
+            0.035397280007600784,
+            -0.004114313051104546,
+            0.0050896890461444855,
+            0.010401054285466671,
+            -0.00304583297111094,
+            0.006583787500858307,
+            -0.012467077933251858,
+            -0.008836018852889538,
+            0.013247378170490265,
+            0.02612234093248844,
+            -0.010392187163233757,
+            -0.019738061353564262,
+            0.0009947726503014565,
+            0.003639925504103303,
+            -0.00837049912661314,
+            -0.0006694625481031835,
+            0.010995146818459034,
+            0.014417828992009163,
+            -0.008587741293013096,
+            -0.006459648720920086,
+            0.024739079177379608,
+            0.013477921485900879,
+            -0.004351506475359201,
+            -0.016891736537218094,
+            0.009022227488458157,
+            0.0002779267088044435,
+            0.006942903157323599,
+            0.0063798450864851475,
+            -0.007151278667151928,
+            -0.004522197414189577,
+            0.023231681436300278,
+            -0.011004013940691948,
+            -0.002135851653292775,
+            -0.01094194408506155,
+            -0.016492720693349838,
+            -0.022717392072081566,
+            0.002875142265111208,
+            -0.024136120453476906,
+            -4.776432615472004e-05,
+            0.025501646101474762,
+            -0.0016969323623925447,
+            -0.009833562187850475,
+            0.0018576477887108922,
+            -0.016891736537218094,
+            0.012857227586209774,
+            6.400904385372996e-05,
+            -0.001757893362082541,
+            -0.0013788269134238362,
+            0.027133185416460037,
+            0.03816379979252815,
+            0.02481001615524292,
+            -0.002146935323253274,
+            0.01658139005303383,
+            -0.007953747175633907,
+            0.012068060226738453,
+            -0.0054488047026097775,
+            0.01391240768134594,
+            0.015428673475980759,
+            0.01116362027823925,
+            0.004335989244282246,
+            -0.004992151167243719,
+            -0.006433047354221344,
+            0.006526151672005653,
+            -0.008663111366331577,
+            0.007758671883493662,
+            0.0223272405564785,
+            0.0027776046190410852,
+            0.01537547167390585,
+            0.012059193104505539,
+            -0.011332094669342041,
+            0.016457252204418182,
+            -0.01728188805282116,
+            0.0027111016679555178,
+            -0.016058234497904778,
+            -0.0028640583623200655,
+            0.035822898149490356,
+            0.015703551471233368,
+            -0.0027088848873972893,
+            -0.0050852554850280285,
+            -0.007137978449463844,
+            0.008184290491044521,
+            -0.019525252282619476,
+            -0.0007836258737370372,
+            0.00467737065628171,
+            0.0001997858053073287,
+            -0.019330177456140518,
+            0.025022825226187706,
+            0.0056660473346710205,
+            -0.02092624641954899,
+            0.009328140877187252,
+            -0.015029655769467354,
+            0.012564615346491337,
+            0.020270084962248802,
+            0.0011560420971363783,
+            0.01045425608754158,
+            0.00598082784563303,
+            0.009425678290426731,
+            -0.0026445987168699503,
+            -0.00044169012107886374,
+            -0.01711341366171837,
+            -0.00751039432361722,
+            -0.0014708224916830659,
+            0.012812892906367779,
+            -0.003662093309685588,
+            0.00984242931008339,
+            -0.010064106434583664,
+            0.009266071021556854,
+            -0.0030125814955681562,
+            -0.005107422824949026,
+            -0.013433586806058884,
+            0.003036965848878026,
+            0.001497423741966486,
+            0.012919297441840172,
+            -0.007173446472734213,
+            -0.0029638127889484167,
+            -0.022256305441260338,
+            0.007120244204998016,
+            0.011677909642457962,
+            -0.01788484677672386,
+            0.017787309363484383,
+            0.004797075875103474,
+            -0.014107482507824898,
+            -0.01023257989436388,
+            0.011562637984752655,
+            -0.0019540770445019007,
+            -0.009381342679262161,
+            0.012546881102025509,
+            -0.018177460879087448,
+            -0.009736024774610996,
+            7.412302693410311e-06,
+            0.0205538310110569,
+            0.0022012460976839066,
+            0.010560660623013973,
+            -0.007696602493524551,
+            0.00886262021958828,
+            0.002527110278606415,
+            0.0010834431741386652,
+            0.010773470625281334,
+            -4.464700032258406e-05,
+            0.010365585796535015,
+            0.007625666446983814,
+            -0.014009945094585419,
+            -0.006707926280796528,
+            -0.021103588864207268,
+            -0.001193727133795619,
+            -0.001488556619733572,
+            -0.014027679339051247,
+            0.0068497988395392895,
+            -0.0012070277007296681,
+            0.006938469596207142,
+            0.016882870346307755,
+            -0.004743873607367277,
+            -0.011500569060444832,
+            0.0004724477184936404,
+            0.011935054324567318,
+            0.006836498621851206,
+            -0.004664070438593626,
+            -0.014497633092105389,
+            0.015747888013720512,
+            0.010613863356411457,
+            -0.0005594556569121778,
+            0.0050852554850280285,
+            -0.01879815384745598,
+            -0.01904643140733242,
+            -0.014568569138646126,
+            -0.0050675212405622005,
+            0.010046372190117836,
+            0.004205200355499983,
+            0.0033273620065301657,
+            -0.0014386795228347182,
+            0.0023076506331562996,
+            -0.00571481604129076,
+            -0.00014935444050934166,
+            0.03057360090315342,
+            -0.0015805523144081235,
+            -0.0003696453059092164,
+            0.012103528715670109,
+            0.014905516989529133,
+            0.012812892906367779,
+            -0.005151758436113596,
+            0.012653285637497902,
+            0.00907986331731081,
+            0.010480857454240322,
+            0.01709567941725254,
+            -0.02693810872733593,
+            0.02135186456143856,
+            -0.011261158622801304,
+            -0.010631597600877285,
+            0.018106523901224136,
+            0.003500269493088126,
+            -0.0022400394082069397,
+            -0.002502725925296545,
+            0.0059098913334310055,
+            0.01674986444413662,
+            -0.003901503747329116,
+            -0.017849380150437355,
+            0.0004815918509848416,
+            -0.013123239390552044,
+            -0.00751039432361722,
+            -0.017503563314676285,
+            -0.017476962879300117,
+            0.0054266368970274925,
+            -0.0028396740090101957,
+            0.008844885975122452,
+            0.003846084466204047,
+            -0.021050386130809784,
+            -0.010463123209774494,
+            -0.005741417407989502,
+            -0.001937451190315187,
+            0.06781522929668427,
+            -0.005169492214918137,
+            0.018008986487984657,
+            -0.0013566592242568731,
+            0.006220238283276558,
+            0.0070936428382992744,
+            -0.01851440779864788,
+            -0.006681324914097786,
+            -0.005626145750284195,
+            -0.03383667767047882,
+            0.006162602454423904,
+            0.013735066168010235,
+            -0.007749804761260748,
+            0.005267030093818903,
+            0.0003879335999954492,
+            -0.020890777930617332,
+            -0.020766640082001686,
+            -0.006996105425059795,
+            0.001219219877384603,
+            -0.008663111366331577,
+            0.021546941250562668,
+            -0.007944880053400993,
+            -0.0011349829146638513,
+            0.006167036015540361,
+            0.004531064536422491,
+            -0.0015739020891487598,
+            -0.017202084884047508,
+            0.005630579311400652,
+            -2.2150315999169834e-05,
+            -0.03630171716213226,
+            0.0036975613329559565,
+            -0.0022810494992882013,
+            -0.012334072031080723,
+            -0.006632556207478046,
+            0.011855251155793667,
+            0.016457252204418182,
+            -0.009611885994672775,
+            0.01975579559803009,
+            -0.020075010135769844,
+            0.0014741476625204086,
+            0.007594631519168615,
+            0.011083817109465599,
+            -0.004322688560932875,
+            -0.002511593047529459,
+            -0.020075010135769844,
+            -0.0007210022886283696,
+            0.029988376423716545,
+            -0.03591156750917435,
+            0.0038106164429336786,
+            -0.010445388965308666,
+            -0.004178598988801241,
+            -0.007816308178007603,
+            0.01842573843896389,
+            -0.002203462878242135,
+            -0.025625785812735558,
+            0.017521297559142113,
+            0.005484272725880146,
+            0.002493858803063631,
+            0.0016492720460519195,
+            -0.022486848756670952,
+            0.004699538461863995,
+            -0.022983403876423836,
+            -0.0008013600017875433,
+            -0.008237493224442005,
+            -0.018674015998840332,
+            0.011509436182677746,
+            -0.00984242931008339,
+            0.010525193065404892,
+            -0.024739079177379608,
+            0.006189203355461359,
+            0.0028241565451025963,
+            -0.012786291539669037,
+            -0.011713378131389618,
+            -0.006446348037570715,
+            0.0054399375803768635,
+            -0.024685878306627274,
+            0.0008146605687215924,
+            0.01657252386212349,
+            0.015570546500384808,
+            0.005958660040050745,
+            0.0057901861146092415,
+            -0.006916301790624857,
+            -0.01710454747080803,
+            0.003584506455808878,
+            -0.0038992869667708874,
+            -0.022894732654094696,
+            0.0013622011756524444,
+            -0.0022267389576882124,
+            0.006521718110889196,
+            0.023533159866929054,
+            0.012812892906367779,
+            0.021919356659054756,
+            0.008481336757540703,
+            0.014763644896447659,
+            0.005847821943461895,
+            0.024934155866503716,
+            0.0042428853921592236,
+            -0.023338085040450096,
+            -0.004136480391025543,
+            -0.000975375936832279,
+            0.0014176202239468694,
+            0.014072014018893242,
+            -0.0063887122087180614,
+            -0.024065183475613594,
+            -0.03890863060951233,
+            -0.0058832899667322636,
+            0.013664130121469498,
+            0.00757246371358633,
+            -0.008303996175527573,
+            0.006809897255152464,
+            -0.0019197171786800027,
+            -0.0035468214191496372,
+            0.0027842549607157707,
+            0.0026845005340874195,
+            -0.020979449152946472,
+            0.028924329206347466,
+            0.014728176407516003,
+            0.046818044036626816,
+            0.03204553201794624,
+            0.009904499165713787,
+            -0.003105685580521822,
+            0.01950751803815365,
+            0.001809987355954945,
+            -0.01186411827802658,
+            0.016528187319636345,
+            0.022965669631958008,
+            -0.02498735673725605,
+            0.02126319520175457,
+            -0.008703012950718403,
+            -0.00784290861338377,
+            -0.00407884456217289,
+            -0.011935054324567318,
+            0.0170868132263422,
+            0.031673114746809006,
+            -0.006583787500858307,
+            0.000389319087844342,
+            -0.02417158894240856,
+            0.01710454747080803,
+            -0.010604996234178543,
+            0.016856269910931587,
+            -0.007244382984936237,
+            0.045044634491205215,
+            0.009665088728070259,
+            0.006557186134159565,
+            0.0035623388830572367,
+            -0.011066082864999771,
+            0.0006689083529636264,
+            0.0055507756769657135,
+            -0.002966029569506645,
+            0.004420226439833641,
+            0.019826732575893402,
+            -0.0053246659226715565,
+            0.023001138120889664,
+            0.005107422824949026,
+            -0.009549817070364952,
+            0.004167515318840742,
+            0.014861182309687138,
+            0.002815289655700326,
+            0.020128212869167328,
+            -0.006220238283276558,
+            -0.013353782705962658,
+            -0.0026290814857929945,
+            0.002706668106839061,
+            0.003090168349444866,
+            -0.0019518602639436722,
+            -0.006685758475214243,
+            -0.001582769094966352,
+            -0.01647498644888401,
+            -0.005608411505818367,
+            -0.018017852678894997,
+            -0.01311437226831913,
+            0.002930561313405633,
+            -0.013814869336783886,
+            0.01824839599430561,
+            0.029349947348237038,
+            -0.004087711684405804,
+            -0.0026645497418940067,
+            0.018283864483237267,
+            -0.014825713820755482,
+            -0.013442453928291798,
+            0.004236234817653894,
+            0.010551794432103634,
+            0.005630579311400652,
+            0.003850518027320504,
+            0.03587609902024269,
+            0.003948055673390627,
+            0.010374452918767929,
+            -0.013273979537189007,
+            -0.02372823655605316,
+            -0.000808564480394125,
+            -0.0043670241720974445,
+            -0.008428134955465794,
+            0.00013272871728986502,
+            -0.009470012970268726,
+            -0.009363608434796333,
+            0.01646611839532852,
+            0.01568581722676754,
+            -0.016758732497692108,
+            -0.021582409739494324,
+            0.0001651488710194826,
+            0.0019540770445019007,
+            -0.0037795817479491234,
+            -0.0311942957341671,
+            -0.011722245253622532
+          ],
+          "title": "Preventing Token Dropping with No-Token-Left-Behind"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "73de5974-272d-474b-8a31-f59c835d8011",
+      "type": "next",
+      "source": {
+        "id": "44ddb1e4-8508-4bcf-9d46-226ebb75db2f",
+        "properties": {
+          "page_content": "B Preventing Token Dropping with No-Token-Left-Behind 29\n\n",
+          "keyphrases": [
+            "Preventing Token Dropping",
+            "No-Token-Left-Behind"
+          ],
+          "embedding": [
+            0.025537114590406418,
+            -0.019454315304756165,
+            -0.016811933368444443,
+            0.022433646023273468,
+            -0.0004264498711563647,
+            -0.0028263733256608248,
+            -0.020252350717782974,
+            0.05167718976736069,
+            -0.02958049066364765,
+            0.00320987356826663,
+            -0.031212029978632927,
+            -0.00974489189684391,
+            0.01094194408506155,
+            -0.049123477190732956,
+            -0.010197111405432224,
+            0.038660354912281036,
+            -0.002062698360532522,
+            0.016102569177746773,
+            0.016013899818062782,
+            -0.003059133654460311,
+            0.010631597600877285,
+            0.013460187241435051,
+            -0.01815972663462162,
+            0.039192378520965576,
+            -0.005244862288236618,
+            0.027842549607157707,
+            -0.010347851552069187,
+            0.005479839164763689,
+            -0.04174608737230301,
+            0.018115390092134476,
+            0.008871487341821194,
+            0.009558684192597866,
+            -0.0016282127471640706,
+            -0.052067339420318604,
+            -0.011119285598397255,
+            0.012387273833155632,
+            0.04429979994893074,
+            0.015579413622617722,
+            -0.07313545793294907,
+            -0.028835657984018326,
+            -0.0168474018573761,
+            0.003110119141638279,
+            -0.05912551283836365,
+            0.0050763883627951145,
+            0.019170571118593216,
+            0.025129230692982674,
+            -0.004646336194127798,
+            -0.01648385263979435,
+            -0.0052049607038497925,
+            0.0036554429680109024,
+            0.017237553372979164,
+            -0.042207177728414536,
+            0.011943921446800232,
+            -0.02472134679555893,
+            0.015464141964912415,
+            0.015251332893967628,
+            0.0375608392059803,
+            0.018283864483237267,
+            0.007479359861463308,
+            -0.01447989884763956,
+            0.01440009567886591,
+            0.012804025784134865,
+            0.034670181572437286,
+            0.004735006485134363,
+            -0.0008046851144172251,
+            -0.008299562148749828,
+            -0.011722245253622532,
+            -0.023072073236107826,
+            0.017343956977128983,
+            -0.021405067294836044,
+            -0.050719548016786575,
+            0.02541297674179077,
+            -0.003491402370855212,
+            -0.006645856890827417,
+            0.023781437426805496,
+            0.010489724576473236,
+            0.07838475704193115,
+            0.05848708748817444,
+            0.0006278982618823647,
+            -0.000581900414545089,
+            -0.005510874092578888,
+            0.021937090903520584,
+            -0.001937451190315187,
+            0.014329158701002598,
+            0.041426874697208405,
+            -0.00011402476957300678,
+            -0.046995386481285095,
+            -0.04749194160103798,
+            0.04476088657975197,
+            -0.021334132179617882,
+            -0.0384475439786911,
+            -0.02028781920671463,
+            -0.02064250037074089,
+            -0.04121406748890877,
+            -0.00320987356826663,
+            0.0588417686522007,
+            0.017228685319423676,
+            -0.019383380189538002,
+            -0.022256305441260338,
+            0.005936492700129747,
+            -0.025022825226187706,
+            -0.0026623329613357782,
+            -0.05469198897480965,
+            0.01869175024330616,
+            0.03926331549882889,
+            0.012874961830675602,
+            -0.0023054340854287148,
+            0.009425678290426731,
+            -0.022185368463397026,
+            0.0798034816980362,
+            -0.018815888091921806,
+            0.009212869219481945,
+            -0.020411957055330276,
+            0.04468994960188866,
+            -0.031247496604919434,
+            -0.016785332933068275,
+            -0.00320987356826663,
+            -0.034067220985889435,
+            -0.036638665944337845,
+            0.011669042520225048,
+            0.0056571802124381065,
+            -0.015269066207110882,
+            -0.012617817148566246,
+            0.013229643926024437,
+            -0.027186386287212372,
+            0.008844885975122452,
+            0.027150917798280716,
+            0.019738061353564262,
+            -0.021405067294836044,
+            -0.01984446682035923,
+            -0.014072014018893242,
+            -0.013149840757250786,
+            -0.058628957718610764,
+            0.012724222615361214,
+            -0.04224264249205589,
+            0.019879935309290886,
+            0.019436581060290337,
+            0.011988257057964802,
+            -0.026211010292172432,
+            0.04163968563079834,
+            -0.022912466898560524,
+            0.007377388887107372,
+            0.028392305597662926,
+            0.04745647311210632,
+            -0.012431609444320202,
+            0.009913366287946701,
+            -0.008441435173153877,
+            0.015747888013720512,
+            -0.018620813265442848,
+            0.021511472761631012,
+            0.02498735673725605,
+            0.004351506475359201,
+            0.014630638994276524,
+            0.03674507141113281,
+            0.005346833262592554,
+            0.01222766749560833,
+            -0.007204481400549412,
+            -0.02152920700609684,
+            0.007794140372425318,
+            -0.007452758494764566,
+            0.016155771911144257,
+            0.007075909059494734,
+            0.03532634302973747,
+            -0.00984242931008339,
+            0.024029714986681938,
+            0.006654724013060331,
+            0.03472338244318962,
+            0.013265112414956093,
+            -0.08313749730587006,
+            0.02347995899617672,
+            -0.0027244023513048887,
+            0.031034687533974648,
+            -0.010223712772130966,
+            0.03021891973912716,
+            -0.007612365763634443,
+            -0.020341021940112114,
+            -0.0064995503053069115,
+            -0.00956755131483078,
+            0.016599124297499657,
+            0.011411897838115692,
+            -0.011713378131389618,
+            0.03709975257515907,
+            -0.009053261950612068,
+            -0.024650409817695618,
+            0.022522317245602608,
+            -0.003440416883677244,
+            0.019206037744879723,
+            -0.0175301656126976,
+            -0.008538972586393356,
+            0.009434545412659645,
+            -0.02516469918191433,
+            0.018461205065250397,
+            0.01779617741703987,
+            -0.00274656992405653,
+            -0.01745922863483429,
+            0.0034071654081344604,
+            -0.014905516989529133,
+            -0.025022825226187706,
+            0.01737942546606064,
+            -0.00849907100200653,
+            -0.0026157808024436235,
+            0.0168474018573761,
+            -0.029917439445853233,
+            -0.00881828460842371,
+            0.0021292013116180897,
+            0.021121321246027946,
+            0.0020128211472183466,
+            0.04458354786038399,
+            0.007740938104689121,
+            0.01790258102118969,
+            -0.0017235336126759648,
+            0.0003020340227521956,
+            -0.007173446472734213,
+            0.012502545490860939,
+            -0.019418848678469658,
+            -0.01103061530739069,
+            -0.03181498870253563,
+            0.018053321167826653,
+            -0.022185368463397026,
+            0.039192378520965576,
+            0.03630171716213226,
+            -0.014976453967392445,
+            -0.022717392072081566,
+            -0.0026645497418940067,
+            -0.003846084466204047,
+            -0.017388291656970978,
+            0.018408004194498062,
+            0.025217900052666664,
+            -0.0026157808024436235,
+            -0.00030425077420659363,
+            0.010826672427356243,
+            0.02401198074221611,
+            -0.05298951268196106,
+            0.0066192555241286755,
+            -0.013309448026120663,
+            -0.009700557217001915,
+            -0.034084953367710114,
+            0.010933076962828636,
+            0.019791264086961746,
+            0.01067593228071928,
+            0.013814869336783886,
+            0.044654481112957,
+            -0.07682415097951889,
+            -0.0029549456667155027,
+            -0.03552141785621643,
+            0.06685758382081985,
+            -0.026636630296707153,
+            0.03678053990006447,
+            -0.03527313843369484,
+            -0.0006434156093746424,
+            -0.018141992390155792,
+            0.017672037705779076,
+            -0.010968545451760292,
+            0.007763105444610119,
+            0.02481001615524292,
+            0.06412653625011444,
+            -0.006934036035090685,
+            -0.025093762204051018,
+            0.001193727133795619,
+            -0.02489868737757206,
+            0.020695703104138374,
+            0.03848301246762276,
+            0.025483911857008934,
+            -0.01815972663462162,
+            0.008995626121759415,
+            -0.009922233410179615,
+            0.016155771911144257,
+            -0.014178418554365635,
+            0.025661254301667213,
+            -0.021688813343644142,
+            0.027062248438596725,
+            0.016368580982089043,
+            -0.03748990222811699,
+            -0.03177952021360397,
+            0.0035667724441736937,
+            0.003726379247382283,
+            -0.00783404242247343,
+            0.0015772271435707808,
+            0.005289197433739901,
+            0.034067220985889435,
+            -0.014870049431920052,
+            0.00894242338836193,
+            0.003912587650120258,
+            -0.033535197377204895,
+            -0.007878377102315426,
+            -0.035237669944763184,
+            -0.0650487095117569,
+            0.03780911862850189,
+            -0.08483996987342834,
+            -0.012910430319607258,
+            -0.017831645905971527,
+            0.01191732008010149,
+            -0.011057215742766857,
+            0.009039961732923985,
+            0.06430387496948242,
+            0.01702474243938923,
+            0.010471990332007408,
+            0.0023896710481494665,
+            -0.04312935099005699,
+            -0.0010507458355277777,
+            -0.007581330835819244,
+            -0.01966712437570095,
+            0.001189293572679162,
+            -0.029988376423716545,
+            -0.03589383512735367,
+            -0.017831645905971527,
+            0.0019474267028272152,
+            -0.027682941406965256,
+            -0.05171265825629234,
+            -0.002861841581761837,
+            0.020802108570933342,
+            -0.02383464016020298,
+            -0.02755880355834961,
+            -0.0328080989420414,
+            -0.040185488760471344,
+            -0.006588221061974764,
+            0.005807920359075069,
+            0.04075298085808754,
+            -0.023072073236107826,
+            -0.02995290793478489,
+            0.02037649042904377,
+            -0.017964651808142662,
+            -0.04057563841342926,
+            0.011243424378335476,
+            0.006925168912857771,
+            -0.0393342487514019,
+            0.027186386287212372,
+            0.04330668970942497,
+            -0.025749923661351204,
+            -0.00035828439285978675,
+            -0.014533101581037045,
+            -0.01913510262966156,
+            0.000581900414545089,
+            -0.029740098863840103,
+            -0.02709771692752838,
+            -0.03126523271203041,
+            0.0014054280472919345,
+            -0.018674015998840332,
+            -0.0025248934980481863,
+            0.007563597057014704,
+            -0.020500628277659416,
+            -0.03731256350874901,
+            0.009815828874707222,
+            0.0009986519580706954,
+            0.0007620124379172921,
+            -0.016253309324383736,
+            0.01440009567886591,
+            -0.021068120375275612,
+            -0.005524174310266972,
+            -0.002821939764544368,
+            0.0011904019629582763,
+            0.01320304349064827,
+            0.05518854409456253,
+            -0.026636630296707153,
+            0.0665738433599472,
+            -0.009141932241618633,
+            -0.01913510262966156,
+            0.008920256048440933,
+            0.006428613793104887,
+            -0.03731256350874901,
+            0.006348810624331236,
+            -0.02605140395462513,
+            0.009532082825899124,
+            0.004101012367755175,
+            -0.0023497692309319973,
+            0.00025395795819349587,
+            -0.012662152759730816,
+            0.01067593228071928,
+            -0.0247036125510931,
+            -0.014905516989529133,
+            -0.018727216869592667,
+            -0.011819782666862011,
+            0.034404169768095016,
+            -0.012724222615361214,
+            0.02417158894240856,
+            -0.027789346873760223,
+            0.03741896525025368,
+            0.015623748302459717,
+            -0.018141992390155792,
+            0.01977352984249592,
+            -0.022557783871889114,
+            -0.021688813343644142,
+            0.024419866502285004,
+            -0.01594296284019947,
+            0.02170654758810997,
+            0.022256305441260338,
+            0.01904643140733242,
+            0.0366741344332695,
+            -0.012600083835422993,
+            0.032116468995809555,
+            -0.03559235483407974,
+            0.02234497480094433,
+            -0.07061721384525299,
+            0.006096099503338337,
+            0.017317356541752815,
+            0.036886945366859436,
+            -0.02835683710873127,
+            0.00717788003385067,
+            0.021227726712822914,
+            -0.013398118317127228,
+            -0.007408423349261284,
+            0.010746869258582592,
+            0.00956755131483078,
+            -0.034510571509599686,
+            -0.012289736419916153,
+            0.028764722868800163,
+            -0.050896890461444855,
+            -0.013637528754770756,
+            -0.0027753878384828568,
+            0.007483793422579765,
+            0.0027776046190410852,
+            -0.047704748809337616,
+            -0.004488945938646793,
+            -0.02878245711326599,
+            -0.007918278686702251,
+            0.010152776725590229,
+            0.02427799254655838,
+            0.021671079099178314,
+            -0.01966712437570095,
+            -0.03284356743097305,
+            0.006645856890827417,
+            0.029545024037361145,
+            0.009993169456720352,
+            0.026441553607583046,
+            0.07192953675985336,
+            -0.036195311695337296,
+            -0.01502078864723444,
+            0.009833562187850475,
+            -0.00788724422454834,
+            -0.016359714791178703,
+            0.03475885093212128,
+            0.03279036656022072,
+            -0.03692241013050079,
+            -0.011057215742766857,
+            -0.020075010135769844,
+            -0.029527289792895317,
+            -0.05774225294589996,
+            0.0027709542773663998,
+            -0.0008872595499269664,
+            -0.02081984281539917,
+            0.00427170330658555,
+            0.014187285676598549,
+            -0.055685099214315414,
+            0.005280330311506987,
+            -0.016528187319636345,
+            -0.021458270028233528,
+            -0.028587380424141884,
+            -0.03569876030087471,
+            -0.013176442123949528,
+            0.008973458781838417,
+            -0.037702713161706924,
+            0.007523695006966591,
+            -0.022575518116354942,
+            0.030502665787935257,
+            0.008574441075325012,
+            -0.008303996175527573,
+            0.025217900052666664,
+            -0.021156789734959602,
+            0.013362649828195572,
+            -0.006867533084005117,
+            -0.04146234318614006,
+            -0.003145587397739291,
+            0.0023608531337231398,
+            -0.011021748185157776,
+            0.0055463421158492565,
+            0.023515427485108376,
+            -0.02791348472237587,
+            0.029970642179250717,
+            0.0017833862220868468,
+            -0.0061404346488416195,
+            0.0061315675266087055,
+            -0.01710454747080803,
+            -0.011748846620321274,
+            -0.021227726712822914,
+            0.0023165177553892136,
+            -0.0034382001031190157,
+            -0.04011455178260803,
+            -0.04394511878490448,
+            -0.048520516604185104,
+            -0.007541429251432419,
+            -0.014001077972352505,
+            -0.00903552770614624,
+            -0.0083217304199934,
+            -0.005462104920297861,
+            -0.004034509416669607,
+            -0.0031123359221965075,
+            -0.007027139887213707,
+            -0.010294649749994278,
+            -0.009372475557029247,
+            -0.004280570428818464,
+            0.02294793538749218,
+            -0.019808998331427574,
+            -0.019188303500413895,
+            -0.016599124297499657,
+            0.00631334213539958,
+            -0.024508535861968994,
+            -0.03545048087835312,
+            0.003611107589676976,
+            0.038412075489759445,
+            0.03373027220368385,
+            0.011465100571513176,
+            -0.018026720732450485,
+            0.0026290814857929945,
+            0.019596189260482788,
+            0.022699657827615738,
+            -0.05909004807472229,
+            -0.01332718227058649,
+            -0.0008351656142622232,
+            -0.02410065196454525,
+            -0.0027798213995993137,
+            -0.015384337864816189,
+            0.0010551793966442347,
+            -0.006526151672005653,
+            -0.0005539137637242675,
+            -0.007155712228268385,
+            -0.020802108570933342,
+            -6.234647298697382e-05,
+            0.0027111016679555178,
+            -0.0008235276327468455,
+            -0.002099274890497327,
+            0.016616858541965485,
+            0.019649391993880272,
+            -0.0164040494710207,
+            0.00757246371358633,
+            -0.011216823011636734,
+            0.018567610532045364,
+            0.03137163445353508,
+            -0.0016437300946563482,
+            0.020713437348604202,
+            -0.025306571274995804,
+            0.003990174271166325,
+            0.022735126316547394,
+            -0.034670181572437286,
+            -0.027168652042746544,
+            -0.0014575219247490168,
+            0.014453297480940819,
+            -0.0006240188959054649,
+            0.01984446682035923,
+            -0.06370091438293457,
+            -0.00586998974904418,
+            -0.025058293715119362,
+            0.00200284575112164,
+            -0.016439517959952354,
+            0.035503681749105453,
+            -0.003141153836622834,
+            0.012245400808751583,
+            0.03259528800845146,
+            0.006415313575416803,
+            0.016714395955204964,
+            -0.01779617741703987,
+            0.014408962801098824,
+            0.0354682132601738,
+            -0.047101788222789764,
+            -0.011881851591169834,
+            -0.006645856890827417,
+            -0.044512610882520676,
+            0.013371516950428486,
+            0.011296626180410385,
+            0.019702592864632607,
+            0.027594272047281265,
+            -0.024384398013353348,
+            0.03165538236498833,
+            -0.007324186619371176,
+            -0.009363608434796333,
+            0.026246478781104088,
+            -0.004805942997336388,
+            0.019472049549221992,
+            0.006362111307680607,
+            -0.0072133480571210384,
+            0.010569527745246887,
+            0.012245400808751583,
+            -0.011216823011636734,
+            0.006876400206238031,
+            -0.029296746477484703,
+            -8.721578342374414e-05,
+            0.012316337786614895,
+            -0.010516325943171978,
+            0.002083757659420371,
+            -0.03390761464834213,
+            0.025448445230722427,
+            -0.013140973635017872,
+            -0.01267101988196373,
+            0.03461697697639465,
+            0.004225151147693396,
+            0.016599124297499657,
+            0.03447510674595833,
+            0.0026889340952038765,
+            -0.009000059217214584,
+            0.006597088184207678,
+            -0.001766760484315455,
+            0.007341920398175716,
+            0.018993228673934937,
+            0.00934587512165308,
+            -0.023550894111394882,
+            -0.02374597080051899,
+            -0.013699597679078579,
+            0.008836018852889538,
+            0.01001090370118618,
+            0.0020848659332841635,
+            0.003216523677110672,
+            0.008707446977496147,
+            0.009159666486084461,
+            0.015792222693562508,
+            -0.013353782705962658,
+            -0.0031256363727152348,
+            0.01763656921684742,
+            -0.03773818165063858,
+            0.0036465758457779884,
+            -0.010285782627761364,
+            0.0010391079122200608,
+            0.00934587512165308,
+            -0.00047992929467000067,
+            0.0473145991563797,
+            0.005643879529088736,
+            0.008405966684222221,
+            0.010436521843075752,
+            0.019365645945072174,
+            -0.006774429231882095,
+            -0.017964651808142662,
+            0.005967527162283659,
+            0.026477022096514702,
+            -0.029456352815032005,
+            0.03766724467277527,
+            -0.00445569446310401,
+            -0.001095081097446382,
+            0.002797555411234498,
+            -0.006539452355355024,
+            -0.018833622336387634,
+            -0.00405446020886302,
+            -0.03509579971432686,
+            -0.009833562187850475,
+            0.01098627969622612,
+            0.027665207162499428,
+            -0.023941045626997948,
+            -0.02622874453663826,
+            0.03138937056064606,
+            0.023107541725039482,
+            -0.0005497573292814195,
+            0.030963752418756485,
+            -0.028232699260115623,
+            0.020624767988920212,
+            0.026920374482870102,
+            0.018815888091921806,
+            0.03748990222811699,
+            -0.030981486663222313,
+            -0.011713378131389618,
+            0.022167634218931198,
+            0.038944099098443985,
+            0.038837697356939316,
+            0.0012380623957142234,
+            -0.03748990222811699,
+            0.02347995899617672,
+            0.03422682732343674,
+            0.03426229581236839,
+            0.00675226142629981,
+            -0.02339128777384758,
+            -0.009638487361371517,
+            0.013469054363667965,
+            0.0005458780215121806,
+            0.010090706869959831,
+            -0.010684799402952194,
+            0.008295129053294659,
+            -0.013593193143606186,
+            -0.016625726595520973,
+            -0.0005375651526264846,
+            -0.018815888091921806,
+            -0.033801209181547165,
+            -0.030857346951961517,
+            0.016944939270615578,
+            -0.014417828992009163,
+            -0.03470565006136894,
+            -0.019879935309290886,
+            -0.013185309246182442,
+            -0.005847821943461895,
+            -0.04834317788481712,
+            -0.02800215594470501,
+            -0.020429691299796104,
+            0.027310525998473167,
+            -0.01879815384745598,
+            -0.007865076884627342,
+            -0.02727505750954151,
+            -0.003212090115994215,
+            0.0007664459408260882,
+            0.0384475439786911,
+            -0.013894673436880112,
+            -0.0001251778594451025,
+            -0.02488095313310623,
+            0.013238511048257351,
+            -0.02984650246798992,
+            0.0065039838664233685,
+            -0.015827691182494164,
+            -0.02638835273683071,
+            0.011305493302643299,
+            -0.011491701938211918,
+            -0.002347552450373769,
+            -0.011172487400472164,
+            0.01377053465694189,
+            -0.005151758436113596,
+            -0.010658198967576027,
+            0.007723203860223293,
+            0.0014763644430786371,
+            -0.008552273735404015,
+            -0.02259325236082077,
+            0.04962003231048584,
+            0.0036598765291273594,
+            -0.0062645734287798405,
+            -0.0013056736206635833,
+            -0.012387273833155632,
+            -0.006291174795478582,
+            -0.008303996175527573,
+            -0.019099634140729904,
+            0.002118117408826947,
+            0.017778443172574043,
+            0.026175541803240776,
+            0.026884907856583595,
+            -0.001146066701039672,
+            0.015419806353747845,
+            -0.00015406505553983152,
+            -0.04600227624177933,
+            -0.022717392072081566,
+            -0.03169085085391998,
+            -0.03169085085391998,
+            0.03197459504008293,
+            -0.019649391993880272,
+            -0.0030480497516691685,
+            -0.011872985400259495,
+            0.02463267557322979,
+            0.02913713827729225,
+            0.0003538508608471602,
+            -0.0030347490683197975,
+            -0.0001627935707801953,
+            0.0027842549607157707,
+            -0.01045425608754158,
+            0.0033827810548245907,
+            -0.016058234497904778,
+            0.024845484644174576,
+            -5.153975143912248e-05,
+            0.01727301999926567,
+            0.032471150159835815,
+            -0.02913713827729225,
+            0.033801209181547165,
+            0.010188245214521885,
+            0.008729614317417145,
+            0.015925228595733643,
+            -0.008135521784424782,
+            -0.0011011771857738495,
+            0.035822898149490356,
+            0.02073117159307003,
+            -0.031673114746809006,
+            0.01457743626087904,
+            -0.02372823655605316,
+            -0.026884907856583595,
+            0.020890777930617332,
+            -0.009806961752474308,
+            -0.028924329206347466,
+            0.029704630374908447,
+            0.020181413739919662,
+            0.026104606688022614,
+            0.032985441386699677,
+            -0.015818823128938675,
+            -0.006654724013060331,
+            -0.002751003485172987,
+            0.0004120409139432013,
+            -0.006344377063214779,
+            0.012600083835422993,
+            0.004087711684405804,
+            -0.0014054280472919345,
+            -0.001630429527722299,
+            0.018478939309716225,
+            0.0006295607890933752,
+            0.011598106473684311,
+            0.006277874112129211,
+            -0.00751039432361722,
+            -0.029030734673142433,
+            -0.008051284588873386,
+            0.011633574962615967,
+            0.03731256350874901,
+            -0.034865256398916245,
+            0.00859217531979084,
+            -0.022699657827615738,
+            -0.019188303500413895,
+            0.01737942546606064,
+            -0.03271942958235741,
+            0.004770474974066019,
+            0.016776464879512787,
+            0.0006539452006109059,
+            0.013903540559113026,
+            0.009425678290426731,
+            0.011598106473684311,
+            0.012963633053004742,
+            0.0057724518701434135,
+            0.002511593047529459,
+            0.015357737429440022,
+            0.012910430319607258,
+            0.07249703258275986,
+            -0.025040559470653534,
+            0.00045665327343158424,
+            0.03802192583680153,
+            -0.027168652042746544,
+            -0.003965789917856455,
+            0.03396081551909447,
+            -0.006468515843153,
+            -0.02401198074221611,
+            0.029562756419181824,
+            -0.01860307902097702,
+            -0.02816176228225231,
+            -0.016794199123978615,
+            -0.030272122472524643,
+            -0.013983343727886677,
+            0.012165597639977932,
+            0.01762770302593708,
+            -0.011145886965095997,
+            0.025005090981721878,
+            0.0008401533705182374,
+            -0.019915401935577393,
+            -0.008787250146269798,
+            0.02259325236082077,
+            -0.012316337786614895,
+            -0.006694625597447157,
+            0.0025780959986150265,
+            0.012520279735326767,
+            0.00962075311690569,
+            0.015340003184974194,
+            -0.041604217141866684,
+            -0.010303516872227192,
+            0.011119285598397255,
+            -0.0031034687999635935,
+            0.0030037143733352423,
+            0.0276474729180336,
+            0.01717548258602619,
+            0.011828649789094925,
+            -0.03163764625787735,
+            0.027168652042746544,
+            0.01853214204311371,
+            -0.023409022018313408,
+            0.03827020525932312,
+            -0.03936971724033356,
+            -0.025005090981721878,
+            -0.006676891352981329,
+            0.006109400186687708,
+            0.0049566831439733505,
+            -0.00631334213539958,
+            -0.010064106434583664,
+            -0.024916421622037888,
+            -0.03279036656022072,
+            -0.019010962918400764,
+            -0.0005746959359385073,
+            0.015295667573809624,
+            0.011802048422396183,
+            0.005235995166003704,
+            0.008286261931061745,
+            0.04057563841342926,
+            -0.0033606134820729494,
+            -0.002994847483932972,
+            -0.031939126551151276,
+            -0.012289736419916153,
+            0.03165538236498833,
+            0.027044514194130898,
+            -0.005945359356701374,
+            0.007204481400549412,
+            -0.005351266823709011,
+            0.00907986331731081,
+            0.0013245161389932036,
+            -0.013486788608133793,
+            0.026264213025569916,
+            0.04632148891687393,
+            -0.0022533400915563107,
+            -0.03279036656022072,
+            -0.006167036015540361,
+            -0.026884907856583595,
+            -0.01566808484494686,
+            0.012733088806271553,
+            -0.026689831167459488,
+            0.0027088848873972893,
+            -0.011988257057964802,
+            -0.04366137459874153,
+            -0.008756215684115887,
+            -0.009443412534892559,
+            0.01753903180360794,
+            -0.00571481604129076,
+            -0.0062468391843140125,
+            -0.0061227004043757915,
+            -0.0053335330449044704,
+            0.00350913661532104,
+            -0.0031832722015678883,
+            -0.03341105952858925,
+            0.020677968859672546,
+            -0.01869175024330616,
+            -0.023089807480573654,
+            -0.02002180740237236,
+            -0.003923671320080757,
+            0.002082649152725935,
+            -0.005572943482547998,
+            -0.010179378092288971,
+            0.014870049431920052,
+            -0.019383380189538002,
+            -0.024526270106434822,
+            -0.03837660700082779,
+            0.012014857493340969,
+            0.010046372190117836,
+            -0.011553770862519741,
+            -0.01186411827802658,
+            -0.0016160205705091357,
+            -0.004899047315120697,
+            0.01568581722676754,
+            -0.02346222475171089,
+            0.0049522495828568935,
+            0.029367681592702866,
+            0.018008986487984657,
+            -0.026477022096514702,
+            0.012050325982272625,
+            -0.01897549442946911,
+            0.02913713827729225,
+            -0.027079982683062553,
+            -0.011819782666862011,
+            0.02443760074675083,
+            0.0022965669631958008,
+            -0.0027044513262808323,
+            -0.025803126394748688,
+            0.015393204987049103,
+            -0.002391887828707695,
+            0.019915401935577393,
+            -0.021511472761631012,
+            0.02773614414036274,
+            0.004276136867702007,
+            -0.030360791832208633,
+            0.015047390013933182,
+            0.026104606688022614,
+            -0.011367563158273697,
+            0.02365729957818985,
+            0.01957845501601696,
+            -0.004899047315120697,
+            -0.010729135014116764,
+            0.0175301656126976,
+            -0.008188724517822266,
+            0.023799171671271324,
+            -0.0407884456217289,
+            0.0064951167441904545,
+            -0.044867292046546936,
+            -0.036106642335653305,
+            0.003602240700274706,
+            -0.006237972527742386,
+            -0.020979449152946472,
+            -0.040220957249403,
+            0.003229824360460043,
+            0.009922233410179615,
+            0.01842573843896389,
+            0.027842549607157707,
+            -0.0218838881701231,
+            0.02878245711326599,
+            0.0034226826392114162,
+            -0.027487866580486298,
+            0.0070936428382992744,
+            -0.0022810494992882013,
+            -0.032205138355493546,
+            -0.009239469654858112,
+            -0.022025762125849724,
+            -0.005586243700236082,
+            -0.049832843244075775,
+            0.013052303344011307,
+            -0.011074949987232685,
+            0.015934094786643982,
+            -0.017521297559142113,
+            -0.02833910472691059,
+            -0.0022489065304398537,
+            0.037773650139570236,
+            0.003945839125663042,
+            -0.036816008388996124,
+            -0.02924354374408722,
+            0.019010962918400764,
+            -0.02984650246798992,
+            -0.007071475498378277,
+            0.005936492700129747,
+            0.020252350717782974,
+            -0.010773470625281334,
+            -0.020411957055330276,
+            -0.010037505067884922,
+            -0.006065064575523138,
+            -0.00823305919766426,
+            0.013070037588477135,
+            0.04163968563079834,
+            -0.004659636877477169,
+            0.010356718674302101,
+            0.0027155352290719748,
+            0.04603774473071098,
+            -0.001809987355954945,
+            -0.018656281754374504,
+            0.028587380424141884,
+            -0.0014364627422764897,
+            -0.0059143248945474625,
+            0.016164638102054596,
+            -0.013238511048257351,
+            0.002917260630056262,
+            -0.0300238449126482,
+            0.028906594961881638,
+            0.011970522813498974,
+            -0.026867173612117767,
+            -0.00939907692372799,
+            -0.02782481536269188,
+            -0.02791348472237587,
+            -0.05022299289703369,
+            -0.011092684231698513,
+            -0.01700701005756855,
+            0.04518650472164154,
+            -0.05593337491154671,
+            -0.01728188805282116,
+            0.006198070477694273,
+            -0.006920735351741314,
+            -0.024224791675806046,
+            0.006645856890827417,
+            -0.01320304349064827,
+            -0.022185368463397026,
+            0.011145886965095997,
+            -0.033269185572862625,
+            -0.003879335941746831,
+            -0.015907494351267815,
+            -0.007315319497138262,
+            0.0017301838379353285,
+            -0.0014353543519973755,
+            0.06015409156680107,
+            0.009957700967788696,
+            0.02862284891307354,
+            -0.009443412534892559,
+            0.009780360385775566,
+            0.026104606688022614,
+            0.020766640082001686,
+            0.023444490507245064,
+            0.009904499165713787,
+            -0.0047305733896791935,
+            -0.019702592864632607,
+            0.0024184889625757933,
+            0.010241447016596794,
+            0.010072972625494003,
+            0.005621712189167738,
+            -0.028410039842128754,
+            -0.02126319520175457,
+            -0.006916301790624857,
+            -0.008667545393109322,
+            -0.004380324389785528,
+            0.031158827245235443,
+            -0.016794199123978615,
+            -0.029367681592702866,
+            -0.023284882307052612,
+            -0.005036486312747002,
+            -0.04635695740580559,
+            0.0003114552528131753,
+            0.015827691182494164,
+            0.002108142012730241,
+            5.7462664699414745e-05,
+            -0.0026623329613357782,
+            0.004570966120809317,
+            -0.011119285598397255,
+            0.014072014018893242,
+            0.0034515007864683867,
+            0.0058034867979586124,
+            -0.00996656809002161,
+            -0.0005685998476110399,
+            0.0024916420225054026,
+            0.013921274803578854,
+            0.006597088184207678,
+            -0.02259325236082077,
+            -0.04802396148443222,
+            -0.027416929602622986,
+            -0.0029615960083901882,
+            -0.013273979537189007,
+            -0.000881163461599499,
+            -0.010853273794054985,
+            -0.0049655502662062645,
+            0.0034382001031190157,
+            -0.0008839344372972846,
+            -0.011855251155793667,
+            -0.008073452860116959,
+            -0.004245101939886808,
+            -0.008937990292906761,
+            -0.023072073236107826,
+            0.023355819284915924,
+            -0.009443412534892559,
+            -0.01621784083545208,
+            -0.020890777930617332,
+            -0.007501527667045593,
+            -0.023426756262779236,
+            -0.010587261989712715,
+            0.009673955850303173,
+            0.006557186134159565,
+            -0.016412915661931038,
+            -0.006716793403029442,
+            -0.004449044354259968,
+            0.008286261931061745,
+            -3.664586984086782e-05,
+            -0.006162602454423904,
+            -0.029083935543894768,
+            0.007523695006966591,
+            0.008711880072951317,
+            0.0083217304199934,
+            -0.0015317834913730621,
+            -0.002019471488893032,
+            0.01470157504081726,
+            0.08065471798181534,
+            -0.012697621248662472,
+            0.009363608434796333,
+            0.0008722963975742459,
+            -0.032400213181972504,
+            0.01244934368878603,
+            -0.03848301246762276,
+            -0.001967377495020628,
+            0.02230950817465782,
+            0.008574441075325012,
+            0.024260258302092552,
+            -0.011606973595917225,
+            0.010090706869959831,
+            0.04827224090695381,
+            -0.006348810624331236,
+            0.0029793300200253725,
+            0.0026800669729709625,
+            2.926820525317453e-05,
+            0.0056616137735545635,
+            0.007031573448330164,
+            -0.0015417590038850904,
+            -0.009682822972536087,
+            -0.019347911700606346,
+            -0.016191240400075912,
+            0.01780504360795021,
+            -0.013850337825715542,
+            -0.0012258702190592885,
+            -0.03177952021360397,
+            -0.004198549780994654,
+            -0.010002036578953266,
+            0.015632616356015205,
+            -0.03057360090315342,
+            0.00029787758830934763,
+            0.012298603542149067,
+            -0.035663291811943054,
+            0.004311604890972376,
+            0.001890899264253676,
+            0.0007936013280414045,
+            0.01835480146110058,
+            -0.003152237506583333,
+            -0.007723203860223293,
+            0.022132165729999542,
+            0.0022522315848618746,
+            -0.013628661632537842,
+            -0.022841529920697212,
+            0.002338685328140855,
+            -0.015987297520041466,
+            -0.03231154382228851,
+            -0.019720327109098434,
+            -0.00039347552228718996,
+            -0.009097597561776638,
+            0.023054338991642,
+            -0.007049307692795992,
+            -0.028729254379868507,
+            -0.01922377198934555,
+            0.007816308178007603,
+            0.004668503999710083,
+            -0.017140014097094536,
+            -0.005834521260112524,
+            0.021812953054904938,
+            -0.03738350048661232,
+            -0.021103588864207268,
+            -0.009443412534892559,
+            0.002622431144118309,
+            0.004641902633011341,
+            -0.008969024755060673,
+            -0.017255285754799843,
+            0.0024362229742109776,
+            0.004056677222251892,
+            -0.015304534696042538,
+            -0.0016359713627025485,
+            0.028285901993513107,
+            0.019418848678469658,
+            0.004522197414189577,
+            -0.00288179237395525,
+            0.004067760892212391,
+            -0.0051916600205004215,
+            0.006211371161043644,
+            0.0025936132296919823,
+            -0.026618896052241325,
+            -0.008290695026516914,
+            0.04848505184054375,
+            0.003999041393399239,
+            0.0057680183090269566,
+            0.004797075875103474,
+            -0.007084776181727648,
+            -0.014541967771947384,
+            0.0005589014617726207,
+            0.020057275891304016,
+            0.013415852561593056,
+            0.022752860561013222,
+            0.015428673475980759,
+            -0.007918278686702251,
+            -0.004167515318840742,
+            0.0002563132729846984,
+            -0.02285926416516304,
+            0.036816008388996124,
+            0.0016891737468540668,
+            -0.0025337606202811003,
+            0.005626145750284195,
+            -0.0063887122087180614,
+            0.00031976812169887125,
+            0.025501646101474762,
+            -0.004087711684405804,
+            -0.014666106551885605,
+            0.02321394719183445,
+            0.013477921485900879,
+            -0.007257683668285608,
+            -0.034510571509599686,
+            0.008840452879667282,
+            0.007044874131679535,
+            0.01585429161787033,
+            -0.01737942546606064,
+            -0.003799532540142536,
+            -0.009115330874919891,
+            -0.013318315148353577,
+            0.03660319745540619,
+            -0.00723994942381978,
+            -0.006597088184207678,
+            -0.0010075189638882875,
+            -0.030449463054537773,
+            0.00680103013291955,
+            0.02383464016020298,
+            0.00819759164005518,
+            -0.027345994487404823,
+            -0.0068542324006557465,
+            0.004083278123289347,
+            -0.020270084962248802,
+            -0.010853273794054985,
+            0.005107422824949026,
+            -0.023781437426805496,
+            0.011278891935944557,
+            -0.0028175064362585545,
+            -0.0030702173244208097,
+            0.0018432388314977288,
+            0.0020050625316798687,
+            0.04784662276506424,
+            -0.02940315008163452,
+            0.01904643140733242,
+            -0.003897070186212659,
+            0.009239469654858112,
+            0.00018468410416971892,
+            0.011190221644937992,
+            -0.007270983885973692,
+            -0.007869509980082512,
+            -0.00019646066357381642,
+            -0.03745443373918533,
+            -0.0007570247398689389,
+            0.010401054285466671,
+            0.038305673748254776,
+            -0.007754238322377205,
+            -0.007115810643881559,
+            0.023621831089258194,
+            0.011314360424876213,
+            0.009106464684009552,
+            -0.008410400710999966,
+            -0.005989694967865944,
+            -0.007563597057014704,
+            -0.0011288868263363838,
+            -0.012564615346491337,
+            0.029296746477484703,
+            -0.00907986331731081,
+            0.003713078796863556,
+            -0.030077045783400536,
+            0.02933221310377121,
+            0.0056571802124381065,
+            0.02401198074221611,
+            -0.004273919854313135,
+            0.0019607271533459425,
+            0.005821220576763153,
+            0.025998201221227646,
+            -0.009274938143789768,
+            -0.023089807480573654,
+            0.0012979150051251054,
+            0.01763656921684742,
+            0.04387418180704117,
+            0.030609069392085075,
+            -0.017476962879300117,
+            0.023444490507245064,
+            -0.03596476837992668,
+            0.0003726933500729501,
+            0.006198070477694273,
+            -0.006570486817508936,
+            -0.0053202323615550995,
+            -0.014027679339051247,
+            0.0004497258923947811,
+            0.004637469071894884,
+            -0.01798238418996334,
+            -0.008175423368811607,
+            -0.010223712772130966,
+            0.01518926303833723,
+            0.01799125224351883,
+            0.004903480876237154,
+            -0.0010851057013496757,
+            7.121352246031165e-05,
+            -0.021724281832575798,
+            0.010081839747726917,
+            -0.026547959074378014,
+            0.03438643366098404,
+            -0.009088730439543724,
+            -0.00010841358744073659,
+            0.007355221081525087,
+            -0.01279515866190195,
+            0.01762770302593708,
+            0.01541093923151493,
+            -0.0019751363433897495,
+            0.0059187584556639194,
+            -0.039653465151786804,
+            0.020890777930617332,
+            -0.009709424339234829,
+            -0.04571852833032608,
+            -0.0011194655671715736,
+            0.011332094669342041,
+            -0.01621784083545208,
+            0.020358756184577942,
+            0.03365933522582054,
+            -0.01603163406252861,
+            0.0136020602658391,
+            -0.015517343766987324,
+            -0.009195134975016117,
+            0.027878016233444214,
+            0.006100533064454794,
+            -0.015818823128938675,
+            0.007975914515554905,
+            0.0057680183090269566,
+            0.02046515978872776,
+            -0.008525672368705273,
+            -2.9960943720652722e-05,
+            0.004695104900747538,
+            -0.010170510970056057,
+            0.040220957249403,
+            5.850177331012674e-05,
+            0.019436581060290337,
+            -0.01895776018500328,
+            -0.010906476527452469,
+            -0.006472949404269457,
+            0.003294110530987382,
+            -0.0067300936207175255,
+            -0.010321250185370445,
+            -0.018727216869592667,
+            -0.009239469654858112,
+            -0.00939907692372799,
+            0.031495776027441025,
+            -0.0014785812236368656,
+            -0.011704511009156704,
+            -0.0002506328164599836,
+            -0.013619794510304928,
+            -0.004240668378770351,
+            -0.029314478859305382,
+            -0.0054177697747945786,
+            -0.007324186619371176,
+            -0.01966712437570095,
+            -0.006295608356595039,
+            -0.01328284665942192,
+            0.005861122626811266,
+            0.014790245331823826,
+            0.008565573953092098,
+            0.008379366248846054,
+            -0.0040079085156321526,
+            -0.0021713196765631437,
+            -0.006827631499618292,
+            -0.037241626530885696,
+            0.017388291656970978,
+            0.0016326463082805276,
+            -0.01586315967142582,
+            0.00886705331504345,
+            -0.03656772896647453,
+            0.03571649268269539,
+            0.02330261655151844,
+            -0.02888886071741581,
+            0.00575915165245533,
+            0.025909531861543655,
+            0.024863218888640404,
+            -0.0036554429680109024,
+            0.009984302334487438,
+            -0.015543945133686066,
+            -0.007412856910377741,
+            -0.0053290994837880135,
+            0.02629968151450157,
+            -0.009727157652378082,
+            0.002292133402079344,
+            0.04206530377268791,
+            -0.0011837516212835908,
+            0.004788209218531847,
+            -0.011376430280506611,
+            0.005360133945941925,
+            -0.008441435173153877,
+            0.0027266191318631172,
+            -0.036354921758174896,
+            -0.006929602473974228,
+            -0.03413815796375275,
+            0.0062601398676633835,
+            0.01191732008010149,
+            -0.005954226478934288,
+            0.02842777408659458,
+            0.002972679678350687,
+            0.002906176960095763,
+            0.022433646023273468,
+            -0.025944998487830162,
+            -0.026796236634254456,
+            0.012378406710922718,
+            0.006951770279556513,
+            0.031123358756303787,
+            0.013637528754770756,
+            0.007222215179353952,
+            0.003555688541382551,
+            -0.0056660473346710205,
+            -0.04518650472164154,
+            0.00819759164005518,
+            -0.013406985439360142,
+            0.024402132257819176,
+            0.0046020010486245155,
+            0.0080601517111063,
+            0.03605344146490097,
+            0.01332718227058649,
+            0.004615301266312599,
+            0.016359714791178703,
+            0.0007514827884733677,
+            0.00819759164005518,
+            0.03004157729446888,
+            0.028658317402005196,
+            -0.007155712228268385,
+            -0.04649883136153221,
+            -0.032027799636125565,
+            0.010383320041000843,
+            -0.0023586363531649113,
+            -0.0006478491122834384,
+            0.010019770823419094,
+            -0.023373553529381752,
+            -0.01033011730760336,
+            0.023515427485108376,
+            -4.7383316996274516e-05,
+            -0.00653058523312211,
+            -0.03156671300530434,
+            -0.012050325982272625,
+            0.01771637424826622,
+            -0.012928164564073086,
+            0.011021748185157776,
+            0.014772512018680573,
+            -0.010871008038520813,
+            -0.015419806353747845,
+            -0.014932118356227875,
+            -0.002702234545722604,
+            0.019986338913440704,
+            0.009860163554549217,
+            -0.020004073157906532,
+            0.022486848756670952,
+            0.016634592786431313,
+            0.0062734405510127544,
+            0.012768557295203209,
+            -0.020571565255522728,
+            0.02638835273683071,
+            -0.013983343727886677,
+            0.014692707918584347,
+            -0.012050325982272625,
+            -0.00686309952288866,
+            0.011890718713402748,
+            0.004101012367755175,
+            -0.0014353543519973755,
+            -0.003888203063979745,
+            0.001440896186977625,
+            -0.012334072031080723,
+            -0.0013622011756524444,
+            -0.020589299499988556,
+            0.009283805266022682,
+            -0.011225690133869648,
+            -0.00047217062092386186,
+            0.023196212947368622,
+            0.01413408387452364,
+            0.003201006446033716,
+            0.0028441075701266527,
+            -0.014045413583517075,
+            -0.01984446682035923,
+            0.0050675212405622005,
+            -0.006308908574283123,
+            -0.0028441075701266527,
+            -0.02073117159307003,
+            -0.0254307109862566,
+            0.017583368346095085,
+            0.03321598470211029,
+            0.016058234497904778,
+            -0.027629738673567772,
+            -0.013876939192414284,
+            -0.004205200355499983,
+            0.014009945094585419,
+            0.011881851591169834,
+            -0.006632556207478046,
+            0.006397579330950975,
+            -0.008193157613277435,
+            -0.0036000239197164774,
+            -0.013433586806058884,
+            0.015996165573596954,
+            0.01711341366171837,
+            0.004468995146453381,
+            0.018301598727703094,
+            0.015632616356015205,
+            0.015446407720446587,
+            -0.030431728810071945,
+            -0.0014043196570128202,
+            0.018283864483237267,
+            -0.010915343649685383,
+            -0.03585836663842201,
+            -0.014692707918584347,
+            0.00228326627984643,
+            0.018461205065250397,
+            -0.0015905278269201517,
+            -0.02135186456143856,
+            -0.014719309285283089,
+            0.0002265255170641467,
+            -0.03373027220368385,
+            0.017335090786218643,
+            0.00018468410416971892,
+            -0.00043060630559921265,
+            0.015526210889220238,
+            0.0074483249336481094,
+            -0.0007398448069579899,
+            0.005107422824949026,
+            -0.019188303500413895,
+            -0.006521718110889196,
+            -0.007545862812548876,
+            0.02959822490811348,
+            -0.00028429992380551994,
+            0.0023209513165056705,
+            -0.005705948919057846,
+            -0.009390209801495075,
+            0.007528128568083048,
+            -0.010383320041000843,
+            0.00022555567556992173,
+            0.012422742322087288,
+            0.025271102786064148,
+            -0.008405966684222221,
+            0.011314360424876213,
+            0.039830803871154785,
+            0.00038350006798282266,
+            -0.03436870127916336,
+            0.00023594676167704165,
+            -0.01435576006770134,
+            -0.005129590630531311,
+            0.009008926339447498,
+            0.007270983885973692,
+            -0.0007564705447293818,
+            -0.001497423741966486,
+            -0.006885267328470945,
+            -0.021919356659054756,
+            -0.012360673397779465,
+            -0.030147982761263847,
+            -0.00988676492124796,
+            -0.0047261398285627365,
+            0.007222215179353952,
+            -0.020482894033193588,
+            0.006477382965385914,
+            -0.01611143723130226,
+            -0.009718290530145168,
+            -0.015242465771734715,
+            -0.002522676717489958,
+            -0.013539991341531277,
+            -0.014391228556632996,
+            0.0070936428382992744,
+            0.006167036015540361,
+            0.00934587512165308,
+            -0.0033716971520334482,
+            -0.009057695046067238,
+            -0.0019197171786800027,
+            -0.012422742322087288,
+            0.014719309285283089,
+            0.00864537712186575,
+            0.01637744903564453,
+            -0.003936972003430128,
+            -0.009115330874919891,
+            0.00031588878482580185,
+            0.01683853566646576,
+            0.012786291539669037,
+            0.010268048383295536,
+            -0.012165597639977932,
+            0.014231621287763119,
+            -0.003688694443553686,
+            0.01700701005756855,
+            -0.004191899672150612,
+            0.010241447016596794,
+            -0.004792642779648304,
+            0.004938948899507523,
+            0.019613923504948616,
+            0.018993228673934937,
+            0.004912347998470068,
+            0.004903480876237154,
+            0.06632556021213531,
+            0.013406985439360142,
+            0.033517464995384216,
+            0.011287759058177471,
+            0.0065039838664233685,
+            -0.01510945986956358,
+            -0.008437002077698708,
+            -0.012422742322087288,
+            0.026743033900856972,
+            0.017246419563889503,
+            0.007270983885973692,
+            -0.018656281754374504,
+            -0.0030236653983592987,
+            0.028658317402005196,
+            0.014657239429652691,
+            0.008219758979976177,
+            0.006286741234362125,
+            0.019454315304756165,
+            0.006082798819988966,
+            0.008366065099835396,
+            0.003460367675870657,
+            0.0060029951855540276,
+            0.006078365258872509,
+            0.015313401818275452,
+            0.012617817148566246,
+            0.01208579447120428,
+            0.005018752533942461,
+            0.00560397794470191,
+            0.02949182130396366,
+            0.014426696114242077,
+            0.029615959152579308,
+            -0.0007819632883183658,
+            0.005710382480174303,
+            0.031939126551151276,
+            0.012147863395512104,
+            0.01421388704329729,
+            0.02614007517695427,
+            -0.0017612185329198837,
+            -0.022486848756670952,
+            0.0037507638335227966,
+            0.02214989997446537,
+            -0.014630638994276524,
+            0.023284882307052612,
+            -0.0033606134820729494,
+            -0.0008224192424677312,
+            -0.026725299656391144,
+            -0.019117368385195732,
+            0.0027776046190410852,
+            -0.027328260242938995,
+            0.009939967654645443,
+            -0.0029926307033747435,
+            -0.007523695006966591,
+            0.005847821943461895,
+            -0.007900545373558998,
+            -0.010516325943171978,
+            0.004562098998576403,
+            0.016874004155397415,
+            -0.005630579311400652,
+            -0.0067389607429504395,
+            0.013300580903887749,
+            -0.000335839664330706,
+            -0.0005372880259528756,
+            0.011819782666862011,
+            -0.00394362211227417,
+            -0.015632616356015205,
+            0.00935474131256342,
+            -0.0013743933523073792,
+            -0.012236534617841244,
+            -0.015606014989316463,
+            0.0013888023095205426,
+            -0.005976394284516573,
+            -0.01569468528032303,
+            0.0029704628977924585,
+            0.007590197958052158,
+            -0.002387454267591238,
+            -0.0031322867143899202,
+            0.007049307692795992,
+            0.012883828952908516,
+            0.011447366327047348,
+            -0.002442873315885663,
+            0.005488706286996603,
+            0.003828350454568863,
+            -0.014178418554365635,
+            -0.010862140916287899,
+            -0.038412075489759445,
+            -0.005816787481307983,
+            0.01922377198934555,
+            0.01860307902097702,
+            0.0006855340907350183,
+            0.003285243408754468,
+            0.009195134975016117,
+            0.006699059158563614,
+            -0.0007453867001459002,
+            0.004801509436219931,
+            -0.025483911857008934,
+            0.002972679678350687,
+            0.012165597639977932,
+            -0.008352764882147312,
+            -0.0007985890260897577,
+            -0.01826613023877144,
+            -0.021316397935152054,
+            0.0041120960377156734,
+            0.003169971751049161,
+            -0.025111496448516846,
+            -0.011793181300163269,
+            0.0029638127889484167,
+            0.008716314099729061,
+            -0.002309867413714528,
+            -0.027612006291747093,
+            0.009070996195077896,
+            -0.025324305519461632,
+            0.013167575001716614,
+            -3.078356940022786e-06,
+            -0.021032651886343956,
+            0.03624851629137993,
+            0.002489425241947174,
+            0.01603163406252861,
+            0.004083278123289347,
+            -0.007705469615757465,
+            -0.009461146779358387,
+            -0.02649475634098053,
+            0.01350452285259962,
+            -0.0066901920363307,
+            0.0014752560527995229,
+            0.019596189260482788,
+            -0.004171948879957199,
+            -0.004588700365275145,
+            -0.013539991341531277,
+            0.003693128004670143,
+            -0.010064106434583664,
+            0.000359392783138901,
+            0.013096638023853302,
+            -0.0006201395881362259,
+            0.00035828439285978675,
+            -0.0025514946319162846,
+            -0.001351117272861302,
+            -0.005475405603647232,
+            0.0001952137245098129,
+            -0.0015495176194235682,
+            -0.009824695996940136,
+            -0.002011712873354554,
+            0.005120723508298397,
+            -0.005032053217291832,
+            0.019170571118593216,
+            0.015703551471233368,
+            -0.016856269910931587,
+            -0.004788209218531847,
+            -0.013797136023640633,
+            0.008969024755060673,
+            -0.022806061431765556,
+            0.008756215684115887,
+            0.0060162958689033985,
+            0.0159606970846653,
+            0.008552273735404015,
+            -0.0020737822633236647,
+            0.004034509416669607,
+            0.017166616395115852,
+            0.017521297559142113,
+            -0.0012602299684658647,
+            -0.006663591135293245,
+            -0.00728871813043952,
+            -0.007160145789384842,
+            -0.011083817109465599,
+            -0.002338685328140855,
+            -0.009673955850303173,
+            0.023338085040450096,
+            0.015242465771734715,
+            0.022664189338684082,
+            0.008490203879773617,
+            -0.0014785812236368656,
+            -0.007248816546052694,
+            0.016182372346520424,
+            0.0029748964589089155,
+            0.005240428727120161,
+            0.006317775696516037,
+            -0.0034581508953124285,
+            0.004801509436219931,
+            -0.004562098998576403,
+            0.008295129053294659,
+            0.008734048344194889,
+            -0.009638487361371517,
+            0.01835480146110058,
+            0.009266071021556854,
+            -0.01808878965675831,
+            0.008632076904177666,
+            -0.003972440026700497,
+            -0.0028928762767463923,
+            0.006823197938501835,
+            0.0044822958298027515,
+            -0.0036066740285605192,
+            0.008561140857636929,
+            0.007430591154843569,
+            -0.007226648740470409,
+            0.007599065080285072,
+            0.007519261445850134,
+            -0.003460367675870657,
+            0.014648373238742352,
+            0.012130129151046276,
+            -0.0003546821535564959,
+            0.014559702016413212,
+            0.0028729254845529795,
+            -0.008663111366331577,
+            -0.011500569060444832,
+            0.016155771911144257,
+            -0.0035401713103055954,
+            0.007417290471494198,
+            0.0036975613329559565,
+            0.03470565006136894,
+            0.006339943502098322,
+            -0.015934094786643982,
+            -0.03392534703016281,
+            0.003786231856793165,
+            -0.0037064284551888704,
+            0.014533101581037045,
+            -0.013238511048257351,
+            0.006699059158563614,
+            0.01462177187204361,
+            -0.00494338246062398,
+            0.026512490585446358,
+            0.004426876548677683,
+            0.007656700909137726,
+            0.006277874112129211,
+            -0.010658198967576027,
+            0.0006594871520064771,
+            -0.0017290754476562142,
+            0.020145945250988007,
+            0.0016060450579971075,
+            -0.013637528754770756,
+            -0.014949852600693703,
+            -0.014178418554365635,
+            0.006344377063214779,
+            0.013797136023640633,
+            -0.010968545451760292,
+            0.020890777930617332,
+            -0.011544903740286827,
+            0.012342939153313637,
+            0.03342879191040993,
+            0.02454400435090065,
+            0.011748846620321274,
+            0.0008745131781324744,
+            0.0004896276514045894,
+            -0.018230661749839783,
+            -0.009718290530145168,
+            -0.00956755131483078,
+            -0.0016891737468540668,
+            0.0017656520940363407,
+            0.011606973595917225,
+            0.005728116724640131,
+            0.012307470664381981,
+            -3.5606761230155826e-05,
+            -0.019418848678469658,
+            0.016350846737623215,
+            -0.02817949652671814,
+            -0.010170510970056057,
+            -0.007426157593727112,
+            0.029704630374908447,
+            0.024207057431340218,
+            -0.01577448844909668,
+            0.011092684231698513,
+            0.010853273794054985,
+            -0.0056483130902051926,
+            0.013832603581249714,
+            -0.010064106434583664,
+            -0.0009210652206093073,
+            0.009372475557029247,
+            0.011305493302643299,
+            0.006118267308920622,
+            0.020571565255522728,
+            0.024136120453476906,
+            0.027434663847088814,
+            -0.006167036015540361,
+            -0.005994128528982401,
+            0.012112395837903023,
+            0.00996656809002161,
+            0.012679887004196644,
+            0.020323287695646286,
+            0.0006567161763086915,
+            -0.016545921564102173,
+            0.03509579971432686,
+            0.009665088728070259,
+            0.003959139343351126,
+            0.0003106239892076701,
+            0.00403007585555315,
+            0.023621831089258194,
+            -0.02161787636578083,
+            0.004287220537662506,
+            0.0048502786085009575,
+            -0.013841470703482628,
+            0.03759630769491196,
+            0.008481336757540703,
+            0.012520279735326767,
+            0.00751039432361722,
+            0.0023564195726066828,
+            0.00026518033700995147,
+            -0.006193636916577816,
+            -0.00881385151296854,
+            0.016785332933068275,
+            -0.009425678290426731,
+            -0.018443472683429718,
+            -0.00962075311690569,
+            -0.01425822265446186,
+            0.0019285841844975948,
+            -0.007975914515554905,
+            -0.020748905837535858,
+            0.0003084072086494416,
+            -0.012068060226738453,
+            0.019897669553756714,
+            -0.004143130965530872,
+            -0.0021247677505016327,
+            -0.010853273794054985,
+            -0.00016237792442552745,
+            0.005630579311400652,
+            0.013460187241435051,
+            -0.018133124336600304,
+            -0.0067433943040668964,
+            0.01904643140733242,
+            -0.014231621287763119,
+            -0.0127153554931283,
+            -0.026335150003433228,
+            -0.020075010135769844,
+            0.001347792218439281,
+            -0.008441435173153877,
+            0.005147324874997139,
+            0.008583308197557926,
+            0.0050808219239115715,
+            -0.0013644179562106729,
+            0.005018752533942461,
+            -0.021458270028233528,
+            0.00033473127405159175,
+            -0.008995626121759415,
+            0.018904559314250946,
+            0.004899047315120697,
+            0.0037928821984678507,
+            -0.014249355532228947,
+            -0.0029881971422582865,
+            -0.010126175358891487,
+            -0.016412915661931038,
+            0.002062698360532522,
+            -0.03197459504008293,
+            0.013220776803791523,
+            -0.005351266823709011,
+            -0.014001077972352505,
+            -0.002966029569506645,
+            -0.012564615346491337,
+            -0.009044394828379154,
+            0.010338984429836273,
+            -0.002567012095823884,
+            -0.025022825226187706,
+            -0.0003854397509712726,
+            0.0034891855902969837,
+            0.018904559314250946,
+            -0.0035734225530177355,
+            0.0030236653983592987,
+            -0.0004976633936166763,
+            -0.007408423349261284,
+            -0.00913306511938572,
+            0.020571565255522728,
+            0.007333053275942802,
+            -0.01431142445653677,
+            0.006162602454423904,
+            0.0053202323615550995,
+            -0.03137163445353508,
+            -0.0205538310110569,
+            -0.0018365884898230433,
+            0.01528680045157671,
+            -0.0006112725241109729,
+            -0.0010607213480398059,
+            0.003118986263871193,
+            -0.015348870307207108,
+            -0.011553770862519741,
+            -0.0138237364590168,
+            0.006942903157323599,
+            -0.013743933290243149,
+            0.007860642857849598,
+            -0.010002036578953266,
+            0.0067256600596010685,
+            0.0014475465286523104,
+            0.006517284549772739,
+            0.021050386130809784,
+            -0.0007021598285064101,
+            0.003285243408754468,
+            -0.005608411505818367,
+            0.00420963391661644,
+            -0.015073991380631924,
+            0.009700557217001915,
+            0.00876951590180397,
+            -0.0018210711423307657,
+            0.017609968781471252,
+            0.002079324098303914,
+            -0.011323227547109127,
+            0.000915523327421397,
+            0.0010956353507936,
+            0.004646336194127798,
+            -0.018017852678894997,
+            -0.019418848678469658,
+            0.015934094786643982,
+            -0.004134263843297958,
+            0.010569527745246887,
+            -0.00086398352868855,
+            0.008751781657338142,
+            -0.004360373597592115,
+            0.00642418023198843,
+            0.025732189416885376,
+            -0.011598106473684311,
+            0.0012956982245668769,
+            -0.0067389607429504395,
+            0.006636989768594503,
+            -0.007412856910377741,
+            0.02958049066364765,
+            0.011270025745034218,
+            0.008804984390735626,
+            0.004393625073134899,
+            0.024313461035490036,
+            0.0026667665224522352,
+            -0.01279515866190195,
+            0.0017346173990517855,
+            0.033623866736888885,
+            0.0026601161807775497,
+            -0.003872685832902789,
+            0.00988676492124796,
+            0.012245400808751583,
+            0.0009875681716948748,
+            0.006747827865183353,
+            0.00675226142629981,
+            0.019525252282619476,
+            -0.0026046971324831247,
+            0.019330177456140518,
+            -0.004154214635491371,
+            -0.006215804722160101,
+            -0.026991311460733414,
+            0.0056660473346710205,
+            -0.02126319520175457,
+            0.01604936644434929,
+            -0.0027753878384828568,
+            0.016359714791178703,
+            0.012502545490860939,
+            -0.001784494612365961,
+            -0.022912466898560524,
+            -0.004404708743095398,
+            0.016811933368444443,
+            -0.0041608652099967,
+            0.011802048422396183,
+            -0.007536995690315962,
+            0.0015794439241290092,
+            0.007568030618131161,
+            -0.017308488488197327,
+            -0.0037418967112898827,
+            0.0006234647007659078,
+            -0.010507458820939064,
+            0.006588221061974764,
+            0.005963093601167202,
+            -0.008875920437276363,
+            0.017317356541752815,
+            -0.013921274803578854,
+            -0.003786231856793165,
+            -0.002456173999235034,
+            0.019276974722743034,
+            0.0006001886795274913,
+            0.008556706830859184,
+            -0.011571505106985569,
+            -0.015366604551672935,
+            6.151518755359575e-05,
+            0.012688754126429558,
+            -0.005630579311400652,
+            -0.0009875681716948748,
+            0.013593193143606186,
+            -0.008210891857743263,
+            0.032985441386699677,
+            -0.017929183319211006,
+            -0.028285901993513107,
+            -0.005874423310160637,
+            -0.0032187404576689005,
+            -0.010569527745246887,
+            0.003150020958855748,
+            0.013469054363667965,
+            0.0161203034222126,
+            -0.005519740749150515,
+            0.01222766749560833,
+            0.01701587624847889,
+            -0.0019906535744667053,
+            -0.023621831089258194,
+            -0.0023542027920484543,
+            -0.001258013304322958,
+            -0.013371516950428486,
+            -0.0045975674875080585,
+            -0.016705529764294624,
+            0.002338685328140855,
+            0.006707926280796528,
+            -0.003854951588436961,
+            -0.004067760892212391,
+            -0.005235995166003704,
+            0.007000538986176252,
+            0.0021624527871608734,
+            -0.019117368385195732,
+            0.00560397794470191,
+            -0.017335090786218643,
+            0.016874004155397415,
+            -0.005156191997230053,
+            -0.009000059217214584,
+            0.021582409739494324,
+            -0.008494637906551361,
+            -0.0056660473346710205,
+            0.001044095610268414,
+            0.0255903173238039,
+            -0.011225690133869648,
+            0.0056882151402533054,
+            -0.008069018833339214,
+            0.017335090786218643,
+            5.936769593972713e-05,
+            0.005284763872623444,
+            -0.0027244023513048887,
+            0.024242525920271873,
+            -0.006601521745324135,
+            -0.02596273273229599,
+            -0.0030657837633043528,
+            0.02895979769527912,
+            -0.02021688222885132,
+            -0.022132165729999542,
+            0.005280330311506987,
+            -0.000599080347456038,
+            0.0055374749936163425,
+            -0.003584506455808878,
+            0.0056837815791368484,
+            0.009815828874707222,
+            0.000171106425113976,
+            -0.006965070962905884,
+            0.024863218888640404,
+            -0.02771840989589691,
+            0.010143909603357315,
+            0.0047305733896791935,
+            -0.002263315487653017,
+            -0.010933076962828636,
+            0.02135186456143856,
+            0.014444430358707905,
+            0.006450781598687172,
+            -0.0017390509601682425,
+            0.01692720502614975,
+            -0.024082917720079422,
+            0.0021569107193499804,
+            0.01452423445880413,
+            0.01718435063958168,
+            -0.0033051942009478807,
+            -0.008241926319897175,
+            0.002852974459528923,
+            -0.0056571802124381065,
+            -0.01510945986956358,
+            0.011802048422396183,
+            -0.0018022286240011454,
+            0.015419806353747845,
+            -0.030857346951961517,
+            0.013185309246182442,
+            0.0006018512649461627,
+            0.014240488409996033,
+            0.002691150875762105,
+            0.007661134470254183,
+            -0.01333604846149683,
+            0.012121262960135937,
+            -0.017955783754587173,
+            0.00302144861780107,
+            0.017166616395115852,
+            0.020961714908480644,
+            -0.0036177579313516617,
+            -0.0031212028115987778,
+            0.01165130827575922,
+            0.014160685241222382,
+            0.016067100688815117,
+            0.003670960199087858,
+            0.0036864776629954576,
+            -0.017689771950244904,
+            -0.0300238449126482,
+            -0.00582565413787961,
+            -0.0014142950531095266,
+            -0.012369539588689804,
+            -0.0009338115924037993,
+            0.019436581060290337,
+            -0.001129995216615498,
+            0.018762685358524323,
+            -0.03536181151866913,
+            0.008069018833339214,
+            -0.007545862812548876,
+            -0.012484811246395111,
+            -0.0032519919332116842,
+            -0.014914384111762047,
+            -0.015127194114029408,
+            0.0067256600596010685,
+            -0.011500569060444832,
+            0.009124197997152805,
+            -0.01666119322180748,
+            -0.025324305519461632,
+            0.008889221586287022,
+            -0.01125229150056839,
+            -0.013007967732846737,
+            0.007856209762394428,
+            -0.01772524043917656,
+            0.007355221081525087,
+            0.007971481420099735,
+            0.0016016116132959723,
+            -0.00631334213539958,
+            0.002755437046289444,
+            0.0076167993247509,
+            0.003537954529747367,
+            0.015809956938028336,
+            -0.0006428614142350852,
+            -0.00048824213445186615,
+            -0.022717392072081566,
+            -0.007803007494658232,
+            -0.018106523901224136,
+            -0.002391887828707695,
+            -0.007036007009446621,
+            -0.03055586665868759,
+            -0.0031877057626843452,
+            0.010746869258582592,
+            0.003923671320080757,
+            0.014063146896660328,
+            0.004553232342004776,
+            -0.009540949948132038,
+            0.011926187202334404,
+            0.005502006970345974,
+            -0.0005051449406892061,
+            -0.0033561799209564924,
+            -0.005927625577896833,
+            0.008264093659818172,
+            0.00026725855423137546,
+            0.008960157632827759,
+            0.0006661374354735017,
+            0.025874063372612,
+            -0.006894134450703859,
+            -0.00030729881837032735,
+            0.010489724576473236,
+            -0.009736024774610996,
+            -0.0011155862594023347,
+            0.0007852884591557086,
+            0.0057768854312598705,
+            0.012662152759730816,
+            -0.021937090903520584,
+            0.012555748224258423,
+            -0.023621831089258194,
+            -0.011216823011636734,
+            0.007705469615757465,
+            0.0020471811294555664,
+            -0.010995146818459034,
+            0.0026002635713666677,
+            -0.01342471968382597,
+            -0.017343956977128983,
+            -0.00894242338836193,
+            0.017148882150650024,
+            -0.018585344776511192,
+            -0.03048493154346943,
+            -0.004703972022980452,
+            -0.007621232885867357,
+            -0.02303660660982132,
+            0.008343897759914398,
+            -0.024136120453476906,
+            -0.0030635669827461243,
+            0.000919956830330193,
+            0.015561679378151894,
+            -0.0011737762251868844,
+            0.014604037627577782,
+            -0.03064453788101673,
+            -0.0005134578095749021,
+            0.022433646023273468,
+            -0.021334132179617882,
+            -0.015162661671638489,
+            0.0061537353321909904,
+            -0.011048348620533943,
+            -0.019117368385195732,
+            0.011997124180197716,
+            -0.0007004972430877388,
+            0.002314300974830985,
+            -0.019986338913440704,
+            0.034067220985889435,
+            -0.037880051881074905,
+            0.005253729410469532,
+            0.009904499165713787,
+            0.023143010213971138,
+            0.008796117268502712,
+            0.016430649906396866,
+            0.02995290793478489,
+            0.0036554429680109024,
+            0.014391228556632996,
+            0.007763105444610119,
+            -0.005138457752764225,
+            -0.004885746631771326,
+            -0.009221736341714859,
+            -6.1956807257956825e-06,
+            -0.002870708703994751,
+            0.0058877235278487206,
+            0.005395602434873581,
+            0.022575518116354942,
+            -0.022912466898560524,
+            0.027416929602622986,
+            0.02488095313310623,
+            0.006769995670765638,
+            -0.00984242931008339,
+            0.0009886764455586672,
+            -0.013451320119202137,
+            0.023710502311587334,
+            -0.00353352096863091,
+            -0.021334132179617882,
+            -0.0012868312187492847,
+            0.019791264086961746,
+            0.015091725625097752,
+            0.0028884427156299353,
+            -0.009869030676782131,
+            -0.009381342679262161,
+            -0.0002915044024121016,
+            -0.03057360090315342,
+            0.007674435153603554,
+            0.012112395837903023,
+            -0.012076927348971367,
+            -0.01699814200401306,
+            -0.011713378131389618,
+            -0.013460187241435051,
+            0.001321190968155861,
+            -0.007333053275942802,
+            0.015224731527268887,
+            -0.013584326021373272,
+            0.0038660354912281036,
+            0.018283864483237267,
+            -0.0007010514382272959,
+            -0.022380443289875984,
+            -0.021901622414588928,
+            -0.004355940036475658,
+            -0.00917740073055029,
+            0.011145886965095997,
+            -0.008649811148643494,
+            -0.017166616395115852,
+            -0.019365645945072174,
+            0.0011859684018418193,
+            -0.02259325236082077,
+            -0.0016160205705091357,
+            -0.015322268940508366,
+            0.0006932927644811571,
+            -0.013619794510304928,
+            0.0014054280472919345,
+            -0.00885818712413311,
+            0.014586303383111954,
+            -0.0033517463598400354,
+            -0.0021114670671522617,
+            0.022628720849752426,
+            -0.011979389935731888,
+            0.017432628199458122,
+            0.003635491942986846,
+            0.0010668174363672733,
+            -0.006903001572936773,
+            -0.017867112532258034,
+            0.009558684192597866,
+            0.0024761247914284468,
+            -0.007767539005726576,
+            0.018230661749839783,
+            -0.012254267930984497,
+            0.0041231801733374596,
+            -0.0023497692309319973,
+            -0.01103061530739069,
+            0.0055507756769657135,
+            0.007355221081525087,
+            0.0013821519678458571,
+            -0.0013655262300744653,
+            0.003917021211236715,
+            -0.010117308236658573,
+            0.012218800373375416,
+            0.01492325123399496,
+            0.00881828460842371,
+            0.004480078816413879,
+            -0.026991311460733414,
+            0.006393145769834518,
+            0.023001138120889664,
+            -0.004515547305345535,
+            -0.0018498890567570925,
+            -0.0002811133163049817,
+            0.014683840796351433,
+            0.022841529920697212,
+            -0.007723203860223293,
+            0.009939967654645443,
+            -0.031141093000769615,
+            0.03349972888827324,
+            0.01629764400422573,
+            -0.013965609483420849,
+            -0.023675033822655678,
+            0.02376370318233967,
+            -0.01125229150056839,
+            0.0029748964589089155,
+            -0.011606973595917225,
+            -0.015517343766987324,
+            -0.024614941328763962,
+            0.01737942546606064,
+            0.015162661671638489,
+            0.007426157593727112,
+            -0.014471031725406647,
+            0.03064453788101673,
+            0.0006894134567119181,
+            0.004650769755244255,
+            -0.007324186619371176,
+            0.005351266823709011,
+            0.021546941250562668,
+            -0.003529087407514453,
+            -0.0031810556538403034,
+            -0.005258162971585989,
+            -0.01541093923151493,
+            0.0070936428382992744,
+            -0.014586303383111954,
+            0.0031345034949481487,
+            -0.010738002136349678,
+            -0.00030480496934615076,
+            0.009062129072844982,
+            0.027434663847088814,
+            0.008951290510594845,
+            0.011926187202334404,
+            0.023710502311587334,
+            -0.014497633092105389,
+            -0.010268048383295536,
+            -0.009558684192597866,
+            0.014125216752290726,
+            0.016368580982089043,
+            0.01984446682035923,
+            -0.0009166316594928503,
+            0.01622670888900757,
+            -0.013442453928291798,
+            -0.009718290530145168,
+            -0.009931100532412529,
+            -0.01744149439036846,
+            0.0007132436148822308,
+            0.00978922750800848,
+            -0.0026445987168699503,
+            -0.005169492214918137,
+            -0.011722245253622532,
+            0.012830627150833607,
+            -0.0015118326991796494,
+            -0.007372955325990915,
+            -0.010897609405219555,
+            0.01510945986956358,
+            -0.005293630994856358,
+            -0.004686237778514624,
+            0.016075968742370605,
+            0.023533159866929054,
+            -0.01993313618004322,
+            0.0026645497418940067,
+            -0.013406985439360142,
+            0.004267269745469093,
+            -0.007971481420099735,
+            0.031212029978632927,
+            -0.017920315265655518,
+            0.0205538310110569,
+            0.0014575219247490168,
+            -0.001433137571439147,
+            -0.009762626141309738,
+            0.0561816543340683,
+            -0.0012702054809778929,
+            -0.028853392228484154,
+            0.024207057431340218,
+            0.007927145808935165,
+            -0.012724222615361214,
+            -0.002011712873354554,
+            0.01262668427079916,
+            0.015650350600481033,
+            0.004085495136678219,
+            -0.011935054324567318,
+            -0.007275417447090149,
+            -0.007004972547292709,
+            -0.017601102590560913,
+            0.030520398169755936,
+            -0.01895776018500328,
+            0.004868012387305498,
+            0.011358696036040783,
+            0.021777484565973282,
+            -0.004686237778514624,
+            -0.01510945986956358,
+            0.00225888192653656,
+            0.007319753058254719,
+            -0.024685878306627274,
+            -0.004198549780994654,
+            0.00016542596858926117,
+            -0.0025315438397228718,
+            -0.004218501038849354,
+            -0.020429691299796104,
+            0.02383464016020298,
+            0.00984242931008339,
+            -0.00750596122816205,
+            -0.017929183319211006,
+            -0.01700701005756855,
+            -0.010844406671822071,
+            -0.017831645905971527,
+            -0.002586962888017297,
+            0.017973517999053,
+            0.016803067177534103,
+            0.001548409229144454,
+            0.013930141925811768,
+            -0.0055463421158492565,
+            0.0049566831439733505,
+            -0.010427655652165413,
+            0.012307470664381981,
+            -0.01753903180360794,
+            0.0038704690523445606,
+            0.01975579559803009,
+            -0.007297585252672434,
+            0.01948978379368782,
+            0.020677968859672546,
+            0.0067433943040668964,
+            -0.008623209781944752,
+            0.00010411861148895696,
+            0.0024207057431340218,
+            0.024047449231147766,
+            -0.0016115870093926787,
+            -0.002083757659420371,
+            0.0024362229742109776,
+            0.007807441055774689,
+            0.028906594961881638,
+            0.009860163554549217,
+            0.008184290491044521,
+            0.025182433426380157,
+            -0.0031677549704909325,
+            -0.009159666486084461,
+            -0.0055419085547327995,
+            -0.014958719722926617,
+            0.0020216882694512606,
+            0.005958660040050745,
+            0.003815049771219492,
+            -0.0029881971422582865,
+            -0.0038616019301116467,
+            -0.001410969882272184,
+            -0.01886909082531929,
+            0.010693666525185108,
+            0.0016670060576871037,
+            0.0020649151410907507,
+            0.02835683710873127,
+            -0.012272002175450325,
+            0.008312863297760487,
+            0.01788484677672386,
+            -0.003032532287761569,
+            0.00930153951048851,
+            0.013043436221778393,
+            -0.008095620200037956,
+            0.008964591659605503,
+            -0.010640464723110199,
+            -0.01853214204311371,
+            -0.004074411001056433,
+            -0.0021790785249322653,
+            0.022185368463397026,
+            -0.010915343649685383,
+            -0.02356862835586071,
+            0.025271102786064148,
+            0.009860163554549217,
+            -0.022894732654094696,
+            -0.009603018872439861,
+            -0.00011146163888042793,
+            0.00801138300448656,
+            0.00962075311690569,
+            0.022274039685726166,
+            -0.01566808484494686,
+            -0.007670001592487097,
+            0.0003970777615904808,
+            0.008804984390735626,
+            0.019170571118593216,
+            0.0070892092771828175,
+            -0.005001018289476633,
+            -0.0011915103532373905,
+            0.01173997949808836,
+            -0.012662152759730816,
+            -0.023320350795984268,
+            0.007355221081525087,
+            0.00723994942381978,
+            -0.006787729449570179,
+            0.011881851591169834,
+            -0.013575458899140358,
+            -0.02399424836039543,
+            -0.012990233488380909,
+            0.004398058634251356,
+            -0.023621831089258194,
+            -0.01518926303833723,
+            -0.0027199687901884317,
+            -0.025874063372612,
+            -0.005151758436113596,
+            0.008335030637681484,
+            -0.007749804761260748,
+            0.009141932241618633,
+            0.004484512377530336,
+            0.006574920378625393,
+            -0.002575879218056798,
+            -0.0026290814857929945,
+            -9.774541103979573e-05,
+            0.0006384278531186283,
+            -0.002926127752289176,
+            0.0009011143120005727,
+            -0.021937090903520584,
+            0.009939967654645443,
+            -0.008707446977496147,
+            0.010480857454240322,
+            -0.009221736341714859,
+            -0.005377868190407753,
+            -0.015898628160357475,
+            0.016013899818062782,
+            -0.018762685358524323,
+            0.01995087042450905,
+            -1.6495836462127045e-05,
+            0.006406446453183889,
+            -0.0002840228262357414,
+            0.018408004194498062,
+            -0.014417828992009163,
+            -0.007607932202517986,
+            0.011562637984752655,
+            0.006428613793104887,
+            0.024473069235682487,
+            -0.01728188805282116,
+            0.0019540770445019007,
+            0.013655262999236584,
+            0.011837516911327839,
+            0.01218333188444376,
+            -0.00036299502244219184,
+            -0.019330177456140518,
+            0.011456233449280262,
+            0.0007354112458415329,
+            0.005240428727120161,
+            -0.004205200355499983,
+            0.014515367336571217,
+            0.020181413739919662,
+            -0.021387333050370216,
+            0.015357737429440022,
+            0.0009304864797741175,
+            0.00236972002312541,
+            0.012121262960135937,
+            0.005612845066934824,
+            -0.01824839599430561,
+            -0.014151818118989468,
+            0.012928164564073086,
+            0.0066901920363307,
+            -0.028197230771183968,
+            0.024331195279955864,
+            -0.014426696114242077,
+            -0.0004627493617590517,
+            0.010613863356411457,
+            -0.001321190968155861,
+            -0.024685878306627274,
+            -0.00837493222206831,
+            0.000817431544419378,
+            -0.004912347998470068,
+            -0.008286261931061745,
+            0.0030524833127856255,
+            0.018301598727703094,
+            0.009647354483604431,
+            0.0017789526609703898,
+            0.020571565255522728,
+            0.005732550285756588,
+            0.012582349590957165,
+            0.03873129189014435,
+            -0.001021927921101451,
+            0.004471211694180965,
+            -0.02019914798438549,
+            0.011580372229218483,
+            0.011385297402739525,
+            -0.006056197918951511,
+            -0.01528680045157671,
+            -0.025377508252859116,
+            -0.002099274890497327,
+            -0.009780360385775566,
+            0.010862140916287899,
+            0.0018709483556449413,
+            -0.011527169495821,
+            0.004300521221011877,
+            0.00701827323064208,
+            -0.010046372190117836,
+            -0.012759690172970295,
+            0.01904643140733242,
+            0.009266071021556854,
+            0.008778383024036884,
+            0.0008296237210743129,
+            -0.001856539398431778,
+            -0.01528680045157671,
+            0.014302557334303856,
+            -0.020979449152946472,
+            -0.002135851653292775,
+            0.00805571861565113,
+            0.004114313051104546,
+            -0.010959678329527378,
+            0.0057724518701434135,
+            -0.009629620239138603,
+            0.0011582588776946068,
+            -0.00793157983571291,
+            0.035397280007600784,
+            -0.004114313051104546,
+            0.0050896890461444855,
+            0.010401054285466671,
+            -0.00304583297111094,
+            0.006583787500858307,
+            -0.012467077933251858,
+            -0.008836018852889538,
+            0.013247378170490265,
+            0.02612234093248844,
+            -0.010392187163233757,
+            -0.019738061353564262,
+            0.0009947726503014565,
+            0.003639925504103303,
+            -0.00837049912661314,
+            -0.0006694625481031835,
+            0.010995146818459034,
+            0.014417828992009163,
+            -0.008587741293013096,
+            -0.006459648720920086,
+            0.024739079177379608,
+            0.013477921485900879,
+            -0.004351506475359201,
+            -0.016891736537218094,
+            0.009022227488458157,
+            0.0002779267088044435,
+            0.006942903157323599,
+            0.0063798450864851475,
+            -0.007151278667151928,
+            -0.004522197414189577,
+            0.023231681436300278,
+            -0.011004013940691948,
+            -0.002135851653292775,
+            -0.01094194408506155,
+            -0.016492720693349838,
+            -0.022717392072081566,
+            0.002875142265111208,
+            -0.024136120453476906,
+            -4.776432615472004e-05,
+            0.025501646101474762,
+            -0.0016969323623925447,
+            -0.009833562187850475,
+            0.0018576477887108922,
+            -0.016891736537218094,
+            0.012857227586209774,
+            6.400904385372996e-05,
+            -0.001757893362082541,
+            -0.0013788269134238362,
+            0.027133185416460037,
+            0.03816379979252815,
+            0.02481001615524292,
+            -0.002146935323253274,
+            0.01658139005303383,
+            -0.007953747175633907,
+            0.012068060226738453,
+            -0.0054488047026097775,
+            0.01391240768134594,
+            0.015428673475980759,
+            0.01116362027823925,
+            0.004335989244282246,
+            -0.004992151167243719,
+            -0.006433047354221344,
+            0.006526151672005653,
+            -0.008663111366331577,
+            0.007758671883493662,
+            0.0223272405564785,
+            0.0027776046190410852,
+            0.01537547167390585,
+            0.012059193104505539,
+            -0.011332094669342041,
+            0.016457252204418182,
+            -0.01728188805282116,
+            0.0027111016679555178,
+            -0.016058234497904778,
+            -0.0028640583623200655,
+            0.035822898149490356,
+            0.015703551471233368,
+            -0.0027088848873972893,
+            -0.0050852554850280285,
+            -0.007137978449463844,
+            0.008184290491044521,
+            -0.019525252282619476,
+            -0.0007836258737370372,
+            0.00467737065628171,
+            0.0001997858053073287,
+            -0.019330177456140518,
+            0.025022825226187706,
+            0.0056660473346710205,
+            -0.02092624641954899,
+            0.009328140877187252,
+            -0.015029655769467354,
+            0.012564615346491337,
+            0.020270084962248802,
+            0.0011560420971363783,
+            0.01045425608754158,
+            0.00598082784563303,
+            0.009425678290426731,
+            -0.0026445987168699503,
+            -0.00044169012107886374,
+            -0.01711341366171837,
+            -0.00751039432361722,
+            -0.0014708224916830659,
+            0.012812892906367779,
+            -0.003662093309685588,
+            0.00984242931008339,
+            -0.010064106434583664,
+            0.009266071021556854,
+            -0.0030125814955681562,
+            -0.005107422824949026,
+            -0.013433586806058884,
+            0.003036965848878026,
+            0.001497423741966486,
+            0.012919297441840172,
+            -0.007173446472734213,
+            -0.0029638127889484167,
+            -0.022256305441260338,
+            0.007120244204998016,
+            0.011677909642457962,
+            -0.01788484677672386,
+            0.017787309363484383,
+            0.004797075875103474,
+            -0.014107482507824898,
+            -0.01023257989436388,
+            0.011562637984752655,
+            -0.0019540770445019007,
+            -0.009381342679262161,
+            0.012546881102025509,
+            -0.018177460879087448,
+            -0.009736024774610996,
+            7.412302693410311e-06,
+            0.0205538310110569,
+            0.0022012460976839066,
+            0.010560660623013973,
+            -0.007696602493524551,
+            0.00886262021958828,
+            0.002527110278606415,
+            0.0010834431741386652,
+            0.010773470625281334,
+            -4.464700032258406e-05,
+            0.010365585796535015,
+            0.007625666446983814,
+            -0.014009945094585419,
+            -0.006707926280796528,
+            -0.021103588864207268,
+            -0.001193727133795619,
+            -0.001488556619733572,
+            -0.014027679339051247,
+            0.0068497988395392895,
+            -0.0012070277007296681,
+            0.006938469596207142,
+            0.016882870346307755,
+            -0.004743873607367277,
+            -0.011500569060444832,
+            0.0004724477184936404,
+            0.011935054324567318,
+            0.006836498621851206,
+            -0.004664070438593626,
+            -0.014497633092105389,
+            0.015747888013720512,
+            0.010613863356411457,
+            -0.0005594556569121778,
+            0.0050852554850280285,
+            -0.01879815384745598,
+            -0.01904643140733242,
+            -0.014568569138646126,
+            -0.0050675212405622005,
+            0.010046372190117836,
+            0.004205200355499983,
+            0.0033273620065301657,
+            -0.0014386795228347182,
+            0.0023076506331562996,
+            -0.00571481604129076,
+            -0.00014935444050934166,
+            0.03057360090315342,
+            -0.0015805523144081235,
+            -0.0003696453059092164,
+            0.012103528715670109,
+            0.014905516989529133,
+            0.012812892906367779,
+            -0.005151758436113596,
+            0.012653285637497902,
+            0.00907986331731081,
+            0.010480857454240322,
+            0.01709567941725254,
+            -0.02693810872733593,
+            0.02135186456143856,
+            -0.011261158622801304,
+            -0.010631597600877285,
+            0.018106523901224136,
+            0.003500269493088126,
+            -0.0022400394082069397,
+            -0.002502725925296545,
+            0.0059098913334310055,
+            0.01674986444413662,
+            -0.003901503747329116,
+            -0.017849380150437355,
+            0.0004815918509848416,
+            -0.013123239390552044,
+            -0.00751039432361722,
+            -0.017503563314676285,
+            -0.017476962879300117,
+            0.0054266368970274925,
+            -0.0028396740090101957,
+            0.008844885975122452,
+            0.003846084466204047,
+            -0.021050386130809784,
+            -0.010463123209774494,
+            -0.005741417407989502,
+            -0.001937451190315187,
+            0.06781522929668427,
+            -0.005169492214918137,
+            0.018008986487984657,
+            -0.0013566592242568731,
+            0.006220238283276558,
+            0.0070936428382992744,
+            -0.01851440779864788,
+            -0.006681324914097786,
+            -0.005626145750284195,
+            -0.03383667767047882,
+            0.006162602454423904,
+            0.013735066168010235,
+            -0.007749804761260748,
+            0.005267030093818903,
+            0.0003879335999954492,
+            -0.020890777930617332,
+            -0.020766640082001686,
+            -0.006996105425059795,
+            0.001219219877384603,
+            -0.008663111366331577,
+            0.021546941250562668,
+            -0.007944880053400993,
+            -0.0011349829146638513,
+            0.006167036015540361,
+            0.004531064536422491,
+            -0.0015739020891487598,
+            -0.017202084884047508,
+            0.005630579311400652,
+            -2.2150315999169834e-05,
+            -0.03630171716213226,
+            0.0036975613329559565,
+            -0.0022810494992882013,
+            -0.012334072031080723,
+            -0.006632556207478046,
+            0.011855251155793667,
+            0.016457252204418182,
+            -0.009611885994672775,
+            0.01975579559803009,
+            -0.020075010135769844,
+            0.0014741476625204086,
+            0.007594631519168615,
+            0.011083817109465599,
+            -0.004322688560932875,
+            -0.002511593047529459,
+            -0.020075010135769844,
+            -0.0007210022886283696,
+            0.029988376423716545,
+            -0.03591156750917435,
+            0.0038106164429336786,
+            -0.010445388965308666,
+            -0.004178598988801241,
+            -0.007816308178007603,
+            0.01842573843896389,
+            -0.002203462878242135,
+            -0.025625785812735558,
+            0.017521297559142113,
+            0.005484272725880146,
+            0.002493858803063631,
+            0.0016492720460519195,
+            -0.022486848756670952,
+            0.004699538461863995,
+            -0.022983403876423836,
+            -0.0008013600017875433,
+            -0.008237493224442005,
+            -0.018674015998840332,
+            0.011509436182677746,
+            -0.00984242931008339,
+            0.010525193065404892,
+            -0.024739079177379608,
+            0.006189203355461359,
+            0.0028241565451025963,
+            -0.012786291539669037,
+            -0.011713378131389618,
+            -0.006446348037570715,
+            0.0054399375803768635,
+            -0.024685878306627274,
+            0.0008146605687215924,
+            0.01657252386212349,
+            0.015570546500384808,
+            0.005958660040050745,
+            0.0057901861146092415,
+            -0.006916301790624857,
+            -0.01710454747080803,
+            0.003584506455808878,
+            -0.0038992869667708874,
+            -0.022894732654094696,
+            0.0013622011756524444,
+            -0.0022267389576882124,
+            0.006521718110889196,
+            0.023533159866929054,
+            0.012812892906367779,
+            0.021919356659054756,
+            0.008481336757540703,
+            0.014763644896447659,
+            0.005847821943461895,
+            0.024934155866503716,
+            0.0042428853921592236,
+            -0.023338085040450096,
+            -0.004136480391025543,
+            -0.000975375936832279,
+            0.0014176202239468694,
+            0.014072014018893242,
+            -0.0063887122087180614,
+            -0.024065183475613594,
+            -0.03890863060951233,
+            -0.0058832899667322636,
+            0.013664130121469498,
+            0.00757246371358633,
+            -0.008303996175527573,
+            0.006809897255152464,
+            -0.0019197171786800027,
+            -0.0035468214191496372,
+            0.0027842549607157707,
+            0.0026845005340874195,
+            -0.020979449152946472,
+            0.028924329206347466,
+            0.014728176407516003,
+            0.046818044036626816,
+            0.03204553201794624,
+            0.009904499165713787,
+            -0.003105685580521822,
+            0.01950751803815365,
+            0.001809987355954945,
+            -0.01186411827802658,
+            0.016528187319636345,
+            0.022965669631958008,
+            -0.02498735673725605,
+            0.02126319520175457,
+            -0.008703012950718403,
+            -0.00784290861338377,
+            -0.00407884456217289,
+            -0.011935054324567318,
+            0.0170868132263422,
+            0.031673114746809006,
+            -0.006583787500858307,
+            0.000389319087844342,
+            -0.02417158894240856,
+            0.01710454747080803,
+            -0.010604996234178543,
+            0.016856269910931587,
+            -0.007244382984936237,
+            0.045044634491205215,
+            0.009665088728070259,
+            0.006557186134159565,
+            0.0035623388830572367,
+            -0.011066082864999771,
+            0.0006689083529636264,
+            0.0055507756769657135,
+            -0.002966029569506645,
+            0.004420226439833641,
+            0.019826732575893402,
+            -0.0053246659226715565,
+            0.023001138120889664,
+            0.005107422824949026,
+            -0.009549817070364952,
+            0.004167515318840742,
+            0.014861182309687138,
+            0.002815289655700326,
+            0.020128212869167328,
+            -0.006220238283276558,
+            -0.013353782705962658,
+            -0.0026290814857929945,
+            0.002706668106839061,
+            0.003090168349444866,
+            -0.0019518602639436722,
+            -0.006685758475214243,
+            -0.001582769094966352,
+            -0.01647498644888401,
+            -0.005608411505818367,
+            -0.018017852678894997,
+            -0.01311437226831913,
+            0.002930561313405633,
+            -0.013814869336783886,
+            0.01824839599430561,
+            0.029349947348237038,
+            -0.004087711684405804,
+            -0.0026645497418940067,
+            0.018283864483237267,
+            -0.014825713820755482,
+            -0.013442453928291798,
+            0.004236234817653894,
+            0.010551794432103634,
+            0.005630579311400652,
+            0.003850518027320504,
+            0.03587609902024269,
+            0.003948055673390627,
+            0.010374452918767929,
+            -0.013273979537189007,
+            -0.02372823655605316,
+            -0.000808564480394125,
+            -0.0043670241720974445,
+            -0.008428134955465794,
+            0.00013272871728986502,
+            -0.009470012970268726,
+            -0.009363608434796333,
+            0.01646611839532852,
+            0.01568581722676754,
+            -0.016758732497692108,
+            -0.021582409739494324,
+            0.0001651488710194826,
+            0.0019540770445019007,
+            -0.0037795817479491234,
+            -0.0311942957341671,
+            -0.011722245253622532
+          ],
+          "title": "Preventing Token Dropping with No-Token-Left-Behind"
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "460c555c-6aea-4625-af79-5ad4ea6dd448",
+        "properties": {
+          "page_content": "C Encouraging Exploration Across Experts 29 ",
+          "keyphrases": [
+            "Encouraging Exploration",
+            "Experts"
+          ],
+          "embedding": [
+            0.029896028339862823,
+            0.007837229408323765,
+            -0.022426677867770195,
+            -0.02060125023126602,
+            -0.0056159826926887035,
+            0.03129303827881813,
+            -0.005033894907683134,
+            0.0729425847530365,
+            -0.001533219125121832,
+            -0.0002334171877009794,
+            -0.04105348512530327,
+            -0.01453822385519743,
+            0.016559232026338577,
+            -0.02227766253054142,
+            -0.02035910077393055,
+            0.05126097798347473,
+            0.010049163363873959,
+            0.008787197060883045,
+            -0.022743333131074905,
+            -0.01933462731540203,
+            0.0470513179898262,
+            -0.040829963982105255,
+            -0.06571538001298904,
+            0.061505720019340515,
+            0.04552391916513443,
+            0.010365818627178669,
+            0.024457000195980072,
+            0.007217888254672289,
+            -0.06351741403341293,
+            0.04153778403997421,
+            0.019949311390519142,
+            0.008465884253382683,
+            -0.008838420733809471,
+            -0.03708597645163536,
+            -0.02613341249525547,
+            -0.024401118978857994,
+            0.018803762272000313,
+            -0.012619663029909134,
+            -0.013113273307681084,
+            0.015842100605368614,
+            -0.007562484126538038,
+            -0.030492085963487625,
+            -0.016037682071328163,
+            0.019148359075188637,
+            0.00031636469066143036,
+            -0.002261993009597063,
+            -0.041463274508714676,
+            -0.036601677536964417,
+            -0.014938700012862682,
+            -0.023302137851715088,
+            -0.02192375436425209,
+            -0.0339939258992672,
+            0.00943913497030735,
+            -0.010915310122072697,
+            0.007962960749864578,
+            0.03365864232182503,
+            0.030436204746365547,
+            -0.008545048534870148,
+            0.03002641536295414,
+            0.04511412978172302,
+            0.022780586034059525,
+            -0.0023202018346637487,
+            -0.02482953481376171,
+            -0.000566662463825196,
+            0.009844268672168255,
+            -0.041463274508714676,
+            -0.01307601947337389,
+            0.011865276843309402,
+            -0.006072339601814747,
+            -0.04101623222231865,
+            -0.014473030343651772,
+            0.010552086867392063,
+            -0.02872253768146038,
+            0.029784265905618668,
+            0.021495336666703224,
+            0.015385744161903858,
+            0.06273508816957474,
+            0.03393804654479027,
+            -0.030249936506152153,
+            -0.0006822068826295435,
+            0.0351860411465168,
+            -0.03609875589609146,
+            0.03326747938990593,
+            0.001204921631142497,
+            0.03285769000649452,
+            -0.01790967583656311,
+            -0.03591248765587807,
+            -0.08165992796421051,
+            0.052415840327739716,
+            0.0031362888403236866,
+            -0.02225903607904911,
+            -0.0011286681983619928,
+            0.003944226540625095,
+            -0.010617280378937721,
+            -0.05465105548501015,
+            0.026040278375148773,
+            0.008135258220136166,
+            0.0008545048185624182,
+            0.0029989161994308233,
+            -0.013578942976891994,
+            0.02455013431608677,
+            0.009993283078074455,
+            -0.015963174402713776,
+            -0.022668825462460518,
+            0.030845994129776955,
+            0.0056159826926887035,
+            0.01548819150775671,
+            0.020508116111159325,
+            -0.011343725956976414,
+            0.08039330691099167,
+            -0.027902958914637566,
+            0.005285356659442186,
+            -0.017024902626872063,
+            0.021420828998088837,
+            -0.013923538848757744,
+            -0.02676672302186489,
+            0.000443550874479115,
+            -0.04485335573554039,
+            0.021718857809901237,
+            0.00806540809571743,
+            -0.01486419327557087,
+            0.01780722849071026,
+            0.01503183413296938,
+            -0.029467610642313957,
+            -0.031423427164554596,
+            0.023767806589603424,
+            -0.010598653927445412,
+            0.014119120314717293,
+            0.020526742562651634,
+            0.04522589221596718,
+            0.030752860009670258,
+            0.0222404096275568,
+            -0.004028047434985638,
+            0.024419745430350304,
+            0.041798558086156845,
+            0.027213767170906067,
+            0.046678781509399414,
+            0.02423347719013691,
+            -0.02546284720301628,
+            -0.002768409438431263,
+            -0.03900453820824623,
+            0.018803762272000313,
+            -0.0029407073743641376,
+            0.01745332032442093,
+            -0.010216804221272469,
+            -0.00982564128935337,
+            0.024997176602482796,
+            0.04455532506108284,
+            -0.04906301200389862,
+            0.01417500153183937,
+            0.008279616013169289,
+            0.04973357915878296,
+            0.014910760335624218,
+            -0.009583492763340473,
+            0.03220575302839279,
+            -0.001704353024251759,
+            -0.014454402960836887,
+            0.0264873206615448,
+            -0.020545369014143944,
+            -0.021886499598622322,
+            0.013467182405292988,
+            -0.008177168667316437,
+            0.005229476373642683,
+            0.012498588301241398,
+            0.011893217451870441,
+            0.02715788595378399,
+            0.0011496233055368066,
+            -0.016913142055273056,
+            -0.07972273975610733,
+            0.04734934866428375,
+            0.0036461977288126945,
+            -0.018338093534111977,
+            0.04943554848432541,
+            0.002908110385760665,
+            0.0007311022491194308,
+            -0.020210087299346924,
+            -0.0017765318043529987,
+            -0.0043516880832612514,
+            -0.00693382928147912,
+            0.015860727056860924,
+            -0.04433180391788483,
+            -0.0029942593537271023,
+            0.022426677867770195,
+            -0.013141212984919548,
+            0.010701101273298264,
+            0.005406430922448635,
+            0.08411866426467896,
+            -0.016335710883140564,
+            0.028163734823465347,
+            0.017742035910487175,
+            0.015246042981743813,
+            -0.031516559422016144,
+            0.003408706048503518,
+            0.018328780308365822,
+            0.013755897991359234,
+            -0.011129518039524555,
+            0.009816328063607216,
+            0.03481350466609001,
+            0.0048429700545966625,
+            -0.025369713082909584,
+            -0.01563720591366291,
+            -0.03447822108864784,
+            0.01683863438665867,
+            0.000613520503975451,
+            0.01678275503218174,
+            0.024307984858751297,
+            -0.009956029243767262,
+            0.02646869421005249,
+            -0.0024750372394919395,
+            -0.01634502410888672,
+            0.016717560589313507,
+            -0.01307601947337389,
+            0.009047972038388252,
+            -0.013914225623011589,
+            -0.005001298151910305,
+            -0.003718376625329256,
+            -0.018487107008695602,
+            0.02516481839120388,
+            -0.0036718095652759075,
+            -0.012265753000974655,
+            0.015702400356531143,
+            0.023972703143954277,
+            0.026226546615362167,
+            -0.014612731523811817,
+            -0.01563720591366291,
+            0.006700994446873665,
+            0.04414553567767143,
+            0.009392567910254002,
+            0.021178681403398514,
+            -0.01552544441074133,
+            -0.005643922835588455,
+            0.03689970821142197,
+            -0.017667528241872787,
+            0.016400905326008797,
+            -0.0005166028859093785,
+            -0.016242576763033867,
+            0.018291525542736053,
+            -0.003939569927752018,
+            0.006426248699426651,
+            0.03639678284525871,
+            0.0035018399357795715,
+            -0.024792281910777092,
+            -0.030417578294873238,
+            0.028312748298048973,
+            0.026617709547281265,
+            0.004137479700148106,
+            -0.027660811319947243,
+            0.04354947805404663,
+            0.026375560089945793,
+            0.014333329163491726,
+            -0.02065712958574295,
+            -0.02587263658642769,
+            -0.010803548619151115,
+            0.011874590069055557,
+            -0.010654534213244915,
+            0.040867216885089874,
+            0.03283906355500221,
+            -0.00010513960296520963,
+            -0.01047758013010025,
+            0.004083927720785141,
+            0.021551217883825302,
+            0.01868268847465515,
+            0.010281998664140701,
+            0.0032131243497133255,
+            -0.01935325376689434,
+            -0.04202207922935486,
+            0.024922668933868408,
+            -0.023767806589603424,
+            0.00702230678871274,
+            -0.03948883339762688,
+            0.006477472838014364,
+            -0.010216804221272469,
+            0.009434478357434273,
+            -0.034385088831186295,
+            0.045337650924921036,
+            0.016754813492298126,
+            -0.006887262221425772,
+            -0.0130946459248662,
+            -0.011800083331763744,
+            0.030175428837537766,
+            -0.002717185765504837,
+            0.02034047432243824,
+            0.027847077697515488,
+            -0.013411302119493484,
+            -0.01632639765739441,
+            0.0054855952039361,
+            -0.052490346133708954,
+            0.02069438435137272,
+            -0.033211600035429,
+            0.009224927052855492,
+            -0.00036293172161094844,
+            -0.04641800746321678,
+            -0.022147275507450104,
+            0.028461763635277748,
+            0.037272244691848755,
+            -0.012424081563949585,
+            -0.016894515603780746,
+            0.004402911756187677,
+            -0.008880331180989742,
+            0.003743988461792469,
+            -0.005853474605828524,
+            -0.02447562664747238,
+            -0.03188909590244293,
+            -0.0067801582626998425,
+            -0.026207920163869858,
+            0.02844313718378544,
+            -0.011902530677616596,
+            0.0119584109634161,
+            -0.03704872354865074,
+            -0.02712063305079937,
+            0.022445304319262505,
+            0.005662549752742052,
+            -0.020470863208174706,
+            -0.04869047552347183,
+            -0.057966627180576324,
+            0.04004763811826706,
+            0.06381544470787048,
+            0.025630488991737366,
+            -0.00016007413796614856,
+            -0.027660811319947243,
+            0.02641281485557556,
+            -0.014938700012862682,
+            -0.037980061024427414,
+            0.01676412671804428,
+            0.011828023009002209,
+            0.007939676754176617,
+            0.01905522495508194,
+            0.028536271303892136,
+            -0.001995396800339222,
+            -0.004321419633924961,
+            -0.031237157061696053,
+            -0.04362398386001587,
+            -0.029467610642313957,
+            -0.02093653194606304,
+            -0.01437058299779892,
+            -0.03574484586715698,
+            -0.01940913498401642,
+            0.024047208949923515,
+            0.005806907545775175,
+            0.005862787831574678,
+            -0.02546284720301628,
+            0.012032918632030487,
+            -0.010980503633618355,
+            0.0053784907795488834,
+            0.010896682739257812,
+            0.0053272671066224575,
+            0.017797915264964104,
+            -0.0036834513302892447,
+            -0.00501061137765646,
+            -0.023786434903740883,
+            0.0012631304562091827,
+            0.011082950979471207,
+            0.0033947357442229986,
+            -0.0089455246925354,
+            0.024270731955766678,
+            0.01614944264292717,
+            -0.03513016179203987,
+            -0.013644137419760227,
+            0.03809182345867157,
+            0.0029290656093508005,
+            0.030827367678284645,
+            -0.07957372814416885,
+            -0.016969023272395134,
+            0.004898850806057453,
+            -0.03652717173099518,
+            -0.0075019472278654575,
+            0.005606669466942549,
+            0.019800297915935516,
+            -0.010570713318884373,
+            -0.0037649436853826046,
+            -0.027902958914637566,
+            -0.009215613827109337,
+            0.02356291189789772,
+            0.011064324527978897,
+            -0.02166297845542431,
+            0.013811778277158737,
+            0.00014312082203105092,
+            -0.01903659850358963,
+            -0.02933722361922264,
+            0.014584790915250778,
+            -0.03540956228971481,
+            -0.002244530478492379,
+            0.009881521575152874,
+            0.000486043281853199,
+            -0.016205323860049248,
+            0.05748233199119568,
+            0.023767806589603424,
+            0.018496420234441757,
+            -2.3683694962528534e-05,
+            0.0024890073109418154,
+            -0.02097378671169281,
+            -0.011734888888895512,
+            -0.05547063425183296,
+            0.002099008532240987,
+            0.02326488308608532,
+            -0.010328564792871475,
+            -0.023395271971821785,
+            -0.018971404060721397,
+            0.01658717356622219,
+            -0.015124968253076077,
+            -0.024699147790670395,
+            0.03514878824353218,
+            -0.010123670101165771,
+            -0.01340198889374733,
+            0.05174527317285538,
+            0.01275005005300045,
+            -0.04347497224807739,
+            -0.00793502014130354,
+            -0.0371604822576046,
+            0.04068094864487648,
+            -0.024028582498431206,
+            -0.03680657222867012,
+            0.0004083345702383667,
+            0.0027055440004915,
+            0.054017744958400726,
+            -0.005108402110636234,
+            0.04623173922300339,
+            0.023991329595446587,
+            0.0029919310472905636,
+            0.004945417400449514,
+            0.05032963678240776,
+            0.011092264205217361,
+            0.051149215549230576,
+            0.01966990903019905,
+            -0.025630488991737366,
+            0.002516947453841567,
+            -0.03645266219973564,
+            0.0035693622194230556,
+            -0.021551217883825302,
+            -0.043288704007864,
+            -0.004991984460502863,
+            0.019204240292310715,
+            -0.01682000793516636,
+            -0.005117715802043676,
+            0.0472375862300396,
+            -0.046678781509399414,
+            -0.03624776750802994,
+            0.01909247785806656,
+            -0.013001511804759502,
+            -0.03159106522798538,
+            0.0020047102589160204,
+            -0.0069245160557329655,
+            -0.031795963644981384,
+            -0.032168496400117874,
+            -5.2824463637080044e-05,
+            0.014230881817638874,
+            -0.029858773574233055,
+            -0.0166058000177145,
+            -0.04515138268470764,
+            0.01451959740370512,
+            -0.036992840468883514,
+            0.03580072522163391,
+            -0.03526054695248604,
+            0.0003108348755631596,
+            0.027847077697515488,
+            -0.009145762771368027,
+            -0.03125578537583351,
+            0.027958840131759644,
+            0.0016065621748566628,
+            -0.01836603321135044,
+            0.010095730423927307,
+            -0.0018673375016078353,
+            0.03453410416841507,
+            -0.030138175934553146,
+            0.0003000662545673549,
+            0.034422341734170914,
+            -0.03295082226395607,
+            -0.02393544837832451,
+            0.01079423539340496,
+            0.05971754714846611,
+            0.008088691160082817,
+            -0.027008872479200363,
+            -0.012358887121081352,
+            0.014808312989771366,
+            -0.009499671868979931,
+            0.009304090403020382,
+            0.020843397825956345,
+            -0.03749576583504677,
+            -0.022426677867770195,
+            -0.007134067825973034,
+            0.00888498779386282,
+            -0.007981587201356888,
+            -0.0022526795510202646,
+            0.036266397684812546,
+            0.017173917964100838,
+            6.915930134709924e-05,
+            0.03369589522480965,
+            -0.020899279043078423,
+            -0.019297374412417412,
+            -0.004482075572013855,
+            0.00969525333493948,
+            -0.017304304987192154,
+            0.00484762666746974,
+            -0.006649770773947239,
+            -0.003913958091288805,
+            -0.04030841216444969,
+            -0.03494389355182648,
+            -0.03477625176310539,
+            0.007203917950391769,
+            0.07160145044326782,
+            -0.009779074229300022,
+            -0.0028918120078742504,
+            0.018058691173791885,
+            0.01743469201028347,
+            -0.001707845483906567,
+            -0.021532591432332993,
+            -0.03198223188519478,
+            -0.028331374749541283,
+            0.008670778945088387,
+            0.00515962578356266,
+            -0.007688215002417564,
+            0.04485335573554039,
+            0.021867873147130013,
+            0.004153778310865164,
+            -0.015395057387650013,
+            0.024736400693655014,
+            -0.024959923699498177,
+            0.03520466759800911,
+            -0.00905262865126133,
+            0.0023958731908351183,
+            -0.0019977251067757607,
+            0.000827728770673275,
+            -0.0044378372840583324,
+            0.03593111410737038,
+            -0.00347389979287982,
+            0.0261892918497324,
+            0.014603417366743088,
+            -0.005308640189468861,
+            0.03168420121073723,
+            -0.0626605823636055,
+            -0.0012316977372393012,
+            0.02004244551062584,
+            0.0047498359344899654,
+            0.001548353466205299,
+            -0.004866253584623337,
+            -0.004782433155924082,
+            0.01258240919560194,
+            0.02354428544640541,
+            -0.003101363545283675,
+            -0.034999772906303406,
+            -0.02067575789988041,
+            0.048876743763685226,
+            0.003289960091933608,
+            0.0009470568038523197,
+            0.02069438435137272,
+            0.009229583665728569,
+            0.03464586287736893,
+            -0.01519016269594431,
+            0.026040278375148773,
+            0.01708078384399414,
+            0.009383254684507847,
+            0.008572988212108612,
+            0.015721026808023453,
+            0.011399607174098492,
+            -0.000368752604117617,
+            -0.026096157729625702,
+            0.018990030512213707,
+            0.01870131492614746,
+            0.0015646519605070353,
+            -0.04675328731536865,
+            0.014975953847169876,
+            0.014510284177958965,
+            -0.0008236541762016714,
+            0.012386827729642391,
+            -0.01875719614326954,
+            0.003408706048503518,
+            -0.011725575663149357,
+            0.016121502965688705,
+            -0.00448673265054822,
+            -0.015003894455730915,
+            -0.00463574705645442,
+            -0.0009453105158172548,
+            -0.017034215852618217,
+            -0.010421698912978172,
+            -0.003874376183375716,
+            0.01773272082209587,
+            0.039190806448459625,
+            -0.045039623975753784,
+            -0.010291311889886856,
+            -0.015553385019302368,
+            -0.011986351571977139,
+            -0.0326155424118042,
+            -0.001544860890135169,
+            -0.00967662688344717,
+            0.006808098405599594,
+            -0.005546132102608681,
+            -0.013523062691092491,
+            -0.04172405228018761,
+            -0.011408920399844646,
+            -0.027344154193997383,
+            -0.013737271539866924,
+            0.02941173128783703,
+            0.031181277707219124,
+            0.0015390400076285005,
+            0.009373941458761692,
+            0.010309938341379166,
+            0.009639373049139977,
+            -0.004787089768797159,
+            -0.015711713582277298,
+            -0.028461763635277748,
+            0.019446387887001038,
+            0.015097028575837612,
+            0.032168496400117874,
+            -0.01337404828518629,
+            0.0002654320269357413,
+            0.019949311390519142,
+            -0.002558857901021838,
+            -0.010729041881859303,
+            -0.0014517268864437938,
+            0.01098981685936451,
+            -0.02028459496796131,
+            -0.004447150509804487,
+            0.03229888528585434,
+            0.0009418179979547858,
+            0.01273142360150814,
+            0.03349100053310394,
+            0.004018733743578196,
+            0.02415897138416767,
+            0.005392461083829403,
+            0.018533675000071526,
+            0.029635252431035042,
+            -0.017816541716456413,
+            0.009220270439982414,
+            0.007990900427103043,
+            0.030175428837537766,
+            0.009169046767055988,
+            -0.014575477689504623,
+            -0.0339939258992672,
+            -0.02004244551062584,
+            0.0001397010637447238,
+            0.03136754408478737,
+            0.0005989682977087796,
+            -0.0033248853869736195,
+            -0.011297158896923065,
+            -0.01771409437060356,
+            0.014612731523811817,
+            0.001167668029665947,
+            0.033360615372657776,
+            0.031162651255726814,
+            -0.01486419327557087,
+            -0.02060125023126602,
+            -0.028275495395064354,
+            0.008647495880723,
+            0.0612821988761425,
+            -0.014240195043385029,
+            -0.011949097737669945,
+            0.029430357739329338,
+            0.015069087967276573,
+            0.020806144922971725,
+            0.003799868980422616,
+            -0.0026450068689882755,
+            0.009909462183713913,
+            0.01965128257870674,
+            0.026617709547281265,
+            -0.0229296013712883,
+            -0.029150955379009247,
+            -0.013653450645506382,
+            -0.012619663029909134,
+            -0.006966426502913237,
+            0.009099195711314678,
+            0.004433180205523968,
+            -0.03811044991016388,
+            0.009117823094129562,
+            0.00342966103926301,
+            -0.012098112143576145,
+            -0.014761745929718018,
+            -0.013588257133960724,
+            0.015804847702383995,
+            -0.01342061534523964,
+            0.02937447652220726,
+            -0.013904912397265434,
+            -0.03028719127178192,
+            -0.013634824194014072,
+            -0.01907385140657425,
+            -0.019763043150305748,
+            -0.034105684608221054,
+            -0.019763043150305748,
+            0.015450937673449516,
+            0.020806144922971725,
+            -0.006165473721921444,
+            -0.017425378784537315,
+            0.0009272657916881144,
+            -0.003071095095947385,
+            0.0020908592268824577,
+            0.028890179470181465,
+            0.009741820394992828,
+            -0.025015803053975105,
+            0.03509290888905525,
+            -0.018198391422629356,
+            0.003418019274249673,
+            -0.011492740362882614,
+            0.022482557222247124,
+            -0.014510284177958965,
+            0.030268562957644463,
+            -0.008018841035664082,
+            -0.006808098405599594,
+            -0.016000429168343544,
+            -0.018179764971137047,
+            0.005909354891628027,
+            0.04008489102125168,
+            -0.009639373049139977,
+            0.008675435557961464,
+            -0.019893432036042213,
+            0.013290227390825748,
+            0.0213463231921196,
+            -0.0006047891802154481,
+            -0.003813839051872492,
+            -0.01159518864005804,
+            -0.024084463715553284,
+            -0.005946608725935221,
+            0.0006787143065594137,
+            0.008852390572428703,
+            -0.0050478652119636536,
+            0.026394188404083252,
+            0.01940913498401642,
+            -0.0019651283510029316,
+            0.01808663085103035,
+            -0.008000214584171772,
+            -0.004433180205523968,
+            0.028610777109861374,
+            -0.002068739850074053,
+            0.0066683972254395485,
+            0.01785379648208618,
+            -0.0118093965575099,
+            -0.022780586034059525,
+            0.0059559219516813755,
+            0.01998656615614891,
+            0.02060125023126602,
+            0.029486237093806267,
+            -0.010570713318884373,
+            0.011129518039524555,
+            -0.014622044749557972,
+            -0.009946716018021107,
+            0.0041235098615288734,
+            0.0004292897356208414,
+            0.01940913498401642,
+            0.010850115679204464,
+            0.022389423102140427,
+            0.0027590959798544645,
+            -0.03609875589609146,
+            -0.002391216577962041,
+            -0.029206834733486176,
+            -0.0017311290139332414,
+            0.0025518727488815784,
+            0.003222437808290124,
+            -0.004312105942517519,
+            0.03416156768798828,
+            -0.011343725956976414,
+            -0.02516481839120388,
+            0.022724706679582596,
+            -0.0038976597134023905,
+            -0.0015809503383934498,
+            0.04656702280044556,
+            -0.030901875346899033,
+            -0.013020139187574387,
+            0.012647602707147598,
+            -0.0008125944877974689,
+            0.012181933037936687,
+            -0.0027102006133645773,
+            0.006118906661868095,
+            0.0175930205732584,
+            0.008503138087689877,
+            0.005401774309575558,
+            0.026375560089945793,
+            0.028331374749541283,
+            0.0010035192826762795,
+            0.009751134552061558,
+            -0.00637968210503459,
+            0.015171535313129425,
+            0.01613081619143486,
+            0.0007421618793159723,
+            -0.0006734755588695407,
+            0.04492786154150963,
+            -0.04392201453447342,
+            0.025891263037919998,
+            -0.024084463715553284,
+            0.018515046685934067,
+            0.005369177553802729,
+            -0.015246042981743813,
+            0.010878056287765503,
+            -0.031814590096473694,
+            -0.011446173302829266,
+            -0.024140343070030212,
+            -0.0009394896333105862,
+            -0.018747882917523384,
+            -0.025313831865787506,
+            0.009993283078074455,
+            0.0009202807559631765,
+            -0.006468159146606922,
+            0.0012631304562091827,
+            -0.008763913065195084,
+            -0.01996793784201145,
+            -0.0021467397455126047,
+            0.010281998664140701,
+            0.029635252431035042,
+            0.01451959740370512,
+            0.007124754134565592,
+            0.008833764120936394,
+            -0.04481610283255577,
+            -0.0322243794798851,
+            0.026729470118880272,
+            0.0029220804572105408,
+            -0.039228059351444244,
+            0.010729041881859303,
+            -0.006836038548499346,
+            -0.01725773885846138,
+            0.03937707468867302,
+            -0.026245173066854477,
+            -0.005704460199922323,
+            0.01733224466443062,
+            0.02393544837832451,
+            -0.008596272207796574,
+            0.0024307984858751297,
+            0.01483625266700983,
+            0.009881521575152874,
+            -0.003434317884966731,
+            0.013271600939333439,
+            -0.013523062691092491,
+            -0.012861811555922031,
+            0.023730553686618805,
+            0.029523491859436035,
+            0.019483640789985657,
+            -0.0024820221588015556,
+            0.0067801582626998425,
+            0.0035903172101825476,
+            0.026394188404083252,
+            0.03619188815355301,
+            -0.02192375436425209,
+            0.03127441182732582,
+            -0.006063025910407305,
+            -0.0355772040784359,
+            0.029430357739329338,
+            -0.0018033079104498029,
+            0.02415897138416767,
+            0.021830620244145393,
+            0.023302137851715088,
+            -0.009061941877007484,
+            -0.015450937673449516,
+            -0.01047758013010025,
+            0.03812907636165619,
+            0.014705865643918514,
+            -0.009583492763340473,
+            -0.015646519139409065,
+            -0.020470863208174706,
+            -0.008512451313436031,
+            0.01909247785806656,
+            -0.015981800854206085,
+            0.026859857141971588,
+            0.02909507416188717,
+            0.016736187040805817,
+            -0.0013271600473672152,
+            0.015469564124941826,
+            -0.02190512605011463,
+            -0.013085332699120045,
+            -0.03423607349395752,
+            -0.006426248699426651,
+            0.0044704340398311615,
+            0.010552086867392063,
+            0.0005579311400651932,
+            -0.006700994446873665,
+            0.02970976009964943,
+            0.01320640742778778,
+            -0.008037467487156391,
+            -0.010831489227712154,
+            0.03168420121073723,
+            -0.01613081619143486,
+            0.00866612233221531,
+            -0.03544681519269943,
+            -0.0072644553147256374,
+            -0.01790967583656311,
+            -0.018012123182415962,
+            0.04753561317920685,
+            -0.0235256589949131,
+            -0.0005762669024989009,
+            0.007483320310711861,
+            -0.008060751482844353,
+            -0.01030062511563301,
+            -0.0185802411288023,
+            0.014901447109878063,
+            0.0013876971788704395,
+            0.0027730660513043404,
+            -0.02522069774568081,
+            -0.0026822604704648256,
+            0.0009488030336797237,
+            -0.049584563821554184,
+            0.004912820644676685,
+            -0.03561445698142052,
+            -0.027251020073890686,
+            -0.00028114838642068207,
+            -0.056327469646930695,
+            0.02132769487798214,
+            -0.019893432036042213,
+            -0.0009668477578088641,
+            -0.025239326059818268,
+            0.012880438007414341,
+            0.01836603321135044,
+            -0.04712582379579544,
+            -0.012275067158043385,
+            -0.011902530677616596,
+            0.011557934805750847,
+            -0.010775608941912651,
+            -0.02553735487163067,
+            0.02484816312789917,
+            -0.038855522871017456,
+            0.007175977807492018,
+            -0.00043103599455207586,
+            0.017211170867085457,
+            0.007613707799464464,
+            0.02259431779384613,
+            0.00892689824104309,
+            0.012275067158043385,
+            -0.011651068925857544,
+            -0.0036881081759929657,
+            -0.03513016179203987,
+            0.004586851689964533,
+            0.008982778526842594,
+            0.02032184787094593,
+            -0.021048294380307198,
+            0.02199826017022133,
+            0.03231751173734665,
+            0.022203154861927032,
+            0.02747454307973385,
+            0.0038347942754626274,
+            0.022519811987876892,
+            0.008107318542897701,
+            -0.03233613818883896,
+            0.017695467919111252,
+            -0.006333115044981241,
+            -0.005192223004996777,
+            -0.020135579630732536,
+            -0.018608180806040764,
+            0.04556117206811905,
+            0.025276578962802887,
+            0.020116953179240227,
+            0.03296945244073868,
+            -0.03226163238286972,
+            -0.014780372381210327,
+            -0.016056308522820473,
+            -0.007231858558952808,
+            -0.013560316525399685,
+            -0.02747454307973385,
+            0.0016461441991850734,
+            -0.011893217451870441,
+            0.007883796468377113,
+            0.005290013737976551,
+            0.02000519260764122,
+            0.012526528909802437,
+            0.042729899287223816,
+            0.013979420065879822,
+            0.0056765200570225716,
+            0.013178466819226742,
+            0.0038813611026853323,
+            -0.009429821744561195,
+            0.03349100053310394,
+            -0.004423866979777813,
+            0.013495123013854027,
+            -0.01972579024732113,
+            0.03585660457611084,
+            0.0030361698009073734,
+            0.01081286184489727,
+            0.01468723826110363,
+            0.00033499152050353587,
+            -0.0336027629673481,
+            -0.005057178437709808,
+            0.027008872479200363,
+            0.036601677536964417,
+            0.012293693609535694,
+            -0.00782791618257761,
+            -0.02455013431608677,
+            0.018077317625284195,
+            -0.006226010620594025,
+            -0.0016985321417450905,
+            0.034105684608221054,
+            0.025090310722589493,
+            0.03103226236999035,
+            -0.014184314757585526,
+            0.0037719286046922207,
+            -0.02000519260764122,
+            0.008018841035664082,
+            0.019781669601798058,
+            0.036955587565898895,
+            -0.01081286184489727,
+            -0.012489275075495243,
+            -0.0006239980575628579,
+            0.03391941636800766,
+            -0.039898622781038284,
+            0.01355100329965353,
+            0.026524575427174568,
+            -0.02553735487163067,
+            0.019856177270412445,
+            -0.007311022374778986,
+            -0.034068431705236435,
+            0.010393759235739708,
+            -0.013774524442851543,
+            0.005350550636649132,
+            0.03255965933203697,
+            0.0006839531124569476,
+            -0.032131243497133255,
+            0.00471025425940752,
+            -0.024661894887685776,
+            -0.019241493195295334,
+            -0.007227201480418444,
+            -0.0002667417284101248,
+            0.03233613818883896,
+            -0.02844313718378544,
+            0.028461763635277748,
+            0.029840147122740746,
+            0.027549048885703087,
+            -0.008805823512375355,
+            0.010356505401432514,
+            -0.0261892918497324,
+            0.037570271641016006,
+            0.011660382151603699,
+            -0.011651068925857544,
+            0.013942166231572628,
+            0.009993283078074455,
+            -0.013513749465346336,
+            0.00011066943261539564,
+            -0.01421225443482399,
+            -0.01320640742778778,
+            -0.02423347719013691,
+            0.002894140314310789,
+            -0.03065972588956356,
+            -0.026897111907601357,
+            0.014500970020890236,
+            0.020824771374464035,
+            -0.004544941242784262,
+            0.00868940632790327,
+            0.00935997162014246,
+            -0.006076996214687824,
+            0.021066920831799507,
+            -0.00177769607398659,
+            0.01084080245345831,
+            -0.012368200346827507,
+            -0.001345786964520812,
+            -0.003157244063913822,
+            0.01854298822581768,
+            -0.07778555154800415,
+            -0.02484816312789917,
+            0.05010611563920975,
+            0.032373394817113876,
+            -0.014789685606956482,
+            -0.002556529361754656,
+            0.010822176001966,
+            -0.0005678266170434654,
+            0.018533675000071526,
+            0.004435508977621794,
+            0.015218102373182774,
+            -0.010961877182126045,
+            -0.011353040114045143,
+            -0.01113883126527071,
+            0.028238240629434586,
+            -0.002343485364690423,
+            -0.01275936421006918,
+            -0.02417759783565998,
+            0.02006107196211815,
+            0.0112692192196846,
+            -0.00757179781794548,
+            -0.0006111921393312514,
+            0.0031619006767868996,
+            -0.01307601947337389,
+            0.00035681980079971254,
+            -0.009145762771368027,
+            -0.01965128257870674,
+            0.019064538180828094,
+            0.025276578962802887,
+            0.009844268672168255,
+            0.01725773885846138,
+            0.029281342402100563,
+            0.008372750133275986,
+            0.01486419327557087,
+            0.005746370181441307,
+            0.007315678987652063,
+            -0.009844268672168255,
+            -0.01047758013010025,
+            0.008274959400296211,
+            0.0032131243497133255,
+            -0.014780372381210327,
+            -0.004163091536611319,
+            0.019614029675722122,
+            0.011399607174098492,
+            0.00034576014149934053,
+            0.009215613827109337,
+            0.022110020741820335,
+            0.013234347105026245,
+            -0.01804937794804573,
+            0.011865276843309402,
+            0.010542773641645908,
+            0.03581935167312622,
+            -0.00903400219976902,
+            -0.02877841889858246,
+            -0.006938486360013485,
+            0.021551217883825302,
+            0.004018733743578196,
+            -0.007418126333504915,
+            0.015683772042393684,
+            -0.01616806909441948,
+            0.07208574563264847,
+            0.0012747722212225199,
+            0.015786219388246536,
+            -0.027027498930692673,
+            -0.014789685606956482,
+            0.007497290149331093,
+            -0.014780372381210327,
+            0.021458083763718605,
+            0.002114142756909132,
+            0.026356933638453484,
+            0.014053926803171635,
+            0.014016672968864441,
+            0.0042399270460009575,
+            0.010542773641645908,
+            -0.0030734234023839235,
+            0.014631357975304127,
+            0.0019709491170942783,
+            0.007078187074512243,
+            -0.00109956378582865,
+            0.012200559489428997,
+            0.004558911547064781,
+            -0.0010244745062664151,
+            -0.0025239326059818268,
+            0.010011909529566765,
+            0.00854039192199707,
+            -0.007143381051719189,
+            0.004463449120521545,
+            -0.003844107501208782,
+            -0.030734233558177948,
+            -0.007418126333504915,
+            0.016037682071328163,
+            0.008503138087689877,
+            0.032392021268606186,
+            -0.0036485260352492332,
+            -0.018570927903056145,
+            0.02840588241815567,
+            0.0003201482759322971,
+            0.010216804221272469,
+            0.0182822123169899,
+            0.005313297268003225,
+            -0.027306901291012764,
+            0.03164694830775261,
+            -0.01905522495508194,
+            0.016652366146445274,
+            -0.023413898423314095,
+            -0.014798998832702637,
+            0.004258553963154554,
+            0.0030478115659207106,
+            -0.029448984190821648,
+            -0.005266730207949877,
+            0.01634502410888672,
+            0.024307984858751297,
+            0.00815388560295105,
+            0.009490358643233776,
+            0.010226117447018623,
+            0.004009420517832041,
+            0.013765211217105389,
+            0.01577690616250038,
+            -0.012796617113053799,
+            -0.007921050302684307,
+            0.0012421752326190472,
+            -0.015646519139409065,
+            0.013904912397265434,
+            -0.00281730480492115,
+            -0.0008748779073357582,
+            0.013886285945773125,
+            -0.02265019901096821,
+            -0.010375131852924824,
+            0.010132983326911926,
+            -0.009834954515099525,
+            -0.004193360451608896,
+            0.038222212344408035,
+            -0.006510069593787193,
+            0.008801166899502277,
+            -0.01565583236515522,
+            0.00038184956065379083,
+            0.012954945676028728,
+            -0.004202673677355051,
+            -0.026580454781651497,
+            -0.04381025210022926,
+            0.004917477257549763,
+            0.02905782125890255,
+            0.013001511804759502,
+            0.04056918993592262,
+            0.0019290389027446508,
+            -0.01996793784201145,
+            -0.025704994797706604,
+            0.02838725596666336,
+            -0.00868940632790327,
+            0.0017846811097115278,
+            -0.025555981323122978,
+            0.013560316525399685,
+            0.00861955527216196,
+            0.012833870947360992,
+            -0.007506603840738535,
+            -0.01501320768147707,
+            0.014277448877692223,
+            0.014072554185986519,
+            -0.03220575302839279,
+            0.007539200596511364,
+            0.004200345370918512,
+            0.0038557492662221193,
+            0.011520680971443653,
+            -0.01844054087996483,
+            -0.009974655695259571,
+            0.0004563567927107215,
+            0.0036881081759929657,
+            -0.012787303887307644,
+            0.0006915202829986811,
+            0.015273982658982277,
+            -0.02130906842648983,
+            -0.016540605574846268,
+            -0.01785379648208618,
+            -0.007674245163798332,
+            -0.003962853457778692,
+            0.00456123985350132,
+            0.01455685030668974,
+            -0.013802465051412582,
+            -0.008475197479128838,
+            -0.005136342253535986,
+            -0.004558911547064781,
+            0.02747454307973385,
+            0.006808098405599594,
+            -0.02613341249525547,
+            0.0068127550184726715,
+            0.012200559489428997,
+            -0.0011408920399844646,
+            -0.011082950979471207,
+            0.024065837264060974,
+            0.0008603257010690868,
+            0.003939569927752018,
+            -0.000613520503975451,
+            0.007902423851191998,
+            0.02229628898203373,
+            -0.00866612233221531,
+            0.007851200178265572,
+            0.014435776509344578,
+            0.013737271539866924,
+            0.00621669739484787,
+            -0.014649984426796436,
+            -0.012293693609535694,
+            -0.011362353339791298,
+            0.017071470618247986,
+            0.013485808856785297,
+            -0.012768677435815334,
+            -0.022110020741820335,
+            -0.012628976255655289,
+            -0.0011630113003775477,
+            0.0070688738487660885,
+            0.00862886942923069,
+            -0.008754599839448929,
+            0.023153122514486313,
+            0.020731637254357338,
+            0.004405240062624216,
+            -0.0046147918328642845,
+            -0.013737271539866924,
+            0.030119549483060837,
+            0.0014261150499805808,
+            0.0351860411465168,
+            0.013485808856785297,
+            0.005709116812795401,
+            0.012722110375761986,
+            -0.005541475489735603,
+            -0.018971404060721397,
+            0.029523491859436035,
+            0.0035181385464966297,
+            -0.010067789815366268,
+            0.06199001893401146,
+            0.0020198444835841656,
+            -0.0015972488326951861,
+            -0.008386720903217793,
+            -0.008866360411047935,
+            0.014426463283598423,
+            -0.004414553754031658,
+            0.01834740675985813,
+            -0.0015169207472354174,
+            0.01563720591366291,
+            -0.00870803277939558,
+            -0.006044399458914995,
+            0.0037020782474428415,
+            0.004223628900945187,
+            -0.007054903544485569,
+            0.014808312989771366,
+            0.02201688662171364,
+            0.006901232525706291,
+            -0.036974214017391205,
+            -0.02741866186261177,
+            -0.014249508269131184,
+            -0.037868302315473557,
+            0.014733805321156979,
+            0.008423973806202412,
+            9.793626668397337e-05,
+            0.014277448877692223,
+            -0.035968367010354996,
+            0.012247126549482346,
+            0.0122564397752285,
+            -0.0027614242862910032,
+            -0.0062120407819747925,
+            -0.0004831328405998647,
+            -0.02028459496796131,
+            -0.0011024741688743234,
+            0.01875719614326954,
+            0.013616196811199188,
+            -0.0007695200038142502,
+            0.02132769487798214,
+            -0.00475449301302433,
+            0.014612731523811817,
+            -0.006654427386820316,
+            0.025630488991737366,
+            -0.013830405659973621,
+            -0.017043529078364372,
+            -0.009788387455046177,
+            0.01468723826110363,
+            0.021532591432332993,
+            0.009467075578868389,
+            0.03034307062625885,
+            -0.007981587201356888,
+            0.009411195293068886,
+            -0.013849032111465931,
+            -0.014994580298662186,
+            0.00266130524687469,
+            0.024103090167045593,
+            0.004801060073077679,
+            0.015422997064888477,
+            0.013830405659973621,
+            -0.011772142723202705,
+            0.007939676754176617,
+            -0.007371559273451567,
+            0.0031130053102970123,
+            -0.008731316775083542,
+            -0.005969892255961895,
+            0.0038487643469125032,
+            -0.0006664904649369419,
+            0.012470648624002934,
+            0.002131605288013816,
+            0.018393972888588905,
+            -0.007595081347972155,
+            -0.005578729324042797,
+            -0.021811991930007935,
+            -0.001249160384759307,
+            0.0257981289178133,
+            -0.028554897755384445,
+            -0.02425210364162922,
+            0.016037682071328163,
+            0.0023073959164321423,
+            -0.0003850510693155229,
+            0.01243339478969574,
+            -0.04101623222231865,
+            -0.033360615372657776,
+            -0.0235256589949131,
+            -0.014854880049824715,
+            -0.0001042664734995924,
+            0.015078401193022728,
+            -0.01611218973994255,
+            -0.010486893355846405,
+            0.0219423808157444,
+            -0.015199475921690464,
+            0.024047208949923515,
+            -0.014286762103438377,
+            -0.0028359314892441034,
+            -0.004363330081105232,
+            0.006030429154634476,
+            0.00011692687985487282,
+            0.019893432036042213,
+            -0.019129732623696327,
+            -0.0030757517088204622,
+            -0.02071301080286503,
+            -0.001526234089396894,
+            0.00614219019189477,
+            0.001925546326674521,
+            -0.024587387219071388,
+            0.0016915469896048307,
+            0.020824771374464035,
+            -0.016615113243460655,
+            0.02715788595378399,
+            -0.01909247785806656,
+            0.020862026140093803,
+            -0.009611433371901512,
+            0.011688321828842163,
+            -0.008228392340242863,
+            -0.024307984858751297,
+            0.06638594716787338,
+            -0.0069245160557329655,
+            -0.00215488881804049,
+            -0.016456784680485725,
+            0.0016473083524033427,
+            -0.021066920831799507,
+            0.02581675723195076,
+            0.006896575912833214,
+            0.019427761435508728,
+            -0.014091180637478828,
+            0.026356933638453484,
+            -0.002856886712834239,
+            0.0064402190037071705,
+            0.02741866186261177,
+            -0.0018719942308962345,
+            -0.016792068257927895,
+            0.005313297268003225,
+            0.02291097491979599,
+            -0.020172834396362305,
+            -0.01747194677591324,
+            -0.021197307854890823,
+            -0.003471571486443281,
+            -0.004633418750017881,
+            -0.011259905993938446,
+            -0.002584469737485051,
+            0.002989602740854025,
+            -0.03349100053310394,
+            0.024680521339178085,
+            0.0036275710444897413,
+            -0.005108402110636234,
+            0.004400583449751139,
+            -0.026710843667387962,
+            0.000846355629619211,
+            0.00281730480492115,
+            -0.012694169767200947,
+            0.011716262437403202,
+            0.01646609790623188,
+            0.008447257801890373,
+            0.002136262133717537,
+            0.011846650391817093,
+            0.017015589401125908,
+            -0.017760662361979485,
+            0.008177168667316437,
+            -0.011362353339791298,
+            0.018654748797416687,
+            -0.010701101273298264,
+            -0.01371864415705204,
+            -0.01569308526813984,
+            0.001958143198862672,
+            -0.02060125023126602,
+            -0.004074614495038986,
+            0.00590469827875495,
+            -0.025071684271097183,
+            -0.0032573631033301353,
+            0.019837550818920135,
+            0.005057178437709808,
+            0.020862026140093803,
+            0.00023560001864098012,
+            -0.00982564128935337,
+            -0.031497932970523834,
+            -0.0025448875967413187,
+            -0.021495336666703224,
+            -0.012265753000974655,
+            0.016298457980155945,
+            0.007604394573718309,
+            -0.015497504733502865,
+            -0.005723086651414633,
+            0.001080936985090375,
+            -0.007734782062470913,
+            -0.035968367010354996,
+            0.03721636161208153,
+            0.006267921067774296,
+            0.014296075329184532,
+            -0.03390078991651535,
+            0.005313297268003225,
+            -0.003185184206813574,
+            -0.0005832519382238388,
+            -0.011511367745697498,
+            0.0032177811954170465,
+            -0.015627892687916756,
+            -0.00793502014130354,
+            -0.01939050666987896,
+            -0.009443791583180428,
+            -0.035670336335897446,
+            -0.021402202546596527,
+            -0.00012332983897067606,
+            0.012666230089962482,
+            0.008228392340242863,
+            0.001559995231218636,
+            0.02030322141945362,
+            -0.025555981323122978,
+            0.019465014338493347,
+            0.0015646519605070353,
+            0.02546284720301628,
+            0.006961769424378872,
+            -0.0007718483684584498,
+            0.008582302369177341,
+            -0.029933281242847443,
+            -0.013727957382798195,
+            -0.01755576767027378,
+            -0.0169038288295269,
+            -0.021551217883825302,
+            0.0064728157594799995,
+            -0.004742851015180349,
+            -0.017695467919111252,
+            -0.012638289481401443,
+            -0.004682314116507769,
+            0.004735866095870733,
+            -0.02287372015416622,
+            0.02255706489086151,
+            0.011278532445430756,
+            0.037942808121442795,
+            -0.01587004028260708,
+            0.02032184787094593,
+            0.00797693058848381,
+            -0.006519382819533348,
+            -0.01325297448784113,
+            -0.027213767170906067,
+            0.002933722222223878,
+            -0.0015553385019302368,
+            -0.014333329163491726,
+            0.004628761671483517,
+            0.012116738595068455,
+            0.026207920163869858,
+            0.027604930102825165,
+            -0.0209551602602005,
+            -0.008717346005141735,
+            -0.011371666565537453,
+            -0.006375025026500225,
+            0.0041048829443752766,
+            -0.019856177270412445,
+            -0.0012002650182694197,
+            -0.036340903490781784,
+            -0.00281730480492115,
+            -0.006351741496473551,
+            0.0003335362998768687,
+            0.006719620898365974,
+            -0.004540284629911184,
+            -0.0007153858314268291,
+            -0.00262405164539814,
+            -0.024680521339178085,
+            0.024028582498431206,
+            -0.021066920831799507,
+            -0.0072644553147256374,
+            -0.018496420234441757,
+            0.028629405423998833,
+            0.0025006490759551525,
+            0.005997832398861647,
+            -0.029914654791355133,
+            -0.011399607174098492,
+            -0.012964258901774883,
+            0.0038301374297589064,
+            0.00043481955071911216,
+            -0.00815388560295105,
+            0.03198223188519478,
+            0.008917584083974361,
+            0.004824343603104353,
+            0.012656915932893753,
+            0.020265966653823853,
+            -0.02004244551062584,
+            -0.019539522007107735,
+            -0.034422341734170914,
+            0.02227766253054142,
+            -0.017779288813471794,
+            -0.006370368413627148,
+            -0.002556529361754656,
+            0.0033318703062832355,
+            -0.007348275743424892,
+            -0.013625510036945343,
+            0.010254058055579662,
+            0.007990900427103043,
+            -0.011259905993938446,
+            -0.002887155395001173,
+            -0.009900148957967758,
+            0.002933722222223878,
+            -0.00788845308125019,
+            0.006458845920860767,
+            -0.009476388804614544,
+            0.007092157378792763,
+            0.0063563985750079155,
+            0.003799868980422616,
+            -0.011474113911390305,
+            0.028908805921673775,
+            0.010859428904950619,
+            0.01871994324028492,
+            0.010011909529566765,
+            -0.017462633550167084,
+            0.013262287713587284,
+            -0.006719620898365974,
+            0.026282425969839096,
+            -0.028461763635277748,
+            -0.021048294380307198,
+            -0.012628976255655289,
+            0.008866360411047935,
+            -0.006314488127827644,
+            -0.0033435120712965727,
+            -0.0006222518277354538,
+            0.0019756059627979994,
+            -0.01737881265580654,
+            -0.0012736080680042505,
+            0.02166297845542431,
+            0.006766187958419323,
+            0.007045590318739414,
+            -0.00759042426943779,
+            0.052788376808166504,
+            0.005886071361601353,
+            0.0219423808157444,
+            0.0003335362998768687,
+            0.01998656615614891,
+            -0.015739653259515762,
+            0.009900148957967758,
+            -0.03231751173734665,
+            -0.00177420349791646,
+            0.007520573679357767,
+            -0.007786005735397339,
+            0.0009982804767787457,
+            -0.003443631110712886,
+            0.009769761003553867,
+            0.002465723780915141,
+            -0.020489489659667015,
+            -0.027027498930692673,
+            -0.012954945676028728,
+            0.01650335267186165,
+            0.009038658812642097,
+            -0.011576561257243156,
+            -0.00351115339435637,
+            -0.00550422165542841,
+            0.0026636335533112288,
+            0.003341183764860034,
+            0.028834300115704536,
+            -0.038892775774002075,
+            0.0011263397755101323,
+            -0.00023865597904659808,
+            0.030920501798391342,
+            0.0037416601553559303,
+            -0.016046995297074318,
+            -0.00021944708714727312,
+            0.012470648624002934,
+            0.008526421152055264,
+            0.007236515171825886,
+            -0.0005649161757901311,
+            0.004423866979777813,
+            0.007031620014458895,
+            -0.009499671868979931,
+            0.014808312989771366,
+            -0.015283295884728432,
+            -0.009318061172962189,
+            -0.004035032354295254,
+            -0.007967617362737656,
+            0.003930256702005863,
+            0.005448341369628906,
+            0.0031293039210140705,
+            0.0007840722100809216,
+            -0.0006030429503880441,
+            -0.0012573095737025142,
+            -0.006468159146606922,
+            0.018328780308365822,
+            0.01998656615614891,
+            -0.0185802411288023,
+            0.0016158756334334612,
+            0.025276578962802887,
+            -0.025053057819604874,
+            0.014147060923278332,
+            0.023041361942887306,
+            0.003681123023852706,
+            -0.002554201055318117,
+            0.005648579448461533,
+            0.0007252813666127622,
+            0.010552086867392063,
+            0.007837229408323765,
+            0.01785379648208618,
+            0.010505519807338715,
+            -0.014482343569397926,
+            0.01771409437060356,
+            -0.006198070477694273,
+            -0.0011414741165935993,
+            -0.009243553504347801,
+            0.010784922167658806,
+            -0.002402858342975378,
+            0.004978014621883631,
+            -0.00016021965711843222,
+            -0.00423527043312788,
+            -0.007217888254672289,
+            0.013197094202041626,
+            0.00355539214797318,
+            0.017173917964100838,
+            -0.008023497648537159,
+            -0.0038720478769391775,
+            -0.02199826017022133,
+            -0.020265966653823853,
+            0.0024447685573250055,
+            0.019800297915935516,
+            0.02326488308608532,
+            -0.01666167937219143,
+            0.0074926335364580154,
+            0.008647495880723,
+            0.02265019901096821,
+            -0.008801166899502277,
+            -0.014147060923278332,
+            0.0034133626613765955,
+            -0.011073637753725052,
+            0.004714910872280598,
+            -0.012694169767200947,
+            -0.015301923267543316,
+            0.016689620912075043,
+            0.0212531890720129,
+            0.002812647959217429,
+            0.013709330931305885,
+            0.00645418930798769,
+            -0.004260882269591093,
+            0.019912058487534523,
+            -0.01678275503218174,
+            -0.017993496730923653,
+            -0.005369177553802729,
+            -0.0030920500867068768,
+            -0.00935997162014246,
+            -0.002068739850074053,
+            0.00021711873705498874,
+            0.0261892918497324,
+            0.007450723554939032,
+            0.003045483259484172,
+            -0.008503138087689877,
+            -0.002073396695777774,
+            -0.009224927052855492,
+            -0.027195140719413757,
+            0.008647495880723,
+            -0.007869826629757881,
+            -0.006742904428392649,
+            0.005103745497763157,
+            -0.01706215739250183,
+            -0.019129732623696327,
+            -0.007725468836724758,
+            -0.0022352170199155807,
+            -0.005681176669895649,
+            0.0166058000177145,
+            0.01771409437060356,
+            -0.022184528410434723,
+            -0.006598547101020813,
+            -0.00918301660567522,
+            -0.0012142350897192955,
+            -0.014957327395677567,
+            0.006724277976900339,
+            0.006826725322753191,
+            0.02199826017022133,
+            -0.023656046018004417,
+            -0.008251676335930824,
+            0.0015576668083667755,
+            -0.0015145923243835568,
+            0.02482953481376171,
+            -0.003285303246229887,
+            -0.017229797318577766,
+            -0.02836862951517105,
+            0.011828023009002209,
+            -0.008111975155770779,
+            -0.00614219019189477,
+            -0.019912058487534523,
+            -0.007450723554939032,
+            0.0017648900393396616,
+            -0.013020139187574387,
+            0.0027777228970080614,
+            -0.02712063305079937,
+            0.012377514503896236,
+            -0.010421698912978172,
+            -0.006673054303973913,
+            -0.0030221997294574976,
+            -0.011446173302829266,
+            0.026561828330159187,
+            0.019818924367427826,
+            0.016866575926542282,
+            0.015003894455730915,
+            0.008638182654976845,
+            0.0006239980575628579,
+            -0.00817251205444336,
+            0.003315571928396821,
+            -0.023693300783634186,
+            0.02548147365450859,
+            -0.002238709479570389,
+            -0.02190512605011463,
+            0.00493610417470336,
+            -0.004659030586481094,
+            -0.01079423539340496,
+            -0.000244331342400983,
+            0.012805930338799953,
+            -0.0003213124582543969,
+            -0.0021642022766172886,
+            0.010747668333351612,
+            0.013336794450879097,
+            0.011362353339791298,
+            0.030510712414979935,
+            0.016084248200058937,
+            -0.013513749465346336,
+            -0.02065712958574295,
+            0.011697635985910892,
+            -0.009723193943500519,
+            0.0044448222033679485,
+            0.025015803053975105,
+            0.017704781144857407,
+            -0.0017800243804231286,
+            -0.009047972038388252,
+            -0.0012771005276590586,
+            0.008079377934336662,
+            0.00935065746307373,
+            0.010552086867392063,
+            -0.012004978023469448,
+            -0.009425165131688118,
+            -0.01049620658159256,
+            -0.0027963495813310146,
+            0.001626353245228529,
+            -0.010859428904950619,
+            -0.0028475732542574406,
+            -0.001629845704883337,
+            0.011110891588032246,
+            0.010254058055579662,
+            -0.006319144740700722,
+            -0.054129503667354584,
+            0.007040933705866337,
+            0.010710414499044418,
+            0.009751134552061558,
+            -0.010505519807338715,
+            0.01870131492614746,
+            0.0012503245379775763,
+            -0.0014796671457588673,
+            0.010775608941912651,
+            0.010766295716166496,
+            0.0024936639238148928,
+            0.013495123013854027,
+            0.01583278737962246,
+            0.011092264205217361,
+            0.006840695161372423,
+            0.029951907694339752,
+            -0.021234560757875443,
+            -0.003932585008442402,
+            -0.022668825462460518,
+            -0.03349100053310394,
+            -0.007064217235893011,
+            0.01422156859189272,
+            0.004244584124535322,
+            0.008982778526842594,
+            0.020433608442544937,
+            0.03494389355182648,
+            0.013672077096998692,
+            -0.002324858447536826,
+            -0.002270142314955592,
+            -0.02162572368979454,
+            0.0037207049317657948,
+            -0.0014249508967623115,
+            0.00477311946451664,
+            0.006971083115786314,
+            -0.004305121023207903,
+            0.002726498991250992,
+            0.0090945390984416,
+            -0.0015588310780003667,
+            -0.00982564128935337,
+            -3.0923412850825116e-05,
+            0.010226117447018623,
+            0.01547887735068798,
+            -0.00524810329079628,
+            -0.017481260001659393,
+            0.007129410747438669,
+            0.022668825462460518,
+            -0.004165419843047857,
+            -0.004298136103898287,
+            -0.006840695161372423,
+            -0.013904912397265434,
+            -0.0024191567208617926,
+            -0.025630488991737366,
+            0.021811991930007935,
+            -0.012982885353267193,
+            0.017648901790380478,
+            0.01664305292069912,
+            -0.009313404560089111,
+            0.00016051069542299956,
+            -0.0027078723069280386,
+            0.03613600879907608,
+            0.0050478652119636536,
+            0.027008872479200363,
+            0.01692245528101921,
+            0.025239326059818268,
+            0.008722002618014812,
+            0.015786219388246536,
+            0.011883903294801712,
+            -0.004025719128549099,
+            0.00954623892903328,
+            0.011101577430963516,
+            0.008479854092001915,
+            -0.0032759900204837322,
+            0.013085332699120045,
+            0.0017660543089732528,
+            -0.018887583166360855,
+            -0.013904912397265434,
+            0.002901125466451049,
+            0.010654534213244915,
+            0.006738247815519571,
+            0.01840328611433506,
+            -0.008260989561676979,
+            -0.01723911054432392,
+            -0.0035088250879198313,
+            0.011697635985910892,
+            0.024661894887685776,
+            -0.01422156859189272,
+            -0.009620746597647667,
+            -0.02747454307973385,
+            0.0028662001714110374,
+            -0.0023341719061136246,
+            0.008503138087689877,
+            -0.0019860833417624235,
+            0.012163305655121803,
+            -0.00026732380501925945,
+            -0.015152908861637115,
+            -0.008614898659288883,
+            0.020396355539560318,
+            -0.026971617713570595,
+            0.017518512904644012,
+            0.0008731316193006933,
+            -0.01483625266700983,
+            0.017406752333045006,
+            0.0021502322051674128,
+            0.014249508269131184,
+            0.007902423851191998,
+            0.0070688738487660885,
+            -0.01245202124118805,
+            0.010207490995526314,
+            -0.024345237761735916,
+            0.010281998664140701,
+            0.011073637753725052,
+            0.0017159946728497744,
+            0.018868956714868546,
+            0.01356962975114584,
+            0.004668343812227249,
+            -0.019427761435508728,
+            0.009066599421203136,
+            -0.02544422075152397,
+            0.013271600939333439,
+            -0.001437756814993918,
+            0.024792281910777092,
+            -0.008600928820669651,
+            -0.0006106100627221167,
+            0.006058369297534227,
+            0.014910760335624218,
+            -0.003485541557893157,
+            -0.0020338145550340414,
+            0.006999023258686066,
+            -0.0006164309452287853,
+            -0.0017707109218463302,
+            0.0005800504586659372,
+            -0.019483640789985657,
+            -0.02294822782278061,
+            -0.0005075805238448083,
+            -0.009089882485568523,
+            0.01613081619143486,
+            -0.02745591476559639,
+            0.001415637438185513,
+            -0.00733430590480566,
+            0.003969838377088308,
+            -0.00020416728511918336,
+            -0.017145976424217224,
+            0.012209872715175152,
+            -0.005266730207949877,
+            0.0016321741277351975,
+            0.023134496062994003,
+            0.03362138941884041,
+            0.00048517013783566654,
+            -0.016223950311541557,
+            0.0175930205732584,
+            0.0066963378340005875,
+            -0.013392675668001175,
+            -0.020768892019987106,
+            -0.01049620658159256,
+            -0.017294991761446,
+            -0.009010718204081059,
+            0.0025518727488815784,
+            0.003837122581899166,
+            -0.0069571128115057945,
+            -0.010803548619151115,
+            0.010682474821805954,
+            -0.013755897991359234,
+            -0.02294822782278061,
+            0.00965800043195486,
+            -0.0012247127015143633,
+            0.016140129417181015,
+            0.0010663848370313644,
+            -0.004260882269591093,
+            0.019632656127214432,
+            -0.01664305292069912,
+            0.009741820394992828,
+            0.017499886453151703,
+            0.02296685427427292,
+            -0.026654962450265884,
+            -0.00935997162014246,
+            -0.012610348872840405,
+            -0.013625510036945343,
+            -0.0008585794712416828,
+            0.02740003541111946,
+            0.012172618880867958,
+            0.004596164915710688,
+            0.015450937673449516,
+            -0.0038417791947722435,
+            0.009089882485568523,
+            -0.025425594300031662,
+            0.016754813492298126,
+            -0.0118093965575099,
+            0.02615203894674778,
+            0.016223950311541557,
+            0.010077103041112423,
+            0.020806144922971725,
+            -0.0028149764984846115,
+            0.0018452182412147522,
+            -0.007380872964859009,
+            0.01773272082209587,
+            0.014715178869664669,
+            0.028312748298048973,
+            -0.011446173302829266,
+            0.016773441806435585,
+            0.005648579448461533,
+            0.005331923719495535,
+            -0.002933722222223878,
+            0.018794449046254158,
+            -0.0038464360404759645,
+            0.002593782963231206,
+            0.011297158896923065,
+            0.00281730480492115,
+            -0.01998656615614891,
+            -0.0023155452217906713,
+            0.012852497398853302,
+            0.0004173569323029369,
+            0.0030315129552036524,
+            0.0005468714516609907,
+            0.010952563025057316,
+            -0.0073575894348323345,
+            -0.018766509369015694,
+            0.017229797318577766,
+            0.016438158228993416,
+            -0.014128434471786022,
+            -0.026524575427174568,
+            -0.007324992213398218,
+            -0.000489535799715668,
+            0.021439457312226295,
+            0.0012293693143874407,
+            0.0047335377894341946,
+            0.014910760335624218,
+            0.010244744829833508,
+            0.003844107501208782,
+            -0.003769600298255682,
+            0.0009255195618607104,
+            -0.018459167331457138,
+            -0.003918614704161882,
+            0.01163244154304266,
+            -0.008330839686095715,
+            -1.7971959096030332e-05,
+            -0.0029593342915177345,
+            -0.008978121913969517,
+            -0.009965342469513416,
+            0.016717560589313507,
+            0.002754439366981387,
+            0.007525230757892132,
+            0.01193047035485506,
+            0.040196653455495834,
+            -0.007157351355999708,
+            -0.004172405228018761,
+            0.024624640122056007,
+            0.004095569718629122,
+            -0.001629845704883337,
+            0.0219423808157444,
+            -0.005178252700716257,
+            -0.01711803674697876,
+            0.0021607098169624805,
+            0.01842191442847252,
+            0.007324992213398218,
+            -0.007082843687385321,
+            0.012181933037936687,
+            -0.014780372381210327,
+            0.020824771374464035,
+            0.004991984460502863,
+            -0.009974655695259571,
+            -0.00937859807163477,
+            0.005313297268003225,
+            0.007939676754176617,
+            0.0043866136111319065,
+            0.024065837264060974,
+            0.019064538180828094,
+            -0.02032184787094593,
+            0.007166664581745863,
+            0.009807014837861061,
+            0.017034215852618217,
+            -0.02097378671169281,
+            -0.001526234089396894,
+            0.009816328063607216,
+            0.01552544441074133,
+            0.0025704996660351753,
+            0.003152587218210101,
+            -0.02004244551062584,
+            -0.005313297268003225,
+            -0.01940913498401642,
+            -0.016810694709420204,
+            -0.016140129417181015,
+            -0.006337771657854319,
+            0.00517359608784318,
+            0.01585141383111477,
+            -0.01646609790623188,
+            0.0033877508249133825,
+            0.0028103196527808905,
+            -0.024047208949923515,
+            -0.01404461357742548,
+            0.0029360507614910603,
+            0.002756767673417926,
+            -0.00086672866018489,
+            -0.0002302157081430778,
+            -0.007529887370765209,
+            -0.018161138519644737,
+            0.016279829666018486,
+            -0.012992198579013348,
+            -0.01404461357742548,
+            0.004889537114650011,
+            0.01402598712593317,
+            0.005993175785988569,
+            -0.0066683972254395485,
+            0.0029057820793241262,
+            -0.015022520907223225,
+            -0.009983968921005726,
+            0.012563781812787056,
+            0.0004697448166552931,
+            -0.02553735487163067,
+            0.00868940632790327,
+            0.02231491729617119,
+            -0.00725979870185256,
+            0.002270142314955592,
+            -0.003681123023852706,
+            -0.01179077010601759,
+            -0.02287372015416622,
+            0.0077813491225242615,
+            -0.005653236526995897,
+            -0.013681390322744846,
+            -0.0002440402895445004,
+            0.03188909590244293,
+            -0.012163305655121803,
+            -0.002957005752250552,
+            0.009723193943500519,
+            -0.0010244745062664151,
+            -0.009862895123660564,
+            -0.0017241439782083035,
+            -0.01826358586549759,
+            0.004225957207381725,
+            0.0022573363967239857,
+            -0.004682314116507769,
+            0.008028154261410236,
+            -0.0109711904078722,
+            -0.018626809120178223,
+            0.01340198889374733,
+            -0.007674245163798332,
+            0.002428470179438591,
+            -0.007511260453611612,
+            -0.006575263570994139,
+            -0.012172618880867958,
+            -0.03466448932886124,
+            0.013299541547894478,
+            -0.012470648624002934,
+            -0.02127181552350521,
+            -0.014584790915250778,
+            -0.005695146508514881,
+            0.014463717117905617,
+            0.004328404553234577,
+            -0.03485075756907463,
+            -0.018077317625284195,
+            0.00970456749200821,
+            0.015441624447703362,
+            -0.009751134552061558,
+            -0.0132436603307724,
+            0.011343725956976414,
+            0.038930028676986694,
+            0.0026776036247611046,
+            -0.007436753250658512,
+            0.000422304670792073,
+            0.01051483303308487,
+            -0.011697635985910892,
+            -0.004032704047858715,
+            0.021849246695637703,
+            0.010086417198181152,
+            -0.0056672063656151295,
+            -0.006193413864821196,
+            0.027828451246023178,
+            0.00942050851881504,
+            0.018338093534111977,
+            -0.006579920183867216,
+            -0.007641647942364216,
+            -0.014761745929718018,
+            -0.004277180880308151,
+            -0.015087714418768883,
+            -0.02097378671169281,
+            0.009834954515099525,
+            0.009862895123660564,
+            -0.011064324527978897,
+            0.006682367529720068,
+            0.0029407073743641376,
+            0.0229296013712883,
+            -0.0072644553147256374,
+            0.003131632227450609,
+            -0.015981800854206085,
+            0.004833656828850508,
+            -0.0036322276573628187,
+            0.0046730004251003265,
+            0.006076996214687824,
+            -0.010309938341379166,
+            0.006426248699426651,
+            -0.017015589401125908,
+            -0.00982564128935337,
+            0.0054390281438827515,
+            -0.005578729324042797,
+            0.00025306266616098583,
+            -0.001689218683168292,
+            0.017015589401125908,
+            -0.002021008636802435,
+            -0.005606669466942549,
+            -0.0004228867474012077,
+            0.012973572127521038,
+            -0.01163244154304266,
+            -0.02158847078680992,
+            0.009266837500035763,
+            0.003974495455622673,
+            0.009881521575152874,
+            -0.005117715802043676,
+            -0.0005270804977044463,
+            -0.009448448196053505,
+            0.0010093401651829481,
+            -0.02713925950229168,
+            -0.01662442646920681,
+            0.003101363545283675,
+            0.0011211010860279202,
+            0.02484816312789917,
+            0.024680521339178085,
+            -0.0012736080680042505,
+            -0.007422782946377993,
+            -0.013765211217105389,
+            -0.0036531828809529543,
+            -0.008116631768643856,
+            0.011706949211657047,
+            -0.01113883126527071,
+            -0.013951479457318783,
+            -7.86727980539581e-07,
+            -0.042096588760614395,
+            0.0005590952932834625,
+            -0.014826939441263676,
+            0.014873506501317024,
+            0.004787089768797159,
+            0.020880652591586113,
+            -0.0036345559637993574,
+            -0.003210796043276787,
+            -0.0045030307956039906,
+            0.008912927471101284,
+            0.009457762353122234,
+            0.013681390322744846,
+            0.004633418750017881,
+            -0.01968853734433651,
+            0.025630488991737366,
+            -0.016261203214526176,
+            -0.03511153534054756,
+            0.016531292349100113,
+            -0.02067575789988041,
+            0.011408920399844646,
+            -0.007874483242630959,
+            0.0025332458317279816,
+            -0.0043586730025708675,
+            0.0209551602602005,
+            -0.01688520237803459,
+            -0.015916608273983,
+            0.00019732775399461389,
+            -0.012610348872840405,
+            0.0008271466940641403,
+            -0.01838465966284275,
+            -0.011539307422935963,
+            -0.006673054303973913,
+            0.02320900373160839,
+            0.006258607842028141,
+            -0.0031805275939404964,
+            -0.025965770706534386,
+            0.015069087967276573,
+            -0.02425210364162922,
+            0.013429928570985794,
+            -0.0004976850468665361,
+            -2.5630051823100075e-05,
+            -0.016931768506765366,
+            0.031069517135620117,
+            0.004633418750017881,
+            0.0031665575224906206,
+            0.010142297483980656,
+            0.023358017206192017,
+            0.005364520940929651,
+            0.005154969170689583,
+            0.0015972488326951861,
+            -1.1186999472556636e-05,
+            -0.0166058000177145,
+            0.01771409437060356,
+            -0.007972273975610733,
+            0.020582623779773712,
+            -0.0111202048137784,
+            0.010067789815366268,
+            0.0037463167682290077,
+            -0.0033202285412698984,
+            0.005825534462928772,
+            -0.005806907545775175,
+            -0.013318167999386787,
+            0.017462633550167084,
+            -0.01974441669881344,
+            0.010933936573565006,
+            -0.006165473721921444,
+            0.005550788715481758,
+            -0.004139808006584644,
+            0.01289906445890665,
+            -0.006901232525706291,
+            -0.02099241316318512,
+            0.008754599839448929,
+            -0.01641021855175495,
+            0.013765211217105389,
+            0.014240195043385029,
+            -0.005718430038541555,
+            -0.013802465051412582,
+            -0.007804632652550936,
+            -0.007324992213398218,
+            0.025127563625574112,
+            -0.018990030512213707,
+            -0.013327481225132942,
+            -0.01062659453600645,
+            0.0006071175448596478,
+            0.006263264454901218,
+            0.014482343569397926,
+            -0.0012922348687425256,
+            0.007059560157358646,
+            0.017248423770070076,
+            -0.012032918632030487,
+            0.0024890073109418154,
+            -0.018114570528268814,
+            0.005704460199922323,
+            -0.0029919310472905636,
+            0.001448234310373664,
+            -0.0013504435773938894,
+            0.026710843667387962,
+            0.014612731523811817,
+            -0.019008658826351166,
+            -0.018487107008695602,
+            -0.006449532229453325,
+            -0.030436204746365547,
+            0.001108295051380992,
+            -0.020135579630732536,
+            -0.02287372015416622,
+            0.03770066052675247,
+            -0.015702400356531143,
+            -0.018971404060721397,
+            0.004533299710601568,
+            -0.005187565926462412,
+            -0.0011059667449444532,
+            -0.004305121023207903,
+            -0.01569308526813984,
+            0.0006711471942253411,
+            -0.018095944076776505,
+            -0.0049547310918569565,
+            0.024643268436193466,
+            0.0021246203687042,
+            0.01417500153183937,
+            -0.002210769336670637,
+            -0.015860727056860924,
+            0.0008812808664515615,
+            -0.005783624015748501,
+            -0.005345894023776054,
+            -0.0016577859641984105,
+            -0.009145762771368027,
+            -0.0023306794464588165,
+            0.013327481225132942,
+            0.011967724189162254,
+            0.0033761090599000454,
+            -0.019166985526680946,
+            -0.005704460199922323,
+            0.021495336666703224,
+            -0.0032317512668669224,
+            -0.013951479457318783,
+            -0.011334412731230259,
+            -0.007050246931612492,
+            0.005611326079815626,
+            0.005201536230742931,
+            0.0028079913463443518,
+            0.001922053750604391,
+            0.011064324527978897,
+            0.013048078864812851,
+            0.00109956378582865,
+            -0.02743728831410408,
+            0.011110891588032246,
+            0.001075698179192841,
+            -0.00645418930798769,
+            -0.006230667233467102,
+            0.00574171356856823,
+            -0.016438158228993416,
+            -0.017565080896019936,
+            -0.01223781332373619,
+            -0.00574171356856823,
+            -0.005518191959708929,
+            0.015078401193022728,
+            -0.030734233558177948,
+            0.010403072461485863,
+            -0.01854298822581768,
+            -0.014053926803171635,
+            0.001118190586566925,
+            0.006896575912833214,
+            -0.0028033347334712744,
+            0.014426463283598423,
+            -0.010132983326911926,
+            0.016475411131978035,
+            -0.01051483303308487,
+            -0.003876704489812255,
+            0.005536818876862526,
+            -0.01688520237803459,
+            -0.0002786745026241988,
+            0.01148342713713646,
+            0.021160054951906204,
+            -0.026282425969839096,
+            0.01514359563589096,
+            -0.0035064967814832926,
+            -0.0053272671066224575,
+            0.025611860677599907,
+            0.00524810329079628,
+            -0.0027777228970080614,
+            -0.002700887154787779,
+            -0.003732346696779132,
+            -0.003078080015257001,
+            -0.012517215684056282,
+            -0.0054855952039361,
+            -0.029635252431035042,
+            -0.011874590069055557,
+            -0.027232393622398376,
+            -0.016056308522820473,
+            0.0017381140496581793,
+            -0.008847733959555626,
+            0.0004837149172089994,
+            -0.01778860203921795,
+            0.008717346005141735,
+            0.024978550150990486,
+            -0.012712797150015831,
+            0.0035530636087059975,
+            0.0028894837014377117,
+            0.008805823512375355,
+            0.005867444444447756,
+            0.007059560157358646,
+            0.012293693609535694,
+            0.0012817572569474578,
+            0.0036718095652759075,
+            0.0029407073743641376,
+            -0.01966990903019905,
+            -0.013606883585453033,
+            -0.010766295716166496,
+            0.018468480557203293,
+            -0.0008836092310957611,
+            -0.027064751833677292,
+            0.0029290656093508005,
+            0.024382492527365685,
+            0.017024902626872063,
+            -0.01713666319847107,
+            0.004731209482997656,
+            0.01648472622036934,
+            0.023488406091928482,
+            0.0036974214017391205,
+            0.041127994656562805,
+            -0.0001826300285756588,
+            0.028461763635277748,
+            0.005415744613856077,
+            0.009397224523127079,
+            0.0002062045969069004,
+            -0.03254103288054466,
+            -0.0020489490125328302,
+            -0.013318167999386787,
+            -0.012051545083522797,
+            0.033397868275642395,
+            0.016615113243460655,
+            -0.017816541716456413,
+            0.0054855952039361,
+            -0.00030937965493649244,
+            0.005588042549788952,
+            -0.02617066539824009,
+            -0.0012631304562091827,
+            0.0069571128115057945,
+            0.006179443560540676,
+            0.00517359608784318,
+            -0.028890179470181465,
+            0.003296945011243224,
+            -0.005210849456489086,
+            -0.0061841001734137535,
+            0.0024075149558484554,
+            -0.009252866730093956,
+            -0.022091394290328026,
+            0.020210087299346924,
+            0.009355314075946808,
+            0.016810694709420204,
+            0.012722110375761986,
+            -0.008940868079662323,
+            -0.008139914833009243,
+            -0.02328351140022278,
+            -0.01066384743899107,
+            0.001526234089396894,
+            0.0019057553727179766,
+            -0.002250351244583726,
+            0.017704781144857407,
+            0.0025379026774317026,
+            -0.00637968210503459,
+            -0.0008271466940641403,
+            -0.0012479961151257157,
+            -0.01579553261399269,
+            -0.003152587218210101,
+            -0.008917584083974361,
+            0.00839137751609087,
+            0.01404461357742548,
+            -0.012619663029909134,
+            -0.007138724438846111,
+            0.014435776509344578,
+            0.005681176669895649,
+            0.024289358407258987,
+            -0.010822176001966,
+            0.0014179657446220517,
+            0.02583538368344307,
+            -0.010878056287765503,
+            -0.010878056287765503,
+            0.012945631518959999,
+            0.015981800854206085,
+            0.014333329163491726,
+            0.0013876971788704395,
+            0.005373834166675806,
+            -0.012377514503896236,
+            -0.007855856791138649,
+            0.009751134552061558,
+            -0.028797045350074768,
+            8.30202698125504e-05,
+            0.00817251205444336,
+            -0.025332460179924965,
+            -0.0009947880171239376,
+            -0.013476495631039143,
+            0.0006007145857438445,
+            -0.010067789815366268,
+            -0.004526314325630665,
+            0.00039320028736256063,
+            0.012042231857776642,
+            -0.007720812223851681,
+            -0.019893432036042213,
+            0.0025774845853447914,
+            -0.020396355539560318,
+            0.021066920831799507,
+            -0.016065621748566628,
+            0.013290227390825748,
+            0.004768462851643562,
+            0.015981800854206085,
+            0.024680521339178085,
+            -0.006831381935626268,
+            -0.005713773425668478,
+            0.01519016269594431,
+            -0.013960792683064938,
+            -0.007557827513664961,
+            -0.0030478115659207106,
+            0.007930363528430462,
+            0.011734888888895512,
+            0.012200559489428997,
+            0.00201169541105628,
+            -0.01179077010601759,
+            0.01081286184489727,
+            -0.00044704339234158397,
+            0.002244530478492379,
+            0.014733805321156979,
+            -0.0072644553147256374,
+            -0.009648687206208706,
+            -0.0029034537728875875,
+            0.0034366461914032698,
+            -0.015199475921690464,
+            -0.00423527043312788,
+            0.0053272671066224575,
+            -0.02939310297369957,
+            -0.013178466819226742,
+            0.0018638450419530272,
+            -0.006896575912833214,
+            -0.03978686407208443,
+            -0.013541690073907375,
+            -0.009006061591207981,
+            -0.008950181305408478,
+            0.0041561066173017025,
+            -0.008475197479128838,
+            -0.012004978023469448,
+            0.021215934306383133,
+            0.027251020073890686,
+            0.024345237761735916,
+            -0.007851200178265572,
+            -0.01131578627973795,
+            0.021495336666703224,
+            -0.02192375436425209,
+            -0.005452997982501984,
+            -0.010980503633618355,
+            -0.0019139045616611838,
+            0.0034506162628531456,
+            0.007795319426804781,
+            0.015758279711008072,
+            -0.0018382332054898143,
+            0.013979420065879822,
+            -0.025053057819604874,
+            0.014202941209077835,
+            -0.004379628226161003,
+            0.0056672063656151295,
+            0.001721815555356443,
+            0.003895331406965852,
+            -0.006887262221425772,
+            0.019297374412417412,
+            -0.002672947011888027,
+            -0.013783838599920273,
+            0.023022735491394997,
+            -0.005839504301548004,
+            -0.002350470284000039,
+            0.0007084007957018912,
+            -0.00501061137765646,
+            0.025239326059818268,
+            0.0111202048137784,
+            -0.011017757467925549,
+            0.016512665897607803,
+            0.006910545751452446,
+            0.014938700012862682,
+            -0.024382492527365685,
+            0.005737056955695152,
+            0.010719728656113148,
+            -0.011706949211657047,
+            0.015786219388246536,
+            -0.02548147365450859,
+            0.018198391422629356,
+            0.0047335377894341946,
+            -0.011949097737669945,
+            -0.019204240292310715,
+            0.030845994129776955,
+            0.02069438435137272,
+            -0.0004354016564320773,
+            0.010803548619151115,
+            -0.017490573227405548,
+            -0.004342374857515097,
+            0.010114356875419617,
+            -0.0020605907775461674,
+            0.00767890177667141,
+            -0.013970105908811092,
+            0.0013131899759173393,
+            -0.009555553086102009,
+            0.007143381051719189,
+            -0.015711713582277298,
+            -0.0010431013070046902,
+            -0.03138617053627968,
+            0.0010221460834145546,
+            -0.01434264238923788,
+            0.0031432739924639463,
+            0.012973572127521038,
+            0.0029593342915177345,
+            0.010281998664140701,
+            0.008116631768643856,
+            0.021644352003932,
+            0.02060125023126602,
+            0.016521979123353958,
+            0.002498320769518614,
+            0.009495015256106853,
+            -0.006193413864821196,
+            0.01178145594894886,
+            0.0013294884702190757,
+            -0.007371559273451567,
+            0.010226117447018623,
+            -0.01852436177432537,
+            -0.013234347105026245,
+            0.023171748965978622,
+            -0.003755630226805806,
+            0.00841000396758318,
+            -0.01709941029548645,
+            -0.006738247815519571,
+            0.01113883126527071,
+            -0.009639373049139977,
+            0.014286762103438377,
+            -0.015897981822490692,
+            0.021122800186276436,
+            0.030529338866472244,
+            -0.016615113243460655,
+            -0.01891552470624447,
+            -0.02646869421005249,
+            0.0064728157594799995,
+            -0.01532986294478178,
+            -0.010095730423927307,
+            -0.002369097201153636,
+            0.0025402309838682413,
+            -3.296072100056335e-05,
+            0.024438371881842613,
+            0.005317953880876303,
+            -0.0013329809298738837,
+            -0.009788387455046177,
+            -0.011744203045964241,
+            0.019129732623696327,
+            0.002738140756264329,
+            -0.00970456749200821,
+            -0.00501061137765646,
+            -0.024121716618537903,
+            -0.0059233251959085464,
+            0.008512451313436031,
+            -0.0027590959798544645,
+            -0.022706080228090286,
+            0.014147060923278332,
+            0.006808098405599594,
+            -0.025555981323122978,
+            -0.004563568159937859,
+            -0.002268978161737323,
+            0.0012584737269207835,
+            -0.007948990911245346,
+            0.007362246047705412,
+            -0.003385422518476844,
+            0.007352932821959257,
+            0.006645113695412874,
+            -0.007511260453611612,
+            0.013998046517372131,
+            0.0005355207249522209,
+            0.01706215739250183,
+            -0.012312320061028004,
+            -0.01569308526813984,
+            -0.01143686007708311,
+            0.014202941209077835,
+            0.0011909515596926212,
+            0.009797701612114906,
+            -0.004184046760201454,
+            -0.0043982551433146,
+            0.0014074882492423058,
+            0.0008958330727182329,
+            0.003196825971826911,
+            -0.01907385140657425,
+            0.03414294123649597,
+            0.009983968921005726,
+            -0.00013373466208577156,
+            7.618073868798092e-05,
+            -0.020414981991052628,
+            0.007460036780685186,
+            0.0005011775647290051,
+            -0.0047521647065877914,
+            -0.004945417400449514,
+            0.0033691239077597857,
+            0.014230881817638874,
+            0.013783838599920273,
+            0.008414660580456257,
+            0.008195796050131321,
+            0.013113273307681084,
+            0.01434264238923788,
+            -0.0012805931037291884,
+            0.011222652159631252,
+            -0.014072554185986519,
+            -0.00235512712970376,
+            -0.016242576763033867,
+            -0.016726873815059662,
+            0.029281342402100563,
+            -0.019465014338493347,
+            0.022799214348196983,
+            0.008177168667316437,
+            0.0024191567208617926,
+            0.024661894887685776,
+            0.006221354007720947,
+            0.025313831865787506,
+            0.017509199678897858,
+            0.015721026808023453,
+            0.0023493061307817698,
+            0.009518299251794815,
+            0.014184314757585526,
+            0.005513535346835852,
+            0.008475197479128838,
+            0.00039640176692046225,
+            -0.005862787831574678,
+            -0.010747668333351612,
+            0.00355539214797318,
+            0.009741820394992828,
+            -0.011883903294801712,
+            -0.008260989561676979,
+            0.017285678535699844,
+            -0.0053272671066224575,
+            -0.011380979791283607,
+            0.00090747483773157,
+            0.005154969170689583,
+            0.012684856541454792,
+            -0.0027404692955315113,
+            -0.007776692509651184,
+            0.002151396358385682,
+            -0.023991329595446587,
+            0.010375131852924824,
+            0.011101577430963516,
+            0.0039046446327120066,
+            0.004917477257549763,
+            0.003913958091288805,
+            0.007720812223851681,
+            0.0004953566822223365,
+            -0.013085332699120045,
+            0.010719728656113148,
+            -0.025891263037919998,
+            -0.008107318542897701,
+            -0.021439457312226295,
+            -0.011585874482989311,
+            -0.007115440908819437,
+            -0.01820770464837551,
+            -0.014817626215517521,
+            0.00671030767261982,
+            -0.021458083763718605,
+            0.0028103196527808905,
+            -0.001522741629742086,
+            -0.01737881265580654,
+            -0.002635693410411477,
+            0.0118093965575099,
+            0.017267052084207535,
+            0.029504865407943726,
+            0.011548621580004692,
+            0.005983862094581127,
+            0.009024688974022865,
+            0.0062446375377476215,
+            -0.007548514287918806,
+            0.03930256515741348,
+            0.0011152802035212517,
+            -0.015758279711008072,
+            0.014016672968864441,
+            -0.014752432703971863,
+            -0.007040933705866337,
+            -0.007036277092993259,
+            0.0053784907795488834,
+            0.0084984814748168,
+            0.020731637254357338,
+            -0.01840328611433506,
+            0.0005104909650981426,
+            0.007562484126538038,
+            0.023153122514486313,
+            0.004745179321616888,
+            0.011716262437403202,
+            -0.0019022627966478467,
+            0.024270731955766678,
+            -0.003932585008442402,
+            -0.0010052655125036836,
+            -0.015376430004835129,
+            -0.008982778526842594,
+            -0.006840695161372423,
+            -0.004731209482997656,
+            -0.017015589401125908,
+            -0.01909247785806656,
+            -0.006095623131841421,
+            0.005303983576595783,
+            -0.007944333367049694,
+            0.010431013070046902,
+            0.007478663697838783,
+            0.014268135651946068,
+            0.018077317625284195,
+            0.025704994797706604,
+            0.010403072461485863,
+            0.005974548868834972,
+            0.010077103041112423,
+            -0.009862895123660564,
+            -0.002845244947820902,
+            -0.006836038548499346,
+            -0.04403377324342728,
+            0.005117715802043676,
+            0.0030245280358940363,
+            -0.01628914475440979,
+            0.0005954757798463106,
+            -0.009965342469513416,
+            0.015376430004835129,
+            0.00629120459780097,
+            -0.011427546851336956,
+            -0.015534758567810059,
+            0.015767592936754227,
+            -0.010440326295793056,
+            0.014873506501317024,
+            0.007422782946377993,
+            0.011604501865804195,
+            -0.006235324312001467,
+            0.019539522007107735,
+            -0.004875567276030779,
+            -0.01535780355334282,
+            0.017481260001659393,
+            0.008633526042103767,
+            -0.0006752217886969447,
+            0.010421698912978172,
+            0.0014191300142556429,
+            -0.02037772908806801,
+            -0.010188864544034004,
+            0.031423427164554596,
+            -0.001983755035325885,
+            -0.015804847702383995,
+            -0.002610081573948264,
+            -0.01062659453600645,
+            0.005769653711467981,
+            0.006645113695412874,
+            -0.004691627342253923,
+            -0.01163244154304266,
+            -0.020526742562651634,
+            -0.018785135820508003,
+            -0.028648031875491142,
+            0.02577950246632099,
+            -0.0017509199678897858,
+            -0.006901232525706291,
+            0.011911843903362751,
+            -5.009592860005796e-05,
+            0.0008358780178241432,
+            -0.0023702613543719053,
+            -0.010542773641645908,
+            0.07178772240877151,
+            -0.019837550818920135,
+            0.010570713318884373,
+            0.00016967857663985342,
+            -0.002754439366981387,
+            0.0017683826154097915,
+            -0.003969838377088308,
+            0.00954623892903328,
+            0.003795212134718895,
+            -0.011744203045964241,
+            -0.01370001770555973,
+            0.003641541115939617,
+            -0.014826939441263676,
+            -0.010524147190153599,
+            -0.0012503245379775763,
+            -0.006575263570994139,
+            -0.019008658826351166,
+            0.007534543983638287,
+            0.0007706842152401805,
+            -0.004507687408477068,
+            -0.008586958982050419,
+            -0.008079377934336662,
+            -0.01776997558772564,
+            0.010235431604087353,
+            -0.0021071576047688723,
+            -0.003415690967813134,
+            -0.03974960744380951,
+            -0.023749180138111115,
+            -0.008871017955243587,
+            -0.028834300115704536,
+            0.00815388560295105,
+            -0.015106341801583767,
+            -0.023786434903740883,
+            0.0068732923828065395,
+            0.007539200596511364,
+            0.006253950763493776,
+            -0.01761164702475071,
+            0.011241278611123562,
+            -0.014053926803171635,
+            -0.002642678329721093,
+            -0.0075531709007918835,
+            0.012889751233160496,
+            0.009145762771368027,
+            -8.149228960974142e-05,
+            -0.013774524442851543,
+            -0.009574179537594318,
+            0.01531123649328947,
+            -0.010086417198181152,
+            0.007893109694123268,
+            -0.004593836609274149,
+            -0.014771059155464172,
+            -0.020526742562651634,
+            0.01755576767027378,
+            0.0033132436219602823,
+            -0.006556636653840542,
+            0.01483625266700983,
+            0.0017450990853831172,
+            -0.00021871947683393955,
+            0.024643268436193466,
+            -0.011185398325324059,
+            -0.011772142723202705,
+            -0.008763913065195084,
+            -0.015544071793556213,
+            0.009630059823393822,
+            -0.019446387887001038,
+            0.011185398325324059,
+            -0.01289906445890665,
+            -0.022352170199155807,
+            -0.005015267990529537,
+            0.00177769607398659,
+            0.02453150600194931,
+            0.0018242630176246166,
+            -0.020563997328281403,
+            -0.012852497398853302,
+            0.006919859442859888,
+            -0.025276578962802887,
+            0.00023836494074203074,
+            0.0002408388099865988,
+            0.015450937673449516,
+            -0.016186697408556938,
+            0.016196010634303093,
+            -0.008600928820669651,
+            -0.0049873278476297855,
+            -0.0028545584063977003,
+            0.016037682071328163,
+            -0.0023621120490133762,
+            -0.024997176602482796,
+            -0.024103090167045593,
+            -0.01273142360150814,
+            -0.013578942976891994,
+            -0.0024075149558484554,
+            0.0226315725594759,
+            0.0024773655459284782,
+            0.026003023609519005,
+            0.007474007084965706,
+            -0.015152908861637115,
+            -0.015711713582277298,
+            -8.636726852273569e-05,
+            0.008265646174550056,
+            0.005411088000983,
+            0.001437756814993918,
+            -0.0017439349321648479,
+            -0.007422782946377993,
+            -0.007762722205370665,
+            -0.0033248853869736195,
+            0.03105088882148266,
+            0.0074414098635315895,
+            -0.0022596647031605244,
+            -0.005029238294810057,
+            -0.002712528919801116,
+            0.005611326079815626,
+            0.0027102006133645773,
+            -0.007162007968872786,
+            0.007767379283905029,
+            -0.004624105058610439,
+            0.0053365807980299,
+            0.025239326059818268,
+            -0.002100172685459256,
+            0.015572011470794678,
+            0.016894515603780746,
+            -0.01533917710185051,
+            -0.026394188404083252,
+            0.025611860677599907,
+            0.0019942326471209526,
+            -0.011520680971443653,
+            -0.011557934805750847,
+            -0.0226315725594759,
+            -0.006337771657854319,
+            -0.004884880501776934,
+            0.006752218119800091,
+            -0.009355314075946808,
+            -0.01695970818400383,
+            -0.006272577680647373,
+            -0.007376216351985931,
+            0.0015658161137253046,
+            -0.003650854341685772,
+            0.017704781144857407,
+            0.01840328611433506,
+            -0.01131578627973795,
+            0.024792281910777092,
+            -0.01532986294478178,
+            0.0567372590303421,
+            0.0015530101954936981,
+            -0.006770845036953688,
+            -0.010142297483980656,
+            -0.012116738595068455,
+            -0.012330947443842888,
+            0.0010780264856293797,
+            0.012507901526987553,
+            -0.0001615293585928157,
+            -0.016261203214526176,
+            -0.008028154261410236,
+            0.00652869651094079,
+            0.007474007084965706,
+            -0.0028010064270347357,
+            0.005308640189468861,
+            0.026952991262078285,
+            0.0002060590632027015,
+            -0.014491656795144081,
+            0.02974701300263405,
+            -0.003955868538469076,
+            0.003459929721429944,
+            0.0024890073109418154,
+            0.008824450895190239,
+            -0.005704460199922323,
+            0.00614219019189477,
+            0.013234347105026245,
+            0.003997778985649347,
+            0.012638289481401443,
+            0.0015611593844369054,
+            -0.0008975793025456369,
+            0.009252866730093956,
+            -0.009555553086102009,
+            0.00508511858060956,
+            -0.002896468620747328,
+            7.610797183588147e-05,
+            -0.003762615378946066,
+            -0.004076942801475525,
+            -0.014985267072916031,
+            0.014491656795144081,
+            -0.006035085767507553,
+            -0.004340046551078558,
+            -0.003755630226805806,
+            -0.011706949211657047,
+            -0.019614029675722122,
+            -0.014333329163491726,
+            0.017937617376446724,
+            -0.016019055619835854,
+            -0.012973572127521038,
+            0.026971617713570595,
+            0.015907295048236847,
+            -0.03125578537583351,
+            -0.013783838599920273,
+            -0.008330839686095715,
+            -0.014566164463758469,
+            0.018626809120178223,
+            -0.0074088131077587605,
+            0.007650961633771658,
+            -0.013280914165079594,
+            0.01453822385519743,
+            0.005876758135855198,
+            -0.021141426637768745,
+            0.010617280378937721,
+            -0.009075912646949291
+          ],
+          "title": "C Encouraging Exploration Across Experts"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "654d315e-0be0-48f5-9d63-6aaba71a1b2e",
+      "type": "next",
+      "source": {
+        "id": "460c555c-6aea-4625-af79-5ad4ea6dd448",
+        "properties": {
+          "page_content": "C Encouraging Exploration Across Experts 29 ",
+          "keyphrases": [
+            "Encouraging Exploration",
+            "Experts"
+          ],
+          "embedding": [
+            0.029896028339862823,
+            0.007837229408323765,
+            -0.022426677867770195,
+            -0.02060125023126602,
+            -0.0056159826926887035,
+            0.03129303827881813,
+            -0.005033894907683134,
+            0.0729425847530365,
+            -0.001533219125121832,
+            -0.0002334171877009794,
+            -0.04105348512530327,
+            -0.01453822385519743,
+            0.016559232026338577,
+            -0.02227766253054142,
+            -0.02035910077393055,
+            0.05126097798347473,
+            0.010049163363873959,
+            0.008787197060883045,
+            -0.022743333131074905,
+            -0.01933462731540203,
+            0.0470513179898262,
+            -0.040829963982105255,
+            -0.06571538001298904,
+            0.061505720019340515,
+            0.04552391916513443,
+            0.010365818627178669,
+            0.024457000195980072,
+            0.007217888254672289,
+            -0.06351741403341293,
+            0.04153778403997421,
+            0.019949311390519142,
+            0.008465884253382683,
+            -0.008838420733809471,
+            -0.03708597645163536,
+            -0.02613341249525547,
+            -0.024401118978857994,
+            0.018803762272000313,
+            -0.012619663029909134,
+            -0.013113273307681084,
+            0.015842100605368614,
+            -0.007562484126538038,
+            -0.030492085963487625,
+            -0.016037682071328163,
+            0.019148359075188637,
+            0.00031636469066143036,
+            -0.002261993009597063,
+            -0.041463274508714676,
+            -0.036601677536964417,
+            -0.014938700012862682,
+            -0.023302137851715088,
+            -0.02192375436425209,
+            -0.0339939258992672,
+            0.00943913497030735,
+            -0.010915310122072697,
+            0.007962960749864578,
+            0.03365864232182503,
+            0.030436204746365547,
+            -0.008545048534870148,
+            0.03002641536295414,
+            0.04511412978172302,
+            0.022780586034059525,
+            -0.0023202018346637487,
+            -0.02482953481376171,
+            -0.000566662463825196,
+            0.009844268672168255,
+            -0.041463274508714676,
+            -0.01307601947337389,
+            0.011865276843309402,
+            -0.006072339601814747,
+            -0.04101623222231865,
+            -0.014473030343651772,
+            0.010552086867392063,
+            -0.02872253768146038,
+            0.029784265905618668,
+            0.021495336666703224,
+            0.015385744161903858,
+            0.06273508816957474,
+            0.03393804654479027,
+            -0.030249936506152153,
+            -0.0006822068826295435,
+            0.0351860411465168,
+            -0.03609875589609146,
+            0.03326747938990593,
+            0.001204921631142497,
+            0.03285769000649452,
+            -0.01790967583656311,
+            -0.03591248765587807,
+            -0.08165992796421051,
+            0.052415840327739716,
+            0.0031362888403236866,
+            -0.02225903607904911,
+            -0.0011286681983619928,
+            0.003944226540625095,
+            -0.010617280378937721,
+            -0.05465105548501015,
+            0.026040278375148773,
+            0.008135258220136166,
+            0.0008545048185624182,
+            0.0029989161994308233,
+            -0.013578942976891994,
+            0.02455013431608677,
+            0.009993283078074455,
+            -0.015963174402713776,
+            -0.022668825462460518,
+            0.030845994129776955,
+            0.0056159826926887035,
+            0.01548819150775671,
+            0.020508116111159325,
+            -0.011343725956976414,
+            0.08039330691099167,
+            -0.027902958914637566,
+            0.005285356659442186,
+            -0.017024902626872063,
+            0.021420828998088837,
+            -0.013923538848757744,
+            -0.02676672302186489,
+            0.000443550874479115,
+            -0.04485335573554039,
+            0.021718857809901237,
+            0.00806540809571743,
+            -0.01486419327557087,
+            0.01780722849071026,
+            0.01503183413296938,
+            -0.029467610642313957,
+            -0.031423427164554596,
+            0.023767806589603424,
+            -0.010598653927445412,
+            0.014119120314717293,
+            0.020526742562651634,
+            0.04522589221596718,
+            0.030752860009670258,
+            0.0222404096275568,
+            -0.004028047434985638,
+            0.024419745430350304,
+            0.041798558086156845,
+            0.027213767170906067,
+            0.046678781509399414,
+            0.02423347719013691,
+            -0.02546284720301628,
+            -0.002768409438431263,
+            -0.03900453820824623,
+            0.018803762272000313,
+            -0.0029407073743641376,
+            0.01745332032442093,
+            -0.010216804221272469,
+            -0.00982564128935337,
+            0.024997176602482796,
+            0.04455532506108284,
+            -0.04906301200389862,
+            0.01417500153183937,
+            0.008279616013169289,
+            0.04973357915878296,
+            0.014910760335624218,
+            -0.009583492763340473,
+            0.03220575302839279,
+            -0.001704353024251759,
+            -0.014454402960836887,
+            0.0264873206615448,
+            -0.020545369014143944,
+            -0.021886499598622322,
+            0.013467182405292988,
+            -0.008177168667316437,
+            0.005229476373642683,
+            0.012498588301241398,
+            0.011893217451870441,
+            0.02715788595378399,
+            0.0011496233055368066,
+            -0.016913142055273056,
+            -0.07972273975610733,
+            0.04734934866428375,
+            0.0036461977288126945,
+            -0.018338093534111977,
+            0.04943554848432541,
+            0.002908110385760665,
+            0.0007311022491194308,
+            -0.020210087299346924,
+            -0.0017765318043529987,
+            -0.0043516880832612514,
+            -0.00693382928147912,
+            0.015860727056860924,
+            -0.04433180391788483,
+            -0.0029942593537271023,
+            0.022426677867770195,
+            -0.013141212984919548,
+            0.010701101273298264,
+            0.005406430922448635,
+            0.08411866426467896,
+            -0.016335710883140564,
+            0.028163734823465347,
+            0.017742035910487175,
+            0.015246042981743813,
+            -0.031516559422016144,
+            0.003408706048503518,
+            0.018328780308365822,
+            0.013755897991359234,
+            -0.011129518039524555,
+            0.009816328063607216,
+            0.03481350466609001,
+            0.0048429700545966625,
+            -0.025369713082909584,
+            -0.01563720591366291,
+            -0.03447822108864784,
+            0.01683863438665867,
+            0.000613520503975451,
+            0.01678275503218174,
+            0.024307984858751297,
+            -0.009956029243767262,
+            0.02646869421005249,
+            -0.0024750372394919395,
+            -0.01634502410888672,
+            0.016717560589313507,
+            -0.01307601947337389,
+            0.009047972038388252,
+            -0.013914225623011589,
+            -0.005001298151910305,
+            -0.003718376625329256,
+            -0.018487107008695602,
+            0.02516481839120388,
+            -0.0036718095652759075,
+            -0.012265753000974655,
+            0.015702400356531143,
+            0.023972703143954277,
+            0.026226546615362167,
+            -0.014612731523811817,
+            -0.01563720591366291,
+            0.006700994446873665,
+            0.04414553567767143,
+            0.009392567910254002,
+            0.021178681403398514,
+            -0.01552544441074133,
+            -0.005643922835588455,
+            0.03689970821142197,
+            -0.017667528241872787,
+            0.016400905326008797,
+            -0.0005166028859093785,
+            -0.016242576763033867,
+            0.018291525542736053,
+            -0.003939569927752018,
+            0.006426248699426651,
+            0.03639678284525871,
+            0.0035018399357795715,
+            -0.024792281910777092,
+            -0.030417578294873238,
+            0.028312748298048973,
+            0.026617709547281265,
+            0.004137479700148106,
+            -0.027660811319947243,
+            0.04354947805404663,
+            0.026375560089945793,
+            0.014333329163491726,
+            -0.02065712958574295,
+            -0.02587263658642769,
+            -0.010803548619151115,
+            0.011874590069055557,
+            -0.010654534213244915,
+            0.040867216885089874,
+            0.03283906355500221,
+            -0.00010513960296520963,
+            -0.01047758013010025,
+            0.004083927720785141,
+            0.021551217883825302,
+            0.01868268847465515,
+            0.010281998664140701,
+            0.0032131243497133255,
+            -0.01935325376689434,
+            -0.04202207922935486,
+            0.024922668933868408,
+            -0.023767806589603424,
+            0.00702230678871274,
+            -0.03948883339762688,
+            0.006477472838014364,
+            -0.010216804221272469,
+            0.009434478357434273,
+            -0.034385088831186295,
+            0.045337650924921036,
+            0.016754813492298126,
+            -0.006887262221425772,
+            -0.0130946459248662,
+            -0.011800083331763744,
+            0.030175428837537766,
+            -0.002717185765504837,
+            0.02034047432243824,
+            0.027847077697515488,
+            -0.013411302119493484,
+            -0.01632639765739441,
+            0.0054855952039361,
+            -0.052490346133708954,
+            0.02069438435137272,
+            -0.033211600035429,
+            0.009224927052855492,
+            -0.00036293172161094844,
+            -0.04641800746321678,
+            -0.022147275507450104,
+            0.028461763635277748,
+            0.037272244691848755,
+            -0.012424081563949585,
+            -0.016894515603780746,
+            0.004402911756187677,
+            -0.008880331180989742,
+            0.003743988461792469,
+            -0.005853474605828524,
+            -0.02447562664747238,
+            -0.03188909590244293,
+            -0.0067801582626998425,
+            -0.026207920163869858,
+            0.02844313718378544,
+            -0.011902530677616596,
+            0.0119584109634161,
+            -0.03704872354865074,
+            -0.02712063305079937,
+            0.022445304319262505,
+            0.005662549752742052,
+            -0.020470863208174706,
+            -0.04869047552347183,
+            -0.057966627180576324,
+            0.04004763811826706,
+            0.06381544470787048,
+            0.025630488991737366,
+            -0.00016007413796614856,
+            -0.027660811319947243,
+            0.02641281485557556,
+            -0.014938700012862682,
+            -0.037980061024427414,
+            0.01676412671804428,
+            0.011828023009002209,
+            0.007939676754176617,
+            0.01905522495508194,
+            0.028536271303892136,
+            -0.001995396800339222,
+            -0.004321419633924961,
+            -0.031237157061696053,
+            -0.04362398386001587,
+            -0.029467610642313957,
+            -0.02093653194606304,
+            -0.01437058299779892,
+            -0.03574484586715698,
+            -0.01940913498401642,
+            0.024047208949923515,
+            0.005806907545775175,
+            0.005862787831574678,
+            -0.02546284720301628,
+            0.012032918632030487,
+            -0.010980503633618355,
+            0.0053784907795488834,
+            0.010896682739257812,
+            0.0053272671066224575,
+            0.017797915264964104,
+            -0.0036834513302892447,
+            -0.00501061137765646,
+            -0.023786434903740883,
+            0.0012631304562091827,
+            0.011082950979471207,
+            0.0033947357442229986,
+            -0.0089455246925354,
+            0.024270731955766678,
+            0.01614944264292717,
+            -0.03513016179203987,
+            -0.013644137419760227,
+            0.03809182345867157,
+            0.0029290656093508005,
+            0.030827367678284645,
+            -0.07957372814416885,
+            -0.016969023272395134,
+            0.004898850806057453,
+            -0.03652717173099518,
+            -0.0075019472278654575,
+            0.005606669466942549,
+            0.019800297915935516,
+            -0.010570713318884373,
+            -0.0037649436853826046,
+            -0.027902958914637566,
+            -0.009215613827109337,
+            0.02356291189789772,
+            0.011064324527978897,
+            -0.02166297845542431,
+            0.013811778277158737,
+            0.00014312082203105092,
+            -0.01903659850358963,
+            -0.02933722361922264,
+            0.014584790915250778,
+            -0.03540956228971481,
+            -0.002244530478492379,
+            0.009881521575152874,
+            0.000486043281853199,
+            -0.016205323860049248,
+            0.05748233199119568,
+            0.023767806589603424,
+            0.018496420234441757,
+            -2.3683694962528534e-05,
+            0.0024890073109418154,
+            -0.02097378671169281,
+            -0.011734888888895512,
+            -0.05547063425183296,
+            0.002099008532240987,
+            0.02326488308608532,
+            -0.010328564792871475,
+            -0.023395271971821785,
+            -0.018971404060721397,
+            0.01658717356622219,
+            -0.015124968253076077,
+            -0.024699147790670395,
+            0.03514878824353218,
+            -0.010123670101165771,
+            -0.01340198889374733,
+            0.05174527317285538,
+            0.01275005005300045,
+            -0.04347497224807739,
+            -0.00793502014130354,
+            -0.0371604822576046,
+            0.04068094864487648,
+            -0.024028582498431206,
+            -0.03680657222867012,
+            0.0004083345702383667,
+            0.0027055440004915,
+            0.054017744958400726,
+            -0.005108402110636234,
+            0.04623173922300339,
+            0.023991329595446587,
+            0.0029919310472905636,
+            0.004945417400449514,
+            0.05032963678240776,
+            0.011092264205217361,
+            0.051149215549230576,
+            0.01966990903019905,
+            -0.025630488991737366,
+            0.002516947453841567,
+            -0.03645266219973564,
+            0.0035693622194230556,
+            -0.021551217883825302,
+            -0.043288704007864,
+            -0.004991984460502863,
+            0.019204240292310715,
+            -0.01682000793516636,
+            -0.005117715802043676,
+            0.0472375862300396,
+            -0.046678781509399414,
+            -0.03624776750802994,
+            0.01909247785806656,
+            -0.013001511804759502,
+            -0.03159106522798538,
+            0.0020047102589160204,
+            -0.0069245160557329655,
+            -0.031795963644981384,
+            -0.032168496400117874,
+            -5.2824463637080044e-05,
+            0.014230881817638874,
+            -0.029858773574233055,
+            -0.0166058000177145,
+            -0.04515138268470764,
+            0.01451959740370512,
+            -0.036992840468883514,
+            0.03580072522163391,
+            -0.03526054695248604,
+            0.0003108348755631596,
+            0.027847077697515488,
+            -0.009145762771368027,
+            -0.03125578537583351,
+            0.027958840131759644,
+            0.0016065621748566628,
+            -0.01836603321135044,
+            0.010095730423927307,
+            -0.0018673375016078353,
+            0.03453410416841507,
+            -0.030138175934553146,
+            0.0003000662545673549,
+            0.034422341734170914,
+            -0.03295082226395607,
+            -0.02393544837832451,
+            0.01079423539340496,
+            0.05971754714846611,
+            0.008088691160082817,
+            -0.027008872479200363,
+            -0.012358887121081352,
+            0.014808312989771366,
+            -0.009499671868979931,
+            0.009304090403020382,
+            0.020843397825956345,
+            -0.03749576583504677,
+            -0.022426677867770195,
+            -0.007134067825973034,
+            0.00888498779386282,
+            -0.007981587201356888,
+            -0.0022526795510202646,
+            0.036266397684812546,
+            0.017173917964100838,
+            6.915930134709924e-05,
+            0.03369589522480965,
+            -0.020899279043078423,
+            -0.019297374412417412,
+            -0.004482075572013855,
+            0.00969525333493948,
+            -0.017304304987192154,
+            0.00484762666746974,
+            -0.006649770773947239,
+            -0.003913958091288805,
+            -0.04030841216444969,
+            -0.03494389355182648,
+            -0.03477625176310539,
+            0.007203917950391769,
+            0.07160145044326782,
+            -0.009779074229300022,
+            -0.0028918120078742504,
+            0.018058691173791885,
+            0.01743469201028347,
+            -0.001707845483906567,
+            -0.021532591432332993,
+            -0.03198223188519478,
+            -0.028331374749541283,
+            0.008670778945088387,
+            0.00515962578356266,
+            -0.007688215002417564,
+            0.04485335573554039,
+            0.021867873147130013,
+            0.004153778310865164,
+            -0.015395057387650013,
+            0.024736400693655014,
+            -0.024959923699498177,
+            0.03520466759800911,
+            -0.00905262865126133,
+            0.0023958731908351183,
+            -0.0019977251067757607,
+            0.000827728770673275,
+            -0.0044378372840583324,
+            0.03593111410737038,
+            -0.00347389979287982,
+            0.0261892918497324,
+            0.014603417366743088,
+            -0.005308640189468861,
+            0.03168420121073723,
+            -0.0626605823636055,
+            -0.0012316977372393012,
+            0.02004244551062584,
+            0.0047498359344899654,
+            0.001548353466205299,
+            -0.004866253584623337,
+            -0.004782433155924082,
+            0.01258240919560194,
+            0.02354428544640541,
+            -0.003101363545283675,
+            -0.034999772906303406,
+            -0.02067575789988041,
+            0.048876743763685226,
+            0.003289960091933608,
+            0.0009470568038523197,
+            0.02069438435137272,
+            0.009229583665728569,
+            0.03464586287736893,
+            -0.01519016269594431,
+            0.026040278375148773,
+            0.01708078384399414,
+            0.009383254684507847,
+            0.008572988212108612,
+            0.015721026808023453,
+            0.011399607174098492,
+            -0.000368752604117617,
+            -0.026096157729625702,
+            0.018990030512213707,
+            0.01870131492614746,
+            0.0015646519605070353,
+            -0.04675328731536865,
+            0.014975953847169876,
+            0.014510284177958965,
+            -0.0008236541762016714,
+            0.012386827729642391,
+            -0.01875719614326954,
+            0.003408706048503518,
+            -0.011725575663149357,
+            0.016121502965688705,
+            -0.00448673265054822,
+            -0.015003894455730915,
+            -0.00463574705645442,
+            -0.0009453105158172548,
+            -0.017034215852618217,
+            -0.010421698912978172,
+            -0.003874376183375716,
+            0.01773272082209587,
+            0.039190806448459625,
+            -0.045039623975753784,
+            -0.010291311889886856,
+            -0.015553385019302368,
+            -0.011986351571977139,
+            -0.0326155424118042,
+            -0.001544860890135169,
+            -0.00967662688344717,
+            0.006808098405599594,
+            -0.005546132102608681,
+            -0.013523062691092491,
+            -0.04172405228018761,
+            -0.011408920399844646,
+            -0.027344154193997383,
+            -0.013737271539866924,
+            0.02941173128783703,
+            0.031181277707219124,
+            0.0015390400076285005,
+            0.009373941458761692,
+            0.010309938341379166,
+            0.009639373049139977,
+            -0.004787089768797159,
+            -0.015711713582277298,
+            -0.028461763635277748,
+            0.019446387887001038,
+            0.015097028575837612,
+            0.032168496400117874,
+            -0.01337404828518629,
+            0.0002654320269357413,
+            0.019949311390519142,
+            -0.002558857901021838,
+            -0.010729041881859303,
+            -0.0014517268864437938,
+            0.01098981685936451,
+            -0.02028459496796131,
+            -0.004447150509804487,
+            0.03229888528585434,
+            0.0009418179979547858,
+            0.01273142360150814,
+            0.03349100053310394,
+            0.004018733743578196,
+            0.02415897138416767,
+            0.005392461083829403,
+            0.018533675000071526,
+            0.029635252431035042,
+            -0.017816541716456413,
+            0.009220270439982414,
+            0.007990900427103043,
+            0.030175428837537766,
+            0.009169046767055988,
+            -0.014575477689504623,
+            -0.0339939258992672,
+            -0.02004244551062584,
+            0.0001397010637447238,
+            0.03136754408478737,
+            0.0005989682977087796,
+            -0.0033248853869736195,
+            -0.011297158896923065,
+            -0.01771409437060356,
+            0.014612731523811817,
+            0.001167668029665947,
+            0.033360615372657776,
+            0.031162651255726814,
+            -0.01486419327557087,
+            -0.02060125023126602,
+            -0.028275495395064354,
+            0.008647495880723,
+            0.0612821988761425,
+            -0.014240195043385029,
+            -0.011949097737669945,
+            0.029430357739329338,
+            0.015069087967276573,
+            0.020806144922971725,
+            0.003799868980422616,
+            -0.0026450068689882755,
+            0.009909462183713913,
+            0.01965128257870674,
+            0.026617709547281265,
+            -0.0229296013712883,
+            -0.029150955379009247,
+            -0.013653450645506382,
+            -0.012619663029909134,
+            -0.006966426502913237,
+            0.009099195711314678,
+            0.004433180205523968,
+            -0.03811044991016388,
+            0.009117823094129562,
+            0.00342966103926301,
+            -0.012098112143576145,
+            -0.014761745929718018,
+            -0.013588257133960724,
+            0.015804847702383995,
+            -0.01342061534523964,
+            0.02937447652220726,
+            -0.013904912397265434,
+            -0.03028719127178192,
+            -0.013634824194014072,
+            -0.01907385140657425,
+            -0.019763043150305748,
+            -0.034105684608221054,
+            -0.019763043150305748,
+            0.015450937673449516,
+            0.020806144922971725,
+            -0.006165473721921444,
+            -0.017425378784537315,
+            0.0009272657916881144,
+            -0.003071095095947385,
+            0.0020908592268824577,
+            0.028890179470181465,
+            0.009741820394992828,
+            -0.025015803053975105,
+            0.03509290888905525,
+            -0.018198391422629356,
+            0.003418019274249673,
+            -0.011492740362882614,
+            0.022482557222247124,
+            -0.014510284177958965,
+            0.030268562957644463,
+            -0.008018841035664082,
+            -0.006808098405599594,
+            -0.016000429168343544,
+            -0.018179764971137047,
+            0.005909354891628027,
+            0.04008489102125168,
+            -0.009639373049139977,
+            0.008675435557961464,
+            -0.019893432036042213,
+            0.013290227390825748,
+            0.0213463231921196,
+            -0.0006047891802154481,
+            -0.003813839051872492,
+            -0.01159518864005804,
+            -0.024084463715553284,
+            -0.005946608725935221,
+            0.0006787143065594137,
+            0.008852390572428703,
+            -0.0050478652119636536,
+            0.026394188404083252,
+            0.01940913498401642,
+            -0.0019651283510029316,
+            0.01808663085103035,
+            -0.008000214584171772,
+            -0.004433180205523968,
+            0.028610777109861374,
+            -0.002068739850074053,
+            0.0066683972254395485,
+            0.01785379648208618,
+            -0.0118093965575099,
+            -0.022780586034059525,
+            0.0059559219516813755,
+            0.01998656615614891,
+            0.02060125023126602,
+            0.029486237093806267,
+            -0.010570713318884373,
+            0.011129518039524555,
+            -0.014622044749557972,
+            -0.009946716018021107,
+            0.0041235098615288734,
+            0.0004292897356208414,
+            0.01940913498401642,
+            0.010850115679204464,
+            0.022389423102140427,
+            0.0027590959798544645,
+            -0.03609875589609146,
+            -0.002391216577962041,
+            -0.029206834733486176,
+            -0.0017311290139332414,
+            0.0025518727488815784,
+            0.003222437808290124,
+            -0.004312105942517519,
+            0.03416156768798828,
+            -0.011343725956976414,
+            -0.02516481839120388,
+            0.022724706679582596,
+            -0.0038976597134023905,
+            -0.0015809503383934498,
+            0.04656702280044556,
+            -0.030901875346899033,
+            -0.013020139187574387,
+            0.012647602707147598,
+            -0.0008125944877974689,
+            0.012181933037936687,
+            -0.0027102006133645773,
+            0.006118906661868095,
+            0.0175930205732584,
+            0.008503138087689877,
+            0.005401774309575558,
+            0.026375560089945793,
+            0.028331374749541283,
+            0.0010035192826762795,
+            0.009751134552061558,
+            -0.00637968210503459,
+            0.015171535313129425,
+            0.01613081619143486,
+            0.0007421618793159723,
+            -0.0006734755588695407,
+            0.04492786154150963,
+            -0.04392201453447342,
+            0.025891263037919998,
+            -0.024084463715553284,
+            0.018515046685934067,
+            0.005369177553802729,
+            -0.015246042981743813,
+            0.010878056287765503,
+            -0.031814590096473694,
+            -0.011446173302829266,
+            -0.024140343070030212,
+            -0.0009394896333105862,
+            -0.018747882917523384,
+            -0.025313831865787506,
+            0.009993283078074455,
+            0.0009202807559631765,
+            -0.006468159146606922,
+            0.0012631304562091827,
+            -0.008763913065195084,
+            -0.01996793784201145,
+            -0.0021467397455126047,
+            0.010281998664140701,
+            0.029635252431035042,
+            0.01451959740370512,
+            0.007124754134565592,
+            0.008833764120936394,
+            -0.04481610283255577,
+            -0.0322243794798851,
+            0.026729470118880272,
+            0.0029220804572105408,
+            -0.039228059351444244,
+            0.010729041881859303,
+            -0.006836038548499346,
+            -0.01725773885846138,
+            0.03937707468867302,
+            -0.026245173066854477,
+            -0.005704460199922323,
+            0.01733224466443062,
+            0.02393544837832451,
+            -0.008596272207796574,
+            0.0024307984858751297,
+            0.01483625266700983,
+            0.009881521575152874,
+            -0.003434317884966731,
+            0.013271600939333439,
+            -0.013523062691092491,
+            -0.012861811555922031,
+            0.023730553686618805,
+            0.029523491859436035,
+            0.019483640789985657,
+            -0.0024820221588015556,
+            0.0067801582626998425,
+            0.0035903172101825476,
+            0.026394188404083252,
+            0.03619188815355301,
+            -0.02192375436425209,
+            0.03127441182732582,
+            -0.006063025910407305,
+            -0.0355772040784359,
+            0.029430357739329338,
+            -0.0018033079104498029,
+            0.02415897138416767,
+            0.021830620244145393,
+            0.023302137851715088,
+            -0.009061941877007484,
+            -0.015450937673449516,
+            -0.01047758013010025,
+            0.03812907636165619,
+            0.014705865643918514,
+            -0.009583492763340473,
+            -0.015646519139409065,
+            -0.020470863208174706,
+            -0.008512451313436031,
+            0.01909247785806656,
+            -0.015981800854206085,
+            0.026859857141971588,
+            0.02909507416188717,
+            0.016736187040805817,
+            -0.0013271600473672152,
+            0.015469564124941826,
+            -0.02190512605011463,
+            -0.013085332699120045,
+            -0.03423607349395752,
+            -0.006426248699426651,
+            0.0044704340398311615,
+            0.010552086867392063,
+            0.0005579311400651932,
+            -0.006700994446873665,
+            0.02970976009964943,
+            0.01320640742778778,
+            -0.008037467487156391,
+            -0.010831489227712154,
+            0.03168420121073723,
+            -0.01613081619143486,
+            0.00866612233221531,
+            -0.03544681519269943,
+            -0.0072644553147256374,
+            -0.01790967583656311,
+            -0.018012123182415962,
+            0.04753561317920685,
+            -0.0235256589949131,
+            -0.0005762669024989009,
+            0.007483320310711861,
+            -0.008060751482844353,
+            -0.01030062511563301,
+            -0.0185802411288023,
+            0.014901447109878063,
+            0.0013876971788704395,
+            0.0027730660513043404,
+            -0.02522069774568081,
+            -0.0026822604704648256,
+            0.0009488030336797237,
+            -0.049584563821554184,
+            0.004912820644676685,
+            -0.03561445698142052,
+            -0.027251020073890686,
+            -0.00028114838642068207,
+            -0.056327469646930695,
+            0.02132769487798214,
+            -0.019893432036042213,
+            -0.0009668477578088641,
+            -0.025239326059818268,
+            0.012880438007414341,
+            0.01836603321135044,
+            -0.04712582379579544,
+            -0.012275067158043385,
+            -0.011902530677616596,
+            0.011557934805750847,
+            -0.010775608941912651,
+            -0.02553735487163067,
+            0.02484816312789917,
+            -0.038855522871017456,
+            0.007175977807492018,
+            -0.00043103599455207586,
+            0.017211170867085457,
+            0.007613707799464464,
+            0.02259431779384613,
+            0.00892689824104309,
+            0.012275067158043385,
+            -0.011651068925857544,
+            -0.0036881081759929657,
+            -0.03513016179203987,
+            0.004586851689964533,
+            0.008982778526842594,
+            0.02032184787094593,
+            -0.021048294380307198,
+            0.02199826017022133,
+            0.03231751173734665,
+            0.022203154861927032,
+            0.02747454307973385,
+            0.0038347942754626274,
+            0.022519811987876892,
+            0.008107318542897701,
+            -0.03233613818883896,
+            0.017695467919111252,
+            -0.006333115044981241,
+            -0.005192223004996777,
+            -0.020135579630732536,
+            -0.018608180806040764,
+            0.04556117206811905,
+            0.025276578962802887,
+            0.020116953179240227,
+            0.03296945244073868,
+            -0.03226163238286972,
+            -0.014780372381210327,
+            -0.016056308522820473,
+            -0.007231858558952808,
+            -0.013560316525399685,
+            -0.02747454307973385,
+            0.0016461441991850734,
+            -0.011893217451870441,
+            0.007883796468377113,
+            0.005290013737976551,
+            0.02000519260764122,
+            0.012526528909802437,
+            0.042729899287223816,
+            0.013979420065879822,
+            0.0056765200570225716,
+            0.013178466819226742,
+            0.0038813611026853323,
+            -0.009429821744561195,
+            0.03349100053310394,
+            -0.004423866979777813,
+            0.013495123013854027,
+            -0.01972579024732113,
+            0.03585660457611084,
+            0.0030361698009073734,
+            0.01081286184489727,
+            0.01468723826110363,
+            0.00033499152050353587,
+            -0.0336027629673481,
+            -0.005057178437709808,
+            0.027008872479200363,
+            0.036601677536964417,
+            0.012293693609535694,
+            -0.00782791618257761,
+            -0.02455013431608677,
+            0.018077317625284195,
+            -0.006226010620594025,
+            -0.0016985321417450905,
+            0.034105684608221054,
+            0.025090310722589493,
+            0.03103226236999035,
+            -0.014184314757585526,
+            0.0037719286046922207,
+            -0.02000519260764122,
+            0.008018841035664082,
+            0.019781669601798058,
+            0.036955587565898895,
+            -0.01081286184489727,
+            -0.012489275075495243,
+            -0.0006239980575628579,
+            0.03391941636800766,
+            -0.039898622781038284,
+            0.01355100329965353,
+            0.026524575427174568,
+            -0.02553735487163067,
+            0.019856177270412445,
+            -0.007311022374778986,
+            -0.034068431705236435,
+            0.010393759235739708,
+            -0.013774524442851543,
+            0.005350550636649132,
+            0.03255965933203697,
+            0.0006839531124569476,
+            -0.032131243497133255,
+            0.00471025425940752,
+            -0.024661894887685776,
+            -0.019241493195295334,
+            -0.007227201480418444,
+            -0.0002667417284101248,
+            0.03233613818883896,
+            -0.02844313718378544,
+            0.028461763635277748,
+            0.029840147122740746,
+            0.027549048885703087,
+            -0.008805823512375355,
+            0.010356505401432514,
+            -0.0261892918497324,
+            0.037570271641016006,
+            0.011660382151603699,
+            -0.011651068925857544,
+            0.013942166231572628,
+            0.009993283078074455,
+            -0.013513749465346336,
+            0.00011066943261539564,
+            -0.01421225443482399,
+            -0.01320640742778778,
+            -0.02423347719013691,
+            0.002894140314310789,
+            -0.03065972588956356,
+            -0.026897111907601357,
+            0.014500970020890236,
+            0.020824771374464035,
+            -0.004544941242784262,
+            0.00868940632790327,
+            0.00935997162014246,
+            -0.006076996214687824,
+            0.021066920831799507,
+            -0.00177769607398659,
+            0.01084080245345831,
+            -0.012368200346827507,
+            -0.001345786964520812,
+            -0.003157244063913822,
+            0.01854298822581768,
+            -0.07778555154800415,
+            -0.02484816312789917,
+            0.05010611563920975,
+            0.032373394817113876,
+            -0.014789685606956482,
+            -0.002556529361754656,
+            0.010822176001966,
+            -0.0005678266170434654,
+            0.018533675000071526,
+            0.004435508977621794,
+            0.015218102373182774,
+            -0.010961877182126045,
+            -0.011353040114045143,
+            -0.01113883126527071,
+            0.028238240629434586,
+            -0.002343485364690423,
+            -0.01275936421006918,
+            -0.02417759783565998,
+            0.02006107196211815,
+            0.0112692192196846,
+            -0.00757179781794548,
+            -0.0006111921393312514,
+            0.0031619006767868996,
+            -0.01307601947337389,
+            0.00035681980079971254,
+            -0.009145762771368027,
+            -0.01965128257870674,
+            0.019064538180828094,
+            0.025276578962802887,
+            0.009844268672168255,
+            0.01725773885846138,
+            0.029281342402100563,
+            0.008372750133275986,
+            0.01486419327557087,
+            0.005746370181441307,
+            0.007315678987652063,
+            -0.009844268672168255,
+            -0.01047758013010025,
+            0.008274959400296211,
+            0.0032131243497133255,
+            -0.014780372381210327,
+            -0.004163091536611319,
+            0.019614029675722122,
+            0.011399607174098492,
+            0.00034576014149934053,
+            0.009215613827109337,
+            0.022110020741820335,
+            0.013234347105026245,
+            -0.01804937794804573,
+            0.011865276843309402,
+            0.010542773641645908,
+            0.03581935167312622,
+            -0.00903400219976902,
+            -0.02877841889858246,
+            -0.006938486360013485,
+            0.021551217883825302,
+            0.004018733743578196,
+            -0.007418126333504915,
+            0.015683772042393684,
+            -0.01616806909441948,
+            0.07208574563264847,
+            0.0012747722212225199,
+            0.015786219388246536,
+            -0.027027498930692673,
+            -0.014789685606956482,
+            0.007497290149331093,
+            -0.014780372381210327,
+            0.021458083763718605,
+            0.002114142756909132,
+            0.026356933638453484,
+            0.014053926803171635,
+            0.014016672968864441,
+            0.0042399270460009575,
+            0.010542773641645908,
+            -0.0030734234023839235,
+            0.014631357975304127,
+            0.0019709491170942783,
+            0.007078187074512243,
+            -0.00109956378582865,
+            0.012200559489428997,
+            0.004558911547064781,
+            -0.0010244745062664151,
+            -0.0025239326059818268,
+            0.010011909529566765,
+            0.00854039192199707,
+            -0.007143381051719189,
+            0.004463449120521545,
+            -0.003844107501208782,
+            -0.030734233558177948,
+            -0.007418126333504915,
+            0.016037682071328163,
+            0.008503138087689877,
+            0.032392021268606186,
+            -0.0036485260352492332,
+            -0.018570927903056145,
+            0.02840588241815567,
+            0.0003201482759322971,
+            0.010216804221272469,
+            0.0182822123169899,
+            0.005313297268003225,
+            -0.027306901291012764,
+            0.03164694830775261,
+            -0.01905522495508194,
+            0.016652366146445274,
+            -0.023413898423314095,
+            -0.014798998832702637,
+            0.004258553963154554,
+            0.0030478115659207106,
+            -0.029448984190821648,
+            -0.005266730207949877,
+            0.01634502410888672,
+            0.024307984858751297,
+            0.00815388560295105,
+            0.009490358643233776,
+            0.010226117447018623,
+            0.004009420517832041,
+            0.013765211217105389,
+            0.01577690616250038,
+            -0.012796617113053799,
+            -0.007921050302684307,
+            0.0012421752326190472,
+            -0.015646519139409065,
+            0.013904912397265434,
+            -0.00281730480492115,
+            -0.0008748779073357582,
+            0.013886285945773125,
+            -0.02265019901096821,
+            -0.010375131852924824,
+            0.010132983326911926,
+            -0.009834954515099525,
+            -0.004193360451608896,
+            0.038222212344408035,
+            -0.006510069593787193,
+            0.008801166899502277,
+            -0.01565583236515522,
+            0.00038184956065379083,
+            0.012954945676028728,
+            -0.004202673677355051,
+            -0.026580454781651497,
+            -0.04381025210022926,
+            0.004917477257549763,
+            0.02905782125890255,
+            0.013001511804759502,
+            0.04056918993592262,
+            0.0019290389027446508,
+            -0.01996793784201145,
+            -0.025704994797706604,
+            0.02838725596666336,
+            -0.00868940632790327,
+            0.0017846811097115278,
+            -0.025555981323122978,
+            0.013560316525399685,
+            0.00861955527216196,
+            0.012833870947360992,
+            -0.007506603840738535,
+            -0.01501320768147707,
+            0.014277448877692223,
+            0.014072554185986519,
+            -0.03220575302839279,
+            0.007539200596511364,
+            0.004200345370918512,
+            0.0038557492662221193,
+            0.011520680971443653,
+            -0.01844054087996483,
+            -0.009974655695259571,
+            0.0004563567927107215,
+            0.0036881081759929657,
+            -0.012787303887307644,
+            0.0006915202829986811,
+            0.015273982658982277,
+            -0.02130906842648983,
+            -0.016540605574846268,
+            -0.01785379648208618,
+            -0.007674245163798332,
+            -0.003962853457778692,
+            0.00456123985350132,
+            0.01455685030668974,
+            -0.013802465051412582,
+            -0.008475197479128838,
+            -0.005136342253535986,
+            -0.004558911547064781,
+            0.02747454307973385,
+            0.006808098405599594,
+            -0.02613341249525547,
+            0.0068127550184726715,
+            0.012200559489428997,
+            -0.0011408920399844646,
+            -0.011082950979471207,
+            0.024065837264060974,
+            0.0008603257010690868,
+            0.003939569927752018,
+            -0.000613520503975451,
+            0.007902423851191998,
+            0.02229628898203373,
+            -0.00866612233221531,
+            0.007851200178265572,
+            0.014435776509344578,
+            0.013737271539866924,
+            0.00621669739484787,
+            -0.014649984426796436,
+            -0.012293693609535694,
+            -0.011362353339791298,
+            0.017071470618247986,
+            0.013485808856785297,
+            -0.012768677435815334,
+            -0.022110020741820335,
+            -0.012628976255655289,
+            -0.0011630113003775477,
+            0.0070688738487660885,
+            0.00862886942923069,
+            -0.008754599839448929,
+            0.023153122514486313,
+            0.020731637254357338,
+            0.004405240062624216,
+            -0.0046147918328642845,
+            -0.013737271539866924,
+            0.030119549483060837,
+            0.0014261150499805808,
+            0.0351860411465168,
+            0.013485808856785297,
+            0.005709116812795401,
+            0.012722110375761986,
+            -0.005541475489735603,
+            -0.018971404060721397,
+            0.029523491859436035,
+            0.0035181385464966297,
+            -0.010067789815366268,
+            0.06199001893401146,
+            0.0020198444835841656,
+            -0.0015972488326951861,
+            -0.008386720903217793,
+            -0.008866360411047935,
+            0.014426463283598423,
+            -0.004414553754031658,
+            0.01834740675985813,
+            -0.0015169207472354174,
+            0.01563720591366291,
+            -0.00870803277939558,
+            -0.006044399458914995,
+            0.0037020782474428415,
+            0.004223628900945187,
+            -0.007054903544485569,
+            0.014808312989771366,
+            0.02201688662171364,
+            0.006901232525706291,
+            -0.036974214017391205,
+            -0.02741866186261177,
+            -0.014249508269131184,
+            -0.037868302315473557,
+            0.014733805321156979,
+            0.008423973806202412,
+            9.793626668397337e-05,
+            0.014277448877692223,
+            -0.035968367010354996,
+            0.012247126549482346,
+            0.0122564397752285,
+            -0.0027614242862910032,
+            -0.0062120407819747925,
+            -0.0004831328405998647,
+            -0.02028459496796131,
+            -0.0011024741688743234,
+            0.01875719614326954,
+            0.013616196811199188,
+            -0.0007695200038142502,
+            0.02132769487798214,
+            -0.00475449301302433,
+            0.014612731523811817,
+            -0.006654427386820316,
+            0.025630488991737366,
+            -0.013830405659973621,
+            -0.017043529078364372,
+            -0.009788387455046177,
+            0.01468723826110363,
+            0.021532591432332993,
+            0.009467075578868389,
+            0.03034307062625885,
+            -0.007981587201356888,
+            0.009411195293068886,
+            -0.013849032111465931,
+            -0.014994580298662186,
+            0.00266130524687469,
+            0.024103090167045593,
+            0.004801060073077679,
+            0.015422997064888477,
+            0.013830405659973621,
+            -0.011772142723202705,
+            0.007939676754176617,
+            -0.007371559273451567,
+            0.0031130053102970123,
+            -0.008731316775083542,
+            -0.005969892255961895,
+            0.0038487643469125032,
+            -0.0006664904649369419,
+            0.012470648624002934,
+            0.002131605288013816,
+            0.018393972888588905,
+            -0.007595081347972155,
+            -0.005578729324042797,
+            -0.021811991930007935,
+            -0.001249160384759307,
+            0.0257981289178133,
+            -0.028554897755384445,
+            -0.02425210364162922,
+            0.016037682071328163,
+            0.0023073959164321423,
+            -0.0003850510693155229,
+            0.01243339478969574,
+            -0.04101623222231865,
+            -0.033360615372657776,
+            -0.0235256589949131,
+            -0.014854880049824715,
+            -0.0001042664734995924,
+            0.015078401193022728,
+            -0.01611218973994255,
+            -0.010486893355846405,
+            0.0219423808157444,
+            -0.015199475921690464,
+            0.024047208949923515,
+            -0.014286762103438377,
+            -0.0028359314892441034,
+            -0.004363330081105232,
+            0.006030429154634476,
+            0.00011692687985487282,
+            0.019893432036042213,
+            -0.019129732623696327,
+            -0.0030757517088204622,
+            -0.02071301080286503,
+            -0.001526234089396894,
+            0.00614219019189477,
+            0.001925546326674521,
+            -0.024587387219071388,
+            0.0016915469896048307,
+            0.020824771374464035,
+            -0.016615113243460655,
+            0.02715788595378399,
+            -0.01909247785806656,
+            0.020862026140093803,
+            -0.009611433371901512,
+            0.011688321828842163,
+            -0.008228392340242863,
+            -0.024307984858751297,
+            0.06638594716787338,
+            -0.0069245160557329655,
+            -0.00215488881804049,
+            -0.016456784680485725,
+            0.0016473083524033427,
+            -0.021066920831799507,
+            0.02581675723195076,
+            0.006896575912833214,
+            0.019427761435508728,
+            -0.014091180637478828,
+            0.026356933638453484,
+            -0.002856886712834239,
+            0.0064402190037071705,
+            0.02741866186261177,
+            -0.0018719942308962345,
+            -0.016792068257927895,
+            0.005313297268003225,
+            0.02291097491979599,
+            -0.020172834396362305,
+            -0.01747194677591324,
+            -0.021197307854890823,
+            -0.003471571486443281,
+            -0.004633418750017881,
+            -0.011259905993938446,
+            -0.002584469737485051,
+            0.002989602740854025,
+            -0.03349100053310394,
+            0.024680521339178085,
+            0.0036275710444897413,
+            -0.005108402110636234,
+            0.004400583449751139,
+            -0.026710843667387962,
+            0.000846355629619211,
+            0.00281730480492115,
+            -0.012694169767200947,
+            0.011716262437403202,
+            0.01646609790623188,
+            0.008447257801890373,
+            0.002136262133717537,
+            0.011846650391817093,
+            0.017015589401125908,
+            -0.017760662361979485,
+            0.008177168667316437,
+            -0.011362353339791298,
+            0.018654748797416687,
+            -0.010701101273298264,
+            -0.01371864415705204,
+            -0.01569308526813984,
+            0.001958143198862672,
+            -0.02060125023126602,
+            -0.004074614495038986,
+            0.00590469827875495,
+            -0.025071684271097183,
+            -0.0032573631033301353,
+            0.019837550818920135,
+            0.005057178437709808,
+            0.020862026140093803,
+            0.00023560001864098012,
+            -0.00982564128935337,
+            -0.031497932970523834,
+            -0.0025448875967413187,
+            -0.021495336666703224,
+            -0.012265753000974655,
+            0.016298457980155945,
+            0.007604394573718309,
+            -0.015497504733502865,
+            -0.005723086651414633,
+            0.001080936985090375,
+            -0.007734782062470913,
+            -0.035968367010354996,
+            0.03721636161208153,
+            0.006267921067774296,
+            0.014296075329184532,
+            -0.03390078991651535,
+            0.005313297268003225,
+            -0.003185184206813574,
+            -0.0005832519382238388,
+            -0.011511367745697498,
+            0.0032177811954170465,
+            -0.015627892687916756,
+            -0.00793502014130354,
+            -0.01939050666987896,
+            -0.009443791583180428,
+            -0.035670336335897446,
+            -0.021402202546596527,
+            -0.00012332983897067606,
+            0.012666230089962482,
+            0.008228392340242863,
+            0.001559995231218636,
+            0.02030322141945362,
+            -0.025555981323122978,
+            0.019465014338493347,
+            0.0015646519605070353,
+            0.02546284720301628,
+            0.006961769424378872,
+            -0.0007718483684584498,
+            0.008582302369177341,
+            -0.029933281242847443,
+            -0.013727957382798195,
+            -0.01755576767027378,
+            -0.0169038288295269,
+            -0.021551217883825302,
+            0.0064728157594799995,
+            -0.004742851015180349,
+            -0.017695467919111252,
+            -0.012638289481401443,
+            -0.004682314116507769,
+            0.004735866095870733,
+            -0.02287372015416622,
+            0.02255706489086151,
+            0.011278532445430756,
+            0.037942808121442795,
+            -0.01587004028260708,
+            0.02032184787094593,
+            0.00797693058848381,
+            -0.006519382819533348,
+            -0.01325297448784113,
+            -0.027213767170906067,
+            0.002933722222223878,
+            -0.0015553385019302368,
+            -0.014333329163491726,
+            0.004628761671483517,
+            0.012116738595068455,
+            0.026207920163869858,
+            0.027604930102825165,
+            -0.0209551602602005,
+            -0.008717346005141735,
+            -0.011371666565537453,
+            -0.006375025026500225,
+            0.0041048829443752766,
+            -0.019856177270412445,
+            -0.0012002650182694197,
+            -0.036340903490781784,
+            -0.00281730480492115,
+            -0.006351741496473551,
+            0.0003335362998768687,
+            0.006719620898365974,
+            -0.004540284629911184,
+            -0.0007153858314268291,
+            -0.00262405164539814,
+            -0.024680521339178085,
+            0.024028582498431206,
+            -0.021066920831799507,
+            -0.0072644553147256374,
+            -0.018496420234441757,
+            0.028629405423998833,
+            0.0025006490759551525,
+            0.005997832398861647,
+            -0.029914654791355133,
+            -0.011399607174098492,
+            -0.012964258901774883,
+            0.0038301374297589064,
+            0.00043481955071911216,
+            -0.00815388560295105,
+            0.03198223188519478,
+            0.008917584083974361,
+            0.004824343603104353,
+            0.012656915932893753,
+            0.020265966653823853,
+            -0.02004244551062584,
+            -0.019539522007107735,
+            -0.034422341734170914,
+            0.02227766253054142,
+            -0.017779288813471794,
+            -0.006370368413627148,
+            -0.002556529361754656,
+            0.0033318703062832355,
+            -0.007348275743424892,
+            -0.013625510036945343,
+            0.010254058055579662,
+            0.007990900427103043,
+            -0.011259905993938446,
+            -0.002887155395001173,
+            -0.009900148957967758,
+            0.002933722222223878,
+            -0.00788845308125019,
+            0.006458845920860767,
+            -0.009476388804614544,
+            0.007092157378792763,
+            0.0063563985750079155,
+            0.003799868980422616,
+            -0.011474113911390305,
+            0.028908805921673775,
+            0.010859428904950619,
+            0.01871994324028492,
+            0.010011909529566765,
+            -0.017462633550167084,
+            0.013262287713587284,
+            -0.006719620898365974,
+            0.026282425969839096,
+            -0.028461763635277748,
+            -0.021048294380307198,
+            -0.012628976255655289,
+            0.008866360411047935,
+            -0.006314488127827644,
+            -0.0033435120712965727,
+            -0.0006222518277354538,
+            0.0019756059627979994,
+            -0.01737881265580654,
+            -0.0012736080680042505,
+            0.02166297845542431,
+            0.006766187958419323,
+            0.007045590318739414,
+            -0.00759042426943779,
+            0.052788376808166504,
+            0.005886071361601353,
+            0.0219423808157444,
+            0.0003335362998768687,
+            0.01998656615614891,
+            -0.015739653259515762,
+            0.009900148957967758,
+            -0.03231751173734665,
+            -0.00177420349791646,
+            0.007520573679357767,
+            -0.007786005735397339,
+            0.0009982804767787457,
+            -0.003443631110712886,
+            0.009769761003553867,
+            0.002465723780915141,
+            -0.020489489659667015,
+            -0.027027498930692673,
+            -0.012954945676028728,
+            0.01650335267186165,
+            0.009038658812642097,
+            -0.011576561257243156,
+            -0.00351115339435637,
+            -0.00550422165542841,
+            0.0026636335533112288,
+            0.003341183764860034,
+            0.028834300115704536,
+            -0.038892775774002075,
+            0.0011263397755101323,
+            -0.00023865597904659808,
+            0.030920501798391342,
+            0.0037416601553559303,
+            -0.016046995297074318,
+            -0.00021944708714727312,
+            0.012470648624002934,
+            0.008526421152055264,
+            0.007236515171825886,
+            -0.0005649161757901311,
+            0.004423866979777813,
+            0.007031620014458895,
+            -0.009499671868979931,
+            0.014808312989771366,
+            -0.015283295884728432,
+            -0.009318061172962189,
+            -0.004035032354295254,
+            -0.007967617362737656,
+            0.003930256702005863,
+            0.005448341369628906,
+            0.0031293039210140705,
+            0.0007840722100809216,
+            -0.0006030429503880441,
+            -0.0012573095737025142,
+            -0.006468159146606922,
+            0.018328780308365822,
+            0.01998656615614891,
+            -0.0185802411288023,
+            0.0016158756334334612,
+            0.025276578962802887,
+            -0.025053057819604874,
+            0.014147060923278332,
+            0.023041361942887306,
+            0.003681123023852706,
+            -0.002554201055318117,
+            0.005648579448461533,
+            0.0007252813666127622,
+            0.010552086867392063,
+            0.007837229408323765,
+            0.01785379648208618,
+            0.010505519807338715,
+            -0.014482343569397926,
+            0.01771409437060356,
+            -0.006198070477694273,
+            -0.0011414741165935993,
+            -0.009243553504347801,
+            0.010784922167658806,
+            -0.002402858342975378,
+            0.004978014621883631,
+            -0.00016021965711843222,
+            -0.00423527043312788,
+            -0.007217888254672289,
+            0.013197094202041626,
+            0.00355539214797318,
+            0.017173917964100838,
+            -0.008023497648537159,
+            -0.0038720478769391775,
+            -0.02199826017022133,
+            -0.020265966653823853,
+            0.0024447685573250055,
+            0.019800297915935516,
+            0.02326488308608532,
+            -0.01666167937219143,
+            0.0074926335364580154,
+            0.008647495880723,
+            0.02265019901096821,
+            -0.008801166899502277,
+            -0.014147060923278332,
+            0.0034133626613765955,
+            -0.011073637753725052,
+            0.004714910872280598,
+            -0.012694169767200947,
+            -0.015301923267543316,
+            0.016689620912075043,
+            0.0212531890720129,
+            0.002812647959217429,
+            0.013709330931305885,
+            0.00645418930798769,
+            -0.004260882269591093,
+            0.019912058487534523,
+            -0.01678275503218174,
+            -0.017993496730923653,
+            -0.005369177553802729,
+            -0.0030920500867068768,
+            -0.00935997162014246,
+            -0.002068739850074053,
+            0.00021711873705498874,
+            0.0261892918497324,
+            0.007450723554939032,
+            0.003045483259484172,
+            -0.008503138087689877,
+            -0.002073396695777774,
+            -0.009224927052855492,
+            -0.027195140719413757,
+            0.008647495880723,
+            -0.007869826629757881,
+            -0.006742904428392649,
+            0.005103745497763157,
+            -0.01706215739250183,
+            -0.019129732623696327,
+            -0.007725468836724758,
+            -0.0022352170199155807,
+            -0.005681176669895649,
+            0.0166058000177145,
+            0.01771409437060356,
+            -0.022184528410434723,
+            -0.006598547101020813,
+            -0.00918301660567522,
+            -0.0012142350897192955,
+            -0.014957327395677567,
+            0.006724277976900339,
+            0.006826725322753191,
+            0.02199826017022133,
+            -0.023656046018004417,
+            -0.008251676335930824,
+            0.0015576668083667755,
+            -0.0015145923243835568,
+            0.02482953481376171,
+            -0.003285303246229887,
+            -0.017229797318577766,
+            -0.02836862951517105,
+            0.011828023009002209,
+            -0.008111975155770779,
+            -0.00614219019189477,
+            -0.019912058487534523,
+            -0.007450723554939032,
+            0.0017648900393396616,
+            -0.013020139187574387,
+            0.0027777228970080614,
+            -0.02712063305079937,
+            0.012377514503896236,
+            -0.010421698912978172,
+            -0.006673054303973913,
+            -0.0030221997294574976,
+            -0.011446173302829266,
+            0.026561828330159187,
+            0.019818924367427826,
+            0.016866575926542282,
+            0.015003894455730915,
+            0.008638182654976845,
+            0.0006239980575628579,
+            -0.00817251205444336,
+            0.003315571928396821,
+            -0.023693300783634186,
+            0.02548147365450859,
+            -0.002238709479570389,
+            -0.02190512605011463,
+            0.00493610417470336,
+            -0.004659030586481094,
+            -0.01079423539340496,
+            -0.000244331342400983,
+            0.012805930338799953,
+            -0.0003213124582543969,
+            -0.0021642022766172886,
+            0.010747668333351612,
+            0.013336794450879097,
+            0.011362353339791298,
+            0.030510712414979935,
+            0.016084248200058937,
+            -0.013513749465346336,
+            -0.02065712958574295,
+            0.011697635985910892,
+            -0.009723193943500519,
+            0.0044448222033679485,
+            0.025015803053975105,
+            0.017704781144857407,
+            -0.0017800243804231286,
+            -0.009047972038388252,
+            -0.0012771005276590586,
+            0.008079377934336662,
+            0.00935065746307373,
+            0.010552086867392063,
+            -0.012004978023469448,
+            -0.009425165131688118,
+            -0.01049620658159256,
+            -0.0027963495813310146,
+            0.001626353245228529,
+            -0.010859428904950619,
+            -0.0028475732542574406,
+            -0.001629845704883337,
+            0.011110891588032246,
+            0.010254058055579662,
+            -0.006319144740700722,
+            -0.054129503667354584,
+            0.007040933705866337,
+            0.010710414499044418,
+            0.009751134552061558,
+            -0.010505519807338715,
+            0.01870131492614746,
+            0.0012503245379775763,
+            -0.0014796671457588673,
+            0.010775608941912651,
+            0.010766295716166496,
+            0.0024936639238148928,
+            0.013495123013854027,
+            0.01583278737962246,
+            0.011092264205217361,
+            0.006840695161372423,
+            0.029951907694339752,
+            -0.021234560757875443,
+            -0.003932585008442402,
+            -0.022668825462460518,
+            -0.03349100053310394,
+            -0.007064217235893011,
+            0.01422156859189272,
+            0.004244584124535322,
+            0.008982778526842594,
+            0.020433608442544937,
+            0.03494389355182648,
+            0.013672077096998692,
+            -0.002324858447536826,
+            -0.002270142314955592,
+            -0.02162572368979454,
+            0.0037207049317657948,
+            -0.0014249508967623115,
+            0.00477311946451664,
+            0.006971083115786314,
+            -0.004305121023207903,
+            0.002726498991250992,
+            0.0090945390984416,
+            -0.0015588310780003667,
+            -0.00982564128935337,
+            -3.0923412850825116e-05,
+            0.010226117447018623,
+            0.01547887735068798,
+            -0.00524810329079628,
+            -0.017481260001659393,
+            0.007129410747438669,
+            0.022668825462460518,
+            -0.004165419843047857,
+            -0.004298136103898287,
+            -0.006840695161372423,
+            -0.013904912397265434,
+            -0.0024191567208617926,
+            -0.025630488991737366,
+            0.021811991930007935,
+            -0.012982885353267193,
+            0.017648901790380478,
+            0.01664305292069912,
+            -0.009313404560089111,
+            0.00016051069542299956,
+            -0.0027078723069280386,
+            0.03613600879907608,
+            0.0050478652119636536,
+            0.027008872479200363,
+            0.01692245528101921,
+            0.025239326059818268,
+            0.008722002618014812,
+            0.015786219388246536,
+            0.011883903294801712,
+            -0.004025719128549099,
+            0.00954623892903328,
+            0.011101577430963516,
+            0.008479854092001915,
+            -0.0032759900204837322,
+            0.013085332699120045,
+            0.0017660543089732528,
+            -0.018887583166360855,
+            -0.013904912397265434,
+            0.002901125466451049,
+            0.010654534213244915,
+            0.006738247815519571,
+            0.01840328611433506,
+            -0.008260989561676979,
+            -0.01723911054432392,
+            -0.0035088250879198313,
+            0.011697635985910892,
+            0.024661894887685776,
+            -0.01422156859189272,
+            -0.009620746597647667,
+            -0.02747454307973385,
+            0.0028662001714110374,
+            -0.0023341719061136246,
+            0.008503138087689877,
+            -0.0019860833417624235,
+            0.012163305655121803,
+            -0.00026732380501925945,
+            -0.015152908861637115,
+            -0.008614898659288883,
+            0.020396355539560318,
+            -0.026971617713570595,
+            0.017518512904644012,
+            0.0008731316193006933,
+            -0.01483625266700983,
+            0.017406752333045006,
+            0.0021502322051674128,
+            0.014249508269131184,
+            0.007902423851191998,
+            0.0070688738487660885,
+            -0.01245202124118805,
+            0.010207490995526314,
+            -0.024345237761735916,
+            0.010281998664140701,
+            0.011073637753725052,
+            0.0017159946728497744,
+            0.018868956714868546,
+            0.01356962975114584,
+            0.004668343812227249,
+            -0.019427761435508728,
+            0.009066599421203136,
+            -0.02544422075152397,
+            0.013271600939333439,
+            -0.001437756814993918,
+            0.024792281910777092,
+            -0.008600928820669651,
+            -0.0006106100627221167,
+            0.006058369297534227,
+            0.014910760335624218,
+            -0.003485541557893157,
+            -0.0020338145550340414,
+            0.006999023258686066,
+            -0.0006164309452287853,
+            -0.0017707109218463302,
+            0.0005800504586659372,
+            -0.019483640789985657,
+            -0.02294822782278061,
+            -0.0005075805238448083,
+            -0.009089882485568523,
+            0.01613081619143486,
+            -0.02745591476559639,
+            0.001415637438185513,
+            -0.00733430590480566,
+            0.003969838377088308,
+            -0.00020416728511918336,
+            -0.017145976424217224,
+            0.012209872715175152,
+            -0.005266730207949877,
+            0.0016321741277351975,
+            0.023134496062994003,
+            0.03362138941884041,
+            0.00048517013783566654,
+            -0.016223950311541557,
+            0.0175930205732584,
+            0.0066963378340005875,
+            -0.013392675668001175,
+            -0.020768892019987106,
+            -0.01049620658159256,
+            -0.017294991761446,
+            -0.009010718204081059,
+            0.0025518727488815784,
+            0.003837122581899166,
+            -0.0069571128115057945,
+            -0.010803548619151115,
+            0.010682474821805954,
+            -0.013755897991359234,
+            -0.02294822782278061,
+            0.00965800043195486,
+            -0.0012247127015143633,
+            0.016140129417181015,
+            0.0010663848370313644,
+            -0.004260882269591093,
+            0.019632656127214432,
+            -0.01664305292069912,
+            0.009741820394992828,
+            0.017499886453151703,
+            0.02296685427427292,
+            -0.026654962450265884,
+            -0.00935997162014246,
+            -0.012610348872840405,
+            -0.013625510036945343,
+            -0.0008585794712416828,
+            0.02740003541111946,
+            0.012172618880867958,
+            0.004596164915710688,
+            0.015450937673449516,
+            -0.0038417791947722435,
+            0.009089882485568523,
+            -0.025425594300031662,
+            0.016754813492298126,
+            -0.0118093965575099,
+            0.02615203894674778,
+            0.016223950311541557,
+            0.010077103041112423,
+            0.020806144922971725,
+            -0.0028149764984846115,
+            0.0018452182412147522,
+            -0.007380872964859009,
+            0.01773272082209587,
+            0.014715178869664669,
+            0.028312748298048973,
+            -0.011446173302829266,
+            0.016773441806435585,
+            0.005648579448461533,
+            0.005331923719495535,
+            -0.002933722222223878,
+            0.018794449046254158,
+            -0.0038464360404759645,
+            0.002593782963231206,
+            0.011297158896923065,
+            0.00281730480492115,
+            -0.01998656615614891,
+            -0.0023155452217906713,
+            0.012852497398853302,
+            0.0004173569323029369,
+            0.0030315129552036524,
+            0.0005468714516609907,
+            0.010952563025057316,
+            -0.0073575894348323345,
+            -0.018766509369015694,
+            0.017229797318577766,
+            0.016438158228993416,
+            -0.014128434471786022,
+            -0.026524575427174568,
+            -0.007324992213398218,
+            -0.000489535799715668,
+            0.021439457312226295,
+            0.0012293693143874407,
+            0.0047335377894341946,
+            0.014910760335624218,
+            0.010244744829833508,
+            0.003844107501208782,
+            -0.003769600298255682,
+            0.0009255195618607104,
+            -0.018459167331457138,
+            -0.003918614704161882,
+            0.01163244154304266,
+            -0.008330839686095715,
+            -1.7971959096030332e-05,
+            -0.0029593342915177345,
+            -0.008978121913969517,
+            -0.009965342469513416,
+            0.016717560589313507,
+            0.002754439366981387,
+            0.007525230757892132,
+            0.01193047035485506,
+            0.040196653455495834,
+            -0.007157351355999708,
+            -0.004172405228018761,
+            0.024624640122056007,
+            0.004095569718629122,
+            -0.001629845704883337,
+            0.0219423808157444,
+            -0.005178252700716257,
+            -0.01711803674697876,
+            0.0021607098169624805,
+            0.01842191442847252,
+            0.007324992213398218,
+            -0.007082843687385321,
+            0.012181933037936687,
+            -0.014780372381210327,
+            0.020824771374464035,
+            0.004991984460502863,
+            -0.009974655695259571,
+            -0.00937859807163477,
+            0.005313297268003225,
+            0.007939676754176617,
+            0.0043866136111319065,
+            0.024065837264060974,
+            0.019064538180828094,
+            -0.02032184787094593,
+            0.007166664581745863,
+            0.009807014837861061,
+            0.017034215852618217,
+            -0.02097378671169281,
+            -0.001526234089396894,
+            0.009816328063607216,
+            0.01552544441074133,
+            0.0025704996660351753,
+            0.003152587218210101,
+            -0.02004244551062584,
+            -0.005313297268003225,
+            -0.01940913498401642,
+            -0.016810694709420204,
+            -0.016140129417181015,
+            -0.006337771657854319,
+            0.00517359608784318,
+            0.01585141383111477,
+            -0.01646609790623188,
+            0.0033877508249133825,
+            0.0028103196527808905,
+            -0.024047208949923515,
+            -0.01404461357742548,
+            0.0029360507614910603,
+            0.002756767673417926,
+            -0.00086672866018489,
+            -0.0002302157081430778,
+            -0.007529887370765209,
+            -0.018161138519644737,
+            0.016279829666018486,
+            -0.012992198579013348,
+            -0.01404461357742548,
+            0.004889537114650011,
+            0.01402598712593317,
+            0.005993175785988569,
+            -0.0066683972254395485,
+            0.0029057820793241262,
+            -0.015022520907223225,
+            -0.009983968921005726,
+            0.012563781812787056,
+            0.0004697448166552931,
+            -0.02553735487163067,
+            0.00868940632790327,
+            0.02231491729617119,
+            -0.00725979870185256,
+            0.002270142314955592,
+            -0.003681123023852706,
+            -0.01179077010601759,
+            -0.02287372015416622,
+            0.0077813491225242615,
+            -0.005653236526995897,
+            -0.013681390322744846,
+            -0.0002440402895445004,
+            0.03188909590244293,
+            -0.012163305655121803,
+            -0.002957005752250552,
+            0.009723193943500519,
+            -0.0010244745062664151,
+            -0.009862895123660564,
+            -0.0017241439782083035,
+            -0.01826358586549759,
+            0.004225957207381725,
+            0.0022573363967239857,
+            -0.004682314116507769,
+            0.008028154261410236,
+            -0.0109711904078722,
+            -0.018626809120178223,
+            0.01340198889374733,
+            -0.007674245163798332,
+            0.002428470179438591,
+            -0.007511260453611612,
+            -0.006575263570994139,
+            -0.012172618880867958,
+            -0.03466448932886124,
+            0.013299541547894478,
+            -0.012470648624002934,
+            -0.02127181552350521,
+            -0.014584790915250778,
+            -0.005695146508514881,
+            0.014463717117905617,
+            0.004328404553234577,
+            -0.03485075756907463,
+            -0.018077317625284195,
+            0.00970456749200821,
+            0.015441624447703362,
+            -0.009751134552061558,
+            -0.0132436603307724,
+            0.011343725956976414,
+            0.038930028676986694,
+            0.0026776036247611046,
+            -0.007436753250658512,
+            0.000422304670792073,
+            0.01051483303308487,
+            -0.011697635985910892,
+            -0.004032704047858715,
+            0.021849246695637703,
+            0.010086417198181152,
+            -0.0056672063656151295,
+            -0.006193413864821196,
+            0.027828451246023178,
+            0.00942050851881504,
+            0.018338093534111977,
+            -0.006579920183867216,
+            -0.007641647942364216,
+            -0.014761745929718018,
+            -0.004277180880308151,
+            -0.015087714418768883,
+            -0.02097378671169281,
+            0.009834954515099525,
+            0.009862895123660564,
+            -0.011064324527978897,
+            0.006682367529720068,
+            0.0029407073743641376,
+            0.0229296013712883,
+            -0.0072644553147256374,
+            0.003131632227450609,
+            -0.015981800854206085,
+            0.004833656828850508,
+            -0.0036322276573628187,
+            0.0046730004251003265,
+            0.006076996214687824,
+            -0.010309938341379166,
+            0.006426248699426651,
+            -0.017015589401125908,
+            -0.00982564128935337,
+            0.0054390281438827515,
+            -0.005578729324042797,
+            0.00025306266616098583,
+            -0.001689218683168292,
+            0.017015589401125908,
+            -0.002021008636802435,
+            -0.005606669466942549,
+            -0.0004228867474012077,
+            0.012973572127521038,
+            -0.01163244154304266,
+            -0.02158847078680992,
+            0.009266837500035763,
+            0.003974495455622673,
+            0.009881521575152874,
+            -0.005117715802043676,
+            -0.0005270804977044463,
+            -0.009448448196053505,
+            0.0010093401651829481,
+            -0.02713925950229168,
+            -0.01662442646920681,
+            0.003101363545283675,
+            0.0011211010860279202,
+            0.02484816312789917,
+            0.024680521339178085,
+            -0.0012736080680042505,
+            -0.007422782946377993,
+            -0.013765211217105389,
+            -0.0036531828809529543,
+            -0.008116631768643856,
+            0.011706949211657047,
+            -0.01113883126527071,
+            -0.013951479457318783,
+            -7.86727980539581e-07,
+            -0.042096588760614395,
+            0.0005590952932834625,
+            -0.014826939441263676,
+            0.014873506501317024,
+            0.004787089768797159,
+            0.020880652591586113,
+            -0.0036345559637993574,
+            -0.003210796043276787,
+            -0.0045030307956039906,
+            0.008912927471101284,
+            0.009457762353122234,
+            0.013681390322744846,
+            0.004633418750017881,
+            -0.01968853734433651,
+            0.025630488991737366,
+            -0.016261203214526176,
+            -0.03511153534054756,
+            0.016531292349100113,
+            -0.02067575789988041,
+            0.011408920399844646,
+            -0.007874483242630959,
+            0.0025332458317279816,
+            -0.0043586730025708675,
+            0.0209551602602005,
+            -0.01688520237803459,
+            -0.015916608273983,
+            0.00019732775399461389,
+            -0.012610348872840405,
+            0.0008271466940641403,
+            -0.01838465966284275,
+            -0.011539307422935963,
+            -0.006673054303973913,
+            0.02320900373160839,
+            0.006258607842028141,
+            -0.0031805275939404964,
+            -0.025965770706534386,
+            0.015069087967276573,
+            -0.02425210364162922,
+            0.013429928570985794,
+            -0.0004976850468665361,
+            -2.5630051823100075e-05,
+            -0.016931768506765366,
+            0.031069517135620117,
+            0.004633418750017881,
+            0.0031665575224906206,
+            0.010142297483980656,
+            0.023358017206192017,
+            0.005364520940929651,
+            0.005154969170689583,
+            0.0015972488326951861,
+            -1.1186999472556636e-05,
+            -0.0166058000177145,
+            0.01771409437060356,
+            -0.007972273975610733,
+            0.020582623779773712,
+            -0.0111202048137784,
+            0.010067789815366268,
+            0.0037463167682290077,
+            -0.0033202285412698984,
+            0.005825534462928772,
+            -0.005806907545775175,
+            -0.013318167999386787,
+            0.017462633550167084,
+            -0.01974441669881344,
+            0.010933936573565006,
+            -0.006165473721921444,
+            0.005550788715481758,
+            -0.004139808006584644,
+            0.01289906445890665,
+            -0.006901232525706291,
+            -0.02099241316318512,
+            0.008754599839448929,
+            -0.01641021855175495,
+            0.013765211217105389,
+            0.014240195043385029,
+            -0.005718430038541555,
+            -0.013802465051412582,
+            -0.007804632652550936,
+            -0.007324992213398218,
+            0.025127563625574112,
+            -0.018990030512213707,
+            -0.013327481225132942,
+            -0.01062659453600645,
+            0.0006071175448596478,
+            0.006263264454901218,
+            0.014482343569397926,
+            -0.0012922348687425256,
+            0.007059560157358646,
+            0.017248423770070076,
+            -0.012032918632030487,
+            0.0024890073109418154,
+            -0.018114570528268814,
+            0.005704460199922323,
+            -0.0029919310472905636,
+            0.001448234310373664,
+            -0.0013504435773938894,
+            0.026710843667387962,
+            0.014612731523811817,
+            -0.019008658826351166,
+            -0.018487107008695602,
+            -0.006449532229453325,
+            -0.030436204746365547,
+            0.001108295051380992,
+            -0.020135579630732536,
+            -0.02287372015416622,
+            0.03770066052675247,
+            -0.015702400356531143,
+            -0.018971404060721397,
+            0.004533299710601568,
+            -0.005187565926462412,
+            -0.0011059667449444532,
+            -0.004305121023207903,
+            -0.01569308526813984,
+            0.0006711471942253411,
+            -0.018095944076776505,
+            -0.0049547310918569565,
+            0.024643268436193466,
+            0.0021246203687042,
+            0.01417500153183937,
+            -0.002210769336670637,
+            -0.015860727056860924,
+            0.0008812808664515615,
+            -0.005783624015748501,
+            -0.005345894023776054,
+            -0.0016577859641984105,
+            -0.009145762771368027,
+            -0.0023306794464588165,
+            0.013327481225132942,
+            0.011967724189162254,
+            0.0033761090599000454,
+            -0.019166985526680946,
+            -0.005704460199922323,
+            0.021495336666703224,
+            -0.0032317512668669224,
+            -0.013951479457318783,
+            -0.011334412731230259,
+            -0.007050246931612492,
+            0.005611326079815626,
+            0.005201536230742931,
+            0.0028079913463443518,
+            0.001922053750604391,
+            0.011064324527978897,
+            0.013048078864812851,
+            0.00109956378582865,
+            -0.02743728831410408,
+            0.011110891588032246,
+            0.001075698179192841,
+            -0.00645418930798769,
+            -0.006230667233467102,
+            0.00574171356856823,
+            -0.016438158228993416,
+            -0.017565080896019936,
+            -0.01223781332373619,
+            -0.00574171356856823,
+            -0.005518191959708929,
+            0.015078401193022728,
+            -0.030734233558177948,
+            0.010403072461485863,
+            -0.01854298822581768,
+            -0.014053926803171635,
+            0.001118190586566925,
+            0.006896575912833214,
+            -0.0028033347334712744,
+            0.014426463283598423,
+            -0.010132983326911926,
+            0.016475411131978035,
+            -0.01051483303308487,
+            -0.003876704489812255,
+            0.005536818876862526,
+            -0.01688520237803459,
+            -0.0002786745026241988,
+            0.01148342713713646,
+            0.021160054951906204,
+            -0.026282425969839096,
+            0.01514359563589096,
+            -0.0035064967814832926,
+            -0.0053272671066224575,
+            0.025611860677599907,
+            0.00524810329079628,
+            -0.0027777228970080614,
+            -0.002700887154787779,
+            -0.003732346696779132,
+            -0.003078080015257001,
+            -0.012517215684056282,
+            -0.0054855952039361,
+            -0.029635252431035042,
+            -0.011874590069055557,
+            -0.027232393622398376,
+            -0.016056308522820473,
+            0.0017381140496581793,
+            -0.008847733959555626,
+            0.0004837149172089994,
+            -0.01778860203921795,
+            0.008717346005141735,
+            0.024978550150990486,
+            -0.012712797150015831,
+            0.0035530636087059975,
+            0.0028894837014377117,
+            0.008805823512375355,
+            0.005867444444447756,
+            0.007059560157358646,
+            0.012293693609535694,
+            0.0012817572569474578,
+            0.0036718095652759075,
+            0.0029407073743641376,
+            -0.01966990903019905,
+            -0.013606883585453033,
+            -0.010766295716166496,
+            0.018468480557203293,
+            -0.0008836092310957611,
+            -0.027064751833677292,
+            0.0029290656093508005,
+            0.024382492527365685,
+            0.017024902626872063,
+            -0.01713666319847107,
+            0.004731209482997656,
+            0.01648472622036934,
+            0.023488406091928482,
+            0.0036974214017391205,
+            0.041127994656562805,
+            -0.0001826300285756588,
+            0.028461763635277748,
+            0.005415744613856077,
+            0.009397224523127079,
+            0.0002062045969069004,
+            -0.03254103288054466,
+            -0.0020489490125328302,
+            -0.013318167999386787,
+            -0.012051545083522797,
+            0.033397868275642395,
+            0.016615113243460655,
+            -0.017816541716456413,
+            0.0054855952039361,
+            -0.00030937965493649244,
+            0.005588042549788952,
+            -0.02617066539824009,
+            -0.0012631304562091827,
+            0.0069571128115057945,
+            0.006179443560540676,
+            0.00517359608784318,
+            -0.028890179470181465,
+            0.003296945011243224,
+            -0.005210849456489086,
+            -0.0061841001734137535,
+            0.0024075149558484554,
+            -0.009252866730093956,
+            -0.022091394290328026,
+            0.020210087299346924,
+            0.009355314075946808,
+            0.016810694709420204,
+            0.012722110375761986,
+            -0.008940868079662323,
+            -0.008139914833009243,
+            -0.02328351140022278,
+            -0.01066384743899107,
+            0.001526234089396894,
+            0.0019057553727179766,
+            -0.002250351244583726,
+            0.017704781144857407,
+            0.0025379026774317026,
+            -0.00637968210503459,
+            -0.0008271466940641403,
+            -0.0012479961151257157,
+            -0.01579553261399269,
+            -0.003152587218210101,
+            -0.008917584083974361,
+            0.00839137751609087,
+            0.01404461357742548,
+            -0.012619663029909134,
+            -0.007138724438846111,
+            0.014435776509344578,
+            0.005681176669895649,
+            0.024289358407258987,
+            -0.010822176001966,
+            0.0014179657446220517,
+            0.02583538368344307,
+            -0.010878056287765503,
+            -0.010878056287765503,
+            0.012945631518959999,
+            0.015981800854206085,
+            0.014333329163491726,
+            0.0013876971788704395,
+            0.005373834166675806,
+            -0.012377514503896236,
+            -0.007855856791138649,
+            0.009751134552061558,
+            -0.028797045350074768,
+            8.30202698125504e-05,
+            0.00817251205444336,
+            -0.025332460179924965,
+            -0.0009947880171239376,
+            -0.013476495631039143,
+            0.0006007145857438445,
+            -0.010067789815366268,
+            -0.004526314325630665,
+            0.00039320028736256063,
+            0.012042231857776642,
+            -0.007720812223851681,
+            -0.019893432036042213,
+            0.0025774845853447914,
+            -0.020396355539560318,
+            0.021066920831799507,
+            -0.016065621748566628,
+            0.013290227390825748,
+            0.004768462851643562,
+            0.015981800854206085,
+            0.024680521339178085,
+            -0.006831381935626268,
+            -0.005713773425668478,
+            0.01519016269594431,
+            -0.013960792683064938,
+            -0.007557827513664961,
+            -0.0030478115659207106,
+            0.007930363528430462,
+            0.011734888888895512,
+            0.012200559489428997,
+            0.00201169541105628,
+            -0.01179077010601759,
+            0.01081286184489727,
+            -0.00044704339234158397,
+            0.002244530478492379,
+            0.014733805321156979,
+            -0.0072644553147256374,
+            -0.009648687206208706,
+            -0.0029034537728875875,
+            0.0034366461914032698,
+            -0.015199475921690464,
+            -0.00423527043312788,
+            0.0053272671066224575,
+            -0.02939310297369957,
+            -0.013178466819226742,
+            0.0018638450419530272,
+            -0.006896575912833214,
+            -0.03978686407208443,
+            -0.013541690073907375,
+            -0.009006061591207981,
+            -0.008950181305408478,
+            0.0041561066173017025,
+            -0.008475197479128838,
+            -0.012004978023469448,
+            0.021215934306383133,
+            0.027251020073890686,
+            0.024345237761735916,
+            -0.007851200178265572,
+            -0.01131578627973795,
+            0.021495336666703224,
+            -0.02192375436425209,
+            -0.005452997982501984,
+            -0.010980503633618355,
+            -0.0019139045616611838,
+            0.0034506162628531456,
+            0.007795319426804781,
+            0.015758279711008072,
+            -0.0018382332054898143,
+            0.013979420065879822,
+            -0.025053057819604874,
+            0.014202941209077835,
+            -0.004379628226161003,
+            0.0056672063656151295,
+            0.001721815555356443,
+            0.003895331406965852,
+            -0.006887262221425772,
+            0.019297374412417412,
+            -0.002672947011888027,
+            -0.013783838599920273,
+            0.023022735491394997,
+            -0.005839504301548004,
+            -0.002350470284000039,
+            0.0007084007957018912,
+            -0.00501061137765646,
+            0.025239326059818268,
+            0.0111202048137784,
+            -0.011017757467925549,
+            0.016512665897607803,
+            0.006910545751452446,
+            0.014938700012862682,
+            -0.024382492527365685,
+            0.005737056955695152,
+            0.010719728656113148,
+            -0.011706949211657047,
+            0.015786219388246536,
+            -0.02548147365450859,
+            0.018198391422629356,
+            0.0047335377894341946,
+            -0.011949097737669945,
+            -0.019204240292310715,
+            0.030845994129776955,
+            0.02069438435137272,
+            -0.0004354016564320773,
+            0.010803548619151115,
+            -0.017490573227405548,
+            -0.004342374857515097,
+            0.010114356875419617,
+            -0.0020605907775461674,
+            0.00767890177667141,
+            -0.013970105908811092,
+            0.0013131899759173393,
+            -0.009555553086102009,
+            0.007143381051719189,
+            -0.015711713582277298,
+            -0.0010431013070046902,
+            -0.03138617053627968,
+            0.0010221460834145546,
+            -0.01434264238923788,
+            0.0031432739924639463,
+            0.012973572127521038,
+            0.0029593342915177345,
+            0.010281998664140701,
+            0.008116631768643856,
+            0.021644352003932,
+            0.02060125023126602,
+            0.016521979123353958,
+            0.002498320769518614,
+            0.009495015256106853,
+            -0.006193413864821196,
+            0.01178145594894886,
+            0.0013294884702190757,
+            -0.007371559273451567,
+            0.010226117447018623,
+            -0.01852436177432537,
+            -0.013234347105026245,
+            0.023171748965978622,
+            -0.003755630226805806,
+            0.00841000396758318,
+            -0.01709941029548645,
+            -0.006738247815519571,
+            0.01113883126527071,
+            -0.009639373049139977,
+            0.014286762103438377,
+            -0.015897981822490692,
+            0.021122800186276436,
+            0.030529338866472244,
+            -0.016615113243460655,
+            -0.01891552470624447,
+            -0.02646869421005249,
+            0.0064728157594799995,
+            -0.01532986294478178,
+            -0.010095730423927307,
+            -0.002369097201153636,
+            0.0025402309838682413,
+            -3.296072100056335e-05,
+            0.024438371881842613,
+            0.005317953880876303,
+            -0.0013329809298738837,
+            -0.009788387455046177,
+            -0.011744203045964241,
+            0.019129732623696327,
+            0.002738140756264329,
+            -0.00970456749200821,
+            -0.00501061137765646,
+            -0.024121716618537903,
+            -0.0059233251959085464,
+            0.008512451313436031,
+            -0.0027590959798544645,
+            -0.022706080228090286,
+            0.014147060923278332,
+            0.006808098405599594,
+            -0.025555981323122978,
+            -0.004563568159937859,
+            -0.002268978161737323,
+            0.0012584737269207835,
+            -0.007948990911245346,
+            0.007362246047705412,
+            -0.003385422518476844,
+            0.007352932821959257,
+            0.006645113695412874,
+            -0.007511260453611612,
+            0.013998046517372131,
+            0.0005355207249522209,
+            0.01706215739250183,
+            -0.012312320061028004,
+            -0.01569308526813984,
+            -0.01143686007708311,
+            0.014202941209077835,
+            0.0011909515596926212,
+            0.009797701612114906,
+            -0.004184046760201454,
+            -0.0043982551433146,
+            0.0014074882492423058,
+            0.0008958330727182329,
+            0.003196825971826911,
+            -0.01907385140657425,
+            0.03414294123649597,
+            0.009983968921005726,
+            -0.00013373466208577156,
+            7.618073868798092e-05,
+            -0.020414981991052628,
+            0.007460036780685186,
+            0.0005011775647290051,
+            -0.0047521647065877914,
+            -0.004945417400449514,
+            0.0033691239077597857,
+            0.014230881817638874,
+            0.013783838599920273,
+            0.008414660580456257,
+            0.008195796050131321,
+            0.013113273307681084,
+            0.01434264238923788,
+            -0.0012805931037291884,
+            0.011222652159631252,
+            -0.014072554185986519,
+            -0.00235512712970376,
+            -0.016242576763033867,
+            -0.016726873815059662,
+            0.029281342402100563,
+            -0.019465014338493347,
+            0.022799214348196983,
+            0.008177168667316437,
+            0.0024191567208617926,
+            0.024661894887685776,
+            0.006221354007720947,
+            0.025313831865787506,
+            0.017509199678897858,
+            0.015721026808023453,
+            0.0023493061307817698,
+            0.009518299251794815,
+            0.014184314757585526,
+            0.005513535346835852,
+            0.008475197479128838,
+            0.00039640176692046225,
+            -0.005862787831574678,
+            -0.010747668333351612,
+            0.00355539214797318,
+            0.009741820394992828,
+            -0.011883903294801712,
+            -0.008260989561676979,
+            0.017285678535699844,
+            -0.0053272671066224575,
+            -0.011380979791283607,
+            0.00090747483773157,
+            0.005154969170689583,
+            0.012684856541454792,
+            -0.0027404692955315113,
+            -0.007776692509651184,
+            0.002151396358385682,
+            -0.023991329595446587,
+            0.010375131852924824,
+            0.011101577430963516,
+            0.0039046446327120066,
+            0.004917477257549763,
+            0.003913958091288805,
+            0.007720812223851681,
+            0.0004953566822223365,
+            -0.013085332699120045,
+            0.010719728656113148,
+            -0.025891263037919998,
+            -0.008107318542897701,
+            -0.021439457312226295,
+            -0.011585874482989311,
+            -0.007115440908819437,
+            -0.01820770464837551,
+            -0.014817626215517521,
+            0.00671030767261982,
+            -0.021458083763718605,
+            0.0028103196527808905,
+            -0.001522741629742086,
+            -0.01737881265580654,
+            -0.002635693410411477,
+            0.0118093965575099,
+            0.017267052084207535,
+            0.029504865407943726,
+            0.011548621580004692,
+            0.005983862094581127,
+            0.009024688974022865,
+            0.0062446375377476215,
+            -0.007548514287918806,
+            0.03930256515741348,
+            0.0011152802035212517,
+            -0.015758279711008072,
+            0.014016672968864441,
+            -0.014752432703971863,
+            -0.007040933705866337,
+            -0.007036277092993259,
+            0.0053784907795488834,
+            0.0084984814748168,
+            0.020731637254357338,
+            -0.01840328611433506,
+            0.0005104909650981426,
+            0.007562484126538038,
+            0.023153122514486313,
+            0.004745179321616888,
+            0.011716262437403202,
+            -0.0019022627966478467,
+            0.024270731955766678,
+            -0.003932585008442402,
+            -0.0010052655125036836,
+            -0.015376430004835129,
+            -0.008982778526842594,
+            -0.006840695161372423,
+            -0.004731209482997656,
+            -0.017015589401125908,
+            -0.01909247785806656,
+            -0.006095623131841421,
+            0.005303983576595783,
+            -0.007944333367049694,
+            0.010431013070046902,
+            0.007478663697838783,
+            0.014268135651946068,
+            0.018077317625284195,
+            0.025704994797706604,
+            0.010403072461485863,
+            0.005974548868834972,
+            0.010077103041112423,
+            -0.009862895123660564,
+            -0.002845244947820902,
+            -0.006836038548499346,
+            -0.04403377324342728,
+            0.005117715802043676,
+            0.0030245280358940363,
+            -0.01628914475440979,
+            0.0005954757798463106,
+            -0.009965342469513416,
+            0.015376430004835129,
+            0.00629120459780097,
+            -0.011427546851336956,
+            -0.015534758567810059,
+            0.015767592936754227,
+            -0.010440326295793056,
+            0.014873506501317024,
+            0.007422782946377993,
+            0.011604501865804195,
+            -0.006235324312001467,
+            0.019539522007107735,
+            -0.004875567276030779,
+            -0.01535780355334282,
+            0.017481260001659393,
+            0.008633526042103767,
+            -0.0006752217886969447,
+            0.010421698912978172,
+            0.0014191300142556429,
+            -0.02037772908806801,
+            -0.010188864544034004,
+            0.031423427164554596,
+            -0.001983755035325885,
+            -0.015804847702383995,
+            -0.002610081573948264,
+            -0.01062659453600645,
+            0.005769653711467981,
+            0.006645113695412874,
+            -0.004691627342253923,
+            -0.01163244154304266,
+            -0.020526742562651634,
+            -0.018785135820508003,
+            -0.028648031875491142,
+            0.02577950246632099,
+            -0.0017509199678897858,
+            -0.006901232525706291,
+            0.011911843903362751,
+            -5.009592860005796e-05,
+            0.0008358780178241432,
+            -0.0023702613543719053,
+            -0.010542773641645908,
+            0.07178772240877151,
+            -0.019837550818920135,
+            0.010570713318884373,
+            0.00016967857663985342,
+            -0.002754439366981387,
+            0.0017683826154097915,
+            -0.003969838377088308,
+            0.00954623892903328,
+            0.003795212134718895,
+            -0.011744203045964241,
+            -0.01370001770555973,
+            0.003641541115939617,
+            -0.014826939441263676,
+            -0.010524147190153599,
+            -0.0012503245379775763,
+            -0.006575263570994139,
+            -0.019008658826351166,
+            0.007534543983638287,
+            0.0007706842152401805,
+            -0.004507687408477068,
+            -0.008586958982050419,
+            -0.008079377934336662,
+            -0.01776997558772564,
+            0.010235431604087353,
+            -0.0021071576047688723,
+            -0.003415690967813134,
+            -0.03974960744380951,
+            -0.023749180138111115,
+            -0.008871017955243587,
+            -0.028834300115704536,
+            0.00815388560295105,
+            -0.015106341801583767,
+            -0.023786434903740883,
+            0.0068732923828065395,
+            0.007539200596511364,
+            0.006253950763493776,
+            -0.01761164702475071,
+            0.011241278611123562,
+            -0.014053926803171635,
+            -0.002642678329721093,
+            -0.0075531709007918835,
+            0.012889751233160496,
+            0.009145762771368027,
+            -8.149228960974142e-05,
+            -0.013774524442851543,
+            -0.009574179537594318,
+            0.01531123649328947,
+            -0.010086417198181152,
+            0.007893109694123268,
+            -0.004593836609274149,
+            -0.014771059155464172,
+            -0.020526742562651634,
+            0.01755576767027378,
+            0.0033132436219602823,
+            -0.006556636653840542,
+            0.01483625266700983,
+            0.0017450990853831172,
+            -0.00021871947683393955,
+            0.024643268436193466,
+            -0.011185398325324059,
+            -0.011772142723202705,
+            -0.008763913065195084,
+            -0.015544071793556213,
+            0.009630059823393822,
+            -0.019446387887001038,
+            0.011185398325324059,
+            -0.01289906445890665,
+            -0.022352170199155807,
+            -0.005015267990529537,
+            0.00177769607398659,
+            0.02453150600194931,
+            0.0018242630176246166,
+            -0.020563997328281403,
+            -0.012852497398853302,
+            0.006919859442859888,
+            -0.025276578962802887,
+            0.00023836494074203074,
+            0.0002408388099865988,
+            0.015450937673449516,
+            -0.016186697408556938,
+            0.016196010634303093,
+            -0.008600928820669651,
+            -0.0049873278476297855,
+            -0.0028545584063977003,
+            0.016037682071328163,
+            -0.0023621120490133762,
+            -0.024997176602482796,
+            -0.024103090167045593,
+            -0.01273142360150814,
+            -0.013578942976891994,
+            -0.0024075149558484554,
+            0.0226315725594759,
+            0.0024773655459284782,
+            0.026003023609519005,
+            0.007474007084965706,
+            -0.015152908861637115,
+            -0.015711713582277298,
+            -8.636726852273569e-05,
+            0.008265646174550056,
+            0.005411088000983,
+            0.001437756814993918,
+            -0.0017439349321648479,
+            -0.007422782946377993,
+            -0.007762722205370665,
+            -0.0033248853869736195,
+            0.03105088882148266,
+            0.0074414098635315895,
+            -0.0022596647031605244,
+            -0.005029238294810057,
+            -0.002712528919801116,
+            0.005611326079815626,
+            0.0027102006133645773,
+            -0.007162007968872786,
+            0.007767379283905029,
+            -0.004624105058610439,
+            0.0053365807980299,
+            0.025239326059818268,
+            -0.002100172685459256,
+            0.015572011470794678,
+            0.016894515603780746,
+            -0.01533917710185051,
+            -0.026394188404083252,
+            0.025611860677599907,
+            0.0019942326471209526,
+            -0.011520680971443653,
+            -0.011557934805750847,
+            -0.0226315725594759,
+            -0.006337771657854319,
+            -0.004884880501776934,
+            0.006752218119800091,
+            -0.009355314075946808,
+            -0.01695970818400383,
+            -0.006272577680647373,
+            -0.007376216351985931,
+            0.0015658161137253046,
+            -0.003650854341685772,
+            0.017704781144857407,
+            0.01840328611433506,
+            -0.01131578627973795,
+            0.024792281910777092,
+            -0.01532986294478178,
+            0.0567372590303421,
+            0.0015530101954936981,
+            -0.006770845036953688,
+            -0.010142297483980656,
+            -0.012116738595068455,
+            -0.012330947443842888,
+            0.0010780264856293797,
+            0.012507901526987553,
+            -0.0001615293585928157,
+            -0.016261203214526176,
+            -0.008028154261410236,
+            0.00652869651094079,
+            0.007474007084965706,
+            -0.0028010064270347357,
+            0.005308640189468861,
+            0.026952991262078285,
+            0.0002060590632027015,
+            -0.014491656795144081,
+            0.02974701300263405,
+            -0.003955868538469076,
+            0.003459929721429944,
+            0.0024890073109418154,
+            0.008824450895190239,
+            -0.005704460199922323,
+            0.00614219019189477,
+            0.013234347105026245,
+            0.003997778985649347,
+            0.012638289481401443,
+            0.0015611593844369054,
+            -0.0008975793025456369,
+            0.009252866730093956,
+            -0.009555553086102009,
+            0.00508511858060956,
+            -0.002896468620747328,
+            7.610797183588147e-05,
+            -0.003762615378946066,
+            -0.004076942801475525,
+            -0.014985267072916031,
+            0.014491656795144081,
+            -0.006035085767507553,
+            -0.004340046551078558,
+            -0.003755630226805806,
+            -0.011706949211657047,
+            -0.019614029675722122,
+            -0.014333329163491726,
+            0.017937617376446724,
+            -0.016019055619835854,
+            -0.012973572127521038,
+            0.026971617713570595,
+            0.015907295048236847,
+            -0.03125578537583351,
+            -0.013783838599920273,
+            -0.008330839686095715,
+            -0.014566164463758469,
+            0.018626809120178223,
+            -0.0074088131077587605,
+            0.007650961633771658,
+            -0.013280914165079594,
+            0.01453822385519743,
+            0.005876758135855198,
+            -0.021141426637768745,
+            0.010617280378937721,
+            -0.009075912646949291
+          ],
+          "title": "C Encouraging Exploration Across Experts"
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "12f04ba2-cf28-4e5a-aaa8-f47613191ec5",
+        "properties": {
+          "page_content": "D Switch Transformers in Lower Compute Regimes 29 ",
+          "embedding": [
+            0.023223955184221268,
+            0.008607305586338043,
+            -0.01453364733606577,
+            0.043360237032175064,
+            -0.007881038822233677,
+            0.03894453123211861,
+            0.03295178711414337,
+            0.07682663947343826,
+            0.0186422448605299,
+            0.02312435209751129,
+            -0.010591053403913975,
+            -0.009802534244954586,
+            0.008665407076478004,
+            0.01330521795898676,
+            -0.00808024313300848,
+            0.023273756727576256,
+            -0.016957305371761322,
+            0.01791182905435562,
+            0.009520327672362328,
+            -0.021547315642237663,
+            0.02274254336953163,
+            0.002058449201285839,
+            -0.04790874570608139,
+            0.03579045459628105,
+            -0.01648419350385666,
+            0.01867544651031494,
+            -0.016533995047211647,
+            -0.012840406969189644,
+            -0.023522762581706047,
+            0.02939930371940136,
+            -0.0005597446579486132,
+            0.009113618172705173,
+            -0.018758447840809822,
+            -0.025431808084249496,
+            -0.048473160713911057,
+            -0.027888666838407516,
+            -0.008001391775906086,
+            -0.017513418570160866,
+            -0.04266301915049553,
+            0.006719010882079601,
+            -0.04774274304509163,
+            -0.01855924353003502,
+            -0.026710039004683495,
+            0.03499363735318184,
+            0.02893449179828167,
+            0.0012471048394218087,
+            -0.011661779135465622,
+            0.01660039648413658,
+            0.01438424363732338,
+            -0.01910705678164959,
+            -0.009603329934179783,
+            -0.057968586683273315,
+            0.004353453870862722,
+            0.01889125257730484,
+            -0.026079224422574043,
+            0.02126510813832283,
+            0.025049999356269836,
+            0.11447633802890778,
+            -0.006486604921519756,
+            -0.0010707256151363254,
+            0.02028568461537361,
+            -0.02486739493906498,
+            -0.01766282320022583,
+            -0.03291858732700348,
+            -0.028585882857441902,
+            -0.01542176865041256,
+            -0.03071073442697525,
+            0.007947440259158611,
+            0.0005597446579486132,
+            -0.03542524576187134,
+            -0.04130178689956665,
+            -0.015604373067617416,
+            0.010881559923291206,
+            0.015371967107057571,
+            0.010873259976506233,
+            0.010449949651956558,
+            0.0406709723174572,
+            0.034628428518772125,
+            -0.02661043591797352,
+            0.01175308134406805,
+            0.01771262288093567,
+            -0.032221369445323944,
+            0.018625644966959953,
+            0.012201291508972645,
+            0.01673319935798645,
+            -0.001978559885174036,
+            -0.052291251718997955,
+            -0.04757673665881157,
+            0.03821411356329918,
+            0.00491371750831604,
+            -0.021846123039722443,
+            -0.02601282112300396,
+            0.003295178757980466,
+            -0.020401887595653534,
+            -0.008972514420747757,
+            0.04877196624875069,
+            -0.006113096140325069,
+            -0.01459174882620573,
+            0.025398606434464455,
+            0.008989115245640278,
+            0.0039674947038292885,
+            0.016807902604341507,
+            0.0043908050283789635,
+            -0.008997415192425251,
+            0.030146321281790733,
+            -0.011819482780992985,
+            -0.014367643743753433,
+            0.04452226310968399,
+            0.0186422448605299,
+            0.048738766461610794,
+            0.013047912158071995,
+            -0.024551987648010254,
+            -0.022344134747982025,
+            0.02325715683400631,
+            -0.003189351176843047,
+            -0.026710039004683495,
+            0.003627186641097069,
+            -0.012159790843725204,
+            -0.000483227166114375,
+            -0.00844960194081068,
+            0.026378029957413673,
+            0.013056212104856968,
+            -0.008881212212145329,
+            -0.01224279310554266,
+            -0.02101610228419304,
+            0.012201291508972645,
+            0.00771088432520628,
+            -0.02647763304412365,
+            0.006582057569175959,
+            0.01943906396627426,
+            0.0009275471675209701,
+            0.015172762796282768,
+            -0.00734982592985034,
+            -0.034163616597652435,
+            0.01629328913986683,
+            0.00948712695389986,
+            0.035724055022001266,
+            0.04681311920285225,
+            -0.001914233318530023,
+            -0.00859900563955307,
+            -0.0402393639087677,
+            -0.0012087164213880897,
+            0.0311423446983099,
+            0.00649075536057353,
+            -0.05415049567818642,
+            0.02388797141611576,
+            -0.005996893160045147,
+            0.03655407577753067,
+            0.004627360496670008,
+            0.005270625930279493,
+            -0.00020218764257151634,
+            0.03871212527155876,
+            -0.018775049597024918,
+            -0.0005281001213006675,
+            0.07596341520547867,
+            0.040139760822057724,
+            -0.02481759339570999,
+            -0.005204224493354559,
+            -0.0001957031199708581,
+            -0.01050805114209652,
+            0.015521370805799961,
+            0.022427136078476906,
+            -0.030810337513685226,
+            -0.024004174396395683,
+            0.005996893160045147,
+            0.004648111294955015,
+            0.04249701648950577,
+            0.007715034298598766,
+            -0.05577733367681503,
+            0.023622365668416023,
+            0.03731769323348999,
+            0.0220287274569273,
+            0.031374748796224594,
+            0.022559938952326775,
+            -0.004332703538239002,
+            0.01850944198668003,
+            -0.006075745448470116,
+            0.010151143185794353,
+            0.023921171203255653,
+            -0.021032702177762985,
+            -0.032885387539863586,
+            -0.016002783551812172,
+            -0.0109064606949687,
+            -0.017563220113515854,
+            0.003915618639439344,
+            0.011238468810915947,
+            0.02104930393397808,
+            -0.001071763108484447,
+            0.02762306109070778,
+            0.003759989980608225,
+            0.005399279296398163,
+            0.008134194649755955,
+            0.01791182905435562,
+            0.010383548215031624,
+            -0.019638270139694214,
+            -0.04359264299273491,
+            -0.00263116299174726,
+            0.035392045974731445,
+            -0.0014027334982529283,
+            -0.0031416250858455896,
+            -0.00401937123388052,
+            0.008715208619832993,
+            0.020385287702083588,
+            0.01194398570805788,
+            -0.016608696430921555,
+            -0.004839015658944845,
+            0.037550099194049835,
+            -0.016251789405941963,
+            0.009097017347812653,
+            -0.0258468184620142,
+            0.018227236345410347,
+            -0.005399279296398163,
+            0.008931013755500317,
+            0.013272017240524292,
+            -0.0014255590504035354,
+            0.00751997996121645,
+            0.025697413831949234,
+            0.009578429162502289,
+            -0.0377161018550396,
+            -0.01973787136375904,
+            0.021995525807142258,
+            0.00754903070628643,
+            0.023240555077791214,
+            -0.03778250515460968,
+            -0.02330695651471615,
+            0.011454273946583271,
+            0.015637574717402458,
+            -0.002007610397413373,
+            -0.002900919411331415,
+            -0.0402393639087677,
+            -0.038778528571128845,
+            0.006465854588896036,
+            0.014475545845925808,
+            -0.013819830492138863,
+            0.00028843191103078425,
+            -0.02727445214986801,
+            -0.00632060132920742,
+            0.05219164863228798,
+            -0.028287075459957123,
+            0.027888666838407516,
+            0.012334095314145088,
+            0.015313866548240185,
+            -0.01762962155044079,
+            -0.015513070859014988,
+            -0.0150399599224329,
+            0.008806510828435421,
+            -0.015919780358672142,
+            0.0648079514503479,
+            0.03321739286184311,
+            -0.013462921604514122,
+            -0.04116898402571678,
+            -0.01347122248262167,
+            0.002583436667919159,
+            -0.004127273801714182,
+            0.0014608348719775677,
+            0.02964830957353115,
+            -0.003932218998670578,
+            -0.015181062743067741,
+            -0.026129024103283882,
+            0.02016948163509369,
+            0.016102384775877,
+            -0.003091824008151889,
+            -0.030561331659555435,
+            0.029930515214800835,
+            -0.017048608511686325,
+            0.030826937407255173,
+            0.03612246364355087,
+            0.00788933876901865,
+            -0.0036126612685620785,
+            0.004760163836181164,
+            0.00200968561694026,
+            -0.0313415490090847,
+            -0.05806818976998329,
+            0.010325446724891663,
+            0.02085009776055813,
+            -0.032852184027433395,
+            -0.026278428733348846,
+            0.01188588421791792,
+            0.003847141982987523,
+            -0.010026639327406883,
+            0.001663152244873345,
+            0.0011174142127856612,
+            -0.00038388418033719063,
+            -0.007300024386495352,
+            -0.00700121745467186,
+            -0.04126858711242676,
+            -0.0063538020476698875,
+            0.04863916337490082,
+            -0.03834691643714905,
+            -0.0377161018550396,
+            0.005249875597655773,
+            -0.01296490989625454,
+            -0.006594507489353418,
+            0.01120526809245348,
+            0.034163616597652435,
+            -0.03489403426647186,
+            0.05401769280433655,
+            -0.027971668168902397,
+            0.02080029807984829,
+            -0.00859900563955307,
+            -0.015205963514745235,
+            -0.0016185387503355742,
+            0.02825387567281723,
+            -0.02443578466773033,
+            -0.028951091691851616,
+            -0.002340656006708741,
+            0.01588658057153225,
+            0.016044283285737038,
+            0.003556634997949004,
+            0.0351596400141716,
+            0.04485427215695381,
+            0.0056192344054579735,
+            -0.004855616018176079,
+            -0.06620237976312637,
+            -0.025979621335864067,
+            -0.010234144516289234,
+            0.020899899303913116,
+            -0.012400496751070023,
+            -0.011620277538895607,
+            -0.00472281314432621,
+            0.008706907741725445,
+            -0.036155663430690765,
+            -0.05584373697638512,
+            0.05268966034054756,
+            0.02085009776055813,
+            -0.03041192702949047,
+            -0.0025626863352954388,
+            0.026876043528318405,
+            -0.011429373174905777,
+            -0.028320277109742165,
+            -0.017264412716031075,
+            -0.019986877217888832,
+            0.012591401115059853,
+            0.007748235482722521,
+            0.025232603773474693,
+            0.026544034481048584,
+            0.0011475024512037635,
+            -0.020451689139008522,
+            -0.044289860874414444,
+            0.017978230491280556,
+            0.036321669816970825,
+            0.02926649898290634,
+            -0.04004015773534775,
+            -0.03456202521920204,
+            -0.03363240510225296,
+            -0.02186272293329239,
+            0.008798209950327873,
+            -0.007138170767575502,
+            -0.02923329919576645,
+            0.008237946778535843,
+            0.0010442687198519707,
+            -0.007573931012302637,
+            0.014774353243410587,
+            -0.03758329898118973,
+            0.062085483223199844,
+            -0.0478755459189415,
+            -0.02508319914340973,
+            -0.004056721925735474,
+            -0.025182802230119705,
+            -0.04824075475335121,
+            0.010815158486366272,
+            -0.021663518622517586,
+            0.0021912523079663515,
+            -0.024518785998225212,
+            -0.03894453123211861,
+            -0.03834691643714905,
+            0.03426321968436241,
+            -0.008773310109972954,
+            -0.021115705370903015,
+            0.021082503721117973,
+            -0.026129024103283882,
+            -0.0016403266927227378,
+            0.009072116576135159,
+            0.027772463858127594,
+            0.010806858539581299,
+            -0.018360039219260216,
+            0.03758329898118973,
+            -0.012491798959672451,
+            -0.02885149046778679,
+            0.008200596086680889,
+            -0.043526239693164825,
+            0.023605763912200928,
+            -0.04203220456838608,
+            -0.04837355762720108,
+            0.07735785096883774,
+            0.00859900563955307,
+            0.026793040335178375,
+            -0.014351042918860912,
+            -0.04143458977341652,
+            0.0336822047829628,
+            -9.992660488933325e-05,
+            -0.01703200675547123,
+            -0.02063429355621338,
+            -0.012566500343382359,
+            0.0347280316054821,
+            0.021281708031892776,
+            -0.0220287274569273,
+            0.014558548107743263,
+            0.007831237278878689,
+            -0.01910705678164959,
+            -0.005888990592211485,
+            0.013379920274019241,
+            -0.03084353730082512,
+            -0.018609045073390007,
+            0.008823110722005367,
+            0.01296490989625454,
+            -0.06839363276958466,
+            -0.013396520167589188,
+            -0.006573757156729698,
+            -0.00281169218942523,
+            -0.00595124252140522,
+            -0.05162723362445831,
+            0.011852683499455452,
+            -0.013562524691224098,
+            0.02765626087784767,
+            0.0010017302120104432,
+            0.0025128850247710943,
+            0.027805665507912636,
+            -0.022725943475961685,
+            -0.023057950660586357,
+            0.009395824745297432,
+            0.047510337084531784,
+            0.01880824938416481,
+            0.015596073120832443,
+            0.033781807869672775,
+            0.015562872402369976,
+            0.03396441042423248,
+            0.022510137408971786,
+            0.020816897973418236,
+            0.0020958001259714365,
+            0.02377176843583584,
+            -0.0013612326001748443,
+            -0.04163379594683647,
+            -0.009312822483479977,
+            -0.04608270153403282,
+            0.02511640079319477,
+            -0.014691350981593132,
+            0.036155663430690765,
+            -0.012973209843039513,
+            0.0017254037084057927,
+            0.028320277109742165,
+            -0.010383548215031624,
+            -0.02486739493906498,
+            0.020999502390623093,
+            -0.04963518679141998,
+            0.028187474235892296,
+            -0.008200596086680889,
+            0.016168786212801933,
+            -0.0037309392355382442,
+            0.013886231929063797,
+            -0.03469482809305191,
+            0.0258468184620142,
+            -0.013844731263816357,
+            -0.022759143263101578,
+            0.02126510813832283,
+            -0.03851291909813881,
+            0.027772463858127594,
+            0.01624348759651184,
+            -0.015322166495025158,
+            -0.03960854560136795,
+            0.03336679935455322,
+            0.033150993287563324,
+            0.032553378492593765,
+            -0.0347280316054821,
+            0.013338418677449226,
+            0.0423642136156559,
+            -0.05255685746669769,
+            -0.005967842880636454,
+            -0.02685944177210331,
+            0.02426978014409542,
+            -0.01311431359499693,
+            -0.00978593435138464,
+            0.008329248987138271,
+            0.05800178647041321,
+            -0.024087175726890564,
+            -0.0041438741609454155,
+            0.0018260437063872814,
+            -0.07616262137889862,
+            -0.014118637889623642,
+            -0.01370362751185894,
+            0.012425397522747517,
+            0.004959368612617254,
+            0.0026871892623603344,
+            -0.018758447840809822,
+            0.03122534602880478,
+            0.019721271470189095,
+            0.018907852470874786,
+            0.0330347903072834,
+            -0.05036560446023941,
+            0.018758447840809822,
+            -0.005880690645426512,
+            -0.026029422879219055,
+            -0.00212277565151453,
+            -0.0235061626881361,
+            -0.011595377698540688,
+            -0.06328071653842926,
+            -0.044123854488134384,
+            -0.04960198700428009,
+            0.025016797706484795,
+            0.03549164906144142,
+            -0.05109602212905884,
+            -0.021331509575247765,
+            -0.005839189514517784,
+            0.008457901887595654,
+            -0.02227773331105709,
+            -0.03409721702337265,
+            0.006283250171691179,
+            0.0009960237657651305,
+            -0.0025045848451554775,
+            0.0005753075238317251,
+            0.05939621850848198,
+            0.0022472788114100695,
+            0.028917891904711723,
+            -0.022941747680306435,
+            -0.027208050712943077,
+            0.006864264141768217,
+            -0.006191947963088751,
+            0.005486431065946817,
+            0.017463617026805878,
+            -0.007441128138452768,
+            0.006158747244626284,
+            -0.0047975145280361176,
+            0.007466028444468975,
+            0.04621550440788269,
+            -0.005984443239867687,
+            0.023057950660586357,
+            0.018077831715345383,
+            -0.009744432754814625,
+            0.04047176614403725,
+            0.009951937943696976,
+            0.007142320740967989,
+            0.024751191958785057,
+            -0.008947613649070263,
+            -0.021630316972732544,
+            -0.008486952632665634,
+            0.03535884618759155,
+            -0.012035287916660309,
+            -0.007835387252271175,
+            -0.026278428733348846,
+            0.020518090575933456,
+            0.01518936362117529,
+            0.04641471058130264,
+            -0.02219473011791706,
+            -0.009080417454242706,
+            -0.007573931012302637,
+            0.005258175544440746,
+            0.014458945952355862,
+            -0.011869283393025398,
+            0.013645526021718979,
+            0.003253677859902382,
+            -0.01791182905435562,
+            -0.0436258427798748,
+            -0.006768811959773302,
+            -0.009453926235437393,
+            -0.006042544264346361,
+            -0.0072585237212479115,
+            0.0011620278237387538,
+            0.023008150979876518,
+            -0.00014642068708781153,
+            -0.01654229499399662,
+            -0.007428677752614021,
+            0.020833497866988182,
+            0.01635139063000679,
+            0.011346370913088322,
+            0.013886231929063797,
+            -0.00018052931409329176,
+            0.01621858775615692,
+            0.04100298136472702,
+            -0.0070634689182043076,
+            -0.011196968145668507,
+            -0.022095128893852234,
+            0.0177956260740757,
+            -0.00961992982774973,
+            0.004436456132680178,
+            0.00422480097040534,
+            0.05521291866898537,
+            -0.026842841878533363,
+            -0.013944333419203758,
+            -0.015737175941467285,
+            0.022061927244067192,
+            0.00933772325515747,
+            -0.03144115209579468,
+            0.011130565777420998,
+            0.006582057569175959,
+            -0.017994830384850502,
+            -0.024452384561300278,
+            -0.006901614833623171,
+            -0.025747215375304222,
+            0.007187971845269203,
+            -0.0018654696177691221,
+            0.006345501635223627,
+            0.04262981936335564,
+            -0.017015406861901283,
+            -0.01546326931566,
+            0.006598657928407192,
+            0.010889860801398754,
+            0.02159711718559265,
+            -0.010806858539581299,
+            0.022808944806456566,
+            -0.04103618115186691,
+            0.015529671683907509,
+            0.02579701691865921,
+            0.013670426793396473,
+            -0.013230516575276852,
+            0.004448906518518925,
+            0.032984986901283264,
+            -0.02592981979250908,
+            -0.0005654510459862649,
+            -0.002218228066340089,
+            0.009819135069847107,
+            0.03592325747013092,
+            -0.00855750497430563,
+            0.020401887595653534,
+            -0.014168438501656055,
+            -0.017546620219945908,
+            0.024933796375989914,
+            -0.046149104833602905,
+            0.024966996163129807,
+            0.0035524850245565176,
+            -0.01711500994861126,
+            0.035392045974731445,
+            -0.017380615696310997,
+            0.011371271684765816,
+            0.024136977270245552,
+            -0.016177086159586906,
+            0.019040655344724655,
+            0.0055113318376243114,
+            -0.02707524783909321,
+            -0.013985834084451199,
+            -0.0018488692585378885,
+            0.021115705370903015,
+            -0.004357604309916496,
+            -0.013653826899826527,
+            -0.04146779328584671,
+            0.021680118516087532,
+            -0.022725943475961685,
+            0.009927037172019482,
+            -0.020186083391308784,
+            0.020601091906428337,
+            0.004635660909116268,
+            -0.007881038822233677,
+            0.017131609842181206,
+            0.032603178173303604,
+            -0.03213836997747421,
+            -0.019472265616059303,
+            0.013396520167589188,
+            0.012848706915974617,
+            0.02028568461537361,
+            0.007478478830307722,
+            0.020899899303913116,
+            0.03456202521920204,
+            0.009860635735094547,
+            0.0175798200070858,
+            0.017845427617430687,
+            0.013446321710944176,
+            -0.008130044676363468,
+            0.0037952656857669353,
+            -0.007619582116603851,
+            -0.012209592387080193,
+            0.0022970798891037703,
+            -0.0389777310192585,
+            -0.009968538768589497,
+            0.028735287487506866,
+            0.021846123039722443,
+            -0.012168090790510178,
+            -0.033067990094423294,
+            0.000248616881435737,
+            0.02151411399245262,
+            -0.01075705699622631,
+            0.024966996163129807,
+            -0.03502683714032173,
+            -0.0343296192586422,
+            0.006984617095440626,
+            -0.026145625859498978,
+            0.005565282888710499,
+            -0.026544034481048584,
+            -0.00019012641860172153,
+            0.026444431394338608,
+            -0.012217892333865166,
+            0.010350347496569157,
+            -0.01919005811214447,
+            0.020451689139008522,
+            0.008574104867875576,
+            -8.741146302781999e-05,
+            0.012309194542467594,
+            0.007374726235866547,
+            -0.041202183812856674,
+            -0.011429373174905777,
+            -0.02672663889825344,
+            0.03366560488939285,
+            -0.011811182834208012,
+            -0.006113096140325069,
+            -0.009669731371104717,
+            0.013346719555556774,
+            -0.019538667052984238,
+            -0.04189940169453621,
+            0.010823458433151245,
+            0.004980118945240974,
+            0.017231212928891182,
+            -0.021763119846582413,
+            0.008266997523605824,
+            -0.01512296125292778,
+            0.003008821979165077,
+            0.010292246006429195,
+            0.007312474772334099,
+            0.003932218998670578,
+            0.0065447064116597176,
+            0.00808024313300848,
+            -0.01419333927333355,
+            -0.012275993824005127,
+            0.011562176048755646,
+            -0.009570129215717316,
+            -0.00560263404622674,
+            0.06656759232282639,
+            0.03258657827973366,
+            0.0025398607831448317,
+            -0.02320735529065132,
+            -0.01512296125292778,
+            -0.03911053389310837,
+            -0.002927894936874509,
+            0.02661043591797352,
+            0.03057793155312538,
+            -0.01957186870276928,
+            -0.0046066101640462875,
+            -0.012441997416317463,
+            -0.015006759203970432,
+            0.03688608109951019,
+            0.004598309751600027,
+            0.024070575833320618,
+            -0.0029341201297938824,
+            0.0147245517000556,
+            0.009669731371104717,
+            -0.02699224464595318,
+            -0.0002613265533000231,
+            -0.0032702782191336155,
+            -0.006017643958330154,
+            0.012068488635122776,
+            0.03529244288802147,
+            0.0013384070480242372,
+            -0.024468984454870224,
+            0.01668339967727661,
+            0.0003351205086801201,
+            -0.016343090683221817,
+            0.02541520819067955,
+            0.001071763108484447,
+            -0.017181411385536194,
+            0.016500795260071754,
+            0.002089574933052063,
+            -0.017845427617430687,
+            0.039376143366098404,
+            -0.015928080305457115,
+            -0.05544532462954521,
+            0.00670241005718708,
+            -0.0008647768991068006,
+            -0.02694244496524334,
+            0.021464312449097633,
+            0.006914065219461918,
+            -0.00192772108130157,
+            0.01400243490934372,
+            0.014483846724033356,
+            -0.002081274753436446,
+            0.022393936291337013,
+            0.00719212181866169,
+            0.029465705156326294,
+            0.02554801106452942,
+            0.011802881956100464,
+            -0.013263717293739319,
+            -0.044920675456523895,
+            0.013902832753956318,
+            0.05833379551768303,
+            0.0381145104765892,
+            0.031059343367815018,
+            0.01434274297207594,
+            -0.004880516789853573,
+            0.0020750495605170727,
+            0.016284989193081856,
+            0.017148209735751152,
+            -0.03794850781559944,
+            0.0013653826899826527,
+            0.021962326020002365,
+            0.00855750497430563,
+            -0.007627882529050112,
+            -0.02792186848819256,
+            0.001390283228829503,
+            -0.017994830384850502,
+            0.008059492334723473,
+            0.02320735529065132,
+            -0.002274254336953163,
+            0.014624949544668198,
+            0.019173458218574524,
+            -0.00793083943426609,
+            0.016666797921061516,
+            -0.013446321710944176,
+            0.010474850423634052,
+            0.016550594940781593,
+            -0.004197825212031603,
+            -0.021613717079162598,
+            0.01158707682043314,
+            -0.045352283865213394,
+            -0.044289860874414444,
+            0.022642942145466805,
+            0.002205777680501342,
+            -0.020916501060128212,
+            0.004739413503557444,
+            0.00719212181866169,
+            -0.007129870355129242,
+            0.018144233152270317,
+            -0.04037216678261757,
+            0.013371619395911694,
+            -0.001614388544112444,
+            0.031241947785019875,
+            -0.0495687834918499,
+            0.019090456888079643,
+            0.036454472690820694,
+            -0.002919594757258892,
+            -0.01050805114209652,
+            -0.0003117762098554522,
+            -0.006308150943368673,
+            0.00015926006017252803,
+            -0.003587760729715228,
+            0.023024750873446465,
+            0.021215306594967842,
+            0.01321391575038433,
+            -0.023987572640180588,
+            -0.017928428947925568,
+            -0.018658846616744995,
+            0.035093240439891815,
+            -0.008798209950327873,
+            0.026212027296423912,
+            0.046912722289562225,
+            -0.01826043613255024,
+            0.00944562628865242,
+            -0.01071555633097887,
+            -0.01557117234915495,
+            -6.3183638303598855e-06,
+            0.02999691665172577,
+            -0.012026987969875336,
+            0.016152186319231987,
+            -0.03914373740553856,
+            0.010723856277763844,
+            0.018758447840809822,
+            0.013587424531579018,
+            -0.015330466441810131,
+            -0.0317731611430645,
+            -0.016774700954556465,
+            -0.015031659044325352,
+            -0.0003994470462203026,
+            0.006685809697955847,
+            0.03582365810871124,
+            0.05262325704097748,
+            0.04774274304509163,
+            0.024253180250525475,
+            0.008026291616261005,
+            -0.004270452074706554,
+            -0.05879860743880272,
+            0.008399800397455692,
+            0.030212722718715668,
+            0.00123361696023494,
+            -0.019306261092424393,
+            -9.441475413041189e-05,
+            -0.007810486946254969,
+            0.02528240531682968,
+            0.05023280158638954,
+            -0.027855465188622475,
+            0.025232603773474693,
+            -0.0026353129651397467,
+            0.0036126612685620785,
+            0.01194398570805788,
+            0.011852683499455452,
+            -0.04329383373260498,
+            -0.05398448929190636,
+            0.017397215589880943,
+            -0.01243369746953249,
+            -0.0034196816850453615,
+            0.013089412823319435,
+            -0.01782882586121559,
+            -0.024236580356955528,
+            -0.025730615481734276,
+            0.037550099194049835,
+            0.036188866943120956,
+            0.005191774107515812,
+            -0.004502857569605112,
+            -0.013462921604514122,
+            0.00810929387807846,
+            0.03665367513895035,
+            0.022891947999596596,
+            -0.0027535909321159124,
+            -0.0003333048371132463,
+            0.02385476976633072,
+            -0.017895227298140526,
+            -0.006536406464874744,
+            -0.013421420939266682,
+            0.026676837354898453,
+            -0.015537971630692482,
+            0.001002248958684504,
+            0.0018716948106884956,
+            0.007673533633351326,
+            0.01012624241411686,
+            0.009561828337609768,
+            -0.013761729001998901,
+            -0.0043078032322227955,
+            -0.039840951561927795,
+            0.020435089245438576,
+            -0.013180715031921864,
+            0.01867544651031494,
+            -0.010806858539581299,
+            0.0005216156132519245,
+            0.02863568440079689,
+            0.023456361144781113,
+            -0.015347067266702652,
+            -0.010109641589224339,
+            -0.015687374398112297,
+            0.02036868780851364,
+            -0.0073539759032428265,
+            0.028370078653097153,
+            0.003654162399470806,
+            0.0042829024605453014,
+            -0.03046172857284546,
+            0.026195425540208817,
+            0.0037662149406969547,
+            -0.021065903827548027,
+            0.0036375620402395725,
+            0.02481759339570999,
+            0.02765626087784767,
+            -0.019937077537178993,
+            -0.018409840762615204,
+            -0.01621858775615692,
+            0.02008648030459881,
+            -0.0020345861557871103,
+            0.03681968152523041,
+            -0.010425048880279064,
+            0.024170177057385445,
+            0.02194572426378727,
+            -0.025448407977819443,
+            0.0258468184620142,
+            -0.01226769294589758,
+            -0.004320253152400255,
+            0.018360039219260216,
+            -0.013886231929063797,
+            -0.0011091140331700444,
+            -0.027390655130147934,
+            0.016210287809371948,
+            0.00700121745467186,
+            -0.02533220499753952,
+            -0.007055168505758047,
+            0.024767791852355003,
+            -0.011346370913088322,
+            0.029515504837036133,
+            -0.007378876209259033,
+            -0.0032267021015286446,
+            0.0036292618606239557,
+            -0.004519457928836346,
+            -0.01012624241411686,
+            0.009478827007114887,
+            -0.006295700557529926,
+            -0.05833379551768303,
+            -0.021032702177762985,
+            0.024983597919344902,
+            -0.012109989300370216,
+            0.023223955184221268,
+            0.037981707602739334,
+            0.0021373010240495205,
+            -0.02617882564663887,
+            0.0007885188679210842,
+            0.022128328680992126,
+            -0.029034094884991646,
+            -0.01673319935798645,
+            -0.020833497866988182,
+            -0.03781570494174957,
+            -0.0019318711711093783,
+            -0.025299005210399628,
+            -0.006324751302599907,
+            0.04511987790465355,
+            0.007685983553528786,
+            -0.012035287916660309,
+            -0.007084219250828028,
+            0.014873955398797989,
+            -0.0004430231056176126,
+            0.006150446832180023,
+            0.008482802659273148,
+            0.029830913990736008,
+            -0.00192253349814564,
+            0.014774353243410587,
+            0.05484770983457565,
+            0.016467593610286713,
+            -0.012184691615402699,
+            -0.006387002766132355,
+            -0.01867544651031494,
+            -0.003214251948520541,
+            -0.004502857569605112,
+            0.0011827782727777958,
+            -0.02338995970785618,
+            0.04040536656975746,
+            -0.02888469025492668,
+            0.014525347389280796,
+            -0.02224453166127205,
+            0.0020491115283221006,
+            -0.05006679892539978,
+            0.010591053403913975,
+            -0.013803229667246342,
+            -0.02036868780851364,
+            -0.009918737225234509,
+            0.0002780566574074328,
+            0.05185963958501816,
+            0.0004969743895344436,
+            0.01867544651031494,
+            -0.004502857569605112,
+            -0.019206659868359566,
+            -0.015454969368875027,
+            -0.02046828903257847,
+            0.005270625930279493,
+            -6.296478386502713e-05,
+            0.018160834908485413,
+            0.0006251087179407477,
+            -0.0006152522400952876,
+            0.02418677881360054,
+            0.0288016889244318,
+            -0.019306261092424393,
+            -0.027506858110427856,
+            0.05899780988693237,
+            -0.013645526021718979,
+            -0.02579701691865921,
+            -0.02020268328487873,
+            0.005864090286195278,
+            -0.003253677859902382,
+            0.012251093052327633,
+            0.004847316071391106,
+            -0.002434033202007413,
+            -0.00297147105447948,
+            -0.02592981979250908,
+            -0.0004222725983709097,
+            -0.013064512051641941,
+            -0.015338766388595104,
+            -0.022676141932606697,
+            0.0008803398231975734,
+            0.0018696197075769305,
+            0.011670079082250595,
+            -0.008540904149413109,
+            -0.019173458218574524,
+            0.00047259253915399313,
+            -0.0006634971359744668,
+            -0.011138866655528545,
+            0.02792186848819256,
+            -0.012002087198197842,
+            -0.03562445193529129,
+            0.019854074344038963,
+            0.0024547837674617767,
+            0.017729224637150764,
+            0.0129234092310071,
+            0.014409144409000874,
+            0.009047216735780239,
+            0.0032184019219130278,
+            0.01812763325870037,
+            -0.02426978014409542,
+            0.002842817921191454,
+            -0.001082138391211629,
+            0.0030025967862457037,
+            -0.02868548594415188,
+            0.023904571309685707,
+            0.003488158341497183,
+            -0.012350695207715034,
+            -0.0005498881218954921,
+            0.002855268307030201,
+            -0.01867544651031494,
+            0.005129522643983364,
+            0.02524920366704464,
+            -0.0015614748699590564,
+            0.012358995154500008,
+            0.016193687915802002,
+            -0.01258310116827488,
+            0.0029133695643395185,
+            -0.0009591917041689157,
+            -0.005187624134123325,
+            -0.0020449613220989704,
+            0.016475893557071686,
+            0.020302286371588707,
+            0.010192643851041794,
+            -0.012026987969875336,
+            -0.03376520797610283,
+            0.00971953198313713,
+            -0.012566500343382359,
+            0.01673319935798645,
+            -0.007656933274120092,
+            0.018575843423604965,
+            -0.029432503506541252,
+            0.028270475566387177,
+            -0.013238816522061825,
+            -0.00683106342330575,
+            -0.015546271577477455,
+            -0.025016797706484795,
+            -0.023373357951641083,
+            0.011238468810915947,
+            0.025365406647324562,
+            -0.006938965991139412,
+            0.015977881848812103,
+            -0.012085088528692722,
+            -0.005245725624263287,
+            0.06291550397872925,
+            0.01586167886853218,
+            0.012649502605199814,
+            -0.023323558270931244,
+            -0.00913021806627512,
+            -0.01385303121060133,
+            -0.0298641137778759,
+            -0.0007330112857744098,
+            0.025066599249839783,
+            0.02754005789756775,
+            0.014857355505228043,
+            0.008615606464445591,
+            0.011761381290853024,
+            0.009578429162502289,
+            0.0034985337406396866,
+            -0.00613384647294879,
+            -0.009644830599427223,
+            2.2695854568155482e-05,
+            0.00281169218942523,
+            -0.008296048268675804,
+            -0.012184691615402699,
+            -0.0036583123728632927,
+            -0.009287921711802483,
+            0.015197663567960262,
+            -0.016633598133921623,
+            0.006611107848584652,
+            -0.011122265830636024,
+            0.0015365742146968842,
+            -0.019538667052984238,
+            -0.01336331944912672,
+            0.011827782727777958,
+            0.01396093424409628,
+            -0.0007003292557783425,
+            -0.00424555130302906,
+            -0.003886567894369364,
+            0.02219473011791706,
+            -0.023921171203255653,
+            0.013537623919546604,
+            0.01910705678164959,
+            0.017845427617430687,
+            -0.012226192280650139,
+            0.023622365668416023,
+            0.008781610056757927,
+            0.008781610056757927,
+            -0.03808131068944931,
+            -0.018277037888765335,
+            0.007225322537124157,
+            -0.0038222414441406727,
+            0.00012969059753231704,
+            0.02016948163509369,
+            0.003577385563403368,
+            0.011570476926863194,
+            -0.03374860808253288,
+            -0.0065322560258209705,
+            -0.011221867986023426,
+            0.006743911188095808,
+            0.006557156797498465,
+            -0.009503726847469807,
+            0.00756148062646389,
+            0.037848904728889465,
+            -0.002983921440318227,
+            0.0122842937707901,
+            0.0069555663503706455,
+            -0.002749440725892782,
+            -0.010084740817546844,
+            0.0147245517000556,
+            -0.016849402338266373,
+            0.013612325303256512,
+            -0.02546500973403454,
+            -0.002585511887446046,
+            -0.016301589086651802,
+            0.01973787136375904,
+            0.004345153924077749,
+            0.018609045073390007,
+            -0.0025917370803654194,
+            -0.020534690469503403,
+            -0.009188319556415081,
+            -0.017513418570160866,
+            0.002475534100085497,
+            -0.006407753098756075,
+            0.0018457566620782018,
+            -0.006357952021062374,
+            -0.00014655037375632674,
+            0.009196619503200054,
+            -0.019771073013544083,
+            -0.018791649490594864,
+            -0.019771073013544083,
+            -0.015488170087337494,
+            0.006250049453228712,
+            -0.014425745233893394,
+            -0.006785412319004536,
+            -0.002328205620869994,
+            0.005843339487910271,
+            0.041833002120256424,
+            -0.020833497866988182,
+            -0.028652284294366837,
+            0.02503339946269989,
+            -0.00022877422452438623,
+            -0.009171719662845135,
+            0.01370362751185894,
+            -0.0012211666908115149,
+            -0.005154423415660858,
+            0.018990853801369667,
+            -0.01889125257730484,
+            0.007229472976177931,
+            0.008673707023262978,
+            0.02109910361468792,
+            -0.008682007901370525,
+            -0.0017918053781613708,
+            0.00472281314432621,
+            0.0207504965364933,
+            -0.00898081436753273,
+            0.006478304974734783,
+            -0.006590357515960932,
+            0.006075745448470116,
+            0.002392532303929329,
+            0.043061431497335434,
+            -0.010109641589224339,
+            0.004353453870862722,
+            -0.006428503897041082,
+            -0.011221867986023426,
+            0.003284803591668606,
+            0.03212176635861397,
+            -0.012350695207715034,
+            -0.009453926235437393,
+            -0.0004974931362085044,
+            0.004258001688867807,
+            -0.0012782305711880326,
+            0.009669731371104717,
+            -0.002780566457659006,
+            -0.019422464072704315,
+            0.009370923973619938,
+            -0.0021808771416544914,
+            -0.010109641589224339,
+            -0.008839711546897888,
+            0.012417096644639969,
+            -0.014873955398797989,
+            0.003652087412774563,
+            -0.009022315964102745,
+            0.0038969432935118675,
+            0.017729224637150764,
+            0.02312435209751129,
+            -0.013545923866331577,
+            0.00015848191105760634,
+            -0.014019034802913666,
+            -0.02991391532123089,
+            -0.022344134747982025,
+            0.004980118945240974,
+            0.001614388544112444,
+            -0.007856138050556183,
+            0.007237772922962904,
+            0.0004977525095455348,
+            -0.011686678975820541,
+            0.007922539487481117,
+            0.001984784845262766,
+            -0.020003478974103928,
+            0.017646221444010735,
+            -0.005183474160730839,
+            0.014425745233893394,
+            0.004206125624477863,
+            0.012691003270447254,
+            0.004094073083251715,
+            -0.0008103068685159087,
+            -0.004677162040024996,
+            0.005909741390496492,
+            -0.01829363778233528,
+            0.02249353751540184,
+            0.04160059615969658,
+            -0.0004712956433650106,
+            0.012035287916660309,
+            0.013628926128149033,
+            0.01829363778233528,
+            -0.005058970768004656,
+            -0.010259045287966728,
+            0.002500434871762991,
+            -0.01744701713323593,
+            -0.005166873335838318,
+            -0.004938618279993534,
+            0.028204074129462242,
+            0.00562338437885046,
+            0.005818439181894064,
+            0.00597199285402894,
+            0.0006686847191303968,
+            0.013745129108428955,
+            0.007490929216146469,
+            0.026842841878533363,
+            0.020617693662643433,
+            -0.020783696323633194,
+            -0.011346370913088322,
+            -0.0039674947038292885,
+            0.018061231821775436,
+            0.018360039219260216,
+            -0.00022812576207797974,
+            -0.048307154327631,
+            0.026361430063843727,
+            -0.007295874413102865,
+            0.02486739493906498,
+            -0.00315615045838058,
+            0.021912524476647377,
+            0.01962166838347912,
+            -0.01523086428642273,
+            0.01834343932569027,
+            0.01923985965549946,
+            0.001934983767569065,
+            0.03384821116924286,
+            -0.028054671362042427,
+            -0.0063413516618311405,
+            -0.010416748933494091,
+            -0.005830889567732811,
+            0.01736401580274105,
+            -0.0006453404203057289,
+            -0.028585882857441902,
+            0.0021539016161113977,
+            0.023954372853040695,
+            -0.0046896119602024555,
+            0.022858746349811554,
+            -0.009495426900684834,
+            -0.028287075459957123,
+            -0.026461033150553703,
+            0.001020405674353242,
+            -0.023788368329405785,
+            -0.018924452364444733,
+            0.004780914168804884,
+            0.02850288152694702,
+            0.0039031682536005974,
+            -0.016334790736436844,
+            0.00545323034748435,
+            -0.02118210680782795,
+            0.013786629773676395,
+            0.005399279296398163,
+            -0.001994122751057148,
+            0.01665019802749157,
+            0.006893314886838198,
+            0.0053038266487419605,
+            0.0021497514098882675,
+            0.03335019573569298,
+            -0.01262460183352232,
+            -0.006266649812459946,
+            -0.008549204096198082,
+            0.003048247890546918,
+            0.00037999346386641264,
+            0.011072465218603611,
+            0.02516620233654976,
+            0.0008331324206665158,
+            0.005581883247941732,
+            -0.0003052916727028787,
+            0.017463617026805878,
+            -0.015388567931950092,
+            -0.03881172835826874,
+            0.00280131702311337,
+            -0.006171197630465031,
+            -0.021680118516087532,
+            0.015396867878735065,
+            -0.02885149046778679,
+            -0.007673533633351326,
+            -0.0008206821512430906,
+            0.0013581200037151575,
+            0.031756557524204254,
+            -0.005353628192096949,
+            -0.017347414046525955,
+            0.0014421595260500908,
+            -0.028702085837721825,
+            -0.0030129719525575638,
+            0.009927037172019482,
+            0.024385983124375343,
+            0.019306261092424393,
+            -0.01804463192820549,
+            0.018028030171990395,
+            0.027772463858127594,
+            -0.003533809445798397,
+            -0.023987572640180588,
+            -0.010707256384193897,
+            -0.009935337118804455,
+            0.011570476926863194,
+            0.0023116052616387606,
+            -0.013313517905771732,
+            0.010615954175591469,
+            -0.01075705699622631,
+            -0.0019412089604884386,
+            -0.0010655380319803953,
+            -0.013794929720461369,
+            0.052888862788677216,
+            0.006005193572491407,
+            0.0049220179207623005,
+            -0.03957534581422806,
+            0.005108772311359644,
+            -0.011180367320775986,
+            0.013911132700741291,
+            0.009736132808029652,
+            0.014873955398797989,
+            0.00668165972456336,
+            -0.001169290510006249,
+            0.01557117234915495,
+            -0.011147166602313519,
+            0.013728528283536434,
+            0.005652435123920441,
+            -0.0300799198448658,
+            0.0023302806075662374,
+            0.0006536406581290066,
+            0.0051253726705908775,
+            -0.004789214581251144,
+            -1.1988275218755007e-05,
+            0.007304174825549126,
+            0.025149600580334663,
+            0.014508746564388275,
+            0.008254547603428364,
+            -0.011578776873648167,
+            -0.023057950660586357,
+            0.01508146058768034,
+            0.009495426900684834,
+            -0.000578420062083751,
+            0.002919594757258892,
+            -0.030594531446695328,
+            0.005615084432065487,
+            0.004050496965646744,
+            0.010740457102656364,
+            0.01347122248262167,
+            -0.010815158486366272,
+            0.024385983124375343,
+            -0.015454969368875027,
+            -0.003757914761081338,
+            0.011163766495883465,
+            -0.05494731292128563,
+            0.00810929387807846,
+            0.03867892548441887,
+            0.004399104975163937,
+            -0.027307653799653053,
+            -0.01543836947530508,
+            -0.01910705678164959,
+            -0.000976829556748271,
+            0.0006375589873641729,
+            0.005527932196855545,
+            0.006988767068833113,
+            -0.016832802444696426,
+            0.011213568039238453,
+            -0.01880824938416481,
+            -0.034163616597652435,
+            -0.011869283393025398,
+            0.019605068489909172,
+            0.012940009124577045,
+            -0.023439761251211166,
+            0.018443040549755096,
+            -0.010425048880279064,
+            0.011279969476163387,
+            -0.01247519813477993,
+            -0.032885387539863586,
+            0.003793190699070692,
+            -0.015737175941467285,
+            0.007283424027264118,
+            0.0165173951536417,
+            -0.03159055486321449,
+            0.007839537225663662,
+            0.006611107848584652,
+            -0.013039612211287022,
+            -0.03029572404921055,
+            0.018160834908485413,
+            -0.0019806348718702793,
+            0.019040655344724655,
+            -0.01982087455689907,
+            0.019040655344724655,
+            -0.007590531371533871,
+            -0.00685181375592947,
+            -0.03313439339399338,
+            -0.007042718585580587,
+            -0.010682355612516403,
+            -0.0283866785466671,
+            -0.011977186426520348,
+            -0.009935337118804455,
+            0.02342315949499607,
+            -0.040438566356897354,
+            0.003598136128857732,
+            -0.012915108352899551,
+            0.017679423093795776,
+            0.014135237783193588,
+            -0.023871371522545815,
+            0.020684095099568367,
+            -0.018625644966959953,
+            -0.013628926128149033,
+            -0.006868414115160704,
+            -0.011304870247840881,
+            0.005648285150527954,
+            -0.000889158749487251,
+            -0.004494557622820139,
+            -0.005295526701956987,
+            -0.02270934358239174,
+            -0.03180636093020439,
+            -0.002793016843497753,
+            0.0006531218532472849,
+            -0.0025979620404541492,
+            0.012857007794082165,
+            0.02846967987716198,
+            -0.008453751914203167,
+            -0.0032163269352167845,
+            0.005855789873749018,
+            0.017679423093795776,
+            0.0008793022716417909,
+            0.023107752203941345,
+            -0.015347067266702652,
+            -0.02363896556198597,
+            0.0012605926021933556,
+            0.01574547588825226,
+            -0.010532951913774014,
+            -0.0017357789911329746,
+            0.01489055622369051,
+            -0.006270799785852432,
+            0.003816016251221299,
+            -0.02134811133146286,
+            -0.0288016889244318,
+            -0.0022701043635606766,
+            -0.002444408368319273,
+            -0.015703976154327393,
+            -0.0013435946311801672,
+            -0.012616301886737347,
+            -0.00893931370228529,
+            0.021447712555527687,
+            -0.01493205688893795,
+            -0.01139617245644331,
+            0.01037524826824665,
+            -0.01826043613255024,
+            0.008872912265360355,
+            0.0013321818551048636,
+            0.017696022987365723,
+            0.02066749334335327,
+            -0.002402907470241189,
+            0.011338070966303349,
+            0.013944333419203758,
+            0.0035794605500996113,
+            -0.0056316847912967205,
+            0.014052236452698708,
+            0.010225844569504261,
+            -0.011528975330293179,
+            0.001997235231101513,
+            0.011578776873648167,
+            0.01423483993858099,
+            -0.03095974028110504,
+            -0.022128328680992126,
+            -0.014575148932635784,
+            0.015181062743067741,
+            0.0192730613052845,
+            0.0013487822143360972,
+            -0.002954870695248246,
+            0.0049925693310797215,
+            -0.009113618172705173,
+            0.005171023774892092,
+            0.003899018280208111,
+            -0.0186422448605299,
+            0.004660561215132475,
+            0.0014815854374319315,
+            0.007685983553528786,
+            -0.006100645754486322,
+            0.010723856277763844,
+            -0.003911468666046858,
+            -0.011553876101970673,
+            -0.007163071073591709,
+            -0.01894105225801468,
+            -0.00889781303703785,
+            0.020020078867673874,
+            0.00022086309036239982,
+            -0.02211172878742218,
+            0.005656585097312927,
+            0.013413120992481709,
+            -0.015023359097540379,
+            -0.02334015816450119,
+            0.010026639327406883,
+            -0.0005804951069876552,
+            0.0008767084800638258,
+            0.010267345234751701,
+            -0.011761381290853024,
+            0.016907503828406334,
+            -0.014276341535151005,
+            0.010964562185108662,
+            0.0068227630108594894,
+            -0.04189940169453621,
+            0.010441649705171585,
+            0.019671469926834106,
+            -0.01224279310554266,
+            -0.01829363778233528,
+            0.0008450639434158802,
+            0.0277392640709877,
+            -0.0012159791076555848,
+            0.011935685761272907,
+            0.019986877217888832,
+            -0.0007024043006822467,
+            -0.012317494489252567,
+            -0.004851466044783592,
+            0.0918997973203659,
+            0.016309890896081924,
+            0.0136621268466115,
+            0.010200943797826767,
+            0.011836082674562931,
+            -0.030129719525575638,
+            -0.007225322537124157,
+            -0.008906112983822823,
+            -0.010823458433151245,
+            -0.017098408192396164,
+            -0.014434045180678368,
+            0.006777111906558275,
+            0.012790605425834656,
+            0.015919780358672142,
+            0.015952982008457184,
+            -0.00895591452717781,
+            0.007781436201184988,
+            0.008204746060073376,
+            0.0037662149406969547,
+            0.013031311333179474,
+            -0.0056316847912967205,
+            -0.0003042541502509266,
+            -0.011230168864130974,
+            0.010939661413431168,
+            0.014417444355785847,
+            0.011230168864130974,
+            -0.00262701278552413,
+            0.004590009804815054,
+            0.02500019781291485,
+            0.00793083943426609,
+            0.016060883179306984,
+            -0.016002783551812172,
+            0.0045941597782075405,
+            -0.013189014978706837,
+            0.040903378278017044,
+            0.015255765058100224,
+            0.0037765903398394585,
+            -0.005241575185209513,
+            0.0122842937707901,
+            0.004681312013417482,
+            0.018243836238980293,
+            -0.014940356835722923,
+            0.013545923866331577,
+            -0.009453926235437393,
+            -0.0038616673555225134,
+            0.03067753277719021,
+            -0.006677509751170874,
+            0.0029112945776432753,
+            -0.014093737117946148,
+            0.004560959059745073,
+            0.007980640977621078,
+            0.003305553924292326,
+            0.004830715712159872,
+            0.016633598133921623,
+            -0.01894105225801468,
+            -0.011138866655528545,
+            0.020551292225718498,
+            -0.013637226074934006,
+            -0.0017949179746210575,
+            0.017613021656870842,
+            0.006100645754486322,
+            -0.0013819830492138863,
+            0.010856659151613712,
+            -0.012334095314145088,
+            -0.007358125876635313,
+            0.008433002047240734,
+            0.006183648016303778,
+            4.756402631755918e-05,
+            0.005577733274549246,
+            0.024452384561300278,
+            0.005320427007973194,
+            -0.017231212928891182,
+            0.012699303217232227,
+            0.0008367637638002634,
+            0.010483150370419025,
+            0.017928428947925568,
+            0.011952285654842854,
+            0.01948886550962925,
+            -0.015977881848812103,
+            0.00719212181866169,
+            0.006100645754486322,
+            0.01572057604789734,
+            -0.02689264342188835,
+            -0.006992917042225599,
+            -0.0011672154068946838,
+            -0.0017814300954341888,
+            0.012715904042124748,
+            0.014840754680335522,
+            -0.00022125216491986066,
+            0.008972514420747757,
+            -0.0024278080090880394,
+            -0.010408448986709118,
+            0.02172992005944252,
+            -0.022725943475961685,
+            -0.02588001824915409,
+            -0.024634988978505135,
+            0.0073664262890815735,
+            -0.012558200396597385,
+            0.008304348215460777,
+            -0.005270625930279493,
+            0.009835734963417053,
+            0.004432306159287691,
+            0.015181062743067741,
+            0.002600037259981036,
+            -0.007586381398141384,
+            -0.014176739379763603,
+            0.021414512768387794,
+            -0.0033906311728060246,
+            0.009254720993340015,
+            -0.009960237890481949,
+            0.014301242306828499,
+            0.0038222414441406727,
+            0.00560263404622674,
+            -0.010242444463074207,
+            0.016641898080706596,
+            -0.014774353243410587,
+            0.002942420309409499,
+            0.016832802444696426,
+            -0.01736401580274105,
+            -0.012566500343382359,
+            -0.006719010882079601,
+            -0.011703279800713062,
+            0.006905765272676945,
+            -0.028071271255612373,
+            -0.011869283393025398,
+            0.007723334711045027,
+            -0.018077831715345383,
+            0.0071464707143604755,
+            -0.00963653065264225,
+            -0.016558896750211716,
+            0.0235061626881361,
+            0.027855465188622475,
+            -0.011379571631550789,
+            -0.004123123362660408,
+            -0.0025564611423760653,
+            0.003811866044998169,
+            -0.026660237461328506,
+            0.0006546781514771283,
+            -0.0056192344054579735,
+            -0.008773310109972954,
+            -0.003048247890546918,
+            0.016849402338266373,
+            0.002066749380901456,
+            0.0013435946311801672,
+            0.013462921604514122,
+            -0.006810312625020742,
+            -0.020534690469503403,
+            0.01109736505895853,
+            0.014458945952355862,
+            -0.01762962155044079,
+            0.003143700072541833,
+            0.0009285847190767527,
+            -0.019389264285564423,
+            0.01547987014055252,
+            0.00855750497430563,
+            0.016110684722661972,
+            -0.0004414668073877692,
+            0.013819830492138863,
+            -0.011819482780992985,
+            -0.005349477753043175,
+            -0.008590705692768097,
+            0.025780417025089264,
+            0.00685181375592947,
+            0.015073160640895367,
+            0.010873259976506233,
+            0.006270799785852432,
+            -0.00476431380957365,
+            0.027390655130147934,
+            0.023738566786050797,
+            -0.0007309362408705056,
+            -0.005361928138881922,
+            0.010076440870761871,
+            -0.005851639900356531,
+            0.0027100148145109415,
+            -0.017231212928891182,
+            0.0080511923879385,
+            0.0024713841266930103,
+            0.010400149039924145,
+            0.029465705156326294,
+            0.012151490896940231,
+            -0.014301242306828499,
+            0.02571401558816433,
+            -0.003058623056858778,
+            -0.023987572640180588,
+            -0.00014538316463585943,
+            -0.01425144076347351,
+            -0.009080417454242706,
+            0.010840059258043766,
+            -0.0014691351680085063,
+            -0.015106361359357834,
+            0.015297265723347664,
+            0.01453364733606577,
+            0.009927037172019482,
+            -0.008105143904685974,
+            0.004266302101314068,
+            0.00011510041076689959,
+            0.01315581426024437,
+            0.004901267122477293,
+            0.014326142147183418,
+            -0.004185375291854143,
+            0.0029237449634820223,
+            -5.317314571584575e-05,
+            -0.0014784728409722447,
+            -0.012201291508972645,
+            0.007881038822233677,
+            0.012358995154500008,
+            0.0033034789375960827,
+            0.013562524691224098,
+            0.00297147105447948,
+            -0.015820179134607315,
+            -0.03290198743343353,
+            -0.0009156156447716057,
+            0.010840059258043766,
+            -0.009794234298169613,
+            -0.01141277328133583,
+            0.010723856277763844,
+            0.007208722177892923,
+            0.009005715139210224,
+            0.013288618065416813,
+            0.003759989980608225,
+            -0.002585511887446046,
+            -0.010275646112859249,
+            0.021148905158042908,
+            0.007453578058630228,
+            -0.002770191291347146,
+            -0.0036686877720057964,
+            -0.010134542360901833,
+            -0.009586729109287262,
+            -0.0034196816850453615,
+            -0.014392544515430927,
+            0.013313517905771732,
+            0.007565631065517664,
+            0.007059318944811821,
+            0.011952285654842854,
+            0.016467593610286713,
+            0.0035919109359383583,
+            0.009022315964102745,
+            -0.010690655559301376,
+            -0.006175347603857517,
+            -0.020136281847953796,
+            -0.01850944198668003,
+            -0.014135237783193588,
+            0.0015510995872318745,
+            -0.006864264141768217,
+            -0.0004572890466079116,
+            -0.004614910576492548,
+            0.02131490968167782,
+            -0.011288270354270935,
+            -0.0012854932574555278,
+            -3.089878737227991e-05,
+            -0.008856311440467834,
+            -0.01298151072114706,
+            0.020916501060128212,
+            0.0016662648413330317,
+            0.0157786775380373,
+            0.007395477034151554,
+            0.026046022772789,
+            0.007026118226349354,
+            0.002187102334573865,
+            -0.015106361359357834,
+            -0.02431958168745041,
+            -0.011487474665045738,
+            0.021082503721117973,
+            -0.004193675238639116,
+            -0.0001918124035000801,
+            0.02334015816450119,
+            -0.011827782727777958,
+            0.007167221512645483,
+            -0.015313866548240185,
+            0.030312323942780495,
+            -0.008856311440467834,
+            0.012275993824005127,
+            0.012765705585479736,
+            0.01826043613255024,
+            -0.0018997079459950328,
+            0.01741381548345089,
+            0.0021829521283507347,
+            -0.008997415192425251,
+            -0.008383200503885746,
+            -0.0017679422162473202,
+            -0.021148905158042908,
+            -0.012782305479049683,
+            0.00771918473765254,
+            0.0050299204885959625,
+            -0.0054158796556293964,
+            0.013504423201084137,
+            0.0015355367213487625,
+            -0.0007049980922602117,
+            -0.00982743501663208,
+            0.00670241005718708,
+            -0.0013653826899826527,
+            -0.01523086428642273,
+            -0.0017181411385536194,
+            0.011454273946583271,
+            -0.027174849063158035,
+            -0.008856311440467834,
+            0.010010039433836937,
+            -0.00751997996121645,
+            -0.010433349758386612,
+            -0.00786028802394867,
+            0.0022763293236494064,
+            0.008154945448040962,
+            0.004158399533480406,
+            -0.009022315964102745,
+            0.0026456883642822504,
+            0.0015687375562265515,
+            -0.00035249904613010585,
+            -0.011968886479735374,
+            -0.005710536614060402,
+            -0.00898081436753273,
+            0.019937077537178993,
+            -0.007565631065517664,
+            -0.00580183882266283,
+            0.003598136128857732,
+            -0.0068227630108594894,
+            0.015056559816002846,
+            -0.013902832753956318,
+            -0.007885188795626163,
+            -0.01812763325870037,
+            0.021364711225032806,
+            -0.014683051034808159,
+            -0.00026443914975970984,
+            -0.008034592494368553,
+            -0.013280317187309265,
+            -0.0020646743942052126,
+            -0.0029693960677832365,
+            0.017214611172676086,
+            -0.0024278080090880394,
+            0.0003117762098554522,
+            -0.010516351088881493,
+            0.002209927886724472,
+            0.00788933876901865,
+            -0.013064512051641941,
+            0.0020677868742495775,
+            -0.0011402397649362683,
+            0.006241749040782452,
+            -0.007627882529050112,
+            -0.012051887810230255,
+            -0.007997240871191025,
+            0.019156858325004578,
+            0.012010387144982815,
+            -0.00773163465783,
+            -0.017081808298826218,
+            -0.007154771126806736,
+            0.013628926128149033,
+            0.014102037064731121,
+            0.005565282888710499,
+            -0.033649004995822906,
+            -0.003977870102971792,
+            -0.004859765991568565,
+            0.009121918119490147,
+            -0.022443735972046852,
+            -0.0074867792427539825,
+            -0.005270625930279493,
+            -0.02363896556198597,
+            -0.0010634629288688302,
+            0.010898160748183727,
+            0.006474155001342297,
+            -0.007470178417861462,
+            -0.009121918119490147,
+            -0.008291898295283318,
+            0.009951937943696976,
+            -0.01198548637330532,
+            0.0051959240809082985,
+            -0.0031416250858455896,
+            0.005805988796055317,
+            0.02126510813832283,
+            -0.018609045073390007,
+            0.0034093065187335014,
+            -0.005025770049542189,
+            -0.00947052612900734,
+            0.006710710469633341,
+            -0.007104970049113035,
+            -0.014649850316345692,
+            -0.006743911188095808,
+            -0.009495426900684834,
+            0.0004225319717079401,
+            0.007042718585580587,
+            0.0013684951700270176,
+            -0.028021469712257385,
+            0.004258001688867807,
+            -0.003836766816675663,
+            0.014832454733550549,
+            -0.00011587855260586366,
+            -0.008881212212145329,
+            0.011960585601627827,
+            -0.002284629736095667,
+            0.0050299204885959625,
+            0.012516699731349945,
+            -0.01071555633097887,
+            -0.023190755397081375,
+            -0.006868414115160704,
+            -0.010076440870761871,
+            0.005067271180450916,
+            -0.004361754283308983,
+            0.022941747680306435,
+            -0.03535884618759155,
+            0.014085437171161175,
+            -0.0074494280852377415,
+            -0.006528106052428484,
+            0.006972166709601879,
+            -0.005345327779650688,
+            0.027722662314772606,
+            0.00489296717569232,
+            0.008433002047240734,
+            -0.003654162399470806,
+            -0.008549204096198082,
+            -0.020385287702083588,
+            0.02443578466773033,
+            0.010574452579021454,
+            -0.0014286716468632221,
+            0.0023468811996281147,
+            0.006760511547327042,
+            0.011562176048755646,
+            0.024468984454870224,
+            -0.024717990309000015,
+            -0.007926689460873604,
+            -0.018111033365130424,
+            -0.013255417346954346,
+            -0.008416401222348213,
+            -0.011113965883851051,
+            -0.0006640158826485276,
+            -0.007237772922962904,
+            -0.0021310760639607906,
+            -0.0026228628121316433,
+            -0.0015479869907721877,
+            -0.004270452074706554,
+            -0.0014743227511644363,
+            -0.003079373622313142,
+            -0.015737175941467285,
+            -0.003106349380686879,
+            0.005224974825978279,
+            0.00948712695389986,
+            -0.004847316071391106,
+            0.0009778671665117145,
+            0.01056615263223648,
+            0.014467245899140835,
+            -0.007237772922962904,
+            -0.021464312449097633,
+            0.02385476976633072,
+            -0.010682355612516403,
+            0.000497233762871474,
+            -0.013238816522061825,
+            -0.009121918119490147,
+            -0.0034508074168115854,
+            0.0024402583949267864,
+            -0.0032267021015286446,
+            -0.015239164233207703,
+            -0.0031810509972274303,
+            -0.011421073228120804,
+            -0.016177086159586906,
+            0.01648419350385666,
+            0.03278578445315361,
+            -0.0015230864519253373,
+            0.016509095206856728,
+            -1.2742101716867182e-05,
+            0.010433349758386612,
+            -0.004805814940482378,
+            0.0005177248967811465,
+            0.003600211115553975,
+            -0.009578429162502289,
+            -0.004946918226778507,
+            0.007511679548770189,
+            -0.004021446220576763,
+            -0.006727310828864574,
+            -0.00961992982774973,
+            -0.017231212928891182,
+            -0.00893931370228529,
+            -0.005440779961645603,
+            -0.009263021871447563,
+            0.012350695207715034,
+            0.004345153924077749,
+            -0.008781610056757927,
+            0.014824154786765575,
+            0.00424555130302906,
+            -0.007034418173134327,
+            -0.011080765165388584,
+            -0.00013040390331298113,
+            0.0230911523103714,
+            0.023140953853726387,
+            -0.00809269305318594,
+            -0.026444431394338608,
+            0.01593638025224209,
+            -0.025597812607884407,
+            -0.004365904256701469,
+            -0.011371271684765816,
+            -0.01012624241411686,
+            -0.03838011622428894,
+            0.0087899100035429,
+            0.011180367320775986,
+            0.01635139063000679,
+            0.002259728964418173,
+            -0.0031167245469987392,
+            0.014608349651098251,
+            -0.008831411600112915,
+            -0.016235187649726868,
+            -0.004556809086352587,
+            0.003867892548441887,
+            -0.008706907741725445,
+            0.015089760534465313,
+            0.005436629988253117,
+            0.007793886587023735,
+            -0.0019754471722990274,
+            0.002940345322713256,
+            0.0036769879516214132,
+            0.021298309788107872,
+            0.004332703538239002,
+            0.004685461986809969,
+            0.0035836107563227415,
+            -0.0004712956433650106,
+            -0.010931361466646194,
+            0.007142320740967989,
+            -0.0013446321245282888,
+            -0.007055168505758047,
+            -0.00889781303703785,
+            0.030743934214115143,
+            0.02948230504989624,
+            0.010649154894053936,
+            -0.008266997523605824,
+            0.009495426900684834,
+            0.00908871740102768,
+            -0.015015059150755405,
+            0.033233996480703354,
+            -0.004955218639224768,
+            0.0014608348719775677,
+            0.012649502605199814,
+            0.016849402338266373,
+            -0.008287748321890831,
+            -0.006374552380293608,
+            0.007295874413102865,
+            -0.007395477034151554,
+            0.006988767068833113,
+            0.025946419686079025,
+            0.007706734351813793,
+            -0.004287052433937788,
+            -0.030528130009770393,
+            0.0038637423422187567,
+            -0.0031623756512999535,
+            -0.009478827007114887,
+            -0.005328727420419455,
+            0.01209338940680027,
+            -0.011670079082250595,
+            0.007818787358701229,
+            0.008466202765703201,
+            -0.0101677430793643,
+            0.0050880215130746365,
+            -0.00849110260605812,
+            0.0030752234160900116,
+            -0.01527236495167017,
+            -0.01654229499399662,
+            -0.005428329575806856,
+            0.01258310116827488,
+            -0.003857517149299383,
+            -0.008831411600112915,
+            0.012541599571704865,
+            0.01330521795898676,
+            0.01796162873506546,
+            -0.006864264141768217,
+            -0.0028677184600383043,
+            0.00908871740102768,
+            0.025863418355584145,
+            0.0016901278868317604,
+            -0.011852683499455452,
+            0.0013799079461023211,
+            0.008848011493682861,
+            0.017397215589880943,
+            0.007291724439710379,
+            -0.004818265326321125,
+            -0.01330521795898676,
+            -0.01741381548345089,
+            -0.001454609795473516,
+            0.003328379476442933,
+            -0.008565804921090603,
+            0.01103926356881857,
+            0.015056559816002846,
+            -0.009155118837952614,
+            0.007773135788738728,
+            0.0006463779718615115,
+            0.015961281955242157,
+            -0.022061927244067192,
+            -0.026212027296423912,
+            0.013197315856814384,
+            -0.003984095063060522,
+            -0.02028568461537361,
+            -0.00384506699629128,
+            0.001266817795112729,
+            0.022393936291337013,
+            -0.00034731143387034535,
+            -0.00038621859857812524,
+            0.00439495500177145,
+            -0.007138170767575502,
+            0.011006062850356102,
+            -0.02888469025492668,
+            -0.004490407183766365,
+            -3.7837231502635404e-05,
+            -0.0040878476575016975,
+            0.00018014023953583091,
+            -0.014898856170475483,
+            0.006727310828864574,
+            0.01733081415295601,
+            -0.008889512158930302,
+            0.01826043613255024,
+            -0.004697912372648716,
+            0.024070575833320618,
+            0.010607653297483921,
+            -0.0012657803017646074,
+            -0.0052332752384245396,
+            0.001739929080940783,
+            -0.028370078653097153,
+            0.004436456132680178,
+            -0.007358125876635313,
+            -0.010342047549784184,
+            -0.00489296717569232,
+            -0.024087175726890564,
+            0.013587424531579018,
+            0.006893314886838198,
+            0.018775049597024918,
+            0.004552658647298813,
+            0.008773310109972954,
+            0.020352086052298546,
+            -0.010856659151613712,
+            -0.0006178460316732526,
+            0.0036292618606239557,
+            0.012674403376877308,
+            0.0069182151928544044,
+            -0.004519457928836346,
+            -0.024800993502140045,
+            -0.030561331659555435,
+            0.0040359715931117535,
+            0.0016081634676083922,
+            -0.00978593435138464,
+            -0.0022763293236494064,
+            -0.011113965883851051,
+            0.0034508074168115854,
+            -0.02749025821685791,
+            -0.010250745341181755,
+            0.007312474772334099,
+            0.030146321281790733,
+            -0.006793712265789509,
+            -0.0005296564195305109,
+            -0.020518090575933456,
+            0.002039773855358362,
+            -0.02699224464595318,
+            -0.014268041588366032,
+            0.008320949040353298,
+            -0.00786028802394867,
+            0.021962326020002365,
+            -0.02999691665172577,
+            -0.001976484665647149,
+            -0.0046066101640462875,
+            -0.011852683499455452,
+            0.004560959059745073,
+            -0.013296918012201786,
+            -0.0063413516618311405,
+            0.0033470550552010536,
+            0.004174999892711639,
+            -0.00864050630480051,
+            0.016998806968331337,
+            -0.018841451033949852,
+            -0.012549900449812412,
+            -0.00456510903313756,
+            -0.01711500994861126,
+            -0.003201801562681794,
+            0.015712276101112366,
+            -0.007528279908001423,
+            0.005590183660387993,
+            0.017812225967645645,
+            0.0006666096742264926,
+            -0.007084219250828028,
+            -0.02672663889825344,
+            0.01593638025224209,
+            0.001115339226089418,
+            -0.004038046579807997,
+            -0.0005029401509091258,
+            0.02833687700331211,
+            0.00314577529206872,
+            0.007590531371533871,
+            0.014027335681021214,
+            -0.006528106052428484,
+            -0.017148209735751152,
+            0.008814810775220394,
+            -0.002479684306308627,
+            -0.01880824938416481,
+            0.010350347496569157,
+            -9.426885299035348e-06,
+            -0.004631510935723782,
+            -0.003811866044998169,
+            -0.016940705478191376,
+            -0.0064492542296648026,
+            -0.012466898187994957,
+            -0.0015282740350812674,
+            0.015330466441810131,
+            -0.005735436920076609,
+            -0.00933772325515747,
+            -0.014467245899140835,
+            -0.021829521283507347,
+            0.018459642305970192,
+            -0.011960585601627827,
+            8.036764484131709e-06,
+            -0.0022161530796438456,
+            -0.006457554176449776,
+            -0.0023323558270931244,
+            -0.012491798959672451,
+            0.000848176539875567,
+            -0.0028635684866458178,
+            -0.02164691686630249,
+            -0.0032059517689049244,
+            0.020318886265158653,
+            0.02850288152694702,
+            0.005743737332522869,
+            -2.742956166912336e-05,
+            -0.010765357874333858,
+            -0.02669343911111355,
+            0.01406053639948368,
+            0.009180019609630108,
+            -0.020435089245438576,
+            -0.01509806141257286,
+            0.01706520840525627,
+            0.0012720053782686591,
+            -0.005573583301156759,
+            -0.02558121085166931,
+            0.0032785783987492323,
+            0.003392706159502268,
+            -0.03125854581594467,
+            -0.0028635684866458178,
+            0.008823110722005367,
+            7.243220170494169e-05,
+            -0.01434274297207594,
+            0.016367990523576736,
+            -0.0017606796463951468,
+            0.011387872509658337,
+            0.007266823668032885,
+            0.0157786775380373,
+            -0.03678647801280022,
+            -0.033233996480703354,
+            0.013844731263816357,
+            0.010516351088881493,
+            -0.018526043742895126,
+            -0.011504075489938259,
+            0.013653826899826527,
+            -0.0002509512996766716,
+            -0.008125894702970982,
+            -0.0022618041839450598,
+            0.01266610249876976,
+            -0.03452882543206215,
+            -0.021447712555527687,
+            -0.009188319556415081,
+            0.011113965883851051,
+            0.002952795708552003,
+            0.011105665937066078,
+            0.014649850316345692,
+            -0.014110337011516094,
+            -0.010043240152299404,
+            -0.03894453123211861,
+            -0.004772614222019911,
+            0.009163418784737587,
+            -0.01733081415295601,
+            -0.0095286276191473,
+            0.003585685743018985,
+            -0.004751863423734903,
+            0.0084205511957407,
+            0.008881212212145329,
+            0.014525347389280796,
+            -0.002135226037353277,
+            0.015181062743067741,
+            0.0007112232269719243,
+            0.01943906396627426,
+            0.01283210702240467,
+            0.002712089801207185,
+            0.019854074344038963,
+            0.0007480554049834609,
+            -0.003438357263803482,
+            -0.004258001688867807,
+            0.002369706751778722,
+            0.009918737225234509,
+            -0.003120874520391226,
+            0.00016457737365271896,
+            0.020152881741523743,
+            0.007968190126121044,
+            0.009769333526492119,
+            -0.0021891773212701082,
+            0.03545844927430153,
+            -0.011769681237637997,
+            0.005814289208501577,
+            -0.005897291004657745,
+            -0.015513070859014988,
+            0.020833497866988182,
+            0.012765705585479736,
+            0.0016413643024861813,
+            0.018326837569475174,
+            0.00528307631611824,
+            0.0007226360030472279,
+            -0.00543248001486063,
+            -0.022227931767702103,
+            0.0013591574970632792,
+            0.017596419900655746,
+            -0.007698433939367533,
+            0.006096495781093836,
+            -0.004536058288067579,
+            0.023406559601426125,
+            0.0005312127177603543,
+            0.006897464860230684,
+            0.016417792066931725,
+            -0.006067445036023855,
+            0.003957119770348072,
+            -0.0006334088975563645,
+            0.0022784045431762934,
+            -0.011329771019518375,
+            -0.003199726575985551,
+            0.016318190842866898,
+            0.0010800632881000638,
+            0.0020034604240208864,
+            0.004614910576492548,
+            0.0002532857470214367,
+            0.0009825360029935837,
+            0.002994296606630087,
+            -0.0062085483223199844,
+            0.010101341642439365,
+            -0.0068601141683757305,
+            -0.006934816017746925,
+            -0.017762424424290657,
+            0.0063164508901536465,
+            0.029764512553811073,
+            -0.01050805114209652,
+            -0.02016948163509369,
+            0.0039052434731274843,
+            -0.005556982941925526,
+            0.003988245502114296,
+            -0.0008751521818339825,
+            -0.005025770049542189,
+            -0.01787862740457058,
+            -0.009121918119490147,
+            0.007914239540696144,
+            -0.0018135933205485344,
+            0.018575843423604965,
+            0.029681509360671043,
+            0.005673185456544161,
+            -7.55447763367556e-05,
+            0.009595029056072235,
+            0.004598309751600027,
+            0.001292755943723023,
+            0.0017752049025148153,
+            0.004760163836181164,
+            0.006922365631908178,
+            -0.020451689139008522,
+            -0.01001833938062191,
+            -0.019754473119974136,
+            -0.003938443958759308,
+            -0.009113618172705173,
+            -0.00839150045067072,
+            0.00650320528075099,
+            -0.002766041085124016,
+            -0.0017285163048654795,
+            0.0005358815542422235,
+            -0.0019080081256106496,
+            0.01054955180734396,
+            -0.019754473119974136,
+            -0.0054158796556293964,
+            -0.021779721602797508,
+            -0.004482107236981392,
+            0.003309704130515456,
+            -0.008354149758815765,
+            -0.016525695100426674,
+            -0.0038077160716056824,
+            0.03320079296827316,
+            -0.017613021656870842,
+            -0.00844960194081068,
+            -0.0062583498656749725,
+            0.0072585237212479115,
+            0.00927962176501751,
+            -0.02071729488670826,
+            0.0012356920633465052,
+            -0.00017806519463192672,
+            -0.01277400553226471,
+            -0.007851988077163696,
+            -0.017945028841495514,
+            -0.02634483017027378,
+            0.004818265326321125,
+            -0.0011962661519646645,
+            -0.01749681867659092,
+            -0.008947613649070263,
+            -0.015687374398112297,
+            -0.006494905333966017,
+            0.00843715202063322,
+            -0.024087175726890564,
+            0.010151143185794353,
+            0.012043587863445282,
+            -0.004193675238639116,
+            0.003143700072541833,
+            0.0063538020476698875,
+            0.016509095206856728,
+            0.005648285150527954,
+            0.005864090286195278,
+            -0.01425144076347351,
+            -0.017314214259386063,
+            0.0022244532592594624,
+            0.005710536614060402,
+            0.014367643743753433,
+            -0.004901267122477293,
+            0.0059180413372814655,
+            0.007980640977621078,
+            0.0020491115283221006,
+            0.01404393557459116,
+            0.025315605103969574,
+            -0.020302286371588707,
+            -0.01065745484083891,
+            0.0044987075962126255,
+            0.015396867878735065,
+            0.003959194757044315,
+            0.007138170767575502,
+            -0.014525347389280796,
+            -0.009935337118804455,
+            -0.001842644065618515,
+            0.007017817813903093,
+            -0.017596419900655746,
+            0.021829521283507347,
+            -0.00685181375592947,
+            0.02546500973403454,
+            0.014757752418518066,
+            0.02282554656267166,
+            0.017546620219945908,
+            -0.010914760641753674,
+            8.55309481266886e-05,
+            -0.0015376117080450058,
+            0.01542176865041256,
+            -0.008362449705600739,
+            -0.006768811959773302,
+            0.0028822438325732946,
+            0.009312822483479977,
+            -0.004901267122477293,
+            0.009379223920404911,
+            0.016807902604341507,
+            -0.005366078112274408,
+            0.0011962661519646645,
+            -0.014351042918860912,
+            -0.004197825212031603,
+            -0.010350347496569157,
+            0.0029361951164901257,
+            0.0033034789375960827,
+            -0.022045327350497246,
+            0.011910784989595413,
+            0.013545923866331577,
+            0.0024859094992280006,
+            -0.026909243315458298,
+            0.0063164508901536465,
+            0.0032578278332948685,
+            0.002357256365939975,
+            0.0122842937707901,
+            0.027224650606513023,
+            0.00406087189912796,
+            -0.006756361573934555,
+            -0.003811866044998169,
+            0.006868414115160704,
+            0.012126590125262737,
+            0.00263738795183599,
+            -0.006793712265789509,
+            -0.021298309788107872,
+            -0.009254720993340015,
+            -0.028917891904711723,
+            0.002919594757258892,
+            -0.026411231607198715,
+            0.0005919078830629587,
+            0.006432653870433569,
+            -0.009702932089567184,
+            -0.024419184774160385,
+            0.01842644065618515,
+            0.016152186319231987,
+            0.0017088033491745591,
+            0.009354324080049992,
+            -0.015703976154327393,
+            -0.00526232598349452,
+            0.015737175941467285,
+            -0.009362624026834965,
+            -0.011869283393025398,
+            -0.029714711010456085,
+            0.009503726847469807,
+            0.0017129534389823675,
+            0.0038741177413612604,
+            -0.010350347496569157,
+            0.0015791127225384116,
+            -0.014757752418518066,
+            0.006196097936481237,
+            -0.0008502515847794712,
+            -0.017247812822461128,
+            0.010200943797826767,
+            0.00368528813123703,
+            0.003071073442697525,
+            -0.003888642881065607,
+            -0.0014660225715488195,
+            0.010275646112859249,
+            0.016658497974276543,
+            -0.0032931037712842226,
+            -0.009960237890481949,
+            -0.0021165506914258003,
+            0.014409144409000874,
+            -0.005781088024377823,
+            0.016450993716716766,
+            -0.0011371271684765816,
+            0.0053162770345807076,
+            -0.003025422338396311,
+            0.002425733022391796,
+            0.0144008444622159,
+            -0.0005436629871837795,
+            0.007864437997341156,
+            0.019588468596339226,
+            -0.006474155001342297,
+            -0.01347122248262167,
+            0.00961992982774973,
+            -0.010649154894053936,
+            0.015977881848812103,
+            0.0031935013830661774,
+            -0.009387524798512459,
+            -0.01018434390425682,
+            0.014683051034808159,
+            0.023987572640180588,
+            -0.022061927244067192,
+            0.009221520274877548,
+            0.009412424638867378,
+            0.004104448016732931,
+            -0.01787862740457058,
+            0.007279274053871632,
+            -0.007075919304043055,
+            -0.0035939859226346016,
+            0.01018434390425682,
+            0.00889781303703785,
+            0.002546085976064205,
+            0.021547315642237663,
+            -0.011371271684765816,
+            -0.0012004162417724729,
+            -0.005573583301156759,
+            0.010840059258043766,
+            -0.017978230491280556,
+            0.0012720053782686591,
+            -0.00913021806627512,
+            -0.004436456132680178,
+            0.005212524440139532,
+            0.018227236345410347,
+            -0.011960585601627827,
+            -0.005980292800813913,
+            0.003371955594047904,
+            -0.014226539991796017,
+            -0.010068140923976898,
+            -0.00997683871537447,
+            -0.0044032554142177105,
+            0.008665407076478004,
+            0.0074369776993989944,
+            -0.018443040549755096,
+            -0.010624254122376442,
+            0.0008336511673405766,
+            -0.02159711718559265,
+            0.0036168114747852087,
+            -0.031109143048524857,
+            0.015322166495025158,
+            -0.0021705019753426313,
+            -0.006059145089238882,
+            -0.0023738567251712084,
+            0.018227236345410347,
+            -0.005540382582694292,
+            0.0023427309934049845,
+            0.0068725645542144775,
+            0.004569259472191334,
+            0.01659209653735161,
+            0.01523086428642273,
+            -0.010516351088881493,
+            -0.01207678858190775,
+            0.025348806753754616,
+            -0.007594681810587645,
+            0.01243369746953249,
+            -0.03698568418622017,
+            0.004237251356244087,
+            -0.014541947282850742,
+            -0.0031395500991493464,
+            -0.018144233152270317,
+            0.013936033472418785,
+            0.003759989980608225,
+            0.005171023774892092,
+            -0.0026436131447553635,
+            0.013338418677449226,
+            0.008715208619832993,
+            -0.008906112983822823,
+            0.008117593824863434,
+            -0.00849110260605812,
+            0.012840406969189644,
+            -0.024883994832634926,
+            0.005025770049542189,
+            0.012383895926177502,
+            -0.0011682529002428055,
+            0.007229472976177931,
+            -0.01662529818713665,
+            0.004868066404014826,
+            0.00908871740102768,
+            -0.021331509575247765,
+            0.02197892591357231,
+            -0.008354149758815765,
+            -0.00166522734798491,
+            -0.020783696323633194,
+            -0.009844035841524601,
+            0.014367643743753433,
+            -0.0006354839424602687,
+            -0.012508398853242397,
+            0.015579472295939922,
+            0.019771073013544083,
+            0.0005976142711006105,
+            0.023057950660586357,
+            -0.017297614365816116,
+            0.010408448986709118,
+            -0.0017492668703198433,
+            -0.00029595394153147936,
+            -0.0074867792427539825,
+            0.0122842937707901,
+            0.0015116736758500338,
+            -0.0006769849569536746,
+            -0.01419333927333355,
+            -0.005133672617375851,
+            -0.013786629773676395,
+            0.02885149046778679,
+            0.02312435209751129,
+            -0.0006583094946108758,
+            0.0031706758309155703,
+            -0.0038077160716056824,
+            -0.0034279818646609783,
+            0.004370054695755243,
+            -0.01914025843143463,
+            -0.01965487003326416,
+            -0.0036707627587020397,
+            -0.013819830492138863,
+            0.018526043742895126,
+            0.018775049597024918,
+            -0.006901614833623171,
+            0.019721271470189095,
+            -0.013745129108428955,
+            0.013778329826891422,
+            -0.0066443090327084064,
+            -0.010051540099084377,
+            -0.003716413863003254,
+            0.002954870695248246,
+            -0.0063164508901536465,
+            0.025149600580334663,
+            0.008233796805143356,
+            -0.0230911523103714,
+            0.013902832753956318,
+            -0.010732156224548817,
+            -0.015703976154327393,
+            0.0015178987523540854,
+            -0.0177956260740757,
+            0.025216002017259598,
+            0.012840406969189644,
+            0.0003945188072975725,
+            -0.01400243490934372,
+            0.016011083498597145,
+            0.008183995261788368,
+            0.002693414455279708,
+            0.011620277538895607,
+            0.02113230526447296,
+            -8.961620187619701e-05,
+            -0.0177956260740757,
+            -0.0012678552884608507,
+            0.019339462742209435,
+            -0.02224453166127205,
+            -0.004177074879407883,
+            0.010640854015946388,
+            0.017729224637150764,
+            0.009893836453557014,
+            -0.0005239500314928591,
+            -0.0070510185323655605,
+            0.0010364872869104147,
+            -0.010781957767903805,
+            -0.01995367743074894,
+            0.01855924353003502,
+            -0.011329771019518375,
+            0.020916501060128212,
+            0.016251789405941963,
+            0.02148091420531273,
+            -0.005843339487910271,
+            -0.021331509575247765,
+            0.005440779961645603,
+            -0.010541251860558987,
+            -0.010914760641753674,
+            0.026361430063843727,
+            -0.009072116576135159,
+            -0.004407405387610197,
+            0.006050844676792622,
+            -0.012275993824005127,
+            0.013537623919546604,
+            -0.014367643743753433,
+            0.0038844929076731205,
+            -0.013413120992481709,
+            -0.022676141932606697,
+            0.009213220328092575,
+            0.004768464248627424,
+            0.013205615803599358,
+            -0.01902405545115471,
+            -0.01662529818713665,
+            0.006976316682994366,
+            0.01812763325870037,
+            -0.00788933876901865,
+            0.008997415192425251,
+            0.013272017240524292,
+            -0.003542109625414014,
+            -0.010350347496569157,
+            0.037417296320199966,
+            0.004872216377407312,
+            -0.002952795708552003,
+            0.006988767068833113,
+            0.006594507489353418,
+            0.024070575833320618,
+            -0.005710536614060402,
+            0.02410377562046051,
+            -0.019970277324318886,
+            7.839797035558149e-05,
+            -0.00401937123388052,
+            -0.0002860974636860192,
+            0.017214611172676086,
+            -0.01736401580274105,
+            0.017812225967645645,
+            -0.0022618041839450598,
+            0.004469656851142645,
+            0.04777594283223152,
+            -0.012715904042124748,
+            -0.028585882857441902,
+            -0.018409840762615204,
+            -0.025481609627604485,
+            -0.0009156156447716057,
+            -0.001976484665647149,
+            -0.01990387588739395,
+            0.0017326663946732879,
+            -0.0029590206686407328,
+            -0.00722117256373167,
+            0.004938618279993534,
+            0.012334095314145088,
+            -0.009653130546212196,
+            0.021331509575247765,
+            0.020551292225718498,
+            0.013446321710944176,
+            0.006951416376978159,
+            0.005648285150527954,
+            -0.005561132915318012,
+            0.0008476577932015061,
+            0.00026612510555423796,
+            -0.0013684951700270176,
+            -0.03213836997747421,
+            0.02172992005944252,
+            -0.015869978815317154,
+            0.00472281314432621,
+            0.011802881956100464,
+            -0.03358260169625282,
+            -0.004668861627578735,
+            -0.006586207542568445,
+            0.009138518944382668,
+            -0.008167395368218422,
+            0.01817743480205536,
+            -0.016741501167416573,
+            -0.00506312120705843,
+            0.008806510828435421,
+            0.013014711439609528,
+            0.00927962176501751,
+            0.015264065004885197,
+            0.016035983338952065,
+            0.010707256384193897,
+            -0.013728528283536434,
+            0.02101610228419304,
+            -0.019007453694939613,
+            -0.0044032554142177105,
+            0.01957186870276928,
+            -0.012358995154500008,
+            0.016002783551812172,
+            0.022974949330091476,
+            -0.00579353841021657,
+            -0.024551987648010254,
+            0.01593638025224209,
+            0.008914412930607796,
+            0.01435934379696846,
+            0.044887471944093704,
+            -0.0012554050190374255,
+            -0.01689920388162136,
+            -0.027324253693223,
+            -0.02224453166127205,
+            -0.01105586439371109,
+            0.013911132700741291,
+            -0.003187276190146804,
+            0.0033657304011285305,
+            0.02194572426378727,
+            -0.0030316475313156843,
+            -0.014840754680335522,
+            0.011869283393025398,
+            0.001804255647584796,
+            0.0643763393163681,
+            0.004258001688867807,
+            -0.005681485868990421,
+            0.018775049597024918,
+            -0.0062583498656749725,
+            -0.012217892333865166,
+            0.006561306770890951,
+            -0.0063164508901536465,
+            -0.012906808406114578,
+            -0.01850944198668003,
+            0.010134542360901833,
+            -0.023140953853726387,
+            0.004278752487152815,
+            0.004756013862788677,
+            0.0025481609627604485,
+            -0.017513418570160866,
+            -0.010109641589224339,
+            -0.022178130224347115,
+            0.0015490244841203094,
+            -0.017812225967645645,
+            -0.018609045073390007,
+            -0.04027256369590759,
+            -0.016691699624061584,
+            0.016401192173361778,
+            -0.00908871740102768,
+            0.015031659044325352,
+            -0.008453751914203167,
+            -0.010350347496569157,
+            -0.009304522536695004,
+            -0.006308150943368673,
+            -0.005374378524720669,
+            0.014981858432292938,
+            0.025863418355584145,
+            -0.004681312013417482,
+            -0.004469656851142645,
+            0.004419855773448944,
+            -0.013753429055213928,
+            0.011487474665045738,
+            -0.010748757049441338,
+            0.011636878363788128,
+            -0.00386996753513813,
+            0.01889125257730484,
+            0.0003467926580924541,
+            -0.008723508566617966,
+            -0.02778906375169754,
+            -0.012740804813802242,
+            -0.0010925136739388108,
+            -0.00773163465783,
+            0.010300545953214169,
+            -0.005229125265032053,
+            0.003774515353143215,
+            0.0015510995872318745,
+            0.0034217569045722485,
+            -0.020534690469503403,
+            -0.007088369689881802,
+            -0.00034860832965932786,
+            0.0030025967862457037,
+            0.008183995261788368,
+            0.006880864500999451,
+            -0.005627534352242947,
+            0.0035732353571802378,
+            -0.017646221444010735,
+            -0.010649154894053936,
+            0.01580357737839222,
+            -0.013628926128149033,
+            -0.002390457084402442,
+            0.008673707023262978,
+            -0.013844731263816357,
+            -0.026577236130833626,
+            0.009578429162502289,
+            0.011421073228120804,
+            0.00367491296492517,
+            -0.021464312449097633,
+            -0.0003724714042618871,
+            0.012989810667932034,
+            0.0008263884810730815,
+            0.020103080198168755,
+            0.022393936291337013,
+            0.00212277565151453,
+            0.00032215146347880363,
+            0.003147850278764963,
+            0.005839189514517784,
+            -0.028270475566387177,
+            0.024850795045495033,
+            0.022775745019316673,
+            -0.011238468810915947,
+            0.015330466441810131,
+            0.00405464693903923,
+            0.0009856485994532704,
+            -0.01948886550962925,
+            0.0042829024605453014,
+            0.021530715748667717,
+            0.00335120502859354,
+            0.007291724439710379,
+            0.0015095985727384686,
+            0.0011630653170868754,
+            0.017397215589880943,
+            0.0006385965389199555,
+            0.004199900198727846,
+            0.0041832998394966125,
+            -0.020816897973418236,
+            0.00827114749699831,
+            -0.011080765165388584,
+            -0.0028594182804226875,
+            -0.007727484684437513,
+            0.0003626149264164269,
+            0.02699224464595318,
+            0.011213568039238453,
+            -0.014309542253613472,
+            0.014450645074248314,
+            0.011578776873648167,
+            -0.00803874246776104,
+            -0.012043587863445282,
+            -0.0007942252559587359,
+            -0.00632060132920742,
+            0.0041438741609454155,
+            0.017264412716031075,
+            -0.0013487822143360972,
+            0.009702932089567184,
+            0.014492146670818329,
+            -0.00175756704993546,
+            -0.012524999678134918,
+            0.01351272314786911,
+            0.011329771019518375,
+            0.013164115138351917,
+            -0.0073622758500278,
+            -0.0023655565455555916,
+            -0.011761381290853024,
+            -0.007424527313560247,
+            0.005627534352242947,
+            0.0022472788114100695,
+            -0.012466898187994957,
+            -0.007735785096883774,
+            -0.0016880529001355171,
+            0.0009555603610351682,
+            0.011728180572390556,
+            -0.007831237278878689,
+            0.0076652332209050655,
+            -0.003666612785309553,
+            0.012491798959672451,
+            0.000912503048311919,
+            0.04883836954832077,
+            0.005610933993011713,
+            0.003147850278764963,
+            0.0052083744667470455,
+            0.0030772986356168985,
+            0.01728101260960102,
+            -0.008026291616261005,
+            0.024120377376675606,
+            0.0076776836067438126,
+            -0.010599353350698948,
+            -0.026710039004683495,
+            -0.015762077644467354,
+            0.021364711225032806,
+            -0.004739413503557444,
+            -0.001546949497424066,
+            0.019389264285564423,
+            0.01574547588825226,
+            0.011761381290853024,
+            0.023705366998910904,
+            0.00614214688539505,
+            0.013039612211287022,
+            0.0011423148680478334,
+            0.02104930393397808,
+            0.004697912372648716,
+            -0.0027556659188121557,
+            0.016011083498597145,
+            0.009586729109287262,
+            0.019156858325004578,
+            -0.004531908314675093,
+            -0.00043420412112027407,
+            0.019920475780963898,
+            -0.0144008444622159,
+            -0.008665407076478004,
+            0.003533809445798397,
+            -0.009163418784737587,
+            -0.0018540568416938186,
+            0.0026124874129891396,
+            -0.022227931767702103,
+            -0.012018687091767788,
+            -0.007715034298598766,
+            0.00489296717569232,
+            0.008922713808715343,
+            0.0010593128390610218,
+            0.02672663889825344,
+            -0.011479174718260765,
+            0.011977186426520348,
+            0.00929622258991003,
+            0.010192643851041794,
+            -0.005370228551328182,
+            0.002778491470962763,
+            0.0021414512302726507,
+            -0.01762962155044079,
+            -0.00683106342330575,
+            -0.018227236345410347,
+            0.007407926954329014,
+            0.01169497985392809,
+            0.026876043528318405,
+            -0.01489055622369051,
+            -0.001508561079390347,
+            -0.011089065112173557,
+            0.010059840977191925,
+            0.014840754680335522,
+            0.005527932196855545
+          ],
+          "keyphrases": [
+            "Switch Transformers",
+            "Lower Compute Regimes"
+          ],
+          "title": "D Switch Transformers in Lower Compute Regimes"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "28c8fb51-cda2-44c3-8c8a-865ab1a61972",
+      "type": "next",
+      "source": {
+        "id": "12f04ba2-cf28-4e5a-aaa8-f47613191ec5",
+        "properties": {
+          "page_content": "D Switch Transformers in Lower Compute Regimes 29 ",
+          "embedding": [
+            0.023223955184221268,
+            0.008607305586338043,
+            -0.01453364733606577,
+            0.043360237032175064,
+            -0.007881038822233677,
+            0.03894453123211861,
+            0.03295178711414337,
+            0.07682663947343826,
+            0.0186422448605299,
+            0.02312435209751129,
+            -0.010591053403913975,
+            -0.009802534244954586,
+            0.008665407076478004,
+            0.01330521795898676,
+            -0.00808024313300848,
+            0.023273756727576256,
+            -0.016957305371761322,
+            0.01791182905435562,
+            0.009520327672362328,
+            -0.021547315642237663,
+            0.02274254336953163,
+            0.002058449201285839,
+            -0.04790874570608139,
+            0.03579045459628105,
+            -0.01648419350385666,
+            0.01867544651031494,
+            -0.016533995047211647,
+            -0.012840406969189644,
+            -0.023522762581706047,
+            0.02939930371940136,
+            -0.0005597446579486132,
+            0.009113618172705173,
+            -0.018758447840809822,
+            -0.025431808084249496,
+            -0.048473160713911057,
+            -0.027888666838407516,
+            -0.008001391775906086,
+            -0.017513418570160866,
+            -0.04266301915049553,
+            0.006719010882079601,
+            -0.04774274304509163,
+            -0.01855924353003502,
+            -0.026710039004683495,
+            0.03499363735318184,
+            0.02893449179828167,
+            0.0012471048394218087,
+            -0.011661779135465622,
+            0.01660039648413658,
+            0.01438424363732338,
+            -0.01910705678164959,
+            -0.009603329934179783,
+            -0.057968586683273315,
+            0.004353453870862722,
+            0.01889125257730484,
+            -0.026079224422574043,
+            0.02126510813832283,
+            0.025049999356269836,
+            0.11447633802890778,
+            -0.006486604921519756,
+            -0.0010707256151363254,
+            0.02028568461537361,
+            -0.02486739493906498,
+            -0.01766282320022583,
+            -0.03291858732700348,
+            -0.028585882857441902,
+            -0.01542176865041256,
+            -0.03071073442697525,
+            0.007947440259158611,
+            0.0005597446579486132,
+            -0.03542524576187134,
+            -0.04130178689956665,
+            -0.015604373067617416,
+            0.010881559923291206,
+            0.015371967107057571,
+            0.010873259976506233,
+            0.010449949651956558,
+            0.0406709723174572,
+            0.034628428518772125,
+            -0.02661043591797352,
+            0.01175308134406805,
+            0.01771262288093567,
+            -0.032221369445323944,
+            0.018625644966959953,
+            0.012201291508972645,
+            0.01673319935798645,
+            -0.001978559885174036,
+            -0.052291251718997955,
+            -0.04757673665881157,
+            0.03821411356329918,
+            0.00491371750831604,
+            -0.021846123039722443,
+            -0.02601282112300396,
+            0.003295178757980466,
+            -0.020401887595653534,
+            -0.008972514420747757,
+            0.04877196624875069,
+            -0.006113096140325069,
+            -0.01459174882620573,
+            0.025398606434464455,
+            0.008989115245640278,
+            0.0039674947038292885,
+            0.016807902604341507,
+            0.0043908050283789635,
+            -0.008997415192425251,
+            0.030146321281790733,
+            -0.011819482780992985,
+            -0.014367643743753433,
+            0.04452226310968399,
+            0.0186422448605299,
+            0.048738766461610794,
+            0.013047912158071995,
+            -0.024551987648010254,
+            -0.022344134747982025,
+            0.02325715683400631,
+            -0.003189351176843047,
+            -0.026710039004683495,
+            0.003627186641097069,
+            -0.012159790843725204,
+            -0.000483227166114375,
+            -0.00844960194081068,
+            0.026378029957413673,
+            0.013056212104856968,
+            -0.008881212212145329,
+            -0.01224279310554266,
+            -0.02101610228419304,
+            0.012201291508972645,
+            0.00771088432520628,
+            -0.02647763304412365,
+            0.006582057569175959,
+            0.01943906396627426,
+            0.0009275471675209701,
+            0.015172762796282768,
+            -0.00734982592985034,
+            -0.034163616597652435,
+            0.01629328913986683,
+            0.00948712695389986,
+            0.035724055022001266,
+            0.04681311920285225,
+            -0.001914233318530023,
+            -0.00859900563955307,
+            -0.0402393639087677,
+            -0.0012087164213880897,
+            0.0311423446983099,
+            0.00649075536057353,
+            -0.05415049567818642,
+            0.02388797141611576,
+            -0.005996893160045147,
+            0.03655407577753067,
+            0.004627360496670008,
+            0.005270625930279493,
+            -0.00020218764257151634,
+            0.03871212527155876,
+            -0.018775049597024918,
+            -0.0005281001213006675,
+            0.07596341520547867,
+            0.040139760822057724,
+            -0.02481759339570999,
+            -0.005204224493354559,
+            -0.0001957031199708581,
+            -0.01050805114209652,
+            0.015521370805799961,
+            0.022427136078476906,
+            -0.030810337513685226,
+            -0.024004174396395683,
+            0.005996893160045147,
+            0.004648111294955015,
+            0.04249701648950577,
+            0.007715034298598766,
+            -0.05577733367681503,
+            0.023622365668416023,
+            0.03731769323348999,
+            0.0220287274569273,
+            0.031374748796224594,
+            0.022559938952326775,
+            -0.004332703538239002,
+            0.01850944198668003,
+            -0.006075745448470116,
+            0.010151143185794353,
+            0.023921171203255653,
+            -0.021032702177762985,
+            -0.032885387539863586,
+            -0.016002783551812172,
+            -0.0109064606949687,
+            -0.017563220113515854,
+            0.003915618639439344,
+            0.011238468810915947,
+            0.02104930393397808,
+            -0.001071763108484447,
+            0.02762306109070778,
+            0.003759989980608225,
+            0.005399279296398163,
+            0.008134194649755955,
+            0.01791182905435562,
+            0.010383548215031624,
+            -0.019638270139694214,
+            -0.04359264299273491,
+            -0.00263116299174726,
+            0.035392045974731445,
+            -0.0014027334982529283,
+            -0.0031416250858455896,
+            -0.00401937123388052,
+            0.008715208619832993,
+            0.020385287702083588,
+            0.01194398570805788,
+            -0.016608696430921555,
+            -0.004839015658944845,
+            0.037550099194049835,
+            -0.016251789405941963,
+            0.009097017347812653,
+            -0.0258468184620142,
+            0.018227236345410347,
+            -0.005399279296398163,
+            0.008931013755500317,
+            0.013272017240524292,
+            -0.0014255590504035354,
+            0.00751997996121645,
+            0.025697413831949234,
+            0.009578429162502289,
+            -0.0377161018550396,
+            -0.01973787136375904,
+            0.021995525807142258,
+            0.00754903070628643,
+            0.023240555077791214,
+            -0.03778250515460968,
+            -0.02330695651471615,
+            0.011454273946583271,
+            0.015637574717402458,
+            -0.002007610397413373,
+            -0.002900919411331415,
+            -0.0402393639087677,
+            -0.038778528571128845,
+            0.006465854588896036,
+            0.014475545845925808,
+            -0.013819830492138863,
+            0.00028843191103078425,
+            -0.02727445214986801,
+            -0.00632060132920742,
+            0.05219164863228798,
+            -0.028287075459957123,
+            0.027888666838407516,
+            0.012334095314145088,
+            0.015313866548240185,
+            -0.01762962155044079,
+            -0.015513070859014988,
+            -0.0150399599224329,
+            0.008806510828435421,
+            -0.015919780358672142,
+            0.0648079514503479,
+            0.03321739286184311,
+            -0.013462921604514122,
+            -0.04116898402571678,
+            -0.01347122248262167,
+            0.002583436667919159,
+            -0.004127273801714182,
+            0.0014608348719775677,
+            0.02964830957353115,
+            -0.003932218998670578,
+            -0.015181062743067741,
+            -0.026129024103283882,
+            0.02016948163509369,
+            0.016102384775877,
+            -0.003091824008151889,
+            -0.030561331659555435,
+            0.029930515214800835,
+            -0.017048608511686325,
+            0.030826937407255173,
+            0.03612246364355087,
+            0.00788933876901865,
+            -0.0036126612685620785,
+            0.004760163836181164,
+            0.00200968561694026,
+            -0.0313415490090847,
+            -0.05806818976998329,
+            0.010325446724891663,
+            0.02085009776055813,
+            -0.032852184027433395,
+            -0.026278428733348846,
+            0.01188588421791792,
+            0.003847141982987523,
+            -0.010026639327406883,
+            0.001663152244873345,
+            0.0011174142127856612,
+            -0.00038388418033719063,
+            -0.007300024386495352,
+            -0.00700121745467186,
+            -0.04126858711242676,
+            -0.0063538020476698875,
+            0.04863916337490082,
+            -0.03834691643714905,
+            -0.0377161018550396,
+            0.005249875597655773,
+            -0.01296490989625454,
+            -0.006594507489353418,
+            0.01120526809245348,
+            0.034163616597652435,
+            -0.03489403426647186,
+            0.05401769280433655,
+            -0.027971668168902397,
+            0.02080029807984829,
+            -0.00859900563955307,
+            -0.015205963514745235,
+            -0.0016185387503355742,
+            0.02825387567281723,
+            -0.02443578466773033,
+            -0.028951091691851616,
+            -0.002340656006708741,
+            0.01588658057153225,
+            0.016044283285737038,
+            0.003556634997949004,
+            0.0351596400141716,
+            0.04485427215695381,
+            0.0056192344054579735,
+            -0.004855616018176079,
+            -0.06620237976312637,
+            -0.025979621335864067,
+            -0.010234144516289234,
+            0.020899899303913116,
+            -0.012400496751070023,
+            -0.011620277538895607,
+            -0.00472281314432621,
+            0.008706907741725445,
+            -0.036155663430690765,
+            -0.05584373697638512,
+            0.05268966034054756,
+            0.02085009776055813,
+            -0.03041192702949047,
+            -0.0025626863352954388,
+            0.026876043528318405,
+            -0.011429373174905777,
+            -0.028320277109742165,
+            -0.017264412716031075,
+            -0.019986877217888832,
+            0.012591401115059853,
+            0.007748235482722521,
+            0.025232603773474693,
+            0.026544034481048584,
+            0.0011475024512037635,
+            -0.020451689139008522,
+            -0.044289860874414444,
+            0.017978230491280556,
+            0.036321669816970825,
+            0.02926649898290634,
+            -0.04004015773534775,
+            -0.03456202521920204,
+            -0.03363240510225296,
+            -0.02186272293329239,
+            0.008798209950327873,
+            -0.007138170767575502,
+            -0.02923329919576645,
+            0.008237946778535843,
+            0.0010442687198519707,
+            -0.007573931012302637,
+            0.014774353243410587,
+            -0.03758329898118973,
+            0.062085483223199844,
+            -0.0478755459189415,
+            -0.02508319914340973,
+            -0.004056721925735474,
+            -0.025182802230119705,
+            -0.04824075475335121,
+            0.010815158486366272,
+            -0.021663518622517586,
+            0.0021912523079663515,
+            -0.024518785998225212,
+            -0.03894453123211861,
+            -0.03834691643714905,
+            0.03426321968436241,
+            -0.008773310109972954,
+            -0.021115705370903015,
+            0.021082503721117973,
+            -0.026129024103283882,
+            -0.0016403266927227378,
+            0.009072116576135159,
+            0.027772463858127594,
+            0.010806858539581299,
+            -0.018360039219260216,
+            0.03758329898118973,
+            -0.012491798959672451,
+            -0.02885149046778679,
+            0.008200596086680889,
+            -0.043526239693164825,
+            0.023605763912200928,
+            -0.04203220456838608,
+            -0.04837355762720108,
+            0.07735785096883774,
+            0.00859900563955307,
+            0.026793040335178375,
+            -0.014351042918860912,
+            -0.04143458977341652,
+            0.0336822047829628,
+            -9.992660488933325e-05,
+            -0.01703200675547123,
+            -0.02063429355621338,
+            -0.012566500343382359,
+            0.0347280316054821,
+            0.021281708031892776,
+            -0.0220287274569273,
+            0.014558548107743263,
+            0.007831237278878689,
+            -0.01910705678164959,
+            -0.005888990592211485,
+            0.013379920274019241,
+            -0.03084353730082512,
+            -0.018609045073390007,
+            0.008823110722005367,
+            0.01296490989625454,
+            -0.06839363276958466,
+            -0.013396520167589188,
+            -0.006573757156729698,
+            -0.00281169218942523,
+            -0.00595124252140522,
+            -0.05162723362445831,
+            0.011852683499455452,
+            -0.013562524691224098,
+            0.02765626087784767,
+            0.0010017302120104432,
+            0.0025128850247710943,
+            0.027805665507912636,
+            -0.022725943475961685,
+            -0.023057950660586357,
+            0.009395824745297432,
+            0.047510337084531784,
+            0.01880824938416481,
+            0.015596073120832443,
+            0.033781807869672775,
+            0.015562872402369976,
+            0.03396441042423248,
+            0.022510137408971786,
+            0.020816897973418236,
+            0.0020958001259714365,
+            0.02377176843583584,
+            -0.0013612326001748443,
+            -0.04163379594683647,
+            -0.009312822483479977,
+            -0.04608270153403282,
+            0.02511640079319477,
+            -0.014691350981593132,
+            0.036155663430690765,
+            -0.012973209843039513,
+            0.0017254037084057927,
+            0.028320277109742165,
+            -0.010383548215031624,
+            -0.02486739493906498,
+            0.020999502390623093,
+            -0.04963518679141998,
+            0.028187474235892296,
+            -0.008200596086680889,
+            0.016168786212801933,
+            -0.0037309392355382442,
+            0.013886231929063797,
+            -0.03469482809305191,
+            0.0258468184620142,
+            -0.013844731263816357,
+            -0.022759143263101578,
+            0.02126510813832283,
+            -0.03851291909813881,
+            0.027772463858127594,
+            0.01624348759651184,
+            -0.015322166495025158,
+            -0.03960854560136795,
+            0.03336679935455322,
+            0.033150993287563324,
+            0.032553378492593765,
+            -0.0347280316054821,
+            0.013338418677449226,
+            0.0423642136156559,
+            -0.05255685746669769,
+            -0.005967842880636454,
+            -0.02685944177210331,
+            0.02426978014409542,
+            -0.01311431359499693,
+            -0.00978593435138464,
+            0.008329248987138271,
+            0.05800178647041321,
+            -0.024087175726890564,
+            -0.0041438741609454155,
+            0.0018260437063872814,
+            -0.07616262137889862,
+            -0.014118637889623642,
+            -0.01370362751185894,
+            0.012425397522747517,
+            0.004959368612617254,
+            0.0026871892623603344,
+            -0.018758447840809822,
+            0.03122534602880478,
+            0.019721271470189095,
+            0.018907852470874786,
+            0.0330347903072834,
+            -0.05036560446023941,
+            0.018758447840809822,
+            -0.005880690645426512,
+            -0.026029422879219055,
+            -0.00212277565151453,
+            -0.0235061626881361,
+            -0.011595377698540688,
+            -0.06328071653842926,
+            -0.044123854488134384,
+            -0.04960198700428009,
+            0.025016797706484795,
+            0.03549164906144142,
+            -0.05109602212905884,
+            -0.021331509575247765,
+            -0.005839189514517784,
+            0.008457901887595654,
+            -0.02227773331105709,
+            -0.03409721702337265,
+            0.006283250171691179,
+            0.0009960237657651305,
+            -0.0025045848451554775,
+            0.0005753075238317251,
+            0.05939621850848198,
+            0.0022472788114100695,
+            0.028917891904711723,
+            -0.022941747680306435,
+            -0.027208050712943077,
+            0.006864264141768217,
+            -0.006191947963088751,
+            0.005486431065946817,
+            0.017463617026805878,
+            -0.007441128138452768,
+            0.006158747244626284,
+            -0.0047975145280361176,
+            0.007466028444468975,
+            0.04621550440788269,
+            -0.005984443239867687,
+            0.023057950660586357,
+            0.018077831715345383,
+            -0.009744432754814625,
+            0.04047176614403725,
+            0.009951937943696976,
+            0.007142320740967989,
+            0.024751191958785057,
+            -0.008947613649070263,
+            -0.021630316972732544,
+            -0.008486952632665634,
+            0.03535884618759155,
+            -0.012035287916660309,
+            -0.007835387252271175,
+            -0.026278428733348846,
+            0.020518090575933456,
+            0.01518936362117529,
+            0.04641471058130264,
+            -0.02219473011791706,
+            -0.009080417454242706,
+            -0.007573931012302637,
+            0.005258175544440746,
+            0.014458945952355862,
+            -0.011869283393025398,
+            0.013645526021718979,
+            0.003253677859902382,
+            -0.01791182905435562,
+            -0.0436258427798748,
+            -0.006768811959773302,
+            -0.009453926235437393,
+            -0.006042544264346361,
+            -0.0072585237212479115,
+            0.0011620278237387538,
+            0.023008150979876518,
+            -0.00014642068708781153,
+            -0.01654229499399662,
+            -0.007428677752614021,
+            0.020833497866988182,
+            0.01635139063000679,
+            0.011346370913088322,
+            0.013886231929063797,
+            -0.00018052931409329176,
+            0.01621858775615692,
+            0.04100298136472702,
+            -0.0070634689182043076,
+            -0.011196968145668507,
+            -0.022095128893852234,
+            0.0177956260740757,
+            -0.00961992982774973,
+            0.004436456132680178,
+            0.00422480097040534,
+            0.05521291866898537,
+            -0.026842841878533363,
+            -0.013944333419203758,
+            -0.015737175941467285,
+            0.022061927244067192,
+            0.00933772325515747,
+            -0.03144115209579468,
+            0.011130565777420998,
+            0.006582057569175959,
+            -0.017994830384850502,
+            -0.024452384561300278,
+            -0.006901614833623171,
+            -0.025747215375304222,
+            0.007187971845269203,
+            -0.0018654696177691221,
+            0.006345501635223627,
+            0.04262981936335564,
+            -0.017015406861901283,
+            -0.01546326931566,
+            0.006598657928407192,
+            0.010889860801398754,
+            0.02159711718559265,
+            -0.010806858539581299,
+            0.022808944806456566,
+            -0.04103618115186691,
+            0.015529671683907509,
+            0.02579701691865921,
+            0.013670426793396473,
+            -0.013230516575276852,
+            0.004448906518518925,
+            0.032984986901283264,
+            -0.02592981979250908,
+            -0.0005654510459862649,
+            -0.002218228066340089,
+            0.009819135069847107,
+            0.03592325747013092,
+            -0.00855750497430563,
+            0.020401887595653534,
+            -0.014168438501656055,
+            -0.017546620219945908,
+            0.024933796375989914,
+            -0.046149104833602905,
+            0.024966996163129807,
+            0.0035524850245565176,
+            -0.01711500994861126,
+            0.035392045974731445,
+            -0.017380615696310997,
+            0.011371271684765816,
+            0.024136977270245552,
+            -0.016177086159586906,
+            0.019040655344724655,
+            0.0055113318376243114,
+            -0.02707524783909321,
+            -0.013985834084451199,
+            -0.0018488692585378885,
+            0.021115705370903015,
+            -0.004357604309916496,
+            -0.013653826899826527,
+            -0.04146779328584671,
+            0.021680118516087532,
+            -0.022725943475961685,
+            0.009927037172019482,
+            -0.020186083391308784,
+            0.020601091906428337,
+            0.004635660909116268,
+            -0.007881038822233677,
+            0.017131609842181206,
+            0.032603178173303604,
+            -0.03213836997747421,
+            -0.019472265616059303,
+            0.013396520167589188,
+            0.012848706915974617,
+            0.02028568461537361,
+            0.007478478830307722,
+            0.020899899303913116,
+            0.03456202521920204,
+            0.009860635735094547,
+            0.0175798200070858,
+            0.017845427617430687,
+            0.013446321710944176,
+            -0.008130044676363468,
+            0.0037952656857669353,
+            -0.007619582116603851,
+            -0.012209592387080193,
+            0.0022970798891037703,
+            -0.0389777310192585,
+            -0.009968538768589497,
+            0.028735287487506866,
+            0.021846123039722443,
+            -0.012168090790510178,
+            -0.033067990094423294,
+            0.000248616881435737,
+            0.02151411399245262,
+            -0.01075705699622631,
+            0.024966996163129807,
+            -0.03502683714032173,
+            -0.0343296192586422,
+            0.006984617095440626,
+            -0.026145625859498978,
+            0.005565282888710499,
+            -0.026544034481048584,
+            -0.00019012641860172153,
+            0.026444431394338608,
+            -0.012217892333865166,
+            0.010350347496569157,
+            -0.01919005811214447,
+            0.020451689139008522,
+            0.008574104867875576,
+            -8.741146302781999e-05,
+            0.012309194542467594,
+            0.007374726235866547,
+            -0.041202183812856674,
+            -0.011429373174905777,
+            -0.02672663889825344,
+            0.03366560488939285,
+            -0.011811182834208012,
+            -0.006113096140325069,
+            -0.009669731371104717,
+            0.013346719555556774,
+            -0.019538667052984238,
+            -0.04189940169453621,
+            0.010823458433151245,
+            0.004980118945240974,
+            0.017231212928891182,
+            -0.021763119846582413,
+            0.008266997523605824,
+            -0.01512296125292778,
+            0.003008821979165077,
+            0.010292246006429195,
+            0.007312474772334099,
+            0.003932218998670578,
+            0.0065447064116597176,
+            0.00808024313300848,
+            -0.01419333927333355,
+            -0.012275993824005127,
+            0.011562176048755646,
+            -0.009570129215717316,
+            -0.00560263404622674,
+            0.06656759232282639,
+            0.03258657827973366,
+            0.0025398607831448317,
+            -0.02320735529065132,
+            -0.01512296125292778,
+            -0.03911053389310837,
+            -0.002927894936874509,
+            0.02661043591797352,
+            0.03057793155312538,
+            -0.01957186870276928,
+            -0.0046066101640462875,
+            -0.012441997416317463,
+            -0.015006759203970432,
+            0.03688608109951019,
+            0.004598309751600027,
+            0.024070575833320618,
+            -0.0029341201297938824,
+            0.0147245517000556,
+            0.009669731371104717,
+            -0.02699224464595318,
+            -0.0002613265533000231,
+            -0.0032702782191336155,
+            -0.006017643958330154,
+            0.012068488635122776,
+            0.03529244288802147,
+            0.0013384070480242372,
+            -0.024468984454870224,
+            0.01668339967727661,
+            0.0003351205086801201,
+            -0.016343090683221817,
+            0.02541520819067955,
+            0.001071763108484447,
+            -0.017181411385536194,
+            0.016500795260071754,
+            0.002089574933052063,
+            -0.017845427617430687,
+            0.039376143366098404,
+            -0.015928080305457115,
+            -0.05544532462954521,
+            0.00670241005718708,
+            -0.0008647768991068006,
+            -0.02694244496524334,
+            0.021464312449097633,
+            0.006914065219461918,
+            -0.00192772108130157,
+            0.01400243490934372,
+            0.014483846724033356,
+            -0.002081274753436446,
+            0.022393936291337013,
+            0.00719212181866169,
+            0.029465705156326294,
+            0.02554801106452942,
+            0.011802881956100464,
+            -0.013263717293739319,
+            -0.044920675456523895,
+            0.013902832753956318,
+            0.05833379551768303,
+            0.0381145104765892,
+            0.031059343367815018,
+            0.01434274297207594,
+            -0.004880516789853573,
+            0.0020750495605170727,
+            0.016284989193081856,
+            0.017148209735751152,
+            -0.03794850781559944,
+            0.0013653826899826527,
+            0.021962326020002365,
+            0.00855750497430563,
+            -0.007627882529050112,
+            -0.02792186848819256,
+            0.001390283228829503,
+            -0.017994830384850502,
+            0.008059492334723473,
+            0.02320735529065132,
+            -0.002274254336953163,
+            0.014624949544668198,
+            0.019173458218574524,
+            -0.00793083943426609,
+            0.016666797921061516,
+            -0.013446321710944176,
+            0.010474850423634052,
+            0.016550594940781593,
+            -0.004197825212031603,
+            -0.021613717079162598,
+            0.01158707682043314,
+            -0.045352283865213394,
+            -0.044289860874414444,
+            0.022642942145466805,
+            0.002205777680501342,
+            -0.020916501060128212,
+            0.004739413503557444,
+            0.00719212181866169,
+            -0.007129870355129242,
+            0.018144233152270317,
+            -0.04037216678261757,
+            0.013371619395911694,
+            -0.001614388544112444,
+            0.031241947785019875,
+            -0.0495687834918499,
+            0.019090456888079643,
+            0.036454472690820694,
+            -0.002919594757258892,
+            -0.01050805114209652,
+            -0.0003117762098554522,
+            -0.006308150943368673,
+            0.00015926006017252803,
+            -0.003587760729715228,
+            0.023024750873446465,
+            0.021215306594967842,
+            0.01321391575038433,
+            -0.023987572640180588,
+            -0.017928428947925568,
+            -0.018658846616744995,
+            0.035093240439891815,
+            -0.008798209950327873,
+            0.026212027296423912,
+            0.046912722289562225,
+            -0.01826043613255024,
+            0.00944562628865242,
+            -0.01071555633097887,
+            -0.01557117234915495,
+            -6.3183638303598855e-06,
+            0.02999691665172577,
+            -0.012026987969875336,
+            0.016152186319231987,
+            -0.03914373740553856,
+            0.010723856277763844,
+            0.018758447840809822,
+            0.013587424531579018,
+            -0.015330466441810131,
+            -0.0317731611430645,
+            -0.016774700954556465,
+            -0.015031659044325352,
+            -0.0003994470462203026,
+            0.006685809697955847,
+            0.03582365810871124,
+            0.05262325704097748,
+            0.04774274304509163,
+            0.024253180250525475,
+            0.008026291616261005,
+            -0.004270452074706554,
+            -0.05879860743880272,
+            0.008399800397455692,
+            0.030212722718715668,
+            0.00123361696023494,
+            -0.019306261092424393,
+            -9.441475413041189e-05,
+            -0.007810486946254969,
+            0.02528240531682968,
+            0.05023280158638954,
+            -0.027855465188622475,
+            0.025232603773474693,
+            -0.0026353129651397467,
+            0.0036126612685620785,
+            0.01194398570805788,
+            0.011852683499455452,
+            -0.04329383373260498,
+            -0.05398448929190636,
+            0.017397215589880943,
+            -0.01243369746953249,
+            -0.0034196816850453615,
+            0.013089412823319435,
+            -0.01782882586121559,
+            -0.024236580356955528,
+            -0.025730615481734276,
+            0.037550099194049835,
+            0.036188866943120956,
+            0.005191774107515812,
+            -0.004502857569605112,
+            -0.013462921604514122,
+            0.00810929387807846,
+            0.03665367513895035,
+            0.022891947999596596,
+            -0.0027535909321159124,
+            -0.0003333048371132463,
+            0.02385476976633072,
+            -0.017895227298140526,
+            -0.006536406464874744,
+            -0.013421420939266682,
+            0.026676837354898453,
+            -0.015537971630692482,
+            0.001002248958684504,
+            0.0018716948106884956,
+            0.007673533633351326,
+            0.01012624241411686,
+            0.009561828337609768,
+            -0.013761729001998901,
+            -0.0043078032322227955,
+            -0.039840951561927795,
+            0.020435089245438576,
+            -0.013180715031921864,
+            0.01867544651031494,
+            -0.010806858539581299,
+            0.0005216156132519245,
+            0.02863568440079689,
+            0.023456361144781113,
+            -0.015347067266702652,
+            -0.010109641589224339,
+            -0.015687374398112297,
+            0.02036868780851364,
+            -0.0073539759032428265,
+            0.028370078653097153,
+            0.003654162399470806,
+            0.0042829024605453014,
+            -0.03046172857284546,
+            0.026195425540208817,
+            0.0037662149406969547,
+            -0.021065903827548027,
+            0.0036375620402395725,
+            0.02481759339570999,
+            0.02765626087784767,
+            -0.019937077537178993,
+            -0.018409840762615204,
+            -0.01621858775615692,
+            0.02008648030459881,
+            -0.0020345861557871103,
+            0.03681968152523041,
+            -0.010425048880279064,
+            0.024170177057385445,
+            0.02194572426378727,
+            -0.025448407977819443,
+            0.0258468184620142,
+            -0.01226769294589758,
+            -0.004320253152400255,
+            0.018360039219260216,
+            -0.013886231929063797,
+            -0.0011091140331700444,
+            -0.027390655130147934,
+            0.016210287809371948,
+            0.00700121745467186,
+            -0.02533220499753952,
+            -0.007055168505758047,
+            0.024767791852355003,
+            -0.011346370913088322,
+            0.029515504837036133,
+            -0.007378876209259033,
+            -0.0032267021015286446,
+            0.0036292618606239557,
+            -0.004519457928836346,
+            -0.01012624241411686,
+            0.009478827007114887,
+            -0.006295700557529926,
+            -0.05833379551768303,
+            -0.021032702177762985,
+            0.024983597919344902,
+            -0.012109989300370216,
+            0.023223955184221268,
+            0.037981707602739334,
+            0.0021373010240495205,
+            -0.02617882564663887,
+            0.0007885188679210842,
+            0.022128328680992126,
+            -0.029034094884991646,
+            -0.01673319935798645,
+            -0.020833497866988182,
+            -0.03781570494174957,
+            -0.0019318711711093783,
+            -0.025299005210399628,
+            -0.006324751302599907,
+            0.04511987790465355,
+            0.007685983553528786,
+            -0.012035287916660309,
+            -0.007084219250828028,
+            0.014873955398797989,
+            -0.0004430231056176126,
+            0.006150446832180023,
+            0.008482802659273148,
+            0.029830913990736008,
+            -0.00192253349814564,
+            0.014774353243410587,
+            0.05484770983457565,
+            0.016467593610286713,
+            -0.012184691615402699,
+            -0.006387002766132355,
+            -0.01867544651031494,
+            -0.003214251948520541,
+            -0.004502857569605112,
+            0.0011827782727777958,
+            -0.02338995970785618,
+            0.04040536656975746,
+            -0.02888469025492668,
+            0.014525347389280796,
+            -0.02224453166127205,
+            0.0020491115283221006,
+            -0.05006679892539978,
+            0.010591053403913975,
+            -0.013803229667246342,
+            -0.02036868780851364,
+            -0.009918737225234509,
+            0.0002780566574074328,
+            0.05185963958501816,
+            0.0004969743895344436,
+            0.01867544651031494,
+            -0.004502857569605112,
+            -0.019206659868359566,
+            -0.015454969368875027,
+            -0.02046828903257847,
+            0.005270625930279493,
+            -6.296478386502713e-05,
+            0.018160834908485413,
+            0.0006251087179407477,
+            -0.0006152522400952876,
+            0.02418677881360054,
+            0.0288016889244318,
+            -0.019306261092424393,
+            -0.027506858110427856,
+            0.05899780988693237,
+            -0.013645526021718979,
+            -0.02579701691865921,
+            -0.02020268328487873,
+            0.005864090286195278,
+            -0.003253677859902382,
+            0.012251093052327633,
+            0.004847316071391106,
+            -0.002434033202007413,
+            -0.00297147105447948,
+            -0.02592981979250908,
+            -0.0004222725983709097,
+            -0.013064512051641941,
+            -0.015338766388595104,
+            -0.022676141932606697,
+            0.0008803398231975734,
+            0.0018696197075769305,
+            0.011670079082250595,
+            -0.008540904149413109,
+            -0.019173458218574524,
+            0.00047259253915399313,
+            -0.0006634971359744668,
+            -0.011138866655528545,
+            0.02792186848819256,
+            -0.012002087198197842,
+            -0.03562445193529129,
+            0.019854074344038963,
+            0.0024547837674617767,
+            0.017729224637150764,
+            0.0129234092310071,
+            0.014409144409000874,
+            0.009047216735780239,
+            0.0032184019219130278,
+            0.01812763325870037,
+            -0.02426978014409542,
+            0.002842817921191454,
+            -0.001082138391211629,
+            0.0030025967862457037,
+            -0.02868548594415188,
+            0.023904571309685707,
+            0.003488158341497183,
+            -0.012350695207715034,
+            -0.0005498881218954921,
+            0.002855268307030201,
+            -0.01867544651031494,
+            0.005129522643983364,
+            0.02524920366704464,
+            -0.0015614748699590564,
+            0.012358995154500008,
+            0.016193687915802002,
+            -0.01258310116827488,
+            0.0029133695643395185,
+            -0.0009591917041689157,
+            -0.005187624134123325,
+            -0.0020449613220989704,
+            0.016475893557071686,
+            0.020302286371588707,
+            0.010192643851041794,
+            -0.012026987969875336,
+            -0.03376520797610283,
+            0.00971953198313713,
+            -0.012566500343382359,
+            0.01673319935798645,
+            -0.007656933274120092,
+            0.018575843423604965,
+            -0.029432503506541252,
+            0.028270475566387177,
+            -0.013238816522061825,
+            -0.00683106342330575,
+            -0.015546271577477455,
+            -0.025016797706484795,
+            -0.023373357951641083,
+            0.011238468810915947,
+            0.025365406647324562,
+            -0.006938965991139412,
+            0.015977881848812103,
+            -0.012085088528692722,
+            -0.005245725624263287,
+            0.06291550397872925,
+            0.01586167886853218,
+            0.012649502605199814,
+            -0.023323558270931244,
+            -0.00913021806627512,
+            -0.01385303121060133,
+            -0.0298641137778759,
+            -0.0007330112857744098,
+            0.025066599249839783,
+            0.02754005789756775,
+            0.014857355505228043,
+            0.008615606464445591,
+            0.011761381290853024,
+            0.009578429162502289,
+            0.0034985337406396866,
+            -0.00613384647294879,
+            -0.009644830599427223,
+            2.2695854568155482e-05,
+            0.00281169218942523,
+            -0.008296048268675804,
+            -0.012184691615402699,
+            -0.0036583123728632927,
+            -0.009287921711802483,
+            0.015197663567960262,
+            -0.016633598133921623,
+            0.006611107848584652,
+            -0.011122265830636024,
+            0.0015365742146968842,
+            -0.019538667052984238,
+            -0.01336331944912672,
+            0.011827782727777958,
+            0.01396093424409628,
+            -0.0007003292557783425,
+            -0.00424555130302906,
+            -0.003886567894369364,
+            0.02219473011791706,
+            -0.023921171203255653,
+            0.013537623919546604,
+            0.01910705678164959,
+            0.017845427617430687,
+            -0.012226192280650139,
+            0.023622365668416023,
+            0.008781610056757927,
+            0.008781610056757927,
+            -0.03808131068944931,
+            -0.018277037888765335,
+            0.007225322537124157,
+            -0.0038222414441406727,
+            0.00012969059753231704,
+            0.02016948163509369,
+            0.003577385563403368,
+            0.011570476926863194,
+            -0.03374860808253288,
+            -0.0065322560258209705,
+            -0.011221867986023426,
+            0.006743911188095808,
+            0.006557156797498465,
+            -0.009503726847469807,
+            0.00756148062646389,
+            0.037848904728889465,
+            -0.002983921440318227,
+            0.0122842937707901,
+            0.0069555663503706455,
+            -0.002749440725892782,
+            -0.010084740817546844,
+            0.0147245517000556,
+            -0.016849402338266373,
+            0.013612325303256512,
+            -0.02546500973403454,
+            -0.002585511887446046,
+            -0.016301589086651802,
+            0.01973787136375904,
+            0.004345153924077749,
+            0.018609045073390007,
+            -0.0025917370803654194,
+            -0.020534690469503403,
+            -0.009188319556415081,
+            -0.017513418570160866,
+            0.002475534100085497,
+            -0.006407753098756075,
+            0.0018457566620782018,
+            -0.006357952021062374,
+            -0.00014655037375632674,
+            0.009196619503200054,
+            -0.019771073013544083,
+            -0.018791649490594864,
+            -0.019771073013544083,
+            -0.015488170087337494,
+            0.006250049453228712,
+            -0.014425745233893394,
+            -0.006785412319004536,
+            -0.002328205620869994,
+            0.005843339487910271,
+            0.041833002120256424,
+            -0.020833497866988182,
+            -0.028652284294366837,
+            0.02503339946269989,
+            -0.00022877422452438623,
+            -0.009171719662845135,
+            0.01370362751185894,
+            -0.0012211666908115149,
+            -0.005154423415660858,
+            0.018990853801369667,
+            -0.01889125257730484,
+            0.007229472976177931,
+            0.008673707023262978,
+            0.02109910361468792,
+            -0.008682007901370525,
+            -0.0017918053781613708,
+            0.00472281314432621,
+            0.0207504965364933,
+            -0.00898081436753273,
+            0.006478304974734783,
+            -0.006590357515960932,
+            0.006075745448470116,
+            0.002392532303929329,
+            0.043061431497335434,
+            -0.010109641589224339,
+            0.004353453870862722,
+            -0.006428503897041082,
+            -0.011221867986023426,
+            0.003284803591668606,
+            0.03212176635861397,
+            -0.012350695207715034,
+            -0.009453926235437393,
+            -0.0004974931362085044,
+            0.004258001688867807,
+            -0.0012782305711880326,
+            0.009669731371104717,
+            -0.002780566457659006,
+            -0.019422464072704315,
+            0.009370923973619938,
+            -0.0021808771416544914,
+            -0.010109641589224339,
+            -0.008839711546897888,
+            0.012417096644639969,
+            -0.014873955398797989,
+            0.003652087412774563,
+            -0.009022315964102745,
+            0.0038969432935118675,
+            0.017729224637150764,
+            0.02312435209751129,
+            -0.013545923866331577,
+            0.00015848191105760634,
+            -0.014019034802913666,
+            -0.02991391532123089,
+            -0.022344134747982025,
+            0.004980118945240974,
+            0.001614388544112444,
+            -0.007856138050556183,
+            0.007237772922962904,
+            0.0004977525095455348,
+            -0.011686678975820541,
+            0.007922539487481117,
+            0.001984784845262766,
+            -0.020003478974103928,
+            0.017646221444010735,
+            -0.005183474160730839,
+            0.014425745233893394,
+            0.004206125624477863,
+            0.012691003270447254,
+            0.004094073083251715,
+            -0.0008103068685159087,
+            -0.004677162040024996,
+            0.005909741390496492,
+            -0.01829363778233528,
+            0.02249353751540184,
+            0.04160059615969658,
+            -0.0004712956433650106,
+            0.012035287916660309,
+            0.013628926128149033,
+            0.01829363778233528,
+            -0.005058970768004656,
+            -0.010259045287966728,
+            0.002500434871762991,
+            -0.01744701713323593,
+            -0.005166873335838318,
+            -0.004938618279993534,
+            0.028204074129462242,
+            0.00562338437885046,
+            0.005818439181894064,
+            0.00597199285402894,
+            0.0006686847191303968,
+            0.013745129108428955,
+            0.007490929216146469,
+            0.026842841878533363,
+            0.020617693662643433,
+            -0.020783696323633194,
+            -0.011346370913088322,
+            -0.0039674947038292885,
+            0.018061231821775436,
+            0.018360039219260216,
+            -0.00022812576207797974,
+            -0.048307154327631,
+            0.026361430063843727,
+            -0.007295874413102865,
+            0.02486739493906498,
+            -0.00315615045838058,
+            0.021912524476647377,
+            0.01962166838347912,
+            -0.01523086428642273,
+            0.01834343932569027,
+            0.01923985965549946,
+            0.001934983767569065,
+            0.03384821116924286,
+            -0.028054671362042427,
+            -0.0063413516618311405,
+            -0.010416748933494091,
+            -0.005830889567732811,
+            0.01736401580274105,
+            -0.0006453404203057289,
+            -0.028585882857441902,
+            0.0021539016161113977,
+            0.023954372853040695,
+            -0.0046896119602024555,
+            0.022858746349811554,
+            -0.009495426900684834,
+            -0.028287075459957123,
+            -0.026461033150553703,
+            0.001020405674353242,
+            -0.023788368329405785,
+            -0.018924452364444733,
+            0.004780914168804884,
+            0.02850288152694702,
+            0.0039031682536005974,
+            -0.016334790736436844,
+            0.00545323034748435,
+            -0.02118210680782795,
+            0.013786629773676395,
+            0.005399279296398163,
+            -0.001994122751057148,
+            0.01665019802749157,
+            0.006893314886838198,
+            0.0053038266487419605,
+            0.0021497514098882675,
+            0.03335019573569298,
+            -0.01262460183352232,
+            -0.006266649812459946,
+            -0.008549204096198082,
+            0.003048247890546918,
+            0.00037999346386641264,
+            0.011072465218603611,
+            0.02516620233654976,
+            0.0008331324206665158,
+            0.005581883247941732,
+            -0.0003052916727028787,
+            0.017463617026805878,
+            -0.015388567931950092,
+            -0.03881172835826874,
+            0.00280131702311337,
+            -0.006171197630465031,
+            -0.021680118516087532,
+            0.015396867878735065,
+            -0.02885149046778679,
+            -0.007673533633351326,
+            -0.0008206821512430906,
+            0.0013581200037151575,
+            0.031756557524204254,
+            -0.005353628192096949,
+            -0.017347414046525955,
+            0.0014421595260500908,
+            -0.028702085837721825,
+            -0.0030129719525575638,
+            0.009927037172019482,
+            0.024385983124375343,
+            0.019306261092424393,
+            -0.01804463192820549,
+            0.018028030171990395,
+            0.027772463858127594,
+            -0.003533809445798397,
+            -0.023987572640180588,
+            -0.010707256384193897,
+            -0.009935337118804455,
+            0.011570476926863194,
+            0.0023116052616387606,
+            -0.013313517905771732,
+            0.010615954175591469,
+            -0.01075705699622631,
+            -0.0019412089604884386,
+            -0.0010655380319803953,
+            -0.013794929720461369,
+            0.052888862788677216,
+            0.006005193572491407,
+            0.0049220179207623005,
+            -0.03957534581422806,
+            0.005108772311359644,
+            -0.011180367320775986,
+            0.013911132700741291,
+            0.009736132808029652,
+            0.014873955398797989,
+            0.00668165972456336,
+            -0.001169290510006249,
+            0.01557117234915495,
+            -0.011147166602313519,
+            0.013728528283536434,
+            0.005652435123920441,
+            -0.0300799198448658,
+            0.0023302806075662374,
+            0.0006536406581290066,
+            0.0051253726705908775,
+            -0.004789214581251144,
+            -1.1988275218755007e-05,
+            0.007304174825549126,
+            0.025149600580334663,
+            0.014508746564388275,
+            0.008254547603428364,
+            -0.011578776873648167,
+            -0.023057950660586357,
+            0.01508146058768034,
+            0.009495426900684834,
+            -0.000578420062083751,
+            0.002919594757258892,
+            -0.030594531446695328,
+            0.005615084432065487,
+            0.004050496965646744,
+            0.010740457102656364,
+            0.01347122248262167,
+            -0.010815158486366272,
+            0.024385983124375343,
+            -0.015454969368875027,
+            -0.003757914761081338,
+            0.011163766495883465,
+            -0.05494731292128563,
+            0.00810929387807846,
+            0.03867892548441887,
+            0.004399104975163937,
+            -0.027307653799653053,
+            -0.01543836947530508,
+            -0.01910705678164959,
+            -0.000976829556748271,
+            0.0006375589873641729,
+            0.005527932196855545,
+            0.006988767068833113,
+            -0.016832802444696426,
+            0.011213568039238453,
+            -0.01880824938416481,
+            -0.034163616597652435,
+            -0.011869283393025398,
+            0.019605068489909172,
+            0.012940009124577045,
+            -0.023439761251211166,
+            0.018443040549755096,
+            -0.010425048880279064,
+            0.011279969476163387,
+            -0.01247519813477993,
+            -0.032885387539863586,
+            0.003793190699070692,
+            -0.015737175941467285,
+            0.007283424027264118,
+            0.0165173951536417,
+            -0.03159055486321449,
+            0.007839537225663662,
+            0.006611107848584652,
+            -0.013039612211287022,
+            -0.03029572404921055,
+            0.018160834908485413,
+            -0.0019806348718702793,
+            0.019040655344724655,
+            -0.01982087455689907,
+            0.019040655344724655,
+            -0.007590531371533871,
+            -0.00685181375592947,
+            -0.03313439339399338,
+            -0.007042718585580587,
+            -0.010682355612516403,
+            -0.0283866785466671,
+            -0.011977186426520348,
+            -0.009935337118804455,
+            0.02342315949499607,
+            -0.040438566356897354,
+            0.003598136128857732,
+            -0.012915108352899551,
+            0.017679423093795776,
+            0.014135237783193588,
+            -0.023871371522545815,
+            0.020684095099568367,
+            -0.018625644966959953,
+            -0.013628926128149033,
+            -0.006868414115160704,
+            -0.011304870247840881,
+            0.005648285150527954,
+            -0.000889158749487251,
+            -0.004494557622820139,
+            -0.005295526701956987,
+            -0.02270934358239174,
+            -0.03180636093020439,
+            -0.002793016843497753,
+            0.0006531218532472849,
+            -0.0025979620404541492,
+            0.012857007794082165,
+            0.02846967987716198,
+            -0.008453751914203167,
+            -0.0032163269352167845,
+            0.005855789873749018,
+            0.017679423093795776,
+            0.0008793022716417909,
+            0.023107752203941345,
+            -0.015347067266702652,
+            -0.02363896556198597,
+            0.0012605926021933556,
+            0.01574547588825226,
+            -0.010532951913774014,
+            -0.0017357789911329746,
+            0.01489055622369051,
+            -0.006270799785852432,
+            0.003816016251221299,
+            -0.02134811133146286,
+            -0.0288016889244318,
+            -0.0022701043635606766,
+            -0.002444408368319273,
+            -0.015703976154327393,
+            -0.0013435946311801672,
+            -0.012616301886737347,
+            -0.00893931370228529,
+            0.021447712555527687,
+            -0.01493205688893795,
+            -0.01139617245644331,
+            0.01037524826824665,
+            -0.01826043613255024,
+            0.008872912265360355,
+            0.0013321818551048636,
+            0.017696022987365723,
+            0.02066749334335327,
+            -0.002402907470241189,
+            0.011338070966303349,
+            0.013944333419203758,
+            0.0035794605500996113,
+            -0.0056316847912967205,
+            0.014052236452698708,
+            0.010225844569504261,
+            -0.011528975330293179,
+            0.001997235231101513,
+            0.011578776873648167,
+            0.01423483993858099,
+            -0.03095974028110504,
+            -0.022128328680992126,
+            -0.014575148932635784,
+            0.015181062743067741,
+            0.0192730613052845,
+            0.0013487822143360972,
+            -0.002954870695248246,
+            0.0049925693310797215,
+            -0.009113618172705173,
+            0.005171023774892092,
+            0.003899018280208111,
+            -0.0186422448605299,
+            0.004660561215132475,
+            0.0014815854374319315,
+            0.007685983553528786,
+            -0.006100645754486322,
+            0.010723856277763844,
+            -0.003911468666046858,
+            -0.011553876101970673,
+            -0.007163071073591709,
+            -0.01894105225801468,
+            -0.00889781303703785,
+            0.020020078867673874,
+            0.00022086309036239982,
+            -0.02211172878742218,
+            0.005656585097312927,
+            0.013413120992481709,
+            -0.015023359097540379,
+            -0.02334015816450119,
+            0.010026639327406883,
+            -0.0005804951069876552,
+            0.0008767084800638258,
+            0.010267345234751701,
+            -0.011761381290853024,
+            0.016907503828406334,
+            -0.014276341535151005,
+            0.010964562185108662,
+            0.0068227630108594894,
+            -0.04189940169453621,
+            0.010441649705171585,
+            0.019671469926834106,
+            -0.01224279310554266,
+            -0.01829363778233528,
+            0.0008450639434158802,
+            0.0277392640709877,
+            -0.0012159791076555848,
+            0.011935685761272907,
+            0.019986877217888832,
+            -0.0007024043006822467,
+            -0.012317494489252567,
+            -0.004851466044783592,
+            0.0918997973203659,
+            0.016309890896081924,
+            0.0136621268466115,
+            0.010200943797826767,
+            0.011836082674562931,
+            -0.030129719525575638,
+            -0.007225322537124157,
+            -0.008906112983822823,
+            -0.010823458433151245,
+            -0.017098408192396164,
+            -0.014434045180678368,
+            0.006777111906558275,
+            0.012790605425834656,
+            0.015919780358672142,
+            0.015952982008457184,
+            -0.00895591452717781,
+            0.007781436201184988,
+            0.008204746060073376,
+            0.0037662149406969547,
+            0.013031311333179474,
+            -0.0056316847912967205,
+            -0.0003042541502509266,
+            -0.011230168864130974,
+            0.010939661413431168,
+            0.014417444355785847,
+            0.011230168864130974,
+            -0.00262701278552413,
+            0.004590009804815054,
+            0.02500019781291485,
+            0.00793083943426609,
+            0.016060883179306984,
+            -0.016002783551812172,
+            0.0045941597782075405,
+            -0.013189014978706837,
+            0.040903378278017044,
+            0.015255765058100224,
+            0.0037765903398394585,
+            -0.005241575185209513,
+            0.0122842937707901,
+            0.004681312013417482,
+            0.018243836238980293,
+            -0.014940356835722923,
+            0.013545923866331577,
+            -0.009453926235437393,
+            -0.0038616673555225134,
+            0.03067753277719021,
+            -0.006677509751170874,
+            0.0029112945776432753,
+            -0.014093737117946148,
+            0.004560959059745073,
+            0.007980640977621078,
+            0.003305553924292326,
+            0.004830715712159872,
+            0.016633598133921623,
+            -0.01894105225801468,
+            -0.011138866655528545,
+            0.020551292225718498,
+            -0.013637226074934006,
+            -0.0017949179746210575,
+            0.017613021656870842,
+            0.006100645754486322,
+            -0.0013819830492138863,
+            0.010856659151613712,
+            -0.012334095314145088,
+            -0.007358125876635313,
+            0.008433002047240734,
+            0.006183648016303778,
+            4.756402631755918e-05,
+            0.005577733274549246,
+            0.024452384561300278,
+            0.005320427007973194,
+            -0.017231212928891182,
+            0.012699303217232227,
+            0.0008367637638002634,
+            0.010483150370419025,
+            0.017928428947925568,
+            0.011952285654842854,
+            0.01948886550962925,
+            -0.015977881848812103,
+            0.00719212181866169,
+            0.006100645754486322,
+            0.01572057604789734,
+            -0.02689264342188835,
+            -0.006992917042225599,
+            -0.0011672154068946838,
+            -0.0017814300954341888,
+            0.012715904042124748,
+            0.014840754680335522,
+            -0.00022125216491986066,
+            0.008972514420747757,
+            -0.0024278080090880394,
+            -0.010408448986709118,
+            0.02172992005944252,
+            -0.022725943475961685,
+            -0.02588001824915409,
+            -0.024634988978505135,
+            0.0073664262890815735,
+            -0.012558200396597385,
+            0.008304348215460777,
+            -0.005270625930279493,
+            0.009835734963417053,
+            0.004432306159287691,
+            0.015181062743067741,
+            0.002600037259981036,
+            -0.007586381398141384,
+            -0.014176739379763603,
+            0.021414512768387794,
+            -0.0033906311728060246,
+            0.009254720993340015,
+            -0.009960237890481949,
+            0.014301242306828499,
+            0.0038222414441406727,
+            0.00560263404622674,
+            -0.010242444463074207,
+            0.016641898080706596,
+            -0.014774353243410587,
+            0.002942420309409499,
+            0.016832802444696426,
+            -0.01736401580274105,
+            -0.012566500343382359,
+            -0.006719010882079601,
+            -0.011703279800713062,
+            0.006905765272676945,
+            -0.028071271255612373,
+            -0.011869283393025398,
+            0.007723334711045027,
+            -0.018077831715345383,
+            0.0071464707143604755,
+            -0.00963653065264225,
+            -0.016558896750211716,
+            0.0235061626881361,
+            0.027855465188622475,
+            -0.011379571631550789,
+            -0.004123123362660408,
+            -0.0025564611423760653,
+            0.003811866044998169,
+            -0.026660237461328506,
+            0.0006546781514771283,
+            -0.0056192344054579735,
+            -0.008773310109972954,
+            -0.003048247890546918,
+            0.016849402338266373,
+            0.002066749380901456,
+            0.0013435946311801672,
+            0.013462921604514122,
+            -0.006810312625020742,
+            -0.020534690469503403,
+            0.01109736505895853,
+            0.014458945952355862,
+            -0.01762962155044079,
+            0.003143700072541833,
+            0.0009285847190767527,
+            -0.019389264285564423,
+            0.01547987014055252,
+            0.00855750497430563,
+            0.016110684722661972,
+            -0.0004414668073877692,
+            0.013819830492138863,
+            -0.011819482780992985,
+            -0.005349477753043175,
+            -0.008590705692768097,
+            0.025780417025089264,
+            0.00685181375592947,
+            0.015073160640895367,
+            0.010873259976506233,
+            0.006270799785852432,
+            -0.00476431380957365,
+            0.027390655130147934,
+            0.023738566786050797,
+            -0.0007309362408705056,
+            -0.005361928138881922,
+            0.010076440870761871,
+            -0.005851639900356531,
+            0.0027100148145109415,
+            -0.017231212928891182,
+            0.0080511923879385,
+            0.0024713841266930103,
+            0.010400149039924145,
+            0.029465705156326294,
+            0.012151490896940231,
+            -0.014301242306828499,
+            0.02571401558816433,
+            -0.003058623056858778,
+            -0.023987572640180588,
+            -0.00014538316463585943,
+            -0.01425144076347351,
+            -0.009080417454242706,
+            0.010840059258043766,
+            -0.0014691351680085063,
+            -0.015106361359357834,
+            0.015297265723347664,
+            0.01453364733606577,
+            0.009927037172019482,
+            -0.008105143904685974,
+            0.004266302101314068,
+            0.00011510041076689959,
+            0.01315581426024437,
+            0.004901267122477293,
+            0.014326142147183418,
+            -0.004185375291854143,
+            0.0029237449634820223,
+            -5.317314571584575e-05,
+            -0.0014784728409722447,
+            -0.012201291508972645,
+            0.007881038822233677,
+            0.012358995154500008,
+            0.0033034789375960827,
+            0.013562524691224098,
+            0.00297147105447948,
+            -0.015820179134607315,
+            -0.03290198743343353,
+            -0.0009156156447716057,
+            0.010840059258043766,
+            -0.009794234298169613,
+            -0.01141277328133583,
+            0.010723856277763844,
+            0.007208722177892923,
+            0.009005715139210224,
+            0.013288618065416813,
+            0.003759989980608225,
+            -0.002585511887446046,
+            -0.010275646112859249,
+            0.021148905158042908,
+            0.007453578058630228,
+            -0.002770191291347146,
+            -0.0036686877720057964,
+            -0.010134542360901833,
+            -0.009586729109287262,
+            -0.0034196816850453615,
+            -0.014392544515430927,
+            0.013313517905771732,
+            0.007565631065517664,
+            0.007059318944811821,
+            0.011952285654842854,
+            0.016467593610286713,
+            0.0035919109359383583,
+            0.009022315964102745,
+            -0.010690655559301376,
+            -0.006175347603857517,
+            -0.020136281847953796,
+            -0.01850944198668003,
+            -0.014135237783193588,
+            0.0015510995872318745,
+            -0.006864264141768217,
+            -0.0004572890466079116,
+            -0.004614910576492548,
+            0.02131490968167782,
+            -0.011288270354270935,
+            -0.0012854932574555278,
+            -3.089878737227991e-05,
+            -0.008856311440467834,
+            -0.01298151072114706,
+            0.020916501060128212,
+            0.0016662648413330317,
+            0.0157786775380373,
+            0.007395477034151554,
+            0.026046022772789,
+            0.007026118226349354,
+            0.002187102334573865,
+            -0.015106361359357834,
+            -0.02431958168745041,
+            -0.011487474665045738,
+            0.021082503721117973,
+            -0.004193675238639116,
+            -0.0001918124035000801,
+            0.02334015816450119,
+            -0.011827782727777958,
+            0.007167221512645483,
+            -0.015313866548240185,
+            0.030312323942780495,
+            -0.008856311440467834,
+            0.012275993824005127,
+            0.012765705585479736,
+            0.01826043613255024,
+            -0.0018997079459950328,
+            0.01741381548345089,
+            0.0021829521283507347,
+            -0.008997415192425251,
+            -0.008383200503885746,
+            -0.0017679422162473202,
+            -0.021148905158042908,
+            -0.012782305479049683,
+            0.00771918473765254,
+            0.0050299204885959625,
+            -0.0054158796556293964,
+            0.013504423201084137,
+            0.0015355367213487625,
+            -0.0007049980922602117,
+            -0.00982743501663208,
+            0.00670241005718708,
+            -0.0013653826899826527,
+            -0.01523086428642273,
+            -0.0017181411385536194,
+            0.011454273946583271,
+            -0.027174849063158035,
+            -0.008856311440467834,
+            0.010010039433836937,
+            -0.00751997996121645,
+            -0.010433349758386612,
+            -0.00786028802394867,
+            0.0022763293236494064,
+            0.008154945448040962,
+            0.004158399533480406,
+            -0.009022315964102745,
+            0.0026456883642822504,
+            0.0015687375562265515,
+            -0.00035249904613010585,
+            -0.011968886479735374,
+            -0.005710536614060402,
+            -0.00898081436753273,
+            0.019937077537178993,
+            -0.007565631065517664,
+            -0.00580183882266283,
+            0.003598136128857732,
+            -0.0068227630108594894,
+            0.015056559816002846,
+            -0.013902832753956318,
+            -0.007885188795626163,
+            -0.01812763325870037,
+            0.021364711225032806,
+            -0.014683051034808159,
+            -0.00026443914975970984,
+            -0.008034592494368553,
+            -0.013280317187309265,
+            -0.0020646743942052126,
+            -0.0029693960677832365,
+            0.017214611172676086,
+            -0.0024278080090880394,
+            0.0003117762098554522,
+            -0.010516351088881493,
+            0.002209927886724472,
+            0.00788933876901865,
+            -0.013064512051641941,
+            0.0020677868742495775,
+            -0.0011402397649362683,
+            0.006241749040782452,
+            -0.007627882529050112,
+            -0.012051887810230255,
+            -0.007997240871191025,
+            0.019156858325004578,
+            0.012010387144982815,
+            -0.00773163465783,
+            -0.017081808298826218,
+            -0.007154771126806736,
+            0.013628926128149033,
+            0.014102037064731121,
+            0.005565282888710499,
+            -0.033649004995822906,
+            -0.003977870102971792,
+            -0.004859765991568565,
+            0.009121918119490147,
+            -0.022443735972046852,
+            -0.0074867792427539825,
+            -0.005270625930279493,
+            -0.02363896556198597,
+            -0.0010634629288688302,
+            0.010898160748183727,
+            0.006474155001342297,
+            -0.007470178417861462,
+            -0.009121918119490147,
+            -0.008291898295283318,
+            0.009951937943696976,
+            -0.01198548637330532,
+            0.0051959240809082985,
+            -0.0031416250858455896,
+            0.005805988796055317,
+            0.02126510813832283,
+            -0.018609045073390007,
+            0.0034093065187335014,
+            -0.005025770049542189,
+            -0.00947052612900734,
+            0.006710710469633341,
+            -0.007104970049113035,
+            -0.014649850316345692,
+            -0.006743911188095808,
+            -0.009495426900684834,
+            0.0004225319717079401,
+            0.007042718585580587,
+            0.0013684951700270176,
+            -0.028021469712257385,
+            0.004258001688867807,
+            -0.003836766816675663,
+            0.014832454733550549,
+            -0.00011587855260586366,
+            -0.008881212212145329,
+            0.011960585601627827,
+            -0.002284629736095667,
+            0.0050299204885959625,
+            0.012516699731349945,
+            -0.01071555633097887,
+            -0.023190755397081375,
+            -0.006868414115160704,
+            -0.010076440870761871,
+            0.005067271180450916,
+            -0.004361754283308983,
+            0.022941747680306435,
+            -0.03535884618759155,
+            0.014085437171161175,
+            -0.0074494280852377415,
+            -0.006528106052428484,
+            0.006972166709601879,
+            -0.005345327779650688,
+            0.027722662314772606,
+            0.00489296717569232,
+            0.008433002047240734,
+            -0.003654162399470806,
+            -0.008549204096198082,
+            -0.020385287702083588,
+            0.02443578466773033,
+            0.010574452579021454,
+            -0.0014286716468632221,
+            0.0023468811996281147,
+            0.006760511547327042,
+            0.011562176048755646,
+            0.024468984454870224,
+            -0.024717990309000015,
+            -0.007926689460873604,
+            -0.018111033365130424,
+            -0.013255417346954346,
+            -0.008416401222348213,
+            -0.011113965883851051,
+            -0.0006640158826485276,
+            -0.007237772922962904,
+            -0.0021310760639607906,
+            -0.0026228628121316433,
+            -0.0015479869907721877,
+            -0.004270452074706554,
+            -0.0014743227511644363,
+            -0.003079373622313142,
+            -0.015737175941467285,
+            -0.003106349380686879,
+            0.005224974825978279,
+            0.00948712695389986,
+            -0.004847316071391106,
+            0.0009778671665117145,
+            0.01056615263223648,
+            0.014467245899140835,
+            -0.007237772922962904,
+            -0.021464312449097633,
+            0.02385476976633072,
+            -0.010682355612516403,
+            0.000497233762871474,
+            -0.013238816522061825,
+            -0.009121918119490147,
+            -0.0034508074168115854,
+            0.0024402583949267864,
+            -0.0032267021015286446,
+            -0.015239164233207703,
+            -0.0031810509972274303,
+            -0.011421073228120804,
+            -0.016177086159586906,
+            0.01648419350385666,
+            0.03278578445315361,
+            -0.0015230864519253373,
+            0.016509095206856728,
+            -1.2742101716867182e-05,
+            0.010433349758386612,
+            -0.004805814940482378,
+            0.0005177248967811465,
+            0.003600211115553975,
+            -0.009578429162502289,
+            -0.004946918226778507,
+            0.007511679548770189,
+            -0.004021446220576763,
+            -0.006727310828864574,
+            -0.00961992982774973,
+            -0.017231212928891182,
+            -0.00893931370228529,
+            -0.005440779961645603,
+            -0.009263021871447563,
+            0.012350695207715034,
+            0.004345153924077749,
+            -0.008781610056757927,
+            0.014824154786765575,
+            0.00424555130302906,
+            -0.007034418173134327,
+            -0.011080765165388584,
+            -0.00013040390331298113,
+            0.0230911523103714,
+            0.023140953853726387,
+            -0.00809269305318594,
+            -0.026444431394338608,
+            0.01593638025224209,
+            -0.025597812607884407,
+            -0.004365904256701469,
+            -0.011371271684765816,
+            -0.01012624241411686,
+            -0.03838011622428894,
+            0.0087899100035429,
+            0.011180367320775986,
+            0.01635139063000679,
+            0.002259728964418173,
+            -0.0031167245469987392,
+            0.014608349651098251,
+            -0.008831411600112915,
+            -0.016235187649726868,
+            -0.004556809086352587,
+            0.003867892548441887,
+            -0.008706907741725445,
+            0.015089760534465313,
+            0.005436629988253117,
+            0.007793886587023735,
+            -0.0019754471722990274,
+            0.002940345322713256,
+            0.0036769879516214132,
+            0.021298309788107872,
+            0.004332703538239002,
+            0.004685461986809969,
+            0.0035836107563227415,
+            -0.0004712956433650106,
+            -0.010931361466646194,
+            0.007142320740967989,
+            -0.0013446321245282888,
+            -0.007055168505758047,
+            -0.00889781303703785,
+            0.030743934214115143,
+            0.02948230504989624,
+            0.010649154894053936,
+            -0.008266997523605824,
+            0.009495426900684834,
+            0.00908871740102768,
+            -0.015015059150755405,
+            0.033233996480703354,
+            -0.004955218639224768,
+            0.0014608348719775677,
+            0.012649502605199814,
+            0.016849402338266373,
+            -0.008287748321890831,
+            -0.006374552380293608,
+            0.007295874413102865,
+            -0.007395477034151554,
+            0.006988767068833113,
+            0.025946419686079025,
+            0.007706734351813793,
+            -0.004287052433937788,
+            -0.030528130009770393,
+            0.0038637423422187567,
+            -0.0031623756512999535,
+            -0.009478827007114887,
+            -0.005328727420419455,
+            0.01209338940680027,
+            -0.011670079082250595,
+            0.007818787358701229,
+            0.008466202765703201,
+            -0.0101677430793643,
+            0.0050880215130746365,
+            -0.00849110260605812,
+            0.0030752234160900116,
+            -0.01527236495167017,
+            -0.01654229499399662,
+            -0.005428329575806856,
+            0.01258310116827488,
+            -0.003857517149299383,
+            -0.008831411600112915,
+            0.012541599571704865,
+            0.01330521795898676,
+            0.01796162873506546,
+            -0.006864264141768217,
+            -0.0028677184600383043,
+            0.00908871740102768,
+            0.025863418355584145,
+            0.0016901278868317604,
+            -0.011852683499455452,
+            0.0013799079461023211,
+            0.008848011493682861,
+            0.017397215589880943,
+            0.007291724439710379,
+            -0.004818265326321125,
+            -0.01330521795898676,
+            -0.01741381548345089,
+            -0.001454609795473516,
+            0.003328379476442933,
+            -0.008565804921090603,
+            0.01103926356881857,
+            0.015056559816002846,
+            -0.009155118837952614,
+            0.007773135788738728,
+            0.0006463779718615115,
+            0.015961281955242157,
+            -0.022061927244067192,
+            -0.026212027296423912,
+            0.013197315856814384,
+            -0.003984095063060522,
+            -0.02028568461537361,
+            -0.00384506699629128,
+            0.001266817795112729,
+            0.022393936291337013,
+            -0.00034731143387034535,
+            -0.00038621859857812524,
+            0.00439495500177145,
+            -0.007138170767575502,
+            0.011006062850356102,
+            -0.02888469025492668,
+            -0.004490407183766365,
+            -3.7837231502635404e-05,
+            -0.0040878476575016975,
+            0.00018014023953583091,
+            -0.014898856170475483,
+            0.006727310828864574,
+            0.01733081415295601,
+            -0.008889512158930302,
+            0.01826043613255024,
+            -0.004697912372648716,
+            0.024070575833320618,
+            0.010607653297483921,
+            -0.0012657803017646074,
+            -0.0052332752384245396,
+            0.001739929080940783,
+            -0.028370078653097153,
+            0.004436456132680178,
+            -0.007358125876635313,
+            -0.010342047549784184,
+            -0.00489296717569232,
+            -0.024087175726890564,
+            0.013587424531579018,
+            0.006893314886838198,
+            0.018775049597024918,
+            0.004552658647298813,
+            0.008773310109972954,
+            0.020352086052298546,
+            -0.010856659151613712,
+            -0.0006178460316732526,
+            0.0036292618606239557,
+            0.012674403376877308,
+            0.0069182151928544044,
+            -0.004519457928836346,
+            -0.024800993502140045,
+            -0.030561331659555435,
+            0.0040359715931117535,
+            0.0016081634676083922,
+            -0.00978593435138464,
+            -0.0022763293236494064,
+            -0.011113965883851051,
+            0.0034508074168115854,
+            -0.02749025821685791,
+            -0.010250745341181755,
+            0.007312474772334099,
+            0.030146321281790733,
+            -0.006793712265789509,
+            -0.0005296564195305109,
+            -0.020518090575933456,
+            0.002039773855358362,
+            -0.02699224464595318,
+            -0.014268041588366032,
+            0.008320949040353298,
+            -0.00786028802394867,
+            0.021962326020002365,
+            -0.02999691665172577,
+            -0.001976484665647149,
+            -0.0046066101640462875,
+            -0.011852683499455452,
+            0.004560959059745073,
+            -0.013296918012201786,
+            -0.0063413516618311405,
+            0.0033470550552010536,
+            0.004174999892711639,
+            -0.00864050630480051,
+            0.016998806968331337,
+            -0.018841451033949852,
+            -0.012549900449812412,
+            -0.00456510903313756,
+            -0.01711500994861126,
+            -0.003201801562681794,
+            0.015712276101112366,
+            -0.007528279908001423,
+            0.005590183660387993,
+            0.017812225967645645,
+            0.0006666096742264926,
+            -0.007084219250828028,
+            -0.02672663889825344,
+            0.01593638025224209,
+            0.001115339226089418,
+            -0.004038046579807997,
+            -0.0005029401509091258,
+            0.02833687700331211,
+            0.00314577529206872,
+            0.007590531371533871,
+            0.014027335681021214,
+            -0.006528106052428484,
+            -0.017148209735751152,
+            0.008814810775220394,
+            -0.002479684306308627,
+            -0.01880824938416481,
+            0.010350347496569157,
+            -9.426885299035348e-06,
+            -0.004631510935723782,
+            -0.003811866044998169,
+            -0.016940705478191376,
+            -0.0064492542296648026,
+            -0.012466898187994957,
+            -0.0015282740350812674,
+            0.015330466441810131,
+            -0.005735436920076609,
+            -0.00933772325515747,
+            -0.014467245899140835,
+            -0.021829521283507347,
+            0.018459642305970192,
+            -0.011960585601627827,
+            8.036764484131709e-06,
+            -0.0022161530796438456,
+            -0.006457554176449776,
+            -0.0023323558270931244,
+            -0.012491798959672451,
+            0.000848176539875567,
+            -0.0028635684866458178,
+            -0.02164691686630249,
+            -0.0032059517689049244,
+            0.020318886265158653,
+            0.02850288152694702,
+            0.005743737332522869,
+            -2.742956166912336e-05,
+            -0.010765357874333858,
+            -0.02669343911111355,
+            0.01406053639948368,
+            0.009180019609630108,
+            -0.020435089245438576,
+            -0.01509806141257286,
+            0.01706520840525627,
+            0.0012720053782686591,
+            -0.005573583301156759,
+            -0.02558121085166931,
+            0.0032785783987492323,
+            0.003392706159502268,
+            -0.03125854581594467,
+            -0.0028635684866458178,
+            0.008823110722005367,
+            7.243220170494169e-05,
+            -0.01434274297207594,
+            0.016367990523576736,
+            -0.0017606796463951468,
+            0.011387872509658337,
+            0.007266823668032885,
+            0.0157786775380373,
+            -0.03678647801280022,
+            -0.033233996480703354,
+            0.013844731263816357,
+            0.010516351088881493,
+            -0.018526043742895126,
+            -0.011504075489938259,
+            0.013653826899826527,
+            -0.0002509512996766716,
+            -0.008125894702970982,
+            -0.0022618041839450598,
+            0.01266610249876976,
+            -0.03452882543206215,
+            -0.021447712555527687,
+            -0.009188319556415081,
+            0.011113965883851051,
+            0.002952795708552003,
+            0.011105665937066078,
+            0.014649850316345692,
+            -0.014110337011516094,
+            -0.010043240152299404,
+            -0.03894453123211861,
+            -0.004772614222019911,
+            0.009163418784737587,
+            -0.01733081415295601,
+            -0.0095286276191473,
+            0.003585685743018985,
+            -0.004751863423734903,
+            0.0084205511957407,
+            0.008881212212145329,
+            0.014525347389280796,
+            -0.002135226037353277,
+            0.015181062743067741,
+            0.0007112232269719243,
+            0.01943906396627426,
+            0.01283210702240467,
+            0.002712089801207185,
+            0.019854074344038963,
+            0.0007480554049834609,
+            -0.003438357263803482,
+            -0.004258001688867807,
+            0.002369706751778722,
+            0.009918737225234509,
+            -0.003120874520391226,
+            0.00016457737365271896,
+            0.020152881741523743,
+            0.007968190126121044,
+            0.009769333526492119,
+            -0.0021891773212701082,
+            0.03545844927430153,
+            -0.011769681237637997,
+            0.005814289208501577,
+            -0.005897291004657745,
+            -0.015513070859014988,
+            0.020833497866988182,
+            0.012765705585479736,
+            0.0016413643024861813,
+            0.018326837569475174,
+            0.00528307631611824,
+            0.0007226360030472279,
+            -0.00543248001486063,
+            -0.022227931767702103,
+            0.0013591574970632792,
+            0.017596419900655746,
+            -0.007698433939367533,
+            0.006096495781093836,
+            -0.004536058288067579,
+            0.023406559601426125,
+            0.0005312127177603543,
+            0.006897464860230684,
+            0.016417792066931725,
+            -0.006067445036023855,
+            0.003957119770348072,
+            -0.0006334088975563645,
+            0.0022784045431762934,
+            -0.011329771019518375,
+            -0.003199726575985551,
+            0.016318190842866898,
+            0.0010800632881000638,
+            0.0020034604240208864,
+            0.004614910576492548,
+            0.0002532857470214367,
+            0.0009825360029935837,
+            0.002994296606630087,
+            -0.0062085483223199844,
+            0.010101341642439365,
+            -0.0068601141683757305,
+            -0.006934816017746925,
+            -0.017762424424290657,
+            0.0063164508901536465,
+            0.029764512553811073,
+            -0.01050805114209652,
+            -0.02016948163509369,
+            0.0039052434731274843,
+            -0.005556982941925526,
+            0.003988245502114296,
+            -0.0008751521818339825,
+            -0.005025770049542189,
+            -0.01787862740457058,
+            -0.009121918119490147,
+            0.007914239540696144,
+            -0.0018135933205485344,
+            0.018575843423604965,
+            0.029681509360671043,
+            0.005673185456544161,
+            -7.55447763367556e-05,
+            0.009595029056072235,
+            0.004598309751600027,
+            0.001292755943723023,
+            0.0017752049025148153,
+            0.004760163836181164,
+            0.006922365631908178,
+            -0.020451689139008522,
+            -0.01001833938062191,
+            -0.019754473119974136,
+            -0.003938443958759308,
+            -0.009113618172705173,
+            -0.00839150045067072,
+            0.00650320528075099,
+            -0.002766041085124016,
+            -0.0017285163048654795,
+            0.0005358815542422235,
+            -0.0019080081256106496,
+            0.01054955180734396,
+            -0.019754473119974136,
+            -0.0054158796556293964,
+            -0.021779721602797508,
+            -0.004482107236981392,
+            0.003309704130515456,
+            -0.008354149758815765,
+            -0.016525695100426674,
+            -0.0038077160716056824,
+            0.03320079296827316,
+            -0.017613021656870842,
+            -0.00844960194081068,
+            -0.0062583498656749725,
+            0.0072585237212479115,
+            0.00927962176501751,
+            -0.02071729488670826,
+            0.0012356920633465052,
+            -0.00017806519463192672,
+            -0.01277400553226471,
+            -0.007851988077163696,
+            -0.017945028841495514,
+            -0.02634483017027378,
+            0.004818265326321125,
+            -0.0011962661519646645,
+            -0.01749681867659092,
+            -0.008947613649070263,
+            -0.015687374398112297,
+            -0.006494905333966017,
+            0.00843715202063322,
+            -0.024087175726890564,
+            0.010151143185794353,
+            0.012043587863445282,
+            -0.004193675238639116,
+            0.003143700072541833,
+            0.0063538020476698875,
+            0.016509095206856728,
+            0.005648285150527954,
+            0.005864090286195278,
+            -0.01425144076347351,
+            -0.017314214259386063,
+            0.0022244532592594624,
+            0.005710536614060402,
+            0.014367643743753433,
+            -0.004901267122477293,
+            0.0059180413372814655,
+            0.007980640977621078,
+            0.0020491115283221006,
+            0.01404393557459116,
+            0.025315605103969574,
+            -0.020302286371588707,
+            -0.01065745484083891,
+            0.0044987075962126255,
+            0.015396867878735065,
+            0.003959194757044315,
+            0.007138170767575502,
+            -0.014525347389280796,
+            -0.009935337118804455,
+            -0.001842644065618515,
+            0.007017817813903093,
+            -0.017596419900655746,
+            0.021829521283507347,
+            -0.00685181375592947,
+            0.02546500973403454,
+            0.014757752418518066,
+            0.02282554656267166,
+            0.017546620219945908,
+            -0.010914760641753674,
+            8.55309481266886e-05,
+            -0.0015376117080450058,
+            0.01542176865041256,
+            -0.008362449705600739,
+            -0.006768811959773302,
+            0.0028822438325732946,
+            0.009312822483479977,
+            -0.004901267122477293,
+            0.009379223920404911,
+            0.016807902604341507,
+            -0.005366078112274408,
+            0.0011962661519646645,
+            -0.014351042918860912,
+            -0.004197825212031603,
+            -0.010350347496569157,
+            0.0029361951164901257,
+            0.0033034789375960827,
+            -0.022045327350497246,
+            0.011910784989595413,
+            0.013545923866331577,
+            0.0024859094992280006,
+            -0.026909243315458298,
+            0.0063164508901536465,
+            0.0032578278332948685,
+            0.002357256365939975,
+            0.0122842937707901,
+            0.027224650606513023,
+            0.00406087189912796,
+            -0.006756361573934555,
+            -0.003811866044998169,
+            0.006868414115160704,
+            0.012126590125262737,
+            0.00263738795183599,
+            -0.006793712265789509,
+            -0.021298309788107872,
+            -0.009254720993340015,
+            -0.028917891904711723,
+            0.002919594757258892,
+            -0.026411231607198715,
+            0.0005919078830629587,
+            0.006432653870433569,
+            -0.009702932089567184,
+            -0.024419184774160385,
+            0.01842644065618515,
+            0.016152186319231987,
+            0.0017088033491745591,
+            0.009354324080049992,
+            -0.015703976154327393,
+            -0.00526232598349452,
+            0.015737175941467285,
+            -0.009362624026834965,
+            -0.011869283393025398,
+            -0.029714711010456085,
+            0.009503726847469807,
+            0.0017129534389823675,
+            0.0038741177413612604,
+            -0.010350347496569157,
+            0.0015791127225384116,
+            -0.014757752418518066,
+            0.006196097936481237,
+            -0.0008502515847794712,
+            -0.017247812822461128,
+            0.010200943797826767,
+            0.00368528813123703,
+            0.003071073442697525,
+            -0.003888642881065607,
+            -0.0014660225715488195,
+            0.010275646112859249,
+            0.016658497974276543,
+            -0.0032931037712842226,
+            -0.009960237890481949,
+            -0.0021165506914258003,
+            0.014409144409000874,
+            -0.005781088024377823,
+            0.016450993716716766,
+            -0.0011371271684765816,
+            0.0053162770345807076,
+            -0.003025422338396311,
+            0.002425733022391796,
+            0.0144008444622159,
+            -0.0005436629871837795,
+            0.007864437997341156,
+            0.019588468596339226,
+            -0.006474155001342297,
+            -0.01347122248262167,
+            0.00961992982774973,
+            -0.010649154894053936,
+            0.015977881848812103,
+            0.0031935013830661774,
+            -0.009387524798512459,
+            -0.01018434390425682,
+            0.014683051034808159,
+            0.023987572640180588,
+            -0.022061927244067192,
+            0.009221520274877548,
+            0.009412424638867378,
+            0.004104448016732931,
+            -0.01787862740457058,
+            0.007279274053871632,
+            -0.007075919304043055,
+            -0.0035939859226346016,
+            0.01018434390425682,
+            0.00889781303703785,
+            0.002546085976064205,
+            0.021547315642237663,
+            -0.011371271684765816,
+            -0.0012004162417724729,
+            -0.005573583301156759,
+            0.010840059258043766,
+            -0.017978230491280556,
+            0.0012720053782686591,
+            -0.00913021806627512,
+            -0.004436456132680178,
+            0.005212524440139532,
+            0.018227236345410347,
+            -0.011960585601627827,
+            -0.005980292800813913,
+            0.003371955594047904,
+            -0.014226539991796017,
+            -0.010068140923976898,
+            -0.00997683871537447,
+            -0.0044032554142177105,
+            0.008665407076478004,
+            0.0074369776993989944,
+            -0.018443040549755096,
+            -0.010624254122376442,
+            0.0008336511673405766,
+            -0.02159711718559265,
+            0.0036168114747852087,
+            -0.031109143048524857,
+            0.015322166495025158,
+            -0.0021705019753426313,
+            -0.006059145089238882,
+            -0.0023738567251712084,
+            0.018227236345410347,
+            -0.005540382582694292,
+            0.0023427309934049845,
+            0.0068725645542144775,
+            0.004569259472191334,
+            0.01659209653735161,
+            0.01523086428642273,
+            -0.010516351088881493,
+            -0.01207678858190775,
+            0.025348806753754616,
+            -0.007594681810587645,
+            0.01243369746953249,
+            -0.03698568418622017,
+            0.004237251356244087,
+            -0.014541947282850742,
+            -0.0031395500991493464,
+            -0.018144233152270317,
+            0.013936033472418785,
+            0.003759989980608225,
+            0.005171023774892092,
+            -0.0026436131447553635,
+            0.013338418677449226,
+            0.008715208619832993,
+            -0.008906112983822823,
+            0.008117593824863434,
+            -0.00849110260605812,
+            0.012840406969189644,
+            -0.024883994832634926,
+            0.005025770049542189,
+            0.012383895926177502,
+            -0.0011682529002428055,
+            0.007229472976177931,
+            -0.01662529818713665,
+            0.004868066404014826,
+            0.00908871740102768,
+            -0.021331509575247765,
+            0.02197892591357231,
+            -0.008354149758815765,
+            -0.00166522734798491,
+            -0.020783696323633194,
+            -0.009844035841524601,
+            0.014367643743753433,
+            -0.0006354839424602687,
+            -0.012508398853242397,
+            0.015579472295939922,
+            0.019771073013544083,
+            0.0005976142711006105,
+            0.023057950660586357,
+            -0.017297614365816116,
+            0.010408448986709118,
+            -0.0017492668703198433,
+            -0.00029595394153147936,
+            -0.0074867792427539825,
+            0.0122842937707901,
+            0.0015116736758500338,
+            -0.0006769849569536746,
+            -0.01419333927333355,
+            -0.005133672617375851,
+            -0.013786629773676395,
+            0.02885149046778679,
+            0.02312435209751129,
+            -0.0006583094946108758,
+            0.0031706758309155703,
+            -0.0038077160716056824,
+            -0.0034279818646609783,
+            0.004370054695755243,
+            -0.01914025843143463,
+            -0.01965487003326416,
+            -0.0036707627587020397,
+            -0.013819830492138863,
+            0.018526043742895126,
+            0.018775049597024918,
+            -0.006901614833623171,
+            0.019721271470189095,
+            -0.013745129108428955,
+            0.013778329826891422,
+            -0.0066443090327084064,
+            -0.010051540099084377,
+            -0.003716413863003254,
+            0.002954870695248246,
+            -0.0063164508901536465,
+            0.025149600580334663,
+            0.008233796805143356,
+            -0.0230911523103714,
+            0.013902832753956318,
+            -0.010732156224548817,
+            -0.015703976154327393,
+            0.0015178987523540854,
+            -0.0177956260740757,
+            0.025216002017259598,
+            0.012840406969189644,
+            0.0003945188072975725,
+            -0.01400243490934372,
+            0.016011083498597145,
+            0.008183995261788368,
+            0.002693414455279708,
+            0.011620277538895607,
+            0.02113230526447296,
+            -8.961620187619701e-05,
+            -0.0177956260740757,
+            -0.0012678552884608507,
+            0.019339462742209435,
+            -0.02224453166127205,
+            -0.004177074879407883,
+            0.010640854015946388,
+            0.017729224637150764,
+            0.009893836453557014,
+            -0.0005239500314928591,
+            -0.0070510185323655605,
+            0.0010364872869104147,
+            -0.010781957767903805,
+            -0.01995367743074894,
+            0.01855924353003502,
+            -0.011329771019518375,
+            0.020916501060128212,
+            0.016251789405941963,
+            0.02148091420531273,
+            -0.005843339487910271,
+            -0.021331509575247765,
+            0.005440779961645603,
+            -0.010541251860558987,
+            -0.010914760641753674,
+            0.026361430063843727,
+            -0.009072116576135159,
+            -0.004407405387610197,
+            0.006050844676792622,
+            -0.012275993824005127,
+            0.013537623919546604,
+            -0.014367643743753433,
+            0.0038844929076731205,
+            -0.013413120992481709,
+            -0.022676141932606697,
+            0.009213220328092575,
+            0.004768464248627424,
+            0.013205615803599358,
+            -0.01902405545115471,
+            -0.01662529818713665,
+            0.006976316682994366,
+            0.01812763325870037,
+            -0.00788933876901865,
+            0.008997415192425251,
+            0.013272017240524292,
+            -0.003542109625414014,
+            -0.010350347496569157,
+            0.037417296320199966,
+            0.004872216377407312,
+            -0.002952795708552003,
+            0.006988767068833113,
+            0.006594507489353418,
+            0.024070575833320618,
+            -0.005710536614060402,
+            0.02410377562046051,
+            -0.019970277324318886,
+            7.839797035558149e-05,
+            -0.00401937123388052,
+            -0.0002860974636860192,
+            0.017214611172676086,
+            -0.01736401580274105,
+            0.017812225967645645,
+            -0.0022618041839450598,
+            0.004469656851142645,
+            0.04777594283223152,
+            -0.012715904042124748,
+            -0.028585882857441902,
+            -0.018409840762615204,
+            -0.025481609627604485,
+            -0.0009156156447716057,
+            -0.001976484665647149,
+            -0.01990387588739395,
+            0.0017326663946732879,
+            -0.0029590206686407328,
+            -0.00722117256373167,
+            0.004938618279993534,
+            0.012334095314145088,
+            -0.009653130546212196,
+            0.021331509575247765,
+            0.020551292225718498,
+            0.013446321710944176,
+            0.006951416376978159,
+            0.005648285150527954,
+            -0.005561132915318012,
+            0.0008476577932015061,
+            0.00026612510555423796,
+            -0.0013684951700270176,
+            -0.03213836997747421,
+            0.02172992005944252,
+            -0.015869978815317154,
+            0.00472281314432621,
+            0.011802881956100464,
+            -0.03358260169625282,
+            -0.004668861627578735,
+            -0.006586207542568445,
+            0.009138518944382668,
+            -0.008167395368218422,
+            0.01817743480205536,
+            -0.016741501167416573,
+            -0.00506312120705843,
+            0.008806510828435421,
+            0.013014711439609528,
+            0.00927962176501751,
+            0.015264065004885197,
+            0.016035983338952065,
+            0.010707256384193897,
+            -0.013728528283536434,
+            0.02101610228419304,
+            -0.019007453694939613,
+            -0.0044032554142177105,
+            0.01957186870276928,
+            -0.012358995154500008,
+            0.016002783551812172,
+            0.022974949330091476,
+            -0.00579353841021657,
+            -0.024551987648010254,
+            0.01593638025224209,
+            0.008914412930607796,
+            0.01435934379696846,
+            0.044887471944093704,
+            -0.0012554050190374255,
+            -0.01689920388162136,
+            -0.027324253693223,
+            -0.02224453166127205,
+            -0.01105586439371109,
+            0.013911132700741291,
+            -0.003187276190146804,
+            0.0033657304011285305,
+            0.02194572426378727,
+            -0.0030316475313156843,
+            -0.014840754680335522,
+            0.011869283393025398,
+            0.001804255647584796,
+            0.0643763393163681,
+            0.004258001688867807,
+            -0.005681485868990421,
+            0.018775049597024918,
+            -0.0062583498656749725,
+            -0.012217892333865166,
+            0.006561306770890951,
+            -0.0063164508901536465,
+            -0.012906808406114578,
+            -0.01850944198668003,
+            0.010134542360901833,
+            -0.023140953853726387,
+            0.004278752487152815,
+            0.004756013862788677,
+            0.0025481609627604485,
+            -0.017513418570160866,
+            -0.010109641589224339,
+            -0.022178130224347115,
+            0.0015490244841203094,
+            -0.017812225967645645,
+            -0.018609045073390007,
+            -0.04027256369590759,
+            -0.016691699624061584,
+            0.016401192173361778,
+            -0.00908871740102768,
+            0.015031659044325352,
+            -0.008453751914203167,
+            -0.010350347496569157,
+            -0.009304522536695004,
+            -0.006308150943368673,
+            -0.005374378524720669,
+            0.014981858432292938,
+            0.025863418355584145,
+            -0.004681312013417482,
+            -0.004469656851142645,
+            0.004419855773448944,
+            -0.013753429055213928,
+            0.011487474665045738,
+            -0.010748757049441338,
+            0.011636878363788128,
+            -0.00386996753513813,
+            0.01889125257730484,
+            0.0003467926580924541,
+            -0.008723508566617966,
+            -0.02778906375169754,
+            -0.012740804813802242,
+            -0.0010925136739388108,
+            -0.00773163465783,
+            0.010300545953214169,
+            -0.005229125265032053,
+            0.003774515353143215,
+            0.0015510995872318745,
+            0.0034217569045722485,
+            -0.020534690469503403,
+            -0.007088369689881802,
+            -0.00034860832965932786,
+            0.0030025967862457037,
+            0.008183995261788368,
+            0.006880864500999451,
+            -0.005627534352242947,
+            0.0035732353571802378,
+            -0.017646221444010735,
+            -0.010649154894053936,
+            0.01580357737839222,
+            -0.013628926128149033,
+            -0.002390457084402442,
+            0.008673707023262978,
+            -0.013844731263816357,
+            -0.026577236130833626,
+            0.009578429162502289,
+            0.011421073228120804,
+            0.00367491296492517,
+            -0.021464312449097633,
+            -0.0003724714042618871,
+            0.012989810667932034,
+            0.0008263884810730815,
+            0.020103080198168755,
+            0.022393936291337013,
+            0.00212277565151453,
+            0.00032215146347880363,
+            0.003147850278764963,
+            0.005839189514517784,
+            -0.028270475566387177,
+            0.024850795045495033,
+            0.022775745019316673,
+            -0.011238468810915947,
+            0.015330466441810131,
+            0.00405464693903923,
+            0.0009856485994532704,
+            -0.01948886550962925,
+            0.0042829024605453014,
+            0.021530715748667717,
+            0.00335120502859354,
+            0.007291724439710379,
+            0.0015095985727384686,
+            0.0011630653170868754,
+            0.017397215589880943,
+            0.0006385965389199555,
+            0.004199900198727846,
+            0.0041832998394966125,
+            -0.020816897973418236,
+            0.00827114749699831,
+            -0.011080765165388584,
+            -0.0028594182804226875,
+            -0.007727484684437513,
+            0.0003626149264164269,
+            0.02699224464595318,
+            0.011213568039238453,
+            -0.014309542253613472,
+            0.014450645074248314,
+            0.011578776873648167,
+            -0.00803874246776104,
+            -0.012043587863445282,
+            -0.0007942252559587359,
+            -0.00632060132920742,
+            0.0041438741609454155,
+            0.017264412716031075,
+            -0.0013487822143360972,
+            0.009702932089567184,
+            0.014492146670818329,
+            -0.00175756704993546,
+            -0.012524999678134918,
+            0.01351272314786911,
+            0.011329771019518375,
+            0.013164115138351917,
+            -0.0073622758500278,
+            -0.0023655565455555916,
+            -0.011761381290853024,
+            -0.007424527313560247,
+            0.005627534352242947,
+            0.0022472788114100695,
+            -0.012466898187994957,
+            -0.007735785096883774,
+            -0.0016880529001355171,
+            0.0009555603610351682,
+            0.011728180572390556,
+            -0.007831237278878689,
+            0.0076652332209050655,
+            -0.003666612785309553,
+            0.012491798959672451,
+            0.000912503048311919,
+            0.04883836954832077,
+            0.005610933993011713,
+            0.003147850278764963,
+            0.0052083744667470455,
+            0.0030772986356168985,
+            0.01728101260960102,
+            -0.008026291616261005,
+            0.024120377376675606,
+            0.0076776836067438126,
+            -0.010599353350698948,
+            -0.026710039004683495,
+            -0.015762077644467354,
+            0.021364711225032806,
+            -0.004739413503557444,
+            -0.001546949497424066,
+            0.019389264285564423,
+            0.01574547588825226,
+            0.011761381290853024,
+            0.023705366998910904,
+            0.00614214688539505,
+            0.013039612211287022,
+            0.0011423148680478334,
+            0.02104930393397808,
+            0.004697912372648716,
+            -0.0027556659188121557,
+            0.016011083498597145,
+            0.009586729109287262,
+            0.019156858325004578,
+            -0.004531908314675093,
+            -0.00043420412112027407,
+            0.019920475780963898,
+            -0.0144008444622159,
+            -0.008665407076478004,
+            0.003533809445798397,
+            -0.009163418784737587,
+            -0.0018540568416938186,
+            0.0026124874129891396,
+            -0.022227931767702103,
+            -0.012018687091767788,
+            -0.007715034298598766,
+            0.00489296717569232,
+            0.008922713808715343,
+            0.0010593128390610218,
+            0.02672663889825344,
+            -0.011479174718260765,
+            0.011977186426520348,
+            0.00929622258991003,
+            0.010192643851041794,
+            -0.005370228551328182,
+            0.002778491470962763,
+            0.0021414512302726507,
+            -0.01762962155044079,
+            -0.00683106342330575,
+            -0.018227236345410347,
+            0.007407926954329014,
+            0.01169497985392809,
+            0.026876043528318405,
+            -0.01489055622369051,
+            -0.001508561079390347,
+            -0.011089065112173557,
+            0.010059840977191925,
+            0.014840754680335522,
+            0.005527932196855545
+          ],
+          "keyphrases": [
+            "Switch Transformers",
+            "Lower Compute Regimes"
+          ],
+          "title": "D Switch Transformers in Lower Compute Regimes"
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "6df469f9-914b-4652-b189-540abdf1161a",
+        "properties": {
+          "page_content": "E Relation of Upstream to Downstream Model Performance 32 ",
+          "embedding": [
+            -0.018082372844219208,
+            0.023730995133519173,
+            -0.010728987865149975,
+            0.047936949878931046,
+            -0.03446846455335617,
+            -0.022373968735337257,
+            6.639356433879584e-05,
+            0.06883515417575836,
+            -0.040982190519571304,
+            -0.011789164505898952,
+            -0.011102170683443546,
+            -0.019473325461149216,
+            0.0032950290478765965,
+            -0.020541982725262642,
+            -0.018150225281715393,
+            0.06174468994140625,
+            -0.013655075803399086,
+            -0.0006122520426288247,
+            0.0013835305580869317,
+            0.010686581023037434,
+            -0.011840052902698517,
+            -0.03555408492684364,
+            -0.012162347324192524,
+            0.02306944504380226,
+            -0.017946671694517136,
+            0.0030490681529045105,
+            0.011161540634930134,
+            -0.005020997021347284,
+            -0.02865021489560604,
+            -0.0051015703938901424,
+            0.03867524489760399,
+            0.03236507251858711,
+            -0.004855609033256769,
+            -0.017844894900918007,
+            -0.03636830300092697,
+            0.01401977613568306,
+            0.01845555566251278,
+            -0.04328913614153862,
+            -0.03602904453873634,
+            0.020287541672587395,
+            -0.04037152975797653,
+            0.025851348415017128,
+            -0.025495128706097603,
+            0.022017749026417732,
+            0.03172048553824425,
+            -0.010008067823946476,
+            0.0009758926462382078,
+            0.005945470649749041,
+            0.01615709252655506,
+            -0.010118326172232628,
+            -0.01730208285152912,
+            -0.0348077192902565,
+            0.03915020450949669,
+            0.004249188117682934,
+            -0.02006702497601509,
+            0.01911710575222969,
+            0.04488363862037659,
+            0.1082228347659111,
+            -0.006721520330756903,
+            0.028073478490114212,
+            0.05166877061128616,
+            -0.012043607421219349,
+            0.002571988618001342,
+            -0.03602904453873634,
+            -0.02803955227136612,
+            -0.012798452749848366,
+            -0.06045551598072052,
+            0.016767755150794983,
+            0.011365094222128391,
+            0.017760079354047775,
+            -0.02951531857252121,
+            0.020253615453839302,
+            0.02417202852666378,
+            0.04423905164003372,
+            -0.014910325407981873,
+            0.030634865164756775,
+            0.02975279837846756,
+            0.021084794774651527,
+            -0.006059969775378704,
+            0.02388366125524044,
+            0.0036957759875804186,
+            0.010949504561722279,
+            0.013510892167687416,
+            0.015546430833637714,
+            0.022730188444256783,
+            -0.011585610918700695,
+            -0.027547631412744522,
+            -0.06934403628110886,
+            0.037114664912223816,
+            0.01806541159749031,
+            -0.023188184946775436,
+            -0.007624790538102388,
+            -0.004177096299827099,
+            0.005703750532120466,
+            -0.023001592606306076,
+            0.04657992348074913,
+            0.020525021478533745,
+            0.0024362860713154078,
+            0.011721313931047916,
+            0.04043937847018242,
+            -0.0042152623645961285,
+            0.03552015870809555,
+            0.005071885418146849,
+            0.030007241293787956,
+            0.01556339394301176,
+            -0.0288537684828043,
+            -0.007315219379961491,
+            0.0028221902903169394,
+            0.003935375716537237,
+            0.10937631130218506,
+            -0.014002813957631588,
+            0.006509484723210335,
+            -0.0004625020665116608,
+            0.015554912388324738,
+            -0.024850541725754738,
+            -0.01725119538605213,
+            0.00812519434839487,
+            -0.037352144718170166,
+            0.04885294288396835,
+            -0.022153452038764954,
+            0.013748371042311192,
+            0.01494425069540739,
+            0.005110051482915878,
+            0.016479386016726494,
+            -0.005339049734175205,
+            0.048513684421777725,
+            0.015936575829982758,
+            -0.001556339324451983,
+            0.0006970661925151944,
+            0.005203347187489271,
+            0.01067809946835041,
+            -0.008668004535138607,
+            -0.06221964955329895,
+            -0.0252746120095253,
+            0.0008497316157445312,
+            0.007446681149303913,
+            0.018438592553138733,
+            0.0020864277612417936,
+            0.011458389461040497,
+            -0.008205767720937729,
+            0.00984692107886076,
+            -0.05506133660674095,
+            0.020474132150411606,
+            -0.02403632551431656,
+            0.01446929108351469,
+            0.04261062294244766,
+            0.0181162990629673,
+            0.06048944219946861,
+            -0.0016040473710745573,
+            0.00034826804767362773,
+            0.0008396599441766739,
+            0.00017042340186890215,
+            0.026071865111589432,
+            0.004707184620201588,
+            0.03619867190718651,
+            -0.004170734900981188,
+            -0.013841667212545872,
+            0.004982830490916967,
+            -0.0059836371801793575,
+            0.018862662836909294,
+            0.0063313753344118595,
+            0.0233578123152256,
+            0.034587204456329346,
+            0.006967481225728989,
+            -0.010627211071550846,
+            -0.0009297749493271112,
+            0.022051675245165825,
+            -0.023815808817744255,
+            -0.023289961740374565,
+            0.02822614461183548,
+            0.028718065470457077,
+            -0.03390869125723839,
+            0.01740385964512825,
+            0.01787881925702095,
+            -0.010499989613890648,
+            -0.01446929108351469,
+            -0.013595705851912498,
+            0.018252002075314522,
+            -0.012722120620310307,
+            0.0033459176775068045,
+            -0.046885255724191666,
+            -0.06876730173826218,
+            -0.02227219194173813,
+            -0.048751164227724075,
+            0.007815622724592686,
+            -0.004187698010355234,
+            0.06045551598072052,
+            0.005635899491608143,
+            -0.013010487891733646,
+            0.013476965948939323,
+            0.02454521134495735,
+            0.018557332456111908,
+            0.026258455589413643,
+            0.008621356450021267,
+            -0.022187378257513046,
+            -0.0016793198883533478,
+            0.0016931021818891168,
+            0.03494342416524887,
+            0.016029872000217438,
+            -0.0022645373828709126,
+            -0.0023875178303569555,
+            -0.020626798272132874,
+            0.01029643602669239,
+            -0.002057802863419056,
+            0.024613061919808388,
+            0.007561180274933577,
+            0.04990463703870773,
+            0.038302063941955566,
+            -0.024731801822781563,
+            -0.04715665802359581,
+            -0.014774622395634651,
+            -0.041083965450525284,
+            -0.04542645066976547,
+            0.03548623248934746,
+            -0.012009681202471256,
+            0.03867524489760399,
+            -0.006895389407873154,
+            0.03418009728193283,
+            -0.03562193736433983,
+            -0.013264930807054043,
+            0.0021479178685694933,
+            -0.01122939120978117,
+            0.024765728041529655,
+            0.012425270862877369,
+            -0.036877185106277466,
+            0.011424464173614979,
+            0.011483834125101566,
+            -0.015376802533864975,
+            0.015453135594725609,
+            -0.009134482592344284,
+            -0.00803613942116499,
+            0.036266524344682693,
+            -0.0024256843607872725,
+            -0.017174862325191498,
+            -0.05119381099939346,
+            -0.00827361922711134,
+            -0.021814195439219475,
+            0.006585817784070969,
+            -0.022577522322535515,
+            0.020100949332118034,
+            -0.01032188069075346,
+            -0.01535135880112648,
+            -0.03378995135426521,
+            0.001905137556605041,
+            0.04837798327207565,
+            -0.003979903180152178,
+            -0.010118326172232628,
+            0.06822448968887329,
+            0.02451128512620926,
+            0.00183834636118263,
+            0.01592809520661831,
+            -0.022781075909733772,
+            0.010457582771778107,
+            -0.012611862272024155,
+            0.03602904453873634,
+            0.03643615171313286,
+            0.003526147687807679,
+            -0.008646801114082336,
+            -0.025732608512043953,
+            -0.00999110471457243,
+            0.002476572757586837,
+            -0.004346724599599838,
+            -0.04895471781492233,
+            0.018268965184688568,
+            -0.005572288762778044,
+            -0.006488281302154064,
+            0.03667363151907921,
+            0.022679299116134644,
+            -0.004376409575343132,
+            0.002985457656905055,
+            0.02875199168920517,
+            0.04389979690313339,
+            -0.0567236952483654,
+            -0.055807702243328094,
+            0.016377609223127365,
+            0.011186984367668629,
+            -0.022611448541283607,
+            -0.02164456807076931,
+            -0.001569061540067196,
+            0.01508843433111906,
+            0.004217382986098528,
+            0.022085601463913918,
+            -0.011314205825328827,
+            0.003244140651077032,
+            -0.010169214569032192,
+            -0.008358432911336422,
+            0.022611448541283607,
+            0.01256097387522459,
+            -0.04423905164003372,
+            -0.0008041440160013735,
+            0.01435055211186409,
+            0.009261703118681908,
+            -0.028429698199033737,
+            0.01769222877919674,
+            0.0023853974416851997,
+            0.011585610918700695,
+            0.022340042516589165,
+            -0.011882460676133633,
+            0.025851348415017128,
+            0.010499989613890648,
+            -0.0031190398149192333,
+            0.011670424602925777,
+            0.008214249275624752,
+            -0.022899815812706947,
+            -0.02259448543190956,
+            -0.010898616164922714,
+            -0.00786651112139225,
+            0.004537556320428848,
+            1.3956232578493655e-05,
+            -0.01950725167989731,
+            0.029549244791269302,
+            -0.01735297217965126,
+            -0.02132227271795273,
+            -0.037216443568468094,
+            -0.011899422854185104,
+            -0.010262510739266872,
+            0.02904035896062851,
+            0.03738607093691826,
+            -0.013799259439110756,
+            0.004073198884725571,
+            0.009482220746576786,
+            0.011068244464695454,
+            -0.0639159306883812,
+            0.053093649446964264,
+            0.022730188444256783,
+            0.012798452749848366,
+            -0.019422436133027077,
+            0.065815769135952,
+            0.0029282080940902233,
+            0.01191638596355915,
+            -0.007900436408817768,
+            -0.02832792140543461,
+            -0.007022610399872065,
+            -0.013188597746193409,
+            0.025003207847476006,
+            -0.05492563545703888,
+            -0.027954738587141037,
+            -0.030787531286478043,
+            -0.020558945834636688,
+            0.013604187406599522,
+            -0.016055315732955933,
+            -0.002962133614346385,
+            -0.03562193736433983,
+            -0.017344491556286812,
+            0.010584804229438305,
+            -0.04563000425696373,
+            -0.017285121604800224,
+            -0.0030617902521044016,
+            0.0011131855426356196,
+            0.04135537147521973,
+            0.0010888014221563935,
+            -0.017522599548101425,
+            0.00937196146696806,
+            0.026903044432401657,
+            0.05119381099939346,
+            -0.003013022243976593,
+            -0.006912352051585913,
+            0.001623130519874394,
+            0.006526447832584381,
+            -0.005326327867805958,
+            0.005203347187489271,
+            1.655697815294843e-05,
+            -0.038302063941955566,
+            0.027852961793541908,
+            -0.025444241240620613,
+            -0.02541031502187252,
+            -0.003863283898681402,
+            0.008252415806055069,
+            0.02179723232984543,
+            -0.00820152647793293,
+            -0.03684325888752937,
+            0.019625991582870483,
+            0.03070271760225296,
+            -0.004711425397545099,
+            -0.0031169194262474775,
+            -0.00047098350478336215,
+            -0.01473221555352211,
+            -0.031025011092424393,
+            -0.03928590565919876,
+            0.013646594248712063,
+            -0.0467156246304512,
+            0.011042800731956959,
+            -0.03514697775244713,
+            0.0077901785261929035,
+            -0.0003336906083859503,
+            -0.004987071268260479,
+            0.012688194401562214,
+            0.05207587778568268,
+            0.005958192981779575,
+            0.02770029567182064,
+            -0.008154879324138165,
+            -0.03314536437392235,
+            0.006424670573323965,
+            -0.03426491096615791,
+            0.07972528785467148,
+            -0.005631658714264631,
+            -2.9949991585453972e-05,
+            0.005415382329374552,
+            0.007684160489588976,
+            -0.03314536437392235,
+            -0.029108211398124695,
+            0.032924845814704895,
+            0.0081039909273386,
+            0.0018171428237110376,
+            0.026156678795814514,
+            0.03207670524716377,
+            -0.007463643793016672,
+            -0.03565586358308792,
+            0.002756459405645728,
+            -0.0015775428619235754,
+            0.0106102479621768,
+            -0.019727768376469612,
+            0.018235038965940475,
+            -0.04077863693237305,
+            0.033772990107536316,
+            0.011186984367668629,
+            -0.021763307973742485,
+            0.0152580626308918,
+            -0.025817422196269035,
+            -0.0146643640473485,
+            -0.02870110422372818,
+            0.03088930808007717,
+            0.02946443110704422,
+            0.028480585664510727,
+            0.0007718086126260459,
+            0.04030367732048035,
+            -0.018608221784234047,
+            0.024731801822781563,
+            0.013756852596998215,
+            -0.0024362860713154078,
+            0.028158292174339294,
+            0.04033760353922844,
+            -0.031109824776649475,
+            0.0006530688260681927,
+            -0.010737469419836998,
+            0.0035855176392942667,
+            -0.03450239077210426,
+            0.0014619836583733559,
+            -0.022255228832364082,
+            -0.022085601463913918,
+            0.00032229372300207615,
+            0.013587224297225475,
+            0.005992118734866381,
+            0.0025889514945447445,
+            0.004779276438057423,
+            0.0024914152454584837,
+            -0.01139901950955391,
+            -0.031245527788996696,
+            -0.05601125583052635,
+            -0.004643573891371489,
+            -0.015766948461532593,
+            0.03714859113097191,
+            -0.040507230907678604,
+            0.014961213804781437,
+            0.018557332456111908,
+            -0.016564199700951576,
+            0.010949504561722279,
+            0.005415382329374552,
+            0.02407025173306465,
+            -0.04963323101401329,
+            -0.028599325567483902,
+            0.015173248946666718,
+            0.041457150131464005,
+            -0.030770568177103996,
+            0.009931734763085842,
+            0.00635257875546813,
+            0.0024723319802433252,
+            0.01610620506107807,
+            -0.0006970661925151944,
+            0.02817525528371334,
+            0.014214849099516869,
+            -0.008795225992798805,
+            -0.0056740655563771725,
+            0.018472518771886826,
+            0.020999979227781296,
+            0.028904657810926437,
+            -0.015105397440493107,
+            -0.05760576203465462,
+            -0.016275832429528236,
+            0.02074553817510605,
+            -0.00039306050166487694,
+            0.02222130261361599,
+            0.004588444717228413,
+            0.03231418505311012,
+            0.037589624524116516,
+            0.0037572660949081182,
+            0.020202726125717163,
+            0.0066960761323571205,
+            -0.056045182049274445,
+            -0.01574150286614895,
+            0.011008874513208866,
+            -0.023951511830091476,
+            -0.015079952776432037,
+            -0.005763120483607054,
+            0.004503630567342043,
+            -0.0410161167383194,
+            0.009736662730574608,
+            -0.00865104142576456,
+            0.026258455589413643,
+            0.02751370519399643,
+            0.00678089028224349,
+            0.011195465922355652,
+            -0.015410728752613068,
+            -0.006013322155922651,
+            -0.02016880176961422,
+            -0.026309344917535782,
+            0.026529861614108086,
+            -0.02317122183740139,
+            -0.025681721046566963,
+            -0.0036258043255656958,
+            0.01223019789904356,
+            0.0015839040279388428,
+            0.02498624473810196,
+            -0.018574295565485954,
+            -0.007200720254331827,
+            0.01993132196366787,
+            -0.0116789061576128,
+            -0.009405887685716152,
+            -0.0017874578479677439,
+            0.006874185521155596,
+            -0.0054662711918354034,
+            -0.015207174234092236,
+            0.004766554571688175,
+            0.03504519909620285,
+            -0.0038590431213378906,
+            0.009109037928283215,
+            -0.01178068295121193,
+            0.0034116485621780157,
+            0.06299994140863419,
+            -0.027089634910225868,
+            -0.04437475651502609,
+            0.04600318521261215,
+            0.051227737218141556,
+            -0.0011746757663786411,
+            -0.01542769093066454,
+            -0.019337622448801994,
+            -0.03356943652033806,
+            0.03823421150445938,
+            -0.043933723121881485,
+            0.014206367544829845,
+            -0.012891748920083046,
+            0.024002401158213615,
+            -0.001378229702822864,
+            -0.001866971142590046,
+            -0.020525021478533745,
+            0.04257669672369957,
+            0.03589334338903427,
+            -0.011331168003380299,
+            0.005279679782688618,
+            -0.0004161193501204252,
+            0.015224137343466282,
+            -0.02245878241956234,
+            0.02803955227136612,
+            0.0029388098046183586,
+            -0.010304917581379414,
+            0.005962433759123087,
+            -0.01664053276181221,
+            -0.012476159259676933,
+            -0.0033607601653784513,
+            0.013748371042311192,
+            -0.007289774715900421,
+            0.035350531339645386,
+            -0.04118574410676956,
+            -0.0035473511088639498,
+            0.0041262079030275345,
+            0.005890341475605965,
+            0.009838439524173737,
+            -0.00034800299908965826,
+            -0.036266524344682693,
+            -0.024494322016835213,
+            -0.029260877519845963,
+            0.015605800785124302,
+            -0.04708880931138992,
+            -0.01988043263554573,
+            0.03504519909620285,
+            0.008663764223456383,
+            -0.01782793179154396,
+            -0.02069464884698391,
+            -0.007603587117046118,
+            -0.004728388041257858,
+            0.013799259439110756,
+            -0.009838439524173737,
+            0.0021447374019771814,
+            -0.013672038912773132,
+            0.023103371262550354,
+            -0.015105397440493107,
+            0.02454521134495735,
+            -0.04183033108711243,
+            0.011144577525556087,
+            -0.014316625893115997,
+            0.010796839371323586,
+            0.03999834507703781,
+            0.017675265669822693,
+            -0.018557332456111908,
+            -0.008455969393253326,
+            -0.028785917907953262,
+            -0.005207587964832783,
+            -0.005614695604890585,
+            0.016097722575068474,
+            -0.02164456807076931,
+            0.010355805978178978,
+            0.01518173050135374,
+            0.011170021258294582,
+            -0.014995139092206955,
+            -0.011025837622582912,
+            0.0367075577378273,
+            -0.002381156897172332,
+            0.021865084767341614,
+            0.014503217302262783,
+            -0.01115305908024311,
+            0.015419209375977516,
+            0.009880846366286278,
+            0.03952338546514511,
+            0.00752301374450326,
+            -0.02937961556017399,
+            -0.02846362441778183,
+            -0.005334808956831694,
+            0.030770568177103996,
+            -0.02917606197297573,
+            -0.0024680914357304573,
+            0.048038724809885025,
+            -0.019761692732572556,
+            0.015673652291297913,
+            -0.02464698813855648,
+            -0.020016135647892952,
+            0.0055468445643782616,
+            -0.022764114663004875,
+            -0.05774146318435669,
+            -0.018489481881260872,
+            -0.019965248182415962,
+            -0.026088828220963478,
+            0.03952338546514511,
+            -0.027157485485076904,
+            -0.048649389296770096,
+            0.014291182160377502,
+            0.012179309502243996,
+            0.009363480843603611,
+            -0.025342464447021484,
+            -0.00943981297314167,
+            -0.02230611816048622,
+            -0.01774311624467373,
+            -0.00170476408675313,
+            0.0032823069486767054,
+            0.004656296223402023,
+            -0.021576715633273125,
+            -0.03976086527109146,
+            0.004270391538739204,
+            0.028141330927610397,
+            0.00511429226025939,
+            0.02435861900448799,
+            0.0042152623645961285,
+            0.013595705851912498,
+            0.008769781328737736,
+            0.01161105465143919,
+            0.009117519482970238,
+            -0.04325520992279053,
+            -0.004329761490225792,
+            -0.006263523828238249,
+            -0.040066197514534,
+            -0.010313399136066437,
+            -0.012484640814363956,
+            0.004085920751094818,
+            0.01854036934673786,
+            0.007993732579052448,
+            0.0026313585694879293,
+            -0.014265737496316433,
+            0.005521400365978479,
+            -0.00511429226025939,
+            0.010389731265604496,
+            0.02188204601407051,
+            -0.0005422803806141019,
+            -0.0357576385140419,
+            0.019388511776924133,
+            0.031584784388542175,
+            0.01644546166062355,
+            -0.014749177731573582,
+            -0.019778655841946602,
+            0.02798866480588913,
+            0.022526634857058525,
+            -0.020558945834636688,
+            -0.04094826430082321,
+            0.02927783876657486,
+            -0.026953931897878647,
+            0.018268965184688568,
+            0.029023397713899612,
+            0.004105004016309977,
+            -0.02354440465569496,
+            0.008163360878825188,
+            7.255584205267951e-05,
+            -0.0025211002212017775,
+            -0.011288761161267757,
+            -0.014146997593343258,
+            0.0001689656637609005,
+            -0.008379636332392693,
+            0.003568554762750864,
+            0.02880288101732731,
+            -0.016716865822672844,
+            -0.014588030986487865,
+            0.02530853822827339,
+            0.01982954517006874,
+            0.0038060343358665705,
+            0.001432298682630062,
+            -0.034298837184906006,
+            -0.005754638928920031,
+            0.010864690877497196,
+            -0.024714838713407516,
+            0.01063569262623787,
+            0.009176889434456825,
+            -0.00511429226025939,
+            -0.03302662447094917,
+            -0.0002263477217638865,
+            -0.01921888254582882,
+            -0.016954345628619194,
+            0.030193831771612167,
+            0.023001592606306076,
+            0.0009414369123987854,
+            0.012806934304535389,
+            -0.027361039072275162,
+            -0.016954345628619194,
+            -0.016606608405709267,
+            0.006607021205127239,
+            -0.0021701816003769636,
+            -0.023714032024145126,
+            -0.00695051858201623,
+            -0.014978175982832909,
+            0.015699096024036407,
+            0.031008047983050346,
+            -0.005360253155231476,
+            -0.00413680961355567,
+            0.00024556342395953834,
+            0.0027331355959177017,
+            0.00522030983120203,
+            -0.026631638407707214,
+            0.0023853974416851997,
+            0.017607415094971657,
+            0.009354999288916588,
+            0.012315012514591217,
+            0.008684967644512653,
+            -0.0008836572524160147,
+            -0.04505326971411705,
+            0.020253615453839302,
+            -0.030380424112081528,
+            -0.008795225992798805,
+            0.03972693905234337,
+            3.036412272194866e-05,
+            -0.010542397387325764,
+            0.019625991582870483,
+            -0.0036872944328933954,
+            -0.04172855615615845,
+            0.016894975677132607,
+            -0.002285741036757827,
+            -0.030787531286478043,
+            0.018676072359085083,
+            -0.020253615453839302,
+            0.008099749684333801,
+            0.014910325407981873,
+            0.026156678795814514,
+            -0.03009205497801304,
+            -0.007018369622528553,
+            0.020609835162758827,
+            -0.028548438102006912,
+            0.0010400332976132631,
+            0.0032632239162921906,
+            0.018659109249711037,
+            0.0036046006716787815,
+            0.007366107776761055,
+            -0.051566995680332184,
+            -0.028768954798579216,
+            0.014893362298607826,
+            -0.0010236005764454603,
+            0.0018341057002544403,
+            0.05794501677155495,
+            0.02412113919854164,
+            -0.07450073957443237,
+            0.01711549237370491,
+            0.026394158601760864,
+            0.025020169094204903,
+            -0.027157485485076904,
+            -0.009142964147031307,
+            0.036300450563430786,
+            0.021050868555903435,
+            0.009126001037657261,
+            -0.005398419685661793,
+            0.0022242506965994835,
+            -0.023103371262550354,
+            0.021525828167796135,
+            0.013290374539792538,
+            -0.014392958953976631,
+            -0.036775410175323486,
+            -0.024138102307915688,
+            -0.025088021531701088,
+            -0.0032971494365483522,
+            -0.015139322727918625,
+            -0.018438592553138733,
+            0.03518090397119522,
+            -0.005597732961177826,
+            0.0011502917623147368,
+            0.029142137616872787,
+            -0.002459609881043434,
+            -0.003973542246967554,
+            0.01532591413706541,
+            0.019863469526171684,
+            -0.014783103950321674,
+            -0.002510498510673642,
+            -0.015453135594725609,
+            0.0020228170324116945,
+            0.026716452091932297,
+            -0.024002401158213615,
+            0.015656689181923866,
+            0.0029282080940902233,
+            0.028904657810926437,
+            -0.00970273744314909,
+            0.011042800731956959,
+            -0.01720878854393959,
+            0.00021826387092005461,
+            -0.014986657537519932,
+            0.034451503306627274,
+            -0.006848741322755814,
+            -0.016284313052892685,
+            -0.036504004150629044,
+            0.03847169131040573,
+            0.020779462531208992,
+            -0.014384477399289608,
+            -0.046037111431360245,
+            -0.01535135880112648,
+            -0.028904657810926437,
+            0.0007373528787866235,
+            0.008328747935593128,
+            0.010720506310462952,
+            0.014867917634546757,
+            0.017030678689479828,
+            -0.007794418837875128,
+            0.0002812118618749082,
+            -0.009066631086170673,
+            -0.03367121145129204,
+            0.010211621411144733,
+            -0.038878798484802246,
+            0.01736145280301571,
+            -0.008167601190507412,
+            0.007785937748849392,
+            0.01697978936135769,
+            -0.038030657917261124,
+            -0.010576322674751282,
+            -0.03156781941652298,
+            0.011195465922355652,
+            -0.011008874513208866,
+            0.015800872817635536,
+            -0.030109018087387085,
+            0.011568647809326649,
+            0.020558945834636688,
+            -0.0038081547245383263,
+            0.033874765038490295,
+            -0.006047247909009457,
+            -0.01859125867486,
+            -0.05183839797973633,
+            -0.008027657866477966,
+            0.028531474992632866,
+            0.03986264392733574,
+            -0.0026186364702880383,
+            0.027394965291023254,
+            -0.004440019838511944,
+            0.012238679453730583,
+            0.00418133707717061,
+            -0.021474938839673996,
+            0.027666371315717697,
+            -0.02354440465569496,
+            0.005682547111064196,
+            -0.049565382301807404,
+            -0.03891272470355034,
+            -0.057809315621852875,
+            -0.03155085816979408,
+            -0.0028836806304752827,
+            -0.010330361314117908,
+            0.01270515751093626,
+            0.015758465975522995,
+            0.02103390544652939,
+            -0.0023557126987725496,
+            0.007035332731902599,
+            -0.004804720636457205,
+            -0.028768954798579216,
+            0.03060094080865383,
+            -0.004745350684970617,
+            0.003000300144776702,
+            -0.013078339397907257,
+            0.04047330468893051,
+            0.011220909655094147,
+            0.0116789061576128,
+            0.00045879147364757955,
+            -0.025003207847476006,
+            -0.05160091817378998,
+            0.007472125347703695,
+            -0.008481413125991821,
+            0.006424670573323965,
+            -0.004749591462314129,
+            -0.027479778975248337,
+            0.005249994806945324,
+            0.01587720587849617,
+            -0.0055001964792609215,
+            -0.0029790964908897877,
+            -0.024239879101514816,
+            -0.008905484341084957,
+            -0.03915020450949669,
+            -0.006318653002381325,
+            -0.005004033911973238,
+            0.008778262883424759,
+            -0.027428891509771347,
+            0.011729794554412365,
+            0.011857016012072563,
+            0.008065824396908283,
+            -0.009948697872459888,
+            -0.025206761434674263,
+            -0.007484847214072943,
+            -0.005097329616546631,
+            -0.01518173050135374,
+            0.01363811269402504,
+            -0.026343271136283875,
+            -0.027479778975248337,
+            -0.02627541869878769,
+            0.014791585505008698,
+            -0.016428498551249504,
+            -0.01921888254582882,
+            -0.005020997021347284,
+            0.024901429191231728,
+            0.020728575065732002,
+            0.02764940820634365,
+            0.0014895482454448938,
+            0.024019362404942513,
+            0.027598518878221512,
+            -0.00041585430153645575,
+            -0.0012404067674651742,
+            0.007200720254331827,
+            -0.0023281481117010117,
+            0.0031529655680060387,
+            -0.01328189391642809,
+            0.010949504561722279,
+            0.003430731827393174,
+            0.009032705798745155,
+            -0.026088828220963478,
+            -0.005691028665751219,
+            -0.00013152816973160952,
+            -0.02016880176961422,
+            0.0123998261988163,
+            -0.04257669672369957,
+            -0.0047368695959448814,
+            0.024392545223236084,
+            -0.014189404435455799,
+            0.025902237743139267,
+            0.031686559319496155,
+            -0.012442233972251415,
+            0.009583997540175915,
+            0.0020567427854985,
+            -0.02751370519399643,
+            -0.010805320926010609,
+            -0.0046817404218018055,
+            0.021288348361849785,
+            9.351089829578996e-06,
+            -0.019049255177378654,
+            -0.008684967644512653,
+            -0.029430504888296127,
+            0.01494425069540739,
+            0.02312033250927925,
+            0.014409921132028103,
+            -0.04420512542128563,
+            0.0029918185900896788,
+            0.0025529053527861834,
+            -0.008337229490280151,
+            -0.01663205213844776,
+            -0.010601767338812351,
+            -0.020813388749957085,
+            0.01731056533753872,
+            -0.016182536259293556,
+            -0.00791315920650959,
+            0.015758465975522995,
+            0.01532591413706541,
+            0.01332430075854063,
+            0.01630127616226673,
+            -0.0074254777282476425,
+            -0.013417595997452736,
+            0.0008804767276160419,
+            0.005771602038294077,
+            0.039082352072000504,
+            0.023849735036492348,
+            0.041457150131464005,
+            -0.006560373120009899,
+            0.0028285514563322067,
+            0.0011555926175788045,
+            -0.019710805267095566,
+            0.0015934455441311002,
+            -0.010525434277951717,
+            -0.0007967227720655501,
+            -0.015648208558559418,
+            -0.00786651112139225,
+            0.005979396402835846,
+            0.024765728041529655,
+            0.006475559435784817,
+            0.01877784915268421,
+            0.007985251024365425,
+            -0.0190662182867527,
+            0.002330268267542124,
+            -0.0035240272991359234,
+            -0.009897809475660324,
+            -0.0021468577906489372,
+            -0.035350531339645386,
+            0.0176413394510746,
+            -0.01753956265747547,
+            0.009176889434456825,
+            0.00035197866964153945,
+            -0.01806541159749031,
+            -0.008210008032619953,
+            0.01077987626194954,
+            -0.030295608565211296,
+            0.013587224297225475,
+            -0.01044061966240406,
+            -0.025783497840166092,
+            -0.04081256315112114,
+            0.038505617529153824,
+            0.01735297217965126,
+            -0.007756252773106098,
+            0.02988850139081478,
+            0.01611468568444252,
+            -0.013315819203853607,
+            -0.014367514289915562,
+            -0.009668811224400997,
+            -0.010508471168577671,
+            0.01032188069075346,
+            0.012586417607963085,
+            0.01854036934673786,
+            0.005491715390235186,
+            0.0019242207054048777,
+            0.003341676900163293,
+            0.003956579603254795,
+            -0.01508843433111906,
+            0.0004513702297117561,
+            -0.007811381947249174,
+            0.0052542355842888355,
+            -0.021763307973742485,
+            0.005330568179488182,
+            -0.04115181788802147,
+            -0.0017514119390398264,
+            0.01387559249997139,
+            0.01921888254582882,
+            0.0008030838216654956,
+            0.00867648608982563,
+            0.014783103950321674,
+            -0.022170415148139,
+            0.007124387193471193,
+            -0.016462422907352448,
+            0.0058097681030631065,
+            -0.015376802533864975,
+            0.012645787559449673,
+            0.01418092381209135,
+            0.017056122422218323,
+            0.013553299009799957,
+            -0.0017132455250248313,
+            0.016886495053768158,
+            0.044510457664728165,
+            -0.01325644925236702,
+            -0.0012393465731292963,
+            0.024155065417289734,
+            0.019371548667550087,
+            -0.005678306333720684,
+            -0.009804514236748219,
+            0.01887962594628334,
+            0.011288761161267757,
+            -0.02417202852666378,
+            0.017844894900918007,
+            0.014749177731573582,
+            0.022204341366887093,
+            0.052890095859766006,
+            -0.005555325653403997,
+            -0.005652862135320902,
+            0.012510084547102451,
+            0.009685774333775043,
+            -0.009634885936975479,
+            -0.012501603923738003,
+            0.025613868609070778,
+            -0.02127138525247574,
+            -0.028633251786231995,
+            -0.010576322674751282,
+            -0.008718892931938171,
+            -0.01456258725374937,
+            -0.014418402686715126,
+            0.009779069572687149,
+            0.009100556373596191,
+            -0.005529881455004215,
+            0.004864090587943792,
+            0.016420016065239906,
+            -0.0031466044019907713,
+            0.031025011092424393,
+            -0.013307337649166584,
+            -0.03265344351530075,
+            0.004554518964141607,
+            0.009736662730574608,
+            -0.00592426722869277,
+            -0.0024532489478588104,
+            -0.01882873848080635,
+            0.0024256843607872725,
+            0.050651002675294876,
+            -0.0063865045085549355,
+            0.0011004634434357285,
+            -0.02398543804883957,
+            -0.004673258867114782,
+            0.007489087991416454,
+            0.006933555472642183,
+            0.005741917062550783,
+            0.022662336006760597,
+            0.021915972232818604,
+            0.00367669272236526,
+            -0.004149531479924917,
+            -0.0003323653945699334,
+            -0.028768954798579216,
+            -0.010550878010690212,
+            0.0036809334997087717,
+            0.016411535441875458,
+            0.005245754029601812,
+            0.013383670710027218,
+            0.007378829643130302,
+            0.007785937748849392,
+            -0.01187397912144661,
+            -0.027004821226000786,
+            -0.0057758428156375885,
+            -0.005724953953176737,
+            0.012806934304535389,
+            -0.01187397912144661,
+            0.0048174429684877396,
+            -0.012162347324192524,
+            -0.021050868555903435,
+            0.03185619041323662,
+            0.009244740940630436,
+            -0.005279679782688618,
+            0.002268778160214424,
+            -0.007065017241984606,
+            0.015105397440493107,
+            -0.010067437775433064,
+            0.0019496649038046598,
+            0.0029133656062185764,
+            0.0357576385140419,
+            -0.007344903890043497,
+            0.012213235720992088,
+            0.02103390544652939,
+            0.01835377886891365,
+            -0.04261062294244766,
+            -0.015410728752613068,
+            0.02985457517206669,
+            -0.024019362404942513,
+            0.01845555566251278,
+            0.0058394530788064,
+            0.007489087991416454,
+            -0.023137295618653297,
+            -0.00018818136595655233,
+            0.00791315920650959,
+            0.0047368695959448814,
+            0.026343271136283875,
+            0.02330692484974861,
+            0.04037152975797653,
+            0.009507664479315281,
+            0.00305754947476089,
+            -0.006420430261641741,
+            0.0064373929053545,
+            -0.0031699282117187977,
+            -0.0003058609727304429,
+            -0.014409921132028103,
+            0.008468691259622574,
+            0.000756436085794121,
+            -0.013103784061968327,
+            -0.007086221128702164,
+            0.00927018467336893,
+            -0.0110852075740695,
+            0.027683334425091743,
+            0.004630851559340954,
+            0.00511429226025939,
+            -0.005860656965523958,
+            -0.007476366125047207,
+            0.020016135647892952,
+            -0.02985457517206669,
+            -0.010313399136066437,
+            -0.017191825434565544,
+            0.0015054509276524186,
+            -0.004073198884725571,
+            -0.007031091954559088,
+            0.012747564353048801,
+            -0.006284727249294519,
+            -0.006072692107409239,
+            -0.004342483822256327,
+            -0.014011295512318611,
+            -0.006276246160268784,
+            0.006725761108100414,
+            -0.02627541869878769,
+            -0.0035473511088639498,
+            -0.01032188069075346,
+            -0.027717258781194687,
+            -0.03448542580008507,
+            -0.01721726916730404,
+            0.03151693195104599,
+            -0.01184853445738554,
+            -0.005033718887716532,
+            0.006310171447694302,
+            -0.0026143956929445267,
+            0.017149418592453003,
+            0.011170021258294582,
+            -0.016224944964051247,
+            0.011738276109099388,
+            -0.0012340457178652287,
+            0.006861463654786348,
+            0.0008582130540162325,
+            -0.00943981297314167,
+            -0.011373575776815414,
+            -0.019388511776924133,
+            -0.00015664110833313316,
+            -0.004550278186798096,
+            -0.012756045907735825,
+            0.004775035660713911,
+            0.007120146416127682,
+            0.012527047656476498,
+            0.008646801114082336,
+            -0.0012404067674651742,
+            0.00803189817816019,
+            -0.012730601243674755,
+            -0.013917999342083931,
+            0.004982830490916967,
+            0.00899029802531004,
+            -0.01821807585656643,
+            -0.00784530770033598,
+            0.04142322391271591,
+            -0.022051675245165825,
+            -0.0028200699016451836,
+            -0.007192238699644804,
+            0.009405887685716152,
+            -0.009075112640857697,
+            -0.0048725721426308155,
+            0.012408307753503323,
+            -0.005644380580633879,
+            -0.009931734763085842,
+            -0.036775410175323486,
+            -0.0013983730459585786,
+            0.043221283704042435,
+            -0.019015328958630562,
+            0.01759045198559761,
+            0.0007219803519546986,
+            0.02651289850473404,
+            -0.001526654465124011,
+            0.013756852596998215,
+            -0.03555408492684364,
+            -0.009762107394635677,
+            0.003136002691462636,
+            0.015487060882151127,
+            -0.006238079629838467,
+            -0.016699902713298798,
+            0.03438365086913109,
+            -0.017794005572795868,
+            -0.022187378257513046,
+            0.00362368393689394,
+            -0.03470594435930252,
+            0.007981009781360626,
+            0.0283618476241827,
+            0.011619536206126213,
+            -0.015156285837292671,
+            -0.013578742742538452,
+            0.008222730830311775,
+            0.00781986303627491,
+            -0.002936689415946603,
+            0.0009637005859985948,
+            0.01740385964512825,
+            0.023476552218198776,
+            0.036741483956575394,
+            -0.03312839940190315,
+            -0.021441012620925903,
+            0.0008216369315050542,
+            0.015190212056040764,
+            -0.004999793134629726,
+            0.029956351965665817,
+            0.030431311577558517,
+            0.012043607421219349,
+            0.010584804229438305,
+            -0.018031485378742218,
+            0.009405887685716152,
+            0.00692507391795516,
+            0.011543204076588154,
+            0.02079642564058304,
+            -0.0011524120345711708,
+            0.024613061919808388,
+            0.01582631841301918,
+            0.005890341475605965,
+            -0.0006186130922287703,
+            -0.02040628157556057,
+            -0.007667197845876217,
+            0.016988271847367287,
+            0.004847127944231033,
+            -9.746999421622604e-05,
+            0.005936989560723305,
+            0.0035240272991359234,
+            0.019337622448801994,
+            0.006004840601235628,
+            -0.006632465403527021,
+            0.008231211453676224,
+            0.0004375879361759871,
+            -0.009329554624855518,
+            -0.0025677478406578302,
+            0.014248774386942387,
+            0.023663144558668137,
+            -0.00216382066719234,
+            0.00714135030284524,
+            -0.021491901949048042,
+            0.01639457233250141,
+            -0.0011566528119146824,
+            -0.0040647173300385475,
+            -0.0034116485621780157,
+            -0.021966861560940742,
+            0.02664860151708126,
+            0.0004627671150956303,
+            0.012348937802016735,
+            -0.0005454609054140747,
+            0.0036194431595504284,
+            0.021661529317498207,
+            -0.02359529212117195,
+            -7.586889114463702e-05,
+            -0.009982624091207981,
+            -0.010788357816636562,
+            -0.028497548773884773,
+            0.0033586397767066956,
+            0.03402743116021156,
+            0.013392152264714241,
+            -0.008735856041312218,
+            -0.0009186430834233761,
+            -0.011746757663786411,
+            0.016767755150794983,
+            0.010279472917318344,
+            -0.002824310678988695,
+            0.0009409068152308464,
+            -0.017912745475769043,
+            -0.003307751379907131,
+            0.0028073478024452925,
+            -0.005623177159577608,
+            -0.007938602939248085,
+            -0.010898616164922714,
+            -0.0037297015078365803,
+            0.018608221784234047,
+            0.006602780427783728,
+            -0.007128627970814705,
+            0.005987877957522869,
+            -0.013002007268369198,
+            0.001110004959627986,
+            -0.000263983994955197,
+            0.013332782313227654,
+            -0.016191018745303154,
+            -0.02398543804883957,
+            0.016428498551249504,
+            -0.0005292932037264109,
+            -0.009176889434456825,
+            -0.019473325461149216,
+            -0.010754432529211044,
+            -0.022390931844711304,
+            0.010041994042694569,
+            -0.006963240448385477,
+            0.006013322155922651,
+            0.001960266614332795,
+            -0.023018555715680122,
+            0.009397406131029129,
+            0.007955566048622131,
+            -0.01122939120978117,
+            0.031160714104771614,
+            -0.024918392300605774,
+            0.018268965184688568,
+            -0.0009165227529592812,
+            0.014698289334774017,
+            0.02069464884698391,
+            -0.0013474845327436924,
+            -0.03255166485905647,
+            0.006670631933957338,
+            -0.009516146034002304,
+            0.01578390970826149,
+            -0.0068402597680687904,
+            -0.002148978179320693,
+            -0.006428911350667477,
+            -0.04077863693237305,
+            -0.017912745475769043,
+            -0.0069717220030725,
+            -0.0007262210128828883,
+            0.055468443781137466,
+            -0.009456776082515717,
+            -0.001471525290980935,
+            -0.022153452038764954,
+            -0.0021839640103280544,
+            -0.033874765038490295,
+            0.0011142457369714975,
+            0.017675265669822693,
+            0.005860656965523958,
+            -0.0045630005188286304,
+            0.010881653986871243,
+            0.010101363062858582,
+            0.0056740655563771725,
+            0.013841667212545872,
+            -0.00018341056420467794,
+            -0.04213566333055496,
+            0.005975155625492334,
+            -0.012951118871569633,
+            -0.006382263731211424,
+            0.021424051374197006,
+            0.011042800731956959,
+            0.0014047340955585241,
+            0.018082372844219208,
+            0.0029833372682332993,
+            0.002404480706900358,
+            -0.010864690877497196,
+            -0.017132455483078957,
+            0.017336009070277214,
+            0.0024150824174284935,
+            0.007662957068532705,
+            -0.004830164834856987,
+            -0.004520593211054802,
+            0.013146190904080868,
+            0.0011661943281069398,
+            0.012603380717337132,
+            0.015173248946666718,
+            -0.008044620975852013,
+            -0.002330268267542124,
+            0.024630025029182434,
+            -0.005330568179488182,
+            0.0022730189375579357,
+            -0.07456858456134796,
+            0.038166359066963196,
+            0.028141330927610397,
+            -0.009329554624855518,
+            -0.00543658621609211,
+            -0.015376802533864975,
+            -0.015512505546212196,
+            -0.0046817404218018055,
+            -0.024494322016835213,
+            -0.014130034483969212,
+            0.02583438530564308,
+            -0.014791585505008698,
+            0.00025165945407934487,
+            0.004363687243312597,
+            0.0048725721426308155,
+            0.005385697353631258,
+            0.006233838852494955,
+            -0.00752301374450326,
+            0.017675265669822693,
+            -0.021542789414525032,
+            0.002406601095572114,
+            -0.010712025687098503,
+            -0.01610620506107807,
+            -0.0016803800826892257,
+            0.007573902141302824,
+            -0.008701929822564125,
+            0.0065773362293839455,
+            0.030380424112081528,
+            -0.015062990598380566,
+            0.031873151659965515,
+            0.003867524676024914,
+            -0.016911938786506653,
+            -0.04420512542128563,
+            0.009999586269259453,
+            0.021474938839673996,
+            0.015266544185578823,
+            -0.029498355463147163,
+            0.0037106184754520655,
+            -0.03165263682603836,
+            -0.013239486142992973,
+            -0.01329885609447956,
+            -0.012815415859222412,
+            -0.024256842210888863,
+            0.00754421716555953,
+            0.010025030933320522,
+            -0.017047641798853874,
+            -0.010008067823946476,
+            -0.02717444859445095,
+            -0.01025402918457985,
+            -0.00856198649853468,
+            -0.023256035521626472,
+            -0.03270433098077774,
+            -0.008239693008363247,
+            0.0036957759875804186,
+            -0.01349392905831337,
+            -0.0123998261988163,
+            -0.019659915938973427,
+            -0.003948098048567772,
+            0.002957893069833517,
+            0.007824104279279709,
+            -0.006089654751121998,
+            0.02030450478196144,
+            -0.01587720587849617,
+            -0.008358432911336422,
+            0.007226164452731609,
+            0.012781490571796894,
+            0.003311991924419999,
+            0.022492708638310432,
+            0.014138516038656235,
+            0.02965102158486843,
+            0.02427380532026291,
+            0.00798949133604765,
+            0.0037805901374667883,
+            0.024307731539011,
+            0.007768974639475346,
+            -0.026241492480039597,
+            -0.019371548667550087,
+            -0.0016570561565458775,
+            0.022390931844711304,
+            -0.005249994806945324,
+            -0.010126807726919651,
+            0.017573488876223564,
+            -0.0007357626454904675,
+            0.015071472153067589,
+            -0.030109018087387085,
+            -0.013951925560832024,
+            0.0027055710088461637,
+            -0.030770568177103996,
+            -0.008595912717282772,
+            -0.0008905483991838992,
+            0.00686570443212986,
+            -0.0012181430356577039,
+            -0.006623983848839998,
+            -0.011144577525556087,
+            0.03155085816979408,
+            0.0074424403719604015,
+            0.0021267144475132227,
+            0.010423657484352589,
+            -0.004864090587943792,
+            0.00592426722869277,
+            0.004486667923629284,
+            -0.032907884567976,
+            0.00645859632641077,
+            -0.004978589713573456,
+            0.014537142589688301,
+            -0.0169289018958807,
+            -0.005589251406490803,
+            0.0024299249053001404,
+            -0.0051015703938901424,
+            -4.8271169362124056e-05,
+            0.02250967174768448,
+            -0.01602138951420784,
+            0.0030639106407761574,
+            0.0016517553012818098,
+            -0.0036618502344936132,
+            0.006967481225728989,
+            0.03619867190718651,
+            0.0044527421705424786,
+            -0.006085413973778486,
+            -0.021254422143101692,
+            -0.005712232086807489,
+            -0.0049446639604866505,
+            -0.007645994424819946,
+            -0.042508844286203384,
+            -0.005996359512209892,
+            -0.0010326120536774397,
+            -0.02588527463376522,
+            -0.024528248235583305,
+            -0.007896196097135544,
+            0.01187397912144661,
+            -0.01253552921116352,
+            -0.008455969393253326,
+            -0.014503217302262783,
+            0.007187997922301292,
+            0.0006684414111077785,
+            0.010991912335157394,
+            -0.015156285837292671,
+            0.008197286166250706,
+            9.607851097825915e-05,
+            -0.0023832772858440876,
+            -0.010008067823946476,
+            0.0110852075740695,
+            0.010728987865149975,
+            0.008061583153903484,
+            0.008040379732847214,
+            -0.008214249275624752,
+            0.015911132097244263,
+            -0.005512918811291456,
+            0.005275439005345106,
+            0.010822284035384655,
+            -0.01223019789904356,
+            -0.013375189155340195,
+            -0.0007044874364510179,
+            -0.009499182924628258,
+            -0.020050061866641045,
+            0.010423657484352589,
+            0.009880846366286278,
+            -0.009736662730574608,
+            0.011042800731956959,
+            0.0027607001829892397,
+            0.0019083180231973529,
+            -0.007484847214072943,
+            0.012968081049621105,
+            0.08813884854316711,
+            -0.0005459910025820136,
+            0.018387705087661743,
+            0.01601290889084339,
+            -0.0056571029126644135,
+            -0.030923234298825264,
+            -0.0014535022201016545,
+            -0.03418009728193283,
+            -0.008566227741539478,
+            -0.014384477399289608,
+            -0.026190605014562607,
+            -0.01898140273988247,
+            -0.014291182160377502,
+            0.01549554243683815,
+            0.006496762856841087,
+            0.0023048240691423416,
+            0.00865528266876936,
+            0.006280486937612295,
+            0.031008047983050346,
+            -0.010839246213436127,
+            -0.01044061966240406,
+            -0.02680126577615738,
+            0.011161540634930134,
+            0.000879946630448103,
+            0.019659915938973427,
+            -0.00891396589577198,
+            0.019439399242401123,
+            0.024528248235583305,
+            0.016199499368667603,
+            0.03136426582932472,
+            0.025393351912498474,
+            0.009388924576342106,
+            0.011806127615272999,
+            -0.005644380580633879,
+            -0.0026822469662874937,
+            0.014757659286260605,
+            -0.014520179480314255,
+            0.001473645563237369,
+            -0.002315425779670477,
+            -0.01387559249997139,
+            0.0033925652969628572,
+            -0.016555719077587128,
+            0.0036682114005088806,
+            0.012315012514591217,
+            -0.003678813111037016,
+            -0.0025762293953448534,
+            0.001580723444931209,
+            -6.443886377383024e-05,
+            -0.005360253155231476,
+            -0.000742653792258352,
+            -0.008540783077478409,
+            -0.008829151280224323,
+            0.00013795548875350505,
+            0.016080759465694427,
+            0.006547651253640652,
+            0.00362368393689394,
+            0.013858629390597343,
+            -0.008451728150248528,
+            -0.013214042410254478,
+            0.0054111420176923275,
+            -0.00035091847530566156,
+            0.003598239738494158,
+            0.014588030986487865,
+            -0.004520593211054802,
+            -0.007226164452731609,
+            0.002864597365260124,
+            0.00047389898099936545,
+            0.0006806334131397307,
+            -0.007768974639475346,
+            0.022950705140829086,
+            0.01935458555817604,
+            0.0010066378163173795,
+            0.012382864020764828,
+            0.01893051527440548,
+            0.0022624169941991568,
+            -0.03173745051026344,
+            -0.003996866289526224,
+            -0.007361866999417543,
+            0.013587224297225475,
+            -0.0021171728149056435,
+            -0.01682712510228157,
+            0.00807854626327753,
+            -0.009583997540175915,
+            -0.005576529540121555,
+            -0.009346517734229565,
+            -0.002582590328529477,
+            0.017912745475769043,
+            0.004461223725229502,
+            -0.024138102307915688,
+            0.01577542908489704,
+            0.011263317428529263,
+            -0.003971422091126442,
+            0.0013326420448720455,
+            -0.015724539756774902,
+            -0.004083800595253706,
+            -0.006602780427783728,
+            0.010813802480697632,
+            -0.0031529655680060387,
+            -0.013383670710027218,
+            -0.01774311624467373,
+            0.029532281681895256,
+            -0.005869138054549694,
+            0.005402660463005304,
+            -0.004821683745831251,
+            0.0011301483027637005,
+            0.02206863835453987,
+            0.014604994095861912,
+            -0.005606214515864849,
+            0.013502410613000393,
+            -0.013443040661513805,
+            -0.0063653006218373775,
+            0.028293995186686516,
+            -0.006051488686352968,
+            -0.007645994424819946,
+            0.007692642044275999,
+            -0.007302497047930956,
+            -0.024239879101514816,
+            0.0036872944328933954,
+            -0.02412113919854164,
+            0.004707184620201588,
+            -0.015122360549867153,
+            0.022238265722990036,
+            0.015164767391979694,
+            -0.008621356450021267,
+            -0.003013022243976593,
+            0.0026546823792159557,
+            -0.002449008170515299,
+            -0.00984692107886076,
+            0.0027925053145736456,
+            0.015945058315992355,
+            -0.004804720636457205,
+            0.015071472153067589,
+            -0.010092882439494133,
+            -0.024324694648385048,
+            -0.0005115352687425911,
+            -0.011068244464695454,
+            -0.01411307230591774,
+            0.0018902950687333941,
+            -0.027581557631492615,
+            0.007081980351358652,
+            -0.019948285073041916,
+            0.000553942343685776,
+            -0.015724539756774902,
+            -0.006992925424128771,
+            -0.007171035278588533,
+            0.020525021478533745,
+            -0.003299269825220108,
+            0.0041198465041816235,
+            0.0022730189375579357,
+            -0.007811381947249174,
+            0.01184853445738554,
+            -0.027021784335374832,
+            0.003909931518137455,
+            0.01677623577415943,
+            0.0033183530904352665,
+            -0.01835377886891365,
+            -0.0013994332402944565,
+            0.023900622501969337,
+            -0.008871558122336864,
+            -0.0029833372682332993,
+            -0.007400033064186573,
+            0.008812189102172852,
+            0.007192238699644804,
+            0.003044827375560999,
+            -0.007764733862131834,
+            0.023442627862095833,
+            0.01701371558010578,
+            0.008468691259622574,
+            0.019422436133027077,
+            -0.012925674207508564,
+            -0.00535177206620574,
+            0.022204341366887093,
+            -0.001378229702822864,
+            -0.014163960702717304,
+            0.0011460509849712253,
+            -0.005271198693662882,
+            -0.009354999288916588,
+            0.006242320407181978,
+            0.01518173050135374,
+            -0.010389731265604496,
+            -0.002245454117655754,
+            -0.014477772638201714,
+            0.005771602038294077,
+            -0.011729794554412365,
+            -0.0025741090066730976,
+            -0.01132268738001585,
+            -0.007730808574706316,
+            0.007349144667387009,
+            -0.005635899491608143,
+            -0.01669142208993435,
+            -0.009100556373596191,
+            -0.006204153876751661,
+            -0.003032105276361108,
+            0.010033512488007545,
+            0.025342464447021484,
+            -0.01387559249997139,
+            0.015716059133410454,
+            -0.0048895347863435745,
+            0.008803707547485828,
+            0.016046835109591484,
+            0.013358226045966148,
+            -0.013061377219855785,
+            -0.01630127616226673,
+            0.0011131855426356196,
+            -0.005805527325719595,
+            0.010762914083898067,
+            0.0016157092759385705,
+            0.010270991362631321,
+            0.0018309251172468066,
+            0.009219296276569366,
+            -0.01625886932015419,
+            0.002569868229329586,
+            0.0031423636246472597,
+            -0.008392359130084515,
+            -0.018760886043310165,
+            -0.010627211071550846,
+            -0.009066631086170673,
+            0.021763307973742485,
+            0.018285928294062614,
+            0.007879232987761497,
+            -0.016852568835020065,
+            0.02106783166527748,
+            -0.007450921926647425,
+            0.007289774715900421,
+            -0.014443847350776196,
+            0.00716255372390151,
+            0.02580045908689499,
+            -0.004596926271915436,
+            0.0035346290096640587,
+            -0.015758465975522995,
+            -0.005805527325719595,
+            -0.0007999032968655229,
+            0.00023284131020773202,
+            0.014647400937974453,
+            0.011186984367668629,
+            0.013773815706372261,
+            0.000118408483103849,
+            0.013197079300880432,
+            -0.020677685737609863,
+            -0.019422436133027077,
+            -0.021474938839673996,
+            -0.013129227794706821,
+            -0.004965867381542921,
+            0.0026292381808161736,
+            0.0047156657092273235,
+            0.002610154915601015,
+            0.009600959718227386,
+            -0.00987236574292183,
+            -0.005843693856149912,
+            -3.207034387742169e-05,
+            0.008379636332392693,
+            -0.0006069511291570961,
+            0.008328747935593128,
+            -0.013790778815746307,
+            0.010245547629892826,
+            0.007484847214072943,
+            -0.0016284313751384616,
+            0.010186177678406239,
+            0.003610961837694049,
+            -0.006878426298499107,
+            -0.008099749684333801,
+            0.00025099681806750596,
+            -0.010423657484352589,
+            0.000959989964030683,
+            -0.009812995791435242,
+            0.006123580504208803,
+            0.009397406131029129,
+            0.01921888254582882,
+            0.02150886505842209,
+            0.014070665463805199,
+            3.816635944531299e-05,
+            0.027717258781194687,
+            0.006450114771723747,
+            0.022051675245165825,
+            -0.008061583153903484,
+            0.030244721099734306,
+            0.0008868377772159874,
+            -0.022713225334882736,
+            0.011967274360358715,
+            0.0088206697255373,
+            0.0026568027678877115,
+            -0.012959599494934082,
+            -0.016267351806163788,
+            -0.006132062058895826,
+            -0.03412920981645584,
+            -0.02098301611840725,
+            0.005177902989089489,
+            0.006657909601926804,
+            0.021339235827326775,
+            0.008964854292571545,
+            -0.00597091531381011,
+            -0.01830288954079151,
+            0.005936989560723305,
+            -0.00392265385016799,
+            -0.010720506310462952,
+            -0.013180116191506386,
+            0.012289567850530148,
+            -0.0028306718450039625,
+            0.015393765643239021,
+            0.01005895622074604,
+            -0.00754421716555953,
+            -0.0031593265011906624,
+            -0.007739289663732052,
+            -0.010313399136066437,
+            0.02135619893670082,
+            -0.024256842210888863,
+            -0.00801069475710392,
+            -0.0072516086511313915,
+            -0.008222730830311775,
+            -0.009092075750231743,
+            0.003127521136775613,
+            0.013332782313227654,
+            -0.0032653443049639463,
+            0.0030787531286478043,
+            0.004486667923629284,
+            -0.0012563093332573771,
+            0.008210008032619953,
+            -0.01225564256310463,
+            -0.037793178111314774,
+            0.024918392300605774,
+            -0.007539976388216019,
+            0.020541982725262642,
+            0.014910325407981873,
+            0.0020323586650192738,
+            -0.0012139022583141923,
+            -0.018523406237363815,
+            0.011721313931047916,
+            -0.03183922544121742,
+            0.009075112640857697,
+            -0.016470905393362045,
+            0.034875571727752686,
+            -0.013366707600653172,
+            -0.020677685737609863,
+            0.01769222877919674,
+            0.0011725553777068853,
+            -0.011687387712299824,
+            -0.027157485485076904,
+            -0.005432345438748598,
+            0.005780083127319813,
+            0.008430524729192257,
+            0.015537949278950691,
+            -0.02746281772851944,
+            -0.010957986116409302,
+            -0.025851348415017128,
+            -0.004618129692971706,
+            0.015597319230437279,
+            -0.007552698720246553,
+            -5.648753813147778e-06,
+            0.0014651642413809896,
+            0.014435365796089172,
+            0.0029515319038182497,
+            0.012035125866532326,
+            0.0008990298374556005,
+            0.0031805301550775766,
+            -0.001866971142590046,
+            0.010220102965831757,
+            0.005729194730520248,
+            0.019965248182415962,
+            -0.01208601426333189,
+            -0.00860863458365202,
+            0.011644980870187283,
+            0.008880039677023888,
+            -0.03041434846818447,
+            -0.00784530770033598,
+            0.002970615169033408,
+            -0.015275025740265846,
+            0.02888769470155239,
+            0.005644380580633879,
+            -0.010194659233093262,
+            0.01263730600476265,
+            -0.023442627862095833,
+            -0.017624378204345703,
+            -0.009473739191889763,
+            -0.0030363460537046194,
+            -0.012968081049621105,
+            -0.0031020769383758307,
+            -0.0016676579834893346,
+            0.025189798325300217,
+            0.020423242822289467,
+            -0.011687387712299824,
+            0.016581162810325623,
+            -0.010330361314117908,
+            0.004957386292517185,
+            -0.005979396402835846,
+            0.019422436133027077,
+            0.024528248235583305,
+            -0.0010442740749567747,
+            0.006908111274242401,
+            -0.0042555490508675575,
+            0.0058097681030631065,
+            -0.010092882439494133,
+            0.011687387712299824,
+            -0.012102977372705936,
+            0.002580470172688365,
+            -0.00922777783125639,
+            -0.004172855522483587,
+            -0.012204754166305065,
+            0.046274591237306595,
+            0.0030617902521044016,
+            0.011695869266986847,
+            0.026920005679130554,
+            0.0027925053145736456,
+            0.009448294527828693,
+            0.030109018087387085,
+            0.020338429138064384,
+            -0.00865528266876936,
+            -0.026869118213653564,
+            0.002639839891344309,
+            -0.014952732250094414,
+            -0.01046606432646513,
+            0.0013220403343439102,
+            -0.009626404382288456,
+            0.013366707600653172,
+            0.0030066610779613256,
+            0.047461990267038345,
+            0.003341676900163293,
+            -0.009923254139721394,
+            -0.015800872817635536,
+            0.00781986303627491,
+            0.0014651642413809896,
+            0.006513725500553846,
+            -0.004999793134629726,
+            -0.01490184385329485,
+            0.0038738856092095375,
+            -0.013332782313227654,
+            -0.017200306057929993,
+            0.01782793179154396,
+            -0.004639333114027977,
+            -0.013807740993797779,
+            -0.006869944743812084,
+            0.01725967600941658,
+            -0.009719699621200562,
+            0.03009205497801304,
+            -0.003854802343994379,
+            -0.013629631139338017,
+            0.022577522322535515,
+            -0.0021309552248567343,
+            0.015800872817635536,
+            -0.0026016735937446356,
+            -0.005122773814946413,
+            -0.017607415094971657,
+            -0.013790778815746307,
+            0.01602138951420784,
+            0.004406094551086426,
+            -0.008332989178597927,
+            -0.0009811935015022755,
+            0.01229804940521717,
+            0.01153472252190113,
+            0.0058648972772061825,
+            0.015393765643239021,
+            -0.020999979227781296,
+            0.011246354319155216,
+            0.02093212865293026,
+            0.003155085723847151,
+            -0.007421236950904131,
+            0.0005931688356213272,
+            0.0012881146976724267,
+            0.0009610501583665609,
+            -0.0022772594820708036,
+            -0.015156285837292671,
+            -0.005402660463005304,
+            0.0017885180423036218,
+            0.013307337649166584,
+            0.007552698720246553,
+            0.005033718887716532,
+            -0.02188204601407051,
+            0.0017196065746247768,
+            -0.013672038912773132,
+            0.006327134557068348,
+            -0.001542557030916214,
+            0.0017259676242247224,
+            0.017624378204345703,
+            0.005593492183834314,
+            0.00413468899205327,
+            0.005551085341721773,
+            -0.02751370519399643,
+            -0.020541982725262642,
+            -0.004864090587943792,
+            0.00905814953148365,
+            -0.006797852925956249,
+            0.011458389461040497,
+            -0.01115305908024311,
+            0.012433752417564392,
+            0.009075112640857697,
+            0.007026851177215576,
+            -0.010126807726919651,
+            -0.006255042273551226,
+            -0.011008874513208866,
+            -0.010805320926010609,
+            4.200950206723064e-05,
+            -0.010974949225783348,
+            -0.011661943979561329,
+            0.015546430833637714,
+            0.00383783970028162,
+            0.004359446465969086,
+            -0.015173248946666718,
+            -0.0025529053527861834,
+            -0.011076726019382477,
+            -0.007018369622528553,
+            0.0062295980751514435,
+            0.018947478383779526,
+            0.0024574894923716784,
+            -0.0027734222821891308,
+            -0.008256656117737293,
+            -0.006445874460041523,
+            -0.0036300448700785637,
+            -0.022424858063459396,
+            -0.03643615171313286,
+            0.01325644925236702,
+            0.007641753647476435,
+            0.008973335847258568,
+            0.001959206536412239,
+            -0.012577936053276062,
+            -0.028735028579831123,
+            -0.005127014592289925,
+            0.0134684843942523,
+            0.02369706891477108,
+            -0.022475745528936386,
+            0.011907904408872128,
+            0.023849735036492348,
+            0.012951118871569633,
+            -0.0031444840133190155,
+            0.022662336006760597,
+            0.01256097387522459,
+            0.01911710575222969,
+            -0.0024532489478588104,
+            -0.006314412225037813,
+            0.0070607769303023815,
+            -0.012391345575451851,
+            -0.0013050774578005075,
+            -0.009354999288916588,
+            0.008829151280224323,
+            0.0027607001829892397,
+            -0.0023832772858440876,
+            0.007917399518191814,
+            -0.030482200905680656,
+            0.010661136358976364,
+            -0.022153452038764954,
+            0.013366707600653172,
+            0.014359032735228539,
+            -0.008557746186852455,
+            0.00946525763720274,
+            0.0007617369410581887,
+            0.0053093647584319115,
+            0.02317122183740139,
+            -0.002353592310100794,
+            0.009185370989143848,
+            -0.010576322674751282,
+            -0.0018553092377260327,
+            -0.018252002075314522,
+            0.003301390213891864,
+            0.003998986445367336,
+            -0.004596926271915436,
+            -0.012425270862877369,
+            0.02059287205338478,
+            -0.00473262881860137,
+            -0.0001673754013609141,
+            -0.024239879101514816,
+            -0.00029923487454652786,
+            -0.005843693856149912,
+            0.021525828167796135,
+            -0.017726154997944832,
+            -0.0005565927713178098,
+            0.009388924576342106,
+            0.010211621411144733,
+            0.0017916986253112555,
+            -0.0319579653441906,
+            0.023001592606306076,
+            -0.02993938885629177,
+            -0.025427278131246567,
+            -0.005025237333029509,
+            -0.004864090587943792,
+            -0.006564613897353411,
+            -0.022323081269860268,
+            0.012459196150302887,
+            0.012331975623965263,
+            -0.004196179565042257,
+            -0.014723733998835087,
+            -0.0018319853115826845,
+            -0.003093595616519451,
+            -0.00624656118452549,
+            -0.005093088839203119,
+            -0.0032377797178924084,
+            -0.005941230338066816,
+            0.0021161127369850874,
+            -0.010432139039039612,
+            -0.007739289663732052,
+            -0.00784106645733118,
+            0.007213442120701075,
+            0.0018574295099824667,
+            0.01206056959927082,
+            -0.002894282341003418,
+            -0.020677685737609863,
+            0.011950311250984669,
+            0.001987831201404333,
+            0.015724539756774902,
+            0.004719906486570835,
+            0.006738482974469662,
+            0.010941023007035255,
+            0.013790778815746307,
+            -0.011526240967214108,
+            -0.00982147641479969,
+            -0.01648786850273609,
+            -0.005907304584980011,
+            0.013739889487624168,
+            -0.012026644311845303,
+            -0.0040074680000543594,
+            -0.00012821512063965201,
+            -0.020083988085389137,
+            0.005440826993435621,
+            0.011102170683443546,
+            0.005152458790689707,
+            -0.011144577525556087,
+            -0.026292381808161736,
+            0.013790778815746307,
+            0.0058097681030631065,
+            -0.013392152264714241,
+            -0.003216576064005494,
+            0.01682712510228157,
+            0.026190605014562607,
+            0.02641112171113491,
+            0.011195465922355652,
+            0.015291988849639893,
+            -0.009007261134684086,
+            0.031143750995397568,
+            0.0076163094490766525,
+            0.007221923675388098,
+            -0.024765728041529655,
+            0.01574150286614895,
+            -0.0015467978082597256,
+            0.0022221303079277277,
+            -0.00020103600400034338,
+            -0.028582364320755005,
+            -0.009728181175887585,
+            -0.036741483956575394,
+            -0.015953538939356804,
+            -0.013205560855567455,
+            -0.004758073017001152,
+            0.0037212201859802008,
+            0.006416189484298229,
+            0.00021481829753611237,
+            0.011840052902698517,
+            -0.0017810967983677983,
+            0.009142964147031307,
+            0.006869944743812084,
+            -0.015444654040038586,
+            0.0025062577333301306,
+            -0.009354999288916588,
+            -0.0076884012669324875,
+            -0.0030087814666330814,
+            -0.03156781941652298,
+            0.0018690915312618017,
+            -0.02483357861638069,
+            -0.010627211071550846,
+            -0.008973335847258568,
+            -0.004953145515173674,
+            -0.0028985231183469296,
+            0.0023896382190287113,
+            -0.017794005572795868,
+            -0.018506444990634918,
+            -0.0110852075740695,
+            -0.020491095259785652,
+            -0.013460002839565277,
+            -0.0070607769303023815,
+            -0.003810275113210082,
+            -0.002542303642258048,
+            0.03251773864030838,
+            0.006717279553413391,
+            -0.003307751379907131,
+            -0.032348111271858215,
+            0.014223330654203892,
+            -0.005381457041949034,
+            -0.014842473901808262,
+            -0.00867648608982563,
+            0.014995139092206955,
+            -0.022662336006760597,
+            0.018319852650165558,
+            0.00943981297314167,
+            -0.021254422143101692,
+            0.013621150515973568,
+            0.027903851121664047,
+            0.025732608512043953,
+            0.010220102965831757,
+            -0.00850685779005289,
+            -6.033896170265507e-06,
+            0.002220009919255972,
+            0.001824564067646861,
+            0.015809355303645134,
+            0.011509277857840061,
+            0.015308951027691364,
+            -0.002446887781843543,
+            0.015436172485351562,
+            -0.02780207432806492,
+            -0.0015902650775387883,
+            -0.0076163094490766525,
+            -0.005894582252949476,
+            0.008405080996453762,
+            -0.008108231239020824,
+            -0.018913552165031433,
+            -0.025953125208616257,
+            -0.00010051800200017169,
+            0.024002401158213615,
+            -0.016886495053768158,
+            -0.014359032735228539,
+            -0.010881653986871243,
+            -0.0004110835143364966,
+            -0.007366107776761055,
+            0.010576322674751282,
+            0.022916778922080994,
+            -0.0005102100549265742,
+            0.0060218037106096745,
+            0.012289567850530148,
+            -0.02717444859445095,
+            0.01340063288807869,
+            -0.017895782366394997,
+            -0.0074254777282476425,
+            -0.00798949133604765,
+            -0.010881653986871243,
+            0.0033374361228197813,
+            -0.011526240967214108,
+            0.014800066128373146,
+            0.01606379635632038,
+            0.0034158893395215273,
+            -0.013137709349393845,
+            -0.0027522186283022165,
+            -0.020423242822289467,
+            0.00011118603288196027,
+            -0.007752011995762587,
+            0.003678813111037016,
+            0.009957179427146912,
+            0.004340363200753927,
+            0.021525828167796135,
+            0.008786744438111782,
+            -0.03680933639407158,
+            -0.0021691215224564075,
+            0.012925674207508564,
+            -0.00213307561352849,
+            -0.028378808870911598,
+            -0.012713639065623283,
+            0.009134482592344284,
+            0.00038484414108097553,
+            0.003937496338039637,
+            0.003950218204408884,
+            -0.004558759741485119,
+            0.017471712082624435,
+            -0.04043937847018242,
+            -0.01840466819703579,
+            -0.00040207200800068676,
+            -0.016470905393362045,
+            -0.007103183772414923,
+            0.008481413125991821,
+            0.0026037939824163914,
+            0.010499989613890648,
+            0.009668811224400997,
+            -0.018438592553138733,
+            0.01178068295121193,
+            -0.0009260643273591995,
+            -0.01373140886425972,
+            0.0013167394790798426,
+            0.0007988431607373059,
+            0.009448294527828693,
+            0.0021055107936263084,
+            -0.00862983800470829,
+            -0.00786651112139225,
+            -0.0100165493786335,
+            0.004842887166887522,
+            0.029684947803616524,
+            -0.00023959993268363178,
+            -0.005856416188180447,
+            0.005016756244003773,
+            -0.010381249710917473,
+            -0.013273412361741066,
+            -0.01432510744780302,
+            0.0006949458038434386,
+            0.013460002839565277,
+            0.0073067378252744675,
+            0.014749177731573582,
+            -0.011170021258294582,
+            -0.005071885418146849,
+            0.011390537954866886,
+            -0.01170435082167387,
+            0.013409114442765713,
+            -0.01625886932015419,
+            -0.015995945781469345,
+            -0.007993732579052448,
+            0.0025401832535862923,
+            0.015139322727918625,
+            0.00922777783125639,
+            -0.000635575910564512,
+            0.0333828441798687,
+            -0.008803707547485828,
+            0.020474132150411606,
+            -0.01425725594162941,
+            -0.01697130873799324,
+            0.005283920560032129,
+            -0.016564199700951576,
+            -0.007531495299190283,
+            -0.004859849810600281,
+            -0.020558945834636688,
+            0.007166794501245022,
+            0.018726961687207222,
+            -0.004189818166196346,
+            0.016148611903190613,
+            0.008167601190507412,
+            -0.0039523388259112835,
+            0.0030893548391759396,
+            -0.00927018467336893,
+            -0.012069051153957844,
+            0.004098643083125353,
+            0.0146643640473485,
+            -0.01759045198559761,
+            0.017522599548101425,
+            0.004516352899372578,
+            0.007285534404218197,
+            0.015512505546212196,
+            0.012985044158995152,
+            -0.02680126577615738,
+            -0.011865497566759586,
+            0.022441819310188293,
+            -0.01206056959927082,
+            0.005597732961177826,
+            0.011840052902698517,
+            0.0038993298076093197,
+            -0.009397406131029129,
+            -0.013510892167687416,
+            0.0011259076418355107,
+            -0.00991477258503437,
+            0.013544817455112934,
+            -0.00015120769967325032,
+            -0.041083965450525284,
+            0.007997972890734673,
+            -0.006441633682698011,
+            0.003487981390208006,
+            0.0027331355959177017,
+            -0.0058394530788064,
+            0.023256035521626472,
+            0.02259448543190956,
+            -0.015504023991525173,
+            -0.00588610116392374,
+            -0.013960406184196472,
+            0.012798452749848366,
+            0.008693449199199677,
+            0.007976769469678402,
+            -0.02493535540997982,
+            -0.003610961837694049,
+            0.007213442120701075,
+            -0.0064373929053545,
+            -0.00473262881860137,
+            0.014545624144375324,
+            -0.009346517734229565,
+            -0.0013951925793662667,
+            -0.003825117601081729,
+            0.0015043907333165407,
+            0.013044414110481739,
+            0.0036703317891806364,
+            0.015529467724263668,
+            -0.01950725167989731,
+            0.013833185657858849,
+            -0.0058648972772061825,
+            0.0010368528310209513,
+            0.009897809475660324,
+            -0.014197885990142822,
+            -0.026139715686440468,
+            0.01630127616226673,
+            0.00692507391795516,
+            0.00037477246951311827,
+            -0.023086408153176308,
+            -0.00645859632641077,
+            -0.01993132196366787,
+            0.0029409301932901144,
+            -0.00028465743525885046,
+            0.0038738856092095375,
+            0.025088021531701088,
+            -0.000675862655043602,
+            -0.015860242769122124,
+            -0.005797046236693859,
+            0.001527714659459889,
+            0.0015923853497952223,
+            0.007145591080188751,
+            -0.0193036962300539,
+            -0.025088021531701088,
+            -0.0050464412197470665,
+            -0.013858629390597343,
+            0.018319852650165558,
+            -0.017760079354047775,
+            0.015716059133410454,
+            -0.003555832663550973,
+            0.008922447450459003,
+            0.00562741793692112,
+            -0.0029812168795615435,
+            0.03294181078672409,
+            -0.015597319230437279,
+            0.010304917581379414,
+            0.007548457942903042,
+            -0.011382057331502438,
+            -0.011449907906353474,
+            0.01774311624467373,
+            -0.010508471168577671,
+            0.0032505018170922995,
+            0.0038463210221379995,
+            -0.00929562933743,
+            0.023476552218198776,
+            0.0027500982396304607,
+            0.014757659286260605,
+            -0.01070354413241148,
+            -0.019914358854293823,
+            0.0032907885033637285,
+            0.010686581023037434,
+            0.0013496049214154482,
+            0.00812519434839487,
+            -0.02888769470155239,
+            0.00483864638954401,
+            -0.009643367491662502,
+            0.02106783166527748,
+            0.009728181175887585,
+            0.0018097215797752142,
+            -0.02999027818441391,
+            0.0008857776410877705,
+            -0.014070665463805199,
+            0.02651289850473404,
+            0.005165180657058954,
+            0.003301390213891864,
+            -0.01139901950955391,
+            -0.003975662402808666,
+            0.012823897413909435,
+            -0.005754638928920031,
+            -0.009643367491662502,
+            -0.0038039139471948147,
+            0.011178502812981606,
+            0.004821683745831251,
+            -0.013426077552139759,
+            -0.013934962451457977,
+            -0.011882460676133633,
+            0.010160733014345169,
+            0.0012520686723291874,
+            0.0037424236070364714,
+            -0.004323400557041168,
+            0.007484847214072943,
+            -0.011763720773160458,
+            0.022950705140829086,
+            0.009668811224400997,
+            0.01592809520661831,
+            0.01893051527440548,
+            -0.008176082745194435,
+            0.009643367491662502,
+            0.008837632834911346,
+            0.01046606432646513,
+            0.016174055635929108,
+            0.022373968735337257,
+            0.021899009123444557,
+            0.019710805267095566,
+            -0.021949898451566696,
+            -0.0010585864074528217,
+            -0.006602780427783728,
+            -0.0073237004689872265,
+            -0.011958792805671692,
+            -0.018998365849256516,
+            0.001702643814496696,
+            -0.03372209891676903,
+            -0.019015328958630562,
+            -0.0017206667689606547,
+            0.007090461906045675,
+            0.013027451001107693,
+            -0.014333589002490044,
+            0.012985044158995152,
+            0.008583190850913525,
+            0.00036417070077732205,
+            -0.014486254192888737,
+            -0.010949504561722279,
+            -0.012544010765850544,
+            0.004367928020656109,
+            0.021610641852021217,
+            0.004153772257268429,
+            0.0049446639604866505,
+            -0.01301896944642067,
+            0.00937196146696806,
+            0.02822614461183548,
+            0.0015351359033957124,
+            -0.00953310914337635,
+            0.0058479346334934235,
+            -0.004073198884725571,
+            0.008752819150686264,
+            -0.012162347324192524,
+            -0.010669617913663387,
+            0.000553412246517837,
+            0.005754638928920031,
+            -0.013748371042311192,
+            0.02140708826482296,
+            -0.004196179565042257,
+            0.018285928294062614,
+            -2.4102455427055247e-06,
+            -0.0007198599632829428,
+            0.0024511285591870546,
+            0.003971422091126442,
+            0.02198382467031479,
+            0.005275439005345106,
+            -0.0008550324710085988,
+            -0.023730995133519173,
+            -0.0005356542533263564,
+            -0.0029939389787614346,
+            -0.020134875550866127,
+            0.004906497895717621,
+            0.0069589996710419655,
+            -0.004359446465969086,
+            0.02717444859445095,
+            0.013960406184196472,
+            0.0013156792847439647,
+            0.005669824779033661,
+            0.0014089748729020357,
+            0.007366107776761055,
+            -0.008498376235365868,
+            -0.00946525763720274,
+            0.0041262079030275345,
+            -0.0027755426708608866,
+            0.0051185330376029015,
+            -0.027089634910225868,
+            0.013570261187851429,
+            0.0050634038634598255,
+            0.0038166360463947058,
+            0.006208394654095173,
+            0.008693449199199677,
+            -0.005996359512209892,
+            0.0012128421803936362,
+            0.006975962780416012,
+            0.008260896429419518,
+            0.001960266614332795,
+            0.031075898557901382,
+            -0.027428891509771347,
+            -0.00036019503022544086,
+            -0.005945470649749041,
+            -0.005233032163232565,
+            -0.0032547423616051674,
+            -0.01816718839108944,
+            -0.013782297261059284,
+            -0.00392053322866559,
+            -0.001568001345731318,
+            0.023900622501969337,
+            0.013578742742538452,
+            0.01611468568444252,
+            0.01149231567978859,
+            -0.006344097200781107,
+            -0.018862662836909294,
+            0.009176889434456825,
+            -0.0035367493983358145,
+            -0.00619991309940815,
+            0.031686559319496155,
+            0.016199499368667603,
+            -0.0036830538883805275,
+            0.01887962594628334,
+            -0.0051354956813156605,
+            0.011178502812981606,
+            -0.016174055635929108,
+            0.001324160723015666,
+            -0.006869944743812084,
+            -0.0031741689890623093,
+            0.013426077552139759,
+            -0.002298463135957718,
+            0.007930121384561062,
+            -0.0024574894923716784,
+            -0.00526695791631937,
+            0.0076714386232197285,
+            -0.0031317619141191244,
+            -0.004162253811955452,
+            0.013977369293570518,
+            -0.003746664384379983,
+            -0.011161540634930134,
+            -0.015368320979177952,
+            0.011127614416182041,
+            -0.01328189391642809,
+            0.006216876208782196,
+            -0.0036745723336935043,
+            -0.007925881072878838,
+            -0.002419323194772005,
+            0.03060094080865383,
+            -0.0020779462065547705,
+            0.017794005572795868,
+            0.004851368721574545,
+            -0.02203471213579178,
+            -0.02083035185933113,
+            -0.008167601190507412,
+            0.03151693195104599,
+            -0.00507612619549036,
+            -0.009159926325082779,
+            -0.019795618951320648,
+            -0.01091557927429676,
+            0.004268271382898092,
+            -0.012823897413909435,
+            -0.0003002950397785753,
+            -5.681884431396611e-05,
+            0.003638526424765587,
+            -0.0007946024416014552,
+            -0.01408762764185667,
+            -0.004656296223402023,
+            0.00031142690568231046,
+            0.01253552921116352,
+            0.005059163086116314,
+            0.002565627684816718,
+            0.014308144338428974,
+            0.005470511503517628,
+            -0.02751370519399643,
+            -0.006551892030984163,
+            0.013315819203853607,
+            0.012527047656476498,
+            0.012348937802016735,
+            -0.002095969393849373,
+            -0.007900436408817768,
+            -0.02659771218895912,
+            -0.010381249710917473,
+            -0.0362325981259346,
+            -0.0013803500914946198,
+            -0.023052481934428215,
+            -0.018896589055657387,
+            -0.02103390544652939,
+            -0.021254422143101692,
+            -0.010330361314117908,
+            -0.01067809946835041,
+            0.025681721046566963,
+            0.010983430780470371,
+            0.0026716452557593584,
+            0.00791315920650959,
+            -0.023374775424599648,
+            0.0027119319420307875,
+            -0.009541589766740799,
+            -0.006963240448385477,
+            0.02359529212117195,
+            0.00011436655768193305,
+            0.016793198883533478,
+            -0.0014216969721019268,
+            -0.011907904408872128,
+            0.007395792752504349,
+            -0.007701123598963022,
+            -0.007662957068532705,
+            0.010932542383670807,
+            -0.012628824450075626,
+            -0.013485447503626347,
+            0.016428498551249504,
+            0.026156678795814514,
+            -0.002173362299799919,
+            0.0005157759878784418,
+            -0.024155065417289734,
+            -0.0011969394981861115,
+            -0.0020355391316115856,
+            0.0009202333749271929,
+            -0.0015828438336029649,
+            0.007421236950904131,
+            0.002241213573142886,
+            0.0034116485621780157,
+            0.02006702497601509,
+            -0.012603380717337132,
+            0.01816718839108944,
+            0.007506051100790501,
+            -0.001392011996358633,
+            0.021491901949048042,
+            0.010949504561722279,
+            0.0016284313751384616,
+            -0.023052481934428215,
+            -0.004363687243312597,
+            -0.010033512488007545,
+            -0.001446080976165831,
+            0.014435365796089172,
+            0.013154672458767891,
+            -0.020016135647892952,
+            -0.025257648900151253,
+            4.452741995919496e-05,
+            -0.001272212015464902,
+            0.009829957969486713,
+            -0.015444654040038586,
+            -0.0020620436407625675,
+            0.0006551892147399485,
+            -0.018557332456111908,
+            0.015571875497698784,
+            0.013417595997452736,
+            0.0033925652969628572,
+            0.01332430075854063,
+            -0.005283920560032129,
+            0.015275025740265846,
+            -0.007743530441075563,
+            0.010203140787780285,
+            -0.003099956549704075,
+            -0.013146190904080868,
+            0.006288968026638031,
+            -0.00784106645733118,
+            -0.01163649931550026,
+            -0.010118326172232628,
+            -0.017369935289025307,
+            -0.021542789414525032,
+            -0.020915165543556213,
+            -0.019235845655202866,
+            0.015291988849639893,
+            -0.011365094222128391,
+            0.009261703118681908,
+            0.004961627069860697,
+            -0.006081173662096262,
+            0.021101756021380424,
+            0.005890341475605965,
+            -0.0017641340382397175,
+            0.007209201343357563,
+            -0.003203853964805603,
+            -0.0008216369315050542,
+            -0.004393372219055891,
+            0.032348111271858215,
+            -0.0013315819669514894,
+            0.024494322016835213,
+            -0.0006445874460041523,
+            0.008205767720937729,
+            0.0016125286929309368,
+            0.0015892048832029104,
+            -0.023137295618653297,
+            0.012908711098134518,
+            -0.011314205825328827,
+            -0.014571068808436394,
+            -0.02412113919854164,
+            -0.00022078178881201893,
+            0.006178709678351879,
+            0.017047641798853874,
+            0.0041431705467402935,
+            -0.0009424970485270023,
+            0.008646801114082336,
+            -0.01950725167989731,
+            0.002160640200600028,
+            0.009007261134684086,
+            -0.00156694115139544,
+            -0.008290581405162811,
+            -0.02030450478196144,
+            -0.0019570861477404833,
+            -0.027394965291023254,
+            0.005568047985434532,
+            0.0007946024416014552,
+            -0.01504602748900652,
+            0.010949504561722279,
+            -0.007904677651822567,
+            0.021339235827326775,
+            0.01830288954079151,
+            0.0011916386429220438,
+            -0.006581577006727457,
+            -0.015300470404326916,
+            -0.015962019562721252,
+            0.013527854345738888,
+            0.015308951027691364,
+            0.011110651306807995,
+            0.006178709678351879,
+            0.005398419685661793,
+            -0.004329761490225792,
+            0.01446929108351469,
+            0.00937196146696806,
+            -0.028599325567483902,
+            1.502469149272656e-05,
+            -0.010983430780470371,
+            0.023680105805397034,
+            -0.02330692484974861,
+            0.014002813957631588,
+            -0.020999979227781296,
+            0.01935458555817604,
+            -0.0003612551954574883,
+            0.012722120620310307,
+            0.009278666228055954,
+            8.89223229023628e-05,
+            -0.013375189155340195,
+            0.005852175410836935,
+            0.001298716408200562,
+            -0.0011555926175788045,
+            0.0048174429684877396,
+            0.022492708638310432,
+            -0.02951531857252121,
+            -0.004893775563687086,
+            -0.005190624855458736,
+            -2.678602686501108e-05,
+            -0.010313399136066437,
+            -0.024188991636037827,
+            0.0033586397767066956,
+            -0.015012102201581001,
+            0.002998179756104946,
+            0.018862662836909294,
+            0.00516094034537673,
+            0.02111871913075447,
+            -0.02330692484974861,
+            -0.009829957969486713,
+            -0.013553299009799957,
+            0.017437785863876343,
+            -0.0029536522924900055,
+            0.0016825003549456596,
+            0.006517966277897358,
+            -0.013909517787396908,
+            0.0058097681030631065,
+            -0.0073237004689872265,
+            -0.00441033486276865,
+            0.06323742121458054,
+            -0.0005393648752942681,
+            -0.007014128845185041,
+            -0.01022858452051878,
+            -0.010092882439494133,
+            0.015317432582378387,
+            0.006929314695298672,
+            -0.003528268076479435,
+            -0.010695062577724457,
+            -0.004397612996399403,
+            -0.023256035521626472,
+            -0.01648786850273609,
+            -0.004217382986098528,
+            -0.0074254777282476425,
+            0.01178068295121193,
+            -0.000947267864830792,
+            -0.009999586269259453,
+            0.003445574315264821,
+            -0.0054111420176923275,
+            -0.00846020970493555,
+            0.006378022953867912,
+            -0.01615709252655506,
+            0.001985711045563221,
+            0.008939409628510475,
+            -0.015554912388324738,
+            0.018252002075314522,
+            -0.014528661035001278,
+            -0.0048174429684877396,
+            0.022170415148139,
+            -0.021288348361849785,
+            0.005394178908318281,
+            0.0011725553777068853,
+            0.01935458555817604,
+            -0.019625991582870483,
+            -0.009456776082515717,
+            0.03173745051026344,
+            -0.012645787559449673,
+            0.008400839753448963,
+            0.00853230245411396,
+            0.006759686395525932,
+            0.0018902950687333941,
+            0.0011736155720427632,
+            0.007412755396217108,
+            0.014859436079859734,
+            -0.01636064611375332,
+            -0.01950725167989731,
+            0.007832585833966732,
+            -0.004448501393198967,
+            -0.0036003601271659136,
+            0.010372769087553024,
+            -0.009278666228055954,
+            0.0025529053527861834,
+            -0.0053560128435492516,
+            0.01470677088946104,
+            -0.005682547111064196,
+            -0.006187191233038902,
+            0.018099335953593254,
+            -0.01039821282029152,
+            0.0053093647584319115,
+            0.02927783876657486,
+            -0.012544010765850544,
+            -0.016911938786506653,
+            -0.0009944457560777664,
+            -0.00522030983120203,
+            -0.013332782313227654,
+            0.01740385964512825,
+            -0.01702219620347023,
+            -0.005525640677660704,
+            -0.02393454872071743,
+            -0.004520593211054802,
+            0.005826731212437153,
+            0.0190662182867527,
+            0.013273412361741066,
+            -0.019337622448801994,
+            0.003812395269051194,
+            -0.011271798983216286,
+            0.023493515327572823,
+            0.0031953726429492235,
+            0.012196272611618042,
+            -0.0010442740749567747,
+            -0.03894665092229843,
+            -0.012764527462422848,
+            -0.03562193736433983,
+            0.014825510792434216,
+            -0.004813202191144228,
+            -0.024952318519353867,
+            -0.02627541869878769,
+            -0.0021161127369850874,
+            -0.014723733998835087,
+            0.007187997922301292,
+            0.012611862272024155,
+            0.01425725594162941,
+            0.012645787559449673,
+            0.03775925189256668,
+            0.0021277745254337788,
+            -0.0004309618379920721,
+            -0.0008476112852804363,
+            0.022289155051112175,
+            0.01149231567978859,
+            -0.01720878854393959,
+            0.009142964147031307,
+            0.0043721687979996204,
+            -0.012187791056931019,
+            -0.004596926271915436,
+            -0.017709191888570786,
+            -0.016479386016726494,
+            0.014842473901808262,
+            -0.01877784915268421,
+            0.007561180274933577,
+            0.01578390970826149,
+            -0.010169214569032192,
+            0.009185370989143848,
+            -0.003369241487234831,
+            -0.005173662211745977,
+            -0.01490184385329485,
+            0.0010071678552776575,
+            -0.008693449199199677,
+            0.01582631841301918,
+            -0.012739082798361778,
+            0.01864214614033699,
+            -0.014172442257404327,
+            -0.00826937798410654,
+            0.021339235827326775,
+            0.022475745528936386,
+            -0.009567034430801868,
+            0.008710411377251148,
+            -0.021525828167796135,
+            0.0008878979715518653,
+            0.014630438759922981,
+            -0.019490288570523262,
+            0.02856540121138096,
+            -0.012925674207508564,
+            0.014613475650548935,
+            0.0011322686914354563,
+            -0.009287147782742977,
+            0.002555025741457939,
+            -0.010550878010690212,
+            0.013672038912773132,
+            -0.0019411834655329585,
+            0.012094495818018913,
+            -0.01615709252655506,
+            0.015809355303645134,
+            -0.011653462424874306,
+            0.012510084547102451,
+            0.009244740940630436,
+            -0.005860656965523958,
+            -0.0005868077860213816,
+            -0.013069857843220234,
+            0.0052542355842888355,
+            -0.003812395269051194,
+            0.023849735036492348,
+            0.006399226374924183,
+            -0.015809355303645134,
+            0.0014821270015090704,
+            0.0028094681911170483,
+            0.015537949278950691,
+            0.011314205825328827,
+            0.007086221128702164,
+            -0.013095302507281303,
+            0.01935458555817604,
+            -0.00987236574292183,
+            0.008998779579997063,
+            -0.0011036439100280404,
+            0.011407501064240932,
+            -0.022577522322535515,
+            -0.00588610116392374,
+            0.007048054598271847,
+            -0.015291988849639893,
+            0.012857822701334953,
+            -0.0076163094490766525,
+            0.0006000599823892117,
+            0.007137109525501728,
+            0.006590058095753193,
+            0.019439399242401123,
+            0.004276752937585115,
+            0.002393878996372223,
+            0.003301390213891864,
+            -0.011619536206126213,
+            -0.0159789826720953,
+            -0.008668004535138607,
+            -0.020185764878988266,
+            0.005088848061859608,
+            0.010237066075205803,
+            0.02064376138150692,
+            0.0008592731901444495,
+            -0.008718892931938171,
+            -0.00801069475710392,
+            -0.02011791244149208,
+            0.00045190032687969506,
+            0.021135682240128517,
+            0.012416789308190346,
+            -0.011543204076588154,
+            -0.011415982618927956,
+            0.020491095259785652,
+            -0.00654341047629714,
+            -0.02103390544652939,
+            -0.012934155762195587,
+            -0.006836019456386566,
+            -0.0027267744299024343,
+            -0.007981009781360626,
+            0.016581162810325623,
+            0.004775035660713911,
+            0.0076332720927894115,
+            0.03613082319498062
+          ],
+          "title": "E Relation of Upstream to Downstream Model Performance",
+          "keyphrases": [
+            "Upstream to Downstream Model Performance"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "690ef77a-eec6-479f-98b8-0c50dba9e84a",
+      "type": "next",
+      "source": {
+        "id": "6df469f9-914b-4652-b189-540abdf1161a",
+        "properties": {
+          "page_content": "E Relation of Upstream to Downstream Model Performance 32 ",
+          "embedding": [
+            -0.018082372844219208,
+            0.023730995133519173,
+            -0.010728987865149975,
+            0.047936949878931046,
+            -0.03446846455335617,
+            -0.022373968735337257,
+            6.639356433879584e-05,
+            0.06883515417575836,
+            -0.040982190519571304,
+            -0.011789164505898952,
+            -0.011102170683443546,
+            -0.019473325461149216,
+            0.0032950290478765965,
+            -0.020541982725262642,
+            -0.018150225281715393,
+            0.06174468994140625,
+            -0.013655075803399086,
+            -0.0006122520426288247,
+            0.0013835305580869317,
+            0.010686581023037434,
+            -0.011840052902698517,
+            -0.03555408492684364,
+            -0.012162347324192524,
+            0.02306944504380226,
+            -0.017946671694517136,
+            0.0030490681529045105,
+            0.011161540634930134,
+            -0.005020997021347284,
+            -0.02865021489560604,
+            -0.0051015703938901424,
+            0.03867524489760399,
+            0.03236507251858711,
+            -0.004855609033256769,
+            -0.017844894900918007,
+            -0.03636830300092697,
+            0.01401977613568306,
+            0.01845555566251278,
+            -0.04328913614153862,
+            -0.03602904453873634,
+            0.020287541672587395,
+            -0.04037152975797653,
+            0.025851348415017128,
+            -0.025495128706097603,
+            0.022017749026417732,
+            0.03172048553824425,
+            -0.010008067823946476,
+            0.0009758926462382078,
+            0.005945470649749041,
+            0.01615709252655506,
+            -0.010118326172232628,
+            -0.01730208285152912,
+            -0.0348077192902565,
+            0.03915020450949669,
+            0.004249188117682934,
+            -0.02006702497601509,
+            0.01911710575222969,
+            0.04488363862037659,
+            0.1082228347659111,
+            -0.006721520330756903,
+            0.028073478490114212,
+            0.05166877061128616,
+            -0.012043607421219349,
+            0.002571988618001342,
+            -0.03602904453873634,
+            -0.02803955227136612,
+            -0.012798452749848366,
+            -0.06045551598072052,
+            0.016767755150794983,
+            0.011365094222128391,
+            0.017760079354047775,
+            -0.02951531857252121,
+            0.020253615453839302,
+            0.02417202852666378,
+            0.04423905164003372,
+            -0.014910325407981873,
+            0.030634865164756775,
+            0.02975279837846756,
+            0.021084794774651527,
+            -0.006059969775378704,
+            0.02388366125524044,
+            0.0036957759875804186,
+            0.010949504561722279,
+            0.013510892167687416,
+            0.015546430833637714,
+            0.022730188444256783,
+            -0.011585610918700695,
+            -0.027547631412744522,
+            -0.06934403628110886,
+            0.037114664912223816,
+            0.01806541159749031,
+            -0.023188184946775436,
+            -0.007624790538102388,
+            -0.004177096299827099,
+            0.005703750532120466,
+            -0.023001592606306076,
+            0.04657992348074913,
+            0.020525021478533745,
+            0.0024362860713154078,
+            0.011721313931047916,
+            0.04043937847018242,
+            -0.0042152623645961285,
+            0.03552015870809555,
+            0.005071885418146849,
+            0.030007241293787956,
+            0.01556339394301176,
+            -0.0288537684828043,
+            -0.007315219379961491,
+            0.0028221902903169394,
+            0.003935375716537237,
+            0.10937631130218506,
+            -0.014002813957631588,
+            0.006509484723210335,
+            -0.0004625020665116608,
+            0.015554912388324738,
+            -0.024850541725754738,
+            -0.01725119538605213,
+            0.00812519434839487,
+            -0.037352144718170166,
+            0.04885294288396835,
+            -0.022153452038764954,
+            0.013748371042311192,
+            0.01494425069540739,
+            0.005110051482915878,
+            0.016479386016726494,
+            -0.005339049734175205,
+            0.048513684421777725,
+            0.015936575829982758,
+            -0.001556339324451983,
+            0.0006970661925151944,
+            0.005203347187489271,
+            0.01067809946835041,
+            -0.008668004535138607,
+            -0.06221964955329895,
+            -0.0252746120095253,
+            0.0008497316157445312,
+            0.007446681149303913,
+            0.018438592553138733,
+            0.0020864277612417936,
+            0.011458389461040497,
+            -0.008205767720937729,
+            0.00984692107886076,
+            -0.05506133660674095,
+            0.020474132150411606,
+            -0.02403632551431656,
+            0.01446929108351469,
+            0.04261062294244766,
+            0.0181162990629673,
+            0.06048944219946861,
+            -0.0016040473710745573,
+            0.00034826804767362773,
+            0.0008396599441766739,
+            0.00017042340186890215,
+            0.026071865111589432,
+            0.004707184620201588,
+            0.03619867190718651,
+            -0.004170734900981188,
+            -0.013841667212545872,
+            0.004982830490916967,
+            -0.0059836371801793575,
+            0.018862662836909294,
+            0.0063313753344118595,
+            0.0233578123152256,
+            0.034587204456329346,
+            0.006967481225728989,
+            -0.010627211071550846,
+            -0.0009297749493271112,
+            0.022051675245165825,
+            -0.023815808817744255,
+            -0.023289961740374565,
+            0.02822614461183548,
+            0.028718065470457077,
+            -0.03390869125723839,
+            0.01740385964512825,
+            0.01787881925702095,
+            -0.010499989613890648,
+            -0.01446929108351469,
+            -0.013595705851912498,
+            0.018252002075314522,
+            -0.012722120620310307,
+            0.0033459176775068045,
+            -0.046885255724191666,
+            -0.06876730173826218,
+            -0.02227219194173813,
+            -0.048751164227724075,
+            0.007815622724592686,
+            -0.004187698010355234,
+            0.06045551598072052,
+            0.005635899491608143,
+            -0.013010487891733646,
+            0.013476965948939323,
+            0.02454521134495735,
+            0.018557332456111908,
+            0.026258455589413643,
+            0.008621356450021267,
+            -0.022187378257513046,
+            -0.0016793198883533478,
+            0.0016931021818891168,
+            0.03494342416524887,
+            0.016029872000217438,
+            -0.0022645373828709126,
+            -0.0023875178303569555,
+            -0.020626798272132874,
+            0.01029643602669239,
+            -0.002057802863419056,
+            0.024613061919808388,
+            0.007561180274933577,
+            0.04990463703870773,
+            0.038302063941955566,
+            -0.024731801822781563,
+            -0.04715665802359581,
+            -0.014774622395634651,
+            -0.041083965450525284,
+            -0.04542645066976547,
+            0.03548623248934746,
+            -0.012009681202471256,
+            0.03867524489760399,
+            -0.006895389407873154,
+            0.03418009728193283,
+            -0.03562193736433983,
+            -0.013264930807054043,
+            0.0021479178685694933,
+            -0.01122939120978117,
+            0.024765728041529655,
+            0.012425270862877369,
+            -0.036877185106277466,
+            0.011424464173614979,
+            0.011483834125101566,
+            -0.015376802533864975,
+            0.015453135594725609,
+            -0.009134482592344284,
+            -0.00803613942116499,
+            0.036266524344682693,
+            -0.0024256843607872725,
+            -0.017174862325191498,
+            -0.05119381099939346,
+            -0.00827361922711134,
+            -0.021814195439219475,
+            0.006585817784070969,
+            -0.022577522322535515,
+            0.020100949332118034,
+            -0.01032188069075346,
+            -0.01535135880112648,
+            -0.03378995135426521,
+            0.001905137556605041,
+            0.04837798327207565,
+            -0.003979903180152178,
+            -0.010118326172232628,
+            0.06822448968887329,
+            0.02451128512620926,
+            0.00183834636118263,
+            0.01592809520661831,
+            -0.022781075909733772,
+            0.010457582771778107,
+            -0.012611862272024155,
+            0.03602904453873634,
+            0.03643615171313286,
+            0.003526147687807679,
+            -0.008646801114082336,
+            -0.025732608512043953,
+            -0.00999110471457243,
+            0.002476572757586837,
+            -0.004346724599599838,
+            -0.04895471781492233,
+            0.018268965184688568,
+            -0.005572288762778044,
+            -0.006488281302154064,
+            0.03667363151907921,
+            0.022679299116134644,
+            -0.004376409575343132,
+            0.002985457656905055,
+            0.02875199168920517,
+            0.04389979690313339,
+            -0.0567236952483654,
+            -0.055807702243328094,
+            0.016377609223127365,
+            0.011186984367668629,
+            -0.022611448541283607,
+            -0.02164456807076931,
+            -0.001569061540067196,
+            0.01508843433111906,
+            0.004217382986098528,
+            0.022085601463913918,
+            -0.011314205825328827,
+            0.003244140651077032,
+            -0.010169214569032192,
+            -0.008358432911336422,
+            0.022611448541283607,
+            0.01256097387522459,
+            -0.04423905164003372,
+            -0.0008041440160013735,
+            0.01435055211186409,
+            0.009261703118681908,
+            -0.028429698199033737,
+            0.01769222877919674,
+            0.0023853974416851997,
+            0.011585610918700695,
+            0.022340042516589165,
+            -0.011882460676133633,
+            0.025851348415017128,
+            0.010499989613890648,
+            -0.0031190398149192333,
+            0.011670424602925777,
+            0.008214249275624752,
+            -0.022899815812706947,
+            -0.02259448543190956,
+            -0.010898616164922714,
+            -0.00786651112139225,
+            0.004537556320428848,
+            1.3956232578493655e-05,
+            -0.01950725167989731,
+            0.029549244791269302,
+            -0.01735297217965126,
+            -0.02132227271795273,
+            -0.037216443568468094,
+            -0.011899422854185104,
+            -0.010262510739266872,
+            0.02904035896062851,
+            0.03738607093691826,
+            -0.013799259439110756,
+            0.004073198884725571,
+            0.009482220746576786,
+            0.011068244464695454,
+            -0.0639159306883812,
+            0.053093649446964264,
+            0.022730188444256783,
+            0.012798452749848366,
+            -0.019422436133027077,
+            0.065815769135952,
+            0.0029282080940902233,
+            0.01191638596355915,
+            -0.007900436408817768,
+            -0.02832792140543461,
+            -0.007022610399872065,
+            -0.013188597746193409,
+            0.025003207847476006,
+            -0.05492563545703888,
+            -0.027954738587141037,
+            -0.030787531286478043,
+            -0.020558945834636688,
+            0.013604187406599522,
+            -0.016055315732955933,
+            -0.002962133614346385,
+            -0.03562193736433983,
+            -0.017344491556286812,
+            0.010584804229438305,
+            -0.04563000425696373,
+            -0.017285121604800224,
+            -0.0030617902521044016,
+            0.0011131855426356196,
+            0.04135537147521973,
+            0.0010888014221563935,
+            -0.017522599548101425,
+            0.00937196146696806,
+            0.026903044432401657,
+            0.05119381099939346,
+            -0.003013022243976593,
+            -0.006912352051585913,
+            0.001623130519874394,
+            0.006526447832584381,
+            -0.005326327867805958,
+            0.005203347187489271,
+            1.655697815294843e-05,
+            -0.038302063941955566,
+            0.027852961793541908,
+            -0.025444241240620613,
+            -0.02541031502187252,
+            -0.003863283898681402,
+            0.008252415806055069,
+            0.02179723232984543,
+            -0.00820152647793293,
+            -0.03684325888752937,
+            0.019625991582870483,
+            0.03070271760225296,
+            -0.004711425397545099,
+            -0.0031169194262474775,
+            -0.00047098350478336215,
+            -0.01473221555352211,
+            -0.031025011092424393,
+            -0.03928590565919876,
+            0.013646594248712063,
+            -0.0467156246304512,
+            0.011042800731956959,
+            -0.03514697775244713,
+            0.0077901785261929035,
+            -0.0003336906083859503,
+            -0.004987071268260479,
+            0.012688194401562214,
+            0.05207587778568268,
+            0.005958192981779575,
+            0.02770029567182064,
+            -0.008154879324138165,
+            -0.03314536437392235,
+            0.006424670573323965,
+            -0.03426491096615791,
+            0.07972528785467148,
+            -0.005631658714264631,
+            -2.9949991585453972e-05,
+            0.005415382329374552,
+            0.007684160489588976,
+            -0.03314536437392235,
+            -0.029108211398124695,
+            0.032924845814704895,
+            0.0081039909273386,
+            0.0018171428237110376,
+            0.026156678795814514,
+            0.03207670524716377,
+            -0.007463643793016672,
+            -0.03565586358308792,
+            0.002756459405645728,
+            -0.0015775428619235754,
+            0.0106102479621768,
+            -0.019727768376469612,
+            0.018235038965940475,
+            -0.04077863693237305,
+            0.033772990107536316,
+            0.011186984367668629,
+            -0.021763307973742485,
+            0.0152580626308918,
+            -0.025817422196269035,
+            -0.0146643640473485,
+            -0.02870110422372818,
+            0.03088930808007717,
+            0.02946443110704422,
+            0.028480585664510727,
+            0.0007718086126260459,
+            0.04030367732048035,
+            -0.018608221784234047,
+            0.024731801822781563,
+            0.013756852596998215,
+            -0.0024362860713154078,
+            0.028158292174339294,
+            0.04033760353922844,
+            -0.031109824776649475,
+            0.0006530688260681927,
+            -0.010737469419836998,
+            0.0035855176392942667,
+            -0.03450239077210426,
+            0.0014619836583733559,
+            -0.022255228832364082,
+            -0.022085601463913918,
+            0.00032229372300207615,
+            0.013587224297225475,
+            0.005992118734866381,
+            0.0025889514945447445,
+            0.004779276438057423,
+            0.0024914152454584837,
+            -0.01139901950955391,
+            -0.031245527788996696,
+            -0.05601125583052635,
+            -0.004643573891371489,
+            -0.015766948461532593,
+            0.03714859113097191,
+            -0.040507230907678604,
+            0.014961213804781437,
+            0.018557332456111908,
+            -0.016564199700951576,
+            0.010949504561722279,
+            0.005415382329374552,
+            0.02407025173306465,
+            -0.04963323101401329,
+            -0.028599325567483902,
+            0.015173248946666718,
+            0.041457150131464005,
+            -0.030770568177103996,
+            0.009931734763085842,
+            0.00635257875546813,
+            0.0024723319802433252,
+            0.01610620506107807,
+            -0.0006970661925151944,
+            0.02817525528371334,
+            0.014214849099516869,
+            -0.008795225992798805,
+            -0.0056740655563771725,
+            0.018472518771886826,
+            0.020999979227781296,
+            0.028904657810926437,
+            -0.015105397440493107,
+            -0.05760576203465462,
+            -0.016275832429528236,
+            0.02074553817510605,
+            -0.00039306050166487694,
+            0.02222130261361599,
+            0.004588444717228413,
+            0.03231418505311012,
+            0.037589624524116516,
+            0.0037572660949081182,
+            0.020202726125717163,
+            0.0066960761323571205,
+            -0.056045182049274445,
+            -0.01574150286614895,
+            0.011008874513208866,
+            -0.023951511830091476,
+            -0.015079952776432037,
+            -0.005763120483607054,
+            0.004503630567342043,
+            -0.0410161167383194,
+            0.009736662730574608,
+            -0.00865104142576456,
+            0.026258455589413643,
+            0.02751370519399643,
+            0.00678089028224349,
+            0.011195465922355652,
+            -0.015410728752613068,
+            -0.006013322155922651,
+            -0.02016880176961422,
+            -0.026309344917535782,
+            0.026529861614108086,
+            -0.02317122183740139,
+            -0.025681721046566963,
+            -0.0036258043255656958,
+            0.01223019789904356,
+            0.0015839040279388428,
+            0.02498624473810196,
+            -0.018574295565485954,
+            -0.007200720254331827,
+            0.01993132196366787,
+            -0.0116789061576128,
+            -0.009405887685716152,
+            -0.0017874578479677439,
+            0.006874185521155596,
+            -0.0054662711918354034,
+            -0.015207174234092236,
+            0.004766554571688175,
+            0.03504519909620285,
+            -0.0038590431213378906,
+            0.009109037928283215,
+            -0.01178068295121193,
+            0.0034116485621780157,
+            0.06299994140863419,
+            -0.027089634910225868,
+            -0.04437475651502609,
+            0.04600318521261215,
+            0.051227737218141556,
+            -0.0011746757663786411,
+            -0.01542769093066454,
+            -0.019337622448801994,
+            -0.03356943652033806,
+            0.03823421150445938,
+            -0.043933723121881485,
+            0.014206367544829845,
+            -0.012891748920083046,
+            0.024002401158213615,
+            -0.001378229702822864,
+            -0.001866971142590046,
+            -0.020525021478533745,
+            0.04257669672369957,
+            0.03589334338903427,
+            -0.011331168003380299,
+            0.005279679782688618,
+            -0.0004161193501204252,
+            0.015224137343466282,
+            -0.02245878241956234,
+            0.02803955227136612,
+            0.0029388098046183586,
+            -0.010304917581379414,
+            0.005962433759123087,
+            -0.01664053276181221,
+            -0.012476159259676933,
+            -0.0033607601653784513,
+            0.013748371042311192,
+            -0.007289774715900421,
+            0.035350531339645386,
+            -0.04118574410676956,
+            -0.0035473511088639498,
+            0.0041262079030275345,
+            0.005890341475605965,
+            0.009838439524173737,
+            -0.00034800299908965826,
+            -0.036266524344682693,
+            -0.024494322016835213,
+            -0.029260877519845963,
+            0.015605800785124302,
+            -0.04708880931138992,
+            -0.01988043263554573,
+            0.03504519909620285,
+            0.008663764223456383,
+            -0.01782793179154396,
+            -0.02069464884698391,
+            -0.007603587117046118,
+            -0.004728388041257858,
+            0.013799259439110756,
+            -0.009838439524173737,
+            0.0021447374019771814,
+            -0.013672038912773132,
+            0.023103371262550354,
+            -0.015105397440493107,
+            0.02454521134495735,
+            -0.04183033108711243,
+            0.011144577525556087,
+            -0.014316625893115997,
+            0.010796839371323586,
+            0.03999834507703781,
+            0.017675265669822693,
+            -0.018557332456111908,
+            -0.008455969393253326,
+            -0.028785917907953262,
+            -0.005207587964832783,
+            -0.005614695604890585,
+            0.016097722575068474,
+            -0.02164456807076931,
+            0.010355805978178978,
+            0.01518173050135374,
+            0.011170021258294582,
+            -0.014995139092206955,
+            -0.011025837622582912,
+            0.0367075577378273,
+            -0.002381156897172332,
+            0.021865084767341614,
+            0.014503217302262783,
+            -0.01115305908024311,
+            0.015419209375977516,
+            0.009880846366286278,
+            0.03952338546514511,
+            0.00752301374450326,
+            -0.02937961556017399,
+            -0.02846362441778183,
+            -0.005334808956831694,
+            0.030770568177103996,
+            -0.02917606197297573,
+            -0.0024680914357304573,
+            0.048038724809885025,
+            -0.019761692732572556,
+            0.015673652291297913,
+            -0.02464698813855648,
+            -0.020016135647892952,
+            0.0055468445643782616,
+            -0.022764114663004875,
+            -0.05774146318435669,
+            -0.018489481881260872,
+            -0.019965248182415962,
+            -0.026088828220963478,
+            0.03952338546514511,
+            -0.027157485485076904,
+            -0.048649389296770096,
+            0.014291182160377502,
+            0.012179309502243996,
+            0.009363480843603611,
+            -0.025342464447021484,
+            -0.00943981297314167,
+            -0.02230611816048622,
+            -0.01774311624467373,
+            -0.00170476408675313,
+            0.0032823069486767054,
+            0.004656296223402023,
+            -0.021576715633273125,
+            -0.03976086527109146,
+            0.004270391538739204,
+            0.028141330927610397,
+            0.00511429226025939,
+            0.02435861900448799,
+            0.0042152623645961285,
+            0.013595705851912498,
+            0.008769781328737736,
+            0.01161105465143919,
+            0.009117519482970238,
+            -0.04325520992279053,
+            -0.004329761490225792,
+            -0.006263523828238249,
+            -0.040066197514534,
+            -0.010313399136066437,
+            -0.012484640814363956,
+            0.004085920751094818,
+            0.01854036934673786,
+            0.007993732579052448,
+            0.0026313585694879293,
+            -0.014265737496316433,
+            0.005521400365978479,
+            -0.00511429226025939,
+            0.010389731265604496,
+            0.02188204601407051,
+            -0.0005422803806141019,
+            -0.0357576385140419,
+            0.019388511776924133,
+            0.031584784388542175,
+            0.01644546166062355,
+            -0.014749177731573582,
+            -0.019778655841946602,
+            0.02798866480588913,
+            0.022526634857058525,
+            -0.020558945834636688,
+            -0.04094826430082321,
+            0.02927783876657486,
+            -0.026953931897878647,
+            0.018268965184688568,
+            0.029023397713899612,
+            0.004105004016309977,
+            -0.02354440465569496,
+            0.008163360878825188,
+            7.255584205267951e-05,
+            -0.0025211002212017775,
+            -0.011288761161267757,
+            -0.014146997593343258,
+            0.0001689656637609005,
+            -0.008379636332392693,
+            0.003568554762750864,
+            0.02880288101732731,
+            -0.016716865822672844,
+            -0.014588030986487865,
+            0.02530853822827339,
+            0.01982954517006874,
+            0.0038060343358665705,
+            0.001432298682630062,
+            -0.034298837184906006,
+            -0.005754638928920031,
+            0.010864690877497196,
+            -0.024714838713407516,
+            0.01063569262623787,
+            0.009176889434456825,
+            -0.00511429226025939,
+            -0.03302662447094917,
+            -0.0002263477217638865,
+            -0.01921888254582882,
+            -0.016954345628619194,
+            0.030193831771612167,
+            0.023001592606306076,
+            0.0009414369123987854,
+            0.012806934304535389,
+            -0.027361039072275162,
+            -0.016954345628619194,
+            -0.016606608405709267,
+            0.006607021205127239,
+            -0.0021701816003769636,
+            -0.023714032024145126,
+            -0.00695051858201623,
+            -0.014978175982832909,
+            0.015699096024036407,
+            0.031008047983050346,
+            -0.005360253155231476,
+            -0.00413680961355567,
+            0.00024556342395953834,
+            0.0027331355959177017,
+            0.00522030983120203,
+            -0.026631638407707214,
+            0.0023853974416851997,
+            0.017607415094971657,
+            0.009354999288916588,
+            0.012315012514591217,
+            0.008684967644512653,
+            -0.0008836572524160147,
+            -0.04505326971411705,
+            0.020253615453839302,
+            -0.030380424112081528,
+            -0.008795225992798805,
+            0.03972693905234337,
+            3.036412272194866e-05,
+            -0.010542397387325764,
+            0.019625991582870483,
+            -0.0036872944328933954,
+            -0.04172855615615845,
+            0.016894975677132607,
+            -0.002285741036757827,
+            -0.030787531286478043,
+            0.018676072359085083,
+            -0.020253615453839302,
+            0.008099749684333801,
+            0.014910325407981873,
+            0.026156678795814514,
+            -0.03009205497801304,
+            -0.007018369622528553,
+            0.020609835162758827,
+            -0.028548438102006912,
+            0.0010400332976132631,
+            0.0032632239162921906,
+            0.018659109249711037,
+            0.0036046006716787815,
+            0.007366107776761055,
+            -0.051566995680332184,
+            -0.028768954798579216,
+            0.014893362298607826,
+            -0.0010236005764454603,
+            0.0018341057002544403,
+            0.05794501677155495,
+            0.02412113919854164,
+            -0.07450073957443237,
+            0.01711549237370491,
+            0.026394158601760864,
+            0.025020169094204903,
+            -0.027157485485076904,
+            -0.009142964147031307,
+            0.036300450563430786,
+            0.021050868555903435,
+            0.009126001037657261,
+            -0.005398419685661793,
+            0.0022242506965994835,
+            -0.023103371262550354,
+            0.021525828167796135,
+            0.013290374539792538,
+            -0.014392958953976631,
+            -0.036775410175323486,
+            -0.024138102307915688,
+            -0.025088021531701088,
+            -0.0032971494365483522,
+            -0.015139322727918625,
+            -0.018438592553138733,
+            0.03518090397119522,
+            -0.005597732961177826,
+            0.0011502917623147368,
+            0.029142137616872787,
+            -0.002459609881043434,
+            -0.003973542246967554,
+            0.01532591413706541,
+            0.019863469526171684,
+            -0.014783103950321674,
+            -0.002510498510673642,
+            -0.015453135594725609,
+            0.0020228170324116945,
+            0.026716452091932297,
+            -0.024002401158213615,
+            0.015656689181923866,
+            0.0029282080940902233,
+            0.028904657810926437,
+            -0.00970273744314909,
+            0.011042800731956959,
+            -0.01720878854393959,
+            0.00021826387092005461,
+            -0.014986657537519932,
+            0.034451503306627274,
+            -0.006848741322755814,
+            -0.016284313052892685,
+            -0.036504004150629044,
+            0.03847169131040573,
+            0.020779462531208992,
+            -0.014384477399289608,
+            -0.046037111431360245,
+            -0.01535135880112648,
+            -0.028904657810926437,
+            0.0007373528787866235,
+            0.008328747935593128,
+            0.010720506310462952,
+            0.014867917634546757,
+            0.017030678689479828,
+            -0.007794418837875128,
+            0.0002812118618749082,
+            -0.009066631086170673,
+            -0.03367121145129204,
+            0.010211621411144733,
+            -0.038878798484802246,
+            0.01736145280301571,
+            -0.008167601190507412,
+            0.007785937748849392,
+            0.01697978936135769,
+            -0.038030657917261124,
+            -0.010576322674751282,
+            -0.03156781941652298,
+            0.011195465922355652,
+            -0.011008874513208866,
+            0.015800872817635536,
+            -0.030109018087387085,
+            0.011568647809326649,
+            0.020558945834636688,
+            -0.0038081547245383263,
+            0.033874765038490295,
+            -0.006047247909009457,
+            -0.01859125867486,
+            -0.05183839797973633,
+            -0.008027657866477966,
+            0.028531474992632866,
+            0.03986264392733574,
+            -0.0026186364702880383,
+            0.027394965291023254,
+            -0.004440019838511944,
+            0.012238679453730583,
+            0.00418133707717061,
+            -0.021474938839673996,
+            0.027666371315717697,
+            -0.02354440465569496,
+            0.005682547111064196,
+            -0.049565382301807404,
+            -0.03891272470355034,
+            -0.057809315621852875,
+            -0.03155085816979408,
+            -0.0028836806304752827,
+            -0.010330361314117908,
+            0.01270515751093626,
+            0.015758465975522995,
+            0.02103390544652939,
+            -0.0023557126987725496,
+            0.007035332731902599,
+            -0.004804720636457205,
+            -0.028768954798579216,
+            0.03060094080865383,
+            -0.004745350684970617,
+            0.003000300144776702,
+            -0.013078339397907257,
+            0.04047330468893051,
+            0.011220909655094147,
+            0.0116789061576128,
+            0.00045879147364757955,
+            -0.025003207847476006,
+            -0.05160091817378998,
+            0.007472125347703695,
+            -0.008481413125991821,
+            0.006424670573323965,
+            -0.004749591462314129,
+            -0.027479778975248337,
+            0.005249994806945324,
+            0.01587720587849617,
+            -0.0055001964792609215,
+            -0.0029790964908897877,
+            -0.024239879101514816,
+            -0.008905484341084957,
+            -0.03915020450949669,
+            -0.006318653002381325,
+            -0.005004033911973238,
+            0.008778262883424759,
+            -0.027428891509771347,
+            0.011729794554412365,
+            0.011857016012072563,
+            0.008065824396908283,
+            -0.009948697872459888,
+            -0.025206761434674263,
+            -0.007484847214072943,
+            -0.005097329616546631,
+            -0.01518173050135374,
+            0.01363811269402504,
+            -0.026343271136283875,
+            -0.027479778975248337,
+            -0.02627541869878769,
+            0.014791585505008698,
+            -0.016428498551249504,
+            -0.01921888254582882,
+            -0.005020997021347284,
+            0.024901429191231728,
+            0.020728575065732002,
+            0.02764940820634365,
+            0.0014895482454448938,
+            0.024019362404942513,
+            0.027598518878221512,
+            -0.00041585430153645575,
+            -0.0012404067674651742,
+            0.007200720254331827,
+            -0.0023281481117010117,
+            0.0031529655680060387,
+            -0.01328189391642809,
+            0.010949504561722279,
+            0.003430731827393174,
+            0.009032705798745155,
+            -0.026088828220963478,
+            -0.005691028665751219,
+            -0.00013152816973160952,
+            -0.02016880176961422,
+            0.0123998261988163,
+            -0.04257669672369957,
+            -0.0047368695959448814,
+            0.024392545223236084,
+            -0.014189404435455799,
+            0.025902237743139267,
+            0.031686559319496155,
+            -0.012442233972251415,
+            0.009583997540175915,
+            0.0020567427854985,
+            -0.02751370519399643,
+            -0.010805320926010609,
+            -0.0046817404218018055,
+            0.021288348361849785,
+            9.351089829578996e-06,
+            -0.019049255177378654,
+            -0.008684967644512653,
+            -0.029430504888296127,
+            0.01494425069540739,
+            0.02312033250927925,
+            0.014409921132028103,
+            -0.04420512542128563,
+            0.0029918185900896788,
+            0.0025529053527861834,
+            -0.008337229490280151,
+            -0.01663205213844776,
+            -0.010601767338812351,
+            -0.020813388749957085,
+            0.01731056533753872,
+            -0.016182536259293556,
+            -0.00791315920650959,
+            0.015758465975522995,
+            0.01532591413706541,
+            0.01332430075854063,
+            0.01630127616226673,
+            -0.0074254777282476425,
+            -0.013417595997452736,
+            0.0008804767276160419,
+            0.005771602038294077,
+            0.039082352072000504,
+            0.023849735036492348,
+            0.041457150131464005,
+            -0.006560373120009899,
+            0.0028285514563322067,
+            0.0011555926175788045,
+            -0.019710805267095566,
+            0.0015934455441311002,
+            -0.010525434277951717,
+            -0.0007967227720655501,
+            -0.015648208558559418,
+            -0.00786651112139225,
+            0.005979396402835846,
+            0.024765728041529655,
+            0.006475559435784817,
+            0.01877784915268421,
+            0.007985251024365425,
+            -0.0190662182867527,
+            0.002330268267542124,
+            -0.0035240272991359234,
+            -0.009897809475660324,
+            -0.0021468577906489372,
+            -0.035350531339645386,
+            0.0176413394510746,
+            -0.01753956265747547,
+            0.009176889434456825,
+            0.00035197866964153945,
+            -0.01806541159749031,
+            -0.008210008032619953,
+            0.01077987626194954,
+            -0.030295608565211296,
+            0.013587224297225475,
+            -0.01044061966240406,
+            -0.025783497840166092,
+            -0.04081256315112114,
+            0.038505617529153824,
+            0.01735297217965126,
+            -0.007756252773106098,
+            0.02988850139081478,
+            0.01611468568444252,
+            -0.013315819203853607,
+            -0.014367514289915562,
+            -0.009668811224400997,
+            -0.010508471168577671,
+            0.01032188069075346,
+            0.012586417607963085,
+            0.01854036934673786,
+            0.005491715390235186,
+            0.0019242207054048777,
+            0.003341676900163293,
+            0.003956579603254795,
+            -0.01508843433111906,
+            0.0004513702297117561,
+            -0.007811381947249174,
+            0.0052542355842888355,
+            -0.021763307973742485,
+            0.005330568179488182,
+            -0.04115181788802147,
+            -0.0017514119390398264,
+            0.01387559249997139,
+            0.01921888254582882,
+            0.0008030838216654956,
+            0.00867648608982563,
+            0.014783103950321674,
+            -0.022170415148139,
+            0.007124387193471193,
+            -0.016462422907352448,
+            0.0058097681030631065,
+            -0.015376802533864975,
+            0.012645787559449673,
+            0.01418092381209135,
+            0.017056122422218323,
+            0.013553299009799957,
+            -0.0017132455250248313,
+            0.016886495053768158,
+            0.044510457664728165,
+            -0.01325644925236702,
+            -0.0012393465731292963,
+            0.024155065417289734,
+            0.019371548667550087,
+            -0.005678306333720684,
+            -0.009804514236748219,
+            0.01887962594628334,
+            0.011288761161267757,
+            -0.02417202852666378,
+            0.017844894900918007,
+            0.014749177731573582,
+            0.022204341366887093,
+            0.052890095859766006,
+            -0.005555325653403997,
+            -0.005652862135320902,
+            0.012510084547102451,
+            0.009685774333775043,
+            -0.009634885936975479,
+            -0.012501603923738003,
+            0.025613868609070778,
+            -0.02127138525247574,
+            -0.028633251786231995,
+            -0.010576322674751282,
+            -0.008718892931938171,
+            -0.01456258725374937,
+            -0.014418402686715126,
+            0.009779069572687149,
+            0.009100556373596191,
+            -0.005529881455004215,
+            0.004864090587943792,
+            0.016420016065239906,
+            -0.0031466044019907713,
+            0.031025011092424393,
+            -0.013307337649166584,
+            -0.03265344351530075,
+            0.004554518964141607,
+            0.009736662730574608,
+            -0.00592426722869277,
+            -0.0024532489478588104,
+            -0.01882873848080635,
+            0.0024256843607872725,
+            0.050651002675294876,
+            -0.0063865045085549355,
+            0.0011004634434357285,
+            -0.02398543804883957,
+            -0.004673258867114782,
+            0.007489087991416454,
+            0.006933555472642183,
+            0.005741917062550783,
+            0.022662336006760597,
+            0.021915972232818604,
+            0.00367669272236526,
+            -0.004149531479924917,
+            -0.0003323653945699334,
+            -0.028768954798579216,
+            -0.010550878010690212,
+            0.0036809334997087717,
+            0.016411535441875458,
+            0.005245754029601812,
+            0.013383670710027218,
+            0.007378829643130302,
+            0.007785937748849392,
+            -0.01187397912144661,
+            -0.027004821226000786,
+            -0.0057758428156375885,
+            -0.005724953953176737,
+            0.012806934304535389,
+            -0.01187397912144661,
+            0.0048174429684877396,
+            -0.012162347324192524,
+            -0.021050868555903435,
+            0.03185619041323662,
+            0.009244740940630436,
+            -0.005279679782688618,
+            0.002268778160214424,
+            -0.007065017241984606,
+            0.015105397440493107,
+            -0.010067437775433064,
+            0.0019496649038046598,
+            0.0029133656062185764,
+            0.0357576385140419,
+            -0.007344903890043497,
+            0.012213235720992088,
+            0.02103390544652939,
+            0.01835377886891365,
+            -0.04261062294244766,
+            -0.015410728752613068,
+            0.02985457517206669,
+            -0.024019362404942513,
+            0.01845555566251278,
+            0.0058394530788064,
+            0.007489087991416454,
+            -0.023137295618653297,
+            -0.00018818136595655233,
+            0.00791315920650959,
+            0.0047368695959448814,
+            0.026343271136283875,
+            0.02330692484974861,
+            0.04037152975797653,
+            0.009507664479315281,
+            0.00305754947476089,
+            -0.006420430261641741,
+            0.0064373929053545,
+            -0.0031699282117187977,
+            -0.0003058609727304429,
+            -0.014409921132028103,
+            0.008468691259622574,
+            0.000756436085794121,
+            -0.013103784061968327,
+            -0.007086221128702164,
+            0.00927018467336893,
+            -0.0110852075740695,
+            0.027683334425091743,
+            0.004630851559340954,
+            0.00511429226025939,
+            -0.005860656965523958,
+            -0.007476366125047207,
+            0.020016135647892952,
+            -0.02985457517206669,
+            -0.010313399136066437,
+            -0.017191825434565544,
+            0.0015054509276524186,
+            -0.004073198884725571,
+            -0.007031091954559088,
+            0.012747564353048801,
+            -0.006284727249294519,
+            -0.006072692107409239,
+            -0.004342483822256327,
+            -0.014011295512318611,
+            -0.006276246160268784,
+            0.006725761108100414,
+            -0.02627541869878769,
+            -0.0035473511088639498,
+            -0.01032188069075346,
+            -0.027717258781194687,
+            -0.03448542580008507,
+            -0.01721726916730404,
+            0.03151693195104599,
+            -0.01184853445738554,
+            -0.005033718887716532,
+            0.006310171447694302,
+            -0.0026143956929445267,
+            0.017149418592453003,
+            0.011170021258294582,
+            -0.016224944964051247,
+            0.011738276109099388,
+            -0.0012340457178652287,
+            0.006861463654786348,
+            0.0008582130540162325,
+            -0.00943981297314167,
+            -0.011373575776815414,
+            -0.019388511776924133,
+            -0.00015664110833313316,
+            -0.004550278186798096,
+            -0.012756045907735825,
+            0.004775035660713911,
+            0.007120146416127682,
+            0.012527047656476498,
+            0.008646801114082336,
+            -0.0012404067674651742,
+            0.00803189817816019,
+            -0.012730601243674755,
+            -0.013917999342083931,
+            0.004982830490916967,
+            0.00899029802531004,
+            -0.01821807585656643,
+            -0.00784530770033598,
+            0.04142322391271591,
+            -0.022051675245165825,
+            -0.0028200699016451836,
+            -0.007192238699644804,
+            0.009405887685716152,
+            -0.009075112640857697,
+            -0.0048725721426308155,
+            0.012408307753503323,
+            -0.005644380580633879,
+            -0.009931734763085842,
+            -0.036775410175323486,
+            -0.0013983730459585786,
+            0.043221283704042435,
+            -0.019015328958630562,
+            0.01759045198559761,
+            0.0007219803519546986,
+            0.02651289850473404,
+            -0.001526654465124011,
+            0.013756852596998215,
+            -0.03555408492684364,
+            -0.009762107394635677,
+            0.003136002691462636,
+            0.015487060882151127,
+            -0.006238079629838467,
+            -0.016699902713298798,
+            0.03438365086913109,
+            -0.017794005572795868,
+            -0.022187378257513046,
+            0.00362368393689394,
+            -0.03470594435930252,
+            0.007981009781360626,
+            0.0283618476241827,
+            0.011619536206126213,
+            -0.015156285837292671,
+            -0.013578742742538452,
+            0.008222730830311775,
+            0.00781986303627491,
+            -0.002936689415946603,
+            0.0009637005859985948,
+            0.01740385964512825,
+            0.023476552218198776,
+            0.036741483956575394,
+            -0.03312839940190315,
+            -0.021441012620925903,
+            0.0008216369315050542,
+            0.015190212056040764,
+            -0.004999793134629726,
+            0.029956351965665817,
+            0.030431311577558517,
+            0.012043607421219349,
+            0.010584804229438305,
+            -0.018031485378742218,
+            0.009405887685716152,
+            0.00692507391795516,
+            0.011543204076588154,
+            0.02079642564058304,
+            -0.0011524120345711708,
+            0.024613061919808388,
+            0.01582631841301918,
+            0.005890341475605965,
+            -0.0006186130922287703,
+            -0.02040628157556057,
+            -0.007667197845876217,
+            0.016988271847367287,
+            0.004847127944231033,
+            -9.746999421622604e-05,
+            0.005936989560723305,
+            0.0035240272991359234,
+            0.019337622448801994,
+            0.006004840601235628,
+            -0.006632465403527021,
+            0.008231211453676224,
+            0.0004375879361759871,
+            -0.009329554624855518,
+            -0.0025677478406578302,
+            0.014248774386942387,
+            0.023663144558668137,
+            -0.00216382066719234,
+            0.00714135030284524,
+            -0.021491901949048042,
+            0.01639457233250141,
+            -0.0011566528119146824,
+            -0.0040647173300385475,
+            -0.0034116485621780157,
+            -0.021966861560940742,
+            0.02664860151708126,
+            0.0004627671150956303,
+            0.012348937802016735,
+            -0.0005454609054140747,
+            0.0036194431595504284,
+            0.021661529317498207,
+            -0.02359529212117195,
+            -7.586889114463702e-05,
+            -0.009982624091207981,
+            -0.010788357816636562,
+            -0.028497548773884773,
+            0.0033586397767066956,
+            0.03402743116021156,
+            0.013392152264714241,
+            -0.008735856041312218,
+            -0.0009186430834233761,
+            -0.011746757663786411,
+            0.016767755150794983,
+            0.010279472917318344,
+            -0.002824310678988695,
+            0.0009409068152308464,
+            -0.017912745475769043,
+            -0.003307751379907131,
+            0.0028073478024452925,
+            -0.005623177159577608,
+            -0.007938602939248085,
+            -0.010898616164922714,
+            -0.0037297015078365803,
+            0.018608221784234047,
+            0.006602780427783728,
+            -0.007128627970814705,
+            0.005987877957522869,
+            -0.013002007268369198,
+            0.001110004959627986,
+            -0.000263983994955197,
+            0.013332782313227654,
+            -0.016191018745303154,
+            -0.02398543804883957,
+            0.016428498551249504,
+            -0.0005292932037264109,
+            -0.009176889434456825,
+            -0.019473325461149216,
+            -0.010754432529211044,
+            -0.022390931844711304,
+            0.010041994042694569,
+            -0.006963240448385477,
+            0.006013322155922651,
+            0.001960266614332795,
+            -0.023018555715680122,
+            0.009397406131029129,
+            0.007955566048622131,
+            -0.01122939120978117,
+            0.031160714104771614,
+            -0.024918392300605774,
+            0.018268965184688568,
+            -0.0009165227529592812,
+            0.014698289334774017,
+            0.02069464884698391,
+            -0.0013474845327436924,
+            -0.03255166485905647,
+            0.006670631933957338,
+            -0.009516146034002304,
+            0.01578390970826149,
+            -0.0068402597680687904,
+            -0.002148978179320693,
+            -0.006428911350667477,
+            -0.04077863693237305,
+            -0.017912745475769043,
+            -0.0069717220030725,
+            -0.0007262210128828883,
+            0.055468443781137466,
+            -0.009456776082515717,
+            -0.001471525290980935,
+            -0.022153452038764954,
+            -0.0021839640103280544,
+            -0.033874765038490295,
+            0.0011142457369714975,
+            0.017675265669822693,
+            0.005860656965523958,
+            -0.0045630005188286304,
+            0.010881653986871243,
+            0.010101363062858582,
+            0.0056740655563771725,
+            0.013841667212545872,
+            -0.00018341056420467794,
+            -0.04213566333055496,
+            0.005975155625492334,
+            -0.012951118871569633,
+            -0.006382263731211424,
+            0.021424051374197006,
+            0.011042800731956959,
+            0.0014047340955585241,
+            0.018082372844219208,
+            0.0029833372682332993,
+            0.002404480706900358,
+            -0.010864690877497196,
+            -0.017132455483078957,
+            0.017336009070277214,
+            0.0024150824174284935,
+            0.007662957068532705,
+            -0.004830164834856987,
+            -0.004520593211054802,
+            0.013146190904080868,
+            0.0011661943281069398,
+            0.012603380717337132,
+            0.015173248946666718,
+            -0.008044620975852013,
+            -0.002330268267542124,
+            0.024630025029182434,
+            -0.005330568179488182,
+            0.0022730189375579357,
+            -0.07456858456134796,
+            0.038166359066963196,
+            0.028141330927610397,
+            -0.009329554624855518,
+            -0.00543658621609211,
+            -0.015376802533864975,
+            -0.015512505546212196,
+            -0.0046817404218018055,
+            -0.024494322016835213,
+            -0.014130034483969212,
+            0.02583438530564308,
+            -0.014791585505008698,
+            0.00025165945407934487,
+            0.004363687243312597,
+            0.0048725721426308155,
+            0.005385697353631258,
+            0.006233838852494955,
+            -0.00752301374450326,
+            0.017675265669822693,
+            -0.021542789414525032,
+            0.002406601095572114,
+            -0.010712025687098503,
+            -0.01610620506107807,
+            -0.0016803800826892257,
+            0.007573902141302824,
+            -0.008701929822564125,
+            0.0065773362293839455,
+            0.030380424112081528,
+            -0.015062990598380566,
+            0.031873151659965515,
+            0.003867524676024914,
+            -0.016911938786506653,
+            -0.04420512542128563,
+            0.009999586269259453,
+            0.021474938839673996,
+            0.015266544185578823,
+            -0.029498355463147163,
+            0.0037106184754520655,
+            -0.03165263682603836,
+            -0.013239486142992973,
+            -0.01329885609447956,
+            -0.012815415859222412,
+            -0.024256842210888863,
+            0.00754421716555953,
+            0.010025030933320522,
+            -0.017047641798853874,
+            -0.010008067823946476,
+            -0.02717444859445095,
+            -0.01025402918457985,
+            -0.00856198649853468,
+            -0.023256035521626472,
+            -0.03270433098077774,
+            -0.008239693008363247,
+            0.0036957759875804186,
+            -0.01349392905831337,
+            -0.0123998261988163,
+            -0.019659915938973427,
+            -0.003948098048567772,
+            0.002957893069833517,
+            0.007824104279279709,
+            -0.006089654751121998,
+            0.02030450478196144,
+            -0.01587720587849617,
+            -0.008358432911336422,
+            0.007226164452731609,
+            0.012781490571796894,
+            0.003311991924419999,
+            0.022492708638310432,
+            0.014138516038656235,
+            0.02965102158486843,
+            0.02427380532026291,
+            0.00798949133604765,
+            0.0037805901374667883,
+            0.024307731539011,
+            0.007768974639475346,
+            -0.026241492480039597,
+            -0.019371548667550087,
+            -0.0016570561565458775,
+            0.022390931844711304,
+            -0.005249994806945324,
+            -0.010126807726919651,
+            0.017573488876223564,
+            -0.0007357626454904675,
+            0.015071472153067589,
+            -0.030109018087387085,
+            -0.013951925560832024,
+            0.0027055710088461637,
+            -0.030770568177103996,
+            -0.008595912717282772,
+            -0.0008905483991838992,
+            0.00686570443212986,
+            -0.0012181430356577039,
+            -0.006623983848839998,
+            -0.011144577525556087,
+            0.03155085816979408,
+            0.0074424403719604015,
+            0.0021267144475132227,
+            0.010423657484352589,
+            -0.004864090587943792,
+            0.00592426722869277,
+            0.004486667923629284,
+            -0.032907884567976,
+            0.00645859632641077,
+            -0.004978589713573456,
+            0.014537142589688301,
+            -0.0169289018958807,
+            -0.005589251406490803,
+            0.0024299249053001404,
+            -0.0051015703938901424,
+            -4.8271169362124056e-05,
+            0.02250967174768448,
+            -0.01602138951420784,
+            0.0030639106407761574,
+            0.0016517553012818098,
+            -0.0036618502344936132,
+            0.006967481225728989,
+            0.03619867190718651,
+            0.0044527421705424786,
+            -0.006085413973778486,
+            -0.021254422143101692,
+            -0.005712232086807489,
+            -0.0049446639604866505,
+            -0.007645994424819946,
+            -0.042508844286203384,
+            -0.005996359512209892,
+            -0.0010326120536774397,
+            -0.02588527463376522,
+            -0.024528248235583305,
+            -0.007896196097135544,
+            0.01187397912144661,
+            -0.01253552921116352,
+            -0.008455969393253326,
+            -0.014503217302262783,
+            0.007187997922301292,
+            0.0006684414111077785,
+            0.010991912335157394,
+            -0.015156285837292671,
+            0.008197286166250706,
+            9.607851097825915e-05,
+            -0.0023832772858440876,
+            -0.010008067823946476,
+            0.0110852075740695,
+            0.010728987865149975,
+            0.008061583153903484,
+            0.008040379732847214,
+            -0.008214249275624752,
+            0.015911132097244263,
+            -0.005512918811291456,
+            0.005275439005345106,
+            0.010822284035384655,
+            -0.01223019789904356,
+            -0.013375189155340195,
+            -0.0007044874364510179,
+            -0.009499182924628258,
+            -0.020050061866641045,
+            0.010423657484352589,
+            0.009880846366286278,
+            -0.009736662730574608,
+            0.011042800731956959,
+            0.0027607001829892397,
+            0.0019083180231973529,
+            -0.007484847214072943,
+            0.012968081049621105,
+            0.08813884854316711,
+            -0.0005459910025820136,
+            0.018387705087661743,
+            0.01601290889084339,
+            -0.0056571029126644135,
+            -0.030923234298825264,
+            -0.0014535022201016545,
+            -0.03418009728193283,
+            -0.008566227741539478,
+            -0.014384477399289608,
+            -0.026190605014562607,
+            -0.01898140273988247,
+            -0.014291182160377502,
+            0.01549554243683815,
+            0.006496762856841087,
+            0.0023048240691423416,
+            0.00865528266876936,
+            0.006280486937612295,
+            0.031008047983050346,
+            -0.010839246213436127,
+            -0.01044061966240406,
+            -0.02680126577615738,
+            0.011161540634930134,
+            0.000879946630448103,
+            0.019659915938973427,
+            -0.00891396589577198,
+            0.019439399242401123,
+            0.024528248235583305,
+            0.016199499368667603,
+            0.03136426582932472,
+            0.025393351912498474,
+            0.009388924576342106,
+            0.011806127615272999,
+            -0.005644380580633879,
+            -0.0026822469662874937,
+            0.014757659286260605,
+            -0.014520179480314255,
+            0.001473645563237369,
+            -0.002315425779670477,
+            -0.01387559249997139,
+            0.0033925652969628572,
+            -0.016555719077587128,
+            0.0036682114005088806,
+            0.012315012514591217,
+            -0.003678813111037016,
+            -0.0025762293953448534,
+            0.001580723444931209,
+            -6.443886377383024e-05,
+            -0.005360253155231476,
+            -0.000742653792258352,
+            -0.008540783077478409,
+            -0.008829151280224323,
+            0.00013795548875350505,
+            0.016080759465694427,
+            0.006547651253640652,
+            0.00362368393689394,
+            0.013858629390597343,
+            -0.008451728150248528,
+            -0.013214042410254478,
+            0.0054111420176923275,
+            -0.00035091847530566156,
+            0.003598239738494158,
+            0.014588030986487865,
+            -0.004520593211054802,
+            -0.007226164452731609,
+            0.002864597365260124,
+            0.00047389898099936545,
+            0.0006806334131397307,
+            -0.007768974639475346,
+            0.022950705140829086,
+            0.01935458555817604,
+            0.0010066378163173795,
+            0.012382864020764828,
+            0.01893051527440548,
+            0.0022624169941991568,
+            -0.03173745051026344,
+            -0.003996866289526224,
+            -0.007361866999417543,
+            0.013587224297225475,
+            -0.0021171728149056435,
+            -0.01682712510228157,
+            0.00807854626327753,
+            -0.009583997540175915,
+            -0.005576529540121555,
+            -0.009346517734229565,
+            -0.002582590328529477,
+            0.017912745475769043,
+            0.004461223725229502,
+            -0.024138102307915688,
+            0.01577542908489704,
+            0.011263317428529263,
+            -0.003971422091126442,
+            0.0013326420448720455,
+            -0.015724539756774902,
+            -0.004083800595253706,
+            -0.006602780427783728,
+            0.010813802480697632,
+            -0.0031529655680060387,
+            -0.013383670710027218,
+            -0.01774311624467373,
+            0.029532281681895256,
+            -0.005869138054549694,
+            0.005402660463005304,
+            -0.004821683745831251,
+            0.0011301483027637005,
+            0.02206863835453987,
+            0.014604994095861912,
+            -0.005606214515864849,
+            0.013502410613000393,
+            -0.013443040661513805,
+            -0.0063653006218373775,
+            0.028293995186686516,
+            -0.006051488686352968,
+            -0.007645994424819946,
+            0.007692642044275999,
+            -0.007302497047930956,
+            -0.024239879101514816,
+            0.0036872944328933954,
+            -0.02412113919854164,
+            0.004707184620201588,
+            -0.015122360549867153,
+            0.022238265722990036,
+            0.015164767391979694,
+            -0.008621356450021267,
+            -0.003013022243976593,
+            0.0026546823792159557,
+            -0.002449008170515299,
+            -0.00984692107886076,
+            0.0027925053145736456,
+            0.015945058315992355,
+            -0.004804720636457205,
+            0.015071472153067589,
+            -0.010092882439494133,
+            -0.024324694648385048,
+            -0.0005115352687425911,
+            -0.011068244464695454,
+            -0.01411307230591774,
+            0.0018902950687333941,
+            -0.027581557631492615,
+            0.007081980351358652,
+            -0.019948285073041916,
+            0.000553942343685776,
+            -0.015724539756774902,
+            -0.006992925424128771,
+            -0.007171035278588533,
+            0.020525021478533745,
+            -0.003299269825220108,
+            0.0041198465041816235,
+            0.0022730189375579357,
+            -0.007811381947249174,
+            0.01184853445738554,
+            -0.027021784335374832,
+            0.003909931518137455,
+            0.01677623577415943,
+            0.0033183530904352665,
+            -0.01835377886891365,
+            -0.0013994332402944565,
+            0.023900622501969337,
+            -0.008871558122336864,
+            -0.0029833372682332993,
+            -0.007400033064186573,
+            0.008812189102172852,
+            0.007192238699644804,
+            0.003044827375560999,
+            -0.007764733862131834,
+            0.023442627862095833,
+            0.01701371558010578,
+            0.008468691259622574,
+            0.019422436133027077,
+            -0.012925674207508564,
+            -0.00535177206620574,
+            0.022204341366887093,
+            -0.001378229702822864,
+            -0.014163960702717304,
+            0.0011460509849712253,
+            -0.005271198693662882,
+            -0.009354999288916588,
+            0.006242320407181978,
+            0.01518173050135374,
+            -0.010389731265604496,
+            -0.002245454117655754,
+            -0.014477772638201714,
+            0.005771602038294077,
+            -0.011729794554412365,
+            -0.0025741090066730976,
+            -0.01132268738001585,
+            -0.007730808574706316,
+            0.007349144667387009,
+            -0.005635899491608143,
+            -0.01669142208993435,
+            -0.009100556373596191,
+            -0.006204153876751661,
+            -0.003032105276361108,
+            0.010033512488007545,
+            0.025342464447021484,
+            -0.01387559249997139,
+            0.015716059133410454,
+            -0.0048895347863435745,
+            0.008803707547485828,
+            0.016046835109591484,
+            0.013358226045966148,
+            -0.013061377219855785,
+            -0.01630127616226673,
+            0.0011131855426356196,
+            -0.005805527325719595,
+            0.010762914083898067,
+            0.0016157092759385705,
+            0.010270991362631321,
+            0.0018309251172468066,
+            0.009219296276569366,
+            -0.01625886932015419,
+            0.002569868229329586,
+            0.0031423636246472597,
+            -0.008392359130084515,
+            -0.018760886043310165,
+            -0.010627211071550846,
+            -0.009066631086170673,
+            0.021763307973742485,
+            0.018285928294062614,
+            0.007879232987761497,
+            -0.016852568835020065,
+            0.02106783166527748,
+            -0.007450921926647425,
+            0.007289774715900421,
+            -0.014443847350776196,
+            0.00716255372390151,
+            0.02580045908689499,
+            -0.004596926271915436,
+            0.0035346290096640587,
+            -0.015758465975522995,
+            -0.005805527325719595,
+            -0.0007999032968655229,
+            0.00023284131020773202,
+            0.014647400937974453,
+            0.011186984367668629,
+            0.013773815706372261,
+            0.000118408483103849,
+            0.013197079300880432,
+            -0.020677685737609863,
+            -0.019422436133027077,
+            -0.021474938839673996,
+            -0.013129227794706821,
+            -0.004965867381542921,
+            0.0026292381808161736,
+            0.0047156657092273235,
+            0.002610154915601015,
+            0.009600959718227386,
+            -0.00987236574292183,
+            -0.005843693856149912,
+            -3.207034387742169e-05,
+            0.008379636332392693,
+            -0.0006069511291570961,
+            0.008328747935593128,
+            -0.013790778815746307,
+            0.010245547629892826,
+            0.007484847214072943,
+            -0.0016284313751384616,
+            0.010186177678406239,
+            0.003610961837694049,
+            -0.006878426298499107,
+            -0.008099749684333801,
+            0.00025099681806750596,
+            -0.010423657484352589,
+            0.000959989964030683,
+            -0.009812995791435242,
+            0.006123580504208803,
+            0.009397406131029129,
+            0.01921888254582882,
+            0.02150886505842209,
+            0.014070665463805199,
+            3.816635944531299e-05,
+            0.027717258781194687,
+            0.006450114771723747,
+            0.022051675245165825,
+            -0.008061583153903484,
+            0.030244721099734306,
+            0.0008868377772159874,
+            -0.022713225334882736,
+            0.011967274360358715,
+            0.0088206697255373,
+            0.0026568027678877115,
+            -0.012959599494934082,
+            -0.016267351806163788,
+            -0.006132062058895826,
+            -0.03412920981645584,
+            -0.02098301611840725,
+            0.005177902989089489,
+            0.006657909601926804,
+            0.021339235827326775,
+            0.008964854292571545,
+            -0.00597091531381011,
+            -0.01830288954079151,
+            0.005936989560723305,
+            -0.00392265385016799,
+            -0.010720506310462952,
+            -0.013180116191506386,
+            0.012289567850530148,
+            -0.0028306718450039625,
+            0.015393765643239021,
+            0.01005895622074604,
+            -0.00754421716555953,
+            -0.0031593265011906624,
+            -0.007739289663732052,
+            -0.010313399136066437,
+            0.02135619893670082,
+            -0.024256842210888863,
+            -0.00801069475710392,
+            -0.0072516086511313915,
+            -0.008222730830311775,
+            -0.009092075750231743,
+            0.003127521136775613,
+            0.013332782313227654,
+            -0.0032653443049639463,
+            0.0030787531286478043,
+            0.004486667923629284,
+            -0.0012563093332573771,
+            0.008210008032619953,
+            -0.01225564256310463,
+            -0.037793178111314774,
+            0.024918392300605774,
+            -0.007539976388216019,
+            0.020541982725262642,
+            0.014910325407981873,
+            0.0020323586650192738,
+            -0.0012139022583141923,
+            -0.018523406237363815,
+            0.011721313931047916,
+            -0.03183922544121742,
+            0.009075112640857697,
+            -0.016470905393362045,
+            0.034875571727752686,
+            -0.013366707600653172,
+            -0.020677685737609863,
+            0.01769222877919674,
+            0.0011725553777068853,
+            -0.011687387712299824,
+            -0.027157485485076904,
+            -0.005432345438748598,
+            0.005780083127319813,
+            0.008430524729192257,
+            0.015537949278950691,
+            -0.02746281772851944,
+            -0.010957986116409302,
+            -0.025851348415017128,
+            -0.004618129692971706,
+            0.015597319230437279,
+            -0.007552698720246553,
+            -5.648753813147778e-06,
+            0.0014651642413809896,
+            0.014435365796089172,
+            0.0029515319038182497,
+            0.012035125866532326,
+            0.0008990298374556005,
+            0.0031805301550775766,
+            -0.001866971142590046,
+            0.010220102965831757,
+            0.005729194730520248,
+            0.019965248182415962,
+            -0.01208601426333189,
+            -0.00860863458365202,
+            0.011644980870187283,
+            0.008880039677023888,
+            -0.03041434846818447,
+            -0.00784530770033598,
+            0.002970615169033408,
+            -0.015275025740265846,
+            0.02888769470155239,
+            0.005644380580633879,
+            -0.010194659233093262,
+            0.01263730600476265,
+            -0.023442627862095833,
+            -0.017624378204345703,
+            -0.009473739191889763,
+            -0.0030363460537046194,
+            -0.012968081049621105,
+            -0.0031020769383758307,
+            -0.0016676579834893346,
+            0.025189798325300217,
+            0.020423242822289467,
+            -0.011687387712299824,
+            0.016581162810325623,
+            -0.010330361314117908,
+            0.004957386292517185,
+            -0.005979396402835846,
+            0.019422436133027077,
+            0.024528248235583305,
+            -0.0010442740749567747,
+            0.006908111274242401,
+            -0.0042555490508675575,
+            0.0058097681030631065,
+            -0.010092882439494133,
+            0.011687387712299824,
+            -0.012102977372705936,
+            0.002580470172688365,
+            -0.00922777783125639,
+            -0.004172855522483587,
+            -0.012204754166305065,
+            0.046274591237306595,
+            0.0030617902521044016,
+            0.011695869266986847,
+            0.026920005679130554,
+            0.0027925053145736456,
+            0.009448294527828693,
+            0.030109018087387085,
+            0.020338429138064384,
+            -0.00865528266876936,
+            -0.026869118213653564,
+            0.002639839891344309,
+            -0.014952732250094414,
+            -0.01046606432646513,
+            0.0013220403343439102,
+            -0.009626404382288456,
+            0.013366707600653172,
+            0.0030066610779613256,
+            0.047461990267038345,
+            0.003341676900163293,
+            -0.009923254139721394,
+            -0.015800872817635536,
+            0.00781986303627491,
+            0.0014651642413809896,
+            0.006513725500553846,
+            -0.004999793134629726,
+            -0.01490184385329485,
+            0.0038738856092095375,
+            -0.013332782313227654,
+            -0.017200306057929993,
+            0.01782793179154396,
+            -0.004639333114027977,
+            -0.013807740993797779,
+            -0.006869944743812084,
+            0.01725967600941658,
+            -0.009719699621200562,
+            0.03009205497801304,
+            -0.003854802343994379,
+            -0.013629631139338017,
+            0.022577522322535515,
+            -0.0021309552248567343,
+            0.015800872817635536,
+            -0.0026016735937446356,
+            -0.005122773814946413,
+            -0.017607415094971657,
+            -0.013790778815746307,
+            0.01602138951420784,
+            0.004406094551086426,
+            -0.008332989178597927,
+            -0.0009811935015022755,
+            0.01229804940521717,
+            0.01153472252190113,
+            0.0058648972772061825,
+            0.015393765643239021,
+            -0.020999979227781296,
+            0.011246354319155216,
+            0.02093212865293026,
+            0.003155085723847151,
+            -0.007421236950904131,
+            0.0005931688356213272,
+            0.0012881146976724267,
+            0.0009610501583665609,
+            -0.0022772594820708036,
+            -0.015156285837292671,
+            -0.005402660463005304,
+            0.0017885180423036218,
+            0.013307337649166584,
+            0.007552698720246553,
+            0.005033718887716532,
+            -0.02188204601407051,
+            0.0017196065746247768,
+            -0.013672038912773132,
+            0.006327134557068348,
+            -0.001542557030916214,
+            0.0017259676242247224,
+            0.017624378204345703,
+            0.005593492183834314,
+            0.00413468899205327,
+            0.005551085341721773,
+            -0.02751370519399643,
+            -0.020541982725262642,
+            -0.004864090587943792,
+            0.00905814953148365,
+            -0.006797852925956249,
+            0.011458389461040497,
+            -0.01115305908024311,
+            0.012433752417564392,
+            0.009075112640857697,
+            0.007026851177215576,
+            -0.010126807726919651,
+            -0.006255042273551226,
+            -0.011008874513208866,
+            -0.010805320926010609,
+            4.200950206723064e-05,
+            -0.010974949225783348,
+            -0.011661943979561329,
+            0.015546430833637714,
+            0.00383783970028162,
+            0.004359446465969086,
+            -0.015173248946666718,
+            -0.0025529053527861834,
+            -0.011076726019382477,
+            -0.007018369622528553,
+            0.0062295980751514435,
+            0.018947478383779526,
+            0.0024574894923716784,
+            -0.0027734222821891308,
+            -0.008256656117737293,
+            -0.006445874460041523,
+            -0.0036300448700785637,
+            -0.022424858063459396,
+            -0.03643615171313286,
+            0.01325644925236702,
+            0.007641753647476435,
+            0.008973335847258568,
+            0.001959206536412239,
+            -0.012577936053276062,
+            -0.028735028579831123,
+            -0.005127014592289925,
+            0.0134684843942523,
+            0.02369706891477108,
+            -0.022475745528936386,
+            0.011907904408872128,
+            0.023849735036492348,
+            0.012951118871569633,
+            -0.0031444840133190155,
+            0.022662336006760597,
+            0.01256097387522459,
+            0.01911710575222969,
+            -0.0024532489478588104,
+            -0.006314412225037813,
+            0.0070607769303023815,
+            -0.012391345575451851,
+            -0.0013050774578005075,
+            -0.009354999288916588,
+            0.008829151280224323,
+            0.0027607001829892397,
+            -0.0023832772858440876,
+            0.007917399518191814,
+            -0.030482200905680656,
+            0.010661136358976364,
+            -0.022153452038764954,
+            0.013366707600653172,
+            0.014359032735228539,
+            -0.008557746186852455,
+            0.00946525763720274,
+            0.0007617369410581887,
+            0.0053093647584319115,
+            0.02317122183740139,
+            -0.002353592310100794,
+            0.009185370989143848,
+            -0.010576322674751282,
+            -0.0018553092377260327,
+            -0.018252002075314522,
+            0.003301390213891864,
+            0.003998986445367336,
+            -0.004596926271915436,
+            -0.012425270862877369,
+            0.02059287205338478,
+            -0.00473262881860137,
+            -0.0001673754013609141,
+            -0.024239879101514816,
+            -0.00029923487454652786,
+            -0.005843693856149912,
+            0.021525828167796135,
+            -0.017726154997944832,
+            -0.0005565927713178098,
+            0.009388924576342106,
+            0.010211621411144733,
+            0.0017916986253112555,
+            -0.0319579653441906,
+            0.023001592606306076,
+            -0.02993938885629177,
+            -0.025427278131246567,
+            -0.005025237333029509,
+            -0.004864090587943792,
+            -0.006564613897353411,
+            -0.022323081269860268,
+            0.012459196150302887,
+            0.012331975623965263,
+            -0.004196179565042257,
+            -0.014723733998835087,
+            -0.0018319853115826845,
+            -0.003093595616519451,
+            -0.00624656118452549,
+            -0.005093088839203119,
+            -0.0032377797178924084,
+            -0.005941230338066816,
+            0.0021161127369850874,
+            -0.010432139039039612,
+            -0.007739289663732052,
+            -0.00784106645733118,
+            0.007213442120701075,
+            0.0018574295099824667,
+            0.01206056959927082,
+            -0.002894282341003418,
+            -0.020677685737609863,
+            0.011950311250984669,
+            0.001987831201404333,
+            0.015724539756774902,
+            0.004719906486570835,
+            0.006738482974469662,
+            0.010941023007035255,
+            0.013790778815746307,
+            -0.011526240967214108,
+            -0.00982147641479969,
+            -0.01648786850273609,
+            -0.005907304584980011,
+            0.013739889487624168,
+            -0.012026644311845303,
+            -0.0040074680000543594,
+            -0.00012821512063965201,
+            -0.020083988085389137,
+            0.005440826993435621,
+            0.011102170683443546,
+            0.005152458790689707,
+            -0.011144577525556087,
+            -0.026292381808161736,
+            0.013790778815746307,
+            0.0058097681030631065,
+            -0.013392152264714241,
+            -0.003216576064005494,
+            0.01682712510228157,
+            0.026190605014562607,
+            0.02641112171113491,
+            0.011195465922355652,
+            0.015291988849639893,
+            -0.009007261134684086,
+            0.031143750995397568,
+            0.0076163094490766525,
+            0.007221923675388098,
+            -0.024765728041529655,
+            0.01574150286614895,
+            -0.0015467978082597256,
+            0.0022221303079277277,
+            -0.00020103600400034338,
+            -0.028582364320755005,
+            -0.009728181175887585,
+            -0.036741483956575394,
+            -0.015953538939356804,
+            -0.013205560855567455,
+            -0.004758073017001152,
+            0.0037212201859802008,
+            0.006416189484298229,
+            0.00021481829753611237,
+            0.011840052902698517,
+            -0.0017810967983677983,
+            0.009142964147031307,
+            0.006869944743812084,
+            -0.015444654040038586,
+            0.0025062577333301306,
+            -0.009354999288916588,
+            -0.0076884012669324875,
+            -0.0030087814666330814,
+            -0.03156781941652298,
+            0.0018690915312618017,
+            -0.02483357861638069,
+            -0.010627211071550846,
+            -0.008973335847258568,
+            -0.004953145515173674,
+            -0.0028985231183469296,
+            0.0023896382190287113,
+            -0.017794005572795868,
+            -0.018506444990634918,
+            -0.0110852075740695,
+            -0.020491095259785652,
+            -0.013460002839565277,
+            -0.0070607769303023815,
+            -0.003810275113210082,
+            -0.002542303642258048,
+            0.03251773864030838,
+            0.006717279553413391,
+            -0.003307751379907131,
+            -0.032348111271858215,
+            0.014223330654203892,
+            -0.005381457041949034,
+            -0.014842473901808262,
+            -0.00867648608982563,
+            0.014995139092206955,
+            -0.022662336006760597,
+            0.018319852650165558,
+            0.00943981297314167,
+            -0.021254422143101692,
+            0.013621150515973568,
+            0.027903851121664047,
+            0.025732608512043953,
+            0.010220102965831757,
+            -0.00850685779005289,
+            -6.033896170265507e-06,
+            0.002220009919255972,
+            0.001824564067646861,
+            0.015809355303645134,
+            0.011509277857840061,
+            0.015308951027691364,
+            -0.002446887781843543,
+            0.015436172485351562,
+            -0.02780207432806492,
+            -0.0015902650775387883,
+            -0.0076163094490766525,
+            -0.005894582252949476,
+            0.008405080996453762,
+            -0.008108231239020824,
+            -0.018913552165031433,
+            -0.025953125208616257,
+            -0.00010051800200017169,
+            0.024002401158213615,
+            -0.016886495053768158,
+            -0.014359032735228539,
+            -0.010881653986871243,
+            -0.0004110835143364966,
+            -0.007366107776761055,
+            0.010576322674751282,
+            0.022916778922080994,
+            -0.0005102100549265742,
+            0.0060218037106096745,
+            0.012289567850530148,
+            -0.02717444859445095,
+            0.01340063288807869,
+            -0.017895782366394997,
+            -0.0074254777282476425,
+            -0.00798949133604765,
+            -0.010881653986871243,
+            0.0033374361228197813,
+            -0.011526240967214108,
+            0.014800066128373146,
+            0.01606379635632038,
+            0.0034158893395215273,
+            -0.013137709349393845,
+            -0.0027522186283022165,
+            -0.020423242822289467,
+            0.00011118603288196027,
+            -0.007752011995762587,
+            0.003678813111037016,
+            0.009957179427146912,
+            0.004340363200753927,
+            0.021525828167796135,
+            0.008786744438111782,
+            -0.03680933639407158,
+            -0.0021691215224564075,
+            0.012925674207508564,
+            -0.00213307561352849,
+            -0.028378808870911598,
+            -0.012713639065623283,
+            0.009134482592344284,
+            0.00038484414108097553,
+            0.003937496338039637,
+            0.003950218204408884,
+            -0.004558759741485119,
+            0.017471712082624435,
+            -0.04043937847018242,
+            -0.01840466819703579,
+            -0.00040207200800068676,
+            -0.016470905393362045,
+            -0.007103183772414923,
+            0.008481413125991821,
+            0.0026037939824163914,
+            0.010499989613890648,
+            0.009668811224400997,
+            -0.018438592553138733,
+            0.01178068295121193,
+            -0.0009260643273591995,
+            -0.01373140886425972,
+            0.0013167394790798426,
+            0.0007988431607373059,
+            0.009448294527828693,
+            0.0021055107936263084,
+            -0.00862983800470829,
+            -0.00786651112139225,
+            -0.0100165493786335,
+            0.004842887166887522,
+            0.029684947803616524,
+            -0.00023959993268363178,
+            -0.005856416188180447,
+            0.005016756244003773,
+            -0.010381249710917473,
+            -0.013273412361741066,
+            -0.01432510744780302,
+            0.0006949458038434386,
+            0.013460002839565277,
+            0.0073067378252744675,
+            0.014749177731573582,
+            -0.011170021258294582,
+            -0.005071885418146849,
+            0.011390537954866886,
+            -0.01170435082167387,
+            0.013409114442765713,
+            -0.01625886932015419,
+            -0.015995945781469345,
+            -0.007993732579052448,
+            0.0025401832535862923,
+            0.015139322727918625,
+            0.00922777783125639,
+            -0.000635575910564512,
+            0.0333828441798687,
+            -0.008803707547485828,
+            0.020474132150411606,
+            -0.01425725594162941,
+            -0.01697130873799324,
+            0.005283920560032129,
+            -0.016564199700951576,
+            -0.007531495299190283,
+            -0.004859849810600281,
+            -0.020558945834636688,
+            0.007166794501245022,
+            0.018726961687207222,
+            -0.004189818166196346,
+            0.016148611903190613,
+            0.008167601190507412,
+            -0.0039523388259112835,
+            0.0030893548391759396,
+            -0.00927018467336893,
+            -0.012069051153957844,
+            0.004098643083125353,
+            0.0146643640473485,
+            -0.01759045198559761,
+            0.017522599548101425,
+            0.004516352899372578,
+            0.007285534404218197,
+            0.015512505546212196,
+            0.012985044158995152,
+            -0.02680126577615738,
+            -0.011865497566759586,
+            0.022441819310188293,
+            -0.01206056959927082,
+            0.005597732961177826,
+            0.011840052902698517,
+            0.0038993298076093197,
+            -0.009397406131029129,
+            -0.013510892167687416,
+            0.0011259076418355107,
+            -0.00991477258503437,
+            0.013544817455112934,
+            -0.00015120769967325032,
+            -0.041083965450525284,
+            0.007997972890734673,
+            -0.006441633682698011,
+            0.003487981390208006,
+            0.0027331355959177017,
+            -0.0058394530788064,
+            0.023256035521626472,
+            0.02259448543190956,
+            -0.015504023991525173,
+            -0.00588610116392374,
+            -0.013960406184196472,
+            0.012798452749848366,
+            0.008693449199199677,
+            0.007976769469678402,
+            -0.02493535540997982,
+            -0.003610961837694049,
+            0.007213442120701075,
+            -0.0064373929053545,
+            -0.00473262881860137,
+            0.014545624144375324,
+            -0.009346517734229565,
+            -0.0013951925793662667,
+            -0.003825117601081729,
+            0.0015043907333165407,
+            0.013044414110481739,
+            0.0036703317891806364,
+            0.015529467724263668,
+            -0.01950725167989731,
+            0.013833185657858849,
+            -0.0058648972772061825,
+            0.0010368528310209513,
+            0.009897809475660324,
+            -0.014197885990142822,
+            -0.026139715686440468,
+            0.01630127616226673,
+            0.00692507391795516,
+            0.00037477246951311827,
+            -0.023086408153176308,
+            -0.00645859632641077,
+            -0.01993132196366787,
+            0.0029409301932901144,
+            -0.00028465743525885046,
+            0.0038738856092095375,
+            0.025088021531701088,
+            -0.000675862655043602,
+            -0.015860242769122124,
+            -0.005797046236693859,
+            0.001527714659459889,
+            0.0015923853497952223,
+            0.007145591080188751,
+            -0.0193036962300539,
+            -0.025088021531701088,
+            -0.0050464412197470665,
+            -0.013858629390597343,
+            0.018319852650165558,
+            -0.017760079354047775,
+            0.015716059133410454,
+            -0.003555832663550973,
+            0.008922447450459003,
+            0.00562741793692112,
+            -0.0029812168795615435,
+            0.03294181078672409,
+            -0.015597319230437279,
+            0.010304917581379414,
+            0.007548457942903042,
+            -0.011382057331502438,
+            -0.011449907906353474,
+            0.01774311624467373,
+            -0.010508471168577671,
+            0.0032505018170922995,
+            0.0038463210221379995,
+            -0.00929562933743,
+            0.023476552218198776,
+            0.0027500982396304607,
+            0.014757659286260605,
+            -0.01070354413241148,
+            -0.019914358854293823,
+            0.0032907885033637285,
+            0.010686581023037434,
+            0.0013496049214154482,
+            0.00812519434839487,
+            -0.02888769470155239,
+            0.00483864638954401,
+            -0.009643367491662502,
+            0.02106783166527748,
+            0.009728181175887585,
+            0.0018097215797752142,
+            -0.02999027818441391,
+            0.0008857776410877705,
+            -0.014070665463805199,
+            0.02651289850473404,
+            0.005165180657058954,
+            0.003301390213891864,
+            -0.01139901950955391,
+            -0.003975662402808666,
+            0.012823897413909435,
+            -0.005754638928920031,
+            -0.009643367491662502,
+            -0.0038039139471948147,
+            0.011178502812981606,
+            0.004821683745831251,
+            -0.013426077552139759,
+            -0.013934962451457977,
+            -0.011882460676133633,
+            0.010160733014345169,
+            0.0012520686723291874,
+            0.0037424236070364714,
+            -0.004323400557041168,
+            0.007484847214072943,
+            -0.011763720773160458,
+            0.022950705140829086,
+            0.009668811224400997,
+            0.01592809520661831,
+            0.01893051527440548,
+            -0.008176082745194435,
+            0.009643367491662502,
+            0.008837632834911346,
+            0.01046606432646513,
+            0.016174055635929108,
+            0.022373968735337257,
+            0.021899009123444557,
+            0.019710805267095566,
+            -0.021949898451566696,
+            -0.0010585864074528217,
+            -0.006602780427783728,
+            -0.0073237004689872265,
+            -0.011958792805671692,
+            -0.018998365849256516,
+            0.001702643814496696,
+            -0.03372209891676903,
+            -0.019015328958630562,
+            -0.0017206667689606547,
+            0.007090461906045675,
+            0.013027451001107693,
+            -0.014333589002490044,
+            0.012985044158995152,
+            0.008583190850913525,
+            0.00036417070077732205,
+            -0.014486254192888737,
+            -0.010949504561722279,
+            -0.012544010765850544,
+            0.004367928020656109,
+            0.021610641852021217,
+            0.004153772257268429,
+            0.0049446639604866505,
+            -0.01301896944642067,
+            0.00937196146696806,
+            0.02822614461183548,
+            0.0015351359033957124,
+            -0.00953310914337635,
+            0.0058479346334934235,
+            -0.004073198884725571,
+            0.008752819150686264,
+            -0.012162347324192524,
+            -0.010669617913663387,
+            0.000553412246517837,
+            0.005754638928920031,
+            -0.013748371042311192,
+            0.02140708826482296,
+            -0.004196179565042257,
+            0.018285928294062614,
+            -2.4102455427055247e-06,
+            -0.0007198599632829428,
+            0.0024511285591870546,
+            0.003971422091126442,
+            0.02198382467031479,
+            0.005275439005345106,
+            -0.0008550324710085988,
+            -0.023730995133519173,
+            -0.0005356542533263564,
+            -0.0029939389787614346,
+            -0.020134875550866127,
+            0.004906497895717621,
+            0.0069589996710419655,
+            -0.004359446465969086,
+            0.02717444859445095,
+            0.013960406184196472,
+            0.0013156792847439647,
+            0.005669824779033661,
+            0.0014089748729020357,
+            0.007366107776761055,
+            -0.008498376235365868,
+            -0.00946525763720274,
+            0.0041262079030275345,
+            -0.0027755426708608866,
+            0.0051185330376029015,
+            -0.027089634910225868,
+            0.013570261187851429,
+            0.0050634038634598255,
+            0.0038166360463947058,
+            0.006208394654095173,
+            0.008693449199199677,
+            -0.005996359512209892,
+            0.0012128421803936362,
+            0.006975962780416012,
+            0.008260896429419518,
+            0.001960266614332795,
+            0.031075898557901382,
+            -0.027428891509771347,
+            -0.00036019503022544086,
+            -0.005945470649749041,
+            -0.005233032163232565,
+            -0.0032547423616051674,
+            -0.01816718839108944,
+            -0.013782297261059284,
+            -0.00392053322866559,
+            -0.001568001345731318,
+            0.023900622501969337,
+            0.013578742742538452,
+            0.01611468568444252,
+            0.01149231567978859,
+            -0.006344097200781107,
+            -0.018862662836909294,
+            0.009176889434456825,
+            -0.0035367493983358145,
+            -0.00619991309940815,
+            0.031686559319496155,
+            0.016199499368667603,
+            -0.0036830538883805275,
+            0.01887962594628334,
+            -0.0051354956813156605,
+            0.011178502812981606,
+            -0.016174055635929108,
+            0.001324160723015666,
+            -0.006869944743812084,
+            -0.0031741689890623093,
+            0.013426077552139759,
+            -0.002298463135957718,
+            0.007930121384561062,
+            -0.0024574894923716784,
+            -0.00526695791631937,
+            0.0076714386232197285,
+            -0.0031317619141191244,
+            -0.004162253811955452,
+            0.013977369293570518,
+            -0.003746664384379983,
+            -0.011161540634930134,
+            -0.015368320979177952,
+            0.011127614416182041,
+            -0.01328189391642809,
+            0.006216876208782196,
+            -0.0036745723336935043,
+            -0.007925881072878838,
+            -0.002419323194772005,
+            0.03060094080865383,
+            -0.0020779462065547705,
+            0.017794005572795868,
+            0.004851368721574545,
+            -0.02203471213579178,
+            -0.02083035185933113,
+            -0.008167601190507412,
+            0.03151693195104599,
+            -0.00507612619549036,
+            -0.009159926325082779,
+            -0.019795618951320648,
+            -0.01091557927429676,
+            0.004268271382898092,
+            -0.012823897413909435,
+            -0.0003002950397785753,
+            -5.681884431396611e-05,
+            0.003638526424765587,
+            -0.0007946024416014552,
+            -0.01408762764185667,
+            -0.004656296223402023,
+            0.00031142690568231046,
+            0.01253552921116352,
+            0.005059163086116314,
+            0.002565627684816718,
+            0.014308144338428974,
+            0.005470511503517628,
+            -0.02751370519399643,
+            -0.006551892030984163,
+            0.013315819203853607,
+            0.012527047656476498,
+            0.012348937802016735,
+            -0.002095969393849373,
+            -0.007900436408817768,
+            -0.02659771218895912,
+            -0.010381249710917473,
+            -0.0362325981259346,
+            -0.0013803500914946198,
+            -0.023052481934428215,
+            -0.018896589055657387,
+            -0.02103390544652939,
+            -0.021254422143101692,
+            -0.010330361314117908,
+            -0.01067809946835041,
+            0.025681721046566963,
+            0.010983430780470371,
+            0.0026716452557593584,
+            0.00791315920650959,
+            -0.023374775424599648,
+            0.0027119319420307875,
+            -0.009541589766740799,
+            -0.006963240448385477,
+            0.02359529212117195,
+            0.00011436655768193305,
+            0.016793198883533478,
+            -0.0014216969721019268,
+            -0.011907904408872128,
+            0.007395792752504349,
+            -0.007701123598963022,
+            -0.007662957068532705,
+            0.010932542383670807,
+            -0.012628824450075626,
+            -0.013485447503626347,
+            0.016428498551249504,
+            0.026156678795814514,
+            -0.002173362299799919,
+            0.0005157759878784418,
+            -0.024155065417289734,
+            -0.0011969394981861115,
+            -0.0020355391316115856,
+            0.0009202333749271929,
+            -0.0015828438336029649,
+            0.007421236950904131,
+            0.002241213573142886,
+            0.0034116485621780157,
+            0.02006702497601509,
+            -0.012603380717337132,
+            0.01816718839108944,
+            0.007506051100790501,
+            -0.001392011996358633,
+            0.021491901949048042,
+            0.010949504561722279,
+            0.0016284313751384616,
+            -0.023052481934428215,
+            -0.004363687243312597,
+            -0.010033512488007545,
+            -0.001446080976165831,
+            0.014435365796089172,
+            0.013154672458767891,
+            -0.020016135647892952,
+            -0.025257648900151253,
+            4.452741995919496e-05,
+            -0.001272212015464902,
+            0.009829957969486713,
+            -0.015444654040038586,
+            -0.0020620436407625675,
+            0.0006551892147399485,
+            -0.018557332456111908,
+            0.015571875497698784,
+            0.013417595997452736,
+            0.0033925652969628572,
+            0.01332430075854063,
+            -0.005283920560032129,
+            0.015275025740265846,
+            -0.007743530441075563,
+            0.010203140787780285,
+            -0.003099956549704075,
+            -0.013146190904080868,
+            0.006288968026638031,
+            -0.00784106645733118,
+            -0.01163649931550026,
+            -0.010118326172232628,
+            -0.017369935289025307,
+            -0.021542789414525032,
+            -0.020915165543556213,
+            -0.019235845655202866,
+            0.015291988849639893,
+            -0.011365094222128391,
+            0.009261703118681908,
+            0.004961627069860697,
+            -0.006081173662096262,
+            0.021101756021380424,
+            0.005890341475605965,
+            -0.0017641340382397175,
+            0.007209201343357563,
+            -0.003203853964805603,
+            -0.0008216369315050542,
+            -0.004393372219055891,
+            0.032348111271858215,
+            -0.0013315819669514894,
+            0.024494322016835213,
+            -0.0006445874460041523,
+            0.008205767720937729,
+            0.0016125286929309368,
+            0.0015892048832029104,
+            -0.023137295618653297,
+            0.012908711098134518,
+            -0.011314205825328827,
+            -0.014571068808436394,
+            -0.02412113919854164,
+            -0.00022078178881201893,
+            0.006178709678351879,
+            0.017047641798853874,
+            0.0041431705467402935,
+            -0.0009424970485270023,
+            0.008646801114082336,
+            -0.01950725167989731,
+            0.002160640200600028,
+            0.009007261134684086,
+            -0.00156694115139544,
+            -0.008290581405162811,
+            -0.02030450478196144,
+            -0.0019570861477404833,
+            -0.027394965291023254,
+            0.005568047985434532,
+            0.0007946024416014552,
+            -0.01504602748900652,
+            0.010949504561722279,
+            -0.007904677651822567,
+            0.021339235827326775,
+            0.01830288954079151,
+            0.0011916386429220438,
+            -0.006581577006727457,
+            -0.015300470404326916,
+            -0.015962019562721252,
+            0.013527854345738888,
+            0.015308951027691364,
+            0.011110651306807995,
+            0.006178709678351879,
+            0.005398419685661793,
+            -0.004329761490225792,
+            0.01446929108351469,
+            0.00937196146696806,
+            -0.028599325567483902,
+            1.502469149272656e-05,
+            -0.010983430780470371,
+            0.023680105805397034,
+            -0.02330692484974861,
+            0.014002813957631588,
+            -0.020999979227781296,
+            0.01935458555817604,
+            -0.0003612551954574883,
+            0.012722120620310307,
+            0.009278666228055954,
+            8.89223229023628e-05,
+            -0.013375189155340195,
+            0.005852175410836935,
+            0.001298716408200562,
+            -0.0011555926175788045,
+            0.0048174429684877396,
+            0.022492708638310432,
+            -0.02951531857252121,
+            -0.004893775563687086,
+            -0.005190624855458736,
+            -2.678602686501108e-05,
+            -0.010313399136066437,
+            -0.024188991636037827,
+            0.0033586397767066956,
+            -0.015012102201581001,
+            0.002998179756104946,
+            0.018862662836909294,
+            0.00516094034537673,
+            0.02111871913075447,
+            -0.02330692484974861,
+            -0.009829957969486713,
+            -0.013553299009799957,
+            0.017437785863876343,
+            -0.0029536522924900055,
+            0.0016825003549456596,
+            0.006517966277897358,
+            -0.013909517787396908,
+            0.0058097681030631065,
+            -0.0073237004689872265,
+            -0.00441033486276865,
+            0.06323742121458054,
+            -0.0005393648752942681,
+            -0.007014128845185041,
+            -0.01022858452051878,
+            -0.010092882439494133,
+            0.015317432582378387,
+            0.006929314695298672,
+            -0.003528268076479435,
+            -0.010695062577724457,
+            -0.004397612996399403,
+            -0.023256035521626472,
+            -0.01648786850273609,
+            -0.004217382986098528,
+            -0.0074254777282476425,
+            0.01178068295121193,
+            -0.000947267864830792,
+            -0.009999586269259453,
+            0.003445574315264821,
+            -0.0054111420176923275,
+            -0.00846020970493555,
+            0.006378022953867912,
+            -0.01615709252655506,
+            0.001985711045563221,
+            0.008939409628510475,
+            -0.015554912388324738,
+            0.018252002075314522,
+            -0.014528661035001278,
+            -0.0048174429684877396,
+            0.022170415148139,
+            -0.021288348361849785,
+            0.005394178908318281,
+            0.0011725553777068853,
+            0.01935458555817604,
+            -0.019625991582870483,
+            -0.009456776082515717,
+            0.03173745051026344,
+            -0.012645787559449673,
+            0.008400839753448963,
+            0.00853230245411396,
+            0.006759686395525932,
+            0.0018902950687333941,
+            0.0011736155720427632,
+            0.007412755396217108,
+            0.014859436079859734,
+            -0.01636064611375332,
+            -0.01950725167989731,
+            0.007832585833966732,
+            -0.004448501393198967,
+            -0.0036003601271659136,
+            0.010372769087553024,
+            -0.009278666228055954,
+            0.0025529053527861834,
+            -0.0053560128435492516,
+            0.01470677088946104,
+            -0.005682547111064196,
+            -0.006187191233038902,
+            0.018099335953593254,
+            -0.01039821282029152,
+            0.0053093647584319115,
+            0.02927783876657486,
+            -0.012544010765850544,
+            -0.016911938786506653,
+            -0.0009944457560777664,
+            -0.00522030983120203,
+            -0.013332782313227654,
+            0.01740385964512825,
+            -0.01702219620347023,
+            -0.005525640677660704,
+            -0.02393454872071743,
+            -0.004520593211054802,
+            0.005826731212437153,
+            0.0190662182867527,
+            0.013273412361741066,
+            -0.019337622448801994,
+            0.003812395269051194,
+            -0.011271798983216286,
+            0.023493515327572823,
+            0.0031953726429492235,
+            0.012196272611618042,
+            -0.0010442740749567747,
+            -0.03894665092229843,
+            -0.012764527462422848,
+            -0.03562193736433983,
+            0.014825510792434216,
+            -0.004813202191144228,
+            -0.024952318519353867,
+            -0.02627541869878769,
+            -0.0021161127369850874,
+            -0.014723733998835087,
+            0.007187997922301292,
+            0.012611862272024155,
+            0.01425725594162941,
+            0.012645787559449673,
+            0.03775925189256668,
+            0.0021277745254337788,
+            -0.0004309618379920721,
+            -0.0008476112852804363,
+            0.022289155051112175,
+            0.01149231567978859,
+            -0.01720878854393959,
+            0.009142964147031307,
+            0.0043721687979996204,
+            -0.012187791056931019,
+            -0.004596926271915436,
+            -0.017709191888570786,
+            -0.016479386016726494,
+            0.014842473901808262,
+            -0.01877784915268421,
+            0.007561180274933577,
+            0.01578390970826149,
+            -0.010169214569032192,
+            0.009185370989143848,
+            -0.003369241487234831,
+            -0.005173662211745977,
+            -0.01490184385329485,
+            0.0010071678552776575,
+            -0.008693449199199677,
+            0.01582631841301918,
+            -0.012739082798361778,
+            0.01864214614033699,
+            -0.014172442257404327,
+            -0.00826937798410654,
+            0.021339235827326775,
+            0.022475745528936386,
+            -0.009567034430801868,
+            0.008710411377251148,
+            -0.021525828167796135,
+            0.0008878979715518653,
+            0.014630438759922981,
+            -0.019490288570523262,
+            0.02856540121138096,
+            -0.012925674207508564,
+            0.014613475650548935,
+            0.0011322686914354563,
+            -0.009287147782742977,
+            0.002555025741457939,
+            -0.010550878010690212,
+            0.013672038912773132,
+            -0.0019411834655329585,
+            0.012094495818018913,
+            -0.01615709252655506,
+            0.015809355303645134,
+            -0.011653462424874306,
+            0.012510084547102451,
+            0.009244740940630436,
+            -0.005860656965523958,
+            -0.0005868077860213816,
+            -0.013069857843220234,
+            0.0052542355842888355,
+            -0.003812395269051194,
+            0.023849735036492348,
+            0.006399226374924183,
+            -0.015809355303645134,
+            0.0014821270015090704,
+            0.0028094681911170483,
+            0.015537949278950691,
+            0.011314205825328827,
+            0.007086221128702164,
+            -0.013095302507281303,
+            0.01935458555817604,
+            -0.00987236574292183,
+            0.008998779579997063,
+            -0.0011036439100280404,
+            0.011407501064240932,
+            -0.022577522322535515,
+            -0.00588610116392374,
+            0.007048054598271847,
+            -0.015291988849639893,
+            0.012857822701334953,
+            -0.0076163094490766525,
+            0.0006000599823892117,
+            0.007137109525501728,
+            0.006590058095753193,
+            0.019439399242401123,
+            0.004276752937585115,
+            0.002393878996372223,
+            0.003301390213891864,
+            -0.011619536206126213,
+            -0.0159789826720953,
+            -0.008668004535138607,
+            -0.020185764878988266,
+            0.005088848061859608,
+            0.010237066075205803,
+            0.02064376138150692,
+            0.0008592731901444495,
+            -0.008718892931938171,
+            -0.00801069475710392,
+            -0.02011791244149208,
+            0.00045190032687969506,
+            0.021135682240128517,
+            0.012416789308190346,
+            -0.011543204076588154,
+            -0.011415982618927956,
+            0.020491095259785652,
+            -0.00654341047629714,
+            -0.02103390544652939,
+            -0.012934155762195587,
+            -0.006836019456386566,
+            -0.0027267744299024343,
+            -0.007981009781360626,
+            0.016581162810325623,
+            0.004775035660713911,
+            0.0076332720927894115,
+            0.03613082319498062
+          ],
+          "title": "E Relation of Upstream to Downstream Model Performance",
+          "keyphrases": [
+            "Upstream to Downstream Model Performance"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "addd7c61-bdd3-470e-8c4f-439af824f89e",
+        "properties": {
+          "page_content": "F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+          "embedding": [
+            0.0002445570513886185,
+            -0.025035678422212698,
+            -0.026056660824720954,
+            0.0021234386221176942,
+            -0.007526403638487191,
+            0.018963965001103752,
+            -0.007728070932730228,
+            0.062213823247851435,
+            -0.00501600735659763,
+            0.031461940272958254,
+            0.014585783857839997,
+            0.010118033107401152,
+            0.009983318452435566,
+            -0.03282616973330728,
+            0.015192449242879527,
+            0.01480596701302375,
+            0.019405922018728892,
+            0.022192910412596127,
+            0.008048967251594976,
+            -0.0367043822293015,
+            0.017737679976556845,
+            -0.022757028540067756,
+            -0.06290354368907101,
+            0.016842087557557506,
+            -0.0050319451760917995,
+            0.019311667659179976,
+            0.008548884527537198,
+            0.02344214718766695,
+            -0.042417710048062586,
+            0.01950972309054391,
+            0.021637767293446648,
+            0.010581067128107188,
+            -0.014214247204390872,
+            0.01016365063627519,
+            -0.025782209417014867,
+            -0.02611397356963557,
+            0.034251067799009986,
+            0.0071402676991737055,
+            -0.020147891665364803,
+            0.056472902950962155,
+            -0.006027534100459245,
+            0.005473493744726429,
+            -0.03139313966858068,
+            0.00043634965151753214,
+            0.013171990459657583,
+            -0.016474119463665594,
+            0.0075792906007691105,
+            -0.0032442837467945356,
+            0.02492845005795393,
+            -0.04366675539501486,
+            -0.006033838756045725,
+            -0.018660330752604155,
+            0.02088611082654532,
+            -0.019387265112916365,
+            -0.01222746328548615,
+            0.005977009213829137,
+            0.0012690574665308455,
+            0.05430319716489232,
+            0.058075720286678886,
+            0.02554562736016709,
+            0.02997758641056458,
+            0.00014178556329408164,
+            0.004139467932725944,
+            -0.004454862143349584,
+            0.031094774362479788,
+            0.0017837745263941452,
+            -0.013542834385584962,
+            0.01983846902748998,
+            0.010136662804503968,
+            -0.0387179389182337,
+            -0.06156798393873734,
+            0.015844092498186847,
+            -0.020820695961906926,
+            -0.02678200120240037,
+            -4.570339833017677e-05,
+            -0.015544616932001905,
+            0.005434882475939711,
+            0.050835936350804696,
+            -0.004985250023269435,
+            0.0016509410141492726,
+            0.022801366331672433,
+            -0.004080282248725238,
+            0.0078367492683837,
+            -0.00034876063727630653,
+            0.039772801289415724,
+            0.0288510926430687,
+            -0.022609202992981695,
+            -0.07091181813750308,
+            0.045194222785472735,
+            -0.03225547853169315,
+            -0.032585463285183205,
+            0.01670145480899693,
+            0.03286138231600811,
+            -0.009733112712376757,
+            -0.030197857075781466,
+            0.0731500616714208,
+            -0.005545147701735842,
+            0.017072915518860705,
+            -0.021002529816498382,
+            0.010525361666209598,
+            -0.012745643075817463,
+            0.025352505498532282,
+            -0.003102906957075941,
+            0.016391641977220053,
+            0.02534354270624064,
+            -0.008108690979345334,
+            -0.006451703723989414,
+            0.0065097623653779324,
+            0.04453924610212246,
+            0.0889545538715994,
+            0.019774398083206815,
+            -0.003981259900744729,
+            -0.046615274629371053,
+            0.00732791197790897,
+            -0.02916202133853672,
+            -0.04303144138453202,
+            -0.04367640508681853,
+            -0.010910501188273782,
+            0.023755744360759886,
+            -0.003399898430475389,
+            0.03910227884474275,
+            0.04137187598162877,
+            0.03363334616934253,
+            0.00916121492790106,
+            -0.031686589382074935,
+            0.026907320660055866,
+            -0.01693827759642556,
+            0.002542422168436765,
+            0.00734783450287368,
+            0.008189080458799796,
+            -0.006352648760821389,
+            -0.002333605867785783,
+            -0.03831808898525347,
+            -0.036762698962852246,
+            0.012773386000855496,
+            -0.008349610778914014,
+            0.027936454763857345,
+            0.07015560257394027,
+            -0.03713244149832974,
+            -0.0066245405611366675,
+            0.02802618921440782,
+            0.0017169880395367697,
+            0.023346396749133396,
+            0.007605522879978981,
+            -0.023484167590494896,
+            0.008594274917753499,
+            -0.0071765874950466295,
+            0.0677537727828362,
+            -0.008126000179550069,
+            0.011046754551416857,
+            -0.019516727128214784,
+            0.031654663874166704,
+            0.0007057013357055938,
+            0.021535694192642306,
+            0.03313372731162823,
+            -0.005794425715079591,
+            -0.03865395674172617,
+            0.032271538528276474,
+            -0.029790294001121286,
+            0.0013647166052139265,
+            0.014194415049797637,
+            -0.004899479294176156,
+            -0.015920769574443834,
+            0.0013708196727143433,
+            0.029757298661028133,
+            0.045717552753231847,
+            0.042911032378569906,
+            0.0016019404554248433,
+            -0.06844613496660937,
+            0.06365820914938357,
+            0.012895947255192629,
+            0.03208498337942363,
+            0.009724409704032419,
+            0.042361427690196844,
+            0.008120380964731038,
+            -0.0004384537671641681,
+            -0.005540416382489673,
+            -0.005125920468958917,
+            -0.021186731704155592,
+            0.03258917022091572,
+            -0.01443327736487225,
+            -0.007107908179884635,
+            -0.00206346150761894,
+            0.014042865018508809,
+            0.028517557440008168,
+            0.017283745982594707,
+            0.0029564774390786045,
+            0.04378212980825259,
+            -0.019454732028746872,
+            -0.009437148551161395,
+            0.018820938021162286,
+            0.0008903198503990673,
+            0.025297040557527045,
+            -0.0011429799732923295,
+            -0.01670583390791835,
+            -0.009421860087826363,
+            -0.0328764813310764,
+            0.009563088287577815,
+            -0.0015347427243049344,
+            0.019228666906882602,
+            -0.007174098839673089,
+            -0.04068735475327182,
+            0.008903031576514094,
+            -0.011472954071988202,
+            0.015735403376626513,
+            -0.007084760348074423,
+            0.008729329609340102,
+            -0.00638454270428521,
+            0.016210007209816298,
+            -0.02460475066044075,
+            -0.009047036293783832,
+            -0.022598561172568143,
+            -0.020144302113939427,
+            -0.010886272544512816,
+            -0.010886770387412563,
+            0.008591903125597681,
+            -0.012216526799178626,
+            0.023956058440750026,
+            -0.017402060516065108,
+            -0.018328740623422317,
+            -0.00046940812181371226,
+            0.022921229116282273,
+            0.009026508358954174,
+            -0.00446208425714566,
+            -0.04421346673289319,
+            -0.01214251475197077,
+            -0.016860336137191526,
+            0.005985059770480569,
+            0.02804423359971541,
+            -0.03510963462663318,
+            0.012588036039507566,
+            -0.0018909139908325232,
+            0.0019847602212350957,
+            0.011374423039138836,
+            0.015486762163219968,
+            -0.027040748338117617,
+            -0.016329845569406535,
+            -0.012600824227928017,
+            -0.01765236203915568,
+            0.01673827406092187,
+            0.0016641690585652725,
+            0.003997011410859195,
+            -0.00499922800986537,
+            -0.02782499674212141,
+            -0.008774389172636544,
+            -0.011506812701874534,
+            -0.009783555199790996,
+            -0.015740884114025612,
+            0.013493654174963239,
+            0.014254624157312108,
+            -0.033270488877124596,
+            -0.018446763439664365,
+            0.02247061755496804,
+            -0.0035396859846893343,
+            -0.03759408156279411,
+            0.04544560120093284,
+            0.052893430652542234,
+            -0.02220682841881151,
+            0.002828355769475809,
+            -0.03469141962159207,
+            0.05950248906327849,
+            -0.03444371252359212,
+            -0.03378776392860449,
+            0.06521072243320349,
+            -0.009713026070923203,
+            0.015249223153809483,
+            0.03131206469454657,
+            0.0034865285730464384,
+            -0.0025997752599464445,
+            0.00912836988165194,
+            0.033031075123542926,
+            -0.014878258361522118,
+            -0.013101723481793925,
+            -0.01797681408884685,
+            -0.002397237505725697,
+            -0.03438738791694146,
+            -0.021713511748922477,
+            0.00836495266415799,
+            0.013064910474172053,
+            0.017110113503472666,
+            0.017514586931181946,
+            0.016195079102773242,
+            -0.0024109742901844617,
+            0.01545592172528996,
+            0.001046760697009891,
+            -8.863738247355372e-05,
+            -0.010979717992873124,
+            -0.017762086340346584,
+            -0.052667758929128256,
+            -0.018996338856872796,
+            -0.04421628171000429,
+            -0.005982494173649518,
+            0.0012420734363470585,
+            0.009886466834005627,
+            0.04949340056515024,
+            -0.01295176306623405,
+            0.003508142432105508,
+            0.010930632194984236,
+            0.01019261004578912,
+            -0.006556730312565122,
+            -0.032765993356945154,
+            -0.006485728779853869,
+            0.025709969056450885,
+            -0.022932446214570953,
+            -0.021792413124305354,
+            -0.007047777916806282,
+            0.0038659532636633486,
+            0.02321123092409853,
+            -0.026003302747277822,
+            0.017643847938685852,
+            0.010129447679208158,
+            0.009551458127273026,
+            -0.017747795509094742,
+            -0.02891499441428287,
+            -0.030321565228144824,
+            0.039350178167852384,
+            0.02899873253152933,
+            0.011338359735211436,
+            -0.033634061116901,
+            -0.013629902520109783,
+            0.0376352218933715,
+            -0.0019251015746424632,
+            -0.07267257827375968,
+            0.036685408572352814,
+            0.0050526642351482985,
+            -0.00019289056191322955,
+            -0.013581714265389013,
+            0.05300479823846402,
+            -0.04139717386779843,
+            -0.03335524761191713,
+            -0.00920987783097114,
+            -0.022616612319255443,
+            -0.005274093640024995,
+            -0.0012667939280239298,
+            0.032502321377997974,
+            -0.020089629343721754,
+            -0.024354491736044894,
+            -0.010206744478678104,
+            -0.02465036256948816,
+            0.017966455981144307,
+            -0.012995995315432267,
+            -0.02135780149071486,
+            0.010967719068923269,
+            -0.03402154331207373,
+            -2.0626689329697705e-05,
+            -0.01092309016851862,
+            -0.018683564965672157,
+            0.01571041043301007,
+            0.016259448102797114,
+            0.015403085986247105,
+            0.0020047641027155683,
+            0.015172380396260792,
+            0.008099191720720747,
+            0.004074377809743208,
+            0.01893667379047588,
+            -0.03384452762579117,
+            -0.038822883991989264,
+            0.050315952008617104,
+            0.0008458969404873289,
+            0.0012049589464131246,
+            0.006406225290437734,
+            -0.0042817559898658255,
+            -0.0411169553122232,
+            0.012933323148113837,
+            -0.05784927886369837,
+            0.006262727192870561,
+            0.012928199517490538,
+            -0.01786898028943474,
+            -0.02291183998443847,
+            0.00046953379111417065,
+            -0.024864892714864046,
+            0.02151263275001755,
+            -0.017247234755598105,
+            0.020405349906524557,
+            -0.023068655052252782,
+            -0.006872857342813033,
+            0.05842934123173711,
+            -0.02308642746390288,
+            -0.04925938088613823,
+            0.032829595473385986,
+            -0.007184802367095134,
+            0.03878513331161335,
+            0.055971017386540985,
+            0.016337316575743225,
+            0.006439501224944926,
+            0.026084366083114555,
+            0.013204436213767278,
+            -0.012028383848550473,
+            -0.0383295105300241,
+            0.015805232298071025,
+            0.030751727443425712,
+            -0.04685643586863969,
+            -0.010258856888449854,
+            -0.02541648783174773,
+            0.01599307149086869,
+            0.020046184897703115,
+            -0.0044598427609889935,
+            -0.0012270315730225135,
+            -0.009664326045898607,
+            -0.01520444928791987,
+            -0.02631191309701645,
+            0.02193635473056903,
+            -0.013740214173212537,
+            0.0007504924364311022,
+            -0.001111971939163715,
+            0.0012797636039507028,
+            0.008478066024469051,
+            0.01501219347281274,
+            0.027440349509054086,
+            -0.022728634568050663,
+            0.006916913421268421,
+            -0.008499438828073956,
+            -0.03788213665656746,
+            -0.006718262893819488,
+            0.029198375723488434,
+            0.006169843558767785,
+            -0.016427547826608675,
+            -0.0049009468913267205,
+            0.0022509753971959087,
+            -0.016307639896946134,
+            0.00372456949880952,
+            0.010347360137213142,
+            0.035374109073612686,
+            0.01856811791368892,
+            0.019247903053478954,
+            0.0035591273657627456,
+            0.010292179556783772,
+            0.004040044337368795,
+            0.025869815038962875,
+            -0.04526297785801751,
+            0.030173313035569235,
+            0.01521363974209499,
+            0.009059160353158267,
+            -0.03914219183565228,
+            -0.02367109515190293,
+            -0.025737979702884085,
+            -0.021654077593318197,
+            0.02346511753268162,
+            -0.02730833269330862,
+            -0.006289840626554075,
+            0.006531934540756005,
+            -0.0006804032161296996,
+            0.003928945341546481,
+            -0.0033962179480786624,
+            -0.037975452991752895,
+            -0.009840510213526213,
+            -0.04023861493405766,
+            -0.0241245373830213,
+            0.003822063151525241,
+            0.003249755356115233,
+            -0.06594891648887229,
+            -0.001768385222533634,
+            -0.06222620899240395,
+            -0.013594921028316139,
+            0.03441942883093182,
+            -0.024438842229617237,
+            0.04820486752403123,
+            -0.05486229161412509,
+            0.018219866569279347,
+            -0.046562594596398506,
+            0.011354841468658838,
+            0.021738373879790916,
+            0.04347115691988908,
+            -0.007572536447300832,
+            0.014626550834376365,
+            0.009906079088662038,
+            -0.04530647508884267,
+            -0.04520033458696902,
+            -0.011400230794947595,
+            0.011239630851440557,
+            0.003287355301854498,
+            -0.012304101523783504,
+            0.02623659451177029,
+            0.015903981896266153,
+            0.04300392046214319,
+            0.03414364576071245,
+            -0.0005691335103219632,
+            -0.049506948352089984,
+            -0.027336474868358992,
+            -0.014910043127825028,
+            0.02552137904202476,
+            -0.03055258097953423,
+            0.003401887357920115,
+            -0.03228200232009397,
+            0.013338682663233296,
+            0.018207379090305786,
+            -0.02224210534533673,
+            0.032569850137027664,
+            0.014446434357859929,
+            -0.040301668913492676,
+            0.0012399866793165108,
+            0.0002450513991178417,
+            -0.038300625347349834,
+            0.008119751344998158,
+            -0.013712452991626932,
+            -0.05381257202269606,
+            -0.022199278450525977,
+            -0.014717114803771204,
+            0.02206967996899982,
+            -0.0023470596291977116,
+            0.017222681546831476,
+            -0.029965631581610256,
+            0.0053103260868707786,
+            0.039279027237044066,
+            -0.001286692479487674,
+            -0.01148541970963179,
+            -0.0008706049086180766,
+            0.022795196067681996,
+            -0.01968201411368477,
+            -0.045534614872904305,
+            0.009877856016973188,
+            -0.020566606587457816,
+            0.016002665913994348,
+            -0.015445999456355206,
+            -0.0052940252527317235,
+            0.010360748741468617,
+            -0.03170100753615497,
+            0.02004511444257144,
+            0.009722541379665509,
+            0.006411072442536516,
+            -0.017460683536795665,
+            -0.0033423349847155888,
+            0.015743901543629382,
+            0.047005401368949463,
+            0.0007329338628268773,
+            0.006153613674047577,
+            0.02501606022399451,
+            -0.013054974002495664,
+            0.05036387123988784,
+            -0.02807526639433245,
+            -0.03099731580589117,
+            0.04344605263104773,
+            -0.019489740746149623,
+            -0.017636713524640496,
+            -0.017960525325193457,
+            -0.009792405274000215,
+            -0.0062801692104839674,
+            0.02801421653619339,
+            -0.00712206060099242,
+            -0.005749470909187103,
+            0.020504942834135256,
+            0.013170883165927672,
+            -0.011218763497707175,
+            0.008523945766715354,
+            0.01511514654097483,
+            0.027931571407773977,
+            0.03214582418497515,
+            -0.02345593941074465,
+            0.0333746849861944,
+            0.009324947617983663,
+            -0.004234113006464689,
+            0.005131158074842622,
+            0.0020437263732432953,
+            0.00176803337042029,
+            -0.029372739581870692,
+            -0.01539406767336474,
+            0.015958225909710817,
+            0.007396588215584237,
+            -0.003291618431912445,
+            -0.0316834931316845,
+            -0.0028368107695447525,
+            0.02614797456533501,
+            0.003832488818435695,
+            0.026426519154286857,
+            -0.015605455255333021,
+            -0.02018445776455915,
+            0.008876565653043164,
+            0.023669947107607083,
+            -0.055401884612295824,
+            0.01882053182190406,
+            0.011046506385076857,
+            -2.6003594081234644e-05,
+            0.01591257026228522,
+            -0.0046432737555961385,
+            0.005094314300671912,
+            0.0482159962414822,
+            0.015126643838581288,
+            -0.013318945197936635,
+            0.0020418779929576307,
+            0.015136735547940301,
+            0.001727083579122356,
+            -0.020116332736417462,
+            0.002896210161586164,
+            0.012858057228188658,
+            0.0092589465483404,
+            -0.012984552574495295,
+            -0.005651788746934606,
+            -0.016856929103895944,
+            0.03249171052801259,
+            -0.005843029913406538,
+            0.02336833175997056,
+            0.04800276014595125,
+            0.02261889070736121,
+            -0.03578655846417886,
+            -0.011451513080556048,
+            0.0077334037670644135,
+            0.009415939321950421,
+            -0.012399032514032437,
+            0.006084821107593102,
+            -0.034655634244304866,
+            0.013104268092544735,
+            -0.0016781301605750378,
+            0.012249966178417542,
+            -0.025567673456449492,
+            -0.023488408901029257,
+            0.007732021927617865,
+            -0.006986715221001125,
+            0.01461433573263053,
+            0.00011845838735106802,
+            -0.005506714145045683,
+            -0.006827219009979501,
+            0.0005242424465709722,
+            0.002885486381705293,
+            -0.0042669493792680474,
+            0.021618593454103052,
+            0.00010034640948597114,
+            0.011602625527297699,
+            -0.023477647311478233,
+            -0.0162254255990209,
+            0.031352657169434826,
+            0.03529846802317018,
+            -0.0007059281329032357,
+            0.02568807107901636,
+            -0.00277252926066966,
+            0.010828396988798257,
+            0.00810821945842862,
+            -0.010307381937016405,
+            0.0003407672690264756,
+            -0.003768030765932326,
+            -0.0027367413382233317,
+            -0.012887542072043335,
+            0.0031930180018598015,
+            -0.014086667286395425,
+            -0.01504545079945656,
+            0.02010182276528891,
+            -0.002519095978589373,
+            0.011477055159845283,
+            0.006165264376334778,
+            -0.01139265727997407,
+            -0.016539552549751473,
+            0.01574258576985793,
+            0.05233886430069725,
+            0.022428487725773775,
+            0.044334901638531425,
+            -0.00706941862193002,
+            0.011585028247041065,
+            -0.016760013805097753,
+            0.001211362507398395,
+            0.048283299647824234,
+            0.015403290676022106,
+            -0.013446978976128664,
+            0.024015090204073844,
+            0.013753743307879434,
+            0.0398341544586196,
+            0.0034302789804899806,
+            -0.020152697770903245,
+            0.041614050886963766,
+            0.00392650113407049,
+            0.013465009394885544,
+            0.020487015678247938,
+            0.002430045518615804,
+            0.0035516244267459565,
+            0.02938659684630997,
+            -0.010142701862398358,
+            0.010335745659719125,
+            -0.033187389861872627,
+            -0.021792906328508686,
+            0.02532137764121119,
+            0.0005114146307176893,
+            0.033795722059780535,
+            0.007069260093915293,
+            -0.017628598898476375,
+            -0.017395163521872672,
+            0.021985468266342154,
+            -0.007987351575783343,
+            -0.025037973555114328,
+            0.002627649768366858,
+            0.008700784263932844,
+            -0.004480915144455101,
+            0.006798551384053197,
+            -0.014580806303786968,
+            -0.009741248878656092,
+            0.025066050175629597,
+            0.007330922027951371,
+            0.010472464570407973,
+            -0.024286928122882124,
+            -0.011159235410800293,
+            0.006107141408585794,
+            -0.04181412282877586,
+            0.008647396691893692,
+            0.00528510107024571,
+            -0.006647986735406744,
+            -0.002548771993161688,
+            -0.0011322362644692566,
+            0.00513401828632921,
+            -0.0035963203271502263,
+            -0.014021183815351008,
+            0.022993860073007393,
+            0.02875036422479212,
+            -0.012996574913859303,
+            0.014373647963439562,
+            -0.0093541571449835,
+            -0.019474535598737786,
+            -0.0233122546137944,
+            0.014269723193443946,
+            0.008784376430480408,
+            0.004317715767550436,
+            -0.008273153355003107,
+            0.0010203136295909875,
+            0.001909823045834306,
+            0.02406426043795662,
+            0.049377235716785324,
+            0.005265955549193235,
+            0.029166436381966566,
+            0.017136545810044906,
+            0.03882940870384413,
+            0.0007533463090350569,
+            0.016233986042907378,
+            -0.03311355052072762,
+            0.03082356484543373,
+            -0.0009431717656252606,
+            0.04077734731808,
+            0.0369941736537998,
+            -0.05477277258937169,
+            -0.014658721668336077,
+            0.028327060073149418,
+            0.010896122046930724,
+            0.01436699901738847,
+            -0.003307011398652852,
+            -0.01163199508062231,
+            0.029659278016495517,
+            -0.023936970669211955,
+            -0.010748756889752312,
+            -0.00767811402724157,
+            -0.010785978668072403,
+            0.009408061520238972,
+            -0.0038320809092061126,
+            0.038383104318077226,
+            0.0070413219383093,
+            -0.009684310528333753,
+            0.019154521368080882,
+            -0.006292471963504016,
+            0.01596901434992771,
+            0.0037167338451907566,
+            0.0019879266696342887,
+            0.03104390318244226,
+            -0.003561483519923973,
+            0.028949607591091022,
+            0.015156617705082505,
+            0.017473301120296506,
+            -0.0029321911929339165,
+            -0.017573294769065502,
+            0.027284173131869906,
+            0.009113069584708108,
+            -0.01971507323375489,
+            0.003667023499937692,
+            -0.0025339283534597176,
+            -0.0077697455666997945,
+            -0.010491575421861018,
+            -9.407001004405342e-05,
+            -0.00211520268689493,
+            0.02087980536494742,
+            -0.005582301812064227,
+            -0.0016376727058019977,
+            -0.0064225805186745605,
+            0.0052835419390265485,
+            -0.019662445004865352,
+            0.015117937189193789,
+            0.007575238218238001,
+            -0.018537762525347407,
+            0.014673223139419801,
+            0.019165766004277593,
+            0.0343151018851732,
+            -0.023949650834192267,
+            -0.0009279394158895472,
+            0.01730410035639405,
+            0.005928527001253011,
+            -0.018975612465080345,
+            -0.033873819454780706,
+            0.00825263613547798,
+            -0.01745801647450136,
+            0.027401724356781083,
+            0.008830481774058282,
+            0.0347823637819511,
+            -0.02309575756624861,
+            0.020559439488958053,
+            -1.2525886083543457e-05,
+            0.01593239963830551,
+            0.01455178268663934,
+            0.027993448238508672,
+            -0.017573784980205873,
+            -0.0379374065982424,
+            -0.0030450459335320053,
+            -0.0029968228865394214,
+            0.016851016821808874,
+            -0.013002468483615099,
+            -0.003337198735398279,
+            -0.0019044899095842511,
+            -0.029686441224288696,
+            -0.021487593939979652,
+            0.027796807698461906,
+            -0.04637191752173605,
+            0.0030176823995715597,
+            0.03281049040516897,
+            -0.030047693785808505,
+            -0.04246000386774685,
+            -0.0014603539776430917,
+            -0.03560639883119047,
+            0.0010602583020630973,
+            -0.01222013383414755,
+            0.029128085483749913,
+            0.007388529661657186,
+            -0.020518571527741426,
+            0.018184513437823,
+            0.0076594003031722,
+            -0.004161472492195519,
+            0.006100954284405263,
+            -0.005239699097592758,
+            -0.028136388241670696,
+            -0.015254134018585082,
+            -0.03037392857155718,
+            -1.6317930920712043e-06,
+            -0.002478230556191563,
+            -0.012845180214134591,
+            -0.024383682688014996,
+            -0.000484419407302015,
+            -0.014810859291916117,
+            3.958274758340876e-06,
+            0.05136363130801806,
+            0.02120810926469503,
+            0.005060247818396135,
+            0.012869784532979883,
+            -0.019665446839172217,
+            0.04743686110578455,
+            0.003514534022065111,
+            0.03583012040901262,
+            0.002062991366066978,
+            -0.02173965243546218,
+            -0.0005002960506349255,
+            0.0319890423992129,
+            -0.0016894669649440992,
+            -0.017249094049398075,
+            0.006136324758346181,
+            -0.009220681391996718,
+            0.00837047468537671,
+            0.012871598576797918,
+            0.01579543926540845,
+            -0.0061999233512262265,
+            0.014725777187858035,
+            -0.003927118678273384,
+            0.009336164668870259,
+            0.03122006739418818,
+            0.019771411252821138,
+            -0.024716950288097537,
+            -0.037736399088695305,
+            -0.012523788001500705,
+            0.001234987976751515,
+            0.034477269953960235,
+            -0.013836458205716058,
+            0.003837278407556882,
+            -0.011027628278972294,
+            -0.0054689562005572675,
+            0.005739209890610896,
+            -0.006985434270987544,
+            0.0015806421600885957,
+            0.012196603223058041,
+            -0.006859901421089723,
+            -0.025670287745499374,
+            -0.004011122451643543,
+            -0.023258896941014143,
+            0.012997299613183496,
+            -0.010144866692641839,
+            -0.006816231069049567,
+            -0.005213359047878286,
+            -0.0014167278420750282,
+            -0.0062535968890290545,
+            0.0006662897611595949,
+            -0.014236807237438571,
+            0.01012846684096945,
+            -0.00899852930003816,
+            0.0016620397335915663,
+            0.007838787681725669,
+            0.007725632848979133,
+            0.0033110678307931205,
+            -0.008206916937522372,
+            -0.008777095103563134,
+            -0.01631820698659747,
+            -0.032412484976539334,
+            -0.03000207979123948,
+            -0.02276516767125653,
+            -0.003870969029785218,
+            0.0034978865240734053,
+            -0.032594545526009384,
+            -0.020635972136797354,
+            -0.002367523739258502,
+            0.0030103817989158062,
+            -0.014587341442812854,
+            0.0014779462725289245,
+            0.007969995140846214,
+            -0.0029319878689507335,
+            0.019355924479361485,
+            -0.005448177691636249,
+            -0.021203447333791942,
+            -0.028956667864566195,
+            0.017851274292380093,
+            -0.015891501231440428,
+            -0.010070179095955473,
+            0.0064388768290910715,
+            -0.005633920123057214,
+            -0.03957042012546069,
+            -0.008945181709739484,
+            0.010992989857329521,
+            0.0035378401849368002,
+            -0.025936727040303652,
+            0.031903561853958325,
+            0.030132441728296204,
+            0.01845519141895959,
+            -0.017902241158092712,
+            0.04710441323879975,
+            -0.006739916600200837,
+            -0.012103058303430145,
+            0.016276540022040643,
+            0.023963592487909288,
+            0.028774476787161263,
+            -0.02643829153200483,
+            -0.005349178006882811,
+            -0.010545703050010008,
+            0.005354643441490569,
+            -0.02271588009302058,
+            0.008468528451368553,
+            -0.013601178278798316,
+            0.011576067943282815,
+            0.019227032134009135,
+            -0.013197207810056759,
+            0.02629307658052319,
+            0.0045263515787155985,
+            -6.067227645639678e-05,
+            -0.0037770406201685248,
+            -0.016287414859552752,
+            -0.012319957965328086,
+            -0.034674880061192896,
+            0.020297690806102293,
+            0.01889209267823917,
+            0.003953684662429981,
+            0.021845046751532583,
+            -0.049434011185276405,
+            0.02806744160678342,
+            0.0017849117483741536,
+            0.021763434864703356,
+            0.013889518910180847,
+            -0.001843068102127134,
+            0.013886691089114936,
+            -0.0032117229736863266,
+            -0.019971320584891997,
+            0.006079730961157539,
+            -0.006352535944493006,
+            -0.03316544192496317,
+            0.014736816574462923,
+            -0.03268321622529288,
+            0.013639134163846678,
+            0.015454170145821088,
+            -0.010170396823729307,
+            -0.021708948733883182,
+            -0.008303375382721087,
+            0.0010570623547459114,
+            -0.0014149688171089135,
+            -0.006849845444850982,
+            -0.04346483266888068,
+            -0.03020319096411844,
+            0.03124837430439466,
+            -0.0016720020681567616,
+            -0.03404132565569305,
+            -0.007209824644083462,
+            -0.02711263625315296,
+            0.004780610743996515,
+            -0.004212647822173262,
+            0.0025676935176963222,
+            -0.048750272936498695,
+            -0.01018434342955251,
+            0.011179560412168381,
+            0.031648611061279724,
+            -0.029298960703078986,
+            0.021370266671953315,
+            3.770954396468699e-05,
+            0.006629905234129844,
+            -0.00943632381701963,
+            -0.00484754585233744,
+            -0.012115337995959813,
+            -0.02607251387584821,
+            0.021868809148577698,
+            -0.0020587964818437345,
+            -0.0013502026980372474,
+            0.0341522429571705,
+            -0.00669289309893371,
+            -0.010113192232031375,
+            0.0005934017402521538,
+            -0.011327797122243258,
+            -0.017269539623150844,
+            -0.010467173785198161,
+            0.0005779464354975677,
+            0.0030406502310922306,
+            -0.024755136158113783,
+            0.004943446717510779,
+            0.04566273647510376,
+            0.013607936503018211,
+            0.02534200532143803,
+            0.017104750405207106,
+            -0.03792550084708064,
+            -0.04394052120951292,
+            0.0012600504352102988,
+            -0.005807815271334456,
+            -0.01944685298932453,
+            -0.01732108197375621,
+            0.0011677414026425442,
+            0.00790730598557849,
+            0.0037240231183698074,
+            -0.011427532982681041,
+            -0.02086964367350831,
+            -0.032539146090370744,
+            -0.0136797717535217,
+            -0.009464326388205681,
+            -0.007302204556505735,
+            -0.032703238152349344,
+            -0.015929357859286307,
+            0.026848214129378756,
+            0.01012098852614113,
+            -0.005646819862636667,
+            -0.0107564788588665,
+            -0.024118155140225247,
+            0.007122339033521904,
+            -0.0027183981945039814,
+            -0.009219030314883272,
+            -0.004672057262936543,
+            -0.016450690196817726,
+            0.02348093022809639,
+            0.009139210149358477,
+            0.024997129282118148,
+            -0.058533911093319214,
+            -0.014411932474734742,
+            0.014256038602455438,
+            0.0020188651815054508,
+            0.001417136751974749,
+            0.029836680491427268,
+            0.0028786535004858044,
+            -0.006526872714072965,
+            -0.02967220833325222,
+            0.028817295504169067,
+            0.027449694157944637,
+            0.01800200378294509,
+            -0.010560094796033294,
+            0.0033996190760546383,
+            0.006004691351249025,
+            -0.005679195734863024,
+            -0.011927786207717698,
+            0.0021260244673648285,
+            0.018998574211603905,
+            0.017174632765532803,
+            0.014659925822127794,
+            0.02455156987384756,
+            -0.005882590537857664,
+            -0.03911277764764406,
+            -0.009781110378157883,
+            -0.0119630929326396,
+            -0.004815125964940769,
+            -0.0026631770014339242,
+            0.008903827530107938,
+            -0.011824767329332145,
+            0.013088406716464123,
+            0.030264448923574067,
+            0.042592268021664714,
+            0.021292401951863188,
+            0.006150997237661769,
+            -0.005463801502425317,
+            0.0007399518998122848,
+            0.0038649546282459707,
+            0.011355675550395135,
+            0.035762872771750435,
+            -0.018147218572765546,
+            -0.019704502923991365,
+            0.0015558311499702044,
+            0.023543487977413267,
+            -0.007150197116420024,
+            0.02608209797565159,
+            0.023985156585794257,
+            -0.0003434024317322991,
+            -0.01271338268931938,
+            0.000985282363050309,
+            0.006710364567868783,
+            0.014540943712513688,
+            -0.02154205198718209,
+            0.004039619025717627,
+            0.001865104196024055,
+            -0.009756600776161738,
+            0.029738871754072076,
+            0.00039154552800346406,
+            -0.012981811693538832,
+            0.03825344802886,
+            0.11025963334575106,
+            0.001476339401950677,
+            0.0031187927813794386,
+            -0.031522639386920355,
+            -0.015113380587160163,
+            0.0021786482865071505,
+            -0.005948756393737655,
+            0.020082591515778573,
+            0.012751062800862873,
+            0.0055847042526916265,
+            0.010100961720464159,
+            0.038084609873482715,
+            -0.009203643420546616,
+            0.009804311800033327,
+            -0.01878272400403416,
+            0.017185477997947226,
+            -0.00972716681599207,
+            -0.00857276597428136,
+            -0.007641511960016827,
+            0.004167416313632853,
+            0.00984005584819001,
+            -0.004596999098606572,
+            0.018810908413859465,
+            0.002496310438422421,
+            -0.01613133688770545,
+            0.011505030137914962,
+            0.007437924384075958,
+            -0.005386885126779478,
+            -0.031755982499042136,
+            -0.013811586180238287,
+            0.01190667400285204,
+            -0.0013786018401773159,
+            0.0007177632889203917,
+            0.012342640512536284,
+            0.01205074763420124,
+            0.031217897495485192,
+            -0.0244324065837541,
+            0.0025071378510626696,
+            0.0070486374045588465,
+            0.0159322020159265,
+            0.01098799814774876,
+            0.00029844505852147526,
+            -0.015456787478044924,
+            0.027139764983358832,
+            -0.011978684728746428,
+            0.010378658252224036,
+            0.01146607671795495,
+            -0.04131658683485702,
+            -0.0009967932006190922,
+            -0.005916133731248287,
+            -0.015698754902727707,
+            0.013905118384587169,
+            -0.040163447416240027,
+            -0.005682084326086952,
+            0.004297114532998809,
+            0.010724201884417054,
+            0.0058749102353125955,
+            -0.017715270733111323,
+            0.009962853888582856,
+            0.00955216765250444,
+            0.0025392532243413593,
+            -0.006356691980285544,
+            -0.006218169970979339,
+            -0.008606942887730427,
+            -0.004280171880086814,
+            -0.015669721147884084,
+            0.0057481312044666884,
+            0.0077227094511494055,
+            -0.018751123856436715,
+            -0.01585864749751703,
+            0.009183099781375234,
+            0.006837438250956817,
+            -0.001445622524955126,
+            0.004828557942941401,
+            -0.010034012200008746,
+            -0.004543834537450213,
+            0.01097540631691429,
+            -0.009334158551770033,
+            -0.004804430123454511,
+            -0.02609985520241612,
+            0.013342299390862539,
+            0.011103136832190223,
+            -0.008261332966534102,
+            -0.006944131083745896,
+            -0.016931502665633764,
+            -0.0031318730206304824,
+            -0.0026508016484082988,
+            0.0016110884325639078,
+            0.017071266355203386,
+            -0.009906320666640836,
+            0.010877416311741906,
+            -0.03496707406301918,
+            -0.015357787514985604,
+            0.01622830705930841,
+            -0.01290003882907745,
+            -0.005625420341281716,
+            -0.009838333350984609,
+            0.0006911383780218238,
+            -0.007545081680638308,
+            0.006074752194220675,
+            -0.041320607445282646,
+            0.0035102345636816475,
+            0.018825308640738208,
+            -0.009723969339534365,
+            -0.020829421583878845,
+            -0.009055117268112308,
+            0.010487235049013567,
+            -0.015831829161845813,
+            -0.01780922120240258,
+            0.0016219998217122911,
+            0.010559783293374071,
+            -0.0055135690971065965,
+            -0.005160232484458576,
+            0.006163424486940105,
+            0.020247627989655837,
+            0.015220964429515309,
+            0.04337432385613251,
+            -0.011915355138717245,
+            0.009617347907919507,
+            0.027812934121040484,
+            -0.030652481946156104,
+            0.01037470561447702,
+            0.02599512377987521,
+            0.0037340561802882417,
+            -0.0013375196064018082,
+            -0.010197691376747194,
+            0.004161692215514214,
+            -0.012254402844820636,
+            0.008715552098450461,
+            0.004697361473725416,
+            -0.027978648968732336,
+            0.017196771853396124,
+            0.019448218659009996,
+            0.0038720194629308794,
+            0.0006995062900842749,
+            -0.01527901684789497,
+            -0.011825577013200781,
+            0.019194236643713274,
+            0.027690065919038755,
+            0.005460028121722145,
+            -0.015512497062157167,
+            0.02370643287056251,
+            0.012916257803027072,
+            -0.004973882005506947,
+            0.0022332716531575122,
+            -0.016984605526823043,
+            -0.01443316577897661,
+            0.030942241604983316,
+            0.011563751997976407,
+            0.01908030186367601,
+            -0.021614212853976106,
+            -0.003982310717524619,
+            0.019931579069642925,
+            -0.021029366993395197,
+            0.013006879711144009,
+            -0.019497570222553013,
+            -0.014067925104548129,
+            -0.010687576618198498,
+            0.02884494104893037,
+            0.009229770955288097,
+            0.026701705044769222,
+            0.012715371448003833,
+            -0.021444875656527786,
+            -0.021050591467198086,
+            -0.0027703267406452336,
+            -0.02721606030674425,
+            0.01688256721881862,
+            -0.055990128879151335,
+            0.0016563858333845547,
+            0.010911810464298247,
+            -0.008604898990983188,
+            0.004039521277648332,
+            -0.007858378479197747,
+            0.010021929060776957,
+            0.020926274809879138,
+            -0.02956229623532218,
+            0.01256060127924585,
+            0.01156331569664224,
+            0.0218688589850221,
+            -0.018861065515394883,
+            0.015339087155577661,
+            -0.00841001512858443,
+            -0.027396412391138852,
+            0.008323898095545889,
+            0.01763697170382491,
+            0.014160184414353919,
+            -0.011992345081843988,
+            0.004485633614765869,
+            -0.04000932898876895,
+            0.027999519977069676,
+            -0.016823500188831896,
+            0.002010511519234648,
+            -0.004537991252901591,
+            -0.0067219583064939085,
+            -0.008523354091034399,
+            2.682477256914193e-05,
+            -0.0005371719103087346,
+            -0.030920653652724776,
+            0.0012939076000836422,
+            -0.04144943703666512,
+            -0.02260198516292792,
+            0.013327519374598822,
+            -0.0012897465780583367,
+            -0.005059170593504863,
+            0.02999548027478466,
+            -0.02863650640718448,
+            -0.027272295036911975,
+            -0.014376307736824656,
+            0.03534844385341626,
+            -0.015358480569131909,
+            -0.000540068394391221,
+            -0.009287193719601908,
+            -0.02236594406404711,
+            0.01091078016405303,
+            -0.006469935602381157,
+            -0.0019272207412132534,
+            -0.01387843655423569,
+            -0.03972775413987396,
+            -0.005948332199741248,
+            0.019500395452013777,
+            0.0006122786247500248,
+            0.003656719755271774,
+            0.014473976840529128,
+            -0.0072278373928449675,
+            -0.0036623750417377524,
+            -0.0038809301345520044,
+            0.026929584793529297,
+            -0.022225170775945557,
+            0.009744440210592443,
+            0.004704883568603746,
+            -0.0005991305039095727,
+            0.017317686588588826,
+            0.000351879037187266,
+            0.0024030356296394478,
+            0.0029060795825935717,
+            -0.0035301907818594274,
+            0.02922907976800865,
+            0.0064070048516525,
+            0.0016291092834589405,
+            0.0001988403049139903,
+            0.004746570638478093,
+            0.01395197992587789,
+            -0.01069245020264236,
+            0.0157940052093005,
+            0.02865126209030693,
+            0.00021287181382353683,
+            0.018345429962063935,
+            -0.015921361819192573,
+            -0.036059435316760584,
+            0.019807131219096722,
+            -0.029642240399677812,
+            -0.003504127677138975,
+            0.022473005924273325,
+            -0.012903852741467797,
+            -0.0031750649075827082,
+            -0.01107957248024579,
+            0.027243132530764567,
+            0.023512079029549708,
+            -0.014856101416774535,
+            -0.015142539116300868,
+            0.004337433267586975,
+            -0.010210607143984678,
+            0.010939153927790107,
+            0.029382995339774957,
+            -0.011230337698862454,
+            0.011717621080350227,
+            0.001161341258722717,
+            -0.003778338045364591,
+            0.019300005069271462,
+            0.0027681684372537606,
+            -0.007401677666787755,
+            0.025628069381915978,
+            -0.008115857591022265,
+            0.013043669211084909,
+            -0.004672358463640539,
+            -0.012902494254220482,
+            -0.003106794914564588,
+            -0.017558145644151636,
+            -0.01225024815282763,
+            0.0011522376202485873,
+            -0.005528597019035285,
+            0.06711658458916447,
+            0.010930891527455397,
+            -0.017888423424597618,
+            -0.0059190199704595775,
+            0.00955567741445378,
+            -0.000966061435587513,
+            -0.009466328431452424,
+            0.027063767833815658,
+            -0.016888493246695876,
+            -0.009681104485277698,
+            -3.814655691584766e-05,
+            -0.0233149747136092,
+            0.0030596902889784496,
+            0.010827987186229144,
+            0.0012052973350181351,
+            -0.013994504803884577,
+            0.005318095523120499,
+            -0.004843763992616558,
+            0.0034188401938723923,
+            0.003678626789714963,
+            -0.0025749484272345573,
+            -0.005333294277998365,
+            0.02307826358450567,
+            -0.012791914224263883,
+            -0.008519982879775168,
+            0.01173424692936401,
+            0.001777123104204978,
+            0.003219553788359312,
+            -0.004564745566711077,
+            -0.002015107048152335,
+            -0.017123545909339455,
+            -0.011278616226036531,
+            0.023713473402287875,
+            -0.02733523566514418,
+            -0.00150943777243135,
+            0.017328382962199716,
+            -0.005705544203971564,
+            0.016650187392502975,
+            9.750950361601131e-05,
+            -0.02100066506382883,
+            0.00010090567939930427,
+            -0.024778741347883073,
+            0.010484659495766551,
+            0.013963183459158656,
+            -0.009531952698151101,
+            -0.020878167882780043,
+            -0.027881362421494915,
+            -0.010190031350698889,
+            -0.027246821338460653,
+            -0.0280692647056191,
+            -0.02020582179742232,
+            0.009778340348478517,
+            0.005715364307462615,
+            0.021929057840860695,
+            -0.006380523858228711,
+            -0.027552858236741976,
+            0.015095163660340326,
+            -0.013269344817111162,
+            0.008282918921343354,
+            -0.019991891695114384,
+            0.01754833974582726,
+            -0.014720830704852575,
+            0.02112161088830394,
+            -0.00041409266970033855,
+            -0.004490472334793758,
+            -0.012076561942010787,
+            0.003976913901514849,
+            0.008101357042612863,
+            0.00176590819486174,
+            -0.008977295353115363,
+            0.026242190932756135,
+            0.01707634775369835,
+            -0.007385093173634561,
+            0.01604386075569055,
+            0.031855080733360075,
+            0.014762206220119051,
+            0.038860660002328094,
+            -0.007238177370595944,
+            -0.009237098188374397,
+            -0.02764495845127417,
+            -0.004407870047633689,
+            -0.006820430763682645,
+            -0.004516837004332235,
+            -0.015251921783911087,
+            0.03164122113689215,
+            -0.010371400055822811,
+            -0.01686972159902524,
+            0.03979035122593764,
+            -0.021846728842184655,
+            -0.007084905708666271,
+            0.008642771511425465,
+            -0.013996636040226265,
+            0.03176661229383477,
+            0.007275612529363987,
+            0.018452818352301472,
+            -0.015039890816023917,
+            -0.015988701055097997,
+            -0.0019911190487748244,
+            -0.002806138139429132,
+            -0.020828646560807865,
+            -0.0063794261154720665,
+            -0.01614570050728432,
+            -0.0026319562705484634,
+            -0.00570753504957681,
+            0.015917008930969468,
+            0.0013916917451795444,
+            -0.015420032290910069,
+            0.04488171544231248,
+            0.005033499525006087,
+            0.022768662319611596,
+            0.010777933460735837,
+            0.012372342667053587,
+            0.015819823672091563,
+            0.006759824420801962,
+            -0.0001668726099910387,
+            -0.02211654164646909,
+            0.004261169256891085,
+            -0.005641717035723983,
+            0.0041005490092982915,
+            0.02012094437568505,
+            -0.024436624754739972,
+            -0.029708548399946657,
+            0.024493078535528394,
+            -0.01833101179919427,
+            0.0030908990956393764,
+            -0.0031025537219982234,
+            -0.010059653447038065,
+            -0.011612233298907742,
+            -0.03293899346571458,
+            -0.0005929867206193162,
+            -0.027438115397399914,
+            -0.03635734152790657,
+            -0.006287687204976032,
+            -0.007850474352061472,
+            -0.003574235211373211,
+            0.02449669755605501,
+            0.004269460596739739,
+            -0.029959856511873897,
+            0.01782698044631415,
+            -0.013252733005984911,
+            -0.013342147906120882,
+            0.0029581361690999564,
+            0.004786570391076097,
+            -0.0027618259744494854,
+            0.010720003765859173,
+            0.007898965393358715,
+            0.007358694001293221,
+            -0.010710502803266644,
+            -0.004049671482838962,
+            0.0013263710826446039,
+            0.0030943253037901135,
+            -0.013325799375903166,
+            0.030553973934607993,
+            -0.018812559793353154,
+            -0.004819166852190956,
+            -0.0006499911473215254,
+            0.0034556893023524674,
+            -0.0110929170801761,
+            -0.014862886414702483,
+            0.021004665462647988,
+            0.0031088550726355926,
+            -0.012570481398485388,
+            0.0096952264731202,
+            -0.0018868240406799254,
+            -0.0024767709620225923,
+            0.01176108203579327,
+            -0.002080730326597438,
+            0.014924914157541781,
+            -0.011040818699356567,
+            0.01490073610670544,
+            0.0034649445207609085,
+            0.008645740731109337,
+            0.003603641937848282,
+            -0.01905231764119889,
+            0.009780124784430865,
+            0.016356631955067272,
+            0.013415470760506568,
+            -0.029496530287207723,
+            0.008881639987471632,
+            -0.005975399013716009,
+            -0.009870580036511721,
+            -0.024372445588075428,
+            0.002590423942920311,
+            0.028960961301291905,
+            0.012044179251559706,
+            0.007934516374398097,
+            -0.0077949340827692054,
+            -0.005480592942515702,
+            -0.019923500717513334,
+            0.009041031662948522,
+            -0.01095651774810384,
+            -0.025157908389819465,
+            -0.007578712371715301,
+            0.03430545904801765,
+            -0.010740874257824011,
+            0.01343875386664532,
+            -0.005526757815283282,
+            0.02436552949831661,
+            -0.0027538610388853284,
+            0.0013690344627799137,
+            0.008416180197734094,
+            -0.006091433734127276,
+            -0.001708607133233843,
+            0.013349212286052506,
+            0.041066250931277824,
+            0.0038375614886832564,
+            -0.0050950709988837216,
+            0.01149207527692021,
+            -0.00326389523728132,
+            -0.005712157453687712,
+            -0.00309579814793649,
+            -0.025491974096530076,
+            0.015133356727314608,
+            0.0010745378801251735,
+            -0.017162228172591873,
+            0.021785292616181687,
+            0.027891598422935242,
+            0.013262622112280296,
+            -0.0178471114368855,
+            0.014456217536546996,
+            -0.009278392107874806,
+            0.01629461595238019,
+            0.02423194076020779,
+            0.010067296472207578,
+            -0.011865648674106681,
+            -0.0070073298858753805,
+            0.008985354045706703,
+            0.008096016282907443,
+            0.0004236756235618935,
+            0.020370823466310733,
+            -0.028334250141960886,
+            0.01483270667238479,
+            0.007137078270349882,
+            0.03314918453195744,
+            -0.0006231629401653488,
+            -0.014397909717816884,
+            -0.008918359605259833,
+            -0.002915473990900902,
+            0.011759189078587877,
+            0.003523992368256188,
+            -0.0006853609664806012,
+            -0.004749148984168426,
+            -0.014181883848149463,
+            -0.006778789302277201,
+            0.012376777107762334,
+            -0.0031646329407784245,
+            0.0030514749941358095,
+            -0.0007326603315688041,
+            -0.007811941797535734,
+            0.015661953163847418,
+            0.001669500869963878,
+            -0.0007143628544145883,
+            0.0016729300148826723,
+            6.604560293656475e-06,
+            -0.00467413979302959,
+            -0.019114577369302277,
+            0.008922356418978664,
+            0.00867683974577237,
+            -0.006578536023513047,
+            -0.006467824108991175,
+            0.009324373505311057,
+            -0.01432602608910974,
+            0.01852478469282692,
+            -0.0061254590933158826,
+            -0.006603842196861297,
+            0.00014277641689784565,
+            -0.012328738339684425,
+            -0.003642202698645936,
+            0.0034895681343535914,
+            0.0051453312116433265,
+            -0.000908593523739809,
+            0.0036062764245630334,
+            0.003940522150829955,
+            0.01083964694261132,
+            0.002824079422783228,
+            0.0015266576108775834,
+            0.007280263953807289,
+            -0.01039293741283943,
+            -0.007403606843286745,
+            -0.004993622657038095,
+            -0.006341662331986892,
+            0.007288958567606623,
+            -0.004364975055335228,
+            0.010457360052582658,
+            -0.0062292104145735515,
+            0.020471836369529147,
+            -0.006133009423930811,
+            -0.0024055488697334888,
+            -0.017695183455939345,
+            -0.007234478249839231,
+            0.007156119488831356,
+            0.007878044165936736,
+            0.009359297231370758,
+            -0.04487830669749157,
+            0.005038301058510894,
+            -0.0054192943462008595,
+            0.02941616633783806,
+            -0.009828941844770915,
+            -0.004335130343692854,
+            0.016067824555917796,
+            -0.019100842323053376,
+            0.0037066021889614513,
+            -0.013965982913833292,
+            0.003695036285467973,
+            0.0034558765116311525,
+            0.014205693020177169,
+            0.031307058208919394,
+            -0.007482180869231769,
+            -0.009383819319265733,
+            -0.006384083847148052,
+            0.026967380724567013,
+            0.018643952476557566,
+            0.023548246269020395,
+            -0.011885146764471029,
+            0.00960518654541979,
+            0.001357650512536872,
+            0.018271473493526338,
+            -0.009663920325675169,
+            -0.011163336807225407,
+            -0.000516828412255004,
+            -0.010288703356943627,
+            -0.0032111641654394615,
+            -0.021092460896394025,
+            -0.0033456309643941965,
+            0.025876107752976232,
+            0.0038584058642743987,
+            -0.0036689325941524813,
+            -0.015624044740149702,
+            2.2644755460272697e-05,
+            -0.011931276845804481,
+            -0.0055870010742218364,
+            -0.015697330226950804,
+            -0.028927689083316704,
+            -0.016192017705678192,
+            -0.002914436546908901,
+            -0.017224654082473787,
+            -0.009460990414684924,
+            0.002071634636546831,
+            -0.015533259213802986,
+            0.004918377471991415,
+            -0.015669642231243106,
+            -0.010915412245035698,
+            -0.00046472075835247,
+            0.004672390433470177,
+            -0.004682173537415046,
+            -0.024109027987987815,
+            -0.013628794248618615,
+            -0.0022054473551170926,
+            0.006500170428351522,
+            0.010016298501187265,
+            0.008324628078950177,
+            -0.0069374343736907625,
+            0.011201167788973098,
+            -0.009746811993090597,
+            0.010155637867574787,
+            -0.02773759960959338,
+            -0.04317847702979315,
+            -0.000791583637601122,
+            0.009241551203900427,
+            0.001979688190287368,
+            -0.01884488538245213,
+            0.0038489955050695704,
+            -0.011280060242688173,
+            -0.00847814532468811,
+            -0.007522173019881079,
+            0.01573091287228822,
+            0.004138348969142638,
+            0.013463997539337106,
+            0.0042065270170965705,
+            0.0285467221116185,
+            0.012635383326761538,
+            0.01030078781909351,
+            -0.003092067773698843,
+            0.004763648809248413,
+            -0.0022835042548997662,
+            0.009077632103272405,
+            0.02331475506951857,
+            0.0076225652102231605,
+            -0.002326207128504976,
+            0.00409858249509217,
+            -0.0005698076375460636,
+            -0.005442088641415082,
+            0.018565262151290427,
+            0.02915691852605287,
+            -0.0069811288298562155,
+            0.008120076286611811,
+            -0.0009110258445641983,
+            -0.009900329477406562,
+            0.0038288023984988134,
+            0.005765264445222657,
+            -0.014230725839572923,
+            -0.0024175374911696653,
+            0.009367987848760041,
+            -0.024664375361024386,
+            -0.004113757123212052,
+            0.020371264792326226,
+            0.0024813232005061487,
+            -0.016599537494340785,
+            0.03488073638324099,
+            -0.003709931703179228,
+            0.003010315958426392,
+            -0.00944047118505417,
+            0.016362178907316286,
+            -0.017989464717541103,
+            0.0005390907615384354,
+            0.013131172782591922,
+            -0.005251614069791733,
+            0.005783823857695169,
+            0.003744510952053123,
+            -0.010417674399178015,
+            -0.010824994224858631,
+            -0.0033951133363820767,
+            0.01085129781083312,
+            0.002642227227069058,
+            -0.024446190958178605,
+            -0.017731767444262785,
+            0.019381370848622046,
+            -0.01001206050737818,
+            -0.008290790009659911,
+            0.0009331678418736739,
+            0.0029270629970877303,
+            -0.004637651319894003,
+            -0.004290917518172777,
+            -0.003574121539057569,
+            0.009121344354139133,
+            -0.009485343005168732,
+            0.01637203851901565,
+            0.004043708359084711,
+            0.01843096334879628,
+            -0.011588935298420727,
+            0.005808551475129831,
+            -0.001368952044937682,
+            -0.007242291047654279,
+            -0.019004856431577073,
+            0.0059537808323522675,
+            0.0073777483129045685,
+            0.016051495408653346,
+            0.011060627996179347,
+            0.018569088420830965,
+            0.011345955812555888,
+            0.0039209921807600205,
+            0.001180658155391396,
+            -0.011006781869601802,
+            -0.016371246428912994,
+            -0.014232536816028922,
+            -0.007416908142074357,
+            -0.019561375671062513,
+            -0.0013253236527051844,
+            0.017798265625542643,
+            0.014528038148955117,
+            0.017704367192118375,
+            -0.0048389172759564884,
+            0.016018290057587425,
+            -2.7789810195444745e-05,
+            0.0009215076153434622,
+            -0.005897137279173464,
+            0.001794778099552753,
+            0.016314258230521846,
+            -0.003313666704140227,
+            0.0063152813149371136,
+            0.0002935873451081568,
+            0.01645203621687452,
+            0.022650990579075773,
+            -0.018899997585540014,
+            -0.0014442758993479705,
+            -0.006467599303099793,
+            -0.0021225224281092863,
+            -0.019894752041664215,
+            0.020266472457121338,
+            0.014019662562577204,
+            -0.005002061958298289,
+            -0.00943116174529331,
+            -0.004354703314807638,
+            0.02678118874395904,
+            0.020206530983694557,
+            0.006044260997975214,
+            0.026222728100119382,
+            0.01976601108150248,
+            0.01984865298844998,
+            0.011918890024498122,
+            0.009052182076567042,
+            -0.013092243192366595,
+            0.018358346459726883,
+            -0.01681770553983246,
+            0.0008504895530001561,
+            -0.003223133403363917,
+            0.002743153478851782,
+            -0.007937179998086906,
+            0.013021430914653432,
+            -0.00030955362267617273,
+            -0.025935204965869492,
+            0.007680669831594167,
+            -0.013183173230049736,
+            0.001375652741144253,
+            6.07690288088255e-06,
+            -0.001857629339163609,
+            -0.017652365850892467,
+            -0.0082709435899296,
+            0.0008448530375262078,
+            -0.01435595518048752,
+            0.023866770609514233,
+            -0.026826645910954506,
+            -0.02140002089501948,
+            0.00988141127158139,
+            -0.004436844338114915,
+            -0.012632978351515892,
+            -0.006972217855086974,
+            0.021864041851582767,
+            -0.005139891275508354,
+            -0.016596866242234234,
+            0.014168148679282804,
+            0.0005869000756971533,
+            0.00412958763467087,
+            0.0012364712526909765,
+            -0.008001798306987228,
+            -0.0030416500521329983,
+            0.006860667487819997,
+            0.028693213258586023,
+            -0.011895509688811421,
+            0.0016632663352826491,
+            -0.013799461077520309,
+            -0.0037922015693749754,
+            -0.02974150200357857,
+            0.027752493864472517,
+            -0.011830981692069491,
+            0.01989293240244598,
+            -0.004540523332199857,
+            -0.007667738626842118,
+            -0.021173122202316107,
+            0.035703796333655495,
+            0.004594443089494466,
+            -0.0006664409717898689,
+            0.007256660335524136,
+            -0.047588498838074554,
+            0.01105469709139337,
+            0.006498789757276137,
+            -0.023261890747840144,
+            -0.000964111432535798,
+            -0.003995007813335293,
+            0.02143067593410374,
+            -0.019461700447829744,
+            -0.012777737727876326,
+            0.0021024453304811634,
+            -0.021925886491102078,
+            -0.007056889535073896,
+            -0.0340604475320119,
+            -0.006137802768771592,
+            0.006503135515471076,
+            0.0038385769709340417,
+            0.0020799029890387023,
+            -0.021308499975514292,
+            -0.0309544649831362,
+            0.013308972238482328,
+            0.01651668347077017,
+            0.010007831710070633,
+            0.01200701454477496,
+            0.0067050757442456005,
+            0.011425518484475204,
+            0.00865023114755134,
+            0.00974582806570851,
+            0.008884712478547614,
+            0.039109252942848706,
+            -0.016122498485808056,
+            0.008671479031657285,
+            0.0005699279397862279,
+            0.02156543966774511,
+            -0.02391743870774471,
+            -0.009877938786506814,
+            0.015475531666380014,
+            0.003045729682805746,
+            0.030465388674890067,
+            -0.01511217342930631,
+            0.005735960079359902,
+            0.0030955813611916066,
+            0.003343849304714193,
+            -0.017685941897160877,
+            -0.019795694199965495,
+            -0.015889472669342385,
+            0.0005064000414834533,
+            -0.0007719666713392906,
+            -0.012817517834744753,
+            0.01370266779873307,
+            0.0029457043321269045,
+            0.004380574233134056,
+            0.003997906783171028,
+            -0.02610528890228495,
+            0.0033844277218836307,
+            -0.008397899012048187,
+            0.013703609594527308,
+            -0.0012702392316682096,
+            2.036849934035612e-05,
+            0.007261982817787643,
+            -0.00406175072535136,
+            0.018353316401139956,
+            -0.007206833180919801,
+            -0.004174154024217347,
+            -0.004097578846173991,
+            -0.00573702992914836,
+            -0.0017184308802689716,
+            0.022375713882789285,
+            -0.008746034802967245,
+            0.032729322137817145,
+            -0.014835515993879445,
+            0.01049714579786462,
+            0.013344412726432523,
+            0.006468296817982878,
+            -0.0036641997047404426,
+            0.0003384623491490165,
+            -0.0035345323089769998,
+            -0.006579417470328358,
+            0.015682816949756224,
+            -0.018934655843627496,
+            0.006860259740057359,
+            0.03727162736659208,
+            -0.002536806673033066,
+            -0.00615871353663963,
+            -0.008853057427430264,
+            -0.012968501800661,
+            0.028268388127570685,
+            0.005935705363468247,
+            -0.0013092005585187737,
+            -0.03024974000885413,
+            0.015180604705329429,
+            -0.0037501687534873948,
+            0.0016387074669205704,
+            0.00860407823411825,
+            0.00044285387135976006,
+            0.020127437993717013,
+            -0.0011272951769363546,
+            0.021290480413673304,
+            0.010032359217391118,
+            -0.0010666180914844765,
+            0.011529468490024128,
+            -0.036558434912810016,
+            -0.008579049583792913,
+            0.0038048395503202733,
+            0.019444233855445248,
+            -0.0006623807457136332,
+            -0.015853103419331053,
+            -0.004242407392496502,
+            0.009029483146616182,
+            -0.01059695082306233,
+            -0.013492380798775181,
+            0.016259633969459952,
+            -0.019096544374819852,
+            0.011560512412214506,
+            0.008617334373339341,
+            0.02512750386847096,
+            0.0041202753709624245,
+            0.002613588241299414,
+            0.0035069606381226207,
+            0.006602772841623022,
+            -0.002625433099855357,
+            0.011789793979724633,
+            -0.011592113209370399,
+            0.02102479518464762,
+            -0.008294039267346376,
+            0.007068661518691129,
+            0.0032541671568032306,
+            -0.022166914870027622,
+            -0.018234159989683987,
+            -0.0023642465680105544,
+            0.021312775708470108,
+            0.006016177710817625,
+            -0.012618763344460982,
+            0.0055411096806088885,
+            0.004690341026678943,
+            -0.00547301082703577,
+            0.009118596747082881,
+            0.024398283179066196,
+            -0.0028098889343668772,
+            0.01068095200128235,
+            -0.007137857371825465,
+            -0.027832295014375272,
+            -0.007421428223393234,
+            0.021707007996690945,
+            0.00020946296332069377,
+            0.016793613517175475,
+            0.014396690762872469,
+            0.012889706504719996,
+            -0.00488698200749025,
+            -0.005354238088985329,
+            0.018317372626432005,
+            0.014340608961907827,
+            -0.0048105230366691145,
+            0.001495716612807014,
+            -0.0018529641162211795,
+            -0.017240719704808234,
+            0.026933447120216628,
+            -0.024232238634425658,
+            -0.003438477892192365,
+            0.00892792234782449,
+            -0.012586276728773793,
+            -0.004923824123059311,
+            -0.023468301644204742,
+            -0.02117100775850015,
+            0.014296432515879623,
+            -0.003314359969711106,
+            -0.01750062790930347,
+            -0.006856892040084997,
+            6.621834660421394e-05,
+            -0.003983545180909168,
+            -0.010826144842427663,
+            -0.0028995230412062357,
+            0.0018029406642546938,
+            -0.01286603500476852,
+            0.00937188453692113,
+            -0.00635685147895941,
+            0.010242542894487791,
+            0.0062482230941642404,
+            -0.022974002980633818,
+            0.005235079251883088,
+            0.020160136045340955,
+            -0.008953506137454955,
+            0.0007027144749750591,
+            -0.015318432934167311,
+            -0.006155069535076527,
+            -8.304773776972749e-05,
+            0.010665400866776814,
+            0.013561082526851403,
+            0.019453190615040503,
+            0.008724224096424808,
+            0.017716608184445207,
+            0.0035170519664773133,
+            -0.013169473712817174,
+            0.009862672596286218,
+            0.011645070824075182,
+            0.011745045118639314,
+            0.011761235635339118,
+            -0.0051506969158396755,
+            0.00628898514628358,
+            -0.0049857719357619505,
+            0.008525048536699957,
+            0.027507594807546505,
+            0.0013209667681023782,
+            0.017047984110417937,
+            0.016368796085341072,
+            0.020105811276523177,
+            -0.003759327296302425,
+            3.4626946806238876e-05,
+            0.00035688794161846735,
+            -0.001063984476108484,
+            0.003291497239446139,
+            -0.018279629509070614,
+            -0.01135428882896519,
+            0.017033095140582537,
+            -5.492833742539044e-05,
+            0.024685530931127835,
+            -0.0010090516352729089,
+            -0.0184210790691242,
+            -0.006072072044463014,
+            0.0019766255786695347,
+            -0.0143465314042477,
+            -0.00209268265697904,
+            0.008867797876246725,
+            0.02090777280180668,
+            0.007785784409274212,
+            -0.014694559759226521,
+            0.03266241399157856,
+            0.017596572028431588,
+            0.02301544904486291,
+            -0.02019106568899837,
+            -0.008116927224141509,
+            -0.007678137378694171,
+            0.020318874678520133,
+            -0.0068566512882342135,
+            -0.002510061555870801,
+            0.021473513752723834,
+            0.006645783892214728,
+            0.011211922100954015,
+            0.012286872605920514,
+            0.0003084092164612384,
+            0.0028492628853667647,
+            -0.015161921733689642,
+            0.011855756525919763,
+            -0.005289433905822296,
+            -0.006904878573181685,
+            0.021517401941638887,
+            -0.010444313612915583,
+            -0.009104739247715266,
+            0.005434542886627656,
+            -0.011460046711120444,
+            0.013324525803587403,
+            -0.0016717915349416276,
+            -0.013225789210352699,
+            -0.014868291792978007,
+            -0.004059908595020105,
+            -0.017086039690741954,
+            0.018935544362274584,
+            0.006810606440007176,
+            -0.0016747502037076753,
+            -0.005949836930379681,
+            0.004577809518378731,
+            -0.002387527459083855,
+            -0.0006415824062693734,
+            -0.015923707971787795,
+            0.0035047065077239213,
+            0.009700583796235219,
+            -0.0031324165789907116,
+            0.010643108490671052,
+            -0.013353561795075988,
+            0.033779753889958034,
+            -0.003236875179603599,
+            0.0033068852921624326,
+            0.005106828066670305,
+            -0.003034255184307467,
+            0.001481657575335246,
+            0.013519816427440722,
+            0.009066434673988428,
+            -0.008339461460990311,
+            -0.0009211583209818409,
+            0.013497039042975171,
+            -0.018668722901146233,
+            0.0013796912320793106,
+            0.013796986913155082,
+            -0.015849997808276967,
+            0.0049427770053857125,
+            0.011400118227502903,
+            -0.00639345486098034,
+            0.015340608222895147,
+            -0.01371015934577743,
+            0.007008681477695071,
+            0.009775206231977338,
+            0.028490233702896408,
+            -0.01966685875736091,
+            0.01928758485650401,
+            -0.025635008258084145,
+            0.031283028009357144,
+            0.012122242939206308,
+            -0.005796910058436286,
+            -0.025521126197222552,
+            -0.011593941553769327,
+            0.01283377684428954,
+            -0.021686338476615465,
+            0.0014883363235708173,
+            0.004930128822274494,
+            -0.013589466747020829,
+            -0.014766714233107183,
+            -0.01202283835955618,
+            0.011738578705966321,
+            0.0006399892787922201,
+            0.01589633486439762,
+            -0.001565909739326631,
+            0.006195453613357957,
+            0.00799709051068248,
+            -0.007262779704915372,
+            -0.006717513726969218,
+            -0.01982600498555257,
+            0.01452043414139846,
+            0.004256754663990435,
+            -0.005262312998149605,
+            -0.0224549938750886,
+            0.009397743096285409,
+            0.017707219262338467,
+            -0.005753255460862127,
+            0.013047488144162397,
+            -0.03383526087792666,
+            0.0009452367103428395,
+            -0.005822570832057675,
+            -0.0027621026437538738,
+            -0.011741384193082396,
+            0.005215022845011562,
+            -0.015301048137828595,
+            0.002594097201808009,
+            -0.006446736651739076,
+            -0.016963075831388622,
+            0.002297125878220545,
+            0.02099207084363147,
+            -0.0032941933998622765,
+            -0.0028818889706579017,
+            0.01784874229400347,
+            0.008499720521713671,
+            -0.01436426626087195,
+            -0.004000179664908013,
+            0.006397941984552902,
+            -0.018873859156915206,
+            0.0011599963694687036,
+            0.02198209878751397,
+            0.012652203334020664,
+            -0.026378312815268385,
+            -0.0016397712646032201,
+            0.013833745513458322,
+            0.008230921549345205,
+            0.008667892596847433,
+            0.0040942632709300665,
+            0.014593014365768302,
+            -0.006350891941908016,
+            -0.001374093276890415,
+            -6.3005496707736715e-06,
+            -0.006943874751919358,
+            0.011912879044321251,
+            0.003688355066059739,
+            0.040741622477833674,
+            -0.010639328654064204,
+            0.011006744363538067,
+            -0.01080331270092793,
+            -0.004477468739500037,
+            -0.004503151926350927,
+            0.0027311865257027917,
+            -0.013139063275489816,
+            -0.0001322676781706513,
+            0.0021051078165230533,
+            -0.0019788024041678294,
+            0.00649268452600941,
+            0.003305903531478554,
+            0.013176851471483939,
+            -0.017229299828998786,
+            -0.0024884474570636707,
+            -0.009910839696360717,
+            0.019347666346683084,
+            -0.010724923526200773,
+            0.01051871403061451,
+            0.03256893707453101,
+            0.013242686936629418,
+            0.00022445440868234016,
+            -0.014947304198470308,
+            -0.008836459838385262,
+            -0.008992977994567497,
+            -0.01739161854615299,
+            -0.012908166671408042,
+            -0.022654409684101404,
+            -0.00294050967213434,
+            0.006390611639719366,
+            -0.0010555036392130776,
+            -0.002785796833161689,
+            -0.027340941664129582,
+            -0.00792516100590229,
+            -0.03210910565574123,
+            -0.013679170743667392,
+            0.0056845041078574104,
+            -0.021558178294402857,
+            0.005195998493021035,
+            0.029445968279023734,
+            -0.008256723194649932,
+            0.018403154314213917,
+            0.006923789005793937,
+            0.011433558571124386,
+            -0.0022324062216246424,
+            -0.011870704203994723,
+            -0.011035736680889242,
+            0.00032924465177187114,
+            0.003455564411617082,
+            0.004938023824535148,
+            0.00916834019939669,
+            -0.01732811783799426,
+            -0.010720420451770168,
+            0.00025951769018448803,
+            0.01732173296763328,
+            -0.0036441011455210317,
+            -0.014654182169779525,
+            -0.009890543675136605,
+            0.00934755871928037,
+            -0.017560953324850772,
+            0.0028803276180087,
+            0.008144645056336303,
+            -0.006844751836577803,
+            0.009686357271083834,
+            0.00910269542748879,
+            -0.003634975911996508,
+            -0.01125116892179341,
+            0.015123292946443665,
+            0.011417498955214564,
+            -0.0005683452620824292,
+            0.009547451569266946,
+            0.0031907584166949686,
+            -0.004914343407341038,
+            0.0004946517266559977,
+            -0.013981094295278457,
+            -0.008438177812073363,
+            -0.009698303451884585,
+            -0.0004873949117926506,
+            0.008310275049300703,
+            -0.011244097579407173,
+            0.002589333020166005,
+            0.021677389796728952,
+            0.0016179416889548232,
+            -0.011231602157838225,
+            -0.0045296258349024725,
+            -0.03198651985714177,
+            -0.005752566699226788,
+            -0.017524199510403467,
+            0.014980719037744585,
+            -0.002529640898744345,
+            0.01028755110586937,
+            -0.0021848874276835985,
+            0.015817733088597956,
+            -0.005450667684071798,
+            0.012212046004369696,
+            -0.0008714643856749736,
+            -0.004560548606687623,
+            0.006746171236132563,
+            0.0034423786329218276,
+            -0.0009995673054631,
+            0.05643134636651925,
+            0.010296478797092773,
+            0.014377556149215558,
+            0.013825497434998718,
+            -0.017485595187900386,
+            0.013330532940316836,
+            0.013333976202885402,
+            0.012417102160127097,
+            0.004536479576179834,
+            -0.017256228281240734,
+            0.024337236913894427,
+            0.012755805964855683,
+            0.009622694631579495,
+            -0.0018516129725815202,
+            -0.0015877067248695194,
+            0.019827793457522933,
+            0.018266833138588977,
+            -0.009808363592736615,
+            0.011083663940891296,
+            0.004458465015627857,
+            -0.01068421391840819,
+            -0.0050807436529387556,
+            0.024563333063926475,
+            -0.01325209293057542,
+            -0.016539133573613984,
+            0.0059168302415277515,
+            -0.02488439856049784,
+            -0.029755917745609557,
+            0.001920092403971772,
+            0.006364691645397147,
+            0.002128416061225683,
+            -0.0021827593501562285,
+            0.00636889353893684,
+            0.01505350329221805,
+            -0.02223481012881443,
+            -0.01913380824474348,
+            0.004876437698740333,
+            -0.0031323018872010133,
+            0.015711985535422395,
+            -0.02142499388661488,
+            -0.011646214273628826,
+            0.003244037052013272,
+            -0.006889665577990765,
+            0.004421548112756594,
+            0.004704830748153097,
+            -0.004955793799612646,
+            -0.005239972361031197,
+            0.0007828188358756139,
+            0.019667585310240637,
+            0.017115475760556154,
+            0.02008255444898991,
+            0.012206187649127842,
+            -0.004915394518125827,
+            -0.00997481157928988,
+            0.002664275191354035,
+            -0.017453428141032505,
+            -0.005114569347643741,
+            0.006659824028894164,
+            0.013233844584497872,
+            -0.023794827807567007,
+            -0.007354308951883939,
+            0.012842257789962674,
+            0.011294252132532885,
+            -0.00824836528452984,
+            0.004256336058369505,
+            -0.011193345143979632,
+            0.009248664100690892,
+            0.010363170772580968,
+            0.010981447978991254,
+            -0.01250390246078834,
+            0.008163162620777558,
+            0.002528845973804195,
+            0.007003579605495149,
+            -0.005812703866552746,
+            0.008764871082692903,
+            0.013124123077340675,
+            -0.012212447674686537,
+            -0.0013940507123620205,
+            -0.001544873017903471,
+            -0.020493969663982807,
+            0.005782189204516662,
+            0.007033733269347455,
+            0.012878926833341835,
+            -0.008338295354334415,
+            0.015844568159504493,
+            -0.013423240193733334,
+            -0.00468920516522183,
+            0.0063319500880227525,
+            -0.004924778574104498,
+            -0.012053197667708071,
+            0.0017742020845283848,
+            -0.0035640186446405536,
+            -0.008666749197879684,
+            -0.0013894177719374007,
+            0.0016400693271668141,
+            0.004501038858023909,
+            0.02043221461176279,
+            0.001901506572332553,
+            -0.0056076433740056926,
+            0.019660750419391273,
+            0.003538773720733224,
+            -0.0018659146421562808,
+            0.005354994676719758,
+            0.0006789623595561136,
+            -0.0021516503031452773,
+            -0.0027566542393029298,
+            -0.010552067426241374,
+            0.01699044655141112,
+            -0.01883386136434379,
+            -0.006428123254736374,
+            -0.012514829515253224,
+            0.006923832110523631,
+            0.023065674483579466,
+            -0.0031719793455512423,
+            -0.006131653355136594,
+            -0.015640226612880566,
+            0.003589774262421098,
+            -0.07234225978543331,
+            -0.010636515019878534,
+            0.005436135654579842,
+            0.030362090167699224,
+            -0.008735845879763853,
+            0.0012226978854869956,
+            -0.00802187367908004,
+            0.010460901462092434,
+            -0.0024797678489701584,
+            0.005049311815982583,
+            0.0037643360764659196,
+            -0.003379308997651945,
+            -0.0029803517660911536,
+            0.0047612984744651015,
+            -0.010726918354479238,
+            -0.018870259480662628,
+            0.0028549213021684043,
+            0.025034678854420084,
+            0.009004274035878184,
+            0.004138885468102832,
+            0.009765532829693857,
+            -0.008537379113922283,
+            -0.0014902115840393893,
+            0.008462743183234337,
+            0.007328272099011593,
+            -0.009076904245691212,
+            -0.0008002047467975742,
+            0.008347212876126893,
+            -0.021799374855548816,
+            0.0020253943807330738,
+            0.012246389250647798,
+            -0.008912465890948629,
+            -0.006244789880293639,
+            0.002413038795382646,
+            0.004202974630810528,
+            -0.004339960146144246,
+            -0.009919301413657872,
+            -0.023862183602380987,
+            0.006400371798756263,
+            -0.007410957922309058,
+            -0.0001606112434129549,
+            0.0031662502228666104,
+            0.00012871401736865864,
+            -0.01687187870925491,
+            0.01374600947628664,
+            -0.008834671549515985,
+            -0.0033344289547729495,
+            0.012937173759658114,
+            -0.003267867578821276,
+            0.01864135643890992,
+            -0.014050228437162755,
+            0.007864998326137568,
+            -0.018988002335416778,
+            -0.003944450943320823,
+            -0.0032084001089808037,
+            -0.0023149926514803745,
+            -0.007750311931438483,
+            -0.045019596490520515,
+            0.013573650825957857,
+            -0.004374086208559833,
+            -0.005869005771503507,
+            -0.01250037134701792,
+            -0.016625210662158708,
+            0.011639132814162461,
+            0.004045997475412364,
+            -0.0014885281342871186,
+            -0.020321856013396592,
+            0.006222318025479057,
+            0.011121009394355636,
+            0.00904225037228108,
+            -0.011585148988670874,
+            -0.0018474735072673144,
+            -0.011325387916132704,
+            -0.0025890917877355725,
+            -0.009148717095934154,
+            0.018495348432506,
+            0.002404017871470538,
+            -0.010232069372010613,
+            0.022941087706292104,
+            -0.0024570420644715457,
+            0.020988341201251853,
+            -0.00442363926883869,
+            -0.004485898239018673,
+            -0.020447941631096866,
+            -0.004158168168969816,
+            -0.019358617104349624,
+            0.00036464612100595666,
+            0.0017749921060331022,
+            0.02078748568527368,
+            0.007256675717496499,
+            0.012612722550842472,
+            0.006032799184116686,
+            0.00631983142330159,
+            0.030151438614535002,
+            -0.010852408231267204,
+            0.0022451749744724233,
+            -0.017863828302314567,
+            0.010653806941450981,
+            0.028289026813870025,
+            0.009882042879445822,
+            -0.006920783006708277,
+            0.014053717954960316,
+            0.004932716665921658,
+            -0.00616100713292227,
+            0.025975267677767474,
+            -0.038836632659249616,
+            -0.021059874446788137,
+            -0.0061122622002863615,
+            0.001439949038430725,
+            -3.496872289060673e-05,
+            0.005030571386336924,
+            -0.011007922189367617,
+            0.009222155733548683,
+            0.0026655561130744807,
+            0.03101907459085608,
+            -0.002317540524509921,
+            -0.007348120500263025,
+            0.003987101796623003,
+            0.001511347369284844,
+            0.021229069065914295,
+            -0.01197457000398312,
+            -0.038429200853184124,
+            -0.0031917847963042626,
+            -0.0039488592790253305,
+            0.008371872140716302,
+            -0.012355524487988753,
+            -0.010839396359961,
+            -0.004574476927098761,
+            -0.01882180934875166,
+            -0.010292867074240718,
+            0.025595330740853118,
+            -0.007162452868646569,
+            0.01788200594893745,
+            -0.0076384034977797525,
+            -0.008112046063674713,
+            0.00683079906571602,
+            0.011572398977364099,
+            -0.01932794114157753,
+            0.0034055065356524052,
+            -0.0027824032743918262,
+            0.01627342501880651,
+            0.011718826886349412,
+            -0.013342112644751174,
+            0.006234666245077415,
+            0.01365078200475753,
+            -0.015476695744659024,
+            -0.0015378873781631448,
+            -0.0069121070205125585,
+            0.014866073481143516,
+            -0.003093423905641203,
+            -0.0033762696975023986,
+            0.011958295888528918,
+            -0.021982979614167224,
+            0.01697002117600754,
+            -0.02183323542526832,
+            0.022295922951583082,
+            -0.0006195453455557405,
+            -0.0006320640234089136,
+            0.008423668060539851,
+            0.008314680408345411,
+            -0.013378756052181496,
+            0.0016194432891316378,
+            -0.03416272433582183,
+            -0.009962172336151867,
+            -0.015051382650973002,
+            0.0038009542554177348,
+            -0.010106196779819915,
+            -0.01203081704994182,
+            -0.021869748479536547,
+            -0.013231917312580397,
+            0.009488867268928662,
+            0.02421155136770049,
+            -0.004349181861466944,
+            0.015665735941744613,
+            -0.008077879618601087,
+            -0.00346140349979159,
+            -0.019022501109520248,
+            -0.020158795455478176,
+            -0.011854460388465065,
+            -0.01570428154296937,
+            -0.007073290993912972,
+            0.004794071576762192,
+            -0.014492538032767992,
+            0.013391092874532436,
+            0.03615243857749714,
+            -0.017029836110560772,
+            0.008609740828901341,
+            0.019179133878629843,
+            0.0009923228421593012,
+            0.015393048602064972,
+            0.003790868078763006,
+            0.011710499244312703,
+            -0.014885526287290077,
+            -0.011829126494406653,
+            -0.0005627435441977781,
+            0.0020582596099707785,
+            0.003728052108395647,
+            0.027914733124690644,
+            0.0038604865752177283,
+            0.005302574635039923,
+            0.012293011832771443,
+            0.013320598642264739,
+            -0.010706185765391238,
+            0.015853648683536427,
+            0.004090821159159474,
+            -0.016980287717038522,
+            -0.0031197192883179663,
+            -0.006894138060845641,
+            -0.012739354473335233,
+            -0.002311031608856898,
+            0.0159372988694303,
+            -0.008091532959711314,
+            0.015169701976466646,
+            -0.004316674756425857,
+            -0.015090694840235596,
+            0.0007287957651103694,
+            -0.03810971752015664,
+            -0.0018729979445690102,
+            -0.01301867399306695,
+            -0.023136174288569147,
+            0.04241471786668668,
+            0.00415464531754789,
+            0.020836251111427635,
+            -0.006703690210920993,
+            0.006147438331654669,
+            -0.020673361951422737,
+            0.011783676068789637,
+            -0.0037516561524537765,
+            0.009702463136945385,
+            0.009366943837185086,
+            -0.018879598518030567,
+            -0.01132474738893142,
+            0.018484010977000243,
+            0.0008964474295952695,
+            0.001899853004269779,
+            0.014452586238316783,
+            0.019746940887134506,
+            0.01136527620507647,
+            -0.0020960934961233306,
+            0.02093114800466563,
+            0.018023169361765227,
+            -0.0002475129435970239,
+            -0.007118840223520255,
+            0.013527604253725194,
+            -0.02531530780276405,
+            0.014601163833111726,
+            0.03268673940622878,
+            0.013556815596615061,
+            0.020396267313433348,
+            -0.016161264086739215,
+            0.016259608464481445,
+            -0.006718110409512467,
+            0.017378727943854342,
+            0.001211924791251576,
+            0.01911265175070918,
+            0.014022657097776944,
+            0.01308266544853342,
+            0.006784756253744209,
+            -0.013367188766230505,
+            0.024435211011756285,
+            -0.008342072323415817,
+            -0.0004209982405190692,
+            -0.002823543536521797,
+            -0.023279946325307846,
+            0.01506543259962226,
+            0.0017095020186682082,
+            0.003411744637061027,
+            -0.0035317155649076414,
+            -0.001957702240702538,
+            0.008037874596341196,
+            0.008176809711854463,
+            -0.015780325305129808,
+            0.0031090160698441006,
+            -0.01924979849975192,
+            -0.008566500568783712,
+            -0.024111269290772643,
+            0.003704817310374187,
+            -0.015182531891766549,
+            -0.009971369164255839,
+            -0.019503345629597463,
+            -0.011275291123690537,
+            -0.010129027262987365,
+            0.00026549411615768147,
+            -0.0077553590287942975,
+            0.014297350661562049,
+            -0.017119422159171904,
+            -0.021341131823214652,
+            0.007945720599865192,
+            0.01439604029848557,
+            0.0037256533134793816,
+            0.005472351446140726,
+            0.008325407758477418,
+            0.009082034215555286,
+            -0.010830069811539129,
+            -0.010352768708989164,
+            0.001902868536460922,
+            0.0010307757364948558,
+            0.011572880679973516,
+            0.01480960939553474,
+            0.011328449881374384,
+            -0.013429607187664061,
+            0.0019326261581148884,
+            -0.014862631307250641,
+            0.004686987038977201,
+            -0.0011432874921856958,
+            -0.007026108142521568,
+            -0.0023122147402109927,
+            -0.01067305755615663,
+            0.008329587642130216,
+            0.0051774766682198365,
+            0.0014058609424645312,
+            0.0175095191841222,
+            -0.0016739751490535205,
+            -0.01295313808457633,
+            -0.012767766732175707,
+            -0.011653259281510757,
+            -0.015653450921388538,
+            0.006739771869002279,
+            -0.006908534840842188,
+            -0.006309735406143942,
+            0.00969748644524724,
+            -0.004300978285892996,
+            -0.0071082865141610775,
+            0.009870599453242536,
+            0.0003007194077865931,
+            -0.0011846789159672513,
+            0.03344372215649544,
+            -0.004482311849310301,
+            0.004153681423879411,
+            0.00022516282287549753,
+            -0.00096173441700326,
+            -0.01608715959652527,
+            -0.026963713856310283,
+            -0.0014323662880336336,
+            -0.02273696636049197,
+            0.017938234567205,
+            -0.0058065575851078514,
+            -0.001006759040376689,
+            0.002493842516193206,
+            -0.0042740143630253385,
+            -6.58295494211919e-05,
+            -0.011768915967276026,
+            0.0022431022211178383,
+            -0.029090994997079067,
+            0.009898341521414653,
+            -0.009544941198003441,
+            -0.02154260971685405,
+            0.01133866603878075,
+            0.040150135462722274,
+            -0.0034082812055093122,
+            0.0006477463440760081,
+            0.0010662318198694314,
+            0.010346430535548179,
+            0.014953186668393513,
+            -0.0019239187336010772,
+            -0.01603518108536549,
+            -0.010550241019652155,
+            0.007810013487665037,
+            0.0003366114893320438,
+            0.0030211673366186986,
+            0.025036691996012227,
+            -0.019097101590128545,
+            -0.016027314610013832,
+            -0.011102216330632247,
+            -0.011450582072890358,
+            0.009254748792585386,
+            0.019696746434919966,
+            0.013185751991632784,
+            -0.020709092762886617,
+            -0.017120964154502993,
+            0.00025737692261907426,
+            -0.02662183259737709,
+            -0.0015175233130328546,
+            0.12569047385271798,
+            -0.004250752280902458,
+            -0.009469510190778525,
+            -0.016304049953896647,
+            -0.00863767582150345,
+            0.02183050591695681,
+            -0.008697910752438377,
+            0.06936580824413162,
+            0.0006861269173917291,
+            -0.0029257096931296273,
+            -0.0020896063525880225,
+            -0.002993306834763656,
+            -0.013871927123029641,
+            0.001864949832879178,
+            -0.016847962023455045,
+            -0.01637518023758693,
+            -0.006332388047537449,
+            -0.013325712872379694,
+            -0.0017155737214029734,
+            -0.0061798136617018905,
+            0.004759205228314821,
+            -0.007830001529147273,
+            0.011495001353183024,
+            -0.026107152853762815,
+            -0.010130127689151806,
+            0.005962957535201191,
+            -0.011637257947784772,
+            0.0011307756612654312,
+            -0.030816554116329904,
+            -0.00441741097001083,
+            7.409036337997905e-05,
+            0.016501246370776126,
+            0.0015748302561177006,
+            -0.00940138978060357,
+            -0.011216292373500447,
+            0.006508185064247167,
+            0.021984022174755303,
+            0.02029850289834145,
+            0.020629050747183544,
+            0.009741513502990842,
+            0.017418812192334685,
+            -0.007031919478404256,
+            0.02625468226435525,
+            0.006196721544177288,
+            0.002206946290342702,
+            0.005596780938975919,
+            -0.01211153541496144,
+            -0.008534755574472074,
+            0.0074101096913398124,
+            -0.0219026467376419,
+            -0.0238583746438936,
+            -0.012260621283748488,
+            0.025739429552984337,
+            -0.009054977254128021,
+            -0.0019759724570709175,
+            -0.0030737356218219617,
+            0.024727831360321823,
+            -0.005149317224364814,
+            0.002332136352952536,
+            0.015587381251136284,
+            -0.028035348333895063,
+            0.008643405636450286,
+            0.007622365990511162,
+            0.02564089666732,
+            -0.0032690983288409517,
+            0.0030897564314706603,
+            -0.009711753474608342,
+            -0.009285462125807194,
+            -0.009613024258506932,
+            0.004240295765592783,
+            0.018961596475867332,
+            0.005797211858953558,
+            0.009922313409983964,
+            -0.006335322307636977,
+            -0.015248479856831972,
+            0.018517674622994848,
+            -0.018526204881631703,
+            -0.017805480495530986,
+            -0.0031014632675223346,
+            -0.023499333544416315,
+            0.007508363381003107,
+            0.006594465447638183,
+            -0.013304818586245387,
+            -0.0232989529893049,
+            0.020407788403667446,
+            0.003426820233878876,
+            -0.006520960224510217,
+            0.0028020004149410366,
+            -0.03282622678411901,
+            -0.018099884128291804,
+            0.004341732255376245,
+            0.012442902960644058,
+            -0.006914419410521592,
+            0.01684289728480369,
+            -0.0213314411022478,
+            0.021170682961259797,
+            -0.018444729425471068,
+            0.015400297173773751,
+            -0.005856807329212788,
+            0.01661347113847185,
+            0.011889835104373263,
+            0.0013585650928227913,
+            -0.019926344687386053,
+            0.02233564528528766,
+            0.0006518460244140545,
+            0.029977976063273147,
+            0.026669882018303386,
+            -0.005623183408671759,
+            -0.004497323576380724,
+            -0.017964594171161242,
+            0.011149712487916564,
+            0.0034380718823694635,
+            0.00295265329839474,
+            0.023835186931253683,
+            0.02803963164846445,
+            -0.028097235508129332,
+            0.017703040589712025,
+            -0.012893023698870347,
+            0.0089348477172039,
+            -0.011630331709306501,
+            0.0096768368423985,
+            0.00669273666692988,
+            0.003825890215203911,
+            0.008289398654583944,
+            0.011807275797649763,
+            0.024893949220958504,
+            0.02893733381281628,
+            0.004069052493622539,
+            0.012465630768347759,
+            -0.003600044227856378,
+            0.004954707230784128,
+            0.005985890100113585,
+            0.0014569711209781388,
+            -0.00648453455551435,
+            0.0031945554565833674,
+            -0.006090168119075504,
+            -0.003744222112779555,
+            0.015266602618825367,
+            -0.00028966361595811053,
+            0.015634624351841753,
+            0.012395295870300452,
+            -0.006577117978669073,
+            -0.003069372773552921,
+            -0.0029369830414480856,
+            -0.007598245840968154,
+            -0.017753359726409172,
+            0.025057809370078198,
+            -0.01606095813358513,
+            -0.019631204097884002,
+            0.0008782653868466668,
+            0.0031106969086903075,
+            -0.013675045629189149,
+            -0.008463196400259825,
+            0.008777824363912681,
+            0.014471929439779247,
+            -0.018478175346087953,
+            0.0017862044141924377,
+            -0.0004910793014860654,
+            0.013862986726603731,
+            -0.003641460118879092,
+            -0.00902817956483427,
+            -0.002076665354355939,
+            0.010393957588184604,
+            0.016413601414994997,
+            0.0038017609859922327,
+            -0.008644753777502864,
+            -0.0005484639867925445,
+            0.006452371249117759,
+            0.007391332924183688,
+            0.0036307379469783137,
+            -4.895894824382588e-05,
+            -0.008129811261078888,
+            -0.013655410823646597,
+            0.018829640339982075,
+            -0.007579460691893014,
+            0.016191139169828096,
+            -0.004984562798832502,
+            -0.011846989328030879,
+            0.004110855640416496,
+            0.0004775146965349116,
+            -0.014308885325290177,
+            0.002778951512640393,
+            -0.003583348080466357,
+            0.0017351729844533707,
+            0.006444402813802743,
+            -0.004945263366121632,
+            0.003803172980003619,
+            -0.008178062077842394,
+            0.02700102856098108,
+            0.015477045085093356,
+            0.005818362056942523,
+            -0.00876269298951217,
+            -0.024697543974427114,
+            0.008183329610246195,
+            0.00518473714350693,
+            0.018661120751406378,
+            0.014464207183253898,
+            0.0019868192486975638,
+            0.009827229842256728,
+            -0.013395371817208287,
+            -0.0022376892192351944,
+            0.009711801873474911,
+            0.017963581533215926,
+            -0.0059700736266576305,
+            0.019169874693002874,
+            -0.010316234316657931,
+            -0.0008125759237295765,
+            0.004003428870809722,
+            0.017109043212624506,
+            -0.004590883969173384,
+            0.013958072401225709
+          ],
+          "title": "F Pseudo Code for Switch Transformers 33",
+          "keyphrases": [
+            "Switch Transformers",
+            "Mixture-of-Experts",
+            "sparsely-activated expert model",
+            "natural language tasks",
+            "pre-training and fine-tuning"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "6e0e0b15-33e1-4483-b7c8-b4337867fe0d",
+      "type": "child",
+      "source": {
+        "id": "14339b1f-53f8-479b-9da7-9430b174b9ee",
+        "properties": {
+          "page_content": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer\n\nNoam Shazeer1, Azalia Mirhoseini∗†1, Krzysztof Maziarz∗2, Andy Davis1, Quoc Le1, Geoffrey Hinton1and Jeff Dean1 1Google Brain, {noam,azalia,andydavis,qvl,geoffhinton,jeff}@google.com 2Jagiellonian University, Cracow, krzysztof.maziarz@student.uj.edu.pl\n\nAbstract\n\nThe capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost.\n\n1 Introduction And Related Work 1.1 Conditional Computation\n\nExploiting scale in both training data and model size has been central to the success of deep learning. When datasets are sufficiently large, increasing the capacity (number of parameters) of neural networks can give much better prediction accuracy. This has been shown in domains such as text (Sutskever et al., 2014; Bahdanau et al., 2014; Jozefowicz et al., 2016; Wu et al., 2016), images (Krizhevsky et al., 2012; Le et al., 2012), and audio (Hinton et al., 2012; Amodei et al., 2015). For typical deep learning models, where the entire model is activated for every example, this leads to a roughly quadratic blow-up in training costs, as both the model size and the number of training examples increase. Unfortunately, the advances in computing power and distributed computation fall short of meeting such demand.\n\nVarious forms of conditional computation have been proposed as a way to increase model capacity without a proportional increase in computational costs (Davis & Arel, 2013; Bengio et al., 2013; Eigen et al., 2013; Ludovic Denoyer, 2014; Cho & Bengio, 2014; Bengio et al., 2015; Almahairi et al., 2015). In these schemes, large parts of a network are active or inactive on a per-example basis. The gating decisions may be binary or sparse and continuous, stochastic or deterministic. Various forms of reinforcement learning and back-propagation are proposed for trarining the gating decisions.\n\nWhile these ideas are promising in theory, no work to date has yet demonstrated massive improvements in model capacity, training time, or model quality. We blame this on a combination of the following challenges:\n\nModern computing devices, especially GPUs, are much faster at arithmetic than at branching. Most of the works above recognize this and propose turning on/off large chunks of the network with each gating decision.\n\nLarge batch sizes are critical for performance, as they amortize the costs of parameter transfers and updates. Conditional computation reduces the batch sizes for the conditionally active chunks of the network.\n\nNetwork bandwidth can be a bottleneck. A cluster of GPUs may have computational power thousands of times greater than the aggregate inter-device network bandwidth. To be computationally efficient, the relative computational versus network demands of an algorithm must exceed this ratio. Embedding layers, which can be seen as a form of conditional computation, are handicapped by this very problem. Since the embeddings generally need to be sent across the network, the number of (example, parameter) interactions is limited by network bandwidth instead of computational capacity.\n\nDepending on the scheme, loss terms may be necessary to achieve the desired level of sparsity per-chunk and/or per example. Bengio et al. (2015) use three such terms. These issues can affect both model quality and load-balancing.\n\nModel capacity is most critical for very large data sets. The existing literature on conditional computation deals with relatively small image recognition data sets consisting of up to 600,000 images. It is hard to imagine that the labels of these images provide a sufficient signal to adequately train a model with millions, let alone billions of parameters. In this work, we for the first time address all of the above challenges and finally realize the promise of conditional computation. We obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets.\n\n1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1). All parts of the network are trained jointly by back-propagation.\n\nWhile the introduced technique is generic, in this paper we focus on language modeling and machine translation tasks, which are known to benefit from very large models. In particular, we apply a MoE convolutionally between stacked LSTM layers (Hochreiter & Schmidhuber, 1997), as in Figure 1.\n\nThe MoE is called once for each position in the text, selecting a potentially different combination of experts at each position. The different experts tend to become highly specialized based on syntax and semantics (see Appendix E Table 9). On both language modeling and machine translation benchmarks, we improve on best published results at a fraction of the computational cost.\n\n1.3 Related Work On Mixtures Of Experts\n\nSince its introduction more than two decades ago (Jacobs et al., 1991; Jordan & Jacobs, 1994), the mixture-of-experts approach has been the subject of much research. Different types of expert architectures hae been proposed such as SVMs (Collobert et al., 2002), Gaussian Processes (Tresp, 2001; Theis & Bethge, 2015; Deisenroth & Ng, 2015), Dirichlet Processes (Shahbaba & Neal, 2009), and deep networks. Other work has focused on different expert configurations such as a hierarchical structure (Yao et al., 2009), infinite numbers of experts (Rasmussen & Ghahramani, 2002), and adding experts sequentially (Aljundi et al., 2016). Garmash & Monz (2016) suggest an ensemble model in the format of mixture of experts for machine translation. The gating network is trained on a pre-trained ensemble NMT model. The works above concern top-level mixtures of experts. The mixture of experts is the whole model. Eigen et al. (2013) introduce the idea of using multiple MoEs with their own gating networks as parts of a deep model. It is intuitive that the latter approach is more powerful, since complex problems may contain many sub-problems each requiring different experts. They also allude in their conclusion to the potential to introduce sparsity, turning MoEs into a vehicle for computational computation.\n\nOur work builds on this use of MoEs as a general purpose neural network component. While Eigen et al. (2013) uses two stacked MoEs allowing for two sets of gating decisions, our convolutional application of the MoE allows for different gating decisions at each position in the text. We also realize sparse gating and demonstrate its use as a practical way to massively increase model capacity.\n\n2 The Structure Of The Mixture-Of-Experts Layer\n\nThe Mixture-of-Experts (MoE) layer consists of a set of n \"expert networks\" E1, · · · , En, and a \"gating network\" G whose output is a sparse n-dimensional vector. Figure 1 shows an overview of the MoE module. The experts are themselves neural networks, each with their own parameters.\n\nAlthough in principle we only require that the experts accept the same sized inputs and produce the same-sized outputs, in our initial investigations in this paper, we restrict ourselves to the case where the models are feed-forward networks with identical architectures, but with separate parameters.\n\nLet us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows:\n\n$$y=\\sum_{i=1}^{n}G(x){i}E{i}(x)$$ $$(\\mathbf{l})$$ G(x)iEi(x) (1) We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example. If the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network. In the following we focus on ordinary MoEs. We provide more details on hierarchical MoEs in Appendix B. Our implementation is related to other models of conditional computation. A MoE whose experts are simple weight matrices is similar to the parameterized weight matrix proposed in (Cho & Bengio, 2014). A MoE whose experts have one hidden layer is similar to the block-wise dropout described in (Bengio et al., 2015), where the dropped-out layer is sandwiched between fully-activated layers.\n\n2.1 Gating Network\n\nSoftmax Gating: A simple choice of non-sparse gating function (Jordan & Jacobs, 1994) is to multiply the input by a trainable weight matrix Wg and then apply the Sof tmax function.\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ $\\eqref{eq:walpha}$. Gσ(x) = Sof tmax(x · Wg) (2) Noisy Top-K Gating: We add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to −∞ (which causes the corresponding gate values to equal 0). The sparsity serves to save computation, as described above. While this form of sparsity creates some theoretically scary discontinuities in the output of gating function, we have not yet observed this to be a problem in practice. The noise term helps with load balancing, as will be discussed in Appendix A. The amount of noise per component is controlled by a second trainable weight matrix Wnoise.\n\n$$G(x)=S o f t m a x(K e e p T o p K(H(x),k))$$ $$(4)$$ $$(S)$$ G(x) = Sof tmax(KeepT opK(H(x), k)) (3) $$H(x){i}=(x\\cdot W{g}){i}+S t a n d a r d N o r m a l()\\cdot S o f t p l u s((x\\cdot W{n o i s e}){i})$$ $KeepTopK(v,k){i}=\\begin{cases}v_{i}&\\text{if}v_{i}\\text{is in the top}k\\text{elements of}v.\\ -\\infty&\\text{otherwise.}\\end{cases}$ Training the Gating Network We train the gating network by simple back-propagation, along with the rest of the model. If we choose k > 1, the gate values for the top k experts have nonzero derivatives with respect to the weights of the gating network. This type of occasionally-sensitive behavior is described in (Bengio et al., 2013) with respect to noisy rectifiers. Gradients also backpropagate through the gating network to its inputs. Our method differs here from (Bengio et al., 2015) who use boolean gates and a REINFORCE-style approach to train the gating network.\n\n3 Addressing Performance Challenges 3.1 The Shrinking Batch Problem\n\nOn modern CPUs and GPUs, large batch sizes are necessary for computational efficiency, so as to amortize the overhead of parameter loads and updates. If the gating network chooses k out of n experts for each example, then for a batch of b examples, each expert receives a much smaller batch of approximately kb n b examples. This causes a naive MoE implementation to become very inefficient as the number of experts increases. The solution to this shrinking batch problem is to make the original batch size as large as possible. However, batch size tends to be limited by the memory necessary to store activations between the forwards and backwards passes. We propose the following techniques for increasing the batch size: Mixing Data Parallelism and Model Parallelism: In a conventional distributed training setting, multiple copies of the model on different devices asynchronously process distinct batches of data, and parameters are synchronized through a set of parameter servers. In our technique, these different batches run synchronously so that they can be combined for the MoE layer. We distribute the standard layers of the model and the gating network according to conventional data-parallel schemes, but keep only one shared copy of each expert. Each expert in the MoE layer receives a combined batch consisting of the relevant examples from all of the data-parallel input batches. The same set of devices function as data-parallel replicas (for the standard layers and the gating networks) and as model-parallel shards (each hosting a subset of the experts). If the model is distributed over d devices, and each device processes a batch of size b, each expert receives a batch of approximately kbd nexamples. Thus, we achieve a factor of d improvement in expert batch size.\n\nIn the case of a hierarchical MoE (Section B), the primary gating network employs data parallelism, and the secondary MoEs employ model parallelism. Each secondary MoE resides on one device.\n\nThis technique allows us to increase the number of experts (and hence the number of parameters) by proportionally increasing the number of devices in the training cluster. The total batch size increases, keeping the batch size per expert constant. The memory and bandwidth requirements per device also remain constant, as do the step times, as does the amount of time necessary to process a number of training examples equal to the number of parameters in the model. It is our goal to train a trillionparameter model on a trillion-word corpus. We have not scaled our systems this far as of the writing of this paper, but it should be possible by adding more hardware.\n\nTaking Advantage of Convolutionality: In our language models, we apply the same MoE to each time step of the previous layer. If we wait for the previous layer to finish, we can apply the MoE to all the time steps together as one big batch. Doing so increases the size of the input batch to the MoE layer by a factor of the number of unrolled time steps.\n\nIncreasing Batch Size for a Recurrent MoE: We suspect that even more powerful models may involve applying a MoE recurrently. For example, the weight matrices of a LSTM or other RNN could be replaced by a MoE. Sadly, such models break the convolutional trick from the last paragraph, since the input to the MoE at one timestep depends on the output of the MoE at the previous timestep. Gruslys et al. (2016) describe a technique for drastically reducing the number of stored activations in an unrolled RNN, at the cost of recomputing forward activations. This would allow for a large increase in batch size.\n\n3.2 Network Bandwidth\n\nAnother major performance concern in distributed computing is network bandwidth. Since the experts are stationary (see above) and the number of gating parameters is small, most of the communication involves sending the inputs and outputs of the experts across the network. To maintain computational efficiency, the ratio of an expert's computation to the size of its input and output must exceed the ratio of computational to network capacity of the computing device. For GPUs, this may be thousands to one. In our experiments, we use experts with one hidden layer containing thousands of RELU-activated units. Since the weight matrices in the expert have sizes input_size×hidden_size and hidden_size × output_size, the ratio of computation to input and output is equal to the size of the hidden layer. Conveniently, we can increase computational efficiency simply by using a larger hidden layer, or more hidden layers.\n\n4 Balancing Expert Utilization\n\nWe have observed that the gating network tends to converge to a state where it always produces large weights for the same few experts. This imbalance is self-reinforcing, as the favored experts are trained more rapidly and thus are selected even more by the gating network. Eigen et al. (2013) describe the same phenomenon, and use a hard constraint at the beginning of training to avoid this local minimum. Bengio et al. (2015) include a soft constraint on the batch-wise average of each gate.1 We take a soft constraint approach. We define the importance of an expert relative to a batch of training examples to be the batchwise sum of the gate values for that expert. We define an additional loss Limportance, which is added to the overall loss function for the model. This loss is equal to the square of the coefficient of variation of the set of importance values, multiplied by a hand-tuned scaling factor wimportance. This additional loss encourages all experts to have equal importance.\n\n$$I m p o r t a n c e(X)=\\sum_{x\\in X}G(x)$$ $$L_{importance}(X)=w_{importance}\\cdot CV(Importance(X))^{2}\\tag{7}$$ $$(6)$$\n\nWhile this loss function can ensure equal importance, experts may still receive very different numbers of examples. For example, one expert may receive a few examples with large weights, and another may receive many examples with small weights. This can cause memory and performance problems on distributed hardware. To solve this problem, we introduce a second loss function, Lload , which ensures balanced loads. Appendix A contains the definition of this function, along with experimental results.\n\n5 Experiments 5.1 1 Billion Word Language Modeling Benchmark\n\nDataset: This dataset, introduced by (Chelba et al., 2013) consists of shuffled unique sentences from news articles, totaling approximately 829 million words, with a vocabulary of 793,471 words.\n\nPrevious State-of-the-Art: The best previously published results (Jozefowicz et al., 2016) use models consisting of one or more stacked Long Short-Term Memory (LSTM) layers (Hochreiter & Schmidhuber, 1997; Gers et al., 2000). The number of parameters in the LSTM layers of these models vary from 2 million to 151 million. Quality increases greatly with parameter count, as do computational costs. Results for these models form the top line of Figure 2-right. MoE Models: Our models consist of two stacked LSTM layers with a MoE layer between them (see Figure 1). We vary the sizes of the layers and the number of experts. For full details on model architecture, training regimen, additional baselines and results, see Appendix C.\n\nLow Computation, Varied Capacity: To investigate the effects of adding capacity, we trained\n\na series of MoE models all with roughly equal computational costs: about 8 million multiply-andadds per training example per timestep in the forwards pass, excluding the softmax layer. We call this metric (ops/timestep). We trained models with flat MoEs containing 4, 32, and 256 experts, and models with hierarchical MoEs containing 256, 1024, and 4096 experts. Each expert had about 1 million parameters. For all the MoE layers, 4 experts were active per input.\n\nThe results of these models are shown in Figure 2-left. The model with 4 always-active experts performed (unsurprisingly) similarly to the computationally-matched baseline models, while the largest of the models (4096 experts) achieved an impressive 24% lower perplexity on the test set. Figure 2: Model comparison on 1-Billion-Word Language-Modeling Benchmark. On the left, we plot test perplexity as a function of model capacity for models with similar computational budgets of approximately 8-million-ops-per-timestep. On the right, we plot test perplexity as a function of computational budget. The top line represents the LSTM models from (Jozefowicz et al., 2016).\n\nThe bottom line represents 4-billion parameter MoE models with different computational budgets.\n\nVaried Computation, High Capacity: In addition to the largest model from the previous section, we trained two more MoE models with similarly high capacity (4 billion parameters), but higher computation budgets. These models had larger LSTMs, and fewer but larger and experts. Details\n\nTest Test #Parameters ops/timestep Training TFLOPS Perplexity Perplexity excluding embedding Time /GPU 10 epochs 100 epochs and softmax layers 10 epochs Best Published Results 34.7 30.6 151 million 151 million 59 hours, 32 k40s 1.09 Low-Budget MoE Model 34.1 4303 million 8.9 million 15 hours, 16 k40s 0.74 Medium-Budget MoE Model 31.3 4313 million 33.8 million 17 hours, 32 k40s 1.22 High-Budget MoE Model 28.0 4371 million 142.7 million 47 hours, 32 k40s 1.56\n\nTable 1: Summary of high-capacity MoE-augmented models with varying computational budgets, vs. best previously published results (Jozefowicz et al., 2016). Details in Appendix C. can be found in Appendix C.2. Results of these three models form the bottom line of Figure 2-right.\n\nTable 1 compares the results of these models to the best previously-published result on this dataset .\n\nEven the fastest of these models beats the best published result (when controlling for the number of training epochs), despite requiring only 6% of the computation.\n\nComputational Efficiency: We trained our models using TensorFlow (Abadi et al., 2016) on clusters containing 16-32 Tesla K40 GPUs. For each of our models, we determine computational efficiency in TFLOPS/GPU by dividing the number of floating point operations required to process one training batch by the observed step time and the number of GPUs in the cluster. The operation counts used here are higher than the ones we report in our ops/timestep numbers in that we include the backwards pass, we include the importance-sampling-based training of the softmax layer, and we count a multiply-and-add as two separate operations. For all of our MoE models, the floating point operations involved in the experts represent between 37% and 46% of the total.\n\nFor our baseline models wtih no MoE, observed computational efficiency ranged from 1.07-1.29 TFLOPS/GPU. For our low-computation MoE models, computation efficiency ranged from 0.740.90 TFLOPS/GPU, except for the 4-expert model which did not make full use of the available parallelism. Our highest-computation MoE model was more efficient at 1.56 TFLOPS/GPU, likely due to the larger matrices. These numbers represent a significant fraction of the theoretical maximum of 4.29 TFLOPS/GPU claimed by NVIDIA. Detailed results are in Appendix C, Table 7.\n\n5.2 100 BILLION WORD GOOGLE NEWS CORPUS\n\nOn the 1-billion-word corpus, adding additional capacity seems to produce diminishing returns as the number of parameters in the MoE layer exceeds 1 billion, as can be seen in Figure 2-left. We hypothesized that for a larger training set, even higher capacities would produce significant quality improvements. We constructed a similar training set consisting of shuffled unique sentences from Google's internal news corpus, totalling roughly 100 billion words. Similarly to the previous section, we tested a series of models with similar computational costs of about 8 million ops/timestep. In addition to a baseline LSTM model, we trained models augmented with MoE layers containing 32, 256, 1024, 4096, 16384, 65536, and 131072 experts. This corresponds to up to 137 billion parameters in the MoE layer. Details on architecture, training, and results are given in Appendix D.\n\nResults: Figure 3 shows test perplexity as a function of capacity after training on 10 billion words (top line) and 100 billion words (bottom line). When training over the full 100 billion words, test perplexity improves significantly up to 65536 experts (68 billion parameters), dropping 39% lower than the computationally matched baseline, but degrades at 131072 experts, possibly a result of too much sparsity. The widening gap between the two lines demonstrates (unsurprisingly) that increased model capacity helps more on larger training sets. Even at 65536 experts (99.994% layer sparsity), computational efficiency for the model stays at a respectable 0.72 TFLOPS/GPU.\n\n5.3 Machine Translation (Single Language Pair)\n\nModel Architecture: Our model was a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decreased the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We inserted MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). Each MoE layer contained up to 2048 experts each with about two million parameters, adding a total of about 8 billion parameters to the models. Further details on model architecture, testing procedure and results can be found in Appendix E.\n\nDatasets: We benchmarked our method on the WMT'14 En→Fr and En→De corpora, whose training sets have 36M sentence pairs and 5M sentence pairs, respectively. The experimental protocols were also similar to those in (Wu et al., 2016): newstest2014 was used as the test set to compare against previous work (Luong et al., 2015a; Zhou et al., 2016; Wu et al., 2016), while the combination of newstest2012 and newstest2013 was used as the development set. We also tested the same model on a Google's Production English to French data.\n\nTable 2: Results on WMT'14 En→ Fr newstest2014 (bold values represent best results). Model Test Test ops/timenstep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.69 40.35 85M 8.7B 3 days/64 k40s MoE with 2048 Experts (longer training) 2.63 40.56 85M 8.7B 6 days/64 k40s GNMT (Wu et al., 2016) 2.79 39.22 214M 278M 6 days/96 k80s GNMT+RL (Wu et al., 2016) 2.96 39.92 214M 278M 6 days/96 k80s PBMT (Durrani et al., 2014) 37.0 LSTM (6-layer) (Luong et al., 2015b) 31.5 LSTM (6-layer+PosUnk) (Luong et al., 2015b) 33.1 DeepAtt (Zhou et al., 2016) 37.7 DeepAtt+PosUnk (Zhou et al., 2016) 39.2\n\nTable 3: Results on WMT'14 En → De newstest2014 (bold values represent best results). Model Test Test ops/timestep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 4.64 26.03 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 5.25 24.91 214M 278M 1 day/96 k80s GNMT +RL (Wu et al., 2016) 8.08 24.66 214M 278M 1 day/96 k80s PBMT (Durrani et al., 2014) 20.7 DeepAtt (Zhou et al., 2016) 20.6\n\nTable 4: Results on the Google Production En→ Fr dataset (bold values represent best results). Model Eval Eval Test Test ops/timestep Total Training Perplexity BLEU Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.60 37.27 2.69 36.57 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 2.78 35.80 2.87 35.56 214M 278M 6 days/96 k80s\n\nResults: Tables 2, 3, and 4 show the results of our largest models, compared with published results. Our approach achieved BLEU scores of 40.56 and 26.03 on the WMT'14 En→Fr and En→De benchmarks. As our models did not use RL refinement, these results constitute significant gains of 1.34 and 1.12 BLEU score on top of the strong baselines in (Wu et al., 2016). The perplexity scores are also better.2 On the Google Production dataset, our model achieved 1.01 higher test BLEU score even after training for only one sixth of the time.\n\n5.4 Multilingual Machine Translation\n\nDataset: (Johnson et al., 2016) train a single GNMT (Wu et al., 2016) model on a very large combined dataset of twelve language pairs. Results are somewhat worse than those for 12 separately trained single-pair GNMT models. This is not surprising, given that the twelve models have 12 times the capacity and twelve times the aggregate training of the one model. We repeat this experiment with a single MoE-augmented model. See Appendix E for details on model architecture.\n\nWe train our model on the same dataset as (Johnson et al., 2016) and process the same number of training examples (about 3 billion sentence pairs). Our training time was shorter due to the lower computational budget of our model.\n\nResults: Results for the single-pair GNMT models, the multilingual GNMT model and the multilingual MoE model are given in Table 5. The MoE model achieves 19% lower perplexity on the dev set than the multilingual GNMT model. On BLEU score, the MoE model significantly beats the multilingual GNMT model on 11 of the 12 language pairs (by as much as 5.84 points), and even beats the monolingual GNMT models on 8 of 12 language pairs. The poor performance on English → Korean seems to be a result of severe overtraining, as for the rarer language pairs a small number of real examples were highly oversampled in the training corpus.\n\nTable 5: Multilingual Machine Translation (bold values represent best results). GNMT-Mono GNMT-Multi MoE-Multi MoE-Multi vs. GNMT-Multi Parameters 278M / model 278M 8.7B ops/timestep 212M 212M 102M training time, hardware various 21 days, 96 k20s 12 days, 64 k40s Perplexity (dev) 4.14 3.35 -19% French → English Test BLEU 36.47 34.40 37.46 +3.06 German → English Test BLEU 31.77 31.17 34.80 +3.63 Japanese → English Test BLEU 23.41 21.62 25.91 +4.29 Korean → English Test BLEU 25.42 22.87 28.71 +5.84 Portuguese → English Test BLEU 44.40 42.53 46.13 +3.60 Spanish → English Test BLEU 38.00 36.04 39.39 +3.35 English → French Test BLEU 35.37 34.00 36.59 +2.59 English → German Test BLEU 26.43 23.15 24.53 +1.38 English → Japanese Test BLEU 23.66 21.10 22.78 +1.68 English → Korean Test BLEU 19.75 18.41 16.62 -1.79 English → Portuguese Test BLEU 38.40 37.35 37.90 +0.55 English → Spanish Test BLEU 34.50 34.25 36.21 +1.96\n\n6 Conclusion\n\nThis work is the first to demonstrate major wins from conditional computation in deep networks.\n\nWe carefully identified the design considerations and challenges of conditional computing and addressed them with a combination of algorithmic and engineering solutions. While we focused on text, conditional computation may help in other domains as well, provided sufficiently large training sets. We look forward to seeing many novel implementations and applications of conditional computation in the years to come.\n\nAcknowledgments\n\nWe would like to thank all of the members of the Google Brain and Google Translate teams who helped us with this project, in particular Zhifeng Chen, Yonghui Wu, and Melvin Johnson. Thanks also to our anonymous ICLR reviewers for the helpful suggestions on making this paper better.\n\n2Reported perplexities relative to the tokenization used by both our models and GNMT.\n\nReferences\n\nMartín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Gordon Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda B. Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. CoRR, abs/1603.04467, 2016. URL http://arxiv.org/abs/1603.04467.\n\nRahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. CoRR, abs/1611.06194, 2016. URL http://arxiv.org/abs/1611.\n\n06194.\n\nA. Almahairi, N. Ballas, T. Cooijmans, Y. Zheng, H. Larochelle, and A. Courville. Dynamic Capacity Networks. ArXiv e-prints, November 2015.\n\nDario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan, Christopher Fougner, Tony Han, Awni Y. Hannun, Billy Jun, Patrick LeGresley, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595, 2015. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.\n\nEmmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297, 2015.\n\nYoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.\n\nCiprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling.\n\narXiv preprint arXiv:1312.3005, 2013.\n\nK. Cho and Y. Bengio. Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning. ArXiv e-prints, June 2014. Ronan Collobert, Samy Bengio, and Yoshua Bengio. A parallel mixture of SVMs for very large scale problems. Neural Computing, 2002.\n\nAndrew Davis and Itamar Arel. Low-rank approximations for conditional feedforward computation in deep neural networks. arXiv preprint arXiv:1312.4461, 2013. Marc Peter Deisenroth and Jun Wei Ng. Distributed Gaussian processes. In ICML, 2015.\n\nJohn Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization, 2010.\n\nNadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. Edinburgh's phrase-based machine translation systems for wmt-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation, 2014.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013. Ekaterina Garmash and Christof Monz. Ensemble learning for multi-source neural machine translation. In staff.science.uva.nl/c.monz, 2016.\n\nFelix A. Gers, Jürgen A. Schmidhuber, and Fred A. Cummins. Learning to forget: Continual prediction with lstm. Neural Computation, 2000.\n\nAudrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. Memory-efficient backpropagation through time. CoRR, abs/1606.03401, 2016. URL http://arxiv.org/ abs/1606.03401. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2015.\n\nGeoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 2012. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.\n\nSergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.\n\nRobert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computing, 1991.\n\nMelvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation.\n\nCoRR, abs/1611.04558, 2016. URL http://arxiv.org/abs/1611.04558.\n\nMichael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm.\n\nNeural Computing, 1994. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.\n\nDiederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.\n\nReinhard Kneser and Hermann. Ney. Improved backingoff for m-gram language modeling., 1995.\n\nAlex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.\n\nQuoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeffrey Dean, and Andrew Y. Ng. Building high-level features using large scale unsupervised learning. In ICML, 2012. Patrick Gallinari Ludovic Denoyer. Deep sequential neural network. arXiv preprint arXiv:1410.0510, 2014.\n\nMinh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attentionbased neural machine translation. EMNLP, 2015a.\n\nMinh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the rare word problem in neural machine translation. ACL, 2015b.\n\nCarl Edward Rasmussen and Zoubin Ghahramani. Infinite mixtures of Gaussian process experts.\n\nNIPS, 2002.\n\nHasim Sak, Andrew W Senior, and Françoise Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH, pp. 338–342, 2014.\n\nMike Schuster and Kaisuke Nakajima. Japanese and Korean voice search. ICASSP, 2012.\n\nBabak Shahbaba and Radford Neal. Nonlinear models using dirichlet process mixtures. JMLR, 2009.\n\nIlya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks.\n\nIn NIPS, 2014.\n\nLucas Theis and Matthias Bethge. Generative image modeling using spatial LSTMs. In NIPS, 2015. Volker Tresp. Mixtures of Gaussian Processes. In NIPS, 2001.\n\nYonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.\n\nBangpeng Yao, Dirk Walther, Diane Beck, and Li Fei-fei. Hierarchical mixture of classification experts uncovers interactions between brain regions. In NIPS. 2009.\n\nWojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization.\n\narXiv preprint arXiv:1409.2329, 2014.\n\nJie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with fast-forward connections for neural machine translation. arXiv preprint arXiv:1606.04199, 2016.\n\nAppendices A Load-Balancing Loss\n\nAs discussed in section 4, for load-balancing purposes, we want to define an additional loss function to encourage experts to receive roughly equal numbers of training examples. Unfortunately, the number of examples received by an expert is a discrete quantity, so it can not be used in backpropagation. Instead, we define a smooth estimator Load(X) of the number of examples assigned to each expert for a batch X of inputs. The smoothness allows us to back-propagate gradients through the estimator. This is the purpose of the noise term in the gating function. We define P(x, i) as the probability that G(x)iis nonzero, given a new random choice of noise on element i, but keeping the already-sampled choices of noise on the other elements. To compute P(x, i), we note that the G(x)iis nonzero if and only if H(x)iis greater than the k th-greatest element of H(x) excluding itself. The probability works out to be:\n\n$$P(x,i)=P r\\Big((x\\cdot W_{g}){i}+S t a n d x N o r a l()\\cdot S o f t p u s((x\\cdot W{n o i s e})_{i})$$ $$>k t h_e x c l u d i n(H(x),k,i)\\Big)$$ $$({\\boldsymbol{8}})$$\n\n$$(9)$$ $$(10)$$ (8) Where kth_excluding(v, k, i) means the kth highest component of v, excluding component i. Simplifying, we get:\n\n$$P(x,i)=\\Phi{\\Big(}{\\frac{(x\\cdot W_{g}){i}-k t h_c x c l u d i n g(H(x),k,i)}{S o f t p l u s((x\\cdot W{n o i s e})_{i})}}{\\Big)}$$\n\nWhere Φ is the CDF of the standard normal distribution.\n\n$$L o a d(X){i}=\\sum{x\\in X}P(x,i)$$\n\n$$(11)$$ P(x, i) (10) We can now define the load loss to be the square of the coefficient of variation of the load vector, multiplied by a hand-tuned scaling factor wload.\n\n$$L_{l o a d}(X)=w_{l o a d}\\cdot C V(L o a d(X))^{2}$$ Lload(X) = wload · CV (Load(X))2(11) Initial Load Imbalance: To avoid out-of-memory errors, we need to initialize the network in a state of approximately equal expert load (since the soft constraints need some time to work). To accomplish this, we initialize the matrices Wg and Wnoise to all zeros, which yields no signal and some noise.\n\nExperiments: We trained a set of models with identical architecture (the MoE-256 model described in Appendix C), using different values of wimportance and wload. We trained each model for 10 epochs, then measured perplexity on the test set. We also measured the coefficients of variation in Importance and Load, as well as ratio of the load on the most overloaded expert to the average load. This last value is significant for load balancing purposes on distributed hardware. All of these metrics were averaged over several training batches.\n\nwimportance wload Test Perplexity CV (Importance(X)) CV (Load(X)) max(Load(X)) mean(Load(X)) 0.0 0.0 39.8 3.04 3.01 17.80 0.2 0.0 35.6 0.06 0.17 1.47 0.0 0.2 35.7 0.22 0.04 1.15 0.1 0.1 35.6 0.06 0.05 1.14 0.01 0.01 35.7 0.48 0.11 1.37 1.0 1.0 35.7 0.03 0.02 1.07\n\nTable 6: Experiments with different combinations of losses.\n\n13 Results: Results are reported in Table 6. All the combinations containing at least one the two losses led to very similar model quality, where having no loss was much worse. Models with higher values of wload had lower loads on the most overloaded expert.\n\nB Hierachical Mixture Of Experts\n\nIf the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network.3If the hierarchical MoE consists of a groups of b experts each, we denote the primary gating network by Gprimary, the secondary gating networks by (G1, G2..Ga), and the expert networks by (E0,0, E0,1..Ea,b). The output of the MoE is given by:\n\n$$y_{H}=\\sum_{i=1}^{a}\\sum_{j=1}^{b}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}\\cdot E{i,j}(x)$$ $$(12)$$ $$(13)$$ $$(14)$$\n\nOur metrics of expert utilization change to the following:\n\n$$I m p o r t a n c e_{H}(X){i,j}=\\sum{x\\in X}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}$$ $$L o a d{H}(X){i,j}={\\frac{L o a d{p r i m a r y}(X){i}\\cdot L o a d{i}(X^{(i)})_{j}}{|X^{(i)}|}}$$\n\nLoadprimary and Loadi deonte the Load functions for the primary gating network and i th secondary gating network respectively. X(i) denotes the subset of X for which Gprimary(x)i > 0.\n\nIt would seem simpler to let LoadH(X)i,j = Loadi(Xi)j , but this would not have a gradient with respect to the primary gating network, so we use the formulation above.\n\nC 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS C.1 8-MILLION-OPERATIONS-PER-TIMESTEP MODELS Model Architecture: Our model consists of five layers: a word embedding layer, a recurrent Long Short-Term Memory (LSTM) layer (Hochreiter & Schmidhuber, 1997; Gers et al., 2000), a MoE layer, a second LSTM layer, and a softmax layer. The dimensionality of the embedding layer, the number of units in each LSTM layer, and the input and output dimensionality of the MoE layer are all equal to 512. For every layer other than the softmax, we apply drouput (Zaremba et al., 2014) to the layer output, dropping each activation with probability DropP rob, otherwise dividing by (1 − DropP rob). After dropout, the output of the previous layer is added to the layer output.\n\nThis residual connection encourages gradient flow (He et al., 2015).\n\nMoE Layer Architecture: Each expert in the MoE layer is a feed forward network with one ReLU-activated hidden layer of size 1024 and an output layer of size 512. Thus, each expert contains [512 ∗ 1024] + [1024 ∗ 512] = 1M parameters. The output of the MoE layer is passed through a sigmoid function before dropout. We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts.\n\nWe call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers. Thus, each example is processed by exactly 4 experts for a total of 4M ops/timestep. The two LSTM layers contribute 2M ops/timestep each for the desired total of 8M.\n\n3 We have not found the need for deeper hierarchies.\n\nComputationally-Matched Baselines: The MoE-4 model does not employ sparsity, since all 4 experts are always used. In addition, we trained four more computationally-matched baseline models with no sparsity:\n\nMoE-1-Wide: The MoE layer consists of a single \"expert\" containing one ReLU-activated hidden layer of size 4096.\n\nMoE-1-Deep: The MoE layer consists of a single \"expert\" containing four ReLU-activated hidden layers, each with size 1024.\n\n4xLSTM-512: We replace the MoE layer with two additional 512-unit LSTM layers.\n\nLSTM-2048-512: The model contains one 2048-unit LSTM layer (and no MoE). The output of the LSTM is projected down to 512 dimensions (Sak et al., 2014). The next timestep of the LSTM receives the projected output. This is identical to one of the models published in (Jozefowicz et al., 2016). We re-ran it to account for differences in training regimen, and obtained results very similar to the published ones. Training: The models were trained on a cluster of 16 K40 GPUs using the synchronous method described in Section 3. Each batch consisted of a set of sentences totaling roughly 300,000 words. In the interest of time, we limited training to 10 epochs, (27,000 steps). Training took 12-16 hours for all models, except for MoE-4, which took 18 hours (since all the expert computation was performed on only 4 of 16 GPUs). We used the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 1000 training steps, and decreased after that so as to be proportional to the inverse square root of the step number. The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016). For each model, we performed a hyper-parmeter search to find the best dropout probability, in increments of 0.1.\n\nTo ensure balanced expert utilization we set wimportance = 0.1 and wload = 0.1, as described in Section 4 and Appendix A.\n\nResults: We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol. Results are reported in Table 7. For each model, we report the test perplexity, the computational budget, the parameter counts, the value of DropP rob, and the computational efficiency.\n\nModel Test Test ops/timestep #Params excluding Total Drop- TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params P rob per GPU 10 epochs (final) (millions) (billions) (observed) Kneser-Ney 5-gram* 67.6 0.00001 1.8 LSTM-512-512* 54.1 2.4 2.4 0.8 0.1 LSTM-1024-512* 48.2 4.7 4.7 0.8 0.1 LSTM-2048-512* 45.0 43.7 9.4 9.4 0.8 0.1 0.61 LSTM-2048-512 44.7 9.4 9.4 0.8 0.1 1.21 4xLSTM-512 46.0 8.4 8.4 0.8 0.1 1.07 MoE-1-Wide 46.1 8.4 8.4 0.8 0.1 1.29 MoE-1-Deep 45.7 8.4 8.4 0.8 0.1 1.29 MoE-4 45.0 8.4 8.4 0.8 0.1 0.52 MoE-32 39.7 8.4 37.8 0.9 0.1 0.87 MoE-256 35.7 8.6 272.9 1.1 0.1 0.81 MoE-256-h 36.0 8.4 272.9 1.1 0.1 0.89 MoE-1024-h 34.6 8.5 1079.0 1.9 0.2 0.90 MoE-4096-h 34.1 8.9 4303.4 5.1 0.2 0.74 2xLSTM-8192-1024* 34.7 30.6 151.0 151.0 1.8 0.25 1.09 MoE-34M 31.3 33.8 4313.9 6.0 0.3 1.22 MoE-143M 28.0 142.7 4371.1 6.0 0.4 1.56\n\nC.2 More Expensive Models\n\nWe ran two additional models (MoE-34M and MoE-143M) to investigate the effects of adding more computation in the presence of a large MoE layer. These models have computation budgets of 34M and 143M ops/timestep. Similar to the models above, these models use a MoE layer between two LSTM layers. The dimensionality of the embedding layer, and the input and output dimensionality of the MoE layer are set to 1024 instead of 512. For MoE-34M, the LSTM layers have 1024 units.\n\nFor MoE-143M, the LSTM layers have 4096 units and an output projection of size 1024 (Sak et al., 2014). MoE-34M uses a hierarchical MoE layer with 1024 experts, each with a hidden layer of size 2048. MoE-143M uses a hierarchical MoE layer with 256 experts, each with a hidden layer of size 8192. Both models have 4B parameters in the MoE layers. We searched for the best DropP rob for each model, and trained each model for 10 epochs.\n\nThe two models achieved test perplexity of 31.3 and 28.0 respectively, showing that even in the presence of a large MoE, more computation is still useful. Results are reported at the bottom of Table 7. The larger of the two models has a similar computational budget to the best published model from the literature, and training times are similar. Comparing after 10 epochs, our model has a lower test perplexity by 18%.\n\nD 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS Model Architecture: The models are similar in structure to the 8-million-operations-per-timestep models described in the previous section. We vary the number of experts between models, using an ordinary MoE layer with 32 experts and hierarchical MoE layers with 256, 1024, 4096, 16384, 65536 and 131072 experts. For the hierarchical MoE layers, the first level branching factors are 32, 32, 64, 128, 256 and 256, respectively.\n\nTraining: Models are trained on a cluster of 32 Tesla K40 GPUs, except for the last two models, which are trained on clusters of 64 and 128 GPUs so as to have enough memory for all the parameters. For all models, training batch sizes are approximately 2.5 million words. Models are trained once-through over about 100 billion words. We implement several memory optimizations in order to fit up to 1 billion parameters per GPU. First, we do not store the activations of the hidden layers of the experts, but instead recompute them on the backwards pass. Secondly, we modify the optimizer on the expert parameters to require less auxiliary storage: The Adam optimizer (Kingma & Ba, 2015) keeps first and second moment estimates of the perparameter gradients. This triples the required memory. To avoid keeping a first-moment estimator, we set β1 = 0. To reduce the size of the second moment estimator, we replace it with a factored approximation. For a matrix of parameters, instead of maintaining a full matrix of second-moment estimators, we maintain vectors of row-wise and column-wise averages of that matrix. At each step, the matrix of estimators is taken to be the outer product of those two vectors divided by the mean of either one. This technique could similarly be applied to Adagrad (Duchi et al., 2010).\n\nModel Test Test ops/timestep #Params excluding Total TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params per GPU .1 epochs 1 epoch (millions) (billions) (observed) Kneser-Ney 5-gram 67.1 45.3 0.00001 76.0 4xLSTM-512 54.5 47.0 8.4 8.4 0.1 1.23 MoE-32 48.5 40.4 8.4 37.8 0.1 0.83 MoE-256-h 42.8 35.3 8.4 272.9 0.4 1.11 MoE-1024-h 40.3 32.7 8.5 1079.0 1.2 1.14 MoE-4096-h 38.9 30.9 8.6 4303.4 4.4 1.07 MoE-16384-h 38.2 29.7 8.8 17201.0 17.3 0.96 MoE-65536-h 38.2 28.9 9.2 68791.0 68.9 0.72 MoE-131072-h 39.8 29.2 9.7 137577.6 137.7 0.30\n\nResults: We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8. Perplexity after 100 billion training words is 39% lower for the 68-billion-parameter MoE model than for the baseline model. It is notable that the measured computational efficiency of the largest model (0.30 TFLOPS/GPU) is very low compared to the other models. This is likely a result of the fact that, for purposes of comparison to the other models, we did not increase the training batch size proportionally to the number of GPUs. For comparison, we include results for a computationally matched baseline model consisting of 4 LSTMs, and for an unpruned 5-gram model with Kneser-Ney smoothing (Kneser & Ney, 1995).4\n\nE Machine Translation - Experimental Details\n\nModel Architecture for Single Language Pair MoE Models: Our model is a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decrease the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We insert MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). We use an attention mechanism between the encoder and decoder, with the first decoder LSTM receiving output from and providing input for the attention 5. All of the layers in our model have input and output dimensionality of 512. Our LSTM layers have 2048 hidden units, with a 512-dimensional output projection. We add residual connections around all LSTM and MoE layers to encourage gradient flow (He et al., 2015). Similar to GNMT, to effectively deal with rare words, we used subword units (also known as \"wordpieces\") (Schuster & Nakajima, 2012) for inputs and outputs in our system.\n\nWe use a shared source and target vocabulary of 32K wordpieces. We also used the same beam search technique as proposed in (Wu et al., 2016).\n\nWe train models with different numbers of experts in the MoE layers. In addition to a baseline model with no MoE layers, we train models with flat MoE layers containing 32 experts, and models with hierarchical MoE layers containing 512 and 2048 experts. The flat MoE layers use k = 4 and the hierarchical MoE models use k = 2 at each level of the gating network. Thus, each input is processed by exactly 4 experts in each MoE layer. Each expert in the MoE layer is a feed forward network with one hidden layer of size 2048 and ReLU activation. Thus, each expert contains [512 ∗ 2048] + [2048 ∗ 512] = 2M parameters. The output of the MoE layer is passed through a sigmoid function. We use the strictly-balanced gating function described in Appendix F.\n\nModel Architecture for Multilingual MoE Model: We used the same model architecture as for the single-language-pair models, with the following exceptions: We used noisy-top-k gating as described in Section 2.1, not the scheme from Appendix F. The MoE layers in the encoder and decoder are non-hierarchical MoEs with n = 512 experts, and k = 2. Each expert has a larger hidden layer of size 8192. This doubles the amount of computation in the MoE layers, raising the computational budget of the entire model from 85M to 102M ops/timestep. Training: We trained our networks using the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 2000 training steps, held constant for an additional 8000 steps, and decreased after that so as to be proportional to the inverse square root of the step number. For the single-language-pair models, similarly to (Wu et al., 2016), we applied dropout (Zaremba et al., 2014) to the output of all embedding, LSTM and MoE layers, using DropP rob = 0.4. Training was done synchronously on a cluster of up to 64 GPUs as described in section 3. Each training batch consisted of a set of sentence pairs containing roughly 16000 words per GPU.\n\nTo ensure balanced expert utilization we set wimportance = 0.01 and wload = 0.01, as described in Section 4 and Appendix A. Metrics: We evaluated our models using the perplexity and the standard BLEU score metric. We reported tokenized BLEU score as computed by the multi-bleu.pl script, downloaded from the public implementation of Moses (on Github), which was also used in (Luong et al., 2015a). Results: Tables 2, 3 and 4 in Section 5.3 show comparisons of our results to other published methods. Figure 4 shows test perplexity as a function of number of words in the (training data's) source sentences processed for models with different numbers of experts. As can be seen from the Figure, as we increased the number of experts to approach 2048, the test perplexity of our model continued to improve.\n\nWe found that the experts indeed become highly specialized by syntax and/or semantics, as can be seen in Table 9. For example, one expert is used when the indefinite article \"a\" introduces the direct object in a verb phrase indicating importance or leadership.\n\nExpert 381 Expert 752 Expert 2004 ... with researchers , ... ... plays a core ... ... with rapidly growing ... ... to innovation . ... plays a critical ... ... under static conditions ... ... tics researchers . ... provides a legislative ... ... to swift ly ... ... the generation of ... ... play a leading ... ... to dras tically ... ... technology innovations is ... ... assume a leadership ... ... the rapid and ... ... technological innovations , ... ... plays a central ... ... the fast est ... ... support innovation throughout ... ... taken a leading ... ... the Quick Method ... ... role innovation will ... ... established a reconciliation ... ... rec urrent ) ... ... research scienti st ... ... played a vital ... ... provides quick access ... ... promoting innovation where ... ... have a central ... ... of volatile organic ... ... ... ...\n\nDue to some peculiarities in our infrastructure which have since been fixed, at the time we ran some of the machine translation experiments, our models ran faster if every expert received exactly the same batch size. To accommodate this, we used a different gating function which we describe below. Recall that we define the softmax gating function to be:\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ Gσ(x) = Sof tmax(x · Wg) (15) Sparse Gating (alternate formulation): To obtain a sparse gating vector, we multiply Gσ(x) component-wise with a sparse mask M(Gσ(x)) and normalize the output. The mask itself is a function of Gσ(x) and specifies which experts are assigned to each input example:\n\n$$(15)$$ $$G(x){i}=\\frac{G{\\sigma}(x){i}M(G{\\sigma}(x)){i}}{\\sum{j=1}^{n}G_{\\sigma}(x){j}M(G{\\sigma}(x))_{j}}\\tag{1}$$ $$(17)$$\n\n$$(16)$$\n\nTop-K Mask: To implement top-k gating in this formulation, we would let M(v) = T opK(v, k), where:\n\n$$T o p K(v,k){i}=\\begin{cases}1&{\\mathrm{if~}}v{i}{\\mathrm{~is~in~the~top~}}k{\\mathrm{~elements~of~}}v.\\ 0&{\\mathrm{otherwise.}}\\end{cases}$$ 0 otherwise. (17) Batchwise Mask: To force each expert to receive the exact same number of examples, we introduce an alternative mask function, Mbatchwise(X, m), which operates over batches of input vectors.\n\nInstead of keeping the top k values per example, we keep the top m values per expert across the training batch, where m = k|X| n, so that each example is sent to an average of k experts.\n\n$$M_{b a t c h w i s e}(X,m){j,i}=\\begin{cases}1&\\text{if}X{j,i}\\text{is in the top}m\\text{values for to expert}i\\ 0&\\text{otherwise}\\end{cases}$$\n\nAs our experiments suggest and also observed in (Ioffe & Szegedy, 2015), using a batchwise function during training (such as Mbatchwise) requires modifications to the inference when we may not have a large batch of examples. Our solution to this is to train a vector T of per-expert threshold values to approximate the effects of the batchwise mask. We use the following mask at inference time:\n\n$$(18)$$ $$M_{threshold}(x,T){i}=\\begin{cases}1&\\text{if}x{i}>T_{i}\\ 0&\\text{otherwise}\\end{cases}\\tag{1}$$ $$(19)$$\n\nTo learn the threshold values, we apply an additional loss at training time which is minimized when the batchwise mask and the threshold mask are identical.\n\n$$L_{batchwise}(X,T,m)=\\sum_{j=1}^{|X|}\\sum_{i=1}^{n}(M_{threshold}(x,T){i}-M{batchwise}(X,m){j,i})(X{j,i}-T_{i})\\tag{20}$$\n\nG ATTENTION FUNCTION The attention mechanism described in GNMT (Wu et al., 2016) involves a learned \"Attention Function\" A(xi, yj ) which takes a \"source vector\" xi and a \"target vector\" yj , and must be computed for every source time step i and target time step j. In GNMT, the attention function is implemented as a feed forward neural network with a hidden layer of size n. It can be expressed as:\n\n$$A_{G N M T}(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}t a n h((x_{i}U){d}+(y{j}W)_{d})$$\n\nWhere U and W are trainable weight matrices and V is a trainable weight vector.\n\nFor performance reasons, in our models, we used a slightly different attention function:\n\n$$A(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}tanh((x_{i}U){d})tanh((y{j}W)_{d})\\tag{22}$$\n\nWith our attention function, we can simultaneously compute the attention function on multiple source time steps and multiple target time steps using optimized matrix multiplications. We found little difference in quality between the two functions.\n\n$$(21)$$",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/1701.06538v1.md"
+          },
+          "headlines": {
+            "1 Introduction And Related Work": [
+              "1.1 Conditional Computation",
+              "1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER",
+              "1.3 Related Work On Mixtures Of Experts"
+            ],
+            "2 The Structure Of The Mixture-Of-Experts Layer": [
+              "2.1 Gating Network"
+            ],
+            "3 Addressing Performance Challenges": [
+              "3.1 The Shrinking Batch Problem",
+              "3.2 Network Bandwidth"
+            ],
+            "4 Balancing Expert Utilization": [],
+            "5 Experiments": [
+              "5.1 1 Billion Word Language Modeling Benchmark",
+              "5.2 100 BILLION WORD GOOGLE NEWS CORPUS",
+              "5.3 Machine Translation (Single Language Pair)",
+              "5.4 Multilingual Machine Translation"
+            ],
+            "6 Conclusion": [],
+            "Appendices": [
+              "A Load-Balancing Loss",
+              "B Hierachical Mixture Of Experts",
+              "C 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS",
+              "D 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS",
+              "E Machine Translation - Experimental Details"
+            ]
+          },
+          "summary": "The paper introduces a Sparsely-Gated Mixture-of-Experts (MoE) layer to enhance neural network capacity without a proportional increase in computation. This approach uses a trainable gating network to select a sparse combination of experts for each input, significantly improving model capacity and performance in language modeling and machine translation tasks. The MoE layer, with up to 137 billion parameters, achieves better results than state-of-the-art models at lower computational costs. The paper addresses challenges like batch size and network bandwidth, proposing solutions like mixing data and model parallelism. Experiments show that MoE models outperform traditional models in various benchmarks, demonstrating the potential of conditional computation in deep learning.",
+          "summary_embedding": [
+            -0.01644345559179783,
+            -0.016995344310998917,
+            -0.016372699290513992,
+            -0.014349107630550861,
+            0.02282555401325226,
+            0.045934125781059265,
+            -0.014646278694272041,
+            0.03563220426440239,
+            -0.004829026758670807,
+            0.005617944523692131,
+            -0.00932550523430109,
+            -0.008143897168338299,
+            0.005575491581112146,
+            0.01538213063031435,
+            -0.005653322208672762,
+            -0.020306676626205444,
+            -0.013832597061991692,
+            -0.023235931992530823,
+            0.028344441205263138,
+            -0.020080260932445526,
+            0.011823155917227268,
+            -0.020943470299243927,
+            -0.025868017226457596,
+            -0.016500059515237808,
+            0.025061409920454025,
+            0.024806693196296692,
+            0.02583971433341503,
+            0.003827843815088272,
+            -0.03492465242743492,
+            0.033651065081357956,
+            0.014674580655992031,
+            0.012488252483308315,
+            -0.031500112265348434,
+            -0.009162768721580505,
+            -0.05351905897259712,
+            -0.05227377265691757,
+            0.04143410921096802,
+            0.022245362401008606,
+            -0.016231190413236618,
+            0.023717066273093224,
+            0.017264213413000107,
+            0.005816058721393347,
+            -0.020632149651646614,
+            -0.020844414830207825,
+            0.003700484987348318,
+            -0.013103820383548737,
+            -0.01983969286084175,
+            -0.013528349809348583,
+            -0.01828308403491974,
+            -0.01577835902571678,
+            0.009551920928061008,
+            0.0002856731880456209,
+            0.0029186424799263477,
+            -0.003424540627747774,
+            -0.047830358147621155,
+            0.0248208437114954,
+            0.019853845238685608,
+            0.03492465242743492,
+            0.04633035510778427,
+            0.013011839240789413,
+            0.01516986545175314,
+            -0.014271277002990246,
+            -0.04021712392568588,
+            0.005497660953551531,
+            -0.00041656987741589546,
+            -0.005380915477871895,
+            -0.005819596350193024,
+            0.008575502783060074,
+            0.008724087849259377,
+            -0.01143400277942419,
+            -0.04202845320105553,
+            0.013952880166471004,
+            -0.020561393350362778,
+            0.00010143388499272987,
+            0.01354957651346922,
+            0.008292482234537601,
+            0.02677368000149727,
+            0.03195294365286827,
+            0.023971782997250557,
+            -0.008724087849259377,
+            -0.01340806670486927,
+            0.019924599677324295,
+            -0.0022676968947052956,
+            0.007018893025815487,
+            0.04570771008729935,
+            -0.04033033177256584,
+            0.02163686975836754,
+            -0.025726506486535072,
+            -0.015339677222073078,
+            -0.005349075887352228,
+            -0.022471778094768524,
+            0.02806142158806324,
+            -0.007344366051256657,
+            0.017801949754357338,
+            0.017094401642680168,
+            0.01594817079603672,
+            0.013330236077308655,
+            -0.001894464367069304,
+            -0.022938761860132217,
+            0.0018360915128141642,
+            -0.01541043259203434,
+            0.02114158496260643,
+            0.0015230007702484727,
+            0.005469358991831541,
+            0.027594437822699547,
+            0.031160488724708557,
+            -0.023490650579333305,
+            0.009502392262220383,
+            0.02422650158405304,
+            0.06967949122190475,
+            0.015424583107233047,
+            -0.006435164716094732,
+            -0.0024640418123453856,
+            0.04143410921096802,
+            0.0015513027319684625,
+            0.0038349193055182695,
+            -0.06826439499855042,
+            -0.03022652305662632,
+            -0.0011365016689524055,
+            -0.02254253253340721,
+            0.02227366529405117,
+            0.01002597901970148,
+            -0.009750034660100937,
+            -0.022457627579569817,
+            -0.04516997188329697,
+            0.05190584436058998,
+            -0.00545874610543251,
+            -0.016259491443634033,
+            -0.005554265342652798,
+            -0.05340585112571716,
+            0.00039136342820711434,
+            0.015042506158351898,
+            -0.018481198698282242,
+            -0.03670767694711685,
+            0.036141637712717056,
+            0.0007398316520266235,
+            0.03741522878408432,
+            0.012459950521588326,
+            -0.01708024926483631,
+            -0.020094411447644234,
+            0.022259512916207314,
+            -0.011639192700386047,
+            -0.0038313816767185926,
+            -0.027339721098542213,
+            0.0065236082300543785,
+            -0.028103873133659363,
+            -0.0014566679019480944,
+            0.06730213016271591,
+            -0.04816998168826103,
+            0.02946236915886402,
+            -0.029660483822226524,
+            0.023660462349653244,
+            -0.008575502783060074,
+            0.0278916098177433,
+            -0.010238244198262691,
+            0.005840822588652372,
+            -0.013457595370709896,
+            -0.0010586712742224336,
+            -0.020745357498526573,
+            0.0034210027661174536,
+            -0.016259491443634033,
+            -0.027750099077820778,
+            -0.007054270710796118,
+            0.031273696571588516,
+            0.01698119193315506,
+            0.025655752047896385,
+            0.025429336354136467,
+            0.01246702577918768,
+            -0.04935866594314575,
+            0.02595292218029499,
+            0.0018926954362541437,
+            0.013514199294149876,
+            0.013181651011109352,
+            -0.0010418669553473592,
+            0.025599148124456406,
+            0.012049571610987186,
+            0.025061409920454025,
+            0.037443529814481735,
+            0.0027682881336659193,
+            0.04737752676010132,
+            -0.015693452209234238,
+            -0.005292471963912249,
+            0.028528403490781784,
+            0.000505455827806145,
+            0.02465103194117546,
+            -0.001686621573753655,
+            0.013330236077308655,
+            0.000776978034991771,
+            0.03834919258952141,
+            -0.006339645478874445,
+            0.029518973082304,
+            0.021608568727970123,
+            0.028316138312220573,
+            -0.014759486541152,
+            0.023122724145650864,
+            -0.019754787907004356,
+            -0.02292460948228836,
+            0.035547297447919846,
+            0.004082561936229467,
+            0.01886327564716339,
+            -0.028457649052143097,
+            -0.030679354444146156,
+            0.015452885068953037,
+            -0.01855195313692093,
+            0.016556663438677788,
+            0.03639635816216469,
+            0.0008273909334093332,
+            -0.0025719432160258293,
+            0.026292547583580017,
+            -0.04245298355817795,
+            0.012856177985668182,
+            -0.01532552670687437,
+            -0.02855670638382435,
+            -0.019528372213244438,
+            -0.00875946506857872,
+            -0.0018095583654940128,
+            -0.013691086322069168,
+            -0.0037570889107882977,
+            -0.007563706487417221,
+            -0.02459442801773548,
+            0.00935380719602108,
+            0.013627407141029835,
+            0.009849091991782188,
+            -0.018665160983800888,
+            0.0019952901639044285,
+            -0.008257105015218258,
+            -0.0160047747194767,
+            -0.03752843663096428,
+            0.015268922783434391,
+            -0.0024888061452656984,
+            0.009191070683300495,
+            0.04078316316008568,
+            0.028485950082540512,
+            -0.005055442452430725,
+            0.0329718142747879,
+            -0.033707667142152786,
+            0.017915157601237297,
+            -0.008320784196257591,
+            0.009191070683300495,
+            0.002989397384226322,
+            -0.025542544201016426,
+            0.027424626052379608,
+            -0.03792466223239899,
+            -0.038151077926158905,
+            0.022995365783572197,
+            0.011858533136546612,
+            0.0017096169758588076,
+            -0.019386861473321915,
+            -0.017717044800519943,
+            0.0008499440737068653,
+            0.002752368338406086,
+            -0.018834972754120827,
+            -0.01019579079002142,
+            0.02227366529405117,
+            -0.025500090792775154,
+            0.036085035651922226,
+            0.023391593247652054,
+            -0.03520767390727997,
+            -0.00608492735773325,
+            -0.01903308741748333,
+            0.059037946164608,
+            -0.0011506526498124003,
+            -0.010301923379302025,
+            0.022938761860132217,
+            -0.027268964797258377,
+            -0.004524780437350273,
+            0.022301966324448586,
+            0.01828308403491974,
+            -0.008278331719338894,
+            0.003983504604548216,
+            -0.022287815809249878,
+            -0.05281151086091995,
+            -0.010577867738902569,
+            -0.0003305583668407053,
+            0.054368119686841965,
+            -0.012367968447506428,
+            0.0020695829298347235,
+            -0.006413938011974096,
+            0.004100250545889139,
+            0.03704730048775673,
+            0.002140337834134698,
+            0.0025082635693252087,
+            -0.013698162510991096,
+            0.0371888130903244,
+            0.022316116839647293,
+            0.005883275531232357,
+            -0.012162779457867146,
+            0.005377377849072218,
+            -0.04899074137210846,
+            -0.0009118546731770039,
+            0.005741765722632408,
+            0.007386818993836641,
+            -0.025910470634698868,
+            0.017717044800519943,
+            0.009205221198499203,
+            -0.0024693484883755445,
+            -0.009792488068342209,
+            0.01173117384314537,
+            0.025429336354136467,
+            -0.019938750192523003,
+            -0.05700020492076874,
+            0.0034705311991274357,
+            0.012014193460345268,
+            -0.004054259974509478,
+            -0.019401011988520622,
+            0.006792477332055569,
+            0.0021934041287750006,
+            0.022698193788528442,
+            -0.011271266266703606,
+            -0.0029221803415566683,
+            0.040585052222013474,
+            0.009516543708741665,
+            -0.0016813150141388178,
+            -0.017717044800519943,
+            -0.011794853024184704,
+            0.018070818856358528,
+            0.004740582779049873,
+            0.0035289041697978973,
+            -0.01751893013715744,
+            -0.029264256358146667,
+            0.02800481766462326,
+            0.019967053085565567,
+            -0.048905834555625916,
+            -0.008200501091778278,
+            -0.019273653626441956,
+            0.0019599127117544413,
+            0.008695785887539387,
+            0.04613224044442177,
+            -0.00590803986415267,
+            0.008313708938658237,
+            -0.020179318264126778,
+            -0.05003791302442551,
+            -0.02708500251173973,
+            0.03707560524344444,
+            -0.028684064745903015,
+            -0.04276430234313011,
+            -0.010252394713461399,
+            0.016740625724196434,
+            -0.009403335861861706,
+            -0.018679311498999596,
+            -0.011738249100744724,
+            -0.04969828948378563,
+            -0.003497064346447587,
+            0.0337359681725502,
+            0.017264213413000107,
+            -0.008441068232059479,
+            -0.033877480775117874,
+            0.019641580060124397,
+            -0.007160402834415436,
+            -0.006474079564213753,
+            0.0007862646016292274,
+            0.008794843219220638,
+            0.02400008589029312,
+            0.011879759840667248,
+            -0.02763689123094082,
+            -0.03427370637655258,
+            -0.0024888061452656984,
+            0.014929298311471939,
+            -0.00878776703029871,
+            0.0245378240942955,
+            0.024679332971572876,
+            -0.022995365783572197,
+            -0.01654251106083393,
+            0.015750056132674217,
+            -0.05779265984892845,
+            -0.010287772864103317,
+            -0.029235953465104103,
+            -0.006183984223753214,
+            0.0009613831643946469,
+            0.0061096916906535625,
+            0.00012304731353651732,
+            0.03846240043640137,
+            0.008801918476819992,
+            0.006014172453433275,
+            0.009191070683300495,
+            0.01913214474916458,
+            0.028471799567341805,
+            0.009155693463981152,
+            -0.03517937287688255,
+            0.02850010246038437,
+            -0.017051948234438896,
+            0.007811348885297775,
+            -0.025754809379577637,
+            -0.013033065013587475,
+            0.00443633645772934,
+            0.02725481428205967,
+            -0.000503244751598686,
+            0.018849123269319534,
+            -0.021919889375567436,
+            0.037500135600566864,
+            -0.029915200546383858,
+            -0.05278320610523224,
+            -0.02005195803940296,
+            -0.008285406976938248,
+            0.015764206647872925,
+            0.017278363928198814,
+            -0.019924599677324295,
+            -0.0286557637155056,
+            -0.021750077605247498,
+            0.006470541935414076,
+            0.010542490519583225,
+            -0.021099131554365158,
+            -0.010627396404743195,
+            0.01502835564315319,
+            -0.00404364662244916,
+            -0.01886327564716339,
+            0.013684011064469814,
+            0.015693452209234238,
+            0.03169822692871094,
+            -0.03308502212166786,
+            0.004287751391530037,
+            -0.025967074558138847,
+            -0.04327373951673508,
+            0.004096712917089462,
+            0.03166992589831352,
+            0.027509532868862152,
+            -0.018636859953403473,
+            0.010754755698144436,
+            0.003990580327808857,
+            0.004234685096889734,
+            -0.0033803186379373074,
+            0.023603858426213264,
+            0.03291521221399307,
+            0.02763689123094082,
+            -0.011653343215584755,
+            0.004921008367091417,
+            -0.01559439580887556,
+            -0.0011108530452474952,
+            0.01865101046860218,
+            -0.017816102132201195,
+            0.033877480775117874,
+            -0.029405765235424042,
+            0.02104252763092518,
+            -0.035122767090797424,
+            0.006141531281173229,
+            -0.012099100276827812,
+            -0.00404364662244916,
+            0.04833979532122612,
+            -0.017646290361881256,
+            -0.0019546060357242823,
+            -0.008101443760097027,
+            -0.014575524255633354,
+            -0.02158026583492756,
+            0.008766540326178074,
+            -0.04033033177256584,
+            -0.012367968447506428,
+            -0.028429346159100533,
+            0.003424540627747774,
+            0.019910449162125587,
+            -0.06277380883693695,
+            -0.06237758323550224,
+            0.012275987304747105,
+            -0.04918885603547096,
+            -0.026419905945658684,
+            0.02963218092918396,
+            -0.002984090941026807,
+            0.0012780115939676762,
+            -0.0024003623984754086,
+            0.0068101659417152405,
+            -0.029207652434706688,
+            0.00425944896414876,
+            -0.000887974863871932,
+            0.03271709755063057,
+            -0.021056678146123886,
+            -0.008122670464217663,
+            0.004878554958850145,
+            -0.06860402226448059,
+            -0.028344441205263138,
+            0.011271266266703606,
+            -0.009459939785301685,
+            -0.0066615804098546505,
+            -0.017915157601237297,
+            -0.004383270163089037,
+            0.0008950503543019295,
+            0.0032830305863171816,
+            0.05801907554268837,
+            -0.01983969286084175,
+            -0.053773775696754456,
+            0.03633975237607956,
+            0.00887267291545868,
+            0.029603879898786545,
+            -0.025556694716215134,
+            -0.017533080652356148,
+            -0.01855195313692093,
+            -0.020504789426922798,
+            -0.04123599827289581,
+            0.00404364662244916,
+            -0.005918653216212988,
+            0.009212297387421131,
+            -0.0009693430620245636,
+            0.02763689123094082,
+            0.029235953465104103,
+            0.023108573630452156,
+            0.012750045396387577,
+            0.01079013291746378,
+            -0.022372720763087273,
+            -0.021594416350126266,
+            0.0012222920777276158,
+            0.04683978855609894,
+            0.024636881425976753,
+            -0.02675952948629856,
+            -0.011504758149385452,
+            0.03953787684440613,
+            0.0419718474149704,
+            -0.01974063739180565,
+            -0.02276895008981228,
+            0.00764861237257719,
+            0.03266049548983574,
+            -0.04477374628186226,
+            0.005932804197072983,
+            0.03410389646887779,
+            0.014405711553990841,
+            -0.005356151144951582,
+            -0.010294848121702671,
+            0.022570835426449776,
+            0.03152841329574585,
+            -0.008724087849259377,
+            0.01876421831548214,
+            0.016910437494516373,
+            0.029349161311984062,
+            -0.0067889392375946045,
+            0.0005766529939137399,
+            -0.024735936895012856,
+            0.055188875645399094,
+            -0.03059444949030876,
+            0.02909444272518158,
+            0.010316074825823307,
+            -0.00593634182587266,
+            0.039056744426488876,
+            -0.016641568392515182,
+            -0.0431322306394577,
+            0.01913214474916458,
+            -0.006979977712035179,
+            -0.0055613406002521515,
+            -0.03444352000951767,
+            -0.007592008449137211,
+            -0.007507102098315954,
+            -0.006714646704494953,
+            -0.022556684911251068,
+            -0.02606613002717495,
+            0.004415110219269991,
+            0.00973588414490223,
+            -0.006484692916274071,
+            0.00698705343529582,
+            0.02092931978404522,
+            -0.013507124036550522,
+            0.02525952458381653,
+            -0.046047333627939224,
+            -0.0008366775000467896,
+            -0.0038596836384385824,
+            -0.025712355971336365,
+            -0.0345001220703125,
+            -0.0005770951975136995,
+            -0.0024251264985650778,
+            -0.024622729048132896,
+            -0.00914154201745987,
+            -0.008752389810979366,
+            -0.0023119186516851187,
+            -0.004938696976751089,
+            -0.055188875645399094,
+            0.008695785887539387,
+            0.010846736840903759,
+            -0.002713453024625778,
+            0.0055507272481918335,
+            -0.010613245889544487,
+            -0.03039633482694626,
+            0.03039633482694626,
+            0.04129260033369064,
+            0.033707667142152786,
+            0.014334957115352154,
+            -0.0074646491557359695,
+            -0.017165156081318855,
+            0.006060163024812937,
+            0.021721776574850082,
+            0.00902833417057991,
+            0.007846726104617119,
+            0.03877372294664383,
+            0.00011331850691931322,
+            -0.008349086157977581,
+            -0.009764186106622219,
+            0.028740668669342995,
+            -0.001085204305127263,
+            0.023603858426213264,
+            0.050802066922187805,
+            0.030424637719988823,
+            0.020589696243405342,
+            -0.01681138016283512,
+            -0.02010856196284294,
+            -0.00851889792829752,
+            -0.02855670638382435,
+            0.0364246591925621,
+            0.012127402238547802,
+            -0.005232329946011305,
+            -0.011540135368704796,
+            0.019061388447880745,
+            -0.033339742571115494,
+            -0.0037358624394983053,
+            0.00048422932741232216,
+            0.013924578204751015,
+            -0.03379257395863533,
+            0.0002480846014805138,
+            0.031613320112228394,
+            0.014957600273191929,
+            -0.016995344310998917,
+            -0.0188774261623621,
+            0.010549565777182579,
+            0.020462337881326675,
+            0.018948180601000786,
+            0.00037743354914709926,
+            0.00897173024713993,
+            -0.00022121983056422323,
+            -0.0005319889169186354,
+            0.01590571738779545,
+            -0.007952858693897724,
+            0.036254845559597015,
+            -0.003525366308167577,
+            -0.01325240544974804,
+            -0.0037500134203583,
+            -0.021438755095005035,
+            -0.024467067793011665,
+            0.043556760996580124,
+            0.023971782997250557,
+            0.04661337286233902,
+            -0.005660397466272116,
+            -0.004245297983288765,
+            0.025825563818216324,
+            0.008221727795898914,
+            -0.03288691118359566,
+            0.000265110022155568,
+            0.005281858611851931,
+            -0.02892463095486164,
+            0.016641568392515182,
+            -0.02476423978805542,
+            -0.016896286979317665,
+            -0.006771250627934933,
+            -0.0015557248843833804,
+            -0.028188779950141907,
+            0.01703779585659504,
+            0.0031309074256569147,
+            -0.0027099153958261013,
+            -0.039764292538166046,
+            0.006880920846015215,
+            0.020419884473085403,
+            0.015311375260353088,
+            0.0037358624394983053,
+            0.011525984853506088,
+            -0.016613267362117767,
+            0.002126186853274703,
+            0.03379257395863533,
+            0.004878554958850145,
+            0.02005195803940296,
+            0.0027647505048662424,
+            0.023773670196533203,
+            0.03633975237607956,
+            -0.01681138016283512,
+            -0.02189158834517002,
+            0.01519816741347313,
+            -0.02233026921749115,
+            0.029943503439426422,
+            0.004889168310910463,
+            -0.012283062562346458,
+            0.022033097222447395,
+            -0.009827865287661552,
+            -0.0027346794959157705,
+            0.01659911684691906,
+            -0.022740647196769714,
+            -0.017717044800519943,
+            0.03183973580598831,
+            -0.039452970027923584,
+            0.012856177985668182,
+            -0.008547200821340084,
+            0.015863263979554176,
+            -0.023066120222210884,
+            0.011172209866344929,
+            0.007100261282175779,
+            -0.01573590561747551,
+            0.010797208175063133,
+            0.009693430736660957,
+            0.00545874610543251,
+            0.0018926954362541437,
+            0.02179253101348877,
+            -0.0006049550138413906,
+            -0.011257115751504898,
+            -0.018155725672841072,
+            -0.004694592207670212,
+            -0.002614396158605814,
+            -0.016556663438677788,
+            0.01865101046860218,
+            -0.041066184639930725,
+            -0.013280707411468029,
+            -0.008879749104380608,
+            -0.013988257385790348,
+            -0.010011828504502773,
+            0.024184048175811768,
+            0.025669902563095093,
+            -0.002361447084695101,
+            -0.031217092648148537,
+            0.01790100708603859,
+            0.024198200553655624,
+            0.003654494183138013,
+            -0.023646311834454536,
+            0.0302831269800663,
+            -0.008936353027820587,
+            0.014971751719713211,
+            0.026136886328458786,
+            0.014094389975070953,
+            0.004595535341650248,
+            -0.010500037111341953,
+            -0.02114158496260643,
+            0.01481609046459198,
+            0.010507113300263882,
+            0.033537857234478,
+            0.014745336025953293,
+            0.02071705460548401,
+            0.01594817079603672,
+            -0.023320838809013367,
+            -0.011214662343263626,
+            -0.003482913365587592,
+            -0.022514231503009796,
+            0.03093407303094864,
+            0.0006129149114713073,
+            0.02114158496260643,
+            0.026618018746376038,
+            -0.03319823369383812,
+            -0.01187268365174532,
+            0.032688796520233154,
+            0.02779255248606205,
+            0.05108508840203285,
+            0.025316128507256508,
+            0.015396281145513058,
+            0.027226513251662254,
+            -0.03585861995816231,
+            -0.036368053406476974,
+            0.001548649393953383,
+            0.04256619140505791,
+            -0.008426916785538197,
+            0.00967928022146225,
+            0.03900013864040375,
+            -0.006781863979995251,
+            -0.01611798256635666,
+            0.019457615911960602,
+            0.0026710000820457935,
+            0.002448122017085552,
+            -0.0033113325480371714,
+            -0.01335146278142929,
+            0.03175482898950577,
+            -0.0051474240608513355,
+            0.013570803217589855,
+            -0.019273653626441956,
+            0.027113303542137146,
+            0.0001799829478841275,
+            -0.003229964291676879,
+            0.028344441205263138,
+            -0.002660386962816119,
+            -0.051990751177072525,
+            0.024212351068854332,
+            0.019811391830444336,
+            -0.011561362072825432,
+            -0.0036332677118480206,
+            -0.003792466362938285,
+            0.04567940905690193,
+            0.034698236733675,
+            0.029009537771344185,
+            -0.0020501252729445696,
+            0.0168679840862751,
+            0.018792519345879555,
+            0.0053667644970119,
+            0.014278353191912174,
+            -0.0017025414854288101,
+            -0.007853801362216473,
+            0.021056678146123886,
+            -0.020787810906767845,
+            0.03226426616311073,
+            -0.016570813953876495,
+            -0.020122714340686798,
+            0.008738238364458084,
+            -0.011172209866344929,
+            -0.0009083169279620051,
+            -0.006311343517154455,
+            -0.0008813415770418942,
+            -0.0029664020985364914,
+            0.016938740387558937,
+            -0.004446949809789658,
+            0.04661337286233902,
+            -0.013931653462350368,
+            -0.008808993734419346,
+            0.006972902454435825,
+            -0.025981225073337555,
+            0.0013346156338229775,
+            0.0028726516757160425,
+            0.027000095695257187,
+            -0.01989629678428173,
+            -0.00970758218318224,
+            0.01606137864291668,
+            0.06039644405245781,
+            -0.008639181964099407,
+            -0.0026267783250659704,
+            0.027226513251662254,
+            -0.002787745790556073,
+            -0.03551899641752243,
+            0.012919857166707516,
+            -0.04494355618953705,
+            0.017589686438441277,
+            0.026349151507019997,
+            -0.04986810311675072,
+            -0.022089701145887375,
+            -0.02347649820148945,
+            -0.02043403498828411,
+            0.023434046655893326,
+            -0.009976450353860855,
+            0.013867974281311035,
+            -0.017801949754357338,
+            -0.009014182724058628,
+            0.020518941804766655,
+            0.009176919236779213,
+            0.03735862299799919,
+            -0.027551984414458275,
+            -0.016783079132437706,
+            -0.027820853516459465,
+            -0.04304732382297516,
+            0.009558996185660362,
+            -0.0278916098177433,
+            0.0027099153958261013,
+            -0.0016291331266984344,
+            -0.040302030742168427,
+            -0.037981268018484116,
+            0.005278320983052254,
+            -0.017985913902521133,
+            0.01516986545175314,
+            0.020915169268846512,
+            0.02731141820549965,
+            -0.0020253609400242567,
+            -0.030198222026228905,
+            0.029235953465104103,
+            -0.007344366051256657,
+            0.03223596513271332,
+            -0.00689507182687521,
+            0.005066055804491043,
+            -0.0070436573587358,
+            0.031160488724708557,
+            -0.023009516298770905,
+            0.018792519345879555,
+            -0.026801982894539833,
+            -0.004086099565029144,
+            -0.018990634009242058,
+            -0.01774534583091736,
+            0.004224071744829416,
+            0.004521242808550596,
+            7.888073741924018e-05,
+            0.0048750173300504684,
+            0.005777143407613039,
+            0.024806693196296692,
+            0.0021421066485345364,
+            0.02589631825685501,
+            -0.027877457439899445,
+            -0.025316128507256508,
+            0.01719345711171627,
+            -0.02077365852892399,
+            0.005678086541593075,
+            -0.006261814851313829,
+            -0.0302831269800663,
+            -0.01202834490686655,
+            0.023391593247652054,
+            0.02162271924316883,
+            -0.005490585695952177,
+            0.002644466934725642,
+            0.02206140011548996,
+            -0.026023678481578827,
+            0.00437973253428936,
+            -0.03781145438551903,
+            -0.018396291881799698,
+            0.029773691669106483,
+            -0.03195294365286827,
+            0.020745357498526573,
+            0.003185742534697056,
+            0.03483974561095238,
+            -0.017292514443397522,
+            -0.01947176828980446,
+            0.01654251106083393,
+            -0.01008258294314146,
+            0.004142703488469124,
+            0.01606137864291668,
+            0.01616043597459793,
+            -0.0015070808585733175,
+            0.05380208045244217,
+            0.0008167776977643371,
+            0.006909222807735205,
+            -0.049075648188591,
+            0.004496478475630283,
+            -0.044858649373054504,
+            0.0002768288250081241,
+            -0.013337312266230583,
+            -0.02514631673693657,
+            0.0014027173165231943,
+            0.001001182827167213,
+            0.001963450340554118,
+            -0.01913214474916458,
+            -0.01989629678428173,
+            0.011596739292144775,
+            -0.015863263979554176,
+            0.004928083624690771,
+            -0.06418890506029129,
+            -0.01855195313692093,
+            -0.004974074196070433,
+            0.0043337419629096985,
+            0.0007699025445617735,
+            0.03342464938759804,
+            -0.01478778850287199,
+            0.010117961093783379,
+            -0.01549533847719431,
+            -0.026363302022218704,
+            0.04010391607880592,
+            0.018750067800283432,
+            -0.023773670196533203,
+            0.024693485349416733,
+            0.012106175534427166,
+            0.03099067695438862,
+            -0.0318114347755909,
+            0.020349130034446716,
+            0.018523650243878365,
+            -0.0029823218937963247,
+            0.006551910191774368,
+            0.008575502783060074,
+            0.01832553744316101,
+            -0.015764206647872925,
+            -0.0003632825391832739,
+            0.011419852264225483,
+            0.010033054277300835,
+            0.031924642622470856,
+            0.01403071079403162,
+            -0.03183973580598831,
+            0.019047237932682037,
+            0.01616043597459793,
+            0.02298121340572834,
+            0.033820874989032745,
+            -0.027481229975819588,
+            0.038575608283281326,
+            -0.010386829264461994,
+            -0.024042539298534393,
+            0.020801961421966553,
+            -0.032688796520233154,
+            -0.006679269019514322,
+            0.030735958367586136,
+            0.04398128762841225,
+            -0.0015566092915832996,
+            -0.044632233679294586,
+            0.01535382866859436,
+            0.01316749956458807,
+            0.0027806703001260757,
+            0.028811423107981682,
+            0.011264191009104252,
+            0.021820832043886185,
+            0.025273675099015236,
+            0.011419852264225483,
+            -0.0037252490874379873,
+            0.024962352588772774,
+            -0.008101443760097027,
+            -0.004446949809789658,
+            -0.0007035697344690561,
+            0.03206615149974823,
+            0.014702882617712021,
+            -0.017603836953639984,
+            -0.036962397396564484,
+            -0.007620310410857201,
+            0.009452863596379757,
+            -0.007903330028057098,
+            -0.027127455919981003,
+            -0.03950957581400871,
+            -0.026787832379341125,
+            -0.005667473189532757,
+            0.01505665760487318,
+            -0.06724552065134048,
+            -0.007620310410857201,
+            -0.012233533896505833,
+            0.017717044800519943,
+            0.0022906921803951263,
+            0.0181132722645998,
+            0.001277127186767757,
+            -0.009403335861861706,
+            0.004209920763969421,
+            0.050802066922187805,
+            -0.03059444949030876,
+            0.014242975041270256,
+            0.011292492970824242,
+            0.0032105068676173687,
+            0.002787745790556073,
+            0.038688816130161285,
+            -0.010988246649503708,
+            -0.030424637719988823,
+            0.011575513519346714,
+            -0.017292514443397522,
+            0.003383856499567628,
+            0.05006621778011322,
+            -0.009481165558099747,
+            -0.027495380491018295,
+            0.020915169268846512,
+            -0.009148617275059223,
+            -0.015523640438914299,
+            -0.0027258351910859346,
+            -0.006035399157553911,
+            0.0006283925613388419,
+            -0.03832089155912399,
+            -0.015750056132674217,
+            0.03212275728583336,
+            0.008617955259978771,
+            0.014603826217353344,
+            0.025344429537653923,
+            0.010209942236542702,
+            -0.030481241643428802,
+            0.027481229975819588,
+            -0.0005678086308762431,
+            0.005221716593950987,
+            0.013323160819709301,
+            0.03424540534615517,
+            -0.014264201745390892,
+            -0.020264223217964172,
+            -0.040245428681373596,
+            -0.013853822834789753,
+            -0.0463869571685791,
+            0.001943992800079286,
+            0.006919836159795523,
+            -0.012919857166707516,
+            -0.04366996884346008,
+            -0.0329718142747879,
+            0.021127434447407722,
+            0.03178313374519348,
+            -0.017023645341396332,
+            0.0025294902734458447,
+            -0.008483520708978176,
+            -0.007882103323936462,
+            -0.0017520699184387922,
+            -0.013217028230428696,
+            -0.006378560326993465,
+            -0.006229975260794163,
+            0.015452885068953037,
+            -0.004652139265090227,
+            -0.0036934094969183207,
+            -0.0498398020863533,
+            -0.018778368830680847,
+            0.004082561936229467,
+            -0.008221727795898914,
+            0.007323139347136021,
+            -0.006955213379114866,
+            -0.002727604005485773,
+            -0.012474101036787033,
+            -0.03829259052872658,
+            0.024184048175811768,
+            0.017278363928198814,
+            0.0035501306410878897,
+            -0.008922201581299305,
+            -0.018367990851402283,
+            0.03294351324439049,
+            0.012523629702627659,
+            -0.017547233030200005,
+            -0.018580254167318344,
+            0.025981225073337555,
+            0.026957644149661064,
+            0.008122670464217663,
+            -0.006137993652373552,
+            0.029886899515986443,
+            0.005695775151252747,
+            -0.022938761860132217,
+            -0.003983504604548216,
+            -0.005055442452430725,
+            -0.0038844477385282516,
+            0.020561393350362778,
+            -0.010556641966104507,
+            0.0300001073628664,
+            0.038717120885849,
+            0.012870329432189465,
+            0.0039198254235088825,
+            -0.0033714743331074715,
+            -0.008922201581299305,
+            0.015976471826434135,
+            0.006481155287474394,
+            0.008525974117219448,
+            0.014759486541152,
+            -0.016358548775315285,
+            -0.020405733957886696,
+            0.006215824279934168,
+            -0.007107336539775133,
+            0.020122714340686798,
+            0.01708024926483631,
+            0.016429303213953972,
+            0.0007610581233166158,
+            -0.028287837281823158,
+            -0.0018024828750640154,
+            0.02411329373717308,
+            0.02010856196284294,
+            -0.004655676893889904,
+            -0.023264234885573387,
+            0.014398636296391487,
+            -0.015806660056114197,
+            -0.009969375096261501,
+            0.0033431723713874817,
+            0.006229975260794163,
+            -0.003083147807046771,
+            0.0571134127676487,
+            -0.020193468779325485,
+            -0.011419852264225483,
+            -0.010287772864103317,
+            -0.010867963545024395,
+            0.0012072566896677017,
+            -0.01828308403491974,
+            0.008964654989540577,
+            0.025613298639655113,
+            -0.009764186106622219,
+            0.03568880632519722,
+            0.0228114016354084,
+            0.006912760436534882,
+            0.005260631907731295,
+            -0.027070851996541023,
+            0.012304289266467094,
+            -0.016047226265072823,
+            0.005660397466272116,
+            0.018467046320438385,
+            0.017320817336440086,
+            0.0076132346875965595,
+            0.026306698098778725,
+            0.019386861473321915,
+            0.016684021800756454,
+            -0.022783100605010986,
+            0.025457637384533882,
+            -0.007832575589418411,
+            0.027877457439899445,
+            -0.01611798256635666,
+            -0.002253545681014657,
+            -0.012990612536668777,
+            0.008681634441018105,
+            0.002416282193735242,
+            -0.0019351483788341284,
+            0.00036129256477579474,
+            0.016783079132437706,
+            -0.014589674770832062,
+            0.01062032114714384,
+            0.013316085562109947,
+            -0.027919910848140717,
+            -0.010372678749263287,
+            0.016344398260116577,
+            -0.00668988237157464,
+            0.006877383217215538,
+            -0.011865608394145966,
+            -0.0017202302115038037,
+            0.011009473353624344,
+            -0.0436982698738575,
+            0.0104929618537426,
+            0.006279503460973501,
+            -0.015155714936554432,
+            0.014801939949393272,
+            -0.009799563325941563,
+            -0.008235878311097622,
+            -0.013592029921710491,
+            0.003792466362938285,
+            -0.00030225637601688504,
+            -0.0018290160223841667,
+            -0.00022995365725364536,
+            0.00389859895221889,
+            0.010047205723822117,
+            -0.0045990729704499245,
+            -0.010450509376823902,
+            -0.0028638073708862066,
+            -0.02265574224293232,
+            -0.0010047205723822117,
+            0.018735915422439575,
+            -0.004591997712850571,
+            0.0005222601466812193,
+            0.0145189194008708,
+            -0.005830209702253342,
+            0.010556641966104507,
+            -0.005221716593950987,
+            -0.008320784196257591,
+            -0.010768906213343143,
+            -0.033764272928237915,
+            0.0026710000820457935,
+            -0.015254771336913109,
+            0.008865597657859325,
+            -0.012544856406748295,
+            0.04027372971177101,
+            0.01022409275174141,
+            -0.00837738811969757,
+            -0.002796590095385909,
+            -0.0228114016354084,
+            -0.004475251771509647,
+            -0.016938740387558937,
+            -0.01184438169002533,
+            0.008342010900378227,
+            0.012629762291908264,
+            0.005614406894892454,
+            -0.018410442396998405,
+            0.003665107535198331,
+            0.0030955299735069275,
+            -0.005575491581112146,
+            -0.008370312862098217,
+            0.010853812098503113,
+            -0.0006248548161238432,
+            -0.01800006441771984,
+            -0.01957082375884056,
+            -0.027424626052379608,
+            0.0150000536814332,
+            0.014589674770832062,
+            -0.01011088490486145,
+            -0.022754797711968422,
+            -0.007471724878996611,
+            0.007107336539775133,
+            -0.02064630016684532,
+            0.0006779210525564849,
+            0.006891534198075533,
+            0.008717012591660023,
+            0.004117939155548811,
+            0.009636826813220978,
+            0.0007699025445617735,
+            0.0121698547154665,
+            0.018566103652119637,
+            0.03271709755063057,
+            -0.0396793857216835,
+            -0.009537769481539726,
+            0.0057028504088521,
+            -0.001465512323193252,
+            -0.013853822834789753,
+            0.009672204032540321,
+            0.0024375086650252342,
+            -0.0026426981203258038,
+            0.024424616247415543,
+            -0.008044839836657047,
+            -0.01505665760487318,
+            0.01184438169002533,
+            0.00887267291545868,
+            -0.019075538963079453,
+            0.02746707946062088,
+            0.006169833242893219,
+            -0.01011088490486145,
+            0.02259913645684719,
+            -0.012360893189907074,
+            0.012658064253628254,
+            -0.005522425286471844,
+            0.023660462349653244,
+            -0.019004784524440765,
+            0.0006306036957539618,
+            0.015665151178836823,
+            0.03883032873272896,
+            -0.00437973253428936,
+            -0.004567233379930258,
+            -0.03421710431575775,
+            0.005115584470331669,
+            0.028853876516222954,
+            0.02666047215461731,
+            -0.0012346742441877723,
+            -0.005596718285232782,
+            -0.0019864458590745926,
+            0.010768906213343143,
+            0.011589664034545422,
+            0.017561383545398712,
+            -0.029547275975346565,
+            0.00022011429246049374,
+            -0.016938740387558937,
+            0.025500090792775154,
+            0.014632128179073334,
+            0.006951675750315189,
+            0.04072656109929085,
+            0.0027789014857262373,
+            0.003841994795948267,
+            0.027452928945422173,
+            -0.00779719790443778,
+            0.003546592779457569,
+            -0.015566092915832996,
+            0.0003175129240844399,
+            0.022853855043649673,
+            0.0016370931407436728,
+            0.0020784272346645594,
+            0.013337312266230583,
+            0.029320860281586647,
+            -0.022471778094768524,
+            -0.023929331451654434,
+            0.003482913365587592,
+            0.007185167167335749,
+            -0.010570792481303215,
+            0.0038738346192985773,
+            0.006275965832173824,
+            -0.006007096730172634,
+            0.008900974877178669,
+            0.006332569755613804,
+            -0.0003166284877806902,
+            0.0039198254235088825,
+            -0.00308845448307693,
+            0.012962310574948788,
+            -0.02702839858829975,
+            0.036849189549684525,
+            0.004439874552190304,
+            0.010075507685542107,
+            0.015268922783434391,
+            0.001627364312298596,
+            0.004917470272630453,
+            0.005869124550372362,
+            0.010188715532422066,
+            0.018636859953403473,
+            -0.00956607237458229,
+            -0.027481229975819588,
+            -0.03166992589831352,
+            0.020363280549645424,
+            -0.01173117384314537,
+            0.0035872769076377153,
+            0.0074646491557359695,
+            -0.01828308403491974,
+            -0.007507102098315954,
+            -0.009976450353860855,
+            0.015764206647872925,
+            -0.03189634159207344,
+            0.012990612536668777,
+            -0.0043231286108493805,
+            -0.028669914230704308,
+            0.00851889792829752,
+            -0.010181640274822712,
+            -0.0045884596183896065,
+            -0.0035094465129077435,
+            -0.029264256358146667,
+            -0.007931631989777088,
+            0.03908504545688629,
+            -0.02141045406460762,
+            0.024184048175811768,
+            0.008257105015218258,
+            -0.00014449491573031992,
+            -0.0040365708991885185,
+            -0.013563727959990501,
+            0.005331386812031269,
+            -0.01778779923915863,
+            0.025457637384533882,
+            0.010294848121702671,
+            -0.013924578204751015,
+            0.0322076641023159,
+            -0.011179285123944283,
+            0.027438776567578316,
+            -0.018580254167318344,
+            0.036198243498802185,
+            0.007422196213155985,
+            0.00674648629501462,
+            0.01974063739180565,
+            -0.004025958012789488,
+            0.008228803053498268,
+            0.04559450224041939,
+            -0.018523650243878365,
+            0.02460857853293419,
+            0.020349130034446716,
+            -0.00022685811563860625,
+            0.017915157601237297,
+            0.0028390432707965374,
+            -0.023165177553892136,
+            0.009495317004621029,
+            -0.01135617308318615,
+            -0.009955224581062794,
+            0.01583496294915676,
+            -0.019316107034683228,
+            -0.018721764907240868,
+            -0.025613298639655113,
+            0.0029416377656161785,
+            -0.003350247861817479,
+            -0.0077618202194571495,
+            -0.0019616815261542797,
+            0.01454722136259079,
+            0.008724087849259377,
+            0.025669902563095093,
+            0.0008326975512318313,
+            0.011971740983426571,
+            0.01511326152831316,
+            -0.00035576484515331686,
+            0.02313687466084957,
+            -0.03229256719350815,
+            0.01246702577918768,
+            -0.004517704714089632,
+            -0.00509789539501071,
+            0.023858575150370598,
+            0.0003747802402358502,
+            -0.0004877670726273209,
+            -0.0024888061452656984,
+            0.0005496777012012899,
+            -0.014589674770832062,
+            -0.0057099261321127415,
+            0.000907432462554425,
+            -0.005175726022571325,
+            0.04299071803689003,
+            -0.00701535539701581,
+            -0.004772422835230827,
+            -0.039169952273368835,
+            -0.01918874867260456,
+            0.00623351288959384,
+            -0.01583496294915676,
+            -0.018778368830680847,
+            -0.006307805422693491,
+            0.005801907740533352,
+            -0.04893413558602333,
+            0.008342010900378227,
+            0.03336804360151291,
+            -0.012219383381307125,
+            0.00259493850171566,
+            -0.010981171391904354,
+            -0.006969364359974861,
+            -0.0029115669894963503,
+            -7.766463386360556e-05,
+            -0.002412744564935565,
+            -0.020830262452363968,
+            -0.005617944523692131,
+            0.003364398842677474,
+            -0.0008795727044343948,
+            -0.025047259405255318,
+            0.0006177793256938457,
+            0.018849123269319534,
+            -0.003979966975748539,
+            -0.0011152751976624131,
+            -0.001825478277169168,
+            0.006399787031114101,
+            -0.01508495956659317,
+            0.008009462617337704,
+            -0.0076344613917171955,
+            0.003841994795948267,
+            -0.0030495391692966223,
+            -0.002214630600064993,
+            0.02844349853694439,
+            0.021594416350126266,
+            -0.04808507859706879,
+            -0.01405901275575161,
+            0.023009516298770905,
+            0.007280686404556036,
+            -0.004740582779049873,
+            0.01178777776658535,
+            -0.031613320112228394,
+            -0.00028943203506059945,
+            -0.026787832379341125,
+            0.02200479619204998,
+            -0.009205221198499203,
+            -0.007386818993836641,
+            0.016967041417956352,
+            -0.026632171124219894,
+            0.016344398260116577,
+            0.00794578343629837,
+            -0.00030181417241692543,
+            -0.000493515923153609,
+            0.0063856360502541065,
+            0.0008972614887170494,
+            -0.01659911684691906,
+            0.006442239973694086,
+            -0.0048502529971301556,
+            0.007959933951497078,
+            -0.005087282508611679,
+            -0.05527378246188164,
+            -0.009933997876942158,
+            -0.005497660953551531,
+            -0.008936353027820587,
+            -0.012820800766348839,
+            0.008136821910738945,
+            0.02827368676662445,
+            0.01606137864291668,
+            -0.009863242506980896,
+            -4.8312376748071983e-05,
+            0.033000119030475616,
+            -0.0008300442132167518,
+            0.009955224581062794,
+            -0.011349096894264221,
+            0.00015709814033471048,
+            -0.0006911876262165606,
+            -0.014660430140793324,
+            -0.019061388447880745,
+            0.024042539298534393,
+            -0.028344441205263138,
+            0.02378782071173191,
+            -0.01843874529004097,
+            -0.0130825936794281,
+            0.01757553406059742,
+            -0.032575588673353195,
+            -0.005108508747071028,
+            0.007450498174875975,
+            0.0090566361322999,
+            0.021608568727970123,
+            0.0032794929575175047,
+            0.020589696243405342,
+            -0.03716050833463669,
+            -0.009134466759860516,
+            -0.010160413570702076,
+            -0.00794578343629837,
+            -0.010231168940663338,
+            -0.011596739292144775,
+            0.021863285452127457,
+            -0.015240620821714401,
+            -0.016952890902757645,
+            0.007054270710796118,
+            0.0030212372075766325,
+            -0.011313719674944878,
+            0.015750056132674217,
+            0.00017810352437663823,
+            0.004712280817329884,
+            0.013011839240789413,
+            -0.0008923970744945109,
+            0.04086806997656822,
+            0.0253019779920578,
+            -0.0030212372075766325,
+            -0.008568426594138145,
+            0.021438755095005035,
+            0.0004285097820684314,
+            0.0150000536814332,
+            0.009375033900141716,
+            0.023264234885573387,
+            -0.02092931978404522,
+            0.009997677057981491,
+            -0.005501199048012495,
+            0.014603826217353344,
+            -0.009495317004621029,
+            -0.015339677222073078,
+            0.00479718716815114,
+            -0.01577835902571678,
+            0.010153338313102722,
+            -0.010860888287425041,
+            0.008221727795898914,
+            0.01913214474916458,
+            0.002467579673975706,
+            -0.00941041111946106,
+            0.01146230474114418,
+            0.02075950801372528,
+            -0.025641601532697678,
+            0.008851447142660618,
+            -0.0031998935155570507,
+            -0.0021916350815445185,
+            0.021127434447407722,
+            0.010301923379302025,
+            -0.01594817079603672,
+            0.01573590561747551,
+            -0.008879749104380608,
+            -0.0015053119277581573,
+            -0.01882082223892212,
+            -0.03291521221399307,
+            -0.0015283073298633099,
+            0.012396270409226418,
+            0.004800724796950817,
+            0.0150000536814332,
+            0.011264191009104252,
+            -0.0008950503543019295,
+            -0.006601438857614994,
+            0.0008433107868768275,
+            -0.013485897332429886,
+            -0.004963461309671402,
+            0.010231168940663338,
+            0.017051948234438896,
+            -0.0037217114586383104,
+            0.015650998800992966,
+            0.0088160689920187,
+            0.006137993652373552,
+            -0.0010480580385774374,
+            0.0036138100549578667,
+            -0.002060738392174244,
+            -0.007783046457916498,
+            0.019938750192523003,
+            0.019075538963079453,
+            -0.018509499728679657,
+            -0.0024870370980352163,
+            -0.02141045406460762,
+            0.025882167741656303,
+            0.022188758477568626,
+            0.011165134608745575,
+            -0.04222656786441803,
+            0.00012050455552525818,
+            0.00752832880243659,
+            -0.0013744152383878827,
+            -0.031103884801268578,
+            0.008172199130058289,
+            0.005221716593950987,
+            0.004330204334110022,
+            0.020844414830207825,
+            -0.018424594774842262,
+            0.010853812098503113,
+            0.0070861103013157845,
+            -0.0005766529939137399,
+            0.01019579079002142,
+            -0.0073372903279960155,
+            0.008688710629940033,
+            0.038717120885849,
+            -0.018410442396998405,
+            0.005303084850311279,
+            0.0062370505183935165,
+            0.023320838809013367,
+            -0.011228813789784908,
+            -0.00031485961517319083,
+            0.00800238735973835,
+            0.021608568727970123,
+            -0.012934008613228798,
+            0.005883275531232357,
+            0.067075714468956,
+            -0.012912781909108162,
+            0.031613320112228394,
+            0.020533092319965363,
+            -0.004970536567270756,
+            -0.0104929618537426,
+            -0.012445799075067043,
+            -0.0181132722645998,
+            0.021283095702528954,
+            0.01175947580486536,
+            0.003419233951717615,
+            0.01941516436636448,
+            -0.01610383205115795,
+            0.006636816076934338,
+            0.017547233030200005,
+            0.012601460330188274,
+            -0.007379743270576,
+            -0.0036173479165881872,
+            0.019004784524440765,
+            0.018509499728679657,
+            -0.0030849166214466095,
+            0.00019424450874794275,
+            0.01508495956659317,
+            -0.022075550630688667,
+            -0.012530704960227013,
+            0.01644345559179783,
+            -0.0037358624394983053,
+            0.017773648723959923,
+            -0.02227366529405117,
+            0.024198200553655624,
+            -0.004531855694949627,
+            -0.017179306596517563,
+            0.004698129836469889,
+            -0.007514177821576595,
+            0.022033097222447395,
+            -0.001578720286488533,
+            0.004857328720390797,
+            0.000821199850179255,
+            -0.0018015984678640962,
+            0.01349297259002924,
+            0.002826661104336381,
+            -0.0005120891146361828,
+            0.010507113300263882,
+            -0.0009702275274321437,
+            -0.0013496510218828917,
+            0.024509521201252937,
+            1.9374700059415773e-05,
+            0.005989408120512962,
+            0.0032936439383774996,
+            -0.009261825121939182,
+            -0.004429261200129986,
+            -0.004443412180989981,
+            0.0011842612875625491,
+            0.003877372248098254,
+            -0.007029506377875805,
+            -0.02384442463517189,
+            0.022471778094768524,
+            -0.008540124632418156,
+            0.01175947580486536,
+            0.011349096894264221,
+            0.003665107535198331,
+            0.006824316922575235,
+            -0.009771261364221573,
+            -0.017320817336440086,
+            0.0006633278680965304,
+            -0.01617458648979664,
+            0.006912760436534882,
+            -0.001656550681218505,
+            0.0020235921256244183,
+            0.009445788338780403,
+            -0.002570174168795347,
+            0.0015123875346034765,
+            0.011306644417345524,
+            0.011497682891786098,
+            0.003254728624597192,
+            0.004577846731990576,
+            -0.018721764907240868,
+            -0.020915169268846512,
+            0.005288933869451284,
+            0.013924578204751015,
+            0.004273600410670042,
+            0.0015309606678783894,
+            0.017603836953639984,
+            -0.006166295614093542,
+            -0.022967062890529633,
+            -0.010117961093783379,
+            -0.01584911346435547,
+            0.039396367967128754,
+            0.01800006441771984,
+            -0.02525952458381653,
+            -0.000530220044311136,
+            -0.009728807955980301,
+            0.010450509376823902,
+            -0.00404364662244916,
+            0.014575524255633354,
+            0.005476434715092182,
+            -0.02426895499229431,
+            -0.016797229647636414,
+            -0.010662773624062538,
+            -0.005257094278931618,
+            0.020080260932445526,
+            0.027014248073101044,
+            -0.004956385586410761,
+            0.0061026159673929214,
+            0.019584976136684418,
+            -0.010938717983663082,
+            0.009849091991782188,
+            0.020957622677087784,
+            0.0031698227394372225,
+            -0.007570781745016575,
+            -0.004553082399070263,
+            -0.016514210030436516,
+            0.024467067793011665,
+            0.014250051230192184,
+            -0.005104971118271351,
+            -0.02465103194117546,
+            0.004110863897949457,
+            -0.02428310550749302,
+            -0.01616043597459793,
+            -0.007301913108676672,
+            -0.014143918640911579,
+            -0.004931621253490448,
+            0.0027594438288360834,
+            -0.004836102016270161,
+            0.011278342455625534,
+            -0.0031592093873769045,
+            -0.004531855694949627,
+            -0.0041851564310491085,
+            -0.025584997609257698,
+            -0.02524537406861782,
+            0.007691065315157175,
+            -0.003005317412316799,
+            0.005596718285232782,
+            0.000650945701636374,
+            -0.013400991447269917,
+            -0.011589664034545422,
+            0.00970758218318224,
+            -0.007988235913217068,
+            0.012813725508749485,
+            0.0031149876303970814,
+            -0.009183995425701141,
+            0.0017193458043038845,
+            -0.0012311364989727736,
+            -0.0059752571396529675,
+            0.019924599677324295,
+            0.00916984397917986,
+            0.011724098585546017,
+            -0.03118879161775112,
+            -0.0021633331198245287,
+            0.008349086157977581,
+            0.01828308403491974,
+            -0.02244347706437111,
+            -0.008313708938658237,
+            0.005179263651371002,
+            -0.011228813789784908,
+            -0.013507124036550522,
+            -0.028952933847904205,
+            -0.0030460015404969454,
+            8.175515540642664e-05,
+            -0.018636859953403473,
+            -0.029886899515986443,
+            0.0009038947173394263,
+            -0.007860877551138401,
+            0.016882136464118958,
+            -0.006643891800194979,
+            0.01790100708603859,
+            0.00870286114513874,
+            0.0036173479165881872,
+            0.011554286815226078,
+            0.0025082635693252087,
+            -0.013266556896269321,
+            0.008426916785538197,
+            -0.02671707607805729,
+            -0.01516986545175314,
+            0.009792488068342209,
+            0.014214673079550266,
+            -0.0026126273442059755,
+            5.074458022136241e-05,
+            0.0058372849598526955,
+            0.024254804477095604,
+            0.013584954664111137,
+            0.012474101036787033,
+            0.003042463678866625,
+            -0.005303084850311279,
+            0.005568416323512793,
+            0.0227123461663723,
+            0.008603804744780064,
+            0.004011806566268206,
+            0.007592008449137211,
+            0.00389859895221889,
+            0.0009932229295372963,
+            -0.005943417549133301,
+            -0.00843399204313755,
+            -0.021537812426686287,
+            -0.008908051066100597,
+            0.019938750192523003,
+            -0.010422207415103912,
+            -0.03005671128630638,
+            0.0006023016758263111,
+            -0.0032777239102870226,
+            0.007988235913217068,
+            0.02595292218029499,
+            -0.014172220602631569,
+            0.004970536567270756,
+            0.005536576267331839,
+            -0.007910405285656452,
+            -0.00614860700443387,
+            0.022967062890529633,
+            0.02639160305261612,
+            -0.0034404604230076075,
+            -0.04800017178058624,
+            -0.01671232469379902,
+            0.004305440001189709,
+            -0.007768895477056503,
+            -0.034047290682792664,
+            -0.015424583107233047,
+            0.002757675014436245,
+            -0.03206615149974823,
+            0.010068432427942753,
+            -0.012948159128427505,
+            -0.01214155275374651,
+            0.020264223217964172,
+            0.03223596513271332,
+            0.005154499784111977,
+            0.00014847487909719348,
+            -0.0214670579880476,
+            -0.0021686397958546877,
+            -0.021382151171565056,
+            0.010174565017223358,
+            -0.008801918476819992,
+            -0.010584943927824497,
+            0.0059611061587929726,
+            -0.012191081419587135,
+            0.013266556896269321,
+            0.010797208175063133,
+            0.024778390303254128,
+            0.01584911346435547,
+            -0.01876421831548214,
+            -0.0024746551644057035,
+            -0.01108730398118496,
+            -0.0016450530383735895,
+            -0.009820790030062199,
+            0.006360871717333794,
+            -0.0031592093873769045,
+            0.011264191009104252,
+            0.015184016898274422,
+            0.004432798828929663,
+            0.00046786724124103785,
+            0.038151077926158905,
+            -2.4487851987942122e-05,
+            -0.0003418349369894713,
+            0.006803090218454599,
+            -0.015750056132674217,
+            0.008837295696139336,
+            0.012191081419587135,
+            0.005051904823631048,
+            -0.007698140572756529,
+            0.006739411037415266,
+            0.03099067695438862,
+            -0.01516986545175314,
+            -0.009014182724058628,
+            -0.023504801094532013,
+            0.007549555506557226,
+            -0.0004656561359297484,
+            0.029971804469823837,
+            0.020533092319965363,
+            -0.027509532868862152,
+            -0.016018925234675407,
+            -0.02179253101348877,
+            0.01267929095774889,
+            -0.011497682891786098,
+            -0.003704022616147995,
+            0.012240610085427761,
+            0.02367461286485195,
+            0.0018838511314243078,
+            -0.014455240219831467,
+            -0.008950503543019295,
+            0.008943428285419941,
+            0.014617976732552052,
+            0.003497064346447587,
+            0.007570781745016575,
+            0.00743634719401598,
+            0.008886824361979961,
+            -0.014801939949393272,
+            -0.012254760600626469,
+            0.00767691433429718,
+            -0.010202866978943348,
+            0.01794346049427986,
+            -0.009403335861861706,
+            0.016768928617239,
+            -0.004645063541829586,
+            -0.011964665725827217,
+            -0.0320095494389534,
+            -0.002292460994794965,
+            0.005037753842771053,
+            0.004627374932169914,
+            0.0034705311991274357,
+            -0.0022217060904949903,
+            -0.013160424306988716,
+            0.004673365969210863,
+            0.005073131527751684,
+            1.875283123808913e-05,
+            -0.008313708938658237,
+            -0.00479718716815114,
+            0.004252373706549406,
+            -0.001031253719702363,
+            -0.0073372903279960155,
+            -0.0059858704917132854,
+            0.018481198698282242,
+            0.0066509670577943325,
+            0.00800238735973835,
+            0.03234917297959328,
+            0.02974538877606392,
+            -0.0016353242099285126,
+            -0.004606148693710566,
+            -0.00023304918431676924,
+            -0.01589156687259674,
+            -0.0057205394841730595,
+            -0.023320838809013367,
+            0.022259512916207314,
+            -0.009665128774940968,
+            0.021820832043886185,
+            0.019457615911960602,
+            0.0039304387755692005,
+            -0.008391539566218853,
+            -0.0011586126638576388,
+            -0.007308988366276026,
+            -0.003560743760317564,
+            0.016641568392515182,
+            0.006742948666214943,
+            0.01226891204714775,
+            0.00030933189555071294,
+            -0.028401045128703117,
+            0.026405755430459976,
+            0.000579748535528779,
+            0.0034015451092272997,
+            0.008908051066100597,
+            0.014129767194390297,
+            0.013209952972829342,
+            -0.01235381793230772,
+            0.01768874190747738,
+            -0.03280200436711311,
+            -0.0033484790474176407,
+            -0.016500059515237808,
+            0.002076658420264721,
+            -0.0018520113080739975,
+            -0.0008145665633492172,
+            -0.05354736000299454,
+            0.0003842879377771169,
+            0.005313698202371597,
+            -0.015339677222073078,
+            0.029349161311984062,
+            -0.020589696243405342,
+            -0.017165156081318855,
+            -0.032858606427907944,
+            -0.010238244198262691,
+            0.018099121749401093,
+            0.005617944523692131,
+            0.00035178486723452806,
+            -0.02660386823117733,
+            0.015750056132674217,
+            -0.0047476585023105145,
+            -0.00743634719401598,
+            -0.029858596622943878,
+            -0.010938717983663082,
+            0.009403335861861706,
+            0.004485865123569965,
+            -0.0130825936794281,
+            0.009898620657622814,
+            -0.002890340518206358,
+            0.010867963545024395,
+            -0.0138467475771904,
+            -0.00420284504070878,
+            0.0010020672343671322,
+            -0.005363226868212223,
+            0.012615610845386982,
+            -0.030537845566868782,
+            0.011391550302505493,
+            -0.01352127455174923,
+            0.0008548084879294038,
+            -0.017858553677797318,
+            -0.027707645669579506,
+            0.0020288988016545773,
+            -0.016132133081555367,
+            0.039990708231925964,
+            0.010188715532422066,
+            -0.01481609046459198,
+            0.016995344310998917,
+            -0.002080196049064398,
+            0.017377421259880066,
+            0.0014955832157284021,
+            0.017023645341396332,
+            -0.011971740983426571,
+            0.021056678146123886,
+            -0.0041745430789887905,
+            0.02156611531972885,
+            0.00600002147257328,
+            0.034047290682792664,
+            -0.02033497765660286,
+            0.0070825726725161076,
+            0.003792466362938285,
+            0.005798369646072388,
+            -0.0026816134341061115,
+            0.021424604579806328,
+            0.017985913902521133,
+            0.019698183983564377,
+            0.0037110981065779924,
+            -0.024084990844130516,
+            -0.006137993652373552,
+            0.004867942072451115,
+            0.023858575150370598,
+            0.011150983162224293,
+            -0.016415152698755264,
+            -0.021056678146123886,
+            0.015226469375193119,
+            -0.007860877551138401,
+            0.006771250627934933,
+            -0.00851889792829752,
+            0.00037544354563578963,
+            -0.013259481638669968,
+            0.012233533896505833,
+            -0.012523629702627659,
+            0.0032335021533071995,
+            0.01043635793030262,
+            -0.027806703001260757,
+            0.01511326152831316,
+            -0.00371817359700799,
+            -0.014313730411231518,
+            0.023320838809013367,
+            -0.015877414494752884,
+            -0.013988257385790348,
+            0.005501199048012495,
+            0.008610880002379417,
+            0.01746232621371746,
+            -0.002761212643235922,
+            0.002669231267645955,
+            -0.020504789426922798,
+            -0.012445799075067043,
+            0.004591997712850571,
+            0.02210385352373123,
+            -0.015820810571312904,
+            0.012431648559868336,
+            -0.018099121749401093,
+            0.01364863384515047,
+            0.016202887520194054,
+            0.007149789482355118,
+            0.00573115237057209,
+            -0.007726442534476519,
+            -0.0011471149045974016,
+            0.032151058316230774,
+            -0.0029876285698264837,
+            0.02704254910349846,
+            -0.014915147796273232,
+            0.009615600109100342,
+            0.00518987700343132,
+            -0.013089669868350029,
+            -0.014589674770832062,
+            0.011108530685305595,
+            0.02428310550749302,
+            -0.009530694223940372,
+            -0.023660462349653244,
+            0.0018484735628589988,
+            0.014264201745390892,
+            0.010337301529943943,
+            0.009403335861861706,
+            -0.001997058978304267,
+            -0.0037358624394983053,
+            0.01484439242631197,
+            -0.0032918748911470175,
+            -0.0286557637155056,
+            -0.01633024774491787,
+            -0.004400959238409996,
+            -0.017490629106760025,
+            0.020745357498526573,
+            0.02367461286485195,
+            0.010507113300263882,
+            0.0037570889107882977,
+            0.005540114361792803,
+            0.018297234550118446,
+            0.01843874529004097,
+            0.013330236077308655,
+            0.009304278530180454,
+            0.012700516730546951,
+            -0.004319590982049704,
+            0.02800481766462326,
+            -0.010860888287425041,
+            -0.01046465989202261,
+            -0.015707602724432945,
+            0.0016636261716485023,
+            -0.005193414632230997,
+            -0.012431648559868336,
+            -0.0069481381215155125,
+            0.004369119182229042,
+            0.013330236077308655,
+            -0.012240610085427761,
+            -0.007149789482355118,
+            0.01016748882830143,
+            0.011971740983426571,
+            0.01403071079403162,
+            0.01719345711171627,
+            -0.01633024774491787,
+            -0.011278342455625534,
+            0.0009834941010922194,
+            -0.02844349853694439,
+            -0.0160047747194767,
+            -0.01767459139227867,
+            0.009580222889780998,
+            -0.014469391666352749,
+            0.03724541515111923,
+            0.028471799567341805,
+            0.0073514413088560104,
+            0.031613320112228394,
+            -0.028401045128703117,
+            0.00970758218318224,
+            -0.014398636296391487,
+            0.007132100872695446,
+            -0.004188694059848785,
+            0.003902136581018567,
+            0.002448122017085552,
+            0.020419884473085403,
+            0.00791748147457838,
+            0.014264201745390892,
+            0.008150972425937653,
+            -0.021934039890766144,
+            -0.00662620272487402,
+            -0.012559006921947002,
+            -0.016401002183556557,
+            -0.02595292218029499,
+            0.013719388283789158,
+            0.005087282508611679,
+            -0.003074303502216935,
+            -0.0002704166399780661,
+            0.006566061172634363,
+            -0.0013195802457630634,
+            0.010443433187901974,
+            -0.027608588337898254,
+            -0.01311089564114809,
+            0.0069269114173948765,
+            0.01449061743915081,
+            -0.02114158496260643,
+            0.0014637433923780918,
+            0.008900974877178669,
+            -0.004602610599249601,
+            0.035660505294799805,
+            -0.014674580655992031,
+            -0.008610880002379417,
+            0.0027806703001260757,
+            0.005062518175691366,
+            -0.017660440877079964,
+            -0.02319347858428955,
+            0.0070648835971951485,
+            0.011575513519346714,
+            0.010867963545024395,
+            -0.028570856899023056,
+            0.02087271586060524,
+            -0.0054339817725121975,
+            0.03832089155912399,
+            0.00401534466072917,
+            -0.013337312266230583,
+            0.003962278366088867,
+            0.0011860302183777094,
+            -0.012360893189907074,
+            0.002060738392174244,
+            -0.000340508297085762,
+            0.003700484987348318,
+            0.011228813789784908,
+            -0.014801939949393272,
+            -0.016967041417956352,
+            0.016910437494516373,
+            -0.011476456187665462,
+            -0.0035395172890275717,
+            -0.00943871308118105,
+            -0.016669871285557747,
+            0.007308988366276026,
+            0.025047259405255318,
+            0.0041745430789887905,
+            0.001242634141817689,
+            -0.011632117442786694,
+            -0.0038596836384385824,
+            -0.008575502783060074,
+            0.0011718792375177145,
+            -0.0207029040902853,
+            0.027396325021982193,
+            -0.01801421493291855,
+            0.002410975517705083,
+            0.021099131554365158,
+            0.008709936402738094,
+            -0.003944589756429195,
+            0.016316095367074013,
+            0.026363302022218704,
+            0.01205664686858654,
+            0.0031539027113467455,
+            -0.00902833417057991,
+            1.648646139074117e-05,
+            0.014957600273191929,
+            0.0015123875346034765,
+            -0.0015663381200283766,
+            0.0032140444964170456,
+            -0.006583749782294035,
+            -0.001859086798503995,
+            -0.017561383545398712,
+            0.005564878229051828,
+            0.02125479280948639,
+            -0.004132090136408806,
+            0.020476488396525383,
+            0.015509488992393017,
+            0.011625041253864765,
+            0.010726453736424446,
+            -0.0012390963966026902,
+            -0.002150951186195016,
+            -0.0021668709814548492,
+            -0.007393894251435995,
+            -0.018297234550118446,
+            0.006470541935414076,
+            0.0034493047278374434,
+            0.01822648011147976,
+            0.020504789426922798,
+            -0.02308027073740959,
+            0.010146263055503368,
+            0.014985902234911919,
+            -0.013011839240789413,
+            0.0235755555331707,
+            -0.02006610855460167,
+            0.0024605041835457087,
+            0.02156611531972885,
+            0.013337312266230583,
+            -0.014271277002990246,
+            -0.009219372645020485,
+            0.004662752617150545,
+            -0.01543873455375433,
+            0.000872939417604357,
+            -0.01478778850287199,
+            -0.010783057659864426,
+            0.014533070847392082,
+            -0.01490099634975195,
+            0.01229721400886774,
+            -0.006969364359974861,
+            0.0070967236533761024,
+            -0.017165156081318855,
+            0.015014204196631908,
+            0.008632106706500053,
+            0.01757553406059742,
+            0.0021279556676745415,
+            -0.02518877014517784,
+            -0.006290116813033819,
+            0.021169887855648994,
+            0.009509467519819736,
+            -0.02497650496661663,
+            0.019613277167081833,
+            0.0020094411447644234,
+            -0.019528372213244438,
+            -0.022401023656129837,
+            -0.012686366215348244,
+            -0.005055442452430725,
+            -0.017264213413000107,
+            -0.0011276573641225696,
+            -0.024792540818452835,
+            0.003972891718149185,
+            0.009106164798140526,
+            -0.011978816241025925,
+            -0.019273653626441956,
+            -0.0002315014135092497,
+            -0.01040805596858263,
+            0.006219361908733845,
+            -0.01132787112146616,
+            -0.019853845238685608,
+            0.0070436573587358,
+            0.012127402238547802,
+            -0.0027647505048662424,
+            -0.005646246485412121,
+            -0.02562745101749897,
+            -0.006919836159795523,
+            0.020179318264126778,
+            -0.003994117956608534,
+            -0.004974074196070433,
+            -0.01703779585659504,
+            0.01108730398118496,
+            0.011165134608745575,
+            0.020901018753647804,
+            0.010931642726063728,
+            0.005674548447132111,
+            0.013634482398629189,
+            -0.0029416377656161785,
+            -0.007156865205615759,
+            -9.645892532716971e-06,
+            0.004284213297069073,
+            0.0069233737885952,
+            0.0016636261716485023,
+            0.020235922187566757,
+            -0.01692458800971508,
+            0.018934030085802078,
+            -0.012290137819945812,
+            0.008794843219220638,
+            -0.01855195313692093,
+            -0.0003155229496769607,
+            -0.01067692507058382,
+            0.00015002264990471303,
+            -0.0038455326575785875,
+            -0.009544845670461655,
+            1.9595809135353193e-05,
+            0.00143986358307302,
+            0.008511822670698166,
+            -0.016783079132437706,
+            0.009615600109100342,
+            -0.010839661583304405,
+            -0.004308977629989386,
+            -0.019542522728443146,
+            0.017448175698518753,
+            0.013344387523829937,
+            0.007308988366276026,
+            -0.008278331719338894,
+            -0.006300730165094137,
+            -0.02282555401325226,
+            0.008582578040659428,
+            -0.0014637433923780918,
+            0.002214630600064993,
+            -0.028301987797021866,
+            -0.004358506295830011,
+            0.0024587351363152266,
+            0.020306676626205444,
+            0.0033661676570773125,
+            -0.010712302289903164,
+            0.008978805504739285,
+            -0.023377442732453346,
+            -0.0030159305315464735,
+            -0.023349139839410782,
+            -0.004075486212968826,
+            -0.005112046375870705,
+            0.010103809647262096,
+            0.008101443760097027,
+            0.01549533847719431,
+            -0.009898620657622814,
+            0.01478778850287199,
+            -0.004793649073690176,
+            0.012049571610987186,
+            -0.0012152165872976184,
+            0.01337976474314928,
+            -0.0003111007681582123,
+            -0.01505665760487318,
+            0.02158026583492756,
+            -0.008738238364458084,
+            0.001741456682793796,
+            -0.0052853962406516075,
+            0.0027346794959157705,
+            -0.017702894285321236,
+            -0.004553082399070263,
+            -0.014455240219831467,
+            0.01957082375884056,
+            -0.022584985941648483,
+            0.004153316840529442,
+            -0.005055442452430725,
+            -0.016075529158115387,
+            0.029434068128466606,
+            0.022896308451890945,
+            -0.0035483618266880512,
+            0.015070809051394463,
+            0.0017989451298490167,
+            0.0015061964513733983,
+            0.016202887520194054,
+            -0.015806660056114197,
+            -0.013025989755988121,
+            0.035066161304712296,
+            0.0176321379840374,
+            0.0004313841927796602,
+            -0.0023260696325451136,
+            -0.0037323245778679848,
+            0.0032193511724472046,
+            0.008023613132536411,
+            -0.010450509376823902,
+            -0.01141277700662613,
+            -0.017051948234438896,
+            0.009997677057981491,
+            -0.006155682262033224,
+            0.004446949809789658,
+            -0.00025891896802932024,
+            -7.617214578203857e-05,
+            -0.01516986545175314,
+            0.022584985941648483,
+            -0.017717044800519943,
+            -0.00602832343429327,
+            0.004687516950070858,
+            0.0207029040902853,
+            -0.02541518583893776,
+            0.01616043597459793,
+            -0.007107336539775133,
+            -0.013429293408989906,
+            0.012573158368468285,
+            -0.015127412974834442,
+            -0.0107335289940238,
+            -0.003175129182636738,
+            -0.008009462617337704,
+            -0.022754797711968422,
+            0.017929309979081154,
+            -0.013627407141029835,
+            0.015650998800992966,
+            -0.00287972716614604,
+            0.000562502013053745,
+            0.002057200763374567,
+            0.0032582662533968687,
+            0.02250008098781109,
+            0.030735958367586136,
+            0.010415131226181984,
+            -0.005465821363031864,
+            -0.021000074222683907,
+            0.012983537279069424,
+            0.0020589695777744055,
+            0.013202877715229988,
+            -0.002253545681014657,
+            -0.002135031158104539,
+            -0.016613267362117767,
+            -0.023122724145650864,
+            -0.0013010069960728288,
+            -0.01761798746883869,
+            -0.02552839368581772,
+            -0.01774534583091736,
+            -0.007047194987535477,
+            0.011349096894264221,
+            0.020405733957886696,
+            0.0009755341452546418,
+            -0.005384453106671572,
+            0.01559439580887556,
+            -0.006941062398254871,
+            0.002080196049064398,
+            -0.006594363134354353,
+            -0.002032436430454254,
+            -0.008200501091778278,
+            -0.021863285452127457,
+            0.029660483822226524,
+            -0.010103809647262096,
+            0.01141277700662613,
+            -0.005472897086292505,
+            -0.00965097825974226,
+            0.005575491581112146,
+            0.00900003220885992,
+            0.024849144741892815,
+            0.006452853325754404,
+            0.03141520544886589,
+            0.020151015371084213,
+            -0.018891576677560806,
+            -0.0006071660900488496,
+            0.022684043273329735,
+            -0.0037570889107882977,
+            -0.004889168310910463,
+            -0.0031786670442670584,
+            -0.017985913902521133,
+            -0.012007118202745914,
+            0.007839650847017765,
+            -0.013471745885908604,
+            -0.006636816076934338,
+            -0.00837738811969757,
+            0.018070818856358528,
+            -0.017773648723959923,
+            0.025698205456137657,
+            -0.006488230545073748,
+            0.005154499784111977,
+            0.031924642622470856,
+            0.014363259077072144,
+            0.019698183983564377,
+            -0.011794853024184704,
+            -0.031075581908226013,
+            -0.00683846790343523,
+            -0.007882103323936462,
+            0.009558996185660362,
+            -0.01062032114714384,
+            0.005204027984291315,
+            -0.02006610855460167,
+            -0.018948180601000786,
+            0.012863253243267536,
+            -0.017476476728916168,
+            0.013924578204751015,
+            -0.009955224581062794,
+            0.027226513251662254,
+            -0.001228483160957694,
+            0.0121698547154665,
+            -0.002614396158605814,
+            -0.017801949754357338,
+            0.01205664686858654,
+            0.004089637193828821,
+            -0.023943481966853142,
+            0.010337301529943943,
+            -0.001864393474534154,
+            -0.014037786051630974,
+            -0.018410442396998405,
+            -0.001228483160957694,
+            -0.005200490355491638,
+            -0.009658053517341614,
+            0.025457637384533882,
+            0.014101465232670307,
+            0.016231190413236618,
+            0.0035359796602278948,
+            0.0016335553955286741,
+            -0.011073152534663677,
+            0.009573147632181644,
+            -0.006803090218454599,
+            0.000612030504271388,
+            -0.0034404604230076075,
+            0.011893910355865955,
+            -0.004252373706549406,
+            -0.01543873455375433,
+            0.002324300818145275,
+            0.008561351336538792,
+            0.0143278818577528,
+            5.947839599684812e-05,
+            0.009587298147380352,
+            0.0009790719486773014,
+            -0.009764186106622219,
+            -0.00046388726332224905,
+            -0.0060884649865329266,
+            0.0069375247694551945,
+            0.02400008589029312,
+            -0.012516554445028305,
+            -0.01681138016283512,
+            0.014504768885672092,
+            0.004033033270388842,
+            -0.017589686438441277,
+            0.018367990851402283,
+            0.004103788174688816,
+            -0.022358570247888565,
+            0.0035412863362580538,
+            0.010662773624062538,
+            -0.009523618966341019,
+            -0.014469391666352749,
+            -0.0010339069413021207,
+            -0.015509488992393017,
+            0.03130199760198593,
+            0.00025073791039176285,
+            0.016910437494516373,
+            -0.01062032114714384,
+            -0.0025967073161154985,
+            -0.013309009373188019,
+            0.013103820383548737,
+            -0.015750056132674217,
+            -0.007273610681295395,
+            0.00035620704875327647,
+            -0.02298121340572834,
+            0.01741987280547619,
+            0.0013425755314528942,
+            0.00794578343629837,
+            0.003436922561377287,
+            -0.010556641966104507,
+            0.009820790030062199,
+            0.0018237093463540077,
+            0.006668656133115292,
+            -0.014915147796273232,
+            0.023618008941411972,
+            0.006413938011974096,
+            0.011235889047384262,
+            -0.01352127455174923,
+            -0.01546703651547432,
+            0.002253545681014657,
+            0.04134920611977577,
+            -0.01306136790663004,
+            0.0002593612007331103,
+            0.01562269777059555,
+            7.081024523358792e-05,
+            0.004156854469329119,
+            -0.0023755980655550957,
+            -0.0034493047278374434,
+            -0.034585028886795044,
+            0.005370302125811577,
+            -0.0006328147719614208,
+            0.008073141798377037,
+            -0.008992956951260567,
+            -0.005430444143712521,
+            -0.0040684109553694725,
+            -0.008625030517578125,
+            -0.00314859626814723,
+            -0.01478778850287199,
+            0.008363237604498863,
+            -0.014533070847392082,
+            -0.028910480439662933,
+            0.03166992589831352,
+            -0.006887996103614569,
+            -0.00533846253529191,
+            -0.021396303549408913,
+            0.007570781745016575,
+            -0.01370523776859045,
+            0.002663924591615796,
+            0.010967019945383072,
+            0.008865597657859325,
+            0.008738238364458084,
+            0.002762981690466404,
+            0.0009516543359495699,
+            -0.004255911335349083,
+            0.010273621417582035,
+            -0.00835616234689951,
+            -0.0005943417199887335,
+            -0.0014186371117830276,
+            -0.02725481428205967,
+            -0.01405901275575161,
+            -0.010592019185423851,
+            -0.007811348885297775,
+            -0.00435143057256937,
+            0.012608535587787628,
+            0.007174553815275431,
+            -0.0017485321732237935,
+            -0.022938761860132217,
+            -0.014745336025953293,
+            -0.0034103896468877792,
+            0.008349086157977581,
+            -0.01838214136660099,
+            0.01064862310886383,
+            0.0016830838285386562,
+            0.009976450353860855,
+            0.014292503707110882,
+            0.00302831269800663,
+            0.0016742395237088203,
+            0.003458149265497923,
+            0.014886845834553242,
+            0.012601460330188274,
+            -0.005801907740533352,
+            -0.021382151171565056,
+            -0.0024605041835457087,
+            -0.007068421691656113,
+            -0.0034687623847275972,
+            -0.012332591228187084,
+            0.03039633482694626,
+            0.0029805530793964863,
+            0.0176321379840374,
+            0.007832575589418411,
+            0.009573147632181644,
+            0.010507113300263882,
+            0.029377464205026627,
+            0.006049550138413906,
+            -0.0029929352458566427,
+            -0.008603804744780064,
+            -0.02141045406460762,
+            -0.013146273791790009,
+            0.009898620657622814,
+            0.013294858857989311,
+            0.03597182780504227,
+            -0.034896351397037506,
+            0.012162779457867146,
+            -0.011101454496383667,
+            -0.008264180272817612,
+            -0.008533049374818802,
+            0.0044717141427099705,
+            0.010068432427942753,
+            0.014370334334671497,
+            -0.0023402206134051085,
+            -0.01175947580486536,
+            0.009070787578821182,
+            0.023547254502773285,
+            -0.012983537279069424,
+            0.03016991913318634,
+            0.027325568720698357,
+            0.023066120222210884,
+            0.01761798746883869,
+            -0.012537780217826366,
+            0.026462359353899956,
+            -6.484030018327758e-05,
+            0.0006261814851313829,
+            -0.018084971234202385,
+            -0.00587973790243268,
+            -0.004294826649129391,
+            -0.004362043924629688,
+            0.013316085562109947,
+            -0.0025471788831055164,
+            -0.0008468485320918262,
+            0.015099111013114452,
+            -0.0037323245778679848,
+            -3.927453508367762e-05,
+            -0.0059752571396529675,
+            0.015127412974834442,
+            0.0002699744363781065,
+            0.019457615911960602,
+            -0.04270770028233528,
+            0.019726485013961792,
+            -0.021396303549408913,
+            -0.003102605463936925,
+            -0.0020890405867248774,
+            0.009523618966341019,
+            0.00764861237257719,
+            -0.007142714224755764,
+            -0.014334957115352154,
+            0.004524780437350273,
+            0.014066088013350964,
+            -0.0006726144347339869,
+            0.005080206785351038,
+            -0.0027541371528059244,
+            0.0017228835495188832,
+            -0.010089659132063389,
+            -0.005862049292773008,
+            5.842812606715597e-05,
+            -0.010570792481303215,
+            -0.008646257221698761,
+            -0.03232087194919586,
+            0.01062032114714384,
+            0.010209942236542702,
+            0.007294837385416031,
+            0.004252373706549406,
+            -0.022018946707248688,
+            -0.012318440712988377,
+            0.012948159128427505,
+            -0.01132787112146616,
+            -0.002570174168795347,
+            -0.0020005968399345875,
+            0.005387990735471249,
+            0.0004705205501522869,
+            0.010160413570702076,
+            -0.003472300246357918,
+            0.010938717983663082,
+            -0.014603826217353344,
+            0.018084971234202385,
+            0.03229256719350815,
+            -0.00533846253529191,
+            -0.019967053085565567,
+            0.000907432462554425,
+            -0.009077862836420536,
+            0.003813692834228277,
+            0.007641536649316549,
+            -0.021962342783808708,
+            -0.02173592709004879,
+            0.005087282508611679,
+            -0.013153349049389362,
+            -0.007783046457916498,
+            0.014830241911113262,
+            0.005221716593950987,
+            0.010839661583304405,
+            0.01751893013715744,
+            0.010889190249145031,
+            0.00840569008141756,
+            -0.026533113792538643,
+            0.0006734988419339061,
+            -0.012686366215348244,
+            -0.00590803986415267,
+            0.002239394700154662,
+            -0.0017591454088687897,
+            0.004591997712850571,
+            -0.00010115749319083989,
+            -0.011200511828064919,
+            -0.0007057808106765151,
+            0.010903340764343739,
+            0.027919910848140717,
+            -0.004146241117268801,
+            -0.0022570835426449776,
+            -0.014257126487791538,
+            0.0010100272484123707,
+            0.036481261253356934,
+            -0.012275987304747105,
+            -0.017434025183320045,
+            0.01757553406059742,
+            0.027452928945422173,
+            0.005954030901193619,
+            -0.0032529598101973534,
+            -0.0027364485431462526,
+            0.00039600671152584255,
+            0.011978816241025925,
+            0.007719367276877165,
+            -0.002692226553335786,
+            -0.002131493529304862,
+            0.004761809483170509,
+            0.035773713141679764,
+            -0.0015203474322333932,
+            -0.010330225341022015,
+            -0.024240652099251747,
+            0.010974096134305,
+            0.00277182599529624,
+            0.004754733759909868,
+            0.002313687466084957,
+            0.023504801094532013,
+            0.019655730575323105,
+            0.014858543872833252,
+            0.00470520555973053,
+            -0.009771261364221573,
+            0.03005671128630638,
+            -0.008172199130058289,
+            -0.004542469047009945,
+            -0.02838689461350441,
+            -0.005087282508611679,
+            -0.015650998800992966,
+            0.008929276838898659,
+            -0.0002221042668679729,
+            -0.005016527138650417,
+            -0.013422218151390553,
+            0.019726485013961792,
+            0.01064862310886383,
+            -0.011073152534663677,
+            0.012608535587787628,
+            -0.027070851996541023,
+            0.02346234768629074,
+            -0.03826428949832916,
+            -0.0071993181481957436,
+            -0.006421013735234737,
+            -0.010514188557863235,
+            -0.02627839520573616,
+            -0.0030353881884366274,
+            0.007372668012976646,
+            0.014250051230192184,
+            0.0022570835426449776,
+            0.0022818478755652905,
+            0.02282555401325226,
+            0.0016415152931585908,
+            -0.01508495956659317,
+            0.017985913902521133,
+            0.03492465242743492,
+            0.012926933355629444,
+            0.00155837822239846,
+            0.015990622341632843,
+            0.0027789014857262373,
+            0.0021633331198245287,
+            0.00788917951285839,
+            -0.0034493047278374434,
+            -0.007938707247376442,
+            0.008582578040659428,
+            0.006990591064095497,
+            -0.02763689123094082,
+            0.003824306186288595,
+            -0.00010408719390397891,
+            -0.009898620657622814,
+            -0.022429324686527252,
+            0.011568437330424786,
+            0.01913214474916458,
+            -0.004224071744829416,
+            -0.006771250627934933,
+            -0.0043337419629096985,
+            0.0006836698739789426,
+            0.006785401608794928,
+            -0.002412744564935565,
+            -0.000515184598043561,
+            -0.0015176940942183137,
+            -0.02368876338005066,
+            -0.010316074825823307,
+            0.013570803217589855,
+            -0.005175726022571325,
+            0.010358527302742004,
+            0.005242943298071623,
+            -0.018792519345879555,
+            -0.016938740387558937,
+            0.006852618884295225,
+            -0.016188737004995346,
+            0.01364863384515047,
+            0.00027948213391937315,
+            -0.0023172253277152777,
+            0.024962352588772774,
+            0.03594352304935455,
+            -0.013188726268708706,
+            0.005119122099131346,
+            -0.019386861473321915,
+            -0.0245378240942955,
+            -0.019004784524440765,
+            0.022613288834691048,
+            0.008306633681058884,
+            -0.018792519345879555,
+            0.012304289266467094,
+            -0.002752368338406086,
+            -0.008865597657859325,
+            -0.0066721937619149685,
+            -0.006845543161034584,
+            -0.008547200821340084,
+            0.020533092319965363,
+            -0.004935159347951412,
+            0.0038950610905885696,
+            0.019047237932682037,
+            0.04075486212968826,
+            -0.008292482234537601,
+            -0.01349297259002924,
+            0.0054339817725121975,
+            0.004878554958850145,
+            0.0018626245437189937,
+            -0.004814875777810812,
+            -0.008016537874937057,
+            -0.0185944065451622,
+            -0.01740572229027748,
+            -0.009615600109100342,
+            -0.0020041344687342644,
+            0.020122714340686798,
+            -0.007768895477056503,
+            -0.01354957651346922,
+            -0.0245378240942955,
+            -0.028188779950141907,
+            -0.006580212153494358,
+            0.02319347858428955,
+            0.017179306596517563,
+            -0.00013200224202591926,
+            -4.3337418901501223e-05,
+            -0.012042495422065258,
+            -0.01968403346836567,
+            0.015509488992393017,
+            0.00037876018905080855,
+            -0.01773119531571865,
+            -0.0065448349341750145,
+            -0.015919867902994156,
+            0.005787756759673357,
+            -0.0051368107087910175,
+            0.004574308637529612,
+            0.05057565122842789,
+            0.0005271245026960969,
+            0.008441068232059479,
+            -0.026575567200779915,
+            0.009106164798140526,
+            -0.01583496294915676,
+            -0.017377421259880066,
+            -0.00012702727690339088,
+            -0.022301966324448586,
+            -0.0030955299735069275,
+            -0.018792519345879555,
+            -0.007386818993836641,
+            0.008568426594138145,
+            0.009799563325941563,
+            -0.013896276243031025,
+            -0.013669860549271107,
+            -0.024693485349416733,
+            -0.0035872769076377153,
+            -0.001539805089123547,
+            0.009311353787779808,
+            0.007825499400496483,
+            -0.03693409264087677,
+            0.002536565763875842,
+            -0.0009507699287496507,
+            0.0005779796629212797,
+            0.01208494883030653,
+            -0.008667483925819397,
+            -0.003063690150156617,
+            -0.0214670579880476,
+            0.006452853325754404,
+            0.007103798910975456,
+            -0.013500047847628593,
+            0.012375044636428356,
+            -0.012863253243267536,
+            -0.007669838611036539,
+            -0.0005426021525636315,
+            0.03379257395863533,
+            0.011794853024184704,
+            -0.0025153392925858498,
+            -0.00654129683971405,
+            -0.011150983162224293,
+            0.02654726430773735,
+            -0.011349096894264221,
+            -0.026405755430459976,
+            0.028910480439662933,
+            0.013245330192148685,
+            0.0005974372616037726,
+            0.020151015371084213,
+            -0.0017936384538188577,
+            -0.01397410687059164,
+            0.009601449593901634,
+            0.004266524687409401,
+            0.012424572370946407,
+            -0.00014460546663030982,
+            0.0214670579880476,
+            0.0018555490532889962,
+            0.0027399861719459295,
+            -0.0036226543597877026,
+            0.004687516950070858,
+            -0.03093407303094864,
+            -0.0006549257086589932,
+            -0.003102605463936925,
+            -0.008398614823818207,
+            0.011207587085664272,
+            -0.002421588869765401,
+            0.006558985915035009,
+            -0.022188758477568626,
+            -0.028485950082540512,
+            0.011342021636664867,
+            -0.016231190413236618,
+            -0.021396303549408913,
+            0.015339677222073078,
+            -0.0007641536649316549,
+            0.00614860700443387,
+            -0.007443422917276621,
+            -0.023278385400772095,
+            -0.019386861473321915,
+            0.02064630016684532,
+            0.012212308123707771,
+            -0.01190098561346531,
+            0.012750045396387577,
+            -0.021113283932209015,
+            -0.012948159128427505,
+            -0.001894464367069304,
+            0.0090566361322999,
+            -0.0019864458590745926,
+            0.01325240544974804,
+            -0.012198156677186489,
+            -0.00334494118578732,
+            0.0031592093873769045,
+            -0.011589664034545422,
+            0.008158047683537006,
+            0.01978308893740177,
+            0.008193425834178925,
+            -0.009339655749499798,
+            0.003422771580517292,
+            -0.011922212317585945,
+            0.013500047847628593,
+            0.015537790954113007,
+            0.018311386927962303,
+            -0.006498843897134066,
+            0.0020412809681147337,
+            -0.01370523776859045,
+            0.0063962494023144245,
+            -0.009827865287661552,
+            -0.004878554958850145,
+            -0.0005333155859261751,
+            0.004921008367091417,
+            -0.011179285123944283,
+            -0.0025772498920559883,
+            -0.026830283924937248,
+            -0.0013823752524331212,
+            -0.020533092319965363,
+            0.005303084850311279,
+            0.023207630962133408,
+            -0.007514177821576595,
+            0.0006045127520337701,
+            -0.0039092120714485645,
+            0.014476466923952103,
+            0.020688753575086594,
+            0.00870286114513874,
+            0.00482548912987113,
+            -0.006081389728933573,
+            0.02179253101348877,
+            0.011603815481066704,
+            -0.017816102132201195,
+            0.00164239970035851,
+            -0.001509734196588397,
+            -0.0014336726162582636,
+            0.011625041253864765,
+            0.004443412180989981,
+            -0.020844414830207825,
+            0.0028602697420865297,
+            -0.010089659132063389,
+            0.007818424142897129,
+            0.014702882617712021,
+            -0.00031906069489195943,
+            -0.02486329711973667,
+            -0.009827865287661552,
+            0.021919889375567436,
+            -0.015070809051394463,
+            -0.0008291598060168326,
+            0.00764861237257719,
+            0.023264234885573387,
+            -0.024792540818452835,
+            -0.006831392180174589,
+            -0.0023455272894352674,
+            -0.003933976404368877,
+            -0.005193414632230997,
+            0.0010489424457773566,
+            0.0214670579880476,
+            0.004415110219269991,
+            -0.004234685096889734,
+            0.005724077112972736,
+            0.006470541935414076,
+            0.02325008250772953,
+            0.0040577976033091545,
+            0.028047269210219383,
+            -0.00943871308118105,
+            -0.0023809047415852547,
+            -0.0019988277927041054,
+            -4.775960405822843e-05,
+            0.0160047747194767,
+            -0.005430444143712521,
+            0.020320827141404152,
+            0.009445788338780403,
+            0.0036934094969183207,
+            0.016514210030436516,
+            0.02319347858428955,
+            -0.006767712999135256,
+            -0.004793649073690176,
+            0.019160445779561996,
+            0.0053667644970119,
+            -0.008858522400259972,
+            -0.02194819226861,
+            0.01141277700662613,
+            0.0014124460285529494,
+            -0.0006956097786314785,
+            0.012106175534427166,
+            0.0024375086650252342,
+            0.006300730165094137,
+            0.008172199130058289,
+            -0.016684021800756454,
+            0.004241760354489088,
+            -0.004698129836469889,
+            -0.01860855706036091,
+            0.010910416021943092,
+            -0.012092024087905884,
+            -5.201595922699198e-05,
+            -0.0010002984199672937,
+            0.014165145345032215,
+            -0.01703779585659504,
+            -0.0019103841623291373,
+            -0.0070719593204557896,
+            -1.3024719009990804e-05,
+            -0.0009162768837995827,
+            0.008320784196257591,
+            -0.006254739128053188,
+            -0.010938717983663082,
+            -0.010938717983663082,
+            0.027509532868862152,
+            -0.004991763271391392,
+            0.003916287794709206,
+            0.019330257549881935
+          ],
+          "embedding": [
+            0.007204997251390758,
+            -0.03173324616100502,
+            -0.020505790884701024,
+            0.0037451608231038273,
+            0.009817085391002229,
+            0.026883177166072393,
+            -0.016477826654817607,
+            0.06949311853773875,
+            -0.01224876667472692,
+            0.031952566873802865,
+            0.013633671951171586,
+            -0.005041437355104989,
+            0.02663392750069342,
+            -0.022099132865267587,
+            0.006665702000169926,
+            0.0030467129715919033,
+            0.007931802508415542,
+            0.009984477355413895,
+            -0.010362836464183985,
+            -0.03180848181251013,
+            -0.013542377333474449,
+            -0.013572345022577053,
+            -0.03979985300694044,
+            -0.007698412130292187,
+            -0.005953452234356532,
+            0.018890228917932694,
+            0.02217446994660065,
+            0.005158357501784182,
+            -0.05468694715835175,
+            0.00789994048345594,
+            0.017130184174786147,
+            -0.0026591475669416114,
+            -0.02081141846859738,
+            -0.008403928318906045,
+            -0.030925346724945978,
+            -0.03855023691059801,
+            0.038853354785097736,
+            0.01612011653434478,
+            -0.035635702395196575,
+            0.026266623757525934,
+            0.043450373922427316,
+            -0.007254334835408145,
+            -0.015376804869420366,
+            -0.0017276202867766907,
+            0.004650414207094434,
+            -0.002365514361535186,
+            0.011906139905776183,
+            -0.0005664194302009289,
+            -0.0191723447249157,
+            -0.025546124518037998,
+            0.01025005257280377,
+            -0.014436532079436437,
+            0.029277529346694118,
+            -0.0045414983847733344,
+            -0.02260315456913697,
+            0.01717918656248626,
+            0.013666132349359846,
+            0.057804090189845364,
+            0.056600576093100735,
+            0.051208022172527734,
+            0.03553760896972431,
+            0.02094599510567085,
+            -0.02438674472832137,
+            -0.0027797634437333432,
+            0.03614248430646102,
+            -0.026729308357113124,
+            -0.002251394534452815,
+            0.018929251842788906,
+            0.00689697483318378,
+            -0.026603606434115974,
+            -0.0541443728344337,
+            0.027164630314707807,
+            -0.004335272389193875,
+            -0.013967321193992958,
+            0.003323121377857836,
+            -0.0245295956412367,
+            0.024489335129134787,
+            0.06212059558640156,
+            0.009214058207162371,
+            -0.002082293435905329,
+            0.013659934201796686,
+            -0.003344037927706573,
+            -0.00689041317919184,
+            0.020235785390418023,
+            0.0212739457082623,
+            0.0016782346585238381,
+            -0.013484221664059555,
+            -0.06465582506604173,
+            0.011882797526607326,
+            -0.032970655964890155,
+            -0.04027700439305653,
+            0.025396847912528105,
+            0.013861157517716707,
+            0.00604596157583048,
+            -0.00744611407420936,
+            0.04038084298991919,
+            0.021507532290328625,
+            0.005983822806623904,
+            -0.012711110369915742,
+            0.02410138142963901,
+            -0.018372791517880082,
+            0.021016674128298246,
+            0.013387285956415756,
+            0.02039383525127615,
+            0.010700292312227496,
+            0.0053934660497349735,
+            -0.014807678014449691,
+            0.009422805142740376,
+            0.0414357625490679,
+            0.12206398267002624,
+            -0.010877509834324155,
+            -0.011079302503786522,
+            -0.04107714853756596,
+            0.006631397986165343,
+            -0.031914267104222384,
+            -0.0179965295457466,
+            -0.057530036803978266,
+            -0.0320488164246567,
+            0.008208202363320509,
+            -0.0020203090915064162,
+            0.03502055780919601,
+            0.04464599164916666,
+            0.019332152079287082,
+            0.017133943087431935,
+            -0.039885565008919065,
+            0.04703059988373572,
+            -0.029370696964304416,
+            0.0008494086835971101,
+            0.0014082189719987074,
+            -0.030973762582071615,
+            -0.006561157852024518,
+            0.011634653995510726,
+            -0.019896826887411713,
+            -0.034275921680252924,
+            0.026274465530039866,
+            -0.007728234125236442,
+            0.04814514040690998,
+            0.06550407296280117,
+            -0.02749051703615343,
+            -0.010041099502187074,
+            0.02019601967763537,
+            -0.01871701432558228,
+            0.008532829759864928,
+            -0.020164138072967613,
+            -0.045196925901010855,
+            -0.004523517988516156,
+            -0.002113469149506399,
+            0.07929650663710812,
+            -0.01111269708042164,
+            0.03358254444720164,
+            -0.019973382759721527,
+            -0.0007145592863325417,
+            -0.0011740862928390678,
+            0.02166182480535788,
+            0.004543228023398081,
+            -0.013377688428761516,
+            -0.01758250073647576,
+            0.009548292014596834,
+            -0.04753617258004735,
+            -0.007125744019792479,
+            -0.009930358894299303,
+            -0.007931672044235996,
+            -0.024561936537429774,
+            0.012979195012881556,
+            0.009209145250061479,
+            0.04586756662401364,
+            0.04311807301838248,
+            -0.0036241105681042675,
+            -0.06579256720896123,
+            0.04967191755416882,
+            0.015475886231466817,
+            0.03157651656768371,
+            0.02515941109300714,
+            0.01773554572598468,
+            0.005824418191900522,
+            0.016317635547562007,
+            0.007650872352957098,
+            0.010015166465492963,
+            -0.033203307142867844,
+            0.040272188258443496,
+            -0.013408423788025047,
+            0.01036004294342104,
+            0.012391622448703885,
+            0.04214857862444935,
+            0.03418032388760535,
+            0.015263018079889338,
+            0.018786704580603317,
+            0.02704685343985817,
+            0.015348111464412869,
+            0.0013031600434754704,
+            0.03241427528588285,
+            0.011689924942195664,
+            0.04544406114742012,
+            0.016186615914968734,
+            0.011624500626103958,
+            -0.008272953078864385,
+            -0.02363080189189375,
+            0.02784298119317616,
+            -0.010585980875564109,
+            0.01455013604305903,
+            0.01749228557615982,
+            -0.054742056843690476,
+            -0.005876213780171779,
+            -0.010556605098730201,
+            0.003967456397644932,
+            0.014663888851767762,
+            0.0015548476559947807,
+            -0.003858367677774835,
+            0.012504286824484335,
+            -0.025806882498628832,
+            -0.014216299453397403,
+            0.0031620924516417956,
+            -0.006448390918604307,
+            -0.02785577502346934,
+            -0.00766402687921159,
+            0.005275000635711884,
+            -0.0090448751942,
+            0.019702662630533,
+            0.007009091837467867,
+            -0.01847307100043082,
+            -0.0006774105206839772,
+            0.032952656101827446,
+            0.015900033044760432,
+            -0.02170121261931376,
+            -0.017964879081469612,
+            -0.005062849745084668,
+            -0.03171847232054206,
+            -0.038177398521606544,
+            0.029426291950046455,
+            -0.025687201683643784,
+            0.030158867757101797,
+            0.015780411479509624,
+            -0.007011484049169482,
+            -0.012114023533118393,
+            0.020661407942701628,
+            -0.025067530380881376,
+            0.009777190971413269,
+            -0.004160771288889298,
+            -0.028825369485727344,
+            0.008307290529469729,
+            -0.014415903546305744,
+            -0.007561235629544945,
+            -0.03201704135271719,
+            -0.015182869388110948,
+            -0.00501561450194676,
+            -0.0003863746169672397,
+            -0.01073403445816663,
+            -0.03321103354401307,
+            -0.002304357889348034,
+            0.017001213960120047,
+            0.00028222146729658097,
+            -0.04002882852650754,
+            0.014509588052058002,
+            -0.004150541038999649,
+            -0.034333837380045466,
+            0.0564137596224324,
+            0.03584898691608747,
+            -0.006830698606679202,
+            0.006121456445207273,
+            -0.04505674213649008,
+            0.04358467937307492,
+            0.010847859073780811,
+            -0.04918983713702805,
+            0.049190442935189385,
+            -0.00528960443684769,
+            0.01419195371129309,
+            0.040316024601793406,
+            0.004600843996974165,
+            -0.006780970176765864,
+            0.005206651850992678,
+            0.03397517915308201,
+            -0.010037223444627569,
+            -0.027377913828275806,
+            -0.03328136706915221,
+            0.02562568440972934,
+            -0.03485770844248482,
+            -0.020069979230687374,
+            -0.0052041408068332524,
+            0.029818953377223505,
+            0.028789460147370508,
+            0.004071274130619537,
+            0.01928375754300958,
+            0.0014970947734202088,
+            0.010502599795027616,
+            0.016914145702515263,
+            0.011942576738188502,
+            -0.014909953528043828,
+            -0.003432489584767786,
+            -0.05733768990013957,
+            0.0030710780388681002,
+            -0.02627280695622545,
+            0.007930906480371135,
+            -0.031227917075485834,
+            0.013132797714872345,
+            0.002514351081823354,
+            -0.007152021475569949,
+            -0.0066245205132327055,
+            -0.005279224195782313,
+            0.028512607222275155,
+            -0.00921960825771865,
+            -0.0359704105589433,
+            -0.021347597388353182,
+            0.011200044645812517,
+            -0.0004171294784927543,
+            -0.016784468873218786,
+            -0.0018746544944553878,
+            0.007749103417240419,
+            0.02390272097796833,
+            -0.005128540108845805,
+            0.006397849195755672,
+            0.01214922062611769,
+            0.01163371266250103,
+            0.012307186722130844,
+            -0.004982589662882946,
+            -0.016343724553037307,
+            0.019806887143115643,
+            0.013139657463054087,
+            0.0205801609265216,
+            -0.03767676007629612,
+            -0.0249879011550972,
+            0.05547392586363707,
+            0.005903313549319051,
+            -0.05568806298774435,
+            0.018827693208840233,
+            -0.007931178941307645,
+            0.008183992600768387,
+            -0.010319022946041053,
+            0.05111352384595183,
+            -0.024897203290807753,
+            -0.02096678205188722,
+            0.006436523335212348,
+            -0.04917084523640266,
+            -0.010459284852442706,
+            -0.011138558192296276,
+            0.014789370043397917,
+            -0.03105725978752438,
+            -0.004934745004818259,
+            0.015595801375817122,
+            -4.687585727052446e-05,
+            0.0028886067775750254,
+            -0.00850352807499054,
+            -0.0034885579171087815,
+            0.016339681487281293,
+            -0.0034050317046826813,
+            -0.008309570448487812,
+            -0.003615165868022203,
+            -0.027638409919916757,
+            0.023352910161586186,
+            0.026364511529991978,
+            0.001249936082895606,
+            0.001987948967948376,
+            0.01756629246122769,
+            0.004656075485833687,
+            -0.006752579977941239,
+            -0.013681608638716132,
+            -0.03405936202918687,
+            -0.0401867651078737,
+            0.022991180183813725,
+            -0.02318908140203705,
+            0.018299512436825923,
+            0.00904081847012441,
+            0.019697091729447287,
+            -0.019236130984371835,
+            0.015159917756323988,
+            -0.0387297415849285,
+            -0.018793498297161176,
+            -0.005256288213225733,
+            -0.010414697711744128,
+            0.0007039894599589104,
+            -0.01739645696111503,
+            -0.013728845328097506,
+            0.022709522955937343,
+            -0.022789243441781803,
+            -0.0036763802589970026,
+            -0.01429671717544519,
+            0.0023674126354206932,
+            0.049448298344058644,
+            -0.009926033487593551,
+            -0.023371285193090733,
+            0.020846792916650956,
+            -0.026996519120163966,
+            0.011246908457705283,
+            0.04091365159640201,
+            0.005469162850197613,
+            0.005640919201612091,
+            0.029575753750229278,
+            -0.0007860452837141593,
+            -0.0052805237594885516,
+            -0.009782745443933828,
+            0.021178503581190618,
+            -0.0009176512672832309,
+            -0.05245650420332918,
+            -0.007698364493410058,
+            -0.039027418406703554,
+            0.005448840777995117,
+            0.017581625031764185,
+            0.025522517097715556,
+            -0.020775492472580457,
+            -0.015906320383873178,
+            -0.000643669839811819,
+            -0.0086896084429604,
+            0.0029399233482444638,
+            0.014381240985647907,
+            0.021086693776922936,
+            0.002436575081260609,
+            -0.004913514868101904,
+            -0.011582101006213432,
+            0.014400297663309155,
+            0.0467181039341492,
+            -0.01722692180137712,
+            0.0011622224560709315,
+            -0.013036540368441797,
+            -0.06208454638917413,
+            -0.019160212331432766,
+            0.0351496273293468,
+            0.02429814055116254,
+            -0.009896662169762772,
+            -0.01647101145477158,
+            -0.002915821099802562,
+            -0.03557863734177486,
+            -0.01786433999615293,
+            -0.010699364747311715,
+            0.02390901423299708,
+            0.007213809581960646,
+            0.0005601364043388203,
+            0.00916614601011734,
+            0.014737491634516238,
+            0.019303599465763203,
+            0.008864535607492504,
+            -0.0575839606069681,
+            0.018051837786406608,
+            0.006315830080225406,
+            0.02525619013082567,
+            -0.015671924543113065,
+            0.003988943108210209,
+            -0.03179878281917948,
+            -0.029440641244429775,
+            0.040586322748158446,
+            -0.03418722657593859,
+            -0.030650542761032924,
+            0.006566745611226466,
+            0.009571089937153852,
+            0.004491158558315032,
+            -0.011352813042040581,
+            -0.02535505987721779,
+            -0.013381948772812133,
+            -0.030918676056735445,
+            -0.02192708443081135,
+            -0.005232667170255381,
+            -0.03342408726436631,
+            -0.07498634115550039,
+            -0.006245638452568239,
+            -0.049946896465756174,
+            -0.03900385398225758,
+            0.04458524125677807,
+            -0.006106489377959481,
+            0.02398410968928338,
+            -0.04615183642824994,
+            -0.01165382599292015,
+            -0.04890013373371541,
+            0.013624324077017377,
+            0.01476398305411351,
+            0.020202554298470666,
+            0.00014553149242532853,
+            0.036224316512987846,
+            0.00021498496267774312,
+            -0.05055208658013767,
+            -0.057482169168904106,
+            -0.003960451575823462,
+            -0.010217078509789989,
+            -0.005593738972150311,
+            -0.034090261748104665,
+            0.05835341009022349,
+            0.0116359661127526,
+            0.04203678184748103,
+            0.03447818335186531,
+            -0.006747564131379555,
+            -0.025756962754841885,
+            -0.028123416424745064,
+            -0.008073625379614702,
+            0.047544241458556825,
+            -0.029605671083240564,
+            0.008171081332993893,
+            -0.026007452205474395,
+            0.0067084589781962805,
+            -0.0070645448617940995,
+            -0.0012298101311792947,
+            0.012372286141738867,
+            0.006377319686763927,
+            -0.015024537877377194,
+            0.007934745664790507,
+            0.026041755081605937,
+            -0.007303731324541215,
+            0.0407000170257006,
+            -0.012990703097937777,
+            -0.033189084013863494,
+            -0.022525679964848666,
+            0.01657717698191603,
+            0.03440284885162039,
+            0.010230633738702899,
+            -0.008707808754455952,
+            -0.0363777020540216,
+            0.011184076269109602,
+            0.031774012431843524,
+            0.01504703222643543,
+            -0.022509833303978505,
+            0.025230974050270107,
+            0.017502662694813216,
+            -0.018700579139537456,
+            -0.03511734747709348,
+            0.015162183119327901,
+            -0.017711638132862063,
+            0.018190335050681287,
+            -0.01590174153215615,
+            0.011103110869700727,
+            0.0025189370696524936,
+            -0.00920305384736575,
+            0.01929652151514404,
+            -0.006196715938358087,
+            0.011426611271994685,
+            0.007058186102245829,
+            -0.018191352599117545,
+            0.0027202185072220937,
+            0.057310566886380025,
+            -0.000531639834386418,
+            0.01494975681909879,
+            0.03845692752400366,
+            -0.001971953300066856,
+            0.045554909803324765,
+            -0.02532332999916258,
+            -0.042616337063845953,
+            0.02633963974172818,
+            -0.02970705577683783,
+            -0.01624154998005974,
+            -0.02620993449678864,
+            -0.021490817420853802,
+            -0.004038424045768319,
+            0.009940887982818171,
+            -0.02692933845395842,
+            -0.023801570856681796,
+            -0.001948460540662095,
+            0.02290911497874141,
+            -0.03300532378029689,
+            0.006525488380791647,
+            0.007039306647666766,
+            -0.003685585408710752,
+            0.03183735852538168,
+            -0.027197064244321705,
+            0.006861928286688311,
+            0.011209869849682412,
+            -0.001094897323854524,
+            -0.005328676992017907,
+            -0.004007474089818478,
+            0.010796946231051235,
+            -0.030543173497906445,
+            -0.012879972141706173,
+            0.00935894527375688,
+            0.0017573071057611829,
+            -0.010050882212831222,
+            -0.03102132152986303,
+            -0.002915141291985554,
+            0.007634137769426993,
+            -0.011094093612593696,
+            0.01199609810193786,
+            -0.010458016758814742,
+            -0.04929347607956551,
+            0.010973132159729983,
+            0.028148117094907874,
+            -0.025504019123994758,
+            0.012470127357818321,
+            0.009182729379865986,
+            -0.016020133756226275,
+            0.016320111225714855,
+            -0.012051271446461976,
+            0.024098195821827578,
+            0.04317620443481784,
+            0.024592886889234396,
+            0.007727787037954994,
+            0.00771746776156926,
+            -0.0003147538723468374,
+            0.022572868970938624,
+            -0.016683340264320543,
+            0.013672085891597506,
+            0.020827021901755415,
+            0.02092309219721837,
+            0.0057130785714458055,
+            -0.003901728981013356,
+            -0.0036968415923910314,
+            0.002227024825579113,
+            -0.02106678364798051,
+            0.014578546755457675,
+            0.03128101864424301,
+            -0.005234627629617124,
+            -0.022394744692442837,
+            0.002430617645746389,
+            -0.015780389432069438,
+            4.7277885648917e-05,
+            0.013357803020920043,
+            0.013777651543690907,
+            -0.027713499926132385,
+            -0.0018999112228756628,
+            0.028364630573666852,
+            0.009146793705085586,
+            -0.009891297866910313,
+            -0.01644269527910728,
+            -0.0003903768506971414,
+            0.006512217314869576,
+            0.02408785204739448,
+            0.0018406499545367782,
+            -0.007360473750031044,
+            0.01484497577088404,
+            0.016723277095327158,
+            0.003629911330658209,
+            -0.0004820567356447713,
+            0.02394406451074397,
+            0.023794019615473026,
+            0.001632266615911801,
+            -0.02064379152040184,
+            -0.02755693521684878,
+            0.027335938400284726,
+            0.02769248863677192,
+            -0.006832353240623308,
+            0.027324973360715134,
+            0.003937207909967385,
+            0.013580154030133622,
+            0.02076818768224654,
+            -0.01784151382870951,
+            -0.017617903734311997,
+            0.0017316476451235418,
+            -0.009605294810027804,
+            -0.009829196696575069,
+            0.004891780008779742,
+            -0.011655402383771472,
+            -0.01124379942906224,
+            0.043205454954628145,
+            0.004713701851783307,
+            0.003560551284131962,
+            0.007894840447501912,
+            -0.018881214526749082,
+            -0.00945538175144614,
+            0.0019040259741796817,
+            0.029505863720301916,
+            -0.001400002868254808,
+            0.03315866433970851,
+            -0.00830542045504805,
+            0.008602408964961391,
+            -0.03202191233287069,
+            -0.006978289755114341,
+            0.0326499698728755,
+            0.015314305362524091,
+            0.015104436300213667,
+            0.0021257042489266787,
+            -0.0010840650756005028,
+            0.02434348355847526,
+            -0.007174433778817674,
+            -0.04839373063710409,
+            0.022696994450018916,
+            -0.004779511451775743,
+            0.023588216956749154,
+            0.0382733980620746,
+            -0.004232286553944853,
+            0.003815507826545929,
+            0.023531223314846965,
+            -0.01887650634213847,
+            0.015055403379782985,
+            -0.01918944132635701,
+            -0.017597900721414636,
+            0.02176630010543207,
+            -0.008896608357798277,
+            0.01732387427588561,
+            0.01764363756727683,
+            -0.012180043444768312,
+            -0.019003360499321047,
+            0.01229139850966224,
+            0.0025710091163655626,
+            -0.017075658234265562,
+            0.01797412486853335,
+            0.018372010308966135,
+            0.006723179570342314,
+            0.015417677746651683,
+            -0.00047634684109332134,
+            0.017608296628666153,
+            -0.001049984114818522,
+            -0.008416990841821452,
+            0.01827827907328822,
+            -0.009568717277044265,
+            -0.01658089563620308,
+            0.009808478422454319,
+            -0.05193977747210182,
+            0.010644335931025957,
+            0.0016781339270411392,
+            -0.01562285100294917,
+            0.00405036330963539,
+            0.007102724536527262,
+            0.005195069570261247,
+            0.0028773630901681767,
+            0.004610474981005765,
+            0.042921486858787584,
+            0.013213672397112759,
+            -0.0064030584930089415,
+            -0.010711457508461704,
+            -0.017480309425763254,
+            -0.027151012983864797,
+            -0.005204425551873271,
+            0.009927884476991038,
+            0.022601230152836282,
+            0.008046297278661014,
+            -0.0031858480011267704,
+            -0.02589628103025927,
+            0.019723430185137954,
+            0.03604835658149313,
+            0.062423099201768716,
+            0.010930268386637612,
+            0.02925564269324381,
+            0.02439517070384333,
+            0.023930432096006114,
+            0.013036828091946605,
+            0.006910311087503408,
+            -0.032995841694293995,
+            0.012813650252889403,
+            0.01333714655191133,
+            0.03678966264802025,
+            0.012571138353467183,
+            -0.05610151838497368,
+            3.0777966587107344e-05,
+            0.009791774904454618,
+            0.041959488802722524,
+            0.01956296954085397,
+            0.000155217569370113,
+            -0.007285602454264675,
+            0.031547870838711856,
+            -0.026518385135879206,
+            -0.012299593280363843,
+            -0.0026551378470968594,
+            -0.006190117573235779,
+            -0.003807907001161504,
+            -0.004991957475076049,
+            0.036983035940964804,
+            -0.015648565314743963,
+            -0.019256463332683493,
+            0.0004300362961407231,
+            -0.0015686065121077472,
+            0.0011311524136286674,
+            0.0032236641835276464,
+            0.013795345278976164,
+            0.015385029265915279,
+            -0.004800448639353721,
+            0.026514926384796363,
+            -0.008699453553651681,
+            0.022186817580012775,
+            -0.013427553742493343,
+            -0.011927259975371873,
+            0.017776228476138103,
+            -0.021243797085501972,
+            -0.01667280589187595,
+            -0.003688045116580565,
+            0.006189970874536939,
+            0.0020911968266404467,
+            0.0006097761986763103,
+            -0.004685468039463817,
+            -0.01011357666327168,
+            0.022631082517271542,
+            0.012398000826637277,
+            -0.006721588925828951,
+            0.018142966317821054,
+            -0.0021330638043572654,
+            -0.011869184647489572,
+            0.025073455250141923,
+            -0.0015392945266926425,
+            -0.0337085890598252,
+            0.01267302113143573,
+            -0.013438601165482187,
+            0.02673290895281225,
+            -0.009586762706902736,
+            -0.011208527168019762,
+            0.019144928964011168,
+            -0.0017591316941248897,
+            -0.005459150067129732,
+            -0.029230079356228025,
+            0.004679497471366242,
+            -0.011723195942933286,
+            0.015899377439148873,
+            0.0022054234777254627,
+            0.029187759252864872,
+            -0.018290694895057157,
+            0.00792533824593637,
+            0.011719198071061566,
+            0.014612375997487872,
+            0.005419943379355563,
+            0.010515265723144554,
+            -0.006893314445458116,
+            -0.04424495426064596,
+            5.8241472764594054e-05,
+            -0.005719143565673274,
+            0.0038220021965914754,
+            0.013106940982933836,
+            -0.01203955335201393,
+            0.011260626096963215,
+            -0.0279896016489392,
+            -0.0295228261396555,
+            0.036474811902902415,
+            -0.04900747926861378,
+            0.019261089742811332,
+            0.00014104646541594166,
+            -0.03057197731717424,
+            -0.04331886560073797,
+            -0.03232957159604667,
+            -0.019401402365464362,
+            0.004102283356333861,
+            0.0002610070704924167,
+            0.0012280372705500212,
+            -0.014025012323929194,
+            -0.014135990809256075,
+            0.013269444858490211,
+            0.00992155580941576,
+            0.011225162609266308,
+            -0.006545044517282357,
+            0.002368862850663672,
+            -0.02063183105457681,
+            -0.026981383275694142,
+            -0.02774194980145225,
+            0.013337303400678109,
+            -0.0025902548982556788,
+            -0.02794339093668326,
+            -0.025606606546251152,
+            -0.0029852498102770587,
+            -0.01906873442473352,
+            0.006508845592362198,
+            0.03268861529625085,
+            0.0355975143313428,
+            0.010066906387952277,
+            0.01690250467284985,
+            -0.017201544758038183,
+            0.048345309114140714,
+            -0.019031171948182872,
+            0.03188329748476941,
+            -0.0005845169448621133,
+            -0.024140202991373587,
+            0.00011597263700677706,
+            0.03203623528903764,
+            -0.019285565857567896,
+            -0.01332755507465857,
+            -0.010914187315481855,
+            -0.003374912271519852,
+            0.02419326154642231,
+            0.02443825785057301,
+            -0.0037923303809811746,
+            0.00022800399604730175,
+            0.015893265651968885,
+            0.003154527829977343,
+            0.007688529734165207,
+            0.030623379985627763,
+            0.014052282574575454,
+            0.01127235345371133,
+            -0.04164707563955819,
+            -0.011811587719246128,
+            -0.003763086694358232,
+            0.005909410979841885,
+            0.002913220159138154,
+            -0.002671613084376487,
+            0.0045765194162907885,
+            -0.007241703279856166,
+            -0.005855949422953439,
+            0.0016973541436777935,
+            -0.001576326424468957,
+            0.008393873711385013,
+            -0.0037217094191171083,
+            -0.03498879678492834,
+            -0.026096985039773694,
+            -0.021280075788049788,
+            0.006437497348154639,
+            0.0070214556591898375,
+            -0.00972721128392027,
+            0.01949553762684561,
+            -0.0029404319702066146,
+            0.000271037702827666,
+            -0.010734867909488861,
+            -0.0029784341515646075,
+            -0.01206464664516104,
+            -0.019544778160755956,
+            0.00035139045159739705,
+            0.013457601448003152,
+            0.00025947376767512006,
+            -0.03165522602526639,
+            0.011979811705146604,
+            0.006642849985850598,
+            0.013181162886675883,
+            -0.024499404540711422,
+            -0.02310714308893576,
+            -0.028028912907092744,
+            0.010570122662701183,
+            -0.0066866321204157725,
+            -0.024617571284817428,
+            -0.004097807033448707,
+            -0.0074576460628928256,
+            -0.0020417256533259887,
+            -0.02470953739335365,
+            -0.011408461668188081,
+            0.023537349682260345,
+            -0.006373652338136973,
+            0.01936338676879376,
+            -0.017636435070085086,
+            -0.017036091876814843,
+            -0.002198685044404603,
+            0.015584758061011048,
+            -0.038943923931820196,
+            -0.002076529978164001,
+            -0.008452513321945455,
+            0.006551055965075869,
+            -0.028840587857625097,
+            -0.007128279504360441,
+            0.0199397499488538,
+            0.012424319366271584,
+            -0.03518875279748495,
+            0.013596027106451115,
+            0.022191403935861113,
+            0.01218507466930854,
+            -0.0038551718784437783,
+            0.03910075115429626,
+            -0.0025215641203030336,
+            -0.005422688956956598,
+            0.016759888275451905,
+            0.04462538468871483,
+            0.023708097134660096,
+            -0.03152743757744915,
+            -0.008668774531128668,
+            -0.015224825785549947,
+            0.004102648962663789,
+            -0.025867795193966987,
+            0.004676262922216584,
+            -0.011739753544166856,
+            0.01645951757031186,
+            0.011253400919230348,
+            0.013958707917471253,
+            0.0376596076542735,
+            -0.005077354977296842,
+            0.02668910096392324,
+            0.0010332284250006117,
+            -0.00856723931523911,
+            0.0014431148305642308,
+            -0.01412579094241589,
+            0.029892393553325205,
+            0.02269599036836208,
+            0.014421409701746036,
+            0.029248195108770955,
+            -0.03588867295960255,
+            0.02458655544632528,
+            -0.0010013539341895672,
+            0.008368648583431747,
+            0.018150734970019018,
+            0.02628022565029391,
+            0.015423867373844644,
+            0.012477080028047823,
+            -0.02513018750675568,
+            -0.022180001014682045,
+            -0.0027550350630288997,
+            -0.031944843677988374,
+            -0.0014668093006914267,
+            -0.009789309083948147,
+            0.026386027599666254,
+            0.0046988411391149,
+            -0.019803160322638583,
+            -0.003931806845108628,
+            0.004482097428470392,
+            -0.00244660100774939,
+            -0.010441732346892935,
+            -0.004649318716095894,
+            -0.01869464992812615,
+            -0.017838643679294407,
+            0.023677878122153204,
+            -0.023958455640817666,
+            -0.025238725541146534,
+            0.014582798102040157,
+            0.0013223590277629197,
+            -0.002295641913710489,
+            0.0008238518461567685,
+            -0.02115533275282518,
+            -0.04214275933987015,
+            -0.027817083427786927,
+            0.0037588803464689412,
+            0.03565056002940463,
+            -0.026788865866928923,
+            0.014162550819567465,
+            -0.011336583458519,
+            0.019175861037658344,
+            -0.005127229083828615,
+            0.015531116410874209,
+            -0.018251290727445846,
+            -0.044630374763679266,
+            0.021833307809708984,
+            -0.012087009688208446,
+            0.02720742725750304,
+            0.021497809277370574,
+            -0.026937653159100156,
+            -0.013572733244188407,
+            0.01358214612023832,
+            0.0020408253706382332,
+            -0.009652962552750464,
+            -0.0063411929391088626,
+            -0.010533309167065193,
+            0.003901271588567942,
+            -0.04496345296706673,
+            -0.008761188928422701,
+            0.06948704056954039,
+            0.005916180375965137,
+            0.03157905519638486,
+            0.014668818318138927,
+            -0.028448496686381258,
+            -0.04008949536631331,
+            -0.00462253589823598,
+            -0.0003701522833020314,
+            -0.00024410762276269235,
+            -0.01850585249309217,
+            0.017699615637510956,
+            -0.002712069912576265,
+            -0.009767338815933488,
+            -0.013119464222297351,
+            -0.013743610418034631,
+            -0.030375365173543157,
+            -0.0030316327877227783,
+            -0.008632689470358149,
+            -0.006565457264917239,
+            -0.0328089275944709,
+            -0.011713996290006943,
+            0.029773910045828737,
+            0.02740100904353622,
+            -0.0061926915497348915,
+            0.00638737826122608,
+            -0.010244154924811975,
+            0.0010396203272677069,
+            -0.001264454445070139,
+            -0.004059969507786729,
+            0.008715730726831364,
+            0.0014767015415289076,
+            0.031074767756160294,
+            0.004761858344125113,
+            0.00987870172044607,
+            -0.059926117793699066,
+            -0.009294021263133636,
+            0.021188432871397475,
+            0.016602826531924377,
+            -0.009125784136292983,
+            0.013946188378030194,
+            -0.0017670794602549428,
+            -0.006837334179368295,
+            -0.028290055353601393,
+            0.024556541126233806,
+            0.04543639597357669,
+            0.0031311343860416473,
+            -0.011199639773247286,
+            0.014965677855595444,
+            0.0018248533506196013,
+            0.015507721118783542,
+            -0.011409402378400194,
+            -0.01338737990904486,
+            0.03063410115541065,
+            0.009305362935335817,
+            0.0025578631174065385,
+            0.012727648343848194,
+            -0.0038680554900795763,
+            -0.01173215289303824,
+            -0.01568862495293563,
+            -0.004101703129244437,
+            -0.03649200408657064,
+            -0.0022107849490954137,
+            0.009659780679763728,
+            0.0034091693454195806,
+            0.008848380026939594,
+            0.01842435611499587,
+            0.03546926214224266,
+            0.013478278853234764,
+            0.005970293629431872,
+            -0.0069197533761813455,
+            0.023196445250565058,
+            0.010403914879205351,
+            0.010017272564548626,
+            0.015076807790462658,
+            -0.029649260409273275,
+            -0.012737152920662164,
+            0.00373276777994811,
+            0.013239909450830258,
+            0.0001682544254742693,
+            0.02030652051051447,
+            0.014143961923579586,
+            0.0010147733405793957,
+            -0.012509446388074923,
+            0.0020437769941487626,
+            0.007405578085797006,
+            0.013458276946120546,
+            0.0006779114540819997,
+            -0.004250745498730309,
+            0.0009149988500358458,
+            -0.008355906072311008,
+            0.040391939952386505,
+            -0.01831218885700067,
+            -0.016793844108223176,
+            0.035064882870542116,
+            0.11416006426189942,
+            0.0010630856112440689,
+            -0.015660014600661857,
+            -0.017142534874931403,
+            -0.0086899013413391,
+            0.016611825572132997,
+            0.01609650977226868,
+            0.009780106186629793,
+            0.009622367309076686,
+            -0.000672926017804395,
+            0.024310949465350565,
+            0.02944698490225121,
+            -0.0037169531700178868,
+            -0.010990838766452194,
+            -0.036819646475041955,
+            0.022923962979125287,
+            -0.004596366586768123,
+            -0.007108041562635549,
+            -0.002604794313552557,
+            0.012641223813290853,
+            0.013668775645501733,
+            0.005125100040230003,
+            0.0077993841846437815,
+            -0.011500851674334447,
+            -0.017334458072700107,
+            0.0056819751688881774,
+            -0.015982234644378358,
+            -0.0076533013358874935,
+            -0.013127172654362694,
+            -0.011837346554842119,
+            -0.006562312362371076,
+            0.0029877420008976425,
+            -0.008572723854050838,
+            0.00854083969164562,
+            -0.010939639783598493,
+            0.016185642759084087,
+            -0.008052282332943615,
+            -0.0011992212936609377,
+            0.0010378646897070758,
+            -0.0025399515501741718,
+            0.010712865639618352,
+            0.01451357949543628,
+            -0.02521401163091652,
+            0.020836357346214075,
+            -0.008252813367155418,
+            0.0006409807249997337,
+            0.015338809580802425,
+            -0.04672526649678653,
+            0.022728876784937357,
+            0.005129225753457052,
+            -0.018372222092794564,
+            0.0066182736895667805,
+            -0.02048375780196863,
+            -0.00896382274306482,
+            0.0021738974344880794,
+            0.004491167150658828,
+            0.0034280799511763956,
+            0.015611133773752265,
+            0.018126362497548308,
+            0.00014383671280220997,
+            0.007927708693242186,
+            -0.006973076292062532,
+            0.008118313055163714,
+            -0.006791836664375275,
+            -0.01164567445668625,
+            -0.010894577039363014,
+            0.00014731542306100506,
+            -0.008865132658571428,
+            -0.011415696755798914,
+            -0.017922318360382588,
+            0.00358538001727405,
+            0.00289327637128695,
+            0.010406054711003452,
+            -0.007757607207626101,
+            -0.010994775479337082,
+            -0.011716096880261664,
+            -0.0013333474316266438,
+            -0.01231144419953554,
+            0.009792603083441917,
+            -0.02884953407490196,
+            0.0029097409399144306,
+            0.025277274938032477,
+            0.009263530315933738,
+            -0.028037993274175277,
+            -0.02532252134134532,
+            -0.0018596440554143638,
+            -0.013847737486669237,
+            0.009910107581119272,
+            0.013952962280235523,
+            -0.011807495289410774,
+            0.003896325037406752,
+            -0.02144921729633576,
+            -0.01628545945817568,
+            -0.0040148671532777005,
+            -0.016032493124255323,
+            0.004580041957001551,
+            -0.011375180364988195,
+            -0.007789996450583554,
+            -0.022041665680945066,
+            -0.005890065942573431,
+            -0.050455058269193866,
+            0.014401045842891128,
+            0.030219548461217365,
+            -0.004580747668649223,
+            -0.028684680969792875,
+            -0.015525870112797619,
+            0.011121401958547706,
+            -0.008879749268609245,
+            -0.019454243014778435,
+            -0.007839434838464604,
+            0.018908534797107714,
+            -0.0019283067432864216,
+            0.0024769961730655835,
+            -0.006732653593696035,
+            0.027604805021030646,
+            0.01366593691107021,
+            0.03740746059041736,
+            -0.014662268336336578,
+            0.02539666271785517,
+            0.02371809783638648,
+            -0.036126848124425154,
+            -0.007627445211958312,
+            0.03657481892016626,
+            0.01092515982671624,
+            0.0011046573579274704,
+            0.012350047525582187,
+            -0.006340802546498992,
+            -0.031330519599469066,
+            -0.0013691360532301728,
+            -0.0014676630169621005,
+            -0.023111265345539388,
+            0.027359543575867225,
+            0.018091458372134273,
+            0.014237156751820934,
+            0.0030993618578507172,
+            -0.0014037082734049385,
+            0.013257269860435359,
+            0.01021315143964527,
+            0.030885417052806473,
+            -0.003225065956381245,
+            -0.012825341172770057,
+            0.013476039084668826,
+            0.01852164843904835,
+            -0.009965495459579544,
+            0.012541079912270015,
+            -0.00824122613985912,
+            -0.018124134280975132,
+            0.03192263766222747,
+            0.0001484308749572035,
+            0.010888733498667343,
+            -0.010649515858317311,
+            -0.013082335214246402,
+            0.032614733822945205,
+            -0.00999075761269065,
+            0.004257643407198955,
+            -0.030723533031261387,
+            -0.003820776505926347,
+            -0.010471792928758869,
+            0.01737608180148319,
+            0.004597677757231315,
+            0.02095148515295301,
+            0.029253627700524396,
+            -0.011604450820008121,
+            -0.01996325509217509,
+            -0.001158626158150122,
+            -0.020583793738536985,
+            0.004168757459099133,
+            -0.03632338333261702,
+            -0.019373660230711246,
+            0.022259216603413172,
+            -0.010364620831688015,
+            -0.004751193403783363,
+            0.00444187509644351,
+            0.0008572202661850934,
+            0.014943779567410408,
+            -0.025408263453313398,
+            0.017755758370543148,
+            0.004627392860417132,
+            0.007402073071525438,
+            -0.024913329140362466,
+            0.013760771254234878,
+            -0.006087346663657887,
+            -0.03780469033982951,
+            0.005509388120163784,
+            0.03756141956381493,
+            0.02381019168406692,
+            -0.024580249929652097,
+            0.01807584361080568,
+            -0.03111039245576191,
+            0.023245542501125073,
+            -0.004868090951368273,
+            5.1544460403943004e-05,
+            0.00640235757127173,
+            -0.004698373089469779,
+            -0.00621383035264185,
+            -0.0025132415458431587,
+            -0.008746634331769936,
+            -0.01611801559493664,
+            -0.01193535623883546,
+            -0.03381789115349148,
+            -0.024119754887378317,
+            0.013881102852385332,
+            0.0019232668966222687,
+            -0.0095263729740736,
+            0.018652966857927805,
+            -0.03775912497511837,
+            -0.012022352025516115,
+            -0.023289377270709736,
+            0.023579672990954228,
+            -0.028605627815260902,
+            0.019235397553197357,
+            -0.025838655372560505,
+            -0.021458526449310157,
+            0.01508157885177254,
+            -0.010064396230734452,
+            -0.0016708624560165368,
+            -0.015042071642751917,
+            -0.03670785416972153,
+            -0.006267614265977843,
+            0.02298763967146357,
+            -0.012506444728541897,
+            0.00029871598584066644,
+            0.02671390618094565,
+            -0.003059865833779805,
+            -0.011103452976820622,
+            -0.0017382684168257506,
+            0.010923055147701782,
+            -0.02938686260440486,
+            0.014300097108092281,
+            0.0038571851420598074,
+            -0.0037614499297117365,
+            0.0010991440540809893,
+            -0.010108413486243776,
+            0.02127952161226619,
+            0.009142545227414532,
+            -0.0030260119510809205,
+            0.040589877142650374,
+            0.008677453889680373,
+            0.0017339508605300814,
+            0.008732950895530873,
+            0.01063352447960556,
+            0.006618635029823979,
+            -0.014986307792626314,
+            0.016905861015681957,
+            0.013101237900847744,
+            0.0002162791516784468,
+            0.019309343995079022,
+            0.0021263376627204313,
+            -0.024516267132902416,
+            0.013339486637573757,
+            -0.02311659751478777,
+            0.0020575982953961246,
+            0.02391695586495566,
+            -0.020440172706421535,
+            -0.001656132876701879,
+            -0.006304133667169861,
+            0.000543457661904495,
+            0.022135707414160034,
+            -0.0230903871986617,
+            -0.023444763940153112,
+            0.009604600033024832,
+            0.01721387169484476,
+            0.01827905276638783,
+            0.02824773106648473,
+            -0.007100625424382998,
+            0.01348510178676837,
+            0.0002604386361122316,
+            -0.011802178189607182,
+            0.007664157334475143,
+            0.0074175377458313594,
+            -0.0044008460682833875,
+            0.025491035291032547,
+            -0.0017015687016376205,
+            -0.006134010425966364,
+            -0.005379657499404915,
+            -0.01584573519366957,
+            0.0034386456145185213,
+            -0.013860742817569676,
+            -0.013355238174583658,
+            0.0040342831827141845,
+            -0.0208160950322342,
+            0.10904592104153767,
+            -0.0037038089210379766,
+            -0.019472012964045247,
+            0.0063554319446707044,
+            0.003716719884684827,
+            -0.003692786470980806,
+            -0.003353843210051379,
+            0.007788605566526459,
+            -0.011254480693190834,
+            -0.0045751956417398095,
+            -0.013976039724381875,
+            -0.001989262807706565,
+            0.010067374586433014,
+            0.009201983771978006,
+            -0.005902980676079493,
+            -0.02029714318363332,
+            0.013031676901335606,
+            -0.016970144750162643,
+            0.008874670945038486,
+            0.0008311255074953139,
+            0.00884167159995975,
+            0.001716603195908536,
+            0.01854951250316076,
+            -0.009310881405486074,
+            -0.01325242738760595,
+            0.029667589863813217,
+            -0.005616511284128333,
+            -0.0038179509972925954,
+            -0.0004690716809300012,
+            -0.005068974756744239,
+            0.0021204231125051274,
+            -0.027958274497127258,
+            0.022127088236172387,
+            -0.017271838492631593,
+            0.010267709087809265,
+            0.01316800560257366,
+            0.005599367128954229,
+            0.01817806970002991,
+            -0.007115594036834107,
+            -0.02627643123342057,
+            0.006542742528003751,
+            -0.003433874053082844,
+            0.006631427284537971,
+            0.02316295746074422,
+            -0.014573351469966297,
+            -0.033520864104177005,
+            -0.013707022143307006,
+            -0.01877574599842729,
+            -0.005992004412448576,
+            -0.0043698881826095435,
+            -0.01481352026460129,
+            0.006424121402963946,
+            0.01382938882387991,
+            0.018136021761515388,
+            0.0012697483790911433,
+            -0.011531136974861559,
+            0.00609187274277241,
+            0.01076133059240578,
+            0.008400887242930113,
+            -0.01533504932731406,
+            0.011439227832642668,
+            -0.028718487721012004,
+            0.01522697877548034,
+            -0.012452665599305595,
+            -0.024938748080573947,
+            -0.0022082450217489366,
+            0.017589399637694264,
+            0.0037824832255756763,
+            -0.004355445812154662,
+            -0.018778668474337727,
+            0.019825087149714933,
+            0.024692878851503508,
+            0.003883497401229232,
+            0.019470881143244197,
+            0.041834635328088364,
+            -0.003544509491805351,
+            0.027139164223766474,
+            -0.02129723143323512,
+            0.01461354860170749,
+            -0.03441729954693155,
+            0.001893713808284104,
+            -0.004222657393633789,
+            0.0013677766871517654,
+            -0.003811928806805752,
+            0.020142937759360904,
+            -0.009158202577174012,
+            0.0030638058587375543,
+            0.024854053046139163,
+            -0.018665378030958756,
+            0.00805903734779081,
+            0.010625429315145887,
+            -0.012856237251902233,
+            0.028535299711292306,
+            -0.006335094045840286,
+            0.00305422126851784,
+            -0.0297883488914759,
+            -0.0029192872993071216,
+            0.0026067044379754047,
+            -0.006097749614529602,
+            -0.00968023644800927,
+            -0.007433916539502931,
+            -0.006202274026852405,
+            0.0038354607448269555,
+            -0.021471361566683354,
+            0.003195646827154078,
+            -0.004390751282171024,
+            -0.003323447423321121,
+            0.02518759723554948,
+            0.025857746688368054,
+            0.016893707217894103,
+            0.012739170140289531,
+            0.008553625305650129,
+            0.019805351588748434,
+            0.01654964927067909,
+            0.011549329129728739,
+            -0.020026013396046632,
+            0.01150644479272541,
+            -0.004159611921476011,
+            -0.0022937297602724912,
+            0.014815466072709131,
+            -0.01572171840394285,
+            -0.02276187015864222,
+            0.024693009795978448,
+            -0.019725315346836404,
+            0.00998768979020729,
+            -1.1702210281215188e-06,
+            -0.011129745730507545,
+            -0.002591293237053776,
+            -0.018034357254990988,
+            0.00012070821244376275,
+            -0.023051045150566527,
+            -0.028465270110388716,
+            -0.00048603008831669926,
+            -0.000574546300981984,
+            -0.012992623897880395,
+            0.02064747165046032,
+            0.014820638761258362,
+            -0.024820174379739057,
+            0.02666518374464728,
+            -0.017430076811109155,
+            -0.01495051200726725,
+            -0.005861181182847845,
+            0.00100881727574559,
+            -0.0055223282819321295,
+            0.0034719859452138017,
+            -0.010180752654175722,
+            -0.009961678082468036,
+            -0.012134423047419183,
+            -0.028073672819317354,
+            0.011574683159318163,
+            0.016120674453124317,
+            -0.015043147648505721,
+            0.031834693135959095,
+            -0.008802805725050891,
+            -0.0018698230152007877,
+            0.016179742266521146,
+            0.005096232332051617,
+            -0.017187077376166815,
+            -0.018999569809271876,
+            0.012251308792871571,
+            0.013279090498325945,
+            -0.009588297039279112,
+            0.006630431200804809,
+            -0.014130176783207386,
+            -0.005092662672039885,
+            0.007020236623084732,
+            -0.005049363863784624,
+            0.0055051712211635915,
+            -0.00995314895304919,
+            0.0052743456988306485,
+            0.005841370337169303,
+            -0.0029434616355351975,
+            0.004677049987338193,
+            -0.018568941264514256,
+            0.020031253265629777,
+            0.0018725390936655962,
+            0.01332532735283085,
+            -0.03518855189467458,
+            -0.0056298723325832025,
+            -0.005369475318146738,
+            -0.004650364775945294,
+            -0.020833510091412353,
+            0.020445280915606132,
+            0.0159991245532085,
+            0.013202409908533162,
+            0.023094507005426774,
+            -0.014905666713082246,
+            0.01572052456975675,
+            -0.020945766915121455,
+            0.013114000985348841,
+            0.003349101658930482,
+            -0.019829605346106578,
+            -0.001423750749031516,
+            0.04557459007627153,
+            -0.005847231973329081,
+            0.011986110554869603,
+            0.0012709842140249691,
+            0.026758922819943853,
+            -0.0001635536762342011,
+            0.0065419519929442995,
+            0.005997692702992206,
+            -0.009935128108338126,
+            -0.0008057798606777003,
+            0.00865712443881613,
+            0.046254320605750854,
+            -0.01181132866321511,
+            0.008170688314239364,
+            0.013099051984057827,
+            0.0034971494510865624,
+            -0.0008386400616584293,
+            -0.015346506644390764,
+            -0.03067678541405639,
+            0.019305962963428017,
+            0.007736916644077731,
+            -0.012756525606221703,
+            0.008734673666457216,
+            0.015137771675476126,
+            0.012891824153867332,
+            0.003895886821685762,
+            0.02155154331885366,
+            -0.012761690789748027,
+            0.018528313792219413,
+            0.024663453412919027,
+            0.009077230086690327,
+            9.184526542175535e-05,
+            -0.000751529849753638,
+            0.006791927267880182,
+            -0.0004433479107716552,
+            -0.009832230053513483,
+            0.015858232096630655,
+            -0.014861341148074698,
+            0.017790924034108806,
+            2.7216806238946785e-05,
+            0.03744841066552559,
+            -0.015216207585985075,
+            -0.010138319613387346,
+            -0.007774325714088762,
+            -0.0016997034352120362,
+            0.003971564314593291,
+            0.0018686537276849653,
+            0.006051669054361943,
+            -0.0008319624938748146,
+            -0.017326561560776794,
+            0.01667737942301944,
+            0.005719808546053257,
+            -0.0007405637872110778,
+            0.013750752065593804,
+            0.0038732323799683453,
+            -0.00367645612814479,
+            0.023580906347714314,
+            0.006238541579764762,
+            -0.005235795826199542,
+            0.0028182323816636243,
+            -0.006443210286549713,
+            -0.016255209299456314,
+            -0.003061441395209219,
+            0.004597970789372988,
+            -0.0016590920570710628,
+            -0.011138261849069419,
+            -0.0019423173609729848,
+            0.020355534503272887,
+            -0.004612410336124759,
+            0.013346750536413418,
+            -0.005400060374036387,
+            0.006252154576727367,
+            0.010601459925435238,
+            -0.012346569079682625,
+            0.0014154620309235822,
+            0.010406076751218612,
+            0.0013983417564564663,
+            -0.007140180075225315,
+            0.0010544410919327675,
+            0.007757599672903709,
+            0.006505350216372801,
+            0.00483063937962671,
+            0.00836630436955631,
+            0.013284451514874498,
+            0.0033749897293287438,
+            -0.006243293993633622,
+            0.0013949831763016042,
+            -0.013269921385800992,
+            -0.006308639321247727,
+            0.0010906169976769569,
+            -0.0010152835709869353,
+            -0.02017142197515583,
+            0.010787630901414118,
+            0.005355784839945777,
+            0.007838485470991034,
+            -0.01706932891696025,
+            0.009576030158827565,
+            -0.0008144311254998525,
+            0.02829832828245172,
+            0.007407953684956539,
+            -0.02769403281764041,
+            0.014266001222243134,
+            -0.010492199740754951,
+            0.022575782293320284,
+            -0.0016349316593207844,
+            -0.005656505606143177,
+            0.008025514398322551,
+            -0.009634281579228592,
+            -0.005991026158213869,
+            -0.022366083313133788,
+            -0.025825209755160946,
+            0.016871087849192924,
+            0.013583173973358155,
+            0.021125252303971605,
+            0.00012555627355712715,
+            0.015347535950809997,
+            0.0020128011217859444,
+            0.015164099887550086,
+            0.0024544903332736167,
+            0.012507136277293888,
+            -0.011798050869794805,
+            0.0104828761281283,
+            -0.0137805761114781,
+            0.014620357150314097,
+            -0.005246639228547017,
+            -0.006986408834884282,
+            -0.008849580196616098,
+            -0.005442165099441538,
+            -0.020891551850452786,
+            -0.009155930286949747,
+            -0.0013590151948670868,
+            0.010665872513236615,
+            -0.0005386423407222223,
+            -0.0027671778465775897,
+            -0.03237311596935737,
+            -0.004875062533405715,
+            -0.006670982068843501,
+            -0.01942244110096111,
+            -0.0045975373348006885,
+            -0.026137783023267008,
+            -0.01657173843643047,
+            -0.0007784046498232684,
+            -0.015037306676062236,
+            -0.006889680260149007,
+            -0.02009705080543122,
+            -0.027192867101205106,
+            3.0161964469866572e-05,
+            -0.009647644858034833,
+            -0.005380165957043753,
+            0.013815092649296373,
+            0.011902366689169847,
+            -0.01572592170758115,
+            -0.022321702398924333,
+            -0.01088937403483412,
+            -0.0165754067023749,
+            0.01646435743868707,
+            0.009914938007210463,
+            0.018834863207293908,
+            -0.003724245912768107,
+            0.005643505694722323,
+            0.0009581443359000331,
+            0.02327704258596552,
+            -0.02532422120223976,
+            -0.028694220894404807,
+            0.0019298778426469338,
+            0.0025970175688368323,
+            0.006086164866310174,
+            -0.009414665394188936,
+            0.001200390869257535,
+            -0.007782899559317819,
+            -0.0228264209954853,
+            -0.00898519580978142,
+            0.008156835270668449,
+            -0.010036630164522355,
+            0.02230664068547531,
+            0.012332230358869292,
+            0.013699197379189934,
+            0.010593587938184292,
+            0.01576174456497367,
+            -0.0001654241831584687,
+            0.014471905257729428,
+            -0.008675915624727887,
+            -0.0031070346099204425,
+            0.0038358404385932777,
+            0.005878417037187274,
+            -0.00625022998235275,
+            0.028077114529659795,
+            -0.0011836453716817525,
+            0.0002962695480296182,
+            0.014405782572641941,
+            0.02697667559474498,
+            -0.004710313427099278,
+            0.0210512043261309,
+            -0.002393165025068831,
+            -0.006135816286212752,
+            -0.008429776018235104,
+            0.008607617299702318,
+            -0.009529291136733401,
+            0.011133497451942684,
+            0.019864404836361926,
+            -0.014041315457579166,
+            -0.007351413712676263,
+            0.02950254974910368,
+            -0.014564388091367897,
+            -0.01825317106331622,
+            0.02325964952838705,
+            -0.007414295676409368,
+            -0.0039042587057605484,
+            0.00377144546378926,
+            0.007169856503233239,
+            -0.004036744966949995,
+            -0.012928341573802416,
+            0.016090527447479415,
+            -0.00011159893712484579,
+            0.014146137810890391,
+            -0.0004655839804281017,
+            -0.006372231942983921,
+            -0.010047382825720416,
+            0.002373224247019062,
+            0.026818813126567614,
+            0.013483627627077692,
+            -0.017895710143841333,
+            -0.013785856764479817,
+            0.010826610341182479,
+            -0.01538893818197323,
+            -0.004425206001945227,
+            0.02155595194812746,
+            0.0028768153668128553,
+            -0.020543027467698483,
+            0.008028975884363618,
+            0.0003835165718138906,
+            0.00037046947047892953,
+            -0.00029363296913641716,
+            0.030135739208689217,
+            0.001343516577762509,
+            0.00018948658255495327,
+            -0.00875053830853609,
+            0.008606573235959595,
+            -0.01280305089741873,
+            0.012786478836532904,
+            -0.02437993139337839,
+            0.007648181775166282,
+            -0.00629541989541625,
+            0.004579605315599212,
+            0.016658386927051592,
+            0.01218098627022323,
+            0.015031618813778413,
+            0.008237200519537417,
+            -0.008942965899756683,
+            0.0009093265012901892,
+            -0.025302175262520477,
+            -0.001333639155622843,
+            -0.017971876771137367,
+            -0.02036433247634014,
+            0.0002463842837918767,
+            0.022144712689630985,
+            0.018559482073124734,
+            0.011579840028464503,
+            -0.008657014440886656,
+            0.0166696911919226,
+            -2.9834620942882283e-05,
+            0.011845827764518911,
+            -0.00037901176785495324,
+            -0.006138374154553732,
+            0.004711058953619512,
+            -0.003622694588720417,
+            0.013542405259906128,
+            -0.014730567462694827,
+            0.012076044672004906,
+            0.01690223095263866,
+            -0.009108632851807137,
+            -0.008946251274440141,
+            -0.013992423874475428,
+            0.00048381889145447367,
+            -0.015909599902931202,
+            0.035489252674454214,
+            0.02252772349379229,
+            -0.01210464970923921,
+            -0.010469120693943236,
+            -0.014164999313775756,
+            0.019496048806752753,
+            0.010427539351244087,
+            0.0028018908451425276,
+            0.017091768406027965,
+            0.021395638416767112,
+            0.008817247593404623,
+            -0.006081063711583319,
+            -0.0021216503501144983,
+            -0.003781810594530787,
+            0.020143098218482827,
+            -0.003078172339607037,
+            -0.011350171884507555,
+            -0.012516379645909518,
+            0.005766102930704529,
+            -0.01626433668691071,
+            0.0019294643928566657,
+            0.008641558521764407,
+            -0.015057688533773374,
+            0.019407986696657772,
+            -0.028126197703647868,
+            0.012364374199657305,
+            -0.0044278434183731225,
+            -0.007251147925369614,
+            -0.03496220945653279,
+            0.006389489026974697,
+            0.008941259259632024,
+            0.003941752277454314,
+            0.023003838655784314,
+            -0.024528514847470928,
+            -0.018361931455105486,
+            0.0037791061321962543,
+            0.00047162060848090104,
+            -0.00711894577052754,
+            0.0010138871688747137,
+            0.004128392769213672,
+            -0.00674546985015164,
+            0.0017307097443508704,
+            0.0033354800298095546,
+            -0.002829898077850263,
+            0.013252947102333348,
+            0.0081724269619999,
+            -0.021449979263734438,
+            0.001368791856860285,
+            0.01590656734732547,
+            0.01452008402429858,
+            -0.02391847163018108,
+            0.006331397860689212,
+            -0.02064812934794864,
+            -0.000631340242718475,
+            -0.031272866000888556,
+            0.016859793097564958,
+            -0.0069008624795683276,
+            0.01899718801899956,
+            -0.0015377436942771224,
+            -0.0065810685260109004,
+            -0.012822620655323715,
+            0.02137604937338179,
+            -0.0019121137078344135,
+            -0.0025598772127157477,
+            0.015678293057066646,
+            -0.025700865215693987,
+            0.017825186616033165,
+            -0.0026230848396158204,
+            -0.017342967698019966,
+            0.017199139867162275,
+            0.004536019178633225,
+            0.007224666811309087,
+            -0.023505589665315567,
+            0.0007208041198255396,
+            0.003900164485347562,
+            -0.017250324025622685,
+            0.004470862192961212,
+            -0.030807400122398566,
+            0.0006579147990327597,
+            -0.013222907940559794,
+            -0.0020514994691204847,
+            -0.0046763235724575445,
+            -0.014050035576183385,
+            -0.039181147615786484,
+            -0.005429639279537522,
+            0.016065742005148733,
+            -0.0009029636981539927,
+            0.01800715198737081,
+            -0.002730931639143803,
+            0.002947680833843522,
+            0.004912692811020822,
+            0.00818378920043427,
+            0.006915895602313347,
+            0.02274116906914758,
+            -0.0057854922886970365,
+            0.00471999900678522,
+            0.007058014277137233,
+            0.014638081356210031,
+            -0.01719565951201012,
+            -0.015170377598372723,
+            -0.016015921063369496,
+            -0.006351213642269936,
+            0.01739665344303718,
+            -0.017488885259028475,
+            0.013303332445465686,
+            -0.00429428898520701,
+            0.014825338166443021,
+            -0.008686427208690176,
+            -0.01575945086724703,
+            0.0028564767084934386,
+            -0.007570417028925067,
+            0.001007670647768344,
+            -0.02323908008484089,
+            0.02019729564706863,
+            -0.00847581038203716,
+            0.010508144473197669,
+            0.01729824052022313,
+            -0.021364529179605813,
+            -0.009742851959432839,
+            -0.014342281441152413,
+            0.029896210049029176,
+            0.013925163184699139,
+            -0.009392252140772913,
+            0.003927230832659267,
+            0.008873561269230905,
+            0.011529536139797032,
+            -0.012209070214071445,
+            -0.0027679974125029362,
+            0.002411914349172495,
+            -0.0062056207925691356,
+            -0.002025687569076897,
+            0.03090134105002816,
+            0.0038285457265343245,
+            0.024877474647546086,
+            -0.013214997316123852,
+            -0.0033763842539861907,
+            0.00023379106197363714,
+            0.005134764372494098,
+            0.01183867975321486,
+            -0.0017966997954747142,
+            -0.008029791503028367,
+            -0.01179500476055557,
+            0.02599911330072515,
+            -0.021852083993691557,
+            0.007024505421035273,
+            0.026784024996764876,
+            0.001390202993990949,
+            0.0029956956045771046,
+            -0.013131691460546865,
+            -0.006298736864509989,
+            0.02523640664202355,
+            0.010030824059858138,
+            0.0041523782603032225,
+            -0.026465819482580182,
+            0.0032128899272377593,
+            -0.0061901409332948115,
+            -0.008681758126710213,
+            0.009866461769956222,
+            -0.0028095454857224164,
+            0.01188012041754054,
+            -0.012191996383095982,
+            0.028459939543805705,
+            0.010711502132045176,
+            0.01047154118931583,
+            0.025888652452698974,
+            -0.02079614305315134,
+            0.0042203106489994615,
+            -0.010829796140534066,
+            0.0006773493086316061,
+            0.005596640511307504,
+            -0.027510562413905437,
+            0.001969898603871003,
+            0.010390866563404346,
+            -0.005642706366801436,
+            -0.01882994179544761,
+            0.01187970346127799,
+            -0.019814385265412387,
+            0.024479579275617887,
+            0.003231341626501863,
+            0.020056914356101588,
+            -0.0053276448188067154,
+            0.009834685132275868,
+            0.0002742942778577165,
+            0.0031361331890882127,
+            -0.0008074598092128465,
+            0.009009012303347016,
+            0.005786777410144894,
+            0.013719989127068968,
+            -0.025528370602188686,
+            0.006821572597883499,
+            -0.0029197127345352626,
+            -0.030539872624857384,
+            -0.013383097680839732,
+            0.010291485622655026,
+            0.012218533188117704,
+            -0.009144425807890594,
+            -0.010378679518611277,
+            0.0030042579827305603,
+            0.005748243116238633,
+            -0.006692403885450407,
+            0.006913672151000981,
+            0.01629516912493545,
+            -0.002533517074771728,
+            -0.0019083178803585312,
+            -0.0035007340321378893,
+            -0.01154609935927537,
+            -0.010016998439072824,
+            0.023193275321990937,
+            -0.00482990342915347,
+            0.017943025772239603,
+            0.025873224580389702,
+            0.0039876567755322,
+            -0.0057577932736143506,
+            -0.005460363755206188,
+            0.01478435999841998,
+            0.014195457012279476,
+            0.004669295894868759,
+            -0.009702220679239768,
+            0.007974305230005956,
+            -0.007760086293456636,
+            0.008375077126777163,
+            -0.020260917110786147,
+            0.002199401871514877,
+            0.0026191373298767766,
+            -0.0022089262577237776,
+            -0.015213807046175918,
+            -0.01956788452420682,
+            -0.021898368819136302,
+            0.012634659463647929,
+            0.00034643102239488474,
+            -0.009241772775582378,
+            -0.013645250997151429,
+            0.0016665106990117016,
+            0.0025630500346914592,
+            0.0002095521560180936,
+            -0.007783022628694597,
+            0.009546147813008391,
+            -0.01265287317178631,
+            0.007445223311736028,
+            -0.004007029181436962,
+            0.0051846833906574515,
+            -0.0016916014415925766,
+            -0.016413938859402438,
+            -0.0210868129634149,
+            0.01920794362224444,
+            -0.012897042227841011,
+            0.02269303737601583,
+            -0.0014935878147854765,
+            3.062680855559304e-05,
+            0.0033132626755401546,
+            0.007274487651476732,
+            0.012110042834251237,
+            0.011426424896797977,
+            0.004307086698938411,
+            0.008206932254416481,
+            0.004673427176334148,
+            -0.013124571552764586,
+            0.004324963759928715,
+            0.0039216033705804084,
+            -0.010449887922072297,
+            0.005298323150690876,
+            -0.0010196552573491,
+            0.006128327172621373,
+            0.012167492341907742,
+            0.01923525204293908,
+            0.01908107550907106,
+            -0.0012031208519043073,
+            -0.0010601912677170903,
+            0.014474374858125607,
+            0.004540954805526089,
+            -0.004883537463424679,
+            -0.002772988629590886,
+            -0.013363713078069854,
+            -0.002572403646338979,
+            0.011541515690646252,
+            -0.02376161747414617,
+            -0.018215231053002656,
+            0.021831219299143226,
+            -0.004991882316041097,
+            0.033796174834149546,
+            -0.013140970283091536,
+            -0.016197177724633614,
+            -0.00464190076219332,
+            0.0026389653040644607,
+            -0.013788049032378498,
+            -0.010300859744205445,
+            0.010545541278382187,
+            0.013848738866656793,
+            0.009903590151120531,
+            -0.0005984085581408575,
+            0.02922816576443645,
+            0.0019900932217491907,
+            0.02245395592422119,
+            0.0018978407151245295,
+            -0.007029294098689854,
+            -0.01440575120349887,
+            0.010840472832356266,
+            0.005529335652955532,
+            0.009050748012161991,
+            0.017791269808484802,
+            0.014502057454389247,
+            0.006265567405788131,
+            0.013928705812292101,
+            0.0012903002372833693,
+            0.0010858377120770652,
+            -0.002355229463060164,
+            -0.01692711767136341,
+            -0.01456786846097011,
+            -0.01627640642218236,
+            0.003873864312554098,
+            0.012620350492241247,
+            0.00525890570023481,
+            0.0007393789113866752,
+            -0.007153210751899716,
+            0.009254566395058412,
+            0.0065087881512835375,
+            -0.012279152412209152,
+            -0.016530068283509593,
+            0.004060671955726967,
+            -0.02210570272789274,
+            0.007849467899009437,
+            0.01554233481902349,
+            0.00219885032769439,
+            -0.002655449524219968,
+            0.0006843524304842362,
+            0.006272944169764281,
+            0.005531927368102191,
+            -0.0036710328198476905,
+            -0.00544113514775027,
+            0.011264661763762068,
+            -0.006379654234440906,
+            0.009537150372938296,
+            -0.00290031548037575,
+            0.03718190949929003,
+            -0.011976080887998194,
+            0.012120408392610773,
+            -0.001751472085435711,
+            0.008481703962951994,
+            0.008067669857949423,
+            0.010772687289903202,
+            -0.003805089040741351,
+            -0.0022164543131735167,
+            -0.008753625032781523,
+            0.006751953442322185,
+            -0.01582177621255226,
+            0.013631301238130705,
+            -0.00030685726238985963,
+            -0.010728958775974582,
+            0.01055472059121978,
+            0.006788316621149835,
+            0.004413382316882087,
+            -0.003262024961860459,
+            -0.01972283203505511,
+            0.014937729522976615,
+            0.0018799533869990667,
+            0.019247008681353865,
+            -0.023276106189004046,
+            0.029043727846427652,
+            -0.02019024213343764,
+            0.023083643395685142,
+            0.01865875504942157,
+            0.001179436167200093,
+            -0.02070329911663634,
+            -0.007618131527214554,
+            0.014871709122115308,
+            -0.02659818889735726,
+            -0.006234719259825804,
+            0.005986829551040766,
+            -0.014639912579831785,
+            0.00037813423546252645,
+            -0.01924149909676064,
+            0.00010832498610532006,
+            0.011013130149723431,
+            -0.011782407678373089,
+            0.00438725832178584,
+            0.01608349750921335,
+            0.019347883652009713,
+            -0.009760017612545174,
+            0.003869246694835168,
+            -0.024240324890794748,
+            0.010958789926998691,
+            0.020511684636649225,
+            0.008883830392748497,
+            -0.01761944522796127,
+            0.021817973960311446,
+            0.010652131228027521,
+            -0.012575679152857703,
+            -0.0012753095453828826,
+            -0.028191274775392804,
+            -0.012421170815054556,
+            -0.02188248722244861,
+            -0.0011670175621882146,
+            -0.014436683528976771,
+            -0.0029783880907228383,
+            0.0009506986717516226,
+            -0.004719165363484719,
+            -0.01179163390107033,
+            -0.019543732196676634,
+            0.016606805875498898,
+            0.0227313413004497,
+            -0.006738735201626204,
+            -0.014547240981002825,
+            0.013866974327185123,
+            0.031694077725890536,
+            -0.016189209297529324,
+            -0.00542110554672736,
+            0.0028061798782555557,
+            -0.012964876141500577,
+            0.018946046750284537,
+            0.0039419760434645495,
+            0.000322115192590907,
+            -0.01991851369210666,
+            0.002489725351392778,
+            0.010835458511052207,
+            0.01684964045708685,
+            0.0005541314524621196,
+            0.010855852688653808,
+            -0.007549882748861744,
+            0.010096800951420057,
+            -0.011832260836468588,
+            -3.6179995603517093e-06,
+            -0.007800366796516945,
+            0.007001049318404058,
+            -0.004424776169203575,
+            0.034356203288485945,
+            -0.02817071276766299,
+            0.004989631403681216,
+            -0.014539644354369607,
+            -0.010888763939716889,
+            -0.0027322086509797843,
+            0.0004865534340981866,
+            -0.010775238981581461,
+            -0.005162220412866082,
+            -0.0011263271110752205,
+            0.00913081242511139,
+            0.005844327320514489,
+            -0.013886529519306775,
+            0.019171827605726503,
+            -0.030984919318656618,
+            0.004469458202753611,
+            -0.008553175891755315,
+            0.010511707988936336,
+            -0.0151185694875566,
+            0.013830711686954931,
+            0.01301959345025159,
+            0.008205984085944315,
+            0.007740367849216328,
+            -0.01532123052721834,
+            -0.01678314222392063,
+            0.001376468747213867,
+            0.0036953018347776244,
+            -0.00931855481896886,
+            -0.033391028922085425,
+            -0.007072913446166969,
+            0.005331350534425606,
+            0.011614453225043,
+            0.008156128603003278,
+            -0.02345206873048703,
+            0.005003870024809975,
+            -0.024178720646704597,
+            -0.012115112684614331,
+            -0.007750557665424973,
+            -0.011580152272614108,
+            -0.006117298954192534,
+            0.026449765548206453,
+            0.007989237291215073,
+            0.016691517078382376,
+            0.004954133565782356,
+            0.02742809127725913,
+            0.011670893621991404,
+            -0.011736840457474895,
+            -0.003349151747880166,
+            0.00035419034843196817,
+            0.005507651806218584,
+            -0.008706590052024251,
+            0.011793241980257321,
+            -0.0013295035978611142,
+            -0.0065061682758755625,
+            -0.004398354347597199,
+            0.009372816666628725,
+            0.014188676762501037,
+            -0.003007980559117429,
+            -0.008724255229333351,
+            0.015749776067989826,
+            -0.030899793617096834,
+            -0.015215033379100091,
+            0.0016152308157299517,
+            -0.007813175908301852,
+            0.017806770680528907,
+            0.01169056345888971,
+            -0.026835412610223673,
+            -0.004538235170332766,
+            0.010532316940606419,
+            0.004755930024691789,
+            0.0095408889025506,
+            0.00954782703036282,
+            0.012079064562163901,
+            -0.007829549014214643,
+            0.01046722658021748,
+            -0.011519369840999348,
+            -0.007092961453184232,
+            -0.007241799314569802,
+            -0.008077934089061052,
+            0.021467547268987214,
+            -0.025816030568238717,
+            -0.006558853543772633,
+            0.01134670818701549,
+            0.013027491377727365,
+            -0.002301991242932548,
+            -0.006299073927206214,
+            -0.017357989759336386,
+            0.008311464809363397,
+            -0.013922568511594478,
+            0.025633907424998367,
+            -0.004854655498817219,
+            -0.005336807899100369,
+            -0.005228056880188717,
+            0.01958682721272445,
+            -0.016774409737540904,
+            0.02072573220949266,
+            -0.014449250679704904,
+            -0.007857351393427215,
+            0.001673929749689194,
+            0.006890008607998747,
+            -0.002948835017234416,
+            0.03734128725793149,
+            -0.00556474133167756,
+            -0.0014544201137417158,
+            0.02209084868209573,
+            -0.01883610080898967,
+            0.017901421953179442,
+            0.01563604483998435,
+            0.012327159100148019,
+            0.0018161034827768648,
+            -0.010253034391420751,
+            0.019979728662282904,
+            0.020823925491027635,
+            0.0139490143219451,
+            0.009577521724445478,
+            -0.013478761809038993,
+            0.014934933784421556,
+            -0.0003725422088504494,
+            -0.0024362115214204105,
+            0.012785534912074068,
+            0.016441944143888537,
+            -0.02164522371903646,
+            -0.013152102018736346,
+            0.0063548333449661805,
+            -0.016547629104776748,
+            -0.024875443975589356,
+            -0.008991095813132973,
+            -0.021885127280977718,
+            -0.023336714382596175,
+            0.00688008599534903,
+            0.016482610587564443,
+            0.0035797836729533367,
+            0.013111012464332918,
+            0.006820471349350837,
+            0.018274678506489214,
+            -0.00980248993077149,
+            -0.01582838461385605,
+            -0.0009454586621769173,
+            -0.0004906493597104733,
+            0.009787688665732045,
+            -0.01835583040075359,
+            -0.013977939077760832,
+            0.004340902272034388,
+            -0.020118484730757877,
+            0.01421736173162507,
+            -0.006428386730058143,
+            0.016756376129345597,
+            -0.0004630315831872392,
+            0.011701948921076007,
+            0.02621773278421472,
+            -0.001391567406453007,
+            0.01262026995055899,
+            0.004696906079976175,
+            -0.0030577364576645635,
+            -0.010624066224426196,
+            0.0009420498021845824,
+            -0.0044079288776429516,
+            -0.0004845606666642234,
+            0.006481113327122954,
+            0.020269778435314,
+            -0.03321232649491074,
+            -0.009425498978427855,
+            0.01246754422779043,
+            -0.001658574527323383,
+            -0.005816985985181387,
+            0.013050062223188965,
+            0.00011195028631856881,
+            0.016573827662919836,
+            -0.0017453203044868104,
+            0.01417371165305189,
+            -0.008165910519539772,
+            -0.0007461950113024975,
+            0.0016492309267666983,
+            0.005076015595439873,
+            0.005829299318055953,
+            0.012146800191035666,
+            0.01148884639506138,
+            -0.021051327863689618,
+            -0.0026357096198729156,
+            0.0036146148489310443,
+            -0.010673282438301611,
+            0.014967444393862347,
+            0.005737933867764028,
+            0.01391411712868596,
+            0.0029556189893790237,
+            0.01167806258173173,
+            -0.024440974650111174,
+            -0.01029944007931845,
+            0.01385509328958837,
+            0.00869671728965227,
+            -0.00998315356917989,
+            -0.016580053751364735,
+            -0.007061005072623325,
+            -0.00477175200214453,
+            -0.0021182980051220428,
+            -0.009871524928312118,
+            -0.0005933941177237234,
+            0.006284920058476413,
+            5.466820734762947e-05,
+            -0.012842295882002122,
+            0.015810530885294202,
+            0.004572385791133295,
+            0.003685966629168175,
+            0.005093714133656017,
+            0.011877541830396259,
+            -0.013130170365831912,
+            0.01258190257102377,
+            0.011433121961378686,
+            0.01142665981860032,
+            -0.013710025169815697,
+            -0.001774030746825879,
+            -0.01109881340948764,
+            0.0065684584345163735,
+            0.009983753894796107,
+            0.005346443477816739,
+            -0.002062387245572686,
+            0.004950044847352647,
+            0.006105199826576752,
+            -0.06391185159544624,
+            -0.02131511418723256,
+            0.0027165737784844413,
+            0.033389688791557785,
+            -0.007167593552251769,
+            -0.0037646833460996797,
+            0.002883510089742145,
+            0.01356887065340832,
+            0.009730171586767032,
+            0.00418566245984083,
+            0.020320934773473808,
+            -0.008814826860651366,
+            -0.003014193747732543,
+            0.001559573791453607,
+            0.003499420226345028,
+            -0.008703837291821111,
+            0.0044844701302276645,
+            0.0053963630238108035,
+            0.027764369548110287,
+            0.001469968540767627,
+            0.015793086297019008,
+            -0.008505290826588508,
+            -0.004075888558568453,
+            -0.009166710762668385,
+            -0.002860047211143503,
+            -0.011127820563172684,
+            0.003937110717404271,
+            0.02054186721680431,
+            -0.006126180963243227,
+            0.009346433819831456,
+            0.01057504086030076,
+            0.0001650851447316893,
+            -0.012033423322524951,
+            0.0049587335923558,
+            -0.0032240390628760535,
+            -0.005568976687946761,
+            -0.0024667040429312737,
+            -0.03474027725033016,
+            0.026855947684394643,
+            -0.004369626674503285,
+            0.009252512430957932,
+            -0.005081834388513357,
+            -0.0037896610731020337,
+            -0.014201609270756739,
+            0.017871088029615106,
+            -0.008249444723433136,
+            -0.020092257756971717,
+            0.009532994544485572,
+            -0.004991989149115383,
+            0.006464417738670472,
+            -0.02643078108339741,
+            0.0029138922065812495,
+            -0.017232410749655366,
+            -0.003999293090834831,
+            -0.006924488971625995,
+            0.01318082244178942,
+            -0.007232695119856389,
+            -0.02660711037908311,
+            0.01979251106803667,
+            -0.019485860712345603,
+            -0.011977904020170568,
+            -0.0177530973032872,
+            -0.014524003566945937,
+            0.004874952064727108,
+            -0.015326044325348929,
+            0.002047771741041311,
+            -0.009697340332342246,
+            -0.003652304998155628,
+            -0.0008878698629674222,
+            0.0017782435347417623,
+            -0.013017776298713044,
+            -0.018272697230002275,
+            -0.01353020425912225,
+            -0.00757239449706798,
+            -0.0008250000619957538,
+            0.01299142281772714,
+            -0.001679606555140765,
+            -0.007329941421303076,
+            0.0044605462460153116,
+            0.010231192277666706,
+            0.0006558644644452857,
+            0.0024231733856020414,
+            -0.010728844680699885,
+            -0.014137434553169443,
+            0.005230053181525096,
+            -0.012086468475542818,
+            -0.009459717916037003,
+            -0.0028185227600210556,
+            0.014984527039312847,
+            0.008189464334213754,
+            -0.009139240774931914,
+            -0.005639157579163322,
+            0.014325949891859377,
+            0.009020303519690971,
+            -0.013310205617445609,
+            0.01567574314514385,
+            -0.025369802629268523,
+            0.021964786021408878,
+            0.030536905527039065,
+            0.004945288795196981,
+            -0.01168303908203322,
+            0.02281128628230144,
+            -0.004072286832145419,
+            0.0028278384348551146,
+            0.01740522162288968,
+            -0.042558465579514926,
+            -0.013146562547084278,
+            0.002903743316339479,
+            -0.015297177099602701,
+            -0.010963312697387696,
+            0.008804938803398296,
+            -0.0026727734429658817,
+            0.005273238046953793,
+            0.004257613402172282,
+            0.03512246556126453,
+            0.002258612643297096,
+            -0.0002885708728599208,
+            -0.0019720190285930487,
+            0.012312613129996533,
+            0.010451003939611476,
+            -0.015556620910136185,
+            -0.02387856865043814,
+            -0.004583090523619507,
+            -0.0037828776396525835,
+            -0.0014588238692793338,
+            -0.004361834167147271,
+            -0.008159679676329573,
+            0.012243319920992457,
+            -0.017279964797769668,
+            -0.013748032758693542,
+            0.018394195628121487,
+            -0.006411113522842359,
+            0.019002611495040334,
+            -0.014970362556522149,
+            -0.0077437581301661946,
+            0.0017311115715410025,
+            0.024439691379769795,
+            -0.025013956798520212,
+            0.007359134924497539,
+            -0.0031406470488477937,
+            0.017574824561755627,
+            0.00419409250132525,
+            -0.0046803792187546656,
+            0.010556972371044426,
+            0.015229789583480183,
+            -0.004608293492221483,
+            -0.014449487985966966,
+            -0.01172850291078532,
+            0.016823024209386785,
+            -0.012304887597699588,
+            0.0011773351576246157,
+            0.002262670226244277,
+            -0.0023810578015456716,
+            0.03477119000959688,
+            -0.03442178291363675,
+            0.004062693040714608,
+            0.0029249583507755944,
+            0.003291349443702718,
+            0.006461525089528308,
+            0.006047777261425826,
+            -0.020642802018978404,
+            0.008916088381876748,
+            -0.03316762661930299,
+            -0.006367254689957291,
+            -0.005404276207844267,
+            0.0009189006423526284,
+            0.009401330264376041,
+            -0.011106443887384533,
+            -0.02259405372588916,
+            -0.006010439054323966,
+            0.0070744551115721095,
+            0.010262607009446879,
+            -0.002856616377476421,
+            0.019719451109535182,
+            0.008426509638139982,
+            0.0015877648339862691,
+            -0.010655792508751512,
+            -0.015527851374834499,
+            -0.015460974859638219,
+            -0.017847462632188386,
+            -0.02236964311096428,
+            0.011841749854626384,
+            -0.00576688153869993,
+            0.009054902130717224,
+            0.014317707346504314,
+            -0.015366657284925976,
+            0.002910211008094564,
+            0.017761685988205086,
+            0.007974806171830457,
+            0.002692237107055498,
+            0.0032405477366987947,
+            0.02685230091638845,
+            -0.0009116884421301454,
+            -0.0032228977180251673,
+            0.003158213042682977,
+            0.015872199488167846,
+            0.0004966905858799325,
+            0.019981401695749582,
+            0.015201538351780685,
+            0.005359075814444414,
+            -0.004668738193285921,
+            0.019853988551405383,
+            -0.009844394286458195,
+            0.013476144827747423,
+            -0.00411789677405284,
+            -0.00830656434595337,
+            -0.008496132520586954,
+            0.006442581751999304,
+            -0.008138786054348571,
+            0.006977337946948795,
+            0.02419128791801387,
+            -0.0005734827808485311,
+            -0.0028842142002271044,
+            0.005503033191779872,
+            -0.01696639779775455,
+            -0.00389835761601045,
+            -0.03366077608692946,
+            -0.009942613075976742,
+            -0.01708630689497001,
+            -0.02910756562213045,
+            0.026536268897199546,
+            0.00219827495662445,
+            0.02586503862171508,
+            -0.01905122645609761,
+            0.008935637454859214,
+            -0.01385806889332683,
+            0.029747093931541325,
+            0.009163959012862627,
+            0.0008611776492791918,
+            0.003598831885031635,
+            -0.02103045281939351,
+            -0.01150941825152158,
+            0.027341774725604333,
+            -0.0006700608710615124,
+            -0.012089044078073347,
+            0.022396400579038037,
+            0.017491565611764764,
+            0.0051760367056280536,
+            -0.00559536785318233,
+            0.01979262595019399,
+            0.002306380546194741,
+            0.011973863166872444,
+            -0.0028136750606972223,
+            0.012911235736067303,
+            -0.01510421118359169,
+            -0.003338602157155174,
+            0.039758808572572274,
+            0.007290642519242007,
+            -0.001802921446521437,
+            -0.013696774990705774,
+            0.003941198181491921,
+            -0.0008308293948698925,
+            0.0200351513103389,
+            -0.00014327587644797734,
+            0.028291423013030337,
+            -0.0001085859276231927,
+            -0.010723064585989723,
+            0.020443848393732245,
+            -0.017860296930396913,
+            0.04540901629084378,
+            -0.018671578423261854,
+            0.000852031295629291,
+            -0.010639853151638835,
+            -0.019017194262730474,
+            0.003861296738239369,
+            0.0010977143012920602,
+            0.0008387482016061222,
+            0.0060155816314618,
+            -0.010780239332444652,
+            0.002152214497259421,
+            0.010556338812873054,
+            -0.027636192450621728,
+            0.0010938992111792307,
+            -0.014385202899929138,
+            -0.01353192238925814,
+            -0.026667778226211785,
+            -0.00017403398236549974,
+            -0.015357674131635848,
+            -0.013956722505869091,
+            -0.01985038526848704,
+            -0.008927903879720431,
+            -0.009580214158077466,
+            -0.0020314441852913386,
+            -0.004755584878514854,
+            -0.00018388548107372635,
+            -0.002918855567842974,
+            -0.0015941852262316128,
+            -0.014230272061190995,
+            0.02133992212037431,
+            0.015283345362445234,
+            0.004642765060980717,
+            0.012829570028541471,
+            0.01708952142208596,
+            -0.004770647543823141,
+            0.006604762788847952,
+            -0.0025369836845081078,
+            0.008368000846861623,
+            0.014785031726639733,
+            0.012074738887486398,
+            0.02124518119032885,
+            -0.015459818988572564,
+            -0.004992865135357168,
+            -0.012722391036055767,
+            -0.012210202603282509,
+            -0.017850065946244383,
+            -0.004300429148015694,
+            -0.0014387298144714717,
+            -0.023128300275416423,
+            0.004468952290723408,
+            -0.0026602931688877443,
+            -0.005768852559887139,
+            0.007815212168719318,
+            0.0005862972228454734,
+            -0.002265719947629544,
+            -0.00019323737674204774,
+            -0.02735820445266249,
+            0.00024009122317215594,
+            -0.00018701993387558173,
+            -0.014663710240203917,
+            -0.00450797561624596,
+            0.010584731989847334,
+            -0.011966527748437568,
+            -0.00602367292291343,
+            0.0111155226518017,
+            0.0015565179432852256,
+            0.006062862961835896,
+            0.016952559697728433,
+            0.0005012157272316869,
+            0.011160668242334889,
+            0.014993782560797772,
+            -0.01591303023837098,
+            -0.015356412848444082,
+            -0.0236025426434855,
+            -0.004514538154596937,
+            -0.009753334017065119,
+            0.007553951535816575,
+            -0.004616595532996391,
+            -0.013023153809236325,
+            0.006040285197946957,
+            -0.0006343451398642784,
+            0.008338564073158257,
+            -0.017608246105292328,
+            -8.346168119689692e-05,
+            -0.020944870308705715,
+            0.009230123321316848,
+            -0.00531237224627929,
+            -0.00445985089300067,
+            0.01410675485522911,
+            0.02895945580213926,
+            -0.007799830351886844,
+            -0.009455128322575888,
+            -0.005663072244883114,
+            0.012817229945992764,
+            0.012474431030896143,
+            -0.004423141749110039,
+            -0.0018294512146253298,
+            -0.012817101093394574,
+            0.0004755435377918658,
+            -0.01129485262081729,
+            0.0062747111125005155,
+            0.017199374798326412,
+            -0.021965117839794063,
+            -0.01296097993683748,
+            -0.016765996091978336,
+            -0.016882855472266523,
+            0.004372970343291638,
+            0.017290565252244258,
+            0.022670788364378462,
+            -0.008456547449078897,
+            -0.019564049850942417,
+            0.006898353991539946,
+            -0.01620125883985687,
+            0.0037108638122592746,
+            0.12887455743463638,
+            -0.0028864852421660366,
+            -0.011487804449328495,
+            -0.022683334123299446,
+            -0.005467192611880829,
+            0.01210557696712196,
+            -0.01555159294496261,
+            0.06709672179195036,
+            -0.015071089552631105,
+            0.004364902817496082,
+            -0.004899872946473657,
+            0.0006310685609937571,
+            -0.017610935588191347,
+            -0.005962991715597585,
+            -0.007768819050529025,
+            -0.02688607975562596,
+            0.000883763319535686,
+            -0.009935313912803587,
+            -0.012883104064962595,
+            0.002126368813790577,
+            -0.0013697265269800346,
+            -0.016131186838419288,
+            0.01575514418037863,
+            -0.020677990768697592,
+            -0.007251917191831732,
+            -0.0029325623780992118,
+            -0.00600710212646933,
+            0.014573093764769918,
+            -0.03253336117394778,
+            0.002973725493330224,
+            -0.013601417220439602,
+            0.011905788192118131,
+            0.001542117126832946,
+            -0.010789486980776022,
+            -0.013862139885514524,
+            0.007086571810314664,
+            0.015658913897173155,
+            0.017039624764452546,
+            0.01901374401756777,
+            0.004797954569087795,
+            0.003928999581093576,
+            -0.008046722893614655,
+            0.021974371139905104,
+            -0.00697335955527197,
+            0.0008975473383295958,
+            -0.00970589295353857,
+            -0.004346869073499034,
+            0.0031279458769925416,
+            0.02303983493541673,
+            -0.012560380384977343,
+            -0.01536378830925542,
+            0.0012035987157572355,
+            0.033313215063595594,
+            -0.010845084209743817,
+            0.013587126023659577,
+            0.0029867620644300073,
+            0.020679673435988446,
+            -0.0006640787253300063,
+            0.0025071780490108306,
+            0.022554193713122565,
+            -0.018082906254599396,
+            0.018230000220500718,
+            -0.004451773302171552,
+            0.01915083025790679,
+            -0.004183521043835777,
+            0.015019320101074747,
+            -0.027014055979204272,
+            -0.005362044589023088,
+            -0.012355559881051253,
+            0.003995309003198048,
+            0.020680455787871383,
+            0.0054391755324872375,
+            0.009865676762244514,
+            -0.01645386866657314,
+            -0.01760657627715638,
+            0.022969828924328126,
+            -0.021566687194482292,
+            -0.015826424849757402,
+            0.00012804774676298237,
+            -0.015021970627042762,
+            0.006000610438452896,
+            0.010126179023323158,
+            -0.02912121211344025,
+            -0.02046150879033127,
+            0.004881850039543051,
+            -0.0023469840669773042,
+            -0.00915786433296249,
+            0.006547377770403066,
+            -0.022329063637464074,
+            -0.015702777677743352,
+            -0.005692904224111613,
+            0.01207999389934878,
+            -0.008429160038192635,
+            0.010075061963131637,
+            -0.020106712664936504,
+            0.0036787045650556907,
+            -0.002764965946892891,
+            0.010307702251974629,
+            0.004697169247387421,
+            0.011192285826515207,
+            0.0030113901179382946,
+            0.0014774681536380047,
+            -0.02778270000997313,
+            0.010043261430740039,
+            -5.178406450222847e-05,
+            0.04216636255860709,
+            0.030977096295679525,
+            0.004479857028809617,
+            -5.798020745008712e-05,
+            -0.009996471945562616,
+            0.012476009017187798,
+            -0.019306907391548302,
+            -0.005092322343599383,
+            0.01856463597985508,
+            0.02195467141594795,
+            -0.02236232264139305,
+            0.0052217593371508145,
+            -0.016766955100332014,
+            0.016133299654723637,
+            -0.011208763137897244,
+            0.006248024409576809,
+            0.011601988603146743,
+            0.0022330878390492527,
+            0.010317781892717903,
+            -0.005895142237225119,
+            0.019670196605183973,
+            0.01186677200787666,
+            -0.002339367025714548,
+            0.02367408853110795,
+            0.006858134385159451,
+            0.004479683673962342,
+            0.004798719889930319,
+            -0.009604173975576537,
+            -0.009045602998213713,
+            0.009230969488015385,
+            0.004083616606774546,
+            -0.007944628589259331,
+            0.011836323287097144,
+            0.015100350195476968,
+            0.01885317701149666,
+            -0.0011707847499298092,
+            -0.00684548433712413,
+            -0.006550059418141426,
+            0.001899758393077906,
+            -0.024577381680573596,
+            -0.02855511854789239,
+            0.021238935710272536,
+            -0.010427135883958709,
+            -0.019187769881105637,
+            -0.0015864640188683347,
+            0.016360037306061392,
+            -0.008853972319021563,
+            -0.01627167079143546,
+            0.001270414822216867,
+            0.02255755055961612,
+            -0.005959411125391464,
+            0.006009654905468478,
+            0.005909233773418581,
+            0.005874361519568704,
+            -0.0027859463941992753,
+            -0.024599789577567636,
+            0.0036612374968223453,
+            0.03221346865585554,
+            0.016461563205582293,
+            0.016139782455492073,
+            -0.007158168059808459,
+            0.001838455290310879,
+            0.007530954381521755,
+            0.005438791523412295,
+            0.015674000912615536,
+            0.008604106101542998,
+            -0.006990462920227135,
+            0.00244736628387359,
+            0.012213048255418866,
+            -0.007329299946651028,
+            0.017424711328323798,
+            0.001973882532418776,
+            -0.0133178791530314,
+            -0.008373452013951822,
+            -0.010462150707776556,
+            -0.014575483571750984,
+            0.0027958704799249897,
+            -0.011401916623878957,
+            0.0008016041012669959,
+            0.003953958717263787,
+            0.010260753041492368,
+            0.013961649189650642,
+            -0.0019275791443041316,
+            0.017019168629298462,
+            0.011534594243860177,
+            -0.0017859096307142257,
+            -0.002970739203219076,
+            -0.03142778833370656,
+            0.010207789755749909,
+            -0.014730553506704018,
+            -0.0037611416875830746,
+            0.01825790347981639,
+            -0.007766890847082858,
+            -0.0032778433418881666,
+            -0.021003405429807113,
+            0.0024512381432602736,
+            0.008768616692363321,
+            0.01558713989712387,
+            -0.012819542714293536,
+            0.009201155584074711,
+            -0.0012254337002053017,
+            -0.0057451574939563045,
+            0.00977251392462179,
+            0.0042356831792011935,
+            -0.0033433823868435786,
+            0.011499764125503874
+          ],
+          "title": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer",
+          "keyphrases": [
+            "Sparsely-Gated Mixture-of-Experts Layer",
+            "conditional computation",
+            "neural network",
+            "language modeling",
+            "machine translation"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "294d1382-75f0-4a45-bc91-9727e57552f3",
+        "properties": {
+          "page_content": "1 Introduction And Related Work 1.1 Conditional Computation\n\nExploiting scale in both training data and model size has been central to the success of deep learning. When datasets are sufficiently large, increasing the capacity (number of parameters) of neural networks can give much better prediction accuracy. This has been shown in domains such as text (Sutskever et al., 2014; Bahdanau et al., 2014; Jozefowicz et al., 2016; Wu et al., 2016), images (Krizhevsky et al., 2012; Le et al., 2012), and audio (Hinton et al., 2012; Amodei et al., 2015). For typical deep learning models, where the entire model is activated for every example, this leads to a roughly quadratic blow-up in training costs, as both the model size and the number of training examples increase. Unfortunately, the advances in computing power and distributed computation fall short of meeting such demand.\n\nVarious forms of conditional computation have been proposed as a way to increase model capacity without a proportional increase in computational costs (Davis & Arel, 2013; Bengio et al., 2013; Eigen et al., 2013; Ludovic Denoyer, 2014; Cho & Bengio, 2014; Bengio et al., 2015; Almahairi et al., 2015). In these schemes, large parts of a network are active or inactive on a per-example basis. The gating decisions may be binary or sparse and continuous, stochastic or deterministic. Various forms of reinforcement learning and back-propagation are proposed for trarining the gating decisions.\n\nWhile these ideas are promising in theory, no work to date has yet demonstrated massive improvements in model capacity, training time, or model quality. We blame this on a combination of the following challenges:\n\nModern computing devices, especially GPUs, are much faster at arithmetic than at branching. Most of the works above recognize this and propose turning on/off large chunks of the network with each gating decision.\n\nLarge batch sizes are critical for performance, as they amortize the costs of parameter transfers and updates. Conditional computation reduces the batch sizes for the conditionally active chunks of the network.\n\nNetwork bandwidth can be a bottleneck. A cluster of GPUs may have computational power thousands of times greater than the aggregate inter-device network bandwidth. To be computationally efficient, the relative computational versus network demands of an algorithm must exceed this ratio. Embedding layers, which can be seen as a form of conditional computation, are handicapped by this very problem. Since the embeddings generally need to be sent across the network, the number of (example, parameter) interactions is limited by network bandwidth instead of computational capacity.\n\nDepending on the scheme, loss terms may be necessary to achieve the desired level of sparsity per-chunk and/or per example. Bengio et al. (2015) use three such terms. These issues can affect both model quality and load-balancing.\n\nModel capacity is most critical for very large data sets. The existing literature on conditional computation deals with relatively small image recognition data sets consisting of up to 600,000 images. It is hard to imagine that the labels of these images provide a sufficient signal to adequately train a model with millions, let alone billions of parameters. In this work, we for the first time address all of the above challenges and finally realize the promise of conditional computation. We obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets.\n\n1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1). All parts of the network are trained jointly by back-propagation.\n\nWhile the introduced technique is generic, in this paper we focus on language modeling and machine translation tasks, which are known to benefit from very large models. In particular, we apply a MoE convolutionally between stacked LSTM layers (Hochreiter & Schmidhuber, 1997), as in Figure 1.\n\nThe MoE is called once for each position in the text, selecting a potentially different combination of experts at each position. The different experts tend to become highly specialized based on syntax and semantics (see Appendix E Table 9). On both language modeling and machine translation benchmarks, we improve on best published results at a fraction of the computational cost.\n\n1.3 Related Work On Mixtures Of Experts\n\nSince its introduction more than two decades ago (Jacobs et al., 1991; Jordan & Jacobs, 1994), the mixture-of-experts approach has been the subject of much research. Different types of expert architectures hae been proposed such as SVMs (Collobert et al., 2002), Gaussian Processes (Tresp, 2001; Theis & Bethge, 2015; Deisenroth & Ng, 2015), Dirichlet Processes (Shahbaba & Neal, 2009), and deep networks. Other work has focused on different expert configurations such as a hierarchical structure (Yao et al., 2009), infinite numbers of experts (Rasmussen & Ghahramani, 2002), and adding experts sequentially (Aljundi et al., 2016). Garmash & Monz (2016) suggest an ensemble model in the format of mixture of experts for machine translation. The gating network is trained on a pre-trained ensemble NMT model. The works above concern top-level mixtures of experts. The mixture of experts is the whole model. Eigen et al. (2013) introduce the idea of using multiple MoEs with their own gating networks as parts of a deep model. It is intuitive that the latter approach is more powerful, since complex problems may contain many sub-problems each requiring different experts. They also allude in their conclusion to the potential to introduce sparsity, turning MoEs into a vehicle for computational computation.\n\nOur work builds on this use of MoEs as a general purpose neural network component. While Eigen et al. (2013) uses two stacked MoEs allowing for two sets of gating decisions, our convolutional application of the MoE allows for different gating decisions at each position in the text. We also realize sparse gating and demonstrate its use as a practical way to massively increase model capacity.\n\n",
+          "embedding": [
+            -0.011026207357645035,
+            -0.0030965988989919424,
+            -0.017856089398264885,
+            0.0010101930238306522,
+            0.02138679102063179,
+            0.0306765865534544,
+            -0.014028750360012054,
+            0.025452885776758194,
+            -0.030126724392175674,
+            0.021791953593492508,
+            0.010809156112372875,
+            -0.003031483618542552,
+            0.0071120485663414,
+            -0.010223117657005787,
+            0.014021515846252441,
+            0.00018878943228628486,
+            0.01843489333987236,
+            0.003255769843235612,
+            0.03244193643331528,
+            -0.030329305678606033,
+            0.006352368742227554,
+            -0.010158002376556396,
+            -0.03186313435435295,
+            -0.031226450577378273,
+            0.018391482532024384,
+            0.023152140900492668,
+            0.040400486439466476,
+            -0.0036120957229286432,
+            -0.027333997189998627,
+            0.01920180767774582,
+            0.016163088381290436,
+            0.000524540722835809,
+            -0.03342590481042862,
+            -0.03348378464579582,
+            -0.058574918657541275,
+            -0.044220589101314545,
+            0.04589911922812462,
+            0.018217841163277626,
+            -0.003402279457077384,
+            0.018926875665783882,
+            0.037419646978378296,
+            -0.001556438859552145,
+            -0.03244193643331528,
+            -0.025858048349618912,
+            0.0014379650820046663,
+            0.016626130789518356,
+            -0.016466960310935974,
+            -0.03426516801118851,
+            -0.015902627259492874,
+            -0.01490419078618288,
+            0.02338366210460663,
+            -0.025742286816239357,
+            0.01846383325755596,
+            -0.000727121951058507,
+            -0.040603067725896835,
+            0.033049680292606354,
+            0.026639433577656746,
+            -0.004460404627025127,
+            0.04691202566027641,
+            0.04639110341668129,
+            0.0038635137025266886,
+            0.006019556894898415,
+            -0.04427846893668175,
+            -0.0017617333214730024,
+            0.02446891926229,
+            -0.015251473523676395,
+            -0.024526799097657204,
+            0.021010568365454674,
+            0.018594063818454742,
+            -0.00322140334174037,
+            -0.037680111825466156,
+            0.00494153518229723,
+            -0.03119751065969467,
+            0.01049804873764515,
+            0.005907413549721241,
+            0.01499101147055626,
+            0.02413610741496086,
+            0.04008214548230171,
+            0.021242089569568634,
+            0.011192613281309605,
+            -0.003957569133490324,
+            0.022862739861011505,
+            -0.006706885993480682,
+            0.010975562036037445,
+            0.027218235656619072,
+            -0.02911381796002388,
+            0.005502250976860523,
+            -0.06227926164865494,
+            0.004485727287828922,
+            -0.013276305980980396,
+            -0.0250043123960495,
+            0.013565707951784134,
+            -0.0021397643722593784,
+            0.0019570793956518173,
+            0.010577634908258915,
+            0.03658038377761841,
+            -0.0005281582125462592,
+            -0.013869579881429672,
+            0.005491398740559816,
+            -0.0063632214441895485,
+            -0.01565663516521454,
+            0.01856512390077114,
+            0.007361657451838255,
+            -0.0035723031032830477,
+            0.014448382891714573,
+            -0.0009423645096831024,
+            -0.023730944842100143,
+            0.026205329224467278,
+            0.008710993453860283,
+            0.08670476824045181,
+            -0.009463437832891941,
+            -0.022775918245315552,
+            -0.001728271134197712,
+            0.03600158169865608,
+            -0.0008876494830474257,
+            -0.011489249765872955,
+            -0.04349708557128906,
+            -0.04118186980485916,
+            0.007216956466436386,
+            -0.022110294550657272,
+            0.03328120335936546,
+            0.03843255341053009,
+            -0.015164652839303017,
+            -0.010158002376556396,
+            -0.05591242015361786,
+            0.07883304357528687,
+            -0.013695938512682915,
+            -0.035162314772605896,
+            -0.008703758008778095,
+            -0.027044596150517464,
+            -0.002848798641934991,
+            0.00745209539309144,
+            -0.01866641454398632,
+            -0.02755104750394821,
+            0.04659368470311165,
+            0.012415335513651371,
+            0.03773799166083336,
+            0.022139234468340874,
+            -0.01235022023320198,
+            -0.013630823232233524,
+            0.03244193643331528,
+            0.0003031935775652528,
+            0.004037154838442802,
+            -0.017783738672733307,
+            -0.007980253547430038,
+            0.005802505649626255,
+            -0.022732509300112724,
+            0.057648833841085434,
+            -0.03794057294726372,
+            0.02171960286796093,
+            -0.030300365760922432,
+            0.031023869290947914,
+            -0.016163088381290436,
+            0.03941652178764343,
+            -0.01902816630899906,
+            0.01389128528535366,
+            -0.0250043123960495,
+            -0.017957380041480064,
+            0.006417484022676945,
+            -0.011178143322467804,
+            -0.016987884417176247,
+            -0.019910842180252075,
+            -0.015613225288689137,
+            0.02323896251618862,
+            0.0025485444348305464,
+            0.03073446825146675,
+            0.0352201946079731,
+            0.019476739689707756,
+            -0.0614689365029335,
+            0.019447797909379005,
+            0.004789599217474461,
+            0.021632781252264977,
+            0.019968722015619278,
+            -0.0019860195461660624,
+            0.03134221211075783,
+            0.009297031909227371,
+            0.02205241471529007,
+            0.03252875804901123,
+            -0.005419048015028238,
+            0.0330207422375679,
+            -0.03001096285879612,
+            -0.01251662615686655,
+            0.02878100611269474,
+            0.008117719553411007,
+            0.025091134011745453,
+            0.008877399377524853,
+            0.015801336616277695,
+            -0.007813847623765469,
+            0.0341494083404541,
+            0.0012118698796257377,
+            0.01969379000365734,
+            0.023962466046214104,
+            0.036725085228681564,
+            -0.030589766800403595,
+            0.0233113132417202,
+            -0.04074776917695999,
+            -0.019375449046492577,
+            0.024092696607112885,
+            0.02338366210460663,
+            0.017002353444695473,
+            -0.03368636593222618,
+            -0.03212359547615051,
+            0.02480173110961914,
+            0.01720493473112583,
+            0.013695938512682915,
+            0.009246386587619781,
+            0.013703173957765102,
+            0.0147884301841259,
+            0.017595628276467323,
+            -0.037882693111896515,
+            0.009969891048967838,
+            0.00405524205416441,
+            -0.009940951131284237,
+            -0.04303404316306114,
+            0.006829881574958563,
+            -0.0030152045655995607,
+            -0.022240525111556053,
+            -0.0033299291972070932,
+            -0.0182033721357584,
+            -0.003758605569601059,
+            -0.002535883104428649,
+            0.01681424304842949,
+            0.021459141746163368,
+            -0.021632781252264977,
+            -0.0020529436878859997,
+            -0.017291756346821785,
+            -0.021835362538695335,
+            -0.03235511854290962,
+            0.003633800894021988,
+            -0.005400960333645344,
+            0.02464256063103676,
+            0.03140009194612503,
+            0.017885029315948486,
+            -0.004778746515512466,
+            0.03938758000731468,
+            -0.03368636593222618,
+            0.016626130789518356,
+            -0.0012697502970695496,
+            0.007488270755857229,
+            0.008862929418683052,
+            -0.024353159591555595,
+            0.04178961366415024,
+            -0.0409214086830616,
+            -0.005690362304449081,
+            0.011330079287290573,
+            0.010035006329417229,
+            -0.010794686153531075,
+            0.0044133770279586315,
+            -0.023354722186923027,
+            -0.02649473212659359,
+            0.0013222043635323644,
+            -0.012762618251144886,
+            -0.016553781926631927,
+            0.02205241471529007,
+            -0.02539500594139099,
+            0.03976380452513695,
+            0.030618706718087196,
+            -0.02490302175283432,
+            -0.01976614072918892,
+            -0.006916702259331942,
+            0.04123975336551666,
+            -0.009108920581638813,
+            -0.01268303208053112,
+            0.02569887787103653,
+            -0.025612056255340576,
+            0.0019950633868575096,
+            0.036956604570150375,
+            0.022587807849049568,
+            -0.004930682480335236,
+            -0.01800079084932804,
+            0.0038237208500504494,
+            -0.05845915898680687,
+            0.0035361277405172586,
+            -0.041471272706985474,
+            0.05614394322037697,
+            -0.03736176714301109,
+            0.0041999430395662785,
+            0.0013303437735885382,
+            -0.012111463584005833,
+            0.04028472676873207,
+            0.022935090586543083,
+            -0.009405557066202164,
+            -0.022804860025644302,
+            0.02480173110961914,
+            0.015468524768948555,
+            -0.004123975057154894,
+            -0.015208062715828419,
+            -0.0024273572489619255,
+            -0.05116623267531395,
+            0.002250098856166005,
+            -0.005104323849081993,
+            -0.002651643706485629,
+            -0.048359036445617676,
+            0.015671106055378914,
+            0.028607364743947983,
+            -0.001820518053136766,
+            -0.011402430012822151,
+            -0.01437603309750557,
+            0.02708800509572029,
+            -0.017566686496138573,
+            -0.042397357523441315,
+            -0.006793706677854061,
+            0.013970870524644852,
+            0.008776108734309673,
+            -0.03209465742111206,
+            0.006746678613126278,
+            -0.003434837330132723,
+            0.02543841488659382,
+            -0.0076691466383636,
+            0.004728101193904877,
+            0.018188901245594025,
+            0.0015220724744722247,
+            0.01614861935377121,
+            -0.01747986674308777,
+            -0.006120847538113594,
+            0.02141573093831539,
+            0.021676192060112953,
+            0.0006932076648809016,
+            -0.016872122883796692,
+            -0.03316544368863106,
+            0.032181475311517715,
+            0.002027621027082205,
+            -0.0637841522693634,
+            -0.01426027249544859,
+            -0.03542277589440346,
+            0.01033164281398058,
+            0.004677455872297287,
+            0.03247087821364403,
+            -0.02228393591940403,
+            0.018246781080961227,
+            -0.011713536456227303,
+            -0.03542277589440346,
+            -0.024512330070137978,
+            0.013666998594999313,
+            -0.030879167839884758,
+            -0.03987956419587135,
+            -0.017118114978075027,
+            0.023991405963897705,
+            -0.013117135502398014,
+            -0.029026998206973076,
+            -0.01969379000365734,
+            -0.031255390495061874,
+            0.0030748937278985977,
+            0.027739159762859344,
+            0.01774032786488533,
+            -0.024946432560682297,
+            -0.034033648669719696,
+            0.0057265376672148705,
+            -0.01178588718175888,
+            -0.0018594063585624099,
+            0.0010210456093773246,
+            0.018261251971125603,
+            0.006489834748208523,
+            -0.011257728561758995,
+            -0.017552217468619347,
+            -0.04413377121090889,
+            -0.008407121524214745,
+            0.0005385585827752948,
+            -0.009904775768518448,
+            0.034699272364377975,
+            0.03180525451898575,
+            -0.012125934474170208,
+            -0.011612245813012123,
+            0.014745020307600498,
+            -0.047201428562402725,
+            -0.015367234125733376,
+            -0.025640996173024178,
+            -0.005306905135512352,
+            0.01684318296611309,
+            0.008928044699132442,
+            -0.003950334154069424,
+            0.042889341711997986,
+            0.002532265381887555,
+            -0.013507827185094357,
+            0.007922373712062836,
+            0.006956494878977537,
+            0.028144322335720062,
+            0.0057265376672148705,
+            -0.03637780249118805,
+            0.04381542652845383,
+            -0.02464256063103676,
+            0.010027770884335041,
+            0.013348656706511974,
+            -0.011728006415069103,
+            -0.013761053793132305,
+            0.018550653010606766,
+            -0.005636099260300398,
+            0.025858048349618912,
+            -0.020779047161340714,
+            0.061526816338300705,
+            -0.006761149037629366,
+            -0.05186079815030098,
+            -0.0147884301841259,
+            -0.04089247062802315,
+            0.021271029487252235,
+            0.013377596624195576,
+            -0.002273612655699253,
+            -0.020344944670796394,
+            -0.013030314818024635,
+            0.019288627430796623,
+            0.004000979475677013,
+            -0.009629843756556511,
+            -0.01235022023320198,
+            0.006366838701069355,
+            -0.01578686572611332,
+            -0.005990616511553526,
+            0.007376127410680056,
+            0.007625736761838198,
+            0.04187643527984619,
+            -0.03142903372645378,
+            -0.0031201126985251904,
+            -0.01594603806734085,
+            -0.048127513378858566,
+            0.0049560051411390305,
+            0.03154479339718819,
+            0.0364067442715168,
+            -0.023947997018694878,
+            0.0003253509057685733,
+            0.013175015337765217,
+            0.0018973903497681022,
+            -0.017653508111834526,
+            0.011113028042018414,
+            0.03336802497506142,
+            0.02457020990550518,
+            -0.0187966451048851,
+            -0.005245407111942768,
+            0.0062438431195914745,
+            -0.0020746488589793444,
+            0.00621852045878768,
+            -0.04031366482377052,
+            0.03637780249118805,
+            -0.01235022023320198,
+            0.02998202294111252,
+            -0.028708655387163162,
+            -0.013102664612233639,
+            -0.03481503203511238,
+            -0.010223117657005787,
+            0.03472821041941643,
+            -0.032876040786504745,
+            -0.007188016548752785,
+            0.017262814566493034,
+            -0.002754742978140712,
+            -0.006724973674863577,
+            0.018550653010606766,
+            -0.04636216163635254,
+            -0.019520148634910583,
+            -0.027536578476428986,
+            0.0008266037912108004,
+            -0.0018811115296557546,
+            -0.05871962010860443,
+            -0.061295293271541595,
+            -0.005350315012037754,
+            -0.054233890026807785,
+            -0.019491208717226982,
+            0.03935864195227623,
+            -0.009477907791733742,
+            -0.002881356282159686,
+            -0.019838491454720497,
+            0.009543023072183132,
+            -0.028694184496998787,
+            -0.0034185582771897316,
+            0.026205329224467278,
+            0.017190465703606606,
+            -0.018319131806492805,
+            -0.006236608140170574,
+            0.001457861391827464,
+            -0.062221381813287735,
+            -0.027507638558745384,
+            0.0053466977551579475,
+            -0.02477279119193554,
+            -0.008609702810645103,
+            -0.019780611619353294,
+            0.0038526610005646944,
+            0.01460755430161953,
+            0.00634151604026556,
+            0.037622228264808655,
+            0.0022898914758116007,
+            -0.06980370730161667,
+            0.013182250782847404,
+            0.024454450234770775,
+            0.023137671872973442,
+            -0.023600714281201363,
+            -0.00623299041762948,
+            -0.02228393591940403,
+            -0.022529928013682365,
+            -0.02636450156569481,
+            0.0061859628185629845,
+            0.004109505098313093,
+            0.005440753418952227,
+            -0.013884049840271473,
+            0.009311501868069172,
+            0.0421079583466053,
+            0.005227319430559874,
+            0.0153382932767272,
+            0.0037296651862561703,
+            -0.011366254650056362,
+            -0.020402824506163597,
+            -0.003644653595983982,
+            0.02924404852092266,
+            0.011460309848189354,
+            -0.03681190311908722,
+            -0.00158537901006639,
+            0.04427846893668175,
+            0.03530701622366905,
+            0.02198006398975849,
+            -0.011764181777834892,
+            0.009043805301189423,
+            0.020706696435809135,
+            -0.06013768911361694,
+            -0.00040742341661825776,
+            0.010042241774499416,
+            0.0057844179682433605,
+            -0.02148808166384697,
+            -0.020373884588479996,
+            -0.008754403330385685,
+            0.01953461952507496,
+            -0.013789994642138481,
+            0.008276890963315964,
+            6.890248914714903e-05,
+            0.033049680292606354,
+            0.011236024089157581,
+            -0.012531096115708351,
+            -0.017103644087910652,
+            0.05825657770037651,
+            -0.02539500594139099,
+            0.02878100611269474,
+            0.026842014864087105,
+            -0.023658594116568565,
+            0.04734613001346588,
+            -0.005386490374803543,
+            -0.04057412967085838,
+            0.015801336616277695,
+            0.009022099897265434,
+            -0.02171960286796093,
+            -0.025293715298175812,
+            0.0006579368491657078,
+            -0.007430390454828739,
+            0.002532265381887555,
+            -0.04642004519701004,
+            -0.010266527533531189,
+            -0.0025811020750552416,
+            0.017624568194150925,
+            -0.015642166137695312,
+            0.006019556894898415,
+            0.02600274793803692,
+            -0.00706140324473381,
+            0.041037172079086304,
+            -0.041934315115213394,
+            -0.008045368827879429,
+            0.00811048410832882,
+            -0.03750646859407425,
+            -0.03180525451898575,
+            -0.0013927459949627519,
+            -0.0012588977115228772,
+            -0.018058670684695244,
+            -0.018709823489189148,
+            0.00033439468825235963,
+            -0.008305830880999565,
+            -0.006869674660265446,
+            -0.04254205897450447,
+            0.011662891134619713,
+            -0.013522298075258732,
+            -0.006417484022676945,
+            0.003910541534423828,
+            -0.010592104867100716,
+            -0.019114986062049866,
+            0.024266337975859642,
+            0.03484397381544113,
+            0.024266337975859642,
+            0.023861175402998924,
+            -0.0021126328501850367,
+            -0.021271029487252235,
+            -0.01455690898001194,
+            0.02685648389160633,
+            0.001517550554126501,
+            0.007206104230135679,
+            0.024845141917467117,
+            -0.004927065223455429,
+            0.0015410644700750709,
+            -0.011293903924524784,
+            0.0363488607108593,
+            -0.008703758008778095,
+            0.00639216136187315,
+            0.04485727474093437,
+            0.02795621007680893,
+            0.003259387332946062,
+            -0.006649005692452192,
+            -0.0039539518766105175,
+            0.005277964752167463,
+            -0.03394682705402374,
+            0.01840595155954361,
+            0.019389918074011803,
+            0.0038237208500504494,
+            -0.0056071593426167965,
+            0.006431954447180033,
+            -0.024179518222808838,
+            0.011496485210955143,
+            0.01183653250336647,
+            0.00784278754144907,
+            -0.03660932183265686,
+            0.012545567005872726,
+            0.019259687513113022,
+            0.0014054073253646493,
+            -0.015974977985024452,
+            -0.0116701265797019,
+            -0.014238567091524601,
+            0.024222927168011665,
+            0.001188355963677168,
+            -0.003324502846226096,
+            0.0009622608777135611,
+            0.030994929373264313,
+            0.012777088209986687,
+            0.016727421432733536,
+            -0.022732509300112724,
+            0.02675519324839115,
+            -0.0074159204959869385,
+            0.0068009416572749615,
+            -0.0010110974544659257,
+            -0.014462853781878948,
+            -0.013363126665353775,
+            0.03565429896116257,
+            -0.0019968722481280565,
+            0.04138445109128952,
+            -0.005755477584898472,
+            0.013384832069277763,
+            0.03770904988050461,
+            -0.0027728306595236063,
+            -0.02600274793803692,
+            -0.007712556980550289,
+            0.0007709843921475112,
+            -0.015309353359043598,
+            -0.016293318942189217,
+            -0.027507638558745384,
+            -0.013269071467220783,
+            -0.009427262470126152,
+            -0.017885029315948486,
+            -0.012342985719442368,
+            0.025062192231416702,
+            -0.0037658405490219593,
+            -0.0182033721357584,
+            -0.03727494925260544,
+            0.028940176591277122,
+            0.0004478944465517998,
+            0.03692766651511192,
+            -0.008240715600550175,
+            0.012994139455258846,
+            -0.008616937324404716,
+            0.013920225203037262,
+            0.016307789832353592,
+            0.013023079372942448,
+            0.022370757535099983,
+            0.0005584549508057535,
+            0.00988307036459446,
+            0.03565429896116257,
+            -0.031486913561820984,
+            -0.03348378464579582,
+            0.014976541511714458,
+            -0.01522253267467022,
+            0.022095825523138046,
+            0.0029211491346359253,
+            -0.0050283558666706085,
+            0.021401260048151016,
+            0.006457277107983828,
+            -0.034207288175821304,
+            0.01966485008597374,
+            -0.012444276362657547,
+            -0.03319438174366951,
+            0.03435198962688446,
+            -0.014802900142967701,
+            0.004131210036575794,
+            -0.012480450794100761,
+            -0.0025485444348305464,
+            -0.024353159591555595,
+            0.026711782440543175,
+            0.019809551537036896,
+            -0.025915928184986115,
+            0.02523583360016346,
+            0.030358245596289635,
+            0.013348656706511974,
+            -0.004142062738537788,
+            0.008450531400740147,
+            -0.0048510972410440445,
+            -0.022443106397986412,
+            -0.02549629658460617,
+            0.002917531644925475,
+            0.011641185730695724,
+            -0.005719302222132683,
+            0.021459141746163368,
+            -0.049661342054605484,
+            -0.007481035776436329,
+            0.0005797079065814614,
+            -0.016438020393252373,
+            -0.004764276556670666,
+            0.00960090383887291,
+            0.013761053793132305,
+            0.0030622323974967003,
+            -0.03154479339718819,
+            0.0318920761346817,
+            0.029258519411087036,
+            0.020070012658834457,
+            -0.008732697926461697,
+            0.017566686496138573,
+            -0.006374074146151543,
+            0.012096993625164032,
+            0.031949955970048904,
+            0.01827572099864483,
+            0.008038134314119816,
+            0.0005860385717824101,
+            -0.027898330241441727,
+            0.02015683241188526,
+            0.011330079287290573,
+            0.035827938467264175,
+            0.02128550037741661,
+            0.021864304319024086,
+            0.02225499600172043,
+            -0.022023474797606468,
+            0.006493452005088329,
+            -0.006558567751199007,
+            -0.016235439106822014,
+            0.01658272184431553,
+            -0.010628280229866505,
+            0.03504655510187149,
+            0.012147638946771622,
+            -0.05186079815030098,
+            -0.012314044870436192,
+            0.019809551537036896,
+            0.022356286644935608,
+            0.04372860863804817,
+            0.017335165292024612,
+            0.0025666318833827972,
+            0.04838797450065613,
+            -0.02642238140106201,
+            -0.0306765865534544,
+            0.005871238186955452,
+            0.02025812305510044,
+            -0.024859612807631493,
+            0.009760074317455292,
+            0.02464256063103676,
+            -0.018217841163277626,
+            -0.016365669667720795,
+            0.015613225288689137,
+            0.0020999715197831392,
+            0.0030459535773843527,
+            -0.014137276448309422,
+            -0.008096014149487019,
+            0.02487408183515072,
+            0.006301723420619965,
+            0.036985546350479126,
+            -0.01771138794720173,
+            0.019143927842378616,
+            0.00183589244261384,
+            -0.01473778486251831,
+            0.0284481942653656,
+            0.0011639377335086465,
+            -0.03362848609685898,
+            0.025742286816239357,
+            0.012965199537575245,
+            0.010512519627809525,
+            0.009912010282278061,
+            -0.0009658784256316721,
+            0.021907713264226913,
+            0.04233947768807411,
+            0.010592104867100716,
+            0.0004639472172129899,
+            0.03319438174366951,
+            0.024946432560682297,
+            0.003083937568590045,
+            0.01415898185223341,
+            -0.006649005692452192,
+            -0.013052019290626049,
+            0.011532660573720932,
+            -0.02061987668275833,
+            0.0278838612139225,
+            -0.023846706375479698,
+            -0.012393631041049957,
+            -0.004572547972202301,
+            -0.0021144417114555836,
+            -0.0011386150727048516,
+            -0.012096993625164032,
+            -0.008074309676885605,
+            -0.0187677051872015,
+            0.020402824506163597,
+            -0.007965783588588238,
+            0.037419646978378296,
+            -0.007188016548752785,
+            -0.0012118698796257377,
+            0.015048892237246037,
+            -0.026610491797327995,
+            0.002241055015474558,
+            -0.0010174281196668744,
+            0.021734071895480156,
+            -0.01571451686322689,
+            -0.005599924363195896,
+            0.012929024174809456,
+            0.03912711888551712,
+            -0.009260856546461582,
+            0.0003251247981097549,
+            0.029504509642720222,
+            0.0017798208864405751,
+            -0.03817209228873253,
+            0.028838885948061943,
+            -0.046506863087415695,
+            0.008667582646012306,
+            0.009116156026721,
+            -0.022168176248669624,
+            -0.019737200811505318,
+            -0.025640996173024178,
+            -0.004406142048537731,
+            0.0033009888138622046,
+            -0.01203911378979683,
+            0.020807987079024315,
+            -0.00955025851726532,
+            -0.025423945859074593,
+            0.02998202294111252,
+            0.014665434136986732,
+            0.009470673277974129,
+            -0.02675519324839115,
+            0.014267507009208202,
+            -0.036754023283720016,
+            -0.024193987250328064,
+            -0.0038273383397608995,
+            -0.012531096115708351,
+            0.0076980870217084885,
+            -0.006909467279911041,
+            -0.027666809037327766,
+            -0.014021515846252441,
+            0.00466660363599658,
+            -0.01584474742412567,
+            0.039705920964479446,
+            0.035625357180833817,
+            0.04349708557128906,
+            0.014035985805094242,
+            -0.035856880247592926,
+            0.02626321092247963,
+            -0.011170907877385616,
+            0.02059093490242958,
+            -0.009159565903246403,
+            0.0009902967140078545,
+            0.000590108276810497,
+            0.03637780249118805,
+            -0.02536606416106224,
+            0.010360583662986755,
+            -0.04254205897450447,
+            -0.008023664355278015,
+            -0.027377407997846603,
+            0.0032756661530584097,
+            -0.006352368742227554,
+            -0.001016523689031601,
+            0.009991596452891827,
+            -0.004362731706351042,
+            0.021502550691366196,
+            0.04236841946840286,
+            0.0016324068419635296,
+            0.02977944165468216,
+            -0.03883771598339081,
+            -0.020214714109897614,
+            0.013312481343746185,
+            -0.008595232851803303,
+            -0.013146075420081615,
+            -0.019114986062049866,
+            -0.012538331560790539,
+            -0.021213149651885033,
+            0.00700352294370532,
+            0.013710408471524715,
+            -0.01200293842703104,
+            0.001774394535459578,
+            0.016698481515049934,
+            -0.017291756346821785,
+            -0.0018847290193662047,
+            -0.03371530771255493,
+            -0.011633951216936111,
+            0.01899922639131546,
+            -0.026350030675530434,
+            0.029504509642720222,
+            -0.0045580780133605,
+            0.027970680966973305,
+            -0.02420845814049244,
+            -0.010751275345683098,
+            -0.0021415730006992817,
+            -0.02805750072002411,
+            -0.0020221949089318514,
+            0.016003917902708054,
+            0.03889559581875801,
+            -0.01761009730398655,
+            0.04714354872703552,
+            0.0003796137170866132,
+            0.013565707951784134,
+            -0.024555740877985954,
+            0.005249024368822575,
+            -0.052439600229263306,
+            0.008942514657974243,
+            -0.008660348132252693,
+            -0.033570606261491776,
+            -0.01581580750644207,
+            -0.004829391837120056,
+            0.013189485296607018,
+            -0.022370757535099983,
+            -0.00500665046274662,
+            0.01044740341603756,
+            -0.012560036964714527,
+            0.00022609512961935252,
+            -0.023021910339593887,
+            -0.020185774192214012,
+            -0.007065020501613617,
+            0.012639622204005718,
+            -0.03440986946225166,
+            0.016669541597366333,
+            -0.0352780744433403,
+            -0.0049704755656421185,
+            -0.027464227750897408,
+            -0.021271029487252235,
+            0.02911381796002388,
+            0.027232706546783447,
+            -0.04508879408240318,
+            0.029895203188061714,
+            0.018926875665783882,
+            0.011576070450246334,
+            -0.03976380452513695,
+            0.03840361535549164,
+            0.00019421572505962104,
+            0.010273762978613377,
+            0.003168949391692877,
+            -0.008645878173410892,
+            0.02228393591940403,
+            -0.03047400526702404,
+            -0.013052019290626049,
+            0.00472448393702507,
+            -0.006377691403031349,
+            0.012654092162847519,
+            -0.0010617427760735154,
+            -0.023586243391036987,
+            0.016307789832353592,
+            0.011482015252113342,
+            0.012871143408119678,
+            0.03339696303009987,
+            -0.025583116337656975,
+            0.0352780744433403,
+            0.0045182849280536175,
+            -0.018550653010606766,
+            0.006793706677854061,
+            -0.013507827185094357,
+            0.0034764388110488653,
+            0.029157228767871857,
+            0.03930075839161873,
+            0.008587997406721115,
+            -0.036956604570150375,
+            0.02397693693637848,
+            0.022226056084036827,
+            -0.0035867730621248484,
+            0.02867971546947956,
+            0.011474779807031155,
+            0.024613620713353157,
+            0.03545171767473221,
+            -0.0023441542871296406,
+            0.003197889542207122,
+            0.026017218828201294,
+            -0.023094261065125465,
+            0.007596796378493309,
+            -0.01246598083525896,
+            0.03186313435435295,
+            0.011149203404784203,
+            -0.021401260048151016,
+            -0.004189090803265572,
+            -0.010577634908258915,
+            0.009304266422986984,
+            0.007122900802642107,
+            -0.014101101085543633,
+            -0.02857842482626438,
+            -0.014933131635189056,
+            -0.004286763723939657,
+            0.010360583662986755,
+            -0.06262654066085815,
+            -0.01440497301518917,
+            -0.018883464857935905,
+            -0.010816390626132488,
+            0.0036555060651153326,
+            0.01889793574810028,
+            -0.009123390540480614,
+            -0.0029772205743938684,
+            0.0153382932767272,
+            0.05191867798566818,
+            -0.008566292002797127,
+            0.00704693328589201,
+            0.000250739511102438,
+            0.01034611277282238,
+            -0.01640908047556877,
+            0.03507549315690994,
+            -0.013544002547860146,
+            -0.048561617732048035,
+            0.008768873289227486,
+            -0.025293715298175812,
+            0.01863747462630272,
+            0.047230370342731476,
+            -0.023296842351555824,
+            -0.03733282908797264,
+            0.020171303302049637,
+            0.009297031909227371,
+            -0.02934533916413784,
+            -0.00019738104310818017,
+            -0.008226245641708374,
+            -0.008132189512252808,
+            -0.021068448200821877,
+            -0.004785981960594654,
+            0.04636216163635254,
+            -0.008537352085113525,
+            0.025742286816239357,
+            0.03394682705402374,
+            0.017016824334859848,
+            -0.03134221211075783,
+            0.02562652714550495,
+            0.005017503164708614,
+            -0.00505729578435421,
+            0.002235628664493561,
+            0.04031366482377052,
+            -0.02941768988966942,
+            -0.012328515760600567,
+            -0.030358245596289635,
+            -0.016032857820391655,
+            -0.03883771598339081,
+            -0.0005697597516700625,
+            0.0024888552725315094,
+            0.008725463412702084,
+            -0.03160267323255539,
+            -0.032904982566833496,
+            0.031081749126315117,
+            0.03704342618584633,
+            -0.01427474245429039,
+            -0.005390108097344637,
+            -0.010953856632113457,
+            -0.005194761790335178,
+            0.006739443633705378,
+            -0.013175015337765217,
+            -0.001295977272093296,
+            -0.00739421509206295,
+            0.0142240971326828,
+            0.0035343191120773554,
+            0.0071120485663414,
+            -0.05504421517252922,
+            -0.018116550520062447,
+            0.018623003736138344,
+            0.0025792932137846947,
+            0.00119197356980294,
+            -0.004453169647604227,
+            -0.0051332637667655945,
+            0.00782831758260727,
+            -0.026104038581252098,
+            0.02318108081817627,
+            0.022211585193872452,
+            0.005509485956281424,
+            -0.002067413879558444,
+            0.007075873203575611,
+            0.03273133933544159,
+            0.030184604227542877,
+            -0.014079395681619644,
+            -0.015468524768948555,
+            0.03154479339718819,
+            -0.0019100516801699996,
+            0.014614789746701717,
+            -0.0023495806381106377,
+            0.01568557508289814,
+            -1.6335372492903844e-05,
+            -0.01853618398308754,
+            -0.008204540237784386,
+            -0.01622096821665764,
+            0.005824210587888956,
+            0.03131327033042908,
+            -0.00011587375774979591,
+            0.01953461952507496,
+            0.015092302113771439,
+            0.017089175060391426,
+            0.013847874477505684,
+            -0.009528553113341331,
+            -0.018926875665783882,
+            0.01794290915131569,
+            0.0029537067748606205,
+            0.026104038581252098,
+            0.007093960884958506,
+            -0.0039394814521074295,
+            -0.008906339295208454,
+            -0.011409664526581764,
+            -0.005035590846091509,
+            0.009246386587619781,
+            0.02125656045973301,
+            0.03380212560296059,
+            -0.008117719553411007,
+            -0.02716035582125187,
+            -0.01501995138823986,
+            0.021430201828479767,
+            0.02138679102063179,
+            -0.01907157711684704,
+            -0.021459141746163368,
+            0.01740751601755619,
+            -0.014130041003227234,
+            0.015931567177176476,
+            0.0019932547584176064,
+            0.003203315893188119,
+            -0.010903211310505867,
+            0.07107707113027573,
+            -0.02759445831179619,
+            -0.004398907069116831,
+            -0.02798515185713768,
+            -0.003205124521628022,
+            0.006652622949331999,
+            -0.014781195670366287,
+            -0.0013737539993599057,
+            0.018623003736138344,
+            -0.003024248406291008,
+            0.022399697452783585,
+            0.01601838879287243,
+            -0.012017408385872841,
+            -0.00755338603630662,
+            -0.028071971610188484,
+            0.027174826711416245,
+            -0.008660348132252693,
+            0.0037405178882181644,
+            -0.0038309558294713497,
+            0.011359019204974174,
+            0.03713024780154228,
+            0.024989843368530273,
+            0.013363126665353775,
+            0.01740751601755619,
+            -0.010671690106391907,
+            0.013442711904644966,
+            0.012856673449277878,
+            0.014665434136986732,
+            -0.026017218828201294,
+            -0.015092302113771439,
+            0.001866641454398632,
+            0.006428336724638939,
+            0.008146659471094608,
+            0.005661421921104193,
+            -0.001539255608804524,
+            0.012762618251144886,
+            -0.006446424406021833,
+            0.00756062101572752,
+            -0.0009215637692250311,
+            -0.0204896442592144,
+            -0.009854130446910858,
+            0.025091134011745453,
+            0.0003529344976413995,
+            0.024179518222808838,
+            0.0011567026376724243,
+            -0.005542044062167406,
+            0.002711332868784666,
+            -0.02061987668275833,
+            0.00019195476488675922,
+            -0.01714705489575863,
+            -0.0063849263824522495,
+            0.0036790198646485806,
+            -0.01648143120110035,
+            -0.01195229310542345,
+            -0.01170630194246769,
+            0.000591917079873383,
+            0.0003979274188168347,
+            0.0056288642808794975,
+            0.00756062101572752,
+            -0.0036844462156295776,
+            0.009058275260031223,
+            -0.01016523689031601,
+            0.01011459156870842,
+            -0.007850022986531258,
+            -0.012871143408119678,
+            -0.007705322001129389,
+            0.011402430012822151,
+            0.009427262470126152,
+            -0.004492962267249823,
+            0.01160501129925251,
+            -0.016365669667720795,
+            0.003754987847059965,
+            -0.010924916714429855,
+            -0.025915928184986115,
+            -0.006124464794993401,
+            -0.02739187702536583,
+            0.004521902650594711,
+            -0.013131605461239815,
+            0.022269466891884804,
+            -0.024266337975859642,
+            0.027232706546783447,
+            0.007770437281578779,
+            -0.009297031909227371,
+            -0.00021286855917423964,
+            -0.03620416298508644,
+            0.011395194567739964,
+            -0.01846383325755596,
+            0.0028940176125615835,
+            0.012299574911594391,
+            -0.003438454819843173,
+            0.0056071593426167965,
+            -0.008645878173410892,
+            -0.010121827013790607,
+            -0.003353442996740341,
+            -0.021531492471694946,
+            0.004098652396351099,
+            0.006768384017050266,
+            -0.0031906545627862215,
+            -0.01499101147055626,
+            -0.01727728545665741,
+            -0.023528363555669785,
+            0.017812678590416908,
+            0.022891679778695107,
+            -0.021010568365454674,
+            -0.020518586039543152,
+            -0.005314140114933252,
+            0.0030712762381881475,
+            -0.02708800509572029,
+            0.0033064151648432016,
+            -0.008710993453860283,
+            0.01016523689031601,
+            -0.009065510705113411,
+            0.010606574825942516,
+            -0.003870748681947589,
+            0.020070012658834457,
+            0.018377011641860008,
+            0.02351389452815056,
+            -0.03620416298508644,
+            0.0017409324645996094,
+            0.010599339380860329,
+            -0.009246386587619781,
+            -0.014014280401170254,
+            0.021242089569568634,
+            -0.010403993539512157,
+            -0.0020999715197831392,
+            0.03090810962021351,
+            0.000624926935415715,
+            -0.008964220061898232,
+            0.008472236804664135,
+            -0.0017662551254034042,
+            -0.021531492471694946,
+            0.026436852291226387,
+            0.010830861516296864,
+            0.00905104074627161,
+            0.022573338821530342,
+            -0.003820103360339999,
+            0.005397343076765537,
+            -0.017494337633252144,
+            0.020981628447771072,
+            -0.0036772112362086773,
+            -0.009072745218873024,
+            0.019679320976138115,
+            0.026089569553732872,
+            -0.008233480155467987,
+            0.011510955169796944,
+            -0.03249981999397278,
+            0.0012028261553496122,
+            0.03892453759908676,
+            0.01255280151963234,
+            0.0009378425893373787,
+            -0.014281976968050003,
+            -0.012075289152562618,
+            0.013449947349727154,
+            0.021198678761720657,
+            -0.004442316945642233,
+            -0.02875206619501114,
+            0.008009194396436214,
+            -0.011257728561758995,
+            0.03322332352399826,
+            0.018594063818454742,
+            -0.00976730976253748,
+            0.027464227750897408,
+            -0.004619575571268797,
+            -0.01830466091632843,
+            0.01681424304842949,
+            -0.005379255395382643,
+            -0.004243353381752968,
+            0.011843767017126083,
+            0.0036482710856944323,
+            0.03044506534934044,
+            0.006837116554379463,
+            -0.0055709839798510075,
+            -0.0014605745673179626,
+            0.01661166176199913,
+            -0.002420122269541025,
+            -0.024165047332644463,
+            0.0048510972410440445,
+            0.010288232937455177,
+            -0.0007777672726660967,
+            -0.012588976882398129,
+            0.011966763064265251,
+            -0.014694374985992908,
+            -0.0030423360876739025,
+            0.0008921714033931494,
+            0.006099142134189606,
+            -0.00047615633229725063,
+            -0.0233113132417202,
+            -0.006576654966920614,
+            -0.03449669107794762,
+            0.028708655387163162,
+            0.009441732428967953,
+            0.007538916077464819,
+            0.013218426145613194,
+            0.005419048015028238,
+            0.008624172769486904,
+            -0.0034185582771897316,
+            -0.007850022986531258,
+            0.004485727287828922,
+            -0.009919245727360249,
+            -0.02831796370446682,
+            -0.03163161501288414,
+            0.016394609585404396,
+            -0.021299969404935837,
+            -0.0066236830316483974,
+            0.0028596511110663414,
+            -0.02433868870139122,
+            -0.02205241471529007,
+            0.003123730421066284,
+            0.01661166176199913,
+            -0.022139234468340874,
+            0.01664060167968273,
+            -0.001658633933402598,
+            -0.007350804749876261,
+            0.023730944842100143,
+            -0.008023664355278015,
+            -0.001483184052631259,
+            -0.0116701265797019,
+            -0.02669731341302395,
+            0.013666998594999313,
+            0.01283496804535389,
+            -0.0002509655896574259,
+            0.009608138352632523,
+            0.0012118698796257377,
+            0.010707865469157696,
+            0.009116156026721,
+            -0.01830466091632843,
+            0.01273367740213871,
+            -0.02716035582125187,
+            0.03345484286546707,
+            0.01910051703453064,
+            -0.028129851445555687,
+            0.0352780744433403,
+            -0.008710993453860283,
+            0.015859216451644897,
+            -0.0035288927610963583,
+            0.01400704588741064,
+            0.010845331475138664,
+            0.017393046990036964,
+            0.01601838879287243,
+            0.01399257592856884,
+            0.0041529154404997826,
+            0.01920180767774582,
+            -0.006088289897888899,
+            0.023065321147441864,
+            0.013949165120720863,
+            -3.0748939025215805e-05,
+            0.013580177910625935,
+            0.003338972805067897,
+            -0.018449362367391586,
+            -0.0015220724744722247,
+            -0.021068448200821877,
+            -0.007654676679521799,
+            0.0030622323974967003,
+            -0.0204317644238472,
+            -0.004561695270240307,
+            -0.030821288004517555,
+            0.006598360370844603,
+            0.0016595382476225495,
+            -0.012458746321499348,
+            -0.009362147189676762,
+            0.010765746235847473,
+            0.002649834845215082,
+            0.022775918245315552,
+            -0.011908882297575474,
+            0.02241416648030281,
+            0.008935279212892056,
+            -0.002969985594972968,
+            0.01049804873764515,
+            -0.02457020990550518,
+            0.014499028213322163,
+            0.0017535937950015068,
+            0.022472048178315163,
+            0.021632781252264977,
+            0.0005114271771162748,
+            0.010150766931474209,
+            0.0016197455115616322,
+            0.015352764166891575,
+            -0.015164652839303017,
+            -0.012429806403815746,
+            -0.00489088986068964,
+            -0.015584285371005535,
+            0.05814081430435181,
+            -0.02171960286796093,
+            -0.012314044870436192,
+            -0.017682448029518127,
+            -0.002385755768045783,
+            0.009311501868069172,
+            -0.030126724392175674,
+            -0.018651943653821945,
+            -0.022168176248669624,
+            -0.006797323934733868,
+            -0.02038835547864437,
+            -0.004579782951623201,
+            0.021603841334581375,
+            -0.0011901648249477148,
+            0.007850022986531258,
+            -0.01953461952507496,
+            -0.0068009416572749615,
+            -0.01409386657178402,
+            -0.003257578704506159,
+            -0.014535203576087952,
+            -0.012024643830955029,
+            -0.0042144134640693665,
+            0.021545961499214172,
+            0.01426027249544859,
+            -0.010859801433980465,
+            0.006486217025667429,
+            0.013580177910625935,
+            -0.012010172940790653,
+            0.003698916407302022,
+            0.01714705489575863,
+            0.011691831052303314,
+            -0.00944896787405014,
+            0.01007841620594263,
+            -0.0026842013467103243,
+            -0.005419048015028238,
+            0.004963240120559931,
+            0.0023749032989144325,
+            0.03244193643331528,
+            0.010512519627809525,
+            -0.04520455747842789,
+            -0.00811048410832882,
+            0.010968326590955257,
+            0.006844351999461651,
+            0.0019697407260537148,
+            0.005400960333645344,
+            -0.015309353359043598,
+            0.004250588361173868,
+            -0.03093704953789711,
+            -0.0001339613663731143,
+            -0.0026263210456818342,
+            -0.008645878173410892,
+            0.006634535733610392,
+            -0.006591125391423702,
+            0.01177141722291708,
+            0.01923074759542942,
+            -0.003143626730889082,
+            0.01457137893885374,
+            0.0030712762381881475,
+            0.013218426145613194,
+            -0.03281816095113754,
+            0.008739933371543884,
+            -0.011380724608898163,
+            0.009029335342347622,
+            -0.012328515760600567,
+            -0.035856880247592926,
+            -0.010867035947740078,
+            0.015728985890746117,
+            -0.002018577419221401,
+            -0.005976146552711725,
+            -0.0021922183223068714,
+            0.02626321092247963,
+            0.019780611619353294,
+            -0.0014280168106779456,
+            -0.0065621850080788136,
+            0.030589766800403595,
+            -0.0006945642526261508,
+            0.0002321997017133981,
+            0.0003368817560840398,
+            0.004934300202876329,
+            -0.011829297058284283,
+            0.005697597283869982,
+            -0.01740751601755619,
+            0.0067177386954426765,
+            -0.008052604272961617,
+            0.026899894699454308,
+            -0.0019064341904595494,
+            -0.014614789746701717,
+            0.0193609781563282,
+            -0.032876040786504745,
+            -0.004398907069116831,
+            -0.006478982046246529,
+            -0.00466660363599658,
+            0.021806422621011734,
+            -0.0019480356713756919,
+            0.025076663121581078,
+            -0.05145563557744026,
+            -0.0014379650820046663,
+            -0.004051624797284603,
+            -0.016365669667720795,
+            0.004283146001398563,
+            -0.02135785110294819,
+            0.014130041003227234,
+            -0.0030459535773843527,
+            -0.015757925808429718,
+            0.002360433107241988,
+            -0.008291360922157764,
+            -0.008595232851803303,
+            0.013175015337765217,
+            0.01578686572611332,
+            -0.003349825507029891,
+            0.014253037050366402,
+            0.0039539518766105175,
+            0.015265943482518196,
+            0.017494337633252144,
+            -0.013066490180790424,
+            -0.013558472506701946,
+            0.01178588718175888,
+            0.0006185962702147663,
+            0.01697341352701187,
+            0.008320300839841366,
+            -0.009991596452891827,
+            -0.02749316766858101,
+            0.005447988398373127,
+            -0.006714120972901583,
+            -0.0036301834043115377,
+            -0.018492773175239563,
+            -0.016626130789518356,
+            -0.001278794021345675,
+            -0.01668401248753071,
+            0.00461595831438899,
+            -0.023036381229758263,
+            -0.0010617427760735154,
+            0.008812284097075462,
+            -0.02338366210460663,
+            -0.004847479518502951,
+            0.013580177910625935,
+            0.016235439106822014,
+            -0.03247087821364403,
+            0.003870748681947589,
+            -0.0037369003985077143,
+            -0.028129851445555687,
+            -0.0035126139409840107,
+            0.0025720582343637943,
+            -0.013732113875448704,
+            0.01500548142939806,
+            -0.011344549246132374,
+            -0.017103644087910652,
+            -0.03397576883435249,
+            -0.022298406809568405,
+            -0.007010757923126221,
+            0.019158396869897842,
+            0.005462458357214928,
+            0.004178238101303577,
+            0.0027185678482055664,
+            0.0003034196561202407,
+            0.0076691466383636,
+            0.0284481942653656,
+            -0.013724878430366516,
+            -6.6804045673052315e-06,
+            -0.001633311272598803,
+            0.02061987668275833,
+            -0.004373584408313036,
+            0.004113122820854187,
+            0.004605105612426996,
+            0.000946886430028826,
+            0.0033425905276089907,
+            -0.005317757371813059,
+            -0.006938407197594643,
+            -0.010874271392822266,
+            0.010483578778803349,
+            0.024888552725315094,
+            -0.01910051703453064,
+            -0.0012679415522143245,
+            -0.02028706483542919,
+            0.029461100697517395,
+            0.030126724392175674,
+            0.00784278754144907,
+            -0.0409214086830616,
+            -0.0032340646721422672,
+            -0.0030170134268701077,
+            -0.0014325387310236692,
+            -0.02420845814049244,
+            0.01203911378979683,
+            0.006196815520524979,
+            0.009702194482088089,
+            0.018753234297037125,
+            -0.006583890412002802,
+            0.017002353444695473,
+            -0.008573527447879314,
+            -0.005968911573290825,
+            0.001627884921617806,
+            -0.007878962904214859,
+            0.015077832154929638,
+            0.04115293174982071,
+            -0.028838885948061943,
+            0.00650068698450923,
+            -0.007857258431613445,
+            0.022674627602100372,
+            -0.009543023072183132,
+            0.009195741266012192,
+            0.02061987668275833,
+            0.017161523923277855,
+            -0.0051151760853827,
+            0.016958944499492645,
+            0.07582326233386993,
+            -0.003993744496256113,
+            0.022009003907442093,
+            0.008768873289227486,
+            -0.000894884520675987,
+            0.0041167400777339935,
+            -0.019187336787581444,
+            -0.02305085025727749,
+            0.017233874648809433,
+            0.0039756568148732185,
+            -0.004492962267249823,
+            0.011496485210955143,
+            -0.015164652839303017,
+            0.006319811102002859,
+            0.017827149480581284,
+            0.005386490374803543,
+            -0.012639622204005718,
+            -0.006927554961293936,
+            0.022906150668859482,
+            0.019158396869897842,
+            0.0056288642808794975,
+            -0.010823626071214676,
+            0.01454243902117014,
+            -0.005646951962262392,
+            -0.01661166176199913,
+            0.016192028298974037,
+            -0.01691553369164467,
+            0.018782174214720726,
+            -0.0024978991132229567,
+            0.024787262082099915,
+            -0.007640206720679998,
+            -0.010172472335398197,
+            -0.0016088929260149598,
+            0.00032648135675117373,
+            0.01684318296611309,
+            0.019635910168290138,
+            0.0036573149263858795,
+            6.426754407584667e-05,
+            0.002027621027082205,
+            0.012364690192043781,
+            0.0006294488557614386,
+            0.0032431085128337145,
+            0.00500665046274662,
+            0.003143626730889082,
+            -0.012965199537575245,
+            0.023962466046214104,
+            0.006048496812582016,
+            0.008132189512252808,
+            -0.00616064015775919,
+            -0.005618012044578791,
+            0.004894507583230734,
+            -0.009904775768518448,
+            0.00650068698450923,
+            0.001803334802389145,
+            -0.003291945206001401,
+            -0.01261791680008173,
+            0.029721561819314957,
+            -0.021198678761720657,
+            0.014281976968050003,
+            0.01923074759542942,
+            0.008219010196626186,
+            0.00978177972137928,
+            -0.013189485296607018,
+            -0.018362542614340782,
+            0.00761850131675601,
+            -0.00027221854543313384,
+            0.005864003207534552,
+            0.0051115588285028934,
+            0.001834083697758615,
+            0.016597190871834755,
+            -0.00244182744063437,
+            0.005495015997439623,
+            0.013225660659372807,
+            0.007864492945373058,
+            0.005976146552711725,
+            0.016134148463606834,
+            -0.030329305678606033,
+            -0.018753234297037125,
+            0.004315703641623259,
+            0.00516582140699029,
+            -0.0005634290864691138,
+            0.0043554967269301414,
+            0.01006394624710083,
+            0.011279433965682983,
+            -0.008399886079132557,
+            -0.015497464686632156,
+            -0.014817370101809502,
+            0.04184749722480774,
+            0.014347092248499393,
+            -0.020750107243657112,
+            -0.011959527619183064,
+            -0.0021650870330631733,
+            0.003226829692721367,
+            -0.007198868785053492,
+            0.014933131635189056,
+            0.014962071552872658,
+            -0.018955815583467484,
+            -0.02590145915746689,
+            -0.027102475985884666,
+            -0.012704737484455109,
+            0.014310917817056179,
+            0.013688703998923302,
+            0.010288232937455177,
+            0.005809740629047155,
+            0.02112632989883423,
+            -0.014846310950815678,
+            0.008428826928138733,
+            0.012089759111404419,
+            0.0034782474394887686,
+            -0.009427262470126152,
+            -0.00600146921351552,
+            -0.014477323740720749,
+            0.015873687341809273,
+            0.008276890963315964,
+            -0.005802505649626255,
+            -0.01611967757344246,
+            0.0034438809379935265,
+            -0.014918660745024681,
+            -0.010411228984594345,
+            -0.006714120972901583,
+            -0.0021994535345584154,
+            -0.012082523666322231,
+            0.004927065223455429,
+            -0.009745604358613491,
+            -0.007090343162417412,
+            0.0033896181266754866,
+            -0.004171003121882677,
+            -0.005140498746186495,
+            -0.02685648389160633,
+            -0.026914363726973534,
+            -0.003196080680936575,
+            -0.0024038434494286776,
+            -0.00600146921351552,
+            -0.003590390784665942,
+            -0.029359810054302216,
+            -0.0065404800698161125,
+            0.009022099897265434,
+            -0.011851002462208271,
+            0.021068448200821877,
+            0.0029573242645710707,
+            -0.008906339295208454,
+            0.0033896181266754866,
+            -0.003604860743507743,
+            -0.029866263270378113,
+            0.009977125562727451,
+            0.005639716982841492,
+            0.02208135463297367,
+            -0.0182033721357584,
+            0.0015962315956130624,
+            0.0061497874557971954,
+            0.023759884759783745,
+            -0.031718432903289795,
+            -0.015902627259492874,
+            -0.007654676679521799,
+            -0.010975562036037445,
+            -0.006674328353255987,
+            -0.014354327693581581,
+            0.005060913506895304,
+            -0.008117719553411007,
+            -0.025423945859074593,
+            -0.02281932905316353,
+            -0.00650068698450923,
+            -0.008038134314119816,
+            0.022124765440821648,
+            1.9359395082574338e-05,
+            0.03684084489941597,
+            0.008660348132252693,
+            0.003986509516835213,
+            0.010852565988898277,
+            -0.0030133959371596575,
+            -0.007546151056885719,
+            0.011851002462208271,
+            -0.017581157386302948,
+            -0.006873291917145252,
+            0.003925011493265629,
+            0.01943332888185978,
+            -0.007654676679521799,
+            -0.002360433107241988,
+            0.011040677316486835,
+            0.016857653856277466,
+            0.008776108734309673,
+            0.0027438905090093613,
+            -0.007289307191967964,
+            -0.005610776599496603,
+            0.02215370535850525,
+            0.027073536068201065,
+            0.004728101193904877,
+            0.003436645958572626,
+            0.007806612644344568,
+            -0.002843372290953994,
+            -0.006674328353255987,
+            0.001284220372326672,
+            -0.006337898783385754,
+            -0.02085139788687229,
+            -0.0038635137025266886,
+            -0.004225265700370073,
+            -0.0025720582343637943,
+            -0.02205241471529007,
+            -0.01661166176199913,
+            -0.014875250868499279,
+            0.008956984616816044,
+            0.014470088295638561,
+            -0.019578030332922935,
+            0.01211869902908802,
+            0.006062967237085104,
+            0.006569419987499714,
+            -0.006435571704059839,
+            0.01753774657845497,
+            0.042223718017339706,
+            0.00023073007469065487,
+            -0.03906923905014992,
+            -0.01707470417022705,
+            0.0009111633989959955,
+            -0.012494921684265137,
+            -0.022935090586543083,
+            -0.00756062101572752,
+            0.0031617144122719765,
+            -0.024411039426922798,
+            0.023398132994771004,
+            -0.001627884921617806,
+            -0.0009658784256316721,
+            0.004330174066126347,
+            0.017060235142707825,
+            -0.002376711927354336,
+            0.005332227796316147,
+            -0.014535203576087952,
+            0.003442072309553623,
+            -0.01417345181107521,
+            0.010548694059252739,
+            -0.016857653856277466,
+            -0.018984755501151085,
+            0.006804558914154768,
+            -0.002042091218754649,
+            0.008920809254050255,
+            0.022341815754771233,
+            0.02652367204427719,
+            0.01028099749237299,
+            -0.027869390323758125,
+            -0.01523700263351202,
+            -0.011807592585682869,
+            0.0026950540486723185,
+            -0.014021515846252441,
+            -0.007321864832192659,
+            -0.00761850131675601,
+            0.003545171581208706,
+            0.015323823317885399,
+            0.015251473523676395,
+            0.005151351448148489,
+            0.030647646635770798,
+            -3.1540272175334394e-05,
+            0.006804558914154768,
+            -0.003615713445469737,
+            -0.02801409177482128,
+            -0.005939971189945936,
+            0.005708449985831976,
+            0.009991596452891827,
+            -0.013254600577056408,
+            0.007936843670904636,
+            0.014209627173841,
+            -0.016872122883796692,
+            -0.011279433965682983,
+            -0.02716035582125187,
+            0.010440168902277946,
+            -0.002913914155215025,
+            0.03093704953789711,
+            0.020070012658834457,
+            -0.02138679102063179,
+            -0.015439583919942379,
+            -0.02112632989883423,
+            0.018984755501151085,
+            -0.010953856632113457,
+            -0.0019154779147356749,
+            0.007973019033670425,
+            0.019983192905783653,
+            -0.007712556980550289,
+            -0.009094450622797012,
+            -0.0077921426855027676,
+            0.013876814395189285,
+            0.018753234297037125,
+            7.57983943913132e-05,
+            0.0010300894500687718,
+            -0.0096515491604805,
+            0.010780216194689274,
+            -0.009977125562727451,
+            -0.010592104867100716,
+            0.0051115588285028934,
+            -0.004529137630015612,
+            0.019968722015619278,
+            -0.01638014055788517,
+            0.022529928013682365,
+            0.010360583662986755,
+            -0.009202975779771805,
+            -0.0398506224155426,
+            -0.006522392388433218,
+            -0.006005086470395327,
+            -0.001984210917726159,
+            0.011995702981948853,
+            -0.0027131414972245693,
+            -0.01771138794720173,
+            0.008971454575657845,
+            0.000733000400941819,
+            -0.003031483618542552,
+            0.007604031357914209,
+            0.008096014149487019,
+            0.0007470183190889657,
+            0.011684596538543701,
+            -0.005046443548053503,
+            -0.009702194482088089,
+            0.02195112407207489,
+            0.0014090248150750995,
+            -7.969853322720155e-05,
+            0.02002660185098648,
+            0.02251545712351799,
+            0.010982797481119633,
+            -0.00988307036459446,
+            0.0034547336399555206,
+            -0.009825189597904682,
+            -0.007538916077464819,
+            -0.030850227922201157,
+            0.027579989284276962,
+            -0.007929608225822449,
+            0.03258663788437843,
+            0.013232896104454994,
+            -0.004037154838442802,
+            -0.00075018365168944,
+            0.0074593303725123405,
+            -0.005458840634673834,
+            -0.0006244747783057392,
+            0.017450926825404167,
+            -0.0009233725140802562,
+            0.006261930800974369,
+            -0.0032340646721422672,
+            -0.02028706483542919,
+            0.025915928184986115,
+            -0.0022754215169698,
+            0.005220084451138973,
+            0.011966763064265251,
+            0.009087215177714825,
+            0.007057785522192717,
+            -0.01607626862823963,
+            -0.0028071971610188484,
+            -0.03608839958906174,
+            -0.008016428910195827,
+            -0.0021108242217451334,
+            -0.005697597283869982,
+            -0.003261196194216609,
+            -0.0070831081829965115,
+            -0.04691202566027641,
+            -0.004189090803265572,
+            0.008544587530195713,
+            -0.009868600405752659,
+            0.01853618398308754,
+            -0.016828712075948715,
+            -0.017957380041480064,
+            -0.02115526981651783,
+            0.007195251528173685,
+            0.02002660185098648,
+            0.013587413355708122,
+            -0.003053188556805253,
+            -0.017986319959163666,
+            0.018550653010606766,
+            -0.011800357140600681,
+            -0.008211774751543999,
+            -0.01635120064020157,
+            -0.020055541768670082,
+            0.014549673534929752,
+            0.016105208545923233,
+            -0.011908882297575474,
+            -0.0032431085128337145,
+            -0.007604031357914209,
+            0.013797229155898094,
+            -0.01720493473112583,
+            -0.008327536284923553,
+            0.008096014149487019,
+            0.0018738764338195324,
+            0.012342985719442368,
+            -0.019172867760062218,
+            0.007108430843800306,
+            -0.015801336616277695,
+            0.009043805301189423,
+            -0.01789950020611286,
+            -0.013218426145613194,
+            0.002179556991904974,
+            0.0010237587848678231,
+            0.039705920964479446,
+            0.022920619696378708,
+            -0.027406347915530205,
+            0.01846383325755596,
+            0.004572547972202301,
+            0.016336729750037193,
+            -0.0031038338784128428,
+            0.01467266958206892,
+            -0.01720493473112583,
+            0.016655072569847107,
+            -0.007216956466436386,
+            0.01761009730398655,
+            0.003404088318347931,
+            0.03293392062187195,
+            -0.017566686496138573,
+            0.010432933457195759,
+            0.0015681958757340908,
+            -0.004634045995771885,
+            -0.00705416826531291,
+            0.016785303130745888,
+            0.010411228984594345,
+            0.014520733617246151,
+            0.00616787513718009,
+            -0.023600714281201363,
+            -0.009608138352632523,
+            0.004091417416930199,
+            0.020475175231695175,
+            0.014781195670366287,
+            -0.00349814398214221,
+            -0.0036573149263858795,
+            0.002850607503205538,
+            -0.01028099749237299,
+            0.0096660191193223,
+            -0.013334186747670174,
+            0.007596796378493309,
+            -0.008493942208588123,
+            0.0077921426855027676,
+            -0.008407121524214745,
+            0.0007913329754956067,
+            0.021473610773682594,
+            -0.03510443493723869,
+            0.01510677207261324,
+            -0.003747752867639065,
+            -0.014231331646442413,
+            0.027044596150517464,
+            -0.017393046990036964,
+            -0.005845915526151657,
+            -0.008964220061898232,
+            -0.0008143946761265397,
+            0.014216861687600613,
+            -0.000501479022204876,
+            0.0035704942420125008,
+            -0.006768384017050266,
+            -0.006949259899556637,
+            0.001438869396224618,
+            0.0091016860678792,
+            -0.011851002462208271,
+            0.006677945610135794,
+            -0.004026302136480808,
+            0.021444670855998993,
+            0.008732697926461697,
+            0.006938407197594643,
+            0.00020653790852520615,
+            -0.002971794456243515,
+            0.006062967237085104,
+            0.02182089351117611,
+            0.0048149218782782555,
+            0.023166611790657043,
+            -0.023658594116568565,
+            0.013486122712492943,
+            -0.00016154497279785573,
+            -0.008096014149487019,
+            -0.01694447360932827,
+            -0.0017870558658614755,
+            0.02543841488659382,
+            0.0020475175697356462,
+            -0.02400587685406208,
+            0.00033281202195212245,
+            0.010143532417714596,
+            -0.0014524351572617888,
+            0.01500548142939806,
+            0.01784161850810051,
+            -0.0014343474758788943,
+            0.007282072212547064,
+            -0.002738464158028364,
+            -0.024092696607112885,
+            -0.01807313971221447,
+            -0.003783928230404854,
+            -0.016785303130745888,
+            0.02271803840994835,
+            0.01691553369164467,
+            0.0051694391295313835,
+            -0.0038526610005646944,
+            0.0017517850501462817,
+            0.01522253267467022,
+            0.020041072741150856,
+            0.0056288642808794975,
+            0.00939832255244255,
+            0.0011594158131629229,
+            -0.010309938341379166,
+            0.0016504944069311023,
+            -0.008052604272961617,
+            -0.008797814138233662,
+            -0.012357455678284168,
+            0.009275326505303383,
+            -0.006240225397050381,
+            -0.013449947349727154,
+            -0.01437603309750557,
+            -0.004619575571268797,
+            0.016090737655758858,
+            -0.007546151056885719,
+            -0.026870954781770706,
+            0.0036265659146010876,
+            0.018377011641860008,
+            0.011612245813012123,
+            0.015642166137695312,
+            -0.009087215177714825,
+            -0.011185378767549992,
+            0.007929608225822449,
+            -0.019476739689707756,
+            -0.0029193402733653784,
+            -0.010729570873081684,
+            0.003085746429860592,
+            -0.021545961499214172,
+            0.02937427908182144,
+            0.008472236804664135,
+            0.027739159762859344,
+            0.024729380384087563,
+            -0.01150372065603733,
+            0.004478492308408022,
+            -0.003693490056321025,
+            -0.0002373998868279159,
+            -0.008956984616816044,
+            3.804050720646046e-05,
+            0.0006158831529319286,
+            0.021545961499214172,
+            0.013175015337765217,
+            0.01245151087641716,
+            0.001984210917726159,
+            -0.0009134243591688573,
+            0.002418313641101122,
+            -7.246348832268268e-05,
+            -0.010042241774499416,
+            -0.017856089398264885,
+            0.005277964752167463,
+            0.017002353444695473,
+            -0.004178238101303577,
+            -0.005639716982841492,
+            0.009969891048967838,
+            0.0003594912705011666,
+            -0.0029555156361311674,
+            -0.011612245813012123,
+            -0.011858237907290459,
+            0.018159961327910423,
+            0.026509201154112816,
+            -0.008609702810645103,
+            -0.006547715049237013,
+            0.02115526981651783,
+            -0.001689382828772068,
+            0.03215253725647926,
+            -0.011098558083176613,
+            -0.010621044784784317,
+            -0.0014596701366826892,
+            -0.005212849471718073,
+            -0.01730622537434101,
+            -0.0238756462931633,
+            0.01255280151963234,
+            0.0035397454630583525,
+            -0.0035361277405172586,
+            -0.03047400526702404,
+            0.015034422278404236,
+            0.006135317496955395,
+            0.032876040786504745,
+            -0.001438869396224618,
+            -0.006460894364863634,
+            -0.000922015926335007,
+            -0.0010653602657839656,
+            -0.0013864153297618032,
+            -0.013811699114739895,
+            0.00905104074627161,
+            -0.0032684311736375093,
+            0.0034746299497783184,
+            -0.0020312387496232986,
+            -0.025554176419973373,
+            -0.005693979561328888,
+            0.002465341240167618,
+            -0.00727121951058507,
+            -0.00756062101572752,
+            -0.013131605461239815,
+            -0.0012145830551162362,
+            0.017335165292024612,
+            0.004416994284838438,
+            -0.0010997267672792077,
+            0.008674818091094494,
+            -0.0077559673227369785,
+            -0.008493942208588123,
+            -0.0060665844939649105,
+            -0.01720493473112583,
+            0.01830466091632843,
+            -0.02105397917330265,
+            0.0017563069704920053,
+            0.026350030675530434,
+            -0.004218030720949173,
+            -0.003295562695711851,
+            0.018695354461669922,
+            0.0091016860678792,
+            0.008587997406721115,
+            -0.0006027696072123945,
+            -0.002096354030072689,
+            0.00361028709448874,
+            0.01177141722291708,
+            0.011590540409088135,
+            -0.002790918340906501,
+            -0.00232968432828784,
+            -0.02089480683207512,
+            0.008602467365562916,
+            -0.01889793574810028,
+            0.024657031521201134,
+            0.02061987668275833,
+            -0.0015374468639492989,
+            0.01187270786613226,
+            0.02646579220890999,
+            0.008349240757524967,
+            0.01488972082734108,
+            0.000975826580543071,
+            0.005737389903515577,
+            -0.0025431180838495493,
+            -0.000960452132858336,
+            -0.023253431543707848,
+            0.006135317496955395,
+            0.005831445567309856,
+            0.004308468662202358,
+            -0.0035940082743763924,
+            -0.01976614072918892,
+            0.017421986907720566,
+            0.014151746407151222,
+            -0.018217841163277626,
+            0.021936653181910515,
+            -0.005737389903515577,
+            0.019838491454720497,
+            0.017103644087910652,
+            0.011959527619183064,
+            -0.007311012130230665,
+            -0.014166216365993023,
+            0.013558472506701946,
+            -0.02085139788687229,
+            -0.0024092698004096746,
+            0.005632482003420591,
+            -0.01727728545665741,
+            0.002420122269541025,
+            -0.02025812305510044,
+            -0.004076947458088398,
+            -0.008334770798683167,
+            0.006956494878977537,
+            -0.008327536284923553,
+            0.00628363573923707,
+            0.00583506328985095,
+            -0.0022844651248306036,
+            -0.001457861391827464,
+            -0.03637780249118805,
+            -0.005845915526151657,
+            0.021777482703328133,
+            0.0026028072461485863,
+            -0.04057412967085838,
+            0.021097388118505478,
+            0.004811304621398449,
+            -0.01684318296611309,
+            -0.015280413441359997,
+            -0.021936653181910515,
+            -0.014064925722777843,
+            -0.007625736761838198,
+            0.0037405178882181644,
+            -0.02102503925561905,
+            -0.0029048703145235777,
+            0.011807592585682869,
+            -0.01902816630899906,
+            -0.0102158822119236,
+            -0.008124954998493195,
+            -0.0031056427396833897,
+            0.019086046144366264,
+            -0.011923353187739849,
+            -0.00951408315449953,
+            0.025814637541770935,
+            0.006916702259331942,
+            -0.0058061229065060616,
+            -0.015323823317885399,
+            -0.016134148463606834,
+            -0.0071409884840250015,
+            0.013804464600980282,
+            -0.0025467355735599995,
+            -0.003921393770724535,
+            -0.030155664309859276,
+            -0.0028705038130283356,
+            0.006949259899556637,
+            0.019476739689707756,
+            -0.00043726799776777625,
+            0.002132529392838478,
+            0.009072745218873024,
+            0.00794407818466425,
+            -0.014535203576087952,
+            -7.447008556482615e-06,
+            0.009072745218873024,
+            0.012494921684265137,
+            -0.00015498821449000388,
+            0.019722729921340942,
+            -0.01833360269665718,
+            0.004937917459756136,
+            -0.008718227967619896,
+            0.005220084451138973,
+            -0.01860853284597397,
+            -0.006439189426600933,
+            -0.01627884991466999,
+            0.008052604272961617,
+            0.009275326505303383,
+            -0.018319131806492805,
+            -0.0019172866595909,
+            -0.006866056937724352,
+            0.017856089398264885,
+            -0.027174826711416245,
+            0.001350240083411336,
+            -0.017494337633252144,
+            -0.0015817615203559399,
+            -0.026350030675530434,
+            0.00311287771910429,
+            0.006471747066825628,
+            0.004561695270240307,
+            0.0003723786794580519,
+            -0.007850022986531258,
+            -0.02457020990550518,
+            0.01198846846818924,
+            -0.002181365853175521,
+            -0.001573622110299766,
+            -0.010526989586651325,
+            -0.0017327930545434356,
+            0.011966763064265251,
+            0.016163088381290436,
+            -0.0042361184023320675,
+            -0.014665434136986732,
+            0.01843489333987236,
+            -0.016105208545923233,
+            -0.019549088552594185,
+            -0.015179122798144817,
+            -0.011185378767549992,
+            -0.004644898232072592,
+            0.012639622204005718,
+            0.013804464600980282,
+            0.020779047161340714,
+            0.00961537379771471,
+            0.010765746235847473,
+            -0.007358039729297161,
+            0.00016572773165535182,
+            -0.006638152990490198,
+            -0.0015085067134350538,
+            0.012574506923556328,
+            -0.0054299007169902325,
+            0.02613298036158085,
+            -0.015410644002258778,
+            0.0015274987090379,
+            -0.00025797454873099923,
+            0.0069637298583984375,
+            -0.007354422472417355,
+            0.00015397078823298216,
+            -0.010411228984594345,
+            0.008392651565372944,
+            -0.03455457091331482,
+            -0.006258313078433275,
+            -0.0033263114746659994,
+            -0.011185378767549992,
+            0.03157373145222664,
+            0.0003567781241144985,
+            0.00756062101572752,
+            0.0048149218782782555,
+            0.011395194567739964,
+            -0.008298595435917377,
+            0.016061797738075256,
+            -0.017783738672733307,
+            0.001122336252592504,
+            0.015526404604315758,
+            0.007770437281578779,
+            -0.0012932640966027975,
+            0.006775618996471167,
+            -0.006041261833161116,
+            0.009123390540480614,
+            -0.0014108336763456464,
+            -0.014361563138663769,
+            -0.01450626365840435,
+            0.00017024963744916022,
+            0.013146075420081615,
+            -0.01727728545665741,
+            -0.006670710630714893,
+            -0.006511539686471224,
+            0.01394193060696125,
+            -0.01409386657178402,
+            0.026914363726973534,
+            -0.001209156820550561,
+            -0.017465395852923393,
+            0.01238639559596777,
+            0.030589766800403595,
+            -0.014383267611265182,
+            0.02059093490242958,
+            -0.007546151056885719,
+            -0.003906923811882734,
+            0.004988562781363726,
+            -0.0015482994494959712,
+            -0.0004268675984349102,
+            0.018217841163277626,
+            -0.008204540237784386,
+            -0.007777672726660967,
+            0.011207083240151405,
+            -0.02218264527618885,
+            0.016090737655758858,
+            -0.01584474742412567,
+            0.0010300894500687718,
+            0.004858332220464945,
+            0.0031110690906643867,
+            0.02028706483542919,
+            0.037622228264808655,
+            -0.00014944888243917376,
+            -0.007878962904214859,
+            -0.012234459631145,
+            0.02089480683207512,
+            0.005871238186955452,
+            0.005853150971233845,
+            -0.004388054367154837,
+            0.002447253791615367,
+            -0.01740751601755619,
+            -0.013681468553841114,
+            0.0076691466383636,
+            -0.010606574825942516,
+            -0.03313650190830231,
+            -0.015077832154929638,
+            0.0051115588285028934,
+            0.0063993968069553375,
+            0.018579592928290367,
+            0.006204050499945879,
+            -0.003986509516835213,
+            0.020967157557606697,
+            -0.003581346943974495,
+            0.011496485210955143,
+            -0.008414356037974358,
+            -0.002881356282159686,
+            -0.004876419901847839,
+            -0.023354722186923027,
+            0.024729380384087563,
+            -0.027637869119644165,
+            -0.003433028468862176,
+            -0.011337313801050186,
+            -0.000320150691550225,
+            0.005914648529142141,
+            -0.0031291565392166376,
+            0.01470884494483471,
+            0.007466565817594528,
+            0.016958944499492645,
+            0.01737857609987259,
+            -0.03166055306792259,
+            0.01707470417022705,
+            0.019303098320961,
+            -0.0020149596966803074,
+            -0.007538916077464819,
+            -0.0032449173741042614,
+            -0.012017408385872841,
+            -0.012762618251144886,
+            0.014296446926891804,
+            -0.0004528685531113297,
+            -0.006073819473385811,
+            -0.005654186941683292,
+            0.02287720888853073,
+            -0.007784907706081867,
+            0.02351389452815056,
+            -0.0046955435536801815,
+            0.0027800656389445066,
+            0.02652367204427719,
+            0.007325482089072466,
+            0.020417295396327972,
+            -0.016452491283416748,
+            -0.02271803840994835,
+            -0.007159076165407896,
+            -0.005249024368822575,
+            0.006916702259331942,
+            -0.005715684965252876,
+            0.00944896787405014,
+            -0.021647252142429352,
+            -0.020402824506163597,
+            0.01850724220275879,
+            -0.0199253112077713,
+            0.010736805386841297,
+            0.0061027598567306995,
+            0.01899922639131546,
+            -0.004000979475677013,
+            0.012531096115708351,
+            -0.010382288135588169,
+            -0.01604732871055603,
+            0.011424134485423565,
+            0.0062221381813287735,
+            -0.006652622949331999,
+            0.003928629215806723,
+            -0.011192613281309605,
+            -0.008790578693151474,
+            -0.01638014055788517,
+            -0.0010120017686858773,
+            -0.010961092077195644,
+            0.008161129429936409,
+            0.01787055842578411,
+            0.004959622863680124,
+            0.014745020307600498,
+            0.0032412998843938112,
+            -0.01250215619802475,
+            -0.010273762978613377,
+            0.0035940082743763924,
+            -0.010932152159512043,
+            -0.003747752867639065,
+            -0.022920619696378708,
+            0.0007212434429675341,
+            -0.014361563138663769,
+            -0.01175694726407528,
+            -0.009289796464145184,
+            0.0048510972410440445,
+            0.015497464686632156,
+            -0.018825585022568703,
+            -0.008457766845822334,
+            -0.008703758008778095,
+            0.0023170229978859425,
+            0.007057785522192717,
+            -0.016322258859872818,
+            0.008682052604854107,
+            0.02931639924645424,
+            -0.0034402634482830763,
+            -0.014795665629208088,
+            0.004442316945642233,
+            0.005614394322037697,
+            -0.008378181606531143,
+            0.018058670684695244,
+            0.011395194567739964,
+            -0.013826169073581696,
+            -0.004275911021977663,
+            0.013399302028119564,
+            -0.00019625057757366449,
+            -0.0005494111683219671,
+            0.0009396513924002647,
+            -0.014412208460271358,
+            0.03157373145222664,
+            0.0022808476351201534,
+            0.021082919090986252,
+            -8.054639329202473e-05,
+            -0.006601977627724409,
+            -0.006920319981873035,
+            -0.002239246154204011,
+            -0.006894997321069241,
+            -0.0034493072889745235,
+            -0.0011214318219572306,
+            -0.005668656900525093,
+            0.016307789832353592,
+            0.00704693328589201,
+            -0.002043900080025196,
+            0.0006629109266214073,
+            -0.016655072569847107,
+            0.00517305638641119,
+            -0.005249024368822575,
+            -0.005151351448148489,
+            -0.01804419979453087,
+            0.031284332275390625,
+            0.01261791680008173,
+            0.006728590931743383,
+            -0.006370456423610449,
+            -0.020301533862948418,
+            -0.00013983984536025673,
+            0.030358245596289635,
+            -0.011236024089157581,
+            0.004091417416930199,
+            0.01488972082734108,
+            -0.005162204150110483,
+            -0.002150616841390729,
+            -0.014281976968050003,
+            -0.0028053882997483015,
+            -0.027768099680542946,
+            0.0023061702959239483,
+            -0.006775618996471167,
+            0.006323428824543953,
+            -0.002731229178607464,
+            -0.013630823232233524,
+            -0.015902627259492874,
+            -0.0204607043415308,
+            -0.0062438431195914745,
+            -0.009275326505303383,
+            -0.009702194482088089,
+            -0.010411228984594345,
+            -0.025062192231416702,
+            0.021314440295100212,
+            -0.011843767017126083,
+            -0.0004286763723939657,
+            -0.024599149823188782,
+            0.0182033721357584,
+            -0.01034611277282238,
+            -0.015642166137695312,
+            0.0066453879699110985,
+            0.009789015166461468,
+            0.016192028298974037,
+            0.006124464794993401,
+            -0.004977710545063019,
+            -0.004181855358183384,
+            0.014064925722777843,
+            -0.0066236830316483974,
+            -0.005328610073775053,
+            -0.002212114864960313,
+            -0.021893244236707687,
+            -0.007202486507594585,
+            -0.010367818176746368,
+            -0.01177141722291708,
+            -0.005853150971233845,
+            0.010801920667290688,
+            0.015150182880461216,
+            0.0013104473473504186,
+            -0.01761009730398655,
+            -0.0054877810180187225,
+            0.0034113232977688313,
+            0.010085651651024818,
+            -0.023195551708340645,
+            0.008378181606531143,
+            -0.008356476202607155,
+            0.013732113875448704,
+            0.030792348086833954,
+            0.00577718298882246,
+            -0.004720866214483976,
+            0.0008682053303346038,
+            0.02102503925561905,
+            0.0030423360876739025,
+            0.00938385259360075,
+            -0.030358245596289635,
+            -0.022906150668859482,
+            -0.003950334154069424,
+            0.004923447500914335,
+            -0.005932736210525036,
+            0.030358245596289635,
+            0.008573527447879314,
+            0.01840595155954361,
+            0.006612830329686403,
+            0.023484952747821808,
+            0.006620065309107304,
+            0.022558867931365967,
+            0.011568835936486721,
+            0.003633800894021988,
+            -0.007517211139202118,
+            -0.024251868948340416,
+            -0.02944662980735302,
+            0.007690852042287588,
+            0.016568250954151154,
+            0.02115526981651783,
+            -0.03553853556513786,
+            0.023600714281201363,
+            -0.0015284031396731734,
+            0.0003922750474885106,
+            -0.014679905027151108,
+            0.007929608225822449,
+            0.005599924363195896,
+            0.022139234468340874,
+            0.0003649175341706723,
+            0.006775618996471167,
+            0.01039675809442997,
+            0.024512330070137978,
+            -0.017798209562897682,
+            0.01902816630899906,
+            0.017335165292024612,
+            0.0329628624022007,
+            0.017725858837366104,
+            -0.010461874306201935,
+            0.02095268853008747,
+            0.014202391728758812,
+            -0.004351879004389048,
+            -0.011807592585682869,
+            0.0037658405490219593,
+            -0.007799377664923668,
+            0.006319811102002859,
+            0.006493452005088329,
+            0.004464022349566221,
+            -0.010375053621828556,
+            0.007368892431259155,
+            -0.00728568946942687,
+            0.005216466728597879,
+            0.003527084132656455,
+            0.012096993625164032,
+            0.0013782759197056293,
+            0.02785491943359375,
+            -0.04306298494338989,
+            0.02161831222474575,
+            -0.02652367204427719,
+            -0.013529532589018345,
+            -0.005382873117923737,
+            0.005158586427569389,
+            0.0030748937278985977,
+            -0.009904775768518448,
+            -0.011532660573720932,
+            -0.01472331490367651,
+            0.014846310950815678,
+            0.0198674313724041,
+            0.0018286574631929398,
+            -0.0071120485663414,
+            0.006938407197594643,
+            -0.00183227495290339,
+            -0.01747986674308777,
+            0.0019010078394785523,
+            -0.008124954998493195,
+            -0.011236024089157581,
+            -0.02208135463297367,
+            0.013059254735708237,
+            0.0027420816477388144,
+            0.006710503716021776,
+            0.00017431934247724712,
+            -0.025858048349618912,
+            -0.01399257592856884,
+            0.009875834919512272,
+            0.0010328025091439486,
+            -0.013102664612233639,
+            -0.013276305980980396,
+            0.006750296335667372,
+            -0.0027583607006818056,
+            0.003986509516835213,
+            0.011199848726391792,
+            0.0102158822119236,
+            -0.006641770713031292,
+            0.0036247570533305407,
+            0.01910051703453064,
+            0.005267112050205469,
+            -0.016452491283416748,
+            0.01648143120110035,
+            -0.016061797738075256,
+            0.0020312387496232986,
+            0.00035768249654211104,
+            -0.02028706483542919,
+            -0.011973998509347439,
+            0.008182834833860397,
+            -0.009079980663955212,
+            -0.0016794346738606691,
+            0.016061797738075256,
+            0.011735241860151291,
+            0.019375449046492577,
+            0.022037943825125694,
+            0.004084182437509298,
+            0.016452491283416748,
+            -0.02310873195528984,
+            0.014130041003227234,
+            -0.021878773346543312,
+            -0.010888741351664066,
+            0.01753774657845497,
+            0.007032462861388922,
+            0.006573037710040808,
+            -0.0012263399548828602,
+            -0.004275911021977663,
+            -0.0060448795557022095,
+            0.014745020307600498,
+            0.0329628624022007,
+            -0.0022247761953622103,
+            -0.01235022023320198,
+            -0.017364105209708214,
+            -0.00755338603630662,
+            0.03666720539331436,
+            -0.01522253267467022,
+            -0.014578614383935928,
+            0.021213149651885033,
+            0.02742081694304943,
+            0.00489812484011054,
+            -0.009593668393790722,
+            -0.0063993968069553375,
+            0.0032539612147957087,
+            0.015265943482518196,
+            0.013464417308568954,
+            -0.010353348217904568,
+            -0.003545171581208706,
+            0.006891379598528147,
+            0.02600274793803692,
+            0.00716992886736989,
+            -0.013695938512682915,
+            -0.025351595133543015,
+            0.0063849263824522495,
+            -0.0023369193077087402,
+            0.012661327607929707,
+            0.0018241355428472161,
+            0.028592893853783607,
+            0.02516348287463188,
+            0.0012191049754619598,
+            0.018854524940252304,
+            -0.014238567091524601,
+            0.04338132590055466,
+            -0.019303098320961,
+            -0.00472448393702507,
+            -0.02801409177482128,
+            0.0010409420356154442,
+            -0.008291360922157764,
+            0.007980253547430038,
+            -0.006544097326695919,
+            0.002765595680102706,
+            -0.020750107243657112,
+            0.01979508064687252,
+            0.02079351618885994,
+            -0.023730944842100143,
+            0.006739443633705378,
+            -0.024859612807631493,
+            0.01417345181107521,
+            -0.029258519411087036,
+            -0.00961537379771471,
+            -0.017190465703606606,
+            -0.006269165780395269,
+            -0.025886988267302513,
+            0.0021433818619698286,
+            0.0009595477604307234,
+            0.02238522656261921,
+            -0.0051332637667655945,
+            0.003635609755292535,
+            0.01187270786613226,
+            -0.01437603309750557,
+            -0.011981233023107052,
+            0.019245218485593796,
+            0.02559758722782135,
+            0.007929608225822449,
+            0.018319131806492805,
+            0.02089480683207512,
+            -0.01440497301518917,
+            -0.04427846893668175,
+            0.0009749222081154585,
+            -0.003903306322172284,
+            -0.001428921241313219,
+            0.004565312992781401,
+            0.012972434051334858,
+            -0.02593039907515049,
+            -0.001539255608804524,
+            -0.004160150419920683,
+            0.0033823831472545862,
+            -0.004536372609436512,
+            0.015902627259492874,
+            0.025481825694441795,
+            -0.013218426145613194,
+            -0.013804464600980282,
+            -0.011366254650056362,
+            -0.005361167713999748,
+            0.008436061441898346,
+            -0.015396174043416977,
+            0.0008849363657645881,
+            -0.0034891001414507627,
+            -0.015511934645473957,
+            -0.00976730976253748,
+            0.002360433107241988,
+            -0.012943494133651257,
+            -0.008566292002797127,
+            0.006844351999461651,
+            -0.027637869119644165,
+            -0.01640908047556877,
+            0.019158396869897842,
+            -0.015164652839303017,
+            0.005118793807923794,
+            0.013746583834290504,
+            0.008812284097075462,
+            0.011149203404784203,
+            0.030097784474492073,
+            -0.00984689500182867,
+            0.0038562784902751446,
+            -0.021763013675808907,
+            -0.016770832240581512,
+            -0.0096804890781641,
+            0.01797184906899929,
+            0.003168949391692877,
+            -0.011829297058284283,
+            0.007647441700100899,
+            -0.012379161082208157,
+            -0.013232896104454994,
+            -0.006949259899556637,
+            0.001656825072132051,
+            -0.008645878173410892,
+            0.015265943482518196,
+            -0.01039675809442997,
+            -0.004131210036575794,
+            0.012075289152562618,
+            0.036696143448352814,
+            -0.0035415540914982557,
+            -0.00988307036459446,
+            -0.0014334431616589427,
+            0.01501995138823986,
+            0.008602467365562916,
+            -0.004351879004389048,
+            -0.020634345710277557,
+            -0.015700045973062515,
+            -0.0022302023135125637,
+            -0.010780216194689274,
+            -0.015627695247530937,
+            0.02271803840994835,
+            -0.021632781252264977,
+            -0.018854524940252304,
+            -0.018782174214720726,
+            -0.0158881563693285,
+            0.00048113043885678053,
+            0.03270240128040314,
+            0.0147739602252841,
+            0.0024780025705695152,
+            0.0038020156789571047,
+            -0.010056711733341217,
+            -0.027478698641061783,
+            0.024584680795669556,
+            0.011380724608898163,
+            -0.030097784474492073,
+            -0.009275326505303383,
+            -0.0187966451048851,
+            -0.004066094756126404,
+            0.007821083068847656,
+            -0.0022591424640268087,
+            0.05417601019144058,
+            -0.0007131040329113603,
+            0.006478982046246529,
+            -0.026682842522859573,
+            0.008298595435917377,
+            -0.015555345453321934,
+            -0.02192218415439129,
+            -0.010990031994879246,
+            -0.012299574911594391,
+            -0.009470673277974129,
+            -0.016336729750037193,
+            -0.002657070057466626,
+            -0.011438604444265366,
+            0.006471747066825628,
+            -0.015092302113771439,
+            -0.0034456897992640734,
+            -0.02407822757959366,
+            -0.004507432691752911,
+            0.005683127325028181,
+            -0.0007497314363718033,
+            0.008964220061898232,
+            -0.01863747462630272,
+            -0.006486217025667429,
+            0.0014054073253646493,
+            0.010794686153531075,
+            0.007886198349297047,
+            -0.0038490435108542442,
+            -0.014115571044385433,
+            -0.013167780824005604,
+            -0.008616937324404716,
+            0.01001330092549324,
+            -0.00699628796428442,
+            0.008096014149487019,
+            -0.01664060167968273,
+            -0.014643729664385319,
+            0.004062477499246597,
+            0.03328120335936546,
+            0.01490419078618288,
+            -0.009065510705113411,
+            -0.00961537379771471,
+            -0.014296446926891804,
+            0.034004706889390945,
+            -0.012777088209986687,
+            -0.014585848897695541,
+            0.012625152245163918,
+            0.022443106397986412,
+            -0.0007632971392013133,
+            0.0045508430339396,
+            0.010440168902277946,
+            -0.00010264718730468303,
+            -0.0007316438714042306,
+            0.0005679509486071765,
+            0.005328610073775053,
+            0.00020608570775948465,
+            0.028144322335720062,
+            -0.009065510705113411,
+            0.014419442974030972,
+            -0.004594252910465002,
+            0.006887761875987053,
+            -0.028100911527872086,
+            -0.0024906639009714127,
+            -0.006522392388433218,
+            -0.005965293850749731,
+            0.004771511536091566,
+            -0.0015076022827997804,
+            0.01774032786488533,
+            -0.03044506534934044,
+            -0.027305057272315025,
+            0.016669541597366333,
+            -0.02112632989883423,
+            -0.007958549074828625,
+            0.005433517973870039,
+            -0.014072161167860031,
+            0.008710993453860283,
+            0.006507922429591417,
+            -0.022327346727252007,
+            -0.01607626862823963,
+            0.0019588882569223642,
+            -0.0023206404875963926,
+            0.0027221853379160166,
+            0.01661166176199913,
+            -0.010548694059252739,
+            -0.012545567005872726,
+            -0.00761850131675601,
+            0.012914554215967655,
+            -0.005212849471718073,
+            0.024165047332644463,
+            -0.013261836022138596,
+            -0.0005720206536352634,
+            0.0009541214676573873,
+            -0.005976146552711725,
+            0.015613225288689137,
+            0.015468524768948555,
+            -0.0035071875900030136,
+            -0.0010002448689192533,
+            0.0003527084190864116,
+            -0.006847969256341457,
+            0.011033442802727222,
+            0.010635514743626118,
+            0.01460755430161953,
+            -0.004102270118892193,
+            0.012531096115708351,
+            -0.004297616425901651,
+            0.0027203767094761133,
+            -0.02079351618885994,
+            0.0040588597767055035,
+            0.002065605018287897,
+            0.018232312053442,
+            -0.020214714109897614,
+            -0.011800357140600681,
+            -0.031110690906643867,
+            -0.004529137630015612,
+            -0.01691553369164467,
+            0.0022012621629983187,
+            0.030155664309859276,
+            0.0015817615203559399,
+            -0.010541459545493126,
+            0.014629259705543518,
+            0.031226450577378273,
+            0.009506847709417343,
+            0.0004411115951370448,
+            0.010830861516296864,
+            -0.011532660573720932,
+            0.010592104867100716,
+            0.012480450794100761,
+            -0.01846383325755596,
+            -0.0015636739553883672,
+            -0.0007429486140608788,
+            -0.003494526259601116,
+            0.009825189597904682,
+            0.007517211139202118,
+            -0.004322939086705446,
+            0.017682448029518127,
+            -0.027015654370188713,
+            0.0036591235548257828,
+            -0.0030115870758891106,
+            -0.004681073594838381,
+            -0.027305057272315025,
+            -0.006352368742227554,
+            0.01747986674308777,
+            -0.010519754141569138,
+            0.058574918657541275,
+            0.002271803794428706,
+            0.007922373712062836,
+            -0.027536578476428986,
+            -0.005230937153100967,
+            -0.008906339295208454,
+            0.0034493072889745235,
+            -0.007184398826211691,
+            0.00761126633733511,
+            0.004008214455097914,
+            0.00022835608979221433,
+            -0.0035885819233953953,
+            0.0020583700388669968,
+            0.0022917003370821476,
+            0.016872122883796692,
+            0.00506453076377511,
+            0.03524913638830185,
+            -0.007727027405053377,
+            -0.005151351448148489,
+            -0.011062382720410824,
+            0.005932736210525036,
+            0.019042637199163437,
+            0.0002742533979471773,
+            0.006956494878977537,
+            0.011156437918543816,
+            0.011395194567739964,
+            0.013985340483486652,
+            0.018261251971125603,
+            -0.0033950444776564837,
+            0.00500665046274662,
+            0.01622096821665764,
+            0.00494153518229723,
+            -0.01410833653062582,
+            -0.011127498000860214,
+            -0.004297616425901651,
+            0.018290191888809204,
+            -0.007466565817594528,
+            0.011207083240151405,
+            0.006761149037629366,
+            0.00582782831043005,
+            0.01016523689031601,
+            -0.01178588718175888,
+            -0.00023604331363458186,
+            0.012777088209986687,
+            -0.030097784474492073,
+            0.012574506923556328,
+            0.0029301929753273726,
+            -0.00789343286305666,
+            0.00244906242005527,
+            -0.0076980870217084885,
+            -0.009065510705113411,
+            -0.0020836926996707916,
+            -0.009145095944404602,
+            0.0005069052567705512,
+            0.015251473523676395,
+            -0.012133168987929821,
+            0.0023658594582229853,
+            -0.003762223059311509,
+            -0.017103644087910652,
+            0.02595933899283409,
+            -0.018290191888809204,
+            0.005183909088373184,
+            0.014368797652423382
+          ],
+          "title": "Introduction And Related Work",
+          "keyphrases": [
+            "conditional computation",
+            "model capacity",
+            "Sparsely-Gated Mixture-of-Experts Layer",
+            "language modeling",
+            "machine translation"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "87fcb982-54d1-455e-8120-6faf8a8ae864",
+      "type": "child",
+      "source": {
+        "id": "14339b1f-53f8-479b-9da7-9430b174b9ee",
+        "properties": {
+          "page_content": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer\n\nNoam Shazeer1, Azalia Mirhoseini∗†1, Krzysztof Maziarz∗2, Andy Davis1, Quoc Le1, Geoffrey Hinton1and Jeff Dean1 1Google Brain, {noam,azalia,andydavis,qvl,geoffhinton,jeff}@google.com 2Jagiellonian University, Cracow, krzysztof.maziarz@student.uj.edu.pl\n\nAbstract\n\nThe capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost.\n\n1 Introduction And Related Work 1.1 Conditional Computation\n\nExploiting scale in both training data and model size has been central to the success of deep learning. When datasets are sufficiently large, increasing the capacity (number of parameters) of neural networks can give much better prediction accuracy. This has been shown in domains such as text (Sutskever et al., 2014; Bahdanau et al., 2014; Jozefowicz et al., 2016; Wu et al., 2016), images (Krizhevsky et al., 2012; Le et al., 2012), and audio (Hinton et al., 2012; Amodei et al., 2015). For typical deep learning models, where the entire model is activated for every example, this leads to a roughly quadratic blow-up in training costs, as both the model size and the number of training examples increase. Unfortunately, the advances in computing power and distributed computation fall short of meeting such demand.\n\nVarious forms of conditional computation have been proposed as a way to increase model capacity without a proportional increase in computational costs (Davis & Arel, 2013; Bengio et al., 2013; Eigen et al., 2013; Ludovic Denoyer, 2014; Cho & Bengio, 2014; Bengio et al., 2015; Almahairi et al., 2015). In these schemes, large parts of a network are active or inactive on a per-example basis. The gating decisions may be binary or sparse and continuous, stochastic or deterministic. Various forms of reinforcement learning and back-propagation are proposed for trarining the gating decisions.\n\nWhile these ideas are promising in theory, no work to date has yet demonstrated massive improvements in model capacity, training time, or model quality. We blame this on a combination of the following challenges:\n\nModern computing devices, especially GPUs, are much faster at arithmetic than at branching. Most of the works above recognize this and propose turning on/off large chunks of the network with each gating decision.\n\nLarge batch sizes are critical for performance, as they amortize the costs of parameter transfers and updates. Conditional computation reduces the batch sizes for the conditionally active chunks of the network.\n\nNetwork bandwidth can be a bottleneck. A cluster of GPUs may have computational power thousands of times greater than the aggregate inter-device network bandwidth. To be computationally efficient, the relative computational versus network demands of an algorithm must exceed this ratio. Embedding layers, which can be seen as a form of conditional computation, are handicapped by this very problem. Since the embeddings generally need to be sent across the network, the number of (example, parameter) interactions is limited by network bandwidth instead of computational capacity.\n\nDepending on the scheme, loss terms may be necessary to achieve the desired level of sparsity per-chunk and/or per example. Bengio et al. (2015) use three such terms. These issues can affect both model quality and load-balancing.\n\nModel capacity is most critical for very large data sets. The existing literature on conditional computation deals with relatively small image recognition data sets consisting of up to 600,000 images. It is hard to imagine that the labels of these images provide a sufficient signal to adequately train a model with millions, let alone billions of parameters. In this work, we for the first time address all of the above challenges and finally realize the promise of conditional computation. We obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets.\n\n1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1). All parts of the network are trained jointly by back-propagation.\n\nWhile the introduced technique is generic, in this paper we focus on language modeling and machine translation tasks, which are known to benefit from very large models. In particular, we apply a MoE convolutionally between stacked LSTM layers (Hochreiter & Schmidhuber, 1997), as in Figure 1.\n\nThe MoE is called once for each position in the text, selecting a potentially different combination of experts at each position. The different experts tend to become highly specialized based on syntax and semantics (see Appendix E Table 9). On both language modeling and machine translation benchmarks, we improve on best published results at a fraction of the computational cost.\n\n1.3 Related Work On Mixtures Of Experts\n\nSince its introduction more than two decades ago (Jacobs et al., 1991; Jordan & Jacobs, 1994), the mixture-of-experts approach has been the subject of much research. Different types of expert architectures hae been proposed such as SVMs (Collobert et al., 2002), Gaussian Processes (Tresp, 2001; Theis & Bethge, 2015; Deisenroth & Ng, 2015), Dirichlet Processes (Shahbaba & Neal, 2009), and deep networks. Other work has focused on different expert configurations such as a hierarchical structure (Yao et al., 2009), infinite numbers of experts (Rasmussen & Ghahramani, 2002), and adding experts sequentially (Aljundi et al., 2016). Garmash & Monz (2016) suggest an ensemble model in the format of mixture of experts for machine translation. The gating network is trained on a pre-trained ensemble NMT model. The works above concern top-level mixtures of experts. The mixture of experts is the whole model. Eigen et al. (2013) introduce the idea of using multiple MoEs with their own gating networks as parts of a deep model. It is intuitive that the latter approach is more powerful, since complex problems may contain many sub-problems each requiring different experts. They also allude in their conclusion to the potential to introduce sparsity, turning MoEs into a vehicle for computational computation.\n\nOur work builds on this use of MoEs as a general purpose neural network component. While Eigen et al. (2013) uses two stacked MoEs allowing for two sets of gating decisions, our convolutional application of the MoE allows for different gating decisions at each position in the text. We also realize sparse gating and demonstrate its use as a practical way to massively increase model capacity.\n\n2 The Structure Of The Mixture-Of-Experts Layer\n\nThe Mixture-of-Experts (MoE) layer consists of a set of n \"expert networks\" E1, · · · , En, and a \"gating network\" G whose output is a sparse n-dimensional vector. Figure 1 shows an overview of the MoE module. The experts are themselves neural networks, each with their own parameters.\n\nAlthough in principle we only require that the experts accept the same sized inputs and produce the same-sized outputs, in our initial investigations in this paper, we restrict ourselves to the case where the models are feed-forward networks with identical architectures, but with separate parameters.\n\nLet us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows:\n\n$$y=\\sum_{i=1}^{n}G(x){i}E{i}(x)$$ $$(\\mathbf{l})$$ G(x)iEi(x) (1) We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example. If the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network. In the following we focus on ordinary MoEs. We provide more details on hierarchical MoEs in Appendix B. Our implementation is related to other models of conditional computation. A MoE whose experts are simple weight matrices is similar to the parameterized weight matrix proposed in (Cho & Bengio, 2014). A MoE whose experts have one hidden layer is similar to the block-wise dropout described in (Bengio et al., 2015), where the dropped-out layer is sandwiched between fully-activated layers.\n\n2.1 Gating Network\n\nSoftmax Gating: A simple choice of non-sparse gating function (Jordan & Jacobs, 1994) is to multiply the input by a trainable weight matrix Wg and then apply the Sof tmax function.\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ $\\eqref{eq:walpha}$. Gσ(x) = Sof tmax(x · Wg) (2) Noisy Top-K Gating: We add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to −∞ (which causes the corresponding gate values to equal 0). The sparsity serves to save computation, as described above. While this form of sparsity creates some theoretically scary discontinuities in the output of gating function, we have not yet observed this to be a problem in practice. The noise term helps with load balancing, as will be discussed in Appendix A. The amount of noise per component is controlled by a second trainable weight matrix Wnoise.\n\n$$G(x)=S o f t m a x(K e e p T o p K(H(x),k))$$ $$(4)$$ $$(S)$$ G(x) = Sof tmax(KeepT opK(H(x), k)) (3) $$H(x){i}=(x\\cdot W{g}){i}+S t a n d a r d N o r m a l()\\cdot S o f t p l u s((x\\cdot W{n o i s e}){i})$$ $KeepTopK(v,k){i}=\\begin{cases}v_{i}&\\text{if}v_{i}\\text{is in the top}k\\text{elements of}v.\\ -\\infty&\\text{otherwise.}\\end{cases}$ Training the Gating Network We train the gating network by simple back-propagation, along with the rest of the model. If we choose k > 1, the gate values for the top k experts have nonzero derivatives with respect to the weights of the gating network. This type of occasionally-sensitive behavior is described in (Bengio et al., 2013) with respect to noisy rectifiers. Gradients also backpropagate through the gating network to its inputs. Our method differs here from (Bengio et al., 2015) who use boolean gates and a REINFORCE-style approach to train the gating network.\n\n3 Addressing Performance Challenges 3.1 The Shrinking Batch Problem\n\nOn modern CPUs and GPUs, large batch sizes are necessary for computational efficiency, so as to amortize the overhead of parameter loads and updates. If the gating network chooses k out of n experts for each example, then for a batch of b examples, each expert receives a much smaller batch of approximately kb n b examples. This causes a naive MoE implementation to become very inefficient as the number of experts increases. The solution to this shrinking batch problem is to make the original batch size as large as possible. However, batch size tends to be limited by the memory necessary to store activations between the forwards and backwards passes. We propose the following techniques for increasing the batch size: Mixing Data Parallelism and Model Parallelism: In a conventional distributed training setting, multiple copies of the model on different devices asynchronously process distinct batches of data, and parameters are synchronized through a set of parameter servers. In our technique, these different batches run synchronously so that they can be combined for the MoE layer. We distribute the standard layers of the model and the gating network according to conventional data-parallel schemes, but keep only one shared copy of each expert. Each expert in the MoE layer receives a combined batch consisting of the relevant examples from all of the data-parallel input batches. The same set of devices function as data-parallel replicas (for the standard layers and the gating networks) and as model-parallel shards (each hosting a subset of the experts). If the model is distributed over d devices, and each device processes a batch of size b, each expert receives a batch of approximately kbd nexamples. Thus, we achieve a factor of d improvement in expert batch size.\n\nIn the case of a hierarchical MoE (Section B), the primary gating network employs data parallelism, and the secondary MoEs employ model parallelism. Each secondary MoE resides on one device.\n\nThis technique allows us to increase the number of experts (and hence the number of parameters) by proportionally increasing the number of devices in the training cluster. The total batch size increases, keeping the batch size per expert constant. The memory and bandwidth requirements per device also remain constant, as do the step times, as does the amount of time necessary to process a number of training examples equal to the number of parameters in the model. It is our goal to train a trillionparameter model on a trillion-word corpus. We have not scaled our systems this far as of the writing of this paper, but it should be possible by adding more hardware.\n\nTaking Advantage of Convolutionality: In our language models, we apply the same MoE to each time step of the previous layer. If we wait for the previous layer to finish, we can apply the MoE to all the time steps together as one big batch. Doing so increases the size of the input batch to the MoE layer by a factor of the number of unrolled time steps.\n\nIncreasing Batch Size for a Recurrent MoE: We suspect that even more powerful models may involve applying a MoE recurrently. For example, the weight matrices of a LSTM or other RNN could be replaced by a MoE. Sadly, such models break the convolutional trick from the last paragraph, since the input to the MoE at one timestep depends on the output of the MoE at the previous timestep. Gruslys et al. (2016) describe a technique for drastically reducing the number of stored activations in an unrolled RNN, at the cost of recomputing forward activations. This would allow for a large increase in batch size.\n\n3.2 Network Bandwidth\n\nAnother major performance concern in distributed computing is network bandwidth. Since the experts are stationary (see above) and the number of gating parameters is small, most of the communication involves sending the inputs and outputs of the experts across the network. To maintain computational efficiency, the ratio of an expert's computation to the size of its input and output must exceed the ratio of computational to network capacity of the computing device. For GPUs, this may be thousands to one. In our experiments, we use experts with one hidden layer containing thousands of RELU-activated units. Since the weight matrices in the expert have sizes input_size×hidden_size and hidden_size × output_size, the ratio of computation to input and output is equal to the size of the hidden layer. Conveniently, we can increase computational efficiency simply by using a larger hidden layer, or more hidden layers.\n\n4 Balancing Expert Utilization\n\nWe have observed that the gating network tends to converge to a state where it always produces large weights for the same few experts. This imbalance is self-reinforcing, as the favored experts are trained more rapidly and thus are selected even more by the gating network. Eigen et al. (2013) describe the same phenomenon, and use a hard constraint at the beginning of training to avoid this local minimum. Bengio et al. (2015) include a soft constraint on the batch-wise average of each gate.1 We take a soft constraint approach. We define the importance of an expert relative to a batch of training examples to be the batchwise sum of the gate values for that expert. We define an additional loss Limportance, which is added to the overall loss function for the model. This loss is equal to the square of the coefficient of variation of the set of importance values, multiplied by a hand-tuned scaling factor wimportance. This additional loss encourages all experts to have equal importance.\n\n$$I m p o r t a n c e(X)=\\sum_{x\\in X}G(x)$$ $$L_{importance}(X)=w_{importance}\\cdot CV(Importance(X))^{2}\\tag{7}$$ $$(6)$$\n\nWhile this loss function can ensure equal importance, experts may still receive very different numbers of examples. For example, one expert may receive a few examples with large weights, and another may receive many examples with small weights. This can cause memory and performance problems on distributed hardware. To solve this problem, we introduce a second loss function, Lload , which ensures balanced loads. Appendix A contains the definition of this function, along with experimental results.\n\n5 Experiments 5.1 1 Billion Word Language Modeling Benchmark\n\nDataset: This dataset, introduced by (Chelba et al., 2013) consists of shuffled unique sentences from news articles, totaling approximately 829 million words, with a vocabulary of 793,471 words.\n\nPrevious State-of-the-Art: The best previously published results (Jozefowicz et al., 2016) use models consisting of one or more stacked Long Short-Term Memory (LSTM) layers (Hochreiter & Schmidhuber, 1997; Gers et al., 2000). The number of parameters in the LSTM layers of these models vary from 2 million to 151 million. Quality increases greatly with parameter count, as do computational costs. Results for these models form the top line of Figure 2-right. MoE Models: Our models consist of two stacked LSTM layers with a MoE layer between them (see Figure 1). We vary the sizes of the layers and the number of experts. For full details on model architecture, training regimen, additional baselines and results, see Appendix C.\n\nLow Computation, Varied Capacity: To investigate the effects of adding capacity, we trained\n\na series of MoE models all with roughly equal computational costs: about 8 million multiply-andadds per training example per timestep in the forwards pass, excluding the softmax layer. We call this metric (ops/timestep). We trained models with flat MoEs containing 4, 32, and 256 experts, and models with hierarchical MoEs containing 256, 1024, and 4096 experts. Each expert had about 1 million parameters. For all the MoE layers, 4 experts were active per input.\n\nThe results of these models are shown in Figure 2-left. The model with 4 always-active experts performed (unsurprisingly) similarly to the computationally-matched baseline models, while the largest of the models (4096 experts) achieved an impressive 24% lower perplexity on the test set. Figure 2: Model comparison on 1-Billion-Word Language-Modeling Benchmark. On the left, we plot test perplexity as a function of model capacity for models with similar computational budgets of approximately 8-million-ops-per-timestep. On the right, we plot test perplexity as a function of computational budget. The top line represents the LSTM models from (Jozefowicz et al., 2016).\n\nThe bottom line represents 4-billion parameter MoE models with different computational budgets.\n\nVaried Computation, High Capacity: In addition to the largest model from the previous section, we trained two more MoE models with similarly high capacity (4 billion parameters), but higher computation budgets. These models had larger LSTMs, and fewer but larger and experts. Details\n\nTest Test #Parameters ops/timestep Training TFLOPS Perplexity Perplexity excluding embedding Time /GPU 10 epochs 100 epochs and softmax layers 10 epochs Best Published Results 34.7 30.6 151 million 151 million 59 hours, 32 k40s 1.09 Low-Budget MoE Model 34.1 4303 million 8.9 million 15 hours, 16 k40s 0.74 Medium-Budget MoE Model 31.3 4313 million 33.8 million 17 hours, 32 k40s 1.22 High-Budget MoE Model 28.0 4371 million 142.7 million 47 hours, 32 k40s 1.56\n\nTable 1: Summary of high-capacity MoE-augmented models with varying computational budgets, vs. best previously published results (Jozefowicz et al., 2016). Details in Appendix C. can be found in Appendix C.2. Results of these three models form the bottom line of Figure 2-right.\n\nTable 1 compares the results of these models to the best previously-published result on this dataset .\n\nEven the fastest of these models beats the best published result (when controlling for the number of training epochs), despite requiring only 6% of the computation.\n\nComputational Efficiency: We trained our models using TensorFlow (Abadi et al., 2016) on clusters containing 16-32 Tesla K40 GPUs. For each of our models, we determine computational efficiency in TFLOPS/GPU by dividing the number of floating point operations required to process one training batch by the observed step time and the number of GPUs in the cluster. The operation counts used here are higher than the ones we report in our ops/timestep numbers in that we include the backwards pass, we include the importance-sampling-based training of the softmax layer, and we count a multiply-and-add as two separate operations. For all of our MoE models, the floating point operations involved in the experts represent between 37% and 46% of the total.\n\nFor our baseline models wtih no MoE, observed computational efficiency ranged from 1.07-1.29 TFLOPS/GPU. For our low-computation MoE models, computation efficiency ranged from 0.740.90 TFLOPS/GPU, except for the 4-expert model which did not make full use of the available parallelism. Our highest-computation MoE model was more efficient at 1.56 TFLOPS/GPU, likely due to the larger matrices. These numbers represent a significant fraction of the theoretical maximum of 4.29 TFLOPS/GPU claimed by NVIDIA. Detailed results are in Appendix C, Table 7.\n\n5.2 100 BILLION WORD GOOGLE NEWS CORPUS\n\nOn the 1-billion-word corpus, adding additional capacity seems to produce diminishing returns as the number of parameters in the MoE layer exceeds 1 billion, as can be seen in Figure 2-left. We hypothesized that for a larger training set, even higher capacities would produce significant quality improvements. We constructed a similar training set consisting of shuffled unique sentences from Google's internal news corpus, totalling roughly 100 billion words. Similarly to the previous section, we tested a series of models with similar computational costs of about 8 million ops/timestep. In addition to a baseline LSTM model, we trained models augmented with MoE layers containing 32, 256, 1024, 4096, 16384, 65536, and 131072 experts. This corresponds to up to 137 billion parameters in the MoE layer. Details on architecture, training, and results are given in Appendix D.\n\nResults: Figure 3 shows test perplexity as a function of capacity after training on 10 billion words (top line) and 100 billion words (bottom line). When training over the full 100 billion words, test perplexity improves significantly up to 65536 experts (68 billion parameters), dropping 39% lower than the computationally matched baseline, but degrades at 131072 experts, possibly a result of too much sparsity. The widening gap between the two lines demonstrates (unsurprisingly) that increased model capacity helps more on larger training sets. Even at 65536 experts (99.994% layer sparsity), computational efficiency for the model stays at a respectable 0.72 TFLOPS/GPU.\n\n5.3 Machine Translation (Single Language Pair)\n\nModel Architecture: Our model was a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decreased the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We inserted MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). Each MoE layer contained up to 2048 experts each with about two million parameters, adding a total of about 8 billion parameters to the models. Further details on model architecture, testing procedure and results can be found in Appendix E.\n\nDatasets: We benchmarked our method on the WMT'14 En→Fr and En→De corpora, whose training sets have 36M sentence pairs and 5M sentence pairs, respectively. The experimental protocols were also similar to those in (Wu et al., 2016): newstest2014 was used as the test set to compare against previous work (Luong et al., 2015a; Zhou et al., 2016; Wu et al., 2016), while the combination of newstest2012 and newstest2013 was used as the development set. We also tested the same model on a Google's Production English to French data.\n\nTable 2: Results on WMT'14 En→ Fr newstest2014 (bold values represent best results). Model Test Test ops/timenstep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.69 40.35 85M 8.7B 3 days/64 k40s MoE with 2048 Experts (longer training) 2.63 40.56 85M 8.7B 6 days/64 k40s GNMT (Wu et al., 2016) 2.79 39.22 214M 278M 6 days/96 k80s GNMT+RL (Wu et al., 2016) 2.96 39.92 214M 278M 6 days/96 k80s PBMT (Durrani et al., 2014) 37.0 LSTM (6-layer) (Luong et al., 2015b) 31.5 LSTM (6-layer+PosUnk) (Luong et al., 2015b) 33.1 DeepAtt (Zhou et al., 2016) 37.7 DeepAtt+PosUnk (Zhou et al., 2016) 39.2\n\nTable 3: Results on WMT'14 En → De newstest2014 (bold values represent best results). Model Test Test ops/timestep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 4.64 26.03 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 5.25 24.91 214M 278M 1 day/96 k80s GNMT +RL (Wu et al., 2016) 8.08 24.66 214M 278M 1 day/96 k80s PBMT (Durrani et al., 2014) 20.7 DeepAtt (Zhou et al., 2016) 20.6\n\nTable 4: Results on the Google Production En→ Fr dataset (bold values represent best results). Model Eval Eval Test Test ops/timestep Total Training Perplexity BLEU Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.60 37.27 2.69 36.57 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 2.78 35.80 2.87 35.56 214M 278M 6 days/96 k80s\n\nResults: Tables 2, 3, and 4 show the results of our largest models, compared with published results. Our approach achieved BLEU scores of 40.56 and 26.03 on the WMT'14 En→Fr and En→De benchmarks. As our models did not use RL refinement, these results constitute significant gains of 1.34 and 1.12 BLEU score on top of the strong baselines in (Wu et al., 2016). The perplexity scores are also better.2 On the Google Production dataset, our model achieved 1.01 higher test BLEU score even after training for only one sixth of the time.\n\n5.4 Multilingual Machine Translation\n\nDataset: (Johnson et al., 2016) train a single GNMT (Wu et al., 2016) model on a very large combined dataset of twelve language pairs. Results are somewhat worse than those for 12 separately trained single-pair GNMT models. This is not surprising, given that the twelve models have 12 times the capacity and twelve times the aggregate training of the one model. We repeat this experiment with a single MoE-augmented model. See Appendix E for details on model architecture.\n\nWe train our model on the same dataset as (Johnson et al., 2016) and process the same number of training examples (about 3 billion sentence pairs). Our training time was shorter due to the lower computational budget of our model.\n\nResults: Results for the single-pair GNMT models, the multilingual GNMT model and the multilingual MoE model are given in Table 5. The MoE model achieves 19% lower perplexity on the dev set than the multilingual GNMT model. On BLEU score, the MoE model significantly beats the multilingual GNMT model on 11 of the 12 language pairs (by as much as 5.84 points), and even beats the monolingual GNMT models on 8 of 12 language pairs. The poor performance on English → Korean seems to be a result of severe overtraining, as for the rarer language pairs a small number of real examples were highly oversampled in the training corpus.\n\nTable 5: Multilingual Machine Translation (bold values represent best results). GNMT-Mono GNMT-Multi MoE-Multi MoE-Multi vs. GNMT-Multi Parameters 278M / model 278M 8.7B ops/timestep 212M 212M 102M training time, hardware various 21 days, 96 k20s 12 days, 64 k40s Perplexity (dev) 4.14 3.35 -19% French → English Test BLEU 36.47 34.40 37.46 +3.06 German → English Test BLEU 31.77 31.17 34.80 +3.63 Japanese → English Test BLEU 23.41 21.62 25.91 +4.29 Korean → English Test BLEU 25.42 22.87 28.71 +5.84 Portuguese → English Test BLEU 44.40 42.53 46.13 +3.60 Spanish → English Test BLEU 38.00 36.04 39.39 +3.35 English → French Test BLEU 35.37 34.00 36.59 +2.59 English → German Test BLEU 26.43 23.15 24.53 +1.38 English → Japanese Test BLEU 23.66 21.10 22.78 +1.68 English → Korean Test BLEU 19.75 18.41 16.62 -1.79 English → Portuguese Test BLEU 38.40 37.35 37.90 +0.55 English → Spanish Test BLEU 34.50 34.25 36.21 +1.96\n\n6 Conclusion\n\nThis work is the first to demonstrate major wins from conditional computation in deep networks.\n\nWe carefully identified the design considerations and challenges of conditional computing and addressed them with a combination of algorithmic and engineering solutions. While we focused on text, conditional computation may help in other domains as well, provided sufficiently large training sets. We look forward to seeing many novel implementations and applications of conditional computation in the years to come.\n\nAcknowledgments\n\nWe would like to thank all of the members of the Google Brain and Google Translate teams who helped us with this project, in particular Zhifeng Chen, Yonghui Wu, and Melvin Johnson. Thanks also to our anonymous ICLR reviewers for the helpful suggestions on making this paper better.\n\n2Reported perplexities relative to the tokenization used by both our models and GNMT.\n\nReferences\n\nMartín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Gordon Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda B. Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. CoRR, abs/1603.04467, 2016. URL http://arxiv.org/abs/1603.04467.\n\nRahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. CoRR, abs/1611.06194, 2016. URL http://arxiv.org/abs/1611.\n\n06194.\n\nA. Almahairi, N. Ballas, T. Cooijmans, Y. Zheng, H. Larochelle, and A. Courville. Dynamic Capacity Networks. ArXiv e-prints, November 2015.\n\nDario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan, Christopher Fougner, Tony Han, Awni Y. Hannun, Billy Jun, Patrick LeGresley, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595, 2015. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.\n\nEmmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297, 2015.\n\nYoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.\n\nCiprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling.\n\narXiv preprint arXiv:1312.3005, 2013.\n\nK. Cho and Y. Bengio. Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning. ArXiv e-prints, June 2014. Ronan Collobert, Samy Bengio, and Yoshua Bengio. A parallel mixture of SVMs for very large scale problems. Neural Computing, 2002.\n\nAndrew Davis and Itamar Arel. Low-rank approximations for conditional feedforward computation in deep neural networks. arXiv preprint arXiv:1312.4461, 2013. Marc Peter Deisenroth and Jun Wei Ng. Distributed Gaussian processes. In ICML, 2015.\n\nJohn Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization, 2010.\n\nNadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. Edinburgh's phrase-based machine translation systems for wmt-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation, 2014.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013. Ekaterina Garmash and Christof Monz. Ensemble learning for multi-source neural machine translation. In staff.science.uva.nl/c.monz, 2016.\n\nFelix A. Gers, Jürgen A. Schmidhuber, and Fred A. Cummins. Learning to forget: Continual prediction with lstm. Neural Computation, 2000.\n\nAudrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. Memory-efficient backpropagation through time. CoRR, abs/1606.03401, 2016. URL http://arxiv.org/ abs/1606.03401. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2015.\n\nGeoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 2012. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.\n\nSergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.\n\nRobert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computing, 1991.\n\nMelvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation.\n\nCoRR, abs/1611.04558, 2016. URL http://arxiv.org/abs/1611.04558.\n\nMichael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm.\n\nNeural Computing, 1994. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.\n\nDiederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.\n\nReinhard Kneser and Hermann. Ney. Improved backingoff for m-gram language modeling., 1995.\n\nAlex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.\n\nQuoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeffrey Dean, and Andrew Y. Ng. Building high-level features using large scale unsupervised learning. In ICML, 2012. Patrick Gallinari Ludovic Denoyer. Deep sequential neural network. arXiv preprint arXiv:1410.0510, 2014.\n\nMinh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attentionbased neural machine translation. EMNLP, 2015a.\n\nMinh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the rare word problem in neural machine translation. ACL, 2015b.\n\nCarl Edward Rasmussen and Zoubin Ghahramani. Infinite mixtures of Gaussian process experts.\n\nNIPS, 2002.\n\nHasim Sak, Andrew W Senior, and Françoise Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH, pp. 338–342, 2014.\n\nMike Schuster and Kaisuke Nakajima. Japanese and Korean voice search. ICASSP, 2012.\n\nBabak Shahbaba and Radford Neal. Nonlinear models using dirichlet process mixtures. JMLR, 2009.\n\nIlya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks.\n\nIn NIPS, 2014.\n\nLucas Theis and Matthias Bethge. Generative image modeling using spatial LSTMs. In NIPS, 2015. Volker Tresp. Mixtures of Gaussian Processes. In NIPS, 2001.\n\nYonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.\n\nBangpeng Yao, Dirk Walther, Diane Beck, and Li Fei-fei. Hierarchical mixture of classification experts uncovers interactions between brain regions. In NIPS. 2009.\n\nWojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization.\n\narXiv preprint arXiv:1409.2329, 2014.\n\nJie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with fast-forward connections for neural machine translation. arXiv preprint arXiv:1606.04199, 2016.\n\nAppendices A Load-Balancing Loss\n\nAs discussed in section 4, for load-balancing purposes, we want to define an additional loss function to encourage experts to receive roughly equal numbers of training examples. Unfortunately, the number of examples received by an expert is a discrete quantity, so it can not be used in backpropagation. Instead, we define a smooth estimator Load(X) of the number of examples assigned to each expert for a batch X of inputs. The smoothness allows us to back-propagate gradients through the estimator. This is the purpose of the noise term in the gating function. We define P(x, i) as the probability that G(x)iis nonzero, given a new random choice of noise on element i, but keeping the already-sampled choices of noise on the other elements. To compute P(x, i), we note that the G(x)iis nonzero if and only if H(x)iis greater than the k th-greatest element of H(x) excluding itself. The probability works out to be:\n\n$$P(x,i)=P r\\Big((x\\cdot W_{g}){i}+S t a n d x N o r a l()\\cdot S o f t p u s((x\\cdot W{n o i s e})_{i})$$ $$>k t h_e x c l u d i n(H(x),k,i)\\Big)$$ $$({\\boldsymbol{8}})$$\n\n$$(9)$$ $$(10)$$ (8) Where kth_excluding(v, k, i) means the kth highest component of v, excluding component i. Simplifying, we get:\n\n$$P(x,i)=\\Phi{\\Big(}{\\frac{(x\\cdot W_{g}){i}-k t h_c x c l u d i n g(H(x),k,i)}{S o f t p l u s((x\\cdot W{n o i s e})_{i})}}{\\Big)}$$\n\nWhere Φ is the CDF of the standard normal distribution.\n\n$$L o a d(X){i}=\\sum{x\\in X}P(x,i)$$\n\n$$(11)$$ P(x, i) (10) We can now define the load loss to be the square of the coefficient of variation of the load vector, multiplied by a hand-tuned scaling factor wload.\n\n$$L_{l o a d}(X)=w_{l o a d}\\cdot C V(L o a d(X))^{2}$$ Lload(X) = wload · CV (Load(X))2(11) Initial Load Imbalance: To avoid out-of-memory errors, we need to initialize the network in a state of approximately equal expert load (since the soft constraints need some time to work). To accomplish this, we initialize the matrices Wg and Wnoise to all zeros, which yields no signal and some noise.\n\nExperiments: We trained a set of models with identical architecture (the MoE-256 model described in Appendix C), using different values of wimportance and wload. We trained each model for 10 epochs, then measured perplexity on the test set. We also measured the coefficients of variation in Importance and Load, as well as ratio of the load on the most overloaded expert to the average load. This last value is significant for load balancing purposes on distributed hardware. All of these metrics were averaged over several training batches.\n\nwimportance wload Test Perplexity CV (Importance(X)) CV (Load(X)) max(Load(X)) mean(Load(X)) 0.0 0.0 39.8 3.04 3.01 17.80 0.2 0.0 35.6 0.06 0.17 1.47 0.0 0.2 35.7 0.22 0.04 1.15 0.1 0.1 35.6 0.06 0.05 1.14 0.01 0.01 35.7 0.48 0.11 1.37 1.0 1.0 35.7 0.03 0.02 1.07\n\nTable 6: Experiments with different combinations of losses.\n\n13 Results: Results are reported in Table 6. All the combinations containing at least one the two losses led to very similar model quality, where having no loss was much worse. Models with higher values of wload had lower loads on the most overloaded expert.\n\nB Hierachical Mixture Of Experts\n\nIf the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network.3If the hierarchical MoE consists of a groups of b experts each, we denote the primary gating network by Gprimary, the secondary gating networks by (G1, G2..Ga), and the expert networks by (E0,0, E0,1..Ea,b). The output of the MoE is given by:\n\n$$y_{H}=\\sum_{i=1}^{a}\\sum_{j=1}^{b}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}\\cdot E{i,j}(x)$$ $$(12)$$ $$(13)$$ $$(14)$$\n\nOur metrics of expert utilization change to the following:\n\n$$I m p o r t a n c e_{H}(X){i,j}=\\sum{x\\in X}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}$$ $$L o a d{H}(X){i,j}={\\frac{L o a d{p r i m a r y}(X){i}\\cdot L o a d{i}(X^{(i)})_{j}}{|X^{(i)}|}}$$\n\nLoadprimary and Loadi deonte the Load functions for the primary gating network and i th secondary gating network respectively. X(i) denotes the subset of X for which Gprimary(x)i > 0.\n\nIt would seem simpler to let LoadH(X)i,j = Loadi(Xi)j , but this would not have a gradient with respect to the primary gating network, so we use the formulation above.\n\nC 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS C.1 8-MILLION-OPERATIONS-PER-TIMESTEP MODELS Model Architecture: Our model consists of five layers: a word embedding layer, a recurrent Long Short-Term Memory (LSTM) layer (Hochreiter & Schmidhuber, 1997; Gers et al., 2000), a MoE layer, a second LSTM layer, and a softmax layer. The dimensionality of the embedding layer, the number of units in each LSTM layer, and the input and output dimensionality of the MoE layer are all equal to 512. For every layer other than the softmax, we apply drouput (Zaremba et al., 2014) to the layer output, dropping each activation with probability DropP rob, otherwise dividing by (1 − DropP rob). After dropout, the output of the previous layer is added to the layer output.\n\nThis residual connection encourages gradient flow (He et al., 2015).\n\nMoE Layer Architecture: Each expert in the MoE layer is a feed forward network with one ReLU-activated hidden layer of size 1024 and an output layer of size 512. Thus, each expert contains [512 ∗ 1024] + [1024 ∗ 512] = 1M parameters. The output of the MoE layer is passed through a sigmoid function before dropout. We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts.\n\nWe call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers. Thus, each example is processed by exactly 4 experts for a total of 4M ops/timestep. The two LSTM layers contribute 2M ops/timestep each for the desired total of 8M.\n\n3 We have not found the need for deeper hierarchies.\n\nComputationally-Matched Baselines: The MoE-4 model does not employ sparsity, since all 4 experts are always used. In addition, we trained four more computationally-matched baseline models with no sparsity:\n\nMoE-1-Wide: The MoE layer consists of a single \"expert\" containing one ReLU-activated hidden layer of size 4096.\n\nMoE-1-Deep: The MoE layer consists of a single \"expert\" containing four ReLU-activated hidden layers, each with size 1024.\n\n4xLSTM-512: We replace the MoE layer with two additional 512-unit LSTM layers.\n\nLSTM-2048-512: The model contains one 2048-unit LSTM layer (and no MoE). The output of the LSTM is projected down to 512 dimensions (Sak et al., 2014). The next timestep of the LSTM receives the projected output. This is identical to one of the models published in (Jozefowicz et al., 2016). We re-ran it to account for differences in training regimen, and obtained results very similar to the published ones. Training: The models were trained on a cluster of 16 K40 GPUs using the synchronous method described in Section 3. Each batch consisted of a set of sentences totaling roughly 300,000 words. In the interest of time, we limited training to 10 epochs, (27,000 steps). Training took 12-16 hours for all models, except for MoE-4, which took 18 hours (since all the expert computation was performed on only 4 of 16 GPUs). We used the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 1000 training steps, and decreased after that so as to be proportional to the inverse square root of the step number. The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016). For each model, we performed a hyper-parmeter search to find the best dropout probability, in increments of 0.1.\n\nTo ensure balanced expert utilization we set wimportance = 0.1 and wload = 0.1, as described in Section 4 and Appendix A.\n\nResults: We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol. Results are reported in Table 7. For each model, we report the test perplexity, the computational budget, the parameter counts, the value of DropP rob, and the computational efficiency.\n\nModel Test Test ops/timestep #Params excluding Total Drop- TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params P rob per GPU 10 epochs (final) (millions) (billions) (observed) Kneser-Ney 5-gram* 67.6 0.00001 1.8 LSTM-512-512* 54.1 2.4 2.4 0.8 0.1 LSTM-1024-512* 48.2 4.7 4.7 0.8 0.1 LSTM-2048-512* 45.0 43.7 9.4 9.4 0.8 0.1 0.61 LSTM-2048-512 44.7 9.4 9.4 0.8 0.1 1.21 4xLSTM-512 46.0 8.4 8.4 0.8 0.1 1.07 MoE-1-Wide 46.1 8.4 8.4 0.8 0.1 1.29 MoE-1-Deep 45.7 8.4 8.4 0.8 0.1 1.29 MoE-4 45.0 8.4 8.4 0.8 0.1 0.52 MoE-32 39.7 8.4 37.8 0.9 0.1 0.87 MoE-256 35.7 8.6 272.9 1.1 0.1 0.81 MoE-256-h 36.0 8.4 272.9 1.1 0.1 0.89 MoE-1024-h 34.6 8.5 1079.0 1.9 0.2 0.90 MoE-4096-h 34.1 8.9 4303.4 5.1 0.2 0.74 2xLSTM-8192-1024* 34.7 30.6 151.0 151.0 1.8 0.25 1.09 MoE-34M 31.3 33.8 4313.9 6.0 0.3 1.22 MoE-143M 28.0 142.7 4371.1 6.0 0.4 1.56\n\nC.2 More Expensive Models\n\nWe ran two additional models (MoE-34M and MoE-143M) to investigate the effects of adding more computation in the presence of a large MoE layer. These models have computation budgets of 34M and 143M ops/timestep. Similar to the models above, these models use a MoE layer between two LSTM layers. The dimensionality of the embedding layer, and the input and output dimensionality of the MoE layer are set to 1024 instead of 512. For MoE-34M, the LSTM layers have 1024 units.\n\nFor MoE-143M, the LSTM layers have 4096 units and an output projection of size 1024 (Sak et al., 2014). MoE-34M uses a hierarchical MoE layer with 1024 experts, each with a hidden layer of size 2048. MoE-143M uses a hierarchical MoE layer with 256 experts, each with a hidden layer of size 8192. Both models have 4B parameters in the MoE layers. We searched for the best DropP rob for each model, and trained each model for 10 epochs.\n\nThe two models achieved test perplexity of 31.3 and 28.0 respectively, showing that even in the presence of a large MoE, more computation is still useful. Results are reported at the bottom of Table 7. The larger of the two models has a similar computational budget to the best published model from the literature, and training times are similar. Comparing after 10 epochs, our model has a lower test perplexity by 18%.\n\nD 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS Model Architecture: The models are similar in structure to the 8-million-operations-per-timestep models described in the previous section. We vary the number of experts between models, using an ordinary MoE layer with 32 experts and hierarchical MoE layers with 256, 1024, 4096, 16384, 65536 and 131072 experts. For the hierarchical MoE layers, the first level branching factors are 32, 32, 64, 128, 256 and 256, respectively.\n\nTraining: Models are trained on a cluster of 32 Tesla K40 GPUs, except for the last two models, which are trained on clusters of 64 and 128 GPUs so as to have enough memory for all the parameters. For all models, training batch sizes are approximately 2.5 million words. Models are trained once-through over about 100 billion words. We implement several memory optimizations in order to fit up to 1 billion parameters per GPU. First, we do not store the activations of the hidden layers of the experts, but instead recompute them on the backwards pass. Secondly, we modify the optimizer on the expert parameters to require less auxiliary storage: The Adam optimizer (Kingma & Ba, 2015) keeps first and second moment estimates of the perparameter gradients. This triples the required memory. To avoid keeping a first-moment estimator, we set β1 = 0. To reduce the size of the second moment estimator, we replace it with a factored approximation. For a matrix of parameters, instead of maintaining a full matrix of second-moment estimators, we maintain vectors of row-wise and column-wise averages of that matrix. At each step, the matrix of estimators is taken to be the outer product of those two vectors divided by the mean of either one. This technique could similarly be applied to Adagrad (Duchi et al., 2010).\n\nModel Test Test ops/timestep #Params excluding Total TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params per GPU .1 epochs 1 epoch (millions) (billions) (observed) Kneser-Ney 5-gram 67.1 45.3 0.00001 76.0 4xLSTM-512 54.5 47.0 8.4 8.4 0.1 1.23 MoE-32 48.5 40.4 8.4 37.8 0.1 0.83 MoE-256-h 42.8 35.3 8.4 272.9 0.4 1.11 MoE-1024-h 40.3 32.7 8.5 1079.0 1.2 1.14 MoE-4096-h 38.9 30.9 8.6 4303.4 4.4 1.07 MoE-16384-h 38.2 29.7 8.8 17201.0 17.3 0.96 MoE-65536-h 38.2 28.9 9.2 68791.0 68.9 0.72 MoE-131072-h 39.8 29.2 9.7 137577.6 137.7 0.30\n\nResults: We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8. Perplexity after 100 billion training words is 39% lower for the 68-billion-parameter MoE model than for the baseline model. It is notable that the measured computational efficiency of the largest model (0.30 TFLOPS/GPU) is very low compared to the other models. This is likely a result of the fact that, for purposes of comparison to the other models, we did not increase the training batch size proportionally to the number of GPUs. For comparison, we include results for a computationally matched baseline model consisting of 4 LSTMs, and for an unpruned 5-gram model with Kneser-Ney smoothing (Kneser & Ney, 1995).4\n\nE Machine Translation - Experimental Details\n\nModel Architecture for Single Language Pair MoE Models: Our model is a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decrease the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We insert MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). We use an attention mechanism between the encoder and decoder, with the first decoder LSTM receiving output from and providing input for the attention 5. All of the layers in our model have input and output dimensionality of 512. Our LSTM layers have 2048 hidden units, with a 512-dimensional output projection. We add residual connections around all LSTM and MoE layers to encourage gradient flow (He et al., 2015). Similar to GNMT, to effectively deal with rare words, we used subword units (also known as \"wordpieces\") (Schuster & Nakajima, 2012) for inputs and outputs in our system.\n\nWe use a shared source and target vocabulary of 32K wordpieces. We also used the same beam search technique as proposed in (Wu et al., 2016).\n\nWe train models with different numbers of experts in the MoE layers. In addition to a baseline model with no MoE layers, we train models with flat MoE layers containing 32 experts, and models with hierarchical MoE layers containing 512 and 2048 experts. The flat MoE layers use k = 4 and the hierarchical MoE models use k = 2 at each level of the gating network. Thus, each input is processed by exactly 4 experts in each MoE layer. Each expert in the MoE layer is a feed forward network with one hidden layer of size 2048 and ReLU activation. Thus, each expert contains [512 ∗ 2048] + [2048 ∗ 512] = 2M parameters. The output of the MoE layer is passed through a sigmoid function. We use the strictly-balanced gating function described in Appendix F.\n\nModel Architecture for Multilingual MoE Model: We used the same model architecture as for the single-language-pair models, with the following exceptions: We used noisy-top-k gating as described in Section 2.1, not the scheme from Appendix F. The MoE layers in the encoder and decoder are non-hierarchical MoEs with n = 512 experts, and k = 2. Each expert has a larger hidden layer of size 8192. This doubles the amount of computation in the MoE layers, raising the computational budget of the entire model from 85M to 102M ops/timestep. Training: We trained our networks using the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 2000 training steps, held constant for an additional 8000 steps, and decreased after that so as to be proportional to the inverse square root of the step number. For the single-language-pair models, similarly to (Wu et al., 2016), we applied dropout (Zaremba et al., 2014) to the output of all embedding, LSTM and MoE layers, using DropP rob = 0.4. Training was done synchronously on a cluster of up to 64 GPUs as described in section 3. Each training batch consisted of a set of sentence pairs containing roughly 16000 words per GPU.\n\nTo ensure balanced expert utilization we set wimportance = 0.01 and wload = 0.01, as described in Section 4 and Appendix A. Metrics: We evaluated our models using the perplexity and the standard BLEU score metric. We reported tokenized BLEU score as computed by the multi-bleu.pl script, downloaded from the public implementation of Moses (on Github), which was also used in (Luong et al., 2015a). Results: Tables 2, 3 and 4 in Section 5.3 show comparisons of our results to other published methods. Figure 4 shows test perplexity as a function of number of words in the (training data's) source sentences processed for models with different numbers of experts. As can be seen from the Figure, as we increased the number of experts to approach 2048, the test perplexity of our model continued to improve.\n\nWe found that the experts indeed become highly specialized by syntax and/or semantics, as can be seen in Table 9. For example, one expert is used when the indefinite article \"a\" introduces the direct object in a verb phrase indicating importance or leadership.\n\nExpert 381 Expert 752 Expert 2004 ... with researchers , ... ... plays a core ... ... with rapidly growing ... ... to innovation . ... plays a critical ... ... under static conditions ... ... tics researchers . ... provides a legislative ... ... to swift ly ... ... the generation of ... ... play a leading ... ... to dras tically ... ... technology innovations is ... ... assume a leadership ... ... the rapid and ... ... technological innovations , ... ... plays a central ... ... the fast est ... ... support innovation throughout ... ... taken a leading ... ... the Quick Method ... ... role innovation will ... ... established a reconciliation ... ... rec urrent ) ... ... research scienti st ... ... played a vital ... ... provides quick access ... ... promoting innovation where ... ... have a central ... ... of volatile organic ... ... ... ...\n\nDue to some peculiarities in our infrastructure which have since been fixed, at the time we ran some of the machine translation experiments, our models ran faster if every expert received exactly the same batch size. To accommodate this, we used a different gating function which we describe below. Recall that we define the softmax gating function to be:\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ Gσ(x) = Sof tmax(x · Wg) (15) Sparse Gating (alternate formulation): To obtain a sparse gating vector, we multiply Gσ(x) component-wise with a sparse mask M(Gσ(x)) and normalize the output. The mask itself is a function of Gσ(x) and specifies which experts are assigned to each input example:\n\n$$(15)$$ $$G(x){i}=\\frac{G{\\sigma}(x){i}M(G{\\sigma}(x)){i}}{\\sum{j=1}^{n}G_{\\sigma}(x){j}M(G{\\sigma}(x))_{j}}\\tag{1}$$ $$(17)$$\n\n$$(16)$$\n\nTop-K Mask: To implement top-k gating in this formulation, we would let M(v) = T opK(v, k), where:\n\n$$T o p K(v,k){i}=\\begin{cases}1&{\\mathrm{if~}}v{i}{\\mathrm{~is~in~the~top~}}k{\\mathrm{~elements~of~}}v.\\ 0&{\\mathrm{otherwise.}}\\end{cases}$$ 0 otherwise. (17) Batchwise Mask: To force each expert to receive the exact same number of examples, we introduce an alternative mask function, Mbatchwise(X, m), which operates over batches of input vectors.\n\nInstead of keeping the top k values per example, we keep the top m values per expert across the training batch, where m = k|X| n, so that each example is sent to an average of k experts.\n\n$$M_{b a t c h w i s e}(X,m){j,i}=\\begin{cases}1&\\text{if}X{j,i}\\text{is in the top}m\\text{values for to expert}i\\ 0&\\text{otherwise}\\end{cases}$$\n\nAs our experiments suggest and also observed in (Ioffe & Szegedy, 2015), using a batchwise function during training (such as Mbatchwise) requires modifications to the inference when we may not have a large batch of examples. Our solution to this is to train a vector T of per-expert threshold values to approximate the effects of the batchwise mask. We use the following mask at inference time:\n\n$$(18)$$ $$M_{threshold}(x,T){i}=\\begin{cases}1&\\text{if}x{i}>T_{i}\\ 0&\\text{otherwise}\\end{cases}\\tag{1}$$ $$(19)$$\n\nTo learn the threshold values, we apply an additional loss at training time which is minimized when the batchwise mask and the threshold mask are identical.\n\n$$L_{batchwise}(X,T,m)=\\sum_{j=1}^{|X|}\\sum_{i=1}^{n}(M_{threshold}(x,T){i}-M{batchwise}(X,m){j,i})(X{j,i}-T_{i})\\tag{20}$$\n\nG ATTENTION FUNCTION The attention mechanism described in GNMT (Wu et al., 2016) involves a learned \"Attention Function\" A(xi, yj ) which takes a \"source vector\" xi and a \"target vector\" yj , and must be computed for every source time step i and target time step j. In GNMT, the attention function is implemented as a feed forward neural network with a hidden layer of size n. It can be expressed as:\n\n$$A_{G N M T}(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}t a n h((x_{i}U){d}+(y{j}W)_{d})$$\n\nWhere U and W are trainable weight matrices and V is a trainable weight vector.\n\nFor performance reasons, in our models, we used a slightly different attention function:\n\n$$A(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}tanh((x_{i}U){d})tanh((y{j}W)_{d})\\tag{22}$$\n\nWith our attention function, we can simultaneously compute the attention function on multiple source time steps and multiple target time steps using optimized matrix multiplications. We found little difference in quality between the two functions.\n\n$$(21)$$",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/1701.06538v1.md"
+          },
+          "headlines": {
+            "1 Introduction And Related Work": [
+              "1.1 Conditional Computation",
+              "1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER",
+              "1.3 Related Work On Mixtures Of Experts"
+            ],
+            "2 The Structure Of The Mixture-Of-Experts Layer": [
+              "2.1 Gating Network"
+            ],
+            "3 Addressing Performance Challenges": [
+              "3.1 The Shrinking Batch Problem",
+              "3.2 Network Bandwidth"
+            ],
+            "4 Balancing Expert Utilization": [],
+            "5 Experiments": [
+              "5.1 1 Billion Word Language Modeling Benchmark",
+              "5.2 100 BILLION WORD GOOGLE NEWS CORPUS",
+              "5.3 Machine Translation (Single Language Pair)",
+              "5.4 Multilingual Machine Translation"
+            ],
+            "6 Conclusion": [],
+            "Appendices": [
+              "A Load-Balancing Loss",
+              "B Hierachical Mixture Of Experts",
+              "C 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS",
+              "D 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS",
+              "E Machine Translation - Experimental Details"
+            ]
+          },
+          "summary": "The paper introduces a Sparsely-Gated Mixture-of-Experts (MoE) layer to enhance neural network capacity without a proportional increase in computation. This approach uses a trainable gating network to select a sparse combination of experts for each input, significantly improving model capacity and performance in language modeling and machine translation tasks. The MoE layer, with up to 137 billion parameters, achieves better results than state-of-the-art models at lower computational costs. The paper addresses challenges like batch size and network bandwidth, proposing solutions like mixing data and model parallelism. Experiments show that MoE models outperform traditional models in various benchmarks, demonstrating the potential of conditional computation in deep learning.",
+          "summary_embedding": [
+            -0.01644345559179783,
+            -0.016995344310998917,
+            -0.016372699290513992,
+            -0.014349107630550861,
+            0.02282555401325226,
+            0.045934125781059265,
+            -0.014646278694272041,
+            0.03563220426440239,
+            -0.004829026758670807,
+            0.005617944523692131,
+            -0.00932550523430109,
+            -0.008143897168338299,
+            0.005575491581112146,
+            0.01538213063031435,
+            -0.005653322208672762,
+            -0.020306676626205444,
+            -0.013832597061991692,
+            -0.023235931992530823,
+            0.028344441205263138,
+            -0.020080260932445526,
+            0.011823155917227268,
+            -0.020943470299243927,
+            -0.025868017226457596,
+            -0.016500059515237808,
+            0.025061409920454025,
+            0.024806693196296692,
+            0.02583971433341503,
+            0.003827843815088272,
+            -0.03492465242743492,
+            0.033651065081357956,
+            0.014674580655992031,
+            0.012488252483308315,
+            -0.031500112265348434,
+            -0.009162768721580505,
+            -0.05351905897259712,
+            -0.05227377265691757,
+            0.04143410921096802,
+            0.022245362401008606,
+            -0.016231190413236618,
+            0.023717066273093224,
+            0.017264213413000107,
+            0.005816058721393347,
+            -0.020632149651646614,
+            -0.020844414830207825,
+            0.003700484987348318,
+            -0.013103820383548737,
+            -0.01983969286084175,
+            -0.013528349809348583,
+            -0.01828308403491974,
+            -0.01577835902571678,
+            0.009551920928061008,
+            0.0002856731880456209,
+            0.0029186424799263477,
+            -0.003424540627747774,
+            -0.047830358147621155,
+            0.0248208437114954,
+            0.019853845238685608,
+            0.03492465242743492,
+            0.04633035510778427,
+            0.013011839240789413,
+            0.01516986545175314,
+            -0.014271277002990246,
+            -0.04021712392568588,
+            0.005497660953551531,
+            -0.00041656987741589546,
+            -0.005380915477871895,
+            -0.005819596350193024,
+            0.008575502783060074,
+            0.008724087849259377,
+            -0.01143400277942419,
+            -0.04202845320105553,
+            0.013952880166471004,
+            -0.020561393350362778,
+            0.00010143388499272987,
+            0.01354957651346922,
+            0.008292482234537601,
+            0.02677368000149727,
+            0.03195294365286827,
+            0.023971782997250557,
+            -0.008724087849259377,
+            -0.01340806670486927,
+            0.019924599677324295,
+            -0.0022676968947052956,
+            0.007018893025815487,
+            0.04570771008729935,
+            -0.04033033177256584,
+            0.02163686975836754,
+            -0.025726506486535072,
+            -0.015339677222073078,
+            -0.005349075887352228,
+            -0.022471778094768524,
+            0.02806142158806324,
+            -0.007344366051256657,
+            0.017801949754357338,
+            0.017094401642680168,
+            0.01594817079603672,
+            0.013330236077308655,
+            -0.001894464367069304,
+            -0.022938761860132217,
+            0.0018360915128141642,
+            -0.01541043259203434,
+            0.02114158496260643,
+            0.0015230007702484727,
+            0.005469358991831541,
+            0.027594437822699547,
+            0.031160488724708557,
+            -0.023490650579333305,
+            0.009502392262220383,
+            0.02422650158405304,
+            0.06967949122190475,
+            0.015424583107233047,
+            -0.006435164716094732,
+            -0.0024640418123453856,
+            0.04143410921096802,
+            0.0015513027319684625,
+            0.0038349193055182695,
+            -0.06826439499855042,
+            -0.03022652305662632,
+            -0.0011365016689524055,
+            -0.02254253253340721,
+            0.02227366529405117,
+            0.01002597901970148,
+            -0.009750034660100937,
+            -0.022457627579569817,
+            -0.04516997188329697,
+            0.05190584436058998,
+            -0.00545874610543251,
+            -0.016259491443634033,
+            -0.005554265342652798,
+            -0.05340585112571716,
+            0.00039136342820711434,
+            0.015042506158351898,
+            -0.018481198698282242,
+            -0.03670767694711685,
+            0.036141637712717056,
+            0.0007398316520266235,
+            0.03741522878408432,
+            0.012459950521588326,
+            -0.01708024926483631,
+            -0.020094411447644234,
+            0.022259512916207314,
+            -0.011639192700386047,
+            -0.0038313816767185926,
+            -0.027339721098542213,
+            0.0065236082300543785,
+            -0.028103873133659363,
+            -0.0014566679019480944,
+            0.06730213016271591,
+            -0.04816998168826103,
+            0.02946236915886402,
+            -0.029660483822226524,
+            0.023660462349653244,
+            -0.008575502783060074,
+            0.0278916098177433,
+            -0.010238244198262691,
+            0.005840822588652372,
+            -0.013457595370709896,
+            -0.0010586712742224336,
+            -0.020745357498526573,
+            0.0034210027661174536,
+            -0.016259491443634033,
+            -0.027750099077820778,
+            -0.007054270710796118,
+            0.031273696571588516,
+            0.01698119193315506,
+            0.025655752047896385,
+            0.025429336354136467,
+            0.01246702577918768,
+            -0.04935866594314575,
+            0.02595292218029499,
+            0.0018926954362541437,
+            0.013514199294149876,
+            0.013181651011109352,
+            -0.0010418669553473592,
+            0.025599148124456406,
+            0.012049571610987186,
+            0.025061409920454025,
+            0.037443529814481735,
+            0.0027682881336659193,
+            0.04737752676010132,
+            -0.015693452209234238,
+            -0.005292471963912249,
+            0.028528403490781784,
+            0.000505455827806145,
+            0.02465103194117546,
+            -0.001686621573753655,
+            0.013330236077308655,
+            0.000776978034991771,
+            0.03834919258952141,
+            -0.006339645478874445,
+            0.029518973082304,
+            0.021608568727970123,
+            0.028316138312220573,
+            -0.014759486541152,
+            0.023122724145650864,
+            -0.019754787907004356,
+            -0.02292460948228836,
+            0.035547297447919846,
+            0.004082561936229467,
+            0.01886327564716339,
+            -0.028457649052143097,
+            -0.030679354444146156,
+            0.015452885068953037,
+            -0.01855195313692093,
+            0.016556663438677788,
+            0.03639635816216469,
+            0.0008273909334093332,
+            -0.0025719432160258293,
+            0.026292547583580017,
+            -0.04245298355817795,
+            0.012856177985668182,
+            -0.01532552670687437,
+            -0.02855670638382435,
+            -0.019528372213244438,
+            -0.00875946506857872,
+            -0.0018095583654940128,
+            -0.013691086322069168,
+            -0.0037570889107882977,
+            -0.007563706487417221,
+            -0.02459442801773548,
+            0.00935380719602108,
+            0.013627407141029835,
+            0.009849091991782188,
+            -0.018665160983800888,
+            0.0019952901639044285,
+            -0.008257105015218258,
+            -0.0160047747194767,
+            -0.03752843663096428,
+            0.015268922783434391,
+            -0.0024888061452656984,
+            0.009191070683300495,
+            0.04078316316008568,
+            0.028485950082540512,
+            -0.005055442452430725,
+            0.0329718142747879,
+            -0.033707667142152786,
+            0.017915157601237297,
+            -0.008320784196257591,
+            0.009191070683300495,
+            0.002989397384226322,
+            -0.025542544201016426,
+            0.027424626052379608,
+            -0.03792466223239899,
+            -0.038151077926158905,
+            0.022995365783572197,
+            0.011858533136546612,
+            0.0017096169758588076,
+            -0.019386861473321915,
+            -0.017717044800519943,
+            0.0008499440737068653,
+            0.002752368338406086,
+            -0.018834972754120827,
+            -0.01019579079002142,
+            0.02227366529405117,
+            -0.025500090792775154,
+            0.036085035651922226,
+            0.023391593247652054,
+            -0.03520767390727997,
+            -0.00608492735773325,
+            -0.01903308741748333,
+            0.059037946164608,
+            -0.0011506526498124003,
+            -0.010301923379302025,
+            0.022938761860132217,
+            -0.027268964797258377,
+            -0.004524780437350273,
+            0.022301966324448586,
+            0.01828308403491974,
+            -0.008278331719338894,
+            0.003983504604548216,
+            -0.022287815809249878,
+            -0.05281151086091995,
+            -0.010577867738902569,
+            -0.0003305583668407053,
+            0.054368119686841965,
+            -0.012367968447506428,
+            0.0020695829298347235,
+            -0.006413938011974096,
+            0.004100250545889139,
+            0.03704730048775673,
+            0.002140337834134698,
+            0.0025082635693252087,
+            -0.013698162510991096,
+            0.0371888130903244,
+            0.022316116839647293,
+            0.005883275531232357,
+            -0.012162779457867146,
+            0.005377377849072218,
+            -0.04899074137210846,
+            -0.0009118546731770039,
+            0.005741765722632408,
+            0.007386818993836641,
+            -0.025910470634698868,
+            0.017717044800519943,
+            0.009205221198499203,
+            -0.0024693484883755445,
+            -0.009792488068342209,
+            0.01173117384314537,
+            0.025429336354136467,
+            -0.019938750192523003,
+            -0.05700020492076874,
+            0.0034705311991274357,
+            0.012014193460345268,
+            -0.004054259974509478,
+            -0.019401011988520622,
+            0.006792477332055569,
+            0.0021934041287750006,
+            0.022698193788528442,
+            -0.011271266266703606,
+            -0.0029221803415566683,
+            0.040585052222013474,
+            0.009516543708741665,
+            -0.0016813150141388178,
+            -0.017717044800519943,
+            -0.011794853024184704,
+            0.018070818856358528,
+            0.004740582779049873,
+            0.0035289041697978973,
+            -0.01751893013715744,
+            -0.029264256358146667,
+            0.02800481766462326,
+            0.019967053085565567,
+            -0.048905834555625916,
+            -0.008200501091778278,
+            -0.019273653626441956,
+            0.0019599127117544413,
+            0.008695785887539387,
+            0.04613224044442177,
+            -0.00590803986415267,
+            0.008313708938658237,
+            -0.020179318264126778,
+            -0.05003791302442551,
+            -0.02708500251173973,
+            0.03707560524344444,
+            -0.028684064745903015,
+            -0.04276430234313011,
+            -0.010252394713461399,
+            0.016740625724196434,
+            -0.009403335861861706,
+            -0.018679311498999596,
+            -0.011738249100744724,
+            -0.04969828948378563,
+            -0.003497064346447587,
+            0.0337359681725502,
+            0.017264213413000107,
+            -0.008441068232059479,
+            -0.033877480775117874,
+            0.019641580060124397,
+            -0.007160402834415436,
+            -0.006474079564213753,
+            0.0007862646016292274,
+            0.008794843219220638,
+            0.02400008589029312,
+            0.011879759840667248,
+            -0.02763689123094082,
+            -0.03427370637655258,
+            -0.0024888061452656984,
+            0.014929298311471939,
+            -0.00878776703029871,
+            0.0245378240942955,
+            0.024679332971572876,
+            -0.022995365783572197,
+            -0.01654251106083393,
+            0.015750056132674217,
+            -0.05779265984892845,
+            -0.010287772864103317,
+            -0.029235953465104103,
+            -0.006183984223753214,
+            0.0009613831643946469,
+            0.0061096916906535625,
+            0.00012304731353651732,
+            0.03846240043640137,
+            0.008801918476819992,
+            0.006014172453433275,
+            0.009191070683300495,
+            0.01913214474916458,
+            0.028471799567341805,
+            0.009155693463981152,
+            -0.03517937287688255,
+            0.02850010246038437,
+            -0.017051948234438896,
+            0.007811348885297775,
+            -0.025754809379577637,
+            -0.013033065013587475,
+            0.00443633645772934,
+            0.02725481428205967,
+            -0.000503244751598686,
+            0.018849123269319534,
+            -0.021919889375567436,
+            0.037500135600566864,
+            -0.029915200546383858,
+            -0.05278320610523224,
+            -0.02005195803940296,
+            -0.008285406976938248,
+            0.015764206647872925,
+            0.017278363928198814,
+            -0.019924599677324295,
+            -0.0286557637155056,
+            -0.021750077605247498,
+            0.006470541935414076,
+            0.010542490519583225,
+            -0.021099131554365158,
+            -0.010627396404743195,
+            0.01502835564315319,
+            -0.00404364662244916,
+            -0.01886327564716339,
+            0.013684011064469814,
+            0.015693452209234238,
+            0.03169822692871094,
+            -0.03308502212166786,
+            0.004287751391530037,
+            -0.025967074558138847,
+            -0.04327373951673508,
+            0.004096712917089462,
+            0.03166992589831352,
+            0.027509532868862152,
+            -0.018636859953403473,
+            0.010754755698144436,
+            0.003990580327808857,
+            0.004234685096889734,
+            -0.0033803186379373074,
+            0.023603858426213264,
+            0.03291521221399307,
+            0.02763689123094082,
+            -0.011653343215584755,
+            0.004921008367091417,
+            -0.01559439580887556,
+            -0.0011108530452474952,
+            0.01865101046860218,
+            -0.017816102132201195,
+            0.033877480775117874,
+            -0.029405765235424042,
+            0.02104252763092518,
+            -0.035122767090797424,
+            0.006141531281173229,
+            -0.012099100276827812,
+            -0.00404364662244916,
+            0.04833979532122612,
+            -0.017646290361881256,
+            -0.0019546060357242823,
+            -0.008101443760097027,
+            -0.014575524255633354,
+            -0.02158026583492756,
+            0.008766540326178074,
+            -0.04033033177256584,
+            -0.012367968447506428,
+            -0.028429346159100533,
+            0.003424540627747774,
+            0.019910449162125587,
+            -0.06277380883693695,
+            -0.06237758323550224,
+            0.012275987304747105,
+            -0.04918885603547096,
+            -0.026419905945658684,
+            0.02963218092918396,
+            -0.002984090941026807,
+            0.0012780115939676762,
+            -0.0024003623984754086,
+            0.0068101659417152405,
+            -0.029207652434706688,
+            0.00425944896414876,
+            -0.000887974863871932,
+            0.03271709755063057,
+            -0.021056678146123886,
+            -0.008122670464217663,
+            0.004878554958850145,
+            -0.06860402226448059,
+            -0.028344441205263138,
+            0.011271266266703606,
+            -0.009459939785301685,
+            -0.0066615804098546505,
+            -0.017915157601237297,
+            -0.004383270163089037,
+            0.0008950503543019295,
+            0.0032830305863171816,
+            0.05801907554268837,
+            -0.01983969286084175,
+            -0.053773775696754456,
+            0.03633975237607956,
+            0.00887267291545868,
+            0.029603879898786545,
+            -0.025556694716215134,
+            -0.017533080652356148,
+            -0.01855195313692093,
+            -0.020504789426922798,
+            -0.04123599827289581,
+            0.00404364662244916,
+            -0.005918653216212988,
+            0.009212297387421131,
+            -0.0009693430620245636,
+            0.02763689123094082,
+            0.029235953465104103,
+            0.023108573630452156,
+            0.012750045396387577,
+            0.01079013291746378,
+            -0.022372720763087273,
+            -0.021594416350126266,
+            0.0012222920777276158,
+            0.04683978855609894,
+            0.024636881425976753,
+            -0.02675952948629856,
+            -0.011504758149385452,
+            0.03953787684440613,
+            0.0419718474149704,
+            -0.01974063739180565,
+            -0.02276895008981228,
+            0.00764861237257719,
+            0.03266049548983574,
+            -0.04477374628186226,
+            0.005932804197072983,
+            0.03410389646887779,
+            0.014405711553990841,
+            -0.005356151144951582,
+            -0.010294848121702671,
+            0.022570835426449776,
+            0.03152841329574585,
+            -0.008724087849259377,
+            0.01876421831548214,
+            0.016910437494516373,
+            0.029349161311984062,
+            -0.0067889392375946045,
+            0.0005766529939137399,
+            -0.024735936895012856,
+            0.055188875645399094,
+            -0.03059444949030876,
+            0.02909444272518158,
+            0.010316074825823307,
+            -0.00593634182587266,
+            0.039056744426488876,
+            -0.016641568392515182,
+            -0.0431322306394577,
+            0.01913214474916458,
+            -0.006979977712035179,
+            -0.0055613406002521515,
+            -0.03444352000951767,
+            -0.007592008449137211,
+            -0.007507102098315954,
+            -0.006714646704494953,
+            -0.022556684911251068,
+            -0.02606613002717495,
+            0.004415110219269991,
+            0.00973588414490223,
+            -0.006484692916274071,
+            0.00698705343529582,
+            0.02092931978404522,
+            -0.013507124036550522,
+            0.02525952458381653,
+            -0.046047333627939224,
+            -0.0008366775000467896,
+            -0.0038596836384385824,
+            -0.025712355971336365,
+            -0.0345001220703125,
+            -0.0005770951975136995,
+            -0.0024251264985650778,
+            -0.024622729048132896,
+            -0.00914154201745987,
+            -0.008752389810979366,
+            -0.0023119186516851187,
+            -0.004938696976751089,
+            -0.055188875645399094,
+            0.008695785887539387,
+            0.010846736840903759,
+            -0.002713453024625778,
+            0.0055507272481918335,
+            -0.010613245889544487,
+            -0.03039633482694626,
+            0.03039633482694626,
+            0.04129260033369064,
+            0.033707667142152786,
+            0.014334957115352154,
+            -0.0074646491557359695,
+            -0.017165156081318855,
+            0.006060163024812937,
+            0.021721776574850082,
+            0.00902833417057991,
+            0.007846726104617119,
+            0.03877372294664383,
+            0.00011331850691931322,
+            -0.008349086157977581,
+            -0.009764186106622219,
+            0.028740668669342995,
+            -0.001085204305127263,
+            0.023603858426213264,
+            0.050802066922187805,
+            0.030424637719988823,
+            0.020589696243405342,
+            -0.01681138016283512,
+            -0.02010856196284294,
+            -0.00851889792829752,
+            -0.02855670638382435,
+            0.0364246591925621,
+            0.012127402238547802,
+            -0.005232329946011305,
+            -0.011540135368704796,
+            0.019061388447880745,
+            -0.033339742571115494,
+            -0.0037358624394983053,
+            0.00048422932741232216,
+            0.013924578204751015,
+            -0.03379257395863533,
+            0.0002480846014805138,
+            0.031613320112228394,
+            0.014957600273191929,
+            -0.016995344310998917,
+            -0.0188774261623621,
+            0.010549565777182579,
+            0.020462337881326675,
+            0.018948180601000786,
+            0.00037743354914709926,
+            0.00897173024713993,
+            -0.00022121983056422323,
+            -0.0005319889169186354,
+            0.01590571738779545,
+            -0.007952858693897724,
+            0.036254845559597015,
+            -0.003525366308167577,
+            -0.01325240544974804,
+            -0.0037500134203583,
+            -0.021438755095005035,
+            -0.024467067793011665,
+            0.043556760996580124,
+            0.023971782997250557,
+            0.04661337286233902,
+            -0.005660397466272116,
+            -0.004245297983288765,
+            0.025825563818216324,
+            0.008221727795898914,
+            -0.03288691118359566,
+            0.000265110022155568,
+            0.005281858611851931,
+            -0.02892463095486164,
+            0.016641568392515182,
+            -0.02476423978805542,
+            -0.016896286979317665,
+            -0.006771250627934933,
+            -0.0015557248843833804,
+            -0.028188779950141907,
+            0.01703779585659504,
+            0.0031309074256569147,
+            -0.0027099153958261013,
+            -0.039764292538166046,
+            0.006880920846015215,
+            0.020419884473085403,
+            0.015311375260353088,
+            0.0037358624394983053,
+            0.011525984853506088,
+            -0.016613267362117767,
+            0.002126186853274703,
+            0.03379257395863533,
+            0.004878554958850145,
+            0.02005195803940296,
+            0.0027647505048662424,
+            0.023773670196533203,
+            0.03633975237607956,
+            -0.01681138016283512,
+            -0.02189158834517002,
+            0.01519816741347313,
+            -0.02233026921749115,
+            0.029943503439426422,
+            0.004889168310910463,
+            -0.012283062562346458,
+            0.022033097222447395,
+            -0.009827865287661552,
+            -0.0027346794959157705,
+            0.01659911684691906,
+            -0.022740647196769714,
+            -0.017717044800519943,
+            0.03183973580598831,
+            -0.039452970027923584,
+            0.012856177985668182,
+            -0.008547200821340084,
+            0.015863263979554176,
+            -0.023066120222210884,
+            0.011172209866344929,
+            0.007100261282175779,
+            -0.01573590561747551,
+            0.010797208175063133,
+            0.009693430736660957,
+            0.00545874610543251,
+            0.0018926954362541437,
+            0.02179253101348877,
+            -0.0006049550138413906,
+            -0.011257115751504898,
+            -0.018155725672841072,
+            -0.004694592207670212,
+            -0.002614396158605814,
+            -0.016556663438677788,
+            0.01865101046860218,
+            -0.041066184639930725,
+            -0.013280707411468029,
+            -0.008879749104380608,
+            -0.013988257385790348,
+            -0.010011828504502773,
+            0.024184048175811768,
+            0.025669902563095093,
+            -0.002361447084695101,
+            -0.031217092648148537,
+            0.01790100708603859,
+            0.024198200553655624,
+            0.003654494183138013,
+            -0.023646311834454536,
+            0.0302831269800663,
+            -0.008936353027820587,
+            0.014971751719713211,
+            0.026136886328458786,
+            0.014094389975070953,
+            0.004595535341650248,
+            -0.010500037111341953,
+            -0.02114158496260643,
+            0.01481609046459198,
+            0.010507113300263882,
+            0.033537857234478,
+            0.014745336025953293,
+            0.02071705460548401,
+            0.01594817079603672,
+            -0.023320838809013367,
+            -0.011214662343263626,
+            -0.003482913365587592,
+            -0.022514231503009796,
+            0.03093407303094864,
+            0.0006129149114713073,
+            0.02114158496260643,
+            0.026618018746376038,
+            -0.03319823369383812,
+            -0.01187268365174532,
+            0.032688796520233154,
+            0.02779255248606205,
+            0.05108508840203285,
+            0.025316128507256508,
+            0.015396281145513058,
+            0.027226513251662254,
+            -0.03585861995816231,
+            -0.036368053406476974,
+            0.001548649393953383,
+            0.04256619140505791,
+            -0.008426916785538197,
+            0.00967928022146225,
+            0.03900013864040375,
+            -0.006781863979995251,
+            -0.01611798256635666,
+            0.019457615911960602,
+            0.0026710000820457935,
+            0.002448122017085552,
+            -0.0033113325480371714,
+            -0.01335146278142929,
+            0.03175482898950577,
+            -0.0051474240608513355,
+            0.013570803217589855,
+            -0.019273653626441956,
+            0.027113303542137146,
+            0.0001799829478841275,
+            -0.003229964291676879,
+            0.028344441205263138,
+            -0.002660386962816119,
+            -0.051990751177072525,
+            0.024212351068854332,
+            0.019811391830444336,
+            -0.011561362072825432,
+            -0.0036332677118480206,
+            -0.003792466362938285,
+            0.04567940905690193,
+            0.034698236733675,
+            0.029009537771344185,
+            -0.0020501252729445696,
+            0.0168679840862751,
+            0.018792519345879555,
+            0.0053667644970119,
+            0.014278353191912174,
+            -0.0017025414854288101,
+            -0.007853801362216473,
+            0.021056678146123886,
+            -0.020787810906767845,
+            0.03226426616311073,
+            -0.016570813953876495,
+            -0.020122714340686798,
+            0.008738238364458084,
+            -0.011172209866344929,
+            -0.0009083169279620051,
+            -0.006311343517154455,
+            -0.0008813415770418942,
+            -0.0029664020985364914,
+            0.016938740387558937,
+            -0.004446949809789658,
+            0.04661337286233902,
+            -0.013931653462350368,
+            -0.008808993734419346,
+            0.006972902454435825,
+            -0.025981225073337555,
+            0.0013346156338229775,
+            0.0028726516757160425,
+            0.027000095695257187,
+            -0.01989629678428173,
+            -0.00970758218318224,
+            0.01606137864291668,
+            0.06039644405245781,
+            -0.008639181964099407,
+            -0.0026267783250659704,
+            0.027226513251662254,
+            -0.002787745790556073,
+            -0.03551899641752243,
+            0.012919857166707516,
+            -0.04494355618953705,
+            0.017589686438441277,
+            0.026349151507019997,
+            -0.04986810311675072,
+            -0.022089701145887375,
+            -0.02347649820148945,
+            -0.02043403498828411,
+            0.023434046655893326,
+            -0.009976450353860855,
+            0.013867974281311035,
+            -0.017801949754357338,
+            -0.009014182724058628,
+            0.020518941804766655,
+            0.009176919236779213,
+            0.03735862299799919,
+            -0.027551984414458275,
+            -0.016783079132437706,
+            -0.027820853516459465,
+            -0.04304732382297516,
+            0.009558996185660362,
+            -0.0278916098177433,
+            0.0027099153958261013,
+            -0.0016291331266984344,
+            -0.040302030742168427,
+            -0.037981268018484116,
+            0.005278320983052254,
+            -0.017985913902521133,
+            0.01516986545175314,
+            0.020915169268846512,
+            0.02731141820549965,
+            -0.0020253609400242567,
+            -0.030198222026228905,
+            0.029235953465104103,
+            -0.007344366051256657,
+            0.03223596513271332,
+            -0.00689507182687521,
+            0.005066055804491043,
+            -0.0070436573587358,
+            0.031160488724708557,
+            -0.023009516298770905,
+            0.018792519345879555,
+            -0.026801982894539833,
+            -0.004086099565029144,
+            -0.018990634009242058,
+            -0.01774534583091736,
+            0.004224071744829416,
+            0.004521242808550596,
+            7.888073741924018e-05,
+            0.0048750173300504684,
+            0.005777143407613039,
+            0.024806693196296692,
+            0.0021421066485345364,
+            0.02589631825685501,
+            -0.027877457439899445,
+            -0.025316128507256508,
+            0.01719345711171627,
+            -0.02077365852892399,
+            0.005678086541593075,
+            -0.006261814851313829,
+            -0.0302831269800663,
+            -0.01202834490686655,
+            0.023391593247652054,
+            0.02162271924316883,
+            -0.005490585695952177,
+            0.002644466934725642,
+            0.02206140011548996,
+            -0.026023678481578827,
+            0.00437973253428936,
+            -0.03781145438551903,
+            -0.018396291881799698,
+            0.029773691669106483,
+            -0.03195294365286827,
+            0.020745357498526573,
+            0.003185742534697056,
+            0.03483974561095238,
+            -0.017292514443397522,
+            -0.01947176828980446,
+            0.01654251106083393,
+            -0.01008258294314146,
+            0.004142703488469124,
+            0.01606137864291668,
+            0.01616043597459793,
+            -0.0015070808585733175,
+            0.05380208045244217,
+            0.0008167776977643371,
+            0.006909222807735205,
+            -0.049075648188591,
+            0.004496478475630283,
+            -0.044858649373054504,
+            0.0002768288250081241,
+            -0.013337312266230583,
+            -0.02514631673693657,
+            0.0014027173165231943,
+            0.001001182827167213,
+            0.001963450340554118,
+            -0.01913214474916458,
+            -0.01989629678428173,
+            0.011596739292144775,
+            -0.015863263979554176,
+            0.004928083624690771,
+            -0.06418890506029129,
+            -0.01855195313692093,
+            -0.004974074196070433,
+            0.0043337419629096985,
+            0.0007699025445617735,
+            0.03342464938759804,
+            -0.01478778850287199,
+            0.010117961093783379,
+            -0.01549533847719431,
+            -0.026363302022218704,
+            0.04010391607880592,
+            0.018750067800283432,
+            -0.023773670196533203,
+            0.024693485349416733,
+            0.012106175534427166,
+            0.03099067695438862,
+            -0.0318114347755909,
+            0.020349130034446716,
+            0.018523650243878365,
+            -0.0029823218937963247,
+            0.006551910191774368,
+            0.008575502783060074,
+            0.01832553744316101,
+            -0.015764206647872925,
+            -0.0003632825391832739,
+            0.011419852264225483,
+            0.010033054277300835,
+            0.031924642622470856,
+            0.01403071079403162,
+            -0.03183973580598831,
+            0.019047237932682037,
+            0.01616043597459793,
+            0.02298121340572834,
+            0.033820874989032745,
+            -0.027481229975819588,
+            0.038575608283281326,
+            -0.010386829264461994,
+            -0.024042539298534393,
+            0.020801961421966553,
+            -0.032688796520233154,
+            -0.006679269019514322,
+            0.030735958367586136,
+            0.04398128762841225,
+            -0.0015566092915832996,
+            -0.044632233679294586,
+            0.01535382866859436,
+            0.01316749956458807,
+            0.0027806703001260757,
+            0.028811423107981682,
+            0.011264191009104252,
+            0.021820832043886185,
+            0.025273675099015236,
+            0.011419852264225483,
+            -0.0037252490874379873,
+            0.024962352588772774,
+            -0.008101443760097027,
+            -0.004446949809789658,
+            -0.0007035697344690561,
+            0.03206615149974823,
+            0.014702882617712021,
+            -0.017603836953639984,
+            -0.036962397396564484,
+            -0.007620310410857201,
+            0.009452863596379757,
+            -0.007903330028057098,
+            -0.027127455919981003,
+            -0.03950957581400871,
+            -0.026787832379341125,
+            -0.005667473189532757,
+            0.01505665760487318,
+            -0.06724552065134048,
+            -0.007620310410857201,
+            -0.012233533896505833,
+            0.017717044800519943,
+            0.0022906921803951263,
+            0.0181132722645998,
+            0.001277127186767757,
+            -0.009403335861861706,
+            0.004209920763969421,
+            0.050802066922187805,
+            -0.03059444949030876,
+            0.014242975041270256,
+            0.011292492970824242,
+            0.0032105068676173687,
+            0.002787745790556073,
+            0.038688816130161285,
+            -0.010988246649503708,
+            -0.030424637719988823,
+            0.011575513519346714,
+            -0.017292514443397522,
+            0.003383856499567628,
+            0.05006621778011322,
+            -0.009481165558099747,
+            -0.027495380491018295,
+            0.020915169268846512,
+            -0.009148617275059223,
+            -0.015523640438914299,
+            -0.0027258351910859346,
+            -0.006035399157553911,
+            0.0006283925613388419,
+            -0.03832089155912399,
+            -0.015750056132674217,
+            0.03212275728583336,
+            0.008617955259978771,
+            0.014603826217353344,
+            0.025344429537653923,
+            0.010209942236542702,
+            -0.030481241643428802,
+            0.027481229975819588,
+            -0.0005678086308762431,
+            0.005221716593950987,
+            0.013323160819709301,
+            0.03424540534615517,
+            -0.014264201745390892,
+            -0.020264223217964172,
+            -0.040245428681373596,
+            -0.013853822834789753,
+            -0.0463869571685791,
+            0.001943992800079286,
+            0.006919836159795523,
+            -0.012919857166707516,
+            -0.04366996884346008,
+            -0.0329718142747879,
+            0.021127434447407722,
+            0.03178313374519348,
+            -0.017023645341396332,
+            0.0025294902734458447,
+            -0.008483520708978176,
+            -0.007882103323936462,
+            -0.0017520699184387922,
+            -0.013217028230428696,
+            -0.006378560326993465,
+            -0.006229975260794163,
+            0.015452885068953037,
+            -0.004652139265090227,
+            -0.0036934094969183207,
+            -0.0498398020863533,
+            -0.018778368830680847,
+            0.004082561936229467,
+            -0.008221727795898914,
+            0.007323139347136021,
+            -0.006955213379114866,
+            -0.002727604005485773,
+            -0.012474101036787033,
+            -0.03829259052872658,
+            0.024184048175811768,
+            0.017278363928198814,
+            0.0035501306410878897,
+            -0.008922201581299305,
+            -0.018367990851402283,
+            0.03294351324439049,
+            0.012523629702627659,
+            -0.017547233030200005,
+            -0.018580254167318344,
+            0.025981225073337555,
+            0.026957644149661064,
+            0.008122670464217663,
+            -0.006137993652373552,
+            0.029886899515986443,
+            0.005695775151252747,
+            -0.022938761860132217,
+            -0.003983504604548216,
+            -0.005055442452430725,
+            -0.0038844477385282516,
+            0.020561393350362778,
+            -0.010556641966104507,
+            0.0300001073628664,
+            0.038717120885849,
+            0.012870329432189465,
+            0.0039198254235088825,
+            -0.0033714743331074715,
+            -0.008922201581299305,
+            0.015976471826434135,
+            0.006481155287474394,
+            0.008525974117219448,
+            0.014759486541152,
+            -0.016358548775315285,
+            -0.020405733957886696,
+            0.006215824279934168,
+            -0.007107336539775133,
+            0.020122714340686798,
+            0.01708024926483631,
+            0.016429303213953972,
+            0.0007610581233166158,
+            -0.028287837281823158,
+            -0.0018024828750640154,
+            0.02411329373717308,
+            0.02010856196284294,
+            -0.004655676893889904,
+            -0.023264234885573387,
+            0.014398636296391487,
+            -0.015806660056114197,
+            -0.009969375096261501,
+            0.0033431723713874817,
+            0.006229975260794163,
+            -0.003083147807046771,
+            0.0571134127676487,
+            -0.020193468779325485,
+            -0.011419852264225483,
+            -0.010287772864103317,
+            -0.010867963545024395,
+            0.0012072566896677017,
+            -0.01828308403491974,
+            0.008964654989540577,
+            0.025613298639655113,
+            -0.009764186106622219,
+            0.03568880632519722,
+            0.0228114016354084,
+            0.006912760436534882,
+            0.005260631907731295,
+            -0.027070851996541023,
+            0.012304289266467094,
+            -0.016047226265072823,
+            0.005660397466272116,
+            0.018467046320438385,
+            0.017320817336440086,
+            0.0076132346875965595,
+            0.026306698098778725,
+            0.019386861473321915,
+            0.016684021800756454,
+            -0.022783100605010986,
+            0.025457637384533882,
+            -0.007832575589418411,
+            0.027877457439899445,
+            -0.01611798256635666,
+            -0.002253545681014657,
+            -0.012990612536668777,
+            0.008681634441018105,
+            0.002416282193735242,
+            -0.0019351483788341284,
+            0.00036129256477579474,
+            0.016783079132437706,
+            -0.014589674770832062,
+            0.01062032114714384,
+            0.013316085562109947,
+            -0.027919910848140717,
+            -0.010372678749263287,
+            0.016344398260116577,
+            -0.00668988237157464,
+            0.006877383217215538,
+            -0.011865608394145966,
+            -0.0017202302115038037,
+            0.011009473353624344,
+            -0.0436982698738575,
+            0.0104929618537426,
+            0.006279503460973501,
+            -0.015155714936554432,
+            0.014801939949393272,
+            -0.009799563325941563,
+            -0.008235878311097622,
+            -0.013592029921710491,
+            0.003792466362938285,
+            -0.00030225637601688504,
+            -0.0018290160223841667,
+            -0.00022995365725364536,
+            0.00389859895221889,
+            0.010047205723822117,
+            -0.0045990729704499245,
+            -0.010450509376823902,
+            -0.0028638073708862066,
+            -0.02265574224293232,
+            -0.0010047205723822117,
+            0.018735915422439575,
+            -0.004591997712850571,
+            0.0005222601466812193,
+            0.0145189194008708,
+            -0.005830209702253342,
+            0.010556641966104507,
+            -0.005221716593950987,
+            -0.008320784196257591,
+            -0.010768906213343143,
+            -0.033764272928237915,
+            0.0026710000820457935,
+            -0.015254771336913109,
+            0.008865597657859325,
+            -0.012544856406748295,
+            0.04027372971177101,
+            0.01022409275174141,
+            -0.00837738811969757,
+            -0.002796590095385909,
+            -0.0228114016354084,
+            -0.004475251771509647,
+            -0.016938740387558937,
+            -0.01184438169002533,
+            0.008342010900378227,
+            0.012629762291908264,
+            0.005614406894892454,
+            -0.018410442396998405,
+            0.003665107535198331,
+            0.0030955299735069275,
+            -0.005575491581112146,
+            -0.008370312862098217,
+            0.010853812098503113,
+            -0.0006248548161238432,
+            -0.01800006441771984,
+            -0.01957082375884056,
+            -0.027424626052379608,
+            0.0150000536814332,
+            0.014589674770832062,
+            -0.01011088490486145,
+            -0.022754797711968422,
+            -0.007471724878996611,
+            0.007107336539775133,
+            -0.02064630016684532,
+            0.0006779210525564849,
+            0.006891534198075533,
+            0.008717012591660023,
+            0.004117939155548811,
+            0.009636826813220978,
+            0.0007699025445617735,
+            0.0121698547154665,
+            0.018566103652119637,
+            0.03271709755063057,
+            -0.0396793857216835,
+            -0.009537769481539726,
+            0.0057028504088521,
+            -0.001465512323193252,
+            -0.013853822834789753,
+            0.009672204032540321,
+            0.0024375086650252342,
+            -0.0026426981203258038,
+            0.024424616247415543,
+            -0.008044839836657047,
+            -0.01505665760487318,
+            0.01184438169002533,
+            0.00887267291545868,
+            -0.019075538963079453,
+            0.02746707946062088,
+            0.006169833242893219,
+            -0.01011088490486145,
+            0.02259913645684719,
+            -0.012360893189907074,
+            0.012658064253628254,
+            -0.005522425286471844,
+            0.023660462349653244,
+            -0.019004784524440765,
+            0.0006306036957539618,
+            0.015665151178836823,
+            0.03883032873272896,
+            -0.00437973253428936,
+            -0.004567233379930258,
+            -0.03421710431575775,
+            0.005115584470331669,
+            0.028853876516222954,
+            0.02666047215461731,
+            -0.0012346742441877723,
+            -0.005596718285232782,
+            -0.0019864458590745926,
+            0.010768906213343143,
+            0.011589664034545422,
+            0.017561383545398712,
+            -0.029547275975346565,
+            0.00022011429246049374,
+            -0.016938740387558937,
+            0.025500090792775154,
+            0.014632128179073334,
+            0.006951675750315189,
+            0.04072656109929085,
+            0.0027789014857262373,
+            0.003841994795948267,
+            0.027452928945422173,
+            -0.00779719790443778,
+            0.003546592779457569,
+            -0.015566092915832996,
+            0.0003175129240844399,
+            0.022853855043649673,
+            0.0016370931407436728,
+            0.0020784272346645594,
+            0.013337312266230583,
+            0.029320860281586647,
+            -0.022471778094768524,
+            -0.023929331451654434,
+            0.003482913365587592,
+            0.007185167167335749,
+            -0.010570792481303215,
+            0.0038738346192985773,
+            0.006275965832173824,
+            -0.006007096730172634,
+            0.008900974877178669,
+            0.006332569755613804,
+            -0.0003166284877806902,
+            0.0039198254235088825,
+            -0.00308845448307693,
+            0.012962310574948788,
+            -0.02702839858829975,
+            0.036849189549684525,
+            0.004439874552190304,
+            0.010075507685542107,
+            0.015268922783434391,
+            0.001627364312298596,
+            0.004917470272630453,
+            0.005869124550372362,
+            0.010188715532422066,
+            0.018636859953403473,
+            -0.00956607237458229,
+            -0.027481229975819588,
+            -0.03166992589831352,
+            0.020363280549645424,
+            -0.01173117384314537,
+            0.0035872769076377153,
+            0.0074646491557359695,
+            -0.01828308403491974,
+            -0.007507102098315954,
+            -0.009976450353860855,
+            0.015764206647872925,
+            -0.03189634159207344,
+            0.012990612536668777,
+            -0.0043231286108493805,
+            -0.028669914230704308,
+            0.00851889792829752,
+            -0.010181640274822712,
+            -0.0045884596183896065,
+            -0.0035094465129077435,
+            -0.029264256358146667,
+            -0.007931631989777088,
+            0.03908504545688629,
+            -0.02141045406460762,
+            0.024184048175811768,
+            0.008257105015218258,
+            -0.00014449491573031992,
+            -0.0040365708991885185,
+            -0.013563727959990501,
+            0.005331386812031269,
+            -0.01778779923915863,
+            0.025457637384533882,
+            0.010294848121702671,
+            -0.013924578204751015,
+            0.0322076641023159,
+            -0.011179285123944283,
+            0.027438776567578316,
+            -0.018580254167318344,
+            0.036198243498802185,
+            0.007422196213155985,
+            0.00674648629501462,
+            0.01974063739180565,
+            -0.004025958012789488,
+            0.008228803053498268,
+            0.04559450224041939,
+            -0.018523650243878365,
+            0.02460857853293419,
+            0.020349130034446716,
+            -0.00022685811563860625,
+            0.017915157601237297,
+            0.0028390432707965374,
+            -0.023165177553892136,
+            0.009495317004621029,
+            -0.01135617308318615,
+            -0.009955224581062794,
+            0.01583496294915676,
+            -0.019316107034683228,
+            -0.018721764907240868,
+            -0.025613298639655113,
+            0.0029416377656161785,
+            -0.003350247861817479,
+            -0.0077618202194571495,
+            -0.0019616815261542797,
+            0.01454722136259079,
+            0.008724087849259377,
+            0.025669902563095093,
+            0.0008326975512318313,
+            0.011971740983426571,
+            0.01511326152831316,
+            -0.00035576484515331686,
+            0.02313687466084957,
+            -0.03229256719350815,
+            0.01246702577918768,
+            -0.004517704714089632,
+            -0.00509789539501071,
+            0.023858575150370598,
+            0.0003747802402358502,
+            -0.0004877670726273209,
+            -0.0024888061452656984,
+            0.0005496777012012899,
+            -0.014589674770832062,
+            -0.0057099261321127415,
+            0.000907432462554425,
+            -0.005175726022571325,
+            0.04299071803689003,
+            -0.00701535539701581,
+            -0.004772422835230827,
+            -0.039169952273368835,
+            -0.01918874867260456,
+            0.00623351288959384,
+            -0.01583496294915676,
+            -0.018778368830680847,
+            -0.006307805422693491,
+            0.005801907740533352,
+            -0.04893413558602333,
+            0.008342010900378227,
+            0.03336804360151291,
+            -0.012219383381307125,
+            0.00259493850171566,
+            -0.010981171391904354,
+            -0.006969364359974861,
+            -0.0029115669894963503,
+            -7.766463386360556e-05,
+            -0.002412744564935565,
+            -0.020830262452363968,
+            -0.005617944523692131,
+            0.003364398842677474,
+            -0.0008795727044343948,
+            -0.025047259405255318,
+            0.0006177793256938457,
+            0.018849123269319534,
+            -0.003979966975748539,
+            -0.0011152751976624131,
+            -0.001825478277169168,
+            0.006399787031114101,
+            -0.01508495956659317,
+            0.008009462617337704,
+            -0.0076344613917171955,
+            0.003841994795948267,
+            -0.0030495391692966223,
+            -0.002214630600064993,
+            0.02844349853694439,
+            0.021594416350126266,
+            -0.04808507859706879,
+            -0.01405901275575161,
+            0.023009516298770905,
+            0.007280686404556036,
+            -0.004740582779049873,
+            0.01178777776658535,
+            -0.031613320112228394,
+            -0.00028943203506059945,
+            -0.026787832379341125,
+            0.02200479619204998,
+            -0.009205221198499203,
+            -0.007386818993836641,
+            0.016967041417956352,
+            -0.026632171124219894,
+            0.016344398260116577,
+            0.00794578343629837,
+            -0.00030181417241692543,
+            -0.000493515923153609,
+            0.0063856360502541065,
+            0.0008972614887170494,
+            -0.01659911684691906,
+            0.006442239973694086,
+            -0.0048502529971301556,
+            0.007959933951497078,
+            -0.005087282508611679,
+            -0.05527378246188164,
+            -0.009933997876942158,
+            -0.005497660953551531,
+            -0.008936353027820587,
+            -0.012820800766348839,
+            0.008136821910738945,
+            0.02827368676662445,
+            0.01606137864291668,
+            -0.009863242506980896,
+            -4.8312376748071983e-05,
+            0.033000119030475616,
+            -0.0008300442132167518,
+            0.009955224581062794,
+            -0.011349096894264221,
+            0.00015709814033471048,
+            -0.0006911876262165606,
+            -0.014660430140793324,
+            -0.019061388447880745,
+            0.024042539298534393,
+            -0.028344441205263138,
+            0.02378782071173191,
+            -0.01843874529004097,
+            -0.0130825936794281,
+            0.01757553406059742,
+            -0.032575588673353195,
+            -0.005108508747071028,
+            0.007450498174875975,
+            0.0090566361322999,
+            0.021608568727970123,
+            0.0032794929575175047,
+            0.020589696243405342,
+            -0.03716050833463669,
+            -0.009134466759860516,
+            -0.010160413570702076,
+            -0.00794578343629837,
+            -0.010231168940663338,
+            -0.011596739292144775,
+            0.021863285452127457,
+            -0.015240620821714401,
+            -0.016952890902757645,
+            0.007054270710796118,
+            0.0030212372075766325,
+            -0.011313719674944878,
+            0.015750056132674217,
+            0.00017810352437663823,
+            0.004712280817329884,
+            0.013011839240789413,
+            -0.0008923970744945109,
+            0.04086806997656822,
+            0.0253019779920578,
+            -0.0030212372075766325,
+            -0.008568426594138145,
+            0.021438755095005035,
+            0.0004285097820684314,
+            0.0150000536814332,
+            0.009375033900141716,
+            0.023264234885573387,
+            -0.02092931978404522,
+            0.009997677057981491,
+            -0.005501199048012495,
+            0.014603826217353344,
+            -0.009495317004621029,
+            -0.015339677222073078,
+            0.00479718716815114,
+            -0.01577835902571678,
+            0.010153338313102722,
+            -0.010860888287425041,
+            0.008221727795898914,
+            0.01913214474916458,
+            0.002467579673975706,
+            -0.00941041111946106,
+            0.01146230474114418,
+            0.02075950801372528,
+            -0.025641601532697678,
+            0.008851447142660618,
+            -0.0031998935155570507,
+            -0.0021916350815445185,
+            0.021127434447407722,
+            0.010301923379302025,
+            -0.01594817079603672,
+            0.01573590561747551,
+            -0.008879749104380608,
+            -0.0015053119277581573,
+            -0.01882082223892212,
+            -0.03291521221399307,
+            -0.0015283073298633099,
+            0.012396270409226418,
+            0.004800724796950817,
+            0.0150000536814332,
+            0.011264191009104252,
+            -0.0008950503543019295,
+            -0.006601438857614994,
+            0.0008433107868768275,
+            -0.013485897332429886,
+            -0.004963461309671402,
+            0.010231168940663338,
+            0.017051948234438896,
+            -0.0037217114586383104,
+            0.015650998800992966,
+            0.0088160689920187,
+            0.006137993652373552,
+            -0.0010480580385774374,
+            0.0036138100549578667,
+            -0.002060738392174244,
+            -0.007783046457916498,
+            0.019938750192523003,
+            0.019075538963079453,
+            -0.018509499728679657,
+            -0.0024870370980352163,
+            -0.02141045406460762,
+            0.025882167741656303,
+            0.022188758477568626,
+            0.011165134608745575,
+            -0.04222656786441803,
+            0.00012050455552525818,
+            0.00752832880243659,
+            -0.0013744152383878827,
+            -0.031103884801268578,
+            0.008172199130058289,
+            0.005221716593950987,
+            0.004330204334110022,
+            0.020844414830207825,
+            -0.018424594774842262,
+            0.010853812098503113,
+            0.0070861103013157845,
+            -0.0005766529939137399,
+            0.01019579079002142,
+            -0.0073372903279960155,
+            0.008688710629940033,
+            0.038717120885849,
+            -0.018410442396998405,
+            0.005303084850311279,
+            0.0062370505183935165,
+            0.023320838809013367,
+            -0.011228813789784908,
+            -0.00031485961517319083,
+            0.00800238735973835,
+            0.021608568727970123,
+            -0.012934008613228798,
+            0.005883275531232357,
+            0.067075714468956,
+            -0.012912781909108162,
+            0.031613320112228394,
+            0.020533092319965363,
+            -0.004970536567270756,
+            -0.0104929618537426,
+            -0.012445799075067043,
+            -0.0181132722645998,
+            0.021283095702528954,
+            0.01175947580486536,
+            0.003419233951717615,
+            0.01941516436636448,
+            -0.01610383205115795,
+            0.006636816076934338,
+            0.017547233030200005,
+            0.012601460330188274,
+            -0.007379743270576,
+            -0.0036173479165881872,
+            0.019004784524440765,
+            0.018509499728679657,
+            -0.0030849166214466095,
+            0.00019424450874794275,
+            0.01508495956659317,
+            -0.022075550630688667,
+            -0.012530704960227013,
+            0.01644345559179783,
+            -0.0037358624394983053,
+            0.017773648723959923,
+            -0.02227366529405117,
+            0.024198200553655624,
+            -0.004531855694949627,
+            -0.017179306596517563,
+            0.004698129836469889,
+            -0.007514177821576595,
+            0.022033097222447395,
+            -0.001578720286488533,
+            0.004857328720390797,
+            0.000821199850179255,
+            -0.0018015984678640962,
+            0.01349297259002924,
+            0.002826661104336381,
+            -0.0005120891146361828,
+            0.010507113300263882,
+            -0.0009702275274321437,
+            -0.0013496510218828917,
+            0.024509521201252937,
+            1.9374700059415773e-05,
+            0.005989408120512962,
+            0.0032936439383774996,
+            -0.009261825121939182,
+            -0.004429261200129986,
+            -0.004443412180989981,
+            0.0011842612875625491,
+            0.003877372248098254,
+            -0.007029506377875805,
+            -0.02384442463517189,
+            0.022471778094768524,
+            -0.008540124632418156,
+            0.01175947580486536,
+            0.011349096894264221,
+            0.003665107535198331,
+            0.006824316922575235,
+            -0.009771261364221573,
+            -0.017320817336440086,
+            0.0006633278680965304,
+            -0.01617458648979664,
+            0.006912760436534882,
+            -0.001656550681218505,
+            0.0020235921256244183,
+            0.009445788338780403,
+            -0.002570174168795347,
+            0.0015123875346034765,
+            0.011306644417345524,
+            0.011497682891786098,
+            0.003254728624597192,
+            0.004577846731990576,
+            -0.018721764907240868,
+            -0.020915169268846512,
+            0.005288933869451284,
+            0.013924578204751015,
+            0.004273600410670042,
+            0.0015309606678783894,
+            0.017603836953639984,
+            -0.006166295614093542,
+            -0.022967062890529633,
+            -0.010117961093783379,
+            -0.01584911346435547,
+            0.039396367967128754,
+            0.01800006441771984,
+            -0.02525952458381653,
+            -0.000530220044311136,
+            -0.009728807955980301,
+            0.010450509376823902,
+            -0.00404364662244916,
+            0.014575524255633354,
+            0.005476434715092182,
+            -0.02426895499229431,
+            -0.016797229647636414,
+            -0.010662773624062538,
+            -0.005257094278931618,
+            0.020080260932445526,
+            0.027014248073101044,
+            -0.004956385586410761,
+            0.0061026159673929214,
+            0.019584976136684418,
+            -0.010938717983663082,
+            0.009849091991782188,
+            0.020957622677087784,
+            0.0031698227394372225,
+            -0.007570781745016575,
+            -0.004553082399070263,
+            -0.016514210030436516,
+            0.024467067793011665,
+            0.014250051230192184,
+            -0.005104971118271351,
+            -0.02465103194117546,
+            0.004110863897949457,
+            -0.02428310550749302,
+            -0.01616043597459793,
+            -0.007301913108676672,
+            -0.014143918640911579,
+            -0.004931621253490448,
+            0.0027594438288360834,
+            -0.004836102016270161,
+            0.011278342455625534,
+            -0.0031592093873769045,
+            -0.004531855694949627,
+            -0.0041851564310491085,
+            -0.025584997609257698,
+            -0.02524537406861782,
+            0.007691065315157175,
+            -0.003005317412316799,
+            0.005596718285232782,
+            0.000650945701636374,
+            -0.013400991447269917,
+            -0.011589664034545422,
+            0.00970758218318224,
+            -0.007988235913217068,
+            0.012813725508749485,
+            0.0031149876303970814,
+            -0.009183995425701141,
+            0.0017193458043038845,
+            -0.0012311364989727736,
+            -0.0059752571396529675,
+            0.019924599677324295,
+            0.00916984397917986,
+            0.011724098585546017,
+            -0.03118879161775112,
+            -0.0021633331198245287,
+            0.008349086157977581,
+            0.01828308403491974,
+            -0.02244347706437111,
+            -0.008313708938658237,
+            0.005179263651371002,
+            -0.011228813789784908,
+            -0.013507124036550522,
+            -0.028952933847904205,
+            -0.0030460015404969454,
+            8.175515540642664e-05,
+            -0.018636859953403473,
+            -0.029886899515986443,
+            0.0009038947173394263,
+            -0.007860877551138401,
+            0.016882136464118958,
+            -0.006643891800194979,
+            0.01790100708603859,
+            0.00870286114513874,
+            0.0036173479165881872,
+            0.011554286815226078,
+            0.0025082635693252087,
+            -0.013266556896269321,
+            0.008426916785538197,
+            -0.02671707607805729,
+            -0.01516986545175314,
+            0.009792488068342209,
+            0.014214673079550266,
+            -0.0026126273442059755,
+            5.074458022136241e-05,
+            0.0058372849598526955,
+            0.024254804477095604,
+            0.013584954664111137,
+            0.012474101036787033,
+            0.003042463678866625,
+            -0.005303084850311279,
+            0.005568416323512793,
+            0.0227123461663723,
+            0.008603804744780064,
+            0.004011806566268206,
+            0.007592008449137211,
+            0.00389859895221889,
+            0.0009932229295372963,
+            -0.005943417549133301,
+            -0.00843399204313755,
+            -0.021537812426686287,
+            -0.008908051066100597,
+            0.019938750192523003,
+            -0.010422207415103912,
+            -0.03005671128630638,
+            0.0006023016758263111,
+            -0.0032777239102870226,
+            0.007988235913217068,
+            0.02595292218029499,
+            -0.014172220602631569,
+            0.004970536567270756,
+            0.005536576267331839,
+            -0.007910405285656452,
+            -0.00614860700443387,
+            0.022967062890529633,
+            0.02639160305261612,
+            -0.0034404604230076075,
+            -0.04800017178058624,
+            -0.01671232469379902,
+            0.004305440001189709,
+            -0.007768895477056503,
+            -0.034047290682792664,
+            -0.015424583107233047,
+            0.002757675014436245,
+            -0.03206615149974823,
+            0.010068432427942753,
+            -0.012948159128427505,
+            -0.01214155275374651,
+            0.020264223217964172,
+            0.03223596513271332,
+            0.005154499784111977,
+            0.00014847487909719348,
+            -0.0214670579880476,
+            -0.0021686397958546877,
+            -0.021382151171565056,
+            0.010174565017223358,
+            -0.008801918476819992,
+            -0.010584943927824497,
+            0.0059611061587929726,
+            -0.012191081419587135,
+            0.013266556896269321,
+            0.010797208175063133,
+            0.024778390303254128,
+            0.01584911346435547,
+            -0.01876421831548214,
+            -0.0024746551644057035,
+            -0.01108730398118496,
+            -0.0016450530383735895,
+            -0.009820790030062199,
+            0.006360871717333794,
+            -0.0031592093873769045,
+            0.011264191009104252,
+            0.015184016898274422,
+            0.004432798828929663,
+            0.00046786724124103785,
+            0.038151077926158905,
+            -2.4487851987942122e-05,
+            -0.0003418349369894713,
+            0.006803090218454599,
+            -0.015750056132674217,
+            0.008837295696139336,
+            0.012191081419587135,
+            0.005051904823631048,
+            -0.007698140572756529,
+            0.006739411037415266,
+            0.03099067695438862,
+            -0.01516986545175314,
+            -0.009014182724058628,
+            -0.023504801094532013,
+            0.007549555506557226,
+            -0.0004656561359297484,
+            0.029971804469823837,
+            0.020533092319965363,
+            -0.027509532868862152,
+            -0.016018925234675407,
+            -0.02179253101348877,
+            0.01267929095774889,
+            -0.011497682891786098,
+            -0.003704022616147995,
+            0.012240610085427761,
+            0.02367461286485195,
+            0.0018838511314243078,
+            -0.014455240219831467,
+            -0.008950503543019295,
+            0.008943428285419941,
+            0.014617976732552052,
+            0.003497064346447587,
+            0.007570781745016575,
+            0.00743634719401598,
+            0.008886824361979961,
+            -0.014801939949393272,
+            -0.012254760600626469,
+            0.00767691433429718,
+            -0.010202866978943348,
+            0.01794346049427986,
+            -0.009403335861861706,
+            0.016768928617239,
+            -0.004645063541829586,
+            -0.011964665725827217,
+            -0.0320095494389534,
+            -0.002292460994794965,
+            0.005037753842771053,
+            0.004627374932169914,
+            0.0034705311991274357,
+            -0.0022217060904949903,
+            -0.013160424306988716,
+            0.004673365969210863,
+            0.005073131527751684,
+            1.875283123808913e-05,
+            -0.008313708938658237,
+            -0.00479718716815114,
+            0.004252373706549406,
+            -0.001031253719702363,
+            -0.0073372903279960155,
+            -0.0059858704917132854,
+            0.018481198698282242,
+            0.0066509670577943325,
+            0.00800238735973835,
+            0.03234917297959328,
+            0.02974538877606392,
+            -0.0016353242099285126,
+            -0.004606148693710566,
+            -0.00023304918431676924,
+            -0.01589156687259674,
+            -0.0057205394841730595,
+            -0.023320838809013367,
+            0.022259512916207314,
+            -0.009665128774940968,
+            0.021820832043886185,
+            0.019457615911960602,
+            0.0039304387755692005,
+            -0.008391539566218853,
+            -0.0011586126638576388,
+            -0.007308988366276026,
+            -0.003560743760317564,
+            0.016641568392515182,
+            0.006742948666214943,
+            0.01226891204714775,
+            0.00030933189555071294,
+            -0.028401045128703117,
+            0.026405755430459976,
+            0.000579748535528779,
+            0.0034015451092272997,
+            0.008908051066100597,
+            0.014129767194390297,
+            0.013209952972829342,
+            -0.01235381793230772,
+            0.01768874190747738,
+            -0.03280200436711311,
+            -0.0033484790474176407,
+            -0.016500059515237808,
+            0.002076658420264721,
+            -0.0018520113080739975,
+            -0.0008145665633492172,
+            -0.05354736000299454,
+            0.0003842879377771169,
+            0.005313698202371597,
+            -0.015339677222073078,
+            0.029349161311984062,
+            -0.020589696243405342,
+            -0.017165156081318855,
+            -0.032858606427907944,
+            -0.010238244198262691,
+            0.018099121749401093,
+            0.005617944523692131,
+            0.00035178486723452806,
+            -0.02660386823117733,
+            0.015750056132674217,
+            -0.0047476585023105145,
+            -0.00743634719401598,
+            -0.029858596622943878,
+            -0.010938717983663082,
+            0.009403335861861706,
+            0.004485865123569965,
+            -0.0130825936794281,
+            0.009898620657622814,
+            -0.002890340518206358,
+            0.010867963545024395,
+            -0.0138467475771904,
+            -0.00420284504070878,
+            0.0010020672343671322,
+            -0.005363226868212223,
+            0.012615610845386982,
+            -0.030537845566868782,
+            0.011391550302505493,
+            -0.01352127455174923,
+            0.0008548084879294038,
+            -0.017858553677797318,
+            -0.027707645669579506,
+            0.0020288988016545773,
+            -0.016132133081555367,
+            0.039990708231925964,
+            0.010188715532422066,
+            -0.01481609046459198,
+            0.016995344310998917,
+            -0.002080196049064398,
+            0.017377421259880066,
+            0.0014955832157284021,
+            0.017023645341396332,
+            -0.011971740983426571,
+            0.021056678146123886,
+            -0.0041745430789887905,
+            0.02156611531972885,
+            0.00600002147257328,
+            0.034047290682792664,
+            -0.02033497765660286,
+            0.0070825726725161076,
+            0.003792466362938285,
+            0.005798369646072388,
+            -0.0026816134341061115,
+            0.021424604579806328,
+            0.017985913902521133,
+            0.019698183983564377,
+            0.0037110981065779924,
+            -0.024084990844130516,
+            -0.006137993652373552,
+            0.004867942072451115,
+            0.023858575150370598,
+            0.011150983162224293,
+            -0.016415152698755264,
+            -0.021056678146123886,
+            0.015226469375193119,
+            -0.007860877551138401,
+            0.006771250627934933,
+            -0.00851889792829752,
+            0.00037544354563578963,
+            -0.013259481638669968,
+            0.012233533896505833,
+            -0.012523629702627659,
+            0.0032335021533071995,
+            0.01043635793030262,
+            -0.027806703001260757,
+            0.01511326152831316,
+            -0.00371817359700799,
+            -0.014313730411231518,
+            0.023320838809013367,
+            -0.015877414494752884,
+            -0.013988257385790348,
+            0.005501199048012495,
+            0.008610880002379417,
+            0.01746232621371746,
+            -0.002761212643235922,
+            0.002669231267645955,
+            -0.020504789426922798,
+            -0.012445799075067043,
+            0.004591997712850571,
+            0.02210385352373123,
+            -0.015820810571312904,
+            0.012431648559868336,
+            -0.018099121749401093,
+            0.01364863384515047,
+            0.016202887520194054,
+            0.007149789482355118,
+            0.00573115237057209,
+            -0.007726442534476519,
+            -0.0011471149045974016,
+            0.032151058316230774,
+            -0.0029876285698264837,
+            0.02704254910349846,
+            -0.014915147796273232,
+            0.009615600109100342,
+            0.00518987700343132,
+            -0.013089669868350029,
+            -0.014589674770832062,
+            0.011108530685305595,
+            0.02428310550749302,
+            -0.009530694223940372,
+            -0.023660462349653244,
+            0.0018484735628589988,
+            0.014264201745390892,
+            0.010337301529943943,
+            0.009403335861861706,
+            -0.001997058978304267,
+            -0.0037358624394983053,
+            0.01484439242631197,
+            -0.0032918748911470175,
+            -0.0286557637155056,
+            -0.01633024774491787,
+            -0.004400959238409996,
+            -0.017490629106760025,
+            0.020745357498526573,
+            0.02367461286485195,
+            0.010507113300263882,
+            0.0037570889107882977,
+            0.005540114361792803,
+            0.018297234550118446,
+            0.01843874529004097,
+            0.013330236077308655,
+            0.009304278530180454,
+            0.012700516730546951,
+            -0.004319590982049704,
+            0.02800481766462326,
+            -0.010860888287425041,
+            -0.01046465989202261,
+            -0.015707602724432945,
+            0.0016636261716485023,
+            -0.005193414632230997,
+            -0.012431648559868336,
+            -0.0069481381215155125,
+            0.004369119182229042,
+            0.013330236077308655,
+            -0.012240610085427761,
+            -0.007149789482355118,
+            0.01016748882830143,
+            0.011971740983426571,
+            0.01403071079403162,
+            0.01719345711171627,
+            -0.01633024774491787,
+            -0.011278342455625534,
+            0.0009834941010922194,
+            -0.02844349853694439,
+            -0.0160047747194767,
+            -0.01767459139227867,
+            0.009580222889780998,
+            -0.014469391666352749,
+            0.03724541515111923,
+            0.028471799567341805,
+            0.0073514413088560104,
+            0.031613320112228394,
+            -0.028401045128703117,
+            0.00970758218318224,
+            -0.014398636296391487,
+            0.007132100872695446,
+            -0.004188694059848785,
+            0.003902136581018567,
+            0.002448122017085552,
+            0.020419884473085403,
+            0.00791748147457838,
+            0.014264201745390892,
+            0.008150972425937653,
+            -0.021934039890766144,
+            -0.00662620272487402,
+            -0.012559006921947002,
+            -0.016401002183556557,
+            -0.02595292218029499,
+            0.013719388283789158,
+            0.005087282508611679,
+            -0.003074303502216935,
+            -0.0002704166399780661,
+            0.006566061172634363,
+            -0.0013195802457630634,
+            0.010443433187901974,
+            -0.027608588337898254,
+            -0.01311089564114809,
+            0.0069269114173948765,
+            0.01449061743915081,
+            -0.02114158496260643,
+            0.0014637433923780918,
+            0.008900974877178669,
+            -0.004602610599249601,
+            0.035660505294799805,
+            -0.014674580655992031,
+            -0.008610880002379417,
+            0.0027806703001260757,
+            0.005062518175691366,
+            -0.017660440877079964,
+            -0.02319347858428955,
+            0.0070648835971951485,
+            0.011575513519346714,
+            0.010867963545024395,
+            -0.028570856899023056,
+            0.02087271586060524,
+            -0.0054339817725121975,
+            0.03832089155912399,
+            0.00401534466072917,
+            -0.013337312266230583,
+            0.003962278366088867,
+            0.0011860302183777094,
+            -0.012360893189907074,
+            0.002060738392174244,
+            -0.000340508297085762,
+            0.003700484987348318,
+            0.011228813789784908,
+            -0.014801939949393272,
+            -0.016967041417956352,
+            0.016910437494516373,
+            -0.011476456187665462,
+            -0.0035395172890275717,
+            -0.00943871308118105,
+            -0.016669871285557747,
+            0.007308988366276026,
+            0.025047259405255318,
+            0.0041745430789887905,
+            0.001242634141817689,
+            -0.011632117442786694,
+            -0.0038596836384385824,
+            -0.008575502783060074,
+            0.0011718792375177145,
+            -0.0207029040902853,
+            0.027396325021982193,
+            -0.01801421493291855,
+            0.002410975517705083,
+            0.021099131554365158,
+            0.008709936402738094,
+            -0.003944589756429195,
+            0.016316095367074013,
+            0.026363302022218704,
+            0.01205664686858654,
+            0.0031539027113467455,
+            -0.00902833417057991,
+            1.648646139074117e-05,
+            0.014957600273191929,
+            0.0015123875346034765,
+            -0.0015663381200283766,
+            0.0032140444964170456,
+            -0.006583749782294035,
+            -0.001859086798503995,
+            -0.017561383545398712,
+            0.005564878229051828,
+            0.02125479280948639,
+            -0.004132090136408806,
+            0.020476488396525383,
+            0.015509488992393017,
+            0.011625041253864765,
+            0.010726453736424446,
+            -0.0012390963966026902,
+            -0.002150951186195016,
+            -0.0021668709814548492,
+            -0.007393894251435995,
+            -0.018297234550118446,
+            0.006470541935414076,
+            0.0034493047278374434,
+            0.01822648011147976,
+            0.020504789426922798,
+            -0.02308027073740959,
+            0.010146263055503368,
+            0.014985902234911919,
+            -0.013011839240789413,
+            0.0235755555331707,
+            -0.02006610855460167,
+            0.0024605041835457087,
+            0.02156611531972885,
+            0.013337312266230583,
+            -0.014271277002990246,
+            -0.009219372645020485,
+            0.004662752617150545,
+            -0.01543873455375433,
+            0.000872939417604357,
+            -0.01478778850287199,
+            -0.010783057659864426,
+            0.014533070847392082,
+            -0.01490099634975195,
+            0.01229721400886774,
+            -0.006969364359974861,
+            0.0070967236533761024,
+            -0.017165156081318855,
+            0.015014204196631908,
+            0.008632106706500053,
+            0.01757553406059742,
+            0.0021279556676745415,
+            -0.02518877014517784,
+            -0.006290116813033819,
+            0.021169887855648994,
+            0.009509467519819736,
+            -0.02497650496661663,
+            0.019613277167081833,
+            0.0020094411447644234,
+            -0.019528372213244438,
+            -0.022401023656129837,
+            -0.012686366215348244,
+            -0.005055442452430725,
+            -0.017264213413000107,
+            -0.0011276573641225696,
+            -0.024792540818452835,
+            0.003972891718149185,
+            0.009106164798140526,
+            -0.011978816241025925,
+            -0.019273653626441956,
+            -0.0002315014135092497,
+            -0.01040805596858263,
+            0.006219361908733845,
+            -0.01132787112146616,
+            -0.019853845238685608,
+            0.0070436573587358,
+            0.012127402238547802,
+            -0.0027647505048662424,
+            -0.005646246485412121,
+            -0.02562745101749897,
+            -0.006919836159795523,
+            0.020179318264126778,
+            -0.003994117956608534,
+            -0.004974074196070433,
+            -0.01703779585659504,
+            0.01108730398118496,
+            0.011165134608745575,
+            0.020901018753647804,
+            0.010931642726063728,
+            0.005674548447132111,
+            0.013634482398629189,
+            -0.0029416377656161785,
+            -0.007156865205615759,
+            -9.645892532716971e-06,
+            0.004284213297069073,
+            0.0069233737885952,
+            0.0016636261716485023,
+            0.020235922187566757,
+            -0.01692458800971508,
+            0.018934030085802078,
+            -0.012290137819945812,
+            0.008794843219220638,
+            -0.01855195313692093,
+            -0.0003155229496769607,
+            -0.01067692507058382,
+            0.00015002264990471303,
+            -0.0038455326575785875,
+            -0.009544845670461655,
+            1.9595809135353193e-05,
+            0.00143986358307302,
+            0.008511822670698166,
+            -0.016783079132437706,
+            0.009615600109100342,
+            -0.010839661583304405,
+            -0.004308977629989386,
+            -0.019542522728443146,
+            0.017448175698518753,
+            0.013344387523829937,
+            0.007308988366276026,
+            -0.008278331719338894,
+            -0.006300730165094137,
+            -0.02282555401325226,
+            0.008582578040659428,
+            -0.0014637433923780918,
+            0.002214630600064993,
+            -0.028301987797021866,
+            -0.004358506295830011,
+            0.0024587351363152266,
+            0.020306676626205444,
+            0.0033661676570773125,
+            -0.010712302289903164,
+            0.008978805504739285,
+            -0.023377442732453346,
+            -0.0030159305315464735,
+            -0.023349139839410782,
+            -0.004075486212968826,
+            -0.005112046375870705,
+            0.010103809647262096,
+            0.008101443760097027,
+            0.01549533847719431,
+            -0.009898620657622814,
+            0.01478778850287199,
+            -0.004793649073690176,
+            0.012049571610987186,
+            -0.0012152165872976184,
+            0.01337976474314928,
+            -0.0003111007681582123,
+            -0.01505665760487318,
+            0.02158026583492756,
+            -0.008738238364458084,
+            0.001741456682793796,
+            -0.0052853962406516075,
+            0.0027346794959157705,
+            -0.017702894285321236,
+            -0.004553082399070263,
+            -0.014455240219831467,
+            0.01957082375884056,
+            -0.022584985941648483,
+            0.004153316840529442,
+            -0.005055442452430725,
+            -0.016075529158115387,
+            0.029434068128466606,
+            0.022896308451890945,
+            -0.0035483618266880512,
+            0.015070809051394463,
+            0.0017989451298490167,
+            0.0015061964513733983,
+            0.016202887520194054,
+            -0.015806660056114197,
+            -0.013025989755988121,
+            0.035066161304712296,
+            0.0176321379840374,
+            0.0004313841927796602,
+            -0.0023260696325451136,
+            -0.0037323245778679848,
+            0.0032193511724472046,
+            0.008023613132536411,
+            -0.010450509376823902,
+            -0.01141277700662613,
+            -0.017051948234438896,
+            0.009997677057981491,
+            -0.006155682262033224,
+            0.004446949809789658,
+            -0.00025891896802932024,
+            -7.617214578203857e-05,
+            -0.01516986545175314,
+            0.022584985941648483,
+            -0.017717044800519943,
+            -0.00602832343429327,
+            0.004687516950070858,
+            0.0207029040902853,
+            -0.02541518583893776,
+            0.01616043597459793,
+            -0.007107336539775133,
+            -0.013429293408989906,
+            0.012573158368468285,
+            -0.015127412974834442,
+            -0.0107335289940238,
+            -0.003175129182636738,
+            -0.008009462617337704,
+            -0.022754797711968422,
+            0.017929309979081154,
+            -0.013627407141029835,
+            0.015650998800992966,
+            -0.00287972716614604,
+            0.000562502013053745,
+            0.002057200763374567,
+            0.0032582662533968687,
+            0.02250008098781109,
+            0.030735958367586136,
+            0.010415131226181984,
+            -0.005465821363031864,
+            -0.021000074222683907,
+            0.012983537279069424,
+            0.0020589695777744055,
+            0.013202877715229988,
+            -0.002253545681014657,
+            -0.002135031158104539,
+            -0.016613267362117767,
+            -0.023122724145650864,
+            -0.0013010069960728288,
+            -0.01761798746883869,
+            -0.02552839368581772,
+            -0.01774534583091736,
+            -0.007047194987535477,
+            0.011349096894264221,
+            0.020405733957886696,
+            0.0009755341452546418,
+            -0.005384453106671572,
+            0.01559439580887556,
+            -0.006941062398254871,
+            0.002080196049064398,
+            -0.006594363134354353,
+            -0.002032436430454254,
+            -0.008200501091778278,
+            -0.021863285452127457,
+            0.029660483822226524,
+            -0.010103809647262096,
+            0.01141277700662613,
+            -0.005472897086292505,
+            -0.00965097825974226,
+            0.005575491581112146,
+            0.00900003220885992,
+            0.024849144741892815,
+            0.006452853325754404,
+            0.03141520544886589,
+            0.020151015371084213,
+            -0.018891576677560806,
+            -0.0006071660900488496,
+            0.022684043273329735,
+            -0.0037570889107882977,
+            -0.004889168310910463,
+            -0.0031786670442670584,
+            -0.017985913902521133,
+            -0.012007118202745914,
+            0.007839650847017765,
+            -0.013471745885908604,
+            -0.006636816076934338,
+            -0.00837738811969757,
+            0.018070818856358528,
+            -0.017773648723959923,
+            0.025698205456137657,
+            -0.006488230545073748,
+            0.005154499784111977,
+            0.031924642622470856,
+            0.014363259077072144,
+            0.019698183983564377,
+            -0.011794853024184704,
+            -0.031075581908226013,
+            -0.00683846790343523,
+            -0.007882103323936462,
+            0.009558996185660362,
+            -0.01062032114714384,
+            0.005204027984291315,
+            -0.02006610855460167,
+            -0.018948180601000786,
+            0.012863253243267536,
+            -0.017476476728916168,
+            0.013924578204751015,
+            -0.009955224581062794,
+            0.027226513251662254,
+            -0.001228483160957694,
+            0.0121698547154665,
+            -0.002614396158605814,
+            -0.017801949754357338,
+            0.01205664686858654,
+            0.004089637193828821,
+            -0.023943481966853142,
+            0.010337301529943943,
+            -0.001864393474534154,
+            -0.014037786051630974,
+            -0.018410442396998405,
+            -0.001228483160957694,
+            -0.005200490355491638,
+            -0.009658053517341614,
+            0.025457637384533882,
+            0.014101465232670307,
+            0.016231190413236618,
+            0.0035359796602278948,
+            0.0016335553955286741,
+            -0.011073152534663677,
+            0.009573147632181644,
+            -0.006803090218454599,
+            0.000612030504271388,
+            -0.0034404604230076075,
+            0.011893910355865955,
+            -0.004252373706549406,
+            -0.01543873455375433,
+            0.002324300818145275,
+            0.008561351336538792,
+            0.0143278818577528,
+            5.947839599684812e-05,
+            0.009587298147380352,
+            0.0009790719486773014,
+            -0.009764186106622219,
+            -0.00046388726332224905,
+            -0.0060884649865329266,
+            0.0069375247694551945,
+            0.02400008589029312,
+            -0.012516554445028305,
+            -0.01681138016283512,
+            0.014504768885672092,
+            0.004033033270388842,
+            -0.017589686438441277,
+            0.018367990851402283,
+            0.004103788174688816,
+            -0.022358570247888565,
+            0.0035412863362580538,
+            0.010662773624062538,
+            -0.009523618966341019,
+            -0.014469391666352749,
+            -0.0010339069413021207,
+            -0.015509488992393017,
+            0.03130199760198593,
+            0.00025073791039176285,
+            0.016910437494516373,
+            -0.01062032114714384,
+            -0.0025967073161154985,
+            -0.013309009373188019,
+            0.013103820383548737,
+            -0.015750056132674217,
+            -0.007273610681295395,
+            0.00035620704875327647,
+            -0.02298121340572834,
+            0.01741987280547619,
+            0.0013425755314528942,
+            0.00794578343629837,
+            0.003436922561377287,
+            -0.010556641966104507,
+            0.009820790030062199,
+            0.0018237093463540077,
+            0.006668656133115292,
+            -0.014915147796273232,
+            0.023618008941411972,
+            0.006413938011974096,
+            0.011235889047384262,
+            -0.01352127455174923,
+            -0.01546703651547432,
+            0.002253545681014657,
+            0.04134920611977577,
+            -0.01306136790663004,
+            0.0002593612007331103,
+            0.01562269777059555,
+            7.081024523358792e-05,
+            0.004156854469329119,
+            -0.0023755980655550957,
+            -0.0034493047278374434,
+            -0.034585028886795044,
+            0.005370302125811577,
+            -0.0006328147719614208,
+            0.008073141798377037,
+            -0.008992956951260567,
+            -0.005430444143712521,
+            -0.0040684109553694725,
+            -0.008625030517578125,
+            -0.00314859626814723,
+            -0.01478778850287199,
+            0.008363237604498863,
+            -0.014533070847392082,
+            -0.028910480439662933,
+            0.03166992589831352,
+            -0.006887996103614569,
+            -0.00533846253529191,
+            -0.021396303549408913,
+            0.007570781745016575,
+            -0.01370523776859045,
+            0.002663924591615796,
+            0.010967019945383072,
+            0.008865597657859325,
+            0.008738238364458084,
+            0.002762981690466404,
+            0.0009516543359495699,
+            -0.004255911335349083,
+            0.010273621417582035,
+            -0.00835616234689951,
+            -0.0005943417199887335,
+            -0.0014186371117830276,
+            -0.02725481428205967,
+            -0.01405901275575161,
+            -0.010592019185423851,
+            -0.007811348885297775,
+            -0.00435143057256937,
+            0.012608535587787628,
+            0.007174553815275431,
+            -0.0017485321732237935,
+            -0.022938761860132217,
+            -0.014745336025953293,
+            -0.0034103896468877792,
+            0.008349086157977581,
+            -0.01838214136660099,
+            0.01064862310886383,
+            0.0016830838285386562,
+            0.009976450353860855,
+            0.014292503707110882,
+            0.00302831269800663,
+            0.0016742395237088203,
+            0.003458149265497923,
+            0.014886845834553242,
+            0.012601460330188274,
+            -0.005801907740533352,
+            -0.021382151171565056,
+            -0.0024605041835457087,
+            -0.007068421691656113,
+            -0.0034687623847275972,
+            -0.012332591228187084,
+            0.03039633482694626,
+            0.0029805530793964863,
+            0.0176321379840374,
+            0.007832575589418411,
+            0.009573147632181644,
+            0.010507113300263882,
+            0.029377464205026627,
+            0.006049550138413906,
+            -0.0029929352458566427,
+            -0.008603804744780064,
+            -0.02141045406460762,
+            -0.013146273791790009,
+            0.009898620657622814,
+            0.013294858857989311,
+            0.03597182780504227,
+            -0.034896351397037506,
+            0.012162779457867146,
+            -0.011101454496383667,
+            -0.008264180272817612,
+            -0.008533049374818802,
+            0.0044717141427099705,
+            0.010068432427942753,
+            0.014370334334671497,
+            -0.0023402206134051085,
+            -0.01175947580486536,
+            0.009070787578821182,
+            0.023547254502773285,
+            -0.012983537279069424,
+            0.03016991913318634,
+            0.027325568720698357,
+            0.023066120222210884,
+            0.01761798746883869,
+            -0.012537780217826366,
+            0.026462359353899956,
+            -6.484030018327758e-05,
+            0.0006261814851313829,
+            -0.018084971234202385,
+            -0.00587973790243268,
+            -0.004294826649129391,
+            -0.004362043924629688,
+            0.013316085562109947,
+            -0.0025471788831055164,
+            -0.0008468485320918262,
+            0.015099111013114452,
+            -0.0037323245778679848,
+            -3.927453508367762e-05,
+            -0.0059752571396529675,
+            0.015127412974834442,
+            0.0002699744363781065,
+            0.019457615911960602,
+            -0.04270770028233528,
+            0.019726485013961792,
+            -0.021396303549408913,
+            -0.003102605463936925,
+            -0.0020890405867248774,
+            0.009523618966341019,
+            0.00764861237257719,
+            -0.007142714224755764,
+            -0.014334957115352154,
+            0.004524780437350273,
+            0.014066088013350964,
+            -0.0006726144347339869,
+            0.005080206785351038,
+            -0.0027541371528059244,
+            0.0017228835495188832,
+            -0.010089659132063389,
+            -0.005862049292773008,
+            5.842812606715597e-05,
+            -0.010570792481303215,
+            -0.008646257221698761,
+            -0.03232087194919586,
+            0.01062032114714384,
+            0.010209942236542702,
+            0.007294837385416031,
+            0.004252373706549406,
+            -0.022018946707248688,
+            -0.012318440712988377,
+            0.012948159128427505,
+            -0.01132787112146616,
+            -0.002570174168795347,
+            -0.0020005968399345875,
+            0.005387990735471249,
+            0.0004705205501522869,
+            0.010160413570702076,
+            -0.003472300246357918,
+            0.010938717983663082,
+            -0.014603826217353344,
+            0.018084971234202385,
+            0.03229256719350815,
+            -0.00533846253529191,
+            -0.019967053085565567,
+            0.000907432462554425,
+            -0.009077862836420536,
+            0.003813692834228277,
+            0.007641536649316549,
+            -0.021962342783808708,
+            -0.02173592709004879,
+            0.005087282508611679,
+            -0.013153349049389362,
+            -0.007783046457916498,
+            0.014830241911113262,
+            0.005221716593950987,
+            0.010839661583304405,
+            0.01751893013715744,
+            0.010889190249145031,
+            0.00840569008141756,
+            -0.026533113792538643,
+            0.0006734988419339061,
+            -0.012686366215348244,
+            -0.00590803986415267,
+            0.002239394700154662,
+            -0.0017591454088687897,
+            0.004591997712850571,
+            -0.00010115749319083989,
+            -0.011200511828064919,
+            -0.0007057808106765151,
+            0.010903340764343739,
+            0.027919910848140717,
+            -0.004146241117268801,
+            -0.0022570835426449776,
+            -0.014257126487791538,
+            0.0010100272484123707,
+            0.036481261253356934,
+            -0.012275987304747105,
+            -0.017434025183320045,
+            0.01757553406059742,
+            0.027452928945422173,
+            0.005954030901193619,
+            -0.0032529598101973534,
+            -0.0027364485431462526,
+            0.00039600671152584255,
+            0.011978816241025925,
+            0.007719367276877165,
+            -0.002692226553335786,
+            -0.002131493529304862,
+            0.004761809483170509,
+            0.035773713141679764,
+            -0.0015203474322333932,
+            -0.010330225341022015,
+            -0.024240652099251747,
+            0.010974096134305,
+            0.00277182599529624,
+            0.004754733759909868,
+            0.002313687466084957,
+            0.023504801094532013,
+            0.019655730575323105,
+            0.014858543872833252,
+            0.00470520555973053,
+            -0.009771261364221573,
+            0.03005671128630638,
+            -0.008172199130058289,
+            -0.004542469047009945,
+            -0.02838689461350441,
+            -0.005087282508611679,
+            -0.015650998800992966,
+            0.008929276838898659,
+            -0.0002221042668679729,
+            -0.005016527138650417,
+            -0.013422218151390553,
+            0.019726485013961792,
+            0.01064862310886383,
+            -0.011073152534663677,
+            0.012608535587787628,
+            -0.027070851996541023,
+            0.02346234768629074,
+            -0.03826428949832916,
+            -0.0071993181481957436,
+            -0.006421013735234737,
+            -0.010514188557863235,
+            -0.02627839520573616,
+            -0.0030353881884366274,
+            0.007372668012976646,
+            0.014250051230192184,
+            0.0022570835426449776,
+            0.0022818478755652905,
+            0.02282555401325226,
+            0.0016415152931585908,
+            -0.01508495956659317,
+            0.017985913902521133,
+            0.03492465242743492,
+            0.012926933355629444,
+            0.00155837822239846,
+            0.015990622341632843,
+            0.0027789014857262373,
+            0.0021633331198245287,
+            0.00788917951285839,
+            -0.0034493047278374434,
+            -0.007938707247376442,
+            0.008582578040659428,
+            0.006990591064095497,
+            -0.02763689123094082,
+            0.003824306186288595,
+            -0.00010408719390397891,
+            -0.009898620657622814,
+            -0.022429324686527252,
+            0.011568437330424786,
+            0.01913214474916458,
+            -0.004224071744829416,
+            -0.006771250627934933,
+            -0.0043337419629096985,
+            0.0006836698739789426,
+            0.006785401608794928,
+            -0.002412744564935565,
+            -0.000515184598043561,
+            -0.0015176940942183137,
+            -0.02368876338005066,
+            -0.010316074825823307,
+            0.013570803217589855,
+            -0.005175726022571325,
+            0.010358527302742004,
+            0.005242943298071623,
+            -0.018792519345879555,
+            -0.016938740387558937,
+            0.006852618884295225,
+            -0.016188737004995346,
+            0.01364863384515047,
+            0.00027948213391937315,
+            -0.0023172253277152777,
+            0.024962352588772774,
+            0.03594352304935455,
+            -0.013188726268708706,
+            0.005119122099131346,
+            -0.019386861473321915,
+            -0.0245378240942955,
+            -0.019004784524440765,
+            0.022613288834691048,
+            0.008306633681058884,
+            -0.018792519345879555,
+            0.012304289266467094,
+            -0.002752368338406086,
+            -0.008865597657859325,
+            -0.0066721937619149685,
+            -0.006845543161034584,
+            -0.008547200821340084,
+            0.020533092319965363,
+            -0.004935159347951412,
+            0.0038950610905885696,
+            0.019047237932682037,
+            0.04075486212968826,
+            -0.008292482234537601,
+            -0.01349297259002924,
+            0.0054339817725121975,
+            0.004878554958850145,
+            0.0018626245437189937,
+            -0.004814875777810812,
+            -0.008016537874937057,
+            -0.0185944065451622,
+            -0.01740572229027748,
+            -0.009615600109100342,
+            -0.0020041344687342644,
+            0.020122714340686798,
+            -0.007768895477056503,
+            -0.01354957651346922,
+            -0.0245378240942955,
+            -0.028188779950141907,
+            -0.006580212153494358,
+            0.02319347858428955,
+            0.017179306596517563,
+            -0.00013200224202591926,
+            -4.3337418901501223e-05,
+            -0.012042495422065258,
+            -0.01968403346836567,
+            0.015509488992393017,
+            0.00037876018905080855,
+            -0.01773119531571865,
+            -0.0065448349341750145,
+            -0.015919867902994156,
+            0.005787756759673357,
+            -0.0051368107087910175,
+            0.004574308637529612,
+            0.05057565122842789,
+            0.0005271245026960969,
+            0.008441068232059479,
+            -0.026575567200779915,
+            0.009106164798140526,
+            -0.01583496294915676,
+            -0.017377421259880066,
+            -0.00012702727690339088,
+            -0.022301966324448586,
+            -0.0030955299735069275,
+            -0.018792519345879555,
+            -0.007386818993836641,
+            0.008568426594138145,
+            0.009799563325941563,
+            -0.013896276243031025,
+            -0.013669860549271107,
+            -0.024693485349416733,
+            -0.0035872769076377153,
+            -0.001539805089123547,
+            0.009311353787779808,
+            0.007825499400496483,
+            -0.03693409264087677,
+            0.002536565763875842,
+            -0.0009507699287496507,
+            0.0005779796629212797,
+            0.01208494883030653,
+            -0.008667483925819397,
+            -0.003063690150156617,
+            -0.0214670579880476,
+            0.006452853325754404,
+            0.007103798910975456,
+            -0.013500047847628593,
+            0.012375044636428356,
+            -0.012863253243267536,
+            -0.007669838611036539,
+            -0.0005426021525636315,
+            0.03379257395863533,
+            0.011794853024184704,
+            -0.0025153392925858498,
+            -0.00654129683971405,
+            -0.011150983162224293,
+            0.02654726430773735,
+            -0.011349096894264221,
+            -0.026405755430459976,
+            0.028910480439662933,
+            0.013245330192148685,
+            0.0005974372616037726,
+            0.020151015371084213,
+            -0.0017936384538188577,
+            -0.01397410687059164,
+            0.009601449593901634,
+            0.004266524687409401,
+            0.012424572370946407,
+            -0.00014460546663030982,
+            0.0214670579880476,
+            0.0018555490532889962,
+            0.0027399861719459295,
+            -0.0036226543597877026,
+            0.004687516950070858,
+            -0.03093407303094864,
+            -0.0006549257086589932,
+            -0.003102605463936925,
+            -0.008398614823818207,
+            0.011207587085664272,
+            -0.002421588869765401,
+            0.006558985915035009,
+            -0.022188758477568626,
+            -0.028485950082540512,
+            0.011342021636664867,
+            -0.016231190413236618,
+            -0.021396303549408913,
+            0.015339677222073078,
+            -0.0007641536649316549,
+            0.00614860700443387,
+            -0.007443422917276621,
+            -0.023278385400772095,
+            -0.019386861473321915,
+            0.02064630016684532,
+            0.012212308123707771,
+            -0.01190098561346531,
+            0.012750045396387577,
+            -0.021113283932209015,
+            -0.012948159128427505,
+            -0.001894464367069304,
+            0.0090566361322999,
+            -0.0019864458590745926,
+            0.01325240544974804,
+            -0.012198156677186489,
+            -0.00334494118578732,
+            0.0031592093873769045,
+            -0.011589664034545422,
+            0.008158047683537006,
+            0.01978308893740177,
+            0.008193425834178925,
+            -0.009339655749499798,
+            0.003422771580517292,
+            -0.011922212317585945,
+            0.013500047847628593,
+            0.015537790954113007,
+            0.018311386927962303,
+            -0.006498843897134066,
+            0.0020412809681147337,
+            -0.01370523776859045,
+            0.0063962494023144245,
+            -0.009827865287661552,
+            -0.004878554958850145,
+            -0.0005333155859261751,
+            0.004921008367091417,
+            -0.011179285123944283,
+            -0.0025772498920559883,
+            -0.026830283924937248,
+            -0.0013823752524331212,
+            -0.020533092319965363,
+            0.005303084850311279,
+            0.023207630962133408,
+            -0.007514177821576595,
+            0.0006045127520337701,
+            -0.0039092120714485645,
+            0.014476466923952103,
+            0.020688753575086594,
+            0.00870286114513874,
+            0.00482548912987113,
+            -0.006081389728933573,
+            0.02179253101348877,
+            0.011603815481066704,
+            -0.017816102132201195,
+            0.00164239970035851,
+            -0.001509734196588397,
+            -0.0014336726162582636,
+            0.011625041253864765,
+            0.004443412180989981,
+            -0.020844414830207825,
+            0.0028602697420865297,
+            -0.010089659132063389,
+            0.007818424142897129,
+            0.014702882617712021,
+            -0.00031906069489195943,
+            -0.02486329711973667,
+            -0.009827865287661552,
+            0.021919889375567436,
+            -0.015070809051394463,
+            -0.0008291598060168326,
+            0.00764861237257719,
+            0.023264234885573387,
+            -0.024792540818452835,
+            -0.006831392180174589,
+            -0.0023455272894352674,
+            -0.003933976404368877,
+            -0.005193414632230997,
+            0.0010489424457773566,
+            0.0214670579880476,
+            0.004415110219269991,
+            -0.004234685096889734,
+            0.005724077112972736,
+            0.006470541935414076,
+            0.02325008250772953,
+            0.0040577976033091545,
+            0.028047269210219383,
+            -0.00943871308118105,
+            -0.0023809047415852547,
+            -0.0019988277927041054,
+            -4.775960405822843e-05,
+            0.0160047747194767,
+            -0.005430444143712521,
+            0.020320827141404152,
+            0.009445788338780403,
+            0.0036934094969183207,
+            0.016514210030436516,
+            0.02319347858428955,
+            -0.006767712999135256,
+            -0.004793649073690176,
+            0.019160445779561996,
+            0.0053667644970119,
+            -0.008858522400259972,
+            -0.02194819226861,
+            0.01141277700662613,
+            0.0014124460285529494,
+            -0.0006956097786314785,
+            0.012106175534427166,
+            0.0024375086650252342,
+            0.006300730165094137,
+            0.008172199130058289,
+            -0.016684021800756454,
+            0.004241760354489088,
+            -0.004698129836469889,
+            -0.01860855706036091,
+            0.010910416021943092,
+            -0.012092024087905884,
+            -5.201595922699198e-05,
+            -0.0010002984199672937,
+            0.014165145345032215,
+            -0.01703779585659504,
+            -0.0019103841623291373,
+            -0.0070719593204557896,
+            -1.3024719009990804e-05,
+            -0.0009162768837995827,
+            0.008320784196257591,
+            -0.006254739128053188,
+            -0.010938717983663082,
+            -0.010938717983663082,
+            0.027509532868862152,
+            -0.004991763271391392,
+            0.003916287794709206,
+            0.019330257549881935
+          ],
+          "embedding": [
+            0.007204997251390758,
+            -0.03173324616100502,
+            -0.020505790884701024,
+            0.0037451608231038273,
+            0.009817085391002229,
+            0.026883177166072393,
+            -0.016477826654817607,
+            0.06949311853773875,
+            -0.01224876667472692,
+            0.031952566873802865,
+            0.013633671951171586,
+            -0.005041437355104989,
+            0.02663392750069342,
+            -0.022099132865267587,
+            0.006665702000169926,
+            0.0030467129715919033,
+            0.007931802508415542,
+            0.009984477355413895,
+            -0.010362836464183985,
+            -0.03180848181251013,
+            -0.013542377333474449,
+            -0.013572345022577053,
+            -0.03979985300694044,
+            -0.007698412130292187,
+            -0.005953452234356532,
+            0.018890228917932694,
+            0.02217446994660065,
+            0.005158357501784182,
+            -0.05468694715835175,
+            0.00789994048345594,
+            0.017130184174786147,
+            -0.0026591475669416114,
+            -0.02081141846859738,
+            -0.008403928318906045,
+            -0.030925346724945978,
+            -0.03855023691059801,
+            0.038853354785097736,
+            0.01612011653434478,
+            -0.035635702395196575,
+            0.026266623757525934,
+            0.043450373922427316,
+            -0.007254334835408145,
+            -0.015376804869420366,
+            -0.0017276202867766907,
+            0.004650414207094434,
+            -0.002365514361535186,
+            0.011906139905776183,
+            -0.0005664194302009289,
+            -0.0191723447249157,
+            -0.025546124518037998,
+            0.01025005257280377,
+            -0.014436532079436437,
+            0.029277529346694118,
+            -0.0045414983847733344,
+            -0.02260315456913697,
+            0.01717918656248626,
+            0.013666132349359846,
+            0.057804090189845364,
+            0.056600576093100735,
+            0.051208022172527734,
+            0.03553760896972431,
+            0.02094599510567085,
+            -0.02438674472832137,
+            -0.0027797634437333432,
+            0.03614248430646102,
+            -0.026729308357113124,
+            -0.002251394534452815,
+            0.018929251842788906,
+            0.00689697483318378,
+            -0.026603606434115974,
+            -0.0541443728344337,
+            0.027164630314707807,
+            -0.004335272389193875,
+            -0.013967321193992958,
+            0.003323121377857836,
+            -0.0245295956412367,
+            0.024489335129134787,
+            0.06212059558640156,
+            0.009214058207162371,
+            -0.002082293435905329,
+            0.013659934201796686,
+            -0.003344037927706573,
+            -0.00689041317919184,
+            0.020235785390418023,
+            0.0212739457082623,
+            0.0016782346585238381,
+            -0.013484221664059555,
+            -0.06465582506604173,
+            0.011882797526607326,
+            -0.032970655964890155,
+            -0.04027700439305653,
+            0.025396847912528105,
+            0.013861157517716707,
+            0.00604596157583048,
+            -0.00744611407420936,
+            0.04038084298991919,
+            0.021507532290328625,
+            0.005983822806623904,
+            -0.012711110369915742,
+            0.02410138142963901,
+            -0.018372791517880082,
+            0.021016674128298246,
+            0.013387285956415756,
+            0.02039383525127615,
+            0.010700292312227496,
+            0.0053934660497349735,
+            -0.014807678014449691,
+            0.009422805142740376,
+            0.0414357625490679,
+            0.12206398267002624,
+            -0.010877509834324155,
+            -0.011079302503786522,
+            -0.04107714853756596,
+            0.006631397986165343,
+            -0.031914267104222384,
+            -0.0179965295457466,
+            -0.057530036803978266,
+            -0.0320488164246567,
+            0.008208202363320509,
+            -0.0020203090915064162,
+            0.03502055780919601,
+            0.04464599164916666,
+            0.019332152079287082,
+            0.017133943087431935,
+            -0.039885565008919065,
+            0.04703059988373572,
+            -0.029370696964304416,
+            0.0008494086835971101,
+            0.0014082189719987074,
+            -0.030973762582071615,
+            -0.006561157852024518,
+            0.011634653995510726,
+            -0.019896826887411713,
+            -0.034275921680252924,
+            0.026274465530039866,
+            -0.007728234125236442,
+            0.04814514040690998,
+            0.06550407296280117,
+            -0.02749051703615343,
+            -0.010041099502187074,
+            0.02019601967763537,
+            -0.01871701432558228,
+            0.008532829759864928,
+            -0.020164138072967613,
+            -0.045196925901010855,
+            -0.004523517988516156,
+            -0.002113469149506399,
+            0.07929650663710812,
+            -0.01111269708042164,
+            0.03358254444720164,
+            -0.019973382759721527,
+            -0.0007145592863325417,
+            -0.0011740862928390678,
+            0.02166182480535788,
+            0.004543228023398081,
+            -0.013377688428761516,
+            -0.01758250073647576,
+            0.009548292014596834,
+            -0.04753617258004735,
+            -0.007125744019792479,
+            -0.009930358894299303,
+            -0.007931672044235996,
+            -0.024561936537429774,
+            0.012979195012881556,
+            0.009209145250061479,
+            0.04586756662401364,
+            0.04311807301838248,
+            -0.0036241105681042675,
+            -0.06579256720896123,
+            0.04967191755416882,
+            0.015475886231466817,
+            0.03157651656768371,
+            0.02515941109300714,
+            0.01773554572598468,
+            0.005824418191900522,
+            0.016317635547562007,
+            0.007650872352957098,
+            0.010015166465492963,
+            -0.033203307142867844,
+            0.040272188258443496,
+            -0.013408423788025047,
+            0.01036004294342104,
+            0.012391622448703885,
+            0.04214857862444935,
+            0.03418032388760535,
+            0.015263018079889338,
+            0.018786704580603317,
+            0.02704685343985817,
+            0.015348111464412869,
+            0.0013031600434754704,
+            0.03241427528588285,
+            0.011689924942195664,
+            0.04544406114742012,
+            0.016186615914968734,
+            0.011624500626103958,
+            -0.008272953078864385,
+            -0.02363080189189375,
+            0.02784298119317616,
+            -0.010585980875564109,
+            0.01455013604305903,
+            0.01749228557615982,
+            -0.054742056843690476,
+            -0.005876213780171779,
+            -0.010556605098730201,
+            0.003967456397644932,
+            0.014663888851767762,
+            0.0015548476559947807,
+            -0.003858367677774835,
+            0.012504286824484335,
+            -0.025806882498628832,
+            -0.014216299453397403,
+            0.0031620924516417956,
+            -0.006448390918604307,
+            -0.02785577502346934,
+            -0.00766402687921159,
+            0.005275000635711884,
+            -0.0090448751942,
+            0.019702662630533,
+            0.007009091837467867,
+            -0.01847307100043082,
+            -0.0006774105206839772,
+            0.032952656101827446,
+            0.015900033044760432,
+            -0.02170121261931376,
+            -0.017964879081469612,
+            -0.005062849745084668,
+            -0.03171847232054206,
+            -0.038177398521606544,
+            0.029426291950046455,
+            -0.025687201683643784,
+            0.030158867757101797,
+            0.015780411479509624,
+            -0.007011484049169482,
+            -0.012114023533118393,
+            0.020661407942701628,
+            -0.025067530380881376,
+            0.009777190971413269,
+            -0.004160771288889298,
+            -0.028825369485727344,
+            0.008307290529469729,
+            -0.014415903546305744,
+            -0.007561235629544945,
+            -0.03201704135271719,
+            -0.015182869388110948,
+            -0.00501561450194676,
+            -0.0003863746169672397,
+            -0.01073403445816663,
+            -0.03321103354401307,
+            -0.002304357889348034,
+            0.017001213960120047,
+            0.00028222146729658097,
+            -0.04002882852650754,
+            0.014509588052058002,
+            -0.004150541038999649,
+            -0.034333837380045466,
+            0.0564137596224324,
+            0.03584898691608747,
+            -0.006830698606679202,
+            0.006121456445207273,
+            -0.04505674213649008,
+            0.04358467937307492,
+            0.010847859073780811,
+            -0.04918983713702805,
+            0.049190442935189385,
+            -0.00528960443684769,
+            0.01419195371129309,
+            0.040316024601793406,
+            0.004600843996974165,
+            -0.006780970176765864,
+            0.005206651850992678,
+            0.03397517915308201,
+            -0.010037223444627569,
+            -0.027377913828275806,
+            -0.03328136706915221,
+            0.02562568440972934,
+            -0.03485770844248482,
+            -0.020069979230687374,
+            -0.0052041408068332524,
+            0.029818953377223505,
+            0.028789460147370508,
+            0.004071274130619537,
+            0.01928375754300958,
+            0.0014970947734202088,
+            0.010502599795027616,
+            0.016914145702515263,
+            0.011942576738188502,
+            -0.014909953528043828,
+            -0.003432489584767786,
+            -0.05733768990013957,
+            0.0030710780388681002,
+            -0.02627280695622545,
+            0.007930906480371135,
+            -0.031227917075485834,
+            0.013132797714872345,
+            0.002514351081823354,
+            -0.007152021475569949,
+            -0.0066245205132327055,
+            -0.005279224195782313,
+            0.028512607222275155,
+            -0.00921960825771865,
+            -0.0359704105589433,
+            -0.021347597388353182,
+            0.011200044645812517,
+            -0.0004171294784927543,
+            -0.016784468873218786,
+            -0.0018746544944553878,
+            0.007749103417240419,
+            0.02390272097796833,
+            -0.005128540108845805,
+            0.006397849195755672,
+            0.01214922062611769,
+            0.01163371266250103,
+            0.012307186722130844,
+            -0.004982589662882946,
+            -0.016343724553037307,
+            0.019806887143115643,
+            0.013139657463054087,
+            0.0205801609265216,
+            -0.03767676007629612,
+            -0.0249879011550972,
+            0.05547392586363707,
+            0.005903313549319051,
+            -0.05568806298774435,
+            0.018827693208840233,
+            -0.007931178941307645,
+            0.008183992600768387,
+            -0.010319022946041053,
+            0.05111352384595183,
+            -0.024897203290807753,
+            -0.02096678205188722,
+            0.006436523335212348,
+            -0.04917084523640266,
+            -0.010459284852442706,
+            -0.011138558192296276,
+            0.014789370043397917,
+            -0.03105725978752438,
+            -0.004934745004818259,
+            0.015595801375817122,
+            -4.687585727052446e-05,
+            0.0028886067775750254,
+            -0.00850352807499054,
+            -0.0034885579171087815,
+            0.016339681487281293,
+            -0.0034050317046826813,
+            -0.008309570448487812,
+            -0.003615165868022203,
+            -0.027638409919916757,
+            0.023352910161586186,
+            0.026364511529991978,
+            0.001249936082895606,
+            0.001987948967948376,
+            0.01756629246122769,
+            0.004656075485833687,
+            -0.006752579977941239,
+            -0.013681608638716132,
+            -0.03405936202918687,
+            -0.0401867651078737,
+            0.022991180183813725,
+            -0.02318908140203705,
+            0.018299512436825923,
+            0.00904081847012441,
+            0.019697091729447287,
+            -0.019236130984371835,
+            0.015159917756323988,
+            -0.0387297415849285,
+            -0.018793498297161176,
+            -0.005256288213225733,
+            -0.010414697711744128,
+            0.0007039894599589104,
+            -0.01739645696111503,
+            -0.013728845328097506,
+            0.022709522955937343,
+            -0.022789243441781803,
+            -0.0036763802589970026,
+            -0.01429671717544519,
+            0.0023674126354206932,
+            0.049448298344058644,
+            -0.009926033487593551,
+            -0.023371285193090733,
+            0.020846792916650956,
+            -0.026996519120163966,
+            0.011246908457705283,
+            0.04091365159640201,
+            0.005469162850197613,
+            0.005640919201612091,
+            0.029575753750229278,
+            -0.0007860452837141593,
+            -0.0052805237594885516,
+            -0.009782745443933828,
+            0.021178503581190618,
+            -0.0009176512672832309,
+            -0.05245650420332918,
+            -0.007698364493410058,
+            -0.039027418406703554,
+            0.005448840777995117,
+            0.017581625031764185,
+            0.025522517097715556,
+            -0.020775492472580457,
+            -0.015906320383873178,
+            -0.000643669839811819,
+            -0.0086896084429604,
+            0.0029399233482444638,
+            0.014381240985647907,
+            0.021086693776922936,
+            0.002436575081260609,
+            -0.004913514868101904,
+            -0.011582101006213432,
+            0.014400297663309155,
+            0.0467181039341492,
+            -0.01722692180137712,
+            0.0011622224560709315,
+            -0.013036540368441797,
+            -0.06208454638917413,
+            -0.019160212331432766,
+            0.0351496273293468,
+            0.02429814055116254,
+            -0.009896662169762772,
+            -0.01647101145477158,
+            -0.002915821099802562,
+            -0.03557863734177486,
+            -0.01786433999615293,
+            -0.010699364747311715,
+            0.02390901423299708,
+            0.007213809581960646,
+            0.0005601364043388203,
+            0.00916614601011734,
+            0.014737491634516238,
+            0.019303599465763203,
+            0.008864535607492504,
+            -0.0575839606069681,
+            0.018051837786406608,
+            0.006315830080225406,
+            0.02525619013082567,
+            -0.015671924543113065,
+            0.003988943108210209,
+            -0.03179878281917948,
+            -0.029440641244429775,
+            0.040586322748158446,
+            -0.03418722657593859,
+            -0.030650542761032924,
+            0.006566745611226466,
+            0.009571089937153852,
+            0.004491158558315032,
+            -0.011352813042040581,
+            -0.02535505987721779,
+            -0.013381948772812133,
+            -0.030918676056735445,
+            -0.02192708443081135,
+            -0.005232667170255381,
+            -0.03342408726436631,
+            -0.07498634115550039,
+            -0.006245638452568239,
+            -0.049946896465756174,
+            -0.03900385398225758,
+            0.04458524125677807,
+            -0.006106489377959481,
+            0.02398410968928338,
+            -0.04615183642824994,
+            -0.01165382599292015,
+            -0.04890013373371541,
+            0.013624324077017377,
+            0.01476398305411351,
+            0.020202554298470666,
+            0.00014553149242532853,
+            0.036224316512987846,
+            0.00021498496267774312,
+            -0.05055208658013767,
+            -0.057482169168904106,
+            -0.003960451575823462,
+            -0.010217078509789989,
+            -0.005593738972150311,
+            -0.034090261748104665,
+            0.05835341009022349,
+            0.0116359661127526,
+            0.04203678184748103,
+            0.03447818335186531,
+            -0.006747564131379555,
+            -0.025756962754841885,
+            -0.028123416424745064,
+            -0.008073625379614702,
+            0.047544241458556825,
+            -0.029605671083240564,
+            0.008171081332993893,
+            -0.026007452205474395,
+            0.0067084589781962805,
+            -0.0070645448617940995,
+            -0.0012298101311792947,
+            0.012372286141738867,
+            0.006377319686763927,
+            -0.015024537877377194,
+            0.007934745664790507,
+            0.026041755081605937,
+            -0.007303731324541215,
+            0.0407000170257006,
+            -0.012990703097937777,
+            -0.033189084013863494,
+            -0.022525679964848666,
+            0.01657717698191603,
+            0.03440284885162039,
+            0.010230633738702899,
+            -0.008707808754455952,
+            -0.0363777020540216,
+            0.011184076269109602,
+            0.031774012431843524,
+            0.01504703222643543,
+            -0.022509833303978505,
+            0.025230974050270107,
+            0.017502662694813216,
+            -0.018700579139537456,
+            -0.03511734747709348,
+            0.015162183119327901,
+            -0.017711638132862063,
+            0.018190335050681287,
+            -0.01590174153215615,
+            0.011103110869700727,
+            0.0025189370696524936,
+            -0.00920305384736575,
+            0.01929652151514404,
+            -0.006196715938358087,
+            0.011426611271994685,
+            0.007058186102245829,
+            -0.018191352599117545,
+            0.0027202185072220937,
+            0.057310566886380025,
+            -0.000531639834386418,
+            0.01494975681909879,
+            0.03845692752400366,
+            -0.001971953300066856,
+            0.045554909803324765,
+            -0.02532332999916258,
+            -0.042616337063845953,
+            0.02633963974172818,
+            -0.02970705577683783,
+            -0.01624154998005974,
+            -0.02620993449678864,
+            -0.021490817420853802,
+            -0.004038424045768319,
+            0.009940887982818171,
+            -0.02692933845395842,
+            -0.023801570856681796,
+            -0.001948460540662095,
+            0.02290911497874141,
+            -0.03300532378029689,
+            0.006525488380791647,
+            0.007039306647666766,
+            -0.003685585408710752,
+            0.03183735852538168,
+            -0.027197064244321705,
+            0.006861928286688311,
+            0.011209869849682412,
+            -0.001094897323854524,
+            -0.005328676992017907,
+            -0.004007474089818478,
+            0.010796946231051235,
+            -0.030543173497906445,
+            -0.012879972141706173,
+            0.00935894527375688,
+            0.0017573071057611829,
+            -0.010050882212831222,
+            -0.03102132152986303,
+            -0.002915141291985554,
+            0.007634137769426993,
+            -0.011094093612593696,
+            0.01199609810193786,
+            -0.010458016758814742,
+            -0.04929347607956551,
+            0.010973132159729983,
+            0.028148117094907874,
+            -0.025504019123994758,
+            0.012470127357818321,
+            0.009182729379865986,
+            -0.016020133756226275,
+            0.016320111225714855,
+            -0.012051271446461976,
+            0.024098195821827578,
+            0.04317620443481784,
+            0.024592886889234396,
+            0.007727787037954994,
+            0.00771746776156926,
+            -0.0003147538723468374,
+            0.022572868970938624,
+            -0.016683340264320543,
+            0.013672085891597506,
+            0.020827021901755415,
+            0.02092309219721837,
+            0.0057130785714458055,
+            -0.003901728981013356,
+            -0.0036968415923910314,
+            0.002227024825579113,
+            -0.02106678364798051,
+            0.014578546755457675,
+            0.03128101864424301,
+            -0.005234627629617124,
+            -0.022394744692442837,
+            0.002430617645746389,
+            -0.015780389432069438,
+            4.7277885648917e-05,
+            0.013357803020920043,
+            0.013777651543690907,
+            -0.027713499926132385,
+            -0.0018999112228756628,
+            0.028364630573666852,
+            0.009146793705085586,
+            -0.009891297866910313,
+            -0.01644269527910728,
+            -0.0003903768506971414,
+            0.006512217314869576,
+            0.02408785204739448,
+            0.0018406499545367782,
+            -0.007360473750031044,
+            0.01484497577088404,
+            0.016723277095327158,
+            0.003629911330658209,
+            -0.0004820567356447713,
+            0.02394406451074397,
+            0.023794019615473026,
+            0.001632266615911801,
+            -0.02064379152040184,
+            -0.02755693521684878,
+            0.027335938400284726,
+            0.02769248863677192,
+            -0.006832353240623308,
+            0.027324973360715134,
+            0.003937207909967385,
+            0.013580154030133622,
+            0.02076818768224654,
+            -0.01784151382870951,
+            -0.017617903734311997,
+            0.0017316476451235418,
+            -0.009605294810027804,
+            -0.009829196696575069,
+            0.004891780008779742,
+            -0.011655402383771472,
+            -0.01124379942906224,
+            0.043205454954628145,
+            0.004713701851783307,
+            0.003560551284131962,
+            0.007894840447501912,
+            -0.018881214526749082,
+            -0.00945538175144614,
+            0.0019040259741796817,
+            0.029505863720301916,
+            -0.001400002868254808,
+            0.03315866433970851,
+            -0.00830542045504805,
+            0.008602408964961391,
+            -0.03202191233287069,
+            -0.006978289755114341,
+            0.0326499698728755,
+            0.015314305362524091,
+            0.015104436300213667,
+            0.0021257042489266787,
+            -0.0010840650756005028,
+            0.02434348355847526,
+            -0.007174433778817674,
+            -0.04839373063710409,
+            0.022696994450018916,
+            -0.004779511451775743,
+            0.023588216956749154,
+            0.0382733980620746,
+            -0.004232286553944853,
+            0.003815507826545929,
+            0.023531223314846965,
+            -0.01887650634213847,
+            0.015055403379782985,
+            -0.01918944132635701,
+            -0.017597900721414636,
+            0.02176630010543207,
+            -0.008896608357798277,
+            0.01732387427588561,
+            0.01764363756727683,
+            -0.012180043444768312,
+            -0.019003360499321047,
+            0.01229139850966224,
+            0.0025710091163655626,
+            -0.017075658234265562,
+            0.01797412486853335,
+            0.018372010308966135,
+            0.006723179570342314,
+            0.015417677746651683,
+            -0.00047634684109332134,
+            0.017608296628666153,
+            -0.001049984114818522,
+            -0.008416990841821452,
+            0.01827827907328822,
+            -0.009568717277044265,
+            -0.01658089563620308,
+            0.009808478422454319,
+            -0.05193977747210182,
+            0.010644335931025957,
+            0.0016781339270411392,
+            -0.01562285100294917,
+            0.00405036330963539,
+            0.007102724536527262,
+            0.005195069570261247,
+            0.0028773630901681767,
+            0.004610474981005765,
+            0.042921486858787584,
+            0.013213672397112759,
+            -0.0064030584930089415,
+            -0.010711457508461704,
+            -0.017480309425763254,
+            -0.027151012983864797,
+            -0.005204425551873271,
+            0.009927884476991038,
+            0.022601230152836282,
+            0.008046297278661014,
+            -0.0031858480011267704,
+            -0.02589628103025927,
+            0.019723430185137954,
+            0.03604835658149313,
+            0.062423099201768716,
+            0.010930268386637612,
+            0.02925564269324381,
+            0.02439517070384333,
+            0.023930432096006114,
+            0.013036828091946605,
+            0.006910311087503408,
+            -0.032995841694293995,
+            0.012813650252889403,
+            0.01333714655191133,
+            0.03678966264802025,
+            0.012571138353467183,
+            -0.05610151838497368,
+            3.0777966587107344e-05,
+            0.009791774904454618,
+            0.041959488802722524,
+            0.01956296954085397,
+            0.000155217569370113,
+            -0.007285602454264675,
+            0.031547870838711856,
+            -0.026518385135879206,
+            -0.012299593280363843,
+            -0.0026551378470968594,
+            -0.006190117573235779,
+            -0.003807907001161504,
+            -0.004991957475076049,
+            0.036983035940964804,
+            -0.015648565314743963,
+            -0.019256463332683493,
+            0.0004300362961407231,
+            -0.0015686065121077472,
+            0.0011311524136286674,
+            0.0032236641835276464,
+            0.013795345278976164,
+            0.015385029265915279,
+            -0.004800448639353721,
+            0.026514926384796363,
+            -0.008699453553651681,
+            0.022186817580012775,
+            -0.013427553742493343,
+            -0.011927259975371873,
+            0.017776228476138103,
+            -0.021243797085501972,
+            -0.01667280589187595,
+            -0.003688045116580565,
+            0.006189970874536939,
+            0.0020911968266404467,
+            0.0006097761986763103,
+            -0.004685468039463817,
+            -0.01011357666327168,
+            0.022631082517271542,
+            0.012398000826637277,
+            -0.006721588925828951,
+            0.018142966317821054,
+            -0.0021330638043572654,
+            -0.011869184647489572,
+            0.025073455250141923,
+            -0.0015392945266926425,
+            -0.0337085890598252,
+            0.01267302113143573,
+            -0.013438601165482187,
+            0.02673290895281225,
+            -0.009586762706902736,
+            -0.011208527168019762,
+            0.019144928964011168,
+            -0.0017591316941248897,
+            -0.005459150067129732,
+            -0.029230079356228025,
+            0.004679497471366242,
+            -0.011723195942933286,
+            0.015899377439148873,
+            0.0022054234777254627,
+            0.029187759252864872,
+            -0.018290694895057157,
+            0.00792533824593637,
+            0.011719198071061566,
+            0.014612375997487872,
+            0.005419943379355563,
+            0.010515265723144554,
+            -0.006893314445458116,
+            -0.04424495426064596,
+            5.8241472764594054e-05,
+            -0.005719143565673274,
+            0.0038220021965914754,
+            0.013106940982933836,
+            -0.01203955335201393,
+            0.011260626096963215,
+            -0.0279896016489392,
+            -0.0295228261396555,
+            0.036474811902902415,
+            -0.04900747926861378,
+            0.019261089742811332,
+            0.00014104646541594166,
+            -0.03057197731717424,
+            -0.04331886560073797,
+            -0.03232957159604667,
+            -0.019401402365464362,
+            0.004102283356333861,
+            0.0002610070704924167,
+            0.0012280372705500212,
+            -0.014025012323929194,
+            -0.014135990809256075,
+            0.013269444858490211,
+            0.00992155580941576,
+            0.011225162609266308,
+            -0.006545044517282357,
+            0.002368862850663672,
+            -0.02063183105457681,
+            -0.026981383275694142,
+            -0.02774194980145225,
+            0.013337303400678109,
+            -0.0025902548982556788,
+            -0.02794339093668326,
+            -0.025606606546251152,
+            -0.0029852498102770587,
+            -0.01906873442473352,
+            0.006508845592362198,
+            0.03268861529625085,
+            0.0355975143313428,
+            0.010066906387952277,
+            0.01690250467284985,
+            -0.017201544758038183,
+            0.048345309114140714,
+            -0.019031171948182872,
+            0.03188329748476941,
+            -0.0005845169448621133,
+            -0.024140202991373587,
+            0.00011597263700677706,
+            0.03203623528903764,
+            -0.019285565857567896,
+            -0.01332755507465857,
+            -0.010914187315481855,
+            -0.003374912271519852,
+            0.02419326154642231,
+            0.02443825785057301,
+            -0.0037923303809811746,
+            0.00022800399604730175,
+            0.015893265651968885,
+            0.003154527829977343,
+            0.007688529734165207,
+            0.030623379985627763,
+            0.014052282574575454,
+            0.01127235345371133,
+            -0.04164707563955819,
+            -0.011811587719246128,
+            -0.003763086694358232,
+            0.005909410979841885,
+            0.002913220159138154,
+            -0.002671613084376487,
+            0.0045765194162907885,
+            -0.007241703279856166,
+            -0.005855949422953439,
+            0.0016973541436777935,
+            -0.001576326424468957,
+            0.008393873711385013,
+            -0.0037217094191171083,
+            -0.03498879678492834,
+            -0.026096985039773694,
+            -0.021280075788049788,
+            0.006437497348154639,
+            0.0070214556591898375,
+            -0.00972721128392027,
+            0.01949553762684561,
+            -0.0029404319702066146,
+            0.000271037702827666,
+            -0.010734867909488861,
+            -0.0029784341515646075,
+            -0.01206464664516104,
+            -0.019544778160755956,
+            0.00035139045159739705,
+            0.013457601448003152,
+            0.00025947376767512006,
+            -0.03165522602526639,
+            0.011979811705146604,
+            0.006642849985850598,
+            0.013181162886675883,
+            -0.024499404540711422,
+            -0.02310714308893576,
+            -0.028028912907092744,
+            0.010570122662701183,
+            -0.0066866321204157725,
+            -0.024617571284817428,
+            -0.004097807033448707,
+            -0.0074576460628928256,
+            -0.0020417256533259887,
+            -0.02470953739335365,
+            -0.011408461668188081,
+            0.023537349682260345,
+            -0.006373652338136973,
+            0.01936338676879376,
+            -0.017636435070085086,
+            -0.017036091876814843,
+            -0.002198685044404603,
+            0.015584758061011048,
+            -0.038943923931820196,
+            -0.002076529978164001,
+            -0.008452513321945455,
+            0.006551055965075869,
+            -0.028840587857625097,
+            -0.007128279504360441,
+            0.0199397499488538,
+            0.012424319366271584,
+            -0.03518875279748495,
+            0.013596027106451115,
+            0.022191403935861113,
+            0.01218507466930854,
+            -0.0038551718784437783,
+            0.03910075115429626,
+            -0.0025215641203030336,
+            -0.005422688956956598,
+            0.016759888275451905,
+            0.04462538468871483,
+            0.023708097134660096,
+            -0.03152743757744915,
+            -0.008668774531128668,
+            -0.015224825785549947,
+            0.004102648962663789,
+            -0.025867795193966987,
+            0.004676262922216584,
+            -0.011739753544166856,
+            0.01645951757031186,
+            0.011253400919230348,
+            0.013958707917471253,
+            0.0376596076542735,
+            -0.005077354977296842,
+            0.02668910096392324,
+            0.0010332284250006117,
+            -0.00856723931523911,
+            0.0014431148305642308,
+            -0.01412579094241589,
+            0.029892393553325205,
+            0.02269599036836208,
+            0.014421409701746036,
+            0.029248195108770955,
+            -0.03588867295960255,
+            0.02458655544632528,
+            -0.0010013539341895672,
+            0.008368648583431747,
+            0.018150734970019018,
+            0.02628022565029391,
+            0.015423867373844644,
+            0.012477080028047823,
+            -0.02513018750675568,
+            -0.022180001014682045,
+            -0.0027550350630288997,
+            -0.031944843677988374,
+            -0.0014668093006914267,
+            -0.009789309083948147,
+            0.026386027599666254,
+            0.0046988411391149,
+            -0.019803160322638583,
+            -0.003931806845108628,
+            0.004482097428470392,
+            -0.00244660100774939,
+            -0.010441732346892935,
+            -0.004649318716095894,
+            -0.01869464992812615,
+            -0.017838643679294407,
+            0.023677878122153204,
+            -0.023958455640817666,
+            -0.025238725541146534,
+            0.014582798102040157,
+            0.0013223590277629197,
+            -0.002295641913710489,
+            0.0008238518461567685,
+            -0.02115533275282518,
+            -0.04214275933987015,
+            -0.027817083427786927,
+            0.0037588803464689412,
+            0.03565056002940463,
+            -0.026788865866928923,
+            0.014162550819567465,
+            -0.011336583458519,
+            0.019175861037658344,
+            -0.005127229083828615,
+            0.015531116410874209,
+            -0.018251290727445846,
+            -0.044630374763679266,
+            0.021833307809708984,
+            -0.012087009688208446,
+            0.02720742725750304,
+            0.021497809277370574,
+            -0.026937653159100156,
+            -0.013572733244188407,
+            0.01358214612023832,
+            0.0020408253706382332,
+            -0.009652962552750464,
+            -0.0063411929391088626,
+            -0.010533309167065193,
+            0.003901271588567942,
+            -0.04496345296706673,
+            -0.008761188928422701,
+            0.06948704056954039,
+            0.005916180375965137,
+            0.03157905519638486,
+            0.014668818318138927,
+            -0.028448496686381258,
+            -0.04008949536631331,
+            -0.00462253589823598,
+            -0.0003701522833020314,
+            -0.00024410762276269235,
+            -0.01850585249309217,
+            0.017699615637510956,
+            -0.002712069912576265,
+            -0.009767338815933488,
+            -0.013119464222297351,
+            -0.013743610418034631,
+            -0.030375365173543157,
+            -0.0030316327877227783,
+            -0.008632689470358149,
+            -0.006565457264917239,
+            -0.0328089275944709,
+            -0.011713996290006943,
+            0.029773910045828737,
+            0.02740100904353622,
+            -0.0061926915497348915,
+            0.00638737826122608,
+            -0.010244154924811975,
+            0.0010396203272677069,
+            -0.001264454445070139,
+            -0.004059969507786729,
+            0.008715730726831364,
+            0.0014767015415289076,
+            0.031074767756160294,
+            0.004761858344125113,
+            0.00987870172044607,
+            -0.059926117793699066,
+            -0.009294021263133636,
+            0.021188432871397475,
+            0.016602826531924377,
+            -0.009125784136292983,
+            0.013946188378030194,
+            -0.0017670794602549428,
+            -0.006837334179368295,
+            -0.028290055353601393,
+            0.024556541126233806,
+            0.04543639597357669,
+            0.0031311343860416473,
+            -0.011199639773247286,
+            0.014965677855595444,
+            0.0018248533506196013,
+            0.015507721118783542,
+            -0.011409402378400194,
+            -0.01338737990904486,
+            0.03063410115541065,
+            0.009305362935335817,
+            0.0025578631174065385,
+            0.012727648343848194,
+            -0.0038680554900795763,
+            -0.01173215289303824,
+            -0.01568862495293563,
+            -0.004101703129244437,
+            -0.03649200408657064,
+            -0.0022107849490954137,
+            0.009659780679763728,
+            0.0034091693454195806,
+            0.008848380026939594,
+            0.01842435611499587,
+            0.03546926214224266,
+            0.013478278853234764,
+            0.005970293629431872,
+            -0.0069197533761813455,
+            0.023196445250565058,
+            0.010403914879205351,
+            0.010017272564548626,
+            0.015076807790462658,
+            -0.029649260409273275,
+            -0.012737152920662164,
+            0.00373276777994811,
+            0.013239909450830258,
+            0.0001682544254742693,
+            0.02030652051051447,
+            0.014143961923579586,
+            0.0010147733405793957,
+            -0.012509446388074923,
+            0.0020437769941487626,
+            0.007405578085797006,
+            0.013458276946120546,
+            0.0006779114540819997,
+            -0.004250745498730309,
+            0.0009149988500358458,
+            -0.008355906072311008,
+            0.040391939952386505,
+            -0.01831218885700067,
+            -0.016793844108223176,
+            0.035064882870542116,
+            0.11416006426189942,
+            0.0010630856112440689,
+            -0.015660014600661857,
+            -0.017142534874931403,
+            -0.0086899013413391,
+            0.016611825572132997,
+            0.01609650977226868,
+            0.009780106186629793,
+            0.009622367309076686,
+            -0.000672926017804395,
+            0.024310949465350565,
+            0.02944698490225121,
+            -0.0037169531700178868,
+            -0.010990838766452194,
+            -0.036819646475041955,
+            0.022923962979125287,
+            -0.004596366586768123,
+            -0.007108041562635549,
+            -0.002604794313552557,
+            0.012641223813290853,
+            0.013668775645501733,
+            0.005125100040230003,
+            0.0077993841846437815,
+            -0.011500851674334447,
+            -0.017334458072700107,
+            0.0056819751688881774,
+            -0.015982234644378358,
+            -0.0076533013358874935,
+            -0.013127172654362694,
+            -0.011837346554842119,
+            -0.006562312362371076,
+            0.0029877420008976425,
+            -0.008572723854050838,
+            0.00854083969164562,
+            -0.010939639783598493,
+            0.016185642759084087,
+            -0.008052282332943615,
+            -0.0011992212936609377,
+            0.0010378646897070758,
+            -0.0025399515501741718,
+            0.010712865639618352,
+            0.01451357949543628,
+            -0.02521401163091652,
+            0.020836357346214075,
+            -0.008252813367155418,
+            0.0006409807249997337,
+            0.015338809580802425,
+            -0.04672526649678653,
+            0.022728876784937357,
+            0.005129225753457052,
+            -0.018372222092794564,
+            0.0066182736895667805,
+            -0.02048375780196863,
+            -0.00896382274306482,
+            0.0021738974344880794,
+            0.004491167150658828,
+            0.0034280799511763956,
+            0.015611133773752265,
+            0.018126362497548308,
+            0.00014383671280220997,
+            0.007927708693242186,
+            -0.006973076292062532,
+            0.008118313055163714,
+            -0.006791836664375275,
+            -0.01164567445668625,
+            -0.010894577039363014,
+            0.00014731542306100506,
+            -0.008865132658571428,
+            -0.011415696755798914,
+            -0.017922318360382588,
+            0.00358538001727405,
+            0.00289327637128695,
+            0.010406054711003452,
+            -0.007757607207626101,
+            -0.010994775479337082,
+            -0.011716096880261664,
+            -0.0013333474316266438,
+            -0.01231144419953554,
+            0.009792603083441917,
+            -0.02884953407490196,
+            0.0029097409399144306,
+            0.025277274938032477,
+            0.009263530315933738,
+            -0.028037993274175277,
+            -0.02532252134134532,
+            -0.0018596440554143638,
+            -0.013847737486669237,
+            0.009910107581119272,
+            0.013952962280235523,
+            -0.011807495289410774,
+            0.003896325037406752,
+            -0.02144921729633576,
+            -0.01628545945817568,
+            -0.0040148671532777005,
+            -0.016032493124255323,
+            0.004580041957001551,
+            -0.011375180364988195,
+            -0.007789996450583554,
+            -0.022041665680945066,
+            -0.005890065942573431,
+            -0.050455058269193866,
+            0.014401045842891128,
+            0.030219548461217365,
+            -0.004580747668649223,
+            -0.028684680969792875,
+            -0.015525870112797619,
+            0.011121401958547706,
+            -0.008879749268609245,
+            -0.019454243014778435,
+            -0.007839434838464604,
+            0.018908534797107714,
+            -0.0019283067432864216,
+            0.0024769961730655835,
+            -0.006732653593696035,
+            0.027604805021030646,
+            0.01366593691107021,
+            0.03740746059041736,
+            -0.014662268336336578,
+            0.02539666271785517,
+            0.02371809783638648,
+            -0.036126848124425154,
+            -0.007627445211958312,
+            0.03657481892016626,
+            0.01092515982671624,
+            0.0011046573579274704,
+            0.012350047525582187,
+            -0.006340802546498992,
+            -0.031330519599469066,
+            -0.0013691360532301728,
+            -0.0014676630169621005,
+            -0.023111265345539388,
+            0.027359543575867225,
+            0.018091458372134273,
+            0.014237156751820934,
+            0.0030993618578507172,
+            -0.0014037082734049385,
+            0.013257269860435359,
+            0.01021315143964527,
+            0.030885417052806473,
+            -0.003225065956381245,
+            -0.012825341172770057,
+            0.013476039084668826,
+            0.01852164843904835,
+            -0.009965495459579544,
+            0.012541079912270015,
+            -0.00824122613985912,
+            -0.018124134280975132,
+            0.03192263766222747,
+            0.0001484308749572035,
+            0.010888733498667343,
+            -0.010649515858317311,
+            -0.013082335214246402,
+            0.032614733822945205,
+            -0.00999075761269065,
+            0.004257643407198955,
+            -0.030723533031261387,
+            -0.003820776505926347,
+            -0.010471792928758869,
+            0.01737608180148319,
+            0.004597677757231315,
+            0.02095148515295301,
+            0.029253627700524396,
+            -0.011604450820008121,
+            -0.01996325509217509,
+            -0.001158626158150122,
+            -0.020583793738536985,
+            0.004168757459099133,
+            -0.03632338333261702,
+            -0.019373660230711246,
+            0.022259216603413172,
+            -0.010364620831688015,
+            -0.004751193403783363,
+            0.00444187509644351,
+            0.0008572202661850934,
+            0.014943779567410408,
+            -0.025408263453313398,
+            0.017755758370543148,
+            0.004627392860417132,
+            0.007402073071525438,
+            -0.024913329140362466,
+            0.013760771254234878,
+            -0.006087346663657887,
+            -0.03780469033982951,
+            0.005509388120163784,
+            0.03756141956381493,
+            0.02381019168406692,
+            -0.024580249929652097,
+            0.01807584361080568,
+            -0.03111039245576191,
+            0.023245542501125073,
+            -0.004868090951368273,
+            5.1544460403943004e-05,
+            0.00640235757127173,
+            -0.004698373089469779,
+            -0.00621383035264185,
+            -0.0025132415458431587,
+            -0.008746634331769936,
+            -0.01611801559493664,
+            -0.01193535623883546,
+            -0.03381789115349148,
+            -0.024119754887378317,
+            0.013881102852385332,
+            0.0019232668966222687,
+            -0.0095263729740736,
+            0.018652966857927805,
+            -0.03775912497511837,
+            -0.012022352025516115,
+            -0.023289377270709736,
+            0.023579672990954228,
+            -0.028605627815260902,
+            0.019235397553197357,
+            -0.025838655372560505,
+            -0.021458526449310157,
+            0.01508157885177254,
+            -0.010064396230734452,
+            -0.0016708624560165368,
+            -0.015042071642751917,
+            -0.03670785416972153,
+            -0.006267614265977843,
+            0.02298763967146357,
+            -0.012506444728541897,
+            0.00029871598584066644,
+            0.02671390618094565,
+            -0.003059865833779805,
+            -0.011103452976820622,
+            -0.0017382684168257506,
+            0.010923055147701782,
+            -0.02938686260440486,
+            0.014300097108092281,
+            0.0038571851420598074,
+            -0.0037614499297117365,
+            0.0010991440540809893,
+            -0.010108413486243776,
+            0.02127952161226619,
+            0.009142545227414532,
+            -0.0030260119510809205,
+            0.040589877142650374,
+            0.008677453889680373,
+            0.0017339508605300814,
+            0.008732950895530873,
+            0.01063352447960556,
+            0.006618635029823979,
+            -0.014986307792626314,
+            0.016905861015681957,
+            0.013101237900847744,
+            0.0002162791516784468,
+            0.019309343995079022,
+            0.0021263376627204313,
+            -0.024516267132902416,
+            0.013339486637573757,
+            -0.02311659751478777,
+            0.0020575982953961246,
+            0.02391695586495566,
+            -0.020440172706421535,
+            -0.001656132876701879,
+            -0.006304133667169861,
+            0.000543457661904495,
+            0.022135707414160034,
+            -0.0230903871986617,
+            -0.023444763940153112,
+            0.009604600033024832,
+            0.01721387169484476,
+            0.01827905276638783,
+            0.02824773106648473,
+            -0.007100625424382998,
+            0.01348510178676837,
+            0.0002604386361122316,
+            -0.011802178189607182,
+            0.007664157334475143,
+            0.0074175377458313594,
+            -0.0044008460682833875,
+            0.025491035291032547,
+            -0.0017015687016376205,
+            -0.006134010425966364,
+            -0.005379657499404915,
+            -0.01584573519366957,
+            0.0034386456145185213,
+            -0.013860742817569676,
+            -0.013355238174583658,
+            0.0040342831827141845,
+            -0.0208160950322342,
+            0.10904592104153767,
+            -0.0037038089210379766,
+            -0.019472012964045247,
+            0.0063554319446707044,
+            0.003716719884684827,
+            -0.003692786470980806,
+            -0.003353843210051379,
+            0.007788605566526459,
+            -0.011254480693190834,
+            -0.0045751956417398095,
+            -0.013976039724381875,
+            -0.001989262807706565,
+            0.010067374586433014,
+            0.009201983771978006,
+            -0.005902980676079493,
+            -0.02029714318363332,
+            0.013031676901335606,
+            -0.016970144750162643,
+            0.008874670945038486,
+            0.0008311255074953139,
+            0.00884167159995975,
+            0.001716603195908536,
+            0.01854951250316076,
+            -0.009310881405486074,
+            -0.01325242738760595,
+            0.029667589863813217,
+            -0.005616511284128333,
+            -0.0038179509972925954,
+            -0.0004690716809300012,
+            -0.005068974756744239,
+            0.0021204231125051274,
+            -0.027958274497127258,
+            0.022127088236172387,
+            -0.017271838492631593,
+            0.010267709087809265,
+            0.01316800560257366,
+            0.005599367128954229,
+            0.01817806970002991,
+            -0.007115594036834107,
+            -0.02627643123342057,
+            0.006542742528003751,
+            -0.003433874053082844,
+            0.006631427284537971,
+            0.02316295746074422,
+            -0.014573351469966297,
+            -0.033520864104177005,
+            -0.013707022143307006,
+            -0.01877574599842729,
+            -0.005992004412448576,
+            -0.0043698881826095435,
+            -0.01481352026460129,
+            0.006424121402963946,
+            0.01382938882387991,
+            0.018136021761515388,
+            0.0012697483790911433,
+            -0.011531136974861559,
+            0.00609187274277241,
+            0.01076133059240578,
+            0.008400887242930113,
+            -0.01533504932731406,
+            0.011439227832642668,
+            -0.028718487721012004,
+            0.01522697877548034,
+            -0.012452665599305595,
+            -0.024938748080573947,
+            -0.0022082450217489366,
+            0.017589399637694264,
+            0.0037824832255756763,
+            -0.004355445812154662,
+            -0.018778668474337727,
+            0.019825087149714933,
+            0.024692878851503508,
+            0.003883497401229232,
+            0.019470881143244197,
+            0.041834635328088364,
+            -0.003544509491805351,
+            0.027139164223766474,
+            -0.02129723143323512,
+            0.01461354860170749,
+            -0.03441729954693155,
+            0.001893713808284104,
+            -0.004222657393633789,
+            0.0013677766871517654,
+            -0.003811928806805752,
+            0.020142937759360904,
+            -0.009158202577174012,
+            0.0030638058587375543,
+            0.024854053046139163,
+            -0.018665378030958756,
+            0.00805903734779081,
+            0.010625429315145887,
+            -0.012856237251902233,
+            0.028535299711292306,
+            -0.006335094045840286,
+            0.00305422126851784,
+            -0.0297883488914759,
+            -0.0029192872993071216,
+            0.0026067044379754047,
+            -0.006097749614529602,
+            -0.00968023644800927,
+            -0.007433916539502931,
+            -0.006202274026852405,
+            0.0038354607448269555,
+            -0.021471361566683354,
+            0.003195646827154078,
+            -0.004390751282171024,
+            -0.003323447423321121,
+            0.02518759723554948,
+            0.025857746688368054,
+            0.016893707217894103,
+            0.012739170140289531,
+            0.008553625305650129,
+            0.019805351588748434,
+            0.01654964927067909,
+            0.011549329129728739,
+            -0.020026013396046632,
+            0.01150644479272541,
+            -0.004159611921476011,
+            -0.0022937297602724912,
+            0.014815466072709131,
+            -0.01572171840394285,
+            -0.02276187015864222,
+            0.024693009795978448,
+            -0.019725315346836404,
+            0.00998768979020729,
+            -1.1702210281215188e-06,
+            -0.011129745730507545,
+            -0.002591293237053776,
+            -0.018034357254990988,
+            0.00012070821244376275,
+            -0.023051045150566527,
+            -0.028465270110388716,
+            -0.00048603008831669926,
+            -0.000574546300981984,
+            -0.012992623897880395,
+            0.02064747165046032,
+            0.014820638761258362,
+            -0.024820174379739057,
+            0.02666518374464728,
+            -0.017430076811109155,
+            -0.01495051200726725,
+            -0.005861181182847845,
+            0.00100881727574559,
+            -0.0055223282819321295,
+            0.0034719859452138017,
+            -0.010180752654175722,
+            -0.009961678082468036,
+            -0.012134423047419183,
+            -0.028073672819317354,
+            0.011574683159318163,
+            0.016120674453124317,
+            -0.015043147648505721,
+            0.031834693135959095,
+            -0.008802805725050891,
+            -0.0018698230152007877,
+            0.016179742266521146,
+            0.005096232332051617,
+            -0.017187077376166815,
+            -0.018999569809271876,
+            0.012251308792871571,
+            0.013279090498325945,
+            -0.009588297039279112,
+            0.006630431200804809,
+            -0.014130176783207386,
+            -0.005092662672039885,
+            0.007020236623084732,
+            -0.005049363863784624,
+            0.0055051712211635915,
+            -0.00995314895304919,
+            0.0052743456988306485,
+            0.005841370337169303,
+            -0.0029434616355351975,
+            0.004677049987338193,
+            -0.018568941264514256,
+            0.020031253265629777,
+            0.0018725390936655962,
+            0.01332532735283085,
+            -0.03518855189467458,
+            -0.0056298723325832025,
+            -0.005369475318146738,
+            -0.004650364775945294,
+            -0.020833510091412353,
+            0.020445280915606132,
+            0.0159991245532085,
+            0.013202409908533162,
+            0.023094507005426774,
+            -0.014905666713082246,
+            0.01572052456975675,
+            -0.020945766915121455,
+            0.013114000985348841,
+            0.003349101658930482,
+            -0.019829605346106578,
+            -0.001423750749031516,
+            0.04557459007627153,
+            -0.005847231973329081,
+            0.011986110554869603,
+            0.0012709842140249691,
+            0.026758922819943853,
+            -0.0001635536762342011,
+            0.0065419519929442995,
+            0.005997692702992206,
+            -0.009935128108338126,
+            -0.0008057798606777003,
+            0.00865712443881613,
+            0.046254320605750854,
+            -0.01181132866321511,
+            0.008170688314239364,
+            0.013099051984057827,
+            0.0034971494510865624,
+            -0.0008386400616584293,
+            -0.015346506644390764,
+            -0.03067678541405639,
+            0.019305962963428017,
+            0.007736916644077731,
+            -0.012756525606221703,
+            0.008734673666457216,
+            0.015137771675476126,
+            0.012891824153867332,
+            0.003895886821685762,
+            0.02155154331885366,
+            -0.012761690789748027,
+            0.018528313792219413,
+            0.024663453412919027,
+            0.009077230086690327,
+            9.184526542175535e-05,
+            -0.000751529849753638,
+            0.006791927267880182,
+            -0.0004433479107716552,
+            -0.009832230053513483,
+            0.015858232096630655,
+            -0.014861341148074698,
+            0.017790924034108806,
+            2.7216806238946785e-05,
+            0.03744841066552559,
+            -0.015216207585985075,
+            -0.010138319613387346,
+            -0.007774325714088762,
+            -0.0016997034352120362,
+            0.003971564314593291,
+            0.0018686537276849653,
+            0.006051669054361943,
+            -0.0008319624938748146,
+            -0.017326561560776794,
+            0.01667737942301944,
+            0.005719808546053257,
+            -0.0007405637872110778,
+            0.013750752065593804,
+            0.0038732323799683453,
+            -0.00367645612814479,
+            0.023580906347714314,
+            0.006238541579764762,
+            -0.005235795826199542,
+            0.0028182323816636243,
+            -0.006443210286549713,
+            -0.016255209299456314,
+            -0.003061441395209219,
+            0.004597970789372988,
+            -0.0016590920570710628,
+            -0.011138261849069419,
+            -0.0019423173609729848,
+            0.020355534503272887,
+            -0.004612410336124759,
+            0.013346750536413418,
+            -0.005400060374036387,
+            0.006252154576727367,
+            0.010601459925435238,
+            -0.012346569079682625,
+            0.0014154620309235822,
+            0.010406076751218612,
+            0.0013983417564564663,
+            -0.007140180075225315,
+            0.0010544410919327675,
+            0.007757599672903709,
+            0.006505350216372801,
+            0.00483063937962671,
+            0.00836630436955631,
+            0.013284451514874498,
+            0.0033749897293287438,
+            -0.006243293993633622,
+            0.0013949831763016042,
+            -0.013269921385800992,
+            -0.006308639321247727,
+            0.0010906169976769569,
+            -0.0010152835709869353,
+            -0.02017142197515583,
+            0.010787630901414118,
+            0.005355784839945777,
+            0.007838485470991034,
+            -0.01706932891696025,
+            0.009576030158827565,
+            -0.0008144311254998525,
+            0.02829832828245172,
+            0.007407953684956539,
+            -0.02769403281764041,
+            0.014266001222243134,
+            -0.010492199740754951,
+            0.022575782293320284,
+            -0.0016349316593207844,
+            -0.005656505606143177,
+            0.008025514398322551,
+            -0.009634281579228592,
+            -0.005991026158213869,
+            -0.022366083313133788,
+            -0.025825209755160946,
+            0.016871087849192924,
+            0.013583173973358155,
+            0.021125252303971605,
+            0.00012555627355712715,
+            0.015347535950809997,
+            0.0020128011217859444,
+            0.015164099887550086,
+            0.0024544903332736167,
+            0.012507136277293888,
+            -0.011798050869794805,
+            0.0104828761281283,
+            -0.0137805761114781,
+            0.014620357150314097,
+            -0.005246639228547017,
+            -0.006986408834884282,
+            -0.008849580196616098,
+            -0.005442165099441538,
+            -0.020891551850452786,
+            -0.009155930286949747,
+            -0.0013590151948670868,
+            0.010665872513236615,
+            -0.0005386423407222223,
+            -0.0027671778465775897,
+            -0.03237311596935737,
+            -0.004875062533405715,
+            -0.006670982068843501,
+            -0.01942244110096111,
+            -0.0045975373348006885,
+            -0.026137783023267008,
+            -0.01657173843643047,
+            -0.0007784046498232684,
+            -0.015037306676062236,
+            -0.006889680260149007,
+            -0.02009705080543122,
+            -0.027192867101205106,
+            3.0161964469866572e-05,
+            -0.009647644858034833,
+            -0.005380165957043753,
+            0.013815092649296373,
+            0.011902366689169847,
+            -0.01572592170758115,
+            -0.022321702398924333,
+            -0.01088937403483412,
+            -0.0165754067023749,
+            0.01646435743868707,
+            0.009914938007210463,
+            0.018834863207293908,
+            -0.003724245912768107,
+            0.005643505694722323,
+            0.0009581443359000331,
+            0.02327704258596552,
+            -0.02532422120223976,
+            -0.028694220894404807,
+            0.0019298778426469338,
+            0.0025970175688368323,
+            0.006086164866310174,
+            -0.009414665394188936,
+            0.001200390869257535,
+            -0.007782899559317819,
+            -0.0228264209954853,
+            -0.00898519580978142,
+            0.008156835270668449,
+            -0.010036630164522355,
+            0.02230664068547531,
+            0.012332230358869292,
+            0.013699197379189934,
+            0.010593587938184292,
+            0.01576174456497367,
+            -0.0001654241831584687,
+            0.014471905257729428,
+            -0.008675915624727887,
+            -0.0031070346099204425,
+            0.0038358404385932777,
+            0.005878417037187274,
+            -0.00625022998235275,
+            0.028077114529659795,
+            -0.0011836453716817525,
+            0.0002962695480296182,
+            0.014405782572641941,
+            0.02697667559474498,
+            -0.004710313427099278,
+            0.0210512043261309,
+            -0.002393165025068831,
+            -0.006135816286212752,
+            -0.008429776018235104,
+            0.008607617299702318,
+            -0.009529291136733401,
+            0.011133497451942684,
+            0.019864404836361926,
+            -0.014041315457579166,
+            -0.007351413712676263,
+            0.02950254974910368,
+            -0.014564388091367897,
+            -0.01825317106331622,
+            0.02325964952838705,
+            -0.007414295676409368,
+            -0.0039042587057605484,
+            0.00377144546378926,
+            0.007169856503233239,
+            -0.004036744966949995,
+            -0.012928341573802416,
+            0.016090527447479415,
+            -0.00011159893712484579,
+            0.014146137810890391,
+            -0.0004655839804281017,
+            -0.006372231942983921,
+            -0.010047382825720416,
+            0.002373224247019062,
+            0.026818813126567614,
+            0.013483627627077692,
+            -0.017895710143841333,
+            -0.013785856764479817,
+            0.010826610341182479,
+            -0.01538893818197323,
+            -0.004425206001945227,
+            0.02155595194812746,
+            0.0028768153668128553,
+            -0.020543027467698483,
+            0.008028975884363618,
+            0.0003835165718138906,
+            0.00037046947047892953,
+            -0.00029363296913641716,
+            0.030135739208689217,
+            0.001343516577762509,
+            0.00018948658255495327,
+            -0.00875053830853609,
+            0.008606573235959595,
+            -0.01280305089741873,
+            0.012786478836532904,
+            -0.02437993139337839,
+            0.007648181775166282,
+            -0.00629541989541625,
+            0.004579605315599212,
+            0.016658386927051592,
+            0.01218098627022323,
+            0.015031618813778413,
+            0.008237200519537417,
+            -0.008942965899756683,
+            0.0009093265012901892,
+            -0.025302175262520477,
+            -0.001333639155622843,
+            -0.017971876771137367,
+            -0.02036433247634014,
+            0.0002463842837918767,
+            0.022144712689630985,
+            0.018559482073124734,
+            0.011579840028464503,
+            -0.008657014440886656,
+            0.0166696911919226,
+            -2.9834620942882283e-05,
+            0.011845827764518911,
+            -0.00037901176785495324,
+            -0.006138374154553732,
+            0.004711058953619512,
+            -0.003622694588720417,
+            0.013542405259906128,
+            -0.014730567462694827,
+            0.012076044672004906,
+            0.01690223095263866,
+            -0.009108632851807137,
+            -0.008946251274440141,
+            -0.013992423874475428,
+            0.00048381889145447367,
+            -0.015909599902931202,
+            0.035489252674454214,
+            0.02252772349379229,
+            -0.01210464970923921,
+            -0.010469120693943236,
+            -0.014164999313775756,
+            0.019496048806752753,
+            0.010427539351244087,
+            0.0028018908451425276,
+            0.017091768406027965,
+            0.021395638416767112,
+            0.008817247593404623,
+            -0.006081063711583319,
+            -0.0021216503501144983,
+            -0.003781810594530787,
+            0.020143098218482827,
+            -0.003078172339607037,
+            -0.011350171884507555,
+            -0.012516379645909518,
+            0.005766102930704529,
+            -0.01626433668691071,
+            0.0019294643928566657,
+            0.008641558521764407,
+            -0.015057688533773374,
+            0.019407986696657772,
+            -0.028126197703647868,
+            0.012364374199657305,
+            -0.0044278434183731225,
+            -0.007251147925369614,
+            -0.03496220945653279,
+            0.006389489026974697,
+            0.008941259259632024,
+            0.003941752277454314,
+            0.023003838655784314,
+            -0.024528514847470928,
+            -0.018361931455105486,
+            0.0037791061321962543,
+            0.00047162060848090104,
+            -0.00711894577052754,
+            0.0010138871688747137,
+            0.004128392769213672,
+            -0.00674546985015164,
+            0.0017307097443508704,
+            0.0033354800298095546,
+            -0.002829898077850263,
+            0.013252947102333348,
+            0.0081724269619999,
+            -0.021449979263734438,
+            0.001368791856860285,
+            0.01590656734732547,
+            0.01452008402429858,
+            -0.02391847163018108,
+            0.006331397860689212,
+            -0.02064812934794864,
+            -0.000631340242718475,
+            -0.031272866000888556,
+            0.016859793097564958,
+            -0.0069008624795683276,
+            0.01899718801899956,
+            -0.0015377436942771224,
+            -0.0065810685260109004,
+            -0.012822620655323715,
+            0.02137604937338179,
+            -0.0019121137078344135,
+            -0.0025598772127157477,
+            0.015678293057066646,
+            -0.025700865215693987,
+            0.017825186616033165,
+            -0.0026230848396158204,
+            -0.017342967698019966,
+            0.017199139867162275,
+            0.004536019178633225,
+            0.007224666811309087,
+            -0.023505589665315567,
+            0.0007208041198255396,
+            0.003900164485347562,
+            -0.017250324025622685,
+            0.004470862192961212,
+            -0.030807400122398566,
+            0.0006579147990327597,
+            -0.013222907940559794,
+            -0.0020514994691204847,
+            -0.0046763235724575445,
+            -0.014050035576183385,
+            -0.039181147615786484,
+            -0.005429639279537522,
+            0.016065742005148733,
+            -0.0009029636981539927,
+            0.01800715198737081,
+            -0.002730931639143803,
+            0.002947680833843522,
+            0.004912692811020822,
+            0.00818378920043427,
+            0.006915895602313347,
+            0.02274116906914758,
+            -0.0057854922886970365,
+            0.00471999900678522,
+            0.007058014277137233,
+            0.014638081356210031,
+            -0.01719565951201012,
+            -0.015170377598372723,
+            -0.016015921063369496,
+            -0.006351213642269936,
+            0.01739665344303718,
+            -0.017488885259028475,
+            0.013303332445465686,
+            -0.00429428898520701,
+            0.014825338166443021,
+            -0.008686427208690176,
+            -0.01575945086724703,
+            0.0028564767084934386,
+            -0.007570417028925067,
+            0.001007670647768344,
+            -0.02323908008484089,
+            0.02019729564706863,
+            -0.00847581038203716,
+            0.010508144473197669,
+            0.01729824052022313,
+            -0.021364529179605813,
+            -0.009742851959432839,
+            -0.014342281441152413,
+            0.029896210049029176,
+            0.013925163184699139,
+            -0.009392252140772913,
+            0.003927230832659267,
+            0.008873561269230905,
+            0.011529536139797032,
+            -0.012209070214071445,
+            -0.0027679974125029362,
+            0.002411914349172495,
+            -0.0062056207925691356,
+            -0.002025687569076897,
+            0.03090134105002816,
+            0.0038285457265343245,
+            0.024877474647546086,
+            -0.013214997316123852,
+            -0.0033763842539861907,
+            0.00023379106197363714,
+            0.005134764372494098,
+            0.01183867975321486,
+            -0.0017966997954747142,
+            -0.008029791503028367,
+            -0.01179500476055557,
+            0.02599911330072515,
+            -0.021852083993691557,
+            0.007024505421035273,
+            0.026784024996764876,
+            0.001390202993990949,
+            0.0029956956045771046,
+            -0.013131691460546865,
+            -0.006298736864509989,
+            0.02523640664202355,
+            0.010030824059858138,
+            0.0041523782603032225,
+            -0.026465819482580182,
+            0.0032128899272377593,
+            -0.0061901409332948115,
+            -0.008681758126710213,
+            0.009866461769956222,
+            -0.0028095454857224164,
+            0.01188012041754054,
+            -0.012191996383095982,
+            0.028459939543805705,
+            0.010711502132045176,
+            0.01047154118931583,
+            0.025888652452698974,
+            -0.02079614305315134,
+            0.0042203106489994615,
+            -0.010829796140534066,
+            0.0006773493086316061,
+            0.005596640511307504,
+            -0.027510562413905437,
+            0.001969898603871003,
+            0.010390866563404346,
+            -0.005642706366801436,
+            -0.01882994179544761,
+            0.01187970346127799,
+            -0.019814385265412387,
+            0.024479579275617887,
+            0.003231341626501863,
+            0.020056914356101588,
+            -0.0053276448188067154,
+            0.009834685132275868,
+            0.0002742942778577165,
+            0.0031361331890882127,
+            -0.0008074598092128465,
+            0.009009012303347016,
+            0.005786777410144894,
+            0.013719989127068968,
+            -0.025528370602188686,
+            0.006821572597883499,
+            -0.0029197127345352626,
+            -0.030539872624857384,
+            -0.013383097680839732,
+            0.010291485622655026,
+            0.012218533188117704,
+            -0.009144425807890594,
+            -0.010378679518611277,
+            0.0030042579827305603,
+            0.005748243116238633,
+            -0.006692403885450407,
+            0.006913672151000981,
+            0.01629516912493545,
+            -0.002533517074771728,
+            -0.0019083178803585312,
+            -0.0035007340321378893,
+            -0.01154609935927537,
+            -0.010016998439072824,
+            0.023193275321990937,
+            -0.00482990342915347,
+            0.017943025772239603,
+            0.025873224580389702,
+            0.0039876567755322,
+            -0.0057577932736143506,
+            -0.005460363755206188,
+            0.01478435999841998,
+            0.014195457012279476,
+            0.004669295894868759,
+            -0.009702220679239768,
+            0.007974305230005956,
+            -0.007760086293456636,
+            0.008375077126777163,
+            -0.020260917110786147,
+            0.002199401871514877,
+            0.0026191373298767766,
+            -0.0022089262577237776,
+            -0.015213807046175918,
+            -0.01956788452420682,
+            -0.021898368819136302,
+            0.012634659463647929,
+            0.00034643102239488474,
+            -0.009241772775582378,
+            -0.013645250997151429,
+            0.0016665106990117016,
+            0.0025630500346914592,
+            0.0002095521560180936,
+            -0.007783022628694597,
+            0.009546147813008391,
+            -0.01265287317178631,
+            0.007445223311736028,
+            -0.004007029181436962,
+            0.0051846833906574515,
+            -0.0016916014415925766,
+            -0.016413938859402438,
+            -0.0210868129634149,
+            0.01920794362224444,
+            -0.012897042227841011,
+            0.02269303737601583,
+            -0.0014935878147854765,
+            3.062680855559304e-05,
+            0.0033132626755401546,
+            0.007274487651476732,
+            0.012110042834251237,
+            0.011426424896797977,
+            0.004307086698938411,
+            0.008206932254416481,
+            0.004673427176334148,
+            -0.013124571552764586,
+            0.004324963759928715,
+            0.0039216033705804084,
+            -0.010449887922072297,
+            0.005298323150690876,
+            -0.0010196552573491,
+            0.006128327172621373,
+            0.012167492341907742,
+            0.01923525204293908,
+            0.01908107550907106,
+            -0.0012031208519043073,
+            -0.0010601912677170903,
+            0.014474374858125607,
+            0.004540954805526089,
+            -0.004883537463424679,
+            -0.002772988629590886,
+            -0.013363713078069854,
+            -0.002572403646338979,
+            0.011541515690646252,
+            -0.02376161747414617,
+            -0.018215231053002656,
+            0.021831219299143226,
+            -0.004991882316041097,
+            0.033796174834149546,
+            -0.013140970283091536,
+            -0.016197177724633614,
+            -0.00464190076219332,
+            0.0026389653040644607,
+            -0.013788049032378498,
+            -0.010300859744205445,
+            0.010545541278382187,
+            0.013848738866656793,
+            0.009903590151120531,
+            -0.0005984085581408575,
+            0.02922816576443645,
+            0.0019900932217491907,
+            0.02245395592422119,
+            0.0018978407151245295,
+            -0.007029294098689854,
+            -0.01440575120349887,
+            0.010840472832356266,
+            0.005529335652955532,
+            0.009050748012161991,
+            0.017791269808484802,
+            0.014502057454389247,
+            0.006265567405788131,
+            0.013928705812292101,
+            0.0012903002372833693,
+            0.0010858377120770652,
+            -0.002355229463060164,
+            -0.01692711767136341,
+            -0.01456786846097011,
+            -0.01627640642218236,
+            0.003873864312554098,
+            0.012620350492241247,
+            0.00525890570023481,
+            0.0007393789113866752,
+            -0.007153210751899716,
+            0.009254566395058412,
+            0.0065087881512835375,
+            -0.012279152412209152,
+            -0.016530068283509593,
+            0.004060671955726967,
+            -0.02210570272789274,
+            0.007849467899009437,
+            0.01554233481902349,
+            0.00219885032769439,
+            -0.002655449524219968,
+            0.0006843524304842362,
+            0.006272944169764281,
+            0.005531927368102191,
+            -0.0036710328198476905,
+            -0.00544113514775027,
+            0.011264661763762068,
+            -0.006379654234440906,
+            0.009537150372938296,
+            -0.00290031548037575,
+            0.03718190949929003,
+            -0.011976080887998194,
+            0.012120408392610773,
+            -0.001751472085435711,
+            0.008481703962951994,
+            0.008067669857949423,
+            0.010772687289903202,
+            -0.003805089040741351,
+            -0.0022164543131735167,
+            -0.008753625032781523,
+            0.006751953442322185,
+            -0.01582177621255226,
+            0.013631301238130705,
+            -0.00030685726238985963,
+            -0.010728958775974582,
+            0.01055472059121978,
+            0.006788316621149835,
+            0.004413382316882087,
+            -0.003262024961860459,
+            -0.01972283203505511,
+            0.014937729522976615,
+            0.0018799533869990667,
+            0.019247008681353865,
+            -0.023276106189004046,
+            0.029043727846427652,
+            -0.02019024213343764,
+            0.023083643395685142,
+            0.01865875504942157,
+            0.001179436167200093,
+            -0.02070329911663634,
+            -0.007618131527214554,
+            0.014871709122115308,
+            -0.02659818889735726,
+            -0.006234719259825804,
+            0.005986829551040766,
+            -0.014639912579831785,
+            0.00037813423546252645,
+            -0.01924149909676064,
+            0.00010832498610532006,
+            0.011013130149723431,
+            -0.011782407678373089,
+            0.00438725832178584,
+            0.01608349750921335,
+            0.019347883652009713,
+            -0.009760017612545174,
+            0.003869246694835168,
+            -0.024240324890794748,
+            0.010958789926998691,
+            0.020511684636649225,
+            0.008883830392748497,
+            -0.01761944522796127,
+            0.021817973960311446,
+            0.010652131228027521,
+            -0.012575679152857703,
+            -0.0012753095453828826,
+            -0.028191274775392804,
+            -0.012421170815054556,
+            -0.02188248722244861,
+            -0.0011670175621882146,
+            -0.014436683528976771,
+            -0.0029783880907228383,
+            0.0009506986717516226,
+            -0.004719165363484719,
+            -0.01179163390107033,
+            -0.019543732196676634,
+            0.016606805875498898,
+            0.0227313413004497,
+            -0.006738735201626204,
+            -0.014547240981002825,
+            0.013866974327185123,
+            0.031694077725890536,
+            -0.016189209297529324,
+            -0.00542110554672736,
+            0.0028061798782555557,
+            -0.012964876141500577,
+            0.018946046750284537,
+            0.0039419760434645495,
+            0.000322115192590907,
+            -0.01991851369210666,
+            0.002489725351392778,
+            0.010835458511052207,
+            0.01684964045708685,
+            0.0005541314524621196,
+            0.010855852688653808,
+            -0.007549882748861744,
+            0.010096800951420057,
+            -0.011832260836468588,
+            -3.6179995603517093e-06,
+            -0.007800366796516945,
+            0.007001049318404058,
+            -0.004424776169203575,
+            0.034356203288485945,
+            -0.02817071276766299,
+            0.004989631403681216,
+            -0.014539644354369607,
+            -0.010888763939716889,
+            -0.0027322086509797843,
+            0.0004865534340981866,
+            -0.010775238981581461,
+            -0.005162220412866082,
+            -0.0011263271110752205,
+            0.00913081242511139,
+            0.005844327320514489,
+            -0.013886529519306775,
+            0.019171827605726503,
+            -0.030984919318656618,
+            0.004469458202753611,
+            -0.008553175891755315,
+            0.010511707988936336,
+            -0.0151185694875566,
+            0.013830711686954931,
+            0.01301959345025159,
+            0.008205984085944315,
+            0.007740367849216328,
+            -0.01532123052721834,
+            -0.01678314222392063,
+            0.001376468747213867,
+            0.0036953018347776244,
+            -0.00931855481896886,
+            -0.033391028922085425,
+            -0.007072913446166969,
+            0.005331350534425606,
+            0.011614453225043,
+            0.008156128603003278,
+            -0.02345206873048703,
+            0.005003870024809975,
+            -0.024178720646704597,
+            -0.012115112684614331,
+            -0.007750557665424973,
+            -0.011580152272614108,
+            -0.006117298954192534,
+            0.026449765548206453,
+            0.007989237291215073,
+            0.016691517078382376,
+            0.004954133565782356,
+            0.02742809127725913,
+            0.011670893621991404,
+            -0.011736840457474895,
+            -0.003349151747880166,
+            0.00035419034843196817,
+            0.005507651806218584,
+            -0.008706590052024251,
+            0.011793241980257321,
+            -0.0013295035978611142,
+            -0.0065061682758755625,
+            -0.004398354347597199,
+            0.009372816666628725,
+            0.014188676762501037,
+            -0.003007980559117429,
+            -0.008724255229333351,
+            0.015749776067989826,
+            -0.030899793617096834,
+            -0.015215033379100091,
+            0.0016152308157299517,
+            -0.007813175908301852,
+            0.017806770680528907,
+            0.01169056345888971,
+            -0.026835412610223673,
+            -0.004538235170332766,
+            0.010532316940606419,
+            0.004755930024691789,
+            0.0095408889025506,
+            0.00954782703036282,
+            0.012079064562163901,
+            -0.007829549014214643,
+            0.01046722658021748,
+            -0.011519369840999348,
+            -0.007092961453184232,
+            -0.007241799314569802,
+            -0.008077934089061052,
+            0.021467547268987214,
+            -0.025816030568238717,
+            -0.006558853543772633,
+            0.01134670818701549,
+            0.013027491377727365,
+            -0.002301991242932548,
+            -0.006299073927206214,
+            -0.017357989759336386,
+            0.008311464809363397,
+            -0.013922568511594478,
+            0.025633907424998367,
+            -0.004854655498817219,
+            -0.005336807899100369,
+            -0.005228056880188717,
+            0.01958682721272445,
+            -0.016774409737540904,
+            0.02072573220949266,
+            -0.014449250679704904,
+            -0.007857351393427215,
+            0.001673929749689194,
+            0.006890008607998747,
+            -0.002948835017234416,
+            0.03734128725793149,
+            -0.00556474133167756,
+            -0.0014544201137417158,
+            0.02209084868209573,
+            -0.01883610080898967,
+            0.017901421953179442,
+            0.01563604483998435,
+            0.012327159100148019,
+            0.0018161034827768648,
+            -0.010253034391420751,
+            0.019979728662282904,
+            0.020823925491027635,
+            0.0139490143219451,
+            0.009577521724445478,
+            -0.013478761809038993,
+            0.014934933784421556,
+            -0.0003725422088504494,
+            -0.0024362115214204105,
+            0.012785534912074068,
+            0.016441944143888537,
+            -0.02164522371903646,
+            -0.013152102018736346,
+            0.0063548333449661805,
+            -0.016547629104776748,
+            -0.024875443975589356,
+            -0.008991095813132973,
+            -0.021885127280977718,
+            -0.023336714382596175,
+            0.00688008599534903,
+            0.016482610587564443,
+            0.0035797836729533367,
+            0.013111012464332918,
+            0.006820471349350837,
+            0.018274678506489214,
+            -0.00980248993077149,
+            -0.01582838461385605,
+            -0.0009454586621769173,
+            -0.0004906493597104733,
+            0.009787688665732045,
+            -0.01835583040075359,
+            -0.013977939077760832,
+            0.004340902272034388,
+            -0.020118484730757877,
+            0.01421736173162507,
+            -0.006428386730058143,
+            0.016756376129345597,
+            -0.0004630315831872392,
+            0.011701948921076007,
+            0.02621773278421472,
+            -0.001391567406453007,
+            0.01262026995055899,
+            0.004696906079976175,
+            -0.0030577364576645635,
+            -0.010624066224426196,
+            0.0009420498021845824,
+            -0.0044079288776429516,
+            -0.0004845606666642234,
+            0.006481113327122954,
+            0.020269778435314,
+            -0.03321232649491074,
+            -0.009425498978427855,
+            0.01246754422779043,
+            -0.001658574527323383,
+            -0.005816985985181387,
+            0.013050062223188965,
+            0.00011195028631856881,
+            0.016573827662919836,
+            -0.0017453203044868104,
+            0.01417371165305189,
+            -0.008165910519539772,
+            -0.0007461950113024975,
+            0.0016492309267666983,
+            0.005076015595439873,
+            0.005829299318055953,
+            0.012146800191035666,
+            0.01148884639506138,
+            -0.021051327863689618,
+            -0.0026357096198729156,
+            0.0036146148489310443,
+            -0.010673282438301611,
+            0.014967444393862347,
+            0.005737933867764028,
+            0.01391411712868596,
+            0.0029556189893790237,
+            0.01167806258173173,
+            -0.024440974650111174,
+            -0.01029944007931845,
+            0.01385509328958837,
+            0.00869671728965227,
+            -0.00998315356917989,
+            -0.016580053751364735,
+            -0.007061005072623325,
+            -0.00477175200214453,
+            -0.0021182980051220428,
+            -0.009871524928312118,
+            -0.0005933941177237234,
+            0.006284920058476413,
+            5.466820734762947e-05,
+            -0.012842295882002122,
+            0.015810530885294202,
+            0.004572385791133295,
+            0.003685966629168175,
+            0.005093714133656017,
+            0.011877541830396259,
+            -0.013130170365831912,
+            0.01258190257102377,
+            0.011433121961378686,
+            0.01142665981860032,
+            -0.013710025169815697,
+            -0.001774030746825879,
+            -0.01109881340948764,
+            0.0065684584345163735,
+            0.009983753894796107,
+            0.005346443477816739,
+            -0.002062387245572686,
+            0.004950044847352647,
+            0.006105199826576752,
+            -0.06391185159544624,
+            -0.02131511418723256,
+            0.0027165737784844413,
+            0.033389688791557785,
+            -0.007167593552251769,
+            -0.0037646833460996797,
+            0.002883510089742145,
+            0.01356887065340832,
+            0.009730171586767032,
+            0.00418566245984083,
+            0.020320934773473808,
+            -0.008814826860651366,
+            -0.003014193747732543,
+            0.001559573791453607,
+            0.003499420226345028,
+            -0.008703837291821111,
+            0.0044844701302276645,
+            0.0053963630238108035,
+            0.027764369548110287,
+            0.001469968540767627,
+            0.015793086297019008,
+            -0.008505290826588508,
+            -0.004075888558568453,
+            -0.009166710762668385,
+            -0.002860047211143503,
+            -0.011127820563172684,
+            0.003937110717404271,
+            0.02054186721680431,
+            -0.006126180963243227,
+            0.009346433819831456,
+            0.01057504086030076,
+            0.0001650851447316893,
+            -0.012033423322524951,
+            0.0049587335923558,
+            -0.0032240390628760535,
+            -0.005568976687946761,
+            -0.0024667040429312737,
+            -0.03474027725033016,
+            0.026855947684394643,
+            -0.004369626674503285,
+            0.009252512430957932,
+            -0.005081834388513357,
+            -0.0037896610731020337,
+            -0.014201609270756739,
+            0.017871088029615106,
+            -0.008249444723433136,
+            -0.020092257756971717,
+            0.009532994544485572,
+            -0.004991989149115383,
+            0.006464417738670472,
+            -0.02643078108339741,
+            0.0029138922065812495,
+            -0.017232410749655366,
+            -0.003999293090834831,
+            -0.006924488971625995,
+            0.01318082244178942,
+            -0.007232695119856389,
+            -0.02660711037908311,
+            0.01979251106803667,
+            -0.019485860712345603,
+            -0.011977904020170568,
+            -0.0177530973032872,
+            -0.014524003566945937,
+            0.004874952064727108,
+            -0.015326044325348929,
+            0.002047771741041311,
+            -0.009697340332342246,
+            -0.003652304998155628,
+            -0.0008878698629674222,
+            0.0017782435347417623,
+            -0.013017776298713044,
+            -0.018272697230002275,
+            -0.01353020425912225,
+            -0.00757239449706798,
+            -0.0008250000619957538,
+            0.01299142281772714,
+            -0.001679606555140765,
+            -0.007329941421303076,
+            0.0044605462460153116,
+            0.010231192277666706,
+            0.0006558644644452857,
+            0.0024231733856020414,
+            -0.010728844680699885,
+            -0.014137434553169443,
+            0.005230053181525096,
+            -0.012086468475542818,
+            -0.009459717916037003,
+            -0.0028185227600210556,
+            0.014984527039312847,
+            0.008189464334213754,
+            -0.009139240774931914,
+            -0.005639157579163322,
+            0.014325949891859377,
+            0.009020303519690971,
+            -0.013310205617445609,
+            0.01567574314514385,
+            -0.025369802629268523,
+            0.021964786021408878,
+            0.030536905527039065,
+            0.004945288795196981,
+            -0.01168303908203322,
+            0.02281128628230144,
+            -0.004072286832145419,
+            0.0028278384348551146,
+            0.01740522162288968,
+            -0.042558465579514926,
+            -0.013146562547084278,
+            0.002903743316339479,
+            -0.015297177099602701,
+            -0.010963312697387696,
+            0.008804938803398296,
+            -0.0026727734429658817,
+            0.005273238046953793,
+            0.004257613402172282,
+            0.03512246556126453,
+            0.002258612643297096,
+            -0.0002885708728599208,
+            -0.0019720190285930487,
+            0.012312613129996533,
+            0.010451003939611476,
+            -0.015556620910136185,
+            -0.02387856865043814,
+            -0.004583090523619507,
+            -0.0037828776396525835,
+            -0.0014588238692793338,
+            -0.004361834167147271,
+            -0.008159679676329573,
+            0.012243319920992457,
+            -0.017279964797769668,
+            -0.013748032758693542,
+            0.018394195628121487,
+            -0.006411113522842359,
+            0.019002611495040334,
+            -0.014970362556522149,
+            -0.0077437581301661946,
+            0.0017311115715410025,
+            0.024439691379769795,
+            -0.025013956798520212,
+            0.007359134924497539,
+            -0.0031406470488477937,
+            0.017574824561755627,
+            0.00419409250132525,
+            -0.0046803792187546656,
+            0.010556972371044426,
+            0.015229789583480183,
+            -0.004608293492221483,
+            -0.014449487985966966,
+            -0.01172850291078532,
+            0.016823024209386785,
+            -0.012304887597699588,
+            0.0011773351576246157,
+            0.002262670226244277,
+            -0.0023810578015456716,
+            0.03477119000959688,
+            -0.03442178291363675,
+            0.004062693040714608,
+            0.0029249583507755944,
+            0.003291349443702718,
+            0.006461525089528308,
+            0.006047777261425826,
+            -0.020642802018978404,
+            0.008916088381876748,
+            -0.03316762661930299,
+            -0.006367254689957291,
+            -0.005404276207844267,
+            0.0009189006423526284,
+            0.009401330264376041,
+            -0.011106443887384533,
+            -0.02259405372588916,
+            -0.006010439054323966,
+            0.0070744551115721095,
+            0.010262607009446879,
+            -0.002856616377476421,
+            0.019719451109535182,
+            0.008426509638139982,
+            0.0015877648339862691,
+            -0.010655792508751512,
+            -0.015527851374834499,
+            -0.015460974859638219,
+            -0.017847462632188386,
+            -0.02236964311096428,
+            0.011841749854626384,
+            -0.00576688153869993,
+            0.009054902130717224,
+            0.014317707346504314,
+            -0.015366657284925976,
+            0.002910211008094564,
+            0.017761685988205086,
+            0.007974806171830457,
+            0.002692237107055498,
+            0.0032405477366987947,
+            0.02685230091638845,
+            -0.0009116884421301454,
+            -0.0032228977180251673,
+            0.003158213042682977,
+            0.015872199488167846,
+            0.0004966905858799325,
+            0.019981401695749582,
+            0.015201538351780685,
+            0.005359075814444414,
+            -0.004668738193285921,
+            0.019853988551405383,
+            -0.009844394286458195,
+            0.013476144827747423,
+            -0.00411789677405284,
+            -0.00830656434595337,
+            -0.008496132520586954,
+            0.006442581751999304,
+            -0.008138786054348571,
+            0.006977337946948795,
+            0.02419128791801387,
+            -0.0005734827808485311,
+            -0.0028842142002271044,
+            0.005503033191779872,
+            -0.01696639779775455,
+            -0.00389835761601045,
+            -0.03366077608692946,
+            -0.009942613075976742,
+            -0.01708630689497001,
+            -0.02910756562213045,
+            0.026536268897199546,
+            0.00219827495662445,
+            0.02586503862171508,
+            -0.01905122645609761,
+            0.008935637454859214,
+            -0.01385806889332683,
+            0.029747093931541325,
+            0.009163959012862627,
+            0.0008611776492791918,
+            0.003598831885031635,
+            -0.02103045281939351,
+            -0.01150941825152158,
+            0.027341774725604333,
+            -0.0006700608710615124,
+            -0.012089044078073347,
+            0.022396400579038037,
+            0.017491565611764764,
+            0.0051760367056280536,
+            -0.00559536785318233,
+            0.01979262595019399,
+            0.002306380546194741,
+            0.011973863166872444,
+            -0.0028136750606972223,
+            0.012911235736067303,
+            -0.01510421118359169,
+            -0.003338602157155174,
+            0.039758808572572274,
+            0.007290642519242007,
+            -0.001802921446521437,
+            -0.013696774990705774,
+            0.003941198181491921,
+            -0.0008308293948698925,
+            0.0200351513103389,
+            -0.00014327587644797734,
+            0.028291423013030337,
+            -0.0001085859276231927,
+            -0.010723064585989723,
+            0.020443848393732245,
+            -0.017860296930396913,
+            0.04540901629084378,
+            -0.018671578423261854,
+            0.000852031295629291,
+            -0.010639853151638835,
+            -0.019017194262730474,
+            0.003861296738239369,
+            0.0010977143012920602,
+            0.0008387482016061222,
+            0.0060155816314618,
+            -0.010780239332444652,
+            0.002152214497259421,
+            0.010556338812873054,
+            -0.027636192450621728,
+            0.0010938992111792307,
+            -0.014385202899929138,
+            -0.01353192238925814,
+            -0.026667778226211785,
+            -0.00017403398236549974,
+            -0.015357674131635848,
+            -0.013956722505869091,
+            -0.01985038526848704,
+            -0.008927903879720431,
+            -0.009580214158077466,
+            -0.0020314441852913386,
+            -0.004755584878514854,
+            -0.00018388548107372635,
+            -0.002918855567842974,
+            -0.0015941852262316128,
+            -0.014230272061190995,
+            0.02133992212037431,
+            0.015283345362445234,
+            0.004642765060980717,
+            0.012829570028541471,
+            0.01708952142208596,
+            -0.004770647543823141,
+            0.006604762788847952,
+            -0.0025369836845081078,
+            0.008368000846861623,
+            0.014785031726639733,
+            0.012074738887486398,
+            0.02124518119032885,
+            -0.015459818988572564,
+            -0.004992865135357168,
+            -0.012722391036055767,
+            -0.012210202603282509,
+            -0.017850065946244383,
+            -0.004300429148015694,
+            -0.0014387298144714717,
+            -0.023128300275416423,
+            0.004468952290723408,
+            -0.0026602931688877443,
+            -0.005768852559887139,
+            0.007815212168719318,
+            0.0005862972228454734,
+            -0.002265719947629544,
+            -0.00019323737674204774,
+            -0.02735820445266249,
+            0.00024009122317215594,
+            -0.00018701993387558173,
+            -0.014663710240203917,
+            -0.00450797561624596,
+            0.010584731989847334,
+            -0.011966527748437568,
+            -0.00602367292291343,
+            0.0111155226518017,
+            0.0015565179432852256,
+            0.006062862961835896,
+            0.016952559697728433,
+            0.0005012157272316869,
+            0.011160668242334889,
+            0.014993782560797772,
+            -0.01591303023837098,
+            -0.015356412848444082,
+            -0.0236025426434855,
+            -0.004514538154596937,
+            -0.009753334017065119,
+            0.007553951535816575,
+            -0.004616595532996391,
+            -0.013023153809236325,
+            0.006040285197946957,
+            -0.0006343451398642784,
+            0.008338564073158257,
+            -0.017608246105292328,
+            -8.346168119689692e-05,
+            -0.020944870308705715,
+            0.009230123321316848,
+            -0.00531237224627929,
+            -0.00445985089300067,
+            0.01410675485522911,
+            0.02895945580213926,
+            -0.007799830351886844,
+            -0.009455128322575888,
+            -0.005663072244883114,
+            0.012817229945992764,
+            0.012474431030896143,
+            -0.004423141749110039,
+            -0.0018294512146253298,
+            -0.012817101093394574,
+            0.0004755435377918658,
+            -0.01129485262081729,
+            0.0062747111125005155,
+            0.017199374798326412,
+            -0.021965117839794063,
+            -0.01296097993683748,
+            -0.016765996091978336,
+            -0.016882855472266523,
+            0.004372970343291638,
+            0.017290565252244258,
+            0.022670788364378462,
+            -0.008456547449078897,
+            -0.019564049850942417,
+            0.006898353991539946,
+            -0.01620125883985687,
+            0.0037108638122592746,
+            0.12887455743463638,
+            -0.0028864852421660366,
+            -0.011487804449328495,
+            -0.022683334123299446,
+            -0.005467192611880829,
+            0.01210557696712196,
+            -0.01555159294496261,
+            0.06709672179195036,
+            -0.015071089552631105,
+            0.004364902817496082,
+            -0.004899872946473657,
+            0.0006310685609937571,
+            -0.017610935588191347,
+            -0.005962991715597585,
+            -0.007768819050529025,
+            -0.02688607975562596,
+            0.000883763319535686,
+            -0.009935313912803587,
+            -0.012883104064962595,
+            0.002126368813790577,
+            -0.0013697265269800346,
+            -0.016131186838419288,
+            0.01575514418037863,
+            -0.020677990768697592,
+            -0.007251917191831732,
+            -0.0029325623780992118,
+            -0.00600710212646933,
+            0.014573093764769918,
+            -0.03253336117394778,
+            0.002973725493330224,
+            -0.013601417220439602,
+            0.011905788192118131,
+            0.001542117126832946,
+            -0.010789486980776022,
+            -0.013862139885514524,
+            0.007086571810314664,
+            0.015658913897173155,
+            0.017039624764452546,
+            0.01901374401756777,
+            0.004797954569087795,
+            0.003928999581093576,
+            -0.008046722893614655,
+            0.021974371139905104,
+            -0.00697335955527197,
+            0.0008975473383295958,
+            -0.00970589295353857,
+            -0.004346869073499034,
+            0.0031279458769925416,
+            0.02303983493541673,
+            -0.012560380384977343,
+            -0.01536378830925542,
+            0.0012035987157572355,
+            0.033313215063595594,
+            -0.010845084209743817,
+            0.013587126023659577,
+            0.0029867620644300073,
+            0.020679673435988446,
+            -0.0006640787253300063,
+            0.0025071780490108306,
+            0.022554193713122565,
+            -0.018082906254599396,
+            0.018230000220500718,
+            -0.004451773302171552,
+            0.01915083025790679,
+            -0.004183521043835777,
+            0.015019320101074747,
+            -0.027014055979204272,
+            -0.005362044589023088,
+            -0.012355559881051253,
+            0.003995309003198048,
+            0.020680455787871383,
+            0.0054391755324872375,
+            0.009865676762244514,
+            -0.01645386866657314,
+            -0.01760657627715638,
+            0.022969828924328126,
+            -0.021566687194482292,
+            -0.015826424849757402,
+            0.00012804774676298237,
+            -0.015021970627042762,
+            0.006000610438452896,
+            0.010126179023323158,
+            -0.02912121211344025,
+            -0.02046150879033127,
+            0.004881850039543051,
+            -0.0023469840669773042,
+            -0.00915786433296249,
+            0.006547377770403066,
+            -0.022329063637464074,
+            -0.015702777677743352,
+            -0.005692904224111613,
+            0.01207999389934878,
+            -0.008429160038192635,
+            0.010075061963131637,
+            -0.020106712664936504,
+            0.0036787045650556907,
+            -0.002764965946892891,
+            0.010307702251974629,
+            0.004697169247387421,
+            0.011192285826515207,
+            0.0030113901179382946,
+            0.0014774681536380047,
+            -0.02778270000997313,
+            0.010043261430740039,
+            -5.178406450222847e-05,
+            0.04216636255860709,
+            0.030977096295679525,
+            0.004479857028809617,
+            -5.798020745008712e-05,
+            -0.009996471945562616,
+            0.012476009017187798,
+            -0.019306907391548302,
+            -0.005092322343599383,
+            0.01856463597985508,
+            0.02195467141594795,
+            -0.02236232264139305,
+            0.0052217593371508145,
+            -0.016766955100332014,
+            0.016133299654723637,
+            -0.011208763137897244,
+            0.006248024409576809,
+            0.011601988603146743,
+            0.0022330878390492527,
+            0.010317781892717903,
+            -0.005895142237225119,
+            0.019670196605183973,
+            0.01186677200787666,
+            -0.002339367025714548,
+            0.02367408853110795,
+            0.006858134385159451,
+            0.004479683673962342,
+            0.004798719889930319,
+            -0.009604173975576537,
+            -0.009045602998213713,
+            0.009230969488015385,
+            0.004083616606774546,
+            -0.007944628589259331,
+            0.011836323287097144,
+            0.015100350195476968,
+            0.01885317701149666,
+            -0.0011707847499298092,
+            -0.00684548433712413,
+            -0.006550059418141426,
+            0.001899758393077906,
+            -0.024577381680573596,
+            -0.02855511854789239,
+            0.021238935710272536,
+            -0.010427135883958709,
+            -0.019187769881105637,
+            -0.0015864640188683347,
+            0.016360037306061392,
+            -0.008853972319021563,
+            -0.01627167079143546,
+            0.001270414822216867,
+            0.02255755055961612,
+            -0.005959411125391464,
+            0.006009654905468478,
+            0.005909233773418581,
+            0.005874361519568704,
+            -0.0027859463941992753,
+            -0.024599789577567636,
+            0.0036612374968223453,
+            0.03221346865585554,
+            0.016461563205582293,
+            0.016139782455492073,
+            -0.007158168059808459,
+            0.001838455290310879,
+            0.007530954381521755,
+            0.005438791523412295,
+            0.015674000912615536,
+            0.008604106101542998,
+            -0.006990462920227135,
+            0.00244736628387359,
+            0.012213048255418866,
+            -0.007329299946651028,
+            0.017424711328323798,
+            0.001973882532418776,
+            -0.0133178791530314,
+            -0.008373452013951822,
+            -0.010462150707776556,
+            -0.014575483571750984,
+            0.0027958704799249897,
+            -0.011401916623878957,
+            0.0008016041012669959,
+            0.003953958717263787,
+            0.010260753041492368,
+            0.013961649189650642,
+            -0.0019275791443041316,
+            0.017019168629298462,
+            0.011534594243860177,
+            -0.0017859096307142257,
+            -0.002970739203219076,
+            -0.03142778833370656,
+            0.010207789755749909,
+            -0.014730553506704018,
+            -0.0037611416875830746,
+            0.01825790347981639,
+            -0.007766890847082858,
+            -0.0032778433418881666,
+            -0.021003405429807113,
+            0.0024512381432602736,
+            0.008768616692363321,
+            0.01558713989712387,
+            -0.012819542714293536,
+            0.009201155584074711,
+            -0.0012254337002053017,
+            -0.0057451574939563045,
+            0.00977251392462179,
+            0.0042356831792011935,
+            -0.0033433823868435786,
+            0.011499764125503874
+          ],
+          "title": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer",
+          "keyphrases": [
+            "Sparsely-Gated Mixture-of-Experts Layer",
+            "conditional computation",
+            "neural network",
+            "language modeling",
+            "machine translation"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "bec4b8f2-cf26-434f-a587-d63e318fec0c",
+        "properties": {
+          "page_content": "2 The Structure Of The Mixture-Of-Experts Layer\n\nThe Mixture-of-Experts (MoE) layer consists of a set of n \"expert networks\" E1, · · · , En, and a \"gating network\" G whose output is a sparse n-dimensional vector. Figure 1 shows an overview of the MoE module. The experts are themselves neural networks, each with their own parameters.\n\nAlthough in principle we only require that the experts accept the same sized inputs and produce the same-sized outputs, in our initial investigations in this paper, we restrict ourselves to the case where the models are feed-forward networks with identical architectures, but with separate parameters.\n\nLet us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows:\n\n$$y=\\sum_{i=1}^{n}G(x){i}E{i}(x)$$ $$(\\mathbf{l})$$ G(x)iEi(x) (1) We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example. If the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network. In the following we focus on ordinary MoEs. We provide more details on hierarchical MoEs in Appendix B. Our implementation is related to other models of conditional computation. A MoE whose experts are simple weight matrices is similar to the parameterized weight matrix proposed in (Cho & Bengio, 2014). A MoE whose experts have one hidden layer is similar to the block-wise dropout described in (Bengio et al., 2015), where the dropped-out layer is sandwiched between fully-activated layers.\n\n2.1 Gating Network\n\nSoftmax Gating: A simple choice of non-sparse gating function (Jordan & Jacobs, 1994) is to multiply the input by a trainable weight matrix Wg and then apply the Sof tmax function.\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ $\\eqref{eq:walpha}$. Gσ(x) = Sof tmax(x · Wg) (2) Noisy Top-K Gating: We add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to −∞ (which causes the corresponding gate values to equal 0). The sparsity serves to save computation, as described above. While this form of sparsity creates some theoretically scary discontinuities in the output of gating function, we have not yet observed this to be a problem in practice. The noise term helps with load balancing, as will be discussed in Appendix A. The amount of noise per component is controlled by a second trainable weight matrix Wnoise.\n\n$$G(x)=S o f t m a x(K e e p T o p K(H(x),k))$$ $$(4)$$ $$(S)$$ G(x) = Sof tmax(KeepT opK(H(x), k)) (3) $$H(x){i}=(x\\cdot W{g}){i}+S t a n d a r d N o r m a l()\\cdot S o f t p l u s((x\\cdot W{n o i s e}){i})$$ $KeepTopK(v,k){i}=\\begin{cases}v_{i}&\\text{if}v_{i}\\text{is in the top}k\\text{elements of}v.\\ -\\infty&\\text{otherwise.}\\end{cases}$ Training the Gating Network We train the gating network by simple back-propagation, along with the rest of the model. If we choose k > 1, the gate values for the top k experts have nonzero derivatives with respect to the weights of the gating network. This type of occasionally-sensitive behavior is described in (Bengio et al., 2013) with respect to noisy rectifiers. Gradients also backpropagate through the gating network to its inputs. Our method differs here from (Bengio et al., 2015) who use boolean gates and a REINFORCE-style approach to train the gating network.\n\n",
+          "embedding": [
+            0.00986587256193161,
+            -0.03260384127497673,
+            -0.016219932585954666,
+            -0.022328028455376625,
+            0.017736708745360374,
+            0.029624948278069496,
+            -0.004546910524368286,
+            0.026386426761746407,
+            -0.00606368575245142,
+            0.009852208197116852,
+            -0.018788885325193405,
+            -0.02659139782190323,
+            0.0016850214451551437,
+            -0.01340501569211483,
+            -0.004304362926632166,
+            -0.00787083525210619,
+            -0.007932325825095177,
+            0.022382687777280807,
+            -0.00348277622833848,
+            -0.029105691239237785,
+            0.03632062301039696,
+            -0.0069143096916377544,
+            -0.021795107051730156,
+            0.0010393669363111258,
+            0.011690102517604828,
+            0.010815565474331379,
+            0.058430016040802,
+            -0.025744188576936722,
+            -0.05607970058917999,
+            0.010234818793833256,
+            0.020223673433065414,
+            0.003211191389709711,
+            -0.022328028455376625,
+            -0.008465247228741646,
+            -0.07581144571304321,
+            -0.04277033731341362,
+            0.026851026341319084,
+            0.04208710417151451,
+            0.0028746994212269783,
+            0.024746669456362724,
+            0.064114511013031,
+            0.0007968195714056492,
+            -0.00416088430210948,
+            -0.0368945375084877,
+            -0.019854728132486343,
+            -0.012175197713077068,
+            -0.025949159637093544,
+            -0.0034093286376446486,
+            -0.01524974312633276,
+            -0.03134669363498688,
+            4.102061939192936e-05,
+            -0.008957174606621265,
+            0.005318962503224611,
+            0.04080262780189514,
+            -0.05337410047650337,
+            0.022874614223837852,
+            0.01740875653922558,
+            0.021439826115965843,
+            0.03134669363498688,
+            0.02275163307785988,
+            0.04361754283308983,
+            0.00478262547403574,
+            -0.05853933468461037,
+            0.008123630657792091,
+            0.019444787874817848,
+            -0.00873170793056488,
+            0.01642490178346634,
+            0.0034161610528826714,
+            -0.00102143210824579,
+            0.009339784272015095,
+            -0.039490822702646255,
+            0.014552845619618893,
+            -0.02160380221903324,
+            0.0027124318294227123,
+            -0.0009351740591228008,
+            -0.006668346468359232,
+            0.02001870423555374,
+            0.04028337076306343,
+            0.028039850294589996,
+            0.009920531883835793,
+            -0.01302923820912838,
+            0.023366540670394897,
+            0.008595061488449574,
+            0.006244742311537266,
+            0.007324249017983675,
+            -0.02907836250960827,
+            -0.01475781574845314,
+            -0.04747097194194794,
+            -0.010487614199519157,
+            0.006070517934858799,
+            -0.006986049469560385,
+            0.02747959829866886,
+            0.011464636772871017,
+            0.002150473417714238,
+            0.017531737685203552,
+            0.02048330195248127,
+            0.045612581074237823,
+            -0.0037577771581709385,
+            -0.006627352442592382,
+            -0.01876155659556389,
+            -0.0019147582352161407,
+            0.026154128834605217,
+            -0.001622676500119269,
+            0.032795146107673645,
+            0.0015825366135686636,
+            -0.0020411561708897352,
+            0.018816214054822922,
+            0.03131936490535736,
+            0.03260384127497673,
+            0.08685247600078583,
+            -0.0060021947138011456,
+            0.00043107682722620666,
+            -0.019267147406935692,
+            0.03517279401421547,
+            -0.02644108608365059,
+            -0.009086987935006618,
+            -0.04812687635421753,
+            -0.054248638451099396,
+            0.010118668898940086,
+            -0.03875293210148811,
+            0.028039850294589996,
+            0.020893240347504616,
+            -0.023626169189810753,
+            0.0012520230375230312,
+            -0.06941638886928558,
+            0.06351326406002045,
+            0.0015082351164892316,
+            0.007604374550282955,
+            -0.010378297418355942,
+            -0.030854765325784683,
+            0.005377037450671196,
+            0.0025945743545889854,
+            -0.01956777088344097,
+            -0.038889575749635696,
+            0.03028085082769394,
+            0.017299439758062363,
+            0.041540518403053284,
+            0.028504448011517525,
+            -0.005674243438988924,
+            -0.029870912432670593,
+            0.029706936329603195,
+            -0.022778961807489395,
+            -0.007385740056633949,
+            -0.028067179024219513,
+            0.011123020201921463,
+            0.012059048749506474,
+            0.04036535695195198,
+            0.06263872981071472,
+            -0.038670942187309265,
+            0.027834879234433174,
+            -0.04525730013847351,
+            0.04080262780189514,
+            -0.012468988075852394,
+            0.036566589027643204,
+            -0.001954043982550502,
+            0.0030711288563907146,
+            -0.01716279238462448,
+            -0.020578954368829727,
+            -0.014580175280570984,
+            -0.02582617662847042,
+            -0.010754074901342392,
+            -0.028559105470776558,
+            0.005889461375772953,
+            0.049356695264577866,
+            0.01159445010125637,
+            0.05118775740265846,
+            0.009421772323548794,
+            0.019977709278464317,
+            -0.05427596718072891,
+            0.014019925147294998,
+            -0.0017371178837493062,
+            0.008916180580854416,
+            0.04585854709148407,
+            0.008383259177207947,
+            0.0039115045219659805,
+            0.002119727898389101,
+            0.02943364344537258,
+            0.031893279403448105,
+            -0.006326730363070965,
+            0.04028337076306343,
+            -0.03375167027115822,
+            -0.017217451706528664,
+            0.01627459190785885,
+            0.006500954274088144,
+            0.009838543832302094,
+            -0.0024391389451920986,
+            0.03284980356693268,
+            -0.01389694306999445,
+            0.005595671944320202,
+            -0.00036745081888511777,
+            0.042743004858493805,
+            0.039572808891534805,
+            0.04610450938344002,
+            -0.022614985704421997,
+            0.017067139968276024,
+            -0.019854728132486343,
+            0.0023076168727129698,
+            0.015755334869027138,
+            -0.0200323686003685,
+            0.018638573586940765,
+            -0.012223023921251297,
+            -0.012421161867678165,
+            0.011491965502500534,
+            -0.025484561920166016,
+            0.01572800613939762,
+            0.015536700375378132,
+            -0.014853468164801598,
+            -0.019239818677306175,
+            0.013917439617216587,
+            -0.040256042033433914,
+            0.019021185114979744,
+            -0.01000935211777687,
+            -0.01973174512386322,
+            -0.02899637445807457,
+            -0.010453452356159687,
+            -0.005004676058888435,
+            -0.00741990189999342,
+            0.017190122976899147,
+            0.03260384127497673,
+            0.003147992305457592,
+            0.0021487653721123934,
+            0.038370318710803986,
+            0.009387610480189323,
+            0.009585747495293617,
+            -0.0007908412953838706,
+            -0.02303859032690525,
+            -0.016821177676320076,
+            -0.025347914546728134,
+            0.012475820258259773,
+            -0.022396352142095566,
+            -0.030772779136896133,
+            0.02335287630558014,
+            0.007727356161922216,
+            0.01336402166634798,
+            0.015331730246543884,
+            -0.013732966966927052,
+            0.013172716833651066,
+            0.014785144478082657,
+            0.02817649580538273,
+            -0.017531737685203552,
+            -0.02198641188442707,
+            0.020893240347504616,
+            -0.06673812121152878,
+            -0.0074745602905750275,
+            0.006716172676533461,
+            0.02793053165078163,
+            -0.013507500290870667,
+            -0.012926753610372543,
+            -0.007481392472982407,
+            -0.004116474185138941,
+            0.012981412000954151,
+            -0.015277071855962276,
+            -0.010501278564333916,
+            0.019021185114979744,
+            -0.015072102658450603,
+            0.046131838113069534,
+            0.011840414255857468,
+            -0.014484522864222527,
+            -0.015454712323844433,
+            -0.00046801407006569207,
+            0.0370585136115551,
+            0.031292036175727844,
+            -0.0269603431224823,
+            0.009852208197116852,
+            0.008533569984138012,
+            -0.025634871795773506,
+            0.0616002157330513,
+            0.015564029105007648,
+            0.01341184787452221,
+            -0.000460754701634869,
+            0.024131761863827705,
+            -0.04222375154495239,
+            0.013036070391535759,
+            -0.021125540137290955,
+            0.05400267243385315,
+            -0.02295660227537155,
+            0.011539791710674763,
+            -0.04708836227655411,
+            -0.0032009428832679987,
+            0.047170352190732956,
+            0.013418680056929588,
+            0.010965877212584019,
+            0.004526413511484861,
+            0.01973174512386322,
+            0.029215008020401,
+            0.015441047959029675,
+            -0.009121149778366089,
+            -0.017572732642292976,
+            -0.05310080572962761,
+            0.008567731827497482,
+            -0.015482041984796524,
+            0.007495057303458452,
+            -0.03342371806502342,
+            0.0015953471884131432,
+            0.02981625311076641,
+            0.014621169306337833,
+            -0.014361540786921978,
+            -0.002075317781418562,
+            0.027807550504803658,
+            -0.042879652231931686,
+            -0.04968464747071266,
+            -0.007317416835576296,
+            -0.011368983425199986,
+            0.007693194784224033,
+            -0.03361502289772034,
+            0.02048330195248127,
+            0.009394442662596703,
+            0.024828657507896423,
+            -0.005103744566440582,
+            -0.011027367785573006,
+            0.01386278122663498,
+            0.002191467210650444,
+            -0.009209970012307167,
+            -0.03366968408226967,
+            -0.029269667342305183,
+            -0.010166495107114315,
+            0.008137295953929424,
+            0.027848543599247932,
+            -0.016219932585954666,
+            -0.012011222541332245,
+            0.027684569358825684,
+            0.022164052352309227,
+            -0.007153441198170185,
+            0.0025689529720693827,
+            -0.024651017040014267,
+            0.005295049399137497,
+            0.015755334869027138,
+            0.028340471908450127,
+            -0.03960013762116432,
+            0.011321157217025757,
+            -0.00564691424369812,
+            -0.022341692820191383,
+            -0.03312309831380844,
+            0.02380380965769291,
+            -0.0134391775354743,
+            -0.03621130809187889,
+            -0.014279552735388279,
+            0.014812474139034748,
+            -0.0044615063816308975,
+            -0.03172930330038071,
+            -0.013650978915393353,
+            0.006398469675332308,
+            -0.014416199177503586,
+            0.01612428016960621,
+            -0.015085767023265362,
+            -0.0035254780668765306,
+            -0.03880758956074715,
+            0.013036070391535759,
+            0.014006259851157665,
+            0.010938547551631927,
+            0.0009035745752044022,
+            0.006791328079998493,
+            0.017572732642292976,
+            -0.016055956482887268,
+            -0.026755372062325478,
+            -0.024159090593457222,
+            0.0017046643188223243,
+            0.006490706000477076,
+            0.006415550597012043,
+            0.010200656950473785,
+            0.016916830092668533,
+            -0.026454750448465347,
+            -0.011573953554034233,
+            0.007986984215676785,
+            -0.054631248116493225,
+            -0.012298179790377617,
+            -0.03878026083111763,
+            -0.019526775926351547,
+            -0.0013092437293380499,
+            0.028941715136170387,
+            -0.0064462958835065365,
+            0.03427092730998993,
+            -0.0056708273477852345,
+            0.00754971569404006,
+            0.017203787341713905,
+            0.018993854522705078,
+            0.018173975870013237,
+            -0.011806252412497997,
+            -0.050340548157691956,
+            0.028887057676911354,
+            -0.03593801334500313,
+            -0.0045229969546198845,
+            0.00282345712184906,
+            -0.002869575284421444,
+            -0.0057049887254834175,
+            0.0338883176445961,
+            0.0015304401749745011,
+            0.023544181138277054,
+            -0.00381243578158319,
+            0.04353555664420128,
+            -0.027055995538830757,
+            -0.049821291118860245,
+            0.00873170793056488,
+            0.027657238766551018,
+            0.024814993143081665,
+            -0.006521451286971569,
+            -0.026536738499999046,
+            -0.02018267847597599,
+            -0.013514332473278046,
+            0.007966487668454647,
+            0.002003578469157219,
+            -0.00044239286216907203,
+            -0.0013263245346024632,
+            0.006548780482262373,
+            -0.034462232142686844,
+            -0.03645727038383484,
+            0.0040549831464886665,
+            0.011813084594905376,
+            0.03213924169540405,
+            -0.02590816468000412,
+            0.017067139968276024,
+            -0.015454712323844433,
+            -0.038588955998420715,
+            0.0006661514053121209,
+            0.023598840460181236,
+            0.035965342074632645,
+            -0.03921752795577049,
+            0.018419940024614334,
+            0.015850987285375595,
+            -0.005602504126727581,
+            0.004188213497400284,
+            0.025265926495194435,
+            0.025347914546728134,
+            0.022273369133472443,
+            -0.00858822837471962,
+            0.04020138084888458,
+            0.014457193203270435,
+            -0.005564926192164421,
+            0.029706936329603195,
+            -0.016766518354415894,
+            0.02998022921383381,
+            -0.0002463906130287796,
+            0.021426161751151085,
+            -0.027916867285966873,
+            0.019007520750164986,
+            -0.0382336750626564,
+            -0.008642887696623802,
+            0.03304110839962959,
+            -0.026085805147886276,
+            0.0034537387546151876,
+            0.022915607318282127,
+            0.01851559244096279,
+            -0.010631092824041843,
+            0.01725844480097294,
+            -0.04451940953731537,
+            -0.029160350561141968,
+            -0.026195121929049492,
+            -0.006822073366492987,
+            -0.02500629797577858,
+            -0.03552807494997978,
+            -0.056134358048439026,
+            0.008649719879031181,
+            -0.047935571521520615,
+            0.004386350512504578,
+            0.03533677011728287,
+            0.010760907083749771,
+            0.010549104772508144,
+            -0.024432383477687836,
+            0.02809450775384903,
+            -0.018351616337895393,
+            -0.00914847943931818,
+            0.021043552085757256,
+            0.024883316829800606,
+            -0.012346005998551846,
+            -0.013842284679412842,
+            0.01612428016960621,
+            -0.03804237022995949,
+            0.006350643467158079,
+            -0.0020326157100498676,
+            -0.024446047842502594,
+            0.019991373643279076,
+            -0.04684239998459816,
+            0.02380380965769291,
+            0.013876445591449738,
+            0.02877774089574814,
+            0.06039772555232048,
+            -0.024651017040014267,
+            -0.06285735964775085,
+            0.020128021016716957,
+            -0.004143803380429745,
+            0.01336402166634798,
+            0.003747528651729226,
+            -0.006169586908072233,
+            0.002331529976800084,
+            -0.01679384708404541,
+            -0.026031147688627243,
+            0.052526891231536865,
+            0.023475859314203262,
+            0.015263407491147518,
+            -0.010023016482591629,
+            0.0013707346515730023,
+            0.0368945375084877,
+            0.023817474022507668,
+            0.01378079317510128,
+            0.016998816281557083,
+            -0.03929951786994934,
+            0.0037782741710543633,
+            0.0064087179489433765,
+            0.022246040403842926,
+            0.027206305414438248,
+            -0.02711065299808979,
+            -0.027083324268460274,
+            0.020920569077134132,
+            0.04528462886810303,
+            0.011908737011253834,
+            -0.0139379370957613,
+            0.016055956482887268,
+            0.03383366018533707,
+            -0.035965342074632645,
+            -0.003212899435311556,
+            0.009551586583256721,
+            0.0011298952158540487,
+            -0.010719913057982922,
+            -0.02041497826576233,
+            0.009961525909602642,
+            0.014402534812688828,
+            -0.02010069042444229,
+            0.01906217820942402,
+            0.010972709394991398,
+            0.028449788689613342,
+            0.0052267261780798435,
+            -0.006750334054231644,
+            -0.029570288956165314,
+            0.04902874305844307,
+            -0.022027406841516495,
+            0.022997595369815826,
+            0.03774174675345421,
+            -0.0027431773487478495,
+            0.024487042799592018,
+            0.002572369296103716,
+            -0.03812435641884804,
+            0.00813046284019947,
+            0.04451940953731537,
+            0.005575174931436777,
+            -0.028039850294589996,
+            -0.016670865938067436,
+            -0.03145601227879524,
+            0.004348773043602705,
+            -0.01738142780959606,
+            -0.02793053165078163,
+            -0.0014125825837254524,
+            0.005551261827349663,
+            0.016684530302882195,
+            -0.002896904479712248,
+            -0.005985114257782698,
+            -0.006132008973509073,
+            0.02507462166249752,
+            -0.049821291118860245,
+            -0.005663995165377855,
+            0.014443528838455677,
+            -0.029406314715743065,
+            -0.028504448011517525,
+            -0.03711317479610443,
+            -0.004229207523167133,
+            -0.013842284679412842,
+            0.0010385129135102034,
+            -0.001186261884868145,
+            -0.0134391775354743,
+            0.011082026176154613,
+            -0.03940883278846741,
+            0.02508828602731228,
+            -0.012100041843950748,
+            -0.010139165446162224,
+            0.0030813771300017834,
+            0.009230467490851879,
+            -0.010788236744701862,
+            -0.008226115256547928,
+            0.04178648069500923,
+            0.02198641188442707,
+            0.024418719112873077,
+            -0.006641016807407141,
+            -0.007071453146636486,
+            0.005745982751250267,
+            0.000374923663912341,
+            0.03809702768921852,
+            -0.001469803275540471,
+            0.03561006113886833,
+            -0.009121149778366089,
+            -0.012660292908549309,
+            0.006883564405143261,
+            0.03752311319112778,
+            -0.0069962977431714535,
+            0.010207489132881165,
+            0.03088209591805935,
+            0.027015000581741333,
+            -0.017121799290180206,
+            -0.015195083804428577,
+            -0.0025894499849528074,
+            -0.004434177186340094,
+            -0.021057216450572014,
+            0.02198641188442707,
+            0.03350570797920227,
+            0.01171059999614954,
+            -0.009353448636829853,
+            0.022573992609977722,
+            -0.04348089545965195,
+            -0.0011435599299147725,
+            0.004113057628273964,
+            0.012455322779715061,
+            -0.05001259595155716,
+            0.016247261315584183,
+            0.014416199177503586,
+            0.017367763444781303,
+            0.00129045476205647,
+            0.006395053584128618,
+            0.023175235837697983,
+            0.02176777832210064,
+            0.024951640516519547,
+            -0.015850987285375595,
+            2.2431901015806943e-05,
+            0.019841063767671585,
+            0.015618688426911831,
+            0.024268407374620438,
+            0.0021436410024762154,
+            0.024978969246149063,
+            -0.017586397007107735,
+            -0.0138559490442276,
+            0.00282345712184906,
+            -0.023380205035209656,
+            -0.010330471210181713,
+            0.05468590557575226,
+            0.011553456075489521,
+            0.051242414861917496,
+            0.01426588837057352,
+            0.00024169338576029986,
+            0.018993854522705078,
+            -0.020510630682110786,
+            -0.02026466652750969,
+            -0.0007050101994536817,
+            0.0027824630960822105,
+            -0.020387649536132812,
+            -0.00945593323558569,
+            -0.010610596276819706,
+            -0.018187642097473145,
+            -0.018939197063446045,
+            -0.006746917963027954,
+            -0.006326730363070965,
+            0.016848506405949593,
+            -0.002319573424756527,
+            -0.013541662134230137,
+            -0.0320572555065155,
+            0.022041071206331253,
+            0.0199503805488348,
+            0.040556661784648895,
+            -0.013575823977589607,
+            0.011239170096814632,
+            -0.0033819994423538446,
+            -0.004560574889183044,
+            0.014621169306337833,
+            -0.01604229211807251,
+            0.017982671037316322,
+            0.009196305647492409,
+            0.00139293959364295,
+            0.011075193993747234,
+            -0.01077457144856453,
+            -0.04684239998459816,
+            -0.010200656950473785,
+            -0.023557845503091812,
+            0.03284980356693268,
+            0.00672300485894084,
+            0.0018566835206001997,
+            0.0039012557826936245,
+            0.0035459750797599554,
+            -0.008116798475384712,
+            0.016821177676320076,
+            0.013275201432406902,
+            -0.03274048864841461,
+            0.017436085268855095,
+            -0.011382648721337318,
+            0.010255315341055393,
+            -0.034598879516124725,
+            -0.013691972941160202,
+            -0.002481841016560793,
+            0.017586397007107735,
+            0.026482079178094864,
+            -0.02869575284421444,
+            0.010610596276819706,
+            0.02131684496998787,
+            -0.002876407466828823,
+            -0.005698156543076038,
+            0.0021231439895927906,
+            0.0014151446521282196,
+            -0.026304440572857857,
+            -0.012024886906147003,
+            0.005301882047206163,
+            0.004659643396735191,
+            -0.03386098891496658,
+            0.005448776762932539,
+            -0.0293789841234684,
+            -0.005595671944320202,
+            -0.011908737011253834,
+            0.003928584977984428,
+            0.0068152411840856075,
+            0.002565536880865693,
+            0.009134814143180847,
+            -0.01831062324345112,
+            -0.007638535927981138,
+            0.024145426228642464,
+            0.004222374875098467,
+            0.015195083804428577,
+            -0.012414328753948212,
+            0.029542960226535797,
+            -0.016684530302882195,
+            0.021959083154797554,
+            0.014402534812688828,
+            0.007857169955968857,
+            -0.002591158030554652,
+            -0.002437430899590254,
+            -0.04471071437001228,
+            -0.01046711765229702,
+            0.010111836716532707,
+            0.02559387870132923,
+            -0.0005986822070553899,
+            0.014539181254804134,
+            0.02477400004863739,
+            -0.008882018737494946,
+            0.015495706349611282,
+            -0.01701248250901699,
+            -0.030144205316901207,
+            0.020305661484599113,
+            -0.0027380529791116714,
+            0.015318065881729126,
+            0.02425474300980568,
+            -0.03148334100842476,
+            -0.02432306669652462,
+            -0.002088982379063964,
+            0.011813084594905376,
+            0.05137906223535538,
+            0.029706936329603195,
+            0.012182029895484447,
+            0.020906904712319374,
+            -0.008157792501151562,
+            -0.02869575284421444,
+            0.012455322779715061,
+            0.01265346072614193,
+            -0.03110072948038578,
+            0.00634722737595439,
+            0.026755372062325478,
+            -0.008601893670856953,
+            -0.02853177674114704,
+            0.02599015273153782,
+            -0.00689381267875433,
+            0.002553580328822136,
+            -0.004796289838850498,
+            -0.020947899669408798,
+            0.004386350512504578,
+            -0.01602862775325775,
+            0.027903202921152115,
+            -0.024418719112873077,
+            0.008595061488449574,
+            0.010084507055580616,
+            -0.009414940141141415,
+            0.032931793481111526,
+            -0.0005137051921337843,
+            -0.01505843736231327,
+            0.010173327289521694,
+            0.021344173699617386,
+            -0.028887057676911354,
+            0.0005017486400902271,
+            0.002992557128891349,
+            0.01339818350970745,
+            0.03968212753534317,
+            0.018939197063446045,
+            0.016698194667696953,
+            0.03295912221074104,
+            0.020169014111161232,
+            -0.01294725015759468,
+            0.025170274078845978,
+            0.01039879396557808,
+            -0.020237337797880173,
+            0.008615558035671711,
+            -0.0060056112706661224,
+            0.014580175280570984,
+            -0.026851026341319084,
+            -0.007050956133753061,
+            -0.002191467210650444,
+            0.0020087026059627533,
+            -0.020373983308672905,
+            -0.005223310086876154,
+            -0.002256374340504408,
+            -0.023680828511714935,
+            0.008150960318744183,
+            0.028668422251939774,
+            0.023393871262669563,
+            -0.00436585396528244,
+            -0.021508149802684784,
+            0.026673385873436928,
+            -0.02410443127155304,
+            -0.012714951299130917,
+            -0.0037782741710543633,
+            0.015796327963471413,
+            -0.022983931005001068,
+            -0.032494522631168365,
+            0.0019437955925241113,
+            0.03509080410003662,
+            0.02823115512728691,
+            -0.004167716484516859,
+            0.044546738266944885,
+            0.02333921194076538,
+            -0.021617466583848,
+            0.02033299021422863,
+            -0.038288332521915436,
+            0.018939197063446045,
+            0.017531737685203552,
+            -0.04684239998459816,
+            -0.030253522098064423,
+            -0.006258407142013311,
+            -0.011895072646439075,
+            0.008123630657792091,
+            -0.017231116071343422,
+            0.003781690262258053,
+            -0.00754971569404006,
+            -0.023147907108068466,
+            0.03470819443464279,
+            -0.016506889835000038,
+            0.03402496501803398,
+            -0.008369594812393188,
+            0.003199234837666154,
+            -0.015632351860404015,
+            -0.016315585002303123,
+            -0.012468988075852394,
+            -0.020770259201526642,
+            -0.01027581188827753,
+            -0.02681003138422966,
+            -0.021521814167499542,
+            -0.011341654695570469,
+            0.004509332589805126,
+            -0.008458415046334267,
+            0.02363983355462551,
+            0.0268783550709486,
+            0.01133482251316309,
+            0.015331730246543884,
+            -0.0207839235663414,
+            0.043644871562719345,
+            -0.026304440572857857,
+            0.022942937910556793,
+            -0.013186381198465824,
+            -0.015372724272310734,
+            0.009681399911642075,
+            0.020879575982689857,
+            -0.04072063788771629,
+            0.0034229932352900505,
+            -0.01611061580479145,
+            -0.007385740056633949,
+            -0.0215354785323143,
+            0.00495001720264554,
+            0.017818694934248924,
+            -0.008793198503553867,
+            0.00791182927787304,
+            0.00041847972897812724,
+            -0.004543493967503309,
+            0.05099645256996155,
+            -0.0027175559662282467,
+            0.02642742171883583,
+            -0.016301920637488365,
+            -0.019171494990587234,
+            0.013261537067592144,
+            -0.029870912432670593,
+            0.013698806054890156,
+            -0.00832176860421896,
+            -0.037085842341184616,
+            -0.00865655206143856,
+            0.01171059999614954,
+            0.018392611294984818,
+            0.0116422763094306,
+            -0.0038875911850482225,
+            -0.009845376014709473,
+            -0.011389480903744698,
+            -0.00627207150682807,
+            -0.03582869842648506,
+            -0.015919310972094536,
+            0.0021436410024762154,
+            -0.019690752029418945,
+            0.03394297510385513,
+            0.0034434902481734753,
+            0.03741379454731941,
+            -0.011409977450966835,
+            -0.014211229979991913,
+            0.0052643041126430035,
+            -0.017518073320388794,
+            0.0038226840551942587,
+            0.0029771842528134584,
+            0.008458415046334267,
+            -0.018993854522705078,
+            0.03962746635079384,
+            0.009052827022969723,
+            0.014511851593852043,
+            -0.03462620824575424,
+            -0.013295698910951614,
+            -0.04501133784651756,
+            0.006654681637883186,
+            -0.017873354256153107,
+            -0.032357875257730484,
+            -0.0004112203896511346,
+            -0.005698156543076038,
+            0.005882629193365574,
+            -0.016534218564629555,
+            -0.013609985820949078,
+            0.006702507846057415,
+            -0.008226115256547928,
+            0.009223634377121925,
+            -0.01896652579307556,
+            0.010180159471929073,
+            0.004000324755907059,
+            0.03192060813307762,
+            -0.021658461540937424,
+            0.034598879516124725,
+            -0.012011222541332245,
+            0.003979827743023634,
+            -0.00013408431550487876,
+            -0.012632963247597218,
+            0.019239818677306175,
+            0.016766518354415894,
+            -0.0372224897146225,
+            0.011464636772871017,
+            0.0477442666888237,
+            0.03563738986849785,
+            -0.036484599113464355,
+            0.012530478648841381,
+            0.007208099588751793,
+            -0.003062588395550847,
+            0.03186595067381859,
+            0.0013647563755512238,
+            0.01671185903251171,
+            -0.0007998087094165385,
+            -0.02582617662847042,
+            -0.024514371529221535,
+            0.0037201992236077785,
+            0.01378762535750866,
+            0.007392572239041328,
+            -0.031073400750756264,
+            0.01941745914518833,
+            0.002908861031755805,
+            0.029925569891929626,
+            0.025812512263655663,
+            -0.012598801404237747,
+            0.031210046261548996,
+            0.01423855870962143,
+            -0.029242338612675667,
+            0.027985190972685814,
+            0.010747242718935013,
+            0.014060918241739273,
+            0.029652277007699013,
+            0.04749830439686775,
+            0.009326119907200336,
+            -0.022027406841516495,
+            0.001334864879027009,
+            0.012960914522409439,
+            -0.009954692795872688,
+            0.016288256272673607,
+            0.014183900319039822,
+            0.026031147688627243,
+            0.010965877212584019,
+            -0.014019925147294998,
+            -0.010863391682505608,
+            0.04430077597498894,
+            -0.01574167050421238,
+            -0.00499101122841239,
+            -0.016766518354415894,
+            0.035200122743844986,
+            0.027452269569039345,
+            -0.02402244508266449,
+            -0.02335287630558014,
+            -0.005202813073992729,
+            -0.006750334054231644,
+            0.016329249367117882,
+            -0.01687583513557911,
+            -0.01378762535750866,
+            -0.02891438640654087,
+            -0.01596030406653881,
+            0.0018754723714664578,
+            -0.03929951786994934,
+            -0.02839512936770916,
+            -0.0008578834240324795,
+            0.011437307111918926,
+            0.0008672778494656086,
+            0.02206839993596077,
+            -0.012714951299130917,
+            -0.019021185114979744,
+            -0.005202813073992729,
+            0.02847711741924286,
+            -0.01125283446162939,
+            0.021508149802684784,
+            -0.004833867773413658,
+            0.005014924332499504,
+            0.005206229165196419,
+            0.041239894926548004,
+            0.0016380491433665156,
+            -0.02847711741924286,
+            0.02794419787824154,
+            -0.008554067462682724,
+            0.009688233025372028,
+            0.03607466071844101,
+            0.0026663136668503284,
+            -0.02235535718500614,
+            0.02418641932308674,
+            -0.0026321520563215017,
+            -0.02025100216269493,
+            -0.008936677128076553,
+            0.008007481694221497,
+            -0.0027500095311552286,
+            -0.019390130415558815,
+            0.009319286793470383,
+            0.042141761630773544,
+            -0.0028405378106981516,
+            0.02387213334441185,
+            0.053046148270368576,
+            -0.02235535718500614,
+            -0.02387213334441185,
+            0.015372724272310734,
+            -0.028504448011517525,
+            -0.007044123951345682,
+            -0.006832322105765343,
+            0.03826100379228592,
+            -0.03041749820113182,
+            -0.010863391682505608,
+            -0.025361578911542892,
+            7.216640369733796e-05,
+            -0.05495919659733772,
+            0.01904851384460926,
+            0.004266784992069006,
+            0.01505843736231327,
+            -0.041895799338817596,
+            -0.0353914275765419,
+            0.028559105470776558,
+            0.029269667342305183,
+            -0.007433566264808178,
+            -0.00429069809615612,
+            -0.011608115397393703,
+            -0.009763387963175774,
+            0.021371502429246902,
+            0.0008967422763817012,
+            -0.01565968245267868,
+            0.014101912267506123,
+            0.03170197457075119,
+            -0.0014373497106134892,
+            -0.015577694401144981,
+            -0.06045238673686981,
+            -0.017299439758062363,
+            0.014908126555383205,
+            0.0023178651463240385,
+            -0.0020394481252878904,
+            -3.75510826415848e-05,
+            0.0011683270568028092,
+            -0.0024288904387503862,
+            -0.025197602808475494,
+            0.02726096473634243,
+            0.008793198503553867,
+            0.014525516889989376,
+            -0.004587904084473848,
+            -0.011416809633374214,
+            0.014894462190568447,
+            0.017682049423456192,
+            -0.011881408281624317,
+            -0.017121799290180206,
+            0.03894423693418503,
+            0.01973174512386322,
+            -0.001998454099521041,
+            -0.0005862986436113715,
+            0.0010333886602893472,
+            -0.004136971198022366,
+            -0.030062217265367508,
+            0.001253731083124876,
+            -0.0009095528512261808,
+            0.00581772206351161,
+            0.02823115512728691,
+            -0.008724874816834927,
+            0.029652277007699013,
+            0.023462193086743355,
+            0.021357838064432144,
+            -0.0032590175978839397,
+            -0.0043897670693695545,
+            -0.0018618077738210559,
+            0.02538890950381756,
+            0.01208637747913599,
+            0.008670216426253319,
+            0.002106063300743699,
+            -0.0038602619897574186,
+            -0.017354097217321396,
+            0.010781404562294483,
+            -0.013391351327300072,
+            -0.015482041984796524,
+            0.016001299023628235,
+            0.027001336216926575,
+            -0.007604374550282955,
+            -0.004222374875098467,
+            0.0025962824001908302,
+            0.01076773926615715,
+            0.033478379249572754,
+            -0.019267147406935692,
+            -0.007652200758457184,
+            0.021795107051730156,
+            -0.00502858916297555,
+            0.01265346072614193,
+            0.010528608225286007,
+            0.00910748541355133,
+            0.027616245672106743,
+            0.06580892205238342,
+            -0.030909424647688866,
+            0.0016628163866698742,
+            -0.01934913545846939,
+            -0.0011239169398322701,
+            0.011512462981045246,
+            -0.023516852408647537,
+            0.034598879516124725,
+            0.003549391170963645,
+            -0.017969006672501564,
+            0.034680865705013275,
+            0.015564029105007648,
+            0.010945379734039307,
+            0.010296309366822243,
+            -0.01657521352171898,
+            -0.004909023176878691,
+            -0.011088858358561993,
+            0.008328600786626339,
+            0.020606283098459244,
+            0.023065919056534767,
+            0.01679384708404541,
+            0.018474599346518517,
+            0.002625319641083479,
+            0.01770937815308571,
+            -0.0200323686003685,
+            0.021029887720942497,
+            -0.003908087965101004,
+            0.005117409396916628,
+            -0.02251933328807354,
+            0.0028319975826889277,
+            0.0025552883744239807,
+            -0.001964292488992214,
+            0.013698806054890156,
+            0.000991540728136897,
+            -0.011000038124620914,
+            0.003528894390910864,
+            0.0021248520351946354,
+            0.01132799033075571,
+            0.009312454611063004,
+            -0.0032931792084127665,
+            0.004249704070389271,
+            0.018652239814400673,
+            0.022874614223837852,
+            0.012530478648841381,
+            0.0013177840737625957,
+            0.012305011972784996,
+            0.019923051819205284,
+            -0.022273369133472443,
+            0.01088388916105032,
+            -0.030854765325784683,
+            -0.003627962898463011,
+            0.0036689569242298603,
+            -0.010330471210181713,
+            -0.005964617244899273,
+            -0.011505629867315292,
+            -0.008301271125674248,
+            0.01295408234000206,
+            -0.0074472310952842236,
+            0.021726783365011215,
+            -0.001455284538678825,
+            -0.006538532208651304,
+            0.02176777832210064,
+            0.0013391351094469428,
+            -0.0044512576423585415,
+            -0.028559105470776558,
+            -0.0099751902744174,
+            0.021043552085757256,
+            -0.002442555036395788,
+            -0.0005747691029682755,
+            0.02559387870132923,
+            -0.015495706349611282,
+            0.01133482251316309,
+            -0.01686217077076435,
+            -0.015304401516914368,
+            -0.0028217490762472153,
+            -0.015468376688659191,
+            0.016452230513095856,
+            -0.004406847525388002,
+            0.00627207150682807,
+            -0.03448956087231636,
+            0.03142867982387543,
+            0.006627352442592382,
+            -0.0016611083410680294,
+            -0.013664644211530685,
+            -0.030253522098064423,
+            0.0095584187656641,
+            -0.021207528188824654,
+            -0.00017358368495479226,
+            0.02176777832210064,
+            0.011471468955278397,
+            -0.0004603276902344078,
+            -0.009066491387784481,
+            0.0024835490621626377,
+            0.0059680333361029625,
+            -0.0009180932538583875,
+            0.005862132180482149,
+            -0.00028332785586826503,
+            0.015304401516914368,
+            -0.016957823187112808,
+            -0.028449788689613342,
+            -0.02303859032690525,
+            0.01306340005248785,
+            0.00862239021807909,
+            -0.004871445707976818,
+            -0.015523036010563374,
+            0.0057767280377447605,
+            0.004181380849331617,
+            -0.0018020248971879482,
+            -0.0034810679499059916,
+            0.007488225121051073,
+            -0.0010103295790031552,
+            0.001913050189614296,
+            0.002036032034084201,
+            -0.012796939350664616,
+            0.020537959411740303,
+            0.035118136554956436,
+            0.006186667364090681,
+            -0.03320508450269699,
+            0.0021624299697577953,
+            0.00015885148604866117,
+            0.0011461219983175397,
+            -0.0069587198086082935,
+            0.026974007487297058,
+            0.008519905619323254,
+            0.01550937071442604,
+            0.030827436596155167,
+            0.0036347953137010336,
+            -0.0031821539159864187,
+            0.012475820258259773,
+            -0.023434864357113838,
+            -0.02425474300980568,
+            0.0027858791872859,
+            0.009244131855666637,
+            0.010405626147985458,
+            0.019226154312491417,
+            -0.012188862077891827,
+            0.004003740847110748,
+            0.01679384708404541,
+            0.019294477999210358,
+            -0.009920531883835793,
+            -0.01816031150519848,
+            0.01516775507479906,
+            0.054084662348032,
+            0.01590564474463463,
+            -0.006565861403942108,
+            -0.01035779993981123,
+            -0.013377686031162739,
+            0.02801251970231533,
+            0.01092488318681717,
+            0.01596030406653881,
+            -0.0003646751865744591,
+            -0.011840414255857468,
+            0.015919310972094536,
+            0.005981697700917721,
+            -0.0014271012041717768,
+            -0.03910820931196213,
+            0.020073361694812775,
+            -0.0058245547115802765,
+            0.02139883302152157,
+            -0.0005619584699161351,
+            -0.0027397610247135162,
+            0.03899889439344406,
+            0.03683988004922867,
+            -0.011491965502500534,
+            0.04443742334842682,
+            -0.007385740056633949,
+            -0.0011409977450966835,
+            -0.0008967422763817012,
+            -0.0026270279195159674,
+            0.01822863519191742,
+            -0.012899423949420452,
+            0.0008950341725721955,
+            -0.0024271823931485415,
+            0.026413757354021072,
+            0.0042702010832726955,
+            -0.014785144478082657,
+            0.0075087216682732105,
+            -0.006442879792302847,
+            -0.02674170769751072,
+            -0.005899710115045309,
+            0.013760296627879143,
+            -0.010685751214623451,
+            0.005783560685813427,
+            0.006521451286971569,
+            0.023065919056534767,
+            0.01738142780959606,
+            -0.019977709278464317,
+            -0.007495057303458452,
+            -0.008526737801730633,
+            0.01261929888278246,
+            0.004813370760530233,
+            -0.012366502545773983,
+            0.022874614223837852,
+            -0.01964975893497467,
+            0.0030284267850220203,
+            0.016165273264050484,
+            -0.007310584653168917,
+            -0.0011854078620672226,
+            -0.02175411395728588,
+            -0.012305011972784996,
+            -0.04001007601618767,
+            0.013849116861820221,
+            -0.016315585002303123,
+            -0.013773960992693901,
+            0.019786404445767403,
+            -0.027848543599247932,
+            0.025197602808475494,
+            -0.009401274845004082,
+            0.024200083687901497,
+            -0.019923051819205284,
+            0.013849116861820221,
+            0.005708404816687107,
+            -0.017531737685203552,
+            0.005339459516108036,
+            -0.012243521399796009,
+            -0.007228596601635218,
+            -0.0020821501966565847,
+            -0.014921790920197964,
+            0.03593801334500313,
+            0.0323852077126503,
+            -0.02568953111767769,
+            0.03142867982387543,
+            0.02034665457904339,
+            0.02071559987962246,
+            -0.005240391008555889,
+            -0.010836062952876091,
+            0.0199503805488348,
+            -0.016903163865208626,
+            0.03323241323232651,
+            0.0027841711416840553,
+            -0.0008284190553240478,
+            0.0015620396006852388,
+            -0.007529218681156635,
+            0.020988892763853073,
+            -0.007761518005281687,
+            0.008806862868368626,
+            0.021275850012898445,
+            -0.003580136690288782,
+            0.010549104772508144,
+            -0.006357475649565458,
+            0.015768999233841896,
+            0.026536738499999046,
+            -0.009968358092010021,
+            0.03733180835843086,
+            0.00813046284019947,
+            -0.00014177068078424782,
+            0.011983892880380154,
+            0.0018310622544959188,
+            -0.019308142364025116,
+            0.006945055443793535,
+            -0.021426161751151085,
+            0.0030403833370655775,
+            0.01434787642210722,
+            -0.004833867773413658,
+            -0.021808771416544914,
+            0.005752814933657646,
+            -0.004150635562837124,
+            -0.010248483158648014,
+            -0.0074745602905750275,
+            -0.01816031150519848,
+            0.011075193993747234,
+            0.018351616337895393,
+            0.028641093522310257,
+            -0.0030335509218275547,
+            -0.002674854127690196,
+            0.007228596601635218,
+            -0.008567731827497482,
+            0.030007557943463326,
+            -0.029187679290771484,
+            0.008308103308081627,
+            -0.0033905399031937122,
+            0.0017012482276186347,
+            0.02139883302152157,
+            -8.753912698011845e-05,
+            -0.010494446381926537,
+            -0.012188862077891827,
+            0.006832322105765343,
+            -0.009818046353757381,
+            -0.007228596601635218,
+            -0.002570661250501871,
+            0.006729837041348219,
+            0.024268407374620438,
+            0.005804057698696852,
+            0.012714951299130917,
+            -0.03402496501803398,
+            0.000319197541102767,
+            -0.003631379222497344,
+            -0.015154089778661728,
+            -0.0037714417558163404,
+            -0.012646627612411976,
+            -0.002927649999037385,
+            -0.030007557943463326,
+            0.008779534138739109,
+            0.001211883150972426,
+            0.006593190599232912,
+            0.004864613059908152,
+            -0.022642314434051514,
+            0.004488835576921701,
+            -0.006641016807407141,
+            -0.0018600996118038893,
+            0.013452841900289059,
+            -0.0033580863382667303,
+            -0.012236688286066055,
+            0.0002893061318900436,
+            0.014019925147294998,
+            -0.014156571589410305,
+            0.006248158402740955,
+            0.005066166631877422,
+            -0.003921752795577049,
+            0.028504448011517525,
+            0.022806290537118912,
+            0.006429214961826801,
+            -0.013678308576345444,
+            0.005694740451872349,
+            -0.002972060116007924,
+            -0.003522061975672841,
+            0.001583390636369586,
+            0.006979216821491718,
+            0.026195121929049492,
+            0.01956777088344097,
+            -0.03629329428076744,
+            -0.01716279238462448,
+            0.004922688007354736,
+            -0.005459025502204895,
+            -0.000369158893590793,
+            0.010077674873173237,
+            -0.012325508520007133,
+            0.001945503638125956,
+            -0.012455322779715061,
+            0.005592255387455225,
+            -0.002287119859829545,
+            -0.010624260641634464,
+            0.01471682172268629,
+            -0.017190122976899147,
+            0.02967960759997368,
+            0.015454712323844433,
+            0.004905607085675001,
+            0.01520874910056591,
+            0.003979827743023634,
+            0.008308103308081627,
+            -0.027602581307291985,
+            0.01479880977421999,
+            -0.008513073436915874,
+            0.007638535927981138,
+            0.004492251668125391,
+            -0.03315042704343796,
+            -0.014443528838455677,
+            0.0014971325872465968,
+            -0.003959330730140209,
+            -0.007399404887109995,
+            0.001276790164411068,
+            0.014662163332104683,
+            0.014375205151736736,
+            -0.004922688007354736,
+            -0.012011222541332245,
+            0.03533677011728287,
+            -0.002997681265696883,
+            0.015386389568448067,
+            -0.001395501778461039,
+            0.013603152707219124,
+            -0.014429864473640919,
+            0.004847532603889704,
+            -0.013275201432406902,
+            -0.001998454099521041,
+            -0.01212053932249546,
+            0.039053551852703094,
+            0.00012116695870645344,
+            -0.00454007787629962,
+            0.0022768713533878326,
+            -0.019267147406935692,
+            -0.006586358416825533,
+            -0.001919882488436997,
+            -0.01647956110537052,
+            0.004181380849331617,
+            0.0200323686003685,
+            0.011758426204323769,
+            -0.03243986517190933,
+            0.024801328778266907,
+            -0.011157182045280933,
+            0.01702614687383175,
+            0.003424701513722539,
+            0.006548780482262373,
+            0.02093423530459404,
+            -0.00820561870932579,
+            -0.033095769584178925,
+            -0.0020052865147590637,
+            -0.002015535021200776,
+            0.004266784992069006,
+            0.03675789386034012,
+            0.02312057837843895,
+            0.020196344703435898,
+            0.011860910803079605,
+            0.007255925796926022,
+            0.01687583513557911,
+            0.018870873376727104,
+            0.008800030685961246,
+            -0.013268369249999523,
+            -0.006603439338505268,
+            -0.030772779136896133,
+            0.015646018087863922,
+            0.023216230794787407,
+            0.015714339911937714,
+            -0.02168579027056694,
+            0.006391637027263641,
+            0.00696555245667696,
+            0.015113096684217453,
+            0.009886370040476322,
+            -0.00903916172683239,
+            -0.010685751214623451,
+            -0.016096951439976692,
+            0.02847711741924286,
+            -0.027602581307291985,
+            0.006569277495145798,
+            0.0008339703199453652,
+            -0.01647956110537052,
+            -0.015577694401144981,
+            0.02545723132789135,
+            0.04391816630959511,
+            -0.02634543366730213,
+            0.003522061975672841,
+            0.009353448636829853,
+            -0.010521776042878628,
+            0.014279552735388279,
+            0.002939606551080942,
+            -0.012673957273364067,
+            0.013145387172698975,
+            -0.019144166260957718,
+            -0.012632963247597218,
+            -0.012359670363366604,
+            -0.0320572555065155,
+            0.0034349497873336077,
+            0.02913302183151245,
+            0.022314364090561867,
+            0.006381388753652573,
+            -0.00011241305037401617,
+            0.016096951439976692,
+            0.0004517872876022011,
+            0.019718080759048462,
+            -0.01627459190785885,
+            0.009954692795872688,
+            -0.0008621536544524133,
+            0.014730486087501049,
+            -0.008465247228741646,
+            0.01596030406653881,
+            -0.0003898693830706179,
+            0.0007088534184731543,
+            0.00986587256193161,
+            -0.0005717799649573863,
+            -0.014812474139034748,
+            0.00993419624865055,
+            0.005144738592207432,
+            0.014949120581150055,
+            -0.006207164376974106,
+            0.0075565483421087265,
+            -0.03011687472462654,
+            0.01627459190785885,
+            0.02462368831038475,
+            0.009066491387784481,
+            -0.026851026341319084,
+            -0.000849770032800734,
+            0.00523355882614851,
+            -0.010740410536527634,
+            -0.0012486068299040198,
+            0.01837894693017006,
+            0.024159090593457222,
+            0.0029771842528134584,
+            0.028750410303473473,
+            0.001836186507716775,
+            0.019021185114979744,
+            -0.012072713114321232,
+            -0.003141160123050213,
+            0.013254704885184765,
+            0.0012725199339911342,
+            0.0027448853943496943,
+            0.02749326452612877,
+            -0.016971487551927567,
+            0.015482041984796524,
+            -0.00021991536777932197,
+            0.020291997119784355,
+            -0.0064804572612047195,
+            0.00903916172683239,
+            -0.00048039763350971043,
+            0.026564067229628563,
+            0.005732318386435509,
+            0.0065897745080292225,
+            0.0582660436630249,
+            -0.021111875772476196,
+            0.019977709278464317,
+            0.006856235209852457,
+            -0.006576110143214464,
+            -0.015400053933262825,
+            -0.0014894461492076516,
+            -0.01989572122693062,
+            0.026482079178094864,
+            0.014293218031525612,
+            0.0025433318223804235,
+            -0.0026663136668503284,
+            -0.007392572239041328,
+            0.011027367785573006,
+            -0.0012528770603239536,
+            0.0039729950949549675,
+            0.004680140409618616,
+            -0.012933585792779922,
+            0.02394045703113079,
+            0.007932325825095177,
+            -0.004051567055284977,
+            -0.010337303392589092,
+            -0.006463376805186272,
+            0.009018665179610252,
+            -0.016629870980978012,
+            0.001118792686611414,
+            -0.00013034789299126714,
+            0.02250566892325878,
+            -0.017572732642292976,
+            0.035282112658023834,
+            -0.014006259851157665,
+            -0.017449749633669853,
+            -0.010754074901342392,
+            0.011655941605567932,
+            0.024719340726733208,
+            0.01381495501846075,
+            0.020373983308672905,
+            0.01596030406653881,
+            -0.0034264095593243837,
+            0.019526775926351547,
+            -0.0067810798063874245,
+            -0.014033589512109756,
+            0.007064620964229107,
+            0.008000649511814117,
+            -0.0039115045219659805,
+            0.021412497386336327,
+            0.009408107027411461,
+            0.0008587374468334019,
+            0.002954979194328189,
+            -0.0030608803499490023,
+            -0.018419940024614334,
+            -0.005691324360668659,
+            -0.0003687318821903318,
+            0.0046767243184149265,
+            0.0023229895159602165,
+            -0.01505843736231327,
+            0.020988892763853073,
+            -0.0016773350071161985,
+            -0.005534180905669928,
+            0.025429902598261833,
+            0.011123020201921463,
+            -0.0039832438342273235,
+            -0.012468988075852394,
+            -0.012018054723739624,
+            -0.004984179046005011,
+            0.0009881245205178857,
+            0.005383869633078575,
+            0.012175197713077068,
+            0.003908087965101004,
+            -0.006350643467158079,
+            0.005735734477639198,
+            -0.0017746955854818225,
+            0.00561275240033865,
+            0.016383908689022064,
+            0.009838543832302094,
+            0.008007481694221497,
+            -0.021180197596549988,
+            -0.015017443336546421,
+            0.015331730246543884,
+            0.005636665504425764,
+            0.005257471930235624,
+            0.00204286421649158,
+            0.024664683267474174,
+            0.02026466652750969,
+            -0.029734265059232712,
+            -0.011444139294326305,
+            -0.012284515425562859,
+            0.03503614664077759,
+            0.015946639701724052,
+            -0.016588877886533737,
+            0.01174476183950901,
+            0.0011068361345678568,
+            0.004492251668125391,
+            -0.0038500134833157063,
+            0.009660903364419937,
+            0.011006870307028294,
+            -0.021822435781359673,
+            -0.028285812586545944,
+            -0.02064727619290352,
+            -0.013323027640581131,
+            0.032658498734235764,
+            0.02529325522482395,
+            -0.02410443127155304,
+            0.004909023176878691,
+            0.03413427993655205,
+            -0.00540095055475831,
+            -0.009654071182012558,
+            0.006104679778218269,
+            -0.00011123873991891742,
+            -0.01479880977421999,
+            -0.008793198503553867,
+            -0.021330509334802628,
+            0.014525516889989376,
+            0.0034691113978624344,
+            0.005090079735964537,
+            -0.004167716484516859,
+            -0.011997557245194912,
+            -0.027465933933854103,
+            -0.009025497362017632,
+            -0.002867867238819599,
+            -0.0003439646970946342,
+            -0.004475170746445656,
+            0.012967746704816818,
+            -0.0011307492386549711,
+            0.004960265941917896,
+            0.001984789501875639,
+            -0.015031108632683754,
+            -0.007823009043931961,
+            -0.027301957830786705,
+            -0.016329249367117882,
+            0.0034810679499059916,
+            -0.013186381198465824,
+            0.004543493967503309,
+            -0.008929844945669174,
+            -0.014484522864222527,
+            -0.010610596276819706,
+            0.011286996304988861,
+            -0.006425798870623112,
+            0.030772779136896133,
+            0.008314935490489006,
+            -0.011464636772871017,
+            0.0015919309807941318,
+            -0.0022051320411264896,
+            -0.023011259734630585,
+            0.019540440291166306,
+            0.005541013088077307,
+            0.009633573703467846,
+            -0.031374022364616394,
+            0.004471754655241966,
+            0.012933585792779922,
+            0.006531700026243925,
+            -0.023065919056534767,
+            -0.010808733291924,
+            -0.0007246531313285232,
+            -0.0035323104821145535,
+            -0.018542921170592308,
+            -0.014675827696919441,
+            -0.009373946115374565,
+            -0.012065880931913853,
+            -0.020729264244437218,
+            -0.019376466050744057,
+            0.008984503336250782,
+            0.00585529999807477,
+            0.009592579677700996,
+            -0.0030420913826674223,
+            0.007645368110388517,
+            0.013159052468836308,
+            0.0050012595020234585,
+            0.022614985704421997,
+            0.004707470070570707,
+            0.0037270316388458014,
+            0.012899423949420452,
+            -0.018433604389429092,
+            -0.004516164772212505,
+            -0.0016730648931115866,
+            0.02138516865670681,
+            0.007003129925578833,
+            -0.0001629081816645339,
+            0.0004842408234253526,
+            0.02305225469172001,
+            0.0030403833370655775,
+            0.01574167050421238,
+            0.013343525119125843,
+            -0.00589287793263793,
+            0.03216657042503357,
+            -0.00024382848641835153,
+            -0.0004850948753301054,
+            -0.002987432759255171,
+            0.021795107051730156,
+            -0.005845051258802414,
+            0.0004086582630407065,
+            0.013760296627879143,
+            -0.01205221563577652,
+            -0.033095769584178925,
+            -0.005387285724282265,
+            0.026919348165392876,
+            0.015236077830195427,
+            -0.020169014111161232,
+            -0.01000935211777687,
+            0.0016312168445438147,
+            -0.005800641607493162,
+            0.015181419439613819,
+            -0.0019164662808179855,
+            0.00870437826961279,
+            0.017723042517900467,
+            -0.008048475719988346,
+            -0.020155349746346474,
+            0.027834879234433174,
+            0.02732928842306137,
+            0.010193824768066406,
+            -0.038452308624982834,
+            -0.014853468164801598,
+            0.010938547551631927,
+            -0.016670865938067436,
+            -0.019704416394233704,
+            -0.014006259851157665,
+            -0.014730486087501049,
+            -0.007536051329225302,
+            0.01694415882229805,
+            0.008328600786626339,
+            -0.004136971198022366,
+            0.020510630682110786,
+            0.026646055281162262,
+            0.004587904084473848,
+            -0.0035767205990850925,
+            -0.017422420904040337,
+            -0.006593190599232912,
+            -0.0208659116178751,
+            0.0065112030133605,
+            -0.022423680871725082,
+            -0.012906256131827831,
+            -0.010111836716532707,
+            0.002377648139372468,
+            0.02809450775384903,
+            0.005482938606292009,
+            0.03110072948038578,
+            0.0063643078319728374,
+            -0.009353448636829853,
+            -0.008526737801730633,
+            -0.0039729950949549675,
+            -0.0022717469837516546,
+            -0.013548494316637516,
+            -0.007986984215676785,
+            -0.0010163078550249338,
+            -0.005725485738366842,
+            0.012359670363366604,
+            0.016206268221139908,
+            0.004543493967503309,
+            0.025197602808475494,
+            -2.132431836798787e-05,
+            -0.002203423762694001,
+            0.025402573868632317,
+            -0.01590564474463463,
+            0.003627962898463011,
+            0.012475820258259773,
+            -0.0021282683592289686,
+            -0.014880796894431114,
+            0.011198176071047783,
+            0.0325491838157177,
+            -0.014566510915756226,
+            -0.011983892880380154,
+            -0.011983892880380154,
+            0.012332341633737087,
+            -0.004229207523167133,
+            0.03517279401421547,
+            0.021426161751151085,
+            -0.03533677011728287,
+            0.0027226803358644247,
+            -0.019034849479794502,
+            -0.0017951925983652472,
+            -0.023817474022507668,
+            0.0020240754820406437,
+            0.015687011182308197,
+            0.021494485437870026,
+            -0.002029199618846178,
+            0.006524867378175259,
+            -0.02130318060517311,
+            0.01299507636576891,
+            0.008677048608660698,
+            -0.0003403350419830531,
+            0.010665254667401314,
+            -0.0038500134833157063,
+            0.007358410861343145,
+            -0.011109355837106705,
+            -0.013152220286428928,
+            -0.002973768161609769,
+            0.01612428016960621,
+            0.01989572122693062,
+            -0.007713691331446171,
+            0.03028085082769394,
+            -0.002451095497235656,
+            -0.006360891740769148,
+            -0.0368945375084877,
+            -0.0032180235721170902,
+            -0.003962746821343899,
+            0.006705923937261105,
+            -0.0013434053398668766,
+            -0.011164014227688313,
+            -0.011034199967980385,
+            -0.015946639701724052,
+            -0.0030079297721385956,
+            0.004885110072791576,
+            -0.0021026472095400095,
+            -0.0053804535418748856,
+            0.002242709742859006,
+            -0.006029524374753237,
+            0.0054897707886993885,
+            -0.015003778971731663,
+            0.0053087142296135426,
+            0.014334211125969887,
+            -0.006705923937261105,
+            0.0207019355148077,
+            0.022683309391140938,
+            0.01859758049249649,
+            0.0016030335100367665,
+            0.019745411351323128,
+            -0.011020535603165627,
+            -0.017436085268855095,
+            -0.01046711765229702,
+            0.01740875653922558,
+            -0.008314935490489006,
+            0.01852925680577755,
+            0.013685140758752823,
+            0.008888850919902325,
+            0.0005380453658290207,
+            -0.013760296627879143,
+            -0.00783667340874672,
+            0.0009069907246157527,
+            0.01572800613939762,
+            0.01889820210635662,
+            0.004393183160573244,
+            0.0009753139456734061,
+            -0.018624909222126007,
+            0.02200007624924183,
+            -0.015618688426911831,
+            0.002056528814136982,
+            -0.004727966617792845,
+            -0.001228109933435917,
+            0.014088247902691364,
+            -0.007727356161922216,
+            0.008950341492891312,
+            -0.02034665457904339,
+            0.011840414255857468,
+            -0.01117767859250307,
+            -0.0033444215077906847,
+            -0.00602610781788826,
+            -0.012844765558838844,
+            -0.01980006881058216,
+            -0.02508828602731228,
+            0.007085117977112532,
+            -0.0023878966458141804,
+            0.019554104655981064,
+            -0.007631703745573759,
+            -0.020811252295970917,
+            -0.014675827696919441,
+            -0.007986984215676785,
+            0.012223023921251297,
+            0.01747708022594452,
+            -0.00873854011297226,
+            -0.018010001629590988,
+            0.029843581840395927,
+            -0.013992595486342907,
+            -0.008970838971436024,
+            -0.01438887044787407,
+            -0.01672552339732647,
+            -0.0017798198387026787,
+            0.006186667364090681,
+            -0.012585137039422989,
+            -0.014675827696919441,
+            0.014279552735388279,
+            0.0023246975615620613,
+            -0.01649322547018528,
+            -0.016698194667696953,
+            0.005281385034322739,
+            -0.011246002279222012,
+            0.02205473557114601,
+            -0.015700675547122955,
+            0.016520554199814796,
+            -0.011450971476733685,
+            0.01596030406653881,
+            -0.02093423530459404,
+            -0.01973174512386322,
+            0.005834802985191345,
+            -0.011799420230090618,
+            0.029488300904631615,
+            0.001263125566765666,
+            -0.017887018620967865,
+            0.008991335518658161,
+            -0.015290736220777035,
+            0.023544181138277054,
+            -0.0016824592603370547,
+            0.01516775507479906,
+            -0.013220543041825294,
+            0.014511851593852043,
+            0.0012964331544935703,
+            0.014853468164801598,
+            0.008198786526918411,
+            0.038588955998420715,
+            -0.006572693586349487,
+            -0.008943509310483932,
+            0.0008578834240324795,
+            0.008157792501151562,
+            0.0027551339007914066,
+            0.007358410861343145,
+            0.024077102541923523,
+            -0.0012938709696754813,
+            0.005575174931436777,
+            -0.01904851384460926,
+            -0.015290736220777035,
+            -0.007317416835576296,
+            0.015359059907495975,
+            0.017818694934248924,
+            -0.011116188019514084,
+            -0.006784495897591114,
+            0.0055273487232625484,
+            -0.023380205035209656,
+            0.010221153497695923,
+            -0.016247261315584183,
+            -0.013589488342404366,
+            -0.010173327289521694,
+            -0.005964617244899273,
+            -0.008540402166545391,
+            0.011047864332795143,
+            0.014252224005758762,
+            -0.025402573868632317,
+            0.01204538345336914,
+            0.016301920637488365,
+            -0.00626865541562438,
+            0.025129280984401703,
+            -0.027547921985387802,
+            -0.01121867261826992,
+            -0.0023520267568528652,
+            0.011512462981045246,
+            -0.0012725199339911342,
+            -0.006432631053030491,
+            0.009592579677700996,
+            -0.012332341633737087,
+            -0.013691972941160202,
+            -0.002029199618846178,
+            -0.006248158402740955,
+            -0.011170846410095692,
+            0.00023870424774941057,
+            -0.0006435193354263902,
+            0.0065521965734660625,
+            0.012414328753948212,
+            0.00544194458052516,
+            -0.00429411418735981,
+            -0.00956525094807148,
+            0.019485782831907272,
+            0.02463735267519951,
+            0.01596030406653881,
+            0.005510267801582813,
+            -0.01604229211807251,
+            -0.00433169212192297,
+            -0.008547235280275345,
+            -0.02545723132789135,
+            -0.014539181254804134,
+            0.01171059999614954,
+            0.0200323686003685,
+            0.0028217490762472153,
+            -0.02839512936770916,
+            -0.004932936280965805,
+            0.0008907640003599226,
+            0.020291997119784355,
+            0.0019677088130265474,
+            -0.004700637422502041,
+            0.004078896250575781,
+            0.008232948370277882,
+            0.0100571783259511,
+            -0.027069659903645515,
+            -0.015850987285375595,
+            -0.010344135574996471,
+            -0.025511890649795532,
+            0.022164052352309227,
+            0.02817649580538273,
+            0.0134391775354743,
+            0.004106225445866585,
+            0.013500668108463287,
+            0.013773960992693901,
+            0.031975265592336655,
+            -0.00647704117000103,
+            0.01754540205001831,
+            6.079698869143613e-05,
+            -0.007802511565387249,
+            0.018952861428260803,
+            -0.00436585396528244,
+            -0.011758426204323769,
+            -0.013111226260662079,
+            0.010152830742299557,
+            -0.0066615138202905655,
+            -0.008403755724430084,
+            -0.019321806728839874,
+            0.012257185764610767,
+            0.014443528838455677,
+            -0.00865655206143856,
+            -0.010590098798274994,
+            0.004956849385052919,
+            0.007392572239041328,
+            0.004024237859994173,
+            -0.0017217451240867376,
+            -0.0067810798063874245,
+            -0.017067139968276024,
+            -0.005404366645962,
+            -0.0146894920617342,
+            -0.009127981960773468,
+            -0.004174548666924238,
+            0.015113096684217453,
+            -0.017067139968276024,
+            0.018884537741541862,
+            0.014593839645385742,
+            0.014470857568085194,
+            0.019977709278464317,
+            -0.008048475719988346,
+            0.010050345212221146,
+            0.01754540205001831,
+            -0.0014373497106134892,
+            0.0019113420275971293,
+            -0.005605920217931271,
+            0.0007951114675961435,
+            0.008055307902395725,
+            0.0027841711416840553,
+            0.0011162306182086468,
+            -0.003083085408434272,
+            -0.025853507220745087,
+            -0.01876155659556389,
+            -0.0037782741710543633,
+            -0.020196344703435898,
+            -0.011430474929511547,
+            0.003417869098484516,
+            0.0028012520633637905,
+            -0.007167105562984943,
+            -0.012530478648841381,
+            0.0022512502036988735,
+            -0.004495667759329081,
+            0.006658097729086876,
+            -0.0031138306949287653,
+            -0.009052827022969723,
+            0.010672086849808693,
+            0.013138554990291595,
+            -0.016534218564629555,
+            -0.015796327963471413,
+            0.005110576748847961,
+            0.0030386752914637327,
+            0.018870873376727104,
+            -0.0013519456842914224,
+            -0.003747528651729226,
+            -0.008601893670856953,
+            -0.001989913871511817,
+            -0.010521776042878628,
+            -0.007003129925578833,
+            0.008759036660194397,
+            0.016288256272673607,
+            0.005568342283368111,
+            -0.024077102541923523,
+            0.030007557943463326,
+            -0.006559029221534729,
+            0.038534294813871384,
+            0.005564926192164421,
+            -0.005674243438988924,
+            -0.0014313714345917106,
+            0.00956525094807148,
+            -0.011170846410095692,
+            0.012387000024318695,
+            0.009168975986540318,
+            0.010023016482591629,
+            -0.006572693586349487,
+            -0.007057788781821728,
+            0.002792711602523923,
+            0.009715561755001545,
+            -0.0008459268719889224,
+            9.453157690586522e-05,
+            -0.016684530302882195,
+            -0.014320546761155128,
+            -0.008000649511814117,
+            0.026085805147886276,
+            0.005978281609714031,
+            0.004188213497400284,
+            -0.006299400702118874,
+            -0.007986984215676785,
+            -0.0112255048006773,
+            -0.0046152337454259396,
+            -0.03418894112110138,
+            0.011157182045280933,
+            -0.028258483856916428,
+            0.004587904084473848,
+            0.007023626938462257,
+            -5.860851160832681e-05,
+            0.002843954134732485,
+            0.0063028172589838505,
+            0.010091339237987995,
+            0.014101912267506123,
+            0.00741990189999342,
+            -0.013534829951822758,
+            0.0015987633960321546,
+            0.017490744590759277,
+            -0.0004169851599726826,
+            0.0012921629240736365,
+            -0.009175808168947697,
+            -0.016657201573252678,
+            -0.0012981412000954151,
+            -0.016069620847702026,
+            0.002094106748700142,
+            0.010979541577398777,
+            -0.000233366503380239,
+            0.005250639282166958,
+            0.01027581188827753,
+            0.004550326615571976,
+            -0.007235428784042597,
+            6.223818490980193e-05,
+            0.010227985680103302,
+            0.012557808309793472,
+            -0.00457765581086278,
+            -0.016602542251348495,
+            0.015687011182308197,
+            0.016383908689022064,
+            0.02040131390094757,
+            0.0016371951205655932,
+            -0.019718080759048462,
+            -0.0025484561920166016,
+            0.009326119907200336,
+            -0.009346616454422474,
+            0.01966342329978943,
+            0.0013510916614905,
+            0.006012443453073502,
+            0.02146715484559536,
+            0.015482041984796524,
+            -0.009585747495293617,
+            -0.014470857568085194,
+            0.005093495827168226,
+            -0.02327088825404644,
+            -0.0005871526664122939,
+            0.011150349862873554,
+            -0.031073400750756264,
+            0.01627459190785885,
+            -0.020975228399038315,
+            0.012209359556436539,
+            -1.091570175049128e-05,
+            -0.022847285494208336,
+            -0.020087026059627533,
+            0.012837933376431465,
+            0.022095728665590286,
+            0.021193861961364746,
+            -0.0077341883443295956,
+            -0.024719340726733208,
+            -0.007119279354810715,
+            0.02674170769751072,
+            0.006972384639084339,
+            -0.016233596950769424,
+            0.025798847898840904,
+            -0.0075087216682732105,
+            -0.030827436596155167,
+            -0.006125176791101694,
+            -0.02764357440173626,
+            -0.007201267406344414,
+            -0.009025497362017632,
+            0.018706897273659706,
+            -0.027315624058246613,
+            -0.009442268870770931,
+            0.014402534812688828,
+            -0.005076415371149778,
+            -0.00630623335018754,
+            -0.00134169717784971,
+            0.014006259851157665,
+            0.003243644954636693,
+            -0.006808409001678228,
+            -0.0030557559803128242,
+            0.014170235954225063,
+            0.015714339911937714,
+            -0.01559135876595974,
+            -0.009018665179610252,
+            -0.013637314550578594,
+            -0.009237299673259258,
+            0.025716859847307205,
+            -0.00025087432004511356,
+            0.0022205046843737364,
+            -0.011717432178556919,
+            -0.008888850919902325,
+            0.007515554316341877,
+            0.02025100216269493,
+            0.00602269172668457,
+            0.001989913871511817,
+            0.014101912267506123,
+            -0.0017798198387026787,
+            -0.011539791710674763,
+            -1.084230825654231e-05,
+            0.002377648139372468,
+            0.01269445475190878,
+            -0.009770220145583153,
+            0.020455971360206604,
+            -0.007167105562984943,
+            0.014006259851157665,
+            -0.022614985704421997,
+            0.007276422809809446,
+            -0.019185159355401993,
+            -0.008431085385382175,
+            -0.0012964331544935703,
+            -0.004444425459951162,
+            0.00139293959364295,
+            -0.011389480903744698,
+            -0.0032077752985060215,
+            0.0099751902744174,
+            0.008123630657792091,
+            -0.025019962340593338,
+            -0.0012938709696754813,
+            -0.01597396843135357,
+            -0.015932975336909294,
+            -0.013514332473278046,
+            0.014662163332104683,
+            0.015632351860404015,
+            0.0028337056282907724,
+            0.0020462805405259132,
+            -0.020291997119784355,
+            -0.011457803659141064,
+            0.009749723598361015,
+            -0.011587617918848991,
+            0.006319897715002298,
+            -0.0191168375313282,
+            -0.022492004558444023,
+            0.01035096775740385,
+            0.04465605691075325,
+            -0.0030096378177404404,
+            -0.015523036010563374,
+            0.013828619383275509,
+            -0.018843544647097588,
+            -0.012632963247597218,
+            -0.009824879467487335,
+            -0.010111836716532707,
+            -0.005769895855337381,
+            -0.0016149900620803237,
+            0.0033000116236507893,
+            0.02522493340075016,
+            -0.0005888607702217996,
+            0.010665254667401314,
+            0.004195045679807663,
+            0.01762739010155201,
+            -0.005947536323219538,
+            0.0009744599228724837,
+            -0.002644108608365059,
+            0.00280466815456748,
+            0.01597396843135357,
+            0.00010648814350133762,
+            0.005674243438988924,
+            0.008376426994800568,
+            0.0011708892416208982,
+            0.015345395542681217,
+            -0.0068972292356193066,
+            -0.025416238233447075,
+            0.02101622335612774,
+            -0.009018665179610252,
+            -0.004635730292648077,
+            -0.011778922751545906,
+            -0.01550937071442604,
+            0.035965342074632645,
+            0.003768025664612651,
+            -0.008964006789028645,
+            0.018570251762866974,
+            0.002572369296103716,
+            0.0058860452845692635,
+            0.007050956133753061,
+            -0.01876155659556389,
+            -0.010173327289521694,
+            0.018392611294984818,
+            -0.011949731037020683,
+            -0.013541662134230137,
+            0.01257830485701561,
+            -0.0016303628217428923,
+            0.007604374550282955,
+            0.0015031108632683754,
+            -0.013015573844313622,
+            -0.009005000814795494,
+            -0.006261823233217001,
+            0.013193213380873203,
+            -0.0048680296167731285,
+            0.018010001629590988,
+            0.00786400306969881,
+            0.008888850919902325,
+            -0.007727356161922216,
+            0.022259704768657684,
+            -0.020510630682110786,
+            -0.007399404887109995,
+            0.0033375893253833055,
+            0.021070880815386772,
+            -0.014019925147294998,
+            0.013008740730583668,
+            -0.0021880511194467545,
+            -0.010631092824041843,
+            0.0033973720856010914,
+            0.0004987595020793378,
+            0.0009172392310574651,
+            -0.0007302043959498405,
+            0.012762777507305145,
+            -0.0029310660902410746,
+            0.002444263081997633,
+            -0.018802549690008163,
+            0.025921829044818878,
+            -0.008724874816834927,
+            0.013384519144892693,
+            -0.007891331799328327,
+            0.007003129925578833,
+            0.02363983355462551,
+            0.03304110839962959,
+            0.0029242339078336954,
+            0.008444749750196934,
+            -0.0029037368949502707,
+            0.01602862775325775,
+            0.004024237859994173,
+            0.00609784759581089,
+            0.007454063277691603,
+            0.014675827696919441,
+            -0.017203787341713905,
+            -0.020660942420363426,
+            0.013316195458173752,
+            -0.015605023130774498,
+            -0.012188862077891827,
+            -0.016165273264050484,
+            -0.00873854011297226,
+            0.018802549690008163,
+            0.0215354785323143,
+            -0.0015210456913337111,
+            0.006142257712781429,
+            0.016465896740555763,
+            -0.007536051329225302,
+            -0.00108206900767982,
+            -0.0013177840737625957,
+            0.006869899574667215,
+            -0.00034887544461525977,
+            -0.016438566148281097,
+            0.01694415882229805,
+            -0.02273796685039997,
+            0.00948326289653778,
+            -0.007146609015762806,
+            0.0061935000121593475,
+            0.014224894344806671,
+            0.01295408234000206,
+            0.015523036010563374,
+            0.015181419439613819,
+            0.0055615101009607315,
+            0.006753750145435333,
+            -0.005653746426105499,
+            0.012981412000954151,
+            0.009162143804132938,
+            0.00820561870932579,
+            0.014129241928458214,
+            -0.0027380529791116714,
+            -0.01934913545846939,
+            -0.005749398842453957,
+            0.01882988028228283,
+            -0.007966487668454647,
+            -0.009244131855666637,
+            0.0139379370957613,
+            0.023680828511714935,
+            -0.021658461540937424,
+            0.02649574540555477,
+            -0.0031428681686520576,
+            -0.014484522864222527,
+            0.035063475370407104,
+            -0.00564691424369812,
+            0.01949944719672203,
+            -0.021439826115965843,
+            -0.037003856152296066,
+            -0.00477920938283205,
+            0.0028337056282907724,
+            0.016752853989601135,
+            0.005113992840051651,
+            0.0012195694725960493,
+            -0.018037330359220505,
+            -0.006849403027445078,
+            0.01672552339732647,
+            -0.01587831601500511,
+            0.01434787642210722,
+            -0.0039729950949549675,
+            0.0207019355148077,
+            0.0014740735059604049,
+            0.015331730246543884,
+            -0.012188862077891827,
+            -0.007570212706923485,
+            0.030253522098064423,
+            0.005178899969905615,
+            -0.010788236744701862,
+            0.017367763444781303,
+            -0.0043795183300971985,
+            -0.002268330892547965,
+            -0.013275201432406902,
+            0.006572693586349487,
+            -0.0007169668097048998,
+            0.0020992308855056763,
+            0.004652811214327812,
+            -0.003959330730140209,
+            0.02230069972574711,
+            -0.005998778622597456,
+            0.010002519004046917,
+            0.005100328475236893,
+            -0.006104679778218269,
+            0.0055717583745718,
+            0.01589198037981987,
+            -0.02115286886692047,
+            0.010528608225286007,
+            -0.0064087179489433765,
+            0.006289152428507805,
+            -0.01175159402191639,
+            0.00022632066975347698,
+            -0.0070304591208696365,
+            0.004755296278744936,
+            0.01178575586527586,
+            -0.013275201432406902,
+            -0.0032795146107673645,
+            0.007522386498749256,
+            -0.004611817188560963,
+            0.025197602808475494,
+            0.021658461540937424,
+            -0.013124890625476837,
+            -0.013664644211530685,
+            0.01382178720086813,
+            -0.003284638747572899,
+            0.006371140480041504,
+            0.005684491712599993,
+            0.004382934421300888,
+            -0.034899499267339706,
+            -0.012018054723739624,
+            0.004099393263459206,
+            -0.016957823187112808,
+            0.001854975474998355,
+            -0.009223634377121925,
+            -0.024282071739435196,
+            0.022396352142095566,
+            -0.004266784992069006,
+            0.021125540137290955,
+            0.0074472310952842236,
+            -0.011840414255857468,
+            -0.016780182719230652,
+            0.014880796894431114,
+            -0.01649322547018528,
+            -0.0020701936446130276,
+            -0.001913050189614296,
+            -0.020223673433065414,
+            0.015345395542681217,
+            -0.004150635562837124,
+            0.010863391682505608,
+            0.001276790164411068,
+            -0.01269445475190878,
+            -0.0036177143920212984,
+            -0.0077341883443295956,
+            0.008301271125674248,
+            -0.018283294513821602,
+            0.0071397763676941395,
+            0.011526127345860004,
+            -0.002435722853988409,
+            -0.019267147406935692,
+            -0.00782984122633934,
+            -2.091064197884407e-05,
+            0.03670323267579079,
+            -0.0261268001049757,
+            -0.0024015612434595823,
+            -7.245997949212324e-06,
+            -0.012373334728181362,
+            0.0035015649627894163,
+            -0.009271460585296154,
+            -0.0037987709511071444,
+            -0.01964975893497467,
+            -0.006617103703320026,
+            0.007501889485865831,
+            0.008000649511814117,
+            -0.013288866728544235,
+            -0.015482041984796524,
+            -0.010453452356159687,
+            -0.025033628568053246,
+            -0.008718042634427547,
+            -0.02967960759997368,
+            -0.017518073320388794,
+            0.0018652238650247455,
+            -0.002572369296103716,
+            0.02235535718500614,
+            -0.019485782831907272,
+            0.010098172351717949,
+            -0.00495001720264554,
+            0.018939197063446045,
+            -0.008595061488449574,
+            -0.004031070042401552,
+            0.006863067392259836,
+            0.0048680296167731285,
+            0.0009257796336896718,
+            -0.008232948370277882,
+            -0.003805603366345167,
+            -0.00786400306969881,
+            -0.00960624497383833,
+            0.013985763303935528,
+            -0.006319897715002298,
+            0.008977671153843403,
+            -0.017190122976899147,
+            -0.000306386937154457,
+            -0.012585137039422989,
+            -0.008048475719988346,
+            -0.009442268870770931,
+            0.003347837831825018,
+            0.014279552735388279,
+            -0.009537921287119389,
+            -0.03774174675345421,
+            -0.0019010936375707388,
+            -0.0051549868658185005,
+            0.0034298256505280733,
+            -0.01874789223074913,
+            0.015782663598656654,
+            -0.011423642747104168,
+            0.011170846410095692,
+            0.011614947579801083,
+            0.005086663644760847,
+            0.008116798475384712,
+            0.006422382779419422,
+            0.015386389568448067,
+            -0.0011025660205632448,
+            -0.004611817188560963,
+            -0.02318890020251274,
+            0.018460934981703758,
+            -0.01249631680548191,
+            -0.008014313876628876,
+            -0.008403755724430084,
+            0.041841138154268265,
+            0.02266964502632618,
+            0.01212053932249546,
+            0.002522834809496999,
+            0.02891438640654087,
+            0.0031240792013704777,
+            0.018132982775568962,
+            0.011813084594905376,
+            -0.004973930306732655,
+            -0.006418966688215733,
+            -0.01295408234000206,
+            -0.029460972175002098,
+            0.016957823187112808,
+            0.008232948370277882,
+            0.029925569891929626,
+            -0.028504448011517525,
+            0.015031108632683754,
+            -0.005462441593408585,
+            -0.01092488318681717,
+            -0.010023016482591629,
+            0.008519905619323254,
+            -0.0024596359580755234,
+            0.031592655926942825,
+            0.00911431759595871,
+            -0.0029515631031244993,
+            0.008191954344511032,
+            0.013145387172698975,
+            -0.012885759584605694,
+            0.01077457144856453,
+            0.01574167050421238,
+            0.03960013762116432,
+            0.009886370040476322,
+            2.0523655621218495e-05,
+            0.035118136554956436,
+            0.016014963388442993,
+            -0.0036792054306715727,
+            -0.0074745602905750275,
+            -0.016206268221139908,
+            -0.012325508520007133,
+            -0.006439463701099157,
+            0.01988205686211586,
+            -0.005602504126727581,
+            0.009162143804132938,
+            0.007563380524516106,
+            0.00783667340874672,
+            -0.02899637445807457,
+            0.013903775252401829,
+            0.003149700351059437,
+            0.013808122836053371,
+            0.014730486087501049,
+            -0.023475859314203262,
+            0.011977060697972775,
+            -0.02176777832210064,
+            -0.0035903851967304945,
+            -0.001383545226417482,
+            -0.0008151814108714461,
+            0.01792801357805729,
+            -0.023079583421349525,
+            -0.013773960992693901,
+            -0.004673308227211237,
+            0.014525516889989376,
+            0.008007481694221497,
+            0.0103919617831707,
+            0.012188862077891827,
+            0.002838829765096307,
+            -0.005377037450671196,
+            -0.0062549905851483345,
+            0.0018788885790854692,
+            -0.003733864054083824,
+            -0.002818332752212882,
+            -0.006497538182884455,
+            0.009018665179610252,
+            0.006091014947742224,
+            -0.004977346397936344,
+            0.009503760375082493,
+            -0.01627459190785885,
+            -0.013596320524811745,
+            0.02582617662847042,
+            0.002526251133531332,
+            -0.0013135139597579837,
+            0.005295049399137497,
+            0.010952211916446686,
+            0.01475781574845314,
+            0.009770220145583153,
+            0.004013989120721817,
+            0.013241039589047432,
+            -0.005595671944320202,
+            -0.007009962573647499,
+            0.019690752029418945,
+            0.0031052904669195414,
+            -0.018406275659799576,
+            0.0026031145825982094,
+            -0.010870223864912987,
+            -0.0012272557942196727,
+            0.010139165446162224,
+            -0.010405626147985458,
+            -0.015796327963471413,
+            0.0011102523421868682,
+            -0.02275163307785988,
+            -0.004645979031920433,
+            0.01709447056055069,
+            0.0073310816660523415,
+            0.024268407374620438,
+            0.002584325848147273,
+            -0.005985114257782698,
+            -0.0006883564637973905,
+            -0.024159090593457222,
+            0.005284801125526428,
+            -0.003614298300817609,
+            -0.008027978241443634,
+            0.014402534812688828,
+            -0.001230672001838684,
+            0.00241010170429945,
+            -0.011123020201921463,
+            -0.00862239021807909,
+            0.006412134040147066,
+            0.012284515425562859,
+            0.03172930330038071,
+            -0.004266784992069006,
+            -0.008718042634427547,
+            -0.029570288956165314,
+            -0.003781690262258053,
+            0.024364059790968895,
+            -0.003819267963990569,
+            -0.020756594836711884,
+            0.01951311156153679,
+            0.02537524327635765,
+            -0.0008514781366102397,
+            -0.0011316033778712153,
+            -0.007153441198170185,
+            -0.004891942720860243,
+            0.007918661460280418,
+            0.019403794780373573,
+            -0.018337951973080635,
+            0.010439787991344929,
+            0.014976450242102146,
+            0.03817901387810707,
+            0.0025006297510117292,
+            -0.004543493967503309,
+            -0.015523036010563374,
+            -0.0014843220124021173,
+            -0.018556587398052216,
+            0.02861376479268074,
+            -0.000738317787181586,
+            0.03304110839962959,
+            0.020811252295970917,
+            0.008492575958371162,
+            0.0028132086154073477,
+            -0.010808733291924,
+            0.059468530118465424,
+            -0.019089506939053535,
+            -0.007782014552503824,
+            -0.011983892880380154,
+            -0.023257223889231682,
+            -0.010419291444122791,
+            -0.0028217490762472153,
+            -0.012878927402198315,
+            0.015318065881729126,
+            -0.01649322547018528,
+            0.00585529999807477,
+            0.005981697700917721,
+            0.0011879700468853116,
+            0.008451581932604313,
+            -0.022232376039028168,
+            0.021180197596549988,
+            -0.02275163307785988,
+            -0.004400015342980623,
+            0.002869575284421444,
+            -0.0003832505608443171,
+            -0.009312454611063004,
+            0.0031104146037250757,
+            0.004693805240094662,
+            0.005455608945339918,
+            0.001044491189531982,
+            -0.01299507636576891,
+            0.024077102541923523,
+            -0.010576434433460236,
+            -0.017203787341713905,
+            0.013247872702777386,
+            0.037304479628801346,
+            0.00696896854788065,
+            0.008355929516255856,
+            0.022710638120770454,
+            0.003508397378027439,
+            -0.03910820931196213,
+            0.004618649836629629,
+            0.005144738592207432,
+            -0.012762777507305145,
+            0.023079583421349525,
+            0.003781690262258053,
+            -0.00325560150668025,
+            -0.01434787642210722,
+            -0.00412330636754632,
+            -0.0012554391287267208,
+            -0.02312057837843895,
+            0.014525516889989376,
+            0.0016021794872358441,
+            -0.018406275659799576,
+            -0.0081782890483737,
+            0.01175159402191639,
+            0.015468376688659191,
+            0.002534791361540556,
+            -0.004943185020238161,
+            0.003178737824782729,
+            0.004232623614370823,
+            -0.013951601460576057,
+            -0.00436927005648613,
+            -0.00046630596625618637,
+            -0.018488263711333275,
+            0.01306340005248785,
+            0.004837283864617348,
+            -0.01559135876595974,
+            -0.007009962573647499,
+            0.020128021016716957,
+            -0.001632070867344737,
+            0.01611061580479145,
+            -0.004058399237692356,
+            0.002451095497235656,
+            0.01777770183980465,
+            0.026318104937672615,
+            -0.005062750540673733,
+            -0.009093821048736572,
+            -0.016014963388442993,
+            -0.013726134784519672,
+            -0.006825489457696676,
+            0.02432306669652462,
+            -0.0011948023457080126,
+            -0.01129382848739624,
+            0.021877095103263855,
+            -0.0011034200433641672,
+            -0.011416809633374214,
+            -0.004659643396735191,
+            -0.0020326157100498676,
+            -0.021262185648083687,
+            0.008157792501151562,
+            -0.002565536880865693,
+            0.009893202222883701,
+            0.020087026059627533,
+            0.031292036175727844,
+            -0.004512748681008816,
+            -0.003405912546440959,
+            -0.006104679778218269,
+            0.005838219076395035,
+            0.00873854011297226,
+            -0.03033551014959812,
+            -0.012407496571540833,
+            -0.014949120581150055,
+            0.006053437478840351,
+            0.018419940024614334,
+            0.007392572239041328,
+            0.015687011182308197,
+            -0.011000038124620914,
+            -0.014921790920197964,
+            -0.02131684496998787,
+            -0.022765297442674637,
+            0.004406847525388002,
+            0.010596930980682373,
+            0.03500881791114807,
+            0.0009266336564905941,
+            -0.010521776042878628,
+            -0.009701897390186787,
+            -0.010419291444122791,
+            0.0015483750030398369,
+            0.006087598856538534,
+            -0.02605847641825676,
+            -0.012721783481538296,
+            -0.016506889835000038,
+            0.004396599251776934,
+            -0.012892591767013073,
+            -0.013357189483940601,
+            0.04971197620034218,
+            -0.006873316131532192,
+            0.0223690215498209,
+            -0.0340522937476635,
+            -0.00457423971965909,
+            -0.0007259342237375677,
+            -0.02250566892325878,
+            -0.001954043982550502,
+            -0.016397573053836823,
+            0.00914847943931818,
+            -0.022123059257864952,
+            -0.0049295201897621155,
+            -0.006466792896389961,
+            -0.005305298138409853,
+            -0.019253483042120934,
+            0.0038807590026408434,
+            -0.020141685381531715,
+            0.01641123741865158,
+            0.0034981488715857267,
+            0.004526413511484861,
+            0.00953108910471201,
+            -0.030390167608857155,
+            0.011929234489798546,
+            -0.006716172676533461,
+            0.007823009043931961,
+            0.003932001534849405,
+            -0.01249631680548191,
+            -0.012216191738843918,
+            -0.004806538578122854,
+            -0.009278293699026108,
+            0.003733864054083824,
+            -0.0032197318505495787,
+            0.018050994724035263,
+            -0.018255963921546936,
+            -0.008895683102309704,
+            -0.004762128461152315,
+            0.019185159355401993,
+            0.01671185903251171,
+            -0.010043513029813766,
+            -0.005127657670527697,
+            -0.02063361182808876,
+            0.025101950392127037,
+            0.005277968943119049,
+            -0.007570212706923485,
+            0.02567586675286293,
+            0.007980152033269405,
+            0.0001476422039559111,
+            0.014142906293272972,
+            0.01739509217441082,
+            -0.017299439758062363,
+            -0.0012588553363457322,
+            0.00824661273509264,
+            0.00758387753739953,
+            -0.00033094058744609356,
+            0.0018088571960106492,
+            0.010412458330392838,
+            0.008232948370277882,
+            -0.004078896250575781,
+            -0.007249093614518642,
+            -0.029324326664209366,
+            -0.008191954344511032,
+            -0.0005337751354090869,
+            0.00011561570136109367,
+            0.016055956482887268,
+            -0.0020906904246658087,
+            0.0054077827371656895,
+            -0.024432383477687836,
+            -0.015236077830195427,
+            0.013732966966927052,
+            -0.022874614223837852,
+            -0.018816214054822922,
+            0.005950952414423227,
+            0.003788522444665432,
+            -0.0107267452403903,
+            0.0036074661184102297,
+            -0.02297026664018631,
+            -0.02109820954501629,
+            -0.00565033033490181,
+            0.007071453146636486,
+            -0.019677087664604187,
+            0.013186381198465824,
+            -0.019294477999210358,
+            -0.030089545994997025,
+            -0.017449749633669853,
+            0.005482938606292009,
+            0.00630623335018754,
+            0.002254666294902563,
+            -0.0006567569798789918,
+            -0.009326119907200336,
+            0.002830289304256439,
+            -0.01656154915690422,
+            0.01209320966154337,
+            0.009271460585296154,
+            0.004806538578122854,
+            -0.01001618430018425,
+            0.002121435943990946,
+            -0.007126112002879381,
+            0.0031326196622103453,
+            0.010979541577398777,
+            0.026195121929049492,
+            0.009920531883835793,
+            -0.006186667364090681,
+            -0.020988892763853073,
+            -0.002758549991995096,
+            -0.013036070391535759,
+            -0.0031582408118993044,
+            0.009783885441720486,
+            0.004208710510283709,
+            -0.019540440291166306,
+            -0.016438566148281097,
+            -0.026031147688627243,
+            -0.011792588047683239,
+            -0.02327088825404644,
+            -0.004283865913748741,
+            0.025730524212121964,
+            -0.010125501081347466,
+            0.0013254705118015409,
+            0.00015564884233754128,
+            0.01626092568039894,
+            -0.0021436410024762154,
+            -0.026987671852111816,
+            0.015044772997498512,
+            -0.008519905619323254,
+            0.0028729913756251335,
+            0.006073934491723776,
+            -0.00412330636754632,
+            0.0008907640003599226,
+            -0.004242871887981892,
+            0.004140387289226055,
+            -0.014320546761155128,
+            0.005506851710379124,
+            -0.007925493642687798,
+            0.019021185114979744,
+            -0.014033589512109756,
+            -0.000921509403269738,
+            0.009059659205377102,
+            -0.0047860415652394295,
+            -0.014402534812688828,
+            -0.017941677942872047,
+            0.024118097499012947,
+            -0.023981450125575066,
+            0.049520671367645264,
+            -0.004136971198022366,
+            0.006138841155916452,
+            -0.02183610200881958,
+            0.003580136690288782,
+            -0.015714339911937714,
+            -0.013992595486342907,
+            -0.01647956110537052,
+            -0.002818332752212882,
+            0.035118136554956436,
+            0.005759647581726313,
+            0.0014843220124021173,
+            0.00045563047751784325,
+            0.004960265941917896,
+            0.03110072948038578,
+            -0.0013271785574033856,
+            0.01956777088344097,
+            -0.0056708273477852345,
+            -0.007686362136155367,
+            -0.007802511565387249,
+            0.010023016482591629,
+            0.016096951439976692,
+            -0.016069620847702026,
+            0.0019677088130265474,
+            -0.0009052826790139079,
+            0.009196305647492409,
+            0.009421772323548794,
+            0.005277968943119049,
+            -0.0048202034085989,
+            -0.009168975986540318,
+            0.00828077457845211,
+            -0.009995686821639538,
+            -0.00832176860421896,
+            -0.017723042517900467,
+            0.005124241579324007,
+            0.01027581188827753,
+            -0.0025484561920166016,
+            0.014525516889989376,
+            -0.005021756514906883,
+            0.004208710510283709,
+            0.005940704140812159,
+            -0.013985763303935528,
+            0.005318962503224611,
+            0.006019275635480881,
+            -0.030554143711924553,
+            0.02727462910115719,
+            -0.01076773926615715,
+            -0.017299439758062363,
+            -0.003011345863342285,
+            0.003069420577958226,
+            -0.004355605226010084,
+            -0.005421447567641735,
+            -0.008984503336250782,
+            -0.018884537741541862,
+            0.006429214961826801,
+            -0.007467728108167648,
+            -0.01844726875424385,
+            -0.001386107294820249,
+            -0.0024579279124736786,
+            0.035883355885744095,
+            -0.003424701513722539,
+            0.010904385708272457,
+            0.016288256272673607
+          ],
+          "title": "The Structure Of The Mixture-Of-Experts Layer",
+          "keyphrases": [
+            "Mixture-of-Experts layer",
+            "gating network",
+            "neural networks",
+            "sparsity",
+            "hierarchical MoE"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "3eec4357-ddcb-410c-be74-055574810bf3",
+      "type": "child",
+      "source": {
+        "id": "14339b1f-53f8-479b-9da7-9430b174b9ee",
+        "properties": {
+          "page_content": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer\n\nNoam Shazeer1, Azalia Mirhoseini∗†1, Krzysztof Maziarz∗2, Andy Davis1, Quoc Le1, Geoffrey Hinton1and Jeff Dean1 1Google Brain, {noam,azalia,andydavis,qvl,geoffhinton,jeff}@google.com 2Jagiellonian University, Cracow, krzysztof.maziarz@student.uj.edu.pl\n\nAbstract\n\nThe capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost.\n\n1 Introduction And Related Work 1.1 Conditional Computation\n\nExploiting scale in both training data and model size has been central to the success of deep learning. When datasets are sufficiently large, increasing the capacity (number of parameters) of neural networks can give much better prediction accuracy. This has been shown in domains such as text (Sutskever et al., 2014; Bahdanau et al., 2014; Jozefowicz et al., 2016; Wu et al., 2016), images (Krizhevsky et al., 2012; Le et al., 2012), and audio (Hinton et al., 2012; Amodei et al., 2015). For typical deep learning models, where the entire model is activated for every example, this leads to a roughly quadratic blow-up in training costs, as both the model size and the number of training examples increase. Unfortunately, the advances in computing power and distributed computation fall short of meeting such demand.\n\nVarious forms of conditional computation have been proposed as a way to increase model capacity without a proportional increase in computational costs (Davis & Arel, 2013; Bengio et al., 2013; Eigen et al., 2013; Ludovic Denoyer, 2014; Cho & Bengio, 2014; Bengio et al., 2015; Almahairi et al., 2015). In these schemes, large parts of a network are active or inactive on a per-example basis. The gating decisions may be binary or sparse and continuous, stochastic or deterministic. Various forms of reinforcement learning and back-propagation are proposed for trarining the gating decisions.\n\nWhile these ideas are promising in theory, no work to date has yet demonstrated massive improvements in model capacity, training time, or model quality. We blame this on a combination of the following challenges:\n\nModern computing devices, especially GPUs, are much faster at arithmetic than at branching. Most of the works above recognize this and propose turning on/off large chunks of the network with each gating decision.\n\nLarge batch sizes are critical for performance, as they amortize the costs of parameter transfers and updates. Conditional computation reduces the batch sizes for the conditionally active chunks of the network.\n\nNetwork bandwidth can be a bottleneck. A cluster of GPUs may have computational power thousands of times greater than the aggregate inter-device network bandwidth. To be computationally efficient, the relative computational versus network demands of an algorithm must exceed this ratio. Embedding layers, which can be seen as a form of conditional computation, are handicapped by this very problem. Since the embeddings generally need to be sent across the network, the number of (example, parameter) interactions is limited by network bandwidth instead of computational capacity.\n\nDepending on the scheme, loss terms may be necessary to achieve the desired level of sparsity per-chunk and/or per example. Bengio et al. (2015) use three such terms. These issues can affect both model quality and load-balancing.\n\nModel capacity is most critical for very large data sets. The existing literature on conditional computation deals with relatively small image recognition data sets consisting of up to 600,000 images. It is hard to imagine that the labels of these images provide a sufficient signal to adequately train a model with millions, let alone billions of parameters. In this work, we for the first time address all of the above challenges and finally realize the promise of conditional computation. We obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets.\n\n1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1). All parts of the network are trained jointly by back-propagation.\n\nWhile the introduced technique is generic, in this paper we focus on language modeling and machine translation tasks, which are known to benefit from very large models. In particular, we apply a MoE convolutionally between stacked LSTM layers (Hochreiter & Schmidhuber, 1997), as in Figure 1.\n\nThe MoE is called once for each position in the text, selecting a potentially different combination of experts at each position. The different experts tend to become highly specialized based on syntax and semantics (see Appendix E Table 9). On both language modeling and machine translation benchmarks, we improve on best published results at a fraction of the computational cost.\n\n1.3 Related Work On Mixtures Of Experts\n\nSince its introduction more than two decades ago (Jacobs et al., 1991; Jordan & Jacobs, 1994), the mixture-of-experts approach has been the subject of much research. Different types of expert architectures hae been proposed such as SVMs (Collobert et al., 2002), Gaussian Processes (Tresp, 2001; Theis & Bethge, 2015; Deisenroth & Ng, 2015), Dirichlet Processes (Shahbaba & Neal, 2009), and deep networks. Other work has focused on different expert configurations such as a hierarchical structure (Yao et al., 2009), infinite numbers of experts (Rasmussen & Ghahramani, 2002), and adding experts sequentially (Aljundi et al., 2016). Garmash & Monz (2016) suggest an ensemble model in the format of mixture of experts for machine translation. The gating network is trained on a pre-trained ensemble NMT model. The works above concern top-level mixtures of experts. The mixture of experts is the whole model. Eigen et al. (2013) introduce the idea of using multiple MoEs with their own gating networks as parts of a deep model. It is intuitive that the latter approach is more powerful, since complex problems may contain many sub-problems each requiring different experts. They also allude in their conclusion to the potential to introduce sparsity, turning MoEs into a vehicle for computational computation.\n\nOur work builds on this use of MoEs as a general purpose neural network component. While Eigen et al. (2013) uses two stacked MoEs allowing for two sets of gating decisions, our convolutional application of the MoE allows for different gating decisions at each position in the text. We also realize sparse gating and demonstrate its use as a practical way to massively increase model capacity.\n\n2 The Structure Of The Mixture-Of-Experts Layer\n\nThe Mixture-of-Experts (MoE) layer consists of a set of n \"expert networks\" E1, · · · , En, and a \"gating network\" G whose output is a sparse n-dimensional vector. Figure 1 shows an overview of the MoE module. The experts are themselves neural networks, each with their own parameters.\n\nAlthough in principle we only require that the experts accept the same sized inputs and produce the same-sized outputs, in our initial investigations in this paper, we restrict ourselves to the case where the models are feed-forward networks with identical architectures, but with separate parameters.\n\nLet us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows:\n\n$$y=\\sum_{i=1}^{n}G(x){i}E{i}(x)$$ $$(\\mathbf{l})$$ G(x)iEi(x) (1) We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example. If the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network. In the following we focus on ordinary MoEs. We provide more details on hierarchical MoEs in Appendix B. Our implementation is related to other models of conditional computation. A MoE whose experts are simple weight matrices is similar to the parameterized weight matrix proposed in (Cho & Bengio, 2014). A MoE whose experts have one hidden layer is similar to the block-wise dropout described in (Bengio et al., 2015), where the dropped-out layer is sandwiched between fully-activated layers.\n\n2.1 Gating Network\n\nSoftmax Gating: A simple choice of non-sparse gating function (Jordan & Jacobs, 1994) is to multiply the input by a trainable weight matrix Wg and then apply the Sof tmax function.\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ $\\eqref{eq:walpha}$. Gσ(x) = Sof tmax(x · Wg) (2) Noisy Top-K Gating: We add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to −∞ (which causes the corresponding gate values to equal 0). The sparsity serves to save computation, as described above. While this form of sparsity creates some theoretically scary discontinuities in the output of gating function, we have not yet observed this to be a problem in practice. The noise term helps with load balancing, as will be discussed in Appendix A. The amount of noise per component is controlled by a second trainable weight matrix Wnoise.\n\n$$G(x)=S o f t m a x(K e e p T o p K(H(x),k))$$ $$(4)$$ $$(S)$$ G(x) = Sof tmax(KeepT opK(H(x), k)) (3) $$H(x){i}=(x\\cdot W{g}){i}+S t a n d a r d N o r m a l()\\cdot S o f t p l u s((x\\cdot W{n o i s e}){i})$$ $KeepTopK(v,k){i}=\\begin{cases}v_{i}&\\text{if}v_{i}\\text{is in the top}k\\text{elements of}v.\\ -\\infty&\\text{otherwise.}\\end{cases}$ Training the Gating Network We train the gating network by simple back-propagation, along with the rest of the model. If we choose k > 1, the gate values for the top k experts have nonzero derivatives with respect to the weights of the gating network. This type of occasionally-sensitive behavior is described in (Bengio et al., 2013) with respect to noisy rectifiers. Gradients also backpropagate through the gating network to its inputs. Our method differs here from (Bengio et al., 2015) who use boolean gates and a REINFORCE-style approach to train the gating network.\n\n3 Addressing Performance Challenges 3.1 The Shrinking Batch Problem\n\nOn modern CPUs and GPUs, large batch sizes are necessary for computational efficiency, so as to amortize the overhead of parameter loads and updates. If the gating network chooses k out of n experts for each example, then for a batch of b examples, each expert receives a much smaller batch of approximately kb n b examples. This causes a naive MoE implementation to become very inefficient as the number of experts increases. The solution to this shrinking batch problem is to make the original batch size as large as possible. However, batch size tends to be limited by the memory necessary to store activations between the forwards and backwards passes. We propose the following techniques for increasing the batch size: Mixing Data Parallelism and Model Parallelism: In a conventional distributed training setting, multiple copies of the model on different devices asynchronously process distinct batches of data, and parameters are synchronized through a set of parameter servers. In our technique, these different batches run synchronously so that they can be combined for the MoE layer. We distribute the standard layers of the model and the gating network according to conventional data-parallel schemes, but keep only one shared copy of each expert. Each expert in the MoE layer receives a combined batch consisting of the relevant examples from all of the data-parallel input batches. The same set of devices function as data-parallel replicas (for the standard layers and the gating networks) and as model-parallel shards (each hosting a subset of the experts). If the model is distributed over d devices, and each device processes a batch of size b, each expert receives a batch of approximately kbd nexamples. Thus, we achieve a factor of d improvement in expert batch size.\n\nIn the case of a hierarchical MoE (Section B), the primary gating network employs data parallelism, and the secondary MoEs employ model parallelism. Each secondary MoE resides on one device.\n\nThis technique allows us to increase the number of experts (and hence the number of parameters) by proportionally increasing the number of devices in the training cluster. The total batch size increases, keeping the batch size per expert constant. The memory and bandwidth requirements per device also remain constant, as do the step times, as does the amount of time necessary to process a number of training examples equal to the number of parameters in the model. It is our goal to train a trillionparameter model on a trillion-word corpus. We have not scaled our systems this far as of the writing of this paper, but it should be possible by adding more hardware.\n\nTaking Advantage of Convolutionality: In our language models, we apply the same MoE to each time step of the previous layer. If we wait for the previous layer to finish, we can apply the MoE to all the time steps together as one big batch. Doing so increases the size of the input batch to the MoE layer by a factor of the number of unrolled time steps.\n\nIncreasing Batch Size for a Recurrent MoE: We suspect that even more powerful models may involve applying a MoE recurrently. For example, the weight matrices of a LSTM or other RNN could be replaced by a MoE. Sadly, such models break the convolutional trick from the last paragraph, since the input to the MoE at one timestep depends on the output of the MoE at the previous timestep. Gruslys et al. (2016) describe a technique for drastically reducing the number of stored activations in an unrolled RNN, at the cost of recomputing forward activations. This would allow for a large increase in batch size.\n\n3.2 Network Bandwidth\n\nAnother major performance concern in distributed computing is network bandwidth. Since the experts are stationary (see above) and the number of gating parameters is small, most of the communication involves sending the inputs and outputs of the experts across the network. To maintain computational efficiency, the ratio of an expert's computation to the size of its input and output must exceed the ratio of computational to network capacity of the computing device. For GPUs, this may be thousands to one. In our experiments, we use experts with one hidden layer containing thousands of RELU-activated units. Since the weight matrices in the expert have sizes input_size×hidden_size and hidden_size × output_size, the ratio of computation to input and output is equal to the size of the hidden layer. Conveniently, we can increase computational efficiency simply by using a larger hidden layer, or more hidden layers.\n\n4 Balancing Expert Utilization\n\nWe have observed that the gating network tends to converge to a state where it always produces large weights for the same few experts. This imbalance is self-reinforcing, as the favored experts are trained more rapidly and thus are selected even more by the gating network. Eigen et al. (2013) describe the same phenomenon, and use a hard constraint at the beginning of training to avoid this local minimum. Bengio et al. (2015) include a soft constraint on the batch-wise average of each gate.1 We take a soft constraint approach. We define the importance of an expert relative to a batch of training examples to be the batchwise sum of the gate values for that expert. We define an additional loss Limportance, which is added to the overall loss function for the model. This loss is equal to the square of the coefficient of variation of the set of importance values, multiplied by a hand-tuned scaling factor wimportance. This additional loss encourages all experts to have equal importance.\n\n$$I m p o r t a n c e(X)=\\sum_{x\\in X}G(x)$$ $$L_{importance}(X)=w_{importance}\\cdot CV(Importance(X))^{2}\\tag{7}$$ $$(6)$$\n\nWhile this loss function can ensure equal importance, experts may still receive very different numbers of examples. For example, one expert may receive a few examples with large weights, and another may receive many examples with small weights. This can cause memory and performance problems on distributed hardware. To solve this problem, we introduce a second loss function, Lload , which ensures balanced loads. Appendix A contains the definition of this function, along with experimental results.\n\n5 Experiments 5.1 1 Billion Word Language Modeling Benchmark\n\nDataset: This dataset, introduced by (Chelba et al., 2013) consists of shuffled unique sentences from news articles, totaling approximately 829 million words, with a vocabulary of 793,471 words.\n\nPrevious State-of-the-Art: The best previously published results (Jozefowicz et al., 2016) use models consisting of one or more stacked Long Short-Term Memory (LSTM) layers (Hochreiter & Schmidhuber, 1997; Gers et al., 2000). The number of parameters in the LSTM layers of these models vary from 2 million to 151 million. Quality increases greatly with parameter count, as do computational costs. Results for these models form the top line of Figure 2-right. MoE Models: Our models consist of two stacked LSTM layers with a MoE layer between them (see Figure 1). We vary the sizes of the layers and the number of experts. For full details on model architecture, training regimen, additional baselines and results, see Appendix C.\n\nLow Computation, Varied Capacity: To investigate the effects of adding capacity, we trained\n\na series of MoE models all with roughly equal computational costs: about 8 million multiply-andadds per training example per timestep in the forwards pass, excluding the softmax layer. We call this metric (ops/timestep). We trained models with flat MoEs containing 4, 32, and 256 experts, and models with hierarchical MoEs containing 256, 1024, and 4096 experts. Each expert had about 1 million parameters. For all the MoE layers, 4 experts were active per input.\n\nThe results of these models are shown in Figure 2-left. The model with 4 always-active experts performed (unsurprisingly) similarly to the computationally-matched baseline models, while the largest of the models (4096 experts) achieved an impressive 24% lower perplexity on the test set. Figure 2: Model comparison on 1-Billion-Word Language-Modeling Benchmark. On the left, we plot test perplexity as a function of model capacity for models with similar computational budgets of approximately 8-million-ops-per-timestep. On the right, we plot test perplexity as a function of computational budget. The top line represents the LSTM models from (Jozefowicz et al., 2016).\n\nThe bottom line represents 4-billion parameter MoE models with different computational budgets.\n\nVaried Computation, High Capacity: In addition to the largest model from the previous section, we trained two more MoE models with similarly high capacity (4 billion parameters), but higher computation budgets. These models had larger LSTMs, and fewer but larger and experts. Details\n\nTest Test #Parameters ops/timestep Training TFLOPS Perplexity Perplexity excluding embedding Time /GPU 10 epochs 100 epochs and softmax layers 10 epochs Best Published Results 34.7 30.6 151 million 151 million 59 hours, 32 k40s 1.09 Low-Budget MoE Model 34.1 4303 million 8.9 million 15 hours, 16 k40s 0.74 Medium-Budget MoE Model 31.3 4313 million 33.8 million 17 hours, 32 k40s 1.22 High-Budget MoE Model 28.0 4371 million 142.7 million 47 hours, 32 k40s 1.56\n\nTable 1: Summary of high-capacity MoE-augmented models with varying computational budgets, vs. best previously published results (Jozefowicz et al., 2016). Details in Appendix C. can be found in Appendix C.2. Results of these three models form the bottom line of Figure 2-right.\n\nTable 1 compares the results of these models to the best previously-published result on this dataset .\n\nEven the fastest of these models beats the best published result (when controlling for the number of training epochs), despite requiring only 6% of the computation.\n\nComputational Efficiency: We trained our models using TensorFlow (Abadi et al., 2016) on clusters containing 16-32 Tesla K40 GPUs. For each of our models, we determine computational efficiency in TFLOPS/GPU by dividing the number of floating point operations required to process one training batch by the observed step time and the number of GPUs in the cluster. The operation counts used here are higher than the ones we report in our ops/timestep numbers in that we include the backwards pass, we include the importance-sampling-based training of the softmax layer, and we count a multiply-and-add as two separate operations. For all of our MoE models, the floating point operations involved in the experts represent between 37% and 46% of the total.\n\nFor our baseline models wtih no MoE, observed computational efficiency ranged from 1.07-1.29 TFLOPS/GPU. For our low-computation MoE models, computation efficiency ranged from 0.740.90 TFLOPS/GPU, except for the 4-expert model which did not make full use of the available parallelism. Our highest-computation MoE model was more efficient at 1.56 TFLOPS/GPU, likely due to the larger matrices. These numbers represent a significant fraction of the theoretical maximum of 4.29 TFLOPS/GPU claimed by NVIDIA. Detailed results are in Appendix C, Table 7.\n\n5.2 100 BILLION WORD GOOGLE NEWS CORPUS\n\nOn the 1-billion-word corpus, adding additional capacity seems to produce diminishing returns as the number of parameters in the MoE layer exceeds 1 billion, as can be seen in Figure 2-left. We hypothesized that for a larger training set, even higher capacities would produce significant quality improvements. We constructed a similar training set consisting of shuffled unique sentences from Google's internal news corpus, totalling roughly 100 billion words. Similarly to the previous section, we tested a series of models with similar computational costs of about 8 million ops/timestep. In addition to a baseline LSTM model, we trained models augmented with MoE layers containing 32, 256, 1024, 4096, 16384, 65536, and 131072 experts. This corresponds to up to 137 billion parameters in the MoE layer. Details on architecture, training, and results are given in Appendix D.\n\nResults: Figure 3 shows test perplexity as a function of capacity after training on 10 billion words (top line) and 100 billion words (bottom line). When training over the full 100 billion words, test perplexity improves significantly up to 65536 experts (68 billion parameters), dropping 39% lower than the computationally matched baseline, but degrades at 131072 experts, possibly a result of too much sparsity. The widening gap between the two lines demonstrates (unsurprisingly) that increased model capacity helps more on larger training sets. Even at 65536 experts (99.994% layer sparsity), computational efficiency for the model stays at a respectable 0.72 TFLOPS/GPU.\n\n5.3 Machine Translation (Single Language Pair)\n\nModel Architecture: Our model was a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decreased the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We inserted MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). Each MoE layer contained up to 2048 experts each with about two million parameters, adding a total of about 8 billion parameters to the models. Further details on model architecture, testing procedure and results can be found in Appendix E.\n\nDatasets: We benchmarked our method on the WMT'14 En→Fr and En→De corpora, whose training sets have 36M sentence pairs and 5M sentence pairs, respectively. The experimental protocols were also similar to those in (Wu et al., 2016): newstest2014 was used as the test set to compare against previous work (Luong et al., 2015a; Zhou et al., 2016; Wu et al., 2016), while the combination of newstest2012 and newstest2013 was used as the development set. We also tested the same model on a Google's Production English to French data.\n\nTable 2: Results on WMT'14 En→ Fr newstest2014 (bold values represent best results). Model Test Test ops/timenstep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.69 40.35 85M 8.7B 3 days/64 k40s MoE with 2048 Experts (longer training) 2.63 40.56 85M 8.7B 6 days/64 k40s GNMT (Wu et al., 2016) 2.79 39.22 214M 278M 6 days/96 k80s GNMT+RL (Wu et al., 2016) 2.96 39.92 214M 278M 6 days/96 k80s PBMT (Durrani et al., 2014) 37.0 LSTM (6-layer) (Luong et al., 2015b) 31.5 LSTM (6-layer+PosUnk) (Luong et al., 2015b) 33.1 DeepAtt (Zhou et al., 2016) 37.7 DeepAtt+PosUnk (Zhou et al., 2016) 39.2\n\nTable 3: Results on WMT'14 En → De newstest2014 (bold values represent best results). Model Test Test ops/timestep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 4.64 26.03 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 5.25 24.91 214M 278M 1 day/96 k80s GNMT +RL (Wu et al., 2016) 8.08 24.66 214M 278M 1 day/96 k80s PBMT (Durrani et al., 2014) 20.7 DeepAtt (Zhou et al., 2016) 20.6\n\nTable 4: Results on the Google Production En→ Fr dataset (bold values represent best results). Model Eval Eval Test Test ops/timestep Total Training Perplexity BLEU Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.60 37.27 2.69 36.57 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 2.78 35.80 2.87 35.56 214M 278M 6 days/96 k80s\n\nResults: Tables 2, 3, and 4 show the results of our largest models, compared with published results. Our approach achieved BLEU scores of 40.56 and 26.03 on the WMT'14 En→Fr and En→De benchmarks. As our models did not use RL refinement, these results constitute significant gains of 1.34 and 1.12 BLEU score on top of the strong baselines in (Wu et al., 2016). The perplexity scores are also better.2 On the Google Production dataset, our model achieved 1.01 higher test BLEU score even after training for only one sixth of the time.\n\n5.4 Multilingual Machine Translation\n\nDataset: (Johnson et al., 2016) train a single GNMT (Wu et al., 2016) model on a very large combined dataset of twelve language pairs. Results are somewhat worse than those for 12 separately trained single-pair GNMT models. This is not surprising, given that the twelve models have 12 times the capacity and twelve times the aggregate training of the one model. We repeat this experiment with a single MoE-augmented model. See Appendix E for details on model architecture.\n\nWe train our model on the same dataset as (Johnson et al., 2016) and process the same number of training examples (about 3 billion sentence pairs). Our training time was shorter due to the lower computational budget of our model.\n\nResults: Results for the single-pair GNMT models, the multilingual GNMT model and the multilingual MoE model are given in Table 5. The MoE model achieves 19% lower perplexity on the dev set than the multilingual GNMT model. On BLEU score, the MoE model significantly beats the multilingual GNMT model on 11 of the 12 language pairs (by as much as 5.84 points), and even beats the monolingual GNMT models on 8 of 12 language pairs. The poor performance on English → Korean seems to be a result of severe overtraining, as for the rarer language pairs a small number of real examples were highly oversampled in the training corpus.\n\nTable 5: Multilingual Machine Translation (bold values represent best results). GNMT-Mono GNMT-Multi MoE-Multi MoE-Multi vs. GNMT-Multi Parameters 278M / model 278M 8.7B ops/timestep 212M 212M 102M training time, hardware various 21 days, 96 k20s 12 days, 64 k40s Perplexity (dev) 4.14 3.35 -19% French → English Test BLEU 36.47 34.40 37.46 +3.06 German → English Test BLEU 31.77 31.17 34.80 +3.63 Japanese → English Test BLEU 23.41 21.62 25.91 +4.29 Korean → English Test BLEU 25.42 22.87 28.71 +5.84 Portuguese → English Test BLEU 44.40 42.53 46.13 +3.60 Spanish → English Test BLEU 38.00 36.04 39.39 +3.35 English → French Test BLEU 35.37 34.00 36.59 +2.59 English → German Test BLEU 26.43 23.15 24.53 +1.38 English → Japanese Test BLEU 23.66 21.10 22.78 +1.68 English → Korean Test BLEU 19.75 18.41 16.62 -1.79 English → Portuguese Test BLEU 38.40 37.35 37.90 +0.55 English → Spanish Test BLEU 34.50 34.25 36.21 +1.96\n\n6 Conclusion\n\nThis work is the first to demonstrate major wins from conditional computation in deep networks.\n\nWe carefully identified the design considerations and challenges of conditional computing and addressed them with a combination of algorithmic and engineering solutions. While we focused on text, conditional computation may help in other domains as well, provided sufficiently large training sets. We look forward to seeing many novel implementations and applications of conditional computation in the years to come.\n\nAcknowledgments\n\nWe would like to thank all of the members of the Google Brain and Google Translate teams who helped us with this project, in particular Zhifeng Chen, Yonghui Wu, and Melvin Johnson. Thanks also to our anonymous ICLR reviewers for the helpful suggestions on making this paper better.\n\n2Reported perplexities relative to the tokenization used by both our models and GNMT.\n\nReferences\n\nMartín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Gordon Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda B. Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. CoRR, abs/1603.04467, 2016. URL http://arxiv.org/abs/1603.04467.\n\nRahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. CoRR, abs/1611.06194, 2016. URL http://arxiv.org/abs/1611.\n\n06194.\n\nA. Almahairi, N. Ballas, T. Cooijmans, Y. Zheng, H. Larochelle, and A. Courville. Dynamic Capacity Networks. ArXiv e-prints, November 2015.\n\nDario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan, Christopher Fougner, Tony Han, Awni Y. Hannun, Billy Jun, Patrick LeGresley, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595, 2015. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.\n\nEmmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297, 2015.\n\nYoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.\n\nCiprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling.\n\narXiv preprint arXiv:1312.3005, 2013.\n\nK. Cho and Y. Bengio. Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning. ArXiv e-prints, June 2014. Ronan Collobert, Samy Bengio, and Yoshua Bengio. A parallel mixture of SVMs for very large scale problems. Neural Computing, 2002.\n\nAndrew Davis and Itamar Arel. Low-rank approximations for conditional feedforward computation in deep neural networks. arXiv preprint arXiv:1312.4461, 2013. Marc Peter Deisenroth and Jun Wei Ng. Distributed Gaussian processes. In ICML, 2015.\n\nJohn Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization, 2010.\n\nNadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. Edinburgh's phrase-based machine translation systems for wmt-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation, 2014.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013. Ekaterina Garmash and Christof Monz. Ensemble learning for multi-source neural machine translation. In staff.science.uva.nl/c.monz, 2016.\n\nFelix A. Gers, Jürgen A. Schmidhuber, and Fred A. Cummins. Learning to forget: Continual prediction with lstm. Neural Computation, 2000.\n\nAudrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. Memory-efficient backpropagation through time. CoRR, abs/1606.03401, 2016. URL http://arxiv.org/ abs/1606.03401. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2015.\n\nGeoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 2012. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.\n\nSergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.\n\nRobert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computing, 1991.\n\nMelvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation.\n\nCoRR, abs/1611.04558, 2016. URL http://arxiv.org/abs/1611.04558.\n\nMichael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm.\n\nNeural Computing, 1994. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.\n\nDiederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.\n\nReinhard Kneser and Hermann. Ney. Improved backingoff for m-gram language modeling., 1995.\n\nAlex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.\n\nQuoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeffrey Dean, and Andrew Y. Ng. Building high-level features using large scale unsupervised learning. In ICML, 2012. Patrick Gallinari Ludovic Denoyer. Deep sequential neural network. arXiv preprint arXiv:1410.0510, 2014.\n\nMinh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attentionbased neural machine translation. EMNLP, 2015a.\n\nMinh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the rare word problem in neural machine translation. ACL, 2015b.\n\nCarl Edward Rasmussen and Zoubin Ghahramani. Infinite mixtures of Gaussian process experts.\n\nNIPS, 2002.\n\nHasim Sak, Andrew W Senior, and Françoise Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH, pp. 338–342, 2014.\n\nMike Schuster and Kaisuke Nakajima. Japanese and Korean voice search. ICASSP, 2012.\n\nBabak Shahbaba and Radford Neal. Nonlinear models using dirichlet process mixtures. JMLR, 2009.\n\nIlya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks.\n\nIn NIPS, 2014.\n\nLucas Theis and Matthias Bethge. Generative image modeling using spatial LSTMs. In NIPS, 2015. Volker Tresp. Mixtures of Gaussian Processes. In NIPS, 2001.\n\nYonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.\n\nBangpeng Yao, Dirk Walther, Diane Beck, and Li Fei-fei. Hierarchical mixture of classification experts uncovers interactions between brain regions. In NIPS. 2009.\n\nWojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization.\n\narXiv preprint arXiv:1409.2329, 2014.\n\nJie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with fast-forward connections for neural machine translation. arXiv preprint arXiv:1606.04199, 2016.\n\nAppendices A Load-Balancing Loss\n\nAs discussed in section 4, for load-balancing purposes, we want to define an additional loss function to encourage experts to receive roughly equal numbers of training examples. Unfortunately, the number of examples received by an expert is a discrete quantity, so it can not be used in backpropagation. Instead, we define a smooth estimator Load(X) of the number of examples assigned to each expert for a batch X of inputs. The smoothness allows us to back-propagate gradients through the estimator. This is the purpose of the noise term in the gating function. We define P(x, i) as the probability that G(x)iis nonzero, given a new random choice of noise on element i, but keeping the already-sampled choices of noise on the other elements. To compute P(x, i), we note that the G(x)iis nonzero if and only if H(x)iis greater than the k th-greatest element of H(x) excluding itself. The probability works out to be:\n\n$$P(x,i)=P r\\Big((x\\cdot W_{g}){i}+S t a n d x N o r a l()\\cdot S o f t p u s((x\\cdot W{n o i s e})_{i})$$ $$>k t h_e x c l u d i n(H(x),k,i)\\Big)$$ $$({\\boldsymbol{8}})$$\n\n$$(9)$$ $$(10)$$ (8) Where kth_excluding(v, k, i) means the kth highest component of v, excluding component i. Simplifying, we get:\n\n$$P(x,i)=\\Phi{\\Big(}{\\frac{(x\\cdot W_{g}){i}-k t h_c x c l u d i n g(H(x),k,i)}{S o f t p l u s((x\\cdot W{n o i s e})_{i})}}{\\Big)}$$\n\nWhere Φ is the CDF of the standard normal distribution.\n\n$$L o a d(X){i}=\\sum{x\\in X}P(x,i)$$\n\n$$(11)$$ P(x, i) (10) We can now define the load loss to be the square of the coefficient of variation of the load vector, multiplied by a hand-tuned scaling factor wload.\n\n$$L_{l o a d}(X)=w_{l o a d}\\cdot C V(L o a d(X))^{2}$$ Lload(X) = wload · CV (Load(X))2(11) Initial Load Imbalance: To avoid out-of-memory errors, we need to initialize the network in a state of approximately equal expert load (since the soft constraints need some time to work). To accomplish this, we initialize the matrices Wg and Wnoise to all zeros, which yields no signal and some noise.\n\nExperiments: We trained a set of models with identical architecture (the MoE-256 model described in Appendix C), using different values of wimportance and wload. We trained each model for 10 epochs, then measured perplexity on the test set. We also measured the coefficients of variation in Importance and Load, as well as ratio of the load on the most overloaded expert to the average load. This last value is significant for load balancing purposes on distributed hardware. All of these metrics were averaged over several training batches.\n\nwimportance wload Test Perplexity CV (Importance(X)) CV (Load(X)) max(Load(X)) mean(Load(X)) 0.0 0.0 39.8 3.04 3.01 17.80 0.2 0.0 35.6 0.06 0.17 1.47 0.0 0.2 35.7 0.22 0.04 1.15 0.1 0.1 35.6 0.06 0.05 1.14 0.01 0.01 35.7 0.48 0.11 1.37 1.0 1.0 35.7 0.03 0.02 1.07\n\nTable 6: Experiments with different combinations of losses.\n\n13 Results: Results are reported in Table 6. All the combinations containing at least one the two losses led to very similar model quality, where having no loss was much worse. Models with higher values of wload had lower loads on the most overloaded expert.\n\nB Hierachical Mixture Of Experts\n\nIf the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network.3If the hierarchical MoE consists of a groups of b experts each, we denote the primary gating network by Gprimary, the secondary gating networks by (G1, G2..Ga), and the expert networks by (E0,0, E0,1..Ea,b). The output of the MoE is given by:\n\n$$y_{H}=\\sum_{i=1}^{a}\\sum_{j=1}^{b}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}\\cdot E{i,j}(x)$$ $$(12)$$ $$(13)$$ $$(14)$$\n\nOur metrics of expert utilization change to the following:\n\n$$I m p o r t a n c e_{H}(X){i,j}=\\sum{x\\in X}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}$$ $$L o a d{H}(X){i,j}={\\frac{L o a d{p r i m a r y}(X){i}\\cdot L o a d{i}(X^{(i)})_{j}}{|X^{(i)}|}}$$\n\nLoadprimary and Loadi deonte the Load functions for the primary gating network and i th secondary gating network respectively. X(i) denotes the subset of X for which Gprimary(x)i > 0.\n\nIt would seem simpler to let LoadH(X)i,j = Loadi(Xi)j , but this would not have a gradient with respect to the primary gating network, so we use the formulation above.\n\nC 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS C.1 8-MILLION-OPERATIONS-PER-TIMESTEP MODELS Model Architecture: Our model consists of five layers: a word embedding layer, a recurrent Long Short-Term Memory (LSTM) layer (Hochreiter & Schmidhuber, 1997; Gers et al., 2000), a MoE layer, a second LSTM layer, and a softmax layer. The dimensionality of the embedding layer, the number of units in each LSTM layer, and the input and output dimensionality of the MoE layer are all equal to 512. For every layer other than the softmax, we apply drouput (Zaremba et al., 2014) to the layer output, dropping each activation with probability DropP rob, otherwise dividing by (1 − DropP rob). After dropout, the output of the previous layer is added to the layer output.\n\nThis residual connection encourages gradient flow (He et al., 2015).\n\nMoE Layer Architecture: Each expert in the MoE layer is a feed forward network with one ReLU-activated hidden layer of size 1024 and an output layer of size 512. Thus, each expert contains [512 ∗ 1024] + [1024 ∗ 512] = 1M parameters. The output of the MoE layer is passed through a sigmoid function before dropout. We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts.\n\nWe call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers. Thus, each example is processed by exactly 4 experts for a total of 4M ops/timestep. The two LSTM layers contribute 2M ops/timestep each for the desired total of 8M.\n\n3 We have not found the need for deeper hierarchies.\n\nComputationally-Matched Baselines: The MoE-4 model does not employ sparsity, since all 4 experts are always used. In addition, we trained four more computationally-matched baseline models with no sparsity:\n\nMoE-1-Wide: The MoE layer consists of a single \"expert\" containing one ReLU-activated hidden layer of size 4096.\n\nMoE-1-Deep: The MoE layer consists of a single \"expert\" containing four ReLU-activated hidden layers, each with size 1024.\n\n4xLSTM-512: We replace the MoE layer with two additional 512-unit LSTM layers.\n\nLSTM-2048-512: The model contains one 2048-unit LSTM layer (and no MoE). The output of the LSTM is projected down to 512 dimensions (Sak et al., 2014). The next timestep of the LSTM receives the projected output. This is identical to one of the models published in (Jozefowicz et al., 2016). We re-ran it to account for differences in training regimen, and obtained results very similar to the published ones. Training: The models were trained on a cluster of 16 K40 GPUs using the synchronous method described in Section 3. Each batch consisted of a set of sentences totaling roughly 300,000 words. In the interest of time, we limited training to 10 epochs, (27,000 steps). Training took 12-16 hours for all models, except for MoE-4, which took 18 hours (since all the expert computation was performed on only 4 of 16 GPUs). We used the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 1000 training steps, and decreased after that so as to be proportional to the inverse square root of the step number. The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016). For each model, we performed a hyper-parmeter search to find the best dropout probability, in increments of 0.1.\n\nTo ensure balanced expert utilization we set wimportance = 0.1 and wload = 0.1, as described in Section 4 and Appendix A.\n\nResults: We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol. Results are reported in Table 7. For each model, we report the test perplexity, the computational budget, the parameter counts, the value of DropP rob, and the computational efficiency.\n\nModel Test Test ops/timestep #Params excluding Total Drop- TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params P rob per GPU 10 epochs (final) (millions) (billions) (observed) Kneser-Ney 5-gram* 67.6 0.00001 1.8 LSTM-512-512* 54.1 2.4 2.4 0.8 0.1 LSTM-1024-512* 48.2 4.7 4.7 0.8 0.1 LSTM-2048-512* 45.0 43.7 9.4 9.4 0.8 0.1 0.61 LSTM-2048-512 44.7 9.4 9.4 0.8 0.1 1.21 4xLSTM-512 46.0 8.4 8.4 0.8 0.1 1.07 MoE-1-Wide 46.1 8.4 8.4 0.8 0.1 1.29 MoE-1-Deep 45.7 8.4 8.4 0.8 0.1 1.29 MoE-4 45.0 8.4 8.4 0.8 0.1 0.52 MoE-32 39.7 8.4 37.8 0.9 0.1 0.87 MoE-256 35.7 8.6 272.9 1.1 0.1 0.81 MoE-256-h 36.0 8.4 272.9 1.1 0.1 0.89 MoE-1024-h 34.6 8.5 1079.0 1.9 0.2 0.90 MoE-4096-h 34.1 8.9 4303.4 5.1 0.2 0.74 2xLSTM-8192-1024* 34.7 30.6 151.0 151.0 1.8 0.25 1.09 MoE-34M 31.3 33.8 4313.9 6.0 0.3 1.22 MoE-143M 28.0 142.7 4371.1 6.0 0.4 1.56\n\nC.2 More Expensive Models\n\nWe ran two additional models (MoE-34M and MoE-143M) to investigate the effects of adding more computation in the presence of a large MoE layer. These models have computation budgets of 34M and 143M ops/timestep. Similar to the models above, these models use a MoE layer between two LSTM layers. The dimensionality of the embedding layer, and the input and output dimensionality of the MoE layer are set to 1024 instead of 512. For MoE-34M, the LSTM layers have 1024 units.\n\nFor MoE-143M, the LSTM layers have 4096 units and an output projection of size 1024 (Sak et al., 2014). MoE-34M uses a hierarchical MoE layer with 1024 experts, each with a hidden layer of size 2048. MoE-143M uses a hierarchical MoE layer with 256 experts, each with a hidden layer of size 8192. Both models have 4B parameters in the MoE layers. We searched for the best DropP rob for each model, and trained each model for 10 epochs.\n\nThe two models achieved test perplexity of 31.3 and 28.0 respectively, showing that even in the presence of a large MoE, more computation is still useful. Results are reported at the bottom of Table 7. The larger of the two models has a similar computational budget to the best published model from the literature, and training times are similar. Comparing after 10 epochs, our model has a lower test perplexity by 18%.\n\nD 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS Model Architecture: The models are similar in structure to the 8-million-operations-per-timestep models described in the previous section. We vary the number of experts between models, using an ordinary MoE layer with 32 experts and hierarchical MoE layers with 256, 1024, 4096, 16384, 65536 and 131072 experts. For the hierarchical MoE layers, the first level branching factors are 32, 32, 64, 128, 256 and 256, respectively.\n\nTraining: Models are trained on a cluster of 32 Tesla K40 GPUs, except for the last two models, which are trained on clusters of 64 and 128 GPUs so as to have enough memory for all the parameters. For all models, training batch sizes are approximately 2.5 million words. Models are trained once-through over about 100 billion words. We implement several memory optimizations in order to fit up to 1 billion parameters per GPU. First, we do not store the activations of the hidden layers of the experts, but instead recompute them on the backwards pass. Secondly, we modify the optimizer on the expert parameters to require less auxiliary storage: The Adam optimizer (Kingma & Ba, 2015) keeps first and second moment estimates of the perparameter gradients. This triples the required memory. To avoid keeping a first-moment estimator, we set β1 = 0. To reduce the size of the second moment estimator, we replace it with a factored approximation. For a matrix of parameters, instead of maintaining a full matrix of second-moment estimators, we maintain vectors of row-wise and column-wise averages of that matrix. At each step, the matrix of estimators is taken to be the outer product of those two vectors divided by the mean of either one. This technique could similarly be applied to Adagrad (Duchi et al., 2010).\n\nModel Test Test ops/timestep #Params excluding Total TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params per GPU .1 epochs 1 epoch (millions) (billions) (observed) Kneser-Ney 5-gram 67.1 45.3 0.00001 76.0 4xLSTM-512 54.5 47.0 8.4 8.4 0.1 1.23 MoE-32 48.5 40.4 8.4 37.8 0.1 0.83 MoE-256-h 42.8 35.3 8.4 272.9 0.4 1.11 MoE-1024-h 40.3 32.7 8.5 1079.0 1.2 1.14 MoE-4096-h 38.9 30.9 8.6 4303.4 4.4 1.07 MoE-16384-h 38.2 29.7 8.8 17201.0 17.3 0.96 MoE-65536-h 38.2 28.9 9.2 68791.0 68.9 0.72 MoE-131072-h 39.8 29.2 9.7 137577.6 137.7 0.30\n\nResults: We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8. Perplexity after 100 billion training words is 39% lower for the 68-billion-parameter MoE model than for the baseline model. It is notable that the measured computational efficiency of the largest model (0.30 TFLOPS/GPU) is very low compared to the other models. This is likely a result of the fact that, for purposes of comparison to the other models, we did not increase the training batch size proportionally to the number of GPUs. For comparison, we include results for a computationally matched baseline model consisting of 4 LSTMs, and for an unpruned 5-gram model with Kneser-Ney smoothing (Kneser & Ney, 1995).4\n\nE Machine Translation - Experimental Details\n\nModel Architecture for Single Language Pair MoE Models: Our model is a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decrease the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We insert MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). We use an attention mechanism between the encoder and decoder, with the first decoder LSTM receiving output from and providing input for the attention 5. All of the layers in our model have input and output dimensionality of 512. Our LSTM layers have 2048 hidden units, with a 512-dimensional output projection. We add residual connections around all LSTM and MoE layers to encourage gradient flow (He et al., 2015). Similar to GNMT, to effectively deal with rare words, we used subword units (also known as \"wordpieces\") (Schuster & Nakajima, 2012) for inputs and outputs in our system.\n\nWe use a shared source and target vocabulary of 32K wordpieces. We also used the same beam search technique as proposed in (Wu et al., 2016).\n\nWe train models with different numbers of experts in the MoE layers. In addition to a baseline model with no MoE layers, we train models with flat MoE layers containing 32 experts, and models with hierarchical MoE layers containing 512 and 2048 experts. The flat MoE layers use k = 4 and the hierarchical MoE models use k = 2 at each level of the gating network. Thus, each input is processed by exactly 4 experts in each MoE layer. Each expert in the MoE layer is a feed forward network with one hidden layer of size 2048 and ReLU activation. Thus, each expert contains [512 ∗ 2048] + [2048 ∗ 512] = 2M parameters. The output of the MoE layer is passed through a sigmoid function. We use the strictly-balanced gating function described in Appendix F.\n\nModel Architecture for Multilingual MoE Model: We used the same model architecture as for the single-language-pair models, with the following exceptions: We used noisy-top-k gating as described in Section 2.1, not the scheme from Appendix F. The MoE layers in the encoder and decoder are non-hierarchical MoEs with n = 512 experts, and k = 2. Each expert has a larger hidden layer of size 8192. This doubles the amount of computation in the MoE layers, raising the computational budget of the entire model from 85M to 102M ops/timestep. Training: We trained our networks using the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 2000 training steps, held constant for an additional 8000 steps, and decreased after that so as to be proportional to the inverse square root of the step number. For the single-language-pair models, similarly to (Wu et al., 2016), we applied dropout (Zaremba et al., 2014) to the output of all embedding, LSTM and MoE layers, using DropP rob = 0.4. Training was done synchronously on a cluster of up to 64 GPUs as described in section 3. Each training batch consisted of a set of sentence pairs containing roughly 16000 words per GPU.\n\nTo ensure balanced expert utilization we set wimportance = 0.01 and wload = 0.01, as described in Section 4 and Appendix A. Metrics: We evaluated our models using the perplexity and the standard BLEU score metric. We reported tokenized BLEU score as computed by the multi-bleu.pl script, downloaded from the public implementation of Moses (on Github), which was also used in (Luong et al., 2015a). Results: Tables 2, 3 and 4 in Section 5.3 show comparisons of our results to other published methods. Figure 4 shows test perplexity as a function of number of words in the (training data's) source sentences processed for models with different numbers of experts. As can be seen from the Figure, as we increased the number of experts to approach 2048, the test perplexity of our model continued to improve.\n\nWe found that the experts indeed become highly specialized by syntax and/or semantics, as can be seen in Table 9. For example, one expert is used when the indefinite article \"a\" introduces the direct object in a verb phrase indicating importance or leadership.\n\nExpert 381 Expert 752 Expert 2004 ... with researchers , ... ... plays a core ... ... with rapidly growing ... ... to innovation . ... plays a critical ... ... under static conditions ... ... tics researchers . ... provides a legislative ... ... to swift ly ... ... the generation of ... ... play a leading ... ... to dras tically ... ... technology innovations is ... ... assume a leadership ... ... the rapid and ... ... technological innovations , ... ... plays a central ... ... the fast est ... ... support innovation throughout ... ... taken a leading ... ... the Quick Method ... ... role innovation will ... ... established a reconciliation ... ... rec urrent ) ... ... research scienti st ... ... played a vital ... ... provides quick access ... ... promoting innovation where ... ... have a central ... ... of volatile organic ... ... ... ...\n\nDue to some peculiarities in our infrastructure which have since been fixed, at the time we ran some of the machine translation experiments, our models ran faster if every expert received exactly the same batch size. To accommodate this, we used a different gating function which we describe below. Recall that we define the softmax gating function to be:\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ Gσ(x) = Sof tmax(x · Wg) (15) Sparse Gating (alternate formulation): To obtain a sparse gating vector, we multiply Gσ(x) component-wise with a sparse mask M(Gσ(x)) and normalize the output. The mask itself is a function of Gσ(x) and specifies which experts are assigned to each input example:\n\n$$(15)$$ $$G(x){i}=\\frac{G{\\sigma}(x){i}M(G{\\sigma}(x)){i}}{\\sum{j=1}^{n}G_{\\sigma}(x){j}M(G{\\sigma}(x))_{j}}\\tag{1}$$ $$(17)$$\n\n$$(16)$$\n\nTop-K Mask: To implement top-k gating in this formulation, we would let M(v) = T opK(v, k), where:\n\n$$T o p K(v,k){i}=\\begin{cases}1&{\\mathrm{if~}}v{i}{\\mathrm{~is~in~the~top~}}k{\\mathrm{~elements~of~}}v.\\ 0&{\\mathrm{otherwise.}}\\end{cases}$$ 0 otherwise. (17) Batchwise Mask: To force each expert to receive the exact same number of examples, we introduce an alternative mask function, Mbatchwise(X, m), which operates over batches of input vectors.\n\nInstead of keeping the top k values per example, we keep the top m values per expert across the training batch, where m = k|X| n, so that each example is sent to an average of k experts.\n\n$$M_{b a t c h w i s e}(X,m){j,i}=\\begin{cases}1&\\text{if}X{j,i}\\text{is in the top}m\\text{values for to expert}i\\ 0&\\text{otherwise}\\end{cases}$$\n\nAs our experiments suggest and also observed in (Ioffe & Szegedy, 2015), using a batchwise function during training (such as Mbatchwise) requires modifications to the inference when we may not have a large batch of examples. Our solution to this is to train a vector T of per-expert threshold values to approximate the effects of the batchwise mask. We use the following mask at inference time:\n\n$$(18)$$ $$M_{threshold}(x,T){i}=\\begin{cases}1&\\text{if}x{i}>T_{i}\\ 0&\\text{otherwise}\\end{cases}\\tag{1}$$ $$(19)$$\n\nTo learn the threshold values, we apply an additional loss at training time which is minimized when the batchwise mask and the threshold mask are identical.\n\n$$L_{batchwise}(X,T,m)=\\sum_{j=1}^{|X|}\\sum_{i=1}^{n}(M_{threshold}(x,T){i}-M{batchwise}(X,m){j,i})(X{j,i}-T_{i})\\tag{20}$$\n\nG ATTENTION FUNCTION The attention mechanism described in GNMT (Wu et al., 2016) involves a learned \"Attention Function\" A(xi, yj ) which takes a \"source vector\" xi and a \"target vector\" yj , and must be computed for every source time step i and target time step j. In GNMT, the attention function is implemented as a feed forward neural network with a hidden layer of size n. It can be expressed as:\n\n$$A_{G N M T}(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}t a n h((x_{i}U){d}+(y{j}W)_{d})$$\n\nWhere U and W are trainable weight matrices and V is a trainable weight vector.\n\nFor performance reasons, in our models, we used a slightly different attention function:\n\n$$A(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}tanh((x_{i}U){d})tanh((y{j}W)_{d})\\tag{22}$$\n\nWith our attention function, we can simultaneously compute the attention function on multiple source time steps and multiple target time steps using optimized matrix multiplications. We found little difference in quality between the two functions.\n\n$$(21)$$",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/1701.06538v1.md"
+          },
+          "headlines": {
+            "1 Introduction And Related Work": [
+              "1.1 Conditional Computation",
+              "1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER",
+              "1.3 Related Work On Mixtures Of Experts"
+            ],
+            "2 The Structure Of The Mixture-Of-Experts Layer": [
+              "2.1 Gating Network"
+            ],
+            "3 Addressing Performance Challenges": [
+              "3.1 The Shrinking Batch Problem",
+              "3.2 Network Bandwidth"
+            ],
+            "4 Balancing Expert Utilization": [],
+            "5 Experiments": [
+              "5.1 1 Billion Word Language Modeling Benchmark",
+              "5.2 100 BILLION WORD GOOGLE NEWS CORPUS",
+              "5.3 Machine Translation (Single Language Pair)",
+              "5.4 Multilingual Machine Translation"
+            ],
+            "6 Conclusion": [],
+            "Appendices": [
+              "A Load-Balancing Loss",
+              "B Hierachical Mixture Of Experts",
+              "C 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS",
+              "D 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS",
+              "E Machine Translation - Experimental Details"
+            ]
+          },
+          "summary": "The paper introduces a Sparsely-Gated Mixture-of-Experts (MoE) layer to enhance neural network capacity without a proportional increase in computation. This approach uses a trainable gating network to select a sparse combination of experts for each input, significantly improving model capacity and performance in language modeling and machine translation tasks. The MoE layer, with up to 137 billion parameters, achieves better results than state-of-the-art models at lower computational costs. The paper addresses challenges like batch size and network bandwidth, proposing solutions like mixing data and model parallelism. Experiments show that MoE models outperform traditional models in various benchmarks, demonstrating the potential of conditional computation in deep learning.",
+          "summary_embedding": [
+            -0.01644345559179783,
+            -0.016995344310998917,
+            -0.016372699290513992,
+            -0.014349107630550861,
+            0.02282555401325226,
+            0.045934125781059265,
+            -0.014646278694272041,
+            0.03563220426440239,
+            -0.004829026758670807,
+            0.005617944523692131,
+            -0.00932550523430109,
+            -0.008143897168338299,
+            0.005575491581112146,
+            0.01538213063031435,
+            -0.005653322208672762,
+            -0.020306676626205444,
+            -0.013832597061991692,
+            -0.023235931992530823,
+            0.028344441205263138,
+            -0.020080260932445526,
+            0.011823155917227268,
+            -0.020943470299243927,
+            -0.025868017226457596,
+            -0.016500059515237808,
+            0.025061409920454025,
+            0.024806693196296692,
+            0.02583971433341503,
+            0.003827843815088272,
+            -0.03492465242743492,
+            0.033651065081357956,
+            0.014674580655992031,
+            0.012488252483308315,
+            -0.031500112265348434,
+            -0.009162768721580505,
+            -0.05351905897259712,
+            -0.05227377265691757,
+            0.04143410921096802,
+            0.022245362401008606,
+            -0.016231190413236618,
+            0.023717066273093224,
+            0.017264213413000107,
+            0.005816058721393347,
+            -0.020632149651646614,
+            -0.020844414830207825,
+            0.003700484987348318,
+            -0.013103820383548737,
+            -0.01983969286084175,
+            -0.013528349809348583,
+            -0.01828308403491974,
+            -0.01577835902571678,
+            0.009551920928061008,
+            0.0002856731880456209,
+            0.0029186424799263477,
+            -0.003424540627747774,
+            -0.047830358147621155,
+            0.0248208437114954,
+            0.019853845238685608,
+            0.03492465242743492,
+            0.04633035510778427,
+            0.013011839240789413,
+            0.01516986545175314,
+            -0.014271277002990246,
+            -0.04021712392568588,
+            0.005497660953551531,
+            -0.00041656987741589546,
+            -0.005380915477871895,
+            -0.005819596350193024,
+            0.008575502783060074,
+            0.008724087849259377,
+            -0.01143400277942419,
+            -0.04202845320105553,
+            0.013952880166471004,
+            -0.020561393350362778,
+            0.00010143388499272987,
+            0.01354957651346922,
+            0.008292482234537601,
+            0.02677368000149727,
+            0.03195294365286827,
+            0.023971782997250557,
+            -0.008724087849259377,
+            -0.01340806670486927,
+            0.019924599677324295,
+            -0.0022676968947052956,
+            0.007018893025815487,
+            0.04570771008729935,
+            -0.04033033177256584,
+            0.02163686975836754,
+            -0.025726506486535072,
+            -0.015339677222073078,
+            -0.005349075887352228,
+            -0.022471778094768524,
+            0.02806142158806324,
+            -0.007344366051256657,
+            0.017801949754357338,
+            0.017094401642680168,
+            0.01594817079603672,
+            0.013330236077308655,
+            -0.001894464367069304,
+            -0.022938761860132217,
+            0.0018360915128141642,
+            -0.01541043259203434,
+            0.02114158496260643,
+            0.0015230007702484727,
+            0.005469358991831541,
+            0.027594437822699547,
+            0.031160488724708557,
+            -0.023490650579333305,
+            0.009502392262220383,
+            0.02422650158405304,
+            0.06967949122190475,
+            0.015424583107233047,
+            -0.006435164716094732,
+            -0.0024640418123453856,
+            0.04143410921096802,
+            0.0015513027319684625,
+            0.0038349193055182695,
+            -0.06826439499855042,
+            -0.03022652305662632,
+            -0.0011365016689524055,
+            -0.02254253253340721,
+            0.02227366529405117,
+            0.01002597901970148,
+            -0.009750034660100937,
+            -0.022457627579569817,
+            -0.04516997188329697,
+            0.05190584436058998,
+            -0.00545874610543251,
+            -0.016259491443634033,
+            -0.005554265342652798,
+            -0.05340585112571716,
+            0.00039136342820711434,
+            0.015042506158351898,
+            -0.018481198698282242,
+            -0.03670767694711685,
+            0.036141637712717056,
+            0.0007398316520266235,
+            0.03741522878408432,
+            0.012459950521588326,
+            -0.01708024926483631,
+            -0.020094411447644234,
+            0.022259512916207314,
+            -0.011639192700386047,
+            -0.0038313816767185926,
+            -0.027339721098542213,
+            0.0065236082300543785,
+            -0.028103873133659363,
+            -0.0014566679019480944,
+            0.06730213016271591,
+            -0.04816998168826103,
+            0.02946236915886402,
+            -0.029660483822226524,
+            0.023660462349653244,
+            -0.008575502783060074,
+            0.0278916098177433,
+            -0.010238244198262691,
+            0.005840822588652372,
+            -0.013457595370709896,
+            -0.0010586712742224336,
+            -0.020745357498526573,
+            0.0034210027661174536,
+            -0.016259491443634033,
+            -0.027750099077820778,
+            -0.007054270710796118,
+            0.031273696571588516,
+            0.01698119193315506,
+            0.025655752047896385,
+            0.025429336354136467,
+            0.01246702577918768,
+            -0.04935866594314575,
+            0.02595292218029499,
+            0.0018926954362541437,
+            0.013514199294149876,
+            0.013181651011109352,
+            -0.0010418669553473592,
+            0.025599148124456406,
+            0.012049571610987186,
+            0.025061409920454025,
+            0.037443529814481735,
+            0.0027682881336659193,
+            0.04737752676010132,
+            -0.015693452209234238,
+            -0.005292471963912249,
+            0.028528403490781784,
+            0.000505455827806145,
+            0.02465103194117546,
+            -0.001686621573753655,
+            0.013330236077308655,
+            0.000776978034991771,
+            0.03834919258952141,
+            -0.006339645478874445,
+            0.029518973082304,
+            0.021608568727970123,
+            0.028316138312220573,
+            -0.014759486541152,
+            0.023122724145650864,
+            -0.019754787907004356,
+            -0.02292460948228836,
+            0.035547297447919846,
+            0.004082561936229467,
+            0.01886327564716339,
+            -0.028457649052143097,
+            -0.030679354444146156,
+            0.015452885068953037,
+            -0.01855195313692093,
+            0.016556663438677788,
+            0.03639635816216469,
+            0.0008273909334093332,
+            -0.0025719432160258293,
+            0.026292547583580017,
+            -0.04245298355817795,
+            0.012856177985668182,
+            -0.01532552670687437,
+            -0.02855670638382435,
+            -0.019528372213244438,
+            -0.00875946506857872,
+            -0.0018095583654940128,
+            -0.013691086322069168,
+            -0.0037570889107882977,
+            -0.007563706487417221,
+            -0.02459442801773548,
+            0.00935380719602108,
+            0.013627407141029835,
+            0.009849091991782188,
+            -0.018665160983800888,
+            0.0019952901639044285,
+            -0.008257105015218258,
+            -0.0160047747194767,
+            -0.03752843663096428,
+            0.015268922783434391,
+            -0.0024888061452656984,
+            0.009191070683300495,
+            0.04078316316008568,
+            0.028485950082540512,
+            -0.005055442452430725,
+            0.0329718142747879,
+            -0.033707667142152786,
+            0.017915157601237297,
+            -0.008320784196257591,
+            0.009191070683300495,
+            0.002989397384226322,
+            -0.025542544201016426,
+            0.027424626052379608,
+            -0.03792466223239899,
+            -0.038151077926158905,
+            0.022995365783572197,
+            0.011858533136546612,
+            0.0017096169758588076,
+            -0.019386861473321915,
+            -0.017717044800519943,
+            0.0008499440737068653,
+            0.002752368338406086,
+            -0.018834972754120827,
+            -0.01019579079002142,
+            0.02227366529405117,
+            -0.025500090792775154,
+            0.036085035651922226,
+            0.023391593247652054,
+            -0.03520767390727997,
+            -0.00608492735773325,
+            -0.01903308741748333,
+            0.059037946164608,
+            -0.0011506526498124003,
+            -0.010301923379302025,
+            0.022938761860132217,
+            -0.027268964797258377,
+            -0.004524780437350273,
+            0.022301966324448586,
+            0.01828308403491974,
+            -0.008278331719338894,
+            0.003983504604548216,
+            -0.022287815809249878,
+            -0.05281151086091995,
+            -0.010577867738902569,
+            -0.0003305583668407053,
+            0.054368119686841965,
+            -0.012367968447506428,
+            0.0020695829298347235,
+            -0.006413938011974096,
+            0.004100250545889139,
+            0.03704730048775673,
+            0.002140337834134698,
+            0.0025082635693252087,
+            -0.013698162510991096,
+            0.0371888130903244,
+            0.022316116839647293,
+            0.005883275531232357,
+            -0.012162779457867146,
+            0.005377377849072218,
+            -0.04899074137210846,
+            -0.0009118546731770039,
+            0.005741765722632408,
+            0.007386818993836641,
+            -0.025910470634698868,
+            0.017717044800519943,
+            0.009205221198499203,
+            -0.0024693484883755445,
+            -0.009792488068342209,
+            0.01173117384314537,
+            0.025429336354136467,
+            -0.019938750192523003,
+            -0.05700020492076874,
+            0.0034705311991274357,
+            0.012014193460345268,
+            -0.004054259974509478,
+            -0.019401011988520622,
+            0.006792477332055569,
+            0.0021934041287750006,
+            0.022698193788528442,
+            -0.011271266266703606,
+            -0.0029221803415566683,
+            0.040585052222013474,
+            0.009516543708741665,
+            -0.0016813150141388178,
+            -0.017717044800519943,
+            -0.011794853024184704,
+            0.018070818856358528,
+            0.004740582779049873,
+            0.0035289041697978973,
+            -0.01751893013715744,
+            -0.029264256358146667,
+            0.02800481766462326,
+            0.019967053085565567,
+            -0.048905834555625916,
+            -0.008200501091778278,
+            -0.019273653626441956,
+            0.0019599127117544413,
+            0.008695785887539387,
+            0.04613224044442177,
+            -0.00590803986415267,
+            0.008313708938658237,
+            -0.020179318264126778,
+            -0.05003791302442551,
+            -0.02708500251173973,
+            0.03707560524344444,
+            -0.028684064745903015,
+            -0.04276430234313011,
+            -0.010252394713461399,
+            0.016740625724196434,
+            -0.009403335861861706,
+            -0.018679311498999596,
+            -0.011738249100744724,
+            -0.04969828948378563,
+            -0.003497064346447587,
+            0.0337359681725502,
+            0.017264213413000107,
+            -0.008441068232059479,
+            -0.033877480775117874,
+            0.019641580060124397,
+            -0.007160402834415436,
+            -0.006474079564213753,
+            0.0007862646016292274,
+            0.008794843219220638,
+            0.02400008589029312,
+            0.011879759840667248,
+            -0.02763689123094082,
+            -0.03427370637655258,
+            -0.0024888061452656984,
+            0.014929298311471939,
+            -0.00878776703029871,
+            0.0245378240942955,
+            0.024679332971572876,
+            -0.022995365783572197,
+            -0.01654251106083393,
+            0.015750056132674217,
+            -0.05779265984892845,
+            -0.010287772864103317,
+            -0.029235953465104103,
+            -0.006183984223753214,
+            0.0009613831643946469,
+            0.0061096916906535625,
+            0.00012304731353651732,
+            0.03846240043640137,
+            0.008801918476819992,
+            0.006014172453433275,
+            0.009191070683300495,
+            0.01913214474916458,
+            0.028471799567341805,
+            0.009155693463981152,
+            -0.03517937287688255,
+            0.02850010246038437,
+            -0.017051948234438896,
+            0.007811348885297775,
+            -0.025754809379577637,
+            -0.013033065013587475,
+            0.00443633645772934,
+            0.02725481428205967,
+            -0.000503244751598686,
+            0.018849123269319534,
+            -0.021919889375567436,
+            0.037500135600566864,
+            -0.029915200546383858,
+            -0.05278320610523224,
+            -0.02005195803940296,
+            -0.008285406976938248,
+            0.015764206647872925,
+            0.017278363928198814,
+            -0.019924599677324295,
+            -0.0286557637155056,
+            -0.021750077605247498,
+            0.006470541935414076,
+            0.010542490519583225,
+            -0.021099131554365158,
+            -0.010627396404743195,
+            0.01502835564315319,
+            -0.00404364662244916,
+            -0.01886327564716339,
+            0.013684011064469814,
+            0.015693452209234238,
+            0.03169822692871094,
+            -0.03308502212166786,
+            0.004287751391530037,
+            -0.025967074558138847,
+            -0.04327373951673508,
+            0.004096712917089462,
+            0.03166992589831352,
+            0.027509532868862152,
+            -0.018636859953403473,
+            0.010754755698144436,
+            0.003990580327808857,
+            0.004234685096889734,
+            -0.0033803186379373074,
+            0.023603858426213264,
+            0.03291521221399307,
+            0.02763689123094082,
+            -0.011653343215584755,
+            0.004921008367091417,
+            -0.01559439580887556,
+            -0.0011108530452474952,
+            0.01865101046860218,
+            -0.017816102132201195,
+            0.033877480775117874,
+            -0.029405765235424042,
+            0.02104252763092518,
+            -0.035122767090797424,
+            0.006141531281173229,
+            -0.012099100276827812,
+            -0.00404364662244916,
+            0.04833979532122612,
+            -0.017646290361881256,
+            -0.0019546060357242823,
+            -0.008101443760097027,
+            -0.014575524255633354,
+            -0.02158026583492756,
+            0.008766540326178074,
+            -0.04033033177256584,
+            -0.012367968447506428,
+            -0.028429346159100533,
+            0.003424540627747774,
+            0.019910449162125587,
+            -0.06277380883693695,
+            -0.06237758323550224,
+            0.012275987304747105,
+            -0.04918885603547096,
+            -0.026419905945658684,
+            0.02963218092918396,
+            -0.002984090941026807,
+            0.0012780115939676762,
+            -0.0024003623984754086,
+            0.0068101659417152405,
+            -0.029207652434706688,
+            0.00425944896414876,
+            -0.000887974863871932,
+            0.03271709755063057,
+            -0.021056678146123886,
+            -0.008122670464217663,
+            0.004878554958850145,
+            -0.06860402226448059,
+            -0.028344441205263138,
+            0.011271266266703606,
+            -0.009459939785301685,
+            -0.0066615804098546505,
+            -0.017915157601237297,
+            -0.004383270163089037,
+            0.0008950503543019295,
+            0.0032830305863171816,
+            0.05801907554268837,
+            -0.01983969286084175,
+            -0.053773775696754456,
+            0.03633975237607956,
+            0.00887267291545868,
+            0.029603879898786545,
+            -0.025556694716215134,
+            -0.017533080652356148,
+            -0.01855195313692093,
+            -0.020504789426922798,
+            -0.04123599827289581,
+            0.00404364662244916,
+            -0.005918653216212988,
+            0.009212297387421131,
+            -0.0009693430620245636,
+            0.02763689123094082,
+            0.029235953465104103,
+            0.023108573630452156,
+            0.012750045396387577,
+            0.01079013291746378,
+            -0.022372720763087273,
+            -0.021594416350126266,
+            0.0012222920777276158,
+            0.04683978855609894,
+            0.024636881425976753,
+            -0.02675952948629856,
+            -0.011504758149385452,
+            0.03953787684440613,
+            0.0419718474149704,
+            -0.01974063739180565,
+            -0.02276895008981228,
+            0.00764861237257719,
+            0.03266049548983574,
+            -0.04477374628186226,
+            0.005932804197072983,
+            0.03410389646887779,
+            0.014405711553990841,
+            -0.005356151144951582,
+            -0.010294848121702671,
+            0.022570835426449776,
+            0.03152841329574585,
+            -0.008724087849259377,
+            0.01876421831548214,
+            0.016910437494516373,
+            0.029349161311984062,
+            -0.0067889392375946045,
+            0.0005766529939137399,
+            -0.024735936895012856,
+            0.055188875645399094,
+            -0.03059444949030876,
+            0.02909444272518158,
+            0.010316074825823307,
+            -0.00593634182587266,
+            0.039056744426488876,
+            -0.016641568392515182,
+            -0.0431322306394577,
+            0.01913214474916458,
+            -0.006979977712035179,
+            -0.0055613406002521515,
+            -0.03444352000951767,
+            -0.007592008449137211,
+            -0.007507102098315954,
+            -0.006714646704494953,
+            -0.022556684911251068,
+            -0.02606613002717495,
+            0.004415110219269991,
+            0.00973588414490223,
+            -0.006484692916274071,
+            0.00698705343529582,
+            0.02092931978404522,
+            -0.013507124036550522,
+            0.02525952458381653,
+            -0.046047333627939224,
+            -0.0008366775000467896,
+            -0.0038596836384385824,
+            -0.025712355971336365,
+            -0.0345001220703125,
+            -0.0005770951975136995,
+            -0.0024251264985650778,
+            -0.024622729048132896,
+            -0.00914154201745987,
+            -0.008752389810979366,
+            -0.0023119186516851187,
+            -0.004938696976751089,
+            -0.055188875645399094,
+            0.008695785887539387,
+            0.010846736840903759,
+            -0.002713453024625778,
+            0.0055507272481918335,
+            -0.010613245889544487,
+            -0.03039633482694626,
+            0.03039633482694626,
+            0.04129260033369064,
+            0.033707667142152786,
+            0.014334957115352154,
+            -0.0074646491557359695,
+            -0.017165156081318855,
+            0.006060163024812937,
+            0.021721776574850082,
+            0.00902833417057991,
+            0.007846726104617119,
+            0.03877372294664383,
+            0.00011331850691931322,
+            -0.008349086157977581,
+            -0.009764186106622219,
+            0.028740668669342995,
+            -0.001085204305127263,
+            0.023603858426213264,
+            0.050802066922187805,
+            0.030424637719988823,
+            0.020589696243405342,
+            -0.01681138016283512,
+            -0.02010856196284294,
+            -0.00851889792829752,
+            -0.02855670638382435,
+            0.0364246591925621,
+            0.012127402238547802,
+            -0.005232329946011305,
+            -0.011540135368704796,
+            0.019061388447880745,
+            -0.033339742571115494,
+            -0.0037358624394983053,
+            0.00048422932741232216,
+            0.013924578204751015,
+            -0.03379257395863533,
+            0.0002480846014805138,
+            0.031613320112228394,
+            0.014957600273191929,
+            -0.016995344310998917,
+            -0.0188774261623621,
+            0.010549565777182579,
+            0.020462337881326675,
+            0.018948180601000786,
+            0.00037743354914709926,
+            0.00897173024713993,
+            -0.00022121983056422323,
+            -0.0005319889169186354,
+            0.01590571738779545,
+            -0.007952858693897724,
+            0.036254845559597015,
+            -0.003525366308167577,
+            -0.01325240544974804,
+            -0.0037500134203583,
+            -0.021438755095005035,
+            -0.024467067793011665,
+            0.043556760996580124,
+            0.023971782997250557,
+            0.04661337286233902,
+            -0.005660397466272116,
+            -0.004245297983288765,
+            0.025825563818216324,
+            0.008221727795898914,
+            -0.03288691118359566,
+            0.000265110022155568,
+            0.005281858611851931,
+            -0.02892463095486164,
+            0.016641568392515182,
+            -0.02476423978805542,
+            -0.016896286979317665,
+            -0.006771250627934933,
+            -0.0015557248843833804,
+            -0.028188779950141907,
+            0.01703779585659504,
+            0.0031309074256569147,
+            -0.0027099153958261013,
+            -0.039764292538166046,
+            0.006880920846015215,
+            0.020419884473085403,
+            0.015311375260353088,
+            0.0037358624394983053,
+            0.011525984853506088,
+            -0.016613267362117767,
+            0.002126186853274703,
+            0.03379257395863533,
+            0.004878554958850145,
+            0.02005195803940296,
+            0.0027647505048662424,
+            0.023773670196533203,
+            0.03633975237607956,
+            -0.01681138016283512,
+            -0.02189158834517002,
+            0.01519816741347313,
+            -0.02233026921749115,
+            0.029943503439426422,
+            0.004889168310910463,
+            -0.012283062562346458,
+            0.022033097222447395,
+            -0.009827865287661552,
+            -0.0027346794959157705,
+            0.01659911684691906,
+            -0.022740647196769714,
+            -0.017717044800519943,
+            0.03183973580598831,
+            -0.039452970027923584,
+            0.012856177985668182,
+            -0.008547200821340084,
+            0.015863263979554176,
+            -0.023066120222210884,
+            0.011172209866344929,
+            0.007100261282175779,
+            -0.01573590561747551,
+            0.010797208175063133,
+            0.009693430736660957,
+            0.00545874610543251,
+            0.0018926954362541437,
+            0.02179253101348877,
+            -0.0006049550138413906,
+            -0.011257115751504898,
+            -0.018155725672841072,
+            -0.004694592207670212,
+            -0.002614396158605814,
+            -0.016556663438677788,
+            0.01865101046860218,
+            -0.041066184639930725,
+            -0.013280707411468029,
+            -0.008879749104380608,
+            -0.013988257385790348,
+            -0.010011828504502773,
+            0.024184048175811768,
+            0.025669902563095093,
+            -0.002361447084695101,
+            -0.031217092648148537,
+            0.01790100708603859,
+            0.024198200553655624,
+            0.003654494183138013,
+            -0.023646311834454536,
+            0.0302831269800663,
+            -0.008936353027820587,
+            0.014971751719713211,
+            0.026136886328458786,
+            0.014094389975070953,
+            0.004595535341650248,
+            -0.010500037111341953,
+            -0.02114158496260643,
+            0.01481609046459198,
+            0.010507113300263882,
+            0.033537857234478,
+            0.014745336025953293,
+            0.02071705460548401,
+            0.01594817079603672,
+            -0.023320838809013367,
+            -0.011214662343263626,
+            -0.003482913365587592,
+            -0.022514231503009796,
+            0.03093407303094864,
+            0.0006129149114713073,
+            0.02114158496260643,
+            0.026618018746376038,
+            -0.03319823369383812,
+            -0.01187268365174532,
+            0.032688796520233154,
+            0.02779255248606205,
+            0.05108508840203285,
+            0.025316128507256508,
+            0.015396281145513058,
+            0.027226513251662254,
+            -0.03585861995816231,
+            -0.036368053406476974,
+            0.001548649393953383,
+            0.04256619140505791,
+            -0.008426916785538197,
+            0.00967928022146225,
+            0.03900013864040375,
+            -0.006781863979995251,
+            -0.01611798256635666,
+            0.019457615911960602,
+            0.0026710000820457935,
+            0.002448122017085552,
+            -0.0033113325480371714,
+            -0.01335146278142929,
+            0.03175482898950577,
+            -0.0051474240608513355,
+            0.013570803217589855,
+            -0.019273653626441956,
+            0.027113303542137146,
+            0.0001799829478841275,
+            -0.003229964291676879,
+            0.028344441205263138,
+            -0.002660386962816119,
+            -0.051990751177072525,
+            0.024212351068854332,
+            0.019811391830444336,
+            -0.011561362072825432,
+            -0.0036332677118480206,
+            -0.003792466362938285,
+            0.04567940905690193,
+            0.034698236733675,
+            0.029009537771344185,
+            -0.0020501252729445696,
+            0.0168679840862751,
+            0.018792519345879555,
+            0.0053667644970119,
+            0.014278353191912174,
+            -0.0017025414854288101,
+            -0.007853801362216473,
+            0.021056678146123886,
+            -0.020787810906767845,
+            0.03226426616311073,
+            -0.016570813953876495,
+            -0.020122714340686798,
+            0.008738238364458084,
+            -0.011172209866344929,
+            -0.0009083169279620051,
+            -0.006311343517154455,
+            -0.0008813415770418942,
+            -0.0029664020985364914,
+            0.016938740387558937,
+            -0.004446949809789658,
+            0.04661337286233902,
+            -0.013931653462350368,
+            -0.008808993734419346,
+            0.006972902454435825,
+            -0.025981225073337555,
+            0.0013346156338229775,
+            0.0028726516757160425,
+            0.027000095695257187,
+            -0.01989629678428173,
+            -0.00970758218318224,
+            0.01606137864291668,
+            0.06039644405245781,
+            -0.008639181964099407,
+            -0.0026267783250659704,
+            0.027226513251662254,
+            -0.002787745790556073,
+            -0.03551899641752243,
+            0.012919857166707516,
+            -0.04494355618953705,
+            0.017589686438441277,
+            0.026349151507019997,
+            -0.04986810311675072,
+            -0.022089701145887375,
+            -0.02347649820148945,
+            -0.02043403498828411,
+            0.023434046655893326,
+            -0.009976450353860855,
+            0.013867974281311035,
+            -0.017801949754357338,
+            -0.009014182724058628,
+            0.020518941804766655,
+            0.009176919236779213,
+            0.03735862299799919,
+            -0.027551984414458275,
+            -0.016783079132437706,
+            -0.027820853516459465,
+            -0.04304732382297516,
+            0.009558996185660362,
+            -0.0278916098177433,
+            0.0027099153958261013,
+            -0.0016291331266984344,
+            -0.040302030742168427,
+            -0.037981268018484116,
+            0.005278320983052254,
+            -0.017985913902521133,
+            0.01516986545175314,
+            0.020915169268846512,
+            0.02731141820549965,
+            -0.0020253609400242567,
+            -0.030198222026228905,
+            0.029235953465104103,
+            -0.007344366051256657,
+            0.03223596513271332,
+            -0.00689507182687521,
+            0.005066055804491043,
+            -0.0070436573587358,
+            0.031160488724708557,
+            -0.023009516298770905,
+            0.018792519345879555,
+            -0.026801982894539833,
+            -0.004086099565029144,
+            -0.018990634009242058,
+            -0.01774534583091736,
+            0.004224071744829416,
+            0.004521242808550596,
+            7.888073741924018e-05,
+            0.0048750173300504684,
+            0.005777143407613039,
+            0.024806693196296692,
+            0.0021421066485345364,
+            0.02589631825685501,
+            -0.027877457439899445,
+            -0.025316128507256508,
+            0.01719345711171627,
+            -0.02077365852892399,
+            0.005678086541593075,
+            -0.006261814851313829,
+            -0.0302831269800663,
+            -0.01202834490686655,
+            0.023391593247652054,
+            0.02162271924316883,
+            -0.005490585695952177,
+            0.002644466934725642,
+            0.02206140011548996,
+            -0.026023678481578827,
+            0.00437973253428936,
+            -0.03781145438551903,
+            -0.018396291881799698,
+            0.029773691669106483,
+            -0.03195294365286827,
+            0.020745357498526573,
+            0.003185742534697056,
+            0.03483974561095238,
+            -0.017292514443397522,
+            -0.01947176828980446,
+            0.01654251106083393,
+            -0.01008258294314146,
+            0.004142703488469124,
+            0.01606137864291668,
+            0.01616043597459793,
+            -0.0015070808585733175,
+            0.05380208045244217,
+            0.0008167776977643371,
+            0.006909222807735205,
+            -0.049075648188591,
+            0.004496478475630283,
+            -0.044858649373054504,
+            0.0002768288250081241,
+            -0.013337312266230583,
+            -0.02514631673693657,
+            0.0014027173165231943,
+            0.001001182827167213,
+            0.001963450340554118,
+            -0.01913214474916458,
+            -0.01989629678428173,
+            0.011596739292144775,
+            -0.015863263979554176,
+            0.004928083624690771,
+            -0.06418890506029129,
+            -0.01855195313692093,
+            -0.004974074196070433,
+            0.0043337419629096985,
+            0.0007699025445617735,
+            0.03342464938759804,
+            -0.01478778850287199,
+            0.010117961093783379,
+            -0.01549533847719431,
+            -0.026363302022218704,
+            0.04010391607880592,
+            0.018750067800283432,
+            -0.023773670196533203,
+            0.024693485349416733,
+            0.012106175534427166,
+            0.03099067695438862,
+            -0.0318114347755909,
+            0.020349130034446716,
+            0.018523650243878365,
+            -0.0029823218937963247,
+            0.006551910191774368,
+            0.008575502783060074,
+            0.01832553744316101,
+            -0.015764206647872925,
+            -0.0003632825391832739,
+            0.011419852264225483,
+            0.010033054277300835,
+            0.031924642622470856,
+            0.01403071079403162,
+            -0.03183973580598831,
+            0.019047237932682037,
+            0.01616043597459793,
+            0.02298121340572834,
+            0.033820874989032745,
+            -0.027481229975819588,
+            0.038575608283281326,
+            -0.010386829264461994,
+            -0.024042539298534393,
+            0.020801961421966553,
+            -0.032688796520233154,
+            -0.006679269019514322,
+            0.030735958367586136,
+            0.04398128762841225,
+            -0.0015566092915832996,
+            -0.044632233679294586,
+            0.01535382866859436,
+            0.01316749956458807,
+            0.0027806703001260757,
+            0.028811423107981682,
+            0.011264191009104252,
+            0.021820832043886185,
+            0.025273675099015236,
+            0.011419852264225483,
+            -0.0037252490874379873,
+            0.024962352588772774,
+            -0.008101443760097027,
+            -0.004446949809789658,
+            -0.0007035697344690561,
+            0.03206615149974823,
+            0.014702882617712021,
+            -0.017603836953639984,
+            -0.036962397396564484,
+            -0.007620310410857201,
+            0.009452863596379757,
+            -0.007903330028057098,
+            -0.027127455919981003,
+            -0.03950957581400871,
+            -0.026787832379341125,
+            -0.005667473189532757,
+            0.01505665760487318,
+            -0.06724552065134048,
+            -0.007620310410857201,
+            -0.012233533896505833,
+            0.017717044800519943,
+            0.0022906921803951263,
+            0.0181132722645998,
+            0.001277127186767757,
+            -0.009403335861861706,
+            0.004209920763969421,
+            0.050802066922187805,
+            -0.03059444949030876,
+            0.014242975041270256,
+            0.011292492970824242,
+            0.0032105068676173687,
+            0.002787745790556073,
+            0.038688816130161285,
+            -0.010988246649503708,
+            -0.030424637719988823,
+            0.011575513519346714,
+            -0.017292514443397522,
+            0.003383856499567628,
+            0.05006621778011322,
+            -0.009481165558099747,
+            -0.027495380491018295,
+            0.020915169268846512,
+            -0.009148617275059223,
+            -0.015523640438914299,
+            -0.0027258351910859346,
+            -0.006035399157553911,
+            0.0006283925613388419,
+            -0.03832089155912399,
+            -0.015750056132674217,
+            0.03212275728583336,
+            0.008617955259978771,
+            0.014603826217353344,
+            0.025344429537653923,
+            0.010209942236542702,
+            -0.030481241643428802,
+            0.027481229975819588,
+            -0.0005678086308762431,
+            0.005221716593950987,
+            0.013323160819709301,
+            0.03424540534615517,
+            -0.014264201745390892,
+            -0.020264223217964172,
+            -0.040245428681373596,
+            -0.013853822834789753,
+            -0.0463869571685791,
+            0.001943992800079286,
+            0.006919836159795523,
+            -0.012919857166707516,
+            -0.04366996884346008,
+            -0.0329718142747879,
+            0.021127434447407722,
+            0.03178313374519348,
+            -0.017023645341396332,
+            0.0025294902734458447,
+            -0.008483520708978176,
+            -0.007882103323936462,
+            -0.0017520699184387922,
+            -0.013217028230428696,
+            -0.006378560326993465,
+            -0.006229975260794163,
+            0.015452885068953037,
+            -0.004652139265090227,
+            -0.0036934094969183207,
+            -0.0498398020863533,
+            -0.018778368830680847,
+            0.004082561936229467,
+            -0.008221727795898914,
+            0.007323139347136021,
+            -0.006955213379114866,
+            -0.002727604005485773,
+            -0.012474101036787033,
+            -0.03829259052872658,
+            0.024184048175811768,
+            0.017278363928198814,
+            0.0035501306410878897,
+            -0.008922201581299305,
+            -0.018367990851402283,
+            0.03294351324439049,
+            0.012523629702627659,
+            -0.017547233030200005,
+            -0.018580254167318344,
+            0.025981225073337555,
+            0.026957644149661064,
+            0.008122670464217663,
+            -0.006137993652373552,
+            0.029886899515986443,
+            0.005695775151252747,
+            -0.022938761860132217,
+            -0.003983504604548216,
+            -0.005055442452430725,
+            -0.0038844477385282516,
+            0.020561393350362778,
+            -0.010556641966104507,
+            0.0300001073628664,
+            0.038717120885849,
+            0.012870329432189465,
+            0.0039198254235088825,
+            -0.0033714743331074715,
+            -0.008922201581299305,
+            0.015976471826434135,
+            0.006481155287474394,
+            0.008525974117219448,
+            0.014759486541152,
+            -0.016358548775315285,
+            -0.020405733957886696,
+            0.006215824279934168,
+            -0.007107336539775133,
+            0.020122714340686798,
+            0.01708024926483631,
+            0.016429303213953972,
+            0.0007610581233166158,
+            -0.028287837281823158,
+            -0.0018024828750640154,
+            0.02411329373717308,
+            0.02010856196284294,
+            -0.004655676893889904,
+            -0.023264234885573387,
+            0.014398636296391487,
+            -0.015806660056114197,
+            -0.009969375096261501,
+            0.0033431723713874817,
+            0.006229975260794163,
+            -0.003083147807046771,
+            0.0571134127676487,
+            -0.020193468779325485,
+            -0.011419852264225483,
+            -0.010287772864103317,
+            -0.010867963545024395,
+            0.0012072566896677017,
+            -0.01828308403491974,
+            0.008964654989540577,
+            0.025613298639655113,
+            -0.009764186106622219,
+            0.03568880632519722,
+            0.0228114016354084,
+            0.006912760436534882,
+            0.005260631907731295,
+            -0.027070851996541023,
+            0.012304289266467094,
+            -0.016047226265072823,
+            0.005660397466272116,
+            0.018467046320438385,
+            0.017320817336440086,
+            0.0076132346875965595,
+            0.026306698098778725,
+            0.019386861473321915,
+            0.016684021800756454,
+            -0.022783100605010986,
+            0.025457637384533882,
+            -0.007832575589418411,
+            0.027877457439899445,
+            -0.01611798256635666,
+            -0.002253545681014657,
+            -0.012990612536668777,
+            0.008681634441018105,
+            0.002416282193735242,
+            -0.0019351483788341284,
+            0.00036129256477579474,
+            0.016783079132437706,
+            -0.014589674770832062,
+            0.01062032114714384,
+            0.013316085562109947,
+            -0.027919910848140717,
+            -0.010372678749263287,
+            0.016344398260116577,
+            -0.00668988237157464,
+            0.006877383217215538,
+            -0.011865608394145966,
+            -0.0017202302115038037,
+            0.011009473353624344,
+            -0.0436982698738575,
+            0.0104929618537426,
+            0.006279503460973501,
+            -0.015155714936554432,
+            0.014801939949393272,
+            -0.009799563325941563,
+            -0.008235878311097622,
+            -0.013592029921710491,
+            0.003792466362938285,
+            -0.00030225637601688504,
+            -0.0018290160223841667,
+            -0.00022995365725364536,
+            0.00389859895221889,
+            0.010047205723822117,
+            -0.0045990729704499245,
+            -0.010450509376823902,
+            -0.0028638073708862066,
+            -0.02265574224293232,
+            -0.0010047205723822117,
+            0.018735915422439575,
+            -0.004591997712850571,
+            0.0005222601466812193,
+            0.0145189194008708,
+            -0.005830209702253342,
+            0.010556641966104507,
+            -0.005221716593950987,
+            -0.008320784196257591,
+            -0.010768906213343143,
+            -0.033764272928237915,
+            0.0026710000820457935,
+            -0.015254771336913109,
+            0.008865597657859325,
+            -0.012544856406748295,
+            0.04027372971177101,
+            0.01022409275174141,
+            -0.00837738811969757,
+            -0.002796590095385909,
+            -0.0228114016354084,
+            -0.004475251771509647,
+            -0.016938740387558937,
+            -0.01184438169002533,
+            0.008342010900378227,
+            0.012629762291908264,
+            0.005614406894892454,
+            -0.018410442396998405,
+            0.003665107535198331,
+            0.0030955299735069275,
+            -0.005575491581112146,
+            -0.008370312862098217,
+            0.010853812098503113,
+            -0.0006248548161238432,
+            -0.01800006441771984,
+            -0.01957082375884056,
+            -0.027424626052379608,
+            0.0150000536814332,
+            0.014589674770832062,
+            -0.01011088490486145,
+            -0.022754797711968422,
+            -0.007471724878996611,
+            0.007107336539775133,
+            -0.02064630016684532,
+            0.0006779210525564849,
+            0.006891534198075533,
+            0.008717012591660023,
+            0.004117939155548811,
+            0.009636826813220978,
+            0.0007699025445617735,
+            0.0121698547154665,
+            0.018566103652119637,
+            0.03271709755063057,
+            -0.0396793857216835,
+            -0.009537769481539726,
+            0.0057028504088521,
+            -0.001465512323193252,
+            -0.013853822834789753,
+            0.009672204032540321,
+            0.0024375086650252342,
+            -0.0026426981203258038,
+            0.024424616247415543,
+            -0.008044839836657047,
+            -0.01505665760487318,
+            0.01184438169002533,
+            0.00887267291545868,
+            -0.019075538963079453,
+            0.02746707946062088,
+            0.006169833242893219,
+            -0.01011088490486145,
+            0.02259913645684719,
+            -0.012360893189907074,
+            0.012658064253628254,
+            -0.005522425286471844,
+            0.023660462349653244,
+            -0.019004784524440765,
+            0.0006306036957539618,
+            0.015665151178836823,
+            0.03883032873272896,
+            -0.00437973253428936,
+            -0.004567233379930258,
+            -0.03421710431575775,
+            0.005115584470331669,
+            0.028853876516222954,
+            0.02666047215461731,
+            -0.0012346742441877723,
+            -0.005596718285232782,
+            -0.0019864458590745926,
+            0.010768906213343143,
+            0.011589664034545422,
+            0.017561383545398712,
+            -0.029547275975346565,
+            0.00022011429246049374,
+            -0.016938740387558937,
+            0.025500090792775154,
+            0.014632128179073334,
+            0.006951675750315189,
+            0.04072656109929085,
+            0.0027789014857262373,
+            0.003841994795948267,
+            0.027452928945422173,
+            -0.00779719790443778,
+            0.003546592779457569,
+            -0.015566092915832996,
+            0.0003175129240844399,
+            0.022853855043649673,
+            0.0016370931407436728,
+            0.0020784272346645594,
+            0.013337312266230583,
+            0.029320860281586647,
+            -0.022471778094768524,
+            -0.023929331451654434,
+            0.003482913365587592,
+            0.007185167167335749,
+            -0.010570792481303215,
+            0.0038738346192985773,
+            0.006275965832173824,
+            -0.006007096730172634,
+            0.008900974877178669,
+            0.006332569755613804,
+            -0.0003166284877806902,
+            0.0039198254235088825,
+            -0.00308845448307693,
+            0.012962310574948788,
+            -0.02702839858829975,
+            0.036849189549684525,
+            0.004439874552190304,
+            0.010075507685542107,
+            0.015268922783434391,
+            0.001627364312298596,
+            0.004917470272630453,
+            0.005869124550372362,
+            0.010188715532422066,
+            0.018636859953403473,
+            -0.00956607237458229,
+            -0.027481229975819588,
+            -0.03166992589831352,
+            0.020363280549645424,
+            -0.01173117384314537,
+            0.0035872769076377153,
+            0.0074646491557359695,
+            -0.01828308403491974,
+            -0.007507102098315954,
+            -0.009976450353860855,
+            0.015764206647872925,
+            -0.03189634159207344,
+            0.012990612536668777,
+            -0.0043231286108493805,
+            -0.028669914230704308,
+            0.00851889792829752,
+            -0.010181640274822712,
+            -0.0045884596183896065,
+            -0.0035094465129077435,
+            -0.029264256358146667,
+            -0.007931631989777088,
+            0.03908504545688629,
+            -0.02141045406460762,
+            0.024184048175811768,
+            0.008257105015218258,
+            -0.00014449491573031992,
+            -0.0040365708991885185,
+            -0.013563727959990501,
+            0.005331386812031269,
+            -0.01778779923915863,
+            0.025457637384533882,
+            0.010294848121702671,
+            -0.013924578204751015,
+            0.0322076641023159,
+            -0.011179285123944283,
+            0.027438776567578316,
+            -0.018580254167318344,
+            0.036198243498802185,
+            0.007422196213155985,
+            0.00674648629501462,
+            0.01974063739180565,
+            -0.004025958012789488,
+            0.008228803053498268,
+            0.04559450224041939,
+            -0.018523650243878365,
+            0.02460857853293419,
+            0.020349130034446716,
+            -0.00022685811563860625,
+            0.017915157601237297,
+            0.0028390432707965374,
+            -0.023165177553892136,
+            0.009495317004621029,
+            -0.01135617308318615,
+            -0.009955224581062794,
+            0.01583496294915676,
+            -0.019316107034683228,
+            -0.018721764907240868,
+            -0.025613298639655113,
+            0.0029416377656161785,
+            -0.003350247861817479,
+            -0.0077618202194571495,
+            -0.0019616815261542797,
+            0.01454722136259079,
+            0.008724087849259377,
+            0.025669902563095093,
+            0.0008326975512318313,
+            0.011971740983426571,
+            0.01511326152831316,
+            -0.00035576484515331686,
+            0.02313687466084957,
+            -0.03229256719350815,
+            0.01246702577918768,
+            -0.004517704714089632,
+            -0.00509789539501071,
+            0.023858575150370598,
+            0.0003747802402358502,
+            -0.0004877670726273209,
+            -0.0024888061452656984,
+            0.0005496777012012899,
+            -0.014589674770832062,
+            -0.0057099261321127415,
+            0.000907432462554425,
+            -0.005175726022571325,
+            0.04299071803689003,
+            -0.00701535539701581,
+            -0.004772422835230827,
+            -0.039169952273368835,
+            -0.01918874867260456,
+            0.00623351288959384,
+            -0.01583496294915676,
+            -0.018778368830680847,
+            -0.006307805422693491,
+            0.005801907740533352,
+            -0.04893413558602333,
+            0.008342010900378227,
+            0.03336804360151291,
+            -0.012219383381307125,
+            0.00259493850171566,
+            -0.010981171391904354,
+            -0.006969364359974861,
+            -0.0029115669894963503,
+            -7.766463386360556e-05,
+            -0.002412744564935565,
+            -0.020830262452363968,
+            -0.005617944523692131,
+            0.003364398842677474,
+            -0.0008795727044343948,
+            -0.025047259405255318,
+            0.0006177793256938457,
+            0.018849123269319534,
+            -0.003979966975748539,
+            -0.0011152751976624131,
+            -0.001825478277169168,
+            0.006399787031114101,
+            -0.01508495956659317,
+            0.008009462617337704,
+            -0.0076344613917171955,
+            0.003841994795948267,
+            -0.0030495391692966223,
+            -0.002214630600064993,
+            0.02844349853694439,
+            0.021594416350126266,
+            -0.04808507859706879,
+            -0.01405901275575161,
+            0.023009516298770905,
+            0.007280686404556036,
+            -0.004740582779049873,
+            0.01178777776658535,
+            -0.031613320112228394,
+            -0.00028943203506059945,
+            -0.026787832379341125,
+            0.02200479619204998,
+            -0.009205221198499203,
+            -0.007386818993836641,
+            0.016967041417956352,
+            -0.026632171124219894,
+            0.016344398260116577,
+            0.00794578343629837,
+            -0.00030181417241692543,
+            -0.000493515923153609,
+            0.0063856360502541065,
+            0.0008972614887170494,
+            -0.01659911684691906,
+            0.006442239973694086,
+            -0.0048502529971301556,
+            0.007959933951497078,
+            -0.005087282508611679,
+            -0.05527378246188164,
+            -0.009933997876942158,
+            -0.005497660953551531,
+            -0.008936353027820587,
+            -0.012820800766348839,
+            0.008136821910738945,
+            0.02827368676662445,
+            0.01606137864291668,
+            -0.009863242506980896,
+            -4.8312376748071983e-05,
+            0.033000119030475616,
+            -0.0008300442132167518,
+            0.009955224581062794,
+            -0.011349096894264221,
+            0.00015709814033471048,
+            -0.0006911876262165606,
+            -0.014660430140793324,
+            -0.019061388447880745,
+            0.024042539298534393,
+            -0.028344441205263138,
+            0.02378782071173191,
+            -0.01843874529004097,
+            -0.0130825936794281,
+            0.01757553406059742,
+            -0.032575588673353195,
+            -0.005108508747071028,
+            0.007450498174875975,
+            0.0090566361322999,
+            0.021608568727970123,
+            0.0032794929575175047,
+            0.020589696243405342,
+            -0.03716050833463669,
+            -0.009134466759860516,
+            -0.010160413570702076,
+            -0.00794578343629837,
+            -0.010231168940663338,
+            -0.011596739292144775,
+            0.021863285452127457,
+            -0.015240620821714401,
+            -0.016952890902757645,
+            0.007054270710796118,
+            0.0030212372075766325,
+            -0.011313719674944878,
+            0.015750056132674217,
+            0.00017810352437663823,
+            0.004712280817329884,
+            0.013011839240789413,
+            -0.0008923970744945109,
+            0.04086806997656822,
+            0.0253019779920578,
+            -0.0030212372075766325,
+            -0.008568426594138145,
+            0.021438755095005035,
+            0.0004285097820684314,
+            0.0150000536814332,
+            0.009375033900141716,
+            0.023264234885573387,
+            -0.02092931978404522,
+            0.009997677057981491,
+            -0.005501199048012495,
+            0.014603826217353344,
+            -0.009495317004621029,
+            -0.015339677222073078,
+            0.00479718716815114,
+            -0.01577835902571678,
+            0.010153338313102722,
+            -0.010860888287425041,
+            0.008221727795898914,
+            0.01913214474916458,
+            0.002467579673975706,
+            -0.00941041111946106,
+            0.01146230474114418,
+            0.02075950801372528,
+            -0.025641601532697678,
+            0.008851447142660618,
+            -0.0031998935155570507,
+            -0.0021916350815445185,
+            0.021127434447407722,
+            0.010301923379302025,
+            -0.01594817079603672,
+            0.01573590561747551,
+            -0.008879749104380608,
+            -0.0015053119277581573,
+            -0.01882082223892212,
+            -0.03291521221399307,
+            -0.0015283073298633099,
+            0.012396270409226418,
+            0.004800724796950817,
+            0.0150000536814332,
+            0.011264191009104252,
+            -0.0008950503543019295,
+            -0.006601438857614994,
+            0.0008433107868768275,
+            -0.013485897332429886,
+            -0.004963461309671402,
+            0.010231168940663338,
+            0.017051948234438896,
+            -0.0037217114586383104,
+            0.015650998800992966,
+            0.0088160689920187,
+            0.006137993652373552,
+            -0.0010480580385774374,
+            0.0036138100549578667,
+            -0.002060738392174244,
+            -0.007783046457916498,
+            0.019938750192523003,
+            0.019075538963079453,
+            -0.018509499728679657,
+            -0.0024870370980352163,
+            -0.02141045406460762,
+            0.025882167741656303,
+            0.022188758477568626,
+            0.011165134608745575,
+            -0.04222656786441803,
+            0.00012050455552525818,
+            0.00752832880243659,
+            -0.0013744152383878827,
+            -0.031103884801268578,
+            0.008172199130058289,
+            0.005221716593950987,
+            0.004330204334110022,
+            0.020844414830207825,
+            -0.018424594774842262,
+            0.010853812098503113,
+            0.0070861103013157845,
+            -0.0005766529939137399,
+            0.01019579079002142,
+            -0.0073372903279960155,
+            0.008688710629940033,
+            0.038717120885849,
+            -0.018410442396998405,
+            0.005303084850311279,
+            0.0062370505183935165,
+            0.023320838809013367,
+            -0.011228813789784908,
+            -0.00031485961517319083,
+            0.00800238735973835,
+            0.021608568727970123,
+            -0.012934008613228798,
+            0.005883275531232357,
+            0.067075714468956,
+            -0.012912781909108162,
+            0.031613320112228394,
+            0.020533092319965363,
+            -0.004970536567270756,
+            -0.0104929618537426,
+            -0.012445799075067043,
+            -0.0181132722645998,
+            0.021283095702528954,
+            0.01175947580486536,
+            0.003419233951717615,
+            0.01941516436636448,
+            -0.01610383205115795,
+            0.006636816076934338,
+            0.017547233030200005,
+            0.012601460330188274,
+            -0.007379743270576,
+            -0.0036173479165881872,
+            0.019004784524440765,
+            0.018509499728679657,
+            -0.0030849166214466095,
+            0.00019424450874794275,
+            0.01508495956659317,
+            -0.022075550630688667,
+            -0.012530704960227013,
+            0.01644345559179783,
+            -0.0037358624394983053,
+            0.017773648723959923,
+            -0.02227366529405117,
+            0.024198200553655624,
+            -0.004531855694949627,
+            -0.017179306596517563,
+            0.004698129836469889,
+            -0.007514177821576595,
+            0.022033097222447395,
+            -0.001578720286488533,
+            0.004857328720390797,
+            0.000821199850179255,
+            -0.0018015984678640962,
+            0.01349297259002924,
+            0.002826661104336381,
+            -0.0005120891146361828,
+            0.010507113300263882,
+            -0.0009702275274321437,
+            -0.0013496510218828917,
+            0.024509521201252937,
+            1.9374700059415773e-05,
+            0.005989408120512962,
+            0.0032936439383774996,
+            -0.009261825121939182,
+            -0.004429261200129986,
+            -0.004443412180989981,
+            0.0011842612875625491,
+            0.003877372248098254,
+            -0.007029506377875805,
+            -0.02384442463517189,
+            0.022471778094768524,
+            -0.008540124632418156,
+            0.01175947580486536,
+            0.011349096894264221,
+            0.003665107535198331,
+            0.006824316922575235,
+            -0.009771261364221573,
+            -0.017320817336440086,
+            0.0006633278680965304,
+            -0.01617458648979664,
+            0.006912760436534882,
+            -0.001656550681218505,
+            0.0020235921256244183,
+            0.009445788338780403,
+            -0.002570174168795347,
+            0.0015123875346034765,
+            0.011306644417345524,
+            0.011497682891786098,
+            0.003254728624597192,
+            0.004577846731990576,
+            -0.018721764907240868,
+            -0.020915169268846512,
+            0.005288933869451284,
+            0.013924578204751015,
+            0.004273600410670042,
+            0.0015309606678783894,
+            0.017603836953639984,
+            -0.006166295614093542,
+            -0.022967062890529633,
+            -0.010117961093783379,
+            -0.01584911346435547,
+            0.039396367967128754,
+            0.01800006441771984,
+            -0.02525952458381653,
+            -0.000530220044311136,
+            -0.009728807955980301,
+            0.010450509376823902,
+            -0.00404364662244916,
+            0.014575524255633354,
+            0.005476434715092182,
+            -0.02426895499229431,
+            -0.016797229647636414,
+            -0.010662773624062538,
+            -0.005257094278931618,
+            0.020080260932445526,
+            0.027014248073101044,
+            -0.004956385586410761,
+            0.0061026159673929214,
+            0.019584976136684418,
+            -0.010938717983663082,
+            0.009849091991782188,
+            0.020957622677087784,
+            0.0031698227394372225,
+            -0.007570781745016575,
+            -0.004553082399070263,
+            -0.016514210030436516,
+            0.024467067793011665,
+            0.014250051230192184,
+            -0.005104971118271351,
+            -0.02465103194117546,
+            0.004110863897949457,
+            -0.02428310550749302,
+            -0.01616043597459793,
+            -0.007301913108676672,
+            -0.014143918640911579,
+            -0.004931621253490448,
+            0.0027594438288360834,
+            -0.004836102016270161,
+            0.011278342455625534,
+            -0.0031592093873769045,
+            -0.004531855694949627,
+            -0.0041851564310491085,
+            -0.025584997609257698,
+            -0.02524537406861782,
+            0.007691065315157175,
+            -0.003005317412316799,
+            0.005596718285232782,
+            0.000650945701636374,
+            -0.013400991447269917,
+            -0.011589664034545422,
+            0.00970758218318224,
+            -0.007988235913217068,
+            0.012813725508749485,
+            0.0031149876303970814,
+            -0.009183995425701141,
+            0.0017193458043038845,
+            -0.0012311364989727736,
+            -0.0059752571396529675,
+            0.019924599677324295,
+            0.00916984397917986,
+            0.011724098585546017,
+            -0.03118879161775112,
+            -0.0021633331198245287,
+            0.008349086157977581,
+            0.01828308403491974,
+            -0.02244347706437111,
+            -0.008313708938658237,
+            0.005179263651371002,
+            -0.011228813789784908,
+            -0.013507124036550522,
+            -0.028952933847904205,
+            -0.0030460015404969454,
+            8.175515540642664e-05,
+            -0.018636859953403473,
+            -0.029886899515986443,
+            0.0009038947173394263,
+            -0.007860877551138401,
+            0.016882136464118958,
+            -0.006643891800194979,
+            0.01790100708603859,
+            0.00870286114513874,
+            0.0036173479165881872,
+            0.011554286815226078,
+            0.0025082635693252087,
+            -0.013266556896269321,
+            0.008426916785538197,
+            -0.02671707607805729,
+            -0.01516986545175314,
+            0.009792488068342209,
+            0.014214673079550266,
+            -0.0026126273442059755,
+            5.074458022136241e-05,
+            0.0058372849598526955,
+            0.024254804477095604,
+            0.013584954664111137,
+            0.012474101036787033,
+            0.003042463678866625,
+            -0.005303084850311279,
+            0.005568416323512793,
+            0.0227123461663723,
+            0.008603804744780064,
+            0.004011806566268206,
+            0.007592008449137211,
+            0.00389859895221889,
+            0.0009932229295372963,
+            -0.005943417549133301,
+            -0.00843399204313755,
+            -0.021537812426686287,
+            -0.008908051066100597,
+            0.019938750192523003,
+            -0.010422207415103912,
+            -0.03005671128630638,
+            0.0006023016758263111,
+            -0.0032777239102870226,
+            0.007988235913217068,
+            0.02595292218029499,
+            -0.014172220602631569,
+            0.004970536567270756,
+            0.005536576267331839,
+            -0.007910405285656452,
+            -0.00614860700443387,
+            0.022967062890529633,
+            0.02639160305261612,
+            -0.0034404604230076075,
+            -0.04800017178058624,
+            -0.01671232469379902,
+            0.004305440001189709,
+            -0.007768895477056503,
+            -0.034047290682792664,
+            -0.015424583107233047,
+            0.002757675014436245,
+            -0.03206615149974823,
+            0.010068432427942753,
+            -0.012948159128427505,
+            -0.01214155275374651,
+            0.020264223217964172,
+            0.03223596513271332,
+            0.005154499784111977,
+            0.00014847487909719348,
+            -0.0214670579880476,
+            -0.0021686397958546877,
+            -0.021382151171565056,
+            0.010174565017223358,
+            -0.008801918476819992,
+            -0.010584943927824497,
+            0.0059611061587929726,
+            -0.012191081419587135,
+            0.013266556896269321,
+            0.010797208175063133,
+            0.024778390303254128,
+            0.01584911346435547,
+            -0.01876421831548214,
+            -0.0024746551644057035,
+            -0.01108730398118496,
+            -0.0016450530383735895,
+            -0.009820790030062199,
+            0.006360871717333794,
+            -0.0031592093873769045,
+            0.011264191009104252,
+            0.015184016898274422,
+            0.004432798828929663,
+            0.00046786724124103785,
+            0.038151077926158905,
+            -2.4487851987942122e-05,
+            -0.0003418349369894713,
+            0.006803090218454599,
+            -0.015750056132674217,
+            0.008837295696139336,
+            0.012191081419587135,
+            0.005051904823631048,
+            -0.007698140572756529,
+            0.006739411037415266,
+            0.03099067695438862,
+            -0.01516986545175314,
+            -0.009014182724058628,
+            -0.023504801094532013,
+            0.007549555506557226,
+            -0.0004656561359297484,
+            0.029971804469823837,
+            0.020533092319965363,
+            -0.027509532868862152,
+            -0.016018925234675407,
+            -0.02179253101348877,
+            0.01267929095774889,
+            -0.011497682891786098,
+            -0.003704022616147995,
+            0.012240610085427761,
+            0.02367461286485195,
+            0.0018838511314243078,
+            -0.014455240219831467,
+            -0.008950503543019295,
+            0.008943428285419941,
+            0.014617976732552052,
+            0.003497064346447587,
+            0.007570781745016575,
+            0.00743634719401598,
+            0.008886824361979961,
+            -0.014801939949393272,
+            -0.012254760600626469,
+            0.00767691433429718,
+            -0.010202866978943348,
+            0.01794346049427986,
+            -0.009403335861861706,
+            0.016768928617239,
+            -0.004645063541829586,
+            -0.011964665725827217,
+            -0.0320095494389534,
+            -0.002292460994794965,
+            0.005037753842771053,
+            0.004627374932169914,
+            0.0034705311991274357,
+            -0.0022217060904949903,
+            -0.013160424306988716,
+            0.004673365969210863,
+            0.005073131527751684,
+            1.875283123808913e-05,
+            -0.008313708938658237,
+            -0.00479718716815114,
+            0.004252373706549406,
+            -0.001031253719702363,
+            -0.0073372903279960155,
+            -0.0059858704917132854,
+            0.018481198698282242,
+            0.0066509670577943325,
+            0.00800238735973835,
+            0.03234917297959328,
+            0.02974538877606392,
+            -0.0016353242099285126,
+            -0.004606148693710566,
+            -0.00023304918431676924,
+            -0.01589156687259674,
+            -0.0057205394841730595,
+            -0.023320838809013367,
+            0.022259512916207314,
+            -0.009665128774940968,
+            0.021820832043886185,
+            0.019457615911960602,
+            0.0039304387755692005,
+            -0.008391539566218853,
+            -0.0011586126638576388,
+            -0.007308988366276026,
+            -0.003560743760317564,
+            0.016641568392515182,
+            0.006742948666214943,
+            0.01226891204714775,
+            0.00030933189555071294,
+            -0.028401045128703117,
+            0.026405755430459976,
+            0.000579748535528779,
+            0.0034015451092272997,
+            0.008908051066100597,
+            0.014129767194390297,
+            0.013209952972829342,
+            -0.01235381793230772,
+            0.01768874190747738,
+            -0.03280200436711311,
+            -0.0033484790474176407,
+            -0.016500059515237808,
+            0.002076658420264721,
+            -0.0018520113080739975,
+            -0.0008145665633492172,
+            -0.05354736000299454,
+            0.0003842879377771169,
+            0.005313698202371597,
+            -0.015339677222073078,
+            0.029349161311984062,
+            -0.020589696243405342,
+            -0.017165156081318855,
+            -0.032858606427907944,
+            -0.010238244198262691,
+            0.018099121749401093,
+            0.005617944523692131,
+            0.00035178486723452806,
+            -0.02660386823117733,
+            0.015750056132674217,
+            -0.0047476585023105145,
+            -0.00743634719401598,
+            -0.029858596622943878,
+            -0.010938717983663082,
+            0.009403335861861706,
+            0.004485865123569965,
+            -0.0130825936794281,
+            0.009898620657622814,
+            -0.002890340518206358,
+            0.010867963545024395,
+            -0.0138467475771904,
+            -0.00420284504070878,
+            0.0010020672343671322,
+            -0.005363226868212223,
+            0.012615610845386982,
+            -0.030537845566868782,
+            0.011391550302505493,
+            -0.01352127455174923,
+            0.0008548084879294038,
+            -0.017858553677797318,
+            -0.027707645669579506,
+            0.0020288988016545773,
+            -0.016132133081555367,
+            0.039990708231925964,
+            0.010188715532422066,
+            -0.01481609046459198,
+            0.016995344310998917,
+            -0.002080196049064398,
+            0.017377421259880066,
+            0.0014955832157284021,
+            0.017023645341396332,
+            -0.011971740983426571,
+            0.021056678146123886,
+            -0.0041745430789887905,
+            0.02156611531972885,
+            0.00600002147257328,
+            0.034047290682792664,
+            -0.02033497765660286,
+            0.0070825726725161076,
+            0.003792466362938285,
+            0.005798369646072388,
+            -0.0026816134341061115,
+            0.021424604579806328,
+            0.017985913902521133,
+            0.019698183983564377,
+            0.0037110981065779924,
+            -0.024084990844130516,
+            -0.006137993652373552,
+            0.004867942072451115,
+            0.023858575150370598,
+            0.011150983162224293,
+            -0.016415152698755264,
+            -0.021056678146123886,
+            0.015226469375193119,
+            -0.007860877551138401,
+            0.006771250627934933,
+            -0.00851889792829752,
+            0.00037544354563578963,
+            -0.013259481638669968,
+            0.012233533896505833,
+            -0.012523629702627659,
+            0.0032335021533071995,
+            0.01043635793030262,
+            -0.027806703001260757,
+            0.01511326152831316,
+            -0.00371817359700799,
+            -0.014313730411231518,
+            0.023320838809013367,
+            -0.015877414494752884,
+            -0.013988257385790348,
+            0.005501199048012495,
+            0.008610880002379417,
+            0.01746232621371746,
+            -0.002761212643235922,
+            0.002669231267645955,
+            -0.020504789426922798,
+            -0.012445799075067043,
+            0.004591997712850571,
+            0.02210385352373123,
+            -0.015820810571312904,
+            0.012431648559868336,
+            -0.018099121749401093,
+            0.01364863384515047,
+            0.016202887520194054,
+            0.007149789482355118,
+            0.00573115237057209,
+            -0.007726442534476519,
+            -0.0011471149045974016,
+            0.032151058316230774,
+            -0.0029876285698264837,
+            0.02704254910349846,
+            -0.014915147796273232,
+            0.009615600109100342,
+            0.00518987700343132,
+            -0.013089669868350029,
+            -0.014589674770832062,
+            0.011108530685305595,
+            0.02428310550749302,
+            -0.009530694223940372,
+            -0.023660462349653244,
+            0.0018484735628589988,
+            0.014264201745390892,
+            0.010337301529943943,
+            0.009403335861861706,
+            -0.001997058978304267,
+            -0.0037358624394983053,
+            0.01484439242631197,
+            -0.0032918748911470175,
+            -0.0286557637155056,
+            -0.01633024774491787,
+            -0.004400959238409996,
+            -0.017490629106760025,
+            0.020745357498526573,
+            0.02367461286485195,
+            0.010507113300263882,
+            0.0037570889107882977,
+            0.005540114361792803,
+            0.018297234550118446,
+            0.01843874529004097,
+            0.013330236077308655,
+            0.009304278530180454,
+            0.012700516730546951,
+            -0.004319590982049704,
+            0.02800481766462326,
+            -0.010860888287425041,
+            -0.01046465989202261,
+            -0.015707602724432945,
+            0.0016636261716485023,
+            -0.005193414632230997,
+            -0.012431648559868336,
+            -0.0069481381215155125,
+            0.004369119182229042,
+            0.013330236077308655,
+            -0.012240610085427761,
+            -0.007149789482355118,
+            0.01016748882830143,
+            0.011971740983426571,
+            0.01403071079403162,
+            0.01719345711171627,
+            -0.01633024774491787,
+            -0.011278342455625534,
+            0.0009834941010922194,
+            -0.02844349853694439,
+            -0.0160047747194767,
+            -0.01767459139227867,
+            0.009580222889780998,
+            -0.014469391666352749,
+            0.03724541515111923,
+            0.028471799567341805,
+            0.0073514413088560104,
+            0.031613320112228394,
+            -0.028401045128703117,
+            0.00970758218318224,
+            -0.014398636296391487,
+            0.007132100872695446,
+            -0.004188694059848785,
+            0.003902136581018567,
+            0.002448122017085552,
+            0.020419884473085403,
+            0.00791748147457838,
+            0.014264201745390892,
+            0.008150972425937653,
+            -0.021934039890766144,
+            -0.00662620272487402,
+            -0.012559006921947002,
+            -0.016401002183556557,
+            -0.02595292218029499,
+            0.013719388283789158,
+            0.005087282508611679,
+            -0.003074303502216935,
+            -0.0002704166399780661,
+            0.006566061172634363,
+            -0.0013195802457630634,
+            0.010443433187901974,
+            -0.027608588337898254,
+            -0.01311089564114809,
+            0.0069269114173948765,
+            0.01449061743915081,
+            -0.02114158496260643,
+            0.0014637433923780918,
+            0.008900974877178669,
+            -0.004602610599249601,
+            0.035660505294799805,
+            -0.014674580655992031,
+            -0.008610880002379417,
+            0.0027806703001260757,
+            0.005062518175691366,
+            -0.017660440877079964,
+            -0.02319347858428955,
+            0.0070648835971951485,
+            0.011575513519346714,
+            0.010867963545024395,
+            -0.028570856899023056,
+            0.02087271586060524,
+            -0.0054339817725121975,
+            0.03832089155912399,
+            0.00401534466072917,
+            -0.013337312266230583,
+            0.003962278366088867,
+            0.0011860302183777094,
+            -0.012360893189907074,
+            0.002060738392174244,
+            -0.000340508297085762,
+            0.003700484987348318,
+            0.011228813789784908,
+            -0.014801939949393272,
+            -0.016967041417956352,
+            0.016910437494516373,
+            -0.011476456187665462,
+            -0.0035395172890275717,
+            -0.00943871308118105,
+            -0.016669871285557747,
+            0.007308988366276026,
+            0.025047259405255318,
+            0.0041745430789887905,
+            0.001242634141817689,
+            -0.011632117442786694,
+            -0.0038596836384385824,
+            -0.008575502783060074,
+            0.0011718792375177145,
+            -0.0207029040902853,
+            0.027396325021982193,
+            -0.01801421493291855,
+            0.002410975517705083,
+            0.021099131554365158,
+            0.008709936402738094,
+            -0.003944589756429195,
+            0.016316095367074013,
+            0.026363302022218704,
+            0.01205664686858654,
+            0.0031539027113467455,
+            -0.00902833417057991,
+            1.648646139074117e-05,
+            0.014957600273191929,
+            0.0015123875346034765,
+            -0.0015663381200283766,
+            0.0032140444964170456,
+            -0.006583749782294035,
+            -0.001859086798503995,
+            -0.017561383545398712,
+            0.005564878229051828,
+            0.02125479280948639,
+            -0.004132090136408806,
+            0.020476488396525383,
+            0.015509488992393017,
+            0.011625041253864765,
+            0.010726453736424446,
+            -0.0012390963966026902,
+            -0.002150951186195016,
+            -0.0021668709814548492,
+            -0.007393894251435995,
+            -0.018297234550118446,
+            0.006470541935414076,
+            0.0034493047278374434,
+            0.01822648011147976,
+            0.020504789426922798,
+            -0.02308027073740959,
+            0.010146263055503368,
+            0.014985902234911919,
+            -0.013011839240789413,
+            0.0235755555331707,
+            -0.02006610855460167,
+            0.0024605041835457087,
+            0.02156611531972885,
+            0.013337312266230583,
+            -0.014271277002990246,
+            -0.009219372645020485,
+            0.004662752617150545,
+            -0.01543873455375433,
+            0.000872939417604357,
+            -0.01478778850287199,
+            -0.010783057659864426,
+            0.014533070847392082,
+            -0.01490099634975195,
+            0.01229721400886774,
+            -0.006969364359974861,
+            0.0070967236533761024,
+            -0.017165156081318855,
+            0.015014204196631908,
+            0.008632106706500053,
+            0.01757553406059742,
+            0.0021279556676745415,
+            -0.02518877014517784,
+            -0.006290116813033819,
+            0.021169887855648994,
+            0.009509467519819736,
+            -0.02497650496661663,
+            0.019613277167081833,
+            0.0020094411447644234,
+            -0.019528372213244438,
+            -0.022401023656129837,
+            -0.012686366215348244,
+            -0.005055442452430725,
+            -0.017264213413000107,
+            -0.0011276573641225696,
+            -0.024792540818452835,
+            0.003972891718149185,
+            0.009106164798140526,
+            -0.011978816241025925,
+            -0.019273653626441956,
+            -0.0002315014135092497,
+            -0.01040805596858263,
+            0.006219361908733845,
+            -0.01132787112146616,
+            -0.019853845238685608,
+            0.0070436573587358,
+            0.012127402238547802,
+            -0.0027647505048662424,
+            -0.005646246485412121,
+            -0.02562745101749897,
+            -0.006919836159795523,
+            0.020179318264126778,
+            -0.003994117956608534,
+            -0.004974074196070433,
+            -0.01703779585659504,
+            0.01108730398118496,
+            0.011165134608745575,
+            0.020901018753647804,
+            0.010931642726063728,
+            0.005674548447132111,
+            0.013634482398629189,
+            -0.0029416377656161785,
+            -0.007156865205615759,
+            -9.645892532716971e-06,
+            0.004284213297069073,
+            0.0069233737885952,
+            0.0016636261716485023,
+            0.020235922187566757,
+            -0.01692458800971508,
+            0.018934030085802078,
+            -0.012290137819945812,
+            0.008794843219220638,
+            -0.01855195313692093,
+            -0.0003155229496769607,
+            -0.01067692507058382,
+            0.00015002264990471303,
+            -0.0038455326575785875,
+            -0.009544845670461655,
+            1.9595809135353193e-05,
+            0.00143986358307302,
+            0.008511822670698166,
+            -0.016783079132437706,
+            0.009615600109100342,
+            -0.010839661583304405,
+            -0.004308977629989386,
+            -0.019542522728443146,
+            0.017448175698518753,
+            0.013344387523829937,
+            0.007308988366276026,
+            -0.008278331719338894,
+            -0.006300730165094137,
+            -0.02282555401325226,
+            0.008582578040659428,
+            -0.0014637433923780918,
+            0.002214630600064993,
+            -0.028301987797021866,
+            -0.004358506295830011,
+            0.0024587351363152266,
+            0.020306676626205444,
+            0.0033661676570773125,
+            -0.010712302289903164,
+            0.008978805504739285,
+            -0.023377442732453346,
+            -0.0030159305315464735,
+            -0.023349139839410782,
+            -0.004075486212968826,
+            -0.005112046375870705,
+            0.010103809647262096,
+            0.008101443760097027,
+            0.01549533847719431,
+            -0.009898620657622814,
+            0.01478778850287199,
+            -0.004793649073690176,
+            0.012049571610987186,
+            -0.0012152165872976184,
+            0.01337976474314928,
+            -0.0003111007681582123,
+            -0.01505665760487318,
+            0.02158026583492756,
+            -0.008738238364458084,
+            0.001741456682793796,
+            -0.0052853962406516075,
+            0.0027346794959157705,
+            -0.017702894285321236,
+            -0.004553082399070263,
+            -0.014455240219831467,
+            0.01957082375884056,
+            -0.022584985941648483,
+            0.004153316840529442,
+            -0.005055442452430725,
+            -0.016075529158115387,
+            0.029434068128466606,
+            0.022896308451890945,
+            -0.0035483618266880512,
+            0.015070809051394463,
+            0.0017989451298490167,
+            0.0015061964513733983,
+            0.016202887520194054,
+            -0.015806660056114197,
+            -0.013025989755988121,
+            0.035066161304712296,
+            0.0176321379840374,
+            0.0004313841927796602,
+            -0.0023260696325451136,
+            -0.0037323245778679848,
+            0.0032193511724472046,
+            0.008023613132536411,
+            -0.010450509376823902,
+            -0.01141277700662613,
+            -0.017051948234438896,
+            0.009997677057981491,
+            -0.006155682262033224,
+            0.004446949809789658,
+            -0.00025891896802932024,
+            -7.617214578203857e-05,
+            -0.01516986545175314,
+            0.022584985941648483,
+            -0.017717044800519943,
+            -0.00602832343429327,
+            0.004687516950070858,
+            0.0207029040902853,
+            -0.02541518583893776,
+            0.01616043597459793,
+            -0.007107336539775133,
+            -0.013429293408989906,
+            0.012573158368468285,
+            -0.015127412974834442,
+            -0.0107335289940238,
+            -0.003175129182636738,
+            -0.008009462617337704,
+            -0.022754797711968422,
+            0.017929309979081154,
+            -0.013627407141029835,
+            0.015650998800992966,
+            -0.00287972716614604,
+            0.000562502013053745,
+            0.002057200763374567,
+            0.0032582662533968687,
+            0.02250008098781109,
+            0.030735958367586136,
+            0.010415131226181984,
+            -0.005465821363031864,
+            -0.021000074222683907,
+            0.012983537279069424,
+            0.0020589695777744055,
+            0.013202877715229988,
+            -0.002253545681014657,
+            -0.002135031158104539,
+            -0.016613267362117767,
+            -0.023122724145650864,
+            -0.0013010069960728288,
+            -0.01761798746883869,
+            -0.02552839368581772,
+            -0.01774534583091736,
+            -0.007047194987535477,
+            0.011349096894264221,
+            0.020405733957886696,
+            0.0009755341452546418,
+            -0.005384453106671572,
+            0.01559439580887556,
+            -0.006941062398254871,
+            0.002080196049064398,
+            -0.006594363134354353,
+            -0.002032436430454254,
+            -0.008200501091778278,
+            -0.021863285452127457,
+            0.029660483822226524,
+            -0.010103809647262096,
+            0.01141277700662613,
+            -0.005472897086292505,
+            -0.00965097825974226,
+            0.005575491581112146,
+            0.00900003220885992,
+            0.024849144741892815,
+            0.006452853325754404,
+            0.03141520544886589,
+            0.020151015371084213,
+            -0.018891576677560806,
+            -0.0006071660900488496,
+            0.022684043273329735,
+            -0.0037570889107882977,
+            -0.004889168310910463,
+            -0.0031786670442670584,
+            -0.017985913902521133,
+            -0.012007118202745914,
+            0.007839650847017765,
+            -0.013471745885908604,
+            -0.006636816076934338,
+            -0.00837738811969757,
+            0.018070818856358528,
+            -0.017773648723959923,
+            0.025698205456137657,
+            -0.006488230545073748,
+            0.005154499784111977,
+            0.031924642622470856,
+            0.014363259077072144,
+            0.019698183983564377,
+            -0.011794853024184704,
+            -0.031075581908226013,
+            -0.00683846790343523,
+            -0.007882103323936462,
+            0.009558996185660362,
+            -0.01062032114714384,
+            0.005204027984291315,
+            -0.02006610855460167,
+            -0.018948180601000786,
+            0.012863253243267536,
+            -0.017476476728916168,
+            0.013924578204751015,
+            -0.009955224581062794,
+            0.027226513251662254,
+            -0.001228483160957694,
+            0.0121698547154665,
+            -0.002614396158605814,
+            -0.017801949754357338,
+            0.01205664686858654,
+            0.004089637193828821,
+            -0.023943481966853142,
+            0.010337301529943943,
+            -0.001864393474534154,
+            -0.014037786051630974,
+            -0.018410442396998405,
+            -0.001228483160957694,
+            -0.005200490355491638,
+            -0.009658053517341614,
+            0.025457637384533882,
+            0.014101465232670307,
+            0.016231190413236618,
+            0.0035359796602278948,
+            0.0016335553955286741,
+            -0.011073152534663677,
+            0.009573147632181644,
+            -0.006803090218454599,
+            0.000612030504271388,
+            -0.0034404604230076075,
+            0.011893910355865955,
+            -0.004252373706549406,
+            -0.01543873455375433,
+            0.002324300818145275,
+            0.008561351336538792,
+            0.0143278818577528,
+            5.947839599684812e-05,
+            0.009587298147380352,
+            0.0009790719486773014,
+            -0.009764186106622219,
+            -0.00046388726332224905,
+            -0.0060884649865329266,
+            0.0069375247694551945,
+            0.02400008589029312,
+            -0.012516554445028305,
+            -0.01681138016283512,
+            0.014504768885672092,
+            0.004033033270388842,
+            -0.017589686438441277,
+            0.018367990851402283,
+            0.004103788174688816,
+            -0.022358570247888565,
+            0.0035412863362580538,
+            0.010662773624062538,
+            -0.009523618966341019,
+            -0.014469391666352749,
+            -0.0010339069413021207,
+            -0.015509488992393017,
+            0.03130199760198593,
+            0.00025073791039176285,
+            0.016910437494516373,
+            -0.01062032114714384,
+            -0.0025967073161154985,
+            -0.013309009373188019,
+            0.013103820383548737,
+            -0.015750056132674217,
+            -0.007273610681295395,
+            0.00035620704875327647,
+            -0.02298121340572834,
+            0.01741987280547619,
+            0.0013425755314528942,
+            0.00794578343629837,
+            0.003436922561377287,
+            -0.010556641966104507,
+            0.009820790030062199,
+            0.0018237093463540077,
+            0.006668656133115292,
+            -0.014915147796273232,
+            0.023618008941411972,
+            0.006413938011974096,
+            0.011235889047384262,
+            -0.01352127455174923,
+            -0.01546703651547432,
+            0.002253545681014657,
+            0.04134920611977577,
+            -0.01306136790663004,
+            0.0002593612007331103,
+            0.01562269777059555,
+            7.081024523358792e-05,
+            0.004156854469329119,
+            -0.0023755980655550957,
+            -0.0034493047278374434,
+            -0.034585028886795044,
+            0.005370302125811577,
+            -0.0006328147719614208,
+            0.008073141798377037,
+            -0.008992956951260567,
+            -0.005430444143712521,
+            -0.0040684109553694725,
+            -0.008625030517578125,
+            -0.00314859626814723,
+            -0.01478778850287199,
+            0.008363237604498863,
+            -0.014533070847392082,
+            -0.028910480439662933,
+            0.03166992589831352,
+            -0.006887996103614569,
+            -0.00533846253529191,
+            -0.021396303549408913,
+            0.007570781745016575,
+            -0.01370523776859045,
+            0.002663924591615796,
+            0.010967019945383072,
+            0.008865597657859325,
+            0.008738238364458084,
+            0.002762981690466404,
+            0.0009516543359495699,
+            -0.004255911335349083,
+            0.010273621417582035,
+            -0.00835616234689951,
+            -0.0005943417199887335,
+            -0.0014186371117830276,
+            -0.02725481428205967,
+            -0.01405901275575161,
+            -0.010592019185423851,
+            -0.007811348885297775,
+            -0.00435143057256937,
+            0.012608535587787628,
+            0.007174553815275431,
+            -0.0017485321732237935,
+            -0.022938761860132217,
+            -0.014745336025953293,
+            -0.0034103896468877792,
+            0.008349086157977581,
+            -0.01838214136660099,
+            0.01064862310886383,
+            0.0016830838285386562,
+            0.009976450353860855,
+            0.014292503707110882,
+            0.00302831269800663,
+            0.0016742395237088203,
+            0.003458149265497923,
+            0.014886845834553242,
+            0.012601460330188274,
+            -0.005801907740533352,
+            -0.021382151171565056,
+            -0.0024605041835457087,
+            -0.007068421691656113,
+            -0.0034687623847275972,
+            -0.012332591228187084,
+            0.03039633482694626,
+            0.0029805530793964863,
+            0.0176321379840374,
+            0.007832575589418411,
+            0.009573147632181644,
+            0.010507113300263882,
+            0.029377464205026627,
+            0.006049550138413906,
+            -0.0029929352458566427,
+            -0.008603804744780064,
+            -0.02141045406460762,
+            -0.013146273791790009,
+            0.009898620657622814,
+            0.013294858857989311,
+            0.03597182780504227,
+            -0.034896351397037506,
+            0.012162779457867146,
+            -0.011101454496383667,
+            -0.008264180272817612,
+            -0.008533049374818802,
+            0.0044717141427099705,
+            0.010068432427942753,
+            0.014370334334671497,
+            -0.0023402206134051085,
+            -0.01175947580486536,
+            0.009070787578821182,
+            0.023547254502773285,
+            -0.012983537279069424,
+            0.03016991913318634,
+            0.027325568720698357,
+            0.023066120222210884,
+            0.01761798746883869,
+            -0.012537780217826366,
+            0.026462359353899956,
+            -6.484030018327758e-05,
+            0.0006261814851313829,
+            -0.018084971234202385,
+            -0.00587973790243268,
+            -0.004294826649129391,
+            -0.004362043924629688,
+            0.013316085562109947,
+            -0.0025471788831055164,
+            -0.0008468485320918262,
+            0.015099111013114452,
+            -0.0037323245778679848,
+            -3.927453508367762e-05,
+            -0.0059752571396529675,
+            0.015127412974834442,
+            0.0002699744363781065,
+            0.019457615911960602,
+            -0.04270770028233528,
+            0.019726485013961792,
+            -0.021396303549408913,
+            -0.003102605463936925,
+            -0.0020890405867248774,
+            0.009523618966341019,
+            0.00764861237257719,
+            -0.007142714224755764,
+            -0.014334957115352154,
+            0.004524780437350273,
+            0.014066088013350964,
+            -0.0006726144347339869,
+            0.005080206785351038,
+            -0.0027541371528059244,
+            0.0017228835495188832,
+            -0.010089659132063389,
+            -0.005862049292773008,
+            5.842812606715597e-05,
+            -0.010570792481303215,
+            -0.008646257221698761,
+            -0.03232087194919586,
+            0.01062032114714384,
+            0.010209942236542702,
+            0.007294837385416031,
+            0.004252373706549406,
+            -0.022018946707248688,
+            -0.012318440712988377,
+            0.012948159128427505,
+            -0.01132787112146616,
+            -0.002570174168795347,
+            -0.0020005968399345875,
+            0.005387990735471249,
+            0.0004705205501522869,
+            0.010160413570702076,
+            -0.003472300246357918,
+            0.010938717983663082,
+            -0.014603826217353344,
+            0.018084971234202385,
+            0.03229256719350815,
+            -0.00533846253529191,
+            -0.019967053085565567,
+            0.000907432462554425,
+            -0.009077862836420536,
+            0.003813692834228277,
+            0.007641536649316549,
+            -0.021962342783808708,
+            -0.02173592709004879,
+            0.005087282508611679,
+            -0.013153349049389362,
+            -0.007783046457916498,
+            0.014830241911113262,
+            0.005221716593950987,
+            0.010839661583304405,
+            0.01751893013715744,
+            0.010889190249145031,
+            0.00840569008141756,
+            -0.026533113792538643,
+            0.0006734988419339061,
+            -0.012686366215348244,
+            -0.00590803986415267,
+            0.002239394700154662,
+            -0.0017591454088687897,
+            0.004591997712850571,
+            -0.00010115749319083989,
+            -0.011200511828064919,
+            -0.0007057808106765151,
+            0.010903340764343739,
+            0.027919910848140717,
+            -0.004146241117268801,
+            -0.0022570835426449776,
+            -0.014257126487791538,
+            0.0010100272484123707,
+            0.036481261253356934,
+            -0.012275987304747105,
+            -0.017434025183320045,
+            0.01757553406059742,
+            0.027452928945422173,
+            0.005954030901193619,
+            -0.0032529598101973534,
+            -0.0027364485431462526,
+            0.00039600671152584255,
+            0.011978816241025925,
+            0.007719367276877165,
+            -0.002692226553335786,
+            -0.002131493529304862,
+            0.004761809483170509,
+            0.035773713141679764,
+            -0.0015203474322333932,
+            -0.010330225341022015,
+            -0.024240652099251747,
+            0.010974096134305,
+            0.00277182599529624,
+            0.004754733759909868,
+            0.002313687466084957,
+            0.023504801094532013,
+            0.019655730575323105,
+            0.014858543872833252,
+            0.00470520555973053,
+            -0.009771261364221573,
+            0.03005671128630638,
+            -0.008172199130058289,
+            -0.004542469047009945,
+            -0.02838689461350441,
+            -0.005087282508611679,
+            -0.015650998800992966,
+            0.008929276838898659,
+            -0.0002221042668679729,
+            -0.005016527138650417,
+            -0.013422218151390553,
+            0.019726485013961792,
+            0.01064862310886383,
+            -0.011073152534663677,
+            0.012608535587787628,
+            -0.027070851996541023,
+            0.02346234768629074,
+            -0.03826428949832916,
+            -0.0071993181481957436,
+            -0.006421013735234737,
+            -0.010514188557863235,
+            -0.02627839520573616,
+            -0.0030353881884366274,
+            0.007372668012976646,
+            0.014250051230192184,
+            0.0022570835426449776,
+            0.0022818478755652905,
+            0.02282555401325226,
+            0.0016415152931585908,
+            -0.01508495956659317,
+            0.017985913902521133,
+            0.03492465242743492,
+            0.012926933355629444,
+            0.00155837822239846,
+            0.015990622341632843,
+            0.0027789014857262373,
+            0.0021633331198245287,
+            0.00788917951285839,
+            -0.0034493047278374434,
+            -0.007938707247376442,
+            0.008582578040659428,
+            0.006990591064095497,
+            -0.02763689123094082,
+            0.003824306186288595,
+            -0.00010408719390397891,
+            -0.009898620657622814,
+            -0.022429324686527252,
+            0.011568437330424786,
+            0.01913214474916458,
+            -0.004224071744829416,
+            -0.006771250627934933,
+            -0.0043337419629096985,
+            0.0006836698739789426,
+            0.006785401608794928,
+            -0.002412744564935565,
+            -0.000515184598043561,
+            -0.0015176940942183137,
+            -0.02368876338005066,
+            -0.010316074825823307,
+            0.013570803217589855,
+            -0.005175726022571325,
+            0.010358527302742004,
+            0.005242943298071623,
+            -0.018792519345879555,
+            -0.016938740387558937,
+            0.006852618884295225,
+            -0.016188737004995346,
+            0.01364863384515047,
+            0.00027948213391937315,
+            -0.0023172253277152777,
+            0.024962352588772774,
+            0.03594352304935455,
+            -0.013188726268708706,
+            0.005119122099131346,
+            -0.019386861473321915,
+            -0.0245378240942955,
+            -0.019004784524440765,
+            0.022613288834691048,
+            0.008306633681058884,
+            -0.018792519345879555,
+            0.012304289266467094,
+            -0.002752368338406086,
+            -0.008865597657859325,
+            -0.0066721937619149685,
+            -0.006845543161034584,
+            -0.008547200821340084,
+            0.020533092319965363,
+            -0.004935159347951412,
+            0.0038950610905885696,
+            0.019047237932682037,
+            0.04075486212968826,
+            -0.008292482234537601,
+            -0.01349297259002924,
+            0.0054339817725121975,
+            0.004878554958850145,
+            0.0018626245437189937,
+            -0.004814875777810812,
+            -0.008016537874937057,
+            -0.0185944065451622,
+            -0.01740572229027748,
+            -0.009615600109100342,
+            -0.0020041344687342644,
+            0.020122714340686798,
+            -0.007768895477056503,
+            -0.01354957651346922,
+            -0.0245378240942955,
+            -0.028188779950141907,
+            -0.006580212153494358,
+            0.02319347858428955,
+            0.017179306596517563,
+            -0.00013200224202591926,
+            -4.3337418901501223e-05,
+            -0.012042495422065258,
+            -0.01968403346836567,
+            0.015509488992393017,
+            0.00037876018905080855,
+            -0.01773119531571865,
+            -0.0065448349341750145,
+            -0.015919867902994156,
+            0.005787756759673357,
+            -0.0051368107087910175,
+            0.004574308637529612,
+            0.05057565122842789,
+            0.0005271245026960969,
+            0.008441068232059479,
+            -0.026575567200779915,
+            0.009106164798140526,
+            -0.01583496294915676,
+            -0.017377421259880066,
+            -0.00012702727690339088,
+            -0.022301966324448586,
+            -0.0030955299735069275,
+            -0.018792519345879555,
+            -0.007386818993836641,
+            0.008568426594138145,
+            0.009799563325941563,
+            -0.013896276243031025,
+            -0.013669860549271107,
+            -0.024693485349416733,
+            -0.0035872769076377153,
+            -0.001539805089123547,
+            0.009311353787779808,
+            0.007825499400496483,
+            -0.03693409264087677,
+            0.002536565763875842,
+            -0.0009507699287496507,
+            0.0005779796629212797,
+            0.01208494883030653,
+            -0.008667483925819397,
+            -0.003063690150156617,
+            -0.0214670579880476,
+            0.006452853325754404,
+            0.007103798910975456,
+            -0.013500047847628593,
+            0.012375044636428356,
+            -0.012863253243267536,
+            -0.007669838611036539,
+            -0.0005426021525636315,
+            0.03379257395863533,
+            0.011794853024184704,
+            -0.0025153392925858498,
+            -0.00654129683971405,
+            -0.011150983162224293,
+            0.02654726430773735,
+            -0.011349096894264221,
+            -0.026405755430459976,
+            0.028910480439662933,
+            0.013245330192148685,
+            0.0005974372616037726,
+            0.020151015371084213,
+            -0.0017936384538188577,
+            -0.01397410687059164,
+            0.009601449593901634,
+            0.004266524687409401,
+            0.012424572370946407,
+            -0.00014460546663030982,
+            0.0214670579880476,
+            0.0018555490532889962,
+            0.0027399861719459295,
+            -0.0036226543597877026,
+            0.004687516950070858,
+            -0.03093407303094864,
+            -0.0006549257086589932,
+            -0.003102605463936925,
+            -0.008398614823818207,
+            0.011207587085664272,
+            -0.002421588869765401,
+            0.006558985915035009,
+            -0.022188758477568626,
+            -0.028485950082540512,
+            0.011342021636664867,
+            -0.016231190413236618,
+            -0.021396303549408913,
+            0.015339677222073078,
+            -0.0007641536649316549,
+            0.00614860700443387,
+            -0.007443422917276621,
+            -0.023278385400772095,
+            -0.019386861473321915,
+            0.02064630016684532,
+            0.012212308123707771,
+            -0.01190098561346531,
+            0.012750045396387577,
+            -0.021113283932209015,
+            -0.012948159128427505,
+            -0.001894464367069304,
+            0.0090566361322999,
+            -0.0019864458590745926,
+            0.01325240544974804,
+            -0.012198156677186489,
+            -0.00334494118578732,
+            0.0031592093873769045,
+            -0.011589664034545422,
+            0.008158047683537006,
+            0.01978308893740177,
+            0.008193425834178925,
+            -0.009339655749499798,
+            0.003422771580517292,
+            -0.011922212317585945,
+            0.013500047847628593,
+            0.015537790954113007,
+            0.018311386927962303,
+            -0.006498843897134066,
+            0.0020412809681147337,
+            -0.01370523776859045,
+            0.0063962494023144245,
+            -0.009827865287661552,
+            -0.004878554958850145,
+            -0.0005333155859261751,
+            0.004921008367091417,
+            -0.011179285123944283,
+            -0.0025772498920559883,
+            -0.026830283924937248,
+            -0.0013823752524331212,
+            -0.020533092319965363,
+            0.005303084850311279,
+            0.023207630962133408,
+            -0.007514177821576595,
+            0.0006045127520337701,
+            -0.0039092120714485645,
+            0.014476466923952103,
+            0.020688753575086594,
+            0.00870286114513874,
+            0.00482548912987113,
+            -0.006081389728933573,
+            0.02179253101348877,
+            0.011603815481066704,
+            -0.017816102132201195,
+            0.00164239970035851,
+            -0.001509734196588397,
+            -0.0014336726162582636,
+            0.011625041253864765,
+            0.004443412180989981,
+            -0.020844414830207825,
+            0.0028602697420865297,
+            -0.010089659132063389,
+            0.007818424142897129,
+            0.014702882617712021,
+            -0.00031906069489195943,
+            -0.02486329711973667,
+            -0.009827865287661552,
+            0.021919889375567436,
+            -0.015070809051394463,
+            -0.0008291598060168326,
+            0.00764861237257719,
+            0.023264234885573387,
+            -0.024792540818452835,
+            -0.006831392180174589,
+            -0.0023455272894352674,
+            -0.003933976404368877,
+            -0.005193414632230997,
+            0.0010489424457773566,
+            0.0214670579880476,
+            0.004415110219269991,
+            -0.004234685096889734,
+            0.005724077112972736,
+            0.006470541935414076,
+            0.02325008250772953,
+            0.0040577976033091545,
+            0.028047269210219383,
+            -0.00943871308118105,
+            -0.0023809047415852547,
+            -0.0019988277927041054,
+            -4.775960405822843e-05,
+            0.0160047747194767,
+            -0.005430444143712521,
+            0.020320827141404152,
+            0.009445788338780403,
+            0.0036934094969183207,
+            0.016514210030436516,
+            0.02319347858428955,
+            -0.006767712999135256,
+            -0.004793649073690176,
+            0.019160445779561996,
+            0.0053667644970119,
+            -0.008858522400259972,
+            -0.02194819226861,
+            0.01141277700662613,
+            0.0014124460285529494,
+            -0.0006956097786314785,
+            0.012106175534427166,
+            0.0024375086650252342,
+            0.006300730165094137,
+            0.008172199130058289,
+            -0.016684021800756454,
+            0.004241760354489088,
+            -0.004698129836469889,
+            -0.01860855706036091,
+            0.010910416021943092,
+            -0.012092024087905884,
+            -5.201595922699198e-05,
+            -0.0010002984199672937,
+            0.014165145345032215,
+            -0.01703779585659504,
+            -0.0019103841623291373,
+            -0.0070719593204557896,
+            -1.3024719009990804e-05,
+            -0.0009162768837995827,
+            0.008320784196257591,
+            -0.006254739128053188,
+            -0.010938717983663082,
+            -0.010938717983663082,
+            0.027509532868862152,
+            -0.004991763271391392,
+            0.003916287794709206,
+            0.019330257549881935
+          ],
+          "embedding": [
+            0.007204997251390758,
+            -0.03173324616100502,
+            -0.020505790884701024,
+            0.0037451608231038273,
+            0.009817085391002229,
+            0.026883177166072393,
+            -0.016477826654817607,
+            0.06949311853773875,
+            -0.01224876667472692,
+            0.031952566873802865,
+            0.013633671951171586,
+            -0.005041437355104989,
+            0.02663392750069342,
+            -0.022099132865267587,
+            0.006665702000169926,
+            0.0030467129715919033,
+            0.007931802508415542,
+            0.009984477355413895,
+            -0.010362836464183985,
+            -0.03180848181251013,
+            -0.013542377333474449,
+            -0.013572345022577053,
+            -0.03979985300694044,
+            -0.007698412130292187,
+            -0.005953452234356532,
+            0.018890228917932694,
+            0.02217446994660065,
+            0.005158357501784182,
+            -0.05468694715835175,
+            0.00789994048345594,
+            0.017130184174786147,
+            -0.0026591475669416114,
+            -0.02081141846859738,
+            -0.008403928318906045,
+            -0.030925346724945978,
+            -0.03855023691059801,
+            0.038853354785097736,
+            0.01612011653434478,
+            -0.035635702395196575,
+            0.026266623757525934,
+            0.043450373922427316,
+            -0.007254334835408145,
+            -0.015376804869420366,
+            -0.0017276202867766907,
+            0.004650414207094434,
+            -0.002365514361535186,
+            0.011906139905776183,
+            -0.0005664194302009289,
+            -0.0191723447249157,
+            -0.025546124518037998,
+            0.01025005257280377,
+            -0.014436532079436437,
+            0.029277529346694118,
+            -0.0045414983847733344,
+            -0.02260315456913697,
+            0.01717918656248626,
+            0.013666132349359846,
+            0.057804090189845364,
+            0.056600576093100735,
+            0.051208022172527734,
+            0.03553760896972431,
+            0.02094599510567085,
+            -0.02438674472832137,
+            -0.0027797634437333432,
+            0.03614248430646102,
+            -0.026729308357113124,
+            -0.002251394534452815,
+            0.018929251842788906,
+            0.00689697483318378,
+            -0.026603606434115974,
+            -0.0541443728344337,
+            0.027164630314707807,
+            -0.004335272389193875,
+            -0.013967321193992958,
+            0.003323121377857836,
+            -0.0245295956412367,
+            0.024489335129134787,
+            0.06212059558640156,
+            0.009214058207162371,
+            -0.002082293435905329,
+            0.013659934201796686,
+            -0.003344037927706573,
+            -0.00689041317919184,
+            0.020235785390418023,
+            0.0212739457082623,
+            0.0016782346585238381,
+            -0.013484221664059555,
+            -0.06465582506604173,
+            0.011882797526607326,
+            -0.032970655964890155,
+            -0.04027700439305653,
+            0.025396847912528105,
+            0.013861157517716707,
+            0.00604596157583048,
+            -0.00744611407420936,
+            0.04038084298991919,
+            0.021507532290328625,
+            0.005983822806623904,
+            -0.012711110369915742,
+            0.02410138142963901,
+            -0.018372791517880082,
+            0.021016674128298246,
+            0.013387285956415756,
+            0.02039383525127615,
+            0.010700292312227496,
+            0.0053934660497349735,
+            -0.014807678014449691,
+            0.009422805142740376,
+            0.0414357625490679,
+            0.12206398267002624,
+            -0.010877509834324155,
+            -0.011079302503786522,
+            -0.04107714853756596,
+            0.006631397986165343,
+            -0.031914267104222384,
+            -0.0179965295457466,
+            -0.057530036803978266,
+            -0.0320488164246567,
+            0.008208202363320509,
+            -0.0020203090915064162,
+            0.03502055780919601,
+            0.04464599164916666,
+            0.019332152079287082,
+            0.017133943087431935,
+            -0.039885565008919065,
+            0.04703059988373572,
+            -0.029370696964304416,
+            0.0008494086835971101,
+            0.0014082189719987074,
+            -0.030973762582071615,
+            -0.006561157852024518,
+            0.011634653995510726,
+            -0.019896826887411713,
+            -0.034275921680252924,
+            0.026274465530039866,
+            -0.007728234125236442,
+            0.04814514040690998,
+            0.06550407296280117,
+            -0.02749051703615343,
+            -0.010041099502187074,
+            0.02019601967763537,
+            -0.01871701432558228,
+            0.008532829759864928,
+            -0.020164138072967613,
+            -0.045196925901010855,
+            -0.004523517988516156,
+            -0.002113469149506399,
+            0.07929650663710812,
+            -0.01111269708042164,
+            0.03358254444720164,
+            -0.019973382759721527,
+            -0.0007145592863325417,
+            -0.0011740862928390678,
+            0.02166182480535788,
+            0.004543228023398081,
+            -0.013377688428761516,
+            -0.01758250073647576,
+            0.009548292014596834,
+            -0.04753617258004735,
+            -0.007125744019792479,
+            -0.009930358894299303,
+            -0.007931672044235996,
+            -0.024561936537429774,
+            0.012979195012881556,
+            0.009209145250061479,
+            0.04586756662401364,
+            0.04311807301838248,
+            -0.0036241105681042675,
+            -0.06579256720896123,
+            0.04967191755416882,
+            0.015475886231466817,
+            0.03157651656768371,
+            0.02515941109300714,
+            0.01773554572598468,
+            0.005824418191900522,
+            0.016317635547562007,
+            0.007650872352957098,
+            0.010015166465492963,
+            -0.033203307142867844,
+            0.040272188258443496,
+            -0.013408423788025047,
+            0.01036004294342104,
+            0.012391622448703885,
+            0.04214857862444935,
+            0.03418032388760535,
+            0.015263018079889338,
+            0.018786704580603317,
+            0.02704685343985817,
+            0.015348111464412869,
+            0.0013031600434754704,
+            0.03241427528588285,
+            0.011689924942195664,
+            0.04544406114742012,
+            0.016186615914968734,
+            0.011624500626103958,
+            -0.008272953078864385,
+            -0.02363080189189375,
+            0.02784298119317616,
+            -0.010585980875564109,
+            0.01455013604305903,
+            0.01749228557615982,
+            -0.054742056843690476,
+            -0.005876213780171779,
+            -0.010556605098730201,
+            0.003967456397644932,
+            0.014663888851767762,
+            0.0015548476559947807,
+            -0.003858367677774835,
+            0.012504286824484335,
+            -0.025806882498628832,
+            -0.014216299453397403,
+            0.0031620924516417956,
+            -0.006448390918604307,
+            -0.02785577502346934,
+            -0.00766402687921159,
+            0.005275000635711884,
+            -0.0090448751942,
+            0.019702662630533,
+            0.007009091837467867,
+            -0.01847307100043082,
+            -0.0006774105206839772,
+            0.032952656101827446,
+            0.015900033044760432,
+            -0.02170121261931376,
+            -0.017964879081469612,
+            -0.005062849745084668,
+            -0.03171847232054206,
+            -0.038177398521606544,
+            0.029426291950046455,
+            -0.025687201683643784,
+            0.030158867757101797,
+            0.015780411479509624,
+            -0.007011484049169482,
+            -0.012114023533118393,
+            0.020661407942701628,
+            -0.025067530380881376,
+            0.009777190971413269,
+            -0.004160771288889298,
+            -0.028825369485727344,
+            0.008307290529469729,
+            -0.014415903546305744,
+            -0.007561235629544945,
+            -0.03201704135271719,
+            -0.015182869388110948,
+            -0.00501561450194676,
+            -0.0003863746169672397,
+            -0.01073403445816663,
+            -0.03321103354401307,
+            -0.002304357889348034,
+            0.017001213960120047,
+            0.00028222146729658097,
+            -0.04002882852650754,
+            0.014509588052058002,
+            -0.004150541038999649,
+            -0.034333837380045466,
+            0.0564137596224324,
+            0.03584898691608747,
+            -0.006830698606679202,
+            0.006121456445207273,
+            -0.04505674213649008,
+            0.04358467937307492,
+            0.010847859073780811,
+            -0.04918983713702805,
+            0.049190442935189385,
+            -0.00528960443684769,
+            0.01419195371129309,
+            0.040316024601793406,
+            0.004600843996974165,
+            -0.006780970176765864,
+            0.005206651850992678,
+            0.03397517915308201,
+            -0.010037223444627569,
+            -0.027377913828275806,
+            -0.03328136706915221,
+            0.02562568440972934,
+            -0.03485770844248482,
+            -0.020069979230687374,
+            -0.0052041408068332524,
+            0.029818953377223505,
+            0.028789460147370508,
+            0.004071274130619537,
+            0.01928375754300958,
+            0.0014970947734202088,
+            0.010502599795027616,
+            0.016914145702515263,
+            0.011942576738188502,
+            -0.014909953528043828,
+            -0.003432489584767786,
+            -0.05733768990013957,
+            0.0030710780388681002,
+            -0.02627280695622545,
+            0.007930906480371135,
+            -0.031227917075485834,
+            0.013132797714872345,
+            0.002514351081823354,
+            -0.007152021475569949,
+            -0.0066245205132327055,
+            -0.005279224195782313,
+            0.028512607222275155,
+            -0.00921960825771865,
+            -0.0359704105589433,
+            -0.021347597388353182,
+            0.011200044645812517,
+            -0.0004171294784927543,
+            -0.016784468873218786,
+            -0.0018746544944553878,
+            0.007749103417240419,
+            0.02390272097796833,
+            -0.005128540108845805,
+            0.006397849195755672,
+            0.01214922062611769,
+            0.01163371266250103,
+            0.012307186722130844,
+            -0.004982589662882946,
+            -0.016343724553037307,
+            0.019806887143115643,
+            0.013139657463054087,
+            0.0205801609265216,
+            -0.03767676007629612,
+            -0.0249879011550972,
+            0.05547392586363707,
+            0.005903313549319051,
+            -0.05568806298774435,
+            0.018827693208840233,
+            -0.007931178941307645,
+            0.008183992600768387,
+            -0.010319022946041053,
+            0.05111352384595183,
+            -0.024897203290807753,
+            -0.02096678205188722,
+            0.006436523335212348,
+            -0.04917084523640266,
+            -0.010459284852442706,
+            -0.011138558192296276,
+            0.014789370043397917,
+            -0.03105725978752438,
+            -0.004934745004818259,
+            0.015595801375817122,
+            -4.687585727052446e-05,
+            0.0028886067775750254,
+            -0.00850352807499054,
+            -0.0034885579171087815,
+            0.016339681487281293,
+            -0.0034050317046826813,
+            -0.008309570448487812,
+            -0.003615165868022203,
+            -0.027638409919916757,
+            0.023352910161586186,
+            0.026364511529991978,
+            0.001249936082895606,
+            0.001987948967948376,
+            0.01756629246122769,
+            0.004656075485833687,
+            -0.006752579977941239,
+            -0.013681608638716132,
+            -0.03405936202918687,
+            -0.0401867651078737,
+            0.022991180183813725,
+            -0.02318908140203705,
+            0.018299512436825923,
+            0.00904081847012441,
+            0.019697091729447287,
+            -0.019236130984371835,
+            0.015159917756323988,
+            -0.0387297415849285,
+            -0.018793498297161176,
+            -0.005256288213225733,
+            -0.010414697711744128,
+            0.0007039894599589104,
+            -0.01739645696111503,
+            -0.013728845328097506,
+            0.022709522955937343,
+            -0.022789243441781803,
+            -0.0036763802589970026,
+            -0.01429671717544519,
+            0.0023674126354206932,
+            0.049448298344058644,
+            -0.009926033487593551,
+            -0.023371285193090733,
+            0.020846792916650956,
+            -0.026996519120163966,
+            0.011246908457705283,
+            0.04091365159640201,
+            0.005469162850197613,
+            0.005640919201612091,
+            0.029575753750229278,
+            -0.0007860452837141593,
+            -0.0052805237594885516,
+            -0.009782745443933828,
+            0.021178503581190618,
+            -0.0009176512672832309,
+            -0.05245650420332918,
+            -0.007698364493410058,
+            -0.039027418406703554,
+            0.005448840777995117,
+            0.017581625031764185,
+            0.025522517097715556,
+            -0.020775492472580457,
+            -0.015906320383873178,
+            -0.000643669839811819,
+            -0.0086896084429604,
+            0.0029399233482444638,
+            0.014381240985647907,
+            0.021086693776922936,
+            0.002436575081260609,
+            -0.004913514868101904,
+            -0.011582101006213432,
+            0.014400297663309155,
+            0.0467181039341492,
+            -0.01722692180137712,
+            0.0011622224560709315,
+            -0.013036540368441797,
+            -0.06208454638917413,
+            -0.019160212331432766,
+            0.0351496273293468,
+            0.02429814055116254,
+            -0.009896662169762772,
+            -0.01647101145477158,
+            -0.002915821099802562,
+            -0.03557863734177486,
+            -0.01786433999615293,
+            -0.010699364747311715,
+            0.02390901423299708,
+            0.007213809581960646,
+            0.0005601364043388203,
+            0.00916614601011734,
+            0.014737491634516238,
+            0.019303599465763203,
+            0.008864535607492504,
+            -0.0575839606069681,
+            0.018051837786406608,
+            0.006315830080225406,
+            0.02525619013082567,
+            -0.015671924543113065,
+            0.003988943108210209,
+            -0.03179878281917948,
+            -0.029440641244429775,
+            0.040586322748158446,
+            -0.03418722657593859,
+            -0.030650542761032924,
+            0.006566745611226466,
+            0.009571089937153852,
+            0.004491158558315032,
+            -0.011352813042040581,
+            -0.02535505987721779,
+            -0.013381948772812133,
+            -0.030918676056735445,
+            -0.02192708443081135,
+            -0.005232667170255381,
+            -0.03342408726436631,
+            -0.07498634115550039,
+            -0.006245638452568239,
+            -0.049946896465756174,
+            -0.03900385398225758,
+            0.04458524125677807,
+            -0.006106489377959481,
+            0.02398410968928338,
+            -0.04615183642824994,
+            -0.01165382599292015,
+            -0.04890013373371541,
+            0.013624324077017377,
+            0.01476398305411351,
+            0.020202554298470666,
+            0.00014553149242532853,
+            0.036224316512987846,
+            0.00021498496267774312,
+            -0.05055208658013767,
+            -0.057482169168904106,
+            -0.003960451575823462,
+            -0.010217078509789989,
+            -0.005593738972150311,
+            -0.034090261748104665,
+            0.05835341009022349,
+            0.0116359661127526,
+            0.04203678184748103,
+            0.03447818335186531,
+            -0.006747564131379555,
+            -0.025756962754841885,
+            -0.028123416424745064,
+            -0.008073625379614702,
+            0.047544241458556825,
+            -0.029605671083240564,
+            0.008171081332993893,
+            -0.026007452205474395,
+            0.0067084589781962805,
+            -0.0070645448617940995,
+            -0.0012298101311792947,
+            0.012372286141738867,
+            0.006377319686763927,
+            -0.015024537877377194,
+            0.007934745664790507,
+            0.026041755081605937,
+            -0.007303731324541215,
+            0.0407000170257006,
+            -0.012990703097937777,
+            -0.033189084013863494,
+            -0.022525679964848666,
+            0.01657717698191603,
+            0.03440284885162039,
+            0.010230633738702899,
+            -0.008707808754455952,
+            -0.0363777020540216,
+            0.011184076269109602,
+            0.031774012431843524,
+            0.01504703222643543,
+            -0.022509833303978505,
+            0.025230974050270107,
+            0.017502662694813216,
+            -0.018700579139537456,
+            -0.03511734747709348,
+            0.015162183119327901,
+            -0.017711638132862063,
+            0.018190335050681287,
+            -0.01590174153215615,
+            0.011103110869700727,
+            0.0025189370696524936,
+            -0.00920305384736575,
+            0.01929652151514404,
+            -0.006196715938358087,
+            0.011426611271994685,
+            0.007058186102245829,
+            -0.018191352599117545,
+            0.0027202185072220937,
+            0.057310566886380025,
+            -0.000531639834386418,
+            0.01494975681909879,
+            0.03845692752400366,
+            -0.001971953300066856,
+            0.045554909803324765,
+            -0.02532332999916258,
+            -0.042616337063845953,
+            0.02633963974172818,
+            -0.02970705577683783,
+            -0.01624154998005974,
+            -0.02620993449678864,
+            -0.021490817420853802,
+            -0.004038424045768319,
+            0.009940887982818171,
+            -0.02692933845395842,
+            -0.023801570856681796,
+            -0.001948460540662095,
+            0.02290911497874141,
+            -0.03300532378029689,
+            0.006525488380791647,
+            0.007039306647666766,
+            -0.003685585408710752,
+            0.03183735852538168,
+            -0.027197064244321705,
+            0.006861928286688311,
+            0.011209869849682412,
+            -0.001094897323854524,
+            -0.005328676992017907,
+            -0.004007474089818478,
+            0.010796946231051235,
+            -0.030543173497906445,
+            -0.012879972141706173,
+            0.00935894527375688,
+            0.0017573071057611829,
+            -0.010050882212831222,
+            -0.03102132152986303,
+            -0.002915141291985554,
+            0.007634137769426993,
+            -0.011094093612593696,
+            0.01199609810193786,
+            -0.010458016758814742,
+            -0.04929347607956551,
+            0.010973132159729983,
+            0.028148117094907874,
+            -0.025504019123994758,
+            0.012470127357818321,
+            0.009182729379865986,
+            -0.016020133756226275,
+            0.016320111225714855,
+            -0.012051271446461976,
+            0.024098195821827578,
+            0.04317620443481784,
+            0.024592886889234396,
+            0.007727787037954994,
+            0.00771746776156926,
+            -0.0003147538723468374,
+            0.022572868970938624,
+            -0.016683340264320543,
+            0.013672085891597506,
+            0.020827021901755415,
+            0.02092309219721837,
+            0.0057130785714458055,
+            -0.003901728981013356,
+            -0.0036968415923910314,
+            0.002227024825579113,
+            -0.02106678364798051,
+            0.014578546755457675,
+            0.03128101864424301,
+            -0.005234627629617124,
+            -0.022394744692442837,
+            0.002430617645746389,
+            -0.015780389432069438,
+            4.7277885648917e-05,
+            0.013357803020920043,
+            0.013777651543690907,
+            -0.027713499926132385,
+            -0.0018999112228756628,
+            0.028364630573666852,
+            0.009146793705085586,
+            -0.009891297866910313,
+            -0.01644269527910728,
+            -0.0003903768506971414,
+            0.006512217314869576,
+            0.02408785204739448,
+            0.0018406499545367782,
+            -0.007360473750031044,
+            0.01484497577088404,
+            0.016723277095327158,
+            0.003629911330658209,
+            -0.0004820567356447713,
+            0.02394406451074397,
+            0.023794019615473026,
+            0.001632266615911801,
+            -0.02064379152040184,
+            -0.02755693521684878,
+            0.027335938400284726,
+            0.02769248863677192,
+            -0.006832353240623308,
+            0.027324973360715134,
+            0.003937207909967385,
+            0.013580154030133622,
+            0.02076818768224654,
+            -0.01784151382870951,
+            -0.017617903734311997,
+            0.0017316476451235418,
+            -0.009605294810027804,
+            -0.009829196696575069,
+            0.004891780008779742,
+            -0.011655402383771472,
+            -0.01124379942906224,
+            0.043205454954628145,
+            0.004713701851783307,
+            0.003560551284131962,
+            0.007894840447501912,
+            -0.018881214526749082,
+            -0.00945538175144614,
+            0.0019040259741796817,
+            0.029505863720301916,
+            -0.001400002868254808,
+            0.03315866433970851,
+            -0.00830542045504805,
+            0.008602408964961391,
+            -0.03202191233287069,
+            -0.006978289755114341,
+            0.0326499698728755,
+            0.015314305362524091,
+            0.015104436300213667,
+            0.0021257042489266787,
+            -0.0010840650756005028,
+            0.02434348355847526,
+            -0.007174433778817674,
+            -0.04839373063710409,
+            0.022696994450018916,
+            -0.004779511451775743,
+            0.023588216956749154,
+            0.0382733980620746,
+            -0.004232286553944853,
+            0.003815507826545929,
+            0.023531223314846965,
+            -0.01887650634213847,
+            0.015055403379782985,
+            -0.01918944132635701,
+            -0.017597900721414636,
+            0.02176630010543207,
+            -0.008896608357798277,
+            0.01732387427588561,
+            0.01764363756727683,
+            -0.012180043444768312,
+            -0.019003360499321047,
+            0.01229139850966224,
+            0.0025710091163655626,
+            -0.017075658234265562,
+            0.01797412486853335,
+            0.018372010308966135,
+            0.006723179570342314,
+            0.015417677746651683,
+            -0.00047634684109332134,
+            0.017608296628666153,
+            -0.001049984114818522,
+            -0.008416990841821452,
+            0.01827827907328822,
+            -0.009568717277044265,
+            -0.01658089563620308,
+            0.009808478422454319,
+            -0.05193977747210182,
+            0.010644335931025957,
+            0.0016781339270411392,
+            -0.01562285100294917,
+            0.00405036330963539,
+            0.007102724536527262,
+            0.005195069570261247,
+            0.0028773630901681767,
+            0.004610474981005765,
+            0.042921486858787584,
+            0.013213672397112759,
+            -0.0064030584930089415,
+            -0.010711457508461704,
+            -0.017480309425763254,
+            -0.027151012983864797,
+            -0.005204425551873271,
+            0.009927884476991038,
+            0.022601230152836282,
+            0.008046297278661014,
+            -0.0031858480011267704,
+            -0.02589628103025927,
+            0.019723430185137954,
+            0.03604835658149313,
+            0.062423099201768716,
+            0.010930268386637612,
+            0.02925564269324381,
+            0.02439517070384333,
+            0.023930432096006114,
+            0.013036828091946605,
+            0.006910311087503408,
+            -0.032995841694293995,
+            0.012813650252889403,
+            0.01333714655191133,
+            0.03678966264802025,
+            0.012571138353467183,
+            -0.05610151838497368,
+            3.0777966587107344e-05,
+            0.009791774904454618,
+            0.041959488802722524,
+            0.01956296954085397,
+            0.000155217569370113,
+            -0.007285602454264675,
+            0.031547870838711856,
+            -0.026518385135879206,
+            -0.012299593280363843,
+            -0.0026551378470968594,
+            -0.006190117573235779,
+            -0.003807907001161504,
+            -0.004991957475076049,
+            0.036983035940964804,
+            -0.015648565314743963,
+            -0.019256463332683493,
+            0.0004300362961407231,
+            -0.0015686065121077472,
+            0.0011311524136286674,
+            0.0032236641835276464,
+            0.013795345278976164,
+            0.015385029265915279,
+            -0.004800448639353721,
+            0.026514926384796363,
+            -0.008699453553651681,
+            0.022186817580012775,
+            -0.013427553742493343,
+            -0.011927259975371873,
+            0.017776228476138103,
+            -0.021243797085501972,
+            -0.01667280589187595,
+            -0.003688045116580565,
+            0.006189970874536939,
+            0.0020911968266404467,
+            0.0006097761986763103,
+            -0.004685468039463817,
+            -0.01011357666327168,
+            0.022631082517271542,
+            0.012398000826637277,
+            -0.006721588925828951,
+            0.018142966317821054,
+            -0.0021330638043572654,
+            -0.011869184647489572,
+            0.025073455250141923,
+            -0.0015392945266926425,
+            -0.0337085890598252,
+            0.01267302113143573,
+            -0.013438601165482187,
+            0.02673290895281225,
+            -0.009586762706902736,
+            -0.011208527168019762,
+            0.019144928964011168,
+            -0.0017591316941248897,
+            -0.005459150067129732,
+            -0.029230079356228025,
+            0.004679497471366242,
+            -0.011723195942933286,
+            0.015899377439148873,
+            0.0022054234777254627,
+            0.029187759252864872,
+            -0.018290694895057157,
+            0.00792533824593637,
+            0.011719198071061566,
+            0.014612375997487872,
+            0.005419943379355563,
+            0.010515265723144554,
+            -0.006893314445458116,
+            -0.04424495426064596,
+            5.8241472764594054e-05,
+            -0.005719143565673274,
+            0.0038220021965914754,
+            0.013106940982933836,
+            -0.01203955335201393,
+            0.011260626096963215,
+            -0.0279896016489392,
+            -0.0295228261396555,
+            0.036474811902902415,
+            -0.04900747926861378,
+            0.019261089742811332,
+            0.00014104646541594166,
+            -0.03057197731717424,
+            -0.04331886560073797,
+            -0.03232957159604667,
+            -0.019401402365464362,
+            0.004102283356333861,
+            0.0002610070704924167,
+            0.0012280372705500212,
+            -0.014025012323929194,
+            -0.014135990809256075,
+            0.013269444858490211,
+            0.00992155580941576,
+            0.011225162609266308,
+            -0.006545044517282357,
+            0.002368862850663672,
+            -0.02063183105457681,
+            -0.026981383275694142,
+            -0.02774194980145225,
+            0.013337303400678109,
+            -0.0025902548982556788,
+            -0.02794339093668326,
+            -0.025606606546251152,
+            -0.0029852498102770587,
+            -0.01906873442473352,
+            0.006508845592362198,
+            0.03268861529625085,
+            0.0355975143313428,
+            0.010066906387952277,
+            0.01690250467284985,
+            -0.017201544758038183,
+            0.048345309114140714,
+            -0.019031171948182872,
+            0.03188329748476941,
+            -0.0005845169448621133,
+            -0.024140202991373587,
+            0.00011597263700677706,
+            0.03203623528903764,
+            -0.019285565857567896,
+            -0.01332755507465857,
+            -0.010914187315481855,
+            -0.003374912271519852,
+            0.02419326154642231,
+            0.02443825785057301,
+            -0.0037923303809811746,
+            0.00022800399604730175,
+            0.015893265651968885,
+            0.003154527829977343,
+            0.007688529734165207,
+            0.030623379985627763,
+            0.014052282574575454,
+            0.01127235345371133,
+            -0.04164707563955819,
+            -0.011811587719246128,
+            -0.003763086694358232,
+            0.005909410979841885,
+            0.002913220159138154,
+            -0.002671613084376487,
+            0.0045765194162907885,
+            -0.007241703279856166,
+            -0.005855949422953439,
+            0.0016973541436777935,
+            -0.001576326424468957,
+            0.008393873711385013,
+            -0.0037217094191171083,
+            -0.03498879678492834,
+            -0.026096985039773694,
+            -0.021280075788049788,
+            0.006437497348154639,
+            0.0070214556591898375,
+            -0.00972721128392027,
+            0.01949553762684561,
+            -0.0029404319702066146,
+            0.000271037702827666,
+            -0.010734867909488861,
+            -0.0029784341515646075,
+            -0.01206464664516104,
+            -0.019544778160755956,
+            0.00035139045159739705,
+            0.013457601448003152,
+            0.00025947376767512006,
+            -0.03165522602526639,
+            0.011979811705146604,
+            0.006642849985850598,
+            0.013181162886675883,
+            -0.024499404540711422,
+            -0.02310714308893576,
+            -0.028028912907092744,
+            0.010570122662701183,
+            -0.0066866321204157725,
+            -0.024617571284817428,
+            -0.004097807033448707,
+            -0.0074576460628928256,
+            -0.0020417256533259887,
+            -0.02470953739335365,
+            -0.011408461668188081,
+            0.023537349682260345,
+            -0.006373652338136973,
+            0.01936338676879376,
+            -0.017636435070085086,
+            -0.017036091876814843,
+            -0.002198685044404603,
+            0.015584758061011048,
+            -0.038943923931820196,
+            -0.002076529978164001,
+            -0.008452513321945455,
+            0.006551055965075869,
+            -0.028840587857625097,
+            -0.007128279504360441,
+            0.0199397499488538,
+            0.012424319366271584,
+            -0.03518875279748495,
+            0.013596027106451115,
+            0.022191403935861113,
+            0.01218507466930854,
+            -0.0038551718784437783,
+            0.03910075115429626,
+            -0.0025215641203030336,
+            -0.005422688956956598,
+            0.016759888275451905,
+            0.04462538468871483,
+            0.023708097134660096,
+            -0.03152743757744915,
+            -0.008668774531128668,
+            -0.015224825785549947,
+            0.004102648962663789,
+            -0.025867795193966987,
+            0.004676262922216584,
+            -0.011739753544166856,
+            0.01645951757031186,
+            0.011253400919230348,
+            0.013958707917471253,
+            0.0376596076542735,
+            -0.005077354977296842,
+            0.02668910096392324,
+            0.0010332284250006117,
+            -0.00856723931523911,
+            0.0014431148305642308,
+            -0.01412579094241589,
+            0.029892393553325205,
+            0.02269599036836208,
+            0.014421409701746036,
+            0.029248195108770955,
+            -0.03588867295960255,
+            0.02458655544632528,
+            -0.0010013539341895672,
+            0.008368648583431747,
+            0.018150734970019018,
+            0.02628022565029391,
+            0.015423867373844644,
+            0.012477080028047823,
+            -0.02513018750675568,
+            -0.022180001014682045,
+            -0.0027550350630288997,
+            -0.031944843677988374,
+            -0.0014668093006914267,
+            -0.009789309083948147,
+            0.026386027599666254,
+            0.0046988411391149,
+            -0.019803160322638583,
+            -0.003931806845108628,
+            0.004482097428470392,
+            -0.00244660100774939,
+            -0.010441732346892935,
+            -0.004649318716095894,
+            -0.01869464992812615,
+            -0.017838643679294407,
+            0.023677878122153204,
+            -0.023958455640817666,
+            -0.025238725541146534,
+            0.014582798102040157,
+            0.0013223590277629197,
+            -0.002295641913710489,
+            0.0008238518461567685,
+            -0.02115533275282518,
+            -0.04214275933987015,
+            -0.027817083427786927,
+            0.0037588803464689412,
+            0.03565056002940463,
+            -0.026788865866928923,
+            0.014162550819567465,
+            -0.011336583458519,
+            0.019175861037658344,
+            -0.005127229083828615,
+            0.015531116410874209,
+            -0.018251290727445846,
+            -0.044630374763679266,
+            0.021833307809708984,
+            -0.012087009688208446,
+            0.02720742725750304,
+            0.021497809277370574,
+            -0.026937653159100156,
+            -0.013572733244188407,
+            0.01358214612023832,
+            0.0020408253706382332,
+            -0.009652962552750464,
+            -0.0063411929391088626,
+            -0.010533309167065193,
+            0.003901271588567942,
+            -0.04496345296706673,
+            -0.008761188928422701,
+            0.06948704056954039,
+            0.005916180375965137,
+            0.03157905519638486,
+            0.014668818318138927,
+            -0.028448496686381258,
+            -0.04008949536631331,
+            -0.00462253589823598,
+            -0.0003701522833020314,
+            -0.00024410762276269235,
+            -0.01850585249309217,
+            0.017699615637510956,
+            -0.002712069912576265,
+            -0.009767338815933488,
+            -0.013119464222297351,
+            -0.013743610418034631,
+            -0.030375365173543157,
+            -0.0030316327877227783,
+            -0.008632689470358149,
+            -0.006565457264917239,
+            -0.0328089275944709,
+            -0.011713996290006943,
+            0.029773910045828737,
+            0.02740100904353622,
+            -0.0061926915497348915,
+            0.00638737826122608,
+            -0.010244154924811975,
+            0.0010396203272677069,
+            -0.001264454445070139,
+            -0.004059969507786729,
+            0.008715730726831364,
+            0.0014767015415289076,
+            0.031074767756160294,
+            0.004761858344125113,
+            0.00987870172044607,
+            -0.059926117793699066,
+            -0.009294021263133636,
+            0.021188432871397475,
+            0.016602826531924377,
+            -0.009125784136292983,
+            0.013946188378030194,
+            -0.0017670794602549428,
+            -0.006837334179368295,
+            -0.028290055353601393,
+            0.024556541126233806,
+            0.04543639597357669,
+            0.0031311343860416473,
+            -0.011199639773247286,
+            0.014965677855595444,
+            0.0018248533506196013,
+            0.015507721118783542,
+            -0.011409402378400194,
+            -0.01338737990904486,
+            0.03063410115541065,
+            0.009305362935335817,
+            0.0025578631174065385,
+            0.012727648343848194,
+            -0.0038680554900795763,
+            -0.01173215289303824,
+            -0.01568862495293563,
+            -0.004101703129244437,
+            -0.03649200408657064,
+            -0.0022107849490954137,
+            0.009659780679763728,
+            0.0034091693454195806,
+            0.008848380026939594,
+            0.01842435611499587,
+            0.03546926214224266,
+            0.013478278853234764,
+            0.005970293629431872,
+            -0.0069197533761813455,
+            0.023196445250565058,
+            0.010403914879205351,
+            0.010017272564548626,
+            0.015076807790462658,
+            -0.029649260409273275,
+            -0.012737152920662164,
+            0.00373276777994811,
+            0.013239909450830258,
+            0.0001682544254742693,
+            0.02030652051051447,
+            0.014143961923579586,
+            0.0010147733405793957,
+            -0.012509446388074923,
+            0.0020437769941487626,
+            0.007405578085797006,
+            0.013458276946120546,
+            0.0006779114540819997,
+            -0.004250745498730309,
+            0.0009149988500358458,
+            -0.008355906072311008,
+            0.040391939952386505,
+            -0.01831218885700067,
+            -0.016793844108223176,
+            0.035064882870542116,
+            0.11416006426189942,
+            0.0010630856112440689,
+            -0.015660014600661857,
+            -0.017142534874931403,
+            -0.0086899013413391,
+            0.016611825572132997,
+            0.01609650977226868,
+            0.009780106186629793,
+            0.009622367309076686,
+            -0.000672926017804395,
+            0.024310949465350565,
+            0.02944698490225121,
+            -0.0037169531700178868,
+            -0.010990838766452194,
+            -0.036819646475041955,
+            0.022923962979125287,
+            -0.004596366586768123,
+            -0.007108041562635549,
+            -0.002604794313552557,
+            0.012641223813290853,
+            0.013668775645501733,
+            0.005125100040230003,
+            0.0077993841846437815,
+            -0.011500851674334447,
+            -0.017334458072700107,
+            0.0056819751688881774,
+            -0.015982234644378358,
+            -0.0076533013358874935,
+            -0.013127172654362694,
+            -0.011837346554842119,
+            -0.006562312362371076,
+            0.0029877420008976425,
+            -0.008572723854050838,
+            0.00854083969164562,
+            -0.010939639783598493,
+            0.016185642759084087,
+            -0.008052282332943615,
+            -0.0011992212936609377,
+            0.0010378646897070758,
+            -0.0025399515501741718,
+            0.010712865639618352,
+            0.01451357949543628,
+            -0.02521401163091652,
+            0.020836357346214075,
+            -0.008252813367155418,
+            0.0006409807249997337,
+            0.015338809580802425,
+            -0.04672526649678653,
+            0.022728876784937357,
+            0.005129225753457052,
+            -0.018372222092794564,
+            0.0066182736895667805,
+            -0.02048375780196863,
+            -0.00896382274306482,
+            0.0021738974344880794,
+            0.004491167150658828,
+            0.0034280799511763956,
+            0.015611133773752265,
+            0.018126362497548308,
+            0.00014383671280220997,
+            0.007927708693242186,
+            -0.006973076292062532,
+            0.008118313055163714,
+            -0.006791836664375275,
+            -0.01164567445668625,
+            -0.010894577039363014,
+            0.00014731542306100506,
+            -0.008865132658571428,
+            -0.011415696755798914,
+            -0.017922318360382588,
+            0.00358538001727405,
+            0.00289327637128695,
+            0.010406054711003452,
+            -0.007757607207626101,
+            -0.010994775479337082,
+            -0.011716096880261664,
+            -0.0013333474316266438,
+            -0.01231144419953554,
+            0.009792603083441917,
+            -0.02884953407490196,
+            0.0029097409399144306,
+            0.025277274938032477,
+            0.009263530315933738,
+            -0.028037993274175277,
+            -0.02532252134134532,
+            -0.0018596440554143638,
+            -0.013847737486669237,
+            0.009910107581119272,
+            0.013952962280235523,
+            -0.011807495289410774,
+            0.003896325037406752,
+            -0.02144921729633576,
+            -0.01628545945817568,
+            -0.0040148671532777005,
+            -0.016032493124255323,
+            0.004580041957001551,
+            -0.011375180364988195,
+            -0.007789996450583554,
+            -0.022041665680945066,
+            -0.005890065942573431,
+            -0.050455058269193866,
+            0.014401045842891128,
+            0.030219548461217365,
+            -0.004580747668649223,
+            -0.028684680969792875,
+            -0.015525870112797619,
+            0.011121401958547706,
+            -0.008879749268609245,
+            -0.019454243014778435,
+            -0.007839434838464604,
+            0.018908534797107714,
+            -0.0019283067432864216,
+            0.0024769961730655835,
+            -0.006732653593696035,
+            0.027604805021030646,
+            0.01366593691107021,
+            0.03740746059041736,
+            -0.014662268336336578,
+            0.02539666271785517,
+            0.02371809783638648,
+            -0.036126848124425154,
+            -0.007627445211958312,
+            0.03657481892016626,
+            0.01092515982671624,
+            0.0011046573579274704,
+            0.012350047525582187,
+            -0.006340802546498992,
+            -0.031330519599469066,
+            -0.0013691360532301728,
+            -0.0014676630169621005,
+            -0.023111265345539388,
+            0.027359543575867225,
+            0.018091458372134273,
+            0.014237156751820934,
+            0.0030993618578507172,
+            -0.0014037082734049385,
+            0.013257269860435359,
+            0.01021315143964527,
+            0.030885417052806473,
+            -0.003225065956381245,
+            -0.012825341172770057,
+            0.013476039084668826,
+            0.01852164843904835,
+            -0.009965495459579544,
+            0.012541079912270015,
+            -0.00824122613985912,
+            -0.018124134280975132,
+            0.03192263766222747,
+            0.0001484308749572035,
+            0.010888733498667343,
+            -0.010649515858317311,
+            -0.013082335214246402,
+            0.032614733822945205,
+            -0.00999075761269065,
+            0.004257643407198955,
+            -0.030723533031261387,
+            -0.003820776505926347,
+            -0.010471792928758869,
+            0.01737608180148319,
+            0.004597677757231315,
+            0.02095148515295301,
+            0.029253627700524396,
+            -0.011604450820008121,
+            -0.01996325509217509,
+            -0.001158626158150122,
+            -0.020583793738536985,
+            0.004168757459099133,
+            -0.03632338333261702,
+            -0.019373660230711246,
+            0.022259216603413172,
+            -0.010364620831688015,
+            -0.004751193403783363,
+            0.00444187509644351,
+            0.0008572202661850934,
+            0.014943779567410408,
+            -0.025408263453313398,
+            0.017755758370543148,
+            0.004627392860417132,
+            0.007402073071525438,
+            -0.024913329140362466,
+            0.013760771254234878,
+            -0.006087346663657887,
+            -0.03780469033982951,
+            0.005509388120163784,
+            0.03756141956381493,
+            0.02381019168406692,
+            -0.024580249929652097,
+            0.01807584361080568,
+            -0.03111039245576191,
+            0.023245542501125073,
+            -0.004868090951368273,
+            5.1544460403943004e-05,
+            0.00640235757127173,
+            -0.004698373089469779,
+            -0.00621383035264185,
+            -0.0025132415458431587,
+            -0.008746634331769936,
+            -0.01611801559493664,
+            -0.01193535623883546,
+            -0.03381789115349148,
+            -0.024119754887378317,
+            0.013881102852385332,
+            0.0019232668966222687,
+            -0.0095263729740736,
+            0.018652966857927805,
+            -0.03775912497511837,
+            -0.012022352025516115,
+            -0.023289377270709736,
+            0.023579672990954228,
+            -0.028605627815260902,
+            0.019235397553197357,
+            -0.025838655372560505,
+            -0.021458526449310157,
+            0.01508157885177254,
+            -0.010064396230734452,
+            -0.0016708624560165368,
+            -0.015042071642751917,
+            -0.03670785416972153,
+            -0.006267614265977843,
+            0.02298763967146357,
+            -0.012506444728541897,
+            0.00029871598584066644,
+            0.02671390618094565,
+            -0.003059865833779805,
+            -0.011103452976820622,
+            -0.0017382684168257506,
+            0.010923055147701782,
+            -0.02938686260440486,
+            0.014300097108092281,
+            0.0038571851420598074,
+            -0.0037614499297117365,
+            0.0010991440540809893,
+            -0.010108413486243776,
+            0.02127952161226619,
+            0.009142545227414532,
+            -0.0030260119510809205,
+            0.040589877142650374,
+            0.008677453889680373,
+            0.0017339508605300814,
+            0.008732950895530873,
+            0.01063352447960556,
+            0.006618635029823979,
+            -0.014986307792626314,
+            0.016905861015681957,
+            0.013101237900847744,
+            0.0002162791516784468,
+            0.019309343995079022,
+            0.0021263376627204313,
+            -0.024516267132902416,
+            0.013339486637573757,
+            -0.02311659751478777,
+            0.0020575982953961246,
+            0.02391695586495566,
+            -0.020440172706421535,
+            -0.001656132876701879,
+            -0.006304133667169861,
+            0.000543457661904495,
+            0.022135707414160034,
+            -0.0230903871986617,
+            -0.023444763940153112,
+            0.009604600033024832,
+            0.01721387169484476,
+            0.01827905276638783,
+            0.02824773106648473,
+            -0.007100625424382998,
+            0.01348510178676837,
+            0.0002604386361122316,
+            -0.011802178189607182,
+            0.007664157334475143,
+            0.0074175377458313594,
+            -0.0044008460682833875,
+            0.025491035291032547,
+            -0.0017015687016376205,
+            -0.006134010425966364,
+            -0.005379657499404915,
+            -0.01584573519366957,
+            0.0034386456145185213,
+            -0.013860742817569676,
+            -0.013355238174583658,
+            0.0040342831827141845,
+            -0.0208160950322342,
+            0.10904592104153767,
+            -0.0037038089210379766,
+            -0.019472012964045247,
+            0.0063554319446707044,
+            0.003716719884684827,
+            -0.003692786470980806,
+            -0.003353843210051379,
+            0.007788605566526459,
+            -0.011254480693190834,
+            -0.0045751956417398095,
+            -0.013976039724381875,
+            -0.001989262807706565,
+            0.010067374586433014,
+            0.009201983771978006,
+            -0.005902980676079493,
+            -0.02029714318363332,
+            0.013031676901335606,
+            -0.016970144750162643,
+            0.008874670945038486,
+            0.0008311255074953139,
+            0.00884167159995975,
+            0.001716603195908536,
+            0.01854951250316076,
+            -0.009310881405486074,
+            -0.01325242738760595,
+            0.029667589863813217,
+            -0.005616511284128333,
+            -0.0038179509972925954,
+            -0.0004690716809300012,
+            -0.005068974756744239,
+            0.0021204231125051274,
+            -0.027958274497127258,
+            0.022127088236172387,
+            -0.017271838492631593,
+            0.010267709087809265,
+            0.01316800560257366,
+            0.005599367128954229,
+            0.01817806970002991,
+            -0.007115594036834107,
+            -0.02627643123342057,
+            0.006542742528003751,
+            -0.003433874053082844,
+            0.006631427284537971,
+            0.02316295746074422,
+            -0.014573351469966297,
+            -0.033520864104177005,
+            -0.013707022143307006,
+            -0.01877574599842729,
+            -0.005992004412448576,
+            -0.0043698881826095435,
+            -0.01481352026460129,
+            0.006424121402963946,
+            0.01382938882387991,
+            0.018136021761515388,
+            0.0012697483790911433,
+            -0.011531136974861559,
+            0.00609187274277241,
+            0.01076133059240578,
+            0.008400887242930113,
+            -0.01533504932731406,
+            0.011439227832642668,
+            -0.028718487721012004,
+            0.01522697877548034,
+            -0.012452665599305595,
+            -0.024938748080573947,
+            -0.0022082450217489366,
+            0.017589399637694264,
+            0.0037824832255756763,
+            -0.004355445812154662,
+            -0.018778668474337727,
+            0.019825087149714933,
+            0.024692878851503508,
+            0.003883497401229232,
+            0.019470881143244197,
+            0.041834635328088364,
+            -0.003544509491805351,
+            0.027139164223766474,
+            -0.02129723143323512,
+            0.01461354860170749,
+            -0.03441729954693155,
+            0.001893713808284104,
+            -0.004222657393633789,
+            0.0013677766871517654,
+            -0.003811928806805752,
+            0.020142937759360904,
+            -0.009158202577174012,
+            0.0030638058587375543,
+            0.024854053046139163,
+            -0.018665378030958756,
+            0.00805903734779081,
+            0.010625429315145887,
+            -0.012856237251902233,
+            0.028535299711292306,
+            -0.006335094045840286,
+            0.00305422126851784,
+            -0.0297883488914759,
+            -0.0029192872993071216,
+            0.0026067044379754047,
+            -0.006097749614529602,
+            -0.00968023644800927,
+            -0.007433916539502931,
+            -0.006202274026852405,
+            0.0038354607448269555,
+            -0.021471361566683354,
+            0.003195646827154078,
+            -0.004390751282171024,
+            -0.003323447423321121,
+            0.02518759723554948,
+            0.025857746688368054,
+            0.016893707217894103,
+            0.012739170140289531,
+            0.008553625305650129,
+            0.019805351588748434,
+            0.01654964927067909,
+            0.011549329129728739,
+            -0.020026013396046632,
+            0.01150644479272541,
+            -0.004159611921476011,
+            -0.0022937297602724912,
+            0.014815466072709131,
+            -0.01572171840394285,
+            -0.02276187015864222,
+            0.024693009795978448,
+            -0.019725315346836404,
+            0.00998768979020729,
+            -1.1702210281215188e-06,
+            -0.011129745730507545,
+            -0.002591293237053776,
+            -0.018034357254990988,
+            0.00012070821244376275,
+            -0.023051045150566527,
+            -0.028465270110388716,
+            -0.00048603008831669926,
+            -0.000574546300981984,
+            -0.012992623897880395,
+            0.02064747165046032,
+            0.014820638761258362,
+            -0.024820174379739057,
+            0.02666518374464728,
+            -0.017430076811109155,
+            -0.01495051200726725,
+            -0.005861181182847845,
+            0.00100881727574559,
+            -0.0055223282819321295,
+            0.0034719859452138017,
+            -0.010180752654175722,
+            -0.009961678082468036,
+            -0.012134423047419183,
+            -0.028073672819317354,
+            0.011574683159318163,
+            0.016120674453124317,
+            -0.015043147648505721,
+            0.031834693135959095,
+            -0.008802805725050891,
+            -0.0018698230152007877,
+            0.016179742266521146,
+            0.005096232332051617,
+            -0.017187077376166815,
+            -0.018999569809271876,
+            0.012251308792871571,
+            0.013279090498325945,
+            -0.009588297039279112,
+            0.006630431200804809,
+            -0.014130176783207386,
+            -0.005092662672039885,
+            0.007020236623084732,
+            -0.005049363863784624,
+            0.0055051712211635915,
+            -0.00995314895304919,
+            0.0052743456988306485,
+            0.005841370337169303,
+            -0.0029434616355351975,
+            0.004677049987338193,
+            -0.018568941264514256,
+            0.020031253265629777,
+            0.0018725390936655962,
+            0.01332532735283085,
+            -0.03518855189467458,
+            -0.0056298723325832025,
+            -0.005369475318146738,
+            -0.004650364775945294,
+            -0.020833510091412353,
+            0.020445280915606132,
+            0.0159991245532085,
+            0.013202409908533162,
+            0.023094507005426774,
+            -0.014905666713082246,
+            0.01572052456975675,
+            -0.020945766915121455,
+            0.013114000985348841,
+            0.003349101658930482,
+            -0.019829605346106578,
+            -0.001423750749031516,
+            0.04557459007627153,
+            -0.005847231973329081,
+            0.011986110554869603,
+            0.0012709842140249691,
+            0.026758922819943853,
+            -0.0001635536762342011,
+            0.0065419519929442995,
+            0.005997692702992206,
+            -0.009935128108338126,
+            -0.0008057798606777003,
+            0.00865712443881613,
+            0.046254320605750854,
+            -0.01181132866321511,
+            0.008170688314239364,
+            0.013099051984057827,
+            0.0034971494510865624,
+            -0.0008386400616584293,
+            -0.015346506644390764,
+            -0.03067678541405639,
+            0.019305962963428017,
+            0.007736916644077731,
+            -0.012756525606221703,
+            0.008734673666457216,
+            0.015137771675476126,
+            0.012891824153867332,
+            0.003895886821685762,
+            0.02155154331885366,
+            -0.012761690789748027,
+            0.018528313792219413,
+            0.024663453412919027,
+            0.009077230086690327,
+            9.184526542175535e-05,
+            -0.000751529849753638,
+            0.006791927267880182,
+            -0.0004433479107716552,
+            -0.009832230053513483,
+            0.015858232096630655,
+            -0.014861341148074698,
+            0.017790924034108806,
+            2.7216806238946785e-05,
+            0.03744841066552559,
+            -0.015216207585985075,
+            -0.010138319613387346,
+            -0.007774325714088762,
+            -0.0016997034352120362,
+            0.003971564314593291,
+            0.0018686537276849653,
+            0.006051669054361943,
+            -0.0008319624938748146,
+            -0.017326561560776794,
+            0.01667737942301944,
+            0.005719808546053257,
+            -0.0007405637872110778,
+            0.013750752065593804,
+            0.0038732323799683453,
+            -0.00367645612814479,
+            0.023580906347714314,
+            0.006238541579764762,
+            -0.005235795826199542,
+            0.0028182323816636243,
+            -0.006443210286549713,
+            -0.016255209299456314,
+            -0.003061441395209219,
+            0.004597970789372988,
+            -0.0016590920570710628,
+            -0.011138261849069419,
+            -0.0019423173609729848,
+            0.020355534503272887,
+            -0.004612410336124759,
+            0.013346750536413418,
+            -0.005400060374036387,
+            0.006252154576727367,
+            0.010601459925435238,
+            -0.012346569079682625,
+            0.0014154620309235822,
+            0.010406076751218612,
+            0.0013983417564564663,
+            -0.007140180075225315,
+            0.0010544410919327675,
+            0.007757599672903709,
+            0.006505350216372801,
+            0.00483063937962671,
+            0.00836630436955631,
+            0.013284451514874498,
+            0.0033749897293287438,
+            -0.006243293993633622,
+            0.0013949831763016042,
+            -0.013269921385800992,
+            -0.006308639321247727,
+            0.0010906169976769569,
+            -0.0010152835709869353,
+            -0.02017142197515583,
+            0.010787630901414118,
+            0.005355784839945777,
+            0.007838485470991034,
+            -0.01706932891696025,
+            0.009576030158827565,
+            -0.0008144311254998525,
+            0.02829832828245172,
+            0.007407953684956539,
+            -0.02769403281764041,
+            0.014266001222243134,
+            -0.010492199740754951,
+            0.022575782293320284,
+            -0.0016349316593207844,
+            -0.005656505606143177,
+            0.008025514398322551,
+            -0.009634281579228592,
+            -0.005991026158213869,
+            -0.022366083313133788,
+            -0.025825209755160946,
+            0.016871087849192924,
+            0.013583173973358155,
+            0.021125252303971605,
+            0.00012555627355712715,
+            0.015347535950809997,
+            0.0020128011217859444,
+            0.015164099887550086,
+            0.0024544903332736167,
+            0.012507136277293888,
+            -0.011798050869794805,
+            0.0104828761281283,
+            -0.0137805761114781,
+            0.014620357150314097,
+            -0.005246639228547017,
+            -0.006986408834884282,
+            -0.008849580196616098,
+            -0.005442165099441538,
+            -0.020891551850452786,
+            -0.009155930286949747,
+            -0.0013590151948670868,
+            0.010665872513236615,
+            -0.0005386423407222223,
+            -0.0027671778465775897,
+            -0.03237311596935737,
+            -0.004875062533405715,
+            -0.006670982068843501,
+            -0.01942244110096111,
+            -0.0045975373348006885,
+            -0.026137783023267008,
+            -0.01657173843643047,
+            -0.0007784046498232684,
+            -0.015037306676062236,
+            -0.006889680260149007,
+            -0.02009705080543122,
+            -0.027192867101205106,
+            3.0161964469866572e-05,
+            -0.009647644858034833,
+            -0.005380165957043753,
+            0.013815092649296373,
+            0.011902366689169847,
+            -0.01572592170758115,
+            -0.022321702398924333,
+            -0.01088937403483412,
+            -0.0165754067023749,
+            0.01646435743868707,
+            0.009914938007210463,
+            0.018834863207293908,
+            -0.003724245912768107,
+            0.005643505694722323,
+            0.0009581443359000331,
+            0.02327704258596552,
+            -0.02532422120223976,
+            -0.028694220894404807,
+            0.0019298778426469338,
+            0.0025970175688368323,
+            0.006086164866310174,
+            -0.009414665394188936,
+            0.001200390869257535,
+            -0.007782899559317819,
+            -0.0228264209954853,
+            -0.00898519580978142,
+            0.008156835270668449,
+            -0.010036630164522355,
+            0.02230664068547531,
+            0.012332230358869292,
+            0.013699197379189934,
+            0.010593587938184292,
+            0.01576174456497367,
+            -0.0001654241831584687,
+            0.014471905257729428,
+            -0.008675915624727887,
+            -0.0031070346099204425,
+            0.0038358404385932777,
+            0.005878417037187274,
+            -0.00625022998235275,
+            0.028077114529659795,
+            -0.0011836453716817525,
+            0.0002962695480296182,
+            0.014405782572641941,
+            0.02697667559474498,
+            -0.004710313427099278,
+            0.0210512043261309,
+            -0.002393165025068831,
+            -0.006135816286212752,
+            -0.008429776018235104,
+            0.008607617299702318,
+            -0.009529291136733401,
+            0.011133497451942684,
+            0.019864404836361926,
+            -0.014041315457579166,
+            -0.007351413712676263,
+            0.02950254974910368,
+            -0.014564388091367897,
+            -0.01825317106331622,
+            0.02325964952838705,
+            -0.007414295676409368,
+            -0.0039042587057605484,
+            0.00377144546378926,
+            0.007169856503233239,
+            -0.004036744966949995,
+            -0.012928341573802416,
+            0.016090527447479415,
+            -0.00011159893712484579,
+            0.014146137810890391,
+            -0.0004655839804281017,
+            -0.006372231942983921,
+            -0.010047382825720416,
+            0.002373224247019062,
+            0.026818813126567614,
+            0.013483627627077692,
+            -0.017895710143841333,
+            -0.013785856764479817,
+            0.010826610341182479,
+            -0.01538893818197323,
+            -0.004425206001945227,
+            0.02155595194812746,
+            0.0028768153668128553,
+            -0.020543027467698483,
+            0.008028975884363618,
+            0.0003835165718138906,
+            0.00037046947047892953,
+            -0.00029363296913641716,
+            0.030135739208689217,
+            0.001343516577762509,
+            0.00018948658255495327,
+            -0.00875053830853609,
+            0.008606573235959595,
+            -0.01280305089741873,
+            0.012786478836532904,
+            -0.02437993139337839,
+            0.007648181775166282,
+            -0.00629541989541625,
+            0.004579605315599212,
+            0.016658386927051592,
+            0.01218098627022323,
+            0.015031618813778413,
+            0.008237200519537417,
+            -0.008942965899756683,
+            0.0009093265012901892,
+            -0.025302175262520477,
+            -0.001333639155622843,
+            -0.017971876771137367,
+            -0.02036433247634014,
+            0.0002463842837918767,
+            0.022144712689630985,
+            0.018559482073124734,
+            0.011579840028464503,
+            -0.008657014440886656,
+            0.0166696911919226,
+            -2.9834620942882283e-05,
+            0.011845827764518911,
+            -0.00037901176785495324,
+            -0.006138374154553732,
+            0.004711058953619512,
+            -0.003622694588720417,
+            0.013542405259906128,
+            -0.014730567462694827,
+            0.012076044672004906,
+            0.01690223095263866,
+            -0.009108632851807137,
+            -0.008946251274440141,
+            -0.013992423874475428,
+            0.00048381889145447367,
+            -0.015909599902931202,
+            0.035489252674454214,
+            0.02252772349379229,
+            -0.01210464970923921,
+            -0.010469120693943236,
+            -0.014164999313775756,
+            0.019496048806752753,
+            0.010427539351244087,
+            0.0028018908451425276,
+            0.017091768406027965,
+            0.021395638416767112,
+            0.008817247593404623,
+            -0.006081063711583319,
+            -0.0021216503501144983,
+            -0.003781810594530787,
+            0.020143098218482827,
+            -0.003078172339607037,
+            -0.011350171884507555,
+            -0.012516379645909518,
+            0.005766102930704529,
+            -0.01626433668691071,
+            0.0019294643928566657,
+            0.008641558521764407,
+            -0.015057688533773374,
+            0.019407986696657772,
+            -0.028126197703647868,
+            0.012364374199657305,
+            -0.0044278434183731225,
+            -0.007251147925369614,
+            -0.03496220945653279,
+            0.006389489026974697,
+            0.008941259259632024,
+            0.003941752277454314,
+            0.023003838655784314,
+            -0.024528514847470928,
+            -0.018361931455105486,
+            0.0037791061321962543,
+            0.00047162060848090104,
+            -0.00711894577052754,
+            0.0010138871688747137,
+            0.004128392769213672,
+            -0.00674546985015164,
+            0.0017307097443508704,
+            0.0033354800298095546,
+            -0.002829898077850263,
+            0.013252947102333348,
+            0.0081724269619999,
+            -0.021449979263734438,
+            0.001368791856860285,
+            0.01590656734732547,
+            0.01452008402429858,
+            -0.02391847163018108,
+            0.006331397860689212,
+            -0.02064812934794864,
+            -0.000631340242718475,
+            -0.031272866000888556,
+            0.016859793097564958,
+            -0.0069008624795683276,
+            0.01899718801899956,
+            -0.0015377436942771224,
+            -0.0065810685260109004,
+            -0.012822620655323715,
+            0.02137604937338179,
+            -0.0019121137078344135,
+            -0.0025598772127157477,
+            0.015678293057066646,
+            -0.025700865215693987,
+            0.017825186616033165,
+            -0.0026230848396158204,
+            -0.017342967698019966,
+            0.017199139867162275,
+            0.004536019178633225,
+            0.007224666811309087,
+            -0.023505589665315567,
+            0.0007208041198255396,
+            0.003900164485347562,
+            -0.017250324025622685,
+            0.004470862192961212,
+            -0.030807400122398566,
+            0.0006579147990327597,
+            -0.013222907940559794,
+            -0.0020514994691204847,
+            -0.0046763235724575445,
+            -0.014050035576183385,
+            -0.039181147615786484,
+            -0.005429639279537522,
+            0.016065742005148733,
+            -0.0009029636981539927,
+            0.01800715198737081,
+            -0.002730931639143803,
+            0.002947680833843522,
+            0.004912692811020822,
+            0.00818378920043427,
+            0.006915895602313347,
+            0.02274116906914758,
+            -0.0057854922886970365,
+            0.00471999900678522,
+            0.007058014277137233,
+            0.014638081356210031,
+            -0.01719565951201012,
+            -0.015170377598372723,
+            -0.016015921063369496,
+            -0.006351213642269936,
+            0.01739665344303718,
+            -0.017488885259028475,
+            0.013303332445465686,
+            -0.00429428898520701,
+            0.014825338166443021,
+            -0.008686427208690176,
+            -0.01575945086724703,
+            0.0028564767084934386,
+            -0.007570417028925067,
+            0.001007670647768344,
+            -0.02323908008484089,
+            0.02019729564706863,
+            -0.00847581038203716,
+            0.010508144473197669,
+            0.01729824052022313,
+            -0.021364529179605813,
+            -0.009742851959432839,
+            -0.014342281441152413,
+            0.029896210049029176,
+            0.013925163184699139,
+            -0.009392252140772913,
+            0.003927230832659267,
+            0.008873561269230905,
+            0.011529536139797032,
+            -0.012209070214071445,
+            -0.0027679974125029362,
+            0.002411914349172495,
+            -0.0062056207925691356,
+            -0.002025687569076897,
+            0.03090134105002816,
+            0.0038285457265343245,
+            0.024877474647546086,
+            -0.013214997316123852,
+            -0.0033763842539861907,
+            0.00023379106197363714,
+            0.005134764372494098,
+            0.01183867975321486,
+            -0.0017966997954747142,
+            -0.008029791503028367,
+            -0.01179500476055557,
+            0.02599911330072515,
+            -0.021852083993691557,
+            0.007024505421035273,
+            0.026784024996764876,
+            0.001390202993990949,
+            0.0029956956045771046,
+            -0.013131691460546865,
+            -0.006298736864509989,
+            0.02523640664202355,
+            0.010030824059858138,
+            0.0041523782603032225,
+            -0.026465819482580182,
+            0.0032128899272377593,
+            -0.0061901409332948115,
+            -0.008681758126710213,
+            0.009866461769956222,
+            -0.0028095454857224164,
+            0.01188012041754054,
+            -0.012191996383095982,
+            0.028459939543805705,
+            0.010711502132045176,
+            0.01047154118931583,
+            0.025888652452698974,
+            -0.02079614305315134,
+            0.0042203106489994615,
+            -0.010829796140534066,
+            0.0006773493086316061,
+            0.005596640511307504,
+            -0.027510562413905437,
+            0.001969898603871003,
+            0.010390866563404346,
+            -0.005642706366801436,
+            -0.01882994179544761,
+            0.01187970346127799,
+            -0.019814385265412387,
+            0.024479579275617887,
+            0.003231341626501863,
+            0.020056914356101588,
+            -0.0053276448188067154,
+            0.009834685132275868,
+            0.0002742942778577165,
+            0.0031361331890882127,
+            -0.0008074598092128465,
+            0.009009012303347016,
+            0.005786777410144894,
+            0.013719989127068968,
+            -0.025528370602188686,
+            0.006821572597883499,
+            -0.0029197127345352626,
+            -0.030539872624857384,
+            -0.013383097680839732,
+            0.010291485622655026,
+            0.012218533188117704,
+            -0.009144425807890594,
+            -0.010378679518611277,
+            0.0030042579827305603,
+            0.005748243116238633,
+            -0.006692403885450407,
+            0.006913672151000981,
+            0.01629516912493545,
+            -0.002533517074771728,
+            -0.0019083178803585312,
+            -0.0035007340321378893,
+            -0.01154609935927537,
+            -0.010016998439072824,
+            0.023193275321990937,
+            -0.00482990342915347,
+            0.017943025772239603,
+            0.025873224580389702,
+            0.0039876567755322,
+            -0.0057577932736143506,
+            -0.005460363755206188,
+            0.01478435999841998,
+            0.014195457012279476,
+            0.004669295894868759,
+            -0.009702220679239768,
+            0.007974305230005956,
+            -0.007760086293456636,
+            0.008375077126777163,
+            -0.020260917110786147,
+            0.002199401871514877,
+            0.0026191373298767766,
+            -0.0022089262577237776,
+            -0.015213807046175918,
+            -0.01956788452420682,
+            -0.021898368819136302,
+            0.012634659463647929,
+            0.00034643102239488474,
+            -0.009241772775582378,
+            -0.013645250997151429,
+            0.0016665106990117016,
+            0.0025630500346914592,
+            0.0002095521560180936,
+            -0.007783022628694597,
+            0.009546147813008391,
+            -0.01265287317178631,
+            0.007445223311736028,
+            -0.004007029181436962,
+            0.0051846833906574515,
+            -0.0016916014415925766,
+            -0.016413938859402438,
+            -0.0210868129634149,
+            0.01920794362224444,
+            -0.012897042227841011,
+            0.02269303737601583,
+            -0.0014935878147854765,
+            3.062680855559304e-05,
+            0.0033132626755401546,
+            0.007274487651476732,
+            0.012110042834251237,
+            0.011426424896797977,
+            0.004307086698938411,
+            0.008206932254416481,
+            0.004673427176334148,
+            -0.013124571552764586,
+            0.004324963759928715,
+            0.0039216033705804084,
+            -0.010449887922072297,
+            0.005298323150690876,
+            -0.0010196552573491,
+            0.006128327172621373,
+            0.012167492341907742,
+            0.01923525204293908,
+            0.01908107550907106,
+            -0.0012031208519043073,
+            -0.0010601912677170903,
+            0.014474374858125607,
+            0.004540954805526089,
+            -0.004883537463424679,
+            -0.002772988629590886,
+            -0.013363713078069854,
+            -0.002572403646338979,
+            0.011541515690646252,
+            -0.02376161747414617,
+            -0.018215231053002656,
+            0.021831219299143226,
+            -0.004991882316041097,
+            0.033796174834149546,
+            -0.013140970283091536,
+            -0.016197177724633614,
+            -0.00464190076219332,
+            0.0026389653040644607,
+            -0.013788049032378498,
+            -0.010300859744205445,
+            0.010545541278382187,
+            0.013848738866656793,
+            0.009903590151120531,
+            -0.0005984085581408575,
+            0.02922816576443645,
+            0.0019900932217491907,
+            0.02245395592422119,
+            0.0018978407151245295,
+            -0.007029294098689854,
+            -0.01440575120349887,
+            0.010840472832356266,
+            0.005529335652955532,
+            0.009050748012161991,
+            0.017791269808484802,
+            0.014502057454389247,
+            0.006265567405788131,
+            0.013928705812292101,
+            0.0012903002372833693,
+            0.0010858377120770652,
+            -0.002355229463060164,
+            -0.01692711767136341,
+            -0.01456786846097011,
+            -0.01627640642218236,
+            0.003873864312554098,
+            0.012620350492241247,
+            0.00525890570023481,
+            0.0007393789113866752,
+            -0.007153210751899716,
+            0.009254566395058412,
+            0.0065087881512835375,
+            -0.012279152412209152,
+            -0.016530068283509593,
+            0.004060671955726967,
+            -0.02210570272789274,
+            0.007849467899009437,
+            0.01554233481902349,
+            0.00219885032769439,
+            -0.002655449524219968,
+            0.0006843524304842362,
+            0.006272944169764281,
+            0.005531927368102191,
+            -0.0036710328198476905,
+            -0.00544113514775027,
+            0.011264661763762068,
+            -0.006379654234440906,
+            0.009537150372938296,
+            -0.00290031548037575,
+            0.03718190949929003,
+            -0.011976080887998194,
+            0.012120408392610773,
+            -0.001751472085435711,
+            0.008481703962951994,
+            0.008067669857949423,
+            0.010772687289903202,
+            -0.003805089040741351,
+            -0.0022164543131735167,
+            -0.008753625032781523,
+            0.006751953442322185,
+            -0.01582177621255226,
+            0.013631301238130705,
+            -0.00030685726238985963,
+            -0.010728958775974582,
+            0.01055472059121978,
+            0.006788316621149835,
+            0.004413382316882087,
+            -0.003262024961860459,
+            -0.01972283203505511,
+            0.014937729522976615,
+            0.0018799533869990667,
+            0.019247008681353865,
+            -0.023276106189004046,
+            0.029043727846427652,
+            -0.02019024213343764,
+            0.023083643395685142,
+            0.01865875504942157,
+            0.001179436167200093,
+            -0.02070329911663634,
+            -0.007618131527214554,
+            0.014871709122115308,
+            -0.02659818889735726,
+            -0.006234719259825804,
+            0.005986829551040766,
+            -0.014639912579831785,
+            0.00037813423546252645,
+            -0.01924149909676064,
+            0.00010832498610532006,
+            0.011013130149723431,
+            -0.011782407678373089,
+            0.00438725832178584,
+            0.01608349750921335,
+            0.019347883652009713,
+            -0.009760017612545174,
+            0.003869246694835168,
+            -0.024240324890794748,
+            0.010958789926998691,
+            0.020511684636649225,
+            0.008883830392748497,
+            -0.01761944522796127,
+            0.021817973960311446,
+            0.010652131228027521,
+            -0.012575679152857703,
+            -0.0012753095453828826,
+            -0.028191274775392804,
+            -0.012421170815054556,
+            -0.02188248722244861,
+            -0.0011670175621882146,
+            -0.014436683528976771,
+            -0.0029783880907228383,
+            0.0009506986717516226,
+            -0.004719165363484719,
+            -0.01179163390107033,
+            -0.019543732196676634,
+            0.016606805875498898,
+            0.0227313413004497,
+            -0.006738735201626204,
+            -0.014547240981002825,
+            0.013866974327185123,
+            0.031694077725890536,
+            -0.016189209297529324,
+            -0.00542110554672736,
+            0.0028061798782555557,
+            -0.012964876141500577,
+            0.018946046750284537,
+            0.0039419760434645495,
+            0.000322115192590907,
+            -0.01991851369210666,
+            0.002489725351392778,
+            0.010835458511052207,
+            0.01684964045708685,
+            0.0005541314524621196,
+            0.010855852688653808,
+            -0.007549882748861744,
+            0.010096800951420057,
+            -0.011832260836468588,
+            -3.6179995603517093e-06,
+            -0.007800366796516945,
+            0.007001049318404058,
+            -0.004424776169203575,
+            0.034356203288485945,
+            -0.02817071276766299,
+            0.004989631403681216,
+            -0.014539644354369607,
+            -0.010888763939716889,
+            -0.0027322086509797843,
+            0.0004865534340981866,
+            -0.010775238981581461,
+            -0.005162220412866082,
+            -0.0011263271110752205,
+            0.00913081242511139,
+            0.005844327320514489,
+            -0.013886529519306775,
+            0.019171827605726503,
+            -0.030984919318656618,
+            0.004469458202753611,
+            -0.008553175891755315,
+            0.010511707988936336,
+            -0.0151185694875566,
+            0.013830711686954931,
+            0.01301959345025159,
+            0.008205984085944315,
+            0.007740367849216328,
+            -0.01532123052721834,
+            -0.01678314222392063,
+            0.001376468747213867,
+            0.0036953018347776244,
+            -0.00931855481896886,
+            -0.033391028922085425,
+            -0.007072913446166969,
+            0.005331350534425606,
+            0.011614453225043,
+            0.008156128603003278,
+            -0.02345206873048703,
+            0.005003870024809975,
+            -0.024178720646704597,
+            -0.012115112684614331,
+            -0.007750557665424973,
+            -0.011580152272614108,
+            -0.006117298954192534,
+            0.026449765548206453,
+            0.007989237291215073,
+            0.016691517078382376,
+            0.004954133565782356,
+            0.02742809127725913,
+            0.011670893621991404,
+            -0.011736840457474895,
+            -0.003349151747880166,
+            0.00035419034843196817,
+            0.005507651806218584,
+            -0.008706590052024251,
+            0.011793241980257321,
+            -0.0013295035978611142,
+            -0.0065061682758755625,
+            -0.004398354347597199,
+            0.009372816666628725,
+            0.014188676762501037,
+            -0.003007980559117429,
+            -0.008724255229333351,
+            0.015749776067989826,
+            -0.030899793617096834,
+            -0.015215033379100091,
+            0.0016152308157299517,
+            -0.007813175908301852,
+            0.017806770680528907,
+            0.01169056345888971,
+            -0.026835412610223673,
+            -0.004538235170332766,
+            0.010532316940606419,
+            0.004755930024691789,
+            0.0095408889025506,
+            0.00954782703036282,
+            0.012079064562163901,
+            -0.007829549014214643,
+            0.01046722658021748,
+            -0.011519369840999348,
+            -0.007092961453184232,
+            -0.007241799314569802,
+            -0.008077934089061052,
+            0.021467547268987214,
+            -0.025816030568238717,
+            -0.006558853543772633,
+            0.01134670818701549,
+            0.013027491377727365,
+            -0.002301991242932548,
+            -0.006299073927206214,
+            -0.017357989759336386,
+            0.008311464809363397,
+            -0.013922568511594478,
+            0.025633907424998367,
+            -0.004854655498817219,
+            -0.005336807899100369,
+            -0.005228056880188717,
+            0.01958682721272445,
+            -0.016774409737540904,
+            0.02072573220949266,
+            -0.014449250679704904,
+            -0.007857351393427215,
+            0.001673929749689194,
+            0.006890008607998747,
+            -0.002948835017234416,
+            0.03734128725793149,
+            -0.00556474133167756,
+            -0.0014544201137417158,
+            0.02209084868209573,
+            -0.01883610080898967,
+            0.017901421953179442,
+            0.01563604483998435,
+            0.012327159100148019,
+            0.0018161034827768648,
+            -0.010253034391420751,
+            0.019979728662282904,
+            0.020823925491027635,
+            0.0139490143219451,
+            0.009577521724445478,
+            -0.013478761809038993,
+            0.014934933784421556,
+            -0.0003725422088504494,
+            -0.0024362115214204105,
+            0.012785534912074068,
+            0.016441944143888537,
+            -0.02164522371903646,
+            -0.013152102018736346,
+            0.0063548333449661805,
+            -0.016547629104776748,
+            -0.024875443975589356,
+            -0.008991095813132973,
+            -0.021885127280977718,
+            -0.023336714382596175,
+            0.00688008599534903,
+            0.016482610587564443,
+            0.0035797836729533367,
+            0.013111012464332918,
+            0.006820471349350837,
+            0.018274678506489214,
+            -0.00980248993077149,
+            -0.01582838461385605,
+            -0.0009454586621769173,
+            -0.0004906493597104733,
+            0.009787688665732045,
+            -0.01835583040075359,
+            -0.013977939077760832,
+            0.004340902272034388,
+            -0.020118484730757877,
+            0.01421736173162507,
+            -0.006428386730058143,
+            0.016756376129345597,
+            -0.0004630315831872392,
+            0.011701948921076007,
+            0.02621773278421472,
+            -0.001391567406453007,
+            0.01262026995055899,
+            0.004696906079976175,
+            -0.0030577364576645635,
+            -0.010624066224426196,
+            0.0009420498021845824,
+            -0.0044079288776429516,
+            -0.0004845606666642234,
+            0.006481113327122954,
+            0.020269778435314,
+            -0.03321232649491074,
+            -0.009425498978427855,
+            0.01246754422779043,
+            -0.001658574527323383,
+            -0.005816985985181387,
+            0.013050062223188965,
+            0.00011195028631856881,
+            0.016573827662919836,
+            -0.0017453203044868104,
+            0.01417371165305189,
+            -0.008165910519539772,
+            -0.0007461950113024975,
+            0.0016492309267666983,
+            0.005076015595439873,
+            0.005829299318055953,
+            0.012146800191035666,
+            0.01148884639506138,
+            -0.021051327863689618,
+            -0.0026357096198729156,
+            0.0036146148489310443,
+            -0.010673282438301611,
+            0.014967444393862347,
+            0.005737933867764028,
+            0.01391411712868596,
+            0.0029556189893790237,
+            0.01167806258173173,
+            -0.024440974650111174,
+            -0.01029944007931845,
+            0.01385509328958837,
+            0.00869671728965227,
+            -0.00998315356917989,
+            -0.016580053751364735,
+            -0.007061005072623325,
+            -0.00477175200214453,
+            -0.0021182980051220428,
+            -0.009871524928312118,
+            -0.0005933941177237234,
+            0.006284920058476413,
+            5.466820734762947e-05,
+            -0.012842295882002122,
+            0.015810530885294202,
+            0.004572385791133295,
+            0.003685966629168175,
+            0.005093714133656017,
+            0.011877541830396259,
+            -0.013130170365831912,
+            0.01258190257102377,
+            0.011433121961378686,
+            0.01142665981860032,
+            -0.013710025169815697,
+            -0.001774030746825879,
+            -0.01109881340948764,
+            0.0065684584345163735,
+            0.009983753894796107,
+            0.005346443477816739,
+            -0.002062387245572686,
+            0.004950044847352647,
+            0.006105199826576752,
+            -0.06391185159544624,
+            -0.02131511418723256,
+            0.0027165737784844413,
+            0.033389688791557785,
+            -0.007167593552251769,
+            -0.0037646833460996797,
+            0.002883510089742145,
+            0.01356887065340832,
+            0.009730171586767032,
+            0.00418566245984083,
+            0.020320934773473808,
+            -0.008814826860651366,
+            -0.003014193747732543,
+            0.001559573791453607,
+            0.003499420226345028,
+            -0.008703837291821111,
+            0.0044844701302276645,
+            0.0053963630238108035,
+            0.027764369548110287,
+            0.001469968540767627,
+            0.015793086297019008,
+            -0.008505290826588508,
+            -0.004075888558568453,
+            -0.009166710762668385,
+            -0.002860047211143503,
+            -0.011127820563172684,
+            0.003937110717404271,
+            0.02054186721680431,
+            -0.006126180963243227,
+            0.009346433819831456,
+            0.01057504086030076,
+            0.0001650851447316893,
+            -0.012033423322524951,
+            0.0049587335923558,
+            -0.0032240390628760535,
+            -0.005568976687946761,
+            -0.0024667040429312737,
+            -0.03474027725033016,
+            0.026855947684394643,
+            -0.004369626674503285,
+            0.009252512430957932,
+            -0.005081834388513357,
+            -0.0037896610731020337,
+            -0.014201609270756739,
+            0.017871088029615106,
+            -0.008249444723433136,
+            -0.020092257756971717,
+            0.009532994544485572,
+            -0.004991989149115383,
+            0.006464417738670472,
+            -0.02643078108339741,
+            0.0029138922065812495,
+            -0.017232410749655366,
+            -0.003999293090834831,
+            -0.006924488971625995,
+            0.01318082244178942,
+            -0.007232695119856389,
+            -0.02660711037908311,
+            0.01979251106803667,
+            -0.019485860712345603,
+            -0.011977904020170568,
+            -0.0177530973032872,
+            -0.014524003566945937,
+            0.004874952064727108,
+            -0.015326044325348929,
+            0.002047771741041311,
+            -0.009697340332342246,
+            -0.003652304998155628,
+            -0.0008878698629674222,
+            0.0017782435347417623,
+            -0.013017776298713044,
+            -0.018272697230002275,
+            -0.01353020425912225,
+            -0.00757239449706798,
+            -0.0008250000619957538,
+            0.01299142281772714,
+            -0.001679606555140765,
+            -0.007329941421303076,
+            0.0044605462460153116,
+            0.010231192277666706,
+            0.0006558644644452857,
+            0.0024231733856020414,
+            -0.010728844680699885,
+            -0.014137434553169443,
+            0.005230053181525096,
+            -0.012086468475542818,
+            -0.009459717916037003,
+            -0.0028185227600210556,
+            0.014984527039312847,
+            0.008189464334213754,
+            -0.009139240774931914,
+            -0.005639157579163322,
+            0.014325949891859377,
+            0.009020303519690971,
+            -0.013310205617445609,
+            0.01567574314514385,
+            -0.025369802629268523,
+            0.021964786021408878,
+            0.030536905527039065,
+            0.004945288795196981,
+            -0.01168303908203322,
+            0.02281128628230144,
+            -0.004072286832145419,
+            0.0028278384348551146,
+            0.01740522162288968,
+            -0.042558465579514926,
+            -0.013146562547084278,
+            0.002903743316339479,
+            -0.015297177099602701,
+            -0.010963312697387696,
+            0.008804938803398296,
+            -0.0026727734429658817,
+            0.005273238046953793,
+            0.004257613402172282,
+            0.03512246556126453,
+            0.002258612643297096,
+            -0.0002885708728599208,
+            -0.0019720190285930487,
+            0.012312613129996533,
+            0.010451003939611476,
+            -0.015556620910136185,
+            -0.02387856865043814,
+            -0.004583090523619507,
+            -0.0037828776396525835,
+            -0.0014588238692793338,
+            -0.004361834167147271,
+            -0.008159679676329573,
+            0.012243319920992457,
+            -0.017279964797769668,
+            -0.013748032758693542,
+            0.018394195628121487,
+            -0.006411113522842359,
+            0.019002611495040334,
+            -0.014970362556522149,
+            -0.0077437581301661946,
+            0.0017311115715410025,
+            0.024439691379769795,
+            -0.025013956798520212,
+            0.007359134924497539,
+            -0.0031406470488477937,
+            0.017574824561755627,
+            0.00419409250132525,
+            -0.0046803792187546656,
+            0.010556972371044426,
+            0.015229789583480183,
+            -0.004608293492221483,
+            -0.014449487985966966,
+            -0.01172850291078532,
+            0.016823024209386785,
+            -0.012304887597699588,
+            0.0011773351576246157,
+            0.002262670226244277,
+            -0.0023810578015456716,
+            0.03477119000959688,
+            -0.03442178291363675,
+            0.004062693040714608,
+            0.0029249583507755944,
+            0.003291349443702718,
+            0.006461525089528308,
+            0.006047777261425826,
+            -0.020642802018978404,
+            0.008916088381876748,
+            -0.03316762661930299,
+            -0.006367254689957291,
+            -0.005404276207844267,
+            0.0009189006423526284,
+            0.009401330264376041,
+            -0.011106443887384533,
+            -0.02259405372588916,
+            -0.006010439054323966,
+            0.0070744551115721095,
+            0.010262607009446879,
+            -0.002856616377476421,
+            0.019719451109535182,
+            0.008426509638139982,
+            0.0015877648339862691,
+            -0.010655792508751512,
+            -0.015527851374834499,
+            -0.015460974859638219,
+            -0.017847462632188386,
+            -0.02236964311096428,
+            0.011841749854626384,
+            -0.00576688153869993,
+            0.009054902130717224,
+            0.014317707346504314,
+            -0.015366657284925976,
+            0.002910211008094564,
+            0.017761685988205086,
+            0.007974806171830457,
+            0.002692237107055498,
+            0.0032405477366987947,
+            0.02685230091638845,
+            -0.0009116884421301454,
+            -0.0032228977180251673,
+            0.003158213042682977,
+            0.015872199488167846,
+            0.0004966905858799325,
+            0.019981401695749582,
+            0.015201538351780685,
+            0.005359075814444414,
+            -0.004668738193285921,
+            0.019853988551405383,
+            -0.009844394286458195,
+            0.013476144827747423,
+            -0.00411789677405284,
+            -0.00830656434595337,
+            -0.008496132520586954,
+            0.006442581751999304,
+            -0.008138786054348571,
+            0.006977337946948795,
+            0.02419128791801387,
+            -0.0005734827808485311,
+            -0.0028842142002271044,
+            0.005503033191779872,
+            -0.01696639779775455,
+            -0.00389835761601045,
+            -0.03366077608692946,
+            -0.009942613075976742,
+            -0.01708630689497001,
+            -0.02910756562213045,
+            0.026536268897199546,
+            0.00219827495662445,
+            0.02586503862171508,
+            -0.01905122645609761,
+            0.008935637454859214,
+            -0.01385806889332683,
+            0.029747093931541325,
+            0.009163959012862627,
+            0.0008611776492791918,
+            0.003598831885031635,
+            -0.02103045281939351,
+            -0.01150941825152158,
+            0.027341774725604333,
+            -0.0006700608710615124,
+            -0.012089044078073347,
+            0.022396400579038037,
+            0.017491565611764764,
+            0.0051760367056280536,
+            -0.00559536785318233,
+            0.01979262595019399,
+            0.002306380546194741,
+            0.011973863166872444,
+            -0.0028136750606972223,
+            0.012911235736067303,
+            -0.01510421118359169,
+            -0.003338602157155174,
+            0.039758808572572274,
+            0.007290642519242007,
+            -0.001802921446521437,
+            -0.013696774990705774,
+            0.003941198181491921,
+            -0.0008308293948698925,
+            0.0200351513103389,
+            -0.00014327587644797734,
+            0.028291423013030337,
+            -0.0001085859276231927,
+            -0.010723064585989723,
+            0.020443848393732245,
+            -0.017860296930396913,
+            0.04540901629084378,
+            -0.018671578423261854,
+            0.000852031295629291,
+            -0.010639853151638835,
+            -0.019017194262730474,
+            0.003861296738239369,
+            0.0010977143012920602,
+            0.0008387482016061222,
+            0.0060155816314618,
+            -0.010780239332444652,
+            0.002152214497259421,
+            0.010556338812873054,
+            -0.027636192450621728,
+            0.0010938992111792307,
+            -0.014385202899929138,
+            -0.01353192238925814,
+            -0.026667778226211785,
+            -0.00017403398236549974,
+            -0.015357674131635848,
+            -0.013956722505869091,
+            -0.01985038526848704,
+            -0.008927903879720431,
+            -0.009580214158077466,
+            -0.0020314441852913386,
+            -0.004755584878514854,
+            -0.00018388548107372635,
+            -0.002918855567842974,
+            -0.0015941852262316128,
+            -0.014230272061190995,
+            0.02133992212037431,
+            0.015283345362445234,
+            0.004642765060980717,
+            0.012829570028541471,
+            0.01708952142208596,
+            -0.004770647543823141,
+            0.006604762788847952,
+            -0.0025369836845081078,
+            0.008368000846861623,
+            0.014785031726639733,
+            0.012074738887486398,
+            0.02124518119032885,
+            -0.015459818988572564,
+            -0.004992865135357168,
+            -0.012722391036055767,
+            -0.012210202603282509,
+            -0.017850065946244383,
+            -0.004300429148015694,
+            -0.0014387298144714717,
+            -0.023128300275416423,
+            0.004468952290723408,
+            -0.0026602931688877443,
+            -0.005768852559887139,
+            0.007815212168719318,
+            0.0005862972228454734,
+            -0.002265719947629544,
+            -0.00019323737674204774,
+            -0.02735820445266249,
+            0.00024009122317215594,
+            -0.00018701993387558173,
+            -0.014663710240203917,
+            -0.00450797561624596,
+            0.010584731989847334,
+            -0.011966527748437568,
+            -0.00602367292291343,
+            0.0111155226518017,
+            0.0015565179432852256,
+            0.006062862961835896,
+            0.016952559697728433,
+            0.0005012157272316869,
+            0.011160668242334889,
+            0.014993782560797772,
+            -0.01591303023837098,
+            -0.015356412848444082,
+            -0.0236025426434855,
+            -0.004514538154596937,
+            -0.009753334017065119,
+            0.007553951535816575,
+            -0.004616595532996391,
+            -0.013023153809236325,
+            0.006040285197946957,
+            -0.0006343451398642784,
+            0.008338564073158257,
+            -0.017608246105292328,
+            -8.346168119689692e-05,
+            -0.020944870308705715,
+            0.009230123321316848,
+            -0.00531237224627929,
+            -0.00445985089300067,
+            0.01410675485522911,
+            0.02895945580213926,
+            -0.007799830351886844,
+            -0.009455128322575888,
+            -0.005663072244883114,
+            0.012817229945992764,
+            0.012474431030896143,
+            -0.004423141749110039,
+            -0.0018294512146253298,
+            -0.012817101093394574,
+            0.0004755435377918658,
+            -0.01129485262081729,
+            0.0062747111125005155,
+            0.017199374798326412,
+            -0.021965117839794063,
+            -0.01296097993683748,
+            -0.016765996091978336,
+            -0.016882855472266523,
+            0.004372970343291638,
+            0.017290565252244258,
+            0.022670788364378462,
+            -0.008456547449078897,
+            -0.019564049850942417,
+            0.006898353991539946,
+            -0.01620125883985687,
+            0.0037108638122592746,
+            0.12887455743463638,
+            -0.0028864852421660366,
+            -0.011487804449328495,
+            -0.022683334123299446,
+            -0.005467192611880829,
+            0.01210557696712196,
+            -0.01555159294496261,
+            0.06709672179195036,
+            -0.015071089552631105,
+            0.004364902817496082,
+            -0.004899872946473657,
+            0.0006310685609937571,
+            -0.017610935588191347,
+            -0.005962991715597585,
+            -0.007768819050529025,
+            -0.02688607975562596,
+            0.000883763319535686,
+            -0.009935313912803587,
+            -0.012883104064962595,
+            0.002126368813790577,
+            -0.0013697265269800346,
+            -0.016131186838419288,
+            0.01575514418037863,
+            -0.020677990768697592,
+            -0.007251917191831732,
+            -0.0029325623780992118,
+            -0.00600710212646933,
+            0.014573093764769918,
+            -0.03253336117394778,
+            0.002973725493330224,
+            -0.013601417220439602,
+            0.011905788192118131,
+            0.001542117126832946,
+            -0.010789486980776022,
+            -0.013862139885514524,
+            0.007086571810314664,
+            0.015658913897173155,
+            0.017039624764452546,
+            0.01901374401756777,
+            0.004797954569087795,
+            0.003928999581093576,
+            -0.008046722893614655,
+            0.021974371139905104,
+            -0.00697335955527197,
+            0.0008975473383295958,
+            -0.00970589295353857,
+            -0.004346869073499034,
+            0.0031279458769925416,
+            0.02303983493541673,
+            -0.012560380384977343,
+            -0.01536378830925542,
+            0.0012035987157572355,
+            0.033313215063595594,
+            -0.010845084209743817,
+            0.013587126023659577,
+            0.0029867620644300073,
+            0.020679673435988446,
+            -0.0006640787253300063,
+            0.0025071780490108306,
+            0.022554193713122565,
+            -0.018082906254599396,
+            0.018230000220500718,
+            -0.004451773302171552,
+            0.01915083025790679,
+            -0.004183521043835777,
+            0.015019320101074747,
+            -0.027014055979204272,
+            -0.005362044589023088,
+            -0.012355559881051253,
+            0.003995309003198048,
+            0.020680455787871383,
+            0.0054391755324872375,
+            0.009865676762244514,
+            -0.01645386866657314,
+            -0.01760657627715638,
+            0.022969828924328126,
+            -0.021566687194482292,
+            -0.015826424849757402,
+            0.00012804774676298237,
+            -0.015021970627042762,
+            0.006000610438452896,
+            0.010126179023323158,
+            -0.02912121211344025,
+            -0.02046150879033127,
+            0.004881850039543051,
+            -0.0023469840669773042,
+            -0.00915786433296249,
+            0.006547377770403066,
+            -0.022329063637464074,
+            -0.015702777677743352,
+            -0.005692904224111613,
+            0.01207999389934878,
+            -0.008429160038192635,
+            0.010075061963131637,
+            -0.020106712664936504,
+            0.0036787045650556907,
+            -0.002764965946892891,
+            0.010307702251974629,
+            0.004697169247387421,
+            0.011192285826515207,
+            0.0030113901179382946,
+            0.0014774681536380047,
+            -0.02778270000997313,
+            0.010043261430740039,
+            -5.178406450222847e-05,
+            0.04216636255860709,
+            0.030977096295679525,
+            0.004479857028809617,
+            -5.798020745008712e-05,
+            -0.009996471945562616,
+            0.012476009017187798,
+            -0.019306907391548302,
+            -0.005092322343599383,
+            0.01856463597985508,
+            0.02195467141594795,
+            -0.02236232264139305,
+            0.0052217593371508145,
+            -0.016766955100332014,
+            0.016133299654723637,
+            -0.011208763137897244,
+            0.006248024409576809,
+            0.011601988603146743,
+            0.0022330878390492527,
+            0.010317781892717903,
+            -0.005895142237225119,
+            0.019670196605183973,
+            0.01186677200787666,
+            -0.002339367025714548,
+            0.02367408853110795,
+            0.006858134385159451,
+            0.004479683673962342,
+            0.004798719889930319,
+            -0.009604173975576537,
+            -0.009045602998213713,
+            0.009230969488015385,
+            0.004083616606774546,
+            -0.007944628589259331,
+            0.011836323287097144,
+            0.015100350195476968,
+            0.01885317701149666,
+            -0.0011707847499298092,
+            -0.00684548433712413,
+            -0.006550059418141426,
+            0.001899758393077906,
+            -0.024577381680573596,
+            -0.02855511854789239,
+            0.021238935710272536,
+            -0.010427135883958709,
+            -0.019187769881105637,
+            -0.0015864640188683347,
+            0.016360037306061392,
+            -0.008853972319021563,
+            -0.01627167079143546,
+            0.001270414822216867,
+            0.02255755055961612,
+            -0.005959411125391464,
+            0.006009654905468478,
+            0.005909233773418581,
+            0.005874361519568704,
+            -0.0027859463941992753,
+            -0.024599789577567636,
+            0.0036612374968223453,
+            0.03221346865585554,
+            0.016461563205582293,
+            0.016139782455492073,
+            -0.007158168059808459,
+            0.001838455290310879,
+            0.007530954381521755,
+            0.005438791523412295,
+            0.015674000912615536,
+            0.008604106101542998,
+            -0.006990462920227135,
+            0.00244736628387359,
+            0.012213048255418866,
+            -0.007329299946651028,
+            0.017424711328323798,
+            0.001973882532418776,
+            -0.0133178791530314,
+            -0.008373452013951822,
+            -0.010462150707776556,
+            -0.014575483571750984,
+            0.0027958704799249897,
+            -0.011401916623878957,
+            0.0008016041012669959,
+            0.003953958717263787,
+            0.010260753041492368,
+            0.013961649189650642,
+            -0.0019275791443041316,
+            0.017019168629298462,
+            0.011534594243860177,
+            -0.0017859096307142257,
+            -0.002970739203219076,
+            -0.03142778833370656,
+            0.010207789755749909,
+            -0.014730553506704018,
+            -0.0037611416875830746,
+            0.01825790347981639,
+            -0.007766890847082858,
+            -0.0032778433418881666,
+            -0.021003405429807113,
+            0.0024512381432602736,
+            0.008768616692363321,
+            0.01558713989712387,
+            -0.012819542714293536,
+            0.009201155584074711,
+            -0.0012254337002053017,
+            -0.0057451574939563045,
+            0.00977251392462179,
+            0.0042356831792011935,
+            -0.0033433823868435786,
+            0.011499764125503874
+          ],
+          "title": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer",
+          "keyphrases": [
+            "Sparsely-Gated Mixture-of-Experts Layer",
+            "conditional computation",
+            "neural network",
+            "language modeling",
+            "machine translation"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "44767af8-7eba-44fe-a7ef-fd48aa105f77",
+        "properties": {
+          "page_content": "3 Addressing Performance Challenges 3.1 The Shrinking Batch Problem\n\nOn modern CPUs and GPUs, large batch sizes are necessary for computational efficiency, so as to amortize the overhead of parameter loads and updates. If the gating network chooses k out of n experts for each example, then for a batch of b examples, each expert receives a much smaller batch of approximately kb n b examples. This causes a naive MoE implementation to become very inefficient as the number of experts increases. The solution to this shrinking batch problem is to make the original batch size as large as possible. However, batch size tends to be limited by the memory necessary to store activations between the forwards and backwards passes. We propose the following techniques for increasing the batch size: Mixing Data Parallelism and Model Parallelism: In a conventional distributed training setting, multiple copies of the model on different devices asynchronously process distinct batches of data, and parameters are synchronized through a set of parameter servers. In our technique, these different batches run synchronously so that they can be combined for the MoE layer. We distribute the standard layers of the model and the gating network according to conventional data-parallel schemes, but keep only one shared copy of each expert. Each expert in the MoE layer receives a combined batch consisting of the relevant examples from all of the data-parallel input batches. The same set of devices function as data-parallel replicas (for the standard layers and the gating networks) and as model-parallel shards (each hosting a subset of the experts). If the model is distributed over d devices, and each device processes a batch of size b, each expert receives a batch of approximately kbd nexamples. Thus, we achieve a factor of d improvement in expert batch size.\n\nIn the case of a hierarchical MoE (Section B), the primary gating network employs data parallelism, and the secondary MoEs employ model parallelism. Each secondary MoE resides on one device.\n\nThis technique allows us to increase the number of experts (and hence the number of parameters) by proportionally increasing the number of devices in the training cluster. The total batch size increases, keeping the batch size per expert constant. The memory and bandwidth requirements per device also remain constant, as do the step times, as does the amount of time necessary to process a number of training examples equal to the number of parameters in the model. It is our goal to train a trillionparameter model on a trillion-word corpus. We have not scaled our systems this far as of the writing of this paper, but it should be possible by adding more hardware.\n\nTaking Advantage of Convolutionality: In our language models, we apply the same MoE to each time step of the previous layer. If we wait for the previous layer to finish, we can apply the MoE to all the time steps together as one big batch. Doing so increases the size of the input batch to the MoE layer by a factor of the number of unrolled time steps.\n\nIncreasing Batch Size for a Recurrent MoE: We suspect that even more powerful models may involve applying a MoE recurrently. For example, the weight matrices of a LSTM or other RNN could be replaced by a MoE. Sadly, such models break the convolutional trick from the last paragraph, since the input to the MoE at one timestep depends on the output of the MoE at the previous timestep. Gruslys et al. (2016) describe a technique for drastically reducing the number of stored activations in an unrolled RNN, at the cost of recomputing forward activations. This would allow for a large increase in batch size.\n\n3.2 Network Bandwidth\n\nAnother major performance concern in distributed computing is network bandwidth. Since the experts are stationary (see above) and the number of gating parameters is small, most of the communication involves sending the inputs and outputs of the experts across the network. To maintain computational efficiency, the ratio of an expert's computation to the size of its input and output must exceed the ratio of computational to network capacity of the computing device. For GPUs, this may be thousands to one. In our experiments, we use experts with one hidden layer containing thousands of RELU-activated units. Since the weight matrices in the expert have sizes input_size×hidden_size and hidden_size × output_size, the ratio of computation to input and output is equal to the size of the hidden layer. Conveniently, we can increase computational efficiency simply by using a larger hidden layer, or more hidden layers.\n\n",
+          "embedding": [
+            0.0014724871143698692,
+            -0.0002654724521562457,
+            -0.015004266053438187,
+            -0.010133732110261917,
+            0.02161158062517643,
+            0.014150978997349739,
+            -0.020856458693742752,
+            0.05349281430244446,
+            -0.016355933621525764,
+            0.02585536427795887,
+            0.004545832052826881,
+            -0.0036283594090491533,
+            0.015313866548240185,
+            0.006528026424348354,
+            -0.01309380866587162,
+            0.013184423558413982,
+            0.010216795839369297,
+            0.007860816083848476,
+            0.012504814192652702,
+            0.009748619981110096,
+            0.023076515644788742,
+            -0.03379924222826958,
+            -0.03869242966175079,
+            -0.013516676612198353,
+            -0.0013828164665028453,
+            0.015766939148306847,
+            0.0215813759714365,
+            0.00212189182639122,
+            -0.04113902524113655,
+            0.013191974721848965,
+            0.018107816576957703,
+            -0.0014149091439321637,
+            -0.03023506887257099,
+            -0.02502473071217537,
+            -0.05300953611731529,
+            -0.029328923672437668,
+            0.05705698952078819,
+            0.01131927315145731,
+            -0.023907151073217392,
+            0.005855968222022057,
+            0.03201715648174286,
+            0.013720559887588024,
+            -0.042709678411483765,
+            -0.008555527776479721,
+            0.0013847042573615909,
+            0.009076561778783798,
+            -0.02502473071217537,
+            0.017443308606743813,
+            0.015933066606521606,
+            -0.04283049702644348,
+            0.0038530079182237387,
+            -0.06312816590070724,
+            0.014974061399698257,
+            0.007317128591239452,
+            -0.04415951296687126,
+            0.0005852192407473922,
+            0.025674134492874146,
+            0.01730738766491413,
+            0.05485203489661217,
+            0.0018349455203860998,
+            0.036547884345054626,
+            0.0009023703169077635,
+            -0.05071396753191948,
+            0.006059851031750441,
+            0.00730957742780447,
+            -0.010163936764001846,
+            -0.05470100790262222,
+            0.03787690028548241,
+            -0.01833435334265232,
+            -0.007256718818098307,
+            -0.04116922989487648,
+            0.0024220526684075594,
+            0.0015055236872285604,
+            0.008336542174220085,
+            0.018711913377046585,
+            -0.010873750783503056,
+            0.04974741116166115,
+            0.04283049702644348,
+            0.01686941646039486,
+            0.0027278768830001354,
+            -0.016099192202091217,
+            0.009688210673630238,
+            -0.012217868119478226,
+            0.01082089263945818,
+            0.038269564509391785,
+            -0.012059291824698448,
+            -0.02571944147348404,
+            -0.0495963878929615,
+            -0.01418873481452465,
+            -0.005553919356316328,
+            -0.021234020590782166,
+            0.014196285977959633,
+            0.013894237577915192,
+            -0.005689841229468584,
+            -0.003618920221924782,
+            0.036245837807655334,
+            -0.012089497409760952,
+            -0.01176479458808899,
+            -0.0005918265669606626,
+            -0.008502669632434845,
+            0.00020046118879690766,
+            0.011228658258914948,
+            0.019451932981610298,
+            -0.000972219102550298,
+            0.02058461494743824,
+            0.00044788149534724653,
+            -0.009514532051980495,
+            0.00590127520263195,
+            0.02843788079917431,
+            0.10390473157167435,
+            0.015555505640804768,
+            -0.015072227455675602,
+            -0.012814413756132126,
+            0.004606241825968027,
+            -0.0477236844599247,
+            -0.010496190749108791,
+            -0.010481088422238827,
+            -0.038209155201911926,
+            -0.004406134597957134,
+            -0.001600857824087143,
+            0.019874799996614456,
+            0.020040927454829216,
+            -0.014906100928783417,
+            -0.0016169041628018022,
+            -0.07424356043338776,
+            0.03325555473566055,
+            -0.01044333167374134,
+            -0.01090395636856556,
+            -0.01703554391860962,
+            -0.03902468457818031,
+            0.005587900057435036,
+            0.017971893772482872,
+            -0.015223251655697823,
+            -0.05627166107296944,
+            0.007860816083848476,
+            0.004719510208815336,
+            0.055607154965400696,
+            0.05231482535600662,
+            0.014755076728761196,
+            0.0007310521323233843,
+            0.017911484465003014,
+            -0.040021445602178574,
+            -0.01766984537243843,
+            -0.005716270767152309,
+            -0.013614842668175697,
+            0.01789638213813305,
+            -0.002208730671554804,
+            0.04739143326878548,
+            -0.03530948609113693,
+            0.027259889990091324,
+            -0.02774316817522049,
+            0.017745357006788254,
+            -8.854980842443183e-05,
+            0.05470100790262222,
+            -0.012142355553805828,
+            -0.016386138275265694,
+            -0.02513044700026512,
+            0.005916377995163202,
+            -0.037937309592962265,
+            -0.011938472278416157,
+            -0.013388305902481079,
+            -0.02419409714639187,
+            -0.014332207851111889,
+            0.023121824488043785,
+            -0.001210082438774407,
+            0.038239359855651855,
+            0.04859962686896324,
+            -0.019980518147349358,
+            -0.08771492540836334,
+            0.021203814074397087,
+            0.005720046348869801,
+            0.007721118628978729,
+            0.01756412908434868,
+            0.026761509478092194,
+            0.017382899299263954,
+            0.02369571663439274,
+            0.016914723441004753,
+            0.01736779697239399,
+            -0.029344025999307632,
+            0.0517711378633976,
+            -0.023182233795523643,
+            -0.016189808025956154,
+            0.021128302440047264,
+            0.012663389556109905,
+            0.033134736120700836,
+            0.0027448670007288456,
+            0.006973547860980034,
+            0.028332162648439407,
+            0.011832755990326405,
+            0.011696834117174149,
+            0.03301391750574112,
+            0.030839167535305023,
+            0.018802527338266373,
+            -0.01302584819495678,
+            0.024405529722571373,
+            -0.019874799996614456,
+            -0.012504814192652702,
+            0.0034074862487614155,
+            0.016612675040960312,
+            -0.00446654437109828,
+            -0.014022608287632465,
+            -0.02960076741874218,
+            -0.021883424371480942,
+            -0.02241200953722,
+            0.02618761733174324,
+            0.033195145428180695,
+            0.019149884581565857,
+            0.026942739263176918,
+            0.03990062698721886,
+            -0.031111011281609535,
+            0.03476579859852791,
+            -0.009250239469110966,
+            -0.046122826635837555,
+            -0.04047451913356781,
+            -0.032168179750442505,
+            0.003192276693880558,
+            -0.011794999241828918,
+            0.039598576724529266,
+            -0.012376443482935429,
+            -0.03467518463730812,
+            -0.016809007152915,
+            0.017488617449998856,
+            0.009854337200522423,
+            -0.016068987548351288,
+            -0.005350036546587944,
+            -0.02041848748922348,
+            -0.00730957742780447,
+            -0.017715152353048325,
+            0.021928731352090836,
+            -0.020735640078783035,
+            -0.013373203575611115,
+            0.035853173583745956,
+            -0.0014498335076496005,
+            0.020327873528003693,
+            0.015238353982567787,
+            -0.016416342929005623,
+            0.021989140659570694,
+            0.015736734494566917,
+            -0.007807957474142313,
+            -0.010586804710328579,
+            -0.024284711107611656,
+            0.006648845970630646,
+            -0.05276789888739586,
+            -0.03703116253018379,
+            0.0047950223088264465,
+            0.012988091446459293,
+            -0.01902906410396099,
+            0.04277008771896362,
+            -0.003701983718201518,
+            0.0010835995199158788,
+            -0.0025844036135822535,
+            -0.05162011459469795,
+            0.005735148675739765,
+            0.0007291642832569778,
+            -0.003358403453603387,
+            0.03237961605191231,
+            0.012134804390370846,
+            -0.02399776503443718,
+            -0.004719510208815336,
+            -0.01756412908434868,
+            0.030023634433746338,
+            -0.004172047134488821,
+            -0.029253412038087845,
+            0.02963097207248211,
+            -0.023574896156787872,
+            -0.00843470823019743,
+            0.03497723489999771,
+            0.04434074088931084,
+            0.016718393191695213,
+            0.006101382430642843,
+            -0.005893724039196968,
+            -0.0357927642762661,
+            0.0015734846238046885,
+            -0.031171420589089394,
+            0.03947775810956955,
+            -0.02907218225300312,
+            0.0018104041228070855,
+            -0.008215723559260368,
+            0.02121891640126705,
+            0.035218872129917145,
+            0.015442237257957458,
+            0.009605146944522858,
+            -0.00013780969311483204,
+            0.01214990671724081,
+            -0.004715734627097845,
+            -0.018847836181521416,
+            -0.02094707265496254,
+            0.00012164537474745885,
+            -0.03198695182800293,
+            -0.015117534436285496,
+            -0.005882397294044495,
+            -0.017020441591739655,
+            -0.01464935950934887,
+            0.005670963320881128,
+            0.04319295659661293,
+            0.005935255903750658,
+            -0.016522061079740524,
+            0.012255623936653137,
+            0.030552219599485397,
+            -0.015125085599720478,
+            -0.044401150196790695,
+            -0.003303657053038478,
+            -0.0013507237890735269,
+            -0.021671989932656288,
+            -0.03639686107635498,
+            -0.015283661894500256,
+            -0.017549026757478714,
+            0.011606219224631786,
+            0.008646142669022083,
+            -0.007611625827848911,
+            0.0310203954577446,
+            0.012044189497828484,
+            0.014807934872806072,
+            -0.03974959999322891,
+            -0.03370862826704979,
+            0.011606219224631786,
+            0.025447597727179527,
+            0.01756412908434868,
+            -0.04557913914322853,
+            -0.04491463303565979,
+            0.03869242966175079,
+            0.017609436064958572,
+            -0.03325555473566055,
+            0.013146666809916496,
+            -0.027561938390135765,
+            -0.00012601091293618083,
+            0.020403385162353516,
+            0.06324898451566696,
+            -0.040927592664957047,
+            -0.008827371522784233,
+            -0.01830414868891239,
+            -0.021762605756521225,
+            -0.034161701798439026,
+            -0.0007310521323233843,
+            0.005889948457479477,
+            -0.04657590016722679,
+            -0.026399051770567894,
+            -0.009363507851958275,
+            -0.004451442044228315,
+            0.013131564483046532,
+            -0.01047353632748127,
+            -0.009627800434827805,
+            -0.012519916519522667,
+            0.03905488923192024,
+            0.0010958702769130468,
+            0.010564151220023632,
+            -0.05920153483748436,
+            0.015721632167696953,
+            0.014989163726568222,
+            -0.006520475260913372,
+            0.0011581677244976163,
+            0.01686941646039486,
+            0.023257745429873466,
+            0.03678952530026436,
+            0.006999977398663759,
+            -0.03023506887257099,
+            -0.015797143802046776,
+            0.025764750316739082,
+            -0.02585536427795887,
+            -0.004817675799131393,
+            0.009129419922828674,
+            -0.013879135251045227,
+            -0.026142310351133347,
+            0.02574964612722397,
+            -0.046787336468696594,
+            -0.0031431936658918858,
+            -0.02078094705939293,
+            -0.008714103139936924,
+            0.012572774663567543,
+            -0.00676966505125165,
+            -0.01786617748439312,
+            0.017684947699308395,
+            -0.006697928532958031,
+            0.00021202399511821568,
+            0.017881279811263084,
+            0.014928754419088364,
+            0.03259104862809181,
+            -0.0055388170294463634,
+            -0.04838819429278374,
+            0.033829446882009506,
+            -0.022396907210350037,
+            0.019074371084570885,
+            0.02105279080569744,
+            -0.015797143802046776,
+            0.013569535687565804,
+            0.035218872129917145,
+            -0.0023125598672777414,
+            0.008268581703305244,
+            -0.02061481960117817,
+            0.04684774577617645,
+            -0.022094858810305595,
+            -0.03301391750574112,
+            -0.010488639585673809,
+            0.004224905278533697,
+            -0.015857553109526634,
+            0.01720166951417923,
+            0.0022748038172721863,
+            -0.04603221267461777,
+            -0.006040973123162985,
+            0.006441187579184771,
+            -0.024813296273350716,
+            0.006543128751218319,
+            0.007698464673012495,
+            0.005440651439130306,
+            -0.010617010295391083,
+            -0.008744308724999428,
+            0.0021577600855380297,
+            -0.007981635630130768,
+            0.04956618323922157,
+            -0.03639686107635498,
+            -0.0022011795081198215,
+            -0.04603221267461777,
+            -0.01892334781587124,
+            -0.008427157066762447,
+            0.03811853751540184,
+            0.016129396855831146,
+            0.0040323492139577866,
+            0.018047407269477844,
+            -0.018349455669522285,
+            -0.006403431296348572,
+            -0.02990281581878662,
+            0.03376903757452965,
+            0.025961080566048622,
+            0.010043117217719555,
+            -0.01055660005658865,
+            0.0031262035481631756,
+            0.008049596101045609,
+            -0.006932016462087631,
+            0.0002463584241922945,
+            -0.021626682952046394,
+            0.020765844732522964,
+            -0.0029147693421691656,
+            0.019119679927825928,
+            -0.027984807267785072,
+            -0.012127253226935863,
+            -0.008464912883937359,
+            0.004938495345413685,
+            0.041984762996435165,
+            -0.03234941139817238,
+            -0.012157457880675793,
+            -0.00476481718942523,
+            0.002593842800706625,
+            -0.0003669419093057513,
+            0.0014762626960873604,
+            -0.03920591250061989,
+            -0.03754464536905289,
+            -0.0429513156414032,
+            -0.015268558636307716,
+            -0.01199888251721859,
+            -0.05337199568748474,
+            -0.07581420987844467,
+            0.00016659870743751526,
+            -0.0720083937048912,
+            -0.009174727834761143,
+            0.0407765656709671,
+            -0.015087329782545567,
+            -0.018319251015782356,
+            -0.03292330354452133,
+            -0.0076833623461425304,
+            -0.01736779697239399,
+            -0.01206684298813343,
+            0.004776144400238991,
+            0.016914723441004753,
+            0.006233528722077608,
+            -0.024571657180786133,
+            0.018636401742696762,
+            -0.027003148570656776,
+            -0.028422776609659195,
+            0.007158552762120962,
+            0.0037246374413371086,
+            -0.038964275270700455,
+            -0.029872611165046692,
+            0.013629944995045662,
+            0.012036638334393501,
+            0.046153031289577484,
+            0.025598622858524323,
+            -0.0014101896667852998,
+            -0.05908071622252464,
+            0.01919519156217575,
+            -0.001417740830220282,
+            0.02940443530678749,
+            0.015631018206477165,
+            -0.005384017247706652,
+            0.01677880249917507,
+            -0.016023680567741394,
+            -0.010511293075978756,
+            0.012542570009827614,
+            -0.006025870330631733,
+            -0.0011996994726359844,
+            -0.02926851436495781,
+            0.017186567187309265,
+            0.03049181029200554,
+            0.005806885194033384,
+            0.025568418204784393,
+            0.004036124795675278,
+            -0.012542570009827614,
+            -0.021128302440047264,
+            0.013645047321915627,
+            0.008540425449609756,
+            0.03159428760409355,
+            -0.03908509388566017,
+            -0.023756125941872597,
+            0.012361341156065464,
+            0.010481088422238827,
+            -0.016295524314045906,
+            -0.0004523650277405977,
+            0.01578204147517681,
+            0.021490762010216713,
+            -0.034222111105918884,
+            -0.03292330354452133,
+            0.008208171464502811,
+            0.028317060321569443,
+            0.019602956250309944,
+            -0.001834001624956727,
+            0.011674179695546627,
+            0.026731304824352264,
+            -0.0360041968524456,
+            0.0064713922329247,
+            0.012731350027024746,
+            0.034161701798439026,
+            1.8229107809020206e-05,
+            0.02393735572695732,
+            -0.02910238690674305,
+            0.03700095787644386,
+            -0.020660126581788063,
+            0.03201715648174286,
+            1.8627315512276255e-05,
+            0.011613770388066769,
+            0.03370862826704979,
+            -0.009733517654240131,
+            -0.026066798716783524,
+            0.004496749024838209,
+            0.005078192800283432,
+            -0.0035887155681848526,
+            -0.04228680953383446,
+            -0.014166081324219704,
+            0.01886293850839138,
+            0.02217037044465542,
+            -0.038632020354270935,
+            -0.011077634058892727,
+            0.006516699679195881,
+            -0.0002836425555869937,
+            0.0028675743378698826,
+            -0.003426364390179515,
+            -0.009831683710217476,
+            0.0036170324310660362,
+            0.034886617213487625,
+            -0.034252315759658813,
+            0.0032753399573266506,
+            0.018047407269477844,
+            -0.013199525885283947,
+            -0.037605054676532745,
+            -0.01055660005658865,
+            -0.0042098029516637325,
+            -0.012429301626980305,
+            -0.016189808025956154,
+            0.019497239962220192,
+            -0.013977301307022572,
+            0.011991331353783607,
+            -0.03361801430583,
+            -0.013305243104696274,
+            0.017820870503783226,
+            -0.03669891133904457,
+            0.009801478125154972,
+            -0.017292285338044167,
+            -0.007494581863284111,
+            0.009990259073674679,
+            0.03872263804078102,
+            -0.0028184913098812103,
+            0.015261007472872734,
+            0.000569172902032733,
+            -0.007853264920413494,
+            0.01453609112650156,
+            -0.013365652412176132,
+            -0.0004693552909884602,
+            0.005002680700272322,
+            0.031171420589089394,
+            -0.01244440395385027,
+            -0.0050215586088597775,
+            0.018153123557567596,
+            0.05237523466348648,
+            -0.031050600111484528,
+            0.007120796944946051,
+            0.014974061399698257,
+            0.028483187779784203,
+            -0.007471928372979164,
+            -0.003031813306733966,
+            -0.01955764926970005,
+            0.013267486356198788,
+            -0.01936131902039051,
+            0.0006673387251794338,
+            0.028226446360349655,
+            -0.02449614554643631,
+            -0.01902906410396099,
+            -0.0015527188079431653,
+            -0.02579495497047901,
+            0.0029411986470222473,
+            0.005127275828272104,
+            0.014468129724264145,
+            -0.030522014945745468,
+            -0.0007551216403953731,
+            0.044370945543050766,
+            0.014702217653393745,
+            -0.01147029735147953,
+            -0.008110006339848042,
+            0.0065997629426419735,
+            0.024239404127001762,
+            0.008125108666718006,
+            -0.0030771205201745033,
+            -0.010941712185740471,
+            0.0032621254213154316,
+            -0.011666628532111645,
+            0.020478898659348488,
+            0.004470319952815771,
+            0.030476707965135574,
+            0.008827371522784233,
+            -0.008351645432412624,
+            -0.004213578533381224,
+            -0.021762605756521225,
+            0.011485399678349495,
+            0.05926194414496422,
+            0.006750787142664194,
+            0.016084089875221252,
+            -0.016763700172305107,
+            0.009076561778783798,
+            0.0341314971446991,
+            -0.0019293357618153095,
+            -0.034886617213487625,
+            0.006263733841478825,
+            -0.008276132866740227,
+            -0.01889314316213131,
+            -0.012686043046414852,
+            -0.004190925043076277,
+            -0.025674134492874146,
+            -0.01329014077782631,
+            -0.007513460237532854,
+            -0.011862960644066334,
+            0.006833850406110287,
+            -0.002359755104407668,
+            -0.005278300028294325,
+            -0.029026875272393227,
+            0.024209199473261833,
+            -0.0003412205842323601,
+            0.05303974077105522,
+            0.00045637661241926253,
+            0.012210316024720669,
+            -0.012293379753828049,
+            0.0024352672044187784,
+            0.0018037967383861542,
+            0.0040323492139577866,
+            0.025734543800354004,
+            0.019134782254695892,
+            0.011794999241828918,
+            0.03684993460774422,
+            -0.012044189497828484,
+            -0.03189633786678314,
+            -0.01123620942234993,
+            -0.0020482672844082117,
+            0.03307432681322098,
+            0.021037688478827477,
+            -0.003971939906477928,
+            0.015600812621414661,
+            -0.011107838712632656,
+            -0.008812269195914268,
+            0.041018206626176834,
+            -0.008804718032479286,
+            -0.025779852643609047,
+            0.043374184519052505,
+            -0.012867271900177002,
+            0.04657590016722679,
+            0.00700375298038125,
+            -0.00858573243021965,
+            0.004292866215109825,
+            0.010133732110261917,
+            -0.003817139659076929,
+            -0.018289046362042427,
+            0.011394784785807133,
+            0.04479381442070007,
+            0.012399096973240376,
+            0.0061240363866090775,
+            0.02041848748922348,
+            0.02224588207900524,
+            -0.027622347697615623,
+            -0.023589998483657837,
+            -0.0034622326493263245,
+            -0.0025466475635766983,
+            -0.015933066606521606,
+            0.016824109479784966,
+            -0.0400516502559185,
+            -0.009393712505698204,
+            0.026942739263176918,
+            -0.0011732701677829027,
+            0.017292285338044167,
+            0.003854895941913128,
+            0.014543642289936543,
+            0.008245928213000298,
+            -0.01803230494260788,
+            0.026006387546658516,
+            0.030204864218831062,
+            0.013365652412176132,
+            -0.02519085630774498,
+            0.031080804765224457,
+            -0.0065997629426419735,
+            0.019089475274086,
+            0.01677880249917507,
+            0.005014007445424795,
+            -0.006010768003761768,
+            -0.00208035996183753,
+            -0.026127208024263382,
+            -0.006814972497522831,
+            -0.007396416272968054,
+            0.027758270502090454,
+            -0.00439103227108717,
+            0.01564612053334713,
+            0.0159783735871315,
+            -0.012708696536719799,
+            -0.0006475168047472835,
+            -0.0010156385833397508,
+            -0.031775519251823425,
+            0.025613725185394287,
+            0.007891020737588406,
+            0.006999977398663759,
+            0.03802792355418205,
+            -0.042045172303915024,
+            -0.01842496730387211,
+            -0.003711422672495246,
+            0.029147693887352943,
+            0.05134826898574829,
+            0.0031431936658918858,
+            -0.0003223425301257521,
+            0.031775519251823425,
+            -0.03479600325226784,
+            -0.032530639320611954,
+            0.011349477805197239,
+            0.03292330354452133,
+            -0.0005526546156033874,
+            -0.003741627559065819,
+            0.04379705339670181,
+            -0.010254551656544209,
+            -0.021762605756521225,
+            0.021762605756521225,
+            0.01839476265013218,
+            -0.002372969640418887,
+            -0.005648309830576181,
+            0.0003556150768417865,
+            0.014166081324219704,
+            -0.0045005246065557,
+            0.04310234263539314,
+            -0.02419409714639187,
+            0.01495140790939331,
+            -0.01603878289461136,
+            0.013501574285328388,
+            0.015049573965370655,
+            0.00010677891259547323,
+            -0.04283049702644348,
+            0.013040950521826744,
+            0.022351600229740143,
+            0.005474631674587727,
+            0.00958249345421791,
+            0.017322489991784096,
+            0.021838117390871048,
+            0.03832997381687164,
+            0.02960076741874218,
+            0.005923929158598185,
+            0.040655747056007385,
+            0.01453609112650156,
+            -0.0005847472930327058,
+            0.02784888446331024,
+            -0.0019302796572446823,
+            -0.006105158012360334,
+            -0.0011307946406304836,
+            -0.0041418420150876045,
+            0.02343897521495819,
+            -0.021883424371480942,
+            -0.015223251655697823,
+            0.02124912291765213,
+            0.0028468084055930376,
+            -0.00019196607172489166,
+            -0.03343678638339043,
+            0.006290162913501263,
+            -0.02790929563343525,
+            0.0220495518296957,
+            -0.0024560331366956234,
+            0.03364821895956993,
+            -0.023227540776133537,
+            -0.0056747389025986195,
+            0.022260984405875206,
+            -0.019482137635350227,
+            0.0033055448438972235,
+            0.01392444223165512,
+            0.008631040342152119,
+            -0.005965460557490587,
+            -0.011492950841784477,
+            -0.013569535687565804,
+            0.047089383006095886,
+            -0.005350036546587944,
+            0.018183328211307526,
+            0.018138021230697632,
+            -0.007709791883826256,
+            -0.019300907850265503,
+            0.028392571955919266,
+            -0.040716156363487244,
+            0.010662317276000977,
+            0.03201715648174286,
+            -0.06475923210382462,
+            -0.034161701798439026,
+            -0.006006992422044277,
+            -0.028996670618653297,
+            0.005489734001457691,
+            0.00033579315640963614,
+            0.008064698427915573,
+            0.010095976293087006,
+            -0.018802527338266373,
+            0.018455171957612038,
+            0.0006678106728941202,
+            0.02973668836057186,
+            -0.0031356425024569035,
+            0.0069131385535001755,
+            -0.021883424371480942,
+            -0.025930875912308693,
+            -0.01952744461596012,
+            -0.0008792447042651474,
+            -0.0036566765047609806,
+            0.00772489421069622,
+            -0.02741091512143612,
+            -0.015721632167696953,
+            0.01936131902039051,
+            -0.009861888363957405,
+            0.04026308283209801,
+            0.0405651330947876,
+            0.022381804883480072,
+            0.005980563350021839,
+            -0.025009628385305405,
+            0.03923611715435982,
+            -0.042014967650175095,
+            0.022019345313310623,
+            -0.0015904748579487205,
+            -0.017262080684304237,
+            -0.01252746768295765,
+            0.031111011281609535,
+            -0.007845713756978512,
+            0.005504836328327656,
+            0.0011883726110681891,
+            0.021234020590782166,
+            -0.02294059470295906,
+            -0.0030922230798751116,
+            -0.00885002501308918,
+            -0.011900716461241245,
+            0.015359173528850079,
+            0.0003539632598403841,
+            0.01958785392343998,
+            0.02549290470778942,
+            0.006343021523207426,
+            0.0027448670007288456,
+            -0.032107770442962646,
+            -0.031201625242829323,
+            0.026353744789958,
+            0.012308482080698013,
+            -0.028452983126044273,
+            0.013146666809916496,
+            -0.045428115874528885,
+            -0.013652598485350609,
+            -0.011213555932044983,
+            0.0075059086084365845,
+            0.013199525885283947,
+            -0.002378633012995124,
+            -0.0008320496417582035,
+            -0.02777337282896042,
+            -5.3418953029904515e-05,
+            -0.05001925677061081,
+            -0.017594333738088608,
+            0.015276109799742699,
+            -0.01102477591484785,
+            0.01866660639643669,
+            0.005478407256305218,
+            0.0107378289103508,
+            -0.01919519156217575,
+            -0.012187662534415722,
+            0.004666651599109173,
+            -0.030869372189044952,
+            0.01581224612891674,
+            -0.018440069630742073,
+            0.013433613814413548,
+            -0.013033399358391762,
+            0.03981001302599907,
+            -0.009174727834761143,
+            0.016748597845435143,
+            -0.025613725185394287,
+            -0.029887713491916656,
+            -0.02565903216600418,
+            0.02294059470295906,
+            -0.004485422279685736,
+            -0.018787425011396408,
+            0.012029087170958519,
+            -0.007875918410718441,
+            0.011719487607479095,
+            -0.02214016579091549,
+            -0.00916717667132616,
+            0.023287950083613396,
+            -0.026263128966093063,
+            0.0023842963855713606,
+            -0.04902249574661255,
+            -0.03709157183766365,
+            -0.00450807623565197,
+            0.02114340476691723,
+            -0.019300907850265503,
+            0.030688142403960228,
+            -0.0005021558608859777,
+            -0.004379705525934696,
+            -0.02399776503443718,
+            -0.028694620355963707,
+            0.01112294103950262,
+            -0.0002989809727296233,
+            -0.02880033850669861,
+            0.05781210958957672,
+            0.025009628385305405,
+            0.04008185490965843,
+            -0.04482401907444,
+            0.026867227628827095,
+            0.0066261920146644115,
+            -0.029283616691827774,
+            0.01291257981210947,
+            0.01079068798571825,
+            0.016401240602135658,
+            -0.009484327398240566,
+            -0.007468152791261673,
+            0.0034414667170494795,
+            -0.009371059015393257,
+            0.007687137927860022,
+            0.005297177936881781,
+            -0.03222858905792236,
+            0.005784231703728437,
+            0.025704339146614075,
+            -0.001376209082081914,
+            0.04014226421713829,
+            0.007196309044957161,
+            0.011568463407456875,
+            -0.004674202762544155,
+            -0.04011205956339836,
+            0.01730738766491413,
+            -0.027622347697615623,
+            -0.03277227655053139,
+            0.03887366130948067,
+            0.020992381498217583,
+            0.011711936444044113,
+            -0.02416389249265194,
+            0.00416071992367506,
+            0.014641808345913887,
+            -0.016431447118520737,
+            0.0081704156473279,
+            0.006524250842630863,
+            0.027863986790180206,
+            0.014709768816828728,
+            -0.007860816083848476,
+            0.00044740954763256013,
+            0.02303120866417885,
+            -0.01564612053334713,
+            0.0028373694512993097,
+            0.0020407161209732294,
+            0.02813583053648472,
+            0.010330063290894032,
+            -0.024586759507656097,
+            -0.017986996099352837,
+            -0.017549026757478714,
+            -0.020040927454829216,
+            0.0009608922409825027,
+            -0.013713008724153042,
+            -0.02177770808339119,
+            -0.04358562082052231,
+            -0.004798797890543938,
+            -0.019497239962220192,
+            -0.03138285502791405,
+            -0.015026920475065708,
+            -0.008125108666718006,
+            -0.02821134403347969,
+            0.011198453605175018,
+            0.001511187176220119,
+            -0.02038828283548355,
+            0.002797725610435009,
+            -0.004813900217413902,
+            0.050140075385570526,
+            -0.019542546942830086,
+            0.013259935192763805,
+            -0.0038643348962068558,
+            -0.006052299868315458,
+            -0.00875185988843441,
+            0.03168490156531334,
+            -0.026610486209392548,
+            -0.04977761581540108,
+            0.026625588536262512,
+            -0.022683853283524513,
+            0.000796181324403733,
+            0.04497504234313965,
+            -0.024254506453871727,
+            -0.01753392443060875,
+            0.02005602978169918,
+            -0.020297668874263763,
+            -0.032440025359392166,
+            0.003915305715054274,
+            -0.010647214949131012,
+            -0.016068987548351288,
+            -0.02704845555126667,
+            -0.010118629783391953,
+            0.052163802087306976,
+            0.011092736385762691,
+            0.027335401624441147,
+            0.03364821895956993,
+            0.00480634905397892,
+            -0.016552265733480453,
+            0.010881302878260612,
+            -0.015540402382612228,
+            -0.026202719658613205,
+            0.010095976293087006,
+            0.02562882751226425,
+            0.01739800162613392,
+            0.007317128591239452,
+            -0.030310582369565964,
+            -0.005980563350021839,
+            -0.03204736113548279,
+            -0.0051763588562607765,
+            0.019044166430830956,
+            -0.003532081376761198,
+            -0.031956747174263,
+            -0.011115389876067638,
+            0.025840261951088905,
+            0.021928731352090836,
+            -0.018606197088956833,
+            -0.016748597845435143,
+            -0.0028807888738811016,
+            0.0007291642832569778,
+            0.016914723441004753,
+            -0.01327503751963377,
+            -0.00476481718942523,
+            0.004330622497946024,
+            0.019814390689134598,
+            -0.013916891068220139,
+            0.0013129677390679717,
+            -0.05950358137488365,
+            -0.01620491035282612,
+            0.0211132001131773,
+            0.002933647483587265,
+            -0.0017735918518155813,
+            -0.010088425129652023,
+            0.007668260019272566,
+            -0.0031054376158863306,
+            -0.030008532106876373,
+            0.010201692581176758,
+            0.0023238868452608585,
+            0.02214016579091549,
+            -0.02602148987352848,
+            0.008842473849654198,
+            0.01686941646039486,
+            0.0015206261305138469,
+            -0.013546881265938282,
+            -0.0036113690584897995,
+            0.0343429297208786,
+            0.010465985164046288,
+            0.003277227748185396,
+            0.004440115299075842,
+            0.014543642289936543,
+            0.004224905278533697,
+            -0.0034320277627557516,
+            0.002644813386723399,
+            -0.0012959775049239397,
+            -0.006856504362076521,
+            0.004021022468805313,
+            0.006724358070641756,
+            0.02363530732691288,
+            0.024058174341917038,
+            0.005059314891695976,
+            0.0008070362382568419,
+            -0.00976372230798006,
+            -0.0057426998391747475,
+            0.01700533926486969,
+            -0.012557672336697578,
+            0.015827348455786705,
+            0.011168248951435089,
+            -0.02072053775191307,
+            -0.023892046883702278,
+            0.0018859162228181958,
+            0.001278043375350535,
+            0.01830414868891239,
+            0.02217037044465542,
+            -0.00047100710798986256,
+            0.0005309449043124914,
+            -0.017443308606743813,
+            0.00964290276169777,
+            0.007664484437555075,
+            0.03597399219870567,
+            -0.0365176796913147,
+            -0.01480038370937109,
+            0.021823015064001083,
+            -0.009673107415437698,
+            0.020342975854873657,
+            0.0003593906876631081,
+            0.006977323908358812,
+            0.0004969644360244274,
+            0.06935036927461624,
+            -0.017986996099352837,
+            -0.00620709965005517,
+            -0.029872611165046692,
+            0.009605146944522858,
+            -0.0056332070380449295,
+            -0.02052420563995838,
+            0.011198453605175018,
+            0.0328930988907814,
+            -0.015283661894500256,
+            0.042347218841314316,
+            -0.0018575991271063685,
+            0.009303098544478416,
+            -0.005417997483164072,
+            -0.019738879054784775,
+            0.013305243104696274,
+            -0.0014536090893670917,
+            0.004942270927131176,
+            0.0032847789116203785,
+            0.012436852790415287,
+            0.02069033309817314,
+            0.01020924374461174,
+            0.014498335309326649,
+            0.004172047134488821,
+            -0.02008623443543911,
+            0.01978418603539467,
+            -0.0019897455349564552,
+            0.000477614434203133,
+            -0.015993475914001465,
+            -0.013335447758436203,
+            0.0067092557437717915,
+            0.01155336108058691,
+            -0.0033546278718858957,
+            -0.0017207333585247397,
+            0.003413149621337652,
+            0.025070037692785263,
+            -0.016416342929005623,
+            0.014694666489958763,
+            -0.012346237897872925,
+            -0.012308482080698013,
+            -0.012640736065804958,
+            0.005251870956271887,
+            0.005417997483164072,
+            -0.0055388170294463634,
+            0.004927168600261211,
+            -0.006184446159750223,
+            0.011953575536608696,
+            -0.03739362210035324,
+            0.014241593889892101,
+            -0.006633743178099394,
+            -0.003290442517027259,
+            0.010201692581176758,
+            -0.012867271900177002,
+            0.007517235819250345,
+            -0.007403967436403036,
+            0.01653716340661049,
+            0.006977323908358812,
+            -0.007528562564402819,
+            0.0017282846383750439,
+            -0.014853241853415966,
+            0.004519402980804443,
+            0.014792832545936108,
+            -0.0018085162155330181,
+            0.01495140790939331,
+            -0.01972377672791481,
+            -0.01474752463400364,
+            0.0006059851148165762,
+            -0.01853068359196186,
+            0.002325774636119604,
+            0.00454205647110939,
+            -0.009710864163935184,
+            0.023952458053827286,
+            -0.0034565692767500877,
+            -0.01026210281997919,
+            -0.021928731352090836,
+            -0.015495095402002335,
+            -0.015661222860217094,
+            -0.020629921928048134,
+            0.006093831267207861,
+            -0.03307432681322098,
+            0.022623443976044655,
+            0.016235115006566048,
+            -0.01733759231865406,
+            -0.01972377672791481,
+            -0.032440025359392166,
+            0.0030676815658807755,
+            -0.006093831267207861,
+            -0.005455753766000271,
+            0.012097048573195934,
+            -0.004153168760240078,
+            0.011893165297806263,
+            0.0031299791298806667,
+            -0.010956814512610435,
+            0.011485399678349495,
+            -0.012285828590393066,
+            -0.006494045723229647,
+            -0.0018009650520980358,
+            -0.003647237317636609,
+            -0.015042022801935673,
+            -0.023741023615002632,
+            -0.03769567236304283,
+            0.020071132108569145,
+            0.013426062650978565,
+            -0.014422822743654251,
+            -0.01529876422137022,
+            -0.002776959678158164,
+            0.01223297044634819,
+            -0.013108910992741585,
+            -0.01264828722923994,
+            0.008940639905631542,
+            -0.008464912883937359,
+            0.01461915485560894,
+            -0.0014413383323699236,
+            -0.0034018228761851788,
+            0.023710818961262703,
+            0.00937861017882824,
+            0.005440651439130306,
+            -0.02479819394648075,
+            -0.012240521609783173,
+            0.01370545756071806,
+            -0.013931993395090103,
+            -0.0017226211493834853,
+            0.0219438336789608,
+            -0.00154139194637537,
+            0.000530472956597805,
+            0.011145595461130142,
+            0.0019265039591118693,
+            -0.02058461494743824,
+            0.0054028951562941074,
+            0.008283684030175209,
+            -0.009854337200522423,
+            0.046092621982097626,
+            0.014098120853304863,
+            0.01603878289461136,
+            0.0025655257049947977,
+            -0.006184446159750223,
+            0.013342998921871185,
+            0.010669868439435959,
+            0.02787908911705017,
+            -0.015094880945980549,
+            -0.007366211153566837,
+            0.013388305902481079,
+            0.04080677032470703,
+            0.00013851761468686163,
+            0.011409887112677097,
+            -0.027592143043875694,
+            -0.02164178527891636,
+            0.028815440833568573,
+            0.004787471145391464,
+            0.01302584819495678,
+            -0.011138043366372585,
+            -0.024012867361307144,
+            0.015079778619110584,
+            -0.018545785918831825,
+            -0.015011817216873169,
+            -0.028558699414134026,
+            0.009627800434827805,
+            -0.027471324428915977,
+            0.02840767428278923,
+            0.0069810994900763035,
+            -0.0006479887524619699,
+            0.02519085630774498,
+            0.000645628955680877,
+            -0.025311676785349846,
+            0.014860793016850948,
+            -0.013607291504740715,
+            0.006562006659805775,
+            0.016688188537955284,
+            -0.03092978149652481,
+            0.0014130213530734181,
+            -0.006569557823240757,
+            -0.004107861779630184,
+            -0.0016848650993779302,
+            0.02436022274196148,
+            -0.016612675040960312,
+            -0.02599128521978855,
+            -0.005682290066033602,
+            -0.006331694778054953,
+            0.002212506253272295,
+            -0.007747547701001167,
+            0.022215677425265312,
+            0.005950358230620623,
+            0.003154520643875003,
+            -0.01789638213813305,
+            -0.009287996217608452,
+            0.020297668874263763,
+            -0.020750742405653,
+            0.0014715432189404964,
+            -0.021958936005830765,
+            0.010133732110261917,
+            0.006629967596381903,
+            0.010186590254306793,
+            0.014762627892196178,
+            6.300545646809042e-05,
+            0.003545295912772417,
+            0.0007820228347554803,
+            0.0026183840818703175,
+            0.013063604012131691,
+            0.01524590514600277,
+            -0.0446125864982605,
+            -0.02257813699543476,
+            0.028679518029093742,
+            -0.016461651772260666,
+            0.00308278389275074,
+            0.008563078939914703,
+            -0.03343678638339043,
+            -0.011779896914958954,
+            -0.005614329129457474,
+            0.035520922392606735,
+            -0.01614449918270111,
+            0.0015149627579376101,
+            -0.001464935950934887,
+            -0.024224301800131798,
+            0.0026882330421358347,
+            -0.020131541416049004,
+            0.008842473849654198,
+            -0.01830414868891239,
+            -0.043434593826532364,
+            -0.0008244984201155603,
+            0.02535698376595974,
+            -0.007449274882674217,
+            0.003824691055342555,
+            0.021354839205741882,
+            -0.018138021230697632,
+            0.003739739768207073,
+            -0.019089475274086,
+            0.013780969195067883,
+            0.003877549432218075,
+            0.028196241706609726,
+            0.008736756630241871,
+            0.004685529507696629,
+            0.006380777806043625,
+            -0.00368688115850091,
+            0.030476707965135574,
+            -0.009227585978806019,
+            -0.012036638334393501,
+            0.021022586151957512,
+            0.021520966663956642,
+            0.02127932757139206,
+            0.002335213590413332,
+            0.010835994966328144,
+            0.035520922392606735,
+            -0.005210339091718197,
+            0.019874799996614456,
+            0.007815509103238583,
+            -6.365439185174182e-05,
+            0.008940639905631542,
+            -0.007698464673012495,
+            -0.03443354368209839,
+            -0.0022653648629784584,
+            -0.022865083068609238,
+            -0.01697513461112976,
+            0.011772345751523972,
+            -0.023454077541828156,
+            -0.02217037044465542,
+            -0.028679518029093742,
+            0.013660149648785591,
+            0.004783695563673973,
+            0.004749714862555265,
+            -0.021234020590782166,
+            0.02777337282896042,
+            0.024647168815135956,
+            0.015585710294544697,
+            0.017715152353048325,
+            0.012754004448652267,
+            0.030386094003915787,
+            0.008336542174220085,
+            0.02641415409743786,
+            0.0065997629426419735,
+            0.012406648136675358,
+            -0.007747547701001167,
+            0.031956747174263,
+            0.017594333738088608,
+            0.006267509423196316,
+            0.007509684655815363,
+            -0.01733759231865406,
+            -0.0032338083256036043,
+            0.0007013191934674978,
+            -0.020478898659348488,
+            -0.011039878241717815,
+            -0.006788543425500393,
+            0.032802481204271317,
+            0.02200424298644066,
+            -0.016325728967785835,
+            -0.014211388304829597,
+            -0.005191461183130741,
+            0.0060560754500329494,
+            -0.023046310991048813,
+            -0.007543664891272783,
+            -0.012285828590393066,
+            -0.0023238868452608585,
+            -0.009340854361653328,
+            0.02343897521495819,
+            0.021626682952046394,
+            0.010760483331978321,
+            0.01833435334265232,
+            0.001749050454236567,
+            0.003915305715054274,
+            -0.010760483331978321,
+            -0.007317128591239452,
+            0.011379682458937168,
+            -0.012421750463545322,
+            -0.025341881439089775,
+            0.008087352849543095,
+            -0.006256182678043842,
+            -0.020267464220523834,
+            0.009567391127347946,
+            0.018379660323262215,
+            0.001985969953238964,
+            0.01252746768295765,
+            0.017216771841049194,
+            0.0014998603146523237,
+            0.0065997629426419735,
+            0.02214016579091549,
+            -0.0028298182878643274,
+            -0.006543128751218319,
+            0.004236232023686171,
+            -0.007981635630130768,
+            0.02230629324913025,
+            0.016461651772260666,
+            -0.03207756578922272,
+            -0.01529876422137022,
+            -0.010798239149153233,
+            0.02960076741874218,
+            -0.0070754894986748695,
+            0.008653693832457066,
+            -0.03265145793557167,
+            -0.006267509423196316,
+            -0.026308437809348106,
+            0.02002582512795925,
+            -0.003362179035320878,
+            -0.006331694778054953,
+            0.004734612535685301,
+            -0.01578204147517681,
+            0.017413103953003883,
+            -0.003756730118766427,
+            -0.009287996217608452,
+            0.012429301626980305,
+            -0.004991353955119848,
+            0.01617470569908619,
+            -0.007441723719239235,
+            0.004323071334511042,
+            0.015525300055742264,
+            0.00722651369869709,
+            0.004723285790532827,
+            -0.03201715648174286,
+            -0.0019878577440977097,
+            -0.014528539963066578,
+            0.0062863873317837715,
+            -0.006701704114675522,
+            0.0009490934899076819,
+            0.024178994819521904,
+            0.027728065848350525,
+            -0.010133732110261917,
+            -0.008011840283870697,
+            0.029419537633657455,
+            0.004070105496793985,
+            0.012029087170958519,
+            -0.009333303198218346,
+            -0.013584638014435768,
+            -0.0012251847656443715,
+            -0.012504814192652702,
+            -0.028890952467918396,
+            -0.014437925070524216,
+            -0.027108866721391678,
+            0.016718393191695213,
+            -0.009310649707913399,
+            -0.02019195258617401,
+            0.0328930988907814,
+            -0.041712917387485504,
+            -0.008412054739892483,
+            -0.00329421809874475,
+            0.0015347846783697605,
+            0.012587876990437508,
+            0.003345188684761524,
+            0.0028260427061468363,
+            -0.028951361775398254,
+            -0.013984852470457554,
+            -0.007026406470686197,
+            0.004081432241946459,
+            -0.000620143604464829,
+            -0.02496432140469551,
+            0.007256718818098307,
+            -0.0038058129139244556,
+            -0.004273988306522369,
+            0.010314960964024067,
+            -0.00484788091853261,
+            -0.024481043219566345,
+            0.02602148987352848,
+            -0.007120796944946051,
+            0.02052420563995838,
+            0.010171487927436829,
+            0.016914723441004753,
+            0.010730277746915817,
+            0.03023506887257099,
+            -0.0007919337949715555,
+            -0.002229496603831649,
+            -0.004357051569968462,
+            -0.009703313000500202,
+            0.01165152620524168,
+            0.014279349707067013,
+            -0.01128906849771738,
+            -0.03706136718392372,
+            0.006018319167196751,
+            -0.02343897521495819,
+            0.019738879054784775,
+            0.0010118630016222596,
+            -0.02999342978000641,
+            -0.007037733215838671,
+            -0.0422264002263546,
+            0.02386184222996235,
+            -0.005300953518599272,
+            -0.016386138275265694,
+            0.017050646245479584,
+            -0.006577109452337027,
+            -0.005557694938033819,
+            0.027199480682611465,
+            0.005131051409989595,
+            -0.030084045603871346,
+            0.013743213377892971,
+            0.010156385600566864,
+            -0.012104599736630917,
+            0.0018141797045245767,
+            -0.0006838570116087794,
+            -0.034252315759658813,
+            0.0071170213632285595,
+            0.0008518715621903539,
+            -0.0034660082310438156,
+            -0.02058461494743824,
+            -0.019633162766695023,
+            -0.00881982035934925,
+            0.018379660323262215,
+            0.018349455669522285,
+            0.01836455799639225,
+            -0.007762650027871132,
+            0.00036127850762568414,
+            0.006799870170652866,
+            0.005331158638000488,
+            0.010911507532000542,
+            0.005501060746610165,
+            -0.00045118515845388174,
+            0.007928776554763317,
+            -0.014150978997349739,
+            0.017413103953003883,
+            -0.020901765674352646,
+            -0.003990817815065384,
+            -0.00019208405865356326,
+            -0.008555527776479721,
+            0.00019845539645757526,
+            -0.016809007152915,
+            0.011681731790304184,
+            0.016461651772260666,
+            -0.006822523660957813,
+            -0.009491878561675549,
+            -0.020705435425043106,
+            0.009997810237109661,
+            0.029525255784392357,
+            0.01020924374461174,
+            -0.041320253163576126,
+            0.021898526698350906,
+            -0.01739800162613392,
+            0.0018160674953833222,
+            -0.017941689118742943,
+            0.025961080566048622,
+            0.014332207851111889,
+            0.0035736130084842443,
+            0.018983757123351097,
+            -0.0049989051185548306,
+            0.0003516034921631217,
+            -0.00704528484493494,
+            0.001961428439244628,
+            0.00045614063856191933,
+            -0.009522083215415478,
+            -0.0040738810785114765,
+            0.02704845555126667,
+            -0.02416389249265194,
+            -0.004334398079663515,
+            0.001532896887511015,
+            0.023121824488043785,
+            -0.008261030539870262,
+            0.012535018846392632,
+            0.017911484465003014,
+            0.018017200753092766,
+            -0.002399398945271969,
+            0.011447643861174583,
+            0.07291454076766968,
+            -0.019089475274086,
+            0.017624538391828537,
+            0.019331112504005432,
+            -0.008245928213000298,
+            -0.01637103594839573,
+            -0.010851097293198109,
+            -0.01988990418612957,
+            0.02144545316696167,
+            -0.0025711890775710344,
+            0.01445302739739418,
+            -0.011296619661152363,
+            -0.009257790632545948,
+            0.007539889309555292,
+            0.0037869347725063562,
+            0.0015168505487963557,
+            0.005285851191729307,
+            0.011742141097784042,
+            0.023876944556832314,
+            0.010118629783391953,
+            -0.005165031645447016,
+            0.0012619970366358757,
+            -0.0011996994726359844,
+            -0.0023805208038538694,
+            -0.01184785831719637,
+            -0.0013554432662203908,
+            0.009386161342263222,
+            0.02340877056121826,
+            0.0019878577440977097,
+            0.035218872129917145,
+            -0.015011817216873169,
+            -0.009121868759393692,
+            0.00916717667132616,
+            0.0006692265160381794,
+            0.004606241825968027,
+            0.014724871143698692,
+            0.012837067246437073,
+            -0.011772345751523972,
+            -0.006176894996315241,
+            0.0076154014095664024,
+            0.0052934023551642895,
+            0.006384553387761116,
+            0.0107378289103508,
+            -0.01112294103950262,
+            -0.016960032284259796,
+            0.03709157183766365,
+            0.009884541854262352,
+            0.0023238868452608585,
+            0.0049309441819787025,
+            0.0009769386379048228,
+            -0.011062531732022762,
+            0.004832778126001358,
+            0.004443890880793333,
+            0.005327383056282997,
+            -0.000515842461027205,
+            -0.0009325752034783363,
+            0.003205491229891777,
+            -0.0056332070380449295,
+            0.0014847578713670373,
+            0.008049596101045609,
+            0.005319831892848015,
+            0.010911507532000542,
+            -0.008396952413022518,
+            -0.019210293889045715,
+            0.0010005361400544643,
+            -0.016657982021570206,
+            0.006679050624370575,
+            0.002169086830690503,
+            -0.0063279191963374615,
+            0.007536113727837801,
+            0.009189830161631107,
+            -0.0073473332449793816,
+            0.014332207851111889,
+            -0.001468711532652378,
+            0.001872701570391655,
+            0.0107378289103508,
+            -0.020327873528003693,
+            -0.007672035600990057,
+            0.0024767988361418247,
+            0.006894260179251432,
+            0.003726525232195854,
+            0.0018321138340979815,
+            -0.008615938015282154,
+            0.014936305582523346,
+            -0.023544691503047943,
+            -0.00908411294221878,
+            -0.0032375839073210955,
+            0.026595383882522583,
+            0.013071155175566673,
+            -0.013003193773329258,
+            0.004406134597957134,
+            0.00012789871834684163,
+            0.0066941529512405396,
+            0.0038058129139244556,
+            -0.004383481107652187,
+            0.02735050395131111,
+            -0.003715198254212737,
+            -0.024420632049441338,
+            -0.008238377049565315,
+            -0.002325774636119604,
+            0.009257790632545948,
+            0.023982662707567215,
+            0.0054028951562941074,
+            0.006497821304947138,
+            0.006988650653511286,
+            0.00502910977229476,
+            0.009892093017697334,
+            0.013629944995045662,
+            -0.003990817815065384,
+            -0.008646142669022083,
+            -0.004002144560217857,
+            -0.011402335949242115,
+            0.015283661894500256,
+            0.016386138275265694,
+            -0.012701145373284817,
+            -0.017881279811263084,
+            -0.0073020257987082005,
+            -0.01456629578024149,
+            -0.0075059086084365845,
+            0.003930408041924238,
+            -0.006048524286597967,
+            0.0011279629543423653,
+            -0.0009250239818356931,
+            -0.009597595781087875,
+            0.01324483286589384,
+            -0.012618082575500011,
+            -0.008087352849543095,
+            -0.008547976613044739,
+            -0.034947026520967484,
+            -0.024375325068831444,
+            0.005727597512304783,
+            -0.014966510236263275,
+            -0.001957652857527137,
+            0.014958959072828293,
+            -0.007358659990131855,
+            0.003715198254212737,
+            0.017624538391828537,
+            0.004987578373402357,
+            0.0060560754500329494,
+            -0.005006456281989813,
+            -0.0038114762865006924,
+            -0.010186590254306793,
+            -0.011938472278416157,
+            -0.01464935950934887,
+            0.0040059201419353485,
+            0.0012421749997884035,
+            0.011228658258914948,
+            -0.01544978842139244,
+            -0.0020444917026907206,
+            -0.003060130402445793,
+            0.019482137635350227,
+            -0.028845645487308502,
+            -0.019497239962220192,
+            0.008706551976501942,
+            -0.0004950765869580209,
+            0.01047353632748127,
+            -0.021369941532611847,
+            -0.0029732913244515657,
+            0.0017046870198100805,
+            -0.008691449649631977,
+            -0.019799288362264633,
+            0.02291039004921913,
+            -0.015125085599720478,
+            0.004534505307674408,
+            -0.008396952413022518,
+            0.027592143043875694,
+            0.012693594209849834,
+            0.013894237577915192,
+            0.0064902701415121555,
+            0.00590127520263195,
+            -0.01647675409913063,
+            -0.0026693549007177353,
+            0.0017905820859596133,
+            -0.006528026424348354,
+            0.0015810359036549926,
+            0.0075059086084365845,
+            -0.002389959990978241,
+            -0.007441723719239235,
+            0.018122918903827667,
+            0.01524590514600277,
+            0.005444427020847797,
+            0.020297668874263763,
+            0.0011619434226304293,
+            0.002886452479287982,
+            -0.00011043653648812324,
+            0.0027656329330056906,
+            0.003494325326755643,
+            0.010919058695435524,
+            0.004489197861403227,
+            -0.019874799996614456,
+            0.0013894237345084548,
+            -0.002399398945271969,
+            -0.007604074664413929,
+            -0.026988046243786812,
+            0.00120158726349473,
+            0.007709791883826256,
+            -0.001681089517660439,
+            -0.01471731998026371,
+            -0.015004266053438187,
+            0.0054972851648926735,
+            -0.001405470073223114,
+            0.013848930597305298,
+            -0.02105279080569744,
+            0.001889691804535687,
+            0.0013582749525085092,
+            -0.01925560086965561,
+            -0.005670963320881128,
+            0.02191362902522087,
+            0.019572751596570015,
+            -0.006550679914653301,
+            -0.03953816741704941,
+            -0.005980563350021839,
+            0.01301074493676424,
+            -0.0058370898477733135,
+            -0.01753392443060875,
+            0.006509148515760899,
+            -0.009053908288478851,
+            -0.006101382430642843,
+            0.018243737518787384,
+            0.0010448995744809508,
+            -0.01544978842139244,
+            0.006188221741467714,
+            0.008283684030175209,
+            0.02399776503443718,
+            0.004685529507696629,
+            -0.009439020417630672,
+            -0.005391568411141634,
+            -0.022200575098395348,
+            0.008427157066762447,
+            0.00046392783406190574,
+            -0.026625588536262512,
+            -0.002437154995277524,
+            -0.004662876017391682,
+            0.03165469691157341,
+            0.01040557585656643,
+            0.021505864337086678,
+            0.011092736385762691,
+            -0.018742118030786514,
+            -0.008593283593654633,
+            -0.008480016142129898,
+            -0.012980540283024311,
+            -0.0034225888084620237,
+            -0.0016697626560926437,
+            0.008615938015282154,
+            0.01581224612891674,
+            0.028030114248394966,
+            0.00020706850045826286,
+            0.011742141097784042,
+            0.01468711532652378,
+            -2.842029971361626e-05,
+            0.011636423878371716,
+            0.001077936147339642,
+            -0.016235115006566048,
+            0.00047147905570454895,
+            0.00476481718942523,
+            -0.008985946886241436,
+            -0.006584660615772009,
+            0.006203324068337679,
+            0.018183328211307526,
+            -0.006116485223174095,
+            -0.005081968382000923,
+            -0.010012912563979626,
+            -0.00026452852762304246,
+            -0.012225419282913208,
+            0.04503545165061951,
+            0.009673107415437698,
+            -0.033497195690870285,
+            -0.001490421243943274,
+            -0.021536068990826607,
+            0.01856089010834694,
+            -0.0037982617504894733,
+            -0.004036124795675278,
+            0.007018855307251215,
+            0.02701825089752674,
+            0.025568418204784393,
+            -0.009046357125043869,
+            -0.011168248951435089,
+            0.0007848545210435987,
+            0.011062531732022762,
+            -0.0004979083314538002,
+            -0.008706551976501942,
+            -0.0012431190116330981,
+            0.0028732377104461193,
+            -0.018711913377046585,
+            0.0030808961018919945,
+            0.008155313320457935,
+            0.00044363393681123853,
+            0.0074379476718604565,
+            -0.014551193453371525,
+            0.026172515004873276,
+            -0.0049309441819787025,
+            -0.01385648176074028,
+            -0.03772587701678276,
+            -0.01995031349360943,
+            0.020237259566783905,
+            0.0068716066889464855,
+            0.002169086830690503,
+            -0.015389378182590008,
+            -0.02011643908917904,
+            0.020433591678738594,
+            -0.011183351278305054,
+            -0.0056520854122936726,
+            -0.010934161022305489,
+            -0.010851097293198109,
+            0.0214756578207016,
+            0.007430396508425474,
+            -0.007875918410718441,
+            -0.012821964919567108,
+            0.012383994646370411,
+            -0.0005762521759606898,
+            0.006826299242675304,
+            0.02369571663439274,
+            0.026761509478092194,
+            0.01819843053817749,
+            -0.009899644181132317,
+            -0.010579253546893597,
+            -0.020509103313088417,
+            -0.003562286263331771,
+            -0.022593239322304726,
+            0.007430396508425474,
+            0.0016357821878045797,
+            0.01427179854363203,
+            0.00722651369869709,
+            -0.0016631553880870342,
+            -0.005603002384305,
+            0.009506980888545513,
+            0.0070490604266524315,
+            -0.012980540283024311,
+            0.018047407269477844,
+            -0.01108518522232771,
+            0.018772322684526443,
+            -0.0030582426115870476,
+            -0.013494023121893406,
+            0.019316010177135468,
+            -0.013750764541327953,
+            0.004409910179674625,
+            -0.004039900843054056,
+            0.008412054739892483,
+            0.00442501250654459,
+            -0.010133732110261917,
+            0.016461651772260666,
+            -0.016612675040960312,
+            0.0002716078015509993,
+            -0.010919058695435524,
+            -0.00476481718942523,
+            -0.014558744616806507,
+            -0.022094858810305595,
+            -0.044672995805740356,
+            -0.005297177936881781,
+            0.019965415820479393,
+            0.0008900996181182563,
+            0.01044333167374134,
+            -0.013826277107000351,
+            -0.0006555399741046131,
+            -0.005950358230620623,
+            -0.008525323122739792,
+            0.012383994646370411,
+            0.01842496730387211,
+            -0.009831683710217476,
+            -0.003926632460206747,
+            0.021264225244522095,
+            0.0021917405538260937,
+            -0.019965415820479393,
+            -0.004651549272239208,
+            -0.006562006659805775,
+            0.0031979400664567947,
+            0.014815486036241055,
+            -0.005376466084271669,
+            -0.023574896156787872,
+            0.0013894237345084548,
+            0.006879157852381468,
+            -0.02724478766322136,
+            -0.0013903676299378276,
+            -0.01720166951417923,
+            -0.01667308434844017,
+            0.006407206878066063,
+            -0.028679518029093742,
+            0.009809029288589954,
+            -0.022955697029829025,
+            -0.010095976293087006,
+            -0.005051763728260994,
+            -0.020796049386262894,
+            0.004610017407685518,
+            -0.008744308724999428,
+            0.031805723905563354,
+            -0.001901962561532855,
+            -0.008691449649631977,
+            0.01158356573432684,
+            0.01753392443060875,
+            0.02200424298644066,
+            0.0051574804820120335,
+            0.025583520531654358,
+            -0.004179598297923803,
+            -0.014974061399698257,
+            -0.0027561939787119627,
+            0.01611429452896118,
+            0.002474911045283079,
+            0.024919012561440468,
+            -0.0158726554363966,
+            0.00037850471562705934,
+            0.00934840552508831,
+            0.01670329086482525,
+            0.0053689144551754,
+            -0.0031733985524624586,
+            0.018409864977002144,
+            0.0007839106256142259,
+            -0.003658564295619726,
+            -0.02771296352148056,
+            -0.0016933601582422853,
+            0.0002349136193515733,
+            0.003620808245614171,
+            0.01032251212745905,
+            -0.0015857553808018565,
+            0.006399655714631081,
+            0.016688188537955284,
+            0.0010722726583480835,
+            -0.007690913509577513,
+            -0.01620491035282612,
+            0.004934719763696194,
+            -0.01955764926970005,
+            0.014702217653393745,
+            -0.004738388117402792,
+            0.015691427513957024,
+            0.002267252653837204,
+            -0.038632020354270935,
+            0.032107770442962646,
+            0.004394807852804661,
+            -0.008842473849654198,
+            0.019814390689134598,
+            -0.02990281581878662,
+            -0.018213532865047455,
+            0.0011175799882039428,
+            0.0022559259086847305,
+            0.018319251015782356,
+            -0.011772345751523972,
+            0.0018283382523804903,
+            0.003532081376761198,
+            -0.0037472909316420555,
+            -0.0039379592053592205,
+            0.009053908288478851,
+            -0.025115344673395157,
+            0.00822327472269535,
+            0.018983757123351097,
+            -0.001961428439244628,
+            0.003881325013935566,
+            0.010858648456633091,
+            -0.0067356848157942295,
+            0.0009245520341210067,
+            0.007690913509577513,
+            0.024647168815135956,
+            -0.005316056311130524,
+            0.017262080684304237,
+            -0.03159428760409355,
+            0.004145617596805096,
+            -0.016235115006566048,
+            -0.017277183011174202,
+            -0.015155291184782982,
+            0.001227072672918439,
+            0.01650695875287056,
+            0.008042044937610626,
+            -0.014369964599609375,
+            -0.00018417887622490525,
+            0.0029147693421691656,
+            0.0049724760465323925,
+            -0.0012723798863589764,
+            0.021203814074397087,
+            1.1548051588761155e-05,
+            0.024420632049441338,
+            -0.00772489421069622,
+            -0.027531733736395836,
+            -0.012089497409760952,
+            0.017881279811263084,
+            -0.00651292409747839,
+            0.010533946566283703,
+            0.01495140790939331,
+            -0.00295441341586411,
+            -0.011070082895457745,
+            -0.002305008703842759,
+            0.018681708723306656,
+            0.00208035996183753,
+            0.0003353211795911193,
+            0.009673107415437698,
+            0.009476776234805584,
+            -0.0017669845838099718,
+            0.022230779752135277,
+            -0.00790612306445837,
+            -0.018107816576957703,
+            0.001507411478087306,
+            0.009907195344567299,
+            0.01988990418612957,
+            -0.0159783735871315,
+            -0.013984852470457554,
+            0.003769944654777646,
+            0.01241419930011034,
+            -0.0033433008939027786,
+            -0.01773025467991829,
+            0.0017169577768072486,
+            0.028785236179828644,
+            0.00374540314078331,
+            -0.0084045035764575,
+            -0.0017112944042310119,
+            -0.017186567187309265,
+            -0.0002583931782282889,
+            -0.011175800114870071,
+            0.010511293075978756,
+            -0.004813900217413902,
+            0.0036226960364729166,
+            -0.017095953226089478,
+            0.014702217653393745,
+            0.013728111051023006,
+            0.021717296913266182,
+            0.02343897521495819,
+            -0.022155268117785454,
+            -0.015963271260261536,
+            0.005372690502554178,
+            0.0052443197928369045,
+            0.004515627399086952,
+            -0.005323607474565506,
+            0.0013752651866525412,
+            0.021264225244522095,
+            0.008555527776479721,
+            -0.0038152518682181835,
+            0.0107378289103508,
+            -0.011115389876067638,
+            -0.014513437636196613,
+            -0.008978395722806454,
+            -0.003983266651630402,
+            -0.018575992435216904,
+            -0.0028241549152880907,
+            0.012142355553805828,
+            0.011077634058892727,
+            -0.01480038370937109,
+            0.009295547381043434,
+            0.015917964279651642,
+            -0.004217354115098715,
+            -0.007706015836447477,
+            -0.010163936764001846,
+            -0.0008150594076141715,
+            0.030023634433746338,
+            0.00019161211093887687,
+            -0.006826299242675304,
+            0.014324656687676907,
+            -0.011930921114981174,
+            0.008208171464502811,
+            -0.02019195258617401,
+            -0.01150050200521946,
+            0.00011356321192579344,
+            -0.01032251212745905,
+            -0.016884518787264824,
+            -0.016748597845435143,
+            0.01916498690843582,
+            0.0055161635391414165,
+            -0.008449810557067394,
+            -0.035581331700086594,
+            0.025432495400309563,
+            -0.0018585430225357413,
+            0.040988001972436905,
+            0.006154241040349007,
+            0.007377538364380598,
+            -0.0005111229256726801,
+            0.015185495838522911,
+            0.011334375478327274,
+            -0.020841356366872787,
+            0.015140187926590443,
+            0.007173655088990927,
+            0.0011477848747745156,
+            0.007222738116979599,
+            -0.0025617501232773066,
+            0.0068036457523703575,
+            -0.0103753712028265,
+            -0.0005738924373872578,
+            -0.014415271580219269,
+            -0.027093762531876564,
+            -0.015412031672894955,
+            0.013222179375588894,
+            -0.00041154128848575056,
+            -0.0036283594090491533,
+            -0.018319251015782356,
+            -0.01789638213813305,
+            -0.01919519156217575,
+            -0.0075927479192614555,
+            -0.024209199473261833,
+            -0.008132659830152988,
+            -0.00958249345421791,
+            0.0043004173785448074,
+            0.015117534436285496,
+            0.010065770708024502,
+            -0.003264013212174177,
+            0.027697861194610596,
+            0.019316010177135468,
+            0.013599740341305733,
+            0.00011427114077378064,
+            -0.01158356573432684,
+            0.013063604012131691,
+            0.02665579319000244,
+            -0.0025277696549892426,
+            0.003892651991918683,
+            0.00730957742780447,
+            -0.006309041287750006,
+            0.014083018526434898,
+            -0.008336542174220085,
+            0.0031318669207394123,
+            0.009091664105653763,
+            -0.016099192202091217,
+            0.0015602700877934694,
+            0.02038828283548355,
+            0.013471369631588459,
+            0.00420225178822875,
+            0.0009920409647747874,
+            0.008985946886241436,
+            0.008110006339848042,
+            -0.006271285004913807,
+            -0.012240521609783173,
+            -0.005006456281989813,
+            0.006441187579184771,
+            0.0109719168394804,
+            0.012165009044110775,
+            -0.020765844732522964,
+            0.017322489991784096,
+            0.016265319660305977,
+            -0.013577086851000786,
+            0.012512365356087685,
+            -0.0038662226870656013,
+            0.003658564295619726,
+            0.015125085599720478,
+            0.008985946886241436,
+            -0.0002326718531548977,
+            -0.013939544558525085,
+            0.012097048573195934,
+            -0.02522106096148491,
+            0.004364602733403444,
+            -0.008729205466806889,
+            -0.0310203954577446,
+            0.012572774663567543,
+            -0.02396756038069725,
+            -0.0006475168047472835,
+            0.005704943556338549,
+            -0.013750764541327953,
+            0.001948213786818087,
+            -0.010752932168543339,
+            0.005131051409989595,
+            0.006071177776902914,
+            -0.007777752820402384,
+            -0.016718393191695213,
+            -0.00514615373685956,
+            1.4519867363560479e-05,
+            -0.000538968073669821,
+            -0.023423872888088226,
+            0.02291039004921913,
+            -0.006173118948936462,
+            -0.017986996099352837,
+            0.0004884693189524114,
+            -0.02291039004921913,
+            -0.018606197088956833,
+            -0.006078728940337896,
+            -0.005923929158598185,
+            -0.021853219717741013,
+            -0.00916717667132616,
+            0.0206450242549181,
+            -0.005478407256305218,
+            -0.012535018846392632,
+            -0.024103481322526932,
+            -0.0015527188079431653,
+            0.000700375298038125,
+            -0.008638591505587101,
+            -0.015268558636307716,
+            0.015155291184782982,
+            0.016960032284259796,
+            -0.019572751596570015,
+            -0.010662317276000977,
+            0.006414758041501045,
+            -0.0009122812771238387,
+            0.01859109476208687,
+            0.0063958801329135895,
+            -0.012361341156065464,
+            -0.011560912244021893,
+            0.014468129724264145,
+            0.00952963437885046,
+            0.021626682952046394,
+            0.006214650813490152,
+            0.01090395636856556,
+            0.0057011679746210575,
+            -0.0025787402410060167,
+            -0.009250239469110966,
+            -9.225167559634428e-06,
+            0.010987019166350365,
+            -0.005384017247706652,
+            -0.009605146944522858,
+            0.0023201112635433674,
+            -0.0031941644847393036,
+            0.00692824088037014,
+            -0.010156385600566864,
+            -0.0064223092049360275,
+            -0.027169276028871536,
+            -0.019270703196525574,
+            -0.019708674401044846,
+            -0.009091664105653763,
+            0.00814021099358797,
+            -0.008653693832457066,
+            -0.014966510236263275,
+            0.0041154129430651665,
+            0.014279349707067013,
+            -0.03543030470609665,
+            -0.009522083215415478,
+            -0.004753490444272757,
+            -0.0029355352744460106,
+            -0.036215633153915405,
+            0.019844595342874527,
+            0.025115344673395157,
+            -0.0005441594985313714,
+            -0.000711230153683573,
+            -0.0030582426115870476,
+            -0.012655838392674923,
+            -0.008978395722806454,
+            0.002072808798402548,
+            0.0020067356526851654,
+            -0.0270786602050066,
+            0.005285851191729307,
+            0.01388668641448021,
+            0.021490762010216713,
+            0.008593283593654633,
+            -0.015495095402002335,
+            0.0036510128993541002,
+            -0.024677373468875885,
+            -0.025402290746569633,
+            -0.0220495518296957,
+            -0.004557158797979355,
+            -0.006018319167196751,
+            0.004606241825968027,
+            0.0056332070380449295,
+            0.01842496730387211,
+            0.00719253346323967,
+            0.005493509583175182,
+            -0.01972377672791481,
+            0.010911507532000542,
+            0.0009377666283398867,
+            0.0005111229256726801,
+            -0.012897477485239506,
+            -0.01412077434360981,
+            0.02343897521495819,
+            -0.00041319310548715293,
+            0.014551193453371525,
+            0.006029645912349224,
+            -0.009650453925132751,
+            -0.007966533303260803,
+            -0.0012940897140651941,
+            -0.026066798716783524,
+            0.016325728967785835,
+            -0.011168248951435089,
+            0.0006970716640353203,
+            -0.015555505640804768,
+            -0.00976372230798006,
+            0.022155268117785454,
+            0.012640736065804958,
+            -0.0025881794281303883,
+            0.015933066606521606,
+            0.018349455669522285,
+            0.0008315776940435171,
+            0.005474631674587727,
+            -0.017971893772482872,
+            -0.007230289280414581,
+            0.008396952413022518,
+            -0.0035188666079193354,
+            0.008615938015282154,
+            -0.023650409653782845,
+            -0.005969236139208078,
+            0.016612675040960312,
+            0.0214756578207016,
+            -0.021430350840091705,
+            0.001019414165057242,
+            0.0068036457523703575,
+            0.009325752034783363,
+            -0.006879157852381468,
+            -0.009733517654240131,
+            -0.015902861952781677,
+            -0.00893308874219656,
+            -0.005923929158598185,
+            0.021173609420657158,
+            -0.01853068359196186,
+            -0.004341949243098497,
+            -0.0025617501232773066,
+            0.005999441258609295,
+            -0.013962198980152607,
+            -0.0039379592053592205,
+            -0.016159603372216225,
+            -0.0009269117726944387,
+            0.013871584087610245,
+            -0.008744308724999428,
+            -0.024752886965870857,
+            0.010201692581176758,
+            0.002148320898413658,
+            -0.005055539309978485,
+            -0.0012166897067800164,
+            -0.018938450142741203,
+            0.025311676785349846,
+            -0.011628872714936733,
+            0.011190902441740036,
+            -0.012950335629284382,
+            -0.0034037106670439243,
+            0.01889314316213131,
+            0.030416298657655716,
+            0.01131927315145731,
+            -0.005587900057435036,
+            -0.018908245489001274,
+            0.015419583767652512,
+            0.0063543482683598995,
+            -0.005610553547739983,
+            -0.004553383216261864,
+            0.0215813759714365,
+            -0.017745357006788254,
+            -0.03461477532982826,
+            0.012361341156065464,
+            -0.0003303656994830817,
+            -0.0306579377502203,
+            -0.010231898166239262,
+            -0.006067402195185423,
+            -0.0015272335149347782,
+            0.016235115006566048,
+            0.006603538524359465,
+            -0.004357051569968462,
+            0.001753769931383431,
+            0.0021879649721086025,
+            0.002920432947576046,
+            -0.008457361720502377,
+            -0.0013611067552119493,
+            0.0006659228820353746,
+            0.0007739996653981507,
+            0.020871561020612717,
+            -0.016235115006566048,
+            -0.004409910179674625,
+            -0.0065997629426419735,
+            0.004776144400238991,
+            0.018047407269477844,
+            0.009801478125154972,
+            0.03265145793557167,
+            0.007396416272968054,
+            0.00351697881706059,
+            -0.0040738810785114765,
+            -0.010020463727414608,
+            0.00741529418155551,
+            0.011893165297806263,
+            -0.0017131821950897574,
+            -0.01047353632748127,
+            -0.011870511807501316,
+            -0.014241593889892101,
+            0.007626728154718876,
+            0.013199525885283947,
+            0.00045307297841645777,
+            -0.008464912883937359,
+            -0.011213555932044983,
+            0.025039833039045334,
+            -0.025900671258568764,
+            0.01892334781587124,
+            -0.006724358070641756,
+            -0.002399398945271969,
+            0.023710818961262703,
+            0.015072227455675602,
+            0.015019369311630726,
+            -0.010684970766305923,
+            -0.026957841590046883,
+            0.01670329086482525,
+            -0.0034414667170494795,
+            -0.002335213590413332,
+            -0.001184597029350698,
+            0.01063966378569603,
+            -0.03259104862809181,
+            -0.012897477485239506,
+            0.012429301626980305,
+            -0.0008688617963343859,
+            0.012225419282913208,
+            -0.01611429452896118,
+            0.02363530732691288,
+            0.010979468002915382,
+            0.01697513461112976,
+            -0.012731350027024746,
+            0.00012223531666677445,
+            0.023227540776133537,
+            -0.00040918152080848813,
+            -0.010231898166239262,
+            -0.004474095534533262,
+            -0.007222738116979599,
+            -0.0006470448570325971,
+            -0.013577086851000786,
+            0.012165009044110775,
+            -0.00468930508941412,
+            0.011221107095479965,
+            0.018273942172527313,
+            0.019738879054784775,
+            0.021264225244522095,
+            0.0031998278573155403,
+            -0.0008315776940435171,
+            -0.006905586924403906,
+            0.005625655874609947,
+            -0.0019595406483858824,
+            -0.00024801024119369686,
+            -0.02582515962421894,
+            0.021369941532611847,
+            -0.008797166869044304,
+            0.009408814832568169,
+            0.004055003169924021,
+            1.6061083442764357e-05,
+            0.015721632167696953,
+            0.0012761554680764675,
+            -0.0018764771521091461,
+            0.004530729725956917,
+            -0.012368892319500446,
+            0.00458358833566308,
+            -0.011636423878371716,
+            0.010465985164046288,
+            0.018455171957612038,
+            -0.007702240254729986,
+            -0.009053908288478851,
+            0.009295547381043434,
+            0.0027146621141582727,
+            -0.006724358070641756,
+            0.0071849822998046875,
+            0.010858648456633091,
+            -0.00734355766326189,
+            -0.005274524446576834,
+            0.006418533623218536,
+            -0.01090395636856556,
+            -0.0007268045446835458,
+            0.006645070388913155,
+            -0.015661222860217094,
+            0.01961805857717991,
+            0.003146969247609377,
+            0.01816822588443756,
+            -0.009635351598262787,
+            0.008827371522784233,
+            -0.006592211779206991,
+            0.013954647816717625,
+            -0.02127932757139206,
+            -0.015585710294544697,
+            0.00866879615932703,
+            -0.008744308724999428,
+            0.017277183011174202,
+            0.008593283593654633,
+            0.011175800114870071,
+            -0.004334398079663515,
+            -0.013796071521937847,
+            0.004768592771142721,
+            -0.0010128068970516324,
+            0.0040474520064890385,
+            -0.01958785392343998,
+            0.026942739263176918,
+            0.01955764926970005,
+            0.020358078181743622,
+            -0.016491856426000595,
+            -0.016295524314045906,
+            0.004104085732251406,
+            0.02685212530195713,
+            -0.01547999307513237,
+            0.0008485679281875491,
+            0.01474752463400364,
+            -0.007577645592391491,
+            -0.004270212724804878,
+            0.02052420563995838,
+            -0.0026561403647065163,
+            -0.0376654677093029,
+            -0.013388305902481079,
+            0.00564453424885869,
+            0.003026149934157729,
+            0.003375393571332097,
+            -0.0374540314078331,
+            0.003987042233347893,
+            -0.025447597727179527,
+            0.004677978344261646,
+            -0.008268581703305244,
+            -0.015631018206477165,
+            0.0003327254671603441,
+            -0.001685808994807303,
+            0.009650453925132751,
+            0.005436875391751528,
+            0.00408520782366395,
+            -0.006392104551196098,
+            0.012210316024720669,
+            -0.006720582488924265,
+            -0.002958188997581601,
+            -0.005546368192881346,
+            0.018485376611351967,
+            0.005127275828272104,
+            0.014958959072828293,
+            0.0043872566893696785,
+            -0.01647675409913063,
+            0.00783816259354353,
+            0.010269653983414173,
+            -0.008162864483892918,
+            -0.00756631838157773,
+            0.002348428126424551,
+            -0.004610017407685518,
+            -0.004081432241946459,
+            -0.02061481960117817,
+            -0.0037888227961957455,
+            0.008291235193610191,
+            0.021958936005830765,
+            -0.0025541989598423243,
+            -0.008646142669022083,
+            -0.0029279841110110283,
+            -0.0034716716036200523,
+            0.010465985164046288,
+            -0.015419583767652512,
+            0.021792810410261154,
+            -0.002814715728163719,
+            0.011092736385762691,
+            0.010805790312588215,
+            0.00801939144730568,
+            0.0055388170294463634,
+            0.008442259393632412,
+            0.004217354115098715,
+            0.00964290276169777,
+            0.005595451220870018,
+            -0.02069033309817314,
+            -0.009038805961608887,
+            0.012429301626980305,
+            -0.004934719763696194,
+            -0.015434686094522476,
+            0.03231920674443245,
+            0.008087352849543095,
+            0.0006078729056753218,
+            0.007672035600990057,
+            0.03159428760409355,
+            0.0015848114853724837,
+            0.005772904958575964,
+            0.01123620942234993,
+            0.010828443802893162,
+            -0.0029997206293046474,
+            -0.012429301626980305,
+            -0.044038690626621246,
+            0.0008792447042651474,
+            0.0015404480509459972,
+            0.020750742405653,
+            -0.033225350081920624,
+            0.009590044617652893,
+            -0.013048501685261726,
+            0.008087352849543095,
+            -0.016990236937999725,
+            0.007830611430108547,
+            0.006331694778054953,
+            0.01822863519191742,
+            0.010609458200633526,
+            -0.011032327078282833,
+            0.008865127339959145,
+            0.0015904748579487205,
+            -0.011334375478327274,
+            0.0014083017595112324,
+            0.02810562588274479,
+            0.020599717274308205,
+            0.011168248951435089,
+            -0.0018028528429567814,
+            0.01853068359196186,
+            0.014807934872806072,
+            -0.009129419922828674,
+            -0.027093762531876564,
+            0.003180949715897441,
+            -0.0031564084347337484,
+            -0.004666651599109173,
+            0.01128906849771738,
+            -0.0043457248248159885,
+            0.006361899431794882,
+            0.01750371977686882,
+            -0.006222201976925135,
+            -0.0006158960750326514,
+            0.010360268875956535,
+            0.008117557503283024,
+            -0.0023371013812720776,
+            0.016250217333436012,
+            -0.015615914948284626,
+            0.021158507093787193,
+            -0.025900671258568764,
+            -0.0014696554280817509,
+            0.012625633738934994,
+            0.006958445534110069,
+            0.017820870503783226,
+            -0.01218011137098074,
+            -0.02519085630774498,
+            -0.00976372230798006,
+            0.01249726302921772,
+            0.012957886792719364,
+            0.00420225178822875,
+            0.009031254798173904,
+            0.01196867786347866,
+            -0.004927168600261211,
+            -0.0032451350707560778,
+            0.0012676604092121124,
+            -0.013237281702458858,
+            -0.00722651369869709,
+            -0.009741068817675114,
+            0.010065770708024502,
+            0.0007268045446835458,
+            -0.01079068798571825,
+            0.007283147890120745,
+            -0.02197403833270073,
+            -0.008351645432412624,
+            0.004602466244250536,
+            -0.0024900136049836874,
+            0.0018670381978154182,
+            -0.015313866548240185,
+            0.0074568260461091995,
+            0.00040257422369904816,
+            -0.003881325013935566,
+            -0.010828443802893162,
+            0.022260984405875206,
+            0.00450807623565197,
+            0.003953061532229185,
+            0.013071155175566673,
+            -0.00681119691580534,
+            -0.011538257822394371,
+            0.01544978842139244,
+            -0.009242688305675983,
+            0.0013658262323588133,
+            -0.005939031485468149,
+            -0.012240521609783173,
+            -0.008736756630241871,
+            -0.010488639585673809,
+            -0.032500434666872025,
+            -0.024420632049441338,
+            0.016386138275265694,
+            0.0015574382850900292,
+            0.012338686734437943,
+            0.017216771841049194,
+            -0.012165009044110775,
+            0.008132659830152988,
+            -0.03636665642261505,
+            0.00128653843421489,
+            -0.02155117131769657,
+            -0.01047353632748127,
+            0.0008839642396196723,
+            0.00810245517641306,
+            0.014143427833914757,
+            -0.008011840283870697,
+            0.00790612306445837,
+            -0.0018962991889566183,
+            0.018787425011396408,
+            0.015615914948284626,
+            0.003647237317636609,
+            0.0012893701205030084,
+            -0.007173655088990927,
+            -0.00011834171164082363,
+            0.022064654156565666,
+            -0.002271028235554695,
+            -0.017443308606743813,
+            0.0176396407186985,
+            0.022215677425265312,
+            0.0025202184915542603,
+            -0.02496432140469551,
+            0.005542592611163855,
+            0.0025277696549892426,
+            0.017020441591739655,
+            -0.0040738810785114765,
+            -0.010843546129763126,
+            -0.004610017407685518,
+            -0.0027580817695707083,
+            0.03881325200200081,
+            0.018409864977002144,
+            -0.005047988146543503,
+            -0.03231920674443245,
+            0.0021709746215492487,
+            -0.0028807888738811016,
+            1.456411337130703e-05,
+            0.01988990418612957,
+            0.02393735572695732,
+            0.023559793829917908,
+            -0.0032470228616148233,
+            0.007739996537566185,
+            -0.019980518147349358,
+            0.04331377521157265,
+            -0.007853264920413494,
+            -0.016884518787264824,
+            -0.015102432109415531,
+            -0.009046357125043869,
+            0.003919081296771765,
+            0.008532874286174774,
+            0.006611089687794447,
+            -0.008261030539870262,
+            -0.004704407416284084,
+            0.010428229346871376,
+            0.008631040342152119,
+            -0.017956791445612907,
+            0.013003193773329258,
+            -0.02393735572695732,
+            0.016718393191695213,
+            -0.01603878289461136,
+            0.00035797484451904893,
+            0.0037982617504894733,
+            -0.012202764861285686,
+            -0.029842406511306763,
+            -0.01176479458808899,
+            0.017186567187309265,
+            0.013448716141283512,
+            0.008547976613044739,
+            0.0035679496359080076,
+            0.005757802166044712,
+            -0.009544737637043,
+            -0.00016801456513348967,
+            0.01191581878811121,
+            0.027365608140826225,
+            0.01703554391860962,
+            0.0129427844658494,
+            0.025432495400309563,
+            -0.00737376231700182,
+            -0.039598576724529266,
+            -0.0035056520719081163,
+            0.00943146925419569,
+            0.007883469574153423,
+            0.002072808798402548,
+            0.002338989172130823,
+            -0.019980518147349358,
+            0.0062184263952076435,
+            -0.0003690656740218401,
+            -0.00681119691580534,
+            -0.020901765674352646,
+            0.007966533303260803,
+            -0.013214628212153912,
+            -0.0005021558608859777,
+            0.0007466264651156962,
+            0.00331120821647346,
+            0.007453050464391708,
+            0.012383994646370411,
+            0.0020916869398206472,
+            -0.0013309018686413765,
+            -0.006826299242675304,
+            -0.008079801686108112,
+            -0.016582470387220383,
+            0.01170438528060913,
+            -0.01362239383161068,
+            -0.005195236764848232,
+            -0.004077656660228968,
+            -0.003949285950511694,
+            0.002650476759299636,
+            0.02088666334748268,
+            0.0042098029516637325,
+            0.0027486425824463367,
+            0.01301074493676424,
+            0.008442259393632412,
+            0.0039115301333367825,
+            0.03476579859852791,
+            -0.01836455799639225,
+            -0.0063279191963374615,
+            -0.014150978997349739,
+            -0.01544978842139244,
+            -0.019013961777091026,
+            0.022019345313310623,
+            -0.004296641796827316,
+            -0.010133732110261917,
+            0.007200084626674652,
+            0.014679564163088799,
+            -0.013788520358502865,
+            3.648771235020831e-05,
+            0.010964365676045418,
+            -0.015117534436285496,
+            0.010896405205130577,
+            -0.009544737637043,
+            0.010609458200633526,
+            0.00810245517641306,
+            0.03956837207078934,
+            -0.009295547381043434,
+            -0.014468129724264145,
+            0.0026523645501583815,
+            0.009861888363957405,
+            0.018470274284482002,
+            -0.024677373468875885,
+            0.0013148555299267173,
+            -0.0012733237817883492,
+            -0.000611648487392813,
+            -0.00662241643294692,
+            -0.006252407096326351,
+            0.014415271580219269,
+            -0.006520475260913372,
+            -0.01055660005658865,
+            -0.024949217215180397,
+            -0.017971893772482872,
+            -0.00961269810795784,
+            0.022653648629784584,
+            0.024284711107611656,
+            -0.005399119574576616,
+            -0.005803109612315893,
+            0.0011978116817772388,
+            -0.01567632518708706,
+            0.01700533926486969,
+            0.013629944995045662,
+            -0.02535698376595974,
+            -0.0019000747706741095,
+            -0.009446571581065655,
+            -0.006365675013512373,
+            -0.014543642289936543,
+            0.00878206454217434,
+            0.05584879592061043,
+            -0.02655007503926754,
+            -0.0003390967904124409,
+            -0.01733759231865406,
+            0.00642986036837101,
+            -0.010722726583480835,
+            -0.005074417218565941,
+            -0.01647675409913063,
+            -0.01040557585656643,
+            -0.006656397134065628,
+            -0.006686601787805557,
+            -0.004311744589358568,
+            0.00869900081306696,
+            -0.007294474635273218,
+            -0.02227608673274517,
+            0.016250217333436012,
+            -0.031050600111484528,
+            0.002148320898413658,
+            0.010382922366261482,
+            -0.0017037431243807077,
+            0.019104577600955963,
+            -0.028030114248394966,
+            0.01165152620524168,
+            0.004927168600261211,
+            0.00137148960493505,
+            0.007083040662109852,
+            -0.014641808345913887,
+            -0.003188501112163067,
+            -0.01683921180665493,
+            -0.003941734787076712,
+            0.009748619981110096,
+            -0.010692521929740906,
+            0.023212438449263573,
+            -0.014483232982456684,
+            -0.011070082895457745,
+            0.015827348455786705,
+            0.013977301307022572,
+            0.014672012999653816,
+            -0.013546881265938282,
+            -0.0030412522610276937,
+            -0.010496190749108791,
+            0.016688188537955284,
+            -0.0021974039264023304,
+            -0.021883424371480942,
+            -0.014234042726457119,
+            0.02055441029369831,
+            -0.008042044937610626,
+            0.022381804883480072,
+            -0.002335213590413332,
+            -0.001269548200070858,
+            -0.008472464047372341,
+            0.005191461183130741,
+            -0.0006215595058165491,
+            -0.0025674134958535433,
+            0.017277183011174202,
+            -0.0025070037227123976,
+            0.026776611804962158,
+            -0.01498161256313324,
+            0.015585710294544697,
+            -0.029162796214222908,
+            -0.009204932488501072,
+            -0.00783816259354353,
+            0.0014545529847964644,
+            -0.00017084626597352326,
+            0.0008608386269770563,
+            0.030340787023305893,
+            -0.01291257981210947,
+            -0.02241200953722,
+            0.023741023615002632,
+            -0.015933066606521606,
+            -0.0328930988907814,
+            0.005587900057435036,
+            -0.001396975014358759,
+            0.004768592771142721,
+            0.008917986415326595,
+            -0.007083040662109852,
+            -0.025734543800354004,
+            0.012844618409872055,
+            0.01806250959634781,
+            0.0030374766793102026,
+            4.574975173454732e-05,
+            -0.015004266053438187,
+            -0.019316010177135468,
+            -0.011281517334282398,
+            0.008366747759282589,
+            -0.0040738810785114765,
+            0.020463796332478523,
+            -0.017262080684304237,
+            -0.00405122758820653,
+            -0.006361899431794882,
+            -0.008495118468999863,
+            0.011424990370869637,
+            0.028966465964913368,
+            0.010601907037198544,
+            0.004417461343109608,
+            -0.00123084825463593,
+            0.0015225139213725924,
+            0.0021728624124079943,
+            0.008064698427915573,
+            0.009325752034783363,
+            -0.006444963160902262,
+            -0.0016839212039485574,
+            -0.009287996217608452,
+            -0.0024088378995656967,
+            -0.01335810124874115,
+            0.003643461735919118,
+            -0.003515091026201844,
+            0.005678514484316111,
+            -0.01975398138165474,
+            0.01020924374461174,
+            -0.010284756310284138,
+            0.0031564084347337484,
+            -0.02088666334748268,
+            -0.002552310936152935,
+            0.017473513260483742,
+            -0.010277205146849155,
+            0.018621299415826797,
+            0.0010618898086249828,
+            0.026097003370523453,
+            0.008042044937610626,
+            -0.0030242621432989836,
+            0.003998368978500366,
+            0.0055652461014688015,
+            0.00476481718942523,
+            0.014173632487654686,
+            0.004153168760240078,
+            -0.012618082575500011,
+            0.006241080351173878,
+            0.012550121173262596,
+            -0.008532874286174774,
+            0.011674179695546627,
+            -0.0009844898013398051,
+            0.01700533926486969,
+            -0.010775585658848286,
+            0.010911507532000542,
+            0.0079363277181983,
+            0.007891020737588406,
+            -0.023378565907478333,
+            -0.0038020373322069645,
+            0.023650409653782845,
+            -0.014543642289936543,
+            0.07092101871967316,
+            -0.005893724039196968,
+            0.025508008897304535,
+            -0.008570630103349686,
+            -0.012104599736630917,
+            -0.008034493774175644,
+            -0.004323071334511042,
+            0.007898571901023388,
+            -0.010601907037198544,
+            0.027456222102046013,
+            -0.011606219224631786,
+            0.00439103227108717,
+            -0.006788543425500393,
+            0.003760505700483918,
+            0.034916821867227554,
+            0.011198453605175018,
+            0.021369941532611847,
+            0.012210316024720669,
+            0.010662317276000977,
+            0.00040918152080848813,
+            -0.004693080671131611,
+            0.010118629783391953,
+            0.006414758041501045,
+            0.012776657938957214,
+            -0.014996714890003204,
+            0.012217868119478226,
+            0.01020924374461174,
+            0.01683921180665493,
+            -0.014611602760851383,
+            -0.004719510208815336,
+            0.011266414076089859,
+            0.0012761554680764675,
+            -0.012482160702347755,
+            -0.011643975041806698,
+            0.015079778619110584,
+            0.015042022801935673,
+            0.008563078939914703,
+            0.00810245517641306,
+            -0.00987699069082737,
+            0.0012563335476443172,
+            0.008744308724999428,
+            -0.01188561413437128,
+            0.005723821930587292,
+            -0.006875382270663977,
+            -0.006346797104924917,
+            0.007951430976390839,
+            0.003971939906477928,
+            -0.002012399025261402,
+            -0.013765866868197918,
+            0.010994570329785347,
+            -0.010254551656544209,
+            -0.01567632518708706,
+            -0.012044189497828484,
+            0.000821666675619781,
+            0.012202764861285686,
+            -0.0076607088558375835,
+            -0.002223833231255412,
+            -0.012587876990437508,
+            0.0036453495267778635,
+            0.017292285338044167,
+            0.007588972337543964,
+            0.020660126581788063,
+            0.023227540776133537
+          ],
+          "title": "Addressing Performance Challenges",
+          "keyphrases": [
+            "Shrinking Batch Problem",
+            "Mixing Data Parallelism and Model Parallelism",
+            "hierarchical MoE",
+            "network bandwidth",
+            "computational efficiency"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "3ad52daa-021b-47da-947f-ed8eb5bbe052",
+      "type": "child",
+      "source": {
+        "id": "14339b1f-53f8-479b-9da7-9430b174b9ee",
+        "properties": {
+          "page_content": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer\n\nNoam Shazeer1, Azalia Mirhoseini∗†1, Krzysztof Maziarz∗2, Andy Davis1, Quoc Le1, Geoffrey Hinton1and Jeff Dean1 1Google Brain, {noam,azalia,andydavis,qvl,geoffhinton,jeff}@google.com 2Jagiellonian University, Cracow, krzysztof.maziarz@student.uj.edu.pl\n\nAbstract\n\nThe capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost.\n\n1 Introduction And Related Work 1.1 Conditional Computation\n\nExploiting scale in both training data and model size has been central to the success of deep learning. When datasets are sufficiently large, increasing the capacity (number of parameters) of neural networks can give much better prediction accuracy. This has been shown in domains such as text (Sutskever et al., 2014; Bahdanau et al., 2014; Jozefowicz et al., 2016; Wu et al., 2016), images (Krizhevsky et al., 2012; Le et al., 2012), and audio (Hinton et al., 2012; Amodei et al., 2015). For typical deep learning models, where the entire model is activated for every example, this leads to a roughly quadratic blow-up in training costs, as both the model size and the number of training examples increase. Unfortunately, the advances in computing power and distributed computation fall short of meeting such demand.\n\nVarious forms of conditional computation have been proposed as a way to increase model capacity without a proportional increase in computational costs (Davis & Arel, 2013; Bengio et al., 2013; Eigen et al., 2013; Ludovic Denoyer, 2014; Cho & Bengio, 2014; Bengio et al., 2015; Almahairi et al., 2015). In these schemes, large parts of a network are active or inactive on a per-example basis. The gating decisions may be binary or sparse and continuous, stochastic or deterministic. Various forms of reinforcement learning and back-propagation are proposed for trarining the gating decisions.\n\nWhile these ideas are promising in theory, no work to date has yet demonstrated massive improvements in model capacity, training time, or model quality. We blame this on a combination of the following challenges:\n\nModern computing devices, especially GPUs, are much faster at arithmetic than at branching. Most of the works above recognize this and propose turning on/off large chunks of the network with each gating decision.\n\nLarge batch sizes are critical for performance, as they amortize the costs of parameter transfers and updates. Conditional computation reduces the batch sizes for the conditionally active chunks of the network.\n\nNetwork bandwidth can be a bottleneck. A cluster of GPUs may have computational power thousands of times greater than the aggregate inter-device network bandwidth. To be computationally efficient, the relative computational versus network demands of an algorithm must exceed this ratio. Embedding layers, which can be seen as a form of conditional computation, are handicapped by this very problem. Since the embeddings generally need to be sent across the network, the number of (example, parameter) interactions is limited by network bandwidth instead of computational capacity.\n\nDepending on the scheme, loss terms may be necessary to achieve the desired level of sparsity per-chunk and/or per example. Bengio et al. (2015) use three such terms. These issues can affect both model quality and load-balancing.\n\nModel capacity is most critical for very large data sets. The existing literature on conditional computation deals with relatively small image recognition data sets consisting of up to 600,000 images. It is hard to imagine that the labels of these images provide a sufficient signal to adequately train a model with millions, let alone billions of parameters. In this work, we for the first time address all of the above challenges and finally realize the promise of conditional computation. We obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets.\n\n1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1). All parts of the network are trained jointly by back-propagation.\n\nWhile the introduced technique is generic, in this paper we focus on language modeling and machine translation tasks, which are known to benefit from very large models. In particular, we apply a MoE convolutionally between stacked LSTM layers (Hochreiter & Schmidhuber, 1997), as in Figure 1.\n\nThe MoE is called once for each position in the text, selecting a potentially different combination of experts at each position. The different experts tend to become highly specialized based on syntax and semantics (see Appendix E Table 9). On both language modeling and machine translation benchmarks, we improve on best published results at a fraction of the computational cost.\n\n1.3 Related Work On Mixtures Of Experts\n\nSince its introduction more than two decades ago (Jacobs et al., 1991; Jordan & Jacobs, 1994), the mixture-of-experts approach has been the subject of much research. Different types of expert architectures hae been proposed such as SVMs (Collobert et al., 2002), Gaussian Processes (Tresp, 2001; Theis & Bethge, 2015; Deisenroth & Ng, 2015), Dirichlet Processes (Shahbaba & Neal, 2009), and deep networks. Other work has focused on different expert configurations such as a hierarchical structure (Yao et al., 2009), infinite numbers of experts (Rasmussen & Ghahramani, 2002), and adding experts sequentially (Aljundi et al., 2016). Garmash & Monz (2016) suggest an ensemble model in the format of mixture of experts for machine translation. The gating network is trained on a pre-trained ensemble NMT model. The works above concern top-level mixtures of experts. The mixture of experts is the whole model. Eigen et al. (2013) introduce the idea of using multiple MoEs with their own gating networks as parts of a deep model. It is intuitive that the latter approach is more powerful, since complex problems may contain many sub-problems each requiring different experts. They also allude in their conclusion to the potential to introduce sparsity, turning MoEs into a vehicle for computational computation.\n\nOur work builds on this use of MoEs as a general purpose neural network component. While Eigen et al. (2013) uses two stacked MoEs allowing for two sets of gating decisions, our convolutional application of the MoE allows for different gating decisions at each position in the text. We also realize sparse gating and demonstrate its use as a practical way to massively increase model capacity.\n\n2 The Structure Of The Mixture-Of-Experts Layer\n\nThe Mixture-of-Experts (MoE) layer consists of a set of n \"expert networks\" E1, · · · , En, and a \"gating network\" G whose output is a sparse n-dimensional vector. Figure 1 shows an overview of the MoE module. The experts are themselves neural networks, each with their own parameters.\n\nAlthough in principle we only require that the experts accept the same sized inputs and produce the same-sized outputs, in our initial investigations in this paper, we restrict ourselves to the case where the models are feed-forward networks with identical architectures, but with separate parameters.\n\nLet us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows:\n\n$$y=\\sum_{i=1}^{n}G(x){i}E{i}(x)$$ $$(\\mathbf{l})$$ G(x)iEi(x) (1) We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example. If the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network. In the following we focus on ordinary MoEs. We provide more details on hierarchical MoEs in Appendix B. Our implementation is related to other models of conditional computation. A MoE whose experts are simple weight matrices is similar to the parameterized weight matrix proposed in (Cho & Bengio, 2014). A MoE whose experts have one hidden layer is similar to the block-wise dropout described in (Bengio et al., 2015), where the dropped-out layer is sandwiched between fully-activated layers.\n\n2.1 Gating Network\n\nSoftmax Gating: A simple choice of non-sparse gating function (Jordan & Jacobs, 1994) is to multiply the input by a trainable weight matrix Wg and then apply the Sof tmax function.\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ $\\eqref{eq:walpha}$. Gσ(x) = Sof tmax(x · Wg) (2) Noisy Top-K Gating: We add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to −∞ (which causes the corresponding gate values to equal 0). The sparsity serves to save computation, as described above. While this form of sparsity creates some theoretically scary discontinuities in the output of gating function, we have not yet observed this to be a problem in practice. The noise term helps with load balancing, as will be discussed in Appendix A. The amount of noise per component is controlled by a second trainable weight matrix Wnoise.\n\n$$G(x)=S o f t m a x(K e e p T o p K(H(x),k))$$ $$(4)$$ $$(S)$$ G(x) = Sof tmax(KeepT opK(H(x), k)) (3) $$H(x){i}=(x\\cdot W{g}){i}+S t a n d a r d N o r m a l()\\cdot S o f t p l u s((x\\cdot W{n o i s e}){i})$$ $KeepTopK(v,k){i}=\\begin{cases}v_{i}&\\text{if}v_{i}\\text{is in the top}k\\text{elements of}v.\\ -\\infty&\\text{otherwise.}\\end{cases}$ Training the Gating Network We train the gating network by simple back-propagation, along with the rest of the model. If we choose k > 1, the gate values for the top k experts have nonzero derivatives with respect to the weights of the gating network. This type of occasionally-sensitive behavior is described in (Bengio et al., 2013) with respect to noisy rectifiers. Gradients also backpropagate through the gating network to its inputs. Our method differs here from (Bengio et al., 2015) who use boolean gates and a REINFORCE-style approach to train the gating network.\n\n3 Addressing Performance Challenges 3.1 The Shrinking Batch Problem\n\nOn modern CPUs and GPUs, large batch sizes are necessary for computational efficiency, so as to amortize the overhead of parameter loads and updates. If the gating network chooses k out of n experts for each example, then for a batch of b examples, each expert receives a much smaller batch of approximately kb n b examples. This causes a naive MoE implementation to become very inefficient as the number of experts increases. The solution to this shrinking batch problem is to make the original batch size as large as possible. However, batch size tends to be limited by the memory necessary to store activations between the forwards and backwards passes. We propose the following techniques for increasing the batch size: Mixing Data Parallelism and Model Parallelism: In a conventional distributed training setting, multiple copies of the model on different devices asynchronously process distinct batches of data, and parameters are synchronized through a set of parameter servers. In our technique, these different batches run synchronously so that they can be combined for the MoE layer. We distribute the standard layers of the model and the gating network according to conventional data-parallel schemes, but keep only one shared copy of each expert. Each expert in the MoE layer receives a combined batch consisting of the relevant examples from all of the data-parallel input batches. The same set of devices function as data-parallel replicas (for the standard layers and the gating networks) and as model-parallel shards (each hosting a subset of the experts). If the model is distributed over d devices, and each device processes a batch of size b, each expert receives a batch of approximately kbd nexamples. Thus, we achieve a factor of d improvement in expert batch size.\n\nIn the case of a hierarchical MoE (Section B), the primary gating network employs data parallelism, and the secondary MoEs employ model parallelism. Each secondary MoE resides on one device.\n\nThis technique allows us to increase the number of experts (and hence the number of parameters) by proportionally increasing the number of devices in the training cluster. The total batch size increases, keeping the batch size per expert constant. The memory and bandwidth requirements per device also remain constant, as do the step times, as does the amount of time necessary to process a number of training examples equal to the number of parameters in the model. It is our goal to train a trillionparameter model on a trillion-word corpus. We have not scaled our systems this far as of the writing of this paper, but it should be possible by adding more hardware.\n\nTaking Advantage of Convolutionality: In our language models, we apply the same MoE to each time step of the previous layer. If we wait for the previous layer to finish, we can apply the MoE to all the time steps together as one big batch. Doing so increases the size of the input batch to the MoE layer by a factor of the number of unrolled time steps.\n\nIncreasing Batch Size for a Recurrent MoE: We suspect that even more powerful models may involve applying a MoE recurrently. For example, the weight matrices of a LSTM or other RNN could be replaced by a MoE. Sadly, such models break the convolutional trick from the last paragraph, since the input to the MoE at one timestep depends on the output of the MoE at the previous timestep. Gruslys et al. (2016) describe a technique for drastically reducing the number of stored activations in an unrolled RNN, at the cost of recomputing forward activations. This would allow for a large increase in batch size.\n\n3.2 Network Bandwidth\n\nAnother major performance concern in distributed computing is network bandwidth. Since the experts are stationary (see above) and the number of gating parameters is small, most of the communication involves sending the inputs and outputs of the experts across the network. To maintain computational efficiency, the ratio of an expert's computation to the size of its input and output must exceed the ratio of computational to network capacity of the computing device. For GPUs, this may be thousands to one. In our experiments, we use experts with one hidden layer containing thousands of RELU-activated units. Since the weight matrices in the expert have sizes input_size×hidden_size and hidden_size × output_size, the ratio of computation to input and output is equal to the size of the hidden layer. Conveniently, we can increase computational efficiency simply by using a larger hidden layer, or more hidden layers.\n\n4 Balancing Expert Utilization\n\nWe have observed that the gating network tends to converge to a state where it always produces large weights for the same few experts. This imbalance is self-reinforcing, as the favored experts are trained more rapidly and thus are selected even more by the gating network. Eigen et al. (2013) describe the same phenomenon, and use a hard constraint at the beginning of training to avoid this local minimum. Bengio et al. (2015) include a soft constraint on the batch-wise average of each gate.1 We take a soft constraint approach. We define the importance of an expert relative to a batch of training examples to be the batchwise sum of the gate values for that expert. We define an additional loss Limportance, which is added to the overall loss function for the model. This loss is equal to the square of the coefficient of variation of the set of importance values, multiplied by a hand-tuned scaling factor wimportance. This additional loss encourages all experts to have equal importance.\n\n$$I m p o r t a n c e(X)=\\sum_{x\\in X}G(x)$$ $$L_{importance}(X)=w_{importance}\\cdot CV(Importance(X))^{2}\\tag{7}$$ $$(6)$$\n\nWhile this loss function can ensure equal importance, experts may still receive very different numbers of examples. For example, one expert may receive a few examples with large weights, and another may receive many examples with small weights. This can cause memory and performance problems on distributed hardware. To solve this problem, we introduce a second loss function, Lload , which ensures balanced loads. Appendix A contains the definition of this function, along with experimental results.\n\n5 Experiments 5.1 1 Billion Word Language Modeling Benchmark\n\nDataset: This dataset, introduced by (Chelba et al., 2013) consists of shuffled unique sentences from news articles, totaling approximately 829 million words, with a vocabulary of 793,471 words.\n\nPrevious State-of-the-Art: The best previously published results (Jozefowicz et al., 2016) use models consisting of one or more stacked Long Short-Term Memory (LSTM) layers (Hochreiter & Schmidhuber, 1997; Gers et al., 2000). The number of parameters in the LSTM layers of these models vary from 2 million to 151 million. Quality increases greatly with parameter count, as do computational costs. Results for these models form the top line of Figure 2-right. MoE Models: Our models consist of two stacked LSTM layers with a MoE layer between them (see Figure 1). We vary the sizes of the layers and the number of experts. For full details on model architecture, training regimen, additional baselines and results, see Appendix C.\n\nLow Computation, Varied Capacity: To investigate the effects of adding capacity, we trained\n\na series of MoE models all with roughly equal computational costs: about 8 million multiply-andadds per training example per timestep in the forwards pass, excluding the softmax layer. We call this metric (ops/timestep). We trained models with flat MoEs containing 4, 32, and 256 experts, and models with hierarchical MoEs containing 256, 1024, and 4096 experts. Each expert had about 1 million parameters. For all the MoE layers, 4 experts were active per input.\n\nThe results of these models are shown in Figure 2-left. The model with 4 always-active experts performed (unsurprisingly) similarly to the computationally-matched baseline models, while the largest of the models (4096 experts) achieved an impressive 24% lower perplexity on the test set. Figure 2: Model comparison on 1-Billion-Word Language-Modeling Benchmark. On the left, we plot test perplexity as a function of model capacity for models with similar computational budgets of approximately 8-million-ops-per-timestep. On the right, we plot test perplexity as a function of computational budget. The top line represents the LSTM models from (Jozefowicz et al., 2016).\n\nThe bottom line represents 4-billion parameter MoE models with different computational budgets.\n\nVaried Computation, High Capacity: In addition to the largest model from the previous section, we trained two more MoE models with similarly high capacity (4 billion parameters), but higher computation budgets. These models had larger LSTMs, and fewer but larger and experts. Details\n\nTest Test #Parameters ops/timestep Training TFLOPS Perplexity Perplexity excluding embedding Time /GPU 10 epochs 100 epochs and softmax layers 10 epochs Best Published Results 34.7 30.6 151 million 151 million 59 hours, 32 k40s 1.09 Low-Budget MoE Model 34.1 4303 million 8.9 million 15 hours, 16 k40s 0.74 Medium-Budget MoE Model 31.3 4313 million 33.8 million 17 hours, 32 k40s 1.22 High-Budget MoE Model 28.0 4371 million 142.7 million 47 hours, 32 k40s 1.56\n\nTable 1: Summary of high-capacity MoE-augmented models with varying computational budgets, vs. best previously published results (Jozefowicz et al., 2016). Details in Appendix C. can be found in Appendix C.2. Results of these three models form the bottom line of Figure 2-right.\n\nTable 1 compares the results of these models to the best previously-published result on this dataset .\n\nEven the fastest of these models beats the best published result (when controlling for the number of training epochs), despite requiring only 6% of the computation.\n\nComputational Efficiency: We trained our models using TensorFlow (Abadi et al., 2016) on clusters containing 16-32 Tesla K40 GPUs. For each of our models, we determine computational efficiency in TFLOPS/GPU by dividing the number of floating point operations required to process one training batch by the observed step time and the number of GPUs in the cluster. The operation counts used here are higher than the ones we report in our ops/timestep numbers in that we include the backwards pass, we include the importance-sampling-based training of the softmax layer, and we count a multiply-and-add as two separate operations. For all of our MoE models, the floating point operations involved in the experts represent between 37% and 46% of the total.\n\nFor our baseline models wtih no MoE, observed computational efficiency ranged from 1.07-1.29 TFLOPS/GPU. For our low-computation MoE models, computation efficiency ranged from 0.740.90 TFLOPS/GPU, except for the 4-expert model which did not make full use of the available parallelism. Our highest-computation MoE model was more efficient at 1.56 TFLOPS/GPU, likely due to the larger matrices. These numbers represent a significant fraction of the theoretical maximum of 4.29 TFLOPS/GPU claimed by NVIDIA. Detailed results are in Appendix C, Table 7.\n\n5.2 100 BILLION WORD GOOGLE NEWS CORPUS\n\nOn the 1-billion-word corpus, adding additional capacity seems to produce diminishing returns as the number of parameters in the MoE layer exceeds 1 billion, as can be seen in Figure 2-left. We hypothesized that for a larger training set, even higher capacities would produce significant quality improvements. We constructed a similar training set consisting of shuffled unique sentences from Google's internal news corpus, totalling roughly 100 billion words. Similarly to the previous section, we tested a series of models with similar computational costs of about 8 million ops/timestep. In addition to a baseline LSTM model, we trained models augmented with MoE layers containing 32, 256, 1024, 4096, 16384, 65536, and 131072 experts. This corresponds to up to 137 billion parameters in the MoE layer. Details on architecture, training, and results are given in Appendix D.\n\nResults: Figure 3 shows test perplexity as a function of capacity after training on 10 billion words (top line) and 100 billion words (bottom line). When training over the full 100 billion words, test perplexity improves significantly up to 65536 experts (68 billion parameters), dropping 39% lower than the computationally matched baseline, but degrades at 131072 experts, possibly a result of too much sparsity. The widening gap between the two lines demonstrates (unsurprisingly) that increased model capacity helps more on larger training sets. Even at 65536 experts (99.994% layer sparsity), computational efficiency for the model stays at a respectable 0.72 TFLOPS/GPU.\n\n5.3 Machine Translation (Single Language Pair)\n\nModel Architecture: Our model was a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decreased the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We inserted MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). Each MoE layer contained up to 2048 experts each with about two million parameters, adding a total of about 8 billion parameters to the models. Further details on model architecture, testing procedure and results can be found in Appendix E.\n\nDatasets: We benchmarked our method on the WMT'14 En→Fr and En→De corpora, whose training sets have 36M sentence pairs and 5M sentence pairs, respectively. The experimental protocols were also similar to those in (Wu et al., 2016): newstest2014 was used as the test set to compare against previous work (Luong et al., 2015a; Zhou et al., 2016; Wu et al., 2016), while the combination of newstest2012 and newstest2013 was used as the development set. We also tested the same model on a Google's Production English to French data.\n\nTable 2: Results on WMT'14 En→ Fr newstest2014 (bold values represent best results). Model Test Test ops/timenstep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.69 40.35 85M 8.7B 3 days/64 k40s MoE with 2048 Experts (longer training) 2.63 40.56 85M 8.7B 6 days/64 k40s GNMT (Wu et al., 2016) 2.79 39.22 214M 278M 6 days/96 k80s GNMT+RL (Wu et al., 2016) 2.96 39.92 214M 278M 6 days/96 k80s PBMT (Durrani et al., 2014) 37.0 LSTM (6-layer) (Luong et al., 2015b) 31.5 LSTM (6-layer+PosUnk) (Luong et al., 2015b) 33.1 DeepAtt (Zhou et al., 2016) 37.7 DeepAtt+PosUnk (Zhou et al., 2016) 39.2\n\nTable 3: Results on WMT'14 En → De newstest2014 (bold values represent best results). Model Test Test ops/timestep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 4.64 26.03 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 5.25 24.91 214M 278M 1 day/96 k80s GNMT +RL (Wu et al., 2016) 8.08 24.66 214M 278M 1 day/96 k80s PBMT (Durrani et al., 2014) 20.7 DeepAtt (Zhou et al., 2016) 20.6\n\nTable 4: Results on the Google Production En→ Fr dataset (bold values represent best results). Model Eval Eval Test Test ops/timestep Total Training Perplexity BLEU Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.60 37.27 2.69 36.57 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 2.78 35.80 2.87 35.56 214M 278M 6 days/96 k80s\n\nResults: Tables 2, 3, and 4 show the results of our largest models, compared with published results. Our approach achieved BLEU scores of 40.56 and 26.03 on the WMT'14 En→Fr and En→De benchmarks. As our models did not use RL refinement, these results constitute significant gains of 1.34 and 1.12 BLEU score on top of the strong baselines in (Wu et al., 2016). The perplexity scores are also better.2 On the Google Production dataset, our model achieved 1.01 higher test BLEU score even after training for only one sixth of the time.\n\n5.4 Multilingual Machine Translation\n\nDataset: (Johnson et al., 2016) train a single GNMT (Wu et al., 2016) model on a very large combined dataset of twelve language pairs. Results are somewhat worse than those for 12 separately trained single-pair GNMT models. This is not surprising, given that the twelve models have 12 times the capacity and twelve times the aggregate training of the one model. We repeat this experiment with a single MoE-augmented model. See Appendix E for details on model architecture.\n\nWe train our model on the same dataset as (Johnson et al., 2016) and process the same number of training examples (about 3 billion sentence pairs). Our training time was shorter due to the lower computational budget of our model.\n\nResults: Results for the single-pair GNMT models, the multilingual GNMT model and the multilingual MoE model are given in Table 5. The MoE model achieves 19% lower perplexity on the dev set than the multilingual GNMT model. On BLEU score, the MoE model significantly beats the multilingual GNMT model on 11 of the 12 language pairs (by as much as 5.84 points), and even beats the monolingual GNMT models on 8 of 12 language pairs. The poor performance on English → Korean seems to be a result of severe overtraining, as for the rarer language pairs a small number of real examples were highly oversampled in the training corpus.\n\nTable 5: Multilingual Machine Translation (bold values represent best results). GNMT-Mono GNMT-Multi MoE-Multi MoE-Multi vs. GNMT-Multi Parameters 278M / model 278M 8.7B ops/timestep 212M 212M 102M training time, hardware various 21 days, 96 k20s 12 days, 64 k40s Perplexity (dev) 4.14 3.35 -19% French → English Test BLEU 36.47 34.40 37.46 +3.06 German → English Test BLEU 31.77 31.17 34.80 +3.63 Japanese → English Test BLEU 23.41 21.62 25.91 +4.29 Korean → English Test BLEU 25.42 22.87 28.71 +5.84 Portuguese → English Test BLEU 44.40 42.53 46.13 +3.60 Spanish → English Test BLEU 38.00 36.04 39.39 +3.35 English → French Test BLEU 35.37 34.00 36.59 +2.59 English → German Test BLEU 26.43 23.15 24.53 +1.38 English → Japanese Test BLEU 23.66 21.10 22.78 +1.68 English → Korean Test BLEU 19.75 18.41 16.62 -1.79 English → Portuguese Test BLEU 38.40 37.35 37.90 +0.55 English → Spanish Test BLEU 34.50 34.25 36.21 +1.96\n\n6 Conclusion\n\nThis work is the first to demonstrate major wins from conditional computation in deep networks.\n\nWe carefully identified the design considerations and challenges of conditional computing and addressed them with a combination of algorithmic and engineering solutions. While we focused on text, conditional computation may help in other domains as well, provided sufficiently large training sets. We look forward to seeing many novel implementations and applications of conditional computation in the years to come.\n\nAcknowledgments\n\nWe would like to thank all of the members of the Google Brain and Google Translate teams who helped us with this project, in particular Zhifeng Chen, Yonghui Wu, and Melvin Johnson. Thanks also to our anonymous ICLR reviewers for the helpful suggestions on making this paper better.\n\n2Reported perplexities relative to the tokenization used by both our models and GNMT.\n\nReferences\n\nMartín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Gordon Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda B. Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. CoRR, abs/1603.04467, 2016. URL http://arxiv.org/abs/1603.04467.\n\nRahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. CoRR, abs/1611.06194, 2016. URL http://arxiv.org/abs/1611.\n\n06194.\n\nA. Almahairi, N. Ballas, T. Cooijmans, Y. Zheng, H. Larochelle, and A. Courville. Dynamic Capacity Networks. ArXiv e-prints, November 2015.\n\nDario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan, Christopher Fougner, Tony Han, Awni Y. Hannun, Billy Jun, Patrick LeGresley, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595, 2015. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.\n\nEmmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297, 2015.\n\nYoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.\n\nCiprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling.\n\narXiv preprint arXiv:1312.3005, 2013.\n\nK. Cho and Y. Bengio. Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning. ArXiv e-prints, June 2014. Ronan Collobert, Samy Bengio, and Yoshua Bengio. A parallel mixture of SVMs for very large scale problems. Neural Computing, 2002.\n\nAndrew Davis and Itamar Arel. Low-rank approximations for conditional feedforward computation in deep neural networks. arXiv preprint arXiv:1312.4461, 2013. Marc Peter Deisenroth and Jun Wei Ng. Distributed Gaussian processes. In ICML, 2015.\n\nJohn Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization, 2010.\n\nNadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. Edinburgh's phrase-based machine translation systems for wmt-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation, 2014.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013. Ekaterina Garmash and Christof Monz. Ensemble learning for multi-source neural machine translation. In staff.science.uva.nl/c.monz, 2016.\n\nFelix A. Gers, Jürgen A. Schmidhuber, and Fred A. Cummins. Learning to forget: Continual prediction with lstm. Neural Computation, 2000.\n\nAudrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. Memory-efficient backpropagation through time. CoRR, abs/1606.03401, 2016. URL http://arxiv.org/ abs/1606.03401. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2015.\n\nGeoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 2012. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.\n\nSergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.\n\nRobert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computing, 1991.\n\nMelvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation.\n\nCoRR, abs/1611.04558, 2016. URL http://arxiv.org/abs/1611.04558.\n\nMichael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm.\n\nNeural Computing, 1994. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.\n\nDiederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.\n\nReinhard Kneser and Hermann. Ney. Improved backingoff for m-gram language modeling., 1995.\n\nAlex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.\n\nQuoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeffrey Dean, and Andrew Y. Ng. Building high-level features using large scale unsupervised learning. In ICML, 2012. Patrick Gallinari Ludovic Denoyer. Deep sequential neural network. arXiv preprint arXiv:1410.0510, 2014.\n\nMinh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attentionbased neural machine translation. EMNLP, 2015a.\n\nMinh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the rare word problem in neural machine translation. ACL, 2015b.\n\nCarl Edward Rasmussen and Zoubin Ghahramani. Infinite mixtures of Gaussian process experts.\n\nNIPS, 2002.\n\nHasim Sak, Andrew W Senior, and Françoise Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH, pp. 338–342, 2014.\n\nMike Schuster and Kaisuke Nakajima. Japanese and Korean voice search. ICASSP, 2012.\n\nBabak Shahbaba and Radford Neal. Nonlinear models using dirichlet process mixtures. JMLR, 2009.\n\nIlya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks.\n\nIn NIPS, 2014.\n\nLucas Theis and Matthias Bethge. Generative image modeling using spatial LSTMs. In NIPS, 2015. Volker Tresp. Mixtures of Gaussian Processes. In NIPS, 2001.\n\nYonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.\n\nBangpeng Yao, Dirk Walther, Diane Beck, and Li Fei-fei. Hierarchical mixture of classification experts uncovers interactions between brain regions. In NIPS. 2009.\n\nWojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization.\n\narXiv preprint arXiv:1409.2329, 2014.\n\nJie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with fast-forward connections for neural machine translation. arXiv preprint arXiv:1606.04199, 2016.\n\nAppendices A Load-Balancing Loss\n\nAs discussed in section 4, for load-balancing purposes, we want to define an additional loss function to encourage experts to receive roughly equal numbers of training examples. Unfortunately, the number of examples received by an expert is a discrete quantity, so it can not be used in backpropagation. Instead, we define a smooth estimator Load(X) of the number of examples assigned to each expert for a batch X of inputs. The smoothness allows us to back-propagate gradients through the estimator. This is the purpose of the noise term in the gating function. We define P(x, i) as the probability that G(x)iis nonzero, given a new random choice of noise on element i, but keeping the already-sampled choices of noise on the other elements. To compute P(x, i), we note that the G(x)iis nonzero if and only if H(x)iis greater than the k th-greatest element of H(x) excluding itself. The probability works out to be:\n\n$$P(x,i)=P r\\Big((x\\cdot W_{g}){i}+S t a n d x N o r a l()\\cdot S o f t p u s((x\\cdot W{n o i s e})_{i})$$ $$>k t h_e x c l u d i n(H(x),k,i)\\Big)$$ $$({\\boldsymbol{8}})$$\n\n$$(9)$$ $$(10)$$ (8) Where kth_excluding(v, k, i) means the kth highest component of v, excluding component i. Simplifying, we get:\n\n$$P(x,i)=\\Phi{\\Big(}{\\frac{(x\\cdot W_{g}){i}-k t h_c x c l u d i n g(H(x),k,i)}{S o f t p l u s((x\\cdot W{n o i s e})_{i})}}{\\Big)}$$\n\nWhere Φ is the CDF of the standard normal distribution.\n\n$$L o a d(X){i}=\\sum{x\\in X}P(x,i)$$\n\n$$(11)$$ P(x, i) (10) We can now define the load loss to be the square of the coefficient of variation of the load vector, multiplied by a hand-tuned scaling factor wload.\n\n$$L_{l o a d}(X)=w_{l o a d}\\cdot C V(L o a d(X))^{2}$$ Lload(X) = wload · CV (Load(X))2(11) Initial Load Imbalance: To avoid out-of-memory errors, we need to initialize the network in a state of approximately equal expert load (since the soft constraints need some time to work). To accomplish this, we initialize the matrices Wg and Wnoise to all zeros, which yields no signal and some noise.\n\nExperiments: We trained a set of models with identical architecture (the MoE-256 model described in Appendix C), using different values of wimportance and wload. We trained each model for 10 epochs, then measured perplexity on the test set. We also measured the coefficients of variation in Importance and Load, as well as ratio of the load on the most overloaded expert to the average load. This last value is significant for load balancing purposes on distributed hardware. All of these metrics were averaged over several training batches.\n\nwimportance wload Test Perplexity CV (Importance(X)) CV (Load(X)) max(Load(X)) mean(Load(X)) 0.0 0.0 39.8 3.04 3.01 17.80 0.2 0.0 35.6 0.06 0.17 1.47 0.0 0.2 35.7 0.22 0.04 1.15 0.1 0.1 35.6 0.06 0.05 1.14 0.01 0.01 35.7 0.48 0.11 1.37 1.0 1.0 35.7 0.03 0.02 1.07\n\nTable 6: Experiments with different combinations of losses.\n\n13 Results: Results are reported in Table 6. All the combinations containing at least one the two losses led to very similar model quality, where having no loss was much worse. Models with higher values of wload had lower loads on the most overloaded expert.\n\nB Hierachical Mixture Of Experts\n\nIf the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network.3If the hierarchical MoE consists of a groups of b experts each, we denote the primary gating network by Gprimary, the secondary gating networks by (G1, G2..Ga), and the expert networks by (E0,0, E0,1..Ea,b). The output of the MoE is given by:\n\n$$y_{H}=\\sum_{i=1}^{a}\\sum_{j=1}^{b}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}\\cdot E{i,j}(x)$$ $$(12)$$ $$(13)$$ $$(14)$$\n\nOur metrics of expert utilization change to the following:\n\n$$I m p o r t a n c e_{H}(X){i,j}=\\sum{x\\in X}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}$$ $$L o a d{H}(X){i,j}={\\frac{L o a d{p r i m a r y}(X){i}\\cdot L o a d{i}(X^{(i)})_{j}}{|X^{(i)}|}}$$\n\nLoadprimary and Loadi deonte the Load functions for the primary gating network and i th secondary gating network respectively. X(i) denotes the subset of X for which Gprimary(x)i > 0.\n\nIt would seem simpler to let LoadH(X)i,j = Loadi(Xi)j , but this would not have a gradient with respect to the primary gating network, so we use the formulation above.\n\nC 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS C.1 8-MILLION-OPERATIONS-PER-TIMESTEP MODELS Model Architecture: Our model consists of five layers: a word embedding layer, a recurrent Long Short-Term Memory (LSTM) layer (Hochreiter & Schmidhuber, 1997; Gers et al., 2000), a MoE layer, a second LSTM layer, and a softmax layer. The dimensionality of the embedding layer, the number of units in each LSTM layer, and the input and output dimensionality of the MoE layer are all equal to 512. For every layer other than the softmax, we apply drouput (Zaremba et al., 2014) to the layer output, dropping each activation with probability DropP rob, otherwise dividing by (1 − DropP rob). After dropout, the output of the previous layer is added to the layer output.\n\nThis residual connection encourages gradient flow (He et al., 2015).\n\nMoE Layer Architecture: Each expert in the MoE layer is a feed forward network with one ReLU-activated hidden layer of size 1024 and an output layer of size 512. Thus, each expert contains [512 ∗ 1024] + [1024 ∗ 512] = 1M parameters. The output of the MoE layer is passed through a sigmoid function before dropout. We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts.\n\nWe call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers. Thus, each example is processed by exactly 4 experts for a total of 4M ops/timestep. The two LSTM layers contribute 2M ops/timestep each for the desired total of 8M.\n\n3 We have not found the need for deeper hierarchies.\n\nComputationally-Matched Baselines: The MoE-4 model does not employ sparsity, since all 4 experts are always used. In addition, we trained four more computationally-matched baseline models with no sparsity:\n\nMoE-1-Wide: The MoE layer consists of a single \"expert\" containing one ReLU-activated hidden layer of size 4096.\n\nMoE-1-Deep: The MoE layer consists of a single \"expert\" containing four ReLU-activated hidden layers, each with size 1024.\n\n4xLSTM-512: We replace the MoE layer with two additional 512-unit LSTM layers.\n\nLSTM-2048-512: The model contains one 2048-unit LSTM layer (and no MoE). The output of the LSTM is projected down to 512 dimensions (Sak et al., 2014). The next timestep of the LSTM receives the projected output. This is identical to one of the models published in (Jozefowicz et al., 2016). We re-ran it to account for differences in training regimen, and obtained results very similar to the published ones. Training: The models were trained on a cluster of 16 K40 GPUs using the synchronous method described in Section 3. Each batch consisted of a set of sentences totaling roughly 300,000 words. In the interest of time, we limited training to 10 epochs, (27,000 steps). Training took 12-16 hours for all models, except for MoE-4, which took 18 hours (since all the expert computation was performed on only 4 of 16 GPUs). We used the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 1000 training steps, and decreased after that so as to be proportional to the inverse square root of the step number. The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016). For each model, we performed a hyper-parmeter search to find the best dropout probability, in increments of 0.1.\n\nTo ensure balanced expert utilization we set wimportance = 0.1 and wload = 0.1, as described in Section 4 and Appendix A.\n\nResults: We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol. Results are reported in Table 7. For each model, we report the test perplexity, the computational budget, the parameter counts, the value of DropP rob, and the computational efficiency.\n\nModel Test Test ops/timestep #Params excluding Total Drop- TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params P rob per GPU 10 epochs (final) (millions) (billions) (observed) Kneser-Ney 5-gram* 67.6 0.00001 1.8 LSTM-512-512* 54.1 2.4 2.4 0.8 0.1 LSTM-1024-512* 48.2 4.7 4.7 0.8 0.1 LSTM-2048-512* 45.0 43.7 9.4 9.4 0.8 0.1 0.61 LSTM-2048-512 44.7 9.4 9.4 0.8 0.1 1.21 4xLSTM-512 46.0 8.4 8.4 0.8 0.1 1.07 MoE-1-Wide 46.1 8.4 8.4 0.8 0.1 1.29 MoE-1-Deep 45.7 8.4 8.4 0.8 0.1 1.29 MoE-4 45.0 8.4 8.4 0.8 0.1 0.52 MoE-32 39.7 8.4 37.8 0.9 0.1 0.87 MoE-256 35.7 8.6 272.9 1.1 0.1 0.81 MoE-256-h 36.0 8.4 272.9 1.1 0.1 0.89 MoE-1024-h 34.6 8.5 1079.0 1.9 0.2 0.90 MoE-4096-h 34.1 8.9 4303.4 5.1 0.2 0.74 2xLSTM-8192-1024* 34.7 30.6 151.0 151.0 1.8 0.25 1.09 MoE-34M 31.3 33.8 4313.9 6.0 0.3 1.22 MoE-143M 28.0 142.7 4371.1 6.0 0.4 1.56\n\nC.2 More Expensive Models\n\nWe ran two additional models (MoE-34M and MoE-143M) to investigate the effects of adding more computation in the presence of a large MoE layer. These models have computation budgets of 34M and 143M ops/timestep. Similar to the models above, these models use a MoE layer between two LSTM layers. The dimensionality of the embedding layer, and the input and output dimensionality of the MoE layer are set to 1024 instead of 512. For MoE-34M, the LSTM layers have 1024 units.\n\nFor MoE-143M, the LSTM layers have 4096 units and an output projection of size 1024 (Sak et al., 2014). MoE-34M uses a hierarchical MoE layer with 1024 experts, each with a hidden layer of size 2048. MoE-143M uses a hierarchical MoE layer with 256 experts, each with a hidden layer of size 8192. Both models have 4B parameters in the MoE layers. We searched for the best DropP rob for each model, and trained each model for 10 epochs.\n\nThe two models achieved test perplexity of 31.3 and 28.0 respectively, showing that even in the presence of a large MoE, more computation is still useful. Results are reported at the bottom of Table 7. The larger of the two models has a similar computational budget to the best published model from the literature, and training times are similar. Comparing after 10 epochs, our model has a lower test perplexity by 18%.\n\nD 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS Model Architecture: The models are similar in structure to the 8-million-operations-per-timestep models described in the previous section. We vary the number of experts between models, using an ordinary MoE layer with 32 experts and hierarchical MoE layers with 256, 1024, 4096, 16384, 65536 and 131072 experts. For the hierarchical MoE layers, the first level branching factors are 32, 32, 64, 128, 256 and 256, respectively.\n\nTraining: Models are trained on a cluster of 32 Tesla K40 GPUs, except for the last two models, which are trained on clusters of 64 and 128 GPUs so as to have enough memory for all the parameters. For all models, training batch sizes are approximately 2.5 million words. Models are trained once-through over about 100 billion words. We implement several memory optimizations in order to fit up to 1 billion parameters per GPU. First, we do not store the activations of the hidden layers of the experts, but instead recompute them on the backwards pass. Secondly, we modify the optimizer on the expert parameters to require less auxiliary storage: The Adam optimizer (Kingma & Ba, 2015) keeps first and second moment estimates of the perparameter gradients. This triples the required memory. To avoid keeping a first-moment estimator, we set β1 = 0. To reduce the size of the second moment estimator, we replace it with a factored approximation. For a matrix of parameters, instead of maintaining a full matrix of second-moment estimators, we maintain vectors of row-wise and column-wise averages of that matrix. At each step, the matrix of estimators is taken to be the outer product of those two vectors divided by the mean of either one. This technique could similarly be applied to Adagrad (Duchi et al., 2010).\n\nModel Test Test ops/timestep #Params excluding Total TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params per GPU .1 epochs 1 epoch (millions) (billions) (observed) Kneser-Ney 5-gram 67.1 45.3 0.00001 76.0 4xLSTM-512 54.5 47.0 8.4 8.4 0.1 1.23 MoE-32 48.5 40.4 8.4 37.8 0.1 0.83 MoE-256-h 42.8 35.3 8.4 272.9 0.4 1.11 MoE-1024-h 40.3 32.7 8.5 1079.0 1.2 1.14 MoE-4096-h 38.9 30.9 8.6 4303.4 4.4 1.07 MoE-16384-h 38.2 29.7 8.8 17201.0 17.3 0.96 MoE-65536-h 38.2 28.9 9.2 68791.0 68.9 0.72 MoE-131072-h 39.8 29.2 9.7 137577.6 137.7 0.30\n\nResults: We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8. Perplexity after 100 billion training words is 39% lower for the 68-billion-parameter MoE model than for the baseline model. It is notable that the measured computational efficiency of the largest model (0.30 TFLOPS/GPU) is very low compared to the other models. This is likely a result of the fact that, for purposes of comparison to the other models, we did not increase the training batch size proportionally to the number of GPUs. For comparison, we include results for a computationally matched baseline model consisting of 4 LSTMs, and for an unpruned 5-gram model with Kneser-Ney smoothing (Kneser & Ney, 1995).4\n\nE Machine Translation - Experimental Details\n\nModel Architecture for Single Language Pair MoE Models: Our model is a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decrease the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We insert MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). We use an attention mechanism between the encoder and decoder, with the first decoder LSTM receiving output from and providing input for the attention 5. All of the layers in our model have input and output dimensionality of 512. Our LSTM layers have 2048 hidden units, with a 512-dimensional output projection. We add residual connections around all LSTM and MoE layers to encourage gradient flow (He et al., 2015). Similar to GNMT, to effectively deal with rare words, we used subword units (also known as \"wordpieces\") (Schuster & Nakajima, 2012) for inputs and outputs in our system.\n\nWe use a shared source and target vocabulary of 32K wordpieces. We also used the same beam search technique as proposed in (Wu et al., 2016).\n\nWe train models with different numbers of experts in the MoE layers. In addition to a baseline model with no MoE layers, we train models with flat MoE layers containing 32 experts, and models with hierarchical MoE layers containing 512 and 2048 experts. The flat MoE layers use k = 4 and the hierarchical MoE models use k = 2 at each level of the gating network. Thus, each input is processed by exactly 4 experts in each MoE layer. Each expert in the MoE layer is a feed forward network with one hidden layer of size 2048 and ReLU activation. Thus, each expert contains [512 ∗ 2048] + [2048 ∗ 512] = 2M parameters. The output of the MoE layer is passed through a sigmoid function. We use the strictly-balanced gating function described in Appendix F.\n\nModel Architecture for Multilingual MoE Model: We used the same model architecture as for the single-language-pair models, with the following exceptions: We used noisy-top-k gating as described in Section 2.1, not the scheme from Appendix F. The MoE layers in the encoder and decoder are non-hierarchical MoEs with n = 512 experts, and k = 2. Each expert has a larger hidden layer of size 8192. This doubles the amount of computation in the MoE layers, raising the computational budget of the entire model from 85M to 102M ops/timestep. Training: We trained our networks using the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 2000 training steps, held constant for an additional 8000 steps, and decreased after that so as to be proportional to the inverse square root of the step number. For the single-language-pair models, similarly to (Wu et al., 2016), we applied dropout (Zaremba et al., 2014) to the output of all embedding, LSTM and MoE layers, using DropP rob = 0.4. Training was done synchronously on a cluster of up to 64 GPUs as described in section 3. Each training batch consisted of a set of sentence pairs containing roughly 16000 words per GPU.\n\nTo ensure balanced expert utilization we set wimportance = 0.01 and wload = 0.01, as described in Section 4 and Appendix A. Metrics: We evaluated our models using the perplexity and the standard BLEU score metric. We reported tokenized BLEU score as computed by the multi-bleu.pl script, downloaded from the public implementation of Moses (on Github), which was also used in (Luong et al., 2015a). Results: Tables 2, 3 and 4 in Section 5.3 show comparisons of our results to other published methods. Figure 4 shows test perplexity as a function of number of words in the (training data's) source sentences processed for models with different numbers of experts. As can be seen from the Figure, as we increased the number of experts to approach 2048, the test perplexity of our model continued to improve.\n\nWe found that the experts indeed become highly specialized by syntax and/or semantics, as can be seen in Table 9. For example, one expert is used when the indefinite article \"a\" introduces the direct object in a verb phrase indicating importance or leadership.\n\nExpert 381 Expert 752 Expert 2004 ... with researchers , ... ... plays a core ... ... with rapidly growing ... ... to innovation . ... plays a critical ... ... under static conditions ... ... tics researchers . ... provides a legislative ... ... to swift ly ... ... the generation of ... ... play a leading ... ... to dras tically ... ... technology innovations is ... ... assume a leadership ... ... the rapid and ... ... technological innovations , ... ... plays a central ... ... the fast est ... ... support innovation throughout ... ... taken a leading ... ... the Quick Method ... ... role innovation will ... ... established a reconciliation ... ... rec urrent ) ... ... research scienti st ... ... played a vital ... ... provides quick access ... ... promoting innovation where ... ... have a central ... ... of volatile organic ... ... ... ...\n\nDue to some peculiarities in our infrastructure which have since been fixed, at the time we ran some of the machine translation experiments, our models ran faster if every expert received exactly the same batch size. To accommodate this, we used a different gating function which we describe below. Recall that we define the softmax gating function to be:\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ Gσ(x) = Sof tmax(x · Wg) (15) Sparse Gating (alternate formulation): To obtain a sparse gating vector, we multiply Gσ(x) component-wise with a sparse mask M(Gσ(x)) and normalize the output. The mask itself is a function of Gσ(x) and specifies which experts are assigned to each input example:\n\n$$(15)$$ $$G(x){i}=\\frac{G{\\sigma}(x){i}M(G{\\sigma}(x)){i}}{\\sum{j=1}^{n}G_{\\sigma}(x){j}M(G{\\sigma}(x))_{j}}\\tag{1}$$ $$(17)$$\n\n$$(16)$$\n\nTop-K Mask: To implement top-k gating in this formulation, we would let M(v) = T opK(v, k), where:\n\n$$T o p K(v,k){i}=\\begin{cases}1&{\\mathrm{if~}}v{i}{\\mathrm{~is~in~the~top~}}k{\\mathrm{~elements~of~}}v.\\ 0&{\\mathrm{otherwise.}}\\end{cases}$$ 0 otherwise. (17) Batchwise Mask: To force each expert to receive the exact same number of examples, we introduce an alternative mask function, Mbatchwise(X, m), which operates over batches of input vectors.\n\nInstead of keeping the top k values per example, we keep the top m values per expert across the training batch, where m = k|X| n, so that each example is sent to an average of k experts.\n\n$$M_{b a t c h w i s e}(X,m){j,i}=\\begin{cases}1&\\text{if}X{j,i}\\text{is in the top}m\\text{values for to expert}i\\ 0&\\text{otherwise}\\end{cases}$$\n\nAs our experiments suggest and also observed in (Ioffe & Szegedy, 2015), using a batchwise function during training (such as Mbatchwise) requires modifications to the inference when we may not have a large batch of examples. Our solution to this is to train a vector T of per-expert threshold values to approximate the effects of the batchwise mask. We use the following mask at inference time:\n\n$$(18)$$ $$M_{threshold}(x,T){i}=\\begin{cases}1&\\text{if}x{i}>T_{i}\\ 0&\\text{otherwise}\\end{cases}\\tag{1}$$ $$(19)$$\n\nTo learn the threshold values, we apply an additional loss at training time which is minimized when the batchwise mask and the threshold mask are identical.\n\n$$L_{batchwise}(X,T,m)=\\sum_{j=1}^{|X|}\\sum_{i=1}^{n}(M_{threshold}(x,T){i}-M{batchwise}(X,m){j,i})(X{j,i}-T_{i})\\tag{20}$$\n\nG ATTENTION FUNCTION The attention mechanism described in GNMT (Wu et al., 2016) involves a learned \"Attention Function\" A(xi, yj ) which takes a \"source vector\" xi and a \"target vector\" yj , and must be computed for every source time step i and target time step j. In GNMT, the attention function is implemented as a feed forward neural network with a hidden layer of size n. It can be expressed as:\n\n$$A_{G N M T}(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}t a n h((x_{i}U){d}+(y{j}W)_{d})$$\n\nWhere U and W are trainable weight matrices and V is a trainable weight vector.\n\nFor performance reasons, in our models, we used a slightly different attention function:\n\n$$A(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}tanh((x_{i}U){d})tanh((y{j}W)_{d})\\tag{22}$$\n\nWith our attention function, we can simultaneously compute the attention function on multiple source time steps and multiple target time steps using optimized matrix multiplications. We found little difference in quality between the two functions.\n\n$$(21)$$",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/1701.06538v1.md"
+          },
+          "headlines": {
+            "1 Introduction And Related Work": [
+              "1.1 Conditional Computation",
+              "1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER",
+              "1.3 Related Work On Mixtures Of Experts"
+            ],
+            "2 The Structure Of The Mixture-Of-Experts Layer": [
+              "2.1 Gating Network"
+            ],
+            "3 Addressing Performance Challenges": [
+              "3.1 The Shrinking Batch Problem",
+              "3.2 Network Bandwidth"
+            ],
+            "4 Balancing Expert Utilization": [],
+            "5 Experiments": [
+              "5.1 1 Billion Word Language Modeling Benchmark",
+              "5.2 100 BILLION WORD GOOGLE NEWS CORPUS",
+              "5.3 Machine Translation (Single Language Pair)",
+              "5.4 Multilingual Machine Translation"
+            ],
+            "6 Conclusion": [],
+            "Appendices": [
+              "A Load-Balancing Loss",
+              "B Hierachical Mixture Of Experts",
+              "C 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS",
+              "D 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS",
+              "E Machine Translation - Experimental Details"
+            ]
+          },
+          "summary": "The paper introduces a Sparsely-Gated Mixture-of-Experts (MoE) layer to enhance neural network capacity without a proportional increase in computation. This approach uses a trainable gating network to select a sparse combination of experts for each input, significantly improving model capacity and performance in language modeling and machine translation tasks. The MoE layer, with up to 137 billion parameters, achieves better results than state-of-the-art models at lower computational costs. The paper addresses challenges like batch size and network bandwidth, proposing solutions like mixing data and model parallelism. Experiments show that MoE models outperform traditional models in various benchmarks, demonstrating the potential of conditional computation in deep learning.",
+          "summary_embedding": [
+            -0.01644345559179783,
+            -0.016995344310998917,
+            -0.016372699290513992,
+            -0.014349107630550861,
+            0.02282555401325226,
+            0.045934125781059265,
+            -0.014646278694272041,
+            0.03563220426440239,
+            -0.004829026758670807,
+            0.005617944523692131,
+            -0.00932550523430109,
+            -0.008143897168338299,
+            0.005575491581112146,
+            0.01538213063031435,
+            -0.005653322208672762,
+            -0.020306676626205444,
+            -0.013832597061991692,
+            -0.023235931992530823,
+            0.028344441205263138,
+            -0.020080260932445526,
+            0.011823155917227268,
+            -0.020943470299243927,
+            -0.025868017226457596,
+            -0.016500059515237808,
+            0.025061409920454025,
+            0.024806693196296692,
+            0.02583971433341503,
+            0.003827843815088272,
+            -0.03492465242743492,
+            0.033651065081357956,
+            0.014674580655992031,
+            0.012488252483308315,
+            -0.031500112265348434,
+            -0.009162768721580505,
+            -0.05351905897259712,
+            -0.05227377265691757,
+            0.04143410921096802,
+            0.022245362401008606,
+            -0.016231190413236618,
+            0.023717066273093224,
+            0.017264213413000107,
+            0.005816058721393347,
+            -0.020632149651646614,
+            -0.020844414830207825,
+            0.003700484987348318,
+            -0.013103820383548737,
+            -0.01983969286084175,
+            -0.013528349809348583,
+            -0.01828308403491974,
+            -0.01577835902571678,
+            0.009551920928061008,
+            0.0002856731880456209,
+            0.0029186424799263477,
+            -0.003424540627747774,
+            -0.047830358147621155,
+            0.0248208437114954,
+            0.019853845238685608,
+            0.03492465242743492,
+            0.04633035510778427,
+            0.013011839240789413,
+            0.01516986545175314,
+            -0.014271277002990246,
+            -0.04021712392568588,
+            0.005497660953551531,
+            -0.00041656987741589546,
+            -0.005380915477871895,
+            -0.005819596350193024,
+            0.008575502783060074,
+            0.008724087849259377,
+            -0.01143400277942419,
+            -0.04202845320105553,
+            0.013952880166471004,
+            -0.020561393350362778,
+            0.00010143388499272987,
+            0.01354957651346922,
+            0.008292482234537601,
+            0.02677368000149727,
+            0.03195294365286827,
+            0.023971782997250557,
+            -0.008724087849259377,
+            -0.01340806670486927,
+            0.019924599677324295,
+            -0.0022676968947052956,
+            0.007018893025815487,
+            0.04570771008729935,
+            -0.04033033177256584,
+            0.02163686975836754,
+            -0.025726506486535072,
+            -0.015339677222073078,
+            -0.005349075887352228,
+            -0.022471778094768524,
+            0.02806142158806324,
+            -0.007344366051256657,
+            0.017801949754357338,
+            0.017094401642680168,
+            0.01594817079603672,
+            0.013330236077308655,
+            -0.001894464367069304,
+            -0.022938761860132217,
+            0.0018360915128141642,
+            -0.01541043259203434,
+            0.02114158496260643,
+            0.0015230007702484727,
+            0.005469358991831541,
+            0.027594437822699547,
+            0.031160488724708557,
+            -0.023490650579333305,
+            0.009502392262220383,
+            0.02422650158405304,
+            0.06967949122190475,
+            0.015424583107233047,
+            -0.006435164716094732,
+            -0.0024640418123453856,
+            0.04143410921096802,
+            0.0015513027319684625,
+            0.0038349193055182695,
+            -0.06826439499855042,
+            -0.03022652305662632,
+            -0.0011365016689524055,
+            -0.02254253253340721,
+            0.02227366529405117,
+            0.01002597901970148,
+            -0.009750034660100937,
+            -0.022457627579569817,
+            -0.04516997188329697,
+            0.05190584436058998,
+            -0.00545874610543251,
+            -0.016259491443634033,
+            -0.005554265342652798,
+            -0.05340585112571716,
+            0.00039136342820711434,
+            0.015042506158351898,
+            -0.018481198698282242,
+            -0.03670767694711685,
+            0.036141637712717056,
+            0.0007398316520266235,
+            0.03741522878408432,
+            0.012459950521588326,
+            -0.01708024926483631,
+            -0.020094411447644234,
+            0.022259512916207314,
+            -0.011639192700386047,
+            -0.0038313816767185926,
+            -0.027339721098542213,
+            0.0065236082300543785,
+            -0.028103873133659363,
+            -0.0014566679019480944,
+            0.06730213016271591,
+            -0.04816998168826103,
+            0.02946236915886402,
+            -0.029660483822226524,
+            0.023660462349653244,
+            -0.008575502783060074,
+            0.0278916098177433,
+            -0.010238244198262691,
+            0.005840822588652372,
+            -0.013457595370709896,
+            -0.0010586712742224336,
+            -0.020745357498526573,
+            0.0034210027661174536,
+            -0.016259491443634033,
+            -0.027750099077820778,
+            -0.007054270710796118,
+            0.031273696571588516,
+            0.01698119193315506,
+            0.025655752047896385,
+            0.025429336354136467,
+            0.01246702577918768,
+            -0.04935866594314575,
+            0.02595292218029499,
+            0.0018926954362541437,
+            0.013514199294149876,
+            0.013181651011109352,
+            -0.0010418669553473592,
+            0.025599148124456406,
+            0.012049571610987186,
+            0.025061409920454025,
+            0.037443529814481735,
+            0.0027682881336659193,
+            0.04737752676010132,
+            -0.015693452209234238,
+            -0.005292471963912249,
+            0.028528403490781784,
+            0.000505455827806145,
+            0.02465103194117546,
+            -0.001686621573753655,
+            0.013330236077308655,
+            0.000776978034991771,
+            0.03834919258952141,
+            -0.006339645478874445,
+            0.029518973082304,
+            0.021608568727970123,
+            0.028316138312220573,
+            -0.014759486541152,
+            0.023122724145650864,
+            -0.019754787907004356,
+            -0.02292460948228836,
+            0.035547297447919846,
+            0.004082561936229467,
+            0.01886327564716339,
+            -0.028457649052143097,
+            -0.030679354444146156,
+            0.015452885068953037,
+            -0.01855195313692093,
+            0.016556663438677788,
+            0.03639635816216469,
+            0.0008273909334093332,
+            -0.0025719432160258293,
+            0.026292547583580017,
+            -0.04245298355817795,
+            0.012856177985668182,
+            -0.01532552670687437,
+            -0.02855670638382435,
+            -0.019528372213244438,
+            -0.00875946506857872,
+            -0.0018095583654940128,
+            -0.013691086322069168,
+            -0.0037570889107882977,
+            -0.007563706487417221,
+            -0.02459442801773548,
+            0.00935380719602108,
+            0.013627407141029835,
+            0.009849091991782188,
+            -0.018665160983800888,
+            0.0019952901639044285,
+            -0.008257105015218258,
+            -0.0160047747194767,
+            -0.03752843663096428,
+            0.015268922783434391,
+            -0.0024888061452656984,
+            0.009191070683300495,
+            0.04078316316008568,
+            0.028485950082540512,
+            -0.005055442452430725,
+            0.0329718142747879,
+            -0.033707667142152786,
+            0.017915157601237297,
+            -0.008320784196257591,
+            0.009191070683300495,
+            0.002989397384226322,
+            -0.025542544201016426,
+            0.027424626052379608,
+            -0.03792466223239899,
+            -0.038151077926158905,
+            0.022995365783572197,
+            0.011858533136546612,
+            0.0017096169758588076,
+            -0.019386861473321915,
+            -0.017717044800519943,
+            0.0008499440737068653,
+            0.002752368338406086,
+            -0.018834972754120827,
+            -0.01019579079002142,
+            0.02227366529405117,
+            -0.025500090792775154,
+            0.036085035651922226,
+            0.023391593247652054,
+            -0.03520767390727997,
+            -0.00608492735773325,
+            -0.01903308741748333,
+            0.059037946164608,
+            -0.0011506526498124003,
+            -0.010301923379302025,
+            0.022938761860132217,
+            -0.027268964797258377,
+            -0.004524780437350273,
+            0.022301966324448586,
+            0.01828308403491974,
+            -0.008278331719338894,
+            0.003983504604548216,
+            -0.022287815809249878,
+            -0.05281151086091995,
+            -0.010577867738902569,
+            -0.0003305583668407053,
+            0.054368119686841965,
+            -0.012367968447506428,
+            0.0020695829298347235,
+            -0.006413938011974096,
+            0.004100250545889139,
+            0.03704730048775673,
+            0.002140337834134698,
+            0.0025082635693252087,
+            -0.013698162510991096,
+            0.0371888130903244,
+            0.022316116839647293,
+            0.005883275531232357,
+            -0.012162779457867146,
+            0.005377377849072218,
+            -0.04899074137210846,
+            -0.0009118546731770039,
+            0.005741765722632408,
+            0.007386818993836641,
+            -0.025910470634698868,
+            0.017717044800519943,
+            0.009205221198499203,
+            -0.0024693484883755445,
+            -0.009792488068342209,
+            0.01173117384314537,
+            0.025429336354136467,
+            -0.019938750192523003,
+            -0.05700020492076874,
+            0.0034705311991274357,
+            0.012014193460345268,
+            -0.004054259974509478,
+            -0.019401011988520622,
+            0.006792477332055569,
+            0.0021934041287750006,
+            0.022698193788528442,
+            -0.011271266266703606,
+            -0.0029221803415566683,
+            0.040585052222013474,
+            0.009516543708741665,
+            -0.0016813150141388178,
+            -0.017717044800519943,
+            -0.011794853024184704,
+            0.018070818856358528,
+            0.004740582779049873,
+            0.0035289041697978973,
+            -0.01751893013715744,
+            -0.029264256358146667,
+            0.02800481766462326,
+            0.019967053085565567,
+            -0.048905834555625916,
+            -0.008200501091778278,
+            -0.019273653626441956,
+            0.0019599127117544413,
+            0.008695785887539387,
+            0.04613224044442177,
+            -0.00590803986415267,
+            0.008313708938658237,
+            -0.020179318264126778,
+            -0.05003791302442551,
+            -0.02708500251173973,
+            0.03707560524344444,
+            -0.028684064745903015,
+            -0.04276430234313011,
+            -0.010252394713461399,
+            0.016740625724196434,
+            -0.009403335861861706,
+            -0.018679311498999596,
+            -0.011738249100744724,
+            -0.04969828948378563,
+            -0.003497064346447587,
+            0.0337359681725502,
+            0.017264213413000107,
+            -0.008441068232059479,
+            -0.033877480775117874,
+            0.019641580060124397,
+            -0.007160402834415436,
+            -0.006474079564213753,
+            0.0007862646016292274,
+            0.008794843219220638,
+            0.02400008589029312,
+            0.011879759840667248,
+            -0.02763689123094082,
+            -0.03427370637655258,
+            -0.0024888061452656984,
+            0.014929298311471939,
+            -0.00878776703029871,
+            0.0245378240942955,
+            0.024679332971572876,
+            -0.022995365783572197,
+            -0.01654251106083393,
+            0.015750056132674217,
+            -0.05779265984892845,
+            -0.010287772864103317,
+            -0.029235953465104103,
+            -0.006183984223753214,
+            0.0009613831643946469,
+            0.0061096916906535625,
+            0.00012304731353651732,
+            0.03846240043640137,
+            0.008801918476819992,
+            0.006014172453433275,
+            0.009191070683300495,
+            0.01913214474916458,
+            0.028471799567341805,
+            0.009155693463981152,
+            -0.03517937287688255,
+            0.02850010246038437,
+            -0.017051948234438896,
+            0.007811348885297775,
+            -0.025754809379577637,
+            -0.013033065013587475,
+            0.00443633645772934,
+            0.02725481428205967,
+            -0.000503244751598686,
+            0.018849123269319534,
+            -0.021919889375567436,
+            0.037500135600566864,
+            -0.029915200546383858,
+            -0.05278320610523224,
+            -0.02005195803940296,
+            -0.008285406976938248,
+            0.015764206647872925,
+            0.017278363928198814,
+            -0.019924599677324295,
+            -0.0286557637155056,
+            -0.021750077605247498,
+            0.006470541935414076,
+            0.010542490519583225,
+            -0.021099131554365158,
+            -0.010627396404743195,
+            0.01502835564315319,
+            -0.00404364662244916,
+            -0.01886327564716339,
+            0.013684011064469814,
+            0.015693452209234238,
+            0.03169822692871094,
+            -0.03308502212166786,
+            0.004287751391530037,
+            -0.025967074558138847,
+            -0.04327373951673508,
+            0.004096712917089462,
+            0.03166992589831352,
+            0.027509532868862152,
+            -0.018636859953403473,
+            0.010754755698144436,
+            0.003990580327808857,
+            0.004234685096889734,
+            -0.0033803186379373074,
+            0.023603858426213264,
+            0.03291521221399307,
+            0.02763689123094082,
+            -0.011653343215584755,
+            0.004921008367091417,
+            -0.01559439580887556,
+            -0.0011108530452474952,
+            0.01865101046860218,
+            -0.017816102132201195,
+            0.033877480775117874,
+            -0.029405765235424042,
+            0.02104252763092518,
+            -0.035122767090797424,
+            0.006141531281173229,
+            -0.012099100276827812,
+            -0.00404364662244916,
+            0.04833979532122612,
+            -0.017646290361881256,
+            -0.0019546060357242823,
+            -0.008101443760097027,
+            -0.014575524255633354,
+            -0.02158026583492756,
+            0.008766540326178074,
+            -0.04033033177256584,
+            -0.012367968447506428,
+            -0.028429346159100533,
+            0.003424540627747774,
+            0.019910449162125587,
+            -0.06277380883693695,
+            -0.06237758323550224,
+            0.012275987304747105,
+            -0.04918885603547096,
+            -0.026419905945658684,
+            0.02963218092918396,
+            -0.002984090941026807,
+            0.0012780115939676762,
+            -0.0024003623984754086,
+            0.0068101659417152405,
+            -0.029207652434706688,
+            0.00425944896414876,
+            -0.000887974863871932,
+            0.03271709755063057,
+            -0.021056678146123886,
+            -0.008122670464217663,
+            0.004878554958850145,
+            -0.06860402226448059,
+            -0.028344441205263138,
+            0.011271266266703606,
+            -0.009459939785301685,
+            -0.0066615804098546505,
+            -0.017915157601237297,
+            -0.004383270163089037,
+            0.0008950503543019295,
+            0.0032830305863171816,
+            0.05801907554268837,
+            -0.01983969286084175,
+            -0.053773775696754456,
+            0.03633975237607956,
+            0.00887267291545868,
+            0.029603879898786545,
+            -0.025556694716215134,
+            -0.017533080652356148,
+            -0.01855195313692093,
+            -0.020504789426922798,
+            -0.04123599827289581,
+            0.00404364662244916,
+            -0.005918653216212988,
+            0.009212297387421131,
+            -0.0009693430620245636,
+            0.02763689123094082,
+            0.029235953465104103,
+            0.023108573630452156,
+            0.012750045396387577,
+            0.01079013291746378,
+            -0.022372720763087273,
+            -0.021594416350126266,
+            0.0012222920777276158,
+            0.04683978855609894,
+            0.024636881425976753,
+            -0.02675952948629856,
+            -0.011504758149385452,
+            0.03953787684440613,
+            0.0419718474149704,
+            -0.01974063739180565,
+            -0.02276895008981228,
+            0.00764861237257719,
+            0.03266049548983574,
+            -0.04477374628186226,
+            0.005932804197072983,
+            0.03410389646887779,
+            0.014405711553990841,
+            -0.005356151144951582,
+            -0.010294848121702671,
+            0.022570835426449776,
+            0.03152841329574585,
+            -0.008724087849259377,
+            0.01876421831548214,
+            0.016910437494516373,
+            0.029349161311984062,
+            -0.0067889392375946045,
+            0.0005766529939137399,
+            -0.024735936895012856,
+            0.055188875645399094,
+            -0.03059444949030876,
+            0.02909444272518158,
+            0.010316074825823307,
+            -0.00593634182587266,
+            0.039056744426488876,
+            -0.016641568392515182,
+            -0.0431322306394577,
+            0.01913214474916458,
+            -0.006979977712035179,
+            -0.0055613406002521515,
+            -0.03444352000951767,
+            -0.007592008449137211,
+            -0.007507102098315954,
+            -0.006714646704494953,
+            -0.022556684911251068,
+            -0.02606613002717495,
+            0.004415110219269991,
+            0.00973588414490223,
+            -0.006484692916274071,
+            0.00698705343529582,
+            0.02092931978404522,
+            -0.013507124036550522,
+            0.02525952458381653,
+            -0.046047333627939224,
+            -0.0008366775000467896,
+            -0.0038596836384385824,
+            -0.025712355971336365,
+            -0.0345001220703125,
+            -0.0005770951975136995,
+            -0.0024251264985650778,
+            -0.024622729048132896,
+            -0.00914154201745987,
+            -0.008752389810979366,
+            -0.0023119186516851187,
+            -0.004938696976751089,
+            -0.055188875645399094,
+            0.008695785887539387,
+            0.010846736840903759,
+            -0.002713453024625778,
+            0.0055507272481918335,
+            -0.010613245889544487,
+            -0.03039633482694626,
+            0.03039633482694626,
+            0.04129260033369064,
+            0.033707667142152786,
+            0.014334957115352154,
+            -0.0074646491557359695,
+            -0.017165156081318855,
+            0.006060163024812937,
+            0.021721776574850082,
+            0.00902833417057991,
+            0.007846726104617119,
+            0.03877372294664383,
+            0.00011331850691931322,
+            -0.008349086157977581,
+            -0.009764186106622219,
+            0.028740668669342995,
+            -0.001085204305127263,
+            0.023603858426213264,
+            0.050802066922187805,
+            0.030424637719988823,
+            0.020589696243405342,
+            -0.01681138016283512,
+            -0.02010856196284294,
+            -0.00851889792829752,
+            -0.02855670638382435,
+            0.0364246591925621,
+            0.012127402238547802,
+            -0.005232329946011305,
+            -0.011540135368704796,
+            0.019061388447880745,
+            -0.033339742571115494,
+            -0.0037358624394983053,
+            0.00048422932741232216,
+            0.013924578204751015,
+            -0.03379257395863533,
+            0.0002480846014805138,
+            0.031613320112228394,
+            0.014957600273191929,
+            -0.016995344310998917,
+            -0.0188774261623621,
+            0.010549565777182579,
+            0.020462337881326675,
+            0.018948180601000786,
+            0.00037743354914709926,
+            0.00897173024713993,
+            -0.00022121983056422323,
+            -0.0005319889169186354,
+            0.01590571738779545,
+            -0.007952858693897724,
+            0.036254845559597015,
+            -0.003525366308167577,
+            -0.01325240544974804,
+            -0.0037500134203583,
+            -0.021438755095005035,
+            -0.024467067793011665,
+            0.043556760996580124,
+            0.023971782997250557,
+            0.04661337286233902,
+            -0.005660397466272116,
+            -0.004245297983288765,
+            0.025825563818216324,
+            0.008221727795898914,
+            -0.03288691118359566,
+            0.000265110022155568,
+            0.005281858611851931,
+            -0.02892463095486164,
+            0.016641568392515182,
+            -0.02476423978805542,
+            -0.016896286979317665,
+            -0.006771250627934933,
+            -0.0015557248843833804,
+            -0.028188779950141907,
+            0.01703779585659504,
+            0.0031309074256569147,
+            -0.0027099153958261013,
+            -0.039764292538166046,
+            0.006880920846015215,
+            0.020419884473085403,
+            0.015311375260353088,
+            0.0037358624394983053,
+            0.011525984853506088,
+            -0.016613267362117767,
+            0.002126186853274703,
+            0.03379257395863533,
+            0.004878554958850145,
+            0.02005195803940296,
+            0.0027647505048662424,
+            0.023773670196533203,
+            0.03633975237607956,
+            -0.01681138016283512,
+            -0.02189158834517002,
+            0.01519816741347313,
+            -0.02233026921749115,
+            0.029943503439426422,
+            0.004889168310910463,
+            -0.012283062562346458,
+            0.022033097222447395,
+            -0.009827865287661552,
+            -0.0027346794959157705,
+            0.01659911684691906,
+            -0.022740647196769714,
+            -0.017717044800519943,
+            0.03183973580598831,
+            -0.039452970027923584,
+            0.012856177985668182,
+            -0.008547200821340084,
+            0.015863263979554176,
+            -0.023066120222210884,
+            0.011172209866344929,
+            0.007100261282175779,
+            -0.01573590561747551,
+            0.010797208175063133,
+            0.009693430736660957,
+            0.00545874610543251,
+            0.0018926954362541437,
+            0.02179253101348877,
+            -0.0006049550138413906,
+            -0.011257115751504898,
+            -0.018155725672841072,
+            -0.004694592207670212,
+            -0.002614396158605814,
+            -0.016556663438677788,
+            0.01865101046860218,
+            -0.041066184639930725,
+            -0.013280707411468029,
+            -0.008879749104380608,
+            -0.013988257385790348,
+            -0.010011828504502773,
+            0.024184048175811768,
+            0.025669902563095093,
+            -0.002361447084695101,
+            -0.031217092648148537,
+            0.01790100708603859,
+            0.024198200553655624,
+            0.003654494183138013,
+            -0.023646311834454536,
+            0.0302831269800663,
+            -0.008936353027820587,
+            0.014971751719713211,
+            0.026136886328458786,
+            0.014094389975070953,
+            0.004595535341650248,
+            -0.010500037111341953,
+            -0.02114158496260643,
+            0.01481609046459198,
+            0.010507113300263882,
+            0.033537857234478,
+            0.014745336025953293,
+            0.02071705460548401,
+            0.01594817079603672,
+            -0.023320838809013367,
+            -0.011214662343263626,
+            -0.003482913365587592,
+            -0.022514231503009796,
+            0.03093407303094864,
+            0.0006129149114713073,
+            0.02114158496260643,
+            0.026618018746376038,
+            -0.03319823369383812,
+            -0.01187268365174532,
+            0.032688796520233154,
+            0.02779255248606205,
+            0.05108508840203285,
+            0.025316128507256508,
+            0.015396281145513058,
+            0.027226513251662254,
+            -0.03585861995816231,
+            -0.036368053406476974,
+            0.001548649393953383,
+            0.04256619140505791,
+            -0.008426916785538197,
+            0.00967928022146225,
+            0.03900013864040375,
+            -0.006781863979995251,
+            -0.01611798256635666,
+            0.019457615911960602,
+            0.0026710000820457935,
+            0.002448122017085552,
+            -0.0033113325480371714,
+            -0.01335146278142929,
+            0.03175482898950577,
+            -0.0051474240608513355,
+            0.013570803217589855,
+            -0.019273653626441956,
+            0.027113303542137146,
+            0.0001799829478841275,
+            -0.003229964291676879,
+            0.028344441205263138,
+            -0.002660386962816119,
+            -0.051990751177072525,
+            0.024212351068854332,
+            0.019811391830444336,
+            -0.011561362072825432,
+            -0.0036332677118480206,
+            -0.003792466362938285,
+            0.04567940905690193,
+            0.034698236733675,
+            0.029009537771344185,
+            -0.0020501252729445696,
+            0.0168679840862751,
+            0.018792519345879555,
+            0.0053667644970119,
+            0.014278353191912174,
+            -0.0017025414854288101,
+            -0.007853801362216473,
+            0.021056678146123886,
+            -0.020787810906767845,
+            0.03226426616311073,
+            -0.016570813953876495,
+            -0.020122714340686798,
+            0.008738238364458084,
+            -0.011172209866344929,
+            -0.0009083169279620051,
+            -0.006311343517154455,
+            -0.0008813415770418942,
+            -0.0029664020985364914,
+            0.016938740387558937,
+            -0.004446949809789658,
+            0.04661337286233902,
+            -0.013931653462350368,
+            -0.008808993734419346,
+            0.006972902454435825,
+            -0.025981225073337555,
+            0.0013346156338229775,
+            0.0028726516757160425,
+            0.027000095695257187,
+            -0.01989629678428173,
+            -0.00970758218318224,
+            0.01606137864291668,
+            0.06039644405245781,
+            -0.008639181964099407,
+            -0.0026267783250659704,
+            0.027226513251662254,
+            -0.002787745790556073,
+            -0.03551899641752243,
+            0.012919857166707516,
+            -0.04494355618953705,
+            0.017589686438441277,
+            0.026349151507019997,
+            -0.04986810311675072,
+            -0.022089701145887375,
+            -0.02347649820148945,
+            -0.02043403498828411,
+            0.023434046655893326,
+            -0.009976450353860855,
+            0.013867974281311035,
+            -0.017801949754357338,
+            -0.009014182724058628,
+            0.020518941804766655,
+            0.009176919236779213,
+            0.03735862299799919,
+            -0.027551984414458275,
+            -0.016783079132437706,
+            -0.027820853516459465,
+            -0.04304732382297516,
+            0.009558996185660362,
+            -0.0278916098177433,
+            0.0027099153958261013,
+            -0.0016291331266984344,
+            -0.040302030742168427,
+            -0.037981268018484116,
+            0.005278320983052254,
+            -0.017985913902521133,
+            0.01516986545175314,
+            0.020915169268846512,
+            0.02731141820549965,
+            -0.0020253609400242567,
+            -0.030198222026228905,
+            0.029235953465104103,
+            -0.007344366051256657,
+            0.03223596513271332,
+            -0.00689507182687521,
+            0.005066055804491043,
+            -0.0070436573587358,
+            0.031160488724708557,
+            -0.023009516298770905,
+            0.018792519345879555,
+            -0.026801982894539833,
+            -0.004086099565029144,
+            -0.018990634009242058,
+            -0.01774534583091736,
+            0.004224071744829416,
+            0.004521242808550596,
+            7.888073741924018e-05,
+            0.0048750173300504684,
+            0.005777143407613039,
+            0.024806693196296692,
+            0.0021421066485345364,
+            0.02589631825685501,
+            -0.027877457439899445,
+            -0.025316128507256508,
+            0.01719345711171627,
+            -0.02077365852892399,
+            0.005678086541593075,
+            -0.006261814851313829,
+            -0.0302831269800663,
+            -0.01202834490686655,
+            0.023391593247652054,
+            0.02162271924316883,
+            -0.005490585695952177,
+            0.002644466934725642,
+            0.02206140011548996,
+            -0.026023678481578827,
+            0.00437973253428936,
+            -0.03781145438551903,
+            -0.018396291881799698,
+            0.029773691669106483,
+            -0.03195294365286827,
+            0.020745357498526573,
+            0.003185742534697056,
+            0.03483974561095238,
+            -0.017292514443397522,
+            -0.01947176828980446,
+            0.01654251106083393,
+            -0.01008258294314146,
+            0.004142703488469124,
+            0.01606137864291668,
+            0.01616043597459793,
+            -0.0015070808585733175,
+            0.05380208045244217,
+            0.0008167776977643371,
+            0.006909222807735205,
+            -0.049075648188591,
+            0.004496478475630283,
+            -0.044858649373054504,
+            0.0002768288250081241,
+            -0.013337312266230583,
+            -0.02514631673693657,
+            0.0014027173165231943,
+            0.001001182827167213,
+            0.001963450340554118,
+            -0.01913214474916458,
+            -0.01989629678428173,
+            0.011596739292144775,
+            -0.015863263979554176,
+            0.004928083624690771,
+            -0.06418890506029129,
+            -0.01855195313692093,
+            -0.004974074196070433,
+            0.0043337419629096985,
+            0.0007699025445617735,
+            0.03342464938759804,
+            -0.01478778850287199,
+            0.010117961093783379,
+            -0.01549533847719431,
+            -0.026363302022218704,
+            0.04010391607880592,
+            0.018750067800283432,
+            -0.023773670196533203,
+            0.024693485349416733,
+            0.012106175534427166,
+            0.03099067695438862,
+            -0.0318114347755909,
+            0.020349130034446716,
+            0.018523650243878365,
+            -0.0029823218937963247,
+            0.006551910191774368,
+            0.008575502783060074,
+            0.01832553744316101,
+            -0.015764206647872925,
+            -0.0003632825391832739,
+            0.011419852264225483,
+            0.010033054277300835,
+            0.031924642622470856,
+            0.01403071079403162,
+            -0.03183973580598831,
+            0.019047237932682037,
+            0.01616043597459793,
+            0.02298121340572834,
+            0.033820874989032745,
+            -0.027481229975819588,
+            0.038575608283281326,
+            -0.010386829264461994,
+            -0.024042539298534393,
+            0.020801961421966553,
+            -0.032688796520233154,
+            -0.006679269019514322,
+            0.030735958367586136,
+            0.04398128762841225,
+            -0.0015566092915832996,
+            -0.044632233679294586,
+            0.01535382866859436,
+            0.01316749956458807,
+            0.0027806703001260757,
+            0.028811423107981682,
+            0.011264191009104252,
+            0.021820832043886185,
+            0.025273675099015236,
+            0.011419852264225483,
+            -0.0037252490874379873,
+            0.024962352588772774,
+            -0.008101443760097027,
+            -0.004446949809789658,
+            -0.0007035697344690561,
+            0.03206615149974823,
+            0.014702882617712021,
+            -0.017603836953639984,
+            -0.036962397396564484,
+            -0.007620310410857201,
+            0.009452863596379757,
+            -0.007903330028057098,
+            -0.027127455919981003,
+            -0.03950957581400871,
+            -0.026787832379341125,
+            -0.005667473189532757,
+            0.01505665760487318,
+            -0.06724552065134048,
+            -0.007620310410857201,
+            -0.012233533896505833,
+            0.017717044800519943,
+            0.0022906921803951263,
+            0.0181132722645998,
+            0.001277127186767757,
+            -0.009403335861861706,
+            0.004209920763969421,
+            0.050802066922187805,
+            -0.03059444949030876,
+            0.014242975041270256,
+            0.011292492970824242,
+            0.0032105068676173687,
+            0.002787745790556073,
+            0.038688816130161285,
+            -0.010988246649503708,
+            -0.030424637719988823,
+            0.011575513519346714,
+            -0.017292514443397522,
+            0.003383856499567628,
+            0.05006621778011322,
+            -0.009481165558099747,
+            -0.027495380491018295,
+            0.020915169268846512,
+            -0.009148617275059223,
+            -0.015523640438914299,
+            -0.0027258351910859346,
+            -0.006035399157553911,
+            0.0006283925613388419,
+            -0.03832089155912399,
+            -0.015750056132674217,
+            0.03212275728583336,
+            0.008617955259978771,
+            0.014603826217353344,
+            0.025344429537653923,
+            0.010209942236542702,
+            -0.030481241643428802,
+            0.027481229975819588,
+            -0.0005678086308762431,
+            0.005221716593950987,
+            0.013323160819709301,
+            0.03424540534615517,
+            -0.014264201745390892,
+            -0.020264223217964172,
+            -0.040245428681373596,
+            -0.013853822834789753,
+            -0.0463869571685791,
+            0.001943992800079286,
+            0.006919836159795523,
+            -0.012919857166707516,
+            -0.04366996884346008,
+            -0.0329718142747879,
+            0.021127434447407722,
+            0.03178313374519348,
+            -0.017023645341396332,
+            0.0025294902734458447,
+            -0.008483520708978176,
+            -0.007882103323936462,
+            -0.0017520699184387922,
+            -0.013217028230428696,
+            -0.006378560326993465,
+            -0.006229975260794163,
+            0.015452885068953037,
+            -0.004652139265090227,
+            -0.0036934094969183207,
+            -0.0498398020863533,
+            -0.018778368830680847,
+            0.004082561936229467,
+            -0.008221727795898914,
+            0.007323139347136021,
+            -0.006955213379114866,
+            -0.002727604005485773,
+            -0.012474101036787033,
+            -0.03829259052872658,
+            0.024184048175811768,
+            0.017278363928198814,
+            0.0035501306410878897,
+            -0.008922201581299305,
+            -0.018367990851402283,
+            0.03294351324439049,
+            0.012523629702627659,
+            -0.017547233030200005,
+            -0.018580254167318344,
+            0.025981225073337555,
+            0.026957644149661064,
+            0.008122670464217663,
+            -0.006137993652373552,
+            0.029886899515986443,
+            0.005695775151252747,
+            -0.022938761860132217,
+            -0.003983504604548216,
+            -0.005055442452430725,
+            -0.0038844477385282516,
+            0.020561393350362778,
+            -0.010556641966104507,
+            0.0300001073628664,
+            0.038717120885849,
+            0.012870329432189465,
+            0.0039198254235088825,
+            -0.0033714743331074715,
+            -0.008922201581299305,
+            0.015976471826434135,
+            0.006481155287474394,
+            0.008525974117219448,
+            0.014759486541152,
+            -0.016358548775315285,
+            -0.020405733957886696,
+            0.006215824279934168,
+            -0.007107336539775133,
+            0.020122714340686798,
+            0.01708024926483631,
+            0.016429303213953972,
+            0.0007610581233166158,
+            -0.028287837281823158,
+            -0.0018024828750640154,
+            0.02411329373717308,
+            0.02010856196284294,
+            -0.004655676893889904,
+            -0.023264234885573387,
+            0.014398636296391487,
+            -0.015806660056114197,
+            -0.009969375096261501,
+            0.0033431723713874817,
+            0.006229975260794163,
+            -0.003083147807046771,
+            0.0571134127676487,
+            -0.020193468779325485,
+            -0.011419852264225483,
+            -0.010287772864103317,
+            -0.010867963545024395,
+            0.0012072566896677017,
+            -0.01828308403491974,
+            0.008964654989540577,
+            0.025613298639655113,
+            -0.009764186106622219,
+            0.03568880632519722,
+            0.0228114016354084,
+            0.006912760436534882,
+            0.005260631907731295,
+            -0.027070851996541023,
+            0.012304289266467094,
+            -0.016047226265072823,
+            0.005660397466272116,
+            0.018467046320438385,
+            0.017320817336440086,
+            0.0076132346875965595,
+            0.026306698098778725,
+            0.019386861473321915,
+            0.016684021800756454,
+            -0.022783100605010986,
+            0.025457637384533882,
+            -0.007832575589418411,
+            0.027877457439899445,
+            -0.01611798256635666,
+            -0.002253545681014657,
+            -0.012990612536668777,
+            0.008681634441018105,
+            0.002416282193735242,
+            -0.0019351483788341284,
+            0.00036129256477579474,
+            0.016783079132437706,
+            -0.014589674770832062,
+            0.01062032114714384,
+            0.013316085562109947,
+            -0.027919910848140717,
+            -0.010372678749263287,
+            0.016344398260116577,
+            -0.00668988237157464,
+            0.006877383217215538,
+            -0.011865608394145966,
+            -0.0017202302115038037,
+            0.011009473353624344,
+            -0.0436982698738575,
+            0.0104929618537426,
+            0.006279503460973501,
+            -0.015155714936554432,
+            0.014801939949393272,
+            -0.009799563325941563,
+            -0.008235878311097622,
+            -0.013592029921710491,
+            0.003792466362938285,
+            -0.00030225637601688504,
+            -0.0018290160223841667,
+            -0.00022995365725364536,
+            0.00389859895221889,
+            0.010047205723822117,
+            -0.0045990729704499245,
+            -0.010450509376823902,
+            -0.0028638073708862066,
+            -0.02265574224293232,
+            -0.0010047205723822117,
+            0.018735915422439575,
+            -0.004591997712850571,
+            0.0005222601466812193,
+            0.0145189194008708,
+            -0.005830209702253342,
+            0.010556641966104507,
+            -0.005221716593950987,
+            -0.008320784196257591,
+            -0.010768906213343143,
+            -0.033764272928237915,
+            0.0026710000820457935,
+            -0.015254771336913109,
+            0.008865597657859325,
+            -0.012544856406748295,
+            0.04027372971177101,
+            0.01022409275174141,
+            -0.00837738811969757,
+            -0.002796590095385909,
+            -0.0228114016354084,
+            -0.004475251771509647,
+            -0.016938740387558937,
+            -0.01184438169002533,
+            0.008342010900378227,
+            0.012629762291908264,
+            0.005614406894892454,
+            -0.018410442396998405,
+            0.003665107535198331,
+            0.0030955299735069275,
+            -0.005575491581112146,
+            -0.008370312862098217,
+            0.010853812098503113,
+            -0.0006248548161238432,
+            -0.01800006441771984,
+            -0.01957082375884056,
+            -0.027424626052379608,
+            0.0150000536814332,
+            0.014589674770832062,
+            -0.01011088490486145,
+            -0.022754797711968422,
+            -0.007471724878996611,
+            0.007107336539775133,
+            -0.02064630016684532,
+            0.0006779210525564849,
+            0.006891534198075533,
+            0.008717012591660023,
+            0.004117939155548811,
+            0.009636826813220978,
+            0.0007699025445617735,
+            0.0121698547154665,
+            0.018566103652119637,
+            0.03271709755063057,
+            -0.0396793857216835,
+            -0.009537769481539726,
+            0.0057028504088521,
+            -0.001465512323193252,
+            -0.013853822834789753,
+            0.009672204032540321,
+            0.0024375086650252342,
+            -0.0026426981203258038,
+            0.024424616247415543,
+            -0.008044839836657047,
+            -0.01505665760487318,
+            0.01184438169002533,
+            0.00887267291545868,
+            -0.019075538963079453,
+            0.02746707946062088,
+            0.006169833242893219,
+            -0.01011088490486145,
+            0.02259913645684719,
+            -0.012360893189907074,
+            0.012658064253628254,
+            -0.005522425286471844,
+            0.023660462349653244,
+            -0.019004784524440765,
+            0.0006306036957539618,
+            0.015665151178836823,
+            0.03883032873272896,
+            -0.00437973253428936,
+            -0.004567233379930258,
+            -0.03421710431575775,
+            0.005115584470331669,
+            0.028853876516222954,
+            0.02666047215461731,
+            -0.0012346742441877723,
+            -0.005596718285232782,
+            -0.0019864458590745926,
+            0.010768906213343143,
+            0.011589664034545422,
+            0.017561383545398712,
+            -0.029547275975346565,
+            0.00022011429246049374,
+            -0.016938740387558937,
+            0.025500090792775154,
+            0.014632128179073334,
+            0.006951675750315189,
+            0.04072656109929085,
+            0.0027789014857262373,
+            0.003841994795948267,
+            0.027452928945422173,
+            -0.00779719790443778,
+            0.003546592779457569,
+            -0.015566092915832996,
+            0.0003175129240844399,
+            0.022853855043649673,
+            0.0016370931407436728,
+            0.0020784272346645594,
+            0.013337312266230583,
+            0.029320860281586647,
+            -0.022471778094768524,
+            -0.023929331451654434,
+            0.003482913365587592,
+            0.007185167167335749,
+            -0.010570792481303215,
+            0.0038738346192985773,
+            0.006275965832173824,
+            -0.006007096730172634,
+            0.008900974877178669,
+            0.006332569755613804,
+            -0.0003166284877806902,
+            0.0039198254235088825,
+            -0.00308845448307693,
+            0.012962310574948788,
+            -0.02702839858829975,
+            0.036849189549684525,
+            0.004439874552190304,
+            0.010075507685542107,
+            0.015268922783434391,
+            0.001627364312298596,
+            0.004917470272630453,
+            0.005869124550372362,
+            0.010188715532422066,
+            0.018636859953403473,
+            -0.00956607237458229,
+            -0.027481229975819588,
+            -0.03166992589831352,
+            0.020363280549645424,
+            -0.01173117384314537,
+            0.0035872769076377153,
+            0.0074646491557359695,
+            -0.01828308403491974,
+            -0.007507102098315954,
+            -0.009976450353860855,
+            0.015764206647872925,
+            -0.03189634159207344,
+            0.012990612536668777,
+            -0.0043231286108493805,
+            -0.028669914230704308,
+            0.00851889792829752,
+            -0.010181640274822712,
+            -0.0045884596183896065,
+            -0.0035094465129077435,
+            -0.029264256358146667,
+            -0.007931631989777088,
+            0.03908504545688629,
+            -0.02141045406460762,
+            0.024184048175811768,
+            0.008257105015218258,
+            -0.00014449491573031992,
+            -0.0040365708991885185,
+            -0.013563727959990501,
+            0.005331386812031269,
+            -0.01778779923915863,
+            0.025457637384533882,
+            0.010294848121702671,
+            -0.013924578204751015,
+            0.0322076641023159,
+            -0.011179285123944283,
+            0.027438776567578316,
+            -0.018580254167318344,
+            0.036198243498802185,
+            0.007422196213155985,
+            0.00674648629501462,
+            0.01974063739180565,
+            -0.004025958012789488,
+            0.008228803053498268,
+            0.04559450224041939,
+            -0.018523650243878365,
+            0.02460857853293419,
+            0.020349130034446716,
+            -0.00022685811563860625,
+            0.017915157601237297,
+            0.0028390432707965374,
+            -0.023165177553892136,
+            0.009495317004621029,
+            -0.01135617308318615,
+            -0.009955224581062794,
+            0.01583496294915676,
+            -0.019316107034683228,
+            -0.018721764907240868,
+            -0.025613298639655113,
+            0.0029416377656161785,
+            -0.003350247861817479,
+            -0.0077618202194571495,
+            -0.0019616815261542797,
+            0.01454722136259079,
+            0.008724087849259377,
+            0.025669902563095093,
+            0.0008326975512318313,
+            0.011971740983426571,
+            0.01511326152831316,
+            -0.00035576484515331686,
+            0.02313687466084957,
+            -0.03229256719350815,
+            0.01246702577918768,
+            -0.004517704714089632,
+            -0.00509789539501071,
+            0.023858575150370598,
+            0.0003747802402358502,
+            -0.0004877670726273209,
+            -0.0024888061452656984,
+            0.0005496777012012899,
+            -0.014589674770832062,
+            -0.0057099261321127415,
+            0.000907432462554425,
+            -0.005175726022571325,
+            0.04299071803689003,
+            -0.00701535539701581,
+            -0.004772422835230827,
+            -0.039169952273368835,
+            -0.01918874867260456,
+            0.00623351288959384,
+            -0.01583496294915676,
+            -0.018778368830680847,
+            -0.006307805422693491,
+            0.005801907740533352,
+            -0.04893413558602333,
+            0.008342010900378227,
+            0.03336804360151291,
+            -0.012219383381307125,
+            0.00259493850171566,
+            -0.010981171391904354,
+            -0.006969364359974861,
+            -0.0029115669894963503,
+            -7.766463386360556e-05,
+            -0.002412744564935565,
+            -0.020830262452363968,
+            -0.005617944523692131,
+            0.003364398842677474,
+            -0.0008795727044343948,
+            -0.025047259405255318,
+            0.0006177793256938457,
+            0.018849123269319534,
+            -0.003979966975748539,
+            -0.0011152751976624131,
+            -0.001825478277169168,
+            0.006399787031114101,
+            -0.01508495956659317,
+            0.008009462617337704,
+            -0.0076344613917171955,
+            0.003841994795948267,
+            -0.0030495391692966223,
+            -0.002214630600064993,
+            0.02844349853694439,
+            0.021594416350126266,
+            -0.04808507859706879,
+            -0.01405901275575161,
+            0.023009516298770905,
+            0.007280686404556036,
+            -0.004740582779049873,
+            0.01178777776658535,
+            -0.031613320112228394,
+            -0.00028943203506059945,
+            -0.026787832379341125,
+            0.02200479619204998,
+            -0.009205221198499203,
+            -0.007386818993836641,
+            0.016967041417956352,
+            -0.026632171124219894,
+            0.016344398260116577,
+            0.00794578343629837,
+            -0.00030181417241692543,
+            -0.000493515923153609,
+            0.0063856360502541065,
+            0.0008972614887170494,
+            -0.01659911684691906,
+            0.006442239973694086,
+            -0.0048502529971301556,
+            0.007959933951497078,
+            -0.005087282508611679,
+            -0.05527378246188164,
+            -0.009933997876942158,
+            -0.005497660953551531,
+            -0.008936353027820587,
+            -0.012820800766348839,
+            0.008136821910738945,
+            0.02827368676662445,
+            0.01606137864291668,
+            -0.009863242506980896,
+            -4.8312376748071983e-05,
+            0.033000119030475616,
+            -0.0008300442132167518,
+            0.009955224581062794,
+            -0.011349096894264221,
+            0.00015709814033471048,
+            -0.0006911876262165606,
+            -0.014660430140793324,
+            -0.019061388447880745,
+            0.024042539298534393,
+            -0.028344441205263138,
+            0.02378782071173191,
+            -0.01843874529004097,
+            -0.0130825936794281,
+            0.01757553406059742,
+            -0.032575588673353195,
+            -0.005108508747071028,
+            0.007450498174875975,
+            0.0090566361322999,
+            0.021608568727970123,
+            0.0032794929575175047,
+            0.020589696243405342,
+            -0.03716050833463669,
+            -0.009134466759860516,
+            -0.010160413570702076,
+            -0.00794578343629837,
+            -0.010231168940663338,
+            -0.011596739292144775,
+            0.021863285452127457,
+            -0.015240620821714401,
+            -0.016952890902757645,
+            0.007054270710796118,
+            0.0030212372075766325,
+            -0.011313719674944878,
+            0.015750056132674217,
+            0.00017810352437663823,
+            0.004712280817329884,
+            0.013011839240789413,
+            -0.0008923970744945109,
+            0.04086806997656822,
+            0.0253019779920578,
+            -0.0030212372075766325,
+            -0.008568426594138145,
+            0.021438755095005035,
+            0.0004285097820684314,
+            0.0150000536814332,
+            0.009375033900141716,
+            0.023264234885573387,
+            -0.02092931978404522,
+            0.009997677057981491,
+            -0.005501199048012495,
+            0.014603826217353344,
+            -0.009495317004621029,
+            -0.015339677222073078,
+            0.00479718716815114,
+            -0.01577835902571678,
+            0.010153338313102722,
+            -0.010860888287425041,
+            0.008221727795898914,
+            0.01913214474916458,
+            0.002467579673975706,
+            -0.00941041111946106,
+            0.01146230474114418,
+            0.02075950801372528,
+            -0.025641601532697678,
+            0.008851447142660618,
+            -0.0031998935155570507,
+            -0.0021916350815445185,
+            0.021127434447407722,
+            0.010301923379302025,
+            -0.01594817079603672,
+            0.01573590561747551,
+            -0.008879749104380608,
+            -0.0015053119277581573,
+            -0.01882082223892212,
+            -0.03291521221399307,
+            -0.0015283073298633099,
+            0.012396270409226418,
+            0.004800724796950817,
+            0.0150000536814332,
+            0.011264191009104252,
+            -0.0008950503543019295,
+            -0.006601438857614994,
+            0.0008433107868768275,
+            -0.013485897332429886,
+            -0.004963461309671402,
+            0.010231168940663338,
+            0.017051948234438896,
+            -0.0037217114586383104,
+            0.015650998800992966,
+            0.0088160689920187,
+            0.006137993652373552,
+            -0.0010480580385774374,
+            0.0036138100549578667,
+            -0.002060738392174244,
+            -0.007783046457916498,
+            0.019938750192523003,
+            0.019075538963079453,
+            -0.018509499728679657,
+            -0.0024870370980352163,
+            -0.02141045406460762,
+            0.025882167741656303,
+            0.022188758477568626,
+            0.011165134608745575,
+            -0.04222656786441803,
+            0.00012050455552525818,
+            0.00752832880243659,
+            -0.0013744152383878827,
+            -0.031103884801268578,
+            0.008172199130058289,
+            0.005221716593950987,
+            0.004330204334110022,
+            0.020844414830207825,
+            -0.018424594774842262,
+            0.010853812098503113,
+            0.0070861103013157845,
+            -0.0005766529939137399,
+            0.01019579079002142,
+            -0.0073372903279960155,
+            0.008688710629940033,
+            0.038717120885849,
+            -0.018410442396998405,
+            0.005303084850311279,
+            0.0062370505183935165,
+            0.023320838809013367,
+            -0.011228813789784908,
+            -0.00031485961517319083,
+            0.00800238735973835,
+            0.021608568727970123,
+            -0.012934008613228798,
+            0.005883275531232357,
+            0.067075714468956,
+            -0.012912781909108162,
+            0.031613320112228394,
+            0.020533092319965363,
+            -0.004970536567270756,
+            -0.0104929618537426,
+            -0.012445799075067043,
+            -0.0181132722645998,
+            0.021283095702528954,
+            0.01175947580486536,
+            0.003419233951717615,
+            0.01941516436636448,
+            -0.01610383205115795,
+            0.006636816076934338,
+            0.017547233030200005,
+            0.012601460330188274,
+            -0.007379743270576,
+            -0.0036173479165881872,
+            0.019004784524440765,
+            0.018509499728679657,
+            -0.0030849166214466095,
+            0.00019424450874794275,
+            0.01508495956659317,
+            -0.022075550630688667,
+            -0.012530704960227013,
+            0.01644345559179783,
+            -0.0037358624394983053,
+            0.017773648723959923,
+            -0.02227366529405117,
+            0.024198200553655624,
+            -0.004531855694949627,
+            -0.017179306596517563,
+            0.004698129836469889,
+            -0.007514177821576595,
+            0.022033097222447395,
+            -0.001578720286488533,
+            0.004857328720390797,
+            0.000821199850179255,
+            -0.0018015984678640962,
+            0.01349297259002924,
+            0.002826661104336381,
+            -0.0005120891146361828,
+            0.010507113300263882,
+            -0.0009702275274321437,
+            -0.0013496510218828917,
+            0.024509521201252937,
+            1.9374700059415773e-05,
+            0.005989408120512962,
+            0.0032936439383774996,
+            -0.009261825121939182,
+            -0.004429261200129986,
+            -0.004443412180989981,
+            0.0011842612875625491,
+            0.003877372248098254,
+            -0.007029506377875805,
+            -0.02384442463517189,
+            0.022471778094768524,
+            -0.008540124632418156,
+            0.01175947580486536,
+            0.011349096894264221,
+            0.003665107535198331,
+            0.006824316922575235,
+            -0.009771261364221573,
+            -0.017320817336440086,
+            0.0006633278680965304,
+            -0.01617458648979664,
+            0.006912760436534882,
+            -0.001656550681218505,
+            0.0020235921256244183,
+            0.009445788338780403,
+            -0.002570174168795347,
+            0.0015123875346034765,
+            0.011306644417345524,
+            0.011497682891786098,
+            0.003254728624597192,
+            0.004577846731990576,
+            -0.018721764907240868,
+            -0.020915169268846512,
+            0.005288933869451284,
+            0.013924578204751015,
+            0.004273600410670042,
+            0.0015309606678783894,
+            0.017603836953639984,
+            -0.006166295614093542,
+            -0.022967062890529633,
+            -0.010117961093783379,
+            -0.01584911346435547,
+            0.039396367967128754,
+            0.01800006441771984,
+            -0.02525952458381653,
+            -0.000530220044311136,
+            -0.009728807955980301,
+            0.010450509376823902,
+            -0.00404364662244916,
+            0.014575524255633354,
+            0.005476434715092182,
+            -0.02426895499229431,
+            -0.016797229647636414,
+            -0.010662773624062538,
+            -0.005257094278931618,
+            0.020080260932445526,
+            0.027014248073101044,
+            -0.004956385586410761,
+            0.0061026159673929214,
+            0.019584976136684418,
+            -0.010938717983663082,
+            0.009849091991782188,
+            0.020957622677087784,
+            0.0031698227394372225,
+            -0.007570781745016575,
+            -0.004553082399070263,
+            -0.016514210030436516,
+            0.024467067793011665,
+            0.014250051230192184,
+            -0.005104971118271351,
+            -0.02465103194117546,
+            0.004110863897949457,
+            -0.02428310550749302,
+            -0.01616043597459793,
+            -0.007301913108676672,
+            -0.014143918640911579,
+            -0.004931621253490448,
+            0.0027594438288360834,
+            -0.004836102016270161,
+            0.011278342455625534,
+            -0.0031592093873769045,
+            -0.004531855694949627,
+            -0.0041851564310491085,
+            -0.025584997609257698,
+            -0.02524537406861782,
+            0.007691065315157175,
+            -0.003005317412316799,
+            0.005596718285232782,
+            0.000650945701636374,
+            -0.013400991447269917,
+            -0.011589664034545422,
+            0.00970758218318224,
+            -0.007988235913217068,
+            0.012813725508749485,
+            0.0031149876303970814,
+            -0.009183995425701141,
+            0.0017193458043038845,
+            -0.0012311364989727736,
+            -0.0059752571396529675,
+            0.019924599677324295,
+            0.00916984397917986,
+            0.011724098585546017,
+            -0.03118879161775112,
+            -0.0021633331198245287,
+            0.008349086157977581,
+            0.01828308403491974,
+            -0.02244347706437111,
+            -0.008313708938658237,
+            0.005179263651371002,
+            -0.011228813789784908,
+            -0.013507124036550522,
+            -0.028952933847904205,
+            -0.0030460015404969454,
+            8.175515540642664e-05,
+            -0.018636859953403473,
+            -0.029886899515986443,
+            0.0009038947173394263,
+            -0.007860877551138401,
+            0.016882136464118958,
+            -0.006643891800194979,
+            0.01790100708603859,
+            0.00870286114513874,
+            0.0036173479165881872,
+            0.011554286815226078,
+            0.0025082635693252087,
+            -0.013266556896269321,
+            0.008426916785538197,
+            -0.02671707607805729,
+            -0.01516986545175314,
+            0.009792488068342209,
+            0.014214673079550266,
+            -0.0026126273442059755,
+            5.074458022136241e-05,
+            0.0058372849598526955,
+            0.024254804477095604,
+            0.013584954664111137,
+            0.012474101036787033,
+            0.003042463678866625,
+            -0.005303084850311279,
+            0.005568416323512793,
+            0.0227123461663723,
+            0.008603804744780064,
+            0.004011806566268206,
+            0.007592008449137211,
+            0.00389859895221889,
+            0.0009932229295372963,
+            -0.005943417549133301,
+            -0.00843399204313755,
+            -0.021537812426686287,
+            -0.008908051066100597,
+            0.019938750192523003,
+            -0.010422207415103912,
+            -0.03005671128630638,
+            0.0006023016758263111,
+            -0.0032777239102870226,
+            0.007988235913217068,
+            0.02595292218029499,
+            -0.014172220602631569,
+            0.004970536567270756,
+            0.005536576267331839,
+            -0.007910405285656452,
+            -0.00614860700443387,
+            0.022967062890529633,
+            0.02639160305261612,
+            -0.0034404604230076075,
+            -0.04800017178058624,
+            -0.01671232469379902,
+            0.004305440001189709,
+            -0.007768895477056503,
+            -0.034047290682792664,
+            -0.015424583107233047,
+            0.002757675014436245,
+            -0.03206615149974823,
+            0.010068432427942753,
+            -0.012948159128427505,
+            -0.01214155275374651,
+            0.020264223217964172,
+            0.03223596513271332,
+            0.005154499784111977,
+            0.00014847487909719348,
+            -0.0214670579880476,
+            -0.0021686397958546877,
+            -0.021382151171565056,
+            0.010174565017223358,
+            -0.008801918476819992,
+            -0.010584943927824497,
+            0.0059611061587929726,
+            -0.012191081419587135,
+            0.013266556896269321,
+            0.010797208175063133,
+            0.024778390303254128,
+            0.01584911346435547,
+            -0.01876421831548214,
+            -0.0024746551644057035,
+            -0.01108730398118496,
+            -0.0016450530383735895,
+            -0.009820790030062199,
+            0.006360871717333794,
+            -0.0031592093873769045,
+            0.011264191009104252,
+            0.015184016898274422,
+            0.004432798828929663,
+            0.00046786724124103785,
+            0.038151077926158905,
+            -2.4487851987942122e-05,
+            -0.0003418349369894713,
+            0.006803090218454599,
+            -0.015750056132674217,
+            0.008837295696139336,
+            0.012191081419587135,
+            0.005051904823631048,
+            -0.007698140572756529,
+            0.006739411037415266,
+            0.03099067695438862,
+            -0.01516986545175314,
+            -0.009014182724058628,
+            -0.023504801094532013,
+            0.007549555506557226,
+            -0.0004656561359297484,
+            0.029971804469823837,
+            0.020533092319965363,
+            -0.027509532868862152,
+            -0.016018925234675407,
+            -0.02179253101348877,
+            0.01267929095774889,
+            -0.011497682891786098,
+            -0.003704022616147995,
+            0.012240610085427761,
+            0.02367461286485195,
+            0.0018838511314243078,
+            -0.014455240219831467,
+            -0.008950503543019295,
+            0.008943428285419941,
+            0.014617976732552052,
+            0.003497064346447587,
+            0.007570781745016575,
+            0.00743634719401598,
+            0.008886824361979961,
+            -0.014801939949393272,
+            -0.012254760600626469,
+            0.00767691433429718,
+            -0.010202866978943348,
+            0.01794346049427986,
+            -0.009403335861861706,
+            0.016768928617239,
+            -0.004645063541829586,
+            -0.011964665725827217,
+            -0.0320095494389534,
+            -0.002292460994794965,
+            0.005037753842771053,
+            0.004627374932169914,
+            0.0034705311991274357,
+            -0.0022217060904949903,
+            -0.013160424306988716,
+            0.004673365969210863,
+            0.005073131527751684,
+            1.875283123808913e-05,
+            -0.008313708938658237,
+            -0.00479718716815114,
+            0.004252373706549406,
+            -0.001031253719702363,
+            -0.0073372903279960155,
+            -0.0059858704917132854,
+            0.018481198698282242,
+            0.0066509670577943325,
+            0.00800238735973835,
+            0.03234917297959328,
+            0.02974538877606392,
+            -0.0016353242099285126,
+            -0.004606148693710566,
+            -0.00023304918431676924,
+            -0.01589156687259674,
+            -0.0057205394841730595,
+            -0.023320838809013367,
+            0.022259512916207314,
+            -0.009665128774940968,
+            0.021820832043886185,
+            0.019457615911960602,
+            0.0039304387755692005,
+            -0.008391539566218853,
+            -0.0011586126638576388,
+            -0.007308988366276026,
+            -0.003560743760317564,
+            0.016641568392515182,
+            0.006742948666214943,
+            0.01226891204714775,
+            0.00030933189555071294,
+            -0.028401045128703117,
+            0.026405755430459976,
+            0.000579748535528779,
+            0.0034015451092272997,
+            0.008908051066100597,
+            0.014129767194390297,
+            0.013209952972829342,
+            -0.01235381793230772,
+            0.01768874190747738,
+            -0.03280200436711311,
+            -0.0033484790474176407,
+            -0.016500059515237808,
+            0.002076658420264721,
+            -0.0018520113080739975,
+            -0.0008145665633492172,
+            -0.05354736000299454,
+            0.0003842879377771169,
+            0.005313698202371597,
+            -0.015339677222073078,
+            0.029349161311984062,
+            -0.020589696243405342,
+            -0.017165156081318855,
+            -0.032858606427907944,
+            -0.010238244198262691,
+            0.018099121749401093,
+            0.005617944523692131,
+            0.00035178486723452806,
+            -0.02660386823117733,
+            0.015750056132674217,
+            -0.0047476585023105145,
+            -0.00743634719401598,
+            -0.029858596622943878,
+            -0.010938717983663082,
+            0.009403335861861706,
+            0.004485865123569965,
+            -0.0130825936794281,
+            0.009898620657622814,
+            -0.002890340518206358,
+            0.010867963545024395,
+            -0.0138467475771904,
+            -0.00420284504070878,
+            0.0010020672343671322,
+            -0.005363226868212223,
+            0.012615610845386982,
+            -0.030537845566868782,
+            0.011391550302505493,
+            -0.01352127455174923,
+            0.0008548084879294038,
+            -0.017858553677797318,
+            -0.027707645669579506,
+            0.0020288988016545773,
+            -0.016132133081555367,
+            0.039990708231925964,
+            0.010188715532422066,
+            -0.01481609046459198,
+            0.016995344310998917,
+            -0.002080196049064398,
+            0.017377421259880066,
+            0.0014955832157284021,
+            0.017023645341396332,
+            -0.011971740983426571,
+            0.021056678146123886,
+            -0.0041745430789887905,
+            0.02156611531972885,
+            0.00600002147257328,
+            0.034047290682792664,
+            -0.02033497765660286,
+            0.0070825726725161076,
+            0.003792466362938285,
+            0.005798369646072388,
+            -0.0026816134341061115,
+            0.021424604579806328,
+            0.017985913902521133,
+            0.019698183983564377,
+            0.0037110981065779924,
+            -0.024084990844130516,
+            -0.006137993652373552,
+            0.004867942072451115,
+            0.023858575150370598,
+            0.011150983162224293,
+            -0.016415152698755264,
+            -0.021056678146123886,
+            0.015226469375193119,
+            -0.007860877551138401,
+            0.006771250627934933,
+            -0.00851889792829752,
+            0.00037544354563578963,
+            -0.013259481638669968,
+            0.012233533896505833,
+            -0.012523629702627659,
+            0.0032335021533071995,
+            0.01043635793030262,
+            -0.027806703001260757,
+            0.01511326152831316,
+            -0.00371817359700799,
+            -0.014313730411231518,
+            0.023320838809013367,
+            -0.015877414494752884,
+            -0.013988257385790348,
+            0.005501199048012495,
+            0.008610880002379417,
+            0.01746232621371746,
+            -0.002761212643235922,
+            0.002669231267645955,
+            -0.020504789426922798,
+            -0.012445799075067043,
+            0.004591997712850571,
+            0.02210385352373123,
+            -0.015820810571312904,
+            0.012431648559868336,
+            -0.018099121749401093,
+            0.01364863384515047,
+            0.016202887520194054,
+            0.007149789482355118,
+            0.00573115237057209,
+            -0.007726442534476519,
+            -0.0011471149045974016,
+            0.032151058316230774,
+            -0.0029876285698264837,
+            0.02704254910349846,
+            -0.014915147796273232,
+            0.009615600109100342,
+            0.00518987700343132,
+            -0.013089669868350029,
+            -0.014589674770832062,
+            0.011108530685305595,
+            0.02428310550749302,
+            -0.009530694223940372,
+            -0.023660462349653244,
+            0.0018484735628589988,
+            0.014264201745390892,
+            0.010337301529943943,
+            0.009403335861861706,
+            -0.001997058978304267,
+            -0.0037358624394983053,
+            0.01484439242631197,
+            -0.0032918748911470175,
+            -0.0286557637155056,
+            -0.01633024774491787,
+            -0.004400959238409996,
+            -0.017490629106760025,
+            0.020745357498526573,
+            0.02367461286485195,
+            0.010507113300263882,
+            0.0037570889107882977,
+            0.005540114361792803,
+            0.018297234550118446,
+            0.01843874529004097,
+            0.013330236077308655,
+            0.009304278530180454,
+            0.012700516730546951,
+            -0.004319590982049704,
+            0.02800481766462326,
+            -0.010860888287425041,
+            -0.01046465989202261,
+            -0.015707602724432945,
+            0.0016636261716485023,
+            -0.005193414632230997,
+            -0.012431648559868336,
+            -0.0069481381215155125,
+            0.004369119182229042,
+            0.013330236077308655,
+            -0.012240610085427761,
+            -0.007149789482355118,
+            0.01016748882830143,
+            0.011971740983426571,
+            0.01403071079403162,
+            0.01719345711171627,
+            -0.01633024774491787,
+            -0.011278342455625534,
+            0.0009834941010922194,
+            -0.02844349853694439,
+            -0.0160047747194767,
+            -0.01767459139227867,
+            0.009580222889780998,
+            -0.014469391666352749,
+            0.03724541515111923,
+            0.028471799567341805,
+            0.0073514413088560104,
+            0.031613320112228394,
+            -0.028401045128703117,
+            0.00970758218318224,
+            -0.014398636296391487,
+            0.007132100872695446,
+            -0.004188694059848785,
+            0.003902136581018567,
+            0.002448122017085552,
+            0.020419884473085403,
+            0.00791748147457838,
+            0.014264201745390892,
+            0.008150972425937653,
+            -0.021934039890766144,
+            -0.00662620272487402,
+            -0.012559006921947002,
+            -0.016401002183556557,
+            -0.02595292218029499,
+            0.013719388283789158,
+            0.005087282508611679,
+            -0.003074303502216935,
+            -0.0002704166399780661,
+            0.006566061172634363,
+            -0.0013195802457630634,
+            0.010443433187901974,
+            -0.027608588337898254,
+            -0.01311089564114809,
+            0.0069269114173948765,
+            0.01449061743915081,
+            -0.02114158496260643,
+            0.0014637433923780918,
+            0.008900974877178669,
+            -0.004602610599249601,
+            0.035660505294799805,
+            -0.014674580655992031,
+            -0.008610880002379417,
+            0.0027806703001260757,
+            0.005062518175691366,
+            -0.017660440877079964,
+            -0.02319347858428955,
+            0.0070648835971951485,
+            0.011575513519346714,
+            0.010867963545024395,
+            -0.028570856899023056,
+            0.02087271586060524,
+            -0.0054339817725121975,
+            0.03832089155912399,
+            0.00401534466072917,
+            -0.013337312266230583,
+            0.003962278366088867,
+            0.0011860302183777094,
+            -0.012360893189907074,
+            0.002060738392174244,
+            -0.000340508297085762,
+            0.003700484987348318,
+            0.011228813789784908,
+            -0.014801939949393272,
+            -0.016967041417956352,
+            0.016910437494516373,
+            -0.011476456187665462,
+            -0.0035395172890275717,
+            -0.00943871308118105,
+            -0.016669871285557747,
+            0.007308988366276026,
+            0.025047259405255318,
+            0.0041745430789887905,
+            0.001242634141817689,
+            -0.011632117442786694,
+            -0.0038596836384385824,
+            -0.008575502783060074,
+            0.0011718792375177145,
+            -0.0207029040902853,
+            0.027396325021982193,
+            -0.01801421493291855,
+            0.002410975517705083,
+            0.021099131554365158,
+            0.008709936402738094,
+            -0.003944589756429195,
+            0.016316095367074013,
+            0.026363302022218704,
+            0.01205664686858654,
+            0.0031539027113467455,
+            -0.00902833417057991,
+            1.648646139074117e-05,
+            0.014957600273191929,
+            0.0015123875346034765,
+            -0.0015663381200283766,
+            0.0032140444964170456,
+            -0.006583749782294035,
+            -0.001859086798503995,
+            -0.017561383545398712,
+            0.005564878229051828,
+            0.02125479280948639,
+            -0.004132090136408806,
+            0.020476488396525383,
+            0.015509488992393017,
+            0.011625041253864765,
+            0.010726453736424446,
+            -0.0012390963966026902,
+            -0.002150951186195016,
+            -0.0021668709814548492,
+            -0.007393894251435995,
+            -0.018297234550118446,
+            0.006470541935414076,
+            0.0034493047278374434,
+            0.01822648011147976,
+            0.020504789426922798,
+            -0.02308027073740959,
+            0.010146263055503368,
+            0.014985902234911919,
+            -0.013011839240789413,
+            0.0235755555331707,
+            -0.02006610855460167,
+            0.0024605041835457087,
+            0.02156611531972885,
+            0.013337312266230583,
+            -0.014271277002990246,
+            -0.009219372645020485,
+            0.004662752617150545,
+            -0.01543873455375433,
+            0.000872939417604357,
+            -0.01478778850287199,
+            -0.010783057659864426,
+            0.014533070847392082,
+            -0.01490099634975195,
+            0.01229721400886774,
+            -0.006969364359974861,
+            0.0070967236533761024,
+            -0.017165156081318855,
+            0.015014204196631908,
+            0.008632106706500053,
+            0.01757553406059742,
+            0.0021279556676745415,
+            -0.02518877014517784,
+            -0.006290116813033819,
+            0.021169887855648994,
+            0.009509467519819736,
+            -0.02497650496661663,
+            0.019613277167081833,
+            0.0020094411447644234,
+            -0.019528372213244438,
+            -0.022401023656129837,
+            -0.012686366215348244,
+            -0.005055442452430725,
+            -0.017264213413000107,
+            -0.0011276573641225696,
+            -0.024792540818452835,
+            0.003972891718149185,
+            0.009106164798140526,
+            -0.011978816241025925,
+            -0.019273653626441956,
+            -0.0002315014135092497,
+            -0.01040805596858263,
+            0.006219361908733845,
+            -0.01132787112146616,
+            -0.019853845238685608,
+            0.0070436573587358,
+            0.012127402238547802,
+            -0.0027647505048662424,
+            -0.005646246485412121,
+            -0.02562745101749897,
+            -0.006919836159795523,
+            0.020179318264126778,
+            -0.003994117956608534,
+            -0.004974074196070433,
+            -0.01703779585659504,
+            0.01108730398118496,
+            0.011165134608745575,
+            0.020901018753647804,
+            0.010931642726063728,
+            0.005674548447132111,
+            0.013634482398629189,
+            -0.0029416377656161785,
+            -0.007156865205615759,
+            -9.645892532716971e-06,
+            0.004284213297069073,
+            0.0069233737885952,
+            0.0016636261716485023,
+            0.020235922187566757,
+            -0.01692458800971508,
+            0.018934030085802078,
+            -0.012290137819945812,
+            0.008794843219220638,
+            -0.01855195313692093,
+            -0.0003155229496769607,
+            -0.01067692507058382,
+            0.00015002264990471303,
+            -0.0038455326575785875,
+            -0.009544845670461655,
+            1.9595809135353193e-05,
+            0.00143986358307302,
+            0.008511822670698166,
+            -0.016783079132437706,
+            0.009615600109100342,
+            -0.010839661583304405,
+            -0.004308977629989386,
+            -0.019542522728443146,
+            0.017448175698518753,
+            0.013344387523829937,
+            0.007308988366276026,
+            -0.008278331719338894,
+            -0.006300730165094137,
+            -0.02282555401325226,
+            0.008582578040659428,
+            -0.0014637433923780918,
+            0.002214630600064993,
+            -0.028301987797021866,
+            -0.004358506295830011,
+            0.0024587351363152266,
+            0.020306676626205444,
+            0.0033661676570773125,
+            -0.010712302289903164,
+            0.008978805504739285,
+            -0.023377442732453346,
+            -0.0030159305315464735,
+            -0.023349139839410782,
+            -0.004075486212968826,
+            -0.005112046375870705,
+            0.010103809647262096,
+            0.008101443760097027,
+            0.01549533847719431,
+            -0.009898620657622814,
+            0.01478778850287199,
+            -0.004793649073690176,
+            0.012049571610987186,
+            -0.0012152165872976184,
+            0.01337976474314928,
+            -0.0003111007681582123,
+            -0.01505665760487318,
+            0.02158026583492756,
+            -0.008738238364458084,
+            0.001741456682793796,
+            -0.0052853962406516075,
+            0.0027346794959157705,
+            -0.017702894285321236,
+            -0.004553082399070263,
+            -0.014455240219831467,
+            0.01957082375884056,
+            -0.022584985941648483,
+            0.004153316840529442,
+            -0.005055442452430725,
+            -0.016075529158115387,
+            0.029434068128466606,
+            0.022896308451890945,
+            -0.0035483618266880512,
+            0.015070809051394463,
+            0.0017989451298490167,
+            0.0015061964513733983,
+            0.016202887520194054,
+            -0.015806660056114197,
+            -0.013025989755988121,
+            0.035066161304712296,
+            0.0176321379840374,
+            0.0004313841927796602,
+            -0.0023260696325451136,
+            -0.0037323245778679848,
+            0.0032193511724472046,
+            0.008023613132536411,
+            -0.010450509376823902,
+            -0.01141277700662613,
+            -0.017051948234438896,
+            0.009997677057981491,
+            -0.006155682262033224,
+            0.004446949809789658,
+            -0.00025891896802932024,
+            -7.617214578203857e-05,
+            -0.01516986545175314,
+            0.022584985941648483,
+            -0.017717044800519943,
+            -0.00602832343429327,
+            0.004687516950070858,
+            0.0207029040902853,
+            -0.02541518583893776,
+            0.01616043597459793,
+            -0.007107336539775133,
+            -0.013429293408989906,
+            0.012573158368468285,
+            -0.015127412974834442,
+            -0.0107335289940238,
+            -0.003175129182636738,
+            -0.008009462617337704,
+            -0.022754797711968422,
+            0.017929309979081154,
+            -0.013627407141029835,
+            0.015650998800992966,
+            -0.00287972716614604,
+            0.000562502013053745,
+            0.002057200763374567,
+            0.0032582662533968687,
+            0.02250008098781109,
+            0.030735958367586136,
+            0.010415131226181984,
+            -0.005465821363031864,
+            -0.021000074222683907,
+            0.012983537279069424,
+            0.0020589695777744055,
+            0.013202877715229988,
+            -0.002253545681014657,
+            -0.002135031158104539,
+            -0.016613267362117767,
+            -0.023122724145650864,
+            -0.0013010069960728288,
+            -0.01761798746883869,
+            -0.02552839368581772,
+            -0.01774534583091736,
+            -0.007047194987535477,
+            0.011349096894264221,
+            0.020405733957886696,
+            0.0009755341452546418,
+            -0.005384453106671572,
+            0.01559439580887556,
+            -0.006941062398254871,
+            0.002080196049064398,
+            -0.006594363134354353,
+            -0.002032436430454254,
+            -0.008200501091778278,
+            -0.021863285452127457,
+            0.029660483822226524,
+            -0.010103809647262096,
+            0.01141277700662613,
+            -0.005472897086292505,
+            -0.00965097825974226,
+            0.005575491581112146,
+            0.00900003220885992,
+            0.024849144741892815,
+            0.006452853325754404,
+            0.03141520544886589,
+            0.020151015371084213,
+            -0.018891576677560806,
+            -0.0006071660900488496,
+            0.022684043273329735,
+            -0.0037570889107882977,
+            -0.004889168310910463,
+            -0.0031786670442670584,
+            -0.017985913902521133,
+            -0.012007118202745914,
+            0.007839650847017765,
+            -0.013471745885908604,
+            -0.006636816076934338,
+            -0.00837738811969757,
+            0.018070818856358528,
+            -0.017773648723959923,
+            0.025698205456137657,
+            -0.006488230545073748,
+            0.005154499784111977,
+            0.031924642622470856,
+            0.014363259077072144,
+            0.019698183983564377,
+            -0.011794853024184704,
+            -0.031075581908226013,
+            -0.00683846790343523,
+            -0.007882103323936462,
+            0.009558996185660362,
+            -0.01062032114714384,
+            0.005204027984291315,
+            -0.02006610855460167,
+            -0.018948180601000786,
+            0.012863253243267536,
+            -0.017476476728916168,
+            0.013924578204751015,
+            -0.009955224581062794,
+            0.027226513251662254,
+            -0.001228483160957694,
+            0.0121698547154665,
+            -0.002614396158605814,
+            -0.017801949754357338,
+            0.01205664686858654,
+            0.004089637193828821,
+            -0.023943481966853142,
+            0.010337301529943943,
+            -0.001864393474534154,
+            -0.014037786051630974,
+            -0.018410442396998405,
+            -0.001228483160957694,
+            -0.005200490355491638,
+            -0.009658053517341614,
+            0.025457637384533882,
+            0.014101465232670307,
+            0.016231190413236618,
+            0.0035359796602278948,
+            0.0016335553955286741,
+            -0.011073152534663677,
+            0.009573147632181644,
+            -0.006803090218454599,
+            0.000612030504271388,
+            -0.0034404604230076075,
+            0.011893910355865955,
+            -0.004252373706549406,
+            -0.01543873455375433,
+            0.002324300818145275,
+            0.008561351336538792,
+            0.0143278818577528,
+            5.947839599684812e-05,
+            0.009587298147380352,
+            0.0009790719486773014,
+            -0.009764186106622219,
+            -0.00046388726332224905,
+            -0.0060884649865329266,
+            0.0069375247694551945,
+            0.02400008589029312,
+            -0.012516554445028305,
+            -0.01681138016283512,
+            0.014504768885672092,
+            0.004033033270388842,
+            -0.017589686438441277,
+            0.018367990851402283,
+            0.004103788174688816,
+            -0.022358570247888565,
+            0.0035412863362580538,
+            0.010662773624062538,
+            -0.009523618966341019,
+            -0.014469391666352749,
+            -0.0010339069413021207,
+            -0.015509488992393017,
+            0.03130199760198593,
+            0.00025073791039176285,
+            0.016910437494516373,
+            -0.01062032114714384,
+            -0.0025967073161154985,
+            -0.013309009373188019,
+            0.013103820383548737,
+            -0.015750056132674217,
+            -0.007273610681295395,
+            0.00035620704875327647,
+            -0.02298121340572834,
+            0.01741987280547619,
+            0.0013425755314528942,
+            0.00794578343629837,
+            0.003436922561377287,
+            -0.010556641966104507,
+            0.009820790030062199,
+            0.0018237093463540077,
+            0.006668656133115292,
+            -0.014915147796273232,
+            0.023618008941411972,
+            0.006413938011974096,
+            0.011235889047384262,
+            -0.01352127455174923,
+            -0.01546703651547432,
+            0.002253545681014657,
+            0.04134920611977577,
+            -0.01306136790663004,
+            0.0002593612007331103,
+            0.01562269777059555,
+            7.081024523358792e-05,
+            0.004156854469329119,
+            -0.0023755980655550957,
+            -0.0034493047278374434,
+            -0.034585028886795044,
+            0.005370302125811577,
+            -0.0006328147719614208,
+            0.008073141798377037,
+            -0.008992956951260567,
+            -0.005430444143712521,
+            -0.0040684109553694725,
+            -0.008625030517578125,
+            -0.00314859626814723,
+            -0.01478778850287199,
+            0.008363237604498863,
+            -0.014533070847392082,
+            -0.028910480439662933,
+            0.03166992589831352,
+            -0.006887996103614569,
+            -0.00533846253529191,
+            -0.021396303549408913,
+            0.007570781745016575,
+            -0.01370523776859045,
+            0.002663924591615796,
+            0.010967019945383072,
+            0.008865597657859325,
+            0.008738238364458084,
+            0.002762981690466404,
+            0.0009516543359495699,
+            -0.004255911335349083,
+            0.010273621417582035,
+            -0.00835616234689951,
+            -0.0005943417199887335,
+            -0.0014186371117830276,
+            -0.02725481428205967,
+            -0.01405901275575161,
+            -0.010592019185423851,
+            -0.007811348885297775,
+            -0.00435143057256937,
+            0.012608535587787628,
+            0.007174553815275431,
+            -0.0017485321732237935,
+            -0.022938761860132217,
+            -0.014745336025953293,
+            -0.0034103896468877792,
+            0.008349086157977581,
+            -0.01838214136660099,
+            0.01064862310886383,
+            0.0016830838285386562,
+            0.009976450353860855,
+            0.014292503707110882,
+            0.00302831269800663,
+            0.0016742395237088203,
+            0.003458149265497923,
+            0.014886845834553242,
+            0.012601460330188274,
+            -0.005801907740533352,
+            -0.021382151171565056,
+            -0.0024605041835457087,
+            -0.007068421691656113,
+            -0.0034687623847275972,
+            -0.012332591228187084,
+            0.03039633482694626,
+            0.0029805530793964863,
+            0.0176321379840374,
+            0.007832575589418411,
+            0.009573147632181644,
+            0.010507113300263882,
+            0.029377464205026627,
+            0.006049550138413906,
+            -0.0029929352458566427,
+            -0.008603804744780064,
+            -0.02141045406460762,
+            -0.013146273791790009,
+            0.009898620657622814,
+            0.013294858857989311,
+            0.03597182780504227,
+            -0.034896351397037506,
+            0.012162779457867146,
+            -0.011101454496383667,
+            -0.008264180272817612,
+            -0.008533049374818802,
+            0.0044717141427099705,
+            0.010068432427942753,
+            0.014370334334671497,
+            -0.0023402206134051085,
+            -0.01175947580486536,
+            0.009070787578821182,
+            0.023547254502773285,
+            -0.012983537279069424,
+            0.03016991913318634,
+            0.027325568720698357,
+            0.023066120222210884,
+            0.01761798746883869,
+            -0.012537780217826366,
+            0.026462359353899956,
+            -6.484030018327758e-05,
+            0.0006261814851313829,
+            -0.018084971234202385,
+            -0.00587973790243268,
+            -0.004294826649129391,
+            -0.004362043924629688,
+            0.013316085562109947,
+            -0.0025471788831055164,
+            -0.0008468485320918262,
+            0.015099111013114452,
+            -0.0037323245778679848,
+            -3.927453508367762e-05,
+            -0.0059752571396529675,
+            0.015127412974834442,
+            0.0002699744363781065,
+            0.019457615911960602,
+            -0.04270770028233528,
+            0.019726485013961792,
+            -0.021396303549408913,
+            -0.003102605463936925,
+            -0.0020890405867248774,
+            0.009523618966341019,
+            0.00764861237257719,
+            -0.007142714224755764,
+            -0.014334957115352154,
+            0.004524780437350273,
+            0.014066088013350964,
+            -0.0006726144347339869,
+            0.005080206785351038,
+            -0.0027541371528059244,
+            0.0017228835495188832,
+            -0.010089659132063389,
+            -0.005862049292773008,
+            5.842812606715597e-05,
+            -0.010570792481303215,
+            -0.008646257221698761,
+            -0.03232087194919586,
+            0.01062032114714384,
+            0.010209942236542702,
+            0.007294837385416031,
+            0.004252373706549406,
+            -0.022018946707248688,
+            -0.012318440712988377,
+            0.012948159128427505,
+            -0.01132787112146616,
+            -0.002570174168795347,
+            -0.0020005968399345875,
+            0.005387990735471249,
+            0.0004705205501522869,
+            0.010160413570702076,
+            -0.003472300246357918,
+            0.010938717983663082,
+            -0.014603826217353344,
+            0.018084971234202385,
+            0.03229256719350815,
+            -0.00533846253529191,
+            -0.019967053085565567,
+            0.000907432462554425,
+            -0.009077862836420536,
+            0.003813692834228277,
+            0.007641536649316549,
+            -0.021962342783808708,
+            -0.02173592709004879,
+            0.005087282508611679,
+            -0.013153349049389362,
+            -0.007783046457916498,
+            0.014830241911113262,
+            0.005221716593950987,
+            0.010839661583304405,
+            0.01751893013715744,
+            0.010889190249145031,
+            0.00840569008141756,
+            -0.026533113792538643,
+            0.0006734988419339061,
+            -0.012686366215348244,
+            -0.00590803986415267,
+            0.002239394700154662,
+            -0.0017591454088687897,
+            0.004591997712850571,
+            -0.00010115749319083989,
+            -0.011200511828064919,
+            -0.0007057808106765151,
+            0.010903340764343739,
+            0.027919910848140717,
+            -0.004146241117268801,
+            -0.0022570835426449776,
+            -0.014257126487791538,
+            0.0010100272484123707,
+            0.036481261253356934,
+            -0.012275987304747105,
+            -0.017434025183320045,
+            0.01757553406059742,
+            0.027452928945422173,
+            0.005954030901193619,
+            -0.0032529598101973534,
+            -0.0027364485431462526,
+            0.00039600671152584255,
+            0.011978816241025925,
+            0.007719367276877165,
+            -0.002692226553335786,
+            -0.002131493529304862,
+            0.004761809483170509,
+            0.035773713141679764,
+            -0.0015203474322333932,
+            -0.010330225341022015,
+            -0.024240652099251747,
+            0.010974096134305,
+            0.00277182599529624,
+            0.004754733759909868,
+            0.002313687466084957,
+            0.023504801094532013,
+            0.019655730575323105,
+            0.014858543872833252,
+            0.00470520555973053,
+            -0.009771261364221573,
+            0.03005671128630638,
+            -0.008172199130058289,
+            -0.004542469047009945,
+            -0.02838689461350441,
+            -0.005087282508611679,
+            -0.015650998800992966,
+            0.008929276838898659,
+            -0.0002221042668679729,
+            -0.005016527138650417,
+            -0.013422218151390553,
+            0.019726485013961792,
+            0.01064862310886383,
+            -0.011073152534663677,
+            0.012608535587787628,
+            -0.027070851996541023,
+            0.02346234768629074,
+            -0.03826428949832916,
+            -0.0071993181481957436,
+            -0.006421013735234737,
+            -0.010514188557863235,
+            -0.02627839520573616,
+            -0.0030353881884366274,
+            0.007372668012976646,
+            0.014250051230192184,
+            0.0022570835426449776,
+            0.0022818478755652905,
+            0.02282555401325226,
+            0.0016415152931585908,
+            -0.01508495956659317,
+            0.017985913902521133,
+            0.03492465242743492,
+            0.012926933355629444,
+            0.00155837822239846,
+            0.015990622341632843,
+            0.0027789014857262373,
+            0.0021633331198245287,
+            0.00788917951285839,
+            -0.0034493047278374434,
+            -0.007938707247376442,
+            0.008582578040659428,
+            0.006990591064095497,
+            -0.02763689123094082,
+            0.003824306186288595,
+            -0.00010408719390397891,
+            -0.009898620657622814,
+            -0.022429324686527252,
+            0.011568437330424786,
+            0.01913214474916458,
+            -0.004224071744829416,
+            -0.006771250627934933,
+            -0.0043337419629096985,
+            0.0006836698739789426,
+            0.006785401608794928,
+            -0.002412744564935565,
+            -0.000515184598043561,
+            -0.0015176940942183137,
+            -0.02368876338005066,
+            -0.010316074825823307,
+            0.013570803217589855,
+            -0.005175726022571325,
+            0.010358527302742004,
+            0.005242943298071623,
+            -0.018792519345879555,
+            -0.016938740387558937,
+            0.006852618884295225,
+            -0.016188737004995346,
+            0.01364863384515047,
+            0.00027948213391937315,
+            -0.0023172253277152777,
+            0.024962352588772774,
+            0.03594352304935455,
+            -0.013188726268708706,
+            0.005119122099131346,
+            -0.019386861473321915,
+            -0.0245378240942955,
+            -0.019004784524440765,
+            0.022613288834691048,
+            0.008306633681058884,
+            -0.018792519345879555,
+            0.012304289266467094,
+            -0.002752368338406086,
+            -0.008865597657859325,
+            -0.0066721937619149685,
+            -0.006845543161034584,
+            -0.008547200821340084,
+            0.020533092319965363,
+            -0.004935159347951412,
+            0.0038950610905885696,
+            0.019047237932682037,
+            0.04075486212968826,
+            -0.008292482234537601,
+            -0.01349297259002924,
+            0.0054339817725121975,
+            0.004878554958850145,
+            0.0018626245437189937,
+            -0.004814875777810812,
+            -0.008016537874937057,
+            -0.0185944065451622,
+            -0.01740572229027748,
+            -0.009615600109100342,
+            -0.0020041344687342644,
+            0.020122714340686798,
+            -0.007768895477056503,
+            -0.01354957651346922,
+            -0.0245378240942955,
+            -0.028188779950141907,
+            -0.006580212153494358,
+            0.02319347858428955,
+            0.017179306596517563,
+            -0.00013200224202591926,
+            -4.3337418901501223e-05,
+            -0.012042495422065258,
+            -0.01968403346836567,
+            0.015509488992393017,
+            0.00037876018905080855,
+            -0.01773119531571865,
+            -0.0065448349341750145,
+            -0.015919867902994156,
+            0.005787756759673357,
+            -0.0051368107087910175,
+            0.004574308637529612,
+            0.05057565122842789,
+            0.0005271245026960969,
+            0.008441068232059479,
+            -0.026575567200779915,
+            0.009106164798140526,
+            -0.01583496294915676,
+            -0.017377421259880066,
+            -0.00012702727690339088,
+            -0.022301966324448586,
+            -0.0030955299735069275,
+            -0.018792519345879555,
+            -0.007386818993836641,
+            0.008568426594138145,
+            0.009799563325941563,
+            -0.013896276243031025,
+            -0.013669860549271107,
+            -0.024693485349416733,
+            -0.0035872769076377153,
+            -0.001539805089123547,
+            0.009311353787779808,
+            0.007825499400496483,
+            -0.03693409264087677,
+            0.002536565763875842,
+            -0.0009507699287496507,
+            0.0005779796629212797,
+            0.01208494883030653,
+            -0.008667483925819397,
+            -0.003063690150156617,
+            -0.0214670579880476,
+            0.006452853325754404,
+            0.007103798910975456,
+            -0.013500047847628593,
+            0.012375044636428356,
+            -0.012863253243267536,
+            -0.007669838611036539,
+            -0.0005426021525636315,
+            0.03379257395863533,
+            0.011794853024184704,
+            -0.0025153392925858498,
+            -0.00654129683971405,
+            -0.011150983162224293,
+            0.02654726430773735,
+            -0.011349096894264221,
+            -0.026405755430459976,
+            0.028910480439662933,
+            0.013245330192148685,
+            0.0005974372616037726,
+            0.020151015371084213,
+            -0.0017936384538188577,
+            -0.01397410687059164,
+            0.009601449593901634,
+            0.004266524687409401,
+            0.012424572370946407,
+            -0.00014460546663030982,
+            0.0214670579880476,
+            0.0018555490532889962,
+            0.0027399861719459295,
+            -0.0036226543597877026,
+            0.004687516950070858,
+            -0.03093407303094864,
+            -0.0006549257086589932,
+            -0.003102605463936925,
+            -0.008398614823818207,
+            0.011207587085664272,
+            -0.002421588869765401,
+            0.006558985915035009,
+            -0.022188758477568626,
+            -0.028485950082540512,
+            0.011342021636664867,
+            -0.016231190413236618,
+            -0.021396303549408913,
+            0.015339677222073078,
+            -0.0007641536649316549,
+            0.00614860700443387,
+            -0.007443422917276621,
+            -0.023278385400772095,
+            -0.019386861473321915,
+            0.02064630016684532,
+            0.012212308123707771,
+            -0.01190098561346531,
+            0.012750045396387577,
+            -0.021113283932209015,
+            -0.012948159128427505,
+            -0.001894464367069304,
+            0.0090566361322999,
+            -0.0019864458590745926,
+            0.01325240544974804,
+            -0.012198156677186489,
+            -0.00334494118578732,
+            0.0031592093873769045,
+            -0.011589664034545422,
+            0.008158047683537006,
+            0.01978308893740177,
+            0.008193425834178925,
+            -0.009339655749499798,
+            0.003422771580517292,
+            -0.011922212317585945,
+            0.013500047847628593,
+            0.015537790954113007,
+            0.018311386927962303,
+            -0.006498843897134066,
+            0.0020412809681147337,
+            -0.01370523776859045,
+            0.0063962494023144245,
+            -0.009827865287661552,
+            -0.004878554958850145,
+            -0.0005333155859261751,
+            0.004921008367091417,
+            -0.011179285123944283,
+            -0.0025772498920559883,
+            -0.026830283924937248,
+            -0.0013823752524331212,
+            -0.020533092319965363,
+            0.005303084850311279,
+            0.023207630962133408,
+            -0.007514177821576595,
+            0.0006045127520337701,
+            -0.0039092120714485645,
+            0.014476466923952103,
+            0.020688753575086594,
+            0.00870286114513874,
+            0.00482548912987113,
+            -0.006081389728933573,
+            0.02179253101348877,
+            0.011603815481066704,
+            -0.017816102132201195,
+            0.00164239970035851,
+            -0.001509734196588397,
+            -0.0014336726162582636,
+            0.011625041253864765,
+            0.004443412180989981,
+            -0.020844414830207825,
+            0.0028602697420865297,
+            -0.010089659132063389,
+            0.007818424142897129,
+            0.014702882617712021,
+            -0.00031906069489195943,
+            -0.02486329711973667,
+            -0.009827865287661552,
+            0.021919889375567436,
+            -0.015070809051394463,
+            -0.0008291598060168326,
+            0.00764861237257719,
+            0.023264234885573387,
+            -0.024792540818452835,
+            -0.006831392180174589,
+            -0.0023455272894352674,
+            -0.003933976404368877,
+            -0.005193414632230997,
+            0.0010489424457773566,
+            0.0214670579880476,
+            0.004415110219269991,
+            -0.004234685096889734,
+            0.005724077112972736,
+            0.006470541935414076,
+            0.02325008250772953,
+            0.0040577976033091545,
+            0.028047269210219383,
+            -0.00943871308118105,
+            -0.0023809047415852547,
+            -0.0019988277927041054,
+            -4.775960405822843e-05,
+            0.0160047747194767,
+            -0.005430444143712521,
+            0.020320827141404152,
+            0.009445788338780403,
+            0.0036934094969183207,
+            0.016514210030436516,
+            0.02319347858428955,
+            -0.006767712999135256,
+            -0.004793649073690176,
+            0.019160445779561996,
+            0.0053667644970119,
+            -0.008858522400259972,
+            -0.02194819226861,
+            0.01141277700662613,
+            0.0014124460285529494,
+            -0.0006956097786314785,
+            0.012106175534427166,
+            0.0024375086650252342,
+            0.006300730165094137,
+            0.008172199130058289,
+            -0.016684021800756454,
+            0.004241760354489088,
+            -0.004698129836469889,
+            -0.01860855706036091,
+            0.010910416021943092,
+            -0.012092024087905884,
+            -5.201595922699198e-05,
+            -0.0010002984199672937,
+            0.014165145345032215,
+            -0.01703779585659504,
+            -0.0019103841623291373,
+            -0.0070719593204557896,
+            -1.3024719009990804e-05,
+            -0.0009162768837995827,
+            0.008320784196257591,
+            -0.006254739128053188,
+            -0.010938717983663082,
+            -0.010938717983663082,
+            0.027509532868862152,
+            -0.004991763271391392,
+            0.003916287794709206,
+            0.019330257549881935
+          ],
+          "embedding": [
+            0.007204997251390758,
+            -0.03173324616100502,
+            -0.020505790884701024,
+            0.0037451608231038273,
+            0.009817085391002229,
+            0.026883177166072393,
+            -0.016477826654817607,
+            0.06949311853773875,
+            -0.01224876667472692,
+            0.031952566873802865,
+            0.013633671951171586,
+            -0.005041437355104989,
+            0.02663392750069342,
+            -0.022099132865267587,
+            0.006665702000169926,
+            0.0030467129715919033,
+            0.007931802508415542,
+            0.009984477355413895,
+            -0.010362836464183985,
+            -0.03180848181251013,
+            -0.013542377333474449,
+            -0.013572345022577053,
+            -0.03979985300694044,
+            -0.007698412130292187,
+            -0.005953452234356532,
+            0.018890228917932694,
+            0.02217446994660065,
+            0.005158357501784182,
+            -0.05468694715835175,
+            0.00789994048345594,
+            0.017130184174786147,
+            -0.0026591475669416114,
+            -0.02081141846859738,
+            -0.008403928318906045,
+            -0.030925346724945978,
+            -0.03855023691059801,
+            0.038853354785097736,
+            0.01612011653434478,
+            -0.035635702395196575,
+            0.026266623757525934,
+            0.043450373922427316,
+            -0.007254334835408145,
+            -0.015376804869420366,
+            -0.0017276202867766907,
+            0.004650414207094434,
+            -0.002365514361535186,
+            0.011906139905776183,
+            -0.0005664194302009289,
+            -0.0191723447249157,
+            -0.025546124518037998,
+            0.01025005257280377,
+            -0.014436532079436437,
+            0.029277529346694118,
+            -0.0045414983847733344,
+            -0.02260315456913697,
+            0.01717918656248626,
+            0.013666132349359846,
+            0.057804090189845364,
+            0.056600576093100735,
+            0.051208022172527734,
+            0.03553760896972431,
+            0.02094599510567085,
+            -0.02438674472832137,
+            -0.0027797634437333432,
+            0.03614248430646102,
+            -0.026729308357113124,
+            -0.002251394534452815,
+            0.018929251842788906,
+            0.00689697483318378,
+            -0.026603606434115974,
+            -0.0541443728344337,
+            0.027164630314707807,
+            -0.004335272389193875,
+            -0.013967321193992958,
+            0.003323121377857836,
+            -0.0245295956412367,
+            0.024489335129134787,
+            0.06212059558640156,
+            0.009214058207162371,
+            -0.002082293435905329,
+            0.013659934201796686,
+            -0.003344037927706573,
+            -0.00689041317919184,
+            0.020235785390418023,
+            0.0212739457082623,
+            0.0016782346585238381,
+            -0.013484221664059555,
+            -0.06465582506604173,
+            0.011882797526607326,
+            -0.032970655964890155,
+            -0.04027700439305653,
+            0.025396847912528105,
+            0.013861157517716707,
+            0.00604596157583048,
+            -0.00744611407420936,
+            0.04038084298991919,
+            0.021507532290328625,
+            0.005983822806623904,
+            -0.012711110369915742,
+            0.02410138142963901,
+            -0.018372791517880082,
+            0.021016674128298246,
+            0.013387285956415756,
+            0.02039383525127615,
+            0.010700292312227496,
+            0.0053934660497349735,
+            -0.014807678014449691,
+            0.009422805142740376,
+            0.0414357625490679,
+            0.12206398267002624,
+            -0.010877509834324155,
+            -0.011079302503786522,
+            -0.04107714853756596,
+            0.006631397986165343,
+            -0.031914267104222384,
+            -0.0179965295457466,
+            -0.057530036803978266,
+            -0.0320488164246567,
+            0.008208202363320509,
+            -0.0020203090915064162,
+            0.03502055780919601,
+            0.04464599164916666,
+            0.019332152079287082,
+            0.017133943087431935,
+            -0.039885565008919065,
+            0.04703059988373572,
+            -0.029370696964304416,
+            0.0008494086835971101,
+            0.0014082189719987074,
+            -0.030973762582071615,
+            -0.006561157852024518,
+            0.011634653995510726,
+            -0.019896826887411713,
+            -0.034275921680252924,
+            0.026274465530039866,
+            -0.007728234125236442,
+            0.04814514040690998,
+            0.06550407296280117,
+            -0.02749051703615343,
+            -0.010041099502187074,
+            0.02019601967763537,
+            -0.01871701432558228,
+            0.008532829759864928,
+            -0.020164138072967613,
+            -0.045196925901010855,
+            -0.004523517988516156,
+            -0.002113469149506399,
+            0.07929650663710812,
+            -0.01111269708042164,
+            0.03358254444720164,
+            -0.019973382759721527,
+            -0.0007145592863325417,
+            -0.0011740862928390678,
+            0.02166182480535788,
+            0.004543228023398081,
+            -0.013377688428761516,
+            -0.01758250073647576,
+            0.009548292014596834,
+            -0.04753617258004735,
+            -0.007125744019792479,
+            -0.009930358894299303,
+            -0.007931672044235996,
+            -0.024561936537429774,
+            0.012979195012881556,
+            0.009209145250061479,
+            0.04586756662401364,
+            0.04311807301838248,
+            -0.0036241105681042675,
+            -0.06579256720896123,
+            0.04967191755416882,
+            0.015475886231466817,
+            0.03157651656768371,
+            0.02515941109300714,
+            0.01773554572598468,
+            0.005824418191900522,
+            0.016317635547562007,
+            0.007650872352957098,
+            0.010015166465492963,
+            -0.033203307142867844,
+            0.040272188258443496,
+            -0.013408423788025047,
+            0.01036004294342104,
+            0.012391622448703885,
+            0.04214857862444935,
+            0.03418032388760535,
+            0.015263018079889338,
+            0.018786704580603317,
+            0.02704685343985817,
+            0.015348111464412869,
+            0.0013031600434754704,
+            0.03241427528588285,
+            0.011689924942195664,
+            0.04544406114742012,
+            0.016186615914968734,
+            0.011624500626103958,
+            -0.008272953078864385,
+            -0.02363080189189375,
+            0.02784298119317616,
+            -0.010585980875564109,
+            0.01455013604305903,
+            0.01749228557615982,
+            -0.054742056843690476,
+            -0.005876213780171779,
+            -0.010556605098730201,
+            0.003967456397644932,
+            0.014663888851767762,
+            0.0015548476559947807,
+            -0.003858367677774835,
+            0.012504286824484335,
+            -0.025806882498628832,
+            -0.014216299453397403,
+            0.0031620924516417956,
+            -0.006448390918604307,
+            -0.02785577502346934,
+            -0.00766402687921159,
+            0.005275000635711884,
+            -0.0090448751942,
+            0.019702662630533,
+            0.007009091837467867,
+            -0.01847307100043082,
+            -0.0006774105206839772,
+            0.032952656101827446,
+            0.015900033044760432,
+            -0.02170121261931376,
+            -0.017964879081469612,
+            -0.005062849745084668,
+            -0.03171847232054206,
+            -0.038177398521606544,
+            0.029426291950046455,
+            -0.025687201683643784,
+            0.030158867757101797,
+            0.015780411479509624,
+            -0.007011484049169482,
+            -0.012114023533118393,
+            0.020661407942701628,
+            -0.025067530380881376,
+            0.009777190971413269,
+            -0.004160771288889298,
+            -0.028825369485727344,
+            0.008307290529469729,
+            -0.014415903546305744,
+            -0.007561235629544945,
+            -0.03201704135271719,
+            -0.015182869388110948,
+            -0.00501561450194676,
+            -0.0003863746169672397,
+            -0.01073403445816663,
+            -0.03321103354401307,
+            -0.002304357889348034,
+            0.017001213960120047,
+            0.00028222146729658097,
+            -0.04002882852650754,
+            0.014509588052058002,
+            -0.004150541038999649,
+            -0.034333837380045466,
+            0.0564137596224324,
+            0.03584898691608747,
+            -0.006830698606679202,
+            0.006121456445207273,
+            -0.04505674213649008,
+            0.04358467937307492,
+            0.010847859073780811,
+            -0.04918983713702805,
+            0.049190442935189385,
+            -0.00528960443684769,
+            0.01419195371129309,
+            0.040316024601793406,
+            0.004600843996974165,
+            -0.006780970176765864,
+            0.005206651850992678,
+            0.03397517915308201,
+            -0.010037223444627569,
+            -0.027377913828275806,
+            -0.03328136706915221,
+            0.02562568440972934,
+            -0.03485770844248482,
+            -0.020069979230687374,
+            -0.0052041408068332524,
+            0.029818953377223505,
+            0.028789460147370508,
+            0.004071274130619537,
+            0.01928375754300958,
+            0.0014970947734202088,
+            0.010502599795027616,
+            0.016914145702515263,
+            0.011942576738188502,
+            -0.014909953528043828,
+            -0.003432489584767786,
+            -0.05733768990013957,
+            0.0030710780388681002,
+            -0.02627280695622545,
+            0.007930906480371135,
+            -0.031227917075485834,
+            0.013132797714872345,
+            0.002514351081823354,
+            -0.007152021475569949,
+            -0.0066245205132327055,
+            -0.005279224195782313,
+            0.028512607222275155,
+            -0.00921960825771865,
+            -0.0359704105589433,
+            -0.021347597388353182,
+            0.011200044645812517,
+            -0.0004171294784927543,
+            -0.016784468873218786,
+            -0.0018746544944553878,
+            0.007749103417240419,
+            0.02390272097796833,
+            -0.005128540108845805,
+            0.006397849195755672,
+            0.01214922062611769,
+            0.01163371266250103,
+            0.012307186722130844,
+            -0.004982589662882946,
+            -0.016343724553037307,
+            0.019806887143115643,
+            0.013139657463054087,
+            0.0205801609265216,
+            -0.03767676007629612,
+            -0.0249879011550972,
+            0.05547392586363707,
+            0.005903313549319051,
+            -0.05568806298774435,
+            0.018827693208840233,
+            -0.007931178941307645,
+            0.008183992600768387,
+            -0.010319022946041053,
+            0.05111352384595183,
+            -0.024897203290807753,
+            -0.02096678205188722,
+            0.006436523335212348,
+            -0.04917084523640266,
+            -0.010459284852442706,
+            -0.011138558192296276,
+            0.014789370043397917,
+            -0.03105725978752438,
+            -0.004934745004818259,
+            0.015595801375817122,
+            -4.687585727052446e-05,
+            0.0028886067775750254,
+            -0.00850352807499054,
+            -0.0034885579171087815,
+            0.016339681487281293,
+            -0.0034050317046826813,
+            -0.008309570448487812,
+            -0.003615165868022203,
+            -0.027638409919916757,
+            0.023352910161586186,
+            0.026364511529991978,
+            0.001249936082895606,
+            0.001987948967948376,
+            0.01756629246122769,
+            0.004656075485833687,
+            -0.006752579977941239,
+            -0.013681608638716132,
+            -0.03405936202918687,
+            -0.0401867651078737,
+            0.022991180183813725,
+            -0.02318908140203705,
+            0.018299512436825923,
+            0.00904081847012441,
+            0.019697091729447287,
+            -0.019236130984371835,
+            0.015159917756323988,
+            -0.0387297415849285,
+            -0.018793498297161176,
+            -0.005256288213225733,
+            -0.010414697711744128,
+            0.0007039894599589104,
+            -0.01739645696111503,
+            -0.013728845328097506,
+            0.022709522955937343,
+            -0.022789243441781803,
+            -0.0036763802589970026,
+            -0.01429671717544519,
+            0.0023674126354206932,
+            0.049448298344058644,
+            -0.009926033487593551,
+            -0.023371285193090733,
+            0.020846792916650956,
+            -0.026996519120163966,
+            0.011246908457705283,
+            0.04091365159640201,
+            0.005469162850197613,
+            0.005640919201612091,
+            0.029575753750229278,
+            -0.0007860452837141593,
+            -0.0052805237594885516,
+            -0.009782745443933828,
+            0.021178503581190618,
+            -0.0009176512672832309,
+            -0.05245650420332918,
+            -0.007698364493410058,
+            -0.039027418406703554,
+            0.005448840777995117,
+            0.017581625031764185,
+            0.025522517097715556,
+            -0.020775492472580457,
+            -0.015906320383873178,
+            -0.000643669839811819,
+            -0.0086896084429604,
+            0.0029399233482444638,
+            0.014381240985647907,
+            0.021086693776922936,
+            0.002436575081260609,
+            -0.004913514868101904,
+            -0.011582101006213432,
+            0.014400297663309155,
+            0.0467181039341492,
+            -0.01722692180137712,
+            0.0011622224560709315,
+            -0.013036540368441797,
+            -0.06208454638917413,
+            -0.019160212331432766,
+            0.0351496273293468,
+            0.02429814055116254,
+            -0.009896662169762772,
+            -0.01647101145477158,
+            -0.002915821099802562,
+            -0.03557863734177486,
+            -0.01786433999615293,
+            -0.010699364747311715,
+            0.02390901423299708,
+            0.007213809581960646,
+            0.0005601364043388203,
+            0.00916614601011734,
+            0.014737491634516238,
+            0.019303599465763203,
+            0.008864535607492504,
+            -0.0575839606069681,
+            0.018051837786406608,
+            0.006315830080225406,
+            0.02525619013082567,
+            -0.015671924543113065,
+            0.003988943108210209,
+            -0.03179878281917948,
+            -0.029440641244429775,
+            0.040586322748158446,
+            -0.03418722657593859,
+            -0.030650542761032924,
+            0.006566745611226466,
+            0.009571089937153852,
+            0.004491158558315032,
+            -0.011352813042040581,
+            -0.02535505987721779,
+            -0.013381948772812133,
+            -0.030918676056735445,
+            -0.02192708443081135,
+            -0.005232667170255381,
+            -0.03342408726436631,
+            -0.07498634115550039,
+            -0.006245638452568239,
+            -0.049946896465756174,
+            -0.03900385398225758,
+            0.04458524125677807,
+            -0.006106489377959481,
+            0.02398410968928338,
+            -0.04615183642824994,
+            -0.01165382599292015,
+            -0.04890013373371541,
+            0.013624324077017377,
+            0.01476398305411351,
+            0.020202554298470666,
+            0.00014553149242532853,
+            0.036224316512987846,
+            0.00021498496267774312,
+            -0.05055208658013767,
+            -0.057482169168904106,
+            -0.003960451575823462,
+            -0.010217078509789989,
+            -0.005593738972150311,
+            -0.034090261748104665,
+            0.05835341009022349,
+            0.0116359661127526,
+            0.04203678184748103,
+            0.03447818335186531,
+            -0.006747564131379555,
+            -0.025756962754841885,
+            -0.028123416424745064,
+            -0.008073625379614702,
+            0.047544241458556825,
+            -0.029605671083240564,
+            0.008171081332993893,
+            -0.026007452205474395,
+            0.0067084589781962805,
+            -0.0070645448617940995,
+            -0.0012298101311792947,
+            0.012372286141738867,
+            0.006377319686763927,
+            -0.015024537877377194,
+            0.007934745664790507,
+            0.026041755081605937,
+            -0.007303731324541215,
+            0.0407000170257006,
+            -0.012990703097937777,
+            -0.033189084013863494,
+            -0.022525679964848666,
+            0.01657717698191603,
+            0.03440284885162039,
+            0.010230633738702899,
+            -0.008707808754455952,
+            -0.0363777020540216,
+            0.011184076269109602,
+            0.031774012431843524,
+            0.01504703222643543,
+            -0.022509833303978505,
+            0.025230974050270107,
+            0.017502662694813216,
+            -0.018700579139537456,
+            -0.03511734747709348,
+            0.015162183119327901,
+            -0.017711638132862063,
+            0.018190335050681287,
+            -0.01590174153215615,
+            0.011103110869700727,
+            0.0025189370696524936,
+            -0.00920305384736575,
+            0.01929652151514404,
+            -0.006196715938358087,
+            0.011426611271994685,
+            0.007058186102245829,
+            -0.018191352599117545,
+            0.0027202185072220937,
+            0.057310566886380025,
+            -0.000531639834386418,
+            0.01494975681909879,
+            0.03845692752400366,
+            -0.001971953300066856,
+            0.045554909803324765,
+            -0.02532332999916258,
+            -0.042616337063845953,
+            0.02633963974172818,
+            -0.02970705577683783,
+            -0.01624154998005974,
+            -0.02620993449678864,
+            -0.021490817420853802,
+            -0.004038424045768319,
+            0.009940887982818171,
+            -0.02692933845395842,
+            -0.023801570856681796,
+            -0.001948460540662095,
+            0.02290911497874141,
+            -0.03300532378029689,
+            0.006525488380791647,
+            0.007039306647666766,
+            -0.003685585408710752,
+            0.03183735852538168,
+            -0.027197064244321705,
+            0.006861928286688311,
+            0.011209869849682412,
+            -0.001094897323854524,
+            -0.005328676992017907,
+            -0.004007474089818478,
+            0.010796946231051235,
+            -0.030543173497906445,
+            -0.012879972141706173,
+            0.00935894527375688,
+            0.0017573071057611829,
+            -0.010050882212831222,
+            -0.03102132152986303,
+            -0.002915141291985554,
+            0.007634137769426993,
+            -0.011094093612593696,
+            0.01199609810193786,
+            -0.010458016758814742,
+            -0.04929347607956551,
+            0.010973132159729983,
+            0.028148117094907874,
+            -0.025504019123994758,
+            0.012470127357818321,
+            0.009182729379865986,
+            -0.016020133756226275,
+            0.016320111225714855,
+            -0.012051271446461976,
+            0.024098195821827578,
+            0.04317620443481784,
+            0.024592886889234396,
+            0.007727787037954994,
+            0.00771746776156926,
+            -0.0003147538723468374,
+            0.022572868970938624,
+            -0.016683340264320543,
+            0.013672085891597506,
+            0.020827021901755415,
+            0.02092309219721837,
+            0.0057130785714458055,
+            -0.003901728981013356,
+            -0.0036968415923910314,
+            0.002227024825579113,
+            -0.02106678364798051,
+            0.014578546755457675,
+            0.03128101864424301,
+            -0.005234627629617124,
+            -0.022394744692442837,
+            0.002430617645746389,
+            -0.015780389432069438,
+            4.7277885648917e-05,
+            0.013357803020920043,
+            0.013777651543690907,
+            -0.027713499926132385,
+            -0.0018999112228756628,
+            0.028364630573666852,
+            0.009146793705085586,
+            -0.009891297866910313,
+            -0.01644269527910728,
+            -0.0003903768506971414,
+            0.006512217314869576,
+            0.02408785204739448,
+            0.0018406499545367782,
+            -0.007360473750031044,
+            0.01484497577088404,
+            0.016723277095327158,
+            0.003629911330658209,
+            -0.0004820567356447713,
+            0.02394406451074397,
+            0.023794019615473026,
+            0.001632266615911801,
+            -0.02064379152040184,
+            -0.02755693521684878,
+            0.027335938400284726,
+            0.02769248863677192,
+            -0.006832353240623308,
+            0.027324973360715134,
+            0.003937207909967385,
+            0.013580154030133622,
+            0.02076818768224654,
+            -0.01784151382870951,
+            -0.017617903734311997,
+            0.0017316476451235418,
+            -0.009605294810027804,
+            -0.009829196696575069,
+            0.004891780008779742,
+            -0.011655402383771472,
+            -0.01124379942906224,
+            0.043205454954628145,
+            0.004713701851783307,
+            0.003560551284131962,
+            0.007894840447501912,
+            -0.018881214526749082,
+            -0.00945538175144614,
+            0.0019040259741796817,
+            0.029505863720301916,
+            -0.001400002868254808,
+            0.03315866433970851,
+            -0.00830542045504805,
+            0.008602408964961391,
+            -0.03202191233287069,
+            -0.006978289755114341,
+            0.0326499698728755,
+            0.015314305362524091,
+            0.015104436300213667,
+            0.0021257042489266787,
+            -0.0010840650756005028,
+            0.02434348355847526,
+            -0.007174433778817674,
+            -0.04839373063710409,
+            0.022696994450018916,
+            -0.004779511451775743,
+            0.023588216956749154,
+            0.0382733980620746,
+            -0.004232286553944853,
+            0.003815507826545929,
+            0.023531223314846965,
+            -0.01887650634213847,
+            0.015055403379782985,
+            -0.01918944132635701,
+            -0.017597900721414636,
+            0.02176630010543207,
+            -0.008896608357798277,
+            0.01732387427588561,
+            0.01764363756727683,
+            -0.012180043444768312,
+            -0.019003360499321047,
+            0.01229139850966224,
+            0.0025710091163655626,
+            -0.017075658234265562,
+            0.01797412486853335,
+            0.018372010308966135,
+            0.006723179570342314,
+            0.015417677746651683,
+            -0.00047634684109332134,
+            0.017608296628666153,
+            -0.001049984114818522,
+            -0.008416990841821452,
+            0.01827827907328822,
+            -0.009568717277044265,
+            -0.01658089563620308,
+            0.009808478422454319,
+            -0.05193977747210182,
+            0.010644335931025957,
+            0.0016781339270411392,
+            -0.01562285100294917,
+            0.00405036330963539,
+            0.007102724536527262,
+            0.005195069570261247,
+            0.0028773630901681767,
+            0.004610474981005765,
+            0.042921486858787584,
+            0.013213672397112759,
+            -0.0064030584930089415,
+            -0.010711457508461704,
+            -0.017480309425763254,
+            -0.027151012983864797,
+            -0.005204425551873271,
+            0.009927884476991038,
+            0.022601230152836282,
+            0.008046297278661014,
+            -0.0031858480011267704,
+            -0.02589628103025927,
+            0.019723430185137954,
+            0.03604835658149313,
+            0.062423099201768716,
+            0.010930268386637612,
+            0.02925564269324381,
+            0.02439517070384333,
+            0.023930432096006114,
+            0.013036828091946605,
+            0.006910311087503408,
+            -0.032995841694293995,
+            0.012813650252889403,
+            0.01333714655191133,
+            0.03678966264802025,
+            0.012571138353467183,
+            -0.05610151838497368,
+            3.0777966587107344e-05,
+            0.009791774904454618,
+            0.041959488802722524,
+            0.01956296954085397,
+            0.000155217569370113,
+            -0.007285602454264675,
+            0.031547870838711856,
+            -0.026518385135879206,
+            -0.012299593280363843,
+            -0.0026551378470968594,
+            -0.006190117573235779,
+            -0.003807907001161504,
+            -0.004991957475076049,
+            0.036983035940964804,
+            -0.015648565314743963,
+            -0.019256463332683493,
+            0.0004300362961407231,
+            -0.0015686065121077472,
+            0.0011311524136286674,
+            0.0032236641835276464,
+            0.013795345278976164,
+            0.015385029265915279,
+            -0.004800448639353721,
+            0.026514926384796363,
+            -0.008699453553651681,
+            0.022186817580012775,
+            -0.013427553742493343,
+            -0.011927259975371873,
+            0.017776228476138103,
+            -0.021243797085501972,
+            -0.01667280589187595,
+            -0.003688045116580565,
+            0.006189970874536939,
+            0.0020911968266404467,
+            0.0006097761986763103,
+            -0.004685468039463817,
+            -0.01011357666327168,
+            0.022631082517271542,
+            0.012398000826637277,
+            -0.006721588925828951,
+            0.018142966317821054,
+            -0.0021330638043572654,
+            -0.011869184647489572,
+            0.025073455250141923,
+            -0.0015392945266926425,
+            -0.0337085890598252,
+            0.01267302113143573,
+            -0.013438601165482187,
+            0.02673290895281225,
+            -0.009586762706902736,
+            -0.011208527168019762,
+            0.019144928964011168,
+            -0.0017591316941248897,
+            -0.005459150067129732,
+            -0.029230079356228025,
+            0.004679497471366242,
+            -0.011723195942933286,
+            0.015899377439148873,
+            0.0022054234777254627,
+            0.029187759252864872,
+            -0.018290694895057157,
+            0.00792533824593637,
+            0.011719198071061566,
+            0.014612375997487872,
+            0.005419943379355563,
+            0.010515265723144554,
+            -0.006893314445458116,
+            -0.04424495426064596,
+            5.8241472764594054e-05,
+            -0.005719143565673274,
+            0.0038220021965914754,
+            0.013106940982933836,
+            -0.01203955335201393,
+            0.011260626096963215,
+            -0.0279896016489392,
+            -0.0295228261396555,
+            0.036474811902902415,
+            -0.04900747926861378,
+            0.019261089742811332,
+            0.00014104646541594166,
+            -0.03057197731717424,
+            -0.04331886560073797,
+            -0.03232957159604667,
+            -0.019401402365464362,
+            0.004102283356333861,
+            0.0002610070704924167,
+            0.0012280372705500212,
+            -0.014025012323929194,
+            -0.014135990809256075,
+            0.013269444858490211,
+            0.00992155580941576,
+            0.011225162609266308,
+            -0.006545044517282357,
+            0.002368862850663672,
+            -0.02063183105457681,
+            -0.026981383275694142,
+            -0.02774194980145225,
+            0.013337303400678109,
+            -0.0025902548982556788,
+            -0.02794339093668326,
+            -0.025606606546251152,
+            -0.0029852498102770587,
+            -0.01906873442473352,
+            0.006508845592362198,
+            0.03268861529625085,
+            0.0355975143313428,
+            0.010066906387952277,
+            0.01690250467284985,
+            -0.017201544758038183,
+            0.048345309114140714,
+            -0.019031171948182872,
+            0.03188329748476941,
+            -0.0005845169448621133,
+            -0.024140202991373587,
+            0.00011597263700677706,
+            0.03203623528903764,
+            -0.019285565857567896,
+            -0.01332755507465857,
+            -0.010914187315481855,
+            -0.003374912271519852,
+            0.02419326154642231,
+            0.02443825785057301,
+            -0.0037923303809811746,
+            0.00022800399604730175,
+            0.015893265651968885,
+            0.003154527829977343,
+            0.007688529734165207,
+            0.030623379985627763,
+            0.014052282574575454,
+            0.01127235345371133,
+            -0.04164707563955819,
+            -0.011811587719246128,
+            -0.003763086694358232,
+            0.005909410979841885,
+            0.002913220159138154,
+            -0.002671613084376487,
+            0.0045765194162907885,
+            -0.007241703279856166,
+            -0.005855949422953439,
+            0.0016973541436777935,
+            -0.001576326424468957,
+            0.008393873711385013,
+            -0.0037217094191171083,
+            -0.03498879678492834,
+            -0.026096985039773694,
+            -0.021280075788049788,
+            0.006437497348154639,
+            0.0070214556591898375,
+            -0.00972721128392027,
+            0.01949553762684561,
+            -0.0029404319702066146,
+            0.000271037702827666,
+            -0.010734867909488861,
+            -0.0029784341515646075,
+            -0.01206464664516104,
+            -0.019544778160755956,
+            0.00035139045159739705,
+            0.013457601448003152,
+            0.00025947376767512006,
+            -0.03165522602526639,
+            0.011979811705146604,
+            0.006642849985850598,
+            0.013181162886675883,
+            -0.024499404540711422,
+            -0.02310714308893576,
+            -0.028028912907092744,
+            0.010570122662701183,
+            -0.0066866321204157725,
+            -0.024617571284817428,
+            -0.004097807033448707,
+            -0.0074576460628928256,
+            -0.0020417256533259887,
+            -0.02470953739335365,
+            -0.011408461668188081,
+            0.023537349682260345,
+            -0.006373652338136973,
+            0.01936338676879376,
+            -0.017636435070085086,
+            -0.017036091876814843,
+            -0.002198685044404603,
+            0.015584758061011048,
+            -0.038943923931820196,
+            -0.002076529978164001,
+            -0.008452513321945455,
+            0.006551055965075869,
+            -0.028840587857625097,
+            -0.007128279504360441,
+            0.0199397499488538,
+            0.012424319366271584,
+            -0.03518875279748495,
+            0.013596027106451115,
+            0.022191403935861113,
+            0.01218507466930854,
+            -0.0038551718784437783,
+            0.03910075115429626,
+            -0.0025215641203030336,
+            -0.005422688956956598,
+            0.016759888275451905,
+            0.04462538468871483,
+            0.023708097134660096,
+            -0.03152743757744915,
+            -0.008668774531128668,
+            -0.015224825785549947,
+            0.004102648962663789,
+            -0.025867795193966987,
+            0.004676262922216584,
+            -0.011739753544166856,
+            0.01645951757031186,
+            0.011253400919230348,
+            0.013958707917471253,
+            0.0376596076542735,
+            -0.005077354977296842,
+            0.02668910096392324,
+            0.0010332284250006117,
+            -0.00856723931523911,
+            0.0014431148305642308,
+            -0.01412579094241589,
+            0.029892393553325205,
+            0.02269599036836208,
+            0.014421409701746036,
+            0.029248195108770955,
+            -0.03588867295960255,
+            0.02458655544632528,
+            -0.0010013539341895672,
+            0.008368648583431747,
+            0.018150734970019018,
+            0.02628022565029391,
+            0.015423867373844644,
+            0.012477080028047823,
+            -0.02513018750675568,
+            -0.022180001014682045,
+            -0.0027550350630288997,
+            -0.031944843677988374,
+            -0.0014668093006914267,
+            -0.009789309083948147,
+            0.026386027599666254,
+            0.0046988411391149,
+            -0.019803160322638583,
+            -0.003931806845108628,
+            0.004482097428470392,
+            -0.00244660100774939,
+            -0.010441732346892935,
+            -0.004649318716095894,
+            -0.01869464992812615,
+            -0.017838643679294407,
+            0.023677878122153204,
+            -0.023958455640817666,
+            -0.025238725541146534,
+            0.014582798102040157,
+            0.0013223590277629197,
+            -0.002295641913710489,
+            0.0008238518461567685,
+            -0.02115533275282518,
+            -0.04214275933987015,
+            -0.027817083427786927,
+            0.0037588803464689412,
+            0.03565056002940463,
+            -0.026788865866928923,
+            0.014162550819567465,
+            -0.011336583458519,
+            0.019175861037658344,
+            -0.005127229083828615,
+            0.015531116410874209,
+            -0.018251290727445846,
+            -0.044630374763679266,
+            0.021833307809708984,
+            -0.012087009688208446,
+            0.02720742725750304,
+            0.021497809277370574,
+            -0.026937653159100156,
+            -0.013572733244188407,
+            0.01358214612023832,
+            0.0020408253706382332,
+            -0.009652962552750464,
+            -0.0063411929391088626,
+            -0.010533309167065193,
+            0.003901271588567942,
+            -0.04496345296706673,
+            -0.008761188928422701,
+            0.06948704056954039,
+            0.005916180375965137,
+            0.03157905519638486,
+            0.014668818318138927,
+            -0.028448496686381258,
+            -0.04008949536631331,
+            -0.00462253589823598,
+            -0.0003701522833020314,
+            -0.00024410762276269235,
+            -0.01850585249309217,
+            0.017699615637510956,
+            -0.002712069912576265,
+            -0.009767338815933488,
+            -0.013119464222297351,
+            -0.013743610418034631,
+            -0.030375365173543157,
+            -0.0030316327877227783,
+            -0.008632689470358149,
+            -0.006565457264917239,
+            -0.0328089275944709,
+            -0.011713996290006943,
+            0.029773910045828737,
+            0.02740100904353622,
+            -0.0061926915497348915,
+            0.00638737826122608,
+            -0.010244154924811975,
+            0.0010396203272677069,
+            -0.001264454445070139,
+            -0.004059969507786729,
+            0.008715730726831364,
+            0.0014767015415289076,
+            0.031074767756160294,
+            0.004761858344125113,
+            0.00987870172044607,
+            -0.059926117793699066,
+            -0.009294021263133636,
+            0.021188432871397475,
+            0.016602826531924377,
+            -0.009125784136292983,
+            0.013946188378030194,
+            -0.0017670794602549428,
+            -0.006837334179368295,
+            -0.028290055353601393,
+            0.024556541126233806,
+            0.04543639597357669,
+            0.0031311343860416473,
+            -0.011199639773247286,
+            0.014965677855595444,
+            0.0018248533506196013,
+            0.015507721118783542,
+            -0.011409402378400194,
+            -0.01338737990904486,
+            0.03063410115541065,
+            0.009305362935335817,
+            0.0025578631174065385,
+            0.012727648343848194,
+            -0.0038680554900795763,
+            -0.01173215289303824,
+            -0.01568862495293563,
+            -0.004101703129244437,
+            -0.03649200408657064,
+            -0.0022107849490954137,
+            0.009659780679763728,
+            0.0034091693454195806,
+            0.008848380026939594,
+            0.01842435611499587,
+            0.03546926214224266,
+            0.013478278853234764,
+            0.005970293629431872,
+            -0.0069197533761813455,
+            0.023196445250565058,
+            0.010403914879205351,
+            0.010017272564548626,
+            0.015076807790462658,
+            -0.029649260409273275,
+            -0.012737152920662164,
+            0.00373276777994811,
+            0.013239909450830258,
+            0.0001682544254742693,
+            0.02030652051051447,
+            0.014143961923579586,
+            0.0010147733405793957,
+            -0.012509446388074923,
+            0.0020437769941487626,
+            0.007405578085797006,
+            0.013458276946120546,
+            0.0006779114540819997,
+            -0.004250745498730309,
+            0.0009149988500358458,
+            -0.008355906072311008,
+            0.040391939952386505,
+            -0.01831218885700067,
+            -0.016793844108223176,
+            0.035064882870542116,
+            0.11416006426189942,
+            0.0010630856112440689,
+            -0.015660014600661857,
+            -0.017142534874931403,
+            -0.0086899013413391,
+            0.016611825572132997,
+            0.01609650977226868,
+            0.009780106186629793,
+            0.009622367309076686,
+            -0.000672926017804395,
+            0.024310949465350565,
+            0.02944698490225121,
+            -0.0037169531700178868,
+            -0.010990838766452194,
+            -0.036819646475041955,
+            0.022923962979125287,
+            -0.004596366586768123,
+            -0.007108041562635549,
+            -0.002604794313552557,
+            0.012641223813290853,
+            0.013668775645501733,
+            0.005125100040230003,
+            0.0077993841846437815,
+            -0.011500851674334447,
+            -0.017334458072700107,
+            0.0056819751688881774,
+            -0.015982234644378358,
+            -0.0076533013358874935,
+            -0.013127172654362694,
+            -0.011837346554842119,
+            -0.006562312362371076,
+            0.0029877420008976425,
+            -0.008572723854050838,
+            0.00854083969164562,
+            -0.010939639783598493,
+            0.016185642759084087,
+            -0.008052282332943615,
+            -0.0011992212936609377,
+            0.0010378646897070758,
+            -0.0025399515501741718,
+            0.010712865639618352,
+            0.01451357949543628,
+            -0.02521401163091652,
+            0.020836357346214075,
+            -0.008252813367155418,
+            0.0006409807249997337,
+            0.015338809580802425,
+            -0.04672526649678653,
+            0.022728876784937357,
+            0.005129225753457052,
+            -0.018372222092794564,
+            0.0066182736895667805,
+            -0.02048375780196863,
+            -0.00896382274306482,
+            0.0021738974344880794,
+            0.004491167150658828,
+            0.0034280799511763956,
+            0.015611133773752265,
+            0.018126362497548308,
+            0.00014383671280220997,
+            0.007927708693242186,
+            -0.006973076292062532,
+            0.008118313055163714,
+            -0.006791836664375275,
+            -0.01164567445668625,
+            -0.010894577039363014,
+            0.00014731542306100506,
+            -0.008865132658571428,
+            -0.011415696755798914,
+            -0.017922318360382588,
+            0.00358538001727405,
+            0.00289327637128695,
+            0.010406054711003452,
+            -0.007757607207626101,
+            -0.010994775479337082,
+            -0.011716096880261664,
+            -0.0013333474316266438,
+            -0.01231144419953554,
+            0.009792603083441917,
+            -0.02884953407490196,
+            0.0029097409399144306,
+            0.025277274938032477,
+            0.009263530315933738,
+            -0.028037993274175277,
+            -0.02532252134134532,
+            -0.0018596440554143638,
+            -0.013847737486669237,
+            0.009910107581119272,
+            0.013952962280235523,
+            -0.011807495289410774,
+            0.003896325037406752,
+            -0.02144921729633576,
+            -0.01628545945817568,
+            -0.0040148671532777005,
+            -0.016032493124255323,
+            0.004580041957001551,
+            -0.011375180364988195,
+            -0.007789996450583554,
+            -0.022041665680945066,
+            -0.005890065942573431,
+            -0.050455058269193866,
+            0.014401045842891128,
+            0.030219548461217365,
+            -0.004580747668649223,
+            -0.028684680969792875,
+            -0.015525870112797619,
+            0.011121401958547706,
+            -0.008879749268609245,
+            -0.019454243014778435,
+            -0.007839434838464604,
+            0.018908534797107714,
+            -0.0019283067432864216,
+            0.0024769961730655835,
+            -0.006732653593696035,
+            0.027604805021030646,
+            0.01366593691107021,
+            0.03740746059041736,
+            -0.014662268336336578,
+            0.02539666271785517,
+            0.02371809783638648,
+            -0.036126848124425154,
+            -0.007627445211958312,
+            0.03657481892016626,
+            0.01092515982671624,
+            0.0011046573579274704,
+            0.012350047525582187,
+            -0.006340802546498992,
+            -0.031330519599469066,
+            -0.0013691360532301728,
+            -0.0014676630169621005,
+            -0.023111265345539388,
+            0.027359543575867225,
+            0.018091458372134273,
+            0.014237156751820934,
+            0.0030993618578507172,
+            -0.0014037082734049385,
+            0.013257269860435359,
+            0.01021315143964527,
+            0.030885417052806473,
+            -0.003225065956381245,
+            -0.012825341172770057,
+            0.013476039084668826,
+            0.01852164843904835,
+            -0.009965495459579544,
+            0.012541079912270015,
+            -0.00824122613985912,
+            -0.018124134280975132,
+            0.03192263766222747,
+            0.0001484308749572035,
+            0.010888733498667343,
+            -0.010649515858317311,
+            -0.013082335214246402,
+            0.032614733822945205,
+            -0.00999075761269065,
+            0.004257643407198955,
+            -0.030723533031261387,
+            -0.003820776505926347,
+            -0.010471792928758869,
+            0.01737608180148319,
+            0.004597677757231315,
+            0.02095148515295301,
+            0.029253627700524396,
+            -0.011604450820008121,
+            -0.01996325509217509,
+            -0.001158626158150122,
+            -0.020583793738536985,
+            0.004168757459099133,
+            -0.03632338333261702,
+            -0.019373660230711246,
+            0.022259216603413172,
+            -0.010364620831688015,
+            -0.004751193403783363,
+            0.00444187509644351,
+            0.0008572202661850934,
+            0.014943779567410408,
+            -0.025408263453313398,
+            0.017755758370543148,
+            0.004627392860417132,
+            0.007402073071525438,
+            -0.024913329140362466,
+            0.013760771254234878,
+            -0.006087346663657887,
+            -0.03780469033982951,
+            0.005509388120163784,
+            0.03756141956381493,
+            0.02381019168406692,
+            -0.024580249929652097,
+            0.01807584361080568,
+            -0.03111039245576191,
+            0.023245542501125073,
+            -0.004868090951368273,
+            5.1544460403943004e-05,
+            0.00640235757127173,
+            -0.004698373089469779,
+            -0.00621383035264185,
+            -0.0025132415458431587,
+            -0.008746634331769936,
+            -0.01611801559493664,
+            -0.01193535623883546,
+            -0.03381789115349148,
+            -0.024119754887378317,
+            0.013881102852385332,
+            0.0019232668966222687,
+            -0.0095263729740736,
+            0.018652966857927805,
+            -0.03775912497511837,
+            -0.012022352025516115,
+            -0.023289377270709736,
+            0.023579672990954228,
+            -0.028605627815260902,
+            0.019235397553197357,
+            -0.025838655372560505,
+            -0.021458526449310157,
+            0.01508157885177254,
+            -0.010064396230734452,
+            -0.0016708624560165368,
+            -0.015042071642751917,
+            -0.03670785416972153,
+            -0.006267614265977843,
+            0.02298763967146357,
+            -0.012506444728541897,
+            0.00029871598584066644,
+            0.02671390618094565,
+            -0.003059865833779805,
+            -0.011103452976820622,
+            -0.0017382684168257506,
+            0.010923055147701782,
+            -0.02938686260440486,
+            0.014300097108092281,
+            0.0038571851420598074,
+            -0.0037614499297117365,
+            0.0010991440540809893,
+            -0.010108413486243776,
+            0.02127952161226619,
+            0.009142545227414532,
+            -0.0030260119510809205,
+            0.040589877142650374,
+            0.008677453889680373,
+            0.0017339508605300814,
+            0.008732950895530873,
+            0.01063352447960556,
+            0.006618635029823979,
+            -0.014986307792626314,
+            0.016905861015681957,
+            0.013101237900847744,
+            0.0002162791516784468,
+            0.019309343995079022,
+            0.0021263376627204313,
+            -0.024516267132902416,
+            0.013339486637573757,
+            -0.02311659751478777,
+            0.0020575982953961246,
+            0.02391695586495566,
+            -0.020440172706421535,
+            -0.001656132876701879,
+            -0.006304133667169861,
+            0.000543457661904495,
+            0.022135707414160034,
+            -0.0230903871986617,
+            -0.023444763940153112,
+            0.009604600033024832,
+            0.01721387169484476,
+            0.01827905276638783,
+            0.02824773106648473,
+            -0.007100625424382998,
+            0.01348510178676837,
+            0.0002604386361122316,
+            -0.011802178189607182,
+            0.007664157334475143,
+            0.0074175377458313594,
+            -0.0044008460682833875,
+            0.025491035291032547,
+            -0.0017015687016376205,
+            -0.006134010425966364,
+            -0.005379657499404915,
+            -0.01584573519366957,
+            0.0034386456145185213,
+            -0.013860742817569676,
+            -0.013355238174583658,
+            0.0040342831827141845,
+            -0.0208160950322342,
+            0.10904592104153767,
+            -0.0037038089210379766,
+            -0.019472012964045247,
+            0.0063554319446707044,
+            0.003716719884684827,
+            -0.003692786470980806,
+            -0.003353843210051379,
+            0.007788605566526459,
+            -0.011254480693190834,
+            -0.0045751956417398095,
+            -0.013976039724381875,
+            -0.001989262807706565,
+            0.010067374586433014,
+            0.009201983771978006,
+            -0.005902980676079493,
+            -0.02029714318363332,
+            0.013031676901335606,
+            -0.016970144750162643,
+            0.008874670945038486,
+            0.0008311255074953139,
+            0.00884167159995975,
+            0.001716603195908536,
+            0.01854951250316076,
+            -0.009310881405486074,
+            -0.01325242738760595,
+            0.029667589863813217,
+            -0.005616511284128333,
+            -0.0038179509972925954,
+            -0.0004690716809300012,
+            -0.005068974756744239,
+            0.0021204231125051274,
+            -0.027958274497127258,
+            0.022127088236172387,
+            -0.017271838492631593,
+            0.010267709087809265,
+            0.01316800560257366,
+            0.005599367128954229,
+            0.01817806970002991,
+            -0.007115594036834107,
+            -0.02627643123342057,
+            0.006542742528003751,
+            -0.003433874053082844,
+            0.006631427284537971,
+            0.02316295746074422,
+            -0.014573351469966297,
+            -0.033520864104177005,
+            -0.013707022143307006,
+            -0.01877574599842729,
+            -0.005992004412448576,
+            -0.0043698881826095435,
+            -0.01481352026460129,
+            0.006424121402963946,
+            0.01382938882387991,
+            0.018136021761515388,
+            0.0012697483790911433,
+            -0.011531136974861559,
+            0.00609187274277241,
+            0.01076133059240578,
+            0.008400887242930113,
+            -0.01533504932731406,
+            0.011439227832642668,
+            -0.028718487721012004,
+            0.01522697877548034,
+            -0.012452665599305595,
+            -0.024938748080573947,
+            -0.0022082450217489366,
+            0.017589399637694264,
+            0.0037824832255756763,
+            -0.004355445812154662,
+            -0.018778668474337727,
+            0.019825087149714933,
+            0.024692878851503508,
+            0.003883497401229232,
+            0.019470881143244197,
+            0.041834635328088364,
+            -0.003544509491805351,
+            0.027139164223766474,
+            -0.02129723143323512,
+            0.01461354860170749,
+            -0.03441729954693155,
+            0.001893713808284104,
+            -0.004222657393633789,
+            0.0013677766871517654,
+            -0.003811928806805752,
+            0.020142937759360904,
+            -0.009158202577174012,
+            0.0030638058587375543,
+            0.024854053046139163,
+            -0.018665378030958756,
+            0.00805903734779081,
+            0.010625429315145887,
+            -0.012856237251902233,
+            0.028535299711292306,
+            -0.006335094045840286,
+            0.00305422126851784,
+            -0.0297883488914759,
+            -0.0029192872993071216,
+            0.0026067044379754047,
+            -0.006097749614529602,
+            -0.00968023644800927,
+            -0.007433916539502931,
+            -0.006202274026852405,
+            0.0038354607448269555,
+            -0.021471361566683354,
+            0.003195646827154078,
+            -0.004390751282171024,
+            -0.003323447423321121,
+            0.02518759723554948,
+            0.025857746688368054,
+            0.016893707217894103,
+            0.012739170140289531,
+            0.008553625305650129,
+            0.019805351588748434,
+            0.01654964927067909,
+            0.011549329129728739,
+            -0.020026013396046632,
+            0.01150644479272541,
+            -0.004159611921476011,
+            -0.0022937297602724912,
+            0.014815466072709131,
+            -0.01572171840394285,
+            -0.02276187015864222,
+            0.024693009795978448,
+            -0.019725315346836404,
+            0.00998768979020729,
+            -1.1702210281215188e-06,
+            -0.011129745730507545,
+            -0.002591293237053776,
+            -0.018034357254990988,
+            0.00012070821244376275,
+            -0.023051045150566527,
+            -0.028465270110388716,
+            -0.00048603008831669926,
+            -0.000574546300981984,
+            -0.012992623897880395,
+            0.02064747165046032,
+            0.014820638761258362,
+            -0.024820174379739057,
+            0.02666518374464728,
+            -0.017430076811109155,
+            -0.01495051200726725,
+            -0.005861181182847845,
+            0.00100881727574559,
+            -0.0055223282819321295,
+            0.0034719859452138017,
+            -0.010180752654175722,
+            -0.009961678082468036,
+            -0.012134423047419183,
+            -0.028073672819317354,
+            0.011574683159318163,
+            0.016120674453124317,
+            -0.015043147648505721,
+            0.031834693135959095,
+            -0.008802805725050891,
+            -0.0018698230152007877,
+            0.016179742266521146,
+            0.005096232332051617,
+            -0.017187077376166815,
+            -0.018999569809271876,
+            0.012251308792871571,
+            0.013279090498325945,
+            -0.009588297039279112,
+            0.006630431200804809,
+            -0.014130176783207386,
+            -0.005092662672039885,
+            0.007020236623084732,
+            -0.005049363863784624,
+            0.0055051712211635915,
+            -0.00995314895304919,
+            0.0052743456988306485,
+            0.005841370337169303,
+            -0.0029434616355351975,
+            0.004677049987338193,
+            -0.018568941264514256,
+            0.020031253265629777,
+            0.0018725390936655962,
+            0.01332532735283085,
+            -0.03518855189467458,
+            -0.0056298723325832025,
+            -0.005369475318146738,
+            -0.004650364775945294,
+            -0.020833510091412353,
+            0.020445280915606132,
+            0.0159991245532085,
+            0.013202409908533162,
+            0.023094507005426774,
+            -0.014905666713082246,
+            0.01572052456975675,
+            -0.020945766915121455,
+            0.013114000985348841,
+            0.003349101658930482,
+            -0.019829605346106578,
+            -0.001423750749031516,
+            0.04557459007627153,
+            -0.005847231973329081,
+            0.011986110554869603,
+            0.0012709842140249691,
+            0.026758922819943853,
+            -0.0001635536762342011,
+            0.0065419519929442995,
+            0.005997692702992206,
+            -0.009935128108338126,
+            -0.0008057798606777003,
+            0.00865712443881613,
+            0.046254320605750854,
+            -0.01181132866321511,
+            0.008170688314239364,
+            0.013099051984057827,
+            0.0034971494510865624,
+            -0.0008386400616584293,
+            -0.015346506644390764,
+            -0.03067678541405639,
+            0.019305962963428017,
+            0.007736916644077731,
+            -0.012756525606221703,
+            0.008734673666457216,
+            0.015137771675476126,
+            0.012891824153867332,
+            0.003895886821685762,
+            0.02155154331885366,
+            -0.012761690789748027,
+            0.018528313792219413,
+            0.024663453412919027,
+            0.009077230086690327,
+            9.184526542175535e-05,
+            -0.000751529849753638,
+            0.006791927267880182,
+            -0.0004433479107716552,
+            -0.009832230053513483,
+            0.015858232096630655,
+            -0.014861341148074698,
+            0.017790924034108806,
+            2.7216806238946785e-05,
+            0.03744841066552559,
+            -0.015216207585985075,
+            -0.010138319613387346,
+            -0.007774325714088762,
+            -0.0016997034352120362,
+            0.003971564314593291,
+            0.0018686537276849653,
+            0.006051669054361943,
+            -0.0008319624938748146,
+            -0.017326561560776794,
+            0.01667737942301944,
+            0.005719808546053257,
+            -0.0007405637872110778,
+            0.013750752065593804,
+            0.0038732323799683453,
+            -0.00367645612814479,
+            0.023580906347714314,
+            0.006238541579764762,
+            -0.005235795826199542,
+            0.0028182323816636243,
+            -0.006443210286549713,
+            -0.016255209299456314,
+            -0.003061441395209219,
+            0.004597970789372988,
+            -0.0016590920570710628,
+            -0.011138261849069419,
+            -0.0019423173609729848,
+            0.020355534503272887,
+            -0.004612410336124759,
+            0.013346750536413418,
+            -0.005400060374036387,
+            0.006252154576727367,
+            0.010601459925435238,
+            -0.012346569079682625,
+            0.0014154620309235822,
+            0.010406076751218612,
+            0.0013983417564564663,
+            -0.007140180075225315,
+            0.0010544410919327675,
+            0.007757599672903709,
+            0.006505350216372801,
+            0.00483063937962671,
+            0.00836630436955631,
+            0.013284451514874498,
+            0.0033749897293287438,
+            -0.006243293993633622,
+            0.0013949831763016042,
+            -0.013269921385800992,
+            -0.006308639321247727,
+            0.0010906169976769569,
+            -0.0010152835709869353,
+            -0.02017142197515583,
+            0.010787630901414118,
+            0.005355784839945777,
+            0.007838485470991034,
+            -0.01706932891696025,
+            0.009576030158827565,
+            -0.0008144311254998525,
+            0.02829832828245172,
+            0.007407953684956539,
+            -0.02769403281764041,
+            0.014266001222243134,
+            -0.010492199740754951,
+            0.022575782293320284,
+            -0.0016349316593207844,
+            -0.005656505606143177,
+            0.008025514398322551,
+            -0.009634281579228592,
+            -0.005991026158213869,
+            -0.022366083313133788,
+            -0.025825209755160946,
+            0.016871087849192924,
+            0.013583173973358155,
+            0.021125252303971605,
+            0.00012555627355712715,
+            0.015347535950809997,
+            0.0020128011217859444,
+            0.015164099887550086,
+            0.0024544903332736167,
+            0.012507136277293888,
+            -0.011798050869794805,
+            0.0104828761281283,
+            -0.0137805761114781,
+            0.014620357150314097,
+            -0.005246639228547017,
+            -0.006986408834884282,
+            -0.008849580196616098,
+            -0.005442165099441538,
+            -0.020891551850452786,
+            -0.009155930286949747,
+            -0.0013590151948670868,
+            0.010665872513236615,
+            -0.0005386423407222223,
+            -0.0027671778465775897,
+            -0.03237311596935737,
+            -0.004875062533405715,
+            -0.006670982068843501,
+            -0.01942244110096111,
+            -0.0045975373348006885,
+            -0.026137783023267008,
+            -0.01657173843643047,
+            -0.0007784046498232684,
+            -0.015037306676062236,
+            -0.006889680260149007,
+            -0.02009705080543122,
+            -0.027192867101205106,
+            3.0161964469866572e-05,
+            -0.009647644858034833,
+            -0.005380165957043753,
+            0.013815092649296373,
+            0.011902366689169847,
+            -0.01572592170758115,
+            -0.022321702398924333,
+            -0.01088937403483412,
+            -0.0165754067023749,
+            0.01646435743868707,
+            0.009914938007210463,
+            0.018834863207293908,
+            -0.003724245912768107,
+            0.005643505694722323,
+            0.0009581443359000331,
+            0.02327704258596552,
+            -0.02532422120223976,
+            -0.028694220894404807,
+            0.0019298778426469338,
+            0.0025970175688368323,
+            0.006086164866310174,
+            -0.009414665394188936,
+            0.001200390869257535,
+            -0.007782899559317819,
+            -0.0228264209954853,
+            -0.00898519580978142,
+            0.008156835270668449,
+            -0.010036630164522355,
+            0.02230664068547531,
+            0.012332230358869292,
+            0.013699197379189934,
+            0.010593587938184292,
+            0.01576174456497367,
+            -0.0001654241831584687,
+            0.014471905257729428,
+            -0.008675915624727887,
+            -0.0031070346099204425,
+            0.0038358404385932777,
+            0.005878417037187274,
+            -0.00625022998235275,
+            0.028077114529659795,
+            -0.0011836453716817525,
+            0.0002962695480296182,
+            0.014405782572641941,
+            0.02697667559474498,
+            -0.004710313427099278,
+            0.0210512043261309,
+            -0.002393165025068831,
+            -0.006135816286212752,
+            -0.008429776018235104,
+            0.008607617299702318,
+            -0.009529291136733401,
+            0.011133497451942684,
+            0.019864404836361926,
+            -0.014041315457579166,
+            -0.007351413712676263,
+            0.02950254974910368,
+            -0.014564388091367897,
+            -0.01825317106331622,
+            0.02325964952838705,
+            -0.007414295676409368,
+            -0.0039042587057605484,
+            0.00377144546378926,
+            0.007169856503233239,
+            -0.004036744966949995,
+            -0.012928341573802416,
+            0.016090527447479415,
+            -0.00011159893712484579,
+            0.014146137810890391,
+            -0.0004655839804281017,
+            -0.006372231942983921,
+            -0.010047382825720416,
+            0.002373224247019062,
+            0.026818813126567614,
+            0.013483627627077692,
+            -0.017895710143841333,
+            -0.013785856764479817,
+            0.010826610341182479,
+            -0.01538893818197323,
+            -0.004425206001945227,
+            0.02155595194812746,
+            0.0028768153668128553,
+            -0.020543027467698483,
+            0.008028975884363618,
+            0.0003835165718138906,
+            0.00037046947047892953,
+            -0.00029363296913641716,
+            0.030135739208689217,
+            0.001343516577762509,
+            0.00018948658255495327,
+            -0.00875053830853609,
+            0.008606573235959595,
+            -0.01280305089741873,
+            0.012786478836532904,
+            -0.02437993139337839,
+            0.007648181775166282,
+            -0.00629541989541625,
+            0.004579605315599212,
+            0.016658386927051592,
+            0.01218098627022323,
+            0.015031618813778413,
+            0.008237200519537417,
+            -0.008942965899756683,
+            0.0009093265012901892,
+            -0.025302175262520477,
+            -0.001333639155622843,
+            -0.017971876771137367,
+            -0.02036433247634014,
+            0.0002463842837918767,
+            0.022144712689630985,
+            0.018559482073124734,
+            0.011579840028464503,
+            -0.008657014440886656,
+            0.0166696911919226,
+            -2.9834620942882283e-05,
+            0.011845827764518911,
+            -0.00037901176785495324,
+            -0.006138374154553732,
+            0.004711058953619512,
+            -0.003622694588720417,
+            0.013542405259906128,
+            -0.014730567462694827,
+            0.012076044672004906,
+            0.01690223095263866,
+            -0.009108632851807137,
+            -0.008946251274440141,
+            -0.013992423874475428,
+            0.00048381889145447367,
+            -0.015909599902931202,
+            0.035489252674454214,
+            0.02252772349379229,
+            -0.01210464970923921,
+            -0.010469120693943236,
+            -0.014164999313775756,
+            0.019496048806752753,
+            0.010427539351244087,
+            0.0028018908451425276,
+            0.017091768406027965,
+            0.021395638416767112,
+            0.008817247593404623,
+            -0.006081063711583319,
+            -0.0021216503501144983,
+            -0.003781810594530787,
+            0.020143098218482827,
+            -0.003078172339607037,
+            -0.011350171884507555,
+            -0.012516379645909518,
+            0.005766102930704529,
+            -0.01626433668691071,
+            0.0019294643928566657,
+            0.008641558521764407,
+            -0.015057688533773374,
+            0.019407986696657772,
+            -0.028126197703647868,
+            0.012364374199657305,
+            -0.0044278434183731225,
+            -0.007251147925369614,
+            -0.03496220945653279,
+            0.006389489026974697,
+            0.008941259259632024,
+            0.003941752277454314,
+            0.023003838655784314,
+            -0.024528514847470928,
+            -0.018361931455105486,
+            0.0037791061321962543,
+            0.00047162060848090104,
+            -0.00711894577052754,
+            0.0010138871688747137,
+            0.004128392769213672,
+            -0.00674546985015164,
+            0.0017307097443508704,
+            0.0033354800298095546,
+            -0.002829898077850263,
+            0.013252947102333348,
+            0.0081724269619999,
+            -0.021449979263734438,
+            0.001368791856860285,
+            0.01590656734732547,
+            0.01452008402429858,
+            -0.02391847163018108,
+            0.006331397860689212,
+            -0.02064812934794864,
+            -0.000631340242718475,
+            -0.031272866000888556,
+            0.016859793097564958,
+            -0.0069008624795683276,
+            0.01899718801899956,
+            -0.0015377436942771224,
+            -0.0065810685260109004,
+            -0.012822620655323715,
+            0.02137604937338179,
+            -0.0019121137078344135,
+            -0.0025598772127157477,
+            0.015678293057066646,
+            -0.025700865215693987,
+            0.017825186616033165,
+            -0.0026230848396158204,
+            -0.017342967698019966,
+            0.017199139867162275,
+            0.004536019178633225,
+            0.007224666811309087,
+            -0.023505589665315567,
+            0.0007208041198255396,
+            0.003900164485347562,
+            -0.017250324025622685,
+            0.004470862192961212,
+            -0.030807400122398566,
+            0.0006579147990327597,
+            -0.013222907940559794,
+            -0.0020514994691204847,
+            -0.0046763235724575445,
+            -0.014050035576183385,
+            -0.039181147615786484,
+            -0.005429639279537522,
+            0.016065742005148733,
+            -0.0009029636981539927,
+            0.01800715198737081,
+            -0.002730931639143803,
+            0.002947680833843522,
+            0.004912692811020822,
+            0.00818378920043427,
+            0.006915895602313347,
+            0.02274116906914758,
+            -0.0057854922886970365,
+            0.00471999900678522,
+            0.007058014277137233,
+            0.014638081356210031,
+            -0.01719565951201012,
+            -0.015170377598372723,
+            -0.016015921063369496,
+            -0.006351213642269936,
+            0.01739665344303718,
+            -0.017488885259028475,
+            0.013303332445465686,
+            -0.00429428898520701,
+            0.014825338166443021,
+            -0.008686427208690176,
+            -0.01575945086724703,
+            0.0028564767084934386,
+            -0.007570417028925067,
+            0.001007670647768344,
+            -0.02323908008484089,
+            0.02019729564706863,
+            -0.00847581038203716,
+            0.010508144473197669,
+            0.01729824052022313,
+            -0.021364529179605813,
+            -0.009742851959432839,
+            -0.014342281441152413,
+            0.029896210049029176,
+            0.013925163184699139,
+            -0.009392252140772913,
+            0.003927230832659267,
+            0.008873561269230905,
+            0.011529536139797032,
+            -0.012209070214071445,
+            -0.0027679974125029362,
+            0.002411914349172495,
+            -0.0062056207925691356,
+            -0.002025687569076897,
+            0.03090134105002816,
+            0.0038285457265343245,
+            0.024877474647546086,
+            -0.013214997316123852,
+            -0.0033763842539861907,
+            0.00023379106197363714,
+            0.005134764372494098,
+            0.01183867975321486,
+            -0.0017966997954747142,
+            -0.008029791503028367,
+            -0.01179500476055557,
+            0.02599911330072515,
+            -0.021852083993691557,
+            0.007024505421035273,
+            0.026784024996764876,
+            0.001390202993990949,
+            0.0029956956045771046,
+            -0.013131691460546865,
+            -0.006298736864509989,
+            0.02523640664202355,
+            0.010030824059858138,
+            0.0041523782603032225,
+            -0.026465819482580182,
+            0.0032128899272377593,
+            -0.0061901409332948115,
+            -0.008681758126710213,
+            0.009866461769956222,
+            -0.0028095454857224164,
+            0.01188012041754054,
+            -0.012191996383095982,
+            0.028459939543805705,
+            0.010711502132045176,
+            0.01047154118931583,
+            0.025888652452698974,
+            -0.02079614305315134,
+            0.0042203106489994615,
+            -0.010829796140534066,
+            0.0006773493086316061,
+            0.005596640511307504,
+            -0.027510562413905437,
+            0.001969898603871003,
+            0.010390866563404346,
+            -0.005642706366801436,
+            -0.01882994179544761,
+            0.01187970346127799,
+            -0.019814385265412387,
+            0.024479579275617887,
+            0.003231341626501863,
+            0.020056914356101588,
+            -0.0053276448188067154,
+            0.009834685132275868,
+            0.0002742942778577165,
+            0.0031361331890882127,
+            -0.0008074598092128465,
+            0.009009012303347016,
+            0.005786777410144894,
+            0.013719989127068968,
+            -0.025528370602188686,
+            0.006821572597883499,
+            -0.0029197127345352626,
+            -0.030539872624857384,
+            -0.013383097680839732,
+            0.010291485622655026,
+            0.012218533188117704,
+            -0.009144425807890594,
+            -0.010378679518611277,
+            0.0030042579827305603,
+            0.005748243116238633,
+            -0.006692403885450407,
+            0.006913672151000981,
+            0.01629516912493545,
+            -0.002533517074771728,
+            -0.0019083178803585312,
+            -0.0035007340321378893,
+            -0.01154609935927537,
+            -0.010016998439072824,
+            0.023193275321990937,
+            -0.00482990342915347,
+            0.017943025772239603,
+            0.025873224580389702,
+            0.0039876567755322,
+            -0.0057577932736143506,
+            -0.005460363755206188,
+            0.01478435999841998,
+            0.014195457012279476,
+            0.004669295894868759,
+            -0.009702220679239768,
+            0.007974305230005956,
+            -0.007760086293456636,
+            0.008375077126777163,
+            -0.020260917110786147,
+            0.002199401871514877,
+            0.0026191373298767766,
+            -0.0022089262577237776,
+            -0.015213807046175918,
+            -0.01956788452420682,
+            -0.021898368819136302,
+            0.012634659463647929,
+            0.00034643102239488474,
+            -0.009241772775582378,
+            -0.013645250997151429,
+            0.0016665106990117016,
+            0.0025630500346914592,
+            0.0002095521560180936,
+            -0.007783022628694597,
+            0.009546147813008391,
+            -0.01265287317178631,
+            0.007445223311736028,
+            -0.004007029181436962,
+            0.0051846833906574515,
+            -0.0016916014415925766,
+            -0.016413938859402438,
+            -0.0210868129634149,
+            0.01920794362224444,
+            -0.012897042227841011,
+            0.02269303737601583,
+            -0.0014935878147854765,
+            3.062680855559304e-05,
+            0.0033132626755401546,
+            0.007274487651476732,
+            0.012110042834251237,
+            0.011426424896797977,
+            0.004307086698938411,
+            0.008206932254416481,
+            0.004673427176334148,
+            -0.013124571552764586,
+            0.004324963759928715,
+            0.0039216033705804084,
+            -0.010449887922072297,
+            0.005298323150690876,
+            -0.0010196552573491,
+            0.006128327172621373,
+            0.012167492341907742,
+            0.01923525204293908,
+            0.01908107550907106,
+            -0.0012031208519043073,
+            -0.0010601912677170903,
+            0.014474374858125607,
+            0.004540954805526089,
+            -0.004883537463424679,
+            -0.002772988629590886,
+            -0.013363713078069854,
+            -0.002572403646338979,
+            0.011541515690646252,
+            -0.02376161747414617,
+            -0.018215231053002656,
+            0.021831219299143226,
+            -0.004991882316041097,
+            0.033796174834149546,
+            -0.013140970283091536,
+            -0.016197177724633614,
+            -0.00464190076219332,
+            0.0026389653040644607,
+            -0.013788049032378498,
+            -0.010300859744205445,
+            0.010545541278382187,
+            0.013848738866656793,
+            0.009903590151120531,
+            -0.0005984085581408575,
+            0.02922816576443645,
+            0.0019900932217491907,
+            0.02245395592422119,
+            0.0018978407151245295,
+            -0.007029294098689854,
+            -0.01440575120349887,
+            0.010840472832356266,
+            0.005529335652955532,
+            0.009050748012161991,
+            0.017791269808484802,
+            0.014502057454389247,
+            0.006265567405788131,
+            0.013928705812292101,
+            0.0012903002372833693,
+            0.0010858377120770652,
+            -0.002355229463060164,
+            -0.01692711767136341,
+            -0.01456786846097011,
+            -0.01627640642218236,
+            0.003873864312554098,
+            0.012620350492241247,
+            0.00525890570023481,
+            0.0007393789113866752,
+            -0.007153210751899716,
+            0.009254566395058412,
+            0.0065087881512835375,
+            -0.012279152412209152,
+            -0.016530068283509593,
+            0.004060671955726967,
+            -0.02210570272789274,
+            0.007849467899009437,
+            0.01554233481902349,
+            0.00219885032769439,
+            -0.002655449524219968,
+            0.0006843524304842362,
+            0.006272944169764281,
+            0.005531927368102191,
+            -0.0036710328198476905,
+            -0.00544113514775027,
+            0.011264661763762068,
+            -0.006379654234440906,
+            0.009537150372938296,
+            -0.00290031548037575,
+            0.03718190949929003,
+            -0.011976080887998194,
+            0.012120408392610773,
+            -0.001751472085435711,
+            0.008481703962951994,
+            0.008067669857949423,
+            0.010772687289903202,
+            -0.003805089040741351,
+            -0.0022164543131735167,
+            -0.008753625032781523,
+            0.006751953442322185,
+            -0.01582177621255226,
+            0.013631301238130705,
+            -0.00030685726238985963,
+            -0.010728958775974582,
+            0.01055472059121978,
+            0.006788316621149835,
+            0.004413382316882087,
+            -0.003262024961860459,
+            -0.01972283203505511,
+            0.014937729522976615,
+            0.0018799533869990667,
+            0.019247008681353865,
+            -0.023276106189004046,
+            0.029043727846427652,
+            -0.02019024213343764,
+            0.023083643395685142,
+            0.01865875504942157,
+            0.001179436167200093,
+            -0.02070329911663634,
+            -0.007618131527214554,
+            0.014871709122115308,
+            -0.02659818889735726,
+            -0.006234719259825804,
+            0.005986829551040766,
+            -0.014639912579831785,
+            0.00037813423546252645,
+            -0.01924149909676064,
+            0.00010832498610532006,
+            0.011013130149723431,
+            -0.011782407678373089,
+            0.00438725832178584,
+            0.01608349750921335,
+            0.019347883652009713,
+            -0.009760017612545174,
+            0.003869246694835168,
+            -0.024240324890794748,
+            0.010958789926998691,
+            0.020511684636649225,
+            0.008883830392748497,
+            -0.01761944522796127,
+            0.021817973960311446,
+            0.010652131228027521,
+            -0.012575679152857703,
+            -0.0012753095453828826,
+            -0.028191274775392804,
+            -0.012421170815054556,
+            -0.02188248722244861,
+            -0.0011670175621882146,
+            -0.014436683528976771,
+            -0.0029783880907228383,
+            0.0009506986717516226,
+            -0.004719165363484719,
+            -0.01179163390107033,
+            -0.019543732196676634,
+            0.016606805875498898,
+            0.0227313413004497,
+            -0.006738735201626204,
+            -0.014547240981002825,
+            0.013866974327185123,
+            0.031694077725890536,
+            -0.016189209297529324,
+            -0.00542110554672736,
+            0.0028061798782555557,
+            -0.012964876141500577,
+            0.018946046750284537,
+            0.0039419760434645495,
+            0.000322115192590907,
+            -0.01991851369210666,
+            0.002489725351392778,
+            0.010835458511052207,
+            0.01684964045708685,
+            0.0005541314524621196,
+            0.010855852688653808,
+            -0.007549882748861744,
+            0.010096800951420057,
+            -0.011832260836468588,
+            -3.6179995603517093e-06,
+            -0.007800366796516945,
+            0.007001049318404058,
+            -0.004424776169203575,
+            0.034356203288485945,
+            -0.02817071276766299,
+            0.004989631403681216,
+            -0.014539644354369607,
+            -0.010888763939716889,
+            -0.0027322086509797843,
+            0.0004865534340981866,
+            -0.010775238981581461,
+            -0.005162220412866082,
+            -0.0011263271110752205,
+            0.00913081242511139,
+            0.005844327320514489,
+            -0.013886529519306775,
+            0.019171827605726503,
+            -0.030984919318656618,
+            0.004469458202753611,
+            -0.008553175891755315,
+            0.010511707988936336,
+            -0.0151185694875566,
+            0.013830711686954931,
+            0.01301959345025159,
+            0.008205984085944315,
+            0.007740367849216328,
+            -0.01532123052721834,
+            -0.01678314222392063,
+            0.001376468747213867,
+            0.0036953018347776244,
+            -0.00931855481896886,
+            -0.033391028922085425,
+            -0.007072913446166969,
+            0.005331350534425606,
+            0.011614453225043,
+            0.008156128603003278,
+            -0.02345206873048703,
+            0.005003870024809975,
+            -0.024178720646704597,
+            -0.012115112684614331,
+            -0.007750557665424973,
+            -0.011580152272614108,
+            -0.006117298954192534,
+            0.026449765548206453,
+            0.007989237291215073,
+            0.016691517078382376,
+            0.004954133565782356,
+            0.02742809127725913,
+            0.011670893621991404,
+            -0.011736840457474895,
+            -0.003349151747880166,
+            0.00035419034843196817,
+            0.005507651806218584,
+            -0.008706590052024251,
+            0.011793241980257321,
+            -0.0013295035978611142,
+            -0.0065061682758755625,
+            -0.004398354347597199,
+            0.009372816666628725,
+            0.014188676762501037,
+            -0.003007980559117429,
+            -0.008724255229333351,
+            0.015749776067989826,
+            -0.030899793617096834,
+            -0.015215033379100091,
+            0.0016152308157299517,
+            -0.007813175908301852,
+            0.017806770680528907,
+            0.01169056345888971,
+            -0.026835412610223673,
+            -0.004538235170332766,
+            0.010532316940606419,
+            0.004755930024691789,
+            0.0095408889025506,
+            0.00954782703036282,
+            0.012079064562163901,
+            -0.007829549014214643,
+            0.01046722658021748,
+            -0.011519369840999348,
+            -0.007092961453184232,
+            -0.007241799314569802,
+            -0.008077934089061052,
+            0.021467547268987214,
+            -0.025816030568238717,
+            -0.006558853543772633,
+            0.01134670818701549,
+            0.013027491377727365,
+            -0.002301991242932548,
+            -0.006299073927206214,
+            -0.017357989759336386,
+            0.008311464809363397,
+            -0.013922568511594478,
+            0.025633907424998367,
+            -0.004854655498817219,
+            -0.005336807899100369,
+            -0.005228056880188717,
+            0.01958682721272445,
+            -0.016774409737540904,
+            0.02072573220949266,
+            -0.014449250679704904,
+            -0.007857351393427215,
+            0.001673929749689194,
+            0.006890008607998747,
+            -0.002948835017234416,
+            0.03734128725793149,
+            -0.00556474133167756,
+            -0.0014544201137417158,
+            0.02209084868209573,
+            -0.01883610080898967,
+            0.017901421953179442,
+            0.01563604483998435,
+            0.012327159100148019,
+            0.0018161034827768648,
+            -0.010253034391420751,
+            0.019979728662282904,
+            0.020823925491027635,
+            0.0139490143219451,
+            0.009577521724445478,
+            -0.013478761809038993,
+            0.014934933784421556,
+            -0.0003725422088504494,
+            -0.0024362115214204105,
+            0.012785534912074068,
+            0.016441944143888537,
+            -0.02164522371903646,
+            -0.013152102018736346,
+            0.0063548333449661805,
+            -0.016547629104776748,
+            -0.024875443975589356,
+            -0.008991095813132973,
+            -0.021885127280977718,
+            -0.023336714382596175,
+            0.00688008599534903,
+            0.016482610587564443,
+            0.0035797836729533367,
+            0.013111012464332918,
+            0.006820471349350837,
+            0.018274678506489214,
+            -0.00980248993077149,
+            -0.01582838461385605,
+            -0.0009454586621769173,
+            -0.0004906493597104733,
+            0.009787688665732045,
+            -0.01835583040075359,
+            -0.013977939077760832,
+            0.004340902272034388,
+            -0.020118484730757877,
+            0.01421736173162507,
+            -0.006428386730058143,
+            0.016756376129345597,
+            -0.0004630315831872392,
+            0.011701948921076007,
+            0.02621773278421472,
+            -0.001391567406453007,
+            0.01262026995055899,
+            0.004696906079976175,
+            -0.0030577364576645635,
+            -0.010624066224426196,
+            0.0009420498021845824,
+            -0.0044079288776429516,
+            -0.0004845606666642234,
+            0.006481113327122954,
+            0.020269778435314,
+            -0.03321232649491074,
+            -0.009425498978427855,
+            0.01246754422779043,
+            -0.001658574527323383,
+            -0.005816985985181387,
+            0.013050062223188965,
+            0.00011195028631856881,
+            0.016573827662919836,
+            -0.0017453203044868104,
+            0.01417371165305189,
+            -0.008165910519539772,
+            -0.0007461950113024975,
+            0.0016492309267666983,
+            0.005076015595439873,
+            0.005829299318055953,
+            0.012146800191035666,
+            0.01148884639506138,
+            -0.021051327863689618,
+            -0.0026357096198729156,
+            0.0036146148489310443,
+            -0.010673282438301611,
+            0.014967444393862347,
+            0.005737933867764028,
+            0.01391411712868596,
+            0.0029556189893790237,
+            0.01167806258173173,
+            -0.024440974650111174,
+            -0.01029944007931845,
+            0.01385509328958837,
+            0.00869671728965227,
+            -0.00998315356917989,
+            -0.016580053751364735,
+            -0.007061005072623325,
+            -0.00477175200214453,
+            -0.0021182980051220428,
+            -0.009871524928312118,
+            -0.0005933941177237234,
+            0.006284920058476413,
+            5.466820734762947e-05,
+            -0.012842295882002122,
+            0.015810530885294202,
+            0.004572385791133295,
+            0.003685966629168175,
+            0.005093714133656017,
+            0.011877541830396259,
+            -0.013130170365831912,
+            0.01258190257102377,
+            0.011433121961378686,
+            0.01142665981860032,
+            -0.013710025169815697,
+            -0.001774030746825879,
+            -0.01109881340948764,
+            0.0065684584345163735,
+            0.009983753894796107,
+            0.005346443477816739,
+            -0.002062387245572686,
+            0.004950044847352647,
+            0.006105199826576752,
+            -0.06391185159544624,
+            -0.02131511418723256,
+            0.0027165737784844413,
+            0.033389688791557785,
+            -0.007167593552251769,
+            -0.0037646833460996797,
+            0.002883510089742145,
+            0.01356887065340832,
+            0.009730171586767032,
+            0.00418566245984083,
+            0.020320934773473808,
+            -0.008814826860651366,
+            -0.003014193747732543,
+            0.001559573791453607,
+            0.003499420226345028,
+            -0.008703837291821111,
+            0.0044844701302276645,
+            0.0053963630238108035,
+            0.027764369548110287,
+            0.001469968540767627,
+            0.015793086297019008,
+            -0.008505290826588508,
+            -0.004075888558568453,
+            -0.009166710762668385,
+            -0.002860047211143503,
+            -0.011127820563172684,
+            0.003937110717404271,
+            0.02054186721680431,
+            -0.006126180963243227,
+            0.009346433819831456,
+            0.01057504086030076,
+            0.0001650851447316893,
+            -0.012033423322524951,
+            0.0049587335923558,
+            -0.0032240390628760535,
+            -0.005568976687946761,
+            -0.0024667040429312737,
+            -0.03474027725033016,
+            0.026855947684394643,
+            -0.004369626674503285,
+            0.009252512430957932,
+            -0.005081834388513357,
+            -0.0037896610731020337,
+            -0.014201609270756739,
+            0.017871088029615106,
+            -0.008249444723433136,
+            -0.020092257756971717,
+            0.009532994544485572,
+            -0.004991989149115383,
+            0.006464417738670472,
+            -0.02643078108339741,
+            0.0029138922065812495,
+            -0.017232410749655366,
+            -0.003999293090834831,
+            -0.006924488971625995,
+            0.01318082244178942,
+            -0.007232695119856389,
+            -0.02660711037908311,
+            0.01979251106803667,
+            -0.019485860712345603,
+            -0.011977904020170568,
+            -0.0177530973032872,
+            -0.014524003566945937,
+            0.004874952064727108,
+            -0.015326044325348929,
+            0.002047771741041311,
+            -0.009697340332342246,
+            -0.003652304998155628,
+            -0.0008878698629674222,
+            0.0017782435347417623,
+            -0.013017776298713044,
+            -0.018272697230002275,
+            -0.01353020425912225,
+            -0.00757239449706798,
+            -0.0008250000619957538,
+            0.01299142281772714,
+            -0.001679606555140765,
+            -0.007329941421303076,
+            0.0044605462460153116,
+            0.010231192277666706,
+            0.0006558644644452857,
+            0.0024231733856020414,
+            -0.010728844680699885,
+            -0.014137434553169443,
+            0.005230053181525096,
+            -0.012086468475542818,
+            -0.009459717916037003,
+            -0.0028185227600210556,
+            0.014984527039312847,
+            0.008189464334213754,
+            -0.009139240774931914,
+            -0.005639157579163322,
+            0.014325949891859377,
+            0.009020303519690971,
+            -0.013310205617445609,
+            0.01567574314514385,
+            -0.025369802629268523,
+            0.021964786021408878,
+            0.030536905527039065,
+            0.004945288795196981,
+            -0.01168303908203322,
+            0.02281128628230144,
+            -0.004072286832145419,
+            0.0028278384348551146,
+            0.01740522162288968,
+            -0.042558465579514926,
+            -0.013146562547084278,
+            0.002903743316339479,
+            -0.015297177099602701,
+            -0.010963312697387696,
+            0.008804938803398296,
+            -0.0026727734429658817,
+            0.005273238046953793,
+            0.004257613402172282,
+            0.03512246556126453,
+            0.002258612643297096,
+            -0.0002885708728599208,
+            -0.0019720190285930487,
+            0.012312613129996533,
+            0.010451003939611476,
+            -0.015556620910136185,
+            -0.02387856865043814,
+            -0.004583090523619507,
+            -0.0037828776396525835,
+            -0.0014588238692793338,
+            -0.004361834167147271,
+            -0.008159679676329573,
+            0.012243319920992457,
+            -0.017279964797769668,
+            -0.013748032758693542,
+            0.018394195628121487,
+            -0.006411113522842359,
+            0.019002611495040334,
+            -0.014970362556522149,
+            -0.0077437581301661946,
+            0.0017311115715410025,
+            0.024439691379769795,
+            -0.025013956798520212,
+            0.007359134924497539,
+            -0.0031406470488477937,
+            0.017574824561755627,
+            0.00419409250132525,
+            -0.0046803792187546656,
+            0.010556972371044426,
+            0.015229789583480183,
+            -0.004608293492221483,
+            -0.014449487985966966,
+            -0.01172850291078532,
+            0.016823024209386785,
+            -0.012304887597699588,
+            0.0011773351576246157,
+            0.002262670226244277,
+            -0.0023810578015456716,
+            0.03477119000959688,
+            -0.03442178291363675,
+            0.004062693040714608,
+            0.0029249583507755944,
+            0.003291349443702718,
+            0.006461525089528308,
+            0.006047777261425826,
+            -0.020642802018978404,
+            0.008916088381876748,
+            -0.03316762661930299,
+            -0.006367254689957291,
+            -0.005404276207844267,
+            0.0009189006423526284,
+            0.009401330264376041,
+            -0.011106443887384533,
+            -0.02259405372588916,
+            -0.006010439054323966,
+            0.0070744551115721095,
+            0.010262607009446879,
+            -0.002856616377476421,
+            0.019719451109535182,
+            0.008426509638139982,
+            0.0015877648339862691,
+            -0.010655792508751512,
+            -0.015527851374834499,
+            -0.015460974859638219,
+            -0.017847462632188386,
+            -0.02236964311096428,
+            0.011841749854626384,
+            -0.00576688153869993,
+            0.009054902130717224,
+            0.014317707346504314,
+            -0.015366657284925976,
+            0.002910211008094564,
+            0.017761685988205086,
+            0.007974806171830457,
+            0.002692237107055498,
+            0.0032405477366987947,
+            0.02685230091638845,
+            -0.0009116884421301454,
+            -0.0032228977180251673,
+            0.003158213042682977,
+            0.015872199488167846,
+            0.0004966905858799325,
+            0.019981401695749582,
+            0.015201538351780685,
+            0.005359075814444414,
+            -0.004668738193285921,
+            0.019853988551405383,
+            -0.009844394286458195,
+            0.013476144827747423,
+            -0.00411789677405284,
+            -0.00830656434595337,
+            -0.008496132520586954,
+            0.006442581751999304,
+            -0.008138786054348571,
+            0.006977337946948795,
+            0.02419128791801387,
+            -0.0005734827808485311,
+            -0.0028842142002271044,
+            0.005503033191779872,
+            -0.01696639779775455,
+            -0.00389835761601045,
+            -0.03366077608692946,
+            -0.009942613075976742,
+            -0.01708630689497001,
+            -0.02910756562213045,
+            0.026536268897199546,
+            0.00219827495662445,
+            0.02586503862171508,
+            -0.01905122645609761,
+            0.008935637454859214,
+            -0.01385806889332683,
+            0.029747093931541325,
+            0.009163959012862627,
+            0.0008611776492791918,
+            0.003598831885031635,
+            -0.02103045281939351,
+            -0.01150941825152158,
+            0.027341774725604333,
+            -0.0006700608710615124,
+            -0.012089044078073347,
+            0.022396400579038037,
+            0.017491565611764764,
+            0.0051760367056280536,
+            -0.00559536785318233,
+            0.01979262595019399,
+            0.002306380546194741,
+            0.011973863166872444,
+            -0.0028136750606972223,
+            0.012911235736067303,
+            -0.01510421118359169,
+            -0.003338602157155174,
+            0.039758808572572274,
+            0.007290642519242007,
+            -0.001802921446521437,
+            -0.013696774990705774,
+            0.003941198181491921,
+            -0.0008308293948698925,
+            0.0200351513103389,
+            -0.00014327587644797734,
+            0.028291423013030337,
+            -0.0001085859276231927,
+            -0.010723064585989723,
+            0.020443848393732245,
+            -0.017860296930396913,
+            0.04540901629084378,
+            -0.018671578423261854,
+            0.000852031295629291,
+            -0.010639853151638835,
+            -0.019017194262730474,
+            0.003861296738239369,
+            0.0010977143012920602,
+            0.0008387482016061222,
+            0.0060155816314618,
+            -0.010780239332444652,
+            0.002152214497259421,
+            0.010556338812873054,
+            -0.027636192450621728,
+            0.0010938992111792307,
+            -0.014385202899929138,
+            -0.01353192238925814,
+            -0.026667778226211785,
+            -0.00017403398236549974,
+            -0.015357674131635848,
+            -0.013956722505869091,
+            -0.01985038526848704,
+            -0.008927903879720431,
+            -0.009580214158077466,
+            -0.0020314441852913386,
+            -0.004755584878514854,
+            -0.00018388548107372635,
+            -0.002918855567842974,
+            -0.0015941852262316128,
+            -0.014230272061190995,
+            0.02133992212037431,
+            0.015283345362445234,
+            0.004642765060980717,
+            0.012829570028541471,
+            0.01708952142208596,
+            -0.004770647543823141,
+            0.006604762788847952,
+            -0.0025369836845081078,
+            0.008368000846861623,
+            0.014785031726639733,
+            0.012074738887486398,
+            0.02124518119032885,
+            -0.015459818988572564,
+            -0.004992865135357168,
+            -0.012722391036055767,
+            -0.012210202603282509,
+            -0.017850065946244383,
+            -0.004300429148015694,
+            -0.0014387298144714717,
+            -0.023128300275416423,
+            0.004468952290723408,
+            -0.0026602931688877443,
+            -0.005768852559887139,
+            0.007815212168719318,
+            0.0005862972228454734,
+            -0.002265719947629544,
+            -0.00019323737674204774,
+            -0.02735820445266249,
+            0.00024009122317215594,
+            -0.00018701993387558173,
+            -0.014663710240203917,
+            -0.00450797561624596,
+            0.010584731989847334,
+            -0.011966527748437568,
+            -0.00602367292291343,
+            0.0111155226518017,
+            0.0015565179432852256,
+            0.006062862961835896,
+            0.016952559697728433,
+            0.0005012157272316869,
+            0.011160668242334889,
+            0.014993782560797772,
+            -0.01591303023837098,
+            -0.015356412848444082,
+            -0.0236025426434855,
+            -0.004514538154596937,
+            -0.009753334017065119,
+            0.007553951535816575,
+            -0.004616595532996391,
+            -0.013023153809236325,
+            0.006040285197946957,
+            -0.0006343451398642784,
+            0.008338564073158257,
+            -0.017608246105292328,
+            -8.346168119689692e-05,
+            -0.020944870308705715,
+            0.009230123321316848,
+            -0.00531237224627929,
+            -0.00445985089300067,
+            0.01410675485522911,
+            0.02895945580213926,
+            -0.007799830351886844,
+            -0.009455128322575888,
+            -0.005663072244883114,
+            0.012817229945992764,
+            0.012474431030896143,
+            -0.004423141749110039,
+            -0.0018294512146253298,
+            -0.012817101093394574,
+            0.0004755435377918658,
+            -0.01129485262081729,
+            0.0062747111125005155,
+            0.017199374798326412,
+            -0.021965117839794063,
+            -0.01296097993683748,
+            -0.016765996091978336,
+            -0.016882855472266523,
+            0.004372970343291638,
+            0.017290565252244258,
+            0.022670788364378462,
+            -0.008456547449078897,
+            -0.019564049850942417,
+            0.006898353991539946,
+            -0.01620125883985687,
+            0.0037108638122592746,
+            0.12887455743463638,
+            -0.0028864852421660366,
+            -0.011487804449328495,
+            -0.022683334123299446,
+            -0.005467192611880829,
+            0.01210557696712196,
+            -0.01555159294496261,
+            0.06709672179195036,
+            -0.015071089552631105,
+            0.004364902817496082,
+            -0.004899872946473657,
+            0.0006310685609937571,
+            -0.017610935588191347,
+            -0.005962991715597585,
+            -0.007768819050529025,
+            -0.02688607975562596,
+            0.000883763319535686,
+            -0.009935313912803587,
+            -0.012883104064962595,
+            0.002126368813790577,
+            -0.0013697265269800346,
+            -0.016131186838419288,
+            0.01575514418037863,
+            -0.020677990768697592,
+            -0.007251917191831732,
+            -0.0029325623780992118,
+            -0.00600710212646933,
+            0.014573093764769918,
+            -0.03253336117394778,
+            0.002973725493330224,
+            -0.013601417220439602,
+            0.011905788192118131,
+            0.001542117126832946,
+            -0.010789486980776022,
+            -0.013862139885514524,
+            0.007086571810314664,
+            0.015658913897173155,
+            0.017039624764452546,
+            0.01901374401756777,
+            0.004797954569087795,
+            0.003928999581093576,
+            -0.008046722893614655,
+            0.021974371139905104,
+            -0.00697335955527197,
+            0.0008975473383295958,
+            -0.00970589295353857,
+            -0.004346869073499034,
+            0.0031279458769925416,
+            0.02303983493541673,
+            -0.012560380384977343,
+            -0.01536378830925542,
+            0.0012035987157572355,
+            0.033313215063595594,
+            -0.010845084209743817,
+            0.013587126023659577,
+            0.0029867620644300073,
+            0.020679673435988446,
+            -0.0006640787253300063,
+            0.0025071780490108306,
+            0.022554193713122565,
+            -0.018082906254599396,
+            0.018230000220500718,
+            -0.004451773302171552,
+            0.01915083025790679,
+            -0.004183521043835777,
+            0.015019320101074747,
+            -0.027014055979204272,
+            -0.005362044589023088,
+            -0.012355559881051253,
+            0.003995309003198048,
+            0.020680455787871383,
+            0.0054391755324872375,
+            0.009865676762244514,
+            -0.01645386866657314,
+            -0.01760657627715638,
+            0.022969828924328126,
+            -0.021566687194482292,
+            -0.015826424849757402,
+            0.00012804774676298237,
+            -0.015021970627042762,
+            0.006000610438452896,
+            0.010126179023323158,
+            -0.02912121211344025,
+            -0.02046150879033127,
+            0.004881850039543051,
+            -0.0023469840669773042,
+            -0.00915786433296249,
+            0.006547377770403066,
+            -0.022329063637464074,
+            -0.015702777677743352,
+            -0.005692904224111613,
+            0.01207999389934878,
+            -0.008429160038192635,
+            0.010075061963131637,
+            -0.020106712664936504,
+            0.0036787045650556907,
+            -0.002764965946892891,
+            0.010307702251974629,
+            0.004697169247387421,
+            0.011192285826515207,
+            0.0030113901179382946,
+            0.0014774681536380047,
+            -0.02778270000997313,
+            0.010043261430740039,
+            -5.178406450222847e-05,
+            0.04216636255860709,
+            0.030977096295679525,
+            0.004479857028809617,
+            -5.798020745008712e-05,
+            -0.009996471945562616,
+            0.012476009017187798,
+            -0.019306907391548302,
+            -0.005092322343599383,
+            0.01856463597985508,
+            0.02195467141594795,
+            -0.02236232264139305,
+            0.0052217593371508145,
+            -0.016766955100332014,
+            0.016133299654723637,
+            -0.011208763137897244,
+            0.006248024409576809,
+            0.011601988603146743,
+            0.0022330878390492527,
+            0.010317781892717903,
+            -0.005895142237225119,
+            0.019670196605183973,
+            0.01186677200787666,
+            -0.002339367025714548,
+            0.02367408853110795,
+            0.006858134385159451,
+            0.004479683673962342,
+            0.004798719889930319,
+            -0.009604173975576537,
+            -0.009045602998213713,
+            0.009230969488015385,
+            0.004083616606774546,
+            -0.007944628589259331,
+            0.011836323287097144,
+            0.015100350195476968,
+            0.01885317701149666,
+            -0.0011707847499298092,
+            -0.00684548433712413,
+            -0.006550059418141426,
+            0.001899758393077906,
+            -0.024577381680573596,
+            -0.02855511854789239,
+            0.021238935710272536,
+            -0.010427135883958709,
+            -0.019187769881105637,
+            -0.0015864640188683347,
+            0.016360037306061392,
+            -0.008853972319021563,
+            -0.01627167079143546,
+            0.001270414822216867,
+            0.02255755055961612,
+            -0.005959411125391464,
+            0.006009654905468478,
+            0.005909233773418581,
+            0.005874361519568704,
+            -0.0027859463941992753,
+            -0.024599789577567636,
+            0.0036612374968223453,
+            0.03221346865585554,
+            0.016461563205582293,
+            0.016139782455492073,
+            -0.007158168059808459,
+            0.001838455290310879,
+            0.007530954381521755,
+            0.005438791523412295,
+            0.015674000912615536,
+            0.008604106101542998,
+            -0.006990462920227135,
+            0.00244736628387359,
+            0.012213048255418866,
+            -0.007329299946651028,
+            0.017424711328323798,
+            0.001973882532418776,
+            -0.0133178791530314,
+            -0.008373452013951822,
+            -0.010462150707776556,
+            -0.014575483571750984,
+            0.0027958704799249897,
+            -0.011401916623878957,
+            0.0008016041012669959,
+            0.003953958717263787,
+            0.010260753041492368,
+            0.013961649189650642,
+            -0.0019275791443041316,
+            0.017019168629298462,
+            0.011534594243860177,
+            -0.0017859096307142257,
+            -0.002970739203219076,
+            -0.03142778833370656,
+            0.010207789755749909,
+            -0.014730553506704018,
+            -0.0037611416875830746,
+            0.01825790347981639,
+            -0.007766890847082858,
+            -0.0032778433418881666,
+            -0.021003405429807113,
+            0.0024512381432602736,
+            0.008768616692363321,
+            0.01558713989712387,
+            -0.012819542714293536,
+            0.009201155584074711,
+            -0.0012254337002053017,
+            -0.0057451574939563045,
+            0.00977251392462179,
+            0.0042356831792011935,
+            -0.0033433823868435786,
+            0.011499764125503874
+          ],
+          "title": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer",
+          "keyphrases": [
+            "Sparsely-Gated Mixture-of-Experts Layer",
+            "conditional computation",
+            "neural network",
+            "language modeling",
+            "machine translation"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "ff7dd6f5-6560-42e5-923e-f4091e278202",
+        "properties": {
+          "page_content": "4 Balancing Expert Utilization\n\nWe have observed that the gating network tends to converge to a state where it always produces large weights for the same few experts. This imbalance is self-reinforcing, as the favored experts are trained more rapidly and thus are selected even more by the gating network. Eigen et al. (2013) describe the same phenomenon, and use a hard constraint at the beginning of training to avoid this local minimum. Bengio et al. (2015) include a soft constraint on the batch-wise average of each gate.1 We take a soft constraint approach. We define the importance of an expert relative to a batch of training examples to be the batchwise sum of the gate values for that expert. We define an additional loss Limportance, which is added to the overall loss function for the model. This loss is equal to the square of the coefficient of variation of the set of importance values, multiplied by a hand-tuned scaling factor wimportance. This additional loss encourages all experts to have equal importance.\n\n$$I m p o r t a n c e(X)=\\sum_{x\\in X}G(x)$$ $$L_{importance}(X)=w_{importance}\\cdot CV(Importance(X))^{2}\\tag{7}$$ $$(6)$$\n\nWhile this loss function can ensure equal importance, experts may still receive very different numbers of examples. For example, one expert may receive a few examples with large weights, and another may receive many examples with small weights. This can cause memory and performance problems on distributed hardware. To solve this problem, we introduce a second loss function, Lload , which ensures balanced loads. Appendix A contains the definition of this function, along with experimental results.\n\n",
+          "embedding": [
+            -0.013197094202041626,
+            -0.002106932457536459,
+            -0.020997902378439903,
+            -0.01955360174179077,
+            -0.0042971898801624775,
+            0.013943051919341087,
+            -0.01379227265715599,
+            0.05729586258530617,
+            -0.02623547427356243,
+            0.009284786880016327,
+            -0.014958823099732399,
+            0.013482780195772648,
+            -0.0004461359931156039,
+            -0.0026227538473904133,
+            0.012546366080641747,
+            0.013776401989161968,
+            0.013578008860349655,
+            -0.0015841671265661716,
+            0.0022577110212296247,
+            -0.008999100886285305,
+            0.036821719259023666,
+            -0.03621860593557358,
+            -0.02348971739411354,
+            0.012839986942708492,
+            -0.011721051298081875,
+            0.029409760609269142,
+            0.025187959894537926,
+            -0.015276252292096615,
+            -0.048630062490701675,
+            0.009760930202901363,
+            0.025886302813887596,
+            -0.002469991333782673,
+            -0.04361468926072121,
+            -0.004309093579649925,
+            -0.04536054655909538,
+            -0.018458474427461624,
+            0.04875703528523445,
+            0.02812417410314083,
+            -0.012403522618114948,
+            0.030568374320864677,
+            0.02133120223879814,
+            0.006888200994580984,
+            -0.03971032053232193,
+            0.01788710243999958,
+            0.009380015544593334,
+            -0.032631661742925644,
+            -0.03777400404214859,
+            -0.0011179439024999738,
+            -0.0026505289133638144,
+            -0.03621860593557358,
+            0.013538329862058163,
+            -0.05221700668334961,
+            0.005737522151321173,
+            -0.018934616819024086,
+            -0.011316330172121525,
+            0.02893361821770668,
+            -0.010998900979757309,
+            0.014236672781407833,
+            0.042598918080329895,
+            0.01918855868279934,
+            0.006015272345393896,
+            0.03012397512793541,
+            -0.055962663143873215,
+            0.021616889163851738,
+            0.023997602984309196,
+            -0.04751906171441078,
+            -0.0064636399038136005,
+            0.01644280180335045,
+            -0.001340143964625895,
+            -0.012998702004551888,
+            -0.06805669516324997,
+            0.02564823254942894,
+            -0.0355837456882,
+            0.025187959894537926,
+            0.005527225788682699,
+            -0.006130340043455362,
+            0.042027547955513,
+            0.03217139095067978,
+            -0.0028231306932866573,
+            0.013181222602725029,
+            -0.026933817192912102,
+            0.009205429814755917,
+            0.004031343385577202,
+            0.022108903154730797,
+            0.04294808954000473,
+            -0.010895737446844578,
+            -0.00926891528069973,
+            -0.05367717891931534,
+            0.01799820177257061,
+            -0.022077159956097603,
+            -0.015966659411787987,
+            0.02117248810827732,
+            0.01120522990822792,
+            -0.012308293953537941,
+            -0.02818766050040722,
+            0.03694869205355644,
+            -0.000555996026378125,
+            -0.004745557904243469,
+            -0.010451337322592735,
+            -0.008872129954397678,
+            -0.02236284501850605,
+            0.006181922275573015,
+            -0.02958434633910656,
+            -0.029520859941840172,
+            0.030203333124518394,
+            -0.022378716617822647,
+            -0.01826801709830761,
+            0.012459073215723038,
+            0.030250946059823036,
+            0.10710041224956512,
+            -0.009221301414072514,
+            -0.009784736670553684,
+            -0.030838189646601677,
+            0.013371679931879044,
+            -0.03891674801707268,
+            -0.037043917924165726,
+            -0.027632160112261772,
+            -0.06120023503899574,
+            0.025362545624375343,
+            -0.023584945127367973,
+            0.023188160732388496,
+            0.04977280646562576,
+            0.012966958805918694,
+            0.01961708813905716,
+            -0.044916149228811264,
+            0.04409083351492882,
+            -0.023997602984309196,
+            0.027775002643465996,
+            0.008102364838123322,
+            0.0019363145111128688,
+            0.0032278522849082947,
+            0.02650528959929943,
+            -0.02269614487886429,
+            -0.044376518577337265,
+            0.006757261697202921,
+            0.013300258666276932,
+            0.06339049339294434,
+            0.02499750256538391,
+            0.01441919431090355,
+            -0.034345775842666626,
+            0.01129252277314663,
+            -0.053645435720682144,
+            -0.019220301881432533,
+            0.015982531011104584,
+            -0.00685645779594779,
+            0.016347574070096016,
+            -0.00514631113037467,
+            0.020442401990294456,
+            -0.016887202858924866,
+            0.010364044457674026,
+            -0.005693875718861818,
+            0.04843960702419281,
+            -0.006166050676256418,
+            0.024108702316880226,
+            0.019061587750911713,
+            0.016506288200616837,
+            -0.028949489817023277,
+            0.025553002953529358,
+            -0.015427030622959137,
+            0.0011467108270153403,
+            -0.0006834634696133435,
+            -0.001254834933206439,
+            -0.015847623348236084,
+            0.02344210259616375,
+            0.024965759366750717,
+            0.06729486584663391,
+            0.01676023006439209,
+            0.013903372921049595,
+            -0.07834137976169586,
+            0.026759231463074684,
+            -0.0036087664775550365,
+            -0.03891674801707268,
+            0.03996426239609718,
+            0.03631383180618286,
+            2.3481657990487292e-05,
+            0.02656877413392067,
+            -0.007566704414784908,
+            0.017760131508111954,
+            -0.014720751903951168,
+            0.03980554640293121,
+            -0.03479017689824104,
+            -0.014744559302926064,
+            0.008058718405663967,
+            0.007725419010967016,
+            0.01649041660130024,
+            0.008308693766593933,
+            0.026743359863758087,
+            0.012879665940999985,
+            -0.019442502409219742,
+            0.004567004274576902,
+            0.028378117829561234,
+            0.01788710243999958,
+            0.014355708844959736,
+            -0.003918259404599667,
+            0.016887202858924866,
+            -0.03574246168136597,
+            0.010935415513813496,
+            -0.03307606279850006,
+            0.0299017746001482,
+            0.0007930755382403731,
+            -0.005070921964943409,
+            -0.02004561759531498,
+            0.013974794186651707,
+            -0.015831751748919487,
+            -0.023997602984309196,
+            0.012403522618114948,
+            0.03732960298657417,
+            0.004404321778565645,
+            0.013189158402383327,
+            -0.03539329022169113,
+            0.009078457951545715,
+            -0.02053763158619404,
+            -0.018093431368470192,
+            0.004916175734251738,
+            -0.05082032084465027,
+            -0.01330819446593523,
+            0.0177760012447834,
+            0.058819521218538284,
+            0.042376719415187836,
+            -0.007765097543597221,
+            -0.026965560391545296,
+            0.02139468863606453,
+            0.031076261773705482,
+            0.00046944714267738163,
+            -0.0034480681642889977,
+            -0.014585844241082668,
+            -0.004142443183809519,
+            -0.013808144256472588,
+            0.008927679620683193,
+            -0.014593780040740967,
+            0.007959522306919098,
+            0.015442902222275734,
+            0.03767877444624901,
+            0.021902574226260185,
+            -0.0007375255227088928,
+            -0.0022081127390265465,
+            -0.012522558681666851,
+            -0.005023307632654905,
+            0.0010033720172941685,
+            -0.03552026301622391,
+            -0.026600517332553864,
+            -0.017474444583058357,
+            -0.03840886056423187,
+            0.0005227652145549655,
+            -0.011403623037040234,
+            0.011356008239090443,
+            -0.00880864355713129,
+            0.03821840509772301,
+            -0.0026088664308190346,
+            -0.011022708378732204,
+            -0.02785436064004898,
+            -0.027394089847803116,
+            -0.008149979636073112,
+            0.021109001711010933,
+            -0.011363944038748741,
+            0.014196994714438915,
+            0.02785436064004898,
+            -0.047614291310310364,
+            0.011840087361633778,
+            -0.0018361260881647468,
+            0.030949288979172707,
+            -0.004781268537044525,
+            -0.022267617285251617,
+            0.01896636001765728,
+            -0.01371291559189558,
+            -0.03599640354514122,
+            0.047233376652002335,
+            0.0371708907186985,
+            0.024235675111413002,
+            -0.005400254391133785,
+            0.020616987720131874,
+            -0.027552803978323936,
+            0.006423961371183395,
+            -0.0549786351621151,
+            0.03675823286175728,
+            -0.03640906140208244,
+            -0.025505388155579567,
+            -0.007392118684947491,
+            -0.014133509248495102,
+            0.050026748329401016,
+            -0.007046915125101805,
+            0.034028347581624985,
+            0.008967358618974686,
+            -0.007400054484605789,
+            0.03644080460071564,
+            -0.012355908751487732,
+            -0.02187083102762699,
+            -0.031330205500125885,
+            -0.047614291310310364,
+            -0.011848023161292076,
+            -0.0003290841996204108,
+            -0.010014872997999191,
+            -0.001375854597426951,
+            0.006765197031199932,
+            0.04707466438412666,
+            -0.00778493657708168,
+            -0.02625134587287903,
+            0.0028211467433720827,
+            0.01669674552977085,
+            -0.019315531477332115,
+            0.005578808020800352,
+            0.0038865164387971163,
+            0.01633170247077942,
+            -0.026759231463074684,
+            -0.02537841722369194,
+            0.006816779263317585,
+            -0.033488716930150986,
+            0.01680784486234188,
+            -0.03137781843543053,
+            0.0004920143401250243,
+            0.015982531011104584,
+            0.0218232162296772,
+            -0.0049836291000247,
+            -0.05637532100081444,
+            -0.04078957810997963,
+            0.01653803139925003,
+            0.015696844086050987,
+            0.03191744536161423,
+            0.0010425546206533909,
+            0.0014323965879157186,
+            0.016633259132504463,
+            0.01586349494755268,
+            -0.03358394652605057,
+            0.013173287734389305,
+            -0.026743359863758087,
+            -0.010165651328861713,
+            0.01029262226074934,
+            0.04564623534679413,
+            -0.04551926255226135,
+            -0.011030644178390503,
+            -0.012752694077789783,
+            -0.0072770509868860245,
+            -0.03983728960156441,
+            -0.013578008860349655,
+            -0.0009036795818246901,
+            -0.015046115964651108,
+            -0.015276252292096615,
+            0.012951087206602097,
+            -0.014054151251912117,
+            -0.047550804913043976,
+            -0.016839588060975075,
+            0.014530294574797153,
+            -0.023505588993430138,
+            0.014696944504976273,
+            0.02780674584209919,
+            0.01058624405413866,
+            -0.020664602518081665,
+            0.0055311936885118484,
+            0.01597459428012371,
+            0.008673736825585365,
+            0.0010921527864411473,
+            0.016014274209737778,
+            2.0304270947235636e-05,
+            0.03221900388598442,
+            -0.0020751894917339087,
+            -0.010633858852088451,
+            -0.0009309586021117866,
+            0.03212377429008484,
+            -0.015807945281267166,
+            -0.016299959272146225,
+            0.014863594435155392,
+            -0.02042653039097786,
+            -0.06027969345450401,
+            -0.007530993781983852,
+            -0.027775002643465996,
+            0.00897529348731041,
+            -0.011641694232821465,
+            -0.020029745995998383,
+            -0.02758454531431198,
+            0.0034659237135201693,
+            -0.012705080211162567,
+            0.0013252644566819072,
+            0.029028845950961113,
+            0.022394588217139244,
+            0.011038579978048801,
+            0.015585744753479958,
+            -0.001685347524471581,
+            0.0027318699285387993,
+            -0.055264320224523544,
+            0.03237771987915039,
+            -0.04529706388711929,
+            0.01939488761126995,
+            0.007729386445134878,
+            -0.010848122648894787,
+            0.02053763158619404,
+            0.027013175189495087,
+            -0.0009607175015844405,
+            0.0074794115498661995,
+            0.019410759210586548,
+            0.04355120658874512,
+            -0.015800008550286293,
+            -0.05570872128009796,
+            -0.0387897752225399,
+            -0.0019601215608417988,
+            -0.011974994093179703,
+            -0.00724927568808198,
+            0.01158614456653595,
+            -0.024426132440567017,
+            -0.004781268537044525,
+            -0.009602215141057968,
+            0.006662033032625914,
+            0.018442602828145027,
+            0.0009369103936478496,
+            0.009308594278991222,
+            -0.020934415981173515,
+            -0.003948018420487642,
+            -0.030742960050702095,
+            0.008443601429462433,
+            0.021521659567952156,
+            -0.00889593642205,
+            0.00948318000882864,
+            -0.002079157391563058,
+            -0.030362047255039215,
+            0.010792572982609272,
+            0.02791784703731537,
+            0.030187461525201797,
+            -0.01660151593387127,
+            0.021902574226260185,
+            -0.010324365459382534,
+            -0.008261079899966717,
+            -0.007530993781983852,
+            0.029647832736372948,
+            0.0678662359714508,
+            0.022029545158147812,
+            0.007546865381300449,
+            -0.02063285931944847,
+            0.0073683117516338825,
+            -0.02155340276658535,
+            0.012657465413212776,
+            -0.028171788901090622,
+            0.03431403264403343,
+            0.03621860593557358,
+            0.012800308875739574,
+            -0.0014046216383576393,
+            0.017283987253904343,
+            -0.029568474739789963,
+            -0.020823316648602486,
+            0.03093341737985611,
+            -0.03282211720943451,
+            -0.029870031401515007,
+            0.014101766049861908,
+            -0.002797339577227831,
+            0.010943351313471794,
+            0.016204729676246643,
+            -0.02920343168079853,
+            -0.042979832738637924,
+            -0.002227952005341649,
+            -0.031155617907643318,
+            -0.02052175998687744,
+            -0.012720951810479164,
+            -0.0625016912817955,
+            -0.002642593113705516,
+            -0.046630263328552246,
+            -0.014720751903951168,
+            0.013133608736097813,
+            -0.008003168739378452,
+            0.011482980102300644,
+            -0.03517109155654907,
+            0.005483579356223345,
+            -0.019966259598731995,
+            -0.026632260531187057,
+            0.015188959427177906,
+            0.045233577489852905,
+            -0.004166250582784414,
+            -0.0059676580131053925,
+            0.027187760919332504,
+            -0.039361149072647095,
+            -0.01750618778169155,
+            0.009784736670553684,
+            0.00778493657708168,
+            -0.014641394838690758,
+            -0.04009123519062996,
+            -0.021839087828993797,
+            0.014244608581066132,
+            0.019315531477332115,
+            0.021632758900523186,
+            -0.004257511347532272,
+            -0.04989977926015854,
+            -0.03520283102989197,
+            -0.030250946059823036,
+            0.039361149072647095,
+            0.0005202853353694081,
+            0.004436064977198839,
+            -0.023616688326001167,
+            -0.009062586352229118,
+            0.004531293176114559,
+            0.022712016478180885,
+            0.009253043681383133,
+            -0.028203532099723816,
+            -0.0049677579663693905,
+            0.0006284094415605068,
+            0.005114568397402763,
+            -0.030139846727252007,
+            0.025441903620958328,
+            0.0013649429893121123,
+            -0.05961309373378754,
+            0.008149979636073112,
+            -0.02926691807806492,
+            0.007344504352658987,
+            0.01409383025020361,
+            -0.010149779729545116,
+            -0.03093341737985611,
+            0.01676023006439209,
+            0.04085306078195572,
+            0.020759830251336098,
+            -0.026759231463074684,
+            0.012713016010820866,
+            0.012927279807627201,
+            -0.030917545780539513,
+            -0.04104351997375488,
+            -0.002900503808632493,
+            0.004801107570528984,
+            -0.009713315404951572,
+            -0.011221101507544518,
+            -0.002594978781417012,
+            0.03475843369960785,
+            -0.009689508005976677,
+            0.0036841558758169413,
+            0.022712016478180885,
+            0.021854959428310394,
+            -0.021680373698472977,
+            0.014577909372746944,
+            -0.01153059396892786,
+            0.033266518265008926,
+            -0.027029046788811684,
+            0.03593291714787483,
+            0.009118136949837208,
+            -0.0022636628709733486,
+            0.031441304832696915,
+            -0.00964189413934946,
+            -0.02042653039097786,
+            0.02321990206837654,
+            -0.0007915875758044422,
+            -0.02026781626045704,
+            -0.029489118605852127,
+            -0.01058624405413866,
+            -0.0069159758277237415,
+            0.026632260531187057,
+            -0.022267617285251617,
+            -0.023315131664276123,
+            0.01298283040523529,
+            0.042027547955513,
+            0.009300658479332924,
+            0.026600517332553864,
+            -0.0004915183526463807,
+            -0.00017086586740333587,
+            0.04964583367109299,
+            -0.03659952059388161,
+            0.02312467433512211,
+            0.018823517486453056,
+            -0.021743860095739365,
+            -0.01182421576231718,
+            -0.0014056136133149266,
+            0.011236973106861115,
+            -0.010284687392413616,
+            -0.006328732706606388,
+            0.003662332659587264,
+            0.005570872221142054,
+            0.0020394788589328527,
+            -0.02112487331032753,
+            0.010252944193780422,
+            0.0006199777708388865,
+            0.005812911316752434,
+            0.017204631119966507,
+            -0.02229936048388481,
+            -0.026473546400666237,
+            0.025394288823008537,
+            0.027092531323432922,
+            -0.010197394527494907,
+            0.014379516243934631,
+            -0.0027001271955668926,
+            -0.01072115171700716,
+            0.01644280180335045,
+            -0.005150279030203819,
+            0.002200176939368248,
+            0.0002386914420640096,
+            0.02650528959929943,
+            -0.022013673558831215,
+            -0.017220502719283104,
+            0.015871429815888405,
+            0.016617387533187866,
+            -0.031758733093738556,
+            0.005511354189366102,
+            0.01745857298374176,
+            0.014030344784259796,
+            -0.04240846261382103,
+            -0.0076976437121629715,
+            -0.023854760453104973,
+            -0.00869754422456026,
+            -0.01462552323937416,
+            -0.03939289227128029,
+            0.03291734680533409,
+            -0.003497666446492076,
+            -0.026917945593595505,
+            -0.010975094512104988,
+            -0.032361846417188644,
+            0.011514722369611263,
+            0.014435065910220146,
+            0.025346674025058746,
+            -0.073262520134449,
+            0.011348072439432144,
+            -0.010729086585342884,
+            0.01441125851124525,
+            -0.007975393906235695,
+            -0.008903872221708298,
+            0.02510860376060009,
+            -0.00015301050734706223,
+            0.021854959428310394,
+            0.009260979481041431,
+            0.0016089662676677108,
+            0.04285286366939545,
+            0.008134108036756516,
+            0.028140045702457428,
+            -0.016823716461658478,
+            0.038123175501823425,
+            0.013276451267302036,
+            0.0025275254156440496,
+            -0.00956253707408905,
+            -0.0061700185760855675,
+            0.03980554640293121,
+            0.06275563687086105,
+            0.00560261495411396,
+            0.03374266251921654,
+            0.0013659349642693996,
+            0.014482680708169937,
+            0.004221800714731216,
+            -0.01107825804501772,
+            -0.02980654686689377,
+            0.0040452308021485806,
+            -0.030917545780539513,
+            -0.01955360174179077,
+            -0.011752794496715069,
+            -0.002979861106723547,
+            -0.021410560235381126,
+            -0.0062454077415168285,
+            0.010768765583634377,
+            -0.01018152292817831,
+            0.044693946838378906,
+            0.009816479869186878,
+            -0.02215651609003544,
+            -0.0037297862581908703,
+            0.061136748641729355,
+            0.0048249149695038795,
+            0.03466320410370827,
+            -0.014593780040740967,
+            0.017125273123383522,
+            -0.00552325788885355,
+            -0.0011695260182023048,
+            0.006328732706606388,
+            -0.010729086585342884,
+            -0.025045117363333702,
+            0.03698043152689934,
+            0.002549348631873727,
+            0.033647432923316956,
+            0.011117937043309212,
+            -0.025362545624375343,
+            0.009554601274430752,
+            -0.012617787346243858,
+            0.015688909217715263,
+            0.01417318731546402,
+            0.0067136152647435665,
+            -0.011363944038748741,
+            0.008657865226268768,
+            -0.025299061089754105,
+            0.015300058759748936,
+            -0.0014631474623456597,
+            0.0001887212274596095,
+            0.02306118793785572,
+            -0.017093529924750328,
+            0.0060589187778532505,
+            -0.01376846618950367,
+            -0.0280130747705698,
+            0.015696844086050987,
+            0.024616587907075882,
+            0.021204231306910515,
+            -0.023077059537172318,
+            -0.011578208766877651,
+            0.04355120658874512,
+            0.013943051919341087,
+            -0.002037494909018278,
+            -0.006757261697202921,
+            -0.0017160983989015222,
+            -0.008221400901675224,
+            -0.01087986584752798,
+            -0.012554301880300045,
+            -0.006435865070670843,
+            -0.011998801492154598,
+            0.0490427203476429,
+            -0.014903273433446884,
+            0.00878483708947897,
+            0.011038579978048801,
+            -0.015276252292096615,
+            -0.002704094862565398,
+            0.0030215235892683268,
+            -0.007769065443426371,
+            -0.0056224544532597065,
+            -0.0039757932536304,
+            0.030489018186926842,
+            0.016506288200616837,
+            0.024600718170404434,
+            -0.037900976836681366,
+            -0.010705280117690563,
+            0.002364843152463436,
+            0.02166450209915638,
+            0.021029645577073097,
+            0.005872429348528385,
+            -0.0029600218404084444,
+            -0.0015623439103364944,
+            -0.02888600341975689,
+            -0.015736524015665054,
+            -0.017093529924750328,
+            0.03459971770644188,
+            -0.025394288823008537,
+            0.03313954547047615,
+            0.02123597450554371,
+            0.008530894294381142,
+            0.0052812183275818825,
+            -0.01511753723025322,
+            -0.034726690500974655,
+            0.015379415825009346,
+            -0.01519689429551363,
+            0.0006452728412114084,
+            0.0290923323482275,
+            -0.04374166205525398,
+            -0.04567797854542732,
+            -0.003692091442644596,
+            0.016665002331137657,
+            0.024553103372454643,
+            0.001200276892632246,
+            0.0011070322943851352,
+            0.012625723145902157,
+            -0.012157515622675419,
+            -0.011808344163000584,
+            0.004654297139495611,
+            0.01880764588713646,
+            -0.005860525649040937,
+            -0.0061501795426011086,
+            0.025981532409787178,
+            0.00932446587830782,
+            -0.03926591947674751,
+            -0.002416425384581089,
+            0.006193825975060463,
+            0.022553302347660065,
+            0.02301357500255108,
+            -0.03142543137073517,
+            0.015855558216571808,
+            0.013189158402383327,
+            0.04459872096776962,
+            -0.012990766204893589,
+            0.008388050831854343,
+            0.038662806153297424,
+            -0.005558968521654606,
+            0.03082231804728508,
+            -0.004634457640349865,
+            -0.03345697373151779,
+            0.02328338846564293,
+            0.0011100082192569971,
+            -0.01831563003361225,
+            -0.012411458417773247,
+            0.007070722058415413,
+            0.004777300637215376,
+            0.027886103838682175,
+            0.025457775220274925,
+            0.0033230807166546583,
+            0.028965361416339874,
+            0.038885004818439484,
+            0.0057176826521754265,
+            0.013236773200333118,
+            0.007237372454255819,
+            -0.015069923363626003,
+            -0.00490427203476429,
+            -0.006729486398398876,
+            0.015323866158723831,
+            -0.04002774879336357,
+            0.020061487331986427,
+            -0.003616702277213335,
+            0.01982341706752777,
+            -0.00529312202706933,
+            -0.012435265816748142,
+            -0.0007657965179532766,
+            -0.02220413088798523,
+            0.02391824498772621,
+            0.0025989466812461615,
+            0.001375854597426951,
+            -0.021077260375022888,
+            -0.024426132440567017,
+            0.00475746113806963,
+            -0.02328338846564293,
+            0.011260779574513435,
+            0.026489417999982834,
+            0.010356108658015728,
+            0.003874612972140312,
+            -0.03713914752006531,
+            0.02123597450554371,
+            0.02979067526757717,
+            -0.0021386751905083656,
+            0.017125273123383522,
+            0.030044617131352425,
+            -0.021886702626943588,
+            -0.015942851081490517,
+            0.015934916213154793,
+            -0.02715601772069931,
+            -0.0019750010687857866,
+            -0.0002596466802060604,
+            -0.05082032084465027,
+            -0.04856657609343529,
+            0.013125672936439514,
+            -0.003858741605654359,
+            0.015379415825009346,
+            -0.010483079589903355,
+            -0.005793072283267975,
+            0.004912207834422588,
+            -0.0005872429464943707,
+            0.006527125835418701,
+            -0.0059676580131053925,
+            0.0056264218874275684,
+            -0.013498651795089245,
+            -0.008515022695064545,
+            -0.02790197543799877,
+            -0.005308993626385927,
+            -0.018712416291236877,
+            -0.013514523394405842,
+            -0.027441702783107758,
+            0.020172588527202606,
+            -0.00964189413934946,
+            -0.012419394217431545,
+            0.019871031865477562,
+            0.013101865537464619,
+            0.020347174257040024,
+            0.010197394527494907,
+            -0.003904371988028288,
+            0.028092432767152786,
+            -0.03409183397889137,
+            0.03615511953830719,
+            0.006328732706606388,
+            0.02677510306239128,
+            -0.002079157391563058,
+            -0.020680474117398262,
+            0.009673637337982655,
+            0.03437751904129982,
+            0.0016109502175822854,
+            -0.013808144256472588,
+            -0.015260380692780018,
+            -0.004376546945422888,
+            -0.005059018265455961,
+            -0.007955554872751236,
+            -0.008872129954397678,
+            0.010744958184659481,
+            -0.0036841558758169413,
+            0.001297489390708506,
+            -0.006884233094751835,
+            0.04193231835961342,
+            0.011713115498423576,
+            -0.004539228975772858,
+            -0.023267516866326332,
+            -0.02758454531431198,
+            0.01999800279736519,
+            -0.00280130747705698,
+            -0.024616587907075882,
+            0.026378316804766655,
+            -0.014054151251912117,
+            -0.004654297139495611,
+            0.00723340455442667,
+            -0.018077559769153595,
+            0.013816080056130886,
+            -0.021362945437431335,
+            0.005979561246931553,
+            -0.041837092489004135,
+            -0.011959122493863106,
+            -0.03253643214702606,
+            -0.005856557749211788,
+            0.0177760012447834,
+            -0.007447668816894293,
+            0.02441026084125042,
+            -0.016728486865758896,
+            -0.0015425046440213919,
+            -0.007332601118832827,
+            -0.013887501321732998,
+            -0.0088880006223917,
+            -0.0050947293639183044,
+            -0.005245507694780827,
+            0.0191250741481781,
+            -0.002251759171485901,
+            -0.02144230343401432,
+            0.027870232239365578,
+            -0.024648331105709076,
+            1.071011593012372e-05,
+            -0.04659852012991905,
+            -0.03640906140208244,
+            -0.04085306078195572,
+            0.02375953085720539,
+            -0.0169665589928627,
+            -0.04942363500595093,
+            -0.009657765738666058,
+            0.00434877211228013,
+            0.0159031730145216,
+            -0.02490227483212948,
+            -0.02434677444398403,
+            0.022220002487301826,
+            -0.011086193844676018,
+            0.0034659237135201693,
+            -0.01642693020403385,
+            -0.011760730296373367,
+            -0.017157016322016716,
+            0.04091654717922211,
+            -0.02348971739411354,
+            0.022870730608701706,
+            -0.01492708083242178,
+            0.007296890020370483,
+            -0.007273083087056875,
+            -0.015450838021934032,
+            0.03383788838982582,
+            0.025299061089754105,
+            -0.04713814705610275,
+            0.04018646106123924,
+            0.020982030779123306,
+            0.03086993284523487,
+            -0.04459872096776962,
+            0.016617387533187866,
+            -0.030187461525201797,
+            8.307702228194103e-05,
+            0.04063086211681366,
+            0.004412257578223944,
+            0.009856157936155796,
+            -0.011498851701617241,
+            -0.020172588527202606,
+            0.003795255906879902,
+            0.010221200995147228,
+            0.000911119335796684,
+            -0.02791784703731537,
+            -0.007130240090191364,
+            0.007455604616552591,
+            0.02080744504928589,
+            0.00272790202870965,
+            0.02537841722369194,
+            0.03475843369960785,
+            -0.014935015700757504,
+            0.006967558059841394,
+            -0.0458049476146698,
+            0.011903572827577591,
+            -0.005582775454968214,
+            0.01741096004843712,
+            0.025314930826425552,
+            0.02214064635336399,
+            0.030044617131352425,
+            -0.03748831897974014,
+            0.024648331105709076,
+            0.0037516094744205475,
+            -0.005562936421483755,
+            0.0323459766805172,
+            0.0088880006223917,
+            0.03183808922767639,
+            -0.015355609357357025,
+            -0.011276651173830032,
+            0.007273083087056875,
+            0.02375953085720539,
+            -0.017442701384425163,
+            0.01669674552977085,
+            -0.005305025726556778,
+            0.04945537820458412,
+            0.013831951655447483,
+            -0.03488540276885033,
+            -0.027298860251903534,
+            -0.020918546244502068,
+            -0.00918955821543932,
+            0.012863794341683388,
+            0.01112587284296751,
+            -0.01129252277314663,
+            -0.0312667191028595,
+            0.02382301725447178,
+            -0.010998900979757309,
+            -0.014530294574797153,
+            -0.04132920503616333,
+            -0.007054850924760103,
+            -0.017823616042733192,
+            0.0024322967510670424,
+            -0.003579007461667061,
+            -0.01676023006439209,
+            -0.0052970899268984795,
+            0.006427929271012545,
+            0.025314930826425552,
+            -0.025743460282683372,
+            -0.008935615420341492,
+            -0.038662806153297424,
+            -0.005039179231971502,
+            -0.0023509557358920574,
+            0.01729985885322094,
+            -0.002940182574093342,
+            -0.038726288825273514,
+            0.019490117207169533,
+            0.010649729520082474,
+            -0.01433190144598484,
+            0.043932121247053146,
+            -0.036345575004816055,
+            0.005872429348528385,
+            0.03215551748871803,
+            -0.00173097790684551,
+            -0.03086993284523487,
+            -0.0017270100070163608,
+            0.007273083087056875,
+            0.0021049485076218843,
+            0.002737821778282523,
+            0.018077559769153595,
+            0.031123874709010124,
+            0.01276856567710638,
+            0.012633658945560455,
+            0.06977080553770065,
+            -0.00043224848923273385,
+            -0.03086993284523487,
+            0.02285485900938511,
+            -0.001691299257799983,
+            -0.01067353691905737,
+            -0.01376846618950367,
+            0.023045316338539124,
+            -0.02139468863606453,
+            0.011522658169269562,
+            -0.019109202548861504,
+            -0.0004731670196633786,
+            -0.0344727486371994,
+            -0.030108103528618813,
+            -0.01239558681845665,
+            0.022029545158147812,
+            -0.04104351997375488,
+            -0.029822418466210365,
+            0.018283888697624207,
+            0.017712516710162163,
+            -0.004582875408232212,
+            0.0018589412793517113,
+            -0.001100088469684124,
+            0.009340336546301842,
+            0.01761728711426258,
+            -0.0020989966578781605,
+            -0.007519090082496405,
+            0.009872029535472393,
+            -0.0074952831491827965,
+            -0.0066064829006791115,
+            0.02123597450554371,
+            -0.07034218311309814,
+            -0.04409083351492882,
+            0.024045217782258987,
+            -0.008626122027635574,
+            0.002261678921058774,
+            0.022886602208018303,
+            -6.004484384902753e-05,
+            -0.01525244489312172,
+            -0.016887202858924866,
+            0.03463146090507507,
+            -0.012101965956389904,
+            0.022759631276130676,
+            -0.0015266332775354385,
+            -0.0020989966578781605,
+            0.01314948033541441,
+            0.011879765428602695,
+            -0.014736623503267765,
+            0.0018877083202823997,
+            0.015284187160432339,
+            0.014760429970920086,
+            0.00878483708947897,
+            0.017283987253904343,
+            -0.008967358618974686,
+            -0.004094828851521015,
+            -0.0009175670566037297,
+            0.001588135026395321,
+            -0.0019204431446269155,
+            0.012839986942708492,
+            0.031980931758880615,
+            0.0013272484065964818,
+            0.03498063236474991,
+            0.029060589149594307,
+            0.02085505984723568,
+            0.0036960593424737453,
+            -0.004614618606865406,
+            -0.009451436810195446,
+            0.023584945127367973,
+            0.016585644334554672,
+            0.010054551064968109,
+            0.0099672582000494,
+            0.005459771957248449,
+            -0.022505687549710274,
+            0.006971525959670544,
+            -0.0042813182808458805,
+            -0.008380115032196045,
+            0.0071500795893371105,
+            0.007320697419345379,
+            -0.018775902688503265,
+            0.012800308875739574,
+            0.013117737136781216,
+            0.0022041448391973972,
+            0.02229936048388481,
+            -0.01833150163292885,
+            -0.012697144411504269,
+            0.004618586506694555,
+            -0.012705080211162567,
+            0.025679973885416985,
+            -0.0030631860718131065,
+            0.0019531778525561094,
+            0.011387751437723637,
+            0.07459572702646255,
+            -0.026854461058974266,
+            -0.0014829868450760841,
+            -0.015522259287536144,
+            -0.02026781626045704,
+            0.0090229082852602,
+            -0.023156417533755302,
+            0.022553302347660065,
+            0.01822040230035782,
+            -0.004110700450837612,
+            0.04574146121740341,
+            0.0020712215919047594,
+            0.019966259598731995,
+            0.024315031245350838,
+            -0.009197494015097618,
+            -0.02053763158619404,
+            0.0045908112078905106,
+            0.012617787346243858,
+            0.00024241130449809134,
+            0.010252944193780422,
+            0.023902375251054764,
+            0.013943051919341087,
+            -0.003702011192217469,
+            -0.003840886289253831,
+            -0.021299459040164948,
+            0.025949789211153984,
+            0.010078358463943005,
+            -0.025124475359916687,
+            -0.038599319756031036,
+            0.0003905859775841236,
+            0.01239558681845665,
+            0.01236384455114603,
+            0.023632559925317764,
+            0.0033786308486014605,
+            -0.011554401367902756,
+            0.012379716150462627,
+            -0.014101766049861908,
+            0.003928178921341896,
+            0.0042813182808458805,
+            -0.007185790222138166,
+            -0.010245008394122124,
+            0.011348072439432144,
+            -0.004729686304926872,
+            0.003174286102876067,
+            0.00020422067609615624,
+            0.0031484949868172407,
+            -0.010038679465651512,
+            -0.010602115653455257,
+            -0.012276551686227322,
+            -0.0029540699906647205,
+            -0.005678004119545221,
+            0.032409459352493286,
+            -0.013109801337122917,
+            -0.003598846960812807,
+            -0.01166550163179636,
+            0.030250946059823036,
+            -0.001897627953439951,
+            -0.006412058137357235,
+            -0.018553702160716057,
+            0.004301157779991627,
+            -0.000162558164447546,
+            -0.009760930202901363,
+            -0.011117937043309212,
+            0.006979461759328842,
+            -0.01998213119804859,
+            -0.008459472097456455,
+            -0.0019710331689566374,
+            0.009173686616122723,
+            0.0026842555962502956,
+            -0.014958823099732399,
+            -0.014149379916489124,
+            0.021966060623526573,
+            -0.011562337167561054,
+            -0.014244608581066132,
+            -0.0010862010531127453,
+            -0.013490715995430946,
+            0.013800208456814289,
+            -0.007900004275143147,
+            -0.0008501135162077844,
+            -0.02144230343401432,
+            0.021093131974339485,
+            0.02333100326359272,
+            -0.017109401524066925,
+            -0.010959222912788391,
+            -0.031060390174388885,
+            -0.0005088777397759259,
+            0.0037278023082762957,
+            0.009443501010537148,
+            0.03628208860754967,
+            0.013744658790528774,
+            0.004245607648044825,
+            0.011459172703325748,
+            -0.007487347349524498,
+            0.008142043836414814,
+            0.009721251204609871,
+            -0.010014872997999191,
+            0.0011248876107856631,
+            2.0846751795033924e-05,
+            -0.03409183397889137,
+            -0.007054850924760103,
+            -0.036186862736940384,
+            -0.0048606256023049355,
+            0.005896236281841993,
+            -0.009816479869186878,
+            -0.0013966858386993408,
+            0.007975393906235695,
+            -0.006138275843113661,
+            -0.015482580289244652,
+            0.000179917537025176,
+            0.008348372764885426,
+            -0.008300757966935635,
+            -0.0010207314044237137,
+            -0.008792772889137268,
+            0.016299959272146225,
+            0.022870730608701706,
+            0.02295008860528469,
+            0.007828583009541035,
+            0.0005460763932205737,
+            -0.004082925617694855,
+            0.007693675812333822,
+            -0.01360181625932455,
+            -0.013466908596456051,
+            0.004305125679820776,
+            0.00950698647648096,
+            -0.0055311936885118484,
+            0.007423861417919397,
+            0.013490715995430946,
+            -0.0029421665240079165,
+            0.018093431368470192,
+            0.00622160080820322,
+            -0.004051182419061661,
+            0.008824515156447887,
+            -0.004019439686089754,
+            0.005578808020800352,
+            -0.008300757966935635,
+            -0.004253543447703123,
+            0.007943650707602501,
+            0.004217832814902067,
+            0.0149508873000741,
+            -0.0032238843850791454,
+            -0.020601116120815277,
+            -0.0033369683660566807,
+            0.04586843401193619,
+            -0.00028419779846444726,
+            0.013958923518657684,
+            -0.020680474117398262,
+            -0.013316130265593529,
+            0.01080050878226757,
+            0.014030344784259796,
+            0.010919543914496899,
+            -0.015125473029911518,
+            -0.016680873930454254,
+            0.02360081672668457,
+            0.00435274001210928,
+            -0.005451836157590151,
+            -0.035329803824424744,
+            0.0036782040260732174,
+            -0.03786923363804817,
+            0.02215651609003544,
+            0.017855359241366386,
+            -0.0035036182962358,
+            -0.004701911471784115,
+            0.017474444583058357,
+            -0.013197094202041626,
+            0.020299559459090233,
+            -0.01637931540608406,
+            0.019632959738373756,
+            0.0002527029428165406,
+            -0.006864393595606089,
+            -0.01676023006439209,
+            -0.02053763158619404,
+            0.004059118218719959,
+            -0.013578008860349655,
+            0.04326551780104637,
+            0.024981630966067314,
+            0.0067334542982280254,
+            0.0017498251982033253,
+            -0.027790874242782593,
+            -0.013141544535756111,
+            0.00934827234596014,
+            0.019109202548861504,
+            -0.010943351313471794,
+            -0.01592698134481907,
+            -0.00505108293145895,
+            0.013895437121391296,
+            -0.0049518863670527935,
+            -0.025886302813887596,
+            0.005253443494439125,
+            -0.014784237369894981,
+            0.013427230529487133,
+            -0.010919543914496899,
+            0.008118236437439919,
+            0.016014274209737778,
+            -0.00552325788885355,
+            -0.0014839787036180496,
+            0.03517109155654907,
+            0.012308293953537941,
+            -0.01322883740067482,
+            0.006781068630516529,
+            -0.018791774287819862,
+            -0.01880764588713646,
+            0.03250468894839287,
+            -0.01503024436533451,
+            0.01988690160214901,
+            0.008991165086627007,
+            -0.0242515467107296,
+            0.017109401524066925,
+            -0.021696245297789574,
+            0.008610251359641552,
+            -0.008681672625243664,
+            0.0042813182808458805,
+            -0.016728486865758896,
+            -0.02268027327954769,
+            -0.01619679480791092,
+            -0.02526731789112091,
+            0.01734747365117073,
+            -0.013125672936439514,
+            -0.0023588913027197123,
+            -0.013514523394405842,
+            0.025584746152162552,
+            0.0040392791852355,
+            0.024203931912779808,
+            0.01622060127556324,
+            0.003977777436375618,
+            0.00544390082359314,
+            -0.033171288669109344,
+            0.028520960360765457,
+            0.002400553785264492,
+            0.020616987720131874,
+            0.013919244520366192,
+            0.006296989973634481,
+            0.0019809529185295105,
+            0.007892068475484848,
+            0.031108003109693527,
+            0.004602714907377958,
+            0.0011982930591329932,
+            0.03158414736390114,
+            -0.00918955821543932,
+            0.013665301725268364,
+            -2.7558011424844153e-05,
+            0.00886419415473938,
+            -0.003112784354016185,
+            -0.009538729675114155,
+            0.02866380289196968,
+            0.0010326349874958396,
+            -0.00019132513261865824,
+            0.02355320379137993,
+            0.0034500521142035723,
+            -0.04018646106123924,
+            -0.003662332659587264,
+            -0.020950287580490112,
+            -0.016522159799933434,
+            0.014554101973772049,
+            0.005908139981329441,
+            0.002900503808632493,
+            0.021966060623526573,
+            0.0004959821817465127,
+            0.01075289398431778,
+            -0.008269015699625015,
+            -0.016617387533187866,
+            -0.021839087828993797,
+            -0.0036881237756460905,
+            0.02839398942887783,
+            0.011308394372463226,
+            -0.011268715374171734,
+            0.015712715685367584,
+            -0.006050982978194952,
+            -0.00435274001210928,
+            0.007110401056706905,
+            -0.0006695759948343039,
+            -0.010951287113130093,
+            0.0014512438792735338,
+            0.0062454077415168285,
+            -0.02175973169505596,
+            0.005872429348528385,
+            -0.0076222545467317104,
+            -0.0003261083038523793,
+            -0.008459472097456455,
+            -0.020013874396681786,
+            -0.011165550909936428,
+            -0.0035809914115816355,
+            0.03685346245765686,
+            0.013347873464226723,
+            -0.020601116120815277,
+            -0.014323965646326542,
+            0.0003261083038523793,
+            -0.0031385754700750113,
+            -0.0008193625835701823,
+            -0.0020930448081344366,
+            -0.0008977277902886271,
+            -0.014569973573088646,
+            0.006265247240662575,
+            -0.0006269214791245759,
+            0.010871930047869682,
+            0.011776601895689964,
+            0.007189758121967316,
+            -0.01606188714504242,
+            -0.0005068937898613513,
+            -0.02434677444398403,
+            -0.023077059537172318,
+            -0.004110700450837612,
+            -0.0026505289133638144,
+            -0.017490316182374954,
+            -0.0022894537542015314,
+            0.014601715840399265,
+            -0.0138795655220747,
+            0.029774803668260574,
+            -0.0006576724117621779,
+            -0.002549348631873727,
+            0.044757433235645294,
+            0.012998702004551888,
+            -0.00030775697086937726,
+            -0.0021724021062254906,
+            0.01917268894612789,
+            -0.005797040183097124,
+            -0.0029203433077782393,
+            0.005642293486744165,
+            0.006047015078365803,
+            0.013704979792237282,
+            0.026584645733237267,
+            -0.02090267464518547,
+            0.0024203930515795946,
+            -0.006047015078365803,
+            0.015387351624667645,
+            -0.014244608581066132,
+            0.0002603906614240259,
+            -0.02155340276658535,
+            -0.01613330841064453,
+            -0.017696645110845566,
+            0.006626322399824858,
+            -0.0072770509868860245,
+            -0.010467207990586758,
+            -0.0016526127001270652,
+            -0.012173387221992016,
+            0.012776501476764679,
+            0.020870931446552277,
+            -0.02936214581131935,
+            0.0199345164000988,
+            -0.006419993471354246,
+            0.0149508873000741,
+            -0.00755876861512661,
+            0.015149280428886414,
+            -0.0020315430592745543,
+            -0.013212965801358223,
+            0.006820747163146734,
+            -0.03752006217837334,
+            -0.017633158713579178,
+            -0.0064953831024467945,
+            0.002751709194853902,
+            -0.03152066096663475,
+            -0.0048764972016215324,
+            0.030901676043868065,
+            0.0035809914115816355,
+            -0.012578108347952366,
+            0.002473959233611822,
+            0.025299061089754105,
+            0.002876696875318885,
+            0.015268316492438316,
+            0.002487846650183201,
+            -0.016887202858924866,
+            -0.004340836312621832,
+            0.02521970309317112,
+            -0.011848023161292076,
+            -0.01026881579309702,
+            -0.024807045236229897,
+            0.03605988994240761,
+            -0.008872129954397678,
+            -0.00140065373852849,
+            0.023949988186359406,
+            -0.006562836468219757,
+            0.005705779418349266,
+            0.0021803376730531454,
+            -0.020553503185510635,
+            0.0231722891330719,
+            0.029520859941840172,
+            0.004197993315756321,
+            -0.02936214581131935,
+            -0.004622553940862417,
+            -0.003910323604941368,
+            0.003116752253845334,
+            0.00015648239059373736,
+            0.002900503808632493,
+            0.002198193222284317,
+            -0.009864093735814095,
+            -0.02079157344996929,
+            -0.00028742168797180057,
+            -0.015720652416348457,
+            -0.010903672315180302,
+            0.018553702160716057,
+            -0.006102565210312605,
+            0.015577808953821659,
+            0.01376846618950367,
+            0.009610150940716267,
+            0.0057494258508086205,
+            0.020886803045868874,
+            0.02893361821770668,
+            -0.014958823099732399,
+            0.00951492227613926,
+            -0.017490316182374954,
+            0.01245113741606474,
+            0.007070722058415413,
+            -0.006669968832284212,
+            -0.03834537789225578,
+            0.008951487019658089,
+            0.007423861417919397,
+            0.014276351779699326,
+            -0.013395487330853939,
+            -0.024775303900241852,
+            -0.036885205656290054,
+            -0.016665002331137657,
+            0.003930163104087114,
+            0.005725618451833725,
+            0.005801007617264986,
+            -0.008522958494722843,
+            -0.005269315093755722,
+            -0.004471775609999895,
+            0.004209897015243769,
+            0.013435166329145432,
+            -0.049106206744909286,
+            -0.019490117207169533,
+            0.006205729208886623,
+            -0.01676023006439209,
+            0.016363445669412613,
+            0.007431797217577696,
+            -0.013236773200333118,
+            0.009737122803926468,
+            -0.016117436811327934,
+            0.000434232410043478,
+            -0.00926891528069973,
+            0.001266738516278565,
+            -0.004194025415927172,
+            0.008967358618974686,
+            0.019585344940423965,
+            0.034948889166116714,
+            -0.007753193844109774,
+            0.003061202121898532,
+            -0.008848322555422783,
+            0.0007211581105366349,
+            0.01029262226074934,
+            0.006332700606435537,
+            -0.005340736359357834,
+            -0.00027403017156757414,
+            -0.02015671692788601,
+            0.015450838021934032,
+            -0.020140845328569412,
+            0.00280130747705698,
+            0.00023273965052794665,
+            -0.0029143914580345154,
+            -0.0019859126769006252,
+            -0.008102364838123322,
+            -0.004928078968077898,
+            -0.007935714907944202,
+            -0.0019630976021289825,
+            0.02042653039097786,
+            -0.02187083102762699,
+            0.00217637000605464,
+            0.0018648931290954351,
+            0.004745557904243469,
+            -0.027870232239365578,
+            -0.0051859901286661625,
+            -0.009705379605293274,
+            -0.014157315716147423,
+            -0.023204030469059944,
+            0.026219602674245834,
+            0.016228538006544113,
+            0.007566704414784908,
+            0.004678104072809219,
+            0.0022954056039452553,
+            -0.0010653698118403554,
+            -0.013657365925610065,
+            0.003325064666569233,
+            -0.001118935877457261,
+            -0.0026683842297643423,
+            -0.0033111772499978542,
+            0.024743560701608658,
+            -0.014823916368186474,
+            0.008634057827293873,
+            -0.0014790189452469349,
+            0.0073683117516338825,
+            0.005900204181671143,
+            0.0121257733553648,
+            0.012030544690787792,
+            0.019966259598731995,
+            0.012117837555706501,
+            0.014593780040740967,
+            0.06742183864116669,
+            -0.0058248150162398815,
+            -0.0032457076013088226,
+            0.0031544468365609646,
+            -0.02080744504928589,
+            -0.01902984455227852,
+            0.004638425540179014,
+            -0.019791673868894577,
+            0.029504990205168724,
+            -0.00918955821543932,
+            0.009713315404951572,
+            0.005725618451833725,
+            -0.007066754624247551,
+            0.017982330173254013,
+            -0.002755677094683051,
+            -0.002307309303432703,
+            0.007927779108285904,
+            0.01438745204359293,
+            0.034028347581624985,
+            0.014681072905659676,
+            -0.016395187005400658,
+            0.0026842555962502956,
+            -0.02252155914902687,
+            0.0028588413260877132,
+            -0.011482980102300644,
+            0.005328832659870386,
+            -0.013744658790528774,
+            0.026584645733237267,
+            0.0011318314354866743,
+            0.02661638893187046,
+            -0.016395187005400658,
+            0.0015365529106929898,
+            -0.014522358775138855,
+            0.0007335576810874045,
+            0.015593680553138256,
+            0.017982330173254013,
+            -0.0038567576557397842,
+            0.006729486398398876,
+            -0.013657365925610065,
+            0.00803094357252121,
+            0.014189058914780617,
+            0.015014373697340488,
+            0.001624837750568986,
+            -0.011340136639773846,
+            -0.011895637027919292,
+            0.02215651609003544,
+            0.0004414241702761501,
+            0.006118436343967915,
+            0.012459073215723038,
+            -0.014474744908511639,
+            0.00011543985601747409,
+            -0.005269315093755722,
+            0.01599840261042118,
+            -0.008026976138353348,
+            0.004376546945422888,
+            -0.006118436343967915,
+            -0.019379016011953354,
+            -0.005495482590049505,
+            -0.02252155914902687,
+            0.01718875952064991,
+            0.00046721522812731564,
+            0.00801110453903675,
+            -0.021839087828993797,
+            -0.01739508844912052,
+            -0.006876297295093536,
+            0.015379415825009346,
+            2.8162488888483495e-05,
+            0.010919543914496899,
+            0.012086094357073307,
+            0.01042752992361784,
+            -0.007400054484605789,
+            0.0004890384734608233,
+            0.02215651609003544,
+            -0.010316429659724236,
+            0.010395786724984646,
+            0.00739608658477664,
+            -0.005435965023934841,
+            -0.032250747084617615,
+            -0.004301157779991627,
+            -0.006820747163146734,
+            0.020236073061823845,
+            0.002410473534837365,
+            -0.01492708083242178,
+            0.009768865071237087,
+            -0.0409482903778553,
+            -0.009451436810195446,
+            -0.0027140146121382713,
+            0.025092732161283493,
+            0.011165550909936428,
+            -0.0069159758277237415,
+            0.01950598880648613,
+            0.008753093890845776,
+            0.013117737136781216,
+            -0.0013619671808555722,
+            -0.007324665319174528,
+            0.0036206701770424843,
+            -0.002962005790323019,
+            0.0028588413260877132,
+            -0.009959322400391102,
+            0.002946134191006422,
+            0.015561937354505062,
+            0.004174186382442713,
+            -0.011411558836698532,
+            -0.005983529146760702,
+            -0.015363545157015324,
+            -0.016117436811327934,
+            0.0026862395461648703,
+            0.00037248263834044337,
+            0.011919444426894188,
+            -0.007903972640633583,
+            -0.004209897015243769,
+            -0.015887301415205002,
+            0.019632959738373756,
+            0.0012320197420194745,
+            0.0009061594610102475,
+            -0.00824520830065012,
+            0.003061202121898532,
+            -0.012657465413212776,
+            -0.007273083087056875,
+            0.008745158091187477,
+            -0.00021451229986269027,
+            -0.0080864941701293,
+            0.024600718170404434,
+            -0.02445787377655506,
+            0.0049677579663693905,
+            -0.010149779729545116,
+            -0.028108302503824234,
+            -0.00940382294356823,
+            -0.03282211720943451,
+            -0.013133608736097813,
+            0.0039202431216835976,
+            -0.011863894760608673,
+            -0.007820647209882736,
+            0.01556987315416336,
+            -0.008074590004980564,
+            -0.005023307632654905,
+            0.015982531011104584,
+            0.003579007461667061,
+            0.0160380806773901,
+            -0.005761329084634781,
+            0.0027715484611690044,
+            -0.01750618778169155,
+            -0.006003368645906448,
+            -0.01463345903903246,
+            0.009824415668845177,
+            0.0016952671576291323,
+            0.013078059069812298,
+            -0.017093529924750328,
+            0.02106138877570629,
+            -0.010006937198340893,
+            0.00474159000441432,
+            -0.03507586196064949,
+            -0.01231622975319624,
+            0.017172887921333313,
+            0.016165051609277725,
+            -0.006003368645906448,
+            -0.00700326869264245,
+            -0.020759830251336098,
+            -0.010744958184659481,
+            -0.014681072905659676,
+            -0.02753693237900734,
+            0.011903572827577591,
+            0.00038339424645528197,
+            0.021188359707593918,
+            0.004142443183809519,
+            0.030457274988293648,
+            0.013046315871179104,
+            0.019839288666844368,
+            -0.0034302128478884697,
+            0.0011020724195986986,
+            -0.0038785808719694614,
+            0.015561937354505062,
+            -0.005804975517094135,
+            -0.012292423285543919,
+            -0.005741490051150322,
+            0.0231722891330719,
+            -0.009308594278991222,
+            0.002333100186660886,
+            0.005908139981329441,
+            0.024553103372454643,
+            0.0038726290222257376,
+            0.01714114472270012,
+            0.004392418544739485,
+            -0.002789404010400176,
+            0.03647254779934883,
+            -0.0027021111454814672,
+            -0.021854959428310394,
+            -0.0009011996444314718,
+            0.011705179698765278,
+            -0.021950189024209976,
+            -0.008856258355081081,
+            0.001166550093330443,
+            -0.022172387689352036,
+            -0.015942851081490517,
+            -2.3326663722400554e-05,
+            -0.00801904033869505,
+            0.027330603450536728,
+            0.0055311936885118484,
+            -0.008594379760324955,
+            -0.016522159799933434,
+            0.010705280117690563,
+            0.025997404009103775,
+            -0.0008962398278526962,
+            0.015173087827861309,
+            -0.014776301570236683,
+            -0.01761728711426258,
+            -0.0054994504898786545,
+            0.013681173324584961,
+            0.004182121716439724,
+            -0.0047336542047560215,
+            -0.05977180600166321,
+            -0.01815691590309143,
+            0.006086693610996008,
+            0.002291437704116106,
+            -0.026394188404083252,
+            -0.008388050831854343,
+            -0.017823616042733192,
+            -0.008911808021366596,
+            0.0029639897402375937,
+            0.02285485900938511,
+            -0.008538829162716866,
+            0.011689308099448681,
+            0.019744059070944786,
+            -0.01745857298374176,
+            -0.0007796840509399772,
+            0.00395992211997509,
+            -0.019379016011953354,
+            -0.005431997124105692,
+            0.0030393789056688547,
+            -0.018188659101724625,
+            -0.005614518653601408,
+            -0.0011754778679460287,
+            -0.007602415047585964,
+            0.012046415358781815,
+            -0.017601415514945984,
+            0.022394588217139244,
+            0.004194025415927172,
+            -0.011681373231112957,
+            -0.017744259908795357,
+            -0.03169524669647217,
+            -0.007741290144622326,
+            -0.002170418156310916,
+            -0.008713414892554283,
+            0.011149680241942406,
+            0.019267916679382324,
+            0.013760530389845371,
+            0.006173986475914717,
+            0.002852889709174633,
+            0.004761429037898779,
+            -2.9216451366664842e-05,
+            0.0005525241722352803,
+            0.003467907663434744,
+            -0.008530894294381142,
+            -0.0003127167874481529,
+            0.0077889044769108295,
+            -0.013506587594747543,
+            -0.021537531167268753,
+            0.009467308409512043,
+            0.03605988994240761,
+            -0.011411558836698532,
+            0.0038865164387971163,
+            -0.005618486553430557,
+            0.007566704414784908,
+            0.006630290299654007,
+            0.03012397512793541,
+            0.01655390113592148,
+            -0.022220002487301826,
+            0.00940382294356823,
+            -0.010959222912788391,
+            0.01425254438072443,
+            -0.0047495258040726185,
+            0.01961708813905716,
+            0.016395187005400658,
+            0.01586349494755268,
+            0.008181722834706306,
+            0.004813011270016432,
+            -0.0101021658629179,
+            -0.0033210967667400837,
+            0.014879466034471989,
+            -0.00872928649187088,
+            -0.011236973106861115,
+            0.0004543196992017329,
+            0.006316829472780228,
+            -0.01276856567710638,
+            -0.0160380806773901,
+            0.0034202933311462402,
+            0.009364143945276737,
+            0.03304431959986687,
+            -0.0036543968599289656,
+            0.003364743199199438,
+            -0.00654696486890316,
+            -0.013355808332562447,
+            -0.02166450209915638,
+            -0.02133120223879814,
+            0.012347972951829433,
+            -0.00739608658477664,
+            -0.0011437350185588002,
+            -0.014363644644618034,
+            -0.020997902378439903,
+            0.0035809914115816355,
+            -0.01441125851124525,
+            -0.00034569960553199053,
+            -0.002469991333782673,
+            -0.0007295898394659162,
+            -0.0028747129254043102,
+            0.0038369183894246817,
+            0.02133120223879814,
+            -0.016299959272146225,
+            0.013141544535756111,
+            0.005614518653601408,
+            0.0039698416367173195,
+            0.02542603202164173,
+            0.007455604616552591,
+            0.009118136949837208,
+            -0.016569772735238075,
+            0.025251446291804314,
+            -0.027600416913628578,
+            -0.014617587439715862,
+            -0.010030743665993214,
+            0.021188359707593918,
+            -0.0049836291000247,
+            0.011030644178390503,
+            0.016665002331137657,
+            0.005384382791817188,
+            -0.001797439530491829,
+            0.026108503341674805,
+            0.02069634571671486,
+            -0.001594086759723723,
+            0.008681672625243664,
+            -0.0005287170060910285,
+            0.02274375967681408,
+            -0.005920043680816889,
+            -0.025156216695904732,
+            0.011522658169269562,
+            0.009332400746643543,
+            0.0051701185293495655,
+            -0.0014075974468141794,
+            -0.013887501321732998,
+            -0.012213066220283508,
+            -0.009499051608145237,
+            -0.0177918728441,
+            -0.019696446135640144,
+            -0.007566704414784908,
+            0.0013074091402813792,
+            -0.01107825804501772,
+            -0.0019333385862410069,
+            -0.01783948764204979,
+            -0.02312467433512211,
+            -0.016474545001983643,
+            0.027568675577640533,
+            -0.0001265994505956769,
+            0.01592698134481907,
+            0.004400353878736496,
+            0.006217632908374071,
+            -0.019490117207169533,
+            -0.005035211332142353,
+            0.023156417533755302,
+            0.01761728711426258,
+            -0.003640509443357587,
+            -0.003360775299370289,
+            0.021886702626943588,
+            0.0017250260571017861,
+            -0.007011204492300749,
+            -0.0017418895149603486,
+            -0.0010385867208242416,
+            0.012990766204893589,
+            0.01207022275775671,
+            -0.008126172237098217,
+            -0.030790574848651886,
+            0.007836518809199333,
+            0.0007940675131976604,
+            -0.010895737446844578,
+            0.012086094357073307,
+            -0.0037416897248476744,
+            -0.013800208456814289,
+            0.005447868723422289,
+            -0.024013474583625793,
+            0.00014643874601460993,
+            -0.005812911316752434,
+            0.008181722834706306,
+            -0.03834537789225578,
+            0.003967857453972101,
+            0.014157315716147423,
+            -0.004495582543313503,
+            -0.007765097543597221,
+            -0.01128458697348833,
+            -0.021045517176389694,
+            0.002924310974776745,
+            0.009999001398682594,
+            0.009459372609853745,
+            -0.0026941753458231688,
+            0.010070422664284706,
+            -0.008840386755764484,
+            0.0012379715917631984,
+            0.006423961371183395,
+            0.012006737291812897,
+            0.012482879683375359,
+            0.020680474117398262,
+            0.0008382099331356585,
+            -0.00832456536591053,
+            0.005876397248357534,
+            -0.002870745025575161,
+            0.02080744504928589,
+            -0.01822040230035782,
+            0.00022976375475991517,
+            8.462696132482961e-05,
+            -0.003707963041961193,
+            -0.022807246074080467,
+            0.0020355109591037035,
+            5.387606506701559e-05,
+            0.010205329395830631,
+            0.014379516243934631,
+            0.005701811518520117,
+            0.002110900357365608,
+            -0.000986508559435606,
+            0.007769065443426371,
+            0.013673237524926662,
+            -0.009149880148470402,
+            -0.015307994559407234,
+            -0.02085505984723568,
+            -0.014070022851228714,
+            0.010451337322592735,
+            -0.0037000272423028946,
+            -0.0029997003730386496,
+            -0.003422277281060815,
+            0.012720951810479164,
+            0.011030644178390503,
+            -0.009165750816464424,
+            0.02236284501850605,
+            -0.016069823876023293,
+            -0.004396386444568634,
+            -0.01087986584752798,
+            0.01203848049044609,
+            0.0015434966189786792,
+            -0.01301457267254591,
+            0.006558868568390608,
+            0.008364243432879448,
+            -0.0008783844532445073,
+            -0.009983129799365997,
+            -0.0023807145189493895,
+            -0.003507586196064949,
+            0.003283402184024453,
+            0.007451636716723442,
+            -0.0007315737311728299,
+            0.002406505635008216,
+            0.017268115654587746,
+            0.003957937937229872,
+            0.005543096922338009,
+            0.013728787191212177,
+            -0.0017895037308335304,
+            -0.013657365925610065,
+            0.00436464324593544,
+            -0.028108302503824234,
+            -0.0047177826054394245,
+            0.011578208766877651,
+            -0.004630489740520716,
+            -0.000541116576641798,
+            -0.006880265194922686,
+            0.0571054071187973,
+            0.00369804329238832,
+            -0.015720652416348457,
+            0.0011873814510181546,
+            0.008507086895406246,
+            0.023410359397530556,
+            -0.0003945538483094424,
+            0.011967058293521404,
+            0.001203252817504108,
+            0.022870730608701706,
+            0.01977580226957798,
+            -0.02560061775147915,
+            -0.005570872221142054,
+            -0.003928178921341896,
+            0.00583275081589818,
+            0.0301715899258852,
+            0.02866380289196968,
+            0.0062612793408334255,
+            0.0006834634696133435,
+            0.005424061324447393,
+            0.021156616508960724,
+            0.003944050520658493,
+            -0.017760131508111954,
+            0.0026306896470487118,
+            0.0024203930515795946,
+            -0.006582675967365503,
+            0.005888300947844982,
+            -0.016887202858924866,
+            -0.029949389398097992,
+            0.018093431368470192,
+            7.718722918070853e-05,
+            -0.002864793175831437,
+            -0.003094929037615657,
+            -0.018839387223124504,
+            0.01222893688827753,
+            0.013593880459666252,
+            0.0017547850729897618,
+            -0.021410560235381126,
+            -0.002325164619833231,
+            0.006689807865768671,
+            -0.012601915746927261,
+            0.013966858386993408,
+            -0.014022408984601498,
+            -0.031869832426309586,
+            0.0034579879138618708,
+            -0.005535161588340998,
+            0.013236773200333118,
+            -0.0015147296944633126,
+            -0.015379415825009346,
+            -0.009356208145618439,
+            0.024219803512096405,
+            -0.01285585854202509,
+            0.0017041948158293962,
+            -0.005943850614130497,
+            -0.01150678750127554,
+            -0.001055450178682804,
+            0.0001287073828279972,
+            0.008292822167277336,
+            7.19174204277806e-05,
+            -0.01253049448132515,
+            0.01282411627471447,
+            0.024442002177238464,
+            0.0009131032275035977,
+            -0.010784637182950974,
+            -0.01107825804501772,
+            -0.01083225104957819,
+            0.002688223496079445,
+            -0.010491015389561653,
+            0.0017954555805772543,
+            -0.021886702626943588,
+            -0.00956253707408905,
+            0.01236384455114603,
+            -0.002618785947561264,
+            -0.0034500521142035723,
+            0.010657665319740772,
+            -0.0010023800423368812,
+            -0.0006889192736707628,
+            0.016903072595596313,
+            -0.013784337788820267,
+            0.0030413628555834293,
+            0.024267416447401047,
+            -0.0010931447613984346,
+            -0.003392518265172839,
+            -0.006400154437869787,
+            0.0050272755324840546,
+            -0.0002998212294187397,
+            -0.01655390113592148,
+            -0.006293022073805332,
+            -0.017522059381008148,
+            -0.009356208145618439,
+            -0.006848522461950779,
+            -0.018633060157299042,
+            -0.002815194893628359,
+            0.001860925229266286,
+            0.012340037152171135,
+            0.014744559302926064,
+            -0.0019144912948831916,
+            0.006126372143626213,
+            0.02053763158619404,
+            -0.01541115902364254,
+            0.014823916368186474,
+            0.00276956451125443,
+            0.007800808176398277,
+            0.005793072283267975,
+            0.017426829785108566,
+            0.009141944348812103,
+            0.0015028261113911867,
+            0.0036186862271279097,
+            -0.006967558059841394,
+            0.002551332348957658,
+            0.00646760780364275,
+            -0.02247394621372223,
+            0.006352540105581284,
+            0.00021810817997902632,
+            -0.00420196121558547,
+            -0.017744259908795357,
+            -0.004257511347532272,
+            0.002378730569034815,
+            0.007737322244793177,
+            -0.004305125679820776,
+            -0.00040571345016360283,
+            -0.02058524452149868,
+            -0.005320896860212088,
+            -0.0020394788589328527,
+            -0.026330703869462013,
+            -0.005785136483609676,
+            0.0008957438403740525,
+            0.013300258666276932,
+            -0.007661933079361916,
+            -0.005551032721996307,
+            0.022489817813038826,
+            0.011967058293521404,
+            -0.013070123270154,
+            -0.0010693375952541828,
+            -0.0029818450566381216,
+            -0.005098697263747454,
+            0.01644280180335045,
+            0.013220901601016521,
+            -0.00030130919185467064,
+            0.011935316026210785,
+            -0.008792772889137268,
+            0.00692391162738204,
+            -0.007939683273434639,
+            -0.0040611024014651775,
+            0.012586044147610664,
+            -0.011808344163000584,
+            0.00404919870197773,
+            0.0140065373852849,
+            0.012792373076081276,
+            -0.017268115654587746,
+            -0.0070270756259560585,
+            -0.01793471723794937,
+            0.0028409860096871853,
+            0.002198193222284317,
+            -0.008713414892554283,
+            0.008356308564543724,
+            0.0033151451498270035,
+            0.007661933079361916,
+            0.012657465413212776,
+            -0.007106433156877756,
+            0.023949988186359406,
+            0.007292922120541334,
+            -0.0019085395615547895,
+            0.01191150862723589,
+            -0.014673137106001377,
+            0.014728687703609467,
+            0.011490915901958942,
+            -0.0009711331804282963,
+            -0.016903072595596313,
+            -0.0016992350574582815,
+            0.0280289463698864,
+            -0.021696245297789574,
+            0.007392118684947491,
+            -0.003473859280347824,
+            -0.01756967417895794,
+            -0.013887501321732998,
+            -0.009792672470211983,
+            0.007098497357219458,
+            -0.010792572982609272,
+            0.0005416125641204417,
+            -0.019363144412636757,
+            -0.009578408673405647,
+            -0.00011240196909056976,
+            0.01745857298374176,
+            -0.001312368898652494,
+            -0.007808743976056576,
+            0.00809839740395546,
+            0.008721350692212582,
+            -0.010689408518373966,
+            -0.03012397512793541,
+            0.02650528959929943,
+            -0.0025810913648456335,
+            -0.008467407897114754,
+            -0.0013748626224696636,
+            -0.0036365415435284376,
+            0.0048606256023049355,
+            -0.018601316958665848,
+            -0.00811030063778162,
+            -0.007812711410224438,
+            -0.007979361340403557,
+            0.005558968521654606,
+            0.00436464324593544,
+            0.0019591297022998333,
+            -0.008721350692212582,
+            0.006217632908374071,
+            -0.005483579356223345,
+            -0.01622060127556324,
+            -0.01691894419491291,
+            0.010737022385001183,
+            -0.006892168894410133,
+            -0.013538329862058163,
+            -0.008050783537328243,
+            0.0038865164387971163,
+            -0.003188173519447446,
+            -0.006102565210312605,
+            0.01691894419491291,
+            -0.005332800559699535,
+            -0.016569772735238075,
+            0.001228051958605647,
+            0.01004661526530981,
+            0.017522059381008148,
+            0.016569772735238075,
+            0.03367917612195015,
+            0.0015682957600802183,
+            -0.017601415514945984,
+            0.010649729520082474,
+            -8.43944690132048e-06,
+            0.0031961093191057444,
+            0.013831951655447483,
+            -0.01849021576344967,
+            0.010697344318032265,
+            -0.019521860405802727,
+            0.010094230063259602,
+            -0.013736722990870476,
+            -0.005162182729691267,
+            -0.018998103216290474,
+            -0.01772838830947876,
+            -0.0008223385084420443,
+            -0.003146511036902666,
+            0.01355420146137476,
+            0.0005530201597139239,
+            -0.012713016010820866,
+            0.008364243432879448,
+            0.017363345250487328,
+            -0.038726288825273514,
+            -0.011133808642625809,
+            -0.004729686304926872,
+            0.0036484450101852417,
+            -0.017268115654587746,
+            0.02042653039097786,
+            0.023838888853788376,
+            -0.003993648570030928,
+            0.003088977187871933,
+            0.00404919870197773,
+            0.005471675656735897,
+            0.010562436655163765,
+            -0.01810930296778679,
+            0.005118536297231913,
+            -0.036885205656290054,
+            0.0010891768615692854,
+            0.016728486865758896,
+            0.02418806031346321,
+            -0.006281118839979172,
+            -0.01642693020403385,
+            -0.012347972951829433,
+            -0.009602215141057968,
+            -0.031012775376439095,
+            -0.006550932768732309,
+            -0.01368910912424326,
+            -0.0026326735969632864,
+            -0.007844454608857632,
+            -0.01236384455114603,
+            0.024600718170404434,
+            0.008427729830145836,
+            -0.0019283788278698921,
+            -0.004686039872467518,
+            0.003116752253845334,
+            -0.024330902844667435,
+            -0.01622060127556324,
+            -0.014204930514097214,
+            0.005801007617264986,
+            0.021426431834697723,
+            -0.0033786308486014605,
+            0.012593979947268963,
+            -0.011395687237381935,
+            0.001903579686768353,
+            0.0024719752836972475,
+            0.004182121716439724,
+            -0.013649430125951767,
+            0.032044418156147,
+            -0.0012171403504908085,
+            -0.0002213320549344644,
+            0.0037238344084471464,
+            -0.004602714907377958,
+            0.014974694699048996,
+            0.006717583164572716,
+            -0.013324066065251827,
+            -0.010657665319740772,
+            0.01741096004843712,
+            0.01549051608890295,
+            0.0019174672197550535,
+            -0.01174485869705677,
+            -0.0005376447224989533,
+            0.008126172237098217,
+            -0.006023207679390907,
+            -0.003297289600595832,
+            -0.00013267523900140077,
+            0.0005698835593648255,
+            0.02769564650952816,
+            0.005487547256052494,
+            0.005090761464089155,
+            -0.011340136639773846,
+            0.015101666562259197,
+            0.017014173790812492,
+            0.005535161588340998,
+            -0.0013084009988233447,
+            -0.020664602518081665,
+            0.004186089616268873,
+            -0.021220102906227112,
+            -0.003416325431317091,
+            -0.008951487019658089,
+            -0.0034778271801769733,
+            0.004559068474918604,
+            -0.0025394288823008537,
+            -0.016791973263025284,
+            0.00607478991150856,
+            -0.009848223067820072,
+            -4.6684326662216336e-05,
+            0.009181622415781021,
+            -0.013347873464226723,
+            0.013022508472204208,
+            0.029552603140473366,
+            0.018823517486453056,
+            0.015514323487877846,
+            -0.0025275254156440496,
+            -0.02241045981645584,
+            0.02321990206837654,
+            0.008761029690504074,
+            0.006114468444138765,
+            0.0010078358463943005,
+            -0.007392118684947491,
+            0.025505388155579567,
+            0.029822418466210365,
+            0.002785436110571027,
+            0.006741390097886324,
+            -0.006332700606435537,
+            0.013514523394405842,
+            0.00864992942661047,
+            -0.006717583164572716,
+            0.0006824714946560562,
+            0.018521958962082863,
+            -0.0066223544999957085,
+            -0.003807159373536706,
+            0.031076261773705482,
+            -0.023156417533755302,
+            -0.012419394217431545,
+            -0.013458972796797752,
+            -0.0021168519742786884,
+            0.00428528618067503,
+            0.017522059381008148,
+            -0.021839087828993797,
+            -0.008459472097456455,
+            0.013141544535756111,
+            0.008792772889137268,
+            0.00013689108891412616,
+            -0.008713414892554283,
+            0.0006621362408623099,
+            0.010713215917348862,
+            -0.006991364993155003,
+            0.0005802991800010204,
+            -0.029917646199464798,
+            -0.0027576610445976257,
+            -0.0023410359863191843,
+            0.0027775003109127283,
+            0.04339249059557915,
+            -0.002136691240593791,
+            0.00878483708947897,
+            0.003140559419989586,
+            0.008078558370471,
+            -0.007185790222138166,
+            -0.014435065910220146,
+            0.0011625823099166155,
+            0.01642693020403385,
+            -0.00584862194955349,
+            0.010435465723276138,
+            -0.0008883041446097195,
+            -0.027346475049853325,
+            0.017331602051854134,
+            0.03659952059388161,
+            0.006082725711166859,
+            -0.005428029224276543,
+            -0.005900204181671143,
+            -0.005150279030203819,
+            -0.012371780350804329,
+            -0.007328633219003677,
+            0.013506587594747543,
+            0.0012657465413212776,
+            0.027235373854637146,
+            -0.015061987563967705,
+            0.005150279030203819,
+            -0.005816879216581583,
+            -0.025632360950112343,
+            0.010824315249919891,
+            0.004309093579649925,
+            0.00013974298781249672,
+            0.01563335955142975,
+            0.020616987720131874,
+            -0.03082231804728508,
+            -0.011030644178390503,
+            -0.005364543292671442,
+            -0.021696245297789574,
+            0.016188859939575195,
+            -0.006864393595606089,
+            0.01503024436533451,
+            -0.002779484260827303,
+            0.007907940074801445,
+            -0.011236973106861115,
+            0.004293221980333328,
+            0.019712315872311592,
+            -0.004086893517524004,
+            0.018982231616973877,
+            -0.009983129799365997,
+            -0.007618286646902561,
+            0.013355808332562447,
+            -0.016633259132504463,
+            0.020775701850652695,
+            -0.005678004119545221,
+            -0.002283502137288451,
+            0.021918445825576782,
+            -0.01347484439611435,
+            0.012355908751487732,
+            0.0030314433388412,
+            -0.01309392973780632,
+            -0.00044415207230485976,
+            -0.0033389520831406116,
+            0.0013252644566819072,
+            -0.009610150940716267,
+            -0.019474245607852936,
+            0.021029645577073097,
+            -0.0180458165705204,
+            0.023807145655155182,
+            0.0015028261113911867,
+            -0.011348072439432144,
+            0.001473067095503211,
+            -0.007971425540745258,
+            -0.023521460592746735,
+            0.0015752394683659077,
+            -0.00280130747705698,
+            0.011816279962658882,
+            -0.009475244209170341,
+            0.020601116120815277,
+            0.0057335542514920235,
+            -0.016165051609277725,
+            0.00047986279241740704,
+            0.011570272967219353,
+            -0.0006963590276427567,
+            0.005158214829862118,
+            0.027711518108844757,
+            -0.001079257228411734,
+            0.0029957324732095003,
+            0.006054950878024101,
+            0.008745158091187477,
+            -0.03352046012878418,
+            -0.009673637337982655,
+            0.004047214984893799,
+            -0.0004498558701016009,
+            0.010149779729545116,
+            0.005221700761467218,
+            -0.010221200995147228,
+            0.022712016478180885,
+            -0.00013354320253711194,
+            -0.017744259908795357,
+            0.008475343696773052,
+            -0.009205429814755917,
+            -0.011602015234529972,
+            -0.009284786880016327,
+            -0.002348971785977483,
+            0.021029645577073097,
+            -0.005820847116410732,
+            0.010030743665993214,
+            0.0017418895149603486,
+            0.003398470114916563,
+            -0.002497766399756074,
+            -0.017966458573937416,
+            0.002624737797304988,
+            -0.004618586506694555,
+            0.020188460126519203,
+            0.015919044613838196,
+            0.015823816880583763,
+            -0.00911020115017891,
+            -0.015593680553138256,
+            0.004725718405097723,
+            0.02796545997262001,
+            -0.009895836934447289,
+            0.0071699186228215694,
+            -0.004011503886431456,
+            0.0013937100302428007,
+            -0.006539029534906149,
+            -0.013133608736097813,
+            -0.015228637494146824,
+            -0.03482192009687424,
+            -0.0019690494518727064,
+            0.002146610990166664,
+            -0.00044415207230485976,
+            -0.012919344007968903,
+            -0.03685346245765686,
+            -0.007911908440291882,
+            -0.02052175998687744,
+            -0.019585344940423965,
+            -0.02807656116783619,
+            -0.02445787377655506,
+            -0.004880465101450682,
+            -0.014784237369894981,
+            0.011244908906519413,
+            -0.0034460844472050667,
+            -0.009792672470211983,
+            0.0037754166405647993,
+            0.023045316338539124,
+            0.00047267103218473494,
+            0.02042653039097786,
+            -0.013760530389845371,
+            0.008880065754055977,
+            -0.007530993781983852,
+            0.007308793719857931,
+            -0.006360475905239582,
+            -0.01649041660130024,
+            0.00654696486890316,
+            0.008070622570812702,
+            0.021140744909644127,
+            -0.01129252277314663,
+            0.0018837404204532504,
+            -0.006285086274147034,
+            0.0013431197730824351,
+            -0.00023447559215128422,
+            0.000928974652197212,
+            0.006709647364914417,
+            0.020188460126519203,
+            -0.02625134587287903,
+            -0.018775902688503265,
+            0.008126172237098217,
+            0.009760930202901363,
+            0.021807344630360603,
+            -0.021632758900523186,
+            0.013284387066960335,
+            -2.7294519895804115e-05,
+            -0.00575339375063777,
+            0.013220901601016521,
+            0.012078158557415009,
+            -0.005277250427752733,
+            0.0024997503496706486,
+            0.02301357500255108,
+            0.012467009015381336,
+            -0.009673637337982655,
+            -0.020601116120815277,
+            0.0088880006223917,
+            0.0006874313112348318,
+            -0.007733354344964027,
+            0.004868561401963234,
+            0.03361568972468376,
+            0.023156417533755302,
+            0.01982341706752777,
+            -0.00039802573155611753,
+            0.031552404165267944,
+            0.0011506787268444896,
+            -0.016744358465075493,
+            0.010705280117690563,
+            -0.0036087664775550365,
+            0.004928078968077898,
+            -0.016315830871462822,
+            -0.04532880708575249,
+            0.004880465101450682,
+            0.002618785947561264,
+            0.009999001398682594,
+            -0.027457574382424355,
+            0.01891874521970749,
+            0.0006244415999390185,
+            -0.0024660234339535236,
+            -0.014958823099732399,
+            0.008126172237098217,
+            -0.010387850925326347,
+            0.00818965770304203,
+            -0.004559068474918604,
+            -0.01463345903903246,
+            0.021474044770002365,
+            0.00630492577329278,
+            -0.004531293176114559,
+            0.011340136639773846,
+            0.022537430748343468,
+            0.01847434602677822,
+            0.012078158557415009,
+            0.010467207990586758,
+            0.04247194901108742,
+            0.011784536764025688,
+            0.0025632360484451056,
+            0.006626322399824858,
+            -0.00334688788279891,
+            -0.0032774503342807293,
+            -0.004908239934593439,
+            0.000227779833949171,
+            -0.004071021918207407,
+            -0.018299760296940804,
+            0.016855459660291672,
+            0.010118036530911922,
+            -0.010729086585342884,
+            0.01714114472270012,
+            -0.01619679480791092,
+            0.012030544690787792,
+            0.03485365957021713,
+            0.00010012640996137634,
+            0.017696645110845566,
+            -0.01901397295296192,
+            -0.009760930202901363,
+            -0.0020573341753333807,
+            -0.011546465568244457,
+            0.0036206701770424843,
+            -0.015823816880583763,
+            -0.007455604616552591,
+            -0.008213465102016926,
+            0.00770557951182127,
+            0.01595078781247139,
+            0.015347673557698727,
+            0.013530394062399864,
+            0.01338755153119564,
+            0.0056065828539431095,
+            -0.009681572206318378,
+            -0.001512745744548738,
+            -0.015379415825009346,
+            -0.012522558681666851,
+            -0.01606188714504242,
+            0.004943950567394495,
+            -0.0025394288823008537,
+            -0.017966458573937416,
+            0.0269496887922287,
+            -0.013054251670837402,
+            0.005487547256052494,
+            0.009959322400391102,
+            0.01876003108918667,
+            -0.0005031739710830152,
+            0.00045853553456254303,
+            -0.005035211332142353,
+            0.0008818563655950129,
+            0.005150279030203819,
+            -0.001076281419955194,
+            0.027394089847803116,
+            -0.004225768148899078,
+            0.004277350381016731,
+            -0.0017498251982033253,
+            -0.014974694699048996,
+            -0.0007028068066574633,
+            0.00840392243117094,
+            -0.020870931446552277,
+            -0.02263266034424305,
+            -0.018283888697624207,
+            -0.005253443494439125,
+            -0.011157616041600704,
+            0.0067334542982280254,
+            -0.011300458572804928,
+            -0.008880065754055977,
+            0.023838888853788376,
+            0.00661441870033741,
+            0.014744559302926064,
+            0.0058446540497243404,
+            -0.0009671652805991471,
+            0.017712516710162163,
+            -0.02306118793785572,
+            -0.014379516243934631,
+            0.0024263449013233185,
+            -0.010744958184659481,
+            0.032568175345659256,
+            0.00097311707213521,
+            0.01301457267254591,
+            -0.010983030311763287,
+            0.010435465723276138,
+            -0.014117637649178505,
+            0.02418806031346321,
+            0.012705080211162567,
+            0.00584862194955349,
+            0.004186089616268873,
+            -0.003910323604941368,
+            -0.011308394372463226,
+            0.03821840509772301,
+            0.0002916375233326107,
+            -0.007225468754768372,
+            0.02004561759531498,
+            0.00832456536591053,
+            0.008721350692212582,
+            -0.008657865226268768,
+            0.012022608891129494,
+            0.016665002331137657,
+            0.0002353435556869954,
+            0.006741390097886324,
+            -0.01826801709830761,
+            -0.0040452308021485806,
+            0.023204030469059944,
+            0.025092732161283493,
+            0.011467108502984047,
+            -0.003124687820672989,
+            -0.002600930631160736,
+            0.012720951810479164,
+            -0.0017408975400030613,
+            0.019410759210586548,
+            0.009300658479332924,
+            0.03863106295466423,
+            0.019601216539740562,
+            -0.014046216383576393,
+            -0.0013530394062399864,
+            0.005110600497573614,
+            0.05183609202504158,
+            -0.010657665319740772,
+            -0.007161982823163271,
+            -0.008269015699625015,
+            -0.014752495102584362,
+            0.015157216228544712,
+            -0.010173587128520012,
+            0.008082525804638863,
+            -0.008657865226268768,
+            0.010863994248211384,
+            0.017823616042733192,
+            0.007892068475484848,
+            -0.009792672470211983,
+            -0.007376247551292181,
+            -0.011117937043309212,
+            0.012562237679958344,
+            0.001430412638001144,
+            -0.00338855036534369,
+            0.011855958960950375,
+            0.01026087999343872,
+            -0.016014274209737778,
+            -0.007677804678678513,
+            0.017855359241366386,
+            -0.022346973419189453,
+            0.0013738706475123763,
+            0.007284986786544323,
+            0.0036008309107273817,
+            -0.00108818500302732,
+            -6.959250185900601e-06,
+            0.00910226535052061,
+            0.028600318357348442,
+            0.0016545966500416398,
+            0.02942563220858574,
+            0.01799820177257061,
+            0.00014445481065195054,
+            -0.024965759366750717,
+            0.005789104383438826,
+            0.01739508844912052,
+            0.004412257578223944,
+            0.024695945903658867,
+            -0.005420093424618244,
+            -0.012459073215723038,
+            -0.0191250741481781,
+            -0.0088880006223917,
+            0.012094030156731606,
+            -0.018394988030195236,
+            0.013197094202041626,
+            -0.005189957562834024,
+            0.007915875874459743,
+            0.005777200683951378,
+            0.015093730762600899,
+            -0.010530694387853146,
+            0.007685740012675524,
+            0.0021902574226260185,
+            -0.011086193844676018,
+            -0.011784536764025688,
+            -0.00032883620588108897,
+            -0.007384182885289192,
+            0.003209996735677123,
+            -0.0035849593114107847,
+            0.011649630032479763,
+            0.023727789521217346,
+            -0.004773332737386227,
+            -0.006947718560695648,
+            0.009991065599024296,
+            0.004654297139495611,
+            -0.011562337167561054,
+            0.02193431742489338,
+            0.009705379605293274,
+            0.004110700450837612,
+            0.007856357842683792,
+            -0.01042752992361784,
+            -0.016069823876023293,
+            -0.023045316338539124,
+            -0.007181822322309017,
+            -0.0231722891330719,
+            0.014974694699048996,
+            -0.006658065132796764,
+            -0.015800008550286293,
+            0.01858544535934925,
+            0.002934230724349618,
+            0.0018480296712368727,
+            -0.00848327949643135,
+            -0.026330703869462013,
+            -0.022331101819872856,
+            0.01026087999343872,
+            -0.00654696486890316,
+            0.005201861262321472,
+            0.02706078812479973,
+            0.030155718326568604,
+            -0.007019140291959047,
+            0.004261479247361422,
+            0.0034500521142035723,
+            0.003930163104087114,
+            -0.0002421633107587695,
+            -0.024981630966067314,
+            -0.0064041223376989365,
+            0.007947619073092937,
+            0.006789004430174828,
+            -0.00544390082359314,
+            -0.00926891528069973,
+            0.011943251825869083,
+            -0.019299659878015518,
+            -0.005920043680816889,
+            -0.009173686616122723,
+            -0.002819162793457508,
+            -0.00032015651231631637,
+            0.017982330173254013,
+            0.015268316492438316,
+            -0.005551032721996307,
+            -0.0025870432145893574,
+            -0.023775402456521988,
+            -0.012181323021650314,
+            0.01939488761126995,
+            0.004884432535618544,
+            0.006574740167707205,
+            -0.01471281610429287,
+            -0.0012141644256189466,
+            -0.0033389520831406116,
+            -0.012498751282691956,
+            0.015173087827861309,
+            0.06513635069131851,
+            -0.016006337478756905,
+            0.02483878843486309,
+            -0.005007436498999596,
+            0.004701911471784115,
+            -0.003416325431317091,
+            -0.021426431834697723,
+            -0.011951187625527382,
+            -0.0060271755792200565,
+            0.0017756163142621517,
+            -0.0188552588224411,
+            -0.0023965861182659864,
+            -0.0006715598865412176,
+            0.002325164619833231,
+            -0.007019140291959047,
+            0.0011953171342611313,
+            -0.02085505984723568,
+            0.016887202858924866,
+            0.006431897170841694,
+            -0.003862709505483508,
+            -0.006447768770158291,
+            -0.018823517486453056,
+            0.00022120805806480348,
+            -0.0016357493586838245,
+            -2.8115991881350055e-05,
+            0.0016784038161858916,
+            -0.025902174413204193,
+            -0.010657665319740772,
+            -0.00964189413934946,
+            -0.015069923363626003,
+            0.0002531989011913538,
+            -0.006380314938724041,
+            0.01463345903903246,
+            -0.0220930315554142,
+            -0.004872529301792383,
+            0.01822040230035782,
+            0.028362246230244637,
+            0.020728088915348053,
+            -0.011213165707886219,
+            0.001291537657380104,
+            -0.02160101756453514,
+            0.02533080242574215,
+            0.01880764588713646,
+            -0.017442701384425163,
+            0.005102664697915316,
+            0.008768965490162373,
+            0.009951386600732803,
+            -0.0019998003263026476,
+            0.030139846727252007,
+            -0.00038463418604806066,
+            -0.015427030622959137,
+            0.005090761464089155,
+            0.018934616819024086,
+            0.002618785947561264,
+            0.002612834330648184,
+            -0.006209697108715773,
+            0.021585145965218544,
+            -0.005582775454968214,
+            0.009633958339691162,
+            -0.044852662831544876,
+            -0.01815691590309143,
+            -0.005519289989024401,
+            -0.004213864915072918,
+            0.007812711410224438,
+            0.004011503886431456,
+            0.008673736825585365,
+            -0.018093431368470192,
+            -0.0021208198741078377,
+            0.025283189490437508,
+            -0.026965560391545296,
+            -0.02823527529835701,
+            -0.005225668661296368,
+            -0.02441026084125042,
+            -0.0007444692892022431,
+            0.0011546466266736388,
+            -0.007959522306919098,
+            -0.026473546400666237,
+            0.008284886367619038,
+            -0.006122404243797064,
+            -0.027679774910211563,
+            0.01768077351152897,
+            -0.005681972019374371,
+            0.000294117460725829,
+            -0.010300558060407639,
+            -0.003963889554142952,
+            -0.005356607958674431,
+            0.016569772735238075,
+            -0.01330819446593523,
+            0.008062686771154404,
+            0.0010266831377521157,
+            -0.0032357878517359495,
+            0.0016873314743861556,
+            0.016093630343675613,
+            0.002900503808632493,
+            0.000488046498503536,
+            -0.006039079278707504,
+            -0.003088977187871933,
+            -0.017871230840682983,
+            0.000921534956432879,
+            0.01891874521970749,
+            0.0036325736436992884,
+            0.02672748826444149,
+            -0.03469494730234146,
+            -0.01642693020403385,
+            -0.005725618451833725,
+            -0.01725224405527115,
+            0.011403623037040234,
+            -0.0036821719259023666,
+            -0.008499151095747948,
+            -0.0019075475865975022,
+            -0.02548951655626297,
+            0.03304431959986687,
+            -0.01939488761126995,
+            -0.00506695406511426,
+            0.004428129177540541,
+            -0.019839288666844368,
+            -0.00044935988262295723,
+            0.008157915435731411,
+            0.01998213119804859,
+            0.010308493860065937,
+            -0.015617487952113152,
+            0.009070522151887417,
+            -0.008538829162716866,
+            -0.008054750971496105,
+            0.0041543468832969666,
+            0.008126172237098217,
+            -0.009681572206318378,
+            -0.0029362146742641926,
+            0.00816585123538971,
+            -0.0030731058213859797,
+            -0.014284287579357624,
+            -0.00878483708947897,
+            0.027219504117965698,
+            -0.008515022695064545,
+            0.00575339375063777,
+            -0.0026386252138763666,
+            -0.001564327860251069,
+            -0.0020394788589328527,
+            0.00064130499958992,
+            0.0062612793408334255,
+            -0.006253343541175127,
+            0.05850209295749664,
+            -0.0003680188092403114,
+            -0.00022405995696317405,
+            -0.01869654469192028,
+            -0.011855958960950375,
+            0.005035211332142353,
+            0.006880265194922686,
+            -0.02220413088798523,
+            -0.006959622260183096,
+            0.025092732161283493,
+            -0.0020533662755042315,
+            0.010808444581925869,
+            0.005408189725130796,
+            -0.001736929640173912,
+            0.0328538604080677,
+            0.006852489896118641,
+            0.0280130747705698,
+            -0.010760829783976078,
+            0.004015471786260605,
+            0.0001645420998102054,
+            0.00755876861512661,
+            0.0069516864605247974,
+            -0.009118136949837208,
+            -0.005003468599170446,
+            0.0040611024014651775,
+            0.014300159178674221,
+            -0.0004451440181583166,
+            -0.008292822167277336,
+            0.0030135877896100283,
+            -0.012109901756048203,
+            -0.0017528011230751872,
+            -0.0008218425209634006,
+            -0.0140065373852849,
+            -0.00576529698446393,
+            0.009173686616122723,
+            0.015458772890269756,
+            0.007324665319174528,
+            0.0010683456202968955,
+            0.004400353878736496,
+            0.004999500699341297,
+            0.005269315093755722,
+            -0.012094030156731606,
+            0.009570472873747349,
+            -0.02667987532913685,
+            -0.009062586352229118,
+            0.01214957982301712,
+            -0.0011348072439432144,
+            -0.00886419415473938,
+            0.023505588993430138,
+            0.0046265218406915665,
+            -0.012332101352512836,
+            -0.011022708378732204,
+            0.006796940229833126,
+            -0.0023172288201749325,
+            0.008991165086627007,
+            -0.007860326208174229,
+            0.0027636128943413496,
+            -0.0007955554756335914,
+            0.01622060127556324,
+            0.01653803139925003,
+            -0.004253543447703123,
+            0.012117837555706501,
+            -0.0004079453647136688
+          ],
+          "title": "Balancing Expert Utilization",
+          "keyphrases": [
+            "gating network",
+            "expert utilization",
+            "soft constraint",
+            "importance of an expert",
+            "balanced loads"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "e7b4a647-c2c4-4a34-8f13-6879240908f3",
+      "type": "child",
+      "source": {
+        "id": "14339b1f-53f8-479b-9da7-9430b174b9ee",
+        "properties": {
+          "page_content": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer\n\nNoam Shazeer1, Azalia Mirhoseini∗†1, Krzysztof Maziarz∗2, Andy Davis1, Quoc Le1, Geoffrey Hinton1and Jeff Dean1 1Google Brain, {noam,azalia,andydavis,qvl,geoffhinton,jeff}@google.com 2Jagiellonian University, Cracow, krzysztof.maziarz@student.uj.edu.pl\n\nAbstract\n\nThe capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost.\n\n1 Introduction And Related Work 1.1 Conditional Computation\n\nExploiting scale in both training data and model size has been central to the success of deep learning. When datasets are sufficiently large, increasing the capacity (number of parameters) of neural networks can give much better prediction accuracy. This has been shown in domains such as text (Sutskever et al., 2014; Bahdanau et al., 2014; Jozefowicz et al., 2016; Wu et al., 2016), images (Krizhevsky et al., 2012; Le et al., 2012), and audio (Hinton et al., 2012; Amodei et al., 2015). For typical deep learning models, where the entire model is activated for every example, this leads to a roughly quadratic blow-up in training costs, as both the model size and the number of training examples increase. Unfortunately, the advances in computing power and distributed computation fall short of meeting such demand.\n\nVarious forms of conditional computation have been proposed as a way to increase model capacity without a proportional increase in computational costs (Davis & Arel, 2013; Bengio et al., 2013; Eigen et al., 2013; Ludovic Denoyer, 2014; Cho & Bengio, 2014; Bengio et al., 2015; Almahairi et al., 2015). In these schemes, large parts of a network are active or inactive on a per-example basis. The gating decisions may be binary or sparse and continuous, stochastic or deterministic. Various forms of reinforcement learning and back-propagation are proposed for trarining the gating decisions.\n\nWhile these ideas are promising in theory, no work to date has yet demonstrated massive improvements in model capacity, training time, or model quality. We blame this on a combination of the following challenges:\n\nModern computing devices, especially GPUs, are much faster at arithmetic than at branching. Most of the works above recognize this and propose turning on/off large chunks of the network with each gating decision.\n\nLarge batch sizes are critical for performance, as they amortize the costs of parameter transfers and updates. Conditional computation reduces the batch sizes for the conditionally active chunks of the network.\n\nNetwork bandwidth can be a bottleneck. A cluster of GPUs may have computational power thousands of times greater than the aggregate inter-device network bandwidth. To be computationally efficient, the relative computational versus network demands of an algorithm must exceed this ratio. Embedding layers, which can be seen as a form of conditional computation, are handicapped by this very problem. Since the embeddings generally need to be sent across the network, the number of (example, parameter) interactions is limited by network bandwidth instead of computational capacity.\n\nDepending on the scheme, loss terms may be necessary to achieve the desired level of sparsity per-chunk and/or per example. Bengio et al. (2015) use three such terms. These issues can affect both model quality and load-balancing.\n\nModel capacity is most critical for very large data sets. The existing literature on conditional computation deals with relatively small image recognition data sets consisting of up to 600,000 images. It is hard to imagine that the labels of these images provide a sufficient signal to adequately train a model with millions, let alone billions of parameters. In this work, we for the first time address all of the above challenges and finally realize the promise of conditional computation. We obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets.\n\n1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1). All parts of the network are trained jointly by back-propagation.\n\nWhile the introduced technique is generic, in this paper we focus on language modeling and machine translation tasks, which are known to benefit from very large models. In particular, we apply a MoE convolutionally between stacked LSTM layers (Hochreiter & Schmidhuber, 1997), as in Figure 1.\n\nThe MoE is called once for each position in the text, selecting a potentially different combination of experts at each position. The different experts tend to become highly specialized based on syntax and semantics (see Appendix E Table 9). On both language modeling and machine translation benchmarks, we improve on best published results at a fraction of the computational cost.\n\n1.3 Related Work On Mixtures Of Experts\n\nSince its introduction more than two decades ago (Jacobs et al., 1991; Jordan & Jacobs, 1994), the mixture-of-experts approach has been the subject of much research. Different types of expert architectures hae been proposed such as SVMs (Collobert et al., 2002), Gaussian Processes (Tresp, 2001; Theis & Bethge, 2015; Deisenroth & Ng, 2015), Dirichlet Processes (Shahbaba & Neal, 2009), and deep networks. Other work has focused on different expert configurations such as a hierarchical structure (Yao et al., 2009), infinite numbers of experts (Rasmussen & Ghahramani, 2002), and adding experts sequentially (Aljundi et al., 2016). Garmash & Monz (2016) suggest an ensemble model in the format of mixture of experts for machine translation. The gating network is trained on a pre-trained ensemble NMT model. The works above concern top-level mixtures of experts. The mixture of experts is the whole model. Eigen et al. (2013) introduce the idea of using multiple MoEs with their own gating networks as parts of a deep model. It is intuitive that the latter approach is more powerful, since complex problems may contain many sub-problems each requiring different experts. They also allude in their conclusion to the potential to introduce sparsity, turning MoEs into a vehicle for computational computation.\n\nOur work builds on this use of MoEs as a general purpose neural network component. While Eigen et al. (2013) uses two stacked MoEs allowing for two sets of gating decisions, our convolutional application of the MoE allows for different gating decisions at each position in the text. We also realize sparse gating and demonstrate its use as a practical way to massively increase model capacity.\n\n2 The Structure Of The Mixture-Of-Experts Layer\n\nThe Mixture-of-Experts (MoE) layer consists of a set of n \"expert networks\" E1, · · · , En, and a \"gating network\" G whose output is a sparse n-dimensional vector. Figure 1 shows an overview of the MoE module. The experts are themselves neural networks, each with their own parameters.\n\nAlthough in principle we only require that the experts accept the same sized inputs and produce the same-sized outputs, in our initial investigations in this paper, we restrict ourselves to the case where the models are feed-forward networks with identical architectures, but with separate parameters.\n\nLet us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows:\n\n$$y=\\sum_{i=1}^{n}G(x){i}E{i}(x)$$ $$(\\mathbf{l})$$ G(x)iEi(x) (1) We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example. If the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network. In the following we focus on ordinary MoEs. We provide more details on hierarchical MoEs in Appendix B. Our implementation is related to other models of conditional computation. A MoE whose experts are simple weight matrices is similar to the parameterized weight matrix proposed in (Cho & Bengio, 2014). A MoE whose experts have one hidden layer is similar to the block-wise dropout described in (Bengio et al., 2015), where the dropped-out layer is sandwiched between fully-activated layers.\n\n2.1 Gating Network\n\nSoftmax Gating: A simple choice of non-sparse gating function (Jordan & Jacobs, 1994) is to multiply the input by a trainable weight matrix Wg and then apply the Sof tmax function.\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ $\\eqref{eq:walpha}$. Gσ(x) = Sof tmax(x · Wg) (2) Noisy Top-K Gating: We add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to −∞ (which causes the corresponding gate values to equal 0). The sparsity serves to save computation, as described above. While this form of sparsity creates some theoretically scary discontinuities in the output of gating function, we have not yet observed this to be a problem in practice. The noise term helps with load balancing, as will be discussed in Appendix A. The amount of noise per component is controlled by a second trainable weight matrix Wnoise.\n\n$$G(x)=S o f t m a x(K e e p T o p K(H(x),k))$$ $$(4)$$ $$(S)$$ G(x) = Sof tmax(KeepT opK(H(x), k)) (3) $$H(x){i}=(x\\cdot W{g}){i}+S t a n d a r d N o r m a l()\\cdot S o f t p l u s((x\\cdot W{n o i s e}){i})$$ $KeepTopK(v,k){i}=\\begin{cases}v_{i}&\\text{if}v_{i}\\text{is in the top}k\\text{elements of}v.\\ -\\infty&\\text{otherwise.}\\end{cases}$ Training the Gating Network We train the gating network by simple back-propagation, along with the rest of the model. If we choose k > 1, the gate values for the top k experts have nonzero derivatives with respect to the weights of the gating network. This type of occasionally-sensitive behavior is described in (Bengio et al., 2013) with respect to noisy rectifiers. Gradients also backpropagate through the gating network to its inputs. Our method differs here from (Bengio et al., 2015) who use boolean gates and a REINFORCE-style approach to train the gating network.\n\n3 Addressing Performance Challenges 3.1 The Shrinking Batch Problem\n\nOn modern CPUs and GPUs, large batch sizes are necessary for computational efficiency, so as to amortize the overhead of parameter loads and updates. If the gating network chooses k out of n experts for each example, then for a batch of b examples, each expert receives a much smaller batch of approximately kb n b examples. This causes a naive MoE implementation to become very inefficient as the number of experts increases. The solution to this shrinking batch problem is to make the original batch size as large as possible. However, batch size tends to be limited by the memory necessary to store activations between the forwards and backwards passes. We propose the following techniques for increasing the batch size: Mixing Data Parallelism and Model Parallelism: In a conventional distributed training setting, multiple copies of the model on different devices asynchronously process distinct batches of data, and parameters are synchronized through a set of parameter servers. In our technique, these different batches run synchronously so that they can be combined for the MoE layer. We distribute the standard layers of the model and the gating network according to conventional data-parallel schemes, but keep only one shared copy of each expert. Each expert in the MoE layer receives a combined batch consisting of the relevant examples from all of the data-parallel input batches. The same set of devices function as data-parallel replicas (for the standard layers and the gating networks) and as model-parallel shards (each hosting a subset of the experts). If the model is distributed over d devices, and each device processes a batch of size b, each expert receives a batch of approximately kbd nexamples. Thus, we achieve a factor of d improvement in expert batch size.\n\nIn the case of a hierarchical MoE (Section B), the primary gating network employs data parallelism, and the secondary MoEs employ model parallelism. Each secondary MoE resides on one device.\n\nThis technique allows us to increase the number of experts (and hence the number of parameters) by proportionally increasing the number of devices in the training cluster. The total batch size increases, keeping the batch size per expert constant. The memory and bandwidth requirements per device also remain constant, as do the step times, as does the amount of time necessary to process a number of training examples equal to the number of parameters in the model. It is our goal to train a trillionparameter model on a trillion-word corpus. We have not scaled our systems this far as of the writing of this paper, but it should be possible by adding more hardware.\n\nTaking Advantage of Convolutionality: In our language models, we apply the same MoE to each time step of the previous layer. If we wait for the previous layer to finish, we can apply the MoE to all the time steps together as one big batch. Doing so increases the size of the input batch to the MoE layer by a factor of the number of unrolled time steps.\n\nIncreasing Batch Size for a Recurrent MoE: We suspect that even more powerful models may involve applying a MoE recurrently. For example, the weight matrices of a LSTM or other RNN could be replaced by a MoE. Sadly, such models break the convolutional trick from the last paragraph, since the input to the MoE at one timestep depends on the output of the MoE at the previous timestep. Gruslys et al. (2016) describe a technique for drastically reducing the number of stored activations in an unrolled RNN, at the cost of recomputing forward activations. This would allow for a large increase in batch size.\n\n3.2 Network Bandwidth\n\nAnother major performance concern in distributed computing is network bandwidth. Since the experts are stationary (see above) and the number of gating parameters is small, most of the communication involves sending the inputs and outputs of the experts across the network. To maintain computational efficiency, the ratio of an expert's computation to the size of its input and output must exceed the ratio of computational to network capacity of the computing device. For GPUs, this may be thousands to one. In our experiments, we use experts with one hidden layer containing thousands of RELU-activated units. Since the weight matrices in the expert have sizes input_size×hidden_size and hidden_size × output_size, the ratio of computation to input and output is equal to the size of the hidden layer. Conveniently, we can increase computational efficiency simply by using a larger hidden layer, or more hidden layers.\n\n4 Balancing Expert Utilization\n\nWe have observed that the gating network tends to converge to a state where it always produces large weights for the same few experts. This imbalance is self-reinforcing, as the favored experts are trained more rapidly and thus are selected even more by the gating network. Eigen et al. (2013) describe the same phenomenon, and use a hard constraint at the beginning of training to avoid this local minimum. Bengio et al. (2015) include a soft constraint on the batch-wise average of each gate.1 We take a soft constraint approach. We define the importance of an expert relative to a batch of training examples to be the batchwise sum of the gate values for that expert. We define an additional loss Limportance, which is added to the overall loss function for the model. This loss is equal to the square of the coefficient of variation of the set of importance values, multiplied by a hand-tuned scaling factor wimportance. This additional loss encourages all experts to have equal importance.\n\n$$I m p o r t a n c e(X)=\\sum_{x\\in X}G(x)$$ $$L_{importance}(X)=w_{importance}\\cdot CV(Importance(X))^{2}\\tag{7}$$ $$(6)$$\n\nWhile this loss function can ensure equal importance, experts may still receive very different numbers of examples. For example, one expert may receive a few examples with large weights, and another may receive many examples with small weights. This can cause memory and performance problems on distributed hardware. To solve this problem, we introduce a second loss function, Lload , which ensures balanced loads. Appendix A contains the definition of this function, along with experimental results.\n\n5 Experiments 5.1 1 Billion Word Language Modeling Benchmark\n\nDataset: This dataset, introduced by (Chelba et al., 2013) consists of shuffled unique sentences from news articles, totaling approximately 829 million words, with a vocabulary of 793,471 words.\n\nPrevious State-of-the-Art: The best previously published results (Jozefowicz et al., 2016) use models consisting of one or more stacked Long Short-Term Memory (LSTM) layers (Hochreiter & Schmidhuber, 1997; Gers et al., 2000). The number of parameters in the LSTM layers of these models vary from 2 million to 151 million. Quality increases greatly with parameter count, as do computational costs. Results for these models form the top line of Figure 2-right. MoE Models: Our models consist of two stacked LSTM layers with a MoE layer between them (see Figure 1). We vary the sizes of the layers and the number of experts. For full details on model architecture, training regimen, additional baselines and results, see Appendix C.\n\nLow Computation, Varied Capacity: To investigate the effects of adding capacity, we trained\n\na series of MoE models all with roughly equal computational costs: about 8 million multiply-andadds per training example per timestep in the forwards pass, excluding the softmax layer. We call this metric (ops/timestep). We trained models with flat MoEs containing 4, 32, and 256 experts, and models with hierarchical MoEs containing 256, 1024, and 4096 experts. Each expert had about 1 million parameters. For all the MoE layers, 4 experts were active per input.\n\nThe results of these models are shown in Figure 2-left. The model with 4 always-active experts performed (unsurprisingly) similarly to the computationally-matched baseline models, while the largest of the models (4096 experts) achieved an impressive 24% lower perplexity on the test set. Figure 2: Model comparison on 1-Billion-Word Language-Modeling Benchmark. On the left, we plot test perplexity as a function of model capacity for models with similar computational budgets of approximately 8-million-ops-per-timestep. On the right, we plot test perplexity as a function of computational budget. The top line represents the LSTM models from (Jozefowicz et al., 2016).\n\nThe bottom line represents 4-billion parameter MoE models with different computational budgets.\n\nVaried Computation, High Capacity: In addition to the largest model from the previous section, we trained two more MoE models with similarly high capacity (4 billion parameters), but higher computation budgets. These models had larger LSTMs, and fewer but larger and experts. Details\n\nTest Test #Parameters ops/timestep Training TFLOPS Perplexity Perplexity excluding embedding Time /GPU 10 epochs 100 epochs and softmax layers 10 epochs Best Published Results 34.7 30.6 151 million 151 million 59 hours, 32 k40s 1.09 Low-Budget MoE Model 34.1 4303 million 8.9 million 15 hours, 16 k40s 0.74 Medium-Budget MoE Model 31.3 4313 million 33.8 million 17 hours, 32 k40s 1.22 High-Budget MoE Model 28.0 4371 million 142.7 million 47 hours, 32 k40s 1.56\n\nTable 1: Summary of high-capacity MoE-augmented models with varying computational budgets, vs. best previously published results (Jozefowicz et al., 2016). Details in Appendix C. can be found in Appendix C.2. Results of these three models form the bottom line of Figure 2-right.\n\nTable 1 compares the results of these models to the best previously-published result on this dataset .\n\nEven the fastest of these models beats the best published result (when controlling for the number of training epochs), despite requiring only 6% of the computation.\n\nComputational Efficiency: We trained our models using TensorFlow (Abadi et al., 2016) on clusters containing 16-32 Tesla K40 GPUs. For each of our models, we determine computational efficiency in TFLOPS/GPU by dividing the number of floating point operations required to process one training batch by the observed step time and the number of GPUs in the cluster. The operation counts used here are higher than the ones we report in our ops/timestep numbers in that we include the backwards pass, we include the importance-sampling-based training of the softmax layer, and we count a multiply-and-add as two separate operations. For all of our MoE models, the floating point operations involved in the experts represent between 37% and 46% of the total.\n\nFor our baseline models wtih no MoE, observed computational efficiency ranged from 1.07-1.29 TFLOPS/GPU. For our low-computation MoE models, computation efficiency ranged from 0.740.90 TFLOPS/GPU, except for the 4-expert model which did not make full use of the available parallelism. Our highest-computation MoE model was more efficient at 1.56 TFLOPS/GPU, likely due to the larger matrices. These numbers represent a significant fraction of the theoretical maximum of 4.29 TFLOPS/GPU claimed by NVIDIA. Detailed results are in Appendix C, Table 7.\n\n5.2 100 BILLION WORD GOOGLE NEWS CORPUS\n\nOn the 1-billion-word corpus, adding additional capacity seems to produce diminishing returns as the number of parameters in the MoE layer exceeds 1 billion, as can be seen in Figure 2-left. We hypothesized that for a larger training set, even higher capacities would produce significant quality improvements. We constructed a similar training set consisting of shuffled unique sentences from Google's internal news corpus, totalling roughly 100 billion words. Similarly to the previous section, we tested a series of models with similar computational costs of about 8 million ops/timestep. In addition to a baseline LSTM model, we trained models augmented with MoE layers containing 32, 256, 1024, 4096, 16384, 65536, and 131072 experts. This corresponds to up to 137 billion parameters in the MoE layer. Details on architecture, training, and results are given in Appendix D.\n\nResults: Figure 3 shows test perplexity as a function of capacity after training on 10 billion words (top line) and 100 billion words (bottom line). When training over the full 100 billion words, test perplexity improves significantly up to 65536 experts (68 billion parameters), dropping 39% lower than the computationally matched baseline, but degrades at 131072 experts, possibly a result of too much sparsity. The widening gap between the two lines demonstrates (unsurprisingly) that increased model capacity helps more on larger training sets. Even at 65536 experts (99.994% layer sparsity), computational efficiency for the model stays at a respectable 0.72 TFLOPS/GPU.\n\n5.3 Machine Translation (Single Language Pair)\n\nModel Architecture: Our model was a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decreased the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We inserted MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). Each MoE layer contained up to 2048 experts each with about two million parameters, adding a total of about 8 billion parameters to the models. Further details on model architecture, testing procedure and results can be found in Appendix E.\n\nDatasets: We benchmarked our method on the WMT'14 En→Fr and En→De corpora, whose training sets have 36M sentence pairs and 5M sentence pairs, respectively. The experimental protocols were also similar to those in (Wu et al., 2016): newstest2014 was used as the test set to compare against previous work (Luong et al., 2015a; Zhou et al., 2016; Wu et al., 2016), while the combination of newstest2012 and newstest2013 was used as the development set. We also tested the same model on a Google's Production English to French data.\n\nTable 2: Results on WMT'14 En→ Fr newstest2014 (bold values represent best results). Model Test Test ops/timenstep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.69 40.35 85M 8.7B 3 days/64 k40s MoE with 2048 Experts (longer training) 2.63 40.56 85M 8.7B 6 days/64 k40s GNMT (Wu et al., 2016) 2.79 39.22 214M 278M 6 days/96 k80s GNMT+RL (Wu et al., 2016) 2.96 39.92 214M 278M 6 days/96 k80s PBMT (Durrani et al., 2014) 37.0 LSTM (6-layer) (Luong et al., 2015b) 31.5 LSTM (6-layer+PosUnk) (Luong et al., 2015b) 33.1 DeepAtt (Zhou et al., 2016) 37.7 DeepAtt+PosUnk (Zhou et al., 2016) 39.2\n\nTable 3: Results on WMT'14 En → De newstest2014 (bold values represent best results). Model Test Test ops/timestep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 4.64 26.03 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 5.25 24.91 214M 278M 1 day/96 k80s GNMT +RL (Wu et al., 2016) 8.08 24.66 214M 278M 1 day/96 k80s PBMT (Durrani et al., 2014) 20.7 DeepAtt (Zhou et al., 2016) 20.6\n\nTable 4: Results on the Google Production En→ Fr dataset (bold values represent best results). Model Eval Eval Test Test ops/timestep Total Training Perplexity BLEU Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.60 37.27 2.69 36.57 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 2.78 35.80 2.87 35.56 214M 278M 6 days/96 k80s\n\nResults: Tables 2, 3, and 4 show the results of our largest models, compared with published results. Our approach achieved BLEU scores of 40.56 and 26.03 on the WMT'14 En→Fr and En→De benchmarks. As our models did not use RL refinement, these results constitute significant gains of 1.34 and 1.12 BLEU score on top of the strong baselines in (Wu et al., 2016). The perplexity scores are also better.2 On the Google Production dataset, our model achieved 1.01 higher test BLEU score even after training for only one sixth of the time.\n\n5.4 Multilingual Machine Translation\n\nDataset: (Johnson et al., 2016) train a single GNMT (Wu et al., 2016) model on a very large combined dataset of twelve language pairs. Results are somewhat worse than those for 12 separately trained single-pair GNMT models. This is not surprising, given that the twelve models have 12 times the capacity and twelve times the aggregate training of the one model. We repeat this experiment with a single MoE-augmented model. See Appendix E for details on model architecture.\n\nWe train our model on the same dataset as (Johnson et al., 2016) and process the same number of training examples (about 3 billion sentence pairs). Our training time was shorter due to the lower computational budget of our model.\n\nResults: Results for the single-pair GNMT models, the multilingual GNMT model and the multilingual MoE model are given in Table 5. The MoE model achieves 19% lower perplexity on the dev set than the multilingual GNMT model. On BLEU score, the MoE model significantly beats the multilingual GNMT model on 11 of the 12 language pairs (by as much as 5.84 points), and even beats the monolingual GNMT models on 8 of 12 language pairs. The poor performance on English → Korean seems to be a result of severe overtraining, as for the rarer language pairs a small number of real examples were highly oversampled in the training corpus.\n\nTable 5: Multilingual Machine Translation (bold values represent best results). GNMT-Mono GNMT-Multi MoE-Multi MoE-Multi vs. GNMT-Multi Parameters 278M / model 278M 8.7B ops/timestep 212M 212M 102M training time, hardware various 21 days, 96 k20s 12 days, 64 k40s Perplexity (dev) 4.14 3.35 -19% French → English Test BLEU 36.47 34.40 37.46 +3.06 German → English Test BLEU 31.77 31.17 34.80 +3.63 Japanese → English Test BLEU 23.41 21.62 25.91 +4.29 Korean → English Test BLEU 25.42 22.87 28.71 +5.84 Portuguese → English Test BLEU 44.40 42.53 46.13 +3.60 Spanish → English Test BLEU 38.00 36.04 39.39 +3.35 English → French Test BLEU 35.37 34.00 36.59 +2.59 English → German Test BLEU 26.43 23.15 24.53 +1.38 English → Japanese Test BLEU 23.66 21.10 22.78 +1.68 English → Korean Test BLEU 19.75 18.41 16.62 -1.79 English → Portuguese Test BLEU 38.40 37.35 37.90 +0.55 English → Spanish Test BLEU 34.50 34.25 36.21 +1.96\n\n6 Conclusion\n\nThis work is the first to demonstrate major wins from conditional computation in deep networks.\n\nWe carefully identified the design considerations and challenges of conditional computing and addressed them with a combination of algorithmic and engineering solutions. While we focused on text, conditional computation may help in other domains as well, provided sufficiently large training sets. We look forward to seeing many novel implementations and applications of conditional computation in the years to come.\n\nAcknowledgments\n\nWe would like to thank all of the members of the Google Brain and Google Translate teams who helped us with this project, in particular Zhifeng Chen, Yonghui Wu, and Melvin Johnson. Thanks also to our anonymous ICLR reviewers for the helpful suggestions on making this paper better.\n\n2Reported perplexities relative to the tokenization used by both our models and GNMT.\n\nReferences\n\nMartín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Gordon Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda B. Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. CoRR, abs/1603.04467, 2016. URL http://arxiv.org/abs/1603.04467.\n\nRahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. CoRR, abs/1611.06194, 2016. URL http://arxiv.org/abs/1611.\n\n06194.\n\nA. Almahairi, N. Ballas, T. Cooijmans, Y. Zheng, H. Larochelle, and A. Courville. Dynamic Capacity Networks. ArXiv e-prints, November 2015.\n\nDario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan, Christopher Fougner, Tony Han, Awni Y. Hannun, Billy Jun, Patrick LeGresley, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595, 2015. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.\n\nEmmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297, 2015.\n\nYoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.\n\nCiprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling.\n\narXiv preprint arXiv:1312.3005, 2013.\n\nK. Cho and Y. Bengio. Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning. ArXiv e-prints, June 2014. Ronan Collobert, Samy Bengio, and Yoshua Bengio. A parallel mixture of SVMs for very large scale problems. Neural Computing, 2002.\n\nAndrew Davis and Itamar Arel. Low-rank approximations for conditional feedforward computation in deep neural networks. arXiv preprint arXiv:1312.4461, 2013. Marc Peter Deisenroth and Jun Wei Ng. Distributed Gaussian processes. In ICML, 2015.\n\nJohn Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization, 2010.\n\nNadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. Edinburgh's phrase-based machine translation systems for wmt-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation, 2014.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013. Ekaterina Garmash and Christof Monz. Ensemble learning for multi-source neural machine translation. In staff.science.uva.nl/c.monz, 2016.\n\nFelix A. Gers, Jürgen A. Schmidhuber, and Fred A. Cummins. Learning to forget: Continual prediction with lstm. Neural Computation, 2000.\n\nAudrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. Memory-efficient backpropagation through time. CoRR, abs/1606.03401, 2016. URL http://arxiv.org/ abs/1606.03401. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2015.\n\nGeoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 2012. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.\n\nSergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.\n\nRobert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computing, 1991.\n\nMelvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation.\n\nCoRR, abs/1611.04558, 2016. URL http://arxiv.org/abs/1611.04558.\n\nMichael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm.\n\nNeural Computing, 1994. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.\n\nDiederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.\n\nReinhard Kneser and Hermann. Ney. Improved backingoff for m-gram language modeling., 1995.\n\nAlex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.\n\nQuoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeffrey Dean, and Andrew Y. Ng. Building high-level features using large scale unsupervised learning. In ICML, 2012. Patrick Gallinari Ludovic Denoyer. Deep sequential neural network. arXiv preprint arXiv:1410.0510, 2014.\n\nMinh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attentionbased neural machine translation. EMNLP, 2015a.\n\nMinh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the rare word problem in neural machine translation. ACL, 2015b.\n\nCarl Edward Rasmussen and Zoubin Ghahramani. Infinite mixtures of Gaussian process experts.\n\nNIPS, 2002.\n\nHasim Sak, Andrew W Senior, and Françoise Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH, pp. 338–342, 2014.\n\nMike Schuster and Kaisuke Nakajima. Japanese and Korean voice search. ICASSP, 2012.\n\nBabak Shahbaba and Radford Neal. Nonlinear models using dirichlet process mixtures. JMLR, 2009.\n\nIlya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks.\n\nIn NIPS, 2014.\n\nLucas Theis and Matthias Bethge. Generative image modeling using spatial LSTMs. In NIPS, 2015. Volker Tresp. Mixtures of Gaussian Processes. In NIPS, 2001.\n\nYonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.\n\nBangpeng Yao, Dirk Walther, Diane Beck, and Li Fei-fei. Hierarchical mixture of classification experts uncovers interactions between brain regions. In NIPS. 2009.\n\nWojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization.\n\narXiv preprint arXiv:1409.2329, 2014.\n\nJie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with fast-forward connections for neural machine translation. arXiv preprint arXiv:1606.04199, 2016.\n\nAppendices A Load-Balancing Loss\n\nAs discussed in section 4, for load-balancing purposes, we want to define an additional loss function to encourage experts to receive roughly equal numbers of training examples. Unfortunately, the number of examples received by an expert is a discrete quantity, so it can not be used in backpropagation. Instead, we define a smooth estimator Load(X) of the number of examples assigned to each expert for a batch X of inputs. The smoothness allows us to back-propagate gradients through the estimator. This is the purpose of the noise term in the gating function. We define P(x, i) as the probability that G(x)iis nonzero, given a new random choice of noise on element i, but keeping the already-sampled choices of noise on the other elements. To compute P(x, i), we note that the G(x)iis nonzero if and only if H(x)iis greater than the k th-greatest element of H(x) excluding itself. The probability works out to be:\n\n$$P(x,i)=P r\\Big((x\\cdot W_{g}){i}+S t a n d x N o r a l()\\cdot S o f t p u s((x\\cdot W{n o i s e})_{i})$$ $$>k t h_e x c l u d i n(H(x),k,i)\\Big)$$ $$({\\boldsymbol{8}})$$\n\n$$(9)$$ $$(10)$$ (8) Where kth_excluding(v, k, i) means the kth highest component of v, excluding component i. Simplifying, we get:\n\n$$P(x,i)=\\Phi{\\Big(}{\\frac{(x\\cdot W_{g}){i}-k t h_c x c l u d i n g(H(x),k,i)}{S o f t p l u s((x\\cdot W{n o i s e})_{i})}}{\\Big)}$$\n\nWhere Φ is the CDF of the standard normal distribution.\n\n$$L o a d(X){i}=\\sum{x\\in X}P(x,i)$$\n\n$$(11)$$ P(x, i) (10) We can now define the load loss to be the square of the coefficient of variation of the load vector, multiplied by a hand-tuned scaling factor wload.\n\n$$L_{l o a d}(X)=w_{l o a d}\\cdot C V(L o a d(X))^{2}$$ Lload(X) = wload · CV (Load(X))2(11) Initial Load Imbalance: To avoid out-of-memory errors, we need to initialize the network in a state of approximately equal expert load (since the soft constraints need some time to work). To accomplish this, we initialize the matrices Wg and Wnoise to all zeros, which yields no signal and some noise.\n\nExperiments: We trained a set of models with identical architecture (the MoE-256 model described in Appendix C), using different values of wimportance and wload. We trained each model for 10 epochs, then measured perplexity on the test set. We also measured the coefficients of variation in Importance and Load, as well as ratio of the load on the most overloaded expert to the average load. This last value is significant for load balancing purposes on distributed hardware. All of these metrics were averaged over several training batches.\n\nwimportance wload Test Perplexity CV (Importance(X)) CV (Load(X)) max(Load(X)) mean(Load(X)) 0.0 0.0 39.8 3.04 3.01 17.80 0.2 0.0 35.6 0.06 0.17 1.47 0.0 0.2 35.7 0.22 0.04 1.15 0.1 0.1 35.6 0.06 0.05 1.14 0.01 0.01 35.7 0.48 0.11 1.37 1.0 1.0 35.7 0.03 0.02 1.07\n\nTable 6: Experiments with different combinations of losses.\n\n13 Results: Results are reported in Table 6. All the combinations containing at least one the two losses led to very similar model quality, where having no loss was much worse. Models with higher values of wload had lower loads on the most overloaded expert.\n\nB Hierachical Mixture Of Experts\n\nIf the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network.3If the hierarchical MoE consists of a groups of b experts each, we denote the primary gating network by Gprimary, the secondary gating networks by (G1, G2..Ga), and the expert networks by (E0,0, E0,1..Ea,b). The output of the MoE is given by:\n\n$$y_{H}=\\sum_{i=1}^{a}\\sum_{j=1}^{b}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}\\cdot E{i,j}(x)$$ $$(12)$$ $$(13)$$ $$(14)$$\n\nOur metrics of expert utilization change to the following:\n\n$$I m p o r t a n c e_{H}(X){i,j}=\\sum{x\\in X}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}$$ $$L o a d{H}(X){i,j}={\\frac{L o a d{p r i m a r y}(X){i}\\cdot L o a d{i}(X^{(i)})_{j}}{|X^{(i)}|}}$$\n\nLoadprimary and Loadi deonte the Load functions for the primary gating network and i th secondary gating network respectively. X(i) denotes the subset of X for which Gprimary(x)i > 0.\n\nIt would seem simpler to let LoadH(X)i,j = Loadi(Xi)j , but this would not have a gradient with respect to the primary gating network, so we use the formulation above.\n\nC 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS C.1 8-MILLION-OPERATIONS-PER-TIMESTEP MODELS Model Architecture: Our model consists of five layers: a word embedding layer, a recurrent Long Short-Term Memory (LSTM) layer (Hochreiter & Schmidhuber, 1997; Gers et al., 2000), a MoE layer, a second LSTM layer, and a softmax layer. The dimensionality of the embedding layer, the number of units in each LSTM layer, and the input and output dimensionality of the MoE layer are all equal to 512. For every layer other than the softmax, we apply drouput (Zaremba et al., 2014) to the layer output, dropping each activation with probability DropP rob, otherwise dividing by (1 − DropP rob). After dropout, the output of the previous layer is added to the layer output.\n\nThis residual connection encourages gradient flow (He et al., 2015).\n\nMoE Layer Architecture: Each expert in the MoE layer is a feed forward network with one ReLU-activated hidden layer of size 1024 and an output layer of size 512. Thus, each expert contains [512 ∗ 1024] + [1024 ∗ 512] = 1M parameters. The output of the MoE layer is passed through a sigmoid function before dropout. We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts.\n\nWe call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers. Thus, each example is processed by exactly 4 experts for a total of 4M ops/timestep. The two LSTM layers contribute 2M ops/timestep each for the desired total of 8M.\n\n3 We have not found the need for deeper hierarchies.\n\nComputationally-Matched Baselines: The MoE-4 model does not employ sparsity, since all 4 experts are always used. In addition, we trained four more computationally-matched baseline models with no sparsity:\n\nMoE-1-Wide: The MoE layer consists of a single \"expert\" containing one ReLU-activated hidden layer of size 4096.\n\nMoE-1-Deep: The MoE layer consists of a single \"expert\" containing four ReLU-activated hidden layers, each with size 1024.\n\n4xLSTM-512: We replace the MoE layer with two additional 512-unit LSTM layers.\n\nLSTM-2048-512: The model contains one 2048-unit LSTM layer (and no MoE). The output of the LSTM is projected down to 512 dimensions (Sak et al., 2014). The next timestep of the LSTM receives the projected output. This is identical to one of the models published in (Jozefowicz et al., 2016). We re-ran it to account for differences in training regimen, and obtained results very similar to the published ones. Training: The models were trained on a cluster of 16 K40 GPUs using the synchronous method described in Section 3. Each batch consisted of a set of sentences totaling roughly 300,000 words. In the interest of time, we limited training to 10 epochs, (27,000 steps). Training took 12-16 hours for all models, except for MoE-4, which took 18 hours (since all the expert computation was performed on only 4 of 16 GPUs). We used the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 1000 training steps, and decreased after that so as to be proportional to the inverse square root of the step number. The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016). For each model, we performed a hyper-parmeter search to find the best dropout probability, in increments of 0.1.\n\nTo ensure balanced expert utilization we set wimportance = 0.1 and wload = 0.1, as described in Section 4 and Appendix A.\n\nResults: We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol. Results are reported in Table 7. For each model, we report the test perplexity, the computational budget, the parameter counts, the value of DropP rob, and the computational efficiency.\n\nModel Test Test ops/timestep #Params excluding Total Drop- TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params P rob per GPU 10 epochs (final) (millions) (billions) (observed) Kneser-Ney 5-gram* 67.6 0.00001 1.8 LSTM-512-512* 54.1 2.4 2.4 0.8 0.1 LSTM-1024-512* 48.2 4.7 4.7 0.8 0.1 LSTM-2048-512* 45.0 43.7 9.4 9.4 0.8 0.1 0.61 LSTM-2048-512 44.7 9.4 9.4 0.8 0.1 1.21 4xLSTM-512 46.0 8.4 8.4 0.8 0.1 1.07 MoE-1-Wide 46.1 8.4 8.4 0.8 0.1 1.29 MoE-1-Deep 45.7 8.4 8.4 0.8 0.1 1.29 MoE-4 45.0 8.4 8.4 0.8 0.1 0.52 MoE-32 39.7 8.4 37.8 0.9 0.1 0.87 MoE-256 35.7 8.6 272.9 1.1 0.1 0.81 MoE-256-h 36.0 8.4 272.9 1.1 0.1 0.89 MoE-1024-h 34.6 8.5 1079.0 1.9 0.2 0.90 MoE-4096-h 34.1 8.9 4303.4 5.1 0.2 0.74 2xLSTM-8192-1024* 34.7 30.6 151.0 151.0 1.8 0.25 1.09 MoE-34M 31.3 33.8 4313.9 6.0 0.3 1.22 MoE-143M 28.0 142.7 4371.1 6.0 0.4 1.56\n\nC.2 More Expensive Models\n\nWe ran two additional models (MoE-34M and MoE-143M) to investigate the effects of adding more computation in the presence of a large MoE layer. These models have computation budgets of 34M and 143M ops/timestep. Similar to the models above, these models use a MoE layer between two LSTM layers. The dimensionality of the embedding layer, and the input and output dimensionality of the MoE layer are set to 1024 instead of 512. For MoE-34M, the LSTM layers have 1024 units.\n\nFor MoE-143M, the LSTM layers have 4096 units and an output projection of size 1024 (Sak et al., 2014). MoE-34M uses a hierarchical MoE layer with 1024 experts, each with a hidden layer of size 2048. MoE-143M uses a hierarchical MoE layer with 256 experts, each with a hidden layer of size 8192. Both models have 4B parameters in the MoE layers. We searched for the best DropP rob for each model, and trained each model for 10 epochs.\n\nThe two models achieved test perplexity of 31.3 and 28.0 respectively, showing that even in the presence of a large MoE, more computation is still useful. Results are reported at the bottom of Table 7. The larger of the two models has a similar computational budget to the best published model from the literature, and training times are similar. Comparing after 10 epochs, our model has a lower test perplexity by 18%.\n\nD 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS Model Architecture: The models are similar in structure to the 8-million-operations-per-timestep models described in the previous section. We vary the number of experts between models, using an ordinary MoE layer with 32 experts and hierarchical MoE layers with 256, 1024, 4096, 16384, 65536 and 131072 experts. For the hierarchical MoE layers, the first level branching factors are 32, 32, 64, 128, 256 and 256, respectively.\n\nTraining: Models are trained on a cluster of 32 Tesla K40 GPUs, except for the last two models, which are trained on clusters of 64 and 128 GPUs so as to have enough memory for all the parameters. For all models, training batch sizes are approximately 2.5 million words. Models are trained once-through over about 100 billion words. We implement several memory optimizations in order to fit up to 1 billion parameters per GPU. First, we do not store the activations of the hidden layers of the experts, but instead recompute them on the backwards pass. Secondly, we modify the optimizer on the expert parameters to require less auxiliary storage: The Adam optimizer (Kingma & Ba, 2015) keeps first and second moment estimates of the perparameter gradients. This triples the required memory. To avoid keeping a first-moment estimator, we set β1 = 0. To reduce the size of the second moment estimator, we replace it with a factored approximation. For a matrix of parameters, instead of maintaining a full matrix of second-moment estimators, we maintain vectors of row-wise and column-wise averages of that matrix. At each step, the matrix of estimators is taken to be the outer product of those two vectors divided by the mean of either one. This technique could similarly be applied to Adagrad (Duchi et al., 2010).\n\nModel Test Test ops/timestep #Params excluding Total TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params per GPU .1 epochs 1 epoch (millions) (billions) (observed) Kneser-Ney 5-gram 67.1 45.3 0.00001 76.0 4xLSTM-512 54.5 47.0 8.4 8.4 0.1 1.23 MoE-32 48.5 40.4 8.4 37.8 0.1 0.83 MoE-256-h 42.8 35.3 8.4 272.9 0.4 1.11 MoE-1024-h 40.3 32.7 8.5 1079.0 1.2 1.14 MoE-4096-h 38.9 30.9 8.6 4303.4 4.4 1.07 MoE-16384-h 38.2 29.7 8.8 17201.0 17.3 0.96 MoE-65536-h 38.2 28.9 9.2 68791.0 68.9 0.72 MoE-131072-h 39.8 29.2 9.7 137577.6 137.7 0.30\n\nResults: We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8. Perplexity after 100 billion training words is 39% lower for the 68-billion-parameter MoE model than for the baseline model. It is notable that the measured computational efficiency of the largest model (0.30 TFLOPS/GPU) is very low compared to the other models. This is likely a result of the fact that, for purposes of comparison to the other models, we did not increase the training batch size proportionally to the number of GPUs. For comparison, we include results for a computationally matched baseline model consisting of 4 LSTMs, and for an unpruned 5-gram model with Kneser-Ney smoothing (Kneser & Ney, 1995).4\n\nE Machine Translation - Experimental Details\n\nModel Architecture for Single Language Pair MoE Models: Our model is a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decrease the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We insert MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). We use an attention mechanism between the encoder and decoder, with the first decoder LSTM receiving output from and providing input for the attention 5. All of the layers in our model have input and output dimensionality of 512. Our LSTM layers have 2048 hidden units, with a 512-dimensional output projection. We add residual connections around all LSTM and MoE layers to encourage gradient flow (He et al., 2015). Similar to GNMT, to effectively deal with rare words, we used subword units (also known as \"wordpieces\") (Schuster & Nakajima, 2012) for inputs and outputs in our system.\n\nWe use a shared source and target vocabulary of 32K wordpieces. We also used the same beam search technique as proposed in (Wu et al., 2016).\n\nWe train models with different numbers of experts in the MoE layers. In addition to a baseline model with no MoE layers, we train models with flat MoE layers containing 32 experts, and models with hierarchical MoE layers containing 512 and 2048 experts. The flat MoE layers use k = 4 and the hierarchical MoE models use k = 2 at each level of the gating network. Thus, each input is processed by exactly 4 experts in each MoE layer. Each expert in the MoE layer is a feed forward network with one hidden layer of size 2048 and ReLU activation. Thus, each expert contains [512 ∗ 2048] + [2048 ∗ 512] = 2M parameters. The output of the MoE layer is passed through a sigmoid function. We use the strictly-balanced gating function described in Appendix F.\n\nModel Architecture for Multilingual MoE Model: We used the same model architecture as for the single-language-pair models, with the following exceptions: We used noisy-top-k gating as described in Section 2.1, not the scheme from Appendix F. The MoE layers in the encoder and decoder are non-hierarchical MoEs with n = 512 experts, and k = 2. Each expert has a larger hidden layer of size 8192. This doubles the amount of computation in the MoE layers, raising the computational budget of the entire model from 85M to 102M ops/timestep. Training: We trained our networks using the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 2000 training steps, held constant for an additional 8000 steps, and decreased after that so as to be proportional to the inverse square root of the step number. For the single-language-pair models, similarly to (Wu et al., 2016), we applied dropout (Zaremba et al., 2014) to the output of all embedding, LSTM and MoE layers, using DropP rob = 0.4. Training was done synchronously on a cluster of up to 64 GPUs as described in section 3. Each training batch consisted of a set of sentence pairs containing roughly 16000 words per GPU.\n\nTo ensure balanced expert utilization we set wimportance = 0.01 and wload = 0.01, as described in Section 4 and Appendix A. Metrics: We evaluated our models using the perplexity and the standard BLEU score metric. We reported tokenized BLEU score as computed by the multi-bleu.pl script, downloaded from the public implementation of Moses (on Github), which was also used in (Luong et al., 2015a). Results: Tables 2, 3 and 4 in Section 5.3 show comparisons of our results to other published methods. Figure 4 shows test perplexity as a function of number of words in the (training data's) source sentences processed for models with different numbers of experts. As can be seen from the Figure, as we increased the number of experts to approach 2048, the test perplexity of our model continued to improve.\n\nWe found that the experts indeed become highly specialized by syntax and/or semantics, as can be seen in Table 9. For example, one expert is used when the indefinite article \"a\" introduces the direct object in a verb phrase indicating importance or leadership.\n\nExpert 381 Expert 752 Expert 2004 ... with researchers , ... ... plays a core ... ... with rapidly growing ... ... to innovation . ... plays a critical ... ... under static conditions ... ... tics researchers . ... provides a legislative ... ... to swift ly ... ... the generation of ... ... play a leading ... ... to dras tically ... ... technology innovations is ... ... assume a leadership ... ... the rapid and ... ... technological innovations , ... ... plays a central ... ... the fast est ... ... support innovation throughout ... ... taken a leading ... ... the Quick Method ... ... role innovation will ... ... established a reconciliation ... ... rec urrent ) ... ... research scienti st ... ... played a vital ... ... provides quick access ... ... promoting innovation where ... ... have a central ... ... of volatile organic ... ... ... ...\n\nDue to some peculiarities in our infrastructure which have since been fixed, at the time we ran some of the machine translation experiments, our models ran faster if every expert received exactly the same batch size. To accommodate this, we used a different gating function which we describe below. Recall that we define the softmax gating function to be:\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ Gσ(x) = Sof tmax(x · Wg) (15) Sparse Gating (alternate formulation): To obtain a sparse gating vector, we multiply Gσ(x) component-wise with a sparse mask M(Gσ(x)) and normalize the output. The mask itself is a function of Gσ(x) and specifies which experts are assigned to each input example:\n\n$$(15)$$ $$G(x){i}=\\frac{G{\\sigma}(x){i}M(G{\\sigma}(x)){i}}{\\sum{j=1}^{n}G_{\\sigma}(x){j}M(G{\\sigma}(x))_{j}}\\tag{1}$$ $$(17)$$\n\n$$(16)$$\n\nTop-K Mask: To implement top-k gating in this formulation, we would let M(v) = T opK(v, k), where:\n\n$$T o p K(v,k){i}=\\begin{cases}1&{\\mathrm{if~}}v{i}{\\mathrm{~is~in~the~top~}}k{\\mathrm{~elements~of~}}v.\\ 0&{\\mathrm{otherwise.}}\\end{cases}$$ 0 otherwise. (17) Batchwise Mask: To force each expert to receive the exact same number of examples, we introduce an alternative mask function, Mbatchwise(X, m), which operates over batches of input vectors.\n\nInstead of keeping the top k values per example, we keep the top m values per expert across the training batch, where m = k|X| n, so that each example is sent to an average of k experts.\n\n$$M_{b a t c h w i s e}(X,m){j,i}=\\begin{cases}1&\\text{if}X{j,i}\\text{is in the top}m\\text{values for to expert}i\\ 0&\\text{otherwise}\\end{cases}$$\n\nAs our experiments suggest and also observed in (Ioffe & Szegedy, 2015), using a batchwise function during training (such as Mbatchwise) requires modifications to the inference when we may not have a large batch of examples. Our solution to this is to train a vector T of per-expert threshold values to approximate the effects of the batchwise mask. We use the following mask at inference time:\n\n$$(18)$$ $$M_{threshold}(x,T){i}=\\begin{cases}1&\\text{if}x{i}>T_{i}\\ 0&\\text{otherwise}\\end{cases}\\tag{1}$$ $$(19)$$\n\nTo learn the threshold values, we apply an additional loss at training time which is minimized when the batchwise mask and the threshold mask are identical.\n\n$$L_{batchwise}(X,T,m)=\\sum_{j=1}^{|X|}\\sum_{i=1}^{n}(M_{threshold}(x,T){i}-M{batchwise}(X,m){j,i})(X{j,i}-T_{i})\\tag{20}$$\n\nG ATTENTION FUNCTION The attention mechanism described in GNMT (Wu et al., 2016) involves a learned \"Attention Function\" A(xi, yj ) which takes a \"source vector\" xi and a \"target vector\" yj , and must be computed for every source time step i and target time step j. In GNMT, the attention function is implemented as a feed forward neural network with a hidden layer of size n. It can be expressed as:\n\n$$A_{G N M T}(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}t a n h((x_{i}U){d}+(y{j}W)_{d})$$\n\nWhere U and W are trainable weight matrices and V is a trainable weight vector.\n\nFor performance reasons, in our models, we used a slightly different attention function:\n\n$$A(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}tanh((x_{i}U){d})tanh((y{j}W)_{d})\\tag{22}$$\n\nWith our attention function, we can simultaneously compute the attention function on multiple source time steps and multiple target time steps using optimized matrix multiplications. We found little difference in quality between the two functions.\n\n$$(21)$$",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/1701.06538v1.md"
+          },
+          "headlines": {
+            "1 Introduction And Related Work": [
+              "1.1 Conditional Computation",
+              "1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER",
+              "1.3 Related Work On Mixtures Of Experts"
+            ],
+            "2 The Structure Of The Mixture-Of-Experts Layer": [
+              "2.1 Gating Network"
+            ],
+            "3 Addressing Performance Challenges": [
+              "3.1 The Shrinking Batch Problem",
+              "3.2 Network Bandwidth"
+            ],
+            "4 Balancing Expert Utilization": [],
+            "5 Experiments": [
+              "5.1 1 Billion Word Language Modeling Benchmark",
+              "5.2 100 BILLION WORD GOOGLE NEWS CORPUS",
+              "5.3 Machine Translation (Single Language Pair)",
+              "5.4 Multilingual Machine Translation"
+            ],
+            "6 Conclusion": [],
+            "Appendices": [
+              "A Load-Balancing Loss",
+              "B Hierachical Mixture Of Experts",
+              "C 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS",
+              "D 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS",
+              "E Machine Translation - Experimental Details"
+            ]
+          },
+          "summary": "The paper introduces a Sparsely-Gated Mixture-of-Experts (MoE) layer to enhance neural network capacity without a proportional increase in computation. This approach uses a trainable gating network to select a sparse combination of experts for each input, significantly improving model capacity and performance in language modeling and machine translation tasks. The MoE layer, with up to 137 billion parameters, achieves better results than state-of-the-art models at lower computational costs. The paper addresses challenges like batch size and network bandwidth, proposing solutions like mixing data and model parallelism. Experiments show that MoE models outperform traditional models in various benchmarks, demonstrating the potential of conditional computation in deep learning.",
+          "summary_embedding": [
+            -0.01644345559179783,
+            -0.016995344310998917,
+            -0.016372699290513992,
+            -0.014349107630550861,
+            0.02282555401325226,
+            0.045934125781059265,
+            -0.014646278694272041,
+            0.03563220426440239,
+            -0.004829026758670807,
+            0.005617944523692131,
+            -0.00932550523430109,
+            -0.008143897168338299,
+            0.005575491581112146,
+            0.01538213063031435,
+            -0.005653322208672762,
+            -0.020306676626205444,
+            -0.013832597061991692,
+            -0.023235931992530823,
+            0.028344441205263138,
+            -0.020080260932445526,
+            0.011823155917227268,
+            -0.020943470299243927,
+            -0.025868017226457596,
+            -0.016500059515237808,
+            0.025061409920454025,
+            0.024806693196296692,
+            0.02583971433341503,
+            0.003827843815088272,
+            -0.03492465242743492,
+            0.033651065081357956,
+            0.014674580655992031,
+            0.012488252483308315,
+            -0.031500112265348434,
+            -0.009162768721580505,
+            -0.05351905897259712,
+            -0.05227377265691757,
+            0.04143410921096802,
+            0.022245362401008606,
+            -0.016231190413236618,
+            0.023717066273093224,
+            0.017264213413000107,
+            0.005816058721393347,
+            -0.020632149651646614,
+            -0.020844414830207825,
+            0.003700484987348318,
+            -0.013103820383548737,
+            -0.01983969286084175,
+            -0.013528349809348583,
+            -0.01828308403491974,
+            -0.01577835902571678,
+            0.009551920928061008,
+            0.0002856731880456209,
+            0.0029186424799263477,
+            -0.003424540627747774,
+            -0.047830358147621155,
+            0.0248208437114954,
+            0.019853845238685608,
+            0.03492465242743492,
+            0.04633035510778427,
+            0.013011839240789413,
+            0.01516986545175314,
+            -0.014271277002990246,
+            -0.04021712392568588,
+            0.005497660953551531,
+            -0.00041656987741589546,
+            -0.005380915477871895,
+            -0.005819596350193024,
+            0.008575502783060074,
+            0.008724087849259377,
+            -0.01143400277942419,
+            -0.04202845320105553,
+            0.013952880166471004,
+            -0.020561393350362778,
+            0.00010143388499272987,
+            0.01354957651346922,
+            0.008292482234537601,
+            0.02677368000149727,
+            0.03195294365286827,
+            0.023971782997250557,
+            -0.008724087849259377,
+            -0.01340806670486927,
+            0.019924599677324295,
+            -0.0022676968947052956,
+            0.007018893025815487,
+            0.04570771008729935,
+            -0.04033033177256584,
+            0.02163686975836754,
+            -0.025726506486535072,
+            -0.015339677222073078,
+            -0.005349075887352228,
+            -0.022471778094768524,
+            0.02806142158806324,
+            -0.007344366051256657,
+            0.017801949754357338,
+            0.017094401642680168,
+            0.01594817079603672,
+            0.013330236077308655,
+            -0.001894464367069304,
+            -0.022938761860132217,
+            0.0018360915128141642,
+            -0.01541043259203434,
+            0.02114158496260643,
+            0.0015230007702484727,
+            0.005469358991831541,
+            0.027594437822699547,
+            0.031160488724708557,
+            -0.023490650579333305,
+            0.009502392262220383,
+            0.02422650158405304,
+            0.06967949122190475,
+            0.015424583107233047,
+            -0.006435164716094732,
+            -0.0024640418123453856,
+            0.04143410921096802,
+            0.0015513027319684625,
+            0.0038349193055182695,
+            -0.06826439499855042,
+            -0.03022652305662632,
+            -0.0011365016689524055,
+            -0.02254253253340721,
+            0.02227366529405117,
+            0.01002597901970148,
+            -0.009750034660100937,
+            -0.022457627579569817,
+            -0.04516997188329697,
+            0.05190584436058998,
+            -0.00545874610543251,
+            -0.016259491443634033,
+            -0.005554265342652798,
+            -0.05340585112571716,
+            0.00039136342820711434,
+            0.015042506158351898,
+            -0.018481198698282242,
+            -0.03670767694711685,
+            0.036141637712717056,
+            0.0007398316520266235,
+            0.03741522878408432,
+            0.012459950521588326,
+            -0.01708024926483631,
+            -0.020094411447644234,
+            0.022259512916207314,
+            -0.011639192700386047,
+            -0.0038313816767185926,
+            -0.027339721098542213,
+            0.0065236082300543785,
+            -0.028103873133659363,
+            -0.0014566679019480944,
+            0.06730213016271591,
+            -0.04816998168826103,
+            0.02946236915886402,
+            -0.029660483822226524,
+            0.023660462349653244,
+            -0.008575502783060074,
+            0.0278916098177433,
+            -0.010238244198262691,
+            0.005840822588652372,
+            -0.013457595370709896,
+            -0.0010586712742224336,
+            -0.020745357498526573,
+            0.0034210027661174536,
+            -0.016259491443634033,
+            -0.027750099077820778,
+            -0.007054270710796118,
+            0.031273696571588516,
+            0.01698119193315506,
+            0.025655752047896385,
+            0.025429336354136467,
+            0.01246702577918768,
+            -0.04935866594314575,
+            0.02595292218029499,
+            0.0018926954362541437,
+            0.013514199294149876,
+            0.013181651011109352,
+            -0.0010418669553473592,
+            0.025599148124456406,
+            0.012049571610987186,
+            0.025061409920454025,
+            0.037443529814481735,
+            0.0027682881336659193,
+            0.04737752676010132,
+            -0.015693452209234238,
+            -0.005292471963912249,
+            0.028528403490781784,
+            0.000505455827806145,
+            0.02465103194117546,
+            -0.001686621573753655,
+            0.013330236077308655,
+            0.000776978034991771,
+            0.03834919258952141,
+            -0.006339645478874445,
+            0.029518973082304,
+            0.021608568727970123,
+            0.028316138312220573,
+            -0.014759486541152,
+            0.023122724145650864,
+            -0.019754787907004356,
+            -0.02292460948228836,
+            0.035547297447919846,
+            0.004082561936229467,
+            0.01886327564716339,
+            -0.028457649052143097,
+            -0.030679354444146156,
+            0.015452885068953037,
+            -0.01855195313692093,
+            0.016556663438677788,
+            0.03639635816216469,
+            0.0008273909334093332,
+            -0.0025719432160258293,
+            0.026292547583580017,
+            -0.04245298355817795,
+            0.012856177985668182,
+            -0.01532552670687437,
+            -0.02855670638382435,
+            -0.019528372213244438,
+            -0.00875946506857872,
+            -0.0018095583654940128,
+            -0.013691086322069168,
+            -0.0037570889107882977,
+            -0.007563706487417221,
+            -0.02459442801773548,
+            0.00935380719602108,
+            0.013627407141029835,
+            0.009849091991782188,
+            -0.018665160983800888,
+            0.0019952901639044285,
+            -0.008257105015218258,
+            -0.0160047747194767,
+            -0.03752843663096428,
+            0.015268922783434391,
+            -0.0024888061452656984,
+            0.009191070683300495,
+            0.04078316316008568,
+            0.028485950082540512,
+            -0.005055442452430725,
+            0.0329718142747879,
+            -0.033707667142152786,
+            0.017915157601237297,
+            -0.008320784196257591,
+            0.009191070683300495,
+            0.002989397384226322,
+            -0.025542544201016426,
+            0.027424626052379608,
+            -0.03792466223239899,
+            -0.038151077926158905,
+            0.022995365783572197,
+            0.011858533136546612,
+            0.0017096169758588076,
+            -0.019386861473321915,
+            -0.017717044800519943,
+            0.0008499440737068653,
+            0.002752368338406086,
+            -0.018834972754120827,
+            -0.01019579079002142,
+            0.02227366529405117,
+            -0.025500090792775154,
+            0.036085035651922226,
+            0.023391593247652054,
+            -0.03520767390727997,
+            -0.00608492735773325,
+            -0.01903308741748333,
+            0.059037946164608,
+            -0.0011506526498124003,
+            -0.010301923379302025,
+            0.022938761860132217,
+            -0.027268964797258377,
+            -0.004524780437350273,
+            0.022301966324448586,
+            0.01828308403491974,
+            -0.008278331719338894,
+            0.003983504604548216,
+            -0.022287815809249878,
+            -0.05281151086091995,
+            -0.010577867738902569,
+            -0.0003305583668407053,
+            0.054368119686841965,
+            -0.012367968447506428,
+            0.0020695829298347235,
+            -0.006413938011974096,
+            0.004100250545889139,
+            0.03704730048775673,
+            0.002140337834134698,
+            0.0025082635693252087,
+            -0.013698162510991096,
+            0.0371888130903244,
+            0.022316116839647293,
+            0.005883275531232357,
+            -0.012162779457867146,
+            0.005377377849072218,
+            -0.04899074137210846,
+            -0.0009118546731770039,
+            0.005741765722632408,
+            0.007386818993836641,
+            -0.025910470634698868,
+            0.017717044800519943,
+            0.009205221198499203,
+            -0.0024693484883755445,
+            -0.009792488068342209,
+            0.01173117384314537,
+            0.025429336354136467,
+            -0.019938750192523003,
+            -0.05700020492076874,
+            0.0034705311991274357,
+            0.012014193460345268,
+            -0.004054259974509478,
+            -0.019401011988520622,
+            0.006792477332055569,
+            0.0021934041287750006,
+            0.022698193788528442,
+            -0.011271266266703606,
+            -0.0029221803415566683,
+            0.040585052222013474,
+            0.009516543708741665,
+            -0.0016813150141388178,
+            -0.017717044800519943,
+            -0.011794853024184704,
+            0.018070818856358528,
+            0.004740582779049873,
+            0.0035289041697978973,
+            -0.01751893013715744,
+            -0.029264256358146667,
+            0.02800481766462326,
+            0.019967053085565567,
+            -0.048905834555625916,
+            -0.008200501091778278,
+            -0.019273653626441956,
+            0.0019599127117544413,
+            0.008695785887539387,
+            0.04613224044442177,
+            -0.00590803986415267,
+            0.008313708938658237,
+            -0.020179318264126778,
+            -0.05003791302442551,
+            -0.02708500251173973,
+            0.03707560524344444,
+            -0.028684064745903015,
+            -0.04276430234313011,
+            -0.010252394713461399,
+            0.016740625724196434,
+            -0.009403335861861706,
+            -0.018679311498999596,
+            -0.011738249100744724,
+            -0.04969828948378563,
+            -0.003497064346447587,
+            0.0337359681725502,
+            0.017264213413000107,
+            -0.008441068232059479,
+            -0.033877480775117874,
+            0.019641580060124397,
+            -0.007160402834415436,
+            -0.006474079564213753,
+            0.0007862646016292274,
+            0.008794843219220638,
+            0.02400008589029312,
+            0.011879759840667248,
+            -0.02763689123094082,
+            -0.03427370637655258,
+            -0.0024888061452656984,
+            0.014929298311471939,
+            -0.00878776703029871,
+            0.0245378240942955,
+            0.024679332971572876,
+            -0.022995365783572197,
+            -0.01654251106083393,
+            0.015750056132674217,
+            -0.05779265984892845,
+            -0.010287772864103317,
+            -0.029235953465104103,
+            -0.006183984223753214,
+            0.0009613831643946469,
+            0.0061096916906535625,
+            0.00012304731353651732,
+            0.03846240043640137,
+            0.008801918476819992,
+            0.006014172453433275,
+            0.009191070683300495,
+            0.01913214474916458,
+            0.028471799567341805,
+            0.009155693463981152,
+            -0.03517937287688255,
+            0.02850010246038437,
+            -0.017051948234438896,
+            0.007811348885297775,
+            -0.025754809379577637,
+            -0.013033065013587475,
+            0.00443633645772934,
+            0.02725481428205967,
+            -0.000503244751598686,
+            0.018849123269319534,
+            -0.021919889375567436,
+            0.037500135600566864,
+            -0.029915200546383858,
+            -0.05278320610523224,
+            -0.02005195803940296,
+            -0.008285406976938248,
+            0.015764206647872925,
+            0.017278363928198814,
+            -0.019924599677324295,
+            -0.0286557637155056,
+            -0.021750077605247498,
+            0.006470541935414076,
+            0.010542490519583225,
+            -0.021099131554365158,
+            -0.010627396404743195,
+            0.01502835564315319,
+            -0.00404364662244916,
+            -0.01886327564716339,
+            0.013684011064469814,
+            0.015693452209234238,
+            0.03169822692871094,
+            -0.03308502212166786,
+            0.004287751391530037,
+            -0.025967074558138847,
+            -0.04327373951673508,
+            0.004096712917089462,
+            0.03166992589831352,
+            0.027509532868862152,
+            -0.018636859953403473,
+            0.010754755698144436,
+            0.003990580327808857,
+            0.004234685096889734,
+            -0.0033803186379373074,
+            0.023603858426213264,
+            0.03291521221399307,
+            0.02763689123094082,
+            -0.011653343215584755,
+            0.004921008367091417,
+            -0.01559439580887556,
+            -0.0011108530452474952,
+            0.01865101046860218,
+            -0.017816102132201195,
+            0.033877480775117874,
+            -0.029405765235424042,
+            0.02104252763092518,
+            -0.035122767090797424,
+            0.006141531281173229,
+            -0.012099100276827812,
+            -0.00404364662244916,
+            0.04833979532122612,
+            -0.017646290361881256,
+            -0.0019546060357242823,
+            -0.008101443760097027,
+            -0.014575524255633354,
+            -0.02158026583492756,
+            0.008766540326178074,
+            -0.04033033177256584,
+            -0.012367968447506428,
+            -0.028429346159100533,
+            0.003424540627747774,
+            0.019910449162125587,
+            -0.06277380883693695,
+            -0.06237758323550224,
+            0.012275987304747105,
+            -0.04918885603547096,
+            -0.026419905945658684,
+            0.02963218092918396,
+            -0.002984090941026807,
+            0.0012780115939676762,
+            -0.0024003623984754086,
+            0.0068101659417152405,
+            -0.029207652434706688,
+            0.00425944896414876,
+            -0.000887974863871932,
+            0.03271709755063057,
+            -0.021056678146123886,
+            -0.008122670464217663,
+            0.004878554958850145,
+            -0.06860402226448059,
+            -0.028344441205263138,
+            0.011271266266703606,
+            -0.009459939785301685,
+            -0.0066615804098546505,
+            -0.017915157601237297,
+            -0.004383270163089037,
+            0.0008950503543019295,
+            0.0032830305863171816,
+            0.05801907554268837,
+            -0.01983969286084175,
+            -0.053773775696754456,
+            0.03633975237607956,
+            0.00887267291545868,
+            0.029603879898786545,
+            -0.025556694716215134,
+            -0.017533080652356148,
+            -0.01855195313692093,
+            -0.020504789426922798,
+            -0.04123599827289581,
+            0.00404364662244916,
+            -0.005918653216212988,
+            0.009212297387421131,
+            -0.0009693430620245636,
+            0.02763689123094082,
+            0.029235953465104103,
+            0.023108573630452156,
+            0.012750045396387577,
+            0.01079013291746378,
+            -0.022372720763087273,
+            -0.021594416350126266,
+            0.0012222920777276158,
+            0.04683978855609894,
+            0.024636881425976753,
+            -0.02675952948629856,
+            -0.011504758149385452,
+            0.03953787684440613,
+            0.0419718474149704,
+            -0.01974063739180565,
+            -0.02276895008981228,
+            0.00764861237257719,
+            0.03266049548983574,
+            -0.04477374628186226,
+            0.005932804197072983,
+            0.03410389646887779,
+            0.014405711553990841,
+            -0.005356151144951582,
+            -0.010294848121702671,
+            0.022570835426449776,
+            0.03152841329574585,
+            -0.008724087849259377,
+            0.01876421831548214,
+            0.016910437494516373,
+            0.029349161311984062,
+            -0.0067889392375946045,
+            0.0005766529939137399,
+            -0.024735936895012856,
+            0.055188875645399094,
+            -0.03059444949030876,
+            0.02909444272518158,
+            0.010316074825823307,
+            -0.00593634182587266,
+            0.039056744426488876,
+            -0.016641568392515182,
+            -0.0431322306394577,
+            0.01913214474916458,
+            -0.006979977712035179,
+            -0.0055613406002521515,
+            -0.03444352000951767,
+            -0.007592008449137211,
+            -0.007507102098315954,
+            -0.006714646704494953,
+            -0.022556684911251068,
+            -0.02606613002717495,
+            0.004415110219269991,
+            0.00973588414490223,
+            -0.006484692916274071,
+            0.00698705343529582,
+            0.02092931978404522,
+            -0.013507124036550522,
+            0.02525952458381653,
+            -0.046047333627939224,
+            -0.0008366775000467896,
+            -0.0038596836384385824,
+            -0.025712355971336365,
+            -0.0345001220703125,
+            -0.0005770951975136995,
+            -0.0024251264985650778,
+            -0.024622729048132896,
+            -0.00914154201745987,
+            -0.008752389810979366,
+            -0.0023119186516851187,
+            -0.004938696976751089,
+            -0.055188875645399094,
+            0.008695785887539387,
+            0.010846736840903759,
+            -0.002713453024625778,
+            0.0055507272481918335,
+            -0.010613245889544487,
+            -0.03039633482694626,
+            0.03039633482694626,
+            0.04129260033369064,
+            0.033707667142152786,
+            0.014334957115352154,
+            -0.0074646491557359695,
+            -0.017165156081318855,
+            0.006060163024812937,
+            0.021721776574850082,
+            0.00902833417057991,
+            0.007846726104617119,
+            0.03877372294664383,
+            0.00011331850691931322,
+            -0.008349086157977581,
+            -0.009764186106622219,
+            0.028740668669342995,
+            -0.001085204305127263,
+            0.023603858426213264,
+            0.050802066922187805,
+            0.030424637719988823,
+            0.020589696243405342,
+            -0.01681138016283512,
+            -0.02010856196284294,
+            -0.00851889792829752,
+            -0.02855670638382435,
+            0.0364246591925621,
+            0.012127402238547802,
+            -0.005232329946011305,
+            -0.011540135368704796,
+            0.019061388447880745,
+            -0.033339742571115494,
+            -0.0037358624394983053,
+            0.00048422932741232216,
+            0.013924578204751015,
+            -0.03379257395863533,
+            0.0002480846014805138,
+            0.031613320112228394,
+            0.014957600273191929,
+            -0.016995344310998917,
+            -0.0188774261623621,
+            0.010549565777182579,
+            0.020462337881326675,
+            0.018948180601000786,
+            0.00037743354914709926,
+            0.00897173024713993,
+            -0.00022121983056422323,
+            -0.0005319889169186354,
+            0.01590571738779545,
+            -0.007952858693897724,
+            0.036254845559597015,
+            -0.003525366308167577,
+            -0.01325240544974804,
+            -0.0037500134203583,
+            -0.021438755095005035,
+            -0.024467067793011665,
+            0.043556760996580124,
+            0.023971782997250557,
+            0.04661337286233902,
+            -0.005660397466272116,
+            -0.004245297983288765,
+            0.025825563818216324,
+            0.008221727795898914,
+            -0.03288691118359566,
+            0.000265110022155568,
+            0.005281858611851931,
+            -0.02892463095486164,
+            0.016641568392515182,
+            -0.02476423978805542,
+            -0.016896286979317665,
+            -0.006771250627934933,
+            -0.0015557248843833804,
+            -0.028188779950141907,
+            0.01703779585659504,
+            0.0031309074256569147,
+            -0.0027099153958261013,
+            -0.039764292538166046,
+            0.006880920846015215,
+            0.020419884473085403,
+            0.015311375260353088,
+            0.0037358624394983053,
+            0.011525984853506088,
+            -0.016613267362117767,
+            0.002126186853274703,
+            0.03379257395863533,
+            0.004878554958850145,
+            0.02005195803940296,
+            0.0027647505048662424,
+            0.023773670196533203,
+            0.03633975237607956,
+            -0.01681138016283512,
+            -0.02189158834517002,
+            0.01519816741347313,
+            -0.02233026921749115,
+            0.029943503439426422,
+            0.004889168310910463,
+            -0.012283062562346458,
+            0.022033097222447395,
+            -0.009827865287661552,
+            -0.0027346794959157705,
+            0.01659911684691906,
+            -0.022740647196769714,
+            -0.017717044800519943,
+            0.03183973580598831,
+            -0.039452970027923584,
+            0.012856177985668182,
+            -0.008547200821340084,
+            0.015863263979554176,
+            -0.023066120222210884,
+            0.011172209866344929,
+            0.007100261282175779,
+            -0.01573590561747551,
+            0.010797208175063133,
+            0.009693430736660957,
+            0.00545874610543251,
+            0.0018926954362541437,
+            0.02179253101348877,
+            -0.0006049550138413906,
+            -0.011257115751504898,
+            -0.018155725672841072,
+            -0.004694592207670212,
+            -0.002614396158605814,
+            -0.016556663438677788,
+            0.01865101046860218,
+            -0.041066184639930725,
+            -0.013280707411468029,
+            -0.008879749104380608,
+            -0.013988257385790348,
+            -0.010011828504502773,
+            0.024184048175811768,
+            0.025669902563095093,
+            -0.002361447084695101,
+            -0.031217092648148537,
+            0.01790100708603859,
+            0.024198200553655624,
+            0.003654494183138013,
+            -0.023646311834454536,
+            0.0302831269800663,
+            -0.008936353027820587,
+            0.014971751719713211,
+            0.026136886328458786,
+            0.014094389975070953,
+            0.004595535341650248,
+            -0.010500037111341953,
+            -0.02114158496260643,
+            0.01481609046459198,
+            0.010507113300263882,
+            0.033537857234478,
+            0.014745336025953293,
+            0.02071705460548401,
+            0.01594817079603672,
+            -0.023320838809013367,
+            -0.011214662343263626,
+            -0.003482913365587592,
+            -0.022514231503009796,
+            0.03093407303094864,
+            0.0006129149114713073,
+            0.02114158496260643,
+            0.026618018746376038,
+            -0.03319823369383812,
+            -0.01187268365174532,
+            0.032688796520233154,
+            0.02779255248606205,
+            0.05108508840203285,
+            0.025316128507256508,
+            0.015396281145513058,
+            0.027226513251662254,
+            -0.03585861995816231,
+            -0.036368053406476974,
+            0.001548649393953383,
+            0.04256619140505791,
+            -0.008426916785538197,
+            0.00967928022146225,
+            0.03900013864040375,
+            -0.006781863979995251,
+            -0.01611798256635666,
+            0.019457615911960602,
+            0.0026710000820457935,
+            0.002448122017085552,
+            -0.0033113325480371714,
+            -0.01335146278142929,
+            0.03175482898950577,
+            -0.0051474240608513355,
+            0.013570803217589855,
+            -0.019273653626441956,
+            0.027113303542137146,
+            0.0001799829478841275,
+            -0.003229964291676879,
+            0.028344441205263138,
+            -0.002660386962816119,
+            -0.051990751177072525,
+            0.024212351068854332,
+            0.019811391830444336,
+            -0.011561362072825432,
+            -0.0036332677118480206,
+            -0.003792466362938285,
+            0.04567940905690193,
+            0.034698236733675,
+            0.029009537771344185,
+            -0.0020501252729445696,
+            0.0168679840862751,
+            0.018792519345879555,
+            0.0053667644970119,
+            0.014278353191912174,
+            -0.0017025414854288101,
+            -0.007853801362216473,
+            0.021056678146123886,
+            -0.020787810906767845,
+            0.03226426616311073,
+            -0.016570813953876495,
+            -0.020122714340686798,
+            0.008738238364458084,
+            -0.011172209866344929,
+            -0.0009083169279620051,
+            -0.006311343517154455,
+            -0.0008813415770418942,
+            -0.0029664020985364914,
+            0.016938740387558937,
+            -0.004446949809789658,
+            0.04661337286233902,
+            -0.013931653462350368,
+            -0.008808993734419346,
+            0.006972902454435825,
+            -0.025981225073337555,
+            0.0013346156338229775,
+            0.0028726516757160425,
+            0.027000095695257187,
+            -0.01989629678428173,
+            -0.00970758218318224,
+            0.01606137864291668,
+            0.06039644405245781,
+            -0.008639181964099407,
+            -0.0026267783250659704,
+            0.027226513251662254,
+            -0.002787745790556073,
+            -0.03551899641752243,
+            0.012919857166707516,
+            -0.04494355618953705,
+            0.017589686438441277,
+            0.026349151507019997,
+            -0.04986810311675072,
+            -0.022089701145887375,
+            -0.02347649820148945,
+            -0.02043403498828411,
+            0.023434046655893326,
+            -0.009976450353860855,
+            0.013867974281311035,
+            -0.017801949754357338,
+            -0.009014182724058628,
+            0.020518941804766655,
+            0.009176919236779213,
+            0.03735862299799919,
+            -0.027551984414458275,
+            -0.016783079132437706,
+            -0.027820853516459465,
+            -0.04304732382297516,
+            0.009558996185660362,
+            -0.0278916098177433,
+            0.0027099153958261013,
+            -0.0016291331266984344,
+            -0.040302030742168427,
+            -0.037981268018484116,
+            0.005278320983052254,
+            -0.017985913902521133,
+            0.01516986545175314,
+            0.020915169268846512,
+            0.02731141820549965,
+            -0.0020253609400242567,
+            -0.030198222026228905,
+            0.029235953465104103,
+            -0.007344366051256657,
+            0.03223596513271332,
+            -0.00689507182687521,
+            0.005066055804491043,
+            -0.0070436573587358,
+            0.031160488724708557,
+            -0.023009516298770905,
+            0.018792519345879555,
+            -0.026801982894539833,
+            -0.004086099565029144,
+            -0.018990634009242058,
+            -0.01774534583091736,
+            0.004224071744829416,
+            0.004521242808550596,
+            7.888073741924018e-05,
+            0.0048750173300504684,
+            0.005777143407613039,
+            0.024806693196296692,
+            0.0021421066485345364,
+            0.02589631825685501,
+            -0.027877457439899445,
+            -0.025316128507256508,
+            0.01719345711171627,
+            -0.02077365852892399,
+            0.005678086541593075,
+            -0.006261814851313829,
+            -0.0302831269800663,
+            -0.01202834490686655,
+            0.023391593247652054,
+            0.02162271924316883,
+            -0.005490585695952177,
+            0.002644466934725642,
+            0.02206140011548996,
+            -0.026023678481578827,
+            0.00437973253428936,
+            -0.03781145438551903,
+            -0.018396291881799698,
+            0.029773691669106483,
+            -0.03195294365286827,
+            0.020745357498526573,
+            0.003185742534697056,
+            0.03483974561095238,
+            -0.017292514443397522,
+            -0.01947176828980446,
+            0.01654251106083393,
+            -0.01008258294314146,
+            0.004142703488469124,
+            0.01606137864291668,
+            0.01616043597459793,
+            -0.0015070808585733175,
+            0.05380208045244217,
+            0.0008167776977643371,
+            0.006909222807735205,
+            -0.049075648188591,
+            0.004496478475630283,
+            -0.044858649373054504,
+            0.0002768288250081241,
+            -0.013337312266230583,
+            -0.02514631673693657,
+            0.0014027173165231943,
+            0.001001182827167213,
+            0.001963450340554118,
+            -0.01913214474916458,
+            -0.01989629678428173,
+            0.011596739292144775,
+            -0.015863263979554176,
+            0.004928083624690771,
+            -0.06418890506029129,
+            -0.01855195313692093,
+            -0.004974074196070433,
+            0.0043337419629096985,
+            0.0007699025445617735,
+            0.03342464938759804,
+            -0.01478778850287199,
+            0.010117961093783379,
+            -0.01549533847719431,
+            -0.026363302022218704,
+            0.04010391607880592,
+            0.018750067800283432,
+            -0.023773670196533203,
+            0.024693485349416733,
+            0.012106175534427166,
+            0.03099067695438862,
+            -0.0318114347755909,
+            0.020349130034446716,
+            0.018523650243878365,
+            -0.0029823218937963247,
+            0.006551910191774368,
+            0.008575502783060074,
+            0.01832553744316101,
+            -0.015764206647872925,
+            -0.0003632825391832739,
+            0.011419852264225483,
+            0.010033054277300835,
+            0.031924642622470856,
+            0.01403071079403162,
+            -0.03183973580598831,
+            0.019047237932682037,
+            0.01616043597459793,
+            0.02298121340572834,
+            0.033820874989032745,
+            -0.027481229975819588,
+            0.038575608283281326,
+            -0.010386829264461994,
+            -0.024042539298534393,
+            0.020801961421966553,
+            -0.032688796520233154,
+            -0.006679269019514322,
+            0.030735958367586136,
+            0.04398128762841225,
+            -0.0015566092915832996,
+            -0.044632233679294586,
+            0.01535382866859436,
+            0.01316749956458807,
+            0.0027806703001260757,
+            0.028811423107981682,
+            0.011264191009104252,
+            0.021820832043886185,
+            0.025273675099015236,
+            0.011419852264225483,
+            -0.0037252490874379873,
+            0.024962352588772774,
+            -0.008101443760097027,
+            -0.004446949809789658,
+            -0.0007035697344690561,
+            0.03206615149974823,
+            0.014702882617712021,
+            -0.017603836953639984,
+            -0.036962397396564484,
+            -0.007620310410857201,
+            0.009452863596379757,
+            -0.007903330028057098,
+            -0.027127455919981003,
+            -0.03950957581400871,
+            -0.026787832379341125,
+            -0.005667473189532757,
+            0.01505665760487318,
+            -0.06724552065134048,
+            -0.007620310410857201,
+            -0.012233533896505833,
+            0.017717044800519943,
+            0.0022906921803951263,
+            0.0181132722645998,
+            0.001277127186767757,
+            -0.009403335861861706,
+            0.004209920763969421,
+            0.050802066922187805,
+            -0.03059444949030876,
+            0.014242975041270256,
+            0.011292492970824242,
+            0.0032105068676173687,
+            0.002787745790556073,
+            0.038688816130161285,
+            -0.010988246649503708,
+            -0.030424637719988823,
+            0.011575513519346714,
+            -0.017292514443397522,
+            0.003383856499567628,
+            0.05006621778011322,
+            -0.009481165558099747,
+            -0.027495380491018295,
+            0.020915169268846512,
+            -0.009148617275059223,
+            -0.015523640438914299,
+            -0.0027258351910859346,
+            -0.006035399157553911,
+            0.0006283925613388419,
+            -0.03832089155912399,
+            -0.015750056132674217,
+            0.03212275728583336,
+            0.008617955259978771,
+            0.014603826217353344,
+            0.025344429537653923,
+            0.010209942236542702,
+            -0.030481241643428802,
+            0.027481229975819588,
+            -0.0005678086308762431,
+            0.005221716593950987,
+            0.013323160819709301,
+            0.03424540534615517,
+            -0.014264201745390892,
+            -0.020264223217964172,
+            -0.040245428681373596,
+            -0.013853822834789753,
+            -0.0463869571685791,
+            0.001943992800079286,
+            0.006919836159795523,
+            -0.012919857166707516,
+            -0.04366996884346008,
+            -0.0329718142747879,
+            0.021127434447407722,
+            0.03178313374519348,
+            -0.017023645341396332,
+            0.0025294902734458447,
+            -0.008483520708978176,
+            -0.007882103323936462,
+            -0.0017520699184387922,
+            -0.013217028230428696,
+            -0.006378560326993465,
+            -0.006229975260794163,
+            0.015452885068953037,
+            -0.004652139265090227,
+            -0.0036934094969183207,
+            -0.0498398020863533,
+            -0.018778368830680847,
+            0.004082561936229467,
+            -0.008221727795898914,
+            0.007323139347136021,
+            -0.006955213379114866,
+            -0.002727604005485773,
+            -0.012474101036787033,
+            -0.03829259052872658,
+            0.024184048175811768,
+            0.017278363928198814,
+            0.0035501306410878897,
+            -0.008922201581299305,
+            -0.018367990851402283,
+            0.03294351324439049,
+            0.012523629702627659,
+            -0.017547233030200005,
+            -0.018580254167318344,
+            0.025981225073337555,
+            0.026957644149661064,
+            0.008122670464217663,
+            -0.006137993652373552,
+            0.029886899515986443,
+            0.005695775151252747,
+            -0.022938761860132217,
+            -0.003983504604548216,
+            -0.005055442452430725,
+            -0.0038844477385282516,
+            0.020561393350362778,
+            -0.010556641966104507,
+            0.0300001073628664,
+            0.038717120885849,
+            0.012870329432189465,
+            0.0039198254235088825,
+            -0.0033714743331074715,
+            -0.008922201581299305,
+            0.015976471826434135,
+            0.006481155287474394,
+            0.008525974117219448,
+            0.014759486541152,
+            -0.016358548775315285,
+            -0.020405733957886696,
+            0.006215824279934168,
+            -0.007107336539775133,
+            0.020122714340686798,
+            0.01708024926483631,
+            0.016429303213953972,
+            0.0007610581233166158,
+            -0.028287837281823158,
+            -0.0018024828750640154,
+            0.02411329373717308,
+            0.02010856196284294,
+            -0.004655676893889904,
+            -0.023264234885573387,
+            0.014398636296391487,
+            -0.015806660056114197,
+            -0.009969375096261501,
+            0.0033431723713874817,
+            0.006229975260794163,
+            -0.003083147807046771,
+            0.0571134127676487,
+            -0.020193468779325485,
+            -0.011419852264225483,
+            -0.010287772864103317,
+            -0.010867963545024395,
+            0.0012072566896677017,
+            -0.01828308403491974,
+            0.008964654989540577,
+            0.025613298639655113,
+            -0.009764186106622219,
+            0.03568880632519722,
+            0.0228114016354084,
+            0.006912760436534882,
+            0.005260631907731295,
+            -0.027070851996541023,
+            0.012304289266467094,
+            -0.016047226265072823,
+            0.005660397466272116,
+            0.018467046320438385,
+            0.017320817336440086,
+            0.0076132346875965595,
+            0.026306698098778725,
+            0.019386861473321915,
+            0.016684021800756454,
+            -0.022783100605010986,
+            0.025457637384533882,
+            -0.007832575589418411,
+            0.027877457439899445,
+            -0.01611798256635666,
+            -0.002253545681014657,
+            -0.012990612536668777,
+            0.008681634441018105,
+            0.002416282193735242,
+            -0.0019351483788341284,
+            0.00036129256477579474,
+            0.016783079132437706,
+            -0.014589674770832062,
+            0.01062032114714384,
+            0.013316085562109947,
+            -0.027919910848140717,
+            -0.010372678749263287,
+            0.016344398260116577,
+            -0.00668988237157464,
+            0.006877383217215538,
+            -0.011865608394145966,
+            -0.0017202302115038037,
+            0.011009473353624344,
+            -0.0436982698738575,
+            0.0104929618537426,
+            0.006279503460973501,
+            -0.015155714936554432,
+            0.014801939949393272,
+            -0.009799563325941563,
+            -0.008235878311097622,
+            -0.013592029921710491,
+            0.003792466362938285,
+            -0.00030225637601688504,
+            -0.0018290160223841667,
+            -0.00022995365725364536,
+            0.00389859895221889,
+            0.010047205723822117,
+            -0.0045990729704499245,
+            -0.010450509376823902,
+            -0.0028638073708862066,
+            -0.02265574224293232,
+            -0.0010047205723822117,
+            0.018735915422439575,
+            -0.004591997712850571,
+            0.0005222601466812193,
+            0.0145189194008708,
+            -0.005830209702253342,
+            0.010556641966104507,
+            -0.005221716593950987,
+            -0.008320784196257591,
+            -0.010768906213343143,
+            -0.033764272928237915,
+            0.0026710000820457935,
+            -0.015254771336913109,
+            0.008865597657859325,
+            -0.012544856406748295,
+            0.04027372971177101,
+            0.01022409275174141,
+            -0.00837738811969757,
+            -0.002796590095385909,
+            -0.0228114016354084,
+            -0.004475251771509647,
+            -0.016938740387558937,
+            -0.01184438169002533,
+            0.008342010900378227,
+            0.012629762291908264,
+            0.005614406894892454,
+            -0.018410442396998405,
+            0.003665107535198331,
+            0.0030955299735069275,
+            -0.005575491581112146,
+            -0.008370312862098217,
+            0.010853812098503113,
+            -0.0006248548161238432,
+            -0.01800006441771984,
+            -0.01957082375884056,
+            -0.027424626052379608,
+            0.0150000536814332,
+            0.014589674770832062,
+            -0.01011088490486145,
+            -0.022754797711968422,
+            -0.007471724878996611,
+            0.007107336539775133,
+            -0.02064630016684532,
+            0.0006779210525564849,
+            0.006891534198075533,
+            0.008717012591660023,
+            0.004117939155548811,
+            0.009636826813220978,
+            0.0007699025445617735,
+            0.0121698547154665,
+            0.018566103652119637,
+            0.03271709755063057,
+            -0.0396793857216835,
+            -0.009537769481539726,
+            0.0057028504088521,
+            -0.001465512323193252,
+            -0.013853822834789753,
+            0.009672204032540321,
+            0.0024375086650252342,
+            -0.0026426981203258038,
+            0.024424616247415543,
+            -0.008044839836657047,
+            -0.01505665760487318,
+            0.01184438169002533,
+            0.00887267291545868,
+            -0.019075538963079453,
+            0.02746707946062088,
+            0.006169833242893219,
+            -0.01011088490486145,
+            0.02259913645684719,
+            -0.012360893189907074,
+            0.012658064253628254,
+            -0.005522425286471844,
+            0.023660462349653244,
+            -0.019004784524440765,
+            0.0006306036957539618,
+            0.015665151178836823,
+            0.03883032873272896,
+            -0.00437973253428936,
+            -0.004567233379930258,
+            -0.03421710431575775,
+            0.005115584470331669,
+            0.028853876516222954,
+            0.02666047215461731,
+            -0.0012346742441877723,
+            -0.005596718285232782,
+            -0.0019864458590745926,
+            0.010768906213343143,
+            0.011589664034545422,
+            0.017561383545398712,
+            -0.029547275975346565,
+            0.00022011429246049374,
+            -0.016938740387558937,
+            0.025500090792775154,
+            0.014632128179073334,
+            0.006951675750315189,
+            0.04072656109929085,
+            0.0027789014857262373,
+            0.003841994795948267,
+            0.027452928945422173,
+            -0.00779719790443778,
+            0.003546592779457569,
+            -0.015566092915832996,
+            0.0003175129240844399,
+            0.022853855043649673,
+            0.0016370931407436728,
+            0.0020784272346645594,
+            0.013337312266230583,
+            0.029320860281586647,
+            -0.022471778094768524,
+            -0.023929331451654434,
+            0.003482913365587592,
+            0.007185167167335749,
+            -0.010570792481303215,
+            0.0038738346192985773,
+            0.006275965832173824,
+            -0.006007096730172634,
+            0.008900974877178669,
+            0.006332569755613804,
+            -0.0003166284877806902,
+            0.0039198254235088825,
+            -0.00308845448307693,
+            0.012962310574948788,
+            -0.02702839858829975,
+            0.036849189549684525,
+            0.004439874552190304,
+            0.010075507685542107,
+            0.015268922783434391,
+            0.001627364312298596,
+            0.004917470272630453,
+            0.005869124550372362,
+            0.010188715532422066,
+            0.018636859953403473,
+            -0.00956607237458229,
+            -0.027481229975819588,
+            -0.03166992589831352,
+            0.020363280549645424,
+            -0.01173117384314537,
+            0.0035872769076377153,
+            0.0074646491557359695,
+            -0.01828308403491974,
+            -0.007507102098315954,
+            -0.009976450353860855,
+            0.015764206647872925,
+            -0.03189634159207344,
+            0.012990612536668777,
+            -0.0043231286108493805,
+            -0.028669914230704308,
+            0.00851889792829752,
+            -0.010181640274822712,
+            -0.0045884596183896065,
+            -0.0035094465129077435,
+            -0.029264256358146667,
+            -0.007931631989777088,
+            0.03908504545688629,
+            -0.02141045406460762,
+            0.024184048175811768,
+            0.008257105015218258,
+            -0.00014449491573031992,
+            -0.0040365708991885185,
+            -0.013563727959990501,
+            0.005331386812031269,
+            -0.01778779923915863,
+            0.025457637384533882,
+            0.010294848121702671,
+            -0.013924578204751015,
+            0.0322076641023159,
+            -0.011179285123944283,
+            0.027438776567578316,
+            -0.018580254167318344,
+            0.036198243498802185,
+            0.007422196213155985,
+            0.00674648629501462,
+            0.01974063739180565,
+            -0.004025958012789488,
+            0.008228803053498268,
+            0.04559450224041939,
+            -0.018523650243878365,
+            0.02460857853293419,
+            0.020349130034446716,
+            -0.00022685811563860625,
+            0.017915157601237297,
+            0.0028390432707965374,
+            -0.023165177553892136,
+            0.009495317004621029,
+            -0.01135617308318615,
+            -0.009955224581062794,
+            0.01583496294915676,
+            -0.019316107034683228,
+            -0.018721764907240868,
+            -0.025613298639655113,
+            0.0029416377656161785,
+            -0.003350247861817479,
+            -0.0077618202194571495,
+            -0.0019616815261542797,
+            0.01454722136259079,
+            0.008724087849259377,
+            0.025669902563095093,
+            0.0008326975512318313,
+            0.011971740983426571,
+            0.01511326152831316,
+            -0.00035576484515331686,
+            0.02313687466084957,
+            -0.03229256719350815,
+            0.01246702577918768,
+            -0.004517704714089632,
+            -0.00509789539501071,
+            0.023858575150370598,
+            0.0003747802402358502,
+            -0.0004877670726273209,
+            -0.0024888061452656984,
+            0.0005496777012012899,
+            -0.014589674770832062,
+            -0.0057099261321127415,
+            0.000907432462554425,
+            -0.005175726022571325,
+            0.04299071803689003,
+            -0.00701535539701581,
+            -0.004772422835230827,
+            -0.039169952273368835,
+            -0.01918874867260456,
+            0.00623351288959384,
+            -0.01583496294915676,
+            -0.018778368830680847,
+            -0.006307805422693491,
+            0.005801907740533352,
+            -0.04893413558602333,
+            0.008342010900378227,
+            0.03336804360151291,
+            -0.012219383381307125,
+            0.00259493850171566,
+            -0.010981171391904354,
+            -0.006969364359974861,
+            -0.0029115669894963503,
+            -7.766463386360556e-05,
+            -0.002412744564935565,
+            -0.020830262452363968,
+            -0.005617944523692131,
+            0.003364398842677474,
+            -0.0008795727044343948,
+            -0.025047259405255318,
+            0.0006177793256938457,
+            0.018849123269319534,
+            -0.003979966975748539,
+            -0.0011152751976624131,
+            -0.001825478277169168,
+            0.006399787031114101,
+            -0.01508495956659317,
+            0.008009462617337704,
+            -0.0076344613917171955,
+            0.003841994795948267,
+            -0.0030495391692966223,
+            -0.002214630600064993,
+            0.02844349853694439,
+            0.021594416350126266,
+            -0.04808507859706879,
+            -0.01405901275575161,
+            0.023009516298770905,
+            0.007280686404556036,
+            -0.004740582779049873,
+            0.01178777776658535,
+            -0.031613320112228394,
+            -0.00028943203506059945,
+            -0.026787832379341125,
+            0.02200479619204998,
+            -0.009205221198499203,
+            -0.007386818993836641,
+            0.016967041417956352,
+            -0.026632171124219894,
+            0.016344398260116577,
+            0.00794578343629837,
+            -0.00030181417241692543,
+            -0.000493515923153609,
+            0.0063856360502541065,
+            0.0008972614887170494,
+            -0.01659911684691906,
+            0.006442239973694086,
+            -0.0048502529971301556,
+            0.007959933951497078,
+            -0.005087282508611679,
+            -0.05527378246188164,
+            -0.009933997876942158,
+            -0.005497660953551531,
+            -0.008936353027820587,
+            -0.012820800766348839,
+            0.008136821910738945,
+            0.02827368676662445,
+            0.01606137864291668,
+            -0.009863242506980896,
+            -4.8312376748071983e-05,
+            0.033000119030475616,
+            -0.0008300442132167518,
+            0.009955224581062794,
+            -0.011349096894264221,
+            0.00015709814033471048,
+            -0.0006911876262165606,
+            -0.014660430140793324,
+            -0.019061388447880745,
+            0.024042539298534393,
+            -0.028344441205263138,
+            0.02378782071173191,
+            -0.01843874529004097,
+            -0.0130825936794281,
+            0.01757553406059742,
+            -0.032575588673353195,
+            -0.005108508747071028,
+            0.007450498174875975,
+            0.0090566361322999,
+            0.021608568727970123,
+            0.0032794929575175047,
+            0.020589696243405342,
+            -0.03716050833463669,
+            -0.009134466759860516,
+            -0.010160413570702076,
+            -0.00794578343629837,
+            -0.010231168940663338,
+            -0.011596739292144775,
+            0.021863285452127457,
+            -0.015240620821714401,
+            -0.016952890902757645,
+            0.007054270710796118,
+            0.0030212372075766325,
+            -0.011313719674944878,
+            0.015750056132674217,
+            0.00017810352437663823,
+            0.004712280817329884,
+            0.013011839240789413,
+            -0.0008923970744945109,
+            0.04086806997656822,
+            0.0253019779920578,
+            -0.0030212372075766325,
+            -0.008568426594138145,
+            0.021438755095005035,
+            0.0004285097820684314,
+            0.0150000536814332,
+            0.009375033900141716,
+            0.023264234885573387,
+            -0.02092931978404522,
+            0.009997677057981491,
+            -0.005501199048012495,
+            0.014603826217353344,
+            -0.009495317004621029,
+            -0.015339677222073078,
+            0.00479718716815114,
+            -0.01577835902571678,
+            0.010153338313102722,
+            -0.010860888287425041,
+            0.008221727795898914,
+            0.01913214474916458,
+            0.002467579673975706,
+            -0.00941041111946106,
+            0.01146230474114418,
+            0.02075950801372528,
+            -0.025641601532697678,
+            0.008851447142660618,
+            -0.0031998935155570507,
+            -0.0021916350815445185,
+            0.021127434447407722,
+            0.010301923379302025,
+            -0.01594817079603672,
+            0.01573590561747551,
+            -0.008879749104380608,
+            -0.0015053119277581573,
+            -0.01882082223892212,
+            -0.03291521221399307,
+            -0.0015283073298633099,
+            0.012396270409226418,
+            0.004800724796950817,
+            0.0150000536814332,
+            0.011264191009104252,
+            -0.0008950503543019295,
+            -0.006601438857614994,
+            0.0008433107868768275,
+            -0.013485897332429886,
+            -0.004963461309671402,
+            0.010231168940663338,
+            0.017051948234438896,
+            -0.0037217114586383104,
+            0.015650998800992966,
+            0.0088160689920187,
+            0.006137993652373552,
+            -0.0010480580385774374,
+            0.0036138100549578667,
+            -0.002060738392174244,
+            -0.007783046457916498,
+            0.019938750192523003,
+            0.019075538963079453,
+            -0.018509499728679657,
+            -0.0024870370980352163,
+            -0.02141045406460762,
+            0.025882167741656303,
+            0.022188758477568626,
+            0.011165134608745575,
+            -0.04222656786441803,
+            0.00012050455552525818,
+            0.00752832880243659,
+            -0.0013744152383878827,
+            -0.031103884801268578,
+            0.008172199130058289,
+            0.005221716593950987,
+            0.004330204334110022,
+            0.020844414830207825,
+            -0.018424594774842262,
+            0.010853812098503113,
+            0.0070861103013157845,
+            -0.0005766529939137399,
+            0.01019579079002142,
+            -0.0073372903279960155,
+            0.008688710629940033,
+            0.038717120885849,
+            -0.018410442396998405,
+            0.005303084850311279,
+            0.0062370505183935165,
+            0.023320838809013367,
+            -0.011228813789784908,
+            -0.00031485961517319083,
+            0.00800238735973835,
+            0.021608568727970123,
+            -0.012934008613228798,
+            0.005883275531232357,
+            0.067075714468956,
+            -0.012912781909108162,
+            0.031613320112228394,
+            0.020533092319965363,
+            -0.004970536567270756,
+            -0.0104929618537426,
+            -0.012445799075067043,
+            -0.0181132722645998,
+            0.021283095702528954,
+            0.01175947580486536,
+            0.003419233951717615,
+            0.01941516436636448,
+            -0.01610383205115795,
+            0.006636816076934338,
+            0.017547233030200005,
+            0.012601460330188274,
+            -0.007379743270576,
+            -0.0036173479165881872,
+            0.019004784524440765,
+            0.018509499728679657,
+            -0.0030849166214466095,
+            0.00019424450874794275,
+            0.01508495956659317,
+            -0.022075550630688667,
+            -0.012530704960227013,
+            0.01644345559179783,
+            -0.0037358624394983053,
+            0.017773648723959923,
+            -0.02227366529405117,
+            0.024198200553655624,
+            -0.004531855694949627,
+            -0.017179306596517563,
+            0.004698129836469889,
+            -0.007514177821576595,
+            0.022033097222447395,
+            -0.001578720286488533,
+            0.004857328720390797,
+            0.000821199850179255,
+            -0.0018015984678640962,
+            0.01349297259002924,
+            0.002826661104336381,
+            -0.0005120891146361828,
+            0.010507113300263882,
+            -0.0009702275274321437,
+            -0.0013496510218828917,
+            0.024509521201252937,
+            1.9374700059415773e-05,
+            0.005989408120512962,
+            0.0032936439383774996,
+            -0.009261825121939182,
+            -0.004429261200129986,
+            -0.004443412180989981,
+            0.0011842612875625491,
+            0.003877372248098254,
+            -0.007029506377875805,
+            -0.02384442463517189,
+            0.022471778094768524,
+            -0.008540124632418156,
+            0.01175947580486536,
+            0.011349096894264221,
+            0.003665107535198331,
+            0.006824316922575235,
+            -0.009771261364221573,
+            -0.017320817336440086,
+            0.0006633278680965304,
+            -0.01617458648979664,
+            0.006912760436534882,
+            -0.001656550681218505,
+            0.0020235921256244183,
+            0.009445788338780403,
+            -0.002570174168795347,
+            0.0015123875346034765,
+            0.011306644417345524,
+            0.011497682891786098,
+            0.003254728624597192,
+            0.004577846731990576,
+            -0.018721764907240868,
+            -0.020915169268846512,
+            0.005288933869451284,
+            0.013924578204751015,
+            0.004273600410670042,
+            0.0015309606678783894,
+            0.017603836953639984,
+            -0.006166295614093542,
+            -0.022967062890529633,
+            -0.010117961093783379,
+            -0.01584911346435547,
+            0.039396367967128754,
+            0.01800006441771984,
+            -0.02525952458381653,
+            -0.000530220044311136,
+            -0.009728807955980301,
+            0.010450509376823902,
+            -0.00404364662244916,
+            0.014575524255633354,
+            0.005476434715092182,
+            -0.02426895499229431,
+            -0.016797229647636414,
+            -0.010662773624062538,
+            -0.005257094278931618,
+            0.020080260932445526,
+            0.027014248073101044,
+            -0.004956385586410761,
+            0.0061026159673929214,
+            0.019584976136684418,
+            -0.010938717983663082,
+            0.009849091991782188,
+            0.020957622677087784,
+            0.0031698227394372225,
+            -0.007570781745016575,
+            -0.004553082399070263,
+            -0.016514210030436516,
+            0.024467067793011665,
+            0.014250051230192184,
+            -0.005104971118271351,
+            -0.02465103194117546,
+            0.004110863897949457,
+            -0.02428310550749302,
+            -0.01616043597459793,
+            -0.007301913108676672,
+            -0.014143918640911579,
+            -0.004931621253490448,
+            0.0027594438288360834,
+            -0.004836102016270161,
+            0.011278342455625534,
+            -0.0031592093873769045,
+            -0.004531855694949627,
+            -0.0041851564310491085,
+            -0.025584997609257698,
+            -0.02524537406861782,
+            0.007691065315157175,
+            -0.003005317412316799,
+            0.005596718285232782,
+            0.000650945701636374,
+            -0.013400991447269917,
+            -0.011589664034545422,
+            0.00970758218318224,
+            -0.007988235913217068,
+            0.012813725508749485,
+            0.0031149876303970814,
+            -0.009183995425701141,
+            0.0017193458043038845,
+            -0.0012311364989727736,
+            -0.0059752571396529675,
+            0.019924599677324295,
+            0.00916984397917986,
+            0.011724098585546017,
+            -0.03118879161775112,
+            -0.0021633331198245287,
+            0.008349086157977581,
+            0.01828308403491974,
+            -0.02244347706437111,
+            -0.008313708938658237,
+            0.005179263651371002,
+            -0.011228813789784908,
+            -0.013507124036550522,
+            -0.028952933847904205,
+            -0.0030460015404969454,
+            8.175515540642664e-05,
+            -0.018636859953403473,
+            -0.029886899515986443,
+            0.0009038947173394263,
+            -0.007860877551138401,
+            0.016882136464118958,
+            -0.006643891800194979,
+            0.01790100708603859,
+            0.00870286114513874,
+            0.0036173479165881872,
+            0.011554286815226078,
+            0.0025082635693252087,
+            -0.013266556896269321,
+            0.008426916785538197,
+            -0.02671707607805729,
+            -0.01516986545175314,
+            0.009792488068342209,
+            0.014214673079550266,
+            -0.0026126273442059755,
+            5.074458022136241e-05,
+            0.0058372849598526955,
+            0.024254804477095604,
+            0.013584954664111137,
+            0.012474101036787033,
+            0.003042463678866625,
+            -0.005303084850311279,
+            0.005568416323512793,
+            0.0227123461663723,
+            0.008603804744780064,
+            0.004011806566268206,
+            0.007592008449137211,
+            0.00389859895221889,
+            0.0009932229295372963,
+            -0.005943417549133301,
+            -0.00843399204313755,
+            -0.021537812426686287,
+            -0.008908051066100597,
+            0.019938750192523003,
+            -0.010422207415103912,
+            -0.03005671128630638,
+            0.0006023016758263111,
+            -0.0032777239102870226,
+            0.007988235913217068,
+            0.02595292218029499,
+            -0.014172220602631569,
+            0.004970536567270756,
+            0.005536576267331839,
+            -0.007910405285656452,
+            -0.00614860700443387,
+            0.022967062890529633,
+            0.02639160305261612,
+            -0.0034404604230076075,
+            -0.04800017178058624,
+            -0.01671232469379902,
+            0.004305440001189709,
+            -0.007768895477056503,
+            -0.034047290682792664,
+            -0.015424583107233047,
+            0.002757675014436245,
+            -0.03206615149974823,
+            0.010068432427942753,
+            -0.012948159128427505,
+            -0.01214155275374651,
+            0.020264223217964172,
+            0.03223596513271332,
+            0.005154499784111977,
+            0.00014847487909719348,
+            -0.0214670579880476,
+            -0.0021686397958546877,
+            -0.021382151171565056,
+            0.010174565017223358,
+            -0.008801918476819992,
+            -0.010584943927824497,
+            0.0059611061587929726,
+            -0.012191081419587135,
+            0.013266556896269321,
+            0.010797208175063133,
+            0.024778390303254128,
+            0.01584911346435547,
+            -0.01876421831548214,
+            -0.0024746551644057035,
+            -0.01108730398118496,
+            -0.0016450530383735895,
+            -0.009820790030062199,
+            0.006360871717333794,
+            -0.0031592093873769045,
+            0.011264191009104252,
+            0.015184016898274422,
+            0.004432798828929663,
+            0.00046786724124103785,
+            0.038151077926158905,
+            -2.4487851987942122e-05,
+            -0.0003418349369894713,
+            0.006803090218454599,
+            -0.015750056132674217,
+            0.008837295696139336,
+            0.012191081419587135,
+            0.005051904823631048,
+            -0.007698140572756529,
+            0.006739411037415266,
+            0.03099067695438862,
+            -0.01516986545175314,
+            -0.009014182724058628,
+            -0.023504801094532013,
+            0.007549555506557226,
+            -0.0004656561359297484,
+            0.029971804469823837,
+            0.020533092319965363,
+            -0.027509532868862152,
+            -0.016018925234675407,
+            -0.02179253101348877,
+            0.01267929095774889,
+            -0.011497682891786098,
+            -0.003704022616147995,
+            0.012240610085427761,
+            0.02367461286485195,
+            0.0018838511314243078,
+            -0.014455240219831467,
+            -0.008950503543019295,
+            0.008943428285419941,
+            0.014617976732552052,
+            0.003497064346447587,
+            0.007570781745016575,
+            0.00743634719401598,
+            0.008886824361979961,
+            -0.014801939949393272,
+            -0.012254760600626469,
+            0.00767691433429718,
+            -0.010202866978943348,
+            0.01794346049427986,
+            -0.009403335861861706,
+            0.016768928617239,
+            -0.004645063541829586,
+            -0.011964665725827217,
+            -0.0320095494389534,
+            -0.002292460994794965,
+            0.005037753842771053,
+            0.004627374932169914,
+            0.0034705311991274357,
+            -0.0022217060904949903,
+            -0.013160424306988716,
+            0.004673365969210863,
+            0.005073131527751684,
+            1.875283123808913e-05,
+            -0.008313708938658237,
+            -0.00479718716815114,
+            0.004252373706549406,
+            -0.001031253719702363,
+            -0.0073372903279960155,
+            -0.0059858704917132854,
+            0.018481198698282242,
+            0.0066509670577943325,
+            0.00800238735973835,
+            0.03234917297959328,
+            0.02974538877606392,
+            -0.0016353242099285126,
+            -0.004606148693710566,
+            -0.00023304918431676924,
+            -0.01589156687259674,
+            -0.0057205394841730595,
+            -0.023320838809013367,
+            0.022259512916207314,
+            -0.009665128774940968,
+            0.021820832043886185,
+            0.019457615911960602,
+            0.0039304387755692005,
+            -0.008391539566218853,
+            -0.0011586126638576388,
+            -0.007308988366276026,
+            -0.003560743760317564,
+            0.016641568392515182,
+            0.006742948666214943,
+            0.01226891204714775,
+            0.00030933189555071294,
+            -0.028401045128703117,
+            0.026405755430459976,
+            0.000579748535528779,
+            0.0034015451092272997,
+            0.008908051066100597,
+            0.014129767194390297,
+            0.013209952972829342,
+            -0.01235381793230772,
+            0.01768874190747738,
+            -0.03280200436711311,
+            -0.0033484790474176407,
+            -0.016500059515237808,
+            0.002076658420264721,
+            -0.0018520113080739975,
+            -0.0008145665633492172,
+            -0.05354736000299454,
+            0.0003842879377771169,
+            0.005313698202371597,
+            -0.015339677222073078,
+            0.029349161311984062,
+            -0.020589696243405342,
+            -0.017165156081318855,
+            -0.032858606427907944,
+            -0.010238244198262691,
+            0.018099121749401093,
+            0.005617944523692131,
+            0.00035178486723452806,
+            -0.02660386823117733,
+            0.015750056132674217,
+            -0.0047476585023105145,
+            -0.00743634719401598,
+            -0.029858596622943878,
+            -0.010938717983663082,
+            0.009403335861861706,
+            0.004485865123569965,
+            -0.0130825936794281,
+            0.009898620657622814,
+            -0.002890340518206358,
+            0.010867963545024395,
+            -0.0138467475771904,
+            -0.00420284504070878,
+            0.0010020672343671322,
+            -0.005363226868212223,
+            0.012615610845386982,
+            -0.030537845566868782,
+            0.011391550302505493,
+            -0.01352127455174923,
+            0.0008548084879294038,
+            -0.017858553677797318,
+            -0.027707645669579506,
+            0.0020288988016545773,
+            -0.016132133081555367,
+            0.039990708231925964,
+            0.010188715532422066,
+            -0.01481609046459198,
+            0.016995344310998917,
+            -0.002080196049064398,
+            0.017377421259880066,
+            0.0014955832157284021,
+            0.017023645341396332,
+            -0.011971740983426571,
+            0.021056678146123886,
+            -0.0041745430789887905,
+            0.02156611531972885,
+            0.00600002147257328,
+            0.034047290682792664,
+            -0.02033497765660286,
+            0.0070825726725161076,
+            0.003792466362938285,
+            0.005798369646072388,
+            -0.0026816134341061115,
+            0.021424604579806328,
+            0.017985913902521133,
+            0.019698183983564377,
+            0.0037110981065779924,
+            -0.024084990844130516,
+            -0.006137993652373552,
+            0.004867942072451115,
+            0.023858575150370598,
+            0.011150983162224293,
+            -0.016415152698755264,
+            -0.021056678146123886,
+            0.015226469375193119,
+            -0.007860877551138401,
+            0.006771250627934933,
+            -0.00851889792829752,
+            0.00037544354563578963,
+            -0.013259481638669968,
+            0.012233533896505833,
+            -0.012523629702627659,
+            0.0032335021533071995,
+            0.01043635793030262,
+            -0.027806703001260757,
+            0.01511326152831316,
+            -0.00371817359700799,
+            -0.014313730411231518,
+            0.023320838809013367,
+            -0.015877414494752884,
+            -0.013988257385790348,
+            0.005501199048012495,
+            0.008610880002379417,
+            0.01746232621371746,
+            -0.002761212643235922,
+            0.002669231267645955,
+            -0.020504789426922798,
+            -0.012445799075067043,
+            0.004591997712850571,
+            0.02210385352373123,
+            -0.015820810571312904,
+            0.012431648559868336,
+            -0.018099121749401093,
+            0.01364863384515047,
+            0.016202887520194054,
+            0.007149789482355118,
+            0.00573115237057209,
+            -0.007726442534476519,
+            -0.0011471149045974016,
+            0.032151058316230774,
+            -0.0029876285698264837,
+            0.02704254910349846,
+            -0.014915147796273232,
+            0.009615600109100342,
+            0.00518987700343132,
+            -0.013089669868350029,
+            -0.014589674770832062,
+            0.011108530685305595,
+            0.02428310550749302,
+            -0.009530694223940372,
+            -0.023660462349653244,
+            0.0018484735628589988,
+            0.014264201745390892,
+            0.010337301529943943,
+            0.009403335861861706,
+            -0.001997058978304267,
+            -0.0037358624394983053,
+            0.01484439242631197,
+            -0.0032918748911470175,
+            -0.0286557637155056,
+            -0.01633024774491787,
+            -0.004400959238409996,
+            -0.017490629106760025,
+            0.020745357498526573,
+            0.02367461286485195,
+            0.010507113300263882,
+            0.0037570889107882977,
+            0.005540114361792803,
+            0.018297234550118446,
+            0.01843874529004097,
+            0.013330236077308655,
+            0.009304278530180454,
+            0.012700516730546951,
+            -0.004319590982049704,
+            0.02800481766462326,
+            -0.010860888287425041,
+            -0.01046465989202261,
+            -0.015707602724432945,
+            0.0016636261716485023,
+            -0.005193414632230997,
+            -0.012431648559868336,
+            -0.0069481381215155125,
+            0.004369119182229042,
+            0.013330236077308655,
+            -0.012240610085427761,
+            -0.007149789482355118,
+            0.01016748882830143,
+            0.011971740983426571,
+            0.01403071079403162,
+            0.01719345711171627,
+            -0.01633024774491787,
+            -0.011278342455625534,
+            0.0009834941010922194,
+            -0.02844349853694439,
+            -0.0160047747194767,
+            -0.01767459139227867,
+            0.009580222889780998,
+            -0.014469391666352749,
+            0.03724541515111923,
+            0.028471799567341805,
+            0.0073514413088560104,
+            0.031613320112228394,
+            -0.028401045128703117,
+            0.00970758218318224,
+            -0.014398636296391487,
+            0.007132100872695446,
+            -0.004188694059848785,
+            0.003902136581018567,
+            0.002448122017085552,
+            0.020419884473085403,
+            0.00791748147457838,
+            0.014264201745390892,
+            0.008150972425937653,
+            -0.021934039890766144,
+            -0.00662620272487402,
+            -0.012559006921947002,
+            -0.016401002183556557,
+            -0.02595292218029499,
+            0.013719388283789158,
+            0.005087282508611679,
+            -0.003074303502216935,
+            -0.0002704166399780661,
+            0.006566061172634363,
+            -0.0013195802457630634,
+            0.010443433187901974,
+            -0.027608588337898254,
+            -0.01311089564114809,
+            0.0069269114173948765,
+            0.01449061743915081,
+            -0.02114158496260643,
+            0.0014637433923780918,
+            0.008900974877178669,
+            -0.004602610599249601,
+            0.035660505294799805,
+            -0.014674580655992031,
+            -0.008610880002379417,
+            0.0027806703001260757,
+            0.005062518175691366,
+            -0.017660440877079964,
+            -0.02319347858428955,
+            0.0070648835971951485,
+            0.011575513519346714,
+            0.010867963545024395,
+            -0.028570856899023056,
+            0.02087271586060524,
+            -0.0054339817725121975,
+            0.03832089155912399,
+            0.00401534466072917,
+            -0.013337312266230583,
+            0.003962278366088867,
+            0.0011860302183777094,
+            -0.012360893189907074,
+            0.002060738392174244,
+            -0.000340508297085762,
+            0.003700484987348318,
+            0.011228813789784908,
+            -0.014801939949393272,
+            -0.016967041417956352,
+            0.016910437494516373,
+            -0.011476456187665462,
+            -0.0035395172890275717,
+            -0.00943871308118105,
+            -0.016669871285557747,
+            0.007308988366276026,
+            0.025047259405255318,
+            0.0041745430789887905,
+            0.001242634141817689,
+            -0.011632117442786694,
+            -0.0038596836384385824,
+            -0.008575502783060074,
+            0.0011718792375177145,
+            -0.0207029040902853,
+            0.027396325021982193,
+            -0.01801421493291855,
+            0.002410975517705083,
+            0.021099131554365158,
+            0.008709936402738094,
+            -0.003944589756429195,
+            0.016316095367074013,
+            0.026363302022218704,
+            0.01205664686858654,
+            0.0031539027113467455,
+            -0.00902833417057991,
+            1.648646139074117e-05,
+            0.014957600273191929,
+            0.0015123875346034765,
+            -0.0015663381200283766,
+            0.0032140444964170456,
+            -0.006583749782294035,
+            -0.001859086798503995,
+            -0.017561383545398712,
+            0.005564878229051828,
+            0.02125479280948639,
+            -0.004132090136408806,
+            0.020476488396525383,
+            0.015509488992393017,
+            0.011625041253864765,
+            0.010726453736424446,
+            -0.0012390963966026902,
+            -0.002150951186195016,
+            -0.0021668709814548492,
+            -0.007393894251435995,
+            -0.018297234550118446,
+            0.006470541935414076,
+            0.0034493047278374434,
+            0.01822648011147976,
+            0.020504789426922798,
+            -0.02308027073740959,
+            0.010146263055503368,
+            0.014985902234911919,
+            -0.013011839240789413,
+            0.0235755555331707,
+            -0.02006610855460167,
+            0.0024605041835457087,
+            0.02156611531972885,
+            0.013337312266230583,
+            -0.014271277002990246,
+            -0.009219372645020485,
+            0.004662752617150545,
+            -0.01543873455375433,
+            0.000872939417604357,
+            -0.01478778850287199,
+            -0.010783057659864426,
+            0.014533070847392082,
+            -0.01490099634975195,
+            0.01229721400886774,
+            -0.006969364359974861,
+            0.0070967236533761024,
+            -0.017165156081318855,
+            0.015014204196631908,
+            0.008632106706500053,
+            0.01757553406059742,
+            0.0021279556676745415,
+            -0.02518877014517784,
+            -0.006290116813033819,
+            0.021169887855648994,
+            0.009509467519819736,
+            -0.02497650496661663,
+            0.019613277167081833,
+            0.0020094411447644234,
+            -0.019528372213244438,
+            -0.022401023656129837,
+            -0.012686366215348244,
+            -0.005055442452430725,
+            -0.017264213413000107,
+            -0.0011276573641225696,
+            -0.024792540818452835,
+            0.003972891718149185,
+            0.009106164798140526,
+            -0.011978816241025925,
+            -0.019273653626441956,
+            -0.0002315014135092497,
+            -0.01040805596858263,
+            0.006219361908733845,
+            -0.01132787112146616,
+            -0.019853845238685608,
+            0.0070436573587358,
+            0.012127402238547802,
+            -0.0027647505048662424,
+            -0.005646246485412121,
+            -0.02562745101749897,
+            -0.006919836159795523,
+            0.020179318264126778,
+            -0.003994117956608534,
+            -0.004974074196070433,
+            -0.01703779585659504,
+            0.01108730398118496,
+            0.011165134608745575,
+            0.020901018753647804,
+            0.010931642726063728,
+            0.005674548447132111,
+            0.013634482398629189,
+            -0.0029416377656161785,
+            -0.007156865205615759,
+            -9.645892532716971e-06,
+            0.004284213297069073,
+            0.0069233737885952,
+            0.0016636261716485023,
+            0.020235922187566757,
+            -0.01692458800971508,
+            0.018934030085802078,
+            -0.012290137819945812,
+            0.008794843219220638,
+            -0.01855195313692093,
+            -0.0003155229496769607,
+            -0.01067692507058382,
+            0.00015002264990471303,
+            -0.0038455326575785875,
+            -0.009544845670461655,
+            1.9595809135353193e-05,
+            0.00143986358307302,
+            0.008511822670698166,
+            -0.016783079132437706,
+            0.009615600109100342,
+            -0.010839661583304405,
+            -0.004308977629989386,
+            -0.019542522728443146,
+            0.017448175698518753,
+            0.013344387523829937,
+            0.007308988366276026,
+            -0.008278331719338894,
+            -0.006300730165094137,
+            -0.02282555401325226,
+            0.008582578040659428,
+            -0.0014637433923780918,
+            0.002214630600064993,
+            -0.028301987797021866,
+            -0.004358506295830011,
+            0.0024587351363152266,
+            0.020306676626205444,
+            0.0033661676570773125,
+            -0.010712302289903164,
+            0.008978805504739285,
+            -0.023377442732453346,
+            -0.0030159305315464735,
+            -0.023349139839410782,
+            -0.004075486212968826,
+            -0.005112046375870705,
+            0.010103809647262096,
+            0.008101443760097027,
+            0.01549533847719431,
+            -0.009898620657622814,
+            0.01478778850287199,
+            -0.004793649073690176,
+            0.012049571610987186,
+            -0.0012152165872976184,
+            0.01337976474314928,
+            -0.0003111007681582123,
+            -0.01505665760487318,
+            0.02158026583492756,
+            -0.008738238364458084,
+            0.001741456682793796,
+            -0.0052853962406516075,
+            0.0027346794959157705,
+            -0.017702894285321236,
+            -0.004553082399070263,
+            -0.014455240219831467,
+            0.01957082375884056,
+            -0.022584985941648483,
+            0.004153316840529442,
+            -0.005055442452430725,
+            -0.016075529158115387,
+            0.029434068128466606,
+            0.022896308451890945,
+            -0.0035483618266880512,
+            0.015070809051394463,
+            0.0017989451298490167,
+            0.0015061964513733983,
+            0.016202887520194054,
+            -0.015806660056114197,
+            -0.013025989755988121,
+            0.035066161304712296,
+            0.0176321379840374,
+            0.0004313841927796602,
+            -0.0023260696325451136,
+            -0.0037323245778679848,
+            0.0032193511724472046,
+            0.008023613132536411,
+            -0.010450509376823902,
+            -0.01141277700662613,
+            -0.017051948234438896,
+            0.009997677057981491,
+            -0.006155682262033224,
+            0.004446949809789658,
+            -0.00025891896802932024,
+            -7.617214578203857e-05,
+            -0.01516986545175314,
+            0.022584985941648483,
+            -0.017717044800519943,
+            -0.00602832343429327,
+            0.004687516950070858,
+            0.0207029040902853,
+            -0.02541518583893776,
+            0.01616043597459793,
+            -0.007107336539775133,
+            -0.013429293408989906,
+            0.012573158368468285,
+            -0.015127412974834442,
+            -0.0107335289940238,
+            -0.003175129182636738,
+            -0.008009462617337704,
+            -0.022754797711968422,
+            0.017929309979081154,
+            -0.013627407141029835,
+            0.015650998800992966,
+            -0.00287972716614604,
+            0.000562502013053745,
+            0.002057200763374567,
+            0.0032582662533968687,
+            0.02250008098781109,
+            0.030735958367586136,
+            0.010415131226181984,
+            -0.005465821363031864,
+            -0.021000074222683907,
+            0.012983537279069424,
+            0.0020589695777744055,
+            0.013202877715229988,
+            -0.002253545681014657,
+            -0.002135031158104539,
+            -0.016613267362117767,
+            -0.023122724145650864,
+            -0.0013010069960728288,
+            -0.01761798746883869,
+            -0.02552839368581772,
+            -0.01774534583091736,
+            -0.007047194987535477,
+            0.011349096894264221,
+            0.020405733957886696,
+            0.0009755341452546418,
+            -0.005384453106671572,
+            0.01559439580887556,
+            -0.006941062398254871,
+            0.002080196049064398,
+            -0.006594363134354353,
+            -0.002032436430454254,
+            -0.008200501091778278,
+            -0.021863285452127457,
+            0.029660483822226524,
+            -0.010103809647262096,
+            0.01141277700662613,
+            -0.005472897086292505,
+            -0.00965097825974226,
+            0.005575491581112146,
+            0.00900003220885992,
+            0.024849144741892815,
+            0.006452853325754404,
+            0.03141520544886589,
+            0.020151015371084213,
+            -0.018891576677560806,
+            -0.0006071660900488496,
+            0.022684043273329735,
+            -0.0037570889107882977,
+            -0.004889168310910463,
+            -0.0031786670442670584,
+            -0.017985913902521133,
+            -0.012007118202745914,
+            0.007839650847017765,
+            -0.013471745885908604,
+            -0.006636816076934338,
+            -0.00837738811969757,
+            0.018070818856358528,
+            -0.017773648723959923,
+            0.025698205456137657,
+            -0.006488230545073748,
+            0.005154499784111977,
+            0.031924642622470856,
+            0.014363259077072144,
+            0.019698183983564377,
+            -0.011794853024184704,
+            -0.031075581908226013,
+            -0.00683846790343523,
+            -0.007882103323936462,
+            0.009558996185660362,
+            -0.01062032114714384,
+            0.005204027984291315,
+            -0.02006610855460167,
+            -0.018948180601000786,
+            0.012863253243267536,
+            -0.017476476728916168,
+            0.013924578204751015,
+            -0.009955224581062794,
+            0.027226513251662254,
+            -0.001228483160957694,
+            0.0121698547154665,
+            -0.002614396158605814,
+            -0.017801949754357338,
+            0.01205664686858654,
+            0.004089637193828821,
+            -0.023943481966853142,
+            0.010337301529943943,
+            -0.001864393474534154,
+            -0.014037786051630974,
+            -0.018410442396998405,
+            -0.001228483160957694,
+            -0.005200490355491638,
+            -0.009658053517341614,
+            0.025457637384533882,
+            0.014101465232670307,
+            0.016231190413236618,
+            0.0035359796602278948,
+            0.0016335553955286741,
+            -0.011073152534663677,
+            0.009573147632181644,
+            -0.006803090218454599,
+            0.000612030504271388,
+            -0.0034404604230076075,
+            0.011893910355865955,
+            -0.004252373706549406,
+            -0.01543873455375433,
+            0.002324300818145275,
+            0.008561351336538792,
+            0.0143278818577528,
+            5.947839599684812e-05,
+            0.009587298147380352,
+            0.0009790719486773014,
+            -0.009764186106622219,
+            -0.00046388726332224905,
+            -0.0060884649865329266,
+            0.0069375247694551945,
+            0.02400008589029312,
+            -0.012516554445028305,
+            -0.01681138016283512,
+            0.014504768885672092,
+            0.004033033270388842,
+            -0.017589686438441277,
+            0.018367990851402283,
+            0.004103788174688816,
+            -0.022358570247888565,
+            0.0035412863362580538,
+            0.010662773624062538,
+            -0.009523618966341019,
+            -0.014469391666352749,
+            -0.0010339069413021207,
+            -0.015509488992393017,
+            0.03130199760198593,
+            0.00025073791039176285,
+            0.016910437494516373,
+            -0.01062032114714384,
+            -0.0025967073161154985,
+            -0.013309009373188019,
+            0.013103820383548737,
+            -0.015750056132674217,
+            -0.007273610681295395,
+            0.00035620704875327647,
+            -0.02298121340572834,
+            0.01741987280547619,
+            0.0013425755314528942,
+            0.00794578343629837,
+            0.003436922561377287,
+            -0.010556641966104507,
+            0.009820790030062199,
+            0.0018237093463540077,
+            0.006668656133115292,
+            -0.014915147796273232,
+            0.023618008941411972,
+            0.006413938011974096,
+            0.011235889047384262,
+            -0.01352127455174923,
+            -0.01546703651547432,
+            0.002253545681014657,
+            0.04134920611977577,
+            -0.01306136790663004,
+            0.0002593612007331103,
+            0.01562269777059555,
+            7.081024523358792e-05,
+            0.004156854469329119,
+            -0.0023755980655550957,
+            -0.0034493047278374434,
+            -0.034585028886795044,
+            0.005370302125811577,
+            -0.0006328147719614208,
+            0.008073141798377037,
+            -0.008992956951260567,
+            -0.005430444143712521,
+            -0.0040684109553694725,
+            -0.008625030517578125,
+            -0.00314859626814723,
+            -0.01478778850287199,
+            0.008363237604498863,
+            -0.014533070847392082,
+            -0.028910480439662933,
+            0.03166992589831352,
+            -0.006887996103614569,
+            -0.00533846253529191,
+            -0.021396303549408913,
+            0.007570781745016575,
+            -0.01370523776859045,
+            0.002663924591615796,
+            0.010967019945383072,
+            0.008865597657859325,
+            0.008738238364458084,
+            0.002762981690466404,
+            0.0009516543359495699,
+            -0.004255911335349083,
+            0.010273621417582035,
+            -0.00835616234689951,
+            -0.0005943417199887335,
+            -0.0014186371117830276,
+            -0.02725481428205967,
+            -0.01405901275575161,
+            -0.010592019185423851,
+            -0.007811348885297775,
+            -0.00435143057256937,
+            0.012608535587787628,
+            0.007174553815275431,
+            -0.0017485321732237935,
+            -0.022938761860132217,
+            -0.014745336025953293,
+            -0.0034103896468877792,
+            0.008349086157977581,
+            -0.01838214136660099,
+            0.01064862310886383,
+            0.0016830838285386562,
+            0.009976450353860855,
+            0.014292503707110882,
+            0.00302831269800663,
+            0.0016742395237088203,
+            0.003458149265497923,
+            0.014886845834553242,
+            0.012601460330188274,
+            -0.005801907740533352,
+            -0.021382151171565056,
+            -0.0024605041835457087,
+            -0.007068421691656113,
+            -0.0034687623847275972,
+            -0.012332591228187084,
+            0.03039633482694626,
+            0.0029805530793964863,
+            0.0176321379840374,
+            0.007832575589418411,
+            0.009573147632181644,
+            0.010507113300263882,
+            0.029377464205026627,
+            0.006049550138413906,
+            -0.0029929352458566427,
+            -0.008603804744780064,
+            -0.02141045406460762,
+            -0.013146273791790009,
+            0.009898620657622814,
+            0.013294858857989311,
+            0.03597182780504227,
+            -0.034896351397037506,
+            0.012162779457867146,
+            -0.011101454496383667,
+            -0.008264180272817612,
+            -0.008533049374818802,
+            0.0044717141427099705,
+            0.010068432427942753,
+            0.014370334334671497,
+            -0.0023402206134051085,
+            -0.01175947580486536,
+            0.009070787578821182,
+            0.023547254502773285,
+            -0.012983537279069424,
+            0.03016991913318634,
+            0.027325568720698357,
+            0.023066120222210884,
+            0.01761798746883869,
+            -0.012537780217826366,
+            0.026462359353899956,
+            -6.484030018327758e-05,
+            0.0006261814851313829,
+            -0.018084971234202385,
+            -0.00587973790243268,
+            -0.004294826649129391,
+            -0.004362043924629688,
+            0.013316085562109947,
+            -0.0025471788831055164,
+            -0.0008468485320918262,
+            0.015099111013114452,
+            -0.0037323245778679848,
+            -3.927453508367762e-05,
+            -0.0059752571396529675,
+            0.015127412974834442,
+            0.0002699744363781065,
+            0.019457615911960602,
+            -0.04270770028233528,
+            0.019726485013961792,
+            -0.021396303549408913,
+            -0.003102605463936925,
+            -0.0020890405867248774,
+            0.009523618966341019,
+            0.00764861237257719,
+            -0.007142714224755764,
+            -0.014334957115352154,
+            0.004524780437350273,
+            0.014066088013350964,
+            -0.0006726144347339869,
+            0.005080206785351038,
+            -0.0027541371528059244,
+            0.0017228835495188832,
+            -0.010089659132063389,
+            -0.005862049292773008,
+            5.842812606715597e-05,
+            -0.010570792481303215,
+            -0.008646257221698761,
+            -0.03232087194919586,
+            0.01062032114714384,
+            0.010209942236542702,
+            0.007294837385416031,
+            0.004252373706549406,
+            -0.022018946707248688,
+            -0.012318440712988377,
+            0.012948159128427505,
+            -0.01132787112146616,
+            -0.002570174168795347,
+            -0.0020005968399345875,
+            0.005387990735471249,
+            0.0004705205501522869,
+            0.010160413570702076,
+            -0.003472300246357918,
+            0.010938717983663082,
+            -0.014603826217353344,
+            0.018084971234202385,
+            0.03229256719350815,
+            -0.00533846253529191,
+            -0.019967053085565567,
+            0.000907432462554425,
+            -0.009077862836420536,
+            0.003813692834228277,
+            0.007641536649316549,
+            -0.021962342783808708,
+            -0.02173592709004879,
+            0.005087282508611679,
+            -0.013153349049389362,
+            -0.007783046457916498,
+            0.014830241911113262,
+            0.005221716593950987,
+            0.010839661583304405,
+            0.01751893013715744,
+            0.010889190249145031,
+            0.00840569008141756,
+            -0.026533113792538643,
+            0.0006734988419339061,
+            -0.012686366215348244,
+            -0.00590803986415267,
+            0.002239394700154662,
+            -0.0017591454088687897,
+            0.004591997712850571,
+            -0.00010115749319083989,
+            -0.011200511828064919,
+            -0.0007057808106765151,
+            0.010903340764343739,
+            0.027919910848140717,
+            -0.004146241117268801,
+            -0.0022570835426449776,
+            -0.014257126487791538,
+            0.0010100272484123707,
+            0.036481261253356934,
+            -0.012275987304747105,
+            -0.017434025183320045,
+            0.01757553406059742,
+            0.027452928945422173,
+            0.005954030901193619,
+            -0.0032529598101973534,
+            -0.0027364485431462526,
+            0.00039600671152584255,
+            0.011978816241025925,
+            0.007719367276877165,
+            -0.002692226553335786,
+            -0.002131493529304862,
+            0.004761809483170509,
+            0.035773713141679764,
+            -0.0015203474322333932,
+            -0.010330225341022015,
+            -0.024240652099251747,
+            0.010974096134305,
+            0.00277182599529624,
+            0.004754733759909868,
+            0.002313687466084957,
+            0.023504801094532013,
+            0.019655730575323105,
+            0.014858543872833252,
+            0.00470520555973053,
+            -0.009771261364221573,
+            0.03005671128630638,
+            -0.008172199130058289,
+            -0.004542469047009945,
+            -0.02838689461350441,
+            -0.005087282508611679,
+            -0.015650998800992966,
+            0.008929276838898659,
+            -0.0002221042668679729,
+            -0.005016527138650417,
+            -0.013422218151390553,
+            0.019726485013961792,
+            0.01064862310886383,
+            -0.011073152534663677,
+            0.012608535587787628,
+            -0.027070851996541023,
+            0.02346234768629074,
+            -0.03826428949832916,
+            -0.0071993181481957436,
+            -0.006421013735234737,
+            -0.010514188557863235,
+            -0.02627839520573616,
+            -0.0030353881884366274,
+            0.007372668012976646,
+            0.014250051230192184,
+            0.0022570835426449776,
+            0.0022818478755652905,
+            0.02282555401325226,
+            0.0016415152931585908,
+            -0.01508495956659317,
+            0.017985913902521133,
+            0.03492465242743492,
+            0.012926933355629444,
+            0.00155837822239846,
+            0.015990622341632843,
+            0.0027789014857262373,
+            0.0021633331198245287,
+            0.00788917951285839,
+            -0.0034493047278374434,
+            -0.007938707247376442,
+            0.008582578040659428,
+            0.006990591064095497,
+            -0.02763689123094082,
+            0.003824306186288595,
+            -0.00010408719390397891,
+            -0.009898620657622814,
+            -0.022429324686527252,
+            0.011568437330424786,
+            0.01913214474916458,
+            -0.004224071744829416,
+            -0.006771250627934933,
+            -0.0043337419629096985,
+            0.0006836698739789426,
+            0.006785401608794928,
+            -0.002412744564935565,
+            -0.000515184598043561,
+            -0.0015176940942183137,
+            -0.02368876338005066,
+            -0.010316074825823307,
+            0.013570803217589855,
+            -0.005175726022571325,
+            0.010358527302742004,
+            0.005242943298071623,
+            -0.018792519345879555,
+            -0.016938740387558937,
+            0.006852618884295225,
+            -0.016188737004995346,
+            0.01364863384515047,
+            0.00027948213391937315,
+            -0.0023172253277152777,
+            0.024962352588772774,
+            0.03594352304935455,
+            -0.013188726268708706,
+            0.005119122099131346,
+            -0.019386861473321915,
+            -0.0245378240942955,
+            -0.019004784524440765,
+            0.022613288834691048,
+            0.008306633681058884,
+            -0.018792519345879555,
+            0.012304289266467094,
+            -0.002752368338406086,
+            -0.008865597657859325,
+            -0.0066721937619149685,
+            -0.006845543161034584,
+            -0.008547200821340084,
+            0.020533092319965363,
+            -0.004935159347951412,
+            0.0038950610905885696,
+            0.019047237932682037,
+            0.04075486212968826,
+            -0.008292482234537601,
+            -0.01349297259002924,
+            0.0054339817725121975,
+            0.004878554958850145,
+            0.0018626245437189937,
+            -0.004814875777810812,
+            -0.008016537874937057,
+            -0.0185944065451622,
+            -0.01740572229027748,
+            -0.009615600109100342,
+            -0.0020041344687342644,
+            0.020122714340686798,
+            -0.007768895477056503,
+            -0.01354957651346922,
+            -0.0245378240942955,
+            -0.028188779950141907,
+            -0.006580212153494358,
+            0.02319347858428955,
+            0.017179306596517563,
+            -0.00013200224202591926,
+            -4.3337418901501223e-05,
+            -0.012042495422065258,
+            -0.01968403346836567,
+            0.015509488992393017,
+            0.00037876018905080855,
+            -0.01773119531571865,
+            -0.0065448349341750145,
+            -0.015919867902994156,
+            0.005787756759673357,
+            -0.0051368107087910175,
+            0.004574308637529612,
+            0.05057565122842789,
+            0.0005271245026960969,
+            0.008441068232059479,
+            -0.026575567200779915,
+            0.009106164798140526,
+            -0.01583496294915676,
+            -0.017377421259880066,
+            -0.00012702727690339088,
+            -0.022301966324448586,
+            -0.0030955299735069275,
+            -0.018792519345879555,
+            -0.007386818993836641,
+            0.008568426594138145,
+            0.009799563325941563,
+            -0.013896276243031025,
+            -0.013669860549271107,
+            -0.024693485349416733,
+            -0.0035872769076377153,
+            -0.001539805089123547,
+            0.009311353787779808,
+            0.007825499400496483,
+            -0.03693409264087677,
+            0.002536565763875842,
+            -0.0009507699287496507,
+            0.0005779796629212797,
+            0.01208494883030653,
+            -0.008667483925819397,
+            -0.003063690150156617,
+            -0.0214670579880476,
+            0.006452853325754404,
+            0.007103798910975456,
+            -0.013500047847628593,
+            0.012375044636428356,
+            -0.012863253243267536,
+            -0.007669838611036539,
+            -0.0005426021525636315,
+            0.03379257395863533,
+            0.011794853024184704,
+            -0.0025153392925858498,
+            -0.00654129683971405,
+            -0.011150983162224293,
+            0.02654726430773735,
+            -0.011349096894264221,
+            -0.026405755430459976,
+            0.028910480439662933,
+            0.013245330192148685,
+            0.0005974372616037726,
+            0.020151015371084213,
+            -0.0017936384538188577,
+            -0.01397410687059164,
+            0.009601449593901634,
+            0.004266524687409401,
+            0.012424572370946407,
+            -0.00014460546663030982,
+            0.0214670579880476,
+            0.0018555490532889962,
+            0.0027399861719459295,
+            -0.0036226543597877026,
+            0.004687516950070858,
+            -0.03093407303094864,
+            -0.0006549257086589932,
+            -0.003102605463936925,
+            -0.008398614823818207,
+            0.011207587085664272,
+            -0.002421588869765401,
+            0.006558985915035009,
+            -0.022188758477568626,
+            -0.028485950082540512,
+            0.011342021636664867,
+            -0.016231190413236618,
+            -0.021396303549408913,
+            0.015339677222073078,
+            -0.0007641536649316549,
+            0.00614860700443387,
+            -0.007443422917276621,
+            -0.023278385400772095,
+            -0.019386861473321915,
+            0.02064630016684532,
+            0.012212308123707771,
+            -0.01190098561346531,
+            0.012750045396387577,
+            -0.021113283932209015,
+            -0.012948159128427505,
+            -0.001894464367069304,
+            0.0090566361322999,
+            -0.0019864458590745926,
+            0.01325240544974804,
+            -0.012198156677186489,
+            -0.00334494118578732,
+            0.0031592093873769045,
+            -0.011589664034545422,
+            0.008158047683537006,
+            0.01978308893740177,
+            0.008193425834178925,
+            -0.009339655749499798,
+            0.003422771580517292,
+            -0.011922212317585945,
+            0.013500047847628593,
+            0.015537790954113007,
+            0.018311386927962303,
+            -0.006498843897134066,
+            0.0020412809681147337,
+            -0.01370523776859045,
+            0.0063962494023144245,
+            -0.009827865287661552,
+            -0.004878554958850145,
+            -0.0005333155859261751,
+            0.004921008367091417,
+            -0.011179285123944283,
+            -0.0025772498920559883,
+            -0.026830283924937248,
+            -0.0013823752524331212,
+            -0.020533092319965363,
+            0.005303084850311279,
+            0.023207630962133408,
+            -0.007514177821576595,
+            0.0006045127520337701,
+            -0.0039092120714485645,
+            0.014476466923952103,
+            0.020688753575086594,
+            0.00870286114513874,
+            0.00482548912987113,
+            -0.006081389728933573,
+            0.02179253101348877,
+            0.011603815481066704,
+            -0.017816102132201195,
+            0.00164239970035851,
+            -0.001509734196588397,
+            -0.0014336726162582636,
+            0.011625041253864765,
+            0.004443412180989981,
+            -0.020844414830207825,
+            0.0028602697420865297,
+            -0.010089659132063389,
+            0.007818424142897129,
+            0.014702882617712021,
+            -0.00031906069489195943,
+            -0.02486329711973667,
+            -0.009827865287661552,
+            0.021919889375567436,
+            -0.015070809051394463,
+            -0.0008291598060168326,
+            0.00764861237257719,
+            0.023264234885573387,
+            -0.024792540818452835,
+            -0.006831392180174589,
+            -0.0023455272894352674,
+            -0.003933976404368877,
+            -0.005193414632230997,
+            0.0010489424457773566,
+            0.0214670579880476,
+            0.004415110219269991,
+            -0.004234685096889734,
+            0.005724077112972736,
+            0.006470541935414076,
+            0.02325008250772953,
+            0.0040577976033091545,
+            0.028047269210219383,
+            -0.00943871308118105,
+            -0.0023809047415852547,
+            -0.0019988277927041054,
+            -4.775960405822843e-05,
+            0.0160047747194767,
+            -0.005430444143712521,
+            0.020320827141404152,
+            0.009445788338780403,
+            0.0036934094969183207,
+            0.016514210030436516,
+            0.02319347858428955,
+            -0.006767712999135256,
+            -0.004793649073690176,
+            0.019160445779561996,
+            0.0053667644970119,
+            -0.008858522400259972,
+            -0.02194819226861,
+            0.01141277700662613,
+            0.0014124460285529494,
+            -0.0006956097786314785,
+            0.012106175534427166,
+            0.0024375086650252342,
+            0.006300730165094137,
+            0.008172199130058289,
+            -0.016684021800756454,
+            0.004241760354489088,
+            -0.004698129836469889,
+            -0.01860855706036091,
+            0.010910416021943092,
+            -0.012092024087905884,
+            -5.201595922699198e-05,
+            -0.0010002984199672937,
+            0.014165145345032215,
+            -0.01703779585659504,
+            -0.0019103841623291373,
+            -0.0070719593204557896,
+            -1.3024719009990804e-05,
+            -0.0009162768837995827,
+            0.008320784196257591,
+            -0.006254739128053188,
+            -0.010938717983663082,
+            -0.010938717983663082,
+            0.027509532868862152,
+            -0.004991763271391392,
+            0.003916287794709206,
+            0.019330257549881935
+          ],
+          "embedding": [
+            0.007204997251390758,
+            -0.03173324616100502,
+            -0.020505790884701024,
+            0.0037451608231038273,
+            0.009817085391002229,
+            0.026883177166072393,
+            -0.016477826654817607,
+            0.06949311853773875,
+            -0.01224876667472692,
+            0.031952566873802865,
+            0.013633671951171586,
+            -0.005041437355104989,
+            0.02663392750069342,
+            -0.022099132865267587,
+            0.006665702000169926,
+            0.0030467129715919033,
+            0.007931802508415542,
+            0.009984477355413895,
+            -0.010362836464183985,
+            -0.03180848181251013,
+            -0.013542377333474449,
+            -0.013572345022577053,
+            -0.03979985300694044,
+            -0.007698412130292187,
+            -0.005953452234356532,
+            0.018890228917932694,
+            0.02217446994660065,
+            0.005158357501784182,
+            -0.05468694715835175,
+            0.00789994048345594,
+            0.017130184174786147,
+            -0.0026591475669416114,
+            -0.02081141846859738,
+            -0.008403928318906045,
+            -0.030925346724945978,
+            -0.03855023691059801,
+            0.038853354785097736,
+            0.01612011653434478,
+            -0.035635702395196575,
+            0.026266623757525934,
+            0.043450373922427316,
+            -0.007254334835408145,
+            -0.015376804869420366,
+            -0.0017276202867766907,
+            0.004650414207094434,
+            -0.002365514361535186,
+            0.011906139905776183,
+            -0.0005664194302009289,
+            -0.0191723447249157,
+            -0.025546124518037998,
+            0.01025005257280377,
+            -0.014436532079436437,
+            0.029277529346694118,
+            -0.0045414983847733344,
+            -0.02260315456913697,
+            0.01717918656248626,
+            0.013666132349359846,
+            0.057804090189845364,
+            0.056600576093100735,
+            0.051208022172527734,
+            0.03553760896972431,
+            0.02094599510567085,
+            -0.02438674472832137,
+            -0.0027797634437333432,
+            0.03614248430646102,
+            -0.026729308357113124,
+            -0.002251394534452815,
+            0.018929251842788906,
+            0.00689697483318378,
+            -0.026603606434115974,
+            -0.0541443728344337,
+            0.027164630314707807,
+            -0.004335272389193875,
+            -0.013967321193992958,
+            0.003323121377857836,
+            -0.0245295956412367,
+            0.024489335129134787,
+            0.06212059558640156,
+            0.009214058207162371,
+            -0.002082293435905329,
+            0.013659934201796686,
+            -0.003344037927706573,
+            -0.00689041317919184,
+            0.020235785390418023,
+            0.0212739457082623,
+            0.0016782346585238381,
+            -0.013484221664059555,
+            -0.06465582506604173,
+            0.011882797526607326,
+            -0.032970655964890155,
+            -0.04027700439305653,
+            0.025396847912528105,
+            0.013861157517716707,
+            0.00604596157583048,
+            -0.00744611407420936,
+            0.04038084298991919,
+            0.021507532290328625,
+            0.005983822806623904,
+            -0.012711110369915742,
+            0.02410138142963901,
+            -0.018372791517880082,
+            0.021016674128298246,
+            0.013387285956415756,
+            0.02039383525127615,
+            0.010700292312227496,
+            0.0053934660497349735,
+            -0.014807678014449691,
+            0.009422805142740376,
+            0.0414357625490679,
+            0.12206398267002624,
+            -0.010877509834324155,
+            -0.011079302503786522,
+            -0.04107714853756596,
+            0.006631397986165343,
+            -0.031914267104222384,
+            -0.0179965295457466,
+            -0.057530036803978266,
+            -0.0320488164246567,
+            0.008208202363320509,
+            -0.0020203090915064162,
+            0.03502055780919601,
+            0.04464599164916666,
+            0.019332152079287082,
+            0.017133943087431935,
+            -0.039885565008919065,
+            0.04703059988373572,
+            -0.029370696964304416,
+            0.0008494086835971101,
+            0.0014082189719987074,
+            -0.030973762582071615,
+            -0.006561157852024518,
+            0.011634653995510726,
+            -0.019896826887411713,
+            -0.034275921680252924,
+            0.026274465530039866,
+            -0.007728234125236442,
+            0.04814514040690998,
+            0.06550407296280117,
+            -0.02749051703615343,
+            -0.010041099502187074,
+            0.02019601967763537,
+            -0.01871701432558228,
+            0.008532829759864928,
+            -0.020164138072967613,
+            -0.045196925901010855,
+            -0.004523517988516156,
+            -0.002113469149506399,
+            0.07929650663710812,
+            -0.01111269708042164,
+            0.03358254444720164,
+            -0.019973382759721527,
+            -0.0007145592863325417,
+            -0.0011740862928390678,
+            0.02166182480535788,
+            0.004543228023398081,
+            -0.013377688428761516,
+            -0.01758250073647576,
+            0.009548292014596834,
+            -0.04753617258004735,
+            -0.007125744019792479,
+            -0.009930358894299303,
+            -0.007931672044235996,
+            -0.024561936537429774,
+            0.012979195012881556,
+            0.009209145250061479,
+            0.04586756662401364,
+            0.04311807301838248,
+            -0.0036241105681042675,
+            -0.06579256720896123,
+            0.04967191755416882,
+            0.015475886231466817,
+            0.03157651656768371,
+            0.02515941109300714,
+            0.01773554572598468,
+            0.005824418191900522,
+            0.016317635547562007,
+            0.007650872352957098,
+            0.010015166465492963,
+            -0.033203307142867844,
+            0.040272188258443496,
+            -0.013408423788025047,
+            0.01036004294342104,
+            0.012391622448703885,
+            0.04214857862444935,
+            0.03418032388760535,
+            0.015263018079889338,
+            0.018786704580603317,
+            0.02704685343985817,
+            0.015348111464412869,
+            0.0013031600434754704,
+            0.03241427528588285,
+            0.011689924942195664,
+            0.04544406114742012,
+            0.016186615914968734,
+            0.011624500626103958,
+            -0.008272953078864385,
+            -0.02363080189189375,
+            0.02784298119317616,
+            -0.010585980875564109,
+            0.01455013604305903,
+            0.01749228557615982,
+            -0.054742056843690476,
+            -0.005876213780171779,
+            -0.010556605098730201,
+            0.003967456397644932,
+            0.014663888851767762,
+            0.0015548476559947807,
+            -0.003858367677774835,
+            0.012504286824484335,
+            -0.025806882498628832,
+            -0.014216299453397403,
+            0.0031620924516417956,
+            -0.006448390918604307,
+            -0.02785577502346934,
+            -0.00766402687921159,
+            0.005275000635711884,
+            -0.0090448751942,
+            0.019702662630533,
+            0.007009091837467867,
+            -0.01847307100043082,
+            -0.0006774105206839772,
+            0.032952656101827446,
+            0.015900033044760432,
+            -0.02170121261931376,
+            -0.017964879081469612,
+            -0.005062849745084668,
+            -0.03171847232054206,
+            -0.038177398521606544,
+            0.029426291950046455,
+            -0.025687201683643784,
+            0.030158867757101797,
+            0.015780411479509624,
+            -0.007011484049169482,
+            -0.012114023533118393,
+            0.020661407942701628,
+            -0.025067530380881376,
+            0.009777190971413269,
+            -0.004160771288889298,
+            -0.028825369485727344,
+            0.008307290529469729,
+            -0.014415903546305744,
+            -0.007561235629544945,
+            -0.03201704135271719,
+            -0.015182869388110948,
+            -0.00501561450194676,
+            -0.0003863746169672397,
+            -0.01073403445816663,
+            -0.03321103354401307,
+            -0.002304357889348034,
+            0.017001213960120047,
+            0.00028222146729658097,
+            -0.04002882852650754,
+            0.014509588052058002,
+            -0.004150541038999649,
+            -0.034333837380045466,
+            0.0564137596224324,
+            0.03584898691608747,
+            -0.006830698606679202,
+            0.006121456445207273,
+            -0.04505674213649008,
+            0.04358467937307492,
+            0.010847859073780811,
+            -0.04918983713702805,
+            0.049190442935189385,
+            -0.00528960443684769,
+            0.01419195371129309,
+            0.040316024601793406,
+            0.004600843996974165,
+            -0.006780970176765864,
+            0.005206651850992678,
+            0.03397517915308201,
+            -0.010037223444627569,
+            -0.027377913828275806,
+            -0.03328136706915221,
+            0.02562568440972934,
+            -0.03485770844248482,
+            -0.020069979230687374,
+            -0.0052041408068332524,
+            0.029818953377223505,
+            0.028789460147370508,
+            0.004071274130619537,
+            0.01928375754300958,
+            0.0014970947734202088,
+            0.010502599795027616,
+            0.016914145702515263,
+            0.011942576738188502,
+            -0.014909953528043828,
+            -0.003432489584767786,
+            -0.05733768990013957,
+            0.0030710780388681002,
+            -0.02627280695622545,
+            0.007930906480371135,
+            -0.031227917075485834,
+            0.013132797714872345,
+            0.002514351081823354,
+            -0.007152021475569949,
+            -0.0066245205132327055,
+            -0.005279224195782313,
+            0.028512607222275155,
+            -0.00921960825771865,
+            -0.0359704105589433,
+            -0.021347597388353182,
+            0.011200044645812517,
+            -0.0004171294784927543,
+            -0.016784468873218786,
+            -0.0018746544944553878,
+            0.007749103417240419,
+            0.02390272097796833,
+            -0.005128540108845805,
+            0.006397849195755672,
+            0.01214922062611769,
+            0.01163371266250103,
+            0.012307186722130844,
+            -0.004982589662882946,
+            -0.016343724553037307,
+            0.019806887143115643,
+            0.013139657463054087,
+            0.0205801609265216,
+            -0.03767676007629612,
+            -0.0249879011550972,
+            0.05547392586363707,
+            0.005903313549319051,
+            -0.05568806298774435,
+            0.018827693208840233,
+            -0.007931178941307645,
+            0.008183992600768387,
+            -0.010319022946041053,
+            0.05111352384595183,
+            -0.024897203290807753,
+            -0.02096678205188722,
+            0.006436523335212348,
+            -0.04917084523640266,
+            -0.010459284852442706,
+            -0.011138558192296276,
+            0.014789370043397917,
+            -0.03105725978752438,
+            -0.004934745004818259,
+            0.015595801375817122,
+            -4.687585727052446e-05,
+            0.0028886067775750254,
+            -0.00850352807499054,
+            -0.0034885579171087815,
+            0.016339681487281293,
+            -0.0034050317046826813,
+            -0.008309570448487812,
+            -0.003615165868022203,
+            -0.027638409919916757,
+            0.023352910161586186,
+            0.026364511529991978,
+            0.001249936082895606,
+            0.001987948967948376,
+            0.01756629246122769,
+            0.004656075485833687,
+            -0.006752579977941239,
+            -0.013681608638716132,
+            -0.03405936202918687,
+            -0.0401867651078737,
+            0.022991180183813725,
+            -0.02318908140203705,
+            0.018299512436825923,
+            0.00904081847012441,
+            0.019697091729447287,
+            -0.019236130984371835,
+            0.015159917756323988,
+            -0.0387297415849285,
+            -0.018793498297161176,
+            -0.005256288213225733,
+            -0.010414697711744128,
+            0.0007039894599589104,
+            -0.01739645696111503,
+            -0.013728845328097506,
+            0.022709522955937343,
+            -0.022789243441781803,
+            -0.0036763802589970026,
+            -0.01429671717544519,
+            0.0023674126354206932,
+            0.049448298344058644,
+            -0.009926033487593551,
+            -0.023371285193090733,
+            0.020846792916650956,
+            -0.026996519120163966,
+            0.011246908457705283,
+            0.04091365159640201,
+            0.005469162850197613,
+            0.005640919201612091,
+            0.029575753750229278,
+            -0.0007860452837141593,
+            -0.0052805237594885516,
+            -0.009782745443933828,
+            0.021178503581190618,
+            -0.0009176512672832309,
+            -0.05245650420332918,
+            -0.007698364493410058,
+            -0.039027418406703554,
+            0.005448840777995117,
+            0.017581625031764185,
+            0.025522517097715556,
+            -0.020775492472580457,
+            -0.015906320383873178,
+            -0.000643669839811819,
+            -0.0086896084429604,
+            0.0029399233482444638,
+            0.014381240985647907,
+            0.021086693776922936,
+            0.002436575081260609,
+            -0.004913514868101904,
+            -0.011582101006213432,
+            0.014400297663309155,
+            0.0467181039341492,
+            -0.01722692180137712,
+            0.0011622224560709315,
+            -0.013036540368441797,
+            -0.06208454638917413,
+            -0.019160212331432766,
+            0.0351496273293468,
+            0.02429814055116254,
+            -0.009896662169762772,
+            -0.01647101145477158,
+            -0.002915821099802562,
+            -0.03557863734177486,
+            -0.01786433999615293,
+            -0.010699364747311715,
+            0.02390901423299708,
+            0.007213809581960646,
+            0.0005601364043388203,
+            0.00916614601011734,
+            0.014737491634516238,
+            0.019303599465763203,
+            0.008864535607492504,
+            -0.0575839606069681,
+            0.018051837786406608,
+            0.006315830080225406,
+            0.02525619013082567,
+            -0.015671924543113065,
+            0.003988943108210209,
+            -0.03179878281917948,
+            -0.029440641244429775,
+            0.040586322748158446,
+            -0.03418722657593859,
+            -0.030650542761032924,
+            0.006566745611226466,
+            0.009571089937153852,
+            0.004491158558315032,
+            -0.011352813042040581,
+            -0.02535505987721779,
+            -0.013381948772812133,
+            -0.030918676056735445,
+            -0.02192708443081135,
+            -0.005232667170255381,
+            -0.03342408726436631,
+            -0.07498634115550039,
+            -0.006245638452568239,
+            -0.049946896465756174,
+            -0.03900385398225758,
+            0.04458524125677807,
+            -0.006106489377959481,
+            0.02398410968928338,
+            -0.04615183642824994,
+            -0.01165382599292015,
+            -0.04890013373371541,
+            0.013624324077017377,
+            0.01476398305411351,
+            0.020202554298470666,
+            0.00014553149242532853,
+            0.036224316512987846,
+            0.00021498496267774312,
+            -0.05055208658013767,
+            -0.057482169168904106,
+            -0.003960451575823462,
+            -0.010217078509789989,
+            -0.005593738972150311,
+            -0.034090261748104665,
+            0.05835341009022349,
+            0.0116359661127526,
+            0.04203678184748103,
+            0.03447818335186531,
+            -0.006747564131379555,
+            -0.025756962754841885,
+            -0.028123416424745064,
+            -0.008073625379614702,
+            0.047544241458556825,
+            -0.029605671083240564,
+            0.008171081332993893,
+            -0.026007452205474395,
+            0.0067084589781962805,
+            -0.0070645448617940995,
+            -0.0012298101311792947,
+            0.012372286141738867,
+            0.006377319686763927,
+            -0.015024537877377194,
+            0.007934745664790507,
+            0.026041755081605937,
+            -0.007303731324541215,
+            0.0407000170257006,
+            -0.012990703097937777,
+            -0.033189084013863494,
+            -0.022525679964848666,
+            0.01657717698191603,
+            0.03440284885162039,
+            0.010230633738702899,
+            -0.008707808754455952,
+            -0.0363777020540216,
+            0.011184076269109602,
+            0.031774012431843524,
+            0.01504703222643543,
+            -0.022509833303978505,
+            0.025230974050270107,
+            0.017502662694813216,
+            -0.018700579139537456,
+            -0.03511734747709348,
+            0.015162183119327901,
+            -0.017711638132862063,
+            0.018190335050681287,
+            -0.01590174153215615,
+            0.011103110869700727,
+            0.0025189370696524936,
+            -0.00920305384736575,
+            0.01929652151514404,
+            -0.006196715938358087,
+            0.011426611271994685,
+            0.007058186102245829,
+            -0.018191352599117545,
+            0.0027202185072220937,
+            0.057310566886380025,
+            -0.000531639834386418,
+            0.01494975681909879,
+            0.03845692752400366,
+            -0.001971953300066856,
+            0.045554909803324765,
+            -0.02532332999916258,
+            -0.042616337063845953,
+            0.02633963974172818,
+            -0.02970705577683783,
+            -0.01624154998005974,
+            -0.02620993449678864,
+            -0.021490817420853802,
+            -0.004038424045768319,
+            0.009940887982818171,
+            -0.02692933845395842,
+            -0.023801570856681796,
+            -0.001948460540662095,
+            0.02290911497874141,
+            -0.03300532378029689,
+            0.006525488380791647,
+            0.007039306647666766,
+            -0.003685585408710752,
+            0.03183735852538168,
+            -0.027197064244321705,
+            0.006861928286688311,
+            0.011209869849682412,
+            -0.001094897323854524,
+            -0.005328676992017907,
+            -0.004007474089818478,
+            0.010796946231051235,
+            -0.030543173497906445,
+            -0.012879972141706173,
+            0.00935894527375688,
+            0.0017573071057611829,
+            -0.010050882212831222,
+            -0.03102132152986303,
+            -0.002915141291985554,
+            0.007634137769426993,
+            -0.011094093612593696,
+            0.01199609810193786,
+            -0.010458016758814742,
+            -0.04929347607956551,
+            0.010973132159729983,
+            0.028148117094907874,
+            -0.025504019123994758,
+            0.012470127357818321,
+            0.009182729379865986,
+            -0.016020133756226275,
+            0.016320111225714855,
+            -0.012051271446461976,
+            0.024098195821827578,
+            0.04317620443481784,
+            0.024592886889234396,
+            0.007727787037954994,
+            0.00771746776156926,
+            -0.0003147538723468374,
+            0.022572868970938624,
+            -0.016683340264320543,
+            0.013672085891597506,
+            0.020827021901755415,
+            0.02092309219721837,
+            0.0057130785714458055,
+            -0.003901728981013356,
+            -0.0036968415923910314,
+            0.002227024825579113,
+            -0.02106678364798051,
+            0.014578546755457675,
+            0.03128101864424301,
+            -0.005234627629617124,
+            -0.022394744692442837,
+            0.002430617645746389,
+            -0.015780389432069438,
+            4.7277885648917e-05,
+            0.013357803020920043,
+            0.013777651543690907,
+            -0.027713499926132385,
+            -0.0018999112228756628,
+            0.028364630573666852,
+            0.009146793705085586,
+            -0.009891297866910313,
+            -0.01644269527910728,
+            -0.0003903768506971414,
+            0.006512217314869576,
+            0.02408785204739448,
+            0.0018406499545367782,
+            -0.007360473750031044,
+            0.01484497577088404,
+            0.016723277095327158,
+            0.003629911330658209,
+            -0.0004820567356447713,
+            0.02394406451074397,
+            0.023794019615473026,
+            0.001632266615911801,
+            -0.02064379152040184,
+            -0.02755693521684878,
+            0.027335938400284726,
+            0.02769248863677192,
+            -0.006832353240623308,
+            0.027324973360715134,
+            0.003937207909967385,
+            0.013580154030133622,
+            0.02076818768224654,
+            -0.01784151382870951,
+            -0.017617903734311997,
+            0.0017316476451235418,
+            -0.009605294810027804,
+            -0.009829196696575069,
+            0.004891780008779742,
+            -0.011655402383771472,
+            -0.01124379942906224,
+            0.043205454954628145,
+            0.004713701851783307,
+            0.003560551284131962,
+            0.007894840447501912,
+            -0.018881214526749082,
+            -0.00945538175144614,
+            0.0019040259741796817,
+            0.029505863720301916,
+            -0.001400002868254808,
+            0.03315866433970851,
+            -0.00830542045504805,
+            0.008602408964961391,
+            -0.03202191233287069,
+            -0.006978289755114341,
+            0.0326499698728755,
+            0.015314305362524091,
+            0.015104436300213667,
+            0.0021257042489266787,
+            -0.0010840650756005028,
+            0.02434348355847526,
+            -0.007174433778817674,
+            -0.04839373063710409,
+            0.022696994450018916,
+            -0.004779511451775743,
+            0.023588216956749154,
+            0.0382733980620746,
+            -0.004232286553944853,
+            0.003815507826545929,
+            0.023531223314846965,
+            -0.01887650634213847,
+            0.015055403379782985,
+            -0.01918944132635701,
+            -0.017597900721414636,
+            0.02176630010543207,
+            -0.008896608357798277,
+            0.01732387427588561,
+            0.01764363756727683,
+            -0.012180043444768312,
+            -0.019003360499321047,
+            0.01229139850966224,
+            0.0025710091163655626,
+            -0.017075658234265562,
+            0.01797412486853335,
+            0.018372010308966135,
+            0.006723179570342314,
+            0.015417677746651683,
+            -0.00047634684109332134,
+            0.017608296628666153,
+            -0.001049984114818522,
+            -0.008416990841821452,
+            0.01827827907328822,
+            -0.009568717277044265,
+            -0.01658089563620308,
+            0.009808478422454319,
+            -0.05193977747210182,
+            0.010644335931025957,
+            0.0016781339270411392,
+            -0.01562285100294917,
+            0.00405036330963539,
+            0.007102724536527262,
+            0.005195069570261247,
+            0.0028773630901681767,
+            0.004610474981005765,
+            0.042921486858787584,
+            0.013213672397112759,
+            -0.0064030584930089415,
+            -0.010711457508461704,
+            -0.017480309425763254,
+            -0.027151012983864797,
+            -0.005204425551873271,
+            0.009927884476991038,
+            0.022601230152836282,
+            0.008046297278661014,
+            -0.0031858480011267704,
+            -0.02589628103025927,
+            0.019723430185137954,
+            0.03604835658149313,
+            0.062423099201768716,
+            0.010930268386637612,
+            0.02925564269324381,
+            0.02439517070384333,
+            0.023930432096006114,
+            0.013036828091946605,
+            0.006910311087503408,
+            -0.032995841694293995,
+            0.012813650252889403,
+            0.01333714655191133,
+            0.03678966264802025,
+            0.012571138353467183,
+            -0.05610151838497368,
+            3.0777966587107344e-05,
+            0.009791774904454618,
+            0.041959488802722524,
+            0.01956296954085397,
+            0.000155217569370113,
+            -0.007285602454264675,
+            0.031547870838711856,
+            -0.026518385135879206,
+            -0.012299593280363843,
+            -0.0026551378470968594,
+            -0.006190117573235779,
+            -0.003807907001161504,
+            -0.004991957475076049,
+            0.036983035940964804,
+            -0.015648565314743963,
+            -0.019256463332683493,
+            0.0004300362961407231,
+            -0.0015686065121077472,
+            0.0011311524136286674,
+            0.0032236641835276464,
+            0.013795345278976164,
+            0.015385029265915279,
+            -0.004800448639353721,
+            0.026514926384796363,
+            -0.008699453553651681,
+            0.022186817580012775,
+            -0.013427553742493343,
+            -0.011927259975371873,
+            0.017776228476138103,
+            -0.021243797085501972,
+            -0.01667280589187595,
+            -0.003688045116580565,
+            0.006189970874536939,
+            0.0020911968266404467,
+            0.0006097761986763103,
+            -0.004685468039463817,
+            -0.01011357666327168,
+            0.022631082517271542,
+            0.012398000826637277,
+            -0.006721588925828951,
+            0.018142966317821054,
+            -0.0021330638043572654,
+            -0.011869184647489572,
+            0.025073455250141923,
+            -0.0015392945266926425,
+            -0.0337085890598252,
+            0.01267302113143573,
+            -0.013438601165482187,
+            0.02673290895281225,
+            -0.009586762706902736,
+            -0.011208527168019762,
+            0.019144928964011168,
+            -0.0017591316941248897,
+            -0.005459150067129732,
+            -0.029230079356228025,
+            0.004679497471366242,
+            -0.011723195942933286,
+            0.015899377439148873,
+            0.0022054234777254627,
+            0.029187759252864872,
+            -0.018290694895057157,
+            0.00792533824593637,
+            0.011719198071061566,
+            0.014612375997487872,
+            0.005419943379355563,
+            0.010515265723144554,
+            -0.006893314445458116,
+            -0.04424495426064596,
+            5.8241472764594054e-05,
+            -0.005719143565673274,
+            0.0038220021965914754,
+            0.013106940982933836,
+            -0.01203955335201393,
+            0.011260626096963215,
+            -0.0279896016489392,
+            -0.0295228261396555,
+            0.036474811902902415,
+            -0.04900747926861378,
+            0.019261089742811332,
+            0.00014104646541594166,
+            -0.03057197731717424,
+            -0.04331886560073797,
+            -0.03232957159604667,
+            -0.019401402365464362,
+            0.004102283356333861,
+            0.0002610070704924167,
+            0.0012280372705500212,
+            -0.014025012323929194,
+            -0.014135990809256075,
+            0.013269444858490211,
+            0.00992155580941576,
+            0.011225162609266308,
+            -0.006545044517282357,
+            0.002368862850663672,
+            -0.02063183105457681,
+            -0.026981383275694142,
+            -0.02774194980145225,
+            0.013337303400678109,
+            -0.0025902548982556788,
+            -0.02794339093668326,
+            -0.025606606546251152,
+            -0.0029852498102770587,
+            -0.01906873442473352,
+            0.006508845592362198,
+            0.03268861529625085,
+            0.0355975143313428,
+            0.010066906387952277,
+            0.01690250467284985,
+            -0.017201544758038183,
+            0.048345309114140714,
+            -0.019031171948182872,
+            0.03188329748476941,
+            -0.0005845169448621133,
+            -0.024140202991373587,
+            0.00011597263700677706,
+            0.03203623528903764,
+            -0.019285565857567896,
+            -0.01332755507465857,
+            -0.010914187315481855,
+            -0.003374912271519852,
+            0.02419326154642231,
+            0.02443825785057301,
+            -0.0037923303809811746,
+            0.00022800399604730175,
+            0.015893265651968885,
+            0.003154527829977343,
+            0.007688529734165207,
+            0.030623379985627763,
+            0.014052282574575454,
+            0.01127235345371133,
+            -0.04164707563955819,
+            -0.011811587719246128,
+            -0.003763086694358232,
+            0.005909410979841885,
+            0.002913220159138154,
+            -0.002671613084376487,
+            0.0045765194162907885,
+            -0.007241703279856166,
+            -0.005855949422953439,
+            0.0016973541436777935,
+            -0.001576326424468957,
+            0.008393873711385013,
+            -0.0037217094191171083,
+            -0.03498879678492834,
+            -0.026096985039773694,
+            -0.021280075788049788,
+            0.006437497348154639,
+            0.0070214556591898375,
+            -0.00972721128392027,
+            0.01949553762684561,
+            -0.0029404319702066146,
+            0.000271037702827666,
+            -0.010734867909488861,
+            -0.0029784341515646075,
+            -0.01206464664516104,
+            -0.019544778160755956,
+            0.00035139045159739705,
+            0.013457601448003152,
+            0.00025947376767512006,
+            -0.03165522602526639,
+            0.011979811705146604,
+            0.006642849985850598,
+            0.013181162886675883,
+            -0.024499404540711422,
+            -0.02310714308893576,
+            -0.028028912907092744,
+            0.010570122662701183,
+            -0.0066866321204157725,
+            -0.024617571284817428,
+            -0.004097807033448707,
+            -0.0074576460628928256,
+            -0.0020417256533259887,
+            -0.02470953739335365,
+            -0.011408461668188081,
+            0.023537349682260345,
+            -0.006373652338136973,
+            0.01936338676879376,
+            -0.017636435070085086,
+            -0.017036091876814843,
+            -0.002198685044404603,
+            0.015584758061011048,
+            -0.038943923931820196,
+            -0.002076529978164001,
+            -0.008452513321945455,
+            0.006551055965075869,
+            -0.028840587857625097,
+            -0.007128279504360441,
+            0.0199397499488538,
+            0.012424319366271584,
+            -0.03518875279748495,
+            0.013596027106451115,
+            0.022191403935861113,
+            0.01218507466930854,
+            -0.0038551718784437783,
+            0.03910075115429626,
+            -0.0025215641203030336,
+            -0.005422688956956598,
+            0.016759888275451905,
+            0.04462538468871483,
+            0.023708097134660096,
+            -0.03152743757744915,
+            -0.008668774531128668,
+            -0.015224825785549947,
+            0.004102648962663789,
+            -0.025867795193966987,
+            0.004676262922216584,
+            -0.011739753544166856,
+            0.01645951757031186,
+            0.011253400919230348,
+            0.013958707917471253,
+            0.0376596076542735,
+            -0.005077354977296842,
+            0.02668910096392324,
+            0.0010332284250006117,
+            -0.00856723931523911,
+            0.0014431148305642308,
+            -0.01412579094241589,
+            0.029892393553325205,
+            0.02269599036836208,
+            0.014421409701746036,
+            0.029248195108770955,
+            -0.03588867295960255,
+            0.02458655544632528,
+            -0.0010013539341895672,
+            0.008368648583431747,
+            0.018150734970019018,
+            0.02628022565029391,
+            0.015423867373844644,
+            0.012477080028047823,
+            -0.02513018750675568,
+            -0.022180001014682045,
+            -0.0027550350630288997,
+            -0.031944843677988374,
+            -0.0014668093006914267,
+            -0.009789309083948147,
+            0.026386027599666254,
+            0.0046988411391149,
+            -0.019803160322638583,
+            -0.003931806845108628,
+            0.004482097428470392,
+            -0.00244660100774939,
+            -0.010441732346892935,
+            -0.004649318716095894,
+            -0.01869464992812615,
+            -0.017838643679294407,
+            0.023677878122153204,
+            -0.023958455640817666,
+            -0.025238725541146534,
+            0.014582798102040157,
+            0.0013223590277629197,
+            -0.002295641913710489,
+            0.0008238518461567685,
+            -0.02115533275282518,
+            -0.04214275933987015,
+            -0.027817083427786927,
+            0.0037588803464689412,
+            0.03565056002940463,
+            -0.026788865866928923,
+            0.014162550819567465,
+            -0.011336583458519,
+            0.019175861037658344,
+            -0.005127229083828615,
+            0.015531116410874209,
+            -0.018251290727445846,
+            -0.044630374763679266,
+            0.021833307809708984,
+            -0.012087009688208446,
+            0.02720742725750304,
+            0.021497809277370574,
+            -0.026937653159100156,
+            -0.013572733244188407,
+            0.01358214612023832,
+            0.0020408253706382332,
+            -0.009652962552750464,
+            -0.0063411929391088626,
+            -0.010533309167065193,
+            0.003901271588567942,
+            -0.04496345296706673,
+            -0.008761188928422701,
+            0.06948704056954039,
+            0.005916180375965137,
+            0.03157905519638486,
+            0.014668818318138927,
+            -0.028448496686381258,
+            -0.04008949536631331,
+            -0.00462253589823598,
+            -0.0003701522833020314,
+            -0.00024410762276269235,
+            -0.01850585249309217,
+            0.017699615637510956,
+            -0.002712069912576265,
+            -0.009767338815933488,
+            -0.013119464222297351,
+            -0.013743610418034631,
+            -0.030375365173543157,
+            -0.0030316327877227783,
+            -0.008632689470358149,
+            -0.006565457264917239,
+            -0.0328089275944709,
+            -0.011713996290006943,
+            0.029773910045828737,
+            0.02740100904353622,
+            -0.0061926915497348915,
+            0.00638737826122608,
+            -0.010244154924811975,
+            0.0010396203272677069,
+            -0.001264454445070139,
+            -0.004059969507786729,
+            0.008715730726831364,
+            0.0014767015415289076,
+            0.031074767756160294,
+            0.004761858344125113,
+            0.00987870172044607,
+            -0.059926117793699066,
+            -0.009294021263133636,
+            0.021188432871397475,
+            0.016602826531924377,
+            -0.009125784136292983,
+            0.013946188378030194,
+            -0.0017670794602549428,
+            -0.006837334179368295,
+            -0.028290055353601393,
+            0.024556541126233806,
+            0.04543639597357669,
+            0.0031311343860416473,
+            -0.011199639773247286,
+            0.014965677855595444,
+            0.0018248533506196013,
+            0.015507721118783542,
+            -0.011409402378400194,
+            -0.01338737990904486,
+            0.03063410115541065,
+            0.009305362935335817,
+            0.0025578631174065385,
+            0.012727648343848194,
+            -0.0038680554900795763,
+            -0.01173215289303824,
+            -0.01568862495293563,
+            -0.004101703129244437,
+            -0.03649200408657064,
+            -0.0022107849490954137,
+            0.009659780679763728,
+            0.0034091693454195806,
+            0.008848380026939594,
+            0.01842435611499587,
+            0.03546926214224266,
+            0.013478278853234764,
+            0.005970293629431872,
+            -0.0069197533761813455,
+            0.023196445250565058,
+            0.010403914879205351,
+            0.010017272564548626,
+            0.015076807790462658,
+            -0.029649260409273275,
+            -0.012737152920662164,
+            0.00373276777994811,
+            0.013239909450830258,
+            0.0001682544254742693,
+            0.02030652051051447,
+            0.014143961923579586,
+            0.0010147733405793957,
+            -0.012509446388074923,
+            0.0020437769941487626,
+            0.007405578085797006,
+            0.013458276946120546,
+            0.0006779114540819997,
+            -0.004250745498730309,
+            0.0009149988500358458,
+            -0.008355906072311008,
+            0.040391939952386505,
+            -0.01831218885700067,
+            -0.016793844108223176,
+            0.035064882870542116,
+            0.11416006426189942,
+            0.0010630856112440689,
+            -0.015660014600661857,
+            -0.017142534874931403,
+            -0.0086899013413391,
+            0.016611825572132997,
+            0.01609650977226868,
+            0.009780106186629793,
+            0.009622367309076686,
+            -0.000672926017804395,
+            0.024310949465350565,
+            0.02944698490225121,
+            -0.0037169531700178868,
+            -0.010990838766452194,
+            -0.036819646475041955,
+            0.022923962979125287,
+            -0.004596366586768123,
+            -0.007108041562635549,
+            -0.002604794313552557,
+            0.012641223813290853,
+            0.013668775645501733,
+            0.005125100040230003,
+            0.0077993841846437815,
+            -0.011500851674334447,
+            -0.017334458072700107,
+            0.0056819751688881774,
+            -0.015982234644378358,
+            -0.0076533013358874935,
+            -0.013127172654362694,
+            -0.011837346554842119,
+            -0.006562312362371076,
+            0.0029877420008976425,
+            -0.008572723854050838,
+            0.00854083969164562,
+            -0.010939639783598493,
+            0.016185642759084087,
+            -0.008052282332943615,
+            -0.0011992212936609377,
+            0.0010378646897070758,
+            -0.0025399515501741718,
+            0.010712865639618352,
+            0.01451357949543628,
+            -0.02521401163091652,
+            0.020836357346214075,
+            -0.008252813367155418,
+            0.0006409807249997337,
+            0.015338809580802425,
+            -0.04672526649678653,
+            0.022728876784937357,
+            0.005129225753457052,
+            -0.018372222092794564,
+            0.0066182736895667805,
+            -0.02048375780196863,
+            -0.00896382274306482,
+            0.0021738974344880794,
+            0.004491167150658828,
+            0.0034280799511763956,
+            0.015611133773752265,
+            0.018126362497548308,
+            0.00014383671280220997,
+            0.007927708693242186,
+            -0.006973076292062532,
+            0.008118313055163714,
+            -0.006791836664375275,
+            -0.01164567445668625,
+            -0.010894577039363014,
+            0.00014731542306100506,
+            -0.008865132658571428,
+            -0.011415696755798914,
+            -0.017922318360382588,
+            0.00358538001727405,
+            0.00289327637128695,
+            0.010406054711003452,
+            -0.007757607207626101,
+            -0.010994775479337082,
+            -0.011716096880261664,
+            -0.0013333474316266438,
+            -0.01231144419953554,
+            0.009792603083441917,
+            -0.02884953407490196,
+            0.0029097409399144306,
+            0.025277274938032477,
+            0.009263530315933738,
+            -0.028037993274175277,
+            -0.02532252134134532,
+            -0.0018596440554143638,
+            -0.013847737486669237,
+            0.009910107581119272,
+            0.013952962280235523,
+            -0.011807495289410774,
+            0.003896325037406752,
+            -0.02144921729633576,
+            -0.01628545945817568,
+            -0.0040148671532777005,
+            -0.016032493124255323,
+            0.004580041957001551,
+            -0.011375180364988195,
+            -0.007789996450583554,
+            -0.022041665680945066,
+            -0.005890065942573431,
+            -0.050455058269193866,
+            0.014401045842891128,
+            0.030219548461217365,
+            -0.004580747668649223,
+            -0.028684680969792875,
+            -0.015525870112797619,
+            0.011121401958547706,
+            -0.008879749268609245,
+            -0.019454243014778435,
+            -0.007839434838464604,
+            0.018908534797107714,
+            -0.0019283067432864216,
+            0.0024769961730655835,
+            -0.006732653593696035,
+            0.027604805021030646,
+            0.01366593691107021,
+            0.03740746059041736,
+            -0.014662268336336578,
+            0.02539666271785517,
+            0.02371809783638648,
+            -0.036126848124425154,
+            -0.007627445211958312,
+            0.03657481892016626,
+            0.01092515982671624,
+            0.0011046573579274704,
+            0.012350047525582187,
+            -0.006340802546498992,
+            -0.031330519599469066,
+            -0.0013691360532301728,
+            -0.0014676630169621005,
+            -0.023111265345539388,
+            0.027359543575867225,
+            0.018091458372134273,
+            0.014237156751820934,
+            0.0030993618578507172,
+            -0.0014037082734049385,
+            0.013257269860435359,
+            0.01021315143964527,
+            0.030885417052806473,
+            -0.003225065956381245,
+            -0.012825341172770057,
+            0.013476039084668826,
+            0.01852164843904835,
+            -0.009965495459579544,
+            0.012541079912270015,
+            -0.00824122613985912,
+            -0.018124134280975132,
+            0.03192263766222747,
+            0.0001484308749572035,
+            0.010888733498667343,
+            -0.010649515858317311,
+            -0.013082335214246402,
+            0.032614733822945205,
+            -0.00999075761269065,
+            0.004257643407198955,
+            -0.030723533031261387,
+            -0.003820776505926347,
+            -0.010471792928758869,
+            0.01737608180148319,
+            0.004597677757231315,
+            0.02095148515295301,
+            0.029253627700524396,
+            -0.011604450820008121,
+            -0.01996325509217509,
+            -0.001158626158150122,
+            -0.020583793738536985,
+            0.004168757459099133,
+            -0.03632338333261702,
+            -0.019373660230711246,
+            0.022259216603413172,
+            -0.010364620831688015,
+            -0.004751193403783363,
+            0.00444187509644351,
+            0.0008572202661850934,
+            0.014943779567410408,
+            -0.025408263453313398,
+            0.017755758370543148,
+            0.004627392860417132,
+            0.007402073071525438,
+            -0.024913329140362466,
+            0.013760771254234878,
+            -0.006087346663657887,
+            -0.03780469033982951,
+            0.005509388120163784,
+            0.03756141956381493,
+            0.02381019168406692,
+            -0.024580249929652097,
+            0.01807584361080568,
+            -0.03111039245576191,
+            0.023245542501125073,
+            -0.004868090951368273,
+            5.1544460403943004e-05,
+            0.00640235757127173,
+            -0.004698373089469779,
+            -0.00621383035264185,
+            -0.0025132415458431587,
+            -0.008746634331769936,
+            -0.01611801559493664,
+            -0.01193535623883546,
+            -0.03381789115349148,
+            -0.024119754887378317,
+            0.013881102852385332,
+            0.0019232668966222687,
+            -0.0095263729740736,
+            0.018652966857927805,
+            -0.03775912497511837,
+            -0.012022352025516115,
+            -0.023289377270709736,
+            0.023579672990954228,
+            -0.028605627815260902,
+            0.019235397553197357,
+            -0.025838655372560505,
+            -0.021458526449310157,
+            0.01508157885177254,
+            -0.010064396230734452,
+            -0.0016708624560165368,
+            -0.015042071642751917,
+            -0.03670785416972153,
+            -0.006267614265977843,
+            0.02298763967146357,
+            -0.012506444728541897,
+            0.00029871598584066644,
+            0.02671390618094565,
+            -0.003059865833779805,
+            -0.011103452976820622,
+            -0.0017382684168257506,
+            0.010923055147701782,
+            -0.02938686260440486,
+            0.014300097108092281,
+            0.0038571851420598074,
+            -0.0037614499297117365,
+            0.0010991440540809893,
+            -0.010108413486243776,
+            0.02127952161226619,
+            0.009142545227414532,
+            -0.0030260119510809205,
+            0.040589877142650374,
+            0.008677453889680373,
+            0.0017339508605300814,
+            0.008732950895530873,
+            0.01063352447960556,
+            0.006618635029823979,
+            -0.014986307792626314,
+            0.016905861015681957,
+            0.013101237900847744,
+            0.0002162791516784468,
+            0.019309343995079022,
+            0.0021263376627204313,
+            -0.024516267132902416,
+            0.013339486637573757,
+            -0.02311659751478777,
+            0.0020575982953961246,
+            0.02391695586495566,
+            -0.020440172706421535,
+            -0.001656132876701879,
+            -0.006304133667169861,
+            0.000543457661904495,
+            0.022135707414160034,
+            -0.0230903871986617,
+            -0.023444763940153112,
+            0.009604600033024832,
+            0.01721387169484476,
+            0.01827905276638783,
+            0.02824773106648473,
+            -0.007100625424382998,
+            0.01348510178676837,
+            0.0002604386361122316,
+            -0.011802178189607182,
+            0.007664157334475143,
+            0.0074175377458313594,
+            -0.0044008460682833875,
+            0.025491035291032547,
+            -0.0017015687016376205,
+            -0.006134010425966364,
+            -0.005379657499404915,
+            -0.01584573519366957,
+            0.0034386456145185213,
+            -0.013860742817569676,
+            -0.013355238174583658,
+            0.0040342831827141845,
+            -0.0208160950322342,
+            0.10904592104153767,
+            -0.0037038089210379766,
+            -0.019472012964045247,
+            0.0063554319446707044,
+            0.003716719884684827,
+            -0.003692786470980806,
+            -0.003353843210051379,
+            0.007788605566526459,
+            -0.011254480693190834,
+            -0.0045751956417398095,
+            -0.013976039724381875,
+            -0.001989262807706565,
+            0.010067374586433014,
+            0.009201983771978006,
+            -0.005902980676079493,
+            -0.02029714318363332,
+            0.013031676901335606,
+            -0.016970144750162643,
+            0.008874670945038486,
+            0.0008311255074953139,
+            0.00884167159995975,
+            0.001716603195908536,
+            0.01854951250316076,
+            -0.009310881405486074,
+            -0.01325242738760595,
+            0.029667589863813217,
+            -0.005616511284128333,
+            -0.0038179509972925954,
+            -0.0004690716809300012,
+            -0.005068974756744239,
+            0.0021204231125051274,
+            -0.027958274497127258,
+            0.022127088236172387,
+            -0.017271838492631593,
+            0.010267709087809265,
+            0.01316800560257366,
+            0.005599367128954229,
+            0.01817806970002991,
+            -0.007115594036834107,
+            -0.02627643123342057,
+            0.006542742528003751,
+            -0.003433874053082844,
+            0.006631427284537971,
+            0.02316295746074422,
+            -0.014573351469966297,
+            -0.033520864104177005,
+            -0.013707022143307006,
+            -0.01877574599842729,
+            -0.005992004412448576,
+            -0.0043698881826095435,
+            -0.01481352026460129,
+            0.006424121402963946,
+            0.01382938882387991,
+            0.018136021761515388,
+            0.0012697483790911433,
+            -0.011531136974861559,
+            0.00609187274277241,
+            0.01076133059240578,
+            0.008400887242930113,
+            -0.01533504932731406,
+            0.011439227832642668,
+            -0.028718487721012004,
+            0.01522697877548034,
+            -0.012452665599305595,
+            -0.024938748080573947,
+            -0.0022082450217489366,
+            0.017589399637694264,
+            0.0037824832255756763,
+            -0.004355445812154662,
+            -0.018778668474337727,
+            0.019825087149714933,
+            0.024692878851503508,
+            0.003883497401229232,
+            0.019470881143244197,
+            0.041834635328088364,
+            -0.003544509491805351,
+            0.027139164223766474,
+            -0.02129723143323512,
+            0.01461354860170749,
+            -0.03441729954693155,
+            0.001893713808284104,
+            -0.004222657393633789,
+            0.0013677766871517654,
+            -0.003811928806805752,
+            0.020142937759360904,
+            -0.009158202577174012,
+            0.0030638058587375543,
+            0.024854053046139163,
+            -0.018665378030958756,
+            0.00805903734779081,
+            0.010625429315145887,
+            -0.012856237251902233,
+            0.028535299711292306,
+            -0.006335094045840286,
+            0.00305422126851784,
+            -0.0297883488914759,
+            -0.0029192872993071216,
+            0.0026067044379754047,
+            -0.006097749614529602,
+            -0.00968023644800927,
+            -0.007433916539502931,
+            -0.006202274026852405,
+            0.0038354607448269555,
+            -0.021471361566683354,
+            0.003195646827154078,
+            -0.004390751282171024,
+            -0.003323447423321121,
+            0.02518759723554948,
+            0.025857746688368054,
+            0.016893707217894103,
+            0.012739170140289531,
+            0.008553625305650129,
+            0.019805351588748434,
+            0.01654964927067909,
+            0.011549329129728739,
+            -0.020026013396046632,
+            0.01150644479272541,
+            -0.004159611921476011,
+            -0.0022937297602724912,
+            0.014815466072709131,
+            -0.01572171840394285,
+            -0.02276187015864222,
+            0.024693009795978448,
+            -0.019725315346836404,
+            0.00998768979020729,
+            -1.1702210281215188e-06,
+            -0.011129745730507545,
+            -0.002591293237053776,
+            -0.018034357254990988,
+            0.00012070821244376275,
+            -0.023051045150566527,
+            -0.028465270110388716,
+            -0.00048603008831669926,
+            -0.000574546300981984,
+            -0.012992623897880395,
+            0.02064747165046032,
+            0.014820638761258362,
+            -0.024820174379739057,
+            0.02666518374464728,
+            -0.017430076811109155,
+            -0.01495051200726725,
+            -0.005861181182847845,
+            0.00100881727574559,
+            -0.0055223282819321295,
+            0.0034719859452138017,
+            -0.010180752654175722,
+            -0.009961678082468036,
+            -0.012134423047419183,
+            -0.028073672819317354,
+            0.011574683159318163,
+            0.016120674453124317,
+            -0.015043147648505721,
+            0.031834693135959095,
+            -0.008802805725050891,
+            -0.0018698230152007877,
+            0.016179742266521146,
+            0.005096232332051617,
+            -0.017187077376166815,
+            -0.018999569809271876,
+            0.012251308792871571,
+            0.013279090498325945,
+            -0.009588297039279112,
+            0.006630431200804809,
+            -0.014130176783207386,
+            -0.005092662672039885,
+            0.007020236623084732,
+            -0.005049363863784624,
+            0.0055051712211635915,
+            -0.00995314895304919,
+            0.0052743456988306485,
+            0.005841370337169303,
+            -0.0029434616355351975,
+            0.004677049987338193,
+            -0.018568941264514256,
+            0.020031253265629777,
+            0.0018725390936655962,
+            0.01332532735283085,
+            -0.03518855189467458,
+            -0.0056298723325832025,
+            -0.005369475318146738,
+            -0.004650364775945294,
+            -0.020833510091412353,
+            0.020445280915606132,
+            0.0159991245532085,
+            0.013202409908533162,
+            0.023094507005426774,
+            -0.014905666713082246,
+            0.01572052456975675,
+            -0.020945766915121455,
+            0.013114000985348841,
+            0.003349101658930482,
+            -0.019829605346106578,
+            -0.001423750749031516,
+            0.04557459007627153,
+            -0.005847231973329081,
+            0.011986110554869603,
+            0.0012709842140249691,
+            0.026758922819943853,
+            -0.0001635536762342011,
+            0.0065419519929442995,
+            0.005997692702992206,
+            -0.009935128108338126,
+            -0.0008057798606777003,
+            0.00865712443881613,
+            0.046254320605750854,
+            -0.01181132866321511,
+            0.008170688314239364,
+            0.013099051984057827,
+            0.0034971494510865624,
+            -0.0008386400616584293,
+            -0.015346506644390764,
+            -0.03067678541405639,
+            0.019305962963428017,
+            0.007736916644077731,
+            -0.012756525606221703,
+            0.008734673666457216,
+            0.015137771675476126,
+            0.012891824153867332,
+            0.003895886821685762,
+            0.02155154331885366,
+            -0.012761690789748027,
+            0.018528313792219413,
+            0.024663453412919027,
+            0.009077230086690327,
+            9.184526542175535e-05,
+            -0.000751529849753638,
+            0.006791927267880182,
+            -0.0004433479107716552,
+            -0.009832230053513483,
+            0.015858232096630655,
+            -0.014861341148074698,
+            0.017790924034108806,
+            2.7216806238946785e-05,
+            0.03744841066552559,
+            -0.015216207585985075,
+            -0.010138319613387346,
+            -0.007774325714088762,
+            -0.0016997034352120362,
+            0.003971564314593291,
+            0.0018686537276849653,
+            0.006051669054361943,
+            -0.0008319624938748146,
+            -0.017326561560776794,
+            0.01667737942301944,
+            0.005719808546053257,
+            -0.0007405637872110778,
+            0.013750752065593804,
+            0.0038732323799683453,
+            -0.00367645612814479,
+            0.023580906347714314,
+            0.006238541579764762,
+            -0.005235795826199542,
+            0.0028182323816636243,
+            -0.006443210286549713,
+            -0.016255209299456314,
+            -0.003061441395209219,
+            0.004597970789372988,
+            -0.0016590920570710628,
+            -0.011138261849069419,
+            -0.0019423173609729848,
+            0.020355534503272887,
+            -0.004612410336124759,
+            0.013346750536413418,
+            -0.005400060374036387,
+            0.006252154576727367,
+            0.010601459925435238,
+            -0.012346569079682625,
+            0.0014154620309235822,
+            0.010406076751218612,
+            0.0013983417564564663,
+            -0.007140180075225315,
+            0.0010544410919327675,
+            0.007757599672903709,
+            0.006505350216372801,
+            0.00483063937962671,
+            0.00836630436955631,
+            0.013284451514874498,
+            0.0033749897293287438,
+            -0.006243293993633622,
+            0.0013949831763016042,
+            -0.013269921385800992,
+            -0.006308639321247727,
+            0.0010906169976769569,
+            -0.0010152835709869353,
+            -0.02017142197515583,
+            0.010787630901414118,
+            0.005355784839945777,
+            0.007838485470991034,
+            -0.01706932891696025,
+            0.009576030158827565,
+            -0.0008144311254998525,
+            0.02829832828245172,
+            0.007407953684956539,
+            -0.02769403281764041,
+            0.014266001222243134,
+            -0.010492199740754951,
+            0.022575782293320284,
+            -0.0016349316593207844,
+            -0.005656505606143177,
+            0.008025514398322551,
+            -0.009634281579228592,
+            -0.005991026158213869,
+            -0.022366083313133788,
+            -0.025825209755160946,
+            0.016871087849192924,
+            0.013583173973358155,
+            0.021125252303971605,
+            0.00012555627355712715,
+            0.015347535950809997,
+            0.0020128011217859444,
+            0.015164099887550086,
+            0.0024544903332736167,
+            0.012507136277293888,
+            -0.011798050869794805,
+            0.0104828761281283,
+            -0.0137805761114781,
+            0.014620357150314097,
+            -0.005246639228547017,
+            -0.006986408834884282,
+            -0.008849580196616098,
+            -0.005442165099441538,
+            -0.020891551850452786,
+            -0.009155930286949747,
+            -0.0013590151948670868,
+            0.010665872513236615,
+            -0.0005386423407222223,
+            -0.0027671778465775897,
+            -0.03237311596935737,
+            -0.004875062533405715,
+            -0.006670982068843501,
+            -0.01942244110096111,
+            -0.0045975373348006885,
+            -0.026137783023267008,
+            -0.01657173843643047,
+            -0.0007784046498232684,
+            -0.015037306676062236,
+            -0.006889680260149007,
+            -0.02009705080543122,
+            -0.027192867101205106,
+            3.0161964469866572e-05,
+            -0.009647644858034833,
+            -0.005380165957043753,
+            0.013815092649296373,
+            0.011902366689169847,
+            -0.01572592170758115,
+            -0.022321702398924333,
+            -0.01088937403483412,
+            -0.0165754067023749,
+            0.01646435743868707,
+            0.009914938007210463,
+            0.018834863207293908,
+            -0.003724245912768107,
+            0.005643505694722323,
+            0.0009581443359000331,
+            0.02327704258596552,
+            -0.02532422120223976,
+            -0.028694220894404807,
+            0.0019298778426469338,
+            0.0025970175688368323,
+            0.006086164866310174,
+            -0.009414665394188936,
+            0.001200390869257535,
+            -0.007782899559317819,
+            -0.0228264209954853,
+            -0.00898519580978142,
+            0.008156835270668449,
+            -0.010036630164522355,
+            0.02230664068547531,
+            0.012332230358869292,
+            0.013699197379189934,
+            0.010593587938184292,
+            0.01576174456497367,
+            -0.0001654241831584687,
+            0.014471905257729428,
+            -0.008675915624727887,
+            -0.0031070346099204425,
+            0.0038358404385932777,
+            0.005878417037187274,
+            -0.00625022998235275,
+            0.028077114529659795,
+            -0.0011836453716817525,
+            0.0002962695480296182,
+            0.014405782572641941,
+            0.02697667559474498,
+            -0.004710313427099278,
+            0.0210512043261309,
+            -0.002393165025068831,
+            -0.006135816286212752,
+            -0.008429776018235104,
+            0.008607617299702318,
+            -0.009529291136733401,
+            0.011133497451942684,
+            0.019864404836361926,
+            -0.014041315457579166,
+            -0.007351413712676263,
+            0.02950254974910368,
+            -0.014564388091367897,
+            -0.01825317106331622,
+            0.02325964952838705,
+            -0.007414295676409368,
+            -0.0039042587057605484,
+            0.00377144546378926,
+            0.007169856503233239,
+            -0.004036744966949995,
+            -0.012928341573802416,
+            0.016090527447479415,
+            -0.00011159893712484579,
+            0.014146137810890391,
+            -0.0004655839804281017,
+            -0.006372231942983921,
+            -0.010047382825720416,
+            0.002373224247019062,
+            0.026818813126567614,
+            0.013483627627077692,
+            -0.017895710143841333,
+            -0.013785856764479817,
+            0.010826610341182479,
+            -0.01538893818197323,
+            -0.004425206001945227,
+            0.02155595194812746,
+            0.0028768153668128553,
+            -0.020543027467698483,
+            0.008028975884363618,
+            0.0003835165718138906,
+            0.00037046947047892953,
+            -0.00029363296913641716,
+            0.030135739208689217,
+            0.001343516577762509,
+            0.00018948658255495327,
+            -0.00875053830853609,
+            0.008606573235959595,
+            -0.01280305089741873,
+            0.012786478836532904,
+            -0.02437993139337839,
+            0.007648181775166282,
+            -0.00629541989541625,
+            0.004579605315599212,
+            0.016658386927051592,
+            0.01218098627022323,
+            0.015031618813778413,
+            0.008237200519537417,
+            -0.008942965899756683,
+            0.0009093265012901892,
+            -0.025302175262520477,
+            -0.001333639155622843,
+            -0.017971876771137367,
+            -0.02036433247634014,
+            0.0002463842837918767,
+            0.022144712689630985,
+            0.018559482073124734,
+            0.011579840028464503,
+            -0.008657014440886656,
+            0.0166696911919226,
+            -2.9834620942882283e-05,
+            0.011845827764518911,
+            -0.00037901176785495324,
+            -0.006138374154553732,
+            0.004711058953619512,
+            -0.003622694588720417,
+            0.013542405259906128,
+            -0.014730567462694827,
+            0.012076044672004906,
+            0.01690223095263866,
+            -0.009108632851807137,
+            -0.008946251274440141,
+            -0.013992423874475428,
+            0.00048381889145447367,
+            -0.015909599902931202,
+            0.035489252674454214,
+            0.02252772349379229,
+            -0.01210464970923921,
+            -0.010469120693943236,
+            -0.014164999313775756,
+            0.019496048806752753,
+            0.010427539351244087,
+            0.0028018908451425276,
+            0.017091768406027965,
+            0.021395638416767112,
+            0.008817247593404623,
+            -0.006081063711583319,
+            -0.0021216503501144983,
+            -0.003781810594530787,
+            0.020143098218482827,
+            -0.003078172339607037,
+            -0.011350171884507555,
+            -0.012516379645909518,
+            0.005766102930704529,
+            -0.01626433668691071,
+            0.0019294643928566657,
+            0.008641558521764407,
+            -0.015057688533773374,
+            0.019407986696657772,
+            -0.028126197703647868,
+            0.012364374199657305,
+            -0.0044278434183731225,
+            -0.007251147925369614,
+            -0.03496220945653279,
+            0.006389489026974697,
+            0.008941259259632024,
+            0.003941752277454314,
+            0.023003838655784314,
+            -0.024528514847470928,
+            -0.018361931455105486,
+            0.0037791061321962543,
+            0.00047162060848090104,
+            -0.00711894577052754,
+            0.0010138871688747137,
+            0.004128392769213672,
+            -0.00674546985015164,
+            0.0017307097443508704,
+            0.0033354800298095546,
+            -0.002829898077850263,
+            0.013252947102333348,
+            0.0081724269619999,
+            -0.021449979263734438,
+            0.001368791856860285,
+            0.01590656734732547,
+            0.01452008402429858,
+            -0.02391847163018108,
+            0.006331397860689212,
+            -0.02064812934794864,
+            -0.000631340242718475,
+            -0.031272866000888556,
+            0.016859793097564958,
+            -0.0069008624795683276,
+            0.01899718801899956,
+            -0.0015377436942771224,
+            -0.0065810685260109004,
+            -0.012822620655323715,
+            0.02137604937338179,
+            -0.0019121137078344135,
+            -0.0025598772127157477,
+            0.015678293057066646,
+            -0.025700865215693987,
+            0.017825186616033165,
+            -0.0026230848396158204,
+            -0.017342967698019966,
+            0.017199139867162275,
+            0.004536019178633225,
+            0.007224666811309087,
+            -0.023505589665315567,
+            0.0007208041198255396,
+            0.003900164485347562,
+            -0.017250324025622685,
+            0.004470862192961212,
+            -0.030807400122398566,
+            0.0006579147990327597,
+            -0.013222907940559794,
+            -0.0020514994691204847,
+            -0.0046763235724575445,
+            -0.014050035576183385,
+            -0.039181147615786484,
+            -0.005429639279537522,
+            0.016065742005148733,
+            -0.0009029636981539927,
+            0.01800715198737081,
+            -0.002730931639143803,
+            0.002947680833843522,
+            0.004912692811020822,
+            0.00818378920043427,
+            0.006915895602313347,
+            0.02274116906914758,
+            -0.0057854922886970365,
+            0.00471999900678522,
+            0.007058014277137233,
+            0.014638081356210031,
+            -0.01719565951201012,
+            -0.015170377598372723,
+            -0.016015921063369496,
+            -0.006351213642269936,
+            0.01739665344303718,
+            -0.017488885259028475,
+            0.013303332445465686,
+            -0.00429428898520701,
+            0.014825338166443021,
+            -0.008686427208690176,
+            -0.01575945086724703,
+            0.0028564767084934386,
+            -0.007570417028925067,
+            0.001007670647768344,
+            -0.02323908008484089,
+            0.02019729564706863,
+            -0.00847581038203716,
+            0.010508144473197669,
+            0.01729824052022313,
+            -0.021364529179605813,
+            -0.009742851959432839,
+            -0.014342281441152413,
+            0.029896210049029176,
+            0.013925163184699139,
+            -0.009392252140772913,
+            0.003927230832659267,
+            0.008873561269230905,
+            0.011529536139797032,
+            -0.012209070214071445,
+            -0.0027679974125029362,
+            0.002411914349172495,
+            -0.0062056207925691356,
+            -0.002025687569076897,
+            0.03090134105002816,
+            0.0038285457265343245,
+            0.024877474647546086,
+            -0.013214997316123852,
+            -0.0033763842539861907,
+            0.00023379106197363714,
+            0.005134764372494098,
+            0.01183867975321486,
+            -0.0017966997954747142,
+            -0.008029791503028367,
+            -0.01179500476055557,
+            0.02599911330072515,
+            -0.021852083993691557,
+            0.007024505421035273,
+            0.026784024996764876,
+            0.001390202993990949,
+            0.0029956956045771046,
+            -0.013131691460546865,
+            -0.006298736864509989,
+            0.02523640664202355,
+            0.010030824059858138,
+            0.0041523782603032225,
+            -0.026465819482580182,
+            0.0032128899272377593,
+            -0.0061901409332948115,
+            -0.008681758126710213,
+            0.009866461769956222,
+            -0.0028095454857224164,
+            0.01188012041754054,
+            -0.012191996383095982,
+            0.028459939543805705,
+            0.010711502132045176,
+            0.01047154118931583,
+            0.025888652452698974,
+            -0.02079614305315134,
+            0.0042203106489994615,
+            -0.010829796140534066,
+            0.0006773493086316061,
+            0.005596640511307504,
+            -0.027510562413905437,
+            0.001969898603871003,
+            0.010390866563404346,
+            -0.005642706366801436,
+            -0.01882994179544761,
+            0.01187970346127799,
+            -0.019814385265412387,
+            0.024479579275617887,
+            0.003231341626501863,
+            0.020056914356101588,
+            -0.0053276448188067154,
+            0.009834685132275868,
+            0.0002742942778577165,
+            0.0031361331890882127,
+            -0.0008074598092128465,
+            0.009009012303347016,
+            0.005786777410144894,
+            0.013719989127068968,
+            -0.025528370602188686,
+            0.006821572597883499,
+            -0.0029197127345352626,
+            -0.030539872624857384,
+            -0.013383097680839732,
+            0.010291485622655026,
+            0.012218533188117704,
+            -0.009144425807890594,
+            -0.010378679518611277,
+            0.0030042579827305603,
+            0.005748243116238633,
+            -0.006692403885450407,
+            0.006913672151000981,
+            0.01629516912493545,
+            -0.002533517074771728,
+            -0.0019083178803585312,
+            -0.0035007340321378893,
+            -0.01154609935927537,
+            -0.010016998439072824,
+            0.023193275321990937,
+            -0.00482990342915347,
+            0.017943025772239603,
+            0.025873224580389702,
+            0.0039876567755322,
+            -0.0057577932736143506,
+            -0.005460363755206188,
+            0.01478435999841998,
+            0.014195457012279476,
+            0.004669295894868759,
+            -0.009702220679239768,
+            0.007974305230005956,
+            -0.007760086293456636,
+            0.008375077126777163,
+            -0.020260917110786147,
+            0.002199401871514877,
+            0.0026191373298767766,
+            -0.0022089262577237776,
+            -0.015213807046175918,
+            -0.01956788452420682,
+            -0.021898368819136302,
+            0.012634659463647929,
+            0.00034643102239488474,
+            -0.009241772775582378,
+            -0.013645250997151429,
+            0.0016665106990117016,
+            0.0025630500346914592,
+            0.0002095521560180936,
+            -0.007783022628694597,
+            0.009546147813008391,
+            -0.01265287317178631,
+            0.007445223311736028,
+            -0.004007029181436962,
+            0.0051846833906574515,
+            -0.0016916014415925766,
+            -0.016413938859402438,
+            -0.0210868129634149,
+            0.01920794362224444,
+            -0.012897042227841011,
+            0.02269303737601583,
+            -0.0014935878147854765,
+            3.062680855559304e-05,
+            0.0033132626755401546,
+            0.007274487651476732,
+            0.012110042834251237,
+            0.011426424896797977,
+            0.004307086698938411,
+            0.008206932254416481,
+            0.004673427176334148,
+            -0.013124571552764586,
+            0.004324963759928715,
+            0.0039216033705804084,
+            -0.010449887922072297,
+            0.005298323150690876,
+            -0.0010196552573491,
+            0.006128327172621373,
+            0.012167492341907742,
+            0.01923525204293908,
+            0.01908107550907106,
+            -0.0012031208519043073,
+            -0.0010601912677170903,
+            0.014474374858125607,
+            0.004540954805526089,
+            -0.004883537463424679,
+            -0.002772988629590886,
+            -0.013363713078069854,
+            -0.002572403646338979,
+            0.011541515690646252,
+            -0.02376161747414617,
+            -0.018215231053002656,
+            0.021831219299143226,
+            -0.004991882316041097,
+            0.033796174834149546,
+            -0.013140970283091536,
+            -0.016197177724633614,
+            -0.00464190076219332,
+            0.0026389653040644607,
+            -0.013788049032378498,
+            -0.010300859744205445,
+            0.010545541278382187,
+            0.013848738866656793,
+            0.009903590151120531,
+            -0.0005984085581408575,
+            0.02922816576443645,
+            0.0019900932217491907,
+            0.02245395592422119,
+            0.0018978407151245295,
+            -0.007029294098689854,
+            -0.01440575120349887,
+            0.010840472832356266,
+            0.005529335652955532,
+            0.009050748012161991,
+            0.017791269808484802,
+            0.014502057454389247,
+            0.006265567405788131,
+            0.013928705812292101,
+            0.0012903002372833693,
+            0.0010858377120770652,
+            -0.002355229463060164,
+            -0.01692711767136341,
+            -0.01456786846097011,
+            -0.01627640642218236,
+            0.003873864312554098,
+            0.012620350492241247,
+            0.00525890570023481,
+            0.0007393789113866752,
+            -0.007153210751899716,
+            0.009254566395058412,
+            0.0065087881512835375,
+            -0.012279152412209152,
+            -0.016530068283509593,
+            0.004060671955726967,
+            -0.02210570272789274,
+            0.007849467899009437,
+            0.01554233481902349,
+            0.00219885032769439,
+            -0.002655449524219968,
+            0.0006843524304842362,
+            0.006272944169764281,
+            0.005531927368102191,
+            -0.0036710328198476905,
+            -0.00544113514775027,
+            0.011264661763762068,
+            -0.006379654234440906,
+            0.009537150372938296,
+            -0.00290031548037575,
+            0.03718190949929003,
+            -0.011976080887998194,
+            0.012120408392610773,
+            -0.001751472085435711,
+            0.008481703962951994,
+            0.008067669857949423,
+            0.010772687289903202,
+            -0.003805089040741351,
+            -0.0022164543131735167,
+            -0.008753625032781523,
+            0.006751953442322185,
+            -0.01582177621255226,
+            0.013631301238130705,
+            -0.00030685726238985963,
+            -0.010728958775974582,
+            0.01055472059121978,
+            0.006788316621149835,
+            0.004413382316882087,
+            -0.003262024961860459,
+            -0.01972283203505511,
+            0.014937729522976615,
+            0.0018799533869990667,
+            0.019247008681353865,
+            -0.023276106189004046,
+            0.029043727846427652,
+            -0.02019024213343764,
+            0.023083643395685142,
+            0.01865875504942157,
+            0.001179436167200093,
+            -0.02070329911663634,
+            -0.007618131527214554,
+            0.014871709122115308,
+            -0.02659818889735726,
+            -0.006234719259825804,
+            0.005986829551040766,
+            -0.014639912579831785,
+            0.00037813423546252645,
+            -0.01924149909676064,
+            0.00010832498610532006,
+            0.011013130149723431,
+            -0.011782407678373089,
+            0.00438725832178584,
+            0.01608349750921335,
+            0.019347883652009713,
+            -0.009760017612545174,
+            0.003869246694835168,
+            -0.024240324890794748,
+            0.010958789926998691,
+            0.020511684636649225,
+            0.008883830392748497,
+            -0.01761944522796127,
+            0.021817973960311446,
+            0.010652131228027521,
+            -0.012575679152857703,
+            -0.0012753095453828826,
+            -0.028191274775392804,
+            -0.012421170815054556,
+            -0.02188248722244861,
+            -0.0011670175621882146,
+            -0.014436683528976771,
+            -0.0029783880907228383,
+            0.0009506986717516226,
+            -0.004719165363484719,
+            -0.01179163390107033,
+            -0.019543732196676634,
+            0.016606805875498898,
+            0.0227313413004497,
+            -0.006738735201626204,
+            -0.014547240981002825,
+            0.013866974327185123,
+            0.031694077725890536,
+            -0.016189209297529324,
+            -0.00542110554672736,
+            0.0028061798782555557,
+            -0.012964876141500577,
+            0.018946046750284537,
+            0.0039419760434645495,
+            0.000322115192590907,
+            -0.01991851369210666,
+            0.002489725351392778,
+            0.010835458511052207,
+            0.01684964045708685,
+            0.0005541314524621196,
+            0.010855852688653808,
+            -0.007549882748861744,
+            0.010096800951420057,
+            -0.011832260836468588,
+            -3.6179995603517093e-06,
+            -0.007800366796516945,
+            0.007001049318404058,
+            -0.004424776169203575,
+            0.034356203288485945,
+            -0.02817071276766299,
+            0.004989631403681216,
+            -0.014539644354369607,
+            -0.010888763939716889,
+            -0.0027322086509797843,
+            0.0004865534340981866,
+            -0.010775238981581461,
+            -0.005162220412866082,
+            -0.0011263271110752205,
+            0.00913081242511139,
+            0.005844327320514489,
+            -0.013886529519306775,
+            0.019171827605726503,
+            -0.030984919318656618,
+            0.004469458202753611,
+            -0.008553175891755315,
+            0.010511707988936336,
+            -0.0151185694875566,
+            0.013830711686954931,
+            0.01301959345025159,
+            0.008205984085944315,
+            0.007740367849216328,
+            -0.01532123052721834,
+            -0.01678314222392063,
+            0.001376468747213867,
+            0.0036953018347776244,
+            -0.00931855481896886,
+            -0.033391028922085425,
+            -0.007072913446166969,
+            0.005331350534425606,
+            0.011614453225043,
+            0.008156128603003278,
+            -0.02345206873048703,
+            0.005003870024809975,
+            -0.024178720646704597,
+            -0.012115112684614331,
+            -0.007750557665424973,
+            -0.011580152272614108,
+            -0.006117298954192534,
+            0.026449765548206453,
+            0.007989237291215073,
+            0.016691517078382376,
+            0.004954133565782356,
+            0.02742809127725913,
+            0.011670893621991404,
+            -0.011736840457474895,
+            -0.003349151747880166,
+            0.00035419034843196817,
+            0.005507651806218584,
+            -0.008706590052024251,
+            0.011793241980257321,
+            -0.0013295035978611142,
+            -0.0065061682758755625,
+            -0.004398354347597199,
+            0.009372816666628725,
+            0.014188676762501037,
+            -0.003007980559117429,
+            -0.008724255229333351,
+            0.015749776067989826,
+            -0.030899793617096834,
+            -0.015215033379100091,
+            0.0016152308157299517,
+            -0.007813175908301852,
+            0.017806770680528907,
+            0.01169056345888971,
+            -0.026835412610223673,
+            -0.004538235170332766,
+            0.010532316940606419,
+            0.004755930024691789,
+            0.0095408889025506,
+            0.00954782703036282,
+            0.012079064562163901,
+            -0.007829549014214643,
+            0.01046722658021748,
+            -0.011519369840999348,
+            -0.007092961453184232,
+            -0.007241799314569802,
+            -0.008077934089061052,
+            0.021467547268987214,
+            -0.025816030568238717,
+            -0.006558853543772633,
+            0.01134670818701549,
+            0.013027491377727365,
+            -0.002301991242932548,
+            -0.006299073927206214,
+            -0.017357989759336386,
+            0.008311464809363397,
+            -0.013922568511594478,
+            0.025633907424998367,
+            -0.004854655498817219,
+            -0.005336807899100369,
+            -0.005228056880188717,
+            0.01958682721272445,
+            -0.016774409737540904,
+            0.02072573220949266,
+            -0.014449250679704904,
+            -0.007857351393427215,
+            0.001673929749689194,
+            0.006890008607998747,
+            -0.002948835017234416,
+            0.03734128725793149,
+            -0.00556474133167756,
+            -0.0014544201137417158,
+            0.02209084868209573,
+            -0.01883610080898967,
+            0.017901421953179442,
+            0.01563604483998435,
+            0.012327159100148019,
+            0.0018161034827768648,
+            -0.010253034391420751,
+            0.019979728662282904,
+            0.020823925491027635,
+            0.0139490143219451,
+            0.009577521724445478,
+            -0.013478761809038993,
+            0.014934933784421556,
+            -0.0003725422088504494,
+            -0.0024362115214204105,
+            0.012785534912074068,
+            0.016441944143888537,
+            -0.02164522371903646,
+            -0.013152102018736346,
+            0.0063548333449661805,
+            -0.016547629104776748,
+            -0.024875443975589356,
+            -0.008991095813132973,
+            -0.021885127280977718,
+            -0.023336714382596175,
+            0.00688008599534903,
+            0.016482610587564443,
+            0.0035797836729533367,
+            0.013111012464332918,
+            0.006820471349350837,
+            0.018274678506489214,
+            -0.00980248993077149,
+            -0.01582838461385605,
+            -0.0009454586621769173,
+            -0.0004906493597104733,
+            0.009787688665732045,
+            -0.01835583040075359,
+            -0.013977939077760832,
+            0.004340902272034388,
+            -0.020118484730757877,
+            0.01421736173162507,
+            -0.006428386730058143,
+            0.016756376129345597,
+            -0.0004630315831872392,
+            0.011701948921076007,
+            0.02621773278421472,
+            -0.001391567406453007,
+            0.01262026995055899,
+            0.004696906079976175,
+            -0.0030577364576645635,
+            -0.010624066224426196,
+            0.0009420498021845824,
+            -0.0044079288776429516,
+            -0.0004845606666642234,
+            0.006481113327122954,
+            0.020269778435314,
+            -0.03321232649491074,
+            -0.009425498978427855,
+            0.01246754422779043,
+            -0.001658574527323383,
+            -0.005816985985181387,
+            0.013050062223188965,
+            0.00011195028631856881,
+            0.016573827662919836,
+            -0.0017453203044868104,
+            0.01417371165305189,
+            -0.008165910519539772,
+            -0.0007461950113024975,
+            0.0016492309267666983,
+            0.005076015595439873,
+            0.005829299318055953,
+            0.012146800191035666,
+            0.01148884639506138,
+            -0.021051327863689618,
+            -0.0026357096198729156,
+            0.0036146148489310443,
+            -0.010673282438301611,
+            0.014967444393862347,
+            0.005737933867764028,
+            0.01391411712868596,
+            0.0029556189893790237,
+            0.01167806258173173,
+            -0.024440974650111174,
+            -0.01029944007931845,
+            0.01385509328958837,
+            0.00869671728965227,
+            -0.00998315356917989,
+            -0.016580053751364735,
+            -0.007061005072623325,
+            -0.00477175200214453,
+            -0.0021182980051220428,
+            -0.009871524928312118,
+            -0.0005933941177237234,
+            0.006284920058476413,
+            5.466820734762947e-05,
+            -0.012842295882002122,
+            0.015810530885294202,
+            0.004572385791133295,
+            0.003685966629168175,
+            0.005093714133656017,
+            0.011877541830396259,
+            -0.013130170365831912,
+            0.01258190257102377,
+            0.011433121961378686,
+            0.01142665981860032,
+            -0.013710025169815697,
+            -0.001774030746825879,
+            -0.01109881340948764,
+            0.0065684584345163735,
+            0.009983753894796107,
+            0.005346443477816739,
+            -0.002062387245572686,
+            0.004950044847352647,
+            0.006105199826576752,
+            -0.06391185159544624,
+            -0.02131511418723256,
+            0.0027165737784844413,
+            0.033389688791557785,
+            -0.007167593552251769,
+            -0.0037646833460996797,
+            0.002883510089742145,
+            0.01356887065340832,
+            0.009730171586767032,
+            0.00418566245984083,
+            0.020320934773473808,
+            -0.008814826860651366,
+            -0.003014193747732543,
+            0.001559573791453607,
+            0.003499420226345028,
+            -0.008703837291821111,
+            0.0044844701302276645,
+            0.0053963630238108035,
+            0.027764369548110287,
+            0.001469968540767627,
+            0.015793086297019008,
+            -0.008505290826588508,
+            -0.004075888558568453,
+            -0.009166710762668385,
+            -0.002860047211143503,
+            -0.011127820563172684,
+            0.003937110717404271,
+            0.02054186721680431,
+            -0.006126180963243227,
+            0.009346433819831456,
+            0.01057504086030076,
+            0.0001650851447316893,
+            -0.012033423322524951,
+            0.0049587335923558,
+            -0.0032240390628760535,
+            -0.005568976687946761,
+            -0.0024667040429312737,
+            -0.03474027725033016,
+            0.026855947684394643,
+            -0.004369626674503285,
+            0.009252512430957932,
+            -0.005081834388513357,
+            -0.0037896610731020337,
+            -0.014201609270756739,
+            0.017871088029615106,
+            -0.008249444723433136,
+            -0.020092257756971717,
+            0.009532994544485572,
+            -0.004991989149115383,
+            0.006464417738670472,
+            -0.02643078108339741,
+            0.0029138922065812495,
+            -0.017232410749655366,
+            -0.003999293090834831,
+            -0.006924488971625995,
+            0.01318082244178942,
+            -0.007232695119856389,
+            -0.02660711037908311,
+            0.01979251106803667,
+            -0.019485860712345603,
+            -0.011977904020170568,
+            -0.0177530973032872,
+            -0.014524003566945937,
+            0.004874952064727108,
+            -0.015326044325348929,
+            0.002047771741041311,
+            -0.009697340332342246,
+            -0.003652304998155628,
+            -0.0008878698629674222,
+            0.0017782435347417623,
+            -0.013017776298713044,
+            -0.018272697230002275,
+            -0.01353020425912225,
+            -0.00757239449706798,
+            -0.0008250000619957538,
+            0.01299142281772714,
+            -0.001679606555140765,
+            -0.007329941421303076,
+            0.0044605462460153116,
+            0.010231192277666706,
+            0.0006558644644452857,
+            0.0024231733856020414,
+            -0.010728844680699885,
+            -0.014137434553169443,
+            0.005230053181525096,
+            -0.012086468475542818,
+            -0.009459717916037003,
+            -0.0028185227600210556,
+            0.014984527039312847,
+            0.008189464334213754,
+            -0.009139240774931914,
+            -0.005639157579163322,
+            0.014325949891859377,
+            0.009020303519690971,
+            -0.013310205617445609,
+            0.01567574314514385,
+            -0.025369802629268523,
+            0.021964786021408878,
+            0.030536905527039065,
+            0.004945288795196981,
+            -0.01168303908203322,
+            0.02281128628230144,
+            -0.004072286832145419,
+            0.0028278384348551146,
+            0.01740522162288968,
+            -0.042558465579514926,
+            -0.013146562547084278,
+            0.002903743316339479,
+            -0.015297177099602701,
+            -0.010963312697387696,
+            0.008804938803398296,
+            -0.0026727734429658817,
+            0.005273238046953793,
+            0.004257613402172282,
+            0.03512246556126453,
+            0.002258612643297096,
+            -0.0002885708728599208,
+            -0.0019720190285930487,
+            0.012312613129996533,
+            0.010451003939611476,
+            -0.015556620910136185,
+            -0.02387856865043814,
+            -0.004583090523619507,
+            -0.0037828776396525835,
+            -0.0014588238692793338,
+            -0.004361834167147271,
+            -0.008159679676329573,
+            0.012243319920992457,
+            -0.017279964797769668,
+            -0.013748032758693542,
+            0.018394195628121487,
+            -0.006411113522842359,
+            0.019002611495040334,
+            -0.014970362556522149,
+            -0.0077437581301661946,
+            0.0017311115715410025,
+            0.024439691379769795,
+            -0.025013956798520212,
+            0.007359134924497539,
+            -0.0031406470488477937,
+            0.017574824561755627,
+            0.00419409250132525,
+            -0.0046803792187546656,
+            0.010556972371044426,
+            0.015229789583480183,
+            -0.004608293492221483,
+            -0.014449487985966966,
+            -0.01172850291078532,
+            0.016823024209386785,
+            -0.012304887597699588,
+            0.0011773351576246157,
+            0.002262670226244277,
+            -0.0023810578015456716,
+            0.03477119000959688,
+            -0.03442178291363675,
+            0.004062693040714608,
+            0.0029249583507755944,
+            0.003291349443702718,
+            0.006461525089528308,
+            0.006047777261425826,
+            -0.020642802018978404,
+            0.008916088381876748,
+            -0.03316762661930299,
+            -0.006367254689957291,
+            -0.005404276207844267,
+            0.0009189006423526284,
+            0.009401330264376041,
+            -0.011106443887384533,
+            -0.02259405372588916,
+            -0.006010439054323966,
+            0.0070744551115721095,
+            0.010262607009446879,
+            -0.002856616377476421,
+            0.019719451109535182,
+            0.008426509638139982,
+            0.0015877648339862691,
+            -0.010655792508751512,
+            -0.015527851374834499,
+            -0.015460974859638219,
+            -0.017847462632188386,
+            -0.02236964311096428,
+            0.011841749854626384,
+            -0.00576688153869993,
+            0.009054902130717224,
+            0.014317707346504314,
+            -0.015366657284925976,
+            0.002910211008094564,
+            0.017761685988205086,
+            0.007974806171830457,
+            0.002692237107055498,
+            0.0032405477366987947,
+            0.02685230091638845,
+            -0.0009116884421301454,
+            -0.0032228977180251673,
+            0.003158213042682977,
+            0.015872199488167846,
+            0.0004966905858799325,
+            0.019981401695749582,
+            0.015201538351780685,
+            0.005359075814444414,
+            -0.004668738193285921,
+            0.019853988551405383,
+            -0.009844394286458195,
+            0.013476144827747423,
+            -0.00411789677405284,
+            -0.00830656434595337,
+            -0.008496132520586954,
+            0.006442581751999304,
+            -0.008138786054348571,
+            0.006977337946948795,
+            0.02419128791801387,
+            -0.0005734827808485311,
+            -0.0028842142002271044,
+            0.005503033191779872,
+            -0.01696639779775455,
+            -0.00389835761601045,
+            -0.03366077608692946,
+            -0.009942613075976742,
+            -0.01708630689497001,
+            -0.02910756562213045,
+            0.026536268897199546,
+            0.00219827495662445,
+            0.02586503862171508,
+            -0.01905122645609761,
+            0.008935637454859214,
+            -0.01385806889332683,
+            0.029747093931541325,
+            0.009163959012862627,
+            0.0008611776492791918,
+            0.003598831885031635,
+            -0.02103045281939351,
+            -0.01150941825152158,
+            0.027341774725604333,
+            -0.0006700608710615124,
+            -0.012089044078073347,
+            0.022396400579038037,
+            0.017491565611764764,
+            0.0051760367056280536,
+            -0.00559536785318233,
+            0.01979262595019399,
+            0.002306380546194741,
+            0.011973863166872444,
+            -0.0028136750606972223,
+            0.012911235736067303,
+            -0.01510421118359169,
+            -0.003338602157155174,
+            0.039758808572572274,
+            0.007290642519242007,
+            -0.001802921446521437,
+            -0.013696774990705774,
+            0.003941198181491921,
+            -0.0008308293948698925,
+            0.0200351513103389,
+            -0.00014327587644797734,
+            0.028291423013030337,
+            -0.0001085859276231927,
+            -0.010723064585989723,
+            0.020443848393732245,
+            -0.017860296930396913,
+            0.04540901629084378,
+            -0.018671578423261854,
+            0.000852031295629291,
+            -0.010639853151638835,
+            -0.019017194262730474,
+            0.003861296738239369,
+            0.0010977143012920602,
+            0.0008387482016061222,
+            0.0060155816314618,
+            -0.010780239332444652,
+            0.002152214497259421,
+            0.010556338812873054,
+            -0.027636192450621728,
+            0.0010938992111792307,
+            -0.014385202899929138,
+            -0.01353192238925814,
+            -0.026667778226211785,
+            -0.00017403398236549974,
+            -0.015357674131635848,
+            -0.013956722505869091,
+            -0.01985038526848704,
+            -0.008927903879720431,
+            -0.009580214158077466,
+            -0.0020314441852913386,
+            -0.004755584878514854,
+            -0.00018388548107372635,
+            -0.002918855567842974,
+            -0.0015941852262316128,
+            -0.014230272061190995,
+            0.02133992212037431,
+            0.015283345362445234,
+            0.004642765060980717,
+            0.012829570028541471,
+            0.01708952142208596,
+            -0.004770647543823141,
+            0.006604762788847952,
+            -0.0025369836845081078,
+            0.008368000846861623,
+            0.014785031726639733,
+            0.012074738887486398,
+            0.02124518119032885,
+            -0.015459818988572564,
+            -0.004992865135357168,
+            -0.012722391036055767,
+            -0.012210202603282509,
+            -0.017850065946244383,
+            -0.004300429148015694,
+            -0.0014387298144714717,
+            -0.023128300275416423,
+            0.004468952290723408,
+            -0.0026602931688877443,
+            -0.005768852559887139,
+            0.007815212168719318,
+            0.0005862972228454734,
+            -0.002265719947629544,
+            -0.00019323737674204774,
+            -0.02735820445266249,
+            0.00024009122317215594,
+            -0.00018701993387558173,
+            -0.014663710240203917,
+            -0.00450797561624596,
+            0.010584731989847334,
+            -0.011966527748437568,
+            -0.00602367292291343,
+            0.0111155226518017,
+            0.0015565179432852256,
+            0.006062862961835896,
+            0.016952559697728433,
+            0.0005012157272316869,
+            0.011160668242334889,
+            0.014993782560797772,
+            -0.01591303023837098,
+            -0.015356412848444082,
+            -0.0236025426434855,
+            -0.004514538154596937,
+            -0.009753334017065119,
+            0.007553951535816575,
+            -0.004616595532996391,
+            -0.013023153809236325,
+            0.006040285197946957,
+            -0.0006343451398642784,
+            0.008338564073158257,
+            -0.017608246105292328,
+            -8.346168119689692e-05,
+            -0.020944870308705715,
+            0.009230123321316848,
+            -0.00531237224627929,
+            -0.00445985089300067,
+            0.01410675485522911,
+            0.02895945580213926,
+            -0.007799830351886844,
+            -0.009455128322575888,
+            -0.005663072244883114,
+            0.012817229945992764,
+            0.012474431030896143,
+            -0.004423141749110039,
+            -0.0018294512146253298,
+            -0.012817101093394574,
+            0.0004755435377918658,
+            -0.01129485262081729,
+            0.0062747111125005155,
+            0.017199374798326412,
+            -0.021965117839794063,
+            -0.01296097993683748,
+            -0.016765996091978336,
+            -0.016882855472266523,
+            0.004372970343291638,
+            0.017290565252244258,
+            0.022670788364378462,
+            -0.008456547449078897,
+            -0.019564049850942417,
+            0.006898353991539946,
+            -0.01620125883985687,
+            0.0037108638122592746,
+            0.12887455743463638,
+            -0.0028864852421660366,
+            -0.011487804449328495,
+            -0.022683334123299446,
+            -0.005467192611880829,
+            0.01210557696712196,
+            -0.01555159294496261,
+            0.06709672179195036,
+            -0.015071089552631105,
+            0.004364902817496082,
+            -0.004899872946473657,
+            0.0006310685609937571,
+            -0.017610935588191347,
+            -0.005962991715597585,
+            -0.007768819050529025,
+            -0.02688607975562596,
+            0.000883763319535686,
+            -0.009935313912803587,
+            -0.012883104064962595,
+            0.002126368813790577,
+            -0.0013697265269800346,
+            -0.016131186838419288,
+            0.01575514418037863,
+            -0.020677990768697592,
+            -0.007251917191831732,
+            -0.0029325623780992118,
+            -0.00600710212646933,
+            0.014573093764769918,
+            -0.03253336117394778,
+            0.002973725493330224,
+            -0.013601417220439602,
+            0.011905788192118131,
+            0.001542117126832946,
+            -0.010789486980776022,
+            -0.013862139885514524,
+            0.007086571810314664,
+            0.015658913897173155,
+            0.017039624764452546,
+            0.01901374401756777,
+            0.004797954569087795,
+            0.003928999581093576,
+            -0.008046722893614655,
+            0.021974371139905104,
+            -0.00697335955527197,
+            0.0008975473383295958,
+            -0.00970589295353857,
+            -0.004346869073499034,
+            0.0031279458769925416,
+            0.02303983493541673,
+            -0.012560380384977343,
+            -0.01536378830925542,
+            0.0012035987157572355,
+            0.033313215063595594,
+            -0.010845084209743817,
+            0.013587126023659577,
+            0.0029867620644300073,
+            0.020679673435988446,
+            -0.0006640787253300063,
+            0.0025071780490108306,
+            0.022554193713122565,
+            -0.018082906254599396,
+            0.018230000220500718,
+            -0.004451773302171552,
+            0.01915083025790679,
+            -0.004183521043835777,
+            0.015019320101074747,
+            -0.027014055979204272,
+            -0.005362044589023088,
+            -0.012355559881051253,
+            0.003995309003198048,
+            0.020680455787871383,
+            0.0054391755324872375,
+            0.009865676762244514,
+            -0.01645386866657314,
+            -0.01760657627715638,
+            0.022969828924328126,
+            -0.021566687194482292,
+            -0.015826424849757402,
+            0.00012804774676298237,
+            -0.015021970627042762,
+            0.006000610438452896,
+            0.010126179023323158,
+            -0.02912121211344025,
+            -0.02046150879033127,
+            0.004881850039543051,
+            -0.0023469840669773042,
+            -0.00915786433296249,
+            0.006547377770403066,
+            -0.022329063637464074,
+            -0.015702777677743352,
+            -0.005692904224111613,
+            0.01207999389934878,
+            -0.008429160038192635,
+            0.010075061963131637,
+            -0.020106712664936504,
+            0.0036787045650556907,
+            -0.002764965946892891,
+            0.010307702251974629,
+            0.004697169247387421,
+            0.011192285826515207,
+            0.0030113901179382946,
+            0.0014774681536380047,
+            -0.02778270000997313,
+            0.010043261430740039,
+            -5.178406450222847e-05,
+            0.04216636255860709,
+            0.030977096295679525,
+            0.004479857028809617,
+            -5.798020745008712e-05,
+            -0.009996471945562616,
+            0.012476009017187798,
+            -0.019306907391548302,
+            -0.005092322343599383,
+            0.01856463597985508,
+            0.02195467141594795,
+            -0.02236232264139305,
+            0.0052217593371508145,
+            -0.016766955100332014,
+            0.016133299654723637,
+            -0.011208763137897244,
+            0.006248024409576809,
+            0.011601988603146743,
+            0.0022330878390492527,
+            0.010317781892717903,
+            -0.005895142237225119,
+            0.019670196605183973,
+            0.01186677200787666,
+            -0.002339367025714548,
+            0.02367408853110795,
+            0.006858134385159451,
+            0.004479683673962342,
+            0.004798719889930319,
+            -0.009604173975576537,
+            -0.009045602998213713,
+            0.009230969488015385,
+            0.004083616606774546,
+            -0.007944628589259331,
+            0.011836323287097144,
+            0.015100350195476968,
+            0.01885317701149666,
+            -0.0011707847499298092,
+            -0.00684548433712413,
+            -0.006550059418141426,
+            0.001899758393077906,
+            -0.024577381680573596,
+            -0.02855511854789239,
+            0.021238935710272536,
+            -0.010427135883958709,
+            -0.019187769881105637,
+            -0.0015864640188683347,
+            0.016360037306061392,
+            -0.008853972319021563,
+            -0.01627167079143546,
+            0.001270414822216867,
+            0.02255755055961612,
+            -0.005959411125391464,
+            0.006009654905468478,
+            0.005909233773418581,
+            0.005874361519568704,
+            -0.0027859463941992753,
+            -0.024599789577567636,
+            0.0036612374968223453,
+            0.03221346865585554,
+            0.016461563205582293,
+            0.016139782455492073,
+            -0.007158168059808459,
+            0.001838455290310879,
+            0.007530954381521755,
+            0.005438791523412295,
+            0.015674000912615536,
+            0.008604106101542998,
+            -0.006990462920227135,
+            0.00244736628387359,
+            0.012213048255418866,
+            -0.007329299946651028,
+            0.017424711328323798,
+            0.001973882532418776,
+            -0.0133178791530314,
+            -0.008373452013951822,
+            -0.010462150707776556,
+            -0.014575483571750984,
+            0.0027958704799249897,
+            -0.011401916623878957,
+            0.0008016041012669959,
+            0.003953958717263787,
+            0.010260753041492368,
+            0.013961649189650642,
+            -0.0019275791443041316,
+            0.017019168629298462,
+            0.011534594243860177,
+            -0.0017859096307142257,
+            -0.002970739203219076,
+            -0.03142778833370656,
+            0.010207789755749909,
+            -0.014730553506704018,
+            -0.0037611416875830746,
+            0.01825790347981639,
+            -0.007766890847082858,
+            -0.0032778433418881666,
+            -0.021003405429807113,
+            0.0024512381432602736,
+            0.008768616692363321,
+            0.01558713989712387,
+            -0.012819542714293536,
+            0.009201155584074711,
+            -0.0012254337002053017,
+            -0.0057451574939563045,
+            0.00977251392462179,
+            0.0042356831792011935,
+            -0.0033433823868435786,
+            0.011499764125503874
+          ],
+          "title": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer",
+          "keyphrases": [
+            "Sparsely-Gated Mixture-of-Experts Layer",
+            "conditional computation",
+            "neural network",
+            "language modeling",
+            "machine translation"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "87b82083-05f3-4d69-9ea0-97d4f551aa37",
+        "properties": {
+          "page_content": "5 Experiments 5.1 1 Billion Word Language Modeling Benchmark\n\nDataset: This dataset, introduced by (Chelba et al., 2013) consists of shuffled unique sentences from news articles, totaling approximately 829 million words, with a vocabulary of 793,471 words.\n\nPrevious State-of-the-Art: The best previously published results (Jozefowicz et al., 2016) use models consisting of one or more stacked Long Short-Term Memory (LSTM) layers (Hochreiter & Schmidhuber, 1997; Gers et al., 2000). The number of parameters in the LSTM layers of these models vary from 2 million to 151 million. Quality increases greatly with parameter count, as do computational costs. Results for these models form the top line of Figure 2-right. MoE Models: Our models consist of two stacked LSTM layers with a MoE layer between them (see Figure 1). We vary the sizes of the layers and the number of experts. For full details on model architecture, training regimen, additional baselines and results, see Appendix C.\n\nLow Computation, Varied Capacity: To investigate the effects of adding capacity, we trained\n\na series of MoE models all with roughly equal computational costs: about 8 million multiply-andadds per training example per timestep in the forwards pass, excluding the softmax layer. We call this metric (ops/timestep). We trained models with flat MoEs containing 4, 32, and 256 experts, and models with hierarchical MoEs containing 256, 1024, and 4096 experts. Each expert had about 1 million parameters. For all the MoE layers, 4 experts were active per input.\n\nThe results of these models are shown in Figure 2-left. The model with 4 always-active experts performed (unsurprisingly) similarly to the computationally-matched baseline models, while the largest of the models (4096 experts) achieved an impressive 24% lower perplexity on the test set. Figure 2: Model comparison on 1-Billion-Word Language-Modeling Benchmark. On the left, we plot test perplexity as a function of model capacity for models with similar computational budgets of approximately 8-million-ops-per-timestep. On the right, we plot test perplexity as a function of computational budget. The top line represents the LSTM models from (Jozefowicz et al., 2016).\n\nThe bottom line represents 4-billion parameter MoE models with different computational budgets.\n\nVaried Computation, High Capacity: In addition to the largest model from the previous section, we trained two more MoE models with similarly high capacity (4 billion parameters), but higher computation budgets. These models had larger LSTMs, and fewer but larger and experts. Details\n\nTest Test #Parameters ops/timestep Training TFLOPS Perplexity Perplexity excluding embedding Time /GPU 10 epochs 100 epochs and softmax layers 10 epochs Best Published Results 34.7 30.6 151 million 151 million 59 hours, 32 k40s 1.09 Low-Budget MoE Model 34.1 4303 million 8.9 million 15 hours, 16 k40s 0.74 Medium-Budget MoE Model 31.3 4313 million 33.8 million 17 hours, 32 k40s 1.22 High-Budget MoE Model 28.0 4371 million 142.7 million 47 hours, 32 k40s 1.56\n\nTable 1: Summary of high-capacity MoE-augmented models with varying computational budgets, vs. best previously published results (Jozefowicz et al., 2016). Details in Appendix C. can be found in Appendix C.2. Results of these three models form the bottom line of Figure 2-right.\n\nTable 1 compares the results of these models to the best previously-published result on this dataset .\n\nEven the fastest of these models beats the best published result (when controlling for the number of training epochs), despite requiring only 6% of the computation.\n\nComputational Efficiency: We trained our models using TensorFlow (Abadi et al., 2016) on clusters containing 16-32 Tesla K40 GPUs. For each of our models, we determine computational efficiency in TFLOPS/GPU by dividing the number of floating point operations required to process one training batch by the observed step time and the number of GPUs in the cluster. The operation counts used here are higher than the ones we report in our ops/timestep numbers in that we include the backwards pass, we include the importance-sampling-based training of the softmax layer, and we count a multiply-and-add as two separate operations. For all of our MoE models, the floating point operations involved in the experts represent between 37% and 46% of the total.\n\nFor our baseline models wtih no MoE, observed computational efficiency ranged from 1.07-1.29 TFLOPS/GPU. For our low-computation MoE models, computation efficiency ranged from 0.740.90 TFLOPS/GPU, except for the 4-expert model which did not make full use of the available parallelism. Our highest-computation MoE model was more efficient at 1.56 TFLOPS/GPU, likely due to the larger matrices. These numbers represent a significant fraction of the theoretical maximum of 4.29 TFLOPS/GPU claimed by NVIDIA. Detailed results are in Appendix C, Table 7.\n\n5.2 100 BILLION WORD GOOGLE NEWS CORPUS\n\nOn the 1-billion-word corpus, adding additional capacity seems to produce diminishing returns as the number of parameters in the MoE layer exceeds 1 billion, as can be seen in Figure 2-left. We hypothesized that for a larger training set, even higher capacities would produce significant quality improvements. We constructed a similar training set consisting of shuffled unique sentences from Google's internal news corpus, totalling roughly 100 billion words. Similarly to the previous section, we tested a series of models with similar computational costs of about 8 million ops/timestep. In addition to a baseline LSTM model, we trained models augmented with MoE layers containing 32, 256, 1024, 4096, 16384, 65536, and 131072 experts. This corresponds to up to 137 billion parameters in the MoE layer. Details on architecture, training, and results are given in Appendix D.\n\nResults: Figure 3 shows test perplexity as a function of capacity after training on 10 billion words (top line) and 100 billion words (bottom line). When training over the full 100 billion words, test perplexity improves significantly up to 65536 experts (68 billion parameters), dropping 39% lower than the computationally matched baseline, but degrades at 131072 experts, possibly a result of too much sparsity. The widening gap between the two lines demonstrates (unsurprisingly) that increased model capacity helps more on larger training sets. Even at 65536 experts (99.994% layer sparsity), computational efficiency for the model stays at a respectable 0.72 TFLOPS/GPU.\n\n5.3 Machine Translation (Single Language Pair)\n\nModel Architecture: Our model was a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decreased the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We inserted MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). Each MoE layer contained up to 2048 experts each with about two million parameters, adding a total of about 8 billion parameters to the models. Further details on model architecture, testing procedure and results can be found in Appendix E.\n\nDatasets: We benchmarked our method on the WMT'14 En→Fr and En→De corpora, whose training sets have 36M sentence pairs and 5M sentence pairs, respectively. The experimental protocols were also similar to those in (Wu et al., 2016): newstest2014 was used as the test set to compare against previous work (Luong et al., 2015a; Zhou et al., 2016; Wu et al., 2016), while the combination of newstest2012 and newstest2013 was used as the development set. We also tested the same model on a Google's Production English to French data.\n\nTable 2: Results on WMT'14 En→ Fr newstest2014 (bold values represent best results). Model Test Test ops/timenstep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.69 40.35 85M 8.7B 3 days/64 k40s MoE with 2048 Experts (longer training) 2.63 40.56 85M 8.7B 6 days/64 k40s GNMT (Wu et al., 2016) 2.79 39.22 214M 278M 6 days/96 k80s GNMT+RL (Wu et al., 2016) 2.96 39.92 214M 278M 6 days/96 k80s PBMT (Durrani et al., 2014) 37.0 LSTM (6-layer) (Luong et al., 2015b) 31.5 LSTM (6-layer+PosUnk) (Luong et al., 2015b) 33.1 DeepAtt (Zhou et al., 2016) 37.7 DeepAtt+PosUnk (Zhou et al., 2016) 39.2\n\nTable 3: Results on WMT'14 En → De newstest2014 (bold values represent best results). Model Test Test ops/timestep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 4.64 26.03 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 5.25 24.91 214M 278M 1 day/96 k80s GNMT +RL (Wu et al., 2016) 8.08 24.66 214M 278M 1 day/96 k80s PBMT (Durrani et al., 2014) 20.7 DeepAtt (Zhou et al., 2016) 20.6\n\nTable 4: Results on the Google Production En→ Fr dataset (bold values represent best results). Model Eval Eval Test Test ops/timestep Total Training Perplexity BLEU Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.60 37.27 2.69 36.57 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 2.78 35.80 2.87 35.56 214M 278M 6 days/96 k80s\n\nResults: Tables 2, 3, and 4 show the results of our largest models, compared with published results. Our approach achieved BLEU scores of 40.56 and 26.03 on the WMT'14 En→Fr and En→De benchmarks. As our models did not use RL refinement, these results constitute significant gains of 1.34 and 1.12 BLEU score on top of the strong baselines in (Wu et al., 2016). The perplexity scores are also better.2 On the Google Production dataset, our model achieved 1.01 higher test BLEU score even after training for only one sixth of the time.\n\n5.4 Multilingual Machine Translation\n\nDataset: (Johnson et al., 2016) train a single GNMT (Wu et al., 2016) model on a very large combined dataset of twelve language pairs. Results are somewhat worse than those for 12 separately trained single-pair GNMT models. This is not surprising, given that the twelve models have 12 times the capacity and twelve times the aggregate training of the one model. We repeat this experiment with a single MoE-augmented model. See Appendix E for details on model architecture.\n\nWe train our model on the same dataset as (Johnson et al., 2016) and process the same number of training examples (about 3 billion sentence pairs). Our training time was shorter due to the lower computational budget of our model.\n\nResults: Results for the single-pair GNMT models, the multilingual GNMT model and the multilingual MoE model are given in Table 5. The MoE model achieves 19% lower perplexity on the dev set than the multilingual GNMT model. On BLEU score, the MoE model significantly beats the multilingual GNMT model on 11 of the 12 language pairs (by as much as 5.84 points), and even beats the monolingual GNMT models on 8 of 12 language pairs. The poor performance on English → Korean seems to be a result of severe overtraining, as for the rarer language pairs a small number of real examples were highly oversampled in the training corpus.\n\nTable 5: Multilingual Machine Translation (bold values represent best results). GNMT-Mono GNMT-Multi MoE-Multi MoE-Multi vs. GNMT-Multi Parameters 278M / model 278M 8.7B ops/timestep 212M 212M 102M training time, hardware various 21 days, 96 k20s 12 days, 64 k40s Perplexity (dev) 4.14 3.35 -19% French → English Test BLEU 36.47 34.40 37.46 +3.06 German → English Test BLEU 31.77 31.17 34.80 +3.63 Japanese → English Test BLEU 23.41 21.62 25.91 +4.29 Korean → English Test BLEU 25.42 22.87 28.71 +5.84 Portuguese → English Test BLEU 44.40 42.53 46.13 +3.60 Spanish → English Test BLEU 38.00 36.04 39.39 +3.35 English → French Test BLEU 35.37 34.00 36.59 +2.59 English → German Test BLEU 26.43 23.15 24.53 +1.38 English → Japanese Test BLEU 23.66 21.10 22.78 +1.68 English → Korean Test BLEU 19.75 18.41 16.62 -1.79 English → Portuguese Test BLEU 38.40 37.35 37.90 +0.55 English → Spanish Test BLEU 34.50 34.25 36.21 +1.96\n\n",
+          "embedding": [
+            -0.009427943266928196,
+            0.0009052514214999974,
+            -0.031882621347904205,
+            0.02155720256268978,
+            -0.0010102161904796958,
+            0.017834007740020752,
+            -0.019673390313982964,
+            0.04869475215673447,
+            -0.022961175069212914,
+            0.03497491776943207,
+            0.036041226238012314,
+            -0.002160385251045227,
+            0.013355513103306293,
+            -0.01921132393181324,
+            -0.013746492564678192,
+            0.007433058228343725,
+            -0.00785069540143013,
+            0.004056415054947138,
+            0.003800945123657584,
+            -0.0150793781504035,
+            -0.03134946897625923,
+            -0.014439593069255352,
+            -0.04698865860700607,
+            -0.015088263899087906,
+            -0.016625525429844856,
+            0.005407072138041258,
+            0.02432960458099842,
+            -0.0009768940508365631,
+            -0.040199827402830124,
+            0.01593242585659027,
+            0.004467388149350882,
+            0.0005736961611546576,
+            0.009525688365101814,
+            -0.008481594733893871,
+            -0.0005428731674328446,
+            -0.017078706994652748,
+            0.07435724139213562,
+            -0.004718414973467588,
+            -0.03138501197099686,
+            0.03760514408349991,
+            0.028381574898958206,
+            -0.006029085721820593,
+            -0.0007786272908560932,
+            0.0035210391506552696,
+            0.0162967462092638,
+            0.003805388230830431,
+            -0.01993996649980545,
+            0.007339756470173597,
+            -0.01571027748286724,
+            -0.037356339395046234,
+            -0.006606669165194035,
+            -0.03152718394994736,
+            0.0029923280235379934,
+            -0.02411634288728237,
+            -0.0463133305311203,
+            0.014768371358513832,
+            0.028292717412114143,
+            0.02548477053642273,
+            0.06902569532394409,
+            0.034797199070453644,
+            0.01878480054438114,
+            0.0006881021545268595,
+            -0.024222973734140396,
+            -0.02600015327334404,
+            0.03351762890815735,
+            -0.012244774959981441,
+            -0.02527150884270668,
+            0.005011649336665869,
+            0.016243431717157364,
+            -0.049334537237882614,
+            -0.03643220290541649,
+            0.037498511373996735,
+            0.009730064310133457,
+            -0.005127166397869587,
+            0.015026062726974487,
+            -0.022961175069212914,
+            0.04986768960952759,
+            0.045104846358299255,
+            -0.012777929194271564,
+            -0.0024103012401610613,
+            0.013133365660905838,
+            0.017505230382084846,
+            -0.008437165059149265,
+            -0.004611784126609564,
+            0.0228723157197237,
+            -0.022374704480171204,
+            -0.014164130203425884,
+            -0.07983095943927765,
+            0.0133999427780509,
+            -0.028097227215766907,
+            -0.03525926545262337,
+            0.015585875138640404,
+            0.015976853668689728,
+            -0.0007325316546484828,
+            -0.002772401785477996,
+            0.031171750277280807,
+            -0.024080798029899597,
+            -0.003567690262570977,
+            0.001345103606581688,
+            0.04762844368815422,
+            -0.01612791419029236,
+            2.6570933187031187e-05,
+            0.04848149046301842,
+            0.01478614378720522,
+            0.027048690244555473,
+            -0.012058171443641186,
+            -0.032824527472257614,
+            0.006117944605648518,
+            0.023796450346708298,
+            0.12625092267990112,
+            0.007517474237829447,
+            -0.004080851096659899,
+            -0.03270012512803078,
+            -0.01029876247048378,
+            -0.03732079640030861,
+            0.002994549460709095,
+            -0.030158756300807,
+            -0.02031317539513111,
+            0.021361712366342545,
+            -0.017620746046304703,
+            0.03206034004688263,
+            0.02752852812409401,
+            0.019246866926550865,
+            -0.020757470279932022,
+            -0.03646774962544441,
+            0.04862366244196892,
+            -0.004369643051177263,
+            0.012031513266265392,
+            0.0041452739387750626,
+            -0.024436235427856445,
+            -0.01317779440432787,
+            0.014368506148457527,
+            -0.004580683074891567,
+            -0.02411634288728237,
+            0.03417518362402916,
+            -0.020064370706677437,
+            0.026195643469691277,
+            0.04286559671163559,
+            -0.01928240992128849,
+            -0.015434814617037773,
+            -0.002169271232560277,
+            -0.013426600024104118,
+            0.0041008442640304565,
+            -0.018535993993282318,
+            -0.05061410740017891,
+            -0.006962105166167021,
+            -0.007939554750919342,
+            0.0924844816327095,
+            -0.012831244617700577,
+            0.023245522752404213,
+            -0.02607124112546444,
+            3.4328742913203314e-05,
+            -0.015274868346750736,
+            0.0298033207654953,
+            -0.013568774797022343,
+            -0.0069754342548549175,
+            -0.01673215627670288,
+            0.02038426324725151,
+            -0.04506930336356163,
+            0.015665847808122635,
+            -0.01008550077676773,
+            -0.010938547551631927,
+            -0.008441608399152756,
+            0.02219698764383793,
+            0.0029367911629378796,
+            0.035170406103134155,
+            0.05530586466193199,
+            -0.003258905140683055,
+            -0.06625329703092575,
+            0.04009319469332695,
+            0.004449616186320782,
+            0.009005863219499588,
+            -0.00478505901992321,
+            0.001390643883496523,
+            0.002912354888394475,
+            0.027155321091413498,
+            -0.004562911577522755,
+            0.016341175884008408,
+            -0.046846482902765274,
+            0.04155048355460167,
+            0.001919355127029121,
+            0.02104181982576847,
+            0.03643220290541649,
+            0.037143077701330185,
+            0.017149792984128,
+            0.006753286812454462,
+            0.03156273066997528,
+            0.02264128252863884,
+            0.006522253155708313,
+            0.015203780494630337,
+            0.006531138904392719,
+            0.011631648056209087,
+            0.0467398539185524,
+            0.004736186470836401,
+            0.0028012811671942472,
+            -0.008152816444635391,
+            -0.014608425088226795,
+            0.060957297682762146,
+            -0.0039808847941458225,
+            0.0034966031089425087,
+            0.011924882419407368,
+            -0.04453614726662636,
+            0.006513367407023907,
+            -0.010334305465221405,
+            0.02148611471056938,
+            0.017647404223680496,
+            0.015585875138640404,
+            0.032895613461732864,
+            0.021237310022115707,
+            -0.01701650582253933,
+            -0.030798541381955147,
+            -0.02534259669482708,
+            -0.02900358848273754,
+            -0.018011726438999176,
+            -0.004891689866781235,
+            0.023831993341445923,
+            -0.008308319374918938,
+            0.020490894094109535,
+            -0.008143930695950985,
+            -0.05004540830850601,
+            0.010680856183171272,
+            0.024347376078367233,
+            0.0011462816037237644,
+            -0.03884917125105858,
+            -0.03179376199841499,
+            -0.01540815643966198,
+            -0.025360368192195892,
+            -0.009223568253219128,
+            0.029110219329595566,
+            -0.01116958074271679,
+            0.02555585838854313,
+            0.02571580372750759,
+            -0.02038426324725151,
+            0.012555781751871109,
+            0.04592235013842583,
+            -0.016270089894533157,
+            0.0072997696697711945,
+            -0.00013884223881177604,
+            -0.025147106498479843,
+            0.009321312420070171,
+            -0.0197977926582098,
+            -0.019673390313982964,
+            -0.0338730625808239,
+            -0.017140908166766167,
+            -0.016909874975681305,
+            -0.02219698764383793,
+            -0.012466922402381897,
+            0.010494251735508442,
+            0.013702062889933586,
+            0.022694597020745277,
+            0.012031513266265392,
+            -0.03536589443683624,
+            0.015665847808122635,
+            0.007130937650799751,
+            -0.022730141878128052,
+            0.06554242223501205,
+            0.045460283756256104,
+            -0.004136388190090656,
+            -0.01041427906602621,
+            -0.046704307198524475,
+            0.0391690619289875,
+            0.032095883041620255,
+            -0.057296305894851685,
+            0.03236246109008789,
+            -0.017913982272148132,
+            0.042403530329465866,
+            0.02841711975634098,
+            0.025680260732769966,
+            -0.014377391897141933,
+            0.01803838461637497,
+            -0.005087179597467184,
+            -0.04777061566710472,
+            -0.034441761672496796,
+            -0.03433513268828392,
+            0.0076729776337742805,
+            -0.003112287726253271,
+            -0.012760157696902752,
+            -0.02900358848273754,
+            0.04133722186088562,
+            0.03214919939637184,
+            0.03563247248530388,
+            0.02104181982576847,
+            0.0030700797215104103,
+            0.00974783580750227,
+            0.016616638749837875,
+            -0.01890920288860798,
+            -0.014812801033258438,
+            -0.00617126002907753,
+            -0.03870699554681778,
+            -0.01540815643966198,
+            -0.017345283180475235,
+            0.002443623496219516,
+            -0.03756960108876228,
+            0.014155244454741478,
+            0.006144602317363024,
+            0.0014572881627827883,
+            0.011214010417461395,
+            0.002150388667359948,
+            0.03664546459913254,
+            -0.0016416707076132298,
+            -0.026906516402959824,
+            0.009943326003849506,
+            0.010405393317341805,
+            0.00527822645381093,
+            -0.029892180114984512,
+            -0.013799807988107204,
+            -0.006024642381817102,
+            0.027812877669930458,
+            -0.026266731321811676,
+            -0.009436829946935177,
+            -0.0020393149461597204,
+            0.0025813549291342497,
+            0.0016538887284696102,
+            -0.02024208754301071,
+            -0.016554437577724457,
+            0.014626197516918182,
+            0.0052115824073553085,
+            0.006637769751250744,
+            -0.04396745190024376,
+            -0.0198688805103302,
+            0.05573238804936409,
+            0.01586133800446987,
+            -0.06284110993146896,
+            0.023725362494587898,
+            -0.01658109575510025,
+            -0.011853795498609543,
+            0.006913233082741499,
+            0.02504047565162182,
+            -0.008406064473092556,
+            -0.01062754075974226,
+            0.0030611937399953604,
+            -0.052995529025793076,
+            -0.019389040768146515,
+            0.0035099319647997618,
+            0.004220804199576378,
+            -0.049263447523117065,
+            -0.019833335652947426,
+            -0.007597447372972965,
+            -0.015719164162874222,
+            0.003709864802658558,
+            0.009658977389335632,
+            -0.017416371032595634,
+            -0.010423164814710617,
+            0.012928989715874195,
+            0.018020613119006157,
+            -0.02299671806395054,
+            -0.03636111691594124,
+            0.027350811287760735,
+            -0.0019337947014719248,
+            -0.008099501021206379,
+            0.0016805464401841164,
+            0.012573553249239922,
+            0.0016838787123560905,
+            0.017771806567907333,
+            -0.005153824109584093,
+            -0.032682351768016815,
+            -0.02818608656525612,
+            0.02752852812409401,
+            -0.021734919399023056,
+            0.004147495608776808,
+            0.02278345637023449,
+            0.008303876966238022,
+            -0.011187352240085602,
+            0.020046597346663475,
+            -0.0527467243373394,
+            -0.008135044947266579,
+            0.0006581122288480401,
+            -0.006269005127251148,
+            0.025307053700089455,
+            -0.016829900443553925,
+            -0.014821686781942844,
+            0.019833335652947426,
+            0.009005863219499588,
+            0.0021626066882163286,
+            0.004014207050204277,
+            0.005966884084045887,
+            0.05512814596295357,
+            0.016154572367668152,
+            -0.01197819784283638,
+            0.017860665917396545,
+            -0.024596180766820908,
+            0.03102957457304001,
+            0.021290624514222145,
+            0.00529599841684103,
+            0.001194043317809701,
+            0.04581571742892265,
+            0.01008550077676773,
+            0.008757057599723339,
+            -0.011507244780659676,
+            0.011516130529344082,
+            -0.007401957642287016,
+            -0.05594564974308014,
+            -0.00806395709514618,
+            -0.022534651681780815,
+            0.002414744347333908,
+            0.0233699269592762,
+            0.031438328325748444,
+            -0.018376048654317856,
+            -0.00924133975058794,
+            -0.0009363520657643676,
+            -0.012102600187063217,
+            0.0010907446267083287,
+            -0.004687313921749592,
+            0.014359620399773121,
+            -0.0027457440737634897,
+            0.007686306722462177,
+            0.0018615968292579055,
+            -0.003058972302824259,
+            0.021343940868973732,
+            -0.03257572278380394,
+            -0.011853795498609543,
+            -0.03788949176669121,
+            -0.059784360229969025,
+            -0.03449507802724838,
+            0.030709682032465935,
+            0.023725362494587898,
+            -0.002539146924391389,
+            -0.004589569289237261,
+            -0.014324076473712921,
+            -0.031438328325748444,
+            -0.021592745557427406,
+            0.0006164595251902938,
+            0.04894355684518814,
+            -0.007868467830121517,
+            -0.02578689157962799,
+            0.01761186122894287,
+            -0.006939890794456005,
+            -0.004389636218547821,
+            -0.004927233327180147,
+            -0.03636111691594124,
+            0.023405469954013824,
+            0.010174359194934368,
+            0.03053196519613266,
+            -0.022143671289086342,
+            0.005575904622673988,
+            -0.006095729768276215,
+            -0.014377391897141933,
+            0.029963266104459763,
+            -0.008859246037900448,
+            0.004727300722151995,
+            0.0058869109489023685,
+            -0.003043422009795904,
+            0.0001384951319778338,
+            -0.005260454956442118,
+            -0.032540179789066315,
+            -0.026017924770712852,
+            -0.041230592876672745,
+            0.002370314672589302,
+            -0.02825717255473137,
+            -0.027137549594044685,
+            -0.06650210171937943,
+            0.024880530312657356,
+            -0.07098059356212616,
+            -0.02498716115951538,
+            0.038529276847839355,
+            -0.008885903283953667,
+            0.01768294721841812,
+            -0.04279451072216034,
+            -0.0006664427346549928,
+            -0.05107617378234863,
+            -0.0009580114856362343,
+            0.035241492092609406,
+            0.0029234623070806265,
+            -0.006393407471477985,
+            0.014368506148457527,
+            -0.00013120591756887734,
+            -0.03389083594083786,
+            -0.04229690134525299,
+            0.0038276028353720903,
+            0.005207139533013105,
+            -0.02301448956131935,
+            -0.014288532547652721,
+            0.04339875280857086,
+            0.00908583588898182,
+            -0.0006070183007977903,
+            0.0300876684486866,
+            0.007939554750919342,
+            -0.027137549594044685,
+            0.019246866926550865,
+            0.02564471773803234,
+            0.038884714245796204,
+            -0.02768847532570362,
+            0.021130679175257683,
+            -0.03200702369213104,
+            -0.005629220046103001,
+            0.011569445952773094,
+            0.0023636503610759974,
+            -0.006011313758790493,
+            0.014457364566624165,
+            -0.0015683620003983378,
+            0.0034077439922839403,
+            0.006842145696282387,
+            -0.006304548587650061,
+            0.032984472811222076,
+            -0.0004387414955999702,
+            -0.016821015626192093,
+            -0.021148450672626495,
+            0.009694520384073257,
+            0.031082890927791595,
+            0.028737012296915054,
+            -0.018304960802197456,
+            -0.014635083265602589,
+            -0.003976441919803619,
+            0.029056904837489128,
+            0.016083484515547752,
+            -0.01384423766285181,
+            0.0067444005981087685,
+            0.02431183122098446,
+            -0.023618731647729874,
+            -0.01586133800446987,
+            0.017398599535226822,
+            -0.006424508057534695,
+            0.012564667500555515,
+            -0.012644641101360321,
+            0.00200932496227324,
+            0.017931753769516945,
+            -0.005367085803300142,
+            0.026693254709243774,
+            -0.017034277319908142,
+            0.010529795661568642,
+            -0.007459715940058231,
+            -0.008681527338922024,
+            0.007917339913547039,
+            0.06948776543140411,
+            -0.020615296438336372,
+            0.022659054026007652,
+            0.018464908003807068,
+            -0.0018671504221856594,
+            0.034726109355688095,
+            -0.04457169026136398,
+            -0.046704307198524475,
+            0.012182573787868023,
+            -0.04314994812011719,
+            -0.0020493115298449993,
+            -0.022410249337553978,
+            -0.023032261058688164,
+            0.008321648463606834,
+            0.011915996670722961,
+            -0.024880530312657356,
+            -0.008717071264982224,
+            0.020864101126790047,
+            0.01577247865498066,
+            -0.02169937640428543,
+            0.019922195002436638,
+            0.00920579582452774,
+            -0.018749255686998367,
+            0.032682351768016815,
+            -0.02205481193959713,
+            -0.0003795946831814945,
+            -0.007099837064743042,
+            -0.0024947174824774265,
+            -0.00668219942599535,
+            -0.004285227041691542,
+            0.008868131786584854,
+            -0.02893250249326229,
+            -0.01622566021978855,
+            0.01680324412882328,
+            -0.00772185018286109,
+            -0.010058842599391937,
+            -0.042403530329465866,
+            0.0032611265778541565,
+            0.013782036490738392,
+            -0.023281067609786987,
+            0.01132064126431942,
+            -0.019477900117635727,
+            -0.032895613461732864,
+            0.01993996649980545,
+            0.03508154675364494,
+            -0.032984472811222076,
+            0.0016561102820560336,
+            0.00873040035367012,
+            -0.009028078056871891,
+            -0.0024502878077328205,
+            -0.01766517572104931,
+            0.007157595362514257,
+            0.03271789476275444,
+            0.014741714112460613,
+            -0.013257768005132675,
+            -0.024222973734140396,
+            -0.017807351425290108,
+            0.021379483863711357,
+            -0.0059313406236469746,
+            0.005162709858268499,
+            0.02562694624066353,
+            0.041514940559864044,
+            0.012173688039183617,
+            0.0008830366423353553,
+            -0.013666519895195961,
+            0.014803915284574032,
+            -0.015887996181845665,
+            0.021646060049533844,
+            0.022072583436965942,
+            -0.01746080070734024,
+            -0.03803166747093201,
+            0.018429363146424294,
+            0.00022339716088026762,
+            0.004554025363177061,
+            0.008112830109894276,
+            0.02768847532570362,
+            -0.016616638749837875,
+            -0.00549593148753047,
+            0.03934678062796593,
+            0.016314519569277763,
+            -0.014892774634063244,
+            -0.01622566021978855,
+            0.016838787123560905,
+            0.0074419439770281315,
+            0.006060186307877302,
+            -0.0018293854082003236,
+            -0.010369849391281605,
+            -0.0016127914423123002,
+            0.009365742094814777,
+            -0.008481594733893871,
+            0.008992534130811691,
+            0.014839459210634232,
+            0.005438172724097967,
+            0.0011035181814804673,
+            -0.01737194135785103,
+            -0.01263575442135334,
+            0.015799136832356453,
+            0.010538681410253048,
+            -0.010023298673331738,
+            0.00927688367664814,
+            0.007961769588291645,
+            0.014013069681823254,
+            0.032166969031095505,
+            -0.0031278380192816257,
+            -0.01671438477933407,
+            -0.0026657709386199713,
+            -0.0006958773010410368,
+            -0.025769120082259178,
+            0.007601890247315168,
+            -0.014892774634063244,
+            -0.02898581698536873,
+            0.018429363146424294,
+            0.02249910682439804,
+            -0.011418386362493038,
+            -0.01802949793636799,
+            -0.0030878514517098665,
+            -0.0043563139624893665,
+            -0.006264561787247658,
+            0.007406400516629219,
+            -0.011276211589574814,
+            0.042616792023181915,
+            -0.001704982714727521,
+            0.0023614289239048958,
+            -0.02703091874718666,
+            -0.0023503215052187443,
+            0.04332766681909561,
+            0.025253737345337868,
+            0.0030545294284820557,
+            0.009116937406361103,
+            0.008010641671717167,
+            0.0467398539185524,
+            -0.013764264062047005,
+            -0.04133722186088562,
+            0.008601554669439793,
+            -0.015319298021495342,
+            0.012173688039183617,
+            0.03266458213329315,
+            -0.009436829946935177,
+            0.010014412924647331,
+            0.010547567158937454,
+            -0.03147387132048607,
+            0.027350811287760735,
+            -0.023796450346708298,
+            -0.022605737671256065,
+            0.030994031578302383,
+            -0.01723865233361721,
+            0.01074305735528469,
+            0.028737012296915054,
+            0.010911889374256134,
+            -0.029838863760232925,
+            0.0013817580183967948,
+            -0.010423164814710617,
+            -0.025218194350600243,
+            0.025911293923854828,
+            0.02863038145005703,
+            0.018642624840140343,
+            0.0063001057133078575,
+            0.0015339291421696544,
+            0.004896132741123438,
+            0.0033699788618832827,
+            -0.01007661409676075,
+            0.013204452581703663,
+            -0.010165473446249962,
+            -0.01768294721841812,
+            -0.0015283754328265786,
+            -0.013568774797022343,
+            -0.00927688367664814,
+            0.0021748249419033527,
+            -0.02607124112546444,
+            0.009134708903729916,
+            0.012351405806839466,
+            0.010458708740770817,
+            -0.018962517380714417,
+            -0.01862485334277153,
+            0.05534140765666962,
+            0.013799807988107204,
+            -0.0032700125593692064,
+            0.004087515641003847,
+            -0.017585203051567078,
+            -0.04318549111485481,
+            0.004571797326207161,
+            0.011418386362493038,
+            0.005562575533986092,
+            0.00020270967797841877,
+            -0.030496420338749886,
+            0.008108386769890785,
+            0.032326918095350266,
+            0.008046185597777367,
+            0.04446506127715111,
+            -0.0012251439038664103,
+            0.030123213306069374,
+            -0.008414950221776962,
+            0.003574354574084282,
+            0.019513443112373352,
+            0.0041230591014027596,
+            -0.019033605232834816,
+            0.03038978949189186,
+            0.01834050565958023,
+            0.018304960802197456,
+            0.006979877129197121,
+            -0.04514038935303688,
+            -0.009872239083051682,
+            0.010236560367047787,
+            0.03182930499315262,
+            0.020259859040379524,
+            -0.017176451161503792,
+            -0.007437501102685928,
+            0.03296670317649841,
+            -0.022232530638575554,
+            -0.004431844223290682,
+            0.0116138756275177,
+            -0.008597111329436302,
+            0.002483610063791275,
+            -0.004687313921749592,
+            0.02249910682439804,
+            -0.039844390004873276,
+            -0.004687313921749592,
+            0.0019349055364727974,
+            0.0057225218042731285,
+            -0.002215922111645341,
+            -0.006939890794456005,
+            0.010405393317341805,
+            0.014261875301599503,
+            0.009076950140297413,
+            0.02943011187016964,
+            -0.0201532281935215,
+            0.015034948475658894,
+            -0.015168237499892712,
+            -0.003134502563625574,
+            0.005749179515987635,
+            -0.03138501197099686,
+            -0.030283158645033836,
+            0.007619662210345268,
+            0.03175821900367737,
+            0.013080050237476826,
+            0.0014817244373261929,
+            -0.004589569289237261,
+            -0.011516130529344082,
+            0.026853200048208237,
+            0.02109513431787491,
+            0.02111290581524372,
+            0.023831993341445923,
+            -0.005087179597467184,
+            -0.0004323547473177314,
+            0.012475808151066303,
+            -0.008650426752865314,
+            -0.024791670963168144,
+            0.008930332958698273,
+            -0.011427272111177444,
+            0.0270842332392931,
+            -0.01724753901362419,
+            -0.003258905140683055,
+            0.005646991543471813,
+            -0.012404721230268478,
+            -0.01128509733825922,
+            -0.039702218025922775,
+            0.025893522426486015,
+            -0.02548477053642273,
+            0.0036343345418572426,
+            -0.019193550571799278,
+            0.01490166038274765,
+            -0.03870699554681778,
+            0.000813615508377552,
+            -0.0018393819918856025,
+            0.010538681410253048,
+            -0.010529795661568642,
+            0.0021770463790744543,
+            -0.03193593770265579,
+            -0.03788949176669121,
+            -0.0116849634796381,
+            0.005073850974440575,
+            0.007899568416178226,
+            -0.0009330198518000543,
+            0.006913233082741499,
+            0.015967968851327896,
+            -0.025093792006373405,
+            -0.03369534760713577,
+            0.018820343539118767,
+            -0.03440621867775917,
+            0.020419806241989136,
+            0.0004870585980825126,
+            -0.02804391086101532,
+            -0.021965952590107918,
+            -0.016492236405611038,
+            -0.015248210169374943,
+            0.021006274968385696,
+            0.022552423179149628,
+            0.017798464745283127,
+            -0.016847673803567886,
+            0.0025124892126768827,
+            0.014039727859199047,
+            0.008121715858578682,
+            0.011880452744662762,
+            0.008481594733893871,
+            0.01825164631009102,
+            -0.016847673803567886,
+            -0.015887996181845665,
+            -0.012244774959981441,
+            -0.002794616622850299,
+            -0.013799807988107204,
+            0.0011179577559232712,
+            -0.038813624531030655,
+            -0.00737529993057251,
+            0.01116958074271679,
+            -0.0020048818551003933,
+            0.06038859859108925,
+            0.0527467243373394,
+            0.013160022906959057,
+            0.007384185679256916,
+            -0.01876702904701233,
+            0.0461711548268795,
+            -0.025804663076996803,
+            0.04986768960952759,
+            0.0034521734341979027,
+            -0.013142251409590244,
+            -0.013462143950164318,
+            0.013426600024104118,
+            -0.029199078679084778,
+            -0.00719313882291317,
+            -0.005424844101071358,
+            -0.009005863219499588,
+            0.018002839758992195,
+            0.00739307189360261,
+            -0.009347970597445965,
+            0.003696535946801305,
+            0.009383514523506165,
+            -0.011258440092206001,
+            0.013222224079072475,
+            0.01612791419029236,
+            0.0017349726986140013,
+            0.015070492401719093,
+            -0.04208363965153694,
+            -0.011231781914830208,
+            -0.00041291682282462716,
+            0.009685634635388851,
+            -0.006220132578164339,
+            0.013168908655643463,
+            -0.006428950931876898,
+            -0.0044229584746062756,
+            -0.007015420589596033,
+            -0.02169937640428543,
+            0.022534651681780815,
+            0.036041226238012314,
+            -0.0016938752960413694,
+            -0.048979099839925766,
+            -0.02614232711493969,
+            -0.050223127007484436,
+            -0.012928989715874195,
+            0.024063026532530785,
+            -0.0050249784253537655,
+            0.015976853668689728,
+            0.004087515641003847,
+            0.0048828041180968285,
+            -0.0030456434469670057,
+            -0.017567431554198265,
+            0.01613680087029934,
+            -0.0232277512550354,
+            0.002739079762250185,
+            0.0003854260721709579,
+            0.007144266273826361,
+            -0.03543698415160179,
+            0.007757393643260002,
+            0.00023422685626428574,
+            0.012093714438378811,
+            -0.03660992160439491,
+            -0.038244929164648056,
+            -0.017398599535226822,
+            0.015941310673952103,
+            -0.002383643528446555,
+            -0.012244774959981441,
+            0.0029501200187951326,
+            0.005033864174038172,
+            0.014741714112460613,
+            -0.017567431554198265,
+            -0.006229018326848745,
+            0.02162828855216503,
+            -0.0032433548476547003,
+            0.007206467911601067,
+            -0.035099320113658905,
+            -0.01555921696126461,
+            -0.01278681494295597,
+            0.0116494195535779,
+            -0.0133999427780509,
+            0.004905018489807844,
+            0.00017799575289245695,
+            0.014848344959318638,
+            -0.015505901537835598,
+            -0.027013147249817848,
+            -0.011231781914830208,
+            0.020099913701415062,
+            -0.03511708974838257,
+            0.036112312227487564,
+            0.002739079762250185,
+            0.02118399366736412,
+            -0.014404049143195152,
+            0.025609172880649567,
+            -0.004036421421915293,
+            -0.015301525592803955,
+            -0.002151499269530177,
+            0.022001497447490692,
+            0.03332213684916496,
+            -0.021361712366342545,
+            0.0013228888856247067,
+            -0.019015833735466003,
+            0.003327770857140422,
+            -0.02351210080087185,
+            0.017345283180475235,
+            -0.004007542505860329,
+            0.01234252005815506,
+            0.017105363309383392,
+            0.013204452581703663,
+            0.05996207520365715,
+            -0.017194222658872604,
+            0.02557362988591194,
+            0.012324748560786247,
+            -0.017265310510993004,
+            -0.007935111410915852,
+            -0.032095883041620255,
+            -0.027333039790391922,
+            0.03133169561624527,
+            0.002228140365332365,
+            0.0015561438631266356,
+            -0.03104734607040882,
+            0.015879109501838684,
+            0.01832273229956627,
+            0.019460128620266914,
+            -0.014288532547652721,
+            -0.015274868346750736,
+            0.016030170023441315,
+            0.026248957961797714,
+            -0.008135044947266579,
+            0.003865367965772748,
+            0.012769043445587158,
+            -0.04076852649450302,
+            0.005282669793814421,
+            0.0029634488746523857,
+            0.012857901863753796,
+            -0.0021548315417021513,
+            0.010183244943618774,
+            -0.03214919939637184,
+            -0.0055137029848992825,
+            -0.01457288209348917,
+            0.02315666526556015,
+            -0.015665847808122635,
+            -0.047806162387132645,
+            -0.03161604329943657,
+            0.025147106498479843,
+            -0.030194299295544624,
+            -0.020988503471016884,
+            0.01311559323221445,
+            0.0022170329466462135,
+            0.015337069518864155,
+            -0.01471505593508482,
+            -0.02747521363198757,
+            -0.038955800235271454,
+            -0.021965952590107918,
+            -0.0006886575138196349,
+            0.0394534096121788,
+            -0.03010544180870056,
+            0.015159351751208305,
+            0.005309327505528927,
+            0.00686436053365469,
+            -0.004869475029408932,
+            0.002365871798247099,
+            -0.029447883367538452,
+            -0.04133722186088562,
+            0.02351210080087185,
+            -0.00494944816455245,
+            -0.0015150465769693255,
+            0.014803915284574032,
+            -0.033055562525987625,
+            -0.023103348910808563,
+            0.003154495730996132,
+            0.00737529993057251,
+            -0.00785069540143013,
+            -0.002352542942389846,
+            0.0013395498972386122,
+            -0.01215591561049223,
+            -0.038600362837314606,
+            -0.004416293930262327,
+            0.03397969529032707,
+            0.02177046425640583,
+            0.031740445643663406,
+            0.011525017209351063,
+            0.00010468705295352265,
+            -0.02886141464114189,
+            -0.013595432043075562,
+            -0.006895461119711399,
+            -0.0009818923426792026,
+            -0.0025769120547920465,
+            0.01848267950117588,
+            0.005291555542498827,
+            0.0034077439922839403,
+            -0.02169937640428543,
+            -0.02783064916729927,
+            -0.03511708974838257,
+            -0.01928240992128849,
+            -0.010049956850707531,
+            -0.00949014537036419,
+            -0.03899134323000908,
+            -0.016741042956709862,
+            0.04485604166984558,
+            0.026995373889803886,
+            -0.009152480401098728,
+            -0.00018729818111751229,
+            -0.0012240331852808595,
+            -0.012973419390618801,
+            0.019886652007699013,
+            -0.013826466165482998,
+            0.0059091257862746716,
+            0.00031656029750593007,
+            0.018607081845402718,
+            0.0017538551473990083,
+            0.0010013302089646459,
+            -0.03419295698404312,
+            -0.009401286020874977,
+            0.031171750277280807,
+            0.00701097771525383,
+            -0.019531216472387314,
+            0.0163678340613842,
+            0.009259111247956753,
+            -0.014466251246631145,
+            -0.047806162387132645,
+            0.026782112196087837,
+            0.04567354544997215,
+            0.02212589979171753,
+            -0.005940226372331381,
+            0.022179214283823967,
+            0.024933844804763794,
+            -0.0012462479062378407,
+            -0.0007541910745203495,
+            -0.004945005290210247,
+            0.026977602392435074,
+            0.006535581778734922,
+            -0.000127387756947428,
+            0.00939240027219057,
+            0.01810947060585022,
+            0.005838038399815559,
+            -0.0007697413675487041,
+            0.007077622227370739,
+            -0.020793013274669647,
+            -0.004913904704153538,
+            -0.013853123411536217,
+            0.010974090546369553,
+            0.00920579582452774,
+            0.041088417172431946,
+            0.008774830028414726,
+            0.013559889048337936,
+            -0.0043563139624893665,
+            -0.010174359194934368,
+            0.028648152947425842,
+            0.014039727859199047,
+            0.001577247865498066,
+            0.008499366231262684,
+            -0.030834084376692772,
+            -0.01586133800446987,
+            -0.013950868509709835,
+            0.01384423766285181,
+            0.0037520728074014187,
+            0.04286559671163559,
+            0.006873246282339096,
+            0.0016794357215985656,
+            -0.02943011187016964,
+            0.0026235629338771105,
+            0.008232789114117622,
+            0.02285454422235489,
+            -0.013266653753817081,
+            -2.6171763238380663e-05,
+            0.0073664141818881035,
+            0.005638105794787407,
+            0.026497764512896538,
+            -0.026195643469691277,
+            -0.017052048817276955,
+            0.019300181418657303,
+            0.09511470794677734,
+            -0.009330199100077152,
+            -0.0233699269592762,
+            -0.026053469628095627,
+            -0.02104181982576847,
+            0.008241675794124603,
+            0.017585203051567078,
+            0.005429286975413561,
+            0.017443029209971428,
+            0.010218788869678974,
+            0.01788732409477234,
+            0.015603646636009216,
+            -0.015976853668689728,
+            -0.007419729605317116,
+            -0.03225582838058472,
+            0.0327356681227684,
+            0.00014911657490301877,
+            -0.0047228578478097916,
+            0.00739307189360261,
+            0.007135380525141954,
+            0.019744478166103363,
+            0.018091699108481407,
+            0.015887996181845665,
+            -0.007979541085660458,
+            -0.011924882419407368,
+            0.020135456696152687,
+            -0.013142251409590244,
+            -0.0034810525830835104,
+            -0.01899806223809719,
+            -0.021592745557427406,
+            -0.005651434417814016,
+            0.014643969014286995,
+            -0.006468937732279301,
+            0.014848344959318638,
+            -0.01132064126431942,
+            0.028310488909482956,
+            -0.021965952590107918,
+            0.011160694994032383,
+            -0.00737529993057251,
+            -0.01526598259806633,
+            0.005216025281697512,
+            0.010245447047054768,
+            -0.007677420508116484,
+            0.010716399177908897,
+            -0.014324076473712921,
+            0.014821686781942844,
+            0.019477900117635727,
+            -0.03650329262018204,
+            0.03328659385442734,
+            0.008126158267259598,
+            -0.009836695156991482,
+            0.00924133975058794,
+            -0.00582470977678895,
+            0.012529123574495316,
+            0.007268669083714485,
+            0.00806395709514618,
+            0.008352749049663544,
+            0.00817947369068861,
+            -0.007659649010747671,
+            0.014883887954056263,
+            0.012138144113123417,
+            0.0007353085093200207,
+            0.010636426508426666,
+            0.014279646798968315,
+            -0.009730064310133457,
+            -0.004696200136095285,
+            0.0038364888168871403,
+            -0.010325419716536999,
+            0.007317541632801294,
+            -0.009108050726354122,
+            0.0060912868939340115,
+            0.024507321417331696,
+            -0.0017083149868994951,
+            -0.016492236405611038,
+            -0.016625525429844856,
+            -0.007935111410915852,
+            -0.021237310022115707,
+            -0.015434814617037773,
+            0.00514938123524189,
+            -0.030834084376692772,
+            0.013053392060101032,
+            0.028665924444794655,
+            0.0233699269592762,
+            -0.006180145777761936,
+            -0.024720583111047745,
+            -0.012413606978952885,
+            -0.0018582645570859313,
+            0.011276211589574814,
+            0.005162709858268499,
+            -0.02454286627471447,
+            0.002643556334078312,
+            -0.011356184259057045,
+            -0.021432798355817795,
+            -0.008277218788862228,
+            -0.01651000790297985,
+            0.003458837978541851,
+            0.0030456434469670057,
+            -0.008441608399152756,
+            -0.010707513429224491,
+            0.003541032550856471,
+            -0.03685872629284859,
+            0.014048613607883453,
+            0.03207810968160629,
+            -0.000934130628593266,
+            -0.03255794942378998,
+            -0.015221552923321724,
+            -0.003250019159168005,
+            -0.017194222658872604,
+            -0.02863038145005703,
+            0.008934776298701763,
+            0.006104615516960621,
+            -0.008454937487840652,
+            0.01252023782581091,
+            -0.004036421421915293,
+            0.02001105435192585,
+            0.0009130265680141747,
+            0.01869594119489193,
+            -0.0408751554787159,
+            0.012564667500555515,
+            0.019158007577061653,
+            -0.019122464582324028,
+            0.006904346868395805,
+            0.023458784446120262,
+            -0.008383849635720253,
+            0.0034699453972280025,
+            0.035756874829530716,
+            0.010369849391281605,
+            -0.042474620044231415,
+            0.01905137673020363,
+            0.006082401145249605,
+            -0.00668219942599535,
+            0.016323404386639595,
+            0.019673390313982964,
+            -0.000788623932749033,
+            0.020757470279932022,
+            -0.006446722894906998,
+            -0.018233874812722206,
+            0.00218260008841753,
+            0.020846329629421234,
+            -0.009330199100077152,
+            -0.015345955267548561,
+            0.01092077512294054,
+            0.0027235294692218304,
+            0.004758401308208704,
+            0.01738082803785801,
+            -0.02607124112546444,
+            -0.018642624840140343,
+            0.019513443112373352,
+            0.012831244617700577,
+            0.017336398363113403,
+            -0.011214010417461395,
+            -0.01402195543050766,
+            0.02219698764383793,
+            -0.005464830435812473,
+            0.010707513429224491,
+            -0.031082890927791595,
+            0.008748171851038933,
+            -0.0019904423970729113,
+            0.01848267950117588,
+            -0.004918347578495741,
+            0.008752615191042423,
+            0.03255794942378998,
+            -0.015514787286520004,
+            -0.025964610278606415,
+            0.009543460793793201,
+            -0.033624257892370224,
+            -0.010094386525452137,
+            -0.029607830569148064,
+            -0.03920460492372513,
+            0.03170490264892578,
+            -0.008046185597777367,
+            -0.0049583339132368565,
+            0.026746569201350212,
+            0.007353085093200207,
+            0.0020270966924726963,
+            -0.03213142603635788,
+            0.021574974060058594,
+            0.006642212625592947,
+            0.007784051354974508,
+            -0.018873659893870354,
+            0.025147106498479843,
+            -0.0030922943260520697,
+            -0.021166222169995308,
+            0.013462143950164318,
+            0.012182573787868023,
+            0.008126158267259598,
+            -0.004534032195806503,
+            0.001586133730597794,
+            -0.03682318329811096,
+            -0.005775837227702141,
+            0.008246118202805519,
+            0.005522589199244976,
+            -0.003236690303310752,
+            0.018802572041749954,
+            -0.016909874975681305,
+            -0.002843489171937108,
+            -0.005629220046103001,
+            -0.022747913375496864,
+            0.0033855291549116373,
+            -0.03248686343431473,
+            -0.020686382427811623,
+            0.019780021160840988,
+            -0.003989770542830229,
+            -0.01825164631009102,
+            0.014581767842173576,
+            -0.03980884701013565,
+            -0.011178466491401196,
+            0.007304212544113398,
+            0.027777334675192833,
+            -0.03447730466723442,
+            0.015719164162874222,
+            -0.01958453096449375,
+            -0.0023769792169332504,
+            0.023618731647729874,
+            -0.011809365823864937,
+            -0.00010725563333835453,
+            -0.012564667500555515,
+            -0.031882621347904205,
+            0.003963112831115723,
+            0.028239401057362556,
+            -0.029234621673822403,
+            0.0009446826297789812,
+            0.031136205419898033,
+            -0.000653669296298176,
+            0.004616227000951767,
+            0.0007586340070702136,
+            0.01113403681665659,
+            -0.015186008997261524,
+            0.015106036327779293,
+            0.005838038399815559,
+            0.00582470977678895,
+            -0.01372872106730938,
+            -0.005202696658670902,
+            0.02710200473666191,
+            0.001230697613209486,
+            -0.008819258771836758,
+            0.0047006430104374886,
+            0.004058636259287596,
+            0.010174359194934368,
+            -0.0016627747099846601,
+            0.014457364566624165,
+            0.018100585788488388,
+            -0.028168313205242157,
+            0.012884560041129589,
+            0.009836695156991482,
+            7.844586798455566e-05,
+            0.013995298184454441,
+            -0.017514115199446678,
+            -0.034352902323007584,
+            0.02161051705479622,
+            -0.022019268944859505,
+            -0.016234545037150383,
+            0.0270842332392931,
+            -0.010867459699511528,
+            -0.006064629182219505,
+            -0.009774493984878063,
+            -0.001900472561828792,
+            0.010609768331050873,
+            -0.009978869929909706,
+            -0.009463487192988396,
+            0.023760905489325523,
+            0.018145015463232994,
+            0.020917417481541634,
+            0.029856635257601738,
+            0.0012118150480091572,
+            0.01570139080286026,
+            -0.0005223245243541896,
+            0.008974762633442879,
+            0.005171595606952906,
+            -0.0018627075478434563,
+            -0.01738971285521984,
+            0.025982381775975227,
+            -0.001804949133656919,
+            0.006264561787247658,
+            0.0063223205506801605,
+            -0.014368506148457527,
+            -0.010867459699511528,
+            -0.026479993015527725,
+            -0.004887246992439032,
+            -0.006580011453479528,
+            -0.016234545037150383,
+            0.0789068192243576,
+            -0.0099344402551651,
+            -0.016554437577724457,
+            0.010494251735508442,
+            -0.004856146406382322,
+            0.0015528115909546614,
+            -0.016847673803567886,
+            0.003736522514373064,
+            -0.008783715777099133,
+            -0.000988556770607829,
+            -0.030176527798175812,
+            -0.0015616975724697113,
+            0.0167943574488163,
+            0.015283754095435143,
+            -0.006557796616107225,
+            -0.018731484189629555,
+            0.008223903365433216,
+            -0.023209979757666588,
+            0.02036648988723755,
+            0.0012418050318956375,
+            0.018447136506438255,
+            0.005087179597467184,
+            0.005082736723124981,
+            -0.014946090057492256,
+            -0.016749927774071693,
+            0.004434065893292427,
+            0.012315861880779266,
+            0.002247022930532694,
+            -0.0090236347168684,
+            -0.009152480401098728,
+            -0.011009634472429752,
+            -0.021859321743249893,
+            0.020544208586215973,
+            -0.01296453271061182,
+            -0.0039142402820289135,
+            -0.003983106464147568,
+            -0.0020726369693875313,
+            0.006331206299364567,
+            0.011356184259057045,
+            -0.02358318865299225,
+            0.0018849222688004375,
+            -0.016972076147794724,
+            0.006913233082741499,
+            0.02504047565162182,
+            0.00981892365962267,
+            -0.04720192030072212,
+            -0.00651781028136611,
+            -0.02754630148410797,
+            0.013746492564678192,
+            -0.011747164651751518,
+            -0.009445715695619583,
+            0.0133999427780509,
+            0.008072842843830585,
+            0.028719240799546242,
+            0.010280990041792393,
+            0.009881124831736088,
+            0.015372613444924355,
+            -0.00032850075513124466,
+            -0.004440730437636375,
+            -0.017647404223680496,
+            0.013568774797022343,
+            -0.023316610604524612,
+            0.020348718389868736,
+            -0.0006103505147621036,
+            -0.017034277319908142,
+            0.004194146487861872,
+            0.002821274334564805,
+            0.013853123411536217,
+            -0.021806007251143456,
+            -0.022481335327029228,
+            0.022730141878128052,
+            0.018216101452708244,
+            0.012600211426615715,
+            0.015123807825148106,
+            0.03182930499315262,
+            0.02139725536108017,
+            0.024294059723615646,
+            -0.025324825197458267,
+            0.0014428485883399844,
+            -0.033855292946100235,
+            -0.005087179597467184,
+            -0.026551079005002975,
+            0.004038643091917038,
+            -0.012973419390618801,
+            0.005638105794787407,
+            -0.022001497447490692,
+            0.001768294838257134,
+            0.017549660056829453,
+            -0.030691910535097122,
+            0.010965204797685146,
+            0.015754707157611847,
+            0.00788623932749033,
+            0.017993954941630363,
+            0.006944333668798208,
+            -0.012831244617700577,
+            -0.01737194135785103,
+            -0.008717071264982224,
+            -0.013782036490738392,
+            0.014812801033258438,
+            0.0020082141272723675,
+            -0.0027679589111357927,
+            -0.00027990597300231457,
+            0.0001258604897884652,
+            -0.014466251246631145,
+            -0.001746080000884831,
+            -0.005180481821298599,
+            -0.013710948638617992,
+            0.005926897749304771,
+            0.0014928317395970225,
+            0.013222224079072475,
+            0.008917003870010376,
+            -0.007286441046744585,
+            0.011293983086943626,
+            0.02578689157962799,
+            0.0005528698093257844,
+            -0.018358277156949043,
+            0.00721091078594327,
+            0.008290547877550125,
+            0.007215353660285473,
+            0.006073514930903912,
+            0.0015172680141404271,
+            -0.012253660708665848,
+            0.027333039790391922,
+            -0.023938624188303947,
+            0.02205481193959713,
+            0.004856146406382322,
+            -0.0008075064979493618,
+            0.0003068413643632084,
+            -0.03518817946314812,
+            0.002401415491476655,
+            -0.024222973734140396,
+            -0.04112396016716957,
+            -0.02411634288728237,
+            -0.01547924429178238,
+            0.0007264225860126317,
+            0.022765684872865677,
+            0.0012862344738095999,
+            -0.027297494933009148,
+            0.022907858714461327,
+            -0.001922687399201095,
+            -0.015959082171320915,
+            0.00420081103220582,
+            -0.001362875453196466,
+            0.0019515665480867028,
+            0.0028745897579938173,
+            -0.0029567843303084373,
+            -0.0081839170306921,
+            -0.017407484352588654,
+            -0.02635558880865574,
+            0.003163381712511182,
+            0.009223568253219128,
+            -0.019815564155578613,
+            0.02249910682439804,
+            -0.009978869929909706,
+            -0.015879109501838684,
+            0.026497764512896538,
+            -0.009312426671385765,
+            -0.021343940868973732,
+            -0.021308396011590958,
+            0.02095296047627926,
+            0.004669542424380779,
+            -0.024418462067842484,
+            0.01643003523349762,
+            0.005562575533986092,
+            -0.017638517543673515,
+            -0.002251465804874897,
+            -0.012040399014949799,
+            0.012351405806839466,
+            -0.01782512292265892,
+            0.0016472243005409837,
+            0.010503137484192848,
+            0.0017738484311848879,
+            0.0030278717167675495,
+            -0.01095631904900074,
+            0.019993282854557037,
+            0.01825164631009102,
+            0.008046185597777367,
+            -0.03200702369213104,
+            0.008819258771836758,
+            -0.007588561624288559,
+            -0.0007630769396200776,
+            -0.020419806241989136,
+            0.029821092262864113,
+            0.013435485772788525,
+            0.007975098676979542,
+            0.01055645290762186,
+            -0.006491152569651604,
+            0.006282333750277758,
+            -0.009685634635388851,
+            -0.002785730641335249,
+            -0.013319969177246094,
+            -0.029678918421268463,
+            -0.007593004498630762,
+            0.023760905489325523,
+            -0.006451165769249201,
+            0.01095631904900074,
+            -0.004316327627748251,
+            0.029270166531205177,
+            -0.015106036327779293,
+            0.000624234729912132,
+            0.02477389946579933,
+            -0.0022370261140167713,
+            0.0058646961115300655,
+            0.011533902958035469,
+            0.06394296139478683,
+            -0.003012321190908551,
+            0.013239996507763863,
+            0.0013817580183967948,
+            0.002856818027794361,
+            0.007948440499603748,
+            -0.015070492401719093,
+            -0.03190039470791817,
+            0.001449513016268611,
+            0.0043718647211790085,
+            -0.013435485772788525,
+            -0.0016072377329692245,
+            -0.0018793685594573617,
+            0.001987110124900937,
+            -0.004909461829811335,
+            0.018153900280594826,
+            -0.020935188978910446,
+            0.028737012296915054,
+            0.019851109012961388,
+            0.012884560041129589,
+            -0.010387620888650417,
+            0.015941310673952103,
+            0.025253737345337868,
+            -0.008121715858578682,
+            -0.0038498176727443933,
+            0.007401957642287016,
+            0.0053937435150146484,
+            0.023760905489325523,
+            0.0027079791761934757,
+            0.029981037601828575,
+            -0.0035343680065125227,
+            0.003134502563625574,
+            -0.014270761050283909,
+            -0.001956009538844228,
+            -0.0033988580107688904,
+            0.005904682911932468,
+            0.003414408303797245,
+            0.0014061941765248775,
+            -0.016403377056121826,
+            0.013648747466504574,
+            0.004176374524831772,
+            0.003367757424712181,
+            0.01456399541348219,
+            -0.0006292330217547715,
+            -0.002503603231161833,
+            0.013444371521472931,
+            -0.002197039546445012,
+            -0.0013151137391105294,
+            0.008143930695950985,
+            -0.012315861880779266,
+            -0.016332291066646576,
+            -0.008255003951489925,
+            0.000503719667904079,
+            -0.006344534922391176,
+            -0.01958453096449375,
+            -0.004847260192036629,
+            0.022232530638575554,
+            0.0022259189281612635,
+            0.031011803075671196,
+            -0.003445509122684598,
+            0.008277218788862228,
+            0.016980960965156555,
+            -0.011489473283290863,
+            -0.0005217691650614142,
+            0.009312426671385765,
+            -0.0034566165413707495,
+            -0.0016960968496277928,
+            -0.004984991624951363,
+            0.00048372638411819935,
+            -0.009001419879496098,
+            3.720972017617896e-05,
+            0.0167232695966959,
+            0.011036291718482971,
+            0.0025458112359046936,
+            -0.005913568660616875,
+            -0.0017027612775564194,
+            -0.011844909749925137,
+            -0.006979877129197121,
+            -0.004487381316721439,
+            0.0017105364240705967,
+            -0.0011201791930943727,
+            0.01182713732123375,
+            0.00040319786057807505,
+            0.010218788869678974,
+            -0.010982976295053959,
+            -0.0011357294861227274,
+            -0.006548910867422819,
+            0.020419806241989136,
+            0.0053937435150146484,
+            -0.0043496498838067055,
+            0.007566346786916256,
+            -0.011027405969798565,
+            0.021379483863711357,
+            0.005291555542498827,
+            -0.00226479466073215,
+            0.0039075762033462524,
+            -0.0017560767009854317,
+            0.01149835903197527,
+            -0.0010962983360514045,
+            -0.013239996507763863,
+            0.02891472913324833,
+            0.017638517543673515,
+            0.027404125779867172,
+            0.0017405262915417552,
+            0.005735850892961025,
+            -0.00888146087527275,
+            0.027510756626725197,
+            0.01585245132446289,
+            -0.0017427478451281786,
+            -0.028150541707873344,
+            0.004243019036948681,
+            -0.012067057192325592,
+            -0.001978224376216531,
+            0.009614547714591026,
+            -0.01095631904900074,
+            -0.009552346542477608,
+            -0.01607459969818592,
+            -0.03138501197099686,
+            -0.026782112196087837,
+            0.0038164956495165825,
+            0.007068736013025045,
+            -0.008312762714922428,
+            0.0008847027784213424,
+            -0.015301525592803955,
+            -0.009543460793793201,
+            -0.025591401383280754,
+            0.002565804636105895,
+            -0.007313098758459091,
+            -0.037356339395046234,
+            -0.02381422184407711,
+            -0.009330199100077152,
+            -0.007166481111198664,
+            0.006611112039536238,
+            -0.0036898714024573565,
+            -0.02461395226418972,
+            0.011729392223060131,
+            -0.00974783580750227,
+            0.011542788706719875,
+            0.017700720578432083,
+            0.007535246200859547,
+            -0.010547567158937454,
+            -0.013559889048337936,
+            -0.009481258690357208,
+            -0.01628786139190197,
+            0.02052643708884716,
+            -0.010360963642597198,
+            0.006215689703822136,
+            -0.0058646961115300655,
+            -0.0018138349987566471,
+            0.0014184123137965798,
+            0.01832273229956627,
+            -0.03796058148145676,
+            -0.027190864086151123,
+            -0.00651781028136611,
+            0.011382842436432838,
+            0.0004651214985642582,
+            -0.012600211426615715,
+            0.014261875301599503,
+            -0.006406736560165882,
+            -0.02031317539513111,
+            -0.014519566670060158,
+            -0.001221811748109758,
+            -0.011054064147174358,
+            0.00840162206441164,
+            -0.011569445952773094,
+            0.01555921696126461,
+            0.0035299251321703196,
+            0.005349313840270042,
+            -0.00651781028136611,
+            0.015967968851327896,
+            0.008921447210013866,
+            -0.021343940868973732,
+            -0.003747629700228572,
+            -0.009463487192988396,
+            0.0017660732846707106,
+            0.01622566021978855,
+            0.007966212928295135,
+            -0.019815564155578613,
+            0.01738082803785801,
+            0.02687097154557705,
+            0.01315113715827465,
+            0.02118399366736412,
+            0.002925683744251728,
+            0.00752636045217514,
+            -0.001412858604453504,
+            0.011062949895858765,
+            -0.004811716731637716,
+            0.0002893472264986485,
+            0.00702874967828393,
+            -0.01628786139190197,
+            -0.011604989878833294,
+            0.0359523668885231,
+            0.000578694452997297,
+            -0.021219536662101746,
+            0.018153900280594826,
+            -0.005753622390329838,
+            -0.009076950140297413,
+            0.005900240037590265,
+            -0.0020581972785294056,
+            0.002081522950902581,
+            -0.010156587697565556,
+            0.009854466654360294,
+            -0.007961769588291645,
+            0.0057003069669008255,
+            0.006557796616107225,
+            0.004651770461350679,
+            -0.016190115362405777,
+            0.01658109575510025,
+            0.029092447832226753,
+            0.02616010047495365,
+            -0.042616792023181915,
+            -0.006753286812454462,
+            0.005424844101071358,
+            -0.018944745883345604,
+            -0.013675405643880367,
+            0.026017924770712852,
+            -0.0022559086792171,
+            -0.011658305302262306,
+            0.014039727859199047,
+            -0.009472372941672802,
+            -0.015434814617037773,
+            -0.0014961640117689967,
+            0.023938624188303947,
+            0.008623769506812096,
+            0.006402293220162392,
+            -0.007939554750919342,
+            0.005220468156039715,
+            -0.021077362820506096,
+            0.006242347415536642,
+            -0.006095729768276215,
+            0.009481258690357208,
+            -0.005349313840270042,
+            -0.005487045273184776,
+            0.012946761213243008,
+            0.005442616064101458,
+            -0.004847260192036629,
+            0.018589310348033905,
+            -0.00038292689714580774,
+            0.001768294838257134,
+            -0.012422492727637291,
+            0.0201532281935215,
+            -0.000900253071449697,
+            -0.009952211752533913,
+            -0.005424844101071358,
+            0.0197267048060894,
+            0.012511352077126503,
+            -0.0030878514517098665,
+            -0.0029145763255655766,
+            0.006055743433535099,
+            -3.049322731385473e-05,
+            0.000538985594175756,
+            -0.003632113104686141,
+            -0.008348306640982628,
+            -0.004402965307235718,
+            -0.007779608480632305,
+            0.015461471863090992,
+            -0.0028101669158786535,
+            -0.012742385268211365,
+            0.011427272111177444,
+            -0.003472166834399104,
+            -0.00031905947253108025,
+            -0.016607753932476044,
+            0.0011729393154382706,
+            0.005815824028104544,
+            0.026017924770712852,
+            0.012760157696902752,
+            -0.014519566670060158,
+            -0.025164877995848656,
+            -0.014484022743999958,
+            0.03270012512803078,
+            0.014377391897141933,
+            -0.004289669916033745,
+            0.01694541797041893,
+            0.005913568660616875,
+            0.021059591323137283,
+            0.005020535551011562,
+            -0.003063415177166462,
+            0.012102600187063217,
+            0.028132770210504532,
+            0.00480283098295331,
+            -0.01577247865498066,
+            -0.0042830053716897964,
+            -0.0017694055568426847,
+            -0.014128586277365685,
+            -0.008930332958698273,
+            -0.006495595443993807,
+            -0.02438291907310486,
+            0.01620788872241974,
+            -0.013497686944901943,
+            0.013666519895195961,
+            -0.014555109664797783,
+            -0.004509596154093742,
+            -0.027261951938271523,
+            0.002392529509961605,
+            0.021450571715831757,
+            -0.0035899050999432802,
+            0.012689069844782352,
+            -0.02738635428249836,
+            -0.020846329629421234,
+            0.019371269270777702,
+            -0.019922195002436638,
+            -0.00027823986602015793,
+            -0.0022559086792171,
+            -0.0034988245461136103,
+            -0.0051449378952383995,
+            -0.0010701960418373346,
+            -0.005375971551984549,
+            -0.0002896249061450362,
+            0.021432798355817795,
+            -0.004505153279751539,
+            0.0051671527326107025,
+            0.00011628037464106455,
+            0.015168237499892712,
+            0.014119700528681278,
+            -0.006544467993080616,
+            -0.007726293057203293,
+            -0.019762249663472176,
+            0.0037720659747719765,
+            -0.02249910682439804,
+            0.022321389988064766,
+            0.005011649336665869,
+            0.016403377056121826,
+            -0.007153152488172054,
+            -0.004263012204319239,
+            -0.006388964597135782,
+            -0.01047648023813963,
+            0.018660398200154305,
+            -0.010325419716536999,
+            0.007059850264340639,
+            -0.008277218788862228,
+            0.03403301164507866,
+            0.005269340705126524,
+            -0.013355513103306293,
+            0.02454286627471447,
+            0.005264897830784321,
+            0.011054064147174358,
+            -0.016741042956709862,
+            0.009427943266928196,
+            0.01022767461836338,
+            -0.016829900443553925,
+            0.01296453271061182,
+            -0.02921685017645359,
+            0.006251233164221048,
+            -0.024720583111047745,
+            -0.013071163557469845,
+            -0.017505230382084846,
+            -0.01914023607969284,
+            -0.04243907332420349,
+            -0.001449513016268611,
+            0.030691910535097122,
+            -0.005420401226729155,
+            0.017505230382084846,
+            -0.009738950058817863,
+            0.011373956687748432,
+            0.00821501761674881,
+            -0.005100508686155081,
+            0.020864101126790047,
+            0.02790173701941967,
+            -0.02052643708884716,
+            -0.006357864011079073,
+            0.0006603337242268026,
+            0.008983648382127285,
+            -0.006073514930903912,
+            -0.02921685017645359,
+            0.0050694081000983715,
+            -0.002747965743765235,
+            0.012644641101360321,
+            -0.01702539063990116,
+            0.0062778908759355545,
+            0.0007464158697985113,
+            0.011427272111177444,
+            -0.008339419960975647,
+            -0.014252989552915096,
+            -0.0180472694337368,
+            0.0016783250030130148,
+            0.0069976490922272205,
+            -0.022232530638575554,
+            0.016039054840803146,
+            -0.023903081193566322,
+            0.0014472915790975094,
+            0.015941310673952103,
+            -0.01956675946712494,
+            -0.006993206217885017,
+            -0.005438172724097967,
+            0.036929816007614136,
+            -0.0023414355237036943,
+            -0.002870146883651614,
+            0.011951540596783161,
+            0.0082194609567523,
+            0.010494251735508442,
+            -0.002934569725766778,
+            0.0051449378952383995,
+            0.0026280060410499573,
+            -0.004336320795118809,
+            0.0017427478451281786,
+            0.02761738747358322,
+            0.002912354888394475,
+            0.02125508151948452,
+            -0.0032966702710837126,
+            -0.002921240869909525,
+            0.0019349055364727974,
+            -0.007170923985540867,
+            0.009810036979615688,
+            7.4350020440761e-05,
+            -0.004203032236546278,
+            -0.017327511683106422,
+            0.015159351751208305,
+            -0.015168237499892712,
+            0.010121043771505356,
+            0.020473120734095573,
+            0.01788732409477234,
+            0.006771058309823275,
+            -0.004878360778093338,
+            -0.010547567158937454,
+            0.014928317628800869,
+            0.017078706994652748,
+            -0.00018424366135150194,
+            -0.020579751580953598,
+            0.0010801926255226135,
+            -0.00869485642760992,
+            0.003989770542830229,
+            -0.0116849634796381,
+            0.009516802616417408,
+            0.008805930614471436,
+            -0.003201146610081196,
+            0.015425928868353367,
+            0.008779272437095642,
+            0.014706170186400414,
+            0.01321333833038807,
+            -0.03317996487021446,
+            -0.0030145428609102964,
+            -0.008459379896521568,
+            -0.0020393149461597204,
+            -0.006384521722793579,
+            -0.017558544874191284,
+            -0.0004920568899251521,
+            0.016927646473050117,
+            -0.005993541795760393,
+            -0.006086844019591808,
+            0.01942458562552929,
+            -0.01420855987817049,
+            0.013853123411536217,
+            0.006180145777761936,
+            0.016305632889270782,
+            -0.02017100155353546,
+            0.003740965388715267,
+            -0.0012007077457383275,
+            -0.0037654016632586718,
+            -0.004905018489807844,
+            0.015665847808122635,
+            0.006224575452506542,
+            0.02475612796843052,
+            -0.02651553601026535,
+            0.006060186307877302,
+            -0.007779608480632305,
+            -0.011542788706719875,
+            -0.018011726438999176,
+            0.011009634472429752,
+            0.00021798232046421617,
+            -0.015372613444924355,
+            -0.01315113715827465,
+            0.006851031444966793,
+            0.011525017209351063,
+            -0.0014006405835971236,
+            0.013195566833019257,
+            0.009685634635388851,
+            0.0018615968292579055,
+            0.009738950058817863,
+            0.008605997078120708,
+            -0.02242802083492279,
+            -0.008059514686465263,
+            0.016554437577724457,
+            -0.0029056903440505266,
+            -0.010014412924647331,
+            0.018873659893870354,
+            0.006548910867422819,
+            -0.011880452744662762,
+            0.006891018245369196,
+            0.01701650582253933,
+            0.007988426834344864,
+            0.006011313758790493,
+            -0.007126494776457548,
+            0.003241133177652955,
+            0.002292563207447529,
+            0.0004895577440038323,
+            -0.020615296438336372,
+            -0.008530467748641968,
+            0.008921447210013866,
+            -0.01092077512294054,
+            0.008570454083383083,
+            -0.005282669793814421,
+            -0.023352153599262238,
+            0.01570139080286026,
+            0.0024325160775333643,
+            -0.008268333040177822,
+            -0.008015085011720657,
+            0.008090615272521973,
+            0.004487381316721439,
+            -0.008281662128865719,
+            0.0009963319171220064,
+            -0.0007880685734562576,
+            -0.012360291555523872,
+            0.022676825523376465,
+            0.00800175592303276,
+            0.0010085500543937087,
+            -0.006428950931876898,
+            -0.009116937406361103,
+            -0.0133643988519907,
+            0.031438328325748444,
+            0.013355513103306293,
+            0.023067805916070938,
+            0.01201374176889658,
+            -0.0033699788618832827,
+            -0.0062556760385632515,
+            -0.002807945478707552,
+            0.01559476088732481,
+            0.012111486867070198,
+            -0.00020451462478376925,
+            0.010831916704773903,
+            0.003245576284825802,
+            0.0037298579700291157,
+            0.006033528596162796,
+            0.005238240119069815,
+            -0.003241133177652955,
+            0.008139487355947495,
+            0.0016305632889270782,
+            0.004811716731637716,
+            0.002556918654590845,
+            0.011729392223060131,
+            0.02541368454694748,
+            -0.006544467993080616,
+            -0.008086171932518482,
+            0.01029876247048378,
+            0.012609097175300121,
+            0.004651770461350679,
+            -0.01569250598549843,
+            0.0066022262908518314,
+            -0.011391728185117245,
+            0.011276211589574814,
+            -0.02877255529165268,
+            -0.006877689156681299,
+            0.009952211752533913,
+            -0.021148450672626495,
+            0.03163381665945053,
+            -0.007419729605317116,
+            -0.024596180766820908,
+            0.007766279857605696,
+            -0.00941905751824379,
+            -0.013302197679877281,
+            -0.018429363146424294,
+            0.020402034744620323,
+            0.017505230382084846,
+            0.020419806241989136,
+            -0.012466922402381897,
+            0.01862485334277153,
+            0.004851703066378832,
+            0.022587966173887253,
+            -0.0037542942445725203,
+            -0.014439593069255352,
+            -0.010574225336313248,
+            0.013391056098043919,
+            0.009614547714591026,
+            -0.007495259400457144,
+            0.029465656727552414,
+            0.014368506148457527,
+            0.005838038399815559,
+            -0.0013339963043108582,
+            -0.00737529993057251,
+            0.004143052268773317,
+            -0.002876811195164919,
+            0.004416293930262327,
+            -0.009161366149783134,
+            -0.02111290581524372,
+            -0.0008302765781991184,
+            0.017265310510993004,
+            0.008619326166808605,
+            0.024933844804763794,
+            -0.015106036327779293,
+            0.0081839170306921,
+            0.002747965743765235,
+            -0.01044093631207943,
+            -0.02095296047627926,
+            0.002816831460222602,
+            -0.010201017372310162,
+            0.023192208260297775,
+            0.007468601688742638,
+            -0.007490816526114941,
+            -0.012022627517580986,
+            0.004451837856322527,
+            0.0034943814389407635,
+            0.024454006925225258,
+            -0.005944669246673584,
+            -0.01029876247048378,
+            0.01420855987817049,
+            0.004776173271238804,
+            0.0015372612979263067,
+            0.004038643091917038,
+            0.013506573624908924,
+            0.00668219942599535,
+            -0.0011362849036231637,
+            -0.001868261257186532,
+            0.0037298579700291157,
+            0.007544131949543953,
+            -0.0010435383301228285,
+            0.0011285097571089864,
+            0.001358432462438941,
+            0.004580683074891567,
+            0.012475808151066303,
+            -0.004098623059689999,
+            0.0027768448926508427,
+            -0.0041230591014027596,
+            -0.00912582315504551,
+            0.0014894995838403702,
+            0.0005156600964255631,
+            -0.012626868672668934,
+            -0.0009224678506143391,
+            -0.00912582315504551,
+            -0.0020126572344452143,
+            0.01417301595211029,
+            0.03367757424712181,
+            -0.010538681410253048,
+            0.022321389988064766,
+            -0.011720506474375725,
+            0.02146834321320057,
+            0.013373284600675106,
+            0.005469273775815964,
+            -0.02228584513068199,
+            -0.015088263899087906,
+            0.01405749935656786,
+            -0.017940638586878777,
+            0.004209696780890226,
+            -0.0052337972447276115,
+            -0.03024761565029621,
+            0.013053392060101032,
+            -0.03170490264892578,
+            -0.0030478648841381073,
+            0.005287112668156624,
+            -0.014332962222397327,
+            -0.009259111247956753,
+            -0.0031789320055395365,
+            -0.002585797803476453,
+            -0.0027368583250790834,
+            0.0064200651831924915,
+            -0.028026139363646507,
+            -0.0133999427780509,
+            0.005540360696613789,
+            0.009996641427278519,
+            -0.01620788872241974,
+            0.010538681410253048,
+            0.004589569289237261,
+            -0.0012617983156815171,
+            0.00462066987529397,
+            -0.018153900280594826,
+            -0.007033192552626133,
+            -0.02512933500111103,
+            -0.003809831105172634,
+            -0.00803729984909296,
+            0.015417042188346386,
+            -0.00045679096365347505,
+            -0.005953555461019278,
+            -0.0009407950565218925,
+            -0.011187352240085602,
+            0.0030834085773676634,
+            0.01055645290762186,
+            0.002639113226905465,
+            -0.008170587942004204,
+            0.01723865233361721,
+            0.02278345637023449,
+            -0.01300007663667202,
+            -0.025662489235401154,
+            0.01921132393181324,
+            -0.01577247865498066,
+            0.021574974060058594,
+            0.0005645325290970504,
+            -0.001508382149040699,
+            -0.0150793781504035,
+            0.015150465071201324,
+            0.0010563117684796453,
+            0.01216480229049921,
+            -0.014795029535889626,
+            0.02825717255473137,
+            -0.0010374292032793164,
+            0.003809831105172634,
+            -0.0073664141818881035,
+            -8.213386536226608e-06,
+            0.00888146087527275,
+            0.0019904423970729113,
+            -0.010369849391281605,
+            0.020508665591478348,
+            -0.012182573787868023,
+            0.0045473612844944,
+            -0.008446050807833672,
+            0.00020104357099626213,
+            -0.008974762633442879,
+            0.0034543948713690042,
+            -0.0017838451312854886,
+            0.012573553249239922,
+            0.012546896003186703,
+            0.015274868346750736,
+            -0.010023298673331738,
+            -0.008948104456067085,
+            0.013782036490738392,
+            -0.02498716115951538,
+            0.0013828687369823456,
+            0.0006231239531189203,
+            0.011951540596783161,
+            -0.02118399366736412,
+            0.006237904075533152,
+            0.026391133666038513,
+            0.005318213254213333,
+            0.009836695156991482,
+            -0.010716399177908897,
+            -0.018002839758992195,
+            -0.007815152406692505,
+            0.007917339913547039,
+            -0.014457364566624165,
+            -0.02059752494096756,
+            -0.013702062889933586,
+            -0.00020479230443015695,
+            -0.0021315061021596193,
+            0.004918347578495741,
+            -0.013630975969135761,
+            -0.002365871798247099,
+            -0.01664329692721367,
+            -0.0024502878077328205,
+            -0.014039727859199047,
+            -0.007042078301310539,
+            0.008090615272521973,
+            0.02855929359793663,
+            0.008077286183834076,
+            0.02402748353779316,
+            0.01237806398421526,
+            0.020917417481541634,
+            0.017834007740020752,
+            -0.004567354451864958,
+            -0.005695864092558622,
+            -0.0002508879406377673,
+            0.018660398200154305,
+            -0.03321550786495209,
+            0.01438627764582634,
+            -0.0077174073085188866,
+            -0.004138609394431114,
+            -0.0054914881475269794,
+            -0.0019304625457152724,
+            0.010796372778713703,
+            0.003976441919803619,
+            -0.004167488776147366,
+            0.010929660871624947,
+            -0.024596180766820908,
+            -0.007135380525141954,
+            -0.00735752796754241,
+            -0.01723865233361721,
+            -0.0013362177414819598,
+            0.02409856952726841,
+            -0.005424844101071358,
+            0.002994549460709095,
+            0.006984320003539324,
+            0.008437165059149265,
+            0.016492236405611038,
+            0.015808021649718285,
+            -0.0016438921447843313,
+            -0.0057669514790177345,
+            0.005398186389356852,
+            -0.004887246992439032,
+            -0.03081631287932396,
+            -0.006899903994053602,
+            0.008379407227039337,
+            0.011729392223060131,
+            -0.025289282202720642,
+            -0.011116265319287777,
+            0.0017338618636131287,
+            0.01540815643966198,
+            0.02081078663468361,
+            -0.0039297910407185555,
+            -0.016083484515547752,
+            -0.004147495608776808,
+            -0.020579751580953598,
+            0.021806007251143456,
+            -0.012804586440324783,
+            0.0005670317332260311,
+            -0.009232454001903534,
+            0.00939240027219057,
+            -0.012564667500555515,
+            0.03317996487021446,
+            -0.014590653590857983,
+            0.002401415491476655,
+            -0.0010685299057513475,
+            0.011098493821918964,
+            0.00476728705689311,
+            0.024134114384651184,
+            -0.01722976751625538,
+            -0.003796502249315381,
+            0.028683695942163467,
+            -0.033624257892370224,
+            0.016696613281965256,
+            0.008743729442358017,
+            0.01084080245345831,
+            -0.009303540922701359,
+            -0.008743729442358017,
+            0.027955051511526108,
+            0.020793013274669647,
+            0.006535581778734922,
+            -0.001412858604453504,
+            -0.019264638423919678,
+            0.023689819499850273,
+            -0.0011829358991235495,
+            0.002683542901650071,
+            -0.002794616622850299,
+            0.013453258201479912,
+            -0.034886058419942856,
+            -0.01657220907509327,
+            0.005562575533986092,
+            -0.008663755841553211,
+            -0.01892697438597679,
+            0.0005864696577191353,
+            -0.0116494195535779,
+            -0.02132616750895977,
+            -0.0032655694521963596,
+            0.01965561881661415,
+            -0.002612455515190959,
+            0.003072301158681512,
+            0.006686642300337553,
+            0.004411851055920124,
+            -0.008605997078120708,
+            -0.024525092914700508,
+            -0.01099186297506094,
+            0.003709864802658558,
+            0.007348642218858004,
+            -0.013017848134040833,
+            -0.012982305139303207,
+            -0.01077860128134489,
+            -0.0030100997537374496,
+            0.013471029698848724,
+            0.002305892063304782,
+            0.02168160490691662,
+            -0.004474052228033543,
+            0.01435073371976614,
+            0.013906438834965229,
+            -0.006851031444966793,
+            -0.008685970678925514,
+            0.013248882256448269,
+            -0.0027457440737634897,
+            -0.007504145614802837,
+            -0.007384185679256916,
+            -0.0016205665888264775,
+            -0.006584454327821732,
+            0.002288120100274682,
+            0.01956675946712494,
+            -0.01970893330872059,
+            0.004820602480322123,
+            0.0017249759985134006,
+            -0.01212925836443901,
+            0.005629220046103001,
+            0.008308319374918938,
+            -0.0011884896084666252,
+            0.011196237988770008,
+            0.003774287411943078,
+            -0.002202593255788088,
+            0.00226479466073215,
+            -0.013071163557469845,
+            0.006575568579137325,
+            0.015417042188346386,
+            0.014190787449479103,
+            0.01077860128134489,
+            0.006366749759763479,
+            -0.034299585968256,
+            -0.006415622308850288,
+            -0.003118952037766576,
+            -0.02374313399195671,
+            0.0014106371672824025,
+            0.0026280060410499573,
+            0.020259859040379524,
+            -0.003076744033023715,
+            0.014937203377485275,
+            -0.005882468074560165,
+            0.0006458940915763378,
+            0.016821015626192093,
+            0.009907782077789307,
+            -0.03573910519480705,
+            -0.010663083754479885,
+            -0.01523043867200613,
+            -0.007979541085660458,
+            -0.015177123248577118,
+            -0.014377391897141933,
+            0.008077286183834076,
+            0.017291968688368797,
+            -0.013648747466504574,
+            -0.012200345285236835,
+            0.011711620725691319,
+            0.013888667337596416,
+            0.007370857056230307,
+            0.006144602317363024,
+            0.016314519569277763,
+            -0.0018127242801710963,
+            0.0006347867310978472,
+            0.00026338372845202684,
+            0.018873659893870354,
+            -0.013026734814047813,
+            -0.0023392140865325928,
+            -0.011418386362493038,
+            0.011676076799631119,
+            0.020864101126790047,
+            -0.01076971460133791,
+            0.014448478817939758,
+            -0.007059850264340639,
+            -0.006779944524168968,
+            -0.047379638999700546,
+            -0.01321333833038807,
+            -0.009436829946935177,
+            0.028132770210504532,
+            -0.004807273857295513,
+            -0.005833595525473356,
+            0.0055803474970161915,
+            0.019602302461862564,
+            0.0063001057133078575,
+            0.008166145533323288,
+            0.01818055845797062,
+            -0.00839273538440466,
+            -0.012804586440324783,
+            0.004160824231803417,
+            0.007859582081437111,
+            -0.00927688367664814,
+            0.0027546300552785397,
+            0.0099344402551651,
+            0.032753441482782364,
+            -0.0057669514790177345,
+            0.0075085884891450405,
+            -0.009108050726354122,
+            0.0022303618025034666,
+            -0.016758814454078674,
+            0.001700539723969996,
+            -0.005473716650158167,
+            -0.006815487984567881,
+            0.014439593069255352,
+            -0.021788235753774643,
+            -0.0014795028837397695,
+            -0.0028968045953661203,
+            0.008406064473092556,
+            -0.01318668108433485,
+            0.011045178398489952,
+            -0.0041230591014027596,
+            -0.004271897953003645,
+            0.014128586277365685,
+            -0.011915996670722961,
+            0.027581844478845596,
+            -0.002074858406558633,
+            0.009969983249902725,
+            -0.007970655336976051,
+            -0.012866788543760777,
+            -0.017807351425290108,
+            0.0025369254872202873,
+            -0.0021348383743315935,
+            -0.008534910157322884,
+            0.008850360289216042,
+            -0.022463563829660416,
+            0.0036543277092278004,
+            -0.004873917903751135,
+            0.0015339291421696544,
+            -0.02592906542122364,
+            0.00015036614786367863,
+            -0.002412522677332163,
+            0.011080721393227577,
+            -0.008810373023152351,
+            -0.013160022906959057,
+            0.01176493614912033,
+            -0.010734171606600285,
+            -0.013888667337596416,
+            -0.01318668108433485,
+            0.008410507813096046,
+            0.004038643091917038,
+            -0.002434737514704466,
+            0.014457364566624165,
+            -0.011009634472429752,
+            0.0008480484248138964,
+            -0.006184588652104139,
+            0.00924133975058794,
+            -0.00047900574281811714,
+            -0.016989847645163536,
+            -4.682454527937807e-05,
+            -0.0002667159424163401,
+            0.012440265156328678,
+            0.012982305139303207,
+            0.018447136506438255,
+            -0.008441608399152756,
+            0.0026168986223638058,
+            0.0034766097087413073,
+            -0.0037809519562870264,
+            0.011187352240085602,
+            0.005775837227702141,
+            -0.013888667337596416,
+            0.0013650968903675675,
+            -0.013684291392564774,
+            0.00854379590600729,
+            -0.00891256146132946,
+            0.010209903120994568,
+            0.009108050726354122,
+            -0.0024769455194473267,
+            -0.010343191213905811,
+            0.011729392223060131,
+            0.01657220907509327,
+            -0.012333634309470654,
+            0.00032655693939886987,
+            -0.007228682283312082,
+            0.025431456044316292,
+            0.016749927774071693,
+            0.010583111084997654,
+            -0.013355513103306293,
+            0.013826466165482998,
+            -0.00025310940691269934,
+            0.0037254150956869125,
+            0.00957011803984642,
+            -0.026408905163407326,
+            -0.02491607330739498,
+            0.003736522514373064,
+            -0.0069310045801103115,
+            -0.0060468572191894054,
+            0.023316610604524612,
+            -0.012138144113123417,
+            0.006184588652104139,
+            -0.011569445952773094,
+            0.028168313205242157,
+            0.01621677353978157,
+            -0.0009307984146289527,
+            -0.015834679827094078,
+            0.0002776844776235521,
+            0.01011215802282095,
+            -0.0013417714508250356,
+            -0.007766279857605696,
+            -0.0022681269329041243,
+            -0.012538010254502296,
+            0.004398522432893515,
+            -0.003783173393458128,
+            0.004776173271238804,
+            0.004402965307235718,
+            -0.012697956524789333,
+            -0.011311755515635014,
+            0.028221629559993744,
+            0.0008724846411496401,
+            0.012973419390618801,
+            -0.004971663001924753,
+            0.0017494122730568051,
+            0.005957998335361481,
+            0.02233916148543358,
+            -0.024276288226246834,
+            0.01905137673020363,
+            0.006739957723766565,
+            0.017771806567907333,
+            0.008903675712645054,
+            -0.00839717872440815,
+            0.01249358057975769,
+            -0.006157930940389633,
+            -0.007415286265313625,
+            -0.010280990041792393,
+            -0.0014750600093975663,
+            -0.0016050162957981229,
+            -0.0076285479590296745,
+            0.009579003788530827,
+            -0.0011290650581941009,
+            -0.004905018489807844,
+            0.01420855987817049,
+            -0.022890087217092514,
+            0.014723941683769226,
+            -0.0013239996042102575,
+            0.01124955341219902,
+            -0.0021781569812446833,
+            0.011125151067972183,
+            -0.02278345637023449,
+            0.0038675894029438496,
+            -0.01797618344426155,
+            -0.0012118150480091572,
+            -0.0040941801853477955,
+            -0.001176271471194923,
+            -0.0030323145911097527,
+            0.007979541085660458,
+            -0.034139640629291534,
+            -0.010067728348076344,
+            0.015488130040466785,
+            0.013933097012341022,
+            0.0026768783573061228,
+            0.013026734814047813,
+            0.006246790289878845,
+            0.0012362513225525618,
+            -0.00615348806604743,
+            -0.009294655174016953,
+            -0.020259859040379524,
+            -0.02219698764383793,
+            -0.013053392060101032,
+            -0.0022725698072463274,
+            -0.0008230567909777164,
+            -0.005242682993412018,
+            0.008286104537546635,
+            -0.018287189304828644,
+            -0.0007325316546484828,
+            -0.0005164931644685566,
+            0.0004479050694499165,
+            0.001358432462438941,
+            0.0018538215663284063,
+            0.016341175884008408,
+            0.008765943348407745,
+            -0.0042363544926047325,
+            -0.007370857056230307,
+            0.025182651355862617,
+            -0.0030345360282808542,
+            0.012147029861807823,
+            0.020028825849294662,
+            0.0025080463383346796,
+            -0.014688398689031601,
+            0.013835351914167404,
+            -0.008290547877550125,
+            0.018642624840140343,
+            -0.012333634309470654,
+            -0.025680260732769966,
+            -0.014857230708003044,
+            0.007610776461660862,
+            -0.011871566995978355,
+            -0.009596775285899639,
+            0.01365763321518898,
+            -0.002821274334564805,
+            0.0001021878924802877,
+            0.007135380525141954,
+            -0.005762508604675531,
+            0.0025369254872202873,
+            -0.03497491776943207,
+            -0.000194934502360411,
+            -0.03001658245921135,
+            -0.030567508190870285,
+            0.014670626260340214,
+            0.006855474319308996,
+            0.021077362820506096,
+            -0.006948776543140411,
+            0.013977525755763054,
+            -0.009099164977669716,
+            0.026053469628095627,
+            0.014946090057492256,
+            -0.0006936558638699353,
+            0.0031789320055395365,
+            -0.007250897120684385,
+            -0.006371192634105682,
+            0.027155321091413498,
+            -0.008081729523837566,
+            -0.0059313406236469746,
+            0.02475612796843052,
+            0.026657709851861,
+            0.01738082803785801,
+            -0.018944745883345604,
+            0.019460128620266914,
+            0.0013017848832532763,
+            0.01789620891213417,
+            -0.016199002042412758,
+            -0.00032850075513124466,
+            -0.015665847808122635,
+            -0.008161702193319798,
+            0.05530586466193199,
+            0.01723865233361721,
+            -0.0007297547999769449,
+            -0.0202243160456419,
+            0.015017176978290081,
+            -0.0029789991676807404,
+            -0.002028207527473569,
+            0.00981892365962267,
+            0.0302298441529274,
+            -0.00920579582452774,
+            -0.013799807988107204,
+            0.013293311931192875,
+            -0.005247125867754221,
+            0.026746569201350212,
+            -0.002260351786389947,
+            0.0020904086995869875,
+            -0.015221552923321724,
+            -0.013133365660905838,
+            0.0013395498972386122,
+            0.009756721556186676,
+            -0.007939554750919342,
+            -0.0011018520453944802,
+            -0.006944333668798208,
+            0.012769043445587158,
+            0.006815487984567881,
+            -0.020277632400393486,
+            -0.003112287726253271,
+            -0.030922943726181984,
+            -0.00480283098295331,
+            -0.035525843501091,
+            -0.005944669246673584,
+            -0.005269340705126524,
+            -0.014626197516918182,
+            -0.032611265778541565,
+            -0.02073969878256321,
+            -0.009996641427278519,
+            0.004660656210035086,
+            0.002425851533189416,
+            0.010041071102023125,
+            0.002474724082276225,
+            0.0005764730158261955,
+            0.0010274326195940375,
+            0.0041230591014027596,
+            0.01303562056273222,
+            0.001622788142412901,
+            0.011729392223060131,
+            0.02233916148543358,
+            -0.018198329955339432,
+            -0.033410996198654175,
+            -0.014484022743999958,
+            0.0050249784253537655,
+            0.005686978343874216,
+            -0.0018749256851151586,
+            0.008601554669439793,
+            0.0015561438631266356,
+            0.011231781914830208,
+            -0.014324076473712921,
+            -0.005020535551011562,
+            -0.009774493984878063,
+            -0.0011773821897804737,
+            0.0026191200595349073,
+            0.003896468784660101,
+            0.0026946503203362226,
+            0.0029412340372800827,
+            -0.011711620725691319,
+            0.01014770194888115,
+            -0.005380414426326752,
+            -0.0009291322785429657,
+            -0.0009024745668284595,
+            -0.012538010254502296,
+            0.004927233327180147,
+            -0.007935111410915852,
+            -0.00753968907520175,
+            -0.008534910157322884,
+            0.0034521734341979027,
+            -0.01600351184606552,
+            -0.009036963805556297,
+            0.01716756634414196,
+            -0.006424508057534695,
+            0.007757393643260002,
+            0.012831244617700577,
+            -0.0013650968903675675,
+            -0.004562911577522755,
+            0.019229095429182053,
+            -0.012742385268211365,
+            -0.01417301595211029,
+            -0.024454006925225258,
+            -0.006446722894906998,
+            0.0005456500221043825,
+            0.015452586114406586,
+            0.0042830053716897964,
+            -0.010903003625571728,
+            -0.0034810525830835104,
+            -0.002579133491963148,
+            0.003554361406713724,
+            -0.012324748560786247,
+            0.005966884084045887,
+            -0.02388530783355236,
+            0.015959082171320915,
+            -0.006677756551653147,
+            -0.009303540922701359,
+            0.018713712692260742,
+            0.04535365104675293,
+            -0.007606333587318659,
+            -0.0177629217505455,
+            0.009014748968183994,
+            0.022836772724986076,
+            0.022676825523376465,
+            -0.007086507976055145,
+            0.013053392060101032,
+            -0.02001105435192585,
+            -0.011880452744662762,
+            -0.005082736723124981,
+            -0.002314777811989188,
+            0.022516880184412003,
+            -0.03104734607040882,
+            -0.019246866926550865,
+            -0.007784051354974508,
+            -0.03241577744483948,
+            -0.006948776543140411,
+            0.028470434248447418,
+            0.004611784126609564,
+            -0.005620333831757307,
+            -0.004143052268773317,
+            -0.002716864924877882,
+            -0.01753188669681549,
+            0.020259859040379524,
+            0.08686859160661697,
+            -0.014635083265602589,
+            -0.008294991217553616,
+            -0.004131945315748453,
+            0.00635342113673687,
+            0.005784722976386547,
+            -0.005056079011410475,
+            0.06181034445762634,
+            -0.023121120408177376,
+            0.00033433211501687765,
+            0.003609898267313838,
+            0.015816908329725266,
+            -0.007170923985540867,
+            -0.0002644944761414081,
+            0.0012107043294236064,
+            -0.005953555461019278,
+            -0.010129929520189762,
+            -0.017105363309383392,
+            -0.0045295893214643,
+            -0.0017949525499716401,
+            -0.00701097771525383,
+            -0.01620788872241974,
+            0.019335726276040077,
+            -0.023849764838814735,
+            -0.006615554913878441,
+            -0.0023347712121903896,
+            -0.006513367407023907,
+            0.009188024327158928,
+            -0.019833335652947426,
+            0.002570247510448098,
+            -0.0011773821897804737,
+            0.005380414426326752,
+            -0.005158266983926296,
+            -0.005575904622673988,
+            -0.0028234957717359066,
+            0.0009313537739217281,
+            -0.0098988963291049,
+            0.01826941780745983,
+            0.013790922239422798,
+            0.009152480401098728,
+            0.010058842599391937,
+            -0.0055803474970161915,
+            0.023121120408177376,
+            -0.005784722976386547,
+            0.0057891663163900375,
+            -0.0067888302728533745,
+            0.0014161908766254783,
+            -0.004789501894265413,
+            0.01839382015168667,
+            -0.020633067935705185,
+            -0.013071163557469845,
+            -0.014875002205371857,
+            0.035170406103134155,
+            -0.02097073197364807,
+            0.014972747303545475,
+            0.0010307647753506899,
+            0.011356184259057045,
+            0.010485365986824036,
+            0.0013051170390099287,
+            0.01702539063990116,
+            -0.012093714438378811,
+            0.016776585951447487,
+            0.002699093194678426,
+            0.015363726764917374,
+            -0.008255003951489925,
+            0.012573553249239922,
+            0.00010274325904902071,
+            0.003809831105172634,
+            -0.008637098595499992,
+            0.0044696093536913395,
+            0.011676076799631119,
+            -0.005780280102044344,
+            0.03541921079158783,
+            -0.0013728720368817449,
+            -0.02226807363331318,
+            0.016741042956709862,
+            -0.015106036327779293,
+            -0.02082855813205242,
+            0.0011740500340238214,
+            -0.014652854762971401,
+            -0.0030278717167675495,
+            0.012315861880779266,
+            -0.028381574898958206,
+            -0.020988503471016884,
+            0.005247125867754221,
+            0.001098519773222506,
+            0.0016572210006415844,
+            0.0006486709462478757,
+            -0.014439593069255352,
+            -0.0018138349987566471,
+            -0.0030656366143375635,
+            0.007957326248288155,
+            -0.014777257107198238,
+            0.006437837146222591,
+            -0.026266731321811676,
+            -0.0031389454379677773,
+            -0.003132280893623829,
+            0.023689819499850273,
+            -0.002907912014052272,
+            0.010432050563395023,
+            -0.005993541795760393,
+            -0.0025502543430775404,
+            -0.016252316534519196,
+            -0.0033522071316838264,
+            -0.003258905140683055,
+            0.018518222495913506,
+            0.026746569201350212,
+            -0.0016761035658419132,
+            -0.0025235966313630342,
+            -6.3693878473714e-05,
+            0.015461471863090992,
+            -0.00858378317207098,
+            -0.01022767461836338,
+            0.01666106842458248,
+            0.014404049143195152,
+            -0.016554437577724457,
+            0.011862681247293949,
+            -0.0022125900723040104,
+            0.011658305302262306,
+            -0.012253660708665848,
+            0.016536666080355644,
+            0.020490894094109535,
+            -0.005282669793814421,
+            0.019193550571799278,
+            -0.01095631904900074,
+            0.015390384942293167,
+            0.018731484189629555,
+            0.006948776543140411,
+            0.014155244454741478,
+            0.012662412598729134,
+            -0.01180048007518053,
+            0.0005836928030475974,
+            -0.005758065264672041,
+            -0.005322656128555536,
+            0.005127166397869587,
+            -0.002612455515190959,
+            -0.019406812265515327,
+            0.0036121197044849396,
+            0.0012462479062378407,
+            0.008508252911269665,
+            0.0034899385645985603,
+            -0.00617126002907753,
+            -0.0008730400004424155,
+            0.013462143950164318,
+            -0.02095296047627926,
+            -0.006077957805246115,
+            0.023938624188303947,
+            -0.006015756633132696,
+            -0.005673649255186319,
+            0.0011407278943806887,
+            0.012093714438378811,
+            -0.01092077512294054,
+            -0.01651889458298683,
+            -0.020508665591478348,
+            0.02841711975634098,
+            0.004509596154093742,
+            0.00887701753526926,
+            0.006762172561138868,
+            -0.008610440418124199,
+            -0.002639113226905465,
+            -0.015950197353959084,
+            -0.001235140603967011,
+            0.018233874812722206,
+            0.01723865233361721,
+            0.011027405969798565,
+            -0.019780021160840988,
+            0.008317206054925919,
+            -0.008486038073897362,
+            3.512708644848317e-05,
+            0.014004183933138847,
+            0.01622566021978855,
+            0.0063223205506801605,
+            -0.0027079791761934757,
+            0.016989847645163536,
+            0.001412858604453504,
+            0.029678918421268463,
+            -0.0006458940915763378,
+            -0.01767406240105629,
+            0.0003993102873209864,
+            0.001091300044208765,
+            -0.01620788872241974,
+            -0.003932012245059013,
+            0.0024769455194473267,
+            0.038813624531030655,
+            -0.008139487355947495,
+            0.007148709148168564,
+            -0.003058972302824259,
+            -0.012422492727637291,
+            0.0025724689476191998,
+            -0.0015883552841842175,
+            -0.002055975841358304,
+            0.02177046425640583,
+            -0.017567431554198265,
+            0.012004856020212173,
+            -0.012591325677931309,
+            0.0056780921295285225,
+            0.0012740164529532194,
+            0.007761836517602205,
+            -0.011356184259057045,
+            -0.03067413903772831,
+            -0.013790922239422798,
+            0.02635558880865574,
+            0.018642624840140343,
+            -0.007135380525141954,
+            0.018145015463232994,
+            -0.014821686781942844,
+            -0.00941905751824379,
+            0.008303876966238022,
+            -0.010982976295053959,
+            0.0009352413471788168,
+            0.007233125623315573
+          ],
+          "keyphrases": [
+            "1 Billion Word Language Modeling Benchmark",
+            "Long Short-Term Memory (LSTM) layers",
+            "Mixture of Experts (MoE) models",
+            "computational efficiency",
+            "Machine Translation"
+          ],
+          "title": "5 Experiments 5.1 1 Billion Word Language Modeling Benchmark"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "36d60b41-5a27-4ba6-9a3a-abebae891d1d",
+      "type": "child",
+      "source": {
+        "id": "14339b1f-53f8-479b-9da7-9430b174b9ee",
+        "properties": {
+          "page_content": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer\n\nNoam Shazeer1, Azalia Mirhoseini∗†1, Krzysztof Maziarz∗2, Andy Davis1, Quoc Le1, Geoffrey Hinton1and Jeff Dean1 1Google Brain, {noam,azalia,andydavis,qvl,geoffhinton,jeff}@google.com 2Jagiellonian University, Cracow, krzysztof.maziarz@student.uj.edu.pl\n\nAbstract\n\nThe capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost.\n\n1 Introduction And Related Work 1.1 Conditional Computation\n\nExploiting scale in both training data and model size has been central to the success of deep learning. When datasets are sufficiently large, increasing the capacity (number of parameters) of neural networks can give much better prediction accuracy. This has been shown in domains such as text (Sutskever et al., 2014; Bahdanau et al., 2014; Jozefowicz et al., 2016; Wu et al., 2016), images (Krizhevsky et al., 2012; Le et al., 2012), and audio (Hinton et al., 2012; Amodei et al., 2015). For typical deep learning models, where the entire model is activated for every example, this leads to a roughly quadratic blow-up in training costs, as both the model size and the number of training examples increase. Unfortunately, the advances in computing power and distributed computation fall short of meeting such demand.\n\nVarious forms of conditional computation have been proposed as a way to increase model capacity without a proportional increase in computational costs (Davis & Arel, 2013; Bengio et al., 2013; Eigen et al., 2013; Ludovic Denoyer, 2014; Cho & Bengio, 2014; Bengio et al., 2015; Almahairi et al., 2015). In these schemes, large parts of a network are active or inactive on a per-example basis. The gating decisions may be binary or sparse and continuous, stochastic or deterministic. Various forms of reinforcement learning and back-propagation are proposed for trarining the gating decisions.\n\nWhile these ideas are promising in theory, no work to date has yet demonstrated massive improvements in model capacity, training time, or model quality. We blame this on a combination of the following challenges:\n\nModern computing devices, especially GPUs, are much faster at arithmetic than at branching. Most of the works above recognize this and propose turning on/off large chunks of the network with each gating decision.\n\nLarge batch sizes are critical for performance, as they amortize the costs of parameter transfers and updates. Conditional computation reduces the batch sizes for the conditionally active chunks of the network.\n\nNetwork bandwidth can be a bottleneck. A cluster of GPUs may have computational power thousands of times greater than the aggregate inter-device network bandwidth. To be computationally efficient, the relative computational versus network demands of an algorithm must exceed this ratio. Embedding layers, which can be seen as a form of conditional computation, are handicapped by this very problem. Since the embeddings generally need to be sent across the network, the number of (example, parameter) interactions is limited by network bandwidth instead of computational capacity.\n\nDepending on the scheme, loss terms may be necessary to achieve the desired level of sparsity per-chunk and/or per example. Bengio et al. (2015) use three such terms. These issues can affect both model quality and load-balancing.\n\nModel capacity is most critical for very large data sets. The existing literature on conditional computation deals with relatively small image recognition data sets consisting of up to 600,000 images. It is hard to imagine that the labels of these images provide a sufficient signal to adequately train a model with millions, let alone billions of parameters. In this work, we for the first time address all of the above challenges and finally realize the promise of conditional computation. We obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets.\n\n1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1). All parts of the network are trained jointly by back-propagation.\n\nWhile the introduced technique is generic, in this paper we focus on language modeling and machine translation tasks, which are known to benefit from very large models. In particular, we apply a MoE convolutionally between stacked LSTM layers (Hochreiter & Schmidhuber, 1997), as in Figure 1.\n\nThe MoE is called once for each position in the text, selecting a potentially different combination of experts at each position. The different experts tend to become highly specialized based on syntax and semantics (see Appendix E Table 9). On both language modeling and machine translation benchmarks, we improve on best published results at a fraction of the computational cost.\n\n1.3 Related Work On Mixtures Of Experts\n\nSince its introduction more than two decades ago (Jacobs et al., 1991; Jordan & Jacobs, 1994), the mixture-of-experts approach has been the subject of much research. Different types of expert architectures hae been proposed such as SVMs (Collobert et al., 2002), Gaussian Processes (Tresp, 2001; Theis & Bethge, 2015; Deisenroth & Ng, 2015), Dirichlet Processes (Shahbaba & Neal, 2009), and deep networks. Other work has focused on different expert configurations such as a hierarchical structure (Yao et al., 2009), infinite numbers of experts (Rasmussen & Ghahramani, 2002), and adding experts sequentially (Aljundi et al., 2016). Garmash & Monz (2016) suggest an ensemble model in the format of mixture of experts for machine translation. The gating network is trained on a pre-trained ensemble NMT model. The works above concern top-level mixtures of experts. The mixture of experts is the whole model. Eigen et al. (2013) introduce the idea of using multiple MoEs with their own gating networks as parts of a deep model. It is intuitive that the latter approach is more powerful, since complex problems may contain many sub-problems each requiring different experts. They also allude in their conclusion to the potential to introduce sparsity, turning MoEs into a vehicle for computational computation.\n\nOur work builds on this use of MoEs as a general purpose neural network component. While Eigen et al. (2013) uses two stacked MoEs allowing for two sets of gating decisions, our convolutional application of the MoE allows for different gating decisions at each position in the text. We also realize sparse gating and demonstrate its use as a practical way to massively increase model capacity.\n\n2 The Structure Of The Mixture-Of-Experts Layer\n\nThe Mixture-of-Experts (MoE) layer consists of a set of n \"expert networks\" E1, · · · , En, and a \"gating network\" G whose output is a sparse n-dimensional vector. Figure 1 shows an overview of the MoE module. The experts are themselves neural networks, each with their own parameters.\n\nAlthough in principle we only require that the experts accept the same sized inputs and produce the same-sized outputs, in our initial investigations in this paper, we restrict ourselves to the case where the models are feed-forward networks with identical architectures, but with separate parameters.\n\nLet us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows:\n\n$$y=\\sum_{i=1}^{n}G(x){i}E{i}(x)$$ $$(\\mathbf{l})$$ G(x)iEi(x) (1) We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example. If the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network. In the following we focus on ordinary MoEs. We provide more details on hierarchical MoEs in Appendix B. Our implementation is related to other models of conditional computation. A MoE whose experts are simple weight matrices is similar to the parameterized weight matrix proposed in (Cho & Bengio, 2014). A MoE whose experts have one hidden layer is similar to the block-wise dropout described in (Bengio et al., 2015), where the dropped-out layer is sandwiched between fully-activated layers.\n\n2.1 Gating Network\n\nSoftmax Gating: A simple choice of non-sparse gating function (Jordan & Jacobs, 1994) is to multiply the input by a trainable weight matrix Wg and then apply the Sof tmax function.\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ $\\eqref{eq:walpha}$. Gσ(x) = Sof tmax(x · Wg) (2) Noisy Top-K Gating: We add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to −∞ (which causes the corresponding gate values to equal 0). The sparsity serves to save computation, as described above. While this form of sparsity creates some theoretically scary discontinuities in the output of gating function, we have not yet observed this to be a problem in practice. The noise term helps with load balancing, as will be discussed in Appendix A. The amount of noise per component is controlled by a second trainable weight matrix Wnoise.\n\n$$G(x)=S o f t m a x(K e e p T o p K(H(x),k))$$ $$(4)$$ $$(S)$$ G(x) = Sof tmax(KeepT opK(H(x), k)) (3) $$H(x){i}=(x\\cdot W{g}){i}+S t a n d a r d N o r m a l()\\cdot S o f t p l u s((x\\cdot W{n o i s e}){i})$$ $KeepTopK(v,k){i}=\\begin{cases}v_{i}&\\text{if}v_{i}\\text{is in the top}k\\text{elements of}v.\\ -\\infty&\\text{otherwise.}\\end{cases}$ Training the Gating Network We train the gating network by simple back-propagation, along with the rest of the model. If we choose k > 1, the gate values for the top k experts have nonzero derivatives with respect to the weights of the gating network. This type of occasionally-sensitive behavior is described in (Bengio et al., 2013) with respect to noisy rectifiers. Gradients also backpropagate through the gating network to its inputs. Our method differs here from (Bengio et al., 2015) who use boolean gates and a REINFORCE-style approach to train the gating network.\n\n3 Addressing Performance Challenges 3.1 The Shrinking Batch Problem\n\nOn modern CPUs and GPUs, large batch sizes are necessary for computational efficiency, so as to amortize the overhead of parameter loads and updates. If the gating network chooses k out of n experts for each example, then for a batch of b examples, each expert receives a much smaller batch of approximately kb n b examples. This causes a naive MoE implementation to become very inefficient as the number of experts increases. The solution to this shrinking batch problem is to make the original batch size as large as possible. However, batch size tends to be limited by the memory necessary to store activations between the forwards and backwards passes. We propose the following techniques for increasing the batch size: Mixing Data Parallelism and Model Parallelism: In a conventional distributed training setting, multiple copies of the model on different devices asynchronously process distinct batches of data, and parameters are synchronized through a set of parameter servers. In our technique, these different batches run synchronously so that they can be combined for the MoE layer. We distribute the standard layers of the model and the gating network according to conventional data-parallel schemes, but keep only one shared copy of each expert. Each expert in the MoE layer receives a combined batch consisting of the relevant examples from all of the data-parallel input batches. The same set of devices function as data-parallel replicas (for the standard layers and the gating networks) and as model-parallel shards (each hosting a subset of the experts). If the model is distributed over d devices, and each device processes a batch of size b, each expert receives a batch of approximately kbd nexamples. Thus, we achieve a factor of d improvement in expert batch size.\n\nIn the case of a hierarchical MoE (Section B), the primary gating network employs data parallelism, and the secondary MoEs employ model parallelism. Each secondary MoE resides on one device.\n\nThis technique allows us to increase the number of experts (and hence the number of parameters) by proportionally increasing the number of devices in the training cluster. The total batch size increases, keeping the batch size per expert constant. The memory and bandwidth requirements per device also remain constant, as do the step times, as does the amount of time necessary to process a number of training examples equal to the number of parameters in the model. It is our goal to train a trillionparameter model on a trillion-word corpus. We have not scaled our systems this far as of the writing of this paper, but it should be possible by adding more hardware.\n\nTaking Advantage of Convolutionality: In our language models, we apply the same MoE to each time step of the previous layer. If we wait for the previous layer to finish, we can apply the MoE to all the time steps together as one big batch. Doing so increases the size of the input batch to the MoE layer by a factor of the number of unrolled time steps.\n\nIncreasing Batch Size for a Recurrent MoE: We suspect that even more powerful models may involve applying a MoE recurrently. For example, the weight matrices of a LSTM or other RNN could be replaced by a MoE. Sadly, such models break the convolutional trick from the last paragraph, since the input to the MoE at one timestep depends on the output of the MoE at the previous timestep. Gruslys et al. (2016) describe a technique for drastically reducing the number of stored activations in an unrolled RNN, at the cost of recomputing forward activations. This would allow for a large increase in batch size.\n\n3.2 Network Bandwidth\n\nAnother major performance concern in distributed computing is network bandwidth. Since the experts are stationary (see above) and the number of gating parameters is small, most of the communication involves sending the inputs and outputs of the experts across the network. To maintain computational efficiency, the ratio of an expert's computation to the size of its input and output must exceed the ratio of computational to network capacity of the computing device. For GPUs, this may be thousands to one. In our experiments, we use experts with one hidden layer containing thousands of RELU-activated units. Since the weight matrices in the expert have sizes input_size×hidden_size and hidden_size × output_size, the ratio of computation to input and output is equal to the size of the hidden layer. Conveniently, we can increase computational efficiency simply by using a larger hidden layer, or more hidden layers.\n\n4 Balancing Expert Utilization\n\nWe have observed that the gating network tends to converge to a state where it always produces large weights for the same few experts. This imbalance is self-reinforcing, as the favored experts are trained more rapidly and thus are selected even more by the gating network. Eigen et al. (2013) describe the same phenomenon, and use a hard constraint at the beginning of training to avoid this local minimum. Bengio et al. (2015) include a soft constraint on the batch-wise average of each gate.1 We take a soft constraint approach. We define the importance of an expert relative to a batch of training examples to be the batchwise sum of the gate values for that expert. We define an additional loss Limportance, which is added to the overall loss function for the model. This loss is equal to the square of the coefficient of variation of the set of importance values, multiplied by a hand-tuned scaling factor wimportance. This additional loss encourages all experts to have equal importance.\n\n$$I m p o r t a n c e(X)=\\sum_{x\\in X}G(x)$$ $$L_{importance}(X)=w_{importance}\\cdot CV(Importance(X))^{2}\\tag{7}$$ $$(6)$$\n\nWhile this loss function can ensure equal importance, experts may still receive very different numbers of examples. For example, one expert may receive a few examples with large weights, and another may receive many examples with small weights. This can cause memory and performance problems on distributed hardware. To solve this problem, we introduce a second loss function, Lload , which ensures balanced loads. Appendix A contains the definition of this function, along with experimental results.\n\n5 Experiments 5.1 1 Billion Word Language Modeling Benchmark\n\nDataset: This dataset, introduced by (Chelba et al., 2013) consists of shuffled unique sentences from news articles, totaling approximately 829 million words, with a vocabulary of 793,471 words.\n\nPrevious State-of-the-Art: The best previously published results (Jozefowicz et al., 2016) use models consisting of one or more stacked Long Short-Term Memory (LSTM) layers (Hochreiter & Schmidhuber, 1997; Gers et al., 2000). The number of parameters in the LSTM layers of these models vary from 2 million to 151 million. Quality increases greatly with parameter count, as do computational costs. Results for these models form the top line of Figure 2-right. MoE Models: Our models consist of two stacked LSTM layers with a MoE layer between them (see Figure 1). We vary the sizes of the layers and the number of experts. For full details on model architecture, training regimen, additional baselines and results, see Appendix C.\n\nLow Computation, Varied Capacity: To investigate the effects of adding capacity, we trained\n\na series of MoE models all with roughly equal computational costs: about 8 million multiply-andadds per training example per timestep in the forwards pass, excluding the softmax layer. We call this metric (ops/timestep). We trained models with flat MoEs containing 4, 32, and 256 experts, and models with hierarchical MoEs containing 256, 1024, and 4096 experts. Each expert had about 1 million parameters. For all the MoE layers, 4 experts were active per input.\n\nThe results of these models are shown in Figure 2-left. The model with 4 always-active experts performed (unsurprisingly) similarly to the computationally-matched baseline models, while the largest of the models (4096 experts) achieved an impressive 24% lower perplexity on the test set. Figure 2: Model comparison on 1-Billion-Word Language-Modeling Benchmark. On the left, we plot test perplexity as a function of model capacity for models with similar computational budgets of approximately 8-million-ops-per-timestep. On the right, we plot test perplexity as a function of computational budget. The top line represents the LSTM models from (Jozefowicz et al., 2016).\n\nThe bottom line represents 4-billion parameter MoE models with different computational budgets.\n\nVaried Computation, High Capacity: In addition to the largest model from the previous section, we trained two more MoE models with similarly high capacity (4 billion parameters), but higher computation budgets. These models had larger LSTMs, and fewer but larger and experts. Details\n\nTest Test #Parameters ops/timestep Training TFLOPS Perplexity Perplexity excluding embedding Time /GPU 10 epochs 100 epochs and softmax layers 10 epochs Best Published Results 34.7 30.6 151 million 151 million 59 hours, 32 k40s 1.09 Low-Budget MoE Model 34.1 4303 million 8.9 million 15 hours, 16 k40s 0.74 Medium-Budget MoE Model 31.3 4313 million 33.8 million 17 hours, 32 k40s 1.22 High-Budget MoE Model 28.0 4371 million 142.7 million 47 hours, 32 k40s 1.56\n\nTable 1: Summary of high-capacity MoE-augmented models with varying computational budgets, vs. best previously published results (Jozefowicz et al., 2016). Details in Appendix C. can be found in Appendix C.2. Results of these three models form the bottom line of Figure 2-right.\n\nTable 1 compares the results of these models to the best previously-published result on this dataset .\n\nEven the fastest of these models beats the best published result (when controlling for the number of training epochs), despite requiring only 6% of the computation.\n\nComputational Efficiency: We trained our models using TensorFlow (Abadi et al., 2016) on clusters containing 16-32 Tesla K40 GPUs. For each of our models, we determine computational efficiency in TFLOPS/GPU by dividing the number of floating point operations required to process one training batch by the observed step time and the number of GPUs in the cluster. The operation counts used here are higher than the ones we report in our ops/timestep numbers in that we include the backwards pass, we include the importance-sampling-based training of the softmax layer, and we count a multiply-and-add as two separate operations. For all of our MoE models, the floating point operations involved in the experts represent between 37% and 46% of the total.\n\nFor our baseline models wtih no MoE, observed computational efficiency ranged from 1.07-1.29 TFLOPS/GPU. For our low-computation MoE models, computation efficiency ranged from 0.740.90 TFLOPS/GPU, except for the 4-expert model which did not make full use of the available parallelism. Our highest-computation MoE model was more efficient at 1.56 TFLOPS/GPU, likely due to the larger matrices. These numbers represent a significant fraction of the theoretical maximum of 4.29 TFLOPS/GPU claimed by NVIDIA. Detailed results are in Appendix C, Table 7.\n\n5.2 100 BILLION WORD GOOGLE NEWS CORPUS\n\nOn the 1-billion-word corpus, adding additional capacity seems to produce diminishing returns as the number of parameters in the MoE layer exceeds 1 billion, as can be seen in Figure 2-left. We hypothesized that for a larger training set, even higher capacities would produce significant quality improvements. We constructed a similar training set consisting of shuffled unique sentences from Google's internal news corpus, totalling roughly 100 billion words. Similarly to the previous section, we tested a series of models with similar computational costs of about 8 million ops/timestep. In addition to a baseline LSTM model, we trained models augmented with MoE layers containing 32, 256, 1024, 4096, 16384, 65536, and 131072 experts. This corresponds to up to 137 billion parameters in the MoE layer. Details on architecture, training, and results are given in Appendix D.\n\nResults: Figure 3 shows test perplexity as a function of capacity after training on 10 billion words (top line) and 100 billion words (bottom line). When training over the full 100 billion words, test perplexity improves significantly up to 65536 experts (68 billion parameters), dropping 39% lower than the computationally matched baseline, but degrades at 131072 experts, possibly a result of too much sparsity. The widening gap between the two lines demonstrates (unsurprisingly) that increased model capacity helps more on larger training sets. Even at 65536 experts (99.994% layer sparsity), computational efficiency for the model stays at a respectable 0.72 TFLOPS/GPU.\n\n5.3 Machine Translation (Single Language Pair)\n\nModel Architecture: Our model was a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decreased the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We inserted MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). Each MoE layer contained up to 2048 experts each with about two million parameters, adding a total of about 8 billion parameters to the models. Further details on model architecture, testing procedure and results can be found in Appendix E.\n\nDatasets: We benchmarked our method on the WMT'14 En→Fr and En→De corpora, whose training sets have 36M sentence pairs and 5M sentence pairs, respectively. The experimental protocols were also similar to those in (Wu et al., 2016): newstest2014 was used as the test set to compare against previous work (Luong et al., 2015a; Zhou et al., 2016; Wu et al., 2016), while the combination of newstest2012 and newstest2013 was used as the development set. We also tested the same model on a Google's Production English to French data.\n\nTable 2: Results on WMT'14 En→ Fr newstest2014 (bold values represent best results). Model Test Test ops/timenstep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.69 40.35 85M 8.7B 3 days/64 k40s MoE with 2048 Experts (longer training) 2.63 40.56 85M 8.7B 6 days/64 k40s GNMT (Wu et al., 2016) 2.79 39.22 214M 278M 6 days/96 k80s GNMT+RL (Wu et al., 2016) 2.96 39.92 214M 278M 6 days/96 k80s PBMT (Durrani et al., 2014) 37.0 LSTM (6-layer) (Luong et al., 2015b) 31.5 LSTM (6-layer+PosUnk) (Luong et al., 2015b) 33.1 DeepAtt (Zhou et al., 2016) 37.7 DeepAtt+PosUnk (Zhou et al., 2016) 39.2\n\nTable 3: Results on WMT'14 En → De newstest2014 (bold values represent best results). Model Test Test ops/timestep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 4.64 26.03 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 5.25 24.91 214M 278M 1 day/96 k80s GNMT +RL (Wu et al., 2016) 8.08 24.66 214M 278M 1 day/96 k80s PBMT (Durrani et al., 2014) 20.7 DeepAtt (Zhou et al., 2016) 20.6\n\nTable 4: Results on the Google Production En→ Fr dataset (bold values represent best results). Model Eval Eval Test Test ops/timestep Total Training Perplexity BLEU Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.60 37.27 2.69 36.57 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 2.78 35.80 2.87 35.56 214M 278M 6 days/96 k80s\n\nResults: Tables 2, 3, and 4 show the results of our largest models, compared with published results. Our approach achieved BLEU scores of 40.56 and 26.03 on the WMT'14 En→Fr and En→De benchmarks. As our models did not use RL refinement, these results constitute significant gains of 1.34 and 1.12 BLEU score on top of the strong baselines in (Wu et al., 2016). The perplexity scores are also better.2 On the Google Production dataset, our model achieved 1.01 higher test BLEU score even after training for only one sixth of the time.\n\n5.4 Multilingual Machine Translation\n\nDataset: (Johnson et al., 2016) train a single GNMT (Wu et al., 2016) model on a very large combined dataset of twelve language pairs. Results are somewhat worse than those for 12 separately trained single-pair GNMT models. This is not surprising, given that the twelve models have 12 times the capacity and twelve times the aggregate training of the one model. We repeat this experiment with a single MoE-augmented model. See Appendix E for details on model architecture.\n\nWe train our model on the same dataset as (Johnson et al., 2016) and process the same number of training examples (about 3 billion sentence pairs). Our training time was shorter due to the lower computational budget of our model.\n\nResults: Results for the single-pair GNMT models, the multilingual GNMT model and the multilingual MoE model are given in Table 5. The MoE model achieves 19% lower perplexity on the dev set than the multilingual GNMT model. On BLEU score, the MoE model significantly beats the multilingual GNMT model on 11 of the 12 language pairs (by as much as 5.84 points), and even beats the monolingual GNMT models on 8 of 12 language pairs. The poor performance on English → Korean seems to be a result of severe overtraining, as for the rarer language pairs a small number of real examples were highly oversampled in the training corpus.\n\nTable 5: Multilingual Machine Translation (bold values represent best results). GNMT-Mono GNMT-Multi MoE-Multi MoE-Multi vs. GNMT-Multi Parameters 278M / model 278M 8.7B ops/timestep 212M 212M 102M training time, hardware various 21 days, 96 k20s 12 days, 64 k40s Perplexity (dev) 4.14 3.35 -19% French → English Test BLEU 36.47 34.40 37.46 +3.06 German → English Test BLEU 31.77 31.17 34.80 +3.63 Japanese → English Test BLEU 23.41 21.62 25.91 +4.29 Korean → English Test BLEU 25.42 22.87 28.71 +5.84 Portuguese → English Test BLEU 44.40 42.53 46.13 +3.60 Spanish → English Test BLEU 38.00 36.04 39.39 +3.35 English → French Test BLEU 35.37 34.00 36.59 +2.59 English → German Test BLEU 26.43 23.15 24.53 +1.38 English → Japanese Test BLEU 23.66 21.10 22.78 +1.68 English → Korean Test BLEU 19.75 18.41 16.62 -1.79 English → Portuguese Test BLEU 38.40 37.35 37.90 +0.55 English → Spanish Test BLEU 34.50 34.25 36.21 +1.96\n\n6 Conclusion\n\nThis work is the first to demonstrate major wins from conditional computation in deep networks.\n\nWe carefully identified the design considerations and challenges of conditional computing and addressed them with a combination of algorithmic and engineering solutions. While we focused on text, conditional computation may help in other domains as well, provided sufficiently large training sets. We look forward to seeing many novel implementations and applications of conditional computation in the years to come.\n\nAcknowledgments\n\nWe would like to thank all of the members of the Google Brain and Google Translate teams who helped us with this project, in particular Zhifeng Chen, Yonghui Wu, and Melvin Johnson. Thanks also to our anonymous ICLR reviewers for the helpful suggestions on making this paper better.\n\n2Reported perplexities relative to the tokenization used by both our models and GNMT.\n\nReferences\n\nMartín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Gordon Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda B. Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. CoRR, abs/1603.04467, 2016. URL http://arxiv.org/abs/1603.04467.\n\nRahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. CoRR, abs/1611.06194, 2016. URL http://arxiv.org/abs/1611.\n\n06194.\n\nA. Almahairi, N. Ballas, T. Cooijmans, Y. Zheng, H. Larochelle, and A. Courville. Dynamic Capacity Networks. ArXiv e-prints, November 2015.\n\nDario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan, Christopher Fougner, Tony Han, Awni Y. Hannun, Billy Jun, Patrick LeGresley, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595, 2015. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.\n\nEmmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297, 2015.\n\nYoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.\n\nCiprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling.\n\narXiv preprint arXiv:1312.3005, 2013.\n\nK. Cho and Y. Bengio. Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning. ArXiv e-prints, June 2014. Ronan Collobert, Samy Bengio, and Yoshua Bengio. A parallel mixture of SVMs for very large scale problems. Neural Computing, 2002.\n\nAndrew Davis and Itamar Arel. Low-rank approximations for conditional feedforward computation in deep neural networks. arXiv preprint arXiv:1312.4461, 2013. Marc Peter Deisenroth and Jun Wei Ng. Distributed Gaussian processes. In ICML, 2015.\n\nJohn Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization, 2010.\n\nNadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. Edinburgh's phrase-based machine translation systems for wmt-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation, 2014.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013. Ekaterina Garmash and Christof Monz. Ensemble learning for multi-source neural machine translation. In staff.science.uva.nl/c.monz, 2016.\n\nFelix A. Gers, Jürgen A. Schmidhuber, and Fred A. Cummins. Learning to forget: Continual prediction with lstm. Neural Computation, 2000.\n\nAudrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. Memory-efficient backpropagation through time. CoRR, abs/1606.03401, 2016. URL http://arxiv.org/ abs/1606.03401. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2015.\n\nGeoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 2012. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.\n\nSergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.\n\nRobert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computing, 1991.\n\nMelvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation.\n\nCoRR, abs/1611.04558, 2016. URL http://arxiv.org/abs/1611.04558.\n\nMichael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm.\n\nNeural Computing, 1994. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.\n\nDiederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.\n\nReinhard Kneser and Hermann. Ney. Improved backingoff for m-gram language modeling., 1995.\n\nAlex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.\n\nQuoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeffrey Dean, and Andrew Y. Ng. Building high-level features using large scale unsupervised learning. In ICML, 2012. Patrick Gallinari Ludovic Denoyer. Deep sequential neural network. arXiv preprint arXiv:1410.0510, 2014.\n\nMinh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attentionbased neural machine translation. EMNLP, 2015a.\n\nMinh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the rare word problem in neural machine translation. ACL, 2015b.\n\nCarl Edward Rasmussen and Zoubin Ghahramani. Infinite mixtures of Gaussian process experts.\n\nNIPS, 2002.\n\nHasim Sak, Andrew W Senior, and Françoise Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH, pp. 338–342, 2014.\n\nMike Schuster and Kaisuke Nakajima. Japanese and Korean voice search. ICASSP, 2012.\n\nBabak Shahbaba and Radford Neal. Nonlinear models using dirichlet process mixtures. JMLR, 2009.\n\nIlya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks.\n\nIn NIPS, 2014.\n\nLucas Theis and Matthias Bethge. Generative image modeling using spatial LSTMs. In NIPS, 2015. Volker Tresp. Mixtures of Gaussian Processes. In NIPS, 2001.\n\nYonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.\n\nBangpeng Yao, Dirk Walther, Diane Beck, and Li Fei-fei. Hierarchical mixture of classification experts uncovers interactions between brain regions. In NIPS. 2009.\n\nWojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization.\n\narXiv preprint arXiv:1409.2329, 2014.\n\nJie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with fast-forward connections for neural machine translation. arXiv preprint arXiv:1606.04199, 2016.\n\nAppendices A Load-Balancing Loss\n\nAs discussed in section 4, for load-balancing purposes, we want to define an additional loss function to encourage experts to receive roughly equal numbers of training examples. Unfortunately, the number of examples received by an expert is a discrete quantity, so it can not be used in backpropagation. Instead, we define a smooth estimator Load(X) of the number of examples assigned to each expert for a batch X of inputs. The smoothness allows us to back-propagate gradients through the estimator. This is the purpose of the noise term in the gating function. We define P(x, i) as the probability that G(x)iis nonzero, given a new random choice of noise on element i, but keeping the already-sampled choices of noise on the other elements. To compute P(x, i), we note that the G(x)iis nonzero if and only if H(x)iis greater than the k th-greatest element of H(x) excluding itself. The probability works out to be:\n\n$$P(x,i)=P r\\Big((x\\cdot W_{g}){i}+S t a n d x N o r a l()\\cdot S o f t p u s((x\\cdot W{n o i s e})_{i})$$ $$>k t h_e x c l u d i n(H(x),k,i)\\Big)$$ $$({\\boldsymbol{8}})$$\n\n$$(9)$$ $$(10)$$ (8) Where kth_excluding(v, k, i) means the kth highest component of v, excluding component i. Simplifying, we get:\n\n$$P(x,i)=\\Phi{\\Big(}{\\frac{(x\\cdot W_{g}){i}-k t h_c x c l u d i n g(H(x),k,i)}{S o f t p l u s((x\\cdot W{n o i s e})_{i})}}{\\Big)}$$\n\nWhere Φ is the CDF of the standard normal distribution.\n\n$$L o a d(X){i}=\\sum{x\\in X}P(x,i)$$\n\n$$(11)$$ P(x, i) (10) We can now define the load loss to be the square of the coefficient of variation of the load vector, multiplied by a hand-tuned scaling factor wload.\n\n$$L_{l o a d}(X)=w_{l o a d}\\cdot C V(L o a d(X))^{2}$$ Lload(X) = wload · CV (Load(X))2(11) Initial Load Imbalance: To avoid out-of-memory errors, we need to initialize the network in a state of approximately equal expert load (since the soft constraints need some time to work). To accomplish this, we initialize the matrices Wg and Wnoise to all zeros, which yields no signal and some noise.\n\nExperiments: We trained a set of models with identical architecture (the MoE-256 model described in Appendix C), using different values of wimportance and wload. We trained each model for 10 epochs, then measured perplexity on the test set. We also measured the coefficients of variation in Importance and Load, as well as ratio of the load on the most overloaded expert to the average load. This last value is significant for load balancing purposes on distributed hardware. All of these metrics were averaged over several training batches.\n\nwimportance wload Test Perplexity CV (Importance(X)) CV (Load(X)) max(Load(X)) mean(Load(X)) 0.0 0.0 39.8 3.04 3.01 17.80 0.2 0.0 35.6 0.06 0.17 1.47 0.0 0.2 35.7 0.22 0.04 1.15 0.1 0.1 35.6 0.06 0.05 1.14 0.01 0.01 35.7 0.48 0.11 1.37 1.0 1.0 35.7 0.03 0.02 1.07\n\nTable 6: Experiments with different combinations of losses.\n\n13 Results: Results are reported in Table 6. All the combinations containing at least one the two losses led to very similar model quality, where having no loss was much worse. Models with higher values of wload had lower loads on the most overloaded expert.\n\nB Hierachical Mixture Of Experts\n\nIf the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network.3If the hierarchical MoE consists of a groups of b experts each, we denote the primary gating network by Gprimary, the secondary gating networks by (G1, G2..Ga), and the expert networks by (E0,0, E0,1..Ea,b). The output of the MoE is given by:\n\n$$y_{H}=\\sum_{i=1}^{a}\\sum_{j=1}^{b}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}\\cdot E{i,j}(x)$$ $$(12)$$ $$(13)$$ $$(14)$$\n\nOur metrics of expert utilization change to the following:\n\n$$I m p o r t a n c e_{H}(X){i,j}=\\sum{x\\in X}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}$$ $$L o a d{H}(X){i,j}={\\frac{L o a d{p r i m a r y}(X){i}\\cdot L o a d{i}(X^{(i)})_{j}}{|X^{(i)}|}}$$\n\nLoadprimary and Loadi deonte the Load functions for the primary gating network and i th secondary gating network respectively. X(i) denotes the subset of X for which Gprimary(x)i > 0.\n\nIt would seem simpler to let LoadH(X)i,j = Loadi(Xi)j , but this would not have a gradient with respect to the primary gating network, so we use the formulation above.\n\nC 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS C.1 8-MILLION-OPERATIONS-PER-TIMESTEP MODELS Model Architecture: Our model consists of five layers: a word embedding layer, a recurrent Long Short-Term Memory (LSTM) layer (Hochreiter & Schmidhuber, 1997; Gers et al., 2000), a MoE layer, a second LSTM layer, and a softmax layer. The dimensionality of the embedding layer, the number of units in each LSTM layer, and the input and output dimensionality of the MoE layer are all equal to 512. For every layer other than the softmax, we apply drouput (Zaremba et al., 2014) to the layer output, dropping each activation with probability DropP rob, otherwise dividing by (1 − DropP rob). After dropout, the output of the previous layer is added to the layer output.\n\nThis residual connection encourages gradient flow (He et al., 2015).\n\nMoE Layer Architecture: Each expert in the MoE layer is a feed forward network with one ReLU-activated hidden layer of size 1024 and an output layer of size 512. Thus, each expert contains [512 ∗ 1024] + [1024 ∗ 512] = 1M parameters. The output of the MoE layer is passed through a sigmoid function before dropout. We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts.\n\nWe call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers. Thus, each example is processed by exactly 4 experts for a total of 4M ops/timestep. The two LSTM layers contribute 2M ops/timestep each for the desired total of 8M.\n\n3 We have not found the need for deeper hierarchies.\n\nComputationally-Matched Baselines: The MoE-4 model does not employ sparsity, since all 4 experts are always used. In addition, we trained four more computationally-matched baseline models with no sparsity:\n\nMoE-1-Wide: The MoE layer consists of a single \"expert\" containing one ReLU-activated hidden layer of size 4096.\n\nMoE-1-Deep: The MoE layer consists of a single \"expert\" containing four ReLU-activated hidden layers, each with size 1024.\n\n4xLSTM-512: We replace the MoE layer with two additional 512-unit LSTM layers.\n\nLSTM-2048-512: The model contains one 2048-unit LSTM layer (and no MoE). The output of the LSTM is projected down to 512 dimensions (Sak et al., 2014). The next timestep of the LSTM receives the projected output. This is identical to one of the models published in (Jozefowicz et al., 2016). We re-ran it to account for differences in training regimen, and obtained results very similar to the published ones. Training: The models were trained on a cluster of 16 K40 GPUs using the synchronous method described in Section 3. Each batch consisted of a set of sentences totaling roughly 300,000 words. In the interest of time, we limited training to 10 epochs, (27,000 steps). Training took 12-16 hours for all models, except for MoE-4, which took 18 hours (since all the expert computation was performed on only 4 of 16 GPUs). We used the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 1000 training steps, and decreased after that so as to be proportional to the inverse square root of the step number. The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016). For each model, we performed a hyper-parmeter search to find the best dropout probability, in increments of 0.1.\n\nTo ensure balanced expert utilization we set wimportance = 0.1 and wload = 0.1, as described in Section 4 and Appendix A.\n\nResults: We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol. Results are reported in Table 7. For each model, we report the test perplexity, the computational budget, the parameter counts, the value of DropP rob, and the computational efficiency.\n\nModel Test Test ops/timestep #Params excluding Total Drop- TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params P rob per GPU 10 epochs (final) (millions) (billions) (observed) Kneser-Ney 5-gram* 67.6 0.00001 1.8 LSTM-512-512* 54.1 2.4 2.4 0.8 0.1 LSTM-1024-512* 48.2 4.7 4.7 0.8 0.1 LSTM-2048-512* 45.0 43.7 9.4 9.4 0.8 0.1 0.61 LSTM-2048-512 44.7 9.4 9.4 0.8 0.1 1.21 4xLSTM-512 46.0 8.4 8.4 0.8 0.1 1.07 MoE-1-Wide 46.1 8.4 8.4 0.8 0.1 1.29 MoE-1-Deep 45.7 8.4 8.4 0.8 0.1 1.29 MoE-4 45.0 8.4 8.4 0.8 0.1 0.52 MoE-32 39.7 8.4 37.8 0.9 0.1 0.87 MoE-256 35.7 8.6 272.9 1.1 0.1 0.81 MoE-256-h 36.0 8.4 272.9 1.1 0.1 0.89 MoE-1024-h 34.6 8.5 1079.0 1.9 0.2 0.90 MoE-4096-h 34.1 8.9 4303.4 5.1 0.2 0.74 2xLSTM-8192-1024* 34.7 30.6 151.0 151.0 1.8 0.25 1.09 MoE-34M 31.3 33.8 4313.9 6.0 0.3 1.22 MoE-143M 28.0 142.7 4371.1 6.0 0.4 1.56\n\nC.2 More Expensive Models\n\nWe ran two additional models (MoE-34M and MoE-143M) to investigate the effects of adding more computation in the presence of a large MoE layer. These models have computation budgets of 34M and 143M ops/timestep. Similar to the models above, these models use a MoE layer between two LSTM layers. The dimensionality of the embedding layer, and the input and output dimensionality of the MoE layer are set to 1024 instead of 512. For MoE-34M, the LSTM layers have 1024 units.\n\nFor MoE-143M, the LSTM layers have 4096 units and an output projection of size 1024 (Sak et al., 2014). MoE-34M uses a hierarchical MoE layer with 1024 experts, each with a hidden layer of size 2048. MoE-143M uses a hierarchical MoE layer with 256 experts, each with a hidden layer of size 8192. Both models have 4B parameters in the MoE layers. We searched for the best DropP rob for each model, and trained each model for 10 epochs.\n\nThe two models achieved test perplexity of 31.3 and 28.0 respectively, showing that even in the presence of a large MoE, more computation is still useful. Results are reported at the bottom of Table 7. The larger of the two models has a similar computational budget to the best published model from the literature, and training times are similar. Comparing after 10 epochs, our model has a lower test perplexity by 18%.\n\nD 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS Model Architecture: The models are similar in structure to the 8-million-operations-per-timestep models described in the previous section. We vary the number of experts between models, using an ordinary MoE layer with 32 experts and hierarchical MoE layers with 256, 1024, 4096, 16384, 65536 and 131072 experts. For the hierarchical MoE layers, the first level branching factors are 32, 32, 64, 128, 256 and 256, respectively.\n\nTraining: Models are trained on a cluster of 32 Tesla K40 GPUs, except for the last two models, which are trained on clusters of 64 and 128 GPUs so as to have enough memory for all the parameters. For all models, training batch sizes are approximately 2.5 million words. Models are trained once-through over about 100 billion words. We implement several memory optimizations in order to fit up to 1 billion parameters per GPU. First, we do not store the activations of the hidden layers of the experts, but instead recompute them on the backwards pass. Secondly, we modify the optimizer on the expert parameters to require less auxiliary storage: The Adam optimizer (Kingma & Ba, 2015) keeps first and second moment estimates of the perparameter gradients. This triples the required memory. To avoid keeping a first-moment estimator, we set β1 = 0. To reduce the size of the second moment estimator, we replace it with a factored approximation. For a matrix of parameters, instead of maintaining a full matrix of second-moment estimators, we maintain vectors of row-wise and column-wise averages of that matrix. At each step, the matrix of estimators is taken to be the outer product of those two vectors divided by the mean of either one. This technique could similarly be applied to Adagrad (Duchi et al., 2010).\n\nModel Test Test ops/timestep #Params excluding Total TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params per GPU .1 epochs 1 epoch (millions) (billions) (observed) Kneser-Ney 5-gram 67.1 45.3 0.00001 76.0 4xLSTM-512 54.5 47.0 8.4 8.4 0.1 1.23 MoE-32 48.5 40.4 8.4 37.8 0.1 0.83 MoE-256-h 42.8 35.3 8.4 272.9 0.4 1.11 MoE-1024-h 40.3 32.7 8.5 1079.0 1.2 1.14 MoE-4096-h 38.9 30.9 8.6 4303.4 4.4 1.07 MoE-16384-h 38.2 29.7 8.8 17201.0 17.3 0.96 MoE-65536-h 38.2 28.9 9.2 68791.0 68.9 0.72 MoE-131072-h 39.8 29.2 9.7 137577.6 137.7 0.30\n\nResults: We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8. Perplexity after 100 billion training words is 39% lower for the 68-billion-parameter MoE model than for the baseline model. It is notable that the measured computational efficiency of the largest model (0.30 TFLOPS/GPU) is very low compared to the other models. This is likely a result of the fact that, for purposes of comparison to the other models, we did not increase the training batch size proportionally to the number of GPUs. For comparison, we include results for a computationally matched baseline model consisting of 4 LSTMs, and for an unpruned 5-gram model with Kneser-Ney smoothing (Kneser & Ney, 1995).4\n\nE Machine Translation - Experimental Details\n\nModel Architecture for Single Language Pair MoE Models: Our model is a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decrease the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We insert MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). We use an attention mechanism between the encoder and decoder, with the first decoder LSTM receiving output from and providing input for the attention 5. All of the layers in our model have input and output dimensionality of 512. Our LSTM layers have 2048 hidden units, with a 512-dimensional output projection. We add residual connections around all LSTM and MoE layers to encourage gradient flow (He et al., 2015). Similar to GNMT, to effectively deal with rare words, we used subword units (also known as \"wordpieces\") (Schuster & Nakajima, 2012) for inputs and outputs in our system.\n\nWe use a shared source and target vocabulary of 32K wordpieces. We also used the same beam search technique as proposed in (Wu et al., 2016).\n\nWe train models with different numbers of experts in the MoE layers. In addition to a baseline model with no MoE layers, we train models with flat MoE layers containing 32 experts, and models with hierarchical MoE layers containing 512 and 2048 experts. The flat MoE layers use k = 4 and the hierarchical MoE models use k = 2 at each level of the gating network. Thus, each input is processed by exactly 4 experts in each MoE layer. Each expert in the MoE layer is a feed forward network with one hidden layer of size 2048 and ReLU activation. Thus, each expert contains [512 ∗ 2048] + [2048 ∗ 512] = 2M parameters. The output of the MoE layer is passed through a sigmoid function. We use the strictly-balanced gating function described in Appendix F.\n\nModel Architecture for Multilingual MoE Model: We used the same model architecture as for the single-language-pair models, with the following exceptions: We used noisy-top-k gating as described in Section 2.1, not the scheme from Appendix F. The MoE layers in the encoder and decoder are non-hierarchical MoEs with n = 512 experts, and k = 2. Each expert has a larger hidden layer of size 8192. This doubles the amount of computation in the MoE layers, raising the computational budget of the entire model from 85M to 102M ops/timestep. Training: We trained our networks using the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 2000 training steps, held constant for an additional 8000 steps, and decreased after that so as to be proportional to the inverse square root of the step number. For the single-language-pair models, similarly to (Wu et al., 2016), we applied dropout (Zaremba et al., 2014) to the output of all embedding, LSTM and MoE layers, using DropP rob = 0.4. Training was done synchronously on a cluster of up to 64 GPUs as described in section 3. Each training batch consisted of a set of sentence pairs containing roughly 16000 words per GPU.\n\nTo ensure balanced expert utilization we set wimportance = 0.01 and wload = 0.01, as described in Section 4 and Appendix A. Metrics: We evaluated our models using the perplexity and the standard BLEU score metric. We reported tokenized BLEU score as computed by the multi-bleu.pl script, downloaded from the public implementation of Moses (on Github), which was also used in (Luong et al., 2015a). Results: Tables 2, 3 and 4 in Section 5.3 show comparisons of our results to other published methods. Figure 4 shows test perplexity as a function of number of words in the (training data's) source sentences processed for models with different numbers of experts. As can be seen from the Figure, as we increased the number of experts to approach 2048, the test perplexity of our model continued to improve.\n\nWe found that the experts indeed become highly specialized by syntax and/or semantics, as can be seen in Table 9. For example, one expert is used when the indefinite article \"a\" introduces the direct object in a verb phrase indicating importance or leadership.\n\nExpert 381 Expert 752 Expert 2004 ... with researchers , ... ... plays a core ... ... with rapidly growing ... ... to innovation . ... plays a critical ... ... under static conditions ... ... tics researchers . ... provides a legislative ... ... to swift ly ... ... the generation of ... ... play a leading ... ... to dras tically ... ... technology innovations is ... ... assume a leadership ... ... the rapid and ... ... technological innovations , ... ... plays a central ... ... the fast est ... ... support innovation throughout ... ... taken a leading ... ... the Quick Method ... ... role innovation will ... ... established a reconciliation ... ... rec urrent ) ... ... research scienti st ... ... played a vital ... ... provides quick access ... ... promoting innovation where ... ... have a central ... ... of volatile organic ... ... ... ...\n\nDue to some peculiarities in our infrastructure which have since been fixed, at the time we ran some of the machine translation experiments, our models ran faster if every expert received exactly the same batch size. To accommodate this, we used a different gating function which we describe below. Recall that we define the softmax gating function to be:\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ Gσ(x) = Sof tmax(x · Wg) (15) Sparse Gating (alternate formulation): To obtain a sparse gating vector, we multiply Gσ(x) component-wise with a sparse mask M(Gσ(x)) and normalize the output. The mask itself is a function of Gσ(x) and specifies which experts are assigned to each input example:\n\n$$(15)$$ $$G(x){i}=\\frac{G{\\sigma}(x){i}M(G{\\sigma}(x)){i}}{\\sum{j=1}^{n}G_{\\sigma}(x){j}M(G{\\sigma}(x))_{j}}\\tag{1}$$ $$(17)$$\n\n$$(16)$$\n\nTop-K Mask: To implement top-k gating in this formulation, we would let M(v) = T opK(v, k), where:\n\n$$T o p K(v,k){i}=\\begin{cases}1&{\\mathrm{if~}}v{i}{\\mathrm{~is~in~the~top~}}k{\\mathrm{~elements~of~}}v.\\ 0&{\\mathrm{otherwise.}}\\end{cases}$$ 0 otherwise. (17) Batchwise Mask: To force each expert to receive the exact same number of examples, we introduce an alternative mask function, Mbatchwise(X, m), which operates over batches of input vectors.\n\nInstead of keeping the top k values per example, we keep the top m values per expert across the training batch, where m = k|X| n, so that each example is sent to an average of k experts.\n\n$$M_{b a t c h w i s e}(X,m){j,i}=\\begin{cases}1&\\text{if}X{j,i}\\text{is in the top}m\\text{values for to expert}i\\ 0&\\text{otherwise}\\end{cases}$$\n\nAs our experiments suggest and also observed in (Ioffe & Szegedy, 2015), using a batchwise function during training (such as Mbatchwise) requires modifications to the inference when we may not have a large batch of examples. Our solution to this is to train a vector T of per-expert threshold values to approximate the effects of the batchwise mask. We use the following mask at inference time:\n\n$$(18)$$ $$M_{threshold}(x,T){i}=\\begin{cases}1&\\text{if}x{i}>T_{i}\\ 0&\\text{otherwise}\\end{cases}\\tag{1}$$ $$(19)$$\n\nTo learn the threshold values, we apply an additional loss at training time which is minimized when the batchwise mask and the threshold mask are identical.\n\n$$L_{batchwise}(X,T,m)=\\sum_{j=1}^{|X|}\\sum_{i=1}^{n}(M_{threshold}(x,T){i}-M{batchwise}(X,m){j,i})(X{j,i}-T_{i})\\tag{20}$$\n\nG ATTENTION FUNCTION The attention mechanism described in GNMT (Wu et al., 2016) involves a learned \"Attention Function\" A(xi, yj ) which takes a \"source vector\" xi and a \"target vector\" yj , and must be computed for every source time step i and target time step j. In GNMT, the attention function is implemented as a feed forward neural network with a hidden layer of size n. It can be expressed as:\n\n$$A_{G N M T}(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}t a n h((x_{i}U){d}+(y{j}W)_{d})$$\n\nWhere U and W are trainable weight matrices and V is a trainable weight vector.\n\nFor performance reasons, in our models, we used a slightly different attention function:\n\n$$A(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}tanh((x_{i}U){d})tanh((y{j}W)_{d})\\tag{22}$$\n\nWith our attention function, we can simultaneously compute the attention function on multiple source time steps and multiple target time steps using optimized matrix multiplications. We found little difference in quality between the two functions.\n\n$$(21)$$",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/1701.06538v1.md"
+          },
+          "headlines": {
+            "1 Introduction And Related Work": [
+              "1.1 Conditional Computation",
+              "1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER",
+              "1.3 Related Work On Mixtures Of Experts"
+            ],
+            "2 The Structure Of The Mixture-Of-Experts Layer": [
+              "2.1 Gating Network"
+            ],
+            "3 Addressing Performance Challenges": [
+              "3.1 The Shrinking Batch Problem",
+              "3.2 Network Bandwidth"
+            ],
+            "4 Balancing Expert Utilization": [],
+            "5 Experiments": [
+              "5.1 1 Billion Word Language Modeling Benchmark",
+              "5.2 100 BILLION WORD GOOGLE NEWS CORPUS",
+              "5.3 Machine Translation (Single Language Pair)",
+              "5.4 Multilingual Machine Translation"
+            ],
+            "6 Conclusion": [],
+            "Appendices": [
+              "A Load-Balancing Loss",
+              "B Hierachical Mixture Of Experts",
+              "C 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS",
+              "D 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS",
+              "E Machine Translation - Experimental Details"
+            ]
+          },
+          "summary": "The paper introduces a Sparsely-Gated Mixture-of-Experts (MoE) layer to enhance neural network capacity without a proportional increase in computation. This approach uses a trainable gating network to select a sparse combination of experts for each input, significantly improving model capacity and performance in language modeling and machine translation tasks. The MoE layer, with up to 137 billion parameters, achieves better results than state-of-the-art models at lower computational costs. The paper addresses challenges like batch size and network bandwidth, proposing solutions like mixing data and model parallelism. Experiments show that MoE models outperform traditional models in various benchmarks, demonstrating the potential of conditional computation in deep learning.",
+          "summary_embedding": [
+            -0.01644345559179783,
+            -0.016995344310998917,
+            -0.016372699290513992,
+            -0.014349107630550861,
+            0.02282555401325226,
+            0.045934125781059265,
+            -0.014646278694272041,
+            0.03563220426440239,
+            -0.004829026758670807,
+            0.005617944523692131,
+            -0.00932550523430109,
+            -0.008143897168338299,
+            0.005575491581112146,
+            0.01538213063031435,
+            -0.005653322208672762,
+            -0.020306676626205444,
+            -0.013832597061991692,
+            -0.023235931992530823,
+            0.028344441205263138,
+            -0.020080260932445526,
+            0.011823155917227268,
+            -0.020943470299243927,
+            -0.025868017226457596,
+            -0.016500059515237808,
+            0.025061409920454025,
+            0.024806693196296692,
+            0.02583971433341503,
+            0.003827843815088272,
+            -0.03492465242743492,
+            0.033651065081357956,
+            0.014674580655992031,
+            0.012488252483308315,
+            -0.031500112265348434,
+            -0.009162768721580505,
+            -0.05351905897259712,
+            -0.05227377265691757,
+            0.04143410921096802,
+            0.022245362401008606,
+            -0.016231190413236618,
+            0.023717066273093224,
+            0.017264213413000107,
+            0.005816058721393347,
+            -0.020632149651646614,
+            -0.020844414830207825,
+            0.003700484987348318,
+            -0.013103820383548737,
+            -0.01983969286084175,
+            -0.013528349809348583,
+            -0.01828308403491974,
+            -0.01577835902571678,
+            0.009551920928061008,
+            0.0002856731880456209,
+            0.0029186424799263477,
+            -0.003424540627747774,
+            -0.047830358147621155,
+            0.0248208437114954,
+            0.019853845238685608,
+            0.03492465242743492,
+            0.04633035510778427,
+            0.013011839240789413,
+            0.01516986545175314,
+            -0.014271277002990246,
+            -0.04021712392568588,
+            0.005497660953551531,
+            -0.00041656987741589546,
+            -0.005380915477871895,
+            -0.005819596350193024,
+            0.008575502783060074,
+            0.008724087849259377,
+            -0.01143400277942419,
+            -0.04202845320105553,
+            0.013952880166471004,
+            -0.020561393350362778,
+            0.00010143388499272987,
+            0.01354957651346922,
+            0.008292482234537601,
+            0.02677368000149727,
+            0.03195294365286827,
+            0.023971782997250557,
+            -0.008724087849259377,
+            -0.01340806670486927,
+            0.019924599677324295,
+            -0.0022676968947052956,
+            0.007018893025815487,
+            0.04570771008729935,
+            -0.04033033177256584,
+            0.02163686975836754,
+            -0.025726506486535072,
+            -0.015339677222073078,
+            -0.005349075887352228,
+            -0.022471778094768524,
+            0.02806142158806324,
+            -0.007344366051256657,
+            0.017801949754357338,
+            0.017094401642680168,
+            0.01594817079603672,
+            0.013330236077308655,
+            -0.001894464367069304,
+            -0.022938761860132217,
+            0.0018360915128141642,
+            -0.01541043259203434,
+            0.02114158496260643,
+            0.0015230007702484727,
+            0.005469358991831541,
+            0.027594437822699547,
+            0.031160488724708557,
+            -0.023490650579333305,
+            0.009502392262220383,
+            0.02422650158405304,
+            0.06967949122190475,
+            0.015424583107233047,
+            -0.006435164716094732,
+            -0.0024640418123453856,
+            0.04143410921096802,
+            0.0015513027319684625,
+            0.0038349193055182695,
+            -0.06826439499855042,
+            -0.03022652305662632,
+            -0.0011365016689524055,
+            -0.02254253253340721,
+            0.02227366529405117,
+            0.01002597901970148,
+            -0.009750034660100937,
+            -0.022457627579569817,
+            -0.04516997188329697,
+            0.05190584436058998,
+            -0.00545874610543251,
+            -0.016259491443634033,
+            -0.005554265342652798,
+            -0.05340585112571716,
+            0.00039136342820711434,
+            0.015042506158351898,
+            -0.018481198698282242,
+            -0.03670767694711685,
+            0.036141637712717056,
+            0.0007398316520266235,
+            0.03741522878408432,
+            0.012459950521588326,
+            -0.01708024926483631,
+            -0.020094411447644234,
+            0.022259512916207314,
+            -0.011639192700386047,
+            -0.0038313816767185926,
+            -0.027339721098542213,
+            0.0065236082300543785,
+            -0.028103873133659363,
+            -0.0014566679019480944,
+            0.06730213016271591,
+            -0.04816998168826103,
+            0.02946236915886402,
+            -0.029660483822226524,
+            0.023660462349653244,
+            -0.008575502783060074,
+            0.0278916098177433,
+            -0.010238244198262691,
+            0.005840822588652372,
+            -0.013457595370709896,
+            -0.0010586712742224336,
+            -0.020745357498526573,
+            0.0034210027661174536,
+            -0.016259491443634033,
+            -0.027750099077820778,
+            -0.007054270710796118,
+            0.031273696571588516,
+            0.01698119193315506,
+            0.025655752047896385,
+            0.025429336354136467,
+            0.01246702577918768,
+            -0.04935866594314575,
+            0.02595292218029499,
+            0.0018926954362541437,
+            0.013514199294149876,
+            0.013181651011109352,
+            -0.0010418669553473592,
+            0.025599148124456406,
+            0.012049571610987186,
+            0.025061409920454025,
+            0.037443529814481735,
+            0.0027682881336659193,
+            0.04737752676010132,
+            -0.015693452209234238,
+            -0.005292471963912249,
+            0.028528403490781784,
+            0.000505455827806145,
+            0.02465103194117546,
+            -0.001686621573753655,
+            0.013330236077308655,
+            0.000776978034991771,
+            0.03834919258952141,
+            -0.006339645478874445,
+            0.029518973082304,
+            0.021608568727970123,
+            0.028316138312220573,
+            -0.014759486541152,
+            0.023122724145650864,
+            -0.019754787907004356,
+            -0.02292460948228836,
+            0.035547297447919846,
+            0.004082561936229467,
+            0.01886327564716339,
+            -0.028457649052143097,
+            -0.030679354444146156,
+            0.015452885068953037,
+            -0.01855195313692093,
+            0.016556663438677788,
+            0.03639635816216469,
+            0.0008273909334093332,
+            -0.0025719432160258293,
+            0.026292547583580017,
+            -0.04245298355817795,
+            0.012856177985668182,
+            -0.01532552670687437,
+            -0.02855670638382435,
+            -0.019528372213244438,
+            -0.00875946506857872,
+            -0.0018095583654940128,
+            -0.013691086322069168,
+            -0.0037570889107882977,
+            -0.007563706487417221,
+            -0.02459442801773548,
+            0.00935380719602108,
+            0.013627407141029835,
+            0.009849091991782188,
+            -0.018665160983800888,
+            0.0019952901639044285,
+            -0.008257105015218258,
+            -0.0160047747194767,
+            -0.03752843663096428,
+            0.015268922783434391,
+            -0.0024888061452656984,
+            0.009191070683300495,
+            0.04078316316008568,
+            0.028485950082540512,
+            -0.005055442452430725,
+            0.0329718142747879,
+            -0.033707667142152786,
+            0.017915157601237297,
+            -0.008320784196257591,
+            0.009191070683300495,
+            0.002989397384226322,
+            -0.025542544201016426,
+            0.027424626052379608,
+            -0.03792466223239899,
+            -0.038151077926158905,
+            0.022995365783572197,
+            0.011858533136546612,
+            0.0017096169758588076,
+            -0.019386861473321915,
+            -0.017717044800519943,
+            0.0008499440737068653,
+            0.002752368338406086,
+            -0.018834972754120827,
+            -0.01019579079002142,
+            0.02227366529405117,
+            -0.025500090792775154,
+            0.036085035651922226,
+            0.023391593247652054,
+            -0.03520767390727997,
+            -0.00608492735773325,
+            -0.01903308741748333,
+            0.059037946164608,
+            -0.0011506526498124003,
+            -0.010301923379302025,
+            0.022938761860132217,
+            -0.027268964797258377,
+            -0.004524780437350273,
+            0.022301966324448586,
+            0.01828308403491974,
+            -0.008278331719338894,
+            0.003983504604548216,
+            -0.022287815809249878,
+            -0.05281151086091995,
+            -0.010577867738902569,
+            -0.0003305583668407053,
+            0.054368119686841965,
+            -0.012367968447506428,
+            0.0020695829298347235,
+            -0.006413938011974096,
+            0.004100250545889139,
+            0.03704730048775673,
+            0.002140337834134698,
+            0.0025082635693252087,
+            -0.013698162510991096,
+            0.0371888130903244,
+            0.022316116839647293,
+            0.005883275531232357,
+            -0.012162779457867146,
+            0.005377377849072218,
+            -0.04899074137210846,
+            -0.0009118546731770039,
+            0.005741765722632408,
+            0.007386818993836641,
+            -0.025910470634698868,
+            0.017717044800519943,
+            0.009205221198499203,
+            -0.0024693484883755445,
+            -0.009792488068342209,
+            0.01173117384314537,
+            0.025429336354136467,
+            -0.019938750192523003,
+            -0.05700020492076874,
+            0.0034705311991274357,
+            0.012014193460345268,
+            -0.004054259974509478,
+            -0.019401011988520622,
+            0.006792477332055569,
+            0.0021934041287750006,
+            0.022698193788528442,
+            -0.011271266266703606,
+            -0.0029221803415566683,
+            0.040585052222013474,
+            0.009516543708741665,
+            -0.0016813150141388178,
+            -0.017717044800519943,
+            -0.011794853024184704,
+            0.018070818856358528,
+            0.004740582779049873,
+            0.0035289041697978973,
+            -0.01751893013715744,
+            -0.029264256358146667,
+            0.02800481766462326,
+            0.019967053085565567,
+            -0.048905834555625916,
+            -0.008200501091778278,
+            -0.019273653626441956,
+            0.0019599127117544413,
+            0.008695785887539387,
+            0.04613224044442177,
+            -0.00590803986415267,
+            0.008313708938658237,
+            -0.020179318264126778,
+            -0.05003791302442551,
+            -0.02708500251173973,
+            0.03707560524344444,
+            -0.028684064745903015,
+            -0.04276430234313011,
+            -0.010252394713461399,
+            0.016740625724196434,
+            -0.009403335861861706,
+            -0.018679311498999596,
+            -0.011738249100744724,
+            -0.04969828948378563,
+            -0.003497064346447587,
+            0.0337359681725502,
+            0.017264213413000107,
+            -0.008441068232059479,
+            -0.033877480775117874,
+            0.019641580060124397,
+            -0.007160402834415436,
+            -0.006474079564213753,
+            0.0007862646016292274,
+            0.008794843219220638,
+            0.02400008589029312,
+            0.011879759840667248,
+            -0.02763689123094082,
+            -0.03427370637655258,
+            -0.0024888061452656984,
+            0.014929298311471939,
+            -0.00878776703029871,
+            0.0245378240942955,
+            0.024679332971572876,
+            -0.022995365783572197,
+            -0.01654251106083393,
+            0.015750056132674217,
+            -0.05779265984892845,
+            -0.010287772864103317,
+            -0.029235953465104103,
+            -0.006183984223753214,
+            0.0009613831643946469,
+            0.0061096916906535625,
+            0.00012304731353651732,
+            0.03846240043640137,
+            0.008801918476819992,
+            0.006014172453433275,
+            0.009191070683300495,
+            0.01913214474916458,
+            0.028471799567341805,
+            0.009155693463981152,
+            -0.03517937287688255,
+            0.02850010246038437,
+            -0.017051948234438896,
+            0.007811348885297775,
+            -0.025754809379577637,
+            -0.013033065013587475,
+            0.00443633645772934,
+            0.02725481428205967,
+            -0.000503244751598686,
+            0.018849123269319534,
+            -0.021919889375567436,
+            0.037500135600566864,
+            -0.029915200546383858,
+            -0.05278320610523224,
+            -0.02005195803940296,
+            -0.008285406976938248,
+            0.015764206647872925,
+            0.017278363928198814,
+            -0.019924599677324295,
+            -0.0286557637155056,
+            -0.021750077605247498,
+            0.006470541935414076,
+            0.010542490519583225,
+            -0.021099131554365158,
+            -0.010627396404743195,
+            0.01502835564315319,
+            -0.00404364662244916,
+            -0.01886327564716339,
+            0.013684011064469814,
+            0.015693452209234238,
+            0.03169822692871094,
+            -0.03308502212166786,
+            0.004287751391530037,
+            -0.025967074558138847,
+            -0.04327373951673508,
+            0.004096712917089462,
+            0.03166992589831352,
+            0.027509532868862152,
+            -0.018636859953403473,
+            0.010754755698144436,
+            0.003990580327808857,
+            0.004234685096889734,
+            -0.0033803186379373074,
+            0.023603858426213264,
+            0.03291521221399307,
+            0.02763689123094082,
+            -0.011653343215584755,
+            0.004921008367091417,
+            -0.01559439580887556,
+            -0.0011108530452474952,
+            0.01865101046860218,
+            -0.017816102132201195,
+            0.033877480775117874,
+            -0.029405765235424042,
+            0.02104252763092518,
+            -0.035122767090797424,
+            0.006141531281173229,
+            -0.012099100276827812,
+            -0.00404364662244916,
+            0.04833979532122612,
+            -0.017646290361881256,
+            -0.0019546060357242823,
+            -0.008101443760097027,
+            -0.014575524255633354,
+            -0.02158026583492756,
+            0.008766540326178074,
+            -0.04033033177256584,
+            -0.012367968447506428,
+            -0.028429346159100533,
+            0.003424540627747774,
+            0.019910449162125587,
+            -0.06277380883693695,
+            -0.06237758323550224,
+            0.012275987304747105,
+            -0.04918885603547096,
+            -0.026419905945658684,
+            0.02963218092918396,
+            -0.002984090941026807,
+            0.0012780115939676762,
+            -0.0024003623984754086,
+            0.0068101659417152405,
+            -0.029207652434706688,
+            0.00425944896414876,
+            -0.000887974863871932,
+            0.03271709755063057,
+            -0.021056678146123886,
+            -0.008122670464217663,
+            0.004878554958850145,
+            -0.06860402226448059,
+            -0.028344441205263138,
+            0.011271266266703606,
+            -0.009459939785301685,
+            -0.0066615804098546505,
+            -0.017915157601237297,
+            -0.004383270163089037,
+            0.0008950503543019295,
+            0.0032830305863171816,
+            0.05801907554268837,
+            -0.01983969286084175,
+            -0.053773775696754456,
+            0.03633975237607956,
+            0.00887267291545868,
+            0.029603879898786545,
+            -0.025556694716215134,
+            -0.017533080652356148,
+            -0.01855195313692093,
+            -0.020504789426922798,
+            -0.04123599827289581,
+            0.00404364662244916,
+            -0.005918653216212988,
+            0.009212297387421131,
+            -0.0009693430620245636,
+            0.02763689123094082,
+            0.029235953465104103,
+            0.023108573630452156,
+            0.012750045396387577,
+            0.01079013291746378,
+            -0.022372720763087273,
+            -0.021594416350126266,
+            0.0012222920777276158,
+            0.04683978855609894,
+            0.024636881425976753,
+            -0.02675952948629856,
+            -0.011504758149385452,
+            0.03953787684440613,
+            0.0419718474149704,
+            -0.01974063739180565,
+            -0.02276895008981228,
+            0.00764861237257719,
+            0.03266049548983574,
+            -0.04477374628186226,
+            0.005932804197072983,
+            0.03410389646887779,
+            0.014405711553990841,
+            -0.005356151144951582,
+            -0.010294848121702671,
+            0.022570835426449776,
+            0.03152841329574585,
+            -0.008724087849259377,
+            0.01876421831548214,
+            0.016910437494516373,
+            0.029349161311984062,
+            -0.0067889392375946045,
+            0.0005766529939137399,
+            -0.024735936895012856,
+            0.055188875645399094,
+            -0.03059444949030876,
+            0.02909444272518158,
+            0.010316074825823307,
+            -0.00593634182587266,
+            0.039056744426488876,
+            -0.016641568392515182,
+            -0.0431322306394577,
+            0.01913214474916458,
+            -0.006979977712035179,
+            -0.0055613406002521515,
+            -0.03444352000951767,
+            -0.007592008449137211,
+            -0.007507102098315954,
+            -0.006714646704494953,
+            -0.022556684911251068,
+            -0.02606613002717495,
+            0.004415110219269991,
+            0.00973588414490223,
+            -0.006484692916274071,
+            0.00698705343529582,
+            0.02092931978404522,
+            -0.013507124036550522,
+            0.02525952458381653,
+            -0.046047333627939224,
+            -0.0008366775000467896,
+            -0.0038596836384385824,
+            -0.025712355971336365,
+            -0.0345001220703125,
+            -0.0005770951975136995,
+            -0.0024251264985650778,
+            -0.024622729048132896,
+            -0.00914154201745987,
+            -0.008752389810979366,
+            -0.0023119186516851187,
+            -0.004938696976751089,
+            -0.055188875645399094,
+            0.008695785887539387,
+            0.010846736840903759,
+            -0.002713453024625778,
+            0.0055507272481918335,
+            -0.010613245889544487,
+            -0.03039633482694626,
+            0.03039633482694626,
+            0.04129260033369064,
+            0.033707667142152786,
+            0.014334957115352154,
+            -0.0074646491557359695,
+            -0.017165156081318855,
+            0.006060163024812937,
+            0.021721776574850082,
+            0.00902833417057991,
+            0.007846726104617119,
+            0.03877372294664383,
+            0.00011331850691931322,
+            -0.008349086157977581,
+            -0.009764186106622219,
+            0.028740668669342995,
+            -0.001085204305127263,
+            0.023603858426213264,
+            0.050802066922187805,
+            0.030424637719988823,
+            0.020589696243405342,
+            -0.01681138016283512,
+            -0.02010856196284294,
+            -0.00851889792829752,
+            -0.02855670638382435,
+            0.0364246591925621,
+            0.012127402238547802,
+            -0.005232329946011305,
+            -0.011540135368704796,
+            0.019061388447880745,
+            -0.033339742571115494,
+            -0.0037358624394983053,
+            0.00048422932741232216,
+            0.013924578204751015,
+            -0.03379257395863533,
+            0.0002480846014805138,
+            0.031613320112228394,
+            0.014957600273191929,
+            -0.016995344310998917,
+            -0.0188774261623621,
+            0.010549565777182579,
+            0.020462337881326675,
+            0.018948180601000786,
+            0.00037743354914709926,
+            0.00897173024713993,
+            -0.00022121983056422323,
+            -0.0005319889169186354,
+            0.01590571738779545,
+            -0.007952858693897724,
+            0.036254845559597015,
+            -0.003525366308167577,
+            -0.01325240544974804,
+            -0.0037500134203583,
+            -0.021438755095005035,
+            -0.024467067793011665,
+            0.043556760996580124,
+            0.023971782997250557,
+            0.04661337286233902,
+            -0.005660397466272116,
+            -0.004245297983288765,
+            0.025825563818216324,
+            0.008221727795898914,
+            -0.03288691118359566,
+            0.000265110022155568,
+            0.005281858611851931,
+            -0.02892463095486164,
+            0.016641568392515182,
+            -0.02476423978805542,
+            -0.016896286979317665,
+            -0.006771250627934933,
+            -0.0015557248843833804,
+            -0.028188779950141907,
+            0.01703779585659504,
+            0.0031309074256569147,
+            -0.0027099153958261013,
+            -0.039764292538166046,
+            0.006880920846015215,
+            0.020419884473085403,
+            0.015311375260353088,
+            0.0037358624394983053,
+            0.011525984853506088,
+            -0.016613267362117767,
+            0.002126186853274703,
+            0.03379257395863533,
+            0.004878554958850145,
+            0.02005195803940296,
+            0.0027647505048662424,
+            0.023773670196533203,
+            0.03633975237607956,
+            -0.01681138016283512,
+            -0.02189158834517002,
+            0.01519816741347313,
+            -0.02233026921749115,
+            0.029943503439426422,
+            0.004889168310910463,
+            -0.012283062562346458,
+            0.022033097222447395,
+            -0.009827865287661552,
+            -0.0027346794959157705,
+            0.01659911684691906,
+            -0.022740647196769714,
+            -0.017717044800519943,
+            0.03183973580598831,
+            -0.039452970027923584,
+            0.012856177985668182,
+            -0.008547200821340084,
+            0.015863263979554176,
+            -0.023066120222210884,
+            0.011172209866344929,
+            0.007100261282175779,
+            -0.01573590561747551,
+            0.010797208175063133,
+            0.009693430736660957,
+            0.00545874610543251,
+            0.0018926954362541437,
+            0.02179253101348877,
+            -0.0006049550138413906,
+            -0.011257115751504898,
+            -0.018155725672841072,
+            -0.004694592207670212,
+            -0.002614396158605814,
+            -0.016556663438677788,
+            0.01865101046860218,
+            -0.041066184639930725,
+            -0.013280707411468029,
+            -0.008879749104380608,
+            -0.013988257385790348,
+            -0.010011828504502773,
+            0.024184048175811768,
+            0.025669902563095093,
+            -0.002361447084695101,
+            -0.031217092648148537,
+            0.01790100708603859,
+            0.024198200553655624,
+            0.003654494183138013,
+            -0.023646311834454536,
+            0.0302831269800663,
+            -0.008936353027820587,
+            0.014971751719713211,
+            0.026136886328458786,
+            0.014094389975070953,
+            0.004595535341650248,
+            -0.010500037111341953,
+            -0.02114158496260643,
+            0.01481609046459198,
+            0.010507113300263882,
+            0.033537857234478,
+            0.014745336025953293,
+            0.02071705460548401,
+            0.01594817079603672,
+            -0.023320838809013367,
+            -0.011214662343263626,
+            -0.003482913365587592,
+            -0.022514231503009796,
+            0.03093407303094864,
+            0.0006129149114713073,
+            0.02114158496260643,
+            0.026618018746376038,
+            -0.03319823369383812,
+            -0.01187268365174532,
+            0.032688796520233154,
+            0.02779255248606205,
+            0.05108508840203285,
+            0.025316128507256508,
+            0.015396281145513058,
+            0.027226513251662254,
+            -0.03585861995816231,
+            -0.036368053406476974,
+            0.001548649393953383,
+            0.04256619140505791,
+            -0.008426916785538197,
+            0.00967928022146225,
+            0.03900013864040375,
+            -0.006781863979995251,
+            -0.01611798256635666,
+            0.019457615911960602,
+            0.0026710000820457935,
+            0.002448122017085552,
+            -0.0033113325480371714,
+            -0.01335146278142929,
+            0.03175482898950577,
+            -0.0051474240608513355,
+            0.013570803217589855,
+            -0.019273653626441956,
+            0.027113303542137146,
+            0.0001799829478841275,
+            -0.003229964291676879,
+            0.028344441205263138,
+            -0.002660386962816119,
+            -0.051990751177072525,
+            0.024212351068854332,
+            0.019811391830444336,
+            -0.011561362072825432,
+            -0.0036332677118480206,
+            -0.003792466362938285,
+            0.04567940905690193,
+            0.034698236733675,
+            0.029009537771344185,
+            -0.0020501252729445696,
+            0.0168679840862751,
+            0.018792519345879555,
+            0.0053667644970119,
+            0.014278353191912174,
+            -0.0017025414854288101,
+            -0.007853801362216473,
+            0.021056678146123886,
+            -0.020787810906767845,
+            0.03226426616311073,
+            -0.016570813953876495,
+            -0.020122714340686798,
+            0.008738238364458084,
+            -0.011172209866344929,
+            -0.0009083169279620051,
+            -0.006311343517154455,
+            -0.0008813415770418942,
+            -0.0029664020985364914,
+            0.016938740387558937,
+            -0.004446949809789658,
+            0.04661337286233902,
+            -0.013931653462350368,
+            -0.008808993734419346,
+            0.006972902454435825,
+            -0.025981225073337555,
+            0.0013346156338229775,
+            0.0028726516757160425,
+            0.027000095695257187,
+            -0.01989629678428173,
+            -0.00970758218318224,
+            0.01606137864291668,
+            0.06039644405245781,
+            -0.008639181964099407,
+            -0.0026267783250659704,
+            0.027226513251662254,
+            -0.002787745790556073,
+            -0.03551899641752243,
+            0.012919857166707516,
+            -0.04494355618953705,
+            0.017589686438441277,
+            0.026349151507019997,
+            -0.04986810311675072,
+            -0.022089701145887375,
+            -0.02347649820148945,
+            -0.02043403498828411,
+            0.023434046655893326,
+            -0.009976450353860855,
+            0.013867974281311035,
+            -0.017801949754357338,
+            -0.009014182724058628,
+            0.020518941804766655,
+            0.009176919236779213,
+            0.03735862299799919,
+            -0.027551984414458275,
+            -0.016783079132437706,
+            -0.027820853516459465,
+            -0.04304732382297516,
+            0.009558996185660362,
+            -0.0278916098177433,
+            0.0027099153958261013,
+            -0.0016291331266984344,
+            -0.040302030742168427,
+            -0.037981268018484116,
+            0.005278320983052254,
+            -0.017985913902521133,
+            0.01516986545175314,
+            0.020915169268846512,
+            0.02731141820549965,
+            -0.0020253609400242567,
+            -0.030198222026228905,
+            0.029235953465104103,
+            -0.007344366051256657,
+            0.03223596513271332,
+            -0.00689507182687521,
+            0.005066055804491043,
+            -0.0070436573587358,
+            0.031160488724708557,
+            -0.023009516298770905,
+            0.018792519345879555,
+            -0.026801982894539833,
+            -0.004086099565029144,
+            -0.018990634009242058,
+            -0.01774534583091736,
+            0.004224071744829416,
+            0.004521242808550596,
+            7.888073741924018e-05,
+            0.0048750173300504684,
+            0.005777143407613039,
+            0.024806693196296692,
+            0.0021421066485345364,
+            0.02589631825685501,
+            -0.027877457439899445,
+            -0.025316128507256508,
+            0.01719345711171627,
+            -0.02077365852892399,
+            0.005678086541593075,
+            -0.006261814851313829,
+            -0.0302831269800663,
+            -0.01202834490686655,
+            0.023391593247652054,
+            0.02162271924316883,
+            -0.005490585695952177,
+            0.002644466934725642,
+            0.02206140011548996,
+            -0.026023678481578827,
+            0.00437973253428936,
+            -0.03781145438551903,
+            -0.018396291881799698,
+            0.029773691669106483,
+            -0.03195294365286827,
+            0.020745357498526573,
+            0.003185742534697056,
+            0.03483974561095238,
+            -0.017292514443397522,
+            -0.01947176828980446,
+            0.01654251106083393,
+            -0.01008258294314146,
+            0.004142703488469124,
+            0.01606137864291668,
+            0.01616043597459793,
+            -0.0015070808585733175,
+            0.05380208045244217,
+            0.0008167776977643371,
+            0.006909222807735205,
+            -0.049075648188591,
+            0.004496478475630283,
+            -0.044858649373054504,
+            0.0002768288250081241,
+            -0.013337312266230583,
+            -0.02514631673693657,
+            0.0014027173165231943,
+            0.001001182827167213,
+            0.001963450340554118,
+            -0.01913214474916458,
+            -0.01989629678428173,
+            0.011596739292144775,
+            -0.015863263979554176,
+            0.004928083624690771,
+            -0.06418890506029129,
+            -0.01855195313692093,
+            -0.004974074196070433,
+            0.0043337419629096985,
+            0.0007699025445617735,
+            0.03342464938759804,
+            -0.01478778850287199,
+            0.010117961093783379,
+            -0.01549533847719431,
+            -0.026363302022218704,
+            0.04010391607880592,
+            0.018750067800283432,
+            -0.023773670196533203,
+            0.024693485349416733,
+            0.012106175534427166,
+            0.03099067695438862,
+            -0.0318114347755909,
+            0.020349130034446716,
+            0.018523650243878365,
+            -0.0029823218937963247,
+            0.006551910191774368,
+            0.008575502783060074,
+            0.01832553744316101,
+            -0.015764206647872925,
+            -0.0003632825391832739,
+            0.011419852264225483,
+            0.010033054277300835,
+            0.031924642622470856,
+            0.01403071079403162,
+            -0.03183973580598831,
+            0.019047237932682037,
+            0.01616043597459793,
+            0.02298121340572834,
+            0.033820874989032745,
+            -0.027481229975819588,
+            0.038575608283281326,
+            -0.010386829264461994,
+            -0.024042539298534393,
+            0.020801961421966553,
+            -0.032688796520233154,
+            -0.006679269019514322,
+            0.030735958367586136,
+            0.04398128762841225,
+            -0.0015566092915832996,
+            -0.044632233679294586,
+            0.01535382866859436,
+            0.01316749956458807,
+            0.0027806703001260757,
+            0.028811423107981682,
+            0.011264191009104252,
+            0.021820832043886185,
+            0.025273675099015236,
+            0.011419852264225483,
+            -0.0037252490874379873,
+            0.024962352588772774,
+            -0.008101443760097027,
+            -0.004446949809789658,
+            -0.0007035697344690561,
+            0.03206615149974823,
+            0.014702882617712021,
+            -0.017603836953639984,
+            -0.036962397396564484,
+            -0.007620310410857201,
+            0.009452863596379757,
+            -0.007903330028057098,
+            -0.027127455919981003,
+            -0.03950957581400871,
+            -0.026787832379341125,
+            -0.005667473189532757,
+            0.01505665760487318,
+            -0.06724552065134048,
+            -0.007620310410857201,
+            -0.012233533896505833,
+            0.017717044800519943,
+            0.0022906921803951263,
+            0.0181132722645998,
+            0.001277127186767757,
+            -0.009403335861861706,
+            0.004209920763969421,
+            0.050802066922187805,
+            -0.03059444949030876,
+            0.014242975041270256,
+            0.011292492970824242,
+            0.0032105068676173687,
+            0.002787745790556073,
+            0.038688816130161285,
+            -0.010988246649503708,
+            -0.030424637719988823,
+            0.011575513519346714,
+            -0.017292514443397522,
+            0.003383856499567628,
+            0.05006621778011322,
+            -0.009481165558099747,
+            -0.027495380491018295,
+            0.020915169268846512,
+            -0.009148617275059223,
+            -0.015523640438914299,
+            -0.0027258351910859346,
+            -0.006035399157553911,
+            0.0006283925613388419,
+            -0.03832089155912399,
+            -0.015750056132674217,
+            0.03212275728583336,
+            0.008617955259978771,
+            0.014603826217353344,
+            0.025344429537653923,
+            0.010209942236542702,
+            -0.030481241643428802,
+            0.027481229975819588,
+            -0.0005678086308762431,
+            0.005221716593950987,
+            0.013323160819709301,
+            0.03424540534615517,
+            -0.014264201745390892,
+            -0.020264223217964172,
+            -0.040245428681373596,
+            -0.013853822834789753,
+            -0.0463869571685791,
+            0.001943992800079286,
+            0.006919836159795523,
+            -0.012919857166707516,
+            -0.04366996884346008,
+            -0.0329718142747879,
+            0.021127434447407722,
+            0.03178313374519348,
+            -0.017023645341396332,
+            0.0025294902734458447,
+            -0.008483520708978176,
+            -0.007882103323936462,
+            -0.0017520699184387922,
+            -0.013217028230428696,
+            -0.006378560326993465,
+            -0.006229975260794163,
+            0.015452885068953037,
+            -0.004652139265090227,
+            -0.0036934094969183207,
+            -0.0498398020863533,
+            -0.018778368830680847,
+            0.004082561936229467,
+            -0.008221727795898914,
+            0.007323139347136021,
+            -0.006955213379114866,
+            -0.002727604005485773,
+            -0.012474101036787033,
+            -0.03829259052872658,
+            0.024184048175811768,
+            0.017278363928198814,
+            0.0035501306410878897,
+            -0.008922201581299305,
+            -0.018367990851402283,
+            0.03294351324439049,
+            0.012523629702627659,
+            -0.017547233030200005,
+            -0.018580254167318344,
+            0.025981225073337555,
+            0.026957644149661064,
+            0.008122670464217663,
+            -0.006137993652373552,
+            0.029886899515986443,
+            0.005695775151252747,
+            -0.022938761860132217,
+            -0.003983504604548216,
+            -0.005055442452430725,
+            -0.0038844477385282516,
+            0.020561393350362778,
+            -0.010556641966104507,
+            0.0300001073628664,
+            0.038717120885849,
+            0.012870329432189465,
+            0.0039198254235088825,
+            -0.0033714743331074715,
+            -0.008922201581299305,
+            0.015976471826434135,
+            0.006481155287474394,
+            0.008525974117219448,
+            0.014759486541152,
+            -0.016358548775315285,
+            -0.020405733957886696,
+            0.006215824279934168,
+            -0.007107336539775133,
+            0.020122714340686798,
+            0.01708024926483631,
+            0.016429303213953972,
+            0.0007610581233166158,
+            -0.028287837281823158,
+            -0.0018024828750640154,
+            0.02411329373717308,
+            0.02010856196284294,
+            -0.004655676893889904,
+            -0.023264234885573387,
+            0.014398636296391487,
+            -0.015806660056114197,
+            -0.009969375096261501,
+            0.0033431723713874817,
+            0.006229975260794163,
+            -0.003083147807046771,
+            0.0571134127676487,
+            -0.020193468779325485,
+            -0.011419852264225483,
+            -0.010287772864103317,
+            -0.010867963545024395,
+            0.0012072566896677017,
+            -0.01828308403491974,
+            0.008964654989540577,
+            0.025613298639655113,
+            -0.009764186106622219,
+            0.03568880632519722,
+            0.0228114016354084,
+            0.006912760436534882,
+            0.005260631907731295,
+            -0.027070851996541023,
+            0.012304289266467094,
+            -0.016047226265072823,
+            0.005660397466272116,
+            0.018467046320438385,
+            0.017320817336440086,
+            0.0076132346875965595,
+            0.026306698098778725,
+            0.019386861473321915,
+            0.016684021800756454,
+            -0.022783100605010986,
+            0.025457637384533882,
+            -0.007832575589418411,
+            0.027877457439899445,
+            -0.01611798256635666,
+            -0.002253545681014657,
+            -0.012990612536668777,
+            0.008681634441018105,
+            0.002416282193735242,
+            -0.0019351483788341284,
+            0.00036129256477579474,
+            0.016783079132437706,
+            -0.014589674770832062,
+            0.01062032114714384,
+            0.013316085562109947,
+            -0.027919910848140717,
+            -0.010372678749263287,
+            0.016344398260116577,
+            -0.00668988237157464,
+            0.006877383217215538,
+            -0.011865608394145966,
+            -0.0017202302115038037,
+            0.011009473353624344,
+            -0.0436982698738575,
+            0.0104929618537426,
+            0.006279503460973501,
+            -0.015155714936554432,
+            0.014801939949393272,
+            -0.009799563325941563,
+            -0.008235878311097622,
+            -0.013592029921710491,
+            0.003792466362938285,
+            -0.00030225637601688504,
+            -0.0018290160223841667,
+            -0.00022995365725364536,
+            0.00389859895221889,
+            0.010047205723822117,
+            -0.0045990729704499245,
+            -0.010450509376823902,
+            -0.0028638073708862066,
+            -0.02265574224293232,
+            -0.0010047205723822117,
+            0.018735915422439575,
+            -0.004591997712850571,
+            0.0005222601466812193,
+            0.0145189194008708,
+            -0.005830209702253342,
+            0.010556641966104507,
+            -0.005221716593950987,
+            -0.008320784196257591,
+            -0.010768906213343143,
+            -0.033764272928237915,
+            0.0026710000820457935,
+            -0.015254771336913109,
+            0.008865597657859325,
+            -0.012544856406748295,
+            0.04027372971177101,
+            0.01022409275174141,
+            -0.00837738811969757,
+            -0.002796590095385909,
+            -0.0228114016354084,
+            -0.004475251771509647,
+            -0.016938740387558937,
+            -0.01184438169002533,
+            0.008342010900378227,
+            0.012629762291908264,
+            0.005614406894892454,
+            -0.018410442396998405,
+            0.003665107535198331,
+            0.0030955299735069275,
+            -0.005575491581112146,
+            -0.008370312862098217,
+            0.010853812098503113,
+            -0.0006248548161238432,
+            -0.01800006441771984,
+            -0.01957082375884056,
+            -0.027424626052379608,
+            0.0150000536814332,
+            0.014589674770832062,
+            -0.01011088490486145,
+            -0.022754797711968422,
+            -0.007471724878996611,
+            0.007107336539775133,
+            -0.02064630016684532,
+            0.0006779210525564849,
+            0.006891534198075533,
+            0.008717012591660023,
+            0.004117939155548811,
+            0.009636826813220978,
+            0.0007699025445617735,
+            0.0121698547154665,
+            0.018566103652119637,
+            0.03271709755063057,
+            -0.0396793857216835,
+            -0.009537769481539726,
+            0.0057028504088521,
+            -0.001465512323193252,
+            -0.013853822834789753,
+            0.009672204032540321,
+            0.0024375086650252342,
+            -0.0026426981203258038,
+            0.024424616247415543,
+            -0.008044839836657047,
+            -0.01505665760487318,
+            0.01184438169002533,
+            0.00887267291545868,
+            -0.019075538963079453,
+            0.02746707946062088,
+            0.006169833242893219,
+            -0.01011088490486145,
+            0.02259913645684719,
+            -0.012360893189907074,
+            0.012658064253628254,
+            -0.005522425286471844,
+            0.023660462349653244,
+            -0.019004784524440765,
+            0.0006306036957539618,
+            0.015665151178836823,
+            0.03883032873272896,
+            -0.00437973253428936,
+            -0.004567233379930258,
+            -0.03421710431575775,
+            0.005115584470331669,
+            0.028853876516222954,
+            0.02666047215461731,
+            -0.0012346742441877723,
+            -0.005596718285232782,
+            -0.0019864458590745926,
+            0.010768906213343143,
+            0.011589664034545422,
+            0.017561383545398712,
+            -0.029547275975346565,
+            0.00022011429246049374,
+            -0.016938740387558937,
+            0.025500090792775154,
+            0.014632128179073334,
+            0.006951675750315189,
+            0.04072656109929085,
+            0.0027789014857262373,
+            0.003841994795948267,
+            0.027452928945422173,
+            -0.00779719790443778,
+            0.003546592779457569,
+            -0.015566092915832996,
+            0.0003175129240844399,
+            0.022853855043649673,
+            0.0016370931407436728,
+            0.0020784272346645594,
+            0.013337312266230583,
+            0.029320860281586647,
+            -0.022471778094768524,
+            -0.023929331451654434,
+            0.003482913365587592,
+            0.007185167167335749,
+            -0.010570792481303215,
+            0.0038738346192985773,
+            0.006275965832173824,
+            -0.006007096730172634,
+            0.008900974877178669,
+            0.006332569755613804,
+            -0.0003166284877806902,
+            0.0039198254235088825,
+            -0.00308845448307693,
+            0.012962310574948788,
+            -0.02702839858829975,
+            0.036849189549684525,
+            0.004439874552190304,
+            0.010075507685542107,
+            0.015268922783434391,
+            0.001627364312298596,
+            0.004917470272630453,
+            0.005869124550372362,
+            0.010188715532422066,
+            0.018636859953403473,
+            -0.00956607237458229,
+            -0.027481229975819588,
+            -0.03166992589831352,
+            0.020363280549645424,
+            -0.01173117384314537,
+            0.0035872769076377153,
+            0.0074646491557359695,
+            -0.01828308403491974,
+            -0.007507102098315954,
+            -0.009976450353860855,
+            0.015764206647872925,
+            -0.03189634159207344,
+            0.012990612536668777,
+            -0.0043231286108493805,
+            -0.028669914230704308,
+            0.00851889792829752,
+            -0.010181640274822712,
+            -0.0045884596183896065,
+            -0.0035094465129077435,
+            -0.029264256358146667,
+            -0.007931631989777088,
+            0.03908504545688629,
+            -0.02141045406460762,
+            0.024184048175811768,
+            0.008257105015218258,
+            -0.00014449491573031992,
+            -0.0040365708991885185,
+            -0.013563727959990501,
+            0.005331386812031269,
+            -0.01778779923915863,
+            0.025457637384533882,
+            0.010294848121702671,
+            -0.013924578204751015,
+            0.0322076641023159,
+            -0.011179285123944283,
+            0.027438776567578316,
+            -0.018580254167318344,
+            0.036198243498802185,
+            0.007422196213155985,
+            0.00674648629501462,
+            0.01974063739180565,
+            -0.004025958012789488,
+            0.008228803053498268,
+            0.04559450224041939,
+            -0.018523650243878365,
+            0.02460857853293419,
+            0.020349130034446716,
+            -0.00022685811563860625,
+            0.017915157601237297,
+            0.0028390432707965374,
+            -0.023165177553892136,
+            0.009495317004621029,
+            -0.01135617308318615,
+            -0.009955224581062794,
+            0.01583496294915676,
+            -0.019316107034683228,
+            -0.018721764907240868,
+            -0.025613298639655113,
+            0.0029416377656161785,
+            -0.003350247861817479,
+            -0.0077618202194571495,
+            -0.0019616815261542797,
+            0.01454722136259079,
+            0.008724087849259377,
+            0.025669902563095093,
+            0.0008326975512318313,
+            0.011971740983426571,
+            0.01511326152831316,
+            -0.00035576484515331686,
+            0.02313687466084957,
+            -0.03229256719350815,
+            0.01246702577918768,
+            -0.004517704714089632,
+            -0.00509789539501071,
+            0.023858575150370598,
+            0.0003747802402358502,
+            -0.0004877670726273209,
+            -0.0024888061452656984,
+            0.0005496777012012899,
+            -0.014589674770832062,
+            -0.0057099261321127415,
+            0.000907432462554425,
+            -0.005175726022571325,
+            0.04299071803689003,
+            -0.00701535539701581,
+            -0.004772422835230827,
+            -0.039169952273368835,
+            -0.01918874867260456,
+            0.00623351288959384,
+            -0.01583496294915676,
+            -0.018778368830680847,
+            -0.006307805422693491,
+            0.005801907740533352,
+            -0.04893413558602333,
+            0.008342010900378227,
+            0.03336804360151291,
+            -0.012219383381307125,
+            0.00259493850171566,
+            -0.010981171391904354,
+            -0.006969364359974861,
+            -0.0029115669894963503,
+            -7.766463386360556e-05,
+            -0.002412744564935565,
+            -0.020830262452363968,
+            -0.005617944523692131,
+            0.003364398842677474,
+            -0.0008795727044343948,
+            -0.025047259405255318,
+            0.0006177793256938457,
+            0.018849123269319534,
+            -0.003979966975748539,
+            -0.0011152751976624131,
+            -0.001825478277169168,
+            0.006399787031114101,
+            -0.01508495956659317,
+            0.008009462617337704,
+            -0.0076344613917171955,
+            0.003841994795948267,
+            -0.0030495391692966223,
+            -0.002214630600064993,
+            0.02844349853694439,
+            0.021594416350126266,
+            -0.04808507859706879,
+            -0.01405901275575161,
+            0.023009516298770905,
+            0.007280686404556036,
+            -0.004740582779049873,
+            0.01178777776658535,
+            -0.031613320112228394,
+            -0.00028943203506059945,
+            -0.026787832379341125,
+            0.02200479619204998,
+            -0.009205221198499203,
+            -0.007386818993836641,
+            0.016967041417956352,
+            -0.026632171124219894,
+            0.016344398260116577,
+            0.00794578343629837,
+            -0.00030181417241692543,
+            -0.000493515923153609,
+            0.0063856360502541065,
+            0.0008972614887170494,
+            -0.01659911684691906,
+            0.006442239973694086,
+            -0.0048502529971301556,
+            0.007959933951497078,
+            -0.005087282508611679,
+            -0.05527378246188164,
+            -0.009933997876942158,
+            -0.005497660953551531,
+            -0.008936353027820587,
+            -0.012820800766348839,
+            0.008136821910738945,
+            0.02827368676662445,
+            0.01606137864291668,
+            -0.009863242506980896,
+            -4.8312376748071983e-05,
+            0.033000119030475616,
+            -0.0008300442132167518,
+            0.009955224581062794,
+            -0.011349096894264221,
+            0.00015709814033471048,
+            -0.0006911876262165606,
+            -0.014660430140793324,
+            -0.019061388447880745,
+            0.024042539298534393,
+            -0.028344441205263138,
+            0.02378782071173191,
+            -0.01843874529004097,
+            -0.0130825936794281,
+            0.01757553406059742,
+            -0.032575588673353195,
+            -0.005108508747071028,
+            0.007450498174875975,
+            0.0090566361322999,
+            0.021608568727970123,
+            0.0032794929575175047,
+            0.020589696243405342,
+            -0.03716050833463669,
+            -0.009134466759860516,
+            -0.010160413570702076,
+            -0.00794578343629837,
+            -0.010231168940663338,
+            -0.011596739292144775,
+            0.021863285452127457,
+            -0.015240620821714401,
+            -0.016952890902757645,
+            0.007054270710796118,
+            0.0030212372075766325,
+            -0.011313719674944878,
+            0.015750056132674217,
+            0.00017810352437663823,
+            0.004712280817329884,
+            0.013011839240789413,
+            -0.0008923970744945109,
+            0.04086806997656822,
+            0.0253019779920578,
+            -0.0030212372075766325,
+            -0.008568426594138145,
+            0.021438755095005035,
+            0.0004285097820684314,
+            0.0150000536814332,
+            0.009375033900141716,
+            0.023264234885573387,
+            -0.02092931978404522,
+            0.009997677057981491,
+            -0.005501199048012495,
+            0.014603826217353344,
+            -0.009495317004621029,
+            -0.015339677222073078,
+            0.00479718716815114,
+            -0.01577835902571678,
+            0.010153338313102722,
+            -0.010860888287425041,
+            0.008221727795898914,
+            0.01913214474916458,
+            0.002467579673975706,
+            -0.00941041111946106,
+            0.01146230474114418,
+            0.02075950801372528,
+            -0.025641601532697678,
+            0.008851447142660618,
+            -0.0031998935155570507,
+            -0.0021916350815445185,
+            0.021127434447407722,
+            0.010301923379302025,
+            -0.01594817079603672,
+            0.01573590561747551,
+            -0.008879749104380608,
+            -0.0015053119277581573,
+            -0.01882082223892212,
+            -0.03291521221399307,
+            -0.0015283073298633099,
+            0.012396270409226418,
+            0.004800724796950817,
+            0.0150000536814332,
+            0.011264191009104252,
+            -0.0008950503543019295,
+            -0.006601438857614994,
+            0.0008433107868768275,
+            -0.013485897332429886,
+            -0.004963461309671402,
+            0.010231168940663338,
+            0.017051948234438896,
+            -0.0037217114586383104,
+            0.015650998800992966,
+            0.0088160689920187,
+            0.006137993652373552,
+            -0.0010480580385774374,
+            0.0036138100549578667,
+            -0.002060738392174244,
+            -0.007783046457916498,
+            0.019938750192523003,
+            0.019075538963079453,
+            -0.018509499728679657,
+            -0.0024870370980352163,
+            -0.02141045406460762,
+            0.025882167741656303,
+            0.022188758477568626,
+            0.011165134608745575,
+            -0.04222656786441803,
+            0.00012050455552525818,
+            0.00752832880243659,
+            -0.0013744152383878827,
+            -0.031103884801268578,
+            0.008172199130058289,
+            0.005221716593950987,
+            0.004330204334110022,
+            0.020844414830207825,
+            -0.018424594774842262,
+            0.010853812098503113,
+            0.0070861103013157845,
+            -0.0005766529939137399,
+            0.01019579079002142,
+            -0.0073372903279960155,
+            0.008688710629940033,
+            0.038717120885849,
+            -0.018410442396998405,
+            0.005303084850311279,
+            0.0062370505183935165,
+            0.023320838809013367,
+            -0.011228813789784908,
+            -0.00031485961517319083,
+            0.00800238735973835,
+            0.021608568727970123,
+            -0.012934008613228798,
+            0.005883275531232357,
+            0.067075714468956,
+            -0.012912781909108162,
+            0.031613320112228394,
+            0.020533092319965363,
+            -0.004970536567270756,
+            -0.0104929618537426,
+            -0.012445799075067043,
+            -0.0181132722645998,
+            0.021283095702528954,
+            0.01175947580486536,
+            0.003419233951717615,
+            0.01941516436636448,
+            -0.01610383205115795,
+            0.006636816076934338,
+            0.017547233030200005,
+            0.012601460330188274,
+            -0.007379743270576,
+            -0.0036173479165881872,
+            0.019004784524440765,
+            0.018509499728679657,
+            -0.0030849166214466095,
+            0.00019424450874794275,
+            0.01508495956659317,
+            -0.022075550630688667,
+            -0.012530704960227013,
+            0.01644345559179783,
+            -0.0037358624394983053,
+            0.017773648723959923,
+            -0.02227366529405117,
+            0.024198200553655624,
+            -0.004531855694949627,
+            -0.017179306596517563,
+            0.004698129836469889,
+            -0.007514177821576595,
+            0.022033097222447395,
+            -0.001578720286488533,
+            0.004857328720390797,
+            0.000821199850179255,
+            -0.0018015984678640962,
+            0.01349297259002924,
+            0.002826661104336381,
+            -0.0005120891146361828,
+            0.010507113300263882,
+            -0.0009702275274321437,
+            -0.0013496510218828917,
+            0.024509521201252937,
+            1.9374700059415773e-05,
+            0.005989408120512962,
+            0.0032936439383774996,
+            -0.009261825121939182,
+            -0.004429261200129986,
+            -0.004443412180989981,
+            0.0011842612875625491,
+            0.003877372248098254,
+            -0.007029506377875805,
+            -0.02384442463517189,
+            0.022471778094768524,
+            -0.008540124632418156,
+            0.01175947580486536,
+            0.011349096894264221,
+            0.003665107535198331,
+            0.006824316922575235,
+            -0.009771261364221573,
+            -0.017320817336440086,
+            0.0006633278680965304,
+            -0.01617458648979664,
+            0.006912760436534882,
+            -0.001656550681218505,
+            0.0020235921256244183,
+            0.009445788338780403,
+            -0.002570174168795347,
+            0.0015123875346034765,
+            0.011306644417345524,
+            0.011497682891786098,
+            0.003254728624597192,
+            0.004577846731990576,
+            -0.018721764907240868,
+            -0.020915169268846512,
+            0.005288933869451284,
+            0.013924578204751015,
+            0.004273600410670042,
+            0.0015309606678783894,
+            0.017603836953639984,
+            -0.006166295614093542,
+            -0.022967062890529633,
+            -0.010117961093783379,
+            -0.01584911346435547,
+            0.039396367967128754,
+            0.01800006441771984,
+            -0.02525952458381653,
+            -0.000530220044311136,
+            -0.009728807955980301,
+            0.010450509376823902,
+            -0.00404364662244916,
+            0.014575524255633354,
+            0.005476434715092182,
+            -0.02426895499229431,
+            -0.016797229647636414,
+            -0.010662773624062538,
+            -0.005257094278931618,
+            0.020080260932445526,
+            0.027014248073101044,
+            -0.004956385586410761,
+            0.0061026159673929214,
+            0.019584976136684418,
+            -0.010938717983663082,
+            0.009849091991782188,
+            0.020957622677087784,
+            0.0031698227394372225,
+            -0.007570781745016575,
+            -0.004553082399070263,
+            -0.016514210030436516,
+            0.024467067793011665,
+            0.014250051230192184,
+            -0.005104971118271351,
+            -0.02465103194117546,
+            0.004110863897949457,
+            -0.02428310550749302,
+            -0.01616043597459793,
+            -0.007301913108676672,
+            -0.014143918640911579,
+            -0.004931621253490448,
+            0.0027594438288360834,
+            -0.004836102016270161,
+            0.011278342455625534,
+            -0.0031592093873769045,
+            -0.004531855694949627,
+            -0.0041851564310491085,
+            -0.025584997609257698,
+            -0.02524537406861782,
+            0.007691065315157175,
+            -0.003005317412316799,
+            0.005596718285232782,
+            0.000650945701636374,
+            -0.013400991447269917,
+            -0.011589664034545422,
+            0.00970758218318224,
+            -0.007988235913217068,
+            0.012813725508749485,
+            0.0031149876303970814,
+            -0.009183995425701141,
+            0.0017193458043038845,
+            -0.0012311364989727736,
+            -0.0059752571396529675,
+            0.019924599677324295,
+            0.00916984397917986,
+            0.011724098585546017,
+            -0.03118879161775112,
+            -0.0021633331198245287,
+            0.008349086157977581,
+            0.01828308403491974,
+            -0.02244347706437111,
+            -0.008313708938658237,
+            0.005179263651371002,
+            -0.011228813789784908,
+            -0.013507124036550522,
+            -0.028952933847904205,
+            -0.0030460015404969454,
+            8.175515540642664e-05,
+            -0.018636859953403473,
+            -0.029886899515986443,
+            0.0009038947173394263,
+            -0.007860877551138401,
+            0.016882136464118958,
+            -0.006643891800194979,
+            0.01790100708603859,
+            0.00870286114513874,
+            0.0036173479165881872,
+            0.011554286815226078,
+            0.0025082635693252087,
+            -0.013266556896269321,
+            0.008426916785538197,
+            -0.02671707607805729,
+            -0.01516986545175314,
+            0.009792488068342209,
+            0.014214673079550266,
+            -0.0026126273442059755,
+            5.074458022136241e-05,
+            0.0058372849598526955,
+            0.024254804477095604,
+            0.013584954664111137,
+            0.012474101036787033,
+            0.003042463678866625,
+            -0.005303084850311279,
+            0.005568416323512793,
+            0.0227123461663723,
+            0.008603804744780064,
+            0.004011806566268206,
+            0.007592008449137211,
+            0.00389859895221889,
+            0.0009932229295372963,
+            -0.005943417549133301,
+            -0.00843399204313755,
+            -0.021537812426686287,
+            -0.008908051066100597,
+            0.019938750192523003,
+            -0.010422207415103912,
+            -0.03005671128630638,
+            0.0006023016758263111,
+            -0.0032777239102870226,
+            0.007988235913217068,
+            0.02595292218029499,
+            -0.014172220602631569,
+            0.004970536567270756,
+            0.005536576267331839,
+            -0.007910405285656452,
+            -0.00614860700443387,
+            0.022967062890529633,
+            0.02639160305261612,
+            -0.0034404604230076075,
+            -0.04800017178058624,
+            -0.01671232469379902,
+            0.004305440001189709,
+            -0.007768895477056503,
+            -0.034047290682792664,
+            -0.015424583107233047,
+            0.002757675014436245,
+            -0.03206615149974823,
+            0.010068432427942753,
+            -0.012948159128427505,
+            -0.01214155275374651,
+            0.020264223217964172,
+            0.03223596513271332,
+            0.005154499784111977,
+            0.00014847487909719348,
+            -0.0214670579880476,
+            -0.0021686397958546877,
+            -0.021382151171565056,
+            0.010174565017223358,
+            -0.008801918476819992,
+            -0.010584943927824497,
+            0.0059611061587929726,
+            -0.012191081419587135,
+            0.013266556896269321,
+            0.010797208175063133,
+            0.024778390303254128,
+            0.01584911346435547,
+            -0.01876421831548214,
+            -0.0024746551644057035,
+            -0.01108730398118496,
+            -0.0016450530383735895,
+            -0.009820790030062199,
+            0.006360871717333794,
+            -0.0031592093873769045,
+            0.011264191009104252,
+            0.015184016898274422,
+            0.004432798828929663,
+            0.00046786724124103785,
+            0.038151077926158905,
+            -2.4487851987942122e-05,
+            -0.0003418349369894713,
+            0.006803090218454599,
+            -0.015750056132674217,
+            0.008837295696139336,
+            0.012191081419587135,
+            0.005051904823631048,
+            -0.007698140572756529,
+            0.006739411037415266,
+            0.03099067695438862,
+            -0.01516986545175314,
+            -0.009014182724058628,
+            -0.023504801094532013,
+            0.007549555506557226,
+            -0.0004656561359297484,
+            0.029971804469823837,
+            0.020533092319965363,
+            -0.027509532868862152,
+            -0.016018925234675407,
+            -0.02179253101348877,
+            0.01267929095774889,
+            -0.011497682891786098,
+            -0.003704022616147995,
+            0.012240610085427761,
+            0.02367461286485195,
+            0.0018838511314243078,
+            -0.014455240219831467,
+            -0.008950503543019295,
+            0.008943428285419941,
+            0.014617976732552052,
+            0.003497064346447587,
+            0.007570781745016575,
+            0.00743634719401598,
+            0.008886824361979961,
+            -0.014801939949393272,
+            -0.012254760600626469,
+            0.00767691433429718,
+            -0.010202866978943348,
+            0.01794346049427986,
+            -0.009403335861861706,
+            0.016768928617239,
+            -0.004645063541829586,
+            -0.011964665725827217,
+            -0.0320095494389534,
+            -0.002292460994794965,
+            0.005037753842771053,
+            0.004627374932169914,
+            0.0034705311991274357,
+            -0.0022217060904949903,
+            -0.013160424306988716,
+            0.004673365969210863,
+            0.005073131527751684,
+            1.875283123808913e-05,
+            -0.008313708938658237,
+            -0.00479718716815114,
+            0.004252373706549406,
+            -0.001031253719702363,
+            -0.0073372903279960155,
+            -0.0059858704917132854,
+            0.018481198698282242,
+            0.0066509670577943325,
+            0.00800238735973835,
+            0.03234917297959328,
+            0.02974538877606392,
+            -0.0016353242099285126,
+            -0.004606148693710566,
+            -0.00023304918431676924,
+            -0.01589156687259674,
+            -0.0057205394841730595,
+            -0.023320838809013367,
+            0.022259512916207314,
+            -0.009665128774940968,
+            0.021820832043886185,
+            0.019457615911960602,
+            0.0039304387755692005,
+            -0.008391539566218853,
+            -0.0011586126638576388,
+            -0.007308988366276026,
+            -0.003560743760317564,
+            0.016641568392515182,
+            0.006742948666214943,
+            0.01226891204714775,
+            0.00030933189555071294,
+            -0.028401045128703117,
+            0.026405755430459976,
+            0.000579748535528779,
+            0.0034015451092272997,
+            0.008908051066100597,
+            0.014129767194390297,
+            0.013209952972829342,
+            -0.01235381793230772,
+            0.01768874190747738,
+            -0.03280200436711311,
+            -0.0033484790474176407,
+            -0.016500059515237808,
+            0.002076658420264721,
+            -0.0018520113080739975,
+            -0.0008145665633492172,
+            -0.05354736000299454,
+            0.0003842879377771169,
+            0.005313698202371597,
+            -0.015339677222073078,
+            0.029349161311984062,
+            -0.020589696243405342,
+            -0.017165156081318855,
+            -0.032858606427907944,
+            -0.010238244198262691,
+            0.018099121749401093,
+            0.005617944523692131,
+            0.00035178486723452806,
+            -0.02660386823117733,
+            0.015750056132674217,
+            -0.0047476585023105145,
+            -0.00743634719401598,
+            -0.029858596622943878,
+            -0.010938717983663082,
+            0.009403335861861706,
+            0.004485865123569965,
+            -0.0130825936794281,
+            0.009898620657622814,
+            -0.002890340518206358,
+            0.010867963545024395,
+            -0.0138467475771904,
+            -0.00420284504070878,
+            0.0010020672343671322,
+            -0.005363226868212223,
+            0.012615610845386982,
+            -0.030537845566868782,
+            0.011391550302505493,
+            -0.01352127455174923,
+            0.0008548084879294038,
+            -0.017858553677797318,
+            -0.027707645669579506,
+            0.0020288988016545773,
+            -0.016132133081555367,
+            0.039990708231925964,
+            0.010188715532422066,
+            -0.01481609046459198,
+            0.016995344310998917,
+            -0.002080196049064398,
+            0.017377421259880066,
+            0.0014955832157284021,
+            0.017023645341396332,
+            -0.011971740983426571,
+            0.021056678146123886,
+            -0.0041745430789887905,
+            0.02156611531972885,
+            0.00600002147257328,
+            0.034047290682792664,
+            -0.02033497765660286,
+            0.0070825726725161076,
+            0.003792466362938285,
+            0.005798369646072388,
+            -0.0026816134341061115,
+            0.021424604579806328,
+            0.017985913902521133,
+            0.019698183983564377,
+            0.0037110981065779924,
+            -0.024084990844130516,
+            -0.006137993652373552,
+            0.004867942072451115,
+            0.023858575150370598,
+            0.011150983162224293,
+            -0.016415152698755264,
+            -0.021056678146123886,
+            0.015226469375193119,
+            -0.007860877551138401,
+            0.006771250627934933,
+            -0.00851889792829752,
+            0.00037544354563578963,
+            -0.013259481638669968,
+            0.012233533896505833,
+            -0.012523629702627659,
+            0.0032335021533071995,
+            0.01043635793030262,
+            -0.027806703001260757,
+            0.01511326152831316,
+            -0.00371817359700799,
+            -0.014313730411231518,
+            0.023320838809013367,
+            -0.015877414494752884,
+            -0.013988257385790348,
+            0.005501199048012495,
+            0.008610880002379417,
+            0.01746232621371746,
+            -0.002761212643235922,
+            0.002669231267645955,
+            -0.020504789426922798,
+            -0.012445799075067043,
+            0.004591997712850571,
+            0.02210385352373123,
+            -0.015820810571312904,
+            0.012431648559868336,
+            -0.018099121749401093,
+            0.01364863384515047,
+            0.016202887520194054,
+            0.007149789482355118,
+            0.00573115237057209,
+            -0.007726442534476519,
+            -0.0011471149045974016,
+            0.032151058316230774,
+            -0.0029876285698264837,
+            0.02704254910349846,
+            -0.014915147796273232,
+            0.009615600109100342,
+            0.00518987700343132,
+            -0.013089669868350029,
+            -0.014589674770832062,
+            0.011108530685305595,
+            0.02428310550749302,
+            -0.009530694223940372,
+            -0.023660462349653244,
+            0.0018484735628589988,
+            0.014264201745390892,
+            0.010337301529943943,
+            0.009403335861861706,
+            -0.001997058978304267,
+            -0.0037358624394983053,
+            0.01484439242631197,
+            -0.0032918748911470175,
+            -0.0286557637155056,
+            -0.01633024774491787,
+            -0.004400959238409996,
+            -0.017490629106760025,
+            0.020745357498526573,
+            0.02367461286485195,
+            0.010507113300263882,
+            0.0037570889107882977,
+            0.005540114361792803,
+            0.018297234550118446,
+            0.01843874529004097,
+            0.013330236077308655,
+            0.009304278530180454,
+            0.012700516730546951,
+            -0.004319590982049704,
+            0.02800481766462326,
+            -0.010860888287425041,
+            -0.01046465989202261,
+            -0.015707602724432945,
+            0.0016636261716485023,
+            -0.005193414632230997,
+            -0.012431648559868336,
+            -0.0069481381215155125,
+            0.004369119182229042,
+            0.013330236077308655,
+            -0.012240610085427761,
+            -0.007149789482355118,
+            0.01016748882830143,
+            0.011971740983426571,
+            0.01403071079403162,
+            0.01719345711171627,
+            -0.01633024774491787,
+            -0.011278342455625534,
+            0.0009834941010922194,
+            -0.02844349853694439,
+            -0.0160047747194767,
+            -0.01767459139227867,
+            0.009580222889780998,
+            -0.014469391666352749,
+            0.03724541515111923,
+            0.028471799567341805,
+            0.0073514413088560104,
+            0.031613320112228394,
+            -0.028401045128703117,
+            0.00970758218318224,
+            -0.014398636296391487,
+            0.007132100872695446,
+            -0.004188694059848785,
+            0.003902136581018567,
+            0.002448122017085552,
+            0.020419884473085403,
+            0.00791748147457838,
+            0.014264201745390892,
+            0.008150972425937653,
+            -0.021934039890766144,
+            -0.00662620272487402,
+            -0.012559006921947002,
+            -0.016401002183556557,
+            -0.02595292218029499,
+            0.013719388283789158,
+            0.005087282508611679,
+            -0.003074303502216935,
+            -0.0002704166399780661,
+            0.006566061172634363,
+            -0.0013195802457630634,
+            0.010443433187901974,
+            -0.027608588337898254,
+            -0.01311089564114809,
+            0.0069269114173948765,
+            0.01449061743915081,
+            -0.02114158496260643,
+            0.0014637433923780918,
+            0.008900974877178669,
+            -0.004602610599249601,
+            0.035660505294799805,
+            -0.014674580655992031,
+            -0.008610880002379417,
+            0.0027806703001260757,
+            0.005062518175691366,
+            -0.017660440877079964,
+            -0.02319347858428955,
+            0.0070648835971951485,
+            0.011575513519346714,
+            0.010867963545024395,
+            -0.028570856899023056,
+            0.02087271586060524,
+            -0.0054339817725121975,
+            0.03832089155912399,
+            0.00401534466072917,
+            -0.013337312266230583,
+            0.003962278366088867,
+            0.0011860302183777094,
+            -0.012360893189907074,
+            0.002060738392174244,
+            -0.000340508297085762,
+            0.003700484987348318,
+            0.011228813789784908,
+            -0.014801939949393272,
+            -0.016967041417956352,
+            0.016910437494516373,
+            -0.011476456187665462,
+            -0.0035395172890275717,
+            -0.00943871308118105,
+            -0.016669871285557747,
+            0.007308988366276026,
+            0.025047259405255318,
+            0.0041745430789887905,
+            0.001242634141817689,
+            -0.011632117442786694,
+            -0.0038596836384385824,
+            -0.008575502783060074,
+            0.0011718792375177145,
+            -0.0207029040902853,
+            0.027396325021982193,
+            -0.01801421493291855,
+            0.002410975517705083,
+            0.021099131554365158,
+            0.008709936402738094,
+            -0.003944589756429195,
+            0.016316095367074013,
+            0.026363302022218704,
+            0.01205664686858654,
+            0.0031539027113467455,
+            -0.00902833417057991,
+            1.648646139074117e-05,
+            0.014957600273191929,
+            0.0015123875346034765,
+            -0.0015663381200283766,
+            0.0032140444964170456,
+            -0.006583749782294035,
+            -0.001859086798503995,
+            -0.017561383545398712,
+            0.005564878229051828,
+            0.02125479280948639,
+            -0.004132090136408806,
+            0.020476488396525383,
+            0.015509488992393017,
+            0.011625041253864765,
+            0.010726453736424446,
+            -0.0012390963966026902,
+            -0.002150951186195016,
+            -0.0021668709814548492,
+            -0.007393894251435995,
+            -0.018297234550118446,
+            0.006470541935414076,
+            0.0034493047278374434,
+            0.01822648011147976,
+            0.020504789426922798,
+            -0.02308027073740959,
+            0.010146263055503368,
+            0.014985902234911919,
+            -0.013011839240789413,
+            0.0235755555331707,
+            -0.02006610855460167,
+            0.0024605041835457087,
+            0.02156611531972885,
+            0.013337312266230583,
+            -0.014271277002990246,
+            -0.009219372645020485,
+            0.004662752617150545,
+            -0.01543873455375433,
+            0.000872939417604357,
+            -0.01478778850287199,
+            -0.010783057659864426,
+            0.014533070847392082,
+            -0.01490099634975195,
+            0.01229721400886774,
+            -0.006969364359974861,
+            0.0070967236533761024,
+            -0.017165156081318855,
+            0.015014204196631908,
+            0.008632106706500053,
+            0.01757553406059742,
+            0.0021279556676745415,
+            -0.02518877014517784,
+            -0.006290116813033819,
+            0.021169887855648994,
+            0.009509467519819736,
+            -0.02497650496661663,
+            0.019613277167081833,
+            0.0020094411447644234,
+            -0.019528372213244438,
+            -0.022401023656129837,
+            -0.012686366215348244,
+            -0.005055442452430725,
+            -0.017264213413000107,
+            -0.0011276573641225696,
+            -0.024792540818452835,
+            0.003972891718149185,
+            0.009106164798140526,
+            -0.011978816241025925,
+            -0.019273653626441956,
+            -0.0002315014135092497,
+            -0.01040805596858263,
+            0.006219361908733845,
+            -0.01132787112146616,
+            -0.019853845238685608,
+            0.0070436573587358,
+            0.012127402238547802,
+            -0.0027647505048662424,
+            -0.005646246485412121,
+            -0.02562745101749897,
+            -0.006919836159795523,
+            0.020179318264126778,
+            -0.003994117956608534,
+            -0.004974074196070433,
+            -0.01703779585659504,
+            0.01108730398118496,
+            0.011165134608745575,
+            0.020901018753647804,
+            0.010931642726063728,
+            0.005674548447132111,
+            0.013634482398629189,
+            -0.0029416377656161785,
+            -0.007156865205615759,
+            -9.645892532716971e-06,
+            0.004284213297069073,
+            0.0069233737885952,
+            0.0016636261716485023,
+            0.020235922187566757,
+            -0.01692458800971508,
+            0.018934030085802078,
+            -0.012290137819945812,
+            0.008794843219220638,
+            -0.01855195313692093,
+            -0.0003155229496769607,
+            -0.01067692507058382,
+            0.00015002264990471303,
+            -0.0038455326575785875,
+            -0.009544845670461655,
+            1.9595809135353193e-05,
+            0.00143986358307302,
+            0.008511822670698166,
+            -0.016783079132437706,
+            0.009615600109100342,
+            -0.010839661583304405,
+            -0.004308977629989386,
+            -0.019542522728443146,
+            0.017448175698518753,
+            0.013344387523829937,
+            0.007308988366276026,
+            -0.008278331719338894,
+            -0.006300730165094137,
+            -0.02282555401325226,
+            0.008582578040659428,
+            -0.0014637433923780918,
+            0.002214630600064993,
+            -0.028301987797021866,
+            -0.004358506295830011,
+            0.0024587351363152266,
+            0.020306676626205444,
+            0.0033661676570773125,
+            -0.010712302289903164,
+            0.008978805504739285,
+            -0.023377442732453346,
+            -0.0030159305315464735,
+            -0.023349139839410782,
+            -0.004075486212968826,
+            -0.005112046375870705,
+            0.010103809647262096,
+            0.008101443760097027,
+            0.01549533847719431,
+            -0.009898620657622814,
+            0.01478778850287199,
+            -0.004793649073690176,
+            0.012049571610987186,
+            -0.0012152165872976184,
+            0.01337976474314928,
+            -0.0003111007681582123,
+            -0.01505665760487318,
+            0.02158026583492756,
+            -0.008738238364458084,
+            0.001741456682793796,
+            -0.0052853962406516075,
+            0.0027346794959157705,
+            -0.017702894285321236,
+            -0.004553082399070263,
+            -0.014455240219831467,
+            0.01957082375884056,
+            -0.022584985941648483,
+            0.004153316840529442,
+            -0.005055442452430725,
+            -0.016075529158115387,
+            0.029434068128466606,
+            0.022896308451890945,
+            -0.0035483618266880512,
+            0.015070809051394463,
+            0.0017989451298490167,
+            0.0015061964513733983,
+            0.016202887520194054,
+            -0.015806660056114197,
+            -0.013025989755988121,
+            0.035066161304712296,
+            0.0176321379840374,
+            0.0004313841927796602,
+            -0.0023260696325451136,
+            -0.0037323245778679848,
+            0.0032193511724472046,
+            0.008023613132536411,
+            -0.010450509376823902,
+            -0.01141277700662613,
+            -0.017051948234438896,
+            0.009997677057981491,
+            -0.006155682262033224,
+            0.004446949809789658,
+            -0.00025891896802932024,
+            -7.617214578203857e-05,
+            -0.01516986545175314,
+            0.022584985941648483,
+            -0.017717044800519943,
+            -0.00602832343429327,
+            0.004687516950070858,
+            0.0207029040902853,
+            -0.02541518583893776,
+            0.01616043597459793,
+            -0.007107336539775133,
+            -0.013429293408989906,
+            0.012573158368468285,
+            -0.015127412974834442,
+            -0.0107335289940238,
+            -0.003175129182636738,
+            -0.008009462617337704,
+            -0.022754797711968422,
+            0.017929309979081154,
+            -0.013627407141029835,
+            0.015650998800992966,
+            -0.00287972716614604,
+            0.000562502013053745,
+            0.002057200763374567,
+            0.0032582662533968687,
+            0.02250008098781109,
+            0.030735958367586136,
+            0.010415131226181984,
+            -0.005465821363031864,
+            -0.021000074222683907,
+            0.012983537279069424,
+            0.0020589695777744055,
+            0.013202877715229988,
+            -0.002253545681014657,
+            -0.002135031158104539,
+            -0.016613267362117767,
+            -0.023122724145650864,
+            -0.0013010069960728288,
+            -0.01761798746883869,
+            -0.02552839368581772,
+            -0.01774534583091736,
+            -0.007047194987535477,
+            0.011349096894264221,
+            0.020405733957886696,
+            0.0009755341452546418,
+            -0.005384453106671572,
+            0.01559439580887556,
+            -0.006941062398254871,
+            0.002080196049064398,
+            -0.006594363134354353,
+            -0.002032436430454254,
+            -0.008200501091778278,
+            -0.021863285452127457,
+            0.029660483822226524,
+            -0.010103809647262096,
+            0.01141277700662613,
+            -0.005472897086292505,
+            -0.00965097825974226,
+            0.005575491581112146,
+            0.00900003220885992,
+            0.024849144741892815,
+            0.006452853325754404,
+            0.03141520544886589,
+            0.020151015371084213,
+            -0.018891576677560806,
+            -0.0006071660900488496,
+            0.022684043273329735,
+            -0.0037570889107882977,
+            -0.004889168310910463,
+            -0.0031786670442670584,
+            -0.017985913902521133,
+            -0.012007118202745914,
+            0.007839650847017765,
+            -0.013471745885908604,
+            -0.006636816076934338,
+            -0.00837738811969757,
+            0.018070818856358528,
+            -0.017773648723959923,
+            0.025698205456137657,
+            -0.006488230545073748,
+            0.005154499784111977,
+            0.031924642622470856,
+            0.014363259077072144,
+            0.019698183983564377,
+            -0.011794853024184704,
+            -0.031075581908226013,
+            -0.00683846790343523,
+            -0.007882103323936462,
+            0.009558996185660362,
+            -0.01062032114714384,
+            0.005204027984291315,
+            -0.02006610855460167,
+            -0.018948180601000786,
+            0.012863253243267536,
+            -0.017476476728916168,
+            0.013924578204751015,
+            -0.009955224581062794,
+            0.027226513251662254,
+            -0.001228483160957694,
+            0.0121698547154665,
+            -0.002614396158605814,
+            -0.017801949754357338,
+            0.01205664686858654,
+            0.004089637193828821,
+            -0.023943481966853142,
+            0.010337301529943943,
+            -0.001864393474534154,
+            -0.014037786051630974,
+            -0.018410442396998405,
+            -0.001228483160957694,
+            -0.005200490355491638,
+            -0.009658053517341614,
+            0.025457637384533882,
+            0.014101465232670307,
+            0.016231190413236618,
+            0.0035359796602278948,
+            0.0016335553955286741,
+            -0.011073152534663677,
+            0.009573147632181644,
+            -0.006803090218454599,
+            0.000612030504271388,
+            -0.0034404604230076075,
+            0.011893910355865955,
+            -0.004252373706549406,
+            -0.01543873455375433,
+            0.002324300818145275,
+            0.008561351336538792,
+            0.0143278818577528,
+            5.947839599684812e-05,
+            0.009587298147380352,
+            0.0009790719486773014,
+            -0.009764186106622219,
+            -0.00046388726332224905,
+            -0.0060884649865329266,
+            0.0069375247694551945,
+            0.02400008589029312,
+            -0.012516554445028305,
+            -0.01681138016283512,
+            0.014504768885672092,
+            0.004033033270388842,
+            -0.017589686438441277,
+            0.018367990851402283,
+            0.004103788174688816,
+            -0.022358570247888565,
+            0.0035412863362580538,
+            0.010662773624062538,
+            -0.009523618966341019,
+            -0.014469391666352749,
+            -0.0010339069413021207,
+            -0.015509488992393017,
+            0.03130199760198593,
+            0.00025073791039176285,
+            0.016910437494516373,
+            -0.01062032114714384,
+            -0.0025967073161154985,
+            -0.013309009373188019,
+            0.013103820383548737,
+            -0.015750056132674217,
+            -0.007273610681295395,
+            0.00035620704875327647,
+            -0.02298121340572834,
+            0.01741987280547619,
+            0.0013425755314528942,
+            0.00794578343629837,
+            0.003436922561377287,
+            -0.010556641966104507,
+            0.009820790030062199,
+            0.0018237093463540077,
+            0.006668656133115292,
+            -0.014915147796273232,
+            0.023618008941411972,
+            0.006413938011974096,
+            0.011235889047384262,
+            -0.01352127455174923,
+            -0.01546703651547432,
+            0.002253545681014657,
+            0.04134920611977577,
+            -0.01306136790663004,
+            0.0002593612007331103,
+            0.01562269777059555,
+            7.081024523358792e-05,
+            0.004156854469329119,
+            -0.0023755980655550957,
+            -0.0034493047278374434,
+            -0.034585028886795044,
+            0.005370302125811577,
+            -0.0006328147719614208,
+            0.008073141798377037,
+            -0.008992956951260567,
+            -0.005430444143712521,
+            -0.0040684109553694725,
+            -0.008625030517578125,
+            -0.00314859626814723,
+            -0.01478778850287199,
+            0.008363237604498863,
+            -0.014533070847392082,
+            -0.028910480439662933,
+            0.03166992589831352,
+            -0.006887996103614569,
+            -0.00533846253529191,
+            -0.021396303549408913,
+            0.007570781745016575,
+            -0.01370523776859045,
+            0.002663924591615796,
+            0.010967019945383072,
+            0.008865597657859325,
+            0.008738238364458084,
+            0.002762981690466404,
+            0.0009516543359495699,
+            -0.004255911335349083,
+            0.010273621417582035,
+            -0.00835616234689951,
+            -0.0005943417199887335,
+            -0.0014186371117830276,
+            -0.02725481428205967,
+            -0.01405901275575161,
+            -0.010592019185423851,
+            -0.007811348885297775,
+            -0.00435143057256937,
+            0.012608535587787628,
+            0.007174553815275431,
+            -0.0017485321732237935,
+            -0.022938761860132217,
+            -0.014745336025953293,
+            -0.0034103896468877792,
+            0.008349086157977581,
+            -0.01838214136660099,
+            0.01064862310886383,
+            0.0016830838285386562,
+            0.009976450353860855,
+            0.014292503707110882,
+            0.00302831269800663,
+            0.0016742395237088203,
+            0.003458149265497923,
+            0.014886845834553242,
+            0.012601460330188274,
+            -0.005801907740533352,
+            -0.021382151171565056,
+            -0.0024605041835457087,
+            -0.007068421691656113,
+            -0.0034687623847275972,
+            -0.012332591228187084,
+            0.03039633482694626,
+            0.0029805530793964863,
+            0.0176321379840374,
+            0.007832575589418411,
+            0.009573147632181644,
+            0.010507113300263882,
+            0.029377464205026627,
+            0.006049550138413906,
+            -0.0029929352458566427,
+            -0.008603804744780064,
+            -0.02141045406460762,
+            -0.013146273791790009,
+            0.009898620657622814,
+            0.013294858857989311,
+            0.03597182780504227,
+            -0.034896351397037506,
+            0.012162779457867146,
+            -0.011101454496383667,
+            -0.008264180272817612,
+            -0.008533049374818802,
+            0.0044717141427099705,
+            0.010068432427942753,
+            0.014370334334671497,
+            -0.0023402206134051085,
+            -0.01175947580486536,
+            0.009070787578821182,
+            0.023547254502773285,
+            -0.012983537279069424,
+            0.03016991913318634,
+            0.027325568720698357,
+            0.023066120222210884,
+            0.01761798746883869,
+            -0.012537780217826366,
+            0.026462359353899956,
+            -6.484030018327758e-05,
+            0.0006261814851313829,
+            -0.018084971234202385,
+            -0.00587973790243268,
+            -0.004294826649129391,
+            -0.004362043924629688,
+            0.013316085562109947,
+            -0.0025471788831055164,
+            -0.0008468485320918262,
+            0.015099111013114452,
+            -0.0037323245778679848,
+            -3.927453508367762e-05,
+            -0.0059752571396529675,
+            0.015127412974834442,
+            0.0002699744363781065,
+            0.019457615911960602,
+            -0.04270770028233528,
+            0.019726485013961792,
+            -0.021396303549408913,
+            -0.003102605463936925,
+            -0.0020890405867248774,
+            0.009523618966341019,
+            0.00764861237257719,
+            -0.007142714224755764,
+            -0.014334957115352154,
+            0.004524780437350273,
+            0.014066088013350964,
+            -0.0006726144347339869,
+            0.005080206785351038,
+            -0.0027541371528059244,
+            0.0017228835495188832,
+            -0.010089659132063389,
+            -0.005862049292773008,
+            5.842812606715597e-05,
+            -0.010570792481303215,
+            -0.008646257221698761,
+            -0.03232087194919586,
+            0.01062032114714384,
+            0.010209942236542702,
+            0.007294837385416031,
+            0.004252373706549406,
+            -0.022018946707248688,
+            -0.012318440712988377,
+            0.012948159128427505,
+            -0.01132787112146616,
+            -0.002570174168795347,
+            -0.0020005968399345875,
+            0.005387990735471249,
+            0.0004705205501522869,
+            0.010160413570702076,
+            -0.003472300246357918,
+            0.010938717983663082,
+            -0.014603826217353344,
+            0.018084971234202385,
+            0.03229256719350815,
+            -0.00533846253529191,
+            -0.019967053085565567,
+            0.000907432462554425,
+            -0.009077862836420536,
+            0.003813692834228277,
+            0.007641536649316549,
+            -0.021962342783808708,
+            -0.02173592709004879,
+            0.005087282508611679,
+            -0.013153349049389362,
+            -0.007783046457916498,
+            0.014830241911113262,
+            0.005221716593950987,
+            0.010839661583304405,
+            0.01751893013715744,
+            0.010889190249145031,
+            0.00840569008141756,
+            -0.026533113792538643,
+            0.0006734988419339061,
+            -0.012686366215348244,
+            -0.00590803986415267,
+            0.002239394700154662,
+            -0.0017591454088687897,
+            0.004591997712850571,
+            -0.00010115749319083989,
+            -0.011200511828064919,
+            -0.0007057808106765151,
+            0.010903340764343739,
+            0.027919910848140717,
+            -0.004146241117268801,
+            -0.0022570835426449776,
+            -0.014257126487791538,
+            0.0010100272484123707,
+            0.036481261253356934,
+            -0.012275987304747105,
+            -0.017434025183320045,
+            0.01757553406059742,
+            0.027452928945422173,
+            0.005954030901193619,
+            -0.0032529598101973534,
+            -0.0027364485431462526,
+            0.00039600671152584255,
+            0.011978816241025925,
+            0.007719367276877165,
+            -0.002692226553335786,
+            -0.002131493529304862,
+            0.004761809483170509,
+            0.035773713141679764,
+            -0.0015203474322333932,
+            -0.010330225341022015,
+            -0.024240652099251747,
+            0.010974096134305,
+            0.00277182599529624,
+            0.004754733759909868,
+            0.002313687466084957,
+            0.023504801094532013,
+            0.019655730575323105,
+            0.014858543872833252,
+            0.00470520555973053,
+            -0.009771261364221573,
+            0.03005671128630638,
+            -0.008172199130058289,
+            -0.004542469047009945,
+            -0.02838689461350441,
+            -0.005087282508611679,
+            -0.015650998800992966,
+            0.008929276838898659,
+            -0.0002221042668679729,
+            -0.005016527138650417,
+            -0.013422218151390553,
+            0.019726485013961792,
+            0.01064862310886383,
+            -0.011073152534663677,
+            0.012608535587787628,
+            -0.027070851996541023,
+            0.02346234768629074,
+            -0.03826428949832916,
+            -0.0071993181481957436,
+            -0.006421013735234737,
+            -0.010514188557863235,
+            -0.02627839520573616,
+            -0.0030353881884366274,
+            0.007372668012976646,
+            0.014250051230192184,
+            0.0022570835426449776,
+            0.0022818478755652905,
+            0.02282555401325226,
+            0.0016415152931585908,
+            -0.01508495956659317,
+            0.017985913902521133,
+            0.03492465242743492,
+            0.012926933355629444,
+            0.00155837822239846,
+            0.015990622341632843,
+            0.0027789014857262373,
+            0.0021633331198245287,
+            0.00788917951285839,
+            -0.0034493047278374434,
+            -0.007938707247376442,
+            0.008582578040659428,
+            0.006990591064095497,
+            -0.02763689123094082,
+            0.003824306186288595,
+            -0.00010408719390397891,
+            -0.009898620657622814,
+            -0.022429324686527252,
+            0.011568437330424786,
+            0.01913214474916458,
+            -0.004224071744829416,
+            -0.006771250627934933,
+            -0.0043337419629096985,
+            0.0006836698739789426,
+            0.006785401608794928,
+            -0.002412744564935565,
+            -0.000515184598043561,
+            -0.0015176940942183137,
+            -0.02368876338005066,
+            -0.010316074825823307,
+            0.013570803217589855,
+            -0.005175726022571325,
+            0.010358527302742004,
+            0.005242943298071623,
+            -0.018792519345879555,
+            -0.016938740387558937,
+            0.006852618884295225,
+            -0.016188737004995346,
+            0.01364863384515047,
+            0.00027948213391937315,
+            -0.0023172253277152777,
+            0.024962352588772774,
+            0.03594352304935455,
+            -0.013188726268708706,
+            0.005119122099131346,
+            -0.019386861473321915,
+            -0.0245378240942955,
+            -0.019004784524440765,
+            0.022613288834691048,
+            0.008306633681058884,
+            -0.018792519345879555,
+            0.012304289266467094,
+            -0.002752368338406086,
+            -0.008865597657859325,
+            -0.0066721937619149685,
+            -0.006845543161034584,
+            -0.008547200821340084,
+            0.020533092319965363,
+            -0.004935159347951412,
+            0.0038950610905885696,
+            0.019047237932682037,
+            0.04075486212968826,
+            -0.008292482234537601,
+            -0.01349297259002924,
+            0.0054339817725121975,
+            0.004878554958850145,
+            0.0018626245437189937,
+            -0.004814875777810812,
+            -0.008016537874937057,
+            -0.0185944065451622,
+            -0.01740572229027748,
+            -0.009615600109100342,
+            -0.0020041344687342644,
+            0.020122714340686798,
+            -0.007768895477056503,
+            -0.01354957651346922,
+            -0.0245378240942955,
+            -0.028188779950141907,
+            -0.006580212153494358,
+            0.02319347858428955,
+            0.017179306596517563,
+            -0.00013200224202591926,
+            -4.3337418901501223e-05,
+            -0.012042495422065258,
+            -0.01968403346836567,
+            0.015509488992393017,
+            0.00037876018905080855,
+            -0.01773119531571865,
+            -0.0065448349341750145,
+            -0.015919867902994156,
+            0.005787756759673357,
+            -0.0051368107087910175,
+            0.004574308637529612,
+            0.05057565122842789,
+            0.0005271245026960969,
+            0.008441068232059479,
+            -0.026575567200779915,
+            0.009106164798140526,
+            -0.01583496294915676,
+            -0.017377421259880066,
+            -0.00012702727690339088,
+            -0.022301966324448586,
+            -0.0030955299735069275,
+            -0.018792519345879555,
+            -0.007386818993836641,
+            0.008568426594138145,
+            0.009799563325941563,
+            -0.013896276243031025,
+            -0.013669860549271107,
+            -0.024693485349416733,
+            -0.0035872769076377153,
+            -0.001539805089123547,
+            0.009311353787779808,
+            0.007825499400496483,
+            -0.03693409264087677,
+            0.002536565763875842,
+            -0.0009507699287496507,
+            0.0005779796629212797,
+            0.01208494883030653,
+            -0.008667483925819397,
+            -0.003063690150156617,
+            -0.0214670579880476,
+            0.006452853325754404,
+            0.007103798910975456,
+            -0.013500047847628593,
+            0.012375044636428356,
+            -0.012863253243267536,
+            -0.007669838611036539,
+            -0.0005426021525636315,
+            0.03379257395863533,
+            0.011794853024184704,
+            -0.0025153392925858498,
+            -0.00654129683971405,
+            -0.011150983162224293,
+            0.02654726430773735,
+            -0.011349096894264221,
+            -0.026405755430459976,
+            0.028910480439662933,
+            0.013245330192148685,
+            0.0005974372616037726,
+            0.020151015371084213,
+            -0.0017936384538188577,
+            -0.01397410687059164,
+            0.009601449593901634,
+            0.004266524687409401,
+            0.012424572370946407,
+            -0.00014460546663030982,
+            0.0214670579880476,
+            0.0018555490532889962,
+            0.0027399861719459295,
+            -0.0036226543597877026,
+            0.004687516950070858,
+            -0.03093407303094864,
+            -0.0006549257086589932,
+            -0.003102605463936925,
+            -0.008398614823818207,
+            0.011207587085664272,
+            -0.002421588869765401,
+            0.006558985915035009,
+            -0.022188758477568626,
+            -0.028485950082540512,
+            0.011342021636664867,
+            -0.016231190413236618,
+            -0.021396303549408913,
+            0.015339677222073078,
+            -0.0007641536649316549,
+            0.00614860700443387,
+            -0.007443422917276621,
+            -0.023278385400772095,
+            -0.019386861473321915,
+            0.02064630016684532,
+            0.012212308123707771,
+            -0.01190098561346531,
+            0.012750045396387577,
+            -0.021113283932209015,
+            -0.012948159128427505,
+            -0.001894464367069304,
+            0.0090566361322999,
+            -0.0019864458590745926,
+            0.01325240544974804,
+            -0.012198156677186489,
+            -0.00334494118578732,
+            0.0031592093873769045,
+            -0.011589664034545422,
+            0.008158047683537006,
+            0.01978308893740177,
+            0.008193425834178925,
+            -0.009339655749499798,
+            0.003422771580517292,
+            -0.011922212317585945,
+            0.013500047847628593,
+            0.015537790954113007,
+            0.018311386927962303,
+            -0.006498843897134066,
+            0.0020412809681147337,
+            -0.01370523776859045,
+            0.0063962494023144245,
+            -0.009827865287661552,
+            -0.004878554958850145,
+            -0.0005333155859261751,
+            0.004921008367091417,
+            -0.011179285123944283,
+            -0.0025772498920559883,
+            -0.026830283924937248,
+            -0.0013823752524331212,
+            -0.020533092319965363,
+            0.005303084850311279,
+            0.023207630962133408,
+            -0.007514177821576595,
+            0.0006045127520337701,
+            -0.0039092120714485645,
+            0.014476466923952103,
+            0.020688753575086594,
+            0.00870286114513874,
+            0.00482548912987113,
+            -0.006081389728933573,
+            0.02179253101348877,
+            0.011603815481066704,
+            -0.017816102132201195,
+            0.00164239970035851,
+            -0.001509734196588397,
+            -0.0014336726162582636,
+            0.011625041253864765,
+            0.004443412180989981,
+            -0.020844414830207825,
+            0.0028602697420865297,
+            -0.010089659132063389,
+            0.007818424142897129,
+            0.014702882617712021,
+            -0.00031906069489195943,
+            -0.02486329711973667,
+            -0.009827865287661552,
+            0.021919889375567436,
+            -0.015070809051394463,
+            -0.0008291598060168326,
+            0.00764861237257719,
+            0.023264234885573387,
+            -0.024792540818452835,
+            -0.006831392180174589,
+            -0.0023455272894352674,
+            -0.003933976404368877,
+            -0.005193414632230997,
+            0.0010489424457773566,
+            0.0214670579880476,
+            0.004415110219269991,
+            -0.004234685096889734,
+            0.005724077112972736,
+            0.006470541935414076,
+            0.02325008250772953,
+            0.0040577976033091545,
+            0.028047269210219383,
+            -0.00943871308118105,
+            -0.0023809047415852547,
+            -0.0019988277927041054,
+            -4.775960405822843e-05,
+            0.0160047747194767,
+            -0.005430444143712521,
+            0.020320827141404152,
+            0.009445788338780403,
+            0.0036934094969183207,
+            0.016514210030436516,
+            0.02319347858428955,
+            -0.006767712999135256,
+            -0.004793649073690176,
+            0.019160445779561996,
+            0.0053667644970119,
+            -0.008858522400259972,
+            -0.02194819226861,
+            0.01141277700662613,
+            0.0014124460285529494,
+            -0.0006956097786314785,
+            0.012106175534427166,
+            0.0024375086650252342,
+            0.006300730165094137,
+            0.008172199130058289,
+            -0.016684021800756454,
+            0.004241760354489088,
+            -0.004698129836469889,
+            -0.01860855706036091,
+            0.010910416021943092,
+            -0.012092024087905884,
+            -5.201595922699198e-05,
+            -0.0010002984199672937,
+            0.014165145345032215,
+            -0.01703779585659504,
+            -0.0019103841623291373,
+            -0.0070719593204557896,
+            -1.3024719009990804e-05,
+            -0.0009162768837995827,
+            0.008320784196257591,
+            -0.006254739128053188,
+            -0.010938717983663082,
+            -0.010938717983663082,
+            0.027509532868862152,
+            -0.004991763271391392,
+            0.003916287794709206,
+            0.019330257549881935
+          ],
+          "embedding": [
+            0.007204997251390758,
+            -0.03173324616100502,
+            -0.020505790884701024,
+            0.0037451608231038273,
+            0.009817085391002229,
+            0.026883177166072393,
+            -0.016477826654817607,
+            0.06949311853773875,
+            -0.01224876667472692,
+            0.031952566873802865,
+            0.013633671951171586,
+            -0.005041437355104989,
+            0.02663392750069342,
+            -0.022099132865267587,
+            0.006665702000169926,
+            0.0030467129715919033,
+            0.007931802508415542,
+            0.009984477355413895,
+            -0.010362836464183985,
+            -0.03180848181251013,
+            -0.013542377333474449,
+            -0.013572345022577053,
+            -0.03979985300694044,
+            -0.007698412130292187,
+            -0.005953452234356532,
+            0.018890228917932694,
+            0.02217446994660065,
+            0.005158357501784182,
+            -0.05468694715835175,
+            0.00789994048345594,
+            0.017130184174786147,
+            -0.0026591475669416114,
+            -0.02081141846859738,
+            -0.008403928318906045,
+            -0.030925346724945978,
+            -0.03855023691059801,
+            0.038853354785097736,
+            0.01612011653434478,
+            -0.035635702395196575,
+            0.026266623757525934,
+            0.043450373922427316,
+            -0.007254334835408145,
+            -0.015376804869420366,
+            -0.0017276202867766907,
+            0.004650414207094434,
+            -0.002365514361535186,
+            0.011906139905776183,
+            -0.0005664194302009289,
+            -0.0191723447249157,
+            -0.025546124518037998,
+            0.01025005257280377,
+            -0.014436532079436437,
+            0.029277529346694118,
+            -0.0045414983847733344,
+            -0.02260315456913697,
+            0.01717918656248626,
+            0.013666132349359846,
+            0.057804090189845364,
+            0.056600576093100735,
+            0.051208022172527734,
+            0.03553760896972431,
+            0.02094599510567085,
+            -0.02438674472832137,
+            -0.0027797634437333432,
+            0.03614248430646102,
+            -0.026729308357113124,
+            -0.002251394534452815,
+            0.018929251842788906,
+            0.00689697483318378,
+            -0.026603606434115974,
+            -0.0541443728344337,
+            0.027164630314707807,
+            -0.004335272389193875,
+            -0.013967321193992958,
+            0.003323121377857836,
+            -0.0245295956412367,
+            0.024489335129134787,
+            0.06212059558640156,
+            0.009214058207162371,
+            -0.002082293435905329,
+            0.013659934201796686,
+            -0.003344037927706573,
+            -0.00689041317919184,
+            0.020235785390418023,
+            0.0212739457082623,
+            0.0016782346585238381,
+            -0.013484221664059555,
+            -0.06465582506604173,
+            0.011882797526607326,
+            -0.032970655964890155,
+            -0.04027700439305653,
+            0.025396847912528105,
+            0.013861157517716707,
+            0.00604596157583048,
+            -0.00744611407420936,
+            0.04038084298991919,
+            0.021507532290328625,
+            0.005983822806623904,
+            -0.012711110369915742,
+            0.02410138142963901,
+            -0.018372791517880082,
+            0.021016674128298246,
+            0.013387285956415756,
+            0.02039383525127615,
+            0.010700292312227496,
+            0.0053934660497349735,
+            -0.014807678014449691,
+            0.009422805142740376,
+            0.0414357625490679,
+            0.12206398267002624,
+            -0.010877509834324155,
+            -0.011079302503786522,
+            -0.04107714853756596,
+            0.006631397986165343,
+            -0.031914267104222384,
+            -0.0179965295457466,
+            -0.057530036803978266,
+            -0.0320488164246567,
+            0.008208202363320509,
+            -0.0020203090915064162,
+            0.03502055780919601,
+            0.04464599164916666,
+            0.019332152079287082,
+            0.017133943087431935,
+            -0.039885565008919065,
+            0.04703059988373572,
+            -0.029370696964304416,
+            0.0008494086835971101,
+            0.0014082189719987074,
+            -0.030973762582071615,
+            -0.006561157852024518,
+            0.011634653995510726,
+            -0.019896826887411713,
+            -0.034275921680252924,
+            0.026274465530039866,
+            -0.007728234125236442,
+            0.04814514040690998,
+            0.06550407296280117,
+            -0.02749051703615343,
+            -0.010041099502187074,
+            0.02019601967763537,
+            -0.01871701432558228,
+            0.008532829759864928,
+            -0.020164138072967613,
+            -0.045196925901010855,
+            -0.004523517988516156,
+            -0.002113469149506399,
+            0.07929650663710812,
+            -0.01111269708042164,
+            0.03358254444720164,
+            -0.019973382759721527,
+            -0.0007145592863325417,
+            -0.0011740862928390678,
+            0.02166182480535788,
+            0.004543228023398081,
+            -0.013377688428761516,
+            -0.01758250073647576,
+            0.009548292014596834,
+            -0.04753617258004735,
+            -0.007125744019792479,
+            -0.009930358894299303,
+            -0.007931672044235996,
+            -0.024561936537429774,
+            0.012979195012881556,
+            0.009209145250061479,
+            0.04586756662401364,
+            0.04311807301838248,
+            -0.0036241105681042675,
+            -0.06579256720896123,
+            0.04967191755416882,
+            0.015475886231466817,
+            0.03157651656768371,
+            0.02515941109300714,
+            0.01773554572598468,
+            0.005824418191900522,
+            0.016317635547562007,
+            0.007650872352957098,
+            0.010015166465492963,
+            -0.033203307142867844,
+            0.040272188258443496,
+            -0.013408423788025047,
+            0.01036004294342104,
+            0.012391622448703885,
+            0.04214857862444935,
+            0.03418032388760535,
+            0.015263018079889338,
+            0.018786704580603317,
+            0.02704685343985817,
+            0.015348111464412869,
+            0.0013031600434754704,
+            0.03241427528588285,
+            0.011689924942195664,
+            0.04544406114742012,
+            0.016186615914968734,
+            0.011624500626103958,
+            -0.008272953078864385,
+            -0.02363080189189375,
+            0.02784298119317616,
+            -0.010585980875564109,
+            0.01455013604305903,
+            0.01749228557615982,
+            -0.054742056843690476,
+            -0.005876213780171779,
+            -0.010556605098730201,
+            0.003967456397644932,
+            0.014663888851767762,
+            0.0015548476559947807,
+            -0.003858367677774835,
+            0.012504286824484335,
+            -0.025806882498628832,
+            -0.014216299453397403,
+            0.0031620924516417956,
+            -0.006448390918604307,
+            -0.02785577502346934,
+            -0.00766402687921159,
+            0.005275000635711884,
+            -0.0090448751942,
+            0.019702662630533,
+            0.007009091837467867,
+            -0.01847307100043082,
+            -0.0006774105206839772,
+            0.032952656101827446,
+            0.015900033044760432,
+            -0.02170121261931376,
+            -0.017964879081469612,
+            -0.005062849745084668,
+            -0.03171847232054206,
+            -0.038177398521606544,
+            0.029426291950046455,
+            -0.025687201683643784,
+            0.030158867757101797,
+            0.015780411479509624,
+            -0.007011484049169482,
+            -0.012114023533118393,
+            0.020661407942701628,
+            -0.025067530380881376,
+            0.009777190971413269,
+            -0.004160771288889298,
+            -0.028825369485727344,
+            0.008307290529469729,
+            -0.014415903546305744,
+            -0.007561235629544945,
+            -0.03201704135271719,
+            -0.015182869388110948,
+            -0.00501561450194676,
+            -0.0003863746169672397,
+            -0.01073403445816663,
+            -0.03321103354401307,
+            -0.002304357889348034,
+            0.017001213960120047,
+            0.00028222146729658097,
+            -0.04002882852650754,
+            0.014509588052058002,
+            -0.004150541038999649,
+            -0.034333837380045466,
+            0.0564137596224324,
+            0.03584898691608747,
+            -0.006830698606679202,
+            0.006121456445207273,
+            -0.04505674213649008,
+            0.04358467937307492,
+            0.010847859073780811,
+            -0.04918983713702805,
+            0.049190442935189385,
+            -0.00528960443684769,
+            0.01419195371129309,
+            0.040316024601793406,
+            0.004600843996974165,
+            -0.006780970176765864,
+            0.005206651850992678,
+            0.03397517915308201,
+            -0.010037223444627569,
+            -0.027377913828275806,
+            -0.03328136706915221,
+            0.02562568440972934,
+            -0.03485770844248482,
+            -0.020069979230687374,
+            -0.0052041408068332524,
+            0.029818953377223505,
+            0.028789460147370508,
+            0.004071274130619537,
+            0.01928375754300958,
+            0.0014970947734202088,
+            0.010502599795027616,
+            0.016914145702515263,
+            0.011942576738188502,
+            -0.014909953528043828,
+            -0.003432489584767786,
+            -0.05733768990013957,
+            0.0030710780388681002,
+            -0.02627280695622545,
+            0.007930906480371135,
+            -0.031227917075485834,
+            0.013132797714872345,
+            0.002514351081823354,
+            -0.007152021475569949,
+            -0.0066245205132327055,
+            -0.005279224195782313,
+            0.028512607222275155,
+            -0.00921960825771865,
+            -0.0359704105589433,
+            -0.021347597388353182,
+            0.011200044645812517,
+            -0.0004171294784927543,
+            -0.016784468873218786,
+            -0.0018746544944553878,
+            0.007749103417240419,
+            0.02390272097796833,
+            -0.005128540108845805,
+            0.006397849195755672,
+            0.01214922062611769,
+            0.01163371266250103,
+            0.012307186722130844,
+            -0.004982589662882946,
+            -0.016343724553037307,
+            0.019806887143115643,
+            0.013139657463054087,
+            0.0205801609265216,
+            -0.03767676007629612,
+            -0.0249879011550972,
+            0.05547392586363707,
+            0.005903313549319051,
+            -0.05568806298774435,
+            0.018827693208840233,
+            -0.007931178941307645,
+            0.008183992600768387,
+            -0.010319022946041053,
+            0.05111352384595183,
+            -0.024897203290807753,
+            -0.02096678205188722,
+            0.006436523335212348,
+            -0.04917084523640266,
+            -0.010459284852442706,
+            -0.011138558192296276,
+            0.014789370043397917,
+            -0.03105725978752438,
+            -0.004934745004818259,
+            0.015595801375817122,
+            -4.687585727052446e-05,
+            0.0028886067775750254,
+            -0.00850352807499054,
+            -0.0034885579171087815,
+            0.016339681487281293,
+            -0.0034050317046826813,
+            -0.008309570448487812,
+            -0.003615165868022203,
+            -0.027638409919916757,
+            0.023352910161586186,
+            0.026364511529991978,
+            0.001249936082895606,
+            0.001987948967948376,
+            0.01756629246122769,
+            0.004656075485833687,
+            -0.006752579977941239,
+            -0.013681608638716132,
+            -0.03405936202918687,
+            -0.0401867651078737,
+            0.022991180183813725,
+            -0.02318908140203705,
+            0.018299512436825923,
+            0.00904081847012441,
+            0.019697091729447287,
+            -0.019236130984371835,
+            0.015159917756323988,
+            -0.0387297415849285,
+            -0.018793498297161176,
+            -0.005256288213225733,
+            -0.010414697711744128,
+            0.0007039894599589104,
+            -0.01739645696111503,
+            -0.013728845328097506,
+            0.022709522955937343,
+            -0.022789243441781803,
+            -0.0036763802589970026,
+            -0.01429671717544519,
+            0.0023674126354206932,
+            0.049448298344058644,
+            -0.009926033487593551,
+            -0.023371285193090733,
+            0.020846792916650956,
+            -0.026996519120163966,
+            0.011246908457705283,
+            0.04091365159640201,
+            0.005469162850197613,
+            0.005640919201612091,
+            0.029575753750229278,
+            -0.0007860452837141593,
+            -0.0052805237594885516,
+            -0.009782745443933828,
+            0.021178503581190618,
+            -0.0009176512672832309,
+            -0.05245650420332918,
+            -0.007698364493410058,
+            -0.039027418406703554,
+            0.005448840777995117,
+            0.017581625031764185,
+            0.025522517097715556,
+            -0.020775492472580457,
+            -0.015906320383873178,
+            -0.000643669839811819,
+            -0.0086896084429604,
+            0.0029399233482444638,
+            0.014381240985647907,
+            0.021086693776922936,
+            0.002436575081260609,
+            -0.004913514868101904,
+            -0.011582101006213432,
+            0.014400297663309155,
+            0.0467181039341492,
+            -0.01722692180137712,
+            0.0011622224560709315,
+            -0.013036540368441797,
+            -0.06208454638917413,
+            -0.019160212331432766,
+            0.0351496273293468,
+            0.02429814055116254,
+            -0.009896662169762772,
+            -0.01647101145477158,
+            -0.002915821099802562,
+            -0.03557863734177486,
+            -0.01786433999615293,
+            -0.010699364747311715,
+            0.02390901423299708,
+            0.007213809581960646,
+            0.0005601364043388203,
+            0.00916614601011734,
+            0.014737491634516238,
+            0.019303599465763203,
+            0.008864535607492504,
+            -0.0575839606069681,
+            0.018051837786406608,
+            0.006315830080225406,
+            0.02525619013082567,
+            -0.015671924543113065,
+            0.003988943108210209,
+            -0.03179878281917948,
+            -0.029440641244429775,
+            0.040586322748158446,
+            -0.03418722657593859,
+            -0.030650542761032924,
+            0.006566745611226466,
+            0.009571089937153852,
+            0.004491158558315032,
+            -0.011352813042040581,
+            -0.02535505987721779,
+            -0.013381948772812133,
+            -0.030918676056735445,
+            -0.02192708443081135,
+            -0.005232667170255381,
+            -0.03342408726436631,
+            -0.07498634115550039,
+            -0.006245638452568239,
+            -0.049946896465756174,
+            -0.03900385398225758,
+            0.04458524125677807,
+            -0.006106489377959481,
+            0.02398410968928338,
+            -0.04615183642824994,
+            -0.01165382599292015,
+            -0.04890013373371541,
+            0.013624324077017377,
+            0.01476398305411351,
+            0.020202554298470666,
+            0.00014553149242532853,
+            0.036224316512987846,
+            0.00021498496267774312,
+            -0.05055208658013767,
+            -0.057482169168904106,
+            -0.003960451575823462,
+            -0.010217078509789989,
+            -0.005593738972150311,
+            -0.034090261748104665,
+            0.05835341009022349,
+            0.0116359661127526,
+            0.04203678184748103,
+            0.03447818335186531,
+            -0.006747564131379555,
+            -0.025756962754841885,
+            -0.028123416424745064,
+            -0.008073625379614702,
+            0.047544241458556825,
+            -0.029605671083240564,
+            0.008171081332993893,
+            -0.026007452205474395,
+            0.0067084589781962805,
+            -0.0070645448617940995,
+            -0.0012298101311792947,
+            0.012372286141738867,
+            0.006377319686763927,
+            -0.015024537877377194,
+            0.007934745664790507,
+            0.026041755081605937,
+            -0.007303731324541215,
+            0.0407000170257006,
+            -0.012990703097937777,
+            -0.033189084013863494,
+            -0.022525679964848666,
+            0.01657717698191603,
+            0.03440284885162039,
+            0.010230633738702899,
+            -0.008707808754455952,
+            -0.0363777020540216,
+            0.011184076269109602,
+            0.031774012431843524,
+            0.01504703222643543,
+            -0.022509833303978505,
+            0.025230974050270107,
+            0.017502662694813216,
+            -0.018700579139537456,
+            -0.03511734747709348,
+            0.015162183119327901,
+            -0.017711638132862063,
+            0.018190335050681287,
+            -0.01590174153215615,
+            0.011103110869700727,
+            0.0025189370696524936,
+            -0.00920305384736575,
+            0.01929652151514404,
+            -0.006196715938358087,
+            0.011426611271994685,
+            0.007058186102245829,
+            -0.018191352599117545,
+            0.0027202185072220937,
+            0.057310566886380025,
+            -0.000531639834386418,
+            0.01494975681909879,
+            0.03845692752400366,
+            -0.001971953300066856,
+            0.045554909803324765,
+            -0.02532332999916258,
+            -0.042616337063845953,
+            0.02633963974172818,
+            -0.02970705577683783,
+            -0.01624154998005974,
+            -0.02620993449678864,
+            -0.021490817420853802,
+            -0.004038424045768319,
+            0.009940887982818171,
+            -0.02692933845395842,
+            -0.023801570856681796,
+            -0.001948460540662095,
+            0.02290911497874141,
+            -0.03300532378029689,
+            0.006525488380791647,
+            0.007039306647666766,
+            -0.003685585408710752,
+            0.03183735852538168,
+            -0.027197064244321705,
+            0.006861928286688311,
+            0.011209869849682412,
+            -0.001094897323854524,
+            -0.005328676992017907,
+            -0.004007474089818478,
+            0.010796946231051235,
+            -0.030543173497906445,
+            -0.012879972141706173,
+            0.00935894527375688,
+            0.0017573071057611829,
+            -0.010050882212831222,
+            -0.03102132152986303,
+            -0.002915141291985554,
+            0.007634137769426993,
+            -0.011094093612593696,
+            0.01199609810193786,
+            -0.010458016758814742,
+            -0.04929347607956551,
+            0.010973132159729983,
+            0.028148117094907874,
+            -0.025504019123994758,
+            0.012470127357818321,
+            0.009182729379865986,
+            -0.016020133756226275,
+            0.016320111225714855,
+            -0.012051271446461976,
+            0.024098195821827578,
+            0.04317620443481784,
+            0.024592886889234396,
+            0.007727787037954994,
+            0.00771746776156926,
+            -0.0003147538723468374,
+            0.022572868970938624,
+            -0.016683340264320543,
+            0.013672085891597506,
+            0.020827021901755415,
+            0.02092309219721837,
+            0.0057130785714458055,
+            -0.003901728981013356,
+            -0.0036968415923910314,
+            0.002227024825579113,
+            -0.02106678364798051,
+            0.014578546755457675,
+            0.03128101864424301,
+            -0.005234627629617124,
+            -0.022394744692442837,
+            0.002430617645746389,
+            -0.015780389432069438,
+            4.7277885648917e-05,
+            0.013357803020920043,
+            0.013777651543690907,
+            -0.027713499926132385,
+            -0.0018999112228756628,
+            0.028364630573666852,
+            0.009146793705085586,
+            -0.009891297866910313,
+            -0.01644269527910728,
+            -0.0003903768506971414,
+            0.006512217314869576,
+            0.02408785204739448,
+            0.0018406499545367782,
+            -0.007360473750031044,
+            0.01484497577088404,
+            0.016723277095327158,
+            0.003629911330658209,
+            -0.0004820567356447713,
+            0.02394406451074397,
+            0.023794019615473026,
+            0.001632266615911801,
+            -0.02064379152040184,
+            -0.02755693521684878,
+            0.027335938400284726,
+            0.02769248863677192,
+            -0.006832353240623308,
+            0.027324973360715134,
+            0.003937207909967385,
+            0.013580154030133622,
+            0.02076818768224654,
+            -0.01784151382870951,
+            -0.017617903734311997,
+            0.0017316476451235418,
+            -0.009605294810027804,
+            -0.009829196696575069,
+            0.004891780008779742,
+            -0.011655402383771472,
+            -0.01124379942906224,
+            0.043205454954628145,
+            0.004713701851783307,
+            0.003560551284131962,
+            0.007894840447501912,
+            -0.018881214526749082,
+            -0.00945538175144614,
+            0.0019040259741796817,
+            0.029505863720301916,
+            -0.001400002868254808,
+            0.03315866433970851,
+            -0.00830542045504805,
+            0.008602408964961391,
+            -0.03202191233287069,
+            -0.006978289755114341,
+            0.0326499698728755,
+            0.015314305362524091,
+            0.015104436300213667,
+            0.0021257042489266787,
+            -0.0010840650756005028,
+            0.02434348355847526,
+            -0.007174433778817674,
+            -0.04839373063710409,
+            0.022696994450018916,
+            -0.004779511451775743,
+            0.023588216956749154,
+            0.0382733980620746,
+            -0.004232286553944853,
+            0.003815507826545929,
+            0.023531223314846965,
+            -0.01887650634213847,
+            0.015055403379782985,
+            -0.01918944132635701,
+            -0.017597900721414636,
+            0.02176630010543207,
+            -0.008896608357798277,
+            0.01732387427588561,
+            0.01764363756727683,
+            -0.012180043444768312,
+            -0.019003360499321047,
+            0.01229139850966224,
+            0.0025710091163655626,
+            -0.017075658234265562,
+            0.01797412486853335,
+            0.018372010308966135,
+            0.006723179570342314,
+            0.015417677746651683,
+            -0.00047634684109332134,
+            0.017608296628666153,
+            -0.001049984114818522,
+            -0.008416990841821452,
+            0.01827827907328822,
+            -0.009568717277044265,
+            -0.01658089563620308,
+            0.009808478422454319,
+            -0.05193977747210182,
+            0.010644335931025957,
+            0.0016781339270411392,
+            -0.01562285100294917,
+            0.00405036330963539,
+            0.007102724536527262,
+            0.005195069570261247,
+            0.0028773630901681767,
+            0.004610474981005765,
+            0.042921486858787584,
+            0.013213672397112759,
+            -0.0064030584930089415,
+            -0.010711457508461704,
+            -0.017480309425763254,
+            -0.027151012983864797,
+            -0.005204425551873271,
+            0.009927884476991038,
+            0.022601230152836282,
+            0.008046297278661014,
+            -0.0031858480011267704,
+            -0.02589628103025927,
+            0.019723430185137954,
+            0.03604835658149313,
+            0.062423099201768716,
+            0.010930268386637612,
+            0.02925564269324381,
+            0.02439517070384333,
+            0.023930432096006114,
+            0.013036828091946605,
+            0.006910311087503408,
+            -0.032995841694293995,
+            0.012813650252889403,
+            0.01333714655191133,
+            0.03678966264802025,
+            0.012571138353467183,
+            -0.05610151838497368,
+            3.0777966587107344e-05,
+            0.009791774904454618,
+            0.041959488802722524,
+            0.01956296954085397,
+            0.000155217569370113,
+            -0.007285602454264675,
+            0.031547870838711856,
+            -0.026518385135879206,
+            -0.012299593280363843,
+            -0.0026551378470968594,
+            -0.006190117573235779,
+            -0.003807907001161504,
+            -0.004991957475076049,
+            0.036983035940964804,
+            -0.015648565314743963,
+            -0.019256463332683493,
+            0.0004300362961407231,
+            -0.0015686065121077472,
+            0.0011311524136286674,
+            0.0032236641835276464,
+            0.013795345278976164,
+            0.015385029265915279,
+            -0.004800448639353721,
+            0.026514926384796363,
+            -0.008699453553651681,
+            0.022186817580012775,
+            -0.013427553742493343,
+            -0.011927259975371873,
+            0.017776228476138103,
+            -0.021243797085501972,
+            -0.01667280589187595,
+            -0.003688045116580565,
+            0.006189970874536939,
+            0.0020911968266404467,
+            0.0006097761986763103,
+            -0.004685468039463817,
+            -0.01011357666327168,
+            0.022631082517271542,
+            0.012398000826637277,
+            -0.006721588925828951,
+            0.018142966317821054,
+            -0.0021330638043572654,
+            -0.011869184647489572,
+            0.025073455250141923,
+            -0.0015392945266926425,
+            -0.0337085890598252,
+            0.01267302113143573,
+            -0.013438601165482187,
+            0.02673290895281225,
+            -0.009586762706902736,
+            -0.011208527168019762,
+            0.019144928964011168,
+            -0.0017591316941248897,
+            -0.005459150067129732,
+            -0.029230079356228025,
+            0.004679497471366242,
+            -0.011723195942933286,
+            0.015899377439148873,
+            0.0022054234777254627,
+            0.029187759252864872,
+            -0.018290694895057157,
+            0.00792533824593637,
+            0.011719198071061566,
+            0.014612375997487872,
+            0.005419943379355563,
+            0.010515265723144554,
+            -0.006893314445458116,
+            -0.04424495426064596,
+            5.8241472764594054e-05,
+            -0.005719143565673274,
+            0.0038220021965914754,
+            0.013106940982933836,
+            -0.01203955335201393,
+            0.011260626096963215,
+            -0.0279896016489392,
+            -0.0295228261396555,
+            0.036474811902902415,
+            -0.04900747926861378,
+            0.019261089742811332,
+            0.00014104646541594166,
+            -0.03057197731717424,
+            -0.04331886560073797,
+            -0.03232957159604667,
+            -0.019401402365464362,
+            0.004102283356333861,
+            0.0002610070704924167,
+            0.0012280372705500212,
+            -0.014025012323929194,
+            -0.014135990809256075,
+            0.013269444858490211,
+            0.00992155580941576,
+            0.011225162609266308,
+            -0.006545044517282357,
+            0.002368862850663672,
+            -0.02063183105457681,
+            -0.026981383275694142,
+            -0.02774194980145225,
+            0.013337303400678109,
+            -0.0025902548982556788,
+            -0.02794339093668326,
+            -0.025606606546251152,
+            -0.0029852498102770587,
+            -0.01906873442473352,
+            0.006508845592362198,
+            0.03268861529625085,
+            0.0355975143313428,
+            0.010066906387952277,
+            0.01690250467284985,
+            -0.017201544758038183,
+            0.048345309114140714,
+            -0.019031171948182872,
+            0.03188329748476941,
+            -0.0005845169448621133,
+            -0.024140202991373587,
+            0.00011597263700677706,
+            0.03203623528903764,
+            -0.019285565857567896,
+            -0.01332755507465857,
+            -0.010914187315481855,
+            -0.003374912271519852,
+            0.02419326154642231,
+            0.02443825785057301,
+            -0.0037923303809811746,
+            0.00022800399604730175,
+            0.015893265651968885,
+            0.003154527829977343,
+            0.007688529734165207,
+            0.030623379985627763,
+            0.014052282574575454,
+            0.01127235345371133,
+            -0.04164707563955819,
+            -0.011811587719246128,
+            -0.003763086694358232,
+            0.005909410979841885,
+            0.002913220159138154,
+            -0.002671613084376487,
+            0.0045765194162907885,
+            -0.007241703279856166,
+            -0.005855949422953439,
+            0.0016973541436777935,
+            -0.001576326424468957,
+            0.008393873711385013,
+            -0.0037217094191171083,
+            -0.03498879678492834,
+            -0.026096985039773694,
+            -0.021280075788049788,
+            0.006437497348154639,
+            0.0070214556591898375,
+            -0.00972721128392027,
+            0.01949553762684561,
+            -0.0029404319702066146,
+            0.000271037702827666,
+            -0.010734867909488861,
+            -0.0029784341515646075,
+            -0.01206464664516104,
+            -0.019544778160755956,
+            0.00035139045159739705,
+            0.013457601448003152,
+            0.00025947376767512006,
+            -0.03165522602526639,
+            0.011979811705146604,
+            0.006642849985850598,
+            0.013181162886675883,
+            -0.024499404540711422,
+            -0.02310714308893576,
+            -0.028028912907092744,
+            0.010570122662701183,
+            -0.0066866321204157725,
+            -0.024617571284817428,
+            -0.004097807033448707,
+            -0.0074576460628928256,
+            -0.0020417256533259887,
+            -0.02470953739335365,
+            -0.011408461668188081,
+            0.023537349682260345,
+            -0.006373652338136973,
+            0.01936338676879376,
+            -0.017636435070085086,
+            -0.017036091876814843,
+            -0.002198685044404603,
+            0.015584758061011048,
+            -0.038943923931820196,
+            -0.002076529978164001,
+            -0.008452513321945455,
+            0.006551055965075869,
+            -0.028840587857625097,
+            -0.007128279504360441,
+            0.0199397499488538,
+            0.012424319366271584,
+            -0.03518875279748495,
+            0.013596027106451115,
+            0.022191403935861113,
+            0.01218507466930854,
+            -0.0038551718784437783,
+            0.03910075115429626,
+            -0.0025215641203030336,
+            -0.005422688956956598,
+            0.016759888275451905,
+            0.04462538468871483,
+            0.023708097134660096,
+            -0.03152743757744915,
+            -0.008668774531128668,
+            -0.015224825785549947,
+            0.004102648962663789,
+            -0.025867795193966987,
+            0.004676262922216584,
+            -0.011739753544166856,
+            0.01645951757031186,
+            0.011253400919230348,
+            0.013958707917471253,
+            0.0376596076542735,
+            -0.005077354977296842,
+            0.02668910096392324,
+            0.0010332284250006117,
+            -0.00856723931523911,
+            0.0014431148305642308,
+            -0.01412579094241589,
+            0.029892393553325205,
+            0.02269599036836208,
+            0.014421409701746036,
+            0.029248195108770955,
+            -0.03588867295960255,
+            0.02458655544632528,
+            -0.0010013539341895672,
+            0.008368648583431747,
+            0.018150734970019018,
+            0.02628022565029391,
+            0.015423867373844644,
+            0.012477080028047823,
+            -0.02513018750675568,
+            -0.022180001014682045,
+            -0.0027550350630288997,
+            -0.031944843677988374,
+            -0.0014668093006914267,
+            -0.009789309083948147,
+            0.026386027599666254,
+            0.0046988411391149,
+            -0.019803160322638583,
+            -0.003931806845108628,
+            0.004482097428470392,
+            -0.00244660100774939,
+            -0.010441732346892935,
+            -0.004649318716095894,
+            -0.01869464992812615,
+            -0.017838643679294407,
+            0.023677878122153204,
+            -0.023958455640817666,
+            -0.025238725541146534,
+            0.014582798102040157,
+            0.0013223590277629197,
+            -0.002295641913710489,
+            0.0008238518461567685,
+            -0.02115533275282518,
+            -0.04214275933987015,
+            -0.027817083427786927,
+            0.0037588803464689412,
+            0.03565056002940463,
+            -0.026788865866928923,
+            0.014162550819567465,
+            -0.011336583458519,
+            0.019175861037658344,
+            -0.005127229083828615,
+            0.015531116410874209,
+            -0.018251290727445846,
+            -0.044630374763679266,
+            0.021833307809708984,
+            -0.012087009688208446,
+            0.02720742725750304,
+            0.021497809277370574,
+            -0.026937653159100156,
+            -0.013572733244188407,
+            0.01358214612023832,
+            0.0020408253706382332,
+            -0.009652962552750464,
+            -0.0063411929391088626,
+            -0.010533309167065193,
+            0.003901271588567942,
+            -0.04496345296706673,
+            -0.008761188928422701,
+            0.06948704056954039,
+            0.005916180375965137,
+            0.03157905519638486,
+            0.014668818318138927,
+            -0.028448496686381258,
+            -0.04008949536631331,
+            -0.00462253589823598,
+            -0.0003701522833020314,
+            -0.00024410762276269235,
+            -0.01850585249309217,
+            0.017699615637510956,
+            -0.002712069912576265,
+            -0.009767338815933488,
+            -0.013119464222297351,
+            -0.013743610418034631,
+            -0.030375365173543157,
+            -0.0030316327877227783,
+            -0.008632689470358149,
+            -0.006565457264917239,
+            -0.0328089275944709,
+            -0.011713996290006943,
+            0.029773910045828737,
+            0.02740100904353622,
+            -0.0061926915497348915,
+            0.00638737826122608,
+            -0.010244154924811975,
+            0.0010396203272677069,
+            -0.001264454445070139,
+            -0.004059969507786729,
+            0.008715730726831364,
+            0.0014767015415289076,
+            0.031074767756160294,
+            0.004761858344125113,
+            0.00987870172044607,
+            -0.059926117793699066,
+            -0.009294021263133636,
+            0.021188432871397475,
+            0.016602826531924377,
+            -0.009125784136292983,
+            0.013946188378030194,
+            -0.0017670794602549428,
+            -0.006837334179368295,
+            -0.028290055353601393,
+            0.024556541126233806,
+            0.04543639597357669,
+            0.0031311343860416473,
+            -0.011199639773247286,
+            0.014965677855595444,
+            0.0018248533506196013,
+            0.015507721118783542,
+            -0.011409402378400194,
+            -0.01338737990904486,
+            0.03063410115541065,
+            0.009305362935335817,
+            0.0025578631174065385,
+            0.012727648343848194,
+            -0.0038680554900795763,
+            -0.01173215289303824,
+            -0.01568862495293563,
+            -0.004101703129244437,
+            -0.03649200408657064,
+            -0.0022107849490954137,
+            0.009659780679763728,
+            0.0034091693454195806,
+            0.008848380026939594,
+            0.01842435611499587,
+            0.03546926214224266,
+            0.013478278853234764,
+            0.005970293629431872,
+            -0.0069197533761813455,
+            0.023196445250565058,
+            0.010403914879205351,
+            0.010017272564548626,
+            0.015076807790462658,
+            -0.029649260409273275,
+            -0.012737152920662164,
+            0.00373276777994811,
+            0.013239909450830258,
+            0.0001682544254742693,
+            0.02030652051051447,
+            0.014143961923579586,
+            0.0010147733405793957,
+            -0.012509446388074923,
+            0.0020437769941487626,
+            0.007405578085797006,
+            0.013458276946120546,
+            0.0006779114540819997,
+            -0.004250745498730309,
+            0.0009149988500358458,
+            -0.008355906072311008,
+            0.040391939952386505,
+            -0.01831218885700067,
+            -0.016793844108223176,
+            0.035064882870542116,
+            0.11416006426189942,
+            0.0010630856112440689,
+            -0.015660014600661857,
+            -0.017142534874931403,
+            -0.0086899013413391,
+            0.016611825572132997,
+            0.01609650977226868,
+            0.009780106186629793,
+            0.009622367309076686,
+            -0.000672926017804395,
+            0.024310949465350565,
+            0.02944698490225121,
+            -0.0037169531700178868,
+            -0.010990838766452194,
+            -0.036819646475041955,
+            0.022923962979125287,
+            -0.004596366586768123,
+            -0.007108041562635549,
+            -0.002604794313552557,
+            0.012641223813290853,
+            0.013668775645501733,
+            0.005125100040230003,
+            0.0077993841846437815,
+            -0.011500851674334447,
+            -0.017334458072700107,
+            0.0056819751688881774,
+            -0.015982234644378358,
+            -0.0076533013358874935,
+            -0.013127172654362694,
+            -0.011837346554842119,
+            -0.006562312362371076,
+            0.0029877420008976425,
+            -0.008572723854050838,
+            0.00854083969164562,
+            -0.010939639783598493,
+            0.016185642759084087,
+            -0.008052282332943615,
+            -0.0011992212936609377,
+            0.0010378646897070758,
+            -0.0025399515501741718,
+            0.010712865639618352,
+            0.01451357949543628,
+            -0.02521401163091652,
+            0.020836357346214075,
+            -0.008252813367155418,
+            0.0006409807249997337,
+            0.015338809580802425,
+            -0.04672526649678653,
+            0.022728876784937357,
+            0.005129225753457052,
+            -0.018372222092794564,
+            0.0066182736895667805,
+            -0.02048375780196863,
+            -0.00896382274306482,
+            0.0021738974344880794,
+            0.004491167150658828,
+            0.0034280799511763956,
+            0.015611133773752265,
+            0.018126362497548308,
+            0.00014383671280220997,
+            0.007927708693242186,
+            -0.006973076292062532,
+            0.008118313055163714,
+            -0.006791836664375275,
+            -0.01164567445668625,
+            -0.010894577039363014,
+            0.00014731542306100506,
+            -0.008865132658571428,
+            -0.011415696755798914,
+            -0.017922318360382588,
+            0.00358538001727405,
+            0.00289327637128695,
+            0.010406054711003452,
+            -0.007757607207626101,
+            -0.010994775479337082,
+            -0.011716096880261664,
+            -0.0013333474316266438,
+            -0.01231144419953554,
+            0.009792603083441917,
+            -0.02884953407490196,
+            0.0029097409399144306,
+            0.025277274938032477,
+            0.009263530315933738,
+            -0.028037993274175277,
+            -0.02532252134134532,
+            -0.0018596440554143638,
+            -0.013847737486669237,
+            0.009910107581119272,
+            0.013952962280235523,
+            -0.011807495289410774,
+            0.003896325037406752,
+            -0.02144921729633576,
+            -0.01628545945817568,
+            -0.0040148671532777005,
+            -0.016032493124255323,
+            0.004580041957001551,
+            -0.011375180364988195,
+            -0.007789996450583554,
+            -0.022041665680945066,
+            -0.005890065942573431,
+            -0.050455058269193866,
+            0.014401045842891128,
+            0.030219548461217365,
+            -0.004580747668649223,
+            -0.028684680969792875,
+            -0.015525870112797619,
+            0.011121401958547706,
+            -0.008879749268609245,
+            -0.019454243014778435,
+            -0.007839434838464604,
+            0.018908534797107714,
+            -0.0019283067432864216,
+            0.0024769961730655835,
+            -0.006732653593696035,
+            0.027604805021030646,
+            0.01366593691107021,
+            0.03740746059041736,
+            -0.014662268336336578,
+            0.02539666271785517,
+            0.02371809783638648,
+            -0.036126848124425154,
+            -0.007627445211958312,
+            0.03657481892016626,
+            0.01092515982671624,
+            0.0011046573579274704,
+            0.012350047525582187,
+            -0.006340802546498992,
+            -0.031330519599469066,
+            -0.0013691360532301728,
+            -0.0014676630169621005,
+            -0.023111265345539388,
+            0.027359543575867225,
+            0.018091458372134273,
+            0.014237156751820934,
+            0.0030993618578507172,
+            -0.0014037082734049385,
+            0.013257269860435359,
+            0.01021315143964527,
+            0.030885417052806473,
+            -0.003225065956381245,
+            -0.012825341172770057,
+            0.013476039084668826,
+            0.01852164843904835,
+            -0.009965495459579544,
+            0.012541079912270015,
+            -0.00824122613985912,
+            -0.018124134280975132,
+            0.03192263766222747,
+            0.0001484308749572035,
+            0.010888733498667343,
+            -0.010649515858317311,
+            -0.013082335214246402,
+            0.032614733822945205,
+            -0.00999075761269065,
+            0.004257643407198955,
+            -0.030723533031261387,
+            -0.003820776505926347,
+            -0.010471792928758869,
+            0.01737608180148319,
+            0.004597677757231315,
+            0.02095148515295301,
+            0.029253627700524396,
+            -0.011604450820008121,
+            -0.01996325509217509,
+            -0.001158626158150122,
+            -0.020583793738536985,
+            0.004168757459099133,
+            -0.03632338333261702,
+            -0.019373660230711246,
+            0.022259216603413172,
+            -0.010364620831688015,
+            -0.004751193403783363,
+            0.00444187509644351,
+            0.0008572202661850934,
+            0.014943779567410408,
+            -0.025408263453313398,
+            0.017755758370543148,
+            0.004627392860417132,
+            0.007402073071525438,
+            -0.024913329140362466,
+            0.013760771254234878,
+            -0.006087346663657887,
+            -0.03780469033982951,
+            0.005509388120163784,
+            0.03756141956381493,
+            0.02381019168406692,
+            -0.024580249929652097,
+            0.01807584361080568,
+            -0.03111039245576191,
+            0.023245542501125073,
+            -0.004868090951368273,
+            5.1544460403943004e-05,
+            0.00640235757127173,
+            -0.004698373089469779,
+            -0.00621383035264185,
+            -0.0025132415458431587,
+            -0.008746634331769936,
+            -0.01611801559493664,
+            -0.01193535623883546,
+            -0.03381789115349148,
+            -0.024119754887378317,
+            0.013881102852385332,
+            0.0019232668966222687,
+            -0.0095263729740736,
+            0.018652966857927805,
+            -0.03775912497511837,
+            -0.012022352025516115,
+            -0.023289377270709736,
+            0.023579672990954228,
+            -0.028605627815260902,
+            0.019235397553197357,
+            -0.025838655372560505,
+            -0.021458526449310157,
+            0.01508157885177254,
+            -0.010064396230734452,
+            -0.0016708624560165368,
+            -0.015042071642751917,
+            -0.03670785416972153,
+            -0.006267614265977843,
+            0.02298763967146357,
+            -0.012506444728541897,
+            0.00029871598584066644,
+            0.02671390618094565,
+            -0.003059865833779805,
+            -0.011103452976820622,
+            -0.0017382684168257506,
+            0.010923055147701782,
+            -0.02938686260440486,
+            0.014300097108092281,
+            0.0038571851420598074,
+            -0.0037614499297117365,
+            0.0010991440540809893,
+            -0.010108413486243776,
+            0.02127952161226619,
+            0.009142545227414532,
+            -0.0030260119510809205,
+            0.040589877142650374,
+            0.008677453889680373,
+            0.0017339508605300814,
+            0.008732950895530873,
+            0.01063352447960556,
+            0.006618635029823979,
+            -0.014986307792626314,
+            0.016905861015681957,
+            0.013101237900847744,
+            0.0002162791516784468,
+            0.019309343995079022,
+            0.0021263376627204313,
+            -0.024516267132902416,
+            0.013339486637573757,
+            -0.02311659751478777,
+            0.0020575982953961246,
+            0.02391695586495566,
+            -0.020440172706421535,
+            -0.001656132876701879,
+            -0.006304133667169861,
+            0.000543457661904495,
+            0.022135707414160034,
+            -0.0230903871986617,
+            -0.023444763940153112,
+            0.009604600033024832,
+            0.01721387169484476,
+            0.01827905276638783,
+            0.02824773106648473,
+            -0.007100625424382998,
+            0.01348510178676837,
+            0.0002604386361122316,
+            -0.011802178189607182,
+            0.007664157334475143,
+            0.0074175377458313594,
+            -0.0044008460682833875,
+            0.025491035291032547,
+            -0.0017015687016376205,
+            -0.006134010425966364,
+            -0.005379657499404915,
+            -0.01584573519366957,
+            0.0034386456145185213,
+            -0.013860742817569676,
+            -0.013355238174583658,
+            0.0040342831827141845,
+            -0.0208160950322342,
+            0.10904592104153767,
+            -0.0037038089210379766,
+            -0.019472012964045247,
+            0.0063554319446707044,
+            0.003716719884684827,
+            -0.003692786470980806,
+            -0.003353843210051379,
+            0.007788605566526459,
+            -0.011254480693190834,
+            -0.0045751956417398095,
+            -0.013976039724381875,
+            -0.001989262807706565,
+            0.010067374586433014,
+            0.009201983771978006,
+            -0.005902980676079493,
+            -0.02029714318363332,
+            0.013031676901335606,
+            -0.016970144750162643,
+            0.008874670945038486,
+            0.0008311255074953139,
+            0.00884167159995975,
+            0.001716603195908536,
+            0.01854951250316076,
+            -0.009310881405486074,
+            -0.01325242738760595,
+            0.029667589863813217,
+            -0.005616511284128333,
+            -0.0038179509972925954,
+            -0.0004690716809300012,
+            -0.005068974756744239,
+            0.0021204231125051274,
+            -0.027958274497127258,
+            0.022127088236172387,
+            -0.017271838492631593,
+            0.010267709087809265,
+            0.01316800560257366,
+            0.005599367128954229,
+            0.01817806970002991,
+            -0.007115594036834107,
+            -0.02627643123342057,
+            0.006542742528003751,
+            -0.003433874053082844,
+            0.006631427284537971,
+            0.02316295746074422,
+            -0.014573351469966297,
+            -0.033520864104177005,
+            -0.013707022143307006,
+            -0.01877574599842729,
+            -0.005992004412448576,
+            -0.0043698881826095435,
+            -0.01481352026460129,
+            0.006424121402963946,
+            0.01382938882387991,
+            0.018136021761515388,
+            0.0012697483790911433,
+            -0.011531136974861559,
+            0.00609187274277241,
+            0.01076133059240578,
+            0.008400887242930113,
+            -0.01533504932731406,
+            0.011439227832642668,
+            -0.028718487721012004,
+            0.01522697877548034,
+            -0.012452665599305595,
+            -0.024938748080573947,
+            -0.0022082450217489366,
+            0.017589399637694264,
+            0.0037824832255756763,
+            -0.004355445812154662,
+            -0.018778668474337727,
+            0.019825087149714933,
+            0.024692878851503508,
+            0.003883497401229232,
+            0.019470881143244197,
+            0.041834635328088364,
+            -0.003544509491805351,
+            0.027139164223766474,
+            -0.02129723143323512,
+            0.01461354860170749,
+            -0.03441729954693155,
+            0.001893713808284104,
+            -0.004222657393633789,
+            0.0013677766871517654,
+            -0.003811928806805752,
+            0.020142937759360904,
+            -0.009158202577174012,
+            0.0030638058587375543,
+            0.024854053046139163,
+            -0.018665378030958756,
+            0.00805903734779081,
+            0.010625429315145887,
+            -0.012856237251902233,
+            0.028535299711292306,
+            -0.006335094045840286,
+            0.00305422126851784,
+            -0.0297883488914759,
+            -0.0029192872993071216,
+            0.0026067044379754047,
+            -0.006097749614529602,
+            -0.00968023644800927,
+            -0.007433916539502931,
+            -0.006202274026852405,
+            0.0038354607448269555,
+            -0.021471361566683354,
+            0.003195646827154078,
+            -0.004390751282171024,
+            -0.003323447423321121,
+            0.02518759723554948,
+            0.025857746688368054,
+            0.016893707217894103,
+            0.012739170140289531,
+            0.008553625305650129,
+            0.019805351588748434,
+            0.01654964927067909,
+            0.011549329129728739,
+            -0.020026013396046632,
+            0.01150644479272541,
+            -0.004159611921476011,
+            -0.0022937297602724912,
+            0.014815466072709131,
+            -0.01572171840394285,
+            -0.02276187015864222,
+            0.024693009795978448,
+            -0.019725315346836404,
+            0.00998768979020729,
+            -1.1702210281215188e-06,
+            -0.011129745730507545,
+            -0.002591293237053776,
+            -0.018034357254990988,
+            0.00012070821244376275,
+            -0.023051045150566527,
+            -0.028465270110388716,
+            -0.00048603008831669926,
+            -0.000574546300981984,
+            -0.012992623897880395,
+            0.02064747165046032,
+            0.014820638761258362,
+            -0.024820174379739057,
+            0.02666518374464728,
+            -0.017430076811109155,
+            -0.01495051200726725,
+            -0.005861181182847845,
+            0.00100881727574559,
+            -0.0055223282819321295,
+            0.0034719859452138017,
+            -0.010180752654175722,
+            -0.009961678082468036,
+            -0.012134423047419183,
+            -0.028073672819317354,
+            0.011574683159318163,
+            0.016120674453124317,
+            -0.015043147648505721,
+            0.031834693135959095,
+            -0.008802805725050891,
+            -0.0018698230152007877,
+            0.016179742266521146,
+            0.005096232332051617,
+            -0.017187077376166815,
+            -0.018999569809271876,
+            0.012251308792871571,
+            0.013279090498325945,
+            -0.009588297039279112,
+            0.006630431200804809,
+            -0.014130176783207386,
+            -0.005092662672039885,
+            0.007020236623084732,
+            -0.005049363863784624,
+            0.0055051712211635915,
+            -0.00995314895304919,
+            0.0052743456988306485,
+            0.005841370337169303,
+            -0.0029434616355351975,
+            0.004677049987338193,
+            -0.018568941264514256,
+            0.020031253265629777,
+            0.0018725390936655962,
+            0.01332532735283085,
+            -0.03518855189467458,
+            -0.0056298723325832025,
+            -0.005369475318146738,
+            -0.004650364775945294,
+            -0.020833510091412353,
+            0.020445280915606132,
+            0.0159991245532085,
+            0.013202409908533162,
+            0.023094507005426774,
+            -0.014905666713082246,
+            0.01572052456975675,
+            -0.020945766915121455,
+            0.013114000985348841,
+            0.003349101658930482,
+            -0.019829605346106578,
+            -0.001423750749031516,
+            0.04557459007627153,
+            -0.005847231973329081,
+            0.011986110554869603,
+            0.0012709842140249691,
+            0.026758922819943853,
+            -0.0001635536762342011,
+            0.0065419519929442995,
+            0.005997692702992206,
+            -0.009935128108338126,
+            -0.0008057798606777003,
+            0.00865712443881613,
+            0.046254320605750854,
+            -0.01181132866321511,
+            0.008170688314239364,
+            0.013099051984057827,
+            0.0034971494510865624,
+            -0.0008386400616584293,
+            -0.015346506644390764,
+            -0.03067678541405639,
+            0.019305962963428017,
+            0.007736916644077731,
+            -0.012756525606221703,
+            0.008734673666457216,
+            0.015137771675476126,
+            0.012891824153867332,
+            0.003895886821685762,
+            0.02155154331885366,
+            -0.012761690789748027,
+            0.018528313792219413,
+            0.024663453412919027,
+            0.009077230086690327,
+            9.184526542175535e-05,
+            -0.000751529849753638,
+            0.006791927267880182,
+            -0.0004433479107716552,
+            -0.009832230053513483,
+            0.015858232096630655,
+            -0.014861341148074698,
+            0.017790924034108806,
+            2.7216806238946785e-05,
+            0.03744841066552559,
+            -0.015216207585985075,
+            -0.010138319613387346,
+            -0.007774325714088762,
+            -0.0016997034352120362,
+            0.003971564314593291,
+            0.0018686537276849653,
+            0.006051669054361943,
+            -0.0008319624938748146,
+            -0.017326561560776794,
+            0.01667737942301944,
+            0.005719808546053257,
+            -0.0007405637872110778,
+            0.013750752065593804,
+            0.0038732323799683453,
+            -0.00367645612814479,
+            0.023580906347714314,
+            0.006238541579764762,
+            -0.005235795826199542,
+            0.0028182323816636243,
+            -0.006443210286549713,
+            -0.016255209299456314,
+            -0.003061441395209219,
+            0.004597970789372988,
+            -0.0016590920570710628,
+            -0.011138261849069419,
+            -0.0019423173609729848,
+            0.020355534503272887,
+            -0.004612410336124759,
+            0.013346750536413418,
+            -0.005400060374036387,
+            0.006252154576727367,
+            0.010601459925435238,
+            -0.012346569079682625,
+            0.0014154620309235822,
+            0.010406076751218612,
+            0.0013983417564564663,
+            -0.007140180075225315,
+            0.0010544410919327675,
+            0.007757599672903709,
+            0.006505350216372801,
+            0.00483063937962671,
+            0.00836630436955631,
+            0.013284451514874498,
+            0.0033749897293287438,
+            -0.006243293993633622,
+            0.0013949831763016042,
+            -0.013269921385800992,
+            -0.006308639321247727,
+            0.0010906169976769569,
+            -0.0010152835709869353,
+            -0.02017142197515583,
+            0.010787630901414118,
+            0.005355784839945777,
+            0.007838485470991034,
+            -0.01706932891696025,
+            0.009576030158827565,
+            -0.0008144311254998525,
+            0.02829832828245172,
+            0.007407953684956539,
+            -0.02769403281764041,
+            0.014266001222243134,
+            -0.010492199740754951,
+            0.022575782293320284,
+            -0.0016349316593207844,
+            -0.005656505606143177,
+            0.008025514398322551,
+            -0.009634281579228592,
+            -0.005991026158213869,
+            -0.022366083313133788,
+            -0.025825209755160946,
+            0.016871087849192924,
+            0.013583173973358155,
+            0.021125252303971605,
+            0.00012555627355712715,
+            0.015347535950809997,
+            0.0020128011217859444,
+            0.015164099887550086,
+            0.0024544903332736167,
+            0.012507136277293888,
+            -0.011798050869794805,
+            0.0104828761281283,
+            -0.0137805761114781,
+            0.014620357150314097,
+            -0.005246639228547017,
+            -0.006986408834884282,
+            -0.008849580196616098,
+            -0.005442165099441538,
+            -0.020891551850452786,
+            -0.009155930286949747,
+            -0.0013590151948670868,
+            0.010665872513236615,
+            -0.0005386423407222223,
+            -0.0027671778465775897,
+            -0.03237311596935737,
+            -0.004875062533405715,
+            -0.006670982068843501,
+            -0.01942244110096111,
+            -0.0045975373348006885,
+            -0.026137783023267008,
+            -0.01657173843643047,
+            -0.0007784046498232684,
+            -0.015037306676062236,
+            -0.006889680260149007,
+            -0.02009705080543122,
+            -0.027192867101205106,
+            3.0161964469866572e-05,
+            -0.009647644858034833,
+            -0.005380165957043753,
+            0.013815092649296373,
+            0.011902366689169847,
+            -0.01572592170758115,
+            -0.022321702398924333,
+            -0.01088937403483412,
+            -0.0165754067023749,
+            0.01646435743868707,
+            0.009914938007210463,
+            0.018834863207293908,
+            -0.003724245912768107,
+            0.005643505694722323,
+            0.0009581443359000331,
+            0.02327704258596552,
+            -0.02532422120223976,
+            -0.028694220894404807,
+            0.0019298778426469338,
+            0.0025970175688368323,
+            0.006086164866310174,
+            -0.009414665394188936,
+            0.001200390869257535,
+            -0.007782899559317819,
+            -0.0228264209954853,
+            -0.00898519580978142,
+            0.008156835270668449,
+            -0.010036630164522355,
+            0.02230664068547531,
+            0.012332230358869292,
+            0.013699197379189934,
+            0.010593587938184292,
+            0.01576174456497367,
+            -0.0001654241831584687,
+            0.014471905257729428,
+            -0.008675915624727887,
+            -0.0031070346099204425,
+            0.0038358404385932777,
+            0.005878417037187274,
+            -0.00625022998235275,
+            0.028077114529659795,
+            -0.0011836453716817525,
+            0.0002962695480296182,
+            0.014405782572641941,
+            0.02697667559474498,
+            -0.004710313427099278,
+            0.0210512043261309,
+            -0.002393165025068831,
+            -0.006135816286212752,
+            -0.008429776018235104,
+            0.008607617299702318,
+            -0.009529291136733401,
+            0.011133497451942684,
+            0.019864404836361926,
+            -0.014041315457579166,
+            -0.007351413712676263,
+            0.02950254974910368,
+            -0.014564388091367897,
+            -0.01825317106331622,
+            0.02325964952838705,
+            -0.007414295676409368,
+            -0.0039042587057605484,
+            0.00377144546378926,
+            0.007169856503233239,
+            -0.004036744966949995,
+            -0.012928341573802416,
+            0.016090527447479415,
+            -0.00011159893712484579,
+            0.014146137810890391,
+            -0.0004655839804281017,
+            -0.006372231942983921,
+            -0.010047382825720416,
+            0.002373224247019062,
+            0.026818813126567614,
+            0.013483627627077692,
+            -0.017895710143841333,
+            -0.013785856764479817,
+            0.010826610341182479,
+            -0.01538893818197323,
+            -0.004425206001945227,
+            0.02155595194812746,
+            0.0028768153668128553,
+            -0.020543027467698483,
+            0.008028975884363618,
+            0.0003835165718138906,
+            0.00037046947047892953,
+            -0.00029363296913641716,
+            0.030135739208689217,
+            0.001343516577762509,
+            0.00018948658255495327,
+            -0.00875053830853609,
+            0.008606573235959595,
+            -0.01280305089741873,
+            0.012786478836532904,
+            -0.02437993139337839,
+            0.007648181775166282,
+            -0.00629541989541625,
+            0.004579605315599212,
+            0.016658386927051592,
+            0.01218098627022323,
+            0.015031618813778413,
+            0.008237200519537417,
+            -0.008942965899756683,
+            0.0009093265012901892,
+            -0.025302175262520477,
+            -0.001333639155622843,
+            -0.017971876771137367,
+            -0.02036433247634014,
+            0.0002463842837918767,
+            0.022144712689630985,
+            0.018559482073124734,
+            0.011579840028464503,
+            -0.008657014440886656,
+            0.0166696911919226,
+            -2.9834620942882283e-05,
+            0.011845827764518911,
+            -0.00037901176785495324,
+            -0.006138374154553732,
+            0.004711058953619512,
+            -0.003622694588720417,
+            0.013542405259906128,
+            -0.014730567462694827,
+            0.012076044672004906,
+            0.01690223095263866,
+            -0.009108632851807137,
+            -0.008946251274440141,
+            -0.013992423874475428,
+            0.00048381889145447367,
+            -0.015909599902931202,
+            0.035489252674454214,
+            0.02252772349379229,
+            -0.01210464970923921,
+            -0.010469120693943236,
+            -0.014164999313775756,
+            0.019496048806752753,
+            0.010427539351244087,
+            0.0028018908451425276,
+            0.017091768406027965,
+            0.021395638416767112,
+            0.008817247593404623,
+            -0.006081063711583319,
+            -0.0021216503501144983,
+            -0.003781810594530787,
+            0.020143098218482827,
+            -0.003078172339607037,
+            -0.011350171884507555,
+            -0.012516379645909518,
+            0.005766102930704529,
+            -0.01626433668691071,
+            0.0019294643928566657,
+            0.008641558521764407,
+            -0.015057688533773374,
+            0.019407986696657772,
+            -0.028126197703647868,
+            0.012364374199657305,
+            -0.0044278434183731225,
+            -0.007251147925369614,
+            -0.03496220945653279,
+            0.006389489026974697,
+            0.008941259259632024,
+            0.003941752277454314,
+            0.023003838655784314,
+            -0.024528514847470928,
+            -0.018361931455105486,
+            0.0037791061321962543,
+            0.00047162060848090104,
+            -0.00711894577052754,
+            0.0010138871688747137,
+            0.004128392769213672,
+            -0.00674546985015164,
+            0.0017307097443508704,
+            0.0033354800298095546,
+            -0.002829898077850263,
+            0.013252947102333348,
+            0.0081724269619999,
+            -0.021449979263734438,
+            0.001368791856860285,
+            0.01590656734732547,
+            0.01452008402429858,
+            -0.02391847163018108,
+            0.006331397860689212,
+            -0.02064812934794864,
+            -0.000631340242718475,
+            -0.031272866000888556,
+            0.016859793097564958,
+            -0.0069008624795683276,
+            0.01899718801899956,
+            -0.0015377436942771224,
+            -0.0065810685260109004,
+            -0.012822620655323715,
+            0.02137604937338179,
+            -0.0019121137078344135,
+            -0.0025598772127157477,
+            0.015678293057066646,
+            -0.025700865215693987,
+            0.017825186616033165,
+            -0.0026230848396158204,
+            -0.017342967698019966,
+            0.017199139867162275,
+            0.004536019178633225,
+            0.007224666811309087,
+            -0.023505589665315567,
+            0.0007208041198255396,
+            0.003900164485347562,
+            -0.017250324025622685,
+            0.004470862192961212,
+            -0.030807400122398566,
+            0.0006579147990327597,
+            -0.013222907940559794,
+            -0.0020514994691204847,
+            -0.0046763235724575445,
+            -0.014050035576183385,
+            -0.039181147615786484,
+            -0.005429639279537522,
+            0.016065742005148733,
+            -0.0009029636981539927,
+            0.01800715198737081,
+            -0.002730931639143803,
+            0.002947680833843522,
+            0.004912692811020822,
+            0.00818378920043427,
+            0.006915895602313347,
+            0.02274116906914758,
+            -0.0057854922886970365,
+            0.00471999900678522,
+            0.007058014277137233,
+            0.014638081356210031,
+            -0.01719565951201012,
+            -0.015170377598372723,
+            -0.016015921063369496,
+            -0.006351213642269936,
+            0.01739665344303718,
+            -0.017488885259028475,
+            0.013303332445465686,
+            -0.00429428898520701,
+            0.014825338166443021,
+            -0.008686427208690176,
+            -0.01575945086724703,
+            0.0028564767084934386,
+            -0.007570417028925067,
+            0.001007670647768344,
+            -0.02323908008484089,
+            0.02019729564706863,
+            -0.00847581038203716,
+            0.010508144473197669,
+            0.01729824052022313,
+            -0.021364529179605813,
+            -0.009742851959432839,
+            -0.014342281441152413,
+            0.029896210049029176,
+            0.013925163184699139,
+            -0.009392252140772913,
+            0.003927230832659267,
+            0.008873561269230905,
+            0.011529536139797032,
+            -0.012209070214071445,
+            -0.0027679974125029362,
+            0.002411914349172495,
+            -0.0062056207925691356,
+            -0.002025687569076897,
+            0.03090134105002816,
+            0.0038285457265343245,
+            0.024877474647546086,
+            -0.013214997316123852,
+            -0.0033763842539861907,
+            0.00023379106197363714,
+            0.005134764372494098,
+            0.01183867975321486,
+            -0.0017966997954747142,
+            -0.008029791503028367,
+            -0.01179500476055557,
+            0.02599911330072515,
+            -0.021852083993691557,
+            0.007024505421035273,
+            0.026784024996764876,
+            0.001390202993990949,
+            0.0029956956045771046,
+            -0.013131691460546865,
+            -0.006298736864509989,
+            0.02523640664202355,
+            0.010030824059858138,
+            0.0041523782603032225,
+            -0.026465819482580182,
+            0.0032128899272377593,
+            -0.0061901409332948115,
+            -0.008681758126710213,
+            0.009866461769956222,
+            -0.0028095454857224164,
+            0.01188012041754054,
+            -0.012191996383095982,
+            0.028459939543805705,
+            0.010711502132045176,
+            0.01047154118931583,
+            0.025888652452698974,
+            -0.02079614305315134,
+            0.0042203106489994615,
+            -0.010829796140534066,
+            0.0006773493086316061,
+            0.005596640511307504,
+            -0.027510562413905437,
+            0.001969898603871003,
+            0.010390866563404346,
+            -0.005642706366801436,
+            -0.01882994179544761,
+            0.01187970346127799,
+            -0.019814385265412387,
+            0.024479579275617887,
+            0.003231341626501863,
+            0.020056914356101588,
+            -0.0053276448188067154,
+            0.009834685132275868,
+            0.0002742942778577165,
+            0.0031361331890882127,
+            -0.0008074598092128465,
+            0.009009012303347016,
+            0.005786777410144894,
+            0.013719989127068968,
+            -0.025528370602188686,
+            0.006821572597883499,
+            -0.0029197127345352626,
+            -0.030539872624857384,
+            -0.013383097680839732,
+            0.010291485622655026,
+            0.012218533188117704,
+            -0.009144425807890594,
+            -0.010378679518611277,
+            0.0030042579827305603,
+            0.005748243116238633,
+            -0.006692403885450407,
+            0.006913672151000981,
+            0.01629516912493545,
+            -0.002533517074771728,
+            -0.0019083178803585312,
+            -0.0035007340321378893,
+            -0.01154609935927537,
+            -0.010016998439072824,
+            0.023193275321990937,
+            -0.00482990342915347,
+            0.017943025772239603,
+            0.025873224580389702,
+            0.0039876567755322,
+            -0.0057577932736143506,
+            -0.005460363755206188,
+            0.01478435999841998,
+            0.014195457012279476,
+            0.004669295894868759,
+            -0.009702220679239768,
+            0.007974305230005956,
+            -0.007760086293456636,
+            0.008375077126777163,
+            -0.020260917110786147,
+            0.002199401871514877,
+            0.0026191373298767766,
+            -0.0022089262577237776,
+            -0.015213807046175918,
+            -0.01956788452420682,
+            -0.021898368819136302,
+            0.012634659463647929,
+            0.00034643102239488474,
+            -0.009241772775582378,
+            -0.013645250997151429,
+            0.0016665106990117016,
+            0.0025630500346914592,
+            0.0002095521560180936,
+            -0.007783022628694597,
+            0.009546147813008391,
+            -0.01265287317178631,
+            0.007445223311736028,
+            -0.004007029181436962,
+            0.0051846833906574515,
+            -0.0016916014415925766,
+            -0.016413938859402438,
+            -0.0210868129634149,
+            0.01920794362224444,
+            -0.012897042227841011,
+            0.02269303737601583,
+            -0.0014935878147854765,
+            3.062680855559304e-05,
+            0.0033132626755401546,
+            0.007274487651476732,
+            0.012110042834251237,
+            0.011426424896797977,
+            0.004307086698938411,
+            0.008206932254416481,
+            0.004673427176334148,
+            -0.013124571552764586,
+            0.004324963759928715,
+            0.0039216033705804084,
+            -0.010449887922072297,
+            0.005298323150690876,
+            -0.0010196552573491,
+            0.006128327172621373,
+            0.012167492341907742,
+            0.01923525204293908,
+            0.01908107550907106,
+            -0.0012031208519043073,
+            -0.0010601912677170903,
+            0.014474374858125607,
+            0.004540954805526089,
+            -0.004883537463424679,
+            -0.002772988629590886,
+            -0.013363713078069854,
+            -0.002572403646338979,
+            0.011541515690646252,
+            -0.02376161747414617,
+            -0.018215231053002656,
+            0.021831219299143226,
+            -0.004991882316041097,
+            0.033796174834149546,
+            -0.013140970283091536,
+            -0.016197177724633614,
+            -0.00464190076219332,
+            0.0026389653040644607,
+            -0.013788049032378498,
+            -0.010300859744205445,
+            0.010545541278382187,
+            0.013848738866656793,
+            0.009903590151120531,
+            -0.0005984085581408575,
+            0.02922816576443645,
+            0.0019900932217491907,
+            0.02245395592422119,
+            0.0018978407151245295,
+            -0.007029294098689854,
+            -0.01440575120349887,
+            0.010840472832356266,
+            0.005529335652955532,
+            0.009050748012161991,
+            0.017791269808484802,
+            0.014502057454389247,
+            0.006265567405788131,
+            0.013928705812292101,
+            0.0012903002372833693,
+            0.0010858377120770652,
+            -0.002355229463060164,
+            -0.01692711767136341,
+            -0.01456786846097011,
+            -0.01627640642218236,
+            0.003873864312554098,
+            0.012620350492241247,
+            0.00525890570023481,
+            0.0007393789113866752,
+            -0.007153210751899716,
+            0.009254566395058412,
+            0.0065087881512835375,
+            -0.012279152412209152,
+            -0.016530068283509593,
+            0.004060671955726967,
+            -0.02210570272789274,
+            0.007849467899009437,
+            0.01554233481902349,
+            0.00219885032769439,
+            -0.002655449524219968,
+            0.0006843524304842362,
+            0.006272944169764281,
+            0.005531927368102191,
+            -0.0036710328198476905,
+            -0.00544113514775027,
+            0.011264661763762068,
+            -0.006379654234440906,
+            0.009537150372938296,
+            -0.00290031548037575,
+            0.03718190949929003,
+            -0.011976080887998194,
+            0.012120408392610773,
+            -0.001751472085435711,
+            0.008481703962951994,
+            0.008067669857949423,
+            0.010772687289903202,
+            -0.003805089040741351,
+            -0.0022164543131735167,
+            -0.008753625032781523,
+            0.006751953442322185,
+            -0.01582177621255226,
+            0.013631301238130705,
+            -0.00030685726238985963,
+            -0.010728958775974582,
+            0.01055472059121978,
+            0.006788316621149835,
+            0.004413382316882087,
+            -0.003262024961860459,
+            -0.01972283203505511,
+            0.014937729522976615,
+            0.0018799533869990667,
+            0.019247008681353865,
+            -0.023276106189004046,
+            0.029043727846427652,
+            -0.02019024213343764,
+            0.023083643395685142,
+            0.01865875504942157,
+            0.001179436167200093,
+            -0.02070329911663634,
+            -0.007618131527214554,
+            0.014871709122115308,
+            -0.02659818889735726,
+            -0.006234719259825804,
+            0.005986829551040766,
+            -0.014639912579831785,
+            0.00037813423546252645,
+            -0.01924149909676064,
+            0.00010832498610532006,
+            0.011013130149723431,
+            -0.011782407678373089,
+            0.00438725832178584,
+            0.01608349750921335,
+            0.019347883652009713,
+            -0.009760017612545174,
+            0.003869246694835168,
+            -0.024240324890794748,
+            0.010958789926998691,
+            0.020511684636649225,
+            0.008883830392748497,
+            -0.01761944522796127,
+            0.021817973960311446,
+            0.010652131228027521,
+            -0.012575679152857703,
+            -0.0012753095453828826,
+            -0.028191274775392804,
+            -0.012421170815054556,
+            -0.02188248722244861,
+            -0.0011670175621882146,
+            -0.014436683528976771,
+            -0.0029783880907228383,
+            0.0009506986717516226,
+            -0.004719165363484719,
+            -0.01179163390107033,
+            -0.019543732196676634,
+            0.016606805875498898,
+            0.0227313413004497,
+            -0.006738735201626204,
+            -0.014547240981002825,
+            0.013866974327185123,
+            0.031694077725890536,
+            -0.016189209297529324,
+            -0.00542110554672736,
+            0.0028061798782555557,
+            -0.012964876141500577,
+            0.018946046750284537,
+            0.0039419760434645495,
+            0.000322115192590907,
+            -0.01991851369210666,
+            0.002489725351392778,
+            0.010835458511052207,
+            0.01684964045708685,
+            0.0005541314524621196,
+            0.010855852688653808,
+            -0.007549882748861744,
+            0.010096800951420057,
+            -0.011832260836468588,
+            -3.6179995603517093e-06,
+            -0.007800366796516945,
+            0.007001049318404058,
+            -0.004424776169203575,
+            0.034356203288485945,
+            -0.02817071276766299,
+            0.004989631403681216,
+            -0.014539644354369607,
+            -0.010888763939716889,
+            -0.0027322086509797843,
+            0.0004865534340981866,
+            -0.010775238981581461,
+            -0.005162220412866082,
+            -0.0011263271110752205,
+            0.00913081242511139,
+            0.005844327320514489,
+            -0.013886529519306775,
+            0.019171827605726503,
+            -0.030984919318656618,
+            0.004469458202753611,
+            -0.008553175891755315,
+            0.010511707988936336,
+            -0.0151185694875566,
+            0.013830711686954931,
+            0.01301959345025159,
+            0.008205984085944315,
+            0.007740367849216328,
+            -0.01532123052721834,
+            -0.01678314222392063,
+            0.001376468747213867,
+            0.0036953018347776244,
+            -0.00931855481896886,
+            -0.033391028922085425,
+            -0.007072913446166969,
+            0.005331350534425606,
+            0.011614453225043,
+            0.008156128603003278,
+            -0.02345206873048703,
+            0.005003870024809975,
+            -0.024178720646704597,
+            -0.012115112684614331,
+            -0.007750557665424973,
+            -0.011580152272614108,
+            -0.006117298954192534,
+            0.026449765548206453,
+            0.007989237291215073,
+            0.016691517078382376,
+            0.004954133565782356,
+            0.02742809127725913,
+            0.011670893621991404,
+            -0.011736840457474895,
+            -0.003349151747880166,
+            0.00035419034843196817,
+            0.005507651806218584,
+            -0.008706590052024251,
+            0.011793241980257321,
+            -0.0013295035978611142,
+            -0.0065061682758755625,
+            -0.004398354347597199,
+            0.009372816666628725,
+            0.014188676762501037,
+            -0.003007980559117429,
+            -0.008724255229333351,
+            0.015749776067989826,
+            -0.030899793617096834,
+            -0.015215033379100091,
+            0.0016152308157299517,
+            -0.007813175908301852,
+            0.017806770680528907,
+            0.01169056345888971,
+            -0.026835412610223673,
+            -0.004538235170332766,
+            0.010532316940606419,
+            0.004755930024691789,
+            0.0095408889025506,
+            0.00954782703036282,
+            0.012079064562163901,
+            -0.007829549014214643,
+            0.01046722658021748,
+            -0.011519369840999348,
+            -0.007092961453184232,
+            -0.007241799314569802,
+            -0.008077934089061052,
+            0.021467547268987214,
+            -0.025816030568238717,
+            -0.006558853543772633,
+            0.01134670818701549,
+            0.013027491377727365,
+            -0.002301991242932548,
+            -0.006299073927206214,
+            -0.017357989759336386,
+            0.008311464809363397,
+            -0.013922568511594478,
+            0.025633907424998367,
+            -0.004854655498817219,
+            -0.005336807899100369,
+            -0.005228056880188717,
+            0.01958682721272445,
+            -0.016774409737540904,
+            0.02072573220949266,
+            -0.014449250679704904,
+            -0.007857351393427215,
+            0.001673929749689194,
+            0.006890008607998747,
+            -0.002948835017234416,
+            0.03734128725793149,
+            -0.00556474133167756,
+            -0.0014544201137417158,
+            0.02209084868209573,
+            -0.01883610080898967,
+            0.017901421953179442,
+            0.01563604483998435,
+            0.012327159100148019,
+            0.0018161034827768648,
+            -0.010253034391420751,
+            0.019979728662282904,
+            0.020823925491027635,
+            0.0139490143219451,
+            0.009577521724445478,
+            -0.013478761809038993,
+            0.014934933784421556,
+            -0.0003725422088504494,
+            -0.0024362115214204105,
+            0.012785534912074068,
+            0.016441944143888537,
+            -0.02164522371903646,
+            -0.013152102018736346,
+            0.0063548333449661805,
+            -0.016547629104776748,
+            -0.024875443975589356,
+            -0.008991095813132973,
+            -0.021885127280977718,
+            -0.023336714382596175,
+            0.00688008599534903,
+            0.016482610587564443,
+            0.0035797836729533367,
+            0.013111012464332918,
+            0.006820471349350837,
+            0.018274678506489214,
+            -0.00980248993077149,
+            -0.01582838461385605,
+            -0.0009454586621769173,
+            -0.0004906493597104733,
+            0.009787688665732045,
+            -0.01835583040075359,
+            -0.013977939077760832,
+            0.004340902272034388,
+            -0.020118484730757877,
+            0.01421736173162507,
+            -0.006428386730058143,
+            0.016756376129345597,
+            -0.0004630315831872392,
+            0.011701948921076007,
+            0.02621773278421472,
+            -0.001391567406453007,
+            0.01262026995055899,
+            0.004696906079976175,
+            -0.0030577364576645635,
+            -0.010624066224426196,
+            0.0009420498021845824,
+            -0.0044079288776429516,
+            -0.0004845606666642234,
+            0.006481113327122954,
+            0.020269778435314,
+            -0.03321232649491074,
+            -0.009425498978427855,
+            0.01246754422779043,
+            -0.001658574527323383,
+            -0.005816985985181387,
+            0.013050062223188965,
+            0.00011195028631856881,
+            0.016573827662919836,
+            -0.0017453203044868104,
+            0.01417371165305189,
+            -0.008165910519539772,
+            -0.0007461950113024975,
+            0.0016492309267666983,
+            0.005076015595439873,
+            0.005829299318055953,
+            0.012146800191035666,
+            0.01148884639506138,
+            -0.021051327863689618,
+            -0.0026357096198729156,
+            0.0036146148489310443,
+            -0.010673282438301611,
+            0.014967444393862347,
+            0.005737933867764028,
+            0.01391411712868596,
+            0.0029556189893790237,
+            0.01167806258173173,
+            -0.024440974650111174,
+            -0.01029944007931845,
+            0.01385509328958837,
+            0.00869671728965227,
+            -0.00998315356917989,
+            -0.016580053751364735,
+            -0.007061005072623325,
+            -0.00477175200214453,
+            -0.0021182980051220428,
+            -0.009871524928312118,
+            -0.0005933941177237234,
+            0.006284920058476413,
+            5.466820734762947e-05,
+            -0.012842295882002122,
+            0.015810530885294202,
+            0.004572385791133295,
+            0.003685966629168175,
+            0.005093714133656017,
+            0.011877541830396259,
+            -0.013130170365831912,
+            0.01258190257102377,
+            0.011433121961378686,
+            0.01142665981860032,
+            -0.013710025169815697,
+            -0.001774030746825879,
+            -0.01109881340948764,
+            0.0065684584345163735,
+            0.009983753894796107,
+            0.005346443477816739,
+            -0.002062387245572686,
+            0.004950044847352647,
+            0.006105199826576752,
+            -0.06391185159544624,
+            -0.02131511418723256,
+            0.0027165737784844413,
+            0.033389688791557785,
+            -0.007167593552251769,
+            -0.0037646833460996797,
+            0.002883510089742145,
+            0.01356887065340832,
+            0.009730171586767032,
+            0.00418566245984083,
+            0.020320934773473808,
+            -0.008814826860651366,
+            -0.003014193747732543,
+            0.001559573791453607,
+            0.003499420226345028,
+            -0.008703837291821111,
+            0.0044844701302276645,
+            0.0053963630238108035,
+            0.027764369548110287,
+            0.001469968540767627,
+            0.015793086297019008,
+            -0.008505290826588508,
+            -0.004075888558568453,
+            -0.009166710762668385,
+            -0.002860047211143503,
+            -0.011127820563172684,
+            0.003937110717404271,
+            0.02054186721680431,
+            -0.006126180963243227,
+            0.009346433819831456,
+            0.01057504086030076,
+            0.0001650851447316893,
+            -0.012033423322524951,
+            0.0049587335923558,
+            -0.0032240390628760535,
+            -0.005568976687946761,
+            -0.0024667040429312737,
+            -0.03474027725033016,
+            0.026855947684394643,
+            -0.004369626674503285,
+            0.009252512430957932,
+            -0.005081834388513357,
+            -0.0037896610731020337,
+            -0.014201609270756739,
+            0.017871088029615106,
+            -0.008249444723433136,
+            -0.020092257756971717,
+            0.009532994544485572,
+            -0.004991989149115383,
+            0.006464417738670472,
+            -0.02643078108339741,
+            0.0029138922065812495,
+            -0.017232410749655366,
+            -0.003999293090834831,
+            -0.006924488971625995,
+            0.01318082244178942,
+            -0.007232695119856389,
+            -0.02660711037908311,
+            0.01979251106803667,
+            -0.019485860712345603,
+            -0.011977904020170568,
+            -0.0177530973032872,
+            -0.014524003566945937,
+            0.004874952064727108,
+            -0.015326044325348929,
+            0.002047771741041311,
+            -0.009697340332342246,
+            -0.003652304998155628,
+            -0.0008878698629674222,
+            0.0017782435347417623,
+            -0.013017776298713044,
+            -0.018272697230002275,
+            -0.01353020425912225,
+            -0.00757239449706798,
+            -0.0008250000619957538,
+            0.01299142281772714,
+            -0.001679606555140765,
+            -0.007329941421303076,
+            0.0044605462460153116,
+            0.010231192277666706,
+            0.0006558644644452857,
+            0.0024231733856020414,
+            -0.010728844680699885,
+            -0.014137434553169443,
+            0.005230053181525096,
+            -0.012086468475542818,
+            -0.009459717916037003,
+            -0.0028185227600210556,
+            0.014984527039312847,
+            0.008189464334213754,
+            -0.009139240774931914,
+            -0.005639157579163322,
+            0.014325949891859377,
+            0.009020303519690971,
+            -0.013310205617445609,
+            0.01567574314514385,
+            -0.025369802629268523,
+            0.021964786021408878,
+            0.030536905527039065,
+            0.004945288795196981,
+            -0.01168303908203322,
+            0.02281128628230144,
+            -0.004072286832145419,
+            0.0028278384348551146,
+            0.01740522162288968,
+            -0.042558465579514926,
+            -0.013146562547084278,
+            0.002903743316339479,
+            -0.015297177099602701,
+            -0.010963312697387696,
+            0.008804938803398296,
+            -0.0026727734429658817,
+            0.005273238046953793,
+            0.004257613402172282,
+            0.03512246556126453,
+            0.002258612643297096,
+            -0.0002885708728599208,
+            -0.0019720190285930487,
+            0.012312613129996533,
+            0.010451003939611476,
+            -0.015556620910136185,
+            -0.02387856865043814,
+            -0.004583090523619507,
+            -0.0037828776396525835,
+            -0.0014588238692793338,
+            -0.004361834167147271,
+            -0.008159679676329573,
+            0.012243319920992457,
+            -0.017279964797769668,
+            -0.013748032758693542,
+            0.018394195628121487,
+            -0.006411113522842359,
+            0.019002611495040334,
+            -0.014970362556522149,
+            -0.0077437581301661946,
+            0.0017311115715410025,
+            0.024439691379769795,
+            -0.025013956798520212,
+            0.007359134924497539,
+            -0.0031406470488477937,
+            0.017574824561755627,
+            0.00419409250132525,
+            -0.0046803792187546656,
+            0.010556972371044426,
+            0.015229789583480183,
+            -0.004608293492221483,
+            -0.014449487985966966,
+            -0.01172850291078532,
+            0.016823024209386785,
+            -0.012304887597699588,
+            0.0011773351576246157,
+            0.002262670226244277,
+            -0.0023810578015456716,
+            0.03477119000959688,
+            -0.03442178291363675,
+            0.004062693040714608,
+            0.0029249583507755944,
+            0.003291349443702718,
+            0.006461525089528308,
+            0.006047777261425826,
+            -0.020642802018978404,
+            0.008916088381876748,
+            -0.03316762661930299,
+            -0.006367254689957291,
+            -0.005404276207844267,
+            0.0009189006423526284,
+            0.009401330264376041,
+            -0.011106443887384533,
+            -0.02259405372588916,
+            -0.006010439054323966,
+            0.0070744551115721095,
+            0.010262607009446879,
+            -0.002856616377476421,
+            0.019719451109535182,
+            0.008426509638139982,
+            0.0015877648339862691,
+            -0.010655792508751512,
+            -0.015527851374834499,
+            -0.015460974859638219,
+            -0.017847462632188386,
+            -0.02236964311096428,
+            0.011841749854626384,
+            -0.00576688153869993,
+            0.009054902130717224,
+            0.014317707346504314,
+            -0.015366657284925976,
+            0.002910211008094564,
+            0.017761685988205086,
+            0.007974806171830457,
+            0.002692237107055498,
+            0.0032405477366987947,
+            0.02685230091638845,
+            -0.0009116884421301454,
+            -0.0032228977180251673,
+            0.003158213042682977,
+            0.015872199488167846,
+            0.0004966905858799325,
+            0.019981401695749582,
+            0.015201538351780685,
+            0.005359075814444414,
+            -0.004668738193285921,
+            0.019853988551405383,
+            -0.009844394286458195,
+            0.013476144827747423,
+            -0.00411789677405284,
+            -0.00830656434595337,
+            -0.008496132520586954,
+            0.006442581751999304,
+            -0.008138786054348571,
+            0.006977337946948795,
+            0.02419128791801387,
+            -0.0005734827808485311,
+            -0.0028842142002271044,
+            0.005503033191779872,
+            -0.01696639779775455,
+            -0.00389835761601045,
+            -0.03366077608692946,
+            -0.009942613075976742,
+            -0.01708630689497001,
+            -0.02910756562213045,
+            0.026536268897199546,
+            0.00219827495662445,
+            0.02586503862171508,
+            -0.01905122645609761,
+            0.008935637454859214,
+            -0.01385806889332683,
+            0.029747093931541325,
+            0.009163959012862627,
+            0.0008611776492791918,
+            0.003598831885031635,
+            -0.02103045281939351,
+            -0.01150941825152158,
+            0.027341774725604333,
+            -0.0006700608710615124,
+            -0.012089044078073347,
+            0.022396400579038037,
+            0.017491565611764764,
+            0.0051760367056280536,
+            -0.00559536785318233,
+            0.01979262595019399,
+            0.002306380546194741,
+            0.011973863166872444,
+            -0.0028136750606972223,
+            0.012911235736067303,
+            -0.01510421118359169,
+            -0.003338602157155174,
+            0.039758808572572274,
+            0.007290642519242007,
+            -0.001802921446521437,
+            -0.013696774990705774,
+            0.003941198181491921,
+            -0.0008308293948698925,
+            0.0200351513103389,
+            -0.00014327587644797734,
+            0.028291423013030337,
+            -0.0001085859276231927,
+            -0.010723064585989723,
+            0.020443848393732245,
+            -0.017860296930396913,
+            0.04540901629084378,
+            -0.018671578423261854,
+            0.000852031295629291,
+            -0.010639853151638835,
+            -0.019017194262730474,
+            0.003861296738239369,
+            0.0010977143012920602,
+            0.0008387482016061222,
+            0.0060155816314618,
+            -0.010780239332444652,
+            0.002152214497259421,
+            0.010556338812873054,
+            -0.027636192450621728,
+            0.0010938992111792307,
+            -0.014385202899929138,
+            -0.01353192238925814,
+            -0.026667778226211785,
+            -0.00017403398236549974,
+            -0.015357674131635848,
+            -0.013956722505869091,
+            -0.01985038526848704,
+            -0.008927903879720431,
+            -0.009580214158077466,
+            -0.0020314441852913386,
+            -0.004755584878514854,
+            -0.00018388548107372635,
+            -0.002918855567842974,
+            -0.0015941852262316128,
+            -0.014230272061190995,
+            0.02133992212037431,
+            0.015283345362445234,
+            0.004642765060980717,
+            0.012829570028541471,
+            0.01708952142208596,
+            -0.004770647543823141,
+            0.006604762788847952,
+            -0.0025369836845081078,
+            0.008368000846861623,
+            0.014785031726639733,
+            0.012074738887486398,
+            0.02124518119032885,
+            -0.015459818988572564,
+            -0.004992865135357168,
+            -0.012722391036055767,
+            -0.012210202603282509,
+            -0.017850065946244383,
+            -0.004300429148015694,
+            -0.0014387298144714717,
+            -0.023128300275416423,
+            0.004468952290723408,
+            -0.0026602931688877443,
+            -0.005768852559887139,
+            0.007815212168719318,
+            0.0005862972228454734,
+            -0.002265719947629544,
+            -0.00019323737674204774,
+            -0.02735820445266249,
+            0.00024009122317215594,
+            -0.00018701993387558173,
+            -0.014663710240203917,
+            -0.00450797561624596,
+            0.010584731989847334,
+            -0.011966527748437568,
+            -0.00602367292291343,
+            0.0111155226518017,
+            0.0015565179432852256,
+            0.006062862961835896,
+            0.016952559697728433,
+            0.0005012157272316869,
+            0.011160668242334889,
+            0.014993782560797772,
+            -0.01591303023837098,
+            -0.015356412848444082,
+            -0.0236025426434855,
+            -0.004514538154596937,
+            -0.009753334017065119,
+            0.007553951535816575,
+            -0.004616595532996391,
+            -0.013023153809236325,
+            0.006040285197946957,
+            -0.0006343451398642784,
+            0.008338564073158257,
+            -0.017608246105292328,
+            -8.346168119689692e-05,
+            -0.020944870308705715,
+            0.009230123321316848,
+            -0.00531237224627929,
+            -0.00445985089300067,
+            0.01410675485522911,
+            0.02895945580213926,
+            -0.007799830351886844,
+            -0.009455128322575888,
+            -0.005663072244883114,
+            0.012817229945992764,
+            0.012474431030896143,
+            -0.004423141749110039,
+            -0.0018294512146253298,
+            -0.012817101093394574,
+            0.0004755435377918658,
+            -0.01129485262081729,
+            0.0062747111125005155,
+            0.017199374798326412,
+            -0.021965117839794063,
+            -0.01296097993683748,
+            -0.016765996091978336,
+            -0.016882855472266523,
+            0.004372970343291638,
+            0.017290565252244258,
+            0.022670788364378462,
+            -0.008456547449078897,
+            -0.019564049850942417,
+            0.006898353991539946,
+            -0.01620125883985687,
+            0.0037108638122592746,
+            0.12887455743463638,
+            -0.0028864852421660366,
+            -0.011487804449328495,
+            -0.022683334123299446,
+            -0.005467192611880829,
+            0.01210557696712196,
+            -0.01555159294496261,
+            0.06709672179195036,
+            -0.015071089552631105,
+            0.004364902817496082,
+            -0.004899872946473657,
+            0.0006310685609937571,
+            -0.017610935588191347,
+            -0.005962991715597585,
+            -0.007768819050529025,
+            -0.02688607975562596,
+            0.000883763319535686,
+            -0.009935313912803587,
+            -0.012883104064962595,
+            0.002126368813790577,
+            -0.0013697265269800346,
+            -0.016131186838419288,
+            0.01575514418037863,
+            -0.020677990768697592,
+            -0.007251917191831732,
+            -0.0029325623780992118,
+            -0.00600710212646933,
+            0.014573093764769918,
+            -0.03253336117394778,
+            0.002973725493330224,
+            -0.013601417220439602,
+            0.011905788192118131,
+            0.001542117126832946,
+            -0.010789486980776022,
+            -0.013862139885514524,
+            0.007086571810314664,
+            0.015658913897173155,
+            0.017039624764452546,
+            0.01901374401756777,
+            0.004797954569087795,
+            0.003928999581093576,
+            -0.008046722893614655,
+            0.021974371139905104,
+            -0.00697335955527197,
+            0.0008975473383295958,
+            -0.00970589295353857,
+            -0.004346869073499034,
+            0.0031279458769925416,
+            0.02303983493541673,
+            -0.012560380384977343,
+            -0.01536378830925542,
+            0.0012035987157572355,
+            0.033313215063595594,
+            -0.010845084209743817,
+            0.013587126023659577,
+            0.0029867620644300073,
+            0.020679673435988446,
+            -0.0006640787253300063,
+            0.0025071780490108306,
+            0.022554193713122565,
+            -0.018082906254599396,
+            0.018230000220500718,
+            -0.004451773302171552,
+            0.01915083025790679,
+            -0.004183521043835777,
+            0.015019320101074747,
+            -0.027014055979204272,
+            -0.005362044589023088,
+            -0.012355559881051253,
+            0.003995309003198048,
+            0.020680455787871383,
+            0.0054391755324872375,
+            0.009865676762244514,
+            -0.01645386866657314,
+            -0.01760657627715638,
+            0.022969828924328126,
+            -0.021566687194482292,
+            -0.015826424849757402,
+            0.00012804774676298237,
+            -0.015021970627042762,
+            0.006000610438452896,
+            0.010126179023323158,
+            -0.02912121211344025,
+            -0.02046150879033127,
+            0.004881850039543051,
+            -0.0023469840669773042,
+            -0.00915786433296249,
+            0.006547377770403066,
+            -0.022329063637464074,
+            -0.015702777677743352,
+            -0.005692904224111613,
+            0.01207999389934878,
+            -0.008429160038192635,
+            0.010075061963131637,
+            -0.020106712664936504,
+            0.0036787045650556907,
+            -0.002764965946892891,
+            0.010307702251974629,
+            0.004697169247387421,
+            0.011192285826515207,
+            0.0030113901179382946,
+            0.0014774681536380047,
+            -0.02778270000997313,
+            0.010043261430740039,
+            -5.178406450222847e-05,
+            0.04216636255860709,
+            0.030977096295679525,
+            0.004479857028809617,
+            -5.798020745008712e-05,
+            -0.009996471945562616,
+            0.012476009017187798,
+            -0.019306907391548302,
+            -0.005092322343599383,
+            0.01856463597985508,
+            0.02195467141594795,
+            -0.02236232264139305,
+            0.0052217593371508145,
+            -0.016766955100332014,
+            0.016133299654723637,
+            -0.011208763137897244,
+            0.006248024409576809,
+            0.011601988603146743,
+            0.0022330878390492527,
+            0.010317781892717903,
+            -0.005895142237225119,
+            0.019670196605183973,
+            0.01186677200787666,
+            -0.002339367025714548,
+            0.02367408853110795,
+            0.006858134385159451,
+            0.004479683673962342,
+            0.004798719889930319,
+            -0.009604173975576537,
+            -0.009045602998213713,
+            0.009230969488015385,
+            0.004083616606774546,
+            -0.007944628589259331,
+            0.011836323287097144,
+            0.015100350195476968,
+            0.01885317701149666,
+            -0.0011707847499298092,
+            -0.00684548433712413,
+            -0.006550059418141426,
+            0.001899758393077906,
+            -0.024577381680573596,
+            -0.02855511854789239,
+            0.021238935710272536,
+            -0.010427135883958709,
+            -0.019187769881105637,
+            -0.0015864640188683347,
+            0.016360037306061392,
+            -0.008853972319021563,
+            -0.01627167079143546,
+            0.001270414822216867,
+            0.02255755055961612,
+            -0.005959411125391464,
+            0.006009654905468478,
+            0.005909233773418581,
+            0.005874361519568704,
+            -0.0027859463941992753,
+            -0.024599789577567636,
+            0.0036612374968223453,
+            0.03221346865585554,
+            0.016461563205582293,
+            0.016139782455492073,
+            -0.007158168059808459,
+            0.001838455290310879,
+            0.007530954381521755,
+            0.005438791523412295,
+            0.015674000912615536,
+            0.008604106101542998,
+            -0.006990462920227135,
+            0.00244736628387359,
+            0.012213048255418866,
+            -0.007329299946651028,
+            0.017424711328323798,
+            0.001973882532418776,
+            -0.0133178791530314,
+            -0.008373452013951822,
+            -0.010462150707776556,
+            -0.014575483571750984,
+            0.0027958704799249897,
+            -0.011401916623878957,
+            0.0008016041012669959,
+            0.003953958717263787,
+            0.010260753041492368,
+            0.013961649189650642,
+            -0.0019275791443041316,
+            0.017019168629298462,
+            0.011534594243860177,
+            -0.0017859096307142257,
+            -0.002970739203219076,
+            -0.03142778833370656,
+            0.010207789755749909,
+            -0.014730553506704018,
+            -0.0037611416875830746,
+            0.01825790347981639,
+            -0.007766890847082858,
+            -0.0032778433418881666,
+            -0.021003405429807113,
+            0.0024512381432602736,
+            0.008768616692363321,
+            0.01558713989712387,
+            -0.012819542714293536,
+            0.009201155584074711,
+            -0.0012254337002053017,
+            -0.0057451574939563045,
+            0.00977251392462179,
+            0.0042356831792011935,
+            -0.0033433823868435786,
+            0.011499764125503874
+          ],
+          "title": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer",
+          "keyphrases": [
+            "Sparsely-Gated Mixture-of-Experts Layer",
+            "conditional computation",
+            "neural network",
+            "language modeling",
+            "machine translation"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "a004def7-e6ff-4a78-ad92-424e0eff3d92",
+        "properties": {
+          "page_content": "6 Conclusion\n\nThis work is the first to demonstrate major wins from conditional computation in deep networks.\n\nWe carefully identified the design considerations and challenges of conditional computing and addressed them with a combination of algorithmic and engineering solutions. While we focused on text, conditional computation may help in other domains as well, provided sufficiently large training sets. We look forward to seeing many novel implementations and applications of conditional computation in the years to come.\n\nAcknowledgments\n\nWe would like to thank all of the members of the Google Brain and Google Translate teams who helped us with this project, in particular Zhifeng Chen, Yonghui Wu, and Melvin Johnson. Thanks also to our anonymous ICLR reviewers for the helpful suggestions on making this paper better.\n\n2Reported perplexities relative to the tokenization used by both our models and GNMT.\n\nReferences\n\nMartín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Gordon Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda B. Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. CoRR, abs/1603.04467, 2016. URL http://arxiv.org/abs/1603.04467.\n\nRahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. CoRR, abs/1611.06194, 2016. URL http://arxiv.org/abs/1611.\n\n06194.\n\nA. Almahairi, N. Ballas, T. Cooijmans, Y. Zheng, H. Larochelle, and A. Courville. Dynamic Capacity Networks. ArXiv e-prints, November 2015.\n\nDario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan, Christopher Fougner, Tony Han, Awni Y. Hannun, Billy Jun, Patrick LeGresley, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595, 2015. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.\n\nEmmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297, 2015.\n\nYoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.\n\nCiprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling.\n\narXiv preprint arXiv:1312.3005, 2013.\n\nK. Cho and Y. Bengio. Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning. ArXiv e-prints, June 2014. Ronan Collobert, Samy Bengio, and Yoshua Bengio. A parallel mixture of SVMs for very large scale problems. Neural Computing, 2002.\n\nAndrew Davis and Itamar Arel. Low-rank approximations for conditional feedforward computation in deep neural networks. arXiv preprint arXiv:1312.4461, 2013. Marc Peter Deisenroth and Jun Wei Ng. Distributed Gaussian processes. In ICML, 2015.\n\nJohn Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization, 2010.\n\nNadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. Edinburgh's phrase-based machine translation systems for wmt-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation, 2014.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013. Ekaterina Garmash and Christof Monz. Ensemble learning for multi-source neural machine translation. In staff.science.uva.nl/c.monz, 2016.\n\nFelix A. Gers, Jürgen A. Schmidhuber, and Fred A. Cummins. Learning to forget: Continual prediction with lstm. Neural Computation, 2000.\n\nAudrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. Memory-efficient backpropagation through time. CoRR, abs/1606.03401, 2016. URL http://arxiv.org/ abs/1606.03401. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2015.\n\nGeoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 2012. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.\n\nSergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.\n\nRobert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computing, 1991.\n\nMelvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation.\n\nCoRR, abs/1611.04558, 2016. URL http://arxiv.org/abs/1611.04558.\n\nMichael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm.\n\nNeural Computing, 1994. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.\n\nDiederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.\n\nReinhard Kneser and Hermann. Ney. Improved backingoff for m-gram language modeling., 1995.\n\nAlex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.\n\nQuoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeffrey Dean, and Andrew Y. Ng. Building high-level features using large scale unsupervised learning. In ICML, 2012. Patrick Gallinari Ludovic Denoyer. Deep sequential neural network. arXiv preprint arXiv:1410.0510, 2014.\n\nMinh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attentionbased neural machine translation. EMNLP, 2015a.\n\nMinh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the rare word problem in neural machine translation. ACL, 2015b.\n\nCarl Edward Rasmussen and Zoubin Ghahramani. Infinite mixtures of Gaussian process experts.\n\nNIPS, 2002.\n\nHasim Sak, Andrew W Senior, and Françoise Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH, pp. 338–342, 2014.\n\nMike Schuster and Kaisuke Nakajima. Japanese and Korean voice search. ICASSP, 2012.\n\nBabak Shahbaba and Radford Neal. Nonlinear models using dirichlet process mixtures. JMLR, 2009.\n\nIlya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks.\n\nIn NIPS, 2014.\n\nLucas Theis and Matthias Bethge. Generative image modeling using spatial LSTMs. In NIPS, 2015. Volker Tresp. Mixtures of Gaussian Processes. In NIPS, 2001.\n\nYonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.\n\nBangpeng Yao, Dirk Walther, Diane Beck, and Li Fei-fei. Hierarchical mixture of classification experts uncovers interactions between brain regions. In NIPS. 2009.\n\nWojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization.\n\narXiv preprint arXiv:1409.2329, 2014.\n\nJie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with fast-forward connections for neural machine translation. arXiv preprint arXiv:1606.04199, 2016.\n\n",
+          "embedding": [
+            0.012222016230225563,
+            0.0018153652781620622,
+            -0.018968041986227036,
+            -0.007018175441771746,
+            0.012403449974954128,
+            0.003484346903860569,
+            0.006766642909497023,
+            0.03404351323843002,
+            -0.030365360900759697,
+            0.03598979860544205,
+            0.007698550820350647,
+            -0.006844989024102688,
+            0.017252670601010323,
+            -0.018192825838923454,
+            0.0035915574990212917,
+            0.014209537766873837,
+            0.014687862247228622,
+            0.004014214966446161,
+            -0.011182897724211216,
+            -0.040674079209566116,
+            -0.03125603497028351,
+            -0.017648525536060333,
+            -0.03161890059709549,
+            -0.009582984261214733,
+            0.016172317788004875,
+            0.03310335800051689,
+            0.022712169215083122,
+            0.017153708264231682,
+            -0.045556288212537766,
+            0.015842437744140625,
+            0.028155172243714333,
+            0.003665780182927847,
+            -0.02747892029583454,
+            -0.023569853976368904,
+            -0.015380607917904854,
+            -0.03267451375722885,
+            0.06508512794971466,
+            0.03876078128814697,
+            -0.014654873870313168,
+            0.019149474799633026,
+            0.04321414604783058,
+            -0.011900384910404682,
+            -0.007348054554313421,
+            -0.015990883111953735,
+            -0.004210080485790968,
+            -0.002177201444283128,
+            -0.014085832983255386,
+            -0.015174433588981628,
+            -0.030051976442337036,
+            -0.007912972010672092,
+            0.028105689212679863,
+            -0.028353098779916763,
+            0.04743659868836403,
+            -0.003750311676412821,
+            -0.016485702246427536,
+            0.04671086370944977,
+            0.010762302204966545,
+            0.008997449651360512,
+            0.06310585141181946,
+            0.04139981418848038,
+            0.003208073088899255,
+            0.018192825838923454,
+            -0.023899732157588005,
+            -0.006424393504858017,
+            0.0322621650993824,
+            -0.02089783363044262,
+            0.0017978404648602009,
+            0.025433670729398727,
+            0.039783407002687454,
+            -0.015347619540989399,
+            -0.03783712163567543,
+            0.029045844450592995,
+            -0.010498398914933205,
+            -0.015957895666360855,
+            -0.00029766425723209977,
+            -0.0036699038464576006,
+            0.029029350727796555,
+            0.05367131158709526,
+            0.005014160647988319,
+            0.004304920788854361,
+            0.012782811187207699,
+            0.017219683155417442,
+            -0.004572947509586811,
+            -0.006494492758065462,
+            0.023157505318522453,
+            0.002239053603261709,
+            -0.005183223634958267,
+            -0.05383625254034996,
+            0.03328479081392288,
+            -0.04740361124277115,
+            -0.039816394448280334,
+            0.018225813284516335,
+            0.006473875138908625,
+            -0.01018501352518797,
+            -0.009624219499528408,
+            0.029656121507287025,
+            -0.00020127772586420178,
+            -0.000498168810736388,
+            -0.018918560817837715,
+            0.014357983134686947,
+            -0.023784274235367775,
+            0.00528631079941988,
+            0.013285876251757145,
+            -0.013665237464010715,
+            0.027577882632613182,
+            0.009541749954223633,
+            -0.013541532680392265,
+            0.004121425561606884,
+            0.01654343120753765,
+            0.09295989573001862,
+            -0.0011030328460037708,
+            -0.022530734539031982,
+            -0.05314350500702858,
+            0.03267451375722885,
+            -0.02602745220065117,
+            -0.019512342289090157,
+            -0.051791001111269,
+            -0.039882369339466095,
+            0.0015978513984009624,
+            -0.00780988484621048,
+            0.03721034899353981,
+            0.03533003851771355,
+            0.034241437911987305,
+            -0.011974607594311237,
+            -0.029342735186219215,
+            0.055914487689733505,
+            -0.002408116590231657,
+            -0.0425214022397995,
+            -0.0034740380942821503,
+            -0.023965708911418915,
+            -0.007587216794490814,
+            0.008593347854912281,
+            -0.040575116872787476,
+            0.003964733332395554,
+            0.03285594657063484,
+            -0.018638163805007935,
+            0.029342735186219215,
+            0.03582485765218735,
+            -0.025417177006602287,
+            -0.0004953339230269194,
+            0.011232379823923111,
+            -0.0033709509298205376,
+            0.016733111813664436,
+            -0.01296424400061369,
+            -0.034208450466394424,
+            -0.0008540773415006697,
+            -0.015990883111953735,
+            0.0541001558303833,
+            -0.009459280408918858,
+            0.030942648649215698,
+            -0.03885974362492561,
+            -0.0009102598414756358,
+            -0.014267266727983952,
+            0.034538328647613525,
+            0.029606638476252556,
+            -0.009046930819749832,
+            0.0019669034518301487,
+            0.009615972638130188,
+            -0.009541749954223633,
+            0.010201508179306984,
+            -0.00514198886230588,
+            -0.03315283730626106,
+            -0.007966578006744385,
+            0.03757321834564209,
+            0.009756171144545078,
+            0.04097096994519234,
+            0.04737062379717827,
+            0.008750040084123611,
+            -0.04882209002971649,
+            0.04700775444507599,
+            0.034439366310834885,
+            0.01601562462747097,
+            0.001114372513256967,
+            -0.018291790038347244,
+            0.01057262159883976,
+            0.01364049594849348,
+            0.033878572285175323,
+            0.009921110235154629,
+            -0.040179260075092316,
+            0.024790406227111816,
+            -0.0025256359949707985,
+            0.010119037702679634,
+            0.04885507747530937,
+            0.02596147730946541,
+            0.040179260075092316,
+            0.003092615632340312,
+            0.029161302372813225,
+            0.028204653412103653,
+            0.029260266572237015,
+            -0.007599587086588144,
+            0.008585100993514061,
+            0.02132667601108551,
+            0.045655250549316406,
+            -0.034769244492053986,
+            0.029903531074523926,
+            -0.039354562759399414,
+            0.0009916987037286162,
+            0.028649989515542984,
+            -0.024526502937078476,
+            0.004700775723904371,
+            0.00834593828767538,
+            -0.055947475135326385,
+            -0.009632466360926628,
+            0.005344039760529995,
+            0.00844490248709917,
+            0.007591340225189924,
+            -0.012667353264987469,
+            0.004255439154803753,
+            0.0053234221413731575,
+            -0.040674079209566116,
+            -0.006354294251650572,
+            0.004333785269409418,
+            -0.003735879436135292,
+            -0.017829960212111473,
+            0.012560142204165459,
+            -0.004383266903460026,
+            -0.003599804360419512,
+            -0.005364656914025545,
+            -0.025631597265601158,
+            -0.009005696512758732,
+            -0.01664239540696144,
+            0.020403016358613968,
+            0.04215853661298752,
+            -0.024856381118297577,
+            -0.016320763155817986,
+            -0.017928922548890114,
+            -0.03556095436215401,
+            -0.030117951333522797,
+            0.017566056922078133,
+            0.00014135829405859113,
+            0.03688047081232071,
+            0.02172253094613552,
+            -0.0031750851776450872,
+            0.004581194370985031,
+            0.02591199427843094,
+            -0.017269164323806763,
+            -0.0021524603944271803,
+            0.004577070940285921,
+            -0.007756279781460762,
+            0.0136487428098917,
+            -0.003177147125825286,
+            0.01680733449757099,
+            -0.00515848258510232,
+            -0.00132157769985497,
+            -0.001624654047191143,
+            -0.01583419181406498,
+            -0.017599044367671013,
+            0.01364049594849348,
+            -0.0008695403812453151,
+            0.010069556534290314,
+            -0.018935054540634155,
+            -0.04262036457657814,
+            0.013896152377128601,
+            -0.010003580711781979,
+            -0.05927925556898117,
+            0.06967044621706009,
+            0.02676968090236187,
+            -0.014621886424720287,
+            -0.00333796301856637,
+            -0.050471484661102295,
+            0.05802571401000023,
+            0.004453366156667471,
+            -0.02221735008060932,
+            0.05353936180472374,
+            -0.007117139175534248,
+            0.03829894959926605,
+            0.05241777002811432,
+            0.022827625274658203,
+            0.010044815018773079,
+            -0.02714904025197029,
+            0.05268167331814766,
+            -0.04205957055091858,
+            -0.00666767917573452,
+            -0.034670282155275345,
+            0.016881557181477547,
+            -0.04278530552983284,
+            -0.004498724825680256,
+            0.018324777483940125,
+            0.010877760127186775,
+            0.03592382371425629,
+            0.023619335144758224,
+            0.015908414497971535,
+            -0.03145396336913109,
+            -0.004395637661218643,
+            -0.00642026960849762,
+            -0.013879658654332161,
+            -0.03737528994679451,
+            -0.0037626822013407946,
+            -0.06696543842554092,
+            -0.004630676470696926,
+            -0.014811567030847073,
+            0.01346730999648571,
+            -0.05264868587255478,
+            0.029458193108439445,
+            0.018291790038347244,
+            0.020419510081410408,
+            0.007104768883436918,
+            -0.024180129170417786,
+            0.019281426444649696,
+            0.011446801014244556,
+            -0.017186695709824562,
+            -0.040113285183906555,
+            0.004412131384015083,
+            -0.01324464101344347,
+            -0.022745156660676003,
+            0.0018205196829512715,
+            0.019842220470309258,
+            0.029606638476252556,
+            0.005950192455202341,
+            0.004560577217489481,
+            -0.0032225053291767836,
+            0.017516573891043663,
+            0.03180033341050148,
+            -0.019693775102496147,
+            -0.03373012691736221,
+            0.040014322847127914,
+            0.025747055187821388,
+            0.003550322726368904,
+            -0.017120718955993652,
+            -0.03123953938484192,
+            0.051164232194423676,
+            -0.0006412022630684078,
+            -0.06333676725625992,
+            0.012329227291047573,
+            -0.006073896773159504,
+            0.002081330167129636,
+            -0.0012401387793943286,
+            0.05159307271242142,
+            0.002733872039243579,
+            0.014935270883142948,
+            0.0019091746071353555,
+            -0.050471484661102295,
+            -0.018522705882787704,
+            -0.007360424846410751,
+            -0.023091528564691544,
+            -0.039156634360551834,
+            -0.008061417378485203,
+            0.03816699981689453,
+            -0.006135749164968729,
+            -0.017401115968823433,
+            -0.03602278605103493,
+            -0.02089783363044262,
+            -0.009211870841681957,
+            -0.001520535908639431,
+            0.004915196914225817,
+            -0.030563287436962128,
+            -0.01624654047191143,
+            -0.0009813900105655193,
+            0.0069480761885643005,
+            0.001200965722091496,
+            0.0013545656111091375,
+            0.015339372679591179,
+            0.005059518851339817,
+            0.006865606643259525,
+            0.010613856837153435,
+            -0.018902065232396126,
+            -0.02114524319767952,
+            0.012180781923234463,
+            -0.007558352313935757,
+            0.012222016230225563,
+            -0.007113015744835138,
+            -0.0006963539053685963,
+            -0.029870541766285896,
+            0.007587216794490814,
+            -0.024411045014858246,
+            -0.03589083254337311,
+            -0.023701805621385574,
+            -0.0026101674884557724,
+            0.023157505318522453,
+            0.001592696993611753,
+            -0.003183332271873951,
+            0.03905767202377319,
+            -0.023982202634215355,
+            -0.011438554152846336,
+            -0.0059172045439481735,
+            -0.010754055343568325,
+            0.03849687799811363,
+            0.0035400139167904854,
+            -0.011158157140016556,
+            0.017928922548890114,
+            -0.025730561465024948,
+            -0.00416884571313858,
+            0.05838857963681221,
+            0.0018669088603928685,
+            0.0066429381258785725,
+            0.02089783363044262,
+            -0.0025297594256699085,
+            -0.0050224075093865395,
+            -0.02139265276491642,
+            0.017945416271686554,
+            0.0020297865848988295,
+            -0.040212247520685196,
+            -0.023239973932504654,
+            -0.039849381893873215,
+            -0.006725407671183348,
+            0.028715966269373894,
+            0.024724429473280907,
+            0.004436872433871031,
+            0.015199174173176289,
+            -0.005067766178399324,
+            -0.002641093684360385,
+            -0.018093861639499664,
+            0.003655471606180072,
+            0.009756171144545078,
+            -0.01081178430467844,
+            0.0016401171451434493,
+            -0.003805978922173381,
+            -0.0052945576608181,
+            0.04126786068081856,
+            0.004010091535747051,
+            -0.009079919196665287,
+            -0.018869077786803246,
+            -0.055287718772888184,
+            -0.034670282155275345,
+            0.034538328647613525,
+            0.014630133286118507,
+            -0.010622103698551655,
+            -0.01332711149007082,
+            0.0016164070693776011,
+            -0.04354402795433998,
+            -0.01296424400061369,
+            -0.009838640689849854,
+            0.02579653635621071,
+            0.014861048199236393,
+            -0.0028287123423069715,
+            0.020551461726427078,
+            0.022959576919674873,
+            0.013558026403188705,
+            -0.007141880225390196,
+            -0.03278997167944908,
+            0.011372578330338001,
+            0.006919211708009243,
+            0.04591915383934975,
+            0.008626335300505161,
+            -0.0047914921306073666,
+            -0.04921794682741165,
+            -0.024575984105467796,
+            0.018159838393330574,
+            -0.03819998726248741,
+            0.001192718744277954,
+            0.013541532680392265,
+            -0.01623004674911499,
+            0.0058677224442362785,
+            0.0003208588750567287,
+            -0.02205241098999977,
+            -0.0161310825496912,
+            -0.004461613483726978,
+            -0.034208450466394424,
+            -0.009929358027875423,
+            -0.04773348942399025,
+            -0.06749323755502701,
+            0.02655525878071785,
+            -0.040278226137161255,
+            -0.0648542121052742,
+            0.03592382371425629,
+            -0.03655059263110161,
+            0.006510986480861902,
+            -0.040278226137161255,
+            0.0021019477862864733,
+            -0.060268890112638474,
+            0.017961911857128143,
+            0.034175463020801544,
+            0.015710486099123955,
+            -0.0021132873371243477,
+            0.022398782894015312,
+            -0.02736346237361431,
+            -0.040443163365125656,
+            -0.06752622872591019,
+            -0.017780477181077003,
+            -0.01351679116487503,
+            -0.0031503443606197834,
+            0.002323585096746683,
+            0.05284661427140236,
+            0.0004986842395737767,
+            0.024674948304891586,
+            0.022547228261828423,
+            -0.0012978676240891218,
+            -0.04885507747530937,
+            -0.024724429473280907,
+            0.011611740104854107,
+            0.03876078128814697,
+            -0.030530299991369247,
+            0.051395148038864136,
+            -0.05258271098136902,
+            0.009665454737842083,
+            -0.009409798309206963,
+            0.006036785431206226,
+            0.019116487354040146,
+            0.007092398125678301,
+            -0.02203591726720333,
+            0.003894633846357465,
+            0.023404914885759354,
+            -0.011595246382057667,
+            0.03585784509778023,
+            -0.012692093849182129,
+            -0.034538328647613525,
+            -0.009442785754799843,
+            -0.007661439478397369,
+            0.03750723972916603,
+            0.011182897724211216,
+            -0.016172317788004875,
+            -0.028798434883356094,
+            0.011784926988184452,
+            0.02111225575208664,
+            0.03866181895136833,
+            -0.02229982055723667,
+            -0.022761650383472443,
+            0.009393304586410522,
+            -0.035692907869815826,
+            0.018027886748313904,
+            -0.017170201987028122,
+            0.005900710355490446,
+            -0.009442785754799843,
+            -0.045226410031318665,
+            -0.0008643860346637666,
+            -0.00010920798376901075,
+            -0.015891920775175095,
+            0.01045716367661953,
+            0.0061604902148246765,
+            0.018638163805007935,
+            0.04113591089844704,
+            -0.02221735008060932,
+            -0.002874070778489113,
+            0.0772576555609703,
+            -0.015537300147116184,
+            0.018819596618413925,
+            0.01659291237592697,
+            -0.01568574644625187,
+            0.02652227133512497,
+            -0.025829525664448738,
+            -0.028419075533747673,
+            0.017467092722654343,
+            -0.01558678224682808,
+            -0.030398348346352577,
+            -0.007467635441571474,
+            -0.003577125258743763,
+            0.010910747572779655,
+            0.02193695306777954,
+            -0.03293841704726219,
+            -0.004898703191429377,
+            -0.019825726747512817,
+            0.03311984986066818,
+            -0.03366415202617645,
+            0.007327436935156584,
+            9.535564458929002e-05,
+            -0.010259236209094524,
+            0.040542129427194595,
+            -0.017763983458280563,
+            -0.006127502303570509,
+            0.006292441859841347,
+            -0.010020074434578419,
+            -0.003263740334659815,
+            0.008411914110183716,
+            0.01359101478010416,
+            -0.02037002705037594,
+            0.0034328033216297626,
+            -0.0009437631815671921,
+            -0.0008262437768280506,
+            -0.004927567206323147,
+            -0.03671552985906601,
+            -0.014687862247228622,
+            0.006238836329430342,
+            -0.023800769820809364,
+            -0.0015256903134286404,
+            0.004939937964081764,
+            -0.03876078128814697,
+            0.0065893325954675674,
+            0.02144213393330574,
+            -0.0028431445825845003,
+            0.006045032292604446,
+            -0.0032431227155029774,
+            -0.005500731989741325,
+            -0.0009633497102186084,
+            -0.0018246431136503816,
+            0.017994899302721024,
+            0.04644696041941643,
+            0.008593347854912281,
+            0.011479788459837437,
+            0.007508870679885149,
+            -0.012469425797462463,
+            0.017516573891043663,
+            -0.022547228261828423,
+            0.007009928580373526,
+            0.03754022717475891,
+            0.02627486176788807,
+            0.011826162226498127,
+            0.003166838316246867,
+            0.0007674840744584799,
+            0.018951548263430595,
+            -0.034208450466394424,
+            0.024872876703739166,
+            0.014234278351068497,
+            -0.000996337621472776,
+            -0.008865498006343842,
+            -0.01110042817890644,
+            2.0150322598055936e-05,
+            0.017945416271686554,
+            0.008939720690250397,
+            -0.004106993321329355,
+            -0.03536302596330643,
+            0.002777168760076165,
+            0.018489716574549675,
+            0.010119037702679634,
+            -0.010894253849983215,
+            -0.024064673110842705,
+            -0.024180129170417786,
+            9.25851782085374e-05,
+            0.02180500142276287,
+            0.014432205818593502,
+            0.019231945276260376,
+            0.0269676074385643,
+            0.029260266572237015,
+            0.02693461999297142,
+            0.008733546361327171,
+            0.012114806100726128,
+            0.011521023698151112,
+            0.006424393504858017,
+            -0.027858281508088112,
+            -0.007657316047698259,
+            0.025466658174991608,
+            0.019594812765717506,
+            0.002787477569654584,
+            0.025450164452195168,
+            0.00653572753071785,
+            -0.005187347065657377,
+            0.020534968003630638,
+            -0.0012937441933900118,
+            -0.0324271060526371,
+            -0.0074758827686309814,
+            -0.015265149995684624,
+            0.008428407832980156,
+            -0.003253431525081396,
+            -0.0032266287598758936,
+            -0.017285659909248352,
+            0.014209537766873837,
+            0.012857033871114254,
+            0.0014988876646384597,
+            0.00264315539970994,
+            -0.0010659215040504932,
+            -0.015017741359770298,
+            -0.013788941316306591,
+            0.00541001558303833,
+            -0.013887905515730381,
+            0.056013453751802444,
+            -0.0214256402105093,
+            -0.016469208523631096,
+            -0.017648525536060333,
+            -0.006787260062992573,
+            0.017747489735484123,
+            0.01654343120753765,
+            0.014102326706051826,
+            0.018308283761143684,
+            0.00041441048961132765,
+            0.04370896518230438,
+            0.00416884571313858,
+            -0.03625370189547539,
+            0.002042157109826803,
+            -0.01590016670525074,
+            0.011512776836752892,
+            0.017302153632044792,
+            0.0042471918277442455,
+            0.002052465919405222,
+            0.01359926164150238,
+            -0.006432640366256237,
+            0.023239973932504654,
+            -0.008758286945521832,
+            -0.0023792521096765995,
+            0.008980955928564072,
+            0.004750257357954979,
+            0.012733329087495804,
+            0.005174976773560047,
+            -0.028748953714966774,
+            -0.014019857160747051,
+            0.01084477175027132,
+            0.005038901697844267,
+            -0.011281860992312431,
+            0.030480818822979927,
+            0.01591666042804718,
+            0.018176332116127014,
+            0.004350278992205858,
+            -0.010077803395688534,
+            -0.012287992052733898,
+            0.0014751775888726115,
+            0.008003689348697662,
+            0.023767780512571335,
+            0.019594812765717506,
+            -0.013162171468138695,
+            0.0056244367733597755,
+            -0.04189463332295418,
+            0.012518907897174358,
+            0.009112906642258167,
+            -0.015875427052378654,
+            -0.0036719655618071556,
+            0.01636199839413166,
+            0.0045440830290317535,
+            0.00045848023728467524,
+            -0.008791275322437286,
+            0.039915356785058975,
+            0.024806899949908257,
+            0.008914980106055737,
+            -0.0063955290243029594,
+            -0.014753838069736958,
+            -0.002406054874882102,
+            0.016634147614240646,
+            0.028946882113814354,
+            0.011570505797863007,
+            0.014061091467738152,
+            0.010795289650559425,
+            -0.023338938131928444,
+            0.012230263091623783,
+            0.028072701767086983,
+            0.044929519295692444,
+            0.016485702246427536,
+            0.03340024873614311,
+            0.027858281508088112,
+            0.0027379957027733326,
+            0.009327328763902187,
+            -0.02068341337144375,
+            -0.014778578653931618,
+            0.003181270556524396,
+            -0.006197601556777954,
+            0.04684281721711159,
+            -0.0071872384287416935,
+            -0.05287960171699524,
+            -0.016502195969223976,
+            0.0025668710004538298,
+            0.02164006233215332,
+            0.020007161423563957,
+            0.004424502141773701,
+            0.0006118224118836224,
+            0.02221735008060932,
+            -0.012065324001014233,
+            -0.013986868783831596,
+            0.011578752659261227,
+            0.0071789915673434734,
+            -0.018588680773973465,
+            0.013945634476840496,
+            0.019067006185650826,
+            -0.003026639809831977,
+            -0.009937604889273643,
+            -0.0036080514546483755,
+            0.024592477828264236,
+            0.006725407671183348,
+            -2.0440254957065918e-05,
+            0.0013968313578516245,
+            0.009269599802792072,
+            0.0027627365197986364,
+            0.04176267981529236,
+            -0.005789376329630613,
+            0.009764418005943298,
+            -0.012271498329937458,
+            -0.004201833624392748,
+            0.03377960994839668,
+            -0.03509912267327309,
+            -0.015496065840125084,
+            0.019248438999056816,
+            0.013665237464010715,
+            0.002488524653017521,
+            0.0017256794963032007,
+            -0.004972925875335932,
+            -0.012213769368827343,
+            0.0316024087369442,
+            -0.0056120664812624454,
+            -0.006605826783925295,
+            0.039651453495025635,
+            -0.006985187530517578,
+            -0.005702782887965441,
+            0.028204653412103653,
+            -0.0015782647533342242,
+            -0.03241061046719551,
+            0.014226031489670277,
+            -0.008552112616598606,
+            0.03552796691656113,
+            0.015792956575751305,
+            -0.012172535061836243,
+            -0.011554011143743992,
+            0.013929140754044056,
+            -0.009879875928163528,
+            -0.023075034841895103,
+            0.0029586022719740868,
+            -0.022580217570066452,
+            0.013986868783831596,
+            -0.009797406382858753,
+            0.022365795448422432,
+            -0.006737778428941965,
+            0.017615538090467453,
+            0.018407247960567474,
+            0.008440778590738773,
+            -0.017417611554265022,
+            0.011306602507829666,
+            -0.02121121808886528,
+            -0.05462796241044998,
+            0.010135532356798649,
+            -0.002533883089199662,
+            0.017153708264231682,
+            -0.011158157140016556,
+            -0.009541749954223633,
+            0.003568878397345543,
+            -0.008659323677420616,
+            -0.05185697600245476,
+            0.0067006670869886875,
+            -0.023718299344182014,
+            0.014308501034975052,
+            -0.02104627899825573,
+            -0.027957243844866753,
+            -0.015776462852954865,
+            -0.04361000284552574,
+            -0.014762084931135178,
+            -0.0130714550614357,
+            0.015289891511201859,
+            0.016238292679190636,
+            -0.011677715927362442,
+            -0.018275296315550804,
+            0.030744722113013268,
+            0.012436438351869583,
+            -0.006754272151738405,
+            -0.016972273588180542,
+            0.017120718955993652,
+            -0.03117356449365616,
+            -0.0440058559179306,
+            -0.013846670277416706,
+            0.02079886943101883,
+            0.007970700971782207,
+            -0.02083185873925686,
+            -0.028353098779916763,
+            -0.005772882141172886,
+            -0.012131299823522568,
+            -0.002649340545758605,
+            0.03356518596410751,
+            0.017335141077637672,
+            0.01619705744087696,
+            0.029227279126644135,
+            -0.03336726129055023,
+            0.03142097219824791,
+            -0.015570288524031639,
+            0.023982202634215355,
+            0.0035894957836717367,
+            -0.011916878633201122,
+            -0.006515109911561012,
+            0.04403884336352348,
+            -0.012716835364699364,
+            0.018110357224941254,
+            -0.022266831248998642,
+            -0.0028967498801648617,
+            0.006172860506922007,
+            0.02137615904211998,
+            -0.006915088277310133,
+            -0.007273831870406866,
+            0.00403689406812191,
+            -0.024196624755859375,
+            0.022547228261828423,
+            0.04295024275779724,
+            0.00791709590703249,
+            0.01558678224682808,
+            -0.019314413890242577,
+            -0.012914762832224369,
+            0.018935054540634155,
+            0.008849004283547401,
+            -0.014085832983255386,
+            -0.012898268178105354,
+            0.01030871830880642,
+            -0.018044380471110344,
+            -0.019330909475684166,
+            -0.019677281379699707,
+            -0.017401115968823433,
+            0.009929358027875423,
+            0.011314849369227886,
+            -0.040212247520685196,
+            -0.011603493243455887,
+            -0.04288426786661148,
+            -0.011842655949294567,
+            0.008766533806920052,
+            -0.006832618732005358,
+            0.03579187020659447,
+            -0.0036740272771567106,
+            0.004214203916490078,
+            -0.015562041662633419,
+            -0.007418153807520866,
+            -0.002801909577101469,
+            -0.035264063626527786,
+            0.0005896586808376014,
+            0.015075469389557838,
+            0.0003703407128341496,
+            -0.04710672050714493,
+            0.017829960212111473,
+            -0.016617653891444206,
+            0.016865063458681107,
+            -0.02708306536078453,
+            -0.008185122162103653,
+            -0.024312080815434456,
+            0.011611740104854107,
+            0.008997449651360512,
+            -0.03605577349662781,
+            -0.014423958957195282,
+            -0.002556562190875411,
+            0.029293254017829895,
+            -0.017351634800434113,
+            0.0006633660523220897,
+            0.017302153632044792,
+            -0.00019715423695743084,
+            -0.010894253849983215,
+            -0.022497747093439102,
+            -0.0022266830783337355,
+            -0.007937713526189327,
+            0.020106123760342598,
+            -0.04156475141644478,
+            -0.005496608559042215,
+            -0.01082003116607666,
+            0.004001844674348831,
+            -0.030150938779115677,
+            -0.009723182767629623,
+            0.006432640366256237,
+            0.02229982055723667,
+            -0.04882209002971649,
+            0.009253106079995632,
+            0.04374195262789726,
+            -0.018489716574549675,
+            -0.01057262159883976,
+            0.025120284408330917,
+            -0.0007772773969918489,
+            0.009459280408918858,
+            -0.009063425473868847,
+            0.016683630645275116,
+            0.02579653635621071,
+            -0.02061743661761284,
+            -0.008020183071494102,
+            0.0003203434171155095,
+            0.002498833229765296,
+            -0.027973737567663193,
+            0.018473222851753235,
+            -0.013261135667562485,
+            0.023289456963539124,
+            0.0077768974006175995,
+            0.010943735018372536,
+            0.05429808050394058,
+            -0.015248656272888184,
+            0.014399217441678047,
+            0.015562041662633419,
+            -0.0074800061993300915,
+            -0.007974824868142605,
+            -0.013772447593510151,
+            0.022563721984624863,
+            0.03704541176557541,
+            0.013582766987383366,
+            0.005649177823215723,
+            -0.039717432111501694,
+            0.01293950341641903,
+            0.0074758827686309814,
+            0.0038637076504528522,
+            0.02096381038427353,
+            0.017763983458280563,
+            0.02196994051337242,
+            0.024163635447621346,
+            -0.02594498172402382,
+            -0.006098637823015451,
+            0.005513102747499943,
+            -0.02762736566364765,
+            -0.003422494512051344,
+            -0.019512342289090157,
+            0.023619335144758224,
+            0.015223915688693523,
+            -0.02660473994910717,
+            0.0008654168923385441,
+            -0.007954207248985767,
+            -0.0006757365190424025,
+            -0.003993597347289324,
+            -0.012667353264987469,
+            -0.012263251468539238,
+            -0.01576821506023407,
+            0.02759437821805477,
+            -0.014753838069736958,
+            -0.024378057569265366,
+            -0.015990883111953735,
+            0.014489934779703617,
+            -0.009088166058063507,
+            0.011224132962524891,
+            -0.030992131680250168,
+            -0.04964678734540939,
+            -0.0216730497777462,
+            0.028419075533747673,
+            0.03302088752388954,
+            -0.025417177006602287,
+            -1.3602676517621148e-05,
+            -0.007042916491627693,
+            0.029755085706710815,
+            -0.03622071444988251,
+            -0.008568606339395046,
+            -0.01332711149007082,
+            -0.04733763635158539,
+            0.0130714550614357,
+            -0.028567520901560783,
+            0.024922357872128487,
+            0.02708306536078453,
+            -0.018308283761143684,
+            -0.03148695081472397,
+            0.0210957620292902,
+            0.020155606791377068,
+            -0.004109055269509554,
+            0.010234495624899864,
+            -0.0036822741385549307,
+            -0.02658824622631073,
+            -0.028270630165934563,
+            -0.003356518689543009,
+            0.04163073003292084,
+            0.014902283437550068,
+            0.03536302596330643,
+            0.04212554544210434,
+            -0.009921110235154629,
+            -0.034241437911987305,
+            0.019099993631243706,
+            0.009945851750671864,
+            -0.007653192616999149,
+            0.00516672944650054,
+            0.003502902574837208,
+            -0.018423741683363914,
+            -0.009137648157775402,
+            -0.008840756490826607,
+            -0.028765447437763214,
+            -0.0441707968711853,
+            0.0012638488551601768,
+            0.0013360099401324987,
+            0.0167083702981472,
+            -0.012518907897174358,
+            -0.006374911405146122,
+            0.035132113844156265,
+            0.03331777825951576,
+            -0.009789158590137959,
+            -2.2904683646629564e-05,
+            -0.017994899302721024,
+            -0.0103004714474082,
+            -0.008131517097353935,
+            -0.002187510021030903,
+            0.023833757266402245,
+            -0.014300254173576832,
+            0.030051976442337036,
+            -0.0015452768420800567,
+            0.015512559562921524,
+            -0.06749323755502701,
+            -0.02066691778600216,
+            0.03155292570590973,
+            0.008725299499928951,
+            -0.009970592334866524,
+            0.002399869728833437,
+            0.005480114836245775,
+            0.0001164240893558599,
+            -0.0220029279589653,
+            0.011694210581481457,
+            0.020023655146360397,
+            0.019974172115325928,
+            -0.020534968003630638,
+            0.022497747093439102,
+            0.01632900908589363,
+            0.019314413890242577,
+            -0.004210080485790968,
+            -0.025268729776144028,
+            0.012444685213267803,
+            -0.007867613807320595,
+            -0.0019627800211310387,
+            0.002861700253561139,
+            0.007463512010872364,
+            -0.008923226967453957,
+            0.01017676666378975,
+            -0.02205241098999977,
+            -0.02726449817419052,
+            -0.011529270559549332,
+            0.017813464626669884,
+            0.010473658330738544,
+            0.007113015744835138,
+            0.008436654694378376,
+            0.0018452606163918972,
+            0.00416884571313858,
+            0.0013267321046441793,
+            -0.019099993631243706,
+            0.028237640857696533,
+            0.001028294675052166,
+            0.013929140754044056,
+            0.013145677745342255,
+            -0.023635828867554665,
+            -0.023058541119098663,
+            -0.008626335300505161,
+            0.01072106696665287,
+            0.0035647547338157892,
+            0.019017523154616356,
+            0.03879376873373985,
+            -0.014918777160346508,
+            -0.014992999844253063,
+            -0.0008478920790366828,
+            0.019759751856327057,
+            0.0056780423037707806,
+            -0.010943735018372536,
+            -0.0028369592037051916,
+            -0.00416884571313858,
+            -0.005925451405346394,
+            0.025747055187821388,
+            -0.008890238590538502,
+            -0.03373012691736221,
+            0.010943735018372536,
+            0.05756388232111931,
+            -0.007059410214424133,
+            -0.0011659160954877734,
+            -0.022514240816235542,
+            -0.0019122672965750098,
+            0.006906841415911913,
+            0.0080655412748456,
+            0.006993434391915798,
+            0.015281643718481064,
+            0.01581769809126854,
+            0.012147793546319008,
+            -0.010102543979883194,
+            -0.012007595039904118,
+            -0.010374694131314754,
+            -0.03787010908126831,
+            0.030018987134099007,
+            -0.018258802592754364,
+            -0.0035358904860913754,
+            0.011331343092024326,
+            0.02188747003674507,
+            0.00010637308878358454,
+            0.016518689692020416,
+            0.013409581035375595,
+            0.004692528396844864,
+            0.0005932667409069836,
+            0.011652975343167782,
+            0.0015494003891944885,
+            -0.013896152377128601,
+            -0.022761650383472443,
+            -0.009533503092825413,
+            0.018918560817837715,
+            0.02705007791519165,
+            -0.004643046762794256,
+            -0.009978839196264744,
+            -0.005673918407410383,
+            0.014687862247228622,
+            -0.004878085572272539,
+            0.011916878633201122,
+            -0.0045317127369344234,
+            0.00045564534957520664,
+            0.030728228390216827,
+            0.013953881338238716,
+            -0.007298572454601526,
+            0.016972273588180542,
+            0.004556453321129084,
+            -0.003414247650653124,
+            0.019479354843497276,
+            -0.04915196821093559,
+            0.01590016670525074,
+            0.0028348974883556366,
+            0.0018029948696494102,
+            0.0036699038464576006,
+            -0.0020679289009422064,
+            -0.013277629390358925,
+            0.012626118026673794,
+            0.007636698428541422,
+            0.003036948386579752,
+            0.010152026079595089,
+            0.010110790841281414,
+            0.00022125086979940534,
+            0.012279745191335678,
+            -0.019776245579123497,
+            0.011512776836752892,
+            -0.015908414497971535,
+            -0.009467527270317078,
+            -0.008477889932692051,
+            -0.0013019911712035537,
+            -0.0037647439166903496,
+            -0.009962345473468304,
+            0.000947886670473963,
+            -0.022398782894015312,
+            -0.006758395582437515,
+            -0.017434105277061462,
+            -0.008799522183835506,
+            -0.0025359448045492172,
+            -0.024081166833639145,
+            0.0076738097704946995,
+            0.004960555117577314,
+            0.011232379823923111,
+            -0.03658358007669449,
+            0.0028328357730060816,
+            0.03272399678826332,
+            0.004506971687078476,
+            -0.018077367916703224,
+            -0.024378057569265366,
+            -0.011702457442879677,
+            -0.02657175250351429,
+            0.013285876251757145,
+            0.011075686663389206,
+            -0.020040148869156837,
+            0.015463077463209629,
+            -0.001200965722091496,
+            -0.023784274235367775,
+            -0.02098030410706997,
+            -0.02073289453983307,
+            -0.0007360425079241395,
+            0.0013566273264586926,
+            0.001249416614882648,
+            0.0007391351391561329,
+            0.004869838710874319,
+            -0.022877108305692673,
+            0.019611306488513947,
+            0.03354869410395622,
+            -0.008486136794090271,
+            -0.0030472571961581707,
+            0.008816015906631947,
+            0.0220029279589653,
+            -0.01660940796136856,
+            0.00010611536708893254,
+            -0.015042481943964958,
+            0.02150811068713665,
+            -0.0027565513737499714,
+            0.012626118026673794,
+            -0.0026946989819407463,
+            0.029639627784490585,
+            -0.004688404966145754,
+            0.0272974856197834,
+            -0.008947967551648617,
+            0.0220029279589653,
+            0.0013782756868749857,
+            -0.034769244492053986,
+            -0.009401551447808743,
+            0.03277347609400749,
+            -0.002628723159432411,
+            -0.013178665190935135,
+            0.022926589474081993,
+            -0.0036967063788324594,
+            -0.024048177525401115,
+            0.00832944456487894,
+            0.0015875425888225436,
+            -0.01375595387071371,
+            0.024955345317721367,
+            0.006370787974447012,
+            0.009549996815621853,
+            0.004742010496556759,
+            0.015529053285717964,
+            0.024031683802604675,
+            -0.000978297321125865,
+            0.03213021531701088,
+            0.009508761577308178,
+            0.0070016817189753056,
+            -0.004346155561506748,
+            0.012403449974954128,
+            -0.005100754089653492,
+            0.024806899949908257,
+            -0.03272399678826332,
+            -0.014118820428848267,
+            0.033746618777513504,
+            0.015232162550091743,
+            0.018159838393330574,
+            -0.022563721984624863,
+            -0.0047873686999082565,
+            0.016386738047003746,
+            0.012733329087495804,
+            -0.01585068553686142,
+            -0.02081536501646042,
+            -0.001431880984455347,
+            0.008510878309607506,
+            0.004280179738998413,
+            -0.017599044367671013,
+            0.00668417289853096,
+            0.03816699981689453,
+            -0.017978405579924583,
+            -0.040047310292720795,
+            0.02596147730946541,
+            -0.015471324324607849,
+            -0.003115294734016061,
+            0.016007378697395325,
+            -0.005389397963881493,
+            0.014473441056907177,
+            -0.008857251144945621,
+            -0.0033029133919626474,
+            0.03628668934106827,
+            0.008914980106055737,
+            0.010481905192136765,
+            -0.030018987134099007,
+            0.02101329155266285,
+            0.002962725702673197,
+            0.007653192616999149,
+            -0.023668818175792694,
+            0.002189571736380458,
+            0.012411696836352348,
+            -0.030051976442337036,
+            -0.003247246379032731,
+            0.03816699981689453,
+            0.008148010820150375,
+            -0.017615538090467453,
+            0.0008721175836399198,
+            -0.02579653635621071,
+            0.02038652077317238,
+            -0.0050801364704966545,
+            0.014580651186406612,
+            0.019462861120700836,
+            -0.012114806100726128,
+            0.018060874193906784,
+            -0.0056161899119615555,
+            -0.011562258936464787,
+            -0.004931691102683544,
+            -0.010943735018372536,
+            -0.02665422298014164,
+            -0.013986868783831596,
+            0.0020029840525239706,
+            -0.01059736218303442,
+            -0.01629602164030075,
+            0.007158373948186636,
+            -0.03641863912343979,
+            -0.014910530298948288,
+            -0.006391405593603849,
+            0.007632574997842312,
+            -0.023569853976368904,
+            0.017450598999857903,
+            -0.024641960859298706,
+            0.0029874665196985006,
+            0.02182149514555931,
+            -0.003570940112695098,
+            0.0024204871151596308,
+            -0.019001029431819916,
+            -0.029276760295033455,
+            0.008366555906832218,
+            0.006506863050162792,
+            -0.00681200111284852,
+            -0.015908414497971535,
+            0.008857251144945621,
+            0.0017669143853709102,
+            -0.0037771144416183233,
+            -0.006255330517888069,
+            0.01293950341641903,
+            -0.023685311898589134,
+            0.017483586445450783,
+            0.019545329734683037,
+            -0.024246105924248695,
+            0.010762302204966545,
+            -0.0069975582882761955,
+            0.017483586445450783,
+            0.01655992493033409,
+            0.0051337420009076595,
+            0.0164609607309103,
+            0.02695111371576786,
+            0.008259344846010208,
+            0.022728662937879562,
+            0.000705116312019527,
+            -0.004383266903460026,
+            -0.019512342289090157,
+            0.027792304754257202,
+            -0.009525256231427193,
+            0.0001148777810158208,
+            0.012568389065563679,
+            -0.009731430560350418,
+            -0.013945634476840496,
+            0.0019308229675516486,
+            4.95140629936941e-05,
+            -0.016172317788004875,
+            -0.007875860668718815,
+            -0.02150811068713665,
+            -0.005768758710473776,
+            -0.018242308869957924,
+            -0.01110042817890644,
+            0.02220085635781288,
+            -0.020023655146360397,
+            -0.0209308210760355,
+            0.00045384131954051554,
+            -0.00031982798827812076,
+            0.019974172115325928,
+            0.020155606791377068,
+            0.009302587248384953,
+            0.0021297812927514315,
+            -0.0025462533812969923,
+            -0.00341012398712337,
+            -0.005686289165169001,
+            0.008366555906832218,
+            0.006510986480861902,
+            0.02220085635781288,
+            -0.0216730497777462,
+            -0.01576821506023407,
+            0.007080027833580971,
+            -0.006840865593403578,
+            0.009764418005943298,
+            -0.019974172115325928,
+            -0.011776680126786232,
+            -0.004704899154603481,
+            -0.017005261033773422,
+            0.0552547313272953,
+            -0.02716553397476673,
+            -0.035297051072120667,
+            -0.01058086846023798,
+            0.002946231747046113,
+            0.006741901859641075,
+            -0.018522705882787704,
+            0.010366447269916534,
+            -0.01323639415204525,
+            -0.014869295060634613,
+            -0.023586347699165344,
+            0.00514611229300499,
+            0.0026081057731062174,
+            0.005933698266744614,
+            -0.009772664867341518,
+            -0.022910095751285553,
+            0.0036121748853474855,
+            -0.017236176878213882,
+            -0.017434105277061462,
+            -0.0020060765091329813,
+            0.01031696517020464,
+            0.015858933329582214,
+            0.018027886748313904,
+            0.007277955301105976,
+            0.002725625177845359,
+            0.0009215994505211711,
+            -0.002183386590331793,
+            0.009145895019173622,
+            -0.00333383958786726,
+            0.003041071817278862,
+            0.008585100993514061,
+            -0.027808798477053642,
+            0.009582984261214733,
+            -0.013574520125985146,
+            0.02705007791519165,
+            0.029161302372813225,
+            0.01603211835026741,
+            0.013137430883944035,
+            0.009244858287274837,
+            -0.040179260075092316,
+            0.011364331468939781,
+            -0.000893765885848552,
+            0.00807378813624382,
+            0.007220226339995861,
+            0.0038637076504528522,
+            -0.007298572454601526,
+            0.0013040528865531087,
+            -0.019116487354040146,
+            -0.004408007953315973,
+            0.000873663870152086,
+            -0.024559490382671356,
+            0.02187097631394863,
+            0.013533285818994045,
+            0.015545547008514404,
+            0.020403016358613968,
+            -0.011801420710980892,
+            0.010886006988584995,
+            0.01297249086201191,
+            0.003876078175380826,
+            -0.020419510081410408,
+            0.005830611102283001,
+            -0.023437902331352234,
+            0.0065893325954675674,
+            -0.010086050257086754,
+            -0.0340765006840229,
+            -0.020122617483139038,
+            0.017846453934907913,
+            0.008432531729340553,
+            -0.009261352941393852,
+            -0.028946882113814354,
+            0.025235742330551147,
+            0.027858281508088112,
+            -0.004498724825680256,
+            0.008956214413046837,
+            0.027792304754257202,
+            0.010358200408518314,
+            0.0446656160056591,
+            -0.0080655412748456,
+            0.009723182767629623,
+            -0.004630676470696926,
+            0.007088274694979191,
+            -0.0051378654316067696,
+            -0.006869730073958635,
+            0.011050946079194546,
+            0.014762084931135178,
+            0.006069773342460394,
+            0.018836090341210365,
+            0.029359230771660805,
+            -0.019281426444649696,
+            -0.007459388580173254,
+            0.013137430883944035,
+            -0.010770549066364765,
+            0.02088133990764618,
+            -0.0001592696935404092,
+            -0.006358417682349682,
+            -0.028303617611527443,
+            0.006976940669119358,
+            0.008774780668318272,
+            -0.007401659619063139,
+            -0.003787423251196742,
+            -0.019842220470309258,
+            -0.005504855420440435,
+            -0.0012762193800881505,
+            -0.020320545881986618,
+            -0.0009262383682653308,
+            -0.004993543028831482,
+            -0.0026699581649154425,
+            0.0047914921306073666,
+            0.018159838393330574,
+            0.008659323677420616,
+            0.009912863373756409,
+            -0.0017308337846770883,
+            0.014861048199236393,
+            0.03381259739398956,
+            -0.005636807065457106,
+            -0.000506158045027405,
+            0.019231945276260376,
+            0.0011978730326518416,
+            0.013937387615442276,
+            -0.004259562585502863,
+            -0.005785252898931503,
+            -0.014894036576151848,
+            0.01571873389184475,
+            -0.004614182282239199,
+            -0.004201833624392748,
+            -0.012774563394486904,
+            -0.020205087959766388,
+            0.012494166381657124,
+            -0.008857251144945621,
+            -0.008923226967453957,
+            -0.023685311898589134,
+            -0.03130551427602768,
+            -0.017450598999857903,
+            -0.0038224728778004646,
+            -0.0009566490771248937,
+            0.014654873870313168,
+            0.02124420739710331,
+            -0.007232596632093191,
+            0.022250337526202202,
+            -0.005962562747299671,
+            -0.03320232033729553,
+            0.009096412919461727,
+            0.014992999844253063,
+            -0.010275730863213539,
+            0.008267591707408428,
+            -0.02221735008060932,
+            -0.009277846664190292,
+            -0.03206423670053482,
+            -0.019776245579123497,
+            -0.0034781615249812603,
+            0.02149161696434021,
+            -0.01636199839413166,
+            0.02688513696193695,
+            -0.013277629390358925,
+            -0.00779339112341404,
+            0.01333535835146904,
+            0.003752373391762376,
+            -0.01606510765850544,
+            -0.004440995864570141,
+            0.011479788459837437,
+            0.01674135960638523,
+            -0.013021972961723804,
+            -0.004560577217489481,
+            -0.00333796301856637,
+            -0.009706689044833183,
+            0.008914980106055737,
+            -0.006857359316200018,
+            0.007640821859240532,
+            -0.019644293934106827,
+            0.010918994434177876,
+            0.005055395420640707,
+            -0.0017143398290500045,
+            -0.0019339155405759811,
+            -0.013673484325408936,
+            0.005711030215024948,
+            0.013656990602612495,
+            0.020551461726427078,
+            -0.03866181895136833,
+            0.004572947509586811,
+            -0.002818403532728553,
+            -0.005224458407610655,
+            -0.018605174496769905,
+            0.012452932074666023,
+            0.016584666445851326,
+            0.015463077463209629,
+            0.013722965493798256,
+            -0.01041592936962843,
+            0.022514240816235542,
+            -0.012395203113555908,
+            -0.013038466684520245,
+            0.00780988484621048,
+            -0.01306320820003748,
+            0.0005324453231878579,
+            0.029045844450592995,
+            -0.007574846502393484,
+            -0.004626553039997816,
+            0.005574955139309168,
+            0.039552491158246994,
+            0.005385274533182383,
+            0.004985296167433262,
+            0.029375724494457245,
+            -0.0013391025131568313,
+            0.006844989024102688,
+            0.03262503072619438,
+            0.017928922548890114,
+            0.007896478287875652,
+            -0.007096521556377411,
+            0.013962128199636936,
+            0.011496283113956451,
+            0.013904399238526821,
+            -0.012510661035776138,
+            -0.04077304154634476,
+            0.019231945276260376,
+            0.004651293624192476,
+            -0.011974607594311237,
+            0.010152026079595089,
+            -0.0024328576400876045,
+            0.007257337681949139,
+            0.017054744064807892,
+            0.00026699580485001206,
+            -0.022811131551861763,
+            0.013426074758172035,
+            0.004170907661318779,
+            0.011298355646431446,
+            -0.005055395420640707,
+            0.000341476290486753,
+            0.02233280800282955,
+            0.007108892314136028,
+            0.0014689923264086246,
+            0.012469425797462463,
+            -0.0128240454941988,
+            0.02145862765610218,
+            0.019825726747512817,
+            0.02716553397476673,
+            -0.0031503443606197834,
+            -0.0031482826452702284,
+            0.000705116312019527,
+            -0.0070016817189753056,
+            -0.012890021316707134,
+            0.009492267854511738,
+            0.0003778145182877779,
+            0.009401551447808743,
+            -0.005851228721439838,
+            0.011685962788760662,
+            0.007570722606033087,
+            0.004849221091717482,
+            0.019974172115325928,
+            0.005253322888165712,
+            -0.0068037542514503,
+            0.028946882113814354,
+            0.0020277248695492744,
+            0.0009226303081959486,
+            -0.011050946079194546,
+            0.0006973848212510347,
+            0.0014669306110590696,
+            -0.001361781731247902,
+            0.007611957844346762,
+            -0.008110899478197098,
+            -0.0130714550614357,
+            -0.01086126547306776,
+            0.028369592502713203,
+            0.00025436762371100485,
+            0.023503877222537994,
+            -0.0053316690027713776,
+            -0.002236991887912154,
+            0.002789539285004139,
+            -0.01618056371808052,
+            -0.005669794976711273,
+            0.015710486099123955,
+            0.0033214690629392862,
+            0.006432640366256237,
+            0.013285876251757145,
+            -0.0053770276717841625,
+            0.016353750601410866,
+            -0.0056203133426606655,
+            0.0036987680941820145,
+            0.013030219823122025,
+            0.005987303797155619,
+            0.008733546361327171,
+            0.0006375942612066865,
+            -0.017796970903873444,
+            -0.019611306488513947,
+            0.01085301861166954,
+            -0.006865606643259525,
+            0.000996337621472776,
+            0.003098800778388977,
+            -0.001747327740304172,
+            0.011248873546719551,
+            -0.015009493567049503,
+            -0.003950301092118025,
+            -0.008007812313735485,
+            0.034967172890901566,
+            -0.006127502303570509,
+            -0.002942108316347003,
+            -0.0063996524550020695,
+            -0.01334360521286726,
+            0.003667841898277402,
+            -0.008585100993514061,
+            -0.00026235688710585237,
+            0.015710486099123955,
+            0.004915196914225817,
+            -0.01703825034201145,
+            -0.013269382528960705,
+            -0.014696109108626842,
+            0.006469751708209515,
+            0.02078237570822239,
+            0.02706657163798809,
+            -0.008783028461039066,
+            0.007108892314136028,
+            -0.008135640993714333,
+            0.017054744064807892,
+            -0.007706797681748867,
+            0.013541532680392265,
+            -0.022695673629641533,
+            0.008898485451936722,
+            -0.01362400222569704,
+            0.016353750601410866,
+            -0.007438771426677704,
+            0.0038905104156583548,
+            -0.01557853538542986,
+            -0.009690195322036743,
+            0.0010999402729794383,
+            -0.011273614130914211,
+            0.009879875928163528,
+            -0.01030871830880642,
+            -0.008535618893802166,
+            0.0043007973581552505,
+            -0.02159057930111885,
+            -0.0021297812927514315,
+            -0.007682057097554207,
+            -0.005636807065457106,
+            -0.009698442183434963,
+            -0.02116173692047596,
+            -0.02140914648771286,
+            0.005694536026567221,
+            -0.009492267854511738,
+            -0.012147793546319008,
+            -0.028979869559407234,
+            -0.03326829522848129,
+            -0.001901958603411913,
+            -0.02180500142276287,
+            -0.014259018935263157,
+            -0.005814117379486561,
+            0.019215451553463936,
+            -0.011166404001414776,
+            -0.012914762832224369,
+            -0.00795008335262537,
+            -0.011125168763101101,
+            0.019462861120700836,
+            0.014234278351068497,
+            0.009954098612070084,
+            -0.001399923930875957,
+            0.016667135059833527,
+            -0.008700557984411716,
+            0.009756171144545078,
+            -0.040443163365125656,
+            -0.017450598999857903,
+            -0.0028369592037051916,
+            0.0026720198802649975,
+            0.008890238590538502,
+            -0.000427811814006418,
+            0.019924690946936607,
+            -0.010102543979883194,
+            -0.04374195262789726,
+            -0.02116173692047596,
+            -0.002880255924537778,
+            -0.011438554152846336,
+            0.022794637829065323,
+            0.003810102352872491,
+            0.039288587868213654,
+            0.022893602028489113,
+            0.015858933329582214,
+            -0.006708913948386908,
+            0.007401659619063139,
+            -0.019001029431819916,
+            0.0015380608383566141,
+            0.016881557181477547,
+            0.0032369375694543123,
+            -0.011174650862812996,
+            0.029359230771660805,
+            -0.009442785754799843,
+            -0.015075469389557838,
+            0.027858281508088112,
+            0.0011803482193499804,
+            0.013285876251757145,
+            0.0005705875810235739,
+            -0.0050760130397975445,
+            -0.010399434715509415,
+            0.0038451519794762135,
+            0.002278226660564542,
+            -0.012700340710580349,
+            0.01596614345908165,
+            0.00808203499764204,
+            4.94818486913573e-05,
+            0.0007891323766671121,
+            0.025153273716568947,
+            -0.0071295094676315784,
+            -0.017681512981653214,
+            0.014894036576151848,
+            0.0032018879428505898,
+            -0.01084477175027132,
+            -0.011215885169804096,
+            -0.009558243677020073,
+            -0.01349205058068037,
+            -0.010927241295576096,
+            0.0027689216658473015,
+            -0.002622538013383746,
+            0.023784274235367775,
+            -0.004022461827844381,
+            0.006370787974447012,
+            -0.00390906585380435,
+            -0.002867885399609804,
+            0.028699472546577454,
+            0.019149474799633026,
+            -0.02643980085849762,
+            -0.017335141077637672,
+            0.0007241874700412154,
+            -0.005508979316800833,
+            -0.011372578330338001,
+            0.02091432735323906,
+            -0.006024415139108896,
+            -0.024724429473280907,
+            0.025070803239941597,
+            -0.0004649232141673565,
+            0.004102869890630245,
+            0.005739894229918718,
+            0.0264727883040905,
+            -0.0025668710004538298,
+            -0.0004610574396792799,
+            -0.003036948386579752,
+            0.011323096230626106,
+            2.6738238375401124e-05,
+            0.007018175441771746,
+            -0.023998696357011795,
+            -0.003014269284904003,
+            0.010803536511957645,
+            0.002383375773206353,
+            0.02747892029583454,
+            0.005459497217088938,
+            0.00514611229300499,
+            0.01324464101344347,
+            -0.016955779865384102,
+            0.008890238590538502,
+            -0.017895935103297234,
+            0.017170201987028122,
+            -0.008750040084123611,
+            -0.025582116097211838,
+            -0.010275730863213539,
+            0.0034740380942821503,
+            0.008675817400217056,
+            0.006898594554513693,
+            0.0043007973581552505,
+            0.019825726747512817,
+            -3.6467092286329716e-05,
+            0.006354294251650572,
+            0.002397808013483882,
+            0.0009432477527298033,
+            -0.0003213743038941175,
+            0.0037709292955696583,
+            0.02660473994910717,
+            -0.008659323677420616,
+            -0.006296565290540457,
+            0.0009757201769389212,
+            -0.0033276542089879513,
+            -0.01324464101344347,
+            0.003115294734016061,
+            -0.003919374663382769,
+            -0.0034596058540046215,
+            0.045787204056978226,
+            0.017104225233197212,
+            -0.007962454110383987,
+            -0.015265149995684624,
+            -0.01598263718187809,
+            0.027759317308664322,
+            -0.001940100803039968,
+            0.011578752659261227,
+            0.019759751856327057,
+            0.0030637511517852545,
+            -0.002773045329377055,
+            -0.0007468666299246252,
+            -0.0016091909492388368,
+            0.001787531771697104,
+            0.024889370426535606,
+            0.008964461274445057,
+            -0.006964570377022028,
+            -0.008469643071293831,
+            0.019215451553463936,
+            -0.01362400222569704,
+            -0.009426292032003403,
+            0.02119472436606884,
+            -0.003034886671230197,
+            0.02079886943101883,
+            -0.007521240971982479,
+            0.01606510765850544,
+            -0.005443003494292498,
+            -0.006964570377022028,
+            -0.02162356860935688,
+            0.00404101749882102,
+            -0.002410178305581212,
+            0.003016331000253558,
+            0.028254134580492973,
+            -0.014465193264186382,
+            -0.002249362412840128,
+            0.0036080514546483755,
+            0.011182897724211216,
+            -0.015108457766473293,
+            0.01031696517020464,
+            0.001118495943956077,
+            0.0007360425079241395,
+            0.025285225361585617,
+            -0.004133795853704214,
+            -0.004692528396844864,
+            0.03130551427602768,
+            -0.007459388580173254,
+            -0.011422060430049896,
+            -0.001281373668462038,
+            0.008069665171205997,
+            0.005661548115313053,
+            -0.02126070111989975,
+            -0.002247300697490573,
+            -0.01019326038658619,
+            0.0021194724831730127,
+            -0.045391347259283066,
+            0.02159057930111885,
+            -0.02165655605494976,
+            0.025367693975567818,
+            -0.005723400507122278,
+            -0.002488524653017521,
+            -0.008593347854912281,
+            0.006201724987477064,
+            0.005574955139309168,
+            -0.012114806100726128,
+            0.007665562909096479,
+            -0.019198957830667496,
+            0.013302369974553585,
+            0.006473875138908625,
+            -0.019231945276260376,
+            0.015693992376327515,
+            -0.007529487833380699,
+            0.006568715441972017,
+            -0.015438336879014969,
+            -0.003880201606079936,
+            0.019347403198480606,
+            -0.0026967606972903013,
+            -0.005772882141172886,
+            -0.03783712163567543,
+            -0.00418121600523591,
+            -0.011389072053134441,
+            0.00163496274035424,
+            0.002814280102029443,
+            -0.015743475407361984,
+            -0.03572589531540871,
+            0.005113124381750822,
+            0.01305496133863926,
+            -0.0009911833330988884,
+            0.022481253370642662,
+            -0.005781129468232393,
+            -0.005925451405346394,
+            0.008024306036531925,
+            0.0025297594256699085,
+            0.018968041986227036,
+            0.014267266727983952,
+            -0.003490532049909234,
+            -0.0045399595983326435,
+            0.0026740815956145525,
+            -0.007698550820350647,
+            -0.017813464626669884,
+            -0.018836090341210365,
+            -0.013021972961723804,
+            0.008832509629428387,
+            0.018489716574549675,
+            -0.018159838393330574,
+            -0.00040281316614709795,
+            -0.004263686016201973,
+            0.015529053285717964,
+            -0.011356084607541561,
+            -0.022398782894015312,
+            -0.009879875928163528,
+            -0.0034451736137270927,
+            0.012329227291047573,
+            -0.024411045014858246,
+            0.014663120731711388,
+            -0.009615972638130188,
+            -0.004276056308299303,
+            0.015809450298547745,
+            -0.0053316690027713776,
+            0.0014968259492889047,
+            0.004915196914225817,
+            0.028897399082779884,
+            0.0019370082300156355,
+            -0.022596711292862892,
+            0.018638163805007935,
+            0.009426292032003403,
+            -0.00026879983488470316,
+            -0.005001790355890989,
+            -0.004214203916490078,
+            -0.018984535709023476,
+            -0.00652748066931963,
+            0.008626335300505161,
+            0.01351679116487503,
+            0.013162171468138695,
+            0.016106341034173965,
+            -0.00528218736872077,
+            0.0013452877756208181,
+            0.004313167650252581,
+            0.0011349898995831609,
+            0.009912863373756409,
+            0.007455265149474144,
+            -0.002552438760176301,
+            -0.007467635441571474,
+            0.022629698738455772,
+            -0.023223480209708214,
+            -0.0032328141387552023,
+            0.023734793066978455,
+            -0.0030451954808086157,
+            0.013788941316306591,
+            -0.00817275233566761,
+            0.002777168760076165,
+            0.015001246705651283,
+            0.007405783515423536,
+            0.009607725776731968,
+            -0.024196624755859375,
+            0.020023655146360397,
+            -0.019858716055750847,
+            0.005723400507122278,
+            0.005038901697844267,
+            0.006774889770895243,
+            0.008494383655488491,
+            -0.010020074434578419,
+            0.0034554824233055115,
+            0.01334360521286726,
+            0.016939286142587662,
+            0.017120718955993652,
+            -0.015562041662633419,
+            -0.001777223078534007,
+            -0.022514240816235542,
+            -0.017236176878213882,
+            0.02137615904211998,
+            -0.013277629390358925,
+            -0.001867939718067646,
+            0.015281643718481064,
+            -0.009640713222324848,
+            -0.009888122789561749,
+            0.023701805621385574,
+            -0.009797406382858753,
+            0.016840321943163872,
+            0.0068862237967550755,
+            0.017994899302721024,
+            0.008011936210095882,
+            -0.0012184905353933573,
+            0.0026534642092883587,
+            -0.010267484001815319,
+            -0.0012442623265087605,
+            -0.004208019003272057,
+            0.011883890256285667,
+            0.0061522433534264565,
+            -0.01041592936962843,
+            -0.001817427109926939,
+            -0.009657206945121288,
+            -0.009780911728739738,
+            -0.017087731510400772,
+            0.006791383493691683,
+            0.005991427227854729,
+            -0.007385165896266699,
+            -0.016700124368071556,
+            0.00011146302131237462,
+            0.01386316493153572,
+            -0.0029359229374676943,
+            -0.00011751939018722624,
+            0.011644728481769562,
+            -0.0076696863397955894,
+            -0.011586999520659447,
+            -0.007937713526189327,
+            -0.040113285183906555,
+            -0.012057077139616013,
+            0.012799304910004139,
+            -0.0007819163147360086,
+            0.014588898047804832,
+            0.034307416528463364,
+            -0.005591448862105608,
+            0.0017401116201654077,
+            -0.007661439478397369,
+            0.014143561944365501,
+            0.022728662937879562,
+            -0.004816233180463314,
+            -0.004465736914426088,
+            -0.001779284793883562,
+            -0.012353967875242233,
+            -0.000558217114303261,
+            -0.016073353588581085,
+            0.0009757201769389212,
+            -0.003036948386579752,
+            -0.007983071729540825,
+            -0.00339775369502604,
+            -0.010382940992712975,
+            -0.014960012398660183,
+            -0.014300254173576832,
+            0.008568606339395046,
+            0.002474092412739992,
+            -0.007748032920062542,
+            -0.0032967282459139824,
+            -0.012642612680792809,
+            0.013937387615442276,
+            0.006197601556777954,
+            0.006869730073958635,
+            0.0037132003344595432,
+            0.009030437096953392,
+            0.006263577379286289,
+            -0.005455373786389828,
+            0.003102924209088087,
+            -0.0023751286789774895,
+            -0.030662251636385918,
+            0.011661222204566002,
+            -0.016551679000258446,
+            0.020436003804206848,
+            0.009937604889273643,
+            -0.0053729042410850525,
+            0.0053729042410850525,
+            0.0024575984571129084,
+            0.02121121808886528,
+            -0.009393304586410522,
+            0.014489934779703617,
+            0.0035894957836717367,
+            0.00821398664265871,
+            -0.00529868109151721,
+            0.004271932877600193,
+            -0.004688404966145754,
+            -0.014992999844253063,
+            0.01641147956252098,
+            0.00795833021402359,
+            0.022448265925049782,
+            0.00416884571313858,
+            0.015438336879014969,
+            0.0035276433918625116,
+            -0.001785470056347549,
+            0.00021802939590997994,
+            0.016551679000258446,
+            0.006078020203858614,
+            -0.015990883111953735,
+            -0.004395637661218643,
+            -0.006201724987477064,
+            0.0128240454941988,
+            0.02630784921348095,
+            -0.01629602164030075,
+            -0.01350029744207859,
+            0.023503877222537994,
+            -0.006981064099818468,
+            0.02629135549068451,
+            -0.004589441232383251,
+            -0.017665019258856773,
+            -0.008832509629428387,
+            -0.0018473223317414522,
+            -0.001506103784777224,
+            -0.0208483524620533,
+            0.019446365535259247,
+            0.01334360521286726,
+            -0.002018447034060955,
+            -0.0035874340683221817,
+            0.010622103698551655,
+            -0.0018493840470910072,
+            0.020073136314749718,
+            -0.024064673110842705,
+            -0.007100645452737808,
+            0.0059130811132490635,
+            0.011125168763101101,
+            0.0071913618594408035,
+            -0.015487818978726864,
+            0.015990883111953735,
+            0.018902065232396126,
+            -0.0021318430081009865,
+            0.020172100514173508,
+            -0.015339372679591179,
+            -0.02091432735323906,
+            0.0014576527755707502,
+            -0.007306819781661034,
+            -0.0026720198802649975,
+            -0.0015751721803098917,
+            -0.0009288155124522746,
+            0.009310834109783173,
+            0.011207638308405876,
+            0.0005079620750620961,
+            -0.012337474152445793,
+            0.00038992727058939636,
+            0.00340600055642426,
+            -0.014382723718881607,
+            -0.019215451553463936,
+            0.009838640689849854,
+            -0.015430090017616749,
+            0.011455047875642776,
+            0.018885571509599686,
+            0.0038245345931500196,
+            -0.008807769045233727,
+            0.015174433588981628,
+            0.017516573891043663,
+            0.0065893325954675674,
+            -0.013384840451180935,
+            0.004230698104947805,
+            -0.0010097390040755272,
+            -0.004737887065857649,
+            0.008824262768030167,
+            -0.004812109749764204,
+            0.020023655146360397,
+            -0.0133600989356637,
+            0.02691812627017498,
+            0.004577070940285921,
+            0.030101457610726357,
+            0.013162171468138695,
+            0.005636807065457106,
+            0.001126742921769619,
+            0.0077232918702065945,
+            0.00780163798481226,
+            0.011982854455709457,
+            -0.0164609607309103,
+            0.013896152377128601,
+            -0.0006267700809985399,
+            0.009813900105655193,
+            -0.019908197224140167,
+            -0.003797731827944517,
+            0.013384840451180935,
+            -0.0033812597393989563,
+            -0.02615940384566784,
+            0.014193043112754822,
+            0.02126070111989975,
+            0.01655992493033409,
+            -0.030530299991369247,
+            0.0164609607309103,
+            -0.013401334173977375,
+            0.015347619540989399,
+            0.0030678745824843645,
+            -0.001799902180209756,
+            -0.02126070111989975,
+            -0.0210957620292902,
+            0.015941401943564415,
+            -0.025301719084382057,
+            -0.009244858287274837,
+            0.008807769045233727,
+            0.0038492754101753235,
+            0.014061091467738152,
+            -0.02617589756846428,
+            -0.008028429932892323,
+            -0.008069665171205997,
+            -0.008296456187963486,
+            0.0036740272771567106,
+            0.002236991887912154,
+            0.006069773342460394,
+            -0.006292441859841347,
+            -0.02149161696434021,
+            -0.018110357224941254,
+            0.0011813790770247579,
+            0.01375595387071371,
+            0.011595246382057667,
+            -0.029441699385643005,
+            0.005748141556978226,
+            0.010886006988584995,
+            -0.0033049751073122025,
+            0.0007545981789007783,
+            -0.024328574538230896,
+            -0.016254786401987076,
+            -0.013549779541790485,
+            -0.006762519478797913,
+            -0.009310834109783173,
+            0.004676034674048424,
+            -0.008424284867942333,
+            -0.004581194370985031,
+            -0.01639498583972454,
+            -0.009665454737842083,
+            0.01297249086201191,
+            0.018687644973397255,
+            -0.005657424684613943,
+            -0.029821060597896576,
+            0.02681916207075119,
+            0.018291790038347244,
+            -0.008824262768030167,
+            -0.01641147956252098,
+            -0.0003654440806712955,
+            -0.016955779865384102,
+            0.008947967551648617,
+            0.004803862888365984,
+            -0.014580651186406612,
+            -0.009112906642258167,
+            -0.010877760127186775,
+            0.013310616835951805,
+            0.020122617483139038,
+            0.004820356611162424,
+            0.0074882530607283115,
+            -0.0035317668225616217,
+            0.023042047396302223,
+            -0.006915088277310133,
+            -2.363756948398077e-06,
+            0.0029833430889993906,
+            -0.00680787768214941,
+            0.007756279781460762,
+            0.023718299344182014,
+            -0.0335816815495491,
+            -0.006853235885500908,
+            -0.002346264198422432,
+            -0.002056589350104332,
+            -0.011949866078794003,
+            -0.011644728481769562,
+            -0.0077356621623039246,
+            0.010778795927762985,
+            0.006750148721039295,
+            -0.0018473223317414522,
+            0.002083391882479191,
+            -0.0133188646286726,
+            0.009129401296377182,
+            -0.024889370426535606,
+            0.002865823684260249,
+            -0.018473222851753235,
+            0.010028321295976639,
+            -0.009657206945121288,
+            -0.00012628179683815688,
+            0.013533285818994045,
+            -0.000562340603210032,
+            0.0015019802376627922,
+            -0.019182464107871056,
+            -0.009805653244256973,
+            0.00791709590703249,
+            0.008436654694378376,
+            0.009294340386986732,
+            -0.02586251311004162,
+            -0.0012298300862312317,
+            0.004601811990141869,
+            0.004170907661318779,
+            -0.0018267049454152584,
+            -0.02726449817419052,
+            -0.0074758827686309814,
+            -0.014894036576151848,
+            -0.02172253094613552,
+            0.0017421734519302845,
+            -0.015149693004786968,
+            -0.013986868783831596,
+            0.024130648002028465,
+            0.02216786891222,
+            0.016551679000258446,
+            -0.004886332433670759,
+            0.01571873389184475,
+            0.006601703353226185,
+            -0.023652324452996254,
+            0.0010679832194000483,
+            -0.014168302528560162,
+            0.01098497025668621,
+            -0.008906732313334942,
+            0.022629698738455772,
+            -0.001510227215476334,
+            -0.010020074434578419,
+            -0.003008084138855338,
+            0.013137430883944035,
+            0.008956214413046837,
+            -0.022382289171218872,
+            -0.009978839196264744,
+            0.003024578094482422,
+            -0.03224566951394081,
+            -0.006622320506721735,
+            0.0025276977103203535,
+            0.0014566219178959727,
+            0.007401659619063139,
+            0.007059410214424133,
+            -0.014539416879415512,
+            -0.0037152620498090982,
+            0.0021524603944271803,
+            -0.01086951233446598,
+            0.005983180366456509,
+            0.015083717182278633,
+            0.017912428826093674,
+            0.011595246382057667,
+            0.007356301415711641,
+            -0.014028104022145271,
+            0.004222451243549585,
+            -0.015174433588981628,
+            -0.0053811511024832726,
+            0.005010037217289209,
+            -0.025252236053347588,
+            -0.012386956252157688,
+            0.02162356860935688,
+            0.02168954350054264,
+            -0.0005592479719780385,
+            -0.023718299344182014,
+            -0.01675785332918167,
+            0.012923009693622589,
+            -0.01309619564563036,
+            0.018885571509599686,
+            -0.0033936300314962864,
+            -0.014259018935263157,
+            -0.0016184687847271562,
+            0.017928922548890114,
+            -0.018275296315550804,
+            0.01598263718187809,
+            -0.02071640081703663,
+            0.009129401296377182,
+            0.002925614360719919,
+            0.0025957352481782436,
+            0.0014875481138005853,
+            0.050273556262254715,
+            -0.00046518092858605087,
+            0.0009396396926604211,
+            0.018737126141786575,
+            -0.033680643886327744,
+            0.02132667601108551,
+            0.003560631303116679,
+            0.00822223350405693,
+            0.0012885897886008024,
+            -0.01319515984505415,
+            0.0105891153216362,
+            0.019677281379699707,
+            0.027874775230884552,
+            -0.010094297118484974,
+            -0.007121262606233358,
+            0.018143344670534134,
+            0.003245184663683176,
+            -0.001950409496203065,
+            0.0029606639873236418,
+            0.004185339901596308,
+            -0.019330909475684166,
+            -0.0036822741385549307,
+            0.011166404001414776,
+            -0.014506428502500057,
+            -0.009624219499528408,
+            -0.014374476857483387,
+            -0.009409798309206963,
+            -0.008964461274445057,
+            -0.010382940992712975,
+            0.0005654331762343645,
+            -0.0026534642092883587,
+            0.014423958957195282,
+            -0.013013726100325584,
+            0.029590144753456116,
+            -0.013302369974553585,
+            -0.024526502937078476,
+            -0.0037544353399425745,
+            -0.007422277238219976,
+            0.02074938826262951,
+            -0.015446583740413189,
+            -0.010127284564077854,
+            -0.0034348650369793177,
+            -0.012535401619970798,
+            0.022448265925049782,
+            0.008931473828852177,
+            -0.006222342606633902,
+            -0.0061604902148246765,
+            0.020172100514173508,
+            0.024295587092638016,
+            -0.0256480909883976,
+            0.02144213393330574,
+            0.020139113068580627,
+            -0.008552112616598606,
+            -0.0033874448854476213,
+            0.002342140767723322,
+            0.005974933039397001,
+            -0.0015050729271024466,
+            0.010646844282746315,
+            0.006226466037333012,
+            -0.013830176554620266,
+            -0.0163784921169281,
+            0.01692279241979122,
+            0.018473222851753235,
+            -0.008750040084123611,
+            0.009063425473868847,
+            -0.003573001828044653,
+            0.002962725702673197,
+            -0.00333796301856637,
+            0.007682057097554207,
+            -0.009987086057662964,
+            -0.017961911857128143,
+            0.011042699217796326,
+            0.0029070586897432804,
+            -0.005715153645724058,
+            0.0037647439166903496,
+            0.023718299344182014,
+            -0.024510009214282036,
+            -0.013566273264586926,
+            0.005488361697643995,
+            -0.02159057930111885,
+            0.014687862247228622,
+            -0.0006107915542088449,
+            0.008824262768030167,
+            0.001291682361625135,
+            0.003952362574636936,
+            -0.024955345317721367,
+            0.009393304586410522,
+            0.008906732313334942,
+            0.0006061526364646852,
+            -0.000622646592091769,
+            -0.009022190235555172,
+            -0.01100146397948265,
+            -0.007381042465567589,
+            -0.006964570377022028,
+            -0.0108365248888731,
+            -0.02061743661761284,
+            0.023883238434791565,
+            0.0022864737547934055,
+            -0.006840865593403578,
+            0.0022947206161916256,
+            0.0136487428098917,
+            -0.0032163201831281185,
+            -0.004317291080951691,
+            0.010077803395688534,
+            -0.017219683155417442,
+            -0.006073896773159504,
+            0.005174976773560047,
+            0.007628451567143202,
+            -0.019710268825292587,
+            -0.009517008438706398,
+            -0.012436438351869583,
+            0.012362214736640453,
+            0.007199608720839024,
+            -0.015512559562921524,
+            -0.01100146397948265,
+            0.00257099443115294,
+            0.005034778267145157,
+            -0.05881742388010025,
+            -0.008543865755200386,
+            0.003925560042262077,
+            0.03879376873373985,
+            -0.003125603310763836,
+            -0.0158836729824543,
+            -0.002467907266691327,
+            0.02736346237361431,
+            0.00016519721248187125,
+            0.02058444917201996,
+            0.006271824240684509,
+            -0.009714935906231403,
+            -0.008477889932692051,
+            0.007591340225189924,
+            -0.005335792433470488,
+            -0.013929140754044056,
+            0.004696652293205261,
+            0.005880093201994896,
+            0.02703358232975006,
+            -0.010514892637729645,
+            0.0005762573564425111,
+            -0.0031379738356918097,
+            -0.001628777477890253,
+            -0.003711138619109988,
+            -0.005253322888165712,
+            0.013162171468138695,
+            -0.015529053285717964,
+            0.014333241619169712,
+            -0.01386316493153572,
+            0.004416254814714193,
+            0.007442894857376814,
+            -0.004511095117777586,
+            -0.002079268451780081,
+            -0.0031585912220180035,
+            0.002257609274238348,
+            0.0071954852901399136,
+            0.004911073483526707,
+            -0.024180129170417786,
+            0.012543648481369019,
+            0.0036987680941820145,
+            0.011784926988184452,
+            0.00779339112341404,
+            -0.023965708911418915,
+            -0.0026163526345044374,
+            0.014168302528560162,
+            0.011438554152846336,
+            0.004100808408111334,
+            0.008494383655488491,
+            -0.008102652616798878,
+            0.015124951489269733,
+            -0.04209255799651146,
+            0.00844490248709917,
+            -0.005789376329630613,
+            0.008275839500129223,
+            -0.011586999520659447,
+            0.009343822486698627,
+            0.007884107530117035,
+            -0.023668818175792694,
+            0.009888122789561749,
+            -0.002232868457213044,
+            -0.014044597744941711,
+            -0.015289891511201859,
+            -0.01055612787604332,
+            -0.00011519993131514639,
+            -0.012551895342767239,
+            0.0017648525536060333,
+            0.0026926372665911913,
+            -0.001965872710570693,
+            -0.024526502937078476,
+            -0.0007612988702021539,
+            -0.009517008438706398,
+            -0.008618088439106941,
+            -0.00555021408945322,
+            -0.009805653244256973,
+            0.012197275646030903,
+            0.008218110539019108,
+            -0.0030039604753255844,
+            -0.01086126547306776,
+            0.02173902466893196,
+            0.0005819271318614483,
+            0.000386576954042539,
+            -0.007987194694578648,
+            -0.004729640204459429,
+            -0.02686864323914051,
+            0.009055178612470627,
+            -0.02206890471279621,
+            -0.0005499701364897192,
+            0.002034940989688039,
+            0.0050760130397975445,
+            -0.014094079844653606,
+            0.0025421299505978823,
+            -0.00138961523771286,
+            0.024163635447621346,
+            0.0031544677913188934,
+            -0.013698224909603596,
+            0.009805653244256973,
+            -0.015496065840125084,
+            0.0108365248888731,
+            0.03278997167944908,
+            0.001166946953162551,
+            0.0011659160954877734,
+            -0.001972057856619358,
+            -0.003702891757711768,
+            0.008424284867942333,
+            0.0019442243501543999,
+            -0.03354869410395622,
+            -0.02655525878071785,
+            -0.009863382205367088,
+            0.011826162226498127,
+            0.0056780423037707806,
+            0.0011174650862812996,
+            -0.010119037702679634,
+            0.01593315601348877,
+            0.006601703353226185,
+            0.02578004263341427,
+            0.014720849692821503,
+            0.013038466684520245,
+            0.005525473039597273,
+            0.0042925504967570305,
+            -0.006275947671383619,
+            -0.009178882464766502,
+            -0.012362214736640453,
+            0.011883890256285667,
+            0.007987194694578648,
+            0.012123052962124348,
+            0.0010772610548883677,
+            -0.00163083930965513,
+            0.010110790841281414,
+            -0.018885571509599686,
+            -0.024114154279232025,
+            0.020007161423563957,
+            0.0020638054702430964,
+            0.002554500475525856,
+            -0.013013726100325584,
+            0.004717269446700811,
+            9.213417069986463e-05,
+            0.008898485451936722,
+            -0.005900710355490446,
+            0.00265140226110816,
+            0.01677434705197811,
+            0.0051502357237041,
+            0.007463512010872364,
+            -0.0012937441933900118,
+            -0.01692279241979122,
+            0.01108393445611,
+            -0.004750257357954979,
+            -0.015487818978726864,
+            -0.014151808805763721,
+            0.01095198281109333,
+            -0.0008185122278518975,
+            0.006226466037333012,
+            -0.0011803482193499804,
+            -0.011133415624499321,
+            0.017368128523230553,
+            -0.024575984105467796,
+            0.006510986480861902,
+            0.001821550540626049,
+            0.024872876703739166,
+            0.0012267375132068992,
+            0.0038616459351032972,
+            -0.020188594236969948,
+            0.028435569256544113,
+            -0.023916225880384445,
+            -0.007945960387587547,
+            -0.011347836814820766,
+            6.86818384565413e-05,
+            -0.0007463512010872364,
+            0.0037853613030165434,
+            -0.008251097984611988,
+            -0.02144213393330574,
+            0.008750040084123611,
+            0.024213118478655815,
+            0.0017081546830013394,
+            0.007748032920062542,
+            0.016386738047003746,
+            0.0016184687847271562,
+            -0.012667353264987469,
+            -0.0105478810146451,
+            -0.011248873546719551,
+            -0.011207638308405876,
+            -0.012255004607141018,
+            0.003812164068222046,
+            -0.018852584064006805,
+            0.015372361056506634,
+            0.007265584543347359,
+            -0.020337039604783058,
+            -0.0015050729271024466,
+            0.006374911405146122,
+            0.013566273264586926,
+            -0.007517117541283369,
+            0.012123052962124348,
+            0.019875209778547287,
+            0.002107101958245039,
+            -0.0006345016299746931,
+            0.00041080242954194546,
+            0.006746025290340185,
+            -0.015661004930734634,
+            0.030018987134099007,
+            0.006869730073958635,
+            0.0063955290243029594,
+            -0.01060560904443264,
+            0.018819596618413925,
+            -0.007261461112648249,
+            0.014481687918305397,
+            -0.018671151250600815,
+            -0.01337659265846014,
+            -0.014745591208338737,
+            0.011166404001414776,
+            -0.010630350559949875,
+            0.003573001828044653,
+            0.03582485765218735,
+            0.008576854132115841,
+            -0.002403993159532547,
+            -0.006016168277710676,
+            -0.015050728805363178,
+            -0.004333785269409418,
+            -0.013953881338238716,
+            -0.000127505962154828,
+            -0.025169767439365387,
+            -0.013409581035375595,
+            0.00807378813624382,
+            0.02122771367430687,
+            0.014836307615041733,
+            0.004671911243349314,
+            0.015067222528159618,
+            -0.007133633363991976,
+            0.01682382822036743,
+            0.018687644973397255,
+            0.01069632638245821,
+            -0.00012067643547197804,
+            -0.009813900105655193,
+            0.0010020073968917131,
+            0.025235742330551147,
+            -0.011199391447007656,
+            -0.00541001558303833,
+            0.02229982055723667,
+            0.02046899124979973,
+            0.0011102489661425352,
+            0.0058594755828380585,
+            0.010127284564077854,
+            0.009079919196665287,
+            0.014168302528560162,
+            -0.004523465409874916,
+            0.012584883719682693,
+            -0.01082003116607666,
+            -0.006506863050162792,
+            0.04816233366727829,
+            0.014448699541389942,
+            -0.0036059897392988205,
+            -0.012197275646030903,
+            -0.0020772067364305258,
+            0.0023895609192550182,
+            0.02107926830649376,
+            0.010069556534290314,
+            0.015289891511201859,
+            0.024608973413705826,
+            -0.007661439478397369,
+            0.014423958957195282,
+            -0.0059172045439481735,
+            0.04410482197999954,
+            -0.017236176878213882,
+            -0.004972925875335932,
+            -0.012535401619970798,
+            0.006840865593403578,
+            -0.00527394050732255,
+            0.018011393025517464,
+            -0.0011545764282345772,
+            0.0007509901188313961,
+            -0.028386086225509644,
+            0.019380390644073486,
+            0.012692093849182129,
+            -0.03829894959926605,
+            0.006234712898731232,
+            -0.009599478915333748,
+            -0.0019493786385282874,
+            -0.02645629458129406,
+            -0.004109055269509554,
+            -0.009987086057662964,
+            -0.0029194289818406105,
+            -0.025351200252771378,
+            0.007302696350961924,
+            -0.025318212807178497,
+            0.009426292032003403,
+            -0.0077851442620158195,
+            -0.0058718458749353886,
+            -0.005661548115313053,
+            -0.009269599802792072,
+            -0.0033523952588438988,
+            0.019495848566293716,
+            0.017203189432621002,
+            0.004878085572272539,
+            0.006783136632293463,
+            0.008914980106055737,
+            -0.011529270559549332,
+            -0.033944547176361084,
+            -0.008519125171005726,
+            -0.01657641865313053,
+            0.00834593828767538,
+            0.014324994757771492,
+            0.02597797103226185,
+            -0.02150811068713665,
+            -0.008799522183835506,
+            -0.009904616512358189,
+            -0.01571873389184475,
+            -0.010663338005542755,
+            0.010102543979883194,
+            -0.004564700648188591,
+            -0.023437902331352234,
+            0.0010468503460288048,
+            -0.005038901697844267,
+            0.004968802444636822,
+            -0.003812164068222046,
+            0.007273831870406866,
+            0.008956214413046837,
+            -0.011652975343167782,
+            -0.03125603497028351,
+            -0.0035297051072120667,
+            -0.001355596468783915,
+            -0.0028452062979340553,
+            -0.025417177006602287,
+            0.007756279781460762,
+            -0.015990883111953735,
+            -0.02112874947488308,
+            0.013294123113155365,
+            -0.007405783515423536,
+            0.006292441859841347,
+            0.018638163805007935,
+            0.013310616835951805,
+            0.010490152053534985,
+            0.033977534621953964,
+            -0.0011525147128850222,
+            -0.014382723718881607,
+            -0.011314849369227886,
+            0.014531169086694717,
+            -0.017005261033773422,
+            0.014011610299348831,
+            -0.007442894857376814,
+            -0.005884216632694006,
+            -0.0002360696526011452,
+            -0.0017968096071854234,
+            0.008857251144945621,
+            -0.009203623980283737,
+            0.00258336472325027,
+            -0.024575984105467796,
+            0.015413595363497734,
+            -0.016155824065208435,
+            -0.009970592334866524,
+            0.010226248763501644,
+            0.031025119125843048,
+            -0.015339372679591179,
+            -0.012634364888072014,
+            -0.0017504204297438264,
+            0.000949948385823518,
+            0.003818349214270711,
+            -0.006242959760129452,
+            -0.008123270235955715,
+            0.0051378654316067696,
+            0.0007159404922276735,
+            -0.016304269433021545,
+            0.0004020400228910148,
+            0.01111692190170288,
+            -0.028485050424933434,
+            -0.017500080168247223,
+            -0.007150127086788416,
+            -0.01566925272345543,
+            0.008267591707408428,
+            0.030101457610726357,
+            0.007084151264280081,
+            -0.006506863050162792,
+            -0.022794637829065323,
+            0.008246975019574165,
+            -0.027643859386444092,
+            0.02098030410706997,
+            0.07903900742530823,
+            -0.004820356611162424,
+            -0.0025441916659474373,
+            -0.018918560817837715,
+            -0.0014267265796661377,
+            0.0032946662977337837,
+            -0.017384622246026993,
+            0.05429808050394058,
+            -0.0014287884114310145,
+            0.004280179738998413,
+            -0.0076738097704946995,
+            0.010886006988584995,
+            -0.0267201978713274,
+            -0.02134316973388195,
+            -0.020205087959766388,
+            -0.010366447269916534,
+            -0.00543475616723299,
+            -0.0036245454102754593,
+            -0.013541532680392265,
+            0.0018225813983008265,
+            0.011958112940192223,
+            -0.00833769142627716,
+            0.0064078993164002895,
+            -0.024114154279232025,
+            -0.007166620809584856,
+            0.011067439801990986,
+            0.014885789714753628,
+            0.027990233153104782,
+            -0.030134445056319237,
+            -0.014588898047804832,
+            -0.007146003656089306,
+            0.008110899478197098,
+            -0.009302587248384953,
+            0.007644945755600929,
+            -0.0208483524620533,
+            -0.005908957216888666,
+            -0.010036568157374859,
+            0.014094079844653606,
+            0.014836307615041733,
+            -0.0022204979322850704,
+            -0.01058086846023798,
+            0.012090064585208893,
+            0.029227279126644135,
+            0.0015421842690557241,
+            0.009690195322036743,
+            -0.011677715927362442,
+            -0.011347836814820766,
+            0.0036925829481333494,
+            0.03151993826031685,
+            0.00652335723862052,
+            -0.00780576141551137,
+            -0.005785252898931503,
+            0.024641960859298706,
+            -0.004191524814814329,
+            0.01100146397948265,
+            -0.008939720690250397,
+            0.023157505318522453,
+            0.0016442405758425593,
+            0.009978839196264744,
+            0.011422060430049896,
+            -0.011933372355997562,
+            0.02076588198542595,
+            -0.0007344961632043123,
+            0.004589441232383251,
+            -0.0011009711306542158,
+            0.0017730995314195752,
+            -0.004737887065857649,
+            0.0040533882565796375,
+            0.003193640848621726,
+            -0.0013184851268306375,
+            0.010770549066364765,
+            3.614494562498294e-05,
+            0.005777005571871996,
+            -0.018951548263430595,
+            -0.012527154758572578,
+            0.013854917138814926,
+            -0.034406378865242004,
+            0.000833459896966815,
+            0.011191144585609436,
+            -0.02094731666147709,
+            0.02206890471279621,
+            -0.0077810208313167095,
+            -0.023256469517946243,
+            -0.007851120084524155,
+            0.00833769142627716,
+            0.004799739457666874,
+            -0.004968802444636822,
+            0.007884107530117035,
+            -0.007063533645123243,
+            0.005946068558841944,
+            -0.0024184253998100758,
+            0.02037002705037594,
+            -0.022431770339608192,
+            0.020534968003630638,
+            0.0005891432520002127,
+            0.01593315601348877,
+            -0.008849004283547401,
+            0.011273614130914211,
+            0.011908631771802902,
+            0.004498724825680256,
+            0.013722965493798256,
+            0.006003797519952059,
+            -0.014976506121456623,
+            0.014432205818593502,
+            0.008378926664590836,
+            0.009145895019173622,
+            0.02211838588118553,
+            0.014704355970025063,
+            0.011240626685321331,
+            -0.009517008438706398,
+            0.008947967551648617,
+            -0.011504529975354671,
+            -0.0015153816202655435,
+            0.01682382822036743,
+            0.02196994051337242,
+            -0.013945634476840496,
+            -0.004902826622128487,
+            -0.019034016877412796,
+            0.015240409411489964,
+            -0.0006375942612066865,
+            0.012692093849182129,
+            0.018390754237771034,
+            0.004882209002971649,
+            0.008980955928564072,
+            0.019743258133530617,
+            0.04215853661298752,
+            0.004799739457666874,
+            -0.0062182191759347916,
+            0.013978621922433376,
+            0.00795833021402359,
+            -0.005785252898931503,
+            0.00679963082075119,
+            -0.01674135960638523,
+            0.0003015300026163459,
+            0.009500514715909958,
+            -0.0025318211410194635,
+            -0.010020074434578419,
+            0.005983180366456509,
+            0.008741793222725391,
+            0.011413812637329102,
+            -0.012205522507429123,
+            0.010086050257086754,
+            -0.000819543085526675,
+            -0.004807986319065094,
+            -0.02063393034040928,
+            -0.013252888806164265,
+            0.03153643012046814,
+            -0.01348380371928215,
+            -0.008102652616798878,
+            0.02162356860935688,
+            0.017054744064807892,
+            0.004585317801684141,
+            -0.017582550644874573,
+            -0.005851228721439838,
+            0.012461178936064243,
+            -0.0011504529975354671,
+            0.023899732157588005,
+            -0.011240626685321331,
+            0.008634582161903381,
+            -0.014357983134686947,
+            -0.009376809932291508,
+            0.0011514838552102447,
+            0.012370462529361248,
+            0.011471541598439217,
+            0.02749541401863098,
+            0.014720849692821503,
+            -0.004086376167833805,
+            -0.0010834463173523545,
+            0.00791709590703249,
+            0.013533285818994045,
+            0.023108022287487984,
+            -0.0026122292038053274,
+            0.005047148559242487,
+            0.020089630037546158,
+            -0.00554609065875411,
+            0.012535401619970798,
+            -0.022382289171218872,
+            -0.014770331792533398,
+            -0.007574846502393484,
+            0.01099321711808443,
+            -0.011974607594311237,
+            0.017467092722654343,
+            0.0013102381490170956,
+            0.02691812627017498,
+            -0.0026823284570127726,
+            0.0028988115955144167,
+            0.0037647439166903496,
+            0.003985350485891104,
+            -0.0012257066555321217,
+            0.00805729441344738,
+            -0.02101329155266285,
+            0.01046541053801775,
+            -0.008519125171005726,
+            0.003183332271873951,
+            -0.01616406999528408,
+            0.0025895501021295786,
+            -0.007954207248985767,
+            -0.024312080815434456,
+            9.664423851063475e-05,
+            -0.014646627008914948,
+            -0.00529043423011899,
+            0.0018431989010423422,
+            0.004704899154603481,
+            -0.00820573978126049,
+            0.007966578006744385,
+            0.007071780972182751,
+            -0.014927024021744728,
+            0.0009128370438702404,
+            -0.00514198886230588,
+            -0.0001724133180687204,
+            0.0067047905176877975
+          ],
+          "keyphrases": [
+            "conditional computation",
+            "deep networks",
+            "Google Brain",
+            "neural machine translation",
+            "algorithmic and engineering solutions"
+          ],
+          "title": "6 Conclusion"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "7d02abfd-7688-4c92-8ea3-a1e3e39fbd07",
+      "type": "child",
+      "source": {
+        "id": "14339b1f-53f8-479b-9da7-9430b174b9ee",
+        "properties": {
+          "page_content": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer\n\nNoam Shazeer1, Azalia Mirhoseini∗†1, Krzysztof Maziarz∗2, Andy Davis1, Quoc Le1, Geoffrey Hinton1and Jeff Dean1 1Google Brain, {noam,azalia,andydavis,qvl,geoffhinton,jeff}@google.com 2Jagiellonian University, Cracow, krzysztof.maziarz@student.uj.edu.pl\n\nAbstract\n\nThe capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost.\n\n1 Introduction And Related Work 1.1 Conditional Computation\n\nExploiting scale in both training data and model size has been central to the success of deep learning. When datasets are sufficiently large, increasing the capacity (number of parameters) of neural networks can give much better prediction accuracy. This has been shown in domains such as text (Sutskever et al., 2014; Bahdanau et al., 2014; Jozefowicz et al., 2016; Wu et al., 2016), images (Krizhevsky et al., 2012; Le et al., 2012), and audio (Hinton et al., 2012; Amodei et al., 2015). For typical deep learning models, where the entire model is activated for every example, this leads to a roughly quadratic blow-up in training costs, as both the model size and the number of training examples increase. Unfortunately, the advances in computing power and distributed computation fall short of meeting such demand.\n\nVarious forms of conditional computation have been proposed as a way to increase model capacity without a proportional increase in computational costs (Davis & Arel, 2013; Bengio et al., 2013; Eigen et al., 2013; Ludovic Denoyer, 2014; Cho & Bengio, 2014; Bengio et al., 2015; Almahairi et al., 2015). In these schemes, large parts of a network are active or inactive on a per-example basis. The gating decisions may be binary or sparse and continuous, stochastic or deterministic. Various forms of reinforcement learning and back-propagation are proposed for trarining the gating decisions.\n\nWhile these ideas are promising in theory, no work to date has yet demonstrated massive improvements in model capacity, training time, or model quality. We blame this on a combination of the following challenges:\n\nModern computing devices, especially GPUs, are much faster at arithmetic than at branching. Most of the works above recognize this and propose turning on/off large chunks of the network with each gating decision.\n\nLarge batch sizes are critical for performance, as they amortize the costs of parameter transfers and updates. Conditional computation reduces the batch sizes for the conditionally active chunks of the network.\n\nNetwork bandwidth can be a bottleneck. A cluster of GPUs may have computational power thousands of times greater than the aggregate inter-device network bandwidth. To be computationally efficient, the relative computational versus network demands of an algorithm must exceed this ratio. Embedding layers, which can be seen as a form of conditional computation, are handicapped by this very problem. Since the embeddings generally need to be sent across the network, the number of (example, parameter) interactions is limited by network bandwidth instead of computational capacity.\n\nDepending on the scheme, loss terms may be necessary to achieve the desired level of sparsity per-chunk and/or per example. Bengio et al. (2015) use three such terms. These issues can affect both model quality and load-balancing.\n\nModel capacity is most critical for very large data sets. The existing literature on conditional computation deals with relatively small image recognition data sets consisting of up to 600,000 images. It is hard to imagine that the labels of these images provide a sufficient signal to adequately train a model with millions, let alone billions of parameters. In this work, we for the first time address all of the above challenges and finally realize the promise of conditional computation. We obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets.\n\n1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1). All parts of the network are trained jointly by back-propagation.\n\nWhile the introduced technique is generic, in this paper we focus on language modeling and machine translation tasks, which are known to benefit from very large models. In particular, we apply a MoE convolutionally between stacked LSTM layers (Hochreiter & Schmidhuber, 1997), as in Figure 1.\n\nThe MoE is called once for each position in the text, selecting a potentially different combination of experts at each position. The different experts tend to become highly specialized based on syntax and semantics (see Appendix E Table 9). On both language modeling and machine translation benchmarks, we improve on best published results at a fraction of the computational cost.\n\n1.3 Related Work On Mixtures Of Experts\n\nSince its introduction more than two decades ago (Jacobs et al., 1991; Jordan & Jacobs, 1994), the mixture-of-experts approach has been the subject of much research. Different types of expert architectures hae been proposed such as SVMs (Collobert et al., 2002), Gaussian Processes (Tresp, 2001; Theis & Bethge, 2015; Deisenroth & Ng, 2015), Dirichlet Processes (Shahbaba & Neal, 2009), and deep networks. Other work has focused on different expert configurations such as a hierarchical structure (Yao et al., 2009), infinite numbers of experts (Rasmussen & Ghahramani, 2002), and adding experts sequentially (Aljundi et al., 2016). Garmash & Monz (2016) suggest an ensemble model in the format of mixture of experts for machine translation. The gating network is trained on a pre-trained ensemble NMT model. The works above concern top-level mixtures of experts. The mixture of experts is the whole model. Eigen et al. (2013) introduce the idea of using multiple MoEs with their own gating networks as parts of a deep model. It is intuitive that the latter approach is more powerful, since complex problems may contain many sub-problems each requiring different experts. They also allude in their conclusion to the potential to introduce sparsity, turning MoEs into a vehicle for computational computation.\n\nOur work builds on this use of MoEs as a general purpose neural network component. While Eigen et al. (2013) uses two stacked MoEs allowing for two sets of gating decisions, our convolutional application of the MoE allows for different gating decisions at each position in the text. We also realize sparse gating and demonstrate its use as a practical way to massively increase model capacity.\n\n2 The Structure Of The Mixture-Of-Experts Layer\n\nThe Mixture-of-Experts (MoE) layer consists of a set of n \"expert networks\" E1, · · · , En, and a \"gating network\" G whose output is a sparse n-dimensional vector. Figure 1 shows an overview of the MoE module. The experts are themselves neural networks, each with their own parameters.\n\nAlthough in principle we only require that the experts accept the same sized inputs and produce the same-sized outputs, in our initial investigations in this paper, we restrict ourselves to the case where the models are feed-forward networks with identical architectures, but with separate parameters.\n\nLet us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows:\n\n$$y=\\sum_{i=1}^{n}G(x){i}E{i}(x)$$ $$(\\mathbf{l})$$ G(x)iEi(x) (1) We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example. If the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network. In the following we focus on ordinary MoEs. We provide more details on hierarchical MoEs in Appendix B. Our implementation is related to other models of conditional computation. A MoE whose experts are simple weight matrices is similar to the parameterized weight matrix proposed in (Cho & Bengio, 2014). A MoE whose experts have one hidden layer is similar to the block-wise dropout described in (Bengio et al., 2015), where the dropped-out layer is sandwiched between fully-activated layers.\n\n2.1 Gating Network\n\nSoftmax Gating: A simple choice of non-sparse gating function (Jordan & Jacobs, 1994) is to multiply the input by a trainable weight matrix Wg and then apply the Sof tmax function.\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ $\\eqref{eq:walpha}$. Gσ(x) = Sof tmax(x · Wg) (2) Noisy Top-K Gating: We add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to −∞ (which causes the corresponding gate values to equal 0). The sparsity serves to save computation, as described above. While this form of sparsity creates some theoretically scary discontinuities in the output of gating function, we have not yet observed this to be a problem in practice. The noise term helps with load balancing, as will be discussed in Appendix A. The amount of noise per component is controlled by a second trainable weight matrix Wnoise.\n\n$$G(x)=S o f t m a x(K e e p T o p K(H(x),k))$$ $$(4)$$ $$(S)$$ G(x) = Sof tmax(KeepT opK(H(x), k)) (3) $$H(x){i}=(x\\cdot W{g}){i}+S t a n d a r d N o r m a l()\\cdot S o f t p l u s((x\\cdot W{n o i s e}){i})$$ $KeepTopK(v,k){i}=\\begin{cases}v_{i}&\\text{if}v_{i}\\text{is in the top}k\\text{elements of}v.\\ -\\infty&\\text{otherwise.}\\end{cases}$ Training the Gating Network We train the gating network by simple back-propagation, along with the rest of the model. If we choose k > 1, the gate values for the top k experts have nonzero derivatives with respect to the weights of the gating network. This type of occasionally-sensitive behavior is described in (Bengio et al., 2013) with respect to noisy rectifiers. Gradients also backpropagate through the gating network to its inputs. Our method differs here from (Bengio et al., 2015) who use boolean gates and a REINFORCE-style approach to train the gating network.\n\n3 Addressing Performance Challenges 3.1 The Shrinking Batch Problem\n\nOn modern CPUs and GPUs, large batch sizes are necessary for computational efficiency, so as to amortize the overhead of parameter loads and updates. If the gating network chooses k out of n experts for each example, then for a batch of b examples, each expert receives a much smaller batch of approximately kb n b examples. This causes a naive MoE implementation to become very inefficient as the number of experts increases. The solution to this shrinking batch problem is to make the original batch size as large as possible. However, batch size tends to be limited by the memory necessary to store activations between the forwards and backwards passes. We propose the following techniques for increasing the batch size: Mixing Data Parallelism and Model Parallelism: In a conventional distributed training setting, multiple copies of the model on different devices asynchronously process distinct batches of data, and parameters are synchronized through a set of parameter servers. In our technique, these different batches run synchronously so that they can be combined for the MoE layer. We distribute the standard layers of the model and the gating network according to conventional data-parallel schemes, but keep only one shared copy of each expert. Each expert in the MoE layer receives a combined batch consisting of the relevant examples from all of the data-parallel input batches. The same set of devices function as data-parallel replicas (for the standard layers and the gating networks) and as model-parallel shards (each hosting a subset of the experts). If the model is distributed over d devices, and each device processes a batch of size b, each expert receives a batch of approximately kbd nexamples. Thus, we achieve a factor of d improvement in expert batch size.\n\nIn the case of a hierarchical MoE (Section B), the primary gating network employs data parallelism, and the secondary MoEs employ model parallelism. Each secondary MoE resides on one device.\n\nThis technique allows us to increase the number of experts (and hence the number of parameters) by proportionally increasing the number of devices in the training cluster. The total batch size increases, keeping the batch size per expert constant. The memory and bandwidth requirements per device also remain constant, as do the step times, as does the amount of time necessary to process a number of training examples equal to the number of parameters in the model. It is our goal to train a trillionparameter model on a trillion-word corpus. We have not scaled our systems this far as of the writing of this paper, but it should be possible by adding more hardware.\n\nTaking Advantage of Convolutionality: In our language models, we apply the same MoE to each time step of the previous layer. If we wait for the previous layer to finish, we can apply the MoE to all the time steps together as one big batch. Doing so increases the size of the input batch to the MoE layer by a factor of the number of unrolled time steps.\n\nIncreasing Batch Size for a Recurrent MoE: We suspect that even more powerful models may involve applying a MoE recurrently. For example, the weight matrices of a LSTM or other RNN could be replaced by a MoE. Sadly, such models break the convolutional trick from the last paragraph, since the input to the MoE at one timestep depends on the output of the MoE at the previous timestep. Gruslys et al. (2016) describe a technique for drastically reducing the number of stored activations in an unrolled RNN, at the cost of recomputing forward activations. This would allow for a large increase in batch size.\n\n3.2 Network Bandwidth\n\nAnother major performance concern in distributed computing is network bandwidth. Since the experts are stationary (see above) and the number of gating parameters is small, most of the communication involves sending the inputs and outputs of the experts across the network. To maintain computational efficiency, the ratio of an expert's computation to the size of its input and output must exceed the ratio of computational to network capacity of the computing device. For GPUs, this may be thousands to one. In our experiments, we use experts with one hidden layer containing thousands of RELU-activated units. Since the weight matrices in the expert have sizes input_size×hidden_size and hidden_size × output_size, the ratio of computation to input and output is equal to the size of the hidden layer. Conveniently, we can increase computational efficiency simply by using a larger hidden layer, or more hidden layers.\n\n4 Balancing Expert Utilization\n\nWe have observed that the gating network tends to converge to a state where it always produces large weights for the same few experts. This imbalance is self-reinforcing, as the favored experts are trained more rapidly and thus are selected even more by the gating network. Eigen et al. (2013) describe the same phenomenon, and use a hard constraint at the beginning of training to avoid this local minimum. Bengio et al. (2015) include a soft constraint on the batch-wise average of each gate.1 We take a soft constraint approach. We define the importance of an expert relative to a batch of training examples to be the batchwise sum of the gate values for that expert. We define an additional loss Limportance, which is added to the overall loss function for the model. This loss is equal to the square of the coefficient of variation of the set of importance values, multiplied by a hand-tuned scaling factor wimportance. This additional loss encourages all experts to have equal importance.\n\n$$I m p o r t a n c e(X)=\\sum_{x\\in X}G(x)$$ $$L_{importance}(X)=w_{importance}\\cdot CV(Importance(X))^{2}\\tag{7}$$ $$(6)$$\n\nWhile this loss function can ensure equal importance, experts may still receive very different numbers of examples. For example, one expert may receive a few examples with large weights, and another may receive many examples with small weights. This can cause memory and performance problems on distributed hardware. To solve this problem, we introduce a second loss function, Lload , which ensures balanced loads. Appendix A contains the definition of this function, along with experimental results.\n\n5 Experiments 5.1 1 Billion Word Language Modeling Benchmark\n\nDataset: This dataset, introduced by (Chelba et al., 2013) consists of shuffled unique sentences from news articles, totaling approximately 829 million words, with a vocabulary of 793,471 words.\n\nPrevious State-of-the-Art: The best previously published results (Jozefowicz et al., 2016) use models consisting of one or more stacked Long Short-Term Memory (LSTM) layers (Hochreiter & Schmidhuber, 1997; Gers et al., 2000). The number of parameters in the LSTM layers of these models vary from 2 million to 151 million. Quality increases greatly with parameter count, as do computational costs. Results for these models form the top line of Figure 2-right. MoE Models: Our models consist of two stacked LSTM layers with a MoE layer between them (see Figure 1). We vary the sizes of the layers and the number of experts. For full details on model architecture, training regimen, additional baselines and results, see Appendix C.\n\nLow Computation, Varied Capacity: To investigate the effects of adding capacity, we trained\n\na series of MoE models all with roughly equal computational costs: about 8 million multiply-andadds per training example per timestep in the forwards pass, excluding the softmax layer. We call this metric (ops/timestep). We trained models with flat MoEs containing 4, 32, and 256 experts, and models with hierarchical MoEs containing 256, 1024, and 4096 experts. Each expert had about 1 million parameters. For all the MoE layers, 4 experts were active per input.\n\nThe results of these models are shown in Figure 2-left. The model with 4 always-active experts performed (unsurprisingly) similarly to the computationally-matched baseline models, while the largest of the models (4096 experts) achieved an impressive 24% lower perplexity on the test set. Figure 2: Model comparison on 1-Billion-Word Language-Modeling Benchmark. On the left, we plot test perplexity as a function of model capacity for models with similar computational budgets of approximately 8-million-ops-per-timestep. On the right, we plot test perplexity as a function of computational budget. The top line represents the LSTM models from (Jozefowicz et al., 2016).\n\nThe bottom line represents 4-billion parameter MoE models with different computational budgets.\n\nVaried Computation, High Capacity: In addition to the largest model from the previous section, we trained two more MoE models with similarly high capacity (4 billion parameters), but higher computation budgets. These models had larger LSTMs, and fewer but larger and experts. Details\n\nTest Test #Parameters ops/timestep Training TFLOPS Perplexity Perplexity excluding embedding Time /GPU 10 epochs 100 epochs and softmax layers 10 epochs Best Published Results 34.7 30.6 151 million 151 million 59 hours, 32 k40s 1.09 Low-Budget MoE Model 34.1 4303 million 8.9 million 15 hours, 16 k40s 0.74 Medium-Budget MoE Model 31.3 4313 million 33.8 million 17 hours, 32 k40s 1.22 High-Budget MoE Model 28.0 4371 million 142.7 million 47 hours, 32 k40s 1.56\n\nTable 1: Summary of high-capacity MoE-augmented models with varying computational budgets, vs. best previously published results (Jozefowicz et al., 2016). Details in Appendix C. can be found in Appendix C.2. Results of these three models form the bottom line of Figure 2-right.\n\nTable 1 compares the results of these models to the best previously-published result on this dataset .\n\nEven the fastest of these models beats the best published result (when controlling for the number of training epochs), despite requiring only 6% of the computation.\n\nComputational Efficiency: We trained our models using TensorFlow (Abadi et al., 2016) on clusters containing 16-32 Tesla K40 GPUs. For each of our models, we determine computational efficiency in TFLOPS/GPU by dividing the number of floating point operations required to process one training batch by the observed step time and the number of GPUs in the cluster. The operation counts used here are higher than the ones we report in our ops/timestep numbers in that we include the backwards pass, we include the importance-sampling-based training of the softmax layer, and we count a multiply-and-add as two separate operations. For all of our MoE models, the floating point operations involved in the experts represent between 37% and 46% of the total.\n\nFor our baseline models wtih no MoE, observed computational efficiency ranged from 1.07-1.29 TFLOPS/GPU. For our low-computation MoE models, computation efficiency ranged from 0.740.90 TFLOPS/GPU, except for the 4-expert model which did not make full use of the available parallelism. Our highest-computation MoE model was more efficient at 1.56 TFLOPS/GPU, likely due to the larger matrices. These numbers represent a significant fraction of the theoretical maximum of 4.29 TFLOPS/GPU claimed by NVIDIA. Detailed results are in Appendix C, Table 7.\n\n5.2 100 BILLION WORD GOOGLE NEWS CORPUS\n\nOn the 1-billion-word corpus, adding additional capacity seems to produce diminishing returns as the number of parameters in the MoE layer exceeds 1 billion, as can be seen in Figure 2-left. We hypothesized that for a larger training set, even higher capacities would produce significant quality improvements. We constructed a similar training set consisting of shuffled unique sentences from Google's internal news corpus, totalling roughly 100 billion words. Similarly to the previous section, we tested a series of models with similar computational costs of about 8 million ops/timestep. In addition to a baseline LSTM model, we trained models augmented with MoE layers containing 32, 256, 1024, 4096, 16384, 65536, and 131072 experts. This corresponds to up to 137 billion parameters in the MoE layer. Details on architecture, training, and results are given in Appendix D.\n\nResults: Figure 3 shows test perplexity as a function of capacity after training on 10 billion words (top line) and 100 billion words (bottom line). When training over the full 100 billion words, test perplexity improves significantly up to 65536 experts (68 billion parameters), dropping 39% lower than the computationally matched baseline, but degrades at 131072 experts, possibly a result of too much sparsity. The widening gap between the two lines demonstrates (unsurprisingly) that increased model capacity helps more on larger training sets. Even at 65536 experts (99.994% layer sparsity), computational efficiency for the model stays at a respectable 0.72 TFLOPS/GPU.\n\n5.3 Machine Translation (Single Language Pair)\n\nModel Architecture: Our model was a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decreased the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We inserted MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). Each MoE layer contained up to 2048 experts each with about two million parameters, adding a total of about 8 billion parameters to the models. Further details on model architecture, testing procedure and results can be found in Appendix E.\n\nDatasets: We benchmarked our method on the WMT'14 En→Fr and En→De corpora, whose training sets have 36M sentence pairs and 5M sentence pairs, respectively. The experimental protocols were also similar to those in (Wu et al., 2016): newstest2014 was used as the test set to compare against previous work (Luong et al., 2015a; Zhou et al., 2016; Wu et al., 2016), while the combination of newstest2012 and newstest2013 was used as the development set. We also tested the same model on a Google's Production English to French data.\n\nTable 2: Results on WMT'14 En→ Fr newstest2014 (bold values represent best results). Model Test Test ops/timenstep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.69 40.35 85M 8.7B 3 days/64 k40s MoE with 2048 Experts (longer training) 2.63 40.56 85M 8.7B 6 days/64 k40s GNMT (Wu et al., 2016) 2.79 39.22 214M 278M 6 days/96 k80s GNMT+RL (Wu et al., 2016) 2.96 39.92 214M 278M 6 days/96 k80s PBMT (Durrani et al., 2014) 37.0 LSTM (6-layer) (Luong et al., 2015b) 31.5 LSTM (6-layer+PosUnk) (Luong et al., 2015b) 33.1 DeepAtt (Zhou et al., 2016) 37.7 DeepAtt+PosUnk (Zhou et al., 2016) 39.2\n\nTable 3: Results on WMT'14 En → De newstest2014 (bold values represent best results). Model Test Test ops/timestep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 4.64 26.03 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 5.25 24.91 214M 278M 1 day/96 k80s GNMT +RL (Wu et al., 2016) 8.08 24.66 214M 278M 1 day/96 k80s PBMT (Durrani et al., 2014) 20.7 DeepAtt (Zhou et al., 2016) 20.6\n\nTable 4: Results on the Google Production En→ Fr dataset (bold values represent best results). Model Eval Eval Test Test ops/timestep Total Training Perplexity BLEU Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.60 37.27 2.69 36.57 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 2.78 35.80 2.87 35.56 214M 278M 6 days/96 k80s\n\nResults: Tables 2, 3, and 4 show the results of our largest models, compared with published results. Our approach achieved BLEU scores of 40.56 and 26.03 on the WMT'14 En→Fr and En→De benchmarks. As our models did not use RL refinement, these results constitute significant gains of 1.34 and 1.12 BLEU score on top of the strong baselines in (Wu et al., 2016). The perplexity scores are also better.2 On the Google Production dataset, our model achieved 1.01 higher test BLEU score even after training for only one sixth of the time.\n\n5.4 Multilingual Machine Translation\n\nDataset: (Johnson et al., 2016) train a single GNMT (Wu et al., 2016) model on a very large combined dataset of twelve language pairs. Results are somewhat worse than those for 12 separately trained single-pair GNMT models. This is not surprising, given that the twelve models have 12 times the capacity and twelve times the aggregate training of the one model. We repeat this experiment with a single MoE-augmented model. See Appendix E for details on model architecture.\n\nWe train our model on the same dataset as (Johnson et al., 2016) and process the same number of training examples (about 3 billion sentence pairs). Our training time was shorter due to the lower computational budget of our model.\n\nResults: Results for the single-pair GNMT models, the multilingual GNMT model and the multilingual MoE model are given in Table 5. The MoE model achieves 19% lower perplexity on the dev set than the multilingual GNMT model. On BLEU score, the MoE model significantly beats the multilingual GNMT model on 11 of the 12 language pairs (by as much as 5.84 points), and even beats the monolingual GNMT models on 8 of 12 language pairs. The poor performance on English → Korean seems to be a result of severe overtraining, as for the rarer language pairs a small number of real examples were highly oversampled in the training corpus.\n\nTable 5: Multilingual Machine Translation (bold values represent best results). GNMT-Mono GNMT-Multi MoE-Multi MoE-Multi vs. GNMT-Multi Parameters 278M / model 278M 8.7B ops/timestep 212M 212M 102M training time, hardware various 21 days, 96 k20s 12 days, 64 k40s Perplexity (dev) 4.14 3.35 -19% French → English Test BLEU 36.47 34.40 37.46 +3.06 German → English Test BLEU 31.77 31.17 34.80 +3.63 Japanese → English Test BLEU 23.41 21.62 25.91 +4.29 Korean → English Test BLEU 25.42 22.87 28.71 +5.84 Portuguese → English Test BLEU 44.40 42.53 46.13 +3.60 Spanish → English Test BLEU 38.00 36.04 39.39 +3.35 English → French Test BLEU 35.37 34.00 36.59 +2.59 English → German Test BLEU 26.43 23.15 24.53 +1.38 English → Japanese Test BLEU 23.66 21.10 22.78 +1.68 English → Korean Test BLEU 19.75 18.41 16.62 -1.79 English → Portuguese Test BLEU 38.40 37.35 37.90 +0.55 English → Spanish Test BLEU 34.50 34.25 36.21 +1.96\n\n6 Conclusion\n\nThis work is the first to demonstrate major wins from conditional computation in deep networks.\n\nWe carefully identified the design considerations and challenges of conditional computing and addressed them with a combination of algorithmic and engineering solutions. While we focused on text, conditional computation may help in other domains as well, provided sufficiently large training sets. We look forward to seeing many novel implementations and applications of conditional computation in the years to come.\n\nAcknowledgments\n\nWe would like to thank all of the members of the Google Brain and Google Translate teams who helped us with this project, in particular Zhifeng Chen, Yonghui Wu, and Melvin Johnson. Thanks also to our anonymous ICLR reviewers for the helpful suggestions on making this paper better.\n\n2Reported perplexities relative to the tokenization used by both our models and GNMT.\n\nReferences\n\nMartín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Gordon Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda B. Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. CoRR, abs/1603.04467, 2016. URL http://arxiv.org/abs/1603.04467.\n\nRahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. CoRR, abs/1611.06194, 2016. URL http://arxiv.org/abs/1611.\n\n06194.\n\nA. Almahairi, N. Ballas, T. Cooijmans, Y. Zheng, H. Larochelle, and A. Courville. Dynamic Capacity Networks. ArXiv e-prints, November 2015.\n\nDario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan, Christopher Fougner, Tony Han, Awni Y. Hannun, Billy Jun, Patrick LeGresley, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595, 2015. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.\n\nEmmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297, 2015.\n\nYoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.\n\nCiprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling.\n\narXiv preprint arXiv:1312.3005, 2013.\n\nK. Cho and Y. Bengio. Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning. ArXiv e-prints, June 2014. Ronan Collobert, Samy Bengio, and Yoshua Bengio. A parallel mixture of SVMs for very large scale problems. Neural Computing, 2002.\n\nAndrew Davis and Itamar Arel. Low-rank approximations for conditional feedforward computation in deep neural networks. arXiv preprint arXiv:1312.4461, 2013. Marc Peter Deisenroth and Jun Wei Ng. Distributed Gaussian processes. In ICML, 2015.\n\nJohn Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization, 2010.\n\nNadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. Edinburgh's phrase-based machine translation systems for wmt-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation, 2014.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013. Ekaterina Garmash and Christof Monz. Ensemble learning for multi-source neural machine translation. In staff.science.uva.nl/c.monz, 2016.\n\nFelix A. Gers, Jürgen A. Schmidhuber, and Fred A. Cummins. Learning to forget: Continual prediction with lstm. Neural Computation, 2000.\n\nAudrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. Memory-efficient backpropagation through time. CoRR, abs/1606.03401, 2016. URL http://arxiv.org/ abs/1606.03401. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2015.\n\nGeoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 2012. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.\n\nSergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.\n\nRobert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computing, 1991.\n\nMelvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation.\n\nCoRR, abs/1611.04558, 2016. URL http://arxiv.org/abs/1611.04558.\n\nMichael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm.\n\nNeural Computing, 1994. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.\n\nDiederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.\n\nReinhard Kneser and Hermann. Ney. Improved backingoff for m-gram language modeling., 1995.\n\nAlex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.\n\nQuoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeffrey Dean, and Andrew Y. Ng. Building high-level features using large scale unsupervised learning. In ICML, 2012. Patrick Gallinari Ludovic Denoyer. Deep sequential neural network. arXiv preprint arXiv:1410.0510, 2014.\n\nMinh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attentionbased neural machine translation. EMNLP, 2015a.\n\nMinh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the rare word problem in neural machine translation. ACL, 2015b.\n\nCarl Edward Rasmussen and Zoubin Ghahramani. Infinite mixtures of Gaussian process experts.\n\nNIPS, 2002.\n\nHasim Sak, Andrew W Senior, and Françoise Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH, pp. 338–342, 2014.\n\nMike Schuster and Kaisuke Nakajima. Japanese and Korean voice search. ICASSP, 2012.\n\nBabak Shahbaba and Radford Neal. Nonlinear models using dirichlet process mixtures. JMLR, 2009.\n\nIlya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks.\n\nIn NIPS, 2014.\n\nLucas Theis and Matthias Bethge. Generative image modeling using spatial LSTMs. In NIPS, 2015. Volker Tresp. Mixtures of Gaussian Processes. In NIPS, 2001.\n\nYonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.\n\nBangpeng Yao, Dirk Walther, Diane Beck, and Li Fei-fei. Hierarchical mixture of classification experts uncovers interactions between brain regions. In NIPS. 2009.\n\nWojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization.\n\narXiv preprint arXiv:1409.2329, 2014.\n\nJie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with fast-forward connections for neural machine translation. arXiv preprint arXiv:1606.04199, 2016.\n\nAppendices A Load-Balancing Loss\n\nAs discussed in section 4, for load-balancing purposes, we want to define an additional loss function to encourage experts to receive roughly equal numbers of training examples. Unfortunately, the number of examples received by an expert is a discrete quantity, so it can not be used in backpropagation. Instead, we define a smooth estimator Load(X) of the number of examples assigned to each expert for a batch X of inputs. The smoothness allows us to back-propagate gradients through the estimator. This is the purpose of the noise term in the gating function. We define P(x, i) as the probability that G(x)iis nonzero, given a new random choice of noise on element i, but keeping the already-sampled choices of noise on the other elements. To compute P(x, i), we note that the G(x)iis nonzero if and only if H(x)iis greater than the k th-greatest element of H(x) excluding itself. The probability works out to be:\n\n$$P(x,i)=P r\\Big((x\\cdot W_{g}){i}+S t a n d x N o r a l()\\cdot S o f t p u s((x\\cdot W{n o i s e})_{i})$$ $$>k t h_e x c l u d i n(H(x),k,i)\\Big)$$ $$({\\boldsymbol{8}})$$\n\n$$(9)$$ $$(10)$$ (8) Where kth_excluding(v, k, i) means the kth highest component of v, excluding component i. Simplifying, we get:\n\n$$P(x,i)=\\Phi{\\Big(}{\\frac{(x\\cdot W_{g}){i}-k t h_c x c l u d i n g(H(x),k,i)}{S o f t p l u s((x\\cdot W{n o i s e})_{i})}}{\\Big)}$$\n\nWhere Φ is the CDF of the standard normal distribution.\n\n$$L o a d(X){i}=\\sum{x\\in X}P(x,i)$$\n\n$$(11)$$ P(x, i) (10) We can now define the load loss to be the square of the coefficient of variation of the load vector, multiplied by a hand-tuned scaling factor wload.\n\n$$L_{l o a d}(X)=w_{l o a d}\\cdot C V(L o a d(X))^{2}$$ Lload(X) = wload · CV (Load(X))2(11) Initial Load Imbalance: To avoid out-of-memory errors, we need to initialize the network in a state of approximately equal expert load (since the soft constraints need some time to work). To accomplish this, we initialize the matrices Wg and Wnoise to all zeros, which yields no signal and some noise.\n\nExperiments: We trained a set of models with identical architecture (the MoE-256 model described in Appendix C), using different values of wimportance and wload. We trained each model for 10 epochs, then measured perplexity on the test set. We also measured the coefficients of variation in Importance and Load, as well as ratio of the load on the most overloaded expert to the average load. This last value is significant for load balancing purposes on distributed hardware. All of these metrics were averaged over several training batches.\n\nwimportance wload Test Perplexity CV (Importance(X)) CV (Load(X)) max(Load(X)) mean(Load(X)) 0.0 0.0 39.8 3.04 3.01 17.80 0.2 0.0 35.6 0.06 0.17 1.47 0.0 0.2 35.7 0.22 0.04 1.15 0.1 0.1 35.6 0.06 0.05 1.14 0.01 0.01 35.7 0.48 0.11 1.37 1.0 1.0 35.7 0.03 0.02 1.07\n\nTable 6: Experiments with different combinations of losses.\n\n13 Results: Results are reported in Table 6. All the combinations containing at least one the two losses led to very similar model quality, where having no loss was much worse. Models with higher values of wload had lower loads on the most overloaded expert.\n\nB Hierachical Mixture Of Experts\n\nIf the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network.3If the hierarchical MoE consists of a groups of b experts each, we denote the primary gating network by Gprimary, the secondary gating networks by (G1, G2..Ga), and the expert networks by (E0,0, E0,1..Ea,b). The output of the MoE is given by:\n\n$$y_{H}=\\sum_{i=1}^{a}\\sum_{j=1}^{b}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}\\cdot E{i,j}(x)$$ $$(12)$$ $$(13)$$ $$(14)$$\n\nOur metrics of expert utilization change to the following:\n\n$$I m p o r t a n c e_{H}(X){i,j}=\\sum{x\\in X}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}$$ $$L o a d{H}(X){i,j}={\\frac{L o a d{p r i m a r y}(X){i}\\cdot L o a d{i}(X^{(i)})_{j}}{|X^{(i)}|}}$$\n\nLoadprimary and Loadi deonte the Load functions for the primary gating network and i th secondary gating network respectively. X(i) denotes the subset of X for which Gprimary(x)i > 0.\n\nIt would seem simpler to let LoadH(X)i,j = Loadi(Xi)j , but this would not have a gradient with respect to the primary gating network, so we use the formulation above.\n\nC 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS C.1 8-MILLION-OPERATIONS-PER-TIMESTEP MODELS Model Architecture: Our model consists of five layers: a word embedding layer, a recurrent Long Short-Term Memory (LSTM) layer (Hochreiter & Schmidhuber, 1997; Gers et al., 2000), a MoE layer, a second LSTM layer, and a softmax layer. The dimensionality of the embedding layer, the number of units in each LSTM layer, and the input and output dimensionality of the MoE layer are all equal to 512. For every layer other than the softmax, we apply drouput (Zaremba et al., 2014) to the layer output, dropping each activation with probability DropP rob, otherwise dividing by (1 − DropP rob). After dropout, the output of the previous layer is added to the layer output.\n\nThis residual connection encourages gradient flow (He et al., 2015).\n\nMoE Layer Architecture: Each expert in the MoE layer is a feed forward network with one ReLU-activated hidden layer of size 1024 and an output layer of size 512. Thus, each expert contains [512 ∗ 1024] + [1024 ∗ 512] = 1M parameters. The output of the MoE layer is passed through a sigmoid function before dropout. We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts.\n\nWe call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers. Thus, each example is processed by exactly 4 experts for a total of 4M ops/timestep. The two LSTM layers contribute 2M ops/timestep each for the desired total of 8M.\n\n3 We have not found the need for deeper hierarchies.\n\nComputationally-Matched Baselines: The MoE-4 model does not employ sparsity, since all 4 experts are always used. In addition, we trained four more computationally-matched baseline models with no sparsity:\n\nMoE-1-Wide: The MoE layer consists of a single \"expert\" containing one ReLU-activated hidden layer of size 4096.\n\nMoE-1-Deep: The MoE layer consists of a single \"expert\" containing four ReLU-activated hidden layers, each with size 1024.\n\n4xLSTM-512: We replace the MoE layer with two additional 512-unit LSTM layers.\n\nLSTM-2048-512: The model contains one 2048-unit LSTM layer (and no MoE). The output of the LSTM is projected down to 512 dimensions (Sak et al., 2014). The next timestep of the LSTM receives the projected output. This is identical to one of the models published in (Jozefowicz et al., 2016). We re-ran it to account for differences in training regimen, and obtained results very similar to the published ones. Training: The models were trained on a cluster of 16 K40 GPUs using the synchronous method described in Section 3. Each batch consisted of a set of sentences totaling roughly 300,000 words. In the interest of time, we limited training to 10 epochs, (27,000 steps). Training took 12-16 hours for all models, except for MoE-4, which took 18 hours (since all the expert computation was performed on only 4 of 16 GPUs). We used the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 1000 training steps, and decreased after that so as to be proportional to the inverse square root of the step number. The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016). For each model, we performed a hyper-parmeter search to find the best dropout probability, in increments of 0.1.\n\nTo ensure balanced expert utilization we set wimportance = 0.1 and wload = 0.1, as described in Section 4 and Appendix A.\n\nResults: We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol. Results are reported in Table 7. For each model, we report the test perplexity, the computational budget, the parameter counts, the value of DropP rob, and the computational efficiency.\n\nModel Test Test ops/timestep #Params excluding Total Drop- TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params P rob per GPU 10 epochs (final) (millions) (billions) (observed) Kneser-Ney 5-gram* 67.6 0.00001 1.8 LSTM-512-512* 54.1 2.4 2.4 0.8 0.1 LSTM-1024-512* 48.2 4.7 4.7 0.8 0.1 LSTM-2048-512* 45.0 43.7 9.4 9.4 0.8 0.1 0.61 LSTM-2048-512 44.7 9.4 9.4 0.8 0.1 1.21 4xLSTM-512 46.0 8.4 8.4 0.8 0.1 1.07 MoE-1-Wide 46.1 8.4 8.4 0.8 0.1 1.29 MoE-1-Deep 45.7 8.4 8.4 0.8 0.1 1.29 MoE-4 45.0 8.4 8.4 0.8 0.1 0.52 MoE-32 39.7 8.4 37.8 0.9 0.1 0.87 MoE-256 35.7 8.6 272.9 1.1 0.1 0.81 MoE-256-h 36.0 8.4 272.9 1.1 0.1 0.89 MoE-1024-h 34.6 8.5 1079.0 1.9 0.2 0.90 MoE-4096-h 34.1 8.9 4303.4 5.1 0.2 0.74 2xLSTM-8192-1024* 34.7 30.6 151.0 151.0 1.8 0.25 1.09 MoE-34M 31.3 33.8 4313.9 6.0 0.3 1.22 MoE-143M 28.0 142.7 4371.1 6.0 0.4 1.56\n\nC.2 More Expensive Models\n\nWe ran two additional models (MoE-34M and MoE-143M) to investigate the effects of adding more computation in the presence of a large MoE layer. These models have computation budgets of 34M and 143M ops/timestep. Similar to the models above, these models use a MoE layer between two LSTM layers. The dimensionality of the embedding layer, and the input and output dimensionality of the MoE layer are set to 1024 instead of 512. For MoE-34M, the LSTM layers have 1024 units.\n\nFor MoE-143M, the LSTM layers have 4096 units and an output projection of size 1024 (Sak et al., 2014). MoE-34M uses a hierarchical MoE layer with 1024 experts, each with a hidden layer of size 2048. MoE-143M uses a hierarchical MoE layer with 256 experts, each with a hidden layer of size 8192. Both models have 4B parameters in the MoE layers. We searched for the best DropP rob for each model, and trained each model for 10 epochs.\n\nThe two models achieved test perplexity of 31.3 and 28.0 respectively, showing that even in the presence of a large MoE, more computation is still useful. Results are reported at the bottom of Table 7. The larger of the two models has a similar computational budget to the best published model from the literature, and training times are similar. Comparing after 10 epochs, our model has a lower test perplexity by 18%.\n\nD 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS Model Architecture: The models are similar in structure to the 8-million-operations-per-timestep models described in the previous section. We vary the number of experts between models, using an ordinary MoE layer with 32 experts and hierarchical MoE layers with 256, 1024, 4096, 16384, 65536 and 131072 experts. For the hierarchical MoE layers, the first level branching factors are 32, 32, 64, 128, 256 and 256, respectively.\n\nTraining: Models are trained on a cluster of 32 Tesla K40 GPUs, except for the last two models, which are trained on clusters of 64 and 128 GPUs so as to have enough memory for all the parameters. For all models, training batch sizes are approximately 2.5 million words. Models are trained once-through over about 100 billion words. We implement several memory optimizations in order to fit up to 1 billion parameters per GPU. First, we do not store the activations of the hidden layers of the experts, but instead recompute them on the backwards pass. Secondly, we modify the optimizer on the expert parameters to require less auxiliary storage: The Adam optimizer (Kingma & Ba, 2015) keeps first and second moment estimates of the perparameter gradients. This triples the required memory. To avoid keeping a first-moment estimator, we set β1 = 0. To reduce the size of the second moment estimator, we replace it with a factored approximation. For a matrix of parameters, instead of maintaining a full matrix of second-moment estimators, we maintain vectors of row-wise and column-wise averages of that matrix. At each step, the matrix of estimators is taken to be the outer product of those two vectors divided by the mean of either one. This technique could similarly be applied to Adagrad (Duchi et al., 2010).\n\nModel Test Test ops/timestep #Params excluding Total TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params per GPU .1 epochs 1 epoch (millions) (billions) (observed) Kneser-Ney 5-gram 67.1 45.3 0.00001 76.0 4xLSTM-512 54.5 47.0 8.4 8.4 0.1 1.23 MoE-32 48.5 40.4 8.4 37.8 0.1 0.83 MoE-256-h 42.8 35.3 8.4 272.9 0.4 1.11 MoE-1024-h 40.3 32.7 8.5 1079.0 1.2 1.14 MoE-4096-h 38.9 30.9 8.6 4303.4 4.4 1.07 MoE-16384-h 38.2 29.7 8.8 17201.0 17.3 0.96 MoE-65536-h 38.2 28.9 9.2 68791.0 68.9 0.72 MoE-131072-h 39.8 29.2 9.7 137577.6 137.7 0.30\n\nResults: We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8. Perplexity after 100 billion training words is 39% lower for the 68-billion-parameter MoE model than for the baseline model. It is notable that the measured computational efficiency of the largest model (0.30 TFLOPS/GPU) is very low compared to the other models. This is likely a result of the fact that, for purposes of comparison to the other models, we did not increase the training batch size proportionally to the number of GPUs. For comparison, we include results for a computationally matched baseline model consisting of 4 LSTMs, and for an unpruned 5-gram model with Kneser-Ney smoothing (Kneser & Ney, 1995).4\n\nE Machine Translation - Experimental Details\n\nModel Architecture for Single Language Pair MoE Models: Our model is a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decrease the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We insert MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). We use an attention mechanism between the encoder and decoder, with the first decoder LSTM receiving output from and providing input for the attention 5. All of the layers in our model have input and output dimensionality of 512. Our LSTM layers have 2048 hidden units, with a 512-dimensional output projection. We add residual connections around all LSTM and MoE layers to encourage gradient flow (He et al., 2015). Similar to GNMT, to effectively deal with rare words, we used subword units (also known as \"wordpieces\") (Schuster & Nakajima, 2012) for inputs and outputs in our system.\n\nWe use a shared source and target vocabulary of 32K wordpieces. We also used the same beam search technique as proposed in (Wu et al., 2016).\n\nWe train models with different numbers of experts in the MoE layers. In addition to a baseline model with no MoE layers, we train models with flat MoE layers containing 32 experts, and models with hierarchical MoE layers containing 512 and 2048 experts. The flat MoE layers use k = 4 and the hierarchical MoE models use k = 2 at each level of the gating network. Thus, each input is processed by exactly 4 experts in each MoE layer. Each expert in the MoE layer is a feed forward network with one hidden layer of size 2048 and ReLU activation. Thus, each expert contains [512 ∗ 2048] + [2048 ∗ 512] = 2M parameters. The output of the MoE layer is passed through a sigmoid function. We use the strictly-balanced gating function described in Appendix F.\n\nModel Architecture for Multilingual MoE Model: We used the same model architecture as for the single-language-pair models, with the following exceptions: We used noisy-top-k gating as described in Section 2.1, not the scheme from Appendix F. The MoE layers in the encoder and decoder are non-hierarchical MoEs with n = 512 experts, and k = 2. Each expert has a larger hidden layer of size 8192. This doubles the amount of computation in the MoE layers, raising the computational budget of the entire model from 85M to 102M ops/timestep. Training: We trained our networks using the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 2000 training steps, held constant for an additional 8000 steps, and decreased after that so as to be proportional to the inverse square root of the step number. For the single-language-pair models, similarly to (Wu et al., 2016), we applied dropout (Zaremba et al., 2014) to the output of all embedding, LSTM and MoE layers, using DropP rob = 0.4. Training was done synchronously on a cluster of up to 64 GPUs as described in section 3. Each training batch consisted of a set of sentence pairs containing roughly 16000 words per GPU.\n\nTo ensure balanced expert utilization we set wimportance = 0.01 and wload = 0.01, as described in Section 4 and Appendix A. Metrics: We evaluated our models using the perplexity and the standard BLEU score metric. We reported tokenized BLEU score as computed by the multi-bleu.pl script, downloaded from the public implementation of Moses (on Github), which was also used in (Luong et al., 2015a). Results: Tables 2, 3 and 4 in Section 5.3 show comparisons of our results to other published methods. Figure 4 shows test perplexity as a function of number of words in the (training data's) source sentences processed for models with different numbers of experts. As can be seen from the Figure, as we increased the number of experts to approach 2048, the test perplexity of our model continued to improve.\n\nWe found that the experts indeed become highly specialized by syntax and/or semantics, as can be seen in Table 9. For example, one expert is used when the indefinite article \"a\" introduces the direct object in a verb phrase indicating importance or leadership.\n\nExpert 381 Expert 752 Expert 2004 ... with researchers , ... ... plays a core ... ... with rapidly growing ... ... to innovation . ... plays a critical ... ... under static conditions ... ... tics researchers . ... provides a legislative ... ... to swift ly ... ... the generation of ... ... play a leading ... ... to dras tically ... ... technology innovations is ... ... assume a leadership ... ... the rapid and ... ... technological innovations , ... ... plays a central ... ... the fast est ... ... support innovation throughout ... ... taken a leading ... ... the Quick Method ... ... role innovation will ... ... established a reconciliation ... ... rec urrent ) ... ... research scienti st ... ... played a vital ... ... provides quick access ... ... promoting innovation where ... ... have a central ... ... of volatile organic ... ... ... ...\n\nDue to some peculiarities in our infrastructure which have since been fixed, at the time we ran some of the machine translation experiments, our models ran faster if every expert received exactly the same batch size. To accommodate this, we used a different gating function which we describe below. Recall that we define the softmax gating function to be:\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ Gσ(x) = Sof tmax(x · Wg) (15) Sparse Gating (alternate formulation): To obtain a sparse gating vector, we multiply Gσ(x) component-wise with a sparse mask M(Gσ(x)) and normalize the output. The mask itself is a function of Gσ(x) and specifies which experts are assigned to each input example:\n\n$$(15)$$ $$G(x){i}=\\frac{G{\\sigma}(x){i}M(G{\\sigma}(x)){i}}{\\sum{j=1}^{n}G_{\\sigma}(x){j}M(G{\\sigma}(x))_{j}}\\tag{1}$$ $$(17)$$\n\n$$(16)$$\n\nTop-K Mask: To implement top-k gating in this formulation, we would let M(v) = T opK(v, k), where:\n\n$$T o p K(v,k){i}=\\begin{cases}1&{\\mathrm{if~}}v{i}{\\mathrm{~is~in~the~top~}}k{\\mathrm{~elements~of~}}v.\\ 0&{\\mathrm{otherwise.}}\\end{cases}$$ 0 otherwise. (17) Batchwise Mask: To force each expert to receive the exact same number of examples, we introduce an alternative mask function, Mbatchwise(X, m), which operates over batches of input vectors.\n\nInstead of keeping the top k values per example, we keep the top m values per expert across the training batch, where m = k|X| n, so that each example is sent to an average of k experts.\n\n$$M_{b a t c h w i s e}(X,m){j,i}=\\begin{cases}1&\\text{if}X{j,i}\\text{is in the top}m\\text{values for to expert}i\\ 0&\\text{otherwise}\\end{cases}$$\n\nAs our experiments suggest and also observed in (Ioffe & Szegedy, 2015), using a batchwise function during training (such as Mbatchwise) requires modifications to the inference when we may not have a large batch of examples. Our solution to this is to train a vector T of per-expert threshold values to approximate the effects of the batchwise mask. We use the following mask at inference time:\n\n$$(18)$$ $$M_{threshold}(x,T){i}=\\begin{cases}1&\\text{if}x{i}>T_{i}\\ 0&\\text{otherwise}\\end{cases}\\tag{1}$$ $$(19)$$\n\nTo learn the threshold values, we apply an additional loss at training time which is minimized when the batchwise mask and the threshold mask are identical.\n\n$$L_{batchwise}(X,T,m)=\\sum_{j=1}^{|X|}\\sum_{i=1}^{n}(M_{threshold}(x,T){i}-M{batchwise}(X,m){j,i})(X{j,i}-T_{i})\\tag{20}$$\n\nG ATTENTION FUNCTION The attention mechanism described in GNMT (Wu et al., 2016) involves a learned \"Attention Function\" A(xi, yj ) which takes a \"source vector\" xi and a \"target vector\" yj , and must be computed for every source time step i and target time step j. In GNMT, the attention function is implemented as a feed forward neural network with a hidden layer of size n. It can be expressed as:\n\n$$A_{G N M T}(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}t a n h((x_{i}U){d}+(y{j}W)_{d})$$\n\nWhere U and W are trainable weight matrices and V is a trainable weight vector.\n\nFor performance reasons, in our models, we used a slightly different attention function:\n\n$$A(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}tanh((x_{i}U){d})tanh((y{j}W)_{d})\\tag{22}$$\n\nWith our attention function, we can simultaneously compute the attention function on multiple source time steps and multiple target time steps using optimized matrix multiplications. We found little difference in quality between the two functions.\n\n$$(21)$$",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/1701.06538v1.md"
+          },
+          "headlines": {
+            "1 Introduction And Related Work": [
+              "1.1 Conditional Computation",
+              "1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER",
+              "1.3 Related Work On Mixtures Of Experts"
+            ],
+            "2 The Structure Of The Mixture-Of-Experts Layer": [
+              "2.1 Gating Network"
+            ],
+            "3 Addressing Performance Challenges": [
+              "3.1 The Shrinking Batch Problem",
+              "3.2 Network Bandwidth"
+            ],
+            "4 Balancing Expert Utilization": [],
+            "5 Experiments": [
+              "5.1 1 Billion Word Language Modeling Benchmark",
+              "5.2 100 BILLION WORD GOOGLE NEWS CORPUS",
+              "5.3 Machine Translation (Single Language Pair)",
+              "5.4 Multilingual Machine Translation"
+            ],
+            "6 Conclusion": [],
+            "Appendices": [
+              "A Load-Balancing Loss",
+              "B Hierachical Mixture Of Experts",
+              "C 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS",
+              "D 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS",
+              "E Machine Translation - Experimental Details"
+            ]
+          },
+          "summary": "The paper introduces a Sparsely-Gated Mixture-of-Experts (MoE) layer to enhance neural network capacity without a proportional increase in computation. This approach uses a trainable gating network to select a sparse combination of experts for each input, significantly improving model capacity and performance in language modeling and machine translation tasks. The MoE layer, with up to 137 billion parameters, achieves better results than state-of-the-art models at lower computational costs. The paper addresses challenges like batch size and network bandwidth, proposing solutions like mixing data and model parallelism. Experiments show that MoE models outperform traditional models in various benchmarks, demonstrating the potential of conditional computation in deep learning.",
+          "summary_embedding": [
+            -0.01644345559179783,
+            -0.016995344310998917,
+            -0.016372699290513992,
+            -0.014349107630550861,
+            0.02282555401325226,
+            0.045934125781059265,
+            -0.014646278694272041,
+            0.03563220426440239,
+            -0.004829026758670807,
+            0.005617944523692131,
+            -0.00932550523430109,
+            -0.008143897168338299,
+            0.005575491581112146,
+            0.01538213063031435,
+            -0.005653322208672762,
+            -0.020306676626205444,
+            -0.013832597061991692,
+            -0.023235931992530823,
+            0.028344441205263138,
+            -0.020080260932445526,
+            0.011823155917227268,
+            -0.020943470299243927,
+            -0.025868017226457596,
+            -0.016500059515237808,
+            0.025061409920454025,
+            0.024806693196296692,
+            0.02583971433341503,
+            0.003827843815088272,
+            -0.03492465242743492,
+            0.033651065081357956,
+            0.014674580655992031,
+            0.012488252483308315,
+            -0.031500112265348434,
+            -0.009162768721580505,
+            -0.05351905897259712,
+            -0.05227377265691757,
+            0.04143410921096802,
+            0.022245362401008606,
+            -0.016231190413236618,
+            0.023717066273093224,
+            0.017264213413000107,
+            0.005816058721393347,
+            -0.020632149651646614,
+            -0.020844414830207825,
+            0.003700484987348318,
+            -0.013103820383548737,
+            -0.01983969286084175,
+            -0.013528349809348583,
+            -0.01828308403491974,
+            -0.01577835902571678,
+            0.009551920928061008,
+            0.0002856731880456209,
+            0.0029186424799263477,
+            -0.003424540627747774,
+            -0.047830358147621155,
+            0.0248208437114954,
+            0.019853845238685608,
+            0.03492465242743492,
+            0.04633035510778427,
+            0.013011839240789413,
+            0.01516986545175314,
+            -0.014271277002990246,
+            -0.04021712392568588,
+            0.005497660953551531,
+            -0.00041656987741589546,
+            -0.005380915477871895,
+            -0.005819596350193024,
+            0.008575502783060074,
+            0.008724087849259377,
+            -0.01143400277942419,
+            -0.04202845320105553,
+            0.013952880166471004,
+            -0.020561393350362778,
+            0.00010143388499272987,
+            0.01354957651346922,
+            0.008292482234537601,
+            0.02677368000149727,
+            0.03195294365286827,
+            0.023971782997250557,
+            -0.008724087849259377,
+            -0.01340806670486927,
+            0.019924599677324295,
+            -0.0022676968947052956,
+            0.007018893025815487,
+            0.04570771008729935,
+            -0.04033033177256584,
+            0.02163686975836754,
+            -0.025726506486535072,
+            -0.015339677222073078,
+            -0.005349075887352228,
+            -0.022471778094768524,
+            0.02806142158806324,
+            -0.007344366051256657,
+            0.017801949754357338,
+            0.017094401642680168,
+            0.01594817079603672,
+            0.013330236077308655,
+            -0.001894464367069304,
+            -0.022938761860132217,
+            0.0018360915128141642,
+            -0.01541043259203434,
+            0.02114158496260643,
+            0.0015230007702484727,
+            0.005469358991831541,
+            0.027594437822699547,
+            0.031160488724708557,
+            -0.023490650579333305,
+            0.009502392262220383,
+            0.02422650158405304,
+            0.06967949122190475,
+            0.015424583107233047,
+            -0.006435164716094732,
+            -0.0024640418123453856,
+            0.04143410921096802,
+            0.0015513027319684625,
+            0.0038349193055182695,
+            -0.06826439499855042,
+            -0.03022652305662632,
+            -0.0011365016689524055,
+            -0.02254253253340721,
+            0.02227366529405117,
+            0.01002597901970148,
+            -0.009750034660100937,
+            -0.022457627579569817,
+            -0.04516997188329697,
+            0.05190584436058998,
+            -0.00545874610543251,
+            -0.016259491443634033,
+            -0.005554265342652798,
+            -0.05340585112571716,
+            0.00039136342820711434,
+            0.015042506158351898,
+            -0.018481198698282242,
+            -0.03670767694711685,
+            0.036141637712717056,
+            0.0007398316520266235,
+            0.03741522878408432,
+            0.012459950521588326,
+            -0.01708024926483631,
+            -0.020094411447644234,
+            0.022259512916207314,
+            -0.011639192700386047,
+            -0.0038313816767185926,
+            -0.027339721098542213,
+            0.0065236082300543785,
+            -0.028103873133659363,
+            -0.0014566679019480944,
+            0.06730213016271591,
+            -0.04816998168826103,
+            0.02946236915886402,
+            -0.029660483822226524,
+            0.023660462349653244,
+            -0.008575502783060074,
+            0.0278916098177433,
+            -0.010238244198262691,
+            0.005840822588652372,
+            -0.013457595370709896,
+            -0.0010586712742224336,
+            -0.020745357498526573,
+            0.0034210027661174536,
+            -0.016259491443634033,
+            -0.027750099077820778,
+            -0.007054270710796118,
+            0.031273696571588516,
+            0.01698119193315506,
+            0.025655752047896385,
+            0.025429336354136467,
+            0.01246702577918768,
+            -0.04935866594314575,
+            0.02595292218029499,
+            0.0018926954362541437,
+            0.013514199294149876,
+            0.013181651011109352,
+            -0.0010418669553473592,
+            0.025599148124456406,
+            0.012049571610987186,
+            0.025061409920454025,
+            0.037443529814481735,
+            0.0027682881336659193,
+            0.04737752676010132,
+            -0.015693452209234238,
+            -0.005292471963912249,
+            0.028528403490781784,
+            0.000505455827806145,
+            0.02465103194117546,
+            -0.001686621573753655,
+            0.013330236077308655,
+            0.000776978034991771,
+            0.03834919258952141,
+            -0.006339645478874445,
+            0.029518973082304,
+            0.021608568727970123,
+            0.028316138312220573,
+            -0.014759486541152,
+            0.023122724145650864,
+            -0.019754787907004356,
+            -0.02292460948228836,
+            0.035547297447919846,
+            0.004082561936229467,
+            0.01886327564716339,
+            -0.028457649052143097,
+            -0.030679354444146156,
+            0.015452885068953037,
+            -0.01855195313692093,
+            0.016556663438677788,
+            0.03639635816216469,
+            0.0008273909334093332,
+            -0.0025719432160258293,
+            0.026292547583580017,
+            -0.04245298355817795,
+            0.012856177985668182,
+            -0.01532552670687437,
+            -0.02855670638382435,
+            -0.019528372213244438,
+            -0.00875946506857872,
+            -0.0018095583654940128,
+            -0.013691086322069168,
+            -0.0037570889107882977,
+            -0.007563706487417221,
+            -0.02459442801773548,
+            0.00935380719602108,
+            0.013627407141029835,
+            0.009849091991782188,
+            -0.018665160983800888,
+            0.0019952901639044285,
+            -0.008257105015218258,
+            -0.0160047747194767,
+            -0.03752843663096428,
+            0.015268922783434391,
+            -0.0024888061452656984,
+            0.009191070683300495,
+            0.04078316316008568,
+            0.028485950082540512,
+            -0.005055442452430725,
+            0.0329718142747879,
+            -0.033707667142152786,
+            0.017915157601237297,
+            -0.008320784196257591,
+            0.009191070683300495,
+            0.002989397384226322,
+            -0.025542544201016426,
+            0.027424626052379608,
+            -0.03792466223239899,
+            -0.038151077926158905,
+            0.022995365783572197,
+            0.011858533136546612,
+            0.0017096169758588076,
+            -0.019386861473321915,
+            -0.017717044800519943,
+            0.0008499440737068653,
+            0.002752368338406086,
+            -0.018834972754120827,
+            -0.01019579079002142,
+            0.02227366529405117,
+            -0.025500090792775154,
+            0.036085035651922226,
+            0.023391593247652054,
+            -0.03520767390727997,
+            -0.00608492735773325,
+            -0.01903308741748333,
+            0.059037946164608,
+            -0.0011506526498124003,
+            -0.010301923379302025,
+            0.022938761860132217,
+            -0.027268964797258377,
+            -0.004524780437350273,
+            0.022301966324448586,
+            0.01828308403491974,
+            -0.008278331719338894,
+            0.003983504604548216,
+            -0.022287815809249878,
+            -0.05281151086091995,
+            -0.010577867738902569,
+            -0.0003305583668407053,
+            0.054368119686841965,
+            -0.012367968447506428,
+            0.0020695829298347235,
+            -0.006413938011974096,
+            0.004100250545889139,
+            0.03704730048775673,
+            0.002140337834134698,
+            0.0025082635693252087,
+            -0.013698162510991096,
+            0.0371888130903244,
+            0.022316116839647293,
+            0.005883275531232357,
+            -0.012162779457867146,
+            0.005377377849072218,
+            -0.04899074137210846,
+            -0.0009118546731770039,
+            0.005741765722632408,
+            0.007386818993836641,
+            -0.025910470634698868,
+            0.017717044800519943,
+            0.009205221198499203,
+            -0.0024693484883755445,
+            -0.009792488068342209,
+            0.01173117384314537,
+            0.025429336354136467,
+            -0.019938750192523003,
+            -0.05700020492076874,
+            0.0034705311991274357,
+            0.012014193460345268,
+            -0.004054259974509478,
+            -0.019401011988520622,
+            0.006792477332055569,
+            0.0021934041287750006,
+            0.022698193788528442,
+            -0.011271266266703606,
+            -0.0029221803415566683,
+            0.040585052222013474,
+            0.009516543708741665,
+            -0.0016813150141388178,
+            -0.017717044800519943,
+            -0.011794853024184704,
+            0.018070818856358528,
+            0.004740582779049873,
+            0.0035289041697978973,
+            -0.01751893013715744,
+            -0.029264256358146667,
+            0.02800481766462326,
+            0.019967053085565567,
+            -0.048905834555625916,
+            -0.008200501091778278,
+            -0.019273653626441956,
+            0.0019599127117544413,
+            0.008695785887539387,
+            0.04613224044442177,
+            -0.00590803986415267,
+            0.008313708938658237,
+            -0.020179318264126778,
+            -0.05003791302442551,
+            -0.02708500251173973,
+            0.03707560524344444,
+            -0.028684064745903015,
+            -0.04276430234313011,
+            -0.010252394713461399,
+            0.016740625724196434,
+            -0.009403335861861706,
+            -0.018679311498999596,
+            -0.011738249100744724,
+            -0.04969828948378563,
+            -0.003497064346447587,
+            0.0337359681725502,
+            0.017264213413000107,
+            -0.008441068232059479,
+            -0.033877480775117874,
+            0.019641580060124397,
+            -0.007160402834415436,
+            -0.006474079564213753,
+            0.0007862646016292274,
+            0.008794843219220638,
+            0.02400008589029312,
+            0.011879759840667248,
+            -0.02763689123094082,
+            -0.03427370637655258,
+            -0.0024888061452656984,
+            0.014929298311471939,
+            -0.00878776703029871,
+            0.0245378240942955,
+            0.024679332971572876,
+            -0.022995365783572197,
+            -0.01654251106083393,
+            0.015750056132674217,
+            -0.05779265984892845,
+            -0.010287772864103317,
+            -0.029235953465104103,
+            -0.006183984223753214,
+            0.0009613831643946469,
+            0.0061096916906535625,
+            0.00012304731353651732,
+            0.03846240043640137,
+            0.008801918476819992,
+            0.006014172453433275,
+            0.009191070683300495,
+            0.01913214474916458,
+            0.028471799567341805,
+            0.009155693463981152,
+            -0.03517937287688255,
+            0.02850010246038437,
+            -0.017051948234438896,
+            0.007811348885297775,
+            -0.025754809379577637,
+            -0.013033065013587475,
+            0.00443633645772934,
+            0.02725481428205967,
+            -0.000503244751598686,
+            0.018849123269319534,
+            -0.021919889375567436,
+            0.037500135600566864,
+            -0.029915200546383858,
+            -0.05278320610523224,
+            -0.02005195803940296,
+            -0.008285406976938248,
+            0.015764206647872925,
+            0.017278363928198814,
+            -0.019924599677324295,
+            -0.0286557637155056,
+            -0.021750077605247498,
+            0.006470541935414076,
+            0.010542490519583225,
+            -0.021099131554365158,
+            -0.010627396404743195,
+            0.01502835564315319,
+            -0.00404364662244916,
+            -0.01886327564716339,
+            0.013684011064469814,
+            0.015693452209234238,
+            0.03169822692871094,
+            -0.03308502212166786,
+            0.004287751391530037,
+            -0.025967074558138847,
+            -0.04327373951673508,
+            0.004096712917089462,
+            0.03166992589831352,
+            0.027509532868862152,
+            -0.018636859953403473,
+            0.010754755698144436,
+            0.003990580327808857,
+            0.004234685096889734,
+            -0.0033803186379373074,
+            0.023603858426213264,
+            0.03291521221399307,
+            0.02763689123094082,
+            -0.011653343215584755,
+            0.004921008367091417,
+            -0.01559439580887556,
+            -0.0011108530452474952,
+            0.01865101046860218,
+            -0.017816102132201195,
+            0.033877480775117874,
+            -0.029405765235424042,
+            0.02104252763092518,
+            -0.035122767090797424,
+            0.006141531281173229,
+            -0.012099100276827812,
+            -0.00404364662244916,
+            0.04833979532122612,
+            -0.017646290361881256,
+            -0.0019546060357242823,
+            -0.008101443760097027,
+            -0.014575524255633354,
+            -0.02158026583492756,
+            0.008766540326178074,
+            -0.04033033177256584,
+            -0.012367968447506428,
+            -0.028429346159100533,
+            0.003424540627747774,
+            0.019910449162125587,
+            -0.06277380883693695,
+            -0.06237758323550224,
+            0.012275987304747105,
+            -0.04918885603547096,
+            -0.026419905945658684,
+            0.02963218092918396,
+            -0.002984090941026807,
+            0.0012780115939676762,
+            -0.0024003623984754086,
+            0.0068101659417152405,
+            -0.029207652434706688,
+            0.00425944896414876,
+            -0.000887974863871932,
+            0.03271709755063057,
+            -0.021056678146123886,
+            -0.008122670464217663,
+            0.004878554958850145,
+            -0.06860402226448059,
+            -0.028344441205263138,
+            0.011271266266703606,
+            -0.009459939785301685,
+            -0.0066615804098546505,
+            -0.017915157601237297,
+            -0.004383270163089037,
+            0.0008950503543019295,
+            0.0032830305863171816,
+            0.05801907554268837,
+            -0.01983969286084175,
+            -0.053773775696754456,
+            0.03633975237607956,
+            0.00887267291545868,
+            0.029603879898786545,
+            -0.025556694716215134,
+            -0.017533080652356148,
+            -0.01855195313692093,
+            -0.020504789426922798,
+            -0.04123599827289581,
+            0.00404364662244916,
+            -0.005918653216212988,
+            0.009212297387421131,
+            -0.0009693430620245636,
+            0.02763689123094082,
+            0.029235953465104103,
+            0.023108573630452156,
+            0.012750045396387577,
+            0.01079013291746378,
+            -0.022372720763087273,
+            -0.021594416350126266,
+            0.0012222920777276158,
+            0.04683978855609894,
+            0.024636881425976753,
+            -0.02675952948629856,
+            -0.011504758149385452,
+            0.03953787684440613,
+            0.0419718474149704,
+            -0.01974063739180565,
+            -0.02276895008981228,
+            0.00764861237257719,
+            0.03266049548983574,
+            -0.04477374628186226,
+            0.005932804197072983,
+            0.03410389646887779,
+            0.014405711553990841,
+            -0.005356151144951582,
+            -0.010294848121702671,
+            0.022570835426449776,
+            0.03152841329574585,
+            -0.008724087849259377,
+            0.01876421831548214,
+            0.016910437494516373,
+            0.029349161311984062,
+            -0.0067889392375946045,
+            0.0005766529939137399,
+            -0.024735936895012856,
+            0.055188875645399094,
+            -0.03059444949030876,
+            0.02909444272518158,
+            0.010316074825823307,
+            -0.00593634182587266,
+            0.039056744426488876,
+            -0.016641568392515182,
+            -0.0431322306394577,
+            0.01913214474916458,
+            -0.006979977712035179,
+            -0.0055613406002521515,
+            -0.03444352000951767,
+            -0.007592008449137211,
+            -0.007507102098315954,
+            -0.006714646704494953,
+            -0.022556684911251068,
+            -0.02606613002717495,
+            0.004415110219269991,
+            0.00973588414490223,
+            -0.006484692916274071,
+            0.00698705343529582,
+            0.02092931978404522,
+            -0.013507124036550522,
+            0.02525952458381653,
+            -0.046047333627939224,
+            -0.0008366775000467896,
+            -0.0038596836384385824,
+            -0.025712355971336365,
+            -0.0345001220703125,
+            -0.0005770951975136995,
+            -0.0024251264985650778,
+            -0.024622729048132896,
+            -0.00914154201745987,
+            -0.008752389810979366,
+            -0.0023119186516851187,
+            -0.004938696976751089,
+            -0.055188875645399094,
+            0.008695785887539387,
+            0.010846736840903759,
+            -0.002713453024625778,
+            0.0055507272481918335,
+            -0.010613245889544487,
+            -0.03039633482694626,
+            0.03039633482694626,
+            0.04129260033369064,
+            0.033707667142152786,
+            0.014334957115352154,
+            -0.0074646491557359695,
+            -0.017165156081318855,
+            0.006060163024812937,
+            0.021721776574850082,
+            0.00902833417057991,
+            0.007846726104617119,
+            0.03877372294664383,
+            0.00011331850691931322,
+            -0.008349086157977581,
+            -0.009764186106622219,
+            0.028740668669342995,
+            -0.001085204305127263,
+            0.023603858426213264,
+            0.050802066922187805,
+            0.030424637719988823,
+            0.020589696243405342,
+            -0.01681138016283512,
+            -0.02010856196284294,
+            -0.00851889792829752,
+            -0.02855670638382435,
+            0.0364246591925621,
+            0.012127402238547802,
+            -0.005232329946011305,
+            -0.011540135368704796,
+            0.019061388447880745,
+            -0.033339742571115494,
+            -0.0037358624394983053,
+            0.00048422932741232216,
+            0.013924578204751015,
+            -0.03379257395863533,
+            0.0002480846014805138,
+            0.031613320112228394,
+            0.014957600273191929,
+            -0.016995344310998917,
+            -0.0188774261623621,
+            0.010549565777182579,
+            0.020462337881326675,
+            0.018948180601000786,
+            0.00037743354914709926,
+            0.00897173024713993,
+            -0.00022121983056422323,
+            -0.0005319889169186354,
+            0.01590571738779545,
+            -0.007952858693897724,
+            0.036254845559597015,
+            -0.003525366308167577,
+            -0.01325240544974804,
+            -0.0037500134203583,
+            -0.021438755095005035,
+            -0.024467067793011665,
+            0.043556760996580124,
+            0.023971782997250557,
+            0.04661337286233902,
+            -0.005660397466272116,
+            -0.004245297983288765,
+            0.025825563818216324,
+            0.008221727795898914,
+            -0.03288691118359566,
+            0.000265110022155568,
+            0.005281858611851931,
+            -0.02892463095486164,
+            0.016641568392515182,
+            -0.02476423978805542,
+            -0.016896286979317665,
+            -0.006771250627934933,
+            -0.0015557248843833804,
+            -0.028188779950141907,
+            0.01703779585659504,
+            0.0031309074256569147,
+            -0.0027099153958261013,
+            -0.039764292538166046,
+            0.006880920846015215,
+            0.020419884473085403,
+            0.015311375260353088,
+            0.0037358624394983053,
+            0.011525984853506088,
+            -0.016613267362117767,
+            0.002126186853274703,
+            0.03379257395863533,
+            0.004878554958850145,
+            0.02005195803940296,
+            0.0027647505048662424,
+            0.023773670196533203,
+            0.03633975237607956,
+            -0.01681138016283512,
+            -0.02189158834517002,
+            0.01519816741347313,
+            -0.02233026921749115,
+            0.029943503439426422,
+            0.004889168310910463,
+            -0.012283062562346458,
+            0.022033097222447395,
+            -0.009827865287661552,
+            -0.0027346794959157705,
+            0.01659911684691906,
+            -0.022740647196769714,
+            -0.017717044800519943,
+            0.03183973580598831,
+            -0.039452970027923584,
+            0.012856177985668182,
+            -0.008547200821340084,
+            0.015863263979554176,
+            -0.023066120222210884,
+            0.011172209866344929,
+            0.007100261282175779,
+            -0.01573590561747551,
+            0.010797208175063133,
+            0.009693430736660957,
+            0.00545874610543251,
+            0.0018926954362541437,
+            0.02179253101348877,
+            -0.0006049550138413906,
+            -0.011257115751504898,
+            -0.018155725672841072,
+            -0.004694592207670212,
+            -0.002614396158605814,
+            -0.016556663438677788,
+            0.01865101046860218,
+            -0.041066184639930725,
+            -0.013280707411468029,
+            -0.008879749104380608,
+            -0.013988257385790348,
+            -0.010011828504502773,
+            0.024184048175811768,
+            0.025669902563095093,
+            -0.002361447084695101,
+            -0.031217092648148537,
+            0.01790100708603859,
+            0.024198200553655624,
+            0.003654494183138013,
+            -0.023646311834454536,
+            0.0302831269800663,
+            -0.008936353027820587,
+            0.014971751719713211,
+            0.026136886328458786,
+            0.014094389975070953,
+            0.004595535341650248,
+            -0.010500037111341953,
+            -0.02114158496260643,
+            0.01481609046459198,
+            0.010507113300263882,
+            0.033537857234478,
+            0.014745336025953293,
+            0.02071705460548401,
+            0.01594817079603672,
+            -0.023320838809013367,
+            -0.011214662343263626,
+            -0.003482913365587592,
+            -0.022514231503009796,
+            0.03093407303094864,
+            0.0006129149114713073,
+            0.02114158496260643,
+            0.026618018746376038,
+            -0.03319823369383812,
+            -0.01187268365174532,
+            0.032688796520233154,
+            0.02779255248606205,
+            0.05108508840203285,
+            0.025316128507256508,
+            0.015396281145513058,
+            0.027226513251662254,
+            -0.03585861995816231,
+            -0.036368053406476974,
+            0.001548649393953383,
+            0.04256619140505791,
+            -0.008426916785538197,
+            0.00967928022146225,
+            0.03900013864040375,
+            -0.006781863979995251,
+            -0.01611798256635666,
+            0.019457615911960602,
+            0.0026710000820457935,
+            0.002448122017085552,
+            -0.0033113325480371714,
+            -0.01335146278142929,
+            0.03175482898950577,
+            -0.0051474240608513355,
+            0.013570803217589855,
+            -0.019273653626441956,
+            0.027113303542137146,
+            0.0001799829478841275,
+            -0.003229964291676879,
+            0.028344441205263138,
+            -0.002660386962816119,
+            -0.051990751177072525,
+            0.024212351068854332,
+            0.019811391830444336,
+            -0.011561362072825432,
+            -0.0036332677118480206,
+            -0.003792466362938285,
+            0.04567940905690193,
+            0.034698236733675,
+            0.029009537771344185,
+            -0.0020501252729445696,
+            0.0168679840862751,
+            0.018792519345879555,
+            0.0053667644970119,
+            0.014278353191912174,
+            -0.0017025414854288101,
+            -0.007853801362216473,
+            0.021056678146123886,
+            -0.020787810906767845,
+            0.03226426616311073,
+            -0.016570813953876495,
+            -0.020122714340686798,
+            0.008738238364458084,
+            -0.011172209866344929,
+            -0.0009083169279620051,
+            -0.006311343517154455,
+            -0.0008813415770418942,
+            -0.0029664020985364914,
+            0.016938740387558937,
+            -0.004446949809789658,
+            0.04661337286233902,
+            -0.013931653462350368,
+            -0.008808993734419346,
+            0.006972902454435825,
+            -0.025981225073337555,
+            0.0013346156338229775,
+            0.0028726516757160425,
+            0.027000095695257187,
+            -0.01989629678428173,
+            -0.00970758218318224,
+            0.01606137864291668,
+            0.06039644405245781,
+            -0.008639181964099407,
+            -0.0026267783250659704,
+            0.027226513251662254,
+            -0.002787745790556073,
+            -0.03551899641752243,
+            0.012919857166707516,
+            -0.04494355618953705,
+            0.017589686438441277,
+            0.026349151507019997,
+            -0.04986810311675072,
+            -0.022089701145887375,
+            -0.02347649820148945,
+            -0.02043403498828411,
+            0.023434046655893326,
+            -0.009976450353860855,
+            0.013867974281311035,
+            -0.017801949754357338,
+            -0.009014182724058628,
+            0.020518941804766655,
+            0.009176919236779213,
+            0.03735862299799919,
+            -0.027551984414458275,
+            -0.016783079132437706,
+            -0.027820853516459465,
+            -0.04304732382297516,
+            0.009558996185660362,
+            -0.0278916098177433,
+            0.0027099153958261013,
+            -0.0016291331266984344,
+            -0.040302030742168427,
+            -0.037981268018484116,
+            0.005278320983052254,
+            -0.017985913902521133,
+            0.01516986545175314,
+            0.020915169268846512,
+            0.02731141820549965,
+            -0.0020253609400242567,
+            -0.030198222026228905,
+            0.029235953465104103,
+            -0.007344366051256657,
+            0.03223596513271332,
+            -0.00689507182687521,
+            0.005066055804491043,
+            -0.0070436573587358,
+            0.031160488724708557,
+            -0.023009516298770905,
+            0.018792519345879555,
+            -0.026801982894539833,
+            -0.004086099565029144,
+            -0.018990634009242058,
+            -0.01774534583091736,
+            0.004224071744829416,
+            0.004521242808550596,
+            7.888073741924018e-05,
+            0.0048750173300504684,
+            0.005777143407613039,
+            0.024806693196296692,
+            0.0021421066485345364,
+            0.02589631825685501,
+            -0.027877457439899445,
+            -0.025316128507256508,
+            0.01719345711171627,
+            -0.02077365852892399,
+            0.005678086541593075,
+            -0.006261814851313829,
+            -0.0302831269800663,
+            -0.01202834490686655,
+            0.023391593247652054,
+            0.02162271924316883,
+            -0.005490585695952177,
+            0.002644466934725642,
+            0.02206140011548996,
+            -0.026023678481578827,
+            0.00437973253428936,
+            -0.03781145438551903,
+            -0.018396291881799698,
+            0.029773691669106483,
+            -0.03195294365286827,
+            0.020745357498526573,
+            0.003185742534697056,
+            0.03483974561095238,
+            -0.017292514443397522,
+            -0.01947176828980446,
+            0.01654251106083393,
+            -0.01008258294314146,
+            0.004142703488469124,
+            0.01606137864291668,
+            0.01616043597459793,
+            -0.0015070808585733175,
+            0.05380208045244217,
+            0.0008167776977643371,
+            0.006909222807735205,
+            -0.049075648188591,
+            0.004496478475630283,
+            -0.044858649373054504,
+            0.0002768288250081241,
+            -0.013337312266230583,
+            -0.02514631673693657,
+            0.0014027173165231943,
+            0.001001182827167213,
+            0.001963450340554118,
+            -0.01913214474916458,
+            -0.01989629678428173,
+            0.011596739292144775,
+            -0.015863263979554176,
+            0.004928083624690771,
+            -0.06418890506029129,
+            -0.01855195313692093,
+            -0.004974074196070433,
+            0.0043337419629096985,
+            0.0007699025445617735,
+            0.03342464938759804,
+            -0.01478778850287199,
+            0.010117961093783379,
+            -0.01549533847719431,
+            -0.026363302022218704,
+            0.04010391607880592,
+            0.018750067800283432,
+            -0.023773670196533203,
+            0.024693485349416733,
+            0.012106175534427166,
+            0.03099067695438862,
+            -0.0318114347755909,
+            0.020349130034446716,
+            0.018523650243878365,
+            -0.0029823218937963247,
+            0.006551910191774368,
+            0.008575502783060074,
+            0.01832553744316101,
+            -0.015764206647872925,
+            -0.0003632825391832739,
+            0.011419852264225483,
+            0.010033054277300835,
+            0.031924642622470856,
+            0.01403071079403162,
+            -0.03183973580598831,
+            0.019047237932682037,
+            0.01616043597459793,
+            0.02298121340572834,
+            0.033820874989032745,
+            -0.027481229975819588,
+            0.038575608283281326,
+            -0.010386829264461994,
+            -0.024042539298534393,
+            0.020801961421966553,
+            -0.032688796520233154,
+            -0.006679269019514322,
+            0.030735958367586136,
+            0.04398128762841225,
+            -0.0015566092915832996,
+            -0.044632233679294586,
+            0.01535382866859436,
+            0.01316749956458807,
+            0.0027806703001260757,
+            0.028811423107981682,
+            0.011264191009104252,
+            0.021820832043886185,
+            0.025273675099015236,
+            0.011419852264225483,
+            -0.0037252490874379873,
+            0.024962352588772774,
+            -0.008101443760097027,
+            -0.004446949809789658,
+            -0.0007035697344690561,
+            0.03206615149974823,
+            0.014702882617712021,
+            -0.017603836953639984,
+            -0.036962397396564484,
+            -0.007620310410857201,
+            0.009452863596379757,
+            -0.007903330028057098,
+            -0.027127455919981003,
+            -0.03950957581400871,
+            -0.026787832379341125,
+            -0.005667473189532757,
+            0.01505665760487318,
+            -0.06724552065134048,
+            -0.007620310410857201,
+            -0.012233533896505833,
+            0.017717044800519943,
+            0.0022906921803951263,
+            0.0181132722645998,
+            0.001277127186767757,
+            -0.009403335861861706,
+            0.004209920763969421,
+            0.050802066922187805,
+            -0.03059444949030876,
+            0.014242975041270256,
+            0.011292492970824242,
+            0.0032105068676173687,
+            0.002787745790556073,
+            0.038688816130161285,
+            -0.010988246649503708,
+            -0.030424637719988823,
+            0.011575513519346714,
+            -0.017292514443397522,
+            0.003383856499567628,
+            0.05006621778011322,
+            -0.009481165558099747,
+            -0.027495380491018295,
+            0.020915169268846512,
+            -0.009148617275059223,
+            -0.015523640438914299,
+            -0.0027258351910859346,
+            -0.006035399157553911,
+            0.0006283925613388419,
+            -0.03832089155912399,
+            -0.015750056132674217,
+            0.03212275728583336,
+            0.008617955259978771,
+            0.014603826217353344,
+            0.025344429537653923,
+            0.010209942236542702,
+            -0.030481241643428802,
+            0.027481229975819588,
+            -0.0005678086308762431,
+            0.005221716593950987,
+            0.013323160819709301,
+            0.03424540534615517,
+            -0.014264201745390892,
+            -0.020264223217964172,
+            -0.040245428681373596,
+            -0.013853822834789753,
+            -0.0463869571685791,
+            0.001943992800079286,
+            0.006919836159795523,
+            -0.012919857166707516,
+            -0.04366996884346008,
+            -0.0329718142747879,
+            0.021127434447407722,
+            0.03178313374519348,
+            -0.017023645341396332,
+            0.0025294902734458447,
+            -0.008483520708978176,
+            -0.007882103323936462,
+            -0.0017520699184387922,
+            -0.013217028230428696,
+            -0.006378560326993465,
+            -0.006229975260794163,
+            0.015452885068953037,
+            -0.004652139265090227,
+            -0.0036934094969183207,
+            -0.0498398020863533,
+            -0.018778368830680847,
+            0.004082561936229467,
+            -0.008221727795898914,
+            0.007323139347136021,
+            -0.006955213379114866,
+            -0.002727604005485773,
+            -0.012474101036787033,
+            -0.03829259052872658,
+            0.024184048175811768,
+            0.017278363928198814,
+            0.0035501306410878897,
+            -0.008922201581299305,
+            -0.018367990851402283,
+            0.03294351324439049,
+            0.012523629702627659,
+            -0.017547233030200005,
+            -0.018580254167318344,
+            0.025981225073337555,
+            0.026957644149661064,
+            0.008122670464217663,
+            -0.006137993652373552,
+            0.029886899515986443,
+            0.005695775151252747,
+            -0.022938761860132217,
+            -0.003983504604548216,
+            -0.005055442452430725,
+            -0.0038844477385282516,
+            0.020561393350362778,
+            -0.010556641966104507,
+            0.0300001073628664,
+            0.038717120885849,
+            0.012870329432189465,
+            0.0039198254235088825,
+            -0.0033714743331074715,
+            -0.008922201581299305,
+            0.015976471826434135,
+            0.006481155287474394,
+            0.008525974117219448,
+            0.014759486541152,
+            -0.016358548775315285,
+            -0.020405733957886696,
+            0.006215824279934168,
+            -0.007107336539775133,
+            0.020122714340686798,
+            0.01708024926483631,
+            0.016429303213953972,
+            0.0007610581233166158,
+            -0.028287837281823158,
+            -0.0018024828750640154,
+            0.02411329373717308,
+            0.02010856196284294,
+            -0.004655676893889904,
+            -0.023264234885573387,
+            0.014398636296391487,
+            -0.015806660056114197,
+            -0.009969375096261501,
+            0.0033431723713874817,
+            0.006229975260794163,
+            -0.003083147807046771,
+            0.0571134127676487,
+            -0.020193468779325485,
+            -0.011419852264225483,
+            -0.010287772864103317,
+            -0.010867963545024395,
+            0.0012072566896677017,
+            -0.01828308403491974,
+            0.008964654989540577,
+            0.025613298639655113,
+            -0.009764186106622219,
+            0.03568880632519722,
+            0.0228114016354084,
+            0.006912760436534882,
+            0.005260631907731295,
+            -0.027070851996541023,
+            0.012304289266467094,
+            -0.016047226265072823,
+            0.005660397466272116,
+            0.018467046320438385,
+            0.017320817336440086,
+            0.0076132346875965595,
+            0.026306698098778725,
+            0.019386861473321915,
+            0.016684021800756454,
+            -0.022783100605010986,
+            0.025457637384533882,
+            -0.007832575589418411,
+            0.027877457439899445,
+            -0.01611798256635666,
+            -0.002253545681014657,
+            -0.012990612536668777,
+            0.008681634441018105,
+            0.002416282193735242,
+            -0.0019351483788341284,
+            0.00036129256477579474,
+            0.016783079132437706,
+            -0.014589674770832062,
+            0.01062032114714384,
+            0.013316085562109947,
+            -0.027919910848140717,
+            -0.010372678749263287,
+            0.016344398260116577,
+            -0.00668988237157464,
+            0.006877383217215538,
+            -0.011865608394145966,
+            -0.0017202302115038037,
+            0.011009473353624344,
+            -0.0436982698738575,
+            0.0104929618537426,
+            0.006279503460973501,
+            -0.015155714936554432,
+            0.014801939949393272,
+            -0.009799563325941563,
+            -0.008235878311097622,
+            -0.013592029921710491,
+            0.003792466362938285,
+            -0.00030225637601688504,
+            -0.0018290160223841667,
+            -0.00022995365725364536,
+            0.00389859895221889,
+            0.010047205723822117,
+            -0.0045990729704499245,
+            -0.010450509376823902,
+            -0.0028638073708862066,
+            -0.02265574224293232,
+            -0.0010047205723822117,
+            0.018735915422439575,
+            -0.004591997712850571,
+            0.0005222601466812193,
+            0.0145189194008708,
+            -0.005830209702253342,
+            0.010556641966104507,
+            -0.005221716593950987,
+            -0.008320784196257591,
+            -0.010768906213343143,
+            -0.033764272928237915,
+            0.0026710000820457935,
+            -0.015254771336913109,
+            0.008865597657859325,
+            -0.012544856406748295,
+            0.04027372971177101,
+            0.01022409275174141,
+            -0.00837738811969757,
+            -0.002796590095385909,
+            -0.0228114016354084,
+            -0.004475251771509647,
+            -0.016938740387558937,
+            -0.01184438169002533,
+            0.008342010900378227,
+            0.012629762291908264,
+            0.005614406894892454,
+            -0.018410442396998405,
+            0.003665107535198331,
+            0.0030955299735069275,
+            -0.005575491581112146,
+            -0.008370312862098217,
+            0.010853812098503113,
+            -0.0006248548161238432,
+            -0.01800006441771984,
+            -0.01957082375884056,
+            -0.027424626052379608,
+            0.0150000536814332,
+            0.014589674770832062,
+            -0.01011088490486145,
+            -0.022754797711968422,
+            -0.007471724878996611,
+            0.007107336539775133,
+            -0.02064630016684532,
+            0.0006779210525564849,
+            0.006891534198075533,
+            0.008717012591660023,
+            0.004117939155548811,
+            0.009636826813220978,
+            0.0007699025445617735,
+            0.0121698547154665,
+            0.018566103652119637,
+            0.03271709755063057,
+            -0.0396793857216835,
+            -0.009537769481539726,
+            0.0057028504088521,
+            -0.001465512323193252,
+            -0.013853822834789753,
+            0.009672204032540321,
+            0.0024375086650252342,
+            -0.0026426981203258038,
+            0.024424616247415543,
+            -0.008044839836657047,
+            -0.01505665760487318,
+            0.01184438169002533,
+            0.00887267291545868,
+            -0.019075538963079453,
+            0.02746707946062088,
+            0.006169833242893219,
+            -0.01011088490486145,
+            0.02259913645684719,
+            -0.012360893189907074,
+            0.012658064253628254,
+            -0.005522425286471844,
+            0.023660462349653244,
+            -0.019004784524440765,
+            0.0006306036957539618,
+            0.015665151178836823,
+            0.03883032873272896,
+            -0.00437973253428936,
+            -0.004567233379930258,
+            -0.03421710431575775,
+            0.005115584470331669,
+            0.028853876516222954,
+            0.02666047215461731,
+            -0.0012346742441877723,
+            -0.005596718285232782,
+            -0.0019864458590745926,
+            0.010768906213343143,
+            0.011589664034545422,
+            0.017561383545398712,
+            -0.029547275975346565,
+            0.00022011429246049374,
+            -0.016938740387558937,
+            0.025500090792775154,
+            0.014632128179073334,
+            0.006951675750315189,
+            0.04072656109929085,
+            0.0027789014857262373,
+            0.003841994795948267,
+            0.027452928945422173,
+            -0.00779719790443778,
+            0.003546592779457569,
+            -0.015566092915832996,
+            0.0003175129240844399,
+            0.022853855043649673,
+            0.0016370931407436728,
+            0.0020784272346645594,
+            0.013337312266230583,
+            0.029320860281586647,
+            -0.022471778094768524,
+            -0.023929331451654434,
+            0.003482913365587592,
+            0.007185167167335749,
+            -0.010570792481303215,
+            0.0038738346192985773,
+            0.006275965832173824,
+            -0.006007096730172634,
+            0.008900974877178669,
+            0.006332569755613804,
+            -0.0003166284877806902,
+            0.0039198254235088825,
+            -0.00308845448307693,
+            0.012962310574948788,
+            -0.02702839858829975,
+            0.036849189549684525,
+            0.004439874552190304,
+            0.010075507685542107,
+            0.015268922783434391,
+            0.001627364312298596,
+            0.004917470272630453,
+            0.005869124550372362,
+            0.010188715532422066,
+            0.018636859953403473,
+            -0.00956607237458229,
+            -0.027481229975819588,
+            -0.03166992589831352,
+            0.020363280549645424,
+            -0.01173117384314537,
+            0.0035872769076377153,
+            0.0074646491557359695,
+            -0.01828308403491974,
+            -0.007507102098315954,
+            -0.009976450353860855,
+            0.015764206647872925,
+            -0.03189634159207344,
+            0.012990612536668777,
+            -0.0043231286108493805,
+            -0.028669914230704308,
+            0.00851889792829752,
+            -0.010181640274822712,
+            -0.0045884596183896065,
+            -0.0035094465129077435,
+            -0.029264256358146667,
+            -0.007931631989777088,
+            0.03908504545688629,
+            -0.02141045406460762,
+            0.024184048175811768,
+            0.008257105015218258,
+            -0.00014449491573031992,
+            -0.0040365708991885185,
+            -0.013563727959990501,
+            0.005331386812031269,
+            -0.01778779923915863,
+            0.025457637384533882,
+            0.010294848121702671,
+            -0.013924578204751015,
+            0.0322076641023159,
+            -0.011179285123944283,
+            0.027438776567578316,
+            -0.018580254167318344,
+            0.036198243498802185,
+            0.007422196213155985,
+            0.00674648629501462,
+            0.01974063739180565,
+            -0.004025958012789488,
+            0.008228803053498268,
+            0.04559450224041939,
+            -0.018523650243878365,
+            0.02460857853293419,
+            0.020349130034446716,
+            -0.00022685811563860625,
+            0.017915157601237297,
+            0.0028390432707965374,
+            -0.023165177553892136,
+            0.009495317004621029,
+            -0.01135617308318615,
+            -0.009955224581062794,
+            0.01583496294915676,
+            -0.019316107034683228,
+            -0.018721764907240868,
+            -0.025613298639655113,
+            0.0029416377656161785,
+            -0.003350247861817479,
+            -0.0077618202194571495,
+            -0.0019616815261542797,
+            0.01454722136259079,
+            0.008724087849259377,
+            0.025669902563095093,
+            0.0008326975512318313,
+            0.011971740983426571,
+            0.01511326152831316,
+            -0.00035576484515331686,
+            0.02313687466084957,
+            -0.03229256719350815,
+            0.01246702577918768,
+            -0.004517704714089632,
+            -0.00509789539501071,
+            0.023858575150370598,
+            0.0003747802402358502,
+            -0.0004877670726273209,
+            -0.0024888061452656984,
+            0.0005496777012012899,
+            -0.014589674770832062,
+            -0.0057099261321127415,
+            0.000907432462554425,
+            -0.005175726022571325,
+            0.04299071803689003,
+            -0.00701535539701581,
+            -0.004772422835230827,
+            -0.039169952273368835,
+            -0.01918874867260456,
+            0.00623351288959384,
+            -0.01583496294915676,
+            -0.018778368830680847,
+            -0.006307805422693491,
+            0.005801907740533352,
+            -0.04893413558602333,
+            0.008342010900378227,
+            0.03336804360151291,
+            -0.012219383381307125,
+            0.00259493850171566,
+            -0.010981171391904354,
+            -0.006969364359974861,
+            -0.0029115669894963503,
+            -7.766463386360556e-05,
+            -0.002412744564935565,
+            -0.020830262452363968,
+            -0.005617944523692131,
+            0.003364398842677474,
+            -0.0008795727044343948,
+            -0.025047259405255318,
+            0.0006177793256938457,
+            0.018849123269319534,
+            -0.003979966975748539,
+            -0.0011152751976624131,
+            -0.001825478277169168,
+            0.006399787031114101,
+            -0.01508495956659317,
+            0.008009462617337704,
+            -0.0076344613917171955,
+            0.003841994795948267,
+            -0.0030495391692966223,
+            -0.002214630600064993,
+            0.02844349853694439,
+            0.021594416350126266,
+            -0.04808507859706879,
+            -0.01405901275575161,
+            0.023009516298770905,
+            0.007280686404556036,
+            -0.004740582779049873,
+            0.01178777776658535,
+            -0.031613320112228394,
+            -0.00028943203506059945,
+            -0.026787832379341125,
+            0.02200479619204998,
+            -0.009205221198499203,
+            -0.007386818993836641,
+            0.016967041417956352,
+            -0.026632171124219894,
+            0.016344398260116577,
+            0.00794578343629837,
+            -0.00030181417241692543,
+            -0.000493515923153609,
+            0.0063856360502541065,
+            0.0008972614887170494,
+            -0.01659911684691906,
+            0.006442239973694086,
+            -0.0048502529971301556,
+            0.007959933951497078,
+            -0.005087282508611679,
+            -0.05527378246188164,
+            -0.009933997876942158,
+            -0.005497660953551531,
+            -0.008936353027820587,
+            -0.012820800766348839,
+            0.008136821910738945,
+            0.02827368676662445,
+            0.01606137864291668,
+            -0.009863242506980896,
+            -4.8312376748071983e-05,
+            0.033000119030475616,
+            -0.0008300442132167518,
+            0.009955224581062794,
+            -0.011349096894264221,
+            0.00015709814033471048,
+            -0.0006911876262165606,
+            -0.014660430140793324,
+            -0.019061388447880745,
+            0.024042539298534393,
+            -0.028344441205263138,
+            0.02378782071173191,
+            -0.01843874529004097,
+            -0.0130825936794281,
+            0.01757553406059742,
+            -0.032575588673353195,
+            -0.005108508747071028,
+            0.007450498174875975,
+            0.0090566361322999,
+            0.021608568727970123,
+            0.0032794929575175047,
+            0.020589696243405342,
+            -0.03716050833463669,
+            -0.009134466759860516,
+            -0.010160413570702076,
+            -0.00794578343629837,
+            -0.010231168940663338,
+            -0.011596739292144775,
+            0.021863285452127457,
+            -0.015240620821714401,
+            -0.016952890902757645,
+            0.007054270710796118,
+            0.0030212372075766325,
+            -0.011313719674944878,
+            0.015750056132674217,
+            0.00017810352437663823,
+            0.004712280817329884,
+            0.013011839240789413,
+            -0.0008923970744945109,
+            0.04086806997656822,
+            0.0253019779920578,
+            -0.0030212372075766325,
+            -0.008568426594138145,
+            0.021438755095005035,
+            0.0004285097820684314,
+            0.0150000536814332,
+            0.009375033900141716,
+            0.023264234885573387,
+            -0.02092931978404522,
+            0.009997677057981491,
+            -0.005501199048012495,
+            0.014603826217353344,
+            -0.009495317004621029,
+            -0.015339677222073078,
+            0.00479718716815114,
+            -0.01577835902571678,
+            0.010153338313102722,
+            -0.010860888287425041,
+            0.008221727795898914,
+            0.01913214474916458,
+            0.002467579673975706,
+            -0.00941041111946106,
+            0.01146230474114418,
+            0.02075950801372528,
+            -0.025641601532697678,
+            0.008851447142660618,
+            -0.0031998935155570507,
+            -0.0021916350815445185,
+            0.021127434447407722,
+            0.010301923379302025,
+            -0.01594817079603672,
+            0.01573590561747551,
+            -0.008879749104380608,
+            -0.0015053119277581573,
+            -0.01882082223892212,
+            -0.03291521221399307,
+            -0.0015283073298633099,
+            0.012396270409226418,
+            0.004800724796950817,
+            0.0150000536814332,
+            0.011264191009104252,
+            -0.0008950503543019295,
+            -0.006601438857614994,
+            0.0008433107868768275,
+            -0.013485897332429886,
+            -0.004963461309671402,
+            0.010231168940663338,
+            0.017051948234438896,
+            -0.0037217114586383104,
+            0.015650998800992966,
+            0.0088160689920187,
+            0.006137993652373552,
+            -0.0010480580385774374,
+            0.0036138100549578667,
+            -0.002060738392174244,
+            -0.007783046457916498,
+            0.019938750192523003,
+            0.019075538963079453,
+            -0.018509499728679657,
+            -0.0024870370980352163,
+            -0.02141045406460762,
+            0.025882167741656303,
+            0.022188758477568626,
+            0.011165134608745575,
+            -0.04222656786441803,
+            0.00012050455552525818,
+            0.00752832880243659,
+            -0.0013744152383878827,
+            -0.031103884801268578,
+            0.008172199130058289,
+            0.005221716593950987,
+            0.004330204334110022,
+            0.020844414830207825,
+            -0.018424594774842262,
+            0.010853812098503113,
+            0.0070861103013157845,
+            -0.0005766529939137399,
+            0.01019579079002142,
+            -0.0073372903279960155,
+            0.008688710629940033,
+            0.038717120885849,
+            -0.018410442396998405,
+            0.005303084850311279,
+            0.0062370505183935165,
+            0.023320838809013367,
+            -0.011228813789784908,
+            -0.00031485961517319083,
+            0.00800238735973835,
+            0.021608568727970123,
+            -0.012934008613228798,
+            0.005883275531232357,
+            0.067075714468956,
+            -0.012912781909108162,
+            0.031613320112228394,
+            0.020533092319965363,
+            -0.004970536567270756,
+            -0.0104929618537426,
+            -0.012445799075067043,
+            -0.0181132722645998,
+            0.021283095702528954,
+            0.01175947580486536,
+            0.003419233951717615,
+            0.01941516436636448,
+            -0.01610383205115795,
+            0.006636816076934338,
+            0.017547233030200005,
+            0.012601460330188274,
+            -0.007379743270576,
+            -0.0036173479165881872,
+            0.019004784524440765,
+            0.018509499728679657,
+            -0.0030849166214466095,
+            0.00019424450874794275,
+            0.01508495956659317,
+            -0.022075550630688667,
+            -0.012530704960227013,
+            0.01644345559179783,
+            -0.0037358624394983053,
+            0.017773648723959923,
+            -0.02227366529405117,
+            0.024198200553655624,
+            -0.004531855694949627,
+            -0.017179306596517563,
+            0.004698129836469889,
+            -0.007514177821576595,
+            0.022033097222447395,
+            -0.001578720286488533,
+            0.004857328720390797,
+            0.000821199850179255,
+            -0.0018015984678640962,
+            0.01349297259002924,
+            0.002826661104336381,
+            -0.0005120891146361828,
+            0.010507113300263882,
+            -0.0009702275274321437,
+            -0.0013496510218828917,
+            0.024509521201252937,
+            1.9374700059415773e-05,
+            0.005989408120512962,
+            0.0032936439383774996,
+            -0.009261825121939182,
+            -0.004429261200129986,
+            -0.004443412180989981,
+            0.0011842612875625491,
+            0.003877372248098254,
+            -0.007029506377875805,
+            -0.02384442463517189,
+            0.022471778094768524,
+            -0.008540124632418156,
+            0.01175947580486536,
+            0.011349096894264221,
+            0.003665107535198331,
+            0.006824316922575235,
+            -0.009771261364221573,
+            -0.017320817336440086,
+            0.0006633278680965304,
+            -0.01617458648979664,
+            0.006912760436534882,
+            -0.001656550681218505,
+            0.0020235921256244183,
+            0.009445788338780403,
+            -0.002570174168795347,
+            0.0015123875346034765,
+            0.011306644417345524,
+            0.011497682891786098,
+            0.003254728624597192,
+            0.004577846731990576,
+            -0.018721764907240868,
+            -0.020915169268846512,
+            0.005288933869451284,
+            0.013924578204751015,
+            0.004273600410670042,
+            0.0015309606678783894,
+            0.017603836953639984,
+            -0.006166295614093542,
+            -0.022967062890529633,
+            -0.010117961093783379,
+            -0.01584911346435547,
+            0.039396367967128754,
+            0.01800006441771984,
+            -0.02525952458381653,
+            -0.000530220044311136,
+            -0.009728807955980301,
+            0.010450509376823902,
+            -0.00404364662244916,
+            0.014575524255633354,
+            0.005476434715092182,
+            -0.02426895499229431,
+            -0.016797229647636414,
+            -0.010662773624062538,
+            -0.005257094278931618,
+            0.020080260932445526,
+            0.027014248073101044,
+            -0.004956385586410761,
+            0.0061026159673929214,
+            0.019584976136684418,
+            -0.010938717983663082,
+            0.009849091991782188,
+            0.020957622677087784,
+            0.0031698227394372225,
+            -0.007570781745016575,
+            -0.004553082399070263,
+            -0.016514210030436516,
+            0.024467067793011665,
+            0.014250051230192184,
+            -0.005104971118271351,
+            -0.02465103194117546,
+            0.004110863897949457,
+            -0.02428310550749302,
+            -0.01616043597459793,
+            -0.007301913108676672,
+            -0.014143918640911579,
+            -0.004931621253490448,
+            0.0027594438288360834,
+            -0.004836102016270161,
+            0.011278342455625534,
+            -0.0031592093873769045,
+            -0.004531855694949627,
+            -0.0041851564310491085,
+            -0.025584997609257698,
+            -0.02524537406861782,
+            0.007691065315157175,
+            -0.003005317412316799,
+            0.005596718285232782,
+            0.000650945701636374,
+            -0.013400991447269917,
+            -0.011589664034545422,
+            0.00970758218318224,
+            -0.007988235913217068,
+            0.012813725508749485,
+            0.0031149876303970814,
+            -0.009183995425701141,
+            0.0017193458043038845,
+            -0.0012311364989727736,
+            -0.0059752571396529675,
+            0.019924599677324295,
+            0.00916984397917986,
+            0.011724098585546017,
+            -0.03118879161775112,
+            -0.0021633331198245287,
+            0.008349086157977581,
+            0.01828308403491974,
+            -0.02244347706437111,
+            -0.008313708938658237,
+            0.005179263651371002,
+            -0.011228813789784908,
+            -0.013507124036550522,
+            -0.028952933847904205,
+            -0.0030460015404969454,
+            8.175515540642664e-05,
+            -0.018636859953403473,
+            -0.029886899515986443,
+            0.0009038947173394263,
+            -0.007860877551138401,
+            0.016882136464118958,
+            -0.006643891800194979,
+            0.01790100708603859,
+            0.00870286114513874,
+            0.0036173479165881872,
+            0.011554286815226078,
+            0.0025082635693252087,
+            -0.013266556896269321,
+            0.008426916785538197,
+            -0.02671707607805729,
+            -0.01516986545175314,
+            0.009792488068342209,
+            0.014214673079550266,
+            -0.0026126273442059755,
+            5.074458022136241e-05,
+            0.0058372849598526955,
+            0.024254804477095604,
+            0.013584954664111137,
+            0.012474101036787033,
+            0.003042463678866625,
+            -0.005303084850311279,
+            0.005568416323512793,
+            0.0227123461663723,
+            0.008603804744780064,
+            0.004011806566268206,
+            0.007592008449137211,
+            0.00389859895221889,
+            0.0009932229295372963,
+            -0.005943417549133301,
+            -0.00843399204313755,
+            -0.021537812426686287,
+            -0.008908051066100597,
+            0.019938750192523003,
+            -0.010422207415103912,
+            -0.03005671128630638,
+            0.0006023016758263111,
+            -0.0032777239102870226,
+            0.007988235913217068,
+            0.02595292218029499,
+            -0.014172220602631569,
+            0.004970536567270756,
+            0.005536576267331839,
+            -0.007910405285656452,
+            -0.00614860700443387,
+            0.022967062890529633,
+            0.02639160305261612,
+            -0.0034404604230076075,
+            -0.04800017178058624,
+            -0.01671232469379902,
+            0.004305440001189709,
+            -0.007768895477056503,
+            -0.034047290682792664,
+            -0.015424583107233047,
+            0.002757675014436245,
+            -0.03206615149974823,
+            0.010068432427942753,
+            -0.012948159128427505,
+            -0.01214155275374651,
+            0.020264223217964172,
+            0.03223596513271332,
+            0.005154499784111977,
+            0.00014847487909719348,
+            -0.0214670579880476,
+            -0.0021686397958546877,
+            -0.021382151171565056,
+            0.010174565017223358,
+            -0.008801918476819992,
+            -0.010584943927824497,
+            0.0059611061587929726,
+            -0.012191081419587135,
+            0.013266556896269321,
+            0.010797208175063133,
+            0.024778390303254128,
+            0.01584911346435547,
+            -0.01876421831548214,
+            -0.0024746551644057035,
+            -0.01108730398118496,
+            -0.0016450530383735895,
+            -0.009820790030062199,
+            0.006360871717333794,
+            -0.0031592093873769045,
+            0.011264191009104252,
+            0.015184016898274422,
+            0.004432798828929663,
+            0.00046786724124103785,
+            0.038151077926158905,
+            -2.4487851987942122e-05,
+            -0.0003418349369894713,
+            0.006803090218454599,
+            -0.015750056132674217,
+            0.008837295696139336,
+            0.012191081419587135,
+            0.005051904823631048,
+            -0.007698140572756529,
+            0.006739411037415266,
+            0.03099067695438862,
+            -0.01516986545175314,
+            -0.009014182724058628,
+            -0.023504801094532013,
+            0.007549555506557226,
+            -0.0004656561359297484,
+            0.029971804469823837,
+            0.020533092319965363,
+            -0.027509532868862152,
+            -0.016018925234675407,
+            -0.02179253101348877,
+            0.01267929095774889,
+            -0.011497682891786098,
+            -0.003704022616147995,
+            0.012240610085427761,
+            0.02367461286485195,
+            0.0018838511314243078,
+            -0.014455240219831467,
+            -0.008950503543019295,
+            0.008943428285419941,
+            0.014617976732552052,
+            0.003497064346447587,
+            0.007570781745016575,
+            0.00743634719401598,
+            0.008886824361979961,
+            -0.014801939949393272,
+            -0.012254760600626469,
+            0.00767691433429718,
+            -0.010202866978943348,
+            0.01794346049427986,
+            -0.009403335861861706,
+            0.016768928617239,
+            -0.004645063541829586,
+            -0.011964665725827217,
+            -0.0320095494389534,
+            -0.002292460994794965,
+            0.005037753842771053,
+            0.004627374932169914,
+            0.0034705311991274357,
+            -0.0022217060904949903,
+            -0.013160424306988716,
+            0.004673365969210863,
+            0.005073131527751684,
+            1.875283123808913e-05,
+            -0.008313708938658237,
+            -0.00479718716815114,
+            0.004252373706549406,
+            -0.001031253719702363,
+            -0.0073372903279960155,
+            -0.0059858704917132854,
+            0.018481198698282242,
+            0.0066509670577943325,
+            0.00800238735973835,
+            0.03234917297959328,
+            0.02974538877606392,
+            -0.0016353242099285126,
+            -0.004606148693710566,
+            -0.00023304918431676924,
+            -0.01589156687259674,
+            -0.0057205394841730595,
+            -0.023320838809013367,
+            0.022259512916207314,
+            -0.009665128774940968,
+            0.021820832043886185,
+            0.019457615911960602,
+            0.0039304387755692005,
+            -0.008391539566218853,
+            -0.0011586126638576388,
+            -0.007308988366276026,
+            -0.003560743760317564,
+            0.016641568392515182,
+            0.006742948666214943,
+            0.01226891204714775,
+            0.00030933189555071294,
+            -0.028401045128703117,
+            0.026405755430459976,
+            0.000579748535528779,
+            0.0034015451092272997,
+            0.008908051066100597,
+            0.014129767194390297,
+            0.013209952972829342,
+            -0.01235381793230772,
+            0.01768874190747738,
+            -0.03280200436711311,
+            -0.0033484790474176407,
+            -0.016500059515237808,
+            0.002076658420264721,
+            -0.0018520113080739975,
+            -0.0008145665633492172,
+            -0.05354736000299454,
+            0.0003842879377771169,
+            0.005313698202371597,
+            -0.015339677222073078,
+            0.029349161311984062,
+            -0.020589696243405342,
+            -0.017165156081318855,
+            -0.032858606427907944,
+            -0.010238244198262691,
+            0.018099121749401093,
+            0.005617944523692131,
+            0.00035178486723452806,
+            -0.02660386823117733,
+            0.015750056132674217,
+            -0.0047476585023105145,
+            -0.00743634719401598,
+            -0.029858596622943878,
+            -0.010938717983663082,
+            0.009403335861861706,
+            0.004485865123569965,
+            -0.0130825936794281,
+            0.009898620657622814,
+            -0.002890340518206358,
+            0.010867963545024395,
+            -0.0138467475771904,
+            -0.00420284504070878,
+            0.0010020672343671322,
+            -0.005363226868212223,
+            0.012615610845386982,
+            -0.030537845566868782,
+            0.011391550302505493,
+            -0.01352127455174923,
+            0.0008548084879294038,
+            -0.017858553677797318,
+            -0.027707645669579506,
+            0.0020288988016545773,
+            -0.016132133081555367,
+            0.039990708231925964,
+            0.010188715532422066,
+            -0.01481609046459198,
+            0.016995344310998917,
+            -0.002080196049064398,
+            0.017377421259880066,
+            0.0014955832157284021,
+            0.017023645341396332,
+            -0.011971740983426571,
+            0.021056678146123886,
+            -0.0041745430789887905,
+            0.02156611531972885,
+            0.00600002147257328,
+            0.034047290682792664,
+            -0.02033497765660286,
+            0.0070825726725161076,
+            0.003792466362938285,
+            0.005798369646072388,
+            -0.0026816134341061115,
+            0.021424604579806328,
+            0.017985913902521133,
+            0.019698183983564377,
+            0.0037110981065779924,
+            -0.024084990844130516,
+            -0.006137993652373552,
+            0.004867942072451115,
+            0.023858575150370598,
+            0.011150983162224293,
+            -0.016415152698755264,
+            -0.021056678146123886,
+            0.015226469375193119,
+            -0.007860877551138401,
+            0.006771250627934933,
+            -0.00851889792829752,
+            0.00037544354563578963,
+            -0.013259481638669968,
+            0.012233533896505833,
+            -0.012523629702627659,
+            0.0032335021533071995,
+            0.01043635793030262,
+            -0.027806703001260757,
+            0.01511326152831316,
+            -0.00371817359700799,
+            -0.014313730411231518,
+            0.023320838809013367,
+            -0.015877414494752884,
+            -0.013988257385790348,
+            0.005501199048012495,
+            0.008610880002379417,
+            0.01746232621371746,
+            -0.002761212643235922,
+            0.002669231267645955,
+            -0.020504789426922798,
+            -0.012445799075067043,
+            0.004591997712850571,
+            0.02210385352373123,
+            -0.015820810571312904,
+            0.012431648559868336,
+            -0.018099121749401093,
+            0.01364863384515047,
+            0.016202887520194054,
+            0.007149789482355118,
+            0.00573115237057209,
+            -0.007726442534476519,
+            -0.0011471149045974016,
+            0.032151058316230774,
+            -0.0029876285698264837,
+            0.02704254910349846,
+            -0.014915147796273232,
+            0.009615600109100342,
+            0.00518987700343132,
+            -0.013089669868350029,
+            -0.014589674770832062,
+            0.011108530685305595,
+            0.02428310550749302,
+            -0.009530694223940372,
+            -0.023660462349653244,
+            0.0018484735628589988,
+            0.014264201745390892,
+            0.010337301529943943,
+            0.009403335861861706,
+            -0.001997058978304267,
+            -0.0037358624394983053,
+            0.01484439242631197,
+            -0.0032918748911470175,
+            -0.0286557637155056,
+            -0.01633024774491787,
+            -0.004400959238409996,
+            -0.017490629106760025,
+            0.020745357498526573,
+            0.02367461286485195,
+            0.010507113300263882,
+            0.0037570889107882977,
+            0.005540114361792803,
+            0.018297234550118446,
+            0.01843874529004097,
+            0.013330236077308655,
+            0.009304278530180454,
+            0.012700516730546951,
+            -0.004319590982049704,
+            0.02800481766462326,
+            -0.010860888287425041,
+            -0.01046465989202261,
+            -0.015707602724432945,
+            0.0016636261716485023,
+            -0.005193414632230997,
+            -0.012431648559868336,
+            -0.0069481381215155125,
+            0.004369119182229042,
+            0.013330236077308655,
+            -0.012240610085427761,
+            -0.007149789482355118,
+            0.01016748882830143,
+            0.011971740983426571,
+            0.01403071079403162,
+            0.01719345711171627,
+            -0.01633024774491787,
+            -0.011278342455625534,
+            0.0009834941010922194,
+            -0.02844349853694439,
+            -0.0160047747194767,
+            -0.01767459139227867,
+            0.009580222889780998,
+            -0.014469391666352749,
+            0.03724541515111923,
+            0.028471799567341805,
+            0.0073514413088560104,
+            0.031613320112228394,
+            -0.028401045128703117,
+            0.00970758218318224,
+            -0.014398636296391487,
+            0.007132100872695446,
+            -0.004188694059848785,
+            0.003902136581018567,
+            0.002448122017085552,
+            0.020419884473085403,
+            0.00791748147457838,
+            0.014264201745390892,
+            0.008150972425937653,
+            -0.021934039890766144,
+            -0.00662620272487402,
+            -0.012559006921947002,
+            -0.016401002183556557,
+            -0.02595292218029499,
+            0.013719388283789158,
+            0.005087282508611679,
+            -0.003074303502216935,
+            -0.0002704166399780661,
+            0.006566061172634363,
+            -0.0013195802457630634,
+            0.010443433187901974,
+            -0.027608588337898254,
+            -0.01311089564114809,
+            0.0069269114173948765,
+            0.01449061743915081,
+            -0.02114158496260643,
+            0.0014637433923780918,
+            0.008900974877178669,
+            -0.004602610599249601,
+            0.035660505294799805,
+            -0.014674580655992031,
+            -0.008610880002379417,
+            0.0027806703001260757,
+            0.005062518175691366,
+            -0.017660440877079964,
+            -0.02319347858428955,
+            0.0070648835971951485,
+            0.011575513519346714,
+            0.010867963545024395,
+            -0.028570856899023056,
+            0.02087271586060524,
+            -0.0054339817725121975,
+            0.03832089155912399,
+            0.00401534466072917,
+            -0.013337312266230583,
+            0.003962278366088867,
+            0.0011860302183777094,
+            -0.012360893189907074,
+            0.002060738392174244,
+            -0.000340508297085762,
+            0.003700484987348318,
+            0.011228813789784908,
+            -0.014801939949393272,
+            -0.016967041417956352,
+            0.016910437494516373,
+            -0.011476456187665462,
+            -0.0035395172890275717,
+            -0.00943871308118105,
+            -0.016669871285557747,
+            0.007308988366276026,
+            0.025047259405255318,
+            0.0041745430789887905,
+            0.001242634141817689,
+            -0.011632117442786694,
+            -0.0038596836384385824,
+            -0.008575502783060074,
+            0.0011718792375177145,
+            -0.0207029040902853,
+            0.027396325021982193,
+            -0.01801421493291855,
+            0.002410975517705083,
+            0.021099131554365158,
+            0.008709936402738094,
+            -0.003944589756429195,
+            0.016316095367074013,
+            0.026363302022218704,
+            0.01205664686858654,
+            0.0031539027113467455,
+            -0.00902833417057991,
+            1.648646139074117e-05,
+            0.014957600273191929,
+            0.0015123875346034765,
+            -0.0015663381200283766,
+            0.0032140444964170456,
+            -0.006583749782294035,
+            -0.001859086798503995,
+            -0.017561383545398712,
+            0.005564878229051828,
+            0.02125479280948639,
+            -0.004132090136408806,
+            0.020476488396525383,
+            0.015509488992393017,
+            0.011625041253864765,
+            0.010726453736424446,
+            -0.0012390963966026902,
+            -0.002150951186195016,
+            -0.0021668709814548492,
+            -0.007393894251435995,
+            -0.018297234550118446,
+            0.006470541935414076,
+            0.0034493047278374434,
+            0.01822648011147976,
+            0.020504789426922798,
+            -0.02308027073740959,
+            0.010146263055503368,
+            0.014985902234911919,
+            -0.013011839240789413,
+            0.0235755555331707,
+            -0.02006610855460167,
+            0.0024605041835457087,
+            0.02156611531972885,
+            0.013337312266230583,
+            -0.014271277002990246,
+            -0.009219372645020485,
+            0.004662752617150545,
+            -0.01543873455375433,
+            0.000872939417604357,
+            -0.01478778850287199,
+            -0.010783057659864426,
+            0.014533070847392082,
+            -0.01490099634975195,
+            0.01229721400886774,
+            -0.006969364359974861,
+            0.0070967236533761024,
+            -0.017165156081318855,
+            0.015014204196631908,
+            0.008632106706500053,
+            0.01757553406059742,
+            0.0021279556676745415,
+            -0.02518877014517784,
+            -0.006290116813033819,
+            0.021169887855648994,
+            0.009509467519819736,
+            -0.02497650496661663,
+            0.019613277167081833,
+            0.0020094411447644234,
+            -0.019528372213244438,
+            -0.022401023656129837,
+            -0.012686366215348244,
+            -0.005055442452430725,
+            -0.017264213413000107,
+            -0.0011276573641225696,
+            -0.024792540818452835,
+            0.003972891718149185,
+            0.009106164798140526,
+            -0.011978816241025925,
+            -0.019273653626441956,
+            -0.0002315014135092497,
+            -0.01040805596858263,
+            0.006219361908733845,
+            -0.01132787112146616,
+            -0.019853845238685608,
+            0.0070436573587358,
+            0.012127402238547802,
+            -0.0027647505048662424,
+            -0.005646246485412121,
+            -0.02562745101749897,
+            -0.006919836159795523,
+            0.020179318264126778,
+            -0.003994117956608534,
+            -0.004974074196070433,
+            -0.01703779585659504,
+            0.01108730398118496,
+            0.011165134608745575,
+            0.020901018753647804,
+            0.010931642726063728,
+            0.005674548447132111,
+            0.013634482398629189,
+            -0.0029416377656161785,
+            -0.007156865205615759,
+            -9.645892532716971e-06,
+            0.004284213297069073,
+            0.0069233737885952,
+            0.0016636261716485023,
+            0.020235922187566757,
+            -0.01692458800971508,
+            0.018934030085802078,
+            -0.012290137819945812,
+            0.008794843219220638,
+            -0.01855195313692093,
+            -0.0003155229496769607,
+            -0.01067692507058382,
+            0.00015002264990471303,
+            -0.0038455326575785875,
+            -0.009544845670461655,
+            1.9595809135353193e-05,
+            0.00143986358307302,
+            0.008511822670698166,
+            -0.016783079132437706,
+            0.009615600109100342,
+            -0.010839661583304405,
+            -0.004308977629989386,
+            -0.019542522728443146,
+            0.017448175698518753,
+            0.013344387523829937,
+            0.007308988366276026,
+            -0.008278331719338894,
+            -0.006300730165094137,
+            -0.02282555401325226,
+            0.008582578040659428,
+            -0.0014637433923780918,
+            0.002214630600064993,
+            -0.028301987797021866,
+            -0.004358506295830011,
+            0.0024587351363152266,
+            0.020306676626205444,
+            0.0033661676570773125,
+            -0.010712302289903164,
+            0.008978805504739285,
+            -0.023377442732453346,
+            -0.0030159305315464735,
+            -0.023349139839410782,
+            -0.004075486212968826,
+            -0.005112046375870705,
+            0.010103809647262096,
+            0.008101443760097027,
+            0.01549533847719431,
+            -0.009898620657622814,
+            0.01478778850287199,
+            -0.004793649073690176,
+            0.012049571610987186,
+            -0.0012152165872976184,
+            0.01337976474314928,
+            -0.0003111007681582123,
+            -0.01505665760487318,
+            0.02158026583492756,
+            -0.008738238364458084,
+            0.001741456682793796,
+            -0.0052853962406516075,
+            0.0027346794959157705,
+            -0.017702894285321236,
+            -0.004553082399070263,
+            -0.014455240219831467,
+            0.01957082375884056,
+            -0.022584985941648483,
+            0.004153316840529442,
+            -0.005055442452430725,
+            -0.016075529158115387,
+            0.029434068128466606,
+            0.022896308451890945,
+            -0.0035483618266880512,
+            0.015070809051394463,
+            0.0017989451298490167,
+            0.0015061964513733983,
+            0.016202887520194054,
+            -0.015806660056114197,
+            -0.013025989755988121,
+            0.035066161304712296,
+            0.0176321379840374,
+            0.0004313841927796602,
+            -0.0023260696325451136,
+            -0.0037323245778679848,
+            0.0032193511724472046,
+            0.008023613132536411,
+            -0.010450509376823902,
+            -0.01141277700662613,
+            -0.017051948234438896,
+            0.009997677057981491,
+            -0.006155682262033224,
+            0.004446949809789658,
+            -0.00025891896802932024,
+            -7.617214578203857e-05,
+            -0.01516986545175314,
+            0.022584985941648483,
+            -0.017717044800519943,
+            -0.00602832343429327,
+            0.004687516950070858,
+            0.0207029040902853,
+            -0.02541518583893776,
+            0.01616043597459793,
+            -0.007107336539775133,
+            -0.013429293408989906,
+            0.012573158368468285,
+            -0.015127412974834442,
+            -0.0107335289940238,
+            -0.003175129182636738,
+            -0.008009462617337704,
+            -0.022754797711968422,
+            0.017929309979081154,
+            -0.013627407141029835,
+            0.015650998800992966,
+            -0.00287972716614604,
+            0.000562502013053745,
+            0.002057200763374567,
+            0.0032582662533968687,
+            0.02250008098781109,
+            0.030735958367586136,
+            0.010415131226181984,
+            -0.005465821363031864,
+            -0.021000074222683907,
+            0.012983537279069424,
+            0.0020589695777744055,
+            0.013202877715229988,
+            -0.002253545681014657,
+            -0.002135031158104539,
+            -0.016613267362117767,
+            -0.023122724145650864,
+            -0.0013010069960728288,
+            -0.01761798746883869,
+            -0.02552839368581772,
+            -0.01774534583091736,
+            -0.007047194987535477,
+            0.011349096894264221,
+            0.020405733957886696,
+            0.0009755341452546418,
+            -0.005384453106671572,
+            0.01559439580887556,
+            -0.006941062398254871,
+            0.002080196049064398,
+            -0.006594363134354353,
+            -0.002032436430454254,
+            -0.008200501091778278,
+            -0.021863285452127457,
+            0.029660483822226524,
+            -0.010103809647262096,
+            0.01141277700662613,
+            -0.005472897086292505,
+            -0.00965097825974226,
+            0.005575491581112146,
+            0.00900003220885992,
+            0.024849144741892815,
+            0.006452853325754404,
+            0.03141520544886589,
+            0.020151015371084213,
+            -0.018891576677560806,
+            -0.0006071660900488496,
+            0.022684043273329735,
+            -0.0037570889107882977,
+            -0.004889168310910463,
+            -0.0031786670442670584,
+            -0.017985913902521133,
+            -0.012007118202745914,
+            0.007839650847017765,
+            -0.013471745885908604,
+            -0.006636816076934338,
+            -0.00837738811969757,
+            0.018070818856358528,
+            -0.017773648723959923,
+            0.025698205456137657,
+            -0.006488230545073748,
+            0.005154499784111977,
+            0.031924642622470856,
+            0.014363259077072144,
+            0.019698183983564377,
+            -0.011794853024184704,
+            -0.031075581908226013,
+            -0.00683846790343523,
+            -0.007882103323936462,
+            0.009558996185660362,
+            -0.01062032114714384,
+            0.005204027984291315,
+            -0.02006610855460167,
+            -0.018948180601000786,
+            0.012863253243267536,
+            -0.017476476728916168,
+            0.013924578204751015,
+            -0.009955224581062794,
+            0.027226513251662254,
+            -0.001228483160957694,
+            0.0121698547154665,
+            -0.002614396158605814,
+            -0.017801949754357338,
+            0.01205664686858654,
+            0.004089637193828821,
+            -0.023943481966853142,
+            0.010337301529943943,
+            -0.001864393474534154,
+            -0.014037786051630974,
+            -0.018410442396998405,
+            -0.001228483160957694,
+            -0.005200490355491638,
+            -0.009658053517341614,
+            0.025457637384533882,
+            0.014101465232670307,
+            0.016231190413236618,
+            0.0035359796602278948,
+            0.0016335553955286741,
+            -0.011073152534663677,
+            0.009573147632181644,
+            -0.006803090218454599,
+            0.000612030504271388,
+            -0.0034404604230076075,
+            0.011893910355865955,
+            -0.004252373706549406,
+            -0.01543873455375433,
+            0.002324300818145275,
+            0.008561351336538792,
+            0.0143278818577528,
+            5.947839599684812e-05,
+            0.009587298147380352,
+            0.0009790719486773014,
+            -0.009764186106622219,
+            -0.00046388726332224905,
+            -0.0060884649865329266,
+            0.0069375247694551945,
+            0.02400008589029312,
+            -0.012516554445028305,
+            -0.01681138016283512,
+            0.014504768885672092,
+            0.004033033270388842,
+            -0.017589686438441277,
+            0.018367990851402283,
+            0.004103788174688816,
+            -0.022358570247888565,
+            0.0035412863362580538,
+            0.010662773624062538,
+            -0.009523618966341019,
+            -0.014469391666352749,
+            -0.0010339069413021207,
+            -0.015509488992393017,
+            0.03130199760198593,
+            0.00025073791039176285,
+            0.016910437494516373,
+            -0.01062032114714384,
+            -0.0025967073161154985,
+            -0.013309009373188019,
+            0.013103820383548737,
+            -0.015750056132674217,
+            -0.007273610681295395,
+            0.00035620704875327647,
+            -0.02298121340572834,
+            0.01741987280547619,
+            0.0013425755314528942,
+            0.00794578343629837,
+            0.003436922561377287,
+            -0.010556641966104507,
+            0.009820790030062199,
+            0.0018237093463540077,
+            0.006668656133115292,
+            -0.014915147796273232,
+            0.023618008941411972,
+            0.006413938011974096,
+            0.011235889047384262,
+            -0.01352127455174923,
+            -0.01546703651547432,
+            0.002253545681014657,
+            0.04134920611977577,
+            -0.01306136790663004,
+            0.0002593612007331103,
+            0.01562269777059555,
+            7.081024523358792e-05,
+            0.004156854469329119,
+            -0.0023755980655550957,
+            -0.0034493047278374434,
+            -0.034585028886795044,
+            0.005370302125811577,
+            -0.0006328147719614208,
+            0.008073141798377037,
+            -0.008992956951260567,
+            -0.005430444143712521,
+            -0.0040684109553694725,
+            -0.008625030517578125,
+            -0.00314859626814723,
+            -0.01478778850287199,
+            0.008363237604498863,
+            -0.014533070847392082,
+            -0.028910480439662933,
+            0.03166992589831352,
+            -0.006887996103614569,
+            -0.00533846253529191,
+            -0.021396303549408913,
+            0.007570781745016575,
+            -0.01370523776859045,
+            0.002663924591615796,
+            0.010967019945383072,
+            0.008865597657859325,
+            0.008738238364458084,
+            0.002762981690466404,
+            0.0009516543359495699,
+            -0.004255911335349083,
+            0.010273621417582035,
+            -0.00835616234689951,
+            -0.0005943417199887335,
+            -0.0014186371117830276,
+            -0.02725481428205967,
+            -0.01405901275575161,
+            -0.010592019185423851,
+            -0.007811348885297775,
+            -0.00435143057256937,
+            0.012608535587787628,
+            0.007174553815275431,
+            -0.0017485321732237935,
+            -0.022938761860132217,
+            -0.014745336025953293,
+            -0.0034103896468877792,
+            0.008349086157977581,
+            -0.01838214136660099,
+            0.01064862310886383,
+            0.0016830838285386562,
+            0.009976450353860855,
+            0.014292503707110882,
+            0.00302831269800663,
+            0.0016742395237088203,
+            0.003458149265497923,
+            0.014886845834553242,
+            0.012601460330188274,
+            -0.005801907740533352,
+            -0.021382151171565056,
+            -0.0024605041835457087,
+            -0.007068421691656113,
+            -0.0034687623847275972,
+            -0.012332591228187084,
+            0.03039633482694626,
+            0.0029805530793964863,
+            0.0176321379840374,
+            0.007832575589418411,
+            0.009573147632181644,
+            0.010507113300263882,
+            0.029377464205026627,
+            0.006049550138413906,
+            -0.0029929352458566427,
+            -0.008603804744780064,
+            -0.02141045406460762,
+            -0.013146273791790009,
+            0.009898620657622814,
+            0.013294858857989311,
+            0.03597182780504227,
+            -0.034896351397037506,
+            0.012162779457867146,
+            -0.011101454496383667,
+            -0.008264180272817612,
+            -0.008533049374818802,
+            0.0044717141427099705,
+            0.010068432427942753,
+            0.014370334334671497,
+            -0.0023402206134051085,
+            -0.01175947580486536,
+            0.009070787578821182,
+            0.023547254502773285,
+            -0.012983537279069424,
+            0.03016991913318634,
+            0.027325568720698357,
+            0.023066120222210884,
+            0.01761798746883869,
+            -0.012537780217826366,
+            0.026462359353899956,
+            -6.484030018327758e-05,
+            0.0006261814851313829,
+            -0.018084971234202385,
+            -0.00587973790243268,
+            -0.004294826649129391,
+            -0.004362043924629688,
+            0.013316085562109947,
+            -0.0025471788831055164,
+            -0.0008468485320918262,
+            0.015099111013114452,
+            -0.0037323245778679848,
+            -3.927453508367762e-05,
+            -0.0059752571396529675,
+            0.015127412974834442,
+            0.0002699744363781065,
+            0.019457615911960602,
+            -0.04270770028233528,
+            0.019726485013961792,
+            -0.021396303549408913,
+            -0.003102605463936925,
+            -0.0020890405867248774,
+            0.009523618966341019,
+            0.00764861237257719,
+            -0.007142714224755764,
+            -0.014334957115352154,
+            0.004524780437350273,
+            0.014066088013350964,
+            -0.0006726144347339869,
+            0.005080206785351038,
+            -0.0027541371528059244,
+            0.0017228835495188832,
+            -0.010089659132063389,
+            -0.005862049292773008,
+            5.842812606715597e-05,
+            -0.010570792481303215,
+            -0.008646257221698761,
+            -0.03232087194919586,
+            0.01062032114714384,
+            0.010209942236542702,
+            0.007294837385416031,
+            0.004252373706549406,
+            -0.022018946707248688,
+            -0.012318440712988377,
+            0.012948159128427505,
+            -0.01132787112146616,
+            -0.002570174168795347,
+            -0.0020005968399345875,
+            0.005387990735471249,
+            0.0004705205501522869,
+            0.010160413570702076,
+            -0.003472300246357918,
+            0.010938717983663082,
+            -0.014603826217353344,
+            0.018084971234202385,
+            0.03229256719350815,
+            -0.00533846253529191,
+            -0.019967053085565567,
+            0.000907432462554425,
+            -0.009077862836420536,
+            0.003813692834228277,
+            0.007641536649316549,
+            -0.021962342783808708,
+            -0.02173592709004879,
+            0.005087282508611679,
+            -0.013153349049389362,
+            -0.007783046457916498,
+            0.014830241911113262,
+            0.005221716593950987,
+            0.010839661583304405,
+            0.01751893013715744,
+            0.010889190249145031,
+            0.00840569008141756,
+            -0.026533113792538643,
+            0.0006734988419339061,
+            -0.012686366215348244,
+            -0.00590803986415267,
+            0.002239394700154662,
+            -0.0017591454088687897,
+            0.004591997712850571,
+            -0.00010115749319083989,
+            -0.011200511828064919,
+            -0.0007057808106765151,
+            0.010903340764343739,
+            0.027919910848140717,
+            -0.004146241117268801,
+            -0.0022570835426449776,
+            -0.014257126487791538,
+            0.0010100272484123707,
+            0.036481261253356934,
+            -0.012275987304747105,
+            -0.017434025183320045,
+            0.01757553406059742,
+            0.027452928945422173,
+            0.005954030901193619,
+            -0.0032529598101973534,
+            -0.0027364485431462526,
+            0.00039600671152584255,
+            0.011978816241025925,
+            0.007719367276877165,
+            -0.002692226553335786,
+            -0.002131493529304862,
+            0.004761809483170509,
+            0.035773713141679764,
+            -0.0015203474322333932,
+            -0.010330225341022015,
+            -0.024240652099251747,
+            0.010974096134305,
+            0.00277182599529624,
+            0.004754733759909868,
+            0.002313687466084957,
+            0.023504801094532013,
+            0.019655730575323105,
+            0.014858543872833252,
+            0.00470520555973053,
+            -0.009771261364221573,
+            0.03005671128630638,
+            -0.008172199130058289,
+            -0.004542469047009945,
+            -0.02838689461350441,
+            -0.005087282508611679,
+            -0.015650998800992966,
+            0.008929276838898659,
+            -0.0002221042668679729,
+            -0.005016527138650417,
+            -0.013422218151390553,
+            0.019726485013961792,
+            0.01064862310886383,
+            -0.011073152534663677,
+            0.012608535587787628,
+            -0.027070851996541023,
+            0.02346234768629074,
+            -0.03826428949832916,
+            -0.0071993181481957436,
+            -0.006421013735234737,
+            -0.010514188557863235,
+            -0.02627839520573616,
+            -0.0030353881884366274,
+            0.007372668012976646,
+            0.014250051230192184,
+            0.0022570835426449776,
+            0.0022818478755652905,
+            0.02282555401325226,
+            0.0016415152931585908,
+            -0.01508495956659317,
+            0.017985913902521133,
+            0.03492465242743492,
+            0.012926933355629444,
+            0.00155837822239846,
+            0.015990622341632843,
+            0.0027789014857262373,
+            0.0021633331198245287,
+            0.00788917951285839,
+            -0.0034493047278374434,
+            -0.007938707247376442,
+            0.008582578040659428,
+            0.006990591064095497,
+            -0.02763689123094082,
+            0.003824306186288595,
+            -0.00010408719390397891,
+            -0.009898620657622814,
+            -0.022429324686527252,
+            0.011568437330424786,
+            0.01913214474916458,
+            -0.004224071744829416,
+            -0.006771250627934933,
+            -0.0043337419629096985,
+            0.0006836698739789426,
+            0.006785401608794928,
+            -0.002412744564935565,
+            -0.000515184598043561,
+            -0.0015176940942183137,
+            -0.02368876338005066,
+            -0.010316074825823307,
+            0.013570803217589855,
+            -0.005175726022571325,
+            0.010358527302742004,
+            0.005242943298071623,
+            -0.018792519345879555,
+            -0.016938740387558937,
+            0.006852618884295225,
+            -0.016188737004995346,
+            0.01364863384515047,
+            0.00027948213391937315,
+            -0.0023172253277152777,
+            0.024962352588772774,
+            0.03594352304935455,
+            -0.013188726268708706,
+            0.005119122099131346,
+            -0.019386861473321915,
+            -0.0245378240942955,
+            -0.019004784524440765,
+            0.022613288834691048,
+            0.008306633681058884,
+            -0.018792519345879555,
+            0.012304289266467094,
+            -0.002752368338406086,
+            -0.008865597657859325,
+            -0.0066721937619149685,
+            -0.006845543161034584,
+            -0.008547200821340084,
+            0.020533092319965363,
+            -0.004935159347951412,
+            0.0038950610905885696,
+            0.019047237932682037,
+            0.04075486212968826,
+            -0.008292482234537601,
+            -0.01349297259002924,
+            0.0054339817725121975,
+            0.004878554958850145,
+            0.0018626245437189937,
+            -0.004814875777810812,
+            -0.008016537874937057,
+            -0.0185944065451622,
+            -0.01740572229027748,
+            -0.009615600109100342,
+            -0.0020041344687342644,
+            0.020122714340686798,
+            -0.007768895477056503,
+            -0.01354957651346922,
+            -0.0245378240942955,
+            -0.028188779950141907,
+            -0.006580212153494358,
+            0.02319347858428955,
+            0.017179306596517563,
+            -0.00013200224202591926,
+            -4.3337418901501223e-05,
+            -0.012042495422065258,
+            -0.01968403346836567,
+            0.015509488992393017,
+            0.00037876018905080855,
+            -0.01773119531571865,
+            -0.0065448349341750145,
+            -0.015919867902994156,
+            0.005787756759673357,
+            -0.0051368107087910175,
+            0.004574308637529612,
+            0.05057565122842789,
+            0.0005271245026960969,
+            0.008441068232059479,
+            -0.026575567200779915,
+            0.009106164798140526,
+            -0.01583496294915676,
+            -0.017377421259880066,
+            -0.00012702727690339088,
+            -0.022301966324448586,
+            -0.0030955299735069275,
+            -0.018792519345879555,
+            -0.007386818993836641,
+            0.008568426594138145,
+            0.009799563325941563,
+            -0.013896276243031025,
+            -0.013669860549271107,
+            -0.024693485349416733,
+            -0.0035872769076377153,
+            -0.001539805089123547,
+            0.009311353787779808,
+            0.007825499400496483,
+            -0.03693409264087677,
+            0.002536565763875842,
+            -0.0009507699287496507,
+            0.0005779796629212797,
+            0.01208494883030653,
+            -0.008667483925819397,
+            -0.003063690150156617,
+            -0.0214670579880476,
+            0.006452853325754404,
+            0.007103798910975456,
+            -0.013500047847628593,
+            0.012375044636428356,
+            -0.012863253243267536,
+            -0.007669838611036539,
+            -0.0005426021525636315,
+            0.03379257395863533,
+            0.011794853024184704,
+            -0.0025153392925858498,
+            -0.00654129683971405,
+            -0.011150983162224293,
+            0.02654726430773735,
+            -0.011349096894264221,
+            -0.026405755430459976,
+            0.028910480439662933,
+            0.013245330192148685,
+            0.0005974372616037726,
+            0.020151015371084213,
+            -0.0017936384538188577,
+            -0.01397410687059164,
+            0.009601449593901634,
+            0.004266524687409401,
+            0.012424572370946407,
+            -0.00014460546663030982,
+            0.0214670579880476,
+            0.0018555490532889962,
+            0.0027399861719459295,
+            -0.0036226543597877026,
+            0.004687516950070858,
+            -0.03093407303094864,
+            -0.0006549257086589932,
+            -0.003102605463936925,
+            -0.008398614823818207,
+            0.011207587085664272,
+            -0.002421588869765401,
+            0.006558985915035009,
+            -0.022188758477568626,
+            -0.028485950082540512,
+            0.011342021636664867,
+            -0.016231190413236618,
+            -0.021396303549408913,
+            0.015339677222073078,
+            -0.0007641536649316549,
+            0.00614860700443387,
+            -0.007443422917276621,
+            -0.023278385400772095,
+            -0.019386861473321915,
+            0.02064630016684532,
+            0.012212308123707771,
+            -0.01190098561346531,
+            0.012750045396387577,
+            -0.021113283932209015,
+            -0.012948159128427505,
+            -0.001894464367069304,
+            0.0090566361322999,
+            -0.0019864458590745926,
+            0.01325240544974804,
+            -0.012198156677186489,
+            -0.00334494118578732,
+            0.0031592093873769045,
+            -0.011589664034545422,
+            0.008158047683537006,
+            0.01978308893740177,
+            0.008193425834178925,
+            -0.009339655749499798,
+            0.003422771580517292,
+            -0.011922212317585945,
+            0.013500047847628593,
+            0.015537790954113007,
+            0.018311386927962303,
+            -0.006498843897134066,
+            0.0020412809681147337,
+            -0.01370523776859045,
+            0.0063962494023144245,
+            -0.009827865287661552,
+            -0.004878554958850145,
+            -0.0005333155859261751,
+            0.004921008367091417,
+            -0.011179285123944283,
+            -0.0025772498920559883,
+            -0.026830283924937248,
+            -0.0013823752524331212,
+            -0.020533092319965363,
+            0.005303084850311279,
+            0.023207630962133408,
+            -0.007514177821576595,
+            0.0006045127520337701,
+            -0.0039092120714485645,
+            0.014476466923952103,
+            0.020688753575086594,
+            0.00870286114513874,
+            0.00482548912987113,
+            -0.006081389728933573,
+            0.02179253101348877,
+            0.011603815481066704,
+            -0.017816102132201195,
+            0.00164239970035851,
+            -0.001509734196588397,
+            -0.0014336726162582636,
+            0.011625041253864765,
+            0.004443412180989981,
+            -0.020844414830207825,
+            0.0028602697420865297,
+            -0.010089659132063389,
+            0.007818424142897129,
+            0.014702882617712021,
+            -0.00031906069489195943,
+            -0.02486329711973667,
+            -0.009827865287661552,
+            0.021919889375567436,
+            -0.015070809051394463,
+            -0.0008291598060168326,
+            0.00764861237257719,
+            0.023264234885573387,
+            -0.024792540818452835,
+            -0.006831392180174589,
+            -0.0023455272894352674,
+            -0.003933976404368877,
+            -0.005193414632230997,
+            0.0010489424457773566,
+            0.0214670579880476,
+            0.004415110219269991,
+            -0.004234685096889734,
+            0.005724077112972736,
+            0.006470541935414076,
+            0.02325008250772953,
+            0.0040577976033091545,
+            0.028047269210219383,
+            -0.00943871308118105,
+            -0.0023809047415852547,
+            -0.0019988277927041054,
+            -4.775960405822843e-05,
+            0.0160047747194767,
+            -0.005430444143712521,
+            0.020320827141404152,
+            0.009445788338780403,
+            0.0036934094969183207,
+            0.016514210030436516,
+            0.02319347858428955,
+            -0.006767712999135256,
+            -0.004793649073690176,
+            0.019160445779561996,
+            0.0053667644970119,
+            -0.008858522400259972,
+            -0.02194819226861,
+            0.01141277700662613,
+            0.0014124460285529494,
+            -0.0006956097786314785,
+            0.012106175534427166,
+            0.0024375086650252342,
+            0.006300730165094137,
+            0.008172199130058289,
+            -0.016684021800756454,
+            0.004241760354489088,
+            -0.004698129836469889,
+            -0.01860855706036091,
+            0.010910416021943092,
+            -0.012092024087905884,
+            -5.201595922699198e-05,
+            -0.0010002984199672937,
+            0.014165145345032215,
+            -0.01703779585659504,
+            -0.0019103841623291373,
+            -0.0070719593204557896,
+            -1.3024719009990804e-05,
+            -0.0009162768837995827,
+            0.008320784196257591,
+            -0.006254739128053188,
+            -0.010938717983663082,
+            -0.010938717983663082,
+            0.027509532868862152,
+            -0.004991763271391392,
+            0.003916287794709206,
+            0.019330257549881935
+          ],
+          "embedding": [
+            0.007204997251390758,
+            -0.03173324616100502,
+            -0.020505790884701024,
+            0.0037451608231038273,
+            0.009817085391002229,
+            0.026883177166072393,
+            -0.016477826654817607,
+            0.06949311853773875,
+            -0.01224876667472692,
+            0.031952566873802865,
+            0.013633671951171586,
+            -0.005041437355104989,
+            0.02663392750069342,
+            -0.022099132865267587,
+            0.006665702000169926,
+            0.0030467129715919033,
+            0.007931802508415542,
+            0.009984477355413895,
+            -0.010362836464183985,
+            -0.03180848181251013,
+            -0.013542377333474449,
+            -0.013572345022577053,
+            -0.03979985300694044,
+            -0.007698412130292187,
+            -0.005953452234356532,
+            0.018890228917932694,
+            0.02217446994660065,
+            0.005158357501784182,
+            -0.05468694715835175,
+            0.00789994048345594,
+            0.017130184174786147,
+            -0.0026591475669416114,
+            -0.02081141846859738,
+            -0.008403928318906045,
+            -0.030925346724945978,
+            -0.03855023691059801,
+            0.038853354785097736,
+            0.01612011653434478,
+            -0.035635702395196575,
+            0.026266623757525934,
+            0.043450373922427316,
+            -0.007254334835408145,
+            -0.015376804869420366,
+            -0.0017276202867766907,
+            0.004650414207094434,
+            -0.002365514361535186,
+            0.011906139905776183,
+            -0.0005664194302009289,
+            -0.0191723447249157,
+            -0.025546124518037998,
+            0.01025005257280377,
+            -0.014436532079436437,
+            0.029277529346694118,
+            -0.0045414983847733344,
+            -0.02260315456913697,
+            0.01717918656248626,
+            0.013666132349359846,
+            0.057804090189845364,
+            0.056600576093100735,
+            0.051208022172527734,
+            0.03553760896972431,
+            0.02094599510567085,
+            -0.02438674472832137,
+            -0.0027797634437333432,
+            0.03614248430646102,
+            -0.026729308357113124,
+            -0.002251394534452815,
+            0.018929251842788906,
+            0.00689697483318378,
+            -0.026603606434115974,
+            -0.0541443728344337,
+            0.027164630314707807,
+            -0.004335272389193875,
+            -0.013967321193992958,
+            0.003323121377857836,
+            -0.0245295956412367,
+            0.024489335129134787,
+            0.06212059558640156,
+            0.009214058207162371,
+            -0.002082293435905329,
+            0.013659934201796686,
+            -0.003344037927706573,
+            -0.00689041317919184,
+            0.020235785390418023,
+            0.0212739457082623,
+            0.0016782346585238381,
+            -0.013484221664059555,
+            -0.06465582506604173,
+            0.011882797526607326,
+            -0.032970655964890155,
+            -0.04027700439305653,
+            0.025396847912528105,
+            0.013861157517716707,
+            0.00604596157583048,
+            -0.00744611407420936,
+            0.04038084298991919,
+            0.021507532290328625,
+            0.005983822806623904,
+            -0.012711110369915742,
+            0.02410138142963901,
+            -0.018372791517880082,
+            0.021016674128298246,
+            0.013387285956415756,
+            0.02039383525127615,
+            0.010700292312227496,
+            0.0053934660497349735,
+            -0.014807678014449691,
+            0.009422805142740376,
+            0.0414357625490679,
+            0.12206398267002624,
+            -0.010877509834324155,
+            -0.011079302503786522,
+            -0.04107714853756596,
+            0.006631397986165343,
+            -0.031914267104222384,
+            -0.0179965295457466,
+            -0.057530036803978266,
+            -0.0320488164246567,
+            0.008208202363320509,
+            -0.0020203090915064162,
+            0.03502055780919601,
+            0.04464599164916666,
+            0.019332152079287082,
+            0.017133943087431935,
+            -0.039885565008919065,
+            0.04703059988373572,
+            -0.029370696964304416,
+            0.0008494086835971101,
+            0.0014082189719987074,
+            -0.030973762582071615,
+            -0.006561157852024518,
+            0.011634653995510726,
+            -0.019896826887411713,
+            -0.034275921680252924,
+            0.026274465530039866,
+            -0.007728234125236442,
+            0.04814514040690998,
+            0.06550407296280117,
+            -0.02749051703615343,
+            -0.010041099502187074,
+            0.02019601967763537,
+            -0.01871701432558228,
+            0.008532829759864928,
+            -0.020164138072967613,
+            -0.045196925901010855,
+            -0.004523517988516156,
+            -0.002113469149506399,
+            0.07929650663710812,
+            -0.01111269708042164,
+            0.03358254444720164,
+            -0.019973382759721527,
+            -0.0007145592863325417,
+            -0.0011740862928390678,
+            0.02166182480535788,
+            0.004543228023398081,
+            -0.013377688428761516,
+            -0.01758250073647576,
+            0.009548292014596834,
+            -0.04753617258004735,
+            -0.007125744019792479,
+            -0.009930358894299303,
+            -0.007931672044235996,
+            -0.024561936537429774,
+            0.012979195012881556,
+            0.009209145250061479,
+            0.04586756662401364,
+            0.04311807301838248,
+            -0.0036241105681042675,
+            -0.06579256720896123,
+            0.04967191755416882,
+            0.015475886231466817,
+            0.03157651656768371,
+            0.02515941109300714,
+            0.01773554572598468,
+            0.005824418191900522,
+            0.016317635547562007,
+            0.007650872352957098,
+            0.010015166465492963,
+            -0.033203307142867844,
+            0.040272188258443496,
+            -0.013408423788025047,
+            0.01036004294342104,
+            0.012391622448703885,
+            0.04214857862444935,
+            0.03418032388760535,
+            0.015263018079889338,
+            0.018786704580603317,
+            0.02704685343985817,
+            0.015348111464412869,
+            0.0013031600434754704,
+            0.03241427528588285,
+            0.011689924942195664,
+            0.04544406114742012,
+            0.016186615914968734,
+            0.011624500626103958,
+            -0.008272953078864385,
+            -0.02363080189189375,
+            0.02784298119317616,
+            -0.010585980875564109,
+            0.01455013604305903,
+            0.01749228557615982,
+            -0.054742056843690476,
+            -0.005876213780171779,
+            -0.010556605098730201,
+            0.003967456397644932,
+            0.014663888851767762,
+            0.0015548476559947807,
+            -0.003858367677774835,
+            0.012504286824484335,
+            -0.025806882498628832,
+            -0.014216299453397403,
+            0.0031620924516417956,
+            -0.006448390918604307,
+            -0.02785577502346934,
+            -0.00766402687921159,
+            0.005275000635711884,
+            -0.0090448751942,
+            0.019702662630533,
+            0.007009091837467867,
+            -0.01847307100043082,
+            -0.0006774105206839772,
+            0.032952656101827446,
+            0.015900033044760432,
+            -0.02170121261931376,
+            -0.017964879081469612,
+            -0.005062849745084668,
+            -0.03171847232054206,
+            -0.038177398521606544,
+            0.029426291950046455,
+            -0.025687201683643784,
+            0.030158867757101797,
+            0.015780411479509624,
+            -0.007011484049169482,
+            -0.012114023533118393,
+            0.020661407942701628,
+            -0.025067530380881376,
+            0.009777190971413269,
+            -0.004160771288889298,
+            -0.028825369485727344,
+            0.008307290529469729,
+            -0.014415903546305744,
+            -0.007561235629544945,
+            -0.03201704135271719,
+            -0.015182869388110948,
+            -0.00501561450194676,
+            -0.0003863746169672397,
+            -0.01073403445816663,
+            -0.03321103354401307,
+            -0.002304357889348034,
+            0.017001213960120047,
+            0.00028222146729658097,
+            -0.04002882852650754,
+            0.014509588052058002,
+            -0.004150541038999649,
+            -0.034333837380045466,
+            0.0564137596224324,
+            0.03584898691608747,
+            -0.006830698606679202,
+            0.006121456445207273,
+            -0.04505674213649008,
+            0.04358467937307492,
+            0.010847859073780811,
+            -0.04918983713702805,
+            0.049190442935189385,
+            -0.00528960443684769,
+            0.01419195371129309,
+            0.040316024601793406,
+            0.004600843996974165,
+            -0.006780970176765864,
+            0.005206651850992678,
+            0.03397517915308201,
+            -0.010037223444627569,
+            -0.027377913828275806,
+            -0.03328136706915221,
+            0.02562568440972934,
+            -0.03485770844248482,
+            -0.020069979230687374,
+            -0.0052041408068332524,
+            0.029818953377223505,
+            0.028789460147370508,
+            0.004071274130619537,
+            0.01928375754300958,
+            0.0014970947734202088,
+            0.010502599795027616,
+            0.016914145702515263,
+            0.011942576738188502,
+            -0.014909953528043828,
+            -0.003432489584767786,
+            -0.05733768990013957,
+            0.0030710780388681002,
+            -0.02627280695622545,
+            0.007930906480371135,
+            -0.031227917075485834,
+            0.013132797714872345,
+            0.002514351081823354,
+            -0.007152021475569949,
+            -0.0066245205132327055,
+            -0.005279224195782313,
+            0.028512607222275155,
+            -0.00921960825771865,
+            -0.0359704105589433,
+            -0.021347597388353182,
+            0.011200044645812517,
+            -0.0004171294784927543,
+            -0.016784468873218786,
+            -0.0018746544944553878,
+            0.007749103417240419,
+            0.02390272097796833,
+            -0.005128540108845805,
+            0.006397849195755672,
+            0.01214922062611769,
+            0.01163371266250103,
+            0.012307186722130844,
+            -0.004982589662882946,
+            -0.016343724553037307,
+            0.019806887143115643,
+            0.013139657463054087,
+            0.0205801609265216,
+            -0.03767676007629612,
+            -0.0249879011550972,
+            0.05547392586363707,
+            0.005903313549319051,
+            -0.05568806298774435,
+            0.018827693208840233,
+            -0.007931178941307645,
+            0.008183992600768387,
+            -0.010319022946041053,
+            0.05111352384595183,
+            -0.024897203290807753,
+            -0.02096678205188722,
+            0.006436523335212348,
+            -0.04917084523640266,
+            -0.010459284852442706,
+            -0.011138558192296276,
+            0.014789370043397917,
+            -0.03105725978752438,
+            -0.004934745004818259,
+            0.015595801375817122,
+            -4.687585727052446e-05,
+            0.0028886067775750254,
+            -0.00850352807499054,
+            -0.0034885579171087815,
+            0.016339681487281293,
+            -0.0034050317046826813,
+            -0.008309570448487812,
+            -0.003615165868022203,
+            -0.027638409919916757,
+            0.023352910161586186,
+            0.026364511529991978,
+            0.001249936082895606,
+            0.001987948967948376,
+            0.01756629246122769,
+            0.004656075485833687,
+            -0.006752579977941239,
+            -0.013681608638716132,
+            -0.03405936202918687,
+            -0.0401867651078737,
+            0.022991180183813725,
+            -0.02318908140203705,
+            0.018299512436825923,
+            0.00904081847012441,
+            0.019697091729447287,
+            -0.019236130984371835,
+            0.015159917756323988,
+            -0.0387297415849285,
+            -0.018793498297161176,
+            -0.005256288213225733,
+            -0.010414697711744128,
+            0.0007039894599589104,
+            -0.01739645696111503,
+            -0.013728845328097506,
+            0.022709522955937343,
+            -0.022789243441781803,
+            -0.0036763802589970026,
+            -0.01429671717544519,
+            0.0023674126354206932,
+            0.049448298344058644,
+            -0.009926033487593551,
+            -0.023371285193090733,
+            0.020846792916650956,
+            -0.026996519120163966,
+            0.011246908457705283,
+            0.04091365159640201,
+            0.005469162850197613,
+            0.005640919201612091,
+            0.029575753750229278,
+            -0.0007860452837141593,
+            -0.0052805237594885516,
+            -0.009782745443933828,
+            0.021178503581190618,
+            -0.0009176512672832309,
+            -0.05245650420332918,
+            -0.007698364493410058,
+            -0.039027418406703554,
+            0.005448840777995117,
+            0.017581625031764185,
+            0.025522517097715556,
+            -0.020775492472580457,
+            -0.015906320383873178,
+            -0.000643669839811819,
+            -0.0086896084429604,
+            0.0029399233482444638,
+            0.014381240985647907,
+            0.021086693776922936,
+            0.002436575081260609,
+            -0.004913514868101904,
+            -0.011582101006213432,
+            0.014400297663309155,
+            0.0467181039341492,
+            -0.01722692180137712,
+            0.0011622224560709315,
+            -0.013036540368441797,
+            -0.06208454638917413,
+            -0.019160212331432766,
+            0.0351496273293468,
+            0.02429814055116254,
+            -0.009896662169762772,
+            -0.01647101145477158,
+            -0.002915821099802562,
+            -0.03557863734177486,
+            -0.01786433999615293,
+            -0.010699364747311715,
+            0.02390901423299708,
+            0.007213809581960646,
+            0.0005601364043388203,
+            0.00916614601011734,
+            0.014737491634516238,
+            0.019303599465763203,
+            0.008864535607492504,
+            -0.0575839606069681,
+            0.018051837786406608,
+            0.006315830080225406,
+            0.02525619013082567,
+            -0.015671924543113065,
+            0.003988943108210209,
+            -0.03179878281917948,
+            -0.029440641244429775,
+            0.040586322748158446,
+            -0.03418722657593859,
+            -0.030650542761032924,
+            0.006566745611226466,
+            0.009571089937153852,
+            0.004491158558315032,
+            -0.011352813042040581,
+            -0.02535505987721779,
+            -0.013381948772812133,
+            -0.030918676056735445,
+            -0.02192708443081135,
+            -0.005232667170255381,
+            -0.03342408726436631,
+            -0.07498634115550039,
+            -0.006245638452568239,
+            -0.049946896465756174,
+            -0.03900385398225758,
+            0.04458524125677807,
+            -0.006106489377959481,
+            0.02398410968928338,
+            -0.04615183642824994,
+            -0.01165382599292015,
+            -0.04890013373371541,
+            0.013624324077017377,
+            0.01476398305411351,
+            0.020202554298470666,
+            0.00014553149242532853,
+            0.036224316512987846,
+            0.00021498496267774312,
+            -0.05055208658013767,
+            -0.057482169168904106,
+            -0.003960451575823462,
+            -0.010217078509789989,
+            -0.005593738972150311,
+            -0.034090261748104665,
+            0.05835341009022349,
+            0.0116359661127526,
+            0.04203678184748103,
+            0.03447818335186531,
+            -0.006747564131379555,
+            -0.025756962754841885,
+            -0.028123416424745064,
+            -0.008073625379614702,
+            0.047544241458556825,
+            -0.029605671083240564,
+            0.008171081332993893,
+            -0.026007452205474395,
+            0.0067084589781962805,
+            -0.0070645448617940995,
+            -0.0012298101311792947,
+            0.012372286141738867,
+            0.006377319686763927,
+            -0.015024537877377194,
+            0.007934745664790507,
+            0.026041755081605937,
+            -0.007303731324541215,
+            0.0407000170257006,
+            -0.012990703097937777,
+            -0.033189084013863494,
+            -0.022525679964848666,
+            0.01657717698191603,
+            0.03440284885162039,
+            0.010230633738702899,
+            -0.008707808754455952,
+            -0.0363777020540216,
+            0.011184076269109602,
+            0.031774012431843524,
+            0.01504703222643543,
+            -0.022509833303978505,
+            0.025230974050270107,
+            0.017502662694813216,
+            -0.018700579139537456,
+            -0.03511734747709348,
+            0.015162183119327901,
+            -0.017711638132862063,
+            0.018190335050681287,
+            -0.01590174153215615,
+            0.011103110869700727,
+            0.0025189370696524936,
+            -0.00920305384736575,
+            0.01929652151514404,
+            -0.006196715938358087,
+            0.011426611271994685,
+            0.007058186102245829,
+            -0.018191352599117545,
+            0.0027202185072220937,
+            0.057310566886380025,
+            -0.000531639834386418,
+            0.01494975681909879,
+            0.03845692752400366,
+            -0.001971953300066856,
+            0.045554909803324765,
+            -0.02532332999916258,
+            -0.042616337063845953,
+            0.02633963974172818,
+            -0.02970705577683783,
+            -0.01624154998005974,
+            -0.02620993449678864,
+            -0.021490817420853802,
+            -0.004038424045768319,
+            0.009940887982818171,
+            -0.02692933845395842,
+            -0.023801570856681796,
+            -0.001948460540662095,
+            0.02290911497874141,
+            -0.03300532378029689,
+            0.006525488380791647,
+            0.007039306647666766,
+            -0.003685585408710752,
+            0.03183735852538168,
+            -0.027197064244321705,
+            0.006861928286688311,
+            0.011209869849682412,
+            -0.001094897323854524,
+            -0.005328676992017907,
+            -0.004007474089818478,
+            0.010796946231051235,
+            -0.030543173497906445,
+            -0.012879972141706173,
+            0.00935894527375688,
+            0.0017573071057611829,
+            -0.010050882212831222,
+            -0.03102132152986303,
+            -0.002915141291985554,
+            0.007634137769426993,
+            -0.011094093612593696,
+            0.01199609810193786,
+            -0.010458016758814742,
+            -0.04929347607956551,
+            0.010973132159729983,
+            0.028148117094907874,
+            -0.025504019123994758,
+            0.012470127357818321,
+            0.009182729379865986,
+            -0.016020133756226275,
+            0.016320111225714855,
+            -0.012051271446461976,
+            0.024098195821827578,
+            0.04317620443481784,
+            0.024592886889234396,
+            0.007727787037954994,
+            0.00771746776156926,
+            -0.0003147538723468374,
+            0.022572868970938624,
+            -0.016683340264320543,
+            0.013672085891597506,
+            0.020827021901755415,
+            0.02092309219721837,
+            0.0057130785714458055,
+            -0.003901728981013356,
+            -0.0036968415923910314,
+            0.002227024825579113,
+            -0.02106678364798051,
+            0.014578546755457675,
+            0.03128101864424301,
+            -0.005234627629617124,
+            -0.022394744692442837,
+            0.002430617645746389,
+            -0.015780389432069438,
+            4.7277885648917e-05,
+            0.013357803020920043,
+            0.013777651543690907,
+            -0.027713499926132385,
+            -0.0018999112228756628,
+            0.028364630573666852,
+            0.009146793705085586,
+            -0.009891297866910313,
+            -0.01644269527910728,
+            -0.0003903768506971414,
+            0.006512217314869576,
+            0.02408785204739448,
+            0.0018406499545367782,
+            -0.007360473750031044,
+            0.01484497577088404,
+            0.016723277095327158,
+            0.003629911330658209,
+            -0.0004820567356447713,
+            0.02394406451074397,
+            0.023794019615473026,
+            0.001632266615911801,
+            -0.02064379152040184,
+            -0.02755693521684878,
+            0.027335938400284726,
+            0.02769248863677192,
+            -0.006832353240623308,
+            0.027324973360715134,
+            0.003937207909967385,
+            0.013580154030133622,
+            0.02076818768224654,
+            -0.01784151382870951,
+            -0.017617903734311997,
+            0.0017316476451235418,
+            -0.009605294810027804,
+            -0.009829196696575069,
+            0.004891780008779742,
+            -0.011655402383771472,
+            -0.01124379942906224,
+            0.043205454954628145,
+            0.004713701851783307,
+            0.003560551284131962,
+            0.007894840447501912,
+            -0.018881214526749082,
+            -0.00945538175144614,
+            0.0019040259741796817,
+            0.029505863720301916,
+            -0.001400002868254808,
+            0.03315866433970851,
+            -0.00830542045504805,
+            0.008602408964961391,
+            -0.03202191233287069,
+            -0.006978289755114341,
+            0.0326499698728755,
+            0.015314305362524091,
+            0.015104436300213667,
+            0.0021257042489266787,
+            -0.0010840650756005028,
+            0.02434348355847526,
+            -0.007174433778817674,
+            -0.04839373063710409,
+            0.022696994450018916,
+            -0.004779511451775743,
+            0.023588216956749154,
+            0.0382733980620746,
+            -0.004232286553944853,
+            0.003815507826545929,
+            0.023531223314846965,
+            -0.01887650634213847,
+            0.015055403379782985,
+            -0.01918944132635701,
+            -0.017597900721414636,
+            0.02176630010543207,
+            -0.008896608357798277,
+            0.01732387427588561,
+            0.01764363756727683,
+            -0.012180043444768312,
+            -0.019003360499321047,
+            0.01229139850966224,
+            0.0025710091163655626,
+            -0.017075658234265562,
+            0.01797412486853335,
+            0.018372010308966135,
+            0.006723179570342314,
+            0.015417677746651683,
+            -0.00047634684109332134,
+            0.017608296628666153,
+            -0.001049984114818522,
+            -0.008416990841821452,
+            0.01827827907328822,
+            -0.009568717277044265,
+            -0.01658089563620308,
+            0.009808478422454319,
+            -0.05193977747210182,
+            0.010644335931025957,
+            0.0016781339270411392,
+            -0.01562285100294917,
+            0.00405036330963539,
+            0.007102724536527262,
+            0.005195069570261247,
+            0.0028773630901681767,
+            0.004610474981005765,
+            0.042921486858787584,
+            0.013213672397112759,
+            -0.0064030584930089415,
+            -0.010711457508461704,
+            -0.017480309425763254,
+            -0.027151012983864797,
+            -0.005204425551873271,
+            0.009927884476991038,
+            0.022601230152836282,
+            0.008046297278661014,
+            -0.0031858480011267704,
+            -0.02589628103025927,
+            0.019723430185137954,
+            0.03604835658149313,
+            0.062423099201768716,
+            0.010930268386637612,
+            0.02925564269324381,
+            0.02439517070384333,
+            0.023930432096006114,
+            0.013036828091946605,
+            0.006910311087503408,
+            -0.032995841694293995,
+            0.012813650252889403,
+            0.01333714655191133,
+            0.03678966264802025,
+            0.012571138353467183,
+            -0.05610151838497368,
+            3.0777966587107344e-05,
+            0.009791774904454618,
+            0.041959488802722524,
+            0.01956296954085397,
+            0.000155217569370113,
+            -0.007285602454264675,
+            0.031547870838711856,
+            -0.026518385135879206,
+            -0.012299593280363843,
+            -0.0026551378470968594,
+            -0.006190117573235779,
+            -0.003807907001161504,
+            -0.004991957475076049,
+            0.036983035940964804,
+            -0.015648565314743963,
+            -0.019256463332683493,
+            0.0004300362961407231,
+            -0.0015686065121077472,
+            0.0011311524136286674,
+            0.0032236641835276464,
+            0.013795345278976164,
+            0.015385029265915279,
+            -0.004800448639353721,
+            0.026514926384796363,
+            -0.008699453553651681,
+            0.022186817580012775,
+            -0.013427553742493343,
+            -0.011927259975371873,
+            0.017776228476138103,
+            -0.021243797085501972,
+            -0.01667280589187595,
+            -0.003688045116580565,
+            0.006189970874536939,
+            0.0020911968266404467,
+            0.0006097761986763103,
+            -0.004685468039463817,
+            -0.01011357666327168,
+            0.022631082517271542,
+            0.012398000826637277,
+            -0.006721588925828951,
+            0.018142966317821054,
+            -0.0021330638043572654,
+            -0.011869184647489572,
+            0.025073455250141923,
+            -0.0015392945266926425,
+            -0.0337085890598252,
+            0.01267302113143573,
+            -0.013438601165482187,
+            0.02673290895281225,
+            -0.009586762706902736,
+            -0.011208527168019762,
+            0.019144928964011168,
+            -0.0017591316941248897,
+            -0.005459150067129732,
+            -0.029230079356228025,
+            0.004679497471366242,
+            -0.011723195942933286,
+            0.015899377439148873,
+            0.0022054234777254627,
+            0.029187759252864872,
+            -0.018290694895057157,
+            0.00792533824593637,
+            0.011719198071061566,
+            0.014612375997487872,
+            0.005419943379355563,
+            0.010515265723144554,
+            -0.006893314445458116,
+            -0.04424495426064596,
+            5.8241472764594054e-05,
+            -0.005719143565673274,
+            0.0038220021965914754,
+            0.013106940982933836,
+            -0.01203955335201393,
+            0.011260626096963215,
+            -0.0279896016489392,
+            -0.0295228261396555,
+            0.036474811902902415,
+            -0.04900747926861378,
+            0.019261089742811332,
+            0.00014104646541594166,
+            -0.03057197731717424,
+            -0.04331886560073797,
+            -0.03232957159604667,
+            -0.019401402365464362,
+            0.004102283356333861,
+            0.0002610070704924167,
+            0.0012280372705500212,
+            -0.014025012323929194,
+            -0.014135990809256075,
+            0.013269444858490211,
+            0.00992155580941576,
+            0.011225162609266308,
+            -0.006545044517282357,
+            0.002368862850663672,
+            -0.02063183105457681,
+            -0.026981383275694142,
+            -0.02774194980145225,
+            0.013337303400678109,
+            -0.0025902548982556788,
+            -0.02794339093668326,
+            -0.025606606546251152,
+            -0.0029852498102770587,
+            -0.01906873442473352,
+            0.006508845592362198,
+            0.03268861529625085,
+            0.0355975143313428,
+            0.010066906387952277,
+            0.01690250467284985,
+            -0.017201544758038183,
+            0.048345309114140714,
+            -0.019031171948182872,
+            0.03188329748476941,
+            -0.0005845169448621133,
+            -0.024140202991373587,
+            0.00011597263700677706,
+            0.03203623528903764,
+            -0.019285565857567896,
+            -0.01332755507465857,
+            -0.010914187315481855,
+            -0.003374912271519852,
+            0.02419326154642231,
+            0.02443825785057301,
+            -0.0037923303809811746,
+            0.00022800399604730175,
+            0.015893265651968885,
+            0.003154527829977343,
+            0.007688529734165207,
+            0.030623379985627763,
+            0.014052282574575454,
+            0.01127235345371133,
+            -0.04164707563955819,
+            -0.011811587719246128,
+            -0.003763086694358232,
+            0.005909410979841885,
+            0.002913220159138154,
+            -0.002671613084376487,
+            0.0045765194162907885,
+            -0.007241703279856166,
+            -0.005855949422953439,
+            0.0016973541436777935,
+            -0.001576326424468957,
+            0.008393873711385013,
+            -0.0037217094191171083,
+            -0.03498879678492834,
+            -0.026096985039773694,
+            -0.021280075788049788,
+            0.006437497348154639,
+            0.0070214556591898375,
+            -0.00972721128392027,
+            0.01949553762684561,
+            -0.0029404319702066146,
+            0.000271037702827666,
+            -0.010734867909488861,
+            -0.0029784341515646075,
+            -0.01206464664516104,
+            -0.019544778160755956,
+            0.00035139045159739705,
+            0.013457601448003152,
+            0.00025947376767512006,
+            -0.03165522602526639,
+            0.011979811705146604,
+            0.006642849985850598,
+            0.013181162886675883,
+            -0.024499404540711422,
+            -0.02310714308893576,
+            -0.028028912907092744,
+            0.010570122662701183,
+            -0.0066866321204157725,
+            -0.024617571284817428,
+            -0.004097807033448707,
+            -0.0074576460628928256,
+            -0.0020417256533259887,
+            -0.02470953739335365,
+            -0.011408461668188081,
+            0.023537349682260345,
+            -0.006373652338136973,
+            0.01936338676879376,
+            -0.017636435070085086,
+            -0.017036091876814843,
+            -0.002198685044404603,
+            0.015584758061011048,
+            -0.038943923931820196,
+            -0.002076529978164001,
+            -0.008452513321945455,
+            0.006551055965075869,
+            -0.028840587857625097,
+            -0.007128279504360441,
+            0.0199397499488538,
+            0.012424319366271584,
+            -0.03518875279748495,
+            0.013596027106451115,
+            0.022191403935861113,
+            0.01218507466930854,
+            -0.0038551718784437783,
+            0.03910075115429626,
+            -0.0025215641203030336,
+            -0.005422688956956598,
+            0.016759888275451905,
+            0.04462538468871483,
+            0.023708097134660096,
+            -0.03152743757744915,
+            -0.008668774531128668,
+            -0.015224825785549947,
+            0.004102648962663789,
+            -0.025867795193966987,
+            0.004676262922216584,
+            -0.011739753544166856,
+            0.01645951757031186,
+            0.011253400919230348,
+            0.013958707917471253,
+            0.0376596076542735,
+            -0.005077354977296842,
+            0.02668910096392324,
+            0.0010332284250006117,
+            -0.00856723931523911,
+            0.0014431148305642308,
+            -0.01412579094241589,
+            0.029892393553325205,
+            0.02269599036836208,
+            0.014421409701746036,
+            0.029248195108770955,
+            -0.03588867295960255,
+            0.02458655544632528,
+            -0.0010013539341895672,
+            0.008368648583431747,
+            0.018150734970019018,
+            0.02628022565029391,
+            0.015423867373844644,
+            0.012477080028047823,
+            -0.02513018750675568,
+            -0.022180001014682045,
+            -0.0027550350630288997,
+            -0.031944843677988374,
+            -0.0014668093006914267,
+            -0.009789309083948147,
+            0.026386027599666254,
+            0.0046988411391149,
+            -0.019803160322638583,
+            -0.003931806845108628,
+            0.004482097428470392,
+            -0.00244660100774939,
+            -0.010441732346892935,
+            -0.004649318716095894,
+            -0.01869464992812615,
+            -0.017838643679294407,
+            0.023677878122153204,
+            -0.023958455640817666,
+            -0.025238725541146534,
+            0.014582798102040157,
+            0.0013223590277629197,
+            -0.002295641913710489,
+            0.0008238518461567685,
+            -0.02115533275282518,
+            -0.04214275933987015,
+            -0.027817083427786927,
+            0.0037588803464689412,
+            0.03565056002940463,
+            -0.026788865866928923,
+            0.014162550819567465,
+            -0.011336583458519,
+            0.019175861037658344,
+            -0.005127229083828615,
+            0.015531116410874209,
+            -0.018251290727445846,
+            -0.044630374763679266,
+            0.021833307809708984,
+            -0.012087009688208446,
+            0.02720742725750304,
+            0.021497809277370574,
+            -0.026937653159100156,
+            -0.013572733244188407,
+            0.01358214612023832,
+            0.0020408253706382332,
+            -0.009652962552750464,
+            -0.0063411929391088626,
+            -0.010533309167065193,
+            0.003901271588567942,
+            -0.04496345296706673,
+            -0.008761188928422701,
+            0.06948704056954039,
+            0.005916180375965137,
+            0.03157905519638486,
+            0.014668818318138927,
+            -0.028448496686381258,
+            -0.04008949536631331,
+            -0.00462253589823598,
+            -0.0003701522833020314,
+            -0.00024410762276269235,
+            -0.01850585249309217,
+            0.017699615637510956,
+            -0.002712069912576265,
+            -0.009767338815933488,
+            -0.013119464222297351,
+            -0.013743610418034631,
+            -0.030375365173543157,
+            -0.0030316327877227783,
+            -0.008632689470358149,
+            -0.006565457264917239,
+            -0.0328089275944709,
+            -0.011713996290006943,
+            0.029773910045828737,
+            0.02740100904353622,
+            -0.0061926915497348915,
+            0.00638737826122608,
+            -0.010244154924811975,
+            0.0010396203272677069,
+            -0.001264454445070139,
+            -0.004059969507786729,
+            0.008715730726831364,
+            0.0014767015415289076,
+            0.031074767756160294,
+            0.004761858344125113,
+            0.00987870172044607,
+            -0.059926117793699066,
+            -0.009294021263133636,
+            0.021188432871397475,
+            0.016602826531924377,
+            -0.009125784136292983,
+            0.013946188378030194,
+            -0.0017670794602549428,
+            -0.006837334179368295,
+            -0.028290055353601393,
+            0.024556541126233806,
+            0.04543639597357669,
+            0.0031311343860416473,
+            -0.011199639773247286,
+            0.014965677855595444,
+            0.0018248533506196013,
+            0.015507721118783542,
+            -0.011409402378400194,
+            -0.01338737990904486,
+            0.03063410115541065,
+            0.009305362935335817,
+            0.0025578631174065385,
+            0.012727648343848194,
+            -0.0038680554900795763,
+            -0.01173215289303824,
+            -0.01568862495293563,
+            -0.004101703129244437,
+            -0.03649200408657064,
+            -0.0022107849490954137,
+            0.009659780679763728,
+            0.0034091693454195806,
+            0.008848380026939594,
+            0.01842435611499587,
+            0.03546926214224266,
+            0.013478278853234764,
+            0.005970293629431872,
+            -0.0069197533761813455,
+            0.023196445250565058,
+            0.010403914879205351,
+            0.010017272564548626,
+            0.015076807790462658,
+            -0.029649260409273275,
+            -0.012737152920662164,
+            0.00373276777994811,
+            0.013239909450830258,
+            0.0001682544254742693,
+            0.02030652051051447,
+            0.014143961923579586,
+            0.0010147733405793957,
+            -0.012509446388074923,
+            0.0020437769941487626,
+            0.007405578085797006,
+            0.013458276946120546,
+            0.0006779114540819997,
+            -0.004250745498730309,
+            0.0009149988500358458,
+            -0.008355906072311008,
+            0.040391939952386505,
+            -0.01831218885700067,
+            -0.016793844108223176,
+            0.035064882870542116,
+            0.11416006426189942,
+            0.0010630856112440689,
+            -0.015660014600661857,
+            -0.017142534874931403,
+            -0.0086899013413391,
+            0.016611825572132997,
+            0.01609650977226868,
+            0.009780106186629793,
+            0.009622367309076686,
+            -0.000672926017804395,
+            0.024310949465350565,
+            0.02944698490225121,
+            -0.0037169531700178868,
+            -0.010990838766452194,
+            -0.036819646475041955,
+            0.022923962979125287,
+            -0.004596366586768123,
+            -0.007108041562635549,
+            -0.002604794313552557,
+            0.012641223813290853,
+            0.013668775645501733,
+            0.005125100040230003,
+            0.0077993841846437815,
+            -0.011500851674334447,
+            -0.017334458072700107,
+            0.0056819751688881774,
+            -0.015982234644378358,
+            -0.0076533013358874935,
+            -0.013127172654362694,
+            -0.011837346554842119,
+            -0.006562312362371076,
+            0.0029877420008976425,
+            -0.008572723854050838,
+            0.00854083969164562,
+            -0.010939639783598493,
+            0.016185642759084087,
+            -0.008052282332943615,
+            -0.0011992212936609377,
+            0.0010378646897070758,
+            -0.0025399515501741718,
+            0.010712865639618352,
+            0.01451357949543628,
+            -0.02521401163091652,
+            0.020836357346214075,
+            -0.008252813367155418,
+            0.0006409807249997337,
+            0.015338809580802425,
+            -0.04672526649678653,
+            0.022728876784937357,
+            0.005129225753457052,
+            -0.018372222092794564,
+            0.0066182736895667805,
+            -0.02048375780196863,
+            -0.00896382274306482,
+            0.0021738974344880794,
+            0.004491167150658828,
+            0.0034280799511763956,
+            0.015611133773752265,
+            0.018126362497548308,
+            0.00014383671280220997,
+            0.007927708693242186,
+            -0.006973076292062532,
+            0.008118313055163714,
+            -0.006791836664375275,
+            -0.01164567445668625,
+            -0.010894577039363014,
+            0.00014731542306100506,
+            -0.008865132658571428,
+            -0.011415696755798914,
+            -0.017922318360382588,
+            0.00358538001727405,
+            0.00289327637128695,
+            0.010406054711003452,
+            -0.007757607207626101,
+            -0.010994775479337082,
+            -0.011716096880261664,
+            -0.0013333474316266438,
+            -0.01231144419953554,
+            0.009792603083441917,
+            -0.02884953407490196,
+            0.0029097409399144306,
+            0.025277274938032477,
+            0.009263530315933738,
+            -0.028037993274175277,
+            -0.02532252134134532,
+            -0.0018596440554143638,
+            -0.013847737486669237,
+            0.009910107581119272,
+            0.013952962280235523,
+            -0.011807495289410774,
+            0.003896325037406752,
+            -0.02144921729633576,
+            -0.01628545945817568,
+            -0.0040148671532777005,
+            -0.016032493124255323,
+            0.004580041957001551,
+            -0.011375180364988195,
+            -0.007789996450583554,
+            -0.022041665680945066,
+            -0.005890065942573431,
+            -0.050455058269193866,
+            0.014401045842891128,
+            0.030219548461217365,
+            -0.004580747668649223,
+            -0.028684680969792875,
+            -0.015525870112797619,
+            0.011121401958547706,
+            -0.008879749268609245,
+            -0.019454243014778435,
+            -0.007839434838464604,
+            0.018908534797107714,
+            -0.0019283067432864216,
+            0.0024769961730655835,
+            -0.006732653593696035,
+            0.027604805021030646,
+            0.01366593691107021,
+            0.03740746059041736,
+            -0.014662268336336578,
+            0.02539666271785517,
+            0.02371809783638648,
+            -0.036126848124425154,
+            -0.007627445211958312,
+            0.03657481892016626,
+            0.01092515982671624,
+            0.0011046573579274704,
+            0.012350047525582187,
+            -0.006340802546498992,
+            -0.031330519599469066,
+            -0.0013691360532301728,
+            -0.0014676630169621005,
+            -0.023111265345539388,
+            0.027359543575867225,
+            0.018091458372134273,
+            0.014237156751820934,
+            0.0030993618578507172,
+            -0.0014037082734049385,
+            0.013257269860435359,
+            0.01021315143964527,
+            0.030885417052806473,
+            -0.003225065956381245,
+            -0.012825341172770057,
+            0.013476039084668826,
+            0.01852164843904835,
+            -0.009965495459579544,
+            0.012541079912270015,
+            -0.00824122613985912,
+            -0.018124134280975132,
+            0.03192263766222747,
+            0.0001484308749572035,
+            0.010888733498667343,
+            -0.010649515858317311,
+            -0.013082335214246402,
+            0.032614733822945205,
+            -0.00999075761269065,
+            0.004257643407198955,
+            -0.030723533031261387,
+            -0.003820776505926347,
+            -0.010471792928758869,
+            0.01737608180148319,
+            0.004597677757231315,
+            0.02095148515295301,
+            0.029253627700524396,
+            -0.011604450820008121,
+            -0.01996325509217509,
+            -0.001158626158150122,
+            -0.020583793738536985,
+            0.004168757459099133,
+            -0.03632338333261702,
+            -0.019373660230711246,
+            0.022259216603413172,
+            -0.010364620831688015,
+            -0.004751193403783363,
+            0.00444187509644351,
+            0.0008572202661850934,
+            0.014943779567410408,
+            -0.025408263453313398,
+            0.017755758370543148,
+            0.004627392860417132,
+            0.007402073071525438,
+            -0.024913329140362466,
+            0.013760771254234878,
+            -0.006087346663657887,
+            -0.03780469033982951,
+            0.005509388120163784,
+            0.03756141956381493,
+            0.02381019168406692,
+            -0.024580249929652097,
+            0.01807584361080568,
+            -0.03111039245576191,
+            0.023245542501125073,
+            -0.004868090951368273,
+            5.1544460403943004e-05,
+            0.00640235757127173,
+            -0.004698373089469779,
+            -0.00621383035264185,
+            -0.0025132415458431587,
+            -0.008746634331769936,
+            -0.01611801559493664,
+            -0.01193535623883546,
+            -0.03381789115349148,
+            -0.024119754887378317,
+            0.013881102852385332,
+            0.0019232668966222687,
+            -0.0095263729740736,
+            0.018652966857927805,
+            -0.03775912497511837,
+            -0.012022352025516115,
+            -0.023289377270709736,
+            0.023579672990954228,
+            -0.028605627815260902,
+            0.019235397553197357,
+            -0.025838655372560505,
+            -0.021458526449310157,
+            0.01508157885177254,
+            -0.010064396230734452,
+            -0.0016708624560165368,
+            -0.015042071642751917,
+            -0.03670785416972153,
+            -0.006267614265977843,
+            0.02298763967146357,
+            -0.012506444728541897,
+            0.00029871598584066644,
+            0.02671390618094565,
+            -0.003059865833779805,
+            -0.011103452976820622,
+            -0.0017382684168257506,
+            0.010923055147701782,
+            -0.02938686260440486,
+            0.014300097108092281,
+            0.0038571851420598074,
+            -0.0037614499297117365,
+            0.0010991440540809893,
+            -0.010108413486243776,
+            0.02127952161226619,
+            0.009142545227414532,
+            -0.0030260119510809205,
+            0.040589877142650374,
+            0.008677453889680373,
+            0.0017339508605300814,
+            0.008732950895530873,
+            0.01063352447960556,
+            0.006618635029823979,
+            -0.014986307792626314,
+            0.016905861015681957,
+            0.013101237900847744,
+            0.0002162791516784468,
+            0.019309343995079022,
+            0.0021263376627204313,
+            -0.024516267132902416,
+            0.013339486637573757,
+            -0.02311659751478777,
+            0.0020575982953961246,
+            0.02391695586495566,
+            -0.020440172706421535,
+            -0.001656132876701879,
+            -0.006304133667169861,
+            0.000543457661904495,
+            0.022135707414160034,
+            -0.0230903871986617,
+            -0.023444763940153112,
+            0.009604600033024832,
+            0.01721387169484476,
+            0.01827905276638783,
+            0.02824773106648473,
+            -0.007100625424382998,
+            0.01348510178676837,
+            0.0002604386361122316,
+            -0.011802178189607182,
+            0.007664157334475143,
+            0.0074175377458313594,
+            -0.0044008460682833875,
+            0.025491035291032547,
+            -0.0017015687016376205,
+            -0.006134010425966364,
+            -0.005379657499404915,
+            -0.01584573519366957,
+            0.0034386456145185213,
+            -0.013860742817569676,
+            -0.013355238174583658,
+            0.0040342831827141845,
+            -0.0208160950322342,
+            0.10904592104153767,
+            -0.0037038089210379766,
+            -0.019472012964045247,
+            0.0063554319446707044,
+            0.003716719884684827,
+            -0.003692786470980806,
+            -0.003353843210051379,
+            0.007788605566526459,
+            -0.011254480693190834,
+            -0.0045751956417398095,
+            -0.013976039724381875,
+            -0.001989262807706565,
+            0.010067374586433014,
+            0.009201983771978006,
+            -0.005902980676079493,
+            -0.02029714318363332,
+            0.013031676901335606,
+            -0.016970144750162643,
+            0.008874670945038486,
+            0.0008311255074953139,
+            0.00884167159995975,
+            0.001716603195908536,
+            0.01854951250316076,
+            -0.009310881405486074,
+            -0.01325242738760595,
+            0.029667589863813217,
+            -0.005616511284128333,
+            -0.0038179509972925954,
+            -0.0004690716809300012,
+            -0.005068974756744239,
+            0.0021204231125051274,
+            -0.027958274497127258,
+            0.022127088236172387,
+            -0.017271838492631593,
+            0.010267709087809265,
+            0.01316800560257366,
+            0.005599367128954229,
+            0.01817806970002991,
+            -0.007115594036834107,
+            -0.02627643123342057,
+            0.006542742528003751,
+            -0.003433874053082844,
+            0.006631427284537971,
+            0.02316295746074422,
+            -0.014573351469966297,
+            -0.033520864104177005,
+            -0.013707022143307006,
+            -0.01877574599842729,
+            -0.005992004412448576,
+            -0.0043698881826095435,
+            -0.01481352026460129,
+            0.006424121402963946,
+            0.01382938882387991,
+            0.018136021761515388,
+            0.0012697483790911433,
+            -0.011531136974861559,
+            0.00609187274277241,
+            0.01076133059240578,
+            0.008400887242930113,
+            -0.01533504932731406,
+            0.011439227832642668,
+            -0.028718487721012004,
+            0.01522697877548034,
+            -0.012452665599305595,
+            -0.024938748080573947,
+            -0.0022082450217489366,
+            0.017589399637694264,
+            0.0037824832255756763,
+            -0.004355445812154662,
+            -0.018778668474337727,
+            0.019825087149714933,
+            0.024692878851503508,
+            0.003883497401229232,
+            0.019470881143244197,
+            0.041834635328088364,
+            -0.003544509491805351,
+            0.027139164223766474,
+            -0.02129723143323512,
+            0.01461354860170749,
+            -0.03441729954693155,
+            0.001893713808284104,
+            -0.004222657393633789,
+            0.0013677766871517654,
+            -0.003811928806805752,
+            0.020142937759360904,
+            -0.009158202577174012,
+            0.0030638058587375543,
+            0.024854053046139163,
+            -0.018665378030958756,
+            0.00805903734779081,
+            0.010625429315145887,
+            -0.012856237251902233,
+            0.028535299711292306,
+            -0.006335094045840286,
+            0.00305422126851784,
+            -0.0297883488914759,
+            -0.0029192872993071216,
+            0.0026067044379754047,
+            -0.006097749614529602,
+            -0.00968023644800927,
+            -0.007433916539502931,
+            -0.006202274026852405,
+            0.0038354607448269555,
+            -0.021471361566683354,
+            0.003195646827154078,
+            -0.004390751282171024,
+            -0.003323447423321121,
+            0.02518759723554948,
+            0.025857746688368054,
+            0.016893707217894103,
+            0.012739170140289531,
+            0.008553625305650129,
+            0.019805351588748434,
+            0.01654964927067909,
+            0.011549329129728739,
+            -0.020026013396046632,
+            0.01150644479272541,
+            -0.004159611921476011,
+            -0.0022937297602724912,
+            0.014815466072709131,
+            -0.01572171840394285,
+            -0.02276187015864222,
+            0.024693009795978448,
+            -0.019725315346836404,
+            0.00998768979020729,
+            -1.1702210281215188e-06,
+            -0.011129745730507545,
+            -0.002591293237053776,
+            -0.018034357254990988,
+            0.00012070821244376275,
+            -0.023051045150566527,
+            -0.028465270110388716,
+            -0.00048603008831669926,
+            -0.000574546300981984,
+            -0.012992623897880395,
+            0.02064747165046032,
+            0.014820638761258362,
+            -0.024820174379739057,
+            0.02666518374464728,
+            -0.017430076811109155,
+            -0.01495051200726725,
+            -0.005861181182847845,
+            0.00100881727574559,
+            -0.0055223282819321295,
+            0.0034719859452138017,
+            -0.010180752654175722,
+            -0.009961678082468036,
+            -0.012134423047419183,
+            -0.028073672819317354,
+            0.011574683159318163,
+            0.016120674453124317,
+            -0.015043147648505721,
+            0.031834693135959095,
+            -0.008802805725050891,
+            -0.0018698230152007877,
+            0.016179742266521146,
+            0.005096232332051617,
+            -0.017187077376166815,
+            -0.018999569809271876,
+            0.012251308792871571,
+            0.013279090498325945,
+            -0.009588297039279112,
+            0.006630431200804809,
+            -0.014130176783207386,
+            -0.005092662672039885,
+            0.007020236623084732,
+            -0.005049363863784624,
+            0.0055051712211635915,
+            -0.00995314895304919,
+            0.0052743456988306485,
+            0.005841370337169303,
+            -0.0029434616355351975,
+            0.004677049987338193,
+            -0.018568941264514256,
+            0.020031253265629777,
+            0.0018725390936655962,
+            0.01332532735283085,
+            -0.03518855189467458,
+            -0.0056298723325832025,
+            -0.005369475318146738,
+            -0.004650364775945294,
+            -0.020833510091412353,
+            0.020445280915606132,
+            0.0159991245532085,
+            0.013202409908533162,
+            0.023094507005426774,
+            -0.014905666713082246,
+            0.01572052456975675,
+            -0.020945766915121455,
+            0.013114000985348841,
+            0.003349101658930482,
+            -0.019829605346106578,
+            -0.001423750749031516,
+            0.04557459007627153,
+            -0.005847231973329081,
+            0.011986110554869603,
+            0.0012709842140249691,
+            0.026758922819943853,
+            -0.0001635536762342011,
+            0.0065419519929442995,
+            0.005997692702992206,
+            -0.009935128108338126,
+            -0.0008057798606777003,
+            0.00865712443881613,
+            0.046254320605750854,
+            -0.01181132866321511,
+            0.008170688314239364,
+            0.013099051984057827,
+            0.0034971494510865624,
+            -0.0008386400616584293,
+            -0.015346506644390764,
+            -0.03067678541405639,
+            0.019305962963428017,
+            0.007736916644077731,
+            -0.012756525606221703,
+            0.008734673666457216,
+            0.015137771675476126,
+            0.012891824153867332,
+            0.003895886821685762,
+            0.02155154331885366,
+            -0.012761690789748027,
+            0.018528313792219413,
+            0.024663453412919027,
+            0.009077230086690327,
+            9.184526542175535e-05,
+            -0.000751529849753638,
+            0.006791927267880182,
+            -0.0004433479107716552,
+            -0.009832230053513483,
+            0.015858232096630655,
+            -0.014861341148074698,
+            0.017790924034108806,
+            2.7216806238946785e-05,
+            0.03744841066552559,
+            -0.015216207585985075,
+            -0.010138319613387346,
+            -0.007774325714088762,
+            -0.0016997034352120362,
+            0.003971564314593291,
+            0.0018686537276849653,
+            0.006051669054361943,
+            -0.0008319624938748146,
+            -0.017326561560776794,
+            0.01667737942301944,
+            0.005719808546053257,
+            -0.0007405637872110778,
+            0.013750752065593804,
+            0.0038732323799683453,
+            -0.00367645612814479,
+            0.023580906347714314,
+            0.006238541579764762,
+            -0.005235795826199542,
+            0.0028182323816636243,
+            -0.006443210286549713,
+            -0.016255209299456314,
+            -0.003061441395209219,
+            0.004597970789372988,
+            -0.0016590920570710628,
+            -0.011138261849069419,
+            -0.0019423173609729848,
+            0.020355534503272887,
+            -0.004612410336124759,
+            0.013346750536413418,
+            -0.005400060374036387,
+            0.006252154576727367,
+            0.010601459925435238,
+            -0.012346569079682625,
+            0.0014154620309235822,
+            0.010406076751218612,
+            0.0013983417564564663,
+            -0.007140180075225315,
+            0.0010544410919327675,
+            0.007757599672903709,
+            0.006505350216372801,
+            0.00483063937962671,
+            0.00836630436955631,
+            0.013284451514874498,
+            0.0033749897293287438,
+            -0.006243293993633622,
+            0.0013949831763016042,
+            -0.013269921385800992,
+            -0.006308639321247727,
+            0.0010906169976769569,
+            -0.0010152835709869353,
+            -0.02017142197515583,
+            0.010787630901414118,
+            0.005355784839945777,
+            0.007838485470991034,
+            -0.01706932891696025,
+            0.009576030158827565,
+            -0.0008144311254998525,
+            0.02829832828245172,
+            0.007407953684956539,
+            -0.02769403281764041,
+            0.014266001222243134,
+            -0.010492199740754951,
+            0.022575782293320284,
+            -0.0016349316593207844,
+            -0.005656505606143177,
+            0.008025514398322551,
+            -0.009634281579228592,
+            -0.005991026158213869,
+            -0.022366083313133788,
+            -0.025825209755160946,
+            0.016871087849192924,
+            0.013583173973358155,
+            0.021125252303971605,
+            0.00012555627355712715,
+            0.015347535950809997,
+            0.0020128011217859444,
+            0.015164099887550086,
+            0.0024544903332736167,
+            0.012507136277293888,
+            -0.011798050869794805,
+            0.0104828761281283,
+            -0.0137805761114781,
+            0.014620357150314097,
+            -0.005246639228547017,
+            -0.006986408834884282,
+            -0.008849580196616098,
+            -0.005442165099441538,
+            -0.020891551850452786,
+            -0.009155930286949747,
+            -0.0013590151948670868,
+            0.010665872513236615,
+            -0.0005386423407222223,
+            -0.0027671778465775897,
+            -0.03237311596935737,
+            -0.004875062533405715,
+            -0.006670982068843501,
+            -0.01942244110096111,
+            -0.0045975373348006885,
+            -0.026137783023267008,
+            -0.01657173843643047,
+            -0.0007784046498232684,
+            -0.015037306676062236,
+            -0.006889680260149007,
+            -0.02009705080543122,
+            -0.027192867101205106,
+            3.0161964469866572e-05,
+            -0.009647644858034833,
+            -0.005380165957043753,
+            0.013815092649296373,
+            0.011902366689169847,
+            -0.01572592170758115,
+            -0.022321702398924333,
+            -0.01088937403483412,
+            -0.0165754067023749,
+            0.01646435743868707,
+            0.009914938007210463,
+            0.018834863207293908,
+            -0.003724245912768107,
+            0.005643505694722323,
+            0.0009581443359000331,
+            0.02327704258596552,
+            -0.02532422120223976,
+            -0.028694220894404807,
+            0.0019298778426469338,
+            0.0025970175688368323,
+            0.006086164866310174,
+            -0.009414665394188936,
+            0.001200390869257535,
+            -0.007782899559317819,
+            -0.0228264209954853,
+            -0.00898519580978142,
+            0.008156835270668449,
+            -0.010036630164522355,
+            0.02230664068547531,
+            0.012332230358869292,
+            0.013699197379189934,
+            0.010593587938184292,
+            0.01576174456497367,
+            -0.0001654241831584687,
+            0.014471905257729428,
+            -0.008675915624727887,
+            -0.0031070346099204425,
+            0.0038358404385932777,
+            0.005878417037187274,
+            -0.00625022998235275,
+            0.028077114529659795,
+            -0.0011836453716817525,
+            0.0002962695480296182,
+            0.014405782572641941,
+            0.02697667559474498,
+            -0.004710313427099278,
+            0.0210512043261309,
+            -0.002393165025068831,
+            -0.006135816286212752,
+            -0.008429776018235104,
+            0.008607617299702318,
+            -0.009529291136733401,
+            0.011133497451942684,
+            0.019864404836361926,
+            -0.014041315457579166,
+            -0.007351413712676263,
+            0.02950254974910368,
+            -0.014564388091367897,
+            -0.01825317106331622,
+            0.02325964952838705,
+            -0.007414295676409368,
+            -0.0039042587057605484,
+            0.00377144546378926,
+            0.007169856503233239,
+            -0.004036744966949995,
+            -0.012928341573802416,
+            0.016090527447479415,
+            -0.00011159893712484579,
+            0.014146137810890391,
+            -0.0004655839804281017,
+            -0.006372231942983921,
+            -0.010047382825720416,
+            0.002373224247019062,
+            0.026818813126567614,
+            0.013483627627077692,
+            -0.017895710143841333,
+            -0.013785856764479817,
+            0.010826610341182479,
+            -0.01538893818197323,
+            -0.004425206001945227,
+            0.02155595194812746,
+            0.0028768153668128553,
+            -0.020543027467698483,
+            0.008028975884363618,
+            0.0003835165718138906,
+            0.00037046947047892953,
+            -0.00029363296913641716,
+            0.030135739208689217,
+            0.001343516577762509,
+            0.00018948658255495327,
+            -0.00875053830853609,
+            0.008606573235959595,
+            -0.01280305089741873,
+            0.012786478836532904,
+            -0.02437993139337839,
+            0.007648181775166282,
+            -0.00629541989541625,
+            0.004579605315599212,
+            0.016658386927051592,
+            0.01218098627022323,
+            0.015031618813778413,
+            0.008237200519537417,
+            -0.008942965899756683,
+            0.0009093265012901892,
+            -0.025302175262520477,
+            -0.001333639155622843,
+            -0.017971876771137367,
+            -0.02036433247634014,
+            0.0002463842837918767,
+            0.022144712689630985,
+            0.018559482073124734,
+            0.011579840028464503,
+            -0.008657014440886656,
+            0.0166696911919226,
+            -2.9834620942882283e-05,
+            0.011845827764518911,
+            -0.00037901176785495324,
+            -0.006138374154553732,
+            0.004711058953619512,
+            -0.003622694588720417,
+            0.013542405259906128,
+            -0.014730567462694827,
+            0.012076044672004906,
+            0.01690223095263866,
+            -0.009108632851807137,
+            -0.008946251274440141,
+            -0.013992423874475428,
+            0.00048381889145447367,
+            -0.015909599902931202,
+            0.035489252674454214,
+            0.02252772349379229,
+            -0.01210464970923921,
+            -0.010469120693943236,
+            -0.014164999313775756,
+            0.019496048806752753,
+            0.010427539351244087,
+            0.0028018908451425276,
+            0.017091768406027965,
+            0.021395638416767112,
+            0.008817247593404623,
+            -0.006081063711583319,
+            -0.0021216503501144983,
+            -0.003781810594530787,
+            0.020143098218482827,
+            -0.003078172339607037,
+            -0.011350171884507555,
+            -0.012516379645909518,
+            0.005766102930704529,
+            -0.01626433668691071,
+            0.0019294643928566657,
+            0.008641558521764407,
+            -0.015057688533773374,
+            0.019407986696657772,
+            -0.028126197703647868,
+            0.012364374199657305,
+            -0.0044278434183731225,
+            -0.007251147925369614,
+            -0.03496220945653279,
+            0.006389489026974697,
+            0.008941259259632024,
+            0.003941752277454314,
+            0.023003838655784314,
+            -0.024528514847470928,
+            -0.018361931455105486,
+            0.0037791061321962543,
+            0.00047162060848090104,
+            -0.00711894577052754,
+            0.0010138871688747137,
+            0.004128392769213672,
+            -0.00674546985015164,
+            0.0017307097443508704,
+            0.0033354800298095546,
+            -0.002829898077850263,
+            0.013252947102333348,
+            0.0081724269619999,
+            -0.021449979263734438,
+            0.001368791856860285,
+            0.01590656734732547,
+            0.01452008402429858,
+            -0.02391847163018108,
+            0.006331397860689212,
+            -0.02064812934794864,
+            -0.000631340242718475,
+            -0.031272866000888556,
+            0.016859793097564958,
+            -0.0069008624795683276,
+            0.01899718801899956,
+            -0.0015377436942771224,
+            -0.0065810685260109004,
+            -0.012822620655323715,
+            0.02137604937338179,
+            -0.0019121137078344135,
+            -0.0025598772127157477,
+            0.015678293057066646,
+            -0.025700865215693987,
+            0.017825186616033165,
+            -0.0026230848396158204,
+            -0.017342967698019966,
+            0.017199139867162275,
+            0.004536019178633225,
+            0.007224666811309087,
+            -0.023505589665315567,
+            0.0007208041198255396,
+            0.003900164485347562,
+            -0.017250324025622685,
+            0.004470862192961212,
+            -0.030807400122398566,
+            0.0006579147990327597,
+            -0.013222907940559794,
+            -0.0020514994691204847,
+            -0.0046763235724575445,
+            -0.014050035576183385,
+            -0.039181147615786484,
+            -0.005429639279537522,
+            0.016065742005148733,
+            -0.0009029636981539927,
+            0.01800715198737081,
+            -0.002730931639143803,
+            0.002947680833843522,
+            0.004912692811020822,
+            0.00818378920043427,
+            0.006915895602313347,
+            0.02274116906914758,
+            -0.0057854922886970365,
+            0.00471999900678522,
+            0.007058014277137233,
+            0.014638081356210031,
+            -0.01719565951201012,
+            -0.015170377598372723,
+            -0.016015921063369496,
+            -0.006351213642269936,
+            0.01739665344303718,
+            -0.017488885259028475,
+            0.013303332445465686,
+            -0.00429428898520701,
+            0.014825338166443021,
+            -0.008686427208690176,
+            -0.01575945086724703,
+            0.0028564767084934386,
+            -0.007570417028925067,
+            0.001007670647768344,
+            -0.02323908008484089,
+            0.02019729564706863,
+            -0.00847581038203716,
+            0.010508144473197669,
+            0.01729824052022313,
+            -0.021364529179605813,
+            -0.009742851959432839,
+            -0.014342281441152413,
+            0.029896210049029176,
+            0.013925163184699139,
+            -0.009392252140772913,
+            0.003927230832659267,
+            0.008873561269230905,
+            0.011529536139797032,
+            -0.012209070214071445,
+            -0.0027679974125029362,
+            0.002411914349172495,
+            -0.0062056207925691356,
+            -0.002025687569076897,
+            0.03090134105002816,
+            0.0038285457265343245,
+            0.024877474647546086,
+            -0.013214997316123852,
+            -0.0033763842539861907,
+            0.00023379106197363714,
+            0.005134764372494098,
+            0.01183867975321486,
+            -0.0017966997954747142,
+            -0.008029791503028367,
+            -0.01179500476055557,
+            0.02599911330072515,
+            -0.021852083993691557,
+            0.007024505421035273,
+            0.026784024996764876,
+            0.001390202993990949,
+            0.0029956956045771046,
+            -0.013131691460546865,
+            -0.006298736864509989,
+            0.02523640664202355,
+            0.010030824059858138,
+            0.0041523782603032225,
+            -0.026465819482580182,
+            0.0032128899272377593,
+            -0.0061901409332948115,
+            -0.008681758126710213,
+            0.009866461769956222,
+            -0.0028095454857224164,
+            0.01188012041754054,
+            -0.012191996383095982,
+            0.028459939543805705,
+            0.010711502132045176,
+            0.01047154118931583,
+            0.025888652452698974,
+            -0.02079614305315134,
+            0.0042203106489994615,
+            -0.010829796140534066,
+            0.0006773493086316061,
+            0.005596640511307504,
+            -0.027510562413905437,
+            0.001969898603871003,
+            0.010390866563404346,
+            -0.005642706366801436,
+            -0.01882994179544761,
+            0.01187970346127799,
+            -0.019814385265412387,
+            0.024479579275617887,
+            0.003231341626501863,
+            0.020056914356101588,
+            -0.0053276448188067154,
+            0.009834685132275868,
+            0.0002742942778577165,
+            0.0031361331890882127,
+            -0.0008074598092128465,
+            0.009009012303347016,
+            0.005786777410144894,
+            0.013719989127068968,
+            -0.025528370602188686,
+            0.006821572597883499,
+            -0.0029197127345352626,
+            -0.030539872624857384,
+            -0.013383097680839732,
+            0.010291485622655026,
+            0.012218533188117704,
+            -0.009144425807890594,
+            -0.010378679518611277,
+            0.0030042579827305603,
+            0.005748243116238633,
+            -0.006692403885450407,
+            0.006913672151000981,
+            0.01629516912493545,
+            -0.002533517074771728,
+            -0.0019083178803585312,
+            -0.0035007340321378893,
+            -0.01154609935927537,
+            -0.010016998439072824,
+            0.023193275321990937,
+            -0.00482990342915347,
+            0.017943025772239603,
+            0.025873224580389702,
+            0.0039876567755322,
+            -0.0057577932736143506,
+            -0.005460363755206188,
+            0.01478435999841998,
+            0.014195457012279476,
+            0.004669295894868759,
+            -0.009702220679239768,
+            0.007974305230005956,
+            -0.007760086293456636,
+            0.008375077126777163,
+            -0.020260917110786147,
+            0.002199401871514877,
+            0.0026191373298767766,
+            -0.0022089262577237776,
+            -0.015213807046175918,
+            -0.01956788452420682,
+            -0.021898368819136302,
+            0.012634659463647929,
+            0.00034643102239488474,
+            -0.009241772775582378,
+            -0.013645250997151429,
+            0.0016665106990117016,
+            0.0025630500346914592,
+            0.0002095521560180936,
+            -0.007783022628694597,
+            0.009546147813008391,
+            -0.01265287317178631,
+            0.007445223311736028,
+            -0.004007029181436962,
+            0.0051846833906574515,
+            -0.0016916014415925766,
+            -0.016413938859402438,
+            -0.0210868129634149,
+            0.01920794362224444,
+            -0.012897042227841011,
+            0.02269303737601583,
+            -0.0014935878147854765,
+            3.062680855559304e-05,
+            0.0033132626755401546,
+            0.007274487651476732,
+            0.012110042834251237,
+            0.011426424896797977,
+            0.004307086698938411,
+            0.008206932254416481,
+            0.004673427176334148,
+            -0.013124571552764586,
+            0.004324963759928715,
+            0.0039216033705804084,
+            -0.010449887922072297,
+            0.005298323150690876,
+            -0.0010196552573491,
+            0.006128327172621373,
+            0.012167492341907742,
+            0.01923525204293908,
+            0.01908107550907106,
+            -0.0012031208519043073,
+            -0.0010601912677170903,
+            0.014474374858125607,
+            0.004540954805526089,
+            -0.004883537463424679,
+            -0.002772988629590886,
+            -0.013363713078069854,
+            -0.002572403646338979,
+            0.011541515690646252,
+            -0.02376161747414617,
+            -0.018215231053002656,
+            0.021831219299143226,
+            -0.004991882316041097,
+            0.033796174834149546,
+            -0.013140970283091536,
+            -0.016197177724633614,
+            -0.00464190076219332,
+            0.0026389653040644607,
+            -0.013788049032378498,
+            -0.010300859744205445,
+            0.010545541278382187,
+            0.013848738866656793,
+            0.009903590151120531,
+            -0.0005984085581408575,
+            0.02922816576443645,
+            0.0019900932217491907,
+            0.02245395592422119,
+            0.0018978407151245295,
+            -0.007029294098689854,
+            -0.01440575120349887,
+            0.010840472832356266,
+            0.005529335652955532,
+            0.009050748012161991,
+            0.017791269808484802,
+            0.014502057454389247,
+            0.006265567405788131,
+            0.013928705812292101,
+            0.0012903002372833693,
+            0.0010858377120770652,
+            -0.002355229463060164,
+            -0.01692711767136341,
+            -0.01456786846097011,
+            -0.01627640642218236,
+            0.003873864312554098,
+            0.012620350492241247,
+            0.00525890570023481,
+            0.0007393789113866752,
+            -0.007153210751899716,
+            0.009254566395058412,
+            0.0065087881512835375,
+            -0.012279152412209152,
+            -0.016530068283509593,
+            0.004060671955726967,
+            -0.02210570272789274,
+            0.007849467899009437,
+            0.01554233481902349,
+            0.00219885032769439,
+            -0.002655449524219968,
+            0.0006843524304842362,
+            0.006272944169764281,
+            0.005531927368102191,
+            -0.0036710328198476905,
+            -0.00544113514775027,
+            0.011264661763762068,
+            -0.006379654234440906,
+            0.009537150372938296,
+            -0.00290031548037575,
+            0.03718190949929003,
+            -0.011976080887998194,
+            0.012120408392610773,
+            -0.001751472085435711,
+            0.008481703962951994,
+            0.008067669857949423,
+            0.010772687289903202,
+            -0.003805089040741351,
+            -0.0022164543131735167,
+            -0.008753625032781523,
+            0.006751953442322185,
+            -0.01582177621255226,
+            0.013631301238130705,
+            -0.00030685726238985963,
+            -0.010728958775974582,
+            0.01055472059121978,
+            0.006788316621149835,
+            0.004413382316882087,
+            -0.003262024961860459,
+            -0.01972283203505511,
+            0.014937729522976615,
+            0.0018799533869990667,
+            0.019247008681353865,
+            -0.023276106189004046,
+            0.029043727846427652,
+            -0.02019024213343764,
+            0.023083643395685142,
+            0.01865875504942157,
+            0.001179436167200093,
+            -0.02070329911663634,
+            -0.007618131527214554,
+            0.014871709122115308,
+            -0.02659818889735726,
+            -0.006234719259825804,
+            0.005986829551040766,
+            -0.014639912579831785,
+            0.00037813423546252645,
+            -0.01924149909676064,
+            0.00010832498610532006,
+            0.011013130149723431,
+            -0.011782407678373089,
+            0.00438725832178584,
+            0.01608349750921335,
+            0.019347883652009713,
+            -0.009760017612545174,
+            0.003869246694835168,
+            -0.024240324890794748,
+            0.010958789926998691,
+            0.020511684636649225,
+            0.008883830392748497,
+            -0.01761944522796127,
+            0.021817973960311446,
+            0.010652131228027521,
+            -0.012575679152857703,
+            -0.0012753095453828826,
+            -0.028191274775392804,
+            -0.012421170815054556,
+            -0.02188248722244861,
+            -0.0011670175621882146,
+            -0.014436683528976771,
+            -0.0029783880907228383,
+            0.0009506986717516226,
+            -0.004719165363484719,
+            -0.01179163390107033,
+            -0.019543732196676634,
+            0.016606805875498898,
+            0.0227313413004497,
+            -0.006738735201626204,
+            -0.014547240981002825,
+            0.013866974327185123,
+            0.031694077725890536,
+            -0.016189209297529324,
+            -0.00542110554672736,
+            0.0028061798782555557,
+            -0.012964876141500577,
+            0.018946046750284537,
+            0.0039419760434645495,
+            0.000322115192590907,
+            -0.01991851369210666,
+            0.002489725351392778,
+            0.010835458511052207,
+            0.01684964045708685,
+            0.0005541314524621196,
+            0.010855852688653808,
+            -0.007549882748861744,
+            0.010096800951420057,
+            -0.011832260836468588,
+            -3.6179995603517093e-06,
+            -0.007800366796516945,
+            0.007001049318404058,
+            -0.004424776169203575,
+            0.034356203288485945,
+            -0.02817071276766299,
+            0.004989631403681216,
+            -0.014539644354369607,
+            -0.010888763939716889,
+            -0.0027322086509797843,
+            0.0004865534340981866,
+            -0.010775238981581461,
+            -0.005162220412866082,
+            -0.0011263271110752205,
+            0.00913081242511139,
+            0.005844327320514489,
+            -0.013886529519306775,
+            0.019171827605726503,
+            -0.030984919318656618,
+            0.004469458202753611,
+            -0.008553175891755315,
+            0.010511707988936336,
+            -0.0151185694875566,
+            0.013830711686954931,
+            0.01301959345025159,
+            0.008205984085944315,
+            0.007740367849216328,
+            -0.01532123052721834,
+            -0.01678314222392063,
+            0.001376468747213867,
+            0.0036953018347776244,
+            -0.00931855481896886,
+            -0.033391028922085425,
+            -0.007072913446166969,
+            0.005331350534425606,
+            0.011614453225043,
+            0.008156128603003278,
+            -0.02345206873048703,
+            0.005003870024809975,
+            -0.024178720646704597,
+            -0.012115112684614331,
+            -0.007750557665424973,
+            -0.011580152272614108,
+            -0.006117298954192534,
+            0.026449765548206453,
+            0.007989237291215073,
+            0.016691517078382376,
+            0.004954133565782356,
+            0.02742809127725913,
+            0.011670893621991404,
+            -0.011736840457474895,
+            -0.003349151747880166,
+            0.00035419034843196817,
+            0.005507651806218584,
+            -0.008706590052024251,
+            0.011793241980257321,
+            -0.0013295035978611142,
+            -0.0065061682758755625,
+            -0.004398354347597199,
+            0.009372816666628725,
+            0.014188676762501037,
+            -0.003007980559117429,
+            -0.008724255229333351,
+            0.015749776067989826,
+            -0.030899793617096834,
+            -0.015215033379100091,
+            0.0016152308157299517,
+            -0.007813175908301852,
+            0.017806770680528907,
+            0.01169056345888971,
+            -0.026835412610223673,
+            -0.004538235170332766,
+            0.010532316940606419,
+            0.004755930024691789,
+            0.0095408889025506,
+            0.00954782703036282,
+            0.012079064562163901,
+            -0.007829549014214643,
+            0.01046722658021748,
+            -0.011519369840999348,
+            -0.007092961453184232,
+            -0.007241799314569802,
+            -0.008077934089061052,
+            0.021467547268987214,
+            -0.025816030568238717,
+            -0.006558853543772633,
+            0.01134670818701549,
+            0.013027491377727365,
+            -0.002301991242932548,
+            -0.006299073927206214,
+            -0.017357989759336386,
+            0.008311464809363397,
+            -0.013922568511594478,
+            0.025633907424998367,
+            -0.004854655498817219,
+            -0.005336807899100369,
+            -0.005228056880188717,
+            0.01958682721272445,
+            -0.016774409737540904,
+            0.02072573220949266,
+            -0.014449250679704904,
+            -0.007857351393427215,
+            0.001673929749689194,
+            0.006890008607998747,
+            -0.002948835017234416,
+            0.03734128725793149,
+            -0.00556474133167756,
+            -0.0014544201137417158,
+            0.02209084868209573,
+            -0.01883610080898967,
+            0.017901421953179442,
+            0.01563604483998435,
+            0.012327159100148019,
+            0.0018161034827768648,
+            -0.010253034391420751,
+            0.019979728662282904,
+            0.020823925491027635,
+            0.0139490143219451,
+            0.009577521724445478,
+            -0.013478761809038993,
+            0.014934933784421556,
+            -0.0003725422088504494,
+            -0.0024362115214204105,
+            0.012785534912074068,
+            0.016441944143888537,
+            -0.02164522371903646,
+            -0.013152102018736346,
+            0.0063548333449661805,
+            -0.016547629104776748,
+            -0.024875443975589356,
+            -0.008991095813132973,
+            -0.021885127280977718,
+            -0.023336714382596175,
+            0.00688008599534903,
+            0.016482610587564443,
+            0.0035797836729533367,
+            0.013111012464332918,
+            0.006820471349350837,
+            0.018274678506489214,
+            -0.00980248993077149,
+            -0.01582838461385605,
+            -0.0009454586621769173,
+            -0.0004906493597104733,
+            0.009787688665732045,
+            -0.01835583040075359,
+            -0.013977939077760832,
+            0.004340902272034388,
+            -0.020118484730757877,
+            0.01421736173162507,
+            -0.006428386730058143,
+            0.016756376129345597,
+            -0.0004630315831872392,
+            0.011701948921076007,
+            0.02621773278421472,
+            -0.001391567406453007,
+            0.01262026995055899,
+            0.004696906079976175,
+            -0.0030577364576645635,
+            -0.010624066224426196,
+            0.0009420498021845824,
+            -0.0044079288776429516,
+            -0.0004845606666642234,
+            0.006481113327122954,
+            0.020269778435314,
+            -0.03321232649491074,
+            -0.009425498978427855,
+            0.01246754422779043,
+            -0.001658574527323383,
+            -0.005816985985181387,
+            0.013050062223188965,
+            0.00011195028631856881,
+            0.016573827662919836,
+            -0.0017453203044868104,
+            0.01417371165305189,
+            -0.008165910519539772,
+            -0.0007461950113024975,
+            0.0016492309267666983,
+            0.005076015595439873,
+            0.005829299318055953,
+            0.012146800191035666,
+            0.01148884639506138,
+            -0.021051327863689618,
+            -0.0026357096198729156,
+            0.0036146148489310443,
+            -0.010673282438301611,
+            0.014967444393862347,
+            0.005737933867764028,
+            0.01391411712868596,
+            0.0029556189893790237,
+            0.01167806258173173,
+            -0.024440974650111174,
+            -0.01029944007931845,
+            0.01385509328958837,
+            0.00869671728965227,
+            -0.00998315356917989,
+            -0.016580053751364735,
+            -0.007061005072623325,
+            -0.00477175200214453,
+            -0.0021182980051220428,
+            -0.009871524928312118,
+            -0.0005933941177237234,
+            0.006284920058476413,
+            5.466820734762947e-05,
+            -0.012842295882002122,
+            0.015810530885294202,
+            0.004572385791133295,
+            0.003685966629168175,
+            0.005093714133656017,
+            0.011877541830396259,
+            -0.013130170365831912,
+            0.01258190257102377,
+            0.011433121961378686,
+            0.01142665981860032,
+            -0.013710025169815697,
+            -0.001774030746825879,
+            -0.01109881340948764,
+            0.0065684584345163735,
+            0.009983753894796107,
+            0.005346443477816739,
+            -0.002062387245572686,
+            0.004950044847352647,
+            0.006105199826576752,
+            -0.06391185159544624,
+            -0.02131511418723256,
+            0.0027165737784844413,
+            0.033389688791557785,
+            -0.007167593552251769,
+            -0.0037646833460996797,
+            0.002883510089742145,
+            0.01356887065340832,
+            0.009730171586767032,
+            0.00418566245984083,
+            0.020320934773473808,
+            -0.008814826860651366,
+            -0.003014193747732543,
+            0.001559573791453607,
+            0.003499420226345028,
+            -0.008703837291821111,
+            0.0044844701302276645,
+            0.0053963630238108035,
+            0.027764369548110287,
+            0.001469968540767627,
+            0.015793086297019008,
+            -0.008505290826588508,
+            -0.004075888558568453,
+            -0.009166710762668385,
+            -0.002860047211143503,
+            -0.011127820563172684,
+            0.003937110717404271,
+            0.02054186721680431,
+            -0.006126180963243227,
+            0.009346433819831456,
+            0.01057504086030076,
+            0.0001650851447316893,
+            -0.012033423322524951,
+            0.0049587335923558,
+            -0.0032240390628760535,
+            -0.005568976687946761,
+            -0.0024667040429312737,
+            -0.03474027725033016,
+            0.026855947684394643,
+            -0.004369626674503285,
+            0.009252512430957932,
+            -0.005081834388513357,
+            -0.0037896610731020337,
+            -0.014201609270756739,
+            0.017871088029615106,
+            -0.008249444723433136,
+            -0.020092257756971717,
+            0.009532994544485572,
+            -0.004991989149115383,
+            0.006464417738670472,
+            -0.02643078108339741,
+            0.0029138922065812495,
+            -0.017232410749655366,
+            -0.003999293090834831,
+            -0.006924488971625995,
+            0.01318082244178942,
+            -0.007232695119856389,
+            -0.02660711037908311,
+            0.01979251106803667,
+            -0.019485860712345603,
+            -0.011977904020170568,
+            -0.0177530973032872,
+            -0.014524003566945937,
+            0.004874952064727108,
+            -0.015326044325348929,
+            0.002047771741041311,
+            -0.009697340332342246,
+            -0.003652304998155628,
+            -0.0008878698629674222,
+            0.0017782435347417623,
+            -0.013017776298713044,
+            -0.018272697230002275,
+            -0.01353020425912225,
+            -0.00757239449706798,
+            -0.0008250000619957538,
+            0.01299142281772714,
+            -0.001679606555140765,
+            -0.007329941421303076,
+            0.0044605462460153116,
+            0.010231192277666706,
+            0.0006558644644452857,
+            0.0024231733856020414,
+            -0.010728844680699885,
+            -0.014137434553169443,
+            0.005230053181525096,
+            -0.012086468475542818,
+            -0.009459717916037003,
+            -0.0028185227600210556,
+            0.014984527039312847,
+            0.008189464334213754,
+            -0.009139240774931914,
+            -0.005639157579163322,
+            0.014325949891859377,
+            0.009020303519690971,
+            -0.013310205617445609,
+            0.01567574314514385,
+            -0.025369802629268523,
+            0.021964786021408878,
+            0.030536905527039065,
+            0.004945288795196981,
+            -0.01168303908203322,
+            0.02281128628230144,
+            -0.004072286832145419,
+            0.0028278384348551146,
+            0.01740522162288968,
+            -0.042558465579514926,
+            -0.013146562547084278,
+            0.002903743316339479,
+            -0.015297177099602701,
+            -0.010963312697387696,
+            0.008804938803398296,
+            -0.0026727734429658817,
+            0.005273238046953793,
+            0.004257613402172282,
+            0.03512246556126453,
+            0.002258612643297096,
+            -0.0002885708728599208,
+            -0.0019720190285930487,
+            0.012312613129996533,
+            0.010451003939611476,
+            -0.015556620910136185,
+            -0.02387856865043814,
+            -0.004583090523619507,
+            -0.0037828776396525835,
+            -0.0014588238692793338,
+            -0.004361834167147271,
+            -0.008159679676329573,
+            0.012243319920992457,
+            -0.017279964797769668,
+            -0.013748032758693542,
+            0.018394195628121487,
+            -0.006411113522842359,
+            0.019002611495040334,
+            -0.014970362556522149,
+            -0.0077437581301661946,
+            0.0017311115715410025,
+            0.024439691379769795,
+            -0.025013956798520212,
+            0.007359134924497539,
+            -0.0031406470488477937,
+            0.017574824561755627,
+            0.00419409250132525,
+            -0.0046803792187546656,
+            0.010556972371044426,
+            0.015229789583480183,
+            -0.004608293492221483,
+            -0.014449487985966966,
+            -0.01172850291078532,
+            0.016823024209386785,
+            -0.012304887597699588,
+            0.0011773351576246157,
+            0.002262670226244277,
+            -0.0023810578015456716,
+            0.03477119000959688,
+            -0.03442178291363675,
+            0.004062693040714608,
+            0.0029249583507755944,
+            0.003291349443702718,
+            0.006461525089528308,
+            0.006047777261425826,
+            -0.020642802018978404,
+            0.008916088381876748,
+            -0.03316762661930299,
+            -0.006367254689957291,
+            -0.005404276207844267,
+            0.0009189006423526284,
+            0.009401330264376041,
+            -0.011106443887384533,
+            -0.02259405372588916,
+            -0.006010439054323966,
+            0.0070744551115721095,
+            0.010262607009446879,
+            -0.002856616377476421,
+            0.019719451109535182,
+            0.008426509638139982,
+            0.0015877648339862691,
+            -0.010655792508751512,
+            -0.015527851374834499,
+            -0.015460974859638219,
+            -0.017847462632188386,
+            -0.02236964311096428,
+            0.011841749854626384,
+            -0.00576688153869993,
+            0.009054902130717224,
+            0.014317707346504314,
+            -0.015366657284925976,
+            0.002910211008094564,
+            0.017761685988205086,
+            0.007974806171830457,
+            0.002692237107055498,
+            0.0032405477366987947,
+            0.02685230091638845,
+            -0.0009116884421301454,
+            -0.0032228977180251673,
+            0.003158213042682977,
+            0.015872199488167846,
+            0.0004966905858799325,
+            0.019981401695749582,
+            0.015201538351780685,
+            0.005359075814444414,
+            -0.004668738193285921,
+            0.019853988551405383,
+            -0.009844394286458195,
+            0.013476144827747423,
+            -0.00411789677405284,
+            -0.00830656434595337,
+            -0.008496132520586954,
+            0.006442581751999304,
+            -0.008138786054348571,
+            0.006977337946948795,
+            0.02419128791801387,
+            -0.0005734827808485311,
+            -0.0028842142002271044,
+            0.005503033191779872,
+            -0.01696639779775455,
+            -0.00389835761601045,
+            -0.03366077608692946,
+            -0.009942613075976742,
+            -0.01708630689497001,
+            -0.02910756562213045,
+            0.026536268897199546,
+            0.00219827495662445,
+            0.02586503862171508,
+            -0.01905122645609761,
+            0.008935637454859214,
+            -0.01385806889332683,
+            0.029747093931541325,
+            0.009163959012862627,
+            0.0008611776492791918,
+            0.003598831885031635,
+            -0.02103045281939351,
+            -0.01150941825152158,
+            0.027341774725604333,
+            -0.0006700608710615124,
+            -0.012089044078073347,
+            0.022396400579038037,
+            0.017491565611764764,
+            0.0051760367056280536,
+            -0.00559536785318233,
+            0.01979262595019399,
+            0.002306380546194741,
+            0.011973863166872444,
+            -0.0028136750606972223,
+            0.012911235736067303,
+            -0.01510421118359169,
+            -0.003338602157155174,
+            0.039758808572572274,
+            0.007290642519242007,
+            -0.001802921446521437,
+            -0.013696774990705774,
+            0.003941198181491921,
+            -0.0008308293948698925,
+            0.0200351513103389,
+            -0.00014327587644797734,
+            0.028291423013030337,
+            -0.0001085859276231927,
+            -0.010723064585989723,
+            0.020443848393732245,
+            -0.017860296930396913,
+            0.04540901629084378,
+            -0.018671578423261854,
+            0.000852031295629291,
+            -0.010639853151638835,
+            -0.019017194262730474,
+            0.003861296738239369,
+            0.0010977143012920602,
+            0.0008387482016061222,
+            0.0060155816314618,
+            -0.010780239332444652,
+            0.002152214497259421,
+            0.010556338812873054,
+            -0.027636192450621728,
+            0.0010938992111792307,
+            -0.014385202899929138,
+            -0.01353192238925814,
+            -0.026667778226211785,
+            -0.00017403398236549974,
+            -0.015357674131635848,
+            -0.013956722505869091,
+            -0.01985038526848704,
+            -0.008927903879720431,
+            -0.009580214158077466,
+            -0.0020314441852913386,
+            -0.004755584878514854,
+            -0.00018388548107372635,
+            -0.002918855567842974,
+            -0.0015941852262316128,
+            -0.014230272061190995,
+            0.02133992212037431,
+            0.015283345362445234,
+            0.004642765060980717,
+            0.012829570028541471,
+            0.01708952142208596,
+            -0.004770647543823141,
+            0.006604762788847952,
+            -0.0025369836845081078,
+            0.008368000846861623,
+            0.014785031726639733,
+            0.012074738887486398,
+            0.02124518119032885,
+            -0.015459818988572564,
+            -0.004992865135357168,
+            -0.012722391036055767,
+            -0.012210202603282509,
+            -0.017850065946244383,
+            -0.004300429148015694,
+            -0.0014387298144714717,
+            -0.023128300275416423,
+            0.004468952290723408,
+            -0.0026602931688877443,
+            -0.005768852559887139,
+            0.007815212168719318,
+            0.0005862972228454734,
+            -0.002265719947629544,
+            -0.00019323737674204774,
+            -0.02735820445266249,
+            0.00024009122317215594,
+            -0.00018701993387558173,
+            -0.014663710240203917,
+            -0.00450797561624596,
+            0.010584731989847334,
+            -0.011966527748437568,
+            -0.00602367292291343,
+            0.0111155226518017,
+            0.0015565179432852256,
+            0.006062862961835896,
+            0.016952559697728433,
+            0.0005012157272316869,
+            0.011160668242334889,
+            0.014993782560797772,
+            -0.01591303023837098,
+            -0.015356412848444082,
+            -0.0236025426434855,
+            -0.004514538154596937,
+            -0.009753334017065119,
+            0.007553951535816575,
+            -0.004616595532996391,
+            -0.013023153809236325,
+            0.006040285197946957,
+            -0.0006343451398642784,
+            0.008338564073158257,
+            -0.017608246105292328,
+            -8.346168119689692e-05,
+            -0.020944870308705715,
+            0.009230123321316848,
+            -0.00531237224627929,
+            -0.00445985089300067,
+            0.01410675485522911,
+            0.02895945580213926,
+            -0.007799830351886844,
+            -0.009455128322575888,
+            -0.005663072244883114,
+            0.012817229945992764,
+            0.012474431030896143,
+            -0.004423141749110039,
+            -0.0018294512146253298,
+            -0.012817101093394574,
+            0.0004755435377918658,
+            -0.01129485262081729,
+            0.0062747111125005155,
+            0.017199374798326412,
+            -0.021965117839794063,
+            -0.01296097993683748,
+            -0.016765996091978336,
+            -0.016882855472266523,
+            0.004372970343291638,
+            0.017290565252244258,
+            0.022670788364378462,
+            -0.008456547449078897,
+            -0.019564049850942417,
+            0.006898353991539946,
+            -0.01620125883985687,
+            0.0037108638122592746,
+            0.12887455743463638,
+            -0.0028864852421660366,
+            -0.011487804449328495,
+            -0.022683334123299446,
+            -0.005467192611880829,
+            0.01210557696712196,
+            -0.01555159294496261,
+            0.06709672179195036,
+            -0.015071089552631105,
+            0.004364902817496082,
+            -0.004899872946473657,
+            0.0006310685609937571,
+            -0.017610935588191347,
+            -0.005962991715597585,
+            -0.007768819050529025,
+            -0.02688607975562596,
+            0.000883763319535686,
+            -0.009935313912803587,
+            -0.012883104064962595,
+            0.002126368813790577,
+            -0.0013697265269800346,
+            -0.016131186838419288,
+            0.01575514418037863,
+            -0.020677990768697592,
+            -0.007251917191831732,
+            -0.0029325623780992118,
+            -0.00600710212646933,
+            0.014573093764769918,
+            -0.03253336117394778,
+            0.002973725493330224,
+            -0.013601417220439602,
+            0.011905788192118131,
+            0.001542117126832946,
+            -0.010789486980776022,
+            -0.013862139885514524,
+            0.007086571810314664,
+            0.015658913897173155,
+            0.017039624764452546,
+            0.01901374401756777,
+            0.004797954569087795,
+            0.003928999581093576,
+            -0.008046722893614655,
+            0.021974371139905104,
+            -0.00697335955527197,
+            0.0008975473383295958,
+            -0.00970589295353857,
+            -0.004346869073499034,
+            0.0031279458769925416,
+            0.02303983493541673,
+            -0.012560380384977343,
+            -0.01536378830925542,
+            0.0012035987157572355,
+            0.033313215063595594,
+            -0.010845084209743817,
+            0.013587126023659577,
+            0.0029867620644300073,
+            0.020679673435988446,
+            -0.0006640787253300063,
+            0.0025071780490108306,
+            0.022554193713122565,
+            -0.018082906254599396,
+            0.018230000220500718,
+            -0.004451773302171552,
+            0.01915083025790679,
+            -0.004183521043835777,
+            0.015019320101074747,
+            -0.027014055979204272,
+            -0.005362044589023088,
+            -0.012355559881051253,
+            0.003995309003198048,
+            0.020680455787871383,
+            0.0054391755324872375,
+            0.009865676762244514,
+            -0.01645386866657314,
+            -0.01760657627715638,
+            0.022969828924328126,
+            -0.021566687194482292,
+            -0.015826424849757402,
+            0.00012804774676298237,
+            -0.015021970627042762,
+            0.006000610438452896,
+            0.010126179023323158,
+            -0.02912121211344025,
+            -0.02046150879033127,
+            0.004881850039543051,
+            -0.0023469840669773042,
+            -0.00915786433296249,
+            0.006547377770403066,
+            -0.022329063637464074,
+            -0.015702777677743352,
+            -0.005692904224111613,
+            0.01207999389934878,
+            -0.008429160038192635,
+            0.010075061963131637,
+            -0.020106712664936504,
+            0.0036787045650556907,
+            -0.002764965946892891,
+            0.010307702251974629,
+            0.004697169247387421,
+            0.011192285826515207,
+            0.0030113901179382946,
+            0.0014774681536380047,
+            -0.02778270000997313,
+            0.010043261430740039,
+            -5.178406450222847e-05,
+            0.04216636255860709,
+            0.030977096295679525,
+            0.004479857028809617,
+            -5.798020745008712e-05,
+            -0.009996471945562616,
+            0.012476009017187798,
+            -0.019306907391548302,
+            -0.005092322343599383,
+            0.01856463597985508,
+            0.02195467141594795,
+            -0.02236232264139305,
+            0.0052217593371508145,
+            -0.016766955100332014,
+            0.016133299654723637,
+            -0.011208763137897244,
+            0.006248024409576809,
+            0.011601988603146743,
+            0.0022330878390492527,
+            0.010317781892717903,
+            -0.005895142237225119,
+            0.019670196605183973,
+            0.01186677200787666,
+            -0.002339367025714548,
+            0.02367408853110795,
+            0.006858134385159451,
+            0.004479683673962342,
+            0.004798719889930319,
+            -0.009604173975576537,
+            -0.009045602998213713,
+            0.009230969488015385,
+            0.004083616606774546,
+            -0.007944628589259331,
+            0.011836323287097144,
+            0.015100350195476968,
+            0.01885317701149666,
+            -0.0011707847499298092,
+            -0.00684548433712413,
+            -0.006550059418141426,
+            0.001899758393077906,
+            -0.024577381680573596,
+            -0.02855511854789239,
+            0.021238935710272536,
+            -0.010427135883958709,
+            -0.019187769881105637,
+            -0.0015864640188683347,
+            0.016360037306061392,
+            -0.008853972319021563,
+            -0.01627167079143546,
+            0.001270414822216867,
+            0.02255755055961612,
+            -0.005959411125391464,
+            0.006009654905468478,
+            0.005909233773418581,
+            0.005874361519568704,
+            -0.0027859463941992753,
+            -0.024599789577567636,
+            0.0036612374968223453,
+            0.03221346865585554,
+            0.016461563205582293,
+            0.016139782455492073,
+            -0.007158168059808459,
+            0.001838455290310879,
+            0.007530954381521755,
+            0.005438791523412295,
+            0.015674000912615536,
+            0.008604106101542998,
+            -0.006990462920227135,
+            0.00244736628387359,
+            0.012213048255418866,
+            -0.007329299946651028,
+            0.017424711328323798,
+            0.001973882532418776,
+            -0.0133178791530314,
+            -0.008373452013951822,
+            -0.010462150707776556,
+            -0.014575483571750984,
+            0.0027958704799249897,
+            -0.011401916623878957,
+            0.0008016041012669959,
+            0.003953958717263787,
+            0.010260753041492368,
+            0.013961649189650642,
+            -0.0019275791443041316,
+            0.017019168629298462,
+            0.011534594243860177,
+            -0.0017859096307142257,
+            -0.002970739203219076,
+            -0.03142778833370656,
+            0.010207789755749909,
+            -0.014730553506704018,
+            -0.0037611416875830746,
+            0.01825790347981639,
+            -0.007766890847082858,
+            -0.0032778433418881666,
+            -0.021003405429807113,
+            0.0024512381432602736,
+            0.008768616692363321,
+            0.01558713989712387,
+            -0.012819542714293536,
+            0.009201155584074711,
+            -0.0012254337002053017,
+            -0.0057451574939563045,
+            0.00977251392462179,
+            0.0042356831792011935,
+            -0.0033433823868435786,
+            0.011499764125503874
+          ],
+          "title": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer",
+          "keyphrases": [
+            "Sparsely-Gated Mixture-of-Experts Layer",
+            "conditional computation",
+            "neural network",
+            "language modeling",
+            "machine translation"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "7f92225a-9c2d-4745-b7b1-2273bcee2b14",
+        "properties": {
+          "page_content": "Appendices A Load-Balancing Loss\n\nAs discussed in section 4, for load-balancing purposes, we want to define an additional loss function to encourage experts to receive roughly equal numbers of training examples. Unfortunately, the number of examples received by an expert is a discrete quantity, so it can not be used in backpropagation. Instead, we define a smooth estimator Load(X) of the number of examples assigned to each expert for a batch X of inputs. The smoothness allows us to back-propagate gradients through the estimator. This is the purpose of the noise term in the gating function. We define P(x, i) as the probability that G(x)iis nonzero, given a new random choice of noise on element i, but keeping the already-sampled choices of noise on the other elements. To compute P(x, i), we note that the G(x)iis nonzero if and only if H(x)iis greater than the k th-greatest element of H(x) excluding itself. The probability works out to be:\n\n$$P(x,i)=P r\\Big((x\\cdot W_{g}){i}+S t a n d x N o r a l()\\cdot S o f t p u s((x\\cdot W{n o i s e})_{i})$$ $$>k t h_e x c l u d i n(H(x),k,i)\\Big)$$ $$({\\boldsymbol{8}})$$\n\n$$(9)$$ $$(10)$$ (8) Where kth_excluding(v, k, i) means the kth highest component of v, excluding component i. Simplifying, we get:\n\n$$P(x,i)=\\Phi{\\Big(}{\\frac{(x\\cdot W_{g}){i}-k t h_c x c l u d i n g(H(x),k,i)}{S o f t p l u s((x\\cdot W{n o i s e})_{i})}}{\\Big)}$$\n\nWhere Φ is the CDF of the standard normal distribution.\n\n$$L o a d(X){i}=\\sum{x\\in X}P(x,i)$$\n\n$$(11)$$ P(x, i) (10) We can now define the load loss to be the square of the coefficient of variation of the load vector, multiplied by a hand-tuned scaling factor wload.\n\n$$L_{l o a d}(X)=w_{l o a d}\\cdot C V(L o a d(X))^{2}$$ Lload(X) = wload · CV (Load(X))2(11) Initial Load Imbalance: To avoid out-of-memory errors, we need to initialize the network in a state of approximately equal expert load (since the soft constraints need some time to work). To accomplish this, we initialize the matrices Wg and Wnoise to all zeros, which yields no signal and some noise.\n\nExperiments: We trained a set of models with identical architecture (the MoE-256 model described in Appendix C), using different values of wimportance and wload. We trained each model for 10 epochs, then measured perplexity on the test set. We also measured the coefficients of variation in Importance and Load, as well as ratio of the load on the most overloaded expert to the average load. This last value is significant for load balancing purposes on distributed hardware. All of these metrics were averaged over several training batches.\n\nwimportance wload Test Perplexity CV (Importance(X)) CV (Load(X)) max(Load(X)) mean(Load(X)) 0.0 0.0 39.8 3.04 3.01 17.80 0.2 0.0 35.6 0.06 0.17 1.47 0.0 0.2 35.7 0.22 0.04 1.15 0.1 0.1 35.6 0.06 0.05 1.14 0.01 0.01 35.7 0.48 0.11 1.37 1.0 1.0 35.7 0.03 0.02 1.07\n\nTable 6: Experiments with different combinations of losses.\n\n13 Results: Results are reported in Table 6. All the combinations containing at least one the two losses led to very similar model quality, where having no loss was much worse. Models with higher values of wload had lower loads on the most overloaded expert.\n\nB Hierachical Mixture Of Experts\n\nIf the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network.3If the hierarchical MoE consists of a groups of b experts each, we denote the primary gating network by Gprimary, the secondary gating networks by (G1, G2..Ga), and the expert networks by (E0,0, E0,1..Ea,b). The output of the MoE is given by:\n\n$$y_{H}=\\sum_{i=1}^{a}\\sum_{j=1}^{b}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}\\cdot E{i,j}(x)$$ $$(12)$$ $$(13)$$ $$(14)$$\n\nOur metrics of expert utilization change to the following:\n\n$$I m p o r t a n c e_{H}(X){i,j}=\\sum{x\\in X}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}$$ $$L o a d{H}(X){i,j}={\\frac{L o a d{p r i m a r y}(X){i}\\cdot L o a d{i}(X^{(i)})_{j}}{|X^{(i)}|}}$$\n\nLoadprimary and Loadi deonte the Load functions for the primary gating network and i th secondary gating network respectively. X(i) denotes the subset of X for which Gprimary(x)i > 0.\n\nIt would seem simpler to let LoadH(X)i,j = Loadi(Xi)j , but this would not have a gradient with respect to the primary gating network, so we use the formulation above.\n\nC 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS C.1 8-MILLION-OPERATIONS-PER-TIMESTEP MODELS Model Architecture: Our model consists of five layers: a word embedding layer, a recurrent Long Short-Term Memory (LSTM) layer (Hochreiter & Schmidhuber, 1997; Gers et al., 2000), a MoE layer, a second LSTM layer, and a softmax layer. The dimensionality of the embedding layer, the number of units in each LSTM layer, and the input and output dimensionality of the MoE layer are all equal to 512. For every layer other than the softmax, we apply drouput (Zaremba et al., 2014) to the layer output, dropping each activation with probability DropP rob, otherwise dividing by (1 − DropP rob). After dropout, the output of the previous layer is added to the layer output.\n\nThis residual connection encourages gradient flow (He et al., 2015).\n\nMoE Layer Architecture: Each expert in the MoE layer is a feed forward network with one ReLU-activated hidden layer of size 1024 and an output layer of size 512. Thus, each expert contains [512 ∗ 1024] + [1024 ∗ 512] = 1M parameters. The output of the MoE layer is passed through a sigmoid function before dropout. We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts.\n\nWe call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers. Thus, each example is processed by exactly 4 experts for a total of 4M ops/timestep. The two LSTM layers contribute 2M ops/timestep each for the desired total of 8M.\n\n3 We have not found the need for deeper hierarchies.\n\nComputationally-Matched Baselines: The MoE-4 model does not employ sparsity, since all 4 experts are always used. In addition, we trained four more computationally-matched baseline models with no sparsity:\n\nMoE-1-Wide: The MoE layer consists of a single \"expert\" containing one ReLU-activated hidden layer of size 4096.\n\nMoE-1-Deep: The MoE layer consists of a single \"expert\" containing four ReLU-activated hidden layers, each with size 1024.\n\n4xLSTM-512: We replace the MoE layer with two additional 512-unit LSTM layers.\n\nLSTM-2048-512: The model contains one 2048-unit LSTM layer (and no MoE). The output of the LSTM is projected down to 512 dimensions (Sak et al., 2014). The next timestep of the LSTM receives the projected output. This is identical to one of the models published in (Jozefowicz et al., 2016). We re-ran it to account for differences in training regimen, and obtained results very similar to the published ones. Training: The models were trained on a cluster of 16 K40 GPUs using the synchronous method described in Section 3. Each batch consisted of a set of sentences totaling roughly 300,000 words. In the interest of time, we limited training to 10 epochs, (27,000 steps). Training took 12-16 hours for all models, except for MoE-4, which took 18 hours (since all the expert computation was performed on only 4 of 16 GPUs). We used the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 1000 training steps, and decreased after that so as to be proportional to the inverse square root of the step number. The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016). For each model, we performed a hyper-parmeter search to find the best dropout probability, in increments of 0.1.\n\nTo ensure balanced expert utilization we set wimportance = 0.1 and wload = 0.1, as described in Section 4 and Appendix A.\n\nResults: We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol. Results are reported in Table 7. For each model, we report the test perplexity, the computational budget, the parameter counts, the value of DropP rob, and the computational efficiency.\n\nModel Test Test ops/timestep #Params excluding Total Drop- TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params P rob per GPU 10 epochs (final) (millions) (billions) (observed) Kneser-Ney 5-gram* 67.6 0.00001 1.8 LSTM-512-512* 54.1 2.4 2.4 0.8 0.1 LSTM-1024-512* 48.2 4.7 4.7 0.8 0.1 LSTM-2048-512* 45.0 43.7 9.4 9.4 0.8 0.1 0.61 LSTM-2048-512 44.7 9.4 9.4 0.8 0.1 1.21 4xLSTM-512 46.0 8.4 8.4 0.8 0.1 1.07 MoE-1-Wide 46.1 8.4 8.4 0.8 0.1 1.29 MoE-1-Deep 45.7 8.4 8.4 0.8 0.1 1.29 MoE-4 45.0 8.4 8.4 0.8 0.1 0.52 MoE-32 39.7 8.4 37.8 0.9 0.1 0.87 MoE-256 35.7 8.6 272.9 1.1 0.1 0.81 MoE-256-h 36.0 8.4 272.9 1.1 0.1 0.89 MoE-1024-h 34.6 8.5 1079.0 1.9 0.2 0.90 MoE-4096-h 34.1 8.9 4303.4 5.1 0.2 0.74 2xLSTM-8192-1024* 34.7 30.6 151.0 151.0 1.8 0.25 1.09 MoE-34M 31.3 33.8 4313.9 6.0 0.3 1.22 MoE-143M 28.0 142.7 4371.1 6.0 0.4 1.56\n\nC.2 More Expensive Models\n\nWe ran two additional models (MoE-34M and MoE-143M) to investigate the effects of adding more computation in the presence of a large MoE layer. These models have computation budgets of 34M and 143M ops/timestep. Similar to the models above, these models use a MoE layer between two LSTM layers. The dimensionality of the embedding layer, and the input and output dimensionality of the MoE layer are set to 1024 instead of 512. For MoE-34M, the LSTM layers have 1024 units.\n\nFor MoE-143M, the LSTM layers have 4096 units and an output projection of size 1024 (Sak et al., 2014). MoE-34M uses a hierarchical MoE layer with 1024 experts, each with a hidden layer of size 2048. MoE-143M uses a hierarchical MoE layer with 256 experts, each with a hidden layer of size 8192. Both models have 4B parameters in the MoE layers. We searched for the best DropP rob for each model, and trained each model for 10 epochs.\n\nThe two models achieved test perplexity of 31.3 and 28.0 respectively, showing that even in the presence of a large MoE, more computation is still useful. Results are reported at the bottom of Table 7. The larger of the two models has a similar computational budget to the best published model from the literature, and training times are similar. Comparing after 10 epochs, our model has a lower test perplexity by 18%.\n\nD 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS Model Architecture: The models are similar in structure to the 8-million-operations-per-timestep models described in the previous section. We vary the number of experts between models, using an ordinary MoE layer with 32 experts and hierarchical MoE layers with 256, 1024, 4096, 16384, 65536 and 131072 experts. For the hierarchical MoE layers, the first level branching factors are 32, 32, 64, 128, 256 and 256, respectively.\n\nTraining: Models are trained on a cluster of 32 Tesla K40 GPUs, except for the last two models, which are trained on clusters of 64 and 128 GPUs so as to have enough memory for all the parameters. For all models, training batch sizes are approximately 2.5 million words. Models are trained once-through over about 100 billion words. We implement several memory optimizations in order to fit up to 1 billion parameters per GPU. First, we do not store the activations of the hidden layers of the experts, but instead recompute them on the backwards pass. Secondly, we modify the optimizer on the expert parameters to require less auxiliary storage: The Adam optimizer (Kingma & Ba, 2015) keeps first and second moment estimates of the perparameter gradients. This triples the required memory. To avoid keeping a first-moment estimator, we set β1 = 0. To reduce the size of the second moment estimator, we replace it with a factored approximation. For a matrix of parameters, instead of maintaining a full matrix of second-moment estimators, we maintain vectors of row-wise and column-wise averages of that matrix. At each step, the matrix of estimators is taken to be the outer product of those two vectors divided by the mean of either one. This technique could similarly be applied to Adagrad (Duchi et al., 2010).\n\nModel Test Test ops/timestep #Params excluding Total TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params per GPU .1 epochs 1 epoch (millions) (billions) (observed) Kneser-Ney 5-gram 67.1 45.3 0.00001 76.0 4xLSTM-512 54.5 47.0 8.4 8.4 0.1 1.23 MoE-32 48.5 40.4 8.4 37.8 0.1 0.83 MoE-256-h 42.8 35.3 8.4 272.9 0.4 1.11 MoE-1024-h 40.3 32.7 8.5 1079.0 1.2 1.14 MoE-4096-h 38.9 30.9 8.6 4303.4 4.4 1.07 MoE-16384-h 38.2 29.7 8.8 17201.0 17.3 0.96 MoE-65536-h 38.2 28.9 9.2 68791.0 68.9 0.72 MoE-131072-h 39.8 29.2 9.7 137577.6 137.7 0.30\n\nResults: We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8. Perplexity after 100 billion training words is 39% lower for the 68-billion-parameter MoE model than for the baseline model. It is notable that the measured computational efficiency of the largest model (0.30 TFLOPS/GPU) is very low compared to the other models. This is likely a result of the fact that, for purposes of comparison to the other models, we did not increase the training batch size proportionally to the number of GPUs. For comparison, we include results for a computationally matched baseline model consisting of 4 LSTMs, and for an unpruned 5-gram model with Kneser-Ney smoothing (Kneser & Ney, 1995).4\n\nE Machine Translation - Experimental Details\n\nModel Architecture for Single Language Pair MoE Models: Our model is a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decrease the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We insert MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). We use an attention mechanism between the encoder and decoder, with the first decoder LSTM receiving output from and providing input for the attention 5. All of the layers in our model have input and output dimensionality of 512. Our LSTM layers have 2048 hidden units, with a 512-dimensional output projection. We add residual connections around all LSTM and MoE layers to encourage gradient flow (He et al., 2015). Similar to GNMT, to effectively deal with rare words, we used subword units (also known as \"wordpieces\") (Schuster & Nakajima, 2012) for inputs and outputs in our system.\n\nWe use a shared source and target vocabulary of 32K wordpieces. We also used the same beam search technique as proposed in (Wu et al., 2016).\n\nWe train models with different numbers of experts in the MoE layers. In addition to a baseline model with no MoE layers, we train models with flat MoE layers containing 32 experts, and models with hierarchical MoE layers containing 512 and 2048 experts. The flat MoE layers use k = 4 and the hierarchical MoE models use k = 2 at each level of the gating network. Thus, each input is processed by exactly 4 experts in each MoE layer. Each expert in the MoE layer is a feed forward network with one hidden layer of size 2048 and ReLU activation. Thus, each expert contains [512 ∗ 2048] + [2048 ∗ 512] = 2M parameters. The output of the MoE layer is passed through a sigmoid function. We use the strictly-balanced gating function described in Appendix F.\n\nModel Architecture for Multilingual MoE Model: We used the same model architecture as for the single-language-pair models, with the following exceptions: We used noisy-top-k gating as described in Section 2.1, not the scheme from Appendix F. The MoE layers in the encoder and decoder are non-hierarchical MoEs with n = 512 experts, and k = 2. Each expert has a larger hidden layer of size 8192. This doubles the amount of computation in the MoE layers, raising the computational budget of the entire model from 85M to 102M ops/timestep. Training: We trained our networks using the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 2000 training steps, held constant for an additional 8000 steps, and decreased after that so as to be proportional to the inverse square root of the step number. For the single-language-pair models, similarly to (Wu et al., 2016), we applied dropout (Zaremba et al., 2014) to the output of all embedding, LSTM and MoE layers, using DropP rob = 0.4. Training was done synchronously on a cluster of up to 64 GPUs as described in section 3. Each training batch consisted of a set of sentence pairs containing roughly 16000 words per GPU.\n\nTo ensure balanced expert utilization we set wimportance = 0.01 and wload = 0.01, as described in Section 4 and Appendix A. Metrics: We evaluated our models using the perplexity and the standard BLEU score metric. We reported tokenized BLEU score as computed by the multi-bleu.pl script, downloaded from the public implementation of Moses (on Github), which was also used in (Luong et al., 2015a). Results: Tables 2, 3 and 4 in Section 5.3 show comparisons of our results to other published methods. Figure 4 shows test perplexity as a function of number of words in the (training data's) source sentences processed for models with different numbers of experts. As can be seen from the Figure, as we increased the number of experts to approach 2048, the test perplexity of our model continued to improve.\n\nWe found that the experts indeed become highly specialized by syntax and/or semantics, as can be seen in Table 9. For example, one expert is used when the indefinite article \"a\" introduces the direct object in a verb phrase indicating importance or leadership.\n\nExpert 381 Expert 752 Expert 2004 ... with researchers , ... ... plays a core ... ... with rapidly growing ... ... to innovation . ... plays a critical ... ... under static conditions ... ... tics researchers . ... provides a legislative ... ... to swift ly ... ... the generation of ... ... play a leading ... ... to dras tically ... ... technology innovations is ... ... assume a leadership ... ... the rapid and ... ... technological innovations , ... ... plays a central ... ... the fast est ... ... support innovation throughout ... ... taken a leading ... ... the Quick Method ... ... role innovation will ... ... established a reconciliation ... ... rec urrent ) ... ... research scienti st ... ... played a vital ... ... provides quick access ... ... promoting innovation where ... ... have a central ... ... of volatile organic ... ... ... ...\n\nDue to some peculiarities in our infrastructure which have since been fixed, at the time we ran some of the machine translation experiments, our models ran faster if every expert received exactly the same batch size. To accommodate this, we used a different gating function which we describe below. Recall that we define the softmax gating function to be:\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ Gσ(x) = Sof tmax(x · Wg) (15) Sparse Gating (alternate formulation): To obtain a sparse gating vector, we multiply Gσ(x) component-wise with a sparse mask M(Gσ(x)) and normalize the output. The mask itself is a function of Gσ(x) and specifies which experts are assigned to each input example:\n\n$$(15)$$ $$G(x){i}=\\frac{G{\\sigma}(x){i}M(G{\\sigma}(x)){i}}{\\sum{j=1}^{n}G_{\\sigma}(x){j}M(G{\\sigma}(x))_{j}}\\tag{1}$$ $$(17)$$\n\n$$(16)$$\n\nTop-K Mask: To implement top-k gating in this formulation, we would let M(v) = T opK(v, k), where:\n\n$$T o p K(v,k){i}=\\begin{cases}1&{\\mathrm{if~}}v{i}{\\mathrm{~is~in~the~top~}}k{\\mathrm{~elements~of~}}v.\\ 0&{\\mathrm{otherwise.}}\\end{cases}$$ 0 otherwise. (17) Batchwise Mask: To force each expert to receive the exact same number of examples, we introduce an alternative mask function, Mbatchwise(X, m), which operates over batches of input vectors.\n\nInstead of keeping the top k values per example, we keep the top m values per expert across the training batch, where m = k|X| n, so that each example is sent to an average of k experts.\n\n$$M_{b a t c h w i s e}(X,m){j,i}=\\begin{cases}1&\\text{if}X{j,i}\\text{is in the top}m\\text{values for to expert}i\\ 0&\\text{otherwise}\\end{cases}$$\n\nAs our experiments suggest and also observed in (Ioffe & Szegedy, 2015), using a batchwise function during training (such as Mbatchwise) requires modifications to the inference when we may not have a large batch of examples. Our solution to this is to train a vector T of per-expert threshold values to approximate the effects of the batchwise mask. We use the following mask at inference time:\n\n$$(18)$$ $$M_{threshold}(x,T){i}=\\begin{cases}1&\\text{if}x{i}>T_{i}\\ 0&\\text{otherwise}\\end{cases}\\tag{1}$$ $$(19)$$\n\nTo learn the threshold values, we apply an additional loss at training time which is minimized when the batchwise mask and the threshold mask are identical.\n\n$$L_{batchwise}(X,T,m)=\\sum_{j=1}^{|X|}\\sum_{i=1}^{n}(M_{threshold}(x,T){i}-M{batchwise}(X,m){j,i})(X{j,i}-T_{i})\\tag{20}$$\n\nG ATTENTION FUNCTION The attention mechanism described in GNMT (Wu et al., 2016) involves a learned \"Attention Function\" A(xi, yj ) which takes a \"source vector\" xi and a \"target vector\" yj , and must be computed for every source time step i and target time step j. In GNMT, the attention function is implemented as a feed forward neural network with a hidden layer of size n. It can be expressed as:\n\n$$A_{G N M T}(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}t a n h((x_{i}U){d}+(y{j}W)_{d})$$\n\nWhere U and W are trainable weight matrices and V is a trainable weight vector.\n\nFor performance reasons, in our models, we used a slightly different attention function:\n\n$$A(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}tanh((x_{i}U){d})tanh((y{j}W)_{d})\\tag{22}$$\n\nWith our attention function, we can simultaneously compute the attention function on multiple source time steps and multiple target time steps using optimized matrix multiplications. We found little difference in quality between the two functions.\n\n$$(21)$$",
+          "embedding": [
+            0.0011764472583308816,
+            -0.018546752631664276,
+            -0.01668562740087509,
+            -0.003858148353174329,
+            0.009172027930617332,
+            0.015837986022233963,
+            -0.013212142512202263,
+            0.06935913860797882,
+            -0.011065399274230003,
+            0.035029683262109756,
+            0.016455290839076042,
+            0.002998990472406149,
+            0.020509224385023117,
+            -0.022867878898978233,
+            0.0126132657751441,
+            -0.002063821302726865,
+            0.02303372137248516,
+            0.015008772723376751,
+            -0.015994615852832794,
+            -0.019071921706199646,
+            -0.008485622704029083,
+            -0.004452418070286512,
+            -0.029298892244696617,
+            -0.004053934942930937,
+            -0.01113910786807537,
+            0.010531017556786537,
+            0.020085405558347702,
+            0.015220683068037033,
+            -0.0586714930832386,
+            -0.0015317423967644572,
+            -0.002752529690042138,
+            -0.0007854498689994216,
+            -0.015248322859406471,
+            -0.012097310274839401,
+            -0.027824735268950462,
+            -0.03810698911547661,
+            0.047799576073884964,
+            -0.00880348775535822,
+            -0.03169440105557442,
+            0.020896190777420998,
+            0.0326526015996933,
+            0.009411578066647053,
+            -0.009347083978354931,
+            -0.00776697089895606,
+            -0.009227308444678783,
+            0.001612360472790897,
+            0.012263152748346329,
+            0.010208545252680779,
+            -0.01241978257894516,
+            -0.031841814517974854,
+            -0.0024001137353479862,
+            -0.020306527614593506,
+            0.015008772723376751,
+            0.0039019123651087284,
+            -0.015156188048422337,
+            -0.008172363974153996,
+            0.0126132657751441,
+            0.048573512583971024,
+            0.05200092867016792,
+            0.04765216261148453,
+            0.038364965468645096,
+            0.01730293035507202,
+            -0.0017505628056824207,
+            -0.00107452308293432,
+            0.031086310744285583,
+            -0.020195966586470604,
+            -0.006030227988958359,
+            0.006214498076587915,
+            -0.016768548637628555,
+            -0.034440018236637115,
+            -0.06047733873128891,
+            0.02084091119468212,
+            0.007868318818509579,
+            -0.015764279291033745,
+            -0.00650011608377099,
+            -0.028635522350668907,
+            0.01830720156431198,
+            0.07252857834100723,
+            0.013009445741772652,
+            0.005095059052109718,
+            0.017883380874991417,
+            0.010816635563969612,
+            -0.015036413446068764,
+            0.01986428163945675,
+            0.024415744468569756,
+            -0.005647868383675814,
+            -0.032431479543447495,
+            -0.06198835000395775,
+            0.008844949305057526,
+            -0.011083826422691345,
+            -0.039323169738054276,
+            0.03440316393971443,
+            0.0019210122991353273,
+            -0.01265933271497488,
+            -0.0008666437352076173,
+            0.03983912244439125,
+            0.03114159032702446,
+            0.0012944950722157955,
+            -0.011793265119194984,
+            0.022093946114182472,
+            -0.002939102705568075,
+            0.0032569682225584984,
+            0.005164160393178463,
+            0.025171250104904175,
+            0.02139372006058693,
+            -0.005099665839225054,
+            -0.007439891807734966,
+            -0.007467532530426979,
+            0.039212606847286224,
+            0.11468950659036636,
+            -0.019606303423643112,
+            -0.0036969122011214495,
+            -0.03038608469069004,
+            -0.0014557312242686749,
+            -0.023678665980696678,
+            -0.015764279291033745,
+            -0.056386549025774,
+            -0.022204507142305374,
+            0.02006697840988636,
+            0.0007134695188142359,
+            0.030109679326415062,
+            0.051153287291526794,
+            0.031473275274038315,
+            -0.0012956467689946294,
+            -0.02800900489091873,
+            0.06519464403390884,
+            -0.02168855257332325,
+            0.025853049010038376,
+            -0.006477082148194313,
+            -0.04636227339506149,
+            -0.0012069669319316745,
+            0.018279561772942543,
+            -0.013875513337552547,
+            -0.031510129570961,
+            0.014760008081793785,
+            0.006771914195269346,
+            0.02799057774245739,
+            0.05008452385663986,
+            -0.029685860499739647,
+            -0.010733714327216148,
+            -0.0009679921204224229,
+            -0.02004855126142502,
+            -0.003448148025199771,
+            -0.012640906497836113,
+            -0.03899148106575012,
+            0.0012334557250142097,
+            0.015543155372142792,
+            0.07444498687982559,
+            -0.022296641021966934,
+            0.029354173690080643,
+            -0.01696203276515007,
+            -0.016519784927368164,
+            0.01671326719224453,
+            0.018887652084231377,
+            -0.008181577548384666,
+            -0.01625259406864643,
+            -0.022130798548460007,
+            0.009333264082670212,
+            -0.04415103793144226,
+            0.0015858716797083616,
+            -0.00802494864910841,
+            -0.008881802670657635,
+            -0.018344055861234665,
+            0.01060472521930933,
+            0.0071035996079444885,
+            0.033481817692518234,
+            0.04308227077126503,
+            -0.006698206067085266,
+            -0.058413516730070114,
+            0.05148497223854065,
+            0.005809104535728693,
+            0.04463013634085655,
+            0.030072826892137527,
+            0.03891777619719505,
+            -0.0029068556614220142,
+            0.01007034257054329,
+            0.010687646456062794,
+            0.008918656967580318,
+            -0.03639327734708786,
+            0.04595687985420227,
+            -0.023181136697530746,
+            0.0117564108222723,
+            0.016556639224290848,
+            0.04440901428461075,
+            0.03666968271136284,
+            0.01958787627518177,
+            0.009793938137590885,
+            0.019716864451766014,
+            -0.011719557456672192,
+            0.010770567692816257,
+            0.02760361135005951,
+            0.017579335719347,
+            0.04927373677492142,
+            0.017081808298826218,
+            0.00880348775535822,
+            -0.00950832013040781,
+            -0.015091693960130215,
+            0.020214393734931946,
+            -0.010365174151957035,
+            0.0126132657751441,
+            0.026387430727481842,
+            -0.03106788359582424,
+            -0.004744946490973234,
+            -0.014068996533751488,
+            -0.003989440388977528,
+            0.00300820404663682,
+            -0.011876186355948448,
+            -0.016694841906428337,
+            0.03066249005496502,
+            -0.01771753840148449,
+            -0.021854395046830177,
+            -0.0038051707670092583,
+            0.00054647505749017,
+            -0.017653044313192368,
+            -0.0017010403098538518,
+            0.01538652554154396,
+            -0.009517533704638481,
+            0.019035067409276962,
+            0.01853753812611103,
+            -0.0024876417592167854,
+            -0.0067212400026619434,
+            0.03226563706994057,
+            0.012631692923605442,
+            -0.003680788679048419,
+            -0.022978439927101135,
+            -0.010236185975372791,
+            -0.03145484998822212,
+            -0.04444586858153343,
+            0.017643829807639122,
+            -0.027179790660738945,
+            0.017579335719347,
+            0.024968553334474564,
+            -0.011102253571152687,
+            -0.012677759863436222,
+            0.02036180905997753,
+            -0.03563777357339859,
+            0.02331012487411499,
+            0.0023482879623770714,
+            -0.023862935602664948,
+            -0.009600454941391945,
+            -0.024913271889090538,
+            -0.01956944912672043,
+            -0.027437768876552582,
+            -0.002099523553624749,
+            -0.0045491596683859825,
+            -0.0019878100138157606,
+            -0.01796630211174488,
+            -0.015856413170695305,
+            -0.011968321166932583,
+            0.013064726255834103,
+            0.024397317320108414,
+            -0.04297171160578728,
+            0.012576411478221416,
+            -0.0023770800326019526,
+            -0.030883613973855972,
+            0.06814295798540115,
+            0.0442984513938427,
+            0.00950832013040781,
+            0.015469446778297424,
+            -0.047541599720716476,
+            0.03315013274550438,
+            0.024765856564044952,
+            -0.05155868083238602,
+            0.047799576073884964,
+            -0.001612360472790897,
+            -0.005776857025921345,
+            0.052885424345731735,
+            -0.004975283518433571,
+            0.0018519111908972263,
+            0.012475063093006611,
+            0.02846967987716198,
+            -0.014833716675639153,
+            -0.03521395102143288,
+            -0.03162069246172905,
+            0.02769574522972107,
+            -0.017312144860625267,
+            -0.03217350319027901,
+            0.0005597193958237767,
+            0.04440901428461075,
+            0.01614203117787838,
+            0.002754833083599806,
+            0.013654389418661594,
+            0.015174615196883678,
+            0.02120945043861866,
+            0.024986980482935905,
+            0.01589326746761799,
+            -0.014004502445459366,
+            9.407835750607774e-05,
+            -0.05056362599134445,
+            0.008273712359368801,
+            -0.020417090505361557,
+            -0.008563937619328499,
+            -0.014704727567732334,
+            -0.005085845477879047,
+            0.015672143548727036,
+            -0.003065788187086582,
+            -0.008149330504238605,
+            0.011949894018471241,
+            0.03455058112740517,
+            -0.03066249005496502,
+            -0.026590127497911453,
+            -0.008250678889453411,
+            -0.0036001706030219793,
+            0.0102730393409729,
+            -0.025558216497302055,
+            0.0033145523630082607,
+            0.004162193275988102,
+            0.029777994379401207,
+            -0.004226687829941511,
+            -0.0012323040282353759,
+            0.02084091119468212,
+            0.006725846324115992,
+            0.008872589096426964,
+            -0.0073661841452121735,
+            -0.016556639224290848,
+            0.008872589096426964,
+            0.0007664470467716455,
+            0.020877765491604805,
+            -0.02331012487411499,
+            -0.024231474846601486,
+            0.037222493439912796,
+            0.015027199871838093,
+            -0.04470384493470192,
+            0.02091461792588234,
+            -0.014336188323795795,
+            0.010328320786356926,
+            -0.012180231511592865,
+            0.0410921573638916,
+            -0.014907424338161945,
+            -0.024028778076171875,
+            0.003388260258361697,
+            -0.04142384231090546,
+            -0.02463686838746071,
+            -0.027253499254584312,
+            0.006085508968681097,
+            -0.02397349663078785,
+            0.005845958366990089,
+            0.0005499300896190107,
+            0.0009127111989073455,
+            -0.004892362281680107,
+            -0.01694360561668873,
+            0.0160959642380476,
+            0.016833042725920677,
+            -0.009052252396941185,
+            -0.013608322478830814,
+            0.009056858718395233,
+            -0.024913271889090538,
+            0.03388721123337746,
+            0.035306088626384735,
+            0.008370454423129559,
+            0.001825422397814691,
+            0.003524159314110875,
+            0.0060578687116503716,
+            -0.01534045860171318,
+            -0.010678432881832123,
+            -0.03687237948179245,
+            -0.04193979874253273,
+            0.013396412134170532,
+            -0.02837754413485527,
+            0.02817484736442566,
+            0.01318450178951025,
+            0.03751732409000397,
+            -0.020675068721175194,
+            0.0005908149178139865,
+            -0.050121378153562546,
+            -0.01739506609737873,
+            -0.0131292212754488,
+            -0.026295294985175133,
+            0.01566293090581894,
+            -0.031491704285144806,
+            -0.017247650772333145,
+            0.026442712172865868,
+            -0.007485959213227034,
+            -0.013479333370923996,
+            -0.01728450506925583,
+            0.003307642415165901,
+            0.05376992002129555,
+            -0.017588550224900246,
+            -0.025650352239608765,
+            0.02463686838746071,
+            -0.03305799514055252,
+            0.0005672053666785359,
+            0.02301529422402382,
+            0.008762027136981487,
+            0.0011442000977694988,
+            0.03316855803132057,
+            -0.0034619681537151337,
+            -0.018758662045001984,
+            0.00160084362141788,
+            0.027751026675105095,
+            -0.00611314969137311,
+            -0.036540694534778595,
+            -0.0026650014333426952,
+            -0.03226563706994057,
+            0.004079272039234638,
+            0.00807101558893919,
+            0.013110794126987457,
+            -0.019090348854660988,
+            -0.005288542248308659,
+            0.008084836415946484,
+            -0.01021775882691145,
+            -0.0018288774881511927,
+            0.011968321166932583,
+            0.032910581678152084,
+            -0.004533036146312952,
+            6.323476554825902e-05,
+            -0.02714293636381626,
+            0.016446076333522797,
+            0.04153440520167351,
+            -0.017183156684041023,
+            0.011636636219918728,
+            -0.012171017937362194,
+            -0.059371720999479294,
+            -0.0513744130730629,
+            0.04098159819841385,
+            0.014833716675639153,
+            -0.017708323895931244,
+            -0.011645849794149399,
+            -0.004203653894364834,
+            -0.037222493439912796,
+            -0.006730453111231327,
+            0.004611351061612368,
+            0.00632966635748744,
+            0.01519304234534502,
+            -0.012788321822881699,
+            0.028230128809809685,
+            0.017026526853442192,
+            0.018251920118927956,
+            0.008001914247870445,
+            -0.05491239205002785,
+            0.016648773103952408,
+            -0.003132586134597659,
+            0.024618441238999367,
+            -0.019809000194072723,
+            -0.005979553796350956,
+            -0.03237619996070862,
+            -0.034716423600912094,
+            0.022757316008210182,
+            -0.026111025363206863,
+            -0.04289800301194191,
+            -0.008757420815527439,
+            0.0358404703438282,
+            -0.0010618545347824693,
+            0.004005563911050558,
+            -0.01921933703124523,
+            -0.021467428654432297,
+            -0.019164055585861206,
+            -0.010503376834094524,
+            -0.013654389418661594,
+            -0.0235128216445446,
+            -0.06497351825237274,
+            -0.008808095008134842,
+            -0.0442984513938427,
+            -0.01725686341524124,
+            0.029114622622728348,
+            -0.010383601300418377,
+            0.028598668053746223,
+            -0.052111491560935974,
+            -0.00685483543202281,
+            -0.04153440520167351,
+            0.008084836415946484,
+            0.025908328592777252,
+            0.023715518414974213,
+            -0.0075274198316037655,
+            0.04319283366203308,
+            0.012134164571762085,
+            -0.030644062906503677,
+            -0.031215298920869827,
+            -0.00568472221493721,
+            -0.013626749627292156,
+            -0.00014914335042703897,
+            -0.024784283712506294,
+            0.06335194408893585,
+            0.02883821912109852,
+            0.05122699588537216,
+            0.02876451052725315,
+            -0.018076865002512932,
+            -0.02616630680859089,
+            -0.028340689837932587,
+            0.006449441891163588,
+            0.05447014421224594,
+            -0.021559562534093857,
+            -0.003945676609873772,
+            -0.013608322478830814,
+            0.0029874735046178102,
+            -0.012327647767961025,
+            0.0146125927567482,
+            0.03449530154466629,
+            -0.012253939174115658,
+            -0.017266077920794487,
+            0.004417867865413427,
+            0.01947731524705887,
+            -0.020232820883393288,
+            0.03545350208878517,
+            -0.005740003194659948,
+            -0.04488811641931534,
+            -0.009545174427330494,
+            0.020011696964502335,
+            0.028193274512887,
+            0.01910877414047718,
+            -0.0073892176151275635,
+            -0.026037318632006645,
+            0.011959107592701912,
+            0.023346979171037674,
+            0.004510002676397562,
+            -0.022775743156671524,
+            0.013774164952337742,
+            0.017597762867808342,
+            -0.006799554452300072,
+            -0.04956856742501259,
+            0.022886306047439575,
+            -0.02894878014922142,
+            0.02600046433508396,
+            -0.003966406919062138,
+            0.01899821311235428,
+            0.001621573930606246,
+            -0.01853753812611103,
+            0.022996867075562477,
+            -0.006675172131508589,
+            0.00989528652280569,
+            -0.0034919120371341705,
+            -0.012134164571762085,
+            -0.0012472759699448943,
+            0.05664452537894249,
+            0.0038535415660589933,
+            0.0047910138964653015,
+            0.0410921573638916,
+            0.013488546945154667,
+            0.013811019249260426,
+            -0.015460233204066753,
+            -0.039028335362672806,
+            0.025576643645763397,
+            -0.019053494557738304,
+            -0.013811019249260426,
+            -0.026903385296463966,
+            -0.028635522350668907,
+            -0.011415512301027775,
+            0.021320011466741562,
+            -0.027106082066893578,
+            -0.014842930249869823,
+            -0.0005844806437380612,
+            0.0168975368142128,
+            -0.0363011434674263,
+            0.009282589890062809,
+            0.019827427342534065,
+            -0.0022642146795988083,
+            0.02426832914352417,
+            -0.03217350319027901,
+            0.0007566577405668795,
+            0.02139372006058693,
+            0.008656072430312634,
+            -0.0001934832544066012,
+            -0.001268006395548582,
+            0.020896190777420998,
+            -0.02893035300076008,
+            -0.00601640809327364,
+            0.0059426999650895596,
+            0.010549443773925304,
+            -0.009379331022500992,
+            -0.01637236960232258,
+            0.009388544596731663,
+            0.0020960685797035694,
+            -0.01022697240114212,
+            0.010190118104219437,
+            -0.020619787275791168,
+            -0.03895463049411774,
+            0.0053944974206388,
+            0.008011127822101116,
+            -0.018979785963892937,
+            0.0069976444356143475,
+            0.0018887651385739446,
+            -0.0028469678945839405,
+            0.026866532862186432,
+            -0.005458991974592209,
+            0.032247208058834076,
+            0.04249260947108269,
+            0.01611439138650894,
+            0.008301353082060814,
+            0.00578146381303668,
+            -0.0067442734725773335,
+            0.02963057905435562,
+            -0.008292139507830143,
+            0.0016273324145004153,
+            0.026018891483545303,
+            0.010881129652261734,
+            -0.006384947337210178,
+            -0.007025284692645073,
+            -0.018251920118927956,
+            -0.0035287661012262106,
+            -0.014750794507563114,
+            0.017892595380544662,
+            0.041129011660814285,
+            -0.00030836393125355244,
+            -0.022867878898978233,
+            0.013470119796693325,
+            -0.02205709181725979,
+            -0.00439483392983675,
+            0.012309220619499683,
+            0.01847304403781891,
+            -0.039875976741313934,
+            -0.010577084496617317,
+            0.023623384535312653,
+            0.021320011466741562,
+            -0.012585625052452087,
+            -0.02876451052725315,
+            0.014022929593920708,
+            0.009185847826302052,
+            0.0167777631431818,
+            -0.002157107926905155,
+            -0.007485959213227034,
+            0.008038768544793129,
+            0.006380340550094843,
+            -0.007928206585347652,
+            -0.011820905841886997,
+            0.018251920118927956,
+            0.01979057304561138,
+            0.0063204532489180565,
+            -0.015644503757357597,
+            -0.023531248793005943,
+            0.024010350927710533,
+            0.03497440367937088,
+            -0.016888324171304703,
+            0.0194036066532135,
+            0.0036347212735563517,
+            0.011056185699999332,
+            0.00874360091984272,
+            -0.009821578860282898,
+            -0.009968994185328484,
+            -0.0004307305789552629,
+            -0.0037061257753521204,
+            -0.011194388382136822,
+            -0.005302362609654665,
+            -0.016860684379935265,
+            -0.012263152748346329,
+            0.053069692105054855,
+            0.010392814874649048,
+            0.014824503101408482,
+            -0.0009841157589107752,
+            -0.019237764179706573,
+            -0.007559667341411114,
+            0.011784051544964314,
+            0.02647956646978855,
+            0.004072362091392279,
+            0.03162069246172905,
+            -0.005058205220848322,
+            0.013479333370923996,
+            -0.024600014090538025,
+            0.0019382876344025135,
+            0.020011696964502335,
+            0.016409222036600113,
+            -0.0023678664583712816,
+            -0.0027502262964844704,
+            0.004544553346931934,
+            0.020896190777420998,
+            -0.007297082804143429,
+            -0.046583399176597595,
+            0.018316414207220078,
+            -0.015745852142572403,
+            0.02664540894329548,
+            0.04234519228339195,
+            0.01513776183128357,
+            -0.000702528515830636,
+            0.030883613973855972,
+            -0.021559562534093857,
+            0.015856413170695305,
+            -0.02577934041619301,
+            -0.027880016714334488,
+            0.014824503101408482,
+            -0.0067442734725773335,
+            0.013921581208705902,
+            0.01074292790144682,
+            -0.018758662045001984,
+            -0.02559507079422474,
+            0.015128548257052898,
+            0.007707083132117987,
+            -0.029685860499739647,
+            0.008421128615736961,
+            0.025060689076781273,
+            0.0011384416138753295,
+            0.01026382576674223,
+            0.004095395561307669,
+            0.01680540293455124,
+            -0.01189461350440979,
+            -0.007601127959787846,
+            0.029114622622728348,
+            -0.010973264463245869,
+            -0.013230569660663605,
+            0.000723258825019002,
+            -0.04853665828704834,
+            -0.007988094352185726,
+            -0.0020649731159210205,
+            -0.02091461792588234,
+            -0.001933680847287178,
+            0.011857759207487106,
+            0.006200677715241909,
+            0.010180904529988766,
+            0.015294390730559826,
+            0.043708790093660355,
+            -0.0009547477238811553,
+            -0.002020057290792465,
+            -0.0018231190042570233,
+            -0.030736196786165237,
+            -0.01744113303720951,
+            -0.011572141200304031,
+            -0.002694945316761732,
+            0.019348325207829475,
+            -0.0029183723963797092,
+            -0.011341804638504982,
+            -0.02292315848171711,
+            0.016657987609505653,
+            0.029372600838541985,
+            0.056902505457401276,
+            0.0031602266244590282,
+            0.043524518609046936,
+            0.0007048318511806428,
+            0.02828541025519371,
+            0.019274618476629257,
+            0.008660678751766682,
+            -0.046104297041893005,
+            0.01223551295697689,
+            0.022130798548460007,
+            0.03217350319027901,
+            0.01805843785405159,
+            -0.047357331961393356,
+            -0.004242811352014542,
+            0.006366520654410124,
+            0.03735148161649704,
+            0.027179790660738945,
+            0.013967648148536682,
+            -0.0034619681537151337,
+            0.022757316008210182,
+            -0.029593724757432938,
+            -0.021080462262034416,
+            -0.0005450354074127972,
+            -0.011931467801332474,
+            0.00437180045992136,
+            -0.018638886511325836,
+            0.029022488743066788,
+            -0.02111731469631195,
+            -0.0356193445622921,
+            -0.006255958694964647,
+            -0.008702139370143414,
+            -0.001766686444170773,
+            -0.002782473573461175,
+            0.009066072292625904,
+            0.028782937675714493,
+            0.004316519480198622,
+            0.027013948187232018,
+            -0.01118517480790615,
+            0.02559507079422474,
+            0.004715002607554197,
+            -0.00549123901873827,
+            0.008296746760606766,
+            -0.018325628712773323,
+            -0.006610678043216467,
+            -0.009351690299808979,
+            -0.007937420159578323,
+            0.00712202675640583,
+            0.006864049006253481,
+            -0.009254949167370796,
+            -0.010927197523415089,
+            0.019551021978259087,
+            0.0032892152667045593,
+            0.010005848482251167,
+            0.026332149282097816,
+            -0.010475736111402512,
+            -0.0011493826750665903,
+            0.024028778076171875,
+            0.002002781955525279,
+            -0.02349439635872841,
+            0.005924272816628218,
+            0.0013843266060575843,
+            0.037222493439912796,
+            -0.021246304735541344,
+            -0.008338207378983498,
+            0.020951472222805023,
+            0.000956475269049406,
+            -0.0028561814688146114,
+            -0.03913889825344086,
+            0.0062052845023572445,
+            -0.013285850174725056,
+            0.01022697240114212,
+            0.007974274456501007,
+            0.016400009393692017,
+            -0.01389394048601389,
+            0.006186857353895903,
+            0.004413261078298092,
+            0.00858236476778984,
+            -0.0075043863616883755,
+            0.018178213387727737,
+            -0.0030151139944791794,
+            -0.05074789375066757,
+            -0.006656745448708534,
+            -0.014971918426454067,
+            0.020509224385023117,
+            0.012825176119804382,
+            -0.004035507794469595,
+            0.020988326519727707,
+            -0.02531866542994976,
+            -0.030128106474876404,
+            0.03499282896518707,
+            -0.03283687308430672,
+            0.014916637912392616,
+            -0.0005496421945281327,
+            -0.032818444073200226,
+            -0.040834181010723114,
+            -0.018961358815431595,
+            -0.00040280219400301576,
+            -0.0052977558225393295,
+            0.004065451677888632,
+            -0.013718884438276291,
+            -0.017109448090195656,
+            -0.02616630680859089,
+            0.02109888754785061,
+            -0.0020039337687194347,
+            0.013682030141353607,
+            0.002022360684350133,
+            -0.012898883782327175,
+            -0.014962704852223396,
+            -0.01801236905157566,
+            -0.012382928282022476,
+            0.01810450479388237,
+            -0.016427649185061455,
+            -0.014262479729950428,
+            -0.017864953726530075,
+            -0.004896969068795443,
+            -0.012935738079249859,
+            0.0010215454967692494,
+            0.03637485206127167,
+            0.03565619885921478,
+            0.009079893119633198,
+            0.018942931666970253,
+            -0.020232820883393288,
+            0.045993734151124954,
+            -0.023936642333865166,
+            0.034532155841588974,
+            0.0001733287499519065,
+            -0.020859338343143463,
+            -0.005606407765299082,
+            0.024194620549678802,
+            -0.016519784927368164,
+            -0.01357146818190813,
+            0.004461631644517183,
+            -0.002443877747282386,
+            0.023549675941467285,
+            0.02159641683101654,
+            0.01447439007461071,
+            -0.004507699050009251,
+            0.02577934041619301,
+            0.004634384531527758,
+            -0.0006092418916523457,
+            0.0303676575422287,
+            0.02493169903755188,
+            0.008642252534627914,
+            -0.030920466408133507,
+            -0.013811019249260426,
+            -0.01584720052778721,
+            0.0071358466520905495,
+            0.012143378145992756,
+            0.001317528891377151,
+            0.0056985425762832165,
+            -0.014354614540934563,
+            0.004228990990668535,
+            -0.008315172977745533,
+            0.008633038960397243,
+            0.02244405820965767,
+            -0.010171690955758095,
+            -0.03093889355659485,
+            -0.028395971283316612,
+            -0.01851911097764969,
+            -0.003362923162057996,
+            -0.00018139055464416742,
+            -0.01666720025241375,
+            0.020582932978868484,
+            -0.0003740100364666432,
+            0.005620228126645088,
+            -0.014032143168151379,
+            0.0027225858066231012,
+            0.005993374157696962,
+            -0.012447422370314598,
+            0.0015064053004607558,
+            0.0055833738297224045,
+            -0.0015156188746914268,
+            -0.041239574551582336,
+            -0.004615957848727703,
+            0.004438598174601793,
+            0.004832474514842033,
+            -0.03455058112740517,
+            -0.03331597521901131,
+            -0.03270788490772247,
+            0.006767307408154011,
+            -0.003961800131946802,
+            -0.023936642333865166,
+            0.0131292212754488,
+            -0.01156292762607336,
+            -0.007416858337819576,
+            -0.023641811683773994,
+            -0.01151686068624258,
+            0.01938517950475216,
+            0.0014292424311861396,
+            0.01685146987438202,
+            -0.008200004696846008,
+            -0.00261663063429296,
+            -0.0204355176538229,
+            0.006993037648499012,
+            -0.03068091720342636,
+            0.0012023602612316608,
+            -0.009913713671267033,
+            -0.0005608710926026106,
+            -0.026921812444925308,
+            0.0022503945510834455,
+            0.0044109574519097805,
+            0.013746525160968304,
+            -0.03534294292330742,
+            0.0065646106377244,
+            0.011194388382136822,
+            0.016630345955491066,
+            -0.00720955478027463,
+            0.03244990482926369,
+            0.01094562467187643,
+            -0.005482025444507599,
+            0.03132586181163788,
+            0.05682879686355591,
+            0.022480912506580353,
+            -0.03364766016602516,
+            -0.01394922100007534,
+            -0.012198658660054207,
+            0.013811019249260426,
+            -0.014723154716193676,
+            -0.010770567692816257,
+            -0.008771240711212158,
+            0.023457542061805725,
+            -0.005362250376492739,
+            0.0075274198316037655,
+            0.04020766168832779,
+            -0.00880348775535822,
+            0.009591241367161274,
+            0.006417194847017527,
+            -0.012327647767961025,
+            0.00601640809327364,
+            -0.00658764410763979,
+            0.01710023358464241,
+            0.030920466408133507,
+            0.013534614816308022,
+            0.023328552022576332,
+            -0.04205036163330078,
+            0.0204355176538229,
+            0.010503376834094524,
+            0.006960790604352951,
+            0.022407203912734985,
+            0.028801364824175835,
+            0.016086751595139503,
+            -0.007195734418928623,
+            -0.026442712172865868,
+            -0.036337997764348984,
+            0.0020718832965940237,
+            -0.03373979404568672,
+            -0.0034435412380844355,
+            -0.014262479729950428,
+            0.023144282400608063,
+            0.004795620683580637,
+            -0.019458888098597527,
+            -0.013534614816308022,
+            0.006960790604352951,
+            -0.02091461792588234,
+            -0.004961463622748852,
+            -0.009867645800113678,
+            -0.021522708237171173,
+            -0.024102484807372093,
+            0.02120945043861866,
+            -0.009351690299808979,
+            -0.029298892244696617,
+            0.010125624015927315,
+            0.001757472869940102,
+            0.0003642207011580467,
+            -0.0007860257173888385,
+            -0.020509224385023117,
+            -0.03943372890353203,
+            -0.02082248404622078,
+            0.0025935969315469265,
+            0.032910581678152084,
+            -0.026313722133636475,
+            0.03554563969373703,
+            -0.007559667341411114,
+            0.007508993148803711,
+            -0.0023701698519289494,
+            0.009185847826302052,
+            -0.01634472794830799,
+            -0.038549236953258514,
+            0.01785574108362198,
+            -0.0010491859866306186,
+            0.026111025363206863,
+            0.006334273144602776,
+            -0.027474623173475266,
+            -0.0058275312185287476,
+            0.025742486119270325,
+            0.005048991646617651,
+            -0.013737311586737633,
+            -0.007375397253781557,
+            -0.005320789758116007,
+            0.011526074260473251,
+            -0.04157125949859619,
+            -0.003858148353174329,
+            0.06368362903594971,
+            -0.013276636600494385,
+            0.02732720598578453,
+            0.019090348854660988,
+            -0.03283687308430672,
+            -0.04824182391166687,
+            -0.021946528926491737,
+            -0.001739045954309404,
+            -0.008467195555567741,
+            -0.024342035874724388,
+            0.019661584869027138,
+            -0.002150197746232152,
+            -0.001446517650038004,
+            0.005910452920943499,
+            -0.0042727552354335785,
+            -0.026240015402436256,
+            0.0004845719086006284,
+            -0.015275963582098484,
+            -0.0006691296002827585,
+            -0.03466114401817322,
+            -0.02874608337879181,
+            0.02319956384599209,
+            0.014935065060853958,
+            -0.0055833738297224045,
+            -0.0008309414843097329,
+            -0.013995288871228695,
+            -0.0037521931808441877,
+            0.017579335719347,
+            -0.0010486101964488626,
+            0.012014389038085938,
+            0.00955438707023859,
+            0.031768109649419785,
+            0.0035517998039722443,
+            0.003777530277147889,
+            -0.049236882477998734,
+            -0.011673489585518837,
+            0.02760361135005951,
+            0.003952586557716131,
+            -0.017957089468836784,
+            0.0008326690294779837,
+            -0.00437180045992136,
+            -0.009517533704638481,
+            -0.038070134818553925,
+            0.030146533623337746,
+            0.046583399176597595,
+            -0.0022319676354527473,
+            -0.003123372560366988,
+            0.019256191328167915,
+            5.466009952215245e-06,
+            0.010300680063664913,
+            -0.005417531356215477,
+            -0.012336861342191696,
+            0.030902039259672165,
+            0.005905846133828163,
+            -0.00028720172122120857,
+            0.014778435230255127,
+            -0.009402364492416382,
+            -0.0024576981086283922,
+            -0.02561349794268608,
+            -0.0004474300367292017,
+            -0.028340689837932587,
+            0.006601464468985796,
+            0.0006265172269195318,
+            0.006467869039624929,
+            0.010660005733370781,
+            0.029391027987003326,
+            0.03668811172246933,
+            0.016768548637628555,
+            -0.00218590022996068,
+            -0.010917983949184418,
+            0.029980691149830818,
+            0.022794170305132866,
+            0.009922927245497704,
+            0.022812597453594208,
+            -0.031970806419849396,
+            -0.010825849138200283,
+            0.01065079215914011,
+            0.0015294391196221113,
+            -0.0008384274551644921,
+            0.008324386551976204,
+            0.009812365286052227,
+            0.0049061826430261135,
+            -0.005882812198251486,
+            0.016105176880955696,
+            0.007347756996750832,
+            0.014953491277992725,
+            -0.0020557595416903496,
+            -0.004404047504067421,
+            0.009738657623529434,
+            -0.012336861342191696,
+            0.03517710044980049,
+            -0.018712595105171204,
+            -0.014253266155719757,
+            0.031123163178563118,
+            0.10658163577318192,
+            0.0036278110928833485,
+            -0.014741581864655018,
+            -0.02034338191151619,
+            -0.0010428517125546932,
+            0.0235128216445446,
+            0.009922927245497704,
+            0.02972271293401718,
+            0.009204274974763393,
+            0.0026488779112696648,
+            0.02426832914352417,
+            0.029649006202816963,
+            0.0031072490382939577,
+            0.0006656745099462569,
+            -0.024950126186013222,
+            0.008715960197150707,
+            0.002158259740099311,
+            -0.005979553796350956,
+            -0.005380677059292793,
+            0.010365174151957035,
+            0.017551695927977562,
+            0.005431351251900196,
+            -0.004035507794469595,
+            -0.011820905841886997,
+            -0.016353942453861237,
+            0.005458991974592209,
+            -0.01881394349038601,
+            -0.009411578066647053,
+            -0.017984729260206223,
+            -0.009784724563360214,
+            0.00013971391308587044,
+            -0.003088821889832616,
+            -0.005947306752204895,
+            0.013580681756138802,
+            -0.019698437303304672,
+            0.01977214589715004,
+            -0.008522477000951767,
+            -0.00026863077073358,
+            0.0012587928213179111,
+            -0.010051915422081947,
+            0.017772819846868515,
+            0.0021167988888919353,
+            -0.029925409704446793,
+            0.013635963201522827,
+            0.008204611018300056,
+            -0.005772250704467297,
+            0.02052765153348446,
+            -0.03344496339559555,
+            0.019809000194072723,
+            -0.0013981468509882689,
+            -0.015561581589281559,
+            0.0040309010073542595,
+            -0.018408549949526787,
+            0.005997980944812298,
+            0.006831801496446133,
+            0.006361913867294788,
+            -0.0014879783848300576,
+            0.009075285866856575,
+            0.005634048022329807,
+            -0.0016054504085332155,
+            -0.002223905874416232,
+            0.006767307408154011,
+            -0.0011056186631321907,
+            -0.00016728240007068962,
+            -0.014824503101408482,
+            -0.011811692267656326,
+            -0.0002376792108407244,
+            -0.00412073265761137,
+            -0.0037452830001711845,
+            -0.01739506609737873,
+            0.003547193016856909,
+            0.00501674460247159,
+            0.013488546945154667,
+            -0.002655788091942668,
+            -0.010733714327216148,
+            0.0019117988413199782,
+            0.007799217943102121,
+            -0.014216412790119648,
+            -0.005099665839225054,
+            -0.030902039259672165,
+            -0.008646858856081963,
+            0.02196495607495308,
+            0.005882812198251486,
+            -0.025152822956442833,
+            -0.021062035113573074,
+            0.004450114909559488,
+            -0.016446076333522797,
+            0.006725846324115992,
+            0.007163487374782562,
+            -0.013046300038695335,
+            0.005578767042607069,
+            -0.010881129652261734,
+            -0.0031602266244590282,
+            -0.0029644398018717766,
+            -0.00865146517753601,
+            0.000947261753026396,
+            -0.010770567692816257,
+            -0.0020465462002903223,
+            -0.012410569004714489,
+            -0.014437536709010601,
+            -0.05399104207754135,
+            0.008223038166761398,
+            0.01753326877951622,
+            -0.0027686532121151686,
+            -0.024802710860967636,
+            -0.013718884438276291,
+            0.013820232823491096,
+            -0.0038788786623626947,
+            -0.01867574080824852,
+            0.005067418795078993,
+            0.01657506637275219,
+            -0.008780454285442829,
+            -0.006108542904257774,
+            -0.014013716019690037,
+            0.027179790660738945,
+            0.018076865002512932,
+            0.02874608337879181,
+            -0.018795516341924667,
+            0.021909674629569054,
+            0.02284945175051689,
+            -0.026866532862186432,
+            -0.010291466489434242,
+            0.03385035693645477,
+            0.01045730896294117,
+            0.004989103879779577,
+            0.006753487046808004,
+            0.0011989051708951592,
+            -0.0346979983150959,
+            -0.007002251222729683,
+            0.0020373326260596514,
+            -0.031196871772408485,
+            0.022407203912734985,
+            0.014898210763931274,
+            0.006449441891163588,
+            0.01332270447164774,
+            -0.014511244371533394,
+            0.010531017556786537,
+            0.015368098393082619,
+            0.021191023290157318,
+            -0.0071266330778598785,
+            -0.010696860030293465,
+            0.013101580552756786,
+            0.01737663894891739,
+            -0.013672816567122936,
+            0.004544553346931934,
+            0.014750794507563114,
+            -0.018482258543372154,
+            0.03764631226658821,
+            -0.010153263807296753,
+            0.006923936307430267,
+            -0.013460906222462654,
+            -0.019698437303304672,
+            0.03283687308430672,
+            -0.0032362379133701324,
+            0.000820000481326133,
+            -0.024507878348231316,
+            0.009241129271686077,
+            0.005002924241125584,
+            0.019753718748688698,
+            0.014603379182517529,
+            0.02646113932132721,
+            0.023162709549069405,
+            0.004090788774192333,
+            -0.02867237664759159,
+            -0.0013992985477671027,
+            -0.020232820883393288,
+            0.00740303797647357,
+            -0.06250430643558502,
+            -0.026516418904066086,
+            0.017708323895931244,
+            -0.01956944912672043,
+            -0.006094722542911768,
+            -0.008955510333180428,
+            0.0022780350409448147,
+            0.025373946875333786,
+            -0.017127875238656998,
+            0.014962704852223396,
+            6.658185156993568e-05,
+            0.011719557456672192,
+            -0.013202928937971592,
+            0.011093039996922016,
+            -0.014326974749565125,
+            -0.04050249606370926,
+            0.01332270447164774,
+            0.035582493990659714,
+            0.02963057905435562,
+            -0.016565851867198944,
+            0.02360495738685131,
+            -0.01899821311235428,
+            0.013276636600494385,
+            0.005283935461193323,
+            -0.009033825248479843,
+            -0.0034020806197077036,
+            0.0004123036051169038,
+            -0.009545174427330494,
+            -0.013046300038695335,
+            -0.006099329330027103,
+            -0.027179790660738945,
+            0.003427417716011405,
+            -0.034145187586545944,
+            -0.019090348854660988,
+            0.018942931666970253,
+            0.0012230905704200268,
+            -0.009683376178145409,
+            0.027345633134245872,
+            -0.04164496809244156,
+            0.005288542248308659,
+            -0.026424285024404526,
+            0.02657170034945011,
+            -0.025834621861577034,
+            0.008877196349203587,
+            -0.016750121489167213,
+            -0.011480006389319897,
+            0.020656641572713852,
+            -0.0010278797708451748,
+            0.0063434867188334465,
+            -0.011618209071457386,
+            -0.03274473920464516,
+            -0.0045583732426166534,
+            0.02979642152786255,
+            -0.010917983949184418,
+            0.005915059708058834,
+            0.021835967898368835,
+            -0.005283935461193323,
+            -0.01428090687841177,
+            -0.00246000150218606,
+            0.007011464796960354,
+            -0.04186609014868736,
+            0.012769894674420357,
+            0.01899821311235428,
+            0.01204202976077795,
+            0.008642252534627914,
+            0.0056985425762832165,
+            0.013626749627292156,
+            0.006085508968681097,
+            -0.006219104863703251,
+            0.048684071749448776,
+            0.0002988625201396644,
+            -0.00037055497523397207,
+            -0.004710395820438862,
+            0.0043902271427214146,
+            0.0002775563334580511,
+            -0.009102926589548588,
+            0.016427649185061455,
+            0.0021617147140204906,
+            0.00016742636216804385,
+            0.007117419969290495,
+            0.006062475498765707,
+            -0.027953723445534706,
+            0.023254845291376114,
+            -0.029980691149830818,
+            0.00029051280580461025,
+            0.03611687570810318,
+            -0.016105176880955696,
+            0.000605211011134088,
+            0.007200341206043959,
+            -0.0022573047317564487,
+            0.016722481697797775,
+            -0.017616190016269684,
+            -0.024673722684383392,
+            0.02120945043861866,
+            0.022186079993844032,
+            0.013663602992892265,
+            0.03326069191098213,
+            -0.020656641572713852,
+            0.02082248404622078,
+            -0.011940681375563145,
+            -0.009420791640877724,
+            0.004238204564899206,
+            0.0032753951381891966,
+            -0.01591169461607933,
+            0.018270347267389297,
+            -0.017708323895931244,
+            -0.0023909001611173153,
+            -0.004590620752424002,
+            -0.016040682792663574,
+            -0.0009904500329867005,
+            -0.010319107212126255,
+            -0.009370117448270321,
+            -0.0026028105057775974,
+            -0.013921581208705902,
+            0.10665534436702728,
+            0.0005424441187642515,
+            -0.0137557378038764,
+            0.014824503101408482,
+            -0.0033352829050272703,
+            -0.014492817223072052,
+            -0.004192137159407139,
+            0.001406208612024784,
+            -0.00640337448567152,
+            -0.007882139645516872,
+            -0.013608322478830814,
+            -0.0004266996693331748,
+            0.001249579363502562,
+            0.01026382576674223,
+            -0.009139780886471272,
+            -0.029188331216573715,
+            0.009420791640877724,
+            -0.02970428578555584,
+            0.01204202976077795,
+            0.005081238690763712,
+            0.015091693960130215,
+            0.006647531874477863,
+            0.009978207759559155,
+            -0.007177307270467281,
+            -0.013737311586737633,
+            0.031178444623947144,
+            -0.007098992820829153,
+            0.0022791866213083267,
+            -0.0024254508316516876,
+            -0.010835062712430954,
+            -0.012640906497836113,
+            -0.02196495607495308,
+            0.024950126186013222,
+            -0.01616045832633972,
+            0.007181914057582617,
+            0.006338879931718111,
+            0.00218590022996068,
+            0.030164960771799088,
+            0.0008608853095211089,
+            -0.02006697840988636,
+            0.00620989128947258,
+            -0.035766761749982834,
+            -0.005260901991277933,
+            0.019937988370656967,
+            -0.014345401898026466,
+            -0.04079732671380043,
+            -0.008923263289034367,
+            -0.009116746485233307,
+            0.0030335411429405212,
+            0.00068352569360286,
+            -0.014686300419270992,
+            0.003503429004922509,
+            -0.003699215594679117,
+            0.018657313659787178,
+            0.008826522156596184,
+            -0.0077485437504947186,
+            0.002402417128905654,
+            -0.0002808674471452832,
+            0.006454048678278923,
+            -0.013313490897417068,
+            0.005776857025921345,
+            -0.019827427342534065,
+            0.018509898334741592,
+            -0.008144724182784557,
+            -0.0204355176538229,
+            0.005095059052109718,
+            0.014253266155719757,
+            0.006592250894755125,
+            -0.003673878498375416,
+            -0.008200004696846008,
+            0.014068996533751488,
+            0.023678665980696678,
+            0.014124277979135513,
+            0.018657313659787178,
+            0.03781215474009514,
+            0.001152837765403092,
+            0.032247208058834076,
+            -0.02255461923778057,
+            0.018592819571495056,
+            -0.04116586595773697,
+            -0.0009167420794256032,
+            -0.009932140819728374,
+            -0.005638654809445143,
+            -0.0016238773241639137,
+            0.02301529422402382,
+            -0.014013716019690037,
+            0.0029644398018717766,
+            0.015985403209924698,
+            -0.010503376834094524,
+            0.012410569004714489,
+            0.01132337749004364,
+            -0.01828877441585064,
+            0.013479333370923996,
+            -0.0068502286449074745,
+            0.004818654619157314,
+            -0.027916869148612022,
+            0.005832138005644083,
+            -0.004450114909559488,
+            -0.005108879413455725,
+            -0.0015386525774374604,
+            -0.0016803100006654859,
+            -0.008273712359368801,
+            -0.009282589890062809,
+            -0.035011254251003265,
+            -0.005270115565508604,
+            -5.72963799641002e-05,
+            -0.006970004178583622,
+            0.024821138009428978,
+            0.029096195474267006,
+            0.011811692267656326,
+            0.00695157703012228,
+            0.000879312283359468,
+            0.012815962545573711,
+            0.005578767042607069,
+            0.0234022606164217,
+            -0.02579776756465435,
+            0.00412764260545373,
+            -0.007140453439205885,
+            -0.013672816567122936,
+            0.013728098012506962,
+            -0.004323429428040981,
+            -0.013350344263017178,
+            0.027769453823566437,
+            -0.02026967518031597,
+            0.007605734746903181,
+            -0.00029713500407524407,
+            -0.002948316279798746,
+            -0.0031302827410399914,
+            -0.01794787496328354,
+            0.0015881750732660294,
+            -0.00242314743809402,
+            -0.026387430727481842,
+            -0.0030128106009215117,
+            -0.008540904149413109,
+            -0.02244405820965767,
+            0.02310742810368538,
+            0.01703573949635029,
+            -0.01623416692018509,
+            0.0262584425508976,
+            -0.006808768026530743,
+            -0.013700457289814949,
+            -0.01228157989680767,
+            -0.0017805066891014576,
+            -0.004086181987076998,
+            0.009720230475068092,
+            -0.019937988370656967,
+            -0.011461579240858555,
+            0.0015121637843549252,
+            -0.02426832914352417,
+            0.015598435886204243,
+            0.017588550224900246,
+            -0.013433266431093216,
+            0.029483163729310036,
+            -0.0046320813708007336,
+            0.008287533186376095,
+            0.017800459638237953,
+            0.009738657623529434,
+            -0.019735291600227356,
+            -0.022609900683164597,
+            0.012788321822881699,
+            0.0051687671802937984,
+            -0.0065646106377244,
+            0.00027035828679800034,
+            -0.02472900226712227,
+            0.0009691438172012568,
+            0.007711689919233322,
+            0.001163778710179031,
+            0.0027133722323924303,
+            0.0036485414020717144,
+            0.0014119670959189534,
+            0.0021525011397898197,
+            -0.007587307598441839,
+            0.006131576374173164,
+            -0.02826698310673237,
+            0.013414839282631874,
+            0.004325732588768005,
+            0.002100675366818905,
+            -0.03475327789783478,
+            -0.001816208939999342,
+            -0.0011989051708951592,
+            -0.00821843184530735,
+            -0.012106523849070072,
+            0.023899788036942482,
+            0.021264731884002686,
+            0.015036413446068764,
+            0.025871476158499718,
+            -0.010816635563969612,
+            0.013746525160968304,
+            -0.011212815530598164,
+            0.018500683829188347,
+            -0.0041253394447267056,
+            -0.010328320786356926,
+            -0.008292139507830143,
+            0.03777530416846275,
+            -0.010171690955758095,
+            0.01260405220091343,
+            0.0024830352049320936,
+            0.023733945563435555,
+            0.0015098603907972574,
+            0.0005044384743086994,
+            0.002237726002931595,
+            -0.008697533048689365,
+            -0.0059196664951741695,
+            0.007513599935919046,
+            0.06983824074268341,
+            -0.011028545908629894,
+            0.005744609981775284,
+            0.010448095388710499,
+            -0.007637981791049242,
+            -0.006034834776073694,
+            -0.005763037130236626,
+            -0.030422938987612724,
+            0.01685146987438202,
+            -0.0017344391671940684,
+            -0.01920090988278389,
+            0.013580681756138802,
+            0.017505627125501633,
+            0.018030796200037003,
+            0.000927107292227447,
+            0.017726751044392586,
+            -0.005472812335938215,
+            0.0155800087377429,
+            0.02703237533569336,
+            0.009411578066647053,
+            -0.008153936825692654,
+            -0.009545174427330494,
+            0.007016071584075689,
+            0.0010779781732708216,
+            -0.01228157989680767,
+            0.003489608643576503,
+            -0.015229896642267704,
+            0.026866532862186432,
+            -0.006025621201843023,
+            0.04182923585176468,
+            -0.012097310274839401,
+            -0.00623292475938797,
+            -0.0011344107333570719,
+            -0.007260228972882032,
+            0.011885399930179119,
+            0.005629441235214472,
+            0.008748207241296768,
+            -0.0021835968364030123,
+            -0.014262479729950428,
+            0.018040010705590248,
+            0.002409327309578657,
+            -0.00040021090535447,
+            0.004966070409864187,
+            -0.00466893520206213,
+            -0.00474955327808857,
+            0.012622479349374771,
+            0.009913713671267033,
+            -0.008614611811935902,
+            0.005399104207754135,
+            -0.007149667013436556,
+            -0.015561581589281559,
+            -0.01065079215914011,
+            0.002605113899335265,
+            -0.008734387345612049,
+            -0.007370790932327509,
+            0.0038604517467319965,
+            0.020472371950745583,
+            0.007168094161897898,
+            0.012134164571762085,
+            -0.020122257992625237,
+            -0.0006063627079129219,
+            0.017266077920794487,
+            -0.010448095388710499,
+            0.001591630163602531,
+            0.004058541730046272,
+            0.011820905841886997,
+            -0.013147647492587566,
+            0.006200677715241909,
+            0.010144051164388657,
+            0.0037153391167521477,
+            0.01618809998035431,
+            0.006154610309749842,
+            0.003945676609873772,
+            0.004721913021057844,
+            -0.0014246356440708041,
+            0.0003927249344997108,
+            -0.009264162741601467,
+            1.6843408957356587e-05,
+            -0.005025957711040974,
+            -0.0023102823179215193,
+            -0.015451019629836082,
+            0.009738657623529434,
+            0.012217085808515549,
+            0.01680540293455124,
+            -0.026626981794834137,
+            0.011959107592701912,
+            -0.010503376834094524,
+            0.02741934172809124,
+            0.018021583557128906,
+            -0.02618473395705223,
+            0.016105176880955696,
+            -0.01079820841550827,
+            0.015626076608896255,
+            0.0018208156106993556,
+            -0.01675933599472046,
+            0.004528429359197617,
+            -0.004590620752424002,
+            0.004249721299856901,
+            -0.01833484135568142,
+            -0.024305181577801704,
+            0.021651698276400566,
+            0.0011275006690993905,
+            0.0155800087377429,
+            0.005205621011555195,
+            0.013700457289814949,
+            0.0007929358398541808,
+            0.01787416823208332,
+            0.007333936635404825,
+            0.005366857163608074,
+            -0.011397085152566433,
+            0.008909443393349648,
+            -0.0047910138964653015,
+            -0.007969667203724384,
+            -0.006785734090954065,
+            -0.010567870922386646,
+            -0.0005983008886687458,
+            -0.01065079215914011,
+            -0.024986980482935905,
+            -0.018565179780125618,
+            -0.007826858200132847,
+            0.02378922700881958,
+            -0.000312682765070349,
+            -0.0018092987593263388,
+            -0.03547193109989166,
+            -0.001534045790322125,
+            -0.003911125939339399,
+            -0.01997484266757965,
+            -0.003017417388036847,
+            -0.0205460786819458,
+            -0.017570123076438904,
+            -0.00611314969137311,
+            -0.02399192377924919,
+            -0.005035171285271645,
+            -0.018122931942343712,
+            -0.01428090687841177,
+            0.006435621529817581,
+            0.005002924241125584,
+            0.01265011914074421,
+            0.028617095202207565,
+            0.01150764711201191,
+            -0.007347756996750832,
+            -0.014686300419270992,
+            -0.01299101859331131,
+            -0.014197985641658306,
+            0.011258882470428944,
+            0.0096465228125453,
+            0.013995288871228695,
+            -1.3226394912635442e-05,
+            0.010844276286661625,
+            0.00018714898033067584,
+            0.02026967518031597,
+            -0.02924361266195774,
+            -0.03038608469069004,
+            0.0026511813048273325,
+            -0.000183118085260503,
+            -0.004219777882099152,
+            -0.0033053390216082335,
+            -0.005500452592968941,
+            -0.00932405050843954,
+            -0.015165401622653008,
+            -0.006292812526226044,
+            2.4365357603528537e-05,
+            -0.006638318300247192,
+            0.024526305496692657,
+            0.020103830844163895,
+            0.006007194519042969,
+            0.0016423043562099338,
+            0.01881394349038601,
+            0.006804161239415407,
+            0.00805719569325447,
+            -0.00289073190651834,
+            0.0043050022795796394,
+            0.0062283179722726345,
+            0.005131912883371115,
+            -0.004648204892873764,
+            0.02905934303998947,
+            0.0014799166237935424,
+            0.011995961889624596,
+            0.014446749351918697,
+            0.03256046772003174,
+            -0.0013359558070078492,
+            0.025963610038161278,
+            0.012161804363131523,
+            -0.009655735455453396,
+            -0.01611439138650894,
+            0.008301353082060814,
+            -0.008135510608553886,
+            0.006251351907849312,
+            0.01632630079984665,
+            -0.02071192115545273,
+            -0.007200341206043959,
+            0.03394249081611633,
+            -0.01389394048601389,
+            -0.012788321822881699,
+            0.017745178192853928,
+            0.007370790932327509,
+            -0.003231631126254797,
+            -0.0008505201549269259,
+            0.004609047435224056,
+            0.001758624566718936,
+            -0.004152979701757431,
+            0.019348325207829475,
+            0.004309609066694975,
+            0.003989440388977528,
+            0.012530344538390636,
+            -0.0019970235880464315,
+            -0.0076103415340185165,
+            -0.0016745515167713165,
+            0.01662113331258297,
+            0.004222081042826176,
+            -0.017791246995329857,
+            -0.006887082476168871,
+            0.013811019249260426,
+            -0.0041575864888727665,
+            -0.004590620752424002,
+            0.025152822956442833,
+            -0.002501462120562792,
+            -0.012576411478221416,
+            0.0011338349431753159,
+            0.006624497938901186,
+            -0.003643934614956379,
+            0.011986748315393925,
+            0.023439114913344383,
+            -0.003038147697225213,
+            -0.0018116021528840065,
+            -0.009056858718395233,
+            0.007923600263893604,
+            -0.008513263426721096,
+            0.013424052856862545,
+            -0.029372600838541985,
+            0.01002427563071251,
+            -0.012327647767961025,
+            0.005095059052109718,
+            0.012318434193730354,
+            0.01768989861011505,
+            0.0204355176538229,
+            0.013801805675029755,
+            -0.007043711841106415,
+            -0.011737984605133533,
+            -0.012917310930788517,
+            -0.0013843266060575843,
+            -0.009172027930617332,
+            -0.01901664026081562,
+            0.004970677196979523,
+            0.019937988370656967,
+            0.017127875238656998,
+            0.009259555488824844,
+            -0.015027199871838093,
+            0.019993269816040993,
+            -2.9781880584778264e-05,
+            0.007711689919233322,
+            0.0028976420871913433,
+            -0.01280674897134304,
+            -0.0020476977806538343,
+            -0.015828773379325867,
+            0.016400009393692017,
+            -0.014326974749565125,
+            0.01628023386001587,
+            0.030349230393767357,
+            -0.01337798498570919,
+            -0.013967648148536682,
+            -0.014262479729950428,
+            -0.00429118238389492,
+            -0.019035067409276962,
+            0.026442712172865868,
+            0.018076865002512932,
+            -0.016980459913611412,
+            -0.014741581864655018,
+            -0.019937988370656967,
+            0.01920090988278389,
+            0.015045627020299435,
+            0.007108206395059824,
+            0.01921933703124523,
+            0.02026967518031597,
+            0.006182250566780567,
+            0.0036761818919330835,
+            -0.01094562467187643,
+            0.0010889191180467606,
+            0.017339784651994705,
+            0.0016388492658734322,
+            -0.011111467145383358,
+            -0.017044954001903534,
+            0.0039963508024811745,
+            -0.02109888754785061,
+            0.008831128478050232,
+            0.006366520654410124,
+            -0.011480006389319897,
+            0.016040682792663574,
+            -0.024544732645154,
+            0.0073892176151275635,
+            -0.012023602612316608,
+            0.0029736533761024475,
+            -0.036724966019392014,
+            0.0002545226307120174,
+            0.003602473996579647,
+            0.006246745120733976,
+            0.023365406319499016,
+            -0.02990698255598545,
+            -0.01733057200908661,
+            0.003339889459311962,
+            -0.006435621529817581,
+            0.003643934614956379,
+            -0.014916637912392616,
+            0.003385957097634673,
+            -0.00484629487618804,
+            -0.0011177113046869636,
+            0.0036900020204484463,
+            -0.003945676609873772,
+            0.009319443255662918,
+            0.015128548257052898,
+            -0.017026526853442192,
+            -0.00016339545254595578,
+            0.017643829807639122,
+            0.014446749351918697,
+            -0.02255461923778057,
+            0.006688992492854595,
+            -0.013626749627292156,
+            0.006454048678278923,
+            -0.027069229632616043,
+            0.011222029104828835,
+            -0.005302362609654665,
+            0.01481528952717781,
+            -0.0031302827410399914,
+            -0.004351069685071707,
+            -0.0076425885781645775,
+            0.014713941141963005,
+            0.004574496764689684,
+            0.007568880915641785,
+            0.017201581969857216,
+            -0.018297987058758736,
+            0.012364501133561134,
+            -0.007158880587667227,
+            -0.016473717987537384,
+            0.006665959022939205,
+            -0.00675809383392334,
+            0.002443877747282386,
+            -0.02331012487411499,
+            -0.005440564826130867,
+            -0.0011453517945483327,
+            -0.019643157720565796,
+            0.008849555626511574,
+            -0.02999911829829216,
+            0.0001314361725235358,
+            -0.01700809970498085,
+            -0.0031072490382939577,
+            -0.007591914385557175,
+            -0.011655062437057495,
+            -0.02915147691965103,
+            -0.009406971745193005,
+            0.023328552022576332,
+            -0.009729444049298763,
+            0.011774837970733643,
+            0.0009916017297655344,
+            0.007914386689662933,
+            0.00585517194122076,
+            -0.0007952392334118485,
+            0.008522477000951767,
+            0.016704054549336433,
+            -0.014400682412087917,
+            -0.0012645513052120805,
+            0.005353036802262068,
+            0.024600014090538025,
+            -0.019422033801674843,
+            -0.01318450178951025,
+            -0.00865146517753601,
+            0.0019198606023564935,
+            0.017643829807639122,
+            -0.02102518081665039,
+            -0.002324102446436882,
+            -0.001895675202831626,
+            0.005486632231622934,
+            -0.006841015070676804,
+            -0.01441910956054926,
+            -0.009609668515622616,
+            -0.00790056586265564,
+            -0.0009622336947359145,
+            -0.019329898059368134,
+            0.01910877414047718,
+            -0.020490799099206924,
+            0.003613990731537342,
+            0.008821914903819561,
+            -0.010890343226492405,
+            -0.005348430015146732,
+            -0.016676414757966995,
+            0.024120911955833435,
+            0.011885399930179119,
+            -0.002489945152774453,
+            0.005002924241125584,
+            -0.00041950165177695453,
+            0.013221356086432934,
+            -0.006338879931718111,
+            0.003779833670705557,
+            0.0003184411907568574,
+            -0.01150764711201191,
+            -0.005707756150513887,
+            0.023328552022576332,
+            -0.007587307598441839,
+            0.021172596141695976,
+            -0.0018023886950686574,
+            0.005417531356215477,
+            0.004599833860993385,
+            -0.004627474583685398,
+            0.0080479821190238,
+            -0.003604777390137315,
+            -0.008577757515013218,
+            -0.019716864451766014,
+            0.02472900226712227,
+            -0.022573046386241913,
+            -0.0025820801965892315,
+            0.025171250104904175,
+            0.0016941301291808486,
+            -0.001815057243220508,
+            -0.012263152748346329,
+            -0.011949894018471241,
+            0.025337092578411102,
+            0.011378658004105091,
+            0.004256631713360548,
+            -0.014502030797302723,
+            -0.007269442547112703,
+            0.0017148605547845364,
+            -0.013506974093616009,
+            0.01222629938274622,
+            -0.012152590788900852,
+            0.021670125424861908,
+            -0.013875513337552547,
+            0.024231474846601486,
+            0.010641579516232014,
+            0.01586562767624855,
+            0.016031470149755478,
+            -0.019053494557738304,
+            0.0022711248602718115,
+            -0.006421801634132862,
+            0.008181577548384666,
+            0.0009587786626070738,
+            -0.026866532862186432,
+            -0.0029229791834950447,
+            0.009453038685023785,
+            0.0037268560845404863,
+            -0.01805843785405159,
+            0.00776697089895606,
+            -0.015322031453251839,
+            0.019716864451766014,
+            0.007301689591258764,
+            -0.002683428581804037,
+            -0.010917983949184418,
+            0.0028653948102146387,
+            -0.001710253767669201,
+            0.008568543940782547,
+            0.0005744034424424171,
+            0.008738993667066097,
+            0.013230569660663605,
+            0.01890607737004757,
+            -0.02883821912109852,
+            -0.0006230621365830302,
+            -0.005790677387267351,
+            -0.027364060282707214,
+            -0.008305959403514862,
+            0.009471465833485126,
+            0.016704054549336433,
+            -0.0007359273731708527,
+            -0.0004033780423924327,
+            -0.004583710338920355,
+            0.0028193274047225714,
+            0.0007566577405668795,
+            0.0004030901181977242,
+            0.013396412134170532,
+            -0.010954837314784527,
+            0.011102253571152687,
+            0.00021853242651559412,
+            -0.0061361831612885,
+            -0.016169672831892967,
+            0.02588990330696106,
+            -0.006122363265603781,
+            0.019993269816040993,
+            0.02447102591395378,
+            0.012576411478221416,
+            0.004205957520753145,
+            -0.014704727567732334,
+            0.019274618476629257,
+            0.012742254883050919,
+            -0.0029943836852908134,
+            -0.017339784651994705,
+            0.0024669114500284195,
+            0.0042612385004758835,
+            0.0005887994775548577,
+            -0.019993269816040993,
+            0.007016071584075689,
+            0.0008320931810885668,
+            -0.008278319612145424,
+            -0.020030124112963676,
+            -0.016842257231473923,
+            -0.024047205224633217,
+            0.02244405820965767,
+            0.0001376984582748264,
+            -0.014575738459825516,
+            -0.012244726531207561,
+            0.00013928202679380774,
+            0.006444835104048252,
+            -0.006887082476168871,
+            -0.005532699637115002,
+            0.009075285866856575,
+            -0.013442480005323887,
+            -0.00310494564473629,
+            0.0066291047260165215,
+            0.009070679545402527,
+            -0.0072418018244206905,
+            -0.01098247803747654,
+            -0.01785574108362198,
+            0.025742486119270325,
+            -0.009259555488824844,
+            0.01648293063044548,
+            0.0009034976828843355,
+            -0.0011361383367329836,
+            0.00893708411604166,
+            0.0072418018244206905,
+            0.005592587403953075,
+            0.021430574357509613,
+            0.010521803982555866,
+            0.01150764711201191,
+            0.0020384842064231634,
+            -0.022112371399998665,
+            0.014115064404904842,
+            -0.007269442547112703,
+            -0.009218094870448112,
+            -0.006182250566780567,
+            -0.0015225289389491081,
+            0.004470845218747854,
+            0.011046973057091236,
+            0.018979785963892937,
+            0.025355519726872444,
+            -0.006652138661593199,
+            -0.004754160065203905,
+            0.01228157989680767,
+            -0.0004039538907818496,
+            0.008660678751766682,
+            -0.0034435412380844355,
+            -0.002821630798280239,
+            -0.004199047107249498,
+            0.0061960709281265736,
+            -0.019127201288938522,
+            -0.005514272954314947,
+            0.008835735730826855,
+            -0.007297082804143429,
+            0.03232091665267944,
+            -0.0140966372564435,
+            -0.02417619340121746,
+            -0.00136705138720572,
+            0.002978260163217783,
+            -0.017496414482593536,
+            -0.006970004178583622,
+            0.016132818534970284,
+            0.015607649460434914,
+            0.01041124202311039,
+            -0.0027133722323924303,
+            0.024286756291985512,
+            0.005790677387267351,
+            0.025926755741238594,
+            -0.00041460696957074106,
+            -0.004364890046417713,
+            -0.022573046386241913,
+            0.017431920394301414,
+            -0.0003259271616116166,
+            0.01357146818190813,
+            0.027069229632616043,
+            0.008292139507830143,
+            0.006357307080179453,
+            0.00783146545290947,
+            0.012253939174115658,
+            -0.000859157822560519,
+            0.00014316897431854159,
+            -0.01663956046104431,
+            -0.013460906222462654,
+            -0.01568135805428028,
+            -0.002111040521413088,
+            0.010908770374953747,
+            0.011820905841886997,
+            0.004827867727726698,
+            0.0028976420871913433,
+            0.012143378145992756,
+            0.008550116792321205,
+            -0.007955847308039665,
+            -0.024802710860967636,
+            -0.00040740895201452076,
+            -0.026221588253974915,
+            0.009950567036867142,
+            0.007476745638996363,
+            -0.0007117419736459851,
+            0.0013705063611268997,
+            0.0015248323325067759,
+            0.003427417716011405,
+            -0.0005240171449258924,
+            -0.0034135973546653986,
+            -0.004279665183275938,
+            0.011102253571152687,
+            -0.006887082476168871,
+            -0.003547193016856909,
+            -0.00171946722548455,
+            0.03639327734708786,
+            -0.019422033801674843,
+            0.006960790604352951,
+            -0.011286523193120956,
+            0.009904500097036362,
+            0.005081238690763712,
+            0.014704727567732334,
+            -0.004265845287591219,
+            -0.009277982637286186,
+            -0.016022255644202232,
+            0.003584047080948949,
+            -0.016860684379935265,
+            0.014299334026873112,
+            0.0008084836299531162,
+            -0.018804728984832764,
+            0.01742270588874817,
+            -0.003095732070505619,
+            0.007720903493463993,
+            -0.0006224862881936133,
+            -0.01003348920494318,
+            0.013838659971952438,
+            0.008144724182784557,
+            0.03552721068263054,
+            -0.01714630238711834,
+            0.023715518414974213,
+            -0.015220683068037033,
+            0.020011696964502335,
+            0.020030124112963676,
+            -0.0013014052528887987,
+            -0.006504722870886326,
+            0.0013877816963940859,
+            0.013940008357167244,
+            -0.03057035431265831,
+            -0.008319780230522156,
+            0.010208545252680779,
+            -0.015091693960130215,
+            -0.011065399274230003,
+            -0.015837986022233963,
+            0.0012138771126046777,
+            0.00650011608377099,
+            -0.011028545908629894,
+            -0.0007267139153555036,
+            0.004288878757506609,
+            0.01742270588874817,
+            -0.008992364630103111,
+            0.003961800131946802,
+            -0.024194620549678802,
+            0.005085845477879047,
+            0.02283102460205555,
+            0.007011464796960354,
+            -0.006868655793368816,
+            0.019827427342534065,
+            0.012954164296388626,
+            -0.007154273800551891,
+            -0.006278992164880037,
+            -0.02273888885974884,
+            -0.015460233204066753,
+            -0.0240840595215559,
+            -0.0005948458565399051,
+            -0.01744113303720951,
+            -0.003581743687391281,
+            0.00019549870921764523,
+            0.0033721367362886667,
+            -0.0007203796412795782,
+            -0.01997484266757965,
+            0.018159786239266396,
+            0.01552472822368145,
+            0.002715675625950098,
+            -0.012557984329760075,
+            0.015626076608896255,
+            0.0234022606164217,
+            -0.01796630211174488,
+            -0.0004338977159932256,
+            0.0029874735046178102,
+            -0.017984729260206223,
+            0.01313843484967947,
+            0.004966070409864187,
+            -0.001163778710179031,
+            -0.022683609277009964,
+            -0.0006858290289528668,
+            0.009821578860282898,
+            0.015358884818851948,
+            0.004360283259302378,
+            0.005187193863093853,
+            -0.007168094161897898,
+            -0.000966840423643589,
+            -0.00818618480116129,
+            -5.551486538024619e-06,
+            -0.013553041033446789,
+            0.007025284692645073,
+            -0.008844949305057526,
+            0.022130798548460007,
+            -0.01958787627518177,
+            0.004318822640925646,
+            -0.02600046433508396,
+            -0.0011966017773374915,
+            0.003358316607773304,
+            0.009913713671267033,
+            -0.009738657623529434,
+            -0.009655735455453396,
+            -0.003991744015365839,
+            0.004171406850218773,
+            0.011599781922996044,
+            -0.009148993529379368,
+            0.010319107212126255,
+            -0.02915147691965103,
+            0.005025957711040974,
+            -0.0040078675374388695,
+            0.008762027136981487,
+            -0.023955069482326508,
+            0.009453038685023785,
+            -0.0010797056602314115,
+            -3.174334779032506e-05,
+            0.008932476863265038,
+            -0.01332270447164774,
+            -0.023770799860358238,
+            0.0043994407169520855,
+            -0.0016849166713654995,
+            -0.013589895330369473,
+            -0.03777530416846275,
+            -0.010871916078031063,
+            0.0010428517125546932,
+            0.0075043863616883755,
+            0.01170113030821085,
+            -0.028709229081869125,
+            -0.0003699791559483856,
+            -0.026332149282097816,
+            -0.006288205739110708,
+            -0.0187218077480793,
+            -0.011461579240858555,
+            0.0037452830001711845,
+            0.01931147091090679,
+            0.004984497092664242,
+            0.011922254227101803,
+            0.009959780611097813,
+            0.03009125404059887,
+            0.008444162085652351,
+            -0.0043994407169520855,
+            0.004109215922653675,
+            -0.005744609981775284,
+            -0.004947643261402845,
+            -0.013645176775753498,
+            0.003895002184435725,
+            -0.005145733244717121,
+            -0.0016238773241639137,
+            -0.003293822053819895,
+            0.01074292790144682,
+            0.011083826422691345,
+            0.00013122022210154682,
+            -0.01441910956054926,
+            0.019698437303304672,
+            -0.014225626364350319,
+            -0.010577084496617317,
+            -0.0021340742241591215,
+            -0.013820232823491096,
+            0.015110121108591557,
+            0.015469446778297424,
+            -0.026055745780467987,
+            -0.005730789620429277,
+            0.013543828390538692,
+            0.01279753539711237,
+            0.0054129245691001415,
+            0.009287196211516857,
+            0.01204202976077795,
+            -0.014068996533751488,
+            -0.0045053958892822266,
+            -0.01537731196731329,
+            -0.006527756340801716,
+            -0.0034435412380844355,
+            -0.0004523246898315847,
+            0.013838659971952438,
+            -0.02102518081665039,
+            -0.004643598105758429,
+            0.003116462379693985,
+            0.008296746760606766,
+            -0.0031878668814897537,
+            -0.0023379225749522448,
+            -0.02025124803185463,
+            0.007591914385557175,
+            -0.012640906497836113,
+            0.016906751319766045,
+            -0.0007146212155930698,
+            -0.0019267707830294967,
+            -0.01189461350440979,
+            0.013672816567122936,
+            -0.021154168993234634,
+            0.030035972595214844,
+            -0.011802478693425655,
+            0.002077641664072871,
+            -0.002655788091942668,
+            -0.001952107879333198,
+            0.00790056586265564,
+            0.03256046772003174,
+            -0.004786407109349966,
+            -0.004164496902376413,
+            0.015497087500989437,
+            -0.023549675941467285,
+            0.02541080117225647,
+            0.007974274456501007,
+            0.013433266431093216,
+            0.0006869807257317007,
+            -0.016013043001294136,
+            0.018942931666970253,
+            0.017883380874991417,
+            0.005288542248308659,
+            0.011046973057091236,
+            -0.01570899784564972,
+            0.016105176880955696,
+            0.007158880587667227,
+            -0.001377416541799903,
+            0.011903827078640461,
+            0.019532594829797745,
+            -0.02848810702562332,
+            -0.016842257231473923,
+            0.014391468837857246,
+            -0.018758662045001984,
+            -0.01995641551911831,
+            -0.012576411478221416,
+            -0.011968321166932583,
+            -0.028635522350668907,
+            0.012567197903990746,
+            0.015607649460434914,
+            0.0002802915987558663,
+            0.015229896642267704,
+            0.0030035972595214844,
+            0.017579335719347,
+            -0.01685146987438202,
+            -0.012244726531207561,
+            -0.0032085974235087633,
+            0.008301353082060814,
+            0.006771914195269346,
+            -0.015220683068037033,
+            -0.00785449892282486,
+            0.00334219285286963,
+            -0.027198217809200287,
+            0.008333600126206875,
+            -0.009664949029684067,
+            0.01910877414047718,
+            0.003918035887181759,
+            0.00503977807238698,
+            0.012742254883050919,
+            0.018620459362864494,
+            0.018500683829188347,
+            0.0050443848595023155,
+            -0.0021363776177167892,
+            0.004026294220238924,
+            -0.0035402828361839056,
+            -0.00210873712785542,
+            0.004005563911050558,
+            0.009236522018909454,
+            0.023918215185403824,
+            -0.0328737273812294,
+            -0.0026903385296463966,
+            0.0072418018244206905,
+            -0.0013992985477671027,
+            -0.0042842719703912735,
+            0.018417762592434883,
+            -0.002002781955525279,
+            0.010448095388710499,
+            0.005177980288863182,
+            0.012880456633865833,
+            -0.009231915697455406,
+            0.003026630962267518,
+            0.015589222311973572,
+            0.014769221656024456,
+            0.014548097737133503,
+            0.011802478693425655,
+            0.0009990877006202936,
+            -0.017597762867808342,
+            0.0075043863616883755,
+            0.004067755304276943,
+            -0.009245735593140125,
+            0.004666632041335106,
+            0.010899556800723076,
+            0.014824503101408482,
+            0.004139159806072712,
+            0.010964050889015198,
+            -0.01890607737004757,
+            -0.009010791778564453,
+            0.01429012045264244,
+            0.005970340687781572,
+            -0.0102730393409729,
+            -0.009775510989129543,
+            -0.004189833998680115,
+            0.0031072490382939577,
+            -0.0028354511596262455,
+            -0.009029218927025795,
+            0.005652475170791149,
+            0.002283793408423662,
+            -0.003855844959616661,
+            -0.015810346230864525,
+            0.014981132000684738,
+            0.004643598105758429,
+            0.006186857353895903,
+            0.007803824730217457,
+            0.007684049196541309,
+            -0.007536633405834436,
+            0.018546752631664276,
+            0.0038166875019669533,
+            0.004613654222339392,
+            -0.012447422370314598,
+            -0.00558798061683774,
+            -0.011572141200304031,
+            0.011747198179364204,
+            0.006647531874477863,
+            0.008964723907411098,
+            0.0024047205224633217,
+            -0.001717163948342204,
+            0.007688655983656645,
+            -0.07857263088226318,
+            -0.014713941141963005,
+            0.006583037320524454,
+            0.03526923432946205,
+            -0.0012610962148755789,
+            -0.005118092987686396,
+            0.0045146094635128975,
+            0.009406971745193005,
+            -0.004072362091392279,
+            -0.0013808716321364045,
+            0.02045394480228424,
+            -0.009305623359978199,
+            -0.013442480005323887,
+            -0.002022360684350133,
+            -0.000278132181847468,
+            -0.00224578776396811,
+            0.002814720617607236,
+            0.0028746083844453096,
+            0.028156420215964317,
+            7.521949737565592e-05,
+            0.017220009118318558,
+            -0.006675172131508589,
+            -0.004781800322234631,
+            -0.007559667341411114,
+            -0.008600790984928608,
+            -0.016261806711554527,
+            0.003406687406823039,
+            0.019845854490995407,
+            -0.011240456253290176,
+            0.017026526853442192,
+            0.0056893290020525455,
+            0.0036830920726060867,
+            -0.0019106471445411444,
+            -0.008863375522196293,
+            -0.011793265119194984,
+            -0.005090452264994383,
+            -0.009379331022500992,
+            -0.031104736030101776,
+            0.02073034830391407,
+            -0.002413933863863349,
+            0.010475736111402512,
+            -0.015358884818851948,
+            0.0033928670454770327,
+            -0.024010350927710533,
+            0.025392374023795128,
+            -0.019827427342534065,
+            -0.019182482734322548,
+            0.0067212400026619434,
+            -0.0019866584334522486,
+            0.006730453111231327,
+            -0.01839933544397354,
+            -0.0006184554076753557,
+            -0.02216765284538269,
+            -0.009245735593140125,
+            -0.0126132657751441,
+            0.012862029485404491,
+            -0.013488546945154667,
+            -0.01968001201748848,
+            0.01968001201748848,
+            -0.014594165608286858,
+            -0.015257536433637142,
+            -0.025816194713115692,
+            -0.016123604029417038,
+            0.0055603403598070145,
+            -0.008421128615736961,
+            0.0023183440789580345,
+            -0.008794275112450123,
+            0.005643261596560478,
+            -0.0036692717112600803,
+            -0.002842361107468605,
+            -0.004910789430141449,
+            -0.018482258543372154,
+            -0.0075274198316037655,
+            -0.00802494864910841,
+            -0.008550116792321205,
+            0.010687646456062794,
+            -0.005196407437324524,
+            -0.003959496505558491,
+            -0.004166800063103437,
+            0.010245399549603462,
+            0.002411630703136325,
+            -0.004613654222339392,
+            -0.009411578066647053,
+            -0.012438209727406502,
+            0.006656745448708534,
+            -0.011111467145383358,
+            -0.015699783340096474,
+            -0.018159786239266396,
+            0.015874840319156647,
+            0.016178885474801064,
+            -0.008001914247870445,
+            -0.005426744464784861,
+            0.008808095008134842,
+            0.0013532311422750354,
+            -0.029649006202816963,
+            0.0215779896825552,
+            -0.02073034830391407,
+            0.019661584869027138,
+            0.021338438615202904,
+            0.0036001706030219793,
+            -0.002823934191837907,
+            0.03268945589661598,
+            -0.008172363974153996,
+            0.0012380625121295452,
+            0.013046300038695335,
+            -0.03620900958776474,
+            -0.011608995497226715,
+            -0.0006720087840221822,
+            -0.027087656781077385,
+            -0.015598435886204243,
+            0.018095292150974274,
+            0.0006242138333618641,
+            0.004360283259302378,
+            0.003075001761317253,
+            0.03136271610856056,
+            0.0036462380085140467,
+            -0.006219104863703251,
+            0.005353036802262068,
+            0.00365314818918705,
+            0.010319107212126255,
+            -0.02120945043861866,
+            -0.02972271293401718,
+            -0.007315509952604771,
+            0.0019290741765871644,
+            -0.0026926419232040644,
+            -0.011848545633256435,
+            -0.0004975283518433571,
+            0.008338207378983498,
+            -0.022038664668798447,
+            -0.012539558112621307,
+            0.02474742941558361,
+            -0.01336877141147852,
+            0.018049223348498344,
+            -0.009784724563360214,
+            -0.009968994185328484,
+            0.006472475826740265,
+            0.023918215185403824,
+            -0.016317088156938553,
+            0.001357837812975049,
+            0.0019071920542046428,
+            0.03027552366256714,
+            0.0013267423491925001,
+            -0.017699111253023148,
+            0.013083153404295444,
+            0.012714614160358906,
+            -0.010623152367770672,
+            -0.015966976061463356,
+            -0.008213824592530727,
+            0.017127875238656998,
+            -0.015165401622653008,
+            0.006366520654410124,
+            -0.0054451716132462025,
+            0.0020269674714654684,
+            0.017044954001903534,
+            -0.03068091720342636,
+            0.005629441235214472,
+            0.010254612192511559,
+            -0.00041950165177695453,
+            0.002570563228800893,
+            0.006923936307430267,
+            -0.028045859187841415,
+            0.005403710994869471,
+            -0.035858895629644394,
+            -0.004445508122444153,
+            0.008886409923434258,
+            -0.009618882089853287,
+            0.01842697709798813,
+            -0.00916742067784071,
+            -0.020122257992625237,
+            -0.005201014224439859,
+            0.008757420815527439,
+            0.013248995877802372,
+            -0.006817981600761414,
+            0.014032143168151379,
+            0.011913040652871132,
+            0.0025912935379892588,
+            -0.003731462871655822,
+            -0.012290793471038342,
+            -0.005205621011555195,
+            -0.02227821573615074,
+            -0.014216412790119648,
+            0.006679778918623924,
+            0.006034834776073694,
+            -0.0010117562487721443,
+            0.009563600644469261,
+            -0.022609900683164597,
+            -0.00016785824846010655,
+            0.01988270878791809,
+            0.005901239346712828,
+            0.0011401692172512412,
+            -0.0016676414525136352,
+            0.028985634446144104,
+            0.008734387345612049,
+            -0.008135510608553886,
+            0.0010992843890562654,
+            0.012751467525959015,
+            0.0017206189222633839,
+            0.01447439007461071,
+            0.01424405351281166,
+            0.006444835104048252,
+            -0.009033825248479843,
+            0.0034803953021764755,
+            -0.003321462543681264,
+            0.00697461050003767,
+            0.002355197910219431,
+            -0.01538652554154396,
+            -0.013497760519385338,
+            0.012917310930788517,
+            -0.009462252259254456,
+            0.002360956510528922,
+            0.01862967386841774,
+            -0.006546183489263058,
+            -0.0028193274047225714,
+            0.0020684280898422003,
+            -0.025926755741238594,
+            -0.0016492144204676151,
+            -0.03191552311182022,
+            -0.010125624015927315,
+            -0.014511244371533394,
+            -0.026534846052527428,
+            0.03335282951593399,
+            -0.0021525011397898197,
+            0.02905934303998947,
+            -0.017524054273962975,
+            0.007739330176264048,
+            -0.015220683068037033,
+            0.024950126186013222,
+            0.005067418795078993,
+            0.00015432592772413045,
+            0.01604989729821682,
+            -0.027916869148612022,
+            -0.01566293090581894,
+            0.02550293505191803,
+            0.009609668515622616,
+            -0.01002427563071251,
+            0.036172155290842056,
+            0.02082248404622078,
+            0.0077485437504947186,
+            -0.006168430671095848,
+            0.023439114913344383,
+            0.0010319106513634324,
+            0.007260228972882032,
+            -0.010660005733370781,
+            0.013847872614860535,
+            -0.012686973437666893,
+            -0.0030151139944791794,
+            0.046767666935920715,
+            0.004477755166590214,
+            -0.0008827673736959696,
+            -0.013801805675029755,
+            0.010844276286661625,
+            -0.0060578687116503716,
+            0.015976188704371452,
+            -0.00862843170762062,
+            0.031583838164806366,
+            -0.005288542248308659,
+            -0.016980459913611412,
+            0.019753718748688698,
+            -0.01442832313477993,
+            0.04374564439058304,
+            -0.02561349794268608,
+            0.003968710079789162,
+            -0.012493490241467953,
+            -0.024544732645154,
+            0.013672816567122936,
+            -0.007707083132117987,
+            -0.003770620096474886,
+            0.008338207378983498,
+            -0.010733714327216148,
+            -0.010715287178754807,
+            0.011341804638504982,
+            -0.010116410441696644,
+            -0.006675172131508589,
+            -0.017275290563702583,
+            -0.009858432225883007,
+            -0.018528325483202934,
+            -0.002324102446436882,
+            -0.007725509814918041,
+            -0.00027582881739363074,
+            -0.00900618452578783,
+            -0.008490229956805706,
+            0.004222081042826176,
+            -0.00027122205938212574,
+            -0.004804834257811308,
+            0.0006040593143552542,
+            -0.003789047012105584,
+            0.00511348620057106,
+            -0.01486135646700859,
+            0.01639079675078392,
+            0.015423379838466644,
+            0.017477987334132195,
+            0.005680115427821875,
+            0.010577084496617317,
+            -0.006103936117142439,
+            -0.016409222036600113,
+            0.0006558852037414908,
+            0.01698967255651951,
+            0.01151686068624258,
+            0.012014389038085938,
+            0.019237764179706573,
+            -0.001894523506052792,
+            0.0007935116882435977,
+            -0.004219777882099152,
+            -0.005832138005644083,
+            -0.01223551295697689,
+            -0.009563600644469261,
+            -0.007587307598441839,
+            -0.02319956384599209,
+            0.0036969122011214495,
+            -0.0010906467214226723,
+            0.0013405625941231847,
+            0.0030427544843405485,
+            0.004763373639434576,
+            0.0072187683545053005,
+            0.001698736916296184,
+            -0.01685146987438202,
+            -0.003547193016856909,
+            -0.002957529854029417,
+            -0.019514167681336403,
+            -0.004247418139129877,
+            0.008702139370143414,
+            -0.010107196867465973,
+            -0.0016250290209427476,
+            0.012382928282022476,
+            0.0025521363131701946,
+            0.016934391111135483,
+            0.012640906497836113,
+            -0.006513936445116997,
+            -0.0006904358160682023,
+            0.009208881296217442,
+            -0.01462180633097887,
+            -0.012309220619499683,
+            -0.025355519726872444,
+            -0.017063381150364876,
+            -0.002549832919612527,
+            0.018749449402093887,
+            -0.00759652117267251,
+            -0.00880348775535822,
+            0.003574833506718278,
+            -0.009425398893654346,
+            0.00511348620057106,
+            -0.021559562534093857,
+            -0.0007624161662533879,
+            -0.020785629749298096,
+            0.005634048022329807,
+            -0.007052925415337086,
+            0.0036393278278410435,
+            0.009821578860282898,
+            0.034347884356975555,
+            0.0023678664583712816,
+            -0.013470119796693325,
+            -0.01265011914074421,
+            0.010881129652261734,
+            0.013212142512202263,
+            -0.013811019249260426,
+            -0.0020949169993400574,
+            -0.008324386551976204,
+            -0.002998990472406149,
+            0.0010509134735912085,
+            0.014492817223072052,
+            0.026203161105513573,
+            -0.01947731524705887,
+            -0.014713941141963005,
+            -0.018270347267389297,
+            -0.024323608726263046,
+            0.003708429168909788,
+            0.009637309238314629,
+            0.02082248404622078,
+            -0.00788674596697092,
+            -0.009148993529379368,
+            -0.005707756150513887,
+            -0.009047646075487137,
+            0.007545846980065107,
+            0.11542658507823944,
+            0.005772250704467297,
+            -0.001601995318196714,
+            -0.01462180633097887,
+            -0.004254328086972237,
+            0.010595511645078659,
+            -0.012760681100189686,
+            0.0664476752281189,
+            -0.013064726255834103,
+            0.005730789620429277,
+            -0.007467532530426979,
+            -0.0006184554076753557,
+            -0.004480058792978525,
+            -0.010973264463245869,
+            0.00079120829468593,
+            -0.018482258543372154,
+            0.000761840317863971,
+            -0.011295736767351627,
+            -0.0063204532489180565,
+            -0.004611351061612368,
+            -0.011885399930179119,
+            -0.016786975786089897,
+            0.01614203117787838,
+            -0.03038608469069004,
+            -0.001484523294493556,
+            0.0005410045268945396,
+            -0.020804056897759438,
+            0.003203990636393428,
+            -0.027013948187232018,
+            0.005726182833313942,
+            -0.0008816156769171357,
+            0.013811019249260426,
+            -0.005647868383675814,
+            -0.0204355176538229,
+            -0.013212142512202263,
+            0.008946297690272331,
+            0.009978207759559155,
+            0.01668562740087509,
+            0.017800459638237953,
+            0.005620228126645088,
+            -0.000780843140091747,
+            -0.011295736767351627,
+            0.010558657348155975,
+            -0.01065079215914011,
+            0.0003250633890274912,
+            -0.010890343226492405,
+            0.0018588212551549077,
+            -0.0021755348425358534,
+            0.02342068776488304,
+            -0.011811692267656326,
+            -0.021670125424861908,
+            0.001484523294493556,
+            0.02940945513546467,
+            -0.0034020806197077036,
+            0.017091020941734314,
+            7.133255712687969e-05,
+            0.009231915697455406,
+            -0.003399777226150036,
+            0.0004618261009454727,
+            0.026829678565263748,
+            -0.018454616889357567,
+            0.00623292475938797,
+            0.004625170957297087,
+            0.013672816567122936,
+            -0.008361240848898888,
+            0.01332270447164774,
+            -0.029943836852908134,
+            -0.005095059052109718,
+            -0.020195966586470604,
+            0.009089106693863869,
+            0.027566757053136826,
+            0.0051825870759785175,
+            0.012198658660054207,
+            -0.01899821311235428,
+            -0.012115737423300743,
+            0.0337950736284256,
+            -0.02006697840988636,
+            -0.010558657348155975,
+            -0.010871916078031063,
+            -0.024342035874724388,
+            0.00790977943688631,
+            0.011019332334399223,
+            -0.023457542061805725,
+            -0.015884054824709892,
+            0.004841688089072704,
+            -0.009664949029684067,
+            -0.017597762867808342,
+            0.0017160122515633702,
+            -0.015073266811668873,
+            -0.014409895986318588,
+            -0.005850565154105425,
+            -0.001894523506052792,
+            -0.011940681375563145,
+            0.00858236476778984,
+            -0.02159641683101654,
+            -0.0016446076333522797,
+            -0.0008729780092835426,
+            0.0024553947150707245,
+            0.0030634847935289145,
+            0.00623292475938797,
+            -0.0023770800326019526,
+            -0.005979553796350956,
+            -0.025171250104904175,
+            0.013064726255834103,
+            -0.0036577549763023853,
+            0.034937549382448196,
+            0.019256191328167915,
+            0.004056238103657961,
+            -0.004758766852319241,
+            -0.00841191504150629,
+            0.019993269816040993,
+            -0.02139372006058693,
+            -0.019145628437399864,
+            0.007776184007525444,
+            0.020398663356900215,
+            -0.02331012487411499,
+            0.00395028293132782,
+            -0.009024611674249172,
+            0.008665286004543304,
+            -0.010236185975372791,
+            -0.005247081629931927,
+            0.007587307598441839,
+            0.00011257731239311397,
+            0.0029045522678643465,
+            -0.006647531874477863,
+            0.01785574108362198,
+            0.013193715363740921,
+            -0.01242899615317583,
+            0.02310742810368538,
+            -0.0021064337342977524,
+            0.0019325291505083442,
+            0.0072510153986513615,
+            -0.009351690299808979,
+            -0.00769326277077198,
+            0.012521130964159966,
+            0.00484629487618804,
+            -0.02045394480228424,
+            0.018777089193463326,
+            0.007803824730217457,
+            0.02828541025519371,
+            -0.0014016019413247705,
+            -0.01662113331258297,
+            -0.0068594422191381454,
+            0.004173710476607084,
+            -0.02168855257332325,
+            -0.027953723445534706,
+            0.017072593793272972,
+            -0.012382928282022476,
+            -0.019182482734322548,
+            -0.00895090401172638,
+            0.010558657348155975,
+            -0.020472371950745583,
+            -0.0068502286449074745,
+            0.0029344961512833834,
+            0.0337950736284256,
+            -0.0038650583010166883,
+            0.002782473573461175,
+            0.011673489585518837,
+            0.004118429496884346,
+            0.001210422022268176,
+            -0.021430574357509613,
+            0.0016549729043617845,
+            0.02905934303998947,
+            0.025631925091147423,
+            0.018565179780125618,
+            -0.018841583281755447,
+            0.0005994525854475796,
+            0.009959780611097813,
+            0.009775510989129543,
+            0.006983824074268341,
+            0.008066409267485142,
+            -0.013405625708401203,
+            -0.0031187657732516527,
+            0.009411578066647053,
+            -0.011922254227101803,
+            0.014170344918966293,
+            -0.0013014052528887987,
+            -0.012180231511592865,
+            -0.004152979701757431,
+            -0.020656641572713852,
+            -0.02187282219529152,
+            -0.001980900065973401,
+            -0.005744609981775284,
+            0.0035978672094643116,
+            3.9805148844607174e-05,
+            0.005362250376492739,
+            0.01593933440744877,
+            -0.013884726911783218,
+            0.018076865002512932,
+            0.009932140819728374,
+            0.0033053390216082335,
+            -0.010917983949184418,
+            -0.0363011434674263,
+            0.01785574108362198,
+            -0.01513776183128357,
+            -0.012336861342191696,
+            0.021743832156062126,
+            -0.006679778918623924,
+            0.013783378526568413,
+            -0.021246304735541344,
+            0.0003751617332454771,
+            0.00769326277077198,
+            0.024581586942076683,
+            -0.017643829807639122,
+            0.00568472221493721,
+            0.0062283179722726345,
+            0.0030012938659638166,
+            0.00974787026643753,
+            0.008706746622920036,
+            0.0035287661012262106,
+            0.015515514649450779
+          ],
+          "title": "Appendices A Load-Balancing Loss",
+          "keyphrases": [
+            "Load-Balancing Loss",
+            "hierarchical MoE",
+            "experts",
+            "perplexity",
+            "attention mechanism"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "16ef9dd0-c391-4f65-84c4-f2e913462563",
+      "type": "next",
+      "source": {
+        "id": "294d1382-75f0-4a45-bc91-9727e57552f3",
+        "properties": {
+          "page_content": "1 Introduction And Related Work 1.1 Conditional Computation\n\nExploiting scale in both training data and model size has been central to the success of deep learning. When datasets are sufficiently large, increasing the capacity (number of parameters) of neural networks can give much better prediction accuracy. This has been shown in domains such as text (Sutskever et al., 2014; Bahdanau et al., 2014; Jozefowicz et al., 2016; Wu et al., 2016), images (Krizhevsky et al., 2012; Le et al., 2012), and audio (Hinton et al., 2012; Amodei et al., 2015). For typical deep learning models, where the entire model is activated for every example, this leads to a roughly quadratic blow-up in training costs, as both the model size and the number of training examples increase. Unfortunately, the advances in computing power and distributed computation fall short of meeting such demand.\n\nVarious forms of conditional computation have been proposed as a way to increase model capacity without a proportional increase in computational costs (Davis & Arel, 2013; Bengio et al., 2013; Eigen et al., 2013; Ludovic Denoyer, 2014; Cho & Bengio, 2014; Bengio et al., 2015; Almahairi et al., 2015). In these schemes, large parts of a network are active or inactive on a per-example basis. The gating decisions may be binary or sparse and continuous, stochastic or deterministic. Various forms of reinforcement learning and back-propagation are proposed for trarining the gating decisions.\n\nWhile these ideas are promising in theory, no work to date has yet demonstrated massive improvements in model capacity, training time, or model quality. We blame this on a combination of the following challenges:\n\nModern computing devices, especially GPUs, are much faster at arithmetic than at branching. Most of the works above recognize this and propose turning on/off large chunks of the network with each gating decision.\n\nLarge batch sizes are critical for performance, as they amortize the costs of parameter transfers and updates. Conditional computation reduces the batch sizes for the conditionally active chunks of the network.\n\nNetwork bandwidth can be a bottleneck. A cluster of GPUs may have computational power thousands of times greater than the aggregate inter-device network bandwidth. To be computationally efficient, the relative computational versus network demands of an algorithm must exceed this ratio. Embedding layers, which can be seen as a form of conditional computation, are handicapped by this very problem. Since the embeddings generally need to be sent across the network, the number of (example, parameter) interactions is limited by network bandwidth instead of computational capacity.\n\nDepending on the scheme, loss terms may be necessary to achieve the desired level of sparsity per-chunk and/or per example. Bengio et al. (2015) use three such terms. These issues can affect both model quality and load-balancing.\n\nModel capacity is most critical for very large data sets. The existing literature on conditional computation deals with relatively small image recognition data sets consisting of up to 600,000 images. It is hard to imagine that the labels of these images provide a sufficient signal to adequately train a model with millions, let alone billions of parameters. In this work, we for the first time address all of the above challenges and finally realize the promise of conditional computation. We obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets.\n\n1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1). All parts of the network are trained jointly by back-propagation.\n\nWhile the introduced technique is generic, in this paper we focus on language modeling and machine translation tasks, which are known to benefit from very large models. In particular, we apply a MoE convolutionally between stacked LSTM layers (Hochreiter & Schmidhuber, 1997), as in Figure 1.\n\nThe MoE is called once for each position in the text, selecting a potentially different combination of experts at each position. The different experts tend to become highly specialized based on syntax and semantics (see Appendix E Table 9). On both language modeling and machine translation benchmarks, we improve on best published results at a fraction of the computational cost.\n\n1.3 Related Work On Mixtures Of Experts\n\nSince its introduction more than two decades ago (Jacobs et al., 1991; Jordan & Jacobs, 1994), the mixture-of-experts approach has been the subject of much research. Different types of expert architectures hae been proposed such as SVMs (Collobert et al., 2002), Gaussian Processes (Tresp, 2001; Theis & Bethge, 2015; Deisenroth & Ng, 2015), Dirichlet Processes (Shahbaba & Neal, 2009), and deep networks. Other work has focused on different expert configurations such as a hierarchical structure (Yao et al., 2009), infinite numbers of experts (Rasmussen & Ghahramani, 2002), and adding experts sequentially (Aljundi et al., 2016). Garmash & Monz (2016) suggest an ensemble model in the format of mixture of experts for machine translation. The gating network is trained on a pre-trained ensemble NMT model. The works above concern top-level mixtures of experts. The mixture of experts is the whole model. Eigen et al. (2013) introduce the idea of using multiple MoEs with their own gating networks as parts of a deep model. It is intuitive that the latter approach is more powerful, since complex problems may contain many sub-problems each requiring different experts. They also allude in their conclusion to the potential to introduce sparsity, turning MoEs into a vehicle for computational computation.\n\nOur work builds on this use of MoEs as a general purpose neural network component. While Eigen et al. (2013) uses two stacked MoEs allowing for two sets of gating decisions, our convolutional application of the MoE allows for different gating decisions at each position in the text. We also realize sparse gating and demonstrate its use as a practical way to massively increase model capacity.\n\n",
+          "embedding": [
+            -0.011026207357645035,
+            -0.0030965988989919424,
+            -0.017856089398264885,
+            0.0010101930238306522,
+            0.02138679102063179,
+            0.0306765865534544,
+            -0.014028750360012054,
+            0.025452885776758194,
+            -0.030126724392175674,
+            0.021791953593492508,
+            0.010809156112372875,
+            -0.003031483618542552,
+            0.0071120485663414,
+            -0.010223117657005787,
+            0.014021515846252441,
+            0.00018878943228628486,
+            0.01843489333987236,
+            0.003255769843235612,
+            0.03244193643331528,
+            -0.030329305678606033,
+            0.006352368742227554,
+            -0.010158002376556396,
+            -0.03186313435435295,
+            -0.031226450577378273,
+            0.018391482532024384,
+            0.023152140900492668,
+            0.040400486439466476,
+            -0.0036120957229286432,
+            -0.027333997189998627,
+            0.01920180767774582,
+            0.016163088381290436,
+            0.000524540722835809,
+            -0.03342590481042862,
+            -0.03348378464579582,
+            -0.058574918657541275,
+            -0.044220589101314545,
+            0.04589911922812462,
+            0.018217841163277626,
+            -0.003402279457077384,
+            0.018926875665783882,
+            0.037419646978378296,
+            -0.001556438859552145,
+            -0.03244193643331528,
+            -0.025858048349618912,
+            0.0014379650820046663,
+            0.016626130789518356,
+            -0.016466960310935974,
+            -0.03426516801118851,
+            -0.015902627259492874,
+            -0.01490419078618288,
+            0.02338366210460663,
+            -0.025742286816239357,
+            0.01846383325755596,
+            -0.000727121951058507,
+            -0.040603067725896835,
+            0.033049680292606354,
+            0.026639433577656746,
+            -0.004460404627025127,
+            0.04691202566027641,
+            0.04639110341668129,
+            0.0038635137025266886,
+            0.006019556894898415,
+            -0.04427846893668175,
+            -0.0017617333214730024,
+            0.02446891926229,
+            -0.015251473523676395,
+            -0.024526799097657204,
+            0.021010568365454674,
+            0.018594063818454742,
+            -0.00322140334174037,
+            -0.037680111825466156,
+            0.00494153518229723,
+            -0.03119751065969467,
+            0.01049804873764515,
+            0.005907413549721241,
+            0.01499101147055626,
+            0.02413610741496086,
+            0.04008214548230171,
+            0.021242089569568634,
+            0.011192613281309605,
+            -0.003957569133490324,
+            0.022862739861011505,
+            -0.006706885993480682,
+            0.010975562036037445,
+            0.027218235656619072,
+            -0.02911381796002388,
+            0.005502250976860523,
+            -0.06227926164865494,
+            0.004485727287828922,
+            -0.013276305980980396,
+            -0.0250043123960495,
+            0.013565707951784134,
+            -0.0021397643722593784,
+            0.0019570793956518173,
+            0.010577634908258915,
+            0.03658038377761841,
+            -0.0005281582125462592,
+            -0.013869579881429672,
+            0.005491398740559816,
+            -0.0063632214441895485,
+            -0.01565663516521454,
+            0.01856512390077114,
+            0.007361657451838255,
+            -0.0035723031032830477,
+            0.014448382891714573,
+            -0.0009423645096831024,
+            -0.023730944842100143,
+            0.026205329224467278,
+            0.008710993453860283,
+            0.08670476824045181,
+            -0.009463437832891941,
+            -0.022775918245315552,
+            -0.001728271134197712,
+            0.03600158169865608,
+            -0.0008876494830474257,
+            -0.011489249765872955,
+            -0.04349708557128906,
+            -0.04118186980485916,
+            0.007216956466436386,
+            -0.022110294550657272,
+            0.03328120335936546,
+            0.03843255341053009,
+            -0.015164652839303017,
+            -0.010158002376556396,
+            -0.05591242015361786,
+            0.07883304357528687,
+            -0.013695938512682915,
+            -0.035162314772605896,
+            -0.008703758008778095,
+            -0.027044596150517464,
+            -0.002848798641934991,
+            0.00745209539309144,
+            -0.01866641454398632,
+            -0.02755104750394821,
+            0.04659368470311165,
+            0.012415335513651371,
+            0.03773799166083336,
+            0.022139234468340874,
+            -0.01235022023320198,
+            -0.013630823232233524,
+            0.03244193643331528,
+            0.0003031935775652528,
+            0.004037154838442802,
+            -0.017783738672733307,
+            -0.007980253547430038,
+            0.005802505649626255,
+            -0.022732509300112724,
+            0.057648833841085434,
+            -0.03794057294726372,
+            0.02171960286796093,
+            -0.030300365760922432,
+            0.031023869290947914,
+            -0.016163088381290436,
+            0.03941652178764343,
+            -0.01902816630899906,
+            0.01389128528535366,
+            -0.0250043123960495,
+            -0.017957380041480064,
+            0.006417484022676945,
+            -0.011178143322467804,
+            -0.016987884417176247,
+            -0.019910842180252075,
+            -0.015613225288689137,
+            0.02323896251618862,
+            0.0025485444348305464,
+            0.03073446825146675,
+            0.0352201946079731,
+            0.019476739689707756,
+            -0.0614689365029335,
+            0.019447797909379005,
+            0.004789599217474461,
+            0.021632781252264977,
+            0.019968722015619278,
+            -0.0019860195461660624,
+            0.03134221211075783,
+            0.009297031909227371,
+            0.02205241471529007,
+            0.03252875804901123,
+            -0.005419048015028238,
+            0.0330207422375679,
+            -0.03001096285879612,
+            -0.01251662615686655,
+            0.02878100611269474,
+            0.008117719553411007,
+            0.025091134011745453,
+            0.008877399377524853,
+            0.015801336616277695,
+            -0.007813847623765469,
+            0.0341494083404541,
+            0.0012118698796257377,
+            0.01969379000365734,
+            0.023962466046214104,
+            0.036725085228681564,
+            -0.030589766800403595,
+            0.0233113132417202,
+            -0.04074776917695999,
+            -0.019375449046492577,
+            0.024092696607112885,
+            0.02338366210460663,
+            0.017002353444695473,
+            -0.03368636593222618,
+            -0.03212359547615051,
+            0.02480173110961914,
+            0.01720493473112583,
+            0.013695938512682915,
+            0.009246386587619781,
+            0.013703173957765102,
+            0.0147884301841259,
+            0.017595628276467323,
+            -0.037882693111896515,
+            0.009969891048967838,
+            0.00405524205416441,
+            -0.009940951131284237,
+            -0.04303404316306114,
+            0.006829881574958563,
+            -0.0030152045655995607,
+            -0.022240525111556053,
+            -0.0033299291972070932,
+            -0.0182033721357584,
+            -0.003758605569601059,
+            -0.002535883104428649,
+            0.01681424304842949,
+            0.021459141746163368,
+            -0.021632781252264977,
+            -0.0020529436878859997,
+            -0.017291756346821785,
+            -0.021835362538695335,
+            -0.03235511854290962,
+            0.003633800894021988,
+            -0.005400960333645344,
+            0.02464256063103676,
+            0.03140009194612503,
+            0.017885029315948486,
+            -0.004778746515512466,
+            0.03938758000731468,
+            -0.03368636593222618,
+            0.016626130789518356,
+            -0.0012697502970695496,
+            0.007488270755857229,
+            0.008862929418683052,
+            -0.024353159591555595,
+            0.04178961366415024,
+            -0.0409214086830616,
+            -0.005690362304449081,
+            0.011330079287290573,
+            0.010035006329417229,
+            -0.010794686153531075,
+            0.0044133770279586315,
+            -0.023354722186923027,
+            -0.02649473212659359,
+            0.0013222043635323644,
+            -0.012762618251144886,
+            -0.016553781926631927,
+            0.02205241471529007,
+            -0.02539500594139099,
+            0.03976380452513695,
+            0.030618706718087196,
+            -0.02490302175283432,
+            -0.01976614072918892,
+            -0.006916702259331942,
+            0.04123975336551666,
+            -0.009108920581638813,
+            -0.01268303208053112,
+            0.02569887787103653,
+            -0.025612056255340576,
+            0.0019950633868575096,
+            0.036956604570150375,
+            0.022587807849049568,
+            -0.004930682480335236,
+            -0.01800079084932804,
+            0.0038237208500504494,
+            -0.05845915898680687,
+            0.0035361277405172586,
+            -0.041471272706985474,
+            0.05614394322037697,
+            -0.03736176714301109,
+            0.0041999430395662785,
+            0.0013303437735885382,
+            -0.012111463584005833,
+            0.04028472676873207,
+            0.022935090586543083,
+            -0.009405557066202164,
+            -0.022804860025644302,
+            0.02480173110961914,
+            0.015468524768948555,
+            -0.004123975057154894,
+            -0.015208062715828419,
+            -0.0024273572489619255,
+            -0.05116623267531395,
+            0.002250098856166005,
+            -0.005104323849081993,
+            -0.002651643706485629,
+            -0.048359036445617676,
+            0.015671106055378914,
+            0.028607364743947983,
+            -0.001820518053136766,
+            -0.011402430012822151,
+            -0.01437603309750557,
+            0.02708800509572029,
+            -0.017566686496138573,
+            -0.042397357523441315,
+            -0.006793706677854061,
+            0.013970870524644852,
+            0.008776108734309673,
+            -0.03209465742111206,
+            0.006746678613126278,
+            -0.003434837330132723,
+            0.02543841488659382,
+            -0.0076691466383636,
+            0.004728101193904877,
+            0.018188901245594025,
+            0.0015220724744722247,
+            0.01614861935377121,
+            -0.01747986674308777,
+            -0.006120847538113594,
+            0.02141573093831539,
+            0.021676192060112953,
+            0.0006932076648809016,
+            -0.016872122883796692,
+            -0.03316544368863106,
+            0.032181475311517715,
+            0.002027621027082205,
+            -0.0637841522693634,
+            -0.01426027249544859,
+            -0.03542277589440346,
+            0.01033164281398058,
+            0.004677455872297287,
+            0.03247087821364403,
+            -0.02228393591940403,
+            0.018246781080961227,
+            -0.011713536456227303,
+            -0.03542277589440346,
+            -0.024512330070137978,
+            0.013666998594999313,
+            -0.030879167839884758,
+            -0.03987956419587135,
+            -0.017118114978075027,
+            0.023991405963897705,
+            -0.013117135502398014,
+            -0.029026998206973076,
+            -0.01969379000365734,
+            -0.031255390495061874,
+            0.0030748937278985977,
+            0.027739159762859344,
+            0.01774032786488533,
+            -0.024946432560682297,
+            -0.034033648669719696,
+            0.0057265376672148705,
+            -0.01178588718175888,
+            -0.0018594063585624099,
+            0.0010210456093773246,
+            0.018261251971125603,
+            0.006489834748208523,
+            -0.011257728561758995,
+            -0.017552217468619347,
+            -0.04413377121090889,
+            -0.008407121524214745,
+            0.0005385585827752948,
+            -0.009904775768518448,
+            0.034699272364377975,
+            0.03180525451898575,
+            -0.012125934474170208,
+            -0.011612245813012123,
+            0.014745020307600498,
+            -0.047201428562402725,
+            -0.015367234125733376,
+            -0.025640996173024178,
+            -0.005306905135512352,
+            0.01684318296611309,
+            0.008928044699132442,
+            -0.003950334154069424,
+            0.042889341711997986,
+            0.002532265381887555,
+            -0.013507827185094357,
+            0.007922373712062836,
+            0.006956494878977537,
+            0.028144322335720062,
+            0.0057265376672148705,
+            -0.03637780249118805,
+            0.04381542652845383,
+            -0.02464256063103676,
+            0.010027770884335041,
+            0.013348656706511974,
+            -0.011728006415069103,
+            -0.013761053793132305,
+            0.018550653010606766,
+            -0.005636099260300398,
+            0.025858048349618912,
+            -0.020779047161340714,
+            0.061526816338300705,
+            -0.006761149037629366,
+            -0.05186079815030098,
+            -0.0147884301841259,
+            -0.04089247062802315,
+            0.021271029487252235,
+            0.013377596624195576,
+            -0.002273612655699253,
+            -0.020344944670796394,
+            -0.013030314818024635,
+            0.019288627430796623,
+            0.004000979475677013,
+            -0.009629843756556511,
+            -0.01235022023320198,
+            0.006366838701069355,
+            -0.01578686572611332,
+            -0.005990616511553526,
+            0.007376127410680056,
+            0.007625736761838198,
+            0.04187643527984619,
+            -0.03142903372645378,
+            -0.0031201126985251904,
+            -0.01594603806734085,
+            -0.048127513378858566,
+            0.0049560051411390305,
+            0.03154479339718819,
+            0.0364067442715168,
+            -0.023947997018694878,
+            0.0003253509057685733,
+            0.013175015337765217,
+            0.0018973903497681022,
+            -0.017653508111834526,
+            0.011113028042018414,
+            0.03336802497506142,
+            0.02457020990550518,
+            -0.0187966451048851,
+            -0.005245407111942768,
+            0.0062438431195914745,
+            -0.0020746488589793444,
+            0.00621852045878768,
+            -0.04031366482377052,
+            0.03637780249118805,
+            -0.01235022023320198,
+            0.02998202294111252,
+            -0.028708655387163162,
+            -0.013102664612233639,
+            -0.03481503203511238,
+            -0.010223117657005787,
+            0.03472821041941643,
+            -0.032876040786504745,
+            -0.007188016548752785,
+            0.017262814566493034,
+            -0.002754742978140712,
+            -0.006724973674863577,
+            0.018550653010606766,
+            -0.04636216163635254,
+            -0.019520148634910583,
+            -0.027536578476428986,
+            0.0008266037912108004,
+            -0.0018811115296557546,
+            -0.05871962010860443,
+            -0.061295293271541595,
+            -0.005350315012037754,
+            -0.054233890026807785,
+            -0.019491208717226982,
+            0.03935864195227623,
+            -0.009477907791733742,
+            -0.002881356282159686,
+            -0.019838491454720497,
+            0.009543023072183132,
+            -0.028694184496998787,
+            -0.0034185582771897316,
+            0.026205329224467278,
+            0.017190465703606606,
+            -0.018319131806492805,
+            -0.006236608140170574,
+            0.001457861391827464,
+            -0.062221381813287735,
+            -0.027507638558745384,
+            0.0053466977551579475,
+            -0.02477279119193554,
+            -0.008609702810645103,
+            -0.019780611619353294,
+            0.0038526610005646944,
+            0.01460755430161953,
+            0.00634151604026556,
+            0.037622228264808655,
+            0.0022898914758116007,
+            -0.06980370730161667,
+            0.013182250782847404,
+            0.024454450234770775,
+            0.023137671872973442,
+            -0.023600714281201363,
+            -0.00623299041762948,
+            -0.02228393591940403,
+            -0.022529928013682365,
+            -0.02636450156569481,
+            0.0061859628185629845,
+            0.004109505098313093,
+            0.005440753418952227,
+            -0.013884049840271473,
+            0.009311501868069172,
+            0.0421079583466053,
+            0.005227319430559874,
+            0.0153382932767272,
+            0.0037296651862561703,
+            -0.011366254650056362,
+            -0.020402824506163597,
+            -0.003644653595983982,
+            0.02924404852092266,
+            0.011460309848189354,
+            -0.03681190311908722,
+            -0.00158537901006639,
+            0.04427846893668175,
+            0.03530701622366905,
+            0.02198006398975849,
+            -0.011764181777834892,
+            0.009043805301189423,
+            0.020706696435809135,
+            -0.06013768911361694,
+            -0.00040742341661825776,
+            0.010042241774499416,
+            0.0057844179682433605,
+            -0.02148808166384697,
+            -0.020373884588479996,
+            -0.008754403330385685,
+            0.01953461952507496,
+            -0.013789994642138481,
+            0.008276890963315964,
+            6.890248914714903e-05,
+            0.033049680292606354,
+            0.011236024089157581,
+            -0.012531096115708351,
+            -0.017103644087910652,
+            0.05825657770037651,
+            -0.02539500594139099,
+            0.02878100611269474,
+            0.026842014864087105,
+            -0.023658594116568565,
+            0.04734613001346588,
+            -0.005386490374803543,
+            -0.04057412967085838,
+            0.015801336616277695,
+            0.009022099897265434,
+            -0.02171960286796093,
+            -0.025293715298175812,
+            0.0006579368491657078,
+            -0.007430390454828739,
+            0.002532265381887555,
+            -0.04642004519701004,
+            -0.010266527533531189,
+            -0.0025811020750552416,
+            0.017624568194150925,
+            -0.015642166137695312,
+            0.006019556894898415,
+            0.02600274793803692,
+            -0.00706140324473381,
+            0.041037172079086304,
+            -0.041934315115213394,
+            -0.008045368827879429,
+            0.00811048410832882,
+            -0.03750646859407425,
+            -0.03180525451898575,
+            -0.0013927459949627519,
+            -0.0012588977115228772,
+            -0.018058670684695244,
+            -0.018709823489189148,
+            0.00033439468825235963,
+            -0.008305830880999565,
+            -0.006869674660265446,
+            -0.04254205897450447,
+            0.011662891134619713,
+            -0.013522298075258732,
+            -0.006417484022676945,
+            0.003910541534423828,
+            -0.010592104867100716,
+            -0.019114986062049866,
+            0.024266337975859642,
+            0.03484397381544113,
+            0.024266337975859642,
+            0.023861175402998924,
+            -0.0021126328501850367,
+            -0.021271029487252235,
+            -0.01455690898001194,
+            0.02685648389160633,
+            0.001517550554126501,
+            0.007206104230135679,
+            0.024845141917467117,
+            -0.004927065223455429,
+            0.0015410644700750709,
+            -0.011293903924524784,
+            0.0363488607108593,
+            -0.008703758008778095,
+            0.00639216136187315,
+            0.04485727474093437,
+            0.02795621007680893,
+            0.003259387332946062,
+            -0.006649005692452192,
+            -0.0039539518766105175,
+            0.005277964752167463,
+            -0.03394682705402374,
+            0.01840595155954361,
+            0.019389918074011803,
+            0.0038237208500504494,
+            -0.0056071593426167965,
+            0.006431954447180033,
+            -0.024179518222808838,
+            0.011496485210955143,
+            0.01183653250336647,
+            0.00784278754144907,
+            -0.03660932183265686,
+            0.012545567005872726,
+            0.019259687513113022,
+            0.0014054073253646493,
+            -0.015974977985024452,
+            -0.0116701265797019,
+            -0.014238567091524601,
+            0.024222927168011665,
+            0.001188355963677168,
+            -0.003324502846226096,
+            0.0009622608777135611,
+            0.030994929373264313,
+            0.012777088209986687,
+            0.016727421432733536,
+            -0.022732509300112724,
+            0.02675519324839115,
+            -0.0074159204959869385,
+            0.0068009416572749615,
+            -0.0010110974544659257,
+            -0.014462853781878948,
+            -0.013363126665353775,
+            0.03565429896116257,
+            -0.0019968722481280565,
+            0.04138445109128952,
+            -0.005755477584898472,
+            0.013384832069277763,
+            0.03770904988050461,
+            -0.0027728306595236063,
+            -0.02600274793803692,
+            -0.007712556980550289,
+            0.0007709843921475112,
+            -0.015309353359043598,
+            -0.016293318942189217,
+            -0.027507638558745384,
+            -0.013269071467220783,
+            -0.009427262470126152,
+            -0.017885029315948486,
+            -0.012342985719442368,
+            0.025062192231416702,
+            -0.0037658405490219593,
+            -0.0182033721357584,
+            -0.03727494925260544,
+            0.028940176591277122,
+            0.0004478944465517998,
+            0.03692766651511192,
+            -0.008240715600550175,
+            0.012994139455258846,
+            -0.008616937324404716,
+            0.013920225203037262,
+            0.016307789832353592,
+            0.013023079372942448,
+            0.022370757535099983,
+            0.0005584549508057535,
+            0.00988307036459446,
+            0.03565429896116257,
+            -0.031486913561820984,
+            -0.03348378464579582,
+            0.014976541511714458,
+            -0.01522253267467022,
+            0.022095825523138046,
+            0.0029211491346359253,
+            -0.0050283558666706085,
+            0.021401260048151016,
+            0.006457277107983828,
+            -0.034207288175821304,
+            0.01966485008597374,
+            -0.012444276362657547,
+            -0.03319438174366951,
+            0.03435198962688446,
+            -0.014802900142967701,
+            0.004131210036575794,
+            -0.012480450794100761,
+            -0.0025485444348305464,
+            -0.024353159591555595,
+            0.026711782440543175,
+            0.019809551537036896,
+            -0.025915928184986115,
+            0.02523583360016346,
+            0.030358245596289635,
+            0.013348656706511974,
+            -0.004142062738537788,
+            0.008450531400740147,
+            -0.0048510972410440445,
+            -0.022443106397986412,
+            -0.02549629658460617,
+            0.002917531644925475,
+            0.011641185730695724,
+            -0.005719302222132683,
+            0.021459141746163368,
+            -0.049661342054605484,
+            -0.007481035776436329,
+            0.0005797079065814614,
+            -0.016438020393252373,
+            -0.004764276556670666,
+            0.00960090383887291,
+            0.013761053793132305,
+            0.0030622323974967003,
+            -0.03154479339718819,
+            0.0318920761346817,
+            0.029258519411087036,
+            0.020070012658834457,
+            -0.008732697926461697,
+            0.017566686496138573,
+            -0.006374074146151543,
+            0.012096993625164032,
+            0.031949955970048904,
+            0.01827572099864483,
+            0.008038134314119816,
+            0.0005860385717824101,
+            -0.027898330241441727,
+            0.02015683241188526,
+            0.011330079287290573,
+            0.035827938467264175,
+            0.02128550037741661,
+            0.021864304319024086,
+            0.02225499600172043,
+            -0.022023474797606468,
+            0.006493452005088329,
+            -0.006558567751199007,
+            -0.016235439106822014,
+            0.01658272184431553,
+            -0.010628280229866505,
+            0.03504655510187149,
+            0.012147638946771622,
+            -0.05186079815030098,
+            -0.012314044870436192,
+            0.019809551537036896,
+            0.022356286644935608,
+            0.04372860863804817,
+            0.017335165292024612,
+            0.0025666318833827972,
+            0.04838797450065613,
+            -0.02642238140106201,
+            -0.0306765865534544,
+            0.005871238186955452,
+            0.02025812305510044,
+            -0.024859612807631493,
+            0.009760074317455292,
+            0.02464256063103676,
+            -0.018217841163277626,
+            -0.016365669667720795,
+            0.015613225288689137,
+            0.0020999715197831392,
+            0.0030459535773843527,
+            -0.014137276448309422,
+            -0.008096014149487019,
+            0.02487408183515072,
+            0.006301723420619965,
+            0.036985546350479126,
+            -0.01771138794720173,
+            0.019143927842378616,
+            0.00183589244261384,
+            -0.01473778486251831,
+            0.0284481942653656,
+            0.0011639377335086465,
+            -0.03362848609685898,
+            0.025742286816239357,
+            0.012965199537575245,
+            0.010512519627809525,
+            0.009912010282278061,
+            -0.0009658784256316721,
+            0.021907713264226913,
+            0.04233947768807411,
+            0.010592104867100716,
+            0.0004639472172129899,
+            0.03319438174366951,
+            0.024946432560682297,
+            0.003083937568590045,
+            0.01415898185223341,
+            -0.006649005692452192,
+            -0.013052019290626049,
+            0.011532660573720932,
+            -0.02061987668275833,
+            0.0278838612139225,
+            -0.023846706375479698,
+            -0.012393631041049957,
+            -0.004572547972202301,
+            -0.0021144417114555836,
+            -0.0011386150727048516,
+            -0.012096993625164032,
+            -0.008074309676885605,
+            -0.0187677051872015,
+            0.020402824506163597,
+            -0.007965783588588238,
+            0.037419646978378296,
+            -0.007188016548752785,
+            -0.0012118698796257377,
+            0.015048892237246037,
+            -0.026610491797327995,
+            0.002241055015474558,
+            -0.0010174281196668744,
+            0.021734071895480156,
+            -0.01571451686322689,
+            -0.005599924363195896,
+            0.012929024174809456,
+            0.03912711888551712,
+            -0.009260856546461582,
+            0.0003251247981097549,
+            0.029504509642720222,
+            0.0017798208864405751,
+            -0.03817209228873253,
+            0.028838885948061943,
+            -0.046506863087415695,
+            0.008667582646012306,
+            0.009116156026721,
+            -0.022168176248669624,
+            -0.019737200811505318,
+            -0.025640996173024178,
+            -0.004406142048537731,
+            0.0033009888138622046,
+            -0.01203911378979683,
+            0.020807987079024315,
+            -0.00955025851726532,
+            -0.025423945859074593,
+            0.02998202294111252,
+            0.014665434136986732,
+            0.009470673277974129,
+            -0.02675519324839115,
+            0.014267507009208202,
+            -0.036754023283720016,
+            -0.024193987250328064,
+            -0.0038273383397608995,
+            -0.012531096115708351,
+            0.0076980870217084885,
+            -0.006909467279911041,
+            -0.027666809037327766,
+            -0.014021515846252441,
+            0.00466660363599658,
+            -0.01584474742412567,
+            0.039705920964479446,
+            0.035625357180833817,
+            0.04349708557128906,
+            0.014035985805094242,
+            -0.035856880247592926,
+            0.02626321092247963,
+            -0.011170907877385616,
+            0.02059093490242958,
+            -0.009159565903246403,
+            0.0009902967140078545,
+            0.000590108276810497,
+            0.03637780249118805,
+            -0.02536606416106224,
+            0.010360583662986755,
+            -0.04254205897450447,
+            -0.008023664355278015,
+            -0.027377407997846603,
+            0.0032756661530584097,
+            -0.006352368742227554,
+            -0.001016523689031601,
+            0.009991596452891827,
+            -0.004362731706351042,
+            0.021502550691366196,
+            0.04236841946840286,
+            0.0016324068419635296,
+            0.02977944165468216,
+            -0.03883771598339081,
+            -0.020214714109897614,
+            0.013312481343746185,
+            -0.008595232851803303,
+            -0.013146075420081615,
+            -0.019114986062049866,
+            -0.012538331560790539,
+            -0.021213149651885033,
+            0.00700352294370532,
+            0.013710408471524715,
+            -0.01200293842703104,
+            0.001774394535459578,
+            0.016698481515049934,
+            -0.017291756346821785,
+            -0.0018847290193662047,
+            -0.03371530771255493,
+            -0.011633951216936111,
+            0.01899922639131546,
+            -0.026350030675530434,
+            0.029504509642720222,
+            -0.0045580780133605,
+            0.027970680966973305,
+            -0.02420845814049244,
+            -0.010751275345683098,
+            -0.0021415730006992817,
+            -0.02805750072002411,
+            -0.0020221949089318514,
+            0.016003917902708054,
+            0.03889559581875801,
+            -0.01761009730398655,
+            0.04714354872703552,
+            0.0003796137170866132,
+            0.013565707951784134,
+            -0.024555740877985954,
+            0.005249024368822575,
+            -0.052439600229263306,
+            0.008942514657974243,
+            -0.008660348132252693,
+            -0.033570606261491776,
+            -0.01581580750644207,
+            -0.004829391837120056,
+            0.013189485296607018,
+            -0.022370757535099983,
+            -0.00500665046274662,
+            0.01044740341603756,
+            -0.012560036964714527,
+            0.00022609512961935252,
+            -0.023021910339593887,
+            -0.020185774192214012,
+            -0.007065020501613617,
+            0.012639622204005718,
+            -0.03440986946225166,
+            0.016669541597366333,
+            -0.0352780744433403,
+            -0.0049704755656421185,
+            -0.027464227750897408,
+            -0.021271029487252235,
+            0.02911381796002388,
+            0.027232706546783447,
+            -0.04508879408240318,
+            0.029895203188061714,
+            0.018926875665783882,
+            0.011576070450246334,
+            -0.03976380452513695,
+            0.03840361535549164,
+            0.00019421572505962104,
+            0.010273762978613377,
+            0.003168949391692877,
+            -0.008645878173410892,
+            0.02228393591940403,
+            -0.03047400526702404,
+            -0.013052019290626049,
+            0.00472448393702507,
+            -0.006377691403031349,
+            0.012654092162847519,
+            -0.0010617427760735154,
+            -0.023586243391036987,
+            0.016307789832353592,
+            0.011482015252113342,
+            0.012871143408119678,
+            0.03339696303009987,
+            -0.025583116337656975,
+            0.0352780744433403,
+            0.0045182849280536175,
+            -0.018550653010606766,
+            0.006793706677854061,
+            -0.013507827185094357,
+            0.0034764388110488653,
+            0.029157228767871857,
+            0.03930075839161873,
+            0.008587997406721115,
+            -0.036956604570150375,
+            0.02397693693637848,
+            0.022226056084036827,
+            -0.0035867730621248484,
+            0.02867971546947956,
+            0.011474779807031155,
+            0.024613620713353157,
+            0.03545171767473221,
+            -0.0023441542871296406,
+            0.003197889542207122,
+            0.026017218828201294,
+            -0.023094261065125465,
+            0.007596796378493309,
+            -0.01246598083525896,
+            0.03186313435435295,
+            0.011149203404784203,
+            -0.021401260048151016,
+            -0.004189090803265572,
+            -0.010577634908258915,
+            0.009304266422986984,
+            0.007122900802642107,
+            -0.014101101085543633,
+            -0.02857842482626438,
+            -0.014933131635189056,
+            -0.004286763723939657,
+            0.010360583662986755,
+            -0.06262654066085815,
+            -0.01440497301518917,
+            -0.018883464857935905,
+            -0.010816390626132488,
+            0.0036555060651153326,
+            0.01889793574810028,
+            -0.009123390540480614,
+            -0.0029772205743938684,
+            0.0153382932767272,
+            0.05191867798566818,
+            -0.008566292002797127,
+            0.00704693328589201,
+            0.000250739511102438,
+            0.01034611277282238,
+            -0.01640908047556877,
+            0.03507549315690994,
+            -0.013544002547860146,
+            -0.048561617732048035,
+            0.008768873289227486,
+            -0.025293715298175812,
+            0.01863747462630272,
+            0.047230370342731476,
+            -0.023296842351555824,
+            -0.03733282908797264,
+            0.020171303302049637,
+            0.009297031909227371,
+            -0.02934533916413784,
+            -0.00019738104310818017,
+            -0.008226245641708374,
+            -0.008132189512252808,
+            -0.021068448200821877,
+            -0.004785981960594654,
+            0.04636216163635254,
+            -0.008537352085113525,
+            0.025742286816239357,
+            0.03394682705402374,
+            0.017016824334859848,
+            -0.03134221211075783,
+            0.02562652714550495,
+            0.005017503164708614,
+            -0.00505729578435421,
+            0.002235628664493561,
+            0.04031366482377052,
+            -0.02941768988966942,
+            -0.012328515760600567,
+            -0.030358245596289635,
+            -0.016032857820391655,
+            -0.03883771598339081,
+            -0.0005697597516700625,
+            0.0024888552725315094,
+            0.008725463412702084,
+            -0.03160267323255539,
+            -0.032904982566833496,
+            0.031081749126315117,
+            0.03704342618584633,
+            -0.01427474245429039,
+            -0.005390108097344637,
+            -0.010953856632113457,
+            -0.005194761790335178,
+            0.006739443633705378,
+            -0.013175015337765217,
+            -0.001295977272093296,
+            -0.00739421509206295,
+            0.0142240971326828,
+            0.0035343191120773554,
+            0.0071120485663414,
+            -0.05504421517252922,
+            -0.018116550520062447,
+            0.018623003736138344,
+            0.0025792932137846947,
+            0.00119197356980294,
+            -0.004453169647604227,
+            -0.0051332637667655945,
+            0.00782831758260727,
+            -0.026104038581252098,
+            0.02318108081817627,
+            0.022211585193872452,
+            0.005509485956281424,
+            -0.002067413879558444,
+            0.007075873203575611,
+            0.03273133933544159,
+            0.030184604227542877,
+            -0.014079395681619644,
+            -0.015468524768948555,
+            0.03154479339718819,
+            -0.0019100516801699996,
+            0.014614789746701717,
+            -0.0023495806381106377,
+            0.01568557508289814,
+            -1.6335372492903844e-05,
+            -0.01853618398308754,
+            -0.008204540237784386,
+            -0.01622096821665764,
+            0.005824210587888956,
+            0.03131327033042908,
+            -0.00011587375774979591,
+            0.01953461952507496,
+            0.015092302113771439,
+            0.017089175060391426,
+            0.013847874477505684,
+            -0.009528553113341331,
+            -0.018926875665783882,
+            0.01794290915131569,
+            0.0029537067748606205,
+            0.026104038581252098,
+            0.007093960884958506,
+            -0.0039394814521074295,
+            -0.008906339295208454,
+            -0.011409664526581764,
+            -0.005035590846091509,
+            0.009246386587619781,
+            0.02125656045973301,
+            0.03380212560296059,
+            -0.008117719553411007,
+            -0.02716035582125187,
+            -0.01501995138823986,
+            0.021430201828479767,
+            0.02138679102063179,
+            -0.01907157711684704,
+            -0.021459141746163368,
+            0.01740751601755619,
+            -0.014130041003227234,
+            0.015931567177176476,
+            0.0019932547584176064,
+            0.003203315893188119,
+            -0.010903211310505867,
+            0.07107707113027573,
+            -0.02759445831179619,
+            -0.004398907069116831,
+            -0.02798515185713768,
+            -0.003205124521628022,
+            0.006652622949331999,
+            -0.014781195670366287,
+            -0.0013737539993599057,
+            0.018623003736138344,
+            -0.003024248406291008,
+            0.022399697452783585,
+            0.01601838879287243,
+            -0.012017408385872841,
+            -0.00755338603630662,
+            -0.028071971610188484,
+            0.027174826711416245,
+            -0.008660348132252693,
+            0.0037405178882181644,
+            -0.0038309558294713497,
+            0.011359019204974174,
+            0.03713024780154228,
+            0.024989843368530273,
+            0.013363126665353775,
+            0.01740751601755619,
+            -0.010671690106391907,
+            0.013442711904644966,
+            0.012856673449277878,
+            0.014665434136986732,
+            -0.026017218828201294,
+            -0.015092302113771439,
+            0.001866641454398632,
+            0.006428336724638939,
+            0.008146659471094608,
+            0.005661421921104193,
+            -0.001539255608804524,
+            0.012762618251144886,
+            -0.006446424406021833,
+            0.00756062101572752,
+            -0.0009215637692250311,
+            -0.0204896442592144,
+            -0.009854130446910858,
+            0.025091134011745453,
+            0.0003529344976413995,
+            0.024179518222808838,
+            0.0011567026376724243,
+            -0.005542044062167406,
+            0.002711332868784666,
+            -0.02061987668275833,
+            0.00019195476488675922,
+            -0.01714705489575863,
+            -0.0063849263824522495,
+            0.0036790198646485806,
+            -0.01648143120110035,
+            -0.01195229310542345,
+            -0.01170630194246769,
+            0.000591917079873383,
+            0.0003979274188168347,
+            0.0056288642808794975,
+            0.00756062101572752,
+            -0.0036844462156295776,
+            0.009058275260031223,
+            -0.01016523689031601,
+            0.01011459156870842,
+            -0.007850022986531258,
+            -0.012871143408119678,
+            -0.007705322001129389,
+            0.011402430012822151,
+            0.009427262470126152,
+            -0.004492962267249823,
+            0.01160501129925251,
+            -0.016365669667720795,
+            0.003754987847059965,
+            -0.010924916714429855,
+            -0.025915928184986115,
+            -0.006124464794993401,
+            -0.02739187702536583,
+            0.004521902650594711,
+            -0.013131605461239815,
+            0.022269466891884804,
+            -0.024266337975859642,
+            0.027232706546783447,
+            0.007770437281578779,
+            -0.009297031909227371,
+            -0.00021286855917423964,
+            -0.03620416298508644,
+            0.011395194567739964,
+            -0.01846383325755596,
+            0.0028940176125615835,
+            0.012299574911594391,
+            -0.003438454819843173,
+            0.0056071593426167965,
+            -0.008645878173410892,
+            -0.010121827013790607,
+            -0.003353442996740341,
+            -0.021531492471694946,
+            0.004098652396351099,
+            0.006768384017050266,
+            -0.0031906545627862215,
+            -0.01499101147055626,
+            -0.01727728545665741,
+            -0.023528363555669785,
+            0.017812678590416908,
+            0.022891679778695107,
+            -0.021010568365454674,
+            -0.020518586039543152,
+            -0.005314140114933252,
+            0.0030712762381881475,
+            -0.02708800509572029,
+            0.0033064151648432016,
+            -0.008710993453860283,
+            0.01016523689031601,
+            -0.009065510705113411,
+            0.010606574825942516,
+            -0.003870748681947589,
+            0.020070012658834457,
+            0.018377011641860008,
+            0.02351389452815056,
+            -0.03620416298508644,
+            0.0017409324645996094,
+            0.010599339380860329,
+            -0.009246386587619781,
+            -0.014014280401170254,
+            0.021242089569568634,
+            -0.010403993539512157,
+            -0.0020999715197831392,
+            0.03090810962021351,
+            0.000624926935415715,
+            -0.008964220061898232,
+            0.008472236804664135,
+            -0.0017662551254034042,
+            -0.021531492471694946,
+            0.026436852291226387,
+            0.010830861516296864,
+            0.00905104074627161,
+            0.022573338821530342,
+            -0.003820103360339999,
+            0.005397343076765537,
+            -0.017494337633252144,
+            0.020981628447771072,
+            -0.0036772112362086773,
+            -0.009072745218873024,
+            0.019679320976138115,
+            0.026089569553732872,
+            -0.008233480155467987,
+            0.011510955169796944,
+            -0.03249981999397278,
+            0.0012028261553496122,
+            0.03892453759908676,
+            0.01255280151963234,
+            0.0009378425893373787,
+            -0.014281976968050003,
+            -0.012075289152562618,
+            0.013449947349727154,
+            0.021198678761720657,
+            -0.004442316945642233,
+            -0.02875206619501114,
+            0.008009194396436214,
+            -0.011257728561758995,
+            0.03322332352399826,
+            0.018594063818454742,
+            -0.00976730976253748,
+            0.027464227750897408,
+            -0.004619575571268797,
+            -0.01830466091632843,
+            0.01681424304842949,
+            -0.005379255395382643,
+            -0.004243353381752968,
+            0.011843767017126083,
+            0.0036482710856944323,
+            0.03044506534934044,
+            0.006837116554379463,
+            -0.0055709839798510075,
+            -0.0014605745673179626,
+            0.01661166176199913,
+            -0.002420122269541025,
+            -0.024165047332644463,
+            0.0048510972410440445,
+            0.010288232937455177,
+            -0.0007777672726660967,
+            -0.012588976882398129,
+            0.011966763064265251,
+            -0.014694374985992908,
+            -0.0030423360876739025,
+            0.0008921714033931494,
+            0.006099142134189606,
+            -0.00047615633229725063,
+            -0.0233113132417202,
+            -0.006576654966920614,
+            -0.03449669107794762,
+            0.028708655387163162,
+            0.009441732428967953,
+            0.007538916077464819,
+            0.013218426145613194,
+            0.005419048015028238,
+            0.008624172769486904,
+            -0.0034185582771897316,
+            -0.007850022986531258,
+            0.004485727287828922,
+            -0.009919245727360249,
+            -0.02831796370446682,
+            -0.03163161501288414,
+            0.016394609585404396,
+            -0.021299969404935837,
+            -0.0066236830316483974,
+            0.0028596511110663414,
+            -0.02433868870139122,
+            -0.02205241471529007,
+            0.003123730421066284,
+            0.01661166176199913,
+            -0.022139234468340874,
+            0.01664060167968273,
+            -0.001658633933402598,
+            -0.007350804749876261,
+            0.023730944842100143,
+            -0.008023664355278015,
+            -0.001483184052631259,
+            -0.0116701265797019,
+            -0.02669731341302395,
+            0.013666998594999313,
+            0.01283496804535389,
+            -0.0002509655896574259,
+            0.009608138352632523,
+            0.0012118698796257377,
+            0.010707865469157696,
+            0.009116156026721,
+            -0.01830466091632843,
+            0.01273367740213871,
+            -0.02716035582125187,
+            0.03345484286546707,
+            0.01910051703453064,
+            -0.028129851445555687,
+            0.0352780744433403,
+            -0.008710993453860283,
+            0.015859216451644897,
+            -0.0035288927610963583,
+            0.01400704588741064,
+            0.010845331475138664,
+            0.017393046990036964,
+            0.01601838879287243,
+            0.01399257592856884,
+            0.0041529154404997826,
+            0.01920180767774582,
+            -0.006088289897888899,
+            0.023065321147441864,
+            0.013949165120720863,
+            -3.0748939025215805e-05,
+            0.013580177910625935,
+            0.003338972805067897,
+            -0.018449362367391586,
+            -0.0015220724744722247,
+            -0.021068448200821877,
+            -0.007654676679521799,
+            0.0030622323974967003,
+            -0.0204317644238472,
+            -0.004561695270240307,
+            -0.030821288004517555,
+            0.006598360370844603,
+            0.0016595382476225495,
+            -0.012458746321499348,
+            -0.009362147189676762,
+            0.010765746235847473,
+            0.002649834845215082,
+            0.022775918245315552,
+            -0.011908882297575474,
+            0.02241416648030281,
+            0.008935279212892056,
+            -0.002969985594972968,
+            0.01049804873764515,
+            -0.02457020990550518,
+            0.014499028213322163,
+            0.0017535937950015068,
+            0.022472048178315163,
+            0.021632781252264977,
+            0.0005114271771162748,
+            0.010150766931474209,
+            0.0016197455115616322,
+            0.015352764166891575,
+            -0.015164652839303017,
+            -0.012429806403815746,
+            -0.00489088986068964,
+            -0.015584285371005535,
+            0.05814081430435181,
+            -0.02171960286796093,
+            -0.012314044870436192,
+            -0.017682448029518127,
+            -0.002385755768045783,
+            0.009311501868069172,
+            -0.030126724392175674,
+            -0.018651943653821945,
+            -0.022168176248669624,
+            -0.006797323934733868,
+            -0.02038835547864437,
+            -0.004579782951623201,
+            0.021603841334581375,
+            -0.0011901648249477148,
+            0.007850022986531258,
+            -0.01953461952507496,
+            -0.0068009416572749615,
+            -0.01409386657178402,
+            -0.003257578704506159,
+            -0.014535203576087952,
+            -0.012024643830955029,
+            -0.0042144134640693665,
+            0.021545961499214172,
+            0.01426027249544859,
+            -0.010859801433980465,
+            0.006486217025667429,
+            0.013580177910625935,
+            -0.012010172940790653,
+            0.003698916407302022,
+            0.01714705489575863,
+            0.011691831052303314,
+            -0.00944896787405014,
+            0.01007841620594263,
+            -0.0026842013467103243,
+            -0.005419048015028238,
+            0.004963240120559931,
+            0.0023749032989144325,
+            0.03244193643331528,
+            0.010512519627809525,
+            -0.04520455747842789,
+            -0.00811048410832882,
+            0.010968326590955257,
+            0.006844351999461651,
+            0.0019697407260537148,
+            0.005400960333645344,
+            -0.015309353359043598,
+            0.004250588361173868,
+            -0.03093704953789711,
+            -0.0001339613663731143,
+            -0.0026263210456818342,
+            -0.008645878173410892,
+            0.006634535733610392,
+            -0.006591125391423702,
+            0.01177141722291708,
+            0.01923074759542942,
+            -0.003143626730889082,
+            0.01457137893885374,
+            0.0030712762381881475,
+            0.013218426145613194,
+            -0.03281816095113754,
+            0.008739933371543884,
+            -0.011380724608898163,
+            0.009029335342347622,
+            -0.012328515760600567,
+            -0.035856880247592926,
+            -0.010867035947740078,
+            0.015728985890746117,
+            -0.002018577419221401,
+            -0.005976146552711725,
+            -0.0021922183223068714,
+            0.02626321092247963,
+            0.019780611619353294,
+            -0.0014280168106779456,
+            -0.0065621850080788136,
+            0.030589766800403595,
+            -0.0006945642526261508,
+            0.0002321997017133981,
+            0.0003368817560840398,
+            0.004934300202876329,
+            -0.011829297058284283,
+            0.005697597283869982,
+            -0.01740751601755619,
+            0.0067177386954426765,
+            -0.008052604272961617,
+            0.026899894699454308,
+            -0.0019064341904595494,
+            -0.014614789746701717,
+            0.0193609781563282,
+            -0.032876040786504745,
+            -0.004398907069116831,
+            -0.006478982046246529,
+            -0.00466660363599658,
+            0.021806422621011734,
+            -0.0019480356713756919,
+            0.025076663121581078,
+            -0.05145563557744026,
+            -0.0014379650820046663,
+            -0.004051624797284603,
+            -0.016365669667720795,
+            0.004283146001398563,
+            -0.02135785110294819,
+            0.014130041003227234,
+            -0.0030459535773843527,
+            -0.015757925808429718,
+            0.002360433107241988,
+            -0.008291360922157764,
+            -0.008595232851803303,
+            0.013175015337765217,
+            0.01578686572611332,
+            -0.003349825507029891,
+            0.014253037050366402,
+            0.0039539518766105175,
+            0.015265943482518196,
+            0.017494337633252144,
+            -0.013066490180790424,
+            -0.013558472506701946,
+            0.01178588718175888,
+            0.0006185962702147663,
+            0.01697341352701187,
+            0.008320300839841366,
+            -0.009991596452891827,
+            -0.02749316766858101,
+            0.005447988398373127,
+            -0.006714120972901583,
+            -0.0036301834043115377,
+            -0.018492773175239563,
+            -0.016626130789518356,
+            -0.001278794021345675,
+            -0.01668401248753071,
+            0.00461595831438899,
+            -0.023036381229758263,
+            -0.0010617427760735154,
+            0.008812284097075462,
+            -0.02338366210460663,
+            -0.004847479518502951,
+            0.013580177910625935,
+            0.016235439106822014,
+            -0.03247087821364403,
+            0.003870748681947589,
+            -0.0037369003985077143,
+            -0.028129851445555687,
+            -0.0035126139409840107,
+            0.0025720582343637943,
+            -0.013732113875448704,
+            0.01500548142939806,
+            -0.011344549246132374,
+            -0.017103644087910652,
+            -0.03397576883435249,
+            -0.022298406809568405,
+            -0.007010757923126221,
+            0.019158396869897842,
+            0.005462458357214928,
+            0.004178238101303577,
+            0.0027185678482055664,
+            0.0003034196561202407,
+            0.0076691466383636,
+            0.0284481942653656,
+            -0.013724878430366516,
+            -6.6804045673052315e-06,
+            -0.001633311272598803,
+            0.02061987668275833,
+            -0.004373584408313036,
+            0.004113122820854187,
+            0.004605105612426996,
+            0.000946886430028826,
+            0.0033425905276089907,
+            -0.005317757371813059,
+            -0.006938407197594643,
+            -0.010874271392822266,
+            0.010483578778803349,
+            0.024888552725315094,
+            -0.01910051703453064,
+            -0.0012679415522143245,
+            -0.02028706483542919,
+            0.029461100697517395,
+            0.030126724392175674,
+            0.00784278754144907,
+            -0.0409214086830616,
+            -0.0032340646721422672,
+            -0.0030170134268701077,
+            -0.0014325387310236692,
+            -0.02420845814049244,
+            0.01203911378979683,
+            0.006196815520524979,
+            0.009702194482088089,
+            0.018753234297037125,
+            -0.006583890412002802,
+            0.017002353444695473,
+            -0.008573527447879314,
+            -0.005968911573290825,
+            0.001627884921617806,
+            -0.007878962904214859,
+            0.015077832154929638,
+            0.04115293174982071,
+            -0.028838885948061943,
+            0.00650068698450923,
+            -0.007857258431613445,
+            0.022674627602100372,
+            -0.009543023072183132,
+            0.009195741266012192,
+            0.02061987668275833,
+            0.017161523923277855,
+            -0.0051151760853827,
+            0.016958944499492645,
+            0.07582326233386993,
+            -0.003993744496256113,
+            0.022009003907442093,
+            0.008768873289227486,
+            -0.000894884520675987,
+            0.0041167400777339935,
+            -0.019187336787581444,
+            -0.02305085025727749,
+            0.017233874648809433,
+            0.0039756568148732185,
+            -0.004492962267249823,
+            0.011496485210955143,
+            -0.015164652839303017,
+            0.006319811102002859,
+            0.017827149480581284,
+            0.005386490374803543,
+            -0.012639622204005718,
+            -0.006927554961293936,
+            0.022906150668859482,
+            0.019158396869897842,
+            0.0056288642808794975,
+            -0.010823626071214676,
+            0.01454243902117014,
+            -0.005646951962262392,
+            -0.01661166176199913,
+            0.016192028298974037,
+            -0.01691553369164467,
+            0.018782174214720726,
+            -0.0024978991132229567,
+            0.024787262082099915,
+            -0.007640206720679998,
+            -0.010172472335398197,
+            -0.0016088929260149598,
+            0.00032648135675117373,
+            0.01684318296611309,
+            0.019635910168290138,
+            0.0036573149263858795,
+            6.426754407584667e-05,
+            0.002027621027082205,
+            0.012364690192043781,
+            0.0006294488557614386,
+            0.0032431085128337145,
+            0.00500665046274662,
+            0.003143626730889082,
+            -0.012965199537575245,
+            0.023962466046214104,
+            0.006048496812582016,
+            0.008132189512252808,
+            -0.00616064015775919,
+            -0.005618012044578791,
+            0.004894507583230734,
+            -0.009904775768518448,
+            0.00650068698450923,
+            0.001803334802389145,
+            -0.003291945206001401,
+            -0.01261791680008173,
+            0.029721561819314957,
+            -0.021198678761720657,
+            0.014281976968050003,
+            0.01923074759542942,
+            0.008219010196626186,
+            0.00978177972137928,
+            -0.013189485296607018,
+            -0.018362542614340782,
+            0.00761850131675601,
+            -0.00027221854543313384,
+            0.005864003207534552,
+            0.0051115588285028934,
+            0.001834083697758615,
+            0.016597190871834755,
+            -0.00244182744063437,
+            0.005495015997439623,
+            0.013225660659372807,
+            0.007864492945373058,
+            0.005976146552711725,
+            0.016134148463606834,
+            -0.030329305678606033,
+            -0.018753234297037125,
+            0.004315703641623259,
+            0.00516582140699029,
+            -0.0005634290864691138,
+            0.0043554967269301414,
+            0.01006394624710083,
+            0.011279433965682983,
+            -0.008399886079132557,
+            -0.015497464686632156,
+            -0.014817370101809502,
+            0.04184749722480774,
+            0.014347092248499393,
+            -0.020750107243657112,
+            -0.011959527619183064,
+            -0.0021650870330631733,
+            0.003226829692721367,
+            -0.007198868785053492,
+            0.014933131635189056,
+            0.014962071552872658,
+            -0.018955815583467484,
+            -0.02590145915746689,
+            -0.027102475985884666,
+            -0.012704737484455109,
+            0.014310917817056179,
+            0.013688703998923302,
+            0.010288232937455177,
+            0.005809740629047155,
+            0.02112632989883423,
+            -0.014846310950815678,
+            0.008428826928138733,
+            0.012089759111404419,
+            0.0034782474394887686,
+            -0.009427262470126152,
+            -0.00600146921351552,
+            -0.014477323740720749,
+            0.015873687341809273,
+            0.008276890963315964,
+            -0.005802505649626255,
+            -0.01611967757344246,
+            0.0034438809379935265,
+            -0.014918660745024681,
+            -0.010411228984594345,
+            -0.006714120972901583,
+            -0.0021994535345584154,
+            -0.012082523666322231,
+            0.004927065223455429,
+            -0.009745604358613491,
+            -0.007090343162417412,
+            0.0033896181266754866,
+            -0.004171003121882677,
+            -0.005140498746186495,
+            -0.02685648389160633,
+            -0.026914363726973534,
+            -0.003196080680936575,
+            -0.0024038434494286776,
+            -0.00600146921351552,
+            -0.003590390784665942,
+            -0.029359810054302216,
+            -0.0065404800698161125,
+            0.009022099897265434,
+            -0.011851002462208271,
+            0.021068448200821877,
+            0.0029573242645710707,
+            -0.008906339295208454,
+            0.0033896181266754866,
+            -0.003604860743507743,
+            -0.029866263270378113,
+            0.009977125562727451,
+            0.005639716982841492,
+            0.02208135463297367,
+            -0.0182033721357584,
+            0.0015962315956130624,
+            0.0061497874557971954,
+            0.023759884759783745,
+            -0.031718432903289795,
+            -0.015902627259492874,
+            -0.007654676679521799,
+            -0.010975562036037445,
+            -0.006674328353255987,
+            -0.014354327693581581,
+            0.005060913506895304,
+            -0.008117719553411007,
+            -0.025423945859074593,
+            -0.02281932905316353,
+            -0.00650068698450923,
+            -0.008038134314119816,
+            0.022124765440821648,
+            1.9359395082574338e-05,
+            0.03684084489941597,
+            0.008660348132252693,
+            0.003986509516835213,
+            0.010852565988898277,
+            -0.0030133959371596575,
+            -0.007546151056885719,
+            0.011851002462208271,
+            -0.017581157386302948,
+            -0.006873291917145252,
+            0.003925011493265629,
+            0.01943332888185978,
+            -0.007654676679521799,
+            -0.002360433107241988,
+            0.011040677316486835,
+            0.016857653856277466,
+            0.008776108734309673,
+            0.0027438905090093613,
+            -0.007289307191967964,
+            -0.005610776599496603,
+            0.02215370535850525,
+            0.027073536068201065,
+            0.004728101193904877,
+            0.003436645958572626,
+            0.007806612644344568,
+            -0.002843372290953994,
+            -0.006674328353255987,
+            0.001284220372326672,
+            -0.006337898783385754,
+            -0.02085139788687229,
+            -0.0038635137025266886,
+            -0.004225265700370073,
+            -0.0025720582343637943,
+            -0.02205241471529007,
+            -0.01661166176199913,
+            -0.014875250868499279,
+            0.008956984616816044,
+            0.014470088295638561,
+            -0.019578030332922935,
+            0.01211869902908802,
+            0.006062967237085104,
+            0.006569419987499714,
+            -0.006435571704059839,
+            0.01753774657845497,
+            0.042223718017339706,
+            0.00023073007469065487,
+            -0.03906923905014992,
+            -0.01707470417022705,
+            0.0009111633989959955,
+            -0.012494921684265137,
+            -0.022935090586543083,
+            -0.00756062101572752,
+            0.0031617144122719765,
+            -0.024411039426922798,
+            0.023398132994771004,
+            -0.001627884921617806,
+            -0.0009658784256316721,
+            0.004330174066126347,
+            0.017060235142707825,
+            -0.002376711927354336,
+            0.005332227796316147,
+            -0.014535203576087952,
+            0.003442072309553623,
+            -0.01417345181107521,
+            0.010548694059252739,
+            -0.016857653856277466,
+            -0.018984755501151085,
+            0.006804558914154768,
+            -0.002042091218754649,
+            0.008920809254050255,
+            0.022341815754771233,
+            0.02652367204427719,
+            0.01028099749237299,
+            -0.027869390323758125,
+            -0.01523700263351202,
+            -0.011807592585682869,
+            0.0026950540486723185,
+            -0.014021515846252441,
+            -0.007321864832192659,
+            -0.00761850131675601,
+            0.003545171581208706,
+            0.015323823317885399,
+            0.015251473523676395,
+            0.005151351448148489,
+            0.030647646635770798,
+            -3.1540272175334394e-05,
+            0.006804558914154768,
+            -0.003615713445469737,
+            -0.02801409177482128,
+            -0.005939971189945936,
+            0.005708449985831976,
+            0.009991596452891827,
+            -0.013254600577056408,
+            0.007936843670904636,
+            0.014209627173841,
+            -0.016872122883796692,
+            -0.011279433965682983,
+            -0.02716035582125187,
+            0.010440168902277946,
+            -0.002913914155215025,
+            0.03093704953789711,
+            0.020070012658834457,
+            -0.02138679102063179,
+            -0.015439583919942379,
+            -0.02112632989883423,
+            0.018984755501151085,
+            -0.010953856632113457,
+            -0.0019154779147356749,
+            0.007973019033670425,
+            0.019983192905783653,
+            -0.007712556980550289,
+            -0.009094450622797012,
+            -0.0077921426855027676,
+            0.013876814395189285,
+            0.018753234297037125,
+            7.57983943913132e-05,
+            0.0010300894500687718,
+            -0.0096515491604805,
+            0.010780216194689274,
+            -0.009977125562727451,
+            -0.010592104867100716,
+            0.0051115588285028934,
+            -0.004529137630015612,
+            0.019968722015619278,
+            -0.01638014055788517,
+            0.022529928013682365,
+            0.010360583662986755,
+            -0.009202975779771805,
+            -0.0398506224155426,
+            -0.006522392388433218,
+            -0.006005086470395327,
+            -0.001984210917726159,
+            0.011995702981948853,
+            -0.0027131414972245693,
+            -0.01771138794720173,
+            0.008971454575657845,
+            0.000733000400941819,
+            -0.003031483618542552,
+            0.007604031357914209,
+            0.008096014149487019,
+            0.0007470183190889657,
+            0.011684596538543701,
+            -0.005046443548053503,
+            -0.009702194482088089,
+            0.02195112407207489,
+            0.0014090248150750995,
+            -7.969853322720155e-05,
+            0.02002660185098648,
+            0.02251545712351799,
+            0.010982797481119633,
+            -0.00988307036459446,
+            0.0034547336399555206,
+            -0.009825189597904682,
+            -0.007538916077464819,
+            -0.030850227922201157,
+            0.027579989284276962,
+            -0.007929608225822449,
+            0.03258663788437843,
+            0.013232896104454994,
+            -0.004037154838442802,
+            -0.00075018365168944,
+            0.0074593303725123405,
+            -0.005458840634673834,
+            -0.0006244747783057392,
+            0.017450926825404167,
+            -0.0009233725140802562,
+            0.006261930800974369,
+            -0.0032340646721422672,
+            -0.02028706483542919,
+            0.025915928184986115,
+            -0.0022754215169698,
+            0.005220084451138973,
+            0.011966763064265251,
+            0.009087215177714825,
+            0.007057785522192717,
+            -0.01607626862823963,
+            -0.0028071971610188484,
+            -0.03608839958906174,
+            -0.008016428910195827,
+            -0.0021108242217451334,
+            -0.005697597283869982,
+            -0.003261196194216609,
+            -0.0070831081829965115,
+            -0.04691202566027641,
+            -0.004189090803265572,
+            0.008544587530195713,
+            -0.009868600405752659,
+            0.01853618398308754,
+            -0.016828712075948715,
+            -0.017957380041480064,
+            -0.02115526981651783,
+            0.007195251528173685,
+            0.02002660185098648,
+            0.013587413355708122,
+            -0.003053188556805253,
+            -0.017986319959163666,
+            0.018550653010606766,
+            -0.011800357140600681,
+            -0.008211774751543999,
+            -0.01635120064020157,
+            -0.020055541768670082,
+            0.014549673534929752,
+            0.016105208545923233,
+            -0.011908882297575474,
+            -0.0032431085128337145,
+            -0.007604031357914209,
+            0.013797229155898094,
+            -0.01720493473112583,
+            -0.008327536284923553,
+            0.008096014149487019,
+            0.0018738764338195324,
+            0.012342985719442368,
+            -0.019172867760062218,
+            0.007108430843800306,
+            -0.015801336616277695,
+            0.009043805301189423,
+            -0.01789950020611286,
+            -0.013218426145613194,
+            0.002179556991904974,
+            0.0010237587848678231,
+            0.039705920964479446,
+            0.022920619696378708,
+            -0.027406347915530205,
+            0.01846383325755596,
+            0.004572547972202301,
+            0.016336729750037193,
+            -0.0031038338784128428,
+            0.01467266958206892,
+            -0.01720493473112583,
+            0.016655072569847107,
+            -0.007216956466436386,
+            0.01761009730398655,
+            0.003404088318347931,
+            0.03293392062187195,
+            -0.017566686496138573,
+            0.010432933457195759,
+            0.0015681958757340908,
+            -0.004634045995771885,
+            -0.00705416826531291,
+            0.016785303130745888,
+            0.010411228984594345,
+            0.014520733617246151,
+            0.00616787513718009,
+            -0.023600714281201363,
+            -0.009608138352632523,
+            0.004091417416930199,
+            0.020475175231695175,
+            0.014781195670366287,
+            -0.00349814398214221,
+            -0.0036573149263858795,
+            0.002850607503205538,
+            -0.01028099749237299,
+            0.0096660191193223,
+            -0.013334186747670174,
+            0.007596796378493309,
+            -0.008493942208588123,
+            0.0077921426855027676,
+            -0.008407121524214745,
+            0.0007913329754956067,
+            0.021473610773682594,
+            -0.03510443493723869,
+            0.01510677207261324,
+            -0.003747752867639065,
+            -0.014231331646442413,
+            0.027044596150517464,
+            -0.017393046990036964,
+            -0.005845915526151657,
+            -0.008964220061898232,
+            -0.0008143946761265397,
+            0.014216861687600613,
+            -0.000501479022204876,
+            0.0035704942420125008,
+            -0.006768384017050266,
+            -0.006949259899556637,
+            0.001438869396224618,
+            0.0091016860678792,
+            -0.011851002462208271,
+            0.006677945610135794,
+            -0.004026302136480808,
+            0.021444670855998993,
+            0.008732697926461697,
+            0.006938407197594643,
+            0.00020653790852520615,
+            -0.002971794456243515,
+            0.006062967237085104,
+            0.02182089351117611,
+            0.0048149218782782555,
+            0.023166611790657043,
+            -0.023658594116568565,
+            0.013486122712492943,
+            -0.00016154497279785573,
+            -0.008096014149487019,
+            -0.01694447360932827,
+            -0.0017870558658614755,
+            0.02543841488659382,
+            0.0020475175697356462,
+            -0.02400587685406208,
+            0.00033281202195212245,
+            0.010143532417714596,
+            -0.0014524351572617888,
+            0.01500548142939806,
+            0.01784161850810051,
+            -0.0014343474758788943,
+            0.007282072212547064,
+            -0.002738464158028364,
+            -0.024092696607112885,
+            -0.01807313971221447,
+            -0.003783928230404854,
+            -0.016785303130745888,
+            0.02271803840994835,
+            0.01691553369164467,
+            0.0051694391295313835,
+            -0.0038526610005646944,
+            0.0017517850501462817,
+            0.01522253267467022,
+            0.020041072741150856,
+            0.0056288642808794975,
+            0.00939832255244255,
+            0.0011594158131629229,
+            -0.010309938341379166,
+            0.0016504944069311023,
+            -0.008052604272961617,
+            -0.008797814138233662,
+            -0.012357455678284168,
+            0.009275326505303383,
+            -0.006240225397050381,
+            -0.013449947349727154,
+            -0.01437603309750557,
+            -0.004619575571268797,
+            0.016090737655758858,
+            -0.007546151056885719,
+            -0.026870954781770706,
+            0.0036265659146010876,
+            0.018377011641860008,
+            0.011612245813012123,
+            0.015642166137695312,
+            -0.009087215177714825,
+            -0.011185378767549992,
+            0.007929608225822449,
+            -0.019476739689707756,
+            -0.0029193402733653784,
+            -0.010729570873081684,
+            0.003085746429860592,
+            -0.021545961499214172,
+            0.02937427908182144,
+            0.008472236804664135,
+            0.027739159762859344,
+            0.024729380384087563,
+            -0.01150372065603733,
+            0.004478492308408022,
+            -0.003693490056321025,
+            -0.0002373998868279159,
+            -0.008956984616816044,
+            3.804050720646046e-05,
+            0.0006158831529319286,
+            0.021545961499214172,
+            0.013175015337765217,
+            0.01245151087641716,
+            0.001984210917726159,
+            -0.0009134243591688573,
+            0.002418313641101122,
+            -7.246348832268268e-05,
+            -0.010042241774499416,
+            -0.017856089398264885,
+            0.005277964752167463,
+            0.017002353444695473,
+            -0.004178238101303577,
+            -0.005639716982841492,
+            0.009969891048967838,
+            0.0003594912705011666,
+            -0.0029555156361311674,
+            -0.011612245813012123,
+            -0.011858237907290459,
+            0.018159961327910423,
+            0.026509201154112816,
+            -0.008609702810645103,
+            -0.006547715049237013,
+            0.02115526981651783,
+            -0.001689382828772068,
+            0.03215253725647926,
+            -0.011098558083176613,
+            -0.010621044784784317,
+            -0.0014596701366826892,
+            -0.005212849471718073,
+            -0.01730622537434101,
+            -0.0238756462931633,
+            0.01255280151963234,
+            0.0035397454630583525,
+            -0.0035361277405172586,
+            -0.03047400526702404,
+            0.015034422278404236,
+            0.006135317496955395,
+            0.032876040786504745,
+            -0.001438869396224618,
+            -0.006460894364863634,
+            -0.000922015926335007,
+            -0.0010653602657839656,
+            -0.0013864153297618032,
+            -0.013811699114739895,
+            0.00905104074627161,
+            -0.0032684311736375093,
+            0.0034746299497783184,
+            -0.0020312387496232986,
+            -0.025554176419973373,
+            -0.005693979561328888,
+            0.002465341240167618,
+            -0.00727121951058507,
+            -0.00756062101572752,
+            -0.013131605461239815,
+            -0.0012145830551162362,
+            0.017335165292024612,
+            0.004416994284838438,
+            -0.0010997267672792077,
+            0.008674818091094494,
+            -0.0077559673227369785,
+            -0.008493942208588123,
+            -0.0060665844939649105,
+            -0.01720493473112583,
+            0.01830466091632843,
+            -0.02105397917330265,
+            0.0017563069704920053,
+            0.026350030675530434,
+            -0.004218030720949173,
+            -0.003295562695711851,
+            0.018695354461669922,
+            0.0091016860678792,
+            0.008587997406721115,
+            -0.0006027696072123945,
+            -0.002096354030072689,
+            0.00361028709448874,
+            0.01177141722291708,
+            0.011590540409088135,
+            -0.002790918340906501,
+            -0.00232968432828784,
+            -0.02089480683207512,
+            0.008602467365562916,
+            -0.01889793574810028,
+            0.024657031521201134,
+            0.02061987668275833,
+            -0.0015374468639492989,
+            0.01187270786613226,
+            0.02646579220890999,
+            0.008349240757524967,
+            0.01488972082734108,
+            0.000975826580543071,
+            0.005737389903515577,
+            -0.0025431180838495493,
+            -0.000960452132858336,
+            -0.023253431543707848,
+            0.006135317496955395,
+            0.005831445567309856,
+            0.004308468662202358,
+            -0.0035940082743763924,
+            -0.01976614072918892,
+            0.017421986907720566,
+            0.014151746407151222,
+            -0.018217841163277626,
+            0.021936653181910515,
+            -0.005737389903515577,
+            0.019838491454720497,
+            0.017103644087910652,
+            0.011959527619183064,
+            -0.007311012130230665,
+            -0.014166216365993023,
+            0.013558472506701946,
+            -0.02085139788687229,
+            -0.0024092698004096746,
+            0.005632482003420591,
+            -0.01727728545665741,
+            0.002420122269541025,
+            -0.02025812305510044,
+            -0.004076947458088398,
+            -0.008334770798683167,
+            0.006956494878977537,
+            -0.008327536284923553,
+            0.00628363573923707,
+            0.00583506328985095,
+            -0.0022844651248306036,
+            -0.001457861391827464,
+            -0.03637780249118805,
+            -0.005845915526151657,
+            0.021777482703328133,
+            0.0026028072461485863,
+            -0.04057412967085838,
+            0.021097388118505478,
+            0.004811304621398449,
+            -0.01684318296611309,
+            -0.015280413441359997,
+            -0.021936653181910515,
+            -0.014064925722777843,
+            -0.007625736761838198,
+            0.0037405178882181644,
+            -0.02102503925561905,
+            -0.0029048703145235777,
+            0.011807592585682869,
+            -0.01902816630899906,
+            -0.0102158822119236,
+            -0.008124954998493195,
+            -0.0031056427396833897,
+            0.019086046144366264,
+            -0.011923353187739849,
+            -0.00951408315449953,
+            0.025814637541770935,
+            0.006916702259331942,
+            -0.0058061229065060616,
+            -0.015323823317885399,
+            -0.016134148463606834,
+            -0.0071409884840250015,
+            0.013804464600980282,
+            -0.0025467355735599995,
+            -0.003921393770724535,
+            -0.030155664309859276,
+            -0.0028705038130283356,
+            0.006949259899556637,
+            0.019476739689707756,
+            -0.00043726799776777625,
+            0.002132529392838478,
+            0.009072745218873024,
+            0.00794407818466425,
+            -0.014535203576087952,
+            -7.447008556482615e-06,
+            0.009072745218873024,
+            0.012494921684265137,
+            -0.00015498821449000388,
+            0.019722729921340942,
+            -0.01833360269665718,
+            0.004937917459756136,
+            -0.008718227967619896,
+            0.005220084451138973,
+            -0.01860853284597397,
+            -0.006439189426600933,
+            -0.01627884991466999,
+            0.008052604272961617,
+            0.009275326505303383,
+            -0.018319131806492805,
+            -0.0019172866595909,
+            -0.006866056937724352,
+            0.017856089398264885,
+            -0.027174826711416245,
+            0.001350240083411336,
+            -0.017494337633252144,
+            -0.0015817615203559399,
+            -0.026350030675530434,
+            0.00311287771910429,
+            0.006471747066825628,
+            0.004561695270240307,
+            0.0003723786794580519,
+            -0.007850022986531258,
+            -0.02457020990550518,
+            0.01198846846818924,
+            -0.002181365853175521,
+            -0.001573622110299766,
+            -0.010526989586651325,
+            -0.0017327930545434356,
+            0.011966763064265251,
+            0.016163088381290436,
+            -0.0042361184023320675,
+            -0.014665434136986732,
+            0.01843489333987236,
+            -0.016105208545923233,
+            -0.019549088552594185,
+            -0.015179122798144817,
+            -0.011185378767549992,
+            -0.004644898232072592,
+            0.012639622204005718,
+            0.013804464600980282,
+            0.020779047161340714,
+            0.00961537379771471,
+            0.010765746235847473,
+            -0.007358039729297161,
+            0.00016572773165535182,
+            -0.006638152990490198,
+            -0.0015085067134350538,
+            0.012574506923556328,
+            -0.0054299007169902325,
+            0.02613298036158085,
+            -0.015410644002258778,
+            0.0015274987090379,
+            -0.00025797454873099923,
+            0.0069637298583984375,
+            -0.007354422472417355,
+            0.00015397078823298216,
+            -0.010411228984594345,
+            0.008392651565372944,
+            -0.03455457091331482,
+            -0.006258313078433275,
+            -0.0033263114746659994,
+            -0.011185378767549992,
+            0.03157373145222664,
+            0.0003567781241144985,
+            0.00756062101572752,
+            0.0048149218782782555,
+            0.011395194567739964,
+            -0.008298595435917377,
+            0.016061797738075256,
+            -0.017783738672733307,
+            0.001122336252592504,
+            0.015526404604315758,
+            0.007770437281578779,
+            -0.0012932640966027975,
+            0.006775618996471167,
+            -0.006041261833161116,
+            0.009123390540480614,
+            -0.0014108336763456464,
+            -0.014361563138663769,
+            -0.01450626365840435,
+            0.00017024963744916022,
+            0.013146075420081615,
+            -0.01727728545665741,
+            -0.006670710630714893,
+            -0.006511539686471224,
+            0.01394193060696125,
+            -0.01409386657178402,
+            0.026914363726973534,
+            -0.001209156820550561,
+            -0.017465395852923393,
+            0.01238639559596777,
+            0.030589766800403595,
+            -0.014383267611265182,
+            0.02059093490242958,
+            -0.007546151056885719,
+            -0.003906923811882734,
+            0.004988562781363726,
+            -0.0015482994494959712,
+            -0.0004268675984349102,
+            0.018217841163277626,
+            -0.008204540237784386,
+            -0.007777672726660967,
+            0.011207083240151405,
+            -0.02218264527618885,
+            0.016090737655758858,
+            -0.01584474742412567,
+            0.0010300894500687718,
+            0.004858332220464945,
+            0.0031110690906643867,
+            0.02028706483542919,
+            0.037622228264808655,
+            -0.00014944888243917376,
+            -0.007878962904214859,
+            -0.012234459631145,
+            0.02089480683207512,
+            0.005871238186955452,
+            0.005853150971233845,
+            -0.004388054367154837,
+            0.002447253791615367,
+            -0.01740751601755619,
+            -0.013681468553841114,
+            0.0076691466383636,
+            -0.010606574825942516,
+            -0.03313650190830231,
+            -0.015077832154929638,
+            0.0051115588285028934,
+            0.0063993968069553375,
+            0.018579592928290367,
+            0.006204050499945879,
+            -0.003986509516835213,
+            0.020967157557606697,
+            -0.003581346943974495,
+            0.011496485210955143,
+            -0.008414356037974358,
+            -0.002881356282159686,
+            -0.004876419901847839,
+            -0.023354722186923027,
+            0.024729380384087563,
+            -0.027637869119644165,
+            -0.003433028468862176,
+            -0.011337313801050186,
+            -0.000320150691550225,
+            0.005914648529142141,
+            -0.0031291565392166376,
+            0.01470884494483471,
+            0.007466565817594528,
+            0.016958944499492645,
+            0.01737857609987259,
+            -0.03166055306792259,
+            0.01707470417022705,
+            0.019303098320961,
+            -0.0020149596966803074,
+            -0.007538916077464819,
+            -0.0032449173741042614,
+            -0.012017408385872841,
+            -0.012762618251144886,
+            0.014296446926891804,
+            -0.0004528685531113297,
+            -0.006073819473385811,
+            -0.005654186941683292,
+            0.02287720888853073,
+            -0.007784907706081867,
+            0.02351389452815056,
+            -0.0046955435536801815,
+            0.0027800656389445066,
+            0.02652367204427719,
+            0.007325482089072466,
+            0.020417295396327972,
+            -0.016452491283416748,
+            -0.02271803840994835,
+            -0.007159076165407896,
+            -0.005249024368822575,
+            0.006916702259331942,
+            -0.005715684965252876,
+            0.00944896787405014,
+            -0.021647252142429352,
+            -0.020402824506163597,
+            0.01850724220275879,
+            -0.0199253112077713,
+            0.010736805386841297,
+            0.0061027598567306995,
+            0.01899922639131546,
+            -0.004000979475677013,
+            0.012531096115708351,
+            -0.010382288135588169,
+            -0.01604732871055603,
+            0.011424134485423565,
+            0.0062221381813287735,
+            -0.006652622949331999,
+            0.003928629215806723,
+            -0.011192613281309605,
+            -0.008790578693151474,
+            -0.01638014055788517,
+            -0.0010120017686858773,
+            -0.010961092077195644,
+            0.008161129429936409,
+            0.01787055842578411,
+            0.004959622863680124,
+            0.014745020307600498,
+            0.0032412998843938112,
+            -0.01250215619802475,
+            -0.010273762978613377,
+            0.0035940082743763924,
+            -0.010932152159512043,
+            -0.003747752867639065,
+            -0.022920619696378708,
+            0.0007212434429675341,
+            -0.014361563138663769,
+            -0.01175694726407528,
+            -0.009289796464145184,
+            0.0048510972410440445,
+            0.015497464686632156,
+            -0.018825585022568703,
+            -0.008457766845822334,
+            -0.008703758008778095,
+            0.0023170229978859425,
+            0.007057785522192717,
+            -0.016322258859872818,
+            0.008682052604854107,
+            0.02931639924645424,
+            -0.0034402634482830763,
+            -0.014795665629208088,
+            0.004442316945642233,
+            0.005614394322037697,
+            -0.008378181606531143,
+            0.018058670684695244,
+            0.011395194567739964,
+            -0.013826169073581696,
+            -0.004275911021977663,
+            0.013399302028119564,
+            -0.00019625057757366449,
+            -0.0005494111683219671,
+            0.0009396513924002647,
+            -0.014412208460271358,
+            0.03157373145222664,
+            0.0022808476351201534,
+            0.021082919090986252,
+            -8.054639329202473e-05,
+            -0.006601977627724409,
+            -0.006920319981873035,
+            -0.002239246154204011,
+            -0.006894997321069241,
+            -0.0034493072889745235,
+            -0.0011214318219572306,
+            -0.005668656900525093,
+            0.016307789832353592,
+            0.00704693328589201,
+            -0.002043900080025196,
+            0.0006629109266214073,
+            -0.016655072569847107,
+            0.00517305638641119,
+            -0.005249024368822575,
+            -0.005151351448148489,
+            -0.01804419979453087,
+            0.031284332275390625,
+            0.01261791680008173,
+            0.006728590931743383,
+            -0.006370456423610449,
+            -0.020301533862948418,
+            -0.00013983984536025673,
+            0.030358245596289635,
+            -0.011236024089157581,
+            0.004091417416930199,
+            0.01488972082734108,
+            -0.005162204150110483,
+            -0.002150616841390729,
+            -0.014281976968050003,
+            -0.0028053882997483015,
+            -0.027768099680542946,
+            0.0023061702959239483,
+            -0.006775618996471167,
+            0.006323428824543953,
+            -0.002731229178607464,
+            -0.013630823232233524,
+            -0.015902627259492874,
+            -0.0204607043415308,
+            -0.0062438431195914745,
+            -0.009275326505303383,
+            -0.009702194482088089,
+            -0.010411228984594345,
+            -0.025062192231416702,
+            0.021314440295100212,
+            -0.011843767017126083,
+            -0.0004286763723939657,
+            -0.024599149823188782,
+            0.0182033721357584,
+            -0.01034611277282238,
+            -0.015642166137695312,
+            0.0066453879699110985,
+            0.009789015166461468,
+            0.016192028298974037,
+            0.006124464794993401,
+            -0.004977710545063019,
+            -0.004181855358183384,
+            0.014064925722777843,
+            -0.0066236830316483974,
+            -0.005328610073775053,
+            -0.002212114864960313,
+            -0.021893244236707687,
+            -0.007202486507594585,
+            -0.010367818176746368,
+            -0.01177141722291708,
+            -0.005853150971233845,
+            0.010801920667290688,
+            0.015150182880461216,
+            0.0013104473473504186,
+            -0.01761009730398655,
+            -0.0054877810180187225,
+            0.0034113232977688313,
+            0.010085651651024818,
+            -0.023195551708340645,
+            0.008378181606531143,
+            -0.008356476202607155,
+            0.013732113875448704,
+            0.030792348086833954,
+            0.00577718298882246,
+            -0.004720866214483976,
+            0.0008682053303346038,
+            0.02102503925561905,
+            0.0030423360876739025,
+            0.00938385259360075,
+            -0.030358245596289635,
+            -0.022906150668859482,
+            -0.003950334154069424,
+            0.004923447500914335,
+            -0.005932736210525036,
+            0.030358245596289635,
+            0.008573527447879314,
+            0.01840595155954361,
+            0.006612830329686403,
+            0.023484952747821808,
+            0.006620065309107304,
+            0.022558867931365967,
+            0.011568835936486721,
+            0.003633800894021988,
+            -0.007517211139202118,
+            -0.024251868948340416,
+            -0.02944662980735302,
+            0.007690852042287588,
+            0.016568250954151154,
+            0.02115526981651783,
+            -0.03553853556513786,
+            0.023600714281201363,
+            -0.0015284031396731734,
+            0.0003922750474885106,
+            -0.014679905027151108,
+            0.007929608225822449,
+            0.005599924363195896,
+            0.022139234468340874,
+            0.0003649175341706723,
+            0.006775618996471167,
+            0.01039675809442997,
+            0.024512330070137978,
+            -0.017798209562897682,
+            0.01902816630899906,
+            0.017335165292024612,
+            0.0329628624022007,
+            0.017725858837366104,
+            -0.010461874306201935,
+            0.02095268853008747,
+            0.014202391728758812,
+            -0.004351879004389048,
+            -0.011807592585682869,
+            0.0037658405490219593,
+            -0.007799377664923668,
+            0.006319811102002859,
+            0.006493452005088329,
+            0.004464022349566221,
+            -0.010375053621828556,
+            0.007368892431259155,
+            -0.00728568946942687,
+            0.005216466728597879,
+            0.003527084132656455,
+            0.012096993625164032,
+            0.0013782759197056293,
+            0.02785491943359375,
+            -0.04306298494338989,
+            0.02161831222474575,
+            -0.02652367204427719,
+            -0.013529532589018345,
+            -0.005382873117923737,
+            0.005158586427569389,
+            0.0030748937278985977,
+            -0.009904775768518448,
+            -0.011532660573720932,
+            -0.01472331490367651,
+            0.014846310950815678,
+            0.0198674313724041,
+            0.0018286574631929398,
+            -0.0071120485663414,
+            0.006938407197594643,
+            -0.00183227495290339,
+            -0.01747986674308777,
+            0.0019010078394785523,
+            -0.008124954998493195,
+            -0.011236024089157581,
+            -0.02208135463297367,
+            0.013059254735708237,
+            0.0027420816477388144,
+            0.006710503716021776,
+            0.00017431934247724712,
+            -0.025858048349618912,
+            -0.01399257592856884,
+            0.009875834919512272,
+            0.0010328025091439486,
+            -0.013102664612233639,
+            -0.013276305980980396,
+            0.006750296335667372,
+            -0.0027583607006818056,
+            0.003986509516835213,
+            0.011199848726391792,
+            0.0102158822119236,
+            -0.006641770713031292,
+            0.0036247570533305407,
+            0.01910051703453064,
+            0.005267112050205469,
+            -0.016452491283416748,
+            0.01648143120110035,
+            -0.016061797738075256,
+            0.0020312387496232986,
+            0.00035768249654211104,
+            -0.02028706483542919,
+            -0.011973998509347439,
+            0.008182834833860397,
+            -0.009079980663955212,
+            -0.0016794346738606691,
+            0.016061797738075256,
+            0.011735241860151291,
+            0.019375449046492577,
+            0.022037943825125694,
+            0.004084182437509298,
+            0.016452491283416748,
+            -0.02310873195528984,
+            0.014130041003227234,
+            -0.021878773346543312,
+            -0.010888741351664066,
+            0.01753774657845497,
+            0.007032462861388922,
+            0.006573037710040808,
+            -0.0012263399548828602,
+            -0.004275911021977663,
+            -0.0060448795557022095,
+            0.014745020307600498,
+            0.0329628624022007,
+            -0.0022247761953622103,
+            -0.01235022023320198,
+            -0.017364105209708214,
+            -0.00755338603630662,
+            0.03666720539331436,
+            -0.01522253267467022,
+            -0.014578614383935928,
+            0.021213149651885033,
+            0.02742081694304943,
+            0.00489812484011054,
+            -0.009593668393790722,
+            -0.0063993968069553375,
+            0.0032539612147957087,
+            0.015265943482518196,
+            0.013464417308568954,
+            -0.010353348217904568,
+            -0.003545171581208706,
+            0.006891379598528147,
+            0.02600274793803692,
+            0.00716992886736989,
+            -0.013695938512682915,
+            -0.025351595133543015,
+            0.0063849263824522495,
+            -0.0023369193077087402,
+            0.012661327607929707,
+            0.0018241355428472161,
+            0.028592893853783607,
+            0.02516348287463188,
+            0.0012191049754619598,
+            0.018854524940252304,
+            -0.014238567091524601,
+            0.04338132590055466,
+            -0.019303098320961,
+            -0.00472448393702507,
+            -0.02801409177482128,
+            0.0010409420356154442,
+            -0.008291360922157764,
+            0.007980253547430038,
+            -0.006544097326695919,
+            0.002765595680102706,
+            -0.020750107243657112,
+            0.01979508064687252,
+            0.02079351618885994,
+            -0.023730944842100143,
+            0.006739443633705378,
+            -0.024859612807631493,
+            0.01417345181107521,
+            -0.029258519411087036,
+            -0.00961537379771471,
+            -0.017190465703606606,
+            -0.006269165780395269,
+            -0.025886988267302513,
+            0.0021433818619698286,
+            0.0009595477604307234,
+            0.02238522656261921,
+            -0.0051332637667655945,
+            0.003635609755292535,
+            0.01187270786613226,
+            -0.01437603309750557,
+            -0.011981233023107052,
+            0.019245218485593796,
+            0.02559758722782135,
+            0.007929608225822449,
+            0.018319131806492805,
+            0.02089480683207512,
+            -0.01440497301518917,
+            -0.04427846893668175,
+            0.0009749222081154585,
+            -0.003903306322172284,
+            -0.001428921241313219,
+            0.004565312992781401,
+            0.012972434051334858,
+            -0.02593039907515049,
+            -0.001539255608804524,
+            -0.004160150419920683,
+            0.0033823831472545862,
+            -0.004536372609436512,
+            0.015902627259492874,
+            0.025481825694441795,
+            -0.013218426145613194,
+            -0.013804464600980282,
+            -0.011366254650056362,
+            -0.005361167713999748,
+            0.008436061441898346,
+            -0.015396174043416977,
+            0.0008849363657645881,
+            -0.0034891001414507627,
+            -0.015511934645473957,
+            -0.00976730976253748,
+            0.002360433107241988,
+            -0.012943494133651257,
+            -0.008566292002797127,
+            0.006844351999461651,
+            -0.027637869119644165,
+            -0.01640908047556877,
+            0.019158396869897842,
+            -0.015164652839303017,
+            0.005118793807923794,
+            0.013746583834290504,
+            0.008812284097075462,
+            0.011149203404784203,
+            0.030097784474492073,
+            -0.00984689500182867,
+            0.0038562784902751446,
+            -0.021763013675808907,
+            -0.016770832240581512,
+            -0.0096804890781641,
+            0.01797184906899929,
+            0.003168949391692877,
+            -0.011829297058284283,
+            0.007647441700100899,
+            -0.012379161082208157,
+            -0.013232896104454994,
+            -0.006949259899556637,
+            0.001656825072132051,
+            -0.008645878173410892,
+            0.015265943482518196,
+            -0.01039675809442997,
+            -0.004131210036575794,
+            0.012075289152562618,
+            0.036696143448352814,
+            -0.0035415540914982557,
+            -0.00988307036459446,
+            -0.0014334431616589427,
+            0.01501995138823986,
+            0.008602467365562916,
+            -0.004351879004389048,
+            -0.020634345710277557,
+            -0.015700045973062515,
+            -0.0022302023135125637,
+            -0.010780216194689274,
+            -0.015627695247530937,
+            0.02271803840994835,
+            -0.021632781252264977,
+            -0.018854524940252304,
+            -0.018782174214720726,
+            -0.0158881563693285,
+            0.00048113043885678053,
+            0.03270240128040314,
+            0.0147739602252841,
+            0.0024780025705695152,
+            0.0038020156789571047,
+            -0.010056711733341217,
+            -0.027478698641061783,
+            0.024584680795669556,
+            0.011380724608898163,
+            -0.030097784474492073,
+            -0.009275326505303383,
+            -0.0187966451048851,
+            -0.004066094756126404,
+            0.007821083068847656,
+            -0.0022591424640268087,
+            0.05417601019144058,
+            -0.0007131040329113603,
+            0.006478982046246529,
+            -0.026682842522859573,
+            0.008298595435917377,
+            -0.015555345453321934,
+            -0.02192218415439129,
+            -0.010990031994879246,
+            -0.012299574911594391,
+            -0.009470673277974129,
+            -0.016336729750037193,
+            -0.002657070057466626,
+            -0.011438604444265366,
+            0.006471747066825628,
+            -0.015092302113771439,
+            -0.0034456897992640734,
+            -0.02407822757959366,
+            -0.004507432691752911,
+            0.005683127325028181,
+            -0.0007497314363718033,
+            0.008964220061898232,
+            -0.01863747462630272,
+            -0.006486217025667429,
+            0.0014054073253646493,
+            0.010794686153531075,
+            0.007886198349297047,
+            -0.0038490435108542442,
+            -0.014115571044385433,
+            -0.013167780824005604,
+            -0.008616937324404716,
+            0.01001330092549324,
+            -0.00699628796428442,
+            0.008096014149487019,
+            -0.01664060167968273,
+            -0.014643729664385319,
+            0.004062477499246597,
+            0.03328120335936546,
+            0.01490419078618288,
+            -0.009065510705113411,
+            -0.00961537379771471,
+            -0.014296446926891804,
+            0.034004706889390945,
+            -0.012777088209986687,
+            -0.014585848897695541,
+            0.012625152245163918,
+            0.022443106397986412,
+            -0.0007632971392013133,
+            0.0045508430339396,
+            0.010440168902277946,
+            -0.00010264718730468303,
+            -0.0007316438714042306,
+            0.0005679509486071765,
+            0.005328610073775053,
+            0.00020608570775948465,
+            0.028144322335720062,
+            -0.009065510705113411,
+            0.014419442974030972,
+            -0.004594252910465002,
+            0.006887761875987053,
+            -0.028100911527872086,
+            -0.0024906639009714127,
+            -0.006522392388433218,
+            -0.005965293850749731,
+            0.004771511536091566,
+            -0.0015076022827997804,
+            0.01774032786488533,
+            -0.03044506534934044,
+            -0.027305057272315025,
+            0.016669541597366333,
+            -0.02112632989883423,
+            -0.007958549074828625,
+            0.005433517973870039,
+            -0.014072161167860031,
+            0.008710993453860283,
+            0.006507922429591417,
+            -0.022327346727252007,
+            -0.01607626862823963,
+            0.0019588882569223642,
+            -0.0023206404875963926,
+            0.0027221853379160166,
+            0.01661166176199913,
+            -0.010548694059252739,
+            -0.012545567005872726,
+            -0.00761850131675601,
+            0.012914554215967655,
+            -0.005212849471718073,
+            0.024165047332644463,
+            -0.013261836022138596,
+            -0.0005720206536352634,
+            0.0009541214676573873,
+            -0.005976146552711725,
+            0.015613225288689137,
+            0.015468524768948555,
+            -0.0035071875900030136,
+            -0.0010002448689192533,
+            0.0003527084190864116,
+            -0.006847969256341457,
+            0.011033442802727222,
+            0.010635514743626118,
+            0.01460755430161953,
+            -0.004102270118892193,
+            0.012531096115708351,
+            -0.004297616425901651,
+            0.0027203767094761133,
+            -0.02079351618885994,
+            0.0040588597767055035,
+            0.002065605018287897,
+            0.018232312053442,
+            -0.020214714109897614,
+            -0.011800357140600681,
+            -0.031110690906643867,
+            -0.004529137630015612,
+            -0.01691553369164467,
+            0.0022012621629983187,
+            0.030155664309859276,
+            0.0015817615203559399,
+            -0.010541459545493126,
+            0.014629259705543518,
+            0.031226450577378273,
+            0.009506847709417343,
+            0.0004411115951370448,
+            0.010830861516296864,
+            -0.011532660573720932,
+            0.010592104867100716,
+            0.012480450794100761,
+            -0.01846383325755596,
+            -0.0015636739553883672,
+            -0.0007429486140608788,
+            -0.003494526259601116,
+            0.009825189597904682,
+            0.007517211139202118,
+            -0.004322939086705446,
+            0.017682448029518127,
+            -0.027015654370188713,
+            0.0036591235548257828,
+            -0.0030115870758891106,
+            -0.004681073594838381,
+            -0.027305057272315025,
+            -0.006352368742227554,
+            0.01747986674308777,
+            -0.010519754141569138,
+            0.058574918657541275,
+            0.002271803794428706,
+            0.007922373712062836,
+            -0.027536578476428986,
+            -0.005230937153100967,
+            -0.008906339295208454,
+            0.0034493072889745235,
+            -0.007184398826211691,
+            0.00761126633733511,
+            0.004008214455097914,
+            0.00022835608979221433,
+            -0.0035885819233953953,
+            0.0020583700388669968,
+            0.0022917003370821476,
+            0.016872122883796692,
+            0.00506453076377511,
+            0.03524913638830185,
+            -0.007727027405053377,
+            -0.005151351448148489,
+            -0.011062382720410824,
+            0.005932736210525036,
+            0.019042637199163437,
+            0.0002742533979471773,
+            0.006956494878977537,
+            0.011156437918543816,
+            0.011395194567739964,
+            0.013985340483486652,
+            0.018261251971125603,
+            -0.0033950444776564837,
+            0.00500665046274662,
+            0.01622096821665764,
+            0.00494153518229723,
+            -0.01410833653062582,
+            -0.011127498000860214,
+            -0.004297616425901651,
+            0.018290191888809204,
+            -0.007466565817594528,
+            0.011207083240151405,
+            0.006761149037629366,
+            0.00582782831043005,
+            0.01016523689031601,
+            -0.01178588718175888,
+            -0.00023604331363458186,
+            0.012777088209986687,
+            -0.030097784474492073,
+            0.012574506923556328,
+            0.0029301929753273726,
+            -0.00789343286305666,
+            0.00244906242005527,
+            -0.0076980870217084885,
+            -0.009065510705113411,
+            -0.0020836926996707916,
+            -0.009145095944404602,
+            0.0005069052567705512,
+            0.015251473523676395,
+            -0.012133168987929821,
+            0.0023658594582229853,
+            -0.003762223059311509,
+            -0.017103644087910652,
+            0.02595933899283409,
+            -0.018290191888809204,
+            0.005183909088373184,
+            0.014368797652423382
+          ],
+          "title": "Introduction And Related Work",
+          "keyphrases": [
+            "conditional computation",
+            "model capacity",
+            "Sparsely-Gated Mixture-of-Experts Layer",
+            "language modeling",
+            "machine translation"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "bec4b8f2-cf26-434f-a587-d63e318fec0c",
+        "properties": {
+          "page_content": "2 The Structure Of The Mixture-Of-Experts Layer\n\nThe Mixture-of-Experts (MoE) layer consists of a set of n \"expert networks\" E1, · · · , En, and a \"gating network\" G whose output is a sparse n-dimensional vector. Figure 1 shows an overview of the MoE module. The experts are themselves neural networks, each with their own parameters.\n\nAlthough in principle we only require that the experts accept the same sized inputs and produce the same-sized outputs, in our initial investigations in this paper, we restrict ourselves to the case where the models are feed-forward networks with identical architectures, but with separate parameters.\n\nLet us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows:\n\n$$y=\\sum_{i=1}^{n}G(x){i}E{i}(x)$$ $$(\\mathbf{l})$$ G(x)iEi(x) (1) We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example. If the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network. In the following we focus on ordinary MoEs. We provide more details on hierarchical MoEs in Appendix B. Our implementation is related to other models of conditional computation. A MoE whose experts are simple weight matrices is similar to the parameterized weight matrix proposed in (Cho & Bengio, 2014). A MoE whose experts have one hidden layer is similar to the block-wise dropout described in (Bengio et al., 2015), where the dropped-out layer is sandwiched between fully-activated layers.\n\n2.1 Gating Network\n\nSoftmax Gating: A simple choice of non-sparse gating function (Jordan & Jacobs, 1994) is to multiply the input by a trainable weight matrix Wg and then apply the Sof tmax function.\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ $\\eqref{eq:walpha}$. Gσ(x) = Sof tmax(x · Wg) (2) Noisy Top-K Gating: We add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to −∞ (which causes the corresponding gate values to equal 0). The sparsity serves to save computation, as described above. While this form of sparsity creates some theoretically scary discontinuities in the output of gating function, we have not yet observed this to be a problem in practice. The noise term helps with load balancing, as will be discussed in Appendix A. The amount of noise per component is controlled by a second trainable weight matrix Wnoise.\n\n$$G(x)=S o f t m a x(K e e p T o p K(H(x),k))$$ $$(4)$$ $$(S)$$ G(x) = Sof tmax(KeepT opK(H(x), k)) (3) $$H(x){i}=(x\\cdot W{g}){i}+S t a n d a r d N o r m a l()\\cdot S o f t p l u s((x\\cdot W{n o i s e}){i})$$ $KeepTopK(v,k){i}=\\begin{cases}v_{i}&\\text{if}v_{i}\\text{is in the top}k\\text{elements of}v.\\ -\\infty&\\text{otherwise.}\\end{cases}$ Training the Gating Network We train the gating network by simple back-propagation, along with the rest of the model. If we choose k > 1, the gate values for the top k experts have nonzero derivatives with respect to the weights of the gating network. This type of occasionally-sensitive behavior is described in (Bengio et al., 2013) with respect to noisy rectifiers. Gradients also backpropagate through the gating network to its inputs. Our method differs here from (Bengio et al., 2015) who use boolean gates and a REINFORCE-style approach to train the gating network.\n\n",
+          "embedding": [
+            0.00986587256193161,
+            -0.03260384127497673,
+            -0.016219932585954666,
+            -0.022328028455376625,
+            0.017736708745360374,
+            0.029624948278069496,
+            -0.004546910524368286,
+            0.026386426761746407,
+            -0.00606368575245142,
+            0.009852208197116852,
+            -0.018788885325193405,
+            -0.02659139782190323,
+            0.0016850214451551437,
+            -0.01340501569211483,
+            -0.004304362926632166,
+            -0.00787083525210619,
+            -0.007932325825095177,
+            0.022382687777280807,
+            -0.00348277622833848,
+            -0.029105691239237785,
+            0.03632062301039696,
+            -0.0069143096916377544,
+            -0.021795107051730156,
+            0.0010393669363111258,
+            0.011690102517604828,
+            0.010815565474331379,
+            0.058430016040802,
+            -0.025744188576936722,
+            -0.05607970058917999,
+            0.010234818793833256,
+            0.020223673433065414,
+            0.003211191389709711,
+            -0.022328028455376625,
+            -0.008465247228741646,
+            -0.07581144571304321,
+            -0.04277033731341362,
+            0.026851026341319084,
+            0.04208710417151451,
+            0.0028746994212269783,
+            0.024746669456362724,
+            0.064114511013031,
+            0.0007968195714056492,
+            -0.00416088430210948,
+            -0.0368945375084877,
+            -0.019854728132486343,
+            -0.012175197713077068,
+            -0.025949159637093544,
+            -0.0034093286376446486,
+            -0.01524974312633276,
+            -0.03134669363498688,
+            4.102061939192936e-05,
+            -0.008957174606621265,
+            0.005318962503224611,
+            0.04080262780189514,
+            -0.05337410047650337,
+            0.022874614223837852,
+            0.01740875653922558,
+            0.021439826115965843,
+            0.03134669363498688,
+            0.02275163307785988,
+            0.04361754283308983,
+            0.00478262547403574,
+            -0.05853933468461037,
+            0.008123630657792091,
+            0.019444787874817848,
+            -0.00873170793056488,
+            0.01642490178346634,
+            0.0034161610528826714,
+            -0.00102143210824579,
+            0.009339784272015095,
+            -0.039490822702646255,
+            0.014552845619618893,
+            -0.02160380221903324,
+            0.0027124318294227123,
+            -0.0009351740591228008,
+            -0.006668346468359232,
+            0.02001870423555374,
+            0.04028337076306343,
+            0.028039850294589996,
+            0.009920531883835793,
+            -0.01302923820912838,
+            0.023366540670394897,
+            0.008595061488449574,
+            0.006244742311537266,
+            0.007324249017983675,
+            -0.02907836250960827,
+            -0.01475781574845314,
+            -0.04747097194194794,
+            -0.010487614199519157,
+            0.006070517934858799,
+            -0.006986049469560385,
+            0.02747959829866886,
+            0.011464636772871017,
+            0.002150473417714238,
+            0.017531737685203552,
+            0.02048330195248127,
+            0.045612581074237823,
+            -0.0037577771581709385,
+            -0.006627352442592382,
+            -0.01876155659556389,
+            -0.0019147582352161407,
+            0.026154128834605217,
+            -0.001622676500119269,
+            0.032795146107673645,
+            0.0015825366135686636,
+            -0.0020411561708897352,
+            0.018816214054822922,
+            0.03131936490535736,
+            0.03260384127497673,
+            0.08685247600078583,
+            -0.0060021947138011456,
+            0.00043107682722620666,
+            -0.019267147406935692,
+            0.03517279401421547,
+            -0.02644108608365059,
+            -0.009086987935006618,
+            -0.04812687635421753,
+            -0.054248638451099396,
+            0.010118668898940086,
+            -0.03875293210148811,
+            0.028039850294589996,
+            0.020893240347504616,
+            -0.023626169189810753,
+            0.0012520230375230312,
+            -0.06941638886928558,
+            0.06351326406002045,
+            0.0015082351164892316,
+            0.007604374550282955,
+            -0.010378297418355942,
+            -0.030854765325784683,
+            0.005377037450671196,
+            0.0025945743545889854,
+            -0.01956777088344097,
+            -0.038889575749635696,
+            0.03028085082769394,
+            0.017299439758062363,
+            0.041540518403053284,
+            0.028504448011517525,
+            -0.005674243438988924,
+            -0.029870912432670593,
+            0.029706936329603195,
+            -0.022778961807489395,
+            -0.007385740056633949,
+            -0.028067179024219513,
+            0.011123020201921463,
+            0.012059048749506474,
+            0.04036535695195198,
+            0.06263872981071472,
+            -0.038670942187309265,
+            0.027834879234433174,
+            -0.04525730013847351,
+            0.04080262780189514,
+            -0.012468988075852394,
+            0.036566589027643204,
+            -0.001954043982550502,
+            0.0030711288563907146,
+            -0.01716279238462448,
+            -0.020578954368829727,
+            -0.014580175280570984,
+            -0.02582617662847042,
+            -0.010754074901342392,
+            -0.028559105470776558,
+            0.005889461375772953,
+            0.049356695264577866,
+            0.01159445010125637,
+            0.05118775740265846,
+            0.009421772323548794,
+            0.019977709278464317,
+            -0.05427596718072891,
+            0.014019925147294998,
+            -0.0017371178837493062,
+            0.008916180580854416,
+            0.04585854709148407,
+            0.008383259177207947,
+            0.0039115045219659805,
+            0.002119727898389101,
+            0.02943364344537258,
+            0.031893279403448105,
+            -0.006326730363070965,
+            0.04028337076306343,
+            -0.03375167027115822,
+            -0.017217451706528664,
+            0.01627459190785885,
+            0.006500954274088144,
+            0.009838543832302094,
+            -0.0024391389451920986,
+            0.03284980356693268,
+            -0.01389694306999445,
+            0.005595671944320202,
+            -0.00036745081888511777,
+            0.042743004858493805,
+            0.039572808891534805,
+            0.04610450938344002,
+            -0.022614985704421997,
+            0.017067139968276024,
+            -0.019854728132486343,
+            0.0023076168727129698,
+            0.015755334869027138,
+            -0.0200323686003685,
+            0.018638573586940765,
+            -0.012223023921251297,
+            -0.012421161867678165,
+            0.011491965502500534,
+            -0.025484561920166016,
+            0.01572800613939762,
+            0.015536700375378132,
+            -0.014853468164801598,
+            -0.019239818677306175,
+            0.013917439617216587,
+            -0.040256042033433914,
+            0.019021185114979744,
+            -0.01000935211777687,
+            -0.01973174512386322,
+            -0.02899637445807457,
+            -0.010453452356159687,
+            -0.005004676058888435,
+            -0.00741990189999342,
+            0.017190122976899147,
+            0.03260384127497673,
+            0.003147992305457592,
+            0.0021487653721123934,
+            0.038370318710803986,
+            0.009387610480189323,
+            0.009585747495293617,
+            -0.0007908412953838706,
+            -0.02303859032690525,
+            -0.016821177676320076,
+            -0.025347914546728134,
+            0.012475820258259773,
+            -0.022396352142095566,
+            -0.030772779136896133,
+            0.02335287630558014,
+            0.007727356161922216,
+            0.01336402166634798,
+            0.015331730246543884,
+            -0.013732966966927052,
+            0.013172716833651066,
+            0.014785144478082657,
+            0.02817649580538273,
+            -0.017531737685203552,
+            -0.02198641188442707,
+            0.020893240347504616,
+            -0.06673812121152878,
+            -0.0074745602905750275,
+            0.006716172676533461,
+            0.02793053165078163,
+            -0.013507500290870667,
+            -0.012926753610372543,
+            -0.007481392472982407,
+            -0.004116474185138941,
+            0.012981412000954151,
+            -0.015277071855962276,
+            -0.010501278564333916,
+            0.019021185114979744,
+            -0.015072102658450603,
+            0.046131838113069534,
+            0.011840414255857468,
+            -0.014484522864222527,
+            -0.015454712323844433,
+            -0.00046801407006569207,
+            0.0370585136115551,
+            0.031292036175727844,
+            -0.0269603431224823,
+            0.009852208197116852,
+            0.008533569984138012,
+            -0.025634871795773506,
+            0.0616002157330513,
+            0.015564029105007648,
+            0.01341184787452221,
+            -0.000460754701634869,
+            0.024131761863827705,
+            -0.04222375154495239,
+            0.013036070391535759,
+            -0.021125540137290955,
+            0.05400267243385315,
+            -0.02295660227537155,
+            0.011539791710674763,
+            -0.04708836227655411,
+            -0.0032009428832679987,
+            0.047170352190732956,
+            0.013418680056929588,
+            0.010965877212584019,
+            0.004526413511484861,
+            0.01973174512386322,
+            0.029215008020401,
+            0.015441047959029675,
+            -0.009121149778366089,
+            -0.017572732642292976,
+            -0.05310080572962761,
+            0.008567731827497482,
+            -0.015482041984796524,
+            0.007495057303458452,
+            -0.03342371806502342,
+            0.0015953471884131432,
+            0.02981625311076641,
+            0.014621169306337833,
+            -0.014361540786921978,
+            -0.002075317781418562,
+            0.027807550504803658,
+            -0.042879652231931686,
+            -0.04968464747071266,
+            -0.007317416835576296,
+            -0.011368983425199986,
+            0.007693194784224033,
+            -0.03361502289772034,
+            0.02048330195248127,
+            0.009394442662596703,
+            0.024828657507896423,
+            -0.005103744566440582,
+            -0.011027367785573006,
+            0.01386278122663498,
+            0.002191467210650444,
+            -0.009209970012307167,
+            -0.03366968408226967,
+            -0.029269667342305183,
+            -0.010166495107114315,
+            0.008137295953929424,
+            0.027848543599247932,
+            -0.016219932585954666,
+            -0.012011222541332245,
+            0.027684569358825684,
+            0.022164052352309227,
+            -0.007153441198170185,
+            0.0025689529720693827,
+            -0.024651017040014267,
+            0.005295049399137497,
+            0.015755334869027138,
+            0.028340471908450127,
+            -0.03960013762116432,
+            0.011321157217025757,
+            -0.00564691424369812,
+            -0.022341692820191383,
+            -0.03312309831380844,
+            0.02380380965769291,
+            -0.0134391775354743,
+            -0.03621130809187889,
+            -0.014279552735388279,
+            0.014812474139034748,
+            -0.0044615063816308975,
+            -0.03172930330038071,
+            -0.013650978915393353,
+            0.006398469675332308,
+            -0.014416199177503586,
+            0.01612428016960621,
+            -0.015085767023265362,
+            -0.0035254780668765306,
+            -0.03880758956074715,
+            0.013036070391535759,
+            0.014006259851157665,
+            0.010938547551631927,
+            0.0009035745752044022,
+            0.006791328079998493,
+            0.017572732642292976,
+            -0.016055956482887268,
+            -0.026755372062325478,
+            -0.024159090593457222,
+            0.0017046643188223243,
+            0.006490706000477076,
+            0.006415550597012043,
+            0.010200656950473785,
+            0.016916830092668533,
+            -0.026454750448465347,
+            -0.011573953554034233,
+            0.007986984215676785,
+            -0.054631248116493225,
+            -0.012298179790377617,
+            -0.03878026083111763,
+            -0.019526775926351547,
+            -0.0013092437293380499,
+            0.028941715136170387,
+            -0.0064462958835065365,
+            0.03427092730998993,
+            -0.0056708273477852345,
+            0.00754971569404006,
+            0.017203787341713905,
+            0.018993854522705078,
+            0.018173975870013237,
+            -0.011806252412497997,
+            -0.050340548157691956,
+            0.028887057676911354,
+            -0.03593801334500313,
+            -0.0045229969546198845,
+            0.00282345712184906,
+            -0.002869575284421444,
+            -0.0057049887254834175,
+            0.0338883176445961,
+            0.0015304401749745011,
+            0.023544181138277054,
+            -0.00381243578158319,
+            0.04353555664420128,
+            -0.027055995538830757,
+            -0.049821291118860245,
+            0.00873170793056488,
+            0.027657238766551018,
+            0.024814993143081665,
+            -0.006521451286971569,
+            -0.026536738499999046,
+            -0.02018267847597599,
+            -0.013514332473278046,
+            0.007966487668454647,
+            0.002003578469157219,
+            -0.00044239286216907203,
+            -0.0013263245346024632,
+            0.006548780482262373,
+            -0.034462232142686844,
+            -0.03645727038383484,
+            0.0040549831464886665,
+            0.011813084594905376,
+            0.03213924169540405,
+            -0.02590816468000412,
+            0.017067139968276024,
+            -0.015454712323844433,
+            -0.038588955998420715,
+            0.0006661514053121209,
+            0.023598840460181236,
+            0.035965342074632645,
+            -0.03921752795577049,
+            0.018419940024614334,
+            0.015850987285375595,
+            -0.005602504126727581,
+            0.004188213497400284,
+            0.025265926495194435,
+            0.025347914546728134,
+            0.022273369133472443,
+            -0.00858822837471962,
+            0.04020138084888458,
+            0.014457193203270435,
+            -0.005564926192164421,
+            0.029706936329603195,
+            -0.016766518354415894,
+            0.02998022921383381,
+            -0.0002463906130287796,
+            0.021426161751151085,
+            -0.027916867285966873,
+            0.019007520750164986,
+            -0.0382336750626564,
+            -0.008642887696623802,
+            0.03304110839962959,
+            -0.026085805147886276,
+            0.0034537387546151876,
+            0.022915607318282127,
+            0.01851559244096279,
+            -0.010631092824041843,
+            0.01725844480097294,
+            -0.04451940953731537,
+            -0.029160350561141968,
+            -0.026195121929049492,
+            -0.006822073366492987,
+            -0.02500629797577858,
+            -0.03552807494997978,
+            -0.056134358048439026,
+            0.008649719879031181,
+            -0.047935571521520615,
+            0.004386350512504578,
+            0.03533677011728287,
+            0.010760907083749771,
+            0.010549104772508144,
+            -0.024432383477687836,
+            0.02809450775384903,
+            -0.018351616337895393,
+            -0.00914847943931818,
+            0.021043552085757256,
+            0.024883316829800606,
+            -0.012346005998551846,
+            -0.013842284679412842,
+            0.01612428016960621,
+            -0.03804237022995949,
+            0.006350643467158079,
+            -0.0020326157100498676,
+            -0.024446047842502594,
+            0.019991373643279076,
+            -0.04684239998459816,
+            0.02380380965769291,
+            0.013876445591449738,
+            0.02877774089574814,
+            0.06039772555232048,
+            -0.024651017040014267,
+            -0.06285735964775085,
+            0.020128021016716957,
+            -0.004143803380429745,
+            0.01336402166634798,
+            0.003747528651729226,
+            -0.006169586908072233,
+            0.002331529976800084,
+            -0.01679384708404541,
+            -0.026031147688627243,
+            0.052526891231536865,
+            0.023475859314203262,
+            0.015263407491147518,
+            -0.010023016482591629,
+            0.0013707346515730023,
+            0.0368945375084877,
+            0.023817474022507668,
+            0.01378079317510128,
+            0.016998816281557083,
+            -0.03929951786994934,
+            0.0037782741710543633,
+            0.0064087179489433765,
+            0.022246040403842926,
+            0.027206305414438248,
+            -0.02711065299808979,
+            -0.027083324268460274,
+            0.020920569077134132,
+            0.04528462886810303,
+            0.011908737011253834,
+            -0.0139379370957613,
+            0.016055956482887268,
+            0.03383366018533707,
+            -0.035965342074632645,
+            -0.003212899435311556,
+            0.009551586583256721,
+            0.0011298952158540487,
+            -0.010719913057982922,
+            -0.02041497826576233,
+            0.009961525909602642,
+            0.014402534812688828,
+            -0.02010069042444229,
+            0.01906217820942402,
+            0.010972709394991398,
+            0.028449788689613342,
+            0.0052267261780798435,
+            -0.006750334054231644,
+            -0.029570288956165314,
+            0.04902874305844307,
+            -0.022027406841516495,
+            0.022997595369815826,
+            0.03774174675345421,
+            -0.0027431773487478495,
+            0.024487042799592018,
+            0.002572369296103716,
+            -0.03812435641884804,
+            0.00813046284019947,
+            0.04451940953731537,
+            0.005575174931436777,
+            -0.028039850294589996,
+            -0.016670865938067436,
+            -0.03145601227879524,
+            0.004348773043602705,
+            -0.01738142780959606,
+            -0.02793053165078163,
+            -0.0014125825837254524,
+            0.005551261827349663,
+            0.016684530302882195,
+            -0.002896904479712248,
+            -0.005985114257782698,
+            -0.006132008973509073,
+            0.02507462166249752,
+            -0.049821291118860245,
+            -0.005663995165377855,
+            0.014443528838455677,
+            -0.029406314715743065,
+            -0.028504448011517525,
+            -0.03711317479610443,
+            -0.004229207523167133,
+            -0.013842284679412842,
+            0.0010385129135102034,
+            -0.001186261884868145,
+            -0.0134391775354743,
+            0.011082026176154613,
+            -0.03940883278846741,
+            0.02508828602731228,
+            -0.012100041843950748,
+            -0.010139165446162224,
+            0.0030813771300017834,
+            0.009230467490851879,
+            -0.010788236744701862,
+            -0.008226115256547928,
+            0.04178648069500923,
+            0.02198641188442707,
+            0.024418719112873077,
+            -0.006641016807407141,
+            -0.007071453146636486,
+            0.005745982751250267,
+            0.000374923663912341,
+            0.03809702768921852,
+            -0.001469803275540471,
+            0.03561006113886833,
+            -0.009121149778366089,
+            -0.012660292908549309,
+            0.006883564405143261,
+            0.03752311319112778,
+            -0.0069962977431714535,
+            0.010207489132881165,
+            0.03088209591805935,
+            0.027015000581741333,
+            -0.017121799290180206,
+            -0.015195083804428577,
+            -0.0025894499849528074,
+            -0.004434177186340094,
+            -0.021057216450572014,
+            0.02198641188442707,
+            0.03350570797920227,
+            0.01171059999614954,
+            -0.009353448636829853,
+            0.022573992609977722,
+            -0.04348089545965195,
+            -0.0011435599299147725,
+            0.004113057628273964,
+            0.012455322779715061,
+            -0.05001259595155716,
+            0.016247261315584183,
+            0.014416199177503586,
+            0.017367763444781303,
+            0.00129045476205647,
+            0.006395053584128618,
+            0.023175235837697983,
+            0.02176777832210064,
+            0.024951640516519547,
+            -0.015850987285375595,
+            2.2431901015806943e-05,
+            0.019841063767671585,
+            0.015618688426911831,
+            0.024268407374620438,
+            0.0021436410024762154,
+            0.024978969246149063,
+            -0.017586397007107735,
+            -0.0138559490442276,
+            0.00282345712184906,
+            -0.023380205035209656,
+            -0.010330471210181713,
+            0.05468590557575226,
+            0.011553456075489521,
+            0.051242414861917496,
+            0.01426588837057352,
+            0.00024169338576029986,
+            0.018993854522705078,
+            -0.020510630682110786,
+            -0.02026466652750969,
+            -0.0007050101994536817,
+            0.0027824630960822105,
+            -0.020387649536132812,
+            -0.00945593323558569,
+            -0.010610596276819706,
+            -0.018187642097473145,
+            -0.018939197063446045,
+            -0.006746917963027954,
+            -0.006326730363070965,
+            0.016848506405949593,
+            -0.002319573424756527,
+            -0.013541662134230137,
+            -0.0320572555065155,
+            0.022041071206331253,
+            0.0199503805488348,
+            0.040556661784648895,
+            -0.013575823977589607,
+            0.011239170096814632,
+            -0.0033819994423538446,
+            -0.004560574889183044,
+            0.014621169306337833,
+            -0.01604229211807251,
+            0.017982671037316322,
+            0.009196305647492409,
+            0.00139293959364295,
+            0.011075193993747234,
+            -0.01077457144856453,
+            -0.04684239998459816,
+            -0.010200656950473785,
+            -0.023557845503091812,
+            0.03284980356693268,
+            0.00672300485894084,
+            0.0018566835206001997,
+            0.0039012557826936245,
+            0.0035459750797599554,
+            -0.008116798475384712,
+            0.016821177676320076,
+            0.013275201432406902,
+            -0.03274048864841461,
+            0.017436085268855095,
+            -0.011382648721337318,
+            0.010255315341055393,
+            -0.034598879516124725,
+            -0.013691972941160202,
+            -0.002481841016560793,
+            0.017586397007107735,
+            0.026482079178094864,
+            -0.02869575284421444,
+            0.010610596276819706,
+            0.02131684496998787,
+            -0.002876407466828823,
+            -0.005698156543076038,
+            0.0021231439895927906,
+            0.0014151446521282196,
+            -0.026304440572857857,
+            -0.012024886906147003,
+            0.005301882047206163,
+            0.004659643396735191,
+            -0.03386098891496658,
+            0.005448776762932539,
+            -0.0293789841234684,
+            -0.005595671944320202,
+            -0.011908737011253834,
+            0.003928584977984428,
+            0.0068152411840856075,
+            0.002565536880865693,
+            0.009134814143180847,
+            -0.01831062324345112,
+            -0.007638535927981138,
+            0.024145426228642464,
+            0.004222374875098467,
+            0.015195083804428577,
+            -0.012414328753948212,
+            0.029542960226535797,
+            -0.016684530302882195,
+            0.021959083154797554,
+            0.014402534812688828,
+            0.007857169955968857,
+            -0.002591158030554652,
+            -0.002437430899590254,
+            -0.04471071437001228,
+            -0.01046711765229702,
+            0.010111836716532707,
+            0.02559387870132923,
+            -0.0005986822070553899,
+            0.014539181254804134,
+            0.02477400004863739,
+            -0.008882018737494946,
+            0.015495706349611282,
+            -0.01701248250901699,
+            -0.030144205316901207,
+            0.020305661484599113,
+            -0.0027380529791116714,
+            0.015318065881729126,
+            0.02425474300980568,
+            -0.03148334100842476,
+            -0.02432306669652462,
+            -0.002088982379063964,
+            0.011813084594905376,
+            0.05137906223535538,
+            0.029706936329603195,
+            0.012182029895484447,
+            0.020906904712319374,
+            -0.008157792501151562,
+            -0.02869575284421444,
+            0.012455322779715061,
+            0.01265346072614193,
+            -0.03110072948038578,
+            0.00634722737595439,
+            0.026755372062325478,
+            -0.008601893670856953,
+            -0.02853177674114704,
+            0.02599015273153782,
+            -0.00689381267875433,
+            0.002553580328822136,
+            -0.004796289838850498,
+            -0.020947899669408798,
+            0.004386350512504578,
+            -0.01602862775325775,
+            0.027903202921152115,
+            -0.024418719112873077,
+            0.008595061488449574,
+            0.010084507055580616,
+            -0.009414940141141415,
+            0.032931793481111526,
+            -0.0005137051921337843,
+            -0.01505843736231327,
+            0.010173327289521694,
+            0.021344173699617386,
+            -0.028887057676911354,
+            0.0005017486400902271,
+            0.002992557128891349,
+            0.01339818350970745,
+            0.03968212753534317,
+            0.018939197063446045,
+            0.016698194667696953,
+            0.03295912221074104,
+            0.020169014111161232,
+            -0.01294725015759468,
+            0.025170274078845978,
+            0.01039879396557808,
+            -0.020237337797880173,
+            0.008615558035671711,
+            -0.0060056112706661224,
+            0.014580175280570984,
+            -0.026851026341319084,
+            -0.007050956133753061,
+            -0.002191467210650444,
+            0.0020087026059627533,
+            -0.020373983308672905,
+            -0.005223310086876154,
+            -0.002256374340504408,
+            -0.023680828511714935,
+            0.008150960318744183,
+            0.028668422251939774,
+            0.023393871262669563,
+            -0.00436585396528244,
+            -0.021508149802684784,
+            0.026673385873436928,
+            -0.02410443127155304,
+            -0.012714951299130917,
+            -0.0037782741710543633,
+            0.015796327963471413,
+            -0.022983931005001068,
+            -0.032494522631168365,
+            0.0019437955925241113,
+            0.03509080410003662,
+            0.02823115512728691,
+            -0.004167716484516859,
+            0.044546738266944885,
+            0.02333921194076538,
+            -0.021617466583848,
+            0.02033299021422863,
+            -0.038288332521915436,
+            0.018939197063446045,
+            0.017531737685203552,
+            -0.04684239998459816,
+            -0.030253522098064423,
+            -0.006258407142013311,
+            -0.011895072646439075,
+            0.008123630657792091,
+            -0.017231116071343422,
+            0.003781690262258053,
+            -0.00754971569404006,
+            -0.023147907108068466,
+            0.03470819443464279,
+            -0.016506889835000038,
+            0.03402496501803398,
+            -0.008369594812393188,
+            0.003199234837666154,
+            -0.015632351860404015,
+            -0.016315585002303123,
+            -0.012468988075852394,
+            -0.020770259201526642,
+            -0.01027581188827753,
+            -0.02681003138422966,
+            -0.021521814167499542,
+            -0.011341654695570469,
+            0.004509332589805126,
+            -0.008458415046334267,
+            0.02363983355462551,
+            0.0268783550709486,
+            0.01133482251316309,
+            0.015331730246543884,
+            -0.0207839235663414,
+            0.043644871562719345,
+            -0.026304440572857857,
+            0.022942937910556793,
+            -0.013186381198465824,
+            -0.015372724272310734,
+            0.009681399911642075,
+            0.020879575982689857,
+            -0.04072063788771629,
+            0.0034229932352900505,
+            -0.01611061580479145,
+            -0.007385740056633949,
+            -0.0215354785323143,
+            0.00495001720264554,
+            0.017818694934248924,
+            -0.008793198503553867,
+            0.00791182927787304,
+            0.00041847972897812724,
+            -0.004543493967503309,
+            0.05099645256996155,
+            -0.0027175559662282467,
+            0.02642742171883583,
+            -0.016301920637488365,
+            -0.019171494990587234,
+            0.013261537067592144,
+            -0.029870912432670593,
+            0.013698806054890156,
+            -0.00832176860421896,
+            -0.037085842341184616,
+            -0.00865655206143856,
+            0.01171059999614954,
+            0.018392611294984818,
+            0.0116422763094306,
+            -0.0038875911850482225,
+            -0.009845376014709473,
+            -0.011389480903744698,
+            -0.00627207150682807,
+            -0.03582869842648506,
+            -0.015919310972094536,
+            0.0021436410024762154,
+            -0.019690752029418945,
+            0.03394297510385513,
+            0.0034434902481734753,
+            0.03741379454731941,
+            -0.011409977450966835,
+            -0.014211229979991913,
+            0.0052643041126430035,
+            -0.017518073320388794,
+            0.0038226840551942587,
+            0.0029771842528134584,
+            0.008458415046334267,
+            -0.018993854522705078,
+            0.03962746635079384,
+            0.009052827022969723,
+            0.014511851593852043,
+            -0.03462620824575424,
+            -0.013295698910951614,
+            -0.04501133784651756,
+            0.006654681637883186,
+            -0.017873354256153107,
+            -0.032357875257730484,
+            -0.0004112203896511346,
+            -0.005698156543076038,
+            0.005882629193365574,
+            -0.016534218564629555,
+            -0.013609985820949078,
+            0.006702507846057415,
+            -0.008226115256547928,
+            0.009223634377121925,
+            -0.01896652579307556,
+            0.010180159471929073,
+            0.004000324755907059,
+            0.03192060813307762,
+            -0.021658461540937424,
+            0.034598879516124725,
+            -0.012011222541332245,
+            0.003979827743023634,
+            -0.00013408431550487876,
+            -0.012632963247597218,
+            0.019239818677306175,
+            0.016766518354415894,
+            -0.0372224897146225,
+            0.011464636772871017,
+            0.0477442666888237,
+            0.03563738986849785,
+            -0.036484599113464355,
+            0.012530478648841381,
+            0.007208099588751793,
+            -0.003062588395550847,
+            0.03186595067381859,
+            0.0013647563755512238,
+            0.01671185903251171,
+            -0.0007998087094165385,
+            -0.02582617662847042,
+            -0.024514371529221535,
+            0.0037201992236077785,
+            0.01378762535750866,
+            0.007392572239041328,
+            -0.031073400750756264,
+            0.01941745914518833,
+            0.002908861031755805,
+            0.029925569891929626,
+            0.025812512263655663,
+            -0.012598801404237747,
+            0.031210046261548996,
+            0.01423855870962143,
+            -0.029242338612675667,
+            0.027985190972685814,
+            0.010747242718935013,
+            0.014060918241739273,
+            0.029652277007699013,
+            0.04749830439686775,
+            0.009326119907200336,
+            -0.022027406841516495,
+            0.001334864879027009,
+            0.012960914522409439,
+            -0.009954692795872688,
+            0.016288256272673607,
+            0.014183900319039822,
+            0.026031147688627243,
+            0.010965877212584019,
+            -0.014019925147294998,
+            -0.010863391682505608,
+            0.04430077597498894,
+            -0.01574167050421238,
+            -0.00499101122841239,
+            -0.016766518354415894,
+            0.035200122743844986,
+            0.027452269569039345,
+            -0.02402244508266449,
+            -0.02335287630558014,
+            -0.005202813073992729,
+            -0.006750334054231644,
+            0.016329249367117882,
+            -0.01687583513557911,
+            -0.01378762535750866,
+            -0.02891438640654087,
+            -0.01596030406653881,
+            0.0018754723714664578,
+            -0.03929951786994934,
+            -0.02839512936770916,
+            -0.0008578834240324795,
+            0.011437307111918926,
+            0.0008672778494656086,
+            0.02206839993596077,
+            -0.012714951299130917,
+            -0.019021185114979744,
+            -0.005202813073992729,
+            0.02847711741924286,
+            -0.01125283446162939,
+            0.021508149802684784,
+            -0.004833867773413658,
+            0.005014924332499504,
+            0.005206229165196419,
+            0.041239894926548004,
+            0.0016380491433665156,
+            -0.02847711741924286,
+            0.02794419787824154,
+            -0.008554067462682724,
+            0.009688233025372028,
+            0.03607466071844101,
+            0.0026663136668503284,
+            -0.02235535718500614,
+            0.02418641932308674,
+            -0.0026321520563215017,
+            -0.02025100216269493,
+            -0.008936677128076553,
+            0.008007481694221497,
+            -0.0027500095311552286,
+            -0.019390130415558815,
+            0.009319286793470383,
+            0.042141761630773544,
+            -0.0028405378106981516,
+            0.02387213334441185,
+            0.053046148270368576,
+            -0.02235535718500614,
+            -0.02387213334441185,
+            0.015372724272310734,
+            -0.028504448011517525,
+            -0.007044123951345682,
+            -0.006832322105765343,
+            0.03826100379228592,
+            -0.03041749820113182,
+            -0.010863391682505608,
+            -0.025361578911542892,
+            7.216640369733796e-05,
+            -0.05495919659733772,
+            0.01904851384460926,
+            0.004266784992069006,
+            0.01505843736231327,
+            -0.041895799338817596,
+            -0.0353914275765419,
+            0.028559105470776558,
+            0.029269667342305183,
+            -0.007433566264808178,
+            -0.00429069809615612,
+            -0.011608115397393703,
+            -0.009763387963175774,
+            0.021371502429246902,
+            0.0008967422763817012,
+            -0.01565968245267868,
+            0.014101912267506123,
+            0.03170197457075119,
+            -0.0014373497106134892,
+            -0.015577694401144981,
+            -0.06045238673686981,
+            -0.017299439758062363,
+            0.014908126555383205,
+            0.0023178651463240385,
+            -0.0020394481252878904,
+            -3.75510826415848e-05,
+            0.0011683270568028092,
+            -0.0024288904387503862,
+            -0.025197602808475494,
+            0.02726096473634243,
+            0.008793198503553867,
+            0.014525516889989376,
+            -0.004587904084473848,
+            -0.011416809633374214,
+            0.014894462190568447,
+            0.017682049423456192,
+            -0.011881408281624317,
+            -0.017121799290180206,
+            0.03894423693418503,
+            0.01973174512386322,
+            -0.001998454099521041,
+            -0.0005862986436113715,
+            0.0010333886602893472,
+            -0.004136971198022366,
+            -0.030062217265367508,
+            0.001253731083124876,
+            -0.0009095528512261808,
+            0.00581772206351161,
+            0.02823115512728691,
+            -0.008724874816834927,
+            0.029652277007699013,
+            0.023462193086743355,
+            0.021357838064432144,
+            -0.0032590175978839397,
+            -0.0043897670693695545,
+            -0.0018618077738210559,
+            0.02538890950381756,
+            0.01208637747913599,
+            0.008670216426253319,
+            0.002106063300743699,
+            -0.0038602619897574186,
+            -0.017354097217321396,
+            0.010781404562294483,
+            -0.013391351327300072,
+            -0.015482041984796524,
+            0.016001299023628235,
+            0.027001336216926575,
+            -0.007604374550282955,
+            -0.004222374875098467,
+            0.0025962824001908302,
+            0.01076773926615715,
+            0.033478379249572754,
+            -0.019267147406935692,
+            -0.007652200758457184,
+            0.021795107051730156,
+            -0.00502858916297555,
+            0.01265346072614193,
+            0.010528608225286007,
+            0.00910748541355133,
+            0.027616245672106743,
+            0.06580892205238342,
+            -0.030909424647688866,
+            0.0016628163866698742,
+            -0.01934913545846939,
+            -0.0011239169398322701,
+            0.011512462981045246,
+            -0.023516852408647537,
+            0.034598879516124725,
+            0.003549391170963645,
+            -0.017969006672501564,
+            0.034680865705013275,
+            0.015564029105007648,
+            0.010945379734039307,
+            0.010296309366822243,
+            -0.01657521352171898,
+            -0.004909023176878691,
+            -0.011088858358561993,
+            0.008328600786626339,
+            0.020606283098459244,
+            0.023065919056534767,
+            0.01679384708404541,
+            0.018474599346518517,
+            0.002625319641083479,
+            0.01770937815308571,
+            -0.0200323686003685,
+            0.021029887720942497,
+            -0.003908087965101004,
+            0.005117409396916628,
+            -0.02251933328807354,
+            0.0028319975826889277,
+            0.0025552883744239807,
+            -0.001964292488992214,
+            0.013698806054890156,
+            0.000991540728136897,
+            -0.011000038124620914,
+            0.003528894390910864,
+            0.0021248520351946354,
+            0.01132799033075571,
+            0.009312454611063004,
+            -0.0032931792084127665,
+            0.004249704070389271,
+            0.018652239814400673,
+            0.022874614223837852,
+            0.012530478648841381,
+            0.0013177840737625957,
+            0.012305011972784996,
+            0.019923051819205284,
+            -0.022273369133472443,
+            0.01088388916105032,
+            -0.030854765325784683,
+            -0.003627962898463011,
+            0.0036689569242298603,
+            -0.010330471210181713,
+            -0.005964617244899273,
+            -0.011505629867315292,
+            -0.008301271125674248,
+            0.01295408234000206,
+            -0.0074472310952842236,
+            0.021726783365011215,
+            -0.001455284538678825,
+            -0.006538532208651304,
+            0.02176777832210064,
+            0.0013391351094469428,
+            -0.0044512576423585415,
+            -0.028559105470776558,
+            -0.0099751902744174,
+            0.021043552085757256,
+            -0.002442555036395788,
+            -0.0005747691029682755,
+            0.02559387870132923,
+            -0.015495706349611282,
+            0.01133482251316309,
+            -0.01686217077076435,
+            -0.015304401516914368,
+            -0.0028217490762472153,
+            -0.015468376688659191,
+            0.016452230513095856,
+            -0.004406847525388002,
+            0.00627207150682807,
+            -0.03448956087231636,
+            0.03142867982387543,
+            0.006627352442592382,
+            -0.0016611083410680294,
+            -0.013664644211530685,
+            -0.030253522098064423,
+            0.0095584187656641,
+            -0.021207528188824654,
+            -0.00017358368495479226,
+            0.02176777832210064,
+            0.011471468955278397,
+            -0.0004603276902344078,
+            -0.009066491387784481,
+            0.0024835490621626377,
+            0.0059680333361029625,
+            -0.0009180932538583875,
+            0.005862132180482149,
+            -0.00028332785586826503,
+            0.015304401516914368,
+            -0.016957823187112808,
+            -0.028449788689613342,
+            -0.02303859032690525,
+            0.01306340005248785,
+            0.00862239021807909,
+            -0.004871445707976818,
+            -0.015523036010563374,
+            0.0057767280377447605,
+            0.004181380849331617,
+            -0.0018020248971879482,
+            -0.0034810679499059916,
+            0.007488225121051073,
+            -0.0010103295790031552,
+            0.001913050189614296,
+            0.002036032034084201,
+            -0.012796939350664616,
+            0.020537959411740303,
+            0.035118136554956436,
+            0.006186667364090681,
+            -0.03320508450269699,
+            0.0021624299697577953,
+            0.00015885148604866117,
+            0.0011461219983175397,
+            -0.0069587198086082935,
+            0.026974007487297058,
+            0.008519905619323254,
+            0.01550937071442604,
+            0.030827436596155167,
+            0.0036347953137010336,
+            -0.0031821539159864187,
+            0.012475820258259773,
+            -0.023434864357113838,
+            -0.02425474300980568,
+            0.0027858791872859,
+            0.009244131855666637,
+            0.010405626147985458,
+            0.019226154312491417,
+            -0.012188862077891827,
+            0.004003740847110748,
+            0.01679384708404541,
+            0.019294477999210358,
+            -0.009920531883835793,
+            -0.01816031150519848,
+            0.01516775507479906,
+            0.054084662348032,
+            0.01590564474463463,
+            -0.006565861403942108,
+            -0.01035779993981123,
+            -0.013377686031162739,
+            0.02801251970231533,
+            0.01092488318681717,
+            0.01596030406653881,
+            -0.0003646751865744591,
+            -0.011840414255857468,
+            0.015919310972094536,
+            0.005981697700917721,
+            -0.0014271012041717768,
+            -0.03910820931196213,
+            0.020073361694812775,
+            -0.0058245547115802765,
+            0.02139883302152157,
+            -0.0005619584699161351,
+            -0.0027397610247135162,
+            0.03899889439344406,
+            0.03683988004922867,
+            -0.011491965502500534,
+            0.04443742334842682,
+            -0.007385740056633949,
+            -0.0011409977450966835,
+            -0.0008967422763817012,
+            -0.0026270279195159674,
+            0.01822863519191742,
+            -0.012899423949420452,
+            0.0008950341725721955,
+            -0.0024271823931485415,
+            0.026413757354021072,
+            0.0042702010832726955,
+            -0.014785144478082657,
+            0.0075087216682732105,
+            -0.006442879792302847,
+            -0.02674170769751072,
+            -0.005899710115045309,
+            0.013760296627879143,
+            -0.010685751214623451,
+            0.005783560685813427,
+            0.006521451286971569,
+            0.023065919056534767,
+            0.01738142780959606,
+            -0.019977709278464317,
+            -0.007495057303458452,
+            -0.008526737801730633,
+            0.01261929888278246,
+            0.004813370760530233,
+            -0.012366502545773983,
+            0.022874614223837852,
+            -0.01964975893497467,
+            0.0030284267850220203,
+            0.016165273264050484,
+            -0.007310584653168917,
+            -0.0011854078620672226,
+            -0.02175411395728588,
+            -0.012305011972784996,
+            -0.04001007601618767,
+            0.013849116861820221,
+            -0.016315585002303123,
+            -0.013773960992693901,
+            0.019786404445767403,
+            -0.027848543599247932,
+            0.025197602808475494,
+            -0.009401274845004082,
+            0.024200083687901497,
+            -0.019923051819205284,
+            0.013849116861820221,
+            0.005708404816687107,
+            -0.017531737685203552,
+            0.005339459516108036,
+            -0.012243521399796009,
+            -0.007228596601635218,
+            -0.0020821501966565847,
+            -0.014921790920197964,
+            0.03593801334500313,
+            0.0323852077126503,
+            -0.02568953111767769,
+            0.03142867982387543,
+            0.02034665457904339,
+            0.02071559987962246,
+            -0.005240391008555889,
+            -0.010836062952876091,
+            0.0199503805488348,
+            -0.016903163865208626,
+            0.03323241323232651,
+            0.0027841711416840553,
+            -0.0008284190553240478,
+            0.0015620396006852388,
+            -0.007529218681156635,
+            0.020988892763853073,
+            -0.007761518005281687,
+            0.008806862868368626,
+            0.021275850012898445,
+            -0.003580136690288782,
+            0.010549104772508144,
+            -0.006357475649565458,
+            0.015768999233841896,
+            0.026536738499999046,
+            -0.009968358092010021,
+            0.03733180835843086,
+            0.00813046284019947,
+            -0.00014177068078424782,
+            0.011983892880380154,
+            0.0018310622544959188,
+            -0.019308142364025116,
+            0.006945055443793535,
+            -0.021426161751151085,
+            0.0030403833370655775,
+            0.01434787642210722,
+            -0.004833867773413658,
+            -0.021808771416544914,
+            0.005752814933657646,
+            -0.004150635562837124,
+            -0.010248483158648014,
+            -0.0074745602905750275,
+            -0.01816031150519848,
+            0.011075193993747234,
+            0.018351616337895393,
+            0.028641093522310257,
+            -0.0030335509218275547,
+            -0.002674854127690196,
+            0.007228596601635218,
+            -0.008567731827497482,
+            0.030007557943463326,
+            -0.029187679290771484,
+            0.008308103308081627,
+            -0.0033905399031937122,
+            0.0017012482276186347,
+            0.02139883302152157,
+            -8.753912698011845e-05,
+            -0.010494446381926537,
+            -0.012188862077891827,
+            0.006832322105765343,
+            -0.009818046353757381,
+            -0.007228596601635218,
+            -0.002570661250501871,
+            0.006729837041348219,
+            0.024268407374620438,
+            0.005804057698696852,
+            0.012714951299130917,
+            -0.03402496501803398,
+            0.000319197541102767,
+            -0.003631379222497344,
+            -0.015154089778661728,
+            -0.0037714417558163404,
+            -0.012646627612411976,
+            -0.002927649999037385,
+            -0.030007557943463326,
+            0.008779534138739109,
+            0.001211883150972426,
+            0.006593190599232912,
+            0.004864613059908152,
+            -0.022642314434051514,
+            0.004488835576921701,
+            -0.006641016807407141,
+            -0.0018600996118038893,
+            0.013452841900289059,
+            -0.0033580863382667303,
+            -0.012236688286066055,
+            0.0002893061318900436,
+            0.014019925147294998,
+            -0.014156571589410305,
+            0.006248158402740955,
+            0.005066166631877422,
+            -0.003921752795577049,
+            0.028504448011517525,
+            0.022806290537118912,
+            0.006429214961826801,
+            -0.013678308576345444,
+            0.005694740451872349,
+            -0.002972060116007924,
+            -0.003522061975672841,
+            0.001583390636369586,
+            0.006979216821491718,
+            0.026195121929049492,
+            0.01956777088344097,
+            -0.03629329428076744,
+            -0.01716279238462448,
+            0.004922688007354736,
+            -0.005459025502204895,
+            -0.000369158893590793,
+            0.010077674873173237,
+            -0.012325508520007133,
+            0.001945503638125956,
+            -0.012455322779715061,
+            0.005592255387455225,
+            -0.002287119859829545,
+            -0.010624260641634464,
+            0.01471682172268629,
+            -0.017190122976899147,
+            0.02967960759997368,
+            0.015454712323844433,
+            0.004905607085675001,
+            0.01520874910056591,
+            0.003979827743023634,
+            0.008308103308081627,
+            -0.027602581307291985,
+            0.01479880977421999,
+            -0.008513073436915874,
+            0.007638535927981138,
+            0.004492251668125391,
+            -0.03315042704343796,
+            -0.014443528838455677,
+            0.0014971325872465968,
+            -0.003959330730140209,
+            -0.007399404887109995,
+            0.001276790164411068,
+            0.014662163332104683,
+            0.014375205151736736,
+            -0.004922688007354736,
+            -0.012011222541332245,
+            0.03533677011728287,
+            -0.002997681265696883,
+            0.015386389568448067,
+            -0.001395501778461039,
+            0.013603152707219124,
+            -0.014429864473640919,
+            0.004847532603889704,
+            -0.013275201432406902,
+            -0.001998454099521041,
+            -0.01212053932249546,
+            0.039053551852703094,
+            0.00012116695870645344,
+            -0.00454007787629962,
+            0.0022768713533878326,
+            -0.019267147406935692,
+            -0.006586358416825533,
+            -0.001919882488436997,
+            -0.01647956110537052,
+            0.004181380849331617,
+            0.0200323686003685,
+            0.011758426204323769,
+            -0.03243986517190933,
+            0.024801328778266907,
+            -0.011157182045280933,
+            0.01702614687383175,
+            0.003424701513722539,
+            0.006548780482262373,
+            0.02093423530459404,
+            -0.00820561870932579,
+            -0.033095769584178925,
+            -0.0020052865147590637,
+            -0.002015535021200776,
+            0.004266784992069006,
+            0.03675789386034012,
+            0.02312057837843895,
+            0.020196344703435898,
+            0.011860910803079605,
+            0.007255925796926022,
+            0.01687583513557911,
+            0.018870873376727104,
+            0.008800030685961246,
+            -0.013268369249999523,
+            -0.006603439338505268,
+            -0.030772779136896133,
+            0.015646018087863922,
+            0.023216230794787407,
+            0.015714339911937714,
+            -0.02168579027056694,
+            0.006391637027263641,
+            0.00696555245667696,
+            0.015113096684217453,
+            0.009886370040476322,
+            -0.00903916172683239,
+            -0.010685751214623451,
+            -0.016096951439976692,
+            0.02847711741924286,
+            -0.027602581307291985,
+            0.006569277495145798,
+            0.0008339703199453652,
+            -0.01647956110537052,
+            -0.015577694401144981,
+            0.02545723132789135,
+            0.04391816630959511,
+            -0.02634543366730213,
+            0.003522061975672841,
+            0.009353448636829853,
+            -0.010521776042878628,
+            0.014279552735388279,
+            0.002939606551080942,
+            -0.012673957273364067,
+            0.013145387172698975,
+            -0.019144166260957718,
+            -0.012632963247597218,
+            -0.012359670363366604,
+            -0.0320572555065155,
+            0.0034349497873336077,
+            0.02913302183151245,
+            0.022314364090561867,
+            0.006381388753652573,
+            -0.00011241305037401617,
+            0.016096951439976692,
+            0.0004517872876022011,
+            0.019718080759048462,
+            -0.01627459190785885,
+            0.009954692795872688,
+            -0.0008621536544524133,
+            0.014730486087501049,
+            -0.008465247228741646,
+            0.01596030406653881,
+            -0.0003898693830706179,
+            0.0007088534184731543,
+            0.00986587256193161,
+            -0.0005717799649573863,
+            -0.014812474139034748,
+            0.00993419624865055,
+            0.005144738592207432,
+            0.014949120581150055,
+            -0.006207164376974106,
+            0.0075565483421087265,
+            -0.03011687472462654,
+            0.01627459190785885,
+            0.02462368831038475,
+            0.009066491387784481,
+            -0.026851026341319084,
+            -0.000849770032800734,
+            0.00523355882614851,
+            -0.010740410536527634,
+            -0.0012486068299040198,
+            0.01837894693017006,
+            0.024159090593457222,
+            0.0029771842528134584,
+            0.028750410303473473,
+            0.001836186507716775,
+            0.019021185114979744,
+            -0.012072713114321232,
+            -0.003141160123050213,
+            0.013254704885184765,
+            0.0012725199339911342,
+            0.0027448853943496943,
+            0.02749326452612877,
+            -0.016971487551927567,
+            0.015482041984796524,
+            -0.00021991536777932197,
+            0.020291997119784355,
+            -0.0064804572612047195,
+            0.00903916172683239,
+            -0.00048039763350971043,
+            0.026564067229628563,
+            0.005732318386435509,
+            0.0065897745080292225,
+            0.0582660436630249,
+            -0.021111875772476196,
+            0.019977709278464317,
+            0.006856235209852457,
+            -0.006576110143214464,
+            -0.015400053933262825,
+            -0.0014894461492076516,
+            -0.01989572122693062,
+            0.026482079178094864,
+            0.014293218031525612,
+            0.0025433318223804235,
+            -0.0026663136668503284,
+            -0.007392572239041328,
+            0.011027367785573006,
+            -0.0012528770603239536,
+            0.0039729950949549675,
+            0.004680140409618616,
+            -0.012933585792779922,
+            0.02394045703113079,
+            0.007932325825095177,
+            -0.004051567055284977,
+            -0.010337303392589092,
+            -0.006463376805186272,
+            0.009018665179610252,
+            -0.016629870980978012,
+            0.001118792686611414,
+            -0.00013034789299126714,
+            0.02250566892325878,
+            -0.017572732642292976,
+            0.035282112658023834,
+            -0.014006259851157665,
+            -0.017449749633669853,
+            -0.010754074901342392,
+            0.011655941605567932,
+            0.024719340726733208,
+            0.01381495501846075,
+            0.020373983308672905,
+            0.01596030406653881,
+            -0.0034264095593243837,
+            0.019526775926351547,
+            -0.0067810798063874245,
+            -0.014033589512109756,
+            0.007064620964229107,
+            0.008000649511814117,
+            -0.0039115045219659805,
+            0.021412497386336327,
+            0.009408107027411461,
+            0.0008587374468334019,
+            0.002954979194328189,
+            -0.0030608803499490023,
+            -0.018419940024614334,
+            -0.005691324360668659,
+            -0.0003687318821903318,
+            0.0046767243184149265,
+            0.0023229895159602165,
+            -0.01505843736231327,
+            0.020988892763853073,
+            -0.0016773350071161985,
+            -0.005534180905669928,
+            0.025429902598261833,
+            0.011123020201921463,
+            -0.0039832438342273235,
+            -0.012468988075852394,
+            -0.012018054723739624,
+            -0.004984179046005011,
+            0.0009881245205178857,
+            0.005383869633078575,
+            0.012175197713077068,
+            0.003908087965101004,
+            -0.006350643467158079,
+            0.005735734477639198,
+            -0.0017746955854818225,
+            0.00561275240033865,
+            0.016383908689022064,
+            0.009838543832302094,
+            0.008007481694221497,
+            -0.021180197596549988,
+            -0.015017443336546421,
+            0.015331730246543884,
+            0.005636665504425764,
+            0.005257471930235624,
+            0.00204286421649158,
+            0.024664683267474174,
+            0.02026466652750969,
+            -0.029734265059232712,
+            -0.011444139294326305,
+            -0.012284515425562859,
+            0.03503614664077759,
+            0.015946639701724052,
+            -0.016588877886533737,
+            0.01174476183950901,
+            0.0011068361345678568,
+            0.004492251668125391,
+            -0.0038500134833157063,
+            0.009660903364419937,
+            0.011006870307028294,
+            -0.021822435781359673,
+            -0.028285812586545944,
+            -0.02064727619290352,
+            -0.013323027640581131,
+            0.032658498734235764,
+            0.02529325522482395,
+            -0.02410443127155304,
+            0.004909023176878691,
+            0.03413427993655205,
+            -0.00540095055475831,
+            -0.009654071182012558,
+            0.006104679778218269,
+            -0.00011123873991891742,
+            -0.01479880977421999,
+            -0.008793198503553867,
+            -0.021330509334802628,
+            0.014525516889989376,
+            0.0034691113978624344,
+            0.005090079735964537,
+            -0.004167716484516859,
+            -0.011997557245194912,
+            -0.027465933933854103,
+            -0.009025497362017632,
+            -0.002867867238819599,
+            -0.0003439646970946342,
+            -0.004475170746445656,
+            0.012967746704816818,
+            -0.0011307492386549711,
+            0.004960265941917896,
+            0.001984789501875639,
+            -0.015031108632683754,
+            -0.007823009043931961,
+            -0.027301957830786705,
+            -0.016329249367117882,
+            0.0034810679499059916,
+            -0.013186381198465824,
+            0.004543493967503309,
+            -0.008929844945669174,
+            -0.014484522864222527,
+            -0.010610596276819706,
+            0.011286996304988861,
+            -0.006425798870623112,
+            0.030772779136896133,
+            0.008314935490489006,
+            -0.011464636772871017,
+            0.0015919309807941318,
+            -0.0022051320411264896,
+            -0.023011259734630585,
+            0.019540440291166306,
+            0.005541013088077307,
+            0.009633573703467846,
+            -0.031374022364616394,
+            0.004471754655241966,
+            0.012933585792779922,
+            0.006531700026243925,
+            -0.023065919056534767,
+            -0.010808733291924,
+            -0.0007246531313285232,
+            -0.0035323104821145535,
+            -0.018542921170592308,
+            -0.014675827696919441,
+            -0.009373946115374565,
+            -0.012065880931913853,
+            -0.020729264244437218,
+            -0.019376466050744057,
+            0.008984503336250782,
+            0.00585529999807477,
+            0.009592579677700996,
+            -0.0030420913826674223,
+            0.007645368110388517,
+            0.013159052468836308,
+            0.0050012595020234585,
+            0.022614985704421997,
+            0.004707470070570707,
+            0.0037270316388458014,
+            0.012899423949420452,
+            -0.018433604389429092,
+            -0.004516164772212505,
+            -0.0016730648931115866,
+            0.02138516865670681,
+            0.007003129925578833,
+            -0.0001629081816645339,
+            0.0004842408234253526,
+            0.02305225469172001,
+            0.0030403833370655775,
+            0.01574167050421238,
+            0.013343525119125843,
+            -0.00589287793263793,
+            0.03216657042503357,
+            -0.00024382848641835153,
+            -0.0004850948753301054,
+            -0.002987432759255171,
+            0.021795107051730156,
+            -0.005845051258802414,
+            0.0004086582630407065,
+            0.013760296627879143,
+            -0.01205221563577652,
+            -0.033095769584178925,
+            -0.005387285724282265,
+            0.026919348165392876,
+            0.015236077830195427,
+            -0.020169014111161232,
+            -0.01000935211777687,
+            0.0016312168445438147,
+            -0.005800641607493162,
+            0.015181419439613819,
+            -0.0019164662808179855,
+            0.00870437826961279,
+            0.017723042517900467,
+            -0.008048475719988346,
+            -0.020155349746346474,
+            0.027834879234433174,
+            0.02732928842306137,
+            0.010193824768066406,
+            -0.038452308624982834,
+            -0.014853468164801598,
+            0.010938547551631927,
+            -0.016670865938067436,
+            -0.019704416394233704,
+            -0.014006259851157665,
+            -0.014730486087501049,
+            -0.007536051329225302,
+            0.01694415882229805,
+            0.008328600786626339,
+            -0.004136971198022366,
+            0.020510630682110786,
+            0.026646055281162262,
+            0.004587904084473848,
+            -0.0035767205990850925,
+            -0.017422420904040337,
+            -0.006593190599232912,
+            -0.0208659116178751,
+            0.0065112030133605,
+            -0.022423680871725082,
+            -0.012906256131827831,
+            -0.010111836716532707,
+            0.002377648139372468,
+            0.02809450775384903,
+            0.005482938606292009,
+            0.03110072948038578,
+            0.0063643078319728374,
+            -0.009353448636829853,
+            -0.008526737801730633,
+            -0.0039729950949549675,
+            -0.0022717469837516546,
+            -0.013548494316637516,
+            -0.007986984215676785,
+            -0.0010163078550249338,
+            -0.005725485738366842,
+            0.012359670363366604,
+            0.016206268221139908,
+            0.004543493967503309,
+            0.025197602808475494,
+            -2.132431836798787e-05,
+            -0.002203423762694001,
+            0.025402573868632317,
+            -0.01590564474463463,
+            0.003627962898463011,
+            0.012475820258259773,
+            -0.0021282683592289686,
+            -0.014880796894431114,
+            0.011198176071047783,
+            0.0325491838157177,
+            -0.014566510915756226,
+            -0.011983892880380154,
+            -0.011983892880380154,
+            0.012332341633737087,
+            -0.004229207523167133,
+            0.03517279401421547,
+            0.021426161751151085,
+            -0.03533677011728287,
+            0.0027226803358644247,
+            -0.019034849479794502,
+            -0.0017951925983652472,
+            -0.023817474022507668,
+            0.0020240754820406437,
+            0.015687011182308197,
+            0.021494485437870026,
+            -0.002029199618846178,
+            0.006524867378175259,
+            -0.02130318060517311,
+            0.01299507636576891,
+            0.008677048608660698,
+            -0.0003403350419830531,
+            0.010665254667401314,
+            -0.0038500134833157063,
+            0.007358410861343145,
+            -0.011109355837106705,
+            -0.013152220286428928,
+            -0.002973768161609769,
+            0.01612428016960621,
+            0.01989572122693062,
+            -0.007713691331446171,
+            0.03028085082769394,
+            -0.002451095497235656,
+            -0.006360891740769148,
+            -0.0368945375084877,
+            -0.0032180235721170902,
+            -0.003962746821343899,
+            0.006705923937261105,
+            -0.0013434053398668766,
+            -0.011164014227688313,
+            -0.011034199967980385,
+            -0.015946639701724052,
+            -0.0030079297721385956,
+            0.004885110072791576,
+            -0.0021026472095400095,
+            -0.0053804535418748856,
+            0.002242709742859006,
+            -0.006029524374753237,
+            0.0054897707886993885,
+            -0.015003778971731663,
+            0.0053087142296135426,
+            0.014334211125969887,
+            -0.006705923937261105,
+            0.0207019355148077,
+            0.022683309391140938,
+            0.01859758049249649,
+            0.0016030335100367665,
+            0.019745411351323128,
+            -0.011020535603165627,
+            -0.017436085268855095,
+            -0.01046711765229702,
+            0.01740875653922558,
+            -0.008314935490489006,
+            0.01852925680577755,
+            0.013685140758752823,
+            0.008888850919902325,
+            0.0005380453658290207,
+            -0.013760296627879143,
+            -0.00783667340874672,
+            0.0009069907246157527,
+            0.01572800613939762,
+            0.01889820210635662,
+            0.004393183160573244,
+            0.0009753139456734061,
+            -0.018624909222126007,
+            0.02200007624924183,
+            -0.015618688426911831,
+            0.002056528814136982,
+            -0.004727966617792845,
+            -0.001228109933435917,
+            0.014088247902691364,
+            -0.007727356161922216,
+            0.008950341492891312,
+            -0.02034665457904339,
+            0.011840414255857468,
+            -0.01117767859250307,
+            -0.0033444215077906847,
+            -0.00602610781788826,
+            -0.012844765558838844,
+            -0.01980006881058216,
+            -0.02508828602731228,
+            0.007085117977112532,
+            -0.0023878966458141804,
+            0.019554104655981064,
+            -0.007631703745573759,
+            -0.020811252295970917,
+            -0.014675827696919441,
+            -0.007986984215676785,
+            0.012223023921251297,
+            0.01747708022594452,
+            -0.00873854011297226,
+            -0.018010001629590988,
+            0.029843581840395927,
+            -0.013992595486342907,
+            -0.008970838971436024,
+            -0.01438887044787407,
+            -0.01672552339732647,
+            -0.0017798198387026787,
+            0.006186667364090681,
+            -0.012585137039422989,
+            -0.014675827696919441,
+            0.014279552735388279,
+            0.0023246975615620613,
+            -0.01649322547018528,
+            -0.016698194667696953,
+            0.005281385034322739,
+            -0.011246002279222012,
+            0.02205473557114601,
+            -0.015700675547122955,
+            0.016520554199814796,
+            -0.011450971476733685,
+            0.01596030406653881,
+            -0.02093423530459404,
+            -0.01973174512386322,
+            0.005834802985191345,
+            -0.011799420230090618,
+            0.029488300904631615,
+            0.001263125566765666,
+            -0.017887018620967865,
+            0.008991335518658161,
+            -0.015290736220777035,
+            0.023544181138277054,
+            -0.0016824592603370547,
+            0.01516775507479906,
+            -0.013220543041825294,
+            0.014511851593852043,
+            0.0012964331544935703,
+            0.014853468164801598,
+            0.008198786526918411,
+            0.038588955998420715,
+            -0.006572693586349487,
+            -0.008943509310483932,
+            0.0008578834240324795,
+            0.008157792501151562,
+            0.0027551339007914066,
+            0.007358410861343145,
+            0.024077102541923523,
+            -0.0012938709696754813,
+            0.005575174931436777,
+            -0.01904851384460926,
+            -0.015290736220777035,
+            -0.007317416835576296,
+            0.015359059907495975,
+            0.017818694934248924,
+            -0.011116188019514084,
+            -0.006784495897591114,
+            0.0055273487232625484,
+            -0.023380205035209656,
+            0.010221153497695923,
+            -0.016247261315584183,
+            -0.013589488342404366,
+            -0.010173327289521694,
+            -0.005964617244899273,
+            -0.008540402166545391,
+            0.011047864332795143,
+            0.014252224005758762,
+            -0.025402573868632317,
+            0.01204538345336914,
+            0.016301920637488365,
+            -0.00626865541562438,
+            0.025129280984401703,
+            -0.027547921985387802,
+            -0.01121867261826992,
+            -0.0023520267568528652,
+            0.011512462981045246,
+            -0.0012725199339911342,
+            -0.006432631053030491,
+            0.009592579677700996,
+            -0.012332341633737087,
+            -0.013691972941160202,
+            -0.002029199618846178,
+            -0.006248158402740955,
+            -0.011170846410095692,
+            0.00023870424774941057,
+            -0.0006435193354263902,
+            0.0065521965734660625,
+            0.012414328753948212,
+            0.00544194458052516,
+            -0.00429411418735981,
+            -0.00956525094807148,
+            0.019485782831907272,
+            0.02463735267519951,
+            0.01596030406653881,
+            0.005510267801582813,
+            -0.01604229211807251,
+            -0.00433169212192297,
+            -0.008547235280275345,
+            -0.02545723132789135,
+            -0.014539181254804134,
+            0.01171059999614954,
+            0.0200323686003685,
+            0.0028217490762472153,
+            -0.02839512936770916,
+            -0.004932936280965805,
+            0.0008907640003599226,
+            0.020291997119784355,
+            0.0019677088130265474,
+            -0.004700637422502041,
+            0.004078896250575781,
+            0.008232948370277882,
+            0.0100571783259511,
+            -0.027069659903645515,
+            -0.015850987285375595,
+            -0.010344135574996471,
+            -0.025511890649795532,
+            0.022164052352309227,
+            0.02817649580538273,
+            0.0134391775354743,
+            0.004106225445866585,
+            0.013500668108463287,
+            0.013773960992693901,
+            0.031975265592336655,
+            -0.00647704117000103,
+            0.01754540205001831,
+            6.079698869143613e-05,
+            -0.007802511565387249,
+            0.018952861428260803,
+            -0.00436585396528244,
+            -0.011758426204323769,
+            -0.013111226260662079,
+            0.010152830742299557,
+            -0.0066615138202905655,
+            -0.008403755724430084,
+            -0.019321806728839874,
+            0.012257185764610767,
+            0.014443528838455677,
+            -0.00865655206143856,
+            -0.010590098798274994,
+            0.004956849385052919,
+            0.007392572239041328,
+            0.004024237859994173,
+            -0.0017217451240867376,
+            -0.0067810798063874245,
+            -0.017067139968276024,
+            -0.005404366645962,
+            -0.0146894920617342,
+            -0.009127981960773468,
+            -0.004174548666924238,
+            0.015113096684217453,
+            -0.017067139968276024,
+            0.018884537741541862,
+            0.014593839645385742,
+            0.014470857568085194,
+            0.019977709278464317,
+            -0.008048475719988346,
+            0.010050345212221146,
+            0.01754540205001831,
+            -0.0014373497106134892,
+            0.0019113420275971293,
+            -0.005605920217931271,
+            0.0007951114675961435,
+            0.008055307902395725,
+            0.0027841711416840553,
+            0.0011162306182086468,
+            -0.003083085408434272,
+            -0.025853507220745087,
+            -0.01876155659556389,
+            -0.0037782741710543633,
+            -0.020196344703435898,
+            -0.011430474929511547,
+            0.003417869098484516,
+            0.0028012520633637905,
+            -0.007167105562984943,
+            -0.012530478648841381,
+            0.0022512502036988735,
+            -0.004495667759329081,
+            0.006658097729086876,
+            -0.0031138306949287653,
+            -0.009052827022969723,
+            0.010672086849808693,
+            0.013138554990291595,
+            -0.016534218564629555,
+            -0.015796327963471413,
+            0.005110576748847961,
+            0.0030386752914637327,
+            0.018870873376727104,
+            -0.0013519456842914224,
+            -0.003747528651729226,
+            -0.008601893670856953,
+            -0.001989913871511817,
+            -0.010521776042878628,
+            -0.007003129925578833,
+            0.008759036660194397,
+            0.016288256272673607,
+            0.005568342283368111,
+            -0.024077102541923523,
+            0.030007557943463326,
+            -0.006559029221534729,
+            0.038534294813871384,
+            0.005564926192164421,
+            -0.005674243438988924,
+            -0.0014313714345917106,
+            0.00956525094807148,
+            -0.011170846410095692,
+            0.012387000024318695,
+            0.009168975986540318,
+            0.010023016482591629,
+            -0.006572693586349487,
+            -0.007057788781821728,
+            0.002792711602523923,
+            0.009715561755001545,
+            -0.0008459268719889224,
+            9.453157690586522e-05,
+            -0.016684530302882195,
+            -0.014320546761155128,
+            -0.008000649511814117,
+            0.026085805147886276,
+            0.005978281609714031,
+            0.004188213497400284,
+            -0.006299400702118874,
+            -0.007986984215676785,
+            -0.0112255048006773,
+            -0.0046152337454259396,
+            -0.03418894112110138,
+            0.011157182045280933,
+            -0.028258483856916428,
+            0.004587904084473848,
+            0.007023626938462257,
+            -5.860851160832681e-05,
+            0.002843954134732485,
+            0.0063028172589838505,
+            0.010091339237987995,
+            0.014101912267506123,
+            0.00741990189999342,
+            -0.013534829951822758,
+            0.0015987633960321546,
+            0.017490744590759277,
+            -0.0004169851599726826,
+            0.0012921629240736365,
+            -0.009175808168947697,
+            -0.016657201573252678,
+            -0.0012981412000954151,
+            -0.016069620847702026,
+            0.002094106748700142,
+            0.010979541577398777,
+            -0.000233366503380239,
+            0.005250639282166958,
+            0.01027581188827753,
+            0.004550326615571976,
+            -0.007235428784042597,
+            6.223818490980193e-05,
+            0.010227985680103302,
+            0.012557808309793472,
+            -0.00457765581086278,
+            -0.016602542251348495,
+            0.015687011182308197,
+            0.016383908689022064,
+            0.02040131390094757,
+            0.0016371951205655932,
+            -0.019718080759048462,
+            -0.0025484561920166016,
+            0.009326119907200336,
+            -0.009346616454422474,
+            0.01966342329978943,
+            0.0013510916614905,
+            0.006012443453073502,
+            0.02146715484559536,
+            0.015482041984796524,
+            -0.009585747495293617,
+            -0.014470857568085194,
+            0.005093495827168226,
+            -0.02327088825404644,
+            -0.0005871526664122939,
+            0.011150349862873554,
+            -0.031073400750756264,
+            0.01627459190785885,
+            -0.020975228399038315,
+            0.012209359556436539,
+            -1.091570175049128e-05,
+            -0.022847285494208336,
+            -0.020087026059627533,
+            0.012837933376431465,
+            0.022095728665590286,
+            0.021193861961364746,
+            -0.0077341883443295956,
+            -0.024719340726733208,
+            -0.007119279354810715,
+            0.02674170769751072,
+            0.006972384639084339,
+            -0.016233596950769424,
+            0.025798847898840904,
+            -0.0075087216682732105,
+            -0.030827436596155167,
+            -0.006125176791101694,
+            -0.02764357440173626,
+            -0.007201267406344414,
+            -0.009025497362017632,
+            0.018706897273659706,
+            -0.027315624058246613,
+            -0.009442268870770931,
+            0.014402534812688828,
+            -0.005076415371149778,
+            -0.00630623335018754,
+            -0.00134169717784971,
+            0.014006259851157665,
+            0.003243644954636693,
+            -0.006808409001678228,
+            -0.0030557559803128242,
+            0.014170235954225063,
+            0.015714339911937714,
+            -0.01559135876595974,
+            -0.009018665179610252,
+            -0.013637314550578594,
+            -0.009237299673259258,
+            0.025716859847307205,
+            -0.00025087432004511356,
+            0.0022205046843737364,
+            -0.011717432178556919,
+            -0.008888850919902325,
+            0.007515554316341877,
+            0.02025100216269493,
+            0.00602269172668457,
+            0.001989913871511817,
+            0.014101912267506123,
+            -0.0017798198387026787,
+            -0.011539791710674763,
+            -1.084230825654231e-05,
+            0.002377648139372468,
+            0.01269445475190878,
+            -0.009770220145583153,
+            0.020455971360206604,
+            -0.007167105562984943,
+            0.014006259851157665,
+            -0.022614985704421997,
+            0.007276422809809446,
+            -0.019185159355401993,
+            -0.008431085385382175,
+            -0.0012964331544935703,
+            -0.004444425459951162,
+            0.00139293959364295,
+            -0.011389480903744698,
+            -0.0032077752985060215,
+            0.0099751902744174,
+            0.008123630657792091,
+            -0.025019962340593338,
+            -0.0012938709696754813,
+            -0.01597396843135357,
+            -0.015932975336909294,
+            -0.013514332473278046,
+            0.014662163332104683,
+            0.015632351860404015,
+            0.0028337056282907724,
+            0.0020462805405259132,
+            -0.020291997119784355,
+            -0.011457803659141064,
+            0.009749723598361015,
+            -0.011587617918848991,
+            0.006319897715002298,
+            -0.0191168375313282,
+            -0.022492004558444023,
+            0.01035096775740385,
+            0.04465605691075325,
+            -0.0030096378177404404,
+            -0.015523036010563374,
+            0.013828619383275509,
+            -0.018843544647097588,
+            -0.012632963247597218,
+            -0.009824879467487335,
+            -0.010111836716532707,
+            -0.005769895855337381,
+            -0.0016149900620803237,
+            0.0033000116236507893,
+            0.02522493340075016,
+            -0.0005888607702217996,
+            0.010665254667401314,
+            0.004195045679807663,
+            0.01762739010155201,
+            -0.005947536323219538,
+            0.0009744599228724837,
+            -0.002644108608365059,
+            0.00280466815456748,
+            0.01597396843135357,
+            0.00010648814350133762,
+            0.005674243438988924,
+            0.008376426994800568,
+            0.0011708892416208982,
+            0.015345395542681217,
+            -0.0068972292356193066,
+            -0.025416238233447075,
+            0.02101622335612774,
+            -0.009018665179610252,
+            -0.004635730292648077,
+            -0.011778922751545906,
+            -0.01550937071442604,
+            0.035965342074632645,
+            0.003768025664612651,
+            -0.008964006789028645,
+            0.018570251762866974,
+            0.002572369296103716,
+            0.0058860452845692635,
+            0.007050956133753061,
+            -0.01876155659556389,
+            -0.010173327289521694,
+            0.018392611294984818,
+            -0.011949731037020683,
+            -0.013541662134230137,
+            0.01257830485701561,
+            -0.0016303628217428923,
+            0.007604374550282955,
+            0.0015031108632683754,
+            -0.013015573844313622,
+            -0.009005000814795494,
+            -0.006261823233217001,
+            0.013193213380873203,
+            -0.0048680296167731285,
+            0.018010001629590988,
+            0.00786400306969881,
+            0.008888850919902325,
+            -0.007727356161922216,
+            0.022259704768657684,
+            -0.020510630682110786,
+            -0.007399404887109995,
+            0.0033375893253833055,
+            0.021070880815386772,
+            -0.014019925147294998,
+            0.013008740730583668,
+            -0.0021880511194467545,
+            -0.010631092824041843,
+            0.0033973720856010914,
+            0.0004987595020793378,
+            0.0009172392310574651,
+            -0.0007302043959498405,
+            0.012762777507305145,
+            -0.0029310660902410746,
+            0.002444263081997633,
+            -0.018802549690008163,
+            0.025921829044818878,
+            -0.008724874816834927,
+            0.013384519144892693,
+            -0.007891331799328327,
+            0.007003129925578833,
+            0.02363983355462551,
+            0.03304110839962959,
+            0.0029242339078336954,
+            0.008444749750196934,
+            -0.0029037368949502707,
+            0.01602862775325775,
+            0.004024237859994173,
+            0.00609784759581089,
+            0.007454063277691603,
+            0.014675827696919441,
+            -0.017203787341713905,
+            -0.020660942420363426,
+            0.013316195458173752,
+            -0.015605023130774498,
+            -0.012188862077891827,
+            -0.016165273264050484,
+            -0.00873854011297226,
+            0.018802549690008163,
+            0.0215354785323143,
+            -0.0015210456913337111,
+            0.006142257712781429,
+            0.016465896740555763,
+            -0.007536051329225302,
+            -0.00108206900767982,
+            -0.0013177840737625957,
+            0.006869899574667215,
+            -0.00034887544461525977,
+            -0.016438566148281097,
+            0.01694415882229805,
+            -0.02273796685039997,
+            0.00948326289653778,
+            -0.007146609015762806,
+            0.0061935000121593475,
+            0.014224894344806671,
+            0.01295408234000206,
+            0.015523036010563374,
+            0.015181419439613819,
+            0.0055615101009607315,
+            0.006753750145435333,
+            -0.005653746426105499,
+            0.012981412000954151,
+            0.009162143804132938,
+            0.00820561870932579,
+            0.014129241928458214,
+            -0.0027380529791116714,
+            -0.01934913545846939,
+            -0.005749398842453957,
+            0.01882988028228283,
+            -0.007966487668454647,
+            -0.009244131855666637,
+            0.0139379370957613,
+            0.023680828511714935,
+            -0.021658461540937424,
+            0.02649574540555477,
+            -0.0031428681686520576,
+            -0.014484522864222527,
+            0.035063475370407104,
+            -0.00564691424369812,
+            0.01949944719672203,
+            -0.021439826115965843,
+            -0.037003856152296066,
+            -0.00477920938283205,
+            0.0028337056282907724,
+            0.016752853989601135,
+            0.005113992840051651,
+            0.0012195694725960493,
+            -0.018037330359220505,
+            -0.006849403027445078,
+            0.01672552339732647,
+            -0.01587831601500511,
+            0.01434787642210722,
+            -0.0039729950949549675,
+            0.0207019355148077,
+            0.0014740735059604049,
+            0.015331730246543884,
+            -0.012188862077891827,
+            -0.007570212706923485,
+            0.030253522098064423,
+            0.005178899969905615,
+            -0.010788236744701862,
+            0.017367763444781303,
+            -0.0043795183300971985,
+            -0.002268330892547965,
+            -0.013275201432406902,
+            0.006572693586349487,
+            -0.0007169668097048998,
+            0.0020992308855056763,
+            0.004652811214327812,
+            -0.003959330730140209,
+            0.02230069972574711,
+            -0.005998778622597456,
+            0.010002519004046917,
+            0.005100328475236893,
+            -0.006104679778218269,
+            0.0055717583745718,
+            0.01589198037981987,
+            -0.02115286886692047,
+            0.010528608225286007,
+            -0.0064087179489433765,
+            0.006289152428507805,
+            -0.01175159402191639,
+            0.00022632066975347698,
+            -0.0070304591208696365,
+            0.004755296278744936,
+            0.01178575586527586,
+            -0.013275201432406902,
+            -0.0032795146107673645,
+            0.007522386498749256,
+            -0.004611817188560963,
+            0.025197602808475494,
+            0.021658461540937424,
+            -0.013124890625476837,
+            -0.013664644211530685,
+            0.01382178720086813,
+            -0.003284638747572899,
+            0.006371140480041504,
+            0.005684491712599993,
+            0.004382934421300888,
+            -0.034899499267339706,
+            -0.012018054723739624,
+            0.004099393263459206,
+            -0.016957823187112808,
+            0.001854975474998355,
+            -0.009223634377121925,
+            -0.024282071739435196,
+            0.022396352142095566,
+            -0.004266784992069006,
+            0.021125540137290955,
+            0.0074472310952842236,
+            -0.011840414255857468,
+            -0.016780182719230652,
+            0.014880796894431114,
+            -0.01649322547018528,
+            -0.0020701936446130276,
+            -0.001913050189614296,
+            -0.020223673433065414,
+            0.015345395542681217,
+            -0.004150635562837124,
+            0.010863391682505608,
+            0.001276790164411068,
+            -0.01269445475190878,
+            -0.0036177143920212984,
+            -0.0077341883443295956,
+            0.008301271125674248,
+            -0.018283294513821602,
+            0.0071397763676941395,
+            0.011526127345860004,
+            -0.002435722853988409,
+            -0.019267147406935692,
+            -0.00782984122633934,
+            -2.091064197884407e-05,
+            0.03670323267579079,
+            -0.0261268001049757,
+            -0.0024015612434595823,
+            -7.245997949212324e-06,
+            -0.012373334728181362,
+            0.0035015649627894163,
+            -0.009271460585296154,
+            -0.0037987709511071444,
+            -0.01964975893497467,
+            -0.006617103703320026,
+            0.007501889485865831,
+            0.008000649511814117,
+            -0.013288866728544235,
+            -0.015482041984796524,
+            -0.010453452356159687,
+            -0.025033628568053246,
+            -0.008718042634427547,
+            -0.02967960759997368,
+            -0.017518073320388794,
+            0.0018652238650247455,
+            -0.002572369296103716,
+            0.02235535718500614,
+            -0.019485782831907272,
+            0.010098172351717949,
+            -0.00495001720264554,
+            0.018939197063446045,
+            -0.008595061488449574,
+            -0.004031070042401552,
+            0.006863067392259836,
+            0.0048680296167731285,
+            0.0009257796336896718,
+            -0.008232948370277882,
+            -0.003805603366345167,
+            -0.00786400306969881,
+            -0.00960624497383833,
+            0.013985763303935528,
+            -0.006319897715002298,
+            0.008977671153843403,
+            -0.017190122976899147,
+            -0.000306386937154457,
+            -0.012585137039422989,
+            -0.008048475719988346,
+            -0.009442268870770931,
+            0.003347837831825018,
+            0.014279552735388279,
+            -0.009537921287119389,
+            -0.03774174675345421,
+            -0.0019010936375707388,
+            -0.0051549868658185005,
+            0.0034298256505280733,
+            -0.01874789223074913,
+            0.015782663598656654,
+            -0.011423642747104168,
+            0.011170846410095692,
+            0.011614947579801083,
+            0.005086663644760847,
+            0.008116798475384712,
+            0.006422382779419422,
+            0.015386389568448067,
+            -0.0011025660205632448,
+            -0.004611817188560963,
+            -0.02318890020251274,
+            0.018460934981703758,
+            -0.01249631680548191,
+            -0.008014313876628876,
+            -0.008403755724430084,
+            0.041841138154268265,
+            0.02266964502632618,
+            0.01212053932249546,
+            0.002522834809496999,
+            0.02891438640654087,
+            0.0031240792013704777,
+            0.018132982775568962,
+            0.011813084594905376,
+            -0.004973930306732655,
+            -0.006418966688215733,
+            -0.01295408234000206,
+            -0.029460972175002098,
+            0.016957823187112808,
+            0.008232948370277882,
+            0.029925569891929626,
+            -0.028504448011517525,
+            0.015031108632683754,
+            -0.005462441593408585,
+            -0.01092488318681717,
+            -0.010023016482591629,
+            0.008519905619323254,
+            -0.0024596359580755234,
+            0.031592655926942825,
+            0.00911431759595871,
+            -0.0029515631031244993,
+            0.008191954344511032,
+            0.013145387172698975,
+            -0.012885759584605694,
+            0.01077457144856453,
+            0.01574167050421238,
+            0.03960013762116432,
+            0.009886370040476322,
+            2.0523655621218495e-05,
+            0.035118136554956436,
+            0.016014963388442993,
+            -0.0036792054306715727,
+            -0.0074745602905750275,
+            -0.016206268221139908,
+            -0.012325508520007133,
+            -0.006439463701099157,
+            0.01988205686211586,
+            -0.005602504126727581,
+            0.009162143804132938,
+            0.007563380524516106,
+            0.00783667340874672,
+            -0.02899637445807457,
+            0.013903775252401829,
+            0.003149700351059437,
+            0.013808122836053371,
+            0.014730486087501049,
+            -0.023475859314203262,
+            0.011977060697972775,
+            -0.02176777832210064,
+            -0.0035903851967304945,
+            -0.001383545226417482,
+            -0.0008151814108714461,
+            0.01792801357805729,
+            -0.023079583421349525,
+            -0.013773960992693901,
+            -0.004673308227211237,
+            0.014525516889989376,
+            0.008007481694221497,
+            0.0103919617831707,
+            0.012188862077891827,
+            0.002838829765096307,
+            -0.005377037450671196,
+            -0.0062549905851483345,
+            0.0018788885790854692,
+            -0.003733864054083824,
+            -0.002818332752212882,
+            -0.006497538182884455,
+            0.009018665179610252,
+            0.006091014947742224,
+            -0.004977346397936344,
+            0.009503760375082493,
+            -0.01627459190785885,
+            -0.013596320524811745,
+            0.02582617662847042,
+            0.002526251133531332,
+            -0.0013135139597579837,
+            0.005295049399137497,
+            0.010952211916446686,
+            0.01475781574845314,
+            0.009770220145583153,
+            0.004013989120721817,
+            0.013241039589047432,
+            -0.005595671944320202,
+            -0.007009962573647499,
+            0.019690752029418945,
+            0.0031052904669195414,
+            -0.018406275659799576,
+            0.0026031145825982094,
+            -0.010870223864912987,
+            -0.0012272557942196727,
+            0.010139165446162224,
+            -0.010405626147985458,
+            -0.015796327963471413,
+            0.0011102523421868682,
+            -0.02275163307785988,
+            -0.004645979031920433,
+            0.01709447056055069,
+            0.0073310816660523415,
+            0.024268407374620438,
+            0.002584325848147273,
+            -0.005985114257782698,
+            -0.0006883564637973905,
+            -0.024159090593457222,
+            0.005284801125526428,
+            -0.003614298300817609,
+            -0.008027978241443634,
+            0.014402534812688828,
+            -0.001230672001838684,
+            0.00241010170429945,
+            -0.011123020201921463,
+            -0.00862239021807909,
+            0.006412134040147066,
+            0.012284515425562859,
+            0.03172930330038071,
+            -0.004266784992069006,
+            -0.008718042634427547,
+            -0.029570288956165314,
+            -0.003781690262258053,
+            0.024364059790968895,
+            -0.003819267963990569,
+            -0.020756594836711884,
+            0.01951311156153679,
+            0.02537524327635765,
+            -0.0008514781366102397,
+            -0.0011316033778712153,
+            -0.007153441198170185,
+            -0.004891942720860243,
+            0.007918661460280418,
+            0.019403794780373573,
+            -0.018337951973080635,
+            0.010439787991344929,
+            0.014976450242102146,
+            0.03817901387810707,
+            0.0025006297510117292,
+            -0.004543493967503309,
+            -0.015523036010563374,
+            -0.0014843220124021173,
+            -0.018556587398052216,
+            0.02861376479268074,
+            -0.000738317787181586,
+            0.03304110839962959,
+            0.020811252295970917,
+            0.008492575958371162,
+            0.0028132086154073477,
+            -0.010808733291924,
+            0.059468530118465424,
+            -0.019089506939053535,
+            -0.007782014552503824,
+            -0.011983892880380154,
+            -0.023257223889231682,
+            -0.010419291444122791,
+            -0.0028217490762472153,
+            -0.012878927402198315,
+            0.015318065881729126,
+            -0.01649322547018528,
+            0.00585529999807477,
+            0.005981697700917721,
+            0.0011879700468853116,
+            0.008451581932604313,
+            -0.022232376039028168,
+            0.021180197596549988,
+            -0.02275163307785988,
+            -0.004400015342980623,
+            0.002869575284421444,
+            -0.0003832505608443171,
+            -0.009312454611063004,
+            0.0031104146037250757,
+            0.004693805240094662,
+            0.005455608945339918,
+            0.001044491189531982,
+            -0.01299507636576891,
+            0.024077102541923523,
+            -0.010576434433460236,
+            -0.017203787341713905,
+            0.013247872702777386,
+            0.037304479628801346,
+            0.00696896854788065,
+            0.008355929516255856,
+            0.022710638120770454,
+            0.003508397378027439,
+            -0.03910820931196213,
+            0.004618649836629629,
+            0.005144738592207432,
+            -0.012762777507305145,
+            0.023079583421349525,
+            0.003781690262258053,
+            -0.00325560150668025,
+            -0.01434787642210722,
+            -0.00412330636754632,
+            -0.0012554391287267208,
+            -0.02312057837843895,
+            0.014525516889989376,
+            0.0016021794872358441,
+            -0.018406275659799576,
+            -0.0081782890483737,
+            0.01175159402191639,
+            0.015468376688659191,
+            0.002534791361540556,
+            -0.004943185020238161,
+            0.003178737824782729,
+            0.004232623614370823,
+            -0.013951601460576057,
+            -0.00436927005648613,
+            -0.00046630596625618637,
+            -0.018488263711333275,
+            0.01306340005248785,
+            0.004837283864617348,
+            -0.01559135876595974,
+            -0.007009962573647499,
+            0.020128021016716957,
+            -0.001632070867344737,
+            0.01611061580479145,
+            -0.004058399237692356,
+            0.002451095497235656,
+            0.01777770183980465,
+            0.026318104937672615,
+            -0.005062750540673733,
+            -0.009093821048736572,
+            -0.016014963388442993,
+            -0.013726134784519672,
+            -0.006825489457696676,
+            0.02432306669652462,
+            -0.0011948023457080126,
+            -0.01129382848739624,
+            0.021877095103263855,
+            -0.0011034200433641672,
+            -0.011416809633374214,
+            -0.004659643396735191,
+            -0.0020326157100498676,
+            -0.021262185648083687,
+            0.008157792501151562,
+            -0.002565536880865693,
+            0.009893202222883701,
+            0.020087026059627533,
+            0.031292036175727844,
+            -0.004512748681008816,
+            -0.003405912546440959,
+            -0.006104679778218269,
+            0.005838219076395035,
+            0.00873854011297226,
+            -0.03033551014959812,
+            -0.012407496571540833,
+            -0.014949120581150055,
+            0.006053437478840351,
+            0.018419940024614334,
+            0.007392572239041328,
+            0.015687011182308197,
+            -0.011000038124620914,
+            -0.014921790920197964,
+            -0.02131684496998787,
+            -0.022765297442674637,
+            0.004406847525388002,
+            0.010596930980682373,
+            0.03500881791114807,
+            0.0009266336564905941,
+            -0.010521776042878628,
+            -0.009701897390186787,
+            -0.010419291444122791,
+            0.0015483750030398369,
+            0.006087598856538534,
+            -0.02605847641825676,
+            -0.012721783481538296,
+            -0.016506889835000038,
+            0.004396599251776934,
+            -0.012892591767013073,
+            -0.013357189483940601,
+            0.04971197620034218,
+            -0.006873316131532192,
+            0.0223690215498209,
+            -0.0340522937476635,
+            -0.00457423971965909,
+            -0.0007259342237375677,
+            -0.02250566892325878,
+            -0.001954043982550502,
+            -0.016397573053836823,
+            0.00914847943931818,
+            -0.022123059257864952,
+            -0.0049295201897621155,
+            -0.006466792896389961,
+            -0.005305298138409853,
+            -0.019253483042120934,
+            0.0038807590026408434,
+            -0.020141685381531715,
+            0.01641123741865158,
+            0.0034981488715857267,
+            0.004526413511484861,
+            0.00953108910471201,
+            -0.030390167608857155,
+            0.011929234489798546,
+            -0.006716172676533461,
+            0.007823009043931961,
+            0.003932001534849405,
+            -0.01249631680548191,
+            -0.012216191738843918,
+            -0.004806538578122854,
+            -0.009278293699026108,
+            0.003733864054083824,
+            -0.0032197318505495787,
+            0.018050994724035263,
+            -0.018255963921546936,
+            -0.008895683102309704,
+            -0.004762128461152315,
+            0.019185159355401993,
+            0.01671185903251171,
+            -0.010043513029813766,
+            -0.005127657670527697,
+            -0.02063361182808876,
+            0.025101950392127037,
+            0.005277968943119049,
+            -0.007570212706923485,
+            0.02567586675286293,
+            0.007980152033269405,
+            0.0001476422039559111,
+            0.014142906293272972,
+            0.01739509217441082,
+            -0.017299439758062363,
+            -0.0012588553363457322,
+            0.00824661273509264,
+            0.00758387753739953,
+            -0.00033094058744609356,
+            0.0018088571960106492,
+            0.010412458330392838,
+            0.008232948370277882,
+            -0.004078896250575781,
+            -0.007249093614518642,
+            -0.029324326664209366,
+            -0.008191954344511032,
+            -0.0005337751354090869,
+            0.00011561570136109367,
+            0.016055956482887268,
+            -0.0020906904246658087,
+            0.0054077827371656895,
+            -0.024432383477687836,
+            -0.015236077830195427,
+            0.013732966966927052,
+            -0.022874614223837852,
+            -0.018816214054822922,
+            0.005950952414423227,
+            0.003788522444665432,
+            -0.0107267452403903,
+            0.0036074661184102297,
+            -0.02297026664018631,
+            -0.02109820954501629,
+            -0.00565033033490181,
+            0.007071453146636486,
+            -0.019677087664604187,
+            0.013186381198465824,
+            -0.019294477999210358,
+            -0.030089545994997025,
+            -0.017449749633669853,
+            0.005482938606292009,
+            0.00630623335018754,
+            0.002254666294902563,
+            -0.0006567569798789918,
+            -0.009326119907200336,
+            0.002830289304256439,
+            -0.01656154915690422,
+            0.01209320966154337,
+            0.009271460585296154,
+            0.004806538578122854,
+            -0.01001618430018425,
+            0.002121435943990946,
+            -0.007126112002879381,
+            0.0031326196622103453,
+            0.010979541577398777,
+            0.026195121929049492,
+            0.009920531883835793,
+            -0.006186667364090681,
+            -0.020988892763853073,
+            -0.002758549991995096,
+            -0.013036070391535759,
+            -0.0031582408118993044,
+            0.009783885441720486,
+            0.004208710510283709,
+            -0.019540440291166306,
+            -0.016438566148281097,
+            -0.026031147688627243,
+            -0.011792588047683239,
+            -0.02327088825404644,
+            -0.004283865913748741,
+            0.025730524212121964,
+            -0.010125501081347466,
+            0.0013254705118015409,
+            0.00015564884233754128,
+            0.01626092568039894,
+            -0.0021436410024762154,
+            -0.026987671852111816,
+            0.015044772997498512,
+            -0.008519905619323254,
+            0.0028729913756251335,
+            0.006073934491723776,
+            -0.00412330636754632,
+            0.0008907640003599226,
+            -0.004242871887981892,
+            0.004140387289226055,
+            -0.014320546761155128,
+            0.005506851710379124,
+            -0.007925493642687798,
+            0.019021185114979744,
+            -0.014033589512109756,
+            -0.000921509403269738,
+            0.009059659205377102,
+            -0.0047860415652394295,
+            -0.014402534812688828,
+            -0.017941677942872047,
+            0.024118097499012947,
+            -0.023981450125575066,
+            0.049520671367645264,
+            -0.004136971198022366,
+            0.006138841155916452,
+            -0.02183610200881958,
+            0.003580136690288782,
+            -0.015714339911937714,
+            -0.013992595486342907,
+            -0.01647956110537052,
+            -0.002818332752212882,
+            0.035118136554956436,
+            0.005759647581726313,
+            0.0014843220124021173,
+            0.00045563047751784325,
+            0.004960265941917896,
+            0.03110072948038578,
+            -0.0013271785574033856,
+            0.01956777088344097,
+            -0.0056708273477852345,
+            -0.007686362136155367,
+            -0.007802511565387249,
+            0.010023016482591629,
+            0.016096951439976692,
+            -0.016069620847702026,
+            0.0019677088130265474,
+            -0.0009052826790139079,
+            0.009196305647492409,
+            0.009421772323548794,
+            0.005277968943119049,
+            -0.0048202034085989,
+            -0.009168975986540318,
+            0.00828077457845211,
+            -0.009995686821639538,
+            -0.00832176860421896,
+            -0.017723042517900467,
+            0.005124241579324007,
+            0.01027581188827753,
+            -0.0025484561920166016,
+            0.014525516889989376,
+            -0.005021756514906883,
+            0.004208710510283709,
+            0.005940704140812159,
+            -0.013985763303935528,
+            0.005318962503224611,
+            0.006019275635480881,
+            -0.030554143711924553,
+            0.02727462910115719,
+            -0.01076773926615715,
+            -0.017299439758062363,
+            -0.003011345863342285,
+            0.003069420577958226,
+            -0.004355605226010084,
+            -0.005421447567641735,
+            -0.008984503336250782,
+            -0.018884537741541862,
+            0.006429214961826801,
+            -0.007467728108167648,
+            -0.01844726875424385,
+            -0.001386107294820249,
+            -0.0024579279124736786,
+            0.035883355885744095,
+            -0.003424701513722539,
+            0.010904385708272457,
+            0.016288256272673607
+          ],
+          "title": "The Structure Of The Mixture-Of-Experts Layer",
+          "keyphrases": [
+            "Mixture-of-Experts layer",
+            "gating network",
+            "neural networks",
+            "sparsity",
+            "hierarchical MoE"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "cbdb8bca-8ea4-4cf7-a1b2-9583066e1c80",
+      "type": "next",
+      "source": {
+        "id": "bec4b8f2-cf26-434f-a587-d63e318fec0c",
+        "properties": {
+          "page_content": "2 The Structure Of The Mixture-Of-Experts Layer\n\nThe Mixture-of-Experts (MoE) layer consists of a set of n \"expert networks\" E1, · · · , En, and a \"gating network\" G whose output is a sparse n-dimensional vector. Figure 1 shows an overview of the MoE module. The experts are themselves neural networks, each with their own parameters.\n\nAlthough in principle we only require that the experts accept the same sized inputs and produce the same-sized outputs, in our initial investigations in this paper, we restrict ourselves to the case where the models are feed-forward networks with identical architectures, but with separate parameters.\n\nLet us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows:\n\n$$y=\\sum_{i=1}^{n}G(x){i}E{i}(x)$$ $$(\\mathbf{l})$$ G(x)iEi(x) (1) We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example. If the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network. In the following we focus on ordinary MoEs. We provide more details on hierarchical MoEs in Appendix B. Our implementation is related to other models of conditional computation. A MoE whose experts are simple weight matrices is similar to the parameterized weight matrix proposed in (Cho & Bengio, 2014). A MoE whose experts have one hidden layer is similar to the block-wise dropout described in (Bengio et al., 2015), where the dropped-out layer is sandwiched between fully-activated layers.\n\n2.1 Gating Network\n\nSoftmax Gating: A simple choice of non-sparse gating function (Jordan & Jacobs, 1994) is to multiply the input by a trainable weight matrix Wg and then apply the Sof tmax function.\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ $\\eqref{eq:walpha}$. Gσ(x) = Sof tmax(x · Wg) (2) Noisy Top-K Gating: We add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to −∞ (which causes the corresponding gate values to equal 0). The sparsity serves to save computation, as described above. While this form of sparsity creates some theoretically scary discontinuities in the output of gating function, we have not yet observed this to be a problem in practice. The noise term helps with load balancing, as will be discussed in Appendix A. The amount of noise per component is controlled by a second trainable weight matrix Wnoise.\n\n$$G(x)=S o f t m a x(K e e p T o p K(H(x),k))$$ $$(4)$$ $$(S)$$ G(x) = Sof tmax(KeepT opK(H(x), k)) (3) $$H(x){i}=(x\\cdot W{g}){i}+S t a n d a r d N o r m a l()\\cdot S o f t p l u s((x\\cdot W{n o i s e}){i})$$ $KeepTopK(v,k){i}=\\begin{cases}v_{i}&\\text{if}v_{i}\\text{is in the top}k\\text{elements of}v.\\ -\\infty&\\text{otherwise.}\\end{cases}$ Training the Gating Network We train the gating network by simple back-propagation, along with the rest of the model. If we choose k > 1, the gate values for the top k experts have nonzero derivatives with respect to the weights of the gating network. This type of occasionally-sensitive behavior is described in (Bengio et al., 2013) with respect to noisy rectifiers. Gradients also backpropagate through the gating network to its inputs. Our method differs here from (Bengio et al., 2015) who use boolean gates and a REINFORCE-style approach to train the gating network.\n\n",
+          "embedding": [
+            0.00986587256193161,
+            -0.03260384127497673,
+            -0.016219932585954666,
+            -0.022328028455376625,
+            0.017736708745360374,
+            0.029624948278069496,
+            -0.004546910524368286,
+            0.026386426761746407,
+            -0.00606368575245142,
+            0.009852208197116852,
+            -0.018788885325193405,
+            -0.02659139782190323,
+            0.0016850214451551437,
+            -0.01340501569211483,
+            -0.004304362926632166,
+            -0.00787083525210619,
+            -0.007932325825095177,
+            0.022382687777280807,
+            -0.00348277622833848,
+            -0.029105691239237785,
+            0.03632062301039696,
+            -0.0069143096916377544,
+            -0.021795107051730156,
+            0.0010393669363111258,
+            0.011690102517604828,
+            0.010815565474331379,
+            0.058430016040802,
+            -0.025744188576936722,
+            -0.05607970058917999,
+            0.010234818793833256,
+            0.020223673433065414,
+            0.003211191389709711,
+            -0.022328028455376625,
+            -0.008465247228741646,
+            -0.07581144571304321,
+            -0.04277033731341362,
+            0.026851026341319084,
+            0.04208710417151451,
+            0.0028746994212269783,
+            0.024746669456362724,
+            0.064114511013031,
+            0.0007968195714056492,
+            -0.00416088430210948,
+            -0.0368945375084877,
+            -0.019854728132486343,
+            -0.012175197713077068,
+            -0.025949159637093544,
+            -0.0034093286376446486,
+            -0.01524974312633276,
+            -0.03134669363498688,
+            4.102061939192936e-05,
+            -0.008957174606621265,
+            0.005318962503224611,
+            0.04080262780189514,
+            -0.05337410047650337,
+            0.022874614223837852,
+            0.01740875653922558,
+            0.021439826115965843,
+            0.03134669363498688,
+            0.02275163307785988,
+            0.04361754283308983,
+            0.00478262547403574,
+            -0.05853933468461037,
+            0.008123630657792091,
+            0.019444787874817848,
+            -0.00873170793056488,
+            0.01642490178346634,
+            0.0034161610528826714,
+            -0.00102143210824579,
+            0.009339784272015095,
+            -0.039490822702646255,
+            0.014552845619618893,
+            -0.02160380221903324,
+            0.0027124318294227123,
+            -0.0009351740591228008,
+            -0.006668346468359232,
+            0.02001870423555374,
+            0.04028337076306343,
+            0.028039850294589996,
+            0.009920531883835793,
+            -0.01302923820912838,
+            0.023366540670394897,
+            0.008595061488449574,
+            0.006244742311537266,
+            0.007324249017983675,
+            -0.02907836250960827,
+            -0.01475781574845314,
+            -0.04747097194194794,
+            -0.010487614199519157,
+            0.006070517934858799,
+            -0.006986049469560385,
+            0.02747959829866886,
+            0.011464636772871017,
+            0.002150473417714238,
+            0.017531737685203552,
+            0.02048330195248127,
+            0.045612581074237823,
+            -0.0037577771581709385,
+            -0.006627352442592382,
+            -0.01876155659556389,
+            -0.0019147582352161407,
+            0.026154128834605217,
+            -0.001622676500119269,
+            0.032795146107673645,
+            0.0015825366135686636,
+            -0.0020411561708897352,
+            0.018816214054822922,
+            0.03131936490535736,
+            0.03260384127497673,
+            0.08685247600078583,
+            -0.0060021947138011456,
+            0.00043107682722620666,
+            -0.019267147406935692,
+            0.03517279401421547,
+            -0.02644108608365059,
+            -0.009086987935006618,
+            -0.04812687635421753,
+            -0.054248638451099396,
+            0.010118668898940086,
+            -0.03875293210148811,
+            0.028039850294589996,
+            0.020893240347504616,
+            -0.023626169189810753,
+            0.0012520230375230312,
+            -0.06941638886928558,
+            0.06351326406002045,
+            0.0015082351164892316,
+            0.007604374550282955,
+            -0.010378297418355942,
+            -0.030854765325784683,
+            0.005377037450671196,
+            0.0025945743545889854,
+            -0.01956777088344097,
+            -0.038889575749635696,
+            0.03028085082769394,
+            0.017299439758062363,
+            0.041540518403053284,
+            0.028504448011517525,
+            -0.005674243438988924,
+            -0.029870912432670593,
+            0.029706936329603195,
+            -0.022778961807489395,
+            -0.007385740056633949,
+            -0.028067179024219513,
+            0.011123020201921463,
+            0.012059048749506474,
+            0.04036535695195198,
+            0.06263872981071472,
+            -0.038670942187309265,
+            0.027834879234433174,
+            -0.04525730013847351,
+            0.04080262780189514,
+            -0.012468988075852394,
+            0.036566589027643204,
+            -0.001954043982550502,
+            0.0030711288563907146,
+            -0.01716279238462448,
+            -0.020578954368829727,
+            -0.014580175280570984,
+            -0.02582617662847042,
+            -0.010754074901342392,
+            -0.028559105470776558,
+            0.005889461375772953,
+            0.049356695264577866,
+            0.01159445010125637,
+            0.05118775740265846,
+            0.009421772323548794,
+            0.019977709278464317,
+            -0.05427596718072891,
+            0.014019925147294998,
+            -0.0017371178837493062,
+            0.008916180580854416,
+            0.04585854709148407,
+            0.008383259177207947,
+            0.0039115045219659805,
+            0.002119727898389101,
+            0.02943364344537258,
+            0.031893279403448105,
+            -0.006326730363070965,
+            0.04028337076306343,
+            -0.03375167027115822,
+            -0.017217451706528664,
+            0.01627459190785885,
+            0.006500954274088144,
+            0.009838543832302094,
+            -0.0024391389451920986,
+            0.03284980356693268,
+            -0.01389694306999445,
+            0.005595671944320202,
+            -0.00036745081888511777,
+            0.042743004858493805,
+            0.039572808891534805,
+            0.04610450938344002,
+            -0.022614985704421997,
+            0.017067139968276024,
+            -0.019854728132486343,
+            0.0023076168727129698,
+            0.015755334869027138,
+            -0.0200323686003685,
+            0.018638573586940765,
+            -0.012223023921251297,
+            -0.012421161867678165,
+            0.011491965502500534,
+            -0.025484561920166016,
+            0.01572800613939762,
+            0.015536700375378132,
+            -0.014853468164801598,
+            -0.019239818677306175,
+            0.013917439617216587,
+            -0.040256042033433914,
+            0.019021185114979744,
+            -0.01000935211777687,
+            -0.01973174512386322,
+            -0.02899637445807457,
+            -0.010453452356159687,
+            -0.005004676058888435,
+            -0.00741990189999342,
+            0.017190122976899147,
+            0.03260384127497673,
+            0.003147992305457592,
+            0.0021487653721123934,
+            0.038370318710803986,
+            0.009387610480189323,
+            0.009585747495293617,
+            -0.0007908412953838706,
+            -0.02303859032690525,
+            -0.016821177676320076,
+            -0.025347914546728134,
+            0.012475820258259773,
+            -0.022396352142095566,
+            -0.030772779136896133,
+            0.02335287630558014,
+            0.007727356161922216,
+            0.01336402166634798,
+            0.015331730246543884,
+            -0.013732966966927052,
+            0.013172716833651066,
+            0.014785144478082657,
+            0.02817649580538273,
+            -0.017531737685203552,
+            -0.02198641188442707,
+            0.020893240347504616,
+            -0.06673812121152878,
+            -0.0074745602905750275,
+            0.006716172676533461,
+            0.02793053165078163,
+            -0.013507500290870667,
+            -0.012926753610372543,
+            -0.007481392472982407,
+            -0.004116474185138941,
+            0.012981412000954151,
+            -0.015277071855962276,
+            -0.010501278564333916,
+            0.019021185114979744,
+            -0.015072102658450603,
+            0.046131838113069534,
+            0.011840414255857468,
+            -0.014484522864222527,
+            -0.015454712323844433,
+            -0.00046801407006569207,
+            0.0370585136115551,
+            0.031292036175727844,
+            -0.0269603431224823,
+            0.009852208197116852,
+            0.008533569984138012,
+            -0.025634871795773506,
+            0.0616002157330513,
+            0.015564029105007648,
+            0.01341184787452221,
+            -0.000460754701634869,
+            0.024131761863827705,
+            -0.04222375154495239,
+            0.013036070391535759,
+            -0.021125540137290955,
+            0.05400267243385315,
+            -0.02295660227537155,
+            0.011539791710674763,
+            -0.04708836227655411,
+            -0.0032009428832679987,
+            0.047170352190732956,
+            0.013418680056929588,
+            0.010965877212584019,
+            0.004526413511484861,
+            0.01973174512386322,
+            0.029215008020401,
+            0.015441047959029675,
+            -0.009121149778366089,
+            -0.017572732642292976,
+            -0.05310080572962761,
+            0.008567731827497482,
+            -0.015482041984796524,
+            0.007495057303458452,
+            -0.03342371806502342,
+            0.0015953471884131432,
+            0.02981625311076641,
+            0.014621169306337833,
+            -0.014361540786921978,
+            -0.002075317781418562,
+            0.027807550504803658,
+            -0.042879652231931686,
+            -0.04968464747071266,
+            -0.007317416835576296,
+            -0.011368983425199986,
+            0.007693194784224033,
+            -0.03361502289772034,
+            0.02048330195248127,
+            0.009394442662596703,
+            0.024828657507896423,
+            -0.005103744566440582,
+            -0.011027367785573006,
+            0.01386278122663498,
+            0.002191467210650444,
+            -0.009209970012307167,
+            -0.03366968408226967,
+            -0.029269667342305183,
+            -0.010166495107114315,
+            0.008137295953929424,
+            0.027848543599247932,
+            -0.016219932585954666,
+            -0.012011222541332245,
+            0.027684569358825684,
+            0.022164052352309227,
+            -0.007153441198170185,
+            0.0025689529720693827,
+            -0.024651017040014267,
+            0.005295049399137497,
+            0.015755334869027138,
+            0.028340471908450127,
+            -0.03960013762116432,
+            0.011321157217025757,
+            -0.00564691424369812,
+            -0.022341692820191383,
+            -0.03312309831380844,
+            0.02380380965769291,
+            -0.0134391775354743,
+            -0.03621130809187889,
+            -0.014279552735388279,
+            0.014812474139034748,
+            -0.0044615063816308975,
+            -0.03172930330038071,
+            -0.013650978915393353,
+            0.006398469675332308,
+            -0.014416199177503586,
+            0.01612428016960621,
+            -0.015085767023265362,
+            -0.0035254780668765306,
+            -0.03880758956074715,
+            0.013036070391535759,
+            0.014006259851157665,
+            0.010938547551631927,
+            0.0009035745752044022,
+            0.006791328079998493,
+            0.017572732642292976,
+            -0.016055956482887268,
+            -0.026755372062325478,
+            -0.024159090593457222,
+            0.0017046643188223243,
+            0.006490706000477076,
+            0.006415550597012043,
+            0.010200656950473785,
+            0.016916830092668533,
+            -0.026454750448465347,
+            -0.011573953554034233,
+            0.007986984215676785,
+            -0.054631248116493225,
+            -0.012298179790377617,
+            -0.03878026083111763,
+            -0.019526775926351547,
+            -0.0013092437293380499,
+            0.028941715136170387,
+            -0.0064462958835065365,
+            0.03427092730998993,
+            -0.0056708273477852345,
+            0.00754971569404006,
+            0.017203787341713905,
+            0.018993854522705078,
+            0.018173975870013237,
+            -0.011806252412497997,
+            -0.050340548157691956,
+            0.028887057676911354,
+            -0.03593801334500313,
+            -0.0045229969546198845,
+            0.00282345712184906,
+            -0.002869575284421444,
+            -0.0057049887254834175,
+            0.0338883176445961,
+            0.0015304401749745011,
+            0.023544181138277054,
+            -0.00381243578158319,
+            0.04353555664420128,
+            -0.027055995538830757,
+            -0.049821291118860245,
+            0.00873170793056488,
+            0.027657238766551018,
+            0.024814993143081665,
+            -0.006521451286971569,
+            -0.026536738499999046,
+            -0.02018267847597599,
+            -0.013514332473278046,
+            0.007966487668454647,
+            0.002003578469157219,
+            -0.00044239286216907203,
+            -0.0013263245346024632,
+            0.006548780482262373,
+            -0.034462232142686844,
+            -0.03645727038383484,
+            0.0040549831464886665,
+            0.011813084594905376,
+            0.03213924169540405,
+            -0.02590816468000412,
+            0.017067139968276024,
+            -0.015454712323844433,
+            -0.038588955998420715,
+            0.0006661514053121209,
+            0.023598840460181236,
+            0.035965342074632645,
+            -0.03921752795577049,
+            0.018419940024614334,
+            0.015850987285375595,
+            -0.005602504126727581,
+            0.004188213497400284,
+            0.025265926495194435,
+            0.025347914546728134,
+            0.022273369133472443,
+            -0.00858822837471962,
+            0.04020138084888458,
+            0.014457193203270435,
+            -0.005564926192164421,
+            0.029706936329603195,
+            -0.016766518354415894,
+            0.02998022921383381,
+            -0.0002463906130287796,
+            0.021426161751151085,
+            -0.027916867285966873,
+            0.019007520750164986,
+            -0.0382336750626564,
+            -0.008642887696623802,
+            0.03304110839962959,
+            -0.026085805147886276,
+            0.0034537387546151876,
+            0.022915607318282127,
+            0.01851559244096279,
+            -0.010631092824041843,
+            0.01725844480097294,
+            -0.04451940953731537,
+            -0.029160350561141968,
+            -0.026195121929049492,
+            -0.006822073366492987,
+            -0.02500629797577858,
+            -0.03552807494997978,
+            -0.056134358048439026,
+            0.008649719879031181,
+            -0.047935571521520615,
+            0.004386350512504578,
+            0.03533677011728287,
+            0.010760907083749771,
+            0.010549104772508144,
+            -0.024432383477687836,
+            0.02809450775384903,
+            -0.018351616337895393,
+            -0.00914847943931818,
+            0.021043552085757256,
+            0.024883316829800606,
+            -0.012346005998551846,
+            -0.013842284679412842,
+            0.01612428016960621,
+            -0.03804237022995949,
+            0.006350643467158079,
+            -0.0020326157100498676,
+            -0.024446047842502594,
+            0.019991373643279076,
+            -0.04684239998459816,
+            0.02380380965769291,
+            0.013876445591449738,
+            0.02877774089574814,
+            0.06039772555232048,
+            -0.024651017040014267,
+            -0.06285735964775085,
+            0.020128021016716957,
+            -0.004143803380429745,
+            0.01336402166634798,
+            0.003747528651729226,
+            -0.006169586908072233,
+            0.002331529976800084,
+            -0.01679384708404541,
+            -0.026031147688627243,
+            0.052526891231536865,
+            0.023475859314203262,
+            0.015263407491147518,
+            -0.010023016482591629,
+            0.0013707346515730023,
+            0.0368945375084877,
+            0.023817474022507668,
+            0.01378079317510128,
+            0.016998816281557083,
+            -0.03929951786994934,
+            0.0037782741710543633,
+            0.0064087179489433765,
+            0.022246040403842926,
+            0.027206305414438248,
+            -0.02711065299808979,
+            -0.027083324268460274,
+            0.020920569077134132,
+            0.04528462886810303,
+            0.011908737011253834,
+            -0.0139379370957613,
+            0.016055956482887268,
+            0.03383366018533707,
+            -0.035965342074632645,
+            -0.003212899435311556,
+            0.009551586583256721,
+            0.0011298952158540487,
+            -0.010719913057982922,
+            -0.02041497826576233,
+            0.009961525909602642,
+            0.014402534812688828,
+            -0.02010069042444229,
+            0.01906217820942402,
+            0.010972709394991398,
+            0.028449788689613342,
+            0.0052267261780798435,
+            -0.006750334054231644,
+            -0.029570288956165314,
+            0.04902874305844307,
+            -0.022027406841516495,
+            0.022997595369815826,
+            0.03774174675345421,
+            -0.0027431773487478495,
+            0.024487042799592018,
+            0.002572369296103716,
+            -0.03812435641884804,
+            0.00813046284019947,
+            0.04451940953731537,
+            0.005575174931436777,
+            -0.028039850294589996,
+            -0.016670865938067436,
+            -0.03145601227879524,
+            0.004348773043602705,
+            -0.01738142780959606,
+            -0.02793053165078163,
+            -0.0014125825837254524,
+            0.005551261827349663,
+            0.016684530302882195,
+            -0.002896904479712248,
+            -0.005985114257782698,
+            -0.006132008973509073,
+            0.02507462166249752,
+            -0.049821291118860245,
+            -0.005663995165377855,
+            0.014443528838455677,
+            -0.029406314715743065,
+            -0.028504448011517525,
+            -0.03711317479610443,
+            -0.004229207523167133,
+            -0.013842284679412842,
+            0.0010385129135102034,
+            -0.001186261884868145,
+            -0.0134391775354743,
+            0.011082026176154613,
+            -0.03940883278846741,
+            0.02508828602731228,
+            -0.012100041843950748,
+            -0.010139165446162224,
+            0.0030813771300017834,
+            0.009230467490851879,
+            -0.010788236744701862,
+            -0.008226115256547928,
+            0.04178648069500923,
+            0.02198641188442707,
+            0.024418719112873077,
+            -0.006641016807407141,
+            -0.007071453146636486,
+            0.005745982751250267,
+            0.000374923663912341,
+            0.03809702768921852,
+            -0.001469803275540471,
+            0.03561006113886833,
+            -0.009121149778366089,
+            -0.012660292908549309,
+            0.006883564405143261,
+            0.03752311319112778,
+            -0.0069962977431714535,
+            0.010207489132881165,
+            0.03088209591805935,
+            0.027015000581741333,
+            -0.017121799290180206,
+            -0.015195083804428577,
+            -0.0025894499849528074,
+            -0.004434177186340094,
+            -0.021057216450572014,
+            0.02198641188442707,
+            0.03350570797920227,
+            0.01171059999614954,
+            -0.009353448636829853,
+            0.022573992609977722,
+            -0.04348089545965195,
+            -0.0011435599299147725,
+            0.004113057628273964,
+            0.012455322779715061,
+            -0.05001259595155716,
+            0.016247261315584183,
+            0.014416199177503586,
+            0.017367763444781303,
+            0.00129045476205647,
+            0.006395053584128618,
+            0.023175235837697983,
+            0.02176777832210064,
+            0.024951640516519547,
+            -0.015850987285375595,
+            2.2431901015806943e-05,
+            0.019841063767671585,
+            0.015618688426911831,
+            0.024268407374620438,
+            0.0021436410024762154,
+            0.024978969246149063,
+            -0.017586397007107735,
+            -0.0138559490442276,
+            0.00282345712184906,
+            -0.023380205035209656,
+            -0.010330471210181713,
+            0.05468590557575226,
+            0.011553456075489521,
+            0.051242414861917496,
+            0.01426588837057352,
+            0.00024169338576029986,
+            0.018993854522705078,
+            -0.020510630682110786,
+            -0.02026466652750969,
+            -0.0007050101994536817,
+            0.0027824630960822105,
+            -0.020387649536132812,
+            -0.00945593323558569,
+            -0.010610596276819706,
+            -0.018187642097473145,
+            -0.018939197063446045,
+            -0.006746917963027954,
+            -0.006326730363070965,
+            0.016848506405949593,
+            -0.002319573424756527,
+            -0.013541662134230137,
+            -0.0320572555065155,
+            0.022041071206331253,
+            0.0199503805488348,
+            0.040556661784648895,
+            -0.013575823977589607,
+            0.011239170096814632,
+            -0.0033819994423538446,
+            -0.004560574889183044,
+            0.014621169306337833,
+            -0.01604229211807251,
+            0.017982671037316322,
+            0.009196305647492409,
+            0.00139293959364295,
+            0.011075193993747234,
+            -0.01077457144856453,
+            -0.04684239998459816,
+            -0.010200656950473785,
+            -0.023557845503091812,
+            0.03284980356693268,
+            0.00672300485894084,
+            0.0018566835206001997,
+            0.0039012557826936245,
+            0.0035459750797599554,
+            -0.008116798475384712,
+            0.016821177676320076,
+            0.013275201432406902,
+            -0.03274048864841461,
+            0.017436085268855095,
+            -0.011382648721337318,
+            0.010255315341055393,
+            -0.034598879516124725,
+            -0.013691972941160202,
+            -0.002481841016560793,
+            0.017586397007107735,
+            0.026482079178094864,
+            -0.02869575284421444,
+            0.010610596276819706,
+            0.02131684496998787,
+            -0.002876407466828823,
+            -0.005698156543076038,
+            0.0021231439895927906,
+            0.0014151446521282196,
+            -0.026304440572857857,
+            -0.012024886906147003,
+            0.005301882047206163,
+            0.004659643396735191,
+            -0.03386098891496658,
+            0.005448776762932539,
+            -0.0293789841234684,
+            -0.005595671944320202,
+            -0.011908737011253834,
+            0.003928584977984428,
+            0.0068152411840856075,
+            0.002565536880865693,
+            0.009134814143180847,
+            -0.01831062324345112,
+            -0.007638535927981138,
+            0.024145426228642464,
+            0.004222374875098467,
+            0.015195083804428577,
+            -0.012414328753948212,
+            0.029542960226535797,
+            -0.016684530302882195,
+            0.021959083154797554,
+            0.014402534812688828,
+            0.007857169955968857,
+            -0.002591158030554652,
+            -0.002437430899590254,
+            -0.04471071437001228,
+            -0.01046711765229702,
+            0.010111836716532707,
+            0.02559387870132923,
+            -0.0005986822070553899,
+            0.014539181254804134,
+            0.02477400004863739,
+            -0.008882018737494946,
+            0.015495706349611282,
+            -0.01701248250901699,
+            -0.030144205316901207,
+            0.020305661484599113,
+            -0.0027380529791116714,
+            0.015318065881729126,
+            0.02425474300980568,
+            -0.03148334100842476,
+            -0.02432306669652462,
+            -0.002088982379063964,
+            0.011813084594905376,
+            0.05137906223535538,
+            0.029706936329603195,
+            0.012182029895484447,
+            0.020906904712319374,
+            -0.008157792501151562,
+            -0.02869575284421444,
+            0.012455322779715061,
+            0.01265346072614193,
+            -0.03110072948038578,
+            0.00634722737595439,
+            0.026755372062325478,
+            -0.008601893670856953,
+            -0.02853177674114704,
+            0.02599015273153782,
+            -0.00689381267875433,
+            0.002553580328822136,
+            -0.004796289838850498,
+            -0.020947899669408798,
+            0.004386350512504578,
+            -0.01602862775325775,
+            0.027903202921152115,
+            -0.024418719112873077,
+            0.008595061488449574,
+            0.010084507055580616,
+            -0.009414940141141415,
+            0.032931793481111526,
+            -0.0005137051921337843,
+            -0.01505843736231327,
+            0.010173327289521694,
+            0.021344173699617386,
+            -0.028887057676911354,
+            0.0005017486400902271,
+            0.002992557128891349,
+            0.01339818350970745,
+            0.03968212753534317,
+            0.018939197063446045,
+            0.016698194667696953,
+            0.03295912221074104,
+            0.020169014111161232,
+            -0.01294725015759468,
+            0.025170274078845978,
+            0.01039879396557808,
+            -0.020237337797880173,
+            0.008615558035671711,
+            -0.0060056112706661224,
+            0.014580175280570984,
+            -0.026851026341319084,
+            -0.007050956133753061,
+            -0.002191467210650444,
+            0.0020087026059627533,
+            -0.020373983308672905,
+            -0.005223310086876154,
+            -0.002256374340504408,
+            -0.023680828511714935,
+            0.008150960318744183,
+            0.028668422251939774,
+            0.023393871262669563,
+            -0.00436585396528244,
+            -0.021508149802684784,
+            0.026673385873436928,
+            -0.02410443127155304,
+            -0.012714951299130917,
+            -0.0037782741710543633,
+            0.015796327963471413,
+            -0.022983931005001068,
+            -0.032494522631168365,
+            0.0019437955925241113,
+            0.03509080410003662,
+            0.02823115512728691,
+            -0.004167716484516859,
+            0.044546738266944885,
+            0.02333921194076538,
+            -0.021617466583848,
+            0.02033299021422863,
+            -0.038288332521915436,
+            0.018939197063446045,
+            0.017531737685203552,
+            -0.04684239998459816,
+            -0.030253522098064423,
+            -0.006258407142013311,
+            -0.011895072646439075,
+            0.008123630657792091,
+            -0.017231116071343422,
+            0.003781690262258053,
+            -0.00754971569404006,
+            -0.023147907108068466,
+            0.03470819443464279,
+            -0.016506889835000038,
+            0.03402496501803398,
+            -0.008369594812393188,
+            0.003199234837666154,
+            -0.015632351860404015,
+            -0.016315585002303123,
+            -0.012468988075852394,
+            -0.020770259201526642,
+            -0.01027581188827753,
+            -0.02681003138422966,
+            -0.021521814167499542,
+            -0.011341654695570469,
+            0.004509332589805126,
+            -0.008458415046334267,
+            0.02363983355462551,
+            0.0268783550709486,
+            0.01133482251316309,
+            0.015331730246543884,
+            -0.0207839235663414,
+            0.043644871562719345,
+            -0.026304440572857857,
+            0.022942937910556793,
+            -0.013186381198465824,
+            -0.015372724272310734,
+            0.009681399911642075,
+            0.020879575982689857,
+            -0.04072063788771629,
+            0.0034229932352900505,
+            -0.01611061580479145,
+            -0.007385740056633949,
+            -0.0215354785323143,
+            0.00495001720264554,
+            0.017818694934248924,
+            -0.008793198503553867,
+            0.00791182927787304,
+            0.00041847972897812724,
+            -0.004543493967503309,
+            0.05099645256996155,
+            -0.0027175559662282467,
+            0.02642742171883583,
+            -0.016301920637488365,
+            -0.019171494990587234,
+            0.013261537067592144,
+            -0.029870912432670593,
+            0.013698806054890156,
+            -0.00832176860421896,
+            -0.037085842341184616,
+            -0.00865655206143856,
+            0.01171059999614954,
+            0.018392611294984818,
+            0.0116422763094306,
+            -0.0038875911850482225,
+            -0.009845376014709473,
+            -0.011389480903744698,
+            -0.00627207150682807,
+            -0.03582869842648506,
+            -0.015919310972094536,
+            0.0021436410024762154,
+            -0.019690752029418945,
+            0.03394297510385513,
+            0.0034434902481734753,
+            0.03741379454731941,
+            -0.011409977450966835,
+            -0.014211229979991913,
+            0.0052643041126430035,
+            -0.017518073320388794,
+            0.0038226840551942587,
+            0.0029771842528134584,
+            0.008458415046334267,
+            -0.018993854522705078,
+            0.03962746635079384,
+            0.009052827022969723,
+            0.014511851593852043,
+            -0.03462620824575424,
+            -0.013295698910951614,
+            -0.04501133784651756,
+            0.006654681637883186,
+            -0.017873354256153107,
+            -0.032357875257730484,
+            -0.0004112203896511346,
+            -0.005698156543076038,
+            0.005882629193365574,
+            -0.016534218564629555,
+            -0.013609985820949078,
+            0.006702507846057415,
+            -0.008226115256547928,
+            0.009223634377121925,
+            -0.01896652579307556,
+            0.010180159471929073,
+            0.004000324755907059,
+            0.03192060813307762,
+            -0.021658461540937424,
+            0.034598879516124725,
+            -0.012011222541332245,
+            0.003979827743023634,
+            -0.00013408431550487876,
+            -0.012632963247597218,
+            0.019239818677306175,
+            0.016766518354415894,
+            -0.0372224897146225,
+            0.011464636772871017,
+            0.0477442666888237,
+            0.03563738986849785,
+            -0.036484599113464355,
+            0.012530478648841381,
+            0.007208099588751793,
+            -0.003062588395550847,
+            0.03186595067381859,
+            0.0013647563755512238,
+            0.01671185903251171,
+            -0.0007998087094165385,
+            -0.02582617662847042,
+            -0.024514371529221535,
+            0.0037201992236077785,
+            0.01378762535750866,
+            0.007392572239041328,
+            -0.031073400750756264,
+            0.01941745914518833,
+            0.002908861031755805,
+            0.029925569891929626,
+            0.025812512263655663,
+            -0.012598801404237747,
+            0.031210046261548996,
+            0.01423855870962143,
+            -0.029242338612675667,
+            0.027985190972685814,
+            0.010747242718935013,
+            0.014060918241739273,
+            0.029652277007699013,
+            0.04749830439686775,
+            0.009326119907200336,
+            -0.022027406841516495,
+            0.001334864879027009,
+            0.012960914522409439,
+            -0.009954692795872688,
+            0.016288256272673607,
+            0.014183900319039822,
+            0.026031147688627243,
+            0.010965877212584019,
+            -0.014019925147294998,
+            -0.010863391682505608,
+            0.04430077597498894,
+            -0.01574167050421238,
+            -0.00499101122841239,
+            -0.016766518354415894,
+            0.035200122743844986,
+            0.027452269569039345,
+            -0.02402244508266449,
+            -0.02335287630558014,
+            -0.005202813073992729,
+            -0.006750334054231644,
+            0.016329249367117882,
+            -0.01687583513557911,
+            -0.01378762535750866,
+            -0.02891438640654087,
+            -0.01596030406653881,
+            0.0018754723714664578,
+            -0.03929951786994934,
+            -0.02839512936770916,
+            -0.0008578834240324795,
+            0.011437307111918926,
+            0.0008672778494656086,
+            0.02206839993596077,
+            -0.012714951299130917,
+            -0.019021185114979744,
+            -0.005202813073992729,
+            0.02847711741924286,
+            -0.01125283446162939,
+            0.021508149802684784,
+            -0.004833867773413658,
+            0.005014924332499504,
+            0.005206229165196419,
+            0.041239894926548004,
+            0.0016380491433665156,
+            -0.02847711741924286,
+            0.02794419787824154,
+            -0.008554067462682724,
+            0.009688233025372028,
+            0.03607466071844101,
+            0.0026663136668503284,
+            -0.02235535718500614,
+            0.02418641932308674,
+            -0.0026321520563215017,
+            -0.02025100216269493,
+            -0.008936677128076553,
+            0.008007481694221497,
+            -0.0027500095311552286,
+            -0.019390130415558815,
+            0.009319286793470383,
+            0.042141761630773544,
+            -0.0028405378106981516,
+            0.02387213334441185,
+            0.053046148270368576,
+            -0.02235535718500614,
+            -0.02387213334441185,
+            0.015372724272310734,
+            -0.028504448011517525,
+            -0.007044123951345682,
+            -0.006832322105765343,
+            0.03826100379228592,
+            -0.03041749820113182,
+            -0.010863391682505608,
+            -0.025361578911542892,
+            7.216640369733796e-05,
+            -0.05495919659733772,
+            0.01904851384460926,
+            0.004266784992069006,
+            0.01505843736231327,
+            -0.041895799338817596,
+            -0.0353914275765419,
+            0.028559105470776558,
+            0.029269667342305183,
+            -0.007433566264808178,
+            -0.00429069809615612,
+            -0.011608115397393703,
+            -0.009763387963175774,
+            0.021371502429246902,
+            0.0008967422763817012,
+            -0.01565968245267868,
+            0.014101912267506123,
+            0.03170197457075119,
+            -0.0014373497106134892,
+            -0.015577694401144981,
+            -0.06045238673686981,
+            -0.017299439758062363,
+            0.014908126555383205,
+            0.0023178651463240385,
+            -0.0020394481252878904,
+            -3.75510826415848e-05,
+            0.0011683270568028092,
+            -0.0024288904387503862,
+            -0.025197602808475494,
+            0.02726096473634243,
+            0.008793198503553867,
+            0.014525516889989376,
+            -0.004587904084473848,
+            -0.011416809633374214,
+            0.014894462190568447,
+            0.017682049423456192,
+            -0.011881408281624317,
+            -0.017121799290180206,
+            0.03894423693418503,
+            0.01973174512386322,
+            -0.001998454099521041,
+            -0.0005862986436113715,
+            0.0010333886602893472,
+            -0.004136971198022366,
+            -0.030062217265367508,
+            0.001253731083124876,
+            -0.0009095528512261808,
+            0.00581772206351161,
+            0.02823115512728691,
+            -0.008724874816834927,
+            0.029652277007699013,
+            0.023462193086743355,
+            0.021357838064432144,
+            -0.0032590175978839397,
+            -0.0043897670693695545,
+            -0.0018618077738210559,
+            0.02538890950381756,
+            0.01208637747913599,
+            0.008670216426253319,
+            0.002106063300743699,
+            -0.0038602619897574186,
+            -0.017354097217321396,
+            0.010781404562294483,
+            -0.013391351327300072,
+            -0.015482041984796524,
+            0.016001299023628235,
+            0.027001336216926575,
+            -0.007604374550282955,
+            -0.004222374875098467,
+            0.0025962824001908302,
+            0.01076773926615715,
+            0.033478379249572754,
+            -0.019267147406935692,
+            -0.007652200758457184,
+            0.021795107051730156,
+            -0.00502858916297555,
+            0.01265346072614193,
+            0.010528608225286007,
+            0.00910748541355133,
+            0.027616245672106743,
+            0.06580892205238342,
+            -0.030909424647688866,
+            0.0016628163866698742,
+            -0.01934913545846939,
+            -0.0011239169398322701,
+            0.011512462981045246,
+            -0.023516852408647537,
+            0.034598879516124725,
+            0.003549391170963645,
+            -0.017969006672501564,
+            0.034680865705013275,
+            0.015564029105007648,
+            0.010945379734039307,
+            0.010296309366822243,
+            -0.01657521352171898,
+            -0.004909023176878691,
+            -0.011088858358561993,
+            0.008328600786626339,
+            0.020606283098459244,
+            0.023065919056534767,
+            0.01679384708404541,
+            0.018474599346518517,
+            0.002625319641083479,
+            0.01770937815308571,
+            -0.0200323686003685,
+            0.021029887720942497,
+            -0.003908087965101004,
+            0.005117409396916628,
+            -0.02251933328807354,
+            0.0028319975826889277,
+            0.0025552883744239807,
+            -0.001964292488992214,
+            0.013698806054890156,
+            0.000991540728136897,
+            -0.011000038124620914,
+            0.003528894390910864,
+            0.0021248520351946354,
+            0.01132799033075571,
+            0.009312454611063004,
+            -0.0032931792084127665,
+            0.004249704070389271,
+            0.018652239814400673,
+            0.022874614223837852,
+            0.012530478648841381,
+            0.0013177840737625957,
+            0.012305011972784996,
+            0.019923051819205284,
+            -0.022273369133472443,
+            0.01088388916105032,
+            -0.030854765325784683,
+            -0.003627962898463011,
+            0.0036689569242298603,
+            -0.010330471210181713,
+            -0.005964617244899273,
+            -0.011505629867315292,
+            -0.008301271125674248,
+            0.01295408234000206,
+            -0.0074472310952842236,
+            0.021726783365011215,
+            -0.001455284538678825,
+            -0.006538532208651304,
+            0.02176777832210064,
+            0.0013391351094469428,
+            -0.0044512576423585415,
+            -0.028559105470776558,
+            -0.0099751902744174,
+            0.021043552085757256,
+            -0.002442555036395788,
+            -0.0005747691029682755,
+            0.02559387870132923,
+            -0.015495706349611282,
+            0.01133482251316309,
+            -0.01686217077076435,
+            -0.015304401516914368,
+            -0.0028217490762472153,
+            -0.015468376688659191,
+            0.016452230513095856,
+            -0.004406847525388002,
+            0.00627207150682807,
+            -0.03448956087231636,
+            0.03142867982387543,
+            0.006627352442592382,
+            -0.0016611083410680294,
+            -0.013664644211530685,
+            -0.030253522098064423,
+            0.0095584187656641,
+            -0.021207528188824654,
+            -0.00017358368495479226,
+            0.02176777832210064,
+            0.011471468955278397,
+            -0.0004603276902344078,
+            -0.009066491387784481,
+            0.0024835490621626377,
+            0.0059680333361029625,
+            -0.0009180932538583875,
+            0.005862132180482149,
+            -0.00028332785586826503,
+            0.015304401516914368,
+            -0.016957823187112808,
+            -0.028449788689613342,
+            -0.02303859032690525,
+            0.01306340005248785,
+            0.00862239021807909,
+            -0.004871445707976818,
+            -0.015523036010563374,
+            0.0057767280377447605,
+            0.004181380849331617,
+            -0.0018020248971879482,
+            -0.0034810679499059916,
+            0.007488225121051073,
+            -0.0010103295790031552,
+            0.001913050189614296,
+            0.002036032034084201,
+            -0.012796939350664616,
+            0.020537959411740303,
+            0.035118136554956436,
+            0.006186667364090681,
+            -0.03320508450269699,
+            0.0021624299697577953,
+            0.00015885148604866117,
+            0.0011461219983175397,
+            -0.0069587198086082935,
+            0.026974007487297058,
+            0.008519905619323254,
+            0.01550937071442604,
+            0.030827436596155167,
+            0.0036347953137010336,
+            -0.0031821539159864187,
+            0.012475820258259773,
+            -0.023434864357113838,
+            -0.02425474300980568,
+            0.0027858791872859,
+            0.009244131855666637,
+            0.010405626147985458,
+            0.019226154312491417,
+            -0.012188862077891827,
+            0.004003740847110748,
+            0.01679384708404541,
+            0.019294477999210358,
+            -0.009920531883835793,
+            -0.01816031150519848,
+            0.01516775507479906,
+            0.054084662348032,
+            0.01590564474463463,
+            -0.006565861403942108,
+            -0.01035779993981123,
+            -0.013377686031162739,
+            0.02801251970231533,
+            0.01092488318681717,
+            0.01596030406653881,
+            -0.0003646751865744591,
+            -0.011840414255857468,
+            0.015919310972094536,
+            0.005981697700917721,
+            -0.0014271012041717768,
+            -0.03910820931196213,
+            0.020073361694812775,
+            -0.0058245547115802765,
+            0.02139883302152157,
+            -0.0005619584699161351,
+            -0.0027397610247135162,
+            0.03899889439344406,
+            0.03683988004922867,
+            -0.011491965502500534,
+            0.04443742334842682,
+            -0.007385740056633949,
+            -0.0011409977450966835,
+            -0.0008967422763817012,
+            -0.0026270279195159674,
+            0.01822863519191742,
+            -0.012899423949420452,
+            0.0008950341725721955,
+            -0.0024271823931485415,
+            0.026413757354021072,
+            0.0042702010832726955,
+            -0.014785144478082657,
+            0.0075087216682732105,
+            -0.006442879792302847,
+            -0.02674170769751072,
+            -0.005899710115045309,
+            0.013760296627879143,
+            -0.010685751214623451,
+            0.005783560685813427,
+            0.006521451286971569,
+            0.023065919056534767,
+            0.01738142780959606,
+            -0.019977709278464317,
+            -0.007495057303458452,
+            -0.008526737801730633,
+            0.01261929888278246,
+            0.004813370760530233,
+            -0.012366502545773983,
+            0.022874614223837852,
+            -0.01964975893497467,
+            0.0030284267850220203,
+            0.016165273264050484,
+            -0.007310584653168917,
+            -0.0011854078620672226,
+            -0.02175411395728588,
+            -0.012305011972784996,
+            -0.04001007601618767,
+            0.013849116861820221,
+            -0.016315585002303123,
+            -0.013773960992693901,
+            0.019786404445767403,
+            -0.027848543599247932,
+            0.025197602808475494,
+            -0.009401274845004082,
+            0.024200083687901497,
+            -0.019923051819205284,
+            0.013849116861820221,
+            0.005708404816687107,
+            -0.017531737685203552,
+            0.005339459516108036,
+            -0.012243521399796009,
+            -0.007228596601635218,
+            -0.0020821501966565847,
+            -0.014921790920197964,
+            0.03593801334500313,
+            0.0323852077126503,
+            -0.02568953111767769,
+            0.03142867982387543,
+            0.02034665457904339,
+            0.02071559987962246,
+            -0.005240391008555889,
+            -0.010836062952876091,
+            0.0199503805488348,
+            -0.016903163865208626,
+            0.03323241323232651,
+            0.0027841711416840553,
+            -0.0008284190553240478,
+            0.0015620396006852388,
+            -0.007529218681156635,
+            0.020988892763853073,
+            -0.007761518005281687,
+            0.008806862868368626,
+            0.021275850012898445,
+            -0.003580136690288782,
+            0.010549104772508144,
+            -0.006357475649565458,
+            0.015768999233841896,
+            0.026536738499999046,
+            -0.009968358092010021,
+            0.03733180835843086,
+            0.00813046284019947,
+            -0.00014177068078424782,
+            0.011983892880380154,
+            0.0018310622544959188,
+            -0.019308142364025116,
+            0.006945055443793535,
+            -0.021426161751151085,
+            0.0030403833370655775,
+            0.01434787642210722,
+            -0.004833867773413658,
+            -0.021808771416544914,
+            0.005752814933657646,
+            -0.004150635562837124,
+            -0.010248483158648014,
+            -0.0074745602905750275,
+            -0.01816031150519848,
+            0.011075193993747234,
+            0.018351616337895393,
+            0.028641093522310257,
+            -0.0030335509218275547,
+            -0.002674854127690196,
+            0.007228596601635218,
+            -0.008567731827497482,
+            0.030007557943463326,
+            -0.029187679290771484,
+            0.008308103308081627,
+            -0.0033905399031937122,
+            0.0017012482276186347,
+            0.02139883302152157,
+            -8.753912698011845e-05,
+            -0.010494446381926537,
+            -0.012188862077891827,
+            0.006832322105765343,
+            -0.009818046353757381,
+            -0.007228596601635218,
+            -0.002570661250501871,
+            0.006729837041348219,
+            0.024268407374620438,
+            0.005804057698696852,
+            0.012714951299130917,
+            -0.03402496501803398,
+            0.000319197541102767,
+            -0.003631379222497344,
+            -0.015154089778661728,
+            -0.0037714417558163404,
+            -0.012646627612411976,
+            -0.002927649999037385,
+            -0.030007557943463326,
+            0.008779534138739109,
+            0.001211883150972426,
+            0.006593190599232912,
+            0.004864613059908152,
+            -0.022642314434051514,
+            0.004488835576921701,
+            -0.006641016807407141,
+            -0.0018600996118038893,
+            0.013452841900289059,
+            -0.0033580863382667303,
+            -0.012236688286066055,
+            0.0002893061318900436,
+            0.014019925147294998,
+            -0.014156571589410305,
+            0.006248158402740955,
+            0.005066166631877422,
+            -0.003921752795577049,
+            0.028504448011517525,
+            0.022806290537118912,
+            0.006429214961826801,
+            -0.013678308576345444,
+            0.005694740451872349,
+            -0.002972060116007924,
+            -0.003522061975672841,
+            0.001583390636369586,
+            0.006979216821491718,
+            0.026195121929049492,
+            0.01956777088344097,
+            -0.03629329428076744,
+            -0.01716279238462448,
+            0.004922688007354736,
+            -0.005459025502204895,
+            -0.000369158893590793,
+            0.010077674873173237,
+            -0.012325508520007133,
+            0.001945503638125956,
+            -0.012455322779715061,
+            0.005592255387455225,
+            -0.002287119859829545,
+            -0.010624260641634464,
+            0.01471682172268629,
+            -0.017190122976899147,
+            0.02967960759997368,
+            0.015454712323844433,
+            0.004905607085675001,
+            0.01520874910056591,
+            0.003979827743023634,
+            0.008308103308081627,
+            -0.027602581307291985,
+            0.01479880977421999,
+            -0.008513073436915874,
+            0.007638535927981138,
+            0.004492251668125391,
+            -0.03315042704343796,
+            -0.014443528838455677,
+            0.0014971325872465968,
+            -0.003959330730140209,
+            -0.007399404887109995,
+            0.001276790164411068,
+            0.014662163332104683,
+            0.014375205151736736,
+            -0.004922688007354736,
+            -0.012011222541332245,
+            0.03533677011728287,
+            -0.002997681265696883,
+            0.015386389568448067,
+            -0.001395501778461039,
+            0.013603152707219124,
+            -0.014429864473640919,
+            0.004847532603889704,
+            -0.013275201432406902,
+            -0.001998454099521041,
+            -0.01212053932249546,
+            0.039053551852703094,
+            0.00012116695870645344,
+            -0.00454007787629962,
+            0.0022768713533878326,
+            -0.019267147406935692,
+            -0.006586358416825533,
+            -0.001919882488436997,
+            -0.01647956110537052,
+            0.004181380849331617,
+            0.0200323686003685,
+            0.011758426204323769,
+            -0.03243986517190933,
+            0.024801328778266907,
+            -0.011157182045280933,
+            0.01702614687383175,
+            0.003424701513722539,
+            0.006548780482262373,
+            0.02093423530459404,
+            -0.00820561870932579,
+            -0.033095769584178925,
+            -0.0020052865147590637,
+            -0.002015535021200776,
+            0.004266784992069006,
+            0.03675789386034012,
+            0.02312057837843895,
+            0.020196344703435898,
+            0.011860910803079605,
+            0.007255925796926022,
+            0.01687583513557911,
+            0.018870873376727104,
+            0.008800030685961246,
+            -0.013268369249999523,
+            -0.006603439338505268,
+            -0.030772779136896133,
+            0.015646018087863922,
+            0.023216230794787407,
+            0.015714339911937714,
+            -0.02168579027056694,
+            0.006391637027263641,
+            0.00696555245667696,
+            0.015113096684217453,
+            0.009886370040476322,
+            -0.00903916172683239,
+            -0.010685751214623451,
+            -0.016096951439976692,
+            0.02847711741924286,
+            -0.027602581307291985,
+            0.006569277495145798,
+            0.0008339703199453652,
+            -0.01647956110537052,
+            -0.015577694401144981,
+            0.02545723132789135,
+            0.04391816630959511,
+            -0.02634543366730213,
+            0.003522061975672841,
+            0.009353448636829853,
+            -0.010521776042878628,
+            0.014279552735388279,
+            0.002939606551080942,
+            -0.012673957273364067,
+            0.013145387172698975,
+            -0.019144166260957718,
+            -0.012632963247597218,
+            -0.012359670363366604,
+            -0.0320572555065155,
+            0.0034349497873336077,
+            0.02913302183151245,
+            0.022314364090561867,
+            0.006381388753652573,
+            -0.00011241305037401617,
+            0.016096951439976692,
+            0.0004517872876022011,
+            0.019718080759048462,
+            -0.01627459190785885,
+            0.009954692795872688,
+            -0.0008621536544524133,
+            0.014730486087501049,
+            -0.008465247228741646,
+            0.01596030406653881,
+            -0.0003898693830706179,
+            0.0007088534184731543,
+            0.00986587256193161,
+            -0.0005717799649573863,
+            -0.014812474139034748,
+            0.00993419624865055,
+            0.005144738592207432,
+            0.014949120581150055,
+            -0.006207164376974106,
+            0.0075565483421087265,
+            -0.03011687472462654,
+            0.01627459190785885,
+            0.02462368831038475,
+            0.009066491387784481,
+            -0.026851026341319084,
+            -0.000849770032800734,
+            0.00523355882614851,
+            -0.010740410536527634,
+            -0.0012486068299040198,
+            0.01837894693017006,
+            0.024159090593457222,
+            0.0029771842528134584,
+            0.028750410303473473,
+            0.001836186507716775,
+            0.019021185114979744,
+            -0.012072713114321232,
+            -0.003141160123050213,
+            0.013254704885184765,
+            0.0012725199339911342,
+            0.0027448853943496943,
+            0.02749326452612877,
+            -0.016971487551927567,
+            0.015482041984796524,
+            -0.00021991536777932197,
+            0.020291997119784355,
+            -0.0064804572612047195,
+            0.00903916172683239,
+            -0.00048039763350971043,
+            0.026564067229628563,
+            0.005732318386435509,
+            0.0065897745080292225,
+            0.0582660436630249,
+            -0.021111875772476196,
+            0.019977709278464317,
+            0.006856235209852457,
+            -0.006576110143214464,
+            -0.015400053933262825,
+            -0.0014894461492076516,
+            -0.01989572122693062,
+            0.026482079178094864,
+            0.014293218031525612,
+            0.0025433318223804235,
+            -0.0026663136668503284,
+            -0.007392572239041328,
+            0.011027367785573006,
+            -0.0012528770603239536,
+            0.0039729950949549675,
+            0.004680140409618616,
+            -0.012933585792779922,
+            0.02394045703113079,
+            0.007932325825095177,
+            -0.004051567055284977,
+            -0.010337303392589092,
+            -0.006463376805186272,
+            0.009018665179610252,
+            -0.016629870980978012,
+            0.001118792686611414,
+            -0.00013034789299126714,
+            0.02250566892325878,
+            -0.017572732642292976,
+            0.035282112658023834,
+            -0.014006259851157665,
+            -0.017449749633669853,
+            -0.010754074901342392,
+            0.011655941605567932,
+            0.024719340726733208,
+            0.01381495501846075,
+            0.020373983308672905,
+            0.01596030406653881,
+            -0.0034264095593243837,
+            0.019526775926351547,
+            -0.0067810798063874245,
+            -0.014033589512109756,
+            0.007064620964229107,
+            0.008000649511814117,
+            -0.0039115045219659805,
+            0.021412497386336327,
+            0.009408107027411461,
+            0.0008587374468334019,
+            0.002954979194328189,
+            -0.0030608803499490023,
+            -0.018419940024614334,
+            -0.005691324360668659,
+            -0.0003687318821903318,
+            0.0046767243184149265,
+            0.0023229895159602165,
+            -0.01505843736231327,
+            0.020988892763853073,
+            -0.0016773350071161985,
+            -0.005534180905669928,
+            0.025429902598261833,
+            0.011123020201921463,
+            -0.0039832438342273235,
+            -0.012468988075852394,
+            -0.012018054723739624,
+            -0.004984179046005011,
+            0.0009881245205178857,
+            0.005383869633078575,
+            0.012175197713077068,
+            0.003908087965101004,
+            -0.006350643467158079,
+            0.005735734477639198,
+            -0.0017746955854818225,
+            0.00561275240033865,
+            0.016383908689022064,
+            0.009838543832302094,
+            0.008007481694221497,
+            -0.021180197596549988,
+            -0.015017443336546421,
+            0.015331730246543884,
+            0.005636665504425764,
+            0.005257471930235624,
+            0.00204286421649158,
+            0.024664683267474174,
+            0.02026466652750969,
+            -0.029734265059232712,
+            -0.011444139294326305,
+            -0.012284515425562859,
+            0.03503614664077759,
+            0.015946639701724052,
+            -0.016588877886533737,
+            0.01174476183950901,
+            0.0011068361345678568,
+            0.004492251668125391,
+            -0.0038500134833157063,
+            0.009660903364419937,
+            0.011006870307028294,
+            -0.021822435781359673,
+            -0.028285812586545944,
+            -0.02064727619290352,
+            -0.013323027640581131,
+            0.032658498734235764,
+            0.02529325522482395,
+            -0.02410443127155304,
+            0.004909023176878691,
+            0.03413427993655205,
+            -0.00540095055475831,
+            -0.009654071182012558,
+            0.006104679778218269,
+            -0.00011123873991891742,
+            -0.01479880977421999,
+            -0.008793198503553867,
+            -0.021330509334802628,
+            0.014525516889989376,
+            0.0034691113978624344,
+            0.005090079735964537,
+            -0.004167716484516859,
+            -0.011997557245194912,
+            -0.027465933933854103,
+            -0.009025497362017632,
+            -0.002867867238819599,
+            -0.0003439646970946342,
+            -0.004475170746445656,
+            0.012967746704816818,
+            -0.0011307492386549711,
+            0.004960265941917896,
+            0.001984789501875639,
+            -0.015031108632683754,
+            -0.007823009043931961,
+            -0.027301957830786705,
+            -0.016329249367117882,
+            0.0034810679499059916,
+            -0.013186381198465824,
+            0.004543493967503309,
+            -0.008929844945669174,
+            -0.014484522864222527,
+            -0.010610596276819706,
+            0.011286996304988861,
+            -0.006425798870623112,
+            0.030772779136896133,
+            0.008314935490489006,
+            -0.011464636772871017,
+            0.0015919309807941318,
+            -0.0022051320411264896,
+            -0.023011259734630585,
+            0.019540440291166306,
+            0.005541013088077307,
+            0.009633573703467846,
+            -0.031374022364616394,
+            0.004471754655241966,
+            0.012933585792779922,
+            0.006531700026243925,
+            -0.023065919056534767,
+            -0.010808733291924,
+            -0.0007246531313285232,
+            -0.0035323104821145535,
+            -0.018542921170592308,
+            -0.014675827696919441,
+            -0.009373946115374565,
+            -0.012065880931913853,
+            -0.020729264244437218,
+            -0.019376466050744057,
+            0.008984503336250782,
+            0.00585529999807477,
+            0.009592579677700996,
+            -0.0030420913826674223,
+            0.007645368110388517,
+            0.013159052468836308,
+            0.0050012595020234585,
+            0.022614985704421997,
+            0.004707470070570707,
+            0.0037270316388458014,
+            0.012899423949420452,
+            -0.018433604389429092,
+            -0.004516164772212505,
+            -0.0016730648931115866,
+            0.02138516865670681,
+            0.007003129925578833,
+            -0.0001629081816645339,
+            0.0004842408234253526,
+            0.02305225469172001,
+            0.0030403833370655775,
+            0.01574167050421238,
+            0.013343525119125843,
+            -0.00589287793263793,
+            0.03216657042503357,
+            -0.00024382848641835153,
+            -0.0004850948753301054,
+            -0.002987432759255171,
+            0.021795107051730156,
+            -0.005845051258802414,
+            0.0004086582630407065,
+            0.013760296627879143,
+            -0.01205221563577652,
+            -0.033095769584178925,
+            -0.005387285724282265,
+            0.026919348165392876,
+            0.015236077830195427,
+            -0.020169014111161232,
+            -0.01000935211777687,
+            0.0016312168445438147,
+            -0.005800641607493162,
+            0.015181419439613819,
+            -0.0019164662808179855,
+            0.00870437826961279,
+            0.017723042517900467,
+            -0.008048475719988346,
+            -0.020155349746346474,
+            0.027834879234433174,
+            0.02732928842306137,
+            0.010193824768066406,
+            -0.038452308624982834,
+            -0.014853468164801598,
+            0.010938547551631927,
+            -0.016670865938067436,
+            -0.019704416394233704,
+            -0.014006259851157665,
+            -0.014730486087501049,
+            -0.007536051329225302,
+            0.01694415882229805,
+            0.008328600786626339,
+            -0.004136971198022366,
+            0.020510630682110786,
+            0.026646055281162262,
+            0.004587904084473848,
+            -0.0035767205990850925,
+            -0.017422420904040337,
+            -0.006593190599232912,
+            -0.0208659116178751,
+            0.0065112030133605,
+            -0.022423680871725082,
+            -0.012906256131827831,
+            -0.010111836716532707,
+            0.002377648139372468,
+            0.02809450775384903,
+            0.005482938606292009,
+            0.03110072948038578,
+            0.0063643078319728374,
+            -0.009353448636829853,
+            -0.008526737801730633,
+            -0.0039729950949549675,
+            -0.0022717469837516546,
+            -0.013548494316637516,
+            -0.007986984215676785,
+            -0.0010163078550249338,
+            -0.005725485738366842,
+            0.012359670363366604,
+            0.016206268221139908,
+            0.004543493967503309,
+            0.025197602808475494,
+            -2.132431836798787e-05,
+            -0.002203423762694001,
+            0.025402573868632317,
+            -0.01590564474463463,
+            0.003627962898463011,
+            0.012475820258259773,
+            -0.0021282683592289686,
+            -0.014880796894431114,
+            0.011198176071047783,
+            0.0325491838157177,
+            -0.014566510915756226,
+            -0.011983892880380154,
+            -0.011983892880380154,
+            0.012332341633737087,
+            -0.004229207523167133,
+            0.03517279401421547,
+            0.021426161751151085,
+            -0.03533677011728287,
+            0.0027226803358644247,
+            -0.019034849479794502,
+            -0.0017951925983652472,
+            -0.023817474022507668,
+            0.0020240754820406437,
+            0.015687011182308197,
+            0.021494485437870026,
+            -0.002029199618846178,
+            0.006524867378175259,
+            -0.02130318060517311,
+            0.01299507636576891,
+            0.008677048608660698,
+            -0.0003403350419830531,
+            0.010665254667401314,
+            -0.0038500134833157063,
+            0.007358410861343145,
+            -0.011109355837106705,
+            -0.013152220286428928,
+            -0.002973768161609769,
+            0.01612428016960621,
+            0.01989572122693062,
+            -0.007713691331446171,
+            0.03028085082769394,
+            -0.002451095497235656,
+            -0.006360891740769148,
+            -0.0368945375084877,
+            -0.0032180235721170902,
+            -0.003962746821343899,
+            0.006705923937261105,
+            -0.0013434053398668766,
+            -0.011164014227688313,
+            -0.011034199967980385,
+            -0.015946639701724052,
+            -0.0030079297721385956,
+            0.004885110072791576,
+            -0.0021026472095400095,
+            -0.0053804535418748856,
+            0.002242709742859006,
+            -0.006029524374753237,
+            0.0054897707886993885,
+            -0.015003778971731663,
+            0.0053087142296135426,
+            0.014334211125969887,
+            -0.006705923937261105,
+            0.0207019355148077,
+            0.022683309391140938,
+            0.01859758049249649,
+            0.0016030335100367665,
+            0.019745411351323128,
+            -0.011020535603165627,
+            -0.017436085268855095,
+            -0.01046711765229702,
+            0.01740875653922558,
+            -0.008314935490489006,
+            0.01852925680577755,
+            0.013685140758752823,
+            0.008888850919902325,
+            0.0005380453658290207,
+            -0.013760296627879143,
+            -0.00783667340874672,
+            0.0009069907246157527,
+            0.01572800613939762,
+            0.01889820210635662,
+            0.004393183160573244,
+            0.0009753139456734061,
+            -0.018624909222126007,
+            0.02200007624924183,
+            -0.015618688426911831,
+            0.002056528814136982,
+            -0.004727966617792845,
+            -0.001228109933435917,
+            0.014088247902691364,
+            -0.007727356161922216,
+            0.008950341492891312,
+            -0.02034665457904339,
+            0.011840414255857468,
+            -0.01117767859250307,
+            -0.0033444215077906847,
+            -0.00602610781788826,
+            -0.012844765558838844,
+            -0.01980006881058216,
+            -0.02508828602731228,
+            0.007085117977112532,
+            -0.0023878966458141804,
+            0.019554104655981064,
+            -0.007631703745573759,
+            -0.020811252295970917,
+            -0.014675827696919441,
+            -0.007986984215676785,
+            0.012223023921251297,
+            0.01747708022594452,
+            -0.00873854011297226,
+            -0.018010001629590988,
+            0.029843581840395927,
+            -0.013992595486342907,
+            -0.008970838971436024,
+            -0.01438887044787407,
+            -0.01672552339732647,
+            -0.0017798198387026787,
+            0.006186667364090681,
+            -0.012585137039422989,
+            -0.014675827696919441,
+            0.014279552735388279,
+            0.0023246975615620613,
+            -0.01649322547018528,
+            -0.016698194667696953,
+            0.005281385034322739,
+            -0.011246002279222012,
+            0.02205473557114601,
+            -0.015700675547122955,
+            0.016520554199814796,
+            -0.011450971476733685,
+            0.01596030406653881,
+            -0.02093423530459404,
+            -0.01973174512386322,
+            0.005834802985191345,
+            -0.011799420230090618,
+            0.029488300904631615,
+            0.001263125566765666,
+            -0.017887018620967865,
+            0.008991335518658161,
+            -0.015290736220777035,
+            0.023544181138277054,
+            -0.0016824592603370547,
+            0.01516775507479906,
+            -0.013220543041825294,
+            0.014511851593852043,
+            0.0012964331544935703,
+            0.014853468164801598,
+            0.008198786526918411,
+            0.038588955998420715,
+            -0.006572693586349487,
+            -0.008943509310483932,
+            0.0008578834240324795,
+            0.008157792501151562,
+            0.0027551339007914066,
+            0.007358410861343145,
+            0.024077102541923523,
+            -0.0012938709696754813,
+            0.005575174931436777,
+            -0.01904851384460926,
+            -0.015290736220777035,
+            -0.007317416835576296,
+            0.015359059907495975,
+            0.017818694934248924,
+            -0.011116188019514084,
+            -0.006784495897591114,
+            0.0055273487232625484,
+            -0.023380205035209656,
+            0.010221153497695923,
+            -0.016247261315584183,
+            -0.013589488342404366,
+            -0.010173327289521694,
+            -0.005964617244899273,
+            -0.008540402166545391,
+            0.011047864332795143,
+            0.014252224005758762,
+            -0.025402573868632317,
+            0.01204538345336914,
+            0.016301920637488365,
+            -0.00626865541562438,
+            0.025129280984401703,
+            -0.027547921985387802,
+            -0.01121867261826992,
+            -0.0023520267568528652,
+            0.011512462981045246,
+            -0.0012725199339911342,
+            -0.006432631053030491,
+            0.009592579677700996,
+            -0.012332341633737087,
+            -0.013691972941160202,
+            -0.002029199618846178,
+            -0.006248158402740955,
+            -0.011170846410095692,
+            0.00023870424774941057,
+            -0.0006435193354263902,
+            0.0065521965734660625,
+            0.012414328753948212,
+            0.00544194458052516,
+            -0.00429411418735981,
+            -0.00956525094807148,
+            0.019485782831907272,
+            0.02463735267519951,
+            0.01596030406653881,
+            0.005510267801582813,
+            -0.01604229211807251,
+            -0.00433169212192297,
+            -0.008547235280275345,
+            -0.02545723132789135,
+            -0.014539181254804134,
+            0.01171059999614954,
+            0.0200323686003685,
+            0.0028217490762472153,
+            -0.02839512936770916,
+            -0.004932936280965805,
+            0.0008907640003599226,
+            0.020291997119784355,
+            0.0019677088130265474,
+            -0.004700637422502041,
+            0.004078896250575781,
+            0.008232948370277882,
+            0.0100571783259511,
+            -0.027069659903645515,
+            -0.015850987285375595,
+            -0.010344135574996471,
+            -0.025511890649795532,
+            0.022164052352309227,
+            0.02817649580538273,
+            0.0134391775354743,
+            0.004106225445866585,
+            0.013500668108463287,
+            0.013773960992693901,
+            0.031975265592336655,
+            -0.00647704117000103,
+            0.01754540205001831,
+            6.079698869143613e-05,
+            -0.007802511565387249,
+            0.018952861428260803,
+            -0.00436585396528244,
+            -0.011758426204323769,
+            -0.013111226260662079,
+            0.010152830742299557,
+            -0.0066615138202905655,
+            -0.008403755724430084,
+            -0.019321806728839874,
+            0.012257185764610767,
+            0.014443528838455677,
+            -0.00865655206143856,
+            -0.010590098798274994,
+            0.004956849385052919,
+            0.007392572239041328,
+            0.004024237859994173,
+            -0.0017217451240867376,
+            -0.0067810798063874245,
+            -0.017067139968276024,
+            -0.005404366645962,
+            -0.0146894920617342,
+            -0.009127981960773468,
+            -0.004174548666924238,
+            0.015113096684217453,
+            -0.017067139968276024,
+            0.018884537741541862,
+            0.014593839645385742,
+            0.014470857568085194,
+            0.019977709278464317,
+            -0.008048475719988346,
+            0.010050345212221146,
+            0.01754540205001831,
+            -0.0014373497106134892,
+            0.0019113420275971293,
+            -0.005605920217931271,
+            0.0007951114675961435,
+            0.008055307902395725,
+            0.0027841711416840553,
+            0.0011162306182086468,
+            -0.003083085408434272,
+            -0.025853507220745087,
+            -0.01876155659556389,
+            -0.0037782741710543633,
+            -0.020196344703435898,
+            -0.011430474929511547,
+            0.003417869098484516,
+            0.0028012520633637905,
+            -0.007167105562984943,
+            -0.012530478648841381,
+            0.0022512502036988735,
+            -0.004495667759329081,
+            0.006658097729086876,
+            -0.0031138306949287653,
+            -0.009052827022969723,
+            0.010672086849808693,
+            0.013138554990291595,
+            -0.016534218564629555,
+            -0.015796327963471413,
+            0.005110576748847961,
+            0.0030386752914637327,
+            0.018870873376727104,
+            -0.0013519456842914224,
+            -0.003747528651729226,
+            -0.008601893670856953,
+            -0.001989913871511817,
+            -0.010521776042878628,
+            -0.007003129925578833,
+            0.008759036660194397,
+            0.016288256272673607,
+            0.005568342283368111,
+            -0.024077102541923523,
+            0.030007557943463326,
+            -0.006559029221534729,
+            0.038534294813871384,
+            0.005564926192164421,
+            -0.005674243438988924,
+            -0.0014313714345917106,
+            0.00956525094807148,
+            -0.011170846410095692,
+            0.012387000024318695,
+            0.009168975986540318,
+            0.010023016482591629,
+            -0.006572693586349487,
+            -0.007057788781821728,
+            0.002792711602523923,
+            0.009715561755001545,
+            -0.0008459268719889224,
+            9.453157690586522e-05,
+            -0.016684530302882195,
+            -0.014320546761155128,
+            -0.008000649511814117,
+            0.026085805147886276,
+            0.005978281609714031,
+            0.004188213497400284,
+            -0.006299400702118874,
+            -0.007986984215676785,
+            -0.0112255048006773,
+            -0.0046152337454259396,
+            -0.03418894112110138,
+            0.011157182045280933,
+            -0.028258483856916428,
+            0.004587904084473848,
+            0.007023626938462257,
+            -5.860851160832681e-05,
+            0.002843954134732485,
+            0.0063028172589838505,
+            0.010091339237987995,
+            0.014101912267506123,
+            0.00741990189999342,
+            -0.013534829951822758,
+            0.0015987633960321546,
+            0.017490744590759277,
+            -0.0004169851599726826,
+            0.0012921629240736365,
+            -0.009175808168947697,
+            -0.016657201573252678,
+            -0.0012981412000954151,
+            -0.016069620847702026,
+            0.002094106748700142,
+            0.010979541577398777,
+            -0.000233366503380239,
+            0.005250639282166958,
+            0.01027581188827753,
+            0.004550326615571976,
+            -0.007235428784042597,
+            6.223818490980193e-05,
+            0.010227985680103302,
+            0.012557808309793472,
+            -0.00457765581086278,
+            -0.016602542251348495,
+            0.015687011182308197,
+            0.016383908689022064,
+            0.02040131390094757,
+            0.0016371951205655932,
+            -0.019718080759048462,
+            -0.0025484561920166016,
+            0.009326119907200336,
+            -0.009346616454422474,
+            0.01966342329978943,
+            0.0013510916614905,
+            0.006012443453073502,
+            0.02146715484559536,
+            0.015482041984796524,
+            -0.009585747495293617,
+            -0.014470857568085194,
+            0.005093495827168226,
+            -0.02327088825404644,
+            -0.0005871526664122939,
+            0.011150349862873554,
+            -0.031073400750756264,
+            0.01627459190785885,
+            -0.020975228399038315,
+            0.012209359556436539,
+            -1.091570175049128e-05,
+            -0.022847285494208336,
+            -0.020087026059627533,
+            0.012837933376431465,
+            0.022095728665590286,
+            0.021193861961364746,
+            -0.0077341883443295956,
+            -0.024719340726733208,
+            -0.007119279354810715,
+            0.02674170769751072,
+            0.006972384639084339,
+            -0.016233596950769424,
+            0.025798847898840904,
+            -0.0075087216682732105,
+            -0.030827436596155167,
+            -0.006125176791101694,
+            -0.02764357440173626,
+            -0.007201267406344414,
+            -0.009025497362017632,
+            0.018706897273659706,
+            -0.027315624058246613,
+            -0.009442268870770931,
+            0.014402534812688828,
+            -0.005076415371149778,
+            -0.00630623335018754,
+            -0.00134169717784971,
+            0.014006259851157665,
+            0.003243644954636693,
+            -0.006808409001678228,
+            -0.0030557559803128242,
+            0.014170235954225063,
+            0.015714339911937714,
+            -0.01559135876595974,
+            -0.009018665179610252,
+            -0.013637314550578594,
+            -0.009237299673259258,
+            0.025716859847307205,
+            -0.00025087432004511356,
+            0.0022205046843737364,
+            -0.011717432178556919,
+            -0.008888850919902325,
+            0.007515554316341877,
+            0.02025100216269493,
+            0.00602269172668457,
+            0.001989913871511817,
+            0.014101912267506123,
+            -0.0017798198387026787,
+            -0.011539791710674763,
+            -1.084230825654231e-05,
+            0.002377648139372468,
+            0.01269445475190878,
+            -0.009770220145583153,
+            0.020455971360206604,
+            -0.007167105562984943,
+            0.014006259851157665,
+            -0.022614985704421997,
+            0.007276422809809446,
+            -0.019185159355401993,
+            -0.008431085385382175,
+            -0.0012964331544935703,
+            -0.004444425459951162,
+            0.00139293959364295,
+            -0.011389480903744698,
+            -0.0032077752985060215,
+            0.0099751902744174,
+            0.008123630657792091,
+            -0.025019962340593338,
+            -0.0012938709696754813,
+            -0.01597396843135357,
+            -0.015932975336909294,
+            -0.013514332473278046,
+            0.014662163332104683,
+            0.015632351860404015,
+            0.0028337056282907724,
+            0.0020462805405259132,
+            -0.020291997119784355,
+            -0.011457803659141064,
+            0.009749723598361015,
+            -0.011587617918848991,
+            0.006319897715002298,
+            -0.0191168375313282,
+            -0.022492004558444023,
+            0.01035096775740385,
+            0.04465605691075325,
+            -0.0030096378177404404,
+            -0.015523036010563374,
+            0.013828619383275509,
+            -0.018843544647097588,
+            -0.012632963247597218,
+            -0.009824879467487335,
+            -0.010111836716532707,
+            -0.005769895855337381,
+            -0.0016149900620803237,
+            0.0033000116236507893,
+            0.02522493340075016,
+            -0.0005888607702217996,
+            0.010665254667401314,
+            0.004195045679807663,
+            0.01762739010155201,
+            -0.005947536323219538,
+            0.0009744599228724837,
+            -0.002644108608365059,
+            0.00280466815456748,
+            0.01597396843135357,
+            0.00010648814350133762,
+            0.005674243438988924,
+            0.008376426994800568,
+            0.0011708892416208982,
+            0.015345395542681217,
+            -0.0068972292356193066,
+            -0.025416238233447075,
+            0.02101622335612774,
+            -0.009018665179610252,
+            -0.004635730292648077,
+            -0.011778922751545906,
+            -0.01550937071442604,
+            0.035965342074632645,
+            0.003768025664612651,
+            -0.008964006789028645,
+            0.018570251762866974,
+            0.002572369296103716,
+            0.0058860452845692635,
+            0.007050956133753061,
+            -0.01876155659556389,
+            -0.010173327289521694,
+            0.018392611294984818,
+            -0.011949731037020683,
+            -0.013541662134230137,
+            0.01257830485701561,
+            -0.0016303628217428923,
+            0.007604374550282955,
+            0.0015031108632683754,
+            -0.013015573844313622,
+            -0.009005000814795494,
+            -0.006261823233217001,
+            0.013193213380873203,
+            -0.0048680296167731285,
+            0.018010001629590988,
+            0.00786400306969881,
+            0.008888850919902325,
+            -0.007727356161922216,
+            0.022259704768657684,
+            -0.020510630682110786,
+            -0.007399404887109995,
+            0.0033375893253833055,
+            0.021070880815386772,
+            -0.014019925147294998,
+            0.013008740730583668,
+            -0.0021880511194467545,
+            -0.010631092824041843,
+            0.0033973720856010914,
+            0.0004987595020793378,
+            0.0009172392310574651,
+            -0.0007302043959498405,
+            0.012762777507305145,
+            -0.0029310660902410746,
+            0.002444263081997633,
+            -0.018802549690008163,
+            0.025921829044818878,
+            -0.008724874816834927,
+            0.013384519144892693,
+            -0.007891331799328327,
+            0.007003129925578833,
+            0.02363983355462551,
+            0.03304110839962959,
+            0.0029242339078336954,
+            0.008444749750196934,
+            -0.0029037368949502707,
+            0.01602862775325775,
+            0.004024237859994173,
+            0.00609784759581089,
+            0.007454063277691603,
+            0.014675827696919441,
+            -0.017203787341713905,
+            -0.020660942420363426,
+            0.013316195458173752,
+            -0.015605023130774498,
+            -0.012188862077891827,
+            -0.016165273264050484,
+            -0.00873854011297226,
+            0.018802549690008163,
+            0.0215354785323143,
+            -0.0015210456913337111,
+            0.006142257712781429,
+            0.016465896740555763,
+            -0.007536051329225302,
+            -0.00108206900767982,
+            -0.0013177840737625957,
+            0.006869899574667215,
+            -0.00034887544461525977,
+            -0.016438566148281097,
+            0.01694415882229805,
+            -0.02273796685039997,
+            0.00948326289653778,
+            -0.007146609015762806,
+            0.0061935000121593475,
+            0.014224894344806671,
+            0.01295408234000206,
+            0.015523036010563374,
+            0.015181419439613819,
+            0.0055615101009607315,
+            0.006753750145435333,
+            -0.005653746426105499,
+            0.012981412000954151,
+            0.009162143804132938,
+            0.00820561870932579,
+            0.014129241928458214,
+            -0.0027380529791116714,
+            -0.01934913545846939,
+            -0.005749398842453957,
+            0.01882988028228283,
+            -0.007966487668454647,
+            -0.009244131855666637,
+            0.0139379370957613,
+            0.023680828511714935,
+            -0.021658461540937424,
+            0.02649574540555477,
+            -0.0031428681686520576,
+            -0.014484522864222527,
+            0.035063475370407104,
+            -0.00564691424369812,
+            0.01949944719672203,
+            -0.021439826115965843,
+            -0.037003856152296066,
+            -0.00477920938283205,
+            0.0028337056282907724,
+            0.016752853989601135,
+            0.005113992840051651,
+            0.0012195694725960493,
+            -0.018037330359220505,
+            -0.006849403027445078,
+            0.01672552339732647,
+            -0.01587831601500511,
+            0.01434787642210722,
+            -0.0039729950949549675,
+            0.0207019355148077,
+            0.0014740735059604049,
+            0.015331730246543884,
+            -0.012188862077891827,
+            -0.007570212706923485,
+            0.030253522098064423,
+            0.005178899969905615,
+            -0.010788236744701862,
+            0.017367763444781303,
+            -0.0043795183300971985,
+            -0.002268330892547965,
+            -0.013275201432406902,
+            0.006572693586349487,
+            -0.0007169668097048998,
+            0.0020992308855056763,
+            0.004652811214327812,
+            -0.003959330730140209,
+            0.02230069972574711,
+            -0.005998778622597456,
+            0.010002519004046917,
+            0.005100328475236893,
+            -0.006104679778218269,
+            0.0055717583745718,
+            0.01589198037981987,
+            -0.02115286886692047,
+            0.010528608225286007,
+            -0.0064087179489433765,
+            0.006289152428507805,
+            -0.01175159402191639,
+            0.00022632066975347698,
+            -0.0070304591208696365,
+            0.004755296278744936,
+            0.01178575586527586,
+            -0.013275201432406902,
+            -0.0032795146107673645,
+            0.007522386498749256,
+            -0.004611817188560963,
+            0.025197602808475494,
+            0.021658461540937424,
+            -0.013124890625476837,
+            -0.013664644211530685,
+            0.01382178720086813,
+            -0.003284638747572899,
+            0.006371140480041504,
+            0.005684491712599993,
+            0.004382934421300888,
+            -0.034899499267339706,
+            -0.012018054723739624,
+            0.004099393263459206,
+            -0.016957823187112808,
+            0.001854975474998355,
+            -0.009223634377121925,
+            -0.024282071739435196,
+            0.022396352142095566,
+            -0.004266784992069006,
+            0.021125540137290955,
+            0.0074472310952842236,
+            -0.011840414255857468,
+            -0.016780182719230652,
+            0.014880796894431114,
+            -0.01649322547018528,
+            -0.0020701936446130276,
+            -0.001913050189614296,
+            -0.020223673433065414,
+            0.015345395542681217,
+            -0.004150635562837124,
+            0.010863391682505608,
+            0.001276790164411068,
+            -0.01269445475190878,
+            -0.0036177143920212984,
+            -0.0077341883443295956,
+            0.008301271125674248,
+            -0.018283294513821602,
+            0.0071397763676941395,
+            0.011526127345860004,
+            -0.002435722853988409,
+            -0.019267147406935692,
+            -0.00782984122633934,
+            -2.091064197884407e-05,
+            0.03670323267579079,
+            -0.0261268001049757,
+            -0.0024015612434595823,
+            -7.245997949212324e-06,
+            -0.012373334728181362,
+            0.0035015649627894163,
+            -0.009271460585296154,
+            -0.0037987709511071444,
+            -0.01964975893497467,
+            -0.006617103703320026,
+            0.007501889485865831,
+            0.008000649511814117,
+            -0.013288866728544235,
+            -0.015482041984796524,
+            -0.010453452356159687,
+            -0.025033628568053246,
+            -0.008718042634427547,
+            -0.02967960759997368,
+            -0.017518073320388794,
+            0.0018652238650247455,
+            -0.002572369296103716,
+            0.02235535718500614,
+            -0.019485782831907272,
+            0.010098172351717949,
+            -0.00495001720264554,
+            0.018939197063446045,
+            -0.008595061488449574,
+            -0.004031070042401552,
+            0.006863067392259836,
+            0.0048680296167731285,
+            0.0009257796336896718,
+            -0.008232948370277882,
+            -0.003805603366345167,
+            -0.00786400306969881,
+            -0.00960624497383833,
+            0.013985763303935528,
+            -0.006319897715002298,
+            0.008977671153843403,
+            -0.017190122976899147,
+            -0.000306386937154457,
+            -0.012585137039422989,
+            -0.008048475719988346,
+            -0.009442268870770931,
+            0.003347837831825018,
+            0.014279552735388279,
+            -0.009537921287119389,
+            -0.03774174675345421,
+            -0.0019010936375707388,
+            -0.0051549868658185005,
+            0.0034298256505280733,
+            -0.01874789223074913,
+            0.015782663598656654,
+            -0.011423642747104168,
+            0.011170846410095692,
+            0.011614947579801083,
+            0.005086663644760847,
+            0.008116798475384712,
+            0.006422382779419422,
+            0.015386389568448067,
+            -0.0011025660205632448,
+            -0.004611817188560963,
+            -0.02318890020251274,
+            0.018460934981703758,
+            -0.01249631680548191,
+            -0.008014313876628876,
+            -0.008403755724430084,
+            0.041841138154268265,
+            0.02266964502632618,
+            0.01212053932249546,
+            0.002522834809496999,
+            0.02891438640654087,
+            0.0031240792013704777,
+            0.018132982775568962,
+            0.011813084594905376,
+            -0.004973930306732655,
+            -0.006418966688215733,
+            -0.01295408234000206,
+            -0.029460972175002098,
+            0.016957823187112808,
+            0.008232948370277882,
+            0.029925569891929626,
+            -0.028504448011517525,
+            0.015031108632683754,
+            -0.005462441593408585,
+            -0.01092488318681717,
+            -0.010023016482591629,
+            0.008519905619323254,
+            -0.0024596359580755234,
+            0.031592655926942825,
+            0.00911431759595871,
+            -0.0029515631031244993,
+            0.008191954344511032,
+            0.013145387172698975,
+            -0.012885759584605694,
+            0.01077457144856453,
+            0.01574167050421238,
+            0.03960013762116432,
+            0.009886370040476322,
+            2.0523655621218495e-05,
+            0.035118136554956436,
+            0.016014963388442993,
+            -0.0036792054306715727,
+            -0.0074745602905750275,
+            -0.016206268221139908,
+            -0.012325508520007133,
+            -0.006439463701099157,
+            0.01988205686211586,
+            -0.005602504126727581,
+            0.009162143804132938,
+            0.007563380524516106,
+            0.00783667340874672,
+            -0.02899637445807457,
+            0.013903775252401829,
+            0.003149700351059437,
+            0.013808122836053371,
+            0.014730486087501049,
+            -0.023475859314203262,
+            0.011977060697972775,
+            -0.02176777832210064,
+            -0.0035903851967304945,
+            -0.001383545226417482,
+            -0.0008151814108714461,
+            0.01792801357805729,
+            -0.023079583421349525,
+            -0.013773960992693901,
+            -0.004673308227211237,
+            0.014525516889989376,
+            0.008007481694221497,
+            0.0103919617831707,
+            0.012188862077891827,
+            0.002838829765096307,
+            -0.005377037450671196,
+            -0.0062549905851483345,
+            0.0018788885790854692,
+            -0.003733864054083824,
+            -0.002818332752212882,
+            -0.006497538182884455,
+            0.009018665179610252,
+            0.006091014947742224,
+            -0.004977346397936344,
+            0.009503760375082493,
+            -0.01627459190785885,
+            -0.013596320524811745,
+            0.02582617662847042,
+            0.002526251133531332,
+            -0.0013135139597579837,
+            0.005295049399137497,
+            0.010952211916446686,
+            0.01475781574845314,
+            0.009770220145583153,
+            0.004013989120721817,
+            0.013241039589047432,
+            -0.005595671944320202,
+            -0.007009962573647499,
+            0.019690752029418945,
+            0.0031052904669195414,
+            -0.018406275659799576,
+            0.0026031145825982094,
+            -0.010870223864912987,
+            -0.0012272557942196727,
+            0.010139165446162224,
+            -0.010405626147985458,
+            -0.015796327963471413,
+            0.0011102523421868682,
+            -0.02275163307785988,
+            -0.004645979031920433,
+            0.01709447056055069,
+            0.0073310816660523415,
+            0.024268407374620438,
+            0.002584325848147273,
+            -0.005985114257782698,
+            -0.0006883564637973905,
+            -0.024159090593457222,
+            0.005284801125526428,
+            -0.003614298300817609,
+            -0.008027978241443634,
+            0.014402534812688828,
+            -0.001230672001838684,
+            0.00241010170429945,
+            -0.011123020201921463,
+            -0.00862239021807909,
+            0.006412134040147066,
+            0.012284515425562859,
+            0.03172930330038071,
+            -0.004266784992069006,
+            -0.008718042634427547,
+            -0.029570288956165314,
+            -0.003781690262258053,
+            0.024364059790968895,
+            -0.003819267963990569,
+            -0.020756594836711884,
+            0.01951311156153679,
+            0.02537524327635765,
+            -0.0008514781366102397,
+            -0.0011316033778712153,
+            -0.007153441198170185,
+            -0.004891942720860243,
+            0.007918661460280418,
+            0.019403794780373573,
+            -0.018337951973080635,
+            0.010439787991344929,
+            0.014976450242102146,
+            0.03817901387810707,
+            0.0025006297510117292,
+            -0.004543493967503309,
+            -0.015523036010563374,
+            -0.0014843220124021173,
+            -0.018556587398052216,
+            0.02861376479268074,
+            -0.000738317787181586,
+            0.03304110839962959,
+            0.020811252295970917,
+            0.008492575958371162,
+            0.0028132086154073477,
+            -0.010808733291924,
+            0.059468530118465424,
+            -0.019089506939053535,
+            -0.007782014552503824,
+            -0.011983892880380154,
+            -0.023257223889231682,
+            -0.010419291444122791,
+            -0.0028217490762472153,
+            -0.012878927402198315,
+            0.015318065881729126,
+            -0.01649322547018528,
+            0.00585529999807477,
+            0.005981697700917721,
+            0.0011879700468853116,
+            0.008451581932604313,
+            -0.022232376039028168,
+            0.021180197596549988,
+            -0.02275163307785988,
+            -0.004400015342980623,
+            0.002869575284421444,
+            -0.0003832505608443171,
+            -0.009312454611063004,
+            0.0031104146037250757,
+            0.004693805240094662,
+            0.005455608945339918,
+            0.001044491189531982,
+            -0.01299507636576891,
+            0.024077102541923523,
+            -0.010576434433460236,
+            -0.017203787341713905,
+            0.013247872702777386,
+            0.037304479628801346,
+            0.00696896854788065,
+            0.008355929516255856,
+            0.022710638120770454,
+            0.003508397378027439,
+            -0.03910820931196213,
+            0.004618649836629629,
+            0.005144738592207432,
+            -0.012762777507305145,
+            0.023079583421349525,
+            0.003781690262258053,
+            -0.00325560150668025,
+            -0.01434787642210722,
+            -0.00412330636754632,
+            -0.0012554391287267208,
+            -0.02312057837843895,
+            0.014525516889989376,
+            0.0016021794872358441,
+            -0.018406275659799576,
+            -0.0081782890483737,
+            0.01175159402191639,
+            0.015468376688659191,
+            0.002534791361540556,
+            -0.004943185020238161,
+            0.003178737824782729,
+            0.004232623614370823,
+            -0.013951601460576057,
+            -0.00436927005648613,
+            -0.00046630596625618637,
+            -0.018488263711333275,
+            0.01306340005248785,
+            0.004837283864617348,
+            -0.01559135876595974,
+            -0.007009962573647499,
+            0.020128021016716957,
+            -0.001632070867344737,
+            0.01611061580479145,
+            -0.004058399237692356,
+            0.002451095497235656,
+            0.01777770183980465,
+            0.026318104937672615,
+            -0.005062750540673733,
+            -0.009093821048736572,
+            -0.016014963388442993,
+            -0.013726134784519672,
+            -0.006825489457696676,
+            0.02432306669652462,
+            -0.0011948023457080126,
+            -0.01129382848739624,
+            0.021877095103263855,
+            -0.0011034200433641672,
+            -0.011416809633374214,
+            -0.004659643396735191,
+            -0.0020326157100498676,
+            -0.021262185648083687,
+            0.008157792501151562,
+            -0.002565536880865693,
+            0.009893202222883701,
+            0.020087026059627533,
+            0.031292036175727844,
+            -0.004512748681008816,
+            -0.003405912546440959,
+            -0.006104679778218269,
+            0.005838219076395035,
+            0.00873854011297226,
+            -0.03033551014959812,
+            -0.012407496571540833,
+            -0.014949120581150055,
+            0.006053437478840351,
+            0.018419940024614334,
+            0.007392572239041328,
+            0.015687011182308197,
+            -0.011000038124620914,
+            -0.014921790920197964,
+            -0.02131684496998787,
+            -0.022765297442674637,
+            0.004406847525388002,
+            0.010596930980682373,
+            0.03500881791114807,
+            0.0009266336564905941,
+            -0.010521776042878628,
+            -0.009701897390186787,
+            -0.010419291444122791,
+            0.0015483750030398369,
+            0.006087598856538534,
+            -0.02605847641825676,
+            -0.012721783481538296,
+            -0.016506889835000038,
+            0.004396599251776934,
+            -0.012892591767013073,
+            -0.013357189483940601,
+            0.04971197620034218,
+            -0.006873316131532192,
+            0.0223690215498209,
+            -0.0340522937476635,
+            -0.00457423971965909,
+            -0.0007259342237375677,
+            -0.02250566892325878,
+            -0.001954043982550502,
+            -0.016397573053836823,
+            0.00914847943931818,
+            -0.022123059257864952,
+            -0.0049295201897621155,
+            -0.006466792896389961,
+            -0.005305298138409853,
+            -0.019253483042120934,
+            0.0038807590026408434,
+            -0.020141685381531715,
+            0.01641123741865158,
+            0.0034981488715857267,
+            0.004526413511484861,
+            0.00953108910471201,
+            -0.030390167608857155,
+            0.011929234489798546,
+            -0.006716172676533461,
+            0.007823009043931961,
+            0.003932001534849405,
+            -0.01249631680548191,
+            -0.012216191738843918,
+            -0.004806538578122854,
+            -0.009278293699026108,
+            0.003733864054083824,
+            -0.0032197318505495787,
+            0.018050994724035263,
+            -0.018255963921546936,
+            -0.008895683102309704,
+            -0.004762128461152315,
+            0.019185159355401993,
+            0.01671185903251171,
+            -0.010043513029813766,
+            -0.005127657670527697,
+            -0.02063361182808876,
+            0.025101950392127037,
+            0.005277968943119049,
+            -0.007570212706923485,
+            0.02567586675286293,
+            0.007980152033269405,
+            0.0001476422039559111,
+            0.014142906293272972,
+            0.01739509217441082,
+            -0.017299439758062363,
+            -0.0012588553363457322,
+            0.00824661273509264,
+            0.00758387753739953,
+            -0.00033094058744609356,
+            0.0018088571960106492,
+            0.010412458330392838,
+            0.008232948370277882,
+            -0.004078896250575781,
+            -0.007249093614518642,
+            -0.029324326664209366,
+            -0.008191954344511032,
+            -0.0005337751354090869,
+            0.00011561570136109367,
+            0.016055956482887268,
+            -0.0020906904246658087,
+            0.0054077827371656895,
+            -0.024432383477687836,
+            -0.015236077830195427,
+            0.013732966966927052,
+            -0.022874614223837852,
+            -0.018816214054822922,
+            0.005950952414423227,
+            0.003788522444665432,
+            -0.0107267452403903,
+            0.0036074661184102297,
+            -0.02297026664018631,
+            -0.02109820954501629,
+            -0.00565033033490181,
+            0.007071453146636486,
+            -0.019677087664604187,
+            0.013186381198465824,
+            -0.019294477999210358,
+            -0.030089545994997025,
+            -0.017449749633669853,
+            0.005482938606292009,
+            0.00630623335018754,
+            0.002254666294902563,
+            -0.0006567569798789918,
+            -0.009326119907200336,
+            0.002830289304256439,
+            -0.01656154915690422,
+            0.01209320966154337,
+            0.009271460585296154,
+            0.004806538578122854,
+            -0.01001618430018425,
+            0.002121435943990946,
+            -0.007126112002879381,
+            0.0031326196622103453,
+            0.010979541577398777,
+            0.026195121929049492,
+            0.009920531883835793,
+            -0.006186667364090681,
+            -0.020988892763853073,
+            -0.002758549991995096,
+            -0.013036070391535759,
+            -0.0031582408118993044,
+            0.009783885441720486,
+            0.004208710510283709,
+            -0.019540440291166306,
+            -0.016438566148281097,
+            -0.026031147688627243,
+            -0.011792588047683239,
+            -0.02327088825404644,
+            -0.004283865913748741,
+            0.025730524212121964,
+            -0.010125501081347466,
+            0.0013254705118015409,
+            0.00015564884233754128,
+            0.01626092568039894,
+            -0.0021436410024762154,
+            -0.026987671852111816,
+            0.015044772997498512,
+            -0.008519905619323254,
+            0.0028729913756251335,
+            0.006073934491723776,
+            -0.00412330636754632,
+            0.0008907640003599226,
+            -0.004242871887981892,
+            0.004140387289226055,
+            -0.014320546761155128,
+            0.005506851710379124,
+            -0.007925493642687798,
+            0.019021185114979744,
+            -0.014033589512109756,
+            -0.000921509403269738,
+            0.009059659205377102,
+            -0.0047860415652394295,
+            -0.014402534812688828,
+            -0.017941677942872047,
+            0.024118097499012947,
+            -0.023981450125575066,
+            0.049520671367645264,
+            -0.004136971198022366,
+            0.006138841155916452,
+            -0.02183610200881958,
+            0.003580136690288782,
+            -0.015714339911937714,
+            -0.013992595486342907,
+            -0.01647956110537052,
+            -0.002818332752212882,
+            0.035118136554956436,
+            0.005759647581726313,
+            0.0014843220124021173,
+            0.00045563047751784325,
+            0.004960265941917896,
+            0.03110072948038578,
+            -0.0013271785574033856,
+            0.01956777088344097,
+            -0.0056708273477852345,
+            -0.007686362136155367,
+            -0.007802511565387249,
+            0.010023016482591629,
+            0.016096951439976692,
+            -0.016069620847702026,
+            0.0019677088130265474,
+            -0.0009052826790139079,
+            0.009196305647492409,
+            0.009421772323548794,
+            0.005277968943119049,
+            -0.0048202034085989,
+            -0.009168975986540318,
+            0.00828077457845211,
+            -0.009995686821639538,
+            -0.00832176860421896,
+            -0.017723042517900467,
+            0.005124241579324007,
+            0.01027581188827753,
+            -0.0025484561920166016,
+            0.014525516889989376,
+            -0.005021756514906883,
+            0.004208710510283709,
+            0.005940704140812159,
+            -0.013985763303935528,
+            0.005318962503224611,
+            0.006019275635480881,
+            -0.030554143711924553,
+            0.02727462910115719,
+            -0.01076773926615715,
+            -0.017299439758062363,
+            -0.003011345863342285,
+            0.003069420577958226,
+            -0.004355605226010084,
+            -0.005421447567641735,
+            -0.008984503336250782,
+            -0.018884537741541862,
+            0.006429214961826801,
+            -0.007467728108167648,
+            -0.01844726875424385,
+            -0.001386107294820249,
+            -0.0024579279124736786,
+            0.035883355885744095,
+            -0.003424701513722539,
+            0.010904385708272457,
+            0.016288256272673607
+          ],
+          "title": "The Structure Of The Mixture-Of-Experts Layer",
+          "keyphrases": [
+            "Mixture-of-Experts layer",
+            "gating network",
+            "neural networks",
+            "sparsity",
+            "hierarchical MoE"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "44767af8-7eba-44fe-a7ef-fd48aa105f77",
+        "properties": {
+          "page_content": "3 Addressing Performance Challenges 3.1 The Shrinking Batch Problem\n\nOn modern CPUs and GPUs, large batch sizes are necessary for computational efficiency, so as to amortize the overhead of parameter loads and updates. If the gating network chooses k out of n experts for each example, then for a batch of b examples, each expert receives a much smaller batch of approximately kb n b examples. This causes a naive MoE implementation to become very inefficient as the number of experts increases. The solution to this shrinking batch problem is to make the original batch size as large as possible. However, batch size tends to be limited by the memory necessary to store activations between the forwards and backwards passes. We propose the following techniques for increasing the batch size: Mixing Data Parallelism and Model Parallelism: In a conventional distributed training setting, multiple copies of the model on different devices asynchronously process distinct batches of data, and parameters are synchronized through a set of parameter servers. In our technique, these different batches run synchronously so that they can be combined for the MoE layer. We distribute the standard layers of the model and the gating network according to conventional data-parallel schemes, but keep only one shared copy of each expert. Each expert in the MoE layer receives a combined batch consisting of the relevant examples from all of the data-parallel input batches. The same set of devices function as data-parallel replicas (for the standard layers and the gating networks) and as model-parallel shards (each hosting a subset of the experts). If the model is distributed over d devices, and each device processes a batch of size b, each expert receives a batch of approximately kbd nexamples. Thus, we achieve a factor of d improvement in expert batch size.\n\nIn the case of a hierarchical MoE (Section B), the primary gating network employs data parallelism, and the secondary MoEs employ model parallelism. Each secondary MoE resides on one device.\n\nThis technique allows us to increase the number of experts (and hence the number of parameters) by proportionally increasing the number of devices in the training cluster. The total batch size increases, keeping the batch size per expert constant. The memory and bandwidth requirements per device also remain constant, as do the step times, as does the amount of time necessary to process a number of training examples equal to the number of parameters in the model. It is our goal to train a trillionparameter model on a trillion-word corpus. We have not scaled our systems this far as of the writing of this paper, but it should be possible by adding more hardware.\n\nTaking Advantage of Convolutionality: In our language models, we apply the same MoE to each time step of the previous layer. If we wait for the previous layer to finish, we can apply the MoE to all the time steps together as one big batch. Doing so increases the size of the input batch to the MoE layer by a factor of the number of unrolled time steps.\n\nIncreasing Batch Size for a Recurrent MoE: We suspect that even more powerful models may involve applying a MoE recurrently. For example, the weight matrices of a LSTM or other RNN could be replaced by a MoE. Sadly, such models break the convolutional trick from the last paragraph, since the input to the MoE at one timestep depends on the output of the MoE at the previous timestep. Gruslys et al. (2016) describe a technique for drastically reducing the number of stored activations in an unrolled RNN, at the cost of recomputing forward activations. This would allow for a large increase in batch size.\n\n3.2 Network Bandwidth\n\nAnother major performance concern in distributed computing is network bandwidth. Since the experts are stationary (see above) and the number of gating parameters is small, most of the communication involves sending the inputs and outputs of the experts across the network. To maintain computational efficiency, the ratio of an expert's computation to the size of its input and output must exceed the ratio of computational to network capacity of the computing device. For GPUs, this may be thousands to one. In our experiments, we use experts with one hidden layer containing thousands of RELU-activated units. Since the weight matrices in the expert have sizes input_size×hidden_size and hidden_size × output_size, the ratio of computation to input and output is equal to the size of the hidden layer. Conveniently, we can increase computational efficiency simply by using a larger hidden layer, or more hidden layers.\n\n",
+          "embedding": [
+            0.0014724871143698692,
+            -0.0002654724521562457,
+            -0.015004266053438187,
+            -0.010133732110261917,
+            0.02161158062517643,
+            0.014150978997349739,
+            -0.020856458693742752,
+            0.05349281430244446,
+            -0.016355933621525764,
+            0.02585536427795887,
+            0.004545832052826881,
+            -0.0036283594090491533,
+            0.015313866548240185,
+            0.006528026424348354,
+            -0.01309380866587162,
+            0.013184423558413982,
+            0.010216795839369297,
+            0.007860816083848476,
+            0.012504814192652702,
+            0.009748619981110096,
+            0.023076515644788742,
+            -0.03379924222826958,
+            -0.03869242966175079,
+            -0.013516676612198353,
+            -0.0013828164665028453,
+            0.015766939148306847,
+            0.0215813759714365,
+            0.00212189182639122,
+            -0.04113902524113655,
+            0.013191974721848965,
+            0.018107816576957703,
+            -0.0014149091439321637,
+            -0.03023506887257099,
+            -0.02502473071217537,
+            -0.05300953611731529,
+            -0.029328923672437668,
+            0.05705698952078819,
+            0.01131927315145731,
+            -0.023907151073217392,
+            0.005855968222022057,
+            0.03201715648174286,
+            0.013720559887588024,
+            -0.042709678411483765,
+            -0.008555527776479721,
+            0.0013847042573615909,
+            0.009076561778783798,
+            -0.02502473071217537,
+            0.017443308606743813,
+            0.015933066606521606,
+            -0.04283049702644348,
+            0.0038530079182237387,
+            -0.06312816590070724,
+            0.014974061399698257,
+            0.007317128591239452,
+            -0.04415951296687126,
+            0.0005852192407473922,
+            0.025674134492874146,
+            0.01730738766491413,
+            0.05485203489661217,
+            0.0018349455203860998,
+            0.036547884345054626,
+            0.0009023703169077635,
+            -0.05071396753191948,
+            0.006059851031750441,
+            0.00730957742780447,
+            -0.010163936764001846,
+            -0.05470100790262222,
+            0.03787690028548241,
+            -0.01833435334265232,
+            -0.007256718818098307,
+            -0.04116922989487648,
+            0.0024220526684075594,
+            0.0015055236872285604,
+            0.008336542174220085,
+            0.018711913377046585,
+            -0.010873750783503056,
+            0.04974741116166115,
+            0.04283049702644348,
+            0.01686941646039486,
+            0.0027278768830001354,
+            -0.016099192202091217,
+            0.009688210673630238,
+            -0.012217868119478226,
+            0.01082089263945818,
+            0.038269564509391785,
+            -0.012059291824698448,
+            -0.02571944147348404,
+            -0.0495963878929615,
+            -0.01418873481452465,
+            -0.005553919356316328,
+            -0.021234020590782166,
+            0.014196285977959633,
+            0.013894237577915192,
+            -0.005689841229468584,
+            -0.003618920221924782,
+            0.036245837807655334,
+            -0.012089497409760952,
+            -0.01176479458808899,
+            -0.0005918265669606626,
+            -0.008502669632434845,
+            0.00020046118879690766,
+            0.011228658258914948,
+            0.019451932981610298,
+            -0.000972219102550298,
+            0.02058461494743824,
+            0.00044788149534724653,
+            -0.009514532051980495,
+            0.00590127520263195,
+            0.02843788079917431,
+            0.10390473157167435,
+            0.015555505640804768,
+            -0.015072227455675602,
+            -0.012814413756132126,
+            0.004606241825968027,
+            -0.0477236844599247,
+            -0.010496190749108791,
+            -0.010481088422238827,
+            -0.038209155201911926,
+            -0.004406134597957134,
+            -0.001600857824087143,
+            0.019874799996614456,
+            0.020040927454829216,
+            -0.014906100928783417,
+            -0.0016169041628018022,
+            -0.07424356043338776,
+            0.03325555473566055,
+            -0.01044333167374134,
+            -0.01090395636856556,
+            -0.01703554391860962,
+            -0.03902468457818031,
+            0.005587900057435036,
+            0.017971893772482872,
+            -0.015223251655697823,
+            -0.05627166107296944,
+            0.007860816083848476,
+            0.004719510208815336,
+            0.055607154965400696,
+            0.05231482535600662,
+            0.014755076728761196,
+            0.0007310521323233843,
+            0.017911484465003014,
+            -0.040021445602178574,
+            -0.01766984537243843,
+            -0.005716270767152309,
+            -0.013614842668175697,
+            0.01789638213813305,
+            -0.002208730671554804,
+            0.04739143326878548,
+            -0.03530948609113693,
+            0.027259889990091324,
+            -0.02774316817522049,
+            0.017745357006788254,
+            -8.854980842443183e-05,
+            0.05470100790262222,
+            -0.012142355553805828,
+            -0.016386138275265694,
+            -0.02513044700026512,
+            0.005916377995163202,
+            -0.037937309592962265,
+            -0.011938472278416157,
+            -0.013388305902481079,
+            -0.02419409714639187,
+            -0.014332207851111889,
+            0.023121824488043785,
+            -0.001210082438774407,
+            0.038239359855651855,
+            0.04859962686896324,
+            -0.019980518147349358,
+            -0.08771492540836334,
+            0.021203814074397087,
+            0.005720046348869801,
+            0.007721118628978729,
+            0.01756412908434868,
+            0.026761509478092194,
+            0.017382899299263954,
+            0.02369571663439274,
+            0.016914723441004753,
+            0.01736779697239399,
+            -0.029344025999307632,
+            0.0517711378633976,
+            -0.023182233795523643,
+            -0.016189808025956154,
+            0.021128302440047264,
+            0.012663389556109905,
+            0.033134736120700836,
+            0.0027448670007288456,
+            0.006973547860980034,
+            0.028332162648439407,
+            0.011832755990326405,
+            0.011696834117174149,
+            0.03301391750574112,
+            0.030839167535305023,
+            0.018802527338266373,
+            -0.01302584819495678,
+            0.024405529722571373,
+            -0.019874799996614456,
+            -0.012504814192652702,
+            0.0034074862487614155,
+            0.016612675040960312,
+            -0.00446654437109828,
+            -0.014022608287632465,
+            -0.02960076741874218,
+            -0.021883424371480942,
+            -0.02241200953722,
+            0.02618761733174324,
+            0.033195145428180695,
+            0.019149884581565857,
+            0.026942739263176918,
+            0.03990062698721886,
+            -0.031111011281609535,
+            0.03476579859852791,
+            -0.009250239469110966,
+            -0.046122826635837555,
+            -0.04047451913356781,
+            -0.032168179750442505,
+            0.003192276693880558,
+            -0.011794999241828918,
+            0.039598576724529266,
+            -0.012376443482935429,
+            -0.03467518463730812,
+            -0.016809007152915,
+            0.017488617449998856,
+            0.009854337200522423,
+            -0.016068987548351288,
+            -0.005350036546587944,
+            -0.02041848748922348,
+            -0.00730957742780447,
+            -0.017715152353048325,
+            0.021928731352090836,
+            -0.020735640078783035,
+            -0.013373203575611115,
+            0.035853173583745956,
+            -0.0014498335076496005,
+            0.020327873528003693,
+            0.015238353982567787,
+            -0.016416342929005623,
+            0.021989140659570694,
+            0.015736734494566917,
+            -0.007807957474142313,
+            -0.010586804710328579,
+            -0.024284711107611656,
+            0.006648845970630646,
+            -0.05276789888739586,
+            -0.03703116253018379,
+            0.0047950223088264465,
+            0.012988091446459293,
+            -0.01902906410396099,
+            0.04277008771896362,
+            -0.003701983718201518,
+            0.0010835995199158788,
+            -0.0025844036135822535,
+            -0.05162011459469795,
+            0.005735148675739765,
+            0.0007291642832569778,
+            -0.003358403453603387,
+            0.03237961605191231,
+            0.012134804390370846,
+            -0.02399776503443718,
+            -0.004719510208815336,
+            -0.01756412908434868,
+            0.030023634433746338,
+            -0.004172047134488821,
+            -0.029253412038087845,
+            0.02963097207248211,
+            -0.023574896156787872,
+            -0.00843470823019743,
+            0.03497723489999771,
+            0.04434074088931084,
+            0.016718393191695213,
+            0.006101382430642843,
+            -0.005893724039196968,
+            -0.0357927642762661,
+            0.0015734846238046885,
+            -0.031171420589089394,
+            0.03947775810956955,
+            -0.02907218225300312,
+            0.0018104041228070855,
+            -0.008215723559260368,
+            0.02121891640126705,
+            0.035218872129917145,
+            0.015442237257957458,
+            0.009605146944522858,
+            -0.00013780969311483204,
+            0.01214990671724081,
+            -0.004715734627097845,
+            -0.018847836181521416,
+            -0.02094707265496254,
+            0.00012164537474745885,
+            -0.03198695182800293,
+            -0.015117534436285496,
+            -0.005882397294044495,
+            -0.017020441591739655,
+            -0.01464935950934887,
+            0.005670963320881128,
+            0.04319295659661293,
+            0.005935255903750658,
+            -0.016522061079740524,
+            0.012255623936653137,
+            0.030552219599485397,
+            -0.015125085599720478,
+            -0.044401150196790695,
+            -0.003303657053038478,
+            -0.0013507237890735269,
+            -0.021671989932656288,
+            -0.03639686107635498,
+            -0.015283661894500256,
+            -0.017549026757478714,
+            0.011606219224631786,
+            0.008646142669022083,
+            -0.007611625827848911,
+            0.0310203954577446,
+            0.012044189497828484,
+            0.014807934872806072,
+            -0.03974959999322891,
+            -0.03370862826704979,
+            0.011606219224631786,
+            0.025447597727179527,
+            0.01756412908434868,
+            -0.04557913914322853,
+            -0.04491463303565979,
+            0.03869242966175079,
+            0.017609436064958572,
+            -0.03325555473566055,
+            0.013146666809916496,
+            -0.027561938390135765,
+            -0.00012601091293618083,
+            0.020403385162353516,
+            0.06324898451566696,
+            -0.040927592664957047,
+            -0.008827371522784233,
+            -0.01830414868891239,
+            -0.021762605756521225,
+            -0.034161701798439026,
+            -0.0007310521323233843,
+            0.005889948457479477,
+            -0.04657590016722679,
+            -0.026399051770567894,
+            -0.009363507851958275,
+            -0.004451442044228315,
+            0.013131564483046532,
+            -0.01047353632748127,
+            -0.009627800434827805,
+            -0.012519916519522667,
+            0.03905488923192024,
+            0.0010958702769130468,
+            0.010564151220023632,
+            -0.05920153483748436,
+            0.015721632167696953,
+            0.014989163726568222,
+            -0.006520475260913372,
+            0.0011581677244976163,
+            0.01686941646039486,
+            0.023257745429873466,
+            0.03678952530026436,
+            0.006999977398663759,
+            -0.03023506887257099,
+            -0.015797143802046776,
+            0.025764750316739082,
+            -0.02585536427795887,
+            -0.004817675799131393,
+            0.009129419922828674,
+            -0.013879135251045227,
+            -0.026142310351133347,
+            0.02574964612722397,
+            -0.046787336468696594,
+            -0.0031431936658918858,
+            -0.02078094705939293,
+            -0.008714103139936924,
+            0.012572774663567543,
+            -0.00676966505125165,
+            -0.01786617748439312,
+            0.017684947699308395,
+            -0.006697928532958031,
+            0.00021202399511821568,
+            0.017881279811263084,
+            0.014928754419088364,
+            0.03259104862809181,
+            -0.0055388170294463634,
+            -0.04838819429278374,
+            0.033829446882009506,
+            -0.022396907210350037,
+            0.019074371084570885,
+            0.02105279080569744,
+            -0.015797143802046776,
+            0.013569535687565804,
+            0.035218872129917145,
+            -0.0023125598672777414,
+            0.008268581703305244,
+            -0.02061481960117817,
+            0.04684774577617645,
+            -0.022094858810305595,
+            -0.03301391750574112,
+            -0.010488639585673809,
+            0.004224905278533697,
+            -0.015857553109526634,
+            0.01720166951417923,
+            0.0022748038172721863,
+            -0.04603221267461777,
+            -0.006040973123162985,
+            0.006441187579184771,
+            -0.024813296273350716,
+            0.006543128751218319,
+            0.007698464673012495,
+            0.005440651439130306,
+            -0.010617010295391083,
+            -0.008744308724999428,
+            0.0021577600855380297,
+            -0.007981635630130768,
+            0.04956618323922157,
+            -0.03639686107635498,
+            -0.0022011795081198215,
+            -0.04603221267461777,
+            -0.01892334781587124,
+            -0.008427157066762447,
+            0.03811853751540184,
+            0.016129396855831146,
+            0.0040323492139577866,
+            0.018047407269477844,
+            -0.018349455669522285,
+            -0.006403431296348572,
+            -0.02990281581878662,
+            0.03376903757452965,
+            0.025961080566048622,
+            0.010043117217719555,
+            -0.01055660005658865,
+            0.0031262035481631756,
+            0.008049596101045609,
+            -0.006932016462087631,
+            0.0002463584241922945,
+            -0.021626682952046394,
+            0.020765844732522964,
+            -0.0029147693421691656,
+            0.019119679927825928,
+            -0.027984807267785072,
+            -0.012127253226935863,
+            -0.008464912883937359,
+            0.004938495345413685,
+            0.041984762996435165,
+            -0.03234941139817238,
+            -0.012157457880675793,
+            -0.00476481718942523,
+            0.002593842800706625,
+            -0.0003669419093057513,
+            0.0014762626960873604,
+            -0.03920591250061989,
+            -0.03754464536905289,
+            -0.0429513156414032,
+            -0.015268558636307716,
+            -0.01199888251721859,
+            -0.05337199568748474,
+            -0.07581420987844467,
+            0.00016659870743751526,
+            -0.0720083937048912,
+            -0.009174727834761143,
+            0.0407765656709671,
+            -0.015087329782545567,
+            -0.018319251015782356,
+            -0.03292330354452133,
+            -0.0076833623461425304,
+            -0.01736779697239399,
+            -0.01206684298813343,
+            0.004776144400238991,
+            0.016914723441004753,
+            0.006233528722077608,
+            -0.024571657180786133,
+            0.018636401742696762,
+            -0.027003148570656776,
+            -0.028422776609659195,
+            0.007158552762120962,
+            0.0037246374413371086,
+            -0.038964275270700455,
+            -0.029872611165046692,
+            0.013629944995045662,
+            0.012036638334393501,
+            0.046153031289577484,
+            0.025598622858524323,
+            -0.0014101896667852998,
+            -0.05908071622252464,
+            0.01919519156217575,
+            -0.001417740830220282,
+            0.02940443530678749,
+            0.015631018206477165,
+            -0.005384017247706652,
+            0.01677880249917507,
+            -0.016023680567741394,
+            -0.010511293075978756,
+            0.012542570009827614,
+            -0.006025870330631733,
+            -0.0011996994726359844,
+            -0.02926851436495781,
+            0.017186567187309265,
+            0.03049181029200554,
+            0.005806885194033384,
+            0.025568418204784393,
+            0.004036124795675278,
+            -0.012542570009827614,
+            -0.021128302440047264,
+            0.013645047321915627,
+            0.008540425449609756,
+            0.03159428760409355,
+            -0.03908509388566017,
+            -0.023756125941872597,
+            0.012361341156065464,
+            0.010481088422238827,
+            -0.016295524314045906,
+            -0.0004523650277405977,
+            0.01578204147517681,
+            0.021490762010216713,
+            -0.034222111105918884,
+            -0.03292330354452133,
+            0.008208171464502811,
+            0.028317060321569443,
+            0.019602956250309944,
+            -0.001834001624956727,
+            0.011674179695546627,
+            0.026731304824352264,
+            -0.0360041968524456,
+            0.0064713922329247,
+            0.012731350027024746,
+            0.034161701798439026,
+            1.8229107809020206e-05,
+            0.02393735572695732,
+            -0.02910238690674305,
+            0.03700095787644386,
+            -0.020660126581788063,
+            0.03201715648174286,
+            1.8627315512276255e-05,
+            0.011613770388066769,
+            0.03370862826704979,
+            -0.009733517654240131,
+            -0.026066798716783524,
+            0.004496749024838209,
+            0.005078192800283432,
+            -0.0035887155681848526,
+            -0.04228680953383446,
+            -0.014166081324219704,
+            0.01886293850839138,
+            0.02217037044465542,
+            -0.038632020354270935,
+            -0.011077634058892727,
+            0.006516699679195881,
+            -0.0002836425555869937,
+            0.0028675743378698826,
+            -0.003426364390179515,
+            -0.009831683710217476,
+            0.0036170324310660362,
+            0.034886617213487625,
+            -0.034252315759658813,
+            0.0032753399573266506,
+            0.018047407269477844,
+            -0.013199525885283947,
+            -0.037605054676532745,
+            -0.01055660005658865,
+            -0.0042098029516637325,
+            -0.012429301626980305,
+            -0.016189808025956154,
+            0.019497239962220192,
+            -0.013977301307022572,
+            0.011991331353783607,
+            -0.03361801430583,
+            -0.013305243104696274,
+            0.017820870503783226,
+            -0.03669891133904457,
+            0.009801478125154972,
+            -0.017292285338044167,
+            -0.007494581863284111,
+            0.009990259073674679,
+            0.03872263804078102,
+            -0.0028184913098812103,
+            0.015261007472872734,
+            0.000569172902032733,
+            -0.007853264920413494,
+            0.01453609112650156,
+            -0.013365652412176132,
+            -0.0004693552909884602,
+            0.005002680700272322,
+            0.031171420589089394,
+            -0.01244440395385027,
+            -0.0050215586088597775,
+            0.018153123557567596,
+            0.05237523466348648,
+            -0.031050600111484528,
+            0.007120796944946051,
+            0.014974061399698257,
+            0.028483187779784203,
+            -0.007471928372979164,
+            -0.003031813306733966,
+            -0.01955764926970005,
+            0.013267486356198788,
+            -0.01936131902039051,
+            0.0006673387251794338,
+            0.028226446360349655,
+            -0.02449614554643631,
+            -0.01902906410396099,
+            -0.0015527188079431653,
+            -0.02579495497047901,
+            0.0029411986470222473,
+            0.005127275828272104,
+            0.014468129724264145,
+            -0.030522014945745468,
+            -0.0007551216403953731,
+            0.044370945543050766,
+            0.014702217653393745,
+            -0.01147029735147953,
+            -0.008110006339848042,
+            0.0065997629426419735,
+            0.024239404127001762,
+            0.008125108666718006,
+            -0.0030771205201745033,
+            -0.010941712185740471,
+            0.0032621254213154316,
+            -0.011666628532111645,
+            0.020478898659348488,
+            0.004470319952815771,
+            0.030476707965135574,
+            0.008827371522784233,
+            -0.008351645432412624,
+            -0.004213578533381224,
+            -0.021762605756521225,
+            0.011485399678349495,
+            0.05926194414496422,
+            0.006750787142664194,
+            0.016084089875221252,
+            -0.016763700172305107,
+            0.009076561778783798,
+            0.0341314971446991,
+            -0.0019293357618153095,
+            -0.034886617213487625,
+            0.006263733841478825,
+            -0.008276132866740227,
+            -0.01889314316213131,
+            -0.012686043046414852,
+            -0.004190925043076277,
+            -0.025674134492874146,
+            -0.01329014077782631,
+            -0.007513460237532854,
+            -0.011862960644066334,
+            0.006833850406110287,
+            -0.002359755104407668,
+            -0.005278300028294325,
+            -0.029026875272393227,
+            0.024209199473261833,
+            -0.0003412205842323601,
+            0.05303974077105522,
+            0.00045637661241926253,
+            0.012210316024720669,
+            -0.012293379753828049,
+            0.0024352672044187784,
+            0.0018037967383861542,
+            0.0040323492139577866,
+            0.025734543800354004,
+            0.019134782254695892,
+            0.011794999241828918,
+            0.03684993460774422,
+            -0.012044189497828484,
+            -0.03189633786678314,
+            -0.01123620942234993,
+            -0.0020482672844082117,
+            0.03307432681322098,
+            0.021037688478827477,
+            -0.003971939906477928,
+            0.015600812621414661,
+            -0.011107838712632656,
+            -0.008812269195914268,
+            0.041018206626176834,
+            -0.008804718032479286,
+            -0.025779852643609047,
+            0.043374184519052505,
+            -0.012867271900177002,
+            0.04657590016722679,
+            0.00700375298038125,
+            -0.00858573243021965,
+            0.004292866215109825,
+            0.010133732110261917,
+            -0.003817139659076929,
+            -0.018289046362042427,
+            0.011394784785807133,
+            0.04479381442070007,
+            0.012399096973240376,
+            0.0061240363866090775,
+            0.02041848748922348,
+            0.02224588207900524,
+            -0.027622347697615623,
+            -0.023589998483657837,
+            -0.0034622326493263245,
+            -0.0025466475635766983,
+            -0.015933066606521606,
+            0.016824109479784966,
+            -0.0400516502559185,
+            -0.009393712505698204,
+            0.026942739263176918,
+            -0.0011732701677829027,
+            0.017292285338044167,
+            0.003854895941913128,
+            0.014543642289936543,
+            0.008245928213000298,
+            -0.01803230494260788,
+            0.026006387546658516,
+            0.030204864218831062,
+            0.013365652412176132,
+            -0.02519085630774498,
+            0.031080804765224457,
+            -0.0065997629426419735,
+            0.019089475274086,
+            0.01677880249917507,
+            0.005014007445424795,
+            -0.006010768003761768,
+            -0.00208035996183753,
+            -0.026127208024263382,
+            -0.006814972497522831,
+            -0.007396416272968054,
+            0.027758270502090454,
+            -0.00439103227108717,
+            0.01564612053334713,
+            0.0159783735871315,
+            -0.012708696536719799,
+            -0.0006475168047472835,
+            -0.0010156385833397508,
+            -0.031775519251823425,
+            0.025613725185394287,
+            0.007891020737588406,
+            0.006999977398663759,
+            0.03802792355418205,
+            -0.042045172303915024,
+            -0.01842496730387211,
+            -0.003711422672495246,
+            0.029147693887352943,
+            0.05134826898574829,
+            0.0031431936658918858,
+            -0.0003223425301257521,
+            0.031775519251823425,
+            -0.03479600325226784,
+            -0.032530639320611954,
+            0.011349477805197239,
+            0.03292330354452133,
+            -0.0005526546156033874,
+            -0.003741627559065819,
+            0.04379705339670181,
+            -0.010254551656544209,
+            -0.021762605756521225,
+            0.021762605756521225,
+            0.01839476265013218,
+            -0.002372969640418887,
+            -0.005648309830576181,
+            0.0003556150768417865,
+            0.014166081324219704,
+            -0.0045005246065557,
+            0.04310234263539314,
+            -0.02419409714639187,
+            0.01495140790939331,
+            -0.01603878289461136,
+            0.013501574285328388,
+            0.015049573965370655,
+            0.00010677891259547323,
+            -0.04283049702644348,
+            0.013040950521826744,
+            0.022351600229740143,
+            0.005474631674587727,
+            0.00958249345421791,
+            0.017322489991784096,
+            0.021838117390871048,
+            0.03832997381687164,
+            0.02960076741874218,
+            0.005923929158598185,
+            0.040655747056007385,
+            0.01453609112650156,
+            -0.0005847472930327058,
+            0.02784888446331024,
+            -0.0019302796572446823,
+            -0.006105158012360334,
+            -0.0011307946406304836,
+            -0.0041418420150876045,
+            0.02343897521495819,
+            -0.021883424371480942,
+            -0.015223251655697823,
+            0.02124912291765213,
+            0.0028468084055930376,
+            -0.00019196607172489166,
+            -0.03343678638339043,
+            0.006290162913501263,
+            -0.02790929563343525,
+            0.0220495518296957,
+            -0.0024560331366956234,
+            0.03364821895956993,
+            -0.023227540776133537,
+            -0.0056747389025986195,
+            0.022260984405875206,
+            -0.019482137635350227,
+            0.0033055448438972235,
+            0.01392444223165512,
+            0.008631040342152119,
+            -0.005965460557490587,
+            -0.011492950841784477,
+            -0.013569535687565804,
+            0.047089383006095886,
+            -0.005350036546587944,
+            0.018183328211307526,
+            0.018138021230697632,
+            -0.007709791883826256,
+            -0.019300907850265503,
+            0.028392571955919266,
+            -0.040716156363487244,
+            0.010662317276000977,
+            0.03201715648174286,
+            -0.06475923210382462,
+            -0.034161701798439026,
+            -0.006006992422044277,
+            -0.028996670618653297,
+            0.005489734001457691,
+            0.00033579315640963614,
+            0.008064698427915573,
+            0.010095976293087006,
+            -0.018802527338266373,
+            0.018455171957612038,
+            0.0006678106728941202,
+            0.02973668836057186,
+            -0.0031356425024569035,
+            0.0069131385535001755,
+            -0.021883424371480942,
+            -0.025930875912308693,
+            -0.01952744461596012,
+            -0.0008792447042651474,
+            -0.0036566765047609806,
+            0.00772489421069622,
+            -0.02741091512143612,
+            -0.015721632167696953,
+            0.01936131902039051,
+            -0.009861888363957405,
+            0.04026308283209801,
+            0.0405651330947876,
+            0.022381804883480072,
+            0.005980563350021839,
+            -0.025009628385305405,
+            0.03923611715435982,
+            -0.042014967650175095,
+            0.022019345313310623,
+            -0.0015904748579487205,
+            -0.017262080684304237,
+            -0.01252746768295765,
+            0.031111011281609535,
+            -0.007845713756978512,
+            0.005504836328327656,
+            0.0011883726110681891,
+            0.021234020590782166,
+            -0.02294059470295906,
+            -0.0030922230798751116,
+            -0.00885002501308918,
+            -0.011900716461241245,
+            0.015359173528850079,
+            0.0003539632598403841,
+            0.01958785392343998,
+            0.02549290470778942,
+            0.006343021523207426,
+            0.0027448670007288456,
+            -0.032107770442962646,
+            -0.031201625242829323,
+            0.026353744789958,
+            0.012308482080698013,
+            -0.028452983126044273,
+            0.013146666809916496,
+            -0.045428115874528885,
+            -0.013652598485350609,
+            -0.011213555932044983,
+            0.0075059086084365845,
+            0.013199525885283947,
+            -0.002378633012995124,
+            -0.0008320496417582035,
+            -0.02777337282896042,
+            -5.3418953029904515e-05,
+            -0.05001925677061081,
+            -0.017594333738088608,
+            0.015276109799742699,
+            -0.01102477591484785,
+            0.01866660639643669,
+            0.005478407256305218,
+            0.0107378289103508,
+            -0.01919519156217575,
+            -0.012187662534415722,
+            0.004666651599109173,
+            -0.030869372189044952,
+            0.01581224612891674,
+            -0.018440069630742073,
+            0.013433613814413548,
+            -0.013033399358391762,
+            0.03981001302599907,
+            -0.009174727834761143,
+            0.016748597845435143,
+            -0.025613725185394287,
+            -0.029887713491916656,
+            -0.02565903216600418,
+            0.02294059470295906,
+            -0.004485422279685736,
+            -0.018787425011396408,
+            0.012029087170958519,
+            -0.007875918410718441,
+            0.011719487607479095,
+            -0.02214016579091549,
+            -0.00916717667132616,
+            0.023287950083613396,
+            -0.026263128966093063,
+            0.0023842963855713606,
+            -0.04902249574661255,
+            -0.03709157183766365,
+            -0.00450807623565197,
+            0.02114340476691723,
+            -0.019300907850265503,
+            0.030688142403960228,
+            -0.0005021558608859777,
+            -0.004379705525934696,
+            -0.02399776503443718,
+            -0.028694620355963707,
+            0.01112294103950262,
+            -0.0002989809727296233,
+            -0.02880033850669861,
+            0.05781210958957672,
+            0.025009628385305405,
+            0.04008185490965843,
+            -0.04482401907444,
+            0.026867227628827095,
+            0.0066261920146644115,
+            -0.029283616691827774,
+            0.01291257981210947,
+            0.01079068798571825,
+            0.016401240602135658,
+            -0.009484327398240566,
+            -0.007468152791261673,
+            0.0034414667170494795,
+            -0.009371059015393257,
+            0.007687137927860022,
+            0.005297177936881781,
+            -0.03222858905792236,
+            0.005784231703728437,
+            0.025704339146614075,
+            -0.001376209082081914,
+            0.04014226421713829,
+            0.007196309044957161,
+            0.011568463407456875,
+            -0.004674202762544155,
+            -0.04011205956339836,
+            0.01730738766491413,
+            -0.027622347697615623,
+            -0.03277227655053139,
+            0.03887366130948067,
+            0.020992381498217583,
+            0.011711936444044113,
+            -0.02416389249265194,
+            0.00416071992367506,
+            0.014641808345913887,
+            -0.016431447118520737,
+            0.0081704156473279,
+            0.006524250842630863,
+            0.027863986790180206,
+            0.014709768816828728,
+            -0.007860816083848476,
+            0.00044740954763256013,
+            0.02303120866417885,
+            -0.01564612053334713,
+            0.0028373694512993097,
+            0.0020407161209732294,
+            0.02813583053648472,
+            0.010330063290894032,
+            -0.024586759507656097,
+            -0.017986996099352837,
+            -0.017549026757478714,
+            -0.020040927454829216,
+            0.0009608922409825027,
+            -0.013713008724153042,
+            -0.02177770808339119,
+            -0.04358562082052231,
+            -0.004798797890543938,
+            -0.019497239962220192,
+            -0.03138285502791405,
+            -0.015026920475065708,
+            -0.008125108666718006,
+            -0.02821134403347969,
+            0.011198453605175018,
+            0.001511187176220119,
+            -0.02038828283548355,
+            0.002797725610435009,
+            -0.004813900217413902,
+            0.050140075385570526,
+            -0.019542546942830086,
+            0.013259935192763805,
+            -0.0038643348962068558,
+            -0.006052299868315458,
+            -0.00875185988843441,
+            0.03168490156531334,
+            -0.026610486209392548,
+            -0.04977761581540108,
+            0.026625588536262512,
+            -0.022683853283524513,
+            0.000796181324403733,
+            0.04497504234313965,
+            -0.024254506453871727,
+            -0.01753392443060875,
+            0.02005602978169918,
+            -0.020297668874263763,
+            -0.032440025359392166,
+            0.003915305715054274,
+            -0.010647214949131012,
+            -0.016068987548351288,
+            -0.02704845555126667,
+            -0.010118629783391953,
+            0.052163802087306976,
+            0.011092736385762691,
+            0.027335401624441147,
+            0.03364821895956993,
+            0.00480634905397892,
+            -0.016552265733480453,
+            0.010881302878260612,
+            -0.015540402382612228,
+            -0.026202719658613205,
+            0.010095976293087006,
+            0.02562882751226425,
+            0.01739800162613392,
+            0.007317128591239452,
+            -0.030310582369565964,
+            -0.005980563350021839,
+            -0.03204736113548279,
+            -0.0051763588562607765,
+            0.019044166430830956,
+            -0.003532081376761198,
+            -0.031956747174263,
+            -0.011115389876067638,
+            0.025840261951088905,
+            0.021928731352090836,
+            -0.018606197088956833,
+            -0.016748597845435143,
+            -0.0028807888738811016,
+            0.0007291642832569778,
+            0.016914723441004753,
+            -0.01327503751963377,
+            -0.00476481718942523,
+            0.004330622497946024,
+            0.019814390689134598,
+            -0.013916891068220139,
+            0.0013129677390679717,
+            -0.05950358137488365,
+            -0.01620491035282612,
+            0.0211132001131773,
+            0.002933647483587265,
+            -0.0017735918518155813,
+            -0.010088425129652023,
+            0.007668260019272566,
+            -0.0031054376158863306,
+            -0.030008532106876373,
+            0.010201692581176758,
+            0.0023238868452608585,
+            0.02214016579091549,
+            -0.02602148987352848,
+            0.008842473849654198,
+            0.01686941646039486,
+            0.0015206261305138469,
+            -0.013546881265938282,
+            -0.0036113690584897995,
+            0.0343429297208786,
+            0.010465985164046288,
+            0.003277227748185396,
+            0.004440115299075842,
+            0.014543642289936543,
+            0.004224905278533697,
+            -0.0034320277627557516,
+            0.002644813386723399,
+            -0.0012959775049239397,
+            -0.006856504362076521,
+            0.004021022468805313,
+            0.006724358070641756,
+            0.02363530732691288,
+            0.024058174341917038,
+            0.005059314891695976,
+            0.0008070362382568419,
+            -0.00976372230798006,
+            -0.0057426998391747475,
+            0.01700533926486969,
+            -0.012557672336697578,
+            0.015827348455786705,
+            0.011168248951435089,
+            -0.02072053775191307,
+            -0.023892046883702278,
+            0.0018859162228181958,
+            0.001278043375350535,
+            0.01830414868891239,
+            0.02217037044465542,
+            -0.00047100710798986256,
+            0.0005309449043124914,
+            -0.017443308606743813,
+            0.00964290276169777,
+            0.007664484437555075,
+            0.03597399219870567,
+            -0.0365176796913147,
+            -0.01480038370937109,
+            0.021823015064001083,
+            -0.009673107415437698,
+            0.020342975854873657,
+            0.0003593906876631081,
+            0.006977323908358812,
+            0.0004969644360244274,
+            0.06935036927461624,
+            -0.017986996099352837,
+            -0.00620709965005517,
+            -0.029872611165046692,
+            0.009605146944522858,
+            -0.0056332070380449295,
+            -0.02052420563995838,
+            0.011198453605175018,
+            0.0328930988907814,
+            -0.015283661894500256,
+            0.042347218841314316,
+            -0.0018575991271063685,
+            0.009303098544478416,
+            -0.005417997483164072,
+            -0.019738879054784775,
+            0.013305243104696274,
+            -0.0014536090893670917,
+            0.004942270927131176,
+            0.0032847789116203785,
+            0.012436852790415287,
+            0.02069033309817314,
+            0.01020924374461174,
+            0.014498335309326649,
+            0.004172047134488821,
+            -0.02008623443543911,
+            0.01978418603539467,
+            -0.0019897455349564552,
+            0.000477614434203133,
+            -0.015993475914001465,
+            -0.013335447758436203,
+            0.0067092557437717915,
+            0.01155336108058691,
+            -0.0033546278718858957,
+            -0.0017207333585247397,
+            0.003413149621337652,
+            0.025070037692785263,
+            -0.016416342929005623,
+            0.014694666489958763,
+            -0.012346237897872925,
+            -0.012308482080698013,
+            -0.012640736065804958,
+            0.005251870956271887,
+            0.005417997483164072,
+            -0.0055388170294463634,
+            0.004927168600261211,
+            -0.006184446159750223,
+            0.011953575536608696,
+            -0.03739362210035324,
+            0.014241593889892101,
+            -0.006633743178099394,
+            -0.003290442517027259,
+            0.010201692581176758,
+            -0.012867271900177002,
+            0.007517235819250345,
+            -0.007403967436403036,
+            0.01653716340661049,
+            0.006977323908358812,
+            -0.007528562564402819,
+            0.0017282846383750439,
+            -0.014853241853415966,
+            0.004519402980804443,
+            0.014792832545936108,
+            -0.0018085162155330181,
+            0.01495140790939331,
+            -0.01972377672791481,
+            -0.01474752463400364,
+            0.0006059851148165762,
+            -0.01853068359196186,
+            0.002325774636119604,
+            0.00454205647110939,
+            -0.009710864163935184,
+            0.023952458053827286,
+            -0.0034565692767500877,
+            -0.01026210281997919,
+            -0.021928731352090836,
+            -0.015495095402002335,
+            -0.015661222860217094,
+            -0.020629921928048134,
+            0.006093831267207861,
+            -0.03307432681322098,
+            0.022623443976044655,
+            0.016235115006566048,
+            -0.01733759231865406,
+            -0.01972377672791481,
+            -0.032440025359392166,
+            0.0030676815658807755,
+            -0.006093831267207861,
+            -0.005455753766000271,
+            0.012097048573195934,
+            -0.004153168760240078,
+            0.011893165297806263,
+            0.0031299791298806667,
+            -0.010956814512610435,
+            0.011485399678349495,
+            -0.012285828590393066,
+            -0.006494045723229647,
+            -0.0018009650520980358,
+            -0.003647237317636609,
+            -0.015042022801935673,
+            -0.023741023615002632,
+            -0.03769567236304283,
+            0.020071132108569145,
+            0.013426062650978565,
+            -0.014422822743654251,
+            -0.01529876422137022,
+            -0.002776959678158164,
+            0.01223297044634819,
+            -0.013108910992741585,
+            -0.01264828722923994,
+            0.008940639905631542,
+            -0.008464912883937359,
+            0.01461915485560894,
+            -0.0014413383323699236,
+            -0.0034018228761851788,
+            0.023710818961262703,
+            0.00937861017882824,
+            0.005440651439130306,
+            -0.02479819394648075,
+            -0.012240521609783173,
+            0.01370545756071806,
+            -0.013931993395090103,
+            -0.0017226211493834853,
+            0.0219438336789608,
+            -0.00154139194637537,
+            0.000530472956597805,
+            0.011145595461130142,
+            0.0019265039591118693,
+            -0.02058461494743824,
+            0.0054028951562941074,
+            0.008283684030175209,
+            -0.009854337200522423,
+            0.046092621982097626,
+            0.014098120853304863,
+            0.01603878289461136,
+            0.0025655257049947977,
+            -0.006184446159750223,
+            0.013342998921871185,
+            0.010669868439435959,
+            0.02787908911705017,
+            -0.015094880945980549,
+            -0.007366211153566837,
+            0.013388305902481079,
+            0.04080677032470703,
+            0.00013851761468686163,
+            0.011409887112677097,
+            -0.027592143043875694,
+            -0.02164178527891636,
+            0.028815440833568573,
+            0.004787471145391464,
+            0.01302584819495678,
+            -0.011138043366372585,
+            -0.024012867361307144,
+            0.015079778619110584,
+            -0.018545785918831825,
+            -0.015011817216873169,
+            -0.028558699414134026,
+            0.009627800434827805,
+            -0.027471324428915977,
+            0.02840767428278923,
+            0.0069810994900763035,
+            -0.0006479887524619699,
+            0.02519085630774498,
+            0.000645628955680877,
+            -0.025311676785349846,
+            0.014860793016850948,
+            -0.013607291504740715,
+            0.006562006659805775,
+            0.016688188537955284,
+            -0.03092978149652481,
+            0.0014130213530734181,
+            -0.006569557823240757,
+            -0.004107861779630184,
+            -0.0016848650993779302,
+            0.02436022274196148,
+            -0.016612675040960312,
+            -0.02599128521978855,
+            -0.005682290066033602,
+            -0.006331694778054953,
+            0.002212506253272295,
+            -0.007747547701001167,
+            0.022215677425265312,
+            0.005950358230620623,
+            0.003154520643875003,
+            -0.01789638213813305,
+            -0.009287996217608452,
+            0.020297668874263763,
+            -0.020750742405653,
+            0.0014715432189404964,
+            -0.021958936005830765,
+            0.010133732110261917,
+            0.006629967596381903,
+            0.010186590254306793,
+            0.014762627892196178,
+            6.300545646809042e-05,
+            0.003545295912772417,
+            0.0007820228347554803,
+            0.0026183840818703175,
+            0.013063604012131691,
+            0.01524590514600277,
+            -0.0446125864982605,
+            -0.02257813699543476,
+            0.028679518029093742,
+            -0.016461651772260666,
+            0.00308278389275074,
+            0.008563078939914703,
+            -0.03343678638339043,
+            -0.011779896914958954,
+            -0.005614329129457474,
+            0.035520922392606735,
+            -0.01614449918270111,
+            0.0015149627579376101,
+            -0.001464935950934887,
+            -0.024224301800131798,
+            0.0026882330421358347,
+            -0.020131541416049004,
+            0.008842473849654198,
+            -0.01830414868891239,
+            -0.043434593826532364,
+            -0.0008244984201155603,
+            0.02535698376595974,
+            -0.007449274882674217,
+            0.003824691055342555,
+            0.021354839205741882,
+            -0.018138021230697632,
+            0.003739739768207073,
+            -0.019089475274086,
+            0.013780969195067883,
+            0.003877549432218075,
+            0.028196241706609726,
+            0.008736756630241871,
+            0.004685529507696629,
+            0.006380777806043625,
+            -0.00368688115850091,
+            0.030476707965135574,
+            -0.009227585978806019,
+            -0.012036638334393501,
+            0.021022586151957512,
+            0.021520966663956642,
+            0.02127932757139206,
+            0.002335213590413332,
+            0.010835994966328144,
+            0.035520922392606735,
+            -0.005210339091718197,
+            0.019874799996614456,
+            0.007815509103238583,
+            -6.365439185174182e-05,
+            0.008940639905631542,
+            -0.007698464673012495,
+            -0.03443354368209839,
+            -0.0022653648629784584,
+            -0.022865083068609238,
+            -0.01697513461112976,
+            0.011772345751523972,
+            -0.023454077541828156,
+            -0.02217037044465542,
+            -0.028679518029093742,
+            0.013660149648785591,
+            0.004783695563673973,
+            0.004749714862555265,
+            -0.021234020590782166,
+            0.02777337282896042,
+            0.024647168815135956,
+            0.015585710294544697,
+            0.017715152353048325,
+            0.012754004448652267,
+            0.030386094003915787,
+            0.008336542174220085,
+            0.02641415409743786,
+            0.0065997629426419735,
+            0.012406648136675358,
+            -0.007747547701001167,
+            0.031956747174263,
+            0.017594333738088608,
+            0.006267509423196316,
+            0.007509684655815363,
+            -0.01733759231865406,
+            -0.0032338083256036043,
+            0.0007013191934674978,
+            -0.020478898659348488,
+            -0.011039878241717815,
+            -0.006788543425500393,
+            0.032802481204271317,
+            0.02200424298644066,
+            -0.016325728967785835,
+            -0.014211388304829597,
+            -0.005191461183130741,
+            0.0060560754500329494,
+            -0.023046310991048813,
+            -0.007543664891272783,
+            -0.012285828590393066,
+            -0.0023238868452608585,
+            -0.009340854361653328,
+            0.02343897521495819,
+            0.021626682952046394,
+            0.010760483331978321,
+            0.01833435334265232,
+            0.001749050454236567,
+            0.003915305715054274,
+            -0.010760483331978321,
+            -0.007317128591239452,
+            0.011379682458937168,
+            -0.012421750463545322,
+            -0.025341881439089775,
+            0.008087352849543095,
+            -0.006256182678043842,
+            -0.020267464220523834,
+            0.009567391127347946,
+            0.018379660323262215,
+            0.001985969953238964,
+            0.01252746768295765,
+            0.017216771841049194,
+            0.0014998603146523237,
+            0.0065997629426419735,
+            0.02214016579091549,
+            -0.0028298182878643274,
+            -0.006543128751218319,
+            0.004236232023686171,
+            -0.007981635630130768,
+            0.02230629324913025,
+            0.016461651772260666,
+            -0.03207756578922272,
+            -0.01529876422137022,
+            -0.010798239149153233,
+            0.02960076741874218,
+            -0.0070754894986748695,
+            0.008653693832457066,
+            -0.03265145793557167,
+            -0.006267509423196316,
+            -0.026308437809348106,
+            0.02002582512795925,
+            -0.003362179035320878,
+            -0.006331694778054953,
+            0.004734612535685301,
+            -0.01578204147517681,
+            0.017413103953003883,
+            -0.003756730118766427,
+            -0.009287996217608452,
+            0.012429301626980305,
+            -0.004991353955119848,
+            0.01617470569908619,
+            -0.007441723719239235,
+            0.004323071334511042,
+            0.015525300055742264,
+            0.00722651369869709,
+            0.004723285790532827,
+            -0.03201715648174286,
+            -0.0019878577440977097,
+            -0.014528539963066578,
+            0.0062863873317837715,
+            -0.006701704114675522,
+            0.0009490934899076819,
+            0.024178994819521904,
+            0.027728065848350525,
+            -0.010133732110261917,
+            -0.008011840283870697,
+            0.029419537633657455,
+            0.004070105496793985,
+            0.012029087170958519,
+            -0.009333303198218346,
+            -0.013584638014435768,
+            -0.0012251847656443715,
+            -0.012504814192652702,
+            -0.028890952467918396,
+            -0.014437925070524216,
+            -0.027108866721391678,
+            0.016718393191695213,
+            -0.009310649707913399,
+            -0.02019195258617401,
+            0.0328930988907814,
+            -0.041712917387485504,
+            -0.008412054739892483,
+            -0.00329421809874475,
+            0.0015347846783697605,
+            0.012587876990437508,
+            0.003345188684761524,
+            0.0028260427061468363,
+            -0.028951361775398254,
+            -0.013984852470457554,
+            -0.007026406470686197,
+            0.004081432241946459,
+            -0.000620143604464829,
+            -0.02496432140469551,
+            0.007256718818098307,
+            -0.0038058129139244556,
+            -0.004273988306522369,
+            0.010314960964024067,
+            -0.00484788091853261,
+            -0.024481043219566345,
+            0.02602148987352848,
+            -0.007120796944946051,
+            0.02052420563995838,
+            0.010171487927436829,
+            0.016914723441004753,
+            0.010730277746915817,
+            0.03023506887257099,
+            -0.0007919337949715555,
+            -0.002229496603831649,
+            -0.004357051569968462,
+            -0.009703313000500202,
+            0.01165152620524168,
+            0.014279349707067013,
+            -0.01128906849771738,
+            -0.03706136718392372,
+            0.006018319167196751,
+            -0.02343897521495819,
+            0.019738879054784775,
+            0.0010118630016222596,
+            -0.02999342978000641,
+            -0.007037733215838671,
+            -0.0422264002263546,
+            0.02386184222996235,
+            -0.005300953518599272,
+            -0.016386138275265694,
+            0.017050646245479584,
+            -0.006577109452337027,
+            -0.005557694938033819,
+            0.027199480682611465,
+            0.005131051409989595,
+            -0.030084045603871346,
+            0.013743213377892971,
+            0.010156385600566864,
+            -0.012104599736630917,
+            0.0018141797045245767,
+            -0.0006838570116087794,
+            -0.034252315759658813,
+            0.0071170213632285595,
+            0.0008518715621903539,
+            -0.0034660082310438156,
+            -0.02058461494743824,
+            -0.019633162766695023,
+            -0.00881982035934925,
+            0.018379660323262215,
+            0.018349455669522285,
+            0.01836455799639225,
+            -0.007762650027871132,
+            0.00036127850762568414,
+            0.006799870170652866,
+            0.005331158638000488,
+            0.010911507532000542,
+            0.005501060746610165,
+            -0.00045118515845388174,
+            0.007928776554763317,
+            -0.014150978997349739,
+            0.017413103953003883,
+            -0.020901765674352646,
+            -0.003990817815065384,
+            -0.00019208405865356326,
+            -0.008555527776479721,
+            0.00019845539645757526,
+            -0.016809007152915,
+            0.011681731790304184,
+            0.016461651772260666,
+            -0.006822523660957813,
+            -0.009491878561675549,
+            -0.020705435425043106,
+            0.009997810237109661,
+            0.029525255784392357,
+            0.01020924374461174,
+            -0.041320253163576126,
+            0.021898526698350906,
+            -0.01739800162613392,
+            0.0018160674953833222,
+            -0.017941689118742943,
+            0.025961080566048622,
+            0.014332207851111889,
+            0.0035736130084842443,
+            0.018983757123351097,
+            -0.0049989051185548306,
+            0.0003516034921631217,
+            -0.00704528484493494,
+            0.001961428439244628,
+            0.00045614063856191933,
+            -0.009522083215415478,
+            -0.0040738810785114765,
+            0.02704845555126667,
+            -0.02416389249265194,
+            -0.004334398079663515,
+            0.001532896887511015,
+            0.023121824488043785,
+            -0.008261030539870262,
+            0.012535018846392632,
+            0.017911484465003014,
+            0.018017200753092766,
+            -0.002399398945271969,
+            0.011447643861174583,
+            0.07291454076766968,
+            -0.019089475274086,
+            0.017624538391828537,
+            0.019331112504005432,
+            -0.008245928213000298,
+            -0.01637103594839573,
+            -0.010851097293198109,
+            -0.01988990418612957,
+            0.02144545316696167,
+            -0.0025711890775710344,
+            0.01445302739739418,
+            -0.011296619661152363,
+            -0.009257790632545948,
+            0.007539889309555292,
+            0.0037869347725063562,
+            0.0015168505487963557,
+            0.005285851191729307,
+            0.011742141097784042,
+            0.023876944556832314,
+            0.010118629783391953,
+            -0.005165031645447016,
+            0.0012619970366358757,
+            -0.0011996994726359844,
+            -0.0023805208038538694,
+            -0.01184785831719637,
+            -0.0013554432662203908,
+            0.009386161342263222,
+            0.02340877056121826,
+            0.0019878577440977097,
+            0.035218872129917145,
+            -0.015011817216873169,
+            -0.009121868759393692,
+            0.00916717667132616,
+            0.0006692265160381794,
+            0.004606241825968027,
+            0.014724871143698692,
+            0.012837067246437073,
+            -0.011772345751523972,
+            -0.006176894996315241,
+            0.0076154014095664024,
+            0.0052934023551642895,
+            0.006384553387761116,
+            0.0107378289103508,
+            -0.01112294103950262,
+            -0.016960032284259796,
+            0.03709157183766365,
+            0.009884541854262352,
+            0.0023238868452608585,
+            0.0049309441819787025,
+            0.0009769386379048228,
+            -0.011062531732022762,
+            0.004832778126001358,
+            0.004443890880793333,
+            0.005327383056282997,
+            -0.000515842461027205,
+            -0.0009325752034783363,
+            0.003205491229891777,
+            -0.0056332070380449295,
+            0.0014847578713670373,
+            0.008049596101045609,
+            0.005319831892848015,
+            0.010911507532000542,
+            -0.008396952413022518,
+            -0.019210293889045715,
+            0.0010005361400544643,
+            -0.016657982021570206,
+            0.006679050624370575,
+            0.002169086830690503,
+            -0.0063279191963374615,
+            0.007536113727837801,
+            0.009189830161631107,
+            -0.0073473332449793816,
+            0.014332207851111889,
+            -0.001468711532652378,
+            0.001872701570391655,
+            0.0107378289103508,
+            -0.020327873528003693,
+            -0.007672035600990057,
+            0.0024767988361418247,
+            0.006894260179251432,
+            0.003726525232195854,
+            0.0018321138340979815,
+            -0.008615938015282154,
+            0.014936305582523346,
+            -0.023544691503047943,
+            -0.00908411294221878,
+            -0.0032375839073210955,
+            0.026595383882522583,
+            0.013071155175566673,
+            -0.013003193773329258,
+            0.004406134597957134,
+            0.00012789871834684163,
+            0.0066941529512405396,
+            0.0038058129139244556,
+            -0.004383481107652187,
+            0.02735050395131111,
+            -0.003715198254212737,
+            -0.024420632049441338,
+            -0.008238377049565315,
+            -0.002325774636119604,
+            0.009257790632545948,
+            0.023982662707567215,
+            0.0054028951562941074,
+            0.006497821304947138,
+            0.006988650653511286,
+            0.00502910977229476,
+            0.009892093017697334,
+            0.013629944995045662,
+            -0.003990817815065384,
+            -0.008646142669022083,
+            -0.004002144560217857,
+            -0.011402335949242115,
+            0.015283661894500256,
+            0.016386138275265694,
+            -0.012701145373284817,
+            -0.017881279811263084,
+            -0.0073020257987082005,
+            -0.01456629578024149,
+            -0.0075059086084365845,
+            0.003930408041924238,
+            -0.006048524286597967,
+            0.0011279629543423653,
+            -0.0009250239818356931,
+            -0.009597595781087875,
+            0.01324483286589384,
+            -0.012618082575500011,
+            -0.008087352849543095,
+            -0.008547976613044739,
+            -0.034947026520967484,
+            -0.024375325068831444,
+            0.005727597512304783,
+            -0.014966510236263275,
+            -0.001957652857527137,
+            0.014958959072828293,
+            -0.007358659990131855,
+            0.003715198254212737,
+            0.017624538391828537,
+            0.004987578373402357,
+            0.0060560754500329494,
+            -0.005006456281989813,
+            -0.0038114762865006924,
+            -0.010186590254306793,
+            -0.011938472278416157,
+            -0.01464935950934887,
+            0.0040059201419353485,
+            0.0012421749997884035,
+            0.011228658258914948,
+            -0.01544978842139244,
+            -0.0020444917026907206,
+            -0.003060130402445793,
+            0.019482137635350227,
+            -0.028845645487308502,
+            -0.019497239962220192,
+            0.008706551976501942,
+            -0.0004950765869580209,
+            0.01047353632748127,
+            -0.021369941532611847,
+            -0.0029732913244515657,
+            0.0017046870198100805,
+            -0.008691449649631977,
+            -0.019799288362264633,
+            0.02291039004921913,
+            -0.015125085599720478,
+            0.004534505307674408,
+            -0.008396952413022518,
+            0.027592143043875694,
+            0.012693594209849834,
+            0.013894237577915192,
+            0.0064902701415121555,
+            0.00590127520263195,
+            -0.01647675409913063,
+            -0.0026693549007177353,
+            0.0017905820859596133,
+            -0.006528026424348354,
+            0.0015810359036549926,
+            0.0075059086084365845,
+            -0.002389959990978241,
+            -0.007441723719239235,
+            0.018122918903827667,
+            0.01524590514600277,
+            0.005444427020847797,
+            0.020297668874263763,
+            0.0011619434226304293,
+            0.002886452479287982,
+            -0.00011043653648812324,
+            0.0027656329330056906,
+            0.003494325326755643,
+            0.010919058695435524,
+            0.004489197861403227,
+            -0.019874799996614456,
+            0.0013894237345084548,
+            -0.002399398945271969,
+            -0.007604074664413929,
+            -0.026988046243786812,
+            0.00120158726349473,
+            0.007709791883826256,
+            -0.001681089517660439,
+            -0.01471731998026371,
+            -0.015004266053438187,
+            0.0054972851648926735,
+            -0.001405470073223114,
+            0.013848930597305298,
+            -0.02105279080569744,
+            0.001889691804535687,
+            0.0013582749525085092,
+            -0.01925560086965561,
+            -0.005670963320881128,
+            0.02191362902522087,
+            0.019572751596570015,
+            -0.006550679914653301,
+            -0.03953816741704941,
+            -0.005980563350021839,
+            0.01301074493676424,
+            -0.0058370898477733135,
+            -0.01753392443060875,
+            0.006509148515760899,
+            -0.009053908288478851,
+            -0.006101382430642843,
+            0.018243737518787384,
+            0.0010448995744809508,
+            -0.01544978842139244,
+            0.006188221741467714,
+            0.008283684030175209,
+            0.02399776503443718,
+            0.004685529507696629,
+            -0.009439020417630672,
+            -0.005391568411141634,
+            -0.022200575098395348,
+            0.008427157066762447,
+            0.00046392783406190574,
+            -0.026625588536262512,
+            -0.002437154995277524,
+            -0.004662876017391682,
+            0.03165469691157341,
+            0.01040557585656643,
+            0.021505864337086678,
+            0.011092736385762691,
+            -0.018742118030786514,
+            -0.008593283593654633,
+            -0.008480016142129898,
+            -0.012980540283024311,
+            -0.0034225888084620237,
+            -0.0016697626560926437,
+            0.008615938015282154,
+            0.01581224612891674,
+            0.028030114248394966,
+            0.00020706850045826286,
+            0.011742141097784042,
+            0.01468711532652378,
+            -2.842029971361626e-05,
+            0.011636423878371716,
+            0.001077936147339642,
+            -0.016235115006566048,
+            0.00047147905570454895,
+            0.00476481718942523,
+            -0.008985946886241436,
+            -0.006584660615772009,
+            0.006203324068337679,
+            0.018183328211307526,
+            -0.006116485223174095,
+            -0.005081968382000923,
+            -0.010012912563979626,
+            -0.00026452852762304246,
+            -0.012225419282913208,
+            0.04503545165061951,
+            0.009673107415437698,
+            -0.033497195690870285,
+            -0.001490421243943274,
+            -0.021536068990826607,
+            0.01856089010834694,
+            -0.0037982617504894733,
+            -0.004036124795675278,
+            0.007018855307251215,
+            0.02701825089752674,
+            0.025568418204784393,
+            -0.009046357125043869,
+            -0.011168248951435089,
+            0.0007848545210435987,
+            0.011062531732022762,
+            -0.0004979083314538002,
+            -0.008706551976501942,
+            -0.0012431190116330981,
+            0.0028732377104461193,
+            -0.018711913377046585,
+            0.0030808961018919945,
+            0.008155313320457935,
+            0.00044363393681123853,
+            0.0074379476718604565,
+            -0.014551193453371525,
+            0.026172515004873276,
+            -0.0049309441819787025,
+            -0.01385648176074028,
+            -0.03772587701678276,
+            -0.01995031349360943,
+            0.020237259566783905,
+            0.0068716066889464855,
+            0.002169086830690503,
+            -0.015389378182590008,
+            -0.02011643908917904,
+            0.020433591678738594,
+            -0.011183351278305054,
+            -0.0056520854122936726,
+            -0.010934161022305489,
+            -0.010851097293198109,
+            0.0214756578207016,
+            0.007430396508425474,
+            -0.007875918410718441,
+            -0.012821964919567108,
+            0.012383994646370411,
+            -0.0005762521759606898,
+            0.006826299242675304,
+            0.02369571663439274,
+            0.026761509478092194,
+            0.01819843053817749,
+            -0.009899644181132317,
+            -0.010579253546893597,
+            -0.020509103313088417,
+            -0.003562286263331771,
+            -0.022593239322304726,
+            0.007430396508425474,
+            0.0016357821878045797,
+            0.01427179854363203,
+            0.00722651369869709,
+            -0.0016631553880870342,
+            -0.005603002384305,
+            0.009506980888545513,
+            0.0070490604266524315,
+            -0.012980540283024311,
+            0.018047407269477844,
+            -0.01108518522232771,
+            0.018772322684526443,
+            -0.0030582426115870476,
+            -0.013494023121893406,
+            0.019316010177135468,
+            -0.013750764541327953,
+            0.004409910179674625,
+            -0.004039900843054056,
+            0.008412054739892483,
+            0.00442501250654459,
+            -0.010133732110261917,
+            0.016461651772260666,
+            -0.016612675040960312,
+            0.0002716078015509993,
+            -0.010919058695435524,
+            -0.00476481718942523,
+            -0.014558744616806507,
+            -0.022094858810305595,
+            -0.044672995805740356,
+            -0.005297177936881781,
+            0.019965415820479393,
+            0.0008900996181182563,
+            0.01044333167374134,
+            -0.013826277107000351,
+            -0.0006555399741046131,
+            -0.005950358230620623,
+            -0.008525323122739792,
+            0.012383994646370411,
+            0.01842496730387211,
+            -0.009831683710217476,
+            -0.003926632460206747,
+            0.021264225244522095,
+            0.0021917405538260937,
+            -0.019965415820479393,
+            -0.004651549272239208,
+            -0.006562006659805775,
+            0.0031979400664567947,
+            0.014815486036241055,
+            -0.005376466084271669,
+            -0.023574896156787872,
+            0.0013894237345084548,
+            0.006879157852381468,
+            -0.02724478766322136,
+            -0.0013903676299378276,
+            -0.01720166951417923,
+            -0.01667308434844017,
+            0.006407206878066063,
+            -0.028679518029093742,
+            0.009809029288589954,
+            -0.022955697029829025,
+            -0.010095976293087006,
+            -0.005051763728260994,
+            -0.020796049386262894,
+            0.004610017407685518,
+            -0.008744308724999428,
+            0.031805723905563354,
+            -0.001901962561532855,
+            -0.008691449649631977,
+            0.01158356573432684,
+            0.01753392443060875,
+            0.02200424298644066,
+            0.0051574804820120335,
+            0.025583520531654358,
+            -0.004179598297923803,
+            -0.014974061399698257,
+            -0.0027561939787119627,
+            0.01611429452896118,
+            0.002474911045283079,
+            0.024919012561440468,
+            -0.0158726554363966,
+            0.00037850471562705934,
+            0.00934840552508831,
+            0.01670329086482525,
+            0.0053689144551754,
+            -0.0031733985524624586,
+            0.018409864977002144,
+            0.0007839106256142259,
+            -0.003658564295619726,
+            -0.02771296352148056,
+            -0.0016933601582422853,
+            0.0002349136193515733,
+            0.003620808245614171,
+            0.01032251212745905,
+            -0.0015857553808018565,
+            0.006399655714631081,
+            0.016688188537955284,
+            0.0010722726583480835,
+            -0.007690913509577513,
+            -0.01620491035282612,
+            0.004934719763696194,
+            -0.01955764926970005,
+            0.014702217653393745,
+            -0.004738388117402792,
+            0.015691427513957024,
+            0.002267252653837204,
+            -0.038632020354270935,
+            0.032107770442962646,
+            0.004394807852804661,
+            -0.008842473849654198,
+            0.019814390689134598,
+            -0.02990281581878662,
+            -0.018213532865047455,
+            0.0011175799882039428,
+            0.0022559259086847305,
+            0.018319251015782356,
+            -0.011772345751523972,
+            0.0018283382523804903,
+            0.003532081376761198,
+            -0.0037472909316420555,
+            -0.0039379592053592205,
+            0.009053908288478851,
+            -0.025115344673395157,
+            0.00822327472269535,
+            0.018983757123351097,
+            -0.001961428439244628,
+            0.003881325013935566,
+            0.010858648456633091,
+            -0.0067356848157942295,
+            0.0009245520341210067,
+            0.007690913509577513,
+            0.024647168815135956,
+            -0.005316056311130524,
+            0.017262080684304237,
+            -0.03159428760409355,
+            0.004145617596805096,
+            -0.016235115006566048,
+            -0.017277183011174202,
+            -0.015155291184782982,
+            0.001227072672918439,
+            0.01650695875287056,
+            0.008042044937610626,
+            -0.014369964599609375,
+            -0.00018417887622490525,
+            0.0029147693421691656,
+            0.0049724760465323925,
+            -0.0012723798863589764,
+            0.021203814074397087,
+            1.1548051588761155e-05,
+            0.024420632049441338,
+            -0.00772489421069622,
+            -0.027531733736395836,
+            -0.012089497409760952,
+            0.017881279811263084,
+            -0.00651292409747839,
+            0.010533946566283703,
+            0.01495140790939331,
+            -0.00295441341586411,
+            -0.011070082895457745,
+            -0.002305008703842759,
+            0.018681708723306656,
+            0.00208035996183753,
+            0.0003353211795911193,
+            0.009673107415437698,
+            0.009476776234805584,
+            -0.0017669845838099718,
+            0.022230779752135277,
+            -0.00790612306445837,
+            -0.018107816576957703,
+            0.001507411478087306,
+            0.009907195344567299,
+            0.01988990418612957,
+            -0.0159783735871315,
+            -0.013984852470457554,
+            0.003769944654777646,
+            0.01241419930011034,
+            -0.0033433008939027786,
+            -0.01773025467991829,
+            0.0017169577768072486,
+            0.028785236179828644,
+            0.00374540314078331,
+            -0.0084045035764575,
+            -0.0017112944042310119,
+            -0.017186567187309265,
+            -0.0002583931782282889,
+            -0.011175800114870071,
+            0.010511293075978756,
+            -0.004813900217413902,
+            0.0036226960364729166,
+            -0.017095953226089478,
+            0.014702217653393745,
+            0.013728111051023006,
+            0.021717296913266182,
+            0.02343897521495819,
+            -0.022155268117785454,
+            -0.015963271260261536,
+            0.005372690502554178,
+            0.0052443197928369045,
+            0.004515627399086952,
+            -0.005323607474565506,
+            0.0013752651866525412,
+            0.021264225244522095,
+            0.008555527776479721,
+            -0.0038152518682181835,
+            0.0107378289103508,
+            -0.011115389876067638,
+            -0.014513437636196613,
+            -0.008978395722806454,
+            -0.003983266651630402,
+            -0.018575992435216904,
+            -0.0028241549152880907,
+            0.012142355553805828,
+            0.011077634058892727,
+            -0.01480038370937109,
+            0.009295547381043434,
+            0.015917964279651642,
+            -0.004217354115098715,
+            -0.007706015836447477,
+            -0.010163936764001846,
+            -0.0008150594076141715,
+            0.030023634433746338,
+            0.00019161211093887687,
+            -0.006826299242675304,
+            0.014324656687676907,
+            -0.011930921114981174,
+            0.008208171464502811,
+            -0.02019195258617401,
+            -0.01150050200521946,
+            0.00011356321192579344,
+            -0.01032251212745905,
+            -0.016884518787264824,
+            -0.016748597845435143,
+            0.01916498690843582,
+            0.0055161635391414165,
+            -0.008449810557067394,
+            -0.035581331700086594,
+            0.025432495400309563,
+            -0.0018585430225357413,
+            0.040988001972436905,
+            0.006154241040349007,
+            0.007377538364380598,
+            -0.0005111229256726801,
+            0.015185495838522911,
+            0.011334375478327274,
+            -0.020841356366872787,
+            0.015140187926590443,
+            0.007173655088990927,
+            0.0011477848747745156,
+            0.007222738116979599,
+            -0.0025617501232773066,
+            0.0068036457523703575,
+            -0.0103753712028265,
+            -0.0005738924373872578,
+            -0.014415271580219269,
+            -0.027093762531876564,
+            -0.015412031672894955,
+            0.013222179375588894,
+            -0.00041154128848575056,
+            -0.0036283594090491533,
+            -0.018319251015782356,
+            -0.01789638213813305,
+            -0.01919519156217575,
+            -0.0075927479192614555,
+            -0.024209199473261833,
+            -0.008132659830152988,
+            -0.00958249345421791,
+            0.0043004173785448074,
+            0.015117534436285496,
+            0.010065770708024502,
+            -0.003264013212174177,
+            0.027697861194610596,
+            0.019316010177135468,
+            0.013599740341305733,
+            0.00011427114077378064,
+            -0.01158356573432684,
+            0.013063604012131691,
+            0.02665579319000244,
+            -0.0025277696549892426,
+            0.003892651991918683,
+            0.00730957742780447,
+            -0.006309041287750006,
+            0.014083018526434898,
+            -0.008336542174220085,
+            0.0031318669207394123,
+            0.009091664105653763,
+            -0.016099192202091217,
+            0.0015602700877934694,
+            0.02038828283548355,
+            0.013471369631588459,
+            0.00420225178822875,
+            0.0009920409647747874,
+            0.008985946886241436,
+            0.008110006339848042,
+            -0.006271285004913807,
+            -0.012240521609783173,
+            -0.005006456281989813,
+            0.006441187579184771,
+            0.0109719168394804,
+            0.012165009044110775,
+            -0.020765844732522964,
+            0.017322489991784096,
+            0.016265319660305977,
+            -0.013577086851000786,
+            0.012512365356087685,
+            -0.0038662226870656013,
+            0.003658564295619726,
+            0.015125085599720478,
+            0.008985946886241436,
+            -0.0002326718531548977,
+            -0.013939544558525085,
+            0.012097048573195934,
+            -0.02522106096148491,
+            0.004364602733403444,
+            -0.008729205466806889,
+            -0.0310203954577446,
+            0.012572774663567543,
+            -0.02396756038069725,
+            -0.0006475168047472835,
+            0.005704943556338549,
+            -0.013750764541327953,
+            0.001948213786818087,
+            -0.010752932168543339,
+            0.005131051409989595,
+            0.006071177776902914,
+            -0.007777752820402384,
+            -0.016718393191695213,
+            -0.00514615373685956,
+            1.4519867363560479e-05,
+            -0.000538968073669821,
+            -0.023423872888088226,
+            0.02291039004921913,
+            -0.006173118948936462,
+            -0.017986996099352837,
+            0.0004884693189524114,
+            -0.02291039004921913,
+            -0.018606197088956833,
+            -0.006078728940337896,
+            -0.005923929158598185,
+            -0.021853219717741013,
+            -0.00916717667132616,
+            0.0206450242549181,
+            -0.005478407256305218,
+            -0.012535018846392632,
+            -0.024103481322526932,
+            -0.0015527188079431653,
+            0.000700375298038125,
+            -0.008638591505587101,
+            -0.015268558636307716,
+            0.015155291184782982,
+            0.016960032284259796,
+            -0.019572751596570015,
+            -0.010662317276000977,
+            0.006414758041501045,
+            -0.0009122812771238387,
+            0.01859109476208687,
+            0.0063958801329135895,
+            -0.012361341156065464,
+            -0.011560912244021893,
+            0.014468129724264145,
+            0.00952963437885046,
+            0.021626682952046394,
+            0.006214650813490152,
+            0.01090395636856556,
+            0.0057011679746210575,
+            -0.0025787402410060167,
+            -0.009250239469110966,
+            -9.225167559634428e-06,
+            0.010987019166350365,
+            -0.005384017247706652,
+            -0.009605146944522858,
+            0.0023201112635433674,
+            -0.0031941644847393036,
+            0.00692824088037014,
+            -0.010156385600566864,
+            -0.0064223092049360275,
+            -0.027169276028871536,
+            -0.019270703196525574,
+            -0.019708674401044846,
+            -0.009091664105653763,
+            0.00814021099358797,
+            -0.008653693832457066,
+            -0.014966510236263275,
+            0.0041154129430651665,
+            0.014279349707067013,
+            -0.03543030470609665,
+            -0.009522083215415478,
+            -0.004753490444272757,
+            -0.0029355352744460106,
+            -0.036215633153915405,
+            0.019844595342874527,
+            0.025115344673395157,
+            -0.0005441594985313714,
+            -0.000711230153683573,
+            -0.0030582426115870476,
+            -0.012655838392674923,
+            -0.008978395722806454,
+            0.002072808798402548,
+            0.0020067356526851654,
+            -0.0270786602050066,
+            0.005285851191729307,
+            0.01388668641448021,
+            0.021490762010216713,
+            0.008593283593654633,
+            -0.015495095402002335,
+            0.0036510128993541002,
+            -0.024677373468875885,
+            -0.025402290746569633,
+            -0.0220495518296957,
+            -0.004557158797979355,
+            -0.006018319167196751,
+            0.004606241825968027,
+            0.0056332070380449295,
+            0.01842496730387211,
+            0.00719253346323967,
+            0.005493509583175182,
+            -0.01972377672791481,
+            0.010911507532000542,
+            0.0009377666283398867,
+            0.0005111229256726801,
+            -0.012897477485239506,
+            -0.01412077434360981,
+            0.02343897521495819,
+            -0.00041319310548715293,
+            0.014551193453371525,
+            0.006029645912349224,
+            -0.009650453925132751,
+            -0.007966533303260803,
+            -0.0012940897140651941,
+            -0.026066798716783524,
+            0.016325728967785835,
+            -0.011168248951435089,
+            0.0006970716640353203,
+            -0.015555505640804768,
+            -0.00976372230798006,
+            0.022155268117785454,
+            0.012640736065804958,
+            -0.0025881794281303883,
+            0.015933066606521606,
+            0.018349455669522285,
+            0.0008315776940435171,
+            0.005474631674587727,
+            -0.017971893772482872,
+            -0.007230289280414581,
+            0.008396952413022518,
+            -0.0035188666079193354,
+            0.008615938015282154,
+            -0.023650409653782845,
+            -0.005969236139208078,
+            0.016612675040960312,
+            0.0214756578207016,
+            -0.021430350840091705,
+            0.001019414165057242,
+            0.0068036457523703575,
+            0.009325752034783363,
+            -0.006879157852381468,
+            -0.009733517654240131,
+            -0.015902861952781677,
+            -0.00893308874219656,
+            -0.005923929158598185,
+            0.021173609420657158,
+            -0.01853068359196186,
+            -0.004341949243098497,
+            -0.0025617501232773066,
+            0.005999441258609295,
+            -0.013962198980152607,
+            -0.0039379592053592205,
+            -0.016159603372216225,
+            -0.0009269117726944387,
+            0.013871584087610245,
+            -0.008744308724999428,
+            -0.024752886965870857,
+            0.010201692581176758,
+            0.002148320898413658,
+            -0.005055539309978485,
+            -0.0012166897067800164,
+            -0.018938450142741203,
+            0.025311676785349846,
+            -0.011628872714936733,
+            0.011190902441740036,
+            -0.012950335629284382,
+            -0.0034037106670439243,
+            0.01889314316213131,
+            0.030416298657655716,
+            0.01131927315145731,
+            -0.005587900057435036,
+            -0.018908245489001274,
+            0.015419583767652512,
+            0.0063543482683598995,
+            -0.005610553547739983,
+            -0.004553383216261864,
+            0.0215813759714365,
+            -0.017745357006788254,
+            -0.03461477532982826,
+            0.012361341156065464,
+            -0.0003303656994830817,
+            -0.0306579377502203,
+            -0.010231898166239262,
+            -0.006067402195185423,
+            -0.0015272335149347782,
+            0.016235115006566048,
+            0.006603538524359465,
+            -0.004357051569968462,
+            0.001753769931383431,
+            0.0021879649721086025,
+            0.002920432947576046,
+            -0.008457361720502377,
+            -0.0013611067552119493,
+            0.0006659228820353746,
+            0.0007739996653981507,
+            0.020871561020612717,
+            -0.016235115006566048,
+            -0.004409910179674625,
+            -0.0065997629426419735,
+            0.004776144400238991,
+            0.018047407269477844,
+            0.009801478125154972,
+            0.03265145793557167,
+            0.007396416272968054,
+            0.00351697881706059,
+            -0.0040738810785114765,
+            -0.010020463727414608,
+            0.00741529418155551,
+            0.011893165297806263,
+            -0.0017131821950897574,
+            -0.01047353632748127,
+            -0.011870511807501316,
+            -0.014241593889892101,
+            0.007626728154718876,
+            0.013199525885283947,
+            0.00045307297841645777,
+            -0.008464912883937359,
+            -0.011213555932044983,
+            0.025039833039045334,
+            -0.025900671258568764,
+            0.01892334781587124,
+            -0.006724358070641756,
+            -0.002399398945271969,
+            0.023710818961262703,
+            0.015072227455675602,
+            0.015019369311630726,
+            -0.010684970766305923,
+            -0.026957841590046883,
+            0.01670329086482525,
+            -0.0034414667170494795,
+            -0.002335213590413332,
+            -0.001184597029350698,
+            0.01063966378569603,
+            -0.03259104862809181,
+            -0.012897477485239506,
+            0.012429301626980305,
+            -0.0008688617963343859,
+            0.012225419282913208,
+            -0.01611429452896118,
+            0.02363530732691288,
+            0.010979468002915382,
+            0.01697513461112976,
+            -0.012731350027024746,
+            0.00012223531666677445,
+            0.023227540776133537,
+            -0.00040918152080848813,
+            -0.010231898166239262,
+            -0.004474095534533262,
+            -0.007222738116979599,
+            -0.0006470448570325971,
+            -0.013577086851000786,
+            0.012165009044110775,
+            -0.00468930508941412,
+            0.011221107095479965,
+            0.018273942172527313,
+            0.019738879054784775,
+            0.021264225244522095,
+            0.0031998278573155403,
+            -0.0008315776940435171,
+            -0.006905586924403906,
+            0.005625655874609947,
+            -0.0019595406483858824,
+            -0.00024801024119369686,
+            -0.02582515962421894,
+            0.021369941532611847,
+            -0.008797166869044304,
+            0.009408814832568169,
+            0.004055003169924021,
+            1.6061083442764357e-05,
+            0.015721632167696953,
+            0.0012761554680764675,
+            -0.0018764771521091461,
+            0.004530729725956917,
+            -0.012368892319500446,
+            0.00458358833566308,
+            -0.011636423878371716,
+            0.010465985164046288,
+            0.018455171957612038,
+            -0.007702240254729986,
+            -0.009053908288478851,
+            0.009295547381043434,
+            0.0027146621141582727,
+            -0.006724358070641756,
+            0.0071849822998046875,
+            0.010858648456633091,
+            -0.00734355766326189,
+            -0.005274524446576834,
+            0.006418533623218536,
+            -0.01090395636856556,
+            -0.0007268045446835458,
+            0.006645070388913155,
+            -0.015661222860217094,
+            0.01961805857717991,
+            0.003146969247609377,
+            0.01816822588443756,
+            -0.009635351598262787,
+            0.008827371522784233,
+            -0.006592211779206991,
+            0.013954647816717625,
+            -0.02127932757139206,
+            -0.015585710294544697,
+            0.00866879615932703,
+            -0.008744308724999428,
+            0.017277183011174202,
+            0.008593283593654633,
+            0.011175800114870071,
+            -0.004334398079663515,
+            -0.013796071521937847,
+            0.004768592771142721,
+            -0.0010128068970516324,
+            0.0040474520064890385,
+            -0.01958785392343998,
+            0.026942739263176918,
+            0.01955764926970005,
+            0.020358078181743622,
+            -0.016491856426000595,
+            -0.016295524314045906,
+            0.004104085732251406,
+            0.02685212530195713,
+            -0.01547999307513237,
+            0.0008485679281875491,
+            0.01474752463400364,
+            -0.007577645592391491,
+            -0.004270212724804878,
+            0.02052420563995838,
+            -0.0026561403647065163,
+            -0.0376654677093029,
+            -0.013388305902481079,
+            0.00564453424885869,
+            0.003026149934157729,
+            0.003375393571332097,
+            -0.0374540314078331,
+            0.003987042233347893,
+            -0.025447597727179527,
+            0.004677978344261646,
+            -0.008268581703305244,
+            -0.015631018206477165,
+            0.0003327254671603441,
+            -0.001685808994807303,
+            0.009650453925132751,
+            0.005436875391751528,
+            0.00408520782366395,
+            -0.006392104551196098,
+            0.012210316024720669,
+            -0.006720582488924265,
+            -0.002958188997581601,
+            -0.005546368192881346,
+            0.018485376611351967,
+            0.005127275828272104,
+            0.014958959072828293,
+            0.0043872566893696785,
+            -0.01647675409913063,
+            0.00783816259354353,
+            0.010269653983414173,
+            -0.008162864483892918,
+            -0.00756631838157773,
+            0.002348428126424551,
+            -0.004610017407685518,
+            -0.004081432241946459,
+            -0.02061481960117817,
+            -0.0037888227961957455,
+            0.008291235193610191,
+            0.021958936005830765,
+            -0.0025541989598423243,
+            -0.008646142669022083,
+            -0.0029279841110110283,
+            -0.0034716716036200523,
+            0.010465985164046288,
+            -0.015419583767652512,
+            0.021792810410261154,
+            -0.002814715728163719,
+            0.011092736385762691,
+            0.010805790312588215,
+            0.00801939144730568,
+            0.0055388170294463634,
+            0.008442259393632412,
+            0.004217354115098715,
+            0.00964290276169777,
+            0.005595451220870018,
+            -0.02069033309817314,
+            -0.009038805961608887,
+            0.012429301626980305,
+            -0.004934719763696194,
+            -0.015434686094522476,
+            0.03231920674443245,
+            0.008087352849543095,
+            0.0006078729056753218,
+            0.007672035600990057,
+            0.03159428760409355,
+            0.0015848114853724837,
+            0.005772904958575964,
+            0.01123620942234993,
+            0.010828443802893162,
+            -0.0029997206293046474,
+            -0.012429301626980305,
+            -0.044038690626621246,
+            0.0008792447042651474,
+            0.0015404480509459972,
+            0.020750742405653,
+            -0.033225350081920624,
+            0.009590044617652893,
+            -0.013048501685261726,
+            0.008087352849543095,
+            -0.016990236937999725,
+            0.007830611430108547,
+            0.006331694778054953,
+            0.01822863519191742,
+            0.010609458200633526,
+            -0.011032327078282833,
+            0.008865127339959145,
+            0.0015904748579487205,
+            -0.011334375478327274,
+            0.0014083017595112324,
+            0.02810562588274479,
+            0.020599717274308205,
+            0.011168248951435089,
+            -0.0018028528429567814,
+            0.01853068359196186,
+            0.014807934872806072,
+            -0.009129419922828674,
+            -0.027093762531876564,
+            0.003180949715897441,
+            -0.0031564084347337484,
+            -0.004666651599109173,
+            0.01128906849771738,
+            -0.0043457248248159885,
+            0.006361899431794882,
+            0.01750371977686882,
+            -0.006222201976925135,
+            -0.0006158960750326514,
+            0.010360268875956535,
+            0.008117557503283024,
+            -0.0023371013812720776,
+            0.016250217333436012,
+            -0.015615914948284626,
+            0.021158507093787193,
+            -0.025900671258568764,
+            -0.0014696554280817509,
+            0.012625633738934994,
+            0.006958445534110069,
+            0.017820870503783226,
+            -0.01218011137098074,
+            -0.02519085630774498,
+            -0.00976372230798006,
+            0.01249726302921772,
+            0.012957886792719364,
+            0.00420225178822875,
+            0.009031254798173904,
+            0.01196867786347866,
+            -0.004927168600261211,
+            -0.0032451350707560778,
+            0.0012676604092121124,
+            -0.013237281702458858,
+            -0.00722651369869709,
+            -0.009741068817675114,
+            0.010065770708024502,
+            0.0007268045446835458,
+            -0.01079068798571825,
+            0.007283147890120745,
+            -0.02197403833270073,
+            -0.008351645432412624,
+            0.004602466244250536,
+            -0.0024900136049836874,
+            0.0018670381978154182,
+            -0.015313866548240185,
+            0.0074568260461091995,
+            0.00040257422369904816,
+            -0.003881325013935566,
+            -0.010828443802893162,
+            0.022260984405875206,
+            0.00450807623565197,
+            0.003953061532229185,
+            0.013071155175566673,
+            -0.00681119691580534,
+            -0.011538257822394371,
+            0.01544978842139244,
+            -0.009242688305675983,
+            0.0013658262323588133,
+            -0.005939031485468149,
+            -0.012240521609783173,
+            -0.008736756630241871,
+            -0.010488639585673809,
+            -0.032500434666872025,
+            -0.024420632049441338,
+            0.016386138275265694,
+            0.0015574382850900292,
+            0.012338686734437943,
+            0.017216771841049194,
+            -0.012165009044110775,
+            0.008132659830152988,
+            -0.03636665642261505,
+            0.00128653843421489,
+            -0.02155117131769657,
+            -0.01047353632748127,
+            0.0008839642396196723,
+            0.00810245517641306,
+            0.014143427833914757,
+            -0.008011840283870697,
+            0.00790612306445837,
+            -0.0018962991889566183,
+            0.018787425011396408,
+            0.015615914948284626,
+            0.003647237317636609,
+            0.0012893701205030084,
+            -0.007173655088990927,
+            -0.00011834171164082363,
+            0.022064654156565666,
+            -0.002271028235554695,
+            -0.017443308606743813,
+            0.0176396407186985,
+            0.022215677425265312,
+            0.0025202184915542603,
+            -0.02496432140469551,
+            0.005542592611163855,
+            0.0025277696549892426,
+            0.017020441591739655,
+            -0.0040738810785114765,
+            -0.010843546129763126,
+            -0.004610017407685518,
+            -0.0027580817695707083,
+            0.03881325200200081,
+            0.018409864977002144,
+            -0.005047988146543503,
+            -0.03231920674443245,
+            0.0021709746215492487,
+            -0.0028807888738811016,
+            1.456411337130703e-05,
+            0.01988990418612957,
+            0.02393735572695732,
+            0.023559793829917908,
+            -0.0032470228616148233,
+            0.007739996537566185,
+            -0.019980518147349358,
+            0.04331377521157265,
+            -0.007853264920413494,
+            -0.016884518787264824,
+            -0.015102432109415531,
+            -0.009046357125043869,
+            0.003919081296771765,
+            0.008532874286174774,
+            0.006611089687794447,
+            -0.008261030539870262,
+            -0.004704407416284084,
+            0.010428229346871376,
+            0.008631040342152119,
+            -0.017956791445612907,
+            0.013003193773329258,
+            -0.02393735572695732,
+            0.016718393191695213,
+            -0.01603878289461136,
+            0.00035797484451904893,
+            0.0037982617504894733,
+            -0.012202764861285686,
+            -0.029842406511306763,
+            -0.01176479458808899,
+            0.017186567187309265,
+            0.013448716141283512,
+            0.008547976613044739,
+            0.0035679496359080076,
+            0.005757802166044712,
+            -0.009544737637043,
+            -0.00016801456513348967,
+            0.01191581878811121,
+            0.027365608140826225,
+            0.01703554391860962,
+            0.0129427844658494,
+            0.025432495400309563,
+            -0.00737376231700182,
+            -0.039598576724529266,
+            -0.0035056520719081163,
+            0.00943146925419569,
+            0.007883469574153423,
+            0.002072808798402548,
+            0.002338989172130823,
+            -0.019980518147349358,
+            0.0062184263952076435,
+            -0.0003690656740218401,
+            -0.00681119691580534,
+            -0.020901765674352646,
+            0.007966533303260803,
+            -0.013214628212153912,
+            -0.0005021558608859777,
+            0.0007466264651156962,
+            0.00331120821647346,
+            0.007453050464391708,
+            0.012383994646370411,
+            0.0020916869398206472,
+            -0.0013309018686413765,
+            -0.006826299242675304,
+            -0.008079801686108112,
+            -0.016582470387220383,
+            0.01170438528060913,
+            -0.01362239383161068,
+            -0.005195236764848232,
+            -0.004077656660228968,
+            -0.003949285950511694,
+            0.002650476759299636,
+            0.02088666334748268,
+            0.0042098029516637325,
+            0.0027486425824463367,
+            0.01301074493676424,
+            0.008442259393632412,
+            0.0039115301333367825,
+            0.03476579859852791,
+            -0.01836455799639225,
+            -0.0063279191963374615,
+            -0.014150978997349739,
+            -0.01544978842139244,
+            -0.019013961777091026,
+            0.022019345313310623,
+            -0.004296641796827316,
+            -0.010133732110261917,
+            0.007200084626674652,
+            0.014679564163088799,
+            -0.013788520358502865,
+            3.648771235020831e-05,
+            0.010964365676045418,
+            -0.015117534436285496,
+            0.010896405205130577,
+            -0.009544737637043,
+            0.010609458200633526,
+            0.00810245517641306,
+            0.03956837207078934,
+            -0.009295547381043434,
+            -0.014468129724264145,
+            0.0026523645501583815,
+            0.009861888363957405,
+            0.018470274284482002,
+            -0.024677373468875885,
+            0.0013148555299267173,
+            -0.0012733237817883492,
+            -0.000611648487392813,
+            -0.00662241643294692,
+            -0.006252407096326351,
+            0.014415271580219269,
+            -0.006520475260913372,
+            -0.01055660005658865,
+            -0.024949217215180397,
+            -0.017971893772482872,
+            -0.00961269810795784,
+            0.022653648629784584,
+            0.024284711107611656,
+            -0.005399119574576616,
+            -0.005803109612315893,
+            0.0011978116817772388,
+            -0.01567632518708706,
+            0.01700533926486969,
+            0.013629944995045662,
+            -0.02535698376595974,
+            -0.0019000747706741095,
+            -0.009446571581065655,
+            -0.006365675013512373,
+            -0.014543642289936543,
+            0.00878206454217434,
+            0.05584879592061043,
+            -0.02655007503926754,
+            -0.0003390967904124409,
+            -0.01733759231865406,
+            0.00642986036837101,
+            -0.010722726583480835,
+            -0.005074417218565941,
+            -0.01647675409913063,
+            -0.01040557585656643,
+            -0.006656397134065628,
+            -0.006686601787805557,
+            -0.004311744589358568,
+            0.00869900081306696,
+            -0.007294474635273218,
+            -0.02227608673274517,
+            0.016250217333436012,
+            -0.031050600111484528,
+            0.002148320898413658,
+            0.010382922366261482,
+            -0.0017037431243807077,
+            0.019104577600955963,
+            -0.028030114248394966,
+            0.01165152620524168,
+            0.004927168600261211,
+            0.00137148960493505,
+            0.007083040662109852,
+            -0.014641808345913887,
+            -0.003188501112163067,
+            -0.01683921180665493,
+            -0.003941734787076712,
+            0.009748619981110096,
+            -0.010692521929740906,
+            0.023212438449263573,
+            -0.014483232982456684,
+            -0.011070082895457745,
+            0.015827348455786705,
+            0.013977301307022572,
+            0.014672012999653816,
+            -0.013546881265938282,
+            -0.0030412522610276937,
+            -0.010496190749108791,
+            0.016688188537955284,
+            -0.0021974039264023304,
+            -0.021883424371480942,
+            -0.014234042726457119,
+            0.02055441029369831,
+            -0.008042044937610626,
+            0.022381804883480072,
+            -0.002335213590413332,
+            -0.001269548200070858,
+            -0.008472464047372341,
+            0.005191461183130741,
+            -0.0006215595058165491,
+            -0.0025674134958535433,
+            0.017277183011174202,
+            -0.0025070037227123976,
+            0.026776611804962158,
+            -0.01498161256313324,
+            0.015585710294544697,
+            -0.029162796214222908,
+            -0.009204932488501072,
+            -0.00783816259354353,
+            0.0014545529847964644,
+            -0.00017084626597352326,
+            0.0008608386269770563,
+            0.030340787023305893,
+            -0.01291257981210947,
+            -0.02241200953722,
+            0.023741023615002632,
+            -0.015933066606521606,
+            -0.0328930988907814,
+            0.005587900057435036,
+            -0.001396975014358759,
+            0.004768592771142721,
+            0.008917986415326595,
+            -0.007083040662109852,
+            -0.025734543800354004,
+            0.012844618409872055,
+            0.01806250959634781,
+            0.0030374766793102026,
+            4.574975173454732e-05,
+            -0.015004266053438187,
+            -0.019316010177135468,
+            -0.011281517334282398,
+            0.008366747759282589,
+            -0.0040738810785114765,
+            0.020463796332478523,
+            -0.017262080684304237,
+            -0.00405122758820653,
+            -0.006361899431794882,
+            -0.008495118468999863,
+            0.011424990370869637,
+            0.028966465964913368,
+            0.010601907037198544,
+            0.004417461343109608,
+            -0.00123084825463593,
+            0.0015225139213725924,
+            0.0021728624124079943,
+            0.008064698427915573,
+            0.009325752034783363,
+            -0.006444963160902262,
+            -0.0016839212039485574,
+            -0.009287996217608452,
+            -0.0024088378995656967,
+            -0.01335810124874115,
+            0.003643461735919118,
+            -0.003515091026201844,
+            0.005678514484316111,
+            -0.01975398138165474,
+            0.01020924374461174,
+            -0.010284756310284138,
+            0.0031564084347337484,
+            -0.02088666334748268,
+            -0.002552310936152935,
+            0.017473513260483742,
+            -0.010277205146849155,
+            0.018621299415826797,
+            0.0010618898086249828,
+            0.026097003370523453,
+            0.008042044937610626,
+            -0.0030242621432989836,
+            0.003998368978500366,
+            0.0055652461014688015,
+            0.00476481718942523,
+            0.014173632487654686,
+            0.004153168760240078,
+            -0.012618082575500011,
+            0.006241080351173878,
+            0.012550121173262596,
+            -0.008532874286174774,
+            0.011674179695546627,
+            -0.0009844898013398051,
+            0.01700533926486969,
+            -0.010775585658848286,
+            0.010911507532000542,
+            0.0079363277181983,
+            0.007891020737588406,
+            -0.023378565907478333,
+            -0.0038020373322069645,
+            0.023650409653782845,
+            -0.014543642289936543,
+            0.07092101871967316,
+            -0.005893724039196968,
+            0.025508008897304535,
+            -0.008570630103349686,
+            -0.012104599736630917,
+            -0.008034493774175644,
+            -0.004323071334511042,
+            0.007898571901023388,
+            -0.010601907037198544,
+            0.027456222102046013,
+            -0.011606219224631786,
+            0.00439103227108717,
+            -0.006788543425500393,
+            0.003760505700483918,
+            0.034916821867227554,
+            0.011198453605175018,
+            0.021369941532611847,
+            0.012210316024720669,
+            0.010662317276000977,
+            0.00040918152080848813,
+            -0.004693080671131611,
+            0.010118629783391953,
+            0.006414758041501045,
+            0.012776657938957214,
+            -0.014996714890003204,
+            0.012217868119478226,
+            0.01020924374461174,
+            0.01683921180665493,
+            -0.014611602760851383,
+            -0.004719510208815336,
+            0.011266414076089859,
+            0.0012761554680764675,
+            -0.012482160702347755,
+            -0.011643975041806698,
+            0.015079778619110584,
+            0.015042022801935673,
+            0.008563078939914703,
+            0.00810245517641306,
+            -0.00987699069082737,
+            0.0012563335476443172,
+            0.008744308724999428,
+            -0.01188561413437128,
+            0.005723821930587292,
+            -0.006875382270663977,
+            -0.006346797104924917,
+            0.007951430976390839,
+            0.003971939906477928,
+            -0.002012399025261402,
+            -0.013765866868197918,
+            0.010994570329785347,
+            -0.010254551656544209,
+            -0.01567632518708706,
+            -0.012044189497828484,
+            0.000821666675619781,
+            0.012202764861285686,
+            -0.0076607088558375835,
+            -0.002223833231255412,
+            -0.012587876990437508,
+            0.0036453495267778635,
+            0.017292285338044167,
+            0.007588972337543964,
+            0.020660126581788063,
+            0.023227540776133537
+          ],
+          "title": "Addressing Performance Challenges",
+          "keyphrases": [
+            "Shrinking Batch Problem",
+            "Mixing Data Parallelism and Model Parallelism",
+            "hierarchical MoE",
+            "network bandwidth",
+            "computational efficiency"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "d99e760c-6f4b-449a-a7bd-0f8653749483",
+      "type": "next",
+      "source": {
+        "id": "44767af8-7eba-44fe-a7ef-fd48aa105f77",
+        "properties": {
+          "page_content": "3 Addressing Performance Challenges 3.1 The Shrinking Batch Problem\n\nOn modern CPUs and GPUs, large batch sizes are necessary for computational efficiency, so as to amortize the overhead of parameter loads and updates. If the gating network chooses k out of n experts for each example, then for a batch of b examples, each expert receives a much smaller batch of approximately kb n b examples. This causes a naive MoE implementation to become very inefficient as the number of experts increases. The solution to this shrinking batch problem is to make the original batch size as large as possible. However, batch size tends to be limited by the memory necessary to store activations between the forwards and backwards passes. We propose the following techniques for increasing the batch size: Mixing Data Parallelism and Model Parallelism: In a conventional distributed training setting, multiple copies of the model on different devices asynchronously process distinct batches of data, and parameters are synchronized through a set of parameter servers. In our technique, these different batches run synchronously so that they can be combined for the MoE layer. We distribute the standard layers of the model and the gating network according to conventional data-parallel schemes, but keep only one shared copy of each expert. Each expert in the MoE layer receives a combined batch consisting of the relevant examples from all of the data-parallel input batches. The same set of devices function as data-parallel replicas (for the standard layers and the gating networks) and as model-parallel shards (each hosting a subset of the experts). If the model is distributed over d devices, and each device processes a batch of size b, each expert receives a batch of approximately kbd nexamples. Thus, we achieve a factor of d improvement in expert batch size.\n\nIn the case of a hierarchical MoE (Section B), the primary gating network employs data parallelism, and the secondary MoEs employ model parallelism. Each secondary MoE resides on one device.\n\nThis technique allows us to increase the number of experts (and hence the number of parameters) by proportionally increasing the number of devices in the training cluster. The total batch size increases, keeping the batch size per expert constant. The memory and bandwidth requirements per device also remain constant, as do the step times, as does the amount of time necessary to process a number of training examples equal to the number of parameters in the model. It is our goal to train a trillionparameter model on a trillion-word corpus. We have not scaled our systems this far as of the writing of this paper, but it should be possible by adding more hardware.\n\nTaking Advantage of Convolutionality: In our language models, we apply the same MoE to each time step of the previous layer. If we wait for the previous layer to finish, we can apply the MoE to all the time steps together as one big batch. Doing so increases the size of the input batch to the MoE layer by a factor of the number of unrolled time steps.\n\nIncreasing Batch Size for a Recurrent MoE: We suspect that even more powerful models may involve applying a MoE recurrently. For example, the weight matrices of a LSTM or other RNN could be replaced by a MoE. Sadly, such models break the convolutional trick from the last paragraph, since the input to the MoE at one timestep depends on the output of the MoE at the previous timestep. Gruslys et al. (2016) describe a technique for drastically reducing the number of stored activations in an unrolled RNN, at the cost of recomputing forward activations. This would allow for a large increase in batch size.\n\n3.2 Network Bandwidth\n\nAnother major performance concern in distributed computing is network bandwidth. Since the experts are stationary (see above) and the number of gating parameters is small, most of the communication involves sending the inputs and outputs of the experts across the network. To maintain computational efficiency, the ratio of an expert's computation to the size of its input and output must exceed the ratio of computational to network capacity of the computing device. For GPUs, this may be thousands to one. In our experiments, we use experts with one hidden layer containing thousands of RELU-activated units. Since the weight matrices in the expert have sizes input_size×hidden_size and hidden_size × output_size, the ratio of computation to input and output is equal to the size of the hidden layer. Conveniently, we can increase computational efficiency simply by using a larger hidden layer, or more hidden layers.\n\n",
+          "embedding": [
+            0.0014724871143698692,
+            -0.0002654724521562457,
+            -0.015004266053438187,
+            -0.010133732110261917,
+            0.02161158062517643,
+            0.014150978997349739,
+            -0.020856458693742752,
+            0.05349281430244446,
+            -0.016355933621525764,
+            0.02585536427795887,
+            0.004545832052826881,
+            -0.0036283594090491533,
+            0.015313866548240185,
+            0.006528026424348354,
+            -0.01309380866587162,
+            0.013184423558413982,
+            0.010216795839369297,
+            0.007860816083848476,
+            0.012504814192652702,
+            0.009748619981110096,
+            0.023076515644788742,
+            -0.03379924222826958,
+            -0.03869242966175079,
+            -0.013516676612198353,
+            -0.0013828164665028453,
+            0.015766939148306847,
+            0.0215813759714365,
+            0.00212189182639122,
+            -0.04113902524113655,
+            0.013191974721848965,
+            0.018107816576957703,
+            -0.0014149091439321637,
+            -0.03023506887257099,
+            -0.02502473071217537,
+            -0.05300953611731529,
+            -0.029328923672437668,
+            0.05705698952078819,
+            0.01131927315145731,
+            -0.023907151073217392,
+            0.005855968222022057,
+            0.03201715648174286,
+            0.013720559887588024,
+            -0.042709678411483765,
+            -0.008555527776479721,
+            0.0013847042573615909,
+            0.009076561778783798,
+            -0.02502473071217537,
+            0.017443308606743813,
+            0.015933066606521606,
+            -0.04283049702644348,
+            0.0038530079182237387,
+            -0.06312816590070724,
+            0.014974061399698257,
+            0.007317128591239452,
+            -0.04415951296687126,
+            0.0005852192407473922,
+            0.025674134492874146,
+            0.01730738766491413,
+            0.05485203489661217,
+            0.0018349455203860998,
+            0.036547884345054626,
+            0.0009023703169077635,
+            -0.05071396753191948,
+            0.006059851031750441,
+            0.00730957742780447,
+            -0.010163936764001846,
+            -0.05470100790262222,
+            0.03787690028548241,
+            -0.01833435334265232,
+            -0.007256718818098307,
+            -0.04116922989487648,
+            0.0024220526684075594,
+            0.0015055236872285604,
+            0.008336542174220085,
+            0.018711913377046585,
+            -0.010873750783503056,
+            0.04974741116166115,
+            0.04283049702644348,
+            0.01686941646039486,
+            0.0027278768830001354,
+            -0.016099192202091217,
+            0.009688210673630238,
+            -0.012217868119478226,
+            0.01082089263945818,
+            0.038269564509391785,
+            -0.012059291824698448,
+            -0.02571944147348404,
+            -0.0495963878929615,
+            -0.01418873481452465,
+            -0.005553919356316328,
+            -0.021234020590782166,
+            0.014196285977959633,
+            0.013894237577915192,
+            -0.005689841229468584,
+            -0.003618920221924782,
+            0.036245837807655334,
+            -0.012089497409760952,
+            -0.01176479458808899,
+            -0.0005918265669606626,
+            -0.008502669632434845,
+            0.00020046118879690766,
+            0.011228658258914948,
+            0.019451932981610298,
+            -0.000972219102550298,
+            0.02058461494743824,
+            0.00044788149534724653,
+            -0.009514532051980495,
+            0.00590127520263195,
+            0.02843788079917431,
+            0.10390473157167435,
+            0.015555505640804768,
+            -0.015072227455675602,
+            -0.012814413756132126,
+            0.004606241825968027,
+            -0.0477236844599247,
+            -0.010496190749108791,
+            -0.010481088422238827,
+            -0.038209155201911926,
+            -0.004406134597957134,
+            -0.001600857824087143,
+            0.019874799996614456,
+            0.020040927454829216,
+            -0.014906100928783417,
+            -0.0016169041628018022,
+            -0.07424356043338776,
+            0.03325555473566055,
+            -0.01044333167374134,
+            -0.01090395636856556,
+            -0.01703554391860962,
+            -0.03902468457818031,
+            0.005587900057435036,
+            0.017971893772482872,
+            -0.015223251655697823,
+            -0.05627166107296944,
+            0.007860816083848476,
+            0.004719510208815336,
+            0.055607154965400696,
+            0.05231482535600662,
+            0.014755076728761196,
+            0.0007310521323233843,
+            0.017911484465003014,
+            -0.040021445602178574,
+            -0.01766984537243843,
+            -0.005716270767152309,
+            -0.013614842668175697,
+            0.01789638213813305,
+            -0.002208730671554804,
+            0.04739143326878548,
+            -0.03530948609113693,
+            0.027259889990091324,
+            -0.02774316817522049,
+            0.017745357006788254,
+            -8.854980842443183e-05,
+            0.05470100790262222,
+            -0.012142355553805828,
+            -0.016386138275265694,
+            -0.02513044700026512,
+            0.005916377995163202,
+            -0.037937309592962265,
+            -0.011938472278416157,
+            -0.013388305902481079,
+            -0.02419409714639187,
+            -0.014332207851111889,
+            0.023121824488043785,
+            -0.001210082438774407,
+            0.038239359855651855,
+            0.04859962686896324,
+            -0.019980518147349358,
+            -0.08771492540836334,
+            0.021203814074397087,
+            0.005720046348869801,
+            0.007721118628978729,
+            0.01756412908434868,
+            0.026761509478092194,
+            0.017382899299263954,
+            0.02369571663439274,
+            0.016914723441004753,
+            0.01736779697239399,
+            -0.029344025999307632,
+            0.0517711378633976,
+            -0.023182233795523643,
+            -0.016189808025956154,
+            0.021128302440047264,
+            0.012663389556109905,
+            0.033134736120700836,
+            0.0027448670007288456,
+            0.006973547860980034,
+            0.028332162648439407,
+            0.011832755990326405,
+            0.011696834117174149,
+            0.03301391750574112,
+            0.030839167535305023,
+            0.018802527338266373,
+            -0.01302584819495678,
+            0.024405529722571373,
+            -0.019874799996614456,
+            -0.012504814192652702,
+            0.0034074862487614155,
+            0.016612675040960312,
+            -0.00446654437109828,
+            -0.014022608287632465,
+            -0.02960076741874218,
+            -0.021883424371480942,
+            -0.02241200953722,
+            0.02618761733174324,
+            0.033195145428180695,
+            0.019149884581565857,
+            0.026942739263176918,
+            0.03990062698721886,
+            -0.031111011281609535,
+            0.03476579859852791,
+            -0.009250239469110966,
+            -0.046122826635837555,
+            -0.04047451913356781,
+            -0.032168179750442505,
+            0.003192276693880558,
+            -0.011794999241828918,
+            0.039598576724529266,
+            -0.012376443482935429,
+            -0.03467518463730812,
+            -0.016809007152915,
+            0.017488617449998856,
+            0.009854337200522423,
+            -0.016068987548351288,
+            -0.005350036546587944,
+            -0.02041848748922348,
+            -0.00730957742780447,
+            -0.017715152353048325,
+            0.021928731352090836,
+            -0.020735640078783035,
+            -0.013373203575611115,
+            0.035853173583745956,
+            -0.0014498335076496005,
+            0.020327873528003693,
+            0.015238353982567787,
+            -0.016416342929005623,
+            0.021989140659570694,
+            0.015736734494566917,
+            -0.007807957474142313,
+            -0.010586804710328579,
+            -0.024284711107611656,
+            0.006648845970630646,
+            -0.05276789888739586,
+            -0.03703116253018379,
+            0.0047950223088264465,
+            0.012988091446459293,
+            -0.01902906410396099,
+            0.04277008771896362,
+            -0.003701983718201518,
+            0.0010835995199158788,
+            -0.0025844036135822535,
+            -0.05162011459469795,
+            0.005735148675739765,
+            0.0007291642832569778,
+            -0.003358403453603387,
+            0.03237961605191231,
+            0.012134804390370846,
+            -0.02399776503443718,
+            -0.004719510208815336,
+            -0.01756412908434868,
+            0.030023634433746338,
+            -0.004172047134488821,
+            -0.029253412038087845,
+            0.02963097207248211,
+            -0.023574896156787872,
+            -0.00843470823019743,
+            0.03497723489999771,
+            0.04434074088931084,
+            0.016718393191695213,
+            0.006101382430642843,
+            -0.005893724039196968,
+            -0.0357927642762661,
+            0.0015734846238046885,
+            -0.031171420589089394,
+            0.03947775810956955,
+            -0.02907218225300312,
+            0.0018104041228070855,
+            -0.008215723559260368,
+            0.02121891640126705,
+            0.035218872129917145,
+            0.015442237257957458,
+            0.009605146944522858,
+            -0.00013780969311483204,
+            0.01214990671724081,
+            -0.004715734627097845,
+            -0.018847836181521416,
+            -0.02094707265496254,
+            0.00012164537474745885,
+            -0.03198695182800293,
+            -0.015117534436285496,
+            -0.005882397294044495,
+            -0.017020441591739655,
+            -0.01464935950934887,
+            0.005670963320881128,
+            0.04319295659661293,
+            0.005935255903750658,
+            -0.016522061079740524,
+            0.012255623936653137,
+            0.030552219599485397,
+            -0.015125085599720478,
+            -0.044401150196790695,
+            -0.003303657053038478,
+            -0.0013507237890735269,
+            -0.021671989932656288,
+            -0.03639686107635498,
+            -0.015283661894500256,
+            -0.017549026757478714,
+            0.011606219224631786,
+            0.008646142669022083,
+            -0.007611625827848911,
+            0.0310203954577446,
+            0.012044189497828484,
+            0.014807934872806072,
+            -0.03974959999322891,
+            -0.03370862826704979,
+            0.011606219224631786,
+            0.025447597727179527,
+            0.01756412908434868,
+            -0.04557913914322853,
+            -0.04491463303565979,
+            0.03869242966175079,
+            0.017609436064958572,
+            -0.03325555473566055,
+            0.013146666809916496,
+            -0.027561938390135765,
+            -0.00012601091293618083,
+            0.020403385162353516,
+            0.06324898451566696,
+            -0.040927592664957047,
+            -0.008827371522784233,
+            -0.01830414868891239,
+            -0.021762605756521225,
+            -0.034161701798439026,
+            -0.0007310521323233843,
+            0.005889948457479477,
+            -0.04657590016722679,
+            -0.026399051770567894,
+            -0.009363507851958275,
+            -0.004451442044228315,
+            0.013131564483046532,
+            -0.01047353632748127,
+            -0.009627800434827805,
+            -0.012519916519522667,
+            0.03905488923192024,
+            0.0010958702769130468,
+            0.010564151220023632,
+            -0.05920153483748436,
+            0.015721632167696953,
+            0.014989163726568222,
+            -0.006520475260913372,
+            0.0011581677244976163,
+            0.01686941646039486,
+            0.023257745429873466,
+            0.03678952530026436,
+            0.006999977398663759,
+            -0.03023506887257099,
+            -0.015797143802046776,
+            0.025764750316739082,
+            -0.02585536427795887,
+            -0.004817675799131393,
+            0.009129419922828674,
+            -0.013879135251045227,
+            -0.026142310351133347,
+            0.02574964612722397,
+            -0.046787336468696594,
+            -0.0031431936658918858,
+            -0.02078094705939293,
+            -0.008714103139936924,
+            0.012572774663567543,
+            -0.00676966505125165,
+            -0.01786617748439312,
+            0.017684947699308395,
+            -0.006697928532958031,
+            0.00021202399511821568,
+            0.017881279811263084,
+            0.014928754419088364,
+            0.03259104862809181,
+            -0.0055388170294463634,
+            -0.04838819429278374,
+            0.033829446882009506,
+            -0.022396907210350037,
+            0.019074371084570885,
+            0.02105279080569744,
+            -0.015797143802046776,
+            0.013569535687565804,
+            0.035218872129917145,
+            -0.0023125598672777414,
+            0.008268581703305244,
+            -0.02061481960117817,
+            0.04684774577617645,
+            -0.022094858810305595,
+            -0.03301391750574112,
+            -0.010488639585673809,
+            0.004224905278533697,
+            -0.015857553109526634,
+            0.01720166951417923,
+            0.0022748038172721863,
+            -0.04603221267461777,
+            -0.006040973123162985,
+            0.006441187579184771,
+            -0.024813296273350716,
+            0.006543128751218319,
+            0.007698464673012495,
+            0.005440651439130306,
+            -0.010617010295391083,
+            -0.008744308724999428,
+            0.0021577600855380297,
+            -0.007981635630130768,
+            0.04956618323922157,
+            -0.03639686107635498,
+            -0.0022011795081198215,
+            -0.04603221267461777,
+            -0.01892334781587124,
+            -0.008427157066762447,
+            0.03811853751540184,
+            0.016129396855831146,
+            0.0040323492139577866,
+            0.018047407269477844,
+            -0.018349455669522285,
+            -0.006403431296348572,
+            -0.02990281581878662,
+            0.03376903757452965,
+            0.025961080566048622,
+            0.010043117217719555,
+            -0.01055660005658865,
+            0.0031262035481631756,
+            0.008049596101045609,
+            -0.006932016462087631,
+            0.0002463584241922945,
+            -0.021626682952046394,
+            0.020765844732522964,
+            -0.0029147693421691656,
+            0.019119679927825928,
+            -0.027984807267785072,
+            -0.012127253226935863,
+            -0.008464912883937359,
+            0.004938495345413685,
+            0.041984762996435165,
+            -0.03234941139817238,
+            -0.012157457880675793,
+            -0.00476481718942523,
+            0.002593842800706625,
+            -0.0003669419093057513,
+            0.0014762626960873604,
+            -0.03920591250061989,
+            -0.03754464536905289,
+            -0.0429513156414032,
+            -0.015268558636307716,
+            -0.01199888251721859,
+            -0.05337199568748474,
+            -0.07581420987844467,
+            0.00016659870743751526,
+            -0.0720083937048912,
+            -0.009174727834761143,
+            0.0407765656709671,
+            -0.015087329782545567,
+            -0.018319251015782356,
+            -0.03292330354452133,
+            -0.0076833623461425304,
+            -0.01736779697239399,
+            -0.01206684298813343,
+            0.004776144400238991,
+            0.016914723441004753,
+            0.006233528722077608,
+            -0.024571657180786133,
+            0.018636401742696762,
+            -0.027003148570656776,
+            -0.028422776609659195,
+            0.007158552762120962,
+            0.0037246374413371086,
+            -0.038964275270700455,
+            -0.029872611165046692,
+            0.013629944995045662,
+            0.012036638334393501,
+            0.046153031289577484,
+            0.025598622858524323,
+            -0.0014101896667852998,
+            -0.05908071622252464,
+            0.01919519156217575,
+            -0.001417740830220282,
+            0.02940443530678749,
+            0.015631018206477165,
+            -0.005384017247706652,
+            0.01677880249917507,
+            -0.016023680567741394,
+            -0.010511293075978756,
+            0.012542570009827614,
+            -0.006025870330631733,
+            -0.0011996994726359844,
+            -0.02926851436495781,
+            0.017186567187309265,
+            0.03049181029200554,
+            0.005806885194033384,
+            0.025568418204784393,
+            0.004036124795675278,
+            -0.012542570009827614,
+            -0.021128302440047264,
+            0.013645047321915627,
+            0.008540425449609756,
+            0.03159428760409355,
+            -0.03908509388566017,
+            -0.023756125941872597,
+            0.012361341156065464,
+            0.010481088422238827,
+            -0.016295524314045906,
+            -0.0004523650277405977,
+            0.01578204147517681,
+            0.021490762010216713,
+            -0.034222111105918884,
+            -0.03292330354452133,
+            0.008208171464502811,
+            0.028317060321569443,
+            0.019602956250309944,
+            -0.001834001624956727,
+            0.011674179695546627,
+            0.026731304824352264,
+            -0.0360041968524456,
+            0.0064713922329247,
+            0.012731350027024746,
+            0.034161701798439026,
+            1.8229107809020206e-05,
+            0.02393735572695732,
+            -0.02910238690674305,
+            0.03700095787644386,
+            -0.020660126581788063,
+            0.03201715648174286,
+            1.8627315512276255e-05,
+            0.011613770388066769,
+            0.03370862826704979,
+            -0.009733517654240131,
+            -0.026066798716783524,
+            0.004496749024838209,
+            0.005078192800283432,
+            -0.0035887155681848526,
+            -0.04228680953383446,
+            -0.014166081324219704,
+            0.01886293850839138,
+            0.02217037044465542,
+            -0.038632020354270935,
+            -0.011077634058892727,
+            0.006516699679195881,
+            -0.0002836425555869937,
+            0.0028675743378698826,
+            -0.003426364390179515,
+            -0.009831683710217476,
+            0.0036170324310660362,
+            0.034886617213487625,
+            -0.034252315759658813,
+            0.0032753399573266506,
+            0.018047407269477844,
+            -0.013199525885283947,
+            -0.037605054676532745,
+            -0.01055660005658865,
+            -0.0042098029516637325,
+            -0.012429301626980305,
+            -0.016189808025956154,
+            0.019497239962220192,
+            -0.013977301307022572,
+            0.011991331353783607,
+            -0.03361801430583,
+            -0.013305243104696274,
+            0.017820870503783226,
+            -0.03669891133904457,
+            0.009801478125154972,
+            -0.017292285338044167,
+            -0.007494581863284111,
+            0.009990259073674679,
+            0.03872263804078102,
+            -0.0028184913098812103,
+            0.015261007472872734,
+            0.000569172902032733,
+            -0.007853264920413494,
+            0.01453609112650156,
+            -0.013365652412176132,
+            -0.0004693552909884602,
+            0.005002680700272322,
+            0.031171420589089394,
+            -0.01244440395385027,
+            -0.0050215586088597775,
+            0.018153123557567596,
+            0.05237523466348648,
+            -0.031050600111484528,
+            0.007120796944946051,
+            0.014974061399698257,
+            0.028483187779784203,
+            -0.007471928372979164,
+            -0.003031813306733966,
+            -0.01955764926970005,
+            0.013267486356198788,
+            -0.01936131902039051,
+            0.0006673387251794338,
+            0.028226446360349655,
+            -0.02449614554643631,
+            -0.01902906410396099,
+            -0.0015527188079431653,
+            -0.02579495497047901,
+            0.0029411986470222473,
+            0.005127275828272104,
+            0.014468129724264145,
+            -0.030522014945745468,
+            -0.0007551216403953731,
+            0.044370945543050766,
+            0.014702217653393745,
+            -0.01147029735147953,
+            -0.008110006339848042,
+            0.0065997629426419735,
+            0.024239404127001762,
+            0.008125108666718006,
+            -0.0030771205201745033,
+            -0.010941712185740471,
+            0.0032621254213154316,
+            -0.011666628532111645,
+            0.020478898659348488,
+            0.004470319952815771,
+            0.030476707965135574,
+            0.008827371522784233,
+            -0.008351645432412624,
+            -0.004213578533381224,
+            -0.021762605756521225,
+            0.011485399678349495,
+            0.05926194414496422,
+            0.006750787142664194,
+            0.016084089875221252,
+            -0.016763700172305107,
+            0.009076561778783798,
+            0.0341314971446991,
+            -0.0019293357618153095,
+            -0.034886617213487625,
+            0.006263733841478825,
+            -0.008276132866740227,
+            -0.01889314316213131,
+            -0.012686043046414852,
+            -0.004190925043076277,
+            -0.025674134492874146,
+            -0.01329014077782631,
+            -0.007513460237532854,
+            -0.011862960644066334,
+            0.006833850406110287,
+            -0.002359755104407668,
+            -0.005278300028294325,
+            -0.029026875272393227,
+            0.024209199473261833,
+            -0.0003412205842323601,
+            0.05303974077105522,
+            0.00045637661241926253,
+            0.012210316024720669,
+            -0.012293379753828049,
+            0.0024352672044187784,
+            0.0018037967383861542,
+            0.0040323492139577866,
+            0.025734543800354004,
+            0.019134782254695892,
+            0.011794999241828918,
+            0.03684993460774422,
+            -0.012044189497828484,
+            -0.03189633786678314,
+            -0.01123620942234993,
+            -0.0020482672844082117,
+            0.03307432681322098,
+            0.021037688478827477,
+            -0.003971939906477928,
+            0.015600812621414661,
+            -0.011107838712632656,
+            -0.008812269195914268,
+            0.041018206626176834,
+            -0.008804718032479286,
+            -0.025779852643609047,
+            0.043374184519052505,
+            -0.012867271900177002,
+            0.04657590016722679,
+            0.00700375298038125,
+            -0.00858573243021965,
+            0.004292866215109825,
+            0.010133732110261917,
+            -0.003817139659076929,
+            -0.018289046362042427,
+            0.011394784785807133,
+            0.04479381442070007,
+            0.012399096973240376,
+            0.0061240363866090775,
+            0.02041848748922348,
+            0.02224588207900524,
+            -0.027622347697615623,
+            -0.023589998483657837,
+            -0.0034622326493263245,
+            -0.0025466475635766983,
+            -0.015933066606521606,
+            0.016824109479784966,
+            -0.0400516502559185,
+            -0.009393712505698204,
+            0.026942739263176918,
+            -0.0011732701677829027,
+            0.017292285338044167,
+            0.003854895941913128,
+            0.014543642289936543,
+            0.008245928213000298,
+            -0.01803230494260788,
+            0.026006387546658516,
+            0.030204864218831062,
+            0.013365652412176132,
+            -0.02519085630774498,
+            0.031080804765224457,
+            -0.0065997629426419735,
+            0.019089475274086,
+            0.01677880249917507,
+            0.005014007445424795,
+            -0.006010768003761768,
+            -0.00208035996183753,
+            -0.026127208024263382,
+            -0.006814972497522831,
+            -0.007396416272968054,
+            0.027758270502090454,
+            -0.00439103227108717,
+            0.01564612053334713,
+            0.0159783735871315,
+            -0.012708696536719799,
+            -0.0006475168047472835,
+            -0.0010156385833397508,
+            -0.031775519251823425,
+            0.025613725185394287,
+            0.007891020737588406,
+            0.006999977398663759,
+            0.03802792355418205,
+            -0.042045172303915024,
+            -0.01842496730387211,
+            -0.003711422672495246,
+            0.029147693887352943,
+            0.05134826898574829,
+            0.0031431936658918858,
+            -0.0003223425301257521,
+            0.031775519251823425,
+            -0.03479600325226784,
+            -0.032530639320611954,
+            0.011349477805197239,
+            0.03292330354452133,
+            -0.0005526546156033874,
+            -0.003741627559065819,
+            0.04379705339670181,
+            -0.010254551656544209,
+            -0.021762605756521225,
+            0.021762605756521225,
+            0.01839476265013218,
+            -0.002372969640418887,
+            -0.005648309830576181,
+            0.0003556150768417865,
+            0.014166081324219704,
+            -0.0045005246065557,
+            0.04310234263539314,
+            -0.02419409714639187,
+            0.01495140790939331,
+            -0.01603878289461136,
+            0.013501574285328388,
+            0.015049573965370655,
+            0.00010677891259547323,
+            -0.04283049702644348,
+            0.013040950521826744,
+            0.022351600229740143,
+            0.005474631674587727,
+            0.00958249345421791,
+            0.017322489991784096,
+            0.021838117390871048,
+            0.03832997381687164,
+            0.02960076741874218,
+            0.005923929158598185,
+            0.040655747056007385,
+            0.01453609112650156,
+            -0.0005847472930327058,
+            0.02784888446331024,
+            -0.0019302796572446823,
+            -0.006105158012360334,
+            -0.0011307946406304836,
+            -0.0041418420150876045,
+            0.02343897521495819,
+            -0.021883424371480942,
+            -0.015223251655697823,
+            0.02124912291765213,
+            0.0028468084055930376,
+            -0.00019196607172489166,
+            -0.03343678638339043,
+            0.006290162913501263,
+            -0.02790929563343525,
+            0.0220495518296957,
+            -0.0024560331366956234,
+            0.03364821895956993,
+            -0.023227540776133537,
+            -0.0056747389025986195,
+            0.022260984405875206,
+            -0.019482137635350227,
+            0.0033055448438972235,
+            0.01392444223165512,
+            0.008631040342152119,
+            -0.005965460557490587,
+            -0.011492950841784477,
+            -0.013569535687565804,
+            0.047089383006095886,
+            -0.005350036546587944,
+            0.018183328211307526,
+            0.018138021230697632,
+            -0.007709791883826256,
+            -0.019300907850265503,
+            0.028392571955919266,
+            -0.040716156363487244,
+            0.010662317276000977,
+            0.03201715648174286,
+            -0.06475923210382462,
+            -0.034161701798439026,
+            -0.006006992422044277,
+            -0.028996670618653297,
+            0.005489734001457691,
+            0.00033579315640963614,
+            0.008064698427915573,
+            0.010095976293087006,
+            -0.018802527338266373,
+            0.018455171957612038,
+            0.0006678106728941202,
+            0.02973668836057186,
+            -0.0031356425024569035,
+            0.0069131385535001755,
+            -0.021883424371480942,
+            -0.025930875912308693,
+            -0.01952744461596012,
+            -0.0008792447042651474,
+            -0.0036566765047609806,
+            0.00772489421069622,
+            -0.02741091512143612,
+            -0.015721632167696953,
+            0.01936131902039051,
+            -0.009861888363957405,
+            0.04026308283209801,
+            0.0405651330947876,
+            0.022381804883480072,
+            0.005980563350021839,
+            -0.025009628385305405,
+            0.03923611715435982,
+            -0.042014967650175095,
+            0.022019345313310623,
+            -0.0015904748579487205,
+            -0.017262080684304237,
+            -0.01252746768295765,
+            0.031111011281609535,
+            -0.007845713756978512,
+            0.005504836328327656,
+            0.0011883726110681891,
+            0.021234020590782166,
+            -0.02294059470295906,
+            -0.0030922230798751116,
+            -0.00885002501308918,
+            -0.011900716461241245,
+            0.015359173528850079,
+            0.0003539632598403841,
+            0.01958785392343998,
+            0.02549290470778942,
+            0.006343021523207426,
+            0.0027448670007288456,
+            -0.032107770442962646,
+            -0.031201625242829323,
+            0.026353744789958,
+            0.012308482080698013,
+            -0.028452983126044273,
+            0.013146666809916496,
+            -0.045428115874528885,
+            -0.013652598485350609,
+            -0.011213555932044983,
+            0.0075059086084365845,
+            0.013199525885283947,
+            -0.002378633012995124,
+            -0.0008320496417582035,
+            -0.02777337282896042,
+            -5.3418953029904515e-05,
+            -0.05001925677061081,
+            -0.017594333738088608,
+            0.015276109799742699,
+            -0.01102477591484785,
+            0.01866660639643669,
+            0.005478407256305218,
+            0.0107378289103508,
+            -0.01919519156217575,
+            -0.012187662534415722,
+            0.004666651599109173,
+            -0.030869372189044952,
+            0.01581224612891674,
+            -0.018440069630742073,
+            0.013433613814413548,
+            -0.013033399358391762,
+            0.03981001302599907,
+            -0.009174727834761143,
+            0.016748597845435143,
+            -0.025613725185394287,
+            -0.029887713491916656,
+            -0.02565903216600418,
+            0.02294059470295906,
+            -0.004485422279685736,
+            -0.018787425011396408,
+            0.012029087170958519,
+            -0.007875918410718441,
+            0.011719487607479095,
+            -0.02214016579091549,
+            -0.00916717667132616,
+            0.023287950083613396,
+            -0.026263128966093063,
+            0.0023842963855713606,
+            -0.04902249574661255,
+            -0.03709157183766365,
+            -0.00450807623565197,
+            0.02114340476691723,
+            -0.019300907850265503,
+            0.030688142403960228,
+            -0.0005021558608859777,
+            -0.004379705525934696,
+            -0.02399776503443718,
+            -0.028694620355963707,
+            0.01112294103950262,
+            -0.0002989809727296233,
+            -0.02880033850669861,
+            0.05781210958957672,
+            0.025009628385305405,
+            0.04008185490965843,
+            -0.04482401907444,
+            0.026867227628827095,
+            0.0066261920146644115,
+            -0.029283616691827774,
+            0.01291257981210947,
+            0.01079068798571825,
+            0.016401240602135658,
+            -0.009484327398240566,
+            -0.007468152791261673,
+            0.0034414667170494795,
+            -0.009371059015393257,
+            0.007687137927860022,
+            0.005297177936881781,
+            -0.03222858905792236,
+            0.005784231703728437,
+            0.025704339146614075,
+            -0.001376209082081914,
+            0.04014226421713829,
+            0.007196309044957161,
+            0.011568463407456875,
+            -0.004674202762544155,
+            -0.04011205956339836,
+            0.01730738766491413,
+            -0.027622347697615623,
+            -0.03277227655053139,
+            0.03887366130948067,
+            0.020992381498217583,
+            0.011711936444044113,
+            -0.02416389249265194,
+            0.00416071992367506,
+            0.014641808345913887,
+            -0.016431447118520737,
+            0.0081704156473279,
+            0.006524250842630863,
+            0.027863986790180206,
+            0.014709768816828728,
+            -0.007860816083848476,
+            0.00044740954763256013,
+            0.02303120866417885,
+            -0.01564612053334713,
+            0.0028373694512993097,
+            0.0020407161209732294,
+            0.02813583053648472,
+            0.010330063290894032,
+            -0.024586759507656097,
+            -0.017986996099352837,
+            -0.017549026757478714,
+            -0.020040927454829216,
+            0.0009608922409825027,
+            -0.013713008724153042,
+            -0.02177770808339119,
+            -0.04358562082052231,
+            -0.004798797890543938,
+            -0.019497239962220192,
+            -0.03138285502791405,
+            -0.015026920475065708,
+            -0.008125108666718006,
+            -0.02821134403347969,
+            0.011198453605175018,
+            0.001511187176220119,
+            -0.02038828283548355,
+            0.002797725610435009,
+            -0.004813900217413902,
+            0.050140075385570526,
+            -0.019542546942830086,
+            0.013259935192763805,
+            -0.0038643348962068558,
+            -0.006052299868315458,
+            -0.00875185988843441,
+            0.03168490156531334,
+            -0.026610486209392548,
+            -0.04977761581540108,
+            0.026625588536262512,
+            -0.022683853283524513,
+            0.000796181324403733,
+            0.04497504234313965,
+            -0.024254506453871727,
+            -0.01753392443060875,
+            0.02005602978169918,
+            -0.020297668874263763,
+            -0.032440025359392166,
+            0.003915305715054274,
+            -0.010647214949131012,
+            -0.016068987548351288,
+            -0.02704845555126667,
+            -0.010118629783391953,
+            0.052163802087306976,
+            0.011092736385762691,
+            0.027335401624441147,
+            0.03364821895956993,
+            0.00480634905397892,
+            -0.016552265733480453,
+            0.010881302878260612,
+            -0.015540402382612228,
+            -0.026202719658613205,
+            0.010095976293087006,
+            0.02562882751226425,
+            0.01739800162613392,
+            0.007317128591239452,
+            -0.030310582369565964,
+            -0.005980563350021839,
+            -0.03204736113548279,
+            -0.0051763588562607765,
+            0.019044166430830956,
+            -0.003532081376761198,
+            -0.031956747174263,
+            -0.011115389876067638,
+            0.025840261951088905,
+            0.021928731352090836,
+            -0.018606197088956833,
+            -0.016748597845435143,
+            -0.0028807888738811016,
+            0.0007291642832569778,
+            0.016914723441004753,
+            -0.01327503751963377,
+            -0.00476481718942523,
+            0.004330622497946024,
+            0.019814390689134598,
+            -0.013916891068220139,
+            0.0013129677390679717,
+            -0.05950358137488365,
+            -0.01620491035282612,
+            0.0211132001131773,
+            0.002933647483587265,
+            -0.0017735918518155813,
+            -0.010088425129652023,
+            0.007668260019272566,
+            -0.0031054376158863306,
+            -0.030008532106876373,
+            0.010201692581176758,
+            0.0023238868452608585,
+            0.02214016579091549,
+            -0.02602148987352848,
+            0.008842473849654198,
+            0.01686941646039486,
+            0.0015206261305138469,
+            -0.013546881265938282,
+            -0.0036113690584897995,
+            0.0343429297208786,
+            0.010465985164046288,
+            0.003277227748185396,
+            0.004440115299075842,
+            0.014543642289936543,
+            0.004224905278533697,
+            -0.0034320277627557516,
+            0.002644813386723399,
+            -0.0012959775049239397,
+            -0.006856504362076521,
+            0.004021022468805313,
+            0.006724358070641756,
+            0.02363530732691288,
+            0.024058174341917038,
+            0.005059314891695976,
+            0.0008070362382568419,
+            -0.00976372230798006,
+            -0.0057426998391747475,
+            0.01700533926486969,
+            -0.012557672336697578,
+            0.015827348455786705,
+            0.011168248951435089,
+            -0.02072053775191307,
+            -0.023892046883702278,
+            0.0018859162228181958,
+            0.001278043375350535,
+            0.01830414868891239,
+            0.02217037044465542,
+            -0.00047100710798986256,
+            0.0005309449043124914,
+            -0.017443308606743813,
+            0.00964290276169777,
+            0.007664484437555075,
+            0.03597399219870567,
+            -0.0365176796913147,
+            -0.01480038370937109,
+            0.021823015064001083,
+            -0.009673107415437698,
+            0.020342975854873657,
+            0.0003593906876631081,
+            0.006977323908358812,
+            0.0004969644360244274,
+            0.06935036927461624,
+            -0.017986996099352837,
+            -0.00620709965005517,
+            -0.029872611165046692,
+            0.009605146944522858,
+            -0.0056332070380449295,
+            -0.02052420563995838,
+            0.011198453605175018,
+            0.0328930988907814,
+            -0.015283661894500256,
+            0.042347218841314316,
+            -0.0018575991271063685,
+            0.009303098544478416,
+            -0.005417997483164072,
+            -0.019738879054784775,
+            0.013305243104696274,
+            -0.0014536090893670917,
+            0.004942270927131176,
+            0.0032847789116203785,
+            0.012436852790415287,
+            0.02069033309817314,
+            0.01020924374461174,
+            0.014498335309326649,
+            0.004172047134488821,
+            -0.02008623443543911,
+            0.01978418603539467,
+            -0.0019897455349564552,
+            0.000477614434203133,
+            -0.015993475914001465,
+            -0.013335447758436203,
+            0.0067092557437717915,
+            0.01155336108058691,
+            -0.0033546278718858957,
+            -0.0017207333585247397,
+            0.003413149621337652,
+            0.025070037692785263,
+            -0.016416342929005623,
+            0.014694666489958763,
+            -0.012346237897872925,
+            -0.012308482080698013,
+            -0.012640736065804958,
+            0.005251870956271887,
+            0.005417997483164072,
+            -0.0055388170294463634,
+            0.004927168600261211,
+            -0.006184446159750223,
+            0.011953575536608696,
+            -0.03739362210035324,
+            0.014241593889892101,
+            -0.006633743178099394,
+            -0.003290442517027259,
+            0.010201692581176758,
+            -0.012867271900177002,
+            0.007517235819250345,
+            -0.007403967436403036,
+            0.01653716340661049,
+            0.006977323908358812,
+            -0.007528562564402819,
+            0.0017282846383750439,
+            -0.014853241853415966,
+            0.004519402980804443,
+            0.014792832545936108,
+            -0.0018085162155330181,
+            0.01495140790939331,
+            -0.01972377672791481,
+            -0.01474752463400364,
+            0.0006059851148165762,
+            -0.01853068359196186,
+            0.002325774636119604,
+            0.00454205647110939,
+            -0.009710864163935184,
+            0.023952458053827286,
+            -0.0034565692767500877,
+            -0.01026210281997919,
+            -0.021928731352090836,
+            -0.015495095402002335,
+            -0.015661222860217094,
+            -0.020629921928048134,
+            0.006093831267207861,
+            -0.03307432681322098,
+            0.022623443976044655,
+            0.016235115006566048,
+            -0.01733759231865406,
+            -0.01972377672791481,
+            -0.032440025359392166,
+            0.0030676815658807755,
+            -0.006093831267207861,
+            -0.005455753766000271,
+            0.012097048573195934,
+            -0.004153168760240078,
+            0.011893165297806263,
+            0.0031299791298806667,
+            -0.010956814512610435,
+            0.011485399678349495,
+            -0.012285828590393066,
+            -0.006494045723229647,
+            -0.0018009650520980358,
+            -0.003647237317636609,
+            -0.015042022801935673,
+            -0.023741023615002632,
+            -0.03769567236304283,
+            0.020071132108569145,
+            0.013426062650978565,
+            -0.014422822743654251,
+            -0.01529876422137022,
+            -0.002776959678158164,
+            0.01223297044634819,
+            -0.013108910992741585,
+            -0.01264828722923994,
+            0.008940639905631542,
+            -0.008464912883937359,
+            0.01461915485560894,
+            -0.0014413383323699236,
+            -0.0034018228761851788,
+            0.023710818961262703,
+            0.00937861017882824,
+            0.005440651439130306,
+            -0.02479819394648075,
+            -0.012240521609783173,
+            0.01370545756071806,
+            -0.013931993395090103,
+            -0.0017226211493834853,
+            0.0219438336789608,
+            -0.00154139194637537,
+            0.000530472956597805,
+            0.011145595461130142,
+            0.0019265039591118693,
+            -0.02058461494743824,
+            0.0054028951562941074,
+            0.008283684030175209,
+            -0.009854337200522423,
+            0.046092621982097626,
+            0.014098120853304863,
+            0.01603878289461136,
+            0.0025655257049947977,
+            -0.006184446159750223,
+            0.013342998921871185,
+            0.010669868439435959,
+            0.02787908911705017,
+            -0.015094880945980549,
+            -0.007366211153566837,
+            0.013388305902481079,
+            0.04080677032470703,
+            0.00013851761468686163,
+            0.011409887112677097,
+            -0.027592143043875694,
+            -0.02164178527891636,
+            0.028815440833568573,
+            0.004787471145391464,
+            0.01302584819495678,
+            -0.011138043366372585,
+            -0.024012867361307144,
+            0.015079778619110584,
+            -0.018545785918831825,
+            -0.015011817216873169,
+            -0.028558699414134026,
+            0.009627800434827805,
+            -0.027471324428915977,
+            0.02840767428278923,
+            0.0069810994900763035,
+            -0.0006479887524619699,
+            0.02519085630774498,
+            0.000645628955680877,
+            -0.025311676785349846,
+            0.014860793016850948,
+            -0.013607291504740715,
+            0.006562006659805775,
+            0.016688188537955284,
+            -0.03092978149652481,
+            0.0014130213530734181,
+            -0.006569557823240757,
+            -0.004107861779630184,
+            -0.0016848650993779302,
+            0.02436022274196148,
+            -0.016612675040960312,
+            -0.02599128521978855,
+            -0.005682290066033602,
+            -0.006331694778054953,
+            0.002212506253272295,
+            -0.007747547701001167,
+            0.022215677425265312,
+            0.005950358230620623,
+            0.003154520643875003,
+            -0.01789638213813305,
+            -0.009287996217608452,
+            0.020297668874263763,
+            -0.020750742405653,
+            0.0014715432189404964,
+            -0.021958936005830765,
+            0.010133732110261917,
+            0.006629967596381903,
+            0.010186590254306793,
+            0.014762627892196178,
+            6.300545646809042e-05,
+            0.003545295912772417,
+            0.0007820228347554803,
+            0.0026183840818703175,
+            0.013063604012131691,
+            0.01524590514600277,
+            -0.0446125864982605,
+            -0.02257813699543476,
+            0.028679518029093742,
+            -0.016461651772260666,
+            0.00308278389275074,
+            0.008563078939914703,
+            -0.03343678638339043,
+            -0.011779896914958954,
+            -0.005614329129457474,
+            0.035520922392606735,
+            -0.01614449918270111,
+            0.0015149627579376101,
+            -0.001464935950934887,
+            -0.024224301800131798,
+            0.0026882330421358347,
+            -0.020131541416049004,
+            0.008842473849654198,
+            -0.01830414868891239,
+            -0.043434593826532364,
+            -0.0008244984201155603,
+            0.02535698376595974,
+            -0.007449274882674217,
+            0.003824691055342555,
+            0.021354839205741882,
+            -0.018138021230697632,
+            0.003739739768207073,
+            -0.019089475274086,
+            0.013780969195067883,
+            0.003877549432218075,
+            0.028196241706609726,
+            0.008736756630241871,
+            0.004685529507696629,
+            0.006380777806043625,
+            -0.00368688115850091,
+            0.030476707965135574,
+            -0.009227585978806019,
+            -0.012036638334393501,
+            0.021022586151957512,
+            0.021520966663956642,
+            0.02127932757139206,
+            0.002335213590413332,
+            0.010835994966328144,
+            0.035520922392606735,
+            -0.005210339091718197,
+            0.019874799996614456,
+            0.007815509103238583,
+            -6.365439185174182e-05,
+            0.008940639905631542,
+            -0.007698464673012495,
+            -0.03443354368209839,
+            -0.0022653648629784584,
+            -0.022865083068609238,
+            -0.01697513461112976,
+            0.011772345751523972,
+            -0.023454077541828156,
+            -0.02217037044465542,
+            -0.028679518029093742,
+            0.013660149648785591,
+            0.004783695563673973,
+            0.004749714862555265,
+            -0.021234020590782166,
+            0.02777337282896042,
+            0.024647168815135956,
+            0.015585710294544697,
+            0.017715152353048325,
+            0.012754004448652267,
+            0.030386094003915787,
+            0.008336542174220085,
+            0.02641415409743786,
+            0.0065997629426419735,
+            0.012406648136675358,
+            -0.007747547701001167,
+            0.031956747174263,
+            0.017594333738088608,
+            0.006267509423196316,
+            0.007509684655815363,
+            -0.01733759231865406,
+            -0.0032338083256036043,
+            0.0007013191934674978,
+            -0.020478898659348488,
+            -0.011039878241717815,
+            -0.006788543425500393,
+            0.032802481204271317,
+            0.02200424298644066,
+            -0.016325728967785835,
+            -0.014211388304829597,
+            -0.005191461183130741,
+            0.0060560754500329494,
+            -0.023046310991048813,
+            -0.007543664891272783,
+            -0.012285828590393066,
+            -0.0023238868452608585,
+            -0.009340854361653328,
+            0.02343897521495819,
+            0.021626682952046394,
+            0.010760483331978321,
+            0.01833435334265232,
+            0.001749050454236567,
+            0.003915305715054274,
+            -0.010760483331978321,
+            -0.007317128591239452,
+            0.011379682458937168,
+            -0.012421750463545322,
+            -0.025341881439089775,
+            0.008087352849543095,
+            -0.006256182678043842,
+            -0.020267464220523834,
+            0.009567391127347946,
+            0.018379660323262215,
+            0.001985969953238964,
+            0.01252746768295765,
+            0.017216771841049194,
+            0.0014998603146523237,
+            0.0065997629426419735,
+            0.02214016579091549,
+            -0.0028298182878643274,
+            -0.006543128751218319,
+            0.004236232023686171,
+            -0.007981635630130768,
+            0.02230629324913025,
+            0.016461651772260666,
+            -0.03207756578922272,
+            -0.01529876422137022,
+            -0.010798239149153233,
+            0.02960076741874218,
+            -0.0070754894986748695,
+            0.008653693832457066,
+            -0.03265145793557167,
+            -0.006267509423196316,
+            -0.026308437809348106,
+            0.02002582512795925,
+            -0.003362179035320878,
+            -0.006331694778054953,
+            0.004734612535685301,
+            -0.01578204147517681,
+            0.017413103953003883,
+            -0.003756730118766427,
+            -0.009287996217608452,
+            0.012429301626980305,
+            -0.004991353955119848,
+            0.01617470569908619,
+            -0.007441723719239235,
+            0.004323071334511042,
+            0.015525300055742264,
+            0.00722651369869709,
+            0.004723285790532827,
+            -0.03201715648174286,
+            -0.0019878577440977097,
+            -0.014528539963066578,
+            0.0062863873317837715,
+            -0.006701704114675522,
+            0.0009490934899076819,
+            0.024178994819521904,
+            0.027728065848350525,
+            -0.010133732110261917,
+            -0.008011840283870697,
+            0.029419537633657455,
+            0.004070105496793985,
+            0.012029087170958519,
+            -0.009333303198218346,
+            -0.013584638014435768,
+            -0.0012251847656443715,
+            -0.012504814192652702,
+            -0.028890952467918396,
+            -0.014437925070524216,
+            -0.027108866721391678,
+            0.016718393191695213,
+            -0.009310649707913399,
+            -0.02019195258617401,
+            0.0328930988907814,
+            -0.041712917387485504,
+            -0.008412054739892483,
+            -0.00329421809874475,
+            0.0015347846783697605,
+            0.012587876990437508,
+            0.003345188684761524,
+            0.0028260427061468363,
+            -0.028951361775398254,
+            -0.013984852470457554,
+            -0.007026406470686197,
+            0.004081432241946459,
+            -0.000620143604464829,
+            -0.02496432140469551,
+            0.007256718818098307,
+            -0.0038058129139244556,
+            -0.004273988306522369,
+            0.010314960964024067,
+            -0.00484788091853261,
+            -0.024481043219566345,
+            0.02602148987352848,
+            -0.007120796944946051,
+            0.02052420563995838,
+            0.010171487927436829,
+            0.016914723441004753,
+            0.010730277746915817,
+            0.03023506887257099,
+            -0.0007919337949715555,
+            -0.002229496603831649,
+            -0.004357051569968462,
+            -0.009703313000500202,
+            0.01165152620524168,
+            0.014279349707067013,
+            -0.01128906849771738,
+            -0.03706136718392372,
+            0.006018319167196751,
+            -0.02343897521495819,
+            0.019738879054784775,
+            0.0010118630016222596,
+            -0.02999342978000641,
+            -0.007037733215838671,
+            -0.0422264002263546,
+            0.02386184222996235,
+            -0.005300953518599272,
+            -0.016386138275265694,
+            0.017050646245479584,
+            -0.006577109452337027,
+            -0.005557694938033819,
+            0.027199480682611465,
+            0.005131051409989595,
+            -0.030084045603871346,
+            0.013743213377892971,
+            0.010156385600566864,
+            -0.012104599736630917,
+            0.0018141797045245767,
+            -0.0006838570116087794,
+            -0.034252315759658813,
+            0.0071170213632285595,
+            0.0008518715621903539,
+            -0.0034660082310438156,
+            -0.02058461494743824,
+            -0.019633162766695023,
+            -0.00881982035934925,
+            0.018379660323262215,
+            0.018349455669522285,
+            0.01836455799639225,
+            -0.007762650027871132,
+            0.00036127850762568414,
+            0.006799870170652866,
+            0.005331158638000488,
+            0.010911507532000542,
+            0.005501060746610165,
+            -0.00045118515845388174,
+            0.007928776554763317,
+            -0.014150978997349739,
+            0.017413103953003883,
+            -0.020901765674352646,
+            -0.003990817815065384,
+            -0.00019208405865356326,
+            -0.008555527776479721,
+            0.00019845539645757526,
+            -0.016809007152915,
+            0.011681731790304184,
+            0.016461651772260666,
+            -0.006822523660957813,
+            -0.009491878561675549,
+            -0.020705435425043106,
+            0.009997810237109661,
+            0.029525255784392357,
+            0.01020924374461174,
+            -0.041320253163576126,
+            0.021898526698350906,
+            -0.01739800162613392,
+            0.0018160674953833222,
+            -0.017941689118742943,
+            0.025961080566048622,
+            0.014332207851111889,
+            0.0035736130084842443,
+            0.018983757123351097,
+            -0.0049989051185548306,
+            0.0003516034921631217,
+            -0.00704528484493494,
+            0.001961428439244628,
+            0.00045614063856191933,
+            -0.009522083215415478,
+            -0.0040738810785114765,
+            0.02704845555126667,
+            -0.02416389249265194,
+            -0.004334398079663515,
+            0.001532896887511015,
+            0.023121824488043785,
+            -0.008261030539870262,
+            0.012535018846392632,
+            0.017911484465003014,
+            0.018017200753092766,
+            -0.002399398945271969,
+            0.011447643861174583,
+            0.07291454076766968,
+            -0.019089475274086,
+            0.017624538391828537,
+            0.019331112504005432,
+            -0.008245928213000298,
+            -0.01637103594839573,
+            -0.010851097293198109,
+            -0.01988990418612957,
+            0.02144545316696167,
+            -0.0025711890775710344,
+            0.01445302739739418,
+            -0.011296619661152363,
+            -0.009257790632545948,
+            0.007539889309555292,
+            0.0037869347725063562,
+            0.0015168505487963557,
+            0.005285851191729307,
+            0.011742141097784042,
+            0.023876944556832314,
+            0.010118629783391953,
+            -0.005165031645447016,
+            0.0012619970366358757,
+            -0.0011996994726359844,
+            -0.0023805208038538694,
+            -0.01184785831719637,
+            -0.0013554432662203908,
+            0.009386161342263222,
+            0.02340877056121826,
+            0.0019878577440977097,
+            0.035218872129917145,
+            -0.015011817216873169,
+            -0.009121868759393692,
+            0.00916717667132616,
+            0.0006692265160381794,
+            0.004606241825968027,
+            0.014724871143698692,
+            0.012837067246437073,
+            -0.011772345751523972,
+            -0.006176894996315241,
+            0.0076154014095664024,
+            0.0052934023551642895,
+            0.006384553387761116,
+            0.0107378289103508,
+            -0.01112294103950262,
+            -0.016960032284259796,
+            0.03709157183766365,
+            0.009884541854262352,
+            0.0023238868452608585,
+            0.0049309441819787025,
+            0.0009769386379048228,
+            -0.011062531732022762,
+            0.004832778126001358,
+            0.004443890880793333,
+            0.005327383056282997,
+            -0.000515842461027205,
+            -0.0009325752034783363,
+            0.003205491229891777,
+            -0.0056332070380449295,
+            0.0014847578713670373,
+            0.008049596101045609,
+            0.005319831892848015,
+            0.010911507532000542,
+            -0.008396952413022518,
+            -0.019210293889045715,
+            0.0010005361400544643,
+            -0.016657982021570206,
+            0.006679050624370575,
+            0.002169086830690503,
+            -0.0063279191963374615,
+            0.007536113727837801,
+            0.009189830161631107,
+            -0.0073473332449793816,
+            0.014332207851111889,
+            -0.001468711532652378,
+            0.001872701570391655,
+            0.0107378289103508,
+            -0.020327873528003693,
+            -0.007672035600990057,
+            0.0024767988361418247,
+            0.006894260179251432,
+            0.003726525232195854,
+            0.0018321138340979815,
+            -0.008615938015282154,
+            0.014936305582523346,
+            -0.023544691503047943,
+            -0.00908411294221878,
+            -0.0032375839073210955,
+            0.026595383882522583,
+            0.013071155175566673,
+            -0.013003193773329258,
+            0.004406134597957134,
+            0.00012789871834684163,
+            0.0066941529512405396,
+            0.0038058129139244556,
+            -0.004383481107652187,
+            0.02735050395131111,
+            -0.003715198254212737,
+            -0.024420632049441338,
+            -0.008238377049565315,
+            -0.002325774636119604,
+            0.009257790632545948,
+            0.023982662707567215,
+            0.0054028951562941074,
+            0.006497821304947138,
+            0.006988650653511286,
+            0.00502910977229476,
+            0.009892093017697334,
+            0.013629944995045662,
+            -0.003990817815065384,
+            -0.008646142669022083,
+            -0.004002144560217857,
+            -0.011402335949242115,
+            0.015283661894500256,
+            0.016386138275265694,
+            -0.012701145373284817,
+            -0.017881279811263084,
+            -0.0073020257987082005,
+            -0.01456629578024149,
+            -0.0075059086084365845,
+            0.003930408041924238,
+            -0.006048524286597967,
+            0.0011279629543423653,
+            -0.0009250239818356931,
+            -0.009597595781087875,
+            0.01324483286589384,
+            -0.012618082575500011,
+            -0.008087352849543095,
+            -0.008547976613044739,
+            -0.034947026520967484,
+            -0.024375325068831444,
+            0.005727597512304783,
+            -0.014966510236263275,
+            -0.001957652857527137,
+            0.014958959072828293,
+            -0.007358659990131855,
+            0.003715198254212737,
+            0.017624538391828537,
+            0.004987578373402357,
+            0.0060560754500329494,
+            -0.005006456281989813,
+            -0.0038114762865006924,
+            -0.010186590254306793,
+            -0.011938472278416157,
+            -0.01464935950934887,
+            0.0040059201419353485,
+            0.0012421749997884035,
+            0.011228658258914948,
+            -0.01544978842139244,
+            -0.0020444917026907206,
+            -0.003060130402445793,
+            0.019482137635350227,
+            -0.028845645487308502,
+            -0.019497239962220192,
+            0.008706551976501942,
+            -0.0004950765869580209,
+            0.01047353632748127,
+            -0.021369941532611847,
+            -0.0029732913244515657,
+            0.0017046870198100805,
+            -0.008691449649631977,
+            -0.019799288362264633,
+            0.02291039004921913,
+            -0.015125085599720478,
+            0.004534505307674408,
+            -0.008396952413022518,
+            0.027592143043875694,
+            0.012693594209849834,
+            0.013894237577915192,
+            0.0064902701415121555,
+            0.00590127520263195,
+            -0.01647675409913063,
+            -0.0026693549007177353,
+            0.0017905820859596133,
+            -0.006528026424348354,
+            0.0015810359036549926,
+            0.0075059086084365845,
+            -0.002389959990978241,
+            -0.007441723719239235,
+            0.018122918903827667,
+            0.01524590514600277,
+            0.005444427020847797,
+            0.020297668874263763,
+            0.0011619434226304293,
+            0.002886452479287982,
+            -0.00011043653648812324,
+            0.0027656329330056906,
+            0.003494325326755643,
+            0.010919058695435524,
+            0.004489197861403227,
+            -0.019874799996614456,
+            0.0013894237345084548,
+            -0.002399398945271969,
+            -0.007604074664413929,
+            -0.026988046243786812,
+            0.00120158726349473,
+            0.007709791883826256,
+            -0.001681089517660439,
+            -0.01471731998026371,
+            -0.015004266053438187,
+            0.0054972851648926735,
+            -0.001405470073223114,
+            0.013848930597305298,
+            -0.02105279080569744,
+            0.001889691804535687,
+            0.0013582749525085092,
+            -0.01925560086965561,
+            -0.005670963320881128,
+            0.02191362902522087,
+            0.019572751596570015,
+            -0.006550679914653301,
+            -0.03953816741704941,
+            -0.005980563350021839,
+            0.01301074493676424,
+            -0.0058370898477733135,
+            -0.01753392443060875,
+            0.006509148515760899,
+            -0.009053908288478851,
+            -0.006101382430642843,
+            0.018243737518787384,
+            0.0010448995744809508,
+            -0.01544978842139244,
+            0.006188221741467714,
+            0.008283684030175209,
+            0.02399776503443718,
+            0.004685529507696629,
+            -0.009439020417630672,
+            -0.005391568411141634,
+            -0.022200575098395348,
+            0.008427157066762447,
+            0.00046392783406190574,
+            -0.026625588536262512,
+            -0.002437154995277524,
+            -0.004662876017391682,
+            0.03165469691157341,
+            0.01040557585656643,
+            0.021505864337086678,
+            0.011092736385762691,
+            -0.018742118030786514,
+            -0.008593283593654633,
+            -0.008480016142129898,
+            -0.012980540283024311,
+            -0.0034225888084620237,
+            -0.0016697626560926437,
+            0.008615938015282154,
+            0.01581224612891674,
+            0.028030114248394966,
+            0.00020706850045826286,
+            0.011742141097784042,
+            0.01468711532652378,
+            -2.842029971361626e-05,
+            0.011636423878371716,
+            0.001077936147339642,
+            -0.016235115006566048,
+            0.00047147905570454895,
+            0.00476481718942523,
+            -0.008985946886241436,
+            -0.006584660615772009,
+            0.006203324068337679,
+            0.018183328211307526,
+            -0.006116485223174095,
+            -0.005081968382000923,
+            -0.010012912563979626,
+            -0.00026452852762304246,
+            -0.012225419282913208,
+            0.04503545165061951,
+            0.009673107415437698,
+            -0.033497195690870285,
+            -0.001490421243943274,
+            -0.021536068990826607,
+            0.01856089010834694,
+            -0.0037982617504894733,
+            -0.004036124795675278,
+            0.007018855307251215,
+            0.02701825089752674,
+            0.025568418204784393,
+            -0.009046357125043869,
+            -0.011168248951435089,
+            0.0007848545210435987,
+            0.011062531732022762,
+            -0.0004979083314538002,
+            -0.008706551976501942,
+            -0.0012431190116330981,
+            0.0028732377104461193,
+            -0.018711913377046585,
+            0.0030808961018919945,
+            0.008155313320457935,
+            0.00044363393681123853,
+            0.0074379476718604565,
+            -0.014551193453371525,
+            0.026172515004873276,
+            -0.0049309441819787025,
+            -0.01385648176074028,
+            -0.03772587701678276,
+            -0.01995031349360943,
+            0.020237259566783905,
+            0.0068716066889464855,
+            0.002169086830690503,
+            -0.015389378182590008,
+            -0.02011643908917904,
+            0.020433591678738594,
+            -0.011183351278305054,
+            -0.0056520854122936726,
+            -0.010934161022305489,
+            -0.010851097293198109,
+            0.0214756578207016,
+            0.007430396508425474,
+            -0.007875918410718441,
+            -0.012821964919567108,
+            0.012383994646370411,
+            -0.0005762521759606898,
+            0.006826299242675304,
+            0.02369571663439274,
+            0.026761509478092194,
+            0.01819843053817749,
+            -0.009899644181132317,
+            -0.010579253546893597,
+            -0.020509103313088417,
+            -0.003562286263331771,
+            -0.022593239322304726,
+            0.007430396508425474,
+            0.0016357821878045797,
+            0.01427179854363203,
+            0.00722651369869709,
+            -0.0016631553880870342,
+            -0.005603002384305,
+            0.009506980888545513,
+            0.0070490604266524315,
+            -0.012980540283024311,
+            0.018047407269477844,
+            -0.01108518522232771,
+            0.018772322684526443,
+            -0.0030582426115870476,
+            -0.013494023121893406,
+            0.019316010177135468,
+            -0.013750764541327953,
+            0.004409910179674625,
+            -0.004039900843054056,
+            0.008412054739892483,
+            0.00442501250654459,
+            -0.010133732110261917,
+            0.016461651772260666,
+            -0.016612675040960312,
+            0.0002716078015509993,
+            -0.010919058695435524,
+            -0.00476481718942523,
+            -0.014558744616806507,
+            -0.022094858810305595,
+            -0.044672995805740356,
+            -0.005297177936881781,
+            0.019965415820479393,
+            0.0008900996181182563,
+            0.01044333167374134,
+            -0.013826277107000351,
+            -0.0006555399741046131,
+            -0.005950358230620623,
+            -0.008525323122739792,
+            0.012383994646370411,
+            0.01842496730387211,
+            -0.009831683710217476,
+            -0.003926632460206747,
+            0.021264225244522095,
+            0.0021917405538260937,
+            -0.019965415820479393,
+            -0.004651549272239208,
+            -0.006562006659805775,
+            0.0031979400664567947,
+            0.014815486036241055,
+            -0.005376466084271669,
+            -0.023574896156787872,
+            0.0013894237345084548,
+            0.006879157852381468,
+            -0.02724478766322136,
+            -0.0013903676299378276,
+            -0.01720166951417923,
+            -0.01667308434844017,
+            0.006407206878066063,
+            -0.028679518029093742,
+            0.009809029288589954,
+            -0.022955697029829025,
+            -0.010095976293087006,
+            -0.005051763728260994,
+            -0.020796049386262894,
+            0.004610017407685518,
+            -0.008744308724999428,
+            0.031805723905563354,
+            -0.001901962561532855,
+            -0.008691449649631977,
+            0.01158356573432684,
+            0.01753392443060875,
+            0.02200424298644066,
+            0.0051574804820120335,
+            0.025583520531654358,
+            -0.004179598297923803,
+            -0.014974061399698257,
+            -0.0027561939787119627,
+            0.01611429452896118,
+            0.002474911045283079,
+            0.024919012561440468,
+            -0.0158726554363966,
+            0.00037850471562705934,
+            0.00934840552508831,
+            0.01670329086482525,
+            0.0053689144551754,
+            -0.0031733985524624586,
+            0.018409864977002144,
+            0.0007839106256142259,
+            -0.003658564295619726,
+            -0.02771296352148056,
+            -0.0016933601582422853,
+            0.0002349136193515733,
+            0.003620808245614171,
+            0.01032251212745905,
+            -0.0015857553808018565,
+            0.006399655714631081,
+            0.016688188537955284,
+            0.0010722726583480835,
+            -0.007690913509577513,
+            -0.01620491035282612,
+            0.004934719763696194,
+            -0.01955764926970005,
+            0.014702217653393745,
+            -0.004738388117402792,
+            0.015691427513957024,
+            0.002267252653837204,
+            -0.038632020354270935,
+            0.032107770442962646,
+            0.004394807852804661,
+            -0.008842473849654198,
+            0.019814390689134598,
+            -0.02990281581878662,
+            -0.018213532865047455,
+            0.0011175799882039428,
+            0.0022559259086847305,
+            0.018319251015782356,
+            -0.011772345751523972,
+            0.0018283382523804903,
+            0.003532081376761198,
+            -0.0037472909316420555,
+            -0.0039379592053592205,
+            0.009053908288478851,
+            -0.025115344673395157,
+            0.00822327472269535,
+            0.018983757123351097,
+            -0.001961428439244628,
+            0.003881325013935566,
+            0.010858648456633091,
+            -0.0067356848157942295,
+            0.0009245520341210067,
+            0.007690913509577513,
+            0.024647168815135956,
+            -0.005316056311130524,
+            0.017262080684304237,
+            -0.03159428760409355,
+            0.004145617596805096,
+            -0.016235115006566048,
+            -0.017277183011174202,
+            -0.015155291184782982,
+            0.001227072672918439,
+            0.01650695875287056,
+            0.008042044937610626,
+            -0.014369964599609375,
+            -0.00018417887622490525,
+            0.0029147693421691656,
+            0.0049724760465323925,
+            -0.0012723798863589764,
+            0.021203814074397087,
+            1.1548051588761155e-05,
+            0.024420632049441338,
+            -0.00772489421069622,
+            -0.027531733736395836,
+            -0.012089497409760952,
+            0.017881279811263084,
+            -0.00651292409747839,
+            0.010533946566283703,
+            0.01495140790939331,
+            -0.00295441341586411,
+            -0.011070082895457745,
+            -0.002305008703842759,
+            0.018681708723306656,
+            0.00208035996183753,
+            0.0003353211795911193,
+            0.009673107415437698,
+            0.009476776234805584,
+            -0.0017669845838099718,
+            0.022230779752135277,
+            -0.00790612306445837,
+            -0.018107816576957703,
+            0.001507411478087306,
+            0.009907195344567299,
+            0.01988990418612957,
+            -0.0159783735871315,
+            -0.013984852470457554,
+            0.003769944654777646,
+            0.01241419930011034,
+            -0.0033433008939027786,
+            -0.01773025467991829,
+            0.0017169577768072486,
+            0.028785236179828644,
+            0.00374540314078331,
+            -0.0084045035764575,
+            -0.0017112944042310119,
+            -0.017186567187309265,
+            -0.0002583931782282889,
+            -0.011175800114870071,
+            0.010511293075978756,
+            -0.004813900217413902,
+            0.0036226960364729166,
+            -0.017095953226089478,
+            0.014702217653393745,
+            0.013728111051023006,
+            0.021717296913266182,
+            0.02343897521495819,
+            -0.022155268117785454,
+            -0.015963271260261536,
+            0.005372690502554178,
+            0.0052443197928369045,
+            0.004515627399086952,
+            -0.005323607474565506,
+            0.0013752651866525412,
+            0.021264225244522095,
+            0.008555527776479721,
+            -0.0038152518682181835,
+            0.0107378289103508,
+            -0.011115389876067638,
+            -0.014513437636196613,
+            -0.008978395722806454,
+            -0.003983266651630402,
+            -0.018575992435216904,
+            -0.0028241549152880907,
+            0.012142355553805828,
+            0.011077634058892727,
+            -0.01480038370937109,
+            0.009295547381043434,
+            0.015917964279651642,
+            -0.004217354115098715,
+            -0.007706015836447477,
+            -0.010163936764001846,
+            -0.0008150594076141715,
+            0.030023634433746338,
+            0.00019161211093887687,
+            -0.006826299242675304,
+            0.014324656687676907,
+            -0.011930921114981174,
+            0.008208171464502811,
+            -0.02019195258617401,
+            -0.01150050200521946,
+            0.00011356321192579344,
+            -0.01032251212745905,
+            -0.016884518787264824,
+            -0.016748597845435143,
+            0.01916498690843582,
+            0.0055161635391414165,
+            -0.008449810557067394,
+            -0.035581331700086594,
+            0.025432495400309563,
+            -0.0018585430225357413,
+            0.040988001972436905,
+            0.006154241040349007,
+            0.007377538364380598,
+            -0.0005111229256726801,
+            0.015185495838522911,
+            0.011334375478327274,
+            -0.020841356366872787,
+            0.015140187926590443,
+            0.007173655088990927,
+            0.0011477848747745156,
+            0.007222738116979599,
+            -0.0025617501232773066,
+            0.0068036457523703575,
+            -0.0103753712028265,
+            -0.0005738924373872578,
+            -0.014415271580219269,
+            -0.027093762531876564,
+            -0.015412031672894955,
+            0.013222179375588894,
+            -0.00041154128848575056,
+            -0.0036283594090491533,
+            -0.018319251015782356,
+            -0.01789638213813305,
+            -0.01919519156217575,
+            -0.0075927479192614555,
+            -0.024209199473261833,
+            -0.008132659830152988,
+            -0.00958249345421791,
+            0.0043004173785448074,
+            0.015117534436285496,
+            0.010065770708024502,
+            -0.003264013212174177,
+            0.027697861194610596,
+            0.019316010177135468,
+            0.013599740341305733,
+            0.00011427114077378064,
+            -0.01158356573432684,
+            0.013063604012131691,
+            0.02665579319000244,
+            -0.0025277696549892426,
+            0.003892651991918683,
+            0.00730957742780447,
+            -0.006309041287750006,
+            0.014083018526434898,
+            -0.008336542174220085,
+            0.0031318669207394123,
+            0.009091664105653763,
+            -0.016099192202091217,
+            0.0015602700877934694,
+            0.02038828283548355,
+            0.013471369631588459,
+            0.00420225178822875,
+            0.0009920409647747874,
+            0.008985946886241436,
+            0.008110006339848042,
+            -0.006271285004913807,
+            -0.012240521609783173,
+            -0.005006456281989813,
+            0.006441187579184771,
+            0.0109719168394804,
+            0.012165009044110775,
+            -0.020765844732522964,
+            0.017322489991784096,
+            0.016265319660305977,
+            -0.013577086851000786,
+            0.012512365356087685,
+            -0.0038662226870656013,
+            0.003658564295619726,
+            0.015125085599720478,
+            0.008985946886241436,
+            -0.0002326718531548977,
+            -0.013939544558525085,
+            0.012097048573195934,
+            -0.02522106096148491,
+            0.004364602733403444,
+            -0.008729205466806889,
+            -0.0310203954577446,
+            0.012572774663567543,
+            -0.02396756038069725,
+            -0.0006475168047472835,
+            0.005704943556338549,
+            -0.013750764541327953,
+            0.001948213786818087,
+            -0.010752932168543339,
+            0.005131051409989595,
+            0.006071177776902914,
+            -0.007777752820402384,
+            -0.016718393191695213,
+            -0.00514615373685956,
+            1.4519867363560479e-05,
+            -0.000538968073669821,
+            -0.023423872888088226,
+            0.02291039004921913,
+            -0.006173118948936462,
+            -0.017986996099352837,
+            0.0004884693189524114,
+            -0.02291039004921913,
+            -0.018606197088956833,
+            -0.006078728940337896,
+            -0.005923929158598185,
+            -0.021853219717741013,
+            -0.00916717667132616,
+            0.0206450242549181,
+            -0.005478407256305218,
+            -0.012535018846392632,
+            -0.024103481322526932,
+            -0.0015527188079431653,
+            0.000700375298038125,
+            -0.008638591505587101,
+            -0.015268558636307716,
+            0.015155291184782982,
+            0.016960032284259796,
+            -0.019572751596570015,
+            -0.010662317276000977,
+            0.006414758041501045,
+            -0.0009122812771238387,
+            0.01859109476208687,
+            0.0063958801329135895,
+            -0.012361341156065464,
+            -0.011560912244021893,
+            0.014468129724264145,
+            0.00952963437885046,
+            0.021626682952046394,
+            0.006214650813490152,
+            0.01090395636856556,
+            0.0057011679746210575,
+            -0.0025787402410060167,
+            -0.009250239469110966,
+            -9.225167559634428e-06,
+            0.010987019166350365,
+            -0.005384017247706652,
+            -0.009605146944522858,
+            0.0023201112635433674,
+            -0.0031941644847393036,
+            0.00692824088037014,
+            -0.010156385600566864,
+            -0.0064223092049360275,
+            -0.027169276028871536,
+            -0.019270703196525574,
+            -0.019708674401044846,
+            -0.009091664105653763,
+            0.00814021099358797,
+            -0.008653693832457066,
+            -0.014966510236263275,
+            0.0041154129430651665,
+            0.014279349707067013,
+            -0.03543030470609665,
+            -0.009522083215415478,
+            -0.004753490444272757,
+            -0.0029355352744460106,
+            -0.036215633153915405,
+            0.019844595342874527,
+            0.025115344673395157,
+            -0.0005441594985313714,
+            -0.000711230153683573,
+            -0.0030582426115870476,
+            -0.012655838392674923,
+            -0.008978395722806454,
+            0.002072808798402548,
+            0.0020067356526851654,
+            -0.0270786602050066,
+            0.005285851191729307,
+            0.01388668641448021,
+            0.021490762010216713,
+            0.008593283593654633,
+            -0.015495095402002335,
+            0.0036510128993541002,
+            -0.024677373468875885,
+            -0.025402290746569633,
+            -0.0220495518296957,
+            -0.004557158797979355,
+            -0.006018319167196751,
+            0.004606241825968027,
+            0.0056332070380449295,
+            0.01842496730387211,
+            0.00719253346323967,
+            0.005493509583175182,
+            -0.01972377672791481,
+            0.010911507532000542,
+            0.0009377666283398867,
+            0.0005111229256726801,
+            -0.012897477485239506,
+            -0.01412077434360981,
+            0.02343897521495819,
+            -0.00041319310548715293,
+            0.014551193453371525,
+            0.006029645912349224,
+            -0.009650453925132751,
+            -0.007966533303260803,
+            -0.0012940897140651941,
+            -0.026066798716783524,
+            0.016325728967785835,
+            -0.011168248951435089,
+            0.0006970716640353203,
+            -0.015555505640804768,
+            -0.00976372230798006,
+            0.022155268117785454,
+            0.012640736065804958,
+            -0.0025881794281303883,
+            0.015933066606521606,
+            0.018349455669522285,
+            0.0008315776940435171,
+            0.005474631674587727,
+            -0.017971893772482872,
+            -0.007230289280414581,
+            0.008396952413022518,
+            -0.0035188666079193354,
+            0.008615938015282154,
+            -0.023650409653782845,
+            -0.005969236139208078,
+            0.016612675040960312,
+            0.0214756578207016,
+            -0.021430350840091705,
+            0.001019414165057242,
+            0.0068036457523703575,
+            0.009325752034783363,
+            -0.006879157852381468,
+            -0.009733517654240131,
+            -0.015902861952781677,
+            -0.00893308874219656,
+            -0.005923929158598185,
+            0.021173609420657158,
+            -0.01853068359196186,
+            -0.004341949243098497,
+            -0.0025617501232773066,
+            0.005999441258609295,
+            -0.013962198980152607,
+            -0.0039379592053592205,
+            -0.016159603372216225,
+            -0.0009269117726944387,
+            0.013871584087610245,
+            -0.008744308724999428,
+            -0.024752886965870857,
+            0.010201692581176758,
+            0.002148320898413658,
+            -0.005055539309978485,
+            -0.0012166897067800164,
+            -0.018938450142741203,
+            0.025311676785349846,
+            -0.011628872714936733,
+            0.011190902441740036,
+            -0.012950335629284382,
+            -0.0034037106670439243,
+            0.01889314316213131,
+            0.030416298657655716,
+            0.01131927315145731,
+            -0.005587900057435036,
+            -0.018908245489001274,
+            0.015419583767652512,
+            0.0063543482683598995,
+            -0.005610553547739983,
+            -0.004553383216261864,
+            0.0215813759714365,
+            -0.017745357006788254,
+            -0.03461477532982826,
+            0.012361341156065464,
+            -0.0003303656994830817,
+            -0.0306579377502203,
+            -0.010231898166239262,
+            -0.006067402195185423,
+            -0.0015272335149347782,
+            0.016235115006566048,
+            0.006603538524359465,
+            -0.004357051569968462,
+            0.001753769931383431,
+            0.0021879649721086025,
+            0.002920432947576046,
+            -0.008457361720502377,
+            -0.0013611067552119493,
+            0.0006659228820353746,
+            0.0007739996653981507,
+            0.020871561020612717,
+            -0.016235115006566048,
+            -0.004409910179674625,
+            -0.0065997629426419735,
+            0.004776144400238991,
+            0.018047407269477844,
+            0.009801478125154972,
+            0.03265145793557167,
+            0.007396416272968054,
+            0.00351697881706059,
+            -0.0040738810785114765,
+            -0.010020463727414608,
+            0.00741529418155551,
+            0.011893165297806263,
+            -0.0017131821950897574,
+            -0.01047353632748127,
+            -0.011870511807501316,
+            -0.014241593889892101,
+            0.007626728154718876,
+            0.013199525885283947,
+            0.00045307297841645777,
+            -0.008464912883937359,
+            -0.011213555932044983,
+            0.025039833039045334,
+            -0.025900671258568764,
+            0.01892334781587124,
+            -0.006724358070641756,
+            -0.002399398945271969,
+            0.023710818961262703,
+            0.015072227455675602,
+            0.015019369311630726,
+            -0.010684970766305923,
+            -0.026957841590046883,
+            0.01670329086482525,
+            -0.0034414667170494795,
+            -0.002335213590413332,
+            -0.001184597029350698,
+            0.01063966378569603,
+            -0.03259104862809181,
+            -0.012897477485239506,
+            0.012429301626980305,
+            -0.0008688617963343859,
+            0.012225419282913208,
+            -0.01611429452896118,
+            0.02363530732691288,
+            0.010979468002915382,
+            0.01697513461112976,
+            -0.012731350027024746,
+            0.00012223531666677445,
+            0.023227540776133537,
+            -0.00040918152080848813,
+            -0.010231898166239262,
+            -0.004474095534533262,
+            -0.007222738116979599,
+            -0.0006470448570325971,
+            -0.013577086851000786,
+            0.012165009044110775,
+            -0.00468930508941412,
+            0.011221107095479965,
+            0.018273942172527313,
+            0.019738879054784775,
+            0.021264225244522095,
+            0.0031998278573155403,
+            -0.0008315776940435171,
+            -0.006905586924403906,
+            0.005625655874609947,
+            -0.0019595406483858824,
+            -0.00024801024119369686,
+            -0.02582515962421894,
+            0.021369941532611847,
+            -0.008797166869044304,
+            0.009408814832568169,
+            0.004055003169924021,
+            1.6061083442764357e-05,
+            0.015721632167696953,
+            0.0012761554680764675,
+            -0.0018764771521091461,
+            0.004530729725956917,
+            -0.012368892319500446,
+            0.00458358833566308,
+            -0.011636423878371716,
+            0.010465985164046288,
+            0.018455171957612038,
+            -0.007702240254729986,
+            -0.009053908288478851,
+            0.009295547381043434,
+            0.0027146621141582727,
+            -0.006724358070641756,
+            0.0071849822998046875,
+            0.010858648456633091,
+            -0.00734355766326189,
+            -0.005274524446576834,
+            0.006418533623218536,
+            -0.01090395636856556,
+            -0.0007268045446835458,
+            0.006645070388913155,
+            -0.015661222860217094,
+            0.01961805857717991,
+            0.003146969247609377,
+            0.01816822588443756,
+            -0.009635351598262787,
+            0.008827371522784233,
+            -0.006592211779206991,
+            0.013954647816717625,
+            -0.02127932757139206,
+            -0.015585710294544697,
+            0.00866879615932703,
+            -0.008744308724999428,
+            0.017277183011174202,
+            0.008593283593654633,
+            0.011175800114870071,
+            -0.004334398079663515,
+            -0.013796071521937847,
+            0.004768592771142721,
+            -0.0010128068970516324,
+            0.0040474520064890385,
+            -0.01958785392343998,
+            0.026942739263176918,
+            0.01955764926970005,
+            0.020358078181743622,
+            -0.016491856426000595,
+            -0.016295524314045906,
+            0.004104085732251406,
+            0.02685212530195713,
+            -0.01547999307513237,
+            0.0008485679281875491,
+            0.01474752463400364,
+            -0.007577645592391491,
+            -0.004270212724804878,
+            0.02052420563995838,
+            -0.0026561403647065163,
+            -0.0376654677093029,
+            -0.013388305902481079,
+            0.00564453424885869,
+            0.003026149934157729,
+            0.003375393571332097,
+            -0.0374540314078331,
+            0.003987042233347893,
+            -0.025447597727179527,
+            0.004677978344261646,
+            -0.008268581703305244,
+            -0.015631018206477165,
+            0.0003327254671603441,
+            -0.001685808994807303,
+            0.009650453925132751,
+            0.005436875391751528,
+            0.00408520782366395,
+            -0.006392104551196098,
+            0.012210316024720669,
+            -0.006720582488924265,
+            -0.002958188997581601,
+            -0.005546368192881346,
+            0.018485376611351967,
+            0.005127275828272104,
+            0.014958959072828293,
+            0.0043872566893696785,
+            -0.01647675409913063,
+            0.00783816259354353,
+            0.010269653983414173,
+            -0.008162864483892918,
+            -0.00756631838157773,
+            0.002348428126424551,
+            -0.004610017407685518,
+            -0.004081432241946459,
+            -0.02061481960117817,
+            -0.0037888227961957455,
+            0.008291235193610191,
+            0.021958936005830765,
+            -0.0025541989598423243,
+            -0.008646142669022083,
+            -0.0029279841110110283,
+            -0.0034716716036200523,
+            0.010465985164046288,
+            -0.015419583767652512,
+            0.021792810410261154,
+            -0.002814715728163719,
+            0.011092736385762691,
+            0.010805790312588215,
+            0.00801939144730568,
+            0.0055388170294463634,
+            0.008442259393632412,
+            0.004217354115098715,
+            0.00964290276169777,
+            0.005595451220870018,
+            -0.02069033309817314,
+            -0.009038805961608887,
+            0.012429301626980305,
+            -0.004934719763696194,
+            -0.015434686094522476,
+            0.03231920674443245,
+            0.008087352849543095,
+            0.0006078729056753218,
+            0.007672035600990057,
+            0.03159428760409355,
+            0.0015848114853724837,
+            0.005772904958575964,
+            0.01123620942234993,
+            0.010828443802893162,
+            -0.0029997206293046474,
+            -0.012429301626980305,
+            -0.044038690626621246,
+            0.0008792447042651474,
+            0.0015404480509459972,
+            0.020750742405653,
+            -0.033225350081920624,
+            0.009590044617652893,
+            -0.013048501685261726,
+            0.008087352849543095,
+            -0.016990236937999725,
+            0.007830611430108547,
+            0.006331694778054953,
+            0.01822863519191742,
+            0.010609458200633526,
+            -0.011032327078282833,
+            0.008865127339959145,
+            0.0015904748579487205,
+            -0.011334375478327274,
+            0.0014083017595112324,
+            0.02810562588274479,
+            0.020599717274308205,
+            0.011168248951435089,
+            -0.0018028528429567814,
+            0.01853068359196186,
+            0.014807934872806072,
+            -0.009129419922828674,
+            -0.027093762531876564,
+            0.003180949715897441,
+            -0.0031564084347337484,
+            -0.004666651599109173,
+            0.01128906849771738,
+            -0.0043457248248159885,
+            0.006361899431794882,
+            0.01750371977686882,
+            -0.006222201976925135,
+            -0.0006158960750326514,
+            0.010360268875956535,
+            0.008117557503283024,
+            -0.0023371013812720776,
+            0.016250217333436012,
+            -0.015615914948284626,
+            0.021158507093787193,
+            -0.025900671258568764,
+            -0.0014696554280817509,
+            0.012625633738934994,
+            0.006958445534110069,
+            0.017820870503783226,
+            -0.01218011137098074,
+            -0.02519085630774498,
+            -0.00976372230798006,
+            0.01249726302921772,
+            0.012957886792719364,
+            0.00420225178822875,
+            0.009031254798173904,
+            0.01196867786347866,
+            -0.004927168600261211,
+            -0.0032451350707560778,
+            0.0012676604092121124,
+            -0.013237281702458858,
+            -0.00722651369869709,
+            -0.009741068817675114,
+            0.010065770708024502,
+            0.0007268045446835458,
+            -0.01079068798571825,
+            0.007283147890120745,
+            -0.02197403833270073,
+            -0.008351645432412624,
+            0.004602466244250536,
+            -0.0024900136049836874,
+            0.0018670381978154182,
+            -0.015313866548240185,
+            0.0074568260461091995,
+            0.00040257422369904816,
+            -0.003881325013935566,
+            -0.010828443802893162,
+            0.022260984405875206,
+            0.00450807623565197,
+            0.003953061532229185,
+            0.013071155175566673,
+            -0.00681119691580534,
+            -0.011538257822394371,
+            0.01544978842139244,
+            -0.009242688305675983,
+            0.0013658262323588133,
+            -0.005939031485468149,
+            -0.012240521609783173,
+            -0.008736756630241871,
+            -0.010488639585673809,
+            -0.032500434666872025,
+            -0.024420632049441338,
+            0.016386138275265694,
+            0.0015574382850900292,
+            0.012338686734437943,
+            0.017216771841049194,
+            -0.012165009044110775,
+            0.008132659830152988,
+            -0.03636665642261505,
+            0.00128653843421489,
+            -0.02155117131769657,
+            -0.01047353632748127,
+            0.0008839642396196723,
+            0.00810245517641306,
+            0.014143427833914757,
+            -0.008011840283870697,
+            0.00790612306445837,
+            -0.0018962991889566183,
+            0.018787425011396408,
+            0.015615914948284626,
+            0.003647237317636609,
+            0.0012893701205030084,
+            -0.007173655088990927,
+            -0.00011834171164082363,
+            0.022064654156565666,
+            -0.002271028235554695,
+            -0.017443308606743813,
+            0.0176396407186985,
+            0.022215677425265312,
+            0.0025202184915542603,
+            -0.02496432140469551,
+            0.005542592611163855,
+            0.0025277696549892426,
+            0.017020441591739655,
+            -0.0040738810785114765,
+            -0.010843546129763126,
+            -0.004610017407685518,
+            -0.0027580817695707083,
+            0.03881325200200081,
+            0.018409864977002144,
+            -0.005047988146543503,
+            -0.03231920674443245,
+            0.0021709746215492487,
+            -0.0028807888738811016,
+            1.456411337130703e-05,
+            0.01988990418612957,
+            0.02393735572695732,
+            0.023559793829917908,
+            -0.0032470228616148233,
+            0.007739996537566185,
+            -0.019980518147349358,
+            0.04331377521157265,
+            -0.007853264920413494,
+            -0.016884518787264824,
+            -0.015102432109415531,
+            -0.009046357125043869,
+            0.003919081296771765,
+            0.008532874286174774,
+            0.006611089687794447,
+            -0.008261030539870262,
+            -0.004704407416284084,
+            0.010428229346871376,
+            0.008631040342152119,
+            -0.017956791445612907,
+            0.013003193773329258,
+            -0.02393735572695732,
+            0.016718393191695213,
+            -0.01603878289461136,
+            0.00035797484451904893,
+            0.0037982617504894733,
+            -0.012202764861285686,
+            -0.029842406511306763,
+            -0.01176479458808899,
+            0.017186567187309265,
+            0.013448716141283512,
+            0.008547976613044739,
+            0.0035679496359080076,
+            0.005757802166044712,
+            -0.009544737637043,
+            -0.00016801456513348967,
+            0.01191581878811121,
+            0.027365608140826225,
+            0.01703554391860962,
+            0.0129427844658494,
+            0.025432495400309563,
+            -0.00737376231700182,
+            -0.039598576724529266,
+            -0.0035056520719081163,
+            0.00943146925419569,
+            0.007883469574153423,
+            0.002072808798402548,
+            0.002338989172130823,
+            -0.019980518147349358,
+            0.0062184263952076435,
+            -0.0003690656740218401,
+            -0.00681119691580534,
+            -0.020901765674352646,
+            0.007966533303260803,
+            -0.013214628212153912,
+            -0.0005021558608859777,
+            0.0007466264651156962,
+            0.00331120821647346,
+            0.007453050464391708,
+            0.012383994646370411,
+            0.0020916869398206472,
+            -0.0013309018686413765,
+            -0.006826299242675304,
+            -0.008079801686108112,
+            -0.016582470387220383,
+            0.01170438528060913,
+            -0.01362239383161068,
+            -0.005195236764848232,
+            -0.004077656660228968,
+            -0.003949285950511694,
+            0.002650476759299636,
+            0.02088666334748268,
+            0.0042098029516637325,
+            0.0027486425824463367,
+            0.01301074493676424,
+            0.008442259393632412,
+            0.0039115301333367825,
+            0.03476579859852791,
+            -0.01836455799639225,
+            -0.0063279191963374615,
+            -0.014150978997349739,
+            -0.01544978842139244,
+            -0.019013961777091026,
+            0.022019345313310623,
+            -0.004296641796827316,
+            -0.010133732110261917,
+            0.007200084626674652,
+            0.014679564163088799,
+            -0.013788520358502865,
+            3.648771235020831e-05,
+            0.010964365676045418,
+            -0.015117534436285496,
+            0.010896405205130577,
+            -0.009544737637043,
+            0.010609458200633526,
+            0.00810245517641306,
+            0.03956837207078934,
+            -0.009295547381043434,
+            -0.014468129724264145,
+            0.0026523645501583815,
+            0.009861888363957405,
+            0.018470274284482002,
+            -0.024677373468875885,
+            0.0013148555299267173,
+            -0.0012733237817883492,
+            -0.000611648487392813,
+            -0.00662241643294692,
+            -0.006252407096326351,
+            0.014415271580219269,
+            -0.006520475260913372,
+            -0.01055660005658865,
+            -0.024949217215180397,
+            -0.017971893772482872,
+            -0.00961269810795784,
+            0.022653648629784584,
+            0.024284711107611656,
+            -0.005399119574576616,
+            -0.005803109612315893,
+            0.0011978116817772388,
+            -0.01567632518708706,
+            0.01700533926486969,
+            0.013629944995045662,
+            -0.02535698376595974,
+            -0.0019000747706741095,
+            -0.009446571581065655,
+            -0.006365675013512373,
+            -0.014543642289936543,
+            0.00878206454217434,
+            0.05584879592061043,
+            -0.02655007503926754,
+            -0.0003390967904124409,
+            -0.01733759231865406,
+            0.00642986036837101,
+            -0.010722726583480835,
+            -0.005074417218565941,
+            -0.01647675409913063,
+            -0.01040557585656643,
+            -0.006656397134065628,
+            -0.006686601787805557,
+            -0.004311744589358568,
+            0.00869900081306696,
+            -0.007294474635273218,
+            -0.02227608673274517,
+            0.016250217333436012,
+            -0.031050600111484528,
+            0.002148320898413658,
+            0.010382922366261482,
+            -0.0017037431243807077,
+            0.019104577600955963,
+            -0.028030114248394966,
+            0.01165152620524168,
+            0.004927168600261211,
+            0.00137148960493505,
+            0.007083040662109852,
+            -0.014641808345913887,
+            -0.003188501112163067,
+            -0.01683921180665493,
+            -0.003941734787076712,
+            0.009748619981110096,
+            -0.010692521929740906,
+            0.023212438449263573,
+            -0.014483232982456684,
+            -0.011070082895457745,
+            0.015827348455786705,
+            0.013977301307022572,
+            0.014672012999653816,
+            -0.013546881265938282,
+            -0.0030412522610276937,
+            -0.010496190749108791,
+            0.016688188537955284,
+            -0.0021974039264023304,
+            -0.021883424371480942,
+            -0.014234042726457119,
+            0.02055441029369831,
+            -0.008042044937610626,
+            0.022381804883480072,
+            -0.002335213590413332,
+            -0.001269548200070858,
+            -0.008472464047372341,
+            0.005191461183130741,
+            -0.0006215595058165491,
+            -0.0025674134958535433,
+            0.017277183011174202,
+            -0.0025070037227123976,
+            0.026776611804962158,
+            -0.01498161256313324,
+            0.015585710294544697,
+            -0.029162796214222908,
+            -0.009204932488501072,
+            -0.00783816259354353,
+            0.0014545529847964644,
+            -0.00017084626597352326,
+            0.0008608386269770563,
+            0.030340787023305893,
+            -0.01291257981210947,
+            -0.02241200953722,
+            0.023741023615002632,
+            -0.015933066606521606,
+            -0.0328930988907814,
+            0.005587900057435036,
+            -0.001396975014358759,
+            0.004768592771142721,
+            0.008917986415326595,
+            -0.007083040662109852,
+            -0.025734543800354004,
+            0.012844618409872055,
+            0.01806250959634781,
+            0.0030374766793102026,
+            4.574975173454732e-05,
+            -0.015004266053438187,
+            -0.019316010177135468,
+            -0.011281517334282398,
+            0.008366747759282589,
+            -0.0040738810785114765,
+            0.020463796332478523,
+            -0.017262080684304237,
+            -0.00405122758820653,
+            -0.006361899431794882,
+            -0.008495118468999863,
+            0.011424990370869637,
+            0.028966465964913368,
+            0.010601907037198544,
+            0.004417461343109608,
+            -0.00123084825463593,
+            0.0015225139213725924,
+            0.0021728624124079943,
+            0.008064698427915573,
+            0.009325752034783363,
+            -0.006444963160902262,
+            -0.0016839212039485574,
+            -0.009287996217608452,
+            -0.0024088378995656967,
+            -0.01335810124874115,
+            0.003643461735919118,
+            -0.003515091026201844,
+            0.005678514484316111,
+            -0.01975398138165474,
+            0.01020924374461174,
+            -0.010284756310284138,
+            0.0031564084347337484,
+            -0.02088666334748268,
+            -0.002552310936152935,
+            0.017473513260483742,
+            -0.010277205146849155,
+            0.018621299415826797,
+            0.0010618898086249828,
+            0.026097003370523453,
+            0.008042044937610626,
+            -0.0030242621432989836,
+            0.003998368978500366,
+            0.0055652461014688015,
+            0.00476481718942523,
+            0.014173632487654686,
+            0.004153168760240078,
+            -0.012618082575500011,
+            0.006241080351173878,
+            0.012550121173262596,
+            -0.008532874286174774,
+            0.011674179695546627,
+            -0.0009844898013398051,
+            0.01700533926486969,
+            -0.010775585658848286,
+            0.010911507532000542,
+            0.0079363277181983,
+            0.007891020737588406,
+            -0.023378565907478333,
+            -0.0038020373322069645,
+            0.023650409653782845,
+            -0.014543642289936543,
+            0.07092101871967316,
+            -0.005893724039196968,
+            0.025508008897304535,
+            -0.008570630103349686,
+            -0.012104599736630917,
+            -0.008034493774175644,
+            -0.004323071334511042,
+            0.007898571901023388,
+            -0.010601907037198544,
+            0.027456222102046013,
+            -0.011606219224631786,
+            0.00439103227108717,
+            -0.006788543425500393,
+            0.003760505700483918,
+            0.034916821867227554,
+            0.011198453605175018,
+            0.021369941532611847,
+            0.012210316024720669,
+            0.010662317276000977,
+            0.00040918152080848813,
+            -0.004693080671131611,
+            0.010118629783391953,
+            0.006414758041501045,
+            0.012776657938957214,
+            -0.014996714890003204,
+            0.012217868119478226,
+            0.01020924374461174,
+            0.01683921180665493,
+            -0.014611602760851383,
+            -0.004719510208815336,
+            0.011266414076089859,
+            0.0012761554680764675,
+            -0.012482160702347755,
+            -0.011643975041806698,
+            0.015079778619110584,
+            0.015042022801935673,
+            0.008563078939914703,
+            0.00810245517641306,
+            -0.00987699069082737,
+            0.0012563335476443172,
+            0.008744308724999428,
+            -0.01188561413437128,
+            0.005723821930587292,
+            -0.006875382270663977,
+            -0.006346797104924917,
+            0.007951430976390839,
+            0.003971939906477928,
+            -0.002012399025261402,
+            -0.013765866868197918,
+            0.010994570329785347,
+            -0.010254551656544209,
+            -0.01567632518708706,
+            -0.012044189497828484,
+            0.000821666675619781,
+            0.012202764861285686,
+            -0.0076607088558375835,
+            -0.002223833231255412,
+            -0.012587876990437508,
+            0.0036453495267778635,
+            0.017292285338044167,
+            0.007588972337543964,
+            0.020660126581788063,
+            0.023227540776133537
+          ],
+          "title": "Addressing Performance Challenges",
+          "keyphrases": [
+            "Shrinking Batch Problem",
+            "Mixing Data Parallelism and Model Parallelism",
+            "hierarchical MoE",
+            "network bandwidth",
+            "computational efficiency"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "ff7dd6f5-6560-42e5-923e-f4091e278202",
+        "properties": {
+          "page_content": "4 Balancing Expert Utilization\n\nWe have observed that the gating network tends to converge to a state where it always produces large weights for the same few experts. This imbalance is self-reinforcing, as the favored experts are trained more rapidly and thus are selected even more by the gating network. Eigen et al. (2013) describe the same phenomenon, and use a hard constraint at the beginning of training to avoid this local minimum. Bengio et al. (2015) include a soft constraint on the batch-wise average of each gate.1 We take a soft constraint approach. We define the importance of an expert relative to a batch of training examples to be the batchwise sum of the gate values for that expert. We define an additional loss Limportance, which is added to the overall loss function for the model. This loss is equal to the square of the coefficient of variation of the set of importance values, multiplied by a hand-tuned scaling factor wimportance. This additional loss encourages all experts to have equal importance.\n\n$$I m p o r t a n c e(X)=\\sum_{x\\in X}G(x)$$ $$L_{importance}(X)=w_{importance}\\cdot CV(Importance(X))^{2}\\tag{7}$$ $$(6)$$\n\nWhile this loss function can ensure equal importance, experts may still receive very different numbers of examples. For example, one expert may receive a few examples with large weights, and another may receive many examples with small weights. This can cause memory and performance problems on distributed hardware. To solve this problem, we introduce a second loss function, Lload , which ensures balanced loads. Appendix A contains the definition of this function, along with experimental results.\n\n",
+          "embedding": [
+            -0.013197094202041626,
+            -0.002106932457536459,
+            -0.020997902378439903,
+            -0.01955360174179077,
+            -0.0042971898801624775,
+            0.013943051919341087,
+            -0.01379227265715599,
+            0.05729586258530617,
+            -0.02623547427356243,
+            0.009284786880016327,
+            -0.014958823099732399,
+            0.013482780195772648,
+            -0.0004461359931156039,
+            -0.0026227538473904133,
+            0.012546366080641747,
+            0.013776401989161968,
+            0.013578008860349655,
+            -0.0015841671265661716,
+            0.0022577110212296247,
+            -0.008999100886285305,
+            0.036821719259023666,
+            -0.03621860593557358,
+            -0.02348971739411354,
+            0.012839986942708492,
+            -0.011721051298081875,
+            0.029409760609269142,
+            0.025187959894537926,
+            -0.015276252292096615,
+            -0.048630062490701675,
+            0.009760930202901363,
+            0.025886302813887596,
+            -0.002469991333782673,
+            -0.04361468926072121,
+            -0.004309093579649925,
+            -0.04536054655909538,
+            -0.018458474427461624,
+            0.04875703528523445,
+            0.02812417410314083,
+            -0.012403522618114948,
+            0.030568374320864677,
+            0.02133120223879814,
+            0.006888200994580984,
+            -0.03971032053232193,
+            0.01788710243999958,
+            0.009380015544593334,
+            -0.032631661742925644,
+            -0.03777400404214859,
+            -0.0011179439024999738,
+            -0.0026505289133638144,
+            -0.03621860593557358,
+            0.013538329862058163,
+            -0.05221700668334961,
+            0.005737522151321173,
+            -0.018934616819024086,
+            -0.011316330172121525,
+            0.02893361821770668,
+            -0.010998900979757309,
+            0.014236672781407833,
+            0.042598918080329895,
+            0.01918855868279934,
+            0.006015272345393896,
+            0.03012397512793541,
+            -0.055962663143873215,
+            0.021616889163851738,
+            0.023997602984309196,
+            -0.04751906171441078,
+            -0.0064636399038136005,
+            0.01644280180335045,
+            -0.001340143964625895,
+            -0.012998702004551888,
+            -0.06805669516324997,
+            0.02564823254942894,
+            -0.0355837456882,
+            0.025187959894537926,
+            0.005527225788682699,
+            -0.006130340043455362,
+            0.042027547955513,
+            0.03217139095067978,
+            -0.0028231306932866573,
+            0.013181222602725029,
+            -0.026933817192912102,
+            0.009205429814755917,
+            0.004031343385577202,
+            0.022108903154730797,
+            0.04294808954000473,
+            -0.010895737446844578,
+            -0.00926891528069973,
+            -0.05367717891931534,
+            0.01799820177257061,
+            -0.022077159956097603,
+            -0.015966659411787987,
+            0.02117248810827732,
+            0.01120522990822792,
+            -0.012308293953537941,
+            -0.02818766050040722,
+            0.03694869205355644,
+            -0.000555996026378125,
+            -0.004745557904243469,
+            -0.010451337322592735,
+            -0.008872129954397678,
+            -0.02236284501850605,
+            0.006181922275573015,
+            -0.02958434633910656,
+            -0.029520859941840172,
+            0.030203333124518394,
+            -0.022378716617822647,
+            -0.01826801709830761,
+            0.012459073215723038,
+            0.030250946059823036,
+            0.10710041224956512,
+            -0.009221301414072514,
+            -0.009784736670553684,
+            -0.030838189646601677,
+            0.013371679931879044,
+            -0.03891674801707268,
+            -0.037043917924165726,
+            -0.027632160112261772,
+            -0.06120023503899574,
+            0.025362545624375343,
+            -0.023584945127367973,
+            0.023188160732388496,
+            0.04977280646562576,
+            0.012966958805918694,
+            0.01961708813905716,
+            -0.044916149228811264,
+            0.04409083351492882,
+            -0.023997602984309196,
+            0.027775002643465996,
+            0.008102364838123322,
+            0.0019363145111128688,
+            0.0032278522849082947,
+            0.02650528959929943,
+            -0.02269614487886429,
+            -0.044376518577337265,
+            0.006757261697202921,
+            0.013300258666276932,
+            0.06339049339294434,
+            0.02499750256538391,
+            0.01441919431090355,
+            -0.034345775842666626,
+            0.01129252277314663,
+            -0.053645435720682144,
+            -0.019220301881432533,
+            0.015982531011104584,
+            -0.00685645779594779,
+            0.016347574070096016,
+            -0.00514631113037467,
+            0.020442401990294456,
+            -0.016887202858924866,
+            0.010364044457674026,
+            -0.005693875718861818,
+            0.04843960702419281,
+            -0.006166050676256418,
+            0.024108702316880226,
+            0.019061587750911713,
+            0.016506288200616837,
+            -0.028949489817023277,
+            0.025553002953529358,
+            -0.015427030622959137,
+            0.0011467108270153403,
+            -0.0006834634696133435,
+            -0.001254834933206439,
+            -0.015847623348236084,
+            0.02344210259616375,
+            0.024965759366750717,
+            0.06729486584663391,
+            0.01676023006439209,
+            0.013903372921049595,
+            -0.07834137976169586,
+            0.026759231463074684,
+            -0.0036087664775550365,
+            -0.03891674801707268,
+            0.03996426239609718,
+            0.03631383180618286,
+            2.3481657990487292e-05,
+            0.02656877413392067,
+            -0.007566704414784908,
+            0.017760131508111954,
+            -0.014720751903951168,
+            0.03980554640293121,
+            -0.03479017689824104,
+            -0.014744559302926064,
+            0.008058718405663967,
+            0.007725419010967016,
+            0.01649041660130024,
+            0.008308693766593933,
+            0.026743359863758087,
+            0.012879665940999985,
+            -0.019442502409219742,
+            0.004567004274576902,
+            0.028378117829561234,
+            0.01788710243999958,
+            0.014355708844959736,
+            -0.003918259404599667,
+            0.016887202858924866,
+            -0.03574246168136597,
+            0.010935415513813496,
+            -0.03307606279850006,
+            0.0299017746001482,
+            0.0007930755382403731,
+            -0.005070921964943409,
+            -0.02004561759531498,
+            0.013974794186651707,
+            -0.015831751748919487,
+            -0.023997602984309196,
+            0.012403522618114948,
+            0.03732960298657417,
+            0.004404321778565645,
+            0.013189158402383327,
+            -0.03539329022169113,
+            0.009078457951545715,
+            -0.02053763158619404,
+            -0.018093431368470192,
+            0.004916175734251738,
+            -0.05082032084465027,
+            -0.01330819446593523,
+            0.0177760012447834,
+            0.058819521218538284,
+            0.042376719415187836,
+            -0.007765097543597221,
+            -0.026965560391545296,
+            0.02139468863606453,
+            0.031076261773705482,
+            0.00046944714267738163,
+            -0.0034480681642889977,
+            -0.014585844241082668,
+            -0.004142443183809519,
+            -0.013808144256472588,
+            0.008927679620683193,
+            -0.014593780040740967,
+            0.007959522306919098,
+            0.015442902222275734,
+            0.03767877444624901,
+            0.021902574226260185,
+            -0.0007375255227088928,
+            -0.0022081127390265465,
+            -0.012522558681666851,
+            -0.005023307632654905,
+            0.0010033720172941685,
+            -0.03552026301622391,
+            -0.026600517332553864,
+            -0.017474444583058357,
+            -0.03840886056423187,
+            0.0005227652145549655,
+            -0.011403623037040234,
+            0.011356008239090443,
+            -0.00880864355713129,
+            0.03821840509772301,
+            -0.0026088664308190346,
+            -0.011022708378732204,
+            -0.02785436064004898,
+            -0.027394089847803116,
+            -0.008149979636073112,
+            0.021109001711010933,
+            -0.011363944038748741,
+            0.014196994714438915,
+            0.02785436064004898,
+            -0.047614291310310364,
+            0.011840087361633778,
+            -0.0018361260881647468,
+            0.030949288979172707,
+            -0.004781268537044525,
+            -0.022267617285251617,
+            0.01896636001765728,
+            -0.01371291559189558,
+            -0.03599640354514122,
+            0.047233376652002335,
+            0.0371708907186985,
+            0.024235675111413002,
+            -0.005400254391133785,
+            0.020616987720131874,
+            -0.027552803978323936,
+            0.006423961371183395,
+            -0.0549786351621151,
+            0.03675823286175728,
+            -0.03640906140208244,
+            -0.025505388155579567,
+            -0.007392118684947491,
+            -0.014133509248495102,
+            0.050026748329401016,
+            -0.007046915125101805,
+            0.034028347581624985,
+            0.008967358618974686,
+            -0.007400054484605789,
+            0.03644080460071564,
+            -0.012355908751487732,
+            -0.02187083102762699,
+            -0.031330205500125885,
+            -0.047614291310310364,
+            -0.011848023161292076,
+            -0.0003290841996204108,
+            -0.010014872997999191,
+            -0.001375854597426951,
+            0.006765197031199932,
+            0.04707466438412666,
+            -0.00778493657708168,
+            -0.02625134587287903,
+            0.0028211467433720827,
+            0.01669674552977085,
+            -0.019315531477332115,
+            0.005578808020800352,
+            0.0038865164387971163,
+            0.01633170247077942,
+            -0.026759231463074684,
+            -0.02537841722369194,
+            0.006816779263317585,
+            -0.033488716930150986,
+            0.01680784486234188,
+            -0.03137781843543053,
+            0.0004920143401250243,
+            0.015982531011104584,
+            0.0218232162296772,
+            -0.0049836291000247,
+            -0.05637532100081444,
+            -0.04078957810997963,
+            0.01653803139925003,
+            0.015696844086050987,
+            0.03191744536161423,
+            0.0010425546206533909,
+            0.0014323965879157186,
+            0.016633259132504463,
+            0.01586349494755268,
+            -0.03358394652605057,
+            0.013173287734389305,
+            -0.026743359863758087,
+            -0.010165651328861713,
+            0.01029262226074934,
+            0.04564623534679413,
+            -0.04551926255226135,
+            -0.011030644178390503,
+            -0.012752694077789783,
+            -0.0072770509868860245,
+            -0.03983728960156441,
+            -0.013578008860349655,
+            -0.0009036795818246901,
+            -0.015046115964651108,
+            -0.015276252292096615,
+            0.012951087206602097,
+            -0.014054151251912117,
+            -0.047550804913043976,
+            -0.016839588060975075,
+            0.014530294574797153,
+            -0.023505588993430138,
+            0.014696944504976273,
+            0.02780674584209919,
+            0.01058624405413866,
+            -0.020664602518081665,
+            0.0055311936885118484,
+            0.01597459428012371,
+            0.008673736825585365,
+            0.0010921527864411473,
+            0.016014274209737778,
+            2.0304270947235636e-05,
+            0.03221900388598442,
+            -0.0020751894917339087,
+            -0.010633858852088451,
+            -0.0009309586021117866,
+            0.03212377429008484,
+            -0.015807945281267166,
+            -0.016299959272146225,
+            0.014863594435155392,
+            -0.02042653039097786,
+            -0.06027969345450401,
+            -0.007530993781983852,
+            -0.027775002643465996,
+            0.00897529348731041,
+            -0.011641694232821465,
+            -0.020029745995998383,
+            -0.02758454531431198,
+            0.0034659237135201693,
+            -0.012705080211162567,
+            0.0013252644566819072,
+            0.029028845950961113,
+            0.022394588217139244,
+            0.011038579978048801,
+            0.015585744753479958,
+            -0.001685347524471581,
+            0.0027318699285387993,
+            -0.055264320224523544,
+            0.03237771987915039,
+            -0.04529706388711929,
+            0.01939488761126995,
+            0.007729386445134878,
+            -0.010848122648894787,
+            0.02053763158619404,
+            0.027013175189495087,
+            -0.0009607175015844405,
+            0.0074794115498661995,
+            0.019410759210586548,
+            0.04355120658874512,
+            -0.015800008550286293,
+            -0.05570872128009796,
+            -0.0387897752225399,
+            -0.0019601215608417988,
+            -0.011974994093179703,
+            -0.00724927568808198,
+            0.01158614456653595,
+            -0.024426132440567017,
+            -0.004781268537044525,
+            -0.009602215141057968,
+            0.006662033032625914,
+            0.018442602828145027,
+            0.0009369103936478496,
+            0.009308594278991222,
+            -0.020934415981173515,
+            -0.003948018420487642,
+            -0.030742960050702095,
+            0.008443601429462433,
+            0.021521659567952156,
+            -0.00889593642205,
+            0.00948318000882864,
+            -0.002079157391563058,
+            -0.030362047255039215,
+            0.010792572982609272,
+            0.02791784703731537,
+            0.030187461525201797,
+            -0.01660151593387127,
+            0.021902574226260185,
+            -0.010324365459382534,
+            -0.008261079899966717,
+            -0.007530993781983852,
+            0.029647832736372948,
+            0.0678662359714508,
+            0.022029545158147812,
+            0.007546865381300449,
+            -0.02063285931944847,
+            0.0073683117516338825,
+            -0.02155340276658535,
+            0.012657465413212776,
+            -0.028171788901090622,
+            0.03431403264403343,
+            0.03621860593557358,
+            0.012800308875739574,
+            -0.0014046216383576393,
+            0.017283987253904343,
+            -0.029568474739789963,
+            -0.020823316648602486,
+            0.03093341737985611,
+            -0.03282211720943451,
+            -0.029870031401515007,
+            0.014101766049861908,
+            -0.002797339577227831,
+            0.010943351313471794,
+            0.016204729676246643,
+            -0.02920343168079853,
+            -0.042979832738637924,
+            -0.002227952005341649,
+            -0.031155617907643318,
+            -0.02052175998687744,
+            -0.012720951810479164,
+            -0.0625016912817955,
+            -0.002642593113705516,
+            -0.046630263328552246,
+            -0.014720751903951168,
+            0.013133608736097813,
+            -0.008003168739378452,
+            0.011482980102300644,
+            -0.03517109155654907,
+            0.005483579356223345,
+            -0.019966259598731995,
+            -0.026632260531187057,
+            0.015188959427177906,
+            0.045233577489852905,
+            -0.004166250582784414,
+            -0.0059676580131053925,
+            0.027187760919332504,
+            -0.039361149072647095,
+            -0.01750618778169155,
+            0.009784736670553684,
+            0.00778493657708168,
+            -0.014641394838690758,
+            -0.04009123519062996,
+            -0.021839087828993797,
+            0.014244608581066132,
+            0.019315531477332115,
+            0.021632758900523186,
+            -0.004257511347532272,
+            -0.04989977926015854,
+            -0.03520283102989197,
+            -0.030250946059823036,
+            0.039361149072647095,
+            0.0005202853353694081,
+            0.004436064977198839,
+            -0.023616688326001167,
+            -0.009062586352229118,
+            0.004531293176114559,
+            0.022712016478180885,
+            0.009253043681383133,
+            -0.028203532099723816,
+            -0.0049677579663693905,
+            0.0006284094415605068,
+            0.005114568397402763,
+            -0.030139846727252007,
+            0.025441903620958328,
+            0.0013649429893121123,
+            -0.05961309373378754,
+            0.008149979636073112,
+            -0.02926691807806492,
+            0.007344504352658987,
+            0.01409383025020361,
+            -0.010149779729545116,
+            -0.03093341737985611,
+            0.01676023006439209,
+            0.04085306078195572,
+            0.020759830251336098,
+            -0.026759231463074684,
+            0.012713016010820866,
+            0.012927279807627201,
+            -0.030917545780539513,
+            -0.04104351997375488,
+            -0.002900503808632493,
+            0.004801107570528984,
+            -0.009713315404951572,
+            -0.011221101507544518,
+            -0.002594978781417012,
+            0.03475843369960785,
+            -0.009689508005976677,
+            0.0036841558758169413,
+            0.022712016478180885,
+            0.021854959428310394,
+            -0.021680373698472977,
+            0.014577909372746944,
+            -0.01153059396892786,
+            0.033266518265008926,
+            -0.027029046788811684,
+            0.03593291714787483,
+            0.009118136949837208,
+            -0.0022636628709733486,
+            0.031441304832696915,
+            -0.00964189413934946,
+            -0.02042653039097786,
+            0.02321990206837654,
+            -0.0007915875758044422,
+            -0.02026781626045704,
+            -0.029489118605852127,
+            -0.01058624405413866,
+            -0.0069159758277237415,
+            0.026632260531187057,
+            -0.022267617285251617,
+            -0.023315131664276123,
+            0.01298283040523529,
+            0.042027547955513,
+            0.009300658479332924,
+            0.026600517332553864,
+            -0.0004915183526463807,
+            -0.00017086586740333587,
+            0.04964583367109299,
+            -0.03659952059388161,
+            0.02312467433512211,
+            0.018823517486453056,
+            -0.021743860095739365,
+            -0.01182421576231718,
+            -0.0014056136133149266,
+            0.011236973106861115,
+            -0.010284687392413616,
+            -0.006328732706606388,
+            0.003662332659587264,
+            0.005570872221142054,
+            0.0020394788589328527,
+            -0.02112487331032753,
+            0.010252944193780422,
+            0.0006199777708388865,
+            0.005812911316752434,
+            0.017204631119966507,
+            -0.02229936048388481,
+            -0.026473546400666237,
+            0.025394288823008537,
+            0.027092531323432922,
+            -0.010197394527494907,
+            0.014379516243934631,
+            -0.0027001271955668926,
+            -0.01072115171700716,
+            0.01644280180335045,
+            -0.005150279030203819,
+            0.002200176939368248,
+            0.0002386914420640096,
+            0.02650528959929943,
+            -0.022013673558831215,
+            -0.017220502719283104,
+            0.015871429815888405,
+            0.016617387533187866,
+            -0.031758733093738556,
+            0.005511354189366102,
+            0.01745857298374176,
+            0.014030344784259796,
+            -0.04240846261382103,
+            -0.0076976437121629715,
+            -0.023854760453104973,
+            -0.00869754422456026,
+            -0.01462552323937416,
+            -0.03939289227128029,
+            0.03291734680533409,
+            -0.003497666446492076,
+            -0.026917945593595505,
+            -0.010975094512104988,
+            -0.032361846417188644,
+            0.011514722369611263,
+            0.014435065910220146,
+            0.025346674025058746,
+            -0.073262520134449,
+            0.011348072439432144,
+            -0.010729086585342884,
+            0.01441125851124525,
+            -0.007975393906235695,
+            -0.008903872221708298,
+            0.02510860376060009,
+            -0.00015301050734706223,
+            0.021854959428310394,
+            0.009260979481041431,
+            0.0016089662676677108,
+            0.04285286366939545,
+            0.008134108036756516,
+            0.028140045702457428,
+            -0.016823716461658478,
+            0.038123175501823425,
+            0.013276451267302036,
+            0.0025275254156440496,
+            -0.00956253707408905,
+            -0.0061700185760855675,
+            0.03980554640293121,
+            0.06275563687086105,
+            0.00560261495411396,
+            0.03374266251921654,
+            0.0013659349642693996,
+            0.014482680708169937,
+            0.004221800714731216,
+            -0.01107825804501772,
+            -0.02980654686689377,
+            0.0040452308021485806,
+            -0.030917545780539513,
+            -0.01955360174179077,
+            -0.011752794496715069,
+            -0.002979861106723547,
+            -0.021410560235381126,
+            -0.0062454077415168285,
+            0.010768765583634377,
+            -0.01018152292817831,
+            0.044693946838378906,
+            0.009816479869186878,
+            -0.02215651609003544,
+            -0.0037297862581908703,
+            0.061136748641729355,
+            0.0048249149695038795,
+            0.03466320410370827,
+            -0.014593780040740967,
+            0.017125273123383522,
+            -0.00552325788885355,
+            -0.0011695260182023048,
+            0.006328732706606388,
+            -0.010729086585342884,
+            -0.025045117363333702,
+            0.03698043152689934,
+            0.002549348631873727,
+            0.033647432923316956,
+            0.011117937043309212,
+            -0.025362545624375343,
+            0.009554601274430752,
+            -0.012617787346243858,
+            0.015688909217715263,
+            0.01417318731546402,
+            0.0067136152647435665,
+            -0.011363944038748741,
+            0.008657865226268768,
+            -0.025299061089754105,
+            0.015300058759748936,
+            -0.0014631474623456597,
+            0.0001887212274596095,
+            0.02306118793785572,
+            -0.017093529924750328,
+            0.0060589187778532505,
+            -0.01376846618950367,
+            -0.0280130747705698,
+            0.015696844086050987,
+            0.024616587907075882,
+            0.021204231306910515,
+            -0.023077059537172318,
+            -0.011578208766877651,
+            0.04355120658874512,
+            0.013943051919341087,
+            -0.002037494909018278,
+            -0.006757261697202921,
+            -0.0017160983989015222,
+            -0.008221400901675224,
+            -0.01087986584752798,
+            -0.012554301880300045,
+            -0.006435865070670843,
+            -0.011998801492154598,
+            0.0490427203476429,
+            -0.014903273433446884,
+            0.00878483708947897,
+            0.011038579978048801,
+            -0.015276252292096615,
+            -0.002704094862565398,
+            0.0030215235892683268,
+            -0.007769065443426371,
+            -0.0056224544532597065,
+            -0.0039757932536304,
+            0.030489018186926842,
+            0.016506288200616837,
+            0.024600718170404434,
+            -0.037900976836681366,
+            -0.010705280117690563,
+            0.002364843152463436,
+            0.02166450209915638,
+            0.021029645577073097,
+            0.005872429348528385,
+            -0.0029600218404084444,
+            -0.0015623439103364944,
+            -0.02888600341975689,
+            -0.015736524015665054,
+            -0.017093529924750328,
+            0.03459971770644188,
+            -0.025394288823008537,
+            0.03313954547047615,
+            0.02123597450554371,
+            0.008530894294381142,
+            0.0052812183275818825,
+            -0.01511753723025322,
+            -0.034726690500974655,
+            0.015379415825009346,
+            -0.01519689429551363,
+            0.0006452728412114084,
+            0.0290923323482275,
+            -0.04374166205525398,
+            -0.04567797854542732,
+            -0.003692091442644596,
+            0.016665002331137657,
+            0.024553103372454643,
+            0.001200276892632246,
+            0.0011070322943851352,
+            0.012625723145902157,
+            -0.012157515622675419,
+            -0.011808344163000584,
+            0.004654297139495611,
+            0.01880764588713646,
+            -0.005860525649040937,
+            -0.0061501795426011086,
+            0.025981532409787178,
+            0.00932446587830782,
+            -0.03926591947674751,
+            -0.002416425384581089,
+            0.006193825975060463,
+            0.022553302347660065,
+            0.02301357500255108,
+            -0.03142543137073517,
+            0.015855558216571808,
+            0.013189158402383327,
+            0.04459872096776962,
+            -0.012990766204893589,
+            0.008388050831854343,
+            0.038662806153297424,
+            -0.005558968521654606,
+            0.03082231804728508,
+            -0.004634457640349865,
+            -0.03345697373151779,
+            0.02328338846564293,
+            0.0011100082192569971,
+            -0.01831563003361225,
+            -0.012411458417773247,
+            0.007070722058415413,
+            0.004777300637215376,
+            0.027886103838682175,
+            0.025457775220274925,
+            0.0033230807166546583,
+            0.028965361416339874,
+            0.038885004818439484,
+            0.0057176826521754265,
+            0.013236773200333118,
+            0.007237372454255819,
+            -0.015069923363626003,
+            -0.00490427203476429,
+            -0.006729486398398876,
+            0.015323866158723831,
+            -0.04002774879336357,
+            0.020061487331986427,
+            -0.003616702277213335,
+            0.01982341706752777,
+            -0.00529312202706933,
+            -0.012435265816748142,
+            -0.0007657965179532766,
+            -0.02220413088798523,
+            0.02391824498772621,
+            0.0025989466812461615,
+            0.001375854597426951,
+            -0.021077260375022888,
+            -0.024426132440567017,
+            0.00475746113806963,
+            -0.02328338846564293,
+            0.011260779574513435,
+            0.026489417999982834,
+            0.010356108658015728,
+            0.003874612972140312,
+            -0.03713914752006531,
+            0.02123597450554371,
+            0.02979067526757717,
+            -0.0021386751905083656,
+            0.017125273123383522,
+            0.030044617131352425,
+            -0.021886702626943588,
+            -0.015942851081490517,
+            0.015934916213154793,
+            -0.02715601772069931,
+            -0.0019750010687857866,
+            -0.0002596466802060604,
+            -0.05082032084465027,
+            -0.04856657609343529,
+            0.013125672936439514,
+            -0.003858741605654359,
+            0.015379415825009346,
+            -0.010483079589903355,
+            -0.005793072283267975,
+            0.004912207834422588,
+            -0.0005872429464943707,
+            0.006527125835418701,
+            -0.0059676580131053925,
+            0.0056264218874275684,
+            -0.013498651795089245,
+            -0.008515022695064545,
+            -0.02790197543799877,
+            -0.005308993626385927,
+            -0.018712416291236877,
+            -0.013514523394405842,
+            -0.027441702783107758,
+            0.020172588527202606,
+            -0.00964189413934946,
+            -0.012419394217431545,
+            0.019871031865477562,
+            0.013101865537464619,
+            0.020347174257040024,
+            0.010197394527494907,
+            -0.003904371988028288,
+            0.028092432767152786,
+            -0.03409183397889137,
+            0.03615511953830719,
+            0.006328732706606388,
+            0.02677510306239128,
+            -0.002079157391563058,
+            -0.020680474117398262,
+            0.009673637337982655,
+            0.03437751904129982,
+            0.0016109502175822854,
+            -0.013808144256472588,
+            -0.015260380692780018,
+            -0.004376546945422888,
+            -0.005059018265455961,
+            -0.007955554872751236,
+            -0.008872129954397678,
+            0.010744958184659481,
+            -0.0036841558758169413,
+            0.001297489390708506,
+            -0.006884233094751835,
+            0.04193231835961342,
+            0.011713115498423576,
+            -0.004539228975772858,
+            -0.023267516866326332,
+            -0.02758454531431198,
+            0.01999800279736519,
+            -0.00280130747705698,
+            -0.024616587907075882,
+            0.026378316804766655,
+            -0.014054151251912117,
+            -0.004654297139495611,
+            0.00723340455442667,
+            -0.018077559769153595,
+            0.013816080056130886,
+            -0.021362945437431335,
+            0.005979561246931553,
+            -0.041837092489004135,
+            -0.011959122493863106,
+            -0.03253643214702606,
+            -0.005856557749211788,
+            0.0177760012447834,
+            -0.007447668816894293,
+            0.02441026084125042,
+            -0.016728486865758896,
+            -0.0015425046440213919,
+            -0.007332601118832827,
+            -0.013887501321732998,
+            -0.0088880006223917,
+            -0.0050947293639183044,
+            -0.005245507694780827,
+            0.0191250741481781,
+            -0.002251759171485901,
+            -0.02144230343401432,
+            0.027870232239365578,
+            -0.024648331105709076,
+            1.071011593012372e-05,
+            -0.04659852012991905,
+            -0.03640906140208244,
+            -0.04085306078195572,
+            0.02375953085720539,
+            -0.0169665589928627,
+            -0.04942363500595093,
+            -0.009657765738666058,
+            0.00434877211228013,
+            0.0159031730145216,
+            -0.02490227483212948,
+            -0.02434677444398403,
+            0.022220002487301826,
+            -0.011086193844676018,
+            0.0034659237135201693,
+            -0.01642693020403385,
+            -0.011760730296373367,
+            -0.017157016322016716,
+            0.04091654717922211,
+            -0.02348971739411354,
+            0.022870730608701706,
+            -0.01492708083242178,
+            0.007296890020370483,
+            -0.007273083087056875,
+            -0.015450838021934032,
+            0.03383788838982582,
+            0.025299061089754105,
+            -0.04713814705610275,
+            0.04018646106123924,
+            0.020982030779123306,
+            0.03086993284523487,
+            -0.04459872096776962,
+            0.016617387533187866,
+            -0.030187461525201797,
+            8.307702228194103e-05,
+            0.04063086211681366,
+            0.004412257578223944,
+            0.009856157936155796,
+            -0.011498851701617241,
+            -0.020172588527202606,
+            0.003795255906879902,
+            0.010221200995147228,
+            0.000911119335796684,
+            -0.02791784703731537,
+            -0.007130240090191364,
+            0.007455604616552591,
+            0.02080744504928589,
+            0.00272790202870965,
+            0.02537841722369194,
+            0.03475843369960785,
+            -0.014935015700757504,
+            0.006967558059841394,
+            -0.0458049476146698,
+            0.011903572827577591,
+            -0.005582775454968214,
+            0.01741096004843712,
+            0.025314930826425552,
+            0.02214064635336399,
+            0.030044617131352425,
+            -0.03748831897974014,
+            0.024648331105709076,
+            0.0037516094744205475,
+            -0.005562936421483755,
+            0.0323459766805172,
+            0.0088880006223917,
+            0.03183808922767639,
+            -0.015355609357357025,
+            -0.011276651173830032,
+            0.007273083087056875,
+            0.02375953085720539,
+            -0.017442701384425163,
+            0.01669674552977085,
+            -0.005305025726556778,
+            0.04945537820458412,
+            0.013831951655447483,
+            -0.03488540276885033,
+            -0.027298860251903534,
+            -0.020918546244502068,
+            -0.00918955821543932,
+            0.012863794341683388,
+            0.01112587284296751,
+            -0.01129252277314663,
+            -0.0312667191028595,
+            0.02382301725447178,
+            -0.010998900979757309,
+            -0.014530294574797153,
+            -0.04132920503616333,
+            -0.007054850924760103,
+            -0.017823616042733192,
+            0.0024322967510670424,
+            -0.003579007461667061,
+            -0.01676023006439209,
+            -0.0052970899268984795,
+            0.006427929271012545,
+            0.025314930826425552,
+            -0.025743460282683372,
+            -0.008935615420341492,
+            -0.038662806153297424,
+            -0.005039179231971502,
+            -0.0023509557358920574,
+            0.01729985885322094,
+            -0.002940182574093342,
+            -0.038726288825273514,
+            0.019490117207169533,
+            0.010649729520082474,
+            -0.01433190144598484,
+            0.043932121247053146,
+            -0.036345575004816055,
+            0.005872429348528385,
+            0.03215551748871803,
+            -0.00173097790684551,
+            -0.03086993284523487,
+            -0.0017270100070163608,
+            0.007273083087056875,
+            0.0021049485076218843,
+            0.002737821778282523,
+            0.018077559769153595,
+            0.031123874709010124,
+            0.01276856567710638,
+            0.012633658945560455,
+            0.06977080553770065,
+            -0.00043224848923273385,
+            -0.03086993284523487,
+            0.02285485900938511,
+            -0.001691299257799983,
+            -0.01067353691905737,
+            -0.01376846618950367,
+            0.023045316338539124,
+            -0.02139468863606453,
+            0.011522658169269562,
+            -0.019109202548861504,
+            -0.0004731670196633786,
+            -0.0344727486371994,
+            -0.030108103528618813,
+            -0.01239558681845665,
+            0.022029545158147812,
+            -0.04104351997375488,
+            -0.029822418466210365,
+            0.018283888697624207,
+            0.017712516710162163,
+            -0.004582875408232212,
+            0.0018589412793517113,
+            -0.001100088469684124,
+            0.009340336546301842,
+            0.01761728711426258,
+            -0.0020989966578781605,
+            -0.007519090082496405,
+            0.009872029535472393,
+            -0.0074952831491827965,
+            -0.0066064829006791115,
+            0.02123597450554371,
+            -0.07034218311309814,
+            -0.04409083351492882,
+            0.024045217782258987,
+            -0.008626122027635574,
+            0.002261678921058774,
+            0.022886602208018303,
+            -6.004484384902753e-05,
+            -0.01525244489312172,
+            -0.016887202858924866,
+            0.03463146090507507,
+            -0.012101965956389904,
+            0.022759631276130676,
+            -0.0015266332775354385,
+            -0.0020989966578781605,
+            0.01314948033541441,
+            0.011879765428602695,
+            -0.014736623503267765,
+            0.0018877083202823997,
+            0.015284187160432339,
+            0.014760429970920086,
+            0.00878483708947897,
+            0.017283987253904343,
+            -0.008967358618974686,
+            -0.004094828851521015,
+            -0.0009175670566037297,
+            0.001588135026395321,
+            -0.0019204431446269155,
+            0.012839986942708492,
+            0.031980931758880615,
+            0.0013272484065964818,
+            0.03498063236474991,
+            0.029060589149594307,
+            0.02085505984723568,
+            0.0036960593424737453,
+            -0.004614618606865406,
+            -0.009451436810195446,
+            0.023584945127367973,
+            0.016585644334554672,
+            0.010054551064968109,
+            0.0099672582000494,
+            0.005459771957248449,
+            -0.022505687549710274,
+            0.006971525959670544,
+            -0.0042813182808458805,
+            -0.008380115032196045,
+            0.0071500795893371105,
+            0.007320697419345379,
+            -0.018775902688503265,
+            0.012800308875739574,
+            0.013117737136781216,
+            0.0022041448391973972,
+            0.02229936048388481,
+            -0.01833150163292885,
+            -0.012697144411504269,
+            0.004618586506694555,
+            -0.012705080211162567,
+            0.025679973885416985,
+            -0.0030631860718131065,
+            0.0019531778525561094,
+            0.011387751437723637,
+            0.07459572702646255,
+            -0.026854461058974266,
+            -0.0014829868450760841,
+            -0.015522259287536144,
+            -0.02026781626045704,
+            0.0090229082852602,
+            -0.023156417533755302,
+            0.022553302347660065,
+            0.01822040230035782,
+            -0.004110700450837612,
+            0.04574146121740341,
+            0.0020712215919047594,
+            0.019966259598731995,
+            0.024315031245350838,
+            -0.009197494015097618,
+            -0.02053763158619404,
+            0.0045908112078905106,
+            0.012617787346243858,
+            0.00024241130449809134,
+            0.010252944193780422,
+            0.023902375251054764,
+            0.013943051919341087,
+            -0.003702011192217469,
+            -0.003840886289253831,
+            -0.021299459040164948,
+            0.025949789211153984,
+            0.010078358463943005,
+            -0.025124475359916687,
+            -0.038599319756031036,
+            0.0003905859775841236,
+            0.01239558681845665,
+            0.01236384455114603,
+            0.023632559925317764,
+            0.0033786308486014605,
+            -0.011554401367902756,
+            0.012379716150462627,
+            -0.014101766049861908,
+            0.003928178921341896,
+            0.0042813182808458805,
+            -0.007185790222138166,
+            -0.010245008394122124,
+            0.011348072439432144,
+            -0.004729686304926872,
+            0.003174286102876067,
+            0.00020422067609615624,
+            0.0031484949868172407,
+            -0.010038679465651512,
+            -0.010602115653455257,
+            -0.012276551686227322,
+            -0.0029540699906647205,
+            -0.005678004119545221,
+            0.032409459352493286,
+            -0.013109801337122917,
+            -0.003598846960812807,
+            -0.01166550163179636,
+            0.030250946059823036,
+            -0.001897627953439951,
+            -0.006412058137357235,
+            -0.018553702160716057,
+            0.004301157779991627,
+            -0.000162558164447546,
+            -0.009760930202901363,
+            -0.011117937043309212,
+            0.006979461759328842,
+            -0.01998213119804859,
+            -0.008459472097456455,
+            -0.0019710331689566374,
+            0.009173686616122723,
+            0.0026842555962502956,
+            -0.014958823099732399,
+            -0.014149379916489124,
+            0.021966060623526573,
+            -0.011562337167561054,
+            -0.014244608581066132,
+            -0.0010862010531127453,
+            -0.013490715995430946,
+            0.013800208456814289,
+            -0.007900004275143147,
+            -0.0008501135162077844,
+            -0.02144230343401432,
+            0.021093131974339485,
+            0.02333100326359272,
+            -0.017109401524066925,
+            -0.010959222912788391,
+            -0.031060390174388885,
+            -0.0005088777397759259,
+            0.0037278023082762957,
+            0.009443501010537148,
+            0.03628208860754967,
+            0.013744658790528774,
+            0.004245607648044825,
+            0.011459172703325748,
+            -0.007487347349524498,
+            0.008142043836414814,
+            0.009721251204609871,
+            -0.010014872997999191,
+            0.0011248876107856631,
+            2.0846751795033924e-05,
+            -0.03409183397889137,
+            -0.007054850924760103,
+            -0.036186862736940384,
+            -0.0048606256023049355,
+            0.005896236281841993,
+            -0.009816479869186878,
+            -0.0013966858386993408,
+            0.007975393906235695,
+            -0.006138275843113661,
+            -0.015482580289244652,
+            0.000179917537025176,
+            0.008348372764885426,
+            -0.008300757966935635,
+            -0.0010207314044237137,
+            -0.008792772889137268,
+            0.016299959272146225,
+            0.022870730608701706,
+            0.02295008860528469,
+            0.007828583009541035,
+            0.0005460763932205737,
+            -0.004082925617694855,
+            0.007693675812333822,
+            -0.01360181625932455,
+            -0.013466908596456051,
+            0.004305125679820776,
+            0.00950698647648096,
+            -0.0055311936885118484,
+            0.007423861417919397,
+            0.013490715995430946,
+            -0.0029421665240079165,
+            0.018093431368470192,
+            0.00622160080820322,
+            -0.004051182419061661,
+            0.008824515156447887,
+            -0.004019439686089754,
+            0.005578808020800352,
+            -0.008300757966935635,
+            -0.004253543447703123,
+            0.007943650707602501,
+            0.004217832814902067,
+            0.0149508873000741,
+            -0.0032238843850791454,
+            -0.020601116120815277,
+            -0.0033369683660566807,
+            0.04586843401193619,
+            -0.00028419779846444726,
+            0.013958923518657684,
+            -0.020680474117398262,
+            -0.013316130265593529,
+            0.01080050878226757,
+            0.014030344784259796,
+            0.010919543914496899,
+            -0.015125473029911518,
+            -0.016680873930454254,
+            0.02360081672668457,
+            0.00435274001210928,
+            -0.005451836157590151,
+            -0.035329803824424744,
+            0.0036782040260732174,
+            -0.03786923363804817,
+            0.02215651609003544,
+            0.017855359241366386,
+            -0.0035036182962358,
+            -0.004701911471784115,
+            0.017474444583058357,
+            -0.013197094202041626,
+            0.020299559459090233,
+            -0.01637931540608406,
+            0.019632959738373756,
+            0.0002527029428165406,
+            -0.006864393595606089,
+            -0.01676023006439209,
+            -0.02053763158619404,
+            0.004059118218719959,
+            -0.013578008860349655,
+            0.04326551780104637,
+            0.024981630966067314,
+            0.0067334542982280254,
+            0.0017498251982033253,
+            -0.027790874242782593,
+            -0.013141544535756111,
+            0.00934827234596014,
+            0.019109202548861504,
+            -0.010943351313471794,
+            -0.01592698134481907,
+            -0.00505108293145895,
+            0.013895437121391296,
+            -0.0049518863670527935,
+            -0.025886302813887596,
+            0.005253443494439125,
+            -0.014784237369894981,
+            0.013427230529487133,
+            -0.010919543914496899,
+            0.008118236437439919,
+            0.016014274209737778,
+            -0.00552325788885355,
+            -0.0014839787036180496,
+            0.03517109155654907,
+            0.012308293953537941,
+            -0.01322883740067482,
+            0.006781068630516529,
+            -0.018791774287819862,
+            -0.01880764588713646,
+            0.03250468894839287,
+            -0.01503024436533451,
+            0.01988690160214901,
+            0.008991165086627007,
+            -0.0242515467107296,
+            0.017109401524066925,
+            -0.021696245297789574,
+            0.008610251359641552,
+            -0.008681672625243664,
+            0.0042813182808458805,
+            -0.016728486865758896,
+            -0.02268027327954769,
+            -0.01619679480791092,
+            -0.02526731789112091,
+            0.01734747365117073,
+            -0.013125672936439514,
+            -0.0023588913027197123,
+            -0.013514523394405842,
+            0.025584746152162552,
+            0.0040392791852355,
+            0.024203931912779808,
+            0.01622060127556324,
+            0.003977777436375618,
+            0.00544390082359314,
+            -0.033171288669109344,
+            0.028520960360765457,
+            0.002400553785264492,
+            0.020616987720131874,
+            0.013919244520366192,
+            0.006296989973634481,
+            0.0019809529185295105,
+            0.007892068475484848,
+            0.031108003109693527,
+            0.004602714907377958,
+            0.0011982930591329932,
+            0.03158414736390114,
+            -0.00918955821543932,
+            0.013665301725268364,
+            -2.7558011424844153e-05,
+            0.00886419415473938,
+            -0.003112784354016185,
+            -0.009538729675114155,
+            0.02866380289196968,
+            0.0010326349874958396,
+            -0.00019132513261865824,
+            0.02355320379137993,
+            0.0034500521142035723,
+            -0.04018646106123924,
+            -0.003662332659587264,
+            -0.020950287580490112,
+            -0.016522159799933434,
+            0.014554101973772049,
+            0.005908139981329441,
+            0.002900503808632493,
+            0.021966060623526573,
+            0.0004959821817465127,
+            0.01075289398431778,
+            -0.008269015699625015,
+            -0.016617387533187866,
+            -0.021839087828993797,
+            -0.0036881237756460905,
+            0.02839398942887783,
+            0.011308394372463226,
+            -0.011268715374171734,
+            0.015712715685367584,
+            -0.006050982978194952,
+            -0.00435274001210928,
+            0.007110401056706905,
+            -0.0006695759948343039,
+            -0.010951287113130093,
+            0.0014512438792735338,
+            0.0062454077415168285,
+            -0.02175973169505596,
+            0.005872429348528385,
+            -0.0076222545467317104,
+            -0.0003261083038523793,
+            -0.008459472097456455,
+            -0.020013874396681786,
+            -0.011165550909936428,
+            -0.0035809914115816355,
+            0.03685346245765686,
+            0.013347873464226723,
+            -0.020601116120815277,
+            -0.014323965646326542,
+            0.0003261083038523793,
+            -0.0031385754700750113,
+            -0.0008193625835701823,
+            -0.0020930448081344366,
+            -0.0008977277902886271,
+            -0.014569973573088646,
+            0.006265247240662575,
+            -0.0006269214791245759,
+            0.010871930047869682,
+            0.011776601895689964,
+            0.007189758121967316,
+            -0.01606188714504242,
+            -0.0005068937898613513,
+            -0.02434677444398403,
+            -0.023077059537172318,
+            -0.004110700450837612,
+            -0.0026505289133638144,
+            -0.017490316182374954,
+            -0.0022894537542015314,
+            0.014601715840399265,
+            -0.0138795655220747,
+            0.029774803668260574,
+            -0.0006576724117621779,
+            -0.002549348631873727,
+            0.044757433235645294,
+            0.012998702004551888,
+            -0.00030775697086937726,
+            -0.0021724021062254906,
+            0.01917268894612789,
+            -0.005797040183097124,
+            -0.0029203433077782393,
+            0.005642293486744165,
+            0.006047015078365803,
+            0.013704979792237282,
+            0.026584645733237267,
+            -0.02090267464518547,
+            0.0024203930515795946,
+            -0.006047015078365803,
+            0.015387351624667645,
+            -0.014244608581066132,
+            0.0002603906614240259,
+            -0.02155340276658535,
+            -0.01613330841064453,
+            -0.017696645110845566,
+            0.006626322399824858,
+            -0.0072770509868860245,
+            -0.010467207990586758,
+            -0.0016526127001270652,
+            -0.012173387221992016,
+            0.012776501476764679,
+            0.020870931446552277,
+            -0.02936214581131935,
+            0.0199345164000988,
+            -0.006419993471354246,
+            0.0149508873000741,
+            -0.00755876861512661,
+            0.015149280428886414,
+            -0.0020315430592745543,
+            -0.013212965801358223,
+            0.006820747163146734,
+            -0.03752006217837334,
+            -0.017633158713579178,
+            -0.0064953831024467945,
+            0.002751709194853902,
+            -0.03152066096663475,
+            -0.0048764972016215324,
+            0.030901676043868065,
+            0.0035809914115816355,
+            -0.012578108347952366,
+            0.002473959233611822,
+            0.025299061089754105,
+            0.002876696875318885,
+            0.015268316492438316,
+            0.002487846650183201,
+            -0.016887202858924866,
+            -0.004340836312621832,
+            0.02521970309317112,
+            -0.011848023161292076,
+            -0.01026881579309702,
+            -0.024807045236229897,
+            0.03605988994240761,
+            -0.008872129954397678,
+            -0.00140065373852849,
+            0.023949988186359406,
+            -0.006562836468219757,
+            0.005705779418349266,
+            0.0021803376730531454,
+            -0.020553503185510635,
+            0.0231722891330719,
+            0.029520859941840172,
+            0.004197993315756321,
+            -0.02936214581131935,
+            -0.004622553940862417,
+            -0.003910323604941368,
+            0.003116752253845334,
+            0.00015648239059373736,
+            0.002900503808632493,
+            0.002198193222284317,
+            -0.009864093735814095,
+            -0.02079157344996929,
+            -0.00028742168797180057,
+            -0.015720652416348457,
+            -0.010903672315180302,
+            0.018553702160716057,
+            -0.006102565210312605,
+            0.015577808953821659,
+            0.01376846618950367,
+            0.009610150940716267,
+            0.0057494258508086205,
+            0.020886803045868874,
+            0.02893361821770668,
+            -0.014958823099732399,
+            0.00951492227613926,
+            -0.017490316182374954,
+            0.01245113741606474,
+            0.007070722058415413,
+            -0.006669968832284212,
+            -0.03834537789225578,
+            0.008951487019658089,
+            0.007423861417919397,
+            0.014276351779699326,
+            -0.013395487330853939,
+            -0.024775303900241852,
+            -0.036885205656290054,
+            -0.016665002331137657,
+            0.003930163104087114,
+            0.005725618451833725,
+            0.005801007617264986,
+            -0.008522958494722843,
+            -0.005269315093755722,
+            -0.004471775609999895,
+            0.004209897015243769,
+            0.013435166329145432,
+            -0.049106206744909286,
+            -0.019490117207169533,
+            0.006205729208886623,
+            -0.01676023006439209,
+            0.016363445669412613,
+            0.007431797217577696,
+            -0.013236773200333118,
+            0.009737122803926468,
+            -0.016117436811327934,
+            0.000434232410043478,
+            -0.00926891528069973,
+            0.001266738516278565,
+            -0.004194025415927172,
+            0.008967358618974686,
+            0.019585344940423965,
+            0.034948889166116714,
+            -0.007753193844109774,
+            0.003061202121898532,
+            -0.008848322555422783,
+            0.0007211581105366349,
+            0.01029262226074934,
+            0.006332700606435537,
+            -0.005340736359357834,
+            -0.00027403017156757414,
+            -0.02015671692788601,
+            0.015450838021934032,
+            -0.020140845328569412,
+            0.00280130747705698,
+            0.00023273965052794665,
+            -0.0029143914580345154,
+            -0.0019859126769006252,
+            -0.008102364838123322,
+            -0.004928078968077898,
+            -0.007935714907944202,
+            -0.0019630976021289825,
+            0.02042653039097786,
+            -0.02187083102762699,
+            0.00217637000605464,
+            0.0018648931290954351,
+            0.004745557904243469,
+            -0.027870232239365578,
+            -0.0051859901286661625,
+            -0.009705379605293274,
+            -0.014157315716147423,
+            -0.023204030469059944,
+            0.026219602674245834,
+            0.016228538006544113,
+            0.007566704414784908,
+            0.004678104072809219,
+            0.0022954056039452553,
+            -0.0010653698118403554,
+            -0.013657365925610065,
+            0.003325064666569233,
+            -0.001118935877457261,
+            -0.0026683842297643423,
+            -0.0033111772499978542,
+            0.024743560701608658,
+            -0.014823916368186474,
+            0.008634057827293873,
+            -0.0014790189452469349,
+            0.0073683117516338825,
+            0.005900204181671143,
+            0.0121257733553648,
+            0.012030544690787792,
+            0.019966259598731995,
+            0.012117837555706501,
+            0.014593780040740967,
+            0.06742183864116669,
+            -0.0058248150162398815,
+            -0.0032457076013088226,
+            0.0031544468365609646,
+            -0.02080744504928589,
+            -0.01902984455227852,
+            0.004638425540179014,
+            -0.019791673868894577,
+            0.029504990205168724,
+            -0.00918955821543932,
+            0.009713315404951572,
+            0.005725618451833725,
+            -0.007066754624247551,
+            0.017982330173254013,
+            -0.002755677094683051,
+            -0.002307309303432703,
+            0.007927779108285904,
+            0.01438745204359293,
+            0.034028347581624985,
+            0.014681072905659676,
+            -0.016395187005400658,
+            0.0026842555962502956,
+            -0.02252155914902687,
+            0.0028588413260877132,
+            -0.011482980102300644,
+            0.005328832659870386,
+            -0.013744658790528774,
+            0.026584645733237267,
+            0.0011318314354866743,
+            0.02661638893187046,
+            -0.016395187005400658,
+            0.0015365529106929898,
+            -0.014522358775138855,
+            0.0007335576810874045,
+            0.015593680553138256,
+            0.017982330173254013,
+            -0.0038567576557397842,
+            0.006729486398398876,
+            -0.013657365925610065,
+            0.00803094357252121,
+            0.014189058914780617,
+            0.015014373697340488,
+            0.001624837750568986,
+            -0.011340136639773846,
+            -0.011895637027919292,
+            0.02215651609003544,
+            0.0004414241702761501,
+            0.006118436343967915,
+            0.012459073215723038,
+            -0.014474744908511639,
+            0.00011543985601747409,
+            -0.005269315093755722,
+            0.01599840261042118,
+            -0.008026976138353348,
+            0.004376546945422888,
+            -0.006118436343967915,
+            -0.019379016011953354,
+            -0.005495482590049505,
+            -0.02252155914902687,
+            0.01718875952064991,
+            0.00046721522812731564,
+            0.00801110453903675,
+            -0.021839087828993797,
+            -0.01739508844912052,
+            -0.006876297295093536,
+            0.015379415825009346,
+            2.8162488888483495e-05,
+            0.010919543914496899,
+            0.012086094357073307,
+            0.01042752992361784,
+            -0.007400054484605789,
+            0.0004890384734608233,
+            0.02215651609003544,
+            -0.010316429659724236,
+            0.010395786724984646,
+            0.00739608658477664,
+            -0.005435965023934841,
+            -0.032250747084617615,
+            -0.004301157779991627,
+            -0.006820747163146734,
+            0.020236073061823845,
+            0.002410473534837365,
+            -0.01492708083242178,
+            0.009768865071237087,
+            -0.0409482903778553,
+            -0.009451436810195446,
+            -0.0027140146121382713,
+            0.025092732161283493,
+            0.011165550909936428,
+            -0.0069159758277237415,
+            0.01950598880648613,
+            0.008753093890845776,
+            0.013117737136781216,
+            -0.0013619671808555722,
+            -0.007324665319174528,
+            0.0036206701770424843,
+            -0.002962005790323019,
+            0.0028588413260877132,
+            -0.009959322400391102,
+            0.002946134191006422,
+            0.015561937354505062,
+            0.004174186382442713,
+            -0.011411558836698532,
+            -0.005983529146760702,
+            -0.015363545157015324,
+            -0.016117436811327934,
+            0.0026862395461648703,
+            0.00037248263834044337,
+            0.011919444426894188,
+            -0.007903972640633583,
+            -0.004209897015243769,
+            -0.015887301415205002,
+            0.019632959738373756,
+            0.0012320197420194745,
+            0.0009061594610102475,
+            -0.00824520830065012,
+            0.003061202121898532,
+            -0.012657465413212776,
+            -0.007273083087056875,
+            0.008745158091187477,
+            -0.00021451229986269027,
+            -0.0080864941701293,
+            0.024600718170404434,
+            -0.02445787377655506,
+            0.0049677579663693905,
+            -0.010149779729545116,
+            -0.028108302503824234,
+            -0.00940382294356823,
+            -0.03282211720943451,
+            -0.013133608736097813,
+            0.0039202431216835976,
+            -0.011863894760608673,
+            -0.007820647209882736,
+            0.01556987315416336,
+            -0.008074590004980564,
+            -0.005023307632654905,
+            0.015982531011104584,
+            0.003579007461667061,
+            0.0160380806773901,
+            -0.005761329084634781,
+            0.0027715484611690044,
+            -0.01750618778169155,
+            -0.006003368645906448,
+            -0.01463345903903246,
+            0.009824415668845177,
+            0.0016952671576291323,
+            0.013078059069812298,
+            -0.017093529924750328,
+            0.02106138877570629,
+            -0.010006937198340893,
+            0.00474159000441432,
+            -0.03507586196064949,
+            -0.01231622975319624,
+            0.017172887921333313,
+            0.016165051609277725,
+            -0.006003368645906448,
+            -0.00700326869264245,
+            -0.020759830251336098,
+            -0.010744958184659481,
+            -0.014681072905659676,
+            -0.02753693237900734,
+            0.011903572827577591,
+            0.00038339424645528197,
+            0.021188359707593918,
+            0.004142443183809519,
+            0.030457274988293648,
+            0.013046315871179104,
+            0.019839288666844368,
+            -0.0034302128478884697,
+            0.0011020724195986986,
+            -0.0038785808719694614,
+            0.015561937354505062,
+            -0.005804975517094135,
+            -0.012292423285543919,
+            -0.005741490051150322,
+            0.0231722891330719,
+            -0.009308594278991222,
+            0.002333100186660886,
+            0.005908139981329441,
+            0.024553103372454643,
+            0.0038726290222257376,
+            0.01714114472270012,
+            0.004392418544739485,
+            -0.002789404010400176,
+            0.03647254779934883,
+            -0.0027021111454814672,
+            -0.021854959428310394,
+            -0.0009011996444314718,
+            0.011705179698765278,
+            -0.021950189024209976,
+            -0.008856258355081081,
+            0.001166550093330443,
+            -0.022172387689352036,
+            -0.015942851081490517,
+            -2.3326663722400554e-05,
+            -0.00801904033869505,
+            0.027330603450536728,
+            0.0055311936885118484,
+            -0.008594379760324955,
+            -0.016522159799933434,
+            0.010705280117690563,
+            0.025997404009103775,
+            -0.0008962398278526962,
+            0.015173087827861309,
+            -0.014776301570236683,
+            -0.01761728711426258,
+            -0.0054994504898786545,
+            0.013681173324584961,
+            0.004182121716439724,
+            -0.0047336542047560215,
+            -0.05977180600166321,
+            -0.01815691590309143,
+            0.006086693610996008,
+            0.002291437704116106,
+            -0.026394188404083252,
+            -0.008388050831854343,
+            -0.017823616042733192,
+            -0.008911808021366596,
+            0.0029639897402375937,
+            0.02285485900938511,
+            -0.008538829162716866,
+            0.011689308099448681,
+            0.019744059070944786,
+            -0.01745857298374176,
+            -0.0007796840509399772,
+            0.00395992211997509,
+            -0.019379016011953354,
+            -0.005431997124105692,
+            0.0030393789056688547,
+            -0.018188659101724625,
+            -0.005614518653601408,
+            -0.0011754778679460287,
+            -0.007602415047585964,
+            0.012046415358781815,
+            -0.017601415514945984,
+            0.022394588217139244,
+            0.004194025415927172,
+            -0.011681373231112957,
+            -0.017744259908795357,
+            -0.03169524669647217,
+            -0.007741290144622326,
+            -0.002170418156310916,
+            -0.008713414892554283,
+            0.011149680241942406,
+            0.019267916679382324,
+            0.013760530389845371,
+            0.006173986475914717,
+            0.002852889709174633,
+            0.004761429037898779,
+            -2.9216451366664842e-05,
+            0.0005525241722352803,
+            0.003467907663434744,
+            -0.008530894294381142,
+            -0.0003127167874481529,
+            0.0077889044769108295,
+            -0.013506587594747543,
+            -0.021537531167268753,
+            0.009467308409512043,
+            0.03605988994240761,
+            -0.011411558836698532,
+            0.0038865164387971163,
+            -0.005618486553430557,
+            0.007566704414784908,
+            0.006630290299654007,
+            0.03012397512793541,
+            0.01655390113592148,
+            -0.022220002487301826,
+            0.00940382294356823,
+            -0.010959222912788391,
+            0.01425254438072443,
+            -0.0047495258040726185,
+            0.01961708813905716,
+            0.016395187005400658,
+            0.01586349494755268,
+            0.008181722834706306,
+            0.004813011270016432,
+            -0.0101021658629179,
+            -0.0033210967667400837,
+            0.014879466034471989,
+            -0.00872928649187088,
+            -0.011236973106861115,
+            0.0004543196992017329,
+            0.006316829472780228,
+            -0.01276856567710638,
+            -0.0160380806773901,
+            0.0034202933311462402,
+            0.009364143945276737,
+            0.03304431959986687,
+            -0.0036543968599289656,
+            0.003364743199199438,
+            -0.00654696486890316,
+            -0.013355808332562447,
+            -0.02166450209915638,
+            -0.02133120223879814,
+            0.012347972951829433,
+            -0.00739608658477664,
+            -0.0011437350185588002,
+            -0.014363644644618034,
+            -0.020997902378439903,
+            0.0035809914115816355,
+            -0.01441125851124525,
+            -0.00034569960553199053,
+            -0.002469991333782673,
+            -0.0007295898394659162,
+            -0.0028747129254043102,
+            0.0038369183894246817,
+            0.02133120223879814,
+            -0.016299959272146225,
+            0.013141544535756111,
+            0.005614518653601408,
+            0.0039698416367173195,
+            0.02542603202164173,
+            0.007455604616552591,
+            0.009118136949837208,
+            -0.016569772735238075,
+            0.025251446291804314,
+            -0.027600416913628578,
+            -0.014617587439715862,
+            -0.010030743665993214,
+            0.021188359707593918,
+            -0.0049836291000247,
+            0.011030644178390503,
+            0.016665002331137657,
+            0.005384382791817188,
+            -0.001797439530491829,
+            0.026108503341674805,
+            0.02069634571671486,
+            -0.001594086759723723,
+            0.008681672625243664,
+            -0.0005287170060910285,
+            0.02274375967681408,
+            -0.005920043680816889,
+            -0.025156216695904732,
+            0.011522658169269562,
+            0.009332400746643543,
+            0.0051701185293495655,
+            -0.0014075974468141794,
+            -0.013887501321732998,
+            -0.012213066220283508,
+            -0.009499051608145237,
+            -0.0177918728441,
+            -0.019696446135640144,
+            -0.007566704414784908,
+            0.0013074091402813792,
+            -0.01107825804501772,
+            -0.0019333385862410069,
+            -0.01783948764204979,
+            -0.02312467433512211,
+            -0.016474545001983643,
+            0.027568675577640533,
+            -0.0001265994505956769,
+            0.01592698134481907,
+            0.004400353878736496,
+            0.006217632908374071,
+            -0.019490117207169533,
+            -0.005035211332142353,
+            0.023156417533755302,
+            0.01761728711426258,
+            -0.003640509443357587,
+            -0.003360775299370289,
+            0.021886702626943588,
+            0.0017250260571017861,
+            -0.007011204492300749,
+            -0.0017418895149603486,
+            -0.0010385867208242416,
+            0.012990766204893589,
+            0.01207022275775671,
+            -0.008126172237098217,
+            -0.030790574848651886,
+            0.007836518809199333,
+            0.0007940675131976604,
+            -0.010895737446844578,
+            0.012086094357073307,
+            -0.0037416897248476744,
+            -0.013800208456814289,
+            0.005447868723422289,
+            -0.024013474583625793,
+            0.00014643874601460993,
+            -0.005812911316752434,
+            0.008181722834706306,
+            -0.03834537789225578,
+            0.003967857453972101,
+            0.014157315716147423,
+            -0.004495582543313503,
+            -0.007765097543597221,
+            -0.01128458697348833,
+            -0.021045517176389694,
+            0.002924310974776745,
+            0.009999001398682594,
+            0.009459372609853745,
+            -0.0026941753458231688,
+            0.010070422664284706,
+            -0.008840386755764484,
+            0.0012379715917631984,
+            0.006423961371183395,
+            0.012006737291812897,
+            0.012482879683375359,
+            0.020680474117398262,
+            0.0008382099331356585,
+            -0.00832456536591053,
+            0.005876397248357534,
+            -0.002870745025575161,
+            0.02080744504928589,
+            -0.01822040230035782,
+            0.00022976375475991517,
+            8.462696132482961e-05,
+            -0.003707963041961193,
+            -0.022807246074080467,
+            0.0020355109591037035,
+            5.387606506701559e-05,
+            0.010205329395830631,
+            0.014379516243934631,
+            0.005701811518520117,
+            0.002110900357365608,
+            -0.000986508559435606,
+            0.007769065443426371,
+            0.013673237524926662,
+            -0.009149880148470402,
+            -0.015307994559407234,
+            -0.02085505984723568,
+            -0.014070022851228714,
+            0.010451337322592735,
+            -0.0037000272423028946,
+            -0.0029997003730386496,
+            -0.003422277281060815,
+            0.012720951810479164,
+            0.011030644178390503,
+            -0.009165750816464424,
+            0.02236284501850605,
+            -0.016069823876023293,
+            -0.004396386444568634,
+            -0.01087986584752798,
+            0.01203848049044609,
+            0.0015434966189786792,
+            -0.01301457267254591,
+            0.006558868568390608,
+            0.008364243432879448,
+            -0.0008783844532445073,
+            -0.009983129799365997,
+            -0.0023807145189493895,
+            -0.003507586196064949,
+            0.003283402184024453,
+            0.007451636716723442,
+            -0.0007315737311728299,
+            0.002406505635008216,
+            0.017268115654587746,
+            0.003957937937229872,
+            0.005543096922338009,
+            0.013728787191212177,
+            -0.0017895037308335304,
+            -0.013657365925610065,
+            0.00436464324593544,
+            -0.028108302503824234,
+            -0.0047177826054394245,
+            0.011578208766877651,
+            -0.004630489740520716,
+            -0.000541116576641798,
+            -0.006880265194922686,
+            0.0571054071187973,
+            0.00369804329238832,
+            -0.015720652416348457,
+            0.0011873814510181546,
+            0.008507086895406246,
+            0.023410359397530556,
+            -0.0003945538483094424,
+            0.011967058293521404,
+            0.001203252817504108,
+            0.022870730608701706,
+            0.01977580226957798,
+            -0.02560061775147915,
+            -0.005570872221142054,
+            -0.003928178921341896,
+            0.00583275081589818,
+            0.0301715899258852,
+            0.02866380289196968,
+            0.0062612793408334255,
+            0.0006834634696133435,
+            0.005424061324447393,
+            0.021156616508960724,
+            0.003944050520658493,
+            -0.017760131508111954,
+            0.0026306896470487118,
+            0.0024203930515795946,
+            -0.006582675967365503,
+            0.005888300947844982,
+            -0.016887202858924866,
+            -0.029949389398097992,
+            0.018093431368470192,
+            7.718722918070853e-05,
+            -0.002864793175831437,
+            -0.003094929037615657,
+            -0.018839387223124504,
+            0.01222893688827753,
+            0.013593880459666252,
+            0.0017547850729897618,
+            -0.021410560235381126,
+            -0.002325164619833231,
+            0.006689807865768671,
+            -0.012601915746927261,
+            0.013966858386993408,
+            -0.014022408984601498,
+            -0.031869832426309586,
+            0.0034579879138618708,
+            -0.005535161588340998,
+            0.013236773200333118,
+            -0.0015147296944633126,
+            -0.015379415825009346,
+            -0.009356208145618439,
+            0.024219803512096405,
+            -0.01285585854202509,
+            0.0017041948158293962,
+            -0.005943850614130497,
+            -0.01150678750127554,
+            -0.001055450178682804,
+            0.0001287073828279972,
+            0.008292822167277336,
+            7.19174204277806e-05,
+            -0.01253049448132515,
+            0.01282411627471447,
+            0.024442002177238464,
+            0.0009131032275035977,
+            -0.010784637182950974,
+            -0.01107825804501772,
+            -0.01083225104957819,
+            0.002688223496079445,
+            -0.010491015389561653,
+            0.0017954555805772543,
+            -0.021886702626943588,
+            -0.00956253707408905,
+            0.01236384455114603,
+            -0.002618785947561264,
+            -0.0034500521142035723,
+            0.010657665319740772,
+            -0.0010023800423368812,
+            -0.0006889192736707628,
+            0.016903072595596313,
+            -0.013784337788820267,
+            0.0030413628555834293,
+            0.024267416447401047,
+            -0.0010931447613984346,
+            -0.003392518265172839,
+            -0.006400154437869787,
+            0.0050272755324840546,
+            -0.0002998212294187397,
+            -0.01655390113592148,
+            -0.006293022073805332,
+            -0.017522059381008148,
+            -0.009356208145618439,
+            -0.006848522461950779,
+            -0.018633060157299042,
+            -0.002815194893628359,
+            0.001860925229266286,
+            0.012340037152171135,
+            0.014744559302926064,
+            -0.0019144912948831916,
+            0.006126372143626213,
+            0.02053763158619404,
+            -0.01541115902364254,
+            0.014823916368186474,
+            0.00276956451125443,
+            0.007800808176398277,
+            0.005793072283267975,
+            0.017426829785108566,
+            0.009141944348812103,
+            0.0015028261113911867,
+            0.0036186862271279097,
+            -0.006967558059841394,
+            0.002551332348957658,
+            0.00646760780364275,
+            -0.02247394621372223,
+            0.006352540105581284,
+            0.00021810817997902632,
+            -0.00420196121558547,
+            -0.017744259908795357,
+            -0.004257511347532272,
+            0.002378730569034815,
+            0.007737322244793177,
+            -0.004305125679820776,
+            -0.00040571345016360283,
+            -0.02058524452149868,
+            -0.005320896860212088,
+            -0.0020394788589328527,
+            -0.026330703869462013,
+            -0.005785136483609676,
+            0.0008957438403740525,
+            0.013300258666276932,
+            -0.007661933079361916,
+            -0.005551032721996307,
+            0.022489817813038826,
+            0.011967058293521404,
+            -0.013070123270154,
+            -0.0010693375952541828,
+            -0.0029818450566381216,
+            -0.005098697263747454,
+            0.01644280180335045,
+            0.013220901601016521,
+            -0.00030130919185467064,
+            0.011935316026210785,
+            -0.008792772889137268,
+            0.00692391162738204,
+            -0.007939683273434639,
+            -0.0040611024014651775,
+            0.012586044147610664,
+            -0.011808344163000584,
+            0.00404919870197773,
+            0.0140065373852849,
+            0.012792373076081276,
+            -0.017268115654587746,
+            -0.0070270756259560585,
+            -0.01793471723794937,
+            0.0028409860096871853,
+            0.002198193222284317,
+            -0.008713414892554283,
+            0.008356308564543724,
+            0.0033151451498270035,
+            0.007661933079361916,
+            0.012657465413212776,
+            -0.007106433156877756,
+            0.023949988186359406,
+            0.007292922120541334,
+            -0.0019085395615547895,
+            0.01191150862723589,
+            -0.014673137106001377,
+            0.014728687703609467,
+            0.011490915901958942,
+            -0.0009711331804282963,
+            -0.016903072595596313,
+            -0.0016992350574582815,
+            0.0280289463698864,
+            -0.021696245297789574,
+            0.007392118684947491,
+            -0.003473859280347824,
+            -0.01756967417895794,
+            -0.013887501321732998,
+            -0.009792672470211983,
+            0.007098497357219458,
+            -0.010792572982609272,
+            0.0005416125641204417,
+            -0.019363144412636757,
+            -0.009578408673405647,
+            -0.00011240196909056976,
+            0.01745857298374176,
+            -0.001312368898652494,
+            -0.007808743976056576,
+            0.00809839740395546,
+            0.008721350692212582,
+            -0.010689408518373966,
+            -0.03012397512793541,
+            0.02650528959929943,
+            -0.0025810913648456335,
+            -0.008467407897114754,
+            -0.0013748626224696636,
+            -0.0036365415435284376,
+            0.0048606256023049355,
+            -0.018601316958665848,
+            -0.00811030063778162,
+            -0.007812711410224438,
+            -0.007979361340403557,
+            0.005558968521654606,
+            0.00436464324593544,
+            0.0019591297022998333,
+            -0.008721350692212582,
+            0.006217632908374071,
+            -0.005483579356223345,
+            -0.01622060127556324,
+            -0.01691894419491291,
+            0.010737022385001183,
+            -0.006892168894410133,
+            -0.013538329862058163,
+            -0.008050783537328243,
+            0.0038865164387971163,
+            -0.003188173519447446,
+            -0.006102565210312605,
+            0.01691894419491291,
+            -0.005332800559699535,
+            -0.016569772735238075,
+            0.001228051958605647,
+            0.01004661526530981,
+            0.017522059381008148,
+            0.016569772735238075,
+            0.03367917612195015,
+            0.0015682957600802183,
+            -0.017601415514945984,
+            0.010649729520082474,
+            -8.43944690132048e-06,
+            0.0031961093191057444,
+            0.013831951655447483,
+            -0.01849021576344967,
+            0.010697344318032265,
+            -0.019521860405802727,
+            0.010094230063259602,
+            -0.013736722990870476,
+            -0.005162182729691267,
+            -0.018998103216290474,
+            -0.01772838830947876,
+            -0.0008223385084420443,
+            -0.003146511036902666,
+            0.01355420146137476,
+            0.0005530201597139239,
+            -0.012713016010820866,
+            0.008364243432879448,
+            0.017363345250487328,
+            -0.038726288825273514,
+            -0.011133808642625809,
+            -0.004729686304926872,
+            0.0036484450101852417,
+            -0.017268115654587746,
+            0.02042653039097786,
+            0.023838888853788376,
+            -0.003993648570030928,
+            0.003088977187871933,
+            0.00404919870197773,
+            0.005471675656735897,
+            0.010562436655163765,
+            -0.01810930296778679,
+            0.005118536297231913,
+            -0.036885205656290054,
+            0.0010891768615692854,
+            0.016728486865758896,
+            0.02418806031346321,
+            -0.006281118839979172,
+            -0.01642693020403385,
+            -0.012347972951829433,
+            -0.009602215141057968,
+            -0.031012775376439095,
+            -0.006550932768732309,
+            -0.01368910912424326,
+            -0.0026326735969632864,
+            -0.007844454608857632,
+            -0.01236384455114603,
+            0.024600718170404434,
+            0.008427729830145836,
+            -0.0019283788278698921,
+            -0.004686039872467518,
+            0.003116752253845334,
+            -0.024330902844667435,
+            -0.01622060127556324,
+            -0.014204930514097214,
+            0.005801007617264986,
+            0.021426431834697723,
+            -0.0033786308486014605,
+            0.012593979947268963,
+            -0.011395687237381935,
+            0.001903579686768353,
+            0.0024719752836972475,
+            0.004182121716439724,
+            -0.013649430125951767,
+            0.032044418156147,
+            -0.0012171403504908085,
+            -0.0002213320549344644,
+            0.0037238344084471464,
+            -0.004602714907377958,
+            0.014974694699048996,
+            0.006717583164572716,
+            -0.013324066065251827,
+            -0.010657665319740772,
+            0.01741096004843712,
+            0.01549051608890295,
+            0.0019174672197550535,
+            -0.01174485869705677,
+            -0.0005376447224989533,
+            0.008126172237098217,
+            -0.006023207679390907,
+            -0.003297289600595832,
+            -0.00013267523900140077,
+            0.0005698835593648255,
+            0.02769564650952816,
+            0.005487547256052494,
+            0.005090761464089155,
+            -0.011340136639773846,
+            0.015101666562259197,
+            0.017014173790812492,
+            0.005535161588340998,
+            -0.0013084009988233447,
+            -0.020664602518081665,
+            0.004186089616268873,
+            -0.021220102906227112,
+            -0.003416325431317091,
+            -0.008951487019658089,
+            -0.0034778271801769733,
+            0.004559068474918604,
+            -0.0025394288823008537,
+            -0.016791973263025284,
+            0.00607478991150856,
+            -0.009848223067820072,
+            -4.6684326662216336e-05,
+            0.009181622415781021,
+            -0.013347873464226723,
+            0.013022508472204208,
+            0.029552603140473366,
+            0.018823517486453056,
+            0.015514323487877846,
+            -0.0025275254156440496,
+            -0.02241045981645584,
+            0.02321990206837654,
+            0.008761029690504074,
+            0.006114468444138765,
+            0.0010078358463943005,
+            -0.007392118684947491,
+            0.025505388155579567,
+            0.029822418466210365,
+            0.002785436110571027,
+            0.006741390097886324,
+            -0.006332700606435537,
+            0.013514523394405842,
+            0.00864992942661047,
+            -0.006717583164572716,
+            0.0006824714946560562,
+            0.018521958962082863,
+            -0.0066223544999957085,
+            -0.003807159373536706,
+            0.031076261773705482,
+            -0.023156417533755302,
+            -0.012419394217431545,
+            -0.013458972796797752,
+            -0.0021168519742786884,
+            0.00428528618067503,
+            0.017522059381008148,
+            -0.021839087828993797,
+            -0.008459472097456455,
+            0.013141544535756111,
+            0.008792772889137268,
+            0.00013689108891412616,
+            -0.008713414892554283,
+            0.0006621362408623099,
+            0.010713215917348862,
+            -0.006991364993155003,
+            0.0005802991800010204,
+            -0.029917646199464798,
+            -0.0027576610445976257,
+            -0.0023410359863191843,
+            0.0027775003109127283,
+            0.04339249059557915,
+            -0.002136691240593791,
+            0.00878483708947897,
+            0.003140559419989586,
+            0.008078558370471,
+            -0.007185790222138166,
+            -0.014435065910220146,
+            0.0011625823099166155,
+            0.01642693020403385,
+            -0.00584862194955349,
+            0.010435465723276138,
+            -0.0008883041446097195,
+            -0.027346475049853325,
+            0.017331602051854134,
+            0.03659952059388161,
+            0.006082725711166859,
+            -0.005428029224276543,
+            -0.005900204181671143,
+            -0.005150279030203819,
+            -0.012371780350804329,
+            -0.007328633219003677,
+            0.013506587594747543,
+            0.0012657465413212776,
+            0.027235373854637146,
+            -0.015061987563967705,
+            0.005150279030203819,
+            -0.005816879216581583,
+            -0.025632360950112343,
+            0.010824315249919891,
+            0.004309093579649925,
+            0.00013974298781249672,
+            0.01563335955142975,
+            0.020616987720131874,
+            -0.03082231804728508,
+            -0.011030644178390503,
+            -0.005364543292671442,
+            -0.021696245297789574,
+            0.016188859939575195,
+            -0.006864393595606089,
+            0.01503024436533451,
+            -0.002779484260827303,
+            0.007907940074801445,
+            -0.011236973106861115,
+            0.004293221980333328,
+            0.019712315872311592,
+            -0.004086893517524004,
+            0.018982231616973877,
+            -0.009983129799365997,
+            -0.007618286646902561,
+            0.013355808332562447,
+            -0.016633259132504463,
+            0.020775701850652695,
+            -0.005678004119545221,
+            -0.002283502137288451,
+            0.021918445825576782,
+            -0.01347484439611435,
+            0.012355908751487732,
+            0.0030314433388412,
+            -0.01309392973780632,
+            -0.00044415207230485976,
+            -0.0033389520831406116,
+            0.0013252644566819072,
+            -0.009610150940716267,
+            -0.019474245607852936,
+            0.021029645577073097,
+            -0.0180458165705204,
+            0.023807145655155182,
+            0.0015028261113911867,
+            -0.011348072439432144,
+            0.001473067095503211,
+            -0.007971425540745258,
+            -0.023521460592746735,
+            0.0015752394683659077,
+            -0.00280130747705698,
+            0.011816279962658882,
+            -0.009475244209170341,
+            0.020601116120815277,
+            0.0057335542514920235,
+            -0.016165051609277725,
+            0.00047986279241740704,
+            0.011570272967219353,
+            -0.0006963590276427567,
+            0.005158214829862118,
+            0.027711518108844757,
+            -0.001079257228411734,
+            0.0029957324732095003,
+            0.006054950878024101,
+            0.008745158091187477,
+            -0.03352046012878418,
+            -0.009673637337982655,
+            0.004047214984893799,
+            -0.0004498558701016009,
+            0.010149779729545116,
+            0.005221700761467218,
+            -0.010221200995147228,
+            0.022712016478180885,
+            -0.00013354320253711194,
+            -0.017744259908795357,
+            0.008475343696773052,
+            -0.009205429814755917,
+            -0.011602015234529972,
+            -0.009284786880016327,
+            -0.002348971785977483,
+            0.021029645577073097,
+            -0.005820847116410732,
+            0.010030743665993214,
+            0.0017418895149603486,
+            0.003398470114916563,
+            -0.002497766399756074,
+            -0.017966458573937416,
+            0.002624737797304988,
+            -0.004618586506694555,
+            0.020188460126519203,
+            0.015919044613838196,
+            0.015823816880583763,
+            -0.00911020115017891,
+            -0.015593680553138256,
+            0.004725718405097723,
+            0.02796545997262001,
+            -0.009895836934447289,
+            0.0071699186228215694,
+            -0.004011503886431456,
+            0.0013937100302428007,
+            -0.006539029534906149,
+            -0.013133608736097813,
+            -0.015228637494146824,
+            -0.03482192009687424,
+            -0.0019690494518727064,
+            0.002146610990166664,
+            -0.00044415207230485976,
+            -0.012919344007968903,
+            -0.03685346245765686,
+            -0.007911908440291882,
+            -0.02052175998687744,
+            -0.019585344940423965,
+            -0.02807656116783619,
+            -0.02445787377655506,
+            -0.004880465101450682,
+            -0.014784237369894981,
+            0.011244908906519413,
+            -0.0034460844472050667,
+            -0.009792672470211983,
+            0.0037754166405647993,
+            0.023045316338539124,
+            0.00047267103218473494,
+            0.02042653039097786,
+            -0.013760530389845371,
+            0.008880065754055977,
+            -0.007530993781983852,
+            0.007308793719857931,
+            -0.006360475905239582,
+            -0.01649041660130024,
+            0.00654696486890316,
+            0.008070622570812702,
+            0.021140744909644127,
+            -0.01129252277314663,
+            0.0018837404204532504,
+            -0.006285086274147034,
+            0.0013431197730824351,
+            -0.00023447559215128422,
+            0.000928974652197212,
+            0.006709647364914417,
+            0.020188460126519203,
+            -0.02625134587287903,
+            -0.018775902688503265,
+            0.008126172237098217,
+            0.009760930202901363,
+            0.021807344630360603,
+            -0.021632758900523186,
+            0.013284387066960335,
+            -2.7294519895804115e-05,
+            -0.00575339375063777,
+            0.013220901601016521,
+            0.012078158557415009,
+            -0.005277250427752733,
+            0.0024997503496706486,
+            0.02301357500255108,
+            0.012467009015381336,
+            -0.009673637337982655,
+            -0.020601116120815277,
+            0.0088880006223917,
+            0.0006874313112348318,
+            -0.007733354344964027,
+            0.004868561401963234,
+            0.03361568972468376,
+            0.023156417533755302,
+            0.01982341706752777,
+            -0.00039802573155611753,
+            0.031552404165267944,
+            0.0011506787268444896,
+            -0.016744358465075493,
+            0.010705280117690563,
+            -0.0036087664775550365,
+            0.004928078968077898,
+            -0.016315830871462822,
+            -0.04532880708575249,
+            0.004880465101450682,
+            0.002618785947561264,
+            0.009999001398682594,
+            -0.027457574382424355,
+            0.01891874521970749,
+            0.0006244415999390185,
+            -0.0024660234339535236,
+            -0.014958823099732399,
+            0.008126172237098217,
+            -0.010387850925326347,
+            0.00818965770304203,
+            -0.004559068474918604,
+            -0.01463345903903246,
+            0.021474044770002365,
+            0.00630492577329278,
+            -0.004531293176114559,
+            0.011340136639773846,
+            0.022537430748343468,
+            0.01847434602677822,
+            0.012078158557415009,
+            0.010467207990586758,
+            0.04247194901108742,
+            0.011784536764025688,
+            0.0025632360484451056,
+            0.006626322399824858,
+            -0.00334688788279891,
+            -0.0032774503342807293,
+            -0.004908239934593439,
+            0.000227779833949171,
+            -0.004071021918207407,
+            -0.018299760296940804,
+            0.016855459660291672,
+            0.010118036530911922,
+            -0.010729086585342884,
+            0.01714114472270012,
+            -0.01619679480791092,
+            0.012030544690787792,
+            0.03485365957021713,
+            0.00010012640996137634,
+            0.017696645110845566,
+            -0.01901397295296192,
+            -0.009760930202901363,
+            -0.0020573341753333807,
+            -0.011546465568244457,
+            0.0036206701770424843,
+            -0.015823816880583763,
+            -0.007455604616552591,
+            -0.008213465102016926,
+            0.00770557951182127,
+            0.01595078781247139,
+            0.015347673557698727,
+            0.013530394062399864,
+            0.01338755153119564,
+            0.0056065828539431095,
+            -0.009681572206318378,
+            -0.001512745744548738,
+            -0.015379415825009346,
+            -0.012522558681666851,
+            -0.01606188714504242,
+            0.004943950567394495,
+            -0.0025394288823008537,
+            -0.017966458573937416,
+            0.0269496887922287,
+            -0.013054251670837402,
+            0.005487547256052494,
+            0.009959322400391102,
+            0.01876003108918667,
+            -0.0005031739710830152,
+            0.00045853553456254303,
+            -0.005035211332142353,
+            0.0008818563655950129,
+            0.005150279030203819,
+            -0.001076281419955194,
+            0.027394089847803116,
+            -0.004225768148899078,
+            0.004277350381016731,
+            -0.0017498251982033253,
+            -0.014974694699048996,
+            -0.0007028068066574633,
+            0.00840392243117094,
+            -0.020870931446552277,
+            -0.02263266034424305,
+            -0.018283888697624207,
+            -0.005253443494439125,
+            -0.011157616041600704,
+            0.0067334542982280254,
+            -0.011300458572804928,
+            -0.008880065754055977,
+            0.023838888853788376,
+            0.00661441870033741,
+            0.014744559302926064,
+            0.0058446540497243404,
+            -0.0009671652805991471,
+            0.017712516710162163,
+            -0.02306118793785572,
+            -0.014379516243934631,
+            0.0024263449013233185,
+            -0.010744958184659481,
+            0.032568175345659256,
+            0.00097311707213521,
+            0.01301457267254591,
+            -0.010983030311763287,
+            0.010435465723276138,
+            -0.014117637649178505,
+            0.02418806031346321,
+            0.012705080211162567,
+            0.00584862194955349,
+            0.004186089616268873,
+            -0.003910323604941368,
+            -0.011308394372463226,
+            0.03821840509772301,
+            0.0002916375233326107,
+            -0.007225468754768372,
+            0.02004561759531498,
+            0.00832456536591053,
+            0.008721350692212582,
+            -0.008657865226268768,
+            0.012022608891129494,
+            0.016665002331137657,
+            0.0002353435556869954,
+            0.006741390097886324,
+            -0.01826801709830761,
+            -0.0040452308021485806,
+            0.023204030469059944,
+            0.025092732161283493,
+            0.011467108502984047,
+            -0.003124687820672989,
+            -0.002600930631160736,
+            0.012720951810479164,
+            -0.0017408975400030613,
+            0.019410759210586548,
+            0.009300658479332924,
+            0.03863106295466423,
+            0.019601216539740562,
+            -0.014046216383576393,
+            -0.0013530394062399864,
+            0.005110600497573614,
+            0.05183609202504158,
+            -0.010657665319740772,
+            -0.007161982823163271,
+            -0.008269015699625015,
+            -0.014752495102584362,
+            0.015157216228544712,
+            -0.010173587128520012,
+            0.008082525804638863,
+            -0.008657865226268768,
+            0.010863994248211384,
+            0.017823616042733192,
+            0.007892068475484848,
+            -0.009792672470211983,
+            -0.007376247551292181,
+            -0.011117937043309212,
+            0.012562237679958344,
+            0.001430412638001144,
+            -0.00338855036534369,
+            0.011855958960950375,
+            0.01026087999343872,
+            -0.016014274209737778,
+            -0.007677804678678513,
+            0.017855359241366386,
+            -0.022346973419189453,
+            0.0013738706475123763,
+            0.007284986786544323,
+            0.0036008309107273817,
+            -0.00108818500302732,
+            -6.959250185900601e-06,
+            0.00910226535052061,
+            0.028600318357348442,
+            0.0016545966500416398,
+            0.02942563220858574,
+            0.01799820177257061,
+            0.00014445481065195054,
+            -0.024965759366750717,
+            0.005789104383438826,
+            0.01739508844912052,
+            0.004412257578223944,
+            0.024695945903658867,
+            -0.005420093424618244,
+            -0.012459073215723038,
+            -0.0191250741481781,
+            -0.0088880006223917,
+            0.012094030156731606,
+            -0.018394988030195236,
+            0.013197094202041626,
+            -0.005189957562834024,
+            0.007915875874459743,
+            0.005777200683951378,
+            0.015093730762600899,
+            -0.010530694387853146,
+            0.007685740012675524,
+            0.0021902574226260185,
+            -0.011086193844676018,
+            -0.011784536764025688,
+            -0.00032883620588108897,
+            -0.007384182885289192,
+            0.003209996735677123,
+            -0.0035849593114107847,
+            0.011649630032479763,
+            0.023727789521217346,
+            -0.004773332737386227,
+            -0.006947718560695648,
+            0.009991065599024296,
+            0.004654297139495611,
+            -0.011562337167561054,
+            0.02193431742489338,
+            0.009705379605293274,
+            0.004110700450837612,
+            0.007856357842683792,
+            -0.01042752992361784,
+            -0.016069823876023293,
+            -0.023045316338539124,
+            -0.007181822322309017,
+            -0.0231722891330719,
+            0.014974694699048996,
+            -0.006658065132796764,
+            -0.015800008550286293,
+            0.01858544535934925,
+            0.002934230724349618,
+            0.0018480296712368727,
+            -0.00848327949643135,
+            -0.026330703869462013,
+            -0.022331101819872856,
+            0.01026087999343872,
+            -0.00654696486890316,
+            0.005201861262321472,
+            0.02706078812479973,
+            0.030155718326568604,
+            -0.007019140291959047,
+            0.004261479247361422,
+            0.0034500521142035723,
+            0.003930163104087114,
+            -0.0002421633107587695,
+            -0.024981630966067314,
+            -0.0064041223376989365,
+            0.007947619073092937,
+            0.006789004430174828,
+            -0.00544390082359314,
+            -0.00926891528069973,
+            0.011943251825869083,
+            -0.019299659878015518,
+            -0.005920043680816889,
+            -0.009173686616122723,
+            -0.002819162793457508,
+            -0.00032015651231631637,
+            0.017982330173254013,
+            0.015268316492438316,
+            -0.005551032721996307,
+            -0.0025870432145893574,
+            -0.023775402456521988,
+            -0.012181323021650314,
+            0.01939488761126995,
+            0.004884432535618544,
+            0.006574740167707205,
+            -0.01471281610429287,
+            -0.0012141644256189466,
+            -0.0033389520831406116,
+            -0.012498751282691956,
+            0.015173087827861309,
+            0.06513635069131851,
+            -0.016006337478756905,
+            0.02483878843486309,
+            -0.005007436498999596,
+            0.004701911471784115,
+            -0.003416325431317091,
+            -0.021426431834697723,
+            -0.011951187625527382,
+            -0.0060271755792200565,
+            0.0017756163142621517,
+            -0.0188552588224411,
+            -0.0023965861182659864,
+            -0.0006715598865412176,
+            0.002325164619833231,
+            -0.007019140291959047,
+            0.0011953171342611313,
+            -0.02085505984723568,
+            0.016887202858924866,
+            0.006431897170841694,
+            -0.003862709505483508,
+            -0.006447768770158291,
+            -0.018823517486453056,
+            0.00022120805806480348,
+            -0.0016357493586838245,
+            -2.8115991881350055e-05,
+            0.0016784038161858916,
+            -0.025902174413204193,
+            -0.010657665319740772,
+            -0.00964189413934946,
+            -0.015069923363626003,
+            0.0002531989011913538,
+            -0.006380314938724041,
+            0.01463345903903246,
+            -0.0220930315554142,
+            -0.004872529301792383,
+            0.01822040230035782,
+            0.028362246230244637,
+            0.020728088915348053,
+            -0.011213165707886219,
+            0.001291537657380104,
+            -0.02160101756453514,
+            0.02533080242574215,
+            0.01880764588713646,
+            -0.017442701384425163,
+            0.005102664697915316,
+            0.008768965490162373,
+            0.009951386600732803,
+            -0.0019998003263026476,
+            0.030139846727252007,
+            -0.00038463418604806066,
+            -0.015427030622959137,
+            0.005090761464089155,
+            0.018934616819024086,
+            0.002618785947561264,
+            0.002612834330648184,
+            -0.006209697108715773,
+            0.021585145965218544,
+            -0.005582775454968214,
+            0.009633958339691162,
+            -0.044852662831544876,
+            -0.01815691590309143,
+            -0.005519289989024401,
+            -0.004213864915072918,
+            0.007812711410224438,
+            0.004011503886431456,
+            0.008673736825585365,
+            -0.018093431368470192,
+            -0.0021208198741078377,
+            0.025283189490437508,
+            -0.026965560391545296,
+            -0.02823527529835701,
+            -0.005225668661296368,
+            -0.02441026084125042,
+            -0.0007444692892022431,
+            0.0011546466266736388,
+            -0.007959522306919098,
+            -0.026473546400666237,
+            0.008284886367619038,
+            -0.006122404243797064,
+            -0.027679774910211563,
+            0.01768077351152897,
+            -0.005681972019374371,
+            0.000294117460725829,
+            -0.010300558060407639,
+            -0.003963889554142952,
+            -0.005356607958674431,
+            0.016569772735238075,
+            -0.01330819446593523,
+            0.008062686771154404,
+            0.0010266831377521157,
+            -0.0032357878517359495,
+            0.0016873314743861556,
+            0.016093630343675613,
+            0.002900503808632493,
+            0.000488046498503536,
+            -0.006039079278707504,
+            -0.003088977187871933,
+            -0.017871230840682983,
+            0.000921534956432879,
+            0.01891874521970749,
+            0.0036325736436992884,
+            0.02672748826444149,
+            -0.03469494730234146,
+            -0.01642693020403385,
+            -0.005725618451833725,
+            -0.01725224405527115,
+            0.011403623037040234,
+            -0.0036821719259023666,
+            -0.008499151095747948,
+            -0.0019075475865975022,
+            -0.02548951655626297,
+            0.03304431959986687,
+            -0.01939488761126995,
+            -0.00506695406511426,
+            0.004428129177540541,
+            -0.019839288666844368,
+            -0.00044935988262295723,
+            0.008157915435731411,
+            0.01998213119804859,
+            0.010308493860065937,
+            -0.015617487952113152,
+            0.009070522151887417,
+            -0.008538829162716866,
+            -0.008054750971496105,
+            0.0041543468832969666,
+            0.008126172237098217,
+            -0.009681572206318378,
+            -0.0029362146742641926,
+            0.00816585123538971,
+            -0.0030731058213859797,
+            -0.014284287579357624,
+            -0.00878483708947897,
+            0.027219504117965698,
+            -0.008515022695064545,
+            0.00575339375063777,
+            -0.0026386252138763666,
+            -0.001564327860251069,
+            -0.0020394788589328527,
+            0.00064130499958992,
+            0.0062612793408334255,
+            -0.006253343541175127,
+            0.05850209295749664,
+            -0.0003680188092403114,
+            -0.00022405995696317405,
+            -0.01869654469192028,
+            -0.011855958960950375,
+            0.005035211332142353,
+            0.006880265194922686,
+            -0.02220413088798523,
+            -0.006959622260183096,
+            0.025092732161283493,
+            -0.0020533662755042315,
+            0.010808444581925869,
+            0.005408189725130796,
+            -0.001736929640173912,
+            0.0328538604080677,
+            0.006852489896118641,
+            0.0280130747705698,
+            -0.010760829783976078,
+            0.004015471786260605,
+            0.0001645420998102054,
+            0.00755876861512661,
+            0.0069516864605247974,
+            -0.009118136949837208,
+            -0.005003468599170446,
+            0.0040611024014651775,
+            0.014300159178674221,
+            -0.0004451440181583166,
+            -0.008292822167277336,
+            0.0030135877896100283,
+            -0.012109901756048203,
+            -0.0017528011230751872,
+            -0.0008218425209634006,
+            -0.0140065373852849,
+            -0.00576529698446393,
+            0.009173686616122723,
+            0.015458772890269756,
+            0.007324665319174528,
+            0.0010683456202968955,
+            0.004400353878736496,
+            0.004999500699341297,
+            0.005269315093755722,
+            -0.012094030156731606,
+            0.009570472873747349,
+            -0.02667987532913685,
+            -0.009062586352229118,
+            0.01214957982301712,
+            -0.0011348072439432144,
+            -0.00886419415473938,
+            0.023505588993430138,
+            0.0046265218406915665,
+            -0.012332101352512836,
+            -0.011022708378732204,
+            0.006796940229833126,
+            -0.0023172288201749325,
+            0.008991165086627007,
+            -0.007860326208174229,
+            0.0027636128943413496,
+            -0.0007955554756335914,
+            0.01622060127556324,
+            0.01653803139925003,
+            -0.004253543447703123,
+            0.012117837555706501,
+            -0.0004079453647136688
+          ],
+          "title": "Balancing Expert Utilization",
+          "keyphrases": [
+            "gating network",
+            "expert utilization",
+            "soft constraint",
+            "importance of an expert",
+            "balanced loads"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "aa1e1bd4-470d-487e-91f8-3df89ce69781",
+      "type": "next",
+      "source": {
+        "id": "ff7dd6f5-6560-42e5-923e-f4091e278202",
+        "properties": {
+          "page_content": "4 Balancing Expert Utilization\n\nWe have observed that the gating network tends to converge to a state where it always produces large weights for the same few experts. This imbalance is self-reinforcing, as the favored experts are trained more rapidly and thus are selected even more by the gating network. Eigen et al. (2013) describe the same phenomenon, and use a hard constraint at the beginning of training to avoid this local minimum. Bengio et al. (2015) include a soft constraint on the batch-wise average of each gate.1 We take a soft constraint approach. We define the importance of an expert relative to a batch of training examples to be the batchwise sum of the gate values for that expert. We define an additional loss Limportance, which is added to the overall loss function for the model. This loss is equal to the square of the coefficient of variation of the set of importance values, multiplied by a hand-tuned scaling factor wimportance. This additional loss encourages all experts to have equal importance.\n\n$$I m p o r t a n c e(X)=\\sum_{x\\in X}G(x)$$ $$L_{importance}(X)=w_{importance}\\cdot CV(Importance(X))^{2}\\tag{7}$$ $$(6)$$\n\nWhile this loss function can ensure equal importance, experts may still receive very different numbers of examples. For example, one expert may receive a few examples with large weights, and another may receive many examples with small weights. This can cause memory and performance problems on distributed hardware. To solve this problem, we introduce a second loss function, Lload , which ensures balanced loads. Appendix A contains the definition of this function, along with experimental results.\n\n",
+          "embedding": [
+            -0.013197094202041626,
+            -0.002106932457536459,
+            -0.020997902378439903,
+            -0.01955360174179077,
+            -0.0042971898801624775,
+            0.013943051919341087,
+            -0.01379227265715599,
+            0.05729586258530617,
+            -0.02623547427356243,
+            0.009284786880016327,
+            -0.014958823099732399,
+            0.013482780195772648,
+            -0.0004461359931156039,
+            -0.0026227538473904133,
+            0.012546366080641747,
+            0.013776401989161968,
+            0.013578008860349655,
+            -0.0015841671265661716,
+            0.0022577110212296247,
+            -0.008999100886285305,
+            0.036821719259023666,
+            -0.03621860593557358,
+            -0.02348971739411354,
+            0.012839986942708492,
+            -0.011721051298081875,
+            0.029409760609269142,
+            0.025187959894537926,
+            -0.015276252292096615,
+            -0.048630062490701675,
+            0.009760930202901363,
+            0.025886302813887596,
+            -0.002469991333782673,
+            -0.04361468926072121,
+            -0.004309093579649925,
+            -0.04536054655909538,
+            -0.018458474427461624,
+            0.04875703528523445,
+            0.02812417410314083,
+            -0.012403522618114948,
+            0.030568374320864677,
+            0.02133120223879814,
+            0.006888200994580984,
+            -0.03971032053232193,
+            0.01788710243999958,
+            0.009380015544593334,
+            -0.032631661742925644,
+            -0.03777400404214859,
+            -0.0011179439024999738,
+            -0.0026505289133638144,
+            -0.03621860593557358,
+            0.013538329862058163,
+            -0.05221700668334961,
+            0.005737522151321173,
+            -0.018934616819024086,
+            -0.011316330172121525,
+            0.02893361821770668,
+            -0.010998900979757309,
+            0.014236672781407833,
+            0.042598918080329895,
+            0.01918855868279934,
+            0.006015272345393896,
+            0.03012397512793541,
+            -0.055962663143873215,
+            0.021616889163851738,
+            0.023997602984309196,
+            -0.04751906171441078,
+            -0.0064636399038136005,
+            0.01644280180335045,
+            -0.001340143964625895,
+            -0.012998702004551888,
+            -0.06805669516324997,
+            0.02564823254942894,
+            -0.0355837456882,
+            0.025187959894537926,
+            0.005527225788682699,
+            -0.006130340043455362,
+            0.042027547955513,
+            0.03217139095067978,
+            -0.0028231306932866573,
+            0.013181222602725029,
+            -0.026933817192912102,
+            0.009205429814755917,
+            0.004031343385577202,
+            0.022108903154730797,
+            0.04294808954000473,
+            -0.010895737446844578,
+            -0.00926891528069973,
+            -0.05367717891931534,
+            0.01799820177257061,
+            -0.022077159956097603,
+            -0.015966659411787987,
+            0.02117248810827732,
+            0.01120522990822792,
+            -0.012308293953537941,
+            -0.02818766050040722,
+            0.03694869205355644,
+            -0.000555996026378125,
+            -0.004745557904243469,
+            -0.010451337322592735,
+            -0.008872129954397678,
+            -0.02236284501850605,
+            0.006181922275573015,
+            -0.02958434633910656,
+            -0.029520859941840172,
+            0.030203333124518394,
+            -0.022378716617822647,
+            -0.01826801709830761,
+            0.012459073215723038,
+            0.030250946059823036,
+            0.10710041224956512,
+            -0.009221301414072514,
+            -0.009784736670553684,
+            -0.030838189646601677,
+            0.013371679931879044,
+            -0.03891674801707268,
+            -0.037043917924165726,
+            -0.027632160112261772,
+            -0.06120023503899574,
+            0.025362545624375343,
+            -0.023584945127367973,
+            0.023188160732388496,
+            0.04977280646562576,
+            0.012966958805918694,
+            0.01961708813905716,
+            -0.044916149228811264,
+            0.04409083351492882,
+            -0.023997602984309196,
+            0.027775002643465996,
+            0.008102364838123322,
+            0.0019363145111128688,
+            0.0032278522849082947,
+            0.02650528959929943,
+            -0.02269614487886429,
+            -0.044376518577337265,
+            0.006757261697202921,
+            0.013300258666276932,
+            0.06339049339294434,
+            0.02499750256538391,
+            0.01441919431090355,
+            -0.034345775842666626,
+            0.01129252277314663,
+            -0.053645435720682144,
+            -0.019220301881432533,
+            0.015982531011104584,
+            -0.00685645779594779,
+            0.016347574070096016,
+            -0.00514631113037467,
+            0.020442401990294456,
+            -0.016887202858924866,
+            0.010364044457674026,
+            -0.005693875718861818,
+            0.04843960702419281,
+            -0.006166050676256418,
+            0.024108702316880226,
+            0.019061587750911713,
+            0.016506288200616837,
+            -0.028949489817023277,
+            0.025553002953529358,
+            -0.015427030622959137,
+            0.0011467108270153403,
+            -0.0006834634696133435,
+            -0.001254834933206439,
+            -0.015847623348236084,
+            0.02344210259616375,
+            0.024965759366750717,
+            0.06729486584663391,
+            0.01676023006439209,
+            0.013903372921049595,
+            -0.07834137976169586,
+            0.026759231463074684,
+            -0.0036087664775550365,
+            -0.03891674801707268,
+            0.03996426239609718,
+            0.03631383180618286,
+            2.3481657990487292e-05,
+            0.02656877413392067,
+            -0.007566704414784908,
+            0.017760131508111954,
+            -0.014720751903951168,
+            0.03980554640293121,
+            -0.03479017689824104,
+            -0.014744559302926064,
+            0.008058718405663967,
+            0.007725419010967016,
+            0.01649041660130024,
+            0.008308693766593933,
+            0.026743359863758087,
+            0.012879665940999985,
+            -0.019442502409219742,
+            0.004567004274576902,
+            0.028378117829561234,
+            0.01788710243999958,
+            0.014355708844959736,
+            -0.003918259404599667,
+            0.016887202858924866,
+            -0.03574246168136597,
+            0.010935415513813496,
+            -0.03307606279850006,
+            0.0299017746001482,
+            0.0007930755382403731,
+            -0.005070921964943409,
+            -0.02004561759531498,
+            0.013974794186651707,
+            -0.015831751748919487,
+            -0.023997602984309196,
+            0.012403522618114948,
+            0.03732960298657417,
+            0.004404321778565645,
+            0.013189158402383327,
+            -0.03539329022169113,
+            0.009078457951545715,
+            -0.02053763158619404,
+            -0.018093431368470192,
+            0.004916175734251738,
+            -0.05082032084465027,
+            -0.01330819446593523,
+            0.0177760012447834,
+            0.058819521218538284,
+            0.042376719415187836,
+            -0.007765097543597221,
+            -0.026965560391545296,
+            0.02139468863606453,
+            0.031076261773705482,
+            0.00046944714267738163,
+            -0.0034480681642889977,
+            -0.014585844241082668,
+            -0.004142443183809519,
+            -0.013808144256472588,
+            0.008927679620683193,
+            -0.014593780040740967,
+            0.007959522306919098,
+            0.015442902222275734,
+            0.03767877444624901,
+            0.021902574226260185,
+            -0.0007375255227088928,
+            -0.0022081127390265465,
+            -0.012522558681666851,
+            -0.005023307632654905,
+            0.0010033720172941685,
+            -0.03552026301622391,
+            -0.026600517332553864,
+            -0.017474444583058357,
+            -0.03840886056423187,
+            0.0005227652145549655,
+            -0.011403623037040234,
+            0.011356008239090443,
+            -0.00880864355713129,
+            0.03821840509772301,
+            -0.0026088664308190346,
+            -0.011022708378732204,
+            -0.02785436064004898,
+            -0.027394089847803116,
+            -0.008149979636073112,
+            0.021109001711010933,
+            -0.011363944038748741,
+            0.014196994714438915,
+            0.02785436064004898,
+            -0.047614291310310364,
+            0.011840087361633778,
+            -0.0018361260881647468,
+            0.030949288979172707,
+            -0.004781268537044525,
+            -0.022267617285251617,
+            0.01896636001765728,
+            -0.01371291559189558,
+            -0.03599640354514122,
+            0.047233376652002335,
+            0.0371708907186985,
+            0.024235675111413002,
+            -0.005400254391133785,
+            0.020616987720131874,
+            -0.027552803978323936,
+            0.006423961371183395,
+            -0.0549786351621151,
+            0.03675823286175728,
+            -0.03640906140208244,
+            -0.025505388155579567,
+            -0.007392118684947491,
+            -0.014133509248495102,
+            0.050026748329401016,
+            -0.007046915125101805,
+            0.034028347581624985,
+            0.008967358618974686,
+            -0.007400054484605789,
+            0.03644080460071564,
+            -0.012355908751487732,
+            -0.02187083102762699,
+            -0.031330205500125885,
+            -0.047614291310310364,
+            -0.011848023161292076,
+            -0.0003290841996204108,
+            -0.010014872997999191,
+            -0.001375854597426951,
+            0.006765197031199932,
+            0.04707466438412666,
+            -0.00778493657708168,
+            -0.02625134587287903,
+            0.0028211467433720827,
+            0.01669674552977085,
+            -0.019315531477332115,
+            0.005578808020800352,
+            0.0038865164387971163,
+            0.01633170247077942,
+            -0.026759231463074684,
+            -0.02537841722369194,
+            0.006816779263317585,
+            -0.033488716930150986,
+            0.01680784486234188,
+            -0.03137781843543053,
+            0.0004920143401250243,
+            0.015982531011104584,
+            0.0218232162296772,
+            -0.0049836291000247,
+            -0.05637532100081444,
+            -0.04078957810997963,
+            0.01653803139925003,
+            0.015696844086050987,
+            0.03191744536161423,
+            0.0010425546206533909,
+            0.0014323965879157186,
+            0.016633259132504463,
+            0.01586349494755268,
+            -0.03358394652605057,
+            0.013173287734389305,
+            -0.026743359863758087,
+            -0.010165651328861713,
+            0.01029262226074934,
+            0.04564623534679413,
+            -0.04551926255226135,
+            -0.011030644178390503,
+            -0.012752694077789783,
+            -0.0072770509868860245,
+            -0.03983728960156441,
+            -0.013578008860349655,
+            -0.0009036795818246901,
+            -0.015046115964651108,
+            -0.015276252292096615,
+            0.012951087206602097,
+            -0.014054151251912117,
+            -0.047550804913043976,
+            -0.016839588060975075,
+            0.014530294574797153,
+            -0.023505588993430138,
+            0.014696944504976273,
+            0.02780674584209919,
+            0.01058624405413866,
+            -0.020664602518081665,
+            0.0055311936885118484,
+            0.01597459428012371,
+            0.008673736825585365,
+            0.0010921527864411473,
+            0.016014274209737778,
+            2.0304270947235636e-05,
+            0.03221900388598442,
+            -0.0020751894917339087,
+            -0.010633858852088451,
+            -0.0009309586021117866,
+            0.03212377429008484,
+            -0.015807945281267166,
+            -0.016299959272146225,
+            0.014863594435155392,
+            -0.02042653039097786,
+            -0.06027969345450401,
+            -0.007530993781983852,
+            -0.027775002643465996,
+            0.00897529348731041,
+            -0.011641694232821465,
+            -0.020029745995998383,
+            -0.02758454531431198,
+            0.0034659237135201693,
+            -0.012705080211162567,
+            0.0013252644566819072,
+            0.029028845950961113,
+            0.022394588217139244,
+            0.011038579978048801,
+            0.015585744753479958,
+            -0.001685347524471581,
+            0.0027318699285387993,
+            -0.055264320224523544,
+            0.03237771987915039,
+            -0.04529706388711929,
+            0.01939488761126995,
+            0.007729386445134878,
+            -0.010848122648894787,
+            0.02053763158619404,
+            0.027013175189495087,
+            -0.0009607175015844405,
+            0.0074794115498661995,
+            0.019410759210586548,
+            0.04355120658874512,
+            -0.015800008550286293,
+            -0.05570872128009796,
+            -0.0387897752225399,
+            -0.0019601215608417988,
+            -0.011974994093179703,
+            -0.00724927568808198,
+            0.01158614456653595,
+            -0.024426132440567017,
+            -0.004781268537044525,
+            -0.009602215141057968,
+            0.006662033032625914,
+            0.018442602828145027,
+            0.0009369103936478496,
+            0.009308594278991222,
+            -0.020934415981173515,
+            -0.003948018420487642,
+            -0.030742960050702095,
+            0.008443601429462433,
+            0.021521659567952156,
+            -0.00889593642205,
+            0.00948318000882864,
+            -0.002079157391563058,
+            -0.030362047255039215,
+            0.010792572982609272,
+            0.02791784703731537,
+            0.030187461525201797,
+            -0.01660151593387127,
+            0.021902574226260185,
+            -0.010324365459382534,
+            -0.008261079899966717,
+            -0.007530993781983852,
+            0.029647832736372948,
+            0.0678662359714508,
+            0.022029545158147812,
+            0.007546865381300449,
+            -0.02063285931944847,
+            0.0073683117516338825,
+            -0.02155340276658535,
+            0.012657465413212776,
+            -0.028171788901090622,
+            0.03431403264403343,
+            0.03621860593557358,
+            0.012800308875739574,
+            -0.0014046216383576393,
+            0.017283987253904343,
+            -0.029568474739789963,
+            -0.020823316648602486,
+            0.03093341737985611,
+            -0.03282211720943451,
+            -0.029870031401515007,
+            0.014101766049861908,
+            -0.002797339577227831,
+            0.010943351313471794,
+            0.016204729676246643,
+            -0.02920343168079853,
+            -0.042979832738637924,
+            -0.002227952005341649,
+            -0.031155617907643318,
+            -0.02052175998687744,
+            -0.012720951810479164,
+            -0.0625016912817955,
+            -0.002642593113705516,
+            -0.046630263328552246,
+            -0.014720751903951168,
+            0.013133608736097813,
+            -0.008003168739378452,
+            0.011482980102300644,
+            -0.03517109155654907,
+            0.005483579356223345,
+            -0.019966259598731995,
+            -0.026632260531187057,
+            0.015188959427177906,
+            0.045233577489852905,
+            -0.004166250582784414,
+            -0.0059676580131053925,
+            0.027187760919332504,
+            -0.039361149072647095,
+            -0.01750618778169155,
+            0.009784736670553684,
+            0.00778493657708168,
+            -0.014641394838690758,
+            -0.04009123519062996,
+            -0.021839087828993797,
+            0.014244608581066132,
+            0.019315531477332115,
+            0.021632758900523186,
+            -0.004257511347532272,
+            -0.04989977926015854,
+            -0.03520283102989197,
+            -0.030250946059823036,
+            0.039361149072647095,
+            0.0005202853353694081,
+            0.004436064977198839,
+            -0.023616688326001167,
+            -0.009062586352229118,
+            0.004531293176114559,
+            0.022712016478180885,
+            0.009253043681383133,
+            -0.028203532099723816,
+            -0.0049677579663693905,
+            0.0006284094415605068,
+            0.005114568397402763,
+            -0.030139846727252007,
+            0.025441903620958328,
+            0.0013649429893121123,
+            -0.05961309373378754,
+            0.008149979636073112,
+            -0.02926691807806492,
+            0.007344504352658987,
+            0.01409383025020361,
+            -0.010149779729545116,
+            -0.03093341737985611,
+            0.01676023006439209,
+            0.04085306078195572,
+            0.020759830251336098,
+            -0.026759231463074684,
+            0.012713016010820866,
+            0.012927279807627201,
+            -0.030917545780539513,
+            -0.04104351997375488,
+            -0.002900503808632493,
+            0.004801107570528984,
+            -0.009713315404951572,
+            -0.011221101507544518,
+            -0.002594978781417012,
+            0.03475843369960785,
+            -0.009689508005976677,
+            0.0036841558758169413,
+            0.022712016478180885,
+            0.021854959428310394,
+            -0.021680373698472977,
+            0.014577909372746944,
+            -0.01153059396892786,
+            0.033266518265008926,
+            -0.027029046788811684,
+            0.03593291714787483,
+            0.009118136949837208,
+            -0.0022636628709733486,
+            0.031441304832696915,
+            -0.00964189413934946,
+            -0.02042653039097786,
+            0.02321990206837654,
+            -0.0007915875758044422,
+            -0.02026781626045704,
+            -0.029489118605852127,
+            -0.01058624405413866,
+            -0.0069159758277237415,
+            0.026632260531187057,
+            -0.022267617285251617,
+            -0.023315131664276123,
+            0.01298283040523529,
+            0.042027547955513,
+            0.009300658479332924,
+            0.026600517332553864,
+            -0.0004915183526463807,
+            -0.00017086586740333587,
+            0.04964583367109299,
+            -0.03659952059388161,
+            0.02312467433512211,
+            0.018823517486453056,
+            -0.021743860095739365,
+            -0.01182421576231718,
+            -0.0014056136133149266,
+            0.011236973106861115,
+            -0.010284687392413616,
+            -0.006328732706606388,
+            0.003662332659587264,
+            0.005570872221142054,
+            0.0020394788589328527,
+            -0.02112487331032753,
+            0.010252944193780422,
+            0.0006199777708388865,
+            0.005812911316752434,
+            0.017204631119966507,
+            -0.02229936048388481,
+            -0.026473546400666237,
+            0.025394288823008537,
+            0.027092531323432922,
+            -0.010197394527494907,
+            0.014379516243934631,
+            -0.0027001271955668926,
+            -0.01072115171700716,
+            0.01644280180335045,
+            -0.005150279030203819,
+            0.002200176939368248,
+            0.0002386914420640096,
+            0.02650528959929943,
+            -0.022013673558831215,
+            -0.017220502719283104,
+            0.015871429815888405,
+            0.016617387533187866,
+            -0.031758733093738556,
+            0.005511354189366102,
+            0.01745857298374176,
+            0.014030344784259796,
+            -0.04240846261382103,
+            -0.0076976437121629715,
+            -0.023854760453104973,
+            -0.00869754422456026,
+            -0.01462552323937416,
+            -0.03939289227128029,
+            0.03291734680533409,
+            -0.003497666446492076,
+            -0.026917945593595505,
+            -0.010975094512104988,
+            -0.032361846417188644,
+            0.011514722369611263,
+            0.014435065910220146,
+            0.025346674025058746,
+            -0.073262520134449,
+            0.011348072439432144,
+            -0.010729086585342884,
+            0.01441125851124525,
+            -0.007975393906235695,
+            -0.008903872221708298,
+            0.02510860376060009,
+            -0.00015301050734706223,
+            0.021854959428310394,
+            0.009260979481041431,
+            0.0016089662676677108,
+            0.04285286366939545,
+            0.008134108036756516,
+            0.028140045702457428,
+            -0.016823716461658478,
+            0.038123175501823425,
+            0.013276451267302036,
+            0.0025275254156440496,
+            -0.00956253707408905,
+            -0.0061700185760855675,
+            0.03980554640293121,
+            0.06275563687086105,
+            0.00560261495411396,
+            0.03374266251921654,
+            0.0013659349642693996,
+            0.014482680708169937,
+            0.004221800714731216,
+            -0.01107825804501772,
+            -0.02980654686689377,
+            0.0040452308021485806,
+            -0.030917545780539513,
+            -0.01955360174179077,
+            -0.011752794496715069,
+            -0.002979861106723547,
+            -0.021410560235381126,
+            -0.0062454077415168285,
+            0.010768765583634377,
+            -0.01018152292817831,
+            0.044693946838378906,
+            0.009816479869186878,
+            -0.02215651609003544,
+            -0.0037297862581908703,
+            0.061136748641729355,
+            0.0048249149695038795,
+            0.03466320410370827,
+            -0.014593780040740967,
+            0.017125273123383522,
+            -0.00552325788885355,
+            -0.0011695260182023048,
+            0.006328732706606388,
+            -0.010729086585342884,
+            -0.025045117363333702,
+            0.03698043152689934,
+            0.002549348631873727,
+            0.033647432923316956,
+            0.011117937043309212,
+            -0.025362545624375343,
+            0.009554601274430752,
+            -0.012617787346243858,
+            0.015688909217715263,
+            0.01417318731546402,
+            0.0067136152647435665,
+            -0.011363944038748741,
+            0.008657865226268768,
+            -0.025299061089754105,
+            0.015300058759748936,
+            -0.0014631474623456597,
+            0.0001887212274596095,
+            0.02306118793785572,
+            -0.017093529924750328,
+            0.0060589187778532505,
+            -0.01376846618950367,
+            -0.0280130747705698,
+            0.015696844086050987,
+            0.024616587907075882,
+            0.021204231306910515,
+            -0.023077059537172318,
+            -0.011578208766877651,
+            0.04355120658874512,
+            0.013943051919341087,
+            -0.002037494909018278,
+            -0.006757261697202921,
+            -0.0017160983989015222,
+            -0.008221400901675224,
+            -0.01087986584752798,
+            -0.012554301880300045,
+            -0.006435865070670843,
+            -0.011998801492154598,
+            0.0490427203476429,
+            -0.014903273433446884,
+            0.00878483708947897,
+            0.011038579978048801,
+            -0.015276252292096615,
+            -0.002704094862565398,
+            0.0030215235892683268,
+            -0.007769065443426371,
+            -0.0056224544532597065,
+            -0.0039757932536304,
+            0.030489018186926842,
+            0.016506288200616837,
+            0.024600718170404434,
+            -0.037900976836681366,
+            -0.010705280117690563,
+            0.002364843152463436,
+            0.02166450209915638,
+            0.021029645577073097,
+            0.005872429348528385,
+            -0.0029600218404084444,
+            -0.0015623439103364944,
+            -0.02888600341975689,
+            -0.015736524015665054,
+            -0.017093529924750328,
+            0.03459971770644188,
+            -0.025394288823008537,
+            0.03313954547047615,
+            0.02123597450554371,
+            0.008530894294381142,
+            0.0052812183275818825,
+            -0.01511753723025322,
+            -0.034726690500974655,
+            0.015379415825009346,
+            -0.01519689429551363,
+            0.0006452728412114084,
+            0.0290923323482275,
+            -0.04374166205525398,
+            -0.04567797854542732,
+            -0.003692091442644596,
+            0.016665002331137657,
+            0.024553103372454643,
+            0.001200276892632246,
+            0.0011070322943851352,
+            0.012625723145902157,
+            -0.012157515622675419,
+            -0.011808344163000584,
+            0.004654297139495611,
+            0.01880764588713646,
+            -0.005860525649040937,
+            -0.0061501795426011086,
+            0.025981532409787178,
+            0.00932446587830782,
+            -0.03926591947674751,
+            -0.002416425384581089,
+            0.006193825975060463,
+            0.022553302347660065,
+            0.02301357500255108,
+            -0.03142543137073517,
+            0.015855558216571808,
+            0.013189158402383327,
+            0.04459872096776962,
+            -0.012990766204893589,
+            0.008388050831854343,
+            0.038662806153297424,
+            -0.005558968521654606,
+            0.03082231804728508,
+            -0.004634457640349865,
+            -0.03345697373151779,
+            0.02328338846564293,
+            0.0011100082192569971,
+            -0.01831563003361225,
+            -0.012411458417773247,
+            0.007070722058415413,
+            0.004777300637215376,
+            0.027886103838682175,
+            0.025457775220274925,
+            0.0033230807166546583,
+            0.028965361416339874,
+            0.038885004818439484,
+            0.0057176826521754265,
+            0.013236773200333118,
+            0.007237372454255819,
+            -0.015069923363626003,
+            -0.00490427203476429,
+            -0.006729486398398876,
+            0.015323866158723831,
+            -0.04002774879336357,
+            0.020061487331986427,
+            -0.003616702277213335,
+            0.01982341706752777,
+            -0.00529312202706933,
+            -0.012435265816748142,
+            -0.0007657965179532766,
+            -0.02220413088798523,
+            0.02391824498772621,
+            0.0025989466812461615,
+            0.001375854597426951,
+            -0.021077260375022888,
+            -0.024426132440567017,
+            0.00475746113806963,
+            -0.02328338846564293,
+            0.011260779574513435,
+            0.026489417999982834,
+            0.010356108658015728,
+            0.003874612972140312,
+            -0.03713914752006531,
+            0.02123597450554371,
+            0.02979067526757717,
+            -0.0021386751905083656,
+            0.017125273123383522,
+            0.030044617131352425,
+            -0.021886702626943588,
+            -0.015942851081490517,
+            0.015934916213154793,
+            -0.02715601772069931,
+            -0.0019750010687857866,
+            -0.0002596466802060604,
+            -0.05082032084465027,
+            -0.04856657609343529,
+            0.013125672936439514,
+            -0.003858741605654359,
+            0.015379415825009346,
+            -0.010483079589903355,
+            -0.005793072283267975,
+            0.004912207834422588,
+            -0.0005872429464943707,
+            0.006527125835418701,
+            -0.0059676580131053925,
+            0.0056264218874275684,
+            -0.013498651795089245,
+            -0.008515022695064545,
+            -0.02790197543799877,
+            -0.005308993626385927,
+            -0.018712416291236877,
+            -0.013514523394405842,
+            -0.027441702783107758,
+            0.020172588527202606,
+            -0.00964189413934946,
+            -0.012419394217431545,
+            0.019871031865477562,
+            0.013101865537464619,
+            0.020347174257040024,
+            0.010197394527494907,
+            -0.003904371988028288,
+            0.028092432767152786,
+            -0.03409183397889137,
+            0.03615511953830719,
+            0.006328732706606388,
+            0.02677510306239128,
+            -0.002079157391563058,
+            -0.020680474117398262,
+            0.009673637337982655,
+            0.03437751904129982,
+            0.0016109502175822854,
+            -0.013808144256472588,
+            -0.015260380692780018,
+            -0.004376546945422888,
+            -0.005059018265455961,
+            -0.007955554872751236,
+            -0.008872129954397678,
+            0.010744958184659481,
+            -0.0036841558758169413,
+            0.001297489390708506,
+            -0.006884233094751835,
+            0.04193231835961342,
+            0.011713115498423576,
+            -0.004539228975772858,
+            -0.023267516866326332,
+            -0.02758454531431198,
+            0.01999800279736519,
+            -0.00280130747705698,
+            -0.024616587907075882,
+            0.026378316804766655,
+            -0.014054151251912117,
+            -0.004654297139495611,
+            0.00723340455442667,
+            -0.018077559769153595,
+            0.013816080056130886,
+            -0.021362945437431335,
+            0.005979561246931553,
+            -0.041837092489004135,
+            -0.011959122493863106,
+            -0.03253643214702606,
+            -0.005856557749211788,
+            0.0177760012447834,
+            -0.007447668816894293,
+            0.02441026084125042,
+            -0.016728486865758896,
+            -0.0015425046440213919,
+            -0.007332601118832827,
+            -0.013887501321732998,
+            -0.0088880006223917,
+            -0.0050947293639183044,
+            -0.005245507694780827,
+            0.0191250741481781,
+            -0.002251759171485901,
+            -0.02144230343401432,
+            0.027870232239365578,
+            -0.024648331105709076,
+            1.071011593012372e-05,
+            -0.04659852012991905,
+            -0.03640906140208244,
+            -0.04085306078195572,
+            0.02375953085720539,
+            -0.0169665589928627,
+            -0.04942363500595093,
+            -0.009657765738666058,
+            0.00434877211228013,
+            0.0159031730145216,
+            -0.02490227483212948,
+            -0.02434677444398403,
+            0.022220002487301826,
+            -0.011086193844676018,
+            0.0034659237135201693,
+            -0.01642693020403385,
+            -0.011760730296373367,
+            -0.017157016322016716,
+            0.04091654717922211,
+            -0.02348971739411354,
+            0.022870730608701706,
+            -0.01492708083242178,
+            0.007296890020370483,
+            -0.007273083087056875,
+            -0.015450838021934032,
+            0.03383788838982582,
+            0.025299061089754105,
+            -0.04713814705610275,
+            0.04018646106123924,
+            0.020982030779123306,
+            0.03086993284523487,
+            -0.04459872096776962,
+            0.016617387533187866,
+            -0.030187461525201797,
+            8.307702228194103e-05,
+            0.04063086211681366,
+            0.004412257578223944,
+            0.009856157936155796,
+            -0.011498851701617241,
+            -0.020172588527202606,
+            0.003795255906879902,
+            0.010221200995147228,
+            0.000911119335796684,
+            -0.02791784703731537,
+            -0.007130240090191364,
+            0.007455604616552591,
+            0.02080744504928589,
+            0.00272790202870965,
+            0.02537841722369194,
+            0.03475843369960785,
+            -0.014935015700757504,
+            0.006967558059841394,
+            -0.0458049476146698,
+            0.011903572827577591,
+            -0.005582775454968214,
+            0.01741096004843712,
+            0.025314930826425552,
+            0.02214064635336399,
+            0.030044617131352425,
+            -0.03748831897974014,
+            0.024648331105709076,
+            0.0037516094744205475,
+            -0.005562936421483755,
+            0.0323459766805172,
+            0.0088880006223917,
+            0.03183808922767639,
+            -0.015355609357357025,
+            -0.011276651173830032,
+            0.007273083087056875,
+            0.02375953085720539,
+            -0.017442701384425163,
+            0.01669674552977085,
+            -0.005305025726556778,
+            0.04945537820458412,
+            0.013831951655447483,
+            -0.03488540276885033,
+            -0.027298860251903534,
+            -0.020918546244502068,
+            -0.00918955821543932,
+            0.012863794341683388,
+            0.01112587284296751,
+            -0.01129252277314663,
+            -0.0312667191028595,
+            0.02382301725447178,
+            -0.010998900979757309,
+            -0.014530294574797153,
+            -0.04132920503616333,
+            -0.007054850924760103,
+            -0.017823616042733192,
+            0.0024322967510670424,
+            -0.003579007461667061,
+            -0.01676023006439209,
+            -0.0052970899268984795,
+            0.006427929271012545,
+            0.025314930826425552,
+            -0.025743460282683372,
+            -0.008935615420341492,
+            -0.038662806153297424,
+            -0.005039179231971502,
+            -0.0023509557358920574,
+            0.01729985885322094,
+            -0.002940182574093342,
+            -0.038726288825273514,
+            0.019490117207169533,
+            0.010649729520082474,
+            -0.01433190144598484,
+            0.043932121247053146,
+            -0.036345575004816055,
+            0.005872429348528385,
+            0.03215551748871803,
+            -0.00173097790684551,
+            -0.03086993284523487,
+            -0.0017270100070163608,
+            0.007273083087056875,
+            0.0021049485076218843,
+            0.002737821778282523,
+            0.018077559769153595,
+            0.031123874709010124,
+            0.01276856567710638,
+            0.012633658945560455,
+            0.06977080553770065,
+            -0.00043224848923273385,
+            -0.03086993284523487,
+            0.02285485900938511,
+            -0.001691299257799983,
+            -0.01067353691905737,
+            -0.01376846618950367,
+            0.023045316338539124,
+            -0.02139468863606453,
+            0.011522658169269562,
+            -0.019109202548861504,
+            -0.0004731670196633786,
+            -0.0344727486371994,
+            -0.030108103528618813,
+            -0.01239558681845665,
+            0.022029545158147812,
+            -0.04104351997375488,
+            -0.029822418466210365,
+            0.018283888697624207,
+            0.017712516710162163,
+            -0.004582875408232212,
+            0.0018589412793517113,
+            -0.001100088469684124,
+            0.009340336546301842,
+            0.01761728711426258,
+            -0.0020989966578781605,
+            -0.007519090082496405,
+            0.009872029535472393,
+            -0.0074952831491827965,
+            -0.0066064829006791115,
+            0.02123597450554371,
+            -0.07034218311309814,
+            -0.04409083351492882,
+            0.024045217782258987,
+            -0.008626122027635574,
+            0.002261678921058774,
+            0.022886602208018303,
+            -6.004484384902753e-05,
+            -0.01525244489312172,
+            -0.016887202858924866,
+            0.03463146090507507,
+            -0.012101965956389904,
+            0.022759631276130676,
+            -0.0015266332775354385,
+            -0.0020989966578781605,
+            0.01314948033541441,
+            0.011879765428602695,
+            -0.014736623503267765,
+            0.0018877083202823997,
+            0.015284187160432339,
+            0.014760429970920086,
+            0.00878483708947897,
+            0.017283987253904343,
+            -0.008967358618974686,
+            -0.004094828851521015,
+            -0.0009175670566037297,
+            0.001588135026395321,
+            -0.0019204431446269155,
+            0.012839986942708492,
+            0.031980931758880615,
+            0.0013272484065964818,
+            0.03498063236474991,
+            0.029060589149594307,
+            0.02085505984723568,
+            0.0036960593424737453,
+            -0.004614618606865406,
+            -0.009451436810195446,
+            0.023584945127367973,
+            0.016585644334554672,
+            0.010054551064968109,
+            0.0099672582000494,
+            0.005459771957248449,
+            -0.022505687549710274,
+            0.006971525959670544,
+            -0.0042813182808458805,
+            -0.008380115032196045,
+            0.0071500795893371105,
+            0.007320697419345379,
+            -0.018775902688503265,
+            0.012800308875739574,
+            0.013117737136781216,
+            0.0022041448391973972,
+            0.02229936048388481,
+            -0.01833150163292885,
+            -0.012697144411504269,
+            0.004618586506694555,
+            -0.012705080211162567,
+            0.025679973885416985,
+            -0.0030631860718131065,
+            0.0019531778525561094,
+            0.011387751437723637,
+            0.07459572702646255,
+            -0.026854461058974266,
+            -0.0014829868450760841,
+            -0.015522259287536144,
+            -0.02026781626045704,
+            0.0090229082852602,
+            -0.023156417533755302,
+            0.022553302347660065,
+            0.01822040230035782,
+            -0.004110700450837612,
+            0.04574146121740341,
+            0.0020712215919047594,
+            0.019966259598731995,
+            0.024315031245350838,
+            -0.009197494015097618,
+            -0.02053763158619404,
+            0.0045908112078905106,
+            0.012617787346243858,
+            0.00024241130449809134,
+            0.010252944193780422,
+            0.023902375251054764,
+            0.013943051919341087,
+            -0.003702011192217469,
+            -0.003840886289253831,
+            -0.021299459040164948,
+            0.025949789211153984,
+            0.010078358463943005,
+            -0.025124475359916687,
+            -0.038599319756031036,
+            0.0003905859775841236,
+            0.01239558681845665,
+            0.01236384455114603,
+            0.023632559925317764,
+            0.0033786308486014605,
+            -0.011554401367902756,
+            0.012379716150462627,
+            -0.014101766049861908,
+            0.003928178921341896,
+            0.0042813182808458805,
+            -0.007185790222138166,
+            -0.010245008394122124,
+            0.011348072439432144,
+            -0.004729686304926872,
+            0.003174286102876067,
+            0.00020422067609615624,
+            0.0031484949868172407,
+            -0.010038679465651512,
+            -0.010602115653455257,
+            -0.012276551686227322,
+            -0.0029540699906647205,
+            -0.005678004119545221,
+            0.032409459352493286,
+            -0.013109801337122917,
+            -0.003598846960812807,
+            -0.01166550163179636,
+            0.030250946059823036,
+            -0.001897627953439951,
+            -0.006412058137357235,
+            -0.018553702160716057,
+            0.004301157779991627,
+            -0.000162558164447546,
+            -0.009760930202901363,
+            -0.011117937043309212,
+            0.006979461759328842,
+            -0.01998213119804859,
+            -0.008459472097456455,
+            -0.0019710331689566374,
+            0.009173686616122723,
+            0.0026842555962502956,
+            -0.014958823099732399,
+            -0.014149379916489124,
+            0.021966060623526573,
+            -0.011562337167561054,
+            -0.014244608581066132,
+            -0.0010862010531127453,
+            -0.013490715995430946,
+            0.013800208456814289,
+            -0.007900004275143147,
+            -0.0008501135162077844,
+            -0.02144230343401432,
+            0.021093131974339485,
+            0.02333100326359272,
+            -0.017109401524066925,
+            -0.010959222912788391,
+            -0.031060390174388885,
+            -0.0005088777397759259,
+            0.0037278023082762957,
+            0.009443501010537148,
+            0.03628208860754967,
+            0.013744658790528774,
+            0.004245607648044825,
+            0.011459172703325748,
+            -0.007487347349524498,
+            0.008142043836414814,
+            0.009721251204609871,
+            -0.010014872997999191,
+            0.0011248876107856631,
+            2.0846751795033924e-05,
+            -0.03409183397889137,
+            -0.007054850924760103,
+            -0.036186862736940384,
+            -0.0048606256023049355,
+            0.005896236281841993,
+            -0.009816479869186878,
+            -0.0013966858386993408,
+            0.007975393906235695,
+            -0.006138275843113661,
+            -0.015482580289244652,
+            0.000179917537025176,
+            0.008348372764885426,
+            -0.008300757966935635,
+            -0.0010207314044237137,
+            -0.008792772889137268,
+            0.016299959272146225,
+            0.022870730608701706,
+            0.02295008860528469,
+            0.007828583009541035,
+            0.0005460763932205737,
+            -0.004082925617694855,
+            0.007693675812333822,
+            -0.01360181625932455,
+            -0.013466908596456051,
+            0.004305125679820776,
+            0.00950698647648096,
+            -0.0055311936885118484,
+            0.007423861417919397,
+            0.013490715995430946,
+            -0.0029421665240079165,
+            0.018093431368470192,
+            0.00622160080820322,
+            -0.004051182419061661,
+            0.008824515156447887,
+            -0.004019439686089754,
+            0.005578808020800352,
+            -0.008300757966935635,
+            -0.004253543447703123,
+            0.007943650707602501,
+            0.004217832814902067,
+            0.0149508873000741,
+            -0.0032238843850791454,
+            -0.020601116120815277,
+            -0.0033369683660566807,
+            0.04586843401193619,
+            -0.00028419779846444726,
+            0.013958923518657684,
+            -0.020680474117398262,
+            -0.013316130265593529,
+            0.01080050878226757,
+            0.014030344784259796,
+            0.010919543914496899,
+            -0.015125473029911518,
+            -0.016680873930454254,
+            0.02360081672668457,
+            0.00435274001210928,
+            -0.005451836157590151,
+            -0.035329803824424744,
+            0.0036782040260732174,
+            -0.03786923363804817,
+            0.02215651609003544,
+            0.017855359241366386,
+            -0.0035036182962358,
+            -0.004701911471784115,
+            0.017474444583058357,
+            -0.013197094202041626,
+            0.020299559459090233,
+            -0.01637931540608406,
+            0.019632959738373756,
+            0.0002527029428165406,
+            -0.006864393595606089,
+            -0.01676023006439209,
+            -0.02053763158619404,
+            0.004059118218719959,
+            -0.013578008860349655,
+            0.04326551780104637,
+            0.024981630966067314,
+            0.0067334542982280254,
+            0.0017498251982033253,
+            -0.027790874242782593,
+            -0.013141544535756111,
+            0.00934827234596014,
+            0.019109202548861504,
+            -0.010943351313471794,
+            -0.01592698134481907,
+            -0.00505108293145895,
+            0.013895437121391296,
+            -0.0049518863670527935,
+            -0.025886302813887596,
+            0.005253443494439125,
+            -0.014784237369894981,
+            0.013427230529487133,
+            -0.010919543914496899,
+            0.008118236437439919,
+            0.016014274209737778,
+            -0.00552325788885355,
+            -0.0014839787036180496,
+            0.03517109155654907,
+            0.012308293953537941,
+            -0.01322883740067482,
+            0.006781068630516529,
+            -0.018791774287819862,
+            -0.01880764588713646,
+            0.03250468894839287,
+            -0.01503024436533451,
+            0.01988690160214901,
+            0.008991165086627007,
+            -0.0242515467107296,
+            0.017109401524066925,
+            -0.021696245297789574,
+            0.008610251359641552,
+            -0.008681672625243664,
+            0.0042813182808458805,
+            -0.016728486865758896,
+            -0.02268027327954769,
+            -0.01619679480791092,
+            -0.02526731789112091,
+            0.01734747365117073,
+            -0.013125672936439514,
+            -0.0023588913027197123,
+            -0.013514523394405842,
+            0.025584746152162552,
+            0.0040392791852355,
+            0.024203931912779808,
+            0.01622060127556324,
+            0.003977777436375618,
+            0.00544390082359314,
+            -0.033171288669109344,
+            0.028520960360765457,
+            0.002400553785264492,
+            0.020616987720131874,
+            0.013919244520366192,
+            0.006296989973634481,
+            0.0019809529185295105,
+            0.007892068475484848,
+            0.031108003109693527,
+            0.004602714907377958,
+            0.0011982930591329932,
+            0.03158414736390114,
+            -0.00918955821543932,
+            0.013665301725268364,
+            -2.7558011424844153e-05,
+            0.00886419415473938,
+            -0.003112784354016185,
+            -0.009538729675114155,
+            0.02866380289196968,
+            0.0010326349874958396,
+            -0.00019132513261865824,
+            0.02355320379137993,
+            0.0034500521142035723,
+            -0.04018646106123924,
+            -0.003662332659587264,
+            -0.020950287580490112,
+            -0.016522159799933434,
+            0.014554101973772049,
+            0.005908139981329441,
+            0.002900503808632493,
+            0.021966060623526573,
+            0.0004959821817465127,
+            0.01075289398431778,
+            -0.008269015699625015,
+            -0.016617387533187866,
+            -0.021839087828993797,
+            -0.0036881237756460905,
+            0.02839398942887783,
+            0.011308394372463226,
+            -0.011268715374171734,
+            0.015712715685367584,
+            -0.006050982978194952,
+            -0.00435274001210928,
+            0.007110401056706905,
+            -0.0006695759948343039,
+            -0.010951287113130093,
+            0.0014512438792735338,
+            0.0062454077415168285,
+            -0.02175973169505596,
+            0.005872429348528385,
+            -0.0076222545467317104,
+            -0.0003261083038523793,
+            -0.008459472097456455,
+            -0.020013874396681786,
+            -0.011165550909936428,
+            -0.0035809914115816355,
+            0.03685346245765686,
+            0.013347873464226723,
+            -0.020601116120815277,
+            -0.014323965646326542,
+            0.0003261083038523793,
+            -0.0031385754700750113,
+            -0.0008193625835701823,
+            -0.0020930448081344366,
+            -0.0008977277902886271,
+            -0.014569973573088646,
+            0.006265247240662575,
+            -0.0006269214791245759,
+            0.010871930047869682,
+            0.011776601895689964,
+            0.007189758121967316,
+            -0.01606188714504242,
+            -0.0005068937898613513,
+            -0.02434677444398403,
+            -0.023077059537172318,
+            -0.004110700450837612,
+            -0.0026505289133638144,
+            -0.017490316182374954,
+            -0.0022894537542015314,
+            0.014601715840399265,
+            -0.0138795655220747,
+            0.029774803668260574,
+            -0.0006576724117621779,
+            -0.002549348631873727,
+            0.044757433235645294,
+            0.012998702004551888,
+            -0.00030775697086937726,
+            -0.0021724021062254906,
+            0.01917268894612789,
+            -0.005797040183097124,
+            -0.0029203433077782393,
+            0.005642293486744165,
+            0.006047015078365803,
+            0.013704979792237282,
+            0.026584645733237267,
+            -0.02090267464518547,
+            0.0024203930515795946,
+            -0.006047015078365803,
+            0.015387351624667645,
+            -0.014244608581066132,
+            0.0002603906614240259,
+            -0.02155340276658535,
+            -0.01613330841064453,
+            -0.017696645110845566,
+            0.006626322399824858,
+            -0.0072770509868860245,
+            -0.010467207990586758,
+            -0.0016526127001270652,
+            -0.012173387221992016,
+            0.012776501476764679,
+            0.020870931446552277,
+            -0.02936214581131935,
+            0.0199345164000988,
+            -0.006419993471354246,
+            0.0149508873000741,
+            -0.00755876861512661,
+            0.015149280428886414,
+            -0.0020315430592745543,
+            -0.013212965801358223,
+            0.006820747163146734,
+            -0.03752006217837334,
+            -0.017633158713579178,
+            -0.0064953831024467945,
+            0.002751709194853902,
+            -0.03152066096663475,
+            -0.0048764972016215324,
+            0.030901676043868065,
+            0.0035809914115816355,
+            -0.012578108347952366,
+            0.002473959233611822,
+            0.025299061089754105,
+            0.002876696875318885,
+            0.015268316492438316,
+            0.002487846650183201,
+            -0.016887202858924866,
+            -0.004340836312621832,
+            0.02521970309317112,
+            -0.011848023161292076,
+            -0.01026881579309702,
+            -0.024807045236229897,
+            0.03605988994240761,
+            -0.008872129954397678,
+            -0.00140065373852849,
+            0.023949988186359406,
+            -0.006562836468219757,
+            0.005705779418349266,
+            0.0021803376730531454,
+            -0.020553503185510635,
+            0.0231722891330719,
+            0.029520859941840172,
+            0.004197993315756321,
+            -0.02936214581131935,
+            -0.004622553940862417,
+            -0.003910323604941368,
+            0.003116752253845334,
+            0.00015648239059373736,
+            0.002900503808632493,
+            0.002198193222284317,
+            -0.009864093735814095,
+            -0.02079157344996929,
+            -0.00028742168797180057,
+            -0.015720652416348457,
+            -0.010903672315180302,
+            0.018553702160716057,
+            -0.006102565210312605,
+            0.015577808953821659,
+            0.01376846618950367,
+            0.009610150940716267,
+            0.0057494258508086205,
+            0.020886803045868874,
+            0.02893361821770668,
+            -0.014958823099732399,
+            0.00951492227613926,
+            -0.017490316182374954,
+            0.01245113741606474,
+            0.007070722058415413,
+            -0.006669968832284212,
+            -0.03834537789225578,
+            0.008951487019658089,
+            0.007423861417919397,
+            0.014276351779699326,
+            -0.013395487330853939,
+            -0.024775303900241852,
+            -0.036885205656290054,
+            -0.016665002331137657,
+            0.003930163104087114,
+            0.005725618451833725,
+            0.005801007617264986,
+            -0.008522958494722843,
+            -0.005269315093755722,
+            -0.004471775609999895,
+            0.004209897015243769,
+            0.013435166329145432,
+            -0.049106206744909286,
+            -0.019490117207169533,
+            0.006205729208886623,
+            -0.01676023006439209,
+            0.016363445669412613,
+            0.007431797217577696,
+            -0.013236773200333118,
+            0.009737122803926468,
+            -0.016117436811327934,
+            0.000434232410043478,
+            -0.00926891528069973,
+            0.001266738516278565,
+            -0.004194025415927172,
+            0.008967358618974686,
+            0.019585344940423965,
+            0.034948889166116714,
+            -0.007753193844109774,
+            0.003061202121898532,
+            -0.008848322555422783,
+            0.0007211581105366349,
+            0.01029262226074934,
+            0.006332700606435537,
+            -0.005340736359357834,
+            -0.00027403017156757414,
+            -0.02015671692788601,
+            0.015450838021934032,
+            -0.020140845328569412,
+            0.00280130747705698,
+            0.00023273965052794665,
+            -0.0029143914580345154,
+            -0.0019859126769006252,
+            -0.008102364838123322,
+            -0.004928078968077898,
+            -0.007935714907944202,
+            -0.0019630976021289825,
+            0.02042653039097786,
+            -0.02187083102762699,
+            0.00217637000605464,
+            0.0018648931290954351,
+            0.004745557904243469,
+            -0.027870232239365578,
+            -0.0051859901286661625,
+            -0.009705379605293274,
+            -0.014157315716147423,
+            -0.023204030469059944,
+            0.026219602674245834,
+            0.016228538006544113,
+            0.007566704414784908,
+            0.004678104072809219,
+            0.0022954056039452553,
+            -0.0010653698118403554,
+            -0.013657365925610065,
+            0.003325064666569233,
+            -0.001118935877457261,
+            -0.0026683842297643423,
+            -0.0033111772499978542,
+            0.024743560701608658,
+            -0.014823916368186474,
+            0.008634057827293873,
+            -0.0014790189452469349,
+            0.0073683117516338825,
+            0.005900204181671143,
+            0.0121257733553648,
+            0.012030544690787792,
+            0.019966259598731995,
+            0.012117837555706501,
+            0.014593780040740967,
+            0.06742183864116669,
+            -0.0058248150162398815,
+            -0.0032457076013088226,
+            0.0031544468365609646,
+            -0.02080744504928589,
+            -0.01902984455227852,
+            0.004638425540179014,
+            -0.019791673868894577,
+            0.029504990205168724,
+            -0.00918955821543932,
+            0.009713315404951572,
+            0.005725618451833725,
+            -0.007066754624247551,
+            0.017982330173254013,
+            -0.002755677094683051,
+            -0.002307309303432703,
+            0.007927779108285904,
+            0.01438745204359293,
+            0.034028347581624985,
+            0.014681072905659676,
+            -0.016395187005400658,
+            0.0026842555962502956,
+            -0.02252155914902687,
+            0.0028588413260877132,
+            -0.011482980102300644,
+            0.005328832659870386,
+            -0.013744658790528774,
+            0.026584645733237267,
+            0.0011318314354866743,
+            0.02661638893187046,
+            -0.016395187005400658,
+            0.0015365529106929898,
+            -0.014522358775138855,
+            0.0007335576810874045,
+            0.015593680553138256,
+            0.017982330173254013,
+            -0.0038567576557397842,
+            0.006729486398398876,
+            -0.013657365925610065,
+            0.00803094357252121,
+            0.014189058914780617,
+            0.015014373697340488,
+            0.001624837750568986,
+            -0.011340136639773846,
+            -0.011895637027919292,
+            0.02215651609003544,
+            0.0004414241702761501,
+            0.006118436343967915,
+            0.012459073215723038,
+            -0.014474744908511639,
+            0.00011543985601747409,
+            -0.005269315093755722,
+            0.01599840261042118,
+            -0.008026976138353348,
+            0.004376546945422888,
+            -0.006118436343967915,
+            -0.019379016011953354,
+            -0.005495482590049505,
+            -0.02252155914902687,
+            0.01718875952064991,
+            0.00046721522812731564,
+            0.00801110453903675,
+            -0.021839087828993797,
+            -0.01739508844912052,
+            -0.006876297295093536,
+            0.015379415825009346,
+            2.8162488888483495e-05,
+            0.010919543914496899,
+            0.012086094357073307,
+            0.01042752992361784,
+            -0.007400054484605789,
+            0.0004890384734608233,
+            0.02215651609003544,
+            -0.010316429659724236,
+            0.010395786724984646,
+            0.00739608658477664,
+            -0.005435965023934841,
+            -0.032250747084617615,
+            -0.004301157779991627,
+            -0.006820747163146734,
+            0.020236073061823845,
+            0.002410473534837365,
+            -0.01492708083242178,
+            0.009768865071237087,
+            -0.0409482903778553,
+            -0.009451436810195446,
+            -0.0027140146121382713,
+            0.025092732161283493,
+            0.011165550909936428,
+            -0.0069159758277237415,
+            0.01950598880648613,
+            0.008753093890845776,
+            0.013117737136781216,
+            -0.0013619671808555722,
+            -0.007324665319174528,
+            0.0036206701770424843,
+            -0.002962005790323019,
+            0.0028588413260877132,
+            -0.009959322400391102,
+            0.002946134191006422,
+            0.015561937354505062,
+            0.004174186382442713,
+            -0.011411558836698532,
+            -0.005983529146760702,
+            -0.015363545157015324,
+            -0.016117436811327934,
+            0.0026862395461648703,
+            0.00037248263834044337,
+            0.011919444426894188,
+            -0.007903972640633583,
+            -0.004209897015243769,
+            -0.015887301415205002,
+            0.019632959738373756,
+            0.0012320197420194745,
+            0.0009061594610102475,
+            -0.00824520830065012,
+            0.003061202121898532,
+            -0.012657465413212776,
+            -0.007273083087056875,
+            0.008745158091187477,
+            -0.00021451229986269027,
+            -0.0080864941701293,
+            0.024600718170404434,
+            -0.02445787377655506,
+            0.0049677579663693905,
+            -0.010149779729545116,
+            -0.028108302503824234,
+            -0.00940382294356823,
+            -0.03282211720943451,
+            -0.013133608736097813,
+            0.0039202431216835976,
+            -0.011863894760608673,
+            -0.007820647209882736,
+            0.01556987315416336,
+            -0.008074590004980564,
+            -0.005023307632654905,
+            0.015982531011104584,
+            0.003579007461667061,
+            0.0160380806773901,
+            -0.005761329084634781,
+            0.0027715484611690044,
+            -0.01750618778169155,
+            -0.006003368645906448,
+            -0.01463345903903246,
+            0.009824415668845177,
+            0.0016952671576291323,
+            0.013078059069812298,
+            -0.017093529924750328,
+            0.02106138877570629,
+            -0.010006937198340893,
+            0.00474159000441432,
+            -0.03507586196064949,
+            -0.01231622975319624,
+            0.017172887921333313,
+            0.016165051609277725,
+            -0.006003368645906448,
+            -0.00700326869264245,
+            -0.020759830251336098,
+            -0.010744958184659481,
+            -0.014681072905659676,
+            -0.02753693237900734,
+            0.011903572827577591,
+            0.00038339424645528197,
+            0.021188359707593918,
+            0.004142443183809519,
+            0.030457274988293648,
+            0.013046315871179104,
+            0.019839288666844368,
+            -0.0034302128478884697,
+            0.0011020724195986986,
+            -0.0038785808719694614,
+            0.015561937354505062,
+            -0.005804975517094135,
+            -0.012292423285543919,
+            -0.005741490051150322,
+            0.0231722891330719,
+            -0.009308594278991222,
+            0.002333100186660886,
+            0.005908139981329441,
+            0.024553103372454643,
+            0.0038726290222257376,
+            0.01714114472270012,
+            0.004392418544739485,
+            -0.002789404010400176,
+            0.03647254779934883,
+            -0.0027021111454814672,
+            -0.021854959428310394,
+            -0.0009011996444314718,
+            0.011705179698765278,
+            -0.021950189024209976,
+            -0.008856258355081081,
+            0.001166550093330443,
+            -0.022172387689352036,
+            -0.015942851081490517,
+            -2.3326663722400554e-05,
+            -0.00801904033869505,
+            0.027330603450536728,
+            0.0055311936885118484,
+            -0.008594379760324955,
+            -0.016522159799933434,
+            0.010705280117690563,
+            0.025997404009103775,
+            -0.0008962398278526962,
+            0.015173087827861309,
+            -0.014776301570236683,
+            -0.01761728711426258,
+            -0.0054994504898786545,
+            0.013681173324584961,
+            0.004182121716439724,
+            -0.0047336542047560215,
+            -0.05977180600166321,
+            -0.01815691590309143,
+            0.006086693610996008,
+            0.002291437704116106,
+            -0.026394188404083252,
+            -0.008388050831854343,
+            -0.017823616042733192,
+            -0.008911808021366596,
+            0.0029639897402375937,
+            0.02285485900938511,
+            -0.008538829162716866,
+            0.011689308099448681,
+            0.019744059070944786,
+            -0.01745857298374176,
+            -0.0007796840509399772,
+            0.00395992211997509,
+            -0.019379016011953354,
+            -0.005431997124105692,
+            0.0030393789056688547,
+            -0.018188659101724625,
+            -0.005614518653601408,
+            -0.0011754778679460287,
+            -0.007602415047585964,
+            0.012046415358781815,
+            -0.017601415514945984,
+            0.022394588217139244,
+            0.004194025415927172,
+            -0.011681373231112957,
+            -0.017744259908795357,
+            -0.03169524669647217,
+            -0.007741290144622326,
+            -0.002170418156310916,
+            -0.008713414892554283,
+            0.011149680241942406,
+            0.019267916679382324,
+            0.013760530389845371,
+            0.006173986475914717,
+            0.002852889709174633,
+            0.004761429037898779,
+            -2.9216451366664842e-05,
+            0.0005525241722352803,
+            0.003467907663434744,
+            -0.008530894294381142,
+            -0.0003127167874481529,
+            0.0077889044769108295,
+            -0.013506587594747543,
+            -0.021537531167268753,
+            0.009467308409512043,
+            0.03605988994240761,
+            -0.011411558836698532,
+            0.0038865164387971163,
+            -0.005618486553430557,
+            0.007566704414784908,
+            0.006630290299654007,
+            0.03012397512793541,
+            0.01655390113592148,
+            -0.022220002487301826,
+            0.00940382294356823,
+            -0.010959222912788391,
+            0.01425254438072443,
+            -0.0047495258040726185,
+            0.01961708813905716,
+            0.016395187005400658,
+            0.01586349494755268,
+            0.008181722834706306,
+            0.004813011270016432,
+            -0.0101021658629179,
+            -0.0033210967667400837,
+            0.014879466034471989,
+            -0.00872928649187088,
+            -0.011236973106861115,
+            0.0004543196992017329,
+            0.006316829472780228,
+            -0.01276856567710638,
+            -0.0160380806773901,
+            0.0034202933311462402,
+            0.009364143945276737,
+            0.03304431959986687,
+            -0.0036543968599289656,
+            0.003364743199199438,
+            -0.00654696486890316,
+            -0.013355808332562447,
+            -0.02166450209915638,
+            -0.02133120223879814,
+            0.012347972951829433,
+            -0.00739608658477664,
+            -0.0011437350185588002,
+            -0.014363644644618034,
+            -0.020997902378439903,
+            0.0035809914115816355,
+            -0.01441125851124525,
+            -0.00034569960553199053,
+            -0.002469991333782673,
+            -0.0007295898394659162,
+            -0.0028747129254043102,
+            0.0038369183894246817,
+            0.02133120223879814,
+            -0.016299959272146225,
+            0.013141544535756111,
+            0.005614518653601408,
+            0.0039698416367173195,
+            0.02542603202164173,
+            0.007455604616552591,
+            0.009118136949837208,
+            -0.016569772735238075,
+            0.025251446291804314,
+            -0.027600416913628578,
+            -0.014617587439715862,
+            -0.010030743665993214,
+            0.021188359707593918,
+            -0.0049836291000247,
+            0.011030644178390503,
+            0.016665002331137657,
+            0.005384382791817188,
+            -0.001797439530491829,
+            0.026108503341674805,
+            0.02069634571671486,
+            -0.001594086759723723,
+            0.008681672625243664,
+            -0.0005287170060910285,
+            0.02274375967681408,
+            -0.005920043680816889,
+            -0.025156216695904732,
+            0.011522658169269562,
+            0.009332400746643543,
+            0.0051701185293495655,
+            -0.0014075974468141794,
+            -0.013887501321732998,
+            -0.012213066220283508,
+            -0.009499051608145237,
+            -0.0177918728441,
+            -0.019696446135640144,
+            -0.007566704414784908,
+            0.0013074091402813792,
+            -0.01107825804501772,
+            -0.0019333385862410069,
+            -0.01783948764204979,
+            -0.02312467433512211,
+            -0.016474545001983643,
+            0.027568675577640533,
+            -0.0001265994505956769,
+            0.01592698134481907,
+            0.004400353878736496,
+            0.006217632908374071,
+            -0.019490117207169533,
+            -0.005035211332142353,
+            0.023156417533755302,
+            0.01761728711426258,
+            -0.003640509443357587,
+            -0.003360775299370289,
+            0.021886702626943588,
+            0.0017250260571017861,
+            -0.007011204492300749,
+            -0.0017418895149603486,
+            -0.0010385867208242416,
+            0.012990766204893589,
+            0.01207022275775671,
+            -0.008126172237098217,
+            -0.030790574848651886,
+            0.007836518809199333,
+            0.0007940675131976604,
+            -0.010895737446844578,
+            0.012086094357073307,
+            -0.0037416897248476744,
+            -0.013800208456814289,
+            0.005447868723422289,
+            -0.024013474583625793,
+            0.00014643874601460993,
+            -0.005812911316752434,
+            0.008181722834706306,
+            -0.03834537789225578,
+            0.003967857453972101,
+            0.014157315716147423,
+            -0.004495582543313503,
+            -0.007765097543597221,
+            -0.01128458697348833,
+            -0.021045517176389694,
+            0.002924310974776745,
+            0.009999001398682594,
+            0.009459372609853745,
+            -0.0026941753458231688,
+            0.010070422664284706,
+            -0.008840386755764484,
+            0.0012379715917631984,
+            0.006423961371183395,
+            0.012006737291812897,
+            0.012482879683375359,
+            0.020680474117398262,
+            0.0008382099331356585,
+            -0.00832456536591053,
+            0.005876397248357534,
+            -0.002870745025575161,
+            0.02080744504928589,
+            -0.01822040230035782,
+            0.00022976375475991517,
+            8.462696132482961e-05,
+            -0.003707963041961193,
+            -0.022807246074080467,
+            0.0020355109591037035,
+            5.387606506701559e-05,
+            0.010205329395830631,
+            0.014379516243934631,
+            0.005701811518520117,
+            0.002110900357365608,
+            -0.000986508559435606,
+            0.007769065443426371,
+            0.013673237524926662,
+            -0.009149880148470402,
+            -0.015307994559407234,
+            -0.02085505984723568,
+            -0.014070022851228714,
+            0.010451337322592735,
+            -0.0037000272423028946,
+            -0.0029997003730386496,
+            -0.003422277281060815,
+            0.012720951810479164,
+            0.011030644178390503,
+            -0.009165750816464424,
+            0.02236284501850605,
+            -0.016069823876023293,
+            -0.004396386444568634,
+            -0.01087986584752798,
+            0.01203848049044609,
+            0.0015434966189786792,
+            -0.01301457267254591,
+            0.006558868568390608,
+            0.008364243432879448,
+            -0.0008783844532445073,
+            -0.009983129799365997,
+            -0.0023807145189493895,
+            -0.003507586196064949,
+            0.003283402184024453,
+            0.007451636716723442,
+            -0.0007315737311728299,
+            0.002406505635008216,
+            0.017268115654587746,
+            0.003957937937229872,
+            0.005543096922338009,
+            0.013728787191212177,
+            -0.0017895037308335304,
+            -0.013657365925610065,
+            0.00436464324593544,
+            -0.028108302503824234,
+            -0.0047177826054394245,
+            0.011578208766877651,
+            -0.004630489740520716,
+            -0.000541116576641798,
+            -0.006880265194922686,
+            0.0571054071187973,
+            0.00369804329238832,
+            -0.015720652416348457,
+            0.0011873814510181546,
+            0.008507086895406246,
+            0.023410359397530556,
+            -0.0003945538483094424,
+            0.011967058293521404,
+            0.001203252817504108,
+            0.022870730608701706,
+            0.01977580226957798,
+            -0.02560061775147915,
+            -0.005570872221142054,
+            -0.003928178921341896,
+            0.00583275081589818,
+            0.0301715899258852,
+            0.02866380289196968,
+            0.0062612793408334255,
+            0.0006834634696133435,
+            0.005424061324447393,
+            0.021156616508960724,
+            0.003944050520658493,
+            -0.017760131508111954,
+            0.0026306896470487118,
+            0.0024203930515795946,
+            -0.006582675967365503,
+            0.005888300947844982,
+            -0.016887202858924866,
+            -0.029949389398097992,
+            0.018093431368470192,
+            7.718722918070853e-05,
+            -0.002864793175831437,
+            -0.003094929037615657,
+            -0.018839387223124504,
+            0.01222893688827753,
+            0.013593880459666252,
+            0.0017547850729897618,
+            -0.021410560235381126,
+            -0.002325164619833231,
+            0.006689807865768671,
+            -0.012601915746927261,
+            0.013966858386993408,
+            -0.014022408984601498,
+            -0.031869832426309586,
+            0.0034579879138618708,
+            -0.005535161588340998,
+            0.013236773200333118,
+            -0.0015147296944633126,
+            -0.015379415825009346,
+            -0.009356208145618439,
+            0.024219803512096405,
+            -0.01285585854202509,
+            0.0017041948158293962,
+            -0.005943850614130497,
+            -0.01150678750127554,
+            -0.001055450178682804,
+            0.0001287073828279972,
+            0.008292822167277336,
+            7.19174204277806e-05,
+            -0.01253049448132515,
+            0.01282411627471447,
+            0.024442002177238464,
+            0.0009131032275035977,
+            -0.010784637182950974,
+            -0.01107825804501772,
+            -0.01083225104957819,
+            0.002688223496079445,
+            -0.010491015389561653,
+            0.0017954555805772543,
+            -0.021886702626943588,
+            -0.00956253707408905,
+            0.01236384455114603,
+            -0.002618785947561264,
+            -0.0034500521142035723,
+            0.010657665319740772,
+            -0.0010023800423368812,
+            -0.0006889192736707628,
+            0.016903072595596313,
+            -0.013784337788820267,
+            0.0030413628555834293,
+            0.024267416447401047,
+            -0.0010931447613984346,
+            -0.003392518265172839,
+            -0.006400154437869787,
+            0.0050272755324840546,
+            -0.0002998212294187397,
+            -0.01655390113592148,
+            -0.006293022073805332,
+            -0.017522059381008148,
+            -0.009356208145618439,
+            -0.006848522461950779,
+            -0.018633060157299042,
+            -0.002815194893628359,
+            0.001860925229266286,
+            0.012340037152171135,
+            0.014744559302926064,
+            -0.0019144912948831916,
+            0.006126372143626213,
+            0.02053763158619404,
+            -0.01541115902364254,
+            0.014823916368186474,
+            0.00276956451125443,
+            0.007800808176398277,
+            0.005793072283267975,
+            0.017426829785108566,
+            0.009141944348812103,
+            0.0015028261113911867,
+            0.0036186862271279097,
+            -0.006967558059841394,
+            0.002551332348957658,
+            0.00646760780364275,
+            -0.02247394621372223,
+            0.006352540105581284,
+            0.00021810817997902632,
+            -0.00420196121558547,
+            -0.017744259908795357,
+            -0.004257511347532272,
+            0.002378730569034815,
+            0.007737322244793177,
+            -0.004305125679820776,
+            -0.00040571345016360283,
+            -0.02058524452149868,
+            -0.005320896860212088,
+            -0.0020394788589328527,
+            -0.026330703869462013,
+            -0.005785136483609676,
+            0.0008957438403740525,
+            0.013300258666276932,
+            -0.007661933079361916,
+            -0.005551032721996307,
+            0.022489817813038826,
+            0.011967058293521404,
+            -0.013070123270154,
+            -0.0010693375952541828,
+            -0.0029818450566381216,
+            -0.005098697263747454,
+            0.01644280180335045,
+            0.013220901601016521,
+            -0.00030130919185467064,
+            0.011935316026210785,
+            -0.008792772889137268,
+            0.00692391162738204,
+            -0.007939683273434639,
+            -0.0040611024014651775,
+            0.012586044147610664,
+            -0.011808344163000584,
+            0.00404919870197773,
+            0.0140065373852849,
+            0.012792373076081276,
+            -0.017268115654587746,
+            -0.0070270756259560585,
+            -0.01793471723794937,
+            0.0028409860096871853,
+            0.002198193222284317,
+            -0.008713414892554283,
+            0.008356308564543724,
+            0.0033151451498270035,
+            0.007661933079361916,
+            0.012657465413212776,
+            -0.007106433156877756,
+            0.023949988186359406,
+            0.007292922120541334,
+            -0.0019085395615547895,
+            0.01191150862723589,
+            -0.014673137106001377,
+            0.014728687703609467,
+            0.011490915901958942,
+            -0.0009711331804282963,
+            -0.016903072595596313,
+            -0.0016992350574582815,
+            0.0280289463698864,
+            -0.021696245297789574,
+            0.007392118684947491,
+            -0.003473859280347824,
+            -0.01756967417895794,
+            -0.013887501321732998,
+            -0.009792672470211983,
+            0.007098497357219458,
+            -0.010792572982609272,
+            0.0005416125641204417,
+            -0.019363144412636757,
+            -0.009578408673405647,
+            -0.00011240196909056976,
+            0.01745857298374176,
+            -0.001312368898652494,
+            -0.007808743976056576,
+            0.00809839740395546,
+            0.008721350692212582,
+            -0.010689408518373966,
+            -0.03012397512793541,
+            0.02650528959929943,
+            -0.0025810913648456335,
+            -0.008467407897114754,
+            -0.0013748626224696636,
+            -0.0036365415435284376,
+            0.0048606256023049355,
+            -0.018601316958665848,
+            -0.00811030063778162,
+            -0.007812711410224438,
+            -0.007979361340403557,
+            0.005558968521654606,
+            0.00436464324593544,
+            0.0019591297022998333,
+            -0.008721350692212582,
+            0.006217632908374071,
+            -0.005483579356223345,
+            -0.01622060127556324,
+            -0.01691894419491291,
+            0.010737022385001183,
+            -0.006892168894410133,
+            -0.013538329862058163,
+            -0.008050783537328243,
+            0.0038865164387971163,
+            -0.003188173519447446,
+            -0.006102565210312605,
+            0.01691894419491291,
+            -0.005332800559699535,
+            -0.016569772735238075,
+            0.001228051958605647,
+            0.01004661526530981,
+            0.017522059381008148,
+            0.016569772735238075,
+            0.03367917612195015,
+            0.0015682957600802183,
+            -0.017601415514945984,
+            0.010649729520082474,
+            -8.43944690132048e-06,
+            0.0031961093191057444,
+            0.013831951655447483,
+            -0.01849021576344967,
+            0.010697344318032265,
+            -0.019521860405802727,
+            0.010094230063259602,
+            -0.013736722990870476,
+            -0.005162182729691267,
+            -0.018998103216290474,
+            -0.01772838830947876,
+            -0.0008223385084420443,
+            -0.003146511036902666,
+            0.01355420146137476,
+            0.0005530201597139239,
+            -0.012713016010820866,
+            0.008364243432879448,
+            0.017363345250487328,
+            -0.038726288825273514,
+            -0.011133808642625809,
+            -0.004729686304926872,
+            0.0036484450101852417,
+            -0.017268115654587746,
+            0.02042653039097786,
+            0.023838888853788376,
+            -0.003993648570030928,
+            0.003088977187871933,
+            0.00404919870197773,
+            0.005471675656735897,
+            0.010562436655163765,
+            -0.01810930296778679,
+            0.005118536297231913,
+            -0.036885205656290054,
+            0.0010891768615692854,
+            0.016728486865758896,
+            0.02418806031346321,
+            -0.006281118839979172,
+            -0.01642693020403385,
+            -0.012347972951829433,
+            -0.009602215141057968,
+            -0.031012775376439095,
+            -0.006550932768732309,
+            -0.01368910912424326,
+            -0.0026326735969632864,
+            -0.007844454608857632,
+            -0.01236384455114603,
+            0.024600718170404434,
+            0.008427729830145836,
+            -0.0019283788278698921,
+            -0.004686039872467518,
+            0.003116752253845334,
+            -0.024330902844667435,
+            -0.01622060127556324,
+            -0.014204930514097214,
+            0.005801007617264986,
+            0.021426431834697723,
+            -0.0033786308486014605,
+            0.012593979947268963,
+            -0.011395687237381935,
+            0.001903579686768353,
+            0.0024719752836972475,
+            0.004182121716439724,
+            -0.013649430125951767,
+            0.032044418156147,
+            -0.0012171403504908085,
+            -0.0002213320549344644,
+            0.0037238344084471464,
+            -0.004602714907377958,
+            0.014974694699048996,
+            0.006717583164572716,
+            -0.013324066065251827,
+            -0.010657665319740772,
+            0.01741096004843712,
+            0.01549051608890295,
+            0.0019174672197550535,
+            -0.01174485869705677,
+            -0.0005376447224989533,
+            0.008126172237098217,
+            -0.006023207679390907,
+            -0.003297289600595832,
+            -0.00013267523900140077,
+            0.0005698835593648255,
+            0.02769564650952816,
+            0.005487547256052494,
+            0.005090761464089155,
+            -0.011340136639773846,
+            0.015101666562259197,
+            0.017014173790812492,
+            0.005535161588340998,
+            -0.0013084009988233447,
+            -0.020664602518081665,
+            0.004186089616268873,
+            -0.021220102906227112,
+            -0.003416325431317091,
+            -0.008951487019658089,
+            -0.0034778271801769733,
+            0.004559068474918604,
+            -0.0025394288823008537,
+            -0.016791973263025284,
+            0.00607478991150856,
+            -0.009848223067820072,
+            -4.6684326662216336e-05,
+            0.009181622415781021,
+            -0.013347873464226723,
+            0.013022508472204208,
+            0.029552603140473366,
+            0.018823517486453056,
+            0.015514323487877846,
+            -0.0025275254156440496,
+            -0.02241045981645584,
+            0.02321990206837654,
+            0.008761029690504074,
+            0.006114468444138765,
+            0.0010078358463943005,
+            -0.007392118684947491,
+            0.025505388155579567,
+            0.029822418466210365,
+            0.002785436110571027,
+            0.006741390097886324,
+            -0.006332700606435537,
+            0.013514523394405842,
+            0.00864992942661047,
+            -0.006717583164572716,
+            0.0006824714946560562,
+            0.018521958962082863,
+            -0.0066223544999957085,
+            -0.003807159373536706,
+            0.031076261773705482,
+            -0.023156417533755302,
+            -0.012419394217431545,
+            -0.013458972796797752,
+            -0.0021168519742786884,
+            0.00428528618067503,
+            0.017522059381008148,
+            -0.021839087828993797,
+            -0.008459472097456455,
+            0.013141544535756111,
+            0.008792772889137268,
+            0.00013689108891412616,
+            -0.008713414892554283,
+            0.0006621362408623099,
+            0.010713215917348862,
+            -0.006991364993155003,
+            0.0005802991800010204,
+            -0.029917646199464798,
+            -0.0027576610445976257,
+            -0.0023410359863191843,
+            0.0027775003109127283,
+            0.04339249059557915,
+            -0.002136691240593791,
+            0.00878483708947897,
+            0.003140559419989586,
+            0.008078558370471,
+            -0.007185790222138166,
+            -0.014435065910220146,
+            0.0011625823099166155,
+            0.01642693020403385,
+            -0.00584862194955349,
+            0.010435465723276138,
+            -0.0008883041446097195,
+            -0.027346475049853325,
+            0.017331602051854134,
+            0.03659952059388161,
+            0.006082725711166859,
+            -0.005428029224276543,
+            -0.005900204181671143,
+            -0.005150279030203819,
+            -0.012371780350804329,
+            -0.007328633219003677,
+            0.013506587594747543,
+            0.0012657465413212776,
+            0.027235373854637146,
+            -0.015061987563967705,
+            0.005150279030203819,
+            -0.005816879216581583,
+            -0.025632360950112343,
+            0.010824315249919891,
+            0.004309093579649925,
+            0.00013974298781249672,
+            0.01563335955142975,
+            0.020616987720131874,
+            -0.03082231804728508,
+            -0.011030644178390503,
+            -0.005364543292671442,
+            -0.021696245297789574,
+            0.016188859939575195,
+            -0.006864393595606089,
+            0.01503024436533451,
+            -0.002779484260827303,
+            0.007907940074801445,
+            -0.011236973106861115,
+            0.004293221980333328,
+            0.019712315872311592,
+            -0.004086893517524004,
+            0.018982231616973877,
+            -0.009983129799365997,
+            -0.007618286646902561,
+            0.013355808332562447,
+            -0.016633259132504463,
+            0.020775701850652695,
+            -0.005678004119545221,
+            -0.002283502137288451,
+            0.021918445825576782,
+            -0.01347484439611435,
+            0.012355908751487732,
+            0.0030314433388412,
+            -0.01309392973780632,
+            -0.00044415207230485976,
+            -0.0033389520831406116,
+            0.0013252644566819072,
+            -0.009610150940716267,
+            -0.019474245607852936,
+            0.021029645577073097,
+            -0.0180458165705204,
+            0.023807145655155182,
+            0.0015028261113911867,
+            -0.011348072439432144,
+            0.001473067095503211,
+            -0.007971425540745258,
+            -0.023521460592746735,
+            0.0015752394683659077,
+            -0.00280130747705698,
+            0.011816279962658882,
+            -0.009475244209170341,
+            0.020601116120815277,
+            0.0057335542514920235,
+            -0.016165051609277725,
+            0.00047986279241740704,
+            0.011570272967219353,
+            -0.0006963590276427567,
+            0.005158214829862118,
+            0.027711518108844757,
+            -0.001079257228411734,
+            0.0029957324732095003,
+            0.006054950878024101,
+            0.008745158091187477,
+            -0.03352046012878418,
+            -0.009673637337982655,
+            0.004047214984893799,
+            -0.0004498558701016009,
+            0.010149779729545116,
+            0.005221700761467218,
+            -0.010221200995147228,
+            0.022712016478180885,
+            -0.00013354320253711194,
+            -0.017744259908795357,
+            0.008475343696773052,
+            -0.009205429814755917,
+            -0.011602015234529972,
+            -0.009284786880016327,
+            -0.002348971785977483,
+            0.021029645577073097,
+            -0.005820847116410732,
+            0.010030743665993214,
+            0.0017418895149603486,
+            0.003398470114916563,
+            -0.002497766399756074,
+            -0.017966458573937416,
+            0.002624737797304988,
+            -0.004618586506694555,
+            0.020188460126519203,
+            0.015919044613838196,
+            0.015823816880583763,
+            -0.00911020115017891,
+            -0.015593680553138256,
+            0.004725718405097723,
+            0.02796545997262001,
+            -0.009895836934447289,
+            0.0071699186228215694,
+            -0.004011503886431456,
+            0.0013937100302428007,
+            -0.006539029534906149,
+            -0.013133608736097813,
+            -0.015228637494146824,
+            -0.03482192009687424,
+            -0.0019690494518727064,
+            0.002146610990166664,
+            -0.00044415207230485976,
+            -0.012919344007968903,
+            -0.03685346245765686,
+            -0.007911908440291882,
+            -0.02052175998687744,
+            -0.019585344940423965,
+            -0.02807656116783619,
+            -0.02445787377655506,
+            -0.004880465101450682,
+            -0.014784237369894981,
+            0.011244908906519413,
+            -0.0034460844472050667,
+            -0.009792672470211983,
+            0.0037754166405647993,
+            0.023045316338539124,
+            0.00047267103218473494,
+            0.02042653039097786,
+            -0.013760530389845371,
+            0.008880065754055977,
+            -0.007530993781983852,
+            0.007308793719857931,
+            -0.006360475905239582,
+            -0.01649041660130024,
+            0.00654696486890316,
+            0.008070622570812702,
+            0.021140744909644127,
+            -0.01129252277314663,
+            0.0018837404204532504,
+            -0.006285086274147034,
+            0.0013431197730824351,
+            -0.00023447559215128422,
+            0.000928974652197212,
+            0.006709647364914417,
+            0.020188460126519203,
+            -0.02625134587287903,
+            -0.018775902688503265,
+            0.008126172237098217,
+            0.009760930202901363,
+            0.021807344630360603,
+            -0.021632758900523186,
+            0.013284387066960335,
+            -2.7294519895804115e-05,
+            -0.00575339375063777,
+            0.013220901601016521,
+            0.012078158557415009,
+            -0.005277250427752733,
+            0.0024997503496706486,
+            0.02301357500255108,
+            0.012467009015381336,
+            -0.009673637337982655,
+            -0.020601116120815277,
+            0.0088880006223917,
+            0.0006874313112348318,
+            -0.007733354344964027,
+            0.004868561401963234,
+            0.03361568972468376,
+            0.023156417533755302,
+            0.01982341706752777,
+            -0.00039802573155611753,
+            0.031552404165267944,
+            0.0011506787268444896,
+            -0.016744358465075493,
+            0.010705280117690563,
+            -0.0036087664775550365,
+            0.004928078968077898,
+            -0.016315830871462822,
+            -0.04532880708575249,
+            0.004880465101450682,
+            0.002618785947561264,
+            0.009999001398682594,
+            -0.027457574382424355,
+            0.01891874521970749,
+            0.0006244415999390185,
+            -0.0024660234339535236,
+            -0.014958823099732399,
+            0.008126172237098217,
+            -0.010387850925326347,
+            0.00818965770304203,
+            -0.004559068474918604,
+            -0.01463345903903246,
+            0.021474044770002365,
+            0.00630492577329278,
+            -0.004531293176114559,
+            0.011340136639773846,
+            0.022537430748343468,
+            0.01847434602677822,
+            0.012078158557415009,
+            0.010467207990586758,
+            0.04247194901108742,
+            0.011784536764025688,
+            0.0025632360484451056,
+            0.006626322399824858,
+            -0.00334688788279891,
+            -0.0032774503342807293,
+            -0.004908239934593439,
+            0.000227779833949171,
+            -0.004071021918207407,
+            -0.018299760296940804,
+            0.016855459660291672,
+            0.010118036530911922,
+            -0.010729086585342884,
+            0.01714114472270012,
+            -0.01619679480791092,
+            0.012030544690787792,
+            0.03485365957021713,
+            0.00010012640996137634,
+            0.017696645110845566,
+            -0.01901397295296192,
+            -0.009760930202901363,
+            -0.0020573341753333807,
+            -0.011546465568244457,
+            0.0036206701770424843,
+            -0.015823816880583763,
+            -0.007455604616552591,
+            -0.008213465102016926,
+            0.00770557951182127,
+            0.01595078781247139,
+            0.015347673557698727,
+            0.013530394062399864,
+            0.01338755153119564,
+            0.0056065828539431095,
+            -0.009681572206318378,
+            -0.001512745744548738,
+            -0.015379415825009346,
+            -0.012522558681666851,
+            -0.01606188714504242,
+            0.004943950567394495,
+            -0.0025394288823008537,
+            -0.017966458573937416,
+            0.0269496887922287,
+            -0.013054251670837402,
+            0.005487547256052494,
+            0.009959322400391102,
+            0.01876003108918667,
+            -0.0005031739710830152,
+            0.00045853553456254303,
+            -0.005035211332142353,
+            0.0008818563655950129,
+            0.005150279030203819,
+            -0.001076281419955194,
+            0.027394089847803116,
+            -0.004225768148899078,
+            0.004277350381016731,
+            -0.0017498251982033253,
+            -0.014974694699048996,
+            -0.0007028068066574633,
+            0.00840392243117094,
+            -0.020870931446552277,
+            -0.02263266034424305,
+            -0.018283888697624207,
+            -0.005253443494439125,
+            -0.011157616041600704,
+            0.0067334542982280254,
+            -0.011300458572804928,
+            -0.008880065754055977,
+            0.023838888853788376,
+            0.00661441870033741,
+            0.014744559302926064,
+            0.0058446540497243404,
+            -0.0009671652805991471,
+            0.017712516710162163,
+            -0.02306118793785572,
+            -0.014379516243934631,
+            0.0024263449013233185,
+            -0.010744958184659481,
+            0.032568175345659256,
+            0.00097311707213521,
+            0.01301457267254591,
+            -0.010983030311763287,
+            0.010435465723276138,
+            -0.014117637649178505,
+            0.02418806031346321,
+            0.012705080211162567,
+            0.00584862194955349,
+            0.004186089616268873,
+            -0.003910323604941368,
+            -0.011308394372463226,
+            0.03821840509772301,
+            0.0002916375233326107,
+            -0.007225468754768372,
+            0.02004561759531498,
+            0.00832456536591053,
+            0.008721350692212582,
+            -0.008657865226268768,
+            0.012022608891129494,
+            0.016665002331137657,
+            0.0002353435556869954,
+            0.006741390097886324,
+            -0.01826801709830761,
+            -0.0040452308021485806,
+            0.023204030469059944,
+            0.025092732161283493,
+            0.011467108502984047,
+            -0.003124687820672989,
+            -0.002600930631160736,
+            0.012720951810479164,
+            -0.0017408975400030613,
+            0.019410759210586548,
+            0.009300658479332924,
+            0.03863106295466423,
+            0.019601216539740562,
+            -0.014046216383576393,
+            -0.0013530394062399864,
+            0.005110600497573614,
+            0.05183609202504158,
+            -0.010657665319740772,
+            -0.007161982823163271,
+            -0.008269015699625015,
+            -0.014752495102584362,
+            0.015157216228544712,
+            -0.010173587128520012,
+            0.008082525804638863,
+            -0.008657865226268768,
+            0.010863994248211384,
+            0.017823616042733192,
+            0.007892068475484848,
+            -0.009792672470211983,
+            -0.007376247551292181,
+            -0.011117937043309212,
+            0.012562237679958344,
+            0.001430412638001144,
+            -0.00338855036534369,
+            0.011855958960950375,
+            0.01026087999343872,
+            -0.016014274209737778,
+            -0.007677804678678513,
+            0.017855359241366386,
+            -0.022346973419189453,
+            0.0013738706475123763,
+            0.007284986786544323,
+            0.0036008309107273817,
+            -0.00108818500302732,
+            -6.959250185900601e-06,
+            0.00910226535052061,
+            0.028600318357348442,
+            0.0016545966500416398,
+            0.02942563220858574,
+            0.01799820177257061,
+            0.00014445481065195054,
+            -0.024965759366750717,
+            0.005789104383438826,
+            0.01739508844912052,
+            0.004412257578223944,
+            0.024695945903658867,
+            -0.005420093424618244,
+            -0.012459073215723038,
+            -0.0191250741481781,
+            -0.0088880006223917,
+            0.012094030156731606,
+            -0.018394988030195236,
+            0.013197094202041626,
+            -0.005189957562834024,
+            0.007915875874459743,
+            0.005777200683951378,
+            0.015093730762600899,
+            -0.010530694387853146,
+            0.007685740012675524,
+            0.0021902574226260185,
+            -0.011086193844676018,
+            -0.011784536764025688,
+            -0.00032883620588108897,
+            -0.007384182885289192,
+            0.003209996735677123,
+            -0.0035849593114107847,
+            0.011649630032479763,
+            0.023727789521217346,
+            -0.004773332737386227,
+            -0.006947718560695648,
+            0.009991065599024296,
+            0.004654297139495611,
+            -0.011562337167561054,
+            0.02193431742489338,
+            0.009705379605293274,
+            0.004110700450837612,
+            0.007856357842683792,
+            -0.01042752992361784,
+            -0.016069823876023293,
+            -0.023045316338539124,
+            -0.007181822322309017,
+            -0.0231722891330719,
+            0.014974694699048996,
+            -0.006658065132796764,
+            -0.015800008550286293,
+            0.01858544535934925,
+            0.002934230724349618,
+            0.0018480296712368727,
+            -0.00848327949643135,
+            -0.026330703869462013,
+            -0.022331101819872856,
+            0.01026087999343872,
+            -0.00654696486890316,
+            0.005201861262321472,
+            0.02706078812479973,
+            0.030155718326568604,
+            -0.007019140291959047,
+            0.004261479247361422,
+            0.0034500521142035723,
+            0.003930163104087114,
+            -0.0002421633107587695,
+            -0.024981630966067314,
+            -0.0064041223376989365,
+            0.007947619073092937,
+            0.006789004430174828,
+            -0.00544390082359314,
+            -0.00926891528069973,
+            0.011943251825869083,
+            -0.019299659878015518,
+            -0.005920043680816889,
+            -0.009173686616122723,
+            -0.002819162793457508,
+            -0.00032015651231631637,
+            0.017982330173254013,
+            0.015268316492438316,
+            -0.005551032721996307,
+            -0.0025870432145893574,
+            -0.023775402456521988,
+            -0.012181323021650314,
+            0.01939488761126995,
+            0.004884432535618544,
+            0.006574740167707205,
+            -0.01471281610429287,
+            -0.0012141644256189466,
+            -0.0033389520831406116,
+            -0.012498751282691956,
+            0.015173087827861309,
+            0.06513635069131851,
+            -0.016006337478756905,
+            0.02483878843486309,
+            -0.005007436498999596,
+            0.004701911471784115,
+            -0.003416325431317091,
+            -0.021426431834697723,
+            -0.011951187625527382,
+            -0.0060271755792200565,
+            0.0017756163142621517,
+            -0.0188552588224411,
+            -0.0023965861182659864,
+            -0.0006715598865412176,
+            0.002325164619833231,
+            -0.007019140291959047,
+            0.0011953171342611313,
+            -0.02085505984723568,
+            0.016887202858924866,
+            0.006431897170841694,
+            -0.003862709505483508,
+            -0.006447768770158291,
+            -0.018823517486453056,
+            0.00022120805806480348,
+            -0.0016357493586838245,
+            -2.8115991881350055e-05,
+            0.0016784038161858916,
+            -0.025902174413204193,
+            -0.010657665319740772,
+            -0.00964189413934946,
+            -0.015069923363626003,
+            0.0002531989011913538,
+            -0.006380314938724041,
+            0.01463345903903246,
+            -0.0220930315554142,
+            -0.004872529301792383,
+            0.01822040230035782,
+            0.028362246230244637,
+            0.020728088915348053,
+            -0.011213165707886219,
+            0.001291537657380104,
+            -0.02160101756453514,
+            0.02533080242574215,
+            0.01880764588713646,
+            -0.017442701384425163,
+            0.005102664697915316,
+            0.008768965490162373,
+            0.009951386600732803,
+            -0.0019998003263026476,
+            0.030139846727252007,
+            -0.00038463418604806066,
+            -0.015427030622959137,
+            0.005090761464089155,
+            0.018934616819024086,
+            0.002618785947561264,
+            0.002612834330648184,
+            -0.006209697108715773,
+            0.021585145965218544,
+            -0.005582775454968214,
+            0.009633958339691162,
+            -0.044852662831544876,
+            -0.01815691590309143,
+            -0.005519289989024401,
+            -0.004213864915072918,
+            0.007812711410224438,
+            0.004011503886431456,
+            0.008673736825585365,
+            -0.018093431368470192,
+            -0.0021208198741078377,
+            0.025283189490437508,
+            -0.026965560391545296,
+            -0.02823527529835701,
+            -0.005225668661296368,
+            -0.02441026084125042,
+            -0.0007444692892022431,
+            0.0011546466266736388,
+            -0.007959522306919098,
+            -0.026473546400666237,
+            0.008284886367619038,
+            -0.006122404243797064,
+            -0.027679774910211563,
+            0.01768077351152897,
+            -0.005681972019374371,
+            0.000294117460725829,
+            -0.010300558060407639,
+            -0.003963889554142952,
+            -0.005356607958674431,
+            0.016569772735238075,
+            -0.01330819446593523,
+            0.008062686771154404,
+            0.0010266831377521157,
+            -0.0032357878517359495,
+            0.0016873314743861556,
+            0.016093630343675613,
+            0.002900503808632493,
+            0.000488046498503536,
+            -0.006039079278707504,
+            -0.003088977187871933,
+            -0.017871230840682983,
+            0.000921534956432879,
+            0.01891874521970749,
+            0.0036325736436992884,
+            0.02672748826444149,
+            -0.03469494730234146,
+            -0.01642693020403385,
+            -0.005725618451833725,
+            -0.01725224405527115,
+            0.011403623037040234,
+            -0.0036821719259023666,
+            -0.008499151095747948,
+            -0.0019075475865975022,
+            -0.02548951655626297,
+            0.03304431959986687,
+            -0.01939488761126995,
+            -0.00506695406511426,
+            0.004428129177540541,
+            -0.019839288666844368,
+            -0.00044935988262295723,
+            0.008157915435731411,
+            0.01998213119804859,
+            0.010308493860065937,
+            -0.015617487952113152,
+            0.009070522151887417,
+            -0.008538829162716866,
+            -0.008054750971496105,
+            0.0041543468832969666,
+            0.008126172237098217,
+            -0.009681572206318378,
+            -0.0029362146742641926,
+            0.00816585123538971,
+            -0.0030731058213859797,
+            -0.014284287579357624,
+            -0.00878483708947897,
+            0.027219504117965698,
+            -0.008515022695064545,
+            0.00575339375063777,
+            -0.0026386252138763666,
+            -0.001564327860251069,
+            -0.0020394788589328527,
+            0.00064130499958992,
+            0.0062612793408334255,
+            -0.006253343541175127,
+            0.05850209295749664,
+            -0.0003680188092403114,
+            -0.00022405995696317405,
+            -0.01869654469192028,
+            -0.011855958960950375,
+            0.005035211332142353,
+            0.006880265194922686,
+            -0.02220413088798523,
+            -0.006959622260183096,
+            0.025092732161283493,
+            -0.0020533662755042315,
+            0.010808444581925869,
+            0.005408189725130796,
+            -0.001736929640173912,
+            0.0328538604080677,
+            0.006852489896118641,
+            0.0280130747705698,
+            -0.010760829783976078,
+            0.004015471786260605,
+            0.0001645420998102054,
+            0.00755876861512661,
+            0.0069516864605247974,
+            -0.009118136949837208,
+            -0.005003468599170446,
+            0.0040611024014651775,
+            0.014300159178674221,
+            -0.0004451440181583166,
+            -0.008292822167277336,
+            0.0030135877896100283,
+            -0.012109901756048203,
+            -0.0017528011230751872,
+            -0.0008218425209634006,
+            -0.0140065373852849,
+            -0.00576529698446393,
+            0.009173686616122723,
+            0.015458772890269756,
+            0.007324665319174528,
+            0.0010683456202968955,
+            0.004400353878736496,
+            0.004999500699341297,
+            0.005269315093755722,
+            -0.012094030156731606,
+            0.009570472873747349,
+            -0.02667987532913685,
+            -0.009062586352229118,
+            0.01214957982301712,
+            -0.0011348072439432144,
+            -0.00886419415473938,
+            0.023505588993430138,
+            0.0046265218406915665,
+            -0.012332101352512836,
+            -0.011022708378732204,
+            0.006796940229833126,
+            -0.0023172288201749325,
+            0.008991165086627007,
+            -0.007860326208174229,
+            0.0027636128943413496,
+            -0.0007955554756335914,
+            0.01622060127556324,
+            0.01653803139925003,
+            -0.004253543447703123,
+            0.012117837555706501,
+            -0.0004079453647136688
+          ],
+          "title": "Balancing Expert Utilization",
+          "keyphrases": [
+            "gating network",
+            "expert utilization",
+            "soft constraint",
+            "importance of an expert",
+            "balanced loads"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "87b82083-05f3-4d69-9ea0-97d4f551aa37",
+        "properties": {
+          "page_content": "5 Experiments 5.1 1 Billion Word Language Modeling Benchmark\n\nDataset: This dataset, introduced by (Chelba et al., 2013) consists of shuffled unique sentences from news articles, totaling approximately 829 million words, with a vocabulary of 793,471 words.\n\nPrevious State-of-the-Art: The best previously published results (Jozefowicz et al., 2016) use models consisting of one or more stacked Long Short-Term Memory (LSTM) layers (Hochreiter & Schmidhuber, 1997; Gers et al., 2000). The number of parameters in the LSTM layers of these models vary from 2 million to 151 million. Quality increases greatly with parameter count, as do computational costs. Results for these models form the top line of Figure 2-right. MoE Models: Our models consist of two stacked LSTM layers with a MoE layer between them (see Figure 1). We vary the sizes of the layers and the number of experts. For full details on model architecture, training regimen, additional baselines and results, see Appendix C.\n\nLow Computation, Varied Capacity: To investigate the effects of adding capacity, we trained\n\na series of MoE models all with roughly equal computational costs: about 8 million multiply-andadds per training example per timestep in the forwards pass, excluding the softmax layer. We call this metric (ops/timestep). We trained models with flat MoEs containing 4, 32, and 256 experts, and models with hierarchical MoEs containing 256, 1024, and 4096 experts. Each expert had about 1 million parameters. For all the MoE layers, 4 experts were active per input.\n\nThe results of these models are shown in Figure 2-left. The model with 4 always-active experts performed (unsurprisingly) similarly to the computationally-matched baseline models, while the largest of the models (4096 experts) achieved an impressive 24% lower perplexity on the test set. Figure 2: Model comparison on 1-Billion-Word Language-Modeling Benchmark. On the left, we plot test perplexity as a function of model capacity for models with similar computational budgets of approximately 8-million-ops-per-timestep. On the right, we plot test perplexity as a function of computational budget. The top line represents the LSTM models from (Jozefowicz et al., 2016).\n\nThe bottom line represents 4-billion parameter MoE models with different computational budgets.\n\nVaried Computation, High Capacity: In addition to the largest model from the previous section, we trained two more MoE models with similarly high capacity (4 billion parameters), but higher computation budgets. These models had larger LSTMs, and fewer but larger and experts. Details\n\nTest Test #Parameters ops/timestep Training TFLOPS Perplexity Perplexity excluding embedding Time /GPU 10 epochs 100 epochs and softmax layers 10 epochs Best Published Results 34.7 30.6 151 million 151 million 59 hours, 32 k40s 1.09 Low-Budget MoE Model 34.1 4303 million 8.9 million 15 hours, 16 k40s 0.74 Medium-Budget MoE Model 31.3 4313 million 33.8 million 17 hours, 32 k40s 1.22 High-Budget MoE Model 28.0 4371 million 142.7 million 47 hours, 32 k40s 1.56\n\nTable 1: Summary of high-capacity MoE-augmented models with varying computational budgets, vs. best previously published results (Jozefowicz et al., 2016). Details in Appendix C. can be found in Appendix C.2. Results of these three models form the bottom line of Figure 2-right.\n\nTable 1 compares the results of these models to the best previously-published result on this dataset .\n\nEven the fastest of these models beats the best published result (when controlling for the number of training epochs), despite requiring only 6% of the computation.\n\nComputational Efficiency: We trained our models using TensorFlow (Abadi et al., 2016) on clusters containing 16-32 Tesla K40 GPUs. For each of our models, we determine computational efficiency in TFLOPS/GPU by dividing the number of floating point operations required to process one training batch by the observed step time and the number of GPUs in the cluster. The operation counts used here are higher than the ones we report in our ops/timestep numbers in that we include the backwards pass, we include the importance-sampling-based training of the softmax layer, and we count a multiply-and-add as two separate operations. For all of our MoE models, the floating point operations involved in the experts represent between 37% and 46% of the total.\n\nFor our baseline models wtih no MoE, observed computational efficiency ranged from 1.07-1.29 TFLOPS/GPU. For our low-computation MoE models, computation efficiency ranged from 0.740.90 TFLOPS/GPU, except for the 4-expert model which did not make full use of the available parallelism. Our highest-computation MoE model was more efficient at 1.56 TFLOPS/GPU, likely due to the larger matrices. These numbers represent a significant fraction of the theoretical maximum of 4.29 TFLOPS/GPU claimed by NVIDIA. Detailed results are in Appendix C, Table 7.\n\n5.2 100 BILLION WORD GOOGLE NEWS CORPUS\n\nOn the 1-billion-word corpus, adding additional capacity seems to produce diminishing returns as the number of parameters in the MoE layer exceeds 1 billion, as can be seen in Figure 2-left. We hypothesized that for a larger training set, even higher capacities would produce significant quality improvements. We constructed a similar training set consisting of shuffled unique sentences from Google's internal news corpus, totalling roughly 100 billion words. Similarly to the previous section, we tested a series of models with similar computational costs of about 8 million ops/timestep. In addition to a baseline LSTM model, we trained models augmented with MoE layers containing 32, 256, 1024, 4096, 16384, 65536, and 131072 experts. This corresponds to up to 137 billion parameters in the MoE layer. Details on architecture, training, and results are given in Appendix D.\n\nResults: Figure 3 shows test perplexity as a function of capacity after training on 10 billion words (top line) and 100 billion words (bottom line). When training over the full 100 billion words, test perplexity improves significantly up to 65536 experts (68 billion parameters), dropping 39% lower than the computationally matched baseline, but degrades at 131072 experts, possibly a result of too much sparsity. The widening gap between the two lines demonstrates (unsurprisingly) that increased model capacity helps more on larger training sets. Even at 65536 experts (99.994% layer sparsity), computational efficiency for the model stays at a respectable 0.72 TFLOPS/GPU.\n\n5.3 Machine Translation (Single Language Pair)\n\nModel Architecture: Our model was a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decreased the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We inserted MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). Each MoE layer contained up to 2048 experts each with about two million parameters, adding a total of about 8 billion parameters to the models. Further details on model architecture, testing procedure and results can be found in Appendix E.\n\nDatasets: We benchmarked our method on the WMT'14 En→Fr and En→De corpora, whose training sets have 36M sentence pairs and 5M sentence pairs, respectively. The experimental protocols were also similar to those in (Wu et al., 2016): newstest2014 was used as the test set to compare against previous work (Luong et al., 2015a; Zhou et al., 2016; Wu et al., 2016), while the combination of newstest2012 and newstest2013 was used as the development set. We also tested the same model on a Google's Production English to French data.\n\nTable 2: Results on WMT'14 En→ Fr newstest2014 (bold values represent best results). Model Test Test ops/timenstep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.69 40.35 85M 8.7B 3 days/64 k40s MoE with 2048 Experts (longer training) 2.63 40.56 85M 8.7B 6 days/64 k40s GNMT (Wu et al., 2016) 2.79 39.22 214M 278M 6 days/96 k80s GNMT+RL (Wu et al., 2016) 2.96 39.92 214M 278M 6 days/96 k80s PBMT (Durrani et al., 2014) 37.0 LSTM (6-layer) (Luong et al., 2015b) 31.5 LSTM (6-layer+PosUnk) (Luong et al., 2015b) 33.1 DeepAtt (Zhou et al., 2016) 37.7 DeepAtt+PosUnk (Zhou et al., 2016) 39.2\n\nTable 3: Results on WMT'14 En → De newstest2014 (bold values represent best results). Model Test Test ops/timestep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 4.64 26.03 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 5.25 24.91 214M 278M 1 day/96 k80s GNMT +RL (Wu et al., 2016) 8.08 24.66 214M 278M 1 day/96 k80s PBMT (Durrani et al., 2014) 20.7 DeepAtt (Zhou et al., 2016) 20.6\n\nTable 4: Results on the Google Production En→ Fr dataset (bold values represent best results). Model Eval Eval Test Test ops/timestep Total Training Perplexity BLEU Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.60 37.27 2.69 36.57 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 2.78 35.80 2.87 35.56 214M 278M 6 days/96 k80s\n\nResults: Tables 2, 3, and 4 show the results of our largest models, compared with published results. Our approach achieved BLEU scores of 40.56 and 26.03 on the WMT'14 En→Fr and En→De benchmarks. As our models did not use RL refinement, these results constitute significant gains of 1.34 and 1.12 BLEU score on top of the strong baselines in (Wu et al., 2016). The perplexity scores are also better.2 On the Google Production dataset, our model achieved 1.01 higher test BLEU score even after training for only one sixth of the time.\n\n5.4 Multilingual Machine Translation\n\nDataset: (Johnson et al., 2016) train a single GNMT (Wu et al., 2016) model on a very large combined dataset of twelve language pairs. Results are somewhat worse than those for 12 separately trained single-pair GNMT models. This is not surprising, given that the twelve models have 12 times the capacity and twelve times the aggregate training of the one model. We repeat this experiment with a single MoE-augmented model. See Appendix E for details on model architecture.\n\nWe train our model on the same dataset as (Johnson et al., 2016) and process the same number of training examples (about 3 billion sentence pairs). Our training time was shorter due to the lower computational budget of our model.\n\nResults: Results for the single-pair GNMT models, the multilingual GNMT model and the multilingual MoE model are given in Table 5. The MoE model achieves 19% lower perplexity on the dev set than the multilingual GNMT model. On BLEU score, the MoE model significantly beats the multilingual GNMT model on 11 of the 12 language pairs (by as much as 5.84 points), and even beats the monolingual GNMT models on 8 of 12 language pairs. The poor performance on English → Korean seems to be a result of severe overtraining, as for the rarer language pairs a small number of real examples were highly oversampled in the training corpus.\n\nTable 5: Multilingual Machine Translation (bold values represent best results). GNMT-Mono GNMT-Multi MoE-Multi MoE-Multi vs. GNMT-Multi Parameters 278M / model 278M 8.7B ops/timestep 212M 212M 102M training time, hardware various 21 days, 96 k20s 12 days, 64 k40s Perplexity (dev) 4.14 3.35 -19% French → English Test BLEU 36.47 34.40 37.46 +3.06 German → English Test BLEU 31.77 31.17 34.80 +3.63 Japanese → English Test BLEU 23.41 21.62 25.91 +4.29 Korean → English Test BLEU 25.42 22.87 28.71 +5.84 Portuguese → English Test BLEU 44.40 42.53 46.13 +3.60 Spanish → English Test BLEU 38.00 36.04 39.39 +3.35 English → French Test BLEU 35.37 34.00 36.59 +2.59 English → German Test BLEU 26.43 23.15 24.53 +1.38 English → Japanese Test BLEU 23.66 21.10 22.78 +1.68 English → Korean Test BLEU 19.75 18.41 16.62 -1.79 English → Portuguese Test BLEU 38.40 37.35 37.90 +0.55 English → Spanish Test BLEU 34.50 34.25 36.21 +1.96\n\n",
+          "embedding": [
+            -0.009427943266928196,
+            0.0009052514214999974,
+            -0.031882621347904205,
+            0.02155720256268978,
+            -0.0010102161904796958,
+            0.017834007740020752,
+            -0.019673390313982964,
+            0.04869475215673447,
+            -0.022961175069212914,
+            0.03497491776943207,
+            0.036041226238012314,
+            -0.002160385251045227,
+            0.013355513103306293,
+            -0.01921132393181324,
+            -0.013746492564678192,
+            0.007433058228343725,
+            -0.00785069540143013,
+            0.004056415054947138,
+            0.003800945123657584,
+            -0.0150793781504035,
+            -0.03134946897625923,
+            -0.014439593069255352,
+            -0.04698865860700607,
+            -0.015088263899087906,
+            -0.016625525429844856,
+            0.005407072138041258,
+            0.02432960458099842,
+            -0.0009768940508365631,
+            -0.040199827402830124,
+            0.01593242585659027,
+            0.004467388149350882,
+            0.0005736961611546576,
+            0.009525688365101814,
+            -0.008481594733893871,
+            -0.0005428731674328446,
+            -0.017078706994652748,
+            0.07435724139213562,
+            -0.004718414973467588,
+            -0.03138501197099686,
+            0.03760514408349991,
+            0.028381574898958206,
+            -0.006029085721820593,
+            -0.0007786272908560932,
+            0.0035210391506552696,
+            0.0162967462092638,
+            0.003805388230830431,
+            -0.01993996649980545,
+            0.007339756470173597,
+            -0.01571027748286724,
+            -0.037356339395046234,
+            -0.006606669165194035,
+            -0.03152718394994736,
+            0.0029923280235379934,
+            -0.02411634288728237,
+            -0.0463133305311203,
+            0.014768371358513832,
+            0.028292717412114143,
+            0.02548477053642273,
+            0.06902569532394409,
+            0.034797199070453644,
+            0.01878480054438114,
+            0.0006881021545268595,
+            -0.024222973734140396,
+            -0.02600015327334404,
+            0.03351762890815735,
+            -0.012244774959981441,
+            -0.02527150884270668,
+            0.005011649336665869,
+            0.016243431717157364,
+            -0.049334537237882614,
+            -0.03643220290541649,
+            0.037498511373996735,
+            0.009730064310133457,
+            -0.005127166397869587,
+            0.015026062726974487,
+            -0.022961175069212914,
+            0.04986768960952759,
+            0.045104846358299255,
+            -0.012777929194271564,
+            -0.0024103012401610613,
+            0.013133365660905838,
+            0.017505230382084846,
+            -0.008437165059149265,
+            -0.004611784126609564,
+            0.0228723157197237,
+            -0.022374704480171204,
+            -0.014164130203425884,
+            -0.07983095943927765,
+            0.0133999427780509,
+            -0.028097227215766907,
+            -0.03525926545262337,
+            0.015585875138640404,
+            0.015976853668689728,
+            -0.0007325316546484828,
+            -0.002772401785477996,
+            0.031171750277280807,
+            -0.024080798029899597,
+            -0.003567690262570977,
+            0.001345103606581688,
+            0.04762844368815422,
+            -0.01612791419029236,
+            2.6570933187031187e-05,
+            0.04848149046301842,
+            0.01478614378720522,
+            0.027048690244555473,
+            -0.012058171443641186,
+            -0.032824527472257614,
+            0.006117944605648518,
+            0.023796450346708298,
+            0.12625092267990112,
+            0.007517474237829447,
+            -0.004080851096659899,
+            -0.03270012512803078,
+            -0.01029876247048378,
+            -0.03732079640030861,
+            0.002994549460709095,
+            -0.030158756300807,
+            -0.02031317539513111,
+            0.021361712366342545,
+            -0.017620746046304703,
+            0.03206034004688263,
+            0.02752852812409401,
+            0.019246866926550865,
+            -0.020757470279932022,
+            -0.03646774962544441,
+            0.04862366244196892,
+            -0.004369643051177263,
+            0.012031513266265392,
+            0.0041452739387750626,
+            -0.024436235427856445,
+            -0.01317779440432787,
+            0.014368506148457527,
+            -0.004580683074891567,
+            -0.02411634288728237,
+            0.03417518362402916,
+            -0.020064370706677437,
+            0.026195643469691277,
+            0.04286559671163559,
+            -0.01928240992128849,
+            -0.015434814617037773,
+            -0.002169271232560277,
+            -0.013426600024104118,
+            0.0041008442640304565,
+            -0.018535993993282318,
+            -0.05061410740017891,
+            -0.006962105166167021,
+            -0.007939554750919342,
+            0.0924844816327095,
+            -0.012831244617700577,
+            0.023245522752404213,
+            -0.02607124112546444,
+            3.4328742913203314e-05,
+            -0.015274868346750736,
+            0.0298033207654953,
+            -0.013568774797022343,
+            -0.0069754342548549175,
+            -0.01673215627670288,
+            0.02038426324725151,
+            -0.04506930336356163,
+            0.015665847808122635,
+            -0.01008550077676773,
+            -0.010938547551631927,
+            -0.008441608399152756,
+            0.02219698764383793,
+            0.0029367911629378796,
+            0.035170406103134155,
+            0.05530586466193199,
+            -0.003258905140683055,
+            -0.06625329703092575,
+            0.04009319469332695,
+            0.004449616186320782,
+            0.009005863219499588,
+            -0.00478505901992321,
+            0.001390643883496523,
+            0.002912354888394475,
+            0.027155321091413498,
+            -0.004562911577522755,
+            0.016341175884008408,
+            -0.046846482902765274,
+            0.04155048355460167,
+            0.001919355127029121,
+            0.02104181982576847,
+            0.03643220290541649,
+            0.037143077701330185,
+            0.017149792984128,
+            0.006753286812454462,
+            0.03156273066997528,
+            0.02264128252863884,
+            0.006522253155708313,
+            0.015203780494630337,
+            0.006531138904392719,
+            0.011631648056209087,
+            0.0467398539185524,
+            0.004736186470836401,
+            0.0028012811671942472,
+            -0.008152816444635391,
+            -0.014608425088226795,
+            0.060957297682762146,
+            -0.0039808847941458225,
+            0.0034966031089425087,
+            0.011924882419407368,
+            -0.04453614726662636,
+            0.006513367407023907,
+            -0.010334305465221405,
+            0.02148611471056938,
+            0.017647404223680496,
+            0.015585875138640404,
+            0.032895613461732864,
+            0.021237310022115707,
+            -0.01701650582253933,
+            -0.030798541381955147,
+            -0.02534259669482708,
+            -0.02900358848273754,
+            -0.018011726438999176,
+            -0.004891689866781235,
+            0.023831993341445923,
+            -0.008308319374918938,
+            0.020490894094109535,
+            -0.008143930695950985,
+            -0.05004540830850601,
+            0.010680856183171272,
+            0.024347376078367233,
+            0.0011462816037237644,
+            -0.03884917125105858,
+            -0.03179376199841499,
+            -0.01540815643966198,
+            -0.025360368192195892,
+            -0.009223568253219128,
+            0.029110219329595566,
+            -0.01116958074271679,
+            0.02555585838854313,
+            0.02571580372750759,
+            -0.02038426324725151,
+            0.012555781751871109,
+            0.04592235013842583,
+            -0.016270089894533157,
+            0.0072997696697711945,
+            -0.00013884223881177604,
+            -0.025147106498479843,
+            0.009321312420070171,
+            -0.0197977926582098,
+            -0.019673390313982964,
+            -0.0338730625808239,
+            -0.017140908166766167,
+            -0.016909874975681305,
+            -0.02219698764383793,
+            -0.012466922402381897,
+            0.010494251735508442,
+            0.013702062889933586,
+            0.022694597020745277,
+            0.012031513266265392,
+            -0.03536589443683624,
+            0.015665847808122635,
+            0.007130937650799751,
+            -0.022730141878128052,
+            0.06554242223501205,
+            0.045460283756256104,
+            -0.004136388190090656,
+            -0.01041427906602621,
+            -0.046704307198524475,
+            0.0391690619289875,
+            0.032095883041620255,
+            -0.057296305894851685,
+            0.03236246109008789,
+            -0.017913982272148132,
+            0.042403530329465866,
+            0.02841711975634098,
+            0.025680260732769966,
+            -0.014377391897141933,
+            0.01803838461637497,
+            -0.005087179597467184,
+            -0.04777061566710472,
+            -0.034441761672496796,
+            -0.03433513268828392,
+            0.0076729776337742805,
+            -0.003112287726253271,
+            -0.012760157696902752,
+            -0.02900358848273754,
+            0.04133722186088562,
+            0.03214919939637184,
+            0.03563247248530388,
+            0.02104181982576847,
+            0.0030700797215104103,
+            0.00974783580750227,
+            0.016616638749837875,
+            -0.01890920288860798,
+            -0.014812801033258438,
+            -0.00617126002907753,
+            -0.03870699554681778,
+            -0.01540815643966198,
+            -0.017345283180475235,
+            0.002443623496219516,
+            -0.03756960108876228,
+            0.014155244454741478,
+            0.006144602317363024,
+            0.0014572881627827883,
+            0.011214010417461395,
+            0.002150388667359948,
+            0.03664546459913254,
+            -0.0016416707076132298,
+            -0.026906516402959824,
+            0.009943326003849506,
+            0.010405393317341805,
+            0.00527822645381093,
+            -0.029892180114984512,
+            -0.013799807988107204,
+            -0.006024642381817102,
+            0.027812877669930458,
+            -0.026266731321811676,
+            -0.009436829946935177,
+            -0.0020393149461597204,
+            0.0025813549291342497,
+            0.0016538887284696102,
+            -0.02024208754301071,
+            -0.016554437577724457,
+            0.014626197516918182,
+            0.0052115824073553085,
+            0.006637769751250744,
+            -0.04396745190024376,
+            -0.0198688805103302,
+            0.05573238804936409,
+            0.01586133800446987,
+            -0.06284110993146896,
+            0.023725362494587898,
+            -0.01658109575510025,
+            -0.011853795498609543,
+            0.006913233082741499,
+            0.02504047565162182,
+            -0.008406064473092556,
+            -0.01062754075974226,
+            0.0030611937399953604,
+            -0.052995529025793076,
+            -0.019389040768146515,
+            0.0035099319647997618,
+            0.004220804199576378,
+            -0.049263447523117065,
+            -0.019833335652947426,
+            -0.007597447372972965,
+            -0.015719164162874222,
+            0.003709864802658558,
+            0.009658977389335632,
+            -0.017416371032595634,
+            -0.010423164814710617,
+            0.012928989715874195,
+            0.018020613119006157,
+            -0.02299671806395054,
+            -0.03636111691594124,
+            0.027350811287760735,
+            -0.0019337947014719248,
+            -0.008099501021206379,
+            0.0016805464401841164,
+            0.012573553249239922,
+            0.0016838787123560905,
+            0.017771806567907333,
+            -0.005153824109584093,
+            -0.032682351768016815,
+            -0.02818608656525612,
+            0.02752852812409401,
+            -0.021734919399023056,
+            0.004147495608776808,
+            0.02278345637023449,
+            0.008303876966238022,
+            -0.011187352240085602,
+            0.020046597346663475,
+            -0.0527467243373394,
+            -0.008135044947266579,
+            0.0006581122288480401,
+            -0.006269005127251148,
+            0.025307053700089455,
+            -0.016829900443553925,
+            -0.014821686781942844,
+            0.019833335652947426,
+            0.009005863219499588,
+            0.0021626066882163286,
+            0.004014207050204277,
+            0.005966884084045887,
+            0.05512814596295357,
+            0.016154572367668152,
+            -0.01197819784283638,
+            0.017860665917396545,
+            -0.024596180766820908,
+            0.03102957457304001,
+            0.021290624514222145,
+            0.00529599841684103,
+            0.001194043317809701,
+            0.04581571742892265,
+            0.01008550077676773,
+            0.008757057599723339,
+            -0.011507244780659676,
+            0.011516130529344082,
+            -0.007401957642287016,
+            -0.05594564974308014,
+            -0.00806395709514618,
+            -0.022534651681780815,
+            0.002414744347333908,
+            0.0233699269592762,
+            0.031438328325748444,
+            -0.018376048654317856,
+            -0.00924133975058794,
+            -0.0009363520657643676,
+            -0.012102600187063217,
+            0.0010907446267083287,
+            -0.004687313921749592,
+            0.014359620399773121,
+            -0.0027457440737634897,
+            0.007686306722462177,
+            0.0018615968292579055,
+            -0.003058972302824259,
+            0.021343940868973732,
+            -0.03257572278380394,
+            -0.011853795498609543,
+            -0.03788949176669121,
+            -0.059784360229969025,
+            -0.03449507802724838,
+            0.030709682032465935,
+            0.023725362494587898,
+            -0.002539146924391389,
+            -0.004589569289237261,
+            -0.014324076473712921,
+            -0.031438328325748444,
+            -0.021592745557427406,
+            0.0006164595251902938,
+            0.04894355684518814,
+            -0.007868467830121517,
+            -0.02578689157962799,
+            0.01761186122894287,
+            -0.006939890794456005,
+            -0.004389636218547821,
+            -0.004927233327180147,
+            -0.03636111691594124,
+            0.023405469954013824,
+            0.010174359194934368,
+            0.03053196519613266,
+            -0.022143671289086342,
+            0.005575904622673988,
+            -0.006095729768276215,
+            -0.014377391897141933,
+            0.029963266104459763,
+            -0.008859246037900448,
+            0.004727300722151995,
+            0.0058869109489023685,
+            -0.003043422009795904,
+            0.0001384951319778338,
+            -0.005260454956442118,
+            -0.032540179789066315,
+            -0.026017924770712852,
+            -0.041230592876672745,
+            0.002370314672589302,
+            -0.02825717255473137,
+            -0.027137549594044685,
+            -0.06650210171937943,
+            0.024880530312657356,
+            -0.07098059356212616,
+            -0.02498716115951538,
+            0.038529276847839355,
+            -0.008885903283953667,
+            0.01768294721841812,
+            -0.04279451072216034,
+            -0.0006664427346549928,
+            -0.05107617378234863,
+            -0.0009580114856362343,
+            0.035241492092609406,
+            0.0029234623070806265,
+            -0.006393407471477985,
+            0.014368506148457527,
+            -0.00013120591756887734,
+            -0.03389083594083786,
+            -0.04229690134525299,
+            0.0038276028353720903,
+            0.005207139533013105,
+            -0.02301448956131935,
+            -0.014288532547652721,
+            0.04339875280857086,
+            0.00908583588898182,
+            -0.0006070183007977903,
+            0.0300876684486866,
+            0.007939554750919342,
+            -0.027137549594044685,
+            0.019246866926550865,
+            0.02564471773803234,
+            0.038884714245796204,
+            -0.02768847532570362,
+            0.021130679175257683,
+            -0.03200702369213104,
+            -0.005629220046103001,
+            0.011569445952773094,
+            0.0023636503610759974,
+            -0.006011313758790493,
+            0.014457364566624165,
+            -0.0015683620003983378,
+            0.0034077439922839403,
+            0.006842145696282387,
+            -0.006304548587650061,
+            0.032984472811222076,
+            -0.0004387414955999702,
+            -0.016821015626192093,
+            -0.021148450672626495,
+            0.009694520384073257,
+            0.031082890927791595,
+            0.028737012296915054,
+            -0.018304960802197456,
+            -0.014635083265602589,
+            -0.003976441919803619,
+            0.029056904837489128,
+            0.016083484515547752,
+            -0.01384423766285181,
+            0.0067444005981087685,
+            0.02431183122098446,
+            -0.023618731647729874,
+            -0.01586133800446987,
+            0.017398599535226822,
+            -0.006424508057534695,
+            0.012564667500555515,
+            -0.012644641101360321,
+            0.00200932496227324,
+            0.017931753769516945,
+            -0.005367085803300142,
+            0.026693254709243774,
+            -0.017034277319908142,
+            0.010529795661568642,
+            -0.007459715940058231,
+            -0.008681527338922024,
+            0.007917339913547039,
+            0.06948776543140411,
+            -0.020615296438336372,
+            0.022659054026007652,
+            0.018464908003807068,
+            -0.0018671504221856594,
+            0.034726109355688095,
+            -0.04457169026136398,
+            -0.046704307198524475,
+            0.012182573787868023,
+            -0.04314994812011719,
+            -0.0020493115298449993,
+            -0.022410249337553978,
+            -0.023032261058688164,
+            0.008321648463606834,
+            0.011915996670722961,
+            -0.024880530312657356,
+            -0.008717071264982224,
+            0.020864101126790047,
+            0.01577247865498066,
+            -0.02169937640428543,
+            0.019922195002436638,
+            0.00920579582452774,
+            -0.018749255686998367,
+            0.032682351768016815,
+            -0.02205481193959713,
+            -0.0003795946831814945,
+            -0.007099837064743042,
+            -0.0024947174824774265,
+            -0.00668219942599535,
+            -0.004285227041691542,
+            0.008868131786584854,
+            -0.02893250249326229,
+            -0.01622566021978855,
+            0.01680324412882328,
+            -0.00772185018286109,
+            -0.010058842599391937,
+            -0.042403530329465866,
+            0.0032611265778541565,
+            0.013782036490738392,
+            -0.023281067609786987,
+            0.01132064126431942,
+            -0.019477900117635727,
+            -0.032895613461732864,
+            0.01993996649980545,
+            0.03508154675364494,
+            -0.032984472811222076,
+            0.0016561102820560336,
+            0.00873040035367012,
+            -0.009028078056871891,
+            -0.0024502878077328205,
+            -0.01766517572104931,
+            0.007157595362514257,
+            0.03271789476275444,
+            0.014741714112460613,
+            -0.013257768005132675,
+            -0.024222973734140396,
+            -0.017807351425290108,
+            0.021379483863711357,
+            -0.0059313406236469746,
+            0.005162709858268499,
+            0.02562694624066353,
+            0.041514940559864044,
+            0.012173688039183617,
+            0.0008830366423353553,
+            -0.013666519895195961,
+            0.014803915284574032,
+            -0.015887996181845665,
+            0.021646060049533844,
+            0.022072583436965942,
+            -0.01746080070734024,
+            -0.03803166747093201,
+            0.018429363146424294,
+            0.00022339716088026762,
+            0.004554025363177061,
+            0.008112830109894276,
+            0.02768847532570362,
+            -0.016616638749837875,
+            -0.00549593148753047,
+            0.03934678062796593,
+            0.016314519569277763,
+            -0.014892774634063244,
+            -0.01622566021978855,
+            0.016838787123560905,
+            0.0074419439770281315,
+            0.006060186307877302,
+            -0.0018293854082003236,
+            -0.010369849391281605,
+            -0.0016127914423123002,
+            0.009365742094814777,
+            -0.008481594733893871,
+            0.008992534130811691,
+            0.014839459210634232,
+            0.005438172724097967,
+            0.0011035181814804673,
+            -0.01737194135785103,
+            -0.01263575442135334,
+            0.015799136832356453,
+            0.010538681410253048,
+            -0.010023298673331738,
+            0.00927688367664814,
+            0.007961769588291645,
+            0.014013069681823254,
+            0.032166969031095505,
+            -0.0031278380192816257,
+            -0.01671438477933407,
+            -0.0026657709386199713,
+            -0.0006958773010410368,
+            -0.025769120082259178,
+            0.007601890247315168,
+            -0.014892774634063244,
+            -0.02898581698536873,
+            0.018429363146424294,
+            0.02249910682439804,
+            -0.011418386362493038,
+            -0.01802949793636799,
+            -0.0030878514517098665,
+            -0.0043563139624893665,
+            -0.006264561787247658,
+            0.007406400516629219,
+            -0.011276211589574814,
+            0.042616792023181915,
+            -0.001704982714727521,
+            0.0023614289239048958,
+            -0.02703091874718666,
+            -0.0023503215052187443,
+            0.04332766681909561,
+            0.025253737345337868,
+            0.0030545294284820557,
+            0.009116937406361103,
+            0.008010641671717167,
+            0.0467398539185524,
+            -0.013764264062047005,
+            -0.04133722186088562,
+            0.008601554669439793,
+            -0.015319298021495342,
+            0.012173688039183617,
+            0.03266458213329315,
+            -0.009436829946935177,
+            0.010014412924647331,
+            0.010547567158937454,
+            -0.03147387132048607,
+            0.027350811287760735,
+            -0.023796450346708298,
+            -0.022605737671256065,
+            0.030994031578302383,
+            -0.01723865233361721,
+            0.01074305735528469,
+            0.028737012296915054,
+            0.010911889374256134,
+            -0.029838863760232925,
+            0.0013817580183967948,
+            -0.010423164814710617,
+            -0.025218194350600243,
+            0.025911293923854828,
+            0.02863038145005703,
+            0.018642624840140343,
+            0.0063001057133078575,
+            0.0015339291421696544,
+            0.004896132741123438,
+            0.0033699788618832827,
+            -0.01007661409676075,
+            0.013204452581703663,
+            -0.010165473446249962,
+            -0.01768294721841812,
+            -0.0015283754328265786,
+            -0.013568774797022343,
+            -0.00927688367664814,
+            0.0021748249419033527,
+            -0.02607124112546444,
+            0.009134708903729916,
+            0.012351405806839466,
+            0.010458708740770817,
+            -0.018962517380714417,
+            -0.01862485334277153,
+            0.05534140765666962,
+            0.013799807988107204,
+            -0.0032700125593692064,
+            0.004087515641003847,
+            -0.017585203051567078,
+            -0.04318549111485481,
+            0.004571797326207161,
+            0.011418386362493038,
+            0.005562575533986092,
+            0.00020270967797841877,
+            -0.030496420338749886,
+            0.008108386769890785,
+            0.032326918095350266,
+            0.008046185597777367,
+            0.04446506127715111,
+            -0.0012251439038664103,
+            0.030123213306069374,
+            -0.008414950221776962,
+            0.003574354574084282,
+            0.019513443112373352,
+            0.0041230591014027596,
+            -0.019033605232834816,
+            0.03038978949189186,
+            0.01834050565958023,
+            0.018304960802197456,
+            0.006979877129197121,
+            -0.04514038935303688,
+            -0.009872239083051682,
+            0.010236560367047787,
+            0.03182930499315262,
+            0.020259859040379524,
+            -0.017176451161503792,
+            -0.007437501102685928,
+            0.03296670317649841,
+            -0.022232530638575554,
+            -0.004431844223290682,
+            0.0116138756275177,
+            -0.008597111329436302,
+            0.002483610063791275,
+            -0.004687313921749592,
+            0.02249910682439804,
+            -0.039844390004873276,
+            -0.004687313921749592,
+            0.0019349055364727974,
+            0.0057225218042731285,
+            -0.002215922111645341,
+            -0.006939890794456005,
+            0.010405393317341805,
+            0.014261875301599503,
+            0.009076950140297413,
+            0.02943011187016964,
+            -0.0201532281935215,
+            0.015034948475658894,
+            -0.015168237499892712,
+            -0.003134502563625574,
+            0.005749179515987635,
+            -0.03138501197099686,
+            -0.030283158645033836,
+            0.007619662210345268,
+            0.03175821900367737,
+            0.013080050237476826,
+            0.0014817244373261929,
+            -0.004589569289237261,
+            -0.011516130529344082,
+            0.026853200048208237,
+            0.02109513431787491,
+            0.02111290581524372,
+            0.023831993341445923,
+            -0.005087179597467184,
+            -0.0004323547473177314,
+            0.012475808151066303,
+            -0.008650426752865314,
+            -0.024791670963168144,
+            0.008930332958698273,
+            -0.011427272111177444,
+            0.0270842332392931,
+            -0.01724753901362419,
+            -0.003258905140683055,
+            0.005646991543471813,
+            -0.012404721230268478,
+            -0.01128509733825922,
+            -0.039702218025922775,
+            0.025893522426486015,
+            -0.02548477053642273,
+            0.0036343345418572426,
+            -0.019193550571799278,
+            0.01490166038274765,
+            -0.03870699554681778,
+            0.000813615508377552,
+            -0.0018393819918856025,
+            0.010538681410253048,
+            -0.010529795661568642,
+            0.0021770463790744543,
+            -0.03193593770265579,
+            -0.03788949176669121,
+            -0.0116849634796381,
+            0.005073850974440575,
+            0.007899568416178226,
+            -0.0009330198518000543,
+            0.006913233082741499,
+            0.015967968851327896,
+            -0.025093792006373405,
+            -0.03369534760713577,
+            0.018820343539118767,
+            -0.03440621867775917,
+            0.020419806241989136,
+            0.0004870585980825126,
+            -0.02804391086101532,
+            -0.021965952590107918,
+            -0.016492236405611038,
+            -0.015248210169374943,
+            0.021006274968385696,
+            0.022552423179149628,
+            0.017798464745283127,
+            -0.016847673803567886,
+            0.0025124892126768827,
+            0.014039727859199047,
+            0.008121715858578682,
+            0.011880452744662762,
+            0.008481594733893871,
+            0.01825164631009102,
+            -0.016847673803567886,
+            -0.015887996181845665,
+            -0.012244774959981441,
+            -0.002794616622850299,
+            -0.013799807988107204,
+            0.0011179577559232712,
+            -0.038813624531030655,
+            -0.00737529993057251,
+            0.01116958074271679,
+            -0.0020048818551003933,
+            0.06038859859108925,
+            0.0527467243373394,
+            0.013160022906959057,
+            0.007384185679256916,
+            -0.01876702904701233,
+            0.0461711548268795,
+            -0.025804663076996803,
+            0.04986768960952759,
+            0.0034521734341979027,
+            -0.013142251409590244,
+            -0.013462143950164318,
+            0.013426600024104118,
+            -0.029199078679084778,
+            -0.00719313882291317,
+            -0.005424844101071358,
+            -0.009005863219499588,
+            0.018002839758992195,
+            0.00739307189360261,
+            -0.009347970597445965,
+            0.003696535946801305,
+            0.009383514523506165,
+            -0.011258440092206001,
+            0.013222224079072475,
+            0.01612791419029236,
+            0.0017349726986140013,
+            0.015070492401719093,
+            -0.04208363965153694,
+            -0.011231781914830208,
+            -0.00041291682282462716,
+            0.009685634635388851,
+            -0.006220132578164339,
+            0.013168908655643463,
+            -0.006428950931876898,
+            -0.0044229584746062756,
+            -0.007015420589596033,
+            -0.02169937640428543,
+            0.022534651681780815,
+            0.036041226238012314,
+            -0.0016938752960413694,
+            -0.048979099839925766,
+            -0.02614232711493969,
+            -0.050223127007484436,
+            -0.012928989715874195,
+            0.024063026532530785,
+            -0.0050249784253537655,
+            0.015976853668689728,
+            0.004087515641003847,
+            0.0048828041180968285,
+            -0.0030456434469670057,
+            -0.017567431554198265,
+            0.01613680087029934,
+            -0.0232277512550354,
+            0.002739079762250185,
+            0.0003854260721709579,
+            0.007144266273826361,
+            -0.03543698415160179,
+            0.007757393643260002,
+            0.00023422685626428574,
+            0.012093714438378811,
+            -0.03660992160439491,
+            -0.038244929164648056,
+            -0.017398599535226822,
+            0.015941310673952103,
+            -0.002383643528446555,
+            -0.012244774959981441,
+            0.0029501200187951326,
+            0.005033864174038172,
+            0.014741714112460613,
+            -0.017567431554198265,
+            -0.006229018326848745,
+            0.02162828855216503,
+            -0.0032433548476547003,
+            0.007206467911601067,
+            -0.035099320113658905,
+            -0.01555921696126461,
+            -0.01278681494295597,
+            0.0116494195535779,
+            -0.0133999427780509,
+            0.004905018489807844,
+            0.00017799575289245695,
+            0.014848344959318638,
+            -0.015505901537835598,
+            -0.027013147249817848,
+            -0.011231781914830208,
+            0.020099913701415062,
+            -0.03511708974838257,
+            0.036112312227487564,
+            0.002739079762250185,
+            0.02118399366736412,
+            -0.014404049143195152,
+            0.025609172880649567,
+            -0.004036421421915293,
+            -0.015301525592803955,
+            -0.002151499269530177,
+            0.022001497447490692,
+            0.03332213684916496,
+            -0.021361712366342545,
+            0.0013228888856247067,
+            -0.019015833735466003,
+            0.003327770857140422,
+            -0.02351210080087185,
+            0.017345283180475235,
+            -0.004007542505860329,
+            0.01234252005815506,
+            0.017105363309383392,
+            0.013204452581703663,
+            0.05996207520365715,
+            -0.017194222658872604,
+            0.02557362988591194,
+            0.012324748560786247,
+            -0.017265310510993004,
+            -0.007935111410915852,
+            -0.032095883041620255,
+            -0.027333039790391922,
+            0.03133169561624527,
+            0.002228140365332365,
+            0.0015561438631266356,
+            -0.03104734607040882,
+            0.015879109501838684,
+            0.01832273229956627,
+            0.019460128620266914,
+            -0.014288532547652721,
+            -0.015274868346750736,
+            0.016030170023441315,
+            0.026248957961797714,
+            -0.008135044947266579,
+            0.003865367965772748,
+            0.012769043445587158,
+            -0.04076852649450302,
+            0.005282669793814421,
+            0.0029634488746523857,
+            0.012857901863753796,
+            -0.0021548315417021513,
+            0.010183244943618774,
+            -0.03214919939637184,
+            -0.0055137029848992825,
+            -0.01457288209348917,
+            0.02315666526556015,
+            -0.015665847808122635,
+            -0.047806162387132645,
+            -0.03161604329943657,
+            0.025147106498479843,
+            -0.030194299295544624,
+            -0.020988503471016884,
+            0.01311559323221445,
+            0.0022170329466462135,
+            0.015337069518864155,
+            -0.01471505593508482,
+            -0.02747521363198757,
+            -0.038955800235271454,
+            -0.021965952590107918,
+            -0.0006886575138196349,
+            0.0394534096121788,
+            -0.03010544180870056,
+            0.015159351751208305,
+            0.005309327505528927,
+            0.00686436053365469,
+            -0.004869475029408932,
+            0.002365871798247099,
+            -0.029447883367538452,
+            -0.04133722186088562,
+            0.02351210080087185,
+            -0.00494944816455245,
+            -0.0015150465769693255,
+            0.014803915284574032,
+            -0.033055562525987625,
+            -0.023103348910808563,
+            0.003154495730996132,
+            0.00737529993057251,
+            -0.00785069540143013,
+            -0.002352542942389846,
+            0.0013395498972386122,
+            -0.01215591561049223,
+            -0.038600362837314606,
+            -0.004416293930262327,
+            0.03397969529032707,
+            0.02177046425640583,
+            0.031740445643663406,
+            0.011525017209351063,
+            0.00010468705295352265,
+            -0.02886141464114189,
+            -0.013595432043075562,
+            -0.006895461119711399,
+            -0.0009818923426792026,
+            -0.0025769120547920465,
+            0.01848267950117588,
+            0.005291555542498827,
+            0.0034077439922839403,
+            -0.02169937640428543,
+            -0.02783064916729927,
+            -0.03511708974838257,
+            -0.01928240992128849,
+            -0.010049956850707531,
+            -0.00949014537036419,
+            -0.03899134323000908,
+            -0.016741042956709862,
+            0.04485604166984558,
+            0.026995373889803886,
+            -0.009152480401098728,
+            -0.00018729818111751229,
+            -0.0012240331852808595,
+            -0.012973419390618801,
+            0.019886652007699013,
+            -0.013826466165482998,
+            0.0059091257862746716,
+            0.00031656029750593007,
+            0.018607081845402718,
+            0.0017538551473990083,
+            0.0010013302089646459,
+            -0.03419295698404312,
+            -0.009401286020874977,
+            0.031171750277280807,
+            0.00701097771525383,
+            -0.019531216472387314,
+            0.0163678340613842,
+            0.009259111247956753,
+            -0.014466251246631145,
+            -0.047806162387132645,
+            0.026782112196087837,
+            0.04567354544997215,
+            0.02212589979171753,
+            -0.005940226372331381,
+            0.022179214283823967,
+            0.024933844804763794,
+            -0.0012462479062378407,
+            -0.0007541910745203495,
+            -0.004945005290210247,
+            0.026977602392435074,
+            0.006535581778734922,
+            -0.000127387756947428,
+            0.00939240027219057,
+            0.01810947060585022,
+            0.005838038399815559,
+            -0.0007697413675487041,
+            0.007077622227370739,
+            -0.020793013274669647,
+            -0.004913904704153538,
+            -0.013853123411536217,
+            0.010974090546369553,
+            0.00920579582452774,
+            0.041088417172431946,
+            0.008774830028414726,
+            0.013559889048337936,
+            -0.0043563139624893665,
+            -0.010174359194934368,
+            0.028648152947425842,
+            0.014039727859199047,
+            0.001577247865498066,
+            0.008499366231262684,
+            -0.030834084376692772,
+            -0.01586133800446987,
+            -0.013950868509709835,
+            0.01384423766285181,
+            0.0037520728074014187,
+            0.04286559671163559,
+            0.006873246282339096,
+            0.0016794357215985656,
+            -0.02943011187016964,
+            0.0026235629338771105,
+            0.008232789114117622,
+            0.02285454422235489,
+            -0.013266653753817081,
+            -2.6171763238380663e-05,
+            0.0073664141818881035,
+            0.005638105794787407,
+            0.026497764512896538,
+            -0.026195643469691277,
+            -0.017052048817276955,
+            0.019300181418657303,
+            0.09511470794677734,
+            -0.009330199100077152,
+            -0.0233699269592762,
+            -0.026053469628095627,
+            -0.02104181982576847,
+            0.008241675794124603,
+            0.017585203051567078,
+            0.005429286975413561,
+            0.017443029209971428,
+            0.010218788869678974,
+            0.01788732409477234,
+            0.015603646636009216,
+            -0.015976853668689728,
+            -0.007419729605317116,
+            -0.03225582838058472,
+            0.0327356681227684,
+            0.00014911657490301877,
+            -0.0047228578478097916,
+            0.00739307189360261,
+            0.007135380525141954,
+            0.019744478166103363,
+            0.018091699108481407,
+            0.015887996181845665,
+            -0.007979541085660458,
+            -0.011924882419407368,
+            0.020135456696152687,
+            -0.013142251409590244,
+            -0.0034810525830835104,
+            -0.01899806223809719,
+            -0.021592745557427406,
+            -0.005651434417814016,
+            0.014643969014286995,
+            -0.006468937732279301,
+            0.014848344959318638,
+            -0.01132064126431942,
+            0.028310488909482956,
+            -0.021965952590107918,
+            0.011160694994032383,
+            -0.00737529993057251,
+            -0.01526598259806633,
+            0.005216025281697512,
+            0.010245447047054768,
+            -0.007677420508116484,
+            0.010716399177908897,
+            -0.014324076473712921,
+            0.014821686781942844,
+            0.019477900117635727,
+            -0.03650329262018204,
+            0.03328659385442734,
+            0.008126158267259598,
+            -0.009836695156991482,
+            0.00924133975058794,
+            -0.00582470977678895,
+            0.012529123574495316,
+            0.007268669083714485,
+            0.00806395709514618,
+            0.008352749049663544,
+            0.00817947369068861,
+            -0.007659649010747671,
+            0.014883887954056263,
+            0.012138144113123417,
+            0.0007353085093200207,
+            0.010636426508426666,
+            0.014279646798968315,
+            -0.009730064310133457,
+            -0.004696200136095285,
+            0.0038364888168871403,
+            -0.010325419716536999,
+            0.007317541632801294,
+            -0.009108050726354122,
+            0.0060912868939340115,
+            0.024507321417331696,
+            -0.0017083149868994951,
+            -0.016492236405611038,
+            -0.016625525429844856,
+            -0.007935111410915852,
+            -0.021237310022115707,
+            -0.015434814617037773,
+            0.00514938123524189,
+            -0.030834084376692772,
+            0.013053392060101032,
+            0.028665924444794655,
+            0.0233699269592762,
+            -0.006180145777761936,
+            -0.024720583111047745,
+            -0.012413606978952885,
+            -0.0018582645570859313,
+            0.011276211589574814,
+            0.005162709858268499,
+            -0.02454286627471447,
+            0.002643556334078312,
+            -0.011356184259057045,
+            -0.021432798355817795,
+            -0.008277218788862228,
+            -0.01651000790297985,
+            0.003458837978541851,
+            0.0030456434469670057,
+            -0.008441608399152756,
+            -0.010707513429224491,
+            0.003541032550856471,
+            -0.03685872629284859,
+            0.014048613607883453,
+            0.03207810968160629,
+            -0.000934130628593266,
+            -0.03255794942378998,
+            -0.015221552923321724,
+            -0.003250019159168005,
+            -0.017194222658872604,
+            -0.02863038145005703,
+            0.008934776298701763,
+            0.006104615516960621,
+            -0.008454937487840652,
+            0.01252023782581091,
+            -0.004036421421915293,
+            0.02001105435192585,
+            0.0009130265680141747,
+            0.01869594119489193,
+            -0.0408751554787159,
+            0.012564667500555515,
+            0.019158007577061653,
+            -0.019122464582324028,
+            0.006904346868395805,
+            0.023458784446120262,
+            -0.008383849635720253,
+            0.0034699453972280025,
+            0.035756874829530716,
+            0.010369849391281605,
+            -0.042474620044231415,
+            0.01905137673020363,
+            0.006082401145249605,
+            -0.00668219942599535,
+            0.016323404386639595,
+            0.019673390313982964,
+            -0.000788623932749033,
+            0.020757470279932022,
+            -0.006446722894906998,
+            -0.018233874812722206,
+            0.00218260008841753,
+            0.020846329629421234,
+            -0.009330199100077152,
+            -0.015345955267548561,
+            0.01092077512294054,
+            0.0027235294692218304,
+            0.004758401308208704,
+            0.01738082803785801,
+            -0.02607124112546444,
+            -0.018642624840140343,
+            0.019513443112373352,
+            0.012831244617700577,
+            0.017336398363113403,
+            -0.011214010417461395,
+            -0.01402195543050766,
+            0.02219698764383793,
+            -0.005464830435812473,
+            0.010707513429224491,
+            -0.031082890927791595,
+            0.008748171851038933,
+            -0.0019904423970729113,
+            0.01848267950117588,
+            -0.004918347578495741,
+            0.008752615191042423,
+            0.03255794942378998,
+            -0.015514787286520004,
+            -0.025964610278606415,
+            0.009543460793793201,
+            -0.033624257892370224,
+            -0.010094386525452137,
+            -0.029607830569148064,
+            -0.03920460492372513,
+            0.03170490264892578,
+            -0.008046185597777367,
+            -0.0049583339132368565,
+            0.026746569201350212,
+            0.007353085093200207,
+            0.0020270966924726963,
+            -0.03213142603635788,
+            0.021574974060058594,
+            0.006642212625592947,
+            0.007784051354974508,
+            -0.018873659893870354,
+            0.025147106498479843,
+            -0.0030922943260520697,
+            -0.021166222169995308,
+            0.013462143950164318,
+            0.012182573787868023,
+            0.008126158267259598,
+            -0.004534032195806503,
+            0.001586133730597794,
+            -0.03682318329811096,
+            -0.005775837227702141,
+            0.008246118202805519,
+            0.005522589199244976,
+            -0.003236690303310752,
+            0.018802572041749954,
+            -0.016909874975681305,
+            -0.002843489171937108,
+            -0.005629220046103001,
+            -0.022747913375496864,
+            0.0033855291549116373,
+            -0.03248686343431473,
+            -0.020686382427811623,
+            0.019780021160840988,
+            -0.003989770542830229,
+            -0.01825164631009102,
+            0.014581767842173576,
+            -0.03980884701013565,
+            -0.011178466491401196,
+            0.007304212544113398,
+            0.027777334675192833,
+            -0.03447730466723442,
+            0.015719164162874222,
+            -0.01958453096449375,
+            -0.0023769792169332504,
+            0.023618731647729874,
+            -0.011809365823864937,
+            -0.00010725563333835453,
+            -0.012564667500555515,
+            -0.031882621347904205,
+            0.003963112831115723,
+            0.028239401057362556,
+            -0.029234621673822403,
+            0.0009446826297789812,
+            0.031136205419898033,
+            -0.000653669296298176,
+            0.004616227000951767,
+            0.0007586340070702136,
+            0.01113403681665659,
+            -0.015186008997261524,
+            0.015106036327779293,
+            0.005838038399815559,
+            0.00582470977678895,
+            -0.01372872106730938,
+            -0.005202696658670902,
+            0.02710200473666191,
+            0.001230697613209486,
+            -0.008819258771836758,
+            0.0047006430104374886,
+            0.004058636259287596,
+            0.010174359194934368,
+            -0.0016627747099846601,
+            0.014457364566624165,
+            0.018100585788488388,
+            -0.028168313205242157,
+            0.012884560041129589,
+            0.009836695156991482,
+            7.844586798455566e-05,
+            0.013995298184454441,
+            -0.017514115199446678,
+            -0.034352902323007584,
+            0.02161051705479622,
+            -0.022019268944859505,
+            -0.016234545037150383,
+            0.0270842332392931,
+            -0.010867459699511528,
+            -0.006064629182219505,
+            -0.009774493984878063,
+            -0.001900472561828792,
+            0.010609768331050873,
+            -0.009978869929909706,
+            -0.009463487192988396,
+            0.023760905489325523,
+            0.018145015463232994,
+            0.020917417481541634,
+            0.029856635257601738,
+            0.0012118150480091572,
+            0.01570139080286026,
+            -0.0005223245243541896,
+            0.008974762633442879,
+            0.005171595606952906,
+            -0.0018627075478434563,
+            -0.01738971285521984,
+            0.025982381775975227,
+            -0.001804949133656919,
+            0.006264561787247658,
+            0.0063223205506801605,
+            -0.014368506148457527,
+            -0.010867459699511528,
+            -0.026479993015527725,
+            -0.004887246992439032,
+            -0.006580011453479528,
+            -0.016234545037150383,
+            0.0789068192243576,
+            -0.0099344402551651,
+            -0.016554437577724457,
+            0.010494251735508442,
+            -0.004856146406382322,
+            0.0015528115909546614,
+            -0.016847673803567886,
+            0.003736522514373064,
+            -0.008783715777099133,
+            -0.000988556770607829,
+            -0.030176527798175812,
+            -0.0015616975724697113,
+            0.0167943574488163,
+            0.015283754095435143,
+            -0.006557796616107225,
+            -0.018731484189629555,
+            0.008223903365433216,
+            -0.023209979757666588,
+            0.02036648988723755,
+            0.0012418050318956375,
+            0.018447136506438255,
+            0.005087179597467184,
+            0.005082736723124981,
+            -0.014946090057492256,
+            -0.016749927774071693,
+            0.004434065893292427,
+            0.012315861880779266,
+            0.002247022930532694,
+            -0.0090236347168684,
+            -0.009152480401098728,
+            -0.011009634472429752,
+            -0.021859321743249893,
+            0.020544208586215973,
+            -0.01296453271061182,
+            -0.0039142402820289135,
+            -0.003983106464147568,
+            -0.0020726369693875313,
+            0.006331206299364567,
+            0.011356184259057045,
+            -0.02358318865299225,
+            0.0018849222688004375,
+            -0.016972076147794724,
+            0.006913233082741499,
+            0.02504047565162182,
+            0.00981892365962267,
+            -0.04720192030072212,
+            -0.00651781028136611,
+            -0.02754630148410797,
+            0.013746492564678192,
+            -0.011747164651751518,
+            -0.009445715695619583,
+            0.0133999427780509,
+            0.008072842843830585,
+            0.028719240799546242,
+            0.010280990041792393,
+            0.009881124831736088,
+            0.015372613444924355,
+            -0.00032850075513124466,
+            -0.004440730437636375,
+            -0.017647404223680496,
+            0.013568774797022343,
+            -0.023316610604524612,
+            0.020348718389868736,
+            -0.0006103505147621036,
+            -0.017034277319908142,
+            0.004194146487861872,
+            0.002821274334564805,
+            0.013853123411536217,
+            -0.021806007251143456,
+            -0.022481335327029228,
+            0.022730141878128052,
+            0.018216101452708244,
+            0.012600211426615715,
+            0.015123807825148106,
+            0.03182930499315262,
+            0.02139725536108017,
+            0.024294059723615646,
+            -0.025324825197458267,
+            0.0014428485883399844,
+            -0.033855292946100235,
+            -0.005087179597467184,
+            -0.026551079005002975,
+            0.004038643091917038,
+            -0.012973419390618801,
+            0.005638105794787407,
+            -0.022001497447490692,
+            0.001768294838257134,
+            0.017549660056829453,
+            -0.030691910535097122,
+            0.010965204797685146,
+            0.015754707157611847,
+            0.00788623932749033,
+            0.017993954941630363,
+            0.006944333668798208,
+            -0.012831244617700577,
+            -0.01737194135785103,
+            -0.008717071264982224,
+            -0.013782036490738392,
+            0.014812801033258438,
+            0.0020082141272723675,
+            -0.0027679589111357927,
+            -0.00027990597300231457,
+            0.0001258604897884652,
+            -0.014466251246631145,
+            -0.001746080000884831,
+            -0.005180481821298599,
+            -0.013710948638617992,
+            0.005926897749304771,
+            0.0014928317395970225,
+            0.013222224079072475,
+            0.008917003870010376,
+            -0.007286441046744585,
+            0.011293983086943626,
+            0.02578689157962799,
+            0.0005528698093257844,
+            -0.018358277156949043,
+            0.00721091078594327,
+            0.008290547877550125,
+            0.007215353660285473,
+            0.006073514930903912,
+            0.0015172680141404271,
+            -0.012253660708665848,
+            0.027333039790391922,
+            -0.023938624188303947,
+            0.02205481193959713,
+            0.004856146406382322,
+            -0.0008075064979493618,
+            0.0003068413643632084,
+            -0.03518817946314812,
+            0.002401415491476655,
+            -0.024222973734140396,
+            -0.04112396016716957,
+            -0.02411634288728237,
+            -0.01547924429178238,
+            0.0007264225860126317,
+            0.022765684872865677,
+            0.0012862344738095999,
+            -0.027297494933009148,
+            0.022907858714461327,
+            -0.001922687399201095,
+            -0.015959082171320915,
+            0.00420081103220582,
+            -0.001362875453196466,
+            0.0019515665480867028,
+            0.0028745897579938173,
+            -0.0029567843303084373,
+            -0.0081839170306921,
+            -0.017407484352588654,
+            -0.02635558880865574,
+            0.003163381712511182,
+            0.009223568253219128,
+            -0.019815564155578613,
+            0.02249910682439804,
+            -0.009978869929909706,
+            -0.015879109501838684,
+            0.026497764512896538,
+            -0.009312426671385765,
+            -0.021343940868973732,
+            -0.021308396011590958,
+            0.02095296047627926,
+            0.004669542424380779,
+            -0.024418462067842484,
+            0.01643003523349762,
+            0.005562575533986092,
+            -0.017638517543673515,
+            -0.002251465804874897,
+            -0.012040399014949799,
+            0.012351405806839466,
+            -0.01782512292265892,
+            0.0016472243005409837,
+            0.010503137484192848,
+            0.0017738484311848879,
+            0.0030278717167675495,
+            -0.01095631904900074,
+            0.019993282854557037,
+            0.01825164631009102,
+            0.008046185597777367,
+            -0.03200702369213104,
+            0.008819258771836758,
+            -0.007588561624288559,
+            -0.0007630769396200776,
+            -0.020419806241989136,
+            0.029821092262864113,
+            0.013435485772788525,
+            0.007975098676979542,
+            0.01055645290762186,
+            -0.006491152569651604,
+            0.006282333750277758,
+            -0.009685634635388851,
+            -0.002785730641335249,
+            -0.013319969177246094,
+            -0.029678918421268463,
+            -0.007593004498630762,
+            0.023760905489325523,
+            -0.006451165769249201,
+            0.01095631904900074,
+            -0.004316327627748251,
+            0.029270166531205177,
+            -0.015106036327779293,
+            0.000624234729912132,
+            0.02477389946579933,
+            -0.0022370261140167713,
+            0.0058646961115300655,
+            0.011533902958035469,
+            0.06394296139478683,
+            -0.003012321190908551,
+            0.013239996507763863,
+            0.0013817580183967948,
+            0.002856818027794361,
+            0.007948440499603748,
+            -0.015070492401719093,
+            -0.03190039470791817,
+            0.001449513016268611,
+            0.0043718647211790085,
+            -0.013435485772788525,
+            -0.0016072377329692245,
+            -0.0018793685594573617,
+            0.001987110124900937,
+            -0.004909461829811335,
+            0.018153900280594826,
+            -0.020935188978910446,
+            0.028737012296915054,
+            0.019851109012961388,
+            0.012884560041129589,
+            -0.010387620888650417,
+            0.015941310673952103,
+            0.025253737345337868,
+            -0.008121715858578682,
+            -0.0038498176727443933,
+            0.007401957642287016,
+            0.0053937435150146484,
+            0.023760905489325523,
+            0.0027079791761934757,
+            0.029981037601828575,
+            -0.0035343680065125227,
+            0.003134502563625574,
+            -0.014270761050283909,
+            -0.001956009538844228,
+            -0.0033988580107688904,
+            0.005904682911932468,
+            0.003414408303797245,
+            0.0014061941765248775,
+            -0.016403377056121826,
+            0.013648747466504574,
+            0.004176374524831772,
+            0.003367757424712181,
+            0.01456399541348219,
+            -0.0006292330217547715,
+            -0.002503603231161833,
+            0.013444371521472931,
+            -0.002197039546445012,
+            -0.0013151137391105294,
+            0.008143930695950985,
+            -0.012315861880779266,
+            -0.016332291066646576,
+            -0.008255003951489925,
+            0.000503719667904079,
+            -0.006344534922391176,
+            -0.01958453096449375,
+            -0.004847260192036629,
+            0.022232530638575554,
+            0.0022259189281612635,
+            0.031011803075671196,
+            -0.003445509122684598,
+            0.008277218788862228,
+            0.016980960965156555,
+            -0.011489473283290863,
+            -0.0005217691650614142,
+            0.009312426671385765,
+            -0.0034566165413707495,
+            -0.0016960968496277928,
+            -0.004984991624951363,
+            0.00048372638411819935,
+            -0.009001419879496098,
+            3.720972017617896e-05,
+            0.0167232695966959,
+            0.011036291718482971,
+            0.0025458112359046936,
+            -0.005913568660616875,
+            -0.0017027612775564194,
+            -0.011844909749925137,
+            -0.006979877129197121,
+            -0.004487381316721439,
+            0.0017105364240705967,
+            -0.0011201791930943727,
+            0.01182713732123375,
+            0.00040319786057807505,
+            0.010218788869678974,
+            -0.010982976295053959,
+            -0.0011357294861227274,
+            -0.006548910867422819,
+            0.020419806241989136,
+            0.0053937435150146484,
+            -0.0043496498838067055,
+            0.007566346786916256,
+            -0.011027405969798565,
+            0.021379483863711357,
+            0.005291555542498827,
+            -0.00226479466073215,
+            0.0039075762033462524,
+            -0.0017560767009854317,
+            0.01149835903197527,
+            -0.0010962983360514045,
+            -0.013239996507763863,
+            0.02891472913324833,
+            0.017638517543673515,
+            0.027404125779867172,
+            0.0017405262915417552,
+            0.005735850892961025,
+            -0.00888146087527275,
+            0.027510756626725197,
+            0.01585245132446289,
+            -0.0017427478451281786,
+            -0.028150541707873344,
+            0.004243019036948681,
+            -0.012067057192325592,
+            -0.001978224376216531,
+            0.009614547714591026,
+            -0.01095631904900074,
+            -0.009552346542477608,
+            -0.01607459969818592,
+            -0.03138501197099686,
+            -0.026782112196087837,
+            0.0038164956495165825,
+            0.007068736013025045,
+            -0.008312762714922428,
+            0.0008847027784213424,
+            -0.015301525592803955,
+            -0.009543460793793201,
+            -0.025591401383280754,
+            0.002565804636105895,
+            -0.007313098758459091,
+            -0.037356339395046234,
+            -0.02381422184407711,
+            -0.009330199100077152,
+            -0.007166481111198664,
+            0.006611112039536238,
+            -0.0036898714024573565,
+            -0.02461395226418972,
+            0.011729392223060131,
+            -0.00974783580750227,
+            0.011542788706719875,
+            0.017700720578432083,
+            0.007535246200859547,
+            -0.010547567158937454,
+            -0.013559889048337936,
+            -0.009481258690357208,
+            -0.01628786139190197,
+            0.02052643708884716,
+            -0.010360963642597198,
+            0.006215689703822136,
+            -0.0058646961115300655,
+            -0.0018138349987566471,
+            0.0014184123137965798,
+            0.01832273229956627,
+            -0.03796058148145676,
+            -0.027190864086151123,
+            -0.00651781028136611,
+            0.011382842436432838,
+            0.0004651214985642582,
+            -0.012600211426615715,
+            0.014261875301599503,
+            -0.006406736560165882,
+            -0.02031317539513111,
+            -0.014519566670060158,
+            -0.001221811748109758,
+            -0.011054064147174358,
+            0.00840162206441164,
+            -0.011569445952773094,
+            0.01555921696126461,
+            0.0035299251321703196,
+            0.005349313840270042,
+            -0.00651781028136611,
+            0.015967968851327896,
+            0.008921447210013866,
+            -0.021343940868973732,
+            -0.003747629700228572,
+            -0.009463487192988396,
+            0.0017660732846707106,
+            0.01622566021978855,
+            0.007966212928295135,
+            -0.019815564155578613,
+            0.01738082803785801,
+            0.02687097154557705,
+            0.01315113715827465,
+            0.02118399366736412,
+            0.002925683744251728,
+            0.00752636045217514,
+            -0.001412858604453504,
+            0.011062949895858765,
+            -0.004811716731637716,
+            0.0002893472264986485,
+            0.00702874967828393,
+            -0.01628786139190197,
+            -0.011604989878833294,
+            0.0359523668885231,
+            0.000578694452997297,
+            -0.021219536662101746,
+            0.018153900280594826,
+            -0.005753622390329838,
+            -0.009076950140297413,
+            0.005900240037590265,
+            -0.0020581972785294056,
+            0.002081522950902581,
+            -0.010156587697565556,
+            0.009854466654360294,
+            -0.007961769588291645,
+            0.0057003069669008255,
+            0.006557796616107225,
+            0.004651770461350679,
+            -0.016190115362405777,
+            0.01658109575510025,
+            0.029092447832226753,
+            0.02616010047495365,
+            -0.042616792023181915,
+            -0.006753286812454462,
+            0.005424844101071358,
+            -0.018944745883345604,
+            -0.013675405643880367,
+            0.026017924770712852,
+            -0.0022559086792171,
+            -0.011658305302262306,
+            0.014039727859199047,
+            -0.009472372941672802,
+            -0.015434814617037773,
+            -0.0014961640117689967,
+            0.023938624188303947,
+            0.008623769506812096,
+            0.006402293220162392,
+            -0.007939554750919342,
+            0.005220468156039715,
+            -0.021077362820506096,
+            0.006242347415536642,
+            -0.006095729768276215,
+            0.009481258690357208,
+            -0.005349313840270042,
+            -0.005487045273184776,
+            0.012946761213243008,
+            0.005442616064101458,
+            -0.004847260192036629,
+            0.018589310348033905,
+            -0.00038292689714580774,
+            0.001768294838257134,
+            -0.012422492727637291,
+            0.0201532281935215,
+            -0.000900253071449697,
+            -0.009952211752533913,
+            -0.005424844101071358,
+            0.0197267048060894,
+            0.012511352077126503,
+            -0.0030878514517098665,
+            -0.0029145763255655766,
+            0.006055743433535099,
+            -3.049322731385473e-05,
+            0.000538985594175756,
+            -0.003632113104686141,
+            -0.008348306640982628,
+            -0.004402965307235718,
+            -0.007779608480632305,
+            0.015461471863090992,
+            -0.0028101669158786535,
+            -0.012742385268211365,
+            0.011427272111177444,
+            -0.003472166834399104,
+            -0.00031905947253108025,
+            -0.016607753932476044,
+            0.0011729393154382706,
+            0.005815824028104544,
+            0.026017924770712852,
+            0.012760157696902752,
+            -0.014519566670060158,
+            -0.025164877995848656,
+            -0.014484022743999958,
+            0.03270012512803078,
+            0.014377391897141933,
+            -0.004289669916033745,
+            0.01694541797041893,
+            0.005913568660616875,
+            0.021059591323137283,
+            0.005020535551011562,
+            -0.003063415177166462,
+            0.012102600187063217,
+            0.028132770210504532,
+            0.00480283098295331,
+            -0.01577247865498066,
+            -0.0042830053716897964,
+            -0.0017694055568426847,
+            -0.014128586277365685,
+            -0.008930332958698273,
+            -0.006495595443993807,
+            -0.02438291907310486,
+            0.01620788872241974,
+            -0.013497686944901943,
+            0.013666519895195961,
+            -0.014555109664797783,
+            -0.004509596154093742,
+            -0.027261951938271523,
+            0.002392529509961605,
+            0.021450571715831757,
+            -0.0035899050999432802,
+            0.012689069844782352,
+            -0.02738635428249836,
+            -0.020846329629421234,
+            0.019371269270777702,
+            -0.019922195002436638,
+            -0.00027823986602015793,
+            -0.0022559086792171,
+            -0.0034988245461136103,
+            -0.0051449378952383995,
+            -0.0010701960418373346,
+            -0.005375971551984549,
+            -0.0002896249061450362,
+            0.021432798355817795,
+            -0.004505153279751539,
+            0.0051671527326107025,
+            0.00011628037464106455,
+            0.015168237499892712,
+            0.014119700528681278,
+            -0.006544467993080616,
+            -0.007726293057203293,
+            -0.019762249663472176,
+            0.0037720659747719765,
+            -0.02249910682439804,
+            0.022321389988064766,
+            0.005011649336665869,
+            0.016403377056121826,
+            -0.007153152488172054,
+            -0.004263012204319239,
+            -0.006388964597135782,
+            -0.01047648023813963,
+            0.018660398200154305,
+            -0.010325419716536999,
+            0.007059850264340639,
+            -0.008277218788862228,
+            0.03403301164507866,
+            0.005269340705126524,
+            -0.013355513103306293,
+            0.02454286627471447,
+            0.005264897830784321,
+            0.011054064147174358,
+            -0.016741042956709862,
+            0.009427943266928196,
+            0.01022767461836338,
+            -0.016829900443553925,
+            0.01296453271061182,
+            -0.02921685017645359,
+            0.006251233164221048,
+            -0.024720583111047745,
+            -0.013071163557469845,
+            -0.017505230382084846,
+            -0.01914023607969284,
+            -0.04243907332420349,
+            -0.001449513016268611,
+            0.030691910535097122,
+            -0.005420401226729155,
+            0.017505230382084846,
+            -0.009738950058817863,
+            0.011373956687748432,
+            0.00821501761674881,
+            -0.005100508686155081,
+            0.020864101126790047,
+            0.02790173701941967,
+            -0.02052643708884716,
+            -0.006357864011079073,
+            0.0006603337242268026,
+            0.008983648382127285,
+            -0.006073514930903912,
+            -0.02921685017645359,
+            0.0050694081000983715,
+            -0.002747965743765235,
+            0.012644641101360321,
+            -0.01702539063990116,
+            0.0062778908759355545,
+            0.0007464158697985113,
+            0.011427272111177444,
+            -0.008339419960975647,
+            -0.014252989552915096,
+            -0.0180472694337368,
+            0.0016783250030130148,
+            0.0069976490922272205,
+            -0.022232530638575554,
+            0.016039054840803146,
+            -0.023903081193566322,
+            0.0014472915790975094,
+            0.015941310673952103,
+            -0.01956675946712494,
+            -0.006993206217885017,
+            -0.005438172724097967,
+            0.036929816007614136,
+            -0.0023414355237036943,
+            -0.002870146883651614,
+            0.011951540596783161,
+            0.0082194609567523,
+            0.010494251735508442,
+            -0.002934569725766778,
+            0.0051449378952383995,
+            0.0026280060410499573,
+            -0.004336320795118809,
+            0.0017427478451281786,
+            0.02761738747358322,
+            0.002912354888394475,
+            0.02125508151948452,
+            -0.0032966702710837126,
+            -0.002921240869909525,
+            0.0019349055364727974,
+            -0.007170923985540867,
+            0.009810036979615688,
+            7.4350020440761e-05,
+            -0.004203032236546278,
+            -0.017327511683106422,
+            0.015159351751208305,
+            -0.015168237499892712,
+            0.010121043771505356,
+            0.020473120734095573,
+            0.01788732409477234,
+            0.006771058309823275,
+            -0.004878360778093338,
+            -0.010547567158937454,
+            0.014928317628800869,
+            0.017078706994652748,
+            -0.00018424366135150194,
+            -0.020579751580953598,
+            0.0010801926255226135,
+            -0.00869485642760992,
+            0.003989770542830229,
+            -0.0116849634796381,
+            0.009516802616417408,
+            0.008805930614471436,
+            -0.003201146610081196,
+            0.015425928868353367,
+            0.008779272437095642,
+            0.014706170186400414,
+            0.01321333833038807,
+            -0.03317996487021446,
+            -0.0030145428609102964,
+            -0.008459379896521568,
+            -0.0020393149461597204,
+            -0.006384521722793579,
+            -0.017558544874191284,
+            -0.0004920568899251521,
+            0.016927646473050117,
+            -0.005993541795760393,
+            -0.006086844019591808,
+            0.01942458562552929,
+            -0.01420855987817049,
+            0.013853123411536217,
+            0.006180145777761936,
+            0.016305632889270782,
+            -0.02017100155353546,
+            0.003740965388715267,
+            -0.0012007077457383275,
+            -0.0037654016632586718,
+            -0.004905018489807844,
+            0.015665847808122635,
+            0.006224575452506542,
+            0.02475612796843052,
+            -0.02651553601026535,
+            0.006060186307877302,
+            -0.007779608480632305,
+            -0.011542788706719875,
+            -0.018011726438999176,
+            0.011009634472429752,
+            0.00021798232046421617,
+            -0.015372613444924355,
+            -0.01315113715827465,
+            0.006851031444966793,
+            0.011525017209351063,
+            -0.0014006405835971236,
+            0.013195566833019257,
+            0.009685634635388851,
+            0.0018615968292579055,
+            0.009738950058817863,
+            0.008605997078120708,
+            -0.02242802083492279,
+            -0.008059514686465263,
+            0.016554437577724457,
+            -0.0029056903440505266,
+            -0.010014412924647331,
+            0.018873659893870354,
+            0.006548910867422819,
+            -0.011880452744662762,
+            0.006891018245369196,
+            0.01701650582253933,
+            0.007988426834344864,
+            0.006011313758790493,
+            -0.007126494776457548,
+            0.003241133177652955,
+            0.002292563207447529,
+            0.0004895577440038323,
+            -0.020615296438336372,
+            -0.008530467748641968,
+            0.008921447210013866,
+            -0.01092077512294054,
+            0.008570454083383083,
+            -0.005282669793814421,
+            -0.023352153599262238,
+            0.01570139080286026,
+            0.0024325160775333643,
+            -0.008268333040177822,
+            -0.008015085011720657,
+            0.008090615272521973,
+            0.004487381316721439,
+            -0.008281662128865719,
+            0.0009963319171220064,
+            -0.0007880685734562576,
+            -0.012360291555523872,
+            0.022676825523376465,
+            0.00800175592303276,
+            0.0010085500543937087,
+            -0.006428950931876898,
+            -0.009116937406361103,
+            -0.0133643988519907,
+            0.031438328325748444,
+            0.013355513103306293,
+            0.023067805916070938,
+            0.01201374176889658,
+            -0.0033699788618832827,
+            -0.0062556760385632515,
+            -0.002807945478707552,
+            0.01559476088732481,
+            0.012111486867070198,
+            -0.00020451462478376925,
+            0.010831916704773903,
+            0.003245576284825802,
+            0.0037298579700291157,
+            0.006033528596162796,
+            0.005238240119069815,
+            -0.003241133177652955,
+            0.008139487355947495,
+            0.0016305632889270782,
+            0.004811716731637716,
+            0.002556918654590845,
+            0.011729392223060131,
+            0.02541368454694748,
+            -0.006544467993080616,
+            -0.008086171932518482,
+            0.01029876247048378,
+            0.012609097175300121,
+            0.004651770461350679,
+            -0.01569250598549843,
+            0.0066022262908518314,
+            -0.011391728185117245,
+            0.011276211589574814,
+            -0.02877255529165268,
+            -0.006877689156681299,
+            0.009952211752533913,
+            -0.021148450672626495,
+            0.03163381665945053,
+            -0.007419729605317116,
+            -0.024596180766820908,
+            0.007766279857605696,
+            -0.00941905751824379,
+            -0.013302197679877281,
+            -0.018429363146424294,
+            0.020402034744620323,
+            0.017505230382084846,
+            0.020419806241989136,
+            -0.012466922402381897,
+            0.01862485334277153,
+            0.004851703066378832,
+            0.022587966173887253,
+            -0.0037542942445725203,
+            -0.014439593069255352,
+            -0.010574225336313248,
+            0.013391056098043919,
+            0.009614547714591026,
+            -0.007495259400457144,
+            0.029465656727552414,
+            0.014368506148457527,
+            0.005838038399815559,
+            -0.0013339963043108582,
+            -0.00737529993057251,
+            0.004143052268773317,
+            -0.002876811195164919,
+            0.004416293930262327,
+            -0.009161366149783134,
+            -0.02111290581524372,
+            -0.0008302765781991184,
+            0.017265310510993004,
+            0.008619326166808605,
+            0.024933844804763794,
+            -0.015106036327779293,
+            0.0081839170306921,
+            0.002747965743765235,
+            -0.01044093631207943,
+            -0.02095296047627926,
+            0.002816831460222602,
+            -0.010201017372310162,
+            0.023192208260297775,
+            0.007468601688742638,
+            -0.007490816526114941,
+            -0.012022627517580986,
+            0.004451837856322527,
+            0.0034943814389407635,
+            0.024454006925225258,
+            -0.005944669246673584,
+            -0.01029876247048378,
+            0.01420855987817049,
+            0.004776173271238804,
+            0.0015372612979263067,
+            0.004038643091917038,
+            0.013506573624908924,
+            0.00668219942599535,
+            -0.0011362849036231637,
+            -0.001868261257186532,
+            0.0037298579700291157,
+            0.007544131949543953,
+            -0.0010435383301228285,
+            0.0011285097571089864,
+            0.001358432462438941,
+            0.004580683074891567,
+            0.012475808151066303,
+            -0.004098623059689999,
+            0.0027768448926508427,
+            -0.0041230591014027596,
+            -0.00912582315504551,
+            0.0014894995838403702,
+            0.0005156600964255631,
+            -0.012626868672668934,
+            -0.0009224678506143391,
+            -0.00912582315504551,
+            -0.0020126572344452143,
+            0.01417301595211029,
+            0.03367757424712181,
+            -0.010538681410253048,
+            0.022321389988064766,
+            -0.011720506474375725,
+            0.02146834321320057,
+            0.013373284600675106,
+            0.005469273775815964,
+            -0.02228584513068199,
+            -0.015088263899087906,
+            0.01405749935656786,
+            -0.017940638586878777,
+            0.004209696780890226,
+            -0.0052337972447276115,
+            -0.03024761565029621,
+            0.013053392060101032,
+            -0.03170490264892578,
+            -0.0030478648841381073,
+            0.005287112668156624,
+            -0.014332962222397327,
+            -0.009259111247956753,
+            -0.0031789320055395365,
+            -0.002585797803476453,
+            -0.0027368583250790834,
+            0.0064200651831924915,
+            -0.028026139363646507,
+            -0.0133999427780509,
+            0.005540360696613789,
+            0.009996641427278519,
+            -0.01620788872241974,
+            0.010538681410253048,
+            0.004589569289237261,
+            -0.0012617983156815171,
+            0.00462066987529397,
+            -0.018153900280594826,
+            -0.007033192552626133,
+            -0.02512933500111103,
+            -0.003809831105172634,
+            -0.00803729984909296,
+            0.015417042188346386,
+            -0.00045679096365347505,
+            -0.005953555461019278,
+            -0.0009407950565218925,
+            -0.011187352240085602,
+            0.0030834085773676634,
+            0.01055645290762186,
+            0.002639113226905465,
+            -0.008170587942004204,
+            0.01723865233361721,
+            0.02278345637023449,
+            -0.01300007663667202,
+            -0.025662489235401154,
+            0.01921132393181324,
+            -0.01577247865498066,
+            0.021574974060058594,
+            0.0005645325290970504,
+            -0.001508382149040699,
+            -0.0150793781504035,
+            0.015150465071201324,
+            0.0010563117684796453,
+            0.01216480229049921,
+            -0.014795029535889626,
+            0.02825717255473137,
+            -0.0010374292032793164,
+            0.003809831105172634,
+            -0.0073664141818881035,
+            -8.213386536226608e-06,
+            0.00888146087527275,
+            0.0019904423970729113,
+            -0.010369849391281605,
+            0.020508665591478348,
+            -0.012182573787868023,
+            0.0045473612844944,
+            -0.008446050807833672,
+            0.00020104357099626213,
+            -0.008974762633442879,
+            0.0034543948713690042,
+            -0.0017838451312854886,
+            0.012573553249239922,
+            0.012546896003186703,
+            0.015274868346750736,
+            -0.010023298673331738,
+            -0.008948104456067085,
+            0.013782036490738392,
+            -0.02498716115951538,
+            0.0013828687369823456,
+            0.0006231239531189203,
+            0.011951540596783161,
+            -0.02118399366736412,
+            0.006237904075533152,
+            0.026391133666038513,
+            0.005318213254213333,
+            0.009836695156991482,
+            -0.010716399177908897,
+            -0.018002839758992195,
+            -0.007815152406692505,
+            0.007917339913547039,
+            -0.014457364566624165,
+            -0.02059752494096756,
+            -0.013702062889933586,
+            -0.00020479230443015695,
+            -0.0021315061021596193,
+            0.004918347578495741,
+            -0.013630975969135761,
+            -0.002365871798247099,
+            -0.01664329692721367,
+            -0.0024502878077328205,
+            -0.014039727859199047,
+            -0.007042078301310539,
+            0.008090615272521973,
+            0.02855929359793663,
+            0.008077286183834076,
+            0.02402748353779316,
+            0.01237806398421526,
+            0.020917417481541634,
+            0.017834007740020752,
+            -0.004567354451864958,
+            -0.005695864092558622,
+            -0.0002508879406377673,
+            0.018660398200154305,
+            -0.03321550786495209,
+            0.01438627764582634,
+            -0.0077174073085188866,
+            -0.004138609394431114,
+            -0.0054914881475269794,
+            -0.0019304625457152724,
+            0.010796372778713703,
+            0.003976441919803619,
+            -0.004167488776147366,
+            0.010929660871624947,
+            -0.024596180766820908,
+            -0.007135380525141954,
+            -0.00735752796754241,
+            -0.01723865233361721,
+            -0.0013362177414819598,
+            0.02409856952726841,
+            -0.005424844101071358,
+            0.002994549460709095,
+            0.006984320003539324,
+            0.008437165059149265,
+            0.016492236405611038,
+            0.015808021649718285,
+            -0.0016438921447843313,
+            -0.0057669514790177345,
+            0.005398186389356852,
+            -0.004887246992439032,
+            -0.03081631287932396,
+            -0.006899903994053602,
+            0.008379407227039337,
+            0.011729392223060131,
+            -0.025289282202720642,
+            -0.011116265319287777,
+            0.0017338618636131287,
+            0.01540815643966198,
+            0.02081078663468361,
+            -0.0039297910407185555,
+            -0.016083484515547752,
+            -0.004147495608776808,
+            -0.020579751580953598,
+            0.021806007251143456,
+            -0.012804586440324783,
+            0.0005670317332260311,
+            -0.009232454001903534,
+            0.00939240027219057,
+            -0.012564667500555515,
+            0.03317996487021446,
+            -0.014590653590857983,
+            0.002401415491476655,
+            -0.0010685299057513475,
+            0.011098493821918964,
+            0.00476728705689311,
+            0.024134114384651184,
+            -0.01722976751625538,
+            -0.003796502249315381,
+            0.028683695942163467,
+            -0.033624257892370224,
+            0.016696613281965256,
+            0.008743729442358017,
+            0.01084080245345831,
+            -0.009303540922701359,
+            -0.008743729442358017,
+            0.027955051511526108,
+            0.020793013274669647,
+            0.006535581778734922,
+            -0.001412858604453504,
+            -0.019264638423919678,
+            0.023689819499850273,
+            -0.0011829358991235495,
+            0.002683542901650071,
+            -0.002794616622850299,
+            0.013453258201479912,
+            -0.034886058419942856,
+            -0.01657220907509327,
+            0.005562575533986092,
+            -0.008663755841553211,
+            -0.01892697438597679,
+            0.0005864696577191353,
+            -0.0116494195535779,
+            -0.02132616750895977,
+            -0.0032655694521963596,
+            0.01965561881661415,
+            -0.002612455515190959,
+            0.003072301158681512,
+            0.006686642300337553,
+            0.004411851055920124,
+            -0.008605997078120708,
+            -0.024525092914700508,
+            -0.01099186297506094,
+            0.003709864802658558,
+            0.007348642218858004,
+            -0.013017848134040833,
+            -0.012982305139303207,
+            -0.01077860128134489,
+            -0.0030100997537374496,
+            0.013471029698848724,
+            0.002305892063304782,
+            0.02168160490691662,
+            -0.004474052228033543,
+            0.01435073371976614,
+            0.013906438834965229,
+            -0.006851031444966793,
+            -0.008685970678925514,
+            0.013248882256448269,
+            -0.0027457440737634897,
+            -0.007504145614802837,
+            -0.007384185679256916,
+            -0.0016205665888264775,
+            -0.006584454327821732,
+            0.002288120100274682,
+            0.01956675946712494,
+            -0.01970893330872059,
+            0.004820602480322123,
+            0.0017249759985134006,
+            -0.01212925836443901,
+            0.005629220046103001,
+            0.008308319374918938,
+            -0.0011884896084666252,
+            0.011196237988770008,
+            0.003774287411943078,
+            -0.002202593255788088,
+            0.00226479466073215,
+            -0.013071163557469845,
+            0.006575568579137325,
+            0.015417042188346386,
+            0.014190787449479103,
+            0.01077860128134489,
+            0.006366749759763479,
+            -0.034299585968256,
+            -0.006415622308850288,
+            -0.003118952037766576,
+            -0.02374313399195671,
+            0.0014106371672824025,
+            0.0026280060410499573,
+            0.020259859040379524,
+            -0.003076744033023715,
+            0.014937203377485275,
+            -0.005882468074560165,
+            0.0006458940915763378,
+            0.016821015626192093,
+            0.009907782077789307,
+            -0.03573910519480705,
+            -0.010663083754479885,
+            -0.01523043867200613,
+            -0.007979541085660458,
+            -0.015177123248577118,
+            -0.014377391897141933,
+            0.008077286183834076,
+            0.017291968688368797,
+            -0.013648747466504574,
+            -0.012200345285236835,
+            0.011711620725691319,
+            0.013888667337596416,
+            0.007370857056230307,
+            0.006144602317363024,
+            0.016314519569277763,
+            -0.0018127242801710963,
+            0.0006347867310978472,
+            0.00026338372845202684,
+            0.018873659893870354,
+            -0.013026734814047813,
+            -0.0023392140865325928,
+            -0.011418386362493038,
+            0.011676076799631119,
+            0.020864101126790047,
+            -0.01076971460133791,
+            0.014448478817939758,
+            -0.007059850264340639,
+            -0.006779944524168968,
+            -0.047379638999700546,
+            -0.01321333833038807,
+            -0.009436829946935177,
+            0.028132770210504532,
+            -0.004807273857295513,
+            -0.005833595525473356,
+            0.0055803474970161915,
+            0.019602302461862564,
+            0.0063001057133078575,
+            0.008166145533323288,
+            0.01818055845797062,
+            -0.00839273538440466,
+            -0.012804586440324783,
+            0.004160824231803417,
+            0.007859582081437111,
+            -0.00927688367664814,
+            0.0027546300552785397,
+            0.0099344402551651,
+            0.032753441482782364,
+            -0.0057669514790177345,
+            0.0075085884891450405,
+            -0.009108050726354122,
+            0.0022303618025034666,
+            -0.016758814454078674,
+            0.001700539723969996,
+            -0.005473716650158167,
+            -0.006815487984567881,
+            0.014439593069255352,
+            -0.021788235753774643,
+            -0.0014795028837397695,
+            -0.0028968045953661203,
+            0.008406064473092556,
+            -0.01318668108433485,
+            0.011045178398489952,
+            -0.0041230591014027596,
+            -0.004271897953003645,
+            0.014128586277365685,
+            -0.011915996670722961,
+            0.027581844478845596,
+            -0.002074858406558633,
+            0.009969983249902725,
+            -0.007970655336976051,
+            -0.012866788543760777,
+            -0.017807351425290108,
+            0.0025369254872202873,
+            -0.0021348383743315935,
+            -0.008534910157322884,
+            0.008850360289216042,
+            -0.022463563829660416,
+            0.0036543277092278004,
+            -0.004873917903751135,
+            0.0015339291421696544,
+            -0.02592906542122364,
+            0.00015036614786367863,
+            -0.002412522677332163,
+            0.011080721393227577,
+            -0.008810373023152351,
+            -0.013160022906959057,
+            0.01176493614912033,
+            -0.010734171606600285,
+            -0.013888667337596416,
+            -0.01318668108433485,
+            0.008410507813096046,
+            0.004038643091917038,
+            -0.002434737514704466,
+            0.014457364566624165,
+            -0.011009634472429752,
+            0.0008480484248138964,
+            -0.006184588652104139,
+            0.00924133975058794,
+            -0.00047900574281811714,
+            -0.016989847645163536,
+            -4.682454527937807e-05,
+            -0.0002667159424163401,
+            0.012440265156328678,
+            0.012982305139303207,
+            0.018447136506438255,
+            -0.008441608399152756,
+            0.0026168986223638058,
+            0.0034766097087413073,
+            -0.0037809519562870264,
+            0.011187352240085602,
+            0.005775837227702141,
+            -0.013888667337596416,
+            0.0013650968903675675,
+            -0.013684291392564774,
+            0.00854379590600729,
+            -0.00891256146132946,
+            0.010209903120994568,
+            0.009108050726354122,
+            -0.0024769455194473267,
+            -0.010343191213905811,
+            0.011729392223060131,
+            0.01657220907509327,
+            -0.012333634309470654,
+            0.00032655693939886987,
+            -0.007228682283312082,
+            0.025431456044316292,
+            0.016749927774071693,
+            0.010583111084997654,
+            -0.013355513103306293,
+            0.013826466165482998,
+            -0.00025310940691269934,
+            0.0037254150956869125,
+            0.00957011803984642,
+            -0.026408905163407326,
+            -0.02491607330739498,
+            0.003736522514373064,
+            -0.0069310045801103115,
+            -0.0060468572191894054,
+            0.023316610604524612,
+            -0.012138144113123417,
+            0.006184588652104139,
+            -0.011569445952773094,
+            0.028168313205242157,
+            0.01621677353978157,
+            -0.0009307984146289527,
+            -0.015834679827094078,
+            0.0002776844776235521,
+            0.01011215802282095,
+            -0.0013417714508250356,
+            -0.007766279857605696,
+            -0.0022681269329041243,
+            -0.012538010254502296,
+            0.004398522432893515,
+            -0.003783173393458128,
+            0.004776173271238804,
+            0.004402965307235718,
+            -0.012697956524789333,
+            -0.011311755515635014,
+            0.028221629559993744,
+            0.0008724846411496401,
+            0.012973419390618801,
+            -0.004971663001924753,
+            0.0017494122730568051,
+            0.005957998335361481,
+            0.02233916148543358,
+            -0.024276288226246834,
+            0.01905137673020363,
+            0.006739957723766565,
+            0.017771806567907333,
+            0.008903675712645054,
+            -0.00839717872440815,
+            0.01249358057975769,
+            -0.006157930940389633,
+            -0.007415286265313625,
+            -0.010280990041792393,
+            -0.0014750600093975663,
+            -0.0016050162957981229,
+            -0.0076285479590296745,
+            0.009579003788530827,
+            -0.0011290650581941009,
+            -0.004905018489807844,
+            0.01420855987817049,
+            -0.022890087217092514,
+            0.014723941683769226,
+            -0.0013239996042102575,
+            0.01124955341219902,
+            -0.0021781569812446833,
+            0.011125151067972183,
+            -0.02278345637023449,
+            0.0038675894029438496,
+            -0.01797618344426155,
+            -0.0012118150480091572,
+            -0.0040941801853477955,
+            -0.001176271471194923,
+            -0.0030323145911097527,
+            0.007979541085660458,
+            -0.034139640629291534,
+            -0.010067728348076344,
+            0.015488130040466785,
+            0.013933097012341022,
+            0.0026768783573061228,
+            0.013026734814047813,
+            0.006246790289878845,
+            0.0012362513225525618,
+            -0.00615348806604743,
+            -0.009294655174016953,
+            -0.020259859040379524,
+            -0.02219698764383793,
+            -0.013053392060101032,
+            -0.0022725698072463274,
+            -0.0008230567909777164,
+            -0.005242682993412018,
+            0.008286104537546635,
+            -0.018287189304828644,
+            -0.0007325316546484828,
+            -0.0005164931644685566,
+            0.0004479050694499165,
+            0.001358432462438941,
+            0.0018538215663284063,
+            0.016341175884008408,
+            0.008765943348407745,
+            -0.0042363544926047325,
+            -0.007370857056230307,
+            0.025182651355862617,
+            -0.0030345360282808542,
+            0.012147029861807823,
+            0.020028825849294662,
+            0.0025080463383346796,
+            -0.014688398689031601,
+            0.013835351914167404,
+            -0.008290547877550125,
+            0.018642624840140343,
+            -0.012333634309470654,
+            -0.025680260732769966,
+            -0.014857230708003044,
+            0.007610776461660862,
+            -0.011871566995978355,
+            -0.009596775285899639,
+            0.01365763321518898,
+            -0.002821274334564805,
+            0.0001021878924802877,
+            0.007135380525141954,
+            -0.005762508604675531,
+            0.0025369254872202873,
+            -0.03497491776943207,
+            -0.000194934502360411,
+            -0.03001658245921135,
+            -0.030567508190870285,
+            0.014670626260340214,
+            0.006855474319308996,
+            0.021077362820506096,
+            -0.006948776543140411,
+            0.013977525755763054,
+            -0.009099164977669716,
+            0.026053469628095627,
+            0.014946090057492256,
+            -0.0006936558638699353,
+            0.0031789320055395365,
+            -0.007250897120684385,
+            -0.006371192634105682,
+            0.027155321091413498,
+            -0.008081729523837566,
+            -0.0059313406236469746,
+            0.02475612796843052,
+            0.026657709851861,
+            0.01738082803785801,
+            -0.018944745883345604,
+            0.019460128620266914,
+            0.0013017848832532763,
+            0.01789620891213417,
+            -0.016199002042412758,
+            -0.00032850075513124466,
+            -0.015665847808122635,
+            -0.008161702193319798,
+            0.05530586466193199,
+            0.01723865233361721,
+            -0.0007297547999769449,
+            -0.0202243160456419,
+            0.015017176978290081,
+            -0.0029789991676807404,
+            -0.002028207527473569,
+            0.00981892365962267,
+            0.0302298441529274,
+            -0.00920579582452774,
+            -0.013799807988107204,
+            0.013293311931192875,
+            -0.005247125867754221,
+            0.026746569201350212,
+            -0.002260351786389947,
+            0.0020904086995869875,
+            -0.015221552923321724,
+            -0.013133365660905838,
+            0.0013395498972386122,
+            0.009756721556186676,
+            -0.007939554750919342,
+            -0.0011018520453944802,
+            -0.006944333668798208,
+            0.012769043445587158,
+            0.006815487984567881,
+            -0.020277632400393486,
+            -0.003112287726253271,
+            -0.030922943726181984,
+            -0.00480283098295331,
+            -0.035525843501091,
+            -0.005944669246673584,
+            -0.005269340705126524,
+            -0.014626197516918182,
+            -0.032611265778541565,
+            -0.02073969878256321,
+            -0.009996641427278519,
+            0.004660656210035086,
+            0.002425851533189416,
+            0.010041071102023125,
+            0.002474724082276225,
+            0.0005764730158261955,
+            0.0010274326195940375,
+            0.0041230591014027596,
+            0.01303562056273222,
+            0.001622788142412901,
+            0.011729392223060131,
+            0.02233916148543358,
+            -0.018198329955339432,
+            -0.033410996198654175,
+            -0.014484022743999958,
+            0.0050249784253537655,
+            0.005686978343874216,
+            -0.0018749256851151586,
+            0.008601554669439793,
+            0.0015561438631266356,
+            0.011231781914830208,
+            -0.014324076473712921,
+            -0.005020535551011562,
+            -0.009774493984878063,
+            -0.0011773821897804737,
+            0.0026191200595349073,
+            0.003896468784660101,
+            0.0026946503203362226,
+            0.0029412340372800827,
+            -0.011711620725691319,
+            0.01014770194888115,
+            -0.005380414426326752,
+            -0.0009291322785429657,
+            -0.0009024745668284595,
+            -0.012538010254502296,
+            0.004927233327180147,
+            -0.007935111410915852,
+            -0.00753968907520175,
+            -0.008534910157322884,
+            0.0034521734341979027,
+            -0.01600351184606552,
+            -0.009036963805556297,
+            0.01716756634414196,
+            -0.006424508057534695,
+            0.007757393643260002,
+            0.012831244617700577,
+            -0.0013650968903675675,
+            -0.004562911577522755,
+            0.019229095429182053,
+            -0.012742385268211365,
+            -0.01417301595211029,
+            -0.024454006925225258,
+            -0.006446722894906998,
+            0.0005456500221043825,
+            0.015452586114406586,
+            0.0042830053716897964,
+            -0.010903003625571728,
+            -0.0034810525830835104,
+            -0.002579133491963148,
+            0.003554361406713724,
+            -0.012324748560786247,
+            0.005966884084045887,
+            -0.02388530783355236,
+            0.015959082171320915,
+            -0.006677756551653147,
+            -0.009303540922701359,
+            0.018713712692260742,
+            0.04535365104675293,
+            -0.007606333587318659,
+            -0.0177629217505455,
+            0.009014748968183994,
+            0.022836772724986076,
+            0.022676825523376465,
+            -0.007086507976055145,
+            0.013053392060101032,
+            -0.02001105435192585,
+            -0.011880452744662762,
+            -0.005082736723124981,
+            -0.002314777811989188,
+            0.022516880184412003,
+            -0.03104734607040882,
+            -0.019246866926550865,
+            -0.007784051354974508,
+            -0.03241577744483948,
+            -0.006948776543140411,
+            0.028470434248447418,
+            0.004611784126609564,
+            -0.005620333831757307,
+            -0.004143052268773317,
+            -0.002716864924877882,
+            -0.01753188669681549,
+            0.020259859040379524,
+            0.08686859160661697,
+            -0.014635083265602589,
+            -0.008294991217553616,
+            -0.004131945315748453,
+            0.00635342113673687,
+            0.005784722976386547,
+            -0.005056079011410475,
+            0.06181034445762634,
+            -0.023121120408177376,
+            0.00033433211501687765,
+            0.003609898267313838,
+            0.015816908329725266,
+            -0.007170923985540867,
+            -0.0002644944761414081,
+            0.0012107043294236064,
+            -0.005953555461019278,
+            -0.010129929520189762,
+            -0.017105363309383392,
+            -0.0045295893214643,
+            -0.0017949525499716401,
+            -0.00701097771525383,
+            -0.01620788872241974,
+            0.019335726276040077,
+            -0.023849764838814735,
+            -0.006615554913878441,
+            -0.0023347712121903896,
+            -0.006513367407023907,
+            0.009188024327158928,
+            -0.019833335652947426,
+            0.002570247510448098,
+            -0.0011773821897804737,
+            0.005380414426326752,
+            -0.005158266983926296,
+            -0.005575904622673988,
+            -0.0028234957717359066,
+            0.0009313537739217281,
+            -0.0098988963291049,
+            0.01826941780745983,
+            0.013790922239422798,
+            0.009152480401098728,
+            0.010058842599391937,
+            -0.0055803474970161915,
+            0.023121120408177376,
+            -0.005784722976386547,
+            0.0057891663163900375,
+            -0.0067888302728533745,
+            0.0014161908766254783,
+            -0.004789501894265413,
+            0.01839382015168667,
+            -0.020633067935705185,
+            -0.013071163557469845,
+            -0.014875002205371857,
+            0.035170406103134155,
+            -0.02097073197364807,
+            0.014972747303545475,
+            0.0010307647753506899,
+            0.011356184259057045,
+            0.010485365986824036,
+            0.0013051170390099287,
+            0.01702539063990116,
+            -0.012093714438378811,
+            0.016776585951447487,
+            0.002699093194678426,
+            0.015363726764917374,
+            -0.008255003951489925,
+            0.012573553249239922,
+            0.00010274325904902071,
+            0.003809831105172634,
+            -0.008637098595499992,
+            0.0044696093536913395,
+            0.011676076799631119,
+            -0.005780280102044344,
+            0.03541921079158783,
+            -0.0013728720368817449,
+            -0.02226807363331318,
+            0.016741042956709862,
+            -0.015106036327779293,
+            -0.02082855813205242,
+            0.0011740500340238214,
+            -0.014652854762971401,
+            -0.0030278717167675495,
+            0.012315861880779266,
+            -0.028381574898958206,
+            -0.020988503471016884,
+            0.005247125867754221,
+            0.001098519773222506,
+            0.0016572210006415844,
+            0.0006486709462478757,
+            -0.014439593069255352,
+            -0.0018138349987566471,
+            -0.0030656366143375635,
+            0.007957326248288155,
+            -0.014777257107198238,
+            0.006437837146222591,
+            -0.026266731321811676,
+            -0.0031389454379677773,
+            -0.003132280893623829,
+            0.023689819499850273,
+            -0.002907912014052272,
+            0.010432050563395023,
+            -0.005993541795760393,
+            -0.0025502543430775404,
+            -0.016252316534519196,
+            -0.0033522071316838264,
+            -0.003258905140683055,
+            0.018518222495913506,
+            0.026746569201350212,
+            -0.0016761035658419132,
+            -0.0025235966313630342,
+            -6.3693878473714e-05,
+            0.015461471863090992,
+            -0.00858378317207098,
+            -0.01022767461836338,
+            0.01666106842458248,
+            0.014404049143195152,
+            -0.016554437577724457,
+            0.011862681247293949,
+            -0.0022125900723040104,
+            0.011658305302262306,
+            -0.012253660708665848,
+            0.016536666080355644,
+            0.020490894094109535,
+            -0.005282669793814421,
+            0.019193550571799278,
+            -0.01095631904900074,
+            0.015390384942293167,
+            0.018731484189629555,
+            0.006948776543140411,
+            0.014155244454741478,
+            0.012662412598729134,
+            -0.01180048007518053,
+            0.0005836928030475974,
+            -0.005758065264672041,
+            -0.005322656128555536,
+            0.005127166397869587,
+            -0.002612455515190959,
+            -0.019406812265515327,
+            0.0036121197044849396,
+            0.0012462479062378407,
+            0.008508252911269665,
+            0.0034899385645985603,
+            -0.00617126002907753,
+            -0.0008730400004424155,
+            0.013462143950164318,
+            -0.02095296047627926,
+            -0.006077957805246115,
+            0.023938624188303947,
+            -0.006015756633132696,
+            -0.005673649255186319,
+            0.0011407278943806887,
+            0.012093714438378811,
+            -0.01092077512294054,
+            -0.01651889458298683,
+            -0.020508665591478348,
+            0.02841711975634098,
+            0.004509596154093742,
+            0.00887701753526926,
+            0.006762172561138868,
+            -0.008610440418124199,
+            -0.002639113226905465,
+            -0.015950197353959084,
+            -0.001235140603967011,
+            0.018233874812722206,
+            0.01723865233361721,
+            0.011027405969798565,
+            -0.019780021160840988,
+            0.008317206054925919,
+            -0.008486038073897362,
+            3.512708644848317e-05,
+            0.014004183933138847,
+            0.01622566021978855,
+            0.0063223205506801605,
+            -0.0027079791761934757,
+            0.016989847645163536,
+            0.001412858604453504,
+            0.029678918421268463,
+            -0.0006458940915763378,
+            -0.01767406240105629,
+            0.0003993102873209864,
+            0.001091300044208765,
+            -0.01620788872241974,
+            -0.003932012245059013,
+            0.0024769455194473267,
+            0.038813624531030655,
+            -0.008139487355947495,
+            0.007148709148168564,
+            -0.003058972302824259,
+            -0.012422492727637291,
+            0.0025724689476191998,
+            -0.0015883552841842175,
+            -0.002055975841358304,
+            0.02177046425640583,
+            -0.017567431554198265,
+            0.012004856020212173,
+            -0.012591325677931309,
+            0.0056780921295285225,
+            0.0012740164529532194,
+            0.007761836517602205,
+            -0.011356184259057045,
+            -0.03067413903772831,
+            -0.013790922239422798,
+            0.02635558880865574,
+            0.018642624840140343,
+            -0.007135380525141954,
+            0.018145015463232994,
+            -0.014821686781942844,
+            -0.00941905751824379,
+            0.008303876966238022,
+            -0.010982976295053959,
+            0.0009352413471788168,
+            0.007233125623315573
+          ],
+          "keyphrases": [
+            "1 Billion Word Language Modeling Benchmark",
+            "Long Short-Term Memory (LSTM) layers",
+            "Mixture of Experts (MoE) models",
+            "computational efficiency",
+            "Machine Translation"
+          ],
+          "title": "5 Experiments 5.1 1 Billion Word Language Modeling Benchmark"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "405ee5d3-61cd-4c9a-8965-e095643d2824",
+      "type": "next",
+      "source": {
+        "id": "87b82083-05f3-4d69-9ea0-97d4f551aa37",
+        "properties": {
+          "page_content": "5 Experiments 5.1 1 Billion Word Language Modeling Benchmark\n\nDataset: This dataset, introduced by (Chelba et al., 2013) consists of shuffled unique sentences from news articles, totaling approximately 829 million words, with a vocabulary of 793,471 words.\n\nPrevious State-of-the-Art: The best previously published results (Jozefowicz et al., 2016) use models consisting of one or more stacked Long Short-Term Memory (LSTM) layers (Hochreiter & Schmidhuber, 1997; Gers et al., 2000). The number of parameters in the LSTM layers of these models vary from 2 million to 151 million. Quality increases greatly with parameter count, as do computational costs. Results for these models form the top line of Figure 2-right. MoE Models: Our models consist of two stacked LSTM layers with a MoE layer between them (see Figure 1). We vary the sizes of the layers and the number of experts. For full details on model architecture, training regimen, additional baselines and results, see Appendix C.\n\nLow Computation, Varied Capacity: To investigate the effects of adding capacity, we trained\n\na series of MoE models all with roughly equal computational costs: about 8 million multiply-andadds per training example per timestep in the forwards pass, excluding the softmax layer. We call this metric (ops/timestep). We trained models with flat MoEs containing 4, 32, and 256 experts, and models with hierarchical MoEs containing 256, 1024, and 4096 experts. Each expert had about 1 million parameters. For all the MoE layers, 4 experts were active per input.\n\nThe results of these models are shown in Figure 2-left. The model with 4 always-active experts performed (unsurprisingly) similarly to the computationally-matched baseline models, while the largest of the models (4096 experts) achieved an impressive 24% lower perplexity on the test set. Figure 2: Model comparison on 1-Billion-Word Language-Modeling Benchmark. On the left, we plot test perplexity as a function of model capacity for models with similar computational budgets of approximately 8-million-ops-per-timestep. On the right, we plot test perplexity as a function of computational budget. The top line represents the LSTM models from (Jozefowicz et al., 2016).\n\nThe bottom line represents 4-billion parameter MoE models with different computational budgets.\n\nVaried Computation, High Capacity: In addition to the largest model from the previous section, we trained two more MoE models with similarly high capacity (4 billion parameters), but higher computation budgets. These models had larger LSTMs, and fewer but larger and experts. Details\n\nTest Test #Parameters ops/timestep Training TFLOPS Perplexity Perplexity excluding embedding Time /GPU 10 epochs 100 epochs and softmax layers 10 epochs Best Published Results 34.7 30.6 151 million 151 million 59 hours, 32 k40s 1.09 Low-Budget MoE Model 34.1 4303 million 8.9 million 15 hours, 16 k40s 0.74 Medium-Budget MoE Model 31.3 4313 million 33.8 million 17 hours, 32 k40s 1.22 High-Budget MoE Model 28.0 4371 million 142.7 million 47 hours, 32 k40s 1.56\n\nTable 1: Summary of high-capacity MoE-augmented models with varying computational budgets, vs. best previously published results (Jozefowicz et al., 2016). Details in Appendix C. can be found in Appendix C.2. Results of these three models form the bottom line of Figure 2-right.\n\nTable 1 compares the results of these models to the best previously-published result on this dataset .\n\nEven the fastest of these models beats the best published result (when controlling for the number of training epochs), despite requiring only 6% of the computation.\n\nComputational Efficiency: We trained our models using TensorFlow (Abadi et al., 2016) on clusters containing 16-32 Tesla K40 GPUs. For each of our models, we determine computational efficiency in TFLOPS/GPU by dividing the number of floating point operations required to process one training batch by the observed step time and the number of GPUs in the cluster. The operation counts used here are higher than the ones we report in our ops/timestep numbers in that we include the backwards pass, we include the importance-sampling-based training of the softmax layer, and we count a multiply-and-add as two separate operations. For all of our MoE models, the floating point operations involved in the experts represent between 37% and 46% of the total.\n\nFor our baseline models wtih no MoE, observed computational efficiency ranged from 1.07-1.29 TFLOPS/GPU. For our low-computation MoE models, computation efficiency ranged from 0.740.90 TFLOPS/GPU, except for the 4-expert model which did not make full use of the available parallelism. Our highest-computation MoE model was more efficient at 1.56 TFLOPS/GPU, likely due to the larger matrices. These numbers represent a significant fraction of the theoretical maximum of 4.29 TFLOPS/GPU claimed by NVIDIA. Detailed results are in Appendix C, Table 7.\n\n5.2 100 BILLION WORD GOOGLE NEWS CORPUS\n\nOn the 1-billion-word corpus, adding additional capacity seems to produce diminishing returns as the number of parameters in the MoE layer exceeds 1 billion, as can be seen in Figure 2-left. We hypothesized that for a larger training set, even higher capacities would produce significant quality improvements. We constructed a similar training set consisting of shuffled unique sentences from Google's internal news corpus, totalling roughly 100 billion words. Similarly to the previous section, we tested a series of models with similar computational costs of about 8 million ops/timestep. In addition to a baseline LSTM model, we trained models augmented with MoE layers containing 32, 256, 1024, 4096, 16384, 65536, and 131072 experts. This corresponds to up to 137 billion parameters in the MoE layer. Details on architecture, training, and results are given in Appendix D.\n\nResults: Figure 3 shows test perplexity as a function of capacity after training on 10 billion words (top line) and 100 billion words (bottom line). When training over the full 100 billion words, test perplexity improves significantly up to 65536 experts (68 billion parameters), dropping 39% lower than the computationally matched baseline, but degrades at 131072 experts, possibly a result of too much sparsity. The widening gap between the two lines demonstrates (unsurprisingly) that increased model capacity helps more on larger training sets. Even at 65536 experts (99.994% layer sparsity), computational efficiency for the model stays at a respectable 0.72 TFLOPS/GPU.\n\n5.3 Machine Translation (Single Language Pair)\n\nModel Architecture: Our model was a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decreased the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We inserted MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). Each MoE layer contained up to 2048 experts each with about two million parameters, adding a total of about 8 billion parameters to the models. Further details on model architecture, testing procedure and results can be found in Appendix E.\n\nDatasets: We benchmarked our method on the WMT'14 En→Fr and En→De corpora, whose training sets have 36M sentence pairs and 5M sentence pairs, respectively. The experimental protocols were also similar to those in (Wu et al., 2016): newstest2014 was used as the test set to compare against previous work (Luong et al., 2015a; Zhou et al., 2016; Wu et al., 2016), while the combination of newstest2012 and newstest2013 was used as the development set. We also tested the same model on a Google's Production English to French data.\n\nTable 2: Results on WMT'14 En→ Fr newstest2014 (bold values represent best results). Model Test Test ops/timenstep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.69 40.35 85M 8.7B 3 days/64 k40s MoE with 2048 Experts (longer training) 2.63 40.56 85M 8.7B 6 days/64 k40s GNMT (Wu et al., 2016) 2.79 39.22 214M 278M 6 days/96 k80s GNMT+RL (Wu et al., 2016) 2.96 39.92 214M 278M 6 days/96 k80s PBMT (Durrani et al., 2014) 37.0 LSTM (6-layer) (Luong et al., 2015b) 31.5 LSTM (6-layer+PosUnk) (Luong et al., 2015b) 33.1 DeepAtt (Zhou et al., 2016) 37.7 DeepAtt+PosUnk (Zhou et al., 2016) 39.2\n\nTable 3: Results on WMT'14 En → De newstest2014 (bold values represent best results). Model Test Test ops/timestep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 4.64 26.03 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 5.25 24.91 214M 278M 1 day/96 k80s GNMT +RL (Wu et al., 2016) 8.08 24.66 214M 278M 1 day/96 k80s PBMT (Durrani et al., 2014) 20.7 DeepAtt (Zhou et al., 2016) 20.6\n\nTable 4: Results on the Google Production En→ Fr dataset (bold values represent best results). Model Eval Eval Test Test ops/timestep Total Training Perplexity BLEU Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.60 37.27 2.69 36.57 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 2.78 35.80 2.87 35.56 214M 278M 6 days/96 k80s\n\nResults: Tables 2, 3, and 4 show the results of our largest models, compared with published results. Our approach achieved BLEU scores of 40.56 and 26.03 on the WMT'14 En→Fr and En→De benchmarks. As our models did not use RL refinement, these results constitute significant gains of 1.34 and 1.12 BLEU score on top of the strong baselines in (Wu et al., 2016). The perplexity scores are also better.2 On the Google Production dataset, our model achieved 1.01 higher test BLEU score even after training for only one sixth of the time.\n\n5.4 Multilingual Machine Translation\n\nDataset: (Johnson et al., 2016) train a single GNMT (Wu et al., 2016) model on a very large combined dataset of twelve language pairs. Results are somewhat worse than those for 12 separately trained single-pair GNMT models. This is not surprising, given that the twelve models have 12 times the capacity and twelve times the aggregate training of the one model. We repeat this experiment with a single MoE-augmented model. See Appendix E for details on model architecture.\n\nWe train our model on the same dataset as (Johnson et al., 2016) and process the same number of training examples (about 3 billion sentence pairs). Our training time was shorter due to the lower computational budget of our model.\n\nResults: Results for the single-pair GNMT models, the multilingual GNMT model and the multilingual MoE model are given in Table 5. The MoE model achieves 19% lower perplexity on the dev set than the multilingual GNMT model. On BLEU score, the MoE model significantly beats the multilingual GNMT model on 11 of the 12 language pairs (by as much as 5.84 points), and even beats the monolingual GNMT models on 8 of 12 language pairs. The poor performance on English → Korean seems to be a result of severe overtraining, as for the rarer language pairs a small number of real examples were highly oversampled in the training corpus.\n\nTable 5: Multilingual Machine Translation (bold values represent best results). GNMT-Mono GNMT-Multi MoE-Multi MoE-Multi vs. GNMT-Multi Parameters 278M / model 278M 8.7B ops/timestep 212M 212M 102M training time, hardware various 21 days, 96 k20s 12 days, 64 k40s Perplexity (dev) 4.14 3.35 -19% French → English Test BLEU 36.47 34.40 37.46 +3.06 German → English Test BLEU 31.77 31.17 34.80 +3.63 Japanese → English Test BLEU 23.41 21.62 25.91 +4.29 Korean → English Test BLEU 25.42 22.87 28.71 +5.84 Portuguese → English Test BLEU 44.40 42.53 46.13 +3.60 Spanish → English Test BLEU 38.00 36.04 39.39 +3.35 English → French Test BLEU 35.37 34.00 36.59 +2.59 English → German Test BLEU 26.43 23.15 24.53 +1.38 English → Japanese Test BLEU 23.66 21.10 22.78 +1.68 English → Korean Test BLEU 19.75 18.41 16.62 -1.79 English → Portuguese Test BLEU 38.40 37.35 37.90 +0.55 English → Spanish Test BLEU 34.50 34.25 36.21 +1.96\n\n",
+          "embedding": [
+            -0.009427943266928196,
+            0.0009052514214999974,
+            -0.031882621347904205,
+            0.02155720256268978,
+            -0.0010102161904796958,
+            0.017834007740020752,
+            -0.019673390313982964,
+            0.04869475215673447,
+            -0.022961175069212914,
+            0.03497491776943207,
+            0.036041226238012314,
+            -0.002160385251045227,
+            0.013355513103306293,
+            -0.01921132393181324,
+            -0.013746492564678192,
+            0.007433058228343725,
+            -0.00785069540143013,
+            0.004056415054947138,
+            0.003800945123657584,
+            -0.0150793781504035,
+            -0.03134946897625923,
+            -0.014439593069255352,
+            -0.04698865860700607,
+            -0.015088263899087906,
+            -0.016625525429844856,
+            0.005407072138041258,
+            0.02432960458099842,
+            -0.0009768940508365631,
+            -0.040199827402830124,
+            0.01593242585659027,
+            0.004467388149350882,
+            0.0005736961611546576,
+            0.009525688365101814,
+            -0.008481594733893871,
+            -0.0005428731674328446,
+            -0.017078706994652748,
+            0.07435724139213562,
+            -0.004718414973467588,
+            -0.03138501197099686,
+            0.03760514408349991,
+            0.028381574898958206,
+            -0.006029085721820593,
+            -0.0007786272908560932,
+            0.0035210391506552696,
+            0.0162967462092638,
+            0.003805388230830431,
+            -0.01993996649980545,
+            0.007339756470173597,
+            -0.01571027748286724,
+            -0.037356339395046234,
+            -0.006606669165194035,
+            -0.03152718394994736,
+            0.0029923280235379934,
+            -0.02411634288728237,
+            -0.0463133305311203,
+            0.014768371358513832,
+            0.028292717412114143,
+            0.02548477053642273,
+            0.06902569532394409,
+            0.034797199070453644,
+            0.01878480054438114,
+            0.0006881021545268595,
+            -0.024222973734140396,
+            -0.02600015327334404,
+            0.03351762890815735,
+            -0.012244774959981441,
+            -0.02527150884270668,
+            0.005011649336665869,
+            0.016243431717157364,
+            -0.049334537237882614,
+            -0.03643220290541649,
+            0.037498511373996735,
+            0.009730064310133457,
+            -0.005127166397869587,
+            0.015026062726974487,
+            -0.022961175069212914,
+            0.04986768960952759,
+            0.045104846358299255,
+            -0.012777929194271564,
+            -0.0024103012401610613,
+            0.013133365660905838,
+            0.017505230382084846,
+            -0.008437165059149265,
+            -0.004611784126609564,
+            0.0228723157197237,
+            -0.022374704480171204,
+            -0.014164130203425884,
+            -0.07983095943927765,
+            0.0133999427780509,
+            -0.028097227215766907,
+            -0.03525926545262337,
+            0.015585875138640404,
+            0.015976853668689728,
+            -0.0007325316546484828,
+            -0.002772401785477996,
+            0.031171750277280807,
+            -0.024080798029899597,
+            -0.003567690262570977,
+            0.001345103606581688,
+            0.04762844368815422,
+            -0.01612791419029236,
+            2.6570933187031187e-05,
+            0.04848149046301842,
+            0.01478614378720522,
+            0.027048690244555473,
+            -0.012058171443641186,
+            -0.032824527472257614,
+            0.006117944605648518,
+            0.023796450346708298,
+            0.12625092267990112,
+            0.007517474237829447,
+            -0.004080851096659899,
+            -0.03270012512803078,
+            -0.01029876247048378,
+            -0.03732079640030861,
+            0.002994549460709095,
+            -0.030158756300807,
+            -0.02031317539513111,
+            0.021361712366342545,
+            -0.017620746046304703,
+            0.03206034004688263,
+            0.02752852812409401,
+            0.019246866926550865,
+            -0.020757470279932022,
+            -0.03646774962544441,
+            0.04862366244196892,
+            -0.004369643051177263,
+            0.012031513266265392,
+            0.0041452739387750626,
+            -0.024436235427856445,
+            -0.01317779440432787,
+            0.014368506148457527,
+            -0.004580683074891567,
+            -0.02411634288728237,
+            0.03417518362402916,
+            -0.020064370706677437,
+            0.026195643469691277,
+            0.04286559671163559,
+            -0.01928240992128849,
+            -0.015434814617037773,
+            -0.002169271232560277,
+            -0.013426600024104118,
+            0.0041008442640304565,
+            -0.018535993993282318,
+            -0.05061410740017891,
+            -0.006962105166167021,
+            -0.007939554750919342,
+            0.0924844816327095,
+            -0.012831244617700577,
+            0.023245522752404213,
+            -0.02607124112546444,
+            3.4328742913203314e-05,
+            -0.015274868346750736,
+            0.0298033207654953,
+            -0.013568774797022343,
+            -0.0069754342548549175,
+            -0.01673215627670288,
+            0.02038426324725151,
+            -0.04506930336356163,
+            0.015665847808122635,
+            -0.01008550077676773,
+            -0.010938547551631927,
+            -0.008441608399152756,
+            0.02219698764383793,
+            0.0029367911629378796,
+            0.035170406103134155,
+            0.05530586466193199,
+            -0.003258905140683055,
+            -0.06625329703092575,
+            0.04009319469332695,
+            0.004449616186320782,
+            0.009005863219499588,
+            -0.00478505901992321,
+            0.001390643883496523,
+            0.002912354888394475,
+            0.027155321091413498,
+            -0.004562911577522755,
+            0.016341175884008408,
+            -0.046846482902765274,
+            0.04155048355460167,
+            0.001919355127029121,
+            0.02104181982576847,
+            0.03643220290541649,
+            0.037143077701330185,
+            0.017149792984128,
+            0.006753286812454462,
+            0.03156273066997528,
+            0.02264128252863884,
+            0.006522253155708313,
+            0.015203780494630337,
+            0.006531138904392719,
+            0.011631648056209087,
+            0.0467398539185524,
+            0.004736186470836401,
+            0.0028012811671942472,
+            -0.008152816444635391,
+            -0.014608425088226795,
+            0.060957297682762146,
+            -0.0039808847941458225,
+            0.0034966031089425087,
+            0.011924882419407368,
+            -0.04453614726662636,
+            0.006513367407023907,
+            -0.010334305465221405,
+            0.02148611471056938,
+            0.017647404223680496,
+            0.015585875138640404,
+            0.032895613461732864,
+            0.021237310022115707,
+            -0.01701650582253933,
+            -0.030798541381955147,
+            -0.02534259669482708,
+            -0.02900358848273754,
+            -0.018011726438999176,
+            -0.004891689866781235,
+            0.023831993341445923,
+            -0.008308319374918938,
+            0.020490894094109535,
+            -0.008143930695950985,
+            -0.05004540830850601,
+            0.010680856183171272,
+            0.024347376078367233,
+            0.0011462816037237644,
+            -0.03884917125105858,
+            -0.03179376199841499,
+            -0.01540815643966198,
+            -0.025360368192195892,
+            -0.009223568253219128,
+            0.029110219329595566,
+            -0.01116958074271679,
+            0.02555585838854313,
+            0.02571580372750759,
+            -0.02038426324725151,
+            0.012555781751871109,
+            0.04592235013842583,
+            -0.016270089894533157,
+            0.0072997696697711945,
+            -0.00013884223881177604,
+            -0.025147106498479843,
+            0.009321312420070171,
+            -0.0197977926582098,
+            -0.019673390313982964,
+            -0.0338730625808239,
+            -0.017140908166766167,
+            -0.016909874975681305,
+            -0.02219698764383793,
+            -0.012466922402381897,
+            0.010494251735508442,
+            0.013702062889933586,
+            0.022694597020745277,
+            0.012031513266265392,
+            -0.03536589443683624,
+            0.015665847808122635,
+            0.007130937650799751,
+            -0.022730141878128052,
+            0.06554242223501205,
+            0.045460283756256104,
+            -0.004136388190090656,
+            -0.01041427906602621,
+            -0.046704307198524475,
+            0.0391690619289875,
+            0.032095883041620255,
+            -0.057296305894851685,
+            0.03236246109008789,
+            -0.017913982272148132,
+            0.042403530329465866,
+            0.02841711975634098,
+            0.025680260732769966,
+            -0.014377391897141933,
+            0.01803838461637497,
+            -0.005087179597467184,
+            -0.04777061566710472,
+            -0.034441761672496796,
+            -0.03433513268828392,
+            0.0076729776337742805,
+            -0.003112287726253271,
+            -0.012760157696902752,
+            -0.02900358848273754,
+            0.04133722186088562,
+            0.03214919939637184,
+            0.03563247248530388,
+            0.02104181982576847,
+            0.0030700797215104103,
+            0.00974783580750227,
+            0.016616638749837875,
+            -0.01890920288860798,
+            -0.014812801033258438,
+            -0.00617126002907753,
+            -0.03870699554681778,
+            -0.01540815643966198,
+            -0.017345283180475235,
+            0.002443623496219516,
+            -0.03756960108876228,
+            0.014155244454741478,
+            0.006144602317363024,
+            0.0014572881627827883,
+            0.011214010417461395,
+            0.002150388667359948,
+            0.03664546459913254,
+            -0.0016416707076132298,
+            -0.026906516402959824,
+            0.009943326003849506,
+            0.010405393317341805,
+            0.00527822645381093,
+            -0.029892180114984512,
+            -0.013799807988107204,
+            -0.006024642381817102,
+            0.027812877669930458,
+            -0.026266731321811676,
+            -0.009436829946935177,
+            -0.0020393149461597204,
+            0.0025813549291342497,
+            0.0016538887284696102,
+            -0.02024208754301071,
+            -0.016554437577724457,
+            0.014626197516918182,
+            0.0052115824073553085,
+            0.006637769751250744,
+            -0.04396745190024376,
+            -0.0198688805103302,
+            0.05573238804936409,
+            0.01586133800446987,
+            -0.06284110993146896,
+            0.023725362494587898,
+            -0.01658109575510025,
+            -0.011853795498609543,
+            0.006913233082741499,
+            0.02504047565162182,
+            -0.008406064473092556,
+            -0.01062754075974226,
+            0.0030611937399953604,
+            -0.052995529025793076,
+            -0.019389040768146515,
+            0.0035099319647997618,
+            0.004220804199576378,
+            -0.049263447523117065,
+            -0.019833335652947426,
+            -0.007597447372972965,
+            -0.015719164162874222,
+            0.003709864802658558,
+            0.009658977389335632,
+            -0.017416371032595634,
+            -0.010423164814710617,
+            0.012928989715874195,
+            0.018020613119006157,
+            -0.02299671806395054,
+            -0.03636111691594124,
+            0.027350811287760735,
+            -0.0019337947014719248,
+            -0.008099501021206379,
+            0.0016805464401841164,
+            0.012573553249239922,
+            0.0016838787123560905,
+            0.017771806567907333,
+            -0.005153824109584093,
+            -0.032682351768016815,
+            -0.02818608656525612,
+            0.02752852812409401,
+            -0.021734919399023056,
+            0.004147495608776808,
+            0.02278345637023449,
+            0.008303876966238022,
+            -0.011187352240085602,
+            0.020046597346663475,
+            -0.0527467243373394,
+            -0.008135044947266579,
+            0.0006581122288480401,
+            -0.006269005127251148,
+            0.025307053700089455,
+            -0.016829900443553925,
+            -0.014821686781942844,
+            0.019833335652947426,
+            0.009005863219499588,
+            0.0021626066882163286,
+            0.004014207050204277,
+            0.005966884084045887,
+            0.05512814596295357,
+            0.016154572367668152,
+            -0.01197819784283638,
+            0.017860665917396545,
+            -0.024596180766820908,
+            0.03102957457304001,
+            0.021290624514222145,
+            0.00529599841684103,
+            0.001194043317809701,
+            0.04581571742892265,
+            0.01008550077676773,
+            0.008757057599723339,
+            -0.011507244780659676,
+            0.011516130529344082,
+            -0.007401957642287016,
+            -0.05594564974308014,
+            -0.00806395709514618,
+            -0.022534651681780815,
+            0.002414744347333908,
+            0.0233699269592762,
+            0.031438328325748444,
+            -0.018376048654317856,
+            -0.00924133975058794,
+            -0.0009363520657643676,
+            -0.012102600187063217,
+            0.0010907446267083287,
+            -0.004687313921749592,
+            0.014359620399773121,
+            -0.0027457440737634897,
+            0.007686306722462177,
+            0.0018615968292579055,
+            -0.003058972302824259,
+            0.021343940868973732,
+            -0.03257572278380394,
+            -0.011853795498609543,
+            -0.03788949176669121,
+            -0.059784360229969025,
+            -0.03449507802724838,
+            0.030709682032465935,
+            0.023725362494587898,
+            -0.002539146924391389,
+            -0.004589569289237261,
+            -0.014324076473712921,
+            -0.031438328325748444,
+            -0.021592745557427406,
+            0.0006164595251902938,
+            0.04894355684518814,
+            -0.007868467830121517,
+            -0.02578689157962799,
+            0.01761186122894287,
+            -0.006939890794456005,
+            -0.004389636218547821,
+            -0.004927233327180147,
+            -0.03636111691594124,
+            0.023405469954013824,
+            0.010174359194934368,
+            0.03053196519613266,
+            -0.022143671289086342,
+            0.005575904622673988,
+            -0.006095729768276215,
+            -0.014377391897141933,
+            0.029963266104459763,
+            -0.008859246037900448,
+            0.004727300722151995,
+            0.0058869109489023685,
+            -0.003043422009795904,
+            0.0001384951319778338,
+            -0.005260454956442118,
+            -0.032540179789066315,
+            -0.026017924770712852,
+            -0.041230592876672745,
+            0.002370314672589302,
+            -0.02825717255473137,
+            -0.027137549594044685,
+            -0.06650210171937943,
+            0.024880530312657356,
+            -0.07098059356212616,
+            -0.02498716115951538,
+            0.038529276847839355,
+            -0.008885903283953667,
+            0.01768294721841812,
+            -0.04279451072216034,
+            -0.0006664427346549928,
+            -0.05107617378234863,
+            -0.0009580114856362343,
+            0.035241492092609406,
+            0.0029234623070806265,
+            -0.006393407471477985,
+            0.014368506148457527,
+            -0.00013120591756887734,
+            -0.03389083594083786,
+            -0.04229690134525299,
+            0.0038276028353720903,
+            0.005207139533013105,
+            -0.02301448956131935,
+            -0.014288532547652721,
+            0.04339875280857086,
+            0.00908583588898182,
+            -0.0006070183007977903,
+            0.0300876684486866,
+            0.007939554750919342,
+            -0.027137549594044685,
+            0.019246866926550865,
+            0.02564471773803234,
+            0.038884714245796204,
+            -0.02768847532570362,
+            0.021130679175257683,
+            -0.03200702369213104,
+            -0.005629220046103001,
+            0.011569445952773094,
+            0.0023636503610759974,
+            -0.006011313758790493,
+            0.014457364566624165,
+            -0.0015683620003983378,
+            0.0034077439922839403,
+            0.006842145696282387,
+            -0.006304548587650061,
+            0.032984472811222076,
+            -0.0004387414955999702,
+            -0.016821015626192093,
+            -0.021148450672626495,
+            0.009694520384073257,
+            0.031082890927791595,
+            0.028737012296915054,
+            -0.018304960802197456,
+            -0.014635083265602589,
+            -0.003976441919803619,
+            0.029056904837489128,
+            0.016083484515547752,
+            -0.01384423766285181,
+            0.0067444005981087685,
+            0.02431183122098446,
+            -0.023618731647729874,
+            -0.01586133800446987,
+            0.017398599535226822,
+            -0.006424508057534695,
+            0.012564667500555515,
+            -0.012644641101360321,
+            0.00200932496227324,
+            0.017931753769516945,
+            -0.005367085803300142,
+            0.026693254709243774,
+            -0.017034277319908142,
+            0.010529795661568642,
+            -0.007459715940058231,
+            -0.008681527338922024,
+            0.007917339913547039,
+            0.06948776543140411,
+            -0.020615296438336372,
+            0.022659054026007652,
+            0.018464908003807068,
+            -0.0018671504221856594,
+            0.034726109355688095,
+            -0.04457169026136398,
+            -0.046704307198524475,
+            0.012182573787868023,
+            -0.04314994812011719,
+            -0.0020493115298449993,
+            -0.022410249337553978,
+            -0.023032261058688164,
+            0.008321648463606834,
+            0.011915996670722961,
+            -0.024880530312657356,
+            -0.008717071264982224,
+            0.020864101126790047,
+            0.01577247865498066,
+            -0.02169937640428543,
+            0.019922195002436638,
+            0.00920579582452774,
+            -0.018749255686998367,
+            0.032682351768016815,
+            -0.02205481193959713,
+            -0.0003795946831814945,
+            -0.007099837064743042,
+            -0.0024947174824774265,
+            -0.00668219942599535,
+            -0.004285227041691542,
+            0.008868131786584854,
+            -0.02893250249326229,
+            -0.01622566021978855,
+            0.01680324412882328,
+            -0.00772185018286109,
+            -0.010058842599391937,
+            -0.042403530329465866,
+            0.0032611265778541565,
+            0.013782036490738392,
+            -0.023281067609786987,
+            0.01132064126431942,
+            -0.019477900117635727,
+            -0.032895613461732864,
+            0.01993996649980545,
+            0.03508154675364494,
+            -0.032984472811222076,
+            0.0016561102820560336,
+            0.00873040035367012,
+            -0.009028078056871891,
+            -0.0024502878077328205,
+            -0.01766517572104931,
+            0.007157595362514257,
+            0.03271789476275444,
+            0.014741714112460613,
+            -0.013257768005132675,
+            -0.024222973734140396,
+            -0.017807351425290108,
+            0.021379483863711357,
+            -0.0059313406236469746,
+            0.005162709858268499,
+            0.02562694624066353,
+            0.041514940559864044,
+            0.012173688039183617,
+            0.0008830366423353553,
+            -0.013666519895195961,
+            0.014803915284574032,
+            -0.015887996181845665,
+            0.021646060049533844,
+            0.022072583436965942,
+            -0.01746080070734024,
+            -0.03803166747093201,
+            0.018429363146424294,
+            0.00022339716088026762,
+            0.004554025363177061,
+            0.008112830109894276,
+            0.02768847532570362,
+            -0.016616638749837875,
+            -0.00549593148753047,
+            0.03934678062796593,
+            0.016314519569277763,
+            -0.014892774634063244,
+            -0.01622566021978855,
+            0.016838787123560905,
+            0.0074419439770281315,
+            0.006060186307877302,
+            -0.0018293854082003236,
+            -0.010369849391281605,
+            -0.0016127914423123002,
+            0.009365742094814777,
+            -0.008481594733893871,
+            0.008992534130811691,
+            0.014839459210634232,
+            0.005438172724097967,
+            0.0011035181814804673,
+            -0.01737194135785103,
+            -0.01263575442135334,
+            0.015799136832356453,
+            0.010538681410253048,
+            -0.010023298673331738,
+            0.00927688367664814,
+            0.007961769588291645,
+            0.014013069681823254,
+            0.032166969031095505,
+            -0.0031278380192816257,
+            -0.01671438477933407,
+            -0.0026657709386199713,
+            -0.0006958773010410368,
+            -0.025769120082259178,
+            0.007601890247315168,
+            -0.014892774634063244,
+            -0.02898581698536873,
+            0.018429363146424294,
+            0.02249910682439804,
+            -0.011418386362493038,
+            -0.01802949793636799,
+            -0.0030878514517098665,
+            -0.0043563139624893665,
+            -0.006264561787247658,
+            0.007406400516629219,
+            -0.011276211589574814,
+            0.042616792023181915,
+            -0.001704982714727521,
+            0.0023614289239048958,
+            -0.02703091874718666,
+            -0.0023503215052187443,
+            0.04332766681909561,
+            0.025253737345337868,
+            0.0030545294284820557,
+            0.009116937406361103,
+            0.008010641671717167,
+            0.0467398539185524,
+            -0.013764264062047005,
+            -0.04133722186088562,
+            0.008601554669439793,
+            -0.015319298021495342,
+            0.012173688039183617,
+            0.03266458213329315,
+            -0.009436829946935177,
+            0.010014412924647331,
+            0.010547567158937454,
+            -0.03147387132048607,
+            0.027350811287760735,
+            -0.023796450346708298,
+            -0.022605737671256065,
+            0.030994031578302383,
+            -0.01723865233361721,
+            0.01074305735528469,
+            0.028737012296915054,
+            0.010911889374256134,
+            -0.029838863760232925,
+            0.0013817580183967948,
+            -0.010423164814710617,
+            -0.025218194350600243,
+            0.025911293923854828,
+            0.02863038145005703,
+            0.018642624840140343,
+            0.0063001057133078575,
+            0.0015339291421696544,
+            0.004896132741123438,
+            0.0033699788618832827,
+            -0.01007661409676075,
+            0.013204452581703663,
+            -0.010165473446249962,
+            -0.01768294721841812,
+            -0.0015283754328265786,
+            -0.013568774797022343,
+            -0.00927688367664814,
+            0.0021748249419033527,
+            -0.02607124112546444,
+            0.009134708903729916,
+            0.012351405806839466,
+            0.010458708740770817,
+            -0.018962517380714417,
+            -0.01862485334277153,
+            0.05534140765666962,
+            0.013799807988107204,
+            -0.0032700125593692064,
+            0.004087515641003847,
+            -0.017585203051567078,
+            -0.04318549111485481,
+            0.004571797326207161,
+            0.011418386362493038,
+            0.005562575533986092,
+            0.00020270967797841877,
+            -0.030496420338749886,
+            0.008108386769890785,
+            0.032326918095350266,
+            0.008046185597777367,
+            0.04446506127715111,
+            -0.0012251439038664103,
+            0.030123213306069374,
+            -0.008414950221776962,
+            0.003574354574084282,
+            0.019513443112373352,
+            0.0041230591014027596,
+            -0.019033605232834816,
+            0.03038978949189186,
+            0.01834050565958023,
+            0.018304960802197456,
+            0.006979877129197121,
+            -0.04514038935303688,
+            -0.009872239083051682,
+            0.010236560367047787,
+            0.03182930499315262,
+            0.020259859040379524,
+            -0.017176451161503792,
+            -0.007437501102685928,
+            0.03296670317649841,
+            -0.022232530638575554,
+            -0.004431844223290682,
+            0.0116138756275177,
+            -0.008597111329436302,
+            0.002483610063791275,
+            -0.004687313921749592,
+            0.02249910682439804,
+            -0.039844390004873276,
+            -0.004687313921749592,
+            0.0019349055364727974,
+            0.0057225218042731285,
+            -0.002215922111645341,
+            -0.006939890794456005,
+            0.010405393317341805,
+            0.014261875301599503,
+            0.009076950140297413,
+            0.02943011187016964,
+            -0.0201532281935215,
+            0.015034948475658894,
+            -0.015168237499892712,
+            -0.003134502563625574,
+            0.005749179515987635,
+            -0.03138501197099686,
+            -0.030283158645033836,
+            0.007619662210345268,
+            0.03175821900367737,
+            0.013080050237476826,
+            0.0014817244373261929,
+            -0.004589569289237261,
+            -0.011516130529344082,
+            0.026853200048208237,
+            0.02109513431787491,
+            0.02111290581524372,
+            0.023831993341445923,
+            -0.005087179597467184,
+            -0.0004323547473177314,
+            0.012475808151066303,
+            -0.008650426752865314,
+            -0.024791670963168144,
+            0.008930332958698273,
+            -0.011427272111177444,
+            0.0270842332392931,
+            -0.01724753901362419,
+            -0.003258905140683055,
+            0.005646991543471813,
+            -0.012404721230268478,
+            -0.01128509733825922,
+            -0.039702218025922775,
+            0.025893522426486015,
+            -0.02548477053642273,
+            0.0036343345418572426,
+            -0.019193550571799278,
+            0.01490166038274765,
+            -0.03870699554681778,
+            0.000813615508377552,
+            -0.0018393819918856025,
+            0.010538681410253048,
+            -0.010529795661568642,
+            0.0021770463790744543,
+            -0.03193593770265579,
+            -0.03788949176669121,
+            -0.0116849634796381,
+            0.005073850974440575,
+            0.007899568416178226,
+            -0.0009330198518000543,
+            0.006913233082741499,
+            0.015967968851327896,
+            -0.025093792006373405,
+            -0.03369534760713577,
+            0.018820343539118767,
+            -0.03440621867775917,
+            0.020419806241989136,
+            0.0004870585980825126,
+            -0.02804391086101532,
+            -0.021965952590107918,
+            -0.016492236405611038,
+            -0.015248210169374943,
+            0.021006274968385696,
+            0.022552423179149628,
+            0.017798464745283127,
+            -0.016847673803567886,
+            0.0025124892126768827,
+            0.014039727859199047,
+            0.008121715858578682,
+            0.011880452744662762,
+            0.008481594733893871,
+            0.01825164631009102,
+            -0.016847673803567886,
+            -0.015887996181845665,
+            -0.012244774959981441,
+            -0.002794616622850299,
+            -0.013799807988107204,
+            0.0011179577559232712,
+            -0.038813624531030655,
+            -0.00737529993057251,
+            0.01116958074271679,
+            -0.0020048818551003933,
+            0.06038859859108925,
+            0.0527467243373394,
+            0.013160022906959057,
+            0.007384185679256916,
+            -0.01876702904701233,
+            0.0461711548268795,
+            -0.025804663076996803,
+            0.04986768960952759,
+            0.0034521734341979027,
+            -0.013142251409590244,
+            -0.013462143950164318,
+            0.013426600024104118,
+            -0.029199078679084778,
+            -0.00719313882291317,
+            -0.005424844101071358,
+            -0.009005863219499588,
+            0.018002839758992195,
+            0.00739307189360261,
+            -0.009347970597445965,
+            0.003696535946801305,
+            0.009383514523506165,
+            -0.011258440092206001,
+            0.013222224079072475,
+            0.01612791419029236,
+            0.0017349726986140013,
+            0.015070492401719093,
+            -0.04208363965153694,
+            -0.011231781914830208,
+            -0.00041291682282462716,
+            0.009685634635388851,
+            -0.006220132578164339,
+            0.013168908655643463,
+            -0.006428950931876898,
+            -0.0044229584746062756,
+            -0.007015420589596033,
+            -0.02169937640428543,
+            0.022534651681780815,
+            0.036041226238012314,
+            -0.0016938752960413694,
+            -0.048979099839925766,
+            -0.02614232711493969,
+            -0.050223127007484436,
+            -0.012928989715874195,
+            0.024063026532530785,
+            -0.0050249784253537655,
+            0.015976853668689728,
+            0.004087515641003847,
+            0.0048828041180968285,
+            -0.0030456434469670057,
+            -0.017567431554198265,
+            0.01613680087029934,
+            -0.0232277512550354,
+            0.002739079762250185,
+            0.0003854260721709579,
+            0.007144266273826361,
+            -0.03543698415160179,
+            0.007757393643260002,
+            0.00023422685626428574,
+            0.012093714438378811,
+            -0.03660992160439491,
+            -0.038244929164648056,
+            -0.017398599535226822,
+            0.015941310673952103,
+            -0.002383643528446555,
+            -0.012244774959981441,
+            0.0029501200187951326,
+            0.005033864174038172,
+            0.014741714112460613,
+            -0.017567431554198265,
+            -0.006229018326848745,
+            0.02162828855216503,
+            -0.0032433548476547003,
+            0.007206467911601067,
+            -0.035099320113658905,
+            -0.01555921696126461,
+            -0.01278681494295597,
+            0.0116494195535779,
+            -0.0133999427780509,
+            0.004905018489807844,
+            0.00017799575289245695,
+            0.014848344959318638,
+            -0.015505901537835598,
+            -0.027013147249817848,
+            -0.011231781914830208,
+            0.020099913701415062,
+            -0.03511708974838257,
+            0.036112312227487564,
+            0.002739079762250185,
+            0.02118399366736412,
+            -0.014404049143195152,
+            0.025609172880649567,
+            -0.004036421421915293,
+            -0.015301525592803955,
+            -0.002151499269530177,
+            0.022001497447490692,
+            0.03332213684916496,
+            -0.021361712366342545,
+            0.0013228888856247067,
+            -0.019015833735466003,
+            0.003327770857140422,
+            -0.02351210080087185,
+            0.017345283180475235,
+            -0.004007542505860329,
+            0.01234252005815506,
+            0.017105363309383392,
+            0.013204452581703663,
+            0.05996207520365715,
+            -0.017194222658872604,
+            0.02557362988591194,
+            0.012324748560786247,
+            -0.017265310510993004,
+            -0.007935111410915852,
+            -0.032095883041620255,
+            -0.027333039790391922,
+            0.03133169561624527,
+            0.002228140365332365,
+            0.0015561438631266356,
+            -0.03104734607040882,
+            0.015879109501838684,
+            0.01832273229956627,
+            0.019460128620266914,
+            -0.014288532547652721,
+            -0.015274868346750736,
+            0.016030170023441315,
+            0.026248957961797714,
+            -0.008135044947266579,
+            0.003865367965772748,
+            0.012769043445587158,
+            -0.04076852649450302,
+            0.005282669793814421,
+            0.0029634488746523857,
+            0.012857901863753796,
+            -0.0021548315417021513,
+            0.010183244943618774,
+            -0.03214919939637184,
+            -0.0055137029848992825,
+            -0.01457288209348917,
+            0.02315666526556015,
+            -0.015665847808122635,
+            -0.047806162387132645,
+            -0.03161604329943657,
+            0.025147106498479843,
+            -0.030194299295544624,
+            -0.020988503471016884,
+            0.01311559323221445,
+            0.0022170329466462135,
+            0.015337069518864155,
+            -0.01471505593508482,
+            -0.02747521363198757,
+            -0.038955800235271454,
+            -0.021965952590107918,
+            -0.0006886575138196349,
+            0.0394534096121788,
+            -0.03010544180870056,
+            0.015159351751208305,
+            0.005309327505528927,
+            0.00686436053365469,
+            -0.004869475029408932,
+            0.002365871798247099,
+            -0.029447883367538452,
+            -0.04133722186088562,
+            0.02351210080087185,
+            -0.00494944816455245,
+            -0.0015150465769693255,
+            0.014803915284574032,
+            -0.033055562525987625,
+            -0.023103348910808563,
+            0.003154495730996132,
+            0.00737529993057251,
+            -0.00785069540143013,
+            -0.002352542942389846,
+            0.0013395498972386122,
+            -0.01215591561049223,
+            -0.038600362837314606,
+            -0.004416293930262327,
+            0.03397969529032707,
+            0.02177046425640583,
+            0.031740445643663406,
+            0.011525017209351063,
+            0.00010468705295352265,
+            -0.02886141464114189,
+            -0.013595432043075562,
+            -0.006895461119711399,
+            -0.0009818923426792026,
+            -0.0025769120547920465,
+            0.01848267950117588,
+            0.005291555542498827,
+            0.0034077439922839403,
+            -0.02169937640428543,
+            -0.02783064916729927,
+            -0.03511708974838257,
+            -0.01928240992128849,
+            -0.010049956850707531,
+            -0.00949014537036419,
+            -0.03899134323000908,
+            -0.016741042956709862,
+            0.04485604166984558,
+            0.026995373889803886,
+            -0.009152480401098728,
+            -0.00018729818111751229,
+            -0.0012240331852808595,
+            -0.012973419390618801,
+            0.019886652007699013,
+            -0.013826466165482998,
+            0.0059091257862746716,
+            0.00031656029750593007,
+            0.018607081845402718,
+            0.0017538551473990083,
+            0.0010013302089646459,
+            -0.03419295698404312,
+            -0.009401286020874977,
+            0.031171750277280807,
+            0.00701097771525383,
+            -0.019531216472387314,
+            0.0163678340613842,
+            0.009259111247956753,
+            -0.014466251246631145,
+            -0.047806162387132645,
+            0.026782112196087837,
+            0.04567354544997215,
+            0.02212589979171753,
+            -0.005940226372331381,
+            0.022179214283823967,
+            0.024933844804763794,
+            -0.0012462479062378407,
+            -0.0007541910745203495,
+            -0.004945005290210247,
+            0.026977602392435074,
+            0.006535581778734922,
+            -0.000127387756947428,
+            0.00939240027219057,
+            0.01810947060585022,
+            0.005838038399815559,
+            -0.0007697413675487041,
+            0.007077622227370739,
+            -0.020793013274669647,
+            -0.004913904704153538,
+            -0.013853123411536217,
+            0.010974090546369553,
+            0.00920579582452774,
+            0.041088417172431946,
+            0.008774830028414726,
+            0.013559889048337936,
+            -0.0043563139624893665,
+            -0.010174359194934368,
+            0.028648152947425842,
+            0.014039727859199047,
+            0.001577247865498066,
+            0.008499366231262684,
+            -0.030834084376692772,
+            -0.01586133800446987,
+            -0.013950868509709835,
+            0.01384423766285181,
+            0.0037520728074014187,
+            0.04286559671163559,
+            0.006873246282339096,
+            0.0016794357215985656,
+            -0.02943011187016964,
+            0.0026235629338771105,
+            0.008232789114117622,
+            0.02285454422235489,
+            -0.013266653753817081,
+            -2.6171763238380663e-05,
+            0.0073664141818881035,
+            0.005638105794787407,
+            0.026497764512896538,
+            -0.026195643469691277,
+            -0.017052048817276955,
+            0.019300181418657303,
+            0.09511470794677734,
+            -0.009330199100077152,
+            -0.0233699269592762,
+            -0.026053469628095627,
+            -0.02104181982576847,
+            0.008241675794124603,
+            0.017585203051567078,
+            0.005429286975413561,
+            0.017443029209971428,
+            0.010218788869678974,
+            0.01788732409477234,
+            0.015603646636009216,
+            -0.015976853668689728,
+            -0.007419729605317116,
+            -0.03225582838058472,
+            0.0327356681227684,
+            0.00014911657490301877,
+            -0.0047228578478097916,
+            0.00739307189360261,
+            0.007135380525141954,
+            0.019744478166103363,
+            0.018091699108481407,
+            0.015887996181845665,
+            -0.007979541085660458,
+            -0.011924882419407368,
+            0.020135456696152687,
+            -0.013142251409590244,
+            -0.0034810525830835104,
+            -0.01899806223809719,
+            -0.021592745557427406,
+            -0.005651434417814016,
+            0.014643969014286995,
+            -0.006468937732279301,
+            0.014848344959318638,
+            -0.01132064126431942,
+            0.028310488909482956,
+            -0.021965952590107918,
+            0.011160694994032383,
+            -0.00737529993057251,
+            -0.01526598259806633,
+            0.005216025281697512,
+            0.010245447047054768,
+            -0.007677420508116484,
+            0.010716399177908897,
+            -0.014324076473712921,
+            0.014821686781942844,
+            0.019477900117635727,
+            -0.03650329262018204,
+            0.03328659385442734,
+            0.008126158267259598,
+            -0.009836695156991482,
+            0.00924133975058794,
+            -0.00582470977678895,
+            0.012529123574495316,
+            0.007268669083714485,
+            0.00806395709514618,
+            0.008352749049663544,
+            0.00817947369068861,
+            -0.007659649010747671,
+            0.014883887954056263,
+            0.012138144113123417,
+            0.0007353085093200207,
+            0.010636426508426666,
+            0.014279646798968315,
+            -0.009730064310133457,
+            -0.004696200136095285,
+            0.0038364888168871403,
+            -0.010325419716536999,
+            0.007317541632801294,
+            -0.009108050726354122,
+            0.0060912868939340115,
+            0.024507321417331696,
+            -0.0017083149868994951,
+            -0.016492236405611038,
+            -0.016625525429844856,
+            -0.007935111410915852,
+            -0.021237310022115707,
+            -0.015434814617037773,
+            0.00514938123524189,
+            -0.030834084376692772,
+            0.013053392060101032,
+            0.028665924444794655,
+            0.0233699269592762,
+            -0.006180145777761936,
+            -0.024720583111047745,
+            -0.012413606978952885,
+            -0.0018582645570859313,
+            0.011276211589574814,
+            0.005162709858268499,
+            -0.02454286627471447,
+            0.002643556334078312,
+            -0.011356184259057045,
+            -0.021432798355817795,
+            -0.008277218788862228,
+            -0.01651000790297985,
+            0.003458837978541851,
+            0.0030456434469670057,
+            -0.008441608399152756,
+            -0.010707513429224491,
+            0.003541032550856471,
+            -0.03685872629284859,
+            0.014048613607883453,
+            0.03207810968160629,
+            -0.000934130628593266,
+            -0.03255794942378998,
+            -0.015221552923321724,
+            -0.003250019159168005,
+            -0.017194222658872604,
+            -0.02863038145005703,
+            0.008934776298701763,
+            0.006104615516960621,
+            -0.008454937487840652,
+            0.01252023782581091,
+            -0.004036421421915293,
+            0.02001105435192585,
+            0.0009130265680141747,
+            0.01869594119489193,
+            -0.0408751554787159,
+            0.012564667500555515,
+            0.019158007577061653,
+            -0.019122464582324028,
+            0.006904346868395805,
+            0.023458784446120262,
+            -0.008383849635720253,
+            0.0034699453972280025,
+            0.035756874829530716,
+            0.010369849391281605,
+            -0.042474620044231415,
+            0.01905137673020363,
+            0.006082401145249605,
+            -0.00668219942599535,
+            0.016323404386639595,
+            0.019673390313982964,
+            -0.000788623932749033,
+            0.020757470279932022,
+            -0.006446722894906998,
+            -0.018233874812722206,
+            0.00218260008841753,
+            0.020846329629421234,
+            -0.009330199100077152,
+            -0.015345955267548561,
+            0.01092077512294054,
+            0.0027235294692218304,
+            0.004758401308208704,
+            0.01738082803785801,
+            -0.02607124112546444,
+            -0.018642624840140343,
+            0.019513443112373352,
+            0.012831244617700577,
+            0.017336398363113403,
+            -0.011214010417461395,
+            -0.01402195543050766,
+            0.02219698764383793,
+            -0.005464830435812473,
+            0.010707513429224491,
+            -0.031082890927791595,
+            0.008748171851038933,
+            -0.0019904423970729113,
+            0.01848267950117588,
+            -0.004918347578495741,
+            0.008752615191042423,
+            0.03255794942378998,
+            -0.015514787286520004,
+            -0.025964610278606415,
+            0.009543460793793201,
+            -0.033624257892370224,
+            -0.010094386525452137,
+            -0.029607830569148064,
+            -0.03920460492372513,
+            0.03170490264892578,
+            -0.008046185597777367,
+            -0.0049583339132368565,
+            0.026746569201350212,
+            0.007353085093200207,
+            0.0020270966924726963,
+            -0.03213142603635788,
+            0.021574974060058594,
+            0.006642212625592947,
+            0.007784051354974508,
+            -0.018873659893870354,
+            0.025147106498479843,
+            -0.0030922943260520697,
+            -0.021166222169995308,
+            0.013462143950164318,
+            0.012182573787868023,
+            0.008126158267259598,
+            -0.004534032195806503,
+            0.001586133730597794,
+            -0.03682318329811096,
+            -0.005775837227702141,
+            0.008246118202805519,
+            0.005522589199244976,
+            -0.003236690303310752,
+            0.018802572041749954,
+            -0.016909874975681305,
+            -0.002843489171937108,
+            -0.005629220046103001,
+            -0.022747913375496864,
+            0.0033855291549116373,
+            -0.03248686343431473,
+            -0.020686382427811623,
+            0.019780021160840988,
+            -0.003989770542830229,
+            -0.01825164631009102,
+            0.014581767842173576,
+            -0.03980884701013565,
+            -0.011178466491401196,
+            0.007304212544113398,
+            0.027777334675192833,
+            -0.03447730466723442,
+            0.015719164162874222,
+            -0.01958453096449375,
+            -0.0023769792169332504,
+            0.023618731647729874,
+            -0.011809365823864937,
+            -0.00010725563333835453,
+            -0.012564667500555515,
+            -0.031882621347904205,
+            0.003963112831115723,
+            0.028239401057362556,
+            -0.029234621673822403,
+            0.0009446826297789812,
+            0.031136205419898033,
+            -0.000653669296298176,
+            0.004616227000951767,
+            0.0007586340070702136,
+            0.01113403681665659,
+            -0.015186008997261524,
+            0.015106036327779293,
+            0.005838038399815559,
+            0.00582470977678895,
+            -0.01372872106730938,
+            -0.005202696658670902,
+            0.02710200473666191,
+            0.001230697613209486,
+            -0.008819258771836758,
+            0.0047006430104374886,
+            0.004058636259287596,
+            0.010174359194934368,
+            -0.0016627747099846601,
+            0.014457364566624165,
+            0.018100585788488388,
+            -0.028168313205242157,
+            0.012884560041129589,
+            0.009836695156991482,
+            7.844586798455566e-05,
+            0.013995298184454441,
+            -0.017514115199446678,
+            -0.034352902323007584,
+            0.02161051705479622,
+            -0.022019268944859505,
+            -0.016234545037150383,
+            0.0270842332392931,
+            -0.010867459699511528,
+            -0.006064629182219505,
+            -0.009774493984878063,
+            -0.001900472561828792,
+            0.010609768331050873,
+            -0.009978869929909706,
+            -0.009463487192988396,
+            0.023760905489325523,
+            0.018145015463232994,
+            0.020917417481541634,
+            0.029856635257601738,
+            0.0012118150480091572,
+            0.01570139080286026,
+            -0.0005223245243541896,
+            0.008974762633442879,
+            0.005171595606952906,
+            -0.0018627075478434563,
+            -0.01738971285521984,
+            0.025982381775975227,
+            -0.001804949133656919,
+            0.006264561787247658,
+            0.0063223205506801605,
+            -0.014368506148457527,
+            -0.010867459699511528,
+            -0.026479993015527725,
+            -0.004887246992439032,
+            -0.006580011453479528,
+            -0.016234545037150383,
+            0.0789068192243576,
+            -0.0099344402551651,
+            -0.016554437577724457,
+            0.010494251735508442,
+            -0.004856146406382322,
+            0.0015528115909546614,
+            -0.016847673803567886,
+            0.003736522514373064,
+            -0.008783715777099133,
+            -0.000988556770607829,
+            -0.030176527798175812,
+            -0.0015616975724697113,
+            0.0167943574488163,
+            0.015283754095435143,
+            -0.006557796616107225,
+            -0.018731484189629555,
+            0.008223903365433216,
+            -0.023209979757666588,
+            0.02036648988723755,
+            0.0012418050318956375,
+            0.018447136506438255,
+            0.005087179597467184,
+            0.005082736723124981,
+            -0.014946090057492256,
+            -0.016749927774071693,
+            0.004434065893292427,
+            0.012315861880779266,
+            0.002247022930532694,
+            -0.0090236347168684,
+            -0.009152480401098728,
+            -0.011009634472429752,
+            -0.021859321743249893,
+            0.020544208586215973,
+            -0.01296453271061182,
+            -0.0039142402820289135,
+            -0.003983106464147568,
+            -0.0020726369693875313,
+            0.006331206299364567,
+            0.011356184259057045,
+            -0.02358318865299225,
+            0.0018849222688004375,
+            -0.016972076147794724,
+            0.006913233082741499,
+            0.02504047565162182,
+            0.00981892365962267,
+            -0.04720192030072212,
+            -0.00651781028136611,
+            -0.02754630148410797,
+            0.013746492564678192,
+            -0.011747164651751518,
+            -0.009445715695619583,
+            0.0133999427780509,
+            0.008072842843830585,
+            0.028719240799546242,
+            0.010280990041792393,
+            0.009881124831736088,
+            0.015372613444924355,
+            -0.00032850075513124466,
+            -0.004440730437636375,
+            -0.017647404223680496,
+            0.013568774797022343,
+            -0.023316610604524612,
+            0.020348718389868736,
+            -0.0006103505147621036,
+            -0.017034277319908142,
+            0.004194146487861872,
+            0.002821274334564805,
+            0.013853123411536217,
+            -0.021806007251143456,
+            -0.022481335327029228,
+            0.022730141878128052,
+            0.018216101452708244,
+            0.012600211426615715,
+            0.015123807825148106,
+            0.03182930499315262,
+            0.02139725536108017,
+            0.024294059723615646,
+            -0.025324825197458267,
+            0.0014428485883399844,
+            -0.033855292946100235,
+            -0.005087179597467184,
+            -0.026551079005002975,
+            0.004038643091917038,
+            -0.012973419390618801,
+            0.005638105794787407,
+            -0.022001497447490692,
+            0.001768294838257134,
+            0.017549660056829453,
+            -0.030691910535097122,
+            0.010965204797685146,
+            0.015754707157611847,
+            0.00788623932749033,
+            0.017993954941630363,
+            0.006944333668798208,
+            -0.012831244617700577,
+            -0.01737194135785103,
+            -0.008717071264982224,
+            -0.013782036490738392,
+            0.014812801033258438,
+            0.0020082141272723675,
+            -0.0027679589111357927,
+            -0.00027990597300231457,
+            0.0001258604897884652,
+            -0.014466251246631145,
+            -0.001746080000884831,
+            -0.005180481821298599,
+            -0.013710948638617992,
+            0.005926897749304771,
+            0.0014928317395970225,
+            0.013222224079072475,
+            0.008917003870010376,
+            -0.007286441046744585,
+            0.011293983086943626,
+            0.02578689157962799,
+            0.0005528698093257844,
+            -0.018358277156949043,
+            0.00721091078594327,
+            0.008290547877550125,
+            0.007215353660285473,
+            0.006073514930903912,
+            0.0015172680141404271,
+            -0.012253660708665848,
+            0.027333039790391922,
+            -0.023938624188303947,
+            0.02205481193959713,
+            0.004856146406382322,
+            -0.0008075064979493618,
+            0.0003068413643632084,
+            -0.03518817946314812,
+            0.002401415491476655,
+            -0.024222973734140396,
+            -0.04112396016716957,
+            -0.02411634288728237,
+            -0.01547924429178238,
+            0.0007264225860126317,
+            0.022765684872865677,
+            0.0012862344738095999,
+            -0.027297494933009148,
+            0.022907858714461327,
+            -0.001922687399201095,
+            -0.015959082171320915,
+            0.00420081103220582,
+            -0.001362875453196466,
+            0.0019515665480867028,
+            0.0028745897579938173,
+            -0.0029567843303084373,
+            -0.0081839170306921,
+            -0.017407484352588654,
+            -0.02635558880865574,
+            0.003163381712511182,
+            0.009223568253219128,
+            -0.019815564155578613,
+            0.02249910682439804,
+            -0.009978869929909706,
+            -0.015879109501838684,
+            0.026497764512896538,
+            -0.009312426671385765,
+            -0.021343940868973732,
+            -0.021308396011590958,
+            0.02095296047627926,
+            0.004669542424380779,
+            -0.024418462067842484,
+            0.01643003523349762,
+            0.005562575533986092,
+            -0.017638517543673515,
+            -0.002251465804874897,
+            -0.012040399014949799,
+            0.012351405806839466,
+            -0.01782512292265892,
+            0.0016472243005409837,
+            0.010503137484192848,
+            0.0017738484311848879,
+            0.0030278717167675495,
+            -0.01095631904900074,
+            0.019993282854557037,
+            0.01825164631009102,
+            0.008046185597777367,
+            -0.03200702369213104,
+            0.008819258771836758,
+            -0.007588561624288559,
+            -0.0007630769396200776,
+            -0.020419806241989136,
+            0.029821092262864113,
+            0.013435485772788525,
+            0.007975098676979542,
+            0.01055645290762186,
+            -0.006491152569651604,
+            0.006282333750277758,
+            -0.009685634635388851,
+            -0.002785730641335249,
+            -0.013319969177246094,
+            -0.029678918421268463,
+            -0.007593004498630762,
+            0.023760905489325523,
+            -0.006451165769249201,
+            0.01095631904900074,
+            -0.004316327627748251,
+            0.029270166531205177,
+            -0.015106036327779293,
+            0.000624234729912132,
+            0.02477389946579933,
+            -0.0022370261140167713,
+            0.0058646961115300655,
+            0.011533902958035469,
+            0.06394296139478683,
+            -0.003012321190908551,
+            0.013239996507763863,
+            0.0013817580183967948,
+            0.002856818027794361,
+            0.007948440499603748,
+            -0.015070492401719093,
+            -0.03190039470791817,
+            0.001449513016268611,
+            0.0043718647211790085,
+            -0.013435485772788525,
+            -0.0016072377329692245,
+            -0.0018793685594573617,
+            0.001987110124900937,
+            -0.004909461829811335,
+            0.018153900280594826,
+            -0.020935188978910446,
+            0.028737012296915054,
+            0.019851109012961388,
+            0.012884560041129589,
+            -0.010387620888650417,
+            0.015941310673952103,
+            0.025253737345337868,
+            -0.008121715858578682,
+            -0.0038498176727443933,
+            0.007401957642287016,
+            0.0053937435150146484,
+            0.023760905489325523,
+            0.0027079791761934757,
+            0.029981037601828575,
+            -0.0035343680065125227,
+            0.003134502563625574,
+            -0.014270761050283909,
+            -0.001956009538844228,
+            -0.0033988580107688904,
+            0.005904682911932468,
+            0.003414408303797245,
+            0.0014061941765248775,
+            -0.016403377056121826,
+            0.013648747466504574,
+            0.004176374524831772,
+            0.003367757424712181,
+            0.01456399541348219,
+            -0.0006292330217547715,
+            -0.002503603231161833,
+            0.013444371521472931,
+            -0.002197039546445012,
+            -0.0013151137391105294,
+            0.008143930695950985,
+            -0.012315861880779266,
+            -0.016332291066646576,
+            -0.008255003951489925,
+            0.000503719667904079,
+            -0.006344534922391176,
+            -0.01958453096449375,
+            -0.004847260192036629,
+            0.022232530638575554,
+            0.0022259189281612635,
+            0.031011803075671196,
+            -0.003445509122684598,
+            0.008277218788862228,
+            0.016980960965156555,
+            -0.011489473283290863,
+            -0.0005217691650614142,
+            0.009312426671385765,
+            -0.0034566165413707495,
+            -0.0016960968496277928,
+            -0.004984991624951363,
+            0.00048372638411819935,
+            -0.009001419879496098,
+            3.720972017617896e-05,
+            0.0167232695966959,
+            0.011036291718482971,
+            0.0025458112359046936,
+            -0.005913568660616875,
+            -0.0017027612775564194,
+            -0.011844909749925137,
+            -0.006979877129197121,
+            -0.004487381316721439,
+            0.0017105364240705967,
+            -0.0011201791930943727,
+            0.01182713732123375,
+            0.00040319786057807505,
+            0.010218788869678974,
+            -0.010982976295053959,
+            -0.0011357294861227274,
+            -0.006548910867422819,
+            0.020419806241989136,
+            0.0053937435150146484,
+            -0.0043496498838067055,
+            0.007566346786916256,
+            -0.011027405969798565,
+            0.021379483863711357,
+            0.005291555542498827,
+            -0.00226479466073215,
+            0.0039075762033462524,
+            -0.0017560767009854317,
+            0.01149835903197527,
+            -0.0010962983360514045,
+            -0.013239996507763863,
+            0.02891472913324833,
+            0.017638517543673515,
+            0.027404125779867172,
+            0.0017405262915417552,
+            0.005735850892961025,
+            -0.00888146087527275,
+            0.027510756626725197,
+            0.01585245132446289,
+            -0.0017427478451281786,
+            -0.028150541707873344,
+            0.004243019036948681,
+            -0.012067057192325592,
+            -0.001978224376216531,
+            0.009614547714591026,
+            -0.01095631904900074,
+            -0.009552346542477608,
+            -0.01607459969818592,
+            -0.03138501197099686,
+            -0.026782112196087837,
+            0.0038164956495165825,
+            0.007068736013025045,
+            -0.008312762714922428,
+            0.0008847027784213424,
+            -0.015301525592803955,
+            -0.009543460793793201,
+            -0.025591401383280754,
+            0.002565804636105895,
+            -0.007313098758459091,
+            -0.037356339395046234,
+            -0.02381422184407711,
+            -0.009330199100077152,
+            -0.007166481111198664,
+            0.006611112039536238,
+            -0.0036898714024573565,
+            -0.02461395226418972,
+            0.011729392223060131,
+            -0.00974783580750227,
+            0.011542788706719875,
+            0.017700720578432083,
+            0.007535246200859547,
+            -0.010547567158937454,
+            -0.013559889048337936,
+            -0.009481258690357208,
+            -0.01628786139190197,
+            0.02052643708884716,
+            -0.010360963642597198,
+            0.006215689703822136,
+            -0.0058646961115300655,
+            -0.0018138349987566471,
+            0.0014184123137965798,
+            0.01832273229956627,
+            -0.03796058148145676,
+            -0.027190864086151123,
+            -0.00651781028136611,
+            0.011382842436432838,
+            0.0004651214985642582,
+            -0.012600211426615715,
+            0.014261875301599503,
+            -0.006406736560165882,
+            -0.02031317539513111,
+            -0.014519566670060158,
+            -0.001221811748109758,
+            -0.011054064147174358,
+            0.00840162206441164,
+            -0.011569445952773094,
+            0.01555921696126461,
+            0.0035299251321703196,
+            0.005349313840270042,
+            -0.00651781028136611,
+            0.015967968851327896,
+            0.008921447210013866,
+            -0.021343940868973732,
+            -0.003747629700228572,
+            -0.009463487192988396,
+            0.0017660732846707106,
+            0.01622566021978855,
+            0.007966212928295135,
+            -0.019815564155578613,
+            0.01738082803785801,
+            0.02687097154557705,
+            0.01315113715827465,
+            0.02118399366736412,
+            0.002925683744251728,
+            0.00752636045217514,
+            -0.001412858604453504,
+            0.011062949895858765,
+            -0.004811716731637716,
+            0.0002893472264986485,
+            0.00702874967828393,
+            -0.01628786139190197,
+            -0.011604989878833294,
+            0.0359523668885231,
+            0.000578694452997297,
+            -0.021219536662101746,
+            0.018153900280594826,
+            -0.005753622390329838,
+            -0.009076950140297413,
+            0.005900240037590265,
+            -0.0020581972785294056,
+            0.002081522950902581,
+            -0.010156587697565556,
+            0.009854466654360294,
+            -0.007961769588291645,
+            0.0057003069669008255,
+            0.006557796616107225,
+            0.004651770461350679,
+            -0.016190115362405777,
+            0.01658109575510025,
+            0.029092447832226753,
+            0.02616010047495365,
+            -0.042616792023181915,
+            -0.006753286812454462,
+            0.005424844101071358,
+            -0.018944745883345604,
+            -0.013675405643880367,
+            0.026017924770712852,
+            -0.0022559086792171,
+            -0.011658305302262306,
+            0.014039727859199047,
+            -0.009472372941672802,
+            -0.015434814617037773,
+            -0.0014961640117689967,
+            0.023938624188303947,
+            0.008623769506812096,
+            0.006402293220162392,
+            -0.007939554750919342,
+            0.005220468156039715,
+            -0.021077362820506096,
+            0.006242347415536642,
+            -0.006095729768276215,
+            0.009481258690357208,
+            -0.005349313840270042,
+            -0.005487045273184776,
+            0.012946761213243008,
+            0.005442616064101458,
+            -0.004847260192036629,
+            0.018589310348033905,
+            -0.00038292689714580774,
+            0.001768294838257134,
+            -0.012422492727637291,
+            0.0201532281935215,
+            -0.000900253071449697,
+            -0.009952211752533913,
+            -0.005424844101071358,
+            0.0197267048060894,
+            0.012511352077126503,
+            -0.0030878514517098665,
+            -0.0029145763255655766,
+            0.006055743433535099,
+            -3.049322731385473e-05,
+            0.000538985594175756,
+            -0.003632113104686141,
+            -0.008348306640982628,
+            -0.004402965307235718,
+            -0.007779608480632305,
+            0.015461471863090992,
+            -0.0028101669158786535,
+            -0.012742385268211365,
+            0.011427272111177444,
+            -0.003472166834399104,
+            -0.00031905947253108025,
+            -0.016607753932476044,
+            0.0011729393154382706,
+            0.005815824028104544,
+            0.026017924770712852,
+            0.012760157696902752,
+            -0.014519566670060158,
+            -0.025164877995848656,
+            -0.014484022743999958,
+            0.03270012512803078,
+            0.014377391897141933,
+            -0.004289669916033745,
+            0.01694541797041893,
+            0.005913568660616875,
+            0.021059591323137283,
+            0.005020535551011562,
+            -0.003063415177166462,
+            0.012102600187063217,
+            0.028132770210504532,
+            0.00480283098295331,
+            -0.01577247865498066,
+            -0.0042830053716897964,
+            -0.0017694055568426847,
+            -0.014128586277365685,
+            -0.008930332958698273,
+            -0.006495595443993807,
+            -0.02438291907310486,
+            0.01620788872241974,
+            -0.013497686944901943,
+            0.013666519895195961,
+            -0.014555109664797783,
+            -0.004509596154093742,
+            -0.027261951938271523,
+            0.002392529509961605,
+            0.021450571715831757,
+            -0.0035899050999432802,
+            0.012689069844782352,
+            -0.02738635428249836,
+            -0.020846329629421234,
+            0.019371269270777702,
+            -0.019922195002436638,
+            -0.00027823986602015793,
+            -0.0022559086792171,
+            -0.0034988245461136103,
+            -0.0051449378952383995,
+            -0.0010701960418373346,
+            -0.005375971551984549,
+            -0.0002896249061450362,
+            0.021432798355817795,
+            -0.004505153279751539,
+            0.0051671527326107025,
+            0.00011628037464106455,
+            0.015168237499892712,
+            0.014119700528681278,
+            -0.006544467993080616,
+            -0.007726293057203293,
+            -0.019762249663472176,
+            0.0037720659747719765,
+            -0.02249910682439804,
+            0.022321389988064766,
+            0.005011649336665869,
+            0.016403377056121826,
+            -0.007153152488172054,
+            -0.004263012204319239,
+            -0.006388964597135782,
+            -0.01047648023813963,
+            0.018660398200154305,
+            -0.010325419716536999,
+            0.007059850264340639,
+            -0.008277218788862228,
+            0.03403301164507866,
+            0.005269340705126524,
+            -0.013355513103306293,
+            0.02454286627471447,
+            0.005264897830784321,
+            0.011054064147174358,
+            -0.016741042956709862,
+            0.009427943266928196,
+            0.01022767461836338,
+            -0.016829900443553925,
+            0.01296453271061182,
+            -0.02921685017645359,
+            0.006251233164221048,
+            -0.024720583111047745,
+            -0.013071163557469845,
+            -0.017505230382084846,
+            -0.01914023607969284,
+            -0.04243907332420349,
+            -0.001449513016268611,
+            0.030691910535097122,
+            -0.005420401226729155,
+            0.017505230382084846,
+            -0.009738950058817863,
+            0.011373956687748432,
+            0.00821501761674881,
+            -0.005100508686155081,
+            0.020864101126790047,
+            0.02790173701941967,
+            -0.02052643708884716,
+            -0.006357864011079073,
+            0.0006603337242268026,
+            0.008983648382127285,
+            -0.006073514930903912,
+            -0.02921685017645359,
+            0.0050694081000983715,
+            -0.002747965743765235,
+            0.012644641101360321,
+            -0.01702539063990116,
+            0.0062778908759355545,
+            0.0007464158697985113,
+            0.011427272111177444,
+            -0.008339419960975647,
+            -0.014252989552915096,
+            -0.0180472694337368,
+            0.0016783250030130148,
+            0.0069976490922272205,
+            -0.022232530638575554,
+            0.016039054840803146,
+            -0.023903081193566322,
+            0.0014472915790975094,
+            0.015941310673952103,
+            -0.01956675946712494,
+            -0.006993206217885017,
+            -0.005438172724097967,
+            0.036929816007614136,
+            -0.0023414355237036943,
+            -0.002870146883651614,
+            0.011951540596783161,
+            0.0082194609567523,
+            0.010494251735508442,
+            -0.002934569725766778,
+            0.0051449378952383995,
+            0.0026280060410499573,
+            -0.004336320795118809,
+            0.0017427478451281786,
+            0.02761738747358322,
+            0.002912354888394475,
+            0.02125508151948452,
+            -0.0032966702710837126,
+            -0.002921240869909525,
+            0.0019349055364727974,
+            -0.007170923985540867,
+            0.009810036979615688,
+            7.4350020440761e-05,
+            -0.004203032236546278,
+            -0.017327511683106422,
+            0.015159351751208305,
+            -0.015168237499892712,
+            0.010121043771505356,
+            0.020473120734095573,
+            0.01788732409477234,
+            0.006771058309823275,
+            -0.004878360778093338,
+            -0.010547567158937454,
+            0.014928317628800869,
+            0.017078706994652748,
+            -0.00018424366135150194,
+            -0.020579751580953598,
+            0.0010801926255226135,
+            -0.00869485642760992,
+            0.003989770542830229,
+            -0.0116849634796381,
+            0.009516802616417408,
+            0.008805930614471436,
+            -0.003201146610081196,
+            0.015425928868353367,
+            0.008779272437095642,
+            0.014706170186400414,
+            0.01321333833038807,
+            -0.03317996487021446,
+            -0.0030145428609102964,
+            -0.008459379896521568,
+            -0.0020393149461597204,
+            -0.006384521722793579,
+            -0.017558544874191284,
+            -0.0004920568899251521,
+            0.016927646473050117,
+            -0.005993541795760393,
+            -0.006086844019591808,
+            0.01942458562552929,
+            -0.01420855987817049,
+            0.013853123411536217,
+            0.006180145777761936,
+            0.016305632889270782,
+            -0.02017100155353546,
+            0.003740965388715267,
+            -0.0012007077457383275,
+            -0.0037654016632586718,
+            -0.004905018489807844,
+            0.015665847808122635,
+            0.006224575452506542,
+            0.02475612796843052,
+            -0.02651553601026535,
+            0.006060186307877302,
+            -0.007779608480632305,
+            -0.011542788706719875,
+            -0.018011726438999176,
+            0.011009634472429752,
+            0.00021798232046421617,
+            -0.015372613444924355,
+            -0.01315113715827465,
+            0.006851031444966793,
+            0.011525017209351063,
+            -0.0014006405835971236,
+            0.013195566833019257,
+            0.009685634635388851,
+            0.0018615968292579055,
+            0.009738950058817863,
+            0.008605997078120708,
+            -0.02242802083492279,
+            -0.008059514686465263,
+            0.016554437577724457,
+            -0.0029056903440505266,
+            -0.010014412924647331,
+            0.018873659893870354,
+            0.006548910867422819,
+            -0.011880452744662762,
+            0.006891018245369196,
+            0.01701650582253933,
+            0.007988426834344864,
+            0.006011313758790493,
+            -0.007126494776457548,
+            0.003241133177652955,
+            0.002292563207447529,
+            0.0004895577440038323,
+            -0.020615296438336372,
+            -0.008530467748641968,
+            0.008921447210013866,
+            -0.01092077512294054,
+            0.008570454083383083,
+            -0.005282669793814421,
+            -0.023352153599262238,
+            0.01570139080286026,
+            0.0024325160775333643,
+            -0.008268333040177822,
+            -0.008015085011720657,
+            0.008090615272521973,
+            0.004487381316721439,
+            -0.008281662128865719,
+            0.0009963319171220064,
+            -0.0007880685734562576,
+            -0.012360291555523872,
+            0.022676825523376465,
+            0.00800175592303276,
+            0.0010085500543937087,
+            -0.006428950931876898,
+            -0.009116937406361103,
+            -0.0133643988519907,
+            0.031438328325748444,
+            0.013355513103306293,
+            0.023067805916070938,
+            0.01201374176889658,
+            -0.0033699788618832827,
+            -0.0062556760385632515,
+            -0.002807945478707552,
+            0.01559476088732481,
+            0.012111486867070198,
+            -0.00020451462478376925,
+            0.010831916704773903,
+            0.003245576284825802,
+            0.0037298579700291157,
+            0.006033528596162796,
+            0.005238240119069815,
+            -0.003241133177652955,
+            0.008139487355947495,
+            0.0016305632889270782,
+            0.004811716731637716,
+            0.002556918654590845,
+            0.011729392223060131,
+            0.02541368454694748,
+            -0.006544467993080616,
+            -0.008086171932518482,
+            0.01029876247048378,
+            0.012609097175300121,
+            0.004651770461350679,
+            -0.01569250598549843,
+            0.0066022262908518314,
+            -0.011391728185117245,
+            0.011276211589574814,
+            -0.02877255529165268,
+            -0.006877689156681299,
+            0.009952211752533913,
+            -0.021148450672626495,
+            0.03163381665945053,
+            -0.007419729605317116,
+            -0.024596180766820908,
+            0.007766279857605696,
+            -0.00941905751824379,
+            -0.013302197679877281,
+            -0.018429363146424294,
+            0.020402034744620323,
+            0.017505230382084846,
+            0.020419806241989136,
+            -0.012466922402381897,
+            0.01862485334277153,
+            0.004851703066378832,
+            0.022587966173887253,
+            -0.0037542942445725203,
+            -0.014439593069255352,
+            -0.010574225336313248,
+            0.013391056098043919,
+            0.009614547714591026,
+            -0.007495259400457144,
+            0.029465656727552414,
+            0.014368506148457527,
+            0.005838038399815559,
+            -0.0013339963043108582,
+            -0.00737529993057251,
+            0.004143052268773317,
+            -0.002876811195164919,
+            0.004416293930262327,
+            -0.009161366149783134,
+            -0.02111290581524372,
+            -0.0008302765781991184,
+            0.017265310510993004,
+            0.008619326166808605,
+            0.024933844804763794,
+            -0.015106036327779293,
+            0.0081839170306921,
+            0.002747965743765235,
+            -0.01044093631207943,
+            -0.02095296047627926,
+            0.002816831460222602,
+            -0.010201017372310162,
+            0.023192208260297775,
+            0.007468601688742638,
+            -0.007490816526114941,
+            -0.012022627517580986,
+            0.004451837856322527,
+            0.0034943814389407635,
+            0.024454006925225258,
+            -0.005944669246673584,
+            -0.01029876247048378,
+            0.01420855987817049,
+            0.004776173271238804,
+            0.0015372612979263067,
+            0.004038643091917038,
+            0.013506573624908924,
+            0.00668219942599535,
+            -0.0011362849036231637,
+            -0.001868261257186532,
+            0.0037298579700291157,
+            0.007544131949543953,
+            -0.0010435383301228285,
+            0.0011285097571089864,
+            0.001358432462438941,
+            0.004580683074891567,
+            0.012475808151066303,
+            -0.004098623059689999,
+            0.0027768448926508427,
+            -0.0041230591014027596,
+            -0.00912582315504551,
+            0.0014894995838403702,
+            0.0005156600964255631,
+            -0.012626868672668934,
+            -0.0009224678506143391,
+            -0.00912582315504551,
+            -0.0020126572344452143,
+            0.01417301595211029,
+            0.03367757424712181,
+            -0.010538681410253048,
+            0.022321389988064766,
+            -0.011720506474375725,
+            0.02146834321320057,
+            0.013373284600675106,
+            0.005469273775815964,
+            -0.02228584513068199,
+            -0.015088263899087906,
+            0.01405749935656786,
+            -0.017940638586878777,
+            0.004209696780890226,
+            -0.0052337972447276115,
+            -0.03024761565029621,
+            0.013053392060101032,
+            -0.03170490264892578,
+            -0.0030478648841381073,
+            0.005287112668156624,
+            -0.014332962222397327,
+            -0.009259111247956753,
+            -0.0031789320055395365,
+            -0.002585797803476453,
+            -0.0027368583250790834,
+            0.0064200651831924915,
+            -0.028026139363646507,
+            -0.0133999427780509,
+            0.005540360696613789,
+            0.009996641427278519,
+            -0.01620788872241974,
+            0.010538681410253048,
+            0.004589569289237261,
+            -0.0012617983156815171,
+            0.00462066987529397,
+            -0.018153900280594826,
+            -0.007033192552626133,
+            -0.02512933500111103,
+            -0.003809831105172634,
+            -0.00803729984909296,
+            0.015417042188346386,
+            -0.00045679096365347505,
+            -0.005953555461019278,
+            -0.0009407950565218925,
+            -0.011187352240085602,
+            0.0030834085773676634,
+            0.01055645290762186,
+            0.002639113226905465,
+            -0.008170587942004204,
+            0.01723865233361721,
+            0.02278345637023449,
+            -0.01300007663667202,
+            -0.025662489235401154,
+            0.01921132393181324,
+            -0.01577247865498066,
+            0.021574974060058594,
+            0.0005645325290970504,
+            -0.001508382149040699,
+            -0.0150793781504035,
+            0.015150465071201324,
+            0.0010563117684796453,
+            0.01216480229049921,
+            -0.014795029535889626,
+            0.02825717255473137,
+            -0.0010374292032793164,
+            0.003809831105172634,
+            -0.0073664141818881035,
+            -8.213386536226608e-06,
+            0.00888146087527275,
+            0.0019904423970729113,
+            -0.010369849391281605,
+            0.020508665591478348,
+            -0.012182573787868023,
+            0.0045473612844944,
+            -0.008446050807833672,
+            0.00020104357099626213,
+            -0.008974762633442879,
+            0.0034543948713690042,
+            -0.0017838451312854886,
+            0.012573553249239922,
+            0.012546896003186703,
+            0.015274868346750736,
+            -0.010023298673331738,
+            -0.008948104456067085,
+            0.013782036490738392,
+            -0.02498716115951538,
+            0.0013828687369823456,
+            0.0006231239531189203,
+            0.011951540596783161,
+            -0.02118399366736412,
+            0.006237904075533152,
+            0.026391133666038513,
+            0.005318213254213333,
+            0.009836695156991482,
+            -0.010716399177908897,
+            -0.018002839758992195,
+            -0.007815152406692505,
+            0.007917339913547039,
+            -0.014457364566624165,
+            -0.02059752494096756,
+            -0.013702062889933586,
+            -0.00020479230443015695,
+            -0.0021315061021596193,
+            0.004918347578495741,
+            -0.013630975969135761,
+            -0.002365871798247099,
+            -0.01664329692721367,
+            -0.0024502878077328205,
+            -0.014039727859199047,
+            -0.007042078301310539,
+            0.008090615272521973,
+            0.02855929359793663,
+            0.008077286183834076,
+            0.02402748353779316,
+            0.01237806398421526,
+            0.020917417481541634,
+            0.017834007740020752,
+            -0.004567354451864958,
+            -0.005695864092558622,
+            -0.0002508879406377673,
+            0.018660398200154305,
+            -0.03321550786495209,
+            0.01438627764582634,
+            -0.0077174073085188866,
+            -0.004138609394431114,
+            -0.0054914881475269794,
+            -0.0019304625457152724,
+            0.010796372778713703,
+            0.003976441919803619,
+            -0.004167488776147366,
+            0.010929660871624947,
+            -0.024596180766820908,
+            -0.007135380525141954,
+            -0.00735752796754241,
+            -0.01723865233361721,
+            -0.0013362177414819598,
+            0.02409856952726841,
+            -0.005424844101071358,
+            0.002994549460709095,
+            0.006984320003539324,
+            0.008437165059149265,
+            0.016492236405611038,
+            0.015808021649718285,
+            -0.0016438921447843313,
+            -0.0057669514790177345,
+            0.005398186389356852,
+            -0.004887246992439032,
+            -0.03081631287932396,
+            -0.006899903994053602,
+            0.008379407227039337,
+            0.011729392223060131,
+            -0.025289282202720642,
+            -0.011116265319287777,
+            0.0017338618636131287,
+            0.01540815643966198,
+            0.02081078663468361,
+            -0.0039297910407185555,
+            -0.016083484515547752,
+            -0.004147495608776808,
+            -0.020579751580953598,
+            0.021806007251143456,
+            -0.012804586440324783,
+            0.0005670317332260311,
+            -0.009232454001903534,
+            0.00939240027219057,
+            -0.012564667500555515,
+            0.03317996487021446,
+            -0.014590653590857983,
+            0.002401415491476655,
+            -0.0010685299057513475,
+            0.011098493821918964,
+            0.00476728705689311,
+            0.024134114384651184,
+            -0.01722976751625538,
+            -0.003796502249315381,
+            0.028683695942163467,
+            -0.033624257892370224,
+            0.016696613281965256,
+            0.008743729442358017,
+            0.01084080245345831,
+            -0.009303540922701359,
+            -0.008743729442358017,
+            0.027955051511526108,
+            0.020793013274669647,
+            0.006535581778734922,
+            -0.001412858604453504,
+            -0.019264638423919678,
+            0.023689819499850273,
+            -0.0011829358991235495,
+            0.002683542901650071,
+            -0.002794616622850299,
+            0.013453258201479912,
+            -0.034886058419942856,
+            -0.01657220907509327,
+            0.005562575533986092,
+            -0.008663755841553211,
+            -0.01892697438597679,
+            0.0005864696577191353,
+            -0.0116494195535779,
+            -0.02132616750895977,
+            -0.0032655694521963596,
+            0.01965561881661415,
+            -0.002612455515190959,
+            0.003072301158681512,
+            0.006686642300337553,
+            0.004411851055920124,
+            -0.008605997078120708,
+            -0.024525092914700508,
+            -0.01099186297506094,
+            0.003709864802658558,
+            0.007348642218858004,
+            -0.013017848134040833,
+            -0.012982305139303207,
+            -0.01077860128134489,
+            -0.0030100997537374496,
+            0.013471029698848724,
+            0.002305892063304782,
+            0.02168160490691662,
+            -0.004474052228033543,
+            0.01435073371976614,
+            0.013906438834965229,
+            -0.006851031444966793,
+            -0.008685970678925514,
+            0.013248882256448269,
+            -0.0027457440737634897,
+            -0.007504145614802837,
+            -0.007384185679256916,
+            -0.0016205665888264775,
+            -0.006584454327821732,
+            0.002288120100274682,
+            0.01956675946712494,
+            -0.01970893330872059,
+            0.004820602480322123,
+            0.0017249759985134006,
+            -0.01212925836443901,
+            0.005629220046103001,
+            0.008308319374918938,
+            -0.0011884896084666252,
+            0.011196237988770008,
+            0.003774287411943078,
+            -0.002202593255788088,
+            0.00226479466073215,
+            -0.013071163557469845,
+            0.006575568579137325,
+            0.015417042188346386,
+            0.014190787449479103,
+            0.01077860128134489,
+            0.006366749759763479,
+            -0.034299585968256,
+            -0.006415622308850288,
+            -0.003118952037766576,
+            -0.02374313399195671,
+            0.0014106371672824025,
+            0.0026280060410499573,
+            0.020259859040379524,
+            -0.003076744033023715,
+            0.014937203377485275,
+            -0.005882468074560165,
+            0.0006458940915763378,
+            0.016821015626192093,
+            0.009907782077789307,
+            -0.03573910519480705,
+            -0.010663083754479885,
+            -0.01523043867200613,
+            -0.007979541085660458,
+            -0.015177123248577118,
+            -0.014377391897141933,
+            0.008077286183834076,
+            0.017291968688368797,
+            -0.013648747466504574,
+            -0.012200345285236835,
+            0.011711620725691319,
+            0.013888667337596416,
+            0.007370857056230307,
+            0.006144602317363024,
+            0.016314519569277763,
+            -0.0018127242801710963,
+            0.0006347867310978472,
+            0.00026338372845202684,
+            0.018873659893870354,
+            -0.013026734814047813,
+            -0.0023392140865325928,
+            -0.011418386362493038,
+            0.011676076799631119,
+            0.020864101126790047,
+            -0.01076971460133791,
+            0.014448478817939758,
+            -0.007059850264340639,
+            -0.006779944524168968,
+            -0.047379638999700546,
+            -0.01321333833038807,
+            -0.009436829946935177,
+            0.028132770210504532,
+            -0.004807273857295513,
+            -0.005833595525473356,
+            0.0055803474970161915,
+            0.019602302461862564,
+            0.0063001057133078575,
+            0.008166145533323288,
+            0.01818055845797062,
+            -0.00839273538440466,
+            -0.012804586440324783,
+            0.004160824231803417,
+            0.007859582081437111,
+            -0.00927688367664814,
+            0.0027546300552785397,
+            0.0099344402551651,
+            0.032753441482782364,
+            -0.0057669514790177345,
+            0.0075085884891450405,
+            -0.009108050726354122,
+            0.0022303618025034666,
+            -0.016758814454078674,
+            0.001700539723969996,
+            -0.005473716650158167,
+            -0.006815487984567881,
+            0.014439593069255352,
+            -0.021788235753774643,
+            -0.0014795028837397695,
+            -0.0028968045953661203,
+            0.008406064473092556,
+            -0.01318668108433485,
+            0.011045178398489952,
+            -0.0041230591014027596,
+            -0.004271897953003645,
+            0.014128586277365685,
+            -0.011915996670722961,
+            0.027581844478845596,
+            -0.002074858406558633,
+            0.009969983249902725,
+            -0.007970655336976051,
+            -0.012866788543760777,
+            -0.017807351425290108,
+            0.0025369254872202873,
+            -0.0021348383743315935,
+            -0.008534910157322884,
+            0.008850360289216042,
+            -0.022463563829660416,
+            0.0036543277092278004,
+            -0.004873917903751135,
+            0.0015339291421696544,
+            -0.02592906542122364,
+            0.00015036614786367863,
+            -0.002412522677332163,
+            0.011080721393227577,
+            -0.008810373023152351,
+            -0.013160022906959057,
+            0.01176493614912033,
+            -0.010734171606600285,
+            -0.013888667337596416,
+            -0.01318668108433485,
+            0.008410507813096046,
+            0.004038643091917038,
+            -0.002434737514704466,
+            0.014457364566624165,
+            -0.011009634472429752,
+            0.0008480484248138964,
+            -0.006184588652104139,
+            0.00924133975058794,
+            -0.00047900574281811714,
+            -0.016989847645163536,
+            -4.682454527937807e-05,
+            -0.0002667159424163401,
+            0.012440265156328678,
+            0.012982305139303207,
+            0.018447136506438255,
+            -0.008441608399152756,
+            0.0026168986223638058,
+            0.0034766097087413073,
+            -0.0037809519562870264,
+            0.011187352240085602,
+            0.005775837227702141,
+            -0.013888667337596416,
+            0.0013650968903675675,
+            -0.013684291392564774,
+            0.00854379590600729,
+            -0.00891256146132946,
+            0.010209903120994568,
+            0.009108050726354122,
+            -0.0024769455194473267,
+            -0.010343191213905811,
+            0.011729392223060131,
+            0.01657220907509327,
+            -0.012333634309470654,
+            0.00032655693939886987,
+            -0.007228682283312082,
+            0.025431456044316292,
+            0.016749927774071693,
+            0.010583111084997654,
+            -0.013355513103306293,
+            0.013826466165482998,
+            -0.00025310940691269934,
+            0.0037254150956869125,
+            0.00957011803984642,
+            -0.026408905163407326,
+            -0.02491607330739498,
+            0.003736522514373064,
+            -0.0069310045801103115,
+            -0.0060468572191894054,
+            0.023316610604524612,
+            -0.012138144113123417,
+            0.006184588652104139,
+            -0.011569445952773094,
+            0.028168313205242157,
+            0.01621677353978157,
+            -0.0009307984146289527,
+            -0.015834679827094078,
+            0.0002776844776235521,
+            0.01011215802282095,
+            -0.0013417714508250356,
+            -0.007766279857605696,
+            -0.0022681269329041243,
+            -0.012538010254502296,
+            0.004398522432893515,
+            -0.003783173393458128,
+            0.004776173271238804,
+            0.004402965307235718,
+            -0.012697956524789333,
+            -0.011311755515635014,
+            0.028221629559993744,
+            0.0008724846411496401,
+            0.012973419390618801,
+            -0.004971663001924753,
+            0.0017494122730568051,
+            0.005957998335361481,
+            0.02233916148543358,
+            -0.024276288226246834,
+            0.01905137673020363,
+            0.006739957723766565,
+            0.017771806567907333,
+            0.008903675712645054,
+            -0.00839717872440815,
+            0.01249358057975769,
+            -0.006157930940389633,
+            -0.007415286265313625,
+            -0.010280990041792393,
+            -0.0014750600093975663,
+            -0.0016050162957981229,
+            -0.0076285479590296745,
+            0.009579003788530827,
+            -0.0011290650581941009,
+            -0.004905018489807844,
+            0.01420855987817049,
+            -0.022890087217092514,
+            0.014723941683769226,
+            -0.0013239996042102575,
+            0.01124955341219902,
+            -0.0021781569812446833,
+            0.011125151067972183,
+            -0.02278345637023449,
+            0.0038675894029438496,
+            -0.01797618344426155,
+            -0.0012118150480091572,
+            -0.0040941801853477955,
+            -0.001176271471194923,
+            -0.0030323145911097527,
+            0.007979541085660458,
+            -0.034139640629291534,
+            -0.010067728348076344,
+            0.015488130040466785,
+            0.013933097012341022,
+            0.0026768783573061228,
+            0.013026734814047813,
+            0.006246790289878845,
+            0.0012362513225525618,
+            -0.00615348806604743,
+            -0.009294655174016953,
+            -0.020259859040379524,
+            -0.02219698764383793,
+            -0.013053392060101032,
+            -0.0022725698072463274,
+            -0.0008230567909777164,
+            -0.005242682993412018,
+            0.008286104537546635,
+            -0.018287189304828644,
+            -0.0007325316546484828,
+            -0.0005164931644685566,
+            0.0004479050694499165,
+            0.001358432462438941,
+            0.0018538215663284063,
+            0.016341175884008408,
+            0.008765943348407745,
+            -0.0042363544926047325,
+            -0.007370857056230307,
+            0.025182651355862617,
+            -0.0030345360282808542,
+            0.012147029861807823,
+            0.020028825849294662,
+            0.0025080463383346796,
+            -0.014688398689031601,
+            0.013835351914167404,
+            -0.008290547877550125,
+            0.018642624840140343,
+            -0.012333634309470654,
+            -0.025680260732769966,
+            -0.014857230708003044,
+            0.007610776461660862,
+            -0.011871566995978355,
+            -0.009596775285899639,
+            0.01365763321518898,
+            -0.002821274334564805,
+            0.0001021878924802877,
+            0.007135380525141954,
+            -0.005762508604675531,
+            0.0025369254872202873,
+            -0.03497491776943207,
+            -0.000194934502360411,
+            -0.03001658245921135,
+            -0.030567508190870285,
+            0.014670626260340214,
+            0.006855474319308996,
+            0.021077362820506096,
+            -0.006948776543140411,
+            0.013977525755763054,
+            -0.009099164977669716,
+            0.026053469628095627,
+            0.014946090057492256,
+            -0.0006936558638699353,
+            0.0031789320055395365,
+            -0.007250897120684385,
+            -0.006371192634105682,
+            0.027155321091413498,
+            -0.008081729523837566,
+            -0.0059313406236469746,
+            0.02475612796843052,
+            0.026657709851861,
+            0.01738082803785801,
+            -0.018944745883345604,
+            0.019460128620266914,
+            0.0013017848832532763,
+            0.01789620891213417,
+            -0.016199002042412758,
+            -0.00032850075513124466,
+            -0.015665847808122635,
+            -0.008161702193319798,
+            0.05530586466193199,
+            0.01723865233361721,
+            -0.0007297547999769449,
+            -0.0202243160456419,
+            0.015017176978290081,
+            -0.0029789991676807404,
+            -0.002028207527473569,
+            0.00981892365962267,
+            0.0302298441529274,
+            -0.00920579582452774,
+            -0.013799807988107204,
+            0.013293311931192875,
+            -0.005247125867754221,
+            0.026746569201350212,
+            -0.002260351786389947,
+            0.0020904086995869875,
+            -0.015221552923321724,
+            -0.013133365660905838,
+            0.0013395498972386122,
+            0.009756721556186676,
+            -0.007939554750919342,
+            -0.0011018520453944802,
+            -0.006944333668798208,
+            0.012769043445587158,
+            0.006815487984567881,
+            -0.020277632400393486,
+            -0.003112287726253271,
+            -0.030922943726181984,
+            -0.00480283098295331,
+            -0.035525843501091,
+            -0.005944669246673584,
+            -0.005269340705126524,
+            -0.014626197516918182,
+            -0.032611265778541565,
+            -0.02073969878256321,
+            -0.009996641427278519,
+            0.004660656210035086,
+            0.002425851533189416,
+            0.010041071102023125,
+            0.002474724082276225,
+            0.0005764730158261955,
+            0.0010274326195940375,
+            0.0041230591014027596,
+            0.01303562056273222,
+            0.001622788142412901,
+            0.011729392223060131,
+            0.02233916148543358,
+            -0.018198329955339432,
+            -0.033410996198654175,
+            -0.014484022743999958,
+            0.0050249784253537655,
+            0.005686978343874216,
+            -0.0018749256851151586,
+            0.008601554669439793,
+            0.0015561438631266356,
+            0.011231781914830208,
+            -0.014324076473712921,
+            -0.005020535551011562,
+            -0.009774493984878063,
+            -0.0011773821897804737,
+            0.0026191200595349073,
+            0.003896468784660101,
+            0.0026946503203362226,
+            0.0029412340372800827,
+            -0.011711620725691319,
+            0.01014770194888115,
+            -0.005380414426326752,
+            -0.0009291322785429657,
+            -0.0009024745668284595,
+            -0.012538010254502296,
+            0.004927233327180147,
+            -0.007935111410915852,
+            -0.00753968907520175,
+            -0.008534910157322884,
+            0.0034521734341979027,
+            -0.01600351184606552,
+            -0.009036963805556297,
+            0.01716756634414196,
+            -0.006424508057534695,
+            0.007757393643260002,
+            0.012831244617700577,
+            -0.0013650968903675675,
+            -0.004562911577522755,
+            0.019229095429182053,
+            -0.012742385268211365,
+            -0.01417301595211029,
+            -0.024454006925225258,
+            -0.006446722894906998,
+            0.0005456500221043825,
+            0.015452586114406586,
+            0.0042830053716897964,
+            -0.010903003625571728,
+            -0.0034810525830835104,
+            -0.002579133491963148,
+            0.003554361406713724,
+            -0.012324748560786247,
+            0.005966884084045887,
+            -0.02388530783355236,
+            0.015959082171320915,
+            -0.006677756551653147,
+            -0.009303540922701359,
+            0.018713712692260742,
+            0.04535365104675293,
+            -0.007606333587318659,
+            -0.0177629217505455,
+            0.009014748968183994,
+            0.022836772724986076,
+            0.022676825523376465,
+            -0.007086507976055145,
+            0.013053392060101032,
+            -0.02001105435192585,
+            -0.011880452744662762,
+            -0.005082736723124981,
+            -0.002314777811989188,
+            0.022516880184412003,
+            -0.03104734607040882,
+            -0.019246866926550865,
+            -0.007784051354974508,
+            -0.03241577744483948,
+            -0.006948776543140411,
+            0.028470434248447418,
+            0.004611784126609564,
+            -0.005620333831757307,
+            -0.004143052268773317,
+            -0.002716864924877882,
+            -0.01753188669681549,
+            0.020259859040379524,
+            0.08686859160661697,
+            -0.014635083265602589,
+            -0.008294991217553616,
+            -0.004131945315748453,
+            0.00635342113673687,
+            0.005784722976386547,
+            -0.005056079011410475,
+            0.06181034445762634,
+            -0.023121120408177376,
+            0.00033433211501687765,
+            0.003609898267313838,
+            0.015816908329725266,
+            -0.007170923985540867,
+            -0.0002644944761414081,
+            0.0012107043294236064,
+            -0.005953555461019278,
+            -0.010129929520189762,
+            -0.017105363309383392,
+            -0.0045295893214643,
+            -0.0017949525499716401,
+            -0.00701097771525383,
+            -0.01620788872241974,
+            0.019335726276040077,
+            -0.023849764838814735,
+            -0.006615554913878441,
+            -0.0023347712121903896,
+            -0.006513367407023907,
+            0.009188024327158928,
+            -0.019833335652947426,
+            0.002570247510448098,
+            -0.0011773821897804737,
+            0.005380414426326752,
+            -0.005158266983926296,
+            -0.005575904622673988,
+            -0.0028234957717359066,
+            0.0009313537739217281,
+            -0.0098988963291049,
+            0.01826941780745983,
+            0.013790922239422798,
+            0.009152480401098728,
+            0.010058842599391937,
+            -0.0055803474970161915,
+            0.023121120408177376,
+            -0.005784722976386547,
+            0.0057891663163900375,
+            -0.0067888302728533745,
+            0.0014161908766254783,
+            -0.004789501894265413,
+            0.01839382015168667,
+            -0.020633067935705185,
+            -0.013071163557469845,
+            -0.014875002205371857,
+            0.035170406103134155,
+            -0.02097073197364807,
+            0.014972747303545475,
+            0.0010307647753506899,
+            0.011356184259057045,
+            0.010485365986824036,
+            0.0013051170390099287,
+            0.01702539063990116,
+            -0.012093714438378811,
+            0.016776585951447487,
+            0.002699093194678426,
+            0.015363726764917374,
+            -0.008255003951489925,
+            0.012573553249239922,
+            0.00010274325904902071,
+            0.003809831105172634,
+            -0.008637098595499992,
+            0.0044696093536913395,
+            0.011676076799631119,
+            -0.005780280102044344,
+            0.03541921079158783,
+            -0.0013728720368817449,
+            -0.02226807363331318,
+            0.016741042956709862,
+            -0.015106036327779293,
+            -0.02082855813205242,
+            0.0011740500340238214,
+            -0.014652854762971401,
+            -0.0030278717167675495,
+            0.012315861880779266,
+            -0.028381574898958206,
+            -0.020988503471016884,
+            0.005247125867754221,
+            0.001098519773222506,
+            0.0016572210006415844,
+            0.0006486709462478757,
+            -0.014439593069255352,
+            -0.0018138349987566471,
+            -0.0030656366143375635,
+            0.007957326248288155,
+            -0.014777257107198238,
+            0.006437837146222591,
+            -0.026266731321811676,
+            -0.0031389454379677773,
+            -0.003132280893623829,
+            0.023689819499850273,
+            -0.002907912014052272,
+            0.010432050563395023,
+            -0.005993541795760393,
+            -0.0025502543430775404,
+            -0.016252316534519196,
+            -0.0033522071316838264,
+            -0.003258905140683055,
+            0.018518222495913506,
+            0.026746569201350212,
+            -0.0016761035658419132,
+            -0.0025235966313630342,
+            -6.3693878473714e-05,
+            0.015461471863090992,
+            -0.00858378317207098,
+            -0.01022767461836338,
+            0.01666106842458248,
+            0.014404049143195152,
+            -0.016554437577724457,
+            0.011862681247293949,
+            -0.0022125900723040104,
+            0.011658305302262306,
+            -0.012253660708665848,
+            0.016536666080355644,
+            0.020490894094109535,
+            -0.005282669793814421,
+            0.019193550571799278,
+            -0.01095631904900074,
+            0.015390384942293167,
+            0.018731484189629555,
+            0.006948776543140411,
+            0.014155244454741478,
+            0.012662412598729134,
+            -0.01180048007518053,
+            0.0005836928030475974,
+            -0.005758065264672041,
+            -0.005322656128555536,
+            0.005127166397869587,
+            -0.002612455515190959,
+            -0.019406812265515327,
+            0.0036121197044849396,
+            0.0012462479062378407,
+            0.008508252911269665,
+            0.0034899385645985603,
+            -0.00617126002907753,
+            -0.0008730400004424155,
+            0.013462143950164318,
+            -0.02095296047627926,
+            -0.006077957805246115,
+            0.023938624188303947,
+            -0.006015756633132696,
+            -0.005673649255186319,
+            0.0011407278943806887,
+            0.012093714438378811,
+            -0.01092077512294054,
+            -0.01651889458298683,
+            -0.020508665591478348,
+            0.02841711975634098,
+            0.004509596154093742,
+            0.00887701753526926,
+            0.006762172561138868,
+            -0.008610440418124199,
+            -0.002639113226905465,
+            -0.015950197353959084,
+            -0.001235140603967011,
+            0.018233874812722206,
+            0.01723865233361721,
+            0.011027405969798565,
+            -0.019780021160840988,
+            0.008317206054925919,
+            -0.008486038073897362,
+            3.512708644848317e-05,
+            0.014004183933138847,
+            0.01622566021978855,
+            0.0063223205506801605,
+            -0.0027079791761934757,
+            0.016989847645163536,
+            0.001412858604453504,
+            0.029678918421268463,
+            -0.0006458940915763378,
+            -0.01767406240105629,
+            0.0003993102873209864,
+            0.001091300044208765,
+            -0.01620788872241974,
+            -0.003932012245059013,
+            0.0024769455194473267,
+            0.038813624531030655,
+            -0.008139487355947495,
+            0.007148709148168564,
+            -0.003058972302824259,
+            -0.012422492727637291,
+            0.0025724689476191998,
+            -0.0015883552841842175,
+            -0.002055975841358304,
+            0.02177046425640583,
+            -0.017567431554198265,
+            0.012004856020212173,
+            -0.012591325677931309,
+            0.0056780921295285225,
+            0.0012740164529532194,
+            0.007761836517602205,
+            -0.011356184259057045,
+            -0.03067413903772831,
+            -0.013790922239422798,
+            0.02635558880865574,
+            0.018642624840140343,
+            -0.007135380525141954,
+            0.018145015463232994,
+            -0.014821686781942844,
+            -0.00941905751824379,
+            0.008303876966238022,
+            -0.010982976295053959,
+            0.0009352413471788168,
+            0.007233125623315573
+          ],
+          "keyphrases": [
+            "1 Billion Word Language Modeling Benchmark",
+            "Long Short-Term Memory (LSTM) layers",
+            "Mixture of Experts (MoE) models",
+            "computational efficiency",
+            "Machine Translation"
+          ],
+          "title": "5 Experiments 5.1 1 Billion Word Language Modeling Benchmark"
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "a004def7-e6ff-4a78-ad92-424e0eff3d92",
+        "properties": {
+          "page_content": "6 Conclusion\n\nThis work is the first to demonstrate major wins from conditional computation in deep networks.\n\nWe carefully identified the design considerations and challenges of conditional computing and addressed them with a combination of algorithmic and engineering solutions. While we focused on text, conditional computation may help in other domains as well, provided sufficiently large training sets. We look forward to seeing many novel implementations and applications of conditional computation in the years to come.\n\nAcknowledgments\n\nWe would like to thank all of the members of the Google Brain and Google Translate teams who helped us with this project, in particular Zhifeng Chen, Yonghui Wu, and Melvin Johnson. Thanks also to our anonymous ICLR reviewers for the helpful suggestions on making this paper better.\n\n2Reported perplexities relative to the tokenization used by both our models and GNMT.\n\nReferences\n\nMartín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Gordon Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda B. Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. CoRR, abs/1603.04467, 2016. URL http://arxiv.org/abs/1603.04467.\n\nRahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. CoRR, abs/1611.06194, 2016. URL http://arxiv.org/abs/1611.\n\n06194.\n\nA. Almahairi, N. Ballas, T. Cooijmans, Y. Zheng, H. Larochelle, and A. Courville. Dynamic Capacity Networks. ArXiv e-prints, November 2015.\n\nDario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan, Christopher Fougner, Tony Han, Awni Y. Hannun, Billy Jun, Patrick LeGresley, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595, 2015. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.\n\nEmmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297, 2015.\n\nYoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.\n\nCiprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling.\n\narXiv preprint arXiv:1312.3005, 2013.\n\nK. Cho and Y. Bengio. Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning. ArXiv e-prints, June 2014. Ronan Collobert, Samy Bengio, and Yoshua Bengio. A parallel mixture of SVMs for very large scale problems. Neural Computing, 2002.\n\nAndrew Davis and Itamar Arel. Low-rank approximations for conditional feedforward computation in deep neural networks. arXiv preprint arXiv:1312.4461, 2013. Marc Peter Deisenroth and Jun Wei Ng. Distributed Gaussian processes. In ICML, 2015.\n\nJohn Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization, 2010.\n\nNadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. Edinburgh's phrase-based machine translation systems for wmt-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation, 2014.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013. Ekaterina Garmash and Christof Monz. Ensemble learning for multi-source neural machine translation. In staff.science.uva.nl/c.monz, 2016.\n\nFelix A. Gers, Jürgen A. Schmidhuber, and Fred A. Cummins. Learning to forget: Continual prediction with lstm. Neural Computation, 2000.\n\nAudrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. Memory-efficient backpropagation through time. CoRR, abs/1606.03401, 2016. URL http://arxiv.org/ abs/1606.03401. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2015.\n\nGeoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 2012. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.\n\nSergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.\n\nRobert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computing, 1991.\n\nMelvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation.\n\nCoRR, abs/1611.04558, 2016. URL http://arxiv.org/abs/1611.04558.\n\nMichael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm.\n\nNeural Computing, 1994. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.\n\nDiederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.\n\nReinhard Kneser and Hermann. Ney. Improved backingoff for m-gram language modeling., 1995.\n\nAlex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.\n\nQuoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeffrey Dean, and Andrew Y. Ng. Building high-level features using large scale unsupervised learning. In ICML, 2012. Patrick Gallinari Ludovic Denoyer. Deep sequential neural network. arXiv preprint arXiv:1410.0510, 2014.\n\nMinh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attentionbased neural machine translation. EMNLP, 2015a.\n\nMinh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the rare word problem in neural machine translation. ACL, 2015b.\n\nCarl Edward Rasmussen and Zoubin Ghahramani. Infinite mixtures of Gaussian process experts.\n\nNIPS, 2002.\n\nHasim Sak, Andrew W Senior, and Françoise Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH, pp. 338–342, 2014.\n\nMike Schuster and Kaisuke Nakajima. Japanese and Korean voice search. ICASSP, 2012.\n\nBabak Shahbaba and Radford Neal. Nonlinear models using dirichlet process mixtures. JMLR, 2009.\n\nIlya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks.\n\nIn NIPS, 2014.\n\nLucas Theis and Matthias Bethge. Generative image modeling using spatial LSTMs. In NIPS, 2015. Volker Tresp. Mixtures of Gaussian Processes. In NIPS, 2001.\n\nYonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.\n\nBangpeng Yao, Dirk Walther, Diane Beck, and Li Fei-fei. Hierarchical mixture of classification experts uncovers interactions between brain regions. In NIPS. 2009.\n\nWojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization.\n\narXiv preprint arXiv:1409.2329, 2014.\n\nJie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with fast-forward connections for neural machine translation. arXiv preprint arXiv:1606.04199, 2016.\n\n",
+          "embedding": [
+            0.012222016230225563,
+            0.0018153652781620622,
+            -0.018968041986227036,
+            -0.007018175441771746,
+            0.012403449974954128,
+            0.003484346903860569,
+            0.006766642909497023,
+            0.03404351323843002,
+            -0.030365360900759697,
+            0.03598979860544205,
+            0.007698550820350647,
+            -0.006844989024102688,
+            0.017252670601010323,
+            -0.018192825838923454,
+            0.0035915574990212917,
+            0.014209537766873837,
+            0.014687862247228622,
+            0.004014214966446161,
+            -0.011182897724211216,
+            -0.040674079209566116,
+            -0.03125603497028351,
+            -0.017648525536060333,
+            -0.03161890059709549,
+            -0.009582984261214733,
+            0.016172317788004875,
+            0.03310335800051689,
+            0.022712169215083122,
+            0.017153708264231682,
+            -0.045556288212537766,
+            0.015842437744140625,
+            0.028155172243714333,
+            0.003665780182927847,
+            -0.02747892029583454,
+            -0.023569853976368904,
+            -0.015380607917904854,
+            -0.03267451375722885,
+            0.06508512794971466,
+            0.03876078128814697,
+            -0.014654873870313168,
+            0.019149474799633026,
+            0.04321414604783058,
+            -0.011900384910404682,
+            -0.007348054554313421,
+            -0.015990883111953735,
+            -0.004210080485790968,
+            -0.002177201444283128,
+            -0.014085832983255386,
+            -0.015174433588981628,
+            -0.030051976442337036,
+            -0.007912972010672092,
+            0.028105689212679863,
+            -0.028353098779916763,
+            0.04743659868836403,
+            -0.003750311676412821,
+            -0.016485702246427536,
+            0.04671086370944977,
+            0.010762302204966545,
+            0.008997449651360512,
+            0.06310585141181946,
+            0.04139981418848038,
+            0.003208073088899255,
+            0.018192825838923454,
+            -0.023899732157588005,
+            -0.006424393504858017,
+            0.0322621650993824,
+            -0.02089783363044262,
+            0.0017978404648602009,
+            0.025433670729398727,
+            0.039783407002687454,
+            -0.015347619540989399,
+            -0.03783712163567543,
+            0.029045844450592995,
+            -0.010498398914933205,
+            -0.015957895666360855,
+            -0.00029766425723209977,
+            -0.0036699038464576006,
+            0.029029350727796555,
+            0.05367131158709526,
+            0.005014160647988319,
+            0.004304920788854361,
+            0.012782811187207699,
+            0.017219683155417442,
+            -0.004572947509586811,
+            -0.006494492758065462,
+            0.023157505318522453,
+            0.002239053603261709,
+            -0.005183223634958267,
+            -0.05383625254034996,
+            0.03328479081392288,
+            -0.04740361124277115,
+            -0.039816394448280334,
+            0.018225813284516335,
+            0.006473875138908625,
+            -0.01018501352518797,
+            -0.009624219499528408,
+            0.029656121507287025,
+            -0.00020127772586420178,
+            -0.000498168810736388,
+            -0.018918560817837715,
+            0.014357983134686947,
+            -0.023784274235367775,
+            0.00528631079941988,
+            0.013285876251757145,
+            -0.013665237464010715,
+            0.027577882632613182,
+            0.009541749954223633,
+            -0.013541532680392265,
+            0.004121425561606884,
+            0.01654343120753765,
+            0.09295989573001862,
+            -0.0011030328460037708,
+            -0.022530734539031982,
+            -0.05314350500702858,
+            0.03267451375722885,
+            -0.02602745220065117,
+            -0.019512342289090157,
+            -0.051791001111269,
+            -0.039882369339466095,
+            0.0015978513984009624,
+            -0.00780988484621048,
+            0.03721034899353981,
+            0.03533003851771355,
+            0.034241437911987305,
+            -0.011974607594311237,
+            -0.029342735186219215,
+            0.055914487689733505,
+            -0.002408116590231657,
+            -0.0425214022397995,
+            -0.0034740380942821503,
+            -0.023965708911418915,
+            -0.007587216794490814,
+            0.008593347854912281,
+            -0.040575116872787476,
+            0.003964733332395554,
+            0.03285594657063484,
+            -0.018638163805007935,
+            0.029342735186219215,
+            0.03582485765218735,
+            -0.025417177006602287,
+            -0.0004953339230269194,
+            0.011232379823923111,
+            -0.0033709509298205376,
+            0.016733111813664436,
+            -0.01296424400061369,
+            -0.034208450466394424,
+            -0.0008540773415006697,
+            -0.015990883111953735,
+            0.0541001558303833,
+            -0.009459280408918858,
+            0.030942648649215698,
+            -0.03885974362492561,
+            -0.0009102598414756358,
+            -0.014267266727983952,
+            0.034538328647613525,
+            0.029606638476252556,
+            -0.009046930819749832,
+            0.0019669034518301487,
+            0.009615972638130188,
+            -0.009541749954223633,
+            0.010201508179306984,
+            -0.00514198886230588,
+            -0.03315283730626106,
+            -0.007966578006744385,
+            0.03757321834564209,
+            0.009756171144545078,
+            0.04097096994519234,
+            0.04737062379717827,
+            0.008750040084123611,
+            -0.04882209002971649,
+            0.04700775444507599,
+            0.034439366310834885,
+            0.01601562462747097,
+            0.001114372513256967,
+            -0.018291790038347244,
+            0.01057262159883976,
+            0.01364049594849348,
+            0.033878572285175323,
+            0.009921110235154629,
+            -0.040179260075092316,
+            0.024790406227111816,
+            -0.0025256359949707985,
+            0.010119037702679634,
+            0.04885507747530937,
+            0.02596147730946541,
+            0.040179260075092316,
+            0.003092615632340312,
+            0.029161302372813225,
+            0.028204653412103653,
+            0.029260266572237015,
+            -0.007599587086588144,
+            0.008585100993514061,
+            0.02132667601108551,
+            0.045655250549316406,
+            -0.034769244492053986,
+            0.029903531074523926,
+            -0.039354562759399414,
+            0.0009916987037286162,
+            0.028649989515542984,
+            -0.024526502937078476,
+            0.004700775723904371,
+            0.00834593828767538,
+            -0.055947475135326385,
+            -0.009632466360926628,
+            0.005344039760529995,
+            0.00844490248709917,
+            0.007591340225189924,
+            -0.012667353264987469,
+            0.004255439154803753,
+            0.0053234221413731575,
+            -0.040674079209566116,
+            -0.006354294251650572,
+            0.004333785269409418,
+            -0.003735879436135292,
+            -0.017829960212111473,
+            0.012560142204165459,
+            -0.004383266903460026,
+            -0.003599804360419512,
+            -0.005364656914025545,
+            -0.025631597265601158,
+            -0.009005696512758732,
+            -0.01664239540696144,
+            0.020403016358613968,
+            0.04215853661298752,
+            -0.024856381118297577,
+            -0.016320763155817986,
+            -0.017928922548890114,
+            -0.03556095436215401,
+            -0.030117951333522797,
+            0.017566056922078133,
+            0.00014135829405859113,
+            0.03688047081232071,
+            0.02172253094613552,
+            -0.0031750851776450872,
+            0.004581194370985031,
+            0.02591199427843094,
+            -0.017269164323806763,
+            -0.0021524603944271803,
+            0.004577070940285921,
+            -0.007756279781460762,
+            0.0136487428098917,
+            -0.003177147125825286,
+            0.01680733449757099,
+            -0.00515848258510232,
+            -0.00132157769985497,
+            -0.001624654047191143,
+            -0.01583419181406498,
+            -0.017599044367671013,
+            0.01364049594849348,
+            -0.0008695403812453151,
+            0.010069556534290314,
+            -0.018935054540634155,
+            -0.04262036457657814,
+            0.013896152377128601,
+            -0.010003580711781979,
+            -0.05927925556898117,
+            0.06967044621706009,
+            0.02676968090236187,
+            -0.014621886424720287,
+            -0.00333796301856637,
+            -0.050471484661102295,
+            0.05802571401000023,
+            0.004453366156667471,
+            -0.02221735008060932,
+            0.05353936180472374,
+            -0.007117139175534248,
+            0.03829894959926605,
+            0.05241777002811432,
+            0.022827625274658203,
+            0.010044815018773079,
+            -0.02714904025197029,
+            0.05268167331814766,
+            -0.04205957055091858,
+            -0.00666767917573452,
+            -0.034670282155275345,
+            0.016881557181477547,
+            -0.04278530552983284,
+            -0.004498724825680256,
+            0.018324777483940125,
+            0.010877760127186775,
+            0.03592382371425629,
+            0.023619335144758224,
+            0.015908414497971535,
+            -0.03145396336913109,
+            -0.004395637661218643,
+            -0.00642026960849762,
+            -0.013879658654332161,
+            -0.03737528994679451,
+            -0.0037626822013407946,
+            -0.06696543842554092,
+            -0.004630676470696926,
+            -0.014811567030847073,
+            0.01346730999648571,
+            -0.05264868587255478,
+            0.029458193108439445,
+            0.018291790038347244,
+            0.020419510081410408,
+            0.007104768883436918,
+            -0.024180129170417786,
+            0.019281426444649696,
+            0.011446801014244556,
+            -0.017186695709824562,
+            -0.040113285183906555,
+            0.004412131384015083,
+            -0.01324464101344347,
+            -0.022745156660676003,
+            0.0018205196829512715,
+            0.019842220470309258,
+            0.029606638476252556,
+            0.005950192455202341,
+            0.004560577217489481,
+            -0.0032225053291767836,
+            0.017516573891043663,
+            0.03180033341050148,
+            -0.019693775102496147,
+            -0.03373012691736221,
+            0.040014322847127914,
+            0.025747055187821388,
+            0.003550322726368904,
+            -0.017120718955993652,
+            -0.03123953938484192,
+            0.051164232194423676,
+            -0.0006412022630684078,
+            -0.06333676725625992,
+            0.012329227291047573,
+            -0.006073896773159504,
+            0.002081330167129636,
+            -0.0012401387793943286,
+            0.05159307271242142,
+            0.002733872039243579,
+            0.014935270883142948,
+            0.0019091746071353555,
+            -0.050471484661102295,
+            -0.018522705882787704,
+            -0.007360424846410751,
+            -0.023091528564691544,
+            -0.039156634360551834,
+            -0.008061417378485203,
+            0.03816699981689453,
+            -0.006135749164968729,
+            -0.017401115968823433,
+            -0.03602278605103493,
+            -0.02089783363044262,
+            -0.009211870841681957,
+            -0.001520535908639431,
+            0.004915196914225817,
+            -0.030563287436962128,
+            -0.01624654047191143,
+            -0.0009813900105655193,
+            0.0069480761885643005,
+            0.001200965722091496,
+            0.0013545656111091375,
+            0.015339372679591179,
+            0.005059518851339817,
+            0.006865606643259525,
+            0.010613856837153435,
+            -0.018902065232396126,
+            -0.02114524319767952,
+            0.012180781923234463,
+            -0.007558352313935757,
+            0.012222016230225563,
+            -0.007113015744835138,
+            -0.0006963539053685963,
+            -0.029870541766285896,
+            0.007587216794490814,
+            -0.024411045014858246,
+            -0.03589083254337311,
+            -0.023701805621385574,
+            -0.0026101674884557724,
+            0.023157505318522453,
+            0.001592696993611753,
+            -0.003183332271873951,
+            0.03905767202377319,
+            -0.023982202634215355,
+            -0.011438554152846336,
+            -0.0059172045439481735,
+            -0.010754055343568325,
+            0.03849687799811363,
+            0.0035400139167904854,
+            -0.011158157140016556,
+            0.017928922548890114,
+            -0.025730561465024948,
+            -0.00416884571313858,
+            0.05838857963681221,
+            0.0018669088603928685,
+            0.0066429381258785725,
+            0.02089783363044262,
+            -0.0025297594256699085,
+            -0.0050224075093865395,
+            -0.02139265276491642,
+            0.017945416271686554,
+            0.0020297865848988295,
+            -0.040212247520685196,
+            -0.023239973932504654,
+            -0.039849381893873215,
+            -0.006725407671183348,
+            0.028715966269373894,
+            0.024724429473280907,
+            0.004436872433871031,
+            0.015199174173176289,
+            -0.005067766178399324,
+            -0.002641093684360385,
+            -0.018093861639499664,
+            0.003655471606180072,
+            0.009756171144545078,
+            -0.01081178430467844,
+            0.0016401171451434493,
+            -0.003805978922173381,
+            -0.0052945576608181,
+            0.04126786068081856,
+            0.004010091535747051,
+            -0.009079919196665287,
+            -0.018869077786803246,
+            -0.055287718772888184,
+            -0.034670282155275345,
+            0.034538328647613525,
+            0.014630133286118507,
+            -0.010622103698551655,
+            -0.01332711149007082,
+            0.0016164070693776011,
+            -0.04354402795433998,
+            -0.01296424400061369,
+            -0.009838640689849854,
+            0.02579653635621071,
+            0.014861048199236393,
+            -0.0028287123423069715,
+            0.020551461726427078,
+            0.022959576919674873,
+            0.013558026403188705,
+            -0.007141880225390196,
+            -0.03278997167944908,
+            0.011372578330338001,
+            0.006919211708009243,
+            0.04591915383934975,
+            0.008626335300505161,
+            -0.0047914921306073666,
+            -0.04921794682741165,
+            -0.024575984105467796,
+            0.018159838393330574,
+            -0.03819998726248741,
+            0.001192718744277954,
+            0.013541532680392265,
+            -0.01623004674911499,
+            0.0058677224442362785,
+            0.0003208588750567287,
+            -0.02205241098999977,
+            -0.0161310825496912,
+            -0.004461613483726978,
+            -0.034208450466394424,
+            -0.009929358027875423,
+            -0.04773348942399025,
+            -0.06749323755502701,
+            0.02655525878071785,
+            -0.040278226137161255,
+            -0.0648542121052742,
+            0.03592382371425629,
+            -0.03655059263110161,
+            0.006510986480861902,
+            -0.040278226137161255,
+            0.0021019477862864733,
+            -0.060268890112638474,
+            0.017961911857128143,
+            0.034175463020801544,
+            0.015710486099123955,
+            -0.0021132873371243477,
+            0.022398782894015312,
+            -0.02736346237361431,
+            -0.040443163365125656,
+            -0.06752622872591019,
+            -0.017780477181077003,
+            -0.01351679116487503,
+            -0.0031503443606197834,
+            0.002323585096746683,
+            0.05284661427140236,
+            0.0004986842395737767,
+            0.024674948304891586,
+            0.022547228261828423,
+            -0.0012978676240891218,
+            -0.04885507747530937,
+            -0.024724429473280907,
+            0.011611740104854107,
+            0.03876078128814697,
+            -0.030530299991369247,
+            0.051395148038864136,
+            -0.05258271098136902,
+            0.009665454737842083,
+            -0.009409798309206963,
+            0.006036785431206226,
+            0.019116487354040146,
+            0.007092398125678301,
+            -0.02203591726720333,
+            0.003894633846357465,
+            0.023404914885759354,
+            -0.011595246382057667,
+            0.03585784509778023,
+            -0.012692093849182129,
+            -0.034538328647613525,
+            -0.009442785754799843,
+            -0.007661439478397369,
+            0.03750723972916603,
+            0.011182897724211216,
+            -0.016172317788004875,
+            -0.028798434883356094,
+            0.011784926988184452,
+            0.02111225575208664,
+            0.03866181895136833,
+            -0.02229982055723667,
+            -0.022761650383472443,
+            0.009393304586410522,
+            -0.035692907869815826,
+            0.018027886748313904,
+            -0.017170201987028122,
+            0.005900710355490446,
+            -0.009442785754799843,
+            -0.045226410031318665,
+            -0.0008643860346637666,
+            -0.00010920798376901075,
+            -0.015891920775175095,
+            0.01045716367661953,
+            0.0061604902148246765,
+            0.018638163805007935,
+            0.04113591089844704,
+            -0.02221735008060932,
+            -0.002874070778489113,
+            0.0772576555609703,
+            -0.015537300147116184,
+            0.018819596618413925,
+            0.01659291237592697,
+            -0.01568574644625187,
+            0.02652227133512497,
+            -0.025829525664448738,
+            -0.028419075533747673,
+            0.017467092722654343,
+            -0.01558678224682808,
+            -0.030398348346352577,
+            -0.007467635441571474,
+            -0.003577125258743763,
+            0.010910747572779655,
+            0.02193695306777954,
+            -0.03293841704726219,
+            -0.004898703191429377,
+            -0.019825726747512817,
+            0.03311984986066818,
+            -0.03366415202617645,
+            0.007327436935156584,
+            9.535564458929002e-05,
+            -0.010259236209094524,
+            0.040542129427194595,
+            -0.017763983458280563,
+            -0.006127502303570509,
+            0.006292441859841347,
+            -0.010020074434578419,
+            -0.003263740334659815,
+            0.008411914110183716,
+            0.01359101478010416,
+            -0.02037002705037594,
+            0.0034328033216297626,
+            -0.0009437631815671921,
+            -0.0008262437768280506,
+            -0.004927567206323147,
+            -0.03671552985906601,
+            -0.014687862247228622,
+            0.006238836329430342,
+            -0.023800769820809364,
+            -0.0015256903134286404,
+            0.004939937964081764,
+            -0.03876078128814697,
+            0.0065893325954675674,
+            0.02144213393330574,
+            -0.0028431445825845003,
+            0.006045032292604446,
+            -0.0032431227155029774,
+            -0.005500731989741325,
+            -0.0009633497102186084,
+            -0.0018246431136503816,
+            0.017994899302721024,
+            0.04644696041941643,
+            0.008593347854912281,
+            0.011479788459837437,
+            0.007508870679885149,
+            -0.012469425797462463,
+            0.017516573891043663,
+            -0.022547228261828423,
+            0.007009928580373526,
+            0.03754022717475891,
+            0.02627486176788807,
+            0.011826162226498127,
+            0.003166838316246867,
+            0.0007674840744584799,
+            0.018951548263430595,
+            -0.034208450466394424,
+            0.024872876703739166,
+            0.014234278351068497,
+            -0.000996337621472776,
+            -0.008865498006343842,
+            -0.01110042817890644,
+            2.0150322598055936e-05,
+            0.017945416271686554,
+            0.008939720690250397,
+            -0.004106993321329355,
+            -0.03536302596330643,
+            0.002777168760076165,
+            0.018489716574549675,
+            0.010119037702679634,
+            -0.010894253849983215,
+            -0.024064673110842705,
+            -0.024180129170417786,
+            9.25851782085374e-05,
+            0.02180500142276287,
+            0.014432205818593502,
+            0.019231945276260376,
+            0.0269676074385643,
+            0.029260266572237015,
+            0.02693461999297142,
+            0.008733546361327171,
+            0.012114806100726128,
+            0.011521023698151112,
+            0.006424393504858017,
+            -0.027858281508088112,
+            -0.007657316047698259,
+            0.025466658174991608,
+            0.019594812765717506,
+            0.002787477569654584,
+            0.025450164452195168,
+            0.00653572753071785,
+            -0.005187347065657377,
+            0.020534968003630638,
+            -0.0012937441933900118,
+            -0.0324271060526371,
+            -0.0074758827686309814,
+            -0.015265149995684624,
+            0.008428407832980156,
+            -0.003253431525081396,
+            -0.0032266287598758936,
+            -0.017285659909248352,
+            0.014209537766873837,
+            0.012857033871114254,
+            0.0014988876646384597,
+            0.00264315539970994,
+            -0.0010659215040504932,
+            -0.015017741359770298,
+            -0.013788941316306591,
+            0.00541001558303833,
+            -0.013887905515730381,
+            0.056013453751802444,
+            -0.0214256402105093,
+            -0.016469208523631096,
+            -0.017648525536060333,
+            -0.006787260062992573,
+            0.017747489735484123,
+            0.01654343120753765,
+            0.014102326706051826,
+            0.018308283761143684,
+            0.00041441048961132765,
+            0.04370896518230438,
+            0.00416884571313858,
+            -0.03625370189547539,
+            0.002042157109826803,
+            -0.01590016670525074,
+            0.011512776836752892,
+            0.017302153632044792,
+            0.0042471918277442455,
+            0.002052465919405222,
+            0.01359926164150238,
+            -0.006432640366256237,
+            0.023239973932504654,
+            -0.008758286945521832,
+            -0.0023792521096765995,
+            0.008980955928564072,
+            0.004750257357954979,
+            0.012733329087495804,
+            0.005174976773560047,
+            -0.028748953714966774,
+            -0.014019857160747051,
+            0.01084477175027132,
+            0.005038901697844267,
+            -0.011281860992312431,
+            0.030480818822979927,
+            0.01591666042804718,
+            0.018176332116127014,
+            0.004350278992205858,
+            -0.010077803395688534,
+            -0.012287992052733898,
+            0.0014751775888726115,
+            0.008003689348697662,
+            0.023767780512571335,
+            0.019594812765717506,
+            -0.013162171468138695,
+            0.0056244367733597755,
+            -0.04189463332295418,
+            0.012518907897174358,
+            0.009112906642258167,
+            -0.015875427052378654,
+            -0.0036719655618071556,
+            0.01636199839413166,
+            0.0045440830290317535,
+            0.00045848023728467524,
+            -0.008791275322437286,
+            0.039915356785058975,
+            0.024806899949908257,
+            0.008914980106055737,
+            -0.0063955290243029594,
+            -0.014753838069736958,
+            -0.002406054874882102,
+            0.016634147614240646,
+            0.028946882113814354,
+            0.011570505797863007,
+            0.014061091467738152,
+            0.010795289650559425,
+            -0.023338938131928444,
+            0.012230263091623783,
+            0.028072701767086983,
+            0.044929519295692444,
+            0.016485702246427536,
+            0.03340024873614311,
+            0.027858281508088112,
+            0.0027379957027733326,
+            0.009327328763902187,
+            -0.02068341337144375,
+            -0.014778578653931618,
+            0.003181270556524396,
+            -0.006197601556777954,
+            0.04684281721711159,
+            -0.0071872384287416935,
+            -0.05287960171699524,
+            -0.016502195969223976,
+            0.0025668710004538298,
+            0.02164006233215332,
+            0.020007161423563957,
+            0.004424502141773701,
+            0.0006118224118836224,
+            0.02221735008060932,
+            -0.012065324001014233,
+            -0.013986868783831596,
+            0.011578752659261227,
+            0.0071789915673434734,
+            -0.018588680773973465,
+            0.013945634476840496,
+            0.019067006185650826,
+            -0.003026639809831977,
+            -0.009937604889273643,
+            -0.0036080514546483755,
+            0.024592477828264236,
+            0.006725407671183348,
+            -2.0440254957065918e-05,
+            0.0013968313578516245,
+            0.009269599802792072,
+            0.0027627365197986364,
+            0.04176267981529236,
+            -0.005789376329630613,
+            0.009764418005943298,
+            -0.012271498329937458,
+            -0.004201833624392748,
+            0.03377960994839668,
+            -0.03509912267327309,
+            -0.015496065840125084,
+            0.019248438999056816,
+            0.013665237464010715,
+            0.002488524653017521,
+            0.0017256794963032007,
+            -0.004972925875335932,
+            -0.012213769368827343,
+            0.0316024087369442,
+            -0.0056120664812624454,
+            -0.006605826783925295,
+            0.039651453495025635,
+            -0.006985187530517578,
+            -0.005702782887965441,
+            0.028204653412103653,
+            -0.0015782647533342242,
+            -0.03241061046719551,
+            0.014226031489670277,
+            -0.008552112616598606,
+            0.03552796691656113,
+            0.015792956575751305,
+            -0.012172535061836243,
+            -0.011554011143743992,
+            0.013929140754044056,
+            -0.009879875928163528,
+            -0.023075034841895103,
+            0.0029586022719740868,
+            -0.022580217570066452,
+            0.013986868783831596,
+            -0.009797406382858753,
+            0.022365795448422432,
+            -0.006737778428941965,
+            0.017615538090467453,
+            0.018407247960567474,
+            0.008440778590738773,
+            -0.017417611554265022,
+            0.011306602507829666,
+            -0.02121121808886528,
+            -0.05462796241044998,
+            0.010135532356798649,
+            -0.002533883089199662,
+            0.017153708264231682,
+            -0.011158157140016556,
+            -0.009541749954223633,
+            0.003568878397345543,
+            -0.008659323677420616,
+            -0.05185697600245476,
+            0.0067006670869886875,
+            -0.023718299344182014,
+            0.014308501034975052,
+            -0.02104627899825573,
+            -0.027957243844866753,
+            -0.015776462852954865,
+            -0.04361000284552574,
+            -0.014762084931135178,
+            -0.0130714550614357,
+            0.015289891511201859,
+            0.016238292679190636,
+            -0.011677715927362442,
+            -0.018275296315550804,
+            0.030744722113013268,
+            0.012436438351869583,
+            -0.006754272151738405,
+            -0.016972273588180542,
+            0.017120718955993652,
+            -0.03117356449365616,
+            -0.0440058559179306,
+            -0.013846670277416706,
+            0.02079886943101883,
+            0.007970700971782207,
+            -0.02083185873925686,
+            -0.028353098779916763,
+            -0.005772882141172886,
+            -0.012131299823522568,
+            -0.002649340545758605,
+            0.03356518596410751,
+            0.017335141077637672,
+            0.01619705744087696,
+            0.029227279126644135,
+            -0.03336726129055023,
+            0.03142097219824791,
+            -0.015570288524031639,
+            0.023982202634215355,
+            0.0035894957836717367,
+            -0.011916878633201122,
+            -0.006515109911561012,
+            0.04403884336352348,
+            -0.012716835364699364,
+            0.018110357224941254,
+            -0.022266831248998642,
+            -0.0028967498801648617,
+            0.006172860506922007,
+            0.02137615904211998,
+            -0.006915088277310133,
+            -0.007273831870406866,
+            0.00403689406812191,
+            -0.024196624755859375,
+            0.022547228261828423,
+            0.04295024275779724,
+            0.00791709590703249,
+            0.01558678224682808,
+            -0.019314413890242577,
+            -0.012914762832224369,
+            0.018935054540634155,
+            0.008849004283547401,
+            -0.014085832983255386,
+            -0.012898268178105354,
+            0.01030871830880642,
+            -0.018044380471110344,
+            -0.019330909475684166,
+            -0.019677281379699707,
+            -0.017401115968823433,
+            0.009929358027875423,
+            0.011314849369227886,
+            -0.040212247520685196,
+            -0.011603493243455887,
+            -0.04288426786661148,
+            -0.011842655949294567,
+            0.008766533806920052,
+            -0.006832618732005358,
+            0.03579187020659447,
+            -0.0036740272771567106,
+            0.004214203916490078,
+            -0.015562041662633419,
+            -0.007418153807520866,
+            -0.002801909577101469,
+            -0.035264063626527786,
+            0.0005896586808376014,
+            0.015075469389557838,
+            0.0003703407128341496,
+            -0.04710672050714493,
+            0.017829960212111473,
+            -0.016617653891444206,
+            0.016865063458681107,
+            -0.02708306536078453,
+            -0.008185122162103653,
+            -0.024312080815434456,
+            0.011611740104854107,
+            0.008997449651360512,
+            -0.03605577349662781,
+            -0.014423958957195282,
+            -0.002556562190875411,
+            0.029293254017829895,
+            -0.017351634800434113,
+            0.0006633660523220897,
+            0.017302153632044792,
+            -0.00019715423695743084,
+            -0.010894253849983215,
+            -0.022497747093439102,
+            -0.0022266830783337355,
+            -0.007937713526189327,
+            0.020106123760342598,
+            -0.04156475141644478,
+            -0.005496608559042215,
+            -0.01082003116607666,
+            0.004001844674348831,
+            -0.030150938779115677,
+            -0.009723182767629623,
+            0.006432640366256237,
+            0.02229982055723667,
+            -0.04882209002971649,
+            0.009253106079995632,
+            0.04374195262789726,
+            -0.018489716574549675,
+            -0.01057262159883976,
+            0.025120284408330917,
+            -0.0007772773969918489,
+            0.009459280408918858,
+            -0.009063425473868847,
+            0.016683630645275116,
+            0.02579653635621071,
+            -0.02061743661761284,
+            -0.008020183071494102,
+            0.0003203434171155095,
+            0.002498833229765296,
+            -0.027973737567663193,
+            0.018473222851753235,
+            -0.013261135667562485,
+            0.023289456963539124,
+            0.0077768974006175995,
+            0.010943735018372536,
+            0.05429808050394058,
+            -0.015248656272888184,
+            0.014399217441678047,
+            0.015562041662633419,
+            -0.0074800061993300915,
+            -0.007974824868142605,
+            -0.013772447593510151,
+            0.022563721984624863,
+            0.03704541176557541,
+            0.013582766987383366,
+            0.005649177823215723,
+            -0.039717432111501694,
+            0.01293950341641903,
+            0.0074758827686309814,
+            0.0038637076504528522,
+            0.02096381038427353,
+            0.017763983458280563,
+            0.02196994051337242,
+            0.024163635447621346,
+            -0.02594498172402382,
+            -0.006098637823015451,
+            0.005513102747499943,
+            -0.02762736566364765,
+            -0.003422494512051344,
+            -0.019512342289090157,
+            0.023619335144758224,
+            0.015223915688693523,
+            -0.02660473994910717,
+            0.0008654168923385441,
+            -0.007954207248985767,
+            -0.0006757365190424025,
+            -0.003993597347289324,
+            -0.012667353264987469,
+            -0.012263251468539238,
+            -0.01576821506023407,
+            0.02759437821805477,
+            -0.014753838069736958,
+            -0.024378057569265366,
+            -0.015990883111953735,
+            0.014489934779703617,
+            -0.009088166058063507,
+            0.011224132962524891,
+            -0.030992131680250168,
+            -0.04964678734540939,
+            -0.0216730497777462,
+            0.028419075533747673,
+            0.03302088752388954,
+            -0.025417177006602287,
+            -1.3602676517621148e-05,
+            -0.007042916491627693,
+            0.029755085706710815,
+            -0.03622071444988251,
+            -0.008568606339395046,
+            -0.01332711149007082,
+            -0.04733763635158539,
+            0.0130714550614357,
+            -0.028567520901560783,
+            0.024922357872128487,
+            0.02708306536078453,
+            -0.018308283761143684,
+            -0.03148695081472397,
+            0.0210957620292902,
+            0.020155606791377068,
+            -0.004109055269509554,
+            0.010234495624899864,
+            -0.0036822741385549307,
+            -0.02658824622631073,
+            -0.028270630165934563,
+            -0.003356518689543009,
+            0.04163073003292084,
+            0.014902283437550068,
+            0.03536302596330643,
+            0.04212554544210434,
+            -0.009921110235154629,
+            -0.034241437911987305,
+            0.019099993631243706,
+            0.009945851750671864,
+            -0.007653192616999149,
+            0.00516672944650054,
+            0.003502902574837208,
+            -0.018423741683363914,
+            -0.009137648157775402,
+            -0.008840756490826607,
+            -0.028765447437763214,
+            -0.0441707968711853,
+            0.0012638488551601768,
+            0.0013360099401324987,
+            0.0167083702981472,
+            -0.012518907897174358,
+            -0.006374911405146122,
+            0.035132113844156265,
+            0.03331777825951576,
+            -0.009789158590137959,
+            -2.2904683646629564e-05,
+            -0.017994899302721024,
+            -0.0103004714474082,
+            -0.008131517097353935,
+            -0.002187510021030903,
+            0.023833757266402245,
+            -0.014300254173576832,
+            0.030051976442337036,
+            -0.0015452768420800567,
+            0.015512559562921524,
+            -0.06749323755502701,
+            -0.02066691778600216,
+            0.03155292570590973,
+            0.008725299499928951,
+            -0.009970592334866524,
+            0.002399869728833437,
+            0.005480114836245775,
+            0.0001164240893558599,
+            -0.0220029279589653,
+            0.011694210581481457,
+            0.020023655146360397,
+            0.019974172115325928,
+            -0.020534968003630638,
+            0.022497747093439102,
+            0.01632900908589363,
+            0.019314413890242577,
+            -0.004210080485790968,
+            -0.025268729776144028,
+            0.012444685213267803,
+            -0.007867613807320595,
+            -0.0019627800211310387,
+            0.002861700253561139,
+            0.007463512010872364,
+            -0.008923226967453957,
+            0.01017676666378975,
+            -0.02205241098999977,
+            -0.02726449817419052,
+            -0.011529270559549332,
+            0.017813464626669884,
+            0.010473658330738544,
+            0.007113015744835138,
+            0.008436654694378376,
+            0.0018452606163918972,
+            0.00416884571313858,
+            0.0013267321046441793,
+            -0.019099993631243706,
+            0.028237640857696533,
+            0.001028294675052166,
+            0.013929140754044056,
+            0.013145677745342255,
+            -0.023635828867554665,
+            -0.023058541119098663,
+            -0.008626335300505161,
+            0.01072106696665287,
+            0.0035647547338157892,
+            0.019017523154616356,
+            0.03879376873373985,
+            -0.014918777160346508,
+            -0.014992999844253063,
+            -0.0008478920790366828,
+            0.019759751856327057,
+            0.0056780423037707806,
+            -0.010943735018372536,
+            -0.0028369592037051916,
+            -0.00416884571313858,
+            -0.005925451405346394,
+            0.025747055187821388,
+            -0.008890238590538502,
+            -0.03373012691736221,
+            0.010943735018372536,
+            0.05756388232111931,
+            -0.007059410214424133,
+            -0.0011659160954877734,
+            -0.022514240816235542,
+            -0.0019122672965750098,
+            0.006906841415911913,
+            0.0080655412748456,
+            0.006993434391915798,
+            0.015281643718481064,
+            0.01581769809126854,
+            0.012147793546319008,
+            -0.010102543979883194,
+            -0.012007595039904118,
+            -0.010374694131314754,
+            -0.03787010908126831,
+            0.030018987134099007,
+            -0.018258802592754364,
+            -0.0035358904860913754,
+            0.011331343092024326,
+            0.02188747003674507,
+            0.00010637308878358454,
+            0.016518689692020416,
+            0.013409581035375595,
+            0.004692528396844864,
+            0.0005932667409069836,
+            0.011652975343167782,
+            0.0015494003891944885,
+            -0.013896152377128601,
+            -0.022761650383472443,
+            -0.009533503092825413,
+            0.018918560817837715,
+            0.02705007791519165,
+            -0.004643046762794256,
+            -0.009978839196264744,
+            -0.005673918407410383,
+            0.014687862247228622,
+            -0.004878085572272539,
+            0.011916878633201122,
+            -0.0045317127369344234,
+            0.00045564534957520664,
+            0.030728228390216827,
+            0.013953881338238716,
+            -0.007298572454601526,
+            0.016972273588180542,
+            0.004556453321129084,
+            -0.003414247650653124,
+            0.019479354843497276,
+            -0.04915196821093559,
+            0.01590016670525074,
+            0.0028348974883556366,
+            0.0018029948696494102,
+            0.0036699038464576006,
+            -0.0020679289009422064,
+            -0.013277629390358925,
+            0.012626118026673794,
+            0.007636698428541422,
+            0.003036948386579752,
+            0.010152026079595089,
+            0.010110790841281414,
+            0.00022125086979940534,
+            0.012279745191335678,
+            -0.019776245579123497,
+            0.011512776836752892,
+            -0.015908414497971535,
+            -0.009467527270317078,
+            -0.008477889932692051,
+            -0.0013019911712035537,
+            -0.0037647439166903496,
+            -0.009962345473468304,
+            0.000947886670473963,
+            -0.022398782894015312,
+            -0.006758395582437515,
+            -0.017434105277061462,
+            -0.008799522183835506,
+            -0.0025359448045492172,
+            -0.024081166833639145,
+            0.0076738097704946995,
+            0.004960555117577314,
+            0.011232379823923111,
+            -0.03658358007669449,
+            0.0028328357730060816,
+            0.03272399678826332,
+            0.004506971687078476,
+            -0.018077367916703224,
+            -0.024378057569265366,
+            -0.011702457442879677,
+            -0.02657175250351429,
+            0.013285876251757145,
+            0.011075686663389206,
+            -0.020040148869156837,
+            0.015463077463209629,
+            -0.001200965722091496,
+            -0.023784274235367775,
+            -0.02098030410706997,
+            -0.02073289453983307,
+            -0.0007360425079241395,
+            0.0013566273264586926,
+            0.001249416614882648,
+            0.0007391351391561329,
+            0.004869838710874319,
+            -0.022877108305692673,
+            0.019611306488513947,
+            0.03354869410395622,
+            -0.008486136794090271,
+            -0.0030472571961581707,
+            0.008816015906631947,
+            0.0220029279589653,
+            -0.01660940796136856,
+            0.00010611536708893254,
+            -0.015042481943964958,
+            0.02150811068713665,
+            -0.0027565513737499714,
+            0.012626118026673794,
+            -0.0026946989819407463,
+            0.029639627784490585,
+            -0.004688404966145754,
+            0.0272974856197834,
+            -0.008947967551648617,
+            0.0220029279589653,
+            0.0013782756868749857,
+            -0.034769244492053986,
+            -0.009401551447808743,
+            0.03277347609400749,
+            -0.002628723159432411,
+            -0.013178665190935135,
+            0.022926589474081993,
+            -0.0036967063788324594,
+            -0.024048177525401115,
+            0.00832944456487894,
+            0.0015875425888225436,
+            -0.01375595387071371,
+            0.024955345317721367,
+            0.006370787974447012,
+            0.009549996815621853,
+            0.004742010496556759,
+            0.015529053285717964,
+            0.024031683802604675,
+            -0.000978297321125865,
+            0.03213021531701088,
+            0.009508761577308178,
+            0.0070016817189753056,
+            -0.004346155561506748,
+            0.012403449974954128,
+            -0.005100754089653492,
+            0.024806899949908257,
+            -0.03272399678826332,
+            -0.014118820428848267,
+            0.033746618777513504,
+            0.015232162550091743,
+            0.018159838393330574,
+            -0.022563721984624863,
+            -0.0047873686999082565,
+            0.016386738047003746,
+            0.012733329087495804,
+            -0.01585068553686142,
+            -0.02081536501646042,
+            -0.001431880984455347,
+            0.008510878309607506,
+            0.004280179738998413,
+            -0.017599044367671013,
+            0.00668417289853096,
+            0.03816699981689453,
+            -0.017978405579924583,
+            -0.040047310292720795,
+            0.02596147730946541,
+            -0.015471324324607849,
+            -0.003115294734016061,
+            0.016007378697395325,
+            -0.005389397963881493,
+            0.014473441056907177,
+            -0.008857251144945621,
+            -0.0033029133919626474,
+            0.03628668934106827,
+            0.008914980106055737,
+            0.010481905192136765,
+            -0.030018987134099007,
+            0.02101329155266285,
+            0.002962725702673197,
+            0.007653192616999149,
+            -0.023668818175792694,
+            0.002189571736380458,
+            0.012411696836352348,
+            -0.030051976442337036,
+            -0.003247246379032731,
+            0.03816699981689453,
+            0.008148010820150375,
+            -0.017615538090467453,
+            0.0008721175836399198,
+            -0.02579653635621071,
+            0.02038652077317238,
+            -0.0050801364704966545,
+            0.014580651186406612,
+            0.019462861120700836,
+            -0.012114806100726128,
+            0.018060874193906784,
+            -0.0056161899119615555,
+            -0.011562258936464787,
+            -0.004931691102683544,
+            -0.010943735018372536,
+            -0.02665422298014164,
+            -0.013986868783831596,
+            0.0020029840525239706,
+            -0.01059736218303442,
+            -0.01629602164030075,
+            0.007158373948186636,
+            -0.03641863912343979,
+            -0.014910530298948288,
+            -0.006391405593603849,
+            0.007632574997842312,
+            -0.023569853976368904,
+            0.017450598999857903,
+            -0.024641960859298706,
+            0.0029874665196985006,
+            0.02182149514555931,
+            -0.003570940112695098,
+            0.0024204871151596308,
+            -0.019001029431819916,
+            -0.029276760295033455,
+            0.008366555906832218,
+            0.006506863050162792,
+            -0.00681200111284852,
+            -0.015908414497971535,
+            0.008857251144945621,
+            0.0017669143853709102,
+            -0.0037771144416183233,
+            -0.006255330517888069,
+            0.01293950341641903,
+            -0.023685311898589134,
+            0.017483586445450783,
+            0.019545329734683037,
+            -0.024246105924248695,
+            0.010762302204966545,
+            -0.0069975582882761955,
+            0.017483586445450783,
+            0.01655992493033409,
+            0.0051337420009076595,
+            0.0164609607309103,
+            0.02695111371576786,
+            0.008259344846010208,
+            0.022728662937879562,
+            0.000705116312019527,
+            -0.004383266903460026,
+            -0.019512342289090157,
+            0.027792304754257202,
+            -0.009525256231427193,
+            0.0001148777810158208,
+            0.012568389065563679,
+            -0.009731430560350418,
+            -0.013945634476840496,
+            0.0019308229675516486,
+            4.95140629936941e-05,
+            -0.016172317788004875,
+            -0.007875860668718815,
+            -0.02150811068713665,
+            -0.005768758710473776,
+            -0.018242308869957924,
+            -0.01110042817890644,
+            0.02220085635781288,
+            -0.020023655146360397,
+            -0.0209308210760355,
+            0.00045384131954051554,
+            -0.00031982798827812076,
+            0.019974172115325928,
+            0.020155606791377068,
+            0.009302587248384953,
+            0.0021297812927514315,
+            -0.0025462533812969923,
+            -0.00341012398712337,
+            -0.005686289165169001,
+            0.008366555906832218,
+            0.006510986480861902,
+            0.02220085635781288,
+            -0.0216730497777462,
+            -0.01576821506023407,
+            0.007080027833580971,
+            -0.006840865593403578,
+            0.009764418005943298,
+            -0.019974172115325928,
+            -0.011776680126786232,
+            -0.004704899154603481,
+            -0.017005261033773422,
+            0.0552547313272953,
+            -0.02716553397476673,
+            -0.035297051072120667,
+            -0.01058086846023798,
+            0.002946231747046113,
+            0.006741901859641075,
+            -0.018522705882787704,
+            0.010366447269916534,
+            -0.01323639415204525,
+            -0.014869295060634613,
+            -0.023586347699165344,
+            0.00514611229300499,
+            0.0026081057731062174,
+            0.005933698266744614,
+            -0.009772664867341518,
+            -0.022910095751285553,
+            0.0036121748853474855,
+            -0.017236176878213882,
+            -0.017434105277061462,
+            -0.0020060765091329813,
+            0.01031696517020464,
+            0.015858933329582214,
+            0.018027886748313904,
+            0.007277955301105976,
+            0.002725625177845359,
+            0.0009215994505211711,
+            -0.002183386590331793,
+            0.009145895019173622,
+            -0.00333383958786726,
+            0.003041071817278862,
+            0.008585100993514061,
+            -0.027808798477053642,
+            0.009582984261214733,
+            -0.013574520125985146,
+            0.02705007791519165,
+            0.029161302372813225,
+            0.01603211835026741,
+            0.013137430883944035,
+            0.009244858287274837,
+            -0.040179260075092316,
+            0.011364331468939781,
+            -0.000893765885848552,
+            0.00807378813624382,
+            0.007220226339995861,
+            0.0038637076504528522,
+            -0.007298572454601526,
+            0.0013040528865531087,
+            -0.019116487354040146,
+            -0.004408007953315973,
+            0.000873663870152086,
+            -0.024559490382671356,
+            0.02187097631394863,
+            0.013533285818994045,
+            0.015545547008514404,
+            0.020403016358613968,
+            -0.011801420710980892,
+            0.010886006988584995,
+            0.01297249086201191,
+            0.003876078175380826,
+            -0.020419510081410408,
+            0.005830611102283001,
+            -0.023437902331352234,
+            0.0065893325954675674,
+            -0.010086050257086754,
+            -0.0340765006840229,
+            -0.020122617483139038,
+            0.017846453934907913,
+            0.008432531729340553,
+            -0.009261352941393852,
+            -0.028946882113814354,
+            0.025235742330551147,
+            0.027858281508088112,
+            -0.004498724825680256,
+            0.008956214413046837,
+            0.027792304754257202,
+            0.010358200408518314,
+            0.0446656160056591,
+            -0.0080655412748456,
+            0.009723182767629623,
+            -0.004630676470696926,
+            0.007088274694979191,
+            -0.0051378654316067696,
+            -0.006869730073958635,
+            0.011050946079194546,
+            0.014762084931135178,
+            0.006069773342460394,
+            0.018836090341210365,
+            0.029359230771660805,
+            -0.019281426444649696,
+            -0.007459388580173254,
+            0.013137430883944035,
+            -0.010770549066364765,
+            0.02088133990764618,
+            -0.0001592696935404092,
+            -0.006358417682349682,
+            -0.028303617611527443,
+            0.006976940669119358,
+            0.008774780668318272,
+            -0.007401659619063139,
+            -0.003787423251196742,
+            -0.019842220470309258,
+            -0.005504855420440435,
+            -0.0012762193800881505,
+            -0.020320545881986618,
+            -0.0009262383682653308,
+            -0.004993543028831482,
+            -0.0026699581649154425,
+            0.0047914921306073666,
+            0.018159838393330574,
+            0.008659323677420616,
+            0.009912863373756409,
+            -0.0017308337846770883,
+            0.014861048199236393,
+            0.03381259739398956,
+            -0.005636807065457106,
+            -0.000506158045027405,
+            0.019231945276260376,
+            0.0011978730326518416,
+            0.013937387615442276,
+            -0.004259562585502863,
+            -0.005785252898931503,
+            -0.014894036576151848,
+            0.01571873389184475,
+            -0.004614182282239199,
+            -0.004201833624392748,
+            -0.012774563394486904,
+            -0.020205087959766388,
+            0.012494166381657124,
+            -0.008857251144945621,
+            -0.008923226967453957,
+            -0.023685311898589134,
+            -0.03130551427602768,
+            -0.017450598999857903,
+            -0.0038224728778004646,
+            -0.0009566490771248937,
+            0.014654873870313168,
+            0.02124420739710331,
+            -0.007232596632093191,
+            0.022250337526202202,
+            -0.005962562747299671,
+            -0.03320232033729553,
+            0.009096412919461727,
+            0.014992999844253063,
+            -0.010275730863213539,
+            0.008267591707408428,
+            -0.02221735008060932,
+            -0.009277846664190292,
+            -0.03206423670053482,
+            -0.019776245579123497,
+            -0.0034781615249812603,
+            0.02149161696434021,
+            -0.01636199839413166,
+            0.02688513696193695,
+            -0.013277629390358925,
+            -0.00779339112341404,
+            0.01333535835146904,
+            0.003752373391762376,
+            -0.01606510765850544,
+            -0.004440995864570141,
+            0.011479788459837437,
+            0.01674135960638523,
+            -0.013021972961723804,
+            -0.004560577217489481,
+            -0.00333796301856637,
+            -0.009706689044833183,
+            0.008914980106055737,
+            -0.006857359316200018,
+            0.007640821859240532,
+            -0.019644293934106827,
+            0.010918994434177876,
+            0.005055395420640707,
+            -0.0017143398290500045,
+            -0.0019339155405759811,
+            -0.013673484325408936,
+            0.005711030215024948,
+            0.013656990602612495,
+            0.020551461726427078,
+            -0.03866181895136833,
+            0.004572947509586811,
+            -0.002818403532728553,
+            -0.005224458407610655,
+            -0.018605174496769905,
+            0.012452932074666023,
+            0.016584666445851326,
+            0.015463077463209629,
+            0.013722965493798256,
+            -0.01041592936962843,
+            0.022514240816235542,
+            -0.012395203113555908,
+            -0.013038466684520245,
+            0.00780988484621048,
+            -0.01306320820003748,
+            0.0005324453231878579,
+            0.029045844450592995,
+            -0.007574846502393484,
+            -0.004626553039997816,
+            0.005574955139309168,
+            0.039552491158246994,
+            0.005385274533182383,
+            0.004985296167433262,
+            0.029375724494457245,
+            -0.0013391025131568313,
+            0.006844989024102688,
+            0.03262503072619438,
+            0.017928922548890114,
+            0.007896478287875652,
+            -0.007096521556377411,
+            0.013962128199636936,
+            0.011496283113956451,
+            0.013904399238526821,
+            -0.012510661035776138,
+            -0.04077304154634476,
+            0.019231945276260376,
+            0.004651293624192476,
+            -0.011974607594311237,
+            0.010152026079595089,
+            -0.0024328576400876045,
+            0.007257337681949139,
+            0.017054744064807892,
+            0.00026699580485001206,
+            -0.022811131551861763,
+            0.013426074758172035,
+            0.004170907661318779,
+            0.011298355646431446,
+            -0.005055395420640707,
+            0.000341476290486753,
+            0.02233280800282955,
+            0.007108892314136028,
+            0.0014689923264086246,
+            0.012469425797462463,
+            -0.0128240454941988,
+            0.02145862765610218,
+            0.019825726747512817,
+            0.02716553397476673,
+            -0.0031503443606197834,
+            -0.0031482826452702284,
+            0.000705116312019527,
+            -0.0070016817189753056,
+            -0.012890021316707134,
+            0.009492267854511738,
+            0.0003778145182877779,
+            0.009401551447808743,
+            -0.005851228721439838,
+            0.011685962788760662,
+            0.007570722606033087,
+            0.004849221091717482,
+            0.019974172115325928,
+            0.005253322888165712,
+            -0.0068037542514503,
+            0.028946882113814354,
+            0.0020277248695492744,
+            0.0009226303081959486,
+            -0.011050946079194546,
+            0.0006973848212510347,
+            0.0014669306110590696,
+            -0.001361781731247902,
+            0.007611957844346762,
+            -0.008110899478197098,
+            -0.0130714550614357,
+            -0.01086126547306776,
+            0.028369592502713203,
+            0.00025436762371100485,
+            0.023503877222537994,
+            -0.0053316690027713776,
+            -0.002236991887912154,
+            0.002789539285004139,
+            -0.01618056371808052,
+            -0.005669794976711273,
+            0.015710486099123955,
+            0.0033214690629392862,
+            0.006432640366256237,
+            0.013285876251757145,
+            -0.0053770276717841625,
+            0.016353750601410866,
+            -0.0056203133426606655,
+            0.0036987680941820145,
+            0.013030219823122025,
+            0.005987303797155619,
+            0.008733546361327171,
+            0.0006375942612066865,
+            -0.017796970903873444,
+            -0.019611306488513947,
+            0.01085301861166954,
+            -0.006865606643259525,
+            0.000996337621472776,
+            0.003098800778388977,
+            -0.001747327740304172,
+            0.011248873546719551,
+            -0.015009493567049503,
+            -0.003950301092118025,
+            -0.008007812313735485,
+            0.034967172890901566,
+            -0.006127502303570509,
+            -0.002942108316347003,
+            -0.0063996524550020695,
+            -0.01334360521286726,
+            0.003667841898277402,
+            -0.008585100993514061,
+            -0.00026235688710585237,
+            0.015710486099123955,
+            0.004915196914225817,
+            -0.01703825034201145,
+            -0.013269382528960705,
+            -0.014696109108626842,
+            0.006469751708209515,
+            0.02078237570822239,
+            0.02706657163798809,
+            -0.008783028461039066,
+            0.007108892314136028,
+            -0.008135640993714333,
+            0.017054744064807892,
+            -0.007706797681748867,
+            0.013541532680392265,
+            -0.022695673629641533,
+            0.008898485451936722,
+            -0.01362400222569704,
+            0.016353750601410866,
+            -0.007438771426677704,
+            0.0038905104156583548,
+            -0.01557853538542986,
+            -0.009690195322036743,
+            0.0010999402729794383,
+            -0.011273614130914211,
+            0.009879875928163528,
+            -0.01030871830880642,
+            -0.008535618893802166,
+            0.0043007973581552505,
+            -0.02159057930111885,
+            -0.0021297812927514315,
+            -0.007682057097554207,
+            -0.005636807065457106,
+            -0.009698442183434963,
+            -0.02116173692047596,
+            -0.02140914648771286,
+            0.005694536026567221,
+            -0.009492267854511738,
+            -0.012147793546319008,
+            -0.028979869559407234,
+            -0.03326829522848129,
+            -0.001901958603411913,
+            -0.02180500142276287,
+            -0.014259018935263157,
+            -0.005814117379486561,
+            0.019215451553463936,
+            -0.011166404001414776,
+            -0.012914762832224369,
+            -0.00795008335262537,
+            -0.011125168763101101,
+            0.019462861120700836,
+            0.014234278351068497,
+            0.009954098612070084,
+            -0.001399923930875957,
+            0.016667135059833527,
+            -0.008700557984411716,
+            0.009756171144545078,
+            -0.040443163365125656,
+            -0.017450598999857903,
+            -0.0028369592037051916,
+            0.0026720198802649975,
+            0.008890238590538502,
+            -0.000427811814006418,
+            0.019924690946936607,
+            -0.010102543979883194,
+            -0.04374195262789726,
+            -0.02116173692047596,
+            -0.002880255924537778,
+            -0.011438554152846336,
+            0.022794637829065323,
+            0.003810102352872491,
+            0.039288587868213654,
+            0.022893602028489113,
+            0.015858933329582214,
+            -0.006708913948386908,
+            0.007401659619063139,
+            -0.019001029431819916,
+            0.0015380608383566141,
+            0.016881557181477547,
+            0.0032369375694543123,
+            -0.011174650862812996,
+            0.029359230771660805,
+            -0.009442785754799843,
+            -0.015075469389557838,
+            0.027858281508088112,
+            0.0011803482193499804,
+            0.013285876251757145,
+            0.0005705875810235739,
+            -0.0050760130397975445,
+            -0.010399434715509415,
+            0.0038451519794762135,
+            0.002278226660564542,
+            -0.012700340710580349,
+            0.01596614345908165,
+            0.00808203499764204,
+            4.94818486913573e-05,
+            0.0007891323766671121,
+            0.025153273716568947,
+            -0.0071295094676315784,
+            -0.017681512981653214,
+            0.014894036576151848,
+            0.0032018879428505898,
+            -0.01084477175027132,
+            -0.011215885169804096,
+            -0.009558243677020073,
+            -0.01349205058068037,
+            -0.010927241295576096,
+            0.0027689216658473015,
+            -0.002622538013383746,
+            0.023784274235367775,
+            -0.004022461827844381,
+            0.006370787974447012,
+            -0.00390906585380435,
+            -0.002867885399609804,
+            0.028699472546577454,
+            0.019149474799633026,
+            -0.02643980085849762,
+            -0.017335141077637672,
+            0.0007241874700412154,
+            -0.005508979316800833,
+            -0.011372578330338001,
+            0.02091432735323906,
+            -0.006024415139108896,
+            -0.024724429473280907,
+            0.025070803239941597,
+            -0.0004649232141673565,
+            0.004102869890630245,
+            0.005739894229918718,
+            0.0264727883040905,
+            -0.0025668710004538298,
+            -0.0004610574396792799,
+            -0.003036948386579752,
+            0.011323096230626106,
+            2.6738238375401124e-05,
+            0.007018175441771746,
+            -0.023998696357011795,
+            -0.003014269284904003,
+            0.010803536511957645,
+            0.002383375773206353,
+            0.02747892029583454,
+            0.005459497217088938,
+            0.00514611229300499,
+            0.01324464101344347,
+            -0.016955779865384102,
+            0.008890238590538502,
+            -0.017895935103297234,
+            0.017170201987028122,
+            -0.008750040084123611,
+            -0.025582116097211838,
+            -0.010275730863213539,
+            0.0034740380942821503,
+            0.008675817400217056,
+            0.006898594554513693,
+            0.0043007973581552505,
+            0.019825726747512817,
+            -3.6467092286329716e-05,
+            0.006354294251650572,
+            0.002397808013483882,
+            0.0009432477527298033,
+            -0.0003213743038941175,
+            0.0037709292955696583,
+            0.02660473994910717,
+            -0.008659323677420616,
+            -0.006296565290540457,
+            0.0009757201769389212,
+            -0.0033276542089879513,
+            -0.01324464101344347,
+            0.003115294734016061,
+            -0.003919374663382769,
+            -0.0034596058540046215,
+            0.045787204056978226,
+            0.017104225233197212,
+            -0.007962454110383987,
+            -0.015265149995684624,
+            -0.01598263718187809,
+            0.027759317308664322,
+            -0.001940100803039968,
+            0.011578752659261227,
+            0.019759751856327057,
+            0.0030637511517852545,
+            -0.002773045329377055,
+            -0.0007468666299246252,
+            -0.0016091909492388368,
+            0.001787531771697104,
+            0.024889370426535606,
+            0.008964461274445057,
+            -0.006964570377022028,
+            -0.008469643071293831,
+            0.019215451553463936,
+            -0.01362400222569704,
+            -0.009426292032003403,
+            0.02119472436606884,
+            -0.003034886671230197,
+            0.02079886943101883,
+            -0.007521240971982479,
+            0.01606510765850544,
+            -0.005443003494292498,
+            -0.006964570377022028,
+            -0.02162356860935688,
+            0.00404101749882102,
+            -0.002410178305581212,
+            0.003016331000253558,
+            0.028254134580492973,
+            -0.014465193264186382,
+            -0.002249362412840128,
+            0.0036080514546483755,
+            0.011182897724211216,
+            -0.015108457766473293,
+            0.01031696517020464,
+            0.001118495943956077,
+            0.0007360425079241395,
+            0.025285225361585617,
+            -0.004133795853704214,
+            -0.004692528396844864,
+            0.03130551427602768,
+            -0.007459388580173254,
+            -0.011422060430049896,
+            -0.001281373668462038,
+            0.008069665171205997,
+            0.005661548115313053,
+            -0.02126070111989975,
+            -0.002247300697490573,
+            -0.01019326038658619,
+            0.0021194724831730127,
+            -0.045391347259283066,
+            0.02159057930111885,
+            -0.02165655605494976,
+            0.025367693975567818,
+            -0.005723400507122278,
+            -0.002488524653017521,
+            -0.008593347854912281,
+            0.006201724987477064,
+            0.005574955139309168,
+            -0.012114806100726128,
+            0.007665562909096479,
+            -0.019198957830667496,
+            0.013302369974553585,
+            0.006473875138908625,
+            -0.019231945276260376,
+            0.015693992376327515,
+            -0.007529487833380699,
+            0.006568715441972017,
+            -0.015438336879014969,
+            -0.003880201606079936,
+            0.019347403198480606,
+            -0.0026967606972903013,
+            -0.005772882141172886,
+            -0.03783712163567543,
+            -0.00418121600523591,
+            -0.011389072053134441,
+            0.00163496274035424,
+            0.002814280102029443,
+            -0.015743475407361984,
+            -0.03572589531540871,
+            0.005113124381750822,
+            0.01305496133863926,
+            -0.0009911833330988884,
+            0.022481253370642662,
+            -0.005781129468232393,
+            -0.005925451405346394,
+            0.008024306036531925,
+            0.0025297594256699085,
+            0.018968041986227036,
+            0.014267266727983952,
+            -0.003490532049909234,
+            -0.0045399595983326435,
+            0.0026740815956145525,
+            -0.007698550820350647,
+            -0.017813464626669884,
+            -0.018836090341210365,
+            -0.013021972961723804,
+            0.008832509629428387,
+            0.018489716574549675,
+            -0.018159838393330574,
+            -0.00040281316614709795,
+            -0.004263686016201973,
+            0.015529053285717964,
+            -0.011356084607541561,
+            -0.022398782894015312,
+            -0.009879875928163528,
+            -0.0034451736137270927,
+            0.012329227291047573,
+            -0.024411045014858246,
+            0.014663120731711388,
+            -0.009615972638130188,
+            -0.004276056308299303,
+            0.015809450298547745,
+            -0.0053316690027713776,
+            0.0014968259492889047,
+            0.004915196914225817,
+            0.028897399082779884,
+            0.0019370082300156355,
+            -0.022596711292862892,
+            0.018638163805007935,
+            0.009426292032003403,
+            -0.00026879983488470316,
+            -0.005001790355890989,
+            -0.004214203916490078,
+            -0.018984535709023476,
+            -0.00652748066931963,
+            0.008626335300505161,
+            0.01351679116487503,
+            0.013162171468138695,
+            0.016106341034173965,
+            -0.00528218736872077,
+            0.0013452877756208181,
+            0.004313167650252581,
+            0.0011349898995831609,
+            0.009912863373756409,
+            0.007455265149474144,
+            -0.002552438760176301,
+            -0.007467635441571474,
+            0.022629698738455772,
+            -0.023223480209708214,
+            -0.0032328141387552023,
+            0.023734793066978455,
+            -0.0030451954808086157,
+            0.013788941316306591,
+            -0.00817275233566761,
+            0.002777168760076165,
+            0.015001246705651283,
+            0.007405783515423536,
+            0.009607725776731968,
+            -0.024196624755859375,
+            0.020023655146360397,
+            -0.019858716055750847,
+            0.005723400507122278,
+            0.005038901697844267,
+            0.006774889770895243,
+            0.008494383655488491,
+            -0.010020074434578419,
+            0.0034554824233055115,
+            0.01334360521286726,
+            0.016939286142587662,
+            0.017120718955993652,
+            -0.015562041662633419,
+            -0.001777223078534007,
+            -0.022514240816235542,
+            -0.017236176878213882,
+            0.02137615904211998,
+            -0.013277629390358925,
+            -0.001867939718067646,
+            0.015281643718481064,
+            -0.009640713222324848,
+            -0.009888122789561749,
+            0.023701805621385574,
+            -0.009797406382858753,
+            0.016840321943163872,
+            0.0068862237967550755,
+            0.017994899302721024,
+            0.008011936210095882,
+            -0.0012184905353933573,
+            0.0026534642092883587,
+            -0.010267484001815319,
+            -0.0012442623265087605,
+            -0.004208019003272057,
+            0.011883890256285667,
+            0.0061522433534264565,
+            -0.01041592936962843,
+            -0.001817427109926939,
+            -0.009657206945121288,
+            -0.009780911728739738,
+            -0.017087731510400772,
+            0.006791383493691683,
+            0.005991427227854729,
+            -0.007385165896266699,
+            -0.016700124368071556,
+            0.00011146302131237462,
+            0.01386316493153572,
+            -0.0029359229374676943,
+            -0.00011751939018722624,
+            0.011644728481769562,
+            -0.0076696863397955894,
+            -0.011586999520659447,
+            -0.007937713526189327,
+            -0.040113285183906555,
+            -0.012057077139616013,
+            0.012799304910004139,
+            -0.0007819163147360086,
+            0.014588898047804832,
+            0.034307416528463364,
+            -0.005591448862105608,
+            0.0017401116201654077,
+            -0.007661439478397369,
+            0.014143561944365501,
+            0.022728662937879562,
+            -0.004816233180463314,
+            -0.004465736914426088,
+            -0.001779284793883562,
+            -0.012353967875242233,
+            -0.000558217114303261,
+            -0.016073353588581085,
+            0.0009757201769389212,
+            -0.003036948386579752,
+            -0.007983071729540825,
+            -0.00339775369502604,
+            -0.010382940992712975,
+            -0.014960012398660183,
+            -0.014300254173576832,
+            0.008568606339395046,
+            0.002474092412739992,
+            -0.007748032920062542,
+            -0.0032967282459139824,
+            -0.012642612680792809,
+            0.013937387615442276,
+            0.006197601556777954,
+            0.006869730073958635,
+            0.0037132003344595432,
+            0.009030437096953392,
+            0.006263577379286289,
+            -0.005455373786389828,
+            0.003102924209088087,
+            -0.0023751286789774895,
+            -0.030662251636385918,
+            0.011661222204566002,
+            -0.016551679000258446,
+            0.020436003804206848,
+            0.009937604889273643,
+            -0.0053729042410850525,
+            0.0053729042410850525,
+            0.0024575984571129084,
+            0.02121121808886528,
+            -0.009393304586410522,
+            0.014489934779703617,
+            0.0035894957836717367,
+            0.00821398664265871,
+            -0.00529868109151721,
+            0.004271932877600193,
+            -0.004688404966145754,
+            -0.014992999844253063,
+            0.01641147956252098,
+            0.00795833021402359,
+            0.022448265925049782,
+            0.00416884571313858,
+            0.015438336879014969,
+            0.0035276433918625116,
+            -0.001785470056347549,
+            0.00021802939590997994,
+            0.016551679000258446,
+            0.006078020203858614,
+            -0.015990883111953735,
+            -0.004395637661218643,
+            -0.006201724987477064,
+            0.0128240454941988,
+            0.02630784921348095,
+            -0.01629602164030075,
+            -0.01350029744207859,
+            0.023503877222537994,
+            -0.006981064099818468,
+            0.02629135549068451,
+            -0.004589441232383251,
+            -0.017665019258856773,
+            -0.008832509629428387,
+            -0.0018473223317414522,
+            -0.001506103784777224,
+            -0.0208483524620533,
+            0.019446365535259247,
+            0.01334360521286726,
+            -0.002018447034060955,
+            -0.0035874340683221817,
+            0.010622103698551655,
+            -0.0018493840470910072,
+            0.020073136314749718,
+            -0.024064673110842705,
+            -0.007100645452737808,
+            0.0059130811132490635,
+            0.011125168763101101,
+            0.0071913618594408035,
+            -0.015487818978726864,
+            0.015990883111953735,
+            0.018902065232396126,
+            -0.0021318430081009865,
+            0.020172100514173508,
+            -0.015339372679591179,
+            -0.02091432735323906,
+            0.0014576527755707502,
+            -0.007306819781661034,
+            -0.0026720198802649975,
+            -0.0015751721803098917,
+            -0.0009288155124522746,
+            0.009310834109783173,
+            0.011207638308405876,
+            0.0005079620750620961,
+            -0.012337474152445793,
+            0.00038992727058939636,
+            0.00340600055642426,
+            -0.014382723718881607,
+            -0.019215451553463936,
+            0.009838640689849854,
+            -0.015430090017616749,
+            0.011455047875642776,
+            0.018885571509599686,
+            0.0038245345931500196,
+            -0.008807769045233727,
+            0.015174433588981628,
+            0.017516573891043663,
+            0.0065893325954675674,
+            -0.013384840451180935,
+            0.004230698104947805,
+            -0.0010097390040755272,
+            -0.004737887065857649,
+            0.008824262768030167,
+            -0.004812109749764204,
+            0.020023655146360397,
+            -0.0133600989356637,
+            0.02691812627017498,
+            0.004577070940285921,
+            0.030101457610726357,
+            0.013162171468138695,
+            0.005636807065457106,
+            0.001126742921769619,
+            0.0077232918702065945,
+            0.00780163798481226,
+            0.011982854455709457,
+            -0.0164609607309103,
+            0.013896152377128601,
+            -0.0006267700809985399,
+            0.009813900105655193,
+            -0.019908197224140167,
+            -0.003797731827944517,
+            0.013384840451180935,
+            -0.0033812597393989563,
+            -0.02615940384566784,
+            0.014193043112754822,
+            0.02126070111989975,
+            0.01655992493033409,
+            -0.030530299991369247,
+            0.0164609607309103,
+            -0.013401334173977375,
+            0.015347619540989399,
+            0.0030678745824843645,
+            -0.001799902180209756,
+            -0.02126070111989975,
+            -0.0210957620292902,
+            0.015941401943564415,
+            -0.025301719084382057,
+            -0.009244858287274837,
+            0.008807769045233727,
+            0.0038492754101753235,
+            0.014061091467738152,
+            -0.02617589756846428,
+            -0.008028429932892323,
+            -0.008069665171205997,
+            -0.008296456187963486,
+            0.0036740272771567106,
+            0.002236991887912154,
+            0.006069773342460394,
+            -0.006292441859841347,
+            -0.02149161696434021,
+            -0.018110357224941254,
+            0.0011813790770247579,
+            0.01375595387071371,
+            0.011595246382057667,
+            -0.029441699385643005,
+            0.005748141556978226,
+            0.010886006988584995,
+            -0.0033049751073122025,
+            0.0007545981789007783,
+            -0.024328574538230896,
+            -0.016254786401987076,
+            -0.013549779541790485,
+            -0.006762519478797913,
+            -0.009310834109783173,
+            0.004676034674048424,
+            -0.008424284867942333,
+            -0.004581194370985031,
+            -0.01639498583972454,
+            -0.009665454737842083,
+            0.01297249086201191,
+            0.018687644973397255,
+            -0.005657424684613943,
+            -0.029821060597896576,
+            0.02681916207075119,
+            0.018291790038347244,
+            -0.008824262768030167,
+            -0.01641147956252098,
+            -0.0003654440806712955,
+            -0.016955779865384102,
+            0.008947967551648617,
+            0.004803862888365984,
+            -0.014580651186406612,
+            -0.009112906642258167,
+            -0.010877760127186775,
+            0.013310616835951805,
+            0.020122617483139038,
+            0.004820356611162424,
+            0.0074882530607283115,
+            -0.0035317668225616217,
+            0.023042047396302223,
+            -0.006915088277310133,
+            -2.363756948398077e-06,
+            0.0029833430889993906,
+            -0.00680787768214941,
+            0.007756279781460762,
+            0.023718299344182014,
+            -0.0335816815495491,
+            -0.006853235885500908,
+            -0.002346264198422432,
+            -0.002056589350104332,
+            -0.011949866078794003,
+            -0.011644728481769562,
+            -0.0077356621623039246,
+            0.010778795927762985,
+            0.006750148721039295,
+            -0.0018473223317414522,
+            0.002083391882479191,
+            -0.0133188646286726,
+            0.009129401296377182,
+            -0.024889370426535606,
+            0.002865823684260249,
+            -0.018473222851753235,
+            0.010028321295976639,
+            -0.009657206945121288,
+            -0.00012628179683815688,
+            0.013533285818994045,
+            -0.000562340603210032,
+            0.0015019802376627922,
+            -0.019182464107871056,
+            -0.009805653244256973,
+            0.00791709590703249,
+            0.008436654694378376,
+            0.009294340386986732,
+            -0.02586251311004162,
+            -0.0012298300862312317,
+            0.004601811990141869,
+            0.004170907661318779,
+            -0.0018267049454152584,
+            -0.02726449817419052,
+            -0.0074758827686309814,
+            -0.014894036576151848,
+            -0.02172253094613552,
+            0.0017421734519302845,
+            -0.015149693004786968,
+            -0.013986868783831596,
+            0.024130648002028465,
+            0.02216786891222,
+            0.016551679000258446,
+            -0.004886332433670759,
+            0.01571873389184475,
+            0.006601703353226185,
+            -0.023652324452996254,
+            0.0010679832194000483,
+            -0.014168302528560162,
+            0.01098497025668621,
+            -0.008906732313334942,
+            0.022629698738455772,
+            -0.001510227215476334,
+            -0.010020074434578419,
+            -0.003008084138855338,
+            0.013137430883944035,
+            0.008956214413046837,
+            -0.022382289171218872,
+            -0.009978839196264744,
+            0.003024578094482422,
+            -0.03224566951394081,
+            -0.006622320506721735,
+            0.0025276977103203535,
+            0.0014566219178959727,
+            0.007401659619063139,
+            0.007059410214424133,
+            -0.014539416879415512,
+            -0.0037152620498090982,
+            0.0021524603944271803,
+            -0.01086951233446598,
+            0.005983180366456509,
+            0.015083717182278633,
+            0.017912428826093674,
+            0.011595246382057667,
+            0.007356301415711641,
+            -0.014028104022145271,
+            0.004222451243549585,
+            -0.015174433588981628,
+            -0.0053811511024832726,
+            0.005010037217289209,
+            -0.025252236053347588,
+            -0.012386956252157688,
+            0.02162356860935688,
+            0.02168954350054264,
+            -0.0005592479719780385,
+            -0.023718299344182014,
+            -0.01675785332918167,
+            0.012923009693622589,
+            -0.01309619564563036,
+            0.018885571509599686,
+            -0.0033936300314962864,
+            -0.014259018935263157,
+            -0.0016184687847271562,
+            0.017928922548890114,
+            -0.018275296315550804,
+            0.01598263718187809,
+            -0.02071640081703663,
+            0.009129401296377182,
+            0.002925614360719919,
+            0.0025957352481782436,
+            0.0014875481138005853,
+            0.050273556262254715,
+            -0.00046518092858605087,
+            0.0009396396926604211,
+            0.018737126141786575,
+            -0.033680643886327744,
+            0.02132667601108551,
+            0.003560631303116679,
+            0.00822223350405693,
+            0.0012885897886008024,
+            -0.01319515984505415,
+            0.0105891153216362,
+            0.019677281379699707,
+            0.027874775230884552,
+            -0.010094297118484974,
+            -0.007121262606233358,
+            0.018143344670534134,
+            0.003245184663683176,
+            -0.001950409496203065,
+            0.0029606639873236418,
+            0.004185339901596308,
+            -0.019330909475684166,
+            -0.0036822741385549307,
+            0.011166404001414776,
+            -0.014506428502500057,
+            -0.009624219499528408,
+            -0.014374476857483387,
+            -0.009409798309206963,
+            -0.008964461274445057,
+            -0.010382940992712975,
+            0.0005654331762343645,
+            -0.0026534642092883587,
+            0.014423958957195282,
+            -0.013013726100325584,
+            0.029590144753456116,
+            -0.013302369974553585,
+            -0.024526502937078476,
+            -0.0037544353399425745,
+            -0.007422277238219976,
+            0.02074938826262951,
+            -0.015446583740413189,
+            -0.010127284564077854,
+            -0.0034348650369793177,
+            -0.012535401619970798,
+            0.022448265925049782,
+            0.008931473828852177,
+            -0.006222342606633902,
+            -0.0061604902148246765,
+            0.020172100514173508,
+            0.024295587092638016,
+            -0.0256480909883976,
+            0.02144213393330574,
+            0.020139113068580627,
+            -0.008552112616598606,
+            -0.0033874448854476213,
+            0.002342140767723322,
+            0.005974933039397001,
+            -0.0015050729271024466,
+            0.010646844282746315,
+            0.006226466037333012,
+            -0.013830176554620266,
+            -0.0163784921169281,
+            0.01692279241979122,
+            0.018473222851753235,
+            -0.008750040084123611,
+            0.009063425473868847,
+            -0.003573001828044653,
+            0.002962725702673197,
+            -0.00333796301856637,
+            0.007682057097554207,
+            -0.009987086057662964,
+            -0.017961911857128143,
+            0.011042699217796326,
+            0.0029070586897432804,
+            -0.005715153645724058,
+            0.0037647439166903496,
+            0.023718299344182014,
+            -0.024510009214282036,
+            -0.013566273264586926,
+            0.005488361697643995,
+            -0.02159057930111885,
+            0.014687862247228622,
+            -0.0006107915542088449,
+            0.008824262768030167,
+            0.001291682361625135,
+            0.003952362574636936,
+            -0.024955345317721367,
+            0.009393304586410522,
+            0.008906732313334942,
+            0.0006061526364646852,
+            -0.000622646592091769,
+            -0.009022190235555172,
+            -0.01100146397948265,
+            -0.007381042465567589,
+            -0.006964570377022028,
+            -0.0108365248888731,
+            -0.02061743661761284,
+            0.023883238434791565,
+            0.0022864737547934055,
+            -0.006840865593403578,
+            0.0022947206161916256,
+            0.0136487428098917,
+            -0.0032163201831281185,
+            -0.004317291080951691,
+            0.010077803395688534,
+            -0.017219683155417442,
+            -0.006073896773159504,
+            0.005174976773560047,
+            0.007628451567143202,
+            -0.019710268825292587,
+            -0.009517008438706398,
+            -0.012436438351869583,
+            0.012362214736640453,
+            0.007199608720839024,
+            -0.015512559562921524,
+            -0.01100146397948265,
+            0.00257099443115294,
+            0.005034778267145157,
+            -0.05881742388010025,
+            -0.008543865755200386,
+            0.003925560042262077,
+            0.03879376873373985,
+            -0.003125603310763836,
+            -0.0158836729824543,
+            -0.002467907266691327,
+            0.02736346237361431,
+            0.00016519721248187125,
+            0.02058444917201996,
+            0.006271824240684509,
+            -0.009714935906231403,
+            -0.008477889932692051,
+            0.007591340225189924,
+            -0.005335792433470488,
+            -0.013929140754044056,
+            0.004696652293205261,
+            0.005880093201994896,
+            0.02703358232975006,
+            -0.010514892637729645,
+            0.0005762573564425111,
+            -0.0031379738356918097,
+            -0.001628777477890253,
+            -0.003711138619109988,
+            -0.005253322888165712,
+            0.013162171468138695,
+            -0.015529053285717964,
+            0.014333241619169712,
+            -0.01386316493153572,
+            0.004416254814714193,
+            0.007442894857376814,
+            -0.004511095117777586,
+            -0.002079268451780081,
+            -0.0031585912220180035,
+            0.002257609274238348,
+            0.0071954852901399136,
+            0.004911073483526707,
+            -0.024180129170417786,
+            0.012543648481369019,
+            0.0036987680941820145,
+            0.011784926988184452,
+            0.00779339112341404,
+            -0.023965708911418915,
+            -0.0026163526345044374,
+            0.014168302528560162,
+            0.011438554152846336,
+            0.004100808408111334,
+            0.008494383655488491,
+            -0.008102652616798878,
+            0.015124951489269733,
+            -0.04209255799651146,
+            0.00844490248709917,
+            -0.005789376329630613,
+            0.008275839500129223,
+            -0.011586999520659447,
+            0.009343822486698627,
+            0.007884107530117035,
+            -0.023668818175792694,
+            0.009888122789561749,
+            -0.002232868457213044,
+            -0.014044597744941711,
+            -0.015289891511201859,
+            -0.01055612787604332,
+            -0.00011519993131514639,
+            -0.012551895342767239,
+            0.0017648525536060333,
+            0.0026926372665911913,
+            -0.001965872710570693,
+            -0.024526502937078476,
+            -0.0007612988702021539,
+            -0.009517008438706398,
+            -0.008618088439106941,
+            -0.00555021408945322,
+            -0.009805653244256973,
+            0.012197275646030903,
+            0.008218110539019108,
+            -0.0030039604753255844,
+            -0.01086126547306776,
+            0.02173902466893196,
+            0.0005819271318614483,
+            0.000386576954042539,
+            -0.007987194694578648,
+            -0.004729640204459429,
+            -0.02686864323914051,
+            0.009055178612470627,
+            -0.02206890471279621,
+            -0.0005499701364897192,
+            0.002034940989688039,
+            0.0050760130397975445,
+            -0.014094079844653606,
+            0.0025421299505978823,
+            -0.00138961523771286,
+            0.024163635447621346,
+            0.0031544677913188934,
+            -0.013698224909603596,
+            0.009805653244256973,
+            -0.015496065840125084,
+            0.0108365248888731,
+            0.03278997167944908,
+            0.001166946953162551,
+            0.0011659160954877734,
+            -0.001972057856619358,
+            -0.003702891757711768,
+            0.008424284867942333,
+            0.0019442243501543999,
+            -0.03354869410395622,
+            -0.02655525878071785,
+            -0.009863382205367088,
+            0.011826162226498127,
+            0.0056780423037707806,
+            0.0011174650862812996,
+            -0.010119037702679634,
+            0.01593315601348877,
+            0.006601703353226185,
+            0.02578004263341427,
+            0.014720849692821503,
+            0.013038466684520245,
+            0.005525473039597273,
+            0.0042925504967570305,
+            -0.006275947671383619,
+            -0.009178882464766502,
+            -0.012362214736640453,
+            0.011883890256285667,
+            0.007987194694578648,
+            0.012123052962124348,
+            0.0010772610548883677,
+            -0.00163083930965513,
+            0.010110790841281414,
+            -0.018885571509599686,
+            -0.024114154279232025,
+            0.020007161423563957,
+            0.0020638054702430964,
+            0.002554500475525856,
+            -0.013013726100325584,
+            0.004717269446700811,
+            9.213417069986463e-05,
+            0.008898485451936722,
+            -0.005900710355490446,
+            0.00265140226110816,
+            0.01677434705197811,
+            0.0051502357237041,
+            0.007463512010872364,
+            -0.0012937441933900118,
+            -0.01692279241979122,
+            0.01108393445611,
+            -0.004750257357954979,
+            -0.015487818978726864,
+            -0.014151808805763721,
+            0.01095198281109333,
+            -0.0008185122278518975,
+            0.006226466037333012,
+            -0.0011803482193499804,
+            -0.011133415624499321,
+            0.017368128523230553,
+            -0.024575984105467796,
+            0.006510986480861902,
+            0.001821550540626049,
+            0.024872876703739166,
+            0.0012267375132068992,
+            0.0038616459351032972,
+            -0.020188594236969948,
+            0.028435569256544113,
+            -0.023916225880384445,
+            -0.007945960387587547,
+            -0.011347836814820766,
+            6.86818384565413e-05,
+            -0.0007463512010872364,
+            0.0037853613030165434,
+            -0.008251097984611988,
+            -0.02144213393330574,
+            0.008750040084123611,
+            0.024213118478655815,
+            0.0017081546830013394,
+            0.007748032920062542,
+            0.016386738047003746,
+            0.0016184687847271562,
+            -0.012667353264987469,
+            -0.0105478810146451,
+            -0.011248873546719551,
+            -0.011207638308405876,
+            -0.012255004607141018,
+            0.003812164068222046,
+            -0.018852584064006805,
+            0.015372361056506634,
+            0.007265584543347359,
+            -0.020337039604783058,
+            -0.0015050729271024466,
+            0.006374911405146122,
+            0.013566273264586926,
+            -0.007517117541283369,
+            0.012123052962124348,
+            0.019875209778547287,
+            0.002107101958245039,
+            -0.0006345016299746931,
+            0.00041080242954194546,
+            0.006746025290340185,
+            -0.015661004930734634,
+            0.030018987134099007,
+            0.006869730073958635,
+            0.0063955290243029594,
+            -0.01060560904443264,
+            0.018819596618413925,
+            -0.007261461112648249,
+            0.014481687918305397,
+            -0.018671151250600815,
+            -0.01337659265846014,
+            -0.014745591208338737,
+            0.011166404001414776,
+            -0.010630350559949875,
+            0.003573001828044653,
+            0.03582485765218735,
+            0.008576854132115841,
+            -0.002403993159532547,
+            -0.006016168277710676,
+            -0.015050728805363178,
+            -0.004333785269409418,
+            -0.013953881338238716,
+            -0.000127505962154828,
+            -0.025169767439365387,
+            -0.013409581035375595,
+            0.00807378813624382,
+            0.02122771367430687,
+            0.014836307615041733,
+            0.004671911243349314,
+            0.015067222528159618,
+            -0.007133633363991976,
+            0.01682382822036743,
+            0.018687644973397255,
+            0.01069632638245821,
+            -0.00012067643547197804,
+            -0.009813900105655193,
+            0.0010020073968917131,
+            0.025235742330551147,
+            -0.011199391447007656,
+            -0.00541001558303833,
+            0.02229982055723667,
+            0.02046899124979973,
+            0.0011102489661425352,
+            0.0058594755828380585,
+            0.010127284564077854,
+            0.009079919196665287,
+            0.014168302528560162,
+            -0.004523465409874916,
+            0.012584883719682693,
+            -0.01082003116607666,
+            -0.006506863050162792,
+            0.04816233366727829,
+            0.014448699541389942,
+            -0.0036059897392988205,
+            -0.012197275646030903,
+            -0.0020772067364305258,
+            0.0023895609192550182,
+            0.02107926830649376,
+            0.010069556534290314,
+            0.015289891511201859,
+            0.024608973413705826,
+            -0.007661439478397369,
+            0.014423958957195282,
+            -0.0059172045439481735,
+            0.04410482197999954,
+            -0.017236176878213882,
+            -0.004972925875335932,
+            -0.012535401619970798,
+            0.006840865593403578,
+            -0.00527394050732255,
+            0.018011393025517464,
+            -0.0011545764282345772,
+            0.0007509901188313961,
+            -0.028386086225509644,
+            0.019380390644073486,
+            0.012692093849182129,
+            -0.03829894959926605,
+            0.006234712898731232,
+            -0.009599478915333748,
+            -0.0019493786385282874,
+            -0.02645629458129406,
+            -0.004109055269509554,
+            -0.009987086057662964,
+            -0.0029194289818406105,
+            -0.025351200252771378,
+            0.007302696350961924,
+            -0.025318212807178497,
+            0.009426292032003403,
+            -0.0077851442620158195,
+            -0.0058718458749353886,
+            -0.005661548115313053,
+            -0.009269599802792072,
+            -0.0033523952588438988,
+            0.019495848566293716,
+            0.017203189432621002,
+            0.004878085572272539,
+            0.006783136632293463,
+            0.008914980106055737,
+            -0.011529270559549332,
+            -0.033944547176361084,
+            -0.008519125171005726,
+            -0.01657641865313053,
+            0.00834593828767538,
+            0.014324994757771492,
+            0.02597797103226185,
+            -0.02150811068713665,
+            -0.008799522183835506,
+            -0.009904616512358189,
+            -0.01571873389184475,
+            -0.010663338005542755,
+            0.010102543979883194,
+            -0.004564700648188591,
+            -0.023437902331352234,
+            0.0010468503460288048,
+            -0.005038901697844267,
+            0.004968802444636822,
+            -0.003812164068222046,
+            0.007273831870406866,
+            0.008956214413046837,
+            -0.011652975343167782,
+            -0.03125603497028351,
+            -0.0035297051072120667,
+            -0.001355596468783915,
+            -0.0028452062979340553,
+            -0.025417177006602287,
+            0.007756279781460762,
+            -0.015990883111953735,
+            -0.02112874947488308,
+            0.013294123113155365,
+            -0.007405783515423536,
+            0.006292441859841347,
+            0.018638163805007935,
+            0.013310616835951805,
+            0.010490152053534985,
+            0.033977534621953964,
+            -0.0011525147128850222,
+            -0.014382723718881607,
+            -0.011314849369227886,
+            0.014531169086694717,
+            -0.017005261033773422,
+            0.014011610299348831,
+            -0.007442894857376814,
+            -0.005884216632694006,
+            -0.0002360696526011452,
+            -0.0017968096071854234,
+            0.008857251144945621,
+            -0.009203623980283737,
+            0.00258336472325027,
+            -0.024575984105467796,
+            0.015413595363497734,
+            -0.016155824065208435,
+            -0.009970592334866524,
+            0.010226248763501644,
+            0.031025119125843048,
+            -0.015339372679591179,
+            -0.012634364888072014,
+            -0.0017504204297438264,
+            0.000949948385823518,
+            0.003818349214270711,
+            -0.006242959760129452,
+            -0.008123270235955715,
+            0.0051378654316067696,
+            0.0007159404922276735,
+            -0.016304269433021545,
+            0.0004020400228910148,
+            0.01111692190170288,
+            -0.028485050424933434,
+            -0.017500080168247223,
+            -0.007150127086788416,
+            -0.01566925272345543,
+            0.008267591707408428,
+            0.030101457610726357,
+            0.007084151264280081,
+            -0.006506863050162792,
+            -0.022794637829065323,
+            0.008246975019574165,
+            -0.027643859386444092,
+            0.02098030410706997,
+            0.07903900742530823,
+            -0.004820356611162424,
+            -0.0025441916659474373,
+            -0.018918560817837715,
+            -0.0014267265796661377,
+            0.0032946662977337837,
+            -0.017384622246026993,
+            0.05429808050394058,
+            -0.0014287884114310145,
+            0.004280179738998413,
+            -0.0076738097704946995,
+            0.010886006988584995,
+            -0.0267201978713274,
+            -0.02134316973388195,
+            -0.020205087959766388,
+            -0.010366447269916534,
+            -0.00543475616723299,
+            -0.0036245454102754593,
+            -0.013541532680392265,
+            0.0018225813983008265,
+            0.011958112940192223,
+            -0.00833769142627716,
+            0.0064078993164002895,
+            -0.024114154279232025,
+            -0.007166620809584856,
+            0.011067439801990986,
+            0.014885789714753628,
+            0.027990233153104782,
+            -0.030134445056319237,
+            -0.014588898047804832,
+            -0.007146003656089306,
+            0.008110899478197098,
+            -0.009302587248384953,
+            0.007644945755600929,
+            -0.0208483524620533,
+            -0.005908957216888666,
+            -0.010036568157374859,
+            0.014094079844653606,
+            0.014836307615041733,
+            -0.0022204979322850704,
+            -0.01058086846023798,
+            0.012090064585208893,
+            0.029227279126644135,
+            0.0015421842690557241,
+            0.009690195322036743,
+            -0.011677715927362442,
+            -0.011347836814820766,
+            0.0036925829481333494,
+            0.03151993826031685,
+            0.00652335723862052,
+            -0.00780576141551137,
+            -0.005785252898931503,
+            0.024641960859298706,
+            -0.004191524814814329,
+            0.01100146397948265,
+            -0.008939720690250397,
+            0.023157505318522453,
+            0.0016442405758425593,
+            0.009978839196264744,
+            0.011422060430049896,
+            -0.011933372355997562,
+            0.02076588198542595,
+            -0.0007344961632043123,
+            0.004589441232383251,
+            -0.0011009711306542158,
+            0.0017730995314195752,
+            -0.004737887065857649,
+            0.0040533882565796375,
+            0.003193640848621726,
+            -0.0013184851268306375,
+            0.010770549066364765,
+            3.614494562498294e-05,
+            0.005777005571871996,
+            -0.018951548263430595,
+            -0.012527154758572578,
+            0.013854917138814926,
+            -0.034406378865242004,
+            0.000833459896966815,
+            0.011191144585609436,
+            -0.02094731666147709,
+            0.02206890471279621,
+            -0.0077810208313167095,
+            -0.023256469517946243,
+            -0.007851120084524155,
+            0.00833769142627716,
+            0.004799739457666874,
+            -0.004968802444636822,
+            0.007884107530117035,
+            -0.007063533645123243,
+            0.005946068558841944,
+            -0.0024184253998100758,
+            0.02037002705037594,
+            -0.022431770339608192,
+            0.020534968003630638,
+            0.0005891432520002127,
+            0.01593315601348877,
+            -0.008849004283547401,
+            0.011273614130914211,
+            0.011908631771802902,
+            0.004498724825680256,
+            0.013722965493798256,
+            0.006003797519952059,
+            -0.014976506121456623,
+            0.014432205818593502,
+            0.008378926664590836,
+            0.009145895019173622,
+            0.02211838588118553,
+            0.014704355970025063,
+            0.011240626685321331,
+            -0.009517008438706398,
+            0.008947967551648617,
+            -0.011504529975354671,
+            -0.0015153816202655435,
+            0.01682382822036743,
+            0.02196994051337242,
+            -0.013945634476840496,
+            -0.004902826622128487,
+            -0.019034016877412796,
+            0.015240409411489964,
+            -0.0006375942612066865,
+            0.012692093849182129,
+            0.018390754237771034,
+            0.004882209002971649,
+            0.008980955928564072,
+            0.019743258133530617,
+            0.04215853661298752,
+            0.004799739457666874,
+            -0.0062182191759347916,
+            0.013978621922433376,
+            0.00795833021402359,
+            -0.005785252898931503,
+            0.00679963082075119,
+            -0.01674135960638523,
+            0.0003015300026163459,
+            0.009500514715909958,
+            -0.0025318211410194635,
+            -0.010020074434578419,
+            0.005983180366456509,
+            0.008741793222725391,
+            0.011413812637329102,
+            -0.012205522507429123,
+            0.010086050257086754,
+            -0.000819543085526675,
+            -0.004807986319065094,
+            -0.02063393034040928,
+            -0.013252888806164265,
+            0.03153643012046814,
+            -0.01348380371928215,
+            -0.008102652616798878,
+            0.02162356860935688,
+            0.017054744064807892,
+            0.004585317801684141,
+            -0.017582550644874573,
+            -0.005851228721439838,
+            0.012461178936064243,
+            -0.0011504529975354671,
+            0.023899732157588005,
+            -0.011240626685321331,
+            0.008634582161903381,
+            -0.014357983134686947,
+            -0.009376809932291508,
+            0.0011514838552102447,
+            0.012370462529361248,
+            0.011471541598439217,
+            0.02749541401863098,
+            0.014720849692821503,
+            -0.004086376167833805,
+            -0.0010834463173523545,
+            0.00791709590703249,
+            0.013533285818994045,
+            0.023108022287487984,
+            -0.0026122292038053274,
+            0.005047148559242487,
+            0.020089630037546158,
+            -0.00554609065875411,
+            0.012535401619970798,
+            -0.022382289171218872,
+            -0.014770331792533398,
+            -0.007574846502393484,
+            0.01099321711808443,
+            -0.011974607594311237,
+            0.017467092722654343,
+            0.0013102381490170956,
+            0.02691812627017498,
+            -0.0026823284570127726,
+            0.0028988115955144167,
+            0.0037647439166903496,
+            0.003985350485891104,
+            -0.0012257066555321217,
+            0.00805729441344738,
+            -0.02101329155266285,
+            0.01046541053801775,
+            -0.008519125171005726,
+            0.003183332271873951,
+            -0.01616406999528408,
+            0.0025895501021295786,
+            -0.007954207248985767,
+            -0.024312080815434456,
+            9.664423851063475e-05,
+            -0.014646627008914948,
+            -0.00529043423011899,
+            0.0018431989010423422,
+            0.004704899154603481,
+            -0.00820573978126049,
+            0.007966578006744385,
+            0.007071780972182751,
+            -0.014927024021744728,
+            0.0009128370438702404,
+            -0.00514198886230588,
+            -0.0001724133180687204,
+            0.0067047905176877975
+          ],
+          "keyphrases": [
+            "conditional computation",
+            "deep networks",
+            "Google Brain",
+            "neural machine translation",
+            "algorithmic and engineering solutions"
+          ],
+          "title": "6 Conclusion"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "d731f785-b03d-4f3b-86e2-f2a1a21970e4",
+      "type": "next",
+      "source": {
+        "id": "a004def7-e6ff-4a78-ad92-424e0eff3d92",
+        "properties": {
+          "page_content": "6 Conclusion\n\nThis work is the first to demonstrate major wins from conditional computation in deep networks.\n\nWe carefully identified the design considerations and challenges of conditional computing and addressed them with a combination of algorithmic and engineering solutions. While we focused on text, conditional computation may help in other domains as well, provided sufficiently large training sets. We look forward to seeing many novel implementations and applications of conditional computation in the years to come.\n\nAcknowledgments\n\nWe would like to thank all of the members of the Google Brain and Google Translate teams who helped us with this project, in particular Zhifeng Chen, Yonghui Wu, and Melvin Johnson. Thanks also to our anonymous ICLR reviewers for the helpful suggestions on making this paper better.\n\n2Reported perplexities relative to the tokenization used by both our models and GNMT.\n\nReferences\n\nMartín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Gordon Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda B. Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. CoRR, abs/1603.04467, 2016. URL http://arxiv.org/abs/1603.04467.\n\nRahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. CoRR, abs/1611.06194, 2016. URL http://arxiv.org/abs/1611.\n\n06194.\n\nA. Almahairi, N. Ballas, T. Cooijmans, Y. Zheng, H. Larochelle, and A. Courville. Dynamic Capacity Networks. ArXiv e-prints, November 2015.\n\nDario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan, Christopher Fougner, Tony Han, Awni Y. Hannun, Billy Jun, Patrick LeGresley, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595, 2015. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.\n\nEmmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297, 2015.\n\nYoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.\n\nCiprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling.\n\narXiv preprint arXiv:1312.3005, 2013.\n\nK. Cho and Y. Bengio. Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning. ArXiv e-prints, June 2014. Ronan Collobert, Samy Bengio, and Yoshua Bengio. A parallel mixture of SVMs for very large scale problems. Neural Computing, 2002.\n\nAndrew Davis and Itamar Arel. Low-rank approximations for conditional feedforward computation in deep neural networks. arXiv preprint arXiv:1312.4461, 2013. Marc Peter Deisenroth and Jun Wei Ng. Distributed Gaussian processes. In ICML, 2015.\n\nJohn Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization, 2010.\n\nNadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. Edinburgh's phrase-based machine translation systems for wmt-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation, 2014.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013. Ekaterina Garmash and Christof Monz. Ensemble learning for multi-source neural machine translation. In staff.science.uva.nl/c.monz, 2016.\n\nFelix A. Gers, Jürgen A. Schmidhuber, and Fred A. Cummins. Learning to forget: Continual prediction with lstm. Neural Computation, 2000.\n\nAudrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. Memory-efficient backpropagation through time. CoRR, abs/1606.03401, 2016. URL http://arxiv.org/ abs/1606.03401. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2015.\n\nGeoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 2012. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.\n\nSergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.\n\nRobert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computing, 1991.\n\nMelvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation.\n\nCoRR, abs/1611.04558, 2016. URL http://arxiv.org/abs/1611.04558.\n\nMichael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm.\n\nNeural Computing, 1994. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.\n\nDiederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.\n\nReinhard Kneser and Hermann. Ney. Improved backingoff for m-gram language modeling., 1995.\n\nAlex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.\n\nQuoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeffrey Dean, and Andrew Y. Ng. Building high-level features using large scale unsupervised learning. In ICML, 2012. Patrick Gallinari Ludovic Denoyer. Deep sequential neural network. arXiv preprint arXiv:1410.0510, 2014.\n\nMinh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attentionbased neural machine translation. EMNLP, 2015a.\n\nMinh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the rare word problem in neural machine translation. ACL, 2015b.\n\nCarl Edward Rasmussen and Zoubin Ghahramani. Infinite mixtures of Gaussian process experts.\n\nNIPS, 2002.\n\nHasim Sak, Andrew W Senior, and Françoise Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH, pp. 338–342, 2014.\n\nMike Schuster and Kaisuke Nakajima. Japanese and Korean voice search. ICASSP, 2012.\n\nBabak Shahbaba and Radford Neal. Nonlinear models using dirichlet process mixtures. JMLR, 2009.\n\nIlya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks.\n\nIn NIPS, 2014.\n\nLucas Theis and Matthias Bethge. Generative image modeling using spatial LSTMs. In NIPS, 2015. Volker Tresp. Mixtures of Gaussian Processes. In NIPS, 2001.\n\nYonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.\n\nBangpeng Yao, Dirk Walther, Diane Beck, and Li Fei-fei. Hierarchical mixture of classification experts uncovers interactions between brain regions. In NIPS. 2009.\n\nWojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization.\n\narXiv preprint arXiv:1409.2329, 2014.\n\nJie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with fast-forward connections for neural machine translation. arXiv preprint arXiv:1606.04199, 2016.\n\n",
+          "embedding": [
+            0.012222016230225563,
+            0.0018153652781620622,
+            -0.018968041986227036,
+            -0.007018175441771746,
+            0.012403449974954128,
+            0.003484346903860569,
+            0.006766642909497023,
+            0.03404351323843002,
+            -0.030365360900759697,
+            0.03598979860544205,
+            0.007698550820350647,
+            -0.006844989024102688,
+            0.017252670601010323,
+            -0.018192825838923454,
+            0.0035915574990212917,
+            0.014209537766873837,
+            0.014687862247228622,
+            0.004014214966446161,
+            -0.011182897724211216,
+            -0.040674079209566116,
+            -0.03125603497028351,
+            -0.017648525536060333,
+            -0.03161890059709549,
+            -0.009582984261214733,
+            0.016172317788004875,
+            0.03310335800051689,
+            0.022712169215083122,
+            0.017153708264231682,
+            -0.045556288212537766,
+            0.015842437744140625,
+            0.028155172243714333,
+            0.003665780182927847,
+            -0.02747892029583454,
+            -0.023569853976368904,
+            -0.015380607917904854,
+            -0.03267451375722885,
+            0.06508512794971466,
+            0.03876078128814697,
+            -0.014654873870313168,
+            0.019149474799633026,
+            0.04321414604783058,
+            -0.011900384910404682,
+            -0.007348054554313421,
+            -0.015990883111953735,
+            -0.004210080485790968,
+            -0.002177201444283128,
+            -0.014085832983255386,
+            -0.015174433588981628,
+            -0.030051976442337036,
+            -0.007912972010672092,
+            0.028105689212679863,
+            -0.028353098779916763,
+            0.04743659868836403,
+            -0.003750311676412821,
+            -0.016485702246427536,
+            0.04671086370944977,
+            0.010762302204966545,
+            0.008997449651360512,
+            0.06310585141181946,
+            0.04139981418848038,
+            0.003208073088899255,
+            0.018192825838923454,
+            -0.023899732157588005,
+            -0.006424393504858017,
+            0.0322621650993824,
+            -0.02089783363044262,
+            0.0017978404648602009,
+            0.025433670729398727,
+            0.039783407002687454,
+            -0.015347619540989399,
+            -0.03783712163567543,
+            0.029045844450592995,
+            -0.010498398914933205,
+            -0.015957895666360855,
+            -0.00029766425723209977,
+            -0.0036699038464576006,
+            0.029029350727796555,
+            0.05367131158709526,
+            0.005014160647988319,
+            0.004304920788854361,
+            0.012782811187207699,
+            0.017219683155417442,
+            -0.004572947509586811,
+            -0.006494492758065462,
+            0.023157505318522453,
+            0.002239053603261709,
+            -0.005183223634958267,
+            -0.05383625254034996,
+            0.03328479081392288,
+            -0.04740361124277115,
+            -0.039816394448280334,
+            0.018225813284516335,
+            0.006473875138908625,
+            -0.01018501352518797,
+            -0.009624219499528408,
+            0.029656121507287025,
+            -0.00020127772586420178,
+            -0.000498168810736388,
+            -0.018918560817837715,
+            0.014357983134686947,
+            -0.023784274235367775,
+            0.00528631079941988,
+            0.013285876251757145,
+            -0.013665237464010715,
+            0.027577882632613182,
+            0.009541749954223633,
+            -0.013541532680392265,
+            0.004121425561606884,
+            0.01654343120753765,
+            0.09295989573001862,
+            -0.0011030328460037708,
+            -0.022530734539031982,
+            -0.05314350500702858,
+            0.03267451375722885,
+            -0.02602745220065117,
+            -0.019512342289090157,
+            -0.051791001111269,
+            -0.039882369339466095,
+            0.0015978513984009624,
+            -0.00780988484621048,
+            0.03721034899353981,
+            0.03533003851771355,
+            0.034241437911987305,
+            -0.011974607594311237,
+            -0.029342735186219215,
+            0.055914487689733505,
+            -0.002408116590231657,
+            -0.0425214022397995,
+            -0.0034740380942821503,
+            -0.023965708911418915,
+            -0.007587216794490814,
+            0.008593347854912281,
+            -0.040575116872787476,
+            0.003964733332395554,
+            0.03285594657063484,
+            -0.018638163805007935,
+            0.029342735186219215,
+            0.03582485765218735,
+            -0.025417177006602287,
+            -0.0004953339230269194,
+            0.011232379823923111,
+            -0.0033709509298205376,
+            0.016733111813664436,
+            -0.01296424400061369,
+            -0.034208450466394424,
+            -0.0008540773415006697,
+            -0.015990883111953735,
+            0.0541001558303833,
+            -0.009459280408918858,
+            0.030942648649215698,
+            -0.03885974362492561,
+            -0.0009102598414756358,
+            -0.014267266727983952,
+            0.034538328647613525,
+            0.029606638476252556,
+            -0.009046930819749832,
+            0.0019669034518301487,
+            0.009615972638130188,
+            -0.009541749954223633,
+            0.010201508179306984,
+            -0.00514198886230588,
+            -0.03315283730626106,
+            -0.007966578006744385,
+            0.03757321834564209,
+            0.009756171144545078,
+            0.04097096994519234,
+            0.04737062379717827,
+            0.008750040084123611,
+            -0.04882209002971649,
+            0.04700775444507599,
+            0.034439366310834885,
+            0.01601562462747097,
+            0.001114372513256967,
+            -0.018291790038347244,
+            0.01057262159883976,
+            0.01364049594849348,
+            0.033878572285175323,
+            0.009921110235154629,
+            -0.040179260075092316,
+            0.024790406227111816,
+            -0.0025256359949707985,
+            0.010119037702679634,
+            0.04885507747530937,
+            0.02596147730946541,
+            0.040179260075092316,
+            0.003092615632340312,
+            0.029161302372813225,
+            0.028204653412103653,
+            0.029260266572237015,
+            -0.007599587086588144,
+            0.008585100993514061,
+            0.02132667601108551,
+            0.045655250549316406,
+            -0.034769244492053986,
+            0.029903531074523926,
+            -0.039354562759399414,
+            0.0009916987037286162,
+            0.028649989515542984,
+            -0.024526502937078476,
+            0.004700775723904371,
+            0.00834593828767538,
+            -0.055947475135326385,
+            -0.009632466360926628,
+            0.005344039760529995,
+            0.00844490248709917,
+            0.007591340225189924,
+            -0.012667353264987469,
+            0.004255439154803753,
+            0.0053234221413731575,
+            -0.040674079209566116,
+            -0.006354294251650572,
+            0.004333785269409418,
+            -0.003735879436135292,
+            -0.017829960212111473,
+            0.012560142204165459,
+            -0.004383266903460026,
+            -0.003599804360419512,
+            -0.005364656914025545,
+            -0.025631597265601158,
+            -0.009005696512758732,
+            -0.01664239540696144,
+            0.020403016358613968,
+            0.04215853661298752,
+            -0.024856381118297577,
+            -0.016320763155817986,
+            -0.017928922548890114,
+            -0.03556095436215401,
+            -0.030117951333522797,
+            0.017566056922078133,
+            0.00014135829405859113,
+            0.03688047081232071,
+            0.02172253094613552,
+            -0.0031750851776450872,
+            0.004581194370985031,
+            0.02591199427843094,
+            -0.017269164323806763,
+            -0.0021524603944271803,
+            0.004577070940285921,
+            -0.007756279781460762,
+            0.0136487428098917,
+            -0.003177147125825286,
+            0.01680733449757099,
+            -0.00515848258510232,
+            -0.00132157769985497,
+            -0.001624654047191143,
+            -0.01583419181406498,
+            -0.017599044367671013,
+            0.01364049594849348,
+            -0.0008695403812453151,
+            0.010069556534290314,
+            -0.018935054540634155,
+            -0.04262036457657814,
+            0.013896152377128601,
+            -0.010003580711781979,
+            -0.05927925556898117,
+            0.06967044621706009,
+            0.02676968090236187,
+            -0.014621886424720287,
+            -0.00333796301856637,
+            -0.050471484661102295,
+            0.05802571401000023,
+            0.004453366156667471,
+            -0.02221735008060932,
+            0.05353936180472374,
+            -0.007117139175534248,
+            0.03829894959926605,
+            0.05241777002811432,
+            0.022827625274658203,
+            0.010044815018773079,
+            -0.02714904025197029,
+            0.05268167331814766,
+            -0.04205957055091858,
+            -0.00666767917573452,
+            -0.034670282155275345,
+            0.016881557181477547,
+            -0.04278530552983284,
+            -0.004498724825680256,
+            0.018324777483940125,
+            0.010877760127186775,
+            0.03592382371425629,
+            0.023619335144758224,
+            0.015908414497971535,
+            -0.03145396336913109,
+            -0.004395637661218643,
+            -0.00642026960849762,
+            -0.013879658654332161,
+            -0.03737528994679451,
+            -0.0037626822013407946,
+            -0.06696543842554092,
+            -0.004630676470696926,
+            -0.014811567030847073,
+            0.01346730999648571,
+            -0.05264868587255478,
+            0.029458193108439445,
+            0.018291790038347244,
+            0.020419510081410408,
+            0.007104768883436918,
+            -0.024180129170417786,
+            0.019281426444649696,
+            0.011446801014244556,
+            -0.017186695709824562,
+            -0.040113285183906555,
+            0.004412131384015083,
+            -0.01324464101344347,
+            -0.022745156660676003,
+            0.0018205196829512715,
+            0.019842220470309258,
+            0.029606638476252556,
+            0.005950192455202341,
+            0.004560577217489481,
+            -0.0032225053291767836,
+            0.017516573891043663,
+            0.03180033341050148,
+            -0.019693775102496147,
+            -0.03373012691736221,
+            0.040014322847127914,
+            0.025747055187821388,
+            0.003550322726368904,
+            -0.017120718955993652,
+            -0.03123953938484192,
+            0.051164232194423676,
+            -0.0006412022630684078,
+            -0.06333676725625992,
+            0.012329227291047573,
+            -0.006073896773159504,
+            0.002081330167129636,
+            -0.0012401387793943286,
+            0.05159307271242142,
+            0.002733872039243579,
+            0.014935270883142948,
+            0.0019091746071353555,
+            -0.050471484661102295,
+            -0.018522705882787704,
+            -0.007360424846410751,
+            -0.023091528564691544,
+            -0.039156634360551834,
+            -0.008061417378485203,
+            0.03816699981689453,
+            -0.006135749164968729,
+            -0.017401115968823433,
+            -0.03602278605103493,
+            -0.02089783363044262,
+            -0.009211870841681957,
+            -0.001520535908639431,
+            0.004915196914225817,
+            -0.030563287436962128,
+            -0.01624654047191143,
+            -0.0009813900105655193,
+            0.0069480761885643005,
+            0.001200965722091496,
+            0.0013545656111091375,
+            0.015339372679591179,
+            0.005059518851339817,
+            0.006865606643259525,
+            0.010613856837153435,
+            -0.018902065232396126,
+            -0.02114524319767952,
+            0.012180781923234463,
+            -0.007558352313935757,
+            0.012222016230225563,
+            -0.007113015744835138,
+            -0.0006963539053685963,
+            -0.029870541766285896,
+            0.007587216794490814,
+            -0.024411045014858246,
+            -0.03589083254337311,
+            -0.023701805621385574,
+            -0.0026101674884557724,
+            0.023157505318522453,
+            0.001592696993611753,
+            -0.003183332271873951,
+            0.03905767202377319,
+            -0.023982202634215355,
+            -0.011438554152846336,
+            -0.0059172045439481735,
+            -0.010754055343568325,
+            0.03849687799811363,
+            0.0035400139167904854,
+            -0.011158157140016556,
+            0.017928922548890114,
+            -0.025730561465024948,
+            -0.00416884571313858,
+            0.05838857963681221,
+            0.0018669088603928685,
+            0.0066429381258785725,
+            0.02089783363044262,
+            -0.0025297594256699085,
+            -0.0050224075093865395,
+            -0.02139265276491642,
+            0.017945416271686554,
+            0.0020297865848988295,
+            -0.040212247520685196,
+            -0.023239973932504654,
+            -0.039849381893873215,
+            -0.006725407671183348,
+            0.028715966269373894,
+            0.024724429473280907,
+            0.004436872433871031,
+            0.015199174173176289,
+            -0.005067766178399324,
+            -0.002641093684360385,
+            -0.018093861639499664,
+            0.003655471606180072,
+            0.009756171144545078,
+            -0.01081178430467844,
+            0.0016401171451434493,
+            -0.003805978922173381,
+            -0.0052945576608181,
+            0.04126786068081856,
+            0.004010091535747051,
+            -0.009079919196665287,
+            -0.018869077786803246,
+            -0.055287718772888184,
+            -0.034670282155275345,
+            0.034538328647613525,
+            0.014630133286118507,
+            -0.010622103698551655,
+            -0.01332711149007082,
+            0.0016164070693776011,
+            -0.04354402795433998,
+            -0.01296424400061369,
+            -0.009838640689849854,
+            0.02579653635621071,
+            0.014861048199236393,
+            -0.0028287123423069715,
+            0.020551461726427078,
+            0.022959576919674873,
+            0.013558026403188705,
+            -0.007141880225390196,
+            -0.03278997167944908,
+            0.011372578330338001,
+            0.006919211708009243,
+            0.04591915383934975,
+            0.008626335300505161,
+            -0.0047914921306073666,
+            -0.04921794682741165,
+            -0.024575984105467796,
+            0.018159838393330574,
+            -0.03819998726248741,
+            0.001192718744277954,
+            0.013541532680392265,
+            -0.01623004674911499,
+            0.0058677224442362785,
+            0.0003208588750567287,
+            -0.02205241098999977,
+            -0.0161310825496912,
+            -0.004461613483726978,
+            -0.034208450466394424,
+            -0.009929358027875423,
+            -0.04773348942399025,
+            -0.06749323755502701,
+            0.02655525878071785,
+            -0.040278226137161255,
+            -0.0648542121052742,
+            0.03592382371425629,
+            -0.03655059263110161,
+            0.006510986480861902,
+            -0.040278226137161255,
+            0.0021019477862864733,
+            -0.060268890112638474,
+            0.017961911857128143,
+            0.034175463020801544,
+            0.015710486099123955,
+            -0.0021132873371243477,
+            0.022398782894015312,
+            -0.02736346237361431,
+            -0.040443163365125656,
+            -0.06752622872591019,
+            -0.017780477181077003,
+            -0.01351679116487503,
+            -0.0031503443606197834,
+            0.002323585096746683,
+            0.05284661427140236,
+            0.0004986842395737767,
+            0.024674948304891586,
+            0.022547228261828423,
+            -0.0012978676240891218,
+            -0.04885507747530937,
+            -0.024724429473280907,
+            0.011611740104854107,
+            0.03876078128814697,
+            -0.030530299991369247,
+            0.051395148038864136,
+            -0.05258271098136902,
+            0.009665454737842083,
+            -0.009409798309206963,
+            0.006036785431206226,
+            0.019116487354040146,
+            0.007092398125678301,
+            -0.02203591726720333,
+            0.003894633846357465,
+            0.023404914885759354,
+            -0.011595246382057667,
+            0.03585784509778023,
+            -0.012692093849182129,
+            -0.034538328647613525,
+            -0.009442785754799843,
+            -0.007661439478397369,
+            0.03750723972916603,
+            0.011182897724211216,
+            -0.016172317788004875,
+            -0.028798434883356094,
+            0.011784926988184452,
+            0.02111225575208664,
+            0.03866181895136833,
+            -0.02229982055723667,
+            -0.022761650383472443,
+            0.009393304586410522,
+            -0.035692907869815826,
+            0.018027886748313904,
+            -0.017170201987028122,
+            0.005900710355490446,
+            -0.009442785754799843,
+            -0.045226410031318665,
+            -0.0008643860346637666,
+            -0.00010920798376901075,
+            -0.015891920775175095,
+            0.01045716367661953,
+            0.0061604902148246765,
+            0.018638163805007935,
+            0.04113591089844704,
+            -0.02221735008060932,
+            -0.002874070778489113,
+            0.0772576555609703,
+            -0.015537300147116184,
+            0.018819596618413925,
+            0.01659291237592697,
+            -0.01568574644625187,
+            0.02652227133512497,
+            -0.025829525664448738,
+            -0.028419075533747673,
+            0.017467092722654343,
+            -0.01558678224682808,
+            -0.030398348346352577,
+            -0.007467635441571474,
+            -0.003577125258743763,
+            0.010910747572779655,
+            0.02193695306777954,
+            -0.03293841704726219,
+            -0.004898703191429377,
+            -0.019825726747512817,
+            0.03311984986066818,
+            -0.03366415202617645,
+            0.007327436935156584,
+            9.535564458929002e-05,
+            -0.010259236209094524,
+            0.040542129427194595,
+            -0.017763983458280563,
+            -0.006127502303570509,
+            0.006292441859841347,
+            -0.010020074434578419,
+            -0.003263740334659815,
+            0.008411914110183716,
+            0.01359101478010416,
+            -0.02037002705037594,
+            0.0034328033216297626,
+            -0.0009437631815671921,
+            -0.0008262437768280506,
+            -0.004927567206323147,
+            -0.03671552985906601,
+            -0.014687862247228622,
+            0.006238836329430342,
+            -0.023800769820809364,
+            -0.0015256903134286404,
+            0.004939937964081764,
+            -0.03876078128814697,
+            0.0065893325954675674,
+            0.02144213393330574,
+            -0.0028431445825845003,
+            0.006045032292604446,
+            -0.0032431227155029774,
+            -0.005500731989741325,
+            -0.0009633497102186084,
+            -0.0018246431136503816,
+            0.017994899302721024,
+            0.04644696041941643,
+            0.008593347854912281,
+            0.011479788459837437,
+            0.007508870679885149,
+            -0.012469425797462463,
+            0.017516573891043663,
+            -0.022547228261828423,
+            0.007009928580373526,
+            0.03754022717475891,
+            0.02627486176788807,
+            0.011826162226498127,
+            0.003166838316246867,
+            0.0007674840744584799,
+            0.018951548263430595,
+            -0.034208450466394424,
+            0.024872876703739166,
+            0.014234278351068497,
+            -0.000996337621472776,
+            -0.008865498006343842,
+            -0.01110042817890644,
+            2.0150322598055936e-05,
+            0.017945416271686554,
+            0.008939720690250397,
+            -0.004106993321329355,
+            -0.03536302596330643,
+            0.002777168760076165,
+            0.018489716574549675,
+            0.010119037702679634,
+            -0.010894253849983215,
+            -0.024064673110842705,
+            -0.024180129170417786,
+            9.25851782085374e-05,
+            0.02180500142276287,
+            0.014432205818593502,
+            0.019231945276260376,
+            0.0269676074385643,
+            0.029260266572237015,
+            0.02693461999297142,
+            0.008733546361327171,
+            0.012114806100726128,
+            0.011521023698151112,
+            0.006424393504858017,
+            -0.027858281508088112,
+            -0.007657316047698259,
+            0.025466658174991608,
+            0.019594812765717506,
+            0.002787477569654584,
+            0.025450164452195168,
+            0.00653572753071785,
+            -0.005187347065657377,
+            0.020534968003630638,
+            -0.0012937441933900118,
+            -0.0324271060526371,
+            -0.0074758827686309814,
+            -0.015265149995684624,
+            0.008428407832980156,
+            -0.003253431525081396,
+            -0.0032266287598758936,
+            -0.017285659909248352,
+            0.014209537766873837,
+            0.012857033871114254,
+            0.0014988876646384597,
+            0.00264315539970994,
+            -0.0010659215040504932,
+            -0.015017741359770298,
+            -0.013788941316306591,
+            0.00541001558303833,
+            -0.013887905515730381,
+            0.056013453751802444,
+            -0.0214256402105093,
+            -0.016469208523631096,
+            -0.017648525536060333,
+            -0.006787260062992573,
+            0.017747489735484123,
+            0.01654343120753765,
+            0.014102326706051826,
+            0.018308283761143684,
+            0.00041441048961132765,
+            0.04370896518230438,
+            0.00416884571313858,
+            -0.03625370189547539,
+            0.002042157109826803,
+            -0.01590016670525074,
+            0.011512776836752892,
+            0.017302153632044792,
+            0.0042471918277442455,
+            0.002052465919405222,
+            0.01359926164150238,
+            -0.006432640366256237,
+            0.023239973932504654,
+            -0.008758286945521832,
+            -0.0023792521096765995,
+            0.008980955928564072,
+            0.004750257357954979,
+            0.012733329087495804,
+            0.005174976773560047,
+            -0.028748953714966774,
+            -0.014019857160747051,
+            0.01084477175027132,
+            0.005038901697844267,
+            -0.011281860992312431,
+            0.030480818822979927,
+            0.01591666042804718,
+            0.018176332116127014,
+            0.004350278992205858,
+            -0.010077803395688534,
+            -0.012287992052733898,
+            0.0014751775888726115,
+            0.008003689348697662,
+            0.023767780512571335,
+            0.019594812765717506,
+            -0.013162171468138695,
+            0.0056244367733597755,
+            -0.04189463332295418,
+            0.012518907897174358,
+            0.009112906642258167,
+            -0.015875427052378654,
+            -0.0036719655618071556,
+            0.01636199839413166,
+            0.0045440830290317535,
+            0.00045848023728467524,
+            -0.008791275322437286,
+            0.039915356785058975,
+            0.024806899949908257,
+            0.008914980106055737,
+            -0.0063955290243029594,
+            -0.014753838069736958,
+            -0.002406054874882102,
+            0.016634147614240646,
+            0.028946882113814354,
+            0.011570505797863007,
+            0.014061091467738152,
+            0.010795289650559425,
+            -0.023338938131928444,
+            0.012230263091623783,
+            0.028072701767086983,
+            0.044929519295692444,
+            0.016485702246427536,
+            0.03340024873614311,
+            0.027858281508088112,
+            0.0027379957027733326,
+            0.009327328763902187,
+            -0.02068341337144375,
+            -0.014778578653931618,
+            0.003181270556524396,
+            -0.006197601556777954,
+            0.04684281721711159,
+            -0.0071872384287416935,
+            -0.05287960171699524,
+            -0.016502195969223976,
+            0.0025668710004538298,
+            0.02164006233215332,
+            0.020007161423563957,
+            0.004424502141773701,
+            0.0006118224118836224,
+            0.02221735008060932,
+            -0.012065324001014233,
+            -0.013986868783831596,
+            0.011578752659261227,
+            0.0071789915673434734,
+            -0.018588680773973465,
+            0.013945634476840496,
+            0.019067006185650826,
+            -0.003026639809831977,
+            -0.009937604889273643,
+            -0.0036080514546483755,
+            0.024592477828264236,
+            0.006725407671183348,
+            -2.0440254957065918e-05,
+            0.0013968313578516245,
+            0.009269599802792072,
+            0.0027627365197986364,
+            0.04176267981529236,
+            -0.005789376329630613,
+            0.009764418005943298,
+            -0.012271498329937458,
+            -0.004201833624392748,
+            0.03377960994839668,
+            -0.03509912267327309,
+            -0.015496065840125084,
+            0.019248438999056816,
+            0.013665237464010715,
+            0.002488524653017521,
+            0.0017256794963032007,
+            -0.004972925875335932,
+            -0.012213769368827343,
+            0.0316024087369442,
+            -0.0056120664812624454,
+            -0.006605826783925295,
+            0.039651453495025635,
+            -0.006985187530517578,
+            -0.005702782887965441,
+            0.028204653412103653,
+            -0.0015782647533342242,
+            -0.03241061046719551,
+            0.014226031489670277,
+            -0.008552112616598606,
+            0.03552796691656113,
+            0.015792956575751305,
+            -0.012172535061836243,
+            -0.011554011143743992,
+            0.013929140754044056,
+            -0.009879875928163528,
+            -0.023075034841895103,
+            0.0029586022719740868,
+            -0.022580217570066452,
+            0.013986868783831596,
+            -0.009797406382858753,
+            0.022365795448422432,
+            -0.006737778428941965,
+            0.017615538090467453,
+            0.018407247960567474,
+            0.008440778590738773,
+            -0.017417611554265022,
+            0.011306602507829666,
+            -0.02121121808886528,
+            -0.05462796241044998,
+            0.010135532356798649,
+            -0.002533883089199662,
+            0.017153708264231682,
+            -0.011158157140016556,
+            -0.009541749954223633,
+            0.003568878397345543,
+            -0.008659323677420616,
+            -0.05185697600245476,
+            0.0067006670869886875,
+            -0.023718299344182014,
+            0.014308501034975052,
+            -0.02104627899825573,
+            -0.027957243844866753,
+            -0.015776462852954865,
+            -0.04361000284552574,
+            -0.014762084931135178,
+            -0.0130714550614357,
+            0.015289891511201859,
+            0.016238292679190636,
+            -0.011677715927362442,
+            -0.018275296315550804,
+            0.030744722113013268,
+            0.012436438351869583,
+            -0.006754272151738405,
+            -0.016972273588180542,
+            0.017120718955993652,
+            -0.03117356449365616,
+            -0.0440058559179306,
+            -0.013846670277416706,
+            0.02079886943101883,
+            0.007970700971782207,
+            -0.02083185873925686,
+            -0.028353098779916763,
+            -0.005772882141172886,
+            -0.012131299823522568,
+            -0.002649340545758605,
+            0.03356518596410751,
+            0.017335141077637672,
+            0.01619705744087696,
+            0.029227279126644135,
+            -0.03336726129055023,
+            0.03142097219824791,
+            -0.015570288524031639,
+            0.023982202634215355,
+            0.0035894957836717367,
+            -0.011916878633201122,
+            -0.006515109911561012,
+            0.04403884336352348,
+            -0.012716835364699364,
+            0.018110357224941254,
+            -0.022266831248998642,
+            -0.0028967498801648617,
+            0.006172860506922007,
+            0.02137615904211998,
+            -0.006915088277310133,
+            -0.007273831870406866,
+            0.00403689406812191,
+            -0.024196624755859375,
+            0.022547228261828423,
+            0.04295024275779724,
+            0.00791709590703249,
+            0.01558678224682808,
+            -0.019314413890242577,
+            -0.012914762832224369,
+            0.018935054540634155,
+            0.008849004283547401,
+            -0.014085832983255386,
+            -0.012898268178105354,
+            0.01030871830880642,
+            -0.018044380471110344,
+            -0.019330909475684166,
+            -0.019677281379699707,
+            -0.017401115968823433,
+            0.009929358027875423,
+            0.011314849369227886,
+            -0.040212247520685196,
+            -0.011603493243455887,
+            -0.04288426786661148,
+            -0.011842655949294567,
+            0.008766533806920052,
+            -0.006832618732005358,
+            0.03579187020659447,
+            -0.0036740272771567106,
+            0.004214203916490078,
+            -0.015562041662633419,
+            -0.007418153807520866,
+            -0.002801909577101469,
+            -0.035264063626527786,
+            0.0005896586808376014,
+            0.015075469389557838,
+            0.0003703407128341496,
+            -0.04710672050714493,
+            0.017829960212111473,
+            -0.016617653891444206,
+            0.016865063458681107,
+            -0.02708306536078453,
+            -0.008185122162103653,
+            -0.024312080815434456,
+            0.011611740104854107,
+            0.008997449651360512,
+            -0.03605577349662781,
+            -0.014423958957195282,
+            -0.002556562190875411,
+            0.029293254017829895,
+            -0.017351634800434113,
+            0.0006633660523220897,
+            0.017302153632044792,
+            -0.00019715423695743084,
+            -0.010894253849983215,
+            -0.022497747093439102,
+            -0.0022266830783337355,
+            -0.007937713526189327,
+            0.020106123760342598,
+            -0.04156475141644478,
+            -0.005496608559042215,
+            -0.01082003116607666,
+            0.004001844674348831,
+            -0.030150938779115677,
+            -0.009723182767629623,
+            0.006432640366256237,
+            0.02229982055723667,
+            -0.04882209002971649,
+            0.009253106079995632,
+            0.04374195262789726,
+            -0.018489716574549675,
+            -0.01057262159883976,
+            0.025120284408330917,
+            -0.0007772773969918489,
+            0.009459280408918858,
+            -0.009063425473868847,
+            0.016683630645275116,
+            0.02579653635621071,
+            -0.02061743661761284,
+            -0.008020183071494102,
+            0.0003203434171155095,
+            0.002498833229765296,
+            -0.027973737567663193,
+            0.018473222851753235,
+            -0.013261135667562485,
+            0.023289456963539124,
+            0.0077768974006175995,
+            0.010943735018372536,
+            0.05429808050394058,
+            -0.015248656272888184,
+            0.014399217441678047,
+            0.015562041662633419,
+            -0.0074800061993300915,
+            -0.007974824868142605,
+            -0.013772447593510151,
+            0.022563721984624863,
+            0.03704541176557541,
+            0.013582766987383366,
+            0.005649177823215723,
+            -0.039717432111501694,
+            0.01293950341641903,
+            0.0074758827686309814,
+            0.0038637076504528522,
+            0.02096381038427353,
+            0.017763983458280563,
+            0.02196994051337242,
+            0.024163635447621346,
+            -0.02594498172402382,
+            -0.006098637823015451,
+            0.005513102747499943,
+            -0.02762736566364765,
+            -0.003422494512051344,
+            -0.019512342289090157,
+            0.023619335144758224,
+            0.015223915688693523,
+            -0.02660473994910717,
+            0.0008654168923385441,
+            -0.007954207248985767,
+            -0.0006757365190424025,
+            -0.003993597347289324,
+            -0.012667353264987469,
+            -0.012263251468539238,
+            -0.01576821506023407,
+            0.02759437821805477,
+            -0.014753838069736958,
+            -0.024378057569265366,
+            -0.015990883111953735,
+            0.014489934779703617,
+            -0.009088166058063507,
+            0.011224132962524891,
+            -0.030992131680250168,
+            -0.04964678734540939,
+            -0.0216730497777462,
+            0.028419075533747673,
+            0.03302088752388954,
+            -0.025417177006602287,
+            -1.3602676517621148e-05,
+            -0.007042916491627693,
+            0.029755085706710815,
+            -0.03622071444988251,
+            -0.008568606339395046,
+            -0.01332711149007082,
+            -0.04733763635158539,
+            0.0130714550614357,
+            -0.028567520901560783,
+            0.024922357872128487,
+            0.02708306536078453,
+            -0.018308283761143684,
+            -0.03148695081472397,
+            0.0210957620292902,
+            0.020155606791377068,
+            -0.004109055269509554,
+            0.010234495624899864,
+            -0.0036822741385549307,
+            -0.02658824622631073,
+            -0.028270630165934563,
+            -0.003356518689543009,
+            0.04163073003292084,
+            0.014902283437550068,
+            0.03536302596330643,
+            0.04212554544210434,
+            -0.009921110235154629,
+            -0.034241437911987305,
+            0.019099993631243706,
+            0.009945851750671864,
+            -0.007653192616999149,
+            0.00516672944650054,
+            0.003502902574837208,
+            -0.018423741683363914,
+            -0.009137648157775402,
+            -0.008840756490826607,
+            -0.028765447437763214,
+            -0.0441707968711853,
+            0.0012638488551601768,
+            0.0013360099401324987,
+            0.0167083702981472,
+            -0.012518907897174358,
+            -0.006374911405146122,
+            0.035132113844156265,
+            0.03331777825951576,
+            -0.009789158590137959,
+            -2.2904683646629564e-05,
+            -0.017994899302721024,
+            -0.0103004714474082,
+            -0.008131517097353935,
+            -0.002187510021030903,
+            0.023833757266402245,
+            -0.014300254173576832,
+            0.030051976442337036,
+            -0.0015452768420800567,
+            0.015512559562921524,
+            -0.06749323755502701,
+            -0.02066691778600216,
+            0.03155292570590973,
+            0.008725299499928951,
+            -0.009970592334866524,
+            0.002399869728833437,
+            0.005480114836245775,
+            0.0001164240893558599,
+            -0.0220029279589653,
+            0.011694210581481457,
+            0.020023655146360397,
+            0.019974172115325928,
+            -0.020534968003630638,
+            0.022497747093439102,
+            0.01632900908589363,
+            0.019314413890242577,
+            -0.004210080485790968,
+            -0.025268729776144028,
+            0.012444685213267803,
+            -0.007867613807320595,
+            -0.0019627800211310387,
+            0.002861700253561139,
+            0.007463512010872364,
+            -0.008923226967453957,
+            0.01017676666378975,
+            -0.02205241098999977,
+            -0.02726449817419052,
+            -0.011529270559549332,
+            0.017813464626669884,
+            0.010473658330738544,
+            0.007113015744835138,
+            0.008436654694378376,
+            0.0018452606163918972,
+            0.00416884571313858,
+            0.0013267321046441793,
+            -0.019099993631243706,
+            0.028237640857696533,
+            0.001028294675052166,
+            0.013929140754044056,
+            0.013145677745342255,
+            -0.023635828867554665,
+            -0.023058541119098663,
+            -0.008626335300505161,
+            0.01072106696665287,
+            0.0035647547338157892,
+            0.019017523154616356,
+            0.03879376873373985,
+            -0.014918777160346508,
+            -0.014992999844253063,
+            -0.0008478920790366828,
+            0.019759751856327057,
+            0.0056780423037707806,
+            -0.010943735018372536,
+            -0.0028369592037051916,
+            -0.00416884571313858,
+            -0.005925451405346394,
+            0.025747055187821388,
+            -0.008890238590538502,
+            -0.03373012691736221,
+            0.010943735018372536,
+            0.05756388232111931,
+            -0.007059410214424133,
+            -0.0011659160954877734,
+            -0.022514240816235542,
+            -0.0019122672965750098,
+            0.006906841415911913,
+            0.0080655412748456,
+            0.006993434391915798,
+            0.015281643718481064,
+            0.01581769809126854,
+            0.012147793546319008,
+            -0.010102543979883194,
+            -0.012007595039904118,
+            -0.010374694131314754,
+            -0.03787010908126831,
+            0.030018987134099007,
+            -0.018258802592754364,
+            -0.0035358904860913754,
+            0.011331343092024326,
+            0.02188747003674507,
+            0.00010637308878358454,
+            0.016518689692020416,
+            0.013409581035375595,
+            0.004692528396844864,
+            0.0005932667409069836,
+            0.011652975343167782,
+            0.0015494003891944885,
+            -0.013896152377128601,
+            -0.022761650383472443,
+            -0.009533503092825413,
+            0.018918560817837715,
+            0.02705007791519165,
+            -0.004643046762794256,
+            -0.009978839196264744,
+            -0.005673918407410383,
+            0.014687862247228622,
+            -0.004878085572272539,
+            0.011916878633201122,
+            -0.0045317127369344234,
+            0.00045564534957520664,
+            0.030728228390216827,
+            0.013953881338238716,
+            -0.007298572454601526,
+            0.016972273588180542,
+            0.004556453321129084,
+            -0.003414247650653124,
+            0.019479354843497276,
+            -0.04915196821093559,
+            0.01590016670525074,
+            0.0028348974883556366,
+            0.0018029948696494102,
+            0.0036699038464576006,
+            -0.0020679289009422064,
+            -0.013277629390358925,
+            0.012626118026673794,
+            0.007636698428541422,
+            0.003036948386579752,
+            0.010152026079595089,
+            0.010110790841281414,
+            0.00022125086979940534,
+            0.012279745191335678,
+            -0.019776245579123497,
+            0.011512776836752892,
+            -0.015908414497971535,
+            -0.009467527270317078,
+            -0.008477889932692051,
+            -0.0013019911712035537,
+            -0.0037647439166903496,
+            -0.009962345473468304,
+            0.000947886670473963,
+            -0.022398782894015312,
+            -0.006758395582437515,
+            -0.017434105277061462,
+            -0.008799522183835506,
+            -0.0025359448045492172,
+            -0.024081166833639145,
+            0.0076738097704946995,
+            0.004960555117577314,
+            0.011232379823923111,
+            -0.03658358007669449,
+            0.0028328357730060816,
+            0.03272399678826332,
+            0.004506971687078476,
+            -0.018077367916703224,
+            -0.024378057569265366,
+            -0.011702457442879677,
+            -0.02657175250351429,
+            0.013285876251757145,
+            0.011075686663389206,
+            -0.020040148869156837,
+            0.015463077463209629,
+            -0.001200965722091496,
+            -0.023784274235367775,
+            -0.02098030410706997,
+            -0.02073289453983307,
+            -0.0007360425079241395,
+            0.0013566273264586926,
+            0.001249416614882648,
+            0.0007391351391561329,
+            0.004869838710874319,
+            -0.022877108305692673,
+            0.019611306488513947,
+            0.03354869410395622,
+            -0.008486136794090271,
+            -0.0030472571961581707,
+            0.008816015906631947,
+            0.0220029279589653,
+            -0.01660940796136856,
+            0.00010611536708893254,
+            -0.015042481943964958,
+            0.02150811068713665,
+            -0.0027565513737499714,
+            0.012626118026673794,
+            -0.0026946989819407463,
+            0.029639627784490585,
+            -0.004688404966145754,
+            0.0272974856197834,
+            -0.008947967551648617,
+            0.0220029279589653,
+            0.0013782756868749857,
+            -0.034769244492053986,
+            -0.009401551447808743,
+            0.03277347609400749,
+            -0.002628723159432411,
+            -0.013178665190935135,
+            0.022926589474081993,
+            -0.0036967063788324594,
+            -0.024048177525401115,
+            0.00832944456487894,
+            0.0015875425888225436,
+            -0.01375595387071371,
+            0.024955345317721367,
+            0.006370787974447012,
+            0.009549996815621853,
+            0.004742010496556759,
+            0.015529053285717964,
+            0.024031683802604675,
+            -0.000978297321125865,
+            0.03213021531701088,
+            0.009508761577308178,
+            0.0070016817189753056,
+            -0.004346155561506748,
+            0.012403449974954128,
+            -0.005100754089653492,
+            0.024806899949908257,
+            -0.03272399678826332,
+            -0.014118820428848267,
+            0.033746618777513504,
+            0.015232162550091743,
+            0.018159838393330574,
+            -0.022563721984624863,
+            -0.0047873686999082565,
+            0.016386738047003746,
+            0.012733329087495804,
+            -0.01585068553686142,
+            -0.02081536501646042,
+            -0.001431880984455347,
+            0.008510878309607506,
+            0.004280179738998413,
+            -0.017599044367671013,
+            0.00668417289853096,
+            0.03816699981689453,
+            -0.017978405579924583,
+            -0.040047310292720795,
+            0.02596147730946541,
+            -0.015471324324607849,
+            -0.003115294734016061,
+            0.016007378697395325,
+            -0.005389397963881493,
+            0.014473441056907177,
+            -0.008857251144945621,
+            -0.0033029133919626474,
+            0.03628668934106827,
+            0.008914980106055737,
+            0.010481905192136765,
+            -0.030018987134099007,
+            0.02101329155266285,
+            0.002962725702673197,
+            0.007653192616999149,
+            -0.023668818175792694,
+            0.002189571736380458,
+            0.012411696836352348,
+            -0.030051976442337036,
+            -0.003247246379032731,
+            0.03816699981689453,
+            0.008148010820150375,
+            -0.017615538090467453,
+            0.0008721175836399198,
+            -0.02579653635621071,
+            0.02038652077317238,
+            -0.0050801364704966545,
+            0.014580651186406612,
+            0.019462861120700836,
+            -0.012114806100726128,
+            0.018060874193906784,
+            -0.0056161899119615555,
+            -0.011562258936464787,
+            -0.004931691102683544,
+            -0.010943735018372536,
+            -0.02665422298014164,
+            -0.013986868783831596,
+            0.0020029840525239706,
+            -0.01059736218303442,
+            -0.01629602164030075,
+            0.007158373948186636,
+            -0.03641863912343979,
+            -0.014910530298948288,
+            -0.006391405593603849,
+            0.007632574997842312,
+            -0.023569853976368904,
+            0.017450598999857903,
+            -0.024641960859298706,
+            0.0029874665196985006,
+            0.02182149514555931,
+            -0.003570940112695098,
+            0.0024204871151596308,
+            -0.019001029431819916,
+            -0.029276760295033455,
+            0.008366555906832218,
+            0.006506863050162792,
+            -0.00681200111284852,
+            -0.015908414497971535,
+            0.008857251144945621,
+            0.0017669143853709102,
+            -0.0037771144416183233,
+            -0.006255330517888069,
+            0.01293950341641903,
+            -0.023685311898589134,
+            0.017483586445450783,
+            0.019545329734683037,
+            -0.024246105924248695,
+            0.010762302204966545,
+            -0.0069975582882761955,
+            0.017483586445450783,
+            0.01655992493033409,
+            0.0051337420009076595,
+            0.0164609607309103,
+            0.02695111371576786,
+            0.008259344846010208,
+            0.022728662937879562,
+            0.000705116312019527,
+            -0.004383266903460026,
+            -0.019512342289090157,
+            0.027792304754257202,
+            -0.009525256231427193,
+            0.0001148777810158208,
+            0.012568389065563679,
+            -0.009731430560350418,
+            -0.013945634476840496,
+            0.0019308229675516486,
+            4.95140629936941e-05,
+            -0.016172317788004875,
+            -0.007875860668718815,
+            -0.02150811068713665,
+            -0.005768758710473776,
+            -0.018242308869957924,
+            -0.01110042817890644,
+            0.02220085635781288,
+            -0.020023655146360397,
+            -0.0209308210760355,
+            0.00045384131954051554,
+            -0.00031982798827812076,
+            0.019974172115325928,
+            0.020155606791377068,
+            0.009302587248384953,
+            0.0021297812927514315,
+            -0.0025462533812969923,
+            -0.00341012398712337,
+            -0.005686289165169001,
+            0.008366555906832218,
+            0.006510986480861902,
+            0.02220085635781288,
+            -0.0216730497777462,
+            -0.01576821506023407,
+            0.007080027833580971,
+            -0.006840865593403578,
+            0.009764418005943298,
+            -0.019974172115325928,
+            -0.011776680126786232,
+            -0.004704899154603481,
+            -0.017005261033773422,
+            0.0552547313272953,
+            -0.02716553397476673,
+            -0.035297051072120667,
+            -0.01058086846023798,
+            0.002946231747046113,
+            0.006741901859641075,
+            -0.018522705882787704,
+            0.010366447269916534,
+            -0.01323639415204525,
+            -0.014869295060634613,
+            -0.023586347699165344,
+            0.00514611229300499,
+            0.0026081057731062174,
+            0.005933698266744614,
+            -0.009772664867341518,
+            -0.022910095751285553,
+            0.0036121748853474855,
+            -0.017236176878213882,
+            -0.017434105277061462,
+            -0.0020060765091329813,
+            0.01031696517020464,
+            0.015858933329582214,
+            0.018027886748313904,
+            0.007277955301105976,
+            0.002725625177845359,
+            0.0009215994505211711,
+            -0.002183386590331793,
+            0.009145895019173622,
+            -0.00333383958786726,
+            0.003041071817278862,
+            0.008585100993514061,
+            -0.027808798477053642,
+            0.009582984261214733,
+            -0.013574520125985146,
+            0.02705007791519165,
+            0.029161302372813225,
+            0.01603211835026741,
+            0.013137430883944035,
+            0.009244858287274837,
+            -0.040179260075092316,
+            0.011364331468939781,
+            -0.000893765885848552,
+            0.00807378813624382,
+            0.007220226339995861,
+            0.0038637076504528522,
+            -0.007298572454601526,
+            0.0013040528865531087,
+            -0.019116487354040146,
+            -0.004408007953315973,
+            0.000873663870152086,
+            -0.024559490382671356,
+            0.02187097631394863,
+            0.013533285818994045,
+            0.015545547008514404,
+            0.020403016358613968,
+            -0.011801420710980892,
+            0.010886006988584995,
+            0.01297249086201191,
+            0.003876078175380826,
+            -0.020419510081410408,
+            0.005830611102283001,
+            -0.023437902331352234,
+            0.0065893325954675674,
+            -0.010086050257086754,
+            -0.0340765006840229,
+            -0.020122617483139038,
+            0.017846453934907913,
+            0.008432531729340553,
+            -0.009261352941393852,
+            -0.028946882113814354,
+            0.025235742330551147,
+            0.027858281508088112,
+            -0.004498724825680256,
+            0.008956214413046837,
+            0.027792304754257202,
+            0.010358200408518314,
+            0.0446656160056591,
+            -0.0080655412748456,
+            0.009723182767629623,
+            -0.004630676470696926,
+            0.007088274694979191,
+            -0.0051378654316067696,
+            -0.006869730073958635,
+            0.011050946079194546,
+            0.014762084931135178,
+            0.006069773342460394,
+            0.018836090341210365,
+            0.029359230771660805,
+            -0.019281426444649696,
+            -0.007459388580173254,
+            0.013137430883944035,
+            -0.010770549066364765,
+            0.02088133990764618,
+            -0.0001592696935404092,
+            -0.006358417682349682,
+            -0.028303617611527443,
+            0.006976940669119358,
+            0.008774780668318272,
+            -0.007401659619063139,
+            -0.003787423251196742,
+            -0.019842220470309258,
+            -0.005504855420440435,
+            -0.0012762193800881505,
+            -0.020320545881986618,
+            -0.0009262383682653308,
+            -0.004993543028831482,
+            -0.0026699581649154425,
+            0.0047914921306073666,
+            0.018159838393330574,
+            0.008659323677420616,
+            0.009912863373756409,
+            -0.0017308337846770883,
+            0.014861048199236393,
+            0.03381259739398956,
+            -0.005636807065457106,
+            -0.000506158045027405,
+            0.019231945276260376,
+            0.0011978730326518416,
+            0.013937387615442276,
+            -0.004259562585502863,
+            -0.005785252898931503,
+            -0.014894036576151848,
+            0.01571873389184475,
+            -0.004614182282239199,
+            -0.004201833624392748,
+            -0.012774563394486904,
+            -0.020205087959766388,
+            0.012494166381657124,
+            -0.008857251144945621,
+            -0.008923226967453957,
+            -0.023685311898589134,
+            -0.03130551427602768,
+            -0.017450598999857903,
+            -0.0038224728778004646,
+            -0.0009566490771248937,
+            0.014654873870313168,
+            0.02124420739710331,
+            -0.007232596632093191,
+            0.022250337526202202,
+            -0.005962562747299671,
+            -0.03320232033729553,
+            0.009096412919461727,
+            0.014992999844253063,
+            -0.010275730863213539,
+            0.008267591707408428,
+            -0.02221735008060932,
+            -0.009277846664190292,
+            -0.03206423670053482,
+            -0.019776245579123497,
+            -0.0034781615249812603,
+            0.02149161696434021,
+            -0.01636199839413166,
+            0.02688513696193695,
+            -0.013277629390358925,
+            -0.00779339112341404,
+            0.01333535835146904,
+            0.003752373391762376,
+            -0.01606510765850544,
+            -0.004440995864570141,
+            0.011479788459837437,
+            0.01674135960638523,
+            -0.013021972961723804,
+            -0.004560577217489481,
+            -0.00333796301856637,
+            -0.009706689044833183,
+            0.008914980106055737,
+            -0.006857359316200018,
+            0.007640821859240532,
+            -0.019644293934106827,
+            0.010918994434177876,
+            0.005055395420640707,
+            -0.0017143398290500045,
+            -0.0019339155405759811,
+            -0.013673484325408936,
+            0.005711030215024948,
+            0.013656990602612495,
+            0.020551461726427078,
+            -0.03866181895136833,
+            0.004572947509586811,
+            -0.002818403532728553,
+            -0.005224458407610655,
+            -0.018605174496769905,
+            0.012452932074666023,
+            0.016584666445851326,
+            0.015463077463209629,
+            0.013722965493798256,
+            -0.01041592936962843,
+            0.022514240816235542,
+            -0.012395203113555908,
+            -0.013038466684520245,
+            0.00780988484621048,
+            -0.01306320820003748,
+            0.0005324453231878579,
+            0.029045844450592995,
+            -0.007574846502393484,
+            -0.004626553039997816,
+            0.005574955139309168,
+            0.039552491158246994,
+            0.005385274533182383,
+            0.004985296167433262,
+            0.029375724494457245,
+            -0.0013391025131568313,
+            0.006844989024102688,
+            0.03262503072619438,
+            0.017928922548890114,
+            0.007896478287875652,
+            -0.007096521556377411,
+            0.013962128199636936,
+            0.011496283113956451,
+            0.013904399238526821,
+            -0.012510661035776138,
+            -0.04077304154634476,
+            0.019231945276260376,
+            0.004651293624192476,
+            -0.011974607594311237,
+            0.010152026079595089,
+            -0.0024328576400876045,
+            0.007257337681949139,
+            0.017054744064807892,
+            0.00026699580485001206,
+            -0.022811131551861763,
+            0.013426074758172035,
+            0.004170907661318779,
+            0.011298355646431446,
+            -0.005055395420640707,
+            0.000341476290486753,
+            0.02233280800282955,
+            0.007108892314136028,
+            0.0014689923264086246,
+            0.012469425797462463,
+            -0.0128240454941988,
+            0.02145862765610218,
+            0.019825726747512817,
+            0.02716553397476673,
+            -0.0031503443606197834,
+            -0.0031482826452702284,
+            0.000705116312019527,
+            -0.0070016817189753056,
+            -0.012890021316707134,
+            0.009492267854511738,
+            0.0003778145182877779,
+            0.009401551447808743,
+            -0.005851228721439838,
+            0.011685962788760662,
+            0.007570722606033087,
+            0.004849221091717482,
+            0.019974172115325928,
+            0.005253322888165712,
+            -0.0068037542514503,
+            0.028946882113814354,
+            0.0020277248695492744,
+            0.0009226303081959486,
+            -0.011050946079194546,
+            0.0006973848212510347,
+            0.0014669306110590696,
+            -0.001361781731247902,
+            0.007611957844346762,
+            -0.008110899478197098,
+            -0.0130714550614357,
+            -0.01086126547306776,
+            0.028369592502713203,
+            0.00025436762371100485,
+            0.023503877222537994,
+            -0.0053316690027713776,
+            -0.002236991887912154,
+            0.002789539285004139,
+            -0.01618056371808052,
+            -0.005669794976711273,
+            0.015710486099123955,
+            0.0033214690629392862,
+            0.006432640366256237,
+            0.013285876251757145,
+            -0.0053770276717841625,
+            0.016353750601410866,
+            -0.0056203133426606655,
+            0.0036987680941820145,
+            0.013030219823122025,
+            0.005987303797155619,
+            0.008733546361327171,
+            0.0006375942612066865,
+            -0.017796970903873444,
+            -0.019611306488513947,
+            0.01085301861166954,
+            -0.006865606643259525,
+            0.000996337621472776,
+            0.003098800778388977,
+            -0.001747327740304172,
+            0.011248873546719551,
+            -0.015009493567049503,
+            -0.003950301092118025,
+            -0.008007812313735485,
+            0.034967172890901566,
+            -0.006127502303570509,
+            -0.002942108316347003,
+            -0.0063996524550020695,
+            -0.01334360521286726,
+            0.003667841898277402,
+            -0.008585100993514061,
+            -0.00026235688710585237,
+            0.015710486099123955,
+            0.004915196914225817,
+            -0.01703825034201145,
+            -0.013269382528960705,
+            -0.014696109108626842,
+            0.006469751708209515,
+            0.02078237570822239,
+            0.02706657163798809,
+            -0.008783028461039066,
+            0.007108892314136028,
+            -0.008135640993714333,
+            0.017054744064807892,
+            -0.007706797681748867,
+            0.013541532680392265,
+            -0.022695673629641533,
+            0.008898485451936722,
+            -0.01362400222569704,
+            0.016353750601410866,
+            -0.007438771426677704,
+            0.0038905104156583548,
+            -0.01557853538542986,
+            -0.009690195322036743,
+            0.0010999402729794383,
+            -0.011273614130914211,
+            0.009879875928163528,
+            -0.01030871830880642,
+            -0.008535618893802166,
+            0.0043007973581552505,
+            -0.02159057930111885,
+            -0.0021297812927514315,
+            -0.007682057097554207,
+            -0.005636807065457106,
+            -0.009698442183434963,
+            -0.02116173692047596,
+            -0.02140914648771286,
+            0.005694536026567221,
+            -0.009492267854511738,
+            -0.012147793546319008,
+            -0.028979869559407234,
+            -0.03326829522848129,
+            -0.001901958603411913,
+            -0.02180500142276287,
+            -0.014259018935263157,
+            -0.005814117379486561,
+            0.019215451553463936,
+            -0.011166404001414776,
+            -0.012914762832224369,
+            -0.00795008335262537,
+            -0.011125168763101101,
+            0.019462861120700836,
+            0.014234278351068497,
+            0.009954098612070084,
+            -0.001399923930875957,
+            0.016667135059833527,
+            -0.008700557984411716,
+            0.009756171144545078,
+            -0.040443163365125656,
+            -0.017450598999857903,
+            -0.0028369592037051916,
+            0.0026720198802649975,
+            0.008890238590538502,
+            -0.000427811814006418,
+            0.019924690946936607,
+            -0.010102543979883194,
+            -0.04374195262789726,
+            -0.02116173692047596,
+            -0.002880255924537778,
+            -0.011438554152846336,
+            0.022794637829065323,
+            0.003810102352872491,
+            0.039288587868213654,
+            0.022893602028489113,
+            0.015858933329582214,
+            -0.006708913948386908,
+            0.007401659619063139,
+            -0.019001029431819916,
+            0.0015380608383566141,
+            0.016881557181477547,
+            0.0032369375694543123,
+            -0.011174650862812996,
+            0.029359230771660805,
+            -0.009442785754799843,
+            -0.015075469389557838,
+            0.027858281508088112,
+            0.0011803482193499804,
+            0.013285876251757145,
+            0.0005705875810235739,
+            -0.0050760130397975445,
+            -0.010399434715509415,
+            0.0038451519794762135,
+            0.002278226660564542,
+            -0.012700340710580349,
+            0.01596614345908165,
+            0.00808203499764204,
+            4.94818486913573e-05,
+            0.0007891323766671121,
+            0.025153273716568947,
+            -0.0071295094676315784,
+            -0.017681512981653214,
+            0.014894036576151848,
+            0.0032018879428505898,
+            -0.01084477175027132,
+            -0.011215885169804096,
+            -0.009558243677020073,
+            -0.01349205058068037,
+            -0.010927241295576096,
+            0.0027689216658473015,
+            -0.002622538013383746,
+            0.023784274235367775,
+            -0.004022461827844381,
+            0.006370787974447012,
+            -0.00390906585380435,
+            -0.002867885399609804,
+            0.028699472546577454,
+            0.019149474799633026,
+            -0.02643980085849762,
+            -0.017335141077637672,
+            0.0007241874700412154,
+            -0.005508979316800833,
+            -0.011372578330338001,
+            0.02091432735323906,
+            -0.006024415139108896,
+            -0.024724429473280907,
+            0.025070803239941597,
+            -0.0004649232141673565,
+            0.004102869890630245,
+            0.005739894229918718,
+            0.0264727883040905,
+            -0.0025668710004538298,
+            -0.0004610574396792799,
+            -0.003036948386579752,
+            0.011323096230626106,
+            2.6738238375401124e-05,
+            0.007018175441771746,
+            -0.023998696357011795,
+            -0.003014269284904003,
+            0.010803536511957645,
+            0.002383375773206353,
+            0.02747892029583454,
+            0.005459497217088938,
+            0.00514611229300499,
+            0.01324464101344347,
+            -0.016955779865384102,
+            0.008890238590538502,
+            -0.017895935103297234,
+            0.017170201987028122,
+            -0.008750040084123611,
+            -0.025582116097211838,
+            -0.010275730863213539,
+            0.0034740380942821503,
+            0.008675817400217056,
+            0.006898594554513693,
+            0.0043007973581552505,
+            0.019825726747512817,
+            -3.6467092286329716e-05,
+            0.006354294251650572,
+            0.002397808013483882,
+            0.0009432477527298033,
+            -0.0003213743038941175,
+            0.0037709292955696583,
+            0.02660473994910717,
+            -0.008659323677420616,
+            -0.006296565290540457,
+            0.0009757201769389212,
+            -0.0033276542089879513,
+            -0.01324464101344347,
+            0.003115294734016061,
+            -0.003919374663382769,
+            -0.0034596058540046215,
+            0.045787204056978226,
+            0.017104225233197212,
+            -0.007962454110383987,
+            -0.015265149995684624,
+            -0.01598263718187809,
+            0.027759317308664322,
+            -0.001940100803039968,
+            0.011578752659261227,
+            0.019759751856327057,
+            0.0030637511517852545,
+            -0.002773045329377055,
+            -0.0007468666299246252,
+            -0.0016091909492388368,
+            0.001787531771697104,
+            0.024889370426535606,
+            0.008964461274445057,
+            -0.006964570377022028,
+            -0.008469643071293831,
+            0.019215451553463936,
+            -0.01362400222569704,
+            -0.009426292032003403,
+            0.02119472436606884,
+            -0.003034886671230197,
+            0.02079886943101883,
+            -0.007521240971982479,
+            0.01606510765850544,
+            -0.005443003494292498,
+            -0.006964570377022028,
+            -0.02162356860935688,
+            0.00404101749882102,
+            -0.002410178305581212,
+            0.003016331000253558,
+            0.028254134580492973,
+            -0.014465193264186382,
+            -0.002249362412840128,
+            0.0036080514546483755,
+            0.011182897724211216,
+            -0.015108457766473293,
+            0.01031696517020464,
+            0.001118495943956077,
+            0.0007360425079241395,
+            0.025285225361585617,
+            -0.004133795853704214,
+            -0.004692528396844864,
+            0.03130551427602768,
+            -0.007459388580173254,
+            -0.011422060430049896,
+            -0.001281373668462038,
+            0.008069665171205997,
+            0.005661548115313053,
+            -0.02126070111989975,
+            -0.002247300697490573,
+            -0.01019326038658619,
+            0.0021194724831730127,
+            -0.045391347259283066,
+            0.02159057930111885,
+            -0.02165655605494976,
+            0.025367693975567818,
+            -0.005723400507122278,
+            -0.002488524653017521,
+            -0.008593347854912281,
+            0.006201724987477064,
+            0.005574955139309168,
+            -0.012114806100726128,
+            0.007665562909096479,
+            -0.019198957830667496,
+            0.013302369974553585,
+            0.006473875138908625,
+            -0.019231945276260376,
+            0.015693992376327515,
+            -0.007529487833380699,
+            0.006568715441972017,
+            -0.015438336879014969,
+            -0.003880201606079936,
+            0.019347403198480606,
+            -0.0026967606972903013,
+            -0.005772882141172886,
+            -0.03783712163567543,
+            -0.00418121600523591,
+            -0.011389072053134441,
+            0.00163496274035424,
+            0.002814280102029443,
+            -0.015743475407361984,
+            -0.03572589531540871,
+            0.005113124381750822,
+            0.01305496133863926,
+            -0.0009911833330988884,
+            0.022481253370642662,
+            -0.005781129468232393,
+            -0.005925451405346394,
+            0.008024306036531925,
+            0.0025297594256699085,
+            0.018968041986227036,
+            0.014267266727983952,
+            -0.003490532049909234,
+            -0.0045399595983326435,
+            0.0026740815956145525,
+            -0.007698550820350647,
+            -0.017813464626669884,
+            -0.018836090341210365,
+            -0.013021972961723804,
+            0.008832509629428387,
+            0.018489716574549675,
+            -0.018159838393330574,
+            -0.00040281316614709795,
+            -0.004263686016201973,
+            0.015529053285717964,
+            -0.011356084607541561,
+            -0.022398782894015312,
+            -0.009879875928163528,
+            -0.0034451736137270927,
+            0.012329227291047573,
+            -0.024411045014858246,
+            0.014663120731711388,
+            -0.009615972638130188,
+            -0.004276056308299303,
+            0.015809450298547745,
+            -0.0053316690027713776,
+            0.0014968259492889047,
+            0.004915196914225817,
+            0.028897399082779884,
+            0.0019370082300156355,
+            -0.022596711292862892,
+            0.018638163805007935,
+            0.009426292032003403,
+            -0.00026879983488470316,
+            -0.005001790355890989,
+            -0.004214203916490078,
+            -0.018984535709023476,
+            -0.00652748066931963,
+            0.008626335300505161,
+            0.01351679116487503,
+            0.013162171468138695,
+            0.016106341034173965,
+            -0.00528218736872077,
+            0.0013452877756208181,
+            0.004313167650252581,
+            0.0011349898995831609,
+            0.009912863373756409,
+            0.007455265149474144,
+            -0.002552438760176301,
+            -0.007467635441571474,
+            0.022629698738455772,
+            -0.023223480209708214,
+            -0.0032328141387552023,
+            0.023734793066978455,
+            -0.0030451954808086157,
+            0.013788941316306591,
+            -0.00817275233566761,
+            0.002777168760076165,
+            0.015001246705651283,
+            0.007405783515423536,
+            0.009607725776731968,
+            -0.024196624755859375,
+            0.020023655146360397,
+            -0.019858716055750847,
+            0.005723400507122278,
+            0.005038901697844267,
+            0.006774889770895243,
+            0.008494383655488491,
+            -0.010020074434578419,
+            0.0034554824233055115,
+            0.01334360521286726,
+            0.016939286142587662,
+            0.017120718955993652,
+            -0.015562041662633419,
+            -0.001777223078534007,
+            -0.022514240816235542,
+            -0.017236176878213882,
+            0.02137615904211998,
+            -0.013277629390358925,
+            -0.001867939718067646,
+            0.015281643718481064,
+            -0.009640713222324848,
+            -0.009888122789561749,
+            0.023701805621385574,
+            -0.009797406382858753,
+            0.016840321943163872,
+            0.0068862237967550755,
+            0.017994899302721024,
+            0.008011936210095882,
+            -0.0012184905353933573,
+            0.0026534642092883587,
+            -0.010267484001815319,
+            -0.0012442623265087605,
+            -0.004208019003272057,
+            0.011883890256285667,
+            0.0061522433534264565,
+            -0.01041592936962843,
+            -0.001817427109926939,
+            -0.009657206945121288,
+            -0.009780911728739738,
+            -0.017087731510400772,
+            0.006791383493691683,
+            0.005991427227854729,
+            -0.007385165896266699,
+            -0.016700124368071556,
+            0.00011146302131237462,
+            0.01386316493153572,
+            -0.0029359229374676943,
+            -0.00011751939018722624,
+            0.011644728481769562,
+            -0.0076696863397955894,
+            -0.011586999520659447,
+            -0.007937713526189327,
+            -0.040113285183906555,
+            -0.012057077139616013,
+            0.012799304910004139,
+            -0.0007819163147360086,
+            0.014588898047804832,
+            0.034307416528463364,
+            -0.005591448862105608,
+            0.0017401116201654077,
+            -0.007661439478397369,
+            0.014143561944365501,
+            0.022728662937879562,
+            -0.004816233180463314,
+            -0.004465736914426088,
+            -0.001779284793883562,
+            -0.012353967875242233,
+            -0.000558217114303261,
+            -0.016073353588581085,
+            0.0009757201769389212,
+            -0.003036948386579752,
+            -0.007983071729540825,
+            -0.00339775369502604,
+            -0.010382940992712975,
+            -0.014960012398660183,
+            -0.014300254173576832,
+            0.008568606339395046,
+            0.002474092412739992,
+            -0.007748032920062542,
+            -0.0032967282459139824,
+            -0.012642612680792809,
+            0.013937387615442276,
+            0.006197601556777954,
+            0.006869730073958635,
+            0.0037132003344595432,
+            0.009030437096953392,
+            0.006263577379286289,
+            -0.005455373786389828,
+            0.003102924209088087,
+            -0.0023751286789774895,
+            -0.030662251636385918,
+            0.011661222204566002,
+            -0.016551679000258446,
+            0.020436003804206848,
+            0.009937604889273643,
+            -0.0053729042410850525,
+            0.0053729042410850525,
+            0.0024575984571129084,
+            0.02121121808886528,
+            -0.009393304586410522,
+            0.014489934779703617,
+            0.0035894957836717367,
+            0.00821398664265871,
+            -0.00529868109151721,
+            0.004271932877600193,
+            -0.004688404966145754,
+            -0.014992999844253063,
+            0.01641147956252098,
+            0.00795833021402359,
+            0.022448265925049782,
+            0.00416884571313858,
+            0.015438336879014969,
+            0.0035276433918625116,
+            -0.001785470056347549,
+            0.00021802939590997994,
+            0.016551679000258446,
+            0.006078020203858614,
+            -0.015990883111953735,
+            -0.004395637661218643,
+            -0.006201724987477064,
+            0.0128240454941988,
+            0.02630784921348095,
+            -0.01629602164030075,
+            -0.01350029744207859,
+            0.023503877222537994,
+            -0.006981064099818468,
+            0.02629135549068451,
+            -0.004589441232383251,
+            -0.017665019258856773,
+            -0.008832509629428387,
+            -0.0018473223317414522,
+            -0.001506103784777224,
+            -0.0208483524620533,
+            0.019446365535259247,
+            0.01334360521286726,
+            -0.002018447034060955,
+            -0.0035874340683221817,
+            0.010622103698551655,
+            -0.0018493840470910072,
+            0.020073136314749718,
+            -0.024064673110842705,
+            -0.007100645452737808,
+            0.0059130811132490635,
+            0.011125168763101101,
+            0.0071913618594408035,
+            -0.015487818978726864,
+            0.015990883111953735,
+            0.018902065232396126,
+            -0.0021318430081009865,
+            0.020172100514173508,
+            -0.015339372679591179,
+            -0.02091432735323906,
+            0.0014576527755707502,
+            -0.007306819781661034,
+            -0.0026720198802649975,
+            -0.0015751721803098917,
+            -0.0009288155124522746,
+            0.009310834109783173,
+            0.011207638308405876,
+            0.0005079620750620961,
+            -0.012337474152445793,
+            0.00038992727058939636,
+            0.00340600055642426,
+            -0.014382723718881607,
+            -0.019215451553463936,
+            0.009838640689849854,
+            -0.015430090017616749,
+            0.011455047875642776,
+            0.018885571509599686,
+            0.0038245345931500196,
+            -0.008807769045233727,
+            0.015174433588981628,
+            0.017516573891043663,
+            0.0065893325954675674,
+            -0.013384840451180935,
+            0.004230698104947805,
+            -0.0010097390040755272,
+            -0.004737887065857649,
+            0.008824262768030167,
+            -0.004812109749764204,
+            0.020023655146360397,
+            -0.0133600989356637,
+            0.02691812627017498,
+            0.004577070940285921,
+            0.030101457610726357,
+            0.013162171468138695,
+            0.005636807065457106,
+            0.001126742921769619,
+            0.0077232918702065945,
+            0.00780163798481226,
+            0.011982854455709457,
+            -0.0164609607309103,
+            0.013896152377128601,
+            -0.0006267700809985399,
+            0.009813900105655193,
+            -0.019908197224140167,
+            -0.003797731827944517,
+            0.013384840451180935,
+            -0.0033812597393989563,
+            -0.02615940384566784,
+            0.014193043112754822,
+            0.02126070111989975,
+            0.01655992493033409,
+            -0.030530299991369247,
+            0.0164609607309103,
+            -0.013401334173977375,
+            0.015347619540989399,
+            0.0030678745824843645,
+            -0.001799902180209756,
+            -0.02126070111989975,
+            -0.0210957620292902,
+            0.015941401943564415,
+            -0.025301719084382057,
+            -0.009244858287274837,
+            0.008807769045233727,
+            0.0038492754101753235,
+            0.014061091467738152,
+            -0.02617589756846428,
+            -0.008028429932892323,
+            -0.008069665171205997,
+            -0.008296456187963486,
+            0.0036740272771567106,
+            0.002236991887912154,
+            0.006069773342460394,
+            -0.006292441859841347,
+            -0.02149161696434021,
+            -0.018110357224941254,
+            0.0011813790770247579,
+            0.01375595387071371,
+            0.011595246382057667,
+            -0.029441699385643005,
+            0.005748141556978226,
+            0.010886006988584995,
+            -0.0033049751073122025,
+            0.0007545981789007783,
+            -0.024328574538230896,
+            -0.016254786401987076,
+            -0.013549779541790485,
+            -0.006762519478797913,
+            -0.009310834109783173,
+            0.004676034674048424,
+            -0.008424284867942333,
+            -0.004581194370985031,
+            -0.01639498583972454,
+            -0.009665454737842083,
+            0.01297249086201191,
+            0.018687644973397255,
+            -0.005657424684613943,
+            -0.029821060597896576,
+            0.02681916207075119,
+            0.018291790038347244,
+            -0.008824262768030167,
+            -0.01641147956252098,
+            -0.0003654440806712955,
+            -0.016955779865384102,
+            0.008947967551648617,
+            0.004803862888365984,
+            -0.014580651186406612,
+            -0.009112906642258167,
+            -0.010877760127186775,
+            0.013310616835951805,
+            0.020122617483139038,
+            0.004820356611162424,
+            0.0074882530607283115,
+            -0.0035317668225616217,
+            0.023042047396302223,
+            -0.006915088277310133,
+            -2.363756948398077e-06,
+            0.0029833430889993906,
+            -0.00680787768214941,
+            0.007756279781460762,
+            0.023718299344182014,
+            -0.0335816815495491,
+            -0.006853235885500908,
+            -0.002346264198422432,
+            -0.002056589350104332,
+            -0.011949866078794003,
+            -0.011644728481769562,
+            -0.0077356621623039246,
+            0.010778795927762985,
+            0.006750148721039295,
+            -0.0018473223317414522,
+            0.002083391882479191,
+            -0.0133188646286726,
+            0.009129401296377182,
+            -0.024889370426535606,
+            0.002865823684260249,
+            -0.018473222851753235,
+            0.010028321295976639,
+            -0.009657206945121288,
+            -0.00012628179683815688,
+            0.013533285818994045,
+            -0.000562340603210032,
+            0.0015019802376627922,
+            -0.019182464107871056,
+            -0.009805653244256973,
+            0.00791709590703249,
+            0.008436654694378376,
+            0.009294340386986732,
+            -0.02586251311004162,
+            -0.0012298300862312317,
+            0.004601811990141869,
+            0.004170907661318779,
+            -0.0018267049454152584,
+            -0.02726449817419052,
+            -0.0074758827686309814,
+            -0.014894036576151848,
+            -0.02172253094613552,
+            0.0017421734519302845,
+            -0.015149693004786968,
+            -0.013986868783831596,
+            0.024130648002028465,
+            0.02216786891222,
+            0.016551679000258446,
+            -0.004886332433670759,
+            0.01571873389184475,
+            0.006601703353226185,
+            -0.023652324452996254,
+            0.0010679832194000483,
+            -0.014168302528560162,
+            0.01098497025668621,
+            -0.008906732313334942,
+            0.022629698738455772,
+            -0.001510227215476334,
+            -0.010020074434578419,
+            -0.003008084138855338,
+            0.013137430883944035,
+            0.008956214413046837,
+            -0.022382289171218872,
+            -0.009978839196264744,
+            0.003024578094482422,
+            -0.03224566951394081,
+            -0.006622320506721735,
+            0.0025276977103203535,
+            0.0014566219178959727,
+            0.007401659619063139,
+            0.007059410214424133,
+            -0.014539416879415512,
+            -0.0037152620498090982,
+            0.0021524603944271803,
+            -0.01086951233446598,
+            0.005983180366456509,
+            0.015083717182278633,
+            0.017912428826093674,
+            0.011595246382057667,
+            0.007356301415711641,
+            -0.014028104022145271,
+            0.004222451243549585,
+            -0.015174433588981628,
+            -0.0053811511024832726,
+            0.005010037217289209,
+            -0.025252236053347588,
+            -0.012386956252157688,
+            0.02162356860935688,
+            0.02168954350054264,
+            -0.0005592479719780385,
+            -0.023718299344182014,
+            -0.01675785332918167,
+            0.012923009693622589,
+            -0.01309619564563036,
+            0.018885571509599686,
+            -0.0033936300314962864,
+            -0.014259018935263157,
+            -0.0016184687847271562,
+            0.017928922548890114,
+            -0.018275296315550804,
+            0.01598263718187809,
+            -0.02071640081703663,
+            0.009129401296377182,
+            0.002925614360719919,
+            0.0025957352481782436,
+            0.0014875481138005853,
+            0.050273556262254715,
+            -0.00046518092858605087,
+            0.0009396396926604211,
+            0.018737126141786575,
+            -0.033680643886327744,
+            0.02132667601108551,
+            0.003560631303116679,
+            0.00822223350405693,
+            0.0012885897886008024,
+            -0.01319515984505415,
+            0.0105891153216362,
+            0.019677281379699707,
+            0.027874775230884552,
+            -0.010094297118484974,
+            -0.007121262606233358,
+            0.018143344670534134,
+            0.003245184663683176,
+            -0.001950409496203065,
+            0.0029606639873236418,
+            0.004185339901596308,
+            -0.019330909475684166,
+            -0.0036822741385549307,
+            0.011166404001414776,
+            -0.014506428502500057,
+            -0.009624219499528408,
+            -0.014374476857483387,
+            -0.009409798309206963,
+            -0.008964461274445057,
+            -0.010382940992712975,
+            0.0005654331762343645,
+            -0.0026534642092883587,
+            0.014423958957195282,
+            -0.013013726100325584,
+            0.029590144753456116,
+            -0.013302369974553585,
+            -0.024526502937078476,
+            -0.0037544353399425745,
+            -0.007422277238219976,
+            0.02074938826262951,
+            -0.015446583740413189,
+            -0.010127284564077854,
+            -0.0034348650369793177,
+            -0.012535401619970798,
+            0.022448265925049782,
+            0.008931473828852177,
+            -0.006222342606633902,
+            -0.0061604902148246765,
+            0.020172100514173508,
+            0.024295587092638016,
+            -0.0256480909883976,
+            0.02144213393330574,
+            0.020139113068580627,
+            -0.008552112616598606,
+            -0.0033874448854476213,
+            0.002342140767723322,
+            0.005974933039397001,
+            -0.0015050729271024466,
+            0.010646844282746315,
+            0.006226466037333012,
+            -0.013830176554620266,
+            -0.0163784921169281,
+            0.01692279241979122,
+            0.018473222851753235,
+            -0.008750040084123611,
+            0.009063425473868847,
+            -0.003573001828044653,
+            0.002962725702673197,
+            -0.00333796301856637,
+            0.007682057097554207,
+            -0.009987086057662964,
+            -0.017961911857128143,
+            0.011042699217796326,
+            0.0029070586897432804,
+            -0.005715153645724058,
+            0.0037647439166903496,
+            0.023718299344182014,
+            -0.024510009214282036,
+            -0.013566273264586926,
+            0.005488361697643995,
+            -0.02159057930111885,
+            0.014687862247228622,
+            -0.0006107915542088449,
+            0.008824262768030167,
+            0.001291682361625135,
+            0.003952362574636936,
+            -0.024955345317721367,
+            0.009393304586410522,
+            0.008906732313334942,
+            0.0006061526364646852,
+            -0.000622646592091769,
+            -0.009022190235555172,
+            -0.01100146397948265,
+            -0.007381042465567589,
+            -0.006964570377022028,
+            -0.0108365248888731,
+            -0.02061743661761284,
+            0.023883238434791565,
+            0.0022864737547934055,
+            -0.006840865593403578,
+            0.0022947206161916256,
+            0.0136487428098917,
+            -0.0032163201831281185,
+            -0.004317291080951691,
+            0.010077803395688534,
+            -0.017219683155417442,
+            -0.006073896773159504,
+            0.005174976773560047,
+            0.007628451567143202,
+            -0.019710268825292587,
+            -0.009517008438706398,
+            -0.012436438351869583,
+            0.012362214736640453,
+            0.007199608720839024,
+            -0.015512559562921524,
+            -0.01100146397948265,
+            0.00257099443115294,
+            0.005034778267145157,
+            -0.05881742388010025,
+            -0.008543865755200386,
+            0.003925560042262077,
+            0.03879376873373985,
+            -0.003125603310763836,
+            -0.0158836729824543,
+            -0.002467907266691327,
+            0.02736346237361431,
+            0.00016519721248187125,
+            0.02058444917201996,
+            0.006271824240684509,
+            -0.009714935906231403,
+            -0.008477889932692051,
+            0.007591340225189924,
+            -0.005335792433470488,
+            -0.013929140754044056,
+            0.004696652293205261,
+            0.005880093201994896,
+            0.02703358232975006,
+            -0.010514892637729645,
+            0.0005762573564425111,
+            -0.0031379738356918097,
+            -0.001628777477890253,
+            -0.003711138619109988,
+            -0.005253322888165712,
+            0.013162171468138695,
+            -0.015529053285717964,
+            0.014333241619169712,
+            -0.01386316493153572,
+            0.004416254814714193,
+            0.007442894857376814,
+            -0.004511095117777586,
+            -0.002079268451780081,
+            -0.0031585912220180035,
+            0.002257609274238348,
+            0.0071954852901399136,
+            0.004911073483526707,
+            -0.024180129170417786,
+            0.012543648481369019,
+            0.0036987680941820145,
+            0.011784926988184452,
+            0.00779339112341404,
+            -0.023965708911418915,
+            -0.0026163526345044374,
+            0.014168302528560162,
+            0.011438554152846336,
+            0.004100808408111334,
+            0.008494383655488491,
+            -0.008102652616798878,
+            0.015124951489269733,
+            -0.04209255799651146,
+            0.00844490248709917,
+            -0.005789376329630613,
+            0.008275839500129223,
+            -0.011586999520659447,
+            0.009343822486698627,
+            0.007884107530117035,
+            -0.023668818175792694,
+            0.009888122789561749,
+            -0.002232868457213044,
+            -0.014044597744941711,
+            -0.015289891511201859,
+            -0.01055612787604332,
+            -0.00011519993131514639,
+            -0.012551895342767239,
+            0.0017648525536060333,
+            0.0026926372665911913,
+            -0.001965872710570693,
+            -0.024526502937078476,
+            -0.0007612988702021539,
+            -0.009517008438706398,
+            -0.008618088439106941,
+            -0.00555021408945322,
+            -0.009805653244256973,
+            0.012197275646030903,
+            0.008218110539019108,
+            -0.0030039604753255844,
+            -0.01086126547306776,
+            0.02173902466893196,
+            0.0005819271318614483,
+            0.000386576954042539,
+            -0.007987194694578648,
+            -0.004729640204459429,
+            -0.02686864323914051,
+            0.009055178612470627,
+            -0.02206890471279621,
+            -0.0005499701364897192,
+            0.002034940989688039,
+            0.0050760130397975445,
+            -0.014094079844653606,
+            0.0025421299505978823,
+            -0.00138961523771286,
+            0.024163635447621346,
+            0.0031544677913188934,
+            -0.013698224909603596,
+            0.009805653244256973,
+            -0.015496065840125084,
+            0.0108365248888731,
+            0.03278997167944908,
+            0.001166946953162551,
+            0.0011659160954877734,
+            -0.001972057856619358,
+            -0.003702891757711768,
+            0.008424284867942333,
+            0.0019442243501543999,
+            -0.03354869410395622,
+            -0.02655525878071785,
+            -0.009863382205367088,
+            0.011826162226498127,
+            0.0056780423037707806,
+            0.0011174650862812996,
+            -0.010119037702679634,
+            0.01593315601348877,
+            0.006601703353226185,
+            0.02578004263341427,
+            0.014720849692821503,
+            0.013038466684520245,
+            0.005525473039597273,
+            0.0042925504967570305,
+            -0.006275947671383619,
+            -0.009178882464766502,
+            -0.012362214736640453,
+            0.011883890256285667,
+            0.007987194694578648,
+            0.012123052962124348,
+            0.0010772610548883677,
+            -0.00163083930965513,
+            0.010110790841281414,
+            -0.018885571509599686,
+            -0.024114154279232025,
+            0.020007161423563957,
+            0.0020638054702430964,
+            0.002554500475525856,
+            -0.013013726100325584,
+            0.004717269446700811,
+            9.213417069986463e-05,
+            0.008898485451936722,
+            -0.005900710355490446,
+            0.00265140226110816,
+            0.01677434705197811,
+            0.0051502357237041,
+            0.007463512010872364,
+            -0.0012937441933900118,
+            -0.01692279241979122,
+            0.01108393445611,
+            -0.004750257357954979,
+            -0.015487818978726864,
+            -0.014151808805763721,
+            0.01095198281109333,
+            -0.0008185122278518975,
+            0.006226466037333012,
+            -0.0011803482193499804,
+            -0.011133415624499321,
+            0.017368128523230553,
+            -0.024575984105467796,
+            0.006510986480861902,
+            0.001821550540626049,
+            0.024872876703739166,
+            0.0012267375132068992,
+            0.0038616459351032972,
+            -0.020188594236969948,
+            0.028435569256544113,
+            -0.023916225880384445,
+            -0.007945960387587547,
+            -0.011347836814820766,
+            6.86818384565413e-05,
+            -0.0007463512010872364,
+            0.0037853613030165434,
+            -0.008251097984611988,
+            -0.02144213393330574,
+            0.008750040084123611,
+            0.024213118478655815,
+            0.0017081546830013394,
+            0.007748032920062542,
+            0.016386738047003746,
+            0.0016184687847271562,
+            -0.012667353264987469,
+            -0.0105478810146451,
+            -0.011248873546719551,
+            -0.011207638308405876,
+            -0.012255004607141018,
+            0.003812164068222046,
+            -0.018852584064006805,
+            0.015372361056506634,
+            0.007265584543347359,
+            -0.020337039604783058,
+            -0.0015050729271024466,
+            0.006374911405146122,
+            0.013566273264586926,
+            -0.007517117541283369,
+            0.012123052962124348,
+            0.019875209778547287,
+            0.002107101958245039,
+            -0.0006345016299746931,
+            0.00041080242954194546,
+            0.006746025290340185,
+            -0.015661004930734634,
+            0.030018987134099007,
+            0.006869730073958635,
+            0.0063955290243029594,
+            -0.01060560904443264,
+            0.018819596618413925,
+            -0.007261461112648249,
+            0.014481687918305397,
+            -0.018671151250600815,
+            -0.01337659265846014,
+            -0.014745591208338737,
+            0.011166404001414776,
+            -0.010630350559949875,
+            0.003573001828044653,
+            0.03582485765218735,
+            0.008576854132115841,
+            -0.002403993159532547,
+            -0.006016168277710676,
+            -0.015050728805363178,
+            -0.004333785269409418,
+            -0.013953881338238716,
+            -0.000127505962154828,
+            -0.025169767439365387,
+            -0.013409581035375595,
+            0.00807378813624382,
+            0.02122771367430687,
+            0.014836307615041733,
+            0.004671911243349314,
+            0.015067222528159618,
+            -0.007133633363991976,
+            0.01682382822036743,
+            0.018687644973397255,
+            0.01069632638245821,
+            -0.00012067643547197804,
+            -0.009813900105655193,
+            0.0010020073968917131,
+            0.025235742330551147,
+            -0.011199391447007656,
+            -0.00541001558303833,
+            0.02229982055723667,
+            0.02046899124979973,
+            0.0011102489661425352,
+            0.0058594755828380585,
+            0.010127284564077854,
+            0.009079919196665287,
+            0.014168302528560162,
+            -0.004523465409874916,
+            0.012584883719682693,
+            -0.01082003116607666,
+            -0.006506863050162792,
+            0.04816233366727829,
+            0.014448699541389942,
+            -0.0036059897392988205,
+            -0.012197275646030903,
+            -0.0020772067364305258,
+            0.0023895609192550182,
+            0.02107926830649376,
+            0.010069556534290314,
+            0.015289891511201859,
+            0.024608973413705826,
+            -0.007661439478397369,
+            0.014423958957195282,
+            -0.0059172045439481735,
+            0.04410482197999954,
+            -0.017236176878213882,
+            -0.004972925875335932,
+            -0.012535401619970798,
+            0.006840865593403578,
+            -0.00527394050732255,
+            0.018011393025517464,
+            -0.0011545764282345772,
+            0.0007509901188313961,
+            -0.028386086225509644,
+            0.019380390644073486,
+            0.012692093849182129,
+            -0.03829894959926605,
+            0.006234712898731232,
+            -0.009599478915333748,
+            -0.0019493786385282874,
+            -0.02645629458129406,
+            -0.004109055269509554,
+            -0.009987086057662964,
+            -0.0029194289818406105,
+            -0.025351200252771378,
+            0.007302696350961924,
+            -0.025318212807178497,
+            0.009426292032003403,
+            -0.0077851442620158195,
+            -0.0058718458749353886,
+            -0.005661548115313053,
+            -0.009269599802792072,
+            -0.0033523952588438988,
+            0.019495848566293716,
+            0.017203189432621002,
+            0.004878085572272539,
+            0.006783136632293463,
+            0.008914980106055737,
+            -0.011529270559549332,
+            -0.033944547176361084,
+            -0.008519125171005726,
+            -0.01657641865313053,
+            0.00834593828767538,
+            0.014324994757771492,
+            0.02597797103226185,
+            -0.02150811068713665,
+            -0.008799522183835506,
+            -0.009904616512358189,
+            -0.01571873389184475,
+            -0.010663338005542755,
+            0.010102543979883194,
+            -0.004564700648188591,
+            -0.023437902331352234,
+            0.0010468503460288048,
+            -0.005038901697844267,
+            0.004968802444636822,
+            -0.003812164068222046,
+            0.007273831870406866,
+            0.008956214413046837,
+            -0.011652975343167782,
+            -0.03125603497028351,
+            -0.0035297051072120667,
+            -0.001355596468783915,
+            -0.0028452062979340553,
+            -0.025417177006602287,
+            0.007756279781460762,
+            -0.015990883111953735,
+            -0.02112874947488308,
+            0.013294123113155365,
+            -0.007405783515423536,
+            0.006292441859841347,
+            0.018638163805007935,
+            0.013310616835951805,
+            0.010490152053534985,
+            0.033977534621953964,
+            -0.0011525147128850222,
+            -0.014382723718881607,
+            -0.011314849369227886,
+            0.014531169086694717,
+            -0.017005261033773422,
+            0.014011610299348831,
+            -0.007442894857376814,
+            -0.005884216632694006,
+            -0.0002360696526011452,
+            -0.0017968096071854234,
+            0.008857251144945621,
+            -0.009203623980283737,
+            0.00258336472325027,
+            -0.024575984105467796,
+            0.015413595363497734,
+            -0.016155824065208435,
+            -0.009970592334866524,
+            0.010226248763501644,
+            0.031025119125843048,
+            -0.015339372679591179,
+            -0.012634364888072014,
+            -0.0017504204297438264,
+            0.000949948385823518,
+            0.003818349214270711,
+            -0.006242959760129452,
+            -0.008123270235955715,
+            0.0051378654316067696,
+            0.0007159404922276735,
+            -0.016304269433021545,
+            0.0004020400228910148,
+            0.01111692190170288,
+            -0.028485050424933434,
+            -0.017500080168247223,
+            -0.007150127086788416,
+            -0.01566925272345543,
+            0.008267591707408428,
+            0.030101457610726357,
+            0.007084151264280081,
+            -0.006506863050162792,
+            -0.022794637829065323,
+            0.008246975019574165,
+            -0.027643859386444092,
+            0.02098030410706997,
+            0.07903900742530823,
+            -0.004820356611162424,
+            -0.0025441916659474373,
+            -0.018918560817837715,
+            -0.0014267265796661377,
+            0.0032946662977337837,
+            -0.017384622246026993,
+            0.05429808050394058,
+            -0.0014287884114310145,
+            0.004280179738998413,
+            -0.0076738097704946995,
+            0.010886006988584995,
+            -0.0267201978713274,
+            -0.02134316973388195,
+            -0.020205087959766388,
+            -0.010366447269916534,
+            -0.00543475616723299,
+            -0.0036245454102754593,
+            -0.013541532680392265,
+            0.0018225813983008265,
+            0.011958112940192223,
+            -0.00833769142627716,
+            0.0064078993164002895,
+            -0.024114154279232025,
+            -0.007166620809584856,
+            0.011067439801990986,
+            0.014885789714753628,
+            0.027990233153104782,
+            -0.030134445056319237,
+            -0.014588898047804832,
+            -0.007146003656089306,
+            0.008110899478197098,
+            -0.009302587248384953,
+            0.007644945755600929,
+            -0.0208483524620533,
+            -0.005908957216888666,
+            -0.010036568157374859,
+            0.014094079844653606,
+            0.014836307615041733,
+            -0.0022204979322850704,
+            -0.01058086846023798,
+            0.012090064585208893,
+            0.029227279126644135,
+            0.0015421842690557241,
+            0.009690195322036743,
+            -0.011677715927362442,
+            -0.011347836814820766,
+            0.0036925829481333494,
+            0.03151993826031685,
+            0.00652335723862052,
+            -0.00780576141551137,
+            -0.005785252898931503,
+            0.024641960859298706,
+            -0.004191524814814329,
+            0.01100146397948265,
+            -0.008939720690250397,
+            0.023157505318522453,
+            0.0016442405758425593,
+            0.009978839196264744,
+            0.011422060430049896,
+            -0.011933372355997562,
+            0.02076588198542595,
+            -0.0007344961632043123,
+            0.004589441232383251,
+            -0.0011009711306542158,
+            0.0017730995314195752,
+            -0.004737887065857649,
+            0.0040533882565796375,
+            0.003193640848621726,
+            -0.0013184851268306375,
+            0.010770549066364765,
+            3.614494562498294e-05,
+            0.005777005571871996,
+            -0.018951548263430595,
+            -0.012527154758572578,
+            0.013854917138814926,
+            -0.034406378865242004,
+            0.000833459896966815,
+            0.011191144585609436,
+            -0.02094731666147709,
+            0.02206890471279621,
+            -0.0077810208313167095,
+            -0.023256469517946243,
+            -0.007851120084524155,
+            0.00833769142627716,
+            0.004799739457666874,
+            -0.004968802444636822,
+            0.007884107530117035,
+            -0.007063533645123243,
+            0.005946068558841944,
+            -0.0024184253998100758,
+            0.02037002705037594,
+            -0.022431770339608192,
+            0.020534968003630638,
+            0.0005891432520002127,
+            0.01593315601348877,
+            -0.008849004283547401,
+            0.011273614130914211,
+            0.011908631771802902,
+            0.004498724825680256,
+            0.013722965493798256,
+            0.006003797519952059,
+            -0.014976506121456623,
+            0.014432205818593502,
+            0.008378926664590836,
+            0.009145895019173622,
+            0.02211838588118553,
+            0.014704355970025063,
+            0.011240626685321331,
+            -0.009517008438706398,
+            0.008947967551648617,
+            -0.011504529975354671,
+            -0.0015153816202655435,
+            0.01682382822036743,
+            0.02196994051337242,
+            -0.013945634476840496,
+            -0.004902826622128487,
+            -0.019034016877412796,
+            0.015240409411489964,
+            -0.0006375942612066865,
+            0.012692093849182129,
+            0.018390754237771034,
+            0.004882209002971649,
+            0.008980955928564072,
+            0.019743258133530617,
+            0.04215853661298752,
+            0.004799739457666874,
+            -0.0062182191759347916,
+            0.013978621922433376,
+            0.00795833021402359,
+            -0.005785252898931503,
+            0.00679963082075119,
+            -0.01674135960638523,
+            0.0003015300026163459,
+            0.009500514715909958,
+            -0.0025318211410194635,
+            -0.010020074434578419,
+            0.005983180366456509,
+            0.008741793222725391,
+            0.011413812637329102,
+            -0.012205522507429123,
+            0.010086050257086754,
+            -0.000819543085526675,
+            -0.004807986319065094,
+            -0.02063393034040928,
+            -0.013252888806164265,
+            0.03153643012046814,
+            -0.01348380371928215,
+            -0.008102652616798878,
+            0.02162356860935688,
+            0.017054744064807892,
+            0.004585317801684141,
+            -0.017582550644874573,
+            -0.005851228721439838,
+            0.012461178936064243,
+            -0.0011504529975354671,
+            0.023899732157588005,
+            -0.011240626685321331,
+            0.008634582161903381,
+            -0.014357983134686947,
+            -0.009376809932291508,
+            0.0011514838552102447,
+            0.012370462529361248,
+            0.011471541598439217,
+            0.02749541401863098,
+            0.014720849692821503,
+            -0.004086376167833805,
+            -0.0010834463173523545,
+            0.00791709590703249,
+            0.013533285818994045,
+            0.023108022287487984,
+            -0.0026122292038053274,
+            0.005047148559242487,
+            0.020089630037546158,
+            -0.00554609065875411,
+            0.012535401619970798,
+            -0.022382289171218872,
+            -0.014770331792533398,
+            -0.007574846502393484,
+            0.01099321711808443,
+            -0.011974607594311237,
+            0.017467092722654343,
+            0.0013102381490170956,
+            0.02691812627017498,
+            -0.0026823284570127726,
+            0.0028988115955144167,
+            0.0037647439166903496,
+            0.003985350485891104,
+            -0.0012257066555321217,
+            0.00805729441344738,
+            -0.02101329155266285,
+            0.01046541053801775,
+            -0.008519125171005726,
+            0.003183332271873951,
+            -0.01616406999528408,
+            0.0025895501021295786,
+            -0.007954207248985767,
+            -0.024312080815434456,
+            9.664423851063475e-05,
+            -0.014646627008914948,
+            -0.00529043423011899,
+            0.0018431989010423422,
+            0.004704899154603481,
+            -0.00820573978126049,
+            0.007966578006744385,
+            0.007071780972182751,
+            -0.014927024021744728,
+            0.0009128370438702404,
+            -0.00514198886230588,
+            -0.0001724133180687204,
+            0.0067047905176877975
+          ],
+          "keyphrases": [
+            "conditional computation",
+            "deep networks",
+            "Google Brain",
+            "neural machine translation",
+            "algorithmic and engineering solutions"
+          ],
+          "title": "6 Conclusion"
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "7f92225a-9c2d-4745-b7b1-2273bcee2b14",
+        "properties": {
+          "page_content": "Appendices A Load-Balancing Loss\n\nAs discussed in section 4, for load-balancing purposes, we want to define an additional loss function to encourage experts to receive roughly equal numbers of training examples. Unfortunately, the number of examples received by an expert is a discrete quantity, so it can not be used in backpropagation. Instead, we define a smooth estimator Load(X) of the number of examples assigned to each expert for a batch X of inputs. The smoothness allows us to back-propagate gradients through the estimator. This is the purpose of the noise term in the gating function. We define P(x, i) as the probability that G(x)iis nonzero, given a new random choice of noise on element i, but keeping the already-sampled choices of noise on the other elements. To compute P(x, i), we note that the G(x)iis nonzero if and only if H(x)iis greater than the k th-greatest element of H(x) excluding itself. The probability works out to be:\n\n$$P(x,i)=P r\\Big((x\\cdot W_{g}){i}+S t a n d x N o r a l()\\cdot S o f t p u s((x\\cdot W{n o i s e})_{i})$$ $$>k t h_e x c l u d i n(H(x),k,i)\\Big)$$ $$({\\boldsymbol{8}})$$\n\n$$(9)$$ $$(10)$$ (8) Where kth_excluding(v, k, i) means the kth highest component of v, excluding component i. Simplifying, we get:\n\n$$P(x,i)=\\Phi{\\Big(}{\\frac{(x\\cdot W_{g}){i}-k t h_c x c l u d i n g(H(x),k,i)}{S o f t p l u s((x\\cdot W{n o i s e})_{i})}}{\\Big)}$$\n\nWhere Φ is the CDF of the standard normal distribution.\n\n$$L o a d(X){i}=\\sum{x\\in X}P(x,i)$$\n\n$$(11)$$ P(x, i) (10) We can now define the load loss to be the square of the coefficient of variation of the load vector, multiplied by a hand-tuned scaling factor wload.\n\n$$L_{l o a d}(X)=w_{l o a d}\\cdot C V(L o a d(X))^{2}$$ Lload(X) = wload · CV (Load(X))2(11) Initial Load Imbalance: To avoid out-of-memory errors, we need to initialize the network in a state of approximately equal expert load (since the soft constraints need some time to work). To accomplish this, we initialize the matrices Wg and Wnoise to all zeros, which yields no signal and some noise.\n\nExperiments: We trained a set of models with identical architecture (the MoE-256 model described in Appendix C), using different values of wimportance and wload. We trained each model for 10 epochs, then measured perplexity on the test set. We also measured the coefficients of variation in Importance and Load, as well as ratio of the load on the most overloaded expert to the average load. This last value is significant for load balancing purposes on distributed hardware. All of these metrics were averaged over several training batches.\n\nwimportance wload Test Perplexity CV (Importance(X)) CV (Load(X)) max(Load(X)) mean(Load(X)) 0.0 0.0 39.8 3.04 3.01 17.80 0.2 0.0 35.6 0.06 0.17 1.47 0.0 0.2 35.7 0.22 0.04 1.15 0.1 0.1 35.6 0.06 0.05 1.14 0.01 0.01 35.7 0.48 0.11 1.37 1.0 1.0 35.7 0.03 0.02 1.07\n\nTable 6: Experiments with different combinations of losses.\n\n13 Results: Results are reported in Table 6. All the combinations containing at least one the two losses led to very similar model quality, where having no loss was much worse. Models with higher values of wload had lower loads on the most overloaded expert.\n\nB Hierachical Mixture Of Experts\n\nIf the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network.3If the hierarchical MoE consists of a groups of b experts each, we denote the primary gating network by Gprimary, the secondary gating networks by (G1, G2..Ga), and the expert networks by (E0,0, E0,1..Ea,b). The output of the MoE is given by:\n\n$$y_{H}=\\sum_{i=1}^{a}\\sum_{j=1}^{b}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}\\cdot E{i,j}(x)$$ $$(12)$$ $$(13)$$ $$(14)$$\n\nOur metrics of expert utilization change to the following:\n\n$$I m p o r t a n c e_{H}(X){i,j}=\\sum{x\\in X}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}$$ $$L o a d{H}(X){i,j}={\\frac{L o a d{p r i m a r y}(X){i}\\cdot L o a d{i}(X^{(i)})_{j}}{|X^{(i)}|}}$$\n\nLoadprimary and Loadi deonte the Load functions for the primary gating network and i th secondary gating network respectively. X(i) denotes the subset of X for which Gprimary(x)i > 0.\n\nIt would seem simpler to let LoadH(X)i,j = Loadi(Xi)j , but this would not have a gradient with respect to the primary gating network, so we use the formulation above.\n\nC 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS C.1 8-MILLION-OPERATIONS-PER-TIMESTEP MODELS Model Architecture: Our model consists of five layers: a word embedding layer, a recurrent Long Short-Term Memory (LSTM) layer (Hochreiter & Schmidhuber, 1997; Gers et al., 2000), a MoE layer, a second LSTM layer, and a softmax layer. The dimensionality of the embedding layer, the number of units in each LSTM layer, and the input and output dimensionality of the MoE layer are all equal to 512. For every layer other than the softmax, we apply drouput (Zaremba et al., 2014) to the layer output, dropping each activation with probability DropP rob, otherwise dividing by (1 − DropP rob). After dropout, the output of the previous layer is added to the layer output.\n\nThis residual connection encourages gradient flow (He et al., 2015).\n\nMoE Layer Architecture: Each expert in the MoE layer is a feed forward network with one ReLU-activated hidden layer of size 1024 and an output layer of size 512. Thus, each expert contains [512 ∗ 1024] + [1024 ∗ 512] = 1M parameters. The output of the MoE layer is passed through a sigmoid function before dropout. We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts.\n\nWe call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers. Thus, each example is processed by exactly 4 experts for a total of 4M ops/timestep. The two LSTM layers contribute 2M ops/timestep each for the desired total of 8M.\n\n3 We have not found the need for deeper hierarchies.\n\nComputationally-Matched Baselines: The MoE-4 model does not employ sparsity, since all 4 experts are always used. In addition, we trained four more computationally-matched baseline models with no sparsity:\n\nMoE-1-Wide: The MoE layer consists of a single \"expert\" containing one ReLU-activated hidden layer of size 4096.\n\nMoE-1-Deep: The MoE layer consists of a single \"expert\" containing four ReLU-activated hidden layers, each with size 1024.\n\n4xLSTM-512: We replace the MoE layer with two additional 512-unit LSTM layers.\n\nLSTM-2048-512: The model contains one 2048-unit LSTM layer (and no MoE). The output of the LSTM is projected down to 512 dimensions (Sak et al., 2014). The next timestep of the LSTM receives the projected output. This is identical to one of the models published in (Jozefowicz et al., 2016). We re-ran it to account for differences in training regimen, and obtained results very similar to the published ones. Training: The models were trained on a cluster of 16 K40 GPUs using the synchronous method described in Section 3. Each batch consisted of a set of sentences totaling roughly 300,000 words. In the interest of time, we limited training to 10 epochs, (27,000 steps). Training took 12-16 hours for all models, except for MoE-4, which took 18 hours (since all the expert computation was performed on only 4 of 16 GPUs). We used the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 1000 training steps, and decreased after that so as to be proportional to the inverse square root of the step number. The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016). For each model, we performed a hyper-parmeter search to find the best dropout probability, in increments of 0.1.\n\nTo ensure balanced expert utilization we set wimportance = 0.1 and wload = 0.1, as described in Section 4 and Appendix A.\n\nResults: We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol. Results are reported in Table 7. For each model, we report the test perplexity, the computational budget, the parameter counts, the value of DropP rob, and the computational efficiency.\n\nModel Test Test ops/timestep #Params excluding Total Drop- TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params P rob per GPU 10 epochs (final) (millions) (billions) (observed) Kneser-Ney 5-gram* 67.6 0.00001 1.8 LSTM-512-512* 54.1 2.4 2.4 0.8 0.1 LSTM-1024-512* 48.2 4.7 4.7 0.8 0.1 LSTM-2048-512* 45.0 43.7 9.4 9.4 0.8 0.1 0.61 LSTM-2048-512 44.7 9.4 9.4 0.8 0.1 1.21 4xLSTM-512 46.0 8.4 8.4 0.8 0.1 1.07 MoE-1-Wide 46.1 8.4 8.4 0.8 0.1 1.29 MoE-1-Deep 45.7 8.4 8.4 0.8 0.1 1.29 MoE-4 45.0 8.4 8.4 0.8 0.1 0.52 MoE-32 39.7 8.4 37.8 0.9 0.1 0.87 MoE-256 35.7 8.6 272.9 1.1 0.1 0.81 MoE-256-h 36.0 8.4 272.9 1.1 0.1 0.89 MoE-1024-h 34.6 8.5 1079.0 1.9 0.2 0.90 MoE-4096-h 34.1 8.9 4303.4 5.1 0.2 0.74 2xLSTM-8192-1024* 34.7 30.6 151.0 151.0 1.8 0.25 1.09 MoE-34M 31.3 33.8 4313.9 6.0 0.3 1.22 MoE-143M 28.0 142.7 4371.1 6.0 0.4 1.56\n\nC.2 More Expensive Models\n\nWe ran two additional models (MoE-34M and MoE-143M) to investigate the effects of adding more computation in the presence of a large MoE layer. These models have computation budgets of 34M and 143M ops/timestep. Similar to the models above, these models use a MoE layer between two LSTM layers. The dimensionality of the embedding layer, and the input and output dimensionality of the MoE layer are set to 1024 instead of 512. For MoE-34M, the LSTM layers have 1024 units.\n\nFor MoE-143M, the LSTM layers have 4096 units and an output projection of size 1024 (Sak et al., 2014). MoE-34M uses a hierarchical MoE layer with 1024 experts, each with a hidden layer of size 2048. MoE-143M uses a hierarchical MoE layer with 256 experts, each with a hidden layer of size 8192. Both models have 4B parameters in the MoE layers. We searched for the best DropP rob for each model, and trained each model for 10 epochs.\n\nThe two models achieved test perplexity of 31.3 and 28.0 respectively, showing that even in the presence of a large MoE, more computation is still useful. Results are reported at the bottom of Table 7. The larger of the two models has a similar computational budget to the best published model from the literature, and training times are similar. Comparing after 10 epochs, our model has a lower test perplexity by 18%.\n\nD 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS Model Architecture: The models are similar in structure to the 8-million-operations-per-timestep models described in the previous section. We vary the number of experts between models, using an ordinary MoE layer with 32 experts and hierarchical MoE layers with 256, 1024, 4096, 16384, 65536 and 131072 experts. For the hierarchical MoE layers, the first level branching factors are 32, 32, 64, 128, 256 and 256, respectively.\n\nTraining: Models are trained on a cluster of 32 Tesla K40 GPUs, except for the last two models, which are trained on clusters of 64 and 128 GPUs so as to have enough memory for all the parameters. For all models, training batch sizes are approximately 2.5 million words. Models are trained once-through over about 100 billion words. We implement several memory optimizations in order to fit up to 1 billion parameters per GPU. First, we do not store the activations of the hidden layers of the experts, but instead recompute them on the backwards pass. Secondly, we modify the optimizer on the expert parameters to require less auxiliary storage: The Adam optimizer (Kingma & Ba, 2015) keeps first and second moment estimates of the perparameter gradients. This triples the required memory. To avoid keeping a first-moment estimator, we set β1 = 0. To reduce the size of the second moment estimator, we replace it with a factored approximation. For a matrix of parameters, instead of maintaining a full matrix of second-moment estimators, we maintain vectors of row-wise and column-wise averages of that matrix. At each step, the matrix of estimators is taken to be the outer product of those two vectors divided by the mean of either one. This technique could similarly be applied to Adagrad (Duchi et al., 2010).\n\nModel Test Test ops/timestep #Params excluding Total TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params per GPU .1 epochs 1 epoch (millions) (billions) (observed) Kneser-Ney 5-gram 67.1 45.3 0.00001 76.0 4xLSTM-512 54.5 47.0 8.4 8.4 0.1 1.23 MoE-32 48.5 40.4 8.4 37.8 0.1 0.83 MoE-256-h 42.8 35.3 8.4 272.9 0.4 1.11 MoE-1024-h 40.3 32.7 8.5 1079.0 1.2 1.14 MoE-4096-h 38.9 30.9 8.6 4303.4 4.4 1.07 MoE-16384-h 38.2 29.7 8.8 17201.0 17.3 0.96 MoE-65536-h 38.2 28.9 9.2 68791.0 68.9 0.72 MoE-131072-h 39.8 29.2 9.7 137577.6 137.7 0.30\n\nResults: We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8. Perplexity after 100 billion training words is 39% lower for the 68-billion-parameter MoE model than for the baseline model. It is notable that the measured computational efficiency of the largest model (0.30 TFLOPS/GPU) is very low compared to the other models. This is likely a result of the fact that, for purposes of comparison to the other models, we did not increase the training batch size proportionally to the number of GPUs. For comparison, we include results for a computationally matched baseline model consisting of 4 LSTMs, and for an unpruned 5-gram model with Kneser-Ney smoothing (Kneser & Ney, 1995).4\n\nE Machine Translation - Experimental Details\n\nModel Architecture for Single Language Pair MoE Models: Our model is a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decrease the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We insert MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). We use an attention mechanism between the encoder and decoder, with the first decoder LSTM receiving output from and providing input for the attention 5. All of the layers in our model have input and output dimensionality of 512. Our LSTM layers have 2048 hidden units, with a 512-dimensional output projection. We add residual connections around all LSTM and MoE layers to encourage gradient flow (He et al., 2015). Similar to GNMT, to effectively deal with rare words, we used subword units (also known as \"wordpieces\") (Schuster & Nakajima, 2012) for inputs and outputs in our system.\n\nWe use a shared source and target vocabulary of 32K wordpieces. We also used the same beam search technique as proposed in (Wu et al., 2016).\n\nWe train models with different numbers of experts in the MoE layers. In addition to a baseline model with no MoE layers, we train models with flat MoE layers containing 32 experts, and models with hierarchical MoE layers containing 512 and 2048 experts. The flat MoE layers use k = 4 and the hierarchical MoE models use k = 2 at each level of the gating network. Thus, each input is processed by exactly 4 experts in each MoE layer. Each expert in the MoE layer is a feed forward network with one hidden layer of size 2048 and ReLU activation. Thus, each expert contains [512 ∗ 2048] + [2048 ∗ 512] = 2M parameters. The output of the MoE layer is passed through a sigmoid function. We use the strictly-balanced gating function described in Appendix F.\n\nModel Architecture for Multilingual MoE Model: We used the same model architecture as for the single-language-pair models, with the following exceptions: We used noisy-top-k gating as described in Section 2.1, not the scheme from Appendix F. The MoE layers in the encoder and decoder are non-hierarchical MoEs with n = 512 experts, and k = 2. Each expert has a larger hidden layer of size 8192. This doubles the amount of computation in the MoE layers, raising the computational budget of the entire model from 85M to 102M ops/timestep. Training: We trained our networks using the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 2000 training steps, held constant for an additional 8000 steps, and decreased after that so as to be proportional to the inverse square root of the step number. For the single-language-pair models, similarly to (Wu et al., 2016), we applied dropout (Zaremba et al., 2014) to the output of all embedding, LSTM and MoE layers, using DropP rob = 0.4. Training was done synchronously on a cluster of up to 64 GPUs as described in section 3. Each training batch consisted of a set of sentence pairs containing roughly 16000 words per GPU.\n\nTo ensure balanced expert utilization we set wimportance = 0.01 and wload = 0.01, as described in Section 4 and Appendix A. Metrics: We evaluated our models using the perplexity and the standard BLEU score metric. We reported tokenized BLEU score as computed by the multi-bleu.pl script, downloaded from the public implementation of Moses (on Github), which was also used in (Luong et al., 2015a). Results: Tables 2, 3 and 4 in Section 5.3 show comparisons of our results to other published methods. Figure 4 shows test perplexity as a function of number of words in the (training data's) source sentences processed for models with different numbers of experts. As can be seen from the Figure, as we increased the number of experts to approach 2048, the test perplexity of our model continued to improve.\n\nWe found that the experts indeed become highly specialized by syntax and/or semantics, as can be seen in Table 9. For example, one expert is used when the indefinite article \"a\" introduces the direct object in a verb phrase indicating importance or leadership.\n\nExpert 381 Expert 752 Expert 2004 ... with researchers , ... ... plays a core ... ... with rapidly growing ... ... to innovation . ... plays a critical ... ... under static conditions ... ... tics researchers . ... provides a legislative ... ... to swift ly ... ... the generation of ... ... play a leading ... ... to dras tically ... ... technology innovations is ... ... assume a leadership ... ... the rapid and ... ... technological innovations , ... ... plays a central ... ... the fast est ... ... support innovation throughout ... ... taken a leading ... ... the Quick Method ... ... role innovation will ... ... established a reconciliation ... ... rec urrent ) ... ... research scienti st ... ... played a vital ... ... provides quick access ... ... promoting innovation where ... ... have a central ... ... of volatile organic ... ... ... ...\n\nDue to some peculiarities in our infrastructure which have since been fixed, at the time we ran some of the machine translation experiments, our models ran faster if every expert received exactly the same batch size. To accommodate this, we used a different gating function which we describe below. Recall that we define the softmax gating function to be:\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ Gσ(x) = Sof tmax(x · Wg) (15) Sparse Gating (alternate formulation): To obtain a sparse gating vector, we multiply Gσ(x) component-wise with a sparse mask M(Gσ(x)) and normalize the output. The mask itself is a function of Gσ(x) and specifies which experts are assigned to each input example:\n\n$$(15)$$ $$G(x){i}=\\frac{G{\\sigma}(x){i}M(G{\\sigma}(x)){i}}{\\sum{j=1}^{n}G_{\\sigma}(x){j}M(G{\\sigma}(x))_{j}}\\tag{1}$$ $$(17)$$\n\n$$(16)$$\n\nTop-K Mask: To implement top-k gating in this formulation, we would let M(v) = T opK(v, k), where:\n\n$$T o p K(v,k){i}=\\begin{cases}1&{\\mathrm{if~}}v{i}{\\mathrm{~is~in~the~top~}}k{\\mathrm{~elements~of~}}v.\\ 0&{\\mathrm{otherwise.}}\\end{cases}$$ 0 otherwise. (17) Batchwise Mask: To force each expert to receive the exact same number of examples, we introduce an alternative mask function, Mbatchwise(X, m), which operates over batches of input vectors.\n\nInstead of keeping the top k values per example, we keep the top m values per expert across the training batch, where m = k|X| n, so that each example is sent to an average of k experts.\n\n$$M_{b a t c h w i s e}(X,m){j,i}=\\begin{cases}1&\\text{if}X{j,i}\\text{is in the top}m\\text{values for to expert}i\\ 0&\\text{otherwise}\\end{cases}$$\n\nAs our experiments suggest and also observed in (Ioffe & Szegedy, 2015), using a batchwise function during training (such as Mbatchwise) requires modifications to the inference when we may not have a large batch of examples. Our solution to this is to train a vector T of per-expert threshold values to approximate the effects of the batchwise mask. We use the following mask at inference time:\n\n$$(18)$$ $$M_{threshold}(x,T){i}=\\begin{cases}1&\\text{if}x{i}>T_{i}\\ 0&\\text{otherwise}\\end{cases}\\tag{1}$$ $$(19)$$\n\nTo learn the threshold values, we apply an additional loss at training time which is minimized when the batchwise mask and the threshold mask are identical.\n\n$$L_{batchwise}(X,T,m)=\\sum_{j=1}^{|X|}\\sum_{i=1}^{n}(M_{threshold}(x,T){i}-M{batchwise}(X,m){j,i})(X{j,i}-T_{i})\\tag{20}$$\n\nG ATTENTION FUNCTION The attention mechanism described in GNMT (Wu et al., 2016) involves a learned \"Attention Function\" A(xi, yj ) which takes a \"source vector\" xi and a \"target vector\" yj , and must be computed for every source time step i and target time step j. In GNMT, the attention function is implemented as a feed forward neural network with a hidden layer of size n. It can be expressed as:\n\n$$A_{G N M T}(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}t a n h((x_{i}U){d}+(y{j}W)_{d})$$\n\nWhere U and W are trainable weight matrices and V is a trainable weight vector.\n\nFor performance reasons, in our models, we used a slightly different attention function:\n\n$$A(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}tanh((x_{i}U){d})tanh((y{j}W)_{d})\\tag{22}$$\n\nWith our attention function, we can simultaneously compute the attention function on multiple source time steps and multiple target time steps using optimized matrix multiplications. We found little difference in quality between the two functions.\n\n$$(21)$$",
+          "embedding": [
+            0.0011764472583308816,
+            -0.018546752631664276,
+            -0.01668562740087509,
+            -0.003858148353174329,
+            0.009172027930617332,
+            0.015837986022233963,
+            -0.013212142512202263,
+            0.06935913860797882,
+            -0.011065399274230003,
+            0.035029683262109756,
+            0.016455290839076042,
+            0.002998990472406149,
+            0.020509224385023117,
+            -0.022867878898978233,
+            0.0126132657751441,
+            -0.002063821302726865,
+            0.02303372137248516,
+            0.015008772723376751,
+            -0.015994615852832794,
+            -0.019071921706199646,
+            -0.008485622704029083,
+            -0.004452418070286512,
+            -0.029298892244696617,
+            -0.004053934942930937,
+            -0.01113910786807537,
+            0.010531017556786537,
+            0.020085405558347702,
+            0.015220683068037033,
+            -0.0586714930832386,
+            -0.0015317423967644572,
+            -0.002752529690042138,
+            -0.0007854498689994216,
+            -0.015248322859406471,
+            -0.012097310274839401,
+            -0.027824735268950462,
+            -0.03810698911547661,
+            0.047799576073884964,
+            -0.00880348775535822,
+            -0.03169440105557442,
+            0.020896190777420998,
+            0.0326526015996933,
+            0.009411578066647053,
+            -0.009347083978354931,
+            -0.00776697089895606,
+            -0.009227308444678783,
+            0.001612360472790897,
+            0.012263152748346329,
+            0.010208545252680779,
+            -0.01241978257894516,
+            -0.031841814517974854,
+            -0.0024001137353479862,
+            -0.020306527614593506,
+            0.015008772723376751,
+            0.0039019123651087284,
+            -0.015156188048422337,
+            -0.008172363974153996,
+            0.0126132657751441,
+            0.048573512583971024,
+            0.05200092867016792,
+            0.04765216261148453,
+            0.038364965468645096,
+            0.01730293035507202,
+            -0.0017505628056824207,
+            -0.00107452308293432,
+            0.031086310744285583,
+            -0.020195966586470604,
+            -0.006030227988958359,
+            0.006214498076587915,
+            -0.016768548637628555,
+            -0.034440018236637115,
+            -0.06047733873128891,
+            0.02084091119468212,
+            0.007868318818509579,
+            -0.015764279291033745,
+            -0.00650011608377099,
+            -0.028635522350668907,
+            0.01830720156431198,
+            0.07252857834100723,
+            0.013009445741772652,
+            0.005095059052109718,
+            0.017883380874991417,
+            0.010816635563969612,
+            -0.015036413446068764,
+            0.01986428163945675,
+            0.024415744468569756,
+            -0.005647868383675814,
+            -0.032431479543447495,
+            -0.06198835000395775,
+            0.008844949305057526,
+            -0.011083826422691345,
+            -0.039323169738054276,
+            0.03440316393971443,
+            0.0019210122991353273,
+            -0.01265933271497488,
+            -0.0008666437352076173,
+            0.03983912244439125,
+            0.03114159032702446,
+            0.0012944950722157955,
+            -0.011793265119194984,
+            0.022093946114182472,
+            -0.002939102705568075,
+            0.0032569682225584984,
+            0.005164160393178463,
+            0.025171250104904175,
+            0.02139372006058693,
+            -0.005099665839225054,
+            -0.007439891807734966,
+            -0.007467532530426979,
+            0.039212606847286224,
+            0.11468950659036636,
+            -0.019606303423643112,
+            -0.0036969122011214495,
+            -0.03038608469069004,
+            -0.0014557312242686749,
+            -0.023678665980696678,
+            -0.015764279291033745,
+            -0.056386549025774,
+            -0.022204507142305374,
+            0.02006697840988636,
+            0.0007134695188142359,
+            0.030109679326415062,
+            0.051153287291526794,
+            0.031473275274038315,
+            -0.0012956467689946294,
+            -0.02800900489091873,
+            0.06519464403390884,
+            -0.02168855257332325,
+            0.025853049010038376,
+            -0.006477082148194313,
+            -0.04636227339506149,
+            -0.0012069669319316745,
+            0.018279561772942543,
+            -0.013875513337552547,
+            -0.031510129570961,
+            0.014760008081793785,
+            0.006771914195269346,
+            0.02799057774245739,
+            0.05008452385663986,
+            -0.029685860499739647,
+            -0.010733714327216148,
+            -0.0009679921204224229,
+            -0.02004855126142502,
+            -0.003448148025199771,
+            -0.012640906497836113,
+            -0.03899148106575012,
+            0.0012334557250142097,
+            0.015543155372142792,
+            0.07444498687982559,
+            -0.022296641021966934,
+            0.029354173690080643,
+            -0.01696203276515007,
+            -0.016519784927368164,
+            0.01671326719224453,
+            0.018887652084231377,
+            -0.008181577548384666,
+            -0.01625259406864643,
+            -0.022130798548460007,
+            0.009333264082670212,
+            -0.04415103793144226,
+            0.0015858716797083616,
+            -0.00802494864910841,
+            -0.008881802670657635,
+            -0.018344055861234665,
+            0.01060472521930933,
+            0.0071035996079444885,
+            0.033481817692518234,
+            0.04308227077126503,
+            -0.006698206067085266,
+            -0.058413516730070114,
+            0.05148497223854065,
+            0.005809104535728693,
+            0.04463013634085655,
+            0.030072826892137527,
+            0.03891777619719505,
+            -0.0029068556614220142,
+            0.01007034257054329,
+            0.010687646456062794,
+            0.008918656967580318,
+            -0.03639327734708786,
+            0.04595687985420227,
+            -0.023181136697530746,
+            0.0117564108222723,
+            0.016556639224290848,
+            0.04440901428461075,
+            0.03666968271136284,
+            0.01958787627518177,
+            0.009793938137590885,
+            0.019716864451766014,
+            -0.011719557456672192,
+            0.010770567692816257,
+            0.02760361135005951,
+            0.017579335719347,
+            0.04927373677492142,
+            0.017081808298826218,
+            0.00880348775535822,
+            -0.00950832013040781,
+            -0.015091693960130215,
+            0.020214393734931946,
+            -0.010365174151957035,
+            0.0126132657751441,
+            0.026387430727481842,
+            -0.03106788359582424,
+            -0.004744946490973234,
+            -0.014068996533751488,
+            -0.003989440388977528,
+            0.00300820404663682,
+            -0.011876186355948448,
+            -0.016694841906428337,
+            0.03066249005496502,
+            -0.01771753840148449,
+            -0.021854395046830177,
+            -0.0038051707670092583,
+            0.00054647505749017,
+            -0.017653044313192368,
+            -0.0017010403098538518,
+            0.01538652554154396,
+            -0.009517533704638481,
+            0.019035067409276962,
+            0.01853753812611103,
+            -0.0024876417592167854,
+            -0.0067212400026619434,
+            0.03226563706994057,
+            0.012631692923605442,
+            -0.003680788679048419,
+            -0.022978439927101135,
+            -0.010236185975372791,
+            -0.03145484998822212,
+            -0.04444586858153343,
+            0.017643829807639122,
+            -0.027179790660738945,
+            0.017579335719347,
+            0.024968553334474564,
+            -0.011102253571152687,
+            -0.012677759863436222,
+            0.02036180905997753,
+            -0.03563777357339859,
+            0.02331012487411499,
+            0.0023482879623770714,
+            -0.023862935602664948,
+            -0.009600454941391945,
+            -0.024913271889090538,
+            -0.01956944912672043,
+            -0.027437768876552582,
+            -0.002099523553624749,
+            -0.0045491596683859825,
+            -0.0019878100138157606,
+            -0.01796630211174488,
+            -0.015856413170695305,
+            -0.011968321166932583,
+            0.013064726255834103,
+            0.024397317320108414,
+            -0.04297171160578728,
+            0.012576411478221416,
+            -0.0023770800326019526,
+            -0.030883613973855972,
+            0.06814295798540115,
+            0.0442984513938427,
+            0.00950832013040781,
+            0.015469446778297424,
+            -0.047541599720716476,
+            0.03315013274550438,
+            0.024765856564044952,
+            -0.05155868083238602,
+            0.047799576073884964,
+            -0.001612360472790897,
+            -0.005776857025921345,
+            0.052885424345731735,
+            -0.004975283518433571,
+            0.0018519111908972263,
+            0.012475063093006611,
+            0.02846967987716198,
+            -0.014833716675639153,
+            -0.03521395102143288,
+            -0.03162069246172905,
+            0.02769574522972107,
+            -0.017312144860625267,
+            -0.03217350319027901,
+            0.0005597193958237767,
+            0.04440901428461075,
+            0.01614203117787838,
+            0.002754833083599806,
+            0.013654389418661594,
+            0.015174615196883678,
+            0.02120945043861866,
+            0.024986980482935905,
+            0.01589326746761799,
+            -0.014004502445459366,
+            9.407835750607774e-05,
+            -0.05056362599134445,
+            0.008273712359368801,
+            -0.020417090505361557,
+            -0.008563937619328499,
+            -0.014704727567732334,
+            -0.005085845477879047,
+            0.015672143548727036,
+            -0.003065788187086582,
+            -0.008149330504238605,
+            0.011949894018471241,
+            0.03455058112740517,
+            -0.03066249005496502,
+            -0.026590127497911453,
+            -0.008250678889453411,
+            -0.0036001706030219793,
+            0.0102730393409729,
+            -0.025558216497302055,
+            0.0033145523630082607,
+            0.004162193275988102,
+            0.029777994379401207,
+            -0.004226687829941511,
+            -0.0012323040282353759,
+            0.02084091119468212,
+            0.006725846324115992,
+            0.008872589096426964,
+            -0.0073661841452121735,
+            -0.016556639224290848,
+            0.008872589096426964,
+            0.0007664470467716455,
+            0.020877765491604805,
+            -0.02331012487411499,
+            -0.024231474846601486,
+            0.037222493439912796,
+            0.015027199871838093,
+            -0.04470384493470192,
+            0.02091461792588234,
+            -0.014336188323795795,
+            0.010328320786356926,
+            -0.012180231511592865,
+            0.0410921573638916,
+            -0.014907424338161945,
+            -0.024028778076171875,
+            0.003388260258361697,
+            -0.04142384231090546,
+            -0.02463686838746071,
+            -0.027253499254584312,
+            0.006085508968681097,
+            -0.02397349663078785,
+            0.005845958366990089,
+            0.0005499300896190107,
+            0.0009127111989073455,
+            -0.004892362281680107,
+            -0.01694360561668873,
+            0.0160959642380476,
+            0.016833042725920677,
+            -0.009052252396941185,
+            -0.013608322478830814,
+            0.009056858718395233,
+            -0.024913271889090538,
+            0.03388721123337746,
+            0.035306088626384735,
+            0.008370454423129559,
+            0.001825422397814691,
+            0.003524159314110875,
+            0.0060578687116503716,
+            -0.01534045860171318,
+            -0.010678432881832123,
+            -0.03687237948179245,
+            -0.04193979874253273,
+            0.013396412134170532,
+            -0.02837754413485527,
+            0.02817484736442566,
+            0.01318450178951025,
+            0.03751732409000397,
+            -0.020675068721175194,
+            0.0005908149178139865,
+            -0.050121378153562546,
+            -0.01739506609737873,
+            -0.0131292212754488,
+            -0.026295294985175133,
+            0.01566293090581894,
+            -0.031491704285144806,
+            -0.017247650772333145,
+            0.026442712172865868,
+            -0.007485959213227034,
+            -0.013479333370923996,
+            -0.01728450506925583,
+            0.003307642415165901,
+            0.05376992002129555,
+            -0.017588550224900246,
+            -0.025650352239608765,
+            0.02463686838746071,
+            -0.03305799514055252,
+            0.0005672053666785359,
+            0.02301529422402382,
+            0.008762027136981487,
+            0.0011442000977694988,
+            0.03316855803132057,
+            -0.0034619681537151337,
+            -0.018758662045001984,
+            0.00160084362141788,
+            0.027751026675105095,
+            -0.00611314969137311,
+            -0.036540694534778595,
+            -0.0026650014333426952,
+            -0.03226563706994057,
+            0.004079272039234638,
+            0.00807101558893919,
+            0.013110794126987457,
+            -0.019090348854660988,
+            -0.005288542248308659,
+            0.008084836415946484,
+            -0.01021775882691145,
+            -0.0018288774881511927,
+            0.011968321166932583,
+            0.032910581678152084,
+            -0.004533036146312952,
+            6.323476554825902e-05,
+            -0.02714293636381626,
+            0.016446076333522797,
+            0.04153440520167351,
+            -0.017183156684041023,
+            0.011636636219918728,
+            -0.012171017937362194,
+            -0.059371720999479294,
+            -0.0513744130730629,
+            0.04098159819841385,
+            0.014833716675639153,
+            -0.017708323895931244,
+            -0.011645849794149399,
+            -0.004203653894364834,
+            -0.037222493439912796,
+            -0.006730453111231327,
+            0.004611351061612368,
+            0.00632966635748744,
+            0.01519304234534502,
+            -0.012788321822881699,
+            0.028230128809809685,
+            0.017026526853442192,
+            0.018251920118927956,
+            0.008001914247870445,
+            -0.05491239205002785,
+            0.016648773103952408,
+            -0.003132586134597659,
+            0.024618441238999367,
+            -0.019809000194072723,
+            -0.005979553796350956,
+            -0.03237619996070862,
+            -0.034716423600912094,
+            0.022757316008210182,
+            -0.026111025363206863,
+            -0.04289800301194191,
+            -0.008757420815527439,
+            0.0358404703438282,
+            -0.0010618545347824693,
+            0.004005563911050558,
+            -0.01921933703124523,
+            -0.021467428654432297,
+            -0.019164055585861206,
+            -0.010503376834094524,
+            -0.013654389418661594,
+            -0.0235128216445446,
+            -0.06497351825237274,
+            -0.008808095008134842,
+            -0.0442984513938427,
+            -0.01725686341524124,
+            0.029114622622728348,
+            -0.010383601300418377,
+            0.028598668053746223,
+            -0.052111491560935974,
+            -0.00685483543202281,
+            -0.04153440520167351,
+            0.008084836415946484,
+            0.025908328592777252,
+            0.023715518414974213,
+            -0.0075274198316037655,
+            0.04319283366203308,
+            0.012134164571762085,
+            -0.030644062906503677,
+            -0.031215298920869827,
+            -0.00568472221493721,
+            -0.013626749627292156,
+            -0.00014914335042703897,
+            -0.024784283712506294,
+            0.06335194408893585,
+            0.02883821912109852,
+            0.05122699588537216,
+            0.02876451052725315,
+            -0.018076865002512932,
+            -0.02616630680859089,
+            -0.028340689837932587,
+            0.006449441891163588,
+            0.05447014421224594,
+            -0.021559562534093857,
+            -0.003945676609873772,
+            -0.013608322478830814,
+            0.0029874735046178102,
+            -0.012327647767961025,
+            0.0146125927567482,
+            0.03449530154466629,
+            -0.012253939174115658,
+            -0.017266077920794487,
+            0.004417867865413427,
+            0.01947731524705887,
+            -0.020232820883393288,
+            0.03545350208878517,
+            -0.005740003194659948,
+            -0.04488811641931534,
+            -0.009545174427330494,
+            0.020011696964502335,
+            0.028193274512887,
+            0.01910877414047718,
+            -0.0073892176151275635,
+            -0.026037318632006645,
+            0.011959107592701912,
+            0.023346979171037674,
+            0.004510002676397562,
+            -0.022775743156671524,
+            0.013774164952337742,
+            0.017597762867808342,
+            -0.006799554452300072,
+            -0.04956856742501259,
+            0.022886306047439575,
+            -0.02894878014922142,
+            0.02600046433508396,
+            -0.003966406919062138,
+            0.01899821311235428,
+            0.001621573930606246,
+            -0.01853753812611103,
+            0.022996867075562477,
+            -0.006675172131508589,
+            0.00989528652280569,
+            -0.0034919120371341705,
+            -0.012134164571762085,
+            -0.0012472759699448943,
+            0.05664452537894249,
+            0.0038535415660589933,
+            0.0047910138964653015,
+            0.0410921573638916,
+            0.013488546945154667,
+            0.013811019249260426,
+            -0.015460233204066753,
+            -0.039028335362672806,
+            0.025576643645763397,
+            -0.019053494557738304,
+            -0.013811019249260426,
+            -0.026903385296463966,
+            -0.028635522350668907,
+            -0.011415512301027775,
+            0.021320011466741562,
+            -0.027106082066893578,
+            -0.014842930249869823,
+            -0.0005844806437380612,
+            0.0168975368142128,
+            -0.0363011434674263,
+            0.009282589890062809,
+            0.019827427342534065,
+            -0.0022642146795988083,
+            0.02426832914352417,
+            -0.03217350319027901,
+            0.0007566577405668795,
+            0.02139372006058693,
+            0.008656072430312634,
+            -0.0001934832544066012,
+            -0.001268006395548582,
+            0.020896190777420998,
+            -0.02893035300076008,
+            -0.00601640809327364,
+            0.0059426999650895596,
+            0.010549443773925304,
+            -0.009379331022500992,
+            -0.01637236960232258,
+            0.009388544596731663,
+            0.0020960685797035694,
+            -0.01022697240114212,
+            0.010190118104219437,
+            -0.020619787275791168,
+            -0.03895463049411774,
+            0.0053944974206388,
+            0.008011127822101116,
+            -0.018979785963892937,
+            0.0069976444356143475,
+            0.0018887651385739446,
+            -0.0028469678945839405,
+            0.026866532862186432,
+            -0.005458991974592209,
+            0.032247208058834076,
+            0.04249260947108269,
+            0.01611439138650894,
+            0.008301353082060814,
+            0.00578146381303668,
+            -0.0067442734725773335,
+            0.02963057905435562,
+            -0.008292139507830143,
+            0.0016273324145004153,
+            0.026018891483545303,
+            0.010881129652261734,
+            -0.006384947337210178,
+            -0.007025284692645073,
+            -0.018251920118927956,
+            -0.0035287661012262106,
+            -0.014750794507563114,
+            0.017892595380544662,
+            0.041129011660814285,
+            -0.00030836393125355244,
+            -0.022867878898978233,
+            0.013470119796693325,
+            -0.02205709181725979,
+            -0.00439483392983675,
+            0.012309220619499683,
+            0.01847304403781891,
+            -0.039875976741313934,
+            -0.010577084496617317,
+            0.023623384535312653,
+            0.021320011466741562,
+            -0.012585625052452087,
+            -0.02876451052725315,
+            0.014022929593920708,
+            0.009185847826302052,
+            0.0167777631431818,
+            -0.002157107926905155,
+            -0.007485959213227034,
+            0.008038768544793129,
+            0.006380340550094843,
+            -0.007928206585347652,
+            -0.011820905841886997,
+            0.018251920118927956,
+            0.01979057304561138,
+            0.0063204532489180565,
+            -0.015644503757357597,
+            -0.023531248793005943,
+            0.024010350927710533,
+            0.03497440367937088,
+            -0.016888324171304703,
+            0.0194036066532135,
+            0.0036347212735563517,
+            0.011056185699999332,
+            0.00874360091984272,
+            -0.009821578860282898,
+            -0.009968994185328484,
+            -0.0004307305789552629,
+            -0.0037061257753521204,
+            -0.011194388382136822,
+            -0.005302362609654665,
+            -0.016860684379935265,
+            -0.012263152748346329,
+            0.053069692105054855,
+            0.010392814874649048,
+            0.014824503101408482,
+            -0.0009841157589107752,
+            -0.019237764179706573,
+            -0.007559667341411114,
+            0.011784051544964314,
+            0.02647956646978855,
+            0.004072362091392279,
+            0.03162069246172905,
+            -0.005058205220848322,
+            0.013479333370923996,
+            -0.024600014090538025,
+            0.0019382876344025135,
+            0.020011696964502335,
+            0.016409222036600113,
+            -0.0023678664583712816,
+            -0.0027502262964844704,
+            0.004544553346931934,
+            0.020896190777420998,
+            -0.007297082804143429,
+            -0.046583399176597595,
+            0.018316414207220078,
+            -0.015745852142572403,
+            0.02664540894329548,
+            0.04234519228339195,
+            0.01513776183128357,
+            -0.000702528515830636,
+            0.030883613973855972,
+            -0.021559562534093857,
+            0.015856413170695305,
+            -0.02577934041619301,
+            -0.027880016714334488,
+            0.014824503101408482,
+            -0.0067442734725773335,
+            0.013921581208705902,
+            0.01074292790144682,
+            -0.018758662045001984,
+            -0.02559507079422474,
+            0.015128548257052898,
+            0.007707083132117987,
+            -0.029685860499739647,
+            0.008421128615736961,
+            0.025060689076781273,
+            0.0011384416138753295,
+            0.01026382576674223,
+            0.004095395561307669,
+            0.01680540293455124,
+            -0.01189461350440979,
+            -0.007601127959787846,
+            0.029114622622728348,
+            -0.010973264463245869,
+            -0.013230569660663605,
+            0.000723258825019002,
+            -0.04853665828704834,
+            -0.007988094352185726,
+            -0.0020649731159210205,
+            -0.02091461792588234,
+            -0.001933680847287178,
+            0.011857759207487106,
+            0.006200677715241909,
+            0.010180904529988766,
+            0.015294390730559826,
+            0.043708790093660355,
+            -0.0009547477238811553,
+            -0.002020057290792465,
+            -0.0018231190042570233,
+            -0.030736196786165237,
+            -0.01744113303720951,
+            -0.011572141200304031,
+            -0.002694945316761732,
+            0.019348325207829475,
+            -0.0029183723963797092,
+            -0.011341804638504982,
+            -0.02292315848171711,
+            0.016657987609505653,
+            0.029372600838541985,
+            0.056902505457401276,
+            0.0031602266244590282,
+            0.043524518609046936,
+            0.0007048318511806428,
+            0.02828541025519371,
+            0.019274618476629257,
+            0.008660678751766682,
+            -0.046104297041893005,
+            0.01223551295697689,
+            0.022130798548460007,
+            0.03217350319027901,
+            0.01805843785405159,
+            -0.047357331961393356,
+            -0.004242811352014542,
+            0.006366520654410124,
+            0.03735148161649704,
+            0.027179790660738945,
+            0.013967648148536682,
+            -0.0034619681537151337,
+            0.022757316008210182,
+            -0.029593724757432938,
+            -0.021080462262034416,
+            -0.0005450354074127972,
+            -0.011931467801332474,
+            0.00437180045992136,
+            -0.018638886511325836,
+            0.029022488743066788,
+            -0.02111731469631195,
+            -0.0356193445622921,
+            -0.006255958694964647,
+            -0.008702139370143414,
+            -0.001766686444170773,
+            -0.002782473573461175,
+            0.009066072292625904,
+            0.028782937675714493,
+            0.004316519480198622,
+            0.027013948187232018,
+            -0.01118517480790615,
+            0.02559507079422474,
+            0.004715002607554197,
+            -0.00549123901873827,
+            0.008296746760606766,
+            -0.018325628712773323,
+            -0.006610678043216467,
+            -0.009351690299808979,
+            -0.007937420159578323,
+            0.00712202675640583,
+            0.006864049006253481,
+            -0.009254949167370796,
+            -0.010927197523415089,
+            0.019551021978259087,
+            0.0032892152667045593,
+            0.010005848482251167,
+            0.026332149282097816,
+            -0.010475736111402512,
+            -0.0011493826750665903,
+            0.024028778076171875,
+            0.002002781955525279,
+            -0.02349439635872841,
+            0.005924272816628218,
+            0.0013843266060575843,
+            0.037222493439912796,
+            -0.021246304735541344,
+            -0.008338207378983498,
+            0.020951472222805023,
+            0.000956475269049406,
+            -0.0028561814688146114,
+            -0.03913889825344086,
+            0.0062052845023572445,
+            -0.013285850174725056,
+            0.01022697240114212,
+            0.007974274456501007,
+            0.016400009393692017,
+            -0.01389394048601389,
+            0.006186857353895903,
+            0.004413261078298092,
+            0.00858236476778984,
+            -0.0075043863616883755,
+            0.018178213387727737,
+            -0.0030151139944791794,
+            -0.05074789375066757,
+            -0.006656745448708534,
+            -0.014971918426454067,
+            0.020509224385023117,
+            0.012825176119804382,
+            -0.004035507794469595,
+            0.020988326519727707,
+            -0.02531866542994976,
+            -0.030128106474876404,
+            0.03499282896518707,
+            -0.03283687308430672,
+            0.014916637912392616,
+            -0.0005496421945281327,
+            -0.032818444073200226,
+            -0.040834181010723114,
+            -0.018961358815431595,
+            -0.00040280219400301576,
+            -0.0052977558225393295,
+            0.004065451677888632,
+            -0.013718884438276291,
+            -0.017109448090195656,
+            -0.02616630680859089,
+            0.02109888754785061,
+            -0.0020039337687194347,
+            0.013682030141353607,
+            0.002022360684350133,
+            -0.012898883782327175,
+            -0.014962704852223396,
+            -0.01801236905157566,
+            -0.012382928282022476,
+            0.01810450479388237,
+            -0.016427649185061455,
+            -0.014262479729950428,
+            -0.017864953726530075,
+            -0.004896969068795443,
+            -0.012935738079249859,
+            0.0010215454967692494,
+            0.03637485206127167,
+            0.03565619885921478,
+            0.009079893119633198,
+            0.018942931666970253,
+            -0.020232820883393288,
+            0.045993734151124954,
+            -0.023936642333865166,
+            0.034532155841588974,
+            0.0001733287499519065,
+            -0.020859338343143463,
+            -0.005606407765299082,
+            0.024194620549678802,
+            -0.016519784927368164,
+            -0.01357146818190813,
+            0.004461631644517183,
+            -0.002443877747282386,
+            0.023549675941467285,
+            0.02159641683101654,
+            0.01447439007461071,
+            -0.004507699050009251,
+            0.02577934041619301,
+            0.004634384531527758,
+            -0.0006092418916523457,
+            0.0303676575422287,
+            0.02493169903755188,
+            0.008642252534627914,
+            -0.030920466408133507,
+            -0.013811019249260426,
+            -0.01584720052778721,
+            0.0071358466520905495,
+            0.012143378145992756,
+            0.001317528891377151,
+            0.0056985425762832165,
+            -0.014354614540934563,
+            0.004228990990668535,
+            -0.008315172977745533,
+            0.008633038960397243,
+            0.02244405820965767,
+            -0.010171690955758095,
+            -0.03093889355659485,
+            -0.028395971283316612,
+            -0.01851911097764969,
+            -0.003362923162057996,
+            -0.00018139055464416742,
+            -0.01666720025241375,
+            0.020582932978868484,
+            -0.0003740100364666432,
+            0.005620228126645088,
+            -0.014032143168151379,
+            0.0027225858066231012,
+            0.005993374157696962,
+            -0.012447422370314598,
+            0.0015064053004607558,
+            0.0055833738297224045,
+            -0.0015156188746914268,
+            -0.041239574551582336,
+            -0.004615957848727703,
+            0.004438598174601793,
+            0.004832474514842033,
+            -0.03455058112740517,
+            -0.03331597521901131,
+            -0.03270788490772247,
+            0.006767307408154011,
+            -0.003961800131946802,
+            -0.023936642333865166,
+            0.0131292212754488,
+            -0.01156292762607336,
+            -0.007416858337819576,
+            -0.023641811683773994,
+            -0.01151686068624258,
+            0.01938517950475216,
+            0.0014292424311861396,
+            0.01685146987438202,
+            -0.008200004696846008,
+            -0.00261663063429296,
+            -0.0204355176538229,
+            0.006993037648499012,
+            -0.03068091720342636,
+            0.0012023602612316608,
+            -0.009913713671267033,
+            -0.0005608710926026106,
+            -0.026921812444925308,
+            0.0022503945510834455,
+            0.0044109574519097805,
+            0.013746525160968304,
+            -0.03534294292330742,
+            0.0065646106377244,
+            0.011194388382136822,
+            0.016630345955491066,
+            -0.00720955478027463,
+            0.03244990482926369,
+            0.01094562467187643,
+            -0.005482025444507599,
+            0.03132586181163788,
+            0.05682879686355591,
+            0.022480912506580353,
+            -0.03364766016602516,
+            -0.01394922100007534,
+            -0.012198658660054207,
+            0.013811019249260426,
+            -0.014723154716193676,
+            -0.010770567692816257,
+            -0.008771240711212158,
+            0.023457542061805725,
+            -0.005362250376492739,
+            0.0075274198316037655,
+            0.04020766168832779,
+            -0.00880348775535822,
+            0.009591241367161274,
+            0.006417194847017527,
+            -0.012327647767961025,
+            0.00601640809327364,
+            -0.00658764410763979,
+            0.01710023358464241,
+            0.030920466408133507,
+            0.013534614816308022,
+            0.023328552022576332,
+            -0.04205036163330078,
+            0.0204355176538229,
+            0.010503376834094524,
+            0.006960790604352951,
+            0.022407203912734985,
+            0.028801364824175835,
+            0.016086751595139503,
+            -0.007195734418928623,
+            -0.026442712172865868,
+            -0.036337997764348984,
+            0.0020718832965940237,
+            -0.03373979404568672,
+            -0.0034435412380844355,
+            -0.014262479729950428,
+            0.023144282400608063,
+            0.004795620683580637,
+            -0.019458888098597527,
+            -0.013534614816308022,
+            0.006960790604352951,
+            -0.02091461792588234,
+            -0.004961463622748852,
+            -0.009867645800113678,
+            -0.021522708237171173,
+            -0.024102484807372093,
+            0.02120945043861866,
+            -0.009351690299808979,
+            -0.029298892244696617,
+            0.010125624015927315,
+            0.001757472869940102,
+            0.0003642207011580467,
+            -0.0007860257173888385,
+            -0.020509224385023117,
+            -0.03943372890353203,
+            -0.02082248404622078,
+            0.0025935969315469265,
+            0.032910581678152084,
+            -0.026313722133636475,
+            0.03554563969373703,
+            -0.007559667341411114,
+            0.007508993148803711,
+            -0.0023701698519289494,
+            0.009185847826302052,
+            -0.01634472794830799,
+            -0.038549236953258514,
+            0.01785574108362198,
+            -0.0010491859866306186,
+            0.026111025363206863,
+            0.006334273144602776,
+            -0.027474623173475266,
+            -0.0058275312185287476,
+            0.025742486119270325,
+            0.005048991646617651,
+            -0.013737311586737633,
+            -0.007375397253781557,
+            -0.005320789758116007,
+            0.011526074260473251,
+            -0.04157125949859619,
+            -0.003858148353174329,
+            0.06368362903594971,
+            -0.013276636600494385,
+            0.02732720598578453,
+            0.019090348854660988,
+            -0.03283687308430672,
+            -0.04824182391166687,
+            -0.021946528926491737,
+            -0.001739045954309404,
+            -0.008467195555567741,
+            -0.024342035874724388,
+            0.019661584869027138,
+            -0.002150197746232152,
+            -0.001446517650038004,
+            0.005910452920943499,
+            -0.0042727552354335785,
+            -0.026240015402436256,
+            0.0004845719086006284,
+            -0.015275963582098484,
+            -0.0006691296002827585,
+            -0.03466114401817322,
+            -0.02874608337879181,
+            0.02319956384599209,
+            0.014935065060853958,
+            -0.0055833738297224045,
+            -0.0008309414843097329,
+            -0.013995288871228695,
+            -0.0037521931808441877,
+            0.017579335719347,
+            -0.0010486101964488626,
+            0.012014389038085938,
+            0.00955438707023859,
+            0.031768109649419785,
+            0.0035517998039722443,
+            0.003777530277147889,
+            -0.049236882477998734,
+            -0.011673489585518837,
+            0.02760361135005951,
+            0.003952586557716131,
+            -0.017957089468836784,
+            0.0008326690294779837,
+            -0.00437180045992136,
+            -0.009517533704638481,
+            -0.038070134818553925,
+            0.030146533623337746,
+            0.046583399176597595,
+            -0.0022319676354527473,
+            -0.003123372560366988,
+            0.019256191328167915,
+            5.466009952215245e-06,
+            0.010300680063664913,
+            -0.005417531356215477,
+            -0.012336861342191696,
+            0.030902039259672165,
+            0.005905846133828163,
+            -0.00028720172122120857,
+            0.014778435230255127,
+            -0.009402364492416382,
+            -0.0024576981086283922,
+            -0.02561349794268608,
+            -0.0004474300367292017,
+            -0.028340689837932587,
+            0.006601464468985796,
+            0.0006265172269195318,
+            0.006467869039624929,
+            0.010660005733370781,
+            0.029391027987003326,
+            0.03668811172246933,
+            0.016768548637628555,
+            -0.00218590022996068,
+            -0.010917983949184418,
+            0.029980691149830818,
+            0.022794170305132866,
+            0.009922927245497704,
+            0.022812597453594208,
+            -0.031970806419849396,
+            -0.010825849138200283,
+            0.01065079215914011,
+            0.0015294391196221113,
+            -0.0008384274551644921,
+            0.008324386551976204,
+            0.009812365286052227,
+            0.0049061826430261135,
+            -0.005882812198251486,
+            0.016105176880955696,
+            0.007347756996750832,
+            0.014953491277992725,
+            -0.0020557595416903496,
+            -0.004404047504067421,
+            0.009738657623529434,
+            -0.012336861342191696,
+            0.03517710044980049,
+            -0.018712595105171204,
+            -0.014253266155719757,
+            0.031123163178563118,
+            0.10658163577318192,
+            0.0036278110928833485,
+            -0.014741581864655018,
+            -0.02034338191151619,
+            -0.0010428517125546932,
+            0.0235128216445446,
+            0.009922927245497704,
+            0.02972271293401718,
+            0.009204274974763393,
+            0.0026488779112696648,
+            0.02426832914352417,
+            0.029649006202816963,
+            0.0031072490382939577,
+            0.0006656745099462569,
+            -0.024950126186013222,
+            0.008715960197150707,
+            0.002158259740099311,
+            -0.005979553796350956,
+            -0.005380677059292793,
+            0.010365174151957035,
+            0.017551695927977562,
+            0.005431351251900196,
+            -0.004035507794469595,
+            -0.011820905841886997,
+            -0.016353942453861237,
+            0.005458991974592209,
+            -0.01881394349038601,
+            -0.009411578066647053,
+            -0.017984729260206223,
+            -0.009784724563360214,
+            0.00013971391308587044,
+            -0.003088821889832616,
+            -0.005947306752204895,
+            0.013580681756138802,
+            -0.019698437303304672,
+            0.01977214589715004,
+            -0.008522477000951767,
+            -0.00026863077073358,
+            0.0012587928213179111,
+            -0.010051915422081947,
+            0.017772819846868515,
+            0.0021167988888919353,
+            -0.029925409704446793,
+            0.013635963201522827,
+            0.008204611018300056,
+            -0.005772250704467297,
+            0.02052765153348446,
+            -0.03344496339559555,
+            0.019809000194072723,
+            -0.0013981468509882689,
+            -0.015561581589281559,
+            0.0040309010073542595,
+            -0.018408549949526787,
+            0.005997980944812298,
+            0.006831801496446133,
+            0.006361913867294788,
+            -0.0014879783848300576,
+            0.009075285866856575,
+            0.005634048022329807,
+            -0.0016054504085332155,
+            -0.002223905874416232,
+            0.006767307408154011,
+            -0.0011056186631321907,
+            -0.00016728240007068962,
+            -0.014824503101408482,
+            -0.011811692267656326,
+            -0.0002376792108407244,
+            -0.00412073265761137,
+            -0.0037452830001711845,
+            -0.01739506609737873,
+            0.003547193016856909,
+            0.00501674460247159,
+            0.013488546945154667,
+            -0.002655788091942668,
+            -0.010733714327216148,
+            0.0019117988413199782,
+            0.007799217943102121,
+            -0.014216412790119648,
+            -0.005099665839225054,
+            -0.030902039259672165,
+            -0.008646858856081963,
+            0.02196495607495308,
+            0.005882812198251486,
+            -0.025152822956442833,
+            -0.021062035113573074,
+            0.004450114909559488,
+            -0.016446076333522797,
+            0.006725846324115992,
+            0.007163487374782562,
+            -0.013046300038695335,
+            0.005578767042607069,
+            -0.010881129652261734,
+            -0.0031602266244590282,
+            -0.0029644398018717766,
+            -0.00865146517753601,
+            0.000947261753026396,
+            -0.010770567692816257,
+            -0.0020465462002903223,
+            -0.012410569004714489,
+            -0.014437536709010601,
+            -0.05399104207754135,
+            0.008223038166761398,
+            0.01753326877951622,
+            -0.0027686532121151686,
+            -0.024802710860967636,
+            -0.013718884438276291,
+            0.013820232823491096,
+            -0.0038788786623626947,
+            -0.01867574080824852,
+            0.005067418795078993,
+            0.01657506637275219,
+            -0.008780454285442829,
+            -0.006108542904257774,
+            -0.014013716019690037,
+            0.027179790660738945,
+            0.018076865002512932,
+            0.02874608337879181,
+            -0.018795516341924667,
+            0.021909674629569054,
+            0.02284945175051689,
+            -0.026866532862186432,
+            -0.010291466489434242,
+            0.03385035693645477,
+            0.01045730896294117,
+            0.004989103879779577,
+            0.006753487046808004,
+            0.0011989051708951592,
+            -0.0346979983150959,
+            -0.007002251222729683,
+            0.0020373326260596514,
+            -0.031196871772408485,
+            0.022407203912734985,
+            0.014898210763931274,
+            0.006449441891163588,
+            0.01332270447164774,
+            -0.014511244371533394,
+            0.010531017556786537,
+            0.015368098393082619,
+            0.021191023290157318,
+            -0.0071266330778598785,
+            -0.010696860030293465,
+            0.013101580552756786,
+            0.01737663894891739,
+            -0.013672816567122936,
+            0.004544553346931934,
+            0.014750794507563114,
+            -0.018482258543372154,
+            0.03764631226658821,
+            -0.010153263807296753,
+            0.006923936307430267,
+            -0.013460906222462654,
+            -0.019698437303304672,
+            0.03283687308430672,
+            -0.0032362379133701324,
+            0.000820000481326133,
+            -0.024507878348231316,
+            0.009241129271686077,
+            0.005002924241125584,
+            0.019753718748688698,
+            0.014603379182517529,
+            0.02646113932132721,
+            0.023162709549069405,
+            0.004090788774192333,
+            -0.02867237664759159,
+            -0.0013992985477671027,
+            -0.020232820883393288,
+            0.00740303797647357,
+            -0.06250430643558502,
+            -0.026516418904066086,
+            0.017708323895931244,
+            -0.01956944912672043,
+            -0.006094722542911768,
+            -0.008955510333180428,
+            0.0022780350409448147,
+            0.025373946875333786,
+            -0.017127875238656998,
+            0.014962704852223396,
+            6.658185156993568e-05,
+            0.011719557456672192,
+            -0.013202928937971592,
+            0.011093039996922016,
+            -0.014326974749565125,
+            -0.04050249606370926,
+            0.01332270447164774,
+            0.035582493990659714,
+            0.02963057905435562,
+            -0.016565851867198944,
+            0.02360495738685131,
+            -0.01899821311235428,
+            0.013276636600494385,
+            0.005283935461193323,
+            -0.009033825248479843,
+            -0.0034020806197077036,
+            0.0004123036051169038,
+            -0.009545174427330494,
+            -0.013046300038695335,
+            -0.006099329330027103,
+            -0.027179790660738945,
+            0.003427417716011405,
+            -0.034145187586545944,
+            -0.019090348854660988,
+            0.018942931666970253,
+            0.0012230905704200268,
+            -0.009683376178145409,
+            0.027345633134245872,
+            -0.04164496809244156,
+            0.005288542248308659,
+            -0.026424285024404526,
+            0.02657170034945011,
+            -0.025834621861577034,
+            0.008877196349203587,
+            -0.016750121489167213,
+            -0.011480006389319897,
+            0.020656641572713852,
+            -0.0010278797708451748,
+            0.0063434867188334465,
+            -0.011618209071457386,
+            -0.03274473920464516,
+            -0.0045583732426166534,
+            0.02979642152786255,
+            -0.010917983949184418,
+            0.005915059708058834,
+            0.021835967898368835,
+            -0.005283935461193323,
+            -0.01428090687841177,
+            -0.00246000150218606,
+            0.007011464796960354,
+            -0.04186609014868736,
+            0.012769894674420357,
+            0.01899821311235428,
+            0.01204202976077795,
+            0.008642252534627914,
+            0.0056985425762832165,
+            0.013626749627292156,
+            0.006085508968681097,
+            -0.006219104863703251,
+            0.048684071749448776,
+            0.0002988625201396644,
+            -0.00037055497523397207,
+            -0.004710395820438862,
+            0.0043902271427214146,
+            0.0002775563334580511,
+            -0.009102926589548588,
+            0.016427649185061455,
+            0.0021617147140204906,
+            0.00016742636216804385,
+            0.007117419969290495,
+            0.006062475498765707,
+            -0.027953723445534706,
+            0.023254845291376114,
+            -0.029980691149830818,
+            0.00029051280580461025,
+            0.03611687570810318,
+            -0.016105176880955696,
+            0.000605211011134088,
+            0.007200341206043959,
+            -0.0022573047317564487,
+            0.016722481697797775,
+            -0.017616190016269684,
+            -0.024673722684383392,
+            0.02120945043861866,
+            0.022186079993844032,
+            0.013663602992892265,
+            0.03326069191098213,
+            -0.020656641572713852,
+            0.02082248404622078,
+            -0.011940681375563145,
+            -0.009420791640877724,
+            0.004238204564899206,
+            0.0032753951381891966,
+            -0.01591169461607933,
+            0.018270347267389297,
+            -0.017708323895931244,
+            -0.0023909001611173153,
+            -0.004590620752424002,
+            -0.016040682792663574,
+            -0.0009904500329867005,
+            -0.010319107212126255,
+            -0.009370117448270321,
+            -0.0026028105057775974,
+            -0.013921581208705902,
+            0.10665534436702728,
+            0.0005424441187642515,
+            -0.0137557378038764,
+            0.014824503101408482,
+            -0.0033352829050272703,
+            -0.014492817223072052,
+            -0.004192137159407139,
+            0.001406208612024784,
+            -0.00640337448567152,
+            -0.007882139645516872,
+            -0.013608322478830814,
+            -0.0004266996693331748,
+            0.001249579363502562,
+            0.01026382576674223,
+            -0.009139780886471272,
+            -0.029188331216573715,
+            0.009420791640877724,
+            -0.02970428578555584,
+            0.01204202976077795,
+            0.005081238690763712,
+            0.015091693960130215,
+            0.006647531874477863,
+            0.009978207759559155,
+            -0.007177307270467281,
+            -0.013737311586737633,
+            0.031178444623947144,
+            -0.007098992820829153,
+            0.0022791866213083267,
+            -0.0024254508316516876,
+            -0.010835062712430954,
+            -0.012640906497836113,
+            -0.02196495607495308,
+            0.024950126186013222,
+            -0.01616045832633972,
+            0.007181914057582617,
+            0.006338879931718111,
+            0.00218590022996068,
+            0.030164960771799088,
+            0.0008608853095211089,
+            -0.02006697840988636,
+            0.00620989128947258,
+            -0.035766761749982834,
+            -0.005260901991277933,
+            0.019937988370656967,
+            -0.014345401898026466,
+            -0.04079732671380043,
+            -0.008923263289034367,
+            -0.009116746485233307,
+            0.0030335411429405212,
+            0.00068352569360286,
+            -0.014686300419270992,
+            0.003503429004922509,
+            -0.003699215594679117,
+            0.018657313659787178,
+            0.008826522156596184,
+            -0.0077485437504947186,
+            0.002402417128905654,
+            -0.0002808674471452832,
+            0.006454048678278923,
+            -0.013313490897417068,
+            0.005776857025921345,
+            -0.019827427342534065,
+            0.018509898334741592,
+            -0.008144724182784557,
+            -0.0204355176538229,
+            0.005095059052109718,
+            0.014253266155719757,
+            0.006592250894755125,
+            -0.003673878498375416,
+            -0.008200004696846008,
+            0.014068996533751488,
+            0.023678665980696678,
+            0.014124277979135513,
+            0.018657313659787178,
+            0.03781215474009514,
+            0.001152837765403092,
+            0.032247208058834076,
+            -0.02255461923778057,
+            0.018592819571495056,
+            -0.04116586595773697,
+            -0.0009167420794256032,
+            -0.009932140819728374,
+            -0.005638654809445143,
+            -0.0016238773241639137,
+            0.02301529422402382,
+            -0.014013716019690037,
+            0.0029644398018717766,
+            0.015985403209924698,
+            -0.010503376834094524,
+            0.012410569004714489,
+            0.01132337749004364,
+            -0.01828877441585064,
+            0.013479333370923996,
+            -0.0068502286449074745,
+            0.004818654619157314,
+            -0.027916869148612022,
+            0.005832138005644083,
+            -0.004450114909559488,
+            -0.005108879413455725,
+            -0.0015386525774374604,
+            -0.0016803100006654859,
+            -0.008273712359368801,
+            -0.009282589890062809,
+            -0.035011254251003265,
+            -0.005270115565508604,
+            -5.72963799641002e-05,
+            -0.006970004178583622,
+            0.024821138009428978,
+            0.029096195474267006,
+            0.011811692267656326,
+            0.00695157703012228,
+            0.000879312283359468,
+            0.012815962545573711,
+            0.005578767042607069,
+            0.0234022606164217,
+            -0.02579776756465435,
+            0.00412764260545373,
+            -0.007140453439205885,
+            -0.013672816567122936,
+            0.013728098012506962,
+            -0.004323429428040981,
+            -0.013350344263017178,
+            0.027769453823566437,
+            -0.02026967518031597,
+            0.007605734746903181,
+            -0.00029713500407524407,
+            -0.002948316279798746,
+            -0.0031302827410399914,
+            -0.01794787496328354,
+            0.0015881750732660294,
+            -0.00242314743809402,
+            -0.026387430727481842,
+            -0.0030128106009215117,
+            -0.008540904149413109,
+            -0.02244405820965767,
+            0.02310742810368538,
+            0.01703573949635029,
+            -0.01623416692018509,
+            0.0262584425508976,
+            -0.006808768026530743,
+            -0.013700457289814949,
+            -0.01228157989680767,
+            -0.0017805066891014576,
+            -0.004086181987076998,
+            0.009720230475068092,
+            -0.019937988370656967,
+            -0.011461579240858555,
+            0.0015121637843549252,
+            -0.02426832914352417,
+            0.015598435886204243,
+            0.017588550224900246,
+            -0.013433266431093216,
+            0.029483163729310036,
+            -0.0046320813708007336,
+            0.008287533186376095,
+            0.017800459638237953,
+            0.009738657623529434,
+            -0.019735291600227356,
+            -0.022609900683164597,
+            0.012788321822881699,
+            0.0051687671802937984,
+            -0.0065646106377244,
+            0.00027035828679800034,
+            -0.02472900226712227,
+            0.0009691438172012568,
+            0.007711689919233322,
+            0.001163778710179031,
+            0.0027133722323924303,
+            0.0036485414020717144,
+            0.0014119670959189534,
+            0.0021525011397898197,
+            -0.007587307598441839,
+            0.006131576374173164,
+            -0.02826698310673237,
+            0.013414839282631874,
+            0.004325732588768005,
+            0.002100675366818905,
+            -0.03475327789783478,
+            -0.001816208939999342,
+            -0.0011989051708951592,
+            -0.00821843184530735,
+            -0.012106523849070072,
+            0.023899788036942482,
+            0.021264731884002686,
+            0.015036413446068764,
+            0.025871476158499718,
+            -0.010816635563969612,
+            0.013746525160968304,
+            -0.011212815530598164,
+            0.018500683829188347,
+            -0.0041253394447267056,
+            -0.010328320786356926,
+            -0.008292139507830143,
+            0.03777530416846275,
+            -0.010171690955758095,
+            0.01260405220091343,
+            0.0024830352049320936,
+            0.023733945563435555,
+            0.0015098603907972574,
+            0.0005044384743086994,
+            0.002237726002931595,
+            -0.008697533048689365,
+            -0.0059196664951741695,
+            0.007513599935919046,
+            0.06983824074268341,
+            -0.011028545908629894,
+            0.005744609981775284,
+            0.010448095388710499,
+            -0.007637981791049242,
+            -0.006034834776073694,
+            -0.005763037130236626,
+            -0.030422938987612724,
+            0.01685146987438202,
+            -0.0017344391671940684,
+            -0.01920090988278389,
+            0.013580681756138802,
+            0.017505627125501633,
+            0.018030796200037003,
+            0.000927107292227447,
+            0.017726751044392586,
+            -0.005472812335938215,
+            0.0155800087377429,
+            0.02703237533569336,
+            0.009411578066647053,
+            -0.008153936825692654,
+            -0.009545174427330494,
+            0.007016071584075689,
+            0.0010779781732708216,
+            -0.01228157989680767,
+            0.003489608643576503,
+            -0.015229896642267704,
+            0.026866532862186432,
+            -0.006025621201843023,
+            0.04182923585176468,
+            -0.012097310274839401,
+            -0.00623292475938797,
+            -0.0011344107333570719,
+            -0.007260228972882032,
+            0.011885399930179119,
+            0.005629441235214472,
+            0.008748207241296768,
+            -0.0021835968364030123,
+            -0.014262479729950428,
+            0.018040010705590248,
+            0.002409327309578657,
+            -0.00040021090535447,
+            0.004966070409864187,
+            -0.00466893520206213,
+            -0.00474955327808857,
+            0.012622479349374771,
+            0.009913713671267033,
+            -0.008614611811935902,
+            0.005399104207754135,
+            -0.007149667013436556,
+            -0.015561581589281559,
+            -0.01065079215914011,
+            0.002605113899335265,
+            -0.008734387345612049,
+            -0.007370790932327509,
+            0.0038604517467319965,
+            0.020472371950745583,
+            0.007168094161897898,
+            0.012134164571762085,
+            -0.020122257992625237,
+            -0.0006063627079129219,
+            0.017266077920794487,
+            -0.010448095388710499,
+            0.001591630163602531,
+            0.004058541730046272,
+            0.011820905841886997,
+            -0.013147647492587566,
+            0.006200677715241909,
+            0.010144051164388657,
+            0.0037153391167521477,
+            0.01618809998035431,
+            0.006154610309749842,
+            0.003945676609873772,
+            0.004721913021057844,
+            -0.0014246356440708041,
+            0.0003927249344997108,
+            -0.009264162741601467,
+            1.6843408957356587e-05,
+            -0.005025957711040974,
+            -0.0023102823179215193,
+            -0.015451019629836082,
+            0.009738657623529434,
+            0.012217085808515549,
+            0.01680540293455124,
+            -0.026626981794834137,
+            0.011959107592701912,
+            -0.010503376834094524,
+            0.02741934172809124,
+            0.018021583557128906,
+            -0.02618473395705223,
+            0.016105176880955696,
+            -0.01079820841550827,
+            0.015626076608896255,
+            0.0018208156106993556,
+            -0.01675933599472046,
+            0.004528429359197617,
+            -0.004590620752424002,
+            0.004249721299856901,
+            -0.01833484135568142,
+            -0.024305181577801704,
+            0.021651698276400566,
+            0.0011275006690993905,
+            0.0155800087377429,
+            0.005205621011555195,
+            0.013700457289814949,
+            0.0007929358398541808,
+            0.01787416823208332,
+            0.007333936635404825,
+            0.005366857163608074,
+            -0.011397085152566433,
+            0.008909443393349648,
+            -0.0047910138964653015,
+            -0.007969667203724384,
+            -0.006785734090954065,
+            -0.010567870922386646,
+            -0.0005983008886687458,
+            -0.01065079215914011,
+            -0.024986980482935905,
+            -0.018565179780125618,
+            -0.007826858200132847,
+            0.02378922700881958,
+            -0.000312682765070349,
+            -0.0018092987593263388,
+            -0.03547193109989166,
+            -0.001534045790322125,
+            -0.003911125939339399,
+            -0.01997484266757965,
+            -0.003017417388036847,
+            -0.0205460786819458,
+            -0.017570123076438904,
+            -0.00611314969137311,
+            -0.02399192377924919,
+            -0.005035171285271645,
+            -0.018122931942343712,
+            -0.01428090687841177,
+            0.006435621529817581,
+            0.005002924241125584,
+            0.01265011914074421,
+            0.028617095202207565,
+            0.01150764711201191,
+            -0.007347756996750832,
+            -0.014686300419270992,
+            -0.01299101859331131,
+            -0.014197985641658306,
+            0.011258882470428944,
+            0.0096465228125453,
+            0.013995288871228695,
+            -1.3226394912635442e-05,
+            0.010844276286661625,
+            0.00018714898033067584,
+            0.02026967518031597,
+            -0.02924361266195774,
+            -0.03038608469069004,
+            0.0026511813048273325,
+            -0.000183118085260503,
+            -0.004219777882099152,
+            -0.0033053390216082335,
+            -0.005500452592968941,
+            -0.00932405050843954,
+            -0.015165401622653008,
+            -0.006292812526226044,
+            2.4365357603528537e-05,
+            -0.006638318300247192,
+            0.024526305496692657,
+            0.020103830844163895,
+            0.006007194519042969,
+            0.0016423043562099338,
+            0.01881394349038601,
+            0.006804161239415407,
+            0.00805719569325447,
+            -0.00289073190651834,
+            0.0043050022795796394,
+            0.0062283179722726345,
+            0.005131912883371115,
+            -0.004648204892873764,
+            0.02905934303998947,
+            0.0014799166237935424,
+            0.011995961889624596,
+            0.014446749351918697,
+            0.03256046772003174,
+            -0.0013359558070078492,
+            0.025963610038161278,
+            0.012161804363131523,
+            -0.009655735455453396,
+            -0.01611439138650894,
+            0.008301353082060814,
+            -0.008135510608553886,
+            0.006251351907849312,
+            0.01632630079984665,
+            -0.02071192115545273,
+            -0.007200341206043959,
+            0.03394249081611633,
+            -0.01389394048601389,
+            -0.012788321822881699,
+            0.017745178192853928,
+            0.007370790932327509,
+            -0.003231631126254797,
+            -0.0008505201549269259,
+            0.004609047435224056,
+            0.001758624566718936,
+            -0.004152979701757431,
+            0.019348325207829475,
+            0.004309609066694975,
+            0.003989440388977528,
+            0.012530344538390636,
+            -0.0019970235880464315,
+            -0.0076103415340185165,
+            -0.0016745515167713165,
+            0.01662113331258297,
+            0.004222081042826176,
+            -0.017791246995329857,
+            -0.006887082476168871,
+            0.013811019249260426,
+            -0.0041575864888727665,
+            -0.004590620752424002,
+            0.025152822956442833,
+            -0.002501462120562792,
+            -0.012576411478221416,
+            0.0011338349431753159,
+            0.006624497938901186,
+            -0.003643934614956379,
+            0.011986748315393925,
+            0.023439114913344383,
+            -0.003038147697225213,
+            -0.0018116021528840065,
+            -0.009056858718395233,
+            0.007923600263893604,
+            -0.008513263426721096,
+            0.013424052856862545,
+            -0.029372600838541985,
+            0.01002427563071251,
+            -0.012327647767961025,
+            0.005095059052109718,
+            0.012318434193730354,
+            0.01768989861011505,
+            0.0204355176538229,
+            0.013801805675029755,
+            -0.007043711841106415,
+            -0.011737984605133533,
+            -0.012917310930788517,
+            -0.0013843266060575843,
+            -0.009172027930617332,
+            -0.01901664026081562,
+            0.004970677196979523,
+            0.019937988370656967,
+            0.017127875238656998,
+            0.009259555488824844,
+            -0.015027199871838093,
+            0.019993269816040993,
+            -2.9781880584778264e-05,
+            0.007711689919233322,
+            0.0028976420871913433,
+            -0.01280674897134304,
+            -0.0020476977806538343,
+            -0.015828773379325867,
+            0.016400009393692017,
+            -0.014326974749565125,
+            0.01628023386001587,
+            0.030349230393767357,
+            -0.01337798498570919,
+            -0.013967648148536682,
+            -0.014262479729950428,
+            -0.00429118238389492,
+            -0.019035067409276962,
+            0.026442712172865868,
+            0.018076865002512932,
+            -0.016980459913611412,
+            -0.014741581864655018,
+            -0.019937988370656967,
+            0.01920090988278389,
+            0.015045627020299435,
+            0.007108206395059824,
+            0.01921933703124523,
+            0.02026967518031597,
+            0.006182250566780567,
+            0.0036761818919330835,
+            -0.01094562467187643,
+            0.0010889191180467606,
+            0.017339784651994705,
+            0.0016388492658734322,
+            -0.011111467145383358,
+            -0.017044954001903534,
+            0.0039963508024811745,
+            -0.02109888754785061,
+            0.008831128478050232,
+            0.006366520654410124,
+            -0.011480006389319897,
+            0.016040682792663574,
+            -0.024544732645154,
+            0.0073892176151275635,
+            -0.012023602612316608,
+            0.0029736533761024475,
+            -0.036724966019392014,
+            0.0002545226307120174,
+            0.003602473996579647,
+            0.006246745120733976,
+            0.023365406319499016,
+            -0.02990698255598545,
+            -0.01733057200908661,
+            0.003339889459311962,
+            -0.006435621529817581,
+            0.003643934614956379,
+            -0.014916637912392616,
+            0.003385957097634673,
+            -0.00484629487618804,
+            -0.0011177113046869636,
+            0.0036900020204484463,
+            -0.003945676609873772,
+            0.009319443255662918,
+            0.015128548257052898,
+            -0.017026526853442192,
+            -0.00016339545254595578,
+            0.017643829807639122,
+            0.014446749351918697,
+            -0.02255461923778057,
+            0.006688992492854595,
+            -0.013626749627292156,
+            0.006454048678278923,
+            -0.027069229632616043,
+            0.011222029104828835,
+            -0.005302362609654665,
+            0.01481528952717781,
+            -0.0031302827410399914,
+            -0.004351069685071707,
+            -0.0076425885781645775,
+            0.014713941141963005,
+            0.004574496764689684,
+            0.007568880915641785,
+            0.017201581969857216,
+            -0.018297987058758736,
+            0.012364501133561134,
+            -0.007158880587667227,
+            -0.016473717987537384,
+            0.006665959022939205,
+            -0.00675809383392334,
+            0.002443877747282386,
+            -0.02331012487411499,
+            -0.005440564826130867,
+            -0.0011453517945483327,
+            -0.019643157720565796,
+            0.008849555626511574,
+            -0.02999911829829216,
+            0.0001314361725235358,
+            -0.01700809970498085,
+            -0.0031072490382939577,
+            -0.007591914385557175,
+            -0.011655062437057495,
+            -0.02915147691965103,
+            -0.009406971745193005,
+            0.023328552022576332,
+            -0.009729444049298763,
+            0.011774837970733643,
+            0.0009916017297655344,
+            0.007914386689662933,
+            0.00585517194122076,
+            -0.0007952392334118485,
+            0.008522477000951767,
+            0.016704054549336433,
+            -0.014400682412087917,
+            -0.0012645513052120805,
+            0.005353036802262068,
+            0.024600014090538025,
+            -0.019422033801674843,
+            -0.01318450178951025,
+            -0.00865146517753601,
+            0.0019198606023564935,
+            0.017643829807639122,
+            -0.02102518081665039,
+            -0.002324102446436882,
+            -0.001895675202831626,
+            0.005486632231622934,
+            -0.006841015070676804,
+            -0.01441910956054926,
+            -0.009609668515622616,
+            -0.00790056586265564,
+            -0.0009622336947359145,
+            -0.019329898059368134,
+            0.01910877414047718,
+            -0.020490799099206924,
+            0.003613990731537342,
+            0.008821914903819561,
+            -0.010890343226492405,
+            -0.005348430015146732,
+            -0.016676414757966995,
+            0.024120911955833435,
+            0.011885399930179119,
+            -0.002489945152774453,
+            0.005002924241125584,
+            -0.00041950165177695453,
+            0.013221356086432934,
+            -0.006338879931718111,
+            0.003779833670705557,
+            0.0003184411907568574,
+            -0.01150764711201191,
+            -0.005707756150513887,
+            0.023328552022576332,
+            -0.007587307598441839,
+            0.021172596141695976,
+            -0.0018023886950686574,
+            0.005417531356215477,
+            0.004599833860993385,
+            -0.004627474583685398,
+            0.0080479821190238,
+            -0.003604777390137315,
+            -0.008577757515013218,
+            -0.019716864451766014,
+            0.02472900226712227,
+            -0.022573046386241913,
+            -0.0025820801965892315,
+            0.025171250104904175,
+            0.0016941301291808486,
+            -0.001815057243220508,
+            -0.012263152748346329,
+            -0.011949894018471241,
+            0.025337092578411102,
+            0.011378658004105091,
+            0.004256631713360548,
+            -0.014502030797302723,
+            -0.007269442547112703,
+            0.0017148605547845364,
+            -0.013506974093616009,
+            0.01222629938274622,
+            -0.012152590788900852,
+            0.021670125424861908,
+            -0.013875513337552547,
+            0.024231474846601486,
+            0.010641579516232014,
+            0.01586562767624855,
+            0.016031470149755478,
+            -0.019053494557738304,
+            0.0022711248602718115,
+            -0.006421801634132862,
+            0.008181577548384666,
+            0.0009587786626070738,
+            -0.026866532862186432,
+            -0.0029229791834950447,
+            0.009453038685023785,
+            0.0037268560845404863,
+            -0.01805843785405159,
+            0.00776697089895606,
+            -0.015322031453251839,
+            0.019716864451766014,
+            0.007301689591258764,
+            -0.002683428581804037,
+            -0.010917983949184418,
+            0.0028653948102146387,
+            -0.001710253767669201,
+            0.008568543940782547,
+            0.0005744034424424171,
+            0.008738993667066097,
+            0.013230569660663605,
+            0.01890607737004757,
+            -0.02883821912109852,
+            -0.0006230621365830302,
+            -0.005790677387267351,
+            -0.027364060282707214,
+            -0.008305959403514862,
+            0.009471465833485126,
+            0.016704054549336433,
+            -0.0007359273731708527,
+            -0.0004033780423924327,
+            -0.004583710338920355,
+            0.0028193274047225714,
+            0.0007566577405668795,
+            0.0004030901181977242,
+            0.013396412134170532,
+            -0.010954837314784527,
+            0.011102253571152687,
+            0.00021853242651559412,
+            -0.0061361831612885,
+            -0.016169672831892967,
+            0.02588990330696106,
+            -0.006122363265603781,
+            0.019993269816040993,
+            0.02447102591395378,
+            0.012576411478221416,
+            0.004205957520753145,
+            -0.014704727567732334,
+            0.019274618476629257,
+            0.012742254883050919,
+            -0.0029943836852908134,
+            -0.017339784651994705,
+            0.0024669114500284195,
+            0.0042612385004758835,
+            0.0005887994775548577,
+            -0.019993269816040993,
+            0.007016071584075689,
+            0.0008320931810885668,
+            -0.008278319612145424,
+            -0.020030124112963676,
+            -0.016842257231473923,
+            -0.024047205224633217,
+            0.02244405820965767,
+            0.0001376984582748264,
+            -0.014575738459825516,
+            -0.012244726531207561,
+            0.00013928202679380774,
+            0.006444835104048252,
+            -0.006887082476168871,
+            -0.005532699637115002,
+            0.009075285866856575,
+            -0.013442480005323887,
+            -0.00310494564473629,
+            0.0066291047260165215,
+            0.009070679545402527,
+            -0.0072418018244206905,
+            -0.01098247803747654,
+            -0.01785574108362198,
+            0.025742486119270325,
+            -0.009259555488824844,
+            0.01648293063044548,
+            0.0009034976828843355,
+            -0.0011361383367329836,
+            0.00893708411604166,
+            0.0072418018244206905,
+            0.005592587403953075,
+            0.021430574357509613,
+            0.010521803982555866,
+            0.01150764711201191,
+            0.0020384842064231634,
+            -0.022112371399998665,
+            0.014115064404904842,
+            -0.007269442547112703,
+            -0.009218094870448112,
+            -0.006182250566780567,
+            -0.0015225289389491081,
+            0.004470845218747854,
+            0.011046973057091236,
+            0.018979785963892937,
+            0.025355519726872444,
+            -0.006652138661593199,
+            -0.004754160065203905,
+            0.01228157989680767,
+            -0.0004039538907818496,
+            0.008660678751766682,
+            -0.0034435412380844355,
+            -0.002821630798280239,
+            -0.004199047107249498,
+            0.0061960709281265736,
+            -0.019127201288938522,
+            -0.005514272954314947,
+            0.008835735730826855,
+            -0.007297082804143429,
+            0.03232091665267944,
+            -0.0140966372564435,
+            -0.02417619340121746,
+            -0.00136705138720572,
+            0.002978260163217783,
+            -0.017496414482593536,
+            -0.006970004178583622,
+            0.016132818534970284,
+            0.015607649460434914,
+            0.01041124202311039,
+            -0.0027133722323924303,
+            0.024286756291985512,
+            0.005790677387267351,
+            0.025926755741238594,
+            -0.00041460696957074106,
+            -0.004364890046417713,
+            -0.022573046386241913,
+            0.017431920394301414,
+            -0.0003259271616116166,
+            0.01357146818190813,
+            0.027069229632616043,
+            0.008292139507830143,
+            0.006357307080179453,
+            0.00783146545290947,
+            0.012253939174115658,
+            -0.000859157822560519,
+            0.00014316897431854159,
+            -0.01663956046104431,
+            -0.013460906222462654,
+            -0.01568135805428028,
+            -0.002111040521413088,
+            0.010908770374953747,
+            0.011820905841886997,
+            0.004827867727726698,
+            0.0028976420871913433,
+            0.012143378145992756,
+            0.008550116792321205,
+            -0.007955847308039665,
+            -0.024802710860967636,
+            -0.00040740895201452076,
+            -0.026221588253974915,
+            0.009950567036867142,
+            0.007476745638996363,
+            -0.0007117419736459851,
+            0.0013705063611268997,
+            0.0015248323325067759,
+            0.003427417716011405,
+            -0.0005240171449258924,
+            -0.0034135973546653986,
+            -0.004279665183275938,
+            0.011102253571152687,
+            -0.006887082476168871,
+            -0.003547193016856909,
+            -0.00171946722548455,
+            0.03639327734708786,
+            -0.019422033801674843,
+            0.006960790604352951,
+            -0.011286523193120956,
+            0.009904500097036362,
+            0.005081238690763712,
+            0.014704727567732334,
+            -0.004265845287591219,
+            -0.009277982637286186,
+            -0.016022255644202232,
+            0.003584047080948949,
+            -0.016860684379935265,
+            0.014299334026873112,
+            0.0008084836299531162,
+            -0.018804728984832764,
+            0.01742270588874817,
+            -0.003095732070505619,
+            0.007720903493463993,
+            -0.0006224862881936133,
+            -0.01003348920494318,
+            0.013838659971952438,
+            0.008144724182784557,
+            0.03552721068263054,
+            -0.01714630238711834,
+            0.023715518414974213,
+            -0.015220683068037033,
+            0.020011696964502335,
+            0.020030124112963676,
+            -0.0013014052528887987,
+            -0.006504722870886326,
+            0.0013877816963940859,
+            0.013940008357167244,
+            -0.03057035431265831,
+            -0.008319780230522156,
+            0.010208545252680779,
+            -0.015091693960130215,
+            -0.011065399274230003,
+            -0.015837986022233963,
+            0.0012138771126046777,
+            0.00650011608377099,
+            -0.011028545908629894,
+            -0.0007267139153555036,
+            0.004288878757506609,
+            0.01742270588874817,
+            -0.008992364630103111,
+            0.003961800131946802,
+            -0.024194620549678802,
+            0.005085845477879047,
+            0.02283102460205555,
+            0.007011464796960354,
+            -0.006868655793368816,
+            0.019827427342534065,
+            0.012954164296388626,
+            -0.007154273800551891,
+            -0.006278992164880037,
+            -0.02273888885974884,
+            -0.015460233204066753,
+            -0.0240840595215559,
+            -0.0005948458565399051,
+            -0.01744113303720951,
+            -0.003581743687391281,
+            0.00019549870921764523,
+            0.0033721367362886667,
+            -0.0007203796412795782,
+            -0.01997484266757965,
+            0.018159786239266396,
+            0.01552472822368145,
+            0.002715675625950098,
+            -0.012557984329760075,
+            0.015626076608896255,
+            0.0234022606164217,
+            -0.01796630211174488,
+            -0.0004338977159932256,
+            0.0029874735046178102,
+            -0.017984729260206223,
+            0.01313843484967947,
+            0.004966070409864187,
+            -0.001163778710179031,
+            -0.022683609277009964,
+            -0.0006858290289528668,
+            0.009821578860282898,
+            0.015358884818851948,
+            0.004360283259302378,
+            0.005187193863093853,
+            -0.007168094161897898,
+            -0.000966840423643589,
+            -0.00818618480116129,
+            -5.551486538024619e-06,
+            -0.013553041033446789,
+            0.007025284692645073,
+            -0.008844949305057526,
+            0.022130798548460007,
+            -0.01958787627518177,
+            0.004318822640925646,
+            -0.02600046433508396,
+            -0.0011966017773374915,
+            0.003358316607773304,
+            0.009913713671267033,
+            -0.009738657623529434,
+            -0.009655735455453396,
+            -0.003991744015365839,
+            0.004171406850218773,
+            0.011599781922996044,
+            -0.009148993529379368,
+            0.010319107212126255,
+            -0.02915147691965103,
+            0.005025957711040974,
+            -0.0040078675374388695,
+            0.008762027136981487,
+            -0.023955069482326508,
+            0.009453038685023785,
+            -0.0010797056602314115,
+            -3.174334779032506e-05,
+            0.008932476863265038,
+            -0.01332270447164774,
+            -0.023770799860358238,
+            0.0043994407169520855,
+            -0.0016849166713654995,
+            -0.013589895330369473,
+            -0.03777530416846275,
+            -0.010871916078031063,
+            0.0010428517125546932,
+            0.0075043863616883755,
+            0.01170113030821085,
+            -0.028709229081869125,
+            -0.0003699791559483856,
+            -0.026332149282097816,
+            -0.006288205739110708,
+            -0.0187218077480793,
+            -0.011461579240858555,
+            0.0037452830001711845,
+            0.01931147091090679,
+            0.004984497092664242,
+            0.011922254227101803,
+            0.009959780611097813,
+            0.03009125404059887,
+            0.008444162085652351,
+            -0.0043994407169520855,
+            0.004109215922653675,
+            -0.005744609981775284,
+            -0.004947643261402845,
+            -0.013645176775753498,
+            0.003895002184435725,
+            -0.005145733244717121,
+            -0.0016238773241639137,
+            -0.003293822053819895,
+            0.01074292790144682,
+            0.011083826422691345,
+            0.00013122022210154682,
+            -0.01441910956054926,
+            0.019698437303304672,
+            -0.014225626364350319,
+            -0.010577084496617317,
+            -0.0021340742241591215,
+            -0.013820232823491096,
+            0.015110121108591557,
+            0.015469446778297424,
+            -0.026055745780467987,
+            -0.005730789620429277,
+            0.013543828390538692,
+            0.01279753539711237,
+            0.0054129245691001415,
+            0.009287196211516857,
+            0.01204202976077795,
+            -0.014068996533751488,
+            -0.0045053958892822266,
+            -0.01537731196731329,
+            -0.006527756340801716,
+            -0.0034435412380844355,
+            -0.0004523246898315847,
+            0.013838659971952438,
+            -0.02102518081665039,
+            -0.004643598105758429,
+            0.003116462379693985,
+            0.008296746760606766,
+            -0.0031878668814897537,
+            -0.0023379225749522448,
+            -0.02025124803185463,
+            0.007591914385557175,
+            -0.012640906497836113,
+            0.016906751319766045,
+            -0.0007146212155930698,
+            -0.0019267707830294967,
+            -0.01189461350440979,
+            0.013672816567122936,
+            -0.021154168993234634,
+            0.030035972595214844,
+            -0.011802478693425655,
+            0.002077641664072871,
+            -0.002655788091942668,
+            -0.001952107879333198,
+            0.00790056586265564,
+            0.03256046772003174,
+            -0.004786407109349966,
+            -0.004164496902376413,
+            0.015497087500989437,
+            -0.023549675941467285,
+            0.02541080117225647,
+            0.007974274456501007,
+            0.013433266431093216,
+            0.0006869807257317007,
+            -0.016013043001294136,
+            0.018942931666970253,
+            0.017883380874991417,
+            0.005288542248308659,
+            0.011046973057091236,
+            -0.01570899784564972,
+            0.016105176880955696,
+            0.007158880587667227,
+            -0.001377416541799903,
+            0.011903827078640461,
+            0.019532594829797745,
+            -0.02848810702562332,
+            -0.016842257231473923,
+            0.014391468837857246,
+            -0.018758662045001984,
+            -0.01995641551911831,
+            -0.012576411478221416,
+            -0.011968321166932583,
+            -0.028635522350668907,
+            0.012567197903990746,
+            0.015607649460434914,
+            0.0002802915987558663,
+            0.015229896642267704,
+            0.0030035972595214844,
+            0.017579335719347,
+            -0.01685146987438202,
+            -0.012244726531207561,
+            -0.0032085974235087633,
+            0.008301353082060814,
+            0.006771914195269346,
+            -0.015220683068037033,
+            -0.00785449892282486,
+            0.00334219285286963,
+            -0.027198217809200287,
+            0.008333600126206875,
+            -0.009664949029684067,
+            0.01910877414047718,
+            0.003918035887181759,
+            0.00503977807238698,
+            0.012742254883050919,
+            0.018620459362864494,
+            0.018500683829188347,
+            0.0050443848595023155,
+            -0.0021363776177167892,
+            0.004026294220238924,
+            -0.0035402828361839056,
+            -0.00210873712785542,
+            0.004005563911050558,
+            0.009236522018909454,
+            0.023918215185403824,
+            -0.0328737273812294,
+            -0.0026903385296463966,
+            0.0072418018244206905,
+            -0.0013992985477671027,
+            -0.0042842719703912735,
+            0.018417762592434883,
+            -0.002002781955525279,
+            0.010448095388710499,
+            0.005177980288863182,
+            0.012880456633865833,
+            -0.009231915697455406,
+            0.003026630962267518,
+            0.015589222311973572,
+            0.014769221656024456,
+            0.014548097737133503,
+            0.011802478693425655,
+            0.0009990877006202936,
+            -0.017597762867808342,
+            0.0075043863616883755,
+            0.004067755304276943,
+            -0.009245735593140125,
+            0.004666632041335106,
+            0.010899556800723076,
+            0.014824503101408482,
+            0.004139159806072712,
+            0.010964050889015198,
+            -0.01890607737004757,
+            -0.009010791778564453,
+            0.01429012045264244,
+            0.005970340687781572,
+            -0.0102730393409729,
+            -0.009775510989129543,
+            -0.004189833998680115,
+            0.0031072490382939577,
+            -0.0028354511596262455,
+            -0.009029218927025795,
+            0.005652475170791149,
+            0.002283793408423662,
+            -0.003855844959616661,
+            -0.015810346230864525,
+            0.014981132000684738,
+            0.004643598105758429,
+            0.006186857353895903,
+            0.007803824730217457,
+            0.007684049196541309,
+            -0.007536633405834436,
+            0.018546752631664276,
+            0.0038166875019669533,
+            0.004613654222339392,
+            -0.012447422370314598,
+            -0.00558798061683774,
+            -0.011572141200304031,
+            0.011747198179364204,
+            0.006647531874477863,
+            0.008964723907411098,
+            0.0024047205224633217,
+            -0.001717163948342204,
+            0.007688655983656645,
+            -0.07857263088226318,
+            -0.014713941141963005,
+            0.006583037320524454,
+            0.03526923432946205,
+            -0.0012610962148755789,
+            -0.005118092987686396,
+            0.0045146094635128975,
+            0.009406971745193005,
+            -0.004072362091392279,
+            -0.0013808716321364045,
+            0.02045394480228424,
+            -0.009305623359978199,
+            -0.013442480005323887,
+            -0.002022360684350133,
+            -0.000278132181847468,
+            -0.00224578776396811,
+            0.002814720617607236,
+            0.0028746083844453096,
+            0.028156420215964317,
+            7.521949737565592e-05,
+            0.017220009118318558,
+            -0.006675172131508589,
+            -0.004781800322234631,
+            -0.007559667341411114,
+            -0.008600790984928608,
+            -0.016261806711554527,
+            0.003406687406823039,
+            0.019845854490995407,
+            -0.011240456253290176,
+            0.017026526853442192,
+            0.0056893290020525455,
+            0.0036830920726060867,
+            -0.0019106471445411444,
+            -0.008863375522196293,
+            -0.011793265119194984,
+            -0.005090452264994383,
+            -0.009379331022500992,
+            -0.031104736030101776,
+            0.02073034830391407,
+            -0.002413933863863349,
+            0.010475736111402512,
+            -0.015358884818851948,
+            0.0033928670454770327,
+            -0.024010350927710533,
+            0.025392374023795128,
+            -0.019827427342534065,
+            -0.019182482734322548,
+            0.0067212400026619434,
+            -0.0019866584334522486,
+            0.006730453111231327,
+            -0.01839933544397354,
+            -0.0006184554076753557,
+            -0.02216765284538269,
+            -0.009245735593140125,
+            -0.0126132657751441,
+            0.012862029485404491,
+            -0.013488546945154667,
+            -0.01968001201748848,
+            0.01968001201748848,
+            -0.014594165608286858,
+            -0.015257536433637142,
+            -0.025816194713115692,
+            -0.016123604029417038,
+            0.0055603403598070145,
+            -0.008421128615736961,
+            0.0023183440789580345,
+            -0.008794275112450123,
+            0.005643261596560478,
+            -0.0036692717112600803,
+            -0.002842361107468605,
+            -0.004910789430141449,
+            -0.018482258543372154,
+            -0.0075274198316037655,
+            -0.00802494864910841,
+            -0.008550116792321205,
+            0.010687646456062794,
+            -0.005196407437324524,
+            -0.003959496505558491,
+            -0.004166800063103437,
+            0.010245399549603462,
+            0.002411630703136325,
+            -0.004613654222339392,
+            -0.009411578066647053,
+            -0.012438209727406502,
+            0.006656745448708534,
+            -0.011111467145383358,
+            -0.015699783340096474,
+            -0.018159786239266396,
+            0.015874840319156647,
+            0.016178885474801064,
+            -0.008001914247870445,
+            -0.005426744464784861,
+            0.008808095008134842,
+            0.0013532311422750354,
+            -0.029649006202816963,
+            0.0215779896825552,
+            -0.02073034830391407,
+            0.019661584869027138,
+            0.021338438615202904,
+            0.0036001706030219793,
+            -0.002823934191837907,
+            0.03268945589661598,
+            -0.008172363974153996,
+            0.0012380625121295452,
+            0.013046300038695335,
+            -0.03620900958776474,
+            -0.011608995497226715,
+            -0.0006720087840221822,
+            -0.027087656781077385,
+            -0.015598435886204243,
+            0.018095292150974274,
+            0.0006242138333618641,
+            0.004360283259302378,
+            0.003075001761317253,
+            0.03136271610856056,
+            0.0036462380085140467,
+            -0.006219104863703251,
+            0.005353036802262068,
+            0.00365314818918705,
+            0.010319107212126255,
+            -0.02120945043861866,
+            -0.02972271293401718,
+            -0.007315509952604771,
+            0.0019290741765871644,
+            -0.0026926419232040644,
+            -0.011848545633256435,
+            -0.0004975283518433571,
+            0.008338207378983498,
+            -0.022038664668798447,
+            -0.012539558112621307,
+            0.02474742941558361,
+            -0.01336877141147852,
+            0.018049223348498344,
+            -0.009784724563360214,
+            -0.009968994185328484,
+            0.006472475826740265,
+            0.023918215185403824,
+            -0.016317088156938553,
+            0.001357837812975049,
+            0.0019071920542046428,
+            0.03027552366256714,
+            0.0013267423491925001,
+            -0.017699111253023148,
+            0.013083153404295444,
+            0.012714614160358906,
+            -0.010623152367770672,
+            -0.015966976061463356,
+            -0.008213824592530727,
+            0.017127875238656998,
+            -0.015165401622653008,
+            0.006366520654410124,
+            -0.0054451716132462025,
+            0.0020269674714654684,
+            0.017044954001903534,
+            -0.03068091720342636,
+            0.005629441235214472,
+            0.010254612192511559,
+            -0.00041950165177695453,
+            0.002570563228800893,
+            0.006923936307430267,
+            -0.028045859187841415,
+            0.005403710994869471,
+            -0.035858895629644394,
+            -0.004445508122444153,
+            0.008886409923434258,
+            -0.009618882089853287,
+            0.01842697709798813,
+            -0.00916742067784071,
+            -0.020122257992625237,
+            -0.005201014224439859,
+            0.008757420815527439,
+            0.013248995877802372,
+            -0.006817981600761414,
+            0.014032143168151379,
+            0.011913040652871132,
+            0.0025912935379892588,
+            -0.003731462871655822,
+            -0.012290793471038342,
+            -0.005205621011555195,
+            -0.02227821573615074,
+            -0.014216412790119648,
+            0.006679778918623924,
+            0.006034834776073694,
+            -0.0010117562487721443,
+            0.009563600644469261,
+            -0.022609900683164597,
+            -0.00016785824846010655,
+            0.01988270878791809,
+            0.005901239346712828,
+            0.0011401692172512412,
+            -0.0016676414525136352,
+            0.028985634446144104,
+            0.008734387345612049,
+            -0.008135510608553886,
+            0.0010992843890562654,
+            0.012751467525959015,
+            0.0017206189222633839,
+            0.01447439007461071,
+            0.01424405351281166,
+            0.006444835104048252,
+            -0.009033825248479843,
+            0.0034803953021764755,
+            -0.003321462543681264,
+            0.00697461050003767,
+            0.002355197910219431,
+            -0.01538652554154396,
+            -0.013497760519385338,
+            0.012917310930788517,
+            -0.009462252259254456,
+            0.002360956510528922,
+            0.01862967386841774,
+            -0.006546183489263058,
+            -0.0028193274047225714,
+            0.0020684280898422003,
+            -0.025926755741238594,
+            -0.0016492144204676151,
+            -0.03191552311182022,
+            -0.010125624015927315,
+            -0.014511244371533394,
+            -0.026534846052527428,
+            0.03335282951593399,
+            -0.0021525011397898197,
+            0.02905934303998947,
+            -0.017524054273962975,
+            0.007739330176264048,
+            -0.015220683068037033,
+            0.024950126186013222,
+            0.005067418795078993,
+            0.00015432592772413045,
+            0.01604989729821682,
+            -0.027916869148612022,
+            -0.01566293090581894,
+            0.02550293505191803,
+            0.009609668515622616,
+            -0.01002427563071251,
+            0.036172155290842056,
+            0.02082248404622078,
+            0.0077485437504947186,
+            -0.006168430671095848,
+            0.023439114913344383,
+            0.0010319106513634324,
+            0.007260228972882032,
+            -0.010660005733370781,
+            0.013847872614860535,
+            -0.012686973437666893,
+            -0.0030151139944791794,
+            0.046767666935920715,
+            0.004477755166590214,
+            -0.0008827673736959696,
+            -0.013801805675029755,
+            0.010844276286661625,
+            -0.0060578687116503716,
+            0.015976188704371452,
+            -0.00862843170762062,
+            0.031583838164806366,
+            -0.005288542248308659,
+            -0.016980459913611412,
+            0.019753718748688698,
+            -0.01442832313477993,
+            0.04374564439058304,
+            -0.02561349794268608,
+            0.003968710079789162,
+            -0.012493490241467953,
+            -0.024544732645154,
+            0.013672816567122936,
+            -0.007707083132117987,
+            -0.003770620096474886,
+            0.008338207378983498,
+            -0.010733714327216148,
+            -0.010715287178754807,
+            0.011341804638504982,
+            -0.010116410441696644,
+            -0.006675172131508589,
+            -0.017275290563702583,
+            -0.009858432225883007,
+            -0.018528325483202934,
+            -0.002324102446436882,
+            -0.007725509814918041,
+            -0.00027582881739363074,
+            -0.00900618452578783,
+            -0.008490229956805706,
+            0.004222081042826176,
+            -0.00027122205938212574,
+            -0.004804834257811308,
+            0.0006040593143552542,
+            -0.003789047012105584,
+            0.00511348620057106,
+            -0.01486135646700859,
+            0.01639079675078392,
+            0.015423379838466644,
+            0.017477987334132195,
+            0.005680115427821875,
+            0.010577084496617317,
+            -0.006103936117142439,
+            -0.016409222036600113,
+            0.0006558852037414908,
+            0.01698967255651951,
+            0.01151686068624258,
+            0.012014389038085938,
+            0.019237764179706573,
+            -0.001894523506052792,
+            0.0007935116882435977,
+            -0.004219777882099152,
+            -0.005832138005644083,
+            -0.01223551295697689,
+            -0.009563600644469261,
+            -0.007587307598441839,
+            -0.02319956384599209,
+            0.0036969122011214495,
+            -0.0010906467214226723,
+            0.0013405625941231847,
+            0.0030427544843405485,
+            0.004763373639434576,
+            0.0072187683545053005,
+            0.001698736916296184,
+            -0.01685146987438202,
+            -0.003547193016856909,
+            -0.002957529854029417,
+            -0.019514167681336403,
+            -0.004247418139129877,
+            0.008702139370143414,
+            -0.010107196867465973,
+            -0.0016250290209427476,
+            0.012382928282022476,
+            0.0025521363131701946,
+            0.016934391111135483,
+            0.012640906497836113,
+            -0.006513936445116997,
+            -0.0006904358160682023,
+            0.009208881296217442,
+            -0.01462180633097887,
+            -0.012309220619499683,
+            -0.025355519726872444,
+            -0.017063381150364876,
+            -0.002549832919612527,
+            0.018749449402093887,
+            -0.00759652117267251,
+            -0.00880348775535822,
+            0.003574833506718278,
+            -0.009425398893654346,
+            0.00511348620057106,
+            -0.021559562534093857,
+            -0.0007624161662533879,
+            -0.020785629749298096,
+            0.005634048022329807,
+            -0.007052925415337086,
+            0.0036393278278410435,
+            0.009821578860282898,
+            0.034347884356975555,
+            0.0023678664583712816,
+            -0.013470119796693325,
+            -0.01265011914074421,
+            0.010881129652261734,
+            0.013212142512202263,
+            -0.013811019249260426,
+            -0.0020949169993400574,
+            -0.008324386551976204,
+            -0.002998990472406149,
+            0.0010509134735912085,
+            0.014492817223072052,
+            0.026203161105513573,
+            -0.01947731524705887,
+            -0.014713941141963005,
+            -0.018270347267389297,
+            -0.024323608726263046,
+            0.003708429168909788,
+            0.009637309238314629,
+            0.02082248404622078,
+            -0.00788674596697092,
+            -0.009148993529379368,
+            -0.005707756150513887,
+            -0.009047646075487137,
+            0.007545846980065107,
+            0.11542658507823944,
+            0.005772250704467297,
+            -0.001601995318196714,
+            -0.01462180633097887,
+            -0.004254328086972237,
+            0.010595511645078659,
+            -0.012760681100189686,
+            0.0664476752281189,
+            -0.013064726255834103,
+            0.005730789620429277,
+            -0.007467532530426979,
+            -0.0006184554076753557,
+            -0.004480058792978525,
+            -0.010973264463245869,
+            0.00079120829468593,
+            -0.018482258543372154,
+            0.000761840317863971,
+            -0.011295736767351627,
+            -0.0063204532489180565,
+            -0.004611351061612368,
+            -0.011885399930179119,
+            -0.016786975786089897,
+            0.01614203117787838,
+            -0.03038608469069004,
+            -0.001484523294493556,
+            0.0005410045268945396,
+            -0.020804056897759438,
+            0.003203990636393428,
+            -0.027013948187232018,
+            0.005726182833313942,
+            -0.0008816156769171357,
+            0.013811019249260426,
+            -0.005647868383675814,
+            -0.0204355176538229,
+            -0.013212142512202263,
+            0.008946297690272331,
+            0.009978207759559155,
+            0.01668562740087509,
+            0.017800459638237953,
+            0.005620228126645088,
+            -0.000780843140091747,
+            -0.011295736767351627,
+            0.010558657348155975,
+            -0.01065079215914011,
+            0.0003250633890274912,
+            -0.010890343226492405,
+            0.0018588212551549077,
+            -0.0021755348425358534,
+            0.02342068776488304,
+            -0.011811692267656326,
+            -0.021670125424861908,
+            0.001484523294493556,
+            0.02940945513546467,
+            -0.0034020806197077036,
+            0.017091020941734314,
+            7.133255712687969e-05,
+            0.009231915697455406,
+            -0.003399777226150036,
+            0.0004618261009454727,
+            0.026829678565263748,
+            -0.018454616889357567,
+            0.00623292475938797,
+            0.004625170957297087,
+            0.013672816567122936,
+            -0.008361240848898888,
+            0.01332270447164774,
+            -0.029943836852908134,
+            -0.005095059052109718,
+            -0.020195966586470604,
+            0.009089106693863869,
+            0.027566757053136826,
+            0.0051825870759785175,
+            0.012198658660054207,
+            -0.01899821311235428,
+            -0.012115737423300743,
+            0.0337950736284256,
+            -0.02006697840988636,
+            -0.010558657348155975,
+            -0.010871916078031063,
+            -0.024342035874724388,
+            0.00790977943688631,
+            0.011019332334399223,
+            -0.023457542061805725,
+            -0.015884054824709892,
+            0.004841688089072704,
+            -0.009664949029684067,
+            -0.017597762867808342,
+            0.0017160122515633702,
+            -0.015073266811668873,
+            -0.014409895986318588,
+            -0.005850565154105425,
+            -0.001894523506052792,
+            -0.011940681375563145,
+            0.00858236476778984,
+            -0.02159641683101654,
+            -0.0016446076333522797,
+            -0.0008729780092835426,
+            0.0024553947150707245,
+            0.0030634847935289145,
+            0.00623292475938797,
+            -0.0023770800326019526,
+            -0.005979553796350956,
+            -0.025171250104904175,
+            0.013064726255834103,
+            -0.0036577549763023853,
+            0.034937549382448196,
+            0.019256191328167915,
+            0.004056238103657961,
+            -0.004758766852319241,
+            -0.00841191504150629,
+            0.019993269816040993,
+            -0.02139372006058693,
+            -0.019145628437399864,
+            0.007776184007525444,
+            0.020398663356900215,
+            -0.02331012487411499,
+            0.00395028293132782,
+            -0.009024611674249172,
+            0.008665286004543304,
+            -0.010236185975372791,
+            -0.005247081629931927,
+            0.007587307598441839,
+            0.00011257731239311397,
+            0.0029045522678643465,
+            -0.006647531874477863,
+            0.01785574108362198,
+            0.013193715363740921,
+            -0.01242899615317583,
+            0.02310742810368538,
+            -0.0021064337342977524,
+            0.0019325291505083442,
+            0.0072510153986513615,
+            -0.009351690299808979,
+            -0.00769326277077198,
+            0.012521130964159966,
+            0.00484629487618804,
+            -0.02045394480228424,
+            0.018777089193463326,
+            0.007803824730217457,
+            0.02828541025519371,
+            -0.0014016019413247705,
+            -0.01662113331258297,
+            -0.0068594422191381454,
+            0.004173710476607084,
+            -0.02168855257332325,
+            -0.027953723445534706,
+            0.017072593793272972,
+            -0.012382928282022476,
+            -0.019182482734322548,
+            -0.00895090401172638,
+            0.010558657348155975,
+            -0.020472371950745583,
+            -0.0068502286449074745,
+            0.0029344961512833834,
+            0.0337950736284256,
+            -0.0038650583010166883,
+            0.002782473573461175,
+            0.011673489585518837,
+            0.004118429496884346,
+            0.001210422022268176,
+            -0.021430574357509613,
+            0.0016549729043617845,
+            0.02905934303998947,
+            0.025631925091147423,
+            0.018565179780125618,
+            -0.018841583281755447,
+            0.0005994525854475796,
+            0.009959780611097813,
+            0.009775510989129543,
+            0.006983824074268341,
+            0.008066409267485142,
+            -0.013405625708401203,
+            -0.0031187657732516527,
+            0.009411578066647053,
+            -0.011922254227101803,
+            0.014170344918966293,
+            -0.0013014052528887987,
+            -0.012180231511592865,
+            -0.004152979701757431,
+            -0.020656641572713852,
+            -0.02187282219529152,
+            -0.001980900065973401,
+            -0.005744609981775284,
+            0.0035978672094643116,
+            3.9805148844607174e-05,
+            0.005362250376492739,
+            0.01593933440744877,
+            -0.013884726911783218,
+            0.018076865002512932,
+            0.009932140819728374,
+            0.0033053390216082335,
+            -0.010917983949184418,
+            -0.0363011434674263,
+            0.01785574108362198,
+            -0.01513776183128357,
+            -0.012336861342191696,
+            0.021743832156062126,
+            -0.006679778918623924,
+            0.013783378526568413,
+            -0.021246304735541344,
+            0.0003751617332454771,
+            0.00769326277077198,
+            0.024581586942076683,
+            -0.017643829807639122,
+            0.00568472221493721,
+            0.0062283179722726345,
+            0.0030012938659638166,
+            0.00974787026643753,
+            0.008706746622920036,
+            0.0035287661012262106,
+            0.015515514649450779
+          ],
+          "title": "Appendices A Load-Balancing Loss",
+          "keyphrases": [
+            "Load-Balancing Loss",
+            "hierarchical MoE",
+            "experts",
+            "perplexity",
+            "attention mechanism"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "a211afc4-8957-43e2-9390-89dcb59a65c4",
+      "type": "child",
+      "source": {
+        "id": "1e687808-8db8-4320-805d-0282a8d679bc",
+        "properties": {
+          "page_content": "Deepseekmoe: Towards Ultimate Expert Specialization In\n\nMixture-Of-Experts Language Models\n\nDamai Dai∗1,2, Chengqi Deng1, Chenggang Zhao∗1,3, R.X. Xu1, Huazuo Gao1, Deli Chen1, Jiashi Li1, Wangding Zeng1, Xingkai Yu∗1,4, Y. Wu1, Zhenda Xie1, Y.K. Li1, Panpan Huang1, Fuli Luo1, Chong Ruan1, Zhifang Sui2, Wenfeng Liang1 1DeepSeek-AI 2National Key Laboratory for Multimedia Information Processing, Peking University 3Institute for Interdisciplinary Information Sciences, Tsinghua University 4National Key Laboratory for Novel Software Technology, Nanjing University {daidamai, szf}@pku.edu.cn, {wenfeng.liang}@deepseek.com https://github.com/deepseek-ai/DeepSeek-MoE\n\nAbstract\n\nIn the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top- out of experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into ones and activating from them, allowing for a more flexible combination of activated experts; (2) isolating experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5× expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations.\n\nFurther, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\n1. Introduction\n\nRecent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023a). It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017) has emerged as a popular solution. It can\n\nenable parameter scaling, while concurrently keeping computational costs at a modest level.\n\nRecent applications of MoE architectures in Transformers (Vaswani et al., 2017) have yielded successful attempts at scaling language models to a substantial size (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022), accompanied with remarkable performance. These achievements underscore the considerable potential and promise of MoE language models.\n\nDespite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts.\n\nThis architecture manifests two potential issues: (1) Knowledge Hybridity: existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. (2) Knowledge Redundancy: tokens assigned to different experts may require common knowledge. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters.\n\nThese issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models.\n\nIn response to the aforementioned issues, we introduce DeepSeekMoE, an innovative MoE architecture specifically designed towards ultimate expert specialization. Our architecture involves two principal strategies: (1) Fine-Grained Expert Segmentation: while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. (2) Shared Expert Isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.\n\nThrough compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized.\n\nStarting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B (Lepikhin et al., 2021) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5× expert parameters and computation. Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE.\n\nThese studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization.\n\nLeveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B (DeepSeek-AI, 2024), a dense model trained on the same 2T corpus. We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B (Touvron et al., 2023b), which has approximately 2.5 times the activated parameters. Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard1. Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. The experimental results still validate its substantial advantages over the GShard architecture consistently. In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nOur contributions are summarized as follows: - Architectural Innovation. We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation.\n\nEmpirical Validation. We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. Experimental results validate the high 1https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models\n\nScalability. We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B.\n\nAlignment for MoE. We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B.\n\nPublic Release. In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization.\n\n2. Preliminaries: Mixture-Of-Experts For Transformers\n\nWe first introduce a generic MoE architecture commonly used in Transformer language models. A standard Transformer language model is constructed by stacking layers of standard Transformer blocks, where each block can be represented as follows:\n\n$$\\mathbf{u}{1:T}^{l}=\\text{Self-Att}\\left(\\mathbf{h}{1:T}^{l-1}\\right)+\\mathbf{h}{1:T}^{l-1},\\tag{1}$$ $$\\mathbf{h}{t}^{l}=\\text{FFN}\\left(\\mathbf{u}{t}^{l}\\right)+\\mathbf{u}{t}^{l},\\tag{2}$$\n\nwhere denotes the sequence length, Self-Att(·) denotes the self-attention module, FFN(·) denotes the Feed-Forward Network (FFN), u 1: ∈ R× are the hidden states of all tokens after the -th attention module, and h\n\n∈ Ris the output hidden state of the -th token after the -th Transformer block. For brevity, we omit the layer normalization in the above formulations.\n\nA typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022). An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. Then, each token will be assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts. If the -th FFN is substituted with an MoE layer, the computation for its output hidden state h\n\nis expressed as: where denotes the total number of experts, FFN(·) is the -th expert FFN, , denotes the gate value for the -th expert, , denotes the token-to-expert affinity, Topk(·, ) denotes the set comprising highest affinity scores among those calculated for the -th token and all experts, and e\n\nis the centroid of the -th expert in the -th layer. Note that ,is sparse, indicating that only out of gate values are nonzero. This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only experts. Also, in the above formulations, we omit the layer normalization operation for brevity.\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{N}\\left(g_{i,t}\\operatorname{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\operatorname{Topk}({s_{j,t}|1\\leqslant j\\leqslant N},K),\\ 0,&\\text{otherwise},\\end{cases}$$ $$s_{i,t}=\\operatorname{Softmax}{i}\\left(\\mathbf{u}{t}^{l}\\,\\mathbf{e}_{i}^{l}\\right),$$\n\n$$(3)$$\n\n(4) $\\binom{4}{5}$ .\n\n3. Deepseekmoe Architecture\n\nOn top of the generic MoE architecture outlined in Section 2, we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. As illustrated in Figure 2, our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. Both of these strategies are designed to elevate the level of expert specialization.\n\n3.1. Fine-Grained Expert Segmentation\n\nIn scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts.\n\nIn pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. The finer expert segmentation enables a more flexible and adaptable combination of activated experts. To be specific, on top of a typical MoE architecture shown in Figure 2(a), we segment each expert FFN into smaller experts by reducing the FFN intermediate hidden dimension to 1 times its original size. Since each expert becomes smaller, in response, we also increase the number of activated experts to times to keep the same computation cost, as illustrated in Figure 2(b). With the fine-grained expert segmentation, the output of an MoE layer can be expressed as:\n\n$$\\mathbf{h}{t}^{i}=\\sum{i=1}^{\\text{mN}}\\left(g_{i,t}\\text{FFN}{i}\\left(\\mathbf{u}{t}^{i}\\right)\\right)+\\mathbf{u}{t}^{i},\\tag{6}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\text{Tokr}(s_{i,t}|1\\leqslant j\\leqslant m\\text{N}),m\\text{K}),\\ 0,&\\text{otherwise},\\end{cases}$$ (7) $$s_{i,t}=\\text{Softmax}{i}\\left(\\mathbf{u}{t}^{T}\\mathbf{e}_{t}^{i}\\right),\\tag{8}$$ where the total number of expert parameters is equal to $N$ times the number of parameters in a\n\n$$(6)$$\n\n(7) (8) $\\frac{1}{2}$\n\nstandard FFN, and denotes the total number of fine-grained experts. With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to .\n\nFrom a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. As an illustrative example, we consider the case where = 16. A typical top-2 routing strategy can yield 16 2 = 120 possible combinations. By contrast, if each expert is split into 4 smaller experts, the fine-grained routing strategy can yield 64 8 = 4, 426, 165, 368 potential combinations. The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition.\n\n3.2. Shared Expert Isolation\n\nWith a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. This alleviation of redundancy will contribute to a more parameterefficient model with more specialized experts.\n\nTowards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate experts to serve as shared experts. Regardless of the router module, each token will be deterministically assigned to these shared experts. In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by , as depicted in Figure 2(c). With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows:\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{K_{s}}\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)+\\sum_{i=K_{s}+1}^{mN}\\left(g_{i,t}\\;\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},\\tag{9}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\mathrm{Topk}{s_{j,t}|K_{s}+1\\leqslant j\\leqslant mN},mK-K_{s}},\\ 0,&\\text{otherwise},\\end{cases}$$ (10) $$s_{i,t}=\\mathrm{Softmax}{i}\\left(\\mathbf{u}{t}^{l,T}e_{i}^{l}\\right).\\tag{11}$$ $\\mathbf{h}\\mathbf{M}\\mathbf{E}$ the number of closed-compact is $K_{s}$ the total number of routed currents.\n\nFinally, in DeepSeekMoE, the number of shared expert is , the total number of routed experts is − , and the number of nonzero gates is − .\n\nIt is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022). The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint.\n\n3.3. Load Balance Consideration\n\nAutomatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. Firstly, there is a risk of routing collapse (Shazeer et al., 2017), i.e., the model always selects only a few experts, preventing other experts from sufficient training.\n\nSecondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.\n\nExpert-Level Balance Loss. In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. The computation of the balance loss is as follows:\n\n$$\\begin{array}{l}{{{\\mathcal L}{\\mathrm{ExpBal}}=\\alpha{1}\\sum_{i=1}^{N^{\\prime}}f_{i}P_{i},}}\\ {{f_{i}=\\frac{N^{\\prime}}{K^{T}}\\sum_{t=1}^{T}\\mathds{1}\\,(\\mathrm{Token}\\;t\\;\\mathrm{s}\\mathrm{lects}\\;\\mathrm{Expert}\\;i),}}\\ {{P_{i}=\\frac{1}{T}\\sum_{t=1}^{T}s_{i,t},}}\\end{array}$$\n\n(12) $$\\begin{array}{l}\\small\\text{(13)}\\end{array}$$ = (14) $$\\begin{array}{l}\\small\\text{(14)}\\end{array}$$ .\n\n7 where 1 is a hyper-parameter called expert-level balance factor, ′is equal to ( − ) and ′ is equal to ( − ) for brevity. 1(·) denotes the indicator function.\n\nDevice-Level Balance Loss. In addition to the expert-level balance loss, we introduce a devicelevel balance loss. When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. Instead, our primary objective is to ensure balanced computation across the devices. If we partition all routed experts into groups {E1, E2, ..., E}, and deploy each group on a single device, the device-level balance loss is computed as follows:\n\n$$\\mathcal{L}{\\text{DevBal}}=\\alpha{2}\\sum_{i=1}^{D}f_{i}^{\\prime}P_{i}^{\\prime},$$ $$f_{i}^{\\prime}=\\frac{1}{|\\mathcal{E}{i}|}\\sum{j\\in\\mathcal{E}{i}}f{j},$$ $$P_{i}^{\\prime}=\\sum_{j\\in\\mathcal{E}{i}}P{j},$$\n\n(15) $$\\begin{array}{l}\\small\\text{(16)}\\end{array}$$ = (17) .\n\nwhere 2 is a hyper-parameter called device-level balance factor. In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices.\n\n4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n6. Alignment For Deepseekmoe 16B\n\nPrevious research indicates that MoE models typically do not emerge significant gains from fine-tuning (Artetxe et al., 2022; Fedus et al., 2021). However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B.\n\n6.1. Experimental Setup\n\nTraining Data. For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios.\n\nHyper-Parameters. During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).\n\nWe employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10−5 without incorporating any learning rate scheduling strategy.\n\nEvaluation Benchmarks. For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3, with the following adjustments: (1) We exclude Pile (Gao et al., 2020) since chat models are seldom employed for pure language modeling. (2) We exclude\n\nMetric # Shot LLaMA2 DeepSeek DeepSeekMoE SFT 7B Chat 7B Chat 16B # Total Params N/A 6.7B 6.9B 16.4B # Activated Params N/A 6.7B 6.9B 2.8B FLOPs per 4K Tokens N/A 187.9T 183.5T 74.4T HellaSwag (Acc.) 0-shot 67.9 71.0 72.2 PIQA (Acc.) 0-shot 76.9 78.4 79.7 ARC-easy (Acc.) 0-shot 69.7 70.2 69.9 ARC-challenge (Acc.) 0-shot 50.8 50.2 50.0 BBH (EM) 3-shot 39.3 43.1 42.2 RACE-middle (Acc.) 5-shot 63.9 66.1 64.8 RACE-high (Acc.) 5-shot 49.6 50.8 50.6 DROP (EM) 1-shot 40.0 41.7 33.8 GSM8K (EM) 0-shot 63.4 62.6 62.2 MATH (EM) 4-shot 13.5 14.7 15.2 HumanEval (Pass@1) 0-shot 35.4 45.1 45.7 MBPP (Pass@1) 3-shot 27.8 39.0 46.2 TriviaQA (EM) 5-shot 60.1 59.5 63.3 NaturalQuestions (EM) 0-shot 35.2 32.7 35.1 MMLU (Acc.) 0-shot 50.0 49.7 47.2 WinoGrande (Acc.) 0-shot 65.1 68.4 69.0 CLUEWSC (EM) 5-shot 48.4 66.2 68.2 CEval (Acc.) 0-shot 35.1 44.7 40.0 CMMLU (Acc.) 0-shot 36.9 51.2 49.3\n\nCHID (Zheng et al., 2019) due to the observed instability of results, hindering the derivation of solid conclusions. (3) We additionally include BBH (Suzgun et al., 2022) to provide a more comprehensive assessment of the reasoning ability of the chat models.\n\nTable 5 | Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations.\n\n6.2. Evaluations\n\nBaselines. In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. Correspondingly, we construct three chat models, including LLaMA2 SFT 7B3, DeepSeek Chat 7B, and DeepSeekMoE Chat 16B.\n\nSubsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks.\n\nResults. The evaluation results are presented in Table 5. Our key observations include: (1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intensive tasks (TriviaQA, NaturalQuestions). (2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. In addition, it also surpasses DeepSeek Chat 7B. (3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1). However, it is worth noting that, after supervised finetuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. (4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations.\n\n7. Deepseekmoe 145B Ongoing\n\nEncouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available.\n\n7.1. Experimental Setup\n\nTraining Data and Tokenization. For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study.\n\nModel Settings. For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps.\n\nSubsequently, the learning rate keeps constant during the remaining training process. The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. Also, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse.\n\nEvaluation Benchmarks. We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3).\n\n7.2. Evaluations\n\nBaselines. Apart from DeepSeekMoE 145B, we consider three additional models for comparison. DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters.\n\nResults. From the evaluation results presented in Table 6, we have the following observations: (1) Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. (2) On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. (3) At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5.\n\n8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\nAppendices A. Overview Of Hyper-Parameters\n\nWe present the overview of hyper-parameters for DeepSeekMoE across various sizes in Table 7.\n\n# Params # Layers Hidden # Attn # Shared # Routed Relative Sequence Batch Size Learning Size Heads Experts Experts Expert Size Length (Sequence) Rate 2.0B 9 1280 10 1 63 (7 activated) 0.25 2048 2048 1.08e-3 16.4B 28 2048 16 2 64 (6 activated) 0.25 4096 4608 4.2e-4 144.6B 62 4096 32 4 128 (12 activated) 0.125 4096 4608 3.0e-4\n\nTable 7 | Overview of hyper-parameters for DeepSeekMoE across various sizes. The relative expert size is in comparison to a standard FFN.\n\nB. Comparing Deepseekmoe With Larger Models\n\nComparisons among DeepSeekMoE, GShard×1.2, and GShard×1.5 are shown in Table 8. Comparisons among DeepSeekMoE, Dense×4, and Dense×16 are shown in Table 9.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 2.3B 2.8B 1.9B # Activated Expert Params N/A 0.28B 0.35B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.824 1.808 1.808 HellaSwag (Acc.) 0-shot 53.7 54.4 54.8 PIQA (Acc.) 0-shot 71.8 71.1 72.3 ARC-easy (Acc.) 0-shot 46.8 47.3 49.4 ARC-challenge (Acc.) 0-shot 31.7 34.1 34.3 RACE-middle (Acc.) 5-shot 43.7 46.4 44.0 RACE-high (Acc.) 5-shot 31.9 32.4 31.7 HumanEval (Pass@1) 0-shot 3.7 3.0 4.9 MBPP (Pass@1) 3-shot 2.4 2.6 2.2 TriviaQA (EM) 5-shot 15.2 15.7 16.6 NaturalQuestions (EM) 5-shot 4.5 4.7 5.7\n\nTable 8 | Comparison between DeepSeekMoE and larger GShard models.\n\nAt a larger scale of 13B total parameters, we also compare DeepSeekMoE with GShard×1.2 and GShard×1.5, and show results in Table 10. At a larger scale, DeepSeekMoE even outperforms GShard×1.5 distinctly.\n\nMetric # Shot Dense×4 Dense×16 DeepSeekMoE Relative Expert Size N/A 1 1 0.25 # Experts N/A 4 + 0 16 + 0 1 + 63 # Activated Experts N/A 4 + 0 16 + 0 1 + 7 # Total Expert Params N/A 0.47B 1.89B 1.89B # Activated Expert Params N/A 0.47B 1.89B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.908 1.806 1.808 HellaSwag (Acc.) 0-shot 47.6 55.1 54.8 PIQA (Acc.) 0-shot 70.0 71.9 72.3 ARC-easy (Acc.) 0-shot 43.9 51.9 49.4 ARC-challenge (Acc.) 0-shot 30.5 33.8 34.3 RACE-middle (Acc.) 5-shot 42.4 46.3 44.0 RACE-high (Acc.) 5-shot 30.7 33.0 31.7 HumanEval (Pass@1) 0-shot 1.8 4.3 4.9 MBPP (Pass@1) 3-shot 0.2 2.2 2.2 TriviaQA (EM) 5-shot 9.9 16.5 16.6 NaturalQuestions (EM) 5-shot 3.0 6.3 5.7\n\nTable 9 | Comparison between DeepSeekMoE and larger dense baselines.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 15.9B 19.8B 13.3B # Activated Expert Params N/A 2.37B 2.82B 2.05B # Training Tokens N/A 100B 100B 100B HellaSwag (Acc.) 0-shot 66.6 67.7 69.1 PIQA (Acc.) 0-shot 75.6 76.0 75.7 ARC-easy (Acc.) 0-shot 56.8 56.8 58.8 ARC-challenge (Acc.) 0-shot 39.9 37.6 38.5 RACE-middle (Acc.) 5-shot 51.6 50.6 52.4 RACE-high (Acc.) 5-shot 37.4 36.3 38.5 HumanEval (Pass@1) 0-shot 6.1 6.1 9.8 MBPP (Pass@1) 3-shot 7.0 11.6 10.6 TriviaQA (EM) 5-shot 36.5 36.7 38.2 NaturalQuestions (EM) 5-shot 12.6 12.1 13.7\n\nTable 10 | Comparison between DeepSeekMoE and larger GShard models at a larger scale.\n\nC. Training Benchmark Curves Of Deepseekmoe 16B\n\nWe present the benchmark curves during training of DeepSeekMoE 16B and DeepSeek 7B (Dense) in Figure 7 for reference.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/DeepSeekMoE_2.md"
+          },
+          "summary": "DeepSeekMoE is an innovative Mixture-of-Experts (MoE) architecture designed to enhance expert specialization in language models. It employs fine-grained expert segmentation and shared expert isolation to improve performance and reduce computational costs. Starting with a model of 2B parameters, DeepSeekMoE demonstrates comparable performance to larger models like GShard 2.9B and LLaMA2 7B, using significantly fewer computations. The architecture scales up to 16B and 145B parameters, maintaining advantages over traditional MoE architectures. DeepSeekMoE 16B achieves similar results to dense models with only 40% of the computations, and its chat model version shows adaptability in bilingual scenarios. The architecture's public release aims to contribute to advancements in large-scale language models.",
+          "headlines": {
+            "1. Introduction": [],
+            "2. Preliminaries: Mixture-Of-Experts For Transformers": [],
+            "3. Deepseekmoe Architecture": [
+              "3.1. Fine-Grained Expert Segmentation",
+              "3.2. Shared Expert Isolation",
+              "3.3. Load Balance Consideration"
+            ],
+            "4. Validation Experiments": [
+              "4.1. Experimental Setup",
+              "4.1.1. Training Data And Tokenization",
+              "4.1.2. Infrastructures",
+              "4.1.3. Hyper-Parameters",
+              "4.1.4. Evaluation Benchmarks",
+              "4.2. Evaluations",
+              "4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models",
+              "4.4. Ablation Studies",
+              "4.5. Analysis On Expert Specialization"
+            ],
+            "5. Scaling Up To Deepseekmoe 16B": [
+              "5.1. Experimental Setup",
+              "5.1.1. Training Data And Tokenization",
+              "5.1.2. Hyper-Parameters",
+              "5.1.3. Evaluation Benchmarks",
+              "5.2. Evaluations",
+              "5.2.1. Internal Comparison With Deepseek 7B",
+              "5.2.2. Comparison With Open Source Models"
+            ],
+            "6. Alignment For Deepseekmoe 16B": [
+              "6.1. Experimental Setup",
+              "6.2. Evaluations"
+            ],
+            "7. Deepseekmoe 145B Ongoing": [
+              "7.1. Experimental Setup",
+              "7.2. Evaluations"
+            ],
+            "8. Related Work": [],
+            "9. Conclusion": [],
+            "Appendices": [
+              "A. Overview Of Hyper-Parameters",
+              "B. Comparing Deepseekmoe With Larger Models",
+              "C. Training Benchmark Curves Of Deepseekmoe 16B"
+            ]
+          },
+          "summary_embedding": [
+            -0.012465757317841053,
+            -0.022808987647294998,
+            -0.018792515620589256,
+            -0.02308654971420765,
+            0.01797616109251976,
+            0.007200260180979967,
+            0.006490030791610479,
+            0.042613785713911057,
+            -0.014049487188458443,
+            0.03187054023146629,
+            -0.021486490964889526,
+            -0.023592690005898476,
+            0.009045224636793137,
+            -0.014441338367760181,
+            0.005865518003702164,
+            -0.007363531738519669,
+            -0.008743173442780972,
+            -0.005163452122360468,
+            0.018841497600078583,
+            -0.03100520372390747,
+            0.011110605672001839,
+            -0.046075135469436646,
+            -0.020017050206661224,
+            -0.021845687180757523,
+            0.016425084322690964,
+            0.037389107048511505,
+            0.022547753527760506,
+            -0.0028062239289283752,
+            -0.06844329088926315,
+            0.034678805619478226,
+            0.035168617963790894,
+            0.02052319049835205,
+            0.0005438972148112953,
+            -0.00031302153365686536,
+            -0.024833550676703453,
+            -0.028556134551763535,
+            0.03347059711813927,
+            -0.004379750229418278,
+            -0.01453930139541626,
+            0.015976088121533394,
+            0.0414055772125721,
+            0.012106560170650482,
+            -0.02754385396838188,
+            0.0014245413476601243,
+            0.015151568688452244,
+            0.0055961208418011665,
+            -0.01678428053855896,
+            -0.0022960014175623655,
+            -0.002424577483907342,
+            -0.021225256845355034,
+            -0.01400050614029169,
+            0.018825169652700424,
+            0.001296985661610961,
+            -0.008865626528859138,
+            -0.0542386956512928,
+            0.007330877240747213,
+            0.03840138763189316,
+            0.05623060464859009,
+            0.044148534536361694,
+            -0.020180322229862213,
+            0.03369917720556259,
+            -0.030025575309991837,
+            -0.02501314878463745,
+            0.011028970591723919,
+            0.003983817528933287,
+            -0.018531281501054764,
+            0.01784554310142994,
+            -0.005636938381940126,
+            -0.0006556359585374594,
+            0.01751900091767311,
+            -0.03941367194056511,
+            0.020751770585775375,
+            -0.01887415163218975,
+            0.031266435980796814,
+            0.02173139713704586,
+            0.0073798587545752525,
+            0.01872720755636692,
+            0.01678428053855896,
+            0.0036858476232737303,
+            -0.04107903689146042,
+            0.009118696674704552,
+            0.016523046419024467,
+            0.014196431264281273,
+            0.016269976273179054,
+            0.06553706526756287,
+            -0.05391215533018112,
+            -0.004485876299440861,
+            -0.05169166624546051,
+            0.0012561678886413574,
+            -0.004971608519554138,
+            -0.03670336678624153,
+            0.01048201136291027,
+            0.028425518423318863,
+            0.021372200921177864,
+            -0.01063711941242218,
+            0.024392718449234962,
+            -0.022743679583072662,
+            0.0032001156359910965,
+            -0.04062187671661377,
+            0.0019653772469609976,
+            0.004518530797213316,
+            0.016531210392713547,
+            0.0070778070949018,
+            0.004273623693734407,
+            -0.016702644526958466,
+            0.006045116577297449,
+            -0.01905374974012375,
+            0.03291547670960426,
+            -0.008465612307190895,
+            0.08653374016284943,
+            0.008751336485147476,
+            0.004738946910947561,
+            0.004281787667423487,
+            0.02398454025387764,
+            0.012131051160395145,
+            -0.00810233410447836,
+            -0.04718538001179695,
+            -0.015690363943576813,
+            0.011061624623835087,
+            -0.0383034273982048,
+            0.018335357308387756,
+            -0.008759500458836555,
+            0.01282495353370905,
+            -0.04107903689146042,
+            -0.06344719231128693,
+            0.01498013362288475,
+            0.001784758409485221,
+            -0.004065453074872494,
+            -0.0002903166168835014,
+            -0.018923133611679077,
+            0.003930754493921995,
+            -0.004200151655822992,
+            -0.010645282454788685,
+            -0.021502818912267685,
+            0.03159298002719879,
+            0.03657275065779686,
+            0.06527583301067352,
+            0.036801330745220184,
+            -0.0030164355412125587,
+            0.005775718949735165,
+            0.019690508022904396,
+            -0.020784424617886543,
+            0.003481758525595069,
+            -0.01214737817645073,
+            -0.021992631256580353,
+            -0.03820546343922615,
+            0.010971825569868088,
+            0.07379858940839767,
+            -0.029029620811343193,
+            0.013437220826745033,
+            -0.017616963014006615,
+            0.04205866530537605,
+            -0.004567512311041355,
+            0.026727497577667236,
+            -0.016669990494847298,
+            0.004059330560266972,
+            -0.025388672947883606,
+            0.01620466820895672,
+            -0.024964168667793274,
+            0.012571883387863636,
+            -0.000885746325366199,
+            -0.04222193360328674,
+            0.00011116473615402356,
+            0.026890767738223076,
+            -0.010971825569868088,
+            0.02261306345462799,
+            0.0004949158756062388,
+            -0.006302268709987402,
+            -0.05097327008843422,
+            0.038727931678295135,
+            -0.003912386484444141,
+            0.018645571544766426,
+            -0.0018959869630634785,
+            0.0163107942789793,
+            -0.0016408757073804736,
+            0.013216804713010788,
+            0.009543201886117458,
+            0.007114543113857508,
+            -0.033503253012895584,
+            0.02697240374982357,
+            -0.02504580281674862,
+            0.028049994260072708,
+            0.03435226157307625,
+            -0.0030184765346348286,
+            0.03255628049373627,
+            0.019249675795435905,
+            0.03840138763189316,
+            0.0327032245695591,
+            0.030335791409015656,
+            -0.00824927818030119,
+            0.013878053054213524,
+            0.008200296200811863,
+            0.03608293831348419,
+            0.013429056853055954,
+            0.019723162055015564,
+            -0.009730963967740536,
+            -0.014041324146091938,
+            0.048785436898469925,
+            0.007318632211536169,
+            -0.0179598331451416,
+            -0.005441013257950544,
+            -0.03510330989956856,
+            -0.002832755446434021,
+            -0.014425011351704597,
+            0.0323440283536911,
+            0.02158445306122303,
+            -0.01647406443953514,
+            0.035135965794324875,
+            0.024980494752526283,
+            -0.03751972317695618,
+            -0.005232842173427343,
+            -0.02026195637881756,
+            -0.023298801854252815,
+            -0.04215662553906441,
+            0.011624909937381744,
+            0.012833117507398129,
+            -0.013976015150547028,
+            -0.007469657808542252,
+            0.020457882434129715,
+            -0.023625344038009644,
+            -0.0018265966791659594,
+            -0.01356783788651228,
+            0.006547175347805023,
+            -0.020898714661598206,
+            -0.03667071461677551,
+            -0.01048201136291027,
+            0.025862159207463264,
+            0.012637191452085972,
+            0.04362606629729271,
+            -0.013518855907022953,
+            -0.005367541220039129,
+            0.033193036913871765,
+            0.012759645469486713,
+            0.010767736472189426,
+            0.03699725493788719,
+            -0.03027048148214817,
+            0.024621298536658287,
+            -0.020294612273573875,
+            -0.0032878739293664694,
+            0.01707816869020462,
+            -0.0030062312725931406,
+            0.01962519995868206,
+            -0.04535674303770065,
+            -0.024196794256567955,
+            0.04395261034369469,
+            -0.0007107399869710207,
+            0.016302630305290222,
+            -0.005959399044513702,
+            -0.012106560170650482,
+            0.0035899258218705654,
+            0.0027715289033949375,
+            -0.01438419334590435,
+            -0.008620720356702805,
+            0.007473739795386791,
+            -0.01962519995868206,
+            0.03660540655255318,
+            0.03794422745704651,
+            -0.004132802598178387,
+            -0.014131123200058937,
+            -0.03324201703071594,
+            0.010016689077019691,
+            -0.017878197133541107,
+            -0.023951886221766472,
+            0.04078514873981476,
+            -0.032115448266267776,
+            -0.00029312283731997013,
+            0.039478980004787445,
+            0.018808843567967415,
+            -0.014898497611284256,
+            0.00868602842092514,
+            -0.01709449477493763,
+            -0.026302991434931755,
+            -0.017159804701805115,
+            0.03356856107711792,
+            0.024800896644592285,
+            -0.010743245482444763,
+            0.010245268233120441,
+            0.005330805201083422,
+            0.02893165871500969,
+            0.024376392364501953,
+            0.00648594880476594,
+            -0.013812744058668613,
+            0.015029114671051502,
+            0.027641816064715385,
+            -0.0028021421749144793,
+            -7.66609373386018e-05,
+            -0.004010349046438932,
+            -0.007796200457960367,
+            -0.006453294772654772,
+            -0.013069860637187958,
+            -0.005089980084449053,
+            0.0009184005321003497,
+            -0.032295044511556625,
+            0.015967924147844315,
+            -0.005890008993446827,
+            0.010359558276832104,
+            -0.011535110883414745,
+            0.01288209855556488,
+            0.01722511276602745,
+            -0.013494365848600864,
+            -0.0716434046626091,
+            0.014612773433327675,
+            0.02458864450454712,
+            -0.006706364918500185,
+            -0.017943505197763443,
+            0.008481939323246479,
+            0.00016148542636074126,
+            0.038597315549850464,
+            -0.014180104248225689,
+            -0.031103165820240974,
+            0.02680913358926773,
+            -0.017437364906072617,
+            -0.015241367742419243,
+            -0.010661610402166843,
+            -0.00580837344750762,
+            0.04649963974952698,
+            0.01889047957956791,
+            0.011420820839703083,
+            -0.04042595252394676,
+            -0.007449248805642128,
+            0.015078096650540829,
+            0.014506647363305092,
+            -0.04705476388335228,
+            0.0052083516493439674,
+            0.015486273914575577,
+            0.019723162055015564,
+            0.024800896644592285,
+            0.04274440184235573,
+            -0.0021531390957534313,
+            0.01661284640431404,
+            -0.015053605660796165,
+            -0.05136512219905853,
+            -0.03340528905391693,
+            0.029943939298391342,
+            -0.017731253057718277,
+            -0.02906227484345436,
+            -0.01693122461438179,
+            -0.006151242647320032,
+            0.0017939424142241478,
+            0.007126788143068552,
+            -0.01916803978383541,
+            -0.03464614972472191,
+            -0.03970756009221077,
+            0.02277633361518383,
+            0.007983962073922157,
+            0.004269542172551155,
+            -0.042777057737112045,
+            0.0013459670590236783,
+            0.009184005670249462,
+            0.012073906138539314,
+            0.0009954441338777542,
+            -0.019984396174550056,
+            0.015494437888264656,
+            0.0032001156359910965,
+            -0.031070511788129807,
+            -0.035593122243881226,
+            -0.03595232218503952,
+            0.0023919232189655304,
+            0.003781769424676895,
+            0.06279410421848297,
+            0.012988224625587463,
+            -0.03840138763189316,
+            -0.04522612690925598,
+            0.029176564887166023,
+            -0.02052319049835205,
+            -0.016678154468536377,
+            0.0059430720284581184,
+            0.012898425571620464,
+            -0.009877908043563366,
+            0.013192313723266125,
+            0.023560035973787308,
+            0.041601505130529404,
+            -0.005987971555441618,
+            0.012914752587676048,
+            0.023037567734718323,
+            0.012955570593476295,
+            0.0362788625061512,
+            -0.00936360377818346,
+            0.0035409443080425262,
+            0.039936136454343796,
+            -0.012269831262528896,
+            0.03175625205039978,
+            -0.02189466916024685,
+            0.0020398697815835476,
+            0.02249877341091633,
+            0.027184655889868736,
+            -0.0006479826406575739,
+            0.02594379521906376,
+            -0.031543996185064316,
+            0.018368011340498924,
+            -0.014498483389616013,
+            -0.04888340085744858,
+            -0.01557607389986515,
+            0.013102514669299126,
+            0.030384771525859833,
+            0.02756018005311489,
+            -0.0041348435916006565,
+            -0.019723162055015564,
+            -0.01155143789947033,
+            -0.010596301406621933,
+            0.0018419033149257302,
+            0.03285016864538193,
+            0.010506502352654934,
+            0.010890189558267593,
+            -0.011935125105082989,
+            -0.004104230087250471,
+            0.03025415539741516,
+            -0.021192602813243866,
+            0.01650671847164631,
+            0.0041511706076562405,
+            -0.002029665280133486,
+            -0.04320156201720238,
+            -0.035266581922769547,
+            0.007845181971788406,
+            0.011747363954782486,
+            0.014874007552862167,
+            0.0368339866399765,
+            0.025094784796237946,
+            -0.01662100851535797,
+            0.030629679560661316,
+            -0.01905374974012375,
+            0.027984686195850372,
+            0.039054471999406815,
+            0.011624909937381744,
+            -0.018351683393120766,
+            0.012237177230417728,
+            0.002424577483907342,
+            -0.025127438828349113,
+            0.01680060662329197,
+            -0.032735876739025116,
+            0.01738838478922844,
+            -0.02354370802640915,
+            0.025143766775727272,
+            -0.02579685114324093,
+            -0.009290131740272045,
+            -0.004428731743246317,
+            -0.006130834110081196,
+            0.04888340085744858,
+            -0.026727497577667236,
+            -0.0142454132437706,
+            -0.0024694770108908415,
+            -0.0024531499948352575,
+            -0.008718682453036308,
+            0.010000362060964108,
+            -0.045781247317790985,
+            -0.01844964735209942,
+            -0.04016471654176712,
+            0.007355368230491877,
+            0.005412440747022629,
+            -0.030907239764928818,
+            -0.05694899708032608,
+            0.025421326979994774,
+            -0.06331657618284225,
+            -0.019886434078216553,
+            0.051626358181238174,
+            -0.018400665372610092,
+            -0.008212542161345482,
+            0.009641164913773537,
+            0.006375740747898817,
+            -0.029290854930877686,
+            0.0041348435916006565,
+            0.02504580281674862,
+            0.04418119043111801,
+            -0.0045797573402523994,
+            -0.02865409664809704,
+            0.0028674507047981024,
+            -0.027788760140538216,
+            -0.013461710885167122,
+            0.005261414684355259,
+            0.016449574381113052,
+            0.024539662525057793,
+            0.007628847379237413,
+            0.03585435822606087,
+            0.007294141221791506,
+            0.006318595726042986,
+            0.044279150664806366,
+            -0.025666235014796257,
+            -0.027380581945180893,
+            0.02519274689257145,
+            0.04045860469341278,
+            0.023151857778429985,
+            -0.018253721296787262,
+            -0.0142454132437706,
+            0.0053063142113387585,
+            0.0009296254720538855,
+            -0.022139575332403183,
+            0.002808264922350645,
+            -0.013902543112635612,
+            0.005143043119460344,
+            0.03892385587096214,
+            -0.001205145614221692,
+            0.014196431264281273,
+            0.007302304729819298,
+            0.006845145486295223,
+            0.004334850702434778,
+            -0.011731036007404327,
+            -0.023592690005898476,
+            0.00032220553839579225,
+            0.034972693771123886,
+            0.015404638834297657,
+            -0.030515389516949654,
+            -0.00995954405516386,
+            0.034711457788944244,
+            -0.00795130804181099,
+            -0.009118696674704552,
+            -0.009771781973540783,
+            0.003385836724191904,
+            0.02638462744653225,
+            -0.01606588624417782,
+            0.006649219896644354,
+            0.03716052696108818,
+            -0.019870106130838394,
+            0.005543057806789875,
+            -0.025405000895261765,
+            0.0018061877926811576,
+            0.02442537248134613,
+            -0.006216551177203655,
+            0.028098974376916885,
+            5.6315813708351925e-05,
+            0.03025415539741516,
+            -0.004677720367908478,
+            0.008432958275079727,
+            -0.020327266305685043,
+            0.03987082839012146,
+            -0.017633290961384773,
+            0.03402572125196457,
+            0.02067013457417488,
+            -0.01258004643023014,
+            0.04783846437931061,
+            -0.036801330745220184,
+            -0.020784424617886543,
+            0.00584919098764658,
+            0.012743317522108555,
+            0.0028919412288814783,
+            -0.0191517136991024,
+            -0.018400665372610092,
+            -0.010996315628290176,
+            0.02380494214594364,
+            -0.022547753527760506,
+            -0.016865916550159454,
+            -0.007441085297614336,
+            -0.01890680566430092,
+            -0.009673818945884705,
+            -0.007988044060766697,
+            -0.01709449477493763,
+            -0.016547536477446556,
+            0.044573038816452026,
+            -0.06334923207759857,
+            0.016996532678604126,
+            0.017159804701805115,
+            0.011477965861558914,
+            -0.034678805619478226,
+            -0.019510909914970398,
+            -0.017633290961384773,
+            -0.002347023691982031,
+            -0.030743969604372978,
+            0.001702102366834879,
+            0.01725776679813862,
+            0.01138000376522541,
+            -0.05022222548723221,
+            0.012335140258073807,
+            0.02772345207631588,
+            -0.012065742164850235,
+            0.0014949520118534565,
+            -0.023135529831051826,
+            -0.017176130786538124,
+            0.0031776658724993467,
+            0.022123249247670174,
+            0.005975726060569286,
+            -0.008188051171600819,
+            -0.016865916550159454,
+            -0.005241005681455135,
+            -0.006045116577297449,
+            -0.002359268954023719,
+            0.005094061605632305,
+            0.011780017986893654,
+            0.019690508022904396,
+            -0.0024633544962853193,
+            -0.004436895251274109,
+            -0.006396149750798941,
+            0.027364253997802734,
+            -0.001450052484869957,
+            -0.008416631259024143,
+            0.050907962024211884,
+            0.03556047007441521,
+            0.006094098091125488,
+            -0.01108611561357975,
+            -0.03990348428487778,
+            0.010661610402166843,
+            -0.022711025550961494,
+            0.0401320643723011,
+            0.01905374974012375,
+            -0.008298259228467941,
+            -0.028866350650787354,
+            0.010294250212609768,
+            -0.02290695160627365,
+            -0.009975871071219444,
+            0.009469729848206043,
+            0.00726556871086359,
+            -0.03263791650533676,
+            -0.013543346896767616,
+            0.03510330989956856,
+            0.011437147855758667,
+            -0.006841063499450684,
+            -0.036050282418727875,
+            0.008588065393269062,
+            0.01646590232849121,
+            0.011869817040860653,
+            -0.019233347848057747,
+            0.004449140280485153,
+            -0.012041252106428146,
+            0.017470018938183784,
+            0.008849299512803555,
+            -0.017796561121940613,
+            0.018237393349409103,
+            0.004710374400019646,
+            -0.008490103296935558,
+            -0.003912386484444141,
+            -0.0163107942789793,
+            -0.02006603218615055,
+            0.03840138763189316,
+            0.024490682408213615,
+            0.026335645467042923,
+            0.005457340274006128,
+            0.02008235827088356,
+            0.03227871656417847,
+            0.011282040737569332,
+            -0.03134807199239731,
+            0.027935704216361046,
+            0.016996532678604126,
+            -0.04604247957468033,
+            0.021551799029111862,
+            -0.02218855731189251,
+            -0.028670424595475197,
+            0.009649328887462616,
+            -0.0017367975087836385,
+            0.011192241683602333,
+            0.0034837995190173388,
+            -0.003642988856881857,
+            -0.03307874873280525,
+            -0.006236960180103779,
+            0.015771998092532158,
+            0.015502601861953735,
+            -0.0048695639707148075,
+            -0.003296037670224905,
+            -0.011886144056916237,
+            0.00487364549189806,
+            0.009796272963285446,
+            0.039054471999406815,
+            0.001381682581268251,
+            -0.004751192405819893,
+            -0.0018765984568744898,
+            0.031854212284088135,
+            0.0271193478256464,
+            -0.017274094745516777,
+            -0.026286665350198746,
+            0.00580837344750762,
+            -0.014253576286137104,
+            0.026156047359108925,
+            -0.00487364549189806,
+            0.011877980083227158,
+            0.008718682453036308,
+            -0.0007908449042588472,
+            -0.001373519073240459,
+            0.025584599003195763,
+            -0.02710302174091339,
+            -0.021764053031802177,
+            0.031233781948685646,
+            -0.025731543079018593,
+            0.028735732659697533,
+            0.013061696663498878,
+            -0.01557607389986515,
+            -0.015225040726363659,
+            -0.010302413254976273,
+            0.009943217039108276,
+            -0.012277995236217976,
+            0.023853924125432968,
+            0.003608293831348419,
+            0.006392067763954401,
+            0.0003584313380997628,
+            0.02770712412893772,
+            0.024833550676703453,
+            0.007820690982043743,
+            -0.018074123188853264,
+            -0.0016816934803500772,
+            -0.02320083975791931,
+            -0.01680060662329197,
+            0.023772288113832474,
+            -0.018090449273586273,
+            -0.03268689662218094,
+            -0.028556134551763535,
+            0.012114724144339561,
+            0.007392104249447584,
+            0.010816717520356178,
+            0.03461349755525589,
+            -0.030384771525859833,
+            -0.0344502255320549,
+            0.01766594499349594,
+            0.026270337402820587,
+            -0.005551221314817667,
+            -0.021535472944378853,
+            0.03085825778543949,
+            -0.005828781984746456,
+            0.017600636929273605,
+            0.0404912605881691,
+            -0.0016225076979026198,
+            0.010228941217064857,
+            -0.042026009410619736,
+            -0.014057651162147522,
+            -0.0066614653915166855,
+            0.006261450704187155,
+            0.034417569637298584,
+            0.01048201136291027,
+            0.01535565685480833,
+            0.03448288142681122,
+            -0.021192602813243866,
+            -0.0002903166168835014,
+            -0.0022449791431427,
+            -0.02818061038851738,
+            0.03347059711813927,
+            0.025715215131640434,
+            0.011918798089027405,
+            0.019706835970282555,
+            -0.03794422745704651,
+            -0.013037205673754215,
+            0.027070365846157074,
+            0.015053605660796165,
+            0.01273515447974205,
+            0.016694480553269386,
+            0.011249386705458164,
+            0.00868602842092514,
+            -0.021633435040712357,
+            -0.05169166624546051,
+            0.01214737817645073,
+            0.01993541419506073,
+            -0.016449574381113052,
+            0.01057181041687727,
+            0.024980494752526283,
+            0.007620683871209621,
+            -0.007045152597129345,
+            0.012808626517653465,
+            -0.02682545967400074,
+            0.002259265398606658,
+            0.0019010890973731875,
+            -0.014482156373560429,
+            0.038270771503448486,
+            -0.016367938369512558,
+            0.011363675817847252,
+            -0.018204739317297935,
+            0.023870250210165977,
+            0.00011531029304023832,
+            0.0007964573451317847,
+            0.014963806606829166,
+            -0.011347348801791668,
+            -0.04235255345702171,
+            0.0021613026037812233,
+            0.03342161700129509,
+            -0.015200549736618996,
+            0.026449935510754585,
+            -0.0020674217958003283,
+            0.032458316534757614,
+            0.042777057737112045,
+            0.017731253057718277,
+            0.0003497575526125729,
+            0.025682561099529266,
+            0.0036144163459539413,
+            0.0073880222626030445,
+            0.012400448322296143,
+            0.009322785772383213,
+            0.007245160173624754,
+            0.030809277668595314,
+            -0.010261595249176025,
+            0.02173139713704586,
+            -0.04760988429188728,
+            -0.014629100449383259,
+            0.012024925090372562,
+            -0.0013316808035597205,
+            -0.012449429370462894,
+            -0.033029764890670776,
+            0.012563719414174557,
+            -0.03503800183534622,
+            0.022074267268180847,
+            -0.0004959363141097128,
+            0.04333217814564705,
+            -0.02187834307551384,
+            -0.005518566817045212,
+            0.016694480553269386,
+            0.0016510800924152136,
+            0.023282473906874657,
+            -0.0012153501156717539,
+            -0.016416920349001884,
+            -0.0067798369564116,
+            -0.009257477708160877,
+            0.004138925112783909,
+            0.04375668615102768,
+            -0.003722583642229438,
+            -0.0001682458823779598,
+            0.01409030519425869,
+            -0.011959616094827652,
+            -0.03250729665160179,
+            0.045193471014499664,
+            -0.0030837850645184517,
+            0.02484987862408161,
+            0.04979771748185158,
+            -0.06410028040409088,
+            -0.019543563947081566,
+            0.013061696663498878,
+            -0.010441194288432598,
+            0.011363675817847252,
+            0.005547139327973127,
+            0.016114868223667145,
+            0.017894525080919266,
+            -0.005485912784934044,
+            0.01619650423526764,
+            0.005106307100504637,
+            0.028098974376916885,
+            -0.02336410991847515,
+            -0.03296445682644844,
+            -0.02636830136179924,
+            -0.017747581005096436,
+            0.0014480114914476871,
+            -0.015796490013599396,
+            -3.563585551091819e-06,
+            -0.0022204883862286806,
+            -0.04388730227947235,
+            -0.025274382904171944,
+            0.040687184780836105,
+            0.006706364918500185,
+            0.02458864450454712,
+            0.0077268099412322044,
+            0.03418898954987526,
+            0.005885927006602287,
+            -0.009037061594426632,
+            0.030058229342103004,
+            -0.020049704238772392,
+            0.03833607956767082,
+            0.022106921300292015,
+            -0.002832755446434021,
+            0.005865518003702164,
+            0.031086837872862816,
+            -0.033797141164541245,
+            0.008086007088422775,
+            -0.034384917467832565,
+            0.0046695563942193985,
+            -0.008620720356702805,
+            -0.015094423666596413,
+            -0.009526874870061874,
+            -0.02504580281674862,
+            0.011216731742024422,
+            -0.021535472944378853,
+            0.01590261608362198,
+            0.010416703298687935,
+            -0.002916431985795498,
+            0.03448288142681122,
+            -0.014204595237970352,
+            -0.0027796924114227295,
+            0.0024531499948352575,
+            -0.027070365846157074,
+            -0.006343086715787649,
+            -0.003377673216164112,
+            -0.03689929470419884,
+            -0.004502203781157732,
+            0.010988152585923672,
+            -0.00024184548237826675,
+            0.016898570582270622,
+            0.02787039615213871,
+            0.0005092020728625357,
+            -0.022302847355604172,
+            0.002340900944545865,
+            -0.04199335351586342,
+            0.0011877981014549732,
+            0.01483318954706192,
+            -0.017600636929273605,
+            -0.008612556383013725,
+            0.033046092838048935,
+            0.023298801854252815,
+            -0.009396257810294628,
+            -0.035005345940589905,
+            0.017796561121940613,
+            -0.011853490024805069,
+            0.027511198073625565,
+            -0.0002630197268445045,
+            0.019298657774925232,
+            -0.0076941559091210365,
+            0.04493223875761032,
+            0.001369437319226563,
+            0.008645210415124893,
+            -0.021959977224469185,
+            0.007020662073045969,
+            -0.020278284326195717,
+            -0.002406209474429488,
+            -0.006171651650220156,
+            -0.012751481495797634,
+            -0.010669773444533348,
+            0.0032735876739025116,
+            0.007730891928076744,
+            -0.0176822729408741,
+            0.0055961208418011665,
+            -0.015959760174155235,
+            -0.02741323597729206,
+            0.007543129846453667,
+            -0.05965929850935936,
+            -0.0179598331451416,
+            -0.03373183310031891,
+            -0.009477893821895123,
+            0.02008235827088356,
+            0.03510330989956856,
+            -0.012759645469486713,
+            0.00497977202758193,
+            -0.014898497611284256,
+            0.004477712791413069,
+            0.03490738570690155,
+            0.044442422688007355,
+            -0.014759717509150505,
+            0.003381754970178008,
+            -0.010955498553812504,
+            0.033029764890670776,
+            -0.03347059711813927,
+            0.0030062312725931406,
+            0.031086837872862816,
+            -0.005057325586676598,
+            0.007959471084177494,
+            0.011624909937381744,
+            0.020474210381507874,
+            -0.019396619871258736,
+            0.00877582747489214,
+            0.002100075827911496,
+            0.019233347848057747,
+            0.02171507105231285,
+            0.020343592390418053,
+            -0.03539719805121422,
+            0.028409190475940704,
+            0.018384337425231934,
+            0.02444170042872429,
+            0.044442422688007355,
+            -0.03974021226167679,
+            -0.004147088620811701,
+            -0.0032225653994828463,
+            -0.02997659333050251,
+            0.02008235827088356,
+            -0.007530884351581335,
+            -0.01841699331998825,
+            -0.007069643586874008,
+            0.03327467292547226,
+            0.014571955427527428,
+            -0.05051611363887787,
+            0.01980479806661606,
+            0.01740471087396145,
+            0.02055584453046322,
+            0.024474354460835457,
+            -0.01123305968940258,
+            -0.0019072118448093534,
+            0.008996243588626385,
+            0.006804327480494976,
+            0.019266001880168915,
+            0.015257694758474827,
+            0.002012317767366767,
+            -0.0022082431241869926,
+            -0.011837163008749485,
+            0.005224678665399551,
+            -0.009077879600226879,
+            0.010808554477989674,
+            -0.02458864450454712,
+            -0.009698309935629368,
+            0.021388528868556023,
+            0.0031164393294602633,
+            -0.013665799982845783,
+            -0.03729114308953285,
+            -0.045944519340991974,
+            0.0033837957307696342,
+            0.004543021321296692,
+            -0.05365091934800148,
+            0.005322641227394342,
+            -0.026547899469733238,
+            -0.002108239568769932,
+            -0.005502239800989628,
+            0.0019214981002733111,
+            0.005636938381940126,
+            0.0167679525911808,
+            -0.013494365848600864,
+            0.02248244546353817,
+            -0.015502601861953735,
+            0.03165828809142113,
+            0.0055961208418011665,
+            0.007375776767730713,
+            -0.0027796924114227295,
+            0.05590406060218811,
+            -0.003712379140779376,
+            -0.03699725493788719,
+            0.023282473906874657,
+            -0.01311067771166563,
+            -0.0001747512142173946,
+            0.030090883374214172,
+            0.00884113647043705,
+            -0.015257694758474827,
+            0.013437220826745033,
+            -0.030613351613283157,
+            -0.03249097242951393,
+            0.010996315628290176,
+            -0.013755599036812782,
+            -0.0015776080545037985,
+            -0.04568328335881233,
+            -0.009339112788438797,
+            0.004269542172551155,
+            -0.018335357308387756,
+            0.010775899514555931,
+            0.01631895825266838,
+            0.013992343097925186,
+            -0.024245774373412132,
+            0.009053388610482216,
+            0.006865554489195347,
+            0.013453547842800617,
+            0.0007331897504627705,
+            0.014049487188458443,
+            -0.008677864447236061,
+            -0.011306531727313995,
+            -0.018319029361009598,
+            0.0032450153958052397,
+            -0.026792805641889572,
+            0.005155288614332676,
+            -0.003155216109007597,
+            -0.03771565109491348,
+            -0.06521052122116089,
+            -0.0344502255320549,
+            0.016294466331601143,
+            0.03396041318774223,
+            -0.009045224636793137,
+            0.0035144127905368805,
+            -0.00023648813657928258,
+            -0.01093917153775692,
+            0.024523336440324783,
+            -0.005930826533585787,
+            -0.0004829256213270128,
+            0.0035389035474509,
+            0.02050686441361904,
+            -0.009722800925374031,
+            0.005845109466463327,
+            -0.051952898502349854,
+            -0.01586179807782173,
+            0.01812310516834259,
+            0.006306350696831942,
+            0.014629100449383259,
+            -0.009690145961940289,
+            0.01693122461438179,
+            -0.030074557289481163,
+            -0.028147956356406212,
+            0.04130761697888374,
+            0.008277850225567818,
+            0.011486129835247993,
+            -0.003055212553590536,
+            -0.01680060662329197,
+            0.02504580281674862,
+            -0.0020633398089557886,
+            -0.006461458280682564,
+            -0.00817988719791174,
+            0.015820980072021484,
+            0.044279150664806366,
+            0.003255219664424658,
+            0.0009821783751249313,
+            0.017045514658093452,
+            -0.002663361607119441,
+            -0.011575928889214993,
+            0.007677828427404165,
+            0.013592327944934368,
+            -0.016384266316890717,
+            0.010375885292887688,
+            -0.025862159207463264,
+            0.0332583449780941,
+            0.03131541982293129,
+            0.011575928889214993,
+            -0.015045441687107086,
+            0.006130834110081196,
+            0.005330805201083422,
+            0.008086007088422775,
+            -0.001369437319226563,
+            -0.0017133272485807538,
+            0.020719116553664207,
+            -0.03276853263378143,
+            -0.01693122461438179,
+            -0.005347132217139006,
+            0.003932795487344265,
+            0.005277741700410843,
+            0.03396041318774223,
+            -0.0033266509417444468,
+            0.017927179113030434,
+            -0.011657563969492912,
+            0.010579974390566349,
+            0.021764053031802177,
+            0.037095218896865845,
+            -0.024996822699904442,
+            -0.029862305149435997,
+            0.012914752587676048,
+            -0.001296985661610961,
+            0.014065815135836601,
+            0.0010173837654292583,
+            0.014139287173748016,
+            0.0029409227427095175,
+            0.060475654900074005,
+            -0.017763907089829445,
+            0.004628738854080439,
+            0.0004035860183648765,
+            -0.01858026348054409,
+            -0.0013531101867556572,
+            0.004261378664523363,
+            -0.010979988612234592,
+            0.03608293831348419,
+            0.002344982698559761,
+            0.039185091853141785,
+            0.013918871060013771,
+            0.009290131740272045,
+            0.0015143405180424452,
+            -0.013592327944934368,
+            0.008355404250323772,
+            0.0015592400450259447,
+            -0.001132694073021412,
+            0.009461566805839539,
+            0.019298657774925232,
+            0.009714636951684952,
+            0.025274382904171944,
+            0.009788108989596367,
+            0.012857607565820217,
+            -0.029470453038811684,
+            0.03510330989956856,
+            -0.0006224715034477413,
+            0.012996388599276543,
+            -0.02040890045464039,
+            -0.003136848099529743,
+            -0.0008240094175562263,
+            0.015649545937776566,
+            -0.015616890974342823,
+            -0.001201063860207796,
+            -0.01297189760953188,
+            0.040099408477544785,
+            -0.018988441675901413,
+            0.013976015150547028,
+            0.009673818945884705,
+            -0.02865409664809704,
+            -0.007596192881464958,
+            0.01995174214243889,
+            0.014294394291937351,
+            -0.0053144777193665504,
+            -0.020490536466240883,
+            0.005832863971590996,
+            0.015380147844552994,
+            -0.028572462499141693,
+            1.3584674888988957e-05,
+            0.014751553535461426,
+            -0.0164903923869133,
+            -0.00025409081717953086,
+            -0.004702210891991854,
+            0.003975654020905495,
+            -0.009265640750527382,
+            0.007028825581073761,
+            0.016057724133133888,
+            -0.024507008492946625,
+            -0.020049704238772392,
+            0.005722655914723873,
+            0.006571666337549686,
+            0.01444950234144926,
+            0.007739055436104536,
+            0.004355259705334902,
+            -0.020033378154039383,
+            -0.01799248717725277,
+            0.0032674651592969894,
+            -0.01649855636060238,
+            -0.0048614004626870155,
+            0.0020531355403363705,
+            -0.019363965839147568,
+            0.01875986158847809,
+            -0.010743245482444763,
+            0.00017538899555802345,
+            -0.003226647386327386,
+            -0.020800752565264702,
+            0.006204306147992611,
+            -0.004865481983870268,
+            0.019870106130838394,
+            0.003555230563506484,
+            0.02785406820476055,
+            0.02310287579894066,
+            -0.004428731743246317,
+            0.017600636929273605,
+            -0.016155686229467392,
+            0.007983962073922157,
+            -0.031135819852352142,
+            -0.001782717532478273,
+            -0.010408539324998856,
+            0.0036103345919400454,
+            -0.015943434089422226,
+            -0.009126860648393631,
+            -0.0048777274787425995,
+            -0.0006678812787868083,
+            -0.004820582456886768,
+            0.0004275664687156677,
+            0.0027817331720143557,
+            0.02231917530298233,
+            -0.013208640739321709,
+            -0.02803366631269455,
+            -0.017437364906072617,
+            0.006045116577297449,
+            0.019396619871258736,
+            -0.026025431230664253,
+            -0.009184005670249462,
+            -0.005012426059693098,
+            -0.004326687194406986,
+            -0.021404854953289032,
+            -0.015167895704507828,
+            -0.005632856860756874,
+            -0.008045189082622528,
+            -0.010522829368710518,
+            0.0046532293781638145,
+            -0.0035715578123927116,
+            -0.011739199981093407,
+            0.021666089072823524,
+            0.018057795241475105,
+            -0.019837452098727226,
+            -0.019837452098727226,
+            0.013641309924423695,
+            -0.006159406621009111,
+            0.0018878233386203647,
+            0.02008235827088356,
+            0.00029286774224601686,
+            0.005824700463563204,
+            0.020882388576865196,
+            -0.0035225762985646725,
+            -0.01797616109251976,
+            0.003577680326998234,
+            -0.0018888438353314996,
+            -0.012122887186706066,
+            0.003212361130863428,
+            0.02773977816104889,
+            0.0008076822850853205,
+            0.023429417982697487,
+            -0.03056436963379383,
+            -0.011412657797336578,
+            0.006061443593353033,
+            0.02369065210223198,
+            0.00211844383738935,
+            -0.0021694661118090153,
+            0.020131340250372887,
+            0.03941367194056511,
+            0.005775718949735165,
+            0.0015755671774968505,
+            -0.04003410041332245,
+            0.01027792226523161,
+            0.009208496659994125,
+            0.024621298536658287,
+            0.008339077234268188,
+            0.0014653591206297278,
+            -0.012906589545309544,
+            -0.009755454957485199,
+            -0.0017939424142241478,
+            0.014702572487294674,
+            -0.023870250210165977,
+            0.0024755997583270073,
+            -0.03157665207982063,
+            0.005094061605632305,
+            0.0021959978621453047,
+            -0.009559528902173042,
+            0.0335359051823616,
+            0.00980443600565195,
+            -0.02065380848944187,
+            0.015151568688452244,
+            0.005767555441707373,
+            0.007330877240747213,
+            0.022449791431427002,
+            -0.008228869177401066,
+            0.030450081452727318,
+            -0.005049162078648806,
+            0.01213921420276165,
+            0.02742956392467022,
+            0.032588932663202286,
+            -0.013102514669299126,
+            -0.003959327004849911,
+            0.0006097159348428249,
+            -0.0002561317232903093,
+            0.0007995187188498676,
+            -0.008204378187656403,
+            0.0037389106582850218,
+            -0.014800535514950752,
+            0.012710663489997387,
+            -0.019543563947081566,
+            -0.0034286954905837774,
+            0.0038532004691660404,
+            0.01991908811032772,
+            0.0016265894519165158,
+            -0.033633869141340256,
+            0.02055584453046322,
+            -0.015420965850353241,
+            0.0135596739128232,
+            0.001290863030590117,
+            -0.011110605672001839,
+            -0.00029210239881649613,
+            0.020588500425219536,
+            0.010947334580123425,
+            -0.007845181971788406,
+            -0.0135596739128232,
+            -0.028376536443829536,
+            -0.021535472944378853,
+            0.00823703221976757,
+            0.009445239789783955,
+            -0.0028837777208536863,
+            0.0021306893322616816,
+            -0.011869817040860653,
+            0.0006520643946714699,
+            -0.0008122742874547839,
+            0.01740471087396145,
+            -0.0335359051823616,
+            0.021943651139736176,
+            -0.019527235999703407,
+            -0.019543563947081566,
+            0.017045514658093452,
+            0.008049271069467068,
+            0.009445239789783955,
+            0.0033552234526723623,
+            -0.01995174214243889,
+            -0.012326976284384727,
+            0.0338951013982296,
+            -0.01138000376522541,
+            0.006298186723142862,
+            0.004408322740346193,
+            0.012914752587676048,
+            -0.008914608508348465,
+            -0.011926962062716484,
+            0.01784554310142994,
+            -0.009861581027507782,
+            0.013951525092124939,
+            0.0010010566329583526,
+            -0.0254703089594841,
+            0.02739690989255905,
+            -0.019233347848057747,
+            0.03487472981214523,
+            -0.021617108955979347,
+            0.00711862463504076,
+            0.0066614653915166855,
+            0.008792154490947723,
+            -0.006690037902444601,
+            -0.013224967755377293,
+            0.0012786176521331072,
+            0.03807484731078148,
+            -0.02623768337070942,
+            0.024180466309189796,
+            0.022400809451937675,
+            -0.00040766780148260295,
+            0.02065380848944187,
+            -0.008367649279534817,
+            -0.026711169630289078,
+            0.0179598331451416,
+            -0.012596373446285725,
+            -0.026123393326997757,
+            0.013257622718811035,
+            -0.022678371518850327,
+            -0.020278284326195717,
+            -0.026302991434931755,
+            0.004947117529809475,
+            -0.0025694805663079023,
+            -0.005151206627488136,
+            0.009004407562315464,
+            0.02561725303530693,
+            0.020915042608976364,
+            0.019576217979192734,
+            0.02111096680164337,
+            0.01161674689501524,
+            0.022449791431427002,
+            -0.007559456862509251,
+            0.04401791840791702,
+            -0.004947117529809475,
+            0.004632820375263691,
+            -0.0015980169409886003,
+            -0.015657709911465645,
+            0.009771781973540783,
+            0.0119106350466609,
+            0.0029572497587651014,
+            -0.029176564887166023,
+            -0.009061552584171295,
+            -0.03660540655255318,
+            0.005053244065493345,
+            0.0020664012990891933,
+            0.0176822729408741,
+            0.032115448266267776,
+            -0.005865518003702164,
+            0.004592002835124731,
+            -0.005102225113660097,
+            -0.013927034102380276,
+            -0.009420748800039291,
+            -0.02739690989255905,
+            -0.025437654927372932,
+            -0.007653337903320789,
+            -0.013739272020757198,
+            -0.030368445441126823,
+            -0.011722872965037823,
+            0.01551892887800932,
+            -0.007645174395292997,
+            0.011918798089027405,
+            -0.016702644526958466,
+            0.01385356206446886,
+            -0.004110352601855993,
+            -0.004763437435030937,
+            -0.004530776292085648,
+            -0.017600636929273605,
+            -0.006420640274882317,
+            0.014367866329848766,
+            -0.009477893821895123,
+            -0.01237595733255148,
+            -0.02215590327978134,
+            -0.0015459742862731218,
+            -0.01574750803411007,
+            0.003489922033622861,
+            -0.014596446417272091,
+            -0.006771673448383808,
+            -0.003908304497599602,
+            -0.018514955416321754,
+            0.01766594499349594,
+            0.014816862531006336,
+            -0.019380291923880577,
+            -0.0090697156265378,
+            0.02638462744653225,
+            0.042777057737112045,
+            -0.03218075633049011,
+            -0.008200296200811863,
+            0.0020378287881612778,
+            0.018090449273586273,
+            0.010171796195209026,
+            0.016702644526958466,
+            -0.024653952568769455,
+            0.0022286521270871162,
+            -0.021649762988090515,
+            -0.001220452249981463,
+            -0.023739634081721306,
+            0.010775899514555931,
+            0.025715215131640434,
+            -0.010971825569868088,
+            0.025976449251174927,
+            0.01634344831109047,
+            -0.01812310516834259,
+            0.012016761116683483,
+            0.007183933164924383,
+            0.014645427465438843,
+            -0.019755816087126732,
+            0.0167679525911808,
+            -0.015796490013599396,
+            0.031233781948685646,
+            0.00892277155071497,
+            -0.04829562455415726,
+            -0.03503800183534622,
+            -0.013388238847255707,
+            0.0013388239312916994,
+            -0.010702427476644516,
+            0.015037278644740582,
+            0.015967924147844315,
+            0.013959688134491444,
+            -0.019380291923880577,
+            -0.015820980072021484,
+            0.011984107084572315,
+            0.010065670125186443,
+            0.027462217956781387,
+            -0.006649219896644354,
+            -0.004212397150695324,
+            -0.00960034690797329,
+            -0.02249877341091633,
+            -0.016098542138934135,
+            0.0209803506731987,
+            -0.018041469156742096,
+            0.015935270115733147,
+            -0.026139721274375916,
+            -0.012449429370462894,
+            0.017453692853450775,
+            -0.05211617052555084,
+            -0.01453113742172718,
+            0.0007612520130351186,
+            0.03012353740632534,
+            0.004751192405819893,
+            0.0035184945445507765,
+            0.01841699331998825,
+            -0.01993541419506073,
+            -0.0039776950143277645,
+            -0.02099667675793171,
+            0.006832899991422892,
+            -0.011853490024805069,
+            -0.013984179124236107,
+            0.014482156373560429,
+            -0.013845399022102356,
+            -0.01601690612733364,
+            0.0034368589986115694,
+            0.007183933164924383,
+            -0.021094640716910362,
+            0.011951452121138573,
+            0.003945040516555309,
+            0.012171869166195393,
+            0.012171869166195393,
+            0.01199227012693882,
+            0.021551799029111862,
+            0.02295593172311783,
+            -0.007334959227591753,
+            -0.004767519421875477,
+            -0.01004117913544178,
+            -0.0015398516552522779,
+            0.008898280560970306,
+            0.0033797139767557383,
+            -0.002989904023706913,
+            0.007837017998099327,
+            0.010033016093075275,
+            -0.01498013362288475,
+            0.033193036913871765,
+            -0.0035878848284482956,
+            -0.04134026914834976,
+            -0.003238892648369074,
+            -0.03800953924655914,
+            0.020033378154039383,
+            -0.023298801854252815,
+            -0.016996532678604126,
+            0.0032776694279164076,
+            -0.00629410520195961,
+            -0.019902760162949562,
+            0.013608654960989952,
+            0.011404493823647499,
+            0.00258580781519413,
+            0.02021297626197338,
+            0.012269831262528896,
+            -0.005134879611432552,
+            0.026172375306487083,
+            -0.014718899503350258,
+            -0.008751336485147476,
+            0.015567909926176071,
+            0.006853308994323015,
+            0.0022347746416926384,
+            0.0011031010653823614,
+            -0.018825169652700424,
+            0.005098143592476845,
+            0.0164903923869133,
+            0.007216587662696838,
+            -0.000566347036510706,
+            0.014049487188458443,
+            0.012514738366007805,
+            -0.015135240741074085,
+            -0.0002213090192526579,
+            -0.026564225554466248,
+            0.0032817514147609472,
+            0.004567512311041355,
+            0.007506393827497959,
+            -0.006771673448383808,
+            0.006281859707087278,
+            0.00658799335360527,
+            0.014759717509150505,
+            -0.010155469179153442,
+            -0.016980206593871117,
+            -0.002510294783860445,
+            0.004285869188606739,
+            0.016694480553269386,
+            0.03157665207982063,
+            -0.010310577228665352,
+            -0.005734901409596205,
+            -0.0386626236140728,
+            0.02070278860628605,
+            0.03332365304231644,
+            0.011175914667546749,
+            -0.028213264420628548,
+            0.012302485294640064,
+            0.0006515541463159025,
+            0.0019276207312941551,
+            -0.026302991434931755,
+            0.02787039615213871,
+            0.00974729098379612,
+            0.0066614653915166855,
+            0.017861871048808098,
+            0.0022306928876787424,
+            0.0029266364872455597,
+            -0.016302630305290222,
+            0.01018812321126461,
+            0.009086042642593384,
+            -0.017306748777627945,
+            0.0009745250572450459,
+            0.02324981987476349,
+            -0.01423724927008152,
+            -0.003208279376849532,
+            0.0018133309204131365,
+            0.02651524543762207,
+            -0.013820908032357693,
+            -0.014294394291937351,
+            0.0005714492290280759,
+            0.01751900091767311,
+            -0.0007979880319908261,
+            0.012098397128283978,
+            0.04898136481642723,
+            -0.012424939312040806,
+            0.019657853990793228,
+            0.005461421795189381,
+            -0.015290348790585995,
+            -0.032605260610580444,
+            -0.004922627005726099,
+            -0.01830270327627659,
+            0.011200404725968838,
+            0.014343376271426678,
+            -0.015763835981488228,
+            0.0007525782566517591,
+            -0.03144603595137596,
+            0.007743136957287788,
+            0.014735226519405842,
+            0.018776189535856247,
+            -0.002440904499962926,
+            0.006134915631264448,
+            0.027674470096826553,
+            0.02458864450454712,
+            -0.007392104249447584,
+            0.013804581016302109,
+            0.02320083975791931,
+            -0.025062130764126778,
+            -0.003069498809054494,
+            0.006820654962211847,
+            0.004104230087250471,
+            0.0053063142113387585,
+            -0.016963878646492958,
+            0.03915243595838547,
+            -0.005922663025557995,
+            -0.01506176870316267,
+            0.0030266400426626205,
+            -0.009028897620737553,
+            0.0063390047289431095,
+            -0.012563719414174557,
+            0.012637191452085972,
+            -0.008114579133689404,
+            0.00022883480414748192,
+            -0.0059430720284581184,
+            -0.011110605672001839,
+            -0.012653518468141556,
+            -0.0007949267164804041,
+            -0.016531210392713547,
+            0.007432921789586544,
+            0.00981259997934103,
+            0.005045080557465553,
+            0.01025343220680952,
+            -0.0036797248758375645,
+            -0.003136848099529743,
+            -0.016751626506447792,
+            -0.009984034113585949,
+            0.0073716952465474606,
+            0.01872720755636692,
+            -0.028588788583874702,
+            -0.018547609448432922,
+            0.009290131740272045,
+            0.0022898786701261997,
+            0.018776189535856247,
+            0.0158944521099329,
+            -0.0002595757250674069,
+            0.012318813242018223,
+            -0.01964152604341507,
+            -0.009184005670249462,
+            0.009690145961940289,
+            -0.003569516818970442,
+            0.01048201136291027,
+            -0.014759717509150505,
+            0.010033016093075275,
+            -0.011159587651491165,
+            -0.011053460650146008,
+            0.01385356206446886,
+            0.01738838478922844,
+            0.008873790502548218,
+            -0.001294944784604013,
+            3.383413059054874e-05,
+            -0.010506502352654934,
+            -0.024033522233366966,
+            0.004771600943058729,
+            0.027021385729312897,
+            -0.013273949734866619,
+            0.003883813973516226,
+            0.008702355436980724,
+            -0.017502672970294952,
+            -0.019821124151349068,
+            -0.012653518468141556,
+            -0.022106921300292015,
+            0.020572172477841377,
+            0.023413091897964478,
+            -0.02026195637881756,
+            0.00877582747489214,
+            -0.004620575346052647,
+            0.014049487188458443,
+            -0.01400050614029169,
+            0.010612628422677517,
+            0.0317889042198658,
+            -0.020000724121928215,
+            -0.0028654097113758326,
+            0.0036633978597819805,
+            -0.015347493812441826,
+            0.018629245460033417,
+            0.03268689662218094,
+            -0.00176945177372545,
+            0.014041324146091938,
+            0.0008138049161061645,
+            -0.009118696674704552,
+            0.01905374974012375,
+            0.011094278655946255,
+            0.005347132217139006,
+            -0.0042491331696510315,
+            -0.012212686240673065,
+            -0.03010721132159233,
+            0.016702644526958466,
+            0.0129637336358428,
+            -0.0037674831692129374,
+            -0.0011551438365131617,
+            -0.006343086715787649,
+            -0.015404638834297657,
+            0.0035123717971146107,
+            -0.013078023679554462,
+            -0.00803294312208891,
+            -0.008759500458836555,
+            0.004314441699534655,
+            -0.0034960447810590267,
+            -0.009053388610482216,
+            0.007236996199935675,
+            0.0009510547970421612,
+            -0.0010602424154058099,
+            -0.01902109570801258,
+            -0.025078458711504936,
+            -0.00995954405516386,
+            -0.016849588602781296,
+            0.0038287099450826645,
+            0.0005234883283264935,
+            -0.027184655889868736,
+            0.0053961132653057575,
+            -0.008139070123434067,
+            -0.008134988136589527,
+            0.007126788143068552,
+            0.006881881505250931,
+            -0.007437003776431084,
+            -0.010816717520356178,
+            -0.00786150898784399,
+            -0.006865554489195347,
+            0.020033378154039383,
+            0.0034348180051892996,
+            0.012082069180905819,
+            -0.010498338378965855,
+            -0.018025141209363937,
+            -0.016963878646492958,
+            0.016718972474336624,
+            -0.008971752598881721,
+            -0.02083340659737587,
+            -0.01444950234144926,
+            -0.008424794301390648,
+            -0.007665583398193121,
+            -0.01527402177453041,
+            -0.021159948781132698,
+            -0.0032103201374411583,
+            -0.011347348801791668,
+            -0.0067431009374558926,
+            -0.0028960229828953743,
+            0.0013571919407695532,
+            0.00951871182769537,
+            0.00546958576887846,
+            0.00438791373744607,
+            0.021666089072823524,
+            0.010122815147042274,
+            -0.005461421795189381,
+            0.016441410407423973,
+            -0.006183897145092487,
+            -0.004440976772457361,
+            -0.01498013362288475,
+            -0.009918726049363613,
+            0.018547609448432922,
+            -0.0055226488038897514,
+            0.008457448333501816,
+            -0.0009536059224046767,
+            -0.00576347392052412,
+            0.015674035996198654,
+            0.0129637336358428,
+            0.016718972474336624,
+            0.004771600943058729,
+            -0.00012168807734269649,
+            0.012416775338351727,
+            -0.0015745466807857156,
+            0.012555556371808052,
+            -0.0038429962005466223,
+            0.014506647363305092,
+            -0.009722800925374031,
+            -0.018988441675901413,
+            0.004157293122261763,
+            0.006265532691031694,
+            -0.012531065382063389,
+            0.009404421783983707,
+            0.007743136957287788,
+            -0.013747435994446278,
+            -0.02261306345462799,
+            -0.000106317616882734,
+            0.00014362763613462448,
+            0.008122743107378483,
+            0.008971752598881721,
+            -0.008073761127889156,
+            0.012890261597931385,
+            0.004616493359208107,
+            -0.007179851643741131,
+            -0.019657853990793228,
+            0.026009103283286095,
+            0.015608727931976318,
+            0.012269831262528896,
+            -0.052508022636175156,
+            -0.0030980713199824095,
+            0.010743245482444763,
+            -0.015951598063111305,
+            -0.026792805641889572,
+            -0.006404313258826733,
+            -0.008759500458836555,
+            -0.04290767386555672,
+            -0.009624837897717953,
+            -0.0005724696675315499,
+            -0.011388166807591915,
+            0.010400376282632351,
+            0.01573934406042099,
+            0.014612773433327675,
+            0.005489994306117296,
+            -0.007232914678752422,
+            -0.017633290961384773,
+            -0.02788672223687172,
+            0.010057506151497364,
+            -0.013380075804889202,
+            -0.006412476766854525,
+            -0.002263347152620554,
+            -0.007690073922276497,
+            0.0016112826997414231,
+            0.01691489666700363,
+            0.042483169585466385,
+            0.023037567734718323,
+            -0.012335140258073807,
+            -0.020131340250372887,
+            -0.012800462543964386,
+            -0.009192168712615967,
+            0.00633492274209857,
+            0.011592255905270576,
+            -0.004738946910947561,
+            0.0096656559035182,
+            -0.004294032696634531,
+            0.012302485294640064,
+            0.012767808511853218,
+            0.01680060662329197,
+            -2.3023791072773747e-05,
+            0.005579793825745583,
+            -0.0024694770108908415,
+            -0.01738838478922844,
+            -0.004379750229418278,
+            0.008261523209512234,
+            -0.0008872769540175796,
+            0.0019317024853080511,
+            0.014204595237970352,
+            0.019102731719613075,
+            -0.009094206616282463,
+            -0.014612773433327675,
+            -0.016098542138934135,
+            -0.005277741700410843,
+            -0.0012622905196622014,
+            0.029927613213658333,
+            0.02037624642252922,
+            -0.019266001880168915,
+            -0.018237393349409103,
+            -0.018351683393120766,
+            0.00922482367604971,
+            -0.003563394071534276,
+            -0.0064369672909379005,
+            0.01905374974012375,
+            0.023021239787340164,
+            0.017910851165652275,
+            -0.003642988856881857,
+            -0.014865843579173088,
+            0.003804219188168645,
+            0.02847449854016304,
+            0.008367649279534817,
+            0.01078406348824501,
+            0.008726846426725388,
+            -0.0011908593587577343,
+            -0.032441988587379456,
+            -0.0113146947696805,
+            0.00951871182769537,
+            -0.008481939323246479,
+            0.026564225554466248,
+            -0.016531210392713547,
+            0.009388094767928123,
+            -0.005077734589576721,
+            0.0019286411115899682,
+            -0.02365799807012081,
+            0.0014929111348465085,
+            0.010122815147042274,
+            -0.024947840720415115,
+            0.00336950970813632,
+            -0.023592690005898476,
+            -0.00013763252354692668,
+            0.008367649279534817,
+            0.0022837561555206776,
+            0.01650671847164631,
+            -0.0113146947696805,
+            0.0051838611252605915,
+            0.02099667675793171,
+            -0.002832755446434021,
+            -0.01512707769870758,
+            -0.0007842120248824358,
+            0.0005092020728625357,
+            -0.001286781276576221,
+            0.02563357912003994,
+            0.015812816098332405,
+            -0.0036756431218236685,
+            0.020882388576865196,
+            0.008677864447236061,
+            0.0019133344758301973,
+            -0.021764053031802177,
+            0.0023796779569238424,
+            -0.032899148762226105,
+            0.017437364906072617,
+            -0.011126932688057423,
+            0.012490247376263142,
+            0.00868602842092514,
+            -0.01341272983700037,
+            -0.0007388022495433688,
+            -0.0107187544927001,
+            -0.0015725058037787676,
+            -0.007934981025755405,
+            0.009779945947229862,
+            -0.004363423213362694,
+            0.008339077234268188,
+            -0.00024579971795901656,
+            -0.01476788055151701,
+            0.018694553524255753,
+            9.678283277025912e-06,
+            -0.00010925139940809458,
+            0.005441013257950544,
+            0.009935053065419197,
+            0.00786559097468853,
+            0.002009256277233362,
+            0.023870250210165977,
+            -0.031119491904973984,
+            -0.016082214191555977,
+            -0.010204450227320194,
+            0.005543057806789875,
+            -0.007922735065221786,
+            -0.018531281501054764,
+            -0.03257260471582413,
+            0.01318415068089962,
+            0.001293924287892878,
+            -0.0009806477464735508,
+            0.01258004643023014,
+            -0.00942891277372837,
+            -0.0013878053287044168,
+            -0.01606588624417782,
+            -0.014808698557317257,
+            0.006416558753699064,
+            0.0033633869607001543,
+            -0.01222901325672865,
+            -0.023739634081721306,
+            0.014759717509150505,
+            0.0007418635650537908,
+            -0.016980206593871117,
+            -0.017502672970294952,
+            0.00018457300029695034,
+            -0.004804255440831184,
+            0.02204161323606968,
+            -0.016678154468536377,
+            0.003373591462150216,
+            0.0007219648687168956,
+            0.009232986718416214,
+            -0.02889900468289852,
+            0.00716352416202426,
+            0.0006107363733462989,
+            0.0035164537839591503,
+            -0.006518602836877108,
+            -0.021013004705309868,
+            0.011371839791536331,
+            -0.0020674217958003283,
+            -0.014751553535461426,
+            -0.020751770585775375,
+            -0.0020796670578420162,
+            -0.007420676294714212,
+            -0.02724996581673622,
+            0.017910851165652275,
+            0.0009985055075958371,
+            -0.01738838478922844,
+            0.008077843114733696,
+            0.006596156861633062,
+            0.009355440735816956,
+            0.02455599047243595,
+            0.010147306136786938,
+            -0.018776189535856247,
+            -0.0008087027235887945,
+            0.006979844067245722,
+            0.009306458756327629,
+            0.0036450298503041267,
+            0.04222193360328674,
+            -0.008130906149744987,
+            -0.006579829845577478,
+            0.0066614653915166855,
+            -0.008449285291135311,
+            -0.013747435994446278,
+            0.026727497577667236,
+            0.005796127952635288,
+            0.015984252095222473,
+            0.016123032197356224,
+            -0.00945340283215046,
+            -0.005008344538509846,
+            -0.0018786393338814378,
+            0.02065380848944187,
+            0.0073798587545752525,
+            -0.005502239800989628,
+            -0.009714636951684952,
+            0.005881845485419035,
+            -0.00898808054625988,
+            0.0021653843577951193,
+            -0.009526874870061874,
+            -0.005824700463563204,
+            -0.008069679141044617,
+            0.016400592401623726,
+            -0.0003523086488712579,
+            0.027315273880958557,
+            0.020751770585775375,
+            -0.008563575334846973,
+            -0.0005683879135176539,
+            0.017208784818649292,
+            -0.005820618476718664,
+            0.015608727931976318,
+            -0.011192241683602333,
+            -0.007706400938332081,
+            0.006408394780009985,
+            0.024017194285988808,
+            0.012237177230417728,
+            -0.0056042843498289585,
+            0.0002885308349505067,
+            -0.0030348035506904125,
+            -0.010612628422677517,
+            0.013388238847255707,
+            0.015323002822697163,
+            -0.01947825588285923,
+            -0.004722619894891977,
+            -0.0016592437168583274,
+            -0.003300119424238801,
+            0.00945340283215046,
+            0.02305389568209648,
+            0.000171817431692034,
+            0.009184005670249462,
+            0.0012765767751261592,
+            0.03862996771931648,
+            -0.0053144777193665504,
+            0.03102152980864048,
+            -0.0010423846542835236,
+            -0.0059430720284581184,
+            -0.009094206616282463,
+            -0.020441556349396706,
+            -0.011486129835247993,
+            -0.00041481092921458185,
+            -0.005040998570621014,
+            -0.0004956811899319291,
+            -0.020327266305685043,
+            0.0026960158720612526,
+            0.014637264423072338,
+            0.007661501411348581,
+            0.017927179113030434,
+            -0.002889900468289852,
+            0.017437364906072617,
+            0.019690508022904396,
+            -0.01400050614029169,
+            -0.029862305149435997,
+            -0.0006515541463159025,
+            0.01498013362288475,
+            -0.01661284640431404,
+            0.019445601850748062,
+            0.004085862077772617,
+            0.0029531680047512054,
+            0.004779764451086521,
+            -0.006049198564141989,
+            0.018482301384210587,
+            0.02039257436990738,
+            0.01453930139541626,
+            -0.013167822733521461,
+            0.014057651162147522,
+            0.009771781973540783,
+            0.006600238848477602,
+            -0.01646590232849121,
+            -0.013535182923078537,
+            0.002555194543674588,
+            0.0009245232213288546,
+            0.024392718449234962,
+            -0.020588500425219536,
+            0.0016602640971541405,
+            0.010588138364255428,
+            0.01769859902560711,
+            -0.004302196204662323,
+            0.008906444534659386,
+            0.0008740111952647567,
+            0.007134951651096344,
+            -0.0035919665824621916,
+            -0.006457376293838024,
+            -0.006673710886389017,
+            -0.010392212308943272,
+            -0.0020959940738976,
+            -0.02070278860628605,
+            0.002198038622736931,
+            -0.002830714685842395,
+            0.018237393349409103,
+            -0.01018812321126461,
+            0.038140155375003815,
+            0.03399306535720825,
+            -0.0007025764207355678,
+            0.011102442629635334,
+            -0.01709449477493763,
+            -0.003945040516555309,
+            -0.021959977224469185,
+            0.01844964735209942,
+            0.016392430290579796,
+            0.012269831262528896,
+            0.005302232690155506,
+            0.017796561121940613,
+            -0.0006301248213276267,
+            0.014490320347249508,
+            0.0029654132667928934,
+            0.007971717044711113,
+            -0.002265388146042824,
+            0.003938918001949787,
+            -0.031103165820240974,
+            -0.017029186710715294,
+            -0.01138000376522541,
+            0.004796091932803392,
+            -0.006375740747898817,
+            -0.0035205355379730463,
+            0.0084737753495574,
+            -0.005902254022657871,
+            0.0004270562494639307,
+            -0.025992777198553085,
+            -0.0157148540019989,
+            -0.010318740271031857,
+            -0.004040962550789118,
+            -0.01169838197529316,
+            -0.0027694879099726677,
+            -0.0022082431241869926,
+            0.0029552089981734753,
+            0.020947696641087532,
+            -0.012808626517653465,
+            -0.016294466331601143,
+            0.004379750229418278,
+            -0.002263347152620554,
+            -0.014016833156347275,
+            -0.016898570582270622,
+            0.006665547378361225,
+            0.012098397128283978,
+            -0.0017470018938183784,
+            -0.013608654960989952,
+            0.011739199981093407,
+            -0.006628811359405518,
+            0.04663025587797165,
+            -0.013584164902567863,
+            -0.0078002819791436195,
+            0.003720542648807168,
+            0.004698128905147314,
+            -0.009396257810294628,
+            0.007330877240747213,
+            0.013020878657698631,
+            0.01830270327627659,
+            0.0095758568495512,
+            -0.008465612307190895,
+            -0.0024388637393712997,
+            0.007796200457960367,
+            -0.016147522255778313,
+            0.006404313258826733,
+            0.0008525818702764809,
+            -0.014400520361959934,
+            0.006110425107181072,
+            0.028980640694499016,
+            0.013151495717465878,
+            0.01288209855556488,
+            -0.022351829335093498,
+            -0.004094025585800409,
+            -0.009355440735816956,
+            -0.004943036008626223,
+            -0.010090161114931107,
+            0.027070365846157074,
+            -0.018155759200453758,
+            0.02651524543762207,
+            0.00898808054625988,
+            0.0017684312770143151,
+            -0.01176369097083807,
+            0.006343086715787649,
+            0.02832755446434021,
+            0.0028592869639396667,
+            -0.020000724121928215,
+            -0.013592327944934368,
+            0.006314514204859734,
+            0.01707816869020462,
+            -0.002647034591063857,
+            -0.005428767763078213,
+            0.019657853990793228,
+            0.01784554310142994,
+            -0.004334850702434778,
+            -0.00936360377818346,
+            0.0008337036124430597,
+            0.00928196869790554,
+            0.016963878646492958,
+            0.015020951628684998,
+            0.009086042642593384,
+            -0.002647034591063857,
+            0.008037025108933449,
+            -0.008800318464636803,
+            -0.016449574381113052,
+            0.007220669183880091,
+            -0.008432958275079727,
+            -0.016963878646492958,
+            0.01872720755636692,
+            0.015257694758474827,
+            0.0181394312530756,
+            0.0024592725094407797,
+            -0.016735298559069633,
+            0.022302847355604172,
+            0.005755309946835041,
+            0.002998067531734705,
+            0.02160078100860119,
+            -0.022221211344003677,
+            0.013012715615332127,
+            0.02037624642252922,
+            0.027609162032604218,
+            -0.01737205684185028,
+            -0.010906516574323177,
+            -0.004857318475842476,
+            -0.006853308994323015,
+            -0.00176945177372545,
+            -0.008253359235823154,
+            -0.012620864436030388,
+            0.0028735732194036245,
+            -0.030466407537460327,
+            0.018106777220964432,
+            -0.005102225113660097,
+            0.010669773444533348,
+            -0.019266001880168915,
+            0.011363675817847252,
+            -0.0035838030744343996,
+            0.011135096661746502,
+            0.0019612954929471016,
+            -0.030041903257369995,
+            -0.008571738377213478,
+            0.015388311818242073,
+            -0.0030368445441126823,
+            -0.013641309924423695,
+            0.02006603218615055,
+            0.00018942011229228228,
+            -0.022270193323493004,
+            -0.004314441699534655,
+            -0.006934944540262222,
+            0.002193956868723035,
+            -0.0029613315127789974,
+            -0.0052981507033109665,
+            -0.030188847333192825,
+            0.020033378154039383,
+            0.009559528902173042,
+            -0.03072764165699482,
+            -0.003255219664424658,
+            -0.014996460638940334,
+            -0.0025184585247188807,
+            0.008881953544914722,
+            -0.004000144544988871,
+            -0.015551582910120487,
+            0.01784554310142994,
+            0.022711025550961494,
+            -0.0023164101876318455,
+            -0.016057724133133888,
+            -0.0090697156265378,
+            -0.027494871988892555,
+            0.030825603753328323,
+            -0.0066696288995444775,
+            0.0022878379095345736,
+            -0.005322641227394342,
+            0.009388094767928123,
+            0.004228724166750908,
+            0.0157148540019989,
+            0.006081852596253157,
+            -0.002647034591063857,
+            0.026596879586577415,
+            -0.01964152604341507,
+            0.009567692875862122,
+            -1.588864870427642e-05,
+            -0.003308282932266593,
+            0.01890680566430092,
+            0.0007939062779769301,
+            0.021649762988090515,
+            -0.008571738377213478,
+            -0.0001475818717153743,
+            -0.0034388997592031956,
+            0.010898353531956673,
+            -0.0028450009413063526,
+            -0.003465431509539485,
+            -0.001531688030809164,
+            0.011118769645690918,
+            0.01013097818940878,
+            0.003804219188168645,
+            -0.01737205684185028,
+            0.003142970846965909,
+            -0.008547247387468815,
+            -0.007028825581073761,
+            0.01302904263138771,
+            -0.011192241683602333,
+            -0.0018990482203662395,
+            -0.03160930797457695,
+            0.016286302357912064,
+            0.0533243790268898,
+            0.008547247387468815,
+            -0.014433175325393677,
+            -0.0018806802108883858,
+            -0.019755816087126732,
+            0.01535565685480833,
+            0.011004479601979256,
+            -0.016816934570670128,
+            -0.011984107084572315,
+            -0.012310649268329144,
+            0.007649256382137537,
+            0.021306892856955528,
+            -0.006522684823721647,
+            -0.008579902350902557,
+            0.025372346863150597,
+            -0.011788181029260159,
+            -0.010882026515901089,
+            -0.01661284640431404,
+            -0.01635977439582348,
+            0.00241641397587955,
+            0.019902760162949562,
+            -0.0169475506991148,
+            0.021176276728510857,
+            0.00936360377818346,
+            0.0010133018950000405,
+            0.003624620847404003,
+            0.008588065393269062,
+            0.0024878450203686953,
+            0.017878197133541107,
+            0.003883813973516226,
+            -0.01439235731959343,
+            0.0021408938337117434,
+            -0.0030286810360848904,
+            0.004624656867235899,
+            3.4320437407586724e-06,
+            -0.002749078907072544,
+            -0.0065635028295218945,
+            0.003710338147357106,
+            -0.021306892856955528,
+            0.021502818912267685,
+            -0.022678371518850327,
+            -0.004245051648467779,
+            0.005555302836000919,
+            -0.005118552595376968,
+            0.013469874858856201,
+            0.018792515620589256,
+            0.01423724927008152,
+            0.008702355436980724,
+            0.01462093647569418,
+            -0.011331021785736084,
+            0.02323349379003048,
+            -0.013282112777233124,
+            0.0005816536722704768,
+            0.012939243577420712,
+            0.01755165494978428,
+            0.018792515620589256,
+            -0.006245123688131571,
+            -0.010384049266576767,
+            -0.0002946535241790116,
+            0.0129637336358428,
+            -0.022286519408226013,
+            -0.012269831262528896,
+            -0.01386172603815794,
+            0.011486129835247993,
+            -0.004743028432130814,
+            0.025453981012105942,
+            -0.00633492274209857,
+            -0.014653591439127922,
+            -0.015918942168354988,
+            0.014196431264281273,
+            -0.048752784729003906,
+            0.0030348035506904125,
+            -0.0024919267743825912,
+            0.03043375350534916,
+            -0.027021385729312897,
+            0.01738838478922844,
+            -0.0028490826953202486,
+            -0.009355440735816956,
+            0.015429128892719746,
+            -0.015992414206266403,
+            -0.0046613928861916065,
+            -0.0016806729836389422,
+            -0.007539048325270414,
+            -0.00112248957157135,
+            0.017616963014006615,
+            -0.015845470130443573,
+            0.01282495353370905,
+            -0.00989423505961895,
+            0.0027021386194974184,
+            -0.004526694305241108,
+            -0.007751300930976868,
+            0.0196088720113039,
+            0.034548189491033554,
+            -0.007053316105157137,
+            0.0014990337658673525,
+            -0.022547753527760506,
+            0.013192313723266125,
+            -0.006730855442583561,
+            0.02351105399429798,
+            -0.005432849284261465,
+            -0.016898570582270622,
+            -0.013976015150547028,
+            -0.009739127941429615,
+            0.00839214026927948,
+            -0.022433463484048843,
+            -0.011110605672001839,
+            -0.00922482367604971,
+            -0.002573562553152442,
+            -0.018694553524255753,
+            0.004865481983870268,
+            -0.002426618244498968,
+            -0.012506574392318726,
+            -0.008816645480692387,
+            0.01048201136291027,
+            -0.009347276762127876,
+            0.003077662317082286,
+            -0.0011929002357646823,
+            -0.011453475803136826,
+            -0.02369065210223198,
+            0.0158944521099329,
+            0.0001974561164388433,
+            -0.010359558276832104,
+            0.008408467285335064,
+            -0.0018276170594617724,
+            0.001211268245242536,
+            0.016963878646492958,
+            -0.0020664012990891933,
+            0.003467472270131111,
+            0.023478399962186813,
+            0.004057289566844702,
+            -0.010971825569868088,
+            -0.00025906547671183944,
+            0.013535182923078537,
+            -0.005118552595376968,
+            -0.010008525103330612,
+            1.6646010408294387e-05,
+            -0.02984597720205784,
+            -0.008351322263479233,
+            0.006094098091125488,
+            0.018629245460033417,
+            -0.010620792396366596,
+            -0.015853634104132652,
+            0.026319319382309914,
+            -0.006359413731843233,
+            0.019070077687501907,
+            0.002889900468289852,
+            0.0025694805663079023,
+            0.01724144071340561,
+            0.013461710885167122,
+            0.00786150898784399,
+            -0.013927034102380276,
+            -0.02429475635290146,
+            0.0016398552106693387,
+            -0.018090449273586273,
+            0.010579974390566349,
+            -0.011028970591723919,
+            0.0032756286673247814,
+            -0.02398454025387764,
+            -0.012237177230417728,
+            0.004934872500598431,
+            -0.020343592390418053,
+            -0.00040588201954960823,
+            -0.0056940834037959576,
+            0.022237539291381836,
+            -0.005310396198183298,
+            0.008669701404869556,
+            0.0018255761824548244,
+            -0.012237177230417728,
+            0.0009877908742055297,
+            0.009469729848206043,
+            -0.029192892834544182,
+            -0.0007127808639779687,
+            -0.003138889092952013,
+            0.0011418780777603388,
+            0.005351213738322258,
+            0.009755454957485199,
+            0.007322713732719421,
+            0.0027062203735113144,
+            0.007253323681652546,
+            0.010449357330799103,
+            -0.00022041612828616053,
+            0.0197884701192379,
+            0.018172085285186768,
+            -0.007351286243647337,
+            0.008710519410669804,
+            0.024817224591970444,
+            -0.013159659691154957,
+            -0.008718682453036308,
+            0.023266147822141647,
+            -0.006461458280682564,
+            -0.0011122850701212883,
+            -0.01483318954706192,
+            0.00815947912633419,
+            0.0017367975087836385,
+            0.028131630271673203,
+            0.019657853990793228,
+            0.005820618476718664,
+            0.000639819074422121,
+            0.015788326039910316,
+            -0.013020878657698631,
+            0.0035286990460008383,
+            0.015461783856153488,
+            -0.003940958995372057,
+            -0.019821124151349068,
+            0.013878053054213524,
+            0.009502384811639786,
+            -0.003228688146919012,
+            0.01551892887800932,
+            0.011412657797336578,
+            -0.030841931700706482,
+            -0.0011898389784619212,
+            0.010498338378965855,
+            -0.0021449755877256393,
+            -0.008751336485147476,
+            -0.00014273475972004235,
+            -0.012384121306240559,
+            0.021388528868556023,
+            -0.004408322740346193,
+            0.012277995236217976,
+            -0.00815947912633419,
+            -0.017127150669693947,
+            -0.011559601873159409,
+            0.007914572022855282,
+            -0.010547320358455181,
+            -0.0006714528426527977,
+            -0.004022594541311264,
+            -0.005779800936579704,
+            0.017747581005096436,
+            0.004706292413175106,
+            -0.0008964610169641674,
+            -0.0018398624379187822,
+            0.007983962073922157,
+            -0.006979844067245722,
+            -0.008726846426725388,
+            0.004816500470042229,
+            -0.012833117507398129,
+            0.003981776535511017,
+            0.005543057806789875,
+            0.01506176870316267,
+            -0.014857680536806583,
+            -0.021159948781132698,
+            0.007020662073045969,
+            0.02695607766509056,
+            -0.02516009286046028,
+            -0.005232842173427343,
+            0.006245123688131571,
+            0.002093953313305974,
+            0.006914535537362099,
+            -0.006934944540262222,
+            -0.007208424154669046,
+            -0.03657275065779686,
+            -0.0020745648071169853,
+            0.0009005427709780633,
+            0.008963589556515217,
+            -0.008049271069467068,
+            -0.0136494729667902,
+            -0.011135096661746502,
+            0.002410291228443384,
+            0.007134951651096344,
+            -0.02204161323606968,
+            0.0049593630246818066,
+            0.009608510881662369,
+            -0.019559891894459724,
+            0.025127438828349113,
+            -0.010890189558267593,
+            -6.964537897147238e-05,
+            -0.00995954405516386,
+            0.021633435040712357,
+            -0.013486201874911785,
+            -0.018368011340498924,
+            -0.0006898208521306515,
+            0.008355404250323772,
+            0.011926962062716484,
+            -0.0017663904000073671,
+            -0.011371839791536331,
+            0.0003752686607185751,
+            0.01646590232849121,
+            0.019135385751724243,
+            0.002744997153058648,
+            -0.0024837632663547993,
+            -0.01019628718495369,
+            -0.009592183865606785,
+            -0.0073961857706308365,
+            -0.0007449248805642128,
+            0.002091912319883704,
+            0.02035992033779621,
+            0.013469874858856201,
+            0.0024837632663547993,
+            -0.015135240741074085,
+            -0.005257333163172007,
+            0.008939098566770554,
+            0.009412584826350212,
+            -0.011690218932926655,
+            -0.00898808054625988,
+            0.002659279853105545,
+            0.02249877341091633,
+            0.017633290961384773,
+            -0.0014000505907461047,
+            0.0032878739293664694,
+            -0.0016592437168583274,
+            0.012506574392318726,
+            -0.007555375341325998,
+            -0.010269759222865105,
+            -0.006943108048290014,
+            -0.014645427465438843,
+            -0.010800390504300594,
+            -0.0023062059190124273,
+            -0.010179960168898106,
+            0.022629389539361,
+            0.007988044060766697,
+            0.009779945947229862,
+            0.008265605196356773,
+            0.010269759222865105,
+            0.002087830565869808,
+            0.025715215131640434,
+            -0.004424649756401777,
+            -0.011984107084572315,
+            0.007236996199935675,
+            -0.022368155419826508,
+            -0.021535472944378853,
+            0.008257441222667694,
+            0.006367577239871025,
+            0.03595232218503952,
+            -0.026890767738223076,
+            0.0052981507033109665,
+            -0.009779945947229862,
+            -0.010400376282632351,
+            0.007996207103133202,
+            -0.0015439334092661738,
+            -0.004245051648467779,
+            0.022368155419826508,
+            0.00726556871086359,
+            0.0035797213204205036,
+            0.0032817514147609472,
+            0.019837452098727226,
+            -0.005685919895768166,
+            0.01573934406042099,
+            0.0259274672716856,
+            0.024621298536658287,
+            0.009388094767928123,
+            -0.009469729848206043,
+            0.02785406820476055,
+            0.0002895512734539807,
+            -0.0034327772445976734,
+            0.007220669183880091,
+            -0.00305113079957664,
+            -0.0009556467994116247,
+            -0.010033016093075275,
+            0.021764053031802177,
+            -0.001455154619179666,
+            0.004359341226518154,
+            0.02081707864999771,
+            -7.774515688652173e-05,
+            0.006220633164048195,
+            0.01057181041687727,
+            0.002181711606681347,
+            0.003396041225641966,
+            0.010775899514555931,
+            -0.039315707981586456,
+            -0.0006571666453965008,
+            -0.025600925087928772,
+            -0.0018286375561729074,
+            0.0007357408758252859,
+            -0.01266984548419714,
+            0.013600491918623447,
+            -0.002015379024669528,
+            -0.010751409456133842,
+            -0.00029337796149775386,
+            0.021649762988090515,
+            0.005416522268205881,
+            0.004071575589478016,
+            -0.004469549283385277,
+            0.018792515620589256,
+            -0.006955353543162346,
+            -0.017861871048808098,
+            -0.012759645469486713,
+            -0.012367794290184975,
+            -0.01918436773121357,
+            -0.01214737817645073,
+            -0.0024551907554268837,
+            -0.002667443361133337,
+            0.004738946910947561,
+            0.002481722505763173,
+            -0.003228688146919012,
+            -0.02070278860628605,
+            0.015037278644740582,
+            -0.01430255826562643,
+            0.012873934581875801,
+            -0.011053460650146008,
+            0.005963481031358242,
+            -0.004122598096728325,
+            0.012726990506052971,
+            -0.021404854953289032,
+            0.02070278860628605,
+            -0.01536382082849741,
+            0.0038532004691660404,
+            0.019592545926570892,
+            0.006551257334649563,
+            -0.008118661120533943,
+            -0.013404566794633865,
+            -0.022074267268180847,
+            0.021633435040712357,
+            -0.011633073911070824,
+            -0.018237393349409103,
+            -0.004608329851180315,
+            0.009649328887462616,
+            -0.017470018938183784,
+            -0.007367613259702921,
+            0.01707816869020462,
+            -0.0018949664663523436,
+            0.012980061583220959,
+            0.023004913702607155,
+            -0.0052981507033109665,
+            0.0036572751123458147,
+            -0.009167678654193878,
+            0.0010286086471751332,
+            -0.02470293454825878,
+            0.013527019880712032,
+            0.008971752598881721,
+            0.006269614677876234,
+            -0.013935198076069355,
+            -0.013306603766977787,
+            0.0036531933583319187,
+            0.004498121794313192,
+            0.011437147855758667,
+            0.009420748800039291,
+            0.004220560658723116,
+            -0.013918871060013771,
+            -0.027788760140538216,
+            -0.027070365846157074,
+            0.03595232218503952,
+            -0.0005002731923013926,
+            -0.020947696641087532,
+            0.03399306535720825,
+            0.016114868223667145,
+            0.0022980421781539917,
+            -0.012449429370462894,
+            -0.004314441699534655,
+            0.024621298536658287,
+            0.016400592401623726,
+            0.010465684346854687,
+            -0.0004742518358398229,
+            0.005571629852056503,
+            0.0016469983384013176,
+            0.03012353740632534,
+            0.015323002822697163,
+            0.0009459525463171303,
+            -0.038597315549850464,
+            0.011943289078772068,
+            -0.007547211833298206,
+            0.0013408648082986474,
+            0.006702283397316933,
+            0.017780235037207603,
+            0.011935125105082989,
+            0.015567909926176071,
+            0.014865843579173088,
+            -0.003065417055040598,
+            0.0029511270113289356,
+            0.0006163488142192364,
+            -0.009624837897717953,
+            -0.001767410896718502,
+            -0.008881953544914722,
+            -0.005714492406696081,
+            -0.002920513739809394,
+            0.005592038854956627,
+            0.007657419890165329,
+            -0.003983817528933287,
+            0.028049994260072708,
+            -0.0006433906382881105,
+            -0.017600636929273605,
+            -0.009061552584171295,
+            -0.03611559048295021,
+            0.00434301421046257,
+            -0.030939893797039986,
+            -0.011257549747824669,
+            0.0033674687147140503,
+            -0.020457882434129715,
+            -0.024768242612481117,
+            0.0010306495241820812,
+            0.014335212297737598,
+            0.007392104249447584,
+            0.0033327736891806126,
+            -0.01004934310913086,
+            0.003816464450210333,
+            -0.001855169073678553,
+            -0.006461458280682564,
+            0.028833696618676186,
+            0.019559891894459724,
+            0.020457882434129715,
+            -0.01190247107297182,
+            0.019298657774925232,
+            0.005845109466463327,
+            0.02035992033779621,
+            0.010465684346854687,
+            -0.0048695639707148075,
+            0.0074859848245978355,
+            0.009567692875862122,
+            -0.006155324634164572,
+            -0.021519144997000694,
+            0.013192313723266125,
+            -0.0010837126756086946,
+            -0.0052083516493439674,
+            -0.016269976273179054,
+            -0.003534821793437004,
+            -0.015812816098332405,
+            0.010547320358455181,
+            -0.017159804701805115,
+            0.012498411349952221,
+            -0.000802069844212383,
+            0.00168475485406816,
+            -0.015469946898519993,
+            -0.013951525092124939,
+            -0.0012439225101843476,
+            -0.01244126632809639,
+            -0.008971752598881721,
+            0.006163488142192364,
+            -0.007539048325270414,
+            -0.008318668231368065,
+            -0.00535937724635005,
+            -0.011437147855758667,
+            -0.010882026515901089,
+            0.004367504734545946,
+            -0.0010653446661308408,
+            0.020931368693709373,
+            0.00011671340325847268,
+            -0.005824700463563204,
+            0.01190247107297182,
+            0.02787039615213871,
+            0.0016704685986042023,
+            0.004583839327096939,
+            -0.014261740259826183,
+            -0.011118769645690918,
+            -0.02279266156256199,
+            0.016572028398513794,
+            0.0049675265327095985,
+            -0.0344502255320549,
+            0.01438419334590435,
+            0.00824927818030119,
+            0.0015102586476132274,
+            -0.006698201410472393,
+            -0.005334886722266674,
+            -0.005608365871012211,
+            0.006306350696831942,
+            -0.0022837561555206776,
+            0.0005107327597215772,
+            0.0035919665824621916,
+            0.034678805619478226,
+            -0.0040674940682947636,
+            -0.007228832691907883,
+            0.00981259997934103,
+            0.0005130287609063089,
+            0.020196648314595222,
+            -0.005518566817045212,
+            0.0037572786677628756,
+            -0.019282329827547073,
+            -0.016563864424824715,
+            0.0018429238116368651,
+            -0.01722511276602745,
+            0.023282473906874657,
+            -0.005392031744122505,
+            -0.01347803883254528,
+            -0.006906372029334307,
+            -0.020115012302994728,
+            0.012637191452085972,
+            0.02310287579894066,
+            0.007016580086201429,
+            0.01033506728708744,
+            0.012131051160395145,
+            -0.02444170042872429,
+            -0.020457882434129715,
+            0.012171869166195393,
+            0.0013571919407695532,
+            -0.008449285291135311,
+            -0.005404277238994837,
+            -0.001783737912774086,
+            0.007783954963088036,
+            -0.0032633834052830935,
+            0.007881917990744114,
+            0.06491663306951523,
+            -0.01093100756406784,
+            -0.002987863030284643,
+            -0.02966637909412384,
+            -0.004396077245473862,
+            -0.006628811359405518,
+            0.0056287748739123344,
+            0.009151351638138294,
+            -0.023135529831051826,
+            0.009232986718416214,
+            -0.030025575309991837,
+            -0.010628955438733101,
+            0.00824111420661211,
+            0.0028490826953202486,
+            -0.01220452319830656,
+            -0.0009592183632776141,
+            -0.013665799982845783,
+            0.0006107363733462989,
+            -0.009412584826350212,
+            -0.00576347392052412,
+            0.012718827463686466,
+            -0.02321716584265232,
+            0.006298186723142862,
+            0.007592111360281706,
+            -0.006514521315693855,
+            -0.013722945004701614,
+            -0.017861871048808098,
+            -0.011967780068516731,
+            -0.013314766809344292,
+            0.002996026771143079,
+            0.01190247107297182,
+            -0.02607441134750843,
+            0.012922916561365128,
+            -0.0037389106582850218,
+            0.00945340283215046,
+            0.004404240753501654,
+            0.013469874858856201,
+            0.018074123188853264,
+            0.009355440735816956,
+            0.0021429345943033695,
+            -0.008024780079722404,
+            0.012702500447630882,
+            -0.01993541419506073,
+            -0.01635161228477955,
+            0.013837235048413277,
+            0.012016761116683483,
+            0.003061335301026702,
+            0.009828926995396614,
+            0.0020959940738976,
+            0.011886144056916237,
+            0.010775899514555931,
+            -0.019739490002393723,
+            -0.0006648199632763863,
+            0.0007326795603148639,
+            0.03280118480324745,
+            -0.0005938990507274866,
+            0.008857463486492634,
+            0.004420568235218525,
+            0.013159659691154957,
+            -0.012784135527908802,
+            -0.01769859902560711,
+            0.0008087027235887945,
+            -0.01057181041687727,
+            0.021257910877466202,
+            -0.008192133158445358,
+            0.006216551177203655,
+            -0.018612917512655258,
+            -0.027086693793535233,
+            0.006392067763954401,
+            -0.016261812299489975,
+            -0.018221067264676094,
+            0.01317598670721054,
+            0.0077268099412322044,
+            0.0017572063952684402,
+            -0.005098143592476845,
+            -0.020457882434129715,
+            -0.009837090037763119,
+            0.028441844508051872,
+            0.011102442629635334,
+            -0.004120557103306055,
+            0.009584019891917706,
+            -0.009094206616282463,
+            -0.012490247376263142,
+            -0.0017541450215503573,
+            0.0036715613678097725,
+            0.008261523209512234,
+            0.005645101889967918,
+            -0.01386172603815794,
+            -0.00029873527819290757,
+            -0.012400448322296143,
+            -0.021225256845355034,
+            -0.008906444534659386,
+            0.016751626506447792,
+            0.007820690982043743,
+            -0.0038960592355579138,
+            0.015812816098332405,
+            -0.004404240753501654,
+            0.002189875114709139,
+            0.014294394291937351,
+            0.005334886722266674,
+            -0.004285869188606739,
+            -0.0095758568495512,
+            -0.004612411838024855,
+            0.004043003544211388,
+            -0.0025960123166441917,
+            0.006457376293838024,
+            0.017192458733916283,
+            0.0072900597006082535,
+            -0.013755599036812782,
+            0.013135168701410294,
+            0.0007791098323650658,
+            0.0049675265327095985,
+            -0.03699725493788719,
+            -0.003247056156396866,
+            0.015796490013599396,
+            -0.013020878657698631,
+            0.02501314878463745,
+            0.005400195252150297,
+            0.022743679583072662,
+            0.015233203768730164,
+            0.008669701404869556,
+            -0.001542912912555039,
+            0.00464098434895277,
+            0.02759283408522606,
+            -0.003151134354993701,
+            -0.008637047372758389,
+            -0.0020602785516530275,
+            0.009241150692105293,
+            0.0002063850115519017,
+            0.003000108525156975,
+            0.008281932212412357,
+            -0.03562577813863754,
+            0.011706545948982239,
+            -0.019576217979192734,
+            0.0024694770108908415,
+            0.003459308762103319,
+            -0.007183933164924383,
+            -0.014865843579173088,
+            -0.005857354495674372,
+            0.022841641679406166,
+            -0.003224606392905116,
+            0.021078312769532204,
+            0.002885818714275956,
+            0.016106704249978065,
+            -0.011575928889214993,
+            -0.009208496659994125,
+            -0.0004130251472815871,
+            -0.005351213738322258,
+            -0.007539048325270414,
+            0.010596301406621933,
+            0.0034409407526254654,
+            0.005534893833100796,
+            0.002265388146042824,
+            0.0036654386203736067,
+            -0.0036878883838653564,
+            0.02773977816104889,
+            0.003563394071534276,
+            0.007996207103133202,
+            -0.011567764915525913,
+            0.014082142151892185,
+            -0.017029186710715294,
+            0.0070778070949018,
+            0.005575711838901043,
+            -0.026156047359108925,
+            0.016849588602781296,
+            0.004775682929903269,
+            0.006628811359405518,
+            0.013812744058668613,
+            0.01618834026157856,
+            -0.010408539324998856,
+            -0.014816862531006336,
+            0.024049850180745125,
+            -0.0032041973900049925,
+            -0.015225040726363659,
+            -0.006775755435228348,
+            4.438935866346583e-05,
+            0.006559420842677355,
+            0.00373074715025723,
+            0.0013020879123359919,
+            -0.0006204306264407933,
+            -0.010743245482444763,
+            0.004645065870136023,
+            -0.01934763789176941,
+            -0.0007163524278439581,
+            -0.016449574381113052,
+            -0.015633217990398407,
+            0.018678225576877594,
+            -0.01830270327627659,
+            0.009633000940084457,
+            0.0012275953777134418,
+            0.028588788583874702,
+            -0.011706545948982239,
+            -0.012155541218817234,
+            0.0015102586476132274,
+            0.010849371552467346,
+            0.0067512644454836845,
+            -0.010033016093075275,
+            -0.007583947852253914,
+            -0.014180104248225689,
+            -0.010163633152842522,
+            0.015502601861953735,
+            -0.014498483389616013,
+            -0.014343376271426678,
+            0.008914608508348465
+          ],
+          "embedding": [
+            -0.006359657085039302,
+            -0.024594786241399234,
+            -0.02553538924848095,
+            0.005360188465088291,
+            0.012801870977589839,
+            0.021680505862506104,
+            -0.016668048966001273,
+            0.06860852474581301,
+            -0.007122272841157724,
+            0.03471775000705488,
+            -0.009004297848861675,
+            0.01059974144678391,
+            0.02565604599426148,
+            -0.022544127646041287,
+            0.010616674976540785,
+            0.013658194823359737,
+            0.016400329743363844,
+            0.02858546964917478,
+            0.0067612676434693686,
+            -0.04174737193606879,
+            0.03184155110488616,
+            -0.02223105127585514,
+            -0.05506239667974977,
+            -0.0015307551410555752,
+            -0.013169788751379287,
+            0.017681749722384007,
+            0.01682706317229353,
+            0.019652295823816928,
+            -0.05623980369271269,
+            0.030698582851533197,
+            0.02286327124296613,
+            0.016019032539689178,
+            0.005508113353872315,
+            8.58985091215149e-05,
+            -0.019206292853495906,
+            -0.018439290483495144,
+            0.03475023176950151,
+            0.01018612942141759,
+            -0.0252551158857426,
+            0.0334562130666827,
+            0.020462153537485785,
+            0.02260508249714211,
+            -0.03323524989392951,
+            -0.002094251009530707,
+            0.015177442347686773,
+            -0.00018698464350038317,
+            -0.0006788648344273403,
+            0.0007777616893581039,
+            0.013352398615059647,
+            -0.030751111528744813,
+            -0.0317733818445192,
+            -0.006729064081710058,
+            0.009413204899986514,
+            -0.005712894407943592,
+            -0.038961826880527066,
+            -0.002827692206530345,
+            0.017863300537973137,
+            0.07918455388563496,
+            0.048967158919816366,
+            0.018002566325350088,
+            0.040264962620950695,
+            -0.019715100359659787,
+            -0.02211350500879408,
+            0.004918148608475574,
+            0.01062585305866606,
+            -0.014478993521696027,
+            0.0034983131226364672,
+            0.000766228629514787,
+            0.009667268771502067,
+            -0.01558439890697323,
+            -0.04984718709870083,
+            0.029613955432821887,
+            -0.009686854114630624,
+            -0.015279319238124568,
+            0.007460629676325322,
+            -0.006235979350809034,
+            0.012514387473717085,
+            0.02800435263741497,
+            -0.008688925946672778,
+            -0.01236301543374469,
+            0.029755705020188972,
+            0.01158037219446542,
+            -0.005242392206024511,
+            0.009982679027728814,
+            0.052414361714483264,
+            -0.018788500381726227,
+            -0.020335831196027204,
+            -0.08583959943356621,
+            0.029309228216812346,
+            -0.0249285282010673,
+            -0.018929396555426307,
+            0.007710663193334574,
+            0.03096798035958051,
+            -0.017089977064591828,
+            -0.028015107605405817,
+            0.06740703271128606,
+            0.009398795412354399,
+            0.01118769022826835,
+            -0.03041417460855722,
+            0.014776002883150835,
+            0.004319194067673161,
+            0.022507636639041203,
+            0.0053562788646786845,
+            0.007548072258943124,
+            0.011123374308563128,
+            -0.006924362115585165,
+            -0.011118507147092024,
+            0.009563232643295304,
+            0.00657178473678069,
+            0.08313027447380966,
+            -0.004239370156003264,
+            -0.00930406163946907,
+            -0.039202672821898946,
+            0.009761029930556298,
+            -0.02827698465424993,
+            -0.037914533246900244,
+            -0.04496050911855815,
+            -0.017711307900236353,
+            0.01934435542287117,
+            0.004335173027873153,
+            0.03800509969714242,
+            0.031198127357716314,
+            0.031311207941680276,
+            -0.012184553894588813,
+            -0.058985863670856876,
+            0.041442017057707635,
+            -0.012456189292721711,
+            -0.0040966351720394135,
+            0.010849179519959421,
+            0.005243123049347636,
+            0.0011495799963862647,
+            0.003534183676934487,
+            -0.03139340512003631,
+            -0.023876261883905306,
+            0.008382957569752911,
+            -0.004778283217011976,
+            0.03703067589888386,
+            0.06888630688839406,
+            -0.03993532296478411,
+            -0.009472167678222372,
+            0.025246872942105057,
+            -0.01524567912700368,
+            0.01595917939878582,
+            -0.003736723164442467,
+            -0.030836797784967753,
+            0.0026246446163368533,
+            0.008894300082148884,
+            0.08342856279154155,
+            -0.0007476580521704578,
+            0.025149116238550588,
+            0.004972542633631468,
+            0.034495632546556576,
+            -0.009173329764686424,
+            0.014009381543772768,
+            0.004427261668151028,
+            -0.013426138607902904,
+            -0.022234568880321808,
+            0.01976282364689966,
+            -0.03592433789490043,
+            -0.010918278220831188,
+            0.013527180289641174,
+            -0.013023113052242677,
+            -0.008304770302495363,
+            0.003677063799167178,
+            0.0013981822663219421,
+            0.047001163242600175,
+            0.01828332714887986,
+            0.002483375662343689,
+            -0.06820498187523893,
+            0.05295312038995555,
+            0.025038753504342162,
+            0.025286870198436735,
+            -0.0029544808216316115,
+            0.029982752196635428,
+            0.0005600481523874767,
+            0.0128290258663539,
+            0.009023805227424919,
+            -0.0019219166367750596,
+            -0.04482843936478531,
+            0.04523479691409549,
+            -0.004678574027241186,
+            0.010486850804218904,
+            0.006736229865018014,
+            0.02134813656476443,
+            0.028083972673690036,
+            0.016857413440125445,
+            0.0189082447820146,
+            0.03843690613820022,
+            -0.011922027011229613,
+            0.016193433524453224,
+            0.03828207911467065,
+            -0.0013055128404331705,
+            0.043198163443700956,
+            0.024025562893260542,
+            -0.001257851946749882,
+            -0.011726713433348844,
+            -0.021907984099443396,
+            0.02752198420597673,
+            0.004431866336999851,
+            0.015798806714413912,
+            -0.012910782191382367,
+            -0.03180549180206364,
+            0.004550247372715741,
+            -0.022778272841118514,
+            0.031242218240017532,
+            -0.006106984949693483,
+            -0.006259993190497421,
+            0.02187449047033339,
+            0.009334146847221406,
+            -0.009381278960016145,
+            -0.020888853823931134,
+            -0.007502374908776156,
+            -0.010390018386728482,
+            -0.04316693435221676,
+            -0.006274513730720055,
+            0.019152443937064258,
+            -0.01668984824845791,
+            0.018573194036505578,
+            0.0035035262183249925,
+            -0.019779911381789077,
+            -0.0027495798165085993,
+            0.02425481236315913,
+            0.006857723545270935,
+            -0.014733535699087731,
+            -0.026326146471941727,
+            -0.019906660231056833,
+            -0.001033294559612807,
+            0.0159154063026895,
+            0.035649534442431415,
+            -0.026210592096240003,
+            0.014397127229517285,
+            0.005114311681443673,
+            -0.0033104236005207077,
+            0.00324988235167305,
+            0.03764004154553922,
+            -0.017822437760737605,
+            -0.008579701063708334,
+            -0.00032410984200965487,
+            0.018450351071676443,
+            0.017283379764846195,
+            0.006384411611332382,
+            -0.005115090309123598,
+            -0.030590047062583125,
+            -0.011776211359193503,
+            0.01183623914060493,
+            0.0016564847796208331,
+            -0.009530766634361575,
+            -0.004810598258154945,
+            0.03876242515526699,
+            0.027076785774401667,
+            0.005324763136165989,
+            -0.03416412480913939,
+            -0.007943016733586697,
+            0.002114307650456928,
+            -0.02961449347300628,
+            0.05142732770518828,
+            0.05566026433842563,
+            -0.0030729309664974704,
+            -0.00938186948393074,
+            -0.04547908981236149,
+            0.0651216359692231,
+            -0.01813722445944094,
+            -0.0482650853560993,
+            0.044045155892545554,
+            -0.014272653003441165,
+            0.013740652306746851,
+            0.04563732439184395,
+            0.02126513340521316,
+            -0.0054845632439384905,
+            -0.011635625101262898,
+            0.030992082069685376,
+            -0.010051935721068779,
+            -0.050967258971854425,
+            -0.009827942095033494,
+            0.013373878556654464,
+            -0.013017627321534534,
+            -0.009361001542607812,
+            -0.01925867665425919,
+            0.020329052915436022,
+            0.023020741670660188,
+            0.019387043947695172,
+            0.005628632318152202,
+            0.018451555161952665,
+            0.016226845400848455,
+            -0.01384033112836216,
+            0.004762589697426251,
+            -0.012718476561725138,
+            -0.024222765127494463,
+            -0.037487144314010595,
+            -0.019971524166659278,
+            -0.015034588610516958,
+            0.009483936496215338,
+            -0.014301924283553591,
+            -0.0024139665139412125,
+            0.013361045353529458,
+            -0.0005870798721107379,
+            -0.011192319340879789,
+            0.013927695434391159,
+            0.021337539669342293,
+            -0.027300952495815437,
+            -0.04765670695199488,
+            -0.00561706224251562,
+            0.0082776639906139,
+            -0.0030984973749633724,
+            -0.026269016057688548,
+            0.007040524859505573,
+            0.0017859008072463442,
+            0.020091542517503808,
+            -0.018716761130867725,
+            0.0035283051960672963,
+            0.008519383548064053,
+            -0.015247417050336866,
+            -0.016917626868166634,
+            -0.018580550690103854,
+            -0.024697180580294884,
+            0.034202028316737125,
+            0.0204395003463227,
+            0.030392895534267717,
+            -0.04527669147753553,
+            -0.018359428092075348,
+            0.05494107499558624,
+            0.015234754907198422,
+            -0.05066008106884831,
+            0.031139824660661085,
+            0.005016615710206006,
+            0.0019000453624929544,
+            -0.006094364589229031,
+            0.0637324238017742,
+            -0.02711381489292883,
+            -0.02187418906973867,
+            0.004601744864532875,
+            -0.04141121277496072,
+            -0.026689585608019588,
+            -0.005568100750837504,
+            0.01291019036002459,
+            -0.031853041426355644,
+            -0.02815800259372657,
+            -0.007964861641790374,
+            -0.010933515363060793,
+            0.013726639077745378,
+            -0.023956800669647334,
+            -0.015672058590455337,
+            -0.023642025374517057,
+            -0.01288435863544158,
+            -0.0042516881552192385,
+            -0.011162214160460695,
+            -0.02412733256710621,
+            0.010088528028211243,
+            0.03310175458095482,
+            0.023064360903748363,
+            0.001996159684818467,
+            0.009478229572933101,
+            0.014478772877286076,
+            0.00371783700962415,
+            0.007864103610320116,
+            -0.048511515124609,
+            -0.031398754206534776,
+            0.020227872561659397,
+            0.012159811587094092,
+            0.0172163448580163,
+            0.009578952796095119,
+            -0.04154545961112031,
+            -0.05095080393682759,
+            0.02968486144855133,
+            -0.03932395023720674,
+            -0.02063219739061335,
+            0.007512423249865194,
+            -0.011911709468952548,
+            -0.00810951734088105,
+            -0.0008079327452253181,
+            -0.01600701450676789,
+            0.023984909975645632,
+            -0.02418072541081482,
+            0.0033984265728439398,
+            -0.0104343782811971,
+            0.01588977364765743,
+            0.05109529246167992,
+            -0.027800269136366997,
+            -0.027889034481533438,
+            0.03706813210665186,
+            -0.02058120652429308,
+            0.030801396468451574,
+            0.03246127847621613,
+            0.008355517405358826,
+            -0.008605467586247932,
+            0.017693344563968573,
+            0.014499487024471321,
+            0.012431277080478217,
+            -0.040103870071440256,
+            0.017328848567062113,
+            0.02723900253727976,
+            -0.054909074068976875,
+            -0.008547596270650966,
+            -0.00777964838164648,
+            0.027134666572242665,
+            0.02765036594970255,
+            -0.002207173084941375,
+            -0.016838740117653725,
+            -0.028289233739722936,
+            -0.019433376533829132,
+            -0.011053380458946764,
+            0.03247641283550457,
+            0.0007514989009289282,
+            0.007919574219000797,
+            0.001316042436488591,
+            -0.0017934342041762042,
+            0.01924145734197619,
+            0.015260199553572459,
+            0.048259602293134723,
+            -0.03383050622747848,
+            0.0021712225589426266,
+            -0.03508567715696004,
+            -0.04578731401250471,
+            -0.008768228935292693,
+            0.010203318819548585,
+            0.024135236371743855,
+            -0.007467140460133838,
+            0.0028907962385387818,
+            -0.007582083854422456,
+            -0.005384477718912037,
+            -0.01867978482547249,
+            0.01641191108381631,
+            0.037168089880543244,
+            0.0031992495663980464,
+            -0.008533890994695162,
+            0.000778843942291118,
+            -0.004620258538699103,
+            0.010572690335163528,
+            0.033233014627100904,
+            -0.030497362482602697,
+            0.02974571152880676,
+            -0.004274442492343003,
+            0.025341347125910928,
+            -0.04307748954707389,
+            -0.018069609916024403,
+            -0.018491122727274337,
+            -0.02069307802429807,
+            0.05372967226264594,
+            -0.030768978301976772,
+            -0.03662045886993282,
+            -0.003205991669059688,
+            0.016815511161004534,
+            -0.0014152644009276566,
+            -0.0016015439520090104,
+            -0.03984716970293867,
+            -0.010828260334550217,
+            -0.05894985435251326,
+            -0.01638035185618134,
+            -0.021819442096404948,
+            -0.005315102328745982,
+            -0.06251579337157112,
+            0.0033713876515840233,
+            -0.07815974854883057,
+            0.001484860071649606,
+            0.04539427129131382,
+            -0.023530943197353476,
+            0.05142250021536681,
+            -0.024746600229063383,
+            -0.0005885740455257141,
+            -0.044698536284421976,
+            0.0026481001040031785,
+            0.02389178603305878,
+            0.051387657872255674,
+            -0.002949198236157656,
+            0.010674377196609455,
+            0.013322075519401089,
+            -0.026245090721950647,
+            -0.01036724558050635,
+            -0.005321884180696542,
+            0.005958798671856245,
+            0.013755178085972732,
+            -0.0074734329537091675,
+            0.038368016046563674,
+            0.010941421833174982,
+            0.03443115781919206,
+            0.034623617136788874,
+            -0.004361926531008214,
+            -0.03348681614127998,
+            -0.00632870804659212,
+            0.024433331840566692,
+            0.0321998807702602,
+            -0.02899983280320611,
+            0.005958471722097962,
+            -0.015602447981516634,
+            0.015633217539747047,
+            0.006321931116948646,
+            -0.011827541320878442,
+            0.02372204564999124,
+            0.015673426811623124,
+            0.006103300725621424,
+            -0.005165006670545588,
+            0.013589526056189892,
+            -0.025411864292353578,
+            0.03041791947375232,
+            -0.004456865298584878,
+            -0.04276058054583262,
+            -0.0024736078733106825,
+            -0.020059309867945227,
+            0.024883721794000956,
+            0.015578237413840115,
+            -0.00536523508503184,
+            -0.005290645798641171,
+            0.008242987948511412,
+            0.020469675635821673,
+            0.006668354647245447,
+            0.00017729517736814364,
+            0.02373604785121342,
+            0.029915687122900485,
+            -0.02432635861789771,
+            -0.029540533943782886,
+            0.008151297444985344,
+            -0.005730733861939129,
+            0.011318816579801282,
+            -0.02644855229221747,
+            0.00033932872948409137,
+            0.0035505684154028996,
+            -0.03570104343808711,
+            0.03408366016010711,
+            -0.009199700348630527,
+            0.005857598530788056,
+            -0.009538177736472163,
+            -0.0017768806148210118,
+            0.0027410153099351694,
+            0.03898108970435115,
+            -0.007367623392751247,
+            0.009827254226889936,
+            0.015465895035342481,
+            0.000981604396724216,
+            0.05143083558797524,
+            -0.027350877485564314,
+            -0.02183036962301852,
+            0.01885468378168636,
+            -0.007285239007489837,
+            -0.016846837665812908,
+            -0.020546383007536074,
+            -0.011863943814684079,
+            -0.016293192750336218,
+            0.01976516340568883,
+            -0.02387328798681798,
+            -0.011892336568452852,
+            0.007170651391639504,
+            0.0013725014050721622,
+            -0.009809669068563384,
+            0.0016617183704652932,
+            -0.006307590349798603,
+            0.02037693835615797,
+            0.031753656615094446,
+            -0.024892675105483313,
+            0.03577919637821846,
+            0.03178303188315581,
+            0.017260898395251155,
+            -0.02102289690701718,
+            -0.009661035772073294,
+            -0.00048542689442823136,
+            -0.014694631967051945,
+            -0.0326379186158631,
+            0.025443206465204822,
+            0.017194626456982255,
+            -0.011381421003142453,
+            -0.03680783761649673,
+            0.0017216115811536721,
+            0.022819859205829785,
+            -0.013600989058799166,
+            0.028141657788496834,
+            -0.01869047298692401,
+            -0.01366833755869951,
+            -0.00782959271979894,
+            0.031227914441058244,
+            -0.05252392878125891,
+            0.016957842026348195,
+            0.00020880696574616416,
+            0.001154370575234767,
+            0.010892067708194324,
+            -0.008368819756628046,
+            0.012224564683325483,
+            0.031700389750709,
+            0.018063405869147564,
+            0.0010794115307202266,
+            -0.002783668216641561,
+            0.01418234057248712,
+            0.017773269840982066,
+            -0.015653443355569053,
+            -0.000497740485657824,
+            0.019890882604100452,
+            0.022672054781919238,
+            -0.00521797370595221,
+            -0.016686942830774276,
+            -0.018443538351123584,
+            0.025270057639201107,
+            -0.010354431070738671,
+            0.03484047753830186,
+            0.04594785214094975,
+            0.0036441959528283004,
+            -0.05695476763071743,
+            -0.009412209619190693,
+            0.0010371110229966086,
+            -0.01099343927053564,
+            0.01606863921935641,
+            0.012470660511773616,
+            -0.03237187923799263,
+            0.006388672213167589,
+            0.012946570716567364,
+            0.022152554488829627,
+            -0.02540317621005867,
+            -0.023183252921885226,
+            -0.0030984019855787655,
+            -0.007890467768631811,
+            0.018648760402624687,
+            -0.008653405980661855,
+            0.0006228315253506687,
+            0.009672526897422602,
+            0.0005436494014569835,
+            0.011632326145022935,
+            0.005655942007108228,
+            0.021736827172951,
+            -0.001256608167230203,
+            0.013124413886287987,
+            -0.01986808920858981,
+            -0.0270450339809516,
+            0.006500423802582728,
+            0.04438242119211055,
+            -0.014813399694814117,
+            0.030621709833336064,
+            0.0092171224268983,
+            0.016433384739642534,
+            0.024875072913970443,
+            -0.00891470467241686,
+            -0.014124009014664287,
+            0.00845777541223868,
+            0.009019007848927453,
+            -0.04147358041382793,
+            0.012513030150663913,
+            -0.01710243399313324,
+            -0.033144939719206276,
+            0.01904674924133127,
+            -0.00534122554342279,
+            0.012367320751926314,
+            0.004270315800911569,
+            -0.002886460711688153,
+            -0.025696286516581048,
+            0.014137901708875033,
+            0.028135087538950946,
+            0.021797869775098042,
+            0.0296173052503884,
+            -0.012757078368361444,
+            0.0005460601780756187,
+            -0.008748930817524899,
+            0.005470975782187157,
+            0.04187083128854477,
+            0.0008687182830616019,
+            -0.0008032802520818105,
+            0.01507726552487313,
+            0.024945744517309208,
+            0.0376130540370386,
+            -0.015415560911023553,
+            -0.03407353556457147,
+            0.01564969972264396,
+            -0.010104443707865262,
+            0.018855792875436674,
+            0.005503259932317811,
+            0.0049634097823351596,
+            0.0038600156268000702,
+            0.019312419743287703,
+            -0.006635193789316492,
+            0.018798045806429395,
+            -0.02500983683871332,
+            -0.028785757874583755,
+            0.01829411751138161,
+            -0.005277972443258226,
+            0.03338623299668019,
+            0.016225885095995672,
+            -0.029522280454919586,
+            -0.01712097344863975,
+            0.01553706187924322,
+            -0.008465766956343768,
+            -0.024842700926275538,
+            0.01895228828512998,
+            0.007076862934523977,
+            0.008956105593078965,
+            0.0024373779389833218,
+            -0.01106503696703573,
+            0.01125466547367417,
+            -0.000818223040067914,
+            -0.015322790791698395,
+            0.014157131555941659,
+            -0.017853386731281715,
+            -0.014994640991761465,
+            0.01775991780643474,
+            -0.037498078532902435,
+            0.009570197712419421,
+            -0.007687297543935123,
+            0.009050494195378254,
+            -0.004222041469820445,
+            0.007460383234429904,
+            -0.0010804379186060264,
+            -0.014987857934522812,
+            -0.014903749199810126,
+            0.03228485430019625,
+            0.022096121825475306,
+            -0.000410816936759558,
+            0.006728733918931589,
+            -3.971636461500232e-05,
+            -0.026719848032785824,
+            -0.011253751790407687,
+            0.02245467425830765,
+            -0.005664635254398481,
+            0.005134496095445943,
+            -0.01109590855846966,
+            -0.0149616794609307,
+            0.0019726017255571673,
+            0.032668637270108894,
+            0.06066635618119076,
+            0.012001732730659747,
+            0.024422093585231484,
+            0.0189499396761026,
+            0.02119276033547528,
+            -0.008489818917377516,
+            0.006413901834269739,
+            -0.03097413287789443,
+            0.03509225573986143,
+            0.005392594635384506,
+            0.03108937318306421,
+            0.026664331248813712,
+            -0.04722525921137681,
+            -0.0058539171830594774,
+            0.017087531454837677,
+            0.022673712717885323,
+            0.0159020681868943,
+            0.0008382498574190522,
+            -0.008938174268517353,
+            0.035650940044285984,
+            -0.03315250299348084,
+            -0.023647111263155,
+            0.005059382024092534,
+            -0.01439537467317451,
+            -0.010488359956731476,
+            -0.006785295580877329,
+            0.02776449681317373,
+            -0.005584052419598926,
+            -0.004582019433008779,
+            0.009697064252638535,
+            -0.014400206558190682,
+            0.008165676294736836,
+            0.0007624778531368436,
+            0.008681263897172326,
+            0.039249866981334126,
+            0.0005918063050375113,
+            0.024275699182767528,
+            -0.004138313072843555,
+            0.011079529739650594,
+            -0.0158972266904151,
+            -0.016985058543605,
+            0.02276982363172991,
+            0.00351101120496393,
+            -0.03244126813432563,
+            -0.00768819222024939,
+            0.00928180979884261,
+            -0.0013793404539641824,
+            0.005266058566444064,
+            -0.00503433670609399,
+            0.003747448261046857,
+            0.03286230666764075,
+            0.0022383592721685153,
+            0.008606190020480513,
+            0.013593947677257625,
+            0.00030951915677929204,
+            -0.01490307680740127,
+            0.013254591138113545,
+            -2.1379308136707722e-05,
+            -0.012900073369514622,
+            0.03357519186606058,
+            0.006705902094864628,
+            0.04243347200485736,
+            -0.03302953674711988,
+            0.009031534935013593,
+            0.01822227379789576,
+            -0.008491770335368761,
+            -0.02725412856390648,
+            -0.03630788717786838,
+            0.008259145747740535,
+            -0.019003260334340084,
+            0.015781275447792516,
+            -0.010039277287105153,
+            0.029665124310477384,
+            -0.03563370794521629,
+            0.004714858110268169,
+            0.004053913355055313,
+            0.0009503673086552651,
+            0.02068360255395961,
+            0.018524093113914366,
+            -0.002785698934204024,
+            -0.031443835341186485,
+            -0.02066365433385339,
+            0.006869321907853112,
+            0.02431260052207595,
+            -0.010367048093621498,
+            -0.005612149485859478,
+            0.0006316058139530722,
+            -0.024369562712568606,
+            -0.016845124984022828,
+            0.030359419843284823,
+            -0.040974427176153246,
+            0.005252911449822132,
+            0.04272936053727608,
+            -0.04599092715016919,
+            -0.03802288716943514,
+            -0.002314988728575789,
+            -0.024124014894379466,
+            -0.006360771882090754,
+            -0.002068030981218464,
+            0.016669991113175926,
+            0.0036232707799735537,
+            -0.028763764013482795,
+            0.00943401978066451,
+            -0.005173988601101547,
+            0.003971420424839974,
+            -0.0031669592494977393,
+            -0.02216336363004636,
+            -0.01657772032717298,
+            -0.02531962163186398,
+            -0.035174102848865116,
+            -0.0015903434876311088,
+            -0.005158703150821607,
+            -0.028703075279214722,
+            -0.03941940849540492,
+            -0.021696000925262537,
+            -0.0160221488026189,
+            -0.00843213135542359,
+            0.040641882367348774,
+            0.021319060666253945,
+            0.019336844555345886,
+            0.0025252167889382917,
+            -0.020346900609125973,
+            0.049440028814233025,
+            -0.007642182460707816,
+            0.035628691886108926,
+            -0.0017882888000877158,
+            -0.013461669667950893,
+            0.00455558389906464,
+            0.03825428026120858,
+            -0.014248434654950971,
+            -0.0009902062564027718,
+            0.00027811655483232545,
+            -0.0011866253757890136,
+            0.0008535155950871286,
+            0.015433650396777696,
+            0.001307651096343185,
+            -0.007112199543314309,
+            0.030380673552683906,
+            -0.012803247194895328,
+            0.01637501752192592,
+            0.018350814869962304,
+            -0.0013359719137099667,
+            0.008064546833360553,
+            -0.021430659268086814,
+            -0.00731572118780415,
+            0.0044578576331236325,
+            0.008422210247967642,
+            -0.015096885054883586,
+            -0.009964588397149773,
+            -0.02359494467299634,
+            -0.01174145797396426,
+            0.0023855749166744366,
+            -0.004559202896782688,
+            0.004902138778592298,
+            0.034595817216889944,
+            -0.016068744500427584,
+            -0.035106033294403614,
+            0.009997207702122002,
+            -0.02663088875167486,
+            0.02222306406130687,
+            3.983384942889326e-05,
+            0.006838221769859366,
+            -0.0023434342406307616,
+            0.001765445386886839,
+            -0.00169484278667461,
+            0.002862331185439823,
+            -0.018217975204395134,
+            0.0031859734246840257,
+            -0.0032119988065673565,
+            0.0009052950782312657,
+            -0.0012459653052875638,
+            0.00687339892509406,
+            0.006332799551164718,
+            -0.0008019538898592668,
+            -0.002968451740267742,
+            -8.541086355097327e-06,
+            -0.020561816931299912,
+            -0.021642512405352372,
+            -0.028259049261410905,
+            -0.000989887278779396,
+            -0.012359733017834338,
+            -0.027804394906433664,
+            0.011443385424045703,
+            -0.006872684999639775,
+            0.008327844771213648,
+            -0.008969527138034904,
+            0.006219994886600349,
+            -0.008415106130309062,
+            -0.0004143996206881767,
+            0.013119286511794273,
+            -0.02925209282917943,
+            -0.02096907906398536,
+            -0.03096513357712367,
+            0.01762223841790055,
+            -0.00925741670612824,
+            -0.002656382260603827,
+            0.0025223527960181835,
+            0.005092659166535706,
+            -0.033606542487570154,
+            -0.006653438998497412,
+            0.007313682256676618,
+            0.009547290751154595,
+            -0.03287788897629887,
+            0.02446728061355753,
+            0.017023222076526946,
+            0.01874233066565012,
+            -0.02417566312664911,
+            0.04049827067897005,
+            0.014426879325647042,
+            -0.021740606525480655,
+            0.016712907564723732,
+            0.027385150577780248,
+            0.03177163221499527,
+            -0.03469098547555545,
+            0.004481376248622418,
+            -0.01518200063132508,
+            0.018918680020516044,
+            -0.025284921244460287,
+            0.020625170445627613,
+            -0.016707629602243063,
+            0.021278719363178687,
+            0.03520334001264928,
+            -0.005524369131116028,
+            0.03608133014180745,
+            -0.019234548780647943,
+            0.0059985801736444555,
+            0.004041778461022664,
+            -0.01645503683802124,
+            0.01024694092614158,
+            -0.0042700351372843386,
+            0.0006059159967286885,
+            0.009406090340935747,
+            0.019133229156115282,
+            0.019007219188041775,
+            -0.04442208547053466,
+            0.028588965123434923,
+            0.00804120101594811,
+            0.021472086070876265,
+            0.014177820346835521,
+            -0.02708807970982664,
+            0.004742981711172693,
+            0.002072673190279628,
+            -0.004659713164250794,
+            0.0061167904730676514,
+            -0.013796604518930328,
+            -0.034540857808309226,
+            0.006338792319814448,
+            -0.018918703778187926,
+            0.015036937364271984,
+            -0.009026670935442384,
+            0.0011667525636770179,
+            -0.01152652811400482,
+            -0.011938679695598347,
+            0.0006170592505446954,
+            -0.005495151258143897,
+            -0.0008075736908949407,
+            -0.03171721592827769,
+            -0.04427658539412448,
+            0.02036132271908089,
+            -0.0021221965958423804,
+            -0.04320396743189005,
+            0.004111179168317842,
+            -0.025352428814134187,
+            -0.01865918186335911,
+            -0.011237699157960728,
+            -0.025047923887906474,
+            -0.04219630696740606,
+            -0.011108496740166623,
+            0.007785006066683012,
+            0.016810642610918525,
+            -0.009081815651299525,
+            0.040965574143584864,
+            0.0073944364393447965,
+            0.015313576815177263,
+            -0.017731360025694955,
+            0.026137777790243826,
+            -0.019314392060123,
+            -0.03108486330987073,
+            0.02676829566273746,
+            -6.259161089416372e-05,
+            0.003053748984540415,
+            0.014358356168574803,
+            2.8186924071913628e-05,
+            -0.006181789640967465,
+            0.01947730374003864,
+            -0.028739300217145797,
+            -0.023460503178221397,
+            0.0012460072421725688,
+            -0.008850395570726385,
+            -0.016271242375881602,
+            -0.035896509136022194,
+            0.005227881053956234,
+            0.03324337105109496,
+            0.004970419987430122,
+            0.008273497228454482,
+            0.01814500594735864,
+            -0.02229593571615308,
+            -0.024230675455380678,
+            0.000639112529814716,
+            -0.010266640429960994,
+            -0.010257247303470066,
+            -0.021302483296731352,
+            0.019150757066356872,
+            0.01651375496287087,
+            -0.00390189952706486,
+            -0.006971692709239087,
+            -0.010621003777733253,
+            -0.01918206725910585,
+            -0.02020345113211036,
+            -0.013827322378214754,
+            -0.017980206764311026,
+            -0.034153741666571154,
+            -0.032195001867229935,
+            0.03225205339201753,
+            0.026354817588919596,
+            0.0014076409241492998,
+            -0.007959081015805435,
+            -0.023070419006792155,
+            -0.004483328183581872,
+            0.02879516595345542,
+            0.002549934782150634,
+            -0.0001541849716285838,
+            -0.00830090759167373,
+            0.015341095409025125,
+            0.00835695019244165,
+            0.005107844984733648,
+            -0.06676974053142512,
+            -0.006027540973808174,
+            0.01655283585235268,
+            0.01351155740684989,
+            -0.002359114145814062,
+            0.02591301169926259,
+            0.01108099264396926,
+            -0.015361038608114896,
+            -0.019765743462262668,
+            0.035195761515422364,
+            0.02408910733949848,
+            0.023468767772473746,
+            -0.0034492999385544727,
+            -0.004472311490618405,
+            0.007030833258457338,
+            -0.010684252427867153,
+            -0.013326964647936429,
+            -0.004558513842504658,
+            0.026925146859337397,
+            0.016099443021514896,
+            -0.0064331198280825575,
+            0.010848931851939124,
+            -0.0008320209125061849,
+            -0.019444491995041537,
+            -0.001841330829156697,
+            0.008730275423516445,
+            0.0012974082030866743,
+            -0.014611902493400719,
+            -0.005911807768916134,
+            -0.019130604101278263,
+            0.024719468317569562,
+            0.0209456329723249,
+            0.03381727436368338,
+            -0.013237902591972769,
+            -0.0003695142331921278,
+            0.005410360755058108,
+            0.0002841632253252816,
+            0.0023352645111704958,
+            0.004801123870325242,
+            0.02623649813035263,
+            -0.029010710171522985,
+            -0.020741774066460415,
+            0.0068426980682519505,
+            0.013445270043414486,
+            0.0022744032984779294,
+            0.033840485942696216,
+            0.035040080489222324,
+            0.004166066423508316,
+            -0.010046666684988869,
+            -0.006148765564658713,
+            0.007071740742464309,
+            0.01865897335451031,
+            -0.023267411753900076,
+            -0.0005936657905496225,
+            0.004833030519903764,
+            -0.004229582648167378,
+            0.03620610659071198,
+            -0.009961238880412731,
+            -0.007197670122096852,
+            0.022212791947347127,
+            0.09819205436926473,
+            0.00047695621444943017,
+            0.0025354521225913395,
+            -0.029358210234742176,
+            -0.023614016211567963,
+            -0.0012564198366457442,
+            0.00242680708457405,
+            0.00830167274308547,
+            0.028076260517106013,
+            0.0004852400049405906,
+            0.011579274037419117,
+            0.02773972145320728,
+            -0.017217305534161797,
+            0.006752070301141707,
+            -0.011376051474975524,
+            0.024577071515205502,
+            -0.012694034026935636,
+            -0.0019809719792138673,
+            -0.016512649783050146,
+            0.015907402258634816,
+            0.004556574687833768,
+            0.004907510649166062,
+            0.011347630984408937,
+            0.009824484201337644,
+            -0.02650731900265866,
+            0.01756773113723491,
+            -0.005671986127099093,
+            0.004449611508541289,
+            -0.027841346572384462,
+            -0.020124233970502267,
+            0.0025986391930124847,
+            -0.0014440758308989809,
+            -0.009579444248630085,
+            0.006005274872617889,
+            -0.0024984235450999127,
+            0.03765664613203687,
+            -0.03765496603611941,
+            0.01409350051861866,
+            0.0066453175608372555,
+            -0.0015390841089599882,
+            0.011806653942640485,
+            0.017306914776093522,
+            -0.008206256939290151,
+            0.02589800448884443,
+            -0.00798213351353373,
+            0.009846274212000523,
+            0.015230070803055296,
+            -0.03449531102975745,
+            0.009174739184806312,
+            0.010145762060174144,
+            -0.024165660737765746,
+            0.012146877603301336,
+            -0.013784371016298891,
+            -0.004818759788054113,
+            0.005930446700613062,
+            0.007349837554764034,
+            0.010733731974181452,
+            -0.02295571464086674,
+            0.016176424826629898,
+            -0.010947841926210105,
+            0.0037766665765835445,
+            0.013559846998199008,
+            0.024892849808979323,
+            0.0012426907454129122,
+            0.0005950515136086193,
+            -0.01379469431504576,
+            0.005534724429523084,
+            -0.015667467221535825,
+            -0.022401586867694584,
+            -0.014192041775234079,
+            -0.006366846766730035,
+            0.01029598687444803,
+            -0.003972386708834858,
+            0.016971476088808737,
+            -0.010309648640697812,
+            -0.010645860303528099,
+            0.0037818310327362103,
+            0.01040806357838676,
+            0.008316605962234849,
+            -0.02855914079113881,
+            -0.005676310556150579,
+            0.004961084138163161,
+            0.00029285971503001265,
+            0.009592497954559435,
+            -0.014556073011576487,
+            0.013689526487160009,
+            -0.013290113354287095,
+            -0.0010726537498214612,
+            0.013201860397060236,
+            -0.012443552043930505,
+            -0.004012305744087929,
+            -0.01592905426965635,
+            -0.028487066798154617,
+            0.008462143660968396,
+            -0.01365614159523485,
+            0.001076882584445294,
+            0.0019503240125505646,
+            0.004730085646042709,
+            -0.016919069570642004,
+            -0.007860046623166481,
+            -0.033515448866521046,
+            0.011065416478687504,
+            0.02314589687882036,
+            -0.006435733825418645,
+            -0.030351727858789472,
+            -0.009546333526852862,
+            0.011327489786114963,
+            -0.0071765122743172715,
+            -0.01676736383051139,
+            -0.0029390038145944096,
+            0.007617910650439855,
+            -0.008174620754097493,
+            -0.00508943241317381,
+            -0.0025173969010414098,
+            0.0217328037301181,
+            0.016094975605203832,
+            0.01769642142712967,
+            -0.021400862603034036,
+            -0.00976992221747187,
+            0.026043879119776864,
+            -0.03275550610951898,
+            0.012441465975914529,
+            0.024587446266013183,
+            0.003446527051428522,
+            0.005492228054838714,
+            0.0030377554109693714,
+            0.005459159163772825,
+            -0.006985179226831048,
+            0.019925583798328282,
+            0.010870421035240845,
+            -0.019709062313471467,
+            0.005385443078244229,
+            0.02086085539375208,
+            0.0037248737640193097,
+            0.012892299156840998,
+            -0.017546870090541213,
+            -0.0020341807891543716,
+            0.015594779287017958,
+            0.04816214672670215,
+            -0.0032851797178091685,
+            -0.010450846098750195,
+            0.029513742590677183,
+            0.023526004023637855,
+            0.005373455940309873,
+            0.0012120777808635214,
+            -0.02525630456877349,
+            -0.013075130701774785,
+            0.01962235086017781,
+            0.016944870494393874,
+            0.013484925533584295,
+            -0.0023255656480479004,
+            -0.019236081983457366,
+            0.021520156296467717,
+            -0.01701733319731161,
+            0.012422659010697477,
+            -0.023764941037007197,
+            -0.001490790833263327,
+            -0.0035419008954878724,
+            0.021998949056452454,
+            0.0021819635638621092,
+            0.008891505304514552,
+            0.024354552127458608,
+            -0.006978548952551799,
+            -0.019526615985982355,
+            -0.0051799020400150305,
+            -0.014578953088901798,
+            0.022506635912517146,
+            -0.024117708614271965,
+            -0.007008641019892289,
+            0.011379795968360019,
+            -0.012362093847309038,
+            -0.0102534051092702,
+            0.001164934138876502,
+            0.017686807394319725,
+            0.0009423993435129515,
+            -0.014784508280754078,
+            0.01944445593807992,
+            -0.0011359271135469456,
+            0.010410915597949927,
+            -0.01921419010977189,
+            0.009390638453648025,
+            -0.016682442516992777,
+            -0.007849184048209236,
+            0.010343008873053859,
+            0.0012604521199837046,
+            0.003948300703645506,
+            0.002592490121754231,
+            0.007773437307969315,
+            -0.03355658426684504,
+            0.022695505487314516,
+            -0.010751606023585792,
+            -0.003538523688786008,
+            -0.0014125330167969218,
+            -0.01678692701672328,
+            -0.0006918162525881059,
+            0.005816502852598842,
+            0.010878358093847386,
+            -0.016045147059003256,
+            -0.00873270513444376,
+            -0.04920244395446931,
+            -0.030314009649638693,
+            0.023156333024481057,
+            0.013733657599720879,
+            -0.010971506342695331,
+            0.018852159415605117,
+            -0.023760853159532587,
+            -0.010587249404421732,
+            0.0011379869499042905,
+            0.023319856329056813,
+            -0.03539290744576506,
+            0.01215369703816749,
+            -0.014493682360886549,
+            -0.02039288669534806,
+            0.009070984596830142,
+            0.011445354735283254,
+            0.0007259852045513916,
+            -0.0011514627555568814,
+            -0.029350945123925453,
+            -0.010246845821727087,
+            0.02391068177425878,
+            -0.00861718589055971,
+            -0.004723044986826333,
+            0.024716520302409263,
+            0.0033721801437248873,
+            -0.012524663490691517,
+            -0.017955758068233754,
+            0.021612671655463236,
+            -0.026077203994996058,
+            0.007645944309444457,
+            0.010032183011253845,
+            -0.0173991263874487,
+            0.021649599230588117,
+            -0.007529701797485131,
+            0.020327763440441377,
+            0.004227390938169872,
+            -0.011249233530251522,
+            0.020352283974471443,
+            0.013644443758970054,
+            0.004216958800094986,
+            -0.004081669404784771,
+            0.006869353098014179,
+            0.02212732880780841,
+            -0.011021739614266063,
+            0.02219580717345023,
+            0.028777213295471067,
+            0.00017328191188631918,
+            0.03260179554743322,
+            -0.021287655520443285,
+            -0.03159728983004563,
+            0.021740163175540124,
+            -0.01035146858119169,
+            -0.021564906188712388,
+            0.028914245422221707,
+            -0.013674731836508768,
+            -0.00786476052825481,
+            -0.0166718429993922,
+            0.00533027311342322,
+            0.016680336311088843,
+            -0.005246190451159961,
+            -0.014360934066628332,
+            0.0029628103834216345,
+            0.019809487418628386,
+            0.014587234544007958,
+            0.029780009648869975,
+            -0.010778199815920887,
+            0.035180242185168895,
+            -0.002320576872571577,
+            0.008990157317473909,
+            0.012415465232073404,
+            0.006489435152972723,
+            -0.011251531176923445,
+            0.013501894212117511,
+            -0.003446759011266526,
+            0.008230548711588447,
+            -0.001449023474280409,
+            -0.035111679577202855,
+            -0.013709382199848678,
+            -0.04709294504969336,
+            -0.0023752976043235697,
+            -0.0025098360125255215,
+            0.0030975417740913154,
+            0.09736815869389318,
+            -0.005723049647039619,
+            -0.011988817571623457,
+            0.001625059666070763,
+            0.009611697638805556,
+            -0.0009896494420056814,
+            -0.007606836952841873,
+            0.015973152482488318,
+            -0.01343292635775299,
+            -0.02140928016038583,
+            -0.010532701190806147,
+            -0.01288594454368977,
+            0.012079198190522715,
+            0.015929716343161652,
+            -0.0005947359502228821,
+            -0.015207331663761012,
+            0.006524010655124294,
+            -0.004566385899752684,
+            0.0064717704997400245,
+            0.008647099975176308,
+            -0.010414668624343924,
+            -0.0071698879102925555,
+            0.030899143952119196,
+            -0.025301269549419674,
+            -0.024361748354052457,
+            -0.0010579931222861095,
+            -0.009055796737486008,
+            -0.010335459737012047,
+            -0.002692653633719985,
+            -0.0017679501322462056,
+            -0.010519994985692081,
+            -0.010019865263564572,
+            0.006537076627706128,
+            -0.015472130358273033,
+            0.004812857776601667,
+            -0.0047457011269495634,
+            -0.013296069496526007,
+            0.017054968086538724,
+            0.022563785357424853,
+            -0.02841491200807702,
+            0.0006755428946168761,
+            -0.017422510783648978,
+            0.012941211381006863,
+            0.008866069010589792,
+            0.006049618782357251,
+            -0.022451528209097505,
+            -0.023622906582403243,
+            -0.015019272754405791,
+            -0.01927535750568903,
+            -0.024973490129133732,
+            -0.009073381957684092,
+            0.0067692803507600945,
+            -0.001267339788153158,
+            0.021918281111033645,
+            0.000816538714279555,
+            -0.01812545613159305,
+            0.0061084913570894905,
+            0.000568371220952269,
+            0.018292497018668842,
+            -0.027468677288882186,
+            0.0033806422195878524,
+            -0.02307582511848465,
+            0.033508234091571124,
+            0.0039195050746594975,
+            -0.026680111514794824,
+            -0.021558668725755275,
+            -0.003311830287529699,
+            0.018994577319066473,
+            -0.0025745912019675805,
+            -0.00724049447539589,
+            0.014775207617625293,
+            0.0155334760323838,
+            -0.004693118176196483,
+            0.002015625406296403,
+            0.027968079002437102,
+            0.01077998866180723,
+            0.0313892866933904,
+            -0.015236169297590675,
+            0.004945828495617267,
+            -0.027215016713379062,
+            0.010214419950212732,
+            -0.01503548137805401,
+            0.008859554812646242,
+            -0.01290890682557532,
+            0.03531384210182567,
+            -0.012005613861257432,
+            -0.004250698950192301,
+            0.03253288796441852,
+            -0.029150986393345252,
+            -0.0035020051536906334,
+            -0.009255437891304041,
+            -0.004142204892160768,
+            0.013240580426083011,
+            -0.001198093874819309,
+            0.008485141356876356,
+            -0.01854370462655459,
+            9.726096864073609e-05,
+            -0.003221852304410318,
+            0.005404595442127598,
+            -0.02005458688172967,
+            -0.003314118809778734,
+            0.0073427568419969814,
+            0.004259518237708634,
+            0.002408630356035373,
+            0.016420646142725346,
+            0.006662121128241274,
+            -0.0029357701493526907,
+            0.03424224243607692,
+            0.008171895850772245,
+            0.0152521424877615,
+            -0.0018638264193202728,
+            0.0030234211261443034,
+            0.019287963516760636,
+            0.019912956661192856,
+            -0.00048443399729935116,
+            -0.01058935153883094,
+            0.00195113175754497,
+            -0.001140732137433033,
+            0.011048741602327437,
+            0.024396046054912645,
+            -0.021158025221644738,
+            -0.02419562092713724,
+            0.016377661005279716,
+            -0.015076145951740961,
+            0.01903712233378994,
+            0.010541463112889985,
+            -0.010099351302095792,
+            -0.00031558616108876247,
+            -0.042316016201379245,
+            0.010964989031040692,
+            -0.02068939391415103,
+            -0.04004120028915868,
+            -0.00244383573783919,
+            0.0009542507389088937,
+            -0.029439456243505052,
+            0.02822814855547978,
+            0.0023755032182188156,
+            -0.017280221740813364,
+            0.018223709137876153,
+            -0.005548654680726505,
+            -0.0024040301239229064,
+            0.005397711787238679,
+            -0.01092026090471121,
+            -0.010648485086769927,
+            0.008235250252660972,
+            0.010957715075632535,
+            0.005425683656535705,
+            -0.005698919020608095,
+            -0.011284911899894062,
+            0.009975882651638084,
+            0.012318901214538077,
+            -0.0019922884373083155,
+            0.010050520101815593,
+            0.0031720752428247446,
+            0.007352859750323051,
+            -0.0073483049386141005,
+            0.0046483587496626965,
+            -0.005093220128411125,
+            -0.026080057085901446,
+            0.012863580653903103,
+            0.0052434400313520774,
+            -0.011863360599637747,
+            0.0103209502958491,
+            -0.01048162756158923,
+            0.005787157766911021,
+            0.001318945460232246,
+            -0.0059333230422061395,
+            0.004821320238893602,
+            -0.0038922239931512784,
+            0.011267634571983924,
+            0.013256611037442042,
+            0.004592529900805719,
+            -0.013076954459595086,
+            -0.02177318635989856,
+            0.009001291224623467,
+            0.01959767163356408,
+            0.024161754192119852,
+            -0.030416109293488064,
+            0.003784634588325551,
+            -0.0014915849060829743,
+            -0.0020550564975080885,
+            -0.015632785947151753,
+            0.01587790981601866,
+            0.01716742056401084,
+            0.012721563202528083,
+            0.01893393006967701,
+            -0.004009617703771589,
+            0.010539100665843704,
+            -0.02080068888791658,
+            0.013642451766699067,
+            0.0008609869904066794,
+            -0.013629067231663643,
+            0.0014620655123815298,
+            0.027046304278766087,
+            -0.011056783825898616,
+            0.006206969699250696,
+            0.008097527175511791,
+            0.022111870736907183,
+            -0.012509479100313492,
+            -0.003735868296238069,
+            0.01516254968167132,
+            0.0014878674796313539,
+            0.0035887079417890352,
+            0.016533149527222273,
+            0.04729052276110659,
+            0.00102751186013391,
+            0.0016713773449738153,
+            0.0192177920520243,
+            -0.005563519063161466,
+            -0.01114821419109643,
+            -0.003549999863325982,
+            -0.021807893906469106,
+            0.010998558936481785,
+            0.00431279785115251,
+            -0.02186324212032519,
+            0.01024093366826206,
+            0.0125495473550584,
+            0.005268320135120931,
+            -0.0031679212410026603,
+            0.02574575247827441,
+            -0.019807879421996258,
+            0.014935295894965128,
+            0.03329450628384787,
+            0.00493356778223407,
+            -0.011423949333937285,
+            0.007263264249871384,
+            0.0027354459678742013,
+            -0.004494406041906684,
+            -0.00606745413660583,
+            0.010497391290734088,
+            -0.008191702034391382,
+            0.009870458191510176,
+            -0.009580130479693,
+            0.039202091799313755,
+            -0.009619018331262844,
+            -0.010782138248435458,
+            -0.004558645539551881,
+            -0.006241773108460662,
+            0.003478414182938609,
+            -0.003395165694642073,
+            0.005812225135633631,
+            -0.0069045407353789945,
+            -0.014607146220271209,
+            0.0008522982757210454,
+            0.00020175973815948303,
+            0.014001922919116725,
+            0.0011804904415274088,
+            -0.004871874532755353,
+            -0.007877217154706205,
+            0.015243277056007988,
+            -0.006047906208304912,
+            0.0006531937033317192,
+            -0.0011653523616313422,
+            -0.0013662556399359335,
+            -0.007401050012179761,
+            -0.015166105795085528,
+            0.00974609244984173,
+            0.016253733908242134,
+            -0.013857834499211056,
+            -0.005142936208363415,
+            0.012274716212832219,
+            0.008540229125193836,
+            0.011742606168187876,
+            0.0007451338650642946,
+            -0.009255813682572937,
+            0.008292531177374192,
+            -0.015020523035432158,
+            -0.008781317965985481,
+            0.0026250469418136603,
+            0.002381147382977664,
+            0.005839723981046825,
+            0.0006240641828668715,
+            0.004502463508605272,
+            0.005798912705218846,
+            -0.00497936644919367,
+            0.01112666823857836,
+            0.008198424171006548,
+            -0.002452143482105151,
+            -0.01017569410837814,
+            -0.001127589984272638,
+            -0.015733298167661346,
+            -0.016758315411028353,
+            -0.0009286266498936771,
+            0.015175988113750882,
+            -0.02352862898718829,
+            0.018996352975391644,
+            -0.0014524363611708201,
+            -0.0033802546171857704,
+            -0.021392428456030312,
+            -0.0025923983911570183,
+            0.0034214188668518796,
+            0.01255768230394476,
+            0.01858853630745928,
+            -0.02729543519465384,
+            0.010575894388563116,
+            -0.007204725245442141,
+            0.030394929424014717,
+            -0.018328782140005044,
+            -0.004022941273610227,
+            0.026915618079621992,
+            -0.02200018306828758,
+            0.0008791185377513978,
+            -0.011454606955158956,
+            -0.008103032693853504,
+            0.019287674837688177,
+            0.023747720480736777,
+            0.02038287242167783,
+            0.01364489294869211,
+            0.0007758044919789085,
+            0.004768030222445397,
+            0.03572705648994403,
+            0.010568241270862248,
+            0.014828087092859878,
+            -0.007542489523110974,
+            9.457808640264811e-05,
+            -0.02554033312739386,
+            0.01635899434369978,
+            2.587383534566272e-05,
+            -0.012370727506722627,
+            0.007162422186621379,
+            -0.013370882403931942,
+            -0.011066483600541058,
+            -0.007151077450730025,
+            -0.01721910018182534,
+            0.02152239849400241,
+            -0.012322440280680164,
+            0.004587729883596212,
+            -0.009079747696669277,
+            -0.013272707286450417,
+            -0.012953682633400515,
+            -0.014202214807187545,
+            -0.01313141988034687,
+            -0.026041403779538307,
+            -0.027109731260915763,
+            -0.006573285295895077,
+            -0.023780366346334524,
+            -0.013799072648505526,
+            0.005588622594136253,
+            -0.022617902666316757,
+            0.012262325666428604,
+            -0.012689827812482627,
+            -0.01226264872059864,
+            0.012805734807835008,
+            0.020274844315616616,
+            -0.011765231803328348,
+            -0.028494489703400308,
+            -0.018583843821043854,
+            -0.019970998470704358,
+            0.012576201137505969,
+            0.002266288538944273,
+            0.013165889075444961,
+            -0.003018437854660793,
+            0.003227412667880239,
+            -0.014895831590298957,
+            0.018570624675927183,
+            -0.03442390703780963,
+            -0.03108003806963536,
+            -0.013611306765348361,
+            0.005182777066476028,
+            -0.0015782033203578553,
+            -0.011429046504256945,
+            -0.0018606648572920801,
+            -0.00433805271182878,
+            -0.025109572024446088,
+            -0.010134856632190067,
+            0.011010662054484161,
+            -0.015205629927469414,
+            0.0072933565255675036,
+            0.0009018832564038863,
+            0.013870902834260887,
+            0.009587432944616442,
+            0.013393301893492882,
+            -0.006095063698384775,
+            0.004356485794113208,
+            -0.0026476612747324386,
+            0.0004894024149759108,
+            0.009336366919474478,
+            0.00839591561027103,
+            0.010551161369079399,
+            0.002260707777260338,
+            0.007830059223001696,
+            -0.0060927446216633265,
+            0.018065002687169148,
+            0.015436944875538481,
+            -0.0002520689442496089,
+            0.013323098485251487,
+            0.0003769862591044255,
+            0.00022293304921127018,
+            -0.0029970303106340466,
+            0.007888437018607987,
+            -0.011785957933642163,
+            -0.001311704406355858,
+            0.015138190893483804,
+            -0.01707247730569978,
+            -0.013045187198628258,
+            0.03107034781283638,
+            0.0018583638488268697,
+            -0.02495564203304796,
+            0.031146554085787762,
+            0.004220047610942943,
+            -0.005835199594223787,
+            -0.013205086624848782,
+            0.0014059047822222418,
+            -0.011570624844100298,
+            0.003584232809362274,
+            0.014295032061446826,
+            -0.02231984223610487,
+            0.010521094801839404,
+            0.008446892441796151,
+            -0.009681535919980559,
+            -0.012337659795586223,
+            0.009847073709209602,
+            0.016809328445200358,
+            0.004472485504034416,
+            -0.03539093810210523,
+            -0.011957694102753411,
+            0.02145470732008716,
+            -0.029237428225009913,
+            -0.00810041956259673,
+            0.007452962400105787,
+            0.0065154832040426065,
+            -0.019353109017609053,
+            -0.008580955020412362,
+            0.005725794605897509,
+            -0.004940390374637084,
+            -0.003892210172017758,
+            0.017541623128534682,
+            0.00485047043626186,
+            0.013698341792101025,
+            -0.01689738578307474,
+            0.009771458057400557,
+            -0.022222817828602557,
+            0.004021882074676753,
+            -0.023373700500461992,
+            -0.009873403167702603,
+            0.002364143609079849,
+            0.012454271200139762,
+            0.014328556467632212,
+            0.027839363143659326,
+            0.030056404238348527,
+            0.013995696971422429,
+            0.007833677585250953,
+            -0.016265199386953717,
+            -0.018134327237616175,
+            -0.006409513840075933,
+            -0.004033173350204784,
+            -0.008714798059218596,
+            0.0066373547199279235,
+            0.01858354212009877,
+            0.004499712802472377,
+            0.02609078206670711,
+            -0.006227189596272876,
+            0.009039744622014585,
+            -3.127526911951914e-05,
+            0.0008833930131827993,
+            -0.004226083827636182,
+            -0.01971685546584527,
+            0.00481003211084742,
+            -0.0013721482162825508,
+            0.025874758880591167,
+            0.00940496860916206,
+            0.0024915013399541847,
+            0.01615978314811047,
+            -0.006378952757397484,
+            -0.01174653607178726,
+            -0.010540107733238588,
+            -0.0061926268398085176,
+            -0.021752582127981104,
+            0.029059100503124387,
+            0.021195595568253612,
+            -0.010800765264123782,
+            -0.013693582166598082,
+            -0.00903369526059487,
+            0.02795767990556847,
+            0.004080965688836953,
+            -0.0013050212744006933,
+            0.016313311850308505,
+            0.012084555958999215,
+            0.012329325712491252,
+            0.009794698207689061,
+            -0.00036157148861933123,
+            0.0015364709122308352,
+            0.03722522141440073,
+            -0.007427028510548186,
+            0.0026463292682538495,
+            -0.005239324411651515,
+            0.0025987932868461313,
+            -0.009705817520422487,
+            0.007242122655628283,
+            -0.004012676385296197,
+            -0.0158760051470016,
+            0.015003444174308607,
+            -0.022443331305169994,
+            -0.0016749787204471238,
+            -0.008150635190952799,
+            0.0016696640640947303,
+            -0.02781023337840043,
+            -0.0025731206607371845,
+            0.006695065701630532,
+            -0.015547340828489069,
+            0.011816010501093885,
+            -0.01740673229051557,
+            -0.011553604323149105,
+            0.007914961618894248,
+            -0.012034157288065447,
+            0.0016684130271324446,
+            -0.0008268957621923278,
+            0.004460864332388434,
+            0.013426055079012884,
+            -0.011008808570221245,
+            0.012166046631826273,
+            -0.005816284310138776,
+            -0.007872091214834688,
+            -0.0016055129487028885,
+            -0.0030049472973942043,
+            -0.0025600746761171543,
+            -0.0009966231170649917,
+            0.034355910234323155,
+            -0.004446570896048543,
+            0.003761242147499512,
+            -0.01362639743248326,
+            0.0019833052855181806,
+            -0.036176376394999044,
+            0.028344464817309477,
+            -0.008642835889481802,
+            0.011013502221008498,
+            -0.0017508365480865596,
+            -0.013243706268954915,
+            -0.01456517309747086,
+            0.015247299501245178,
+            0.007602596572394442,
+            -0.005854531057349665,
+            0.013558248087347848,
+            -0.019724373614448298,
+            0.008530500822910953,
+            0.004463694543554244,
+            -0.015561397976987216,
+            0.004668780190328835,
+            -0.007479912658259757,
+            0.012657201279639194,
+            -0.01774730033362932,
+            0.004314594593150765,
+            0.00958571617187387,
+            -0.01892876084898881,
+            0.009901041800918215,
+            -0.028675879564512306,
+            -0.004631920242208613,
+            -0.012323455524366223,
+            0.0053804608827285865,
+            -0.000660864620708182,
+            -0.02389088252276507,
+            -0.028944280660230762,
+            -0.0002099667627277945,
+            0.00458533068247518,
+            0.0018878351997997833,
+            0.011621407953503755,
+            0.005970807015151215,
+            0.008542958629199003,
+            0.01023896054756221,
+            -0.00205207247588841,
+            0.009402752124876478,
+            0.04169706217414899,
+            -0.022899912321317197,
+            -0.011574970753549647,
+            0.005321872977369877,
+            0.011499271505460519,
+            -0.02647653483798284,
+            -0.011039791758347888,
+            0.0033424345263339144,
+            -0.005219877979550397,
+            0.03609469691834737,
+            -0.02448746405881972,
+            0.019020618652020673,
+            -0.011392297116547224,
+            0.007625475937968714,
+            -0.021465240076078376,
+            -0.003921088264697311,
+            -0.014265745991522477,
+            -0.00011221615798127438,
+            -0.0031478236827989703,
+            -0.0076053733863387755,
+            0.007485039021296698,
+            -0.001907678661303243,
+            0.0020348246022161925,
+            0.006291436821831794,
+            -0.02173305455485444,
+            -0.0019125499969493854,
+            -0.016432460924913166,
+            0.030367263349759286,
+            -0.016242095671439496,
+            0.005072805486919414,
+            0.0053258139728101506,
+            -0.002810639419737258,
+            0.012368771814660942,
+            0.004165967191536186,
+            0.00513754440091394,
+            -0.004199767241827414,
+            -0.005142642360546972,
+            -0.0053886448467479215,
+            0.017868874549760376,
+            -0.013280048598849825,
+            0.035024701466710065,
+            -0.0100959186045929,
+            -0.007094001877261568,
+            0.0019733042325232155,
+            0.010380430710282744,
+            4.2084202709296443e-05,
+            0.01053565670076933,
+            0.0034270489770133685,
+            -0.0006311078385359693,
+            0.0019774779328212756,
+            -0.017162448473652843,
+            0.003089758682621882,
+            0.01682656561693936,
+            0.005463677441664502,
+            0.004469261499451007,
+            -0.013332130891003062,
+            -0.012050233723282738,
+            0.004538268704442613,
+            -0.007533940408255447,
+            0.0003675737140813486,
+            -0.018816953644773076,
+            0.013972696748575243,
+            -0.0011096683686631858,
+            0.0002715683052236706,
+            0.00524678369365015,
+            0.023056514997908218,
+            0.01813167860994747,
+            -0.002553352602551003,
+            0.03155164978556312,
+            0.0174738879054593,
+            0.0029268039949260673,
+            0.010637295711777793,
+            -0.033222781195984666,
+            -0.013301873512972214,
+            -0.0013721703891642354,
+            0.02238421933149517,
+            -0.01530189703335162,
+            -0.014059029663525449,
+            -0.0037393534403749947,
+            0.0010394601933309767,
+            -0.005177359933828505,
+            0.0042489306011644495,
+            0.011039889511148341,
+            -0.014697595240068863,
+            0.007597149702441915,
+            0.0026780496822907634,
+            0.02092734730183164,
+            0.0004688144432243786,
+            0.017181410417231698,
+            -0.0005083970646221274,
+            0.00749639055342427,
+            0.007319420826337891,
+            0.017541463125355265,
+            -0.01002811389296805,
+            0.023582113845081534,
+            -0.008823933397729605,
+            0.0019889258902730535,
+            -0.01257106158422629,
+            -0.028159594067544737,
+            -0.012624750220149823,
+            0.015247564545279736,
+            0.005765911899842925,
+            0.006478299226263732,
+            -0.016537090297144265,
+            0.006276997440048981,
+            0.015647122171767974,
+            -0.002251683151059454,
+            0.010295358869874306,
+            0.009730378299068289,
+            0.00012816405890877098,
+            0.010181120688239766,
+            -0.00801256937783534,
+            -0.00734257767723293,
+            -0.004497520124843868,
+            0.023982754195729983,
+            -0.005443550092842393,
+            0.022434177122972028,
+            0.012647958478747348,
+            0.00616791857573897,
+            -0.010853355357954727,
+            -0.003340111254357573,
+            0.023389634440057473,
+            0.011863246790550376,
+            0.00029642699874398923,
+            -0.009846034060495678,
+            0.0019458876579524617,
+            0.0054010373765486005,
+            0.012983913038716568,
+            -0.02049427270240938,
+            -0.004344571352918878,
+            0.008461947538684664,
+            -0.008472919299762712,
+            0.004318309005722874,
+            -0.030156444522730416,
+            -0.005812303237222271,
+            0.021354352236942434,
+            0.008392281015122372,
+            -0.010232039722494646,
+            -0.004395682831498155,
+            -0.0006527587802367882,
+            0.0040601850269714945,
+            -0.0024323167053212176,
+            -0.0023729138545912176,
+            -0.008541233856441836,
+            -0.006470933419694742,
+            0.014035696617797647,
+            -0.0052691760528871924,
+            0.008597459268477037,
+            0.00012084859681905111,
+            -0.0017255958688825231,
+            -0.007323529232751411,
+            0.03382792341957058,
+            0.008087732655845718,
+            0.005980666974986827,
+            0.0013613625176327891,
+            0.0042244330945601215,
+            0.000616791811672998,
+            -0.0028374856646453707,
+            0.018075360591135542,
+            0.0156738939725117,
+            0.005054696911814988,
+            0.009038525980341447,
+            0.014672270369917403,
+            -0.019700405219230957,
+            0.012625337070556061,
+            0.012184185186281482,
+            0.019382321278525114,
+            0.0007219498737851579,
+            0.002564970959687783,
+            -0.010601873823737926,
+            -0.0029963537053764736,
+            -0.002070567089566093,
+            0.029563028293936092,
+            0.0005903976560069006,
+            -0.010509758568838935,
+            0.010992421945232951,
+            0.020603868018231348,
+            -0.02101157589451638,
+            -0.001971974816402967,
+            0.005963911670920596,
+            -0.010704207628592966,
+            -0.014466113540150018,
+            -0.021269176952661614,
+            -0.013331247081394942,
+            0.0071597345125287174,
+            -0.008162102923896894,
+            0.02367549407145414,
+            -0.014256771777502395,
+            -0.03960796112042958,
+            -0.014803626428839152,
+            -0.00864320347698291,
+            -0.027531337207277483,
+            -0.007305441146142331,
+            0.00965056093889974,
+            0.016698589600806273,
+            -0.003330867073501157,
+            -0.025198703196464594,
+            0.030483790406960656,
+            0.008872821868559751,
+            0.023708422767352917,
+            -0.021464374508407488,
+            -0.0018617153194555344,
+            -0.00886376784514331,
+            0.014505918224188001,
+            -0.017588761818663975,
+            -0.011756112651591795,
+            0.020014632992762026,
+            0.014356419813769714,
+            0.007246557500044434,
+            -0.004355132555323843,
+            0.0067299398462013936,
+            0.007649499235136303,
+            -0.012515181210363267,
+            0.012239792251154642,
+            -0.013499277313916029,
+            -0.024858295126228433,
+            0.014430561986516884,
+            0.002229494935179828,
+            0.009563004428064055,
+            -0.0015984171085659323,
+            -0.016197659608768088,
+            0.010078159490026602,
+            0.00808729175515505,
+            -0.017508840440512225,
+            -0.004793934415922065,
+            0.007133527615305415,
+            -0.020818869260580315,
+            0.031234686259160864,
+            0.011872765152279094,
+            -0.010692949770503467,
+            -0.001989039629661815,
+            0.004431079028880923,
+            0.016511544608975148,
+            0.007893700385963683,
+            -0.01997314388229276,
+            -0.003708006454566117,
+            0.01530128306203401,
+            0.00822588305477249,
+            0.004861703074728471,
+            -0.010836328783594584,
+            0.03434874763691911,
+            0.006082873924472328,
+            4.192481378506604e-05,
+            -0.0052997095110967115,
+            -0.0076108924981838355,
+            0.004196112457289409,
+            0.018905391806961686,
+            0.013330078288599763,
+            -0.004200590783114919,
+            -0.006630107527854746,
+            0.007778448128358027,
+            -0.01706991160178345,
+            0.007329056462125937,
+            0.009390327499494826,
+            -0.012870674959022891,
+            -0.0017571189976160295,
+            0.016462608870101713,
+            0.00209846661139665,
+            0.004942714850185376,
+            -0.011305050739130339,
+            0.004252776146827917,
+            0.015841821944109317,
+            0.018727567181082646,
+            -0.008399642221255733,
+            0.021448055799259867,
+            -0.020972459569228434,
+            0.027243639465535856,
+            0.028499400850980086,
+            0.016573144403275928,
+            -0.01847096443459798,
+            -0.0022409373140116146,
+            -0.0041737366139895414,
+            -0.014761716560267328,
+            0.0034546020161337894,
+            0.005125870074458691,
+            -0.01174186608185822,
+            -0.009577921232380895,
+            -0.024331993794416026,
+            0.011075959943215117,
+            0.007918147902741738,
+            -0.0015045185909419564,
+            -0.011791155919533844,
+            0.016019077577552835,
+            0.010141868950981925,
+            -0.010938402325619172,
+            -0.005635351352447256,
+            -0.019967361901734836,
+            0.009750427567651833,
+            0.009128294017175602,
+            -0.00300556760891084,
+            -0.011496103417643132,
+            0.012977890524465716,
+            0.0014080059843484115,
+            -0.01292380658817521,
+            0.010841954636959441,
+            -0.03142011651894961,
+            -0.005510537354374977,
+            -0.011027026734432255,
+            -0.003620482919145167,
+            -0.014415606143566519,
+            0.003608572766363589,
+            0.0005034611321129689,
+            -0.011470488351158329,
+            -0.0035005774696278,
+            -0.016207453351921627,
+            0.004676327282262429,
+            0.015021513761697039,
+            0.001679614645967629,
+            -0.006917375094634678,
+            0.021994274901326064,
+            0.018518524483306403,
+            -0.021903538417919984,
+            -0.020116013379669034,
+            0.004708084983787958,
+            -0.03261690161800251,
+            0.015019348187107588,
+            0.00966124471577622,
+            0.011676937880751252,
+            -0.028768229226408037,
+            0.007114534979415504,
+            0.00900420165933673,
+            0.016398927481511824,
+            0.0050879301160439245,
+            0.009453255486450082,
+            0.01222758288108498,
+            -0.0104657614414275,
+            -0.003164502778525433,
+            -5.410382753651127e-06,
+            -0.005618225986727351,
+            0.015607620507415484,
+            0.0008527697935056332,
+            0.03603666488542239,
+            -0.005277207843139309,
+            -0.0009799879252482288,
+            -0.011735731235560851,
+            0.00812046801749312,
+            -0.011713349784345638,
+            0.006334172347602515,
+            -0.0024643499727789167,
+            0.0001240009282616789,
+            0.002965067818077541,
+            0.009442572075213767,
+            -0.01086095783353607,
+            -0.007595379345224076,
+            0.007444897256325813,
+            -0.01482814340478338,
+            -0.00019984717819268013,
+            -0.009339343730980451,
+            0.0019680804106275156,
+            -0.02241031895876346,
+            0.006629838013824249,
+            0.040935644507527846,
+            0.02140216465404127,
+            -0.005481040712269146,
+            -0.012431659209559418,
+            -0.004680230194268222,
+            -0.007229300034633229,
+            -0.006482401275991039,
+            -0.004794698160419071,
+            -0.022762405726589375,
+            -0.006928553505112732,
+            0.004320321244630314,
+            0.013300138724212552,
+            0.009741018117644266,
+            -0.008842135469729511,
+            0.004531760574809748,
+            -0.02662650543210291,
+            -0.016578081411548973,
+            -0.0060021425889846495,
+            -0.015840130031005147,
+            0.006125301421471048,
+            0.027521841635711787,
+            -0.0063432924325983525,
+            0.01789356321527744,
+            0.005142029893259461,
+            0.013034833065163448,
+            -0.005338869351330258,
+            0.00035250243782976347,
+            -0.006010938630154189,
+            0.0032958656042048292,
+            0.003074861351445954,
+            -0.0015947217852680714,
+            0.004151590621941346,
+            -0.007139285822575166,
+            -0.0026091937868967284,
+            0.0011629380311555433,
+            0.0117841884590861,
+            0.006877419488576261,
+            -0.00016252653621991807,
+            -0.01761285866765538,
+            0.022221104524808997,
+            -0.018933508559881868,
+            -0.008840161102111973,
+            0.011092684029552064,
+            -0.009660487856185668,
+            0.00702597023411946,
+            0.020387386977378873,
+            -0.006854213562767843,
+            -0.009583981919303215,
+            0.013028181087649146,
+            -0.0036507672754997286,
+            0.004464368543493108,
+            0.009867428978560914,
+            0.006036303973414791,
+            0.002482944923100156,
+            0.013244650094892797,
+            0.004530294330688855,
+            -0.0032625482538717137,
+            -0.020238533752799943,
+            -0.005186498720507101,
+            0.019015752262933585,
+            -0.02252458872825473,
+            -0.0027641215988942563,
+            0.0044506144202026456,
+            -0.0030317840657635457,
+            -0.01733962766050954,
+            0.004205861489436576,
+            -0.018770275720867473,
+            -0.0033883519277721433,
+            -0.008244668812709337,
+            0.009948090119054079,
+            -0.023700732627507225,
+            0.009109315916173436,
+            -0.011739192106106307,
+            0.02356509175177699,
+            -0.003925851077921686,
+            0.024734433836185024,
+            0.0015142645677087268,
+            -0.013482933030628261,
+            0.0055323884713736554,
+            -0.0008930661843169134,
+            -0.004477731568313757,
+            0.029338490404300612,
+            -0.005418258701133154,
+            0.002249996261871466,
+            0.013491124734151694,
+            -0.021332541202417246,
+            0.021773687217263173,
+            0.0046662696076723836,
+            0.01628697326539432,
+            -0.00986238951830334,
+            -0.013799209313714441,
+            0.025178874553620743,
+            0.03324555035648896,
+            0.008398796308070151,
+            0.011261509865513953,
+            -0.01226890254177634,
+            0.018966175575083295,
+            -0.0038328997984309597,
+            0.004344837325292706,
+            0.00878668880458693,
+            -0.007200842513742962,
+            -0.011500562922303248,
+            -0.004533014670236755,
+            0.027639894892383943,
+            -0.021838848944317424,
+            -0.008920629621611768,
+            -0.005739181836252042,
+            -0.01593272996094732,
+            -0.038167534267554606,
+            -0.004371701247332816,
+            0.019604405563120094,
+            -0.005974358141737469,
+            -0.0018341151594086288,
+            0.014334266690716505,
+            0.00402620072221496,
+            -0.002852569009185304,
+            -0.008195218707838913,
+            -0.0027944153624309856,
+            -0.0059390492900516225,
+            0.016651206007022145,
+            -0.004961864528562493,
+            -0.007929830585906699,
+            0.01145431100211762,
+            -0.005731957927601062,
+            0.012130694936714224,
+            0.0004848418056781251,
+            8.392048424584131e-05,
+            -0.0015174120840537315,
+            0.00822932723013239,
+            0.018620869354789082,
+            -0.0055803454666417465,
+            0.025682778991091174,
+            0.006956936033581638,
+            -0.003495709501259274,
+            -0.005489431577052735,
+            -0.0044154293119170445,
+            -0.024561016942073285,
+            -0.0020864636079347357,
+            0.006593060072668424,
+            0.024333813203596122,
+            -0.02515989187381735,
+            -0.004953266000463339,
+            0.020957806164031402,
+            -0.0012192724005613,
+            -0.011978722479596745,
+            0.014690559986694138,
+            0.0041346025614047875,
+            0.009881584250499358,
+            0.012176393203318002,
+            0.015764530703333238,
+            -0.004923548157246672,
+            -0.0015192775000207833,
+            0.012766482928380375,
+            0.0005745973040874939,
+            0.00997809510696241,
+            0.016495444272181515,
+            0.010467514633877887,
+            -0.021108281220544853,
+            0.0015078843789125156,
+            0.00522593579521476,
+            -0.017481430635971396,
+            0.0034362821987740755,
+            0.005397819166037461,
+            0.011626527721895694,
+            0.004048632382026344,
+            0.013207335578216437,
+            -0.015768003434275563,
+            -0.005916168437120099,
+            0.008153798523776654,
+            0.004319451296835019,
+            -0.02599474886669325,
+            -0.01485009085106089,
+            -0.0006781964116146299,
+            -0.004444144921332929,
+            0.013556265879295017,
+            0.01285768162554067,
+            0.02033626001285354,
+            0.024637349990579514,
+            -0.013833060730799217,
+            -0.00842819657792762,
+            0.009436072455719896,
+            0.015560357465711765,
+            0.005798181411384387,
+            0.007874051721372273,
+            0.009297440071118126,
+            0.0077320823375042,
+            -0.004073209576661068,
+            -0.002973928843567532,
+            0.020205003097225747,
+            -0.019725382393637726,
+            0.0006746659838642147,
+            -0.018417443693472958,
+            0.007530796896254817,
+            0.01140696723506382,
+            0.01129515042745398,
+            0.002816883643872417,
+            -0.005274757308627299,
+            0.005716917047812153,
+            -0.06254973656832136,
+            -0.009191453047502128,
+            0.003384751551032943,
+            0.027256466708746898,
+            -0.010885336140129142,
+            -0.003415707910562931,
+            0.007309577090125959,
+            0.005511970708769456,
+            0.0005572239548625122,
+            0.003186427278034852,
+            0.019698716902282903,
+            -0.02052581021967601,
+            -0.010793968505805883,
+            0.004219269922138548,
+            0.0033111837923746746,
+            -0.007269521534951044,
+            0.004805737092302134,
+            0.014143978208501908,
+            0.01986656063456352,
+            -0.0024048854610842748,
+            0.01629335842915328,
+            -0.0015498603704589015,
+            -0.014496994444493684,
+            -0.002671064325909516,
+            0.0037875833047555925,
+            -0.022466433279066646,
+            -0.005980897597195858,
+            0.0043262697442044435,
+            -0.011226121648007667,
+            0.006978495397818936,
+            0.008452177557316863,
+            -0.003018545465557218,
+            -0.0008303307644324117,
+            0.017796789945838845,
+            0.006591364879905198,
+            -0.013464486184025026,
+            0.0028668426952326957,
+            -0.023377605222213174,
+            0.004325185794796361,
+            0.001416245546006914,
+            0.007809449561523174,
+            -0.008431335139468173,
+            -0.014774499666285603,
+            -0.005933810712528226,
+            0.017746847020610673,
+            -0.01965188730040385,
+            -0.002406875192894708,
+            0.01683009662984443,
+            -0.0002518220110313279,
+            0.005456812494894182,
+            -0.02011241622618092,
+            -0.0009782974741561381,
+            -0.02933296756078644,
+            -0.01171823509606186,
+            0.001162345317872531,
+            0.00892221099991475,
+            -0.006551308022163148,
+            -0.03934356275100932,
+            0.00427623835776202,
+            -0.013111664387776261,
+            -0.008752857644809231,
+            -0.010214546496134218,
+            -0.00518087722852466,
+            0.015240297918667867,
+            -0.012498100033235827,
+            0.01086490299272505,
+            -0.00917777212602538,
+            0.005895055740265625,
+            0.0090554651297836,
+            0.014212841798768113,
+            -0.005544047915369086,
+            -0.0033589999440643385,
+            -0.007924830667307956,
+            0.0022291797354234005,
+            -0.0032976226056924502,
+            0.009615674261248298,
+            -0.0051932155872883,
+            -0.0065636389655770455,
+            0.02426106498596339,
+            0.011445591067574932,
+            0.013312596099532533,
+            0.0055118624272110965,
+            -0.007220620072349783,
+            -0.01916551187613823,
+            -0.003977616240384473,
+            -0.013204506708246246,
+            -0.006315966975649096,
+            0.01434099854765068,
+            0.012588249778961223,
+            0.006357972880218866,
+            0.0007747002406996095,
+            -0.004794439624135275,
+            0.006975593520375694,
+            0.0262335496080837,
+            -0.016481909971520166,
+            -0.0013084662210086592,
+            -0.01369313900978409,
+            0.01921748474212319,
+            0.03333761672336437,
+            -0.0008918419419301244,
+            -0.007699525186523558,
+            0.013757216065348164,
+            0.00819045303047094,
+            -0.007411380297164911,
+            0.02453611080891859,
+            -0.025045462880463647,
+            -0.015989319900023054,
+            -0.011650886878708265,
+            -0.011407253985176972,
+            -0.008386367959227559,
+            0.016331961716602287,
+            -0.00141225438918211,
+            -0.0018505754298268654,
+            0.009557291617446645,
+            0.03357924917013024,
+            0.006916860901290756,
+            0.007030308953738567,
+            -0.002739345982346728,
+            -0.009376522438071562,
+            0.011482637968411055,
+            -0.007259054391879614,
+            -0.04075661435928125,
+            0.004250195164583401,
+            -0.005160505473755424,
+            0.02405623169060588,
+            -0.02197586561375289,
+            -0.00619175761282745,
+            -0.006736129049831973,
+            -0.028200015908139763,
+            -0.005867722104196303,
+            0.02805071895018145,
+            -0.015402198567775478,
+            0.014773294766214671,
+            -0.009502185346558072,
+            -0.0013419858580438616,
+            0.004443749010136947,
+            0.013008783570355656,
+            -0.02289662952366337,
+            0.0013834888740540108,
+            -0.00048549197205067054,
+            0.027969842651517314,
+            0.010437729090975158,
+            -0.0047705656249623744,
+            0.014202218020412936,
+            0.008240861154198411,
+            -0.018637315941614406,
+            0.007314019317343154,
+            -0.005747966765101792,
+            0.014833896739762147,
+            -0.01314847940891154,
+            0.007762100494499689,
+            0.00705026393499352,
+            -0.01278023669088385,
+            0.03016342404000438,
+            -0.015204129985039077,
+            0.007776946006562982,
+            0.009278778571285689,
+            0.009656723862179986,
+            0.01257686993982689,
+            0.004362439504889069,
+            -0.02898418193036562,
+            0.0002144431781454467,
+            -0.03828100004144633,
+            -0.00833748467467171,
+            -0.002611937107022567,
+            -0.002145329666219461,
+            0.007050630951707937,
+            -0.009383645162261629,
+            -0.033976442057557676,
+            0.0006429978367640709,
+            0.01603027598131823,
+            0.017501067413108286,
+            0.0002196997337451293,
+            0.013075379412332973,
+            -0.006197537539554383,
+            -0.021148120787551232,
+            -0.016986448660412497,
+            -0.010207705210051413,
+            -0.02122194744719993,
+            -0.020550470201191853,
+            -0.00908564461857315,
+            0.012090079127547193,
+            -0.009927390005879807,
+            0.013888458967497627,
+            0.029513250508042935,
+            -0.024529175134769883,
+            0.0011144447737101862,
+            0.021977677894839922,
+            -0.00198308986575084,
+            0.014108189311291326,
+            0.001696775006906642,
+            0.014122064251288995,
+            -0.0004166140484018266,
+            -0.010556016685519523,
+            -0.009729362492661684,
+            0.017487929296614076,
+            -0.0026043257634797124,
+            0.016715798948897194,
+            0.012183487338360088,
+            0.003771721868999568,
+            -0.0030887189945795796,
+            0.010667776697461199,
+            -0.026471157050402344,
+            0.014249996582064706,
+            -0.006690961709923224,
+            -0.0181382691440242,
+            -0.008729104191096461,
+            0.003882731567105777,
+            -0.005609384377642956,
+            -0.007794812949125115,
+            0.012967089730551055,
+            -0.00944374111323783,
+            0.02088525981049842,
+            0.0023400121055514958,
+            -0.012196925240774426,
+            0.003648688547842322,
+            -0.03322041209304968,
+            0.0015191765655963997,
+            -0.018402382776013956,
+            -0.024659631472836927,
+            0.03840173429666435,
+            0.011991452465253198,
+            0.006639702644606365,
+            -0.01675838115087676,
+            0.007953102302963526,
+            0.0004257479565099406,
+            0.017484338819046517,
+            0.011068864013925452,
+            0.006466915839538296,
+            0.008364454017976605,
+            -0.023316137291380285,
+            -0.010898421318668592,
+            0.016204569093730564,
+            0.012099473359791148,
+            -0.01017229369980751,
+            0.019431669546223253,
+            0.017955119269367787,
+            0.007300372071724306,
+            -0.006898619924677706,
+            0.01647927363640854,
+            0.021012103522831126,
+            0.010291016618557653,
+            -0.00047955552103474145,
+            0.0008684297441778696,
+            -0.010534247928997246,
+            0.005037930235889528,
+            0.035417626740307205,
+            0.014712595692093886,
+            0.010650306363963653,
+            -0.03626584475133921,
+            0.021764456453493054,
+            -0.011961521673661574,
+            0.009092981192874177,
+            0.006089104887183293,
+            0.017982801101846038,
+            0.006916969397519662,
+            0.014196254958707752,
+            0.011745069030657836,
+            -0.006689305450954901,
+            0.025205280281804708,
+            -0.001601414806100461,
+            -0.011992525142592405,
+            0.003386930324498371,
+            -0.01859193251438437,
+            0.005303261319773131,
+            -0.0041710536965100615,
+            -0.004459393120257307,
+            0.0032461828144552565,
+            -0.0018650956576455287,
+            0.015678225251799887,
+            0.003894142628084471,
+            -0.02355212701026675,
+            -0.0014873755115206368,
+            -0.024284295126348117,
+            0.0027260592623277375,
+            -0.022185571998195344,
+            0.00022883605560765203,
+            -0.005557568353382136,
+            -0.009749382813201403,
+            -0.025385857431333838,
+            -0.009853073898348951,
+            -0.0035843255959309307,
+            0.007661137888845632,
+            -0.010521037433807653,
+            -0.004246698084422667,
+            -0.0019920499321964815,
+            -0.019465113363548715,
+            0.009014255846983305,
+            0.027122533756959888,
+            0.016848221859975445,
+            0.027170883580103264,
+            -0.00019554112591107156,
+            0.013614675849485926,
+            -0.002066822813103284,
+            -0.01603063005944364,
+            0.008824663429777616,
+            0.0075653522381218495,
+            -0.00531780740880296,
+            0.014643348459785112,
+            0.0072634484937344704,
+            -0.012800149015319017,
+            0.009859503660384916,
+            -0.007161712176469609,
+            -0.006437857349742842,
+            -0.009764856421740622,
+            -0.02058868349990357,
+            -0.014866137971603676,
+            0.0012450207572602557,
+            -0.002123265406747006,
+            0.006000473862248991,
+            -0.004054016790850226,
+            0.009584879012937315,
+            -0.00608816807208866,
+            -0.00925801057439376,
+            -0.013962683072792106,
+            -0.008924356118158748,
+            -0.004210064145949253,
+            -0.004381709838058668,
+            -0.011732699976353612,
+            -0.005479523097469261,
+            0.0010680258077037758,
+            -0.0019401640391035755,
+            -0.01397652201595301,
+            0.011650839243603377,
+            0.0058731838291845645,
+            -0.002462743134993977,
+            0.02219008688925881,
+            0.01097316136828334,
+            0.0018083001685932406,
+            0.01366095798812196,
+            0.0003942601242216818,
+            -0.0064063532958756175,
+            -0.03343312148366622,
+            -0.0005083355836126291,
+            -0.02815586367527057,
+            0.025770078162165123,
+            -0.0069114949102666145,
+            -0.010322087105114614,
+            -0.002519047562374146,
+            -0.006063669644021698,
+            0.005873609611264377,
+            -0.002722940652283699,
+            0.0021229525800901114,
+            -0.023673778230768945,
+            0.0028629715379660113,
+            -0.003953653437464966,
+            -0.0036366957271968823,
+            0.007469664864329332,
+            0.0346520690389124,
+            -0.004054282838167587,
+            -0.0034765253405897676,
+            0.005159466707114532,
+            0.011642913651743502,
+            0.027845571220425726,
+            -0.004445196354520585,
+            -0.0032038508043339413,
+            -0.02233838584152591,
+            0.0054112576531374175,
+            -0.00019544838850471866,
+            -0.004733192299951088,
+            0.016664217348411003,
+            -0.016719175418727594,
+            -0.01631658832047864,
+            -0.00545969677717507,
+            -0.014570848053102338,
+            -0.00022620258237138677,
+            0.014454101124666536,
+            0.011056312437637014,
+            -0.004035526363810078,
+            -0.012459378363360557,
+            -0.0022595493558020706,
+            -0.030370952502579162,
+            0.012766010284884204,
+            0.1141689619126009,
+            -0.00766197047210299,
+            -0.00896655658602649,
+            -0.009481112393590025,
+            0.001435269335646347,
+            0.001338755472877713,
+            -0.011739143028767118,
+            0.07166096084053437,
+            -0.0077855767070652395,
+            0.0002139580430342728,
+            -0.011312198831181402,
+            0.001844005266872301,
+            -0.001825412693163496,
+            0.01371932421024605,
+            -0.005025814503687022,
+            -0.018213360991353643,
+            -0.0033612481616769416,
+            -0.02941403830641238,
+            -0.0041137865872933515,
+            -0.005683630256743215,
+            -0.004288198570156578,
+            -0.014982999363445122,
+            0.0067149665791261335,
+            -0.019661973497372313,
+            -0.012688556129040908,
+            0.001124391650175233,
+            -0.0033623453063430534,
+            0.00996944383866679,
+            -0.020496336814603385,
+            -0.0044354321069609536,
+            0.0019875840782527733,
+            0.009706675240563883,
+            0.0011600364428219214,
+            -0.012755178230821602,
+            -0.0026514442584739412,
+            0.004056350166031111,
+            0.01909951327913368,
+            0.014360281326789544,
+            0.003732344102045803,
+            0.008623457695718206,
+            0.014461868085333023,
+            -0.001692149528228451,
+            0.01948035376572433,
+            0.01172550239477637,
+            0.0031966531144076173,
+            0.0004779413449255626,
+            0.0030912562273611517,
+            -0.004597633235071031,
+            0.01582547400214653,
+            -0.014811177637844808,
+            -0.02411202772171917,
+            -0.0071272238837422185,
+            0.018015541137524113,
+            -0.013081287852028074,
+            1.8837689271256874e-05,
+            0.004062803284646432,
+            0.027569726868200254,
+            0.0010564850664255057,
+            -0.01179755720247972,
+            0.0028876134800690334,
+            -0.01638215925131053,
+            0.022944183106912663,
+            0.0026043713064243836,
+            0.03279173899194346,
+            -0.005103903273968361,
+            0.00578213669463383,
+            -0.020744194169889544,
+            -0.007588793365216579,
+            -0.00819862611489168,
+            0.0017401442433921013,
+            0.010993500174794084,
+            0.005411238350013134,
+            0.01620925462604661,
+            -0.00641615069699833,
+            -0.020751730783437995,
+            0.014776878596707538,
+            -0.009552193304486048,
+            -0.03393319554458277,
+            0.008881987295131347,
+            -0.029102297123590998,
+            -0.0045267103602711,
+            0.00368971742585232,
+            -0.014269341759285646,
+            -0.016264809810970175,
+            0.012837302095591106,
+            -0.0011890334180728564,
+            -0.008937992768323412,
+            0.0014444613725248846,
+            -0.017110187133760626,
+            -0.02385668037818897,
+            0.006017405364919834,
+            0.00676670672597817,
+            -0.0077994500447595,
+            0.018312452858615755,
+            -0.027015660037024585,
+            0.006123912473734307,
+            -0.012973676426738582,
+            0.009041974762537394,
+            -0.01580183887345168,
+            0.01163516332546347,
+            0.010887662470499361,
+            -0.001503467664709027,
+            -0.0076234124970223326,
+            0.01231202541338883,
+            0.004852720134894811,
+            0.033760365210093034,
+            0.028658326381800395,
+            0.000995999783627898,
+            -0.0025605367058226737,
+            -0.0020356794893366384,
+            0.011667525096213252,
+            -0.014049700287539817,
+            0.011218246246280464,
+            0.02658113044439717,
+            0.017568631325838108,
+            -0.019133062654008885,
+            0.018050678772708717,
+            -0.010356118256625172,
+            0.012872052420999813,
+            -0.025620154415048252,
+            -0.005135768485044442,
+            0.004936490920616527,
+            0.005163008729256001,
+            0.007425278892536464,
+            0.001410212306439329,
+            0.021891387289347516,
+            0.014999616600766989,
+            0.00027613811156650413,
+            0.00558422822386443,
+            0.01477009526981092,
+            0.010497782599293994,
+            -0.0005567894749233204,
+            -0.00467776506604011,
+            -0.016998035358377343,
+            0.00957271797745408,
+            -0.006065116857898469,
+            0.005138070401846389,
+            0.01698914855850691,
+            -0.01230257809343985,
+            0.011109844802663414,
+            0.007518553660463051,
+            -0.015504773927595823,
+            -0.017169933945839305,
+            0.005552829003043926,
+            -0.02252633790984703,
+            -0.01306722099657118,
+            0.02715689072730398,
+            -0.010513941399990838,
+            -0.02790582732974509,
+            -0.0009847768069205477,
+            0.009797865241108237,
+            -0.002772952073769185,
+            -0.0027968741526000576,
+            0.0037541733651479613,
+            0.010801137722284016,
+            -0.014658156970099418,
+            0.002075415233345957,
+            0.004735264309137368,
+            0.01930804315461151,
+            -0.0014151522239496115,
+            -0.015046118414100617,
+            -0.002288495894227231,
+            0.021592829677937263,
+            0.017174609713578173,
+            0.00265560378681856,
+            -0.016499825201963983,
+            -3.965296928410772e-05,
+            -0.0074142747014567635,
+            0.00021360696629503726,
+            0.00027431174904249525,
+            -0.0056391089282546054,
+            -0.018931681700770225,
+            -0.003888248121485032,
+            0.011181675419027736,
+            -0.00247252920175839,
+            0.01977860312223846,
+            -0.0011527575128223392,
+            -0.015763587433200915,
+            0.001960734212846797,
+            -0.005280860341480037,
+            -0.010372829570478649,
+            0.01351900743232294,
+            -0.009359942374603997,
+            0.006788746813046239,
+            -0.003212015225284119,
+            0.004611143085183367,
+            -0.0003308099698088422,
+            -0.017017265064185255,
+            0.02924142709831441,
+            -0.0018238549856433466,
+            -0.004769274041546295,
+            -0.0014058363976586562,
+            -0.033393331166741945,
+            0.015017165991303838,
+            -0.014574579250749552,
+            0.006745699409061803,
+            0.002705853802154516,
+            0.01272741296341735,
+            0.003253095246601883,
+            -0.008898616461779179,
+            0.0027485606928249326,
+            0.017040360927723885,
+            0.013756402798365262,
+            -0.007292864293090161,
+            0.008694855886257706,
+            -0.0028934389922887676,
+            -0.0037483208383542015,
+            0.01363963339139356,
+            0.004170611039133969,
+            -0.006282657671540625,
+            -0.0009784372731140945
+          ],
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "Mixture-of-Experts (MoE)",
+            "fine-grained expert segmentation",
+            "shared expert isolation"
+          ],
+          "title": "Deepseekmoe: Towards Ultimate Expert Specialization In Mixture-Of-Experts Language Models"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "4ef9096a-b51d-4ed6-b15e-5f4caff268e6",
+        "properties": {
+          "page_content": "1. Introduction\n\nRecent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023a). It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017) has emerged as a popular solution. It can\n\nenable parameter scaling, while concurrently keeping computational costs at a modest level.\n\nRecent applications of MoE architectures in Transformers (Vaswani et al., 2017) have yielded successful attempts at scaling language models to a substantial size (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022), accompanied with remarkable performance. These achievements underscore the considerable potential and promise of MoE language models.\n\nDespite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts.\n\nThis architecture manifests two potential issues: (1) Knowledge Hybridity: existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. (2) Knowledge Redundancy: tokens assigned to different experts may require common knowledge. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters.\n\nThese issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models.\n\nIn response to the aforementioned issues, we introduce DeepSeekMoE, an innovative MoE architecture specifically designed towards ultimate expert specialization. Our architecture involves two principal strategies: (1) Fine-Grained Expert Segmentation: while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. (2) Shared Expert Isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.\n\nThrough compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized.\n\nStarting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B (Lepikhin et al., 2021) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5× expert parameters and computation. Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE.\n\nThese studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization.\n\nLeveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B (DeepSeek-AI, 2024), a dense model trained on the same 2T corpus. We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B (Touvron et al., 2023b), which has approximately 2.5 times the activated parameters. Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard1. Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. The experimental results still validate its substantial advantages over the GShard architecture consistently. In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nOur contributions are summarized as follows: - Architectural Innovation. We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation.\n\nEmpirical Validation. We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. Experimental results validate the high 1https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models\n\nScalability. We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B.\n\nAlignment for MoE. We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B.\n\nPublic Release. In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization.\n\n",
+          "embedding": [
+            -0.007097613997757435,
+            -0.024122508242726326,
+            -0.01228793989866972,
+            -0.023231398314237595,
+            0.03164222836494446,
+            -0.012272306717932224,
+            -0.00579612422734499,
+            0.04649406298995018,
+            -0.012303573079407215,
+            0.0454309843480587,
+            -0.02185564860701561,
+            -0.010310300625860691,
+            0.03736409172415733,
+            -0.01593836396932602,
+            0.0027690858114510775,
+            0.013272850774228573,
+            0.008512446656823158,
+            0.021105241030454636,
+            0.017853470519185066,
+            -0.029391001909971237,
+            0.004928464535623789,
+            -0.04117866978049278,
+            -0.02241845615208149,
+            -0.022606058046221733,
+            0.011209228076040745,
+            0.016493353992700577,
+            0.025873461738228798,
+            0.010279033333063126,
+            -0.04374256357550621,
+            0.036988887935876846,
+            0.026092329993844032,
+            0.026905274018645287,
+            0.012608426623046398,
+            0.006147878244519234,
+            -0.01596963219344616,
+            -0.022027617320418358,
+            0.03842717036604881,
+            0.020776936784386635,
+            -0.013640238903462887,
+            0.023888004943728447,
+            0.03273657336831093,
+            0.01783783733844757,
+            -0.028609326109290123,
+            -0.008363928645849228,
+            0.020730035379529,
+            0.000511997495777905,
+            -0.039646584540605545,
+            0.003970911726355553,
+            0.006726318504661322,
+            -0.012373924255371094,
+            -0.017962904646992683,
+            0.0026518343947827816,
+            -0.0023176681715995073,
+            0.005772673990577459,
+            -0.04880782216787338,
+            0.008121608756482601,
+            0.03786436468362808,
+            0.007570527959614992,
+            0.06381599605083466,
+            -0.00819195993244648,
+            0.02632683329284191,
+            -0.028968896716833115,
+            -0.011349929496645927,
+            0.005518629681318998,
+            0.012342656962573528,
+            -0.03111068718135357,
+            -0.009755311533808708,
+            0.004252315033227205,
+            -0.001601457828655839,
+            0.008856384083628654,
+            -0.03999052196741104,
+            0.02268422581255436,
+            -0.021433543413877487,
+            0.01341355312615633,
+            0.022637324407696724,
+            0.0019063112558797002,
+            0.03542553633451462,
+            0.012115971185266972,
+            0.00941919069737196,
+            -0.028968896716833115,
+            0.010451002046465874,
+            0.03836463764309883,
+            0.0045532602816820145,
+            0.023387733846902847,
+            0.046181391924619675,
+            -0.033017974346876144,
+            -0.01883838139474392,
+            -0.07066347450017929,
+            0.019416820257902145,
+            -0.0036641042679548264,
+            -0.036301013082265854,
+            -0.0024915907997637987,
+            0.014390647411346436,
+            -0.017790935933589935,
+            -0.015133239328861237,
+            0.030250845476984978,
+            -0.009114337153732777,
+            -0.009606792591512203,
+            -0.04149134084582329,
+            0.0034120138734579086,
+            0.0041936892084777355,
+            0.012100338004529476,
+            0.0006917828577570617,
+            -0.01863514631986618,
+            -0.017353197559714317,
+            0.007664328906685114,
+            -0.01122486125677824,
+            0.027843283489346504,
+            -0.009442640468478203,
+            0.07210175693035126,
+            0.011662599630653858,
+            -0.013014898635447025,
+            -0.00774249667301774,
+            0.01697799377143383,
+            -0.01190491858869791,
+            -0.024028707295656204,
+            -0.042460616677999496,
+            -0.03342444822192192,
+            0.007097613997757435,
+            -0.03351824730634689,
+            0.0031052061822265387,
+            -0.007050713524222374,
+            0.022011984139680862,
+            -0.02262169122695923,
+            -0.07103867828845978,
+            0.041272468864917755,
+            -0.008317028172314167,
+            -0.01979202590882778,
+            -0.002986000617966056,
+            -0.007121064234524965,
+            0.0018261895747855306,
+            0.004967548418790102,
+            -0.03423738852143288,
+            -0.027171043679118156,
+            0.033737119287252426,
+            0.028562424704432487,
+            0.04386763274669647,
+            0.06197123974561691,
+            -0.014328112825751305,
+            0.018463177606463432,
+            0.016243217512965202,
+            -0.01963569037616253,
+            0.007547077722847462,
+            -0.013358835130929947,
+            -0.029656771570444107,
+            -0.01833810843527317,
+            0.01499253697693348,
+            0.06312812119722366,
+            -0.00027554063126444817,
+            0.005131700076162815,
+            -0.03043844737112522,
+            0.05531136319041252,
+            -0.02162114717066288,
+            0.02626429870724678,
+            -0.001939532463438809,
+            0.002087073866277933,
+            -0.018291208893060684,
+            0.014077977277338505,
+            -0.019745124503970146,
+            0.003943553194403648,
+            0.01255370955914259,
+            -0.0338309183716774,
+            0.00569450668990612,
+            0.01824430748820305,
+            0.004725228995084763,
+            0.03333064541220665,
+            0.006179145537316799,
+            0.0007611566106788814,
+            -0.06187743693590164,
+            0.02789018489420414,
+            0.007937915623188019,
+            0.02185564860701561,
+            -0.011428097262978554,
+            0.008035624399781227,
+            0.0017568158218637109,
+            0.0012887875782325864,
+            0.02079256996512413,
+            0.0060306270606815815,
+            -0.042304281145334244,
+            0.016868559643626213,
+            -0.028703127056360245,
+            0.024951083585619926,
+            0.027937084436416626,
+            0.01647772081196308,
+            0.029172131791710854,
+            0.01813487336039543,
+            0.04702560231089592,
+            0.028796928003430367,
+            0.017259396612644196,
+            0.01966695673763752,
+            0.01883838139474392,
+            0.007789397146552801,
+            0.03207996487617493,
+            0.011084159836173058,
+            0.010638604871928692,
+            -0.026076696813106537,
+            -0.0023919271770864725,
+            0.025123052299022675,
+            0.022371554747223854,
+            0.00022302180877886713,
+            -0.0031208398286253214,
+            -0.027405545115470886,
+            0.0038243476301431656,
+            -0.006437098141759634,
+            0.011725133284926414,
+            -0.0005163944442756474,
+            -0.003394426079466939,
+            0.02990690805017948,
+            0.02507615275681019,
+            -0.02055806666612625,
+            -0.013718406669795513,
+            -0.013202500529587269,
+            -0.03495652973651886,
+            -0.05943860858678818,
+            0.00731257488951087,
+            0.005628064274787903,
+            -0.002409514971077442,
+            0.012209772132337093,
+            0.024497712031006813,
+            -0.03864603862166405,
+            -0.006269038189202547,
+            -0.005651514511555433,
+            0.015563160181045532,
+            -0.02762441523373127,
+            -0.0013708635233342648,
+            -0.0062494962476193905,
+            0.012584976851940155,
+            0.0018232582369819283,
+            0.0441177673637867,
+            -0.0175720676779747,
+            -0.0032107324805110693,
+            0.018682045862078667,
+            0.00900490302592516,
+            0.02573275938630104,
+            0.03989672288298607,
+            -0.02431011013686657,
+            0.012209772132337093,
+            -0.0007171873585321009,
+            0.011154510080814362,
+            0.017368830740451813,
+            0.00593682611361146,
+            -0.002163287252187729,
+            -0.030500980094075203,
+            -0.018181772902607918,
+            0.035269200801849365,
+            0.02026103064417839,
+            0.00014326647215057164,
+            0.04396143555641174,
+            0.0065035405568778515,
+            0.021496077999472618,
+            0.013390102423727512,
+            -0.016165051609277725,
+            -0.008066891692578793,
+            0.006788852158933878,
+            -0.02623303234577179,
+            0.04271075502038002,
+            0.034831464290618896,
+            -0.0005466843722388148,
+            -0.010341567918658257,
+            -0.03958405181765556,
+            0.019682591781020164,
+            -0.028265388682484627,
+            -0.016071248799562454,
+            0.04943316429853439,
+            -0.01714996248483658,
+            0.02401307411491871,
+            0.04552478343248367,
+            0.01860387809574604,
+            -0.009575525298714638,
+            -0.0058977422304451466,
+            0.0014216724084690213,
+            -0.011396829970180988,
+            -0.02925029955804348,
+            0.013546437956392765,
+            0.003519494319334626,
+            -0.03176729381084442,
+            0.024169407784938812,
+            -0.0020089060999453068,
+            0.01830684207379818,
+            0.01532084122300148,
+            0.02158987894654274,
+            -0.009817845188081264,
+            0.02543572336435318,
+            0.010239950381219387,
+            -0.019651323556900024,
+            -0.0199014600366354,
+            -0.008270127698779106,
+            -0.04264821857213974,
+            -0.01573512889444828,
+            -0.013640238903462887,
+            -0.012413008138537407,
+            0.012037803418934345,
+            -0.05046497657895088,
+            0.004490726161748171,
+            0.00925503857433796,
+            -0.003300625132396817,
+            -0.017165595665574074,
+            0.020089061930775642,
+            0.004693961702287197,
+            -0.0008246677462011576,
+            -0.06691142916679382,
+            0.008473362773656845,
+            0.011529714800417423,
+            0.001870158826932311,
+            -0.02576402761042118,
+            0.023622235283255577,
+            -0.012209772132337093,
+            0.047901079058647156,
+            -0.01425776258111,
+            -0.03664495050907135,
+            0.0108574740588665,
+            -0.014296846464276314,
+            -0.0075079938396811485,
+            -0.007574436254799366,
+            0.0036054786760360003,
+            0.058719467371702194,
+            0.04042826220393181,
+            0.013007081113755703,
+            -0.03176729381084442,
+            -0.020683135837316513,
+            0.022699858993291855,
+            0.0027397728990763426,
+            -0.04705686867237091,
+            0.019057249650359154,
+            0.015461542643606663,
+            -0.010826206766068935,
+            0.024450812488794327,
+            0.04302342236042023,
+            -0.023794203996658325,
+            0.017024893313646317,
+            -0.011193593963980675,
+            -0.032580237835645676,
+            -0.031251389533281326,
+            0.012358291074633598,
+            -0.005334935616701841,
+            -0.04430537298321724,
+            -0.0290157962590456,
+            -0.0036191579420119524,
+            -0.007433734834194183,
+            0.009817845188081264,
+            -0.015180139802396297,
+            -0.039052512496709824,
+            -0.036394815891981125,
+            0.031517159193754196,
+            0.0017431365558877587,
+            0.0038946985732764006,
+            -0.03601961210370064,
+            0.0033651133999228477,
+            0.005702323280274868,
+            0.00542873702943325,
+            0.001134406658820808,
+            0.009090886451303959,
+            0.009294122457504272,
+            0.0013835657155141234,
+            -0.01187365222722292,
+            -0.038614772260189056,
+            -0.029203400015830994,
+            -0.0011451547034084797,
+            0.011209228076040745,
+            0.04055332764983177,
+            0.03014140948653221,
+            -0.05312267318367958,
+            -0.04149134084582329,
+            0.02957860380411148,
+            -0.03705142065882683,
+            -0.017228130251169205,
+            0.0033592507243156433,
+            0.00022241112310439348,
+            0.002487682504579425,
+            0.010724589228630066,
+            0.00532321073114872,
+            0.04427410289645195,
+            -0.0007357521099038422,
+            0.012710045091807842,
+            0.01567259430885315,
+            0.013460453599691391,
+            0.02424757555127144,
+            -0.015680411830544472,
+            -0.020354831591248512,
+            0.04755714163184166,
+            -0.023669136688113213,
+            0.01641518622636795,
+            0.011365562677383423,
+            0.011037259362637997,
+            0.013906008563935757,
+            0.017290662974119186,
+            0.001894586137495935,
+            0.011865834705531597,
+            -0.058844536542892456,
+            0.014375013299286366,
+            0.004736953880637884,
+            -0.031954895704984665,
+            -0.024482078850269318,
+            -0.014695500954985619,
+            0.02729611098766327,
+            0.03207996487617493,
+            0.00952080823481083,
+            -0.024466445669531822,
+            -0.022668590769171715,
+            -0.00555380480363965,
+            0.007304758299142122,
+            0.04596252366900444,
+            -0.0008617973071523011,
+            0.02049553394317627,
+            -0.015492809936404228,
+            0.010279033333063126,
+            0.01527394074946642,
+            0.01324940100312233,
+            0.026655137538909912,
+            -0.00941919069737196,
+            -3.502273102640174e-05,
+            -0.043930165469646454,
+            -0.027139775454998016,
+            -0.013937275856733322,
+            0.006112703122198582,
+            0.02712414227426052,
+            0.026389367878437042,
+            0.006374564487487078,
+            -0.003955278545618057,
+            0.022011984139680862,
+            -0.030500980094075203,
+            0.016165051609277725,
+            0.03608214482665062,
+            0.026670770719647408,
+            -0.016665322706103325,
+            0.0023587059695273638,
+            0.01933865435421467,
+            0.0032713122200220823,
+            0.01016178261488676,
+            -0.0252012200653553,
+            0.028859462589025497,
+            -0.022996895015239716,
+            0.029422268271446228,
+            -0.0237160362303257,
+            -0.027217943221330643,
+            -0.030954353511333466,
+            -0.003388563636690378,
+            0.03458132594823837,
+            -0.013046164996922016,
+            -0.01112324371933937,
+            -0.008449913002550602,
+            0.02380983717739582,
+            -0.014961270615458488,
+            -0.0068513862788677216,
+            -0.03702015429735184,
+            -0.017900370061397552,
+            -0.05071511119604111,
+            0.0032380910124629736,
+            0.003027038648724556,
+            -0.022199586033821106,
+            -0.044680576771497726,
+            0.014445364475250244,
+            -0.09092450141906738,
+            -0.004307032562792301,
+            0.032580237835645676,
+            -0.020683135837316513,
+            0.006538716144859791,
+            0.0038204393349587917,
+            0.011740767396986485,
+            -0.03877110779285431,
+            0.0010366971837356687,
+            0.03226756677031517,
+            0.0514029860496521,
+            -0.008637514896690845,
+            -0.01996399462223053,
+            0.00903616938740015,
+            -0.0172437634319067,
+            -0.01904161646962166,
+            0.008434279821813107,
+            0.0228874608874321,
+            0.020135963335633278,
+            0.0036289289128035307,
+            0.026358099654316902,
+            0.01718122884631157,
+            0.030860550701618195,
+            0.03336191177368164,
+            -0.01271786168217659,
+            -0.04918302595615387,
+            0.02768694795668125,
+            0.039083778858184814,
+            0.009739677421748638,
+            -0.015281757339835167,
+            -0.01543809287250042,
+            -0.0034159221686422825,
+            -0.0024661864154040813,
+            0.0010112927993759513,
+            0.0008114769589155912,
+            -0.00029141842969693244,
+            0.01521922368556261,
+            0.028109053149819374,
+            -0.009481724351644516,
+            0.026780204847455025,
+            0.003990453667938709,
+            0.02194944955408573,
+            0.004506359808146954,
+            -0.02513868547976017,
+            -0.010255583561956882,
+            -0.0037227298598736525,
+            0.02481038309633732,
+            0.005276310257613659,
+            -0.03505033254623413,
+            0.006155695300549269,
+            0.029406635090708733,
+            -0.01286637969315052,
+            0.008465546183288097,
+            0.013452637009322643,
+            -0.00037764699663966894,
+            0.016337020322680473,
+            -0.01910415105521679,
+            -0.0001468084374209866,
+            0.02330956608057022,
+            -0.01797853782773018,
+            0.008160692639648914,
+            -0.028593692928552628,
+            -0.012061254121363163,
+            0.007492360193282366,
+            -0.011396829970180988,
+            0.017556432634592056,
+            -0.001251657959073782,
+            0.015680411830544472,
+            -0.005045715719461441,
+            0.0027983984909951687,
+            -0.010654238052666187,
+            0.03501906618475914,
+            -0.00401390390470624,
+            0.03150152415037155,
+            0.021214675158262253,
+            -0.015250490047037601,
+            0.044524241238832474,
+            -0.03295544162392616,
+            -0.025873461738228798,
+            0.01159224845468998,
+            0.006386289373040199,
+            -0.011537531390786171,
+            -0.015336474403738976,
+            -0.013616788201034069,
+            -0.00037911266554147005,
+            0.01718122884631157,
+            -0.02209015190601349,
+            -0.024153774604201317,
+            -0.020010894164443016,
+            -0.010318117216229439,
+            -0.0031638317741453648,
+            0.000499783840496093,
+            -0.0059446427039802074,
+            -0.014593882486224174,
+            0.04880782216787338,
+            -0.05543643236160278,
+            0.01943245530128479,
+            0.047838546335697174,
+            0.013171233236789703,
+            -0.04252314940094948,
+            -0.023997440934181213,
+            -0.014539165422320366,
+            -0.0075509860180318356,
+            -0.041835278272628784,
+            0.014109243638813496,
+            0.002925420878455043,
+            0.005772673990577459,
+            -0.048995424062013626,
+            -0.0009712319006212056,
+            0.005784399341791868,
+            -0.022996895015239716,
+            0.014015442691743374,
+            -0.022934360429644585,
+            -0.022606058046221733,
+            0.01424212846904993,
+            0.019416820257902145,
+            -0.006104886066168547,
+            0.013741856440901756,
+            -0.0035761657636612654,
+            -0.007980907335877419,
+            -0.0021456994581967592,
+            -0.0009365450241602957,
+            0.0011207273928448558,
+            0.019229218363761902,
+            0.004365657921880484,
+            -0.00630812207236886,
+            -0.0013132150052115321,
+            -0.005002723541110754,
+            0.030688581988215446,
+            -0.008864200673997402,
+            -0.009661509655416012,
+            0.03958405181765556,
+            0.035800740122795105,
+            -0.010724589228630066,
+            -0.02218395285308361,
+            -0.023825472220778465,
+            0.014797118492424488,
+            -0.02948480285704136,
+            0.028265388682484627,
+            0.036426082253456116,
+            -0.011670416221022606,
+            -0.03162659332156181,
+            0.005577255040407181,
+            -0.022105785086750984,
+            -0.003996316343545914,
+            0.01704052835702896,
+            0.01804107241332531,
+            -0.04418030381202698,
+            -0.006003268528729677,
+            0.01688419282436371,
+            0.01238955743610859,
+            -0.009137787856161594,
+            -0.03064168244600296,
+            0.008653149008750916,
+            0.01904161646962166,
+            0.013335385359823704,
+            -0.0027808109298348427,
+            0.00702335499227047,
+            -0.0165871549397707,
+            -0.0011353837326169014,
+            -0.000332700670696795,
+            -0.013429186306893826,
+            0.03573820739984512,
+            0.0010484223021194339,
+            0.010294667445123196,
+            -0.00013398408191278577,
+            -0.02005779556930065,
+            -0.012944547459483147,
+            0.03551933914422989,
+            0.01399199292063713,
+            0.038614772260189056,
+            0.0019502805080264807,
+            0.015578794293105602,
+            0.03817703574895859,
+            -0.002221912844106555,
+            -0.0273586455732584,
+            0.021668046712875366,
+            0.0010064073139801621,
+            -0.0441177673637867,
+            0.00657389173284173,
+            -0.03128265589475632,
+            -0.04593125730752945,
+            0.023653503507375717,
+            -0.013624604791402817,
+            0.022793659940361977,
+            -0.005432645324617624,
+            -0.004381291568279266,
+            -0.0318298302590847,
+            0.0035038606729358435,
+            0.02188691683113575,
+            0.015664778649806976,
+            0.04424283653497696,
+            -0.0051629673689603806,
+            -0.005643697455525398,
+            0.011217044666409492,
+            -0.001060147536918521,
+            0.037989433854818344,
+            0.01185801811516285,
+            -0.014281212352216244,
+            0.013859108090400696,
+            0.030266478657722473,
+            0.04761967435479164,
+            -0.02928156778216362,
+            -0.03539426997303963,
+            0.008676598779857159,
+            0.0044633676297962666,
+            0.041303738951683044,
+            0.006831844337284565,
+            0.006155695300549269,
+            0.016790391877293587,
+            0.014695500954985619,
+            -0.008567164652049541,
+            0.024028707295656204,
+            -0.03014140948653221,
+            -0.034768927842378616,
+            0.03670748323202133,
+            -0.008145059458911419,
+            0.012858563102781773,
+            0.01800980605185032,
+            -0.01425776258111,
+            -0.016821658238768578,
+            0.005639789160341024,
+            0.01091219112277031,
+            -0.015344290994107723,
+            0.030798017978668213,
+            0.03389345109462738,
+            0.02398180589079857,
+            -0.0048581138253211975,
+            0.012694410979747772,
+            0.0009091863757930696,
+            0.002092936309054494,
+            -0.01744699850678444,
+            0.007003813050687313,
+            -0.0008837819332256913,
+            -0.01618068479001522,
+            0.043992701917886734,
+            -0.0290157962590456,
+            -0.012928914278745651,
+            -0.0022844469640403986,
+            -0.002614704892039299,
+            0.005502996034920216,
+            0.003972866106778383,
+            0.018619511276483536,
+            -0.012788211926817894,
+            -0.018916549161076546,
+            0.017368830740451813,
+            0.024747848510742188,
+            0.002688963897526264,
+            -0.0072461324743926525,
+            0.023966172710061073,
+            -0.007937915623188019,
+            0.009278489276766777,
+            0.02304379642009735,
+            0.004572802223265171,
+            0.012366107665002346,
+            -0.021542979404330254,
+            -0.006890470162034035,
+            -0.0059876348823308945,
+            0.0075079938396811485,
+            0.044680576771497726,
+            0.0025365371257066727,
+            0.027186676859855652,
+            0.028296655043959618,
+            -0.004799488000571728,
+            -0.009028352797031403,
+            0.0014529394684359431,
+            -0.013859108090400696,
+            0.046806734055280685,
+            0.0006092183757573366,
+            0.007265674415975809,
+            0.013468270190060139,
+            -0.05134045332670212,
+            -0.010779306292533875,
+            0.016024349257349968,
+            0.0012506808852776885,
+            0.03339318186044693,
+            -0.003781355684623122,
+            -0.0033846551086753607,
+            0.014804935082793236,
+            -0.035863276571035385,
+            -0.039677850902080536,
+            0.012366107665002346,
+            0.006018901709467173,
+            -0.011959636583924294,
+            -0.004299215506762266,
+            0.013749673031270504,
+            -0.005788307636976242,
+            -0.004346115980297327,
+            0.009184688329696655,
+            -0.006624700501561165,
+            0.008965819142758846,
+            0.009215954691171646,
+            -0.008739132434129715,
+            0.03986545279622078,
+            -0.009606792591512203,
+            0.03727029263973236,
+            0.0007645764271728694,
+            0.0010112927993759513,
+            0.006867019925266504,
+            -0.01979202590882778,
+            0.03345571458339691,
+            -0.0003559066681191325,
+            -0.04533718153834343,
+            0.0027827650774270296,
+            0.02082383632659912,
+            -0.008231043815612793,
+            0.009825661778450012,
+            -0.006233862601220608,
+            0.015711678192019463,
+            0.03952151536941528,
+            0.0055928886868059635,
+            0.01132647879421711,
+            0.017259396612644196,
+            0.0048581138253211975,
+            0.0018779755337163806,
+            0.031188854947686195,
+            0.0011090021580457687,
+            -0.009231588803231716,
+            0.029672404751181602,
+            -0.018713314086198807,
+            0.037457894533872604,
+            -0.047807276248931885,
+            -0.007414192892611027,
+            0.03561313822865486,
+            0.0005046692676842213,
+            -0.015383374877274036,
+            -0.04196034371852875,
+            0.025029251351952553,
+            -0.03229883313179016,
+            0.021183408796787262,
+            -0.003157969331368804,
+            0.033612050116062164,
+            -0.023825472220778465,
+            0.002618613187223673,
+            0.01883838139474392,
+            -0.013077432289719582,
+            0.02477911487221718,
+            0.014875286258757114,
+            -0.01064642146229744,
+            0.0021417911630123854,
+            -0.013225950300693512,
+            0.006738043390214443,
+            0.04646279662847519,
+            -0.009888196364045143,
+            0.004029537551105022,
+            0.007511902134865522,
+            -0.02224648743867874,
+            -0.018353741616010666,
+            0.04987090080976486,
+            0.0013239630497992039,
+            0.023293932899832726,
+            0.03661368414759636,
+            -0.041897810995578766,
+            -0.02168367989361286,
+            0.02135537751019001,
+            0.004912830889225006,
+            0.0037325008306652308,
+            0.014031076803803444,
+            0.016258852556347847,
+            0.014867469668388367,
+            -0.020386097952723503,
+            0.03411232307553291,
+            0.011131060309708118,
+            0.00669895950704813,
+            -0.019651323556900024,
+            -0.024169407784938812,
+            -0.020776936784386635,
+            -0.03398725390434265,
+            -0.02102707326412201,
+            -0.00927067268639803,
+            0.010200866498053074,
+            -0.008473362773656845,
+            -0.04883908852934837,
+            -0.017790935933589935,
+            0.026358099654316902,
+            -0.0069412789307534695,
+            0.04583745449781418,
+            0.0015369695611298084,
+            0.036957621574401855,
+            0.0012604518560692668,
+            -0.016993626952171326,
+            0.050590042024850845,
+            -0.027937084436416626,
+            0.04158513993024826,
+            0.022840559482574463,
+            -0.004025629255920649,
+            -0.008410829119384289,
+            0.04752587527036667,
+            -0.024122508242726326,
+            0.0032537246588617563,
+            -0.029688037931919098,
+            -0.003392471931874752,
+            -0.024951083585619926,
+            -0.0011197502026334405,
+            -0.0028746118769049644,
+            -0.008684415370225906,
+            0.024357011541724205,
+            -0.015242673456668854,
+            0.020401732996106148,
+            0.019385553896427155,
+            -0.0021437453106045723,
+            0.008739132434129715,
+            -0.020089061930775642,
+            0.012561526149511337,
+            -0.016399553045630455,
+            0.004568893928080797,
+            -0.009919462725520134,
+            0.001229184796102345,
+            -0.03279910609126091,
+            -0.00606189388781786,
+            -0.0067224097438156605,
+            -0.011193593963980675,
+            0.015367741696536541,
+            0.038552239537239075,
+            -0.0045884354040026665,
+            -0.035206668078899384,
+            0.007781580090522766,
+            -0.04008432477712631,
+            0.010169599205255508,
+            0.007144514936953783,
+            0.002470094710588455,
+            -0.009161237627267838,
+            0.01877584680914879,
+            0.022011984139680862,
+            -0.012772578746080399,
+            -0.033737119287252426,
+            0.009724044241011143,
+            -0.006038443651050329,
+            0.016524622216820717,
+            0.0029469169676303864,
+            0.0334869809448719,
+            -0.004009995609521866,
+            0.01880711503326893,
+            -0.012358291074633598,
+            -0.004693961702287197,
+            -0.017556432634592056,
+            0.0023293932899832726,
+            -0.02380983717739582,
+            -0.00526458490639925,
+            -0.00030778476502746344,
+            -0.013358835130929947,
+            -0.00802780780941248,
+            0.0027280477806925774,
+            0.011146693490445614,
+            -0.014156145043671131,
+            0.010732405818998814,
+            -0.019057249650359154,
+            -0.010044530965387821,
+            0.0072109573520720005,
+            -0.03226756677031517,
+            -0.014070160686969757,
+            -0.038552239537239075,
+            -0.003859523218125105,
+            0.013499537482857704,
+            0.03358078375458717,
+            -0.007562711369246244,
+            0.010724589228630066,
+            -0.030250845476984978,
+            -0.0007328208303079009,
+            0.023590968921780586,
+            0.023856738582253456,
+            -0.03545680269598961,
+            0.003523402614519,
+            0.025123052299022675,
+            0.0293441005051136,
+            -0.035206668078899384,
+            0.015625694766640663,
+            0.0351441353559494,
+            0.007093705702573061,
+            0.021371010690927505,
+            -0.0105291698127985,
+            0.042992155998945236,
+            -0.03786436468362808,
+            0.01160788256675005,
+            -0.004826846532523632,
+            0.024294476956129074,
+            0.007840205915272236,
+            0.024591512978076935,
+            -0.02049553394317627,
+            0.0328303724527359,
+            0.033612050116062164,
+            0.02032356522977352,
+            0.04418030381202698,
+            -0.03658241778612137,
+            -0.017415732145309448,
+            0.012538076378405094,
+            -0.01966695673763752,
+            0.01245209202170372,
+            -0.0023606601171195507,
+            -0.02481038309633732,
+            0.009755311533808708,
+            0.024325743317604065,
+            0.008637514896690845,
+            -0.059376075863838196,
+            0.023528434336185455,
+            0.02165241353213787,
+            0.0131008829921484,
+            0.02374730445444584,
+            -0.0371452234685421,
+            0.0004274788370821625,
+            0.015930548310279846,
+            0.015242673456668854,
+            0.020933272317051888,
+            0.017337564378976822,
+            -0.010349384509027004,
+            -0.0008661942556500435,
+            -0.023137597367167473,
+            0.006769310217350721,
+            -0.0035722574684768915,
+            0.010052347555756569,
+            -0.02646753564476967,
+            -0.028187220916152,
+            0.00872349925339222,
+            0.004994906950742006,
+            -0.0021535162813961506,
+            -0.03723902255296707,
+            -0.03933391347527504,
+            0.013749673031270504,
+            0.00819195993244648,
+            -0.044618040323257446,
+            -0.012921096757054329,
+            -0.03823956847190857,
+            -0.01447663176804781,
+            -0.021636780351400375,
+            0.0018261895747855306,
+            -0.010185232385993004,
+            0.0026928724255412817,
+            -0.005202050786465406,
+            0.025513891130685806,
+            -0.010451002046465874,
+            0.04552478343248367,
+            0.006566074676811695,
+            -0.003556623822078109,
+            -0.030125776305794716,
+            0.04802614822983742,
+            -0.007433734834194183,
+            -0.03933391347527504,
+            0.02454461343586445,
+            -0.017790935933589935,
+            0.011662599630653858,
+            0.03187672793865204,
+            -0.007949640974402428,
+            -0.014883102849125862,
+            0.013851291500031948,
+            -0.023262664675712585,
+            -0.042398083955049515,
+            0.007074163760989904,
+            -0.016337020322680473,
+            -0.0293441005051136,
+            -0.041460070759058,
+            -0.004834663588553667,
+            0.018713314086198807,
+            -0.025779660791158676,
+            0.004701778758317232,
+            0.01824430748820305,
+            0.019354287534952164,
+            -0.038520973175764084,
+            0.008551530539989471,
+            -0.00254044565372169,
+            -0.0017480220412835479,
+            -0.010787122882902622,
+            0.009591159410774708,
+            -0.0032380910124629736,
+            -0.004393016919493675,
+            -0.021449178457260132,
+            -0.005784399341791868,
+            -0.029203400015830994,
+            -0.010279033333063126,
+            -0.012006537057459354,
+            -0.027905818074941635,
+            -0.04589999094605446,
+            -0.04483691230416298,
+            0.025482622906565666,
+            0.03126702457666397,
+            -0.003568348940461874,
+            0.0028277114033699036,
+            -0.008379561826586723,
+            -0.012647510506212711,
+            0.019526256248354912,
+            0.00017880828818306327,
+            0.010771489702165127,
+            -0.0133510185405612,
+            0.016634056344628334,
+            -0.010279033333063126,
+            0.00540528679266572,
+            -0.04336735978722572,
+            -0.020370464771986008,
+            0.032986707985401154,
+            0.007062438875436783,
+            -0.0007914465386420488,
+            -0.004721320234239101,
+            0.013468270190060139,
+            -0.02434137649834156,
+            -0.01286637969315052,
+            0.03392472118139267,
+            0.01721249520778656,
+            0.01452353224158287,
+            -0.015000354498624802,
+            -0.008817300200462341,
+            0.003703187918290496,
+            -0.011412463150918484,
+            0.0049362811259925365,
+            -0.004615794401615858,
+            0.009075253270566463,
+            0.013804391026496887,
+            0.013569887727499008,
+            0.013898191973567009,
+            0.02235592156648636,
+            -0.009043985977768898,
+            -0.00649181567132473,
+            -0.003738363506272435,
+            0.008465546183288097,
+            -0.006272946484386921,
+            -0.006894378457218409,
+            -0.025873461738228798,
+            0.033674582839012146,
+            0.01890091598033905,
+            0.010998175479471684,
+            -0.015227040275931358,
+            -0.003757905215024948,
+            0.009356657043099403,
+            -0.009958546608686447,
+            -0.005823483224958181,
+            0.0037930808030068874,
+            0.025029251351952553,
+            -0.02194944955408573,
+            -0.013061799108982086,
+            -0.010865290649235249,
+            -0.0026479260995984077,
+            0.006870928220450878,
+            0.03773929551243782,
+            0.018431909382343292,
+            -0.008528080768883228,
+            -0.008488996885716915,
+            0.002384110586717725,
+            0.017337564378976822,
+            0.023919273167848587,
+            -0.03539426997303963,
+            -0.030751116573810577,
+            0.008105975575745106,
+            -0.0038829734548926353,
+            0.021136507391929626,
+            0.00532321073114872,
+            0.003744225949048996,
+            0.003537081880494952,
+            0.0722893550992012,
+            -0.0008554462110623717,
+            0.010067981667816639,
+            -0.017900370061397552,
+            -0.01520358957350254,
+            -0.0013493674341589212,
+            -0.005565530154854059,
+            -0.01308524888008833,
+            0.036957621574401855,
+            0.001335688168182969,
+            0.044618040323257446,
+            0.011865834705531597,
+            -0.006296396721154451,
+            -0.01190491858869791,
+            -0.0006805462762713432,
+            0.012084703892469406,
+            -0.0005608522333204746,
+            -0.0018662504153326154,
+            -0.0025658500380814075,
+            0.008973635733127594,
+            0.006499632261693478,
+            0.02570149302482605,
+            0.0005916306981816888,
+            0.017384463921189308,
+            -0.02424757555127144,
+            0.03601961210370064,
+            0.01711869426071644,
+            0.012842929922044277,
+            -0.019416820257902145,
+            -0.008207593113183975,
+            0.0028003526385873556,
+            0.0044633676297962666,
+            -0.007707321085035801,
+            -0.0006507448852062225,
+            -0.008660965599119663,
+            0.030454080551862717,
+            -0.03007887676358223,
+            0.01668095588684082,
+            0.011725133284926414,
+            -0.018197407945990562,
+            -0.005940734408795834,
+            0.015352108515799046,
+            -0.002077302895486355,
+            0.004459459334611893,
+            -0.006112703122198582,
+            0.0052450429648160934,
+            0.024357011541724205,
+            -0.028187220916152,
+            -0.006128336302936077,
+            0.010458819568157196,
+            -0.01493781991302967,
+            -0.0036445623263716698,
+            0.004221048206090927,
+            0.00956770870834589,
+            -0.014875286258757114,
+            0.011428097262978554,
+            0.02434137649834156,
+            -0.027937084436416626,
+            -0.009450457990169525,
+            0.011310845613479614,
+            -0.00010235064837615937,
+            0.01866641268134117,
+            0.023403367027640343,
+            0.008739132434129715,
+            -0.006448823492974043,
+            -0.0031423356849700212,
+            0.010341567918658257,
+            -0.009677143767476082,
+            -0.007468909956514835,
+            -0.0069842711091041565,
+            -0.010716771706938744,
+            0.01940118707716465,
+            -0.01591491512954235,
+            -0.0026928724255412817,
+            -0.003937690518796444,
+            -0.0106933219358325,
+            0.001270222826860845,
+            -0.013069615699350834,
+            0.017071794718503952,
+            0.0013249401235952973,
+            0.017931638285517693,
+            0.010951275005936623,
+            0.004846388474106789,
+            0.023590968921780586,
+            -0.012655327096581459,
+            0.012248856015503407,
+            -0.032017432153224945,
+            -0.0018027393380180001,
+            0.009145604446530342,
+            0.006022810470312834,
+            -0.01635265350341797,
+            -0.019948361441493034,
+            -0.013194683939218521,
+            0.0032244117464870214,
+            -0.009372290223836899,
+            0.0065465327352285385,
+            0.014296846464276314,
+            0.037926897406578064,
+            -0.016133783385157585,
+            -0.030469713732600212,
+            -0.015094155445694923,
+            0.012905463576316833,
+            0.016243217512965202,
+            -0.023888004943728447,
+            -0.009669327177107334,
+            -0.007269582711160183,
+            0.0034432809334248304,
+            -0.013171233236789703,
+            -0.01833810843527317,
+            -0.01668095588684082,
+            -0.003011405235156417,
+            -0.025889094918966293,
+            0.0013493674341589212,
+            -0.0043773832730948925,
+            0.0067224097438156605,
+            0.019479354843497276,
+            -0.0034725936129689217,
+            -0.02179311402142048,
+            -0.02490418404340744,
+            0.0175720676779747,
+            -0.02434137649834156,
+            0.008231043815612793,
+            0.025623325258493423,
+            -0.017853470519185066,
+            0.004928464535623789,
+            0.026670770719647408,
+            0.004948006477206945,
+            -0.009333206340670586,
+            0.012670961208641529,
+            0.009778761304914951,
+            0.006362839136272669,
+            0.008700049482285976,
+            0.0190728846937418,
+            0.012061254121363163,
+            0.020933272317051888,
+            -0.021605512127280235,
+            -0.013593338429927826,
+            -0.004025629255920649,
+            0.03626974672079086,
+            -0.0010327888885512948,
+            -0.00878603383898735,
+            0.014617333188652992,
+            0.0361446775496006,
+            0.005663239397108555,
+            -0.005585072096437216,
+            -0.01786910369992256,
+            0.0029586420860141516,
+            0.014601700007915497,
+            0.005291943438351154,
+            0.002679192926734686,
+            -0.00567105645313859,
+            -0.01048226933926344,
+            -0.00022790727962274104,
+            -0.0009438732522539794,
+            0.004697869997471571,
+            -0.026389367878437042,
+            0.018697679042816162,
+            -0.023419000208377838,
+            0.014171778224408627,
+            -0.0013063752558082342,
+            0.0007059507770463824,
+            0.03093871846795082,
+            -0.0022023709025233984,
+            -0.022340288385748863,
+            0.002501361770555377,
+            0.0019502805080264807,
+            0.013601155020296574,
+            0.022230852395296097,
+            -0.010943458415567875,
+            0.01261624414473772,
+            -0.003331892192363739,
+            0.0014382830122485757,
+            0.007461093366146088,
+            0.030469713732600212,
+            0.002923466730862856,
+            -0.011045075953006744,
+            0.020354831591248512,
+            0.009638059884309769,
+            0.0007425918010994792,
+            -0.007703412789851427,
+            0.020886370912194252,
+            -0.015602244064211845,
+            0.012491175904870033,
+            -0.012663144618272781,
+            -0.006323755253106356,
+            3.625936528806051e-07,
+            0.0010777352144941688,
+            -0.0023626144975423813,
+            -0.030688581988215446,
+            0.022965628653764725,
+            0.0005041807307861745,
+            0.0177440345287323,
+            -0.009505175054073334,
+            -0.019307386130094528,
+            0.002489636652171612,
+            0.020933272317051888,
+            -0.006855294574052095,
+            -0.01261624414473772,
+            -0.014304663054645061,
+            -0.014335930347442627,
+            -0.020073428750038147,
+            0.006695051211863756,
+            0.004060804378241301,
+            -0.015039438381791115,
+            0.0053740194998681545,
+            -0.003746180096641183,
+            -0.019119784235954285,
+            0.005280218552798033,
+            0.02410687506198883,
+            -0.027765115723013878,
+            0.03162659332156181,
+            -0.015860198065638542,
+            -0.015297390520572662,
+            0.009075253270566463,
+            0.0035175401717424393,
+            0.0005569438217207789,
+            0.005502996034920216,
+            -0.02135537751019001,
+            0.0020694860722869635,
+            0.020417366176843643,
+            -0.01704052835702896,
+            -0.0019932726863771677,
+            0.010075798258185387,
+            0.017493899911642075,
+            0.0007289124769158661,
+            -0.018025439232587814,
+            0.02490418404340744,
+            -0.016227584332227707,
+            0.02454461343586445,
+            0.02126157470047474,
+            -0.041804008185863495,
+            0.042116679251194,
+            -0.024654047563672066,
+            0.015828929841518402,
+            0.002237546257674694,
+            -0.006437098141759634,
+            0.013069615699350834,
+            0.018885280936956406,
+            -0.005256768316030502,
+            -0.0012780395336449146,
+            0.002804261166602373,
+            0.03167349472641945,
+            -0.023403367027640343,
+            0.024528980255126953,
+            0.02132410928606987,
+            -0.00010387735528638586,
+            0.02481038309633732,
+            -0.021230308338999748,
+            -0.029766205698251724,
+            0.019620057195425034,
+            -0.023465899750590324,
+            -0.034174855798482895,
+            0.020042162388563156,
+            -0.008418645709753036,
+            -0.008184143342077732,
+            -0.020339198410511017,
+            0.006386289373040199,
+            0.0054639121517539024,
+            -0.013007081113755703,
+            0.002849207492545247,
+            0.0027397728990763426,
+            0.02606106363236904,
+            0.022574789822101593,
+            0.01520358957350254,
+            -0.003320167073979974,
+            0.018166139721870422,
+            -0.0015731221064925194,
+            0.032580237835645676,
+            -0.0020304021891206503,
+            0.0004528832796495408,
+            -0.011052892543375492,
+            -0.0021456994581967592,
+            0.0034198306966573,
+            0.0039220573380589485,
+            0.017384463921189308,
+            -0.02069876901805401,
+            -0.015868013724684715,
+            -0.0351441353559494,
+            -0.0025971170980483294,
+            -0.003539036260917783,
+            0.010708955116569996,
+            0.03508159890770912,
+            -0.00903616938740015,
+            -0.013999809511005878,
+            0.009192504920065403,
+            -0.0068083941005170345,
+            -0.0002442736295051873,
+            -0.02330956608057022,
+            -0.020354831591248512,
+            -0.01493781991302967,
+            -0.013272850774228573,
+            -0.008856384083628654,
+            -0.020042162388563156,
+            0.012960180640220642,
+            -0.010599520988762379,
+            0.01048226933926344,
+            -0.02795271761715412,
+            0.014406280592083931,
+            -0.012280123308300972,
+            0.009059620089828968,
+            -0.005456095561385155,
+            -0.024200676009058952,
+            -0.004357841331511736,
+            0.02653006836771965,
+            -0.0034745479933917522,
+            -0.021449178457260132,
+            -0.01786910369992256,
+            0.005202050786465406,
+            -0.014844018965959549,
+            -0.0016180684324353933,
+            -0.015101972036063671,
+            -0.009614609181880951,
+            0.0008300417684949934,
+            0.0009849112248048186,
+            0.005475637502968311,
+            0.010443185456097126,
+            -0.0029332374688237906,
+            -0.018150506541132927,
+            0.014015442691743374,
+            0.04139753803610802,
+            -0.04386763274669647,
+            -0.017134329304099083,
+            0.012076887302100658,
+            0.010404101572930813,
+            -0.0036308830603957176,
+            0.010474452748894691,
+            -0.013554254546761513,
+            -0.003349479753524065,
+            -0.024388277903199196,
+            -0.01308524888008833,
+            -0.02513868547976017,
+            0.00749626848846674,
+            0.017931638285517693,
+            -0.008137242868542671,
+            0.03273657336831093,
+            0.030391545966267586,
+            -0.01877584680914879,
+            -0.0013884512009099126,
+            0.010583886876702309,
+            0.015281757339835167,
+            -0.029031431302428246,
+            0.016368286684155464,
+            -0.009911646135151386,
+            0.021167773753404617,
+            0.00968496035784483,
+            -0.021152140572667122,
+            -0.0146720502525568,
+            -0.016524622216820717,
+            0.007011629641056061,
+            -0.013874741271138191,
+            -3.935858694603667e-05,
+            0.027264844626188278,
+            0.015500626526772976,
+            -0.007484543602913618,
+            -0.014109243638813496,
+            0.018650779500603676,
+            0.013335385359823704,
+            0.019573155790567398,
+            -0.00540528679266572,
+            -0.0038399812765419483,
+            -0.007801122032105923,
+            -0.00946609117090702,
+            -0.018947815522551537,
+            0.00927067268639803,
+            -0.02984437346458435,
+            0.019057249650359154,
+            -0.01854134537279606,
+            -0.007437643129378557,
+            0.03273657336831093,
+            -0.04652532935142517,
+            -0.012913280166685581,
+            -0.013280668295919895,
+            0.02895326353609562,
+            0.015977447852492332,
+            0.004623610991984606,
+            0.009919462725520134,
+            -0.021449178457260132,
+            0.0033729299902915955,
+            -0.00321659492328763,
+            0.003099343739449978,
+            -0.016837291419506073,
+            -0.016399553045630455,
+            0.012413008138537407,
+            0.002137882634997368,
+            -0.004228864796459675,
+            0.0069061038084328175,
+            -0.005420919973403215,
+            -0.013538620434701443,
+            0.002679192926734686,
+            -0.0007513856398873031,
+            0.0015975494170561433,
+            0.008817300200462341,
+            0.002515041036531329,
+            0.01495345402508974,
+            0.01112324371933937,
+            -0.008660965599119663,
+            -0.005647606216371059,
+            -0.005647606216371059,
+            0.00088085065362975,
+            0.010153966024518013,
+            0.005284126847982407,
+            -0.006398014724254608,
+            -0.011631332337856293,
+            0.003376838518306613,
+            -0.007996541447937489,
+            0.026186130940914154,
+            -0.008418645709753036,
+            -0.042335547506809235,
+            0.002751498017460108,
+            -0.042366817593574524,
+            0.02354406751692295,
+            -0.02573275938630104,
+            -0.020651869475841522,
+            -0.009974180720746517,
+            -0.003695371327921748,
+            -0.019979627802968025,
+            0.009380106814205647,
+            0.007121064234524965,
+            -0.007941823452711105,
+            0.00792619027197361,
+            0.001134406658820808,
+            -0.015023804269731045,
+            0.0105291698127985,
+            -0.0101148821413517,
+            -0.010552620515227318,
+            0.005710139870643616,
+            0.0073868343606591225,
+            0.008848567493259907,
+            -0.006272946484386921,
+            -0.007472818251699209,
+            0.009505175054073334,
+            0.014734584838151932,
+            0.005229409784078598,
+            0.0053740194998681545,
+            0.020667502656579018,
+            0.013319752179086208,
+            -0.022496622055768967,
+            0.0023606601171195507,
+            -0.008019991219043732,
+            0.0038380271289497614,
+            0.003476502140983939,
+            -0.005932917352765799,
+            -0.009192504920065403,
+            0.003963095135986805,
+            -0.0003766699228435755,
+            4.7236411774065346e-05,
+            -0.0018340062815696,
+            -0.01614941656589508,
+            -0.005491270683705807,
+            -0.012006537057459354,
+            0.004221048206090927,
+            0.0300163421779871,
+            -0.006233862601220608,
+            -0.006398014724254608,
+            -0.019682591781020164,
+            0.018494443967938423,
+            0.03404978662729263,
+            0.017556432634592056,
+            -0.028671860694885254,
+            0.0005080891423858702,
+            -0.006417556665837765,
+            0.003300625132396817,
+            -0.025513891130685806,
+            0.022856194525957108,
+            0.018744580447673798,
+            0.009458274580538273,
+            0.011834568344056606,
+            -0.0025502166245132685,
+            0.005721865221858025,
+            -0.01548499334603548,
+            0.012858563102781773,
+            -0.004557168576866388,
+            -0.012014353647828102,
+            0.005006631836295128,
+            0.03942771628499031,
+            -0.015993082895874977,
+            -0.009677143767476082,
+            0.0030192218255251646,
+            0.02424757555127144,
+            -0.011889285407960415,
+            -0.007238315884023905,
+            0.011717316694557667,
+            0.009700593538582325,
+            0.004580618813633919,
+            0.00878603383898735,
+            0.06472273916006088,
+            0.0021867374889552593,
+            0.01638391986489296,
+            0.01283511333167553,
+            -0.021136507391929626,
+            -0.01919795200228691,
+            -0.005780491046607494,
+            -0.020354831591248512,
+            0.009012719616293907,
+            0.00900490302592516,
+            -0.011576615273952484,
+            -0.004068621434271336,
+            -0.020214131101965904,
+            0.0060736192390322685,
+            0.00731257488951087,
+            0.011936185881495476,
+            -0.0024192859418690205,
+            0.006870928220450878,
+            0.04274202138185501,
+            0.019448088482022285,
+            -0.0017196862027049065,
+            0.008105975575745106,
+            0.01419522799551487,
+            -0.009489541873335838,
+            -0.012420824728906155,
+            -0.002659651217982173,
+            0.0026107963640242815,
+            0.00877039972692728,
+            -0.010927824303507805,
+            0.033205579966306686,
+            -0.010419735684990883,
+            -0.015774212777614594,
+            0.011005992069840431,
+            -0.008066891692578793,
+            0.011013808660209179,
+            -0.012311390601098537,
+            0.017916003242135048,
+            -0.016665322706103325,
+            -0.0033221212215721607,
+            -0.0002916626981459558,
+            0.00019004486966878176,
+            0.017603334039449692,
+            0.0018095789710059762,
+            -0.010497902520000935,
+            0.0016815796261653304,
+            0.013851291500031948,
+            -0.008590614423155785,
+            0.003435464110225439,
+            -0.006261221133172512,
+            0.003959186840802431,
+            -0.011420279741287231,
+            -0.015555343590676785,
+            0.007191415410488844,
+            0.018822748214006424,
+            -0.026154864579439163,
+            -0.013210317119956017,
+            0.026576969772577286,
+            0.0071601481176912785,
+            0.002671376336365938,
+            0.013593338429927826,
+            -0.0058117578737437725,
+            0.0030954352114349604,
+            -0.016243217512965202,
+            -0.01777530275285244,
+            0.0061635118909180164,
+            -0.0048932889476418495,
+            0.019557522609829903,
+            -0.0024056066758930683,
+            0.004514176398515701,
+            -0.002837482374161482,
+            -0.008535897359251976,
+            0.019510623067617416,
+            0.015430275350809097,
+            -0.005170783959329128,
+            -0.006269038189202547,
+            0.011459363624453545,
+            -0.01827557571232319,
+            -0.020948905497789383,
+            0.003243953688070178,
+            0.020401732996106148,
+            -0.0065113576129078865,
+            0.004994906950742006,
+            -0.00030558629077859223,
+            -0.0012457953998818994,
+            -0.014664233662188053,
+            -0.009974180720746517,
+            -0.020902004092931747,
+            0.01919795200228691,
+            0.01336665265262127,
+            -0.02251225709915161,
+            0.0030192218255251646,
+            -0.005932917352765799,
+            0.02052680030465126,
+            -0.017587700858712196,
+            -0.003423738991841674,
+            0.033080510795116425,
+            -0.017759669572114944,
+            -0.004557168576866388,
+            -0.008231043815612793,
+            -0.014930003322660923,
+            0.016790391877293587,
+            0.03867730870842934,
+            0.0012037804117426276,
+            0.01014614850282669,
+            0.0029156499076634645,
+            -0.00452590174973011,
+            0.03558187186717987,
+            0.012647510506212711,
+            0.0004885472590103745,
+            -0.0032771748956292868,
+            -0.013835657387971878,
+            -0.02626429870724678,
+            0.0049440981820225716,
+            0.0026518343947827816,
+            -0.002016722923144698,
+            -5.20302819495555e-05,
+            -0.006014993414282799,
+            -0.007906648330390453,
+            0.006659876089543104,
+            -0.021668046712875366,
+            0.007797213736921549,
+            -0.016931092366576195,
+            -0.00276517728343606,
+            -0.00231375964358449,
+            -0.012796029448509216,
+            -0.005303668789565563,
+            0.010802756063640118,
+            -0.0016786482883617282,
+            -0.026123598217964172,
+            -0.03404978662729263,
+            -0.011670416221022606,
+            -0.012311390601098537,
+            -0.0007938892231322825,
+            0.00760179478675127,
+            -0.0267020370811224,
+            0.010224316269159317,
+            -0.0095364423468709,
+            -0.0014138557016849518,
+            0.0013679323019459844,
+            0.011983086355030537,
+            -0.006968637928366661,
+            0.0020362648647278547,
+            -0.02925029955804348,
+            -0.023434633389115334,
+            0.02135537751019001,
+            -0.003925965633243322,
+            0.012303573079407215,
+            -0.012538076378405094,
+            -0.004846388474106789,
+            -0.012592793442308903,
+            0.0177440345287323,
+            -0.017603334039449692,
+            -0.014007626101374626,
+            -0.014617333188652992,
+            -0.0005115089588798583,
+            -0.005366202909499407,
+            -0.009231588803231716,
+            -0.010083614848554134,
+            -0.010638604871928692,
+            -0.021152140572667122,
+            -0.014812751673161983,
+            -0.008442096412181854,
+            0.001930738682858646,
+            0.020151596516370773,
+            -0.006237770896404982,
+            0.026311200112104416,
+            0.02126157470047474,
+            0.00823886040598154,
+            -0.00520595908164978,
+            0.01425776258111,
+            -0.006937370635569096,
+            0.007679962553083897,
+            -0.00952080823481083,
+            0.0003239068028051406,
+            0.01671222411096096,
+            0.012850746512413025,
+            0.008410829119384289,
+            -0.004197597969323397,
+            0.002706551691517234,
+            0.01916668564081192,
+            0.015078521333634853,
+            0.01664968952536583,
+            0.005291943438351154,
+            0.008731315843760967,
+            0.010615154169499874,
+            0.0015399008989334106,
+            -0.005377927795052528,
+            0.0008955070516094565,
+            0.011701683513820171,
+            -0.011568798683583736,
+            -0.02005779556930065,
+            0.01329630147665739,
+            0.00930193904787302,
+            -0.018916549161076546,
+            0.009215954691171646,
+            0.0006575845764018595,
+            -0.009645876474678516,
+            -0.01836937665939331,
+            -0.014586065895855427,
+            -0.007805030792951584,
+            0.015430275350809097,
+            0.007918373681604862,
+            -0.022637324407696724,
+            0.01567259430885315,
+            0.010153966024518013,
+            -0.0015311070019379258,
+            -0.02712414227426052,
+            0.018619511276483536,
+            0.014054526574909687,
+            0.010638604871928692,
+            -0.04821375012397766,
+            -0.006069710943847895,
+            0.020730035379529,
+            -0.01329630147665739,
+            -0.026076696813106537,
+            -0.007625245023518801,
+            -0.008879834786057472,
+            -0.03902124613523483,
+            -0.0030758932698518038,
+            0.0029586420860141516,
+            -0.014844018965959549,
+            -0.003974820487201214,
+            0.015539710409939289,
+            0.00391228636726737,
+            0.018979083746671677,
+            -0.007781580090522766,
+            -0.01336665265262127,
+            -0.019213585183024406,
+            0.005202050786465406,
+            -0.010787122882902622,
+            -0.024263210594654083,
+            0.0027554063126444817,
+            -0.008465546183288097,
+            0.0022180043160915375,
+            0.01700926013290882,
+            0.042835820466279984,
+            0.01786910369992256,
+            -0.019917093217372894,
+            -0.025357555598020554,
+            -0.019307386130094528,
+            -0.007953548803925514,
+            0.009137787856161594,
+            0.007179690059274435,
+            -0.0021456994581967592,
+            0.003595707705244422,
+            0.006593433208763599,
+            0.031923629343509674,
+            0.0009536442230455577,
+            0.020886370912194252,
+            -3.169450064888224e-05,
+            -0.003814576892182231,
+            -0.002132020192220807,
+            -0.018979083746671677,
+            -0.0016053662402555346,
+            0.0029664586763828993,
+            0.013139965943992138,
+            0.004826846532523632,
+            0.005385744851082563,
+            0.016665322706103325,
+            -0.005628064274787903,
+            -0.0036035242956131697,
+            -0.017056161537766457,
+            -0.004428192041814327,
+            -0.004330482799559832,
+            0.03717648983001709,
+            0.020636234432458878,
+            -0.0199014600366354,
+            -0.0165871549397707,
+            -0.017431365326046944,
+            0.02194944955408573,
+            -0.00830921158194542,
+            -0.008997085504233837,
+            0.02099580504000187,
+            0.010239950381219387,
+            0.0015311070019379258,
+            0.0027280477806925774,
+            -0.006612975150346756,
+            0.011357746087014675,
+            0.02570149302482605,
+            0.012561526149511337,
+            -0.001964936964213848,
+            -0.004002179019153118,
+            -0.00016915948071982712,
+            -0.019323019310832024,
+            -0.01641518622636795,
+            0.00010302239388693124,
+            -0.012928914278745651,
+            0.025982895866036415,
+            -0.023106329143047333,
+            0.004314849153161049,
+            -0.01424212846904993,
+            -0.0024075608234852552,
+            -0.021996350958943367,
+            -0.011514080688357353,
+            0.014758034609258175,
+            -0.0237160362303257,
+            0.0054170116782188416,
+            -0.016759123653173447,
+            0.0010660100961104035,
+            0.011209228076040745,
+            -0.005159058608114719,
+            0.012561526149511337,
+            -0.00763697037473321,
+            0.004084255080670118,
+            0.004287490621209145,
+            0.00872349925339222,
+            -0.009145604446530342,
+            0.0015164506621658802,
+            0.005538171622902155,
+            -0.014578249305486679,
+            0.009239405393600464,
+            0.00925503857433796,
+            -0.004748679231852293,
+            0.03917757794260979,
+            0.015281757339835167,
+            0.00014949544856790453,
+            -0.009395739994943142,
+            -0.0038673398084938526,
+            -0.03140772506594658,
+            0.027014708146452904,
+            -0.012663144618272781,
+            0.021386643871665,
+            0.005983726587146521,
+            -0.01490655355155468,
+            -0.0016161142848432064,
+            -0.002688963897526264,
+            0.013194683939218521,
+            -0.009935096837580204,
+            0.0008969727205112576,
+            -0.013624604791402817,
+            0.011990902945399284,
+            0.00476822117343545,
+            -0.01016178261488676,
+            0.012295756489038467,
+            -0.015305208042263985,
+            0.0016571521991863847,
+            -0.007183598354458809,
+            0.0022844469640403986,
+            0.01588364690542221,
+            -0.0033416631631553173,
+            0.0028589784633368254,
+            -0.034675128757953644,
+            -0.015570977702736855,
+            -0.011529714800417423,
+            -8.244234777521342e-05,
+            0.007976999506354332,
+            -0.01754079945385456,
+            -0.028077786788344383,
+            -0.015977447852492332,
+            0.0099507300183177,
+            -0.0005926077719777822,
+            0.00028848715010099113,
+            -0.004658786579966545,
+            0.005225501023232937,
+            0.0009604838560335338,
+            -0.007468909956514835,
+            0.017525166273117065,
+            0.014875286258757114,
+            -0.026358099654316902,
+            -0.03081365115940571,
+            0.004564985167235136,
+            -0.0003722730034496635,
+            -0.027608782052993774,
+            -0.014077977277338505,
+            0.001066987169906497,
+            -0.0006072642281651497,
+            0.021199041977524757,
+            -0.020776936784386635,
+            0.0014588020276278257,
+            -0.00043260856182314456,
+            0.0030055425595492125,
+            -0.036394815891981125,
+            0.0030954352114349604,
+            -0.003144290065392852,
+            -0.0029273750260472298,
+            0.005565530154854059,
+            -0.01542245876044035,
+            0.0013278713449835777,
+            -0.008864200673997402,
+            -0.011428097262978554,
+            -0.025748392567038536,
+            -0.004592344164848328,
+            0.001534038339741528,
+            -8.647285721963271e-05,
+            0.023121964186429977,
+            -0.00622604601085186,
+            -0.019776392728090286,
+            0.002987954765558243,
+            -0.0019424638012424111,
+            -0.0030758932698518038,
+            0.018447542563080788,
+            0.020010894164443016,
+            -0.00651526590809226,
+            -0.0015105879865586758,
+            -0.0016258851392194629,
+            -0.001716754981316626,
+            -0.0004516619083005935,
+            0.039146311581134796,
+            -0.008551530539989471,
+            -0.006882653571665287,
+            0.0050261737778782845,
+            0.0016591063467785716,
+            -0.005444370210170746,
+            0.012100338004529476,
+            0.0015594427241012454,
+            0.007918373681604862,
+            0.0018388917669653893,
+            -0.010130515322089195,
+            -0.01824430748820305,
+            0.002528720535337925,
+            0.013851291500031948,
+            0.00581957446411252,
+            0.004197597969323397,
+            -0.014562616124749184,
+            0.008082525804638863,
+            -0.0075978864915668964,
+            -0.008778216317296028,
+            -0.010771489702165127,
+            0.007816756144165993,
+            -0.00444382568821311,
+            0.019526256248354912,
+            0.0003461357264313847,
+            0.021480444818735123,
+            0.022340288385748863,
+            -0.008442096412181854,
+            0.017024893313646317,
+            0.031485892832279205,
+            8.806063851807266e-05,
+            0.004744770936667919,
+            -0.011365562677383423,
+            -0.014679866842925549,
+            -0.013233767822384834,
+            0.03389345109462738,
+            0.004541534930467606,
+            -0.004267948679625988,
+            0.0018193499417975545,
+            -0.007316483184695244,
+            -0.01283511333167553,
+            0.012413008138537407,
+            0.003392471931874752,
+            -0.018556978553533554,
+            -0.004740862175822258,
+            0.009239405393600464,
+            0.006878744810819626,
+            0.012928914278745651,
+            0.02158987894654274,
+            -0.003869294188916683,
+            0.0028316196985542774,
+            0.004959731362760067,
+            0.029953807592391968,
+            -0.006374564487487078,
+            0.028593692928552628,
+            -0.0062494962476193905,
+            -0.0034823645837605,
+            -0.01107634324580431,
+            -0.015750762075185776,
+            -0.008426462300121784,
+            0.0029312833212316036,
+            0.0030602598562836647,
+            0.006186962127685547,
+            -0.01833810843527317,
+            0.006257312837988138,
+            0.02809341996908188,
+            0.0030055425595492125,
+            0.014734584838151932,
+            0.009278489276766777,
+            0.012741311453282833,
+            0.017321931198239326,
+            -0.00479167141020298,
+            -0.014851835556328297,
+            -0.003451097756624222,
+            0.020808203145861626,
+            -0.006683326326310635,
+            0.024654047563672066,
+            -0.0034745479933917522,
+            0.004350024741142988,
+            -0.00021972411195747554,
+            -0.0038810193073004484,
+            0.020479900762438774,
+            0.01500817108899355,
+            0.01664968952536583,
+            -0.001603411976248026,
+            0.00850463006645441,
+            0.0018349833553656936,
+            0.0017480220412835479,
+            -0.012913280166685581,
+            -0.016508987173438072,
+            0.01786910369992256,
+            0.0038477980997413397,
+            0.023168863728642464,
+            -0.022199586033821106,
+            -0.0007113247411325574,
+            0.006018901709467173,
+            0.016430821269750595,
+            0.004131155554205179,
+            0.010552620515227318,
+            -0.005444370210170746,
+            0.009544258937239647,
+            -0.005905558820813894,
+            0.0016913504805415869,
+            -0.01824430748820305,
+            -0.011600065045058727,
+            0.011631332337856293,
+            -0.00819195993244648,
+            0.010685505345463753,
+            0.001274131122045219,
+            0.009622426703572273,
+            -0.013843473978340626,
+            0.029672404751181602,
+            0.021402277052402496,
+            0.015234856866300106,
+            0.01499253697693348,
+            -0.0024779115337878466,
+            -0.007910557091236115,
+            -0.012420824728906155,
+            0.024560246616601944,
+            0.006800577510148287,
+            0.0029156499076634645,
+            -0.004475092515349388,
+            0.024153774604201317,
+            -0.0004384711501188576,
+            0.005151242017745972,
+            -0.0005886994185857475,
+            0.012967998161911964,
+            -0.01638391986489296,
+            0.008825116790831089,
+            -0.023559702560305595,
+            -0.013210317119956017,
+            -0.01913541741669178,
+            0.019588788971304893,
+            0.012741311453282833,
+            -0.0047721294686198235,
+            0.01543809287250042,
+            0.017603334039449692,
+            -0.0027319560758769512,
+            -0.027280477806925774,
+            -0.009786577895283699,
+            1.4351075151353143e-05,
+            -0.0032380910124629736,
+            -0.015727313235402107,
+            -0.0099507300183177,
+            0.002714368514716625,
+            -0.004885472357273102,
+            0.015547527000308037,
+            -0.0037637678906321526,
+            -0.024591512978076935,
+            -0.005686689633876085,
+            -0.007175781764090061,
+            -0.016555888578295708,
+            -0.013577704317867756,
+            0.008184143342077732,
+            0.005749223753809929,
+            -0.011506264097988605,
+            -0.018916549161076546,
+            0.018494443967938423,
+            -0.004693961702287197,
+            0.03214249759912491,
+            -0.015664778649806976,
+            -0.002808169461786747,
+            0.0053740194998681545,
+            0.01940118707716465,
+            0.0002865329443011433,
+            -0.0021183406934142113,
+            0.009638059884309769,
+            0.010615154169499874,
+            0.003232228569686413,
+            -0.007734679616987705,
+            0.0012917189160361886,
+            -0.0002904413267970085,
+            -0.019385553896427155,
+            0.005893833935260773,
+            -0.0005007609142921865,
+            -0.014070160686969757,
+            0.003351433901116252,
+            0.014335930347442627,
+            0.009622426703572273,
+            0.005452187266200781,
+            -0.02789018489420414,
+            0.012897646985948086,
+            -0.008582797832787037,
+            -0.011256128549575806,
+            -0.0016356561100110412,
+            0.01851007714867592,
+            -0.021965082734823227,
+            0.027937084436416626,
+            0.01419522799551487,
+            -0.00941919069737196,
+            0.005166875664144754,
+            0.019948361441493034,
+            0.017024893313646317,
+            0.011474997736513615,
+            -0.023794203996658325,
+            -0.004221048206090927,
+            0.01425776258111,
+            0.006304213311523199,
+            -0.0006717524374835193,
+            -0.0020206314511597157,
+            0.02463841438293457,
+            0.0034139680210500956,
+            -0.0018310750601813197,
+            -0.0070272632874548435,
+            -0.0018193499417975545,
+            0.009309755638241768,
+            0.021183408796787262,
+            0.009927279315888882,
+            0.018947815522551537,
+            -0.00131028366740793,
+            0.01578984595835209,
+            -0.004295307211577892,
+            -0.009669327177107334,
+            0.014844018965959549,
+            -0.0005882108816877007,
+            -0.009841295890510082,
+            0.01668095588684082,
+            0.008418645709753036,
+            0.012373924255371094,
+            -0.0008901330293156207,
+            -0.009872562251985073,
+            0.02871876023709774,
+            0.015109788626432419,
+            -0.0008701026090420783,
+            0.02046426571905613,
+            -0.02685837261378765,
+            0.024951083585619926,
+            0.012772578746080399,
+            0.0160868838429451,
+            -0.022340288385748863,
+            -0.004475092515349388,
+            0.00048488311585970223,
+            -0.018259940668940544,
+            -0.008700049482285976,
+            0.0037950349505990744,
+            -0.009989813901484013,
+            -0.0020264938939362764,
+            -0.037395358085632324,
+            0.01303053181618452,
+            -0.009427007287740707,
+            0.003249816130846739,
+            -0.015227040275931358,
+            -0.001274131122045219,
+            0.00532321073114872,
+            0.008543713949620724,
+            -0.007746404968202114,
+            -0.030469713732600212,
+            -0.004744770936667919,
+            0.016759123653173447,
+            -0.00835611205548048,
+            -0.019354287534952164,
+            0.024716582149267197,
+            -0.0004824404022656381,
+            -0.021699313074350357,
+            0.0041155219078063965,
+            -0.010544802993535995,
+            -0.007363383658230305,
+            -0.005518629681318998,
+            -0.008105975575745106,
+            -0.02424757555127144,
+            0.013843473978340626,
+            0.0019102196674793959,
+            -0.02046426571905613,
+            0.010232132859528065,
+            -0.01378094032406807,
+            0.004111613612622023,
+            0.008551530539989471,
+            -0.008895467966794968,
+            -0.011748583987355232,
+            0.027749482542276382,
+            0.022324655205011368,
+            -0.017618967220187187,
+            -0.015742946416139603,
+            9.386213787365705e-05,
+            -0.025060519576072693,
+            0.023700403049588203,
+            -0.00470959534868598,
+            0.0019825247582048178,
+            -0.011131060309708118,
+            0.011600065045058727,
+            0.010826206766068935,
+            0.012624060735106468,
+            -0.003554669674485922,
+            -0.0005579209537245333,
+            0.026514435186982155,
+            -0.009090886451303959,
+            0.003509723348543048,
+            -8.564843483327422e-06,
+            -0.012413008138537407,
+            0.02384110540151596,
+            -0.004486817866563797,
+            0.023950539529323578,
+            -0.006237770896404982,
+            -0.0006483022007159889,
+            -0.014961270615458488,
+            0.0026440175715833902,
+            -0.0031462442129850388,
+            -0.0069842711091041565,
+            -0.010286850854754448,
+            0.01283511333167553,
+            0.01213942188769579,
+            0.006835752632468939,
+            -0.010255583561956882,
+            0.0012184367515146732,
+            0.00038790650432929397,
+            -0.016696590930223465,
+            0.0023489349987357855,
+            -0.019713858142495155,
+            0.0009829570772126317,
+            -0.03329937905073166,
+            0.009708410128951073,
+            0.04430537298321724,
+            0.016634056344628334,
+            -0.008129426278173923,
+            -0.003537081880494952,
+            -0.01611815020442009,
+            0.005225501023232937,
+            -0.005706231575459242,
+            -0.011725133284926414,
+            -0.0136793227866292,
+            -0.01605561561882496,
+            0.0035312194377183914,
+            0.017916003242135048,
+            -0.0026518343947827816,
+            -0.020745670422911644,
+            0.02298126183450222,
+            -0.014500081539154053,
+            -0.026655137538909912,
+            -0.01754079945385456,
+            -0.02096453867852688,
+            0.007324300240725279,
+            0.020745670422911644,
+            -0.00579612422734499,
+            0.026576969772577286,
+            0.012428641319274902,
+            0.0028140321373939514,
+            -0.0010904374066740274,
+            0.01813487336039543,
+            -0.011349929496645927,
+            0.00452590174973011,
+            -0.0013816115679219365,
+            -0.020354831591248512,
+            0.0014783439692109823,
+            -0.008715682663023472,
+            -0.005600705277174711,
+            0.001242864178493619,
+            -0.005624155513942242,
+            -0.00260297954082489,
+            -0.0075978864915668964,
+            -0.017728401347994804,
+            0.02327829785645008,
+            -0.02348153479397297,
+            -0.002851161640137434,
+            -0.0005007609142921865,
+            -0.009591159410774708,
+            0.0111076096072793,
+            0.023950539529323578,
+            0.01261624414473772,
+            -0.015430275350809097,
+            0.01833810843527317,
+            -0.00037154016899876297,
+            0.026545701548457146,
+            -0.004662694875150919,
+            0.005585072096437216,
+            0.012694410979747772,
+            0.0008129425696097314,
+            0.01685292460024357,
+            -0.00710543105378747,
+            -0.011029441840946674,
+            0.0042913989163935184,
+            -0.00018686932162381709,
+            -0.021402277052402496,
+            -0.009755311533808708,
+            -0.0023782479111105204,
+            0.012217588722705841,
+            -0.017165595665574074,
+            0.015727313235402107,
+            -0.012741311453282833,
+            -0.001535992487333715,
+            -0.00387124833650887,
+            0.01397635880857706,
+            -0.04855768755078316,
+            -0.004963639657944441,
+            -0.008199776522815228,
+            0.03489399701356888,
+            -0.012444275431334972,
+            0.022434089332818985,
+            0.0040002246387302876,
+            -0.007621336728334427,
+            0.011834568344056606,
+            -0.010130515322089195,
+            -0.0004521504743024707,
+            0.015070704743266106,
+            -0.007347750477492809,
+            0.0017431365558877587,
+            0.016993626952171326,
+            -0.01836937665939331,
+            0.01674349047243595,
+            -0.019541889429092407,
+            0.0031071603298187256,
+            -0.012061254121363163,
+            -0.00640973960980773,
+            0.021386643871665,
+            0.0267020370811224,
+            -0.005815666168928146,
+            0.007398559246212244,
+            -0.009184688329696655,
+            0.007691687438637018,
+            -0.004252315033227205,
+            0.011545347981154919,
+            0.0010748038766905665,
+            -0.010232132859528065,
+            -0.0019317157566547394,
+            -0.009661509655416012,
+            0.020651869475841522,
+            -0.013382285833358765,
+            -0.012491175904870033,
+            -0.009559892117977142,
+            -0.004803396295756102,
+            -0.03351824730634689,
+            -0.005983726587146521,
+            -0.008137242868542671,
+            -0.012483359314501286,
+            -0.008426462300121784,
+            0.012733494862914085,
+            0.0010884832590818405,
+            0.0024290569126605988,
+            -0.009544258937239647,
+            -0.01144373044371605,
+            -0.013866924680769444,
+            0.015977447852492332,
+            0.004052987787872553,
+            -0.004111613612622023,
+            0.0038380271289497614,
+            0.00037056306609883904,
+            0.004795579705387354,
+            -0.00015438091941177845,
+            -0.00807470828294754,
+            -0.0032830373384058475,
+            0.012381740845739841,
+            0.006398014724254608,
+            -0.021011440083384514,
+            0.010388468392193317,
+            0.01893218234181404,
+            -0.0033572965767234564,
+            -0.006057985592633486,
+            -0.0064605483785271645,
+            -0.013382285833358765,
+            -0.004557168576866388,
+            0.0017665867926552892,
+            0.020917639136314392,
+            -0.01661842316389084,
+            -0.0013845429057255387,
+            0.019541889429092407,
+            -0.0027749482542276382,
+            0.023028161376714706,
+            0.0025384915061295033,
+            0.009935096837580204,
+            0.00988037884235382,
+            0.011740767396986485,
+            0.0062924884259700775,
+            -0.010951275005936623,
+            -0.02162114717066288,
+            0.001587778446264565,
+            -0.022434089332818985,
+            0.01588364690542221,
+            0.0023333015851676464,
+            -0.005100433249026537,
+            -0.021840015426278114,
+            -0.012506809085607529,
+            0.009512991644442081,
+            -0.02971930429339409,
+            -0.0007259811973199248,
+            -0.00737120071426034,
+            0.014265579171478748,
+            -0.007879289798438549,
+            0.0033866094890981913,
+            0.006042351946234703,
+            -0.007566619664430618,
+            0.00028799858409911394,
+            -0.0024505530018359423,
+            -0.01800980605185032,
+            -0.0045962524600327015,
+            0.00026381551288068295,
+            0.010372835211455822,
+            0.0038575688377022743,
+            0.015539710409939289,
+            0.023231398314237595,
+            0.031689126044511795,
+            -0.009653693065047264,
+            0.0018340062815696,
+            0.0009956592693924904,
+            0.016806025058031082,
+            0.005616338923573494,
+            0.0006727295112796128,
+            0.001513519324362278,
+            0.02510741911828518,
+            -0.011670416221022606,
+            -0.015899281948804855,
+            0.01596963219344616,
+            -0.020182862877845764,
+            -0.00010515979374758899,
+            -0.02029229700565338,
+            0.012084703892469406,
+            0.0011334295850247145,
+            0.013335385359823704,
+            0.013796573504805565,
+            -0.001073826802894473,
+            0.00819195993244648,
+            0.024794749915599823,
+            -0.005995451472699642,
+            0.0027358646038919687,
+            0.027233576402068138,
+            -0.00872349925339222,
+            -0.004779946058988571,
+            -0.000544730166438967,
+            0.002515041036531329,
+            0.00041257814154960215,
+            0.010599520988762379,
+            0.017822202295064926,
+            -0.027608782052993774,
+            -0.006034535355865955,
+            0.01240519154816866,
+            0.001964936964213848,
+            -0.006558258086442947,
+            0.004705687053501606,
+            0.0033553424291312695,
+            0.014851835556328297,
+            0.0003659218782559037,
+            0.01688419282436371,
+            -0.004819029942154884,
+            -0.02543572336435318,
+            -0.00930193904787302,
+            0.005835208110511303,
+            -0.012506809085607529,
+            -0.0032263658940792084,
+            -0.0034999523777514696,
+            -0.009512991644442081,
+            0.014265579171478748,
+            -0.008285760879516602,
+            0.0012340702814981341,
+            0.009395739994943142,
+            0.003802851540967822,
+            -0.007418101187795401,
+            -0.00215156190097332,
+            0.011881468817591667,
+            -0.012678777799010277,
+            0.004752587527036667,
+            0.022309020161628723,
+            0.003931828308850527,
+            -0.016821658238768578,
+            -0.018025439232587814,
+            0.006323755253106356,
+            0.03017267771065235,
+            -0.012194138951599598,
+            -0.0028765660244971514,
+            0.018916549161076546,
+            -0.0006644242093898356,
+            0.008215410634875298,
+            -0.002986000617966056,
+            -0.015023804269731045,
+            -0.03773929551243782,
+            -0.0019072884460911155,
+            -0.0018975174752995372,
+            0.005268493201583624,
+            -0.0003336777735967189,
+            -0.02374730445444584,
+            -0.007199232000857592,
+            -0.0103650176897645,
+            -0.002759314840659499,
+            -0.011021625250577927,
+            0.002632292453199625,
+            0.0008007289143279195,
+            -0.023684769868850708,
+            0.014031076803803444,
+            -0.012319207191467285,
+            0.010888740420341492,
+            -0.006100977770984173,
+            0.0190728846937418,
+            -0.0025169954169541597,
+            -0.0015457634581252933,
+            -0.008950185030698776,
+            0.006175236776471138,
+            0.013788756914436817,
+            0.0068513862788677216,
+            -0.011998720467090607,
+            0.0012467725900933146,
+            0.022856194525957108,
+            0.011818934231996536,
+            0.006179145537316799,
+            0.006683326326310635,
+            -0.011310845613479614,
+            -0.017884736880660057,
+            0.0020108604803681374,
+            -0.010451002046465874,
+            0.0033260295167565346,
+            0.007781580090522766,
+            0.014851835556328297,
+            0.005663239397108555,
+            -0.013694955967366695,
+            0.004760404117405415,
+            0.0103650176897645,
+            0.01694672554731369,
+            -0.014593882486224174,
+            -0.0034725936129689217,
+            -0.00925503857433796,
+            0.012483359314501286,
+            0.01916668564081192,
+            0.0042366813868284225,
+            0.006147878244519234,
+            0.004092071671038866,
+            0.015774212777614594,
+            -0.004228864796459675,
+            0.008332661353051662,
+            -0.013741856440901756,
+            -0.016493353992700577,
+            -0.019182318821549416,
+            0.004932372830808163,
+            0.00021899129205849022,
+            0.02745244652032852,
+            0.004557168576866388,
+            0.01976075768470764,
+            0.007457185070961714,
+            0.012749128974974155,
+            0.0020812111906707287,
+            0.022434089332818985,
+            -0.0009277511853724718,
+            -0.024153774604201317,
+            0.01206907071173191,
+            -0.021871281787753105,
+            -0.03070421703159809,
+            0.006843569688498974,
+            0.006612975150346756,
+            0.0381457693874836,
+            -0.022590424865484238,
+            0.011834568344056606,
+            -0.003156015183776617,
+            -0.023387733846902847,
+            -0.0022844469640403986,
+            -0.001389428274706006,
+            -0.0038380271289497614,
+            0.026358099654316902,
+            -0.0005198142607696354,
+            0.006315938662737608,
+            0.003521448466926813,
+            0.009810028597712517,
+            -0.00983347836881876,
+            0.021761847659945488,
+            0.01632138527929783,
+            0.018150506541132927,
+            0.005788307636976242,
+            -0.01112324371933937,
+            0.016993626952171326,
+            0.0010904374066740274,
+            -0.004955823067575693,
+            0.004568893928080797,
+            -0.005022265482693911,
+            0.0028257572557777166,
+            -0.0005564552848227322,
+            0.01474240142852068,
+            0.008707866072654724,
+            -0.015133239328861237,
+            0.010036714375019073,
+            -0.004568893928080797,
+            0.007898831740021706,
+            0.01949498802423477,
+            0.011975269764661789,
+            -0.002401698147878051,
+            0.009614609181880951,
+            -0.04430537298321724,
+            0.006140061654150486,
+            -0.03339318186044693,
+            -0.01233484037220478,
+            0.00270264339633286,
+            -0.017400098964571953,
+            0.0021105241030454636,
+            -0.01042755227535963,
+            -0.0175720676779747,
+            -0.006859203334897757,
+            0.019307386130094528,
+            0.01877584680914879,
+            0.004092071671038866,
+            0.00378917227499187,
+            0.010443185456097126,
+            -0.010873107239603996,
+            -0.013812207616865635,
+            -0.01394509244710207,
+            -0.012944547459483147,
+            -0.02049553394317627,
+            -0.0032556788064539433,
+            0.0009736746433191001,
+            -0.006441006902605295,
+            -0.002018677070736885,
+            0.009129970334470272,
+            -0.022043250501155853,
+            -0.010974724777042866,
+            0.014109243638813496,
+            -0.003392471931874752,
+            0.0006732181063853204,
+            -0.019119784235954285,
+            0.004256223328411579,
+            -0.003243953688070178,
+            0.00798481609672308,
+            -0.00581957446411252,
+            0.02431011013686657,
+            -0.01382784079760313,
+            -0.005053532775491476,
+            0.016821658238768578,
+            0.005491270683705807,
+            -0.007097613997757435,
+            -0.013546437956392765,
+            -0.03439372405409813,
+            0.025826560333371162,
+            -0.009770944714546204,
+            -0.01727502979338169,
+            -0.013288484886288643,
+            0.014984720386564732,
+            -0.009333206340670586,
+            -0.0018574565183371305,
+            0.011005992069840431,
+            0.0014988628681749105,
+            0.019713858142495155,
+            0.024826016277074814,
+            -0.010349384509027004,
+            0.0014206953346729279,
+            -0.01542245876044035,
+            0.012569342739880085,
+            -0.024700947105884552,
+            -0.009966363199055195,
+            0.013874741271138191,
+            0.012029986828565598,
+            0.002149607753381133,
+            -0.01661842316389084,
+            0.009661509655416012,
+            0.008363928645849228,
+            0.013906008563935757,
+            0.016993626952171326,
+            0.005467820446938276,
+            -0.010873107239603996,
+            -0.007914464920759201,
+            -0.024951083585619926,
+            0.0240443404763937,
+            0.005909467115998268,
+            -0.010974724777042866,
+            0.04368003085255623,
+            0.01993272639811039,
+            0.011998720467090607,
+            -0.01783783733844757,
+            -0.010552620515227318,
+            0.01664968952536583,
+            0.01218632236123085,
+            0.010802756063640118,
+            -0.00024036523245740682,
+            0.004932372830808163,
+            -0.0024603239726275206,
+            0.022168319672346115,
+            0.018213041126728058,
+            0.010185232385993004,
+            -0.05137171968817711,
+            0.014976903796195984,
+            0.002907833084464073,
+            -0.00520595908164978,
+            0.01685292460024357,
+            0.019948361441493034,
+            0.01217068824917078,
+            0.009606792591512203,
+            0.019854558631777763,
+            -0.00018735785852186382,
+            0.02194944955408573,
+            -0.009083069860935211,
+            -0.015758579596877098,
+            -0.00031950988341122866,
+            -0.013804391026496887,
+            -0.002776902401819825,
+            -0.019291752949357033,
+            0.009903829544782639,
+            0.00450245151296258,
+            -0.005237226374447346,
+            0.0240443404763937,
+            0.004209322854876518,
+            -0.029078330844640732,
+            -0.007656512316316366,
+            -0.029031431302428246,
+            0.0013796574203297496,
+            -0.019854558631777763,
+            -0.003818485187366605,
+            -0.014859652146697044,
+            -0.014421914704144001,
+            -0.026842739433050156,
+            -0.012241039425134659,
+            -0.00030363211408257484,
+            0.015023804269731045,
+            -0.001865273341536522,
+            -0.015633512288331985,
+            0.004529810044914484,
+            -0.03839590400457382,
+            -0.008582797832787037,
+            0.03645734861493111,
+            0.00819195993244648,
+            0.031954895704984665,
+            -0.008614065125584602,
+            0.01446099765598774,
+            -0.0010142240207642317,
+            -0.04261695221066475,
+            -0.0031657861545681953,
+            -0.003800897393375635,
+            0.00893455184996128,
+            0.007488451898097992,
+            0.0048932889476418495,
+            -0.012147238478064537,
+            0.016368286684155464,
+            -0.0014314433792605996,
+            0.00898926891386509,
+            0.0030524430330842733,
+            0.005151242017745972,
+            -0.010966908186674118,
+            0.010521353222429752,
+            -0.016368286684155464,
+            0.004248406738042831,
+            0.0019346470944583416,
+            0.004174147732555866,
+            -0.016868559643626213,
+            -0.016430821269750595,
+            -0.0037051422987133265,
+            -0.004576710518449545,
+            -0.012889830395579338,
+            -0.002970367204397917,
+            -0.011670416221022606,
+            -0.006655967328697443,
+            -0.008856384083628654,
+            -0.01431247964501381,
+            -0.011146693490445614,
+            0.0049362811259925365,
+            0.0027573604602366686,
+            0.008840750902891159,
+            0.014234311878681183,
+            0.003171648597344756,
+            0.007961365394294262,
+            0.02573275938630104,
+            0.001182284322567284,
+            -0.001272176974453032,
+            -0.01671222411096096,
+            0.001556511502712965,
+            -0.030157042667269707,
+            0.014453181065618992,
+            0.011748583987355232,
+            -0.025748392567038536,
+            0.006093161180615425,
+            -1.3885428415960632e-05,
+            0.0028570243157446384,
+            0.002175012370571494,
+            -0.0007220727857202291,
+            -0.014062343165278435,
+            0.003333846339955926,
+            -0.0180723387748003,
+            -0.00032488390570506454,
+            0.012467725202441216,
+            0.0401155911386013,
+            3.362579036547686e-06,
+            -0.0008657057187519968,
+            0.014429731294512749,
+            0.007707321085035801,
+            0.022074518725275993,
+            -0.01596963219344616,
+            -0.007015538401901722,
+            -0.03129829093813896,
+            -0.005342752672731876,
+            -0.005925100762397051,
+            -0.02198071777820587,
+            0.012241039425134659,
+            -0.0012545892968773842,
+            -0.010341567918658257,
+            -0.01366368867456913,
+            -0.02238718792796135,
+            0.012217588722705841,
+            0.02241845615208149,
+            0.008825116790831089,
+            0.007183598354458809,
+            0.008707866072654724,
+            -0.024450812488794327,
+            -0.020808203145861626,
+            0.029562970623373985,
+            0.013616788201034069,
+            -0.009231588803231716,
+            -0.01664968952536583,
+            -0.013585521839559078,
+            0.0031247481238096952,
+            -0.0006238748319447041,
+            0.0004951425944454968,
+            0.06219010800123215,
+            -0.018760213628411293,
+            0.0021847831085324287,
+            -0.034768927842378616,
+            -0.000998102012090385,
+            -0.009208138100802898,
+            0.013179049827158451,
+            0.005135608371347189,
+            -0.030188310891389847,
+            0.01618068479001522,
+            -0.030626049265265465,
+            -0.007257857825607061,
+            0.0015750762540847063,
+            -0.008137242868542671,
+            -0.010615154169499874,
+            -0.0023919271770864725,
+            -0.01827557571232319,
+            0.00013215202488936484,
+            -0.013327568769454956,
+            0.0021144323982298374,
+            0.002075348747894168,
+            -0.024700947105884552,
+            0.0015144963981583714,
+            0.006120519712567329,
+            -0.0046314275823533535,
+            -0.004350024741142988,
+            -0.019510623067617416,
+            -0.02019849605858326,
+            -0.014156145043671131,
+            -0.002032356569543481,
+            0.018025439232587814,
+            -0.02424757555127144,
+            0.01697799377143383,
+            -0.0032087783329188824,
+            0.004721320234239101,
+            0.003218549070879817,
+            0.008598431013524532,
+            0.018119240179657936,
+            0.01336665265262127,
+            0.0007044851081445813,
+            -0.010466636158525944,
+            0.01797853782773018,
+            -0.017947271466255188,
+            -0.02188691683113575,
+            0.001725548878312111,
+            0.0034081055782735348,
+            4.7083740355446935e-05,
+            0.0025932088028639555,
+            0.0045962524600327015,
+            0.013569887727499008,
+            0.007500177249312401,
+            -0.016196317970752716,
+            -0.002585391979664564,
+            -0.0027827650774270296,
+            0.029828740283846855,
+            0.0024603239726275206,
+            0.03276783972978592,
+            0.0012545892968773842,
+            0.015782030299305916,
+            -0.0014167869230732322,
+            -0.014570432715117931,
+            -0.017431365326046944,
+            -0.01314778346568346,
+            0.018728947266936302,
+            0.0016972130397334695,
+            0.007629153784364462,
+            -0.023700403049588203,
+            -0.01596963219344616,
+            0.003118885448202491,
+            -0.004049079492688179,
+            -0.01704052835702896,
+            0.0180723387748003,
+            -0.0177440345287323,
+            0.011521898210048676,
+            0.00032415110035799444,
+            -0.015023804269731045,
+            -0.017525166273117065,
+            0.014367196708917618,
+            0.0016923275543376803,
+            -0.000899904000107199,
+            0.012076887302100658,
+            -0.013257217593491077,
+            -0.011826751753687859,
+            0.002767131431028247,
+            0.00219259993173182,
+            -0.006824027746915817,
+            0.013616788201034069,
+            -0.016430821269750595,
+            0.005604614038020372,
+            -0.006824027746915817,
+            -0.0017802660586312413,
+            -0.010849656537175179,
+            0.013921641744673252,
+            0.012921096757054329,
+            -0.0015057026175782084,
+            0.012108154594898224,
+            0.012053437530994415,
+            0.003245907835662365,
+            0.014703317545354366,
+            0.005835208110511303,
+            -0.006378472782671452,
+            -0.010028897784650326,
+            0.0021124782506376505,
+            0.008676598779857159,
+            -0.004088163375854492,
+            -0.00383021030575037,
+            0.033080510795116425,
+            0.01668095588684082,
+            -0.017759669572114944,
+            0.021496077999472618,
+            -0.0038946985732764006,
+            0.007179690059274435,
+            -0.03186109662055969,
+            0.0004763335455209017,
+            0.013655872084200382,
+            -0.010873107239603996,
+            0.008223227225244045,
+            0.007816756144165993,
+            0.028171587735414505,
+            0.012194138951599598,
+            0.00819195993244648,
+            -0.0050769830122590065,
+            0.0021105241030454636,
+            0.013186866417527199,
+            0.0027964443434029818,
+            -0.002470094710588455,
+            -0.0026967807207256556,
+            0.01996399462223053,
+            -0.016337020322680473,
+            0.020370464771986008,
+            0.013335385359823704,
+            -0.029375368729233742,
+            0.005217684432864189,
+            -0.019010350108146667,
+            0.003961140755563974,
+            -0.006319846957921982,
+            -0.0037168674170970917,
+            -0.021386643871665,
+            0.007339933421462774,
+            0.02945353463292122,
+            -0.011772033758461475,
+            0.029734939336776733,
+            0.004326574504375458,
+            0.02016722969710827,
+            -0.009450457990169525,
+            -0.006198687478899956,
+            -0.009208138100802898,
+            0.0010054302401840687,
+            -0.008371745236217976,
+            0.011154510080814362,
+            0.0002304721565451473,
+            0.005846933461725712,
+            0.01424212846904993,
+            0.004060804378241301,
+            -0.003523402614519,
+            0.028828194364905357,
+            0.005925100762397051,
+            0.008231043815612793,
+            -0.013007081113755703,
+            0.005159058608114719,
+            -0.007863656617701054,
+            0.0038243476301431656,
+            0.0019854558631777763,
+            -0.008637514896690845,
+            9.306825086241588e-05,
+            -0.0021730579901486635,
+            0.013585521839559078,
+            0.009614609181880951,
+            0.013960725627839565,
+            -0.004678328521549702,
+            -0.008121608756482601,
+            0.025685859844088554,
+            0.001968845259398222,
+            -0.004310940857976675,
+            0.004729137290269136,
+            0.004881564062088728,
+            0.016931092366576195,
+            0.012960180640220642,
+            0.0006043329485692084,
+            -0.0021574245765805244,
+            -0.025185586884617805,
+            0.01212378777563572,
+            -0.018353741616010666,
+            -0.00039841028046794236,
+            -0.005698414985090494,
+            -0.022731125354766846,
+            0.013053981587290764,
+            -0.005248951260000467,
+            -0.003607432823628187,
+            -0.005542079918086529,
+            0.022965628653764725,
+            0.004064713139086962,
+            -0.014343746937811375,
+            -0.00399240804836154,
+            0.007468909956514835,
+            0.02765568159520626,
+            -0.0022121418733149767,
+            -0.00042381472303532064,
+            -0.010622970759868622,
+            -0.015289573930203915,
+            0.012749128974974155,
+            -0.015742946416139603,
+            -0.013624604791402817,
+            0.003101297887042165
+          ],
+          "title": "DeepSeekMoE: Achieving Ultimate Expert Specialization in Mixture-of-Experts Architectures",
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "Mixture-of-Experts (MoE)",
+            "expert specialization",
+            "fine-grained expert segmentation",
+            "shared expert isolation"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "29bb28da-d077-47a5-a332-b20f4caf8ab4",
+      "type": "child",
+      "source": {
+        "id": "1e687808-8db8-4320-805d-0282a8d679bc",
+        "properties": {
+          "page_content": "Deepseekmoe: Towards Ultimate Expert Specialization In\n\nMixture-Of-Experts Language Models\n\nDamai Dai∗1,2, Chengqi Deng1, Chenggang Zhao∗1,3, R.X. Xu1, Huazuo Gao1, Deli Chen1, Jiashi Li1, Wangding Zeng1, Xingkai Yu∗1,4, Y. Wu1, Zhenda Xie1, Y.K. Li1, Panpan Huang1, Fuli Luo1, Chong Ruan1, Zhifang Sui2, Wenfeng Liang1 1DeepSeek-AI 2National Key Laboratory for Multimedia Information Processing, Peking University 3Institute for Interdisciplinary Information Sciences, Tsinghua University 4National Key Laboratory for Novel Software Technology, Nanjing University {daidamai, szf}@pku.edu.cn, {wenfeng.liang}@deepseek.com https://github.com/deepseek-ai/DeepSeek-MoE\n\nAbstract\n\nIn the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top- out of experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into ones and activating from them, allowing for a more flexible combination of activated experts; (2) isolating experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5× expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations.\n\nFurther, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\n1. Introduction\n\nRecent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023a). It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017) has emerged as a popular solution. It can\n\nenable parameter scaling, while concurrently keeping computational costs at a modest level.\n\nRecent applications of MoE architectures in Transformers (Vaswani et al., 2017) have yielded successful attempts at scaling language models to a substantial size (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022), accompanied with remarkable performance. These achievements underscore the considerable potential and promise of MoE language models.\n\nDespite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts.\n\nThis architecture manifests two potential issues: (1) Knowledge Hybridity: existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. (2) Knowledge Redundancy: tokens assigned to different experts may require common knowledge. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters.\n\nThese issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models.\n\nIn response to the aforementioned issues, we introduce DeepSeekMoE, an innovative MoE architecture specifically designed towards ultimate expert specialization. Our architecture involves two principal strategies: (1) Fine-Grained Expert Segmentation: while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. (2) Shared Expert Isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.\n\nThrough compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized.\n\nStarting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B (Lepikhin et al., 2021) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5× expert parameters and computation. Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE.\n\nThese studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization.\n\nLeveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B (DeepSeek-AI, 2024), a dense model trained on the same 2T corpus. We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B (Touvron et al., 2023b), which has approximately 2.5 times the activated parameters. Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard1. Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. The experimental results still validate its substantial advantages over the GShard architecture consistently. In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nOur contributions are summarized as follows: - Architectural Innovation. We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation.\n\nEmpirical Validation. We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. Experimental results validate the high 1https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models\n\nScalability. We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B.\n\nAlignment for MoE. We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B.\n\nPublic Release. In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization.\n\n2. Preliminaries: Mixture-Of-Experts For Transformers\n\nWe first introduce a generic MoE architecture commonly used in Transformer language models. A standard Transformer language model is constructed by stacking layers of standard Transformer blocks, where each block can be represented as follows:\n\n$$\\mathbf{u}{1:T}^{l}=\\text{Self-Att}\\left(\\mathbf{h}{1:T}^{l-1}\\right)+\\mathbf{h}{1:T}^{l-1},\\tag{1}$$ $$\\mathbf{h}{t}^{l}=\\text{FFN}\\left(\\mathbf{u}{t}^{l}\\right)+\\mathbf{u}{t}^{l},\\tag{2}$$\n\nwhere denotes the sequence length, Self-Att(·) denotes the self-attention module, FFN(·) denotes the Feed-Forward Network (FFN), u 1: ∈ R× are the hidden states of all tokens after the -th attention module, and h\n\n∈ Ris the output hidden state of the -th token after the -th Transformer block. For brevity, we omit the layer normalization in the above formulations.\n\nA typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022). An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. Then, each token will be assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts. If the -th FFN is substituted with an MoE layer, the computation for its output hidden state h\n\nis expressed as: where denotes the total number of experts, FFN(·) is the -th expert FFN, , denotes the gate value for the -th expert, , denotes the token-to-expert affinity, Topk(·, ) denotes the set comprising highest affinity scores among those calculated for the -th token and all experts, and e\n\nis the centroid of the -th expert in the -th layer. Note that ,is sparse, indicating that only out of gate values are nonzero. This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only experts. Also, in the above formulations, we omit the layer normalization operation for brevity.\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{N}\\left(g_{i,t}\\operatorname{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\operatorname{Topk}({s_{j,t}|1\\leqslant j\\leqslant N},K),\\ 0,&\\text{otherwise},\\end{cases}$$ $$s_{i,t}=\\operatorname{Softmax}{i}\\left(\\mathbf{u}{t}^{l}\\,\\mathbf{e}_{i}^{l}\\right),$$\n\n$$(3)$$\n\n(4) $\\binom{4}{5}$ .\n\n3. Deepseekmoe Architecture\n\nOn top of the generic MoE architecture outlined in Section 2, we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. As illustrated in Figure 2, our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. Both of these strategies are designed to elevate the level of expert specialization.\n\n3.1. Fine-Grained Expert Segmentation\n\nIn scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts.\n\nIn pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. The finer expert segmentation enables a more flexible and adaptable combination of activated experts. To be specific, on top of a typical MoE architecture shown in Figure 2(a), we segment each expert FFN into smaller experts by reducing the FFN intermediate hidden dimension to 1 times its original size. Since each expert becomes smaller, in response, we also increase the number of activated experts to times to keep the same computation cost, as illustrated in Figure 2(b). With the fine-grained expert segmentation, the output of an MoE layer can be expressed as:\n\n$$\\mathbf{h}{t}^{i}=\\sum{i=1}^{\\text{mN}}\\left(g_{i,t}\\text{FFN}{i}\\left(\\mathbf{u}{t}^{i}\\right)\\right)+\\mathbf{u}{t}^{i},\\tag{6}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\text{Tokr}(s_{i,t}|1\\leqslant j\\leqslant m\\text{N}),m\\text{K}),\\ 0,&\\text{otherwise},\\end{cases}$$ (7) $$s_{i,t}=\\text{Softmax}{i}\\left(\\mathbf{u}{t}^{T}\\mathbf{e}_{t}^{i}\\right),\\tag{8}$$ where the total number of expert parameters is equal to $N$ times the number of parameters in a\n\n$$(6)$$\n\n(7) (8) $\\frac{1}{2}$\n\nstandard FFN, and denotes the total number of fine-grained experts. With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to .\n\nFrom a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. As an illustrative example, we consider the case where = 16. A typical top-2 routing strategy can yield 16 2 = 120 possible combinations. By contrast, if each expert is split into 4 smaller experts, the fine-grained routing strategy can yield 64 8 = 4, 426, 165, 368 potential combinations. The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition.\n\n3.2. Shared Expert Isolation\n\nWith a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. This alleviation of redundancy will contribute to a more parameterefficient model with more specialized experts.\n\nTowards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate experts to serve as shared experts. Regardless of the router module, each token will be deterministically assigned to these shared experts. In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by , as depicted in Figure 2(c). With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows:\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{K_{s}}\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)+\\sum_{i=K_{s}+1}^{mN}\\left(g_{i,t}\\;\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},\\tag{9}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\mathrm{Topk}{s_{j,t}|K_{s}+1\\leqslant j\\leqslant mN},mK-K_{s}},\\ 0,&\\text{otherwise},\\end{cases}$$ (10) $$s_{i,t}=\\mathrm{Softmax}{i}\\left(\\mathbf{u}{t}^{l,T}e_{i}^{l}\\right).\\tag{11}$$ $\\mathbf{h}\\mathbf{M}\\mathbf{E}$ the number of closed-compact is $K_{s}$ the total number of routed currents.\n\nFinally, in DeepSeekMoE, the number of shared expert is , the total number of routed experts is − , and the number of nonzero gates is − .\n\nIt is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022). The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint.\n\n3.3. Load Balance Consideration\n\nAutomatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. Firstly, there is a risk of routing collapse (Shazeer et al., 2017), i.e., the model always selects only a few experts, preventing other experts from sufficient training.\n\nSecondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.\n\nExpert-Level Balance Loss. In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. The computation of the balance loss is as follows:\n\n$$\\begin{array}{l}{{{\\mathcal L}{\\mathrm{ExpBal}}=\\alpha{1}\\sum_{i=1}^{N^{\\prime}}f_{i}P_{i},}}\\ {{f_{i}=\\frac{N^{\\prime}}{K^{T}}\\sum_{t=1}^{T}\\mathds{1}\\,(\\mathrm{Token}\\;t\\;\\mathrm{s}\\mathrm{lects}\\;\\mathrm{Expert}\\;i),}}\\ {{P_{i}=\\frac{1}{T}\\sum_{t=1}^{T}s_{i,t},}}\\end{array}$$\n\n(12) $$\\begin{array}{l}\\small\\text{(13)}\\end{array}$$ = (14) $$\\begin{array}{l}\\small\\text{(14)}\\end{array}$$ .\n\n7 where 1 is a hyper-parameter called expert-level balance factor, ′is equal to ( − ) and ′ is equal to ( − ) for brevity. 1(·) denotes the indicator function.\n\nDevice-Level Balance Loss. In addition to the expert-level balance loss, we introduce a devicelevel balance loss. When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. Instead, our primary objective is to ensure balanced computation across the devices. If we partition all routed experts into groups {E1, E2, ..., E}, and deploy each group on a single device, the device-level balance loss is computed as follows:\n\n$$\\mathcal{L}{\\text{DevBal}}=\\alpha{2}\\sum_{i=1}^{D}f_{i}^{\\prime}P_{i}^{\\prime},$$ $$f_{i}^{\\prime}=\\frac{1}{|\\mathcal{E}{i}|}\\sum{j\\in\\mathcal{E}{i}}f{j},$$ $$P_{i}^{\\prime}=\\sum_{j\\in\\mathcal{E}{i}}P{j},$$\n\n(15) $$\\begin{array}{l}\\small\\text{(16)}\\end{array}$$ = (17) .\n\nwhere 2 is a hyper-parameter called device-level balance factor. In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices.\n\n4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n6. Alignment For Deepseekmoe 16B\n\nPrevious research indicates that MoE models typically do not emerge significant gains from fine-tuning (Artetxe et al., 2022; Fedus et al., 2021). However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B.\n\n6.1. Experimental Setup\n\nTraining Data. For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios.\n\nHyper-Parameters. During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).\n\nWe employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10−5 without incorporating any learning rate scheduling strategy.\n\nEvaluation Benchmarks. For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3, with the following adjustments: (1) We exclude Pile (Gao et al., 2020) since chat models are seldom employed for pure language modeling. (2) We exclude\n\nMetric # Shot LLaMA2 DeepSeek DeepSeekMoE SFT 7B Chat 7B Chat 16B # Total Params N/A 6.7B 6.9B 16.4B # Activated Params N/A 6.7B 6.9B 2.8B FLOPs per 4K Tokens N/A 187.9T 183.5T 74.4T HellaSwag (Acc.) 0-shot 67.9 71.0 72.2 PIQA (Acc.) 0-shot 76.9 78.4 79.7 ARC-easy (Acc.) 0-shot 69.7 70.2 69.9 ARC-challenge (Acc.) 0-shot 50.8 50.2 50.0 BBH (EM) 3-shot 39.3 43.1 42.2 RACE-middle (Acc.) 5-shot 63.9 66.1 64.8 RACE-high (Acc.) 5-shot 49.6 50.8 50.6 DROP (EM) 1-shot 40.0 41.7 33.8 GSM8K (EM) 0-shot 63.4 62.6 62.2 MATH (EM) 4-shot 13.5 14.7 15.2 HumanEval (Pass@1) 0-shot 35.4 45.1 45.7 MBPP (Pass@1) 3-shot 27.8 39.0 46.2 TriviaQA (EM) 5-shot 60.1 59.5 63.3 NaturalQuestions (EM) 0-shot 35.2 32.7 35.1 MMLU (Acc.) 0-shot 50.0 49.7 47.2 WinoGrande (Acc.) 0-shot 65.1 68.4 69.0 CLUEWSC (EM) 5-shot 48.4 66.2 68.2 CEval (Acc.) 0-shot 35.1 44.7 40.0 CMMLU (Acc.) 0-shot 36.9 51.2 49.3\n\nCHID (Zheng et al., 2019) due to the observed instability of results, hindering the derivation of solid conclusions. (3) We additionally include BBH (Suzgun et al., 2022) to provide a more comprehensive assessment of the reasoning ability of the chat models.\n\nTable 5 | Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations.\n\n6.2. Evaluations\n\nBaselines. In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. Correspondingly, we construct three chat models, including LLaMA2 SFT 7B3, DeepSeek Chat 7B, and DeepSeekMoE Chat 16B.\n\nSubsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks.\n\nResults. The evaluation results are presented in Table 5. Our key observations include: (1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intensive tasks (TriviaQA, NaturalQuestions). (2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. In addition, it also surpasses DeepSeek Chat 7B. (3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1). However, it is worth noting that, after supervised finetuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. (4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations.\n\n7. Deepseekmoe 145B Ongoing\n\nEncouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available.\n\n7.1. Experimental Setup\n\nTraining Data and Tokenization. For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study.\n\nModel Settings. For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps.\n\nSubsequently, the learning rate keeps constant during the remaining training process. The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. Also, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse.\n\nEvaluation Benchmarks. We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3).\n\n7.2. Evaluations\n\nBaselines. Apart from DeepSeekMoE 145B, we consider three additional models for comparison. DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters.\n\nResults. From the evaluation results presented in Table 6, we have the following observations: (1) Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. (2) On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. (3) At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5.\n\n8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\nAppendices A. Overview Of Hyper-Parameters\n\nWe present the overview of hyper-parameters for DeepSeekMoE across various sizes in Table 7.\n\n# Params # Layers Hidden # Attn # Shared # Routed Relative Sequence Batch Size Learning Size Heads Experts Experts Expert Size Length (Sequence) Rate 2.0B 9 1280 10 1 63 (7 activated) 0.25 2048 2048 1.08e-3 16.4B 28 2048 16 2 64 (6 activated) 0.25 4096 4608 4.2e-4 144.6B 62 4096 32 4 128 (12 activated) 0.125 4096 4608 3.0e-4\n\nTable 7 | Overview of hyper-parameters for DeepSeekMoE across various sizes. The relative expert size is in comparison to a standard FFN.\n\nB. Comparing Deepseekmoe With Larger Models\n\nComparisons among DeepSeekMoE, GShard×1.2, and GShard×1.5 are shown in Table 8. Comparisons among DeepSeekMoE, Dense×4, and Dense×16 are shown in Table 9.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 2.3B 2.8B 1.9B # Activated Expert Params N/A 0.28B 0.35B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.824 1.808 1.808 HellaSwag (Acc.) 0-shot 53.7 54.4 54.8 PIQA (Acc.) 0-shot 71.8 71.1 72.3 ARC-easy (Acc.) 0-shot 46.8 47.3 49.4 ARC-challenge (Acc.) 0-shot 31.7 34.1 34.3 RACE-middle (Acc.) 5-shot 43.7 46.4 44.0 RACE-high (Acc.) 5-shot 31.9 32.4 31.7 HumanEval (Pass@1) 0-shot 3.7 3.0 4.9 MBPP (Pass@1) 3-shot 2.4 2.6 2.2 TriviaQA (EM) 5-shot 15.2 15.7 16.6 NaturalQuestions (EM) 5-shot 4.5 4.7 5.7\n\nTable 8 | Comparison between DeepSeekMoE and larger GShard models.\n\nAt a larger scale of 13B total parameters, we also compare DeepSeekMoE with GShard×1.2 and GShard×1.5, and show results in Table 10. At a larger scale, DeepSeekMoE even outperforms GShard×1.5 distinctly.\n\nMetric # Shot Dense×4 Dense×16 DeepSeekMoE Relative Expert Size N/A 1 1 0.25 # Experts N/A 4 + 0 16 + 0 1 + 63 # Activated Experts N/A 4 + 0 16 + 0 1 + 7 # Total Expert Params N/A 0.47B 1.89B 1.89B # Activated Expert Params N/A 0.47B 1.89B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.908 1.806 1.808 HellaSwag (Acc.) 0-shot 47.6 55.1 54.8 PIQA (Acc.) 0-shot 70.0 71.9 72.3 ARC-easy (Acc.) 0-shot 43.9 51.9 49.4 ARC-challenge (Acc.) 0-shot 30.5 33.8 34.3 RACE-middle (Acc.) 5-shot 42.4 46.3 44.0 RACE-high (Acc.) 5-shot 30.7 33.0 31.7 HumanEval (Pass@1) 0-shot 1.8 4.3 4.9 MBPP (Pass@1) 3-shot 0.2 2.2 2.2 TriviaQA (EM) 5-shot 9.9 16.5 16.6 NaturalQuestions (EM) 5-shot 3.0 6.3 5.7\n\nTable 9 | Comparison between DeepSeekMoE and larger dense baselines.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 15.9B 19.8B 13.3B # Activated Expert Params N/A 2.37B 2.82B 2.05B # Training Tokens N/A 100B 100B 100B HellaSwag (Acc.) 0-shot 66.6 67.7 69.1 PIQA (Acc.) 0-shot 75.6 76.0 75.7 ARC-easy (Acc.) 0-shot 56.8 56.8 58.8 ARC-challenge (Acc.) 0-shot 39.9 37.6 38.5 RACE-middle (Acc.) 5-shot 51.6 50.6 52.4 RACE-high (Acc.) 5-shot 37.4 36.3 38.5 HumanEval (Pass@1) 0-shot 6.1 6.1 9.8 MBPP (Pass@1) 3-shot 7.0 11.6 10.6 TriviaQA (EM) 5-shot 36.5 36.7 38.2 NaturalQuestions (EM) 5-shot 12.6 12.1 13.7\n\nTable 10 | Comparison between DeepSeekMoE and larger GShard models at a larger scale.\n\nC. Training Benchmark Curves Of Deepseekmoe 16B\n\nWe present the benchmark curves during training of DeepSeekMoE 16B and DeepSeek 7B (Dense) in Figure 7 for reference.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/DeepSeekMoE_2.md"
+          },
+          "summary": "DeepSeekMoE is an innovative Mixture-of-Experts (MoE) architecture designed to enhance expert specialization in language models. It employs fine-grained expert segmentation and shared expert isolation to improve performance and reduce computational costs. Starting with a model of 2B parameters, DeepSeekMoE demonstrates comparable performance to larger models like GShard 2.9B and LLaMA2 7B, using significantly fewer computations. The architecture scales up to 16B and 145B parameters, maintaining advantages over traditional MoE architectures. DeepSeekMoE 16B achieves similar results to dense models with only 40% of the computations, and its chat model version shows adaptability in bilingual scenarios. The architecture's public release aims to contribute to advancements in large-scale language models.",
+          "headlines": {
+            "1. Introduction": [],
+            "2. Preliminaries: Mixture-Of-Experts For Transformers": [],
+            "3. Deepseekmoe Architecture": [
+              "3.1. Fine-Grained Expert Segmentation",
+              "3.2. Shared Expert Isolation",
+              "3.3. Load Balance Consideration"
+            ],
+            "4. Validation Experiments": [
+              "4.1. Experimental Setup",
+              "4.1.1. Training Data And Tokenization",
+              "4.1.2. Infrastructures",
+              "4.1.3. Hyper-Parameters",
+              "4.1.4. Evaluation Benchmarks",
+              "4.2. Evaluations",
+              "4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models",
+              "4.4. Ablation Studies",
+              "4.5. Analysis On Expert Specialization"
+            ],
+            "5. Scaling Up To Deepseekmoe 16B": [
+              "5.1. Experimental Setup",
+              "5.1.1. Training Data And Tokenization",
+              "5.1.2. Hyper-Parameters",
+              "5.1.3. Evaluation Benchmarks",
+              "5.2. Evaluations",
+              "5.2.1. Internal Comparison With Deepseek 7B",
+              "5.2.2. Comparison With Open Source Models"
+            ],
+            "6. Alignment For Deepseekmoe 16B": [
+              "6.1. Experimental Setup",
+              "6.2. Evaluations"
+            ],
+            "7. Deepseekmoe 145B Ongoing": [
+              "7.1. Experimental Setup",
+              "7.2. Evaluations"
+            ],
+            "8. Related Work": [],
+            "9. Conclusion": [],
+            "Appendices": [
+              "A. Overview Of Hyper-Parameters",
+              "B. Comparing Deepseekmoe With Larger Models",
+              "C. Training Benchmark Curves Of Deepseekmoe 16B"
+            ]
+          },
+          "summary_embedding": [
+            -0.012465757317841053,
+            -0.022808987647294998,
+            -0.018792515620589256,
+            -0.02308654971420765,
+            0.01797616109251976,
+            0.007200260180979967,
+            0.006490030791610479,
+            0.042613785713911057,
+            -0.014049487188458443,
+            0.03187054023146629,
+            -0.021486490964889526,
+            -0.023592690005898476,
+            0.009045224636793137,
+            -0.014441338367760181,
+            0.005865518003702164,
+            -0.007363531738519669,
+            -0.008743173442780972,
+            -0.005163452122360468,
+            0.018841497600078583,
+            -0.03100520372390747,
+            0.011110605672001839,
+            -0.046075135469436646,
+            -0.020017050206661224,
+            -0.021845687180757523,
+            0.016425084322690964,
+            0.037389107048511505,
+            0.022547753527760506,
+            -0.0028062239289283752,
+            -0.06844329088926315,
+            0.034678805619478226,
+            0.035168617963790894,
+            0.02052319049835205,
+            0.0005438972148112953,
+            -0.00031302153365686536,
+            -0.024833550676703453,
+            -0.028556134551763535,
+            0.03347059711813927,
+            -0.004379750229418278,
+            -0.01453930139541626,
+            0.015976088121533394,
+            0.0414055772125721,
+            0.012106560170650482,
+            -0.02754385396838188,
+            0.0014245413476601243,
+            0.015151568688452244,
+            0.0055961208418011665,
+            -0.01678428053855896,
+            -0.0022960014175623655,
+            -0.002424577483907342,
+            -0.021225256845355034,
+            -0.01400050614029169,
+            0.018825169652700424,
+            0.001296985661610961,
+            -0.008865626528859138,
+            -0.0542386956512928,
+            0.007330877240747213,
+            0.03840138763189316,
+            0.05623060464859009,
+            0.044148534536361694,
+            -0.020180322229862213,
+            0.03369917720556259,
+            -0.030025575309991837,
+            -0.02501314878463745,
+            0.011028970591723919,
+            0.003983817528933287,
+            -0.018531281501054764,
+            0.01784554310142994,
+            -0.005636938381940126,
+            -0.0006556359585374594,
+            0.01751900091767311,
+            -0.03941367194056511,
+            0.020751770585775375,
+            -0.01887415163218975,
+            0.031266435980796814,
+            0.02173139713704586,
+            0.0073798587545752525,
+            0.01872720755636692,
+            0.01678428053855896,
+            0.0036858476232737303,
+            -0.04107903689146042,
+            0.009118696674704552,
+            0.016523046419024467,
+            0.014196431264281273,
+            0.016269976273179054,
+            0.06553706526756287,
+            -0.05391215533018112,
+            -0.004485876299440861,
+            -0.05169166624546051,
+            0.0012561678886413574,
+            -0.004971608519554138,
+            -0.03670336678624153,
+            0.01048201136291027,
+            0.028425518423318863,
+            0.021372200921177864,
+            -0.01063711941242218,
+            0.024392718449234962,
+            -0.022743679583072662,
+            0.0032001156359910965,
+            -0.04062187671661377,
+            0.0019653772469609976,
+            0.004518530797213316,
+            0.016531210392713547,
+            0.0070778070949018,
+            0.004273623693734407,
+            -0.016702644526958466,
+            0.006045116577297449,
+            -0.01905374974012375,
+            0.03291547670960426,
+            -0.008465612307190895,
+            0.08653374016284943,
+            0.008751336485147476,
+            0.004738946910947561,
+            0.004281787667423487,
+            0.02398454025387764,
+            0.012131051160395145,
+            -0.00810233410447836,
+            -0.04718538001179695,
+            -0.015690363943576813,
+            0.011061624623835087,
+            -0.0383034273982048,
+            0.018335357308387756,
+            -0.008759500458836555,
+            0.01282495353370905,
+            -0.04107903689146042,
+            -0.06344719231128693,
+            0.01498013362288475,
+            0.001784758409485221,
+            -0.004065453074872494,
+            -0.0002903166168835014,
+            -0.018923133611679077,
+            0.003930754493921995,
+            -0.004200151655822992,
+            -0.010645282454788685,
+            -0.021502818912267685,
+            0.03159298002719879,
+            0.03657275065779686,
+            0.06527583301067352,
+            0.036801330745220184,
+            -0.0030164355412125587,
+            0.005775718949735165,
+            0.019690508022904396,
+            -0.020784424617886543,
+            0.003481758525595069,
+            -0.01214737817645073,
+            -0.021992631256580353,
+            -0.03820546343922615,
+            0.010971825569868088,
+            0.07379858940839767,
+            -0.029029620811343193,
+            0.013437220826745033,
+            -0.017616963014006615,
+            0.04205866530537605,
+            -0.004567512311041355,
+            0.026727497577667236,
+            -0.016669990494847298,
+            0.004059330560266972,
+            -0.025388672947883606,
+            0.01620466820895672,
+            -0.024964168667793274,
+            0.012571883387863636,
+            -0.000885746325366199,
+            -0.04222193360328674,
+            0.00011116473615402356,
+            0.026890767738223076,
+            -0.010971825569868088,
+            0.02261306345462799,
+            0.0004949158756062388,
+            -0.006302268709987402,
+            -0.05097327008843422,
+            0.038727931678295135,
+            -0.003912386484444141,
+            0.018645571544766426,
+            -0.0018959869630634785,
+            0.0163107942789793,
+            -0.0016408757073804736,
+            0.013216804713010788,
+            0.009543201886117458,
+            0.007114543113857508,
+            -0.033503253012895584,
+            0.02697240374982357,
+            -0.02504580281674862,
+            0.028049994260072708,
+            0.03435226157307625,
+            -0.0030184765346348286,
+            0.03255628049373627,
+            0.019249675795435905,
+            0.03840138763189316,
+            0.0327032245695591,
+            0.030335791409015656,
+            -0.00824927818030119,
+            0.013878053054213524,
+            0.008200296200811863,
+            0.03608293831348419,
+            0.013429056853055954,
+            0.019723162055015564,
+            -0.009730963967740536,
+            -0.014041324146091938,
+            0.048785436898469925,
+            0.007318632211536169,
+            -0.0179598331451416,
+            -0.005441013257950544,
+            -0.03510330989956856,
+            -0.002832755446434021,
+            -0.014425011351704597,
+            0.0323440283536911,
+            0.02158445306122303,
+            -0.01647406443953514,
+            0.035135965794324875,
+            0.024980494752526283,
+            -0.03751972317695618,
+            -0.005232842173427343,
+            -0.02026195637881756,
+            -0.023298801854252815,
+            -0.04215662553906441,
+            0.011624909937381744,
+            0.012833117507398129,
+            -0.013976015150547028,
+            -0.007469657808542252,
+            0.020457882434129715,
+            -0.023625344038009644,
+            -0.0018265966791659594,
+            -0.01356783788651228,
+            0.006547175347805023,
+            -0.020898714661598206,
+            -0.03667071461677551,
+            -0.01048201136291027,
+            0.025862159207463264,
+            0.012637191452085972,
+            0.04362606629729271,
+            -0.013518855907022953,
+            -0.005367541220039129,
+            0.033193036913871765,
+            0.012759645469486713,
+            0.010767736472189426,
+            0.03699725493788719,
+            -0.03027048148214817,
+            0.024621298536658287,
+            -0.020294612273573875,
+            -0.0032878739293664694,
+            0.01707816869020462,
+            -0.0030062312725931406,
+            0.01962519995868206,
+            -0.04535674303770065,
+            -0.024196794256567955,
+            0.04395261034369469,
+            -0.0007107399869710207,
+            0.016302630305290222,
+            -0.005959399044513702,
+            -0.012106560170650482,
+            0.0035899258218705654,
+            0.0027715289033949375,
+            -0.01438419334590435,
+            -0.008620720356702805,
+            0.007473739795386791,
+            -0.01962519995868206,
+            0.03660540655255318,
+            0.03794422745704651,
+            -0.004132802598178387,
+            -0.014131123200058937,
+            -0.03324201703071594,
+            0.010016689077019691,
+            -0.017878197133541107,
+            -0.023951886221766472,
+            0.04078514873981476,
+            -0.032115448266267776,
+            -0.00029312283731997013,
+            0.039478980004787445,
+            0.018808843567967415,
+            -0.014898497611284256,
+            0.00868602842092514,
+            -0.01709449477493763,
+            -0.026302991434931755,
+            -0.017159804701805115,
+            0.03356856107711792,
+            0.024800896644592285,
+            -0.010743245482444763,
+            0.010245268233120441,
+            0.005330805201083422,
+            0.02893165871500969,
+            0.024376392364501953,
+            0.00648594880476594,
+            -0.013812744058668613,
+            0.015029114671051502,
+            0.027641816064715385,
+            -0.0028021421749144793,
+            -7.66609373386018e-05,
+            -0.004010349046438932,
+            -0.007796200457960367,
+            -0.006453294772654772,
+            -0.013069860637187958,
+            -0.005089980084449053,
+            0.0009184005321003497,
+            -0.032295044511556625,
+            0.015967924147844315,
+            -0.005890008993446827,
+            0.010359558276832104,
+            -0.011535110883414745,
+            0.01288209855556488,
+            0.01722511276602745,
+            -0.013494365848600864,
+            -0.0716434046626091,
+            0.014612773433327675,
+            0.02458864450454712,
+            -0.006706364918500185,
+            -0.017943505197763443,
+            0.008481939323246479,
+            0.00016148542636074126,
+            0.038597315549850464,
+            -0.014180104248225689,
+            -0.031103165820240974,
+            0.02680913358926773,
+            -0.017437364906072617,
+            -0.015241367742419243,
+            -0.010661610402166843,
+            -0.00580837344750762,
+            0.04649963974952698,
+            0.01889047957956791,
+            0.011420820839703083,
+            -0.04042595252394676,
+            -0.007449248805642128,
+            0.015078096650540829,
+            0.014506647363305092,
+            -0.04705476388335228,
+            0.0052083516493439674,
+            0.015486273914575577,
+            0.019723162055015564,
+            0.024800896644592285,
+            0.04274440184235573,
+            -0.0021531390957534313,
+            0.01661284640431404,
+            -0.015053605660796165,
+            -0.05136512219905853,
+            -0.03340528905391693,
+            0.029943939298391342,
+            -0.017731253057718277,
+            -0.02906227484345436,
+            -0.01693122461438179,
+            -0.006151242647320032,
+            0.0017939424142241478,
+            0.007126788143068552,
+            -0.01916803978383541,
+            -0.03464614972472191,
+            -0.03970756009221077,
+            0.02277633361518383,
+            0.007983962073922157,
+            0.004269542172551155,
+            -0.042777057737112045,
+            0.0013459670590236783,
+            0.009184005670249462,
+            0.012073906138539314,
+            0.0009954441338777542,
+            -0.019984396174550056,
+            0.015494437888264656,
+            0.0032001156359910965,
+            -0.031070511788129807,
+            -0.035593122243881226,
+            -0.03595232218503952,
+            0.0023919232189655304,
+            0.003781769424676895,
+            0.06279410421848297,
+            0.012988224625587463,
+            -0.03840138763189316,
+            -0.04522612690925598,
+            0.029176564887166023,
+            -0.02052319049835205,
+            -0.016678154468536377,
+            0.0059430720284581184,
+            0.012898425571620464,
+            -0.009877908043563366,
+            0.013192313723266125,
+            0.023560035973787308,
+            0.041601505130529404,
+            -0.005987971555441618,
+            0.012914752587676048,
+            0.023037567734718323,
+            0.012955570593476295,
+            0.0362788625061512,
+            -0.00936360377818346,
+            0.0035409443080425262,
+            0.039936136454343796,
+            -0.012269831262528896,
+            0.03175625205039978,
+            -0.02189466916024685,
+            0.0020398697815835476,
+            0.02249877341091633,
+            0.027184655889868736,
+            -0.0006479826406575739,
+            0.02594379521906376,
+            -0.031543996185064316,
+            0.018368011340498924,
+            -0.014498483389616013,
+            -0.04888340085744858,
+            -0.01557607389986515,
+            0.013102514669299126,
+            0.030384771525859833,
+            0.02756018005311489,
+            -0.0041348435916006565,
+            -0.019723162055015564,
+            -0.01155143789947033,
+            -0.010596301406621933,
+            0.0018419033149257302,
+            0.03285016864538193,
+            0.010506502352654934,
+            0.010890189558267593,
+            -0.011935125105082989,
+            -0.004104230087250471,
+            0.03025415539741516,
+            -0.021192602813243866,
+            0.01650671847164631,
+            0.0041511706076562405,
+            -0.002029665280133486,
+            -0.04320156201720238,
+            -0.035266581922769547,
+            0.007845181971788406,
+            0.011747363954782486,
+            0.014874007552862167,
+            0.0368339866399765,
+            0.025094784796237946,
+            -0.01662100851535797,
+            0.030629679560661316,
+            -0.01905374974012375,
+            0.027984686195850372,
+            0.039054471999406815,
+            0.011624909937381744,
+            -0.018351683393120766,
+            0.012237177230417728,
+            0.002424577483907342,
+            -0.025127438828349113,
+            0.01680060662329197,
+            -0.032735876739025116,
+            0.01738838478922844,
+            -0.02354370802640915,
+            0.025143766775727272,
+            -0.02579685114324093,
+            -0.009290131740272045,
+            -0.004428731743246317,
+            -0.006130834110081196,
+            0.04888340085744858,
+            -0.026727497577667236,
+            -0.0142454132437706,
+            -0.0024694770108908415,
+            -0.0024531499948352575,
+            -0.008718682453036308,
+            0.010000362060964108,
+            -0.045781247317790985,
+            -0.01844964735209942,
+            -0.04016471654176712,
+            0.007355368230491877,
+            0.005412440747022629,
+            -0.030907239764928818,
+            -0.05694899708032608,
+            0.025421326979994774,
+            -0.06331657618284225,
+            -0.019886434078216553,
+            0.051626358181238174,
+            -0.018400665372610092,
+            -0.008212542161345482,
+            0.009641164913773537,
+            0.006375740747898817,
+            -0.029290854930877686,
+            0.0041348435916006565,
+            0.02504580281674862,
+            0.04418119043111801,
+            -0.0045797573402523994,
+            -0.02865409664809704,
+            0.0028674507047981024,
+            -0.027788760140538216,
+            -0.013461710885167122,
+            0.005261414684355259,
+            0.016449574381113052,
+            0.024539662525057793,
+            0.007628847379237413,
+            0.03585435822606087,
+            0.007294141221791506,
+            0.006318595726042986,
+            0.044279150664806366,
+            -0.025666235014796257,
+            -0.027380581945180893,
+            0.02519274689257145,
+            0.04045860469341278,
+            0.023151857778429985,
+            -0.018253721296787262,
+            -0.0142454132437706,
+            0.0053063142113387585,
+            0.0009296254720538855,
+            -0.022139575332403183,
+            0.002808264922350645,
+            -0.013902543112635612,
+            0.005143043119460344,
+            0.03892385587096214,
+            -0.001205145614221692,
+            0.014196431264281273,
+            0.007302304729819298,
+            0.006845145486295223,
+            0.004334850702434778,
+            -0.011731036007404327,
+            -0.023592690005898476,
+            0.00032220553839579225,
+            0.034972693771123886,
+            0.015404638834297657,
+            -0.030515389516949654,
+            -0.00995954405516386,
+            0.034711457788944244,
+            -0.00795130804181099,
+            -0.009118696674704552,
+            -0.009771781973540783,
+            0.003385836724191904,
+            0.02638462744653225,
+            -0.01606588624417782,
+            0.006649219896644354,
+            0.03716052696108818,
+            -0.019870106130838394,
+            0.005543057806789875,
+            -0.025405000895261765,
+            0.0018061877926811576,
+            0.02442537248134613,
+            -0.006216551177203655,
+            0.028098974376916885,
+            5.6315813708351925e-05,
+            0.03025415539741516,
+            -0.004677720367908478,
+            0.008432958275079727,
+            -0.020327266305685043,
+            0.03987082839012146,
+            -0.017633290961384773,
+            0.03402572125196457,
+            0.02067013457417488,
+            -0.01258004643023014,
+            0.04783846437931061,
+            -0.036801330745220184,
+            -0.020784424617886543,
+            0.00584919098764658,
+            0.012743317522108555,
+            0.0028919412288814783,
+            -0.0191517136991024,
+            -0.018400665372610092,
+            -0.010996315628290176,
+            0.02380494214594364,
+            -0.022547753527760506,
+            -0.016865916550159454,
+            -0.007441085297614336,
+            -0.01890680566430092,
+            -0.009673818945884705,
+            -0.007988044060766697,
+            -0.01709449477493763,
+            -0.016547536477446556,
+            0.044573038816452026,
+            -0.06334923207759857,
+            0.016996532678604126,
+            0.017159804701805115,
+            0.011477965861558914,
+            -0.034678805619478226,
+            -0.019510909914970398,
+            -0.017633290961384773,
+            -0.002347023691982031,
+            -0.030743969604372978,
+            0.001702102366834879,
+            0.01725776679813862,
+            0.01138000376522541,
+            -0.05022222548723221,
+            0.012335140258073807,
+            0.02772345207631588,
+            -0.012065742164850235,
+            0.0014949520118534565,
+            -0.023135529831051826,
+            -0.017176130786538124,
+            0.0031776658724993467,
+            0.022123249247670174,
+            0.005975726060569286,
+            -0.008188051171600819,
+            -0.016865916550159454,
+            -0.005241005681455135,
+            -0.006045116577297449,
+            -0.002359268954023719,
+            0.005094061605632305,
+            0.011780017986893654,
+            0.019690508022904396,
+            -0.0024633544962853193,
+            -0.004436895251274109,
+            -0.006396149750798941,
+            0.027364253997802734,
+            -0.001450052484869957,
+            -0.008416631259024143,
+            0.050907962024211884,
+            0.03556047007441521,
+            0.006094098091125488,
+            -0.01108611561357975,
+            -0.03990348428487778,
+            0.010661610402166843,
+            -0.022711025550961494,
+            0.0401320643723011,
+            0.01905374974012375,
+            -0.008298259228467941,
+            -0.028866350650787354,
+            0.010294250212609768,
+            -0.02290695160627365,
+            -0.009975871071219444,
+            0.009469729848206043,
+            0.00726556871086359,
+            -0.03263791650533676,
+            -0.013543346896767616,
+            0.03510330989956856,
+            0.011437147855758667,
+            -0.006841063499450684,
+            -0.036050282418727875,
+            0.008588065393269062,
+            0.01646590232849121,
+            0.011869817040860653,
+            -0.019233347848057747,
+            0.004449140280485153,
+            -0.012041252106428146,
+            0.017470018938183784,
+            0.008849299512803555,
+            -0.017796561121940613,
+            0.018237393349409103,
+            0.004710374400019646,
+            -0.008490103296935558,
+            -0.003912386484444141,
+            -0.0163107942789793,
+            -0.02006603218615055,
+            0.03840138763189316,
+            0.024490682408213615,
+            0.026335645467042923,
+            0.005457340274006128,
+            0.02008235827088356,
+            0.03227871656417847,
+            0.011282040737569332,
+            -0.03134807199239731,
+            0.027935704216361046,
+            0.016996532678604126,
+            -0.04604247957468033,
+            0.021551799029111862,
+            -0.02218855731189251,
+            -0.028670424595475197,
+            0.009649328887462616,
+            -0.0017367975087836385,
+            0.011192241683602333,
+            0.0034837995190173388,
+            -0.003642988856881857,
+            -0.03307874873280525,
+            -0.006236960180103779,
+            0.015771998092532158,
+            0.015502601861953735,
+            -0.0048695639707148075,
+            -0.003296037670224905,
+            -0.011886144056916237,
+            0.00487364549189806,
+            0.009796272963285446,
+            0.039054471999406815,
+            0.001381682581268251,
+            -0.004751192405819893,
+            -0.0018765984568744898,
+            0.031854212284088135,
+            0.0271193478256464,
+            -0.017274094745516777,
+            -0.026286665350198746,
+            0.00580837344750762,
+            -0.014253576286137104,
+            0.026156047359108925,
+            -0.00487364549189806,
+            0.011877980083227158,
+            0.008718682453036308,
+            -0.0007908449042588472,
+            -0.001373519073240459,
+            0.025584599003195763,
+            -0.02710302174091339,
+            -0.021764053031802177,
+            0.031233781948685646,
+            -0.025731543079018593,
+            0.028735732659697533,
+            0.013061696663498878,
+            -0.01557607389986515,
+            -0.015225040726363659,
+            -0.010302413254976273,
+            0.009943217039108276,
+            -0.012277995236217976,
+            0.023853924125432968,
+            0.003608293831348419,
+            0.006392067763954401,
+            0.0003584313380997628,
+            0.02770712412893772,
+            0.024833550676703453,
+            0.007820690982043743,
+            -0.018074123188853264,
+            -0.0016816934803500772,
+            -0.02320083975791931,
+            -0.01680060662329197,
+            0.023772288113832474,
+            -0.018090449273586273,
+            -0.03268689662218094,
+            -0.028556134551763535,
+            0.012114724144339561,
+            0.007392104249447584,
+            0.010816717520356178,
+            0.03461349755525589,
+            -0.030384771525859833,
+            -0.0344502255320549,
+            0.01766594499349594,
+            0.026270337402820587,
+            -0.005551221314817667,
+            -0.021535472944378853,
+            0.03085825778543949,
+            -0.005828781984746456,
+            0.017600636929273605,
+            0.0404912605881691,
+            -0.0016225076979026198,
+            0.010228941217064857,
+            -0.042026009410619736,
+            -0.014057651162147522,
+            -0.0066614653915166855,
+            0.006261450704187155,
+            0.034417569637298584,
+            0.01048201136291027,
+            0.01535565685480833,
+            0.03448288142681122,
+            -0.021192602813243866,
+            -0.0002903166168835014,
+            -0.0022449791431427,
+            -0.02818061038851738,
+            0.03347059711813927,
+            0.025715215131640434,
+            0.011918798089027405,
+            0.019706835970282555,
+            -0.03794422745704651,
+            -0.013037205673754215,
+            0.027070365846157074,
+            0.015053605660796165,
+            0.01273515447974205,
+            0.016694480553269386,
+            0.011249386705458164,
+            0.00868602842092514,
+            -0.021633435040712357,
+            -0.05169166624546051,
+            0.01214737817645073,
+            0.01993541419506073,
+            -0.016449574381113052,
+            0.01057181041687727,
+            0.024980494752526283,
+            0.007620683871209621,
+            -0.007045152597129345,
+            0.012808626517653465,
+            -0.02682545967400074,
+            0.002259265398606658,
+            0.0019010890973731875,
+            -0.014482156373560429,
+            0.038270771503448486,
+            -0.016367938369512558,
+            0.011363675817847252,
+            -0.018204739317297935,
+            0.023870250210165977,
+            0.00011531029304023832,
+            0.0007964573451317847,
+            0.014963806606829166,
+            -0.011347348801791668,
+            -0.04235255345702171,
+            0.0021613026037812233,
+            0.03342161700129509,
+            -0.015200549736618996,
+            0.026449935510754585,
+            -0.0020674217958003283,
+            0.032458316534757614,
+            0.042777057737112045,
+            0.017731253057718277,
+            0.0003497575526125729,
+            0.025682561099529266,
+            0.0036144163459539413,
+            0.0073880222626030445,
+            0.012400448322296143,
+            0.009322785772383213,
+            0.007245160173624754,
+            0.030809277668595314,
+            -0.010261595249176025,
+            0.02173139713704586,
+            -0.04760988429188728,
+            -0.014629100449383259,
+            0.012024925090372562,
+            -0.0013316808035597205,
+            -0.012449429370462894,
+            -0.033029764890670776,
+            0.012563719414174557,
+            -0.03503800183534622,
+            0.022074267268180847,
+            -0.0004959363141097128,
+            0.04333217814564705,
+            -0.02187834307551384,
+            -0.005518566817045212,
+            0.016694480553269386,
+            0.0016510800924152136,
+            0.023282473906874657,
+            -0.0012153501156717539,
+            -0.016416920349001884,
+            -0.0067798369564116,
+            -0.009257477708160877,
+            0.004138925112783909,
+            0.04375668615102768,
+            -0.003722583642229438,
+            -0.0001682458823779598,
+            0.01409030519425869,
+            -0.011959616094827652,
+            -0.03250729665160179,
+            0.045193471014499664,
+            -0.0030837850645184517,
+            0.02484987862408161,
+            0.04979771748185158,
+            -0.06410028040409088,
+            -0.019543563947081566,
+            0.013061696663498878,
+            -0.010441194288432598,
+            0.011363675817847252,
+            0.005547139327973127,
+            0.016114868223667145,
+            0.017894525080919266,
+            -0.005485912784934044,
+            0.01619650423526764,
+            0.005106307100504637,
+            0.028098974376916885,
+            -0.02336410991847515,
+            -0.03296445682644844,
+            -0.02636830136179924,
+            -0.017747581005096436,
+            0.0014480114914476871,
+            -0.015796490013599396,
+            -3.563585551091819e-06,
+            -0.0022204883862286806,
+            -0.04388730227947235,
+            -0.025274382904171944,
+            0.040687184780836105,
+            0.006706364918500185,
+            0.02458864450454712,
+            0.0077268099412322044,
+            0.03418898954987526,
+            0.005885927006602287,
+            -0.009037061594426632,
+            0.030058229342103004,
+            -0.020049704238772392,
+            0.03833607956767082,
+            0.022106921300292015,
+            -0.002832755446434021,
+            0.005865518003702164,
+            0.031086837872862816,
+            -0.033797141164541245,
+            0.008086007088422775,
+            -0.034384917467832565,
+            0.0046695563942193985,
+            -0.008620720356702805,
+            -0.015094423666596413,
+            -0.009526874870061874,
+            -0.02504580281674862,
+            0.011216731742024422,
+            -0.021535472944378853,
+            0.01590261608362198,
+            0.010416703298687935,
+            -0.002916431985795498,
+            0.03448288142681122,
+            -0.014204595237970352,
+            -0.0027796924114227295,
+            0.0024531499948352575,
+            -0.027070365846157074,
+            -0.006343086715787649,
+            -0.003377673216164112,
+            -0.03689929470419884,
+            -0.004502203781157732,
+            0.010988152585923672,
+            -0.00024184548237826675,
+            0.016898570582270622,
+            0.02787039615213871,
+            0.0005092020728625357,
+            -0.022302847355604172,
+            0.002340900944545865,
+            -0.04199335351586342,
+            0.0011877981014549732,
+            0.01483318954706192,
+            -0.017600636929273605,
+            -0.008612556383013725,
+            0.033046092838048935,
+            0.023298801854252815,
+            -0.009396257810294628,
+            -0.035005345940589905,
+            0.017796561121940613,
+            -0.011853490024805069,
+            0.027511198073625565,
+            -0.0002630197268445045,
+            0.019298657774925232,
+            -0.0076941559091210365,
+            0.04493223875761032,
+            0.001369437319226563,
+            0.008645210415124893,
+            -0.021959977224469185,
+            0.007020662073045969,
+            -0.020278284326195717,
+            -0.002406209474429488,
+            -0.006171651650220156,
+            -0.012751481495797634,
+            -0.010669773444533348,
+            0.0032735876739025116,
+            0.007730891928076744,
+            -0.0176822729408741,
+            0.0055961208418011665,
+            -0.015959760174155235,
+            -0.02741323597729206,
+            0.007543129846453667,
+            -0.05965929850935936,
+            -0.0179598331451416,
+            -0.03373183310031891,
+            -0.009477893821895123,
+            0.02008235827088356,
+            0.03510330989956856,
+            -0.012759645469486713,
+            0.00497977202758193,
+            -0.014898497611284256,
+            0.004477712791413069,
+            0.03490738570690155,
+            0.044442422688007355,
+            -0.014759717509150505,
+            0.003381754970178008,
+            -0.010955498553812504,
+            0.033029764890670776,
+            -0.03347059711813927,
+            0.0030062312725931406,
+            0.031086837872862816,
+            -0.005057325586676598,
+            0.007959471084177494,
+            0.011624909937381744,
+            0.020474210381507874,
+            -0.019396619871258736,
+            0.00877582747489214,
+            0.002100075827911496,
+            0.019233347848057747,
+            0.02171507105231285,
+            0.020343592390418053,
+            -0.03539719805121422,
+            0.028409190475940704,
+            0.018384337425231934,
+            0.02444170042872429,
+            0.044442422688007355,
+            -0.03974021226167679,
+            -0.004147088620811701,
+            -0.0032225653994828463,
+            -0.02997659333050251,
+            0.02008235827088356,
+            -0.007530884351581335,
+            -0.01841699331998825,
+            -0.007069643586874008,
+            0.03327467292547226,
+            0.014571955427527428,
+            -0.05051611363887787,
+            0.01980479806661606,
+            0.01740471087396145,
+            0.02055584453046322,
+            0.024474354460835457,
+            -0.01123305968940258,
+            -0.0019072118448093534,
+            0.008996243588626385,
+            0.006804327480494976,
+            0.019266001880168915,
+            0.015257694758474827,
+            0.002012317767366767,
+            -0.0022082431241869926,
+            -0.011837163008749485,
+            0.005224678665399551,
+            -0.009077879600226879,
+            0.010808554477989674,
+            -0.02458864450454712,
+            -0.009698309935629368,
+            0.021388528868556023,
+            0.0031164393294602633,
+            -0.013665799982845783,
+            -0.03729114308953285,
+            -0.045944519340991974,
+            0.0033837957307696342,
+            0.004543021321296692,
+            -0.05365091934800148,
+            0.005322641227394342,
+            -0.026547899469733238,
+            -0.002108239568769932,
+            -0.005502239800989628,
+            0.0019214981002733111,
+            0.005636938381940126,
+            0.0167679525911808,
+            -0.013494365848600864,
+            0.02248244546353817,
+            -0.015502601861953735,
+            0.03165828809142113,
+            0.0055961208418011665,
+            0.007375776767730713,
+            -0.0027796924114227295,
+            0.05590406060218811,
+            -0.003712379140779376,
+            -0.03699725493788719,
+            0.023282473906874657,
+            -0.01311067771166563,
+            -0.0001747512142173946,
+            0.030090883374214172,
+            0.00884113647043705,
+            -0.015257694758474827,
+            0.013437220826745033,
+            -0.030613351613283157,
+            -0.03249097242951393,
+            0.010996315628290176,
+            -0.013755599036812782,
+            -0.0015776080545037985,
+            -0.04568328335881233,
+            -0.009339112788438797,
+            0.004269542172551155,
+            -0.018335357308387756,
+            0.010775899514555931,
+            0.01631895825266838,
+            0.013992343097925186,
+            -0.024245774373412132,
+            0.009053388610482216,
+            0.006865554489195347,
+            0.013453547842800617,
+            0.0007331897504627705,
+            0.014049487188458443,
+            -0.008677864447236061,
+            -0.011306531727313995,
+            -0.018319029361009598,
+            0.0032450153958052397,
+            -0.026792805641889572,
+            0.005155288614332676,
+            -0.003155216109007597,
+            -0.03771565109491348,
+            -0.06521052122116089,
+            -0.0344502255320549,
+            0.016294466331601143,
+            0.03396041318774223,
+            -0.009045224636793137,
+            0.0035144127905368805,
+            -0.00023648813657928258,
+            -0.01093917153775692,
+            0.024523336440324783,
+            -0.005930826533585787,
+            -0.0004829256213270128,
+            0.0035389035474509,
+            0.02050686441361904,
+            -0.009722800925374031,
+            0.005845109466463327,
+            -0.051952898502349854,
+            -0.01586179807782173,
+            0.01812310516834259,
+            0.006306350696831942,
+            0.014629100449383259,
+            -0.009690145961940289,
+            0.01693122461438179,
+            -0.030074557289481163,
+            -0.028147956356406212,
+            0.04130761697888374,
+            0.008277850225567818,
+            0.011486129835247993,
+            -0.003055212553590536,
+            -0.01680060662329197,
+            0.02504580281674862,
+            -0.0020633398089557886,
+            -0.006461458280682564,
+            -0.00817988719791174,
+            0.015820980072021484,
+            0.044279150664806366,
+            0.003255219664424658,
+            0.0009821783751249313,
+            0.017045514658093452,
+            -0.002663361607119441,
+            -0.011575928889214993,
+            0.007677828427404165,
+            0.013592327944934368,
+            -0.016384266316890717,
+            0.010375885292887688,
+            -0.025862159207463264,
+            0.0332583449780941,
+            0.03131541982293129,
+            0.011575928889214993,
+            -0.015045441687107086,
+            0.006130834110081196,
+            0.005330805201083422,
+            0.008086007088422775,
+            -0.001369437319226563,
+            -0.0017133272485807538,
+            0.020719116553664207,
+            -0.03276853263378143,
+            -0.01693122461438179,
+            -0.005347132217139006,
+            0.003932795487344265,
+            0.005277741700410843,
+            0.03396041318774223,
+            -0.0033266509417444468,
+            0.017927179113030434,
+            -0.011657563969492912,
+            0.010579974390566349,
+            0.021764053031802177,
+            0.037095218896865845,
+            -0.024996822699904442,
+            -0.029862305149435997,
+            0.012914752587676048,
+            -0.001296985661610961,
+            0.014065815135836601,
+            0.0010173837654292583,
+            0.014139287173748016,
+            0.0029409227427095175,
+            0.060475654900074005,
+            -0.017763907089829445,
+            0.004628738854080439,
+            0.0004035860183648765,
+            -0.01858026348054409,
+            -0.0013531101867556572,
+            0.004261378664523363,
+            -0.010979988612234592,
+            0.03608293831348419,
+            0.002344982698559761,
+            0.039185091853141785,
+            0.013918871060013771,
+            0.009290131740272045,
+            0.0015143405180424452,
+            -0.013592327944934368,
+            0.008355404250323772,
+            0.0015592400450259447,
+            -0.001132694073021412,
+            0.009461566805839539,
+            0.019298657774925232,
+            0.009714636951684952,
+            0.025274382904171944,
+            0.009788108989596367,
+            0.012857607565820217,
+            -0.029470453038811684,
+            0.03510330989956856,
+            -0.0006224715034477413,
+            0.012996388599276543,
+            -0.02040890045464039,
+            -0.003136848099529743,
+            -0.0008240094175562263,
+            0.015649545937776566,
+            -0.015616890974342823,
+            -0.001201063860207796,
+            -0.01297189760953188,
+            0.040099408477544785,
+            -0.018988441675901413,
+            0.013976015150547028,
+            0.009673818945884705,
+            -0.02865409664809704,
+            -0.007596192881464958,
+            0.01995174214243889,
+            0.014294394291937351,
+            -0.0053144777193665504,
+            -0.020490536466240883,
+            0.005832863971590996,
+            0.015380147844552994,
+            -0.028572462499141693,
+            1.3584674888988957e-05,
+            0.014751553535461426,
+            -0.0164903923869133,
+            -0.00025409081717953086,
+            -0.004702210891991854,
+            0.003975654020905495,
+            -0.009265640750527382,
+            0.007028825581073761,
+            0.016057724133133888,
+            -0.024507008492946625,
+            -0.020049704238772392,
+            0.005722655914723873,
+            0.006571666337549686,
+            0.01444950234144926,
+            0.007739055436104536,
+            0.004355259705334902,
+            -0.020033378154039383,
+            -0.01799248717725277,
+            0.0032674651592969894,
+            -0.01649855636060238,
+            -0.0048614004626870155,
+            0.0020531355403363705,
+            -0.019363965839147568,
+            0.01875986158847809,
+            -0.010743245482444763,
+            0.00017538899555802345,
+            -0.003226647386327386,
+            -0.020800752565264702,
+            0.006204306147992611,
+            -0.004865481983870268,
+            0.019870106130838394,
+            0.003555230563506484,
+            0.02785406820476055,
+            0.02310287579894066,
+            -0.004428731743246317,
+            0.017600636929273605,
+            -0.016155686229467392,
+            0.007983962073922157,
+            -0.031135819852352142,
+            -0.001782717532478273,
+            -0.010408539324998856,
+            0.0036103345919400454,
+            -0.015943434089422226,
+            -0.009126860648393631,
+            -0.0048777274787425995,
+            -0.0006678812787868083,
+            -0.004820582456886768,
+            0.0004275664687156677,
+            0.0027817331720143557,
+            0.02231917530298233,
+            -0.013208640739321709,
+            -0.02803366631269455,
+            -0.017437364906072617,
+            0.006045116577297449,
+            0.019396619871258736,
+            -0.026025431230664253,
+            -0.009184005670249462,
+            -0.005012426059693098,
+            -0.004326687194406986,
+            -0.021404854953289032,
+            -0.015167895704507828,
+            -0.005632856860756874,
+            -0.008045189082622528,
+            -0.010522829368710518,
+            0.0046532293781638145,
+            -0.0035715578123927116,
+            -0.011739199981093407,
+            0.021666089072823524,
+            0.018057795241475105,
+            -0.019837452098727226,
+            -0.019837452098727226,
+            0.013641309924423695,
+            -0.006159406621009111,
+            0.0018878233386203647,
+            0.02008235827088356,
+            0.00029286774224601686,
+            0.005824700463563204,
+            0.020882388576865196,
+            -0.0035225762985646725,
+            -0.01797616109251976,
+            0.003577680326998234,
+            -0.0018888438353314996,
+            -0.012122887186706066,
+            0.003212361130863428,
+            0.02773977816104889,
+            0.0008076822850853205,
+            0.023429417982697487,
+            -0.03056436963379383,
+            -0.011412657797336578,
+            0.006061443593353033,
+            0.02369065210223198,
+            0.00211844383738935,
+            -0.0021694661118090153,
+            0.020131340250372887,
+            0.03941367194056511,
+            0.005775718949735165,
+            0.0015755671774968505,
+            -0.04003410041332245,
+            0.01027792226523161,
+            0.009208496659994125,
+            0.024621298536658287,
+            0.008339077234268188,
+            0.0014653591206297278,
+            -0.012906589545309544,
+            -0.009755454957485199,
+            -0.0017939424142241478,
+            0.014702572487294674,
+            -0.023870250210165977,
+            0.0024755997583270073,
+            -0.03157665207982063,
+            0.005094061605632305,
+            0.0021959978621453047,
+            -0.009559528902173042,
+            0.0335359051823616,
+            0.00980443600565195,
+            -0.02065380848944187,
+            0.015151568688452244,
+            0.005767555441707373,
+            0.007330877240747213,
+            0.022449791431427002,
+            -0.008228869177401066,
+            0.030450081452727318,
+            -0.005049162078648806,
+            0.01213921420276165,
+            0.02742956392467022,
+            0.032588932663202286,
+            -0.013102514669299126,
+            -0.003959327004849911,
+            0.0006097159348428249,
+            -0.0002561317232903093,
+            0.0007995187188498676,
+            -0.008204378187656403,
+            0.0037389106582850218,
+            -0.014800535514950752,
+            0.012710663489997387,
+            -0.019543563947081566,
+            -0.0034286954905837774,
+            0.0038532004691660404,
+            0.01991908811032772,
+            0.0016265894519165158,
+            -0.033633869141340256,
+            0.02055584453046322,
+            -0.015420965850353241,
+            0.0135596739128232,
+            0.001290863030590117,
+            -0.011110605672001839,
+            -0.00029210239881649613,
+            0.020588500425219536,
+            0.010947334580123425,
+            -0.007845181971788406,
+            -0.0135596739128232,
+            -0.028376536443829536,
+            -0.021535472944378853,
+            0.00823703221976757,
+            0.009445239789783955,
+            -0.0028837777208536863,
+            0.0021306893322616816,
+            -0.011869817040860653,
+            0.0006520643946714699,
+            -0.0008122742874547839,
+            0.01740471087396145,
+            -0.0335359051823616,
+            0.021943651139736176,
+            -0.019527235999703407,
+            -0.019543563947081566,
+            0.017045514658093452,
+            0.008049271069467068,
+            0.009445239789783955,
+            0.0033552234526723623,
+            -0.01995174214243889,
+            -0.012326976284384727,
+            0.0338951013982296,
+            -0.01138000376522541,
+            0.006298186723142862,
+            0.004408322740346193,
+            0.012914752587676048,
+            -0.008914608508348465,
+            -0.011926962062716484,
+            0.01784554310142994,
+            -0.009861581027507782,
+            0.013951525092124939,
+            0.0010010566329583526,
+            -0.0254703089594841,
+            0.02739690989255905,
+            -0.019233347848057747,
+            0.03487472981214523,
+            -0.021617108955979347,
+            0.00711862463504076,
+            0.0066614653915166855,
+            0.008792154490947723,
+            -0.006690037902444601,
+            -0.013224967755377293,
+            0.0012786176521331072,
+            0.03807484731078148,
+            -0.02623768337070942,
+            0.024180466309189796,
+            0.022400809451937675,
+            -0.00040766780148260295,
+            0.02065380848944187,
+            -0.008367649279534817,
+            -0.026711169630289078,
+            0.0179598331451416,
+            -0.012596373446285725,
+            -0.026123393326997757,
+            0.013257622718811035,
+            -0.022678371518850327,
+            -0.020278284326195717,
+            -0.026302991434931755,
+            0.004947117529809475,
+            -0.0025694805663079023,
+            -0.005151206627488136,
+            0.009004407562315464,
+            0.02561725303530693,
+            0.020915042608976364,
+            0.019576217979192734,
+            0.02111096680164337,
+            0.01161674689501524,
+            0.022449791431427002,
+            -0.007559456862509251,
+            0.04401791840791702,
+            -0.004947117529809475,
+            0.004632820375263691,
+            -0.0015980169409886003,
+            -0.015657709911465645,
+            0.009771781973540783,
+            0.0119106350466609,
+            0.0029572497587651014,
+            -0.029176564887166023,
+            -0.009061552584171295,
+            -0.03660540655255318,
+            0.005053244065493345,
+            0.0020664012990891933,
+            0.0176822729408741,
+            0.032115448266267776,
+            -0.005865518003702164,
+            0.004592002835124731,
+            -0.005102225113660097,
+            -0.013927034102380276,
+            -0.009420748800039291,
+            -0.02739690989255905,
+            -0.025437654927372932,
+            -0.007653337903320789,
+            -0.013739272020757198,
+            -0.030368445441126823,
+            -0.011722872965037823,
+            0.01551892887800932,
+            -0.007645174395292997,
+            0.011918798089027405,
+            -0.016702644526958466,
+            0.01385356206446886,
+            -0.004110352601855993,
+            -0.004763437435030937,
+            -0.004530776292085648,
+            -0.017600636929273605,
+            -0.006420640274882317,
+            0.014367866329848766,
+            -0.009477893821895123,
+            -0.01237595733255148,
+            -0.02215590327978134,
+            -0.0015459742862731218,
+            -0.01574750803411007,
+            0.003489922033622861,
+            -0.014596446417272091,
+            -0.006771673448383808,
+            -0.003908304497599602,
+            -0.018514955416321754,
+            0.01766594499349594,
+            0.014816862531006336,
+            -0.019380291923880577,
+            -0.0090697156265378,
+            0.02638462744653225,
+            0.042777057737112045,
+            -0.03218075633049011,
+            -0.008200296200811863,
+            0.0020378287881612778,
+            0.018090449273586273,
+            0.010171796195209026,
+            0.016702644526958466,
+            -0.024653952568769455,
+            0.0022286521270871162,
+            -0.021649762988090515,
+            -0.001220452249981463,
+            -0.023739634081721306,
+            0.010775899514555931,
+            0.025715215131640434,
+            -0.010971825569868088,
+            0.025976449251174927,
+            0.01634344831109047,
+            -0.01812310516834259,
+            0.012016761116683483,
+            0.007183933164924383,
+            0.014645427465438843,
+            -0.019755816087126732,
+            0.0167679525911808,
+            -0.015796490013599396,
+            0.031233781948685646,
+            0.00892277155071497,
+            -0.04829562455415726,
+            -0.03503800183534622,
+            -0.013388238847255707,
+            0.0013388239312916994,
+            -0.010702427476644516,
+            0.015037278644740582,
+            0.015967924147844315,
+            0.013959688134491444,
+            -0.019380291923880577,
+            -0.015820980072021484,
+            0.011984107084572315,
+            0.010065670125186443,
+            0.027462217956781387,
+            -0.006649219896644354,
+            -0.004212397150695324,
+            -0.00960034690797329,
+            -0.02249877341091633,
+            -0.016098542138934135,
+            0.0209803506731987,
+            -0.018041469156742096,
+            0.015935270115733147,
+            -0.026139721274375916,
+            -0.012449429370462894,
+            0.017453692853450775,
+            -0.05211617052555084,
+            -0.01453113742172718,
+            0.0007612520130351186,
+            0.03012353740632534,
+            0.004751192405819893,
+            0.0035184945445507765,
+            0.01841699331998825,
+            -0.01993541419506073,
+            -0.0039776950143277645,
+            -0.02099667675793171,
+            0.006832899991422892,
+            -0.011853490024805069,
+            -0.013984179124236107,
+            0.014482156373560429,
+            -0.013845399022102356,
+            -0.01601690612733364,
+            0.0034368589986115694,
+            0.007183933164924383,
+            -0.021094640716910362,
+            0.011951452121138573,
+            0.003945040516555309,
+            0.012171869166195393,
+            0.012171869166195393,
+            0.01199227012693882,
+            0.021551799029111862,
+            0.02295593172311783,
+            -0.007334959227591753,
+            -0.004767519421875477,
+            -0.01004117913544178,
+            -0.0015398516552522779,
+            0.008898280560970306,
+            0.0033797139767557383,
+            -0.002989904023706913,
+            0.007837017998099327,
+            0.010033016093075275,
+            -0.01498013362288475,
+            0.033193036913871765,
+            -0.0035878848284482956,
+            -0.04134026914834976,
+            -0.003238892648369074,
+            -0.03800953924655914,
+            0.020033378154039383,
+            -0.023298801854252815,
+            -0.016996532678604126,
+            0.0032776694279164076,
+            -0.00629410520195961,
+            -0.019902760162949562,
+            0.013608654960989952,
+            0.011404493823647499,
+            0.00258580781519413,
+            0.02021297626197338,
+            0.012269831262528896,
+            -0.005134879611432552,
+            0.026172375306487083,
+            -0.014718899503350258,
+            -0.008751336485147476,
+            0.015567909926176071,
+            0.006853308994323015,
+            0.0022347746416926384,
+            0.0011031010653823614,
+            -0.018825169652700424,
+            0.005098143592476845,
+            0.0164903923869133,
+            0.007216587662696838,
+            -0.000566347036510706,
+            0.014049487188458443,
+            0.012514738366007805,
+            -0.015135240741074085,
+            -0.0002213090192526579,
+            -0.026564225554466248,
+            0.0032817514147609472,
+            0.004567512311041355,
+            0.007506393827497959,
+            -0.006771673448383808,
+            0.006281859707087278,
+            0.00658799335360527,
+            0.014759717509150505,
+            -0.010155469179153442,
+            -0.016980206593871117,
+            -0.002510294783860445,
+            0.004285869188606739,
+            0.016694480553269386,
+            0.03157665207982063,
+            -0.010310577228665352,
+            -0.005734901409596205,
+            -0.0386626236140728,
+            0.02070278860628605,
+            0.03332365304231644,
+            0.011175914667546749,
+            -0.028213264420628548,
+            0.012302485294640064,
+            0.0006515541463159025,
+            0.0019276207312941551,
+            -0.026302991434931755,
+            0.02787039615213871,
+            0.00974729098379612,
+            0.0066614653915166855,
+            0.017861871048808098,
+            0.0022306928876787424,
+            0.0029266364872455597,
+            -0.016302630305290222,
+            0.01018812321126461,
+            0.009086042642593384,
+            -0.017306748777627945,
+            0.0009745250572450459,
+            0.02324981987476349,
+            -0.01423724927008152,
+            -0.003208279376849532,
+            0.0018133309204131365,
+            0.02651524543762207,
+            -0.013820908032357693,
+            -0.014294394291937351,
+            0.0005714492290280759,
+            0.01751900091767311,
+            -0.0007979880319908261,
+            0.012098397128283978,
+            0.04898136481642723,
+            -0.012424939312040806,
+            0.019657853990793228,
+            0.005461421795189381,
+            -0.015290348790585995,
+            -0.032605260610580444,
+            -0.004922627005726099,
+            -0.01830270327627659,
+            0.011200404725968838,
+            0.014343376271426678,
+            -0.015763835981488228,
+            0.0007525782566517591,
+            -0.03144603595137596,
+            0.007743136957287788,
+            0.014735226519405842,
+            0.018776189535856247,
+            -0.002440904499962926,
+            0.006134915631264448,
+            0.027674470096826553,
+            0.02458864450454712,
+            -0.007392104249447584,
+            0.013804581016302109,
+            0.02320083975791931,
+            -0.025062130764126778,
+            -0.003069498809054494,
+            0.006820654962211847,
+            0.004104230087250471,
+            0.0053063142113387585,
+            -0.016963878646492958,
+            0.03915243595838547,
+            -0.005922663025557995,
+            -0.01506176870316267,
+            0.0030266400426626205,
+            -0.009028897620737553,
+            0.0063390047289431095,
+            -0.012563719414174557,
+            0.012637191452085972,
+            -0.008114579133689404,
+            0.00022883480414748192,
+            -0.0059430720284581184,
+            -0.011110605672001839,
+            -0.012653518468141556,
+            -0.0007949267164804041,
+            -0.016531210392713547,
+            0.007432921789586544,
+            0.00981259997934103,
+            0.005045080557465553,
+            0.01025343220680952,
+            -0.0036797248758375645,
+            -0.003136848099529743,
+            -0.016751626506447792,
+            -0.009984034113585949,
+            0.0073716952465474606,
+            0.01872720755636692,
+            -0.028588788583874702,
+            -0.018547609448432922,
+            0.009290131740272045,
+            0.0022898786701261997,
+            0.018776189535856247,
+            0.0158944521099329,
+            -0.0002595757250674069,
+            0.012318813242018223,
+            -0.01964152604341507,
+            -0.009184005670249462,
+            0.009690145961940289,
+            -0.003569516818970442,
+            0.01048201136291027,
+            -0.014759717509150505,
+            0.010033016093075275,
+            -0.011159587651491165,
+            -0.011053460650146008,
+            0.01385356206446886,
+            0.01738838478922844,
+            0.008873790502548218,
+            -0.001294944784604013,
+            3.383413059054874e-05,
+            -0.010506502352654934,
+            -0.024033522233366966,
+            0.004771600943058729,
+            0.027021385729312897,
+            -0.013273949734866619,
+            0.003883813973516226,
+            0.008702355436980724,
+            -0.017502672970294952,
+            -0.019821124151349068,
+            -0.012653518468141556,
+            -0.022106921300292015,
+            0.020572172477841377,
+            0.023413091897964478,
+            -0.02026195637881756,
+            0.00877582747489214,
+            -0.004620575346052647,
+            0.014049487188458443,
+            -0.01400050614029169,
+            0.010612628422677517,
+            0.0317889042198658,
+            -0.020000724121928215,
+            -0.0028654097113758326,
+            0.0036633978597819805,
+            -0.015347493812441826,
+            0.018629245460033417,
+            0.03268689662218094,
+            -0.00176945177372545,
+            0.014041324146091938,
+            0.0008138049161061645,
+            -0.009118696674704552,
+            0.01905374974012375,
+            0.011094278655946255,
+            0.005347132217139006,
+            -0.0042491331696510315,
+            -0.012212686240673065,
+            -0.03010721132159233,
+            0.016702644526958466,
+            0.0129637336358428,
+            -0.0037674831692129374,
+            -0.0011551438365131617,
+            -0.006343086715787649,
+            -0.015404638834297657,
+            0.0035123717971146107,
+            -0.013078023679554462,
+            -0.00803294312208891,
+            -0.008759500458836555,
+            0.004314441699534655,
+            -0.0034960447810590267,
+            -0.009053388610482216,
+            0.007236996199935675,
+            0.0009510547970421612,
+            -0.0010602424154058099,
+            -0.01902109570801258,
+            -0.025078458711504936,
+            -0.00995954405516386,
+            -0.016849588602781296,
+            0.0038287099450826645,
+            0.0005234883283264935,
+            -0.027184655889868736,
+            0.0053961132653057575,
+            -0.008139070123434067,
+            -0.008134988136589527,
+            0.007126788143068552,
+            0.006881881505250931,
+            -0.007437003776431084,
+            -0.010816717520356178,
+            -0.00786150898784399,
+            -0.006865554489195347,
+            0.020033378154039383,
+            0.0034348180051892996,
+            0.012082069180905819,
+            -0.010498338378965855,
+            -0.018025141209363937,
+            -0.016963878646492958,
+            0.016718972474336624,
+            -0.008971752598881721,
+            -0.02083340659737587,
+            -0.01444950234144926,
+            -0.008424794301390648,
+            -0.007665583398193121,
+            -0.01527402177453041,
+            -0.021159948781132698,
+            -0.0032103201374411583,
+            -0.011347348801791668,
+            -0.0067431009374558926,
+            -0.0028960229828953743,
+            0.0013571919407695532,
+            0.00951871182769537,
+            0.00546958576887846,
+            0.00438791373744607,
+            0.021666089072823524,
+            0.010122815147042274,
+            -0.005461421795189381,
+            0.016441410407423973,
+            -0.006183897145092487,
+            -0.004440976772457361,
+            -0.01498013362288475,
+            -0.009918726049363613,
+            0.018547609448432922,
+            -0.0055226488038897514,
+            0.008457448333501816,
+            -0.0009536059224046767,
+            -0.00576347392052412,
+            0.015674035996198654,
+            0.0129637336358428,
+            0.016718972474336624,
+            0.004771600943058729,
+            -0.00012168807734269649,
+            0.012416775338351727,
+            -0.0015745466807857156,
+            0.012555556371808052,
+            -0.0038429962005466223,
+            0.014506647363305092,
+            -0.009722800925374031,
+            -0.018988441675901413,
+            0.004157293122261763,
+            0.006265532691031694,
+            -0.012531065382063389,
+            0.009404421783983707,
+            0.007743136957287788,
+            -0.013747435994446278,
+            -0.02261306345462799,
+            -0.000106317616882734,
+            0.00014362763613462448,
+            0.008122743107378483,
+            0.008971752598881721,
+            -0.008073761127889156,
+            0.012890261597931385,
+            0.004616493359208107,
+            -0.007179851643741131,
+            -0.019657853990793228,
+            0.026009103283286095,
+            0.015608727931976318,
+            0.012269831262528896,
+            -0.052508022636175156,
+            -0.0030980713199824095,
+            0.010743245482444763,
+            -0.015951598063111305,
+            -0.026792805641889572,
+            -0.006404313258826733,
+            -0.008759500458836555,
+            -0.04290767386555672,
+            -0.009624837897717953,
+            -0.0005724696675315499,
+            -0.011388166807591915,
+            0.010400376282632351,
+            0.01573934406042099,
+            0.014612773433327675,
+            0.005489994306117296,
+            -0.007232914678752422,
+            -0.017633290961384773,
+            -0.02788672223687172,
+            0.010057506151497364,
+            -0.013380075804889202,
+            -0.006412476766854525,
+            -0.002263347152620554,
+            -0.007690073922276497,
+            0.0016112826997414231,
+            0.01691489666700363,
+            0.042483169585466385,
+            0.023037567734718323,
+            -0.012335140258073807,
+            -0.020131340250372887,
+            -0.012800462543964386,
+            -0.009192168712615967,
+            0.00633492274209857,
+            0.011592255905270576,
+            -0.004738946910947561,
+            0.0096656559035182,
+            -0.004294032696634531,
+            0.012302485294640064,
+            0.012767808511853218,
+            0.01680060662329197,
+            -2.3023791072773747e-05,
+            0.005579793825745583,
+            -0.0024694770108908415,
+            -0.01738838478922844,
+            -0.004379750229418278,
+            0.008261523209512234,
+            -0.0008872769540175796,
+            0.0019317024853080511,
+            0.014204595237970352,
+            0.019102731719613075,
+            -0.009094206616282463,
+            -0.014612773433327675,
+            -0.016098542138934135,
+            -0.005277741700410843,
+            -0.0012622905196622014,
+            0.029927613213658333,
+            0.02037624642252922,
+            -0.019266001880168915,
+            -0.018237393349409103,
+            -0.018351683393120766,
+            0.00922482367604971,
+            -0.003563394071534276,
+            -0.0064369672909379005,
+            0.01905374974012375,
+            0.023021239787340164,
+            0.017910851165652275,
+            -0.003642988856881857,
+            -0.014865843579173088,
+            0.003804219188168645,
+            0.02847449854016304,
+            0.008367649279534817,
+            0.01078406348824501,
+            0.008726846426725388,
+            -0.0011908593587577343,
+            -0.032441988587379456,
+            -0.0113146947696805,
+            0.00951871182769537,
+            -0.008481939323246479,
+            0.026564225554466248,
+            -0.016531210392713547,
+            0.009388094767928123,
+            -0.005077734589576721,
+            0.0019286411115899682,
+            -0.02365799807012081,
+            0.0014929111348465085,
+            0.010122815147042274,
+            -0.024947840720415115,
+            0.00336950970813632,
+            -0.023592690005898476,
+            -0.00013763252354692668,
+            0.008367649279534817,
+            0.0022837561555206776,
+            0.01650671847164631,
+            -0.0113146947696805,
+            0.0051838611252605915,
+            0.02099667675793171,
+            -0.002832755446434021,
+            -0.01512707769870758,
+            -0.0007842120248824358,
+            0.0005092020728625357,
+            -0.001286781276576221,
+            0.02563357912003994,
+            0.015812816098332405,
+            -0.0036756431218236685,
+            0.020882388576865196,
+            0.008677864447236061,
+            0.0019133344758301973,
+            -0.021764053031802177,
+            0.0023796779569238424,
+            -0.032899148762226105,
+            0.017437364906072617,
+            -0.011126932688057423,
+            0.012490247376263142,
+            0.00868602842092514,
+            -0.01341272983700037,
+            -0.0007388022495433688,
+            -0.0107187544927001,
+            -0.0015725058037787676,
+            -0.007934981025755405,
+            0.009779945947229862,
+            -0.004363423213362694,
+            0.008339077234268188,
+            -0.00024579971795901656,
+            -0.01476788055151701,
+            0.018694553524255753,
+            9.678283277025912e-06,
+            -0.00010925139940809458,
+            0.005441013257950544,
+            0.009935053065419197,
+            0.00786559097468853,
+            0.002009256277233362,
+            0.023870250210165977,
+            -0.031119491904973984,
+            -0.016082214191555977,
+            -0.010204450227320194,
+            0.005543057806789875,
+            -0.007922735065221786,
+            -0.018531281501054764,
+            -0.03257260471582413,
+            0.01318415068089962,
+            0.001293924287892878,
+            -0.0009806477464735508,
+            0.01258004643023014,
+            -0.00942891277372837,
+            -0.0013878053287044168,
+            -0.01606588624417782,
+            -0.014808698557317257,
+            0.006416558753699064,
+            0.0033633869607001543,
+            -0.01222901325672865,
+            -0.023739634081721306,
+            0.014759717509150505,
+            0.0007418635650537908,
+            -0.016980206593871117,
+            -0.017502672970294952,
+            0.00018457300029695034,
+            -0.004804255440831184,
+            0.02204161323606968,
+            -0.016678154468536377,
+            0.003373591462150216,
+            0.0007219648687168956,
+            0.009232986718416214,
+            -0.02889900468289852,
+            0.00716352416202426,
+            0.0006107363733462989,
+            0.0035164537839591503,
+            -0.006518602836877108,
+            -0.021013004705309868,
+            0.011371839791536331,
+            -0.0020674217958003283,
+            -0.014751553535461426,
+            -0.020751770585775375,
+            -0.0020796670578420162,
+            -0.007420676294714212,
+            -0.02724996581673622,
+            0.017910851165652275,
+            0.0009985055075958371,
+            -0.01738838478922844,
+            0.008077843114733696,
+            0.006596156861633062,
+            0.009355440735816956,
+            0.02455599047243595,
+            0.010147306136786938,
+            -0.018776189535856247,
+            -0.0008087027235887945,
+            0.006979844067245722,
+            0.009306458756327629,
+            0.0036450298503041267,
+            0.04222193360328674,
+            -0.008130906149744987,
+            -0.006579829845577478,
+            0.0066614653915166855,
+            -0.008449285291135311,
+            -0.013747435994446278,
+            0.026727497577667236,
+            0.005796127952635288,
+            0.015984252095222473,
+            0.016123032197356224,
+            -0.00945340283215046,
+            -0.005008344538509846,
+            -0.0018786393338814378,
+            0.02065380848944187,
+            0.0073798587545752525,
+            -0.005502239800989628,
+            -0.009714636951684952,
+            0.005881845485419035,
+            -0.00898808054625988,
+            0.0021653843577951193,
+            -0.009526874870061874,
+            -0.005824700463563204,
+            -0.008069679141044617,
+            0.016400592401623726,
+            -0.0003523086488712579,
+            0.027315273880958557,
+            0.020751770585775375,
+            -0.008563575334846973,
+            -0.0005683879135176539,
+            0.017208784818649292,
+            -0.005820618476718664,
+            0.015608727931976318,
+            -0.011192241683602333,
+            -0.007706400938332081,
+            0.006408394780009985,
+            0.024017194285988808,
+            0.012237177230417728,
+            -0.0056042843498289585,
+            0.0002885308349505067,
+            -0.0030348035506904125,
+            -0.010612628422677517,
+            0.013388238847255707,
+            0.015323002822697163,
+            -0.01947825588285923,
+            -0.004722619894891977,
+            -0.0016592437168583274,
+            -0.003300119424238801,
+            0.00945340283215046,
+            0.02305389568209648,
+            0.000171817431692034,
+            0.009184005670249462,
+            0.0012765767751261592,
+            0.03862996771931648,
+            -0.0053144777193665504,
+            0.03102152980864048,
+            -0.0010423846542835236,
+            -0.0059430720284581184,
+            -0.009094206616282463,
+            -0.020441556349396706,
+            -0.011486129835247993,
+            -0.00041481092921458185,
+            -0.005040998570621014,
+            -0.0004956811899319291,
+            -0.020327266305685043,
+            0.0026960158720612526,
+            0.014637264423072338,
+            0.007661501411348581,
+            0.017927179113030434,
+            -0.002889900468289852,
+            0.017437364906072617,
+            0.019690508022904396,
+            -0.01400050614029169,
+            -0.029862305149435997,
+            -0.0006515541463159025,
+            0.01498013362288475,
+            -0.01661284640431404,
+            0.019445601850748062,
+            0.004085862077772617,
+            0.0029531680047512054,
+            0.004779764451086521,
+            -0.006049198564141989,
+            0.018482301384210587,
+            0.02039257436990738,
+            0.01453930139541626,
+            -0.013167822733521461,
+            0.014057651162147522,
+            0.009771781973540783,
+            0.006600238848477602,
+            -0.01646590232849121,
+            -0.013535182923078537,
+            0.002555194543674588,
+            0.0009245232213288546,
+            0.024392718449234962,
+            -0.020588500425219536,
+            0.0016602640971541405,
+            0.010588138364255428,
+            0.01769859902560711,
+            -0.004302196204662323,
+            0.008906444534659386,
+            0.0008740111952647567,
+            0.007134951651096344,
+            -0.0035919665824621916,
+            -0.006457376293838024,
+            -0.006673710886389017,
+            -0.010392212308943272,
+            -0.0020959940738976,
+            -0.02070278860628605,
+            0.002198038622736931,
+            -0.002830714685842395,
+            0.018237393349409103,
+            -0.01018812321126461,
+            0.038140155375003815,
+            0.03399306535720825,
+            -0.0007025764207355678,
+            0.011102442629635334,
+            -0.01709449477493763,
+            -0.003945040516555309,
+            -0.021959977224469185,
+            0.01844964735209942,
+            0.016392430290579796,
+            0.012269831262528896,
+            0.005302232690155506,
+            0.017796561121940613,
+            -0.0006301248213276267,
+            0.014490320347249508,
+            0.0029654132667928934,
+            0.007971717044711113,
+            -0.002265388146042824,
+            0.003938918001949787,
+            -0.031103165820240974,
+            -0.017029186710715294,
+            -0.01138000376522541,
+            0.004796091932803392,
+            -0.006375740747898817,
+            -0.0035205355379730463,
+            0.0084737753495574,
+            -0.005902254022657871,
+            0.0004270562494639307,
+            -0.025992777198553085,
+            -0.0157148540019989,
+            -0.010318740271031857,
+            -0.004040962550789118,
+            -0.01169838197529316,
+            -0.0027694879099726677,
+            -0.0022082431241869926,
+            0.0029552089981734753,
+            0.020947696641087532,
+            -0.012808626517653465,
+            -0.016294466331601143,
+            0.004379750229418278,
+            -0.002263347152620554,
+            -0.014016833156347275,
+            -0.016898570582270622,
+            0.006665547378361225,
+            0.012098397128283978,
+            -0.0017470018938183784,
+            -0.013608654960989952,
+            0.011739199981093407,
+            -0.006628811359405518,
+            0.04663025587797165,
+            -0.013584164902567863,
+            -0.0078002819791436195,
+            0.003720542648807168,
+            0.004698128905147314,
+            -0.009396257810294628,
+            0.007330877240747213,
+            0.013020878657698631,
+            0.01830270327627659,
+            0.0095758568495512,
+            -0.008465612307190895,
+            -0.0024388637393712997,
+            0.007796200457960367,
+            -0.016147522255778313,
+            0.006404313258826733,
+            0.0008525818702764809,
+            -0.014400520361959934,
+            0.006110425107181072,
+            0.028980640694499016,
+            0.013151495717465878,
+            0.01288209855556488,
+            -0.022351829335093498,
+            -0.004094025585800409,
+            -0.009355440735816956,
+            -0.004943036008626223,
+            -0.010090161114931107,
+            0.027070365846157074,
+            -0.018155759200453758,
+            0.02651524543762207,
+            0.00898808054625988,
+            0.0017684312770143151,
+            -0.01176369097083807,
+            0.006343086715787649,
+            0.02832755446434021,
+            0.0028592869639396667,
+            -0.020000724121928215,
+            -0.013592327944934368,
+            0.006314514204859734,
+            0.01707816869020462,
+            -0.002647034591063857,
+            -0.005428767763078213,
+            0.019657853990793228,
+            0.01784554310142994,
+            -0.004334850702434778,
+            -0.00936360377818346,
+            0.0008337036124430597,
+            0.00928196869790554,
+            0.016963878646492958,
+            0.015020951628684998,
+            0.009086042642593384,
+            -0.002647034591063857,
+            0.008037025108933449,
+            -0.008800318464636803,
+            -0.016449574381113052,
+            0.007220669183880091,
+            -0.008432958275079727,
+            -0.016963878646492958,
+            0.01872720755636692,
+            0.015257694758474827,
+            0.0181394312530756,
+            0.0024592725094407797,
+            -0.016735298559069633,
+            0.022302847355604172,
+            0.005755309946835041,
+            0.002998067531734705,
+            0.02160078100860119,
+            -0.022221211344003677,
+            0.013012715615332127,
+            0.02037624642252922,
+            0.027609162032604218,
+            -0.01737205684185028,
+            -0.010906516574323177,
+            -0.004857318475842476,
+            -0.006853308994323015,
+            -0.00176945177372545,
+            -0.008253359235823154,
+            -0.012620864436030388,
+            0.0028735732194036245,
+            -0.030466407537460327,
+            0.018106777220964432,
+            -0.005102225113660097,
+            0.010669773444533348,
+            -0.019266001880168915,
+            0.011363675817847252,
+            -0.0035838030744343996,
+            0.011135096661746502,
+            0.0019612954929471016,
+            -0.030041903257369995,
+            -0.008571738377213478,
+            0.015388311818242073,
+            -0.0030368445441126823,
+            -0.013641309924423695,
+            0.02006603218615055,
+            0.00018942011229228228,
+            -0.022270193323493004,
+            -0.004314441699534655,
+            -0.006934944540262222,
+            0.002193956868723035,
+            -0.0029613315127789974,
+            -0.0052981507033109665,
+            -0.030188847333192825,
+            0.020033378154039383,
+            0.009559528902173042,
+            -0.03072764165699482,
+            -0.003255219664424658,
+            -0.014996460638940334,
+            -0.0025184585247188807,
+            0.008881953544914722,
+            -0.004000144544988871,
+            -0.015551582910120487,
+            0.01784554310142994,
+            0.022711025550961494,
+            -0.0023164101876318455,
+            -0.016057724133133888,
+            -0.0090697156265378,
+            -0.027494871988892555,
+            0.030825603753328323,
+            -0.0066696288995444775,
+            0.0022878379095345736,
+            -0.005322641227394342,
+            0.009388094767928123,
+            0.004228724166750908,
+            0.0157148540019989,
+            0.006081852596253157,
+            -0.002647034591063857,
+            0.026596879586577415,
+            -0.01964152604341507,
+            0.009567692875862122,
+            -1.588864870427642e-05,
+            -0.003308282932266593,
+            0.01890680566430092,
+            0.0007939062779769301,
+            0.021649762988090515,
+            -0.008571738377213478,
+            -0.0001475818717153743,
+            -0.0034388997592031956,
+            0.010898353531956673,
+            -0.0028450009413063526,
+            -0.003465431509539485,
+            -0.001531688030809164,
+            0.011118769645690918,
+            0.01013097818940878,
+            0.003804219188168645,
+            -0.01737205684185028,
+            0.003142970846965909,
+            -0.008547247387468815,
+            -0.007028825581073761,
+            0.01302904263138771,
+            -0.011192241683602333,
+            -0.0018990482203662395,
+            -0.03160930797457695,
+            0.016286302357912064,
+            0.0533243790268898,
+            0.008547247387468815,
+            -0.014433175325393677,
+            -0.0018806802108883858,
+            -0.019755816087126732,
+            0.01535565685480833,
+            0.011004479601979256,
+            -0.016816934570670128,
+            -0.011984107084572315,
+            -0.012310649268329144,
+            0.007649256382137537,
+            0.021306892856955528,
+            -0.006522684823721647,
+            -0.008579902350902557,
+            0.025372346863150597,
+            -0.011788181029260159,
+            -0.010882026515901089,
+            -0.01661284640431404,
+            -0.01635977439582348,
+            0.00241641397587955,
+            0.019902760162949562,
+            -0.0169475506991148,
+            0.021176276728510857,
+            0.00936360377818346,
+            0.0010133018950000405,
+            0.003624620847404003,
+            0.008588065393269062,
+            0.0024878450203686953,
+            0.017878197133541107,
+            0.003883813973516226,
+            -0.01439235731959343,
+            0.0021408938337117434,
+            -0.0030286810360848904,
+            0.004624656867235899,
+            3.4320437407586724e-06,
+            -0.002749078907072544,
+            -0.0065635028295218945,
+            0.003710338147357106,
+            -0.021306892856955528,
+            0.021502818912267685,
+            -0.022678371518850327,
+            -0.004245051648467779,
+            0.005555302836000919,
+            -0.005118552595376968,
+            0.013469874858856201,
+            0.018792515620589256,
+            0.01423724927008152,
+            0.008702355436980724,
+            0.01462093647569418,
+            -0.011331021785736084,
+            0.02323349379003048,
+            -0.013282112777233124,
+            0.0005816536722704768,
+            0.012939243577420712,
+            0.01755165494978428,
+            0.018792515620589256,
+            -0.006245123688131571,
+            -0.010384049266576767,
+            -0.0002946535241790116,
+            0.0129637336358428,
+            -0.022286519408226013,
+            -0.012269831262528896,
+            -0.01386172603815794,
+            0.011486129835247993,
+            -0.004743028432130814,
+            0.025453981012105942,
+            -0.00633492274209857,
+            -0.014653591439127922,
+            -0.015918942168354988,
+            0.014196431264281273,
+            -0.048752784729003906,
+            0.0030348035506904125,
+            -0.0024919267743825912,
+            0.03043375350534916,
+            -0.027021385729312897,
+            0.01738838478922844,
+            -0.0028490826953202486,
+            -0.009355440735816956,
+            0.015429128892719746,
+            -0.015992414206266403,
+            -0.0046613928861916065,
+            -0.0016806729836389422,
+            -0.007539048325270414,
+            -0.00112248957157135,
+            0.017616963014006615,
+            -0.015845470130443573,
+            0.01282495353370905,
+            -0.00989423505961895,
+            0.0027021386194974184,
+            -0.004526694305241108,
+            -0.007751300930976868,
+            0.0196088720113039,
+            0.034548189491033554,
+            -0.007053316105157137,
+            0.0014990337658673525,
+            -0.022547753527760506,
+            0.013192313723266125,
+            -0.006730855442583561,
+            0.02351105399429798,
+            -0.005432849284261465,
+            -0.016898570582270622,
+            -0.013976015150547028,
+            -0.009739127941429615,
+            0.00839214026927948,
+            -0.022433463484048843,
+            -0.011110605672001839,
+            -0.00922482367604971,
+            -0.002573562553152442,
+            -0.018694553524255753,
+            0.004865481983870268,
+            -0.002426618244498968,
+            -0.012506574392318726,
+            -0.008816645480692387,
+            0.01048201136291027,
+            -0.009347276762127876,
+            0.003077662317082286,
+            -0.0011929002357646823,
+            -0.011453475803136826,
+            -0.02369065210223198,
+            0.0158944521099329,
+            0.0001974561164388433,
+            -0.010359558276832104,
+            0.008408467285335064,
+            -0.0018276170594617724,
+            0.001211268245242536,
+            0.016963878646492958,
+            -0.0020664012990891933,
+            0.003467472270131111,
+            0.023478399962186813,
+            0.004057289566844702,
+            -0.010971825569868088,
+            -0.00025906547671183944,
+            0.013535182923078537,
+            -0.005118552595376968,
+            -0.010008525103330612,
+            1.6646010408294387e-05,
+            -0.02984597720205784,
+            -0.008351322263479233,
+            0.006094098091125488,
+            0.018629245460033417,
+            -0.010620792396366596,
+            -0.015853634104132652,
+            0.026319319382309914,
+            -0.006359413731843233,
+            0.019070077687501907,
+            0.002889900468289852,
+            0.0025694805663079023,
+            0.01724144071340561,
+            0.013461710885167122,
+            0.00786150898784399,
+            -0.013927034102380276,
+            -0.02429475635290146,
+            0.0016398552106693387,
+            -0.018090449273586273,
+            0.010579974390566349,
+            -0.011028970591723919,
+            0.0032756286673247814,
+            -0.02398454025387764,
+            -0.012237177230417728,
+            0.004934872500598431,
+            -0.020343592390418053,
+            -0.00040588201954960823,
+            -0.0056940834037959576,
+            0.022237539291381836,
+            -0.005310396198183298,
+            0.008669701404869556,
+            0.0018255761824548244,
+            -0.012237177230417728,
+            0.0009877908742055297,
+            0.009469729848206043,
+            -0.029192892834544182,
+            -0.0007127808639779687,
+            -0.003138889092952013,
+            0.0011418780777603388,
+            0.005351213738322258,
+            0.009755454957485199,
+            0.007322713732719421,
+            0.0027062203735113144,
+            0.007253323681652546,
+            0.010449357330799103,
+            -0.00022041612828616053,
+            0.0197884701192379,
+            0.018172085285186768,
+            -0.007351286243647337,
+            0.008710519410669804,
+            0.024817224591970444,
+            -0.013159659691154957,
+            -0.008718682453036308,
+            0.023266147822141647,
+            -0.006461458280682564,
+            -0.0011122850701212883,
+            -0.01483318954706192,
+            0.00815947912633419,
+            0.0017367975087836385,
+            0.028131630271673203,
+            0.019657853990793228,
+            0.005820618476718664,
+            0.000639819074422121,
+            0.015788326039910316,
+            -0.013020878657698631,
+            0.0035286990460008383,
+            0.015461783856153488,
+            -0.003940958995372057,
+            -0.019821124151349068,
+            0.013878053054213524,
+            0.009502384811639786,
+            -0.003228688146919012,
+            0.01551892887800932,
+            0.011412657797336578,
+            -0.030841931700706482,
+            -0.0011898389784619212,
+            0.010498338378965855,
+            -0.0021449755877256393,
+            -0.008751336485147476,
+            -0.00014273475972004235,
+            -0.012384121306240559,
+            0.021388528868556023,
+            -0.004408322740346193,
+            0.012277995236217976,
+            -0.00815947912633419,
+            -0.017127150669693947,
+            -0.011559601873159409,
+            0.007914572022855282,
+            -0.010547320358455181,
+            -0.0006714528426527977,
+            -0.004022594541311264,
+            -0.005779800936579704,
+            0.017747581005096436,
+            0.004706292413175106,
+            -0.0008964610169641674,
+            -0.0018398624379187822,
+            0.007983962073922157,
+            -0.006979844067245722,
+            -0.008726846426725388,
+            0.004816500470042229,
+            -0.012833117507398129,
+            0.003981776535511017,
+            0.005543057806789875,
+            0.01506176870316267,
+            -0.014857680536806583,
+            -0.021159948781132698,
+            0.007020662073045969,
+            0.02695607766509056,
+            -0.02516009286046028,
+            -0.005232842173427343,
+            0.006245123688131571,
+            0.002093953313305974,
+            0.006914535537362099,
+            -0.006934944540262222,
+            -0.007208424154669046,
+            -0.03657275065779686,
+            -0.0020745648071169853,
+            0.0009005427709780633,
+            0.008963589556515217,
+            -0.008049271069467068,
+            -0.0136494729667902,
+            -0.011135096661746502,
+            0.002410291228443384,
+            0.007134951651096344,
+            -0.02204161323606968,
+            0.0049593630246818066,
+            0.009608510881662369,
+            -0.019559891894459724,
+            0.025127438828349113,
+            -0.010890189558267593,
+            -6.964537897147238e-05,
+            -0.00995954405516386,
+            0.021633435040712357,
+            -0.013486201874911785,
+            -0.018368011340498924,
+            -0.0006898208521306515,
+            0.008355404250323772,
+            0.011926962062716484,
+            -0.0017663904000073671,
+            -0.011371839791536331,
+            0.0003752686607185751,
+            0.01646590232849121,
+            0.019135385751724243,
+            0.002744997153058648,
+            -0.0024837632663547993,
+            -0.01019628718495369,
+            -0.009592183865606785,
+            -0.0073961857706308365,
+            -0.0007449248805642128,
+            0.002091912319883704,
+            0.02035992033779621,
+            0.013469874858856201,
+            0.0024837632663547993,
+            -0.015135240741074085,
+            -0.005257333163172007,
+            0.008939098566770554,
+            0.009412584826350212,
+            -0.011690218932926655,
+            -0.00898808054625988,
+            0.002659279853105545,
+            0.02249877341091633,
+            0.017633290961384773,
+            -0.0014000505907461047,
+            0.0032878739293664694,
+            -0.0016592437168583274,
+            0.012506574392318726,
+            -0.007555375341325998,
+            -0.010269759222865105,
+            -0.006943108048290014,
+            -0.014645427465438843,
+            -0.010800390504300594,
+            -0.0023062059190124273,
+            -0.010179960168898106,
+            0.022629389539361,
+            0.007988044060766697,
+            0.009779945947229862,
+            0.008265605196356773,
+            0.010269759222865105,
+            0.002087830565869808,
+            0.025715215131640434,
+            -0.004424649756401777,
+            -0.011984107084572315,
+            0.007236996199935675,
+            -0.022368155419826508,
+            -0.021535472944378853,
+            0.008257441222667694,
+            0.006367577239871025,
+            0.03595232218503952,
+            -0.026890767738223076,
+            0.0052981507033109665,
+            -0.009779945947229862,
+            -0.010400376282632351,
+            0.007996207103133202,
+            -0.0015439334092661738,
+            -0.004245051648467779,
+            0.022368155419826508,
+            0.00726556871086359,
+            0.0035797213204205036,
+            0.0032817514147609472,
+            0.019837452098727226,
+            -0.005685919895768166,
+            0.01573934406042099,
+            0.0259274672716856,
+            0.024621298536658287,
+            0.009388094767928123,
+            -0.009469729848206043,
+            0.02785406820476055,
+            0.0002895512734539807,
+            -0.0034327772445976734,
+            0.007220669183880091,
+            -0.00305113079957664,
+            -0.0009556467994116247,
+            -0.010033016093075275,
+            0.021764053031802177,
+            -0.001455154619179666,
+            0.004359341226518154,
+            0.02081707864999771,
+            -7.774515688652173e-05,
+            0.006220633164048195,
+            0.01057181041687727,
+            0.002181711606681347,
+            0.003396041225641966,
+            0.010775899514555931,
+            -0.039315707981586456,
+            -0.0006571666453965008,
+            -0.025600925087928772,
+            -0.0018286375561729074,
+            0.0007357408758252859,
+            -0.01266984548419714,
+            0.013600491918623447,
+            -0.002015379024669528,
+            -0.010751409456133842,
+            -0.00029337796149775386,
+            0.021649762988090515,
+            0.005416522268205881,
+            0.004071575589478016,
+            -0.004469549283385277,
+            0.018792515620589256,
+            -0.006955353543162346,
+            -0.017861871048808098,
+            -0.012759645469486713,
+            -0.012367794290184975,
+            -0.01918436773121357,
+            -0.01214737817645073,
+            -0.0024551907554268837,
+            -0.002667443361133337,
+            0.004738946910947561,
+            0.002481722505763173,
+            -0.003228688146919012,
+            -0.02070278860628605,
+            0.015037278644740582,
+            -0.01430255826562643,
+            0.012873934581875801,
+            -0.011053460650146008,
+            0.005963481031358242,
+            -0.004122598096728325,
+            0.012726990506052971,
+            -0.021404854953289032,
+            0.02070278860628605,
+            -0.01536382082849741,
+            0.0038532004691660404,
+            0.019592545926570892,
+            0.006551257334649563,
+            -0.008118661120533943,
+            -0.013404566794633865,
+            -0.022074267268180847,
+            0.021633435040712357,
+            -0.011633073911070824,
+            -0.018237393349409103,
+            -0.004608329851180315,
+            0.009649328887462616,
+            -0.017470018938183784,
+            -0.007367613259702921,
+            0.01707816869020462,
+            -0.0018949664663523436,
+            0.012980061583220959,
+            0.023004913702607155,
+            -0.0052981507033109665,
+            0.0036572751123458147,
+            -0.009167678654193878,
+            0.0010286086471751332,
+            -0.02470293454825878,
+            0.013527019880712032,
+            0.008971752598881721,
+            0.006269614677876234,
+            -0.013935198076069355,
+            -0.013306603766977787,
+            0.0036531933583319187,
+            0.004498121794313192,
+            0.011437147855758667,
+            0.009420748800039291,
+            0.004220560658723116,
+            -0.013918871060013771,
+            -0.027788760140538216,
+            -0.027070365846157074,
+            0.03595232218503952,
+            -0.0005002731923013926,
+            -0.020947696641087532,
+            0.03399306535720825,
+            0.016114868223667145,
+            0.0022980421781539917,
+            -0.012449429370462894,
+            -0.004314441699534655,
+            0.024621298536658287,
+            0.016400592401623726,
+            0.010465684346854687,
+            -0.0004742518358398229,
+            0.005571629852056503,
+            0.0016469983384013176,
+            0.03012353740632534,
+            0.015323002822697163,
+            0.0009459525463171303,
+            -0.038597315549850464,
+            0.011943289078772068,
+            -0.007547211833298206,
+            0.0013408648082986474,
+            0.006702283397316933,
+            0.017780235037207603,
+            0.011935125105082989,
+            0.015567909926176071,
+            0.014865843579173088,
+            -0.003065417055040598,
+            0.0029511270113289356,
+            0.0006163488142192364,
+            -0.009624837897717953,
+            -0.001767410896718502,
+            -0.008881953544914722,
+            -0.005714492406696081,
+            -0.002920513739809394,
+            0.005592038854956627,
+            0.007657419890165329,
+            -0.003983817528933287,
+            0.028049994260072708,
+            -0.0006433906382881105,
+            -0.017600636929273605,
+            -0.009061552584171295,
+            -0.03611559048295021,
+            0.00434301421046257,
+            -0.030939893797039986,
+            -0.011257549747824669,
+            0.0033674687147140503,
+            -0.020457882434129715,
+            -0.024768242612481117,
+            0.0010306495241820812,
+            0.014335212297737598,
+            0.007392104249447584,
+            0.0033327736891806126,
+            -0.01004934310913086,
+            0.003816464450210333,
+            -0.001855169073678553,
+            -0.006461458280682564,
+            0.028833696618676186,
+            0.019559891894459724,
+            0.020457882434129715,
+            -0.01190247107297182,
+            0.019298657774925232,
+            0.005845109466463327,
+            0.02035992033779621,
+            0.010465684346854687,
+            -0.0048695639707148075,
+            0.0074859848245978355,
+            0.009567692875862122,
+            -0.006155324634164572,
+            -0.021519144997000694,
+            0.013192313723266125,
+            -0.0010837126756086946,
+            -0.0052083516493439674,
+            -0.016269976273179054,
+            -0.003534821793437004,
+            -0.015812816098332405,
+            0.010547320358455181,
+            -0.017159804701805115,
+            0.012498411349952221,
+            -0.000802069844212383,
+            0.00168475485406816,
+            -0.015469946898519993,
+            -0.013951525092124939,
+            -0.0012439225101843476,
+            -0.01244126632809639,
+            -0.008971752598881721,
+            0.006163488142192364,
+            -0.007539048325270414,
+            -0.008318668231368065,
+            -0.00535937724635005,
+            -0.011437147855758667,
+            -0.010882026515901089,
+            0.004367504734545946,
+            -0.0010653446661308408,
+            0.020931368693709373,
+            0.00011671340325847268,
+            -0.005824700463563204,
+            0.01190247107297182,
+            0.02787039615213871,
+            0.0016704685986042023,
+            0.004583839327096939,
+            -0.014261740259826183,
+            -0.011118769645690918,
+            -0.02279266156256199,
+            0.016572028398513794,
+            0.0049675265327095985,
+            -0.0344502255320549,
+            0.01438419334590435,
+            0.00824927818030119,
+            0.0015102586476132274,
+            -0.006698201410472393,
+            -0.005334886722266674,
+            -0.005608365871012211,
+            0.006306350696831942,
+            -0.0022837561555206776,
+            0.0005107327597215772,
+            0.0035919665824621916,
+            0.034678805619478226,
+            -0.0040674940682947636,
+            -0.007228832691907883,
+            0.00981259997934103,
+            0.0005130287609063089,
+            0.020196648314595222,
+            -0.005518566817045212,
+            0.0037572786677628756,
+            -0.019282329827547073,
+            -0.016563864424824715,
+            0.0018429238116368651,
+            -0.01722511276602745,
+            0.023282473906874657,
+            -0.005392031744122505,
+            -0.01347803883254528,
+            -0.006906372029334307,
+            -0.020115012302994728,
+            0.012637191452085972,
+            0.02310287579894066,
+            0.007016580086201429,
+            0.01033506728708744,
+            0.012131051160395145,
+            -0.02444170042872429,
+            -0.020457882434129715,
+            0.012171869166195393,
+            0.0013571919407695532,
+            -0.008449285291135311,
+            -0.005404277238994837,
+            -0.001783737912774086,
+            0.007783954963088036,
+            -0.0032633834052830935,
+            0.007881917990744114,
+            0.06491663306951523,
+            -0.01093100756406784,
+            -0.002987863030284643,
+            -0.02966637909412384,
+            -0.004396077245473862,
+            -0.006628811359405518,
+            0.0056287748739123344,
+            0.009151351638138294,
+            -0.023135529831051826,
+            0.009232986718416214,
+            -0.030025575309991837,
+            -0.010628955438733101,
+            0.00824111420661211,
+            0.0028490826953202486,
+            -0.01220452319830656,
+            -0.0009592183632776141,
+            -0.013665799982845783,
+            0.0006107363733462989,
+            -0.009412584826350212,
+            -0.00576347392052412,
+            0.012718827463686466,
+            -0.02321716584265232,
+            0.006298186723142862,
+            0.007592111360281706,
+            -0.006514521315693855,
+            -0.013722945004701614,
+            -0.017861871048808098,
+            -0.011967780068516731,
+            -0.013314766809344292,
+            0.002996026771143079,
+            0.01190247107297182,
+            -0.02607441134750843,
+            0.012922916561365128,
+            -0.0037389106582850218,
+            0.00945340283215046,
+            0.004404240753501654,
+            0.013469874858856201,
+            0.018074123188853264,
+            0.009355440735816956,
+            0.0021429345943033695,
+            -0.008024780079722404,
+            0.012702500447630882,
+            -0.01993541419506073,
+            -0.01635161228477955,
+            0.013837235048413277,
+            0.012016761116683483,
+            0.003061335301026702,
+            0.009828926995396614,
+            0.0020959940738976,
+            0.011886144056916237,
+            0.010775899514555931,
+            -0.019739490002393723,
+            -0.0006648199632763863,
+            0.0007326795603148639,
+            0.03280118480324745,
+            -0.0005938990507274866,
+            0.008857463486492634,
+            0.004420568235218525,
+            0.013159659691154957,
+            -0.012784135527908802,
+            -0.01769859902560711,
+            0.0008087027235887945,
+            -0.01057181041687727,
+            0.021257910877466202,
+            -0.008192133158445358,
+            0.006216551177203655,
+            -0.018612917512655258,
+            -0.027086693793535233,
+            0.006392067763954401,
+            -0.016261812299489975,
+            -0.018221067264676094,
+            0.01317598670721054,
+            0.0077268099412322044,
+            0.0017572063952684402,
+            -0.005098143592476845,
+            -0.020457882434129715,
+            -0.009837090037763119,
+            0.028441844508051872,
+            0.011102442629635334,
+            -0.004120557103306055,
+            0.009584019891917706,
+            -0.009094206616282463,
+            -0.012490247376263142,
+            -0.0017541450215503573,
+            0.0036715613678097725,
+            0.008261523209512234,
+            0.005645101889967918,
+            -0.01386172603815794,
+            -0.00029873527819290757,
+            -0.012400448322296143,
+            -0.021225256845355034,
+            -0.008906444534659386,
+            0.016751626506447792,
+            0.007820690982043743,
+            -0.0038960592355579138,
+            0.015812816098332405,
+            -0.004404240753501654,
+            0.002189875114709139,
+            0.014294394291937351,
+            0.005334886722266674,
+            -0.004285869188606739,
+            -0.0095758568495512,
+            -0.004612411838024855,
+            0.004043003544211388,
+            -0.0025960123166441917,
+            0.006457376293838024,
+            0.017192458733916283,
+            0.0072900597006082535,
+            -0.013755599036812782,
+            0.013135168701410294,
+            0.0007791098323650658,
+            0.0049675265327095985,
+            -0.03699725493788719,
+            -0.003247056156396866,
+            0.015796490013599396,
+            -0.013020878657698631,
+            0.02501314878463745,
+            0.005400195252150297,
+            0.022743679583072662,
+            0.015233203768730164,
+            0.008669701404869556,
+            -0.001542912912555039,
+            0.00464098434895277,
+            0.02759283408522606,
+            -0.003151134354993701,
+            -0.008637047372758389,
+            -0.0020602785516530275,
+            0.009241150692105293,
+            0.0002063850115519017,
+            0.003000108525156975,
+            0.008281932212412357,
+            -0.03562577813863754,
+            0.011706545948982239,
+            -0.019576217979192734,
+            0.0024694770108908415,
+            0.003459308762103319,
+            -0.007183933164924383,
+            -0.014865843579173088,
+            -0.005857354495674372,
+            0.022841641679406166,
+            -0.003224606392905116,
+            0.021078312769532204,
+            0.002885818714275956,
+            0.016106704249978065,
+            -0.011575928889214993,
+            -0.009208496659994125,
+            -0.0004130251472815871,
+            -0.005351213738322258,
+            -0.007539048325270414,
+            0.010596301406621933,
+            0.0034409407526254654,
+            0.005534893833100796,
+            0.002265388146042824,
+            0.0036654386203736067,
+            -0.0036878883838653564,
+            0.02773977816104889,
+            0.003563394071534276,
+            0.007996207103133202,
+            -0.011567764915525913,
+            0.014082142151892185,
+            -0.017029186710715294,
+            0.0070778070949018,
+            0.005575711838901043,
+            -0.026156047359108925,
+            0.016849588602781296,
+            0.004775682929903269,
+            0.006628811359405518,
+            0.013812744058668613,
+            0.01618834026157856,
+            -0.010408539324998856,
+            -0.014816862531006336,
+            0.024049850180745125,
+            -0.0032041973900049925,
+            -0.015225040726363659,
+            -0.006775755435228348,
+            4.438935866346583e-05,
+            0.006559420842677355,
+            0.00373074715025723,
+            0.0013020879123359919,
+            -0.0006204306264407933,
+            -0.010743245482444763,
+            0.004645065870136023,
+            -0.01934763789176941,
+            -0.0007163524278439581,
+            -0.016449574381113052,
+            -0.015633217990398407,
+            0.018678225576877594,
+            -0.01830270327627659,
+            0.009633000940084457,
+            0.0012275953777134418,
+            0.028588788583874702,
+            -0.011706545948982239,
+            -0.012155541218817234,
+            0.0015102586476132274,
+            0.010849371552467346,
+            0.0067512644454836845,
+            -0.010033016093075275,
+            -0.007583947852253914,
+            -0.014180104248225689,
+            -0.010163633152842522,
+            0.015502601861953735,
+            -0.014498483389616013,
+            -0.014343376271426678,
+            0.008914608508348465
+          ],
+          "embedding": [
+            -0.006359657085039302,
+            -0.024594786241399234,
+            -0.02553538924848095,
+            0.005360188465088291,
+            0.012801870977589839,
+            0.021680505862506104,
+            -0.016668048966001273,
+            0.06860852474581301,
+            -0.007122272841157724,
+            0.03471775000705488,
+            -0.009004297848861675,
+            0.01059974144678391,
+            0.02565604599426148,
+            -0.022544127646041287,
+            0.010616674976540785,
+            0.013658194823359737,
+            0.016400329743363844,
+            0.02858546964917478,
+            0.0067612676434693686,
+            -0.04174737193606879,
+            0.03184155110488616,
+            -0.02223105127585514,
+            -0.05506239667974977,
+            -0.0015307551410555752,
+            -0.013169788751379287,
+            0.017681749722384007,
+            0.01682706317229353,
+            0.019652295823816928,
+            -0.05623980369271269,
+            0.030698582851533197,
+            0.02286327124296613,
+            0.016019032539689178,
+            0.005508113353872315,
+            8.58985091215149e-05,
+            -0.019206292853495906,
+            -0.018439290483495144,
+            0.03475023176950151,
+            0.01018612942141759,
+            -0.0252551158857426,
+            0.0334562130666827,
+            0.020462153537485785,
+            0.02260508249714211,
+            -0.03323524989392951,
+            -0.002094251009530707,
+            0.015177442347686773,
+            -0.00018698464350038317,
+            -0.0006788648344273403,
+            0.0007777616893581039,
+            0.013352398615059647,
+            -0.030751111528744813,
+            -0.0317733818445192,
+            -0.006729064081710058,
+            0.009413204899986514,
+            -0.005712894407943592,
+            -0.038961826880527066,
+            -0.002827692206530345,
+            0.017863300537973137,
+            0.07918455388563496,
+            0.048967158919816366,
+            0.018002566325350088,
+            0.040264962620950695,
+            -0.019715100359659787,
+            -0.02211350500879408,
+            0.004918148608475574,
+            0.01062585305866606,
+            -0.014478993521696027,
+            0.0034983131226364672,
+            0.000766228629514787,
+            0.009667268771502067,
+            -0.01558439890697323,
+            -0.04984718709870083,
+            0.029613955432821887,
+            -0.009686854114630624,
+            -0.015279319238124568,
+            0.007460629676325322,
+            -0.006235979350809034,
+            0.012514387473717085,
+            0.02800435263741497,
+            -0.008688925946672778,
+            -0.01236301543374469,
+            0.029755705020188972,
+            0.01158037219446542,
+            -0.005242392206024511,
+            0.009982679027728814,
+            0.052414361714483264,
+            -0.018788500381726227,
+            -0.020335831196027204,
+            -0.08583959943356621,
+            0.029309228216812346,
+            -0.0249285282010673,
+            -0.018929396555426307,
+            0.007710663193334574,
+            0.03096798035958051,
+            -0.017089977064591828,
+            -0.028015107605405817,
+            0.06740703271128606,
+            0.009398795412354399,
+            0.01118769022826835,
+            -0.03041417460855722,
+            0.014776002883150835,
+            0.004319194067673161,
+            0.022507636639041203,
+            0.0053562788646786845,
+            0.007548072258943124,
+            0.011123374308563128,
+            -0.006924362115585165,
+            -0.011118507147092024,
+            0.009563232643295304,
+            0.00657178473678069,
+            0.08313027447380966,
+            -0.004239370156003264,
+            -0.00930406163946907,
+            -0.039202672821898946,
+            0.009761029930556298,
+            -0.02827698465424993,
+            -0.037914533246900244,
+            -0.04496050911855815,
+            -0.017711307900236353,
+            0.01934435542287117,
+            0.004335173027873153,
+            0.03800509969714242,
+            0.031198127357716314,
+            0.031311207941680276,
+            -0.012184553894588813,
+            -0.058985863670856876,
+            0.041442017057707635,
+            -0.012456189292721711,
+            -0.0040966351720394135,
+            0.010849179519959421,
+            0.005243123049347636,
+            0.0011495799963862647,
+            0.003534183676934487,
+            -0.03139340512003631,
+            -0.023876261883905306,
+            0.008382957569752911,
+            -0.004778283217011976,
+            0.03703067589888386,
+            0.06888630688839406,
+            -0.03993532296478411,
+            -0.009472167678222372,
+            0.025246872942105057,
+            -0.01524567912700368,
+            0.01595917939878582,
+            -0.003736723164442467,
+            -0.030836797784967753,
+            0.0026246446163368533,
+            0.008894300082148884,
+            0.08342856279154155,
+            -0.0007476580521704578,
+            0.025149116238550588,
+            0.004972542633631468,
+            0.034495632546556576,
+            -0.009173329764686424,
+            0.014009381543772768,
+            0.004427261668151028,
+            -0.013426138607902904,
+            -0.022234568880321808,
+            0.01976282364689966,
+            -0.03592433789490043,
+            -0.010918278220831188,
+            0.013527180289641174,
+            -0.013023113052242677,
+            -0.008304770302495363,
+            0.003677063799167178,
+            0.0013981822663219421,
+            0.047001163242600175,
+            0.01828332714887986,
+            0.002483375662343689,
+            -0.06820498187523893,
+            0.05295312038995555,
+            0.025038753504342162,
+            0.025286870198436735,
+            -0.0029544808216316115,
+            0.029982752196635428,
+            0.0005600481523874767,
+            0.0128290258663539,
+            0.009023805227424919,
+            -0.0019219166367750596,
+            -0.04482843936478531,
+            0.04523479691409549,
+            -0.004678574027241186,
+            0.010486850804218904,
+            0.006736229865018014,
+            0.02134813656476443,
+            0.028083972673690036,
+            0.016857413440125445,
+            0.0189082447820146,
+            0.03843690613820022,
+            -0.011922027011229613,
+            0.016193433524453224,
+            0.03828207911467065,
+            -0.0013055128404331705,
+            0.043198163443700956,
+            0.024025562893260542,
+            -0.001257851946749882,
+            -0.011726713433348844,
+            -0.021907984099443396,
+            0.02752198420597673,
+            0.004431866336999851,
+            0.015798806714413912,
+            -0.012910782191382367,
+            -0.03180549180206364,
+            0.004550247372715741,
+            -0.022778272841118514,
+            0.031242218240017532,
+            -0.006106984949693483,
+            -0.006259993190497421,
+            0.02187449047033339,
+            0.009334146847221406,
+            -0.009381278960016145,
+            -0.020888853823931134,
+            -0.007502374908776156,
+            -0.010390018386728482,
+            -0.04316693435221676,
+            -0.006274513730720055,
+            0.019152443937064258,
+            -0.01668984824845791,
+            0.018573194036505578,
+            0.0035035262183249925,
+            -0.019779911381789077,
+            -0.0027495798165085993,
+            0.02425481236315913,
+            0.006857723545270935,
+            -0.014733535699087731,
+            -0.026326146471941727,
+            -0.019906660231056833,
+            -0.001033294559612807,
+            0.0159154063026895,
+            0.035649534442431415,
+            -0.026210592096240003,
+            0.014397127229517285,
+            0.005114311681443673,
+            -0.0033104236005207077,
+            0.00324988235167305,
+            0.03764004154553922,
+            -0.017822437760737605,
+            -0.008579701063708334,
+            -0.00032410984200965487,
+            0.018450351071676443,
+            0.017283379764846195,
+            0.006384411611332382,
+            -0.005115090309123598,
+            -0.030590047062583125,
+            -0.011776211359193503,
+            0.01183623914060493,
+            0.0016564847796208331,
+            -0.009530766634361575,
+            -0.004810598258154945,
+            0.03876242515526699,
+            0.027076785774401667,
+            0.005324763136165989,
+            -0.03416412480913939,
+            -0.007943016733586697,
+            0.002114307650456928,
+            -0.02961449347300628,
+            0.05142732770518828,
+            0.05566026433842563,
+            -0.0030729309664974704,
+            -0.00938186948393074,
+            -0.04547908981236149,
+            0.0651216359692231,
+            -0.01813722445944094,
+            -0.0482650853560993,
+            0.044045155892545554,
+            -0.014272653003441165,
+            0.013740652306746851,
+            0.04563732439184395,
+            0.02126513340521316,
+            -0.0054845632439384905,
+            -0.011635625101262898,
+            0.030992082069685376,
+            -0.010051935721068779,
+            -0.050967258971854425,
+            -0.009827942095033494,
+            0.013373878556654464,
+            -0.013017627321534534,
+            -0.009361001542607812,
+            -0.01925867665425919,
+            0.020329052915436022,
+            0.023020741670660188,
+            0.019387043947695172,
+            0.005628632318152202,
+            0.018451555161952665,
+            0.016226845400848455,
+            -0.01384033112836216,
+            0.004762589697426251,
+            -0.012718476561725138,
+            -0.024222765127494463,
+            -0.037487144314010595,
+            -0.019971524166659278,
+            -0.015034588610516958,
+            0.009483936496215338,
+            -0.014301924283553591,
+            -0.0024139665139412125,
+            0.013361045353529458,
+            -0.0005870798721107379,
+            -0.011192319340879789,
+            0.013927695434391159,
+            0.021337539669342293,
+            -0.027300952495815437,
+            -0.04765670695199488,
+            -0.00561706224251562,
+            0.0082776639906139,
+            -0.0030984973749633724,
+            -0.026269016057688548,
+            0.007040524859505573,
+            0.0017859008072463442,
+            0.020091542517503808,
+            -0.018716761130867725,
+            0.0035283051960672963,
+            0.008519383548064053,
+            -0.015247417050336866,
+            -0.016917626868166634,
+            -0.018580550690103854,
+            -0.024697180580294884,
+            0.034202028316737125,
+            0.0204395003463227,
+            0.030392895534267717,
+            -0.04527669147753553,
+            -0.018359428092075348,
+            0.05494107499558624,
+            0.015234754907198422,
+            -0.05066008106884831,
+            0.031139824660661085,
+            0.005016615710206006,
+            0.0019000453624929544,
+            -0.006094364589229031,
+            0.0637324238017742,
+            -0.02711381489292883,
+            -0.02187418906973867,
+            0.004601744864532875,
+            -0.04141121277496072,
+            -0.026689585608019588,
+            -0.005568100750837504,
+            0.01291019036002459,
+            -0.031853041426355644,
+            -0.02815800259372657,
+            -0.007964861641790374,
+            -0.010933515363060793,
+            0.013726639077745378,
+            -0.023956800669647334,
+            -0.015672058590455337,
+            -0.023642025374517057,
+            -0.01288435863544158,
+            -0.0042516881552192385,
+            -0.011162214160460695,
+            -0.02412733256710621,
+            0.010088528028211243,
+            0.03310175458095482,
+            0.023064360903748363,
+            0.001996159684818467,
+            0.009478229572933101,
+            0.014478772877286076,
+            0.00371783700962415,
+            0.007864103610320116,
+            -0.048511515124609,
+            -0.031398754206534776,
+            0.020227872561659397,
+            0.012159811587094092,
+            0.0172163448580163,
+            0.009578952796095119,
+            -0.04154545961112031,
+            -0.05095080393682759,
+            0.02968486144855133,
+            -0.03932395023720674,
+            -0.02063219739061335,
+            0.007512423249865194,
+            -0.011911709468952548,
+            -0.00810951734088105,
+            -0.0008079327452253181,
+            -0.01600701450676789,
+            0.023984909975645632,
+            -0.02418072541081482,
+            0.0033984265728439398,
+            -0.0104343782811971,
+            0.01588977364765743,
+            0.05109529246167992,
+            -0.027800269136366997,
+            -0.027889034481533438,
+            0.03706813210665186,
+            -0.02058120652429308,
+            0.030801396468451574,
+            0.03246127847621613,
+            0.008355517405358826,
+            -0.008605467586247932,
+            0.017693344563968573,
+            0.014499487024471321,
+            0.012431277080478217,
+            -0.040103870071440256,
+            0.017328848567062113,
+            0.02723900253727976,
+            -0.054909074068976875,
+            -0.008547596270650966,
+            -0.00777964838164648,
+            0.027134666572242665,
+            0.02765036594970255,
+            -0.002207173084941375,
+            -0.016838740117653725,
+            -0.028289233739722936,
+            -0.019433376533829132,
+            -0.011053380458946764,
+            0.03247641283550457,
+            0.0007514989009289282,
+            0.007919574219000797,
+            0.001316042436488591,
+            -0.0017934342041762042,
+            0.01924145734197619,
+            0.015260199553572459,
+            0.048259602293134723,
+            -0.03383050622747848,
+            0.0021712225589426266,
+            -0.03508567715696004,
+            -0.04578731401250471,
+            -0.008768228935292693,
+            0.010203318819548585,
+            0.024135236371743855,
+            -0.007467140460133838,
+            0.0028907962385387818,
+            -0.007582083854422456,
+            -0.005384477718912037,
+            -0.01867978482547249,
+            0.01641191108381631,
+            0.037168089880543244,
+            0.0031992495663980464,
+            -0.008533890994695162,
+            0.000778843942291118,
+            -0.004620258538699103,
+            0.010572690335163528,
+            0.033233014627100904,
+            -0.030497362482602697,
+            0.02974571152880676,
+            -0.004274442492343003,
+            0.025341347125910928,
+            -0.04307748954707389,
+            -0.018069609916024403,
+            -0.018491122727274337,
+            -0.02069307802429807,
+            0.05372967226264594,
+            -0.030768978301976772,
+            -0.03662045886993282,
+            -0.003205991669059688,
+            0.016815511161004534,
+            -0.0014152644009276566,
+            -0.0016015439520090104,
+            -0.03984716970293867,
+            -0.010828260334550217,
+            -0.05894985435251326,
+            -0.01638035185618134,
+            -0.021819442096404948,
+            -0.005315102328745982,
+            -0.06251579337157112,
+            0.0033713876515840233,
+            -0.07815974854883057,
+            0.001484860071649606,
+            0.04539427129131382,
+            -0.023530943197353476,
+            0.05142250021536681,
+            -0.024746600229063383,
+            -0.0005885740455257141,
+            -0.044698536284421976,
+            0.0026481001040031785,
+            0.02389178603305878,
+            0.051387657872255674,
+            -0.002949198236157656,
+            0.010674377196609455,
+            0.013322075519401089,
+            -0.026245090721950647,
+            -0.01036724558050635,
+            -0.005321884180696542,
+            0.005958798671856245,
+            0.013755178085972732,
+            -0.0074734329537091675,
+            0.038368016046563674,
+            0.010941421833174982,
+            0.03443115781919206,
+            0.034623617136788874,
+            -0.004361926531008214,
+            -0.03348681614127998,
+            -0.00632870804659212,
+            0.024433331840566692,
+            0.0321998807702602,
+            -0.02899983280320611,
+            0.005958471722097962,
+            -0.015602447981516634,
+            0.015633217539747047,
+            0.006321931116948646,
+            -0.011827541320878442,
+            0.02372204564999124,
+            0.015673426811623124,
+            0.006103300725621424,
+            -0.005165006670545588,
+            0.013589526056189892,
+            -0.025411864292353578,
+            0.03041791947375232,
+            -0.004456865298584878,
+            -0.04276058054583262,
+            -0.0024736078733106825,
+            -0.020059309867945227,
+            0.024883721794000956,
+            0.015578237413840115,
+            -0.00536523508503184,
+            -0.005290645798641171,
+            0.008242987948511412,
+            0.020469675635821673,
+            0.006668354647245447,
+            0.00017729517736814364,
+            0.02373604785121342,
+            0.029915687122900485,
+            -0.02432635861789771,
+            -0.029540533943782886,
+            0.008151297444985344,
+            -0.005730733861939129,
+            0.011318816579801282,
+            -0.02644855229221747,
+            0.00033932872948409137,
+            0.0035505684154028996,
+            -0.03570104343808711,
+            0.03408366016010711,
+            -0.009199700348630527,
+            0.005857598530788056,
+            -0.009538177736472163,
+            -0.0017768806148210118,
+            0.0027410153099351694,
+            0.03898108970435115,
+            -0.007367623392751247,
+            0.009827254226889936,
+            0.015465895035342481,
+            0.000981604396724216,
+            0.05143083558797524,
+            -0.027350877485564314,
+            -0.02183036962301852,
+            0.01885468378168636,
+            -0.007285239007489837,
+            -0.016846837665812908,
+            -0.020546383007536074,
+            -0.011863943814684079,
+            -0.016293192750336218,
+            0.01976516340568883,
+            -0.02387328798681798,
+            -0.011892336568452852,
+            0.007170651391639504,
+            0.0013725014050721622,
+            -0.009809669068563384,
+            0.0016617183704652932,
+            -0.006307590349798603,
+            0.02037693835615797,
+            0.031753656615094446,
+            -0.024892675105483313,
+            0.03577919637821846,
+            0.03178303188315581,
+            0.017260898395251155,
+            -0.02102289690701718,
+            -0.009661035772073294,
+            -0.00048542689442823136,
+            -0.014694631967051945,
+            -0.0326379186158631,
+            0.025443206465204822,
+            0.017194626456982255,
+            -0.011381421003142453,
+            -0.03680783761649673,
+            0.0017216115811536721,
+            0.022819859205829785,
+            -0.013600989058799166,
+            0.028141657788496834,
+            -0.01869047298692401,
+            -0.01366833755869951,
+            -0.00782959271979894,
+            0.031227914441058244,
+            -0.05252392878125891,
+            0.016957842026348195,
+            0.00020880696574616416,
+            0.001154370575234767,
+            0.010892067708194324,
+            -0.008368819756628046,
+            0.012224564683325483,
+            0.031700389750709,
+            0.018063405869147564,
+            0.0010794115307202266,
+            -0.002783668216641561,
+            0.01418234057248712,
+            0.017773269840982066,
+            -0.015653443355569053,
+            -0.000497740485657824,
+            0.019890882604100452,
+            0.022672054781919238,
+            -0.00521797370595221,
+            -0.016686942830774276,
+            -0.018443538351123584,
+            0.025270057639201107,
+            -0.010354431070738671,
+            0.03484047753830186,
+            0.04594785214094975,
+            0.0036441959528283004,
+            -0.05695476763071743,
+            -0.009412209619190693,
+            0.0010371110229966086,
+            -0.01099343927053564,
+            0.01606863921935641,
+            0.012470660511773616,
+            -0.03237187923799263,
+            0.006388672213167589,
+            0.012946570716567364,
+            0.022152554488829627,
+            -0.02540317621005867,
+            -0.023183252921885226,
+            -0.0030984019855787655,
+            -0.007890467768631811,
+            0.018648760402624687,
+            -0.008653405980661855,
+            0.0006228315253506687,
+            0.009672526897422602,
+            0.0005436494014569835,
+            0.011632326145022935,
+            0.005655942007108228,
+            0.021736827172951,
+            -0.001256608167230203,
+            0.013124413886287987,
+            -0.01986808920858981,
+            -0.0270450339809516,
+            0.006500423802582728,
+            0.04438242119211055,
+            -0.014813399694814117,
+            0.030621709833336064,
+            0.0092171224268983,
+            0.016433384739642534,
+            0.024875072913970443,
+            -0.00891470467241686,
+            -0.014124009014664287,
+            0.00845777541223868,
+            0.009019007848927453,
+            -0.04147358041382793,
+            0.012513030150663913,
+            -0.01710243399313324,
+            -0.033144939719206276,
+            0.01904674924133127,
+            -0.00534122554342279,
+            0.012367320751926314,
+            0.004270315800911569,
+            -0.002886460711688153,
+            -0.025696286516581048,
+            0.014137901708875033,
+            0.028135087538950946,
+            0.021797869775098042,
+            0.0296173052503884,
+            -0.012757078368361444,
+            0.0005460601780756187,
+            -0.008748930817524899,
+            0.005470975782187157,
+            0.04187083128854477,
+            0.0008687182830616019,
+            -0.0008032802520818105,
+            0.01507726552487313,
+            0.024945744517309208,
+            0.0376130540370386,
+            -0.015415560911023553,
+            -0.03407353556457147,
+            0.01564969972264396,
+            -0.010104443707865262,
+            0.018855792875436674,
+            0.005503259932317811,
+            0.0049634097823351596,
+            0.0038600156268000702,
+            0.019312419743287703,
+            -0.006635193789316492,
+            0.018798045806429395,
+            -0.02500983683871332,
+            -0.028785757874583755,
+            0.01829411751138161,
+            -0.005277972443258226,
+            0.03338623299668019,
+            0.016225885095995672,
+            -0.029522280454919586,
+            -0.01712097344863975,
+            0.01553706187924322,
+            -0.008465766956343768,
+            -0.024842700926275538,
+            0.01895228828512998,
+            0.007076862934523977,
+            0.008956105593078965,
+            0.0024373779389833218,
+            -0.01106503696703573,
+            0.01125466547367417,
+            -0.000818223040067914,
+            -0.015322790791698395,
+            0.014157131555941659,
+            -0.017853386731281715,
+            -0.014994640991761465,
+            0.01775991780643474,
+            -0.037498078532902435,
+            0.009570197712419421,
+            -0.007687297543935123,
+            0.009050494195378254,
+            -0.004222041469820445,
+            0.007460383234429904,
+            -0.0010804379186060264,
+            -0.014987857934522812,
+            -0.014903749199810126,
+            0.03228485430019625,
+            0.022096121825475306,
+            -0.000410816936759558,
+            0.006728733918931589,
+            -3.971636461500232e-05,
+            -0.026719848032785824,
+            -0.011253751790407687,
+            0.02245467425830765,
+            -0.005664635254398481,
+            0.005134496095445943,
+            -0.01109590855846966,
+            -0.0149616794609307,
+            0.0019726017255571673,
+            0.032668637270108894,
+            0.06066635618119076,
+            0.012001732730659747,
+            0.024422093585231484,
+            0.0189499396761026,
+            0.02119276033547528,
+            -0.008489818917377516,
+            0.006413901834269739,
+            -0.03097413287789443,
+            0.03509225573986143,
+            0.005392594635384506,
+            0.03108937318306421,
+            0.026664331248813712,
+            -0.04722525921137681,
+            -0.0058539171830594774,
+            0.017087531454837677,
+            0.022673712717885323,
+            0.0159020681868943,
+            0.0008382498574190522,
+            -0.008938174268517353,
+            0.035650940044285984,
+            -0.03315250299348084,
+            -0.023647111263155,
+            0.005059382024092534,
+            -0.01439537467317451,
+            -0.010488359956731476,
+            -0.006785295580877329,
+            0.02776449681317373,
+            -0.005584052419598926,
+            -0.004582019433008779,
+            0.009697064252638535,
+            -0.014400206558190682,
+            0.008165676294736836,
+            0.0007624778531368436,
+            0.008681263897172326,
+            0.039249866981334126,
+            0.0005918063050375113,
+            0.024275699182767528,
+            -0.004138313072843555,
+            0.011079529739650594,
+            -0.0158972266904151,
+            -0.016985058543605,
+            0.02276982363172991,
+            0.00351101120496393,
+            -0.03244126813432563,
+            -0.00768819222024939,
+            0.00928180979884261,
+            -0.0013793404539641824,
+            0.005266058566444064,
+            -0.00503433670609399,
+            0.003747448261046857,
+            0.03286230666764075,
+            0.0022383592721685153,
+            0.008606190020480513,
+            0.013593947677257625,
+            0.00030951915677929204,
+            -0.01490307680740127,
+            0.013254591138113545,
+            -2.1379308136707722e-05,
+            -0.012900073369514622,
+            0.03357519186606058,
+            0.006705902094864628,
+            0.04243347200485736,
+            -0.03302953674711988,
+            0.009031534935013593,
+            0.01822227379789576,
+            -0.008491770335368761,
+            -0.02725412856390648,
+            -0.03630788717786838,
+            0.008259145747740535,
+            -0.019003260334340084,
+            0.015781275447792516,
+            -0.010039277287105153,
+            0.029665124310477384,
+            -0.03563370794521629,
+            0.004714858110268169,
+            0.004053913355055313,
+            0.0009503673086552651,
+            0.02068360255395961,
+            0.018524093113914366,
+            -0.002785698934204024,
+            -0.031443835341186485,
+            -0.02066365433385339,
+            0.006869321907853112,
+            0.02431260052207595,
+            -0.010367048093621498,
+            -0.005612149485859478,
+            0.0006316058139530722,
+            -0.024369562712568606,
+            -0.016845124984022828,
+            0.030359419843284823,
+            -0.040974427176153246,
+            0.005252911449822132,
+            0.04272936053727608,
+            -0.04599092715016919,
+            -0.03802288716943514,
+            -0.002314988728575789,
+            -0.024124014894379466,
+            -0.006360771882090754,
+            -0.002068030981218464,
+            0.016669991113175926,
+            0.0036232707799735537,
+            -0.028763764013482795,
+            0.00943401978066451,
+            -0.005173988601101547,
+            0.003971420424839974,
+            -0.0031669592494977393,
+            -0.02216336363004636,
+            -0.01657772032717298,
+            -0.02531962163186398,
+            -0.035174102848865116,
+            -0.0015903434876311088,
+            -0.005158703150821607,
+            -0.028703075279214722,
+            -0.03941940849540492,
+            -0.021696000925262537,
+            -0.0160221488026189,
+            -0.00843213135542359,
+            0.040641882367348774,
+            0.021319060666253945,
+            0.019336844555345886,
+            0.0025252167889382917,
+            -0.020346900609125973,
+            0.049440028814233025,
+            -0.007642182460707816,
+            0.035628691886108926,
+            -0.0017882888000877158,
+            -0.013461669667950893,
+            0.00455558389906464,
+            0.03825428026120858,
+            -0.014248434654950971,
+            -0.0009902062564027718,
+            0.00027811655483232545,
+            -0.0011866253757890136,
+            0.0008535155950871286,
+            0.015433650396777696,
+            0.001307651096343185,
+            -0.007112199543314309,
+            0.030380673552683906,
+            -0.012803247194895328,
+            0.01637501752192592,
+            0.018350814869962304,
+            -0.0013359719137099667,
+            0.008064546833360553,
+            -0.021430659268086814,
+            -0.00731572118780415,
+            0.0044578576331236325,
+            0.008422210247967642,
+            -0.015096885054883586,
+            -0.009964588397149773,
+            -0.02359494467299634,
+            -0.01174145797396426,
+            0.0023855749166744366,
+            -0.004559202896782688,
+            0.004902138778592298,
+            0.034595817216889944,
+            -0.016068744500427584,
+            -0.035106033294403614,
+            0.009997207702122002,
+            -0.02663088875167486,
+            0.02222306406130687,
+            3.983384942889326e-05,
+            0.006838221769859366,
+            -0.0023434342406307616,
+            0.001765445386886839,
+            -0.00169484278667461,
+            0.002862331185439823,
+            -0.018217975204395134,
+            0.0031859734246840257,
+            -0.0032119988065673565,
+            0.0009052950782312657,
+            -0.0012459653052875638,
+            0.00687339892509406,
+            0.006332799551164718,
+            -0.0008019538898592668,
+            -0.002968451740267742,
+            -8.541086355097327e-06,
+            -0.020561816931299912,
+            -0.021642512405352372,
+            -0.028259049261410905,
+            -0.000989887278779396,
+            -0.012359733017834338,
+            -0.027804394906433664,
+            0.011443385424045703,
+            -0.006872684999639775,
+            0.008327844771213648,
+            -0.008969527138034904,
+            0.006219994886600349,
+            -0.008415106130309062,
+            -0.0004143996206881767,
+            0.013119286511794273,
+            -0.02925209282917943,
+            -0.02096907906398536,
+            -0.03096513357712367,
+            0.01762223841790055,
+            -0.00925741670612824,
+            -0.002656382260603827,
+            0.0025223527960181835,
+            0.005092659166535706,
+            -0.033606542487570154,
+            -0.006653438998497412,
+            0.007313682256676618,
+            0.009547290751154595,
+            -0.03287788897629887,
+            0.02446728061355753,
+            0.017023222076526946,
+            0.01874233066565012,
+            -0.02417566312664911,
+            0.04049827067897005,
+            0.014426879325647042,
+            -0.021740606525480655,
+            0.016712907564723732,
+            0.027385150577780248,
+            0.03177163221499527,
+            -0.03469098547555545,
+            0.004481376248622418,
+            -0.01518200063132508,
+            0.018918680020516044,
+            -0.025284921244460287,
+            0.020625170445627613,
+            -0.016707629602243063,
+            0.021278719363178687,
+            0.03520334001264928,
+            -0.005524369131116028,
+            0.03608133014180745,
+            -0.019234548780647943,
+            0.0059985801736444555,
+            0.004041778461022664,
+            -0.01645503683802124,
+            0.01024694092614158,
+            -0.0042700351372843386,
+            0.0006059159967286885,
+            0.009406090340935747,
+            0.019133229156115282,
+            0.019007219188041775,
+            -0.04442208547053466,
+            0.028588965123434923,
+            0.00804120101594811,
+            0.021472086070876265,
+            0.014177820346835521,
+            -0.02708807970982664,
+            0.004742981711172693,
+            0.002072673190279628,
+            -0.004659713164250794,
+            0.0061167904730676514,
+            -0.013796604518930328,
+            -0.034540857808309226,
+            0.006338792319814448,
+            -0.018918703778187926,
+            0.015036937364271984,
+            -0.009026670935442384,
+            0.0011667525636770179,
+            -0.01152652811400482,
+            -0.011938679695598347,
+            0.0006170592505446954,
+            -0.005495151258143897,
+            -0.0008075736908949407,
+            -0.03171721592827769,
+            -0.04427658539412448,
+            0.02036132271908089,
+            -0.0021221965958423804,
+            -0.04320396743189005,
+            0.004111179168317842,
+            -0.025352428814134187,
+            -0.01865918186335911,
+            -0.011237699157960728,
+            -0.025047923887906474,
+            -0.04219630696740606,
+            -0.011108496740166623,
+            0.007785006066683012,
+            0.016810642610918525,
+            -0.009081815651299525,
+            0.040965574143584864,
+            0.0073944364393447965,
+            0.015313576815177263,
+            -0.017731360025694955,
+            0.026137777790243826,
+            -0.019314392060123,
+            -0.03108486330987073,
+            0.02676829566273746,
+            -6.259161089416372e-05,
+            0.003053748984540415,
+            0.014358356168574803,
+            2.8186924071913628e-05,
+            -0.006181789640967465,
+            0.01947730374003864,
+            -0.028739300217145797,
+            -0.023460503178221397,
+            0.0012460072421725688,
+            -0.008850395570726385,
+            -0.016271242375881602,
+            -0.035896509136022194,
+            0.005227881053956234,
+            0.03324337105109496,
+            0.004970419987430122,
+            0.008273497228454482,
+            0.01814500594735864,
+            -0.02229593571615308,
+            -0.024230675455380678,
+            0.000639112529814716,
+            -0.010266640429960994,
+            -0.010257247303470066,
+            -0.021302483296731352,
+            0.019150757066356872,
+            0.01651375496287087,
+            -0.00390189952706486,
+            -0.006971692709239087,
+            -0.010621003777733253,
+            -0.01918206725910585,
+            -0.02020345113211036,
+            -0.013827322378214754,
+            -0.017980206764311026,
+            -0.034153741666571154,
+            -0.032195001867229935,
+            0.03225205339201753,
+            0.026354817588919596,
+            0.0014076409241492998,
+            -0.007959081015805435,
+            -0.023070419006792155,
+            -0.004483328183581872,
+            0.02879516595345542,
+            0.002549934782150634,
+            -0.0001541849716285838,
+            -0.00830090759167373,
+            0.015341095409025125,
+            0.00835695019244165,
+            0.005107844984733648,
+            -0.06676974053142512,
+            -0.006027540973808174,
+            0.01655283585235268,
+            0.01351155740684989,
+            -0.002359114145814062,
+            0.02591301169926259,
+            0.01108099264396926,
+            -0.015361038608114896,
+            -0.019765743462262668,
+            0.035195761515422364,
+            0.02408910733949848,
+            0.023468767772473746,
+            -0.0034492999385544727,
+            -0.004472311490618405,
+            0.007030833258457338,
+            -0.010684252427867153,
+            -0.013326964647936429,
+            -0.004558513842504658,
+            0.026925146859337397,
+            0.016099443021514896,
+            -0.0064331198280825575,
+            0.010848931851939124,
+            -0.0008320209125061849,
+            -0.019444491995041537,
+            -0.001841330829156697,
+            0.008730275423516445,
+            0.0012974082030866743,
+            -0.014611902493400719,
+            -0.005911807768916134,
+            -0.019130604101278263,
+            0.024719468317569562,
+            0.0209456329723249,
+            0.03381727436368338,
+            -0.013237902591972769,
+            -0.0003695142331921278,
+            0.005410360755058108,
+            0.0002841632253252816,
+            0.0023352645111704958,
+            0.004801123870325242,
+            0.02623649813035263,
+            -0.029010710171522985,
+            -0.020741774066460415,
+            0.0068426980682519505,
+            0.013445270043414486,
+            0.0022744032984779294,
+            0.033840485942696216,
+            0.035040080489222324,
+            0.004166066423508316,
+            -0.010046666684988869,
+            -0.006148765564658713,
+            0.007071740742464309,
+            0.01865897335451031,
+            -0.023267411753900076,
+            -0.0005936657905496225,
+            0.004833030519903764,
+            -0.004229582648167378,
+            0.03620610659071198,
+            -0.009961238880412731,
+            -0.007197670122096852,
+            0.022212791947347127,
+            0.09819205436926473,
+            0.00047695621444943017,
+            0.0025354521225913395,
+            -0.029358210234742176,
+            -0.023614016211567963,
+            -0.0012564198366457442,
+            0.00242680708457405,
+            0.00830167274308547,
+            0.028076260517106013,
+            0.0004852400049405906,
+            0.011579274037419117,
+            0.02773972145320728,
+            -0.017217305534161797,
+            0.006752070301141707,
+            -0.011376051474975524,
+            0.024577071515205502,
+            -0.012694034026935636,
+            -0.0019809719792138673,
+            -0.016512649783050146,
+            0.015907402258634816,
+            0.004556574687833768,
+            0.004907510649166062,
+            0.011347630984408937,
+            0.009824484201337644,
+            -0.02650731900265866,
+            0.01756773113723491,
+            -0.005671986127099093,
+            0.004449611508541289,
+            -0.027841346572384462,
+            -0.020124233970502267,
+            0.0025986391930124847,
+            -0.0014440758308989809,
+            -0.009579444248630085,
+            0.006005274872617889,
+            -0.0024984235450999127,
+            0.03765664613203687,
+            -0.03765496603611941,
+            0.01409350051861866,
+            0.0066453175608372555,
+            -0.0015390841089599882,
+            0.011806653942640485,
+            0.017306914776093522,
+            -0.008206256939290151,
+            0.02589800448884443,
+            -0.00798213351353373,
+            0.009846274212000523,
+            0.015230070803055296,
+            -0.03449531102975745,
+            0.009174739184806312,
+            0.010145762060174144,
+            -0.024165660737765746,
+            0.012146877603301336,
+            -0.013784371016298891,
+            -0.004818759788054113,
+            0.005930446700613062,
+            0.007349837554764034,
+            0.010733731974181452,
+            -0.02295571464086674,
+            0.016176424826629898,
+            -0.010947841926210105,
+            0.0037766665765835445,
+            0.013559846998199008,
+            0.024892849808979323,
+            0.0012426907454129122,
+            0.0005950515136086193,
+            -0.01379469431504576,
+            0.005534724429523084,
+            -0.015667467221535825,
+            -0.022401586867694584,
+            -0.014192041775234079,
+            -0.006366846766730035,
+            0.01029598687444803,
+            -0.003972386708834858,
+            0.016971476088808737,
+            -0.010309648640697812,
+            -0.010645860303528099,
+            0.0037818310327362103,
+            0.01040806357838676,
+            0.008316605962234849,
+            -0.02855914079113881,
+            -0.005676310556150579,
+            0.004961084138163161,
+            0.00029285971503001265,
+            0.009592497954559435,
+            -0.014556073011576487,
+            0.013689526487160009,
+            -0.013290113354287095,
+            -0.0010726537498214612,
+            0.013201860397060236,
+            -0.012443552043930505,
+            -0.004012305744087929,
+            -0.01592905426965635,
+            -0.028487066798154617,
+            0.008462143660968396,
+            -0.01365614159523485,
+            0.001076882584445294,
+            0.0019503240125505646,
+            0.004730085646042709,
+            -0.016919069570642004,
+            -0.007860046623166481,
+            -0.033515448866521046,
+            0.011065416478687504,
+            0.02314589687882036,
+            -0.006435733825418645,
+            -0.030351727858789472,
+            -0.009546333526852862,
+            0.011327489786114963,
+            -0.0071765122743172715,
+            -0.01676736383051139,
+            -0.0029390038145944096,
+            0.007617910650439855,
+            -0.008174620754097493,
+            -0.00508943241317381,
+            -0.0025173969010414098,
+            0.0217328037301181,
+            0.016094975605203832,
+            0.01769642142712967,
+            -0.021400862603034036,
+            -0.00976992221747187,
+            0.026043879119776864,
+            -0.03275550610951898,
+            0.012441465975914529,
+            0.024587446266013183,
+            0.003446527051428522,
+            0.005492228054838714,
+            0.0030377554109693714,
+            0.005459159163772825,
+            -0.006985179226831048,
+            0.019925583798328282,
+            0.010870421035240845,
+            -0.019709062313471467,
+            0.005385443078244229,
+            0.02086085539375208,
+            0.0037248737640193097,
+            0.012892299156840998,
+            -0.017546870090541213,
+            -0.0020341807891543716,
+            0.015594779287017958,
+            0.04816214672670215,
+            -0.0032851797178091685,
+            -0.010450846098750195,
+            0.029513742590677183,
+            0.023526004023637855,
+            0.005373455940309873,
+            0.0012120777808635214,
+            -0.02525630456877349,
+            -0.013075130701774785,
+            0.01962235086017781,
+            0.016944870494393874,
+            0.013484925533584295,
+            -0.0023255656480479004,
+            -0.019236081983457366,
+            0.021520156296467717,
+            -0.01701733319731161,
+            0.012422659010697477,
+            -0.023764941037007197,
+            -0.001490790833263327,
+            -0.0035419008954878724,
+            0.021998949056452454,
+            0.0021819635638621092,
+            0.008891505304514552,
+            0.024354552127458608,
+            -0.006978548952551799,
+            -0.019526615985982355,
+            -0.0051799020400150305,
+            -0.014578953088901798,
+            0.022506635912517146,
+            -0.024117708614271965,
+            -0.007008641019892289,
+            0.011379795968360019,
+            -0.012362093847309038,
+            -0.0102534051092702,
+            0.001164934138876502,
+            0.017686807394319725,
+            0.0009423993435129515,
+            -0.014784508280754078,
+            0.01944445593807992,
+            -0.0011359271135469456,
+            0.010410915597949927,
+            -0.01921419010977189,
+            0.009390638453648025,
+            -0.016682442516992777,
+            -0.007849184048209236,
+            0.010343008873053859,
+            0.0012604521199837046,
+            0.003948300703645506,
+            0.002592490121754231,
+            0.007773437307969315,
+            -0.03355658426684504,
+            0.022695505487314516,
+            -0.010751606023585792,
+            -0.003538523688786008,
+            -0.0014125330167969218,
+            -0.01678692701672328,
+            -0.0006918162525881059,
+            0.005816502852598842,
+            0.010878358093847386,
+            -0.016045147059003256,
+            -0.00873270513444376,
+            -0.04920244395446931,
+            -0.030314009649638693,
+            0.023156333024481057,
+            0.013733657599720879,
+            -0.010971506342695331,
+            0.018852159415605117,
+            -0.023760853159532587,
+            -0.010587249404421732,
+            0.0011379869499042905,
+            0.023319856329056813,
+            -0.03539290744576506,
+            0.01215369703816749,
+            -0.014493682360886549,
+            -0.02039288669534806,
+            0.009070984596830142,
+            0.011445354735283254,
+            0.0007259852045513916,
+            -0.0011514627555568814,
+            -0.029350945123925453,
+            -0.010246845821727087,
+            0.02391068177425878,
+            -0.00861718589055971,
+            -0.004723044986826333,
+            0.024716520302409263,
+            0.0033721801437248873,
+            -0.012524663490691517,
+            -0.017955758068233754,
+            0.021612671655463236,
+            -0.026077203994996058,
+            0.007645944309444457,
+            0.010032183011253845,
+            -0.0173991263874487,
+            0.021649599230588117,
+            -0.007529701797485131,
+            0.020327763440441377,
+            0.004227390938169872,
+            -0.011249233530251522,
+            0.020352283974471443,
+            0.013644443758970054,
+            0.004216958800094986,
+            -0.004081669404784771,
+            0.006869353098014179,
+            0.02212732880780841,
+            -0.011021739614266063,
+            0.02219580717345023,
+            0.028777213295471067,
+            0.00017328191188631918,
+            0.03260179554743322,
+            -0.021287655520443285,
+            -0.03159728983004563,
+            0.021740163175540124,
+            -0.01035146858119169,
+            -0.021564906188712388,
+            0.028914245422221707,
+            -0.013674731836508768,
+            -0.00786476052825481,
+            -0.0166718429993922,
+            0.00533027311342322,
+            0.016680336311088843,
+            -0.005246190451159961,
+            -0.014360934066628332,
+            0.0029628103834216345,
+            0.019809487418628386,
+            0.014587234544007958,
+            0.029780009648869975,
+            -0.010778199815920887,
+            0.035180242185168895,
+            -0.002320576872571577,
+            0.008990157317473909,
+            0.012415465232073404,
+            0.006489435152972723,
+            -0.011251531176923445,
+            0.013501894212117511,
+            -0.003446759011266526,
+            0.008230548711588447,
+            -0.001449023474280409,
+            -0.035111679577202855,
+            -0.013709382199848678,
+            -0.04709294504969336,
+            -0.0023752976043235697,
+            -0.0025098360125255215,
+            0.0030975417740913154,
+            0.09736815869389318,
+            -0.005723049647039619,
+            -0.011988817571623457,
+            0.001625059666070763,
+            0.009611697638805556,
+            -0.0009896494420056814,
+            -0.007606836952841873,
+            0.015973152482488318,
+            -0.01343292635775299,
+            -0.02140928016038583,
+            -0.010532701190806147,
+            -0.01288594454368977,
+            0.012079198190522715,
+            0.015929716343161652,
+            -0.0005947359502228821,
+            -0.015207331663761012,
+            0.006524010655124294,
+            -0.004566385899752684,
+            0.0064717704997400245,
+            0.008647099975176308,
+            -0.010414668624343924,
+            -0.0071698879102925555,
+            0.030899143952119196,
+            -0.025301269549419674,
+            -0.024361748354052457,
+            -0.0010579931222861095,
+            -0.009055796737486008,
+            -0.010335459737012047,
+            -0.002692653633719985,
+            -0.0017679501322462056,
+            -0.010519994985692081,
+            -0.010019865263564572,
+            0.006537076627706128,
+            -0.015472130358273033,
+            0.004812857776601667,
+            -0.0047457011269495634,
+            -0.013296069496526007,
+            0.017054968086538724,
+            0.022563785357424853,
+            -0.02841491200807702,
+            0.0006755428946168761,
+            -0.017422510783648978,
+            0.012941211381006863,
+            0.008866069010589792,
+            0.006049618782357251,
+            -0.022451528209097505,
+            -0.023622906582403243,
+            -0.015019272754405791,
+            -0.01927535750568903,
+            -0.024973490129133732,
+            -0.009073381957684092,
+            0.0067692803507600945,
+            -0.001267339788153158,
+            0.021918281111033645,
+            0.000816538714279555,
+            -0.01812545613159305,
+            0.0061084913570894905,
+            0.000568371220952269,
+            0.018292497018668842,
+            -0.027468677288882186,
+            0.0033806422195878524,
+            -0.02307582511848465,
+            0.033508234091571124,
+            0.0039195050746594975,
+            -0.026680111514794824,
+            -0.021558668725755275,
+            -0.003311830287529699,
+            0.018994577319066473,
+            -0.0025745912019675805,
+            -0.00724049447539589,
+            0.014775207617625293,
+            0.0155334760323838,
+            -0.004693118176196483,
+            0.002015625406296403,
+            0.027968079002437102,
+            0.01077998866180723,
+            0.0313892866933904,
+            -0.015236169297590675,
+            0.004945828495617267,
+            -0.027215016713379062,
+            0.010214419950212732,
+            -0.01503548137805401,
+            0.008859554812646242,
+            -0.01290890682557532,
+            0.03531384210182567,
+            -0.012005613861257432,
+            -0.004250698950192301,
+            0.03253288796441852,
+            -0.029150986393345252,
+            -0.0035020051536906334,
+            -0.009255437891304041,
+            -0.004142204892160768,
+            0.013240580426083011,
+            -0.001198093874819309,
+            0.008485141356876356,
+            -0.01854370462655459,
+            9.726096864073609e-05,
+            -0.003221852304410318,
+            0.005404595442127598,
+            -0.02005458688172967,
+            -0.003314118809778734,
+            0.0073427568419969814,
+            0.004259518237708634,
+            0.002408630356035373,
+            0.016420646142725346,
+            0.006662121128241274,
+            -0.0029357701493526907,
+            0.03424224243607692,
+            0.008171895850772245,
+            0.0152521424877615,
+            -0.0018638264193202728,
+            0.0030234211261443034,
+            0.019287963516760636,
+            0.019912956661192856,
+            -0.00048443399729935116,
+            -0.01058935153883094,
+            0.00195113175754497,
+            -0.001140732137433033,
+            0.011048741602327437,
+            0.024396046054912645,
+            -0.021158025221644738,
+            -0.02419562092713724,
+            0.016377661005279716,
+            -0.015076145951740961,
+            0.01903712233378994,
+            0.010541463112889985,
+            -0.010099351302095792,
+            -0.00031558616108876247,
+            -0.042316016201379245,
+            0.010964989031040692,
+            -0.02068939391415103,
+            -0.04004120028915868,
+            -0.00244383573783919,
+            0.0009542507389088937,
+            -0.029439456243505052,
+            0.02822814855547978,
+            0.0023755032182188156,
+            -0.017280221740813364,
+            0.018223709137876153,
+            -0.005548654680726505,
+            -0.0024040301239229064,
+            0.005397711787238679,
+            -0.01092026090471121,
+            -0.010648485086769927,
+            0.008235250252660972,
+            0.010957715075632535,
+            0.005425683656535705,
+            -0.005698919020608095,
+            -0.011284911899894062,
+            0.009975882651638084,
+            0.012318901214538077,
+            -0.0019922884373083155,
+            0.010050520101815593,
+            0.0031720752428247446,
+            0.007352859750323051,
+            -0.0073483049386141005,
+            0.0046483587496626965,
+            -0.005093220128411125,
+            -0.026080057085901446,
+            0.012863580653903103,
+            0.0052434400313520774,
+            -0.011863360599637747,
+            0.0103209502958491,
+            -0.01048162756158923,
+            0.005787157766911021,
+            0.001318945460232246,
+            -0.0059333230422061395,
+            0.004821320238893602,
+            -0.0038922239931512784,
+            0.011267634571983924,
+            0.013256611037442042,
+            0.004592529900805719,
+            -0.013076954459595086,
+            -0.02177318635989856,
+            0.009001291224623467,
+            0.01959767163356408,
+            0.024161754192119852,
+            -0.030416109293488064,
+            0.003784634588325551,
+            -0.0014915849060829743,
+            -0.0020550564975080885,
+            -0.015632785947151753,
+            0.01587790981601866,
+            0.01716742056401084,
+            0.012721563202528083,
+            0.01893393006967701,
+            -0.004009617703771589,
+            0.010539100665843704,
+            -0.02080068888791658,
+            0.013642451766699067,
+            0.0008609869904066794,
+            -0.013629067231663643,
+            0.0014620655123815298,
+            0.027046304278766087,
+            -0.011056783825898616,
+            0.006206969699250696,
+            0.008097527175511791,
+            0.022111870736907183,
+            -0.012509479100313492,
+            -0.003735868296238069,
+            0.01516254968167132,
+            0.0014878674796313539,
+            0.0035887079417890352,
+            0.016533149527222273,
+            0.04729052276110659,
+            0.00102751186013391,
+            0.0016713773449738153,
+            0.0192177920520243,
+            -0.005563519063161466,
+            -0.01114821419109643,
+            -0.003549999863325982,
+            -0.021807893906469106,
+            0.010998558936481785,
+            0.00431279785115251,
+            -0.02186324212032519,
+            0.01024093366826206,
+            0.0125495473550584,
+            0.005268320135120931,
+            -0.0031679212410026603,
+            0.02574575247827441,
+            -0.019807879421996258,
+            0.014935295894965128,
+            0.03329450628384787,
+            0.00493356778223407,
+            -0.011423949333937285,
+            0.007263264249871384,
+            0.0027354459678742013,
+            -0.004494406041906684,
+            -0.00606745413660583,
+            0.010497391290734088,
+            -0.008191702034391382,
+            0.009870458191510176,
+            -0.009580130479693,
+            0.039202091799313755,
+            -0.009619018331262844,
+            -0.010782138248435458,
+            -0.004558645539551881,
+            -0.006241773108460662,
+            0.003478414182938609,
+            -0.003395165694642073,
+            0.005812225135633631,
+            -0.0069045407353789945,
+            -0.014607146220271209,
+            0.0008522982757210454,
+            0.00020175973815948303,
+            0.014001922919116725,
+            0.0011804904415274088,
+            -0.004871874532755353,
+            -0.007877217154706205,
+            0.015243277056007988,
+            -0.006047906208304912,
+            0.0006531937033317192,
+            -0.0011653523616313422,
+            -0.0013662556399359335,
+            -0.007401050012179761,
+            -0.015166105795085528,
+            0.00974609244984173,
+            0.016253733908242134,
+            -0.013857834499211056,
+            -0.005142936208363415,
+            0.012274716212832219,
+            0.008540229125193836,
+            0.011742606168187876,
+            0.0007451338650642946,
+            -0.009255813682572937,
+            0.008292531177374192,
+            -0.015020523035432158,
+            -0.008781317965985481,
+            0.0026250469418136603,
+            0.002381147382977664,
+            0.005839723981046825,
+            0.0006240641828668715,
+            0.004502463508605272,
+            0.005798912705218846,
+            -0.00497936644919367,
+            0.01112666823857836,
+            0.008198424171006548,
+            -0.002452143482105151,
+            -0.01017569410837814,
+            -0.001127589984272638,
+            -0.015733298167661346,
+            -0.016758315411028353,
+            -0.0009286266498936771,
+            0.015175988113750882,
+            -0.02352862898718829,
+            0.018996352975391644,
+            -0.0014524363611708201,
+            -0.0033802546171857704,
+            -0.021392428456030312,
+            -0.0025923983911570183,
+            0.0034214188668518796,
+            0.01255768230394476,
+            0.01858853630745928,
+            -0.02729543519465384,
+            0.010575894388563116,
+            -0.007204725245442141,
+            0.030394929424014717,
+            -0.018328782140005044,
+            -0.004022941273610227,
+            0.026915618079621992,
+            -0.02200018306828758,
+            0.0008791185377513978,
+            -0.011454606955158956,
+            -0.008103032693853504,
+            0.019287674837688177,
+            0.023747720480736777,
+            0.02038287242167783,
+            0.01364489294869211,
+            0.0007758044919789085,
+            0.004768030222445397,
+            0.03572705648994403,
+            0.010568241270862248,
+            0.014828087092859878,
+            -0.007542489523110974,
+            9.457808640264811e-05,
+            -0.02554033312739386,
+            0.01635899434369978,
+            2.587383534566272e-05,
+            -0.012370727506722627,
+            0.007162422186621379,
+            -0.013370882403931942,
+            -0.011066483600541058,
+            -0.007151077450730025,
+            -0.01721910018182534,
+            0.02152239849400241,
+            -0.012322440280680164,
+            0.004587729883596212,
+            -0.009079747696669277,
+            -0.013272707286450417,
+            -0.012953682633400515,
+            -0.014202214807187545,
+            -0.01313141988034687,
+            -0.026041403779538307,
+            -0.027109731260915763,
+            -0.006573285295895077,
+            -0.023780366346334524,
+            -0.013799072648505526,
+            0.005588622594136253,
+            -0.022617902666316757,
+            0.012262325666428604,
+            -0.012689827812482627,
+            -0.01226264872059864,
+            0.012805734807835008,
+            0.020274844315616616,
+            -0.011765231803328348,
+            -0.028494489703400308,
+            -0.018583843821043854,
+            -0.019970998470704358,
+            0.012576201137505969,
+            0.002266288538944273,
+            0.013165889075444961,
+            -0.003018437854660793,
+            0.003227412667880239,
+            -0.014895831590298957,
+            0.018570624675927183,
+            -0.03442390703780963,
+            -0.03108003806963536,
+            -0.013611306765348361,
+            0.005182777066476028,
+            -0.0015782033203578553,
+            -0.011429046504256945,
+            -0.0018606648572920801,
+            -0.00433805271182878,
+            -0.025109572024446088,
+            -0.010134856632190067,
+            0.011010662054484161,
+            -0.015205629927469414,
+            0.0072933565255675036,
+            0.0009018832564038863,
+            0.013870902834260887,
+            0.009587432944616442,
+            0.013393301893492882,
+            -0.006095063698384775,
+            0.004356485794113208,
+            -0.0026476612747324386,
+            0.0004894024149759108,
+            0.009336366919474478,
+            0.00839591561027103,
+            0.010551161369079399,
+            0.002260707777260338,
+            0.007830059223001696,
+            -0.0060927446216633265,
+            0.018065002687169148,
+            0.015436944875538481,
+            -0.0002520689442496089,
+            0.013323098485251487,
+            0.0003769862591044255,
+            0.00022293304921127018,
+            -0.0029970303106340466,
+            0.007888437018607987,
+            -0.011785957933642163,
+            -0.001311704406355858,
+            0.015138190893483804,
+            -0.01707247730569978,
+            -0.013045187198628258,
+            0.03107034781283638,
+            0.0018583638488268697,
+            -0.02495564203304796,
+            0.031146554085787762,
+            0.004220047610942943,
+            -0.005835199594223787,
+            -0.013205086624848782,
+            0.0014059047822222418,
+            -0.011570624844100298,
+            0.003584232809362274,
+            0.014295032061446826,
+            -0.02231984223610487,
+            0.010521094801839404,
+            0.008446892441796151,
+            -0.009681535919980559,
+            -0.012337659795586223,
+            0.009847073709209602,
+            0.016809328445200358,
+            0.004472485504034416,
+            -0.03539093810210523,
+            -0.011957694102753411,
+            0.02145470732008716,
+            -0.029237428225009913,
+            -0.00810041956259673,
+            0.007452962400105787,
+            0.0065154832040426065,
+            -0.019353109017609053,
+            -0.008580955020412362,
+            0.005725794605897509,
+            -0.004940390374637084,
+            -0.003892210172017758,
+            0.017541623128534682,
+            0.00485047043626186,
+            0.013698341792101025,
+            -0.01689738578307474,
+            0.009771458057400557,
+            -0.022222817828602557,
+            0.004021882074676753,
+            -0.023373700500461992,
+            -0.009873403167702603,
+            0.002364143609079849,
+            0.012454271200139762,
+            0.014328556467632212,
+            0.027839363143659326,
+            0.030056404238348527,
+            0.013995696971422429,
+            0.007833677585250953,
+            -0.016265199386953717,
+            -0.018134327237616175,
+            -0.006409513840075933,
+            -0.004033173350204784,
+            -0.008714798059218596,
+            0.0066373547199279235,
+            0.01858354212009877,
+            0.004499712802472377,
+            0.02609078206670711,
+            -0.006227189596272876,
+            0.009039744622014585,
+            -3.127526911951914e-05,
+            0.0008833930131827993,
+            -0.004226083827636182,
+            -0.01971685546584527,
+            0.00481003211084742,
+            -0.0013721482162825508,
+            0.025874758880591167,
+            0.00940496860916206,
+            0.0024915013399541847,
+            0.01615978314811047,
+            -0.006378952757397484,
+            -0.01174653607178726,
+            -0.010540107733238588,
+            -0.0061926268398085176,
+            -0.021752582127981104,
+            0.029059100503124387,
+            0.021195595568253612,
+            -0.010800765264123782,
+            -0.013693582166598082,
+            -0.00903369526059487,
+            0.02795767990556847,
+            0.004080965688836953,
+            -0.0013050212744006933,
+            0.016313311850308505,
+            0.012084555958999215,
+            0.012329325712491252,
+            0.009794698207689061,
+            -0.00036157148861933123,
+            0.0015364709122308352,
+            0.03722522141440073,
+            -0.007427028510548186,
+            0.0026463292682538495,
+            -0.005239324411651515,
+            0.0025987932868461313,
+            -0.009705817520422487,
+            0.007242122655628283,
+            -0.004012676385296197,
+            -0.0158760051470016,
+            0.015003444174308607,
+            -0.022443331305169994,
+            -0.0016749787204471238,
+            -0.008150635190952799,
+            0.0016696640640947303,
+            -0.02781023337840043,
+            -0.0025731206607371845,
+            0.006695065701630532,
+            -0.015547340828489069,
+            0.011816010501093885,
+            -0.01740673229051557,
+            -0.011553604323149105,
+            0.007914961618894248,
+            -0.012034157288065447,
+            0.0016684130271324446,
+            -0.0008268957621923278,
+            0.004460864332388434,
+            0.013426055079012884,
+            -0.011008808570221245,
+            0.012166046631826273,
+            -0.005816284310138776,
+            -0.007872091214834688,
+            -0.0016055129487028885,
+            -0.0030049472973942043,
+            -0.0025600746761171543,
+            -0.0009966231170649917,
+            0.034355910234323155,
+            -0.004446570896048543,
+            0.003761242147499512,
+            -0.01362639743248326,
+            0.0019833052855181806,
+            -0.036176376394999044,
+            0.028344464817309477,
+            -0.008642835889481802,
+            0.011013502221008498,
+            -0.0017508365480865596,
+            -0.013243706268954915,
+            -0.01456517309747086,
+            0.015247299501245178,
+            0.007602596572394442,
+            -0.005854531057349665,
+            0.013558248087347848,
+            -0.019724373614448298,
+            0.008530500822910953,
+            0.004463694543554244,
+            -0.015561397976987216,
+            0.004668780190328835,
+            -0.007479912658259757,
+            0.012657201279639194,
+            -0.01774730033362932,
+            0.004314594593150765,
+            0.00958571617187387,
+            -0.01892876084898881,
+            0.009901041800918215,
+            -0.028675879564512306,
+            -0.004631920242208613,
+            -0.012323455524366223,
+            0.0053804608827285865,
+            -0.000660864620708182,
+            -0.02389088252276507,
+            -0.028944280660230762,
+            -0.0002099667627277945,
+            0.00458533068247518,
+            0.0018878351997997833,
+            0.011621407953503755,
+            0.005970807015151215,
+            0.008542958629199003,
+            0.01023896054756221,
+            -0.00205207247588841,
+            0.009402752124876478,
+            0.04169706217414899,
+            -0.022899912321317197,
+            -0.011574970753549647,
+            0.005321872977369877,
+            0.011499271505460519,
+            -0.02647653483798284,
+            -0.011039791758347888,
+            0.0033424345263339144,
+            -0.005219877979550397,
+            0.03609469691834737,
+            -0.02448746405881972,
+            0.019020618652020673,
+            -0.011392297116547224,
+            0.007625475937968714,
+            -0.021465240076078376,
+            -0.003921088264697311,
+            -0.014265745991522477,
+            -0.00011221615798127438,
+            -0.0031478236827989703,
+            -0.0076053733863387755,
+            0.007485039021296698,
+            -0.001907678661303243,
+            0.0020348246022161925,
+            0.006291436821831794,
+            -0.02173305455485444,
+            -0.0019125499969493854,
+            -0.016432460924913166,
+            0.030367263349759286,
+            -0.016242095671439496,
+            0.005072805486919414,
+            0.0053258139728101506,
+            -0.002810639419737258,
+            0.012368771814660942,
+            0.004165967191536186,
+            0.00513754440091394,
+            -0.004199767241827414,
+            -0.005142642360546972,
+            -0.0053886448467479215,
+            0.017868874549760376,
+            -0.013280048598849825,
+            0.035024701466710065,
+            -0.0100959186045929,
+            -0.007094001877261568,
+            0.0019733042325232155,
+            0.010380430710282744,
+            4.2084202709296443e-05,
+            0.01053565670076933,
+            0.0034270489770133685,
+            -0.0006311078385359693,
+            0.0019774779328212756,
+            -0.017162448473652843,
+            0.003089758682621882,
+            0.01682656561693936,
+            0.005463677441664502,
+            0.004469261499451007,
+            -0.013332130891003062,
+            -0.012050233723282738,
+            0.004538268704442613,
+            -0.007533940408255447,
+            0.0003675737140813486,
+            -0.018816953644773076,
+            0.013972696748575243,
+            -0.0011096683686631858,
+            0.0002715683052236706,
+            0.00524678369365015,
+            0.023056514997908218,
+            0.01813167860994747,
+            -0.002553352602551003,
+            0.03155164978556312,
+            0.0174738879054593,
+            0.0029268039949260673,
+            0.010637295711777793,
+            -0.033222781195984666,
+            -0.013301873512972214,
+            -0.0013721703891642354,
+            0.02238421933149517,
+            -0.01530189703335162,
+            -0.014059029663525449,
+            -0.0037393534403749947,
+            0.0010394601933309767,
+            -0.005177359933828505,
+            0.0042489306011644495,
+            0.011039889511148341,
+            -0.014697595240068863,
+            0.007597149702441915,
+            0.0026780496822907634,
+            0.02092734730183164,
+            0.0004688144432243786,
+            0.017181410417231698,
+            -0.0005083970646221274,
+            0.00749639055342427,
+            0.007319420826337891,
+            0.017541463125355265,
+            -0.01002811389296805,
+            0.023582113845081534,
+            -0.008823933397729605,
+            0.0019889258902730535,
+            -0.01257106158422629,
+            -0.028159594067544737,
+            -0.012624750220149823,
+            0.015247564545279736,
+            0.005765911899842925,
+            0.006478299226263732,
+            -0.016537090297144265,
+            0.006276997440048981,
+            0.015647122171767974,
+            -0.002251683151059454,
+            0.010295358869874306,
+            0.009730378299068289,
+            0.00012816405890877098,
+            0.010181120688239766,
+            -0.00801256937783534,
+            -0.00734257767723293,
+            -0.004497520124843868,
+            0.023982754195729983,
+            -0.005443550092842393,
+            0.022434177122972028,
+            0.012647958478747348,
+            0.00616791857573897,
+            -0.010853355357954727,
+            -0.003340111254357573,
+            0.023389634440057473,
+            0.011863246790550376,
+            0.00029642699874398923,
+            -0.009846034060495678,
+            0.0019458876579524617,
+            0.0054010373765486005,
+            0.012983913038716568,
+            -0.02049427270240938,
+            -0.004344571352918878,
+            0.008461947538684664,
+            -0.008472919299762712,
+            0.004318309005722874,
+            -0.030156444522730416,
+            -0.005812303237222271,
+            0.021354352236942434,
+            0.008392281015122372,
+            -0.010232039722494646,
+            -0.004395682831498155,
+            -0.0006527587802367882,
+            0.0040601850269714945,
+            -0.0024323167053212176,
+            -0.0023729138545912176,
+            -0.008541233856441836,
+            -0.006470933419694742,
+            0.014035696617797647,
+            -0.0052691760528871924,
+            0.008597459268477037,
+            0.00012084859681905111,
+            -0.0017255958688825231,
+            -0.007323529232751411,
+            0.03382792341957058,
+            0.008087732655845718,
+            0.005980666974986827,
+            0.0013613625176327891,
+            0.0042244330945601215,
+            0.000616791811672998,
+            -0.0028374856646453707,
+            0.018075360591135542,
+            0.0156738939725117,
+            0.005054696911814988,
+            0.009038525980341447,
+            0.014672270369917403,
+            -0.019700405219230957,
+            0.012625337070556061,
+            0.012184185186281482,
+            0.019382321278525114,
+            0.0007219498737851579,
+            0.002564970959687783,
+            -0.010601873823737926,
+            -0.0029963537053764736,
+            -0.002070567089566093,
+            0.029563028293936092,
+            0.0005903976560069006,
+            -0.010509758568838935,
+            0.010992421945232951,
+            0.020603868018231348,
+            -0.02101157589451638,
+            -0.001971974816402967,
+            0.005963911670920596,
+            -0.010704207628592966,
+            -0.014466113540150018,
+            -0.021269176952661614,
+            -0.013331247081394942,
+            0.0071597345125287174,
+            -0.008162102923896894,
+            0.02367549407145414,
+            -0.014256771777502395,
+            -0.03960796112042958,
+            -0.014803626428839152,
+            -0.00864320347698291,
+            -0.027531337207277483,
+            -0.007305441146142331,
+            0.00965056093889974,
+            0.016698589600806273,
+            -0.003330867073501157,
+            -0.025198703196464594,
+            0.030483790406960656,
+            0.008872821868559751,
+            0.023708422767352917,
+            -0.021464374508407488,
+            -0.0018617153194555344,
+            -0.00886376784514331,
+            0.014505918224188001,
+            -0.017588761818663975,
+            -0.011756112651591795,
+            0.020014632992762026,
+            0.014356419813769714,
+            0.007246557500044434,
+            -0.004355132555323843,
+            0.0067299398462013936,
+            0.007649499235136303,
+            -0.012515181210363267,
+            0.012239792251154642,
+            -0.013499277313916029,
+            -0.024858295126228433,
+            0.014430561986516884,
+            0.002229494935179828,
+            0.009563004428064055,
+            -0.0015984171085659323,
+            -0.016197659608768088,
+            0.010078159490026602,
+            0.00808729175515505,
+            -0.017508840440512225,
+            -0.004793934415922065,
+            0.007133527615305415,
+            -0.020818869260580315,
+            0.031234686259160864,
+            0.011872765152279094,
+            -0.010692949770503467,
+            -0.001989039629661815,
+            0.004431079028880923,
+            0.016511544608975148,
+            0.007893700385963683,
+            -0.01997314388229276,
+            -0.003708006454566117,
+            0.01530128306203401,
+            0.00822588305477249,
+            0.004861703074728471,
+            -0.010836328783594584,
+            0.03434874763691911,
+            0.006082873924472328,
+            4.192481378506604e-05,
+            -0.0052997095110967115,
+            -0.0076108924981838355,
+            0.004196112457289409,
+            0.018905391806961686,
+            0.013330078288599763,
+            -0.004200590783114919,
+            -0.006630107527854746,
+            0.007778448128358027,
+            -0.01706991160178345,
+            0.007329056462125937,
+            0.009390327499494826,
+            -0.012870674959022891,
+            -0.0017571189976160295,
+            0.016462608870101713,
+            0.00209846661139665,
+            0.004942714850185376,
+            -0.011305050739130339,
+            0.004252776146827917,
+            0.015841821944109317,
+            0.018727567181082646,
+            -0.008399642221255733,
+            0.021448055799259867,
+            -0.020972459569228434,
+            0.027243639465535856,
+            0.028499400850980086,
+            0.016573144403275928,
+            -0.01847096443459798,
+            -0.0022409373140116146,
+            -0.0041737366139895414,
+            -0.014761716560267328,
+            0.0034546020161337894,
+            0.005125870074458691,
+            -0.01174186608185822,
+            -0.009577921232380895,
+            -0.024331993794416026,
+            0.011075959943215117,
+            0.007918147902741738,
+            -0.0015045185909419564,
+            -0.011791155919533844,
+            0.016019077577552835,
+            0.010141868950981925,
+            -0.010938402325619172,
+            -0.005635351352447256,
+            -0.019967361901734836,
+            0.009750427567651833,
+            0.009128294017175602,
+            -0.00300556760891084,
+            -0.011496103417643132,
+            0.012977890524465716,
+            0.0014080059843484115,
+            -0.01292380658817521,
+            0.010841954636959441,
+            -0.03142011651894961,
+            -0.005510537354374977,
+            -0.011027026734432255,
+            -0.003620482919145167,
+            -0.014415606143566519,
+            0.003608572766363589,
+            0.0005034611321129689,
+            -0.011470488351158329,
+            -0.0035005774696278,
+            -0.016207453351921627,
+            0.004676327282262429,
+            0.015021513761697039,
+            0.001679614645967629,
+            -0.006917375094634678,
+            0.021994274901326064,
+            0.018518524483306403,
+            -0.021903538417919984,
+            -0.020116013379669034,
+            0.004708084983787958,
+            -0.03261690161800251,
+            0.015019348187107588,
+            0.00966124471577622,
+            0.011676937880751252,
+            -0.028768229226408037,
+            0.007114534979415504,
+            0.00900420165933673,
+            0.016398927481511824,
+            0.0050879301160439245,
+            0.009453255486450082,
+            0.01222758288108498,
+            -0.0104657614414275,
+            -0.003164502778525433,
+            -5.410382753651127e-06,
+            -0.005618225986727351,
+            0.015607620507415484,
+            0.0008527697935056332,
+            0.03603666488542239,
+            -0.005277207843139309,
+            -0.0009799879252482288,
+            -0.011735731235560851,
+            0.00812046801749312,
+            -0.011713349784345638,
+            0.006334172347602515,
+            -0.0024643499727789167,
+            0.0001240009282616789,
+            0.002965067818077541,
+            0.009442572075213767,
+            -0.01086095783353607,
+            -0.007595379345224076,
+            0.007444897256325813,
+            -0.01482814340478338,
+            -0.00019984717819268013,
+            -0.009339343730980451,
+            0.0019680804106275156,
+            -0.02241031895876346,
+            0.006629838013824249,
+            0.040935644507527846,
+            0.02140216465404127,
+            -0.005481040712269146,
+            -0.012431659209559418,
+            -0.004680230194268222,
+            -0.007229300034633229,
+            -0.006482401275991039,
+            -0.004794698160419071,
+            -0.022762405726589375,
+            -0.006928553505112732,
+            0.004320321244630314,
+            0.013300138724212552,
+            0.009741018117644266,
+            -0.008842135469729511,
+            0.004531760574809748,
+            -0.02662650543210291,
+            -0.016578081411548973,
+            -0.0060021425889846495,
+            -0.015840130031005147,
+            0.006125301421471048,
+            0.027521841635711787,
+            -0.0063432924325983525,
+            0.01789356321527744,
+            0.005142029893259461,
+            0.013034833065163448,
+            -0.005338869351330258,
+            0.00035250243782976347,
+            -0.006010938630154189,
+            0.0032958656042048292,
+            0.003074861351445954,
+            -0.0015947217852680714,
+            0.004151590621941346,
+            -0.007139285822575166,
+            -0.0026091937868967284,
+            0.0011629380311555433,
+            0.0117841884590861,
+            0.006877419488576261,
+            -0.00016252653621991807,
+            -0.01761285866765538,
+            0.022221104524808997,
+            -0.018933508559881868,
+            -0.008840161102111973,
+            0.011092684029552064,
+            -0.009660487856185668,
+            0.00702597023411946,
+            0.020387386977378873,
+            -0.006854213562767843,
+            -0.009583981919303215,
+            0.013028181087649146,
+            -0.0036507672754997286,
+            0.004464368543493108,
+            0.009867428978560914,
+            0.006036303973414791,
+            0.002482944923100156,
+            0.013244650094892797,
+            0.004530294330688855,
+            -0.0032625482538717137,
+            -0.020238533752799943,
+            -0.005186498720507101,
+            0.019015752262933585,
+            -0.02252458872825473,
+            -0.0027641215988942563,
+            0.0044506144202026456,
+            -0.0030317840657635457,
+            -0.01733962766050954,
+            0.004205861489436576,
+            -0.018770275720867473,
+            -0.0033883519277721433,
+            -0.008244668812709337,
+            0.009948090119054079,
+            -0.023700732627507225,
+            0.009109315916173436,
+            -0.011739192106106307,
+            0.02356509175177699,
+            -0.003925851077921686,
+            0.024734433836185024,
+            0.0015142645677087268,
+            -0.013482933030628261,
+            0.0055323884713736554,
+            -0.0008930661843169134,
+            -0.004477731568313757,
+            0.029338490404300612,
+            -0.005418258701133154,
+            0.002249996261871466,
+            0.013491124734151694,
+            -0.021332541202417246,
+            0.021773687217263173,
+            0.0046662696076723836,
+            0.01628697326539432,
+            -0.00986238951830334,
+            -0.013799209313714441,
+            0.025178874553620743,
+            0.03324555035648896,
+            0.008398796308070151,
+            0.011261509865513953,
+            -0.01226890254177634,
+            0.018966175575083295,
+            -0.0038328997984309597,
+            0.004344837325292706,
+            0.00878668880458693,
+            -0.007200842513742962,
+            -0.011500562922303248,
+            -0.004533014670236755,
+            0.027639894892383943,
+            -0.021838848944317424,
+            -0.008920629621611768,
+            -0.005739181836252042,
+            -0.01593272996094732,
+            -0.038167534267554606,
+            -0.004371701247332816,
+            0.019604405563120094,
+            -0.005974358141737469,
+            -0.0018341151594086288,
+            0.014334266690716505,
+            0.00402620072221496,
+            -0.002852569009185304,
+            -0.008195218707838913,
+            -0.0027944153624309856,
+            -0.0059390492900516225,
+            0.016651206007022145,
+            -0.004961864528562493,
+            -0.007929830585906699,
+            0.01145431100211762,
+            -0.005731957927601062,
+            0.012130694936714224,
+            0.0004848418056781251,
+            8.392048424584131e-05,
+            -0.0015174120840537315,
+            0.00822932723013239,
+            0.018620869354789082,
+            -0.0055803454666417465,
+            0.025682778991091174,
+            0.006956936033581638,
+            -0.003495709501259274,
+            -0.005489431577052735,
+            -0.0044154293119170445,
+            -0.024561016942073285,
+            -0.0020864636079347357,
+            0.006593060072668424,
+            0.024333813203596122,
+            -0.02515989187381735,
+            -0.004953266000463339,
+            0.020957806164031402,
+            -0.0012192724005613,
+            -0.011978722479596745,
+            0.014690559986694138,
+            0.0041346025614047875,
+            0.009881584250499358,
+            0.012176393203318002,
+            0.015764530703333238,
+            -0.004923548157246672,
+            -0.0015192775000207833,
+            0.012766482928380375,
+            0.0005745973040874939,
+            0.00997809510696241,
+            0.016495444272181515,
+            0.010467514633877887,
+            -0.021108281220544853,
+            0.0015078843789125156,
+            0.00522593579521476,
+            -0.017481430635971396,
+            0.0034362821987740755,
+            0.005397819166037461,
+            0.011626527721895694,
+            0.004048632382026344,
+            0.013207335578216437,
+            -0.015768003434275563,
+            -0.005916168437120099,
+            0.008153798523776654,
+            0.004319451296835019,
+            -0.02599474886669325,
+            -0.01485009085106089,
+            -0.0006781964116146299,
+            -0.004444144921332929,
+            0.013556265879295017,
+            0.01285768162554067,
+            0.02033626001285354,
+            0.024637349990579514,
+            -0.013833060730799217,
+            -0.00842819657792762,
+            0.009436072455719896,
+            0.015560357465711765,
+            0.005798181411384387,
+            0.007874051721372273,
+            0.009297440071118126,
+            0.0077320823375042,
+            -0.004073209576661068,
+            -0.002973928843567532,
+            0.020205003097225747,
+            -0.019725382393637726,
+            0.0006746659838642147,
+            -0.018417443693472958,
+            0.007530796896254817,
+            0.01140696723506382,
+            0.01129515042745398,
+            0.002816883643872417,
+            -0.005274757308627299,
+            0.005716917047812153,
+            -0.06254973656832136,
+            -0.009191453047502128,
+            0.003384751551032943,
+            0.027256466708746898,
+            -0.010885336140129142,
+            -0.003415707910562931,
+            0.007309577090125959,
+            0.005511970708769456,
+            0.0005572239548625122,
+            0.003186427278034852,
+            0.019698716902282903,
+            -0.02052581021967601,
+            -0.010793968505805883,
+            0.004219269922138548,
+            0.0033111837923746746,
+            -0.007269521534951044,
+            0.004805737092302134,
+            0.014143978208501908,
+            0.01986656063456352,
+            -0.0024048854610842748,
+            0.01629335842915328,
+            -0.0015498603704589015,
+            -0.014496994444493684,
+            -0.002671064325909516,
+            0.0037875833047555925,
+            -0.022466433279066646,
+            -0.005980897597195858,
+            0.0043262697442044435,
+            -0.011226121648007667,
+            0.006978495397818936,
+            0.008452177557316863,
+            -0.003018545465557218,
+            -0.0008303307644324117,
+            0.017796789945838845,
+            0.006591364879905198,
+            -0.013464486184025026,
+            0.0028668426952326957,
+            -0.023377605222213174,
+            0.004325185794796361,
+            0.001416245546006914,
+            0.007809449561523174,
+            -0.008431335139468173,
+            -0.014774499666285603,
+            -0.005933810712528226,
+            0.017746847020610673,
+            -0.01965188730040385,
+            -0.002406875192894708,
+            0.01683009662984443,
+            -0.0002518220110313279,
+            0.005456812494894182,
+            -0.02011241622618092,
+            -0.0009782974741561381,
+            -0.02933296756078644,
+            -0.01171823509606186,
+            0.001162345317872531,
+            0.00892221099991475,
+            -0.006551308022163148,
+            -0.03934356275100932,
+            0.00427623835776202,
+            -0.013111664387776261,
+            -0.008752857644809231,
+            -0.010214546496134218,
+            -0.00518087722852466,
+            0.015240297918667867,
+            -0.012498100033235827,
+            0.01086490299272505,
+            -0.00917777212602538,
+            0.005895055740265625,
+            0.0090554651297836,
+            0.014212841798768113,
+            -0.005544047915369086,
+            -0.0033589999440643385,
+            -0.007924830667307956,
+            0.0022291797354234005,
+            -0.0032976226056924502,
+            0.009615674261248298,
+            -0.0051932155872883,
+            -0.0065636389655770455,
+            0.02426106498596339,
+            0.011445591067574932,
+            0.013312596099532533,
+            0.0055118624272110965,
+            -0.007220620072349783,
+            -0.01916551187613823,
+            -0.003977616240384473,
+            -0.013204506708246246,
+            -0.006315966975649096,
+            0.01434099854765068,
+            0.012588249778961223,
+            0.006357972880218866,
+            0.0007747002406996095,
+            -0.004794439624135275,
+            0.006975593520375694,
+            0.0262335496080837,
+            -0.016481909971520166,
+            -0.0013084662210086592,
+            -0.01369313900978409,
+            0.01921748474212319,
+            0.03333761672336437,
+            -0.0008918419419301244,
+            -0.007699525186523558,
+            0.013757216065348164,
+            0.00819045303047094,
+            -0.007411380297164911,
+            0.02453611080891859,
+            -0.025045462880463647,
+            -0.015989319900023054,
+            -0.011650886878708265,
+            -0.011407253985176972,
+            -0.008386367959227559,
+            0.016331961716602287,
+            -0.00141225438918211,
+            -0.0018505754298268654,
+            0.009557291617446645,
+            0.03357924917013024,
+            0.006916860901290756,
+            0.007030308953738567,
+            -0.002739345982346728,
+            -0.009376522438071562,
+            0.011482637968411055,
+            -0.007259054391879614,
+            -0.04075661435928125,
+            0.004250195164583401,
+            -0.005160505473755424,
+            0.02405623169060588,
+            -0.02197586561375289,
+            -0.00619175761282745,
+            -0.006736129049831973,
+            -0.028200015908139763,
+            -0.005867722104196303,
+            0.02805071895018145,
+            -0.015402198567775478,
+            0.014773294766214671,
+            -0.009502185346558072,
+            -0.0013419858580438616,
+            0.004443749010136947,
+            0.013008783570355656,
+            -0.02289662952366337,
+            0.0013834888740540108,
+            -0.00048549197205067054,
+            0.027969842651517314,
+            0.010437729090975158,
+            -0.0047705656249623744,
+            0.014202218020412936,
+            0.008240861154198411,
+            -0.018637315941614406,
+            0.007314019317343154,
+            -0.005747966765101792,
+            0.014833896739762147,
+            -0.01314847940891154,
+            0.007762100494499689,
+            0.00705026393499352,
+            -0.01278023669088385,
+            0.03016342404000438,
+            -0.015204129985039077,
+            0.007776946006562982,
+            0.009278778571285689,
+            0.009656723862179986,
+            0.01257686993982689,
+            0.004362439504889069,
+            -0.02898418193036562,
+            0.0002144431781454467,
+            -0.03828100004144633,
+            -0.00833748467467171,
+            -0.002611937107022567,
+            -0.002145329666219461,
+            0.007050630951707937,
+            -0.009383645162261629,
+            -0.033976442057557676,
+            0.0006429978367640709,
+            0.01603027598131823,
+            0.017501067413108286,
+            0.0002196997337451293,
+            0.013075379412332973,
+            -0.006197537539554383,
+            -0.021148120787551232,
+            -0.016986448660412497,
+            -0.010207705210051413,
+            -0.02122194744719993,
+            -0.020550470201191853,
+            -0.00908564461857315,
+            0.012090079127547193,
+            -0.009927390005879807,
+            0.013888458967497627,
+            0.029513250508042935,
+            -0.024529175134769883,
+            0.0011144447737101862,
+            0.021977677894839922,
+            -0.00198308986575084,
+            0.014108189311291326,
+            0.001696775006906642,
+            0.014122064251288995,
+            -0.0004166140484018266,
+            -0.010556016685519523,
+            -0.009729362492661684,
+            0.017487929296614076,
+            -0.0026043257634797124,
+            0.016715798948897194,
+            0.012183487338360088,
+            0.003771721868999568,
+            -0.0030887189945795796,
+            0.010667776697461199,
+            -0.026471157050402344,
+            0.014249996582064706,
+            -0.006690961709923224,
+            -0.0181382691440242,
+            -0.008729104191096461,
+            0.003882731567105777,
+            -0.005609384377642956,
+            -0.007794812949125115,
+            0.012967089730551055,
+            -0.00944374111323783,
+            0.02088525981049842,
+            0.0023400121055514958,
+            -0.012196925240774426,
+            0.003648688547842322,
+            -0.03322041209304968,
+            0.0015191765655963997,
+            -0.018402382776013956,
+            -0.024659631472836927,
+            0.03840173429666435,
+            0.011991452465253198,
+            0.006639702644606365,
+            -0.01675838115087676,
+            0.007953102302963526,
+            0.0004257479565099406,
+            0.017484338819046517,
+            0.011068864013925452,
+            0.006466915839538296,
+            0.008364454017976605,
+            -0.023316137291380285,
+            -0.010898421318668592,
+            0.016204569093730564,
+            0.012099473359791148,
+            -0.01017229369980751,
+            0.019431669546223253,
+            0.017955119269367787,
+            0.007300372071724306,
+            -0.006898619924677706,
+            0.01647927363640854,
+            0.021012103522831126,
+            0.010291016618557653,
+            -0.00047955552103474145,
+            0.0008684297441778696,
+            -0.010534247928997246,
+            0.005037930235889528,
+            0.035417626740307205,
+            0.014712595692093886,
+            0.010650306363963653,
+            -0.03626584475133921,
+            0.021764456453493054,
+            -0.011961521673661574,
+            0.009092981192874177,
+            0.006089104887183293,
+            0.017982801101846038,
+            0.006916969397519662,
+            0.014196254958707752,
+            0.011745069030657836,
+            -0.006689305450954901,
+            0.025205280281804708,
+            -0.001601414806100461,
+            -0.011992525142592405,
+            0.003386930324498371,
+            -0.01859193251438437,
+            0.005303261319773131,
+            -0.0041710536965100615,
+            -0.004459393120257307,
+            0.0032461828144552565,
+            -0.0018650956576455287,
+            0.015678225251799887,
+            0.003894142628084471,
+            -0.02355212701026675,
+            -0.0014873755115206368,
+            -0.024284295126348117,
+            0.0027260592623277375,
+            -0.022185571998195344,
+            0.00022883605560765203,
+            -0.005557568353382136,
+            -0.009749382813201403,
+            -0.025385857431333838,
+            -0.009853073898348951,
+            -0.0035843255959309307,
+            0.007661137888845632,
+            -0.010521037433807653,
+            -0.004246698084422667,
+            -0.0019920499321964815,
+            -0.019465113363548715,
+            0.009014255846983305,
+            0.027122533756959888,
+            0.016848221859975445,
+            0.027170883580103264,
+            -0.00019554112591107156,
+            0.013614675849485926,
+            -0.002066822813103284,
+            -0.01603063005944364,
+            0.008824663429777616,
+            0.0075653522381218495,
+            -0.00531780740880296,
+            0.014643348459785112,
+            0.0072634484937344704,
+            -0.012800149015319017,
+            0.009859503660384916,
+            -0.007161712176469609,
+            -0.006437857349742842,
+            -0.009764856421740622,
+            -0.02058868349990357,
+            -0.014866137971603676,
+            0.0012450207572602557,
+            -0.002123265406747006,
+            0.006000473862248991,
+            -0.004054016790850226,
+            0.009584879012937315,
+            -0.00608816807208866,
+            -0.00925801057439376,
+            -0.013962683072792106,
+            -0.008924356118158748,
+            -0.004210064145949253,
+            -0.004381709838058668,
+            -0.011732699976353612,
+            -0.005479523097469261,
+            0.0010680258077037758,
+            -0.0019401640391035755,
+            -0.01397652201595301,
+            0.011650839243603377,
+            0.0058731838291845645,
+            -0.002462743134993977,
+            0.02219008688925881,
+            0.01097316136828334,
+            0.0018083001685932406,
+            0.01366095798812196,
+            0.0003942601242216818,
+            -0.0064063532958756175,
+            -0.03343312148366622,
+            -0.0005083355836126291,
+            -0.02815586367527057,
+            0.025770078162165123,
+            -0.0069114949102666145,
+            -0.010322087105114614,
+            -0.002519047562374146,
+            -0.006063669644021698,
+            0.005873609611264377,
+            -0.002722940652283699,
+            0.0021229525800901114,
+            -0.023673778230768945,
+            0.0028629715379660113,
+            -0.003953653437464966,
+            -0.0036366957271968823,
+            0.007469664864329332,
+            0.0346520690389124,
+            -0.004054282838167587,
+            -0.0034765253405897676,
+            0.005159466707114532,
+            0.011642913651743502,
+            0.027845571220425726,
+            -0.004445196354520585,
+            -0.0032038508043339413,
+            -0.02233838584152591,
+            0.0054112576531374175,
+            -0.00019544838850471866,
+            -0.004733192299951088,
+            0.016664217348411003,
+            -0.016719175418727594,
+            -0.01631658832047864,
+            -0.00545969677717507,
+            -0.014570848053102338,
+            -0.00022620258237138677,
+            0.014454101124666536,
+            0.011056312437637014,
+            -0.004035526363810078,
+            -0.012459378363360557,
+            -0.0022595493558020706,
+            -0.030370952502579162,
+            0.012766010284884204,
+            0.1141689619126009,
+            -0.00766197047210299,
+            -0.00896655658602649,
+            -0.009481112393590025,
+            0.001435269335646347,
+            0.001338755472877713,
+            -0.011739143028767118,
+            0.07166096084053437,
+            -0.0077855767070652395,
+            0.0002139580430342728,
+            -0.011312198831181402,
+            0.001844005266872301,
+            -0.001825412693163496,
+            0.01371932421024605,
+            -0.005025814503687022,
+            -0.018213360991353643,
+            -0.0033612481616769416,
+            -0.02941403830641238,
+            -0.0041137865872933515,
+            -0.005683630256743215,
+            -0.004288198570156578,
+            -0.014982999363445122,
+            0.0067149665791261335,
+            -0.019661973497372313,
+            -0.012688556129040908,
+            0.001124391650175233,
+            -0.0033623453063430534,
+            0.00996944383866679,
+            -0.020496336814603385,
+            -0.0044354321069609536,
+            0.0019875840782527733,
+            0.009706675240563883,
+            0.0011600364428219214,
+            -0.012755178230821602,
+            -0.0026514442584739412,
+            0.004056350166031111,
+            0.01909951327913368,
+            0.014360281326789544,
+            0.003732344102045803,
+            0.008623457695718206,
+            0.014461868085333023,
+            -0.001692149528228451,
+            0.01948035376572433,
+            0.01172550239477637,
+            0.0031966531144076173,
+            0.0004779413449255626,
+            0.0030912562273611517,
+            -0.004597633235071031,
+            0.01582547400214653,
+            -0.014811177637844808,
+            -0.02411202772171917,
+            -0.0071272238837422185,
+            0.018015541137524113,
+            -0.013081287852028074,
+            1.8837689271256874e-05,
+            0.004062803284646432,
+            0.027569726868200254,
+            0.0010564850664255057,
+            -0.01179755720247972,
+            0.0028876134800690334,
+            -0.01638215925131053,
+            0.022944183106912663,
+            0.0026043713064243836,
+            0.03279173899194346,
+            -0.005103903273968361,
+            0.00578213669463383,
+            -0.020744194169889544,
+            -0.007588793365216579,
+            -0.00819862611489168,
+            0.0017401442433921013,
+            0.010993500174794084,
+            0.005411238350013134,
+            0.01620925462604661,
+            -0.00641615069699833,
+            -0.020751730783437995,
+            0.014776878596707538,
+            -0.009552193304486048,
+            -0.03393319554458277,
+            0.008881987295131347,
+            -0.029102297123590998,
+            -0.0045267103602711,
+            0.00368971742585232,
+            -0.014269341759285646,
+            -0.016264809810970175,
+            0.012837302095591106,
+            -0.0011890334180728564,
+            -0.008937992768323412,
+            0.0014444613725248846,
+            -0.017110187133760626,
+            -0.02385668037818897,
+            0.006017405364919834,
+            0.00676670672597817,
+            -0.0077994500447595,
+            0.018312452858615755,
+            -0.027015660037024585,
+            0.006123912473734307,
+            -0.012973676426738582,
+            0.009041974762537394,
+            -0.01580183887345168,
+            0.01163516332546347,
+            0.010887662470499361,
+            -0.001503467664709027,
+            -0.0076234124970223326,
+            0.01231202541338883,
+            0.004852720134894811,
+            0.033760365210093034,
+            0.028658326381800395,
+            0.000995999783627898,
+            -0.0025605367058226737,
+            -0.0020356794893366384,
+            0.011667525096213252,
+            -0.014049700287539817,
+            0.011218246246280464,
+            0.02658113044439717,
+            0.017568631325838108,
+            -0.019133062654008885,
+            0.018050678772708717,
+            -0.010356118256625172,
+            0.012872052420999813,
+            -0.025620154415048252,
+            -0.005135768485044442,
+            0.004936490920616527,
+            0.005163008729256001,
+            0.007425278892536464,
+            0.001410212306439329,
+            0.021891387289347516,
+            0.014999616600766989,
+            0.00027613811156650413,
+            0.00558422822386443,
+            0.01477009526981092,
+            0.010497782599293994,
+            -0.0005567894749233204,
+            -0.00467776506604011,
+            -0.016998035358377343,
+            0.00957271797745408,
+            -0.006065116857898469,
+            0.005138070401846389,
+            0.01698914855850691,
+            -0.01230257809343985,
+            0.011109844802663414,
+            0.007518553660463051,
+            -0.015504773927595823,
+            -0.017169933945839305,
+            0.005552829003043926,
+            -0.02252633790984703,
+            -0.01306722099657118,
+            0.02715689072730398,
+            -0.010513941399990838,
+            -0.02790582732974509,
+            -0.0009847768069205477,
+            0.009797865241108237,
+            -0.002772952073769185,
+            -0.0027968741526000576,
+            0.0037541733651479613,
+            0.010801137722284016,
+            -0.014658156970099418,
+            0.002075415233345957,
+            0.004735264309137368,
+            0.01930804315461151,
+            -0.0014151522239496115,
+            -0.015046118414100617,
+            -0.002288495894227231,
+            0.021592829677937263,
+            0.017174609713578173,
+            0.00265560378681856,
+            -0.016499825201963983,
+            -3.965296928410772e-05,
+            -0.0074142747014567635,
+            0.00021360696629503726,
+            0.00027431174904249525,
+            -0.0056391089282546054,
+            -0.018931681700770225,
+            -0.003888248121485032,
+            0.011181675419027736,
+            -0.00247252920175839,
+            0.01977860312223846,
+            -0.0011527575128223392,
+            -0.015763587433200915,
+            0.001960734212846797,
+            -0.005280860341480037,
+            -0.010372829570478649,
+            0.01351900743232294,
+            -0.009359942374603997,
+            0.006788746813046239,
+            -0.003212015225284119,
+            0.004611143085183367,
+            -0.0003308099698088422,
+            -0.017017265064185255,
+            0.02924142709831441,
+            -0.0018238549856433466,
+            -0.004769274041546295,
+            -0.0014058363976586562,
+            -0.033393331166741945,
+            0.015017165991303838,
+            -0.014574579250749552,
+            0.006745699409061803,
+            0.002705853802154516,
+            0.01272741296341735,
+            0.003253095246601883,
+            -0.008898616461779179,
+            0.0027485606928249326,
+            0.017040360927723885,
+            0.013756402798365262,
+            -0.007292864293090161,
+            0.008694855886257706,
+            -0.0028934389922887676,
+            -0.0037483208383542015,
+            0.01363963339139356,
+            0.004170611039133969,
+            -0.006282657671540625,
+            -0.0009784372731140945
+          ],
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "Mixture-of-Experts (MoE)",
+            "fine-grained expert segmentation",
+            "shared expert isolation"
+          ],
+          "title": "Deepseekmoe: Towards Ultimate Expert Specialization In Mixture-Of-Experts Language Models"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "767dcc8f-1e79-439d-92f3-8f20092ad68e",
+        "properties": {
+          "page_content": "2. Preliminaries: Mixture-Of-Experts For Transformers\n\nWe first introduce a generic MoE architecture commonly used in Transformer language models. A standard Transformer language model is constructed by stacking layers of standard Transformer blocks, where each block can be represented as follows:\n\n$$\\mathbf{u}{1:T}^{l}=\\text{Self-Att}\\left(\\mathbf{h}{1:T}^{l-1}\\right)+\\mathbf{h}{1:T}^{l-1},\\tag{1}$$ $$\\mathbf{h}{t}^{l}=\\text{FFN}\\left(\\mathbf{u}{t}^{l}\\right)+\\mathbf{u}{t}^{l},\\tag{2}$$\n\nwhere denotes the sequence length, Self-Att(·) denotes the self-attention module, FFN(·) denotes the Feed-Forward Network (FFN), u 1: ∈ R× are the hidden states of all tokens after the -th attention module, and h\n\n∈ Ris the output hidden state of the -th token after the -th Transformer block. For brevity, we omit the layer normalization in the above formulations.\n\nA typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022). An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. Then, each token will be assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts. If the -th FFN is substituted with an MoE layer, the computation for its output hidden state h\n\nis expressed as: where denotes the total number of experts, FFN(·) is the -th expert FFN, , denotes the gate value for the -th expert, , denotes the token-to-expert affinity, Topk(·, ) denotes the set comprising highest affinity scores among those calculated for the -th token and all experts, and e\n\nis the centroid of the -th expert in the -th layer. Note that ,is sparse, indicating that only out of gate values are nonzero. This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only experts. Also, in the above formulations, we omit the layer normalization operation for brevity.\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{N}\\left(g_{i,t}\\operatorname{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\operatorname{Topk}({s_{j,t}|1\\leqslant j\\leqslant N},K),\\ 0,&\\text{otherwise},\\end{cases}$$ $$s_{i,t}=\\operatorname{Softmax}{i}\\left(\\mathbf{u}{t}^{l}\\,\\mathbf{e}_{i}^{l}\\right),$$\n\n$$(3)$$\n\n(4) $\\binom{4}{5}$ .\n\n",
+          "title": "Preliminaries: Mixture-Of-Experts For Transformers",
+          "embedding": [
+            0.0012184951920062304,
+            -0.02724841982126236,
+            -0.02786005660891533,
+            -0.019281849265098572,
+            0.028028255328536034,
+            0.023196324706077576,
+            0.0037118708714842796,
+            0.03721809759736061,
+            -0.0036163025069981813,
+            0.03367060422897339,
+            -0.006009331438690424,
+            0.0015032885130494833,
+            -0.013440718874335289,
+            0.0020107559394091368,
+            -0.02350214309990406,
+            0.012936118058860302,
+            0.0041056121699512005,
+            0.036545298993587494,
+            0.003880070988088846,
+            -0.017630429938435555,
+            0.0016657545929774642,
+            -0.0020566286984831095,
+            -0.0359642431139946,
+            0.003516911529004574,
+            0.014763383194804192,
+            -0.0010483836522325873,
+            0.04584217816591263,
+            -0.011781653389334679,
+            -0.016330702230334282,
+            0.026667363941669464,
+            0.02402203530073166,
+            -0.003767300397157669,
+            0.002329953946173191,
+            -0.03529144451022148,
+            -0.05529196560382843,
+            -0.002939679427072406,
+            0.046514976769685745,
+            0.04406842961907387,
+            -0.03510795161128044,
+            0.027737729251384735,
+            0.019327722489833832,
+            0.008410005830228329,
+            -0.016453029587864876,
+            -0.02949618361890316,
+            -0.015283274464309216,
+            -0.009480370208621025,
+            -0.03651471808552742,
+            0.004227939527481794,
+            0.014755737967789173,
+            -0.027324873954057693,
+            -0.015206819400191307,
+            -0.008906960487365723,
+            0.009029287844896317,
+            0.025061817839741707,
+            -0.05318181961774826,
+            0.03244733065366745,
+            0.0012739247176796198,
+            0.0359642431139946,
+            0.053120654076337814,
+            0.006900027394294739,
+            0.025627581402659416,
+            -0.04003162682056427,
+            -0.020000522956252098,
+            -0.0015720976516604424,
+            0.040092792361974716,
+            0.0058067268691957,
+            0.019817031919956207,
+            -0.02495478093624115,
+            0.0116134537383914,
+            0.033640023320913315,
+            -0.03782973438501358,
+            0.008088896982371807,
+            -0.015275629237294197,
+            -0.018976030871272087,
+            -0.004931321833282709,
+            0.008203578181564808,
+            0.015994302928447723,
+            0.029465602710843086,
+            0.035230278968811035,
+            0.004912207834422588,
+            -0.02949618361890316,
+            0.027080219238996506,
+            -0.0006469970685429871,
+            0.021086178719997406,
+            0.028288202360272408,
+            -0.03168278560042381,
+            -0.01886899583041668,
+            -0.051530398428440094,
+            0.005971104372292757,
+            0.02536763623356819,
+            -0.023884417489171028,
+            0.01834910362958908,
+            -0.003622036660090089,
+            0.015084492042660713,
+            0.0006288390723057091,
+            0.038043808192014694,
+            0.055597785860300064,
+            -0.0010588961886242032,
+            -0.02565816417336464,
+            -0.011582871899008751,
+            0.010038488544523716,
+            0.010871844366192818,
+            0.005653817672282457,
+            0.023624470457434654,
+            0.017385775223374367,
+            -0.011712844483554363,
+            -0.006502463482320309,
+            0.042111191898584366,
+            0.034435153007507324,
+            0.08899315446615219,
+            0.007599587086588144,
+            0.0032454978208988905,
+            0.0005939566763117909,
+            0.043517958372831345,
+            -0.01177400816231966,
+            -0.012309190817177296,
+            -0.04550577700138092,
+            -0.057188041508197784,
+            0.003386938711628318,
+            -0.030214857310056686,
+            0.021223796531558037,
+            -0.003618213813751936,
+            0.009954389184713364,
+            -0.01877724938094616,
+            -0.05923702195286751,
+            0.037890899926424026,
+            -0.0064260088838636875,
+            0.012798500247299671,
+            -0.011085917241871357,
+            0.005256253760308027,
+            -0.001563496538437903,
+            0.020765069872140884,
+            -0.022125961259007454,
+            -0.03244733065366745,
+            0.036300644278526306,
+            0.02734016440808773,
+            0.023578599095344543,
+            0.020581578835844994,
+            0.0029587929602712393,
+            0.0038514004554599524,
+            0.016131920740008354,
+            -0.013822991400957108,
+            -0.0076875099912285805,
+            0.0016886909725144506,
+            -0.009725024923682213,
+            0.003293281886726618,
+            0.031147602945566177,
+            0.0689314678311348,
+            -0.0224776528775692,
+            0.011712844483554363,
+            -0.0219577606767416,
+            0.020520415157079697,
+            -0.02931269258260727,
+            0.01315783616155386,
+            -0.00795892346650362,
+            0.01703408546745777,
+            -0.007068227510899305,
+            0.009182197041809559,
+            -0.016208374872803688,
+            0.0015692306915298104,
+            -0.003788325469940901,
+            -0.030964111909270287,
+            0.0006613322766497731,
+            0.025168854743242264,
+            -0.009121033363044262,
+            0.018318522721529007,
+            0.04755476117134094,
+            0.024220816791057587,
+            -0.04868628829717636,
+            0.024419598281383514,
+            0.0011879133526235819,
+            0.03076533041894436,
+            0.040245700627565384,
+            0.0068847364746034145,
+            -0.015573801472783089,
+            0.012347417883574963,
+            0.007928341627120972,
+            0.021835433319211006,
+            0.011468189768493176,
+            0.025031235069036484,
+            -0.03929766267538071,
+            -0.00842529721558094,
+            0.032600242644548416,
+            0.007763964589685202,
+            0.03489387780427933,
+            0.018333813175559044,
+            0.037340424954891205,
+            0.019602959975600243,
+            0.010619543492794037,
+            0.015260337851941586,
+            0.03189685940742493,
+            0.011980435810983181,
+            0.03969522938132286,
+            -0.009633279405534267,
+            0.0167129747569561,
+            -0.03156045824289322,
+            -0.003220649901777506,
+            0.020994432270526886,
+            0.01137644425034523,
+            0.02050512284040451,
+            -0.0067088911309838295,
+            -0.01113943476229906,
+            0.003945057280361652,
+            -0.013287809677422047,
+            0.004407607484608889,
+            0.010000261478126049,
+            -0.0019381240708753467,
+            0.020902687683701515,
+            0.00837942399084568,
+            -0.04324272274971008,
+            0.02116263285279274,
+            -0.03422107920050621,
+            0.012278608977794647,
+            -0.009029287844896317,
+            0.006785345729440451,
+            0.0038552230689674616,
+            -0.017171703279018402,
+            0.0024656609166413546,
+            0.029664384201169014,
+            -0.017339903861284256,
+            -0.0012356974184513092,
+            0.00038083948311395943,
+            0.013058445416390896,
+            -0.0007425653166137636,
+            -0.004059739410877228,
+            -0.013196064159274101,
+            -0.017446938902139664,
+            -0.014763383194804192,
+            -0.0004931321600452065,
+            -0.006059026811271906,
+            -0.018456140533089638,
+            0.041407812386751175,
+            0.011903980746865273,
+            0.007400805130600929,
+            0.01877724938094616,
+            -0.010329016484320164,
+            0.021651942282915115,
+            0.008371778763830662,
+            0.028961002826690674,
+            -0.004357912112027407,
+            -0.02227886952459812,
+            0.02552054449915886,
+            -0.044404830783605576,
+            -0.03917533531785011,
+            0.025734618306159973,
+            0.026009853929281235,
+            -0.027798892930150032,
+            -0.0033391546458005905,
+            -0.011743426322937012,
+            0.0074275643564760685,
+            0.012714399956166744,
+            0.003622036660090089,
+            0.0030142224859446287,
+            0.005325062666088343,
+            -0.007278477773070335,
+            0.047585342079401016,
+            0.018456140533089638,
+            -0.017905667424201965,
+            -0.019924068823456764,
+            -0.01984761469066143,
+            0.028012964874505997,
+            0.018884286284446716,
+            0.001693469355814159,
+            0.017875084653496742,
+            0.02238590642809868,
+            -0.0010416939621791244,
+            0.05654582008719444,
+            -0.018945449963212013,
+            -0.01573435589671135,
+            -0.006972659379243851,
+            -0.002561229048296809,
+            -0.04853338003158569,
+            -0.02200363390147686,
+            0.007935987785458565,
+            0.05214203521609306,
+            -0.02327278070151806,
+            -0.0057073356583714485,
+            -0.0008553358493372798,
+            0.043426211923360825,
+            0.023869125172495842,
+            0.03519969806075096,
+            -2.6818839614861645e-05,
+            -0.004858689848333597,
+            0.020719196647405624,
+            0.03149929642677307,
+            -0.002643417799845338,
+            -0.016468320041894913,
+            -0.005168330855667591,
+            -0.03773799166083336,
+            -0.031346384435892105,
+            -0.017431648448109627,
+            -0.009686797857284546,
+            -0.03474096953868866,
+            0.002180867362767458,
+            0.017920957878232002,
+            0.02163665182888508,
+            -0.003543670754879713,
+            0.008333551697432995,
+            0.03177453204989433,
+            -0.02597927302122116,
+            -0.05841131508350372,
+            0.0032875477336347103,
+            0.004744007717818022,
+            0.0005552515503950417,
+            -0.04905327036976814,
+            0.0002306778624188155,
+            0.012561490759253502,
+            0.017446938902139664,
+            -0.02107088826596737,
+            -0.0047975261695683,
+            0.03090294823050499,
+            -0.007175263948738575,
+            -0.0014297009911388159,
+            -0.01163639035075903,
+            -0.03029131144285202,
+            -0.0015510726952925324,
+            0.011689907871186733,
+            -0.008494106121361256,
+            -0.026957891881465912,
+            -0.028823383152484894,
+            -0.007851887494325638,
+            0.009801479987800121,
+            -0.027645982801914215,
+            -0.015497347339987755,
+            0.006441299803555012,
+            0.012737336568534374,
+            0.021896596997976303,
+            0.02967967465519905,
+            -0.015520283952355385,
+            0.017064666375517845,
+            -0.0179821215569973,
+            -0.040704429149627686,
+            -0.028853965923190117,
+            0.0022038037423044443,
+            -0.008348842151463032,
+            -0.032508496195077896,
+            -0.004514643922448158,
+            0.0007502107764594257,
+            -0.01689646579325199,
+            -0.03498562425374985,
+            -0.023655053228139877,
+            -0.00943449791520834,
+            -0.014289364218711853,
+            0.011062980629503727,
+            -0.0020470719318836927,
+            -0.007836596108973026,
+            -0.01605546660721302,
+            0.011322925798594952,
+            0.007974214851856232,
+            0.010092006996273994,
+            0.0009609387489035726,
+            -0.0016925137024372816,
+            0.0062272269278764725,
+            -0.0016389954835176468,
+            -0.024465471506118774,
+            -0.03431282564997673,
+            -0.00899106077849865,
+            -0.015856683254241943,
+            0.007660750765353441,
+            0.0349244624376297,
+            -0.007974214851856232,
+            -0.03718751668930054,
+            -0.021988343447446823,
+            -0.0025115336757153273,
+            -0.0648946613073349,
+            -0.030214857310056686,
+            -0.0349244624376297,
+            -0.030092529952526093,
+            -0.0007473436999134719,
+            -0.0029301224276423454,
+            -0.007836596108973026,
+            0.024832453578710556,
+            -0.0006197600741870701,
+            -0.01240858156234026,
+            0.014778673648834229,
+            0.01587197557091713,
+            0.021560197696089745,
+            -0.011873398907482624,
+            -0.01793624833226204,
+            0.044404830783605576,
+            -0.022951669991016388,
+            -0.001025447272695601,
+            0.01998523250222206,
+            0.003316218266263604,
+            0.0026204814203083515,
+            0.021468451246619225,
+            0.0033601797185838223,
+            -0.007030000444501638,
+            -0.04639264941215515,
+            0.040704429149627686,
+            -0.00032756017753854394,
+            -0.02893042005598545,
+            0.009212778881192207,
+            0.007259364239871502,
+            0.011621098965406418,
+            0.02833407372236252,
+            -0.034343406558036804,
+            -0.014801610261201859,
+            -0.012018662877380848,
+            0.0012089383089914918,
+            -0.021713105961680412,
+            0.012125699780881405,
+            -0.015436183661222458,
+            -0.013134900480508804,
+            -0.023150453343987465,
+            -0.013815346173942089,
+            0.0033468001056462526,
+            0.011720489710569382,
+            0.03247791528701782,
+            -0.018379686400294304,
+            0.0033028386533260345,
+            -0.017676303163170815,
+            -0.0368817001581192,
+            -0.03029131144285202,
+            0.019113650545477867,
+            0.006926786620169878,
+            -0.02734016440808773,
+            0.024939490482211113,
+            -0.003333420492708683,
+            0.007385514210909605,
+            0.02293637953698635,
+            0.038563698530197144,
+            0.033028386533260345,
+            0.02935856580734253,
+            0.0001604352000867948,
+            0.02902216650545597,
+            -0.010665416717529297,
+            -0.010191397741436958,
+            -0.00784424226731062,
+            -0.0010264030424878001,
+            0.02425139956176281,
+            -0.007282300386577845,
+            0.05874771252274513,
+            -0.021789560094475746,
+            -0.009189843200147152,
+            -0.02541350945830345,
+            -0.006376313511282206,
+            0.01276027224957943,
+            -0.0184408500790596,
+            0.002058540005236864,
+            0.029832584783434868,
+            0.02082623355090618,
+            -0.01839497685432434,
+            0.015795519575476646,
+            -0.02223299816250801,
+            -0.02382325381040573,
+            -0.02786005660891533,
+            -0.0015548954252153635,
+            -0.011903980746865273,
+            -0.025153562426567078,
+            -0.03001607581973076,
+            -0.004839576315134764,
+            -0.09296879172325134,
+            -0.004101789090782404,
+            0.04321213811635971,
+            -0.005795258563011885,
+            0.032325003296136856,
+            -0.008937542326748371,
+            0.004151484929025173,
+            -0.03935882821679115,
+            0.00418971199542284,
+            0.03605598956346512,
+            0.039634063839912415,
+            0.0006541646434925497,
+            -0.005692044738680124,
+            -0.0009050313383340836,
+            -0.04675963148474693,
+            0.020428668707609177,
+            -0.005944345146417618,
+            -0.004289103206247091,
+            0.0002034409117186442,
+            -0.030964111909270287,
+            0.02293637953698635,
+            0.02620863728225231,
+            0.03749333694577217,
+            0.03908359259366989,
+            -0.017049375921487808,
+            -0.06532280892133713,
+            -0.0015376930823549628,
+            -0.000322303909342736,
+            0.025811072438955307,
+            0.009556825272738934,
+            -0.004442012403160334,
+            -0.009801479987800121,
+            -0.005252431146800518,
+            -0.01998523250222206,
+            0.016040174290537834,
+            0.01282908208668232,
+            0.013356618583202362,
+            0.0304289311170578,
+            -0.011491126380860806,
+            0.002372003858909011,
+            0.024312563240528107,
+            0.031071148812770844,
+            0.0031212589237838984,
+            -0.050582364201545715,
+            -0.017125830054283142,
+            -0.009319815784692764,
+            0.04883919656276703,
+            0.011361153796315193,
+            -0.021850723773241043,
+            -0.011116499081254005,
+            0.02256939746439457,
+            0.026147473603487015,
+            0.0020967673044651747,
+            -0.018853703513741493,
+            -0.015482055954635143,
+            0.011177662760019302,
+            -0.018700795248150826,
+            0.007309059612452984,
+            -0.0019916424062103033,
+            -0.010497216135263443,
+            -0.002899540588259697,
+            -0.0003841843572445214,
+            -0.018043285235762596,
+            0.01909835822880268,
+            -0.00835648737847805,
+            0.04859454184770584,
+            -0.01074187085032463,
+            0.02055099606513977,
+            -0.003369736485183239,
+            -0.0021827789023518562,
+            -0.0023758267052471638,
+            0.05113283544778824,
+            -0.00784424226731062,
+            0.01582610234618187,
+            0.04097966477274895,
+            -0.002408319851383567,
+            0.04657614231109619,
+            -0.011452899314463139,
+            -0.033120132982730865,
+            -0.0026663541793823242,
+            0.012225090526044369,
+            0.006150772329419851,
+            0.004617857746779919,
+            -0.022829342633485794,
+            -0.00800479669123888,
+            0.004526112228631973,
+            -0.029236238449811935,
+            -0.01591784693300724,
+            -0.013089027255773544,
+            -0.012125699780881405,
+            0.006842686794698238,
+            0.004338798578828573,
+            0.009212778881192207,
+            -0.021009724587202072,
+            0.044496577233076096,
+            -0.028395237401127815,
+            0.02266114391386509,
+            0.02659090980887413,
+            -0.008287678472697735,
+            -0.0008639369625598192,
+            -0.023945581167936325,
+            0.012852017767727375,
+            -0.009870288893580437,
+            -0.022309452295303345,
+            -0.01694233901798725,
+            0.006475704722106457,
+            0.0011238825973123312,
+            -0.045995086431503296,
+            0.03464922308921814,
+            0.01666710339486599,
+            -0.016300121322274208,
+            0.004560516681522131,
+            -0.0048013487830758095,
+            0.016743557527661324,
+            -0.002679733792319894,
+            0.02411377988755703,
+            0.013861218467354774,
+            0.02574990876019001,
+            -0.010458989068865776,
+            0.018884286284446716,
+            0.003998575732111931,
+            0.022997543215751648,
+            0.015053910203278065,
+            0.03253907710313797,
+            0.05663756653666496,
+            -0.005600299220532179,
+            -0.007989505305886269,
+            -0.012507972307503223,
+            0.019419468939304352,
+            -0.016315411776304245,
+            -0.00784424226731062,
+            0.02977142110466957,
+            0.015994302928447723,
+            -0.006575095467269421,
+            -0.0032990160398185253,
+            -0.0159637201577425,
+            0.013165482319891453,
+            -0.017768049612641335,
+            0.006120190490037203,
+            0.035750169306993484,
+            0.003279902273789048,
+            -0.02162136137485504,
+            0.03061242215335369,
+            -0.0040329801850020885,
+            0.034007005393505096,
+            0.00630750460550189,
+            0.02168252505362034,
+            -0.038838937878608704,
+            -0.013517173007130623,
+            0.017661012709140778,
+            0.021040305495262146,
+            -0.009052224457263947,
+            -0.01782921329140663,
+            0.028899839147925377,
+            0.018135031685233116,
+            -0.002549760974943638,
+            -0.01418232824653387,
+            -0.004430544096976519,
+            0.02851756475865841,
+            -0.020673323422670364,
+            0.023532725870609283,
+            0.005581185687333345,
+            0.02781418338418007,
+            -0.0010273586958646774,
+            -0.0013360441662371159,
+            0.004040625412017107,
+            -0.016544776037335396,
+            0.004805171396583319,
+            0.03825788199901581,
+            0.014503437094390392,
+            0.0266826543956995,
+            -4.62012758362107e-05,
+            0.011942208744585514,
+            0.03149929642677307,
+            -0.010657771490514278,
+            -0.02139199711382389,
+            0.0038838936015963554,
+            -0.0033487114124000072,
+            -0.045016467571258545,
+            0.00017799586930777878,
+            -0.007114100269973278,
+            -0.03090294823050499,
+            -0.013945318758487701,
+            0.0004625503206625581,
+            0.01500803790986538,
+            0.008654660545289516,
+            0.03685111552476883,
+            -0.022080088034272194,
+            -0.03180511295795441,
+            0.016774138435721397,
+            0.019281849265098572,
+            0.0334259495139122,
+            -0.008341196924448013,
+            0.005841131322085857,
+            -0.0055200220085680485,
+            0.007026177830994129,
+            0.024939490482211113,
+            0.0031308159232139587,
+            -0.0046255034394562244,
+            0.008585851639509201,
+            0.013899446465075016,
+            0.035750169306993484,
+            -0.024052616208791733,
+            -0.01591784693300724,
+            -0.03263082355260849,
+            -0.032233260571956635,
+            0.029389148578047752,
+            0.003199625061824918,
+            -0.014327592216432095,
+            0.0159637201577425,
+            0.020092269405722618,
+            -0.016590647399425507,
+            0.012890245765447617,
+            -0.014923937618732452,
+            -0.04293690249323845,
+            0.015130365267395973,
+            -0.02200363390147686,
+            0.007351109758019447,
+            -0.003146106842905283,
+            -0.004487885162234306,
+            -0.010504862293601036,
+            0.006452768109738827,
+            0.012248027138411999,
+            -0.03467980772256851,
+            0.012018662877380848,
+            0.03510795161128044,
+            -0.006406895350664854,
+            -0.01221744529902935,
+            -0.01402177382260561,
+            0.011598162353038788,
+            -0.019067777320742607,
+            0.00490456260740757,
+            0.00750401895493269,
+            0.004866335075348616,
+            -0.007018532138317823,
+            -0.003495886456221342,
+            -0.02050512284040451,
+            -0.007557536941021681,
+            0.013456009328365326,
+            -0.011491126380860806,
+            -0.00448406208306551,
+            0.001542471582069993,
+            0.007890114560723305,
+            0.0008448233129456639,
+            -0.008310615085065365,
+            0.013517173007130623,
+            0.019862905144691467,
+            0.016544776037335396,
+            0.0024580154567956924,
+            0.01474809180945158,
+            -0.027141382917761803,
+            -0.00588700408115983,
+            0.03168278560042381,
+            -0.0051989126950502396,
+            0.005764676723629236,
+            -0.011605808511376381,
+            -0.02752365544438362,
+            -0.003889627754688263,
+            -0.0031480181496590376,
+            0.029557347297668457,
+            0.0014134544180706143,
+            0.019740577787160873,
+            0.025015944615006447,
+            -0.0036832003388553858,
+            0.012553845532238483,
+            0.01121588982641697,
+            -0.037340424954891205,
+            0.034159913659095764,
+            -0.00941156130284071,
+            0.014151746407151222,
+            0.03431282564997673,
+            -0.015650255605578423,
+            -0.011743426322937012,
+            -0.002968349726870656,
+            0.03061242215335369,
+            0.04914501681923866,
+            0.013662436977028847,
+            0.005523844622075558,
+            0.04357912018895149,
+            -0.018716085702180862,
+            -0.04461890459060669,
+            0.03056654892861843,
+            0.020398087799549103,
+            -0.007645459845662117,
+            -0.00011969923070864752,
+            0.02659090980887413,
+            -0.017676303163170815,
+            0.003742452710866928,
+            0.04119373857975006,
+            -0.0024121426977217197,
+            0.010535444132983685,
+            -0.01909835822880268,
+            -0.024419598281383514,
+            0.04452715814113617,
+            -0.027263710275292397,
+            0.032508496195077896,
+            -0.015000392682850361,
+            0.007851887494325638,
+            0.04657614231109619,
+            -0.004671376198530197,
+            0.023471562191843987,
+            0.03113231249153614,
+            -0.03287547826766968,
+            0.0006670663715340197,
+            0.018899576738476753,
+            -0.011491126380860806,
+            -0.003715693484991789,
+            0.003880070988088846,
+            0.021667232736945152,
+            0.012064536102116108,
+            0.0036774661857634783,
+            0.021407287567853928,
+            0.0028632248286157846,
+            -0.0029836406465619802,
+            -0.014128809794783592,
+            0.018012704327702522,
+            -0.003092588623985648,
+            -0.015237401239573956,
+            0.014641055837273598,
+            0.00230510625988245,
+            0.022966962307691574,
+            -0.03437398746609688,
+            -0.0035723410546779633,
+            0.0010627189185470343,
+            0.0034404569305479527,
+            -0.044037848711013794,
+            -0.005202735308557749,
+            -0.015176237560808659,
+            -0.023762090131640434,
+            -0.013004926964640617,
+            0.006922964006662369,
+            0.016911758109927177,
+            -0.023471562191843987,
+            -0.007897759787738323,
+            0.02906803786754608,
+            -0.03376235067844391,
+            -0.004052093718200922,
+            0.00697648199275136,
+            0.005879358854144812,
+            -0.019419468939304352,
+            -0.032233260571956635,
+            0.020489832386374474,
+            0.05275367200374603,
+            0.0014975544763728976,
+            -0.0008338329498656094,
+            0.029832584783434868,
+            -0.012377999722957611,
+            -0.027370747178792953,
+            0.029939621686935425,
+            -0.01775275729596615,
+            0.022768178954720497,
+            0.03159103915095329,
+            -0.0449553057551384,
+            -0.013081382028758526,
+            -0.003134638536721468,
+            -0.03052067570388317,
+            0.005351821891963482,
+            -0.014847483485937119,
+            -0.0031289043836295605,
+            -0.01062718965113163,
+            -0.02438901737332344,
+            0.02536763623356819,
+            -0.009289233945310116,
+            0.01784450374543667,
+            -0.021239086985588074,
+            -8.111355418805033e-05,
+            -0.019327722489833832,
+            -0.022401196882128716,
+            0.010092006996273994,
+            -0.011414671316742897,
+            0.01159051712602377,
+            0.012095117941498756,
+            -0.024465471506118774,
+            -0.021009724587202072,
+            0.012270962819457054,
+            0.008708178997039795,
+            0.013066090643405914,
+            0.003167131682857871,
+            0.01639186590909958,
+            -0.010917716659605503,
+            -0.0099696796387434,
+            0.03174395114183426,
+            0.00028455446590669453,
+            0.022905798628926277,
+            -0.017446938902139664,
+            -0.030979404225945473,
+            0.0025574064347893,
+            0.01077245268970728,
+            -0.03847195580601692,
+            -0.004040625412017107,
+            -0.02986316569149494,
+            0.00022434646962210536,
+            -0.034343406558036804,
+            0.007461968809366226,
+            0.015390310436487198,
+            -0.007182909641414881,
+            -0.0007105499389581382,
+            -0.00035431928699836135,
+            0.02266114391386509,
+            0.021132051944732666,
+            0.000617370882537216,
+            0.014541665092110634,
+            -0.005787613335996866,
+            -0.01156758051365614,
+            -0.015673192217946053,
+            0.0003980417677666992,
+            0.021927179768681526,
+            -0.003945057280361652,
+            -0.04709603264927864,
+            -0.016682393848896027,
+            -0.004958080593496561,
+            -0.01769159361720085,
+            0.01261500921100378,
+            0.007599587086588144,
+            0.01102475356310606,
+            0.0032340295147150755,
+            0.009335106238722801,
+            -0.06055204197764397,
+            -0.030000785365700722,
+            0.02204950712621212,
+            -0.018135031685233116,
+            0.023563306778669357,
+            0.010420762002468109,
+            0.023089289665222168,
+            -0.01243151817470789,
+            0.00023462004901375622,
+            0.023379815742373466,
+            -0.005107167176902294,
+            0.018746668472886086,
+            0.0024866857565939426,
+            0.02434314414858818,
+            -0.0259333997964859,
+            0.030321894213557243,
+            -0.009862643666565418,
+            0.019954649731516838,
+            -0.02097914181649685,
+            -0.038838937878608704,
+            -0.033548276871442795,
+            0.018761958926916122,
+            0.00335253425873816,
+            -0.02697318233549595,
+            -0.0017575001111254096,
+            0.0068962047807872295,
+            0.02547467313706875,
+            -0.021560197696089745,
+            0.004575807601213455,
+            -0.0033104841131716967,
+            0.005833486095070839,
+            0.015887266024947166,
+            -0.015948429703712463,
+            -0.01966412365436554,
+            -0.01680472120642662,
+            0.050918761640787125,
+            0.009006352163851261,
+            0.02565816417336464,
+            -0.011468189768493176,
+            0.007091164123266935,
+            0.008868733420968056,
+            -0.020749777555465698,
+            0.005867890547960997,
+            0.02692730911076069,
+            -0.04807465150952339,
+            0.008081250824034214,
+            0.032569658011198044,
+            0.04663730412721634,
+            -0.02536763623356819,
+            0.02458779886364937,
+            0.0029702612664550543,
+            -0.003264611354097724,
+            0.031300511211156845,
+            0.02541350945830345,
+            0.030643003061413765,
+            -0.0167129747569561,
+            -0.009625634178519249,
+            -0.01957237720489502,
+            0.01975586824119091,
+            0.01998523250222206,
+            0.003597188973799348,
+            -0.044588323682546616,
+            0.022034214809536934,
+            -0.010611898265779018,
+            0.013471300713717937,
+            0.01848672144114971,
+            -0.005340353585779667,
+            0.032416749745607376,
+            0.013731245882809162,
+            -0.028762219473719597,
+            0.007549891714006662,
+            0.001983996946364641,
+            -0.0006933476543053985,
+            0.019817031919956207,
+            0.040184538811445236,
+            -0.005653817672282457,
+            -0.02163665182888508,
+            -0.010092006996273994,
+            0.014511083252727985,
+            -0.0045910985209047794,
+            0.022309452295303345,
+            -0.0043540894985198975,
+            0.019725287333130836,
+            0.004334975965321064,
+            -0.006598032079637051,
+            0.009526243433356285,
+            0.03498562425374985,
+            -0.006857977714389563,
+            0.015115073882043362,
+            -0.014251137152314186,
+            0.01821148581802845,
+            0.022814052179455757,
+            -0.02607101760804653,
+            -0.026835564523935318,
+            -0.011903980746865273,
+            -0.024450181052088737,
+            0.01058131642639637,
+            -0.003994752652943134,
+            -0.04143839329481125,
+            -0.03608657047152519,
+            -0.0056767538189888,
+            -0.018517304211854935,
+            -0.03718751668930054,
+            -0.021835433319211006,
+            -0.027508364990353584,
+            0.0008424341212958097,
+            0.004162952769547701,
+            0.011101207695901394,
+            -0.016177793964743614,
+            -0.015459120273590088,
+            0.009656216017901897,
+            0.031101731583476067,
+            -0.027508364990353584,
+            0.043884940445423126,
+            0.012492681853473186,
+            -0.02139199711382389,
+            0.009228070266544819,
+            0.033915258944034576,
+            0.0002752365544438362,
+            -0.0244348905980587,
+            0.019878195598721504,
+            0.0028288201428949833,
+            0.017294030636548996,
+            0.022645851597189903,
+            -0.011277053505182266,
+            0.0030008431058377028,
+            0.004526112228631973,
+            -0.005290658213198185,
+            -0.04269224777817726,
+            -0.0023203971795737743,
+            0.003195802215486765,
+            -0.025015944615006447,
+            0.0007052936707623303,
+            0.005164508242160082,
+            0.026804981753230095,
+            -0.0018005057936534286,
+            0.030031366273760796,
+            0.05813607573509216,
+            -0.020703906193375587,
+            -0.04452715814113617,
+            0.005088053643703461,
+            -0.010137880221009254,
+            0.0021942469757050276,
+            -0.029465602710843086,
+            0.03801322728395462,
+            -0.013868864625692368,
+            0.0033926728647202253,
+            -0.005003953352570534,
+            -0.0036239479668438435,
+            -0.029649093747138977,
+            0.021223796531558037,
+            0.0019161433447152376,
+            -0.006575095467269421,
+            -0.036117151379585266,
+            -0.0338846780359745,
+            0.02472541667521,
+            0.019511213526129723,
+            -0.022814052179455757,
+            -0.021284960210323334,
+            -0.0010426496155560017,
+            -0.011804590001702309,
+            0.026330964639782906,
+            -0.01834910362958908,
+            -0.01675884798169136,
+            0.0116134537383914,
+            0.023578599095344543,
+            0.018104448914527893,
+            -0.015451474115252495,
+            -0.06046029552817345,
+            -0.02191188745200634,
+            0.02125437930226326,
+            -0.00647188164293766,
+            -0.003503531916067004,
+            -0.016774138435721397,
+            0.007286123465746641,
+            -0.019220685586333275,
+            -0.025826362892985344,
+            0.038563698530197144,
+            -0.004239407368004322,
+            0.0227070152759552,
+            0.0004496486217249185,
+            0.0018645365489646792,
+            0.04786057770252228,
+            0.008784633129835129,
+            -0.020612159743905067,
+            -0.010619543492794037,
+            0.01376182772219181,
+            0.041744209825992584,
+            -0.01966412365436554,
+            0.005634704139083624,
+            0.01296669989824295,
+            -0.01102475356310606,
+            -0.013639500364661217,
+            0.0007310971268452704,
+            0.0099696796387434,
+            -0.002119703684002161,
+            0.004755476024001837,
+            -0.020765069872140884,
+            0.015115073882043362,
+            0.02088739722967148,
+            0.012118053622543812,
+            -0.003795970929786563,
+            -0.0037692117039114237,
+            -0.00027021922869607806,
+            0.01418232824653387,
+            0.016269538551568985,
+            0.02411377988755703,
+            0.024786580353975296,
+            -0.001051250728778541,
+            -0.01896074041724205,
+            -0.004182066768407822,
+            -0.003694668412208557,
+            0.001217539538629353,
+            0.02992432937026024,
+            0.032936640083789825,
+            -0.008738760836422443,
+            0.006357199978083372,
+            -0.018792539834976196,
+            0.012844372540712357,
+            0.021973051130771637,
+            -0.018884286284446716,
+            -0.018196195363998413,
+            0.025168854743242264,
+            -0.003528379835188389,
+            0.01441933773458004,
+            0.002433167537674308,
+            0.006364845205098391,
+            0.0206886138767004,
+            0.06813634186983109,
+            -0.015979010611772537,
+            -0.008050668984651566,
+            -0.022018924355506897,
+            0.0031136134639382362,
+            0.025444090366363525,
+            -0.026147473603487015,
+            0.013119609095156193,
+            0.00559265399351716,
+            0.02350214309990406,
+            0.016605939716100693,
+            0.004923676140606403,
+            0.0023280426394194365,
+            0.01214863546192646,
+            0.0025153562892228365,
+            0.016254248097538948,
+            0.0004979106015525758,
+            -0.00819593295454979,
+            0.0035207343753427267,
+            0.01591784693300724,
+            0.015160947106778622,
+            0.027982383966445923,
+            -0.0007554669864475727,
+            0.025734618306159973,
+            -0.03076533041894436,
+            0.022064797580242157,
+            -0.007599587086588144,
+            0.0020031104795634747,
+            -0.01802799478173256,
+            -0.008937542326748371,
+            0.011552290059626102,
+            -0.001057940535247326,
+            -0.0001701114815659821,
+            0.0025669632013887167,
+            -0.007010886911302805,
+            0.03090294823050499,
+            3.2761989132268354e-05,
+            0.011445253156125546,
+            0.008478814736008644,
+            -0.007419918663799763,
+            0.004308216739445925,
+            -0.001004422316327691,
+            0.01773746684193611,
+            -0.029832584783434868,
+            -0.010825971141457558,
+            0.012936118058860302,
+            0.016361285001039505,
+            -0.009059869684278965,
+            0.005130103789269924,
+            -0.023425688967108727,
+            0.0007463880465365946,
+            0.012951409444212914,
+            -0.010321371257305145,
+            0.006773877423256636,
+            -0.024037325754761696,
+            -0.011277053505182266,
+            0.005703513044863939,
+            -0.034526895731687546,
+            -0.006678309291601181,
+            0.008761697448790073,
+            -0.003845666302368045,
+            0.020749777555465698,
+            0.0036832003388553858,
+            0.01116237137466669,
+            -0.022783471271395683,
+            -0.011934562586247921,
+            -0.0022745244204998016,
+            0.0032951931934803724,
+            0.0010808769147843122,
+            0.006987950298935175,
+            -0.01605546660721302,
+            0.028961002826690674,
+            -0.024924200028181076,
+            -0.015252692624926567,
+            -0.004101789090782404,
+            -0.0024637493770569563,
+            0.0008410005830228329,
+            -0.01463341061025858,
+            0.004380848724395037,
+            -0.007710446137934923,
+            0.034007005393505096,
+            0.035322025418281555,
+            -0.005003953352570534,
+            -0.01077245268970728,
+            -0.03217209503054619,
+            0.015619674697518349,
+            -0.018563177436590195,
+            -0.0034939751494675875,
+            -0.0050498261116445065,
+            -0.006815927568823099,
+            0.0005284924409352243,
+            -0.02533705346286297,
+            0.0113535076379776,
+            0.002578431274741888,
+            0.0018893842352554202,
+            0.0011420405935496092,
+            0.011147080920636654,
+            0.015092138200998306,
+            -0.02003110572695732,
+            -0.040184538811445236,
+            -0.024878326803445816,
+            -0.002654885873198509,
+            0.009801479987800121,
+            -0.016254248097538948,
+            -0.008310615085065365,
+            -0.004820462316274643,
+            0.0116134537383914,
+            -0.006716536357998848,
+            0.006942077539861202,
+            0.003516911529004574,
+            0.0025841654278337955,
+            -0.010092006996273994,
+            -0.019358305260539055,
+            -0.010474280454218388,
+            0.01568848453462124,
+            0.020306341350078583,
+            0.007492550648748875,
+            -0.04672905057668686,
+            -0.0023643583990633488,
+            0.002708404092118144,
+            0.007473437115550041,
+            -0.006215759087353945,
+            0.03529144451022148,
+            0.0007846153457649052,
+            2.241374022560194e-05,
+            0.03737100958824158,
+            0.011062980629503727,
+            -0.016743557527661324,
+            0.012309190817177296,
+            -0.015459120273590088,
+            -0.002947324886918068,
+            0.00021455071691889316,
+            0.03168278560042381,
+            0.005290658213198185,
+            0.028074128553271294,
+            -0.0264532919973135,
+            -8.410005830228329e-05,
+            0.003337243339046836,
+            0.02827291004359722,
+            -0.015841392800211906,
+            -0.023762090131640434,
+            0.024006744846701622,
+            0.03929766267538071,
+            -0.007905405946075916,
+            -0.010634834878146648,
+            -0.03330362215638161,
+            -0.025918109342455864,
+            0.047829996794462204,
+            0.025872236117720604,
+            0.004598744213581085,
+            0.02102501504123211,
+            -0.021804852411150932,
+            0.015023328363895416,
+            0.015099783428013325,
+            0.0008868733420968056,
+            -0.030918240547180176,
+            0.03932824730873108,
+            -0.013478945940732956,
+            0.03526085987687111,
+            0.002372003858909011,
+            -0.003767300397157669,
+            0.020948560908436775,
+            0.023884417489171028,
+            -0.029893748462200165,
+            0.009579760953783989,
+            -0.0045643397606909275,
+            -0.013402490876615047,
+            0.015436183661222458,
+            0.007798369042575359,
+            0.02331865206360817,
+            -0.01479396503418684,
+            0.00012184951629023999,
+            -0.0013341327430680394,
+            0.03819671645760536,
+            0.008509396575391293,
+            -0.02350214309990406,
+            0.03263082355260849,
+            -0.012729690410196781,
+            0.00769133260473609,
+            -0.015046264976263046,
+            0.027049636468291283,
+            -0.011292343959212303,
+            -0.004055916331708431,
+            0.008348842151463032,
+            0.01568848453462124,
+            0.01784450374543667,
+            -0.015076846815645695,
+            -0.008318260312080383,
+            -0.03905300796031952,
+            0.01582610234618187,
+            0.012225090526044369,
+            -0.0003877681738231331,
+            0.02448076196014881,
+            -0.0072173140943050385,
+            -0.007851887494325638,
+            -0.0036832003388553858,
+            -0.007324350532144308,
+            -0.012653236277401447,
+            -0.003279902273789048,
+            -0.03510795161128044,
+            -0.03305896744132042,
+            0.011429962702095509,
+            -0.021697815507650375,
+            -0.01587197557091713,
+            0.011651680804789066,
+            -0.02672852762043476,
+            0.02214125171303749,
+            -0.007733382750302553,
+            0.03730984404683113,
+            -0.02289050631225109,
+            0.012683818116784096,
+            0.010374888777732849,
+            -0.004059739410877228,
+            -0.001281570177525282,
+            -0.012706754729151726,
+            -0.004006220959126949,
+            0.0061660632491111755,
+            0.01044369861483574,
+            0.02831878326833248,
+            0.03211093321442604,
+            -0.03458806127309799,
+            0.026346255093812943,
+            -0.007913051173090935,
+            0.010543089359998703,
+            0.00279632699675858,
+            0.001090433681383729,
+            0.03128522261977196,
+            -0.03379293158650398,
+            0.009166906587779522,
+            0.015497347339987755,
+            -0.0015682749217376113,
+            0.022584687918424606,
+            0.008043023757636547,
+            0.0003877681738231331,
+            -0.013853573240339756,
+            0.005068940110504627,
+            0.021269669756293297,
+            -0.0066591957584023476,
+            0.01503097452223301,
+            -0.01439640112221241,
+            0.026881437748670578,
+            0.03507737070322037,
+            -0.009296879172325134,
+            0.02266114391386509,
+            0.008486460894346237,
+            -0.0002391595480730757,
+            0.013249581679701805,
+            -0.007297591306269169,
+            -0.0349244624376297,
+            -0.001130572403781116,
+            -0.02715667337179184,
+            -0.010267852805554867,
+            -0.0009332239278592169,
+            -0.00590994069352746,
+            -0.023578599095344543,
+            -0.0033926728647202253,
+            0.016820011660456657,
+            -0.0007244073203764856,
+            -0.0059787495993077755,
+            -0.011536998674273491,
+            0.020076977089047432,
+            0.015749648213386536,
+            0.025061817839741707,
+            0.020245177671313286,
+            0.0020413377787917852,
+            0.014725156128406525,
+            0.0014421248342841864,
+            0.02438901737332344,
+            -0.006116367876529694,
+            -0.0017288295784965158,
+            -0.009059869684278965,
+            0.00859349686652422,
+            0.010711289010941982,
+            0.008501751348376274,
+            0.011582871899008751,
+            -0.021514324471354485,
+            -0.010902426205575466,
+            -0.012423872016370296,
+            0.004472594242542982,
+            -0.00540534034371376,
+            0.02036750502884388,
+            0.03529144451022148,
+            0.0011860019294545054,
+            -0.005305949132889509,
+            -0.016131920740008354,
+            -0.004671376198530197,
+            0.009747961536049843,
+            -0.032508496195077896,
+            0.0031919796019792557,
+            -0.015933139249682426,
+            0.0009059869917109609,
+            -0.020994432270526886,
+            0.0007736249826848507,
+            0.009579760953783989,
+            0.0030103998724371195,
+            0.016361285001039505,
+            -0.020443959161639214,
+            0.022447070106863976,
+            -0.0005710202967748046,
+            -0.008203578181564808,
+            0.013593628071248531,
+            -0.02093326859176159,
+            0.009304524399340153,
+            0.022737598046660423,
+            0.024985363706946373,
+            -0.016590647399425507,
+            -0.0008175863767974079,
+            0.00418971199542284,
+            0.006326618138700724,
+            0.00769133260473609,
+            0.0038265527691692114,
+            0.013310745358467102,
+            -0.016590647399425507,
+            0.013196064159274101,
+            0.013540109619498253,
+            0.004690489731729031,
+            5.689297177013941e-05,
+            -0.004786057863384485,
+            0.01854788511991501,
+            0.03776857256889343,
+            -0.02827291004359722,
+            -0.017095249146223068,
+            -0.0012796588707715273,
+            -0.009587407112121582,
+            0.000816152838524431,
+            0.015856683254241943,
+            -0.029847875237464905,
+            0.00437702564522624,
+            0.003228295361623168,
+            0.004445835016667843,
+            -0.021988343447446823,
+            0.017339903861284256,
+            0.008371778763830662,
+            -0.008020087145268917,
+            0.03137696906924248,
+            0.023288071155548096,
+            -0.0016514193266630173,
+            0.04088791832327843,
+            0.002578431274741888,
+            0.007087341509759426,
+            -0.026942601427435875,
+            0.031346384435892105,
+            -0.010971235111355782,
+            0.005864067934453487,
+            0.005367112811654806,
+            -0.02631567232310772,
+            -0.007068227510899305,
+            -0.010237270966172218,
+            0.004078852944076061,
+            -0.007595764473080635,
+            0.011842817068099976,
+            0.02552054449915886,
+            0.024465471506118774,
+            0.002127349143847823,
+            -0.019358305260539055,
+            0.0224776528775692,
+            0.018043285235762596,
+            0.013272518292069435,
+            -0.0011640213197097182,
+            -0.004350266885012388,
+            -0.017401067540049553,
+            -0.0032512317411601543,
+            -0.020719196647405624,
+            -0.01680472120642662,
+            -0.012278608977794647,
+            0.02279876172542572,
+            -0.010642480105161667,
+            -0.0004214559739921242,
+            -0.005183621775358915,
+            -0.02579578198492527,
+            -0.004629326052963734,
+            0.007660750765353441,
+            -0.00488162599503994,
+            0.010451343841850758,
+            0.026621490716934204,
+            0.010359598323702812,
+            -0.03269198536872864,
+            0.0068962047807872295,
+            -0.011017107404768467,
+            -0.007351109758019447,
+            0.0008854398038238287,
+            0.0025593177415430546,
+            0.01923597790300846,
+            -0.014663992449641228,
+            -0.022538816556334496,
+            0.020872104912996292,
+            0.005955813452601433,
+            -0.016590647399425507,
+            0.016376575455069542,
+            0.020765069872140884,
+            -0.008845796808600426,
+            0.010986525565385818,
+            -0.003868602681905031,
+            0.0204592514783144,
+            0.02761540189385414,
+            -0.024710126221179962,
+            9.509040683042258e-05,
+            -0.00469431234523654,
+            -0.005095698870718479,
+            0.0060016862116754055,
+            0.010734225623309612,
+            -0.008073605597019196,
+            -0.027447201311588287,
+            -0.0002871826000045985,
+            0.006991772912442684,
+            0.02561229094862938,
+            0.015275629237294197,
+            -0.03229442238807678,
+            -0.008731115609407425,
+            -0.016743557527661324,
+            0.03590308129787445,
+            -0.0404597744345665,
+            0.0001355874555883929,
+            0.0043311528861522675,
+            -0.010038488544523716,
+            -0.018471430987119675,
+            0.012316836044192314,
+            0.026483872905373573,
+            -0.008501751348376274,
+            -0.006555981934070587,
+            0.003174777142703533,
+            -0.014855128712952137,
+            0.027233127504587173,
+            0.033640023320913315,
+            -0.018471430987119675,
+            0.01775275729596615,
+            0.0037118708714842796,
+            0.006953545846045017,
+            -0.003455747850239277,
+            -0.040796175599098206,
+            0.011483481153845787,
+            0.028991583734750748,
+            0.0013828725786879659,
+            -0.004919853527098894,
+            0.00777160981670022,
+            0.009105742909014225,
+            -0.0037367185577750206,
+            0.010023198090493679,
+            -0.015933139249682426,
+            -0.0035666071344166994,
+            -0.003482507076114416,
+            0.01177400816231966,
+            -0.008417651057243347,
+            0.005936699453741312,
+            -0.008272388018667698,
+            -0.003377381945028901,
+            0.007202023174613714,
+            -0.00863937009125948,
+            -0.00014406914124265313,
+            -0.0025593177415430546,
+            0.0030428930185735226,
+            0.013563046231865883,
+            -0.009725024923682213,
+            -0.011644035577774048,
+            -0.03529144451022148,
+            0.016086047515273094,
+            0.028028255328536034,
+            0.0014908646699041128,
+            -0.030168984085321426,
+            0.006716536357998848,
+            -0.014128809794783592,
+            -0.01018375251442194,
+            -0.0060628498904407024,
+            0.020902687683701515,
+            0.03208034858107567,
+            0.020291050896048546,
+            0.01782921329140663,
+            0.007890114560723305,
+            0.019251268357038498,
+            -0.0029951089527457952,
+            0.008226514793932438,
+            -0.0038552230689674616,
+            -0.018716085702180862,
+            0.0015740090748295188,
+            0.017523394897580147,
+            -0.010351953096687794,
+            0.007056759670376778,
+            0.000946125655900687,
+            0.012561490759253502,
+            0.002589899580925703,
+            -0.0024064085446298122,
+            0.005053649190813303,
+            0.015489702112972736,
+            -0.0021827789023518562,
+            0.008853442966938019,
+            0.07370223104953766,
+            -0.01219450868666172,
+            0.010833616368472576,
+            0.008685242384672165,
+            -0.00578379025682807,
+            -0.017309321090579033,
+            -0.005535312928259373,
+            -0.022492943331599236,
+            0.027967091649770737,
+            0.00353984790854156,
+            -0.02116263285279274,
+            -0.004514643922448158,
+            0.00234715617261827,
+            0.016957629472017288,
+            0.015657901763916016,
+            -0.0027905928436666727,
+            -0.0026701767928898335,
+            -0.0023490674793720245,
+            0.017217576503753662,
+            0.0076263463124632835,
+            -0.025597000494599342,
+            -0.00450699869543314,
+            -0.0018196194432675838,
+            0.012324481271207333,
+            0.002180867362767458,
+            -0.004686667118221521,
+            0.0016428182134404778,
+            0.024603089317679405,
+            -0.005455035716295242,
+            0.029572639614343643,
+            -0.014243491925299168,
+            -0.017538685351610184,
+            0.014381109736859798,
+            -0.0012557668378576636,
+            0.021758979186415672,
+            0.00488162599503994,
+            0.018287939950823784,
+            -0.013654791750013828,
+            -0.005581185687333345,
+            0.01322664599865675,
+            -0.012622654438018799,
+            -0.0026625313330441713,
+            0.007935987785458565,
+            -0.00046756764641031623,
+            -0.002867047442123294,
+            0.006269277073442936,
+            -0.014426982961595058,
+            0.005894649773836136,
+            -0.01573435589671135,
+            -0.004537580534815788,
+            -0.007806014735251665,
+            -0.008188287727534771,
+            0.01463341061025858,
+            -0.0013886066153645515,
+            -0.03403758630156517,
+            -0.0024733063764870167,
+            0.034618642181158066,
+            0.003358268178999424,
+            0.00620429078117013,
+            0.01957237720489502,
+            0.005760854110121727,
+            0.014503437094390392,
+            -0.021132051944732666,
+            -0.0002107279869960621,
+            -0.0041056121699512005,
+            0.005554426461458206,
+            -0.006995595991611481,
+            0.0109788803383708,
+            0.005462680943310261,
+            -0.00017082825070247054,
+            -0.0028402884490787983,
+            -0.006330440752208233,
+            0.0032416749745607376,
+            0.008165351115167141,
+            0.008058315142989159,
+            0.01900661364197731,
+            -0.02266114391386509,
+            -0.01081068068742752,
+            0.009725024923682213,
+            0.009839707054197788,
+            0.01315783616155386,
+            0.008585851639509201,
+            0.0199393592774868,
+            -0.003065829398110509,
+            -0.019832322373986244,
+            -0.006127836182713509,
+            -0.0028880725149065256,
+            0.032325003296136856,
+            0.018150322139263153,
+            -0.025015944615006447,
+            -0.008647015318274498,
+            0.0018100626766681671,
+            0.002362447092309594,
+            -0.004247053060680628,
+            0.008708178997039795,
+            0.007874824106693268,
+            -0.02018401399254799,
+            -0.0008080295519903302,
+            -0.007014709524810314,
+            -0.009319815784692764,
+            0.009648570790886879,
+            0.019205395132303238,
+            -0.02223299816250801,
+            0.01435052789747715,
+            0.006819750182330608,
+            -0.02561229094862938,
+            0.024755999445915222,
+            0.008440587669610977,
+            -0.0028039724566042423,
+            -0.006804459262639284,
+            0.006173708941787481,
+            -0.03608657047152519,
+            0.01811973936855793,
+            0.003975639119744301,
+            -0.0035666071344166994,
+            0.0003598144685383886,
+            0.0019954650197178125,
+            -0.001083743991330266,
+            -0.010329016484320164,
+            0.00010416939039714634,
+            0.018425557762384415,
+            -0.008394715376198292,
+            0.01079538930207491,
+            -0.0035818980541080236,
+            0.008111832663416862,
+            -0.0066477274522185326,
+            -0.006953545846045017,
+            -0.004285280127078295,
+            -0.03324246034026146,
+            -0.01542853843420744,
+            -0.012041599489748478,
+            -0.02561229094862938,
+            0.005500908475369215,
+            0.001043605268932879,
+            -0.013234291225671768,
+            0.005669108591973782,
+            0.0013236202066764235,
+            0.0018177081365138292,
+            -0.004545225761830807,
+            0.010894780047237873,
+            -0.0040826755575835705,
+            0.016865884885191917,
+            0.008884024806320667,
+            -0.013662436977028847,
+            0.020168723538517952,
+            0.0009609387489035726,
+            0.002477128989994526,
+            -0.012660881504416466,
+            0.0002587510389275849,
+            0.003929766360670328,
+            0.009923807345330715,
+            -0.01934301294386387,
+            -0.005229494534432888,
+            -0.00601697713136673,
+            -0.011942208744585514,
+            -0.0022707015741616488,
+            -0.027645982801914215,
+            -0.007905405946075916,
+            -0.010466634295880795,
+            -0.0145798921585083,
+            -0.013700664043426514,
+            0.0022879038006067276,
+            -0.0006871357327327132,
+            0.01821148581802845,
+            -0.015130365267395973,
+            0.011108852922916412,
+            0.01280614547431469,
+            -0.0007903494406491518,
+            0.005657640285789967,
+            -0.007611055392771959,
+            -0.013004926964640617,
+            0.01854788511991501,
+            -0.017079956829547882,
+            -0.008394715376198292,
+            0.0206886138767004,
+            0.012225090526044369,
+            0.002939679427072406,
+            -0.014434628188610077,
+            -0.004201180301606655,
+            0.029052747413516045,
+            0.007733382750302553,
+            0.010864198207855225,
+            0.0032856364268809557,
+            0.003398407017812133,
+            0.02756952866911888,
+            0.0013809611555188894,
+            -0.014144101180136204,
+            0.006231050007045269,
+            0.004686667118221521,
+            -0.01102475356310606,
+            -0.013142545707523823,
+            0.017156412824988365,
+            -0.006926786620169878,
+            -0.015887266024947166,
+            0.024098489433526993,
+            0.023700926452875137,
+            0.0112388264387846,
+            -0.011926917359232903,
+            -0.020902687683701515,
+            0.0003727161674760282,
+            0.01001555286347866,
+            0.011812235228717327,
+            -0.01179694477468729,
+            -0.004224116448312998,
+            0.02457250840961933,
+            0.00901399739086628,
+            -0.022355325520038605,
+            0.008723469451069832,
+            0.030780620872974396,
+            0.015604383312165737,
+            -0.04544461518526077,
+            -0.01694233901798725,
+            -0.0007138948421925306,
+            -0.019251268357038498,
+            -0.02036750502884388,
+            -0.023609180003404617,
+            -0.011942208744585514,
+            0.0009485148475505412,
+            0.005615590140223503,
+            -0.0007291857618838549,
+            -0.017951540648937225,
+            0.01556615624576807,
+            0.017951540648937225,
+            0.011697554029524326,
+            -0.007760141976177692,
+            -0.018456140533089638,
+            -0.012530908919870853,
+            -0.01830323040485382,
+            0.01078774407505989,
+            -0.018089158460497856,
+            -0.012890245765447617,
+            0.006712713744491339,
+            -0.006854154635220766,
+            0.013983545824885368,
+            0.01554321963340044,
+            0.03076533041894436,
+            0.016040174290537834,
+            -0.021376706659793854,
+            -0.019373595714569092,
+            -0.005841131322085857,
+            -0.01074187085032463,
+            -0.004170598462224007,
+            -0.0013045065570622683,
+            0.005370935425162315,
+            0.009579760953783989,
+            0.005191267468035221,
+            0.02082623355090618,
+            -0.009610342793166637,
+            0.02776831015944481,
+            -2.6236470148433e-05,
+            -0.007362578064203262,
+            0.004201180301606655,
+            -0.018930159509181976,
+            0.009266297332942486,
+            -0.008302969858050346,
+            0.01120824459940195,
+            0.012263317592442036,
+            0.007599587086588144,
+            0.036392390727996826,
+            -0.011491126380860806,
+            -0.0013226645532995462,
+            -0.028012964874505997,
+            -0.0013417782029137015,
+            -0.003987107425928116,
+            0.021804852411150932,
+            0.005997863132506609,
+            -0.035841915756464005,
+            -0.020566286519169807,
+            -0.012959054671227932,
+            0.019281849265098572,
+            0.0035742525942623615,
+            -0.0053327083587646484,
+            0.020520415157079697,
+            0.025719327852129936,
+            -0.004679021425545216,
+            -0.0018263092497363687,
+            -0.027462491765618324,
+            -0.009686797857284546,
+            0.03287547826766968,
+            -0.005088053643703461,
+            0.001989730866625905,
+            0.0010636745719239116,
+            -0.015115073882043362,
+            -0.017309321090579033,
+            -0.012263317592442036,
+            0.003694668412208557,
+            0.007190554868429899,
+            0.027095509693026543,
+            -0.004766944330185652,
+            0.022768178954720497,
+            -0.01538266520947218,
+            0.008677597157657146,
+            -0.026850854977965355,
+            -0.012538554146885872,
+            0.018976030871272087,
+            -0.0009489927324466407,
+            0.014060000889003277,
+            -0.018058575689792633,
+            -0.006747118197381496,
+            -0.006024622358381748,
+            0.007419918663799763,
+            -0.005340353585779667,
+            -0.0017030262388288975,
+            -0.007744851056486368,
+            0.01479396503418684,
+            0.0009509040974080563,
+            0.007565182633697987,
+            -0.02275288850069046,
+            0.008318260312080383,
+            0.00087206024909392,
+            0.00670506851747632,
+            0.01882312260568142,
+            -0.00034858519211411476,
+            0.03951173648238182,
+            0.0072287824004888535,
+            0.011644035577774048,
+            -0.012003372423350811,
+            0.004560516681522131,
+            -0.008440587669610977,
+            0.012561490759253502,
+            0.002387294778600335,
+            0.010168462060391903,
+            0.007534600794315338,
+            -0.0029205656610429287,
+            -0.008975770324468613,
+            -0.0013245759764686227,
+            0.0014430806040763855,
+            -0.007706623524427414,
+            0.005703513044863939,
+            0.005493262782692909,
+            0.013570691458880901,
+            -0.01181988138705492,
+            -0.012653236277401447,
+            0.019037194550037384,
+            -0.021040305495262146,
+            -0.006479527335613966,
+            -0.00015768760931678116,
+            -0.006253986153751612,
+            0.004101789090782404,
+            0.0073664006777107716,
+            0.013066090643405914,
+            -0.03608657047152519,
+            0.005493262782692909,
+            -0.0164836123585701,
+            0.006987950298935175,
+            0.0005366157274693251,
+            -0.027829473838210106,
+            -0.024266690015792847,
+            0.0002518223482184112,
+            0.009725024923682213,
+            -0.00527918990701437,
+            0.005615590140223503,
+            -0.017920957878232002,
+            -0.0040329801850020885,
+            -0.004950435366481543,
+            -0.019878195598721504,
+            0.008348842151463032,
+            0.027584819123148918,
+            -0.018242066726088524,
+            -0.010657771490514278,
+            0.008723469451069832,
+            -0.005240962840616703,
+            -0.01989348605275154,
+            -0.006322795525193214,
+            -0.012821435928344727,
+            0.0035341137554496527,
+            0.019862905144691467,
+            -0.0028441110625863075,
+            -0.0113535076379776,
+            0.017248157411813736,
+            0.0066859545186161995,
+            -0.015367374755442142,
+            0.001213716808706522,
+            -0.0012194508453831077,
+            -0.012714399956166744,
+            0.01519152894616127,
+            -0.024266690015792847,
+            0.01578022912144661,
+            -0.0065101091749966145,
+            -0.003471038769930601,
+            -0.031621623784303665,
+            -0.006506286561489105,
+            0.008624078705906868,
+            0.0025688745081424713,
+            0.01793624833226204,
+            0.012393290176987648,
+            -0.020336924120783806,
+            0.01441933773458004,
+            -0.0058029042556881905,
+            0.005286835599690676,
+            -0.006357199978083372,
+            0.042386431246995926,
+            -0.010290789417922497,
+            -0.006517754402011633,
+            -0.00033520563738420606,
+            0.0015338703524321318,
+            0.0033850274048745632,
+            0.02761540189385414,
+            -0.004487885162234306,
+            -0.004308216739445925,
+            -0.0032493204344063997,
+            0.004434366710484028,
+            0.005611767526715994,
+            -0.0006713669281452894,
+            0.015160947106778622,
+            -0.006032268051058054,
+            0.008891670033335686,
+            -0.0224164891988039,
+            -0.0030428930185735226,
+            0.0159637201577425,
+            0.0032130044419318438,
+            -0.0012060713488608599,
+            -0.007672219071537256,
+            -0.03177453204989433,
+            -0.000630272610578686,
+            -0.023073997348546982,
+            0.006269277073442936,
+            -0.010351953096687794,
+            0.0017431648448109627,
+            -0.01415939163416624,
+            0.01081068068742752,
+            0.012744981795549393,
+            0.003799793543294072,
+            0.01460282877087593,
+            -0.0219577606767416,
+            0.0159637201577425,
+            0.007408450823277235,
+            -0.005890826694667339,
+            0.011865753680467606,
+            -0.015176237560808659,
+            -0.009717379696667194,
+            0.0017173613887280226,
+            0.02761540189385414,
+            -0.0012834816006943583,
+            -0.006047558970749378,
+            -0.00348824099637568,
+            -0.013532464392483234,
+            -0.0005409162840805948,
+            0.0010703643783926964,
+            -0.010451343841850758,
+            -0.001181223546154797,
+            -0.006655372679233551,
+            0.009396269917488098,
+            0.010543089359998703,
+            0.000875882979016751,
+            0.006093431729823351,
+            -0.009931452572345734,
+            0.00037367184995673597,
+            0.015099783428013325,
+            0.038563698530197144,
+            -0.0012959054438397288,
+            0.011659326031804085,
+            -0.01952650398015976,
+            -0.004407607484608889,
+            -0.0021789560560137033,
+            -0.029878458008170128,
+            -0.010703643783926964,
+            0.00819593295454979,
+            0.014105873182415962,
+            0.006506286561489105,
+            -0.015764938667416573,
+            0.005003953352570534,
+            0.01778334006667137,
+            0.0095950523391366,
+            0.0047898804768919945,
+            0.0071790870279073715,
+            0.003386938711628318,
+            0.006234872620552778,
+            0.00641454104334116,
+            -0.025306472554802895,
+            -0.011154726147651672,
+            -0.010084361769258976,
+            -0.021177923306822777,
+            0.013769472949206829,
+            0.01718699373304844,
+            0.01573435589671135,
+            0.013524818234145641,
+            -0.010382534936070442,
+            0.03299780562520027,
+            0.023853834718465805,
+            -0.00400239834561944,
+            0.003767300397157669,
+            0.004025334492325783,
+            0.005229494534432888,
+            0.013326036743819714,
+            -0.0028517565224319696,
+            -0.00418971199542284,
+            -0.00036602639011107385,
+            0.013165482319891453,
+            -0.0014535930240526795,
+            -0.004197357688099146,
+            -0.010199043899774551,
+            0.0032493204344063997,
+            0.0005714981234632432,
+            -0.004698134958744049,
+            -0.00477458955720067,
+            -0.006383959203958511,
+            -0.0007339641451835632,
+            -0.00025230017490684986,
+            0.013570691458880901,
+            -0.005523844622075558,
+            -0.023119870573282242,
+            -0.002480951836332679,
+            -0.0017555886879563332,
+            -0.003763477550819516,
+            -0.005088053643703461,
+            -0.005944345146417618,
+            -0.0006622879882343113,
+            0.019908778369426727,
+            0.03217209503054619,
+            0.01966412365436554,
+            0.01675884798169136,
+            0.016070757061243057,
+            0.008249451406300068,
+            -0.011070625856518745,
+            0.010803034529089928,
+            0.002651063259691,
+            -0.0030543613247573376,
+            0.005030712578445673,
+            0.008669951930642128,
+            0.006024622358381748,
+            0.009740316309034824,
+            -0.01782921329140663,
+            0.005485617555677891,
+            -0.017355194315314293,
+            0.00641454104334116,
+            -0.02481716312468052,
+            -0.007851887494325638,
+            -0.006533045321702957,
+            -0.010887134820222855,
+            -0.008914606645703316,
+            0.004407607484608889,
+            0.024327853694558144,
+            -0.006655372679233551,
+            0.01479396503418684,
+            -0.02243177965283394,
+            0.013509527780115604,
+            0.007312882225960493,
+            -0.006399250123649836,
+            -0.014694574289023876,
+            -0.006540691014379263,
+            -0.004847221542149782,
+            -0.008692887611687183,
+            0.012202153913676739,
+            -0.0036640865728259087,
+            -0.016498902812600136,
+            -4.0691170397622045e-06,
+            -0.012163926847279072,
+            -0.013249581679701805,
+            -0.005795258563011885,
+            0.020153433084487915,
+            0.013165482319891453,
+            0.005951990373432636,
+            -0.020627450197935104,
+            0.004174421075731516,
+            -0.0014937317464500666,
+            0.02827291004359722,
+            -0.004950435366481543,
+            -0.0038017048500478268,
+            0.0022343855816870928,
+            0.01402177382260561,
+            -0.007882469333708286,
+            -0.0025765199679881334,
+            0.00270267017185688,
+            0.02130025066435337,
+            0.01016081590205431,
+            0.004438189323991537,
+            -0.007966569624841213,
+            0.021514324471354485,
+            -0.012852017767727375,
+            -0.00269311317242682,
+            -0.003715693484991789,
+            -0.005867890547960997,
+            -0.013647145591676235,
+            0.015573801472783089,
+            0.010145525448024273,
+            0.0027657451573759317,
+            -0.005558249540627003,
+            0.001599812530912459,
+            -0.009327461011707783,
+            -0.010688353329896927,
+            -0.018379686400294304,
+            0.008899315260350704,
+            -0.024358434602618217,
+            0.006598032079637051,
+            0.003727161791175604,
+            0.00022745243040844798,
+            -0.00588700408115983,
+            -0.003430900163948536,
+            0.0027389859315007925,
+            0.01116237137466669,
+            -0.005879358854144812,
+            -0.017003502696752548,
+            -0.002113969763740897,
+            0.02940443903207779,
+            -0.011911626905202866,
+            -0.0015376930823549628,
+            -0.003889627754688263,
+            -0.0037443640176206827,
+            0.002712226938456297,
+            -0.01591784693300724,
+            -0.0048242853954434395,
+            0.0013618475059047341,
+            -6.134287104941905e-05,
+            0.0076034097000956535,
+            0.01675884798169136,
+            -0.007056759670376778,
+            0.016743557527661324,
+            -0.009709734469652176,
+            -0.0015262248925864697,
+            0.017920957878232002,
+            -0.0009126767981797457,
+            -0.01789037697017193,
+            0.008142414502799511,
+            -0.007905405946075916,
+            0.030367767438292503,
+            -0.009189843200147152,
+            -0.027462491765618324,
+            0.025811072438955307,
+            0.02986316569149494,
+            -0.010107298381626606,
+            0.01857846789062023,
+            -0.006590386386960745,
+            0.017997412011027336,
+            0.028915129601955414,
+            -0.0016552420565858483,
+            -0.014319946058094501,
+            0.004927498754113913,
+            -0.007943633012473583,
+            -0.014702219516038895,
+            -0.0034481023903936148,
+            0.01643773913383484,
+            -0.026376836001873016,
+            0.0071446821093559265,
+            -0.025581708177924156,
+            0.0191901046782732,
+            -0.016728267073631287,
+            -0.025994563475251198,
+            -0.008371778763830662,
+            0.012576781213283539,
+            0.007305236998945475,
+            0.029985493049025536,
+            -0.0035455820616334677,
+            -0.034098751842975616,
+            -0.0011793122394010425,
+            0.015329146757721901,
+            -0.008685242384672165,
+            -0.01095594372600317,
+            0.03165220469236374,
+            0.01081832591444254,
+            -0.032325003296136856,
+            0.008188287727534771,
+            -0.026239218190312386,
+            0.000751166429836303,
+            -0.01943475939333439,
+            0.012248027138411999,
+            -0.03984813764691353,
+            -0.0024025856982916594,
+            -0.0013006838271394372,
+            -0.0009853086667135358,
+            -0.004078852944076061,
+            0.005267722066491842,
+            0.0034328114707022905,
+            0.014495791867375374,
+            -0.017676303163170815,
+            -0.027401328086853027,
+            0.011246471665799618,
+            0.008700533770024776,
+            -0.0232269074767828,
+            -0.01737048476934433,
+            0.0035474933683872223,
+            -0.019602959975600243,
+            0.026835564523935318,
+            -0.007037645671516657,
+            -0.001758455764502287,
+            -0.018379686400294304,
+            -0.0043540894985198975,
+            -0.0019314342644065619,
+            -0.01564261130988598,
+            0.014304655604064465,
+            0.008287678472697735,
+            0.01179694477468729,
+            -0.0037214276380836964,
+            -0.01341013703495264,
+            -9.332837180409115e-06,
+            0.011911626905202866,
+            0.011429962702095509,
+            -0.005504731088876724,
+            0.009610342793166637,
+            -0.007943633012473583,
+            -0.0021464629098773003,
+            -0.015390310436487198,
+            0.013777119107544422,
+            -0.0026204814203083515,
+            -0.008914606645703316,
+            -0.0019381240708753467,
+            0.012110408395528793,
+            0.008127124048769474,
+            0.003438545623794198,
+            -0.01018375251442194,
+            0.005302126519382,
+            -0.006835041102021933,
+            -0.01540560182183981,
+            -0.01156758051365614,
+            -0.019083067774772644,
+            -0.007786900736391544,
+            -0.020306341350078583,
+            0.01119295321404934,
+            0.032844897359609604,
+            0.005347999278455973,
+            -0.009228070266544819,
+            -0.025107691064476967,
+            -0.027355454862117767,
+            0.017951540648937225,
+            0.0026529745664447546,
+            -0.002366269938647747,
+            -0.02354801632463932,
+            -0.019159521907567978,
+            0.015596738085150719,
+            0.03529144451022148,
+            -0.0109788803383708,
+            -0.016177793964743614,
+            0.004090321250259876,
+            -0.013027863577008247,
+            -0.014281718991696835,
+            -0.00896812416613102,
+            -0.013035508804023266,
+            -0.0005246697110123932,
+            0.0010464723454788327,
+            -0.007572827860713005,
+            0.028731638565659523,
+            0.009877934120595455,
+            0.009342752397060394,
+            0.011200598441064358,
+            0.0031652203761041164,
+            0.005604122299700975,
+            -0.007278477773070335,
+            -0.0009117210865952075,
+            -0.007033823058009148,
+            0.023884417489171028,
+            -0.003960348200052977,
+            -0.00041811109986156225,
+            0.001845422899350524,
+            -0.004705780651420355,
+            0.012446808628737926,
+            0.00029530588653869927,
+            -0.0329672247171402,
+            0.03501620516180992,
+            -0.006395427044481039,
+            0.0016045909142121673,
+            -0.02597927302122116,
+            -0.011605808511376381,
+            0.021177923306822777,
+            0.006280745379626751,
+            0.001812929636798799,
+            -0.006762409117072821,
+            0.0037290730979293585,
+            0.009121033363044262,
+            0.020658032968640327,
+            -0.01079538930207491,
+            0.002070008311420679,
+            0.015252692624926567,
+            0.012270962819457054,
+            0.00031131357536651194,
+            0.0043540894985198975,
+            0.0057417405769228935,
+            -0.005328885745257139,
+            0.007542246021330357,
+            -0.022492943331599236,
+            -0.01142231747508049,
+            -0.011582871899008751,
+            0.009472724981606007,
+            -0.001574964728206396,
+            0.006112545263022184,
+            0.004935144446790218,
+            0.0034595706965774298,
+            -0.010535444132983685,
+            0.008899315260350704,
+            -0.020795650780200958,
+            -0.004147661849856377,
+            0.00032182608265429735,
+            0.027875347062945366,
+            -0.013004926964640617,
+            0.005856422241777182,
+            -0.009656216017901897,
+            0.009342752397060394,
+            0.020917978137731552,
+            -0.0006183265941217542,
+            0.0026778222527354956,
+            0.0006971703842282295,
+            -0.004254698287695646,
+            0.005615590140223503,
+            -0.003937412053346634,
+            -0.017263447865843773,
+            0.017477521672844887,
+            0.00015051999071147293,
+            0.004549048840999603,
+            -0.002289815340191126,
+            0.0010311814257875085,
+            0.01802799478173256,
+            0.020352214574813843,
+            -0.00835648737847805,
+            -0.002268790267407894,
+            -0.010688353329896927,
+            0.03819671645760536,
+            -0.004491707775741816,
+            0.01793624833226204,
+            0.004938967060297728,
+            -0.004533757921308279,
+            0.0009389580227434635,
+            -0.005340353585779667,
+            0.012936118058860302,
+            -0.00842529721558094,
+            -0.015634965151548386,
+            -0.010703643783926964,
+            -0.008234160020947456,
+            -0.013517173007130623,
+            0.019786451011896133,
+            -0.023853834718465805,
+            0.016957629472017288,
+            0.0017708796076476574,
+            0.009923807345330715,
+            0.010351953096687794,
+            -0.00010811158426804468,
+            0.0049848398193717,
+            0.0003978028253186494,
+            -0.018410267308354378,
+            0.026529746130108833,
+            -0.016407156363129616,
+            0.01877724938094616,
+            -0.01175107154995203,
+            -0.0026032791938632727,
+            0.013585981912910938,
+            0.006995595991611481,
+            0.026330964639782906,
+            -0.004461125936359167,
+            -0.0050115990452468395,
+            0.015657901763916016,
+            0.008494106121361256,
+            0.029694966971874237,
+            0.015795519575476646,
+            0.0059787495993077755,
+            -0.0009169773547910154,
+            0.013257227838039398,
+            -0.01666710339486599,
+            -0.015573801472783089,
+            0.008104187436401844,
+            -0.0007229737821035087,
+            -0.004545225761830807,
+            0.0022305629681795835,
+            0.0349244624376297,
+            -0.0010416939621791244,
+            0.011942208744585514,
+            0.009258652105927467,
+            -0.019205395132303238,
+            0.04177479445934296,
+            0.009977324865758419,
+            0.014969810843467712,
+            -0.0199393592774868,
+            -0.04474123194813728,
+            -0.015466765500605106,
+            -0.008876378647983074,
+            0.005795258563011885,
+            0.0005676753935404122,
+            0.0032665226608514786,
+            -0.028716348111629486,
+            -0.019067777320742607,
+            0.006857977714389563,
+            -0.01675884798169136,
+            0.015887266024947166,
+            -0.0061813541688025,
+            0.0047287167981266975,
+            0.005799081176519394,
+            0.0017326523084193468,
+            -0.0065215774811804295,
+            -0.007569005247205496,
+            0.017079956829547882,
+            0.0033468001056462526,
+            -0.006146949715912342,
+            0.021040305495262146,
+            -0.01058131642639637,
+            0.005669108591973782,
+            -0.0009160216432064772,
+            0.007358754985034466,
+            0.016453029587864876,
+            0.01301257312297821,
+            0.0028001496102660894,
+            0.004327330272644758,
+            0.014381109736859798,
+            0.02992432937026024,
+            0.0014918203232809901,
+            -0.010244916193187237,
+            -0.003870513988658786,
+            0.016223665326833725,
+            0.001654286403208971,
+            -0.019373595714569092,
+            0.014686928130686283,
+            -0.027049636468291283,
+            -0.003406052477657795,
+            -0.01811973936855793,
+            0.002033692318946123,
+            -0.0018301319796591997,
+            -0.007630168925970793,
+            2.1741776436101645e-05,
+            0.007905405946075916,
+            -0.000541871995665133,
+            0.012959054671227932,
+            -0.003409875091165304,
+            0.009916161186993122,
+            0.021651942282915115,
+            -0.008478814736008644,
+            -0.012974345125257969,
+            0.017966831102967262,
+            0.010848907753825188,
+            -0.015038619749248028,
+            0.0008300102199427783,
+            -0.000931790447793901,
+            -0.03137696906924248,
+            -0.0037462753243744373,
+            0.012706754729151726,
+            -0.0012557668378576636,
+            0.00488162599503994,
+            -0.0092204250395298,
+            -0.018058575689792633,
+            0.024220816791057587,
+            0.008784633129835129,
+            0.01456460077315569,
+            0.009044579230248928,
+            -0.010244916193187237,
+            0.0001916143373819068,
+            0.015810811892151833,
+            -0.009885579347610474,
+            -0.0052600763738155365,
+            0.01498510129749775,
+            -0.022676434367895126,
+            0.0113535076379776,
+            -0.008769342675805092,
+            0.0023242197930812836,
+            -0.002817352069541812,
+            -0.01121588982641697,
+            -0.008868733420968056,
+            -0.010925361886620522,
+            0.013333681970834732,
+            -0.023914998397231102,
+            -0.0005160685395821929,
+            0.008081250824034214,
+            0.0029205656610429287,
+            -0.035138532519340515,
+            -0.0033391546458005905,
+            0.014946874231100082,
+            0.029572639614343643,
+            -0.01782921329140663,
+            -0.01102475356310606,
+            0.01582610234618187,
+            -0.007859532721340656,
+            -0.000128658750327304,
+            0.0019199660746380687,
+            -0.006915318313986063,
+            -0.02223299816250801,
+            -0.010566025972366333,
+            -0.021651942282915115,
+            0.007867177948355675,
+            0.003247409127652645,
+            -0.008700533770024776,
+            -0.007492550648748875,
+            -0.014319946058094501,
+            -0.011689907871186733,
+            -0.024129072204232216,
+            -0.0031919796019792557,
+            -0.0010770541848614812,
+            -0.017095249146223068,
+            0.02682027406990528,
+            -0.004036802798509598,
+            0.014908647164702415,
+            -0.020336924120783806,
+            0.04064326360821724,
+            -0.015764938667416573,
+            -0.0016514193266630173,
+            -0.010107298381626606,
+            0.01278320886194706,
+            0.0053441766649484634,
+            -2.7714791940525174e-05,
+            -0.010321371257305145,
+            -0.015979010611772537,
+            0.015451474115252495,
+            0.008631723932921886,
+            0.0029090975876897573,
+            -0.008624078705906868,
+            -0.020168723538517952,
+            -0.009977324865758419,
+            -0.009273942559957504,
+            -0.031193476170301437,
+            -0.0016017238376662135,
+            -0.005787613335996866,
+            0.014648701064288616,
+            -0.008960478939116001,
+            -0.01652948372066021,
+            0.01240858156234026,
+            0.021468451246619225,
+            0.009029287844896317,
+            -0.01639186590909958,
+            0.01118530798703432,
+            -0.010864198207855225,
+            0.004808994475752115,
+            0.017248157411813736,
+            -0.004250875674188137,
+            0.009342752397060394,
+            -0.0038380208425223827,
+            -0.0023127517197281122,
+            -0.004464948549866676,
+            -0.007882469333708286,
+            -0.01217157207429409,
+            -0.007798369042575359,
+            -0.029832584783434868,
+            0.007974214851856232,
+            -0.008792279288172722,
+            0.03351769596338272,
+            0.01240858156234026,
+            0.024496054276823997,
+            -0.01119295321404934,
+            0.023899707943201065,
+            -0.0164836123585701,
+            0.023838544264435768,
+            0.005974926985800266,
+            -0.02570403553545475,
+            0.016835302114486694,
+            -0.010642480105161667,
+            -0.03501620516180992,
+            0.0005934788496233523,
+            -0.011766362935304642,
+            0.03458806127309799,
+            -0.0359642431139946,
+            0.011414671316742897,
+            0.0043540894985198975,
+            -0.020443959161639214,
+            -0.004323507659137249,
+            0.006028444971889257,
+            0.00038275084807537496,
+            0.02139199711382389,
+            -0.009640924632549286,
+            0.004961903672665358,
+            0.014327592216432095,
+            0.022156542167067528,
+            -0.012278608977794647,
+            0.008302969858050346,
+            0.023838544264435768,
+            0.021514324471354485,
+            0.010856552980840206,
+            -0.01680472120642662,
+            0.02883867546916008,
+            -0.0037080480251461267,
+            0.0008797057089395821,
+            0.004235584754496813,
+            -0.02706492878496647,
+            -0.006311327219009399,
+            -0.002654885873198509,
+            0.011621098965406418,
+            0.008845796808600426,
+            0.0015701863449066877,
+            0.013249581679701805,
+            0.01652948372066021,
+            -0.01315783616155386,
+            0.004300571046769619,
+            0.005042180884629488,
+            0.006104899570345879,
+            0.01764572225511074,
+            -0.030750039964914322,
+            0.006028444971889257,
+            -0.021361414343118668,
+            -0.005156862549483776,
+            0.0071561504155397415,
+            0.011957499198615551,
+            0.015856683254241943,
+            -0.03694286197423935,
+            0.0011821791995316744,
+            -0.003293281886726618,
+            0.013096672482788563,
+            0.020841524004936218,
+            0.01479396503418684,
+            -0.01296669989824295,
+            0.008555269800126553,
+            -0.02186601608991623,
+            -0.00314228399656713,
+            -0.020963851362466812,
+            0.002131171990185976,
+            -0.019358305260539055,
+            0.006861800327897072,
+            0.006865622941404581,
+            -0.00559647660702467,
+            -0.0013341327430680394,
+            0.01685059443116188,
+            -0.022921089082956314,
+            -0.008302969858050346,
+            0.012377999722957611,
+            -0.0036507071927189827,
+            -0.004495530389249325,
+            -0.008624078705906868,
+            0.0012376088416203856,
+            0.012997281737625599,
+            0.0069688367657363415,
+            -0.006735650356858969,
+            0.010244916193187237,
+            0.0014268339145928621,
+            0.0025057995226234198,
+            0.006318972446024418,
+            0.004063562024384737,
+            -0.01652948372066021,
+            -0.0014956430532038212,
+            -5.632553802570328e-05,
+            0.012240380980074406,
+            0.008104187436401844,
+            -0.017309321090579033,
+            -0.02111675962805748,
+            0.019281849265098572,
+            -0.028578728437423706,
+            -0.003803616389632225,
+            -0.0008457790245302022,
+            -0.0021445516031235456,
+            0.014518728479743004,
+            0.012821435928344727,
+            -0.01058131642639637,
+            -0.0017498546512797475,
+            -0.01825735904276371,
+            0.0037080480251461267,
+            0.0009833972435444593,
+            -0.010650125332176685,
+            0.004701958037912846,
+            0.010527797974646091,
+            0.011888690292835236,
+            0.0025516722816973925,
+            -0.018242066726088524,
+            0.00994674302637577,
+            0.02152961492538452,
+            0.014411691576242447,
+            -0.004629326052963734,
+            -0.014725156128406525,
+            -0.03614773601293564,
+            -0.023012833669781685,
+            0.026193344965577126,
+            -0.009839707054197788,
+            -0.02157548815011978,
+            0.017523394897580147,
+            0.011644035577774048,
+            0.022034214809536934,
+            -0.009380979463458061,
+            -0.006796814035624266,
+            -0.0019228331511840224,
+            0.017615139484405518,
+            0.004403784871101379,
+            -0.0027389859315007925,
+            0.0017890376038849354,
+            -0.0025153562892228365,
+            0.04116315767168999,
+            0.0021005901508033276,
+            0.005336530972272158,
+            -0.0187313761562109,
+            0.008257096633315086,
+            -0.02477128989994526,
+            0.009442143142223358,
+            -0.011368799023330212,
+            0.029419729486107826,
+            0.019465340301394463,
+            0.0071676187217235565,
+            0.015657901763916016,
+            -0.00901399739086628,
+            0.03669820725917816,
+            -0.014847483485937119,
+            -0.013310745358467102,
+            -0.0024140540044754744,
+            -0.018761958926916122,
+            -0.009029287844896317,
+            -0.0037940593902021646,
+            -0.005546781234443188,
+            0.006953545846045017,
+            -0.0046255034394562244,
+            0.006563627161085606,
+            0.002496242756024003,
+            -0.017584558576345444,
+            0.011284698732197285,
+            -0.022263579070568085,
+            0.01779863052070141,
+            -0.01243151817470789,
+            -0.021269669756293297,
+            0.0014669726369902492,
+            0.01830323040485382,
+            -0.017966831102967262,
+            0.00038824600051157176,
+            -0.0021292606834322214,
+            0.02552054449915886,
+            0.008547624573111534,
+            -0.005642349366098642,
+            0.008272388018667698,
+            -0.004067384637892246,
+            -0.006173708941787481,
+            0.01905248686671257,
+            0.028945710510015488,
+            0.019419468939304352,
+            -0.007282300386577845,
+            0.014992746524512768,
+            -0.004350266885012388,
+            -0.02977142110466957,
+            0.012202153913676739,
+            -0.010803034529089928,
+            0.006586563773453236,
+            0.007301414385437965,
+            -0.009556825272738934,
+            -0.015887266024947166,
+            0.008562915027141571,
+            -0.0149545194581151,
+            0.006387781817466021,
+            0.0014077202649787068,
+            0.0035264682956039906,
+            -0.003073474857956171,
+            0.001033092732541263,
+            -0.008792279288172722,
+            0.027493074536323547,
+            0.023379815742373466,
+            0.012951409444212914,
+            -0.023807961493730545,
+            0.005435922183096409,
+            -0.02448076196014881,
+            -0.0031250817701220512,
+            -0.00686944555491209,
+            -0.0036239479668438435,
+            -0.008432942442595959,
+            -0.0071561504155397415,
+            -0.007159973029047251,
+            -0.01732461154460907,
+            -0.0010178018128499389,
+            0.013777119107544422,
+            -0.001823442173190415,
+            0.006880913861095905,
+            0.0031441953033208847,
+            -0.003029513405635953,
+            -0.0006799680995754898,
+            0.029511475935578346,
+            0.00861643347889185,
+            -0.008792279288172722,
+            -0.012018662877380848,
+            -0.01078774407505989,
+            -0.013035508804023266,
+            0.027003765106201172,
+            0.006032268051058054,
+            -0.013754182495176792,
+            0.012225090526044369,
+            -0.0071790870279073715,
+            -0.021606069058179855,
+            -0.003543670754879713,
+            0.009709734469652176,
+            -0.03403758630156517,
+            0.034007005393505096,
+            -0.004113257396966219,
+            0.0057417405769228935,
+            0.0036602639593183994,
+            0.05174447223544121,
+            -0.005890826694667339,
+            -0.007068227510899305,
+            0.017095249146223068,
+            0.014595182612538338,
+            0.011911626905202866,
+            -0.007756318897008896,
+            0.00559265399351716,
+            -0.023395108059048653,
+            0.007171441335231066,
+            -0.01764572225511074,
+            -0.012003372423350811,
+            0.02822703868150711,
+            0.0010244916193187237,
+            -0.017171703279018402,
+            -0.01117001660168171,
+            -0.015466765500605106,
+            0.007882469333708286,
+            0.029740838333964348,
+            0.022538816556334496,
+            -0.018364394083619118,
+            -0.00609725434333086,
+            -0.01458753738552332,
+            -0.016972921788692474,
+            -0.010604253038764,
+            0.00385713460855186,
+            -0.012423872016370296,
+            -0.0027313404716551304,
+            -0.006158418022096157,
+            0.0006900027510710061,
+            -0.0016848682425916195,
+            -0.000514635001309216,
+            0.06348790228366852,
+            -0.007935987785458565,
+            0.006219581700861454,
+            -0.016070757061243057,
+            0.0007707579061388969,
+            -0.0034538365434855223,
+            -0.008150060661137104,
+            0.0007822260959073901,
+            -0.01759984903037548,
+            0.0287010557949543,
+            -0.024939490482211113,
+            0.0010627189185470343,
+            -0.014992746524512768,
+            0.006877091247588396,
+            -0.029465602710843086,
+            -0.0013752271188423038,
+            -0.01943475939333439,
+            0.004598744213581085,
+            -0.0011038132943212986,
+            -0.01016081590205431,
+            0.015313856303691864,
+            -0.0311781857162714,
+            -0.014113519340753555,
+            0.007572827860713005,
+            0.010436052456498146,
+            0.015084492042660713,
+            -0.020627450197935104,
+            -0.007278477773070335,
+            -0.01877724938094616,
+            -0.023670343682169914,
+            0.008792279288172722,
+            -0.0037500981707125902,
+            0.02481716312468052,
+            -0.01923597790300846,
+            0.0037711230106651783,
+            -0.016820011660456657,
+            0.011147080920636654,
+            0.01402177382260561,
+            0.004487885162234306,
+            -0.0029129202011972666,
+            -0.010428407229483128,
+            0.023838544264435768,
+            -0.00758429616689682,
+            -0.034618642181158066,
+            0.007094986736774445,
+            0.004606389440596104,
+            -0.00779454642906785,
+            -0.011598162353038788,
+            0.0113535076379776,
+            -0.0028613132890313864,
+            0.006292213685810566,
+            -0.02050512284040451,
+            -0.0011028576409444213,
+            -0.014312300831079483,
+            0.012454453855752945,
+            0.013754182495176792,
+            0.01357833668589592,
+            3.586795719456859e-05,
+            -0.010749517008662224,
+            -0.0264532919973135,
+            -0.0004510821308940649,
+            -0.004908385220915079,
+            0.0013647145824506879,
+            0.008700533770024776,
+            0.0031652203761041164,
+            0.022599980235099792,
+            -0.016024883836507797,
+            -0.019495923072099686,
+            0.0002515834057703614,
+            -0.01853259466588497,
+            -0.02804354764521122,
+            0.005290658213198185,
+            -0.004120903089642525,
+            -0.004178243689239025,
+            0.003673643572255969,
+            -0.011108852922916412,
+            -0.010420762002468109,
+            0.007282300386577845,
+            0.0024943312164396048,
+            -0.01639186590909958,
+            0.026835564523935318,
+            -0.0076378146186470985,
+            -0.014060000889003277,
+            -0.006322795525193214,
+            0.008364133536815643,
+            -0.009725024923682213,
+            0.0057302722707390785,
+            -0.03038305789232254,
+            0.0033276863396167755,
+            -0.007159973029047251,
+            -0.012874954380095005,
+            -0.0040100435726344585,
+            0.01591784693300724,
+            0.010260207578539848,
+            -0.007030000444501638,
+            -0.004143839236348867,
+            -0.010130234062671661,
+            -0.0053212400525808334,
+            0.007446677889674902,
+            0.03052067570388317,
+            0.001286348677240312,
+            -0.010458989068865776,
+            -0.014809255488216877,
+            0.017875084653496742,
+            -0.010038488544523716,
+            -0.0287010557949543,
+            0.014281718991696835,
+            0.014732801355421543,
+            -0.03394584357738495,
+            0.0059175859205424786,
+            -0.02073448710143566,
+            -0.011666972190141678,
+            -0.01948063261806965,
+            -0.009625634178519249,
+            0.02393028885126114,
+            -0.02134612388908863,
+            0.009877934120595455,
+            0.005068940110504627,
+            0.03324246034026146,
+            0.01639186590909958,
+            -0.01938888616859913,
+            0.00936568807810545,
+            -0.0037787684705108404,
+            0.006441299803555012,
+            0.008906960487365723,
+            -0.014251137152314186,
+            -0.0002635294513311237,
+            0.016269538551568985,
+            -0.0013704487355425954,
+            0.0018177081365138292,
+            -0.003918298054486513,
+            -0.03211093321442604,
+            0.012630299665033817,
+            -0.02837994694709778,
+            -0.0037787684705108404,
+            0.010573671199381351,
+            -0.01859375834465027,
+            -0.02617805451154709,
+            -0.0024580154567956924,
+            0.012339772656559944,
+            -0.029373856261372566,
+            0.05997098609805107,
+            0.0014268339145928621,
+            0.021147342398762703,
+            -0.01456460077315569,
+            -0.008539978414773941,
+            -0.0047287167981266975,
+            0.0018224865198135376,
+            -0.001679134089499712,
+            0.00024632716667838395,
+            0.014686928130686283,
+            -0.008998706005513668,
+            0.017951540648937225,
+            0.010046134702861309,
+            -0.001892251311801374,
+            0.02477128989994526,
+            0.00678916834294796,
+            0.00739698251709342,
+            0.005034535191953182,
+            0.021055595949292183,
+            -0.0023452448658645153,
+            0.016820011660456657,
+            0.009327461011707783,
+            -0.0109788803383708,
+            0.001719272811897099,
+            -0.008792279288172722,
+            0.014281718991696835,
+            0.011040044017136097,
+            0.002257321961224079,
+            -0.008211224339902401,
+            -0.0032550545874983072,
+            0.021407287567853928,
+            -0.010757162235677242,
+            -0.022034214809536934,
+            -0.010267852805554867,
+            0.0059175859205424786,
+            0.007718091830611229,
+            0.01718699373304844,
+            0.01718699373304844,
+            -0.007458146195858717,
+            -0.011888690292835236,
+            0.022492943331599236,
+            -0.012645591050386429,
+            0.014725156128406525,
+            0.0036602639593183994,
+            -0.017706885933876038,
+            0.02200363390147686,
+            -0.011040044017136097,
+            -0.008295323699712753,
+            -0.0006183265941217542,
+            2.241374022560194e-05,
+            -0.006938254926353693,
+            -0.01587197557091713,
+            -0.020658032968640327,
+            -0.006762409117072821,
+            0.009136324748396873,
+            -0.014037064276635647,
+            0.0032187385950237513,
+            -0.0060016862116754055,
+            0.009755606763064861,
+            0.013272518292069435,
+            0.012668526731431484,
+            -0.003195802215486765,
+            -0.007297591306269169
+          ],
+          "keyphrases": [
+            "Mixture-Of-Experts",
+            "Transformer language models",
+            "self-attention module",
+            "Feed-Forward Network",
+            "computational efficiency"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "e6e4be35-22c1-4d7f-8cbf-cf0caefa3298",
+      "type": "child",
+      "source": {
+        "id": "1e687808-8db8-4320-805d-0282a8d679bc",
+        "properties": {
+          "page_content": "Deepseekmoe: Towards Ultimate Expert Specialization In\n\nMixture-Of-Experts Language Models\n\nDamai Dai∗1,2, Chengqi Deng1, Chenggang Zhao∗1,3, R.X. Xu1, Huazuo Gao1, Deli Chen1, Jiashi Li1, Wangding Zeng1, Xingkai Yu∗1,4, Y. Wu1, Zhenda Xie1, Y.K. Li1, Panpan Huang1, Fuli Luo1, Chong Ruan1, Zhifang Sui2, Wenfeng Liang1 1DeepSeek-AI 2National Key Laboratory for Multimedia Information Processing, Peking University 3Institute for Interdisciplinary Information Sciences, Tsinghua University 4National Key Laboratory for Novel Software Technology, Nanjing University {daidamai, szf}@pku.edu.cn, {wenfeng.liang}@deepseek.com https://github.com/deepseek-ai/DeepSeek-MoE\n\nAbstract\n\nIn the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top- out of experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into ones and activating from them, allowing for a more flexible combination of activated experts; (2) isolating experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5× expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations.\n\nFurther, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\n1. Introduction\n\nRecent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023a). It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017) has emerged as a popular solution. It can\n\nenable parameter scaling, while concurrently keeping computational costs at a modest level.\n\nRecent applications of MoE architectures in Transformers (Vaswani et al., 2017) have yielded successful attempts at scaling language models to a substantial size (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022), accompanied with remarkable performance. These achievements underscore the considerable potential and promise of MoE language models.\n\nDespite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts.\n\nThis architecture manifests two potential issues: (1) Knowledge Hybridity: existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. (2) Knowledge Redundancy: tokens assigned to different experts may require common knowledge. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters.\n\nThese issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models.\n\nIn response to the aforementioned issues, we introduce DeepSeekMoE, an innovative MoE architecture specifically designed towards ultimate expert specialization. Our architecture involves two principal strategies: (1) Fine-Grained Expert Segmentation: while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. (2) Shared Expert Isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.\n\nThrough compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized.\n\nStarting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B (Lepikhin et al., 2021) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5× expert parameters and computation. Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE.\n\nThese studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization.\n\nLeveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B (DeepSeek-AI, 2024), a dense model trained on the same 2T corpus. We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B (Touvron et al., 2023b), which has approximately 2.5 times the activated parameters. Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard1. Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. The experimental results still validate its substantial advantages over the GShard architecture consistently. In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nOur contributions are summarized as follows: - Architectural Innovation. We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation.\n\nEmpirical Validation. We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. Experimental results validate the high 1https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models\n\nScalability. We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B.\n\nAlignment for MoE. We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B.\n\nPublic Release. In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization.\n\n2. Preliminaries: Mixture-Of-Experts For Transformers\n\nWe first introduce a generic MoE architecture commonly used in Transformer language models. A standard Transformer language model is constructed by stacking layers of standard Transformer blocks, where each block can be represented as follows:\n\n$$\\mathbf{u}{1:T}^{l}=\\text{Self-Att}\\left(\\mathbf{h}{1:T}^{l-1}\\right)+\\mathbf{h}{1:T}^{l-1},\\tag{1}$$ $$\\mathbf{h}{t}^{l}=\\text{FFN}\\left(\\mathbf{u}{t}^{l}\\right)+\\mathbf{u}{t}^{l},\\tag{2}$$\n\nwhere denotes the sequence length, Self-Att(·) denotes the self-attention module, FFN(·) denotes the Feed-Forward Network (FFN), u 1: ∈ R× are the hidden states of all tokens after the -th attention module, and h\n\n∈ Ris the output hidden state of the -th token after the -th Transformer block. For brevity, we omit the layer normalization in the above formulations.\n\nA typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022). An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. Then, each token will be assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts. If the -th FFN is substituted with an MoE layer, the computation for its output hidden state h\n\nis expressed as: where denotes the total number of experts, FFN(·) is the -th expert FFN, , denotes the gate value for the -th expert, , denotes the token-to-expert affinity, Topk(·, ) denotes the set comprising highest affinity scores among those calculated for the -th token and all experts, and e\n\nis the centroid of the -th expert in the -th layer. Note that ,is sparse, indicating that only out of gate values are nonzero. This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only experts. Also, in the above formulations, we omit the layer normalization operation for brevity.\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{N}\\left(g_{i,t}\\operatorname{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\operatorname{Topk}({s_{j,t}|1\\leqslant j\\leqslant N},K),\\ 0,&\\text{otherwise},\\end{cases}$$ $$s_{i,t}=\\operatorname{Softmax}{i}\\left(\\mathbf{u}{t}^{l}\\,\\mathbf{e}_{i}^{l}\\right),$$\n\n$$(3)$$\n\n(4) $\\binom{4}{5}$ .\n\n3. Deepseekmoe Architecture\n\nOn top of the generic MoE architecture outlined in Section 2, we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. As illustrated in Figure 2, our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. Both of these strategies are designed to elevate the level of expert specialization.\n\n3.1. Fine-Grained Expert Segmentation\n\nIn scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts.\n\nIn pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. The finer expert segmentation enables a more flexible and adaptable combination of activated experts. To be specific, on top of a typical MoE architecture shown in Figure 2(a), we segment each expert FFN into smaller experts by reducing the FFN intermediate hidden dimension to 1 times its original size. Since each expert becomes smaller, in response, we also increase the number of activated experts to times to keep the same computation cost, as illustrated in Figure 2(b). With the fine-grained expert segmentation, the output of an MoE layer can be expressed as:\n\n$$\\mathbf{h}{t}^{i}=\\sum{i=1}^{\\text{mN}}\\left(g_{i,t}\\text{FFN}{i}\\left(\\mathbf{u}{t}^{i}\\right)\\right)+\\mathbf{u}{t}^{i},\\tag{6}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\text{Tokr}(s_{i,t}|1\\leqslant j\\leqslant m\\text{N}),m\\text{K}),\\ 0,&\\text{otherwise},\\end{cases}$$ (7) $$s_{i,t}=\\text{Softmax}{i}\\left(\\mathbf{u}{t}^{T}\\mathbf{e}_{t}^{i}\\right),\\tag{8}$$ where the total number of expert parameters is equal to $N$ times the number of parameters in a\n\n$$(6)$$\n\n(7) (8) $\\frac{1}{2}$\n\nstandard FFN, and denotes the total number of fine-grained experts. With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to .\n\nFrom a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. As an illustrative example, we consider the case where = 16. A typical top-2 routing strategy can yield 16 2 = 120 possible combinations. By contrast, if each expert is split into 4 smaller experts, the fine-grained routing strategy can yield 64 8 = 4, 426, 165, 368 potential combinations. The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition.\n\n3.2. Shared Expert Isolation\n\nWith a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. This alleviation of redundancy will contribute to a more parameterefficient model with more specialized experts.\n\nTowards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate experts to serve as shared experts. Regardless of the router module, each token will be deterministically assigned to these shared experts. In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by , as depicted in Figure 2(c). With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows:\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{K_{s}}\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)+\\sum_{i=K_{s}+1}^{mN}\\left(g_{i,t}\\;\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},\\tag{9}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\mathrm{Topk}{s_{j,t}|K_{s}+1\\leqslant j\\leqslant mN},mK-K_{s}},\\ 0,&\\text{otherwise},\\end{cases}$$ (10) $$s_{i,t}=\\mathrm{Softmax}{i}\\left(\\mathbf{u}{t}^{l,T}e_{i}^{l}\\right).\\tag{11}$$ $\\mathbf{h}\\mathbf{M}\\mathbf{E}$ the number of closed-compact is $K_{s}$ the total number of routed currents.\n\nFinally, in DeepSeekMoE, the number of shared expert is , the total number of routed experts is − , and the number of nonzero gates is − .\n\nIt is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022). The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint.\n\n3.3. Load Balance Consideration\n\nAutomatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. Firstly, there is a risk of routing collapse (Shazeer et al., 2017), i.e., the model always selects only a few experts, preventing other experts from sufficient training.\n\nSecondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.\n\nExpert-Level Balance Loss. In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. The computation of the balance loss is as follows:\n\n$$\\begin{array}{l}{{{\\mathcal L}{\\mathrm{ExpBal}}=\\alpha{1}\\sum_{i=1}^{N^{\\prime}}f_{i}P_{i},}}\\ {{f_{i}=\\frac{N^{\\prime}}{K^{T}}\\sum_{t=1}^{T}\\mathds{1}\\,(\\mathrm{Token}\\;t\\;\\mathrm{s}\\mathrm{lects}\\;\\mathrm{Expert}\\;i),}}\\ {{P_{i}=\\frac{1}{T}\\sum_{t=1}^{T}s_{i,t},}}\\end{array}$$\n\n(12) $$\\begin{array}{l}\\small\\text{(13)}\\end{array}$$ = (14) $$\\begin{array}{l}\\small\\text{(14)}\\end{array}$$ .\n\n7 where 1 is a hyper-parameter called expert-level balance factor, ′is equal to ( − ) and ′ is equal to ( − ) for brevity. 1(·) denotes the indicator function.\n\nDevice-Level Balance Loss. In addition to the expert-level balance loss, we introduce a devicelevel balance loss. When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. Instead, our primary objective is to ensure balanced computation across the devices. If we partition all routed experts into groups {E1, E2, ..., E}, and deploy each group on a single device, the device-level balance loss is computed as follows:\n\n$$\\mathcal{L}{\\text{DevBal}}=\\alpha{2}\\sum_{i=1}^{D}f_{i}^{\\prime}P_{i}^{\\prime},$$ $$f_{i}^{\\prime}=\\frac{1}{|\\mathcal{E}{i}|}\\sum{j\\in\\mathcal{E}{i}}f{j},$$ $$P_{i}^{\\prime}=\\sum_{j\\in\\mathcal{E}{i}}P{j},$$\n\n(15) $$\\begin{array}{l}\\small\\text{(16)}\\end{array}$$ = (17) .\n\nwhere 2 is a hyper-parameter called device-level balance factor. In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices.\n\n4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n6. Alignment For Deepseekmoe 16B\n\nPrevious research indicates that MoE models typically do not emerge significant gains from fine-tuning (Artetxe et al., 2022; Fedus et al., 2021). However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B.\n\n6.1. Experimental Setup\n\nTraining Data. For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios.\n\nHyper-Parameters. During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).\n\nWe employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10−5 without incorporating any learning rate scheduling strategy.\n\nEvaluation Benchmarks. For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3, with the following adjustments: (1) We exclude Pile (Gao et al., 2020) since chat models are seldom employed for pure language modeling. (2) We exclude\n\nMetric # Shot LLaMA2 DeepSeek DeepSeekMoE SFT 7B Chat 7B Chat 16B # Total Params N/A 6.7B 6.9B 16.4B # Activated Params N/A 6.7B 6.9B 2.8B FLOPs per 4K Tokens N/A 187.9T 183.5T 74.4T HellaSwag (Acc.) 0-shot 67.9 71.0 72.2 PIQA (Acc.) 0-shot 76.9 78.4 79.7 ARC-easy (Acc.) 0-shot 69.7 70.2 69.9 ARC-challenge (Acc.) 0-shot 50.8 50.2 50.0 BBH (EM) 3-shot 39.3 43.1 42.2 RACE-middle (Acc.) 5-shot 63.9 66.1 64.8 RACE-high (Acc.) 5-shot 49.6 50.8 50.6 DROP (EM) 1-shot 40.0 41.7 33.8 GSM8K (EM) 0-shot 63.4 62.6 62.2 MATH (EM) 4-shot 13.5 14.7 15.2 HumanEval (Pass@1) 0-shot 35.4 45.1 45.7 MBPP (Pass@1) 3-shot 27.8 39.0 46.2 TriviaQA (EM) 5-shot 60.1 59.5 63.3 NaturalQuestions (EM) 0-shot 35.2 32.7 35.1 MMLU (Acc.) 0-shot 50.0 49.7 47.2 WinoGrande (Acc.) 0-shot 65.1 68.4 69.0 CLUEWSC (EM) 5-shot 48.4 66.2 68.2 CEval (Acc.) 0-shot 35.1 44.7 40.0 CMMLU (Acc.) 0-shot 36.9 51.2 49.3\n\nCHID (Zheng et al., 2019) due to the observed instability of results, hindering the derivation of solid conclusions. (3) We additionally include BBH (Suzgun et al., 2022) to provide a more comprehensive assessment of the reasoning ability of the chat models.\n\nTable 5 | Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations.\n\n6.2. Evaluations\n\nBaselines. In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. Correspondingly, we construct three chat models, including LLaMA2 SFT 7B3, DeepSeek Chat 7B, and DeepSeekMoE Chat 16B.\n\nSubsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks.\n\nResults. The evaluation results are presented in Table 5. Our key observations include: (1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intensive tasks (TriviaQA, NaturalQuestions). (2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. In addition, it also surpasses DeepSeek Chat 7B. (3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1). However, it is worth noting that, after supervised finetuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. (4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations.\n\n7. Deepseekmoe 145B Ongoing\n\nEncouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available.\n\n7.1. Experimental Setup\n\nTraining Data and Tokenization. For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study.\n\nModel Settings. For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps.\n\nSubsequently, the learning rate keeps constant during the remaining training process. The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. Also, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse.\n\nEvaluation Benchmarks. We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3).\n\n7.2. Evaluations\n\nBaselines. Apart from DeepSeekMoE 145B, we consider three additional models for comparison. DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters.\n\nResults. From the evaluation results presented in Table 6, we have the following observations: (1) Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. (2) On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. (3) At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5.\n\n8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\nAppendices A. Overview Of Hyper-Parameters\n\nWe present the overview of hyper-parameters for DeepSeekMoE across various sizes in Table 7.\n\n# Params # Layers Hidden # Attn # Shared # Routed Relative Sequence Batch Size Learning Size Heads Experts Experts Expert Size Length (Sequence) Rate 2.0B 9 1280 10 1 63 (7 activated) 0.25 2048 2048 1.08e-3 16.4B 28 2048 16 2 64 (6 activated) 0.25 4096 4608 4.2e-4 144.6B 62 4096 32 4 128 (12 activated) 0.125 4096 4608 3.0e-4\n\nTable 7 | Overview of hyper-parameters for DeepSeekMoE across various sizes. The relative expert size is in comparison to a standard FFN.\n\nB. Comparing Deepseekmoe With Larger Models\n\nComparisons among DeepSeekMoE, GShard×1.2, and GShard×1.5 are shown in Table 8. Comparisons among DeepSeekMoE, Dense×4, and Dense×16 are shown in Table 9.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 2.3B 2.8B 1.9B # Activated Expert Params N/A 0.28B 0.35B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.824 1.808 1.808 HellaSwag (Acc.) 0-shot 53.7 54.4 54.8 PIQA (Acc.) 0-shot 71.8 71.1 72.3 ARC-easy (Acc.) 0-shot 46.8 47.3 49.4 ARC-challenge (Acc.) 0-shot 31.7 34.1 34.3 RACE-middle (Acc.) 5-shot 43.7 46.4 44.0 RACE-high (Acc.) 5-shot 31.9 32.4 31.7 HumanEval (Pass@1) 0-shot 3.7 3.0 4.9 MBPP (Pass@1) 3-shot 2.4 2.6 2.2 TriviaQA (EM) 5-shot 15.2 15.7 16.6 NaturalQuestions (EM) 5-shot 4.5 4.7 5.7\n\nTable 8 | Comparison between DeepSeekMoE and larger GShard models.\n\nAt a larger scale of 13B total parameters, we also compare DeepSeekMoE with GShard×1.2 and GShard×1.5, and show results in Table 10. At a larger scale, DeepSeekMoE even outperforms GShard×1.5 distinctly.\n\nMetric # Shot Dense×4 Dense×16 DeepSeekMoE Relative Expert Size N/A 1 1 0.25 # Experts N/A 4 + 0 16 + 0 1 + 63 # Activated Experts N/A 4 + 0 16 + 0 1 + 7 # Total Expert Params N/A 0.47B 1.89B 1.89B # Activated Expert Params N/A 0.47B 1.89B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.908 1.806 1.808 HellaSwag (Acc.) 0-shot 47.6 55.1 54.8 PIQA (Acc.) 0-shot 70.0 71.9 72.3 ARC-easy (Acc.) 0-shot 43.9 51.9 49.4 ARC-challenge (Acc.) 0-shot 30.5 33.8 34.3 RACE-middle (Acc.) 5-shot 42.4 46.3 44.0 RACE-high (Acc.) 5-shot 30.7 33.0 31.7 HumanEval (Pass@1) 0-shot 1.8 4.3 4.9 MBPP (Pass@1) 3-shot 0.2 2.2 2.2 TriviaQA (EM) 5-shot 9.9 16.5 16.6 NaturalQuestions (EM) 5-shot 3.0 6.3 5.7\n\nTable 9 | Comparison between DeepSeekMoE and larger dense baselines.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 15.9B 19.8B 13.3B # Activated Expert Params N/A 2.37B 2.82B 2.05B # Training Tokens N/A 100B 100B 100B HellaSwag (Acc.) 0-shot 66.6 67.7 69.1 PIQA (Acc.) 0-shot 75.6 76.0 75.7 ARC-easy (Acc.) 0-shot 56.8 56.8 58.8 ARC-challenge (Acc.) 0-shot 39.9 37.6 38.5 RACE-middle (Acc.) 5-shot 51.6 50.6 52.4 RACE-high (Acc.) 5-shot 37.4 36.3 38.5 HumanEval (Pass@1) 0-shot 6.1 6.1 9.8 MBPP (Pass@1) 3-shot 7.0 11.6 10.6 TriviaQA (EM) 5-shot 36.5 36.7 38.2 NaturalQuestions (EM) 5-shot 12.6 12.1 13.7\n\nTable 10 | Comparison between DeepSeekMoE and larger GShard models at a larger scale.\n\nC. Training Benchmark Curves Of Deepseekmoe 16B\n\nWe present the benchmark curves during training of DeepSeekMoE 16B and DeepSeek 7B (Dense) in Figure 7 for reference.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/DeepSeekMoE_2.md"
+          },
+          "summary": "DeepSeekMoE is an innovative Mixture-of-Experts (MoE) architecture designed to enhance expert specialization in language models. It employs fine-grained expert segmentation and shared expert isolation to improve performance and reduce computational costs. Starting with a model of 2B parameters, DeepSeekMoE demonstrates comparable performance to larger models like GShard 2.9B and LLaMA2 7B, using significantly fewer computations. The architecture scales up to 16B and 145B parameters, maintaining advantages over traditional MoE architectures. DeepSeekMoE 16B achieves similar results to dense models with only 40% of the computations, and its chat model version shows adaptability in bilingual scenarios. The architecture's public release aims to contribute to advancements in large-scale language models.",
+          "headlines": {
+            "1. Introduction": [],
+            "2. Preliminaries: Mixture-Of-Experts For Transformers": [],
+            "3. Deepseekmoe Architecture": [
+              "3.1. Fine-Grained Expert Segmentation",
+              "3.2. Shared Expert Isolation",
+              "3.3. Load Balance Consideration"
+            ],
+            "4. Validation Experiments": [
+              "4.1. Experimental Setup",
+              "4.1.1. Training Data And Tokenization",
+              "4.1.2. Infrastructures",
+              "4.1.3. Hyper-Parameters",
+              "4.1.4. Evaluation Benchmarks",
+              "4.2. Evaluations",
+              "4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models",
+              "4.4. Ablation Studies",
+              "4.5. Analysis On Expert Specialization"
+            ],
+            "5. Scaling Up To Deepseekmoe 16B": [
+              "5.1. Experimental Setup",
+              "5.1.1. Training Data And Tokenization",
+              "5.1.2. Hyper-Parameters",
+              "5.1.3. Evaluation Benchmarks",
+              "5.2. Evaluations",
+              "5.2.1. Internal Comparison With Deepseek 7B",
+              "5.2.2. Comparison With Open Source Models"
+            ],
+            "6. Alignment For Deepseekmoe 16B": [
+              "6.1. Experimental Setup",
+              "6.2. Evaluations"
+            ],
+            "7. Deepseekmoe 145B Ongoing": [
+              "7.1. Experimental Setup",
+              "7.2. Evaluations"
+            ],
+            "8. Related Work": [],
+            "9. Conclusion": [],
+            "Appendices": [
+              "A. Overview Of Hyper-Parameters",
+              "B. Comparing Deepseekmoe With Larger Models",
+              "C. Training Benchmark Curves Of Deepseekmoe 16B"
+            ]
+          },
+          "summary_embedding": [
+            -0.012465757317841053,
+            -0.022808987647294998,
+            -0.018792515620589256,
+            -0.02308654971420765,
+            0.01797616109251976,
+            0.007200260180979967,
+            0.006490030791610479,
+            0.042613785713911057,
+            -0.014049487188458443,
+            0.03187054023146629,
+            -0.021486490964889526,
+            -0.023592690005898476,
+            0.009045224636793137,
+            -0.014441338367760181,
+            0.005865518003702164,
+            -0.007363531738519669,
+            -0.008743173442780972,
+            -0.005163452122360468,
+            0.018841497600078583,
+            -0.03100520372390747,
+            0.011110605672001839,
+            -0.046075135469436646,
+            -0.020017050206661224,
+            -0.021845687180757523,
+            0.016425084322690964,
+            0.037389107048511505,
+            0.022547753527760506,
+            -0.0028062239289283752,
+            -0.06844329088926315,
+            0.034678805619478226,
+            0.035168617963790894,
+            0.02052319049835205,
+            0.0005438972148112953,
+            -0.00031302153365686536,
+            -0.024833550676703453,
+            -0.028556134551763535,
+            0.03347059711813927,
+            -0.004379750229418278,
+            -0.01453930139541626,
+            0.015976088121533394,
+            0.0414055772125721,
+            0.012106560170650482,
+            -0.02754385396838188,
+            0.0014245413476601243,
+            0.015151568688452244,
+            0.0055961208418011665,
+            -0.01678428053855896,
+            -0.0022960014175623655,
+            -0.002424577483907342,
+            -0.021225256845355034,
+            -0.01400050614029169,
+            0.018825169652700424,
+            0.001296985661610961,
+            -0.008865626528859138,
+            -0.0542386956512928,
+            0.007330877240747213,
+            0.03840138763189316,
+            0.05623060464859009,
+            0.044148534536361694,
+            -0.020180322229862213,
+            0.03369917720556259,
+            -0.030025575309991837,
+            -0.02501314878463745,
+            0.011028970591723919,
+            0.003983817528933287,
+            -0.018531281501054764,
+            0.01784554310142994,
+            -0.005636938381940126,
+            -0.0006556359585374594,
+            0.01751900091767311,
+            -0.03941367194056511,
+            0.020751770585775375,
+            -0.01887415163218975,
+            0.031266435980796814,
+            0.02173139713704586,
+            0.0073798587545752525,
+            0.01872720755636692,
+            0.01678428053855896,
+            0.0036858476232737303,
+            -0.04107903689146042,
+            0.009118696674704552,
+            0.016523046419024467,
+            0.014196431264281273,
+            0.016269976273179054,
+            0.06553706526756287,
+            -0.05391215533018112,
+            -0.004485876299440861,
+            -0.05169166624546051,
+            0.0012561678886413574,
+            -0.004971608519554138,
+            -0.03670336678624153,
+            0.01048201136291027,
+            0.028425518423318863,
+            0.021372200921177864,
+            -0.01063711941242218,
+            0.024392718449234962,
+            -0.022743679583072662,
+            0.0032001156359910965,
+            -0.04062187671661377,
+            0.0019653772469609976,
+            0.004518530797213316,
+            0.016531210392713547,
+            0.0070778070949018,
+            0.004273623693734407,
+            -0.016702644526958466,
+            0.006045116577297449,
+            -0.01905374974012375,
+            0.03291547670960426,
+            -0.008465612307190895,
+            0.08653374016284943,
+            0.008751336485147476,
+            0.004738946910947561,
+            0.004281787667423487,
+            0.02398454025387764,
+            0.012131051160395145,
+            -0.00810233410447836,
+            -0.04718538001179695,
+            -0.015690363943576813,
+            0.011061624623835087,
+            -0.0383034273982048,
+            0.018335357308387756,
+            -0.008759500458836555,
+            0.01282495353370905,
+            -0.04107903689146042,
+            -0.06344719231128693,
+            0.01498013362288475,
+            0.001784758409485221,
+            -0.004065453074872494,
+            -0.0002903166168835014,
+            -0.018923133611679077,
+            0.003930754493921995,
+            -0.004200151655822992,
+            -0.010645282454788685,
+            -0.021502818912267685,
+            0.03159298002719879,
+            0.03657275065779686,
+            0.06527583301067352,
+            0.036801330745220184,
+            -0.0030164355412125587,
+            0.005775718949735165,
+            0.019690508022904396,
+            -0.020784424617886543,
+            0.003481758525595069,
+            -0.01214737817645073,
+            -0.021992631256580353,
+            -0.03820546343922615,
+            0.010971825569868088,
+            0.07379858940839767,
+            -0.029029620811343193,
+            0.013437220826745033,
+            -0.017616963014006615,
+            0.04205866530537605,
+            -0.004567512311041355,
+            0.026727497577667236,
+            -0.016669990494847298,
+            0.004059330560266972,
+            -0.025388672947883606,
+            0.01620466820895672,
+            -0.024964168667793274,
+            0.012571883387863636,
+            -0.000885746325366199,
+            -0.04222193360328674,
+            0.00011116473615402356,
+            0.026890767738223076,
+            -0.010971825569868088,
+            0.02261306345462799,
+            0.0004949158756062388,
+            -0.006302268709987402,
+            -0.05097327008843422,
+            0.038727931678295135,
+            -0.003912386484444141,
+            0.018645571544766426,
+            -0.0018959869630634785,
+            0.0163107942789793,
+            -0.0016408757073804736,
+            0.013216804713010788,
+            0.009543201886117458,
+            0.007114543113857508,
+            -0.033503253012895584,
+            0.02697240374982357,
+            -0.02504580281674862,
+            0.028049994260072708,
+            0.03435226157307625,
+            -0.0030184765346348286,
+            0.03255628049373627,
+            0.019249675795435905,
+            0.03840138763189316,
+            0.0327032245695591,
+            0.030335791409015656,
+            -0.00824927818030119,
+            0.013878053054213524,
+            0.008200296200811863,
+            0.03608293831348419,
+            0.013429056853055954,
+            0.019723162055015564,
+            -0.009730963967740536,
+            -0.014041324146091938,
+            0.048785436898469925,
+            0.007318632211536169,
+            -0.0179598331451416,
+            -0.005441013257950544,
+            -0.03510330989956856,
+            -0.002832755446434021,
+            -0.014425011351704597,
+            0.0323440283536911,
+            0.02158445306122303,
+            -0.01647406443953514,
+            0.035135965794324875,
+            0.024980494752526283,
+            -0.03751972317695618,
+            -0.005232842173427343,
+            -0.02026195637881756,
+            -0.023298801854252815,
+            -0.04215662553906441,
+            0.011624909937381744,
+            0.012833117507398129,
+            -0.013976015150547028,
+            -0.007469657808542252,
+            0.020457882434129715,
+            -0.023625344038009644,
+            -0.0018265966791659594,
+            -0.01356783788651228,
+            0.006547175347805023,
+            -0.020898714661598206,
+            -0.03667071461677551,
+            -0.01048201136291027,
+            0.025862159207463264,
+            0.012637191452085972,
+            0.04362606629729271,
+            -0.013518855907022953,
+            -0.005367541220039129,
+            0.033193036913871765,
+            0.012759645469486713,
+            0.010767736472189426,
+            0.03699725493788719,
+            -0.03027048148214817,
+            0.024621298536658287,
+            -0.020294612273573875,
+            -0.0032878739293664694,
+            0.01707816869020462,
+            -0.0030062312725931406,
+            0.01962519995868206,
+            -0.04535674303770065,
+            -0.024196794256567955,
+            0.04395261034369469,
+            -0.0007107399869710207,
+            0.016302630305290222,
+            -0.005959399044513702,
+            -0.012106560170650482,
+            0.0035899258218705654,
+            0.0027715289033949375,
+            -0.01438419334590435,
+            -0.008620720356702805,
+            0.007473739795386791,
+            -0.01962519995868206,
+            0.03660540655255318,
+            0.03794422745704651,
+            -0.004132802598178387,
+            -0.014131123200058937,
+            -0.03324201703071594,
+            0.010016689077019691,
+            -0.017878197133541107,
+            -0.023951886221766472,
+            0.04078514873981476,
+            -0.032115448266267776,
+            -0.00029312283731997013,
+            0.039478980004787445,
+            0.018808843567967415,
+            -0.014898497611284256,
+            0.00868602842092514,
+            -0.01709449477493763,
+            -0.026302991434931755,
+            -0.017159804701805115,
+            0.03356856107711792,
+            0.024800896644592285,
+            -0.010743245482444763,
+            0.010245268233120441,
+            0.005330805201083422,
+            0.02893165871500969,
+            0.024376392364501953,
+            0.00648594880476594,
+            -0.013812744058668613,
+            0.015029114671051502,
+            0.027641816064715385,
+            -0.0028021421749144793,
+            -7.66609373386018e-05,
+            -0.004010349046438932,
+            -0.007796200457960367,
+            -0.006453294772654772,
+            -0.013069860637187958,
+            -0.005089980084449053,
+            0.0009184005321003497,
+            -0.032295044511556625,
+            0.015967924147844315,
+            -0.005890008993446827,
+            0.010359558276832104,
+            -0.011535110883414745,
+            0.01288209855556488,
+            0.01722511276602745,
+            -0.013494365848600864,
+            -0.0716434046626091,
+            0.014612773433327675,
+            0.02458864450454712,
+            -0.006706364918500185,
+            -0.017943505197763443,
+            0.008481939323246479,
+            0.00016148542636074126,
+            0.038597315549850464,
+            -0.014180104248225689,
+            -0.031103165820240974,
+            0.02680913358926773,
+            -0.017437364906072617,
+            -0.015241367742419243,
+            -0.010661610402166843,
+            -0.00580837344750762,
+            0.04649963974952698,
+            0.01889047957956791,
+            0.011420820839703083,
+            -0.04042595252394676,
+            -0.007449248805642128,
+            0.015078096650540829,
+            0.014506647363305092,
+            -0.04705476388335228,
+            0.0052083516493439674,
+            0.015486273914575577,
+            0.019723162055015564,
+            0.024800896644592285,
+            0.04274440184235573,
+            -0.0021531390957534313,
+            0.01661284640431404,
+            -0.015053605660796165,
+            -0.05136512219905853,
+            -0.03340528905391693,
+            0.029943939298391342,
+            -0.017731253057718277,
+            -0.02906227484345436,
+            -0.01693122461438179,
+            -0.006151242647320032,
+            0.0017939424142241478,
+            0.007126788143068552,
+            -0.01916803978383541,
+            -0.03464614972472191,
+            -0.03970756009221077,
+            0.02277633361518383,
+            0.007983962073922157,
+            0.004269542172551155,
+            -0.042777057737112045,
+            0.0013459670590236783,
+            0.009184005670249462,
+            0.012073906138539314,
+            0.0009954441338777542,
+            -0.019984396174550056,
+            0.015494437888264656,
+            0.0032001156359910965,
+            -0.031070511788129807,
+            -0.035593122243881226,
+            -0.03595232218503952,
+            0.0023919232189655304,
+            0.003781769424676895,
+            0.06279410421848297,
+            0.012988224625587463,
+            -0.03840138763189316,
+            -0.04522612690925598,
+            0.029176564887166023,
+            -0.02052319049835205,
+            -0.016678154468536377,
+            0.0059430720284581184,
+            0.012898425571620464,
+            -0.009877908043563366,
+            0.013192313723266125,
+            0.023560035973787308,
+            0.041601505130529404,
+            -0.005987971555441618,
+            0.012914752587676048,
+            0.023037567734718323,
+            0.012955570593476295,
+            0.0362788625061512,
+            -0.00936360377818346,
+            0.0035409443080425262,
+            0.039936136454343796,
+            -0.012269831262528896,
+            0.03175625205039978,
+            -0.02189466916024685,
+            0.0020398697815835476,
+            0.02249877341091633,
+            0.027184655889868736,
+            -0.0006479826406575739,
+            0.02594379521906376,
+            -0.031543996185064316,
+            0.018368011340498924,
+            -0.014498483389616013,
+            -0.04888340085744858,
+            -0.01557607389986515,
+            0.013102514669299126,
+            0.030384771525859833,
+            0.02756018005311489,
+            -0.0041348435916006565,
+            -0.019723162055015564,
+            -0.01155143789947033,
+            -0.010596301406621933,
+            0.0018419033149257302,
+            0.03285016864538193,
+            0.010506502352654934,
+            0.010890189558267593,
+            -0.011935125105082989,
+            -0.004104230087250471,
+            0.03025415539741516,
+            -0.021192602813243866,
+            0.01650671847164631,
+            0.0041511706076562405,
+            -0.002029665280133486,
+            -0.04320156201720238,
+            -0.035266581922769547,
+            0.007845181971788406,
+            0.011747363954782486,
+            0.014874007552862167,
+            0.0368339866399765,
+            0.025094784796237946,
+            -0.01662100851535797,
+            0.030629679560661316,
+            -0.01905374974012375,
+            0.027984686195850372,
+            0.039054471999406815,
+            0.011624909937381744,
+            -0.018351683393120766,
+            0.012237177230417728,
+            0.002424577483907342,
+            -0.025127438828349113,
+            0.01680060662329197,
+            -0.032735876739025116,
+            0.01738838478922844,
+            -0.02354370802640915,
+            0.025143766775727272,
+            -0.02579685114324093,
+            -0.009290131740272045,
+            -0.004428731743246317,
+            -0.006130834110081196,
+            0.04888340085744858,
+            -0.026727497577667236,
+            -0.0142454132437706,
+            -0.0024694770108908415,
+            -0.0024531499948352575,
+            -0.008718682453036308,
+            0.010000362060964108,
+            -0.045781247317790985,
+            -0.01844964735209942,
+            -0.04016471654176712,
+            0.007355368230491877,
+            0.005412440747022629,
+            -0.030907239764928818,
+            -0.05694899708032608,
+            0.025421326979994774,
+            -0.06331657618284225,
+            -0.019886434078216553,
+            0.051626358181238174,
+            -0.018400665372610092,
+            -0.008212542161345482,
+            0.009641164913773537,
+            0.006375740747898817,
+            -0.029290854930877686,
+            0.0041348435916006565,
+            0.02504580281674862,
+            0.04418119043111801,
+            -0.0045797573402523994,
+            -0.02865409664809704,
+            0.0028674507047981024,
+            -0.027788760140538216,
+            -0.013461710885167122,
+            0.005261414684355259,
+            0.016449574381113052,
+            0.024539662525057793,
+            0.007628847379237413,
+            0.03585435822606087,
+            0.007294141221791506,
+            0.006318595726042986,
+            0.044279150664806366,
+            -0.025666235014796257,
+            -0.027380581945180893,
+            0.02519274689257145,
+            0.04045860469341278,
+            0.023151857778429985,
+            -0.018253721296787262,
+            -0.0142454132437706,
+            0.0053063142113387585,
+            0.0009296254720538855,
+            -0.022139575332403183,
+            0.002808264922350645,
+            -0.013902543112635612,
+            0.005143043119460344,
+            0.03892385587096214,
+            -0.001205145614221692,
+            0.014196431264281273,
+            0.007302304729819298,
+            0.006845145486295223,
+            0.004334850702434778,
+            -0.011731036007404327,
+            -0.023592690005898476,
+            0.00032220553839579225,
+            0.034972693771123886,
+            0.015404638834297657,
+            -0.030515389516949654,
+            -0.00995954405516386,
+            0.034711457788944244,
+            -0.00795130804181099,
+            -0.009118696674704552,
+            -0.009771781973540783,
+            0.003385836724191904,
+            0.02638462744653225,
+            -0.01606588624417782,
+            0.006649219896644354,
+            0.03716052696108818,
+            -0.019870106130838394,
+            0.005543057806789875,
+            -0.025405000895261765,
+            0.0018061877926811576,
+            0.02442537248134613,
+            -0.006216551177203655,
+            0.028098974376916885,
+            5.6315813708351925e-05,
+            0.03025415539741516,
+            -0.004677720367908478,
+            0.008432958275079727,
+            -0.020327266305685043,
+            0.03987082839012146,
+            -0.017633290961384773,
+            0.03402572125196457,
+            0.02067013457417488,
+            -0.01258004643023014,
+            0.04783846437931061,
+            -0.036801330745220184,
+            -0.020784424617886543,
+            0.00584919098764658,
+            0.012743317522108555,
+            0.0028919412288814783,
+            -0.0191517136991024,
+            -0.018400665372610092,
+            -0.010996315628290176,
+            0.02380494214594364,
+            -0.022547753527760506,
+            -0.016865916550159454,
+            -0.007441085297614336,
+            -0.01890680566430092,
+            -0.009673818945884705,
+            -0.007988044060766697,
+            -0.01709449477493763,
+            -0.016547536477446556,
+            0.044573038816452026,
+            -0.06334923207759857,
+            0.016996532678604126,
+            0.017159804701805115,
+            0.011477965861558914,
+            -0.034678805619478226,
+            -0.019510909914970398,
+            -0.017633290961384773,
+            -0.002347023691982031,
+            -0.030743969604372978,
+            0.001702102366834879,
+            0.01725776679813862,
+            0.01138000376522541,
+            -0.05022222548723221,
+            0.012335140258073807,
+            0.02772345207631588,
+            -0.012065742164850235,
+            0.0014949520118534565,
+            -0.023135529831051826,
+            -0.017176130786538124,
+            0.0031776658724993467,
+            0.022123249247670174,
+            0.005975726060569286,
+            -0.008188051171600819,
+            -0.016865916550159454,
+            -0.005241005681455135,
+            -0.006045116577297449,
+            -0.002359268954023719,
+            0.005094061605632305,
+            0.011780017986893654,
+            0.019690508022904396,
+            -0.0024633544962853193,
+            -0.004436895251274109,
+            -0.006396149750798941,
+            0.027364253997802734,
+            -0.001450052484869957,
+            -0.008416631259024143,
+            0.050907962024211884,
+            0.03556047007441521,
+            0.006094098091125488,
+            -0.01108611561357975,
+            -0.03990348428487778,
+            0.010661610402166843,
+            -0.022711025550961494,
+            0.0401320643723011,
+            0.01905374974012375,
+            -0.008298259228467941,
+            -0.028866350650787354,
+            0.010294250212609768,
+            -0.02290695160627365,
+            -0.009975871071219444,
+            0.009469729848206043,
+            0.00726556871086359,
+            -0.03263791650533676,
+            -0.013543346896767616,
+            0.03510330989956856,
+            0.011437147855758667,
+            -0.006841063499450684,
+            -0.036050282418727875,
+            0.008588065393269062,
+            0.01646590232849121,
+            0.011869817040860653,
+            -0.019233347848057747,
+            0.004449140280485153,
+            -0.012041252106428146,
+            0.017470018938183784,
+            0.008849299512803555,
+            -0.017796561121940613,
+            0.018237393349409103,
+            0.004710374400019646,
+            -0.008490103296935558,
+            -0.003912386484444141,
+            -0.0163107942789793,
+            -0.02006603218615055,
+            0.03840138763189316,
+            0.024490682408213615,
+            0.026335645467042923,
+            0.005457340274006128,
+            0.02008235827088356,
+            0.03227871656417847,
+            0.011282040737569332,
+            -0.03134807199239731,
+            0.027935704216361046,
+            0.016996532678604126,
+            -0.04604247957468033,
+            0.021551799029111862,
+            -0.02218855731189251,
+            -0.028670424595475197,
+            0.009649328887462616,
+            -0.0017367975087836385,
+            0.011192241683602333,
+            0.0034837995190173388,
+            -0.003642988856881857,
+            -0.03307874873280525,
+            -0.006236960180103779,
+            0.015771998092532158,
+            0.015502601861953735,
+            -0.0048695639707148075,
+            -0.003296037670224905,
+            -0.011886144056916237,
+            0.00487364549189806,
+            0.009796272963285446,
+            0.039054471999406815,
+            0.001381682581268251,
+            -0.004751192405819893,
+            -0.0018765984568744898,
+            0.031854212284088135,
+            0.0271193478256464,
+            -0.017274094745516777,
+            -0.026286665350198746,
+            0.00580837344750762,
+            -0.014253576286137104,
+            0.026156047359108925,
+            -0.00487364549189806,
+            0.011877980083227158,
+            0.008718682453036308,
+            -0.0007908449042588472,
+            -0.001373519073240459,
+            0.025584599003195763,
+            -0.02710302174091339,
+            -0.021764053031802177,
+            0.031233781948685646,
+            -0.025731543079018593,
+            0.028735732659697533,
+            0.013061696663498878,
+            -0.01557607389986515,
+            -0.015225040726363659,
+            -0.010302413254976273,
+            0.009943217039108276,
+            -0.012277995236217976,
+            0.023853924125432968,
+            0.003608293831348419,
+            0.006392067763954401,
+            0.0003584313380997628,
+            0.02770712412893772,
+            0.024833550676703453,
+            0.007820690982043743,
+            -0.018074123188853264,
+            -0.0016816934803500772,
+            -0.02320083975791931,
+            -0.01680060662329197,
+            0.023772288113832474,
+            -0.018090449273586273,
+            -0.03268689662218094,
+            -0.028556134551763535,
+            0.012114724144339561,
+            0.007392104249447584,
+            0.010816717520356178,
+            0.03461349755525589,
+            -0.030384771525859833,
+            -0.0344502255320549,
+            0.01766594499349594,
+            0.026270337402820587,
+            -0.005551221314817667,
+            -0.021535472944378853,
+            0.03085825778543949,
+            -0.005828781984746456,
+            0.017600636929273605,
+            0.0404912605881691,
+            -0.0016225076979026198,
+            0.010228941217064857,
+            -0.042026009410619736,
+            -0.014057651162147522,
+            -0.0066614653915166855,
+            0.006261450704187155,
+            0.034417569637298584,
+            0.01048201136291027,
+            0.01535565685480833,
+            0.03448288142681122,
+            -0.021192602813243866,
+            -0.0002903166168835014,
+            -0.0022449791431427,
+            -0.02818061038851738,
+            0.03347059711813927,
+            0.025715215131640434,
+            0.011918798089027405,
+            0.019706835970282555,
+            -0.03794422745704651,
+            -0.013037205673754215,
+            0.027070365846157074,
+            0.015053605660796165,
+            0.01273515447974205,
+            0.016694480553269386,
+            0.011249386705458164,
+            0.00868602842092514,
+            -0.021633435040712357,
+            -0.05169166624546051,
+            0.01214737817645073,
+            0.01993541419506073,
+            -0.016449574381113052,
+            0.01057181041687727,
+            0.024980494752526283,
+            0.007620683871209621,
+            -0.007045152597129345,
+            0.012808626517653465,
+            -0.02682545967400074,
+            0.002259265398606658,
+            0.0019010890973731875,
+            -0.014482156373560429,
+            0.038270771503448486,
+            -0.016367938369512558,
+            0.011363675817847252,
+            -0.018204739317297935,
+            0.023870250210165977,
+            0.00011531029304023832,
+            0.0007964573451317847,
+            0.014963806606829166,
+            -0.011347348801791668,
+            -0.04235255345702171,
+            0.0021613026037812233,
+            0.03342161700129509,
+            -0.015200549736618996,
+            0.026449935510754585,
+            -0.0020674217958003283,
+            0.032458316534757614,
+            0.042777057737112045,
+            0.017731253057718277,
+            0.0003497575526125729,
+            0.025682561099529266,
+            0.0036144163459539413,
+            0.0073880222626030445,
+            0.012400448322296143,
+            0.009322785772383213,
+            0.007245160173624754,
+            0.030809277668595314,
+            -0.010261595249176025,
+            0.02173139713704586,
+            -0.04760988429188728,
+            -0.014629100449383259,
+            0.012024925090372562,
+            -0.0013316808035597205,
+            -0.012449429370462894,
+            -0.033029764890670776,
+            0.012563719414174557,
+            -0.03503800183534622,
+            0.022074267268180847,
+            -0.0004959363141097128,
+            0.04333217814564705,
+            -0.02187834307551384,
+            -0.005518566817045212,
+            0.016694480553269386,
+            0.0016510800924152136,
+            0.023282473906874657,
+            -0.0012153501156717539,
+            -0.016416920349001884,
+            -0.0067798369564116,
+            -0.009257477708160877,
+            0.004138925112783909,
+            0.04375668615102768,
+            -0.003722583642229438,
+            -0.0001682458823779598,
+            0.01409030519425869,
+            -0.011959616094827652,
+            -0.03250729665160179,
+            0.045193471014499664,
+            -0.0030837850645184517,
+            0.02484987862408161,
+            0.04979771748185158,
+            -0.06410028040409088,
+            -0.019543563947081566,
+            0.013061696663498878,
+            -0.010441194288432598,
+            0.011363675817847252,
+            0.005547139327973127,
+            0.016114868223667145,
+            0.017894525080919266,
+            -0.005485912784934044,
+            0.01619650423526764,
+            0.005106307100504637,
+            0.028098974376916885,
+            -0.02336410991847515,
+            -0.03296445682644844,
+            -0.02636830136179924,
+            -0.017747581005096436,
+            0.0014480114914476871,
+            -0.015796490013599396,
+            -3.563585551091819e-06,
+            -0.0022204883862286806,
+            -0.04388730227947235,
+            -0.025274382904171944,
+            0.040687184780836105,
+            0.006706364918500185,
+            0.02458864450454712,
+            0.0077268099412322044,
+            0.03418898954987526,
+            0.005885927006602287,
+            -0.009037061594426632,
+            0.030058229342103004,
+            -0.020049704238772392,
+            0.03833607956767082,
+            0.022106921300292015,
+            -0.002832755446434021,
+            0.005865518003702164,
+            0.031086837872862816,
+            -0.033797141164541245,
+            0.008086007088422775,
+            -0.034384917467832565,
+            0.0046695563942193985,
+            -0.008620720356702805,
+            -0.015094423666596413,
+            -0.009526874870061874,
+            -0.02504580281674862,
+            0.011216731742024422,
+            -0.021535472944378853,
+            0.01590261608362198,
+            0.010416703298687935,
+            -0.002916431985795498,
+            0.03448288142681122,
+            -0.014204595237970352,
+            -0.0027796924114227295,
+            0.0024531499948352575,
+            -0.027070365846157074,
+            -0.006343086715787649,
+            -0.003377673216164112,
+            -0.03689929470419884,
+            -0.004502203781157732,
+            0.010988152585923672,
+            -0.00024184548237826675,
+            0.016898570582270622,
+            0.02787039615213871,
+            0.0005092020728625357,
+            -0.022302847355604172,
+            0.002340900944545865,
+            -0.04199335351586342,
+            0.0011877981014549732,
+            0.01483318954706192,
+            -0.017600636929273605,
+            -0.008612556383013725,
+            0.033046092838048935,
+            0.023298801854252815,
+            -0.009396257810294628,
+            -0.035005345940589905,
+            0.017796561121940613,
+            -0.011853490024805069,
+            0.027511198073625565,
+            -0.0002630197268445045,
+            0.019298657774925232,
+            -0.0076941559091210365,
+            0.04493223875761032,
+            0.001369437319226563,
+            0.008645210415124893,
+            -0.021959977224469185,
+            0.007020662073045969,
+            -0.020278284326195717,
+            -0.002406209474429488,
+            -0.006171651650220156,
+            -0.012751481495797634,
+            -0.010669773444533348,
+            0.0032735876739025116,
+            0.007730891928076744,
+            -0.0176822729408741,
+            0.0055961208418011665,
+            -0.015959760174155235,
+            -0.02741323597729206,
+            0.007543129846453667,
+            -0.05965929850935936,
+            -0.0179598331451416,
+            -0.03373183310031891,
+            -0.009477893821895123,
+            0.02008235827088356,
+            0.03510330989956856,
+            -0.012759645469486713,
+            0.00497977202758193,
+            -0.014898497611284256,
+            0.004477712791413069,
+            0.03490738570690155,
+            0.044442422688007355,
+            -0.014759717509150505,
+            0.003381754970178008,
+            -0.010955498553812504,
+            0.033029764890670776,
+            -0.03347059711813927,
+            0.0030062312725931406,
+            0.031086837872862816,
+            -0.005057325586676598,
+            0.007959471084177494,
+            0.011624909937381744,
+            0.020474210381507874,
+            -0.019396619871258736,
+            0.00877582747489214,
+            0.002100075827911496,
+            0.019233347848057747,
+            0.02171507105231285,
+            0.020343592390418053,
+            -0.03539719805121422,
+            0.028409190475940704,
+            0.018384337425231934,
+            0.02444170042872429,
+            0.044442422688007355,
+            -0.03974021226167679,
+            -0.004147088620811701,
+            -0.0032225653994828463,
+            -0.02997659333050251,
+            0.02008235827088356,
+            -0.007530884351581335,
+            -0.01841699331998825,
+            -0.007069643586874008,
+            0.03327467292547226,
+            0.014571955427527428,
+            -0.05051611363887787,
+            0.01980479806661606,
+            0.01740471087396145,
+            0.02055584453046322,
+            0.024474354460835457,
+            -0.01123305968940258,
+            -0.0019072118448093534,
+            0.008996243588626385,
+            0.006804327480494976,
+            0.019266001880168915,
+            0.015257694758474827,
+            0.002012317767366767,
+            -0.0022082431241869926,
+            -0.011837163008749485,
+            0.005224678665399551,
+            -0.009077879600226879,
+            0.010808554477989674,
+            -0.02458864450454712,
+            -0.009698309935629368,
+            0.021388528868556023,
+            0.0031164393294602633,
+            -0.013665799982845783,
+            -0.03729114308953285,
+            -0.045944519340991974,
+            0.0033837957307696342,
+            0.004543021321296692,
+            -0.05365091934800148,
+            0.005322641227394342,
+            -0.026547899469733238,
+            -0.002108239568769932,
+            -0.005502239800989628,
+            0.0019214981002733111,
+            0.005636938381940126,
+            0.0167679525911808,
+            -0.013494365848600864,
+            0.02248244546353817,
+            -0.015502601861953735,
+            0.03165828809142113,
+            0.0055961208418011665,
+            0.007375776767730713,
+            -0.0027796924114227295,
+            0.05590406060218811,
+            -0.003712379140779376,
+            -0.03699725493788719,
+            0.023282473906874657,
+            -0.01311067771166563,
+            -0.0001747512142173946,
+            0.030090883374214172,
+            0.00884113647043705,
+            -0.015257694758474827,
+            0.013437220826745033,
+            -0.030613351613283157,
+            -0.03249097242951393,
+            0.010996315628290176,
+            -0.013755599036812782,
+            -0.0015776080545037985,
+            -0.04568328335881233,
+            -0.009339112788438797,
+            0.004269542172551155,
+            -0.018335357308387756,
+            0.010775899514555931,
+            0.01631895825266838,
+            0.013992343097925186,
+            -0.024245774373412132,
+            0.009053388610482216,
+            0.006865554489195347,
+            0.013453547842800617,
+            0.0007331897504627705,
+            0.014049487188458443,
+            -0.008677864447236061,
+            -0.011306531727313995,
+            -0.018319029361009598,
+            0.0032450153958052397,
+            -0.026792805641889572,
+            0.005155288614332676,
+            -0.003155216109007597,
+            -0.03771565109491348,
+            -0.06521052122116089,
+            -0.0344502255320549,
+            0.016294466331601143,
+            0.03396041318774223,
+            -0.009045224636793137,
+            0.0035144127905368805,
+            -0.00023648813657928258,
+            -0.01093917153775692,
+            0.024523336440324783,
+            -0.005930826533585787,
+            -0.0004829256213270128,
+            0.0035389035474509,
+            0.02050686441361904,
+            -0.009722800925374031,
+            0.005845109466463327,
+            -0.051952898502349854,
+            -0.01586179807782173,
+            0.01812310516834259,
+            0.006306350696831942,
+            0.014629100449383259,
+            -0.009690145961940289,
+            0.01693122461438179,
+            -0.030074557289481163,
+            -0.028147956356406212,
+            0.04130761697888374,
+            0.008277850225567818,
+            0.011486129835247993,
+            -0.003055212553590536,
+            -0.01680060662329197,
+            0.02504580281674862,
+            -0.0020633398089557886,
+            -0.006461458280682564,
+            -0.00817988719791174,
+            0.015820980072021484,
+            0.044279150664806366,
+            0.003255219664424658,
+            0.0009821783751249313,
+            0.017045514658093452,
+            -0.002663361607119441,
+            -0.011575928889214993,
+            0.007677828427404165,
+            0.013592327944934368,
+            -0.016384266316890717,
+            0.010375885292887688,
+            -0.025862159207463264,
+            0.0332583449780941,
+            0.03131541982293129,
+            0.011575928889214993,
+            -0.015045441687107086,
+            0.006130834110081196,
+            0.005330805201083422,
+            0.008086007088422775,
+            -0.001369437319226563,
+            -0.0017133272485807538,
+            0.020719116553664207,
+            -0.03276853263378143,
+            -0.01693122461438179,
+            -0.005347132217139006,
+            0.003932795487344265,
+            0.005277741700410843,
+            0.03396041318774223,
+            -0.0033266509417444468,
+            0.017927179113030434,
+            -0.011657563969492912,
+            0.010579974390566349,
+            0.021764053031802177,
+            0.037095218896865845,
+            -0.024996822699904442,
+            -0.029862305149435997,
+            0.012914752587676048,
+            -0.001296985661610961,
+            0.014065815135836601,
+            0.0010173837654292583,
+            0.014139287173748016,
+            0.0029409227427095175,
+            0.060475654900074005,
+            -0.017763907089829445,
+            0.004628738854080439,
+            0.0004035860183648765,
+            -0.01858026348054409,
+            -0.0013531101867556572,
+            0.004261378664523363,
+            -0.010979988612234592,
+            0.03608293831348419,
+            0.002344982698559761,
+            0.039185091853141785,
+            0.013918871060013771,
+            0.009290131740272045,
+            0.0015143405180424452,
+            -0.013592327944934368,
+            0.008355404250323772,
+            0.0015592400450259447,
+            -0.001132694073021412,
+            0.009461566805839539,
+            0.019298657774925232,
+            0.009714636951684952,
+            0.025274382904171944,
+            0.009788108989596367,
+            0.012857607565820217,
+            -0.029470453038811684,
+            0.03510330989956856,
+            -0.0006224715034477413,
+            0.012996388599276543,
+            -0.02040890045464039,
+            -0.003136848099529743,
+            -0.0008240094175562263,
+            0.015649545937776566,
+            -0.015616890974342823,
+            -0.001201063860207796,
+            -0.01297189760953188,
+            0.040099408477544785,
+            -0.018988441675901413,
+            0.013976015150547028,
+            0.009673818945884705,
+            -0.02865409664809704,
+            -0.007596192881464958,
+            0.01995174214243889,
+            0.014294394291937351,
+            -0.0053144777193665504,
+            -0.020490536466240883,
+            0.005832863971590996,
+            0.015380147844552994,
+            -0.028572462499141693,
+            1.3584674888988957e-05,
+            0.014751553535461426,
+            -0.0164903923869133,
+            -0.00025409081717953086,
+            -0.004702210891991854,
+            0.003975654020905495,
+            -0.009265640750527382,
+            0.007028825581073761,
+            0.016057724133133888,
+            -0.024507008492946625,
+            -0.020049704238772392,
+            0.005722655914723873,
+            0.006571666337549686,
+            0.01444950234144926,
+            0.007739055436104536,
+            0.004355259705334902,
+            -0.020033378154039383,
+            -0.01799248717725277,
+            0.0032674651592969894,
+            -0.01649855636060238,
+            -0.0048614004626870155,
+            0.0020531355403363705,
+            -0.019363965839147568,
+            0.01875986158847809,
+            -0.010743245482444763,
+            0.00017538899555802345,
+            -0.003226647386327386,
+            -0.020800752565264702,
+            0.006204306147992611,
+            -0.004865481983870268,
+            0.019870106130838394,
+            0.003555230563506484,
+            0.02785406820476055,
+            0.02310287579894066,
+            -0.004428731743246317,
+            0.017600636929273605,
+            -0.016155686229467392,
+            0.007983962073922157,
+            -0.031135819852352142,
+            -0.001782717532478273,
+            -0.010408539324998856,
+            0.0036103345919400454,
+            -0.015943434089422226,
+            -0.009126860648393631,
+            -0.0048777274787425995,
+            -0.0006678812787868083,
+            -0.004820582456886768,
+            0.0004275664687156677,
+            0.0027817331720143557,
+            0.02231917530298233,
+            -0.013208640739321709,
+            -0.02803366631269455,
+            -0.017437364906072617,
+            0.006045116577297449,
+            0.019396619871258736,
+            -0.026025431230664253,
+            -0.009184005670249462,
+            -0.005012426059693098,
+            -0.004326687194406986,
+            -0.021404854953289032,
+            -0.015167895704507828,
+            -0.005632856860756874,
+            -0.008045189082622528,
+            -0.010522829368710518,
+            0.0046532293781638145,
+            -0.0035715578123927116,
+            -0.011739199981093407,
+            0.021666089072823524,
+            0.018057795241475105,
+            -0.019837452098727226,
+            -0.019837452098727226,
+            0.013641309924423695,
+            -0.006159406621009111,
+            0.0018878233386203647,
+            0.02008235827088356,
+            0.00029286774224601686,
+            0.005824700463563204,
+            0.020882388576865196,
+            -0.0035225762985646725,
+            -0.01797616109251976,
+            0.003577680326998234,
+            -0.0018888438353314996,
+            -0.012122887186706066,
+            0.003212361130863428,
+            0.02773977816104889,
+            0.0008076822850853205,
+            0.023429417982697487,
+            -0.03056436963379383,
+            -0.011412657797336578,
+            0.006061443593353033,
+            0.02369065210223198,
+            0.00211844383738935,
+            -0.0021694661118090153,
+            0.020131340250372887,
+            0.03941367194056511,
+            0.005775718949735165,
+            0.0015755671774968505,
+            -0.04003410041332245,
+            0.01027792226523161,
+            0.009208496659994125,
+            0.024621298536658287,
+            0.008339077234268188,
+            0.0014653591206297278,
+            -0.012906589545309544,
+            -0.009755454957485199,
+            -0.0017939424142241478,
+            0.014702572487294674,
+            -0.023870250210165977,
+            0.0024755997583270073,
+            -0.03157665207982063,
+            0.005094061605632305,
+            0.0021959978621453047,
+            -0.009559528902173042,
+            0.0335359051823616,
+            0.00980443600565195,
+            -0.02065380848944187,
+            0.015151568688452244,
+            0.005767555441707373,
+            0.007330877240747213,
+            0.022449791431427002,
+            -0.008228869177401066,
+            0.030450081452727318,
+            -0.005049162078648806,
+            0.01213921420276165,
+            0.02742956392467022,
+            0.032588932663202286,
+            -0.013102514669299126,
+            -0.003959327004849911,
+            0.0006097159348428249,
+            -0.0002561317232903093,
+            0.0007995187188498676,
+            -0.008204378187656403,
+            0.0037389106582850218,
+            -0.014800535514950752,
+            0.012710663489997387,
+            -0.019543563947081566,
+            -0.0034286954905837774,
+            0.0038532004691660404,
+            0.01991908811032772,
+            0.0016265894519165158,
+            -0.033633869141340256,
+            0.02055584453046322,
+            -0.015420965850353241,
+            0.0135596739128232,
+            0.001290863030590117,
+            -0.011110605672001839,
+            -0.00029210239881649613,
+            0.020588500425219536,
+            0.010947334580123425,
+            -0.007845181971788406,
+            -0.0135596739128232,
+            -0.028376536443829536,
+            -0.021535472944378853,
+            0.00823703221976757,
+            0.009445239789783955,
+            -0.0028837777208536863,
+            0.0021306893322616816,
+            -0.011869817040860653,
+            0.0006520643946714699,
+            -0.0008122742874547839,
+            0.01740471087396145,
+            -0.0335359051823616,
+            0.021943651139736176,
+            -0.019527235999703407,
+            -0.019543563947081566,
+            0.017045514658093452,
+            0.008049271069467068,
+            0.009445239789783955,
+            0.0033552234526723623,
+            -0.01995174214243889,
+            -0.012326976284384727,
+            0.0338951013982296,
+            -0.01138000376522541,
+            0.006298186723142862,
+            0.004408322740346193,
+            0.012914752587676048,
+            -0.008914608508348465,
+            -0.011926962062716484,
+            0.01784554310142994,
+            -0.009861581027507782,
+            0.013951525092124939,
+            0.0010010566329583526,
+            -0.0254703089594841,
+            0.02739690989255905,
+            -0.019233347848057747,
+            0.03487472981214523,
+            -0.021617108955979347,
+            0.00711862463504076,
+            0.0066614653915166855,
+            0.008792154490947723,
+            -0.006690037902444601,
+            -0.013224967755377293,
+            0.0012786176521331072,
+            0.03807484731078148,
+            -0.02623768337070942,
+            0.024180466309189796,
+            0.022400809451937675,
+            -0.00040766780148260295,
+            0.02065380848944187,
+            -0.008367649279534817,
+            -0.026711169630289078,
+            0.0179598331451416,
+            -0.012596373446285725,
+            -0.026123393326997757,
+            0.013257622718811035,
+            -0.022678371518850327,
+            -0.020278284326195717,
+            -0.026302991434931755,
+            0.004947117529809475,
+            -0.0025694805663079023,
+            -0.005151206627488136,
+            0.009004407562315464,
+            0.02561725303530693,
+            0.020915042608976364,
+            0.019576217979192734,
+            0.02111096680164337,
+            0.01161674689501524,
+            0.022449791431427002,
+            -0.007559456862509251,
+            0.04401791840791702,
+            -0.004947117529809475,
+            0.004632820375263691,
+            -0.0015980169409886003,
+            -0.015657709911465645,
+            0.009771781973540783,
+            0.0119106350466609,
+            0.0029572497587651014,
+            -0.029176564887166023,
+            -0.009061552584171295,
+            -0.03660540655255318,
+            0.005053244065493345,
+            0.0020664012990891933,
+            0.0176822729408741,
+            0.032115448266267776,
+            -0.005865518003702164,
+            0.004592002835124731,
+            -0.005102225113660097,
+            -0.013927034102380276,
+            -0.009420748800039291,
+            -0.02739690989255905,
+            -0.025437654927372932,
+            -0.007653337903320789,
+            -0.013739272020757198,
+            -0.030368445441126823,
+            -0.011722872965037823,
+            0.01551892887800932,
+            -0.007645174395292997,
+            0.011918798089027405,
+            -0.016702644526958466,
+            0.01385356206446886,
+            -0.004110352601855993,
+            -0.004763437435030937,
+            -0.004530776292085648,
+            -0.017600636929273605,
+            -0.006420640274882317,
+            0.014367866329848766,
+            -0.009477893821895123,
+            -0.01237595733255148,
+            -0.02215590327978134,
+            -0.0015459742862731218,
+            -0.01574750803411007,
+            0.003489922033622861,
+            -0.014596446417272091,
+            -0.006771673448383808,
+            -0.003908304497599602,
+            -0.018514955416321754,
+            0.01766594499349594,
+            0.014816862531006336,
+            -0.019380291923880577,
+            -0.0090697156265378,
+            0.02638462744653225,
+            0.042777057737112045,
+            -0.03218075633049011,
+            -0.008200296200811863,
+            0.0020378287881612778,
+            0.018090449273586273,
+            0.010171796195209026,
+            0.016702644526958466,
+            -0.024653952568769455,
+            0.0022286521270871162,
+            -0.021649762988090515,
+            -0.001220452249981463,
+            -0.023739634081721306,
+            0.010775899514555931,
+            0.025715215131640434,
+            -0.010971825569868088,
+            0.025976449251174927,
+            0.01634344831109047,
+            -0.01812310516834259,
+            0.012016761116683483,
+            0.007183933164924383,
+            0.014645427465438843,
+            -0.019755816087126732,
+            0.0167679525911808,
+            -0.015796490013599396,
+            0.031233781948685646,
+            0.00892277155071497,
+            -0.04829562455415726,
+            -0.03503800183534622,
+            -0.013388238847255707,
+            0.0013388239312916994,
+            -0.010702427476644516,
+            0.015037278644740582,
+            0.015967924147844315,
+            0.013959688134491444,
+            -0.019380291923880577,
+            -0.015820980072021484,
+            0.011984107084572315,
+            0.010065670125186443,
+            0.027462217956781387,
+            -0.006649219896644354,
+            -0.004212397150695324,
+            -0.00960034690797329,
+            -0.02249877341091633,
+            -0.016098542138934135,
+            0.0209803506731987,
+            -0.018041469156742096,
+            0.015935270115733147,
+            -0.026139721274375916,
+            -0.012449429370462894,
+            0.017453692853450775,
+            -0.05211617052555084,
+            -0.01453113742172718,
+            0.0007612520130351186,
+            0.03012353740632534,
+            0.004751192405819893,
+            0.0035184945445507765,
+            0.01841699331998825,
+            -0.01993541419506073,
+            -0.0039776950143277645,
+            -0.02099667675793171,
+            0.006832899991422892,
+            -0.011853490024805069,
+            -0.013984179124236107,
+            0.014482156373560429,
+            -0.013845399022102356,
+            -0.01601690612733364,
+            0.0034368589986115694,
+            0.007183933164924383,
+            -0.021094640716910362,
+            0.011951452121138573,
+            0.003945040516555309,
+            0.012171869166195393,
+            0.012171869166195393,
+            0.01199227012693882,
+            0.021551799029111862,
+            0.02295593172311783,
+            -0.007334959227591753,
+            -0.004767519421875477,
+            -0.01004117913544178,
+            -0.0015398516552522779,
+            0.008898280560970306,
+            0.0033797139767557383,
+            -0.002989904023706913,
+            0.007837017998099327,
+            0.010033016093075275,
+            -0.01498013362288475,
+            0.033193036913871765,
+            -0.0035878848284482956,
+            -0.04134026914834976,
+            -0.003238892648369074,
+            -0.03800953924655914,
+            0.020033378154039383,
+            -0.023298801854252815,
+            -0.016996532678604126,
+            0.0032776694279164076,
+            -0.00629410520195961,
+            -0.019902760162949562,
+            0.013608654960989952,
+            0.011404493823647499,
+            0.00258580781519413,
+            0.02021297626197338,
+            0.012269831262528896,
+            -0.005134879611432552,
+            0.026172375306487083,
+            -0.014718899503350258,
+            -0.008751336485147476,
+            0.015567909926176071,
+            0.006853308994323015,
+            0.0022347746416926384,
+            0.0011031010653823614,
+            -0.018825169652700424,
+            0.005098143592476845,
+            0.0164903923869133,
+            0.007216587662696838,
+            -0.000566347036510706,
+            0.014049487188458443,
+            0.012514738366007805,
+            -0.015135240741074085,
+            -0.0002213090192526579,
+            -0.026564225554466248,
+            0.0032817514147609472,
+            0.004567512311041355,
+            0.007506393827497959,
+            -0.006771673448383808,
+            0.006281859707087278,
+            0.00658799335360527,
+            0.014759717509150505,
+            -0.010155469179153442,
+            -0.016980206593871117,
+            -0.002510294783860445,
+            0.004285869188606739,
+            0.016694480553269386,
+            0.03157665207982063,
+            -0.010310577228665352,
+            -0.005734901409596205,
+            -0.0386626236140728,
+            0.02070278860628605,
+            0.03332365304231644,
+            0.011175914667546749,
+            -0.028213264420628548,
+            0.012302485294640064,
+            0.0006515541463159025,
+            0.0019276207312941551,
+            -0.026302991434931755,
+            0.02787039615213871,
+            0.00974729098379612,
+            0.0066614653915166855,
+            0.017861871048808098,
+            0.0022306928876787424,
+            0.0029266364872455597,
+            -0.016302630305290222,
+            0.01018812321126461,
+            0.009086042642593384,
+            -0.017306748777627945,
+            0.0009745250572450459,
+            0.02324981987476349,
+            -0.01423724927008152,
+            -0.003208279376849532,
+            0.0018133309204131365,
+            0.02651524543762207,
+            -0.013820908032357693,
+            -0.014294394291937351,
+            0.0005714492290280759,
+            0.01751900091767311,
+            -0.0007979880319908261,
+            0.012098397128283978,
+            0.04898136481642723,
+            -0.012424939312040806,
+            0.019657853990793228,
+            0.005461421795189381,
+            -0.015290348790585995,
+            -0.032605260610580444,
+            -0.004922627005726099,
+            -0.01830270327627659,
+            0.011200404725968838,
+            0.014343376271426678,
+            -0.015763835981488228,
+            0.0007525782566517591,
+            -0.03144603595137596,
+            0.007743136957287788,
+            0.014735226519405842,
+            0.018776189535856247,
+            -0.002440904499962926,
+            0.006134915631264448,
+            0.027674470096826553,
+            0.02458864450454712,
+            -0.007392104249447584,
+            0.013804581016302109,
+            0.02320083975791931,
+            -0.025062130764126778,
+            -0.003069498809054494,
+            0.006820654962211847,
+            0.004104230087250471,
+            0.0053063142113387585,
+            -0.016963878646492958,
+            0.03915243595838547,
+            -0.005922663025557995,
+            -0.01506176870316267,
+            0.0030266400426626205,
+            -0.009028897620737553,
+            0.0063390047289431095,
+            -0.012563719414174557,
+            0.012637191452085972,
+            -0.008114579133689404,
+            0.00022883480414748192,
+            -0.0059430720284581184,
+            -0.011110605672001839,
+            -0.012653518468141556,
+            -0.0007949267164804041,
+            -0.016531210392713547,
+            0.007432921789586544,
+            0.00981259997934103,
+            0.005045080557465553,
+            0.01025343220680952,
+            -0.0036797248758375645,
+            -0.003136848099529743,
+            -0.016751626506447792,
+            -0.009984034113585949,
+            0.0073716952465474606,
+            0.01872720755636692,
+            -0.028588788583874702,
+            -0.018547609448432922,
+            0.009290131740272045,
+            0.0022898786701261997,
+            0.018776189535856247,
+            0.0158944521099329,
+            -0.0002595757250674069,
+            0.012318813242018223,
+            -0.01964152604341507,
+            -0.009184005670249462,
+            0.009690145961940289,
+            -0.003569516818970442,
+            0.01048201136291027,
+            -0.014759717509150505,
+            0.010033016093075275,
+            -0.011159587651491165,
+            -0.011053460650146008,
+            0.01385356206446886,
+            0.01738838478922844,
+            0.008873790502548218,
+            -0.001294944784604013,
+            3.383413059054874e-05,
+            -0.010506502352654934,
+            -0.024033522233366966,
+            0.004771600943058729,
+            0.027021385729312897,
+            -0.013273949734866619,
+            0.003883813973516226,
+            0.008702355436980724,
+            -0.017502672970294952,
+            -0.019821124151349068,
+            -0.012653518468141556,
+            -0.022106921300292015,
+            0.020572172477841377,
+            0.023413091897964478,
+            -0.02026195637881756,
+            0.00877582747489214,
+            -0.004620575346052647,
+            0.014049487188458443,
+            -0.01400050614029169,
+            0.010612628422677517,
+            0.0317889042198658,
+            -0.020000724121928215,
+            -0.0028654097113758326,
+            0.0036633978597819805,
+            -0.015347493812441826,
+            0.018629245460033417,
+            0.03268689662218094,
+            -0.00176945177372545,
+            0.014041324146091938,
+            0.0008138049161061645,
+            -0.009118696674704552,
+            0.01905374974012375,
+            0.011094278655946255,
+            0.005347132217139006,
+            -0.0042491331696510315,
+            -0.012212686240673065,
+            -0.03010721132159233,
+            0.016702644526958466,
+            0.0129637336358428,
+            -0.0037674831692129374,
+            -0.0011551438365131617,
+            -0.006343086715787649,
+            -0.015404638834297657,
+            0.0035123717971146107,
+            -0.013078023679554462,
+            -0.00803294312208891,
+            -0.008759500458836555,
+            0.004314441699534655,
+            -0.0034960447810590267,
+            -0.009053388610482216,
+            0.007236996199935675,
+            0.0009510547970421612,
+            -0.0010602424154058099,
+            -0.01902109570801258,
+            -0.025078458711504936,
+            -0.00995954405516386,
+            -0.016849588602781296,
+            0.0038287099450826645,
+            0.0005234883283264935,
+            -0.027184655889868736,
+            0.0053961132653057575,
+            -0.008139070123434067,
+            -0.008134988136589527,
+            0.007126788143068552,
+            0.006881881505250931,
+            -0.007437003776431084,
+            -0.010816717520356178,
+            -0.00786150898784399,
+            -0.006865554489195347,
+            0.020033378154039383,
+            0.0034348180051892996,
+            0.012082069180905819,
+            -0.010498338378965855,
+            -0.018025141209363937,
+            -0.016963878646492958,
+            0.016718972474336624,
+            -0.008971752598881721,
+            -0.02083340659737587,
+            -0.01444950234144926,
+            -0.008424794301390648,
+            -0.007665583398193121,
+            -0.01527402177453041,
+            -0.021159948781132698,
+            -0.0032103201374411583,
+            -0.011347348801791668,
+            -0.0067431009374558926,
+            -0.0028960229828953743,
+            0.0013571919407695532,
+            0.00951871182769537,
+            0.00546958576887846,
+            0.00438791373744607,
+            0.021666089072823524,
+            0.010122815147042274,
+            -0.005461421795189381,
+            0.016441410407423973,
+            -0.006183897145092487,
+            -0.004440976772457361,
+            -0.01498013362288475,
+            -0.009918726049363613,
+            0.018547609448432922,
+            -0.0055226488038897514,
+            0.008457448333501816,
+            -0.0009536059224046767,
+            -0.00576347392052412,
+            0.015674035996198654,
+            0.0129637336358428,
+            0.016718972474336624,
+            0.004771600943058729,
+            -0.00012168807734269649,
+            0.012416775338351727,
+            -0.0015745466807857156,
+            0.012555556371808052,
+            -0.0038429962005466223,
+            0.014506647363305092,
+            -0.009722800925374031,
+            -0.018988441675901413,
+            0.004157293122261763,
+            0.006265532691031694,
+            -0.012531065382063389,
+            0.009404421783983707,
+            0.007743136957287788,
+            -0.013747435994446278,
+            -0.02261306345462799,
+            -0.000106317616882734,
+            0.00014362763613462448,
+            0.008122743107378483,
+            0.008971752598881721,
+            -0.008073761127889156,
+            0.012890261597931385,
+            0.004616493359208107,
+            -0.007179851643741131,
+            -0.019657853990793228,
+            0.026009103283286095,
+            0.015608727931976318,
+            0.012269831262528896,
+            -0.052508022636175156,
+            -0.0030980713199824095,
+            0.010743245482444763,
+            -0.015951598063111305,
+            -0.026792805641889572,
+            -0.006404313258826733,
+            -0.008759500458836555,
+            -0.04290767386555672,
+            -0.009624837897717953,
+            -0.0005724696675315499,
+            -0.011388166807591915,
+            0.010400376282632351,
+            0.01573934406042099,
+            0.014612773433327675,
+            0.005489994306117296,
+            -0.007232914678752422,
+            -0.017633290961384773,
+            -0.02788672223687172,
+            0.010057506151497364,
+            -0.013380075804889202,
+            -0.006412476766854525,
+            -0.002263347152620554,
+            -0.007690073922276497,
+            0.0016112826997414231,
+            0.01691489666700363,
+            0.042483169585466385,
+            0.023037567734718323,
+            -0.012335140258073807,
+            -0.020131340250372887,
+            -0.012800462543964386,
+            -0.009192168712615967,
+            0.00633492274209857,
+            0.011592255905270576,
+            -0.004738946910947561,
+            0.0096656559035182,
+            -0.004294032696634531,
+            0.012302485294640064,
+            0.012767808511853218,
+            0.01680060662329197,
+            -2.3023791072773747e-05,
+            0.005579793825745583,
+            -0.0024694770108908415,
+            -0.01738838478922844,
+            -0.004379750229418278,
+            0.008261523209512234,
+            -0.0008872769540175796,
+            0.0019317024853080511,
+            0.014204595237970352,
+            0.019102731719613075,
+            -0.009094206616282463,
+            -0.014612773433327675,
+            -0.016098542138934135,
+            -0.005277741700410843,
+            -0.0012622905196622014,
+            0.029927613213658333,
+            0.02037624642252922,
+            -0.019266001880168915,
+            -0.018237393349409103,
+            -0.018351683393120766,
+            0.00922482367604971,
+            -0.003563394071534276,
+            -0.0064369672909379005,
+            0.01905374974012375,
+            0.023021239787340164,
+            0.017910851165652275,
+            -0.003642988856881857,
+            -0.014865843579173088,
+            0.003804219188168645,
+            0.02847449854016304,
+            0.008367649279534817,
+            0.01078406348824501,
+            0.008726846426725388,
+            -0.0011908593587577343,
+            -0.032441988587379456,
+            -0.0113146947696805,
+            0.00951871182769537,
+            -0.008481939323246479,
+            0.026564225554466248,
+            -0.016531210392713547,
+            0.009388094767928123,
+            -0.005077734589576721,
+            0.0019286411115899682,
+            -0.02365799807012081,
+            0.0014929111348465085,
+            0.010122815147042274,
+            -0.024947840720415115,
+            0.00336950970813632,
+            -0.023592690005898476,
+            -0.00013763252354692668,
+            0.008367649279534817,
+            0.0022837561555206776,
+            0.01650671847164631,
+            -0.0113146947696805,
+            0.0051838611252605915,
+            0.02099667675793171,
+            -0.002832755446434021,
+            -0.01512707769870758,
+            -0.0007842120248824358,
+            0.0005092020728625357,
+            -0.001286781276576221,
+            0.02563357912003994,
+            0.015812816098332405,
+            -0.0036756431218236685,
+            0.020882388576865196,
+            0.008677864447236061,
+            0.0019133344758301973,
+            -0.021764053031802177,
+            0.0023796779569238424,
+            -0.032899148762226105,
+            0.017437364906072617,
+            -0.011126932688057423,
+            0.012490247376263142,
+            0.00868602842092514,
+            -0.01341272983700037,
+            -0.0007388022495433688,
+            -0.0107187544927001,
+            -0.0015725058037787676,
+            -0.007934981025755405,
+            0.009779945947229862,
+            -0.004363423213362694,
+            0.008339077234268188,
+            -0.00024579971795901656,
+            -0.01476788055151701,
+            0.018694553524255753,
+            9.678283277025912e-06,
+            -0.00010925139940809458,
+            0.005441013257950544,
+            0.009935053065419197,
+            0.00786559097468853,
+            0.002009256277233362,
+            0.023870250210165977,
+            -0.031119491904973984,
+            -0.016082214191555977,
+            -0.010204450227320194,
+            0.005543057806789875,
+            -0.007922735065221786,
+            -0.018531281501054764,
+            -0.03257260471582413,
+            0.01318415068089962,
+            0.001293924287892878,
+            -0.0009806477464735508,
+            0.01258004643023014,
+            -0.00942891277372837,
+            -0.0013878053287044168,
+            -0.01606588624417782,
+            -0.014808698557317257,
+            0.006416558753699064,
+            0.0033633869607001543,
+            -0.01222901325672865,
+            -0.023739634081721306,
+            0.014759717509150505,
+            0.0007418635650537908,
+            -0.016980206593871117,
+            -0.017502672970294952,
+            0.00018457300029695034,
+            -0.004804255440831184,
+            0.02204161323606968,
+            -0.016678154468536377,
+            0.003373591462150216,
+            0.0007219648687168956,
+            0.009232986718416214,
+            -0.02889900468289852,
+            0.00716352416202426,
+            0.0006107363733462989,
+            0.0035164537839591503,
+            -0.006518602836877108,
+            -0.021013004705309868,
+            0.011371839791536331,
+            -0.0020674217958003283,
+            -0.014751553535461426,
+            -0.020751770585775375,
+            -0.0020796670578420162,
+            -0.007420676294714212,
+            -0.02724996581673622,
+            0.017910851165652275,
+            0.0009985055075958371,
+            -0.01738838478922844,
+            0.008077843114733696,
+            0.006596156861633062,
+            0.009355440735816956,
+            0.02455599047243595,
+            0.010147306136786938,
+            -0.018776189535856247,
+            -0.0008087027235887945,
+            0.006979844067245722,
+            0.009306458756327629,
+            0.0036450298503041267,
+            0.04222193360328674,
+            -0.008130906149744987,
+            -0.006579829845577478,
+            0.0066614653915166855,
+            -0.008449285291135311,
+            -0.013747435994446278,
+            0.026727497577667236,
+            0.005796127952635288,
+            0.015984252095222473,
+            0.016123032197356224,
+            -0.00945340283215046,
+            -0.005008344538509846,
+            -0.0018786393338814378,
+            0.02065380848944187,
+            0.0073798587545752525,
+            -0.005502239800989628,
+            -0.009714636951684952,
+            0.005881845485419035,
+            -0.00898808054625988,
+            0.0021653843577951193,
+            -0.009526874870061874,
+            -0.005824700463563204,
+            -0.008069679141044617,
+            0.016400592401623726,
+            -0.0003523086488712579,
+            0.027315273880958557,
+            0.020751770585775375,
+            -0.008563575334846973,
+            -0.0005683879135176539,
+            0.017208784818649292,
+            -0.005820618476718664,
+            0.015608727931976318,
+            -0.011192241683602333,
+            -0.007706400938332081,
+            0.006408394780009985,
+            0.024017194285988808,
+            0.012237177230417728,
+            -0.0056042843498289585,
+            0.0002885308349505067,
+            -0.0030348035506904125,
+            -0.010612628422677517,
+            0.013388238847255707,
+            0.015323002822697163,
+            -0.01947825588285923,
+            -0.004722619894891977,
+            -0.0016592437168583274,
+            -0.003300119424238801,
+            0.00945340283215046,
+            0.02305389568209648,
+            0.000171817431692034,
+            0.009184005670249462,
+            0.0012765767751261592,
+            0.03862996771931648,
+            -0.0053144777193665504,
+            0.03102152980864048,
+            -0.0010423846542835236,
+            -0.0059430720284581184,
+            -0.009094206616282463,
+            -0.020441556349396706,
+            -0.011486129835247993,
+            -0.00041481092921458185,
+            -0.005040998570621014,
+            -0.0004956811899319291,
+            -0.020327266305685043,
+            0.0026960158720612526,
+            0.014637264423072338,
+            0.007661501411348581,
+            0.017927179113030434,
+            -0.002889900468289852,
+            0.017437364906072617,
+            0.019690508022904396,
+            -0.01400050614029169,
+            -0.029862305149435997,
+            -0.0006515541463159025,
+            0.01498013362288475,
+            -0.01661284640431404,
+            0.019445601850748062,
+            0.004085862077772617,
+            0.0029531680047512054,
+            0.004779764451086521,
+            -0.006049198564141989,
+            0.018482301384210587,
+            0.02039257436990738,
+            0.01453930139541626,
+            -0.013167822733521461,
+            0.014057651162147522,
+            0.009771781973540783,
+            0.006600238848477602,
+            -0.01646590232849121,
+            -0.013535182923078537,
+            0.002555194543674588,
+            0.0009245232213288546,
+            0.024392718449234962,
+            -0.020588500425219536,
+            0.0016602640971541405,
+            0.010588138364255428,
+            0.01769859902560711,
+            -0.004302196204662323,
+            0.008906444534659386,
+            0.0008740111952647567,
+            0.007134951651096344,
+            -0.0035919665824621916,
+            -0.006457376293838024,
+            -0.006673710886389017,
+            -0.010392212308943272,
+            -0.0020959940738976,
+            -0.02070278860628605,
+            0.002198038622736931,
+            -0.002830714685842395,
+            0.018237393349409103,
+            -0.01018812321126461,
+            0.038140155375003815,
+            0.03399306535720825,
+            -0.0007025764207355678,
+            0.011102442629635334,
+            -0.01709449477493763,
+            -0.003945040516555309,
+            -0.021959977224469185,
+            0.01844964735209942,
+            0.016392430290579796,
+            0.012269831262528896,
+            0.005302232690155506,
+            0.017796561121940613,
+            -0.0006301248213276267,
+            0.014490320347249508,
+            0.0029654132667928934,
+            0.007971717044711113,
+            -0.002265388146042824,
+            0.003938918001949787,
+            -0.031103165820240974,
+            -0.017029186710715294,
+            -0.01138000376522541,
+            0.004796091932803392,
+            -0.006375740747898817,
+            -0.0035205355379730463,
+            0.0084737753495574,
+            -0.005902254022657871,
+            0.0004270562494639307,
+            -0.025992777198553085,
+            -0.0157148540019989,
+            -0.010318740271031857,
+            -0.004040962550789118,
+            -0.01169838197529316,
+            -0.0027694879099726677,
+            -0.0022082431241869926,
+            0.0029552089981734753,
+            0.020947696641087532,
+            -0.012808626517653465,
+            -0.016294466331601143,
+            0.004379750229418278,
+            -0.002263347152620554,
+            -0.014016833156347275,
+            -0.016898570582270622,
+            0.006665547378361225,
+            0.012098397128283978,
+            -0.0017470018938183784,
+            -0.013608654960989952,
+            0.011739199981093407,
+            -0.006628811359405518,
+            0.04663025587797165,
+            -0.013584164902567863,
+            -0.0078002819791436195,
+            0.003720542648807168,
+            0.004698128905147314,
+            -0.009396257810294628,
+            0.007330877240747213,
+            0.013020878657698631,
+            0.01830270327627659,
+            0.0095758568495512,
+            -0.008465612307190895,
+            -0.0024388637393712997,
+            0.007796200457960367,
+            -0.016147522255778313,
+            0.006404313258826733,
+            0.0008525818702764809,
+            -0.014400520361959934,
+            0.006110425107181072,
+            0.028980640694499016,
+            0.013151495717465878,
+            0.01288209855556488,
+            -0.022351829335093498,
+            -0.004094025585800409,
+            -0.009355440735816956,
+            -0.004943036008626223,
+            -0.010090161114931107,
+            0.027070365846157074,
+            -0.018155759200453758,
+            0.02651524543762207,
+            0.00898808054625988,
+            0.0017684312770143151,
+            -0.01176369097083807,
+            0.006343086715787649,
+            0.02832755446434021,
+            0.0028592869639396667,
+            -0.020000724121928215,
+            -0.013592327944934368,
+            0.006314514204859734,
+            0.01707816869020462,
+            -0.002647034591063857,
+            -0.005428767763078213,
+            0.019657853990793228,
+            0.01784554310142994,
+            -0.004334850702434778,
+            -0.00936360377818346,
+            0.0008337036124430597,
+            0.00928196869790554,
+            0.016963878646492958,
+            0.015020951628684998,
+            0.009086042642593384,
+            -0.002647034591063857,
+            0.008037025108933449,
+            -0.008800318464636803,
+            -0.016449574381113052,
+            0.007220669183880091,
+            -0.008432958275079727,
+            -0.016963878646492958,
+            0.01872720755636692,
+            0.015257694758474827,
+            0.0181394312530756,
+            0.0024592725094407797,
+            -0.016735298559069633,
+            0.022302847355604172,
+            0.005755309946835041,
+            0.002998067531734705,
+            0.02160078100860119,
+            -0.022221211344003677,
+            0.013012715615332127,
+            0.02037624642252922,
+            0.027609162032604218,
+            -0.01737205684185028,
+            -0.010906516574323177,
+            -0.004857318475842476,
+            -0.006853308994323015,
+            -0.00176945177372545,
+            -0.008253359235823154,
+            -0.012620864436030388,
+            0.0028735732194036245,
+            -0.030466407537460327,
+            0.018106777220964432,
+            -0.005102225113660097,
+            0.010669773444533348,
+            -0.019266001880168915,
+            0.011363675817847252,
+            -0.0035838030744343996,
+            0.011135096661746502,
+            0.0019612954929471016,
+            -0.030041903257369995,
+            -0.008571738377213478,
+            0.015388311818242073,
+            -0.0030368445441126823,
+            -0.013641309924423695,
+            0.02006603218615055,
+            0.00018942011229228228,
+            -0.022270193323493004,
+            -0.004314441699534655,
+            -0.006934944540262222,
+            0.002193956868723035,
+            -0.0029613315127789974,
+            -0.0052981507033109665,
+            -0.030188847333192825,
+            0.020033378154039383,
+            0.009559528902173042,
+            -0.03072764165699482,
+            -0.003255219664424658,
+            -0.014996460638940334,
+            -0.0025184585247188807,
+            0.008881953544914722,
+            -0.004000144544988871,
+            -0.015551582910120487,
+            0.01784554310142994,
+            0.022711025550961494,
+            -0.0023164101876318455,
+            -0.016057724133133888,
+            -0.0090697156265378,
+            -0.027494871988892555,
+            0.030825603753328323,
+            -0.0066696288995444775,
+            0.0022878379095345736,
+            -0.005322641227394342,
+            0.009388094767928123,
+            0.004228724166750908,
+            0.0157148540019989,
+            0.006081852596253157,
+            -0.002647034591063857,
+            0.026596879586577415,
+            -0.01964152604341507,
+            0.009567692875862122,
+            -1.588864870427642e-05,
+            -0.003308282932266593,
+            0.01890680566430092,
+            0.0007939062779769301,
+            0.021649762988090515,
+            -0.008571738377213478,
+            -0.0001475818717153743,
+            -0.0034388997592031956,
+            0.010898353531956673,
+            -0.0028450009413063526,
+            -0.003465431509539485,
+            -0.001531688030809164,
+            0.011118769645690918,
+            0.01013097818940878,
+            0.003804219188168645,
+            -0.01737205684185028,
+            0.003142970846965909,
+            -0.008547247387468815,
+            -0.007028825581073761,
+            0.01302904263138771,
+            -0.011192241683602333,
+            -0.0018990482203662395,
+            -0.03160930797457695,
+            0.016286302357912064,
+            0.0533243790268898,
+            0.008547247387468815,
+            -0.014433175325393677,
+            -0.0018806802108883858,
+            -0.019755816087126732,
+            0.01535565685480833,
+            0.011004479601979256,
+            -0.016816934570670128,
+            -0.011984107084572315,
+            -0.012310649268329144,
+            0.007649256382137537,
+            0.021306892856955528,
+            -0.006522684823721647,
+            -0.008579902350902557,
+            0.025372346863150597,
+            -0.011788181029260159,
+            -0.010882026515901089,
+            -0.01661284640431404,
+            -0.01635977439582348,
+            0.00241641397587955,
+            0.019902760162949562,
+            -0.0169475506991148,
+            0.021176276728510857,
+            0.00936360377818346,
+            0.0010133018950000405,
+            0.003624620847404003,
+            0.008588065393269062,
+            0.0024878450203686953,
+            0.017878197133541107,
+            0.003883813973516226,
+            -0.01439235731959343,
+            0.0021408938337117434,
+            -0.0030286810360848904,
+            0.004624656867235899,
+            3.4320437407586724e-06,
+            -0.002749078907072544,
+            -0.0065635028295218945,
+            0.003710338147357106,
+            -0.021306892856955528,
+            0.021502818912267685,
+            -0.022678371518850327,
+            -0.004245051648467779,
+            0.005555302836000919,
+            -0.005118552595376968,
+            0.013469874858856201,
+            0.018792515620589256,
+            0.01423724927008152,
+            0.008702355436980724,
+            0.01462093647569418,
+            -0.011331021785736084,
+            0.02323349379003048,
+            -0.013282112777233124,
+            0.0005816536722704768,
+            0.012939243577420712,
+            0.01755165494978428,
+            0.018792515620589256,
+            -0.006245123688131571,
+            -0.010384049266576767,
+            -0.0002946535241790116,
+            0.0129637336358428,
+            -0.022286519408226013,
+            -0.012269831262528896,
+            -0.01386172603815794,
+            0.011486129835247993,
+            -0.004743028432130814,
+            0.025453981012105942,
+            -0.00633492274209857,
+            -0.014653591439127922,
+            -0.015918942168354988,
+            0.014196431264281273,
+            -0.048752784729003906,
+            0.0030348035506904125,
+            -0.0024919267743825912,
+            0.03043375350534916,
+            -0.027021385729312897,
+            0.01738838478922844,
+            -0.0028490826953202486,
+            -0.009355440735816956,
+            0.015429128892719746,
+            -0.015992414206266403,
+            -0.0046613928861916065,
+            -0.0016806729836389422,
+            -0.007539048325270414,
+            -0.00112248957157135,
+            0.017616963014006615,
+            -0.015845470130443573,
+            0.01282495353370905,
+            -0.00989423505961895,
+            0.0027021386194974184,
+            -0.004526694305241108,
+            -0.007751300930976868,
+            0.0196088720113039,
+            0.034548189491033554,
+            -0.007053316105157137,
+            0.0014990337658673525,
+            -0.022547753527760506,
+            0.013192313723266125,
+            -0.006730855442583561,
+            0.02351105399429798,
+            -0.005432849284261465,
+            -0.016898570582270622,
+            -0.013976015150547028,
+            -0.009739127941429615,
+            0.00839214026927948,
+            -0.022433463484048843,
+            -0.011110605672001839,
+            -0.00922482367604971,
+            -0.002573562553152442,
+            -0.018694553524255753,
+            0.004865481983870268,
+            -0.002426618244498968,
+            -0.012506574392318726,
+            -0.008816645480692387,
+            0.01048201136291027,
+            -0.009347276762127876,
+            0.003077662317082286,
+            -0.0011929002357646823,
+            -0.011453475803136826,
+            -0.02369065210223198,
+            0.0158944521099329,
+            0.0001974561164388433,
+            -0.010359558276832104,
+            0.008408467285335064,
+            -0.0018276170594617724,
+            0.001211268245242536,
+            0.016963878646492958,
+            -0.0020664012990891933,
+            0.003467472270131111,
+            0.023478399962186813,
+            0.004057289566844702,
+            -0.010971825569868088,
+            -0.00025906547671183944,
+            0.013535182923078537,
+            -0.005118552595376968,
+            -0.010008525103330612,
+            1.6646010408294387e-05,
+            -0.02984597720205784,
+            -0.008351322263479233,
+            0.006094098091125488,
+            0.018629245460033417,
+            -0.010620792396366596,
+            -0.015853634104132652,
+            0.026319319382309914,
+            -0.006359413731843233,
+            0.019070077687501907,
+            0.002889900468289852,
+            0.0025694805663079023,
+            0.01724144071340561,
+            0.013461710885167122,
+            0.00786150898784399,
+            -0.013927034102380276,
+            -0.02429475635290146,
+            0.0016398552106693387,
+            -0.018090449273586273,
+            0.010579974390566349,
+            -0.011028970591723919,
+            0.0032756286673247814,
+            -0.02398454025387764,
+            -0.012237177230417728,
+            0.004934872500598431,
+            -0.020343592390418053,
+            -0.00040588201954960823,
+            -0.0056940834037959576,
+            0.022237539291381836,
+            -0.005310396198183298,
+            0.008669701404869556,
+            0.0018255761824548244,
+            -0.012237177230417728,
+            0.0009877908742055297,
+            0.009469729848206043,
+            -0.029192892834544182,
+            -0.0007127808639779687,
+            -0.003138889092952013,
+            0.0011418780777603388,
+            0.005351213738322258,
+            0.009755454957485199,
+            0.007322713732719421,
+            0.0027062203735113144,
+            0.007253323681652546,
+            0.010449357330799103,
+            -0.00022041612828616053,
+            0.0197884701192379,
+            0.018172085285186768,
+            -0.007351286243647337,
+            0.008710519410669804,
+            0.024817224591970444,
+            -0.013159659691154957,
+            -0.008718682453036308,
+            0.023266147822141647,
+            -0.006461458280682564,
+            -0.0011122850701212883,
+            -0.01483318954706192,
+            0.00815947912633419,
+            0.0017367975087836385,
+            0.028131630271673203,
+            0.019657853990793228,
+            0.005820618476718664,
+            0.000639819074422121,
+            0.015788326039910316,
+            -0.013020878657698631,
+            0.0035286990460008383,
+            0.015461783856153488,
+            -0.003940958995372057,
+            -0.019821124151349068,
+            0.013878053054213524,
+            0.009502384811639786,
+            -0.003228688146919012,
+            0.01551892887800932,
+            0.011412657797336578,
+            -0.030841931700706482,
+            -0.0011898389784619212,
+            0.010498338378965855,
+            -0.0021449755877256393,
+            -0.008751336485147476,
+            -0.00014273475972004235,
+            -0.012384121306240559,
+            0.021388528868556023,
+            -0.004408322740346193,
+            0.012277995236217976,
+            -0.00815947912633419,
+            -0.017127150669693947,
+            -0.011559601873159409,
+            0.007914572022855282,
+            -0.010547320358455181,
+            -0.0006714528426527977,
+            -0.004022594541311264,
+            -0.005779800936579704,
+            0.017747581005096436,
+            0.004706292413175106,
+            -0.0008964610169641674,
+            -0.0018398624379187822,
+            0.007983962073922157,
+            -0.006979844067245722,
+            -0.008726846426725388,
+            0.004816500470042229,
+            -0.012833117507398129,
+            0.003981776535511017,
+            0.005543057806789875,
+            0.01506176870316267,
+            -0.014857680536806583,
+            -0.021159948781132698,
+            0.007020662073045969,
+            0.02695607766509056,
+            -0.02516009286046028,
+            -0.005232842173427343,
+            0.006245123688131571,
+            0.002093953313305974,
+            0.006914535537362099,
+            -0.006934944540262222,
+            -0.007208424154669046,
+            -0.03657275065779686,
+            -0.0020745648071169853,
+            0.0009005427709780633,
+            0.008963589556515217,
+            -0.008049271069467068,
+            -0.0136494729667902,
+            -0.011135096661746502,
+            0.002410291228443384,
+            0.007134951651096344,
+            -0.02204161323606968,
+            0.0049593630246818066,
+            0.009608510881662369,
+            -0.019559891894459724,
+            0.025127438828349113,
+            -0.010890189558267593,
+            -6.964537897147238e-05,
+            -0.00995954405516386,
+            0.021633435040712357,
+            -0.013486201874911785,
+            -0.018368011340498924,
+            -0.0006898208521306515,
+            0.008355404250323772,
+            0.011926962062716484,
+            -0.0017663904000073671,
+            -0.011371839791536331,
+            0.0003752686607185751,
+            0.01646590232849121,
+            0.019135385751724243,
+            0.002744997153058648,
+            -0.0024837632663547993,
+            -0.01019628718495369,
+            -0.009592183865606785,
+            -0.0073961857706308365,
+            -0.0007449248805642128,
+            0.002091912319883704,
+            0.02035992033779621,
+            0.013469874858856201,
+            0.0024837632663547993,
+            -0.015135240741074085,
+            -0.005257333163172007,
+            0.008939098566770554,
+            0.009412584826350212,
+            -0.011690218932926655,
+            -0.00898808054625988,
+            0.002659279853105545,
+            0.02249877341091633,
+            0.017633290961384773,
+            -0.0014000505907461047,
+            0.0032878739293664694,
+            -0.0016592437168583274,
+            0.012506574392318726,
+            -0.007555375341325998,
+            -0.010269759222865105,
+            -0.006943108048290014,
+            -0.014645427465438843,
+            -0.010800390504300594,
+            -0.0023062059190124273,
+            -0.010179960168898106,
+            0.022629389539361,
+            0.007988044060766697,
+            0.009779945947229862,
+            0.008265605196356773,
+            0.010269759222865105,
+            0.002087830565869808,
+            0.025715215131640434,
+            -0.004424649756401777,
+            -0.011984107084572315,
+            0.007236996199935675,
+            -0.022368155419826508,
+            -0.021535472944378853,
+            0.008257441222667694,
+            0.006367577239871025,
+            0.03595232218503952,
+            -0.026890767738223076,
+            0.0052981507033109665,
+            -0.009779945947229862,
+            -0.010400376282632351,
+            0.007996207103133202,
+            -0.0015439334092661738,
+            -0.004245051648467779,
+            0.022368155419826508,
+            0.00726556871086359,
+            0.0035797213204205036,
+            0.0032817514147609472,
+            0.019837452098727226,
+            -0.005685919895768166,
+            0.01573934406042099,
+            0.0259274672716856,
+            0.024621298536658287,
+            0.009388094767928123,
+            -0.009469729848206043,
+            0.02785406820476055,
+            0.0002895512734539807,
+            -0.0034327772445976734,
+            0.007220669183880091,
+            -0.00305113079957664,
+            -0.0009556467994116247,
+            -0.010033016093075275,
+            0.021764053031802177,
+            -0.001455154619179666,
+            0.004359341226518154,
+            0.02081707864999771,
+            -7.774515688652173e-05,
+            0.006220633164048195,
+            0.01057181041687727,
+            0.002181711606681347,
+            0.003396041225641966,
+            0.010775899514555931,
+            -0.039315707981586456,
+            -0.0006571666453965008,
+            -0.025600925087928772,
+            -0.0018286375561729074,
+            0.0007357408758252859,
+            -0.01266984548419714,
+            0.013600491918623447,
+            -0.002015379024669528,
+            -0.010751409456133842,
+            -0.00029337796149775386,
+            0.021649762988090515,
+            0.005416522268205881,
+            0.004071575589478016,
+            -0.004469549283385277,
+            0.018792515620589256,
+            -0.006955353543162346,
+            -0.017861871048808098,
+            -0.012759645469486713,
+            -0.012367794290184975,
+            -0.01918436773121357,
+            -0.01214737817645073,
+            -0.0024551907554268837,
+            -0.002667443361133337,
+            0.004738946910947561,
+            0.002481722505763173,
+            -0.003228688146919012,
+            -0.02070278860628605,
+            0.015037278644740582,
+            -0.01430255826562643,
+            0.012873934581875801,
+            -0.011053460650146008,
+            0.005963481031358242,
+            -0.004122598096728325,
+            0.012726990506052971,
+            -0.021404854953289032,
+            0.02070278860628605,
+            -0.01536382082849741,
+            0.0038532004691660404,
+            0.019592545926570892,
+            0.006551257334649563,
+            -0.008118661120533943,
+            -0.013404566794633865,
+            -0.022074267268180847,
+            0.021633435040712357,
+            -0.011633073911070824,
+            -0.018237393349409103,
+            -0.004608329851180315,
+            0.009649328887462616,
+            -0.017470018938183784,
+            -0.007367613259702921,
+            0.01707816869020462,
+            -0.0018949664663523436,
+            0.012980061583220959,
+            0.023004913702607155,
+            -0.0052981507033109665,
+            0.0036572751123458147,
+            -0.009167678654193878,
+            0.0010286086471751332,
+            -0.02470293454825878,
+            0.013527019880712032,
+            0.008971752598881721,
+            0.006269614677876234,
+            -0.013935198076069355,
+            -0.013306603766977787,
+            0.0036531933583319187,
+            0.004498121794313192,
+            0.011437147855758667,
+            0.009420748800039291,
+            0.004220560658723116,
+            -0.013918871060013771,
+            -0.027788760140538216,
+            -0.027070365846157074,
+            0.03595232218503952,
+            -0.0005002731923013926,
+            -0.020947696641087532,
+            0.03399306535720825,
+            0.016114868223667145,
+            0.0022980421781539917,
+            -0.012449429370462894,
+            -0.004314441699534655,
+            0.024621298536658287,
+            0.016400592401623726,
+            0.010465684346854687,
+            -0.0004742518358398229,
+            0.005571629852056503,
+            0.0016469983384013176,
+            0.03012353740632534,
+            0.015323002822697163,
+            0.0009459525463171303,
+            -0.038597315549850464,
+            0.011943289078772068,
+            -0.007547211833298206,
+            0.0013408648082986474,
+            0.006702283397316933,
+            0.017780235037207603,
+            0.011935125105082989,
+            0.015567909926176071,
+            0.014865843579173088,
+            -0.003065417055040598,
+            0.0029511270113289356,
+            0.0006163488142192364,
+            -0.009624837897717953,
+            -0.001767410896718502,
+            -0.008881953544914722,
+            -0.005714492406696081,
+            -0.002920513739809394,
+            0.005592038854956627,
+            0.007657419890165329,
+            -0.003983817528933287,
+            0.028049994260072708,
+            -0.0006433906382881105,
+            -0.017600636929273605,
+            -0.009061552584171295,
+            -0.03611559048295021,
+            0.00434301421046257,
+            -0.030939893797039986,
+            -0.011257549747824669,
+            0.0033674687147140503,
+            -0.020457882434129715,
+            -0.024768242612481117,
+            0.0010306495241820812,
+            0.014335212297737598,
+            0.007392104249447584,
+            0.0033327736891806126,
+            -0.01004934310913086,
+            0.003816464450210333,
+            -0.001855169073678553,
+            -0.006461458280682564,
+            0.028833696618676186,
+            0.019559891894459724,
+            0.020457882434129715,
+            -0.01190247107297182,
+            0.019298657774925232,
+            0.005845109466463327,
+            0.02035992033779621,
+            0.010465684346854687,
+            -0.0048695639707148075,
+            0.0074859848245978355,
+            0.009567692875862122,
+            -0.006155324634164572,
+            -0.021519144997000694,
+            0.013192313723266125,
+            -0.0010837126756086946,
+            -0.0052083516493439674,
+            -0.016269976273179054,
+            -0.003534821793437004,
+            -0.015812816098332405,
+            0.010547320358455181,
+            -0.017159804701805115,
+            0.012498411349952221,
+            -0.000802069844212383,
+            0.00168475485406816,
+            -0.015469946898519993,
+            -0.013951525092124939,
+            -0.0012439225101843476,
+            -0.01244126632809639,
+            -0.008971752598881721,
+            0.006163488142192364,
+            -0.007539048325270414,
+            -0.008318668231368065,
+            -0.00535937724635005,
+            -0.011437147855758667,
+            -0.010882026515901089,
+            0.004367504734545946,
+            -0.0010653446661308408,
+            0.020931368693709373,
+            0.00011671340325847268,
+            -0.005824700463563204,
+            0.01190247107297182,
+            0.02787039615213871,
+            0.0016704685986042023,
+            0.004583839327096939,
+            -0.014261740259826183,
+            -0.011118769645690918,
+            -0.02279266156256199,
+            0.016572028398513794,
+            0.0049675265327095985,
+            -0.0344502255320549,
+            0.01438419334590435,
+            0.00824927818030119,
+            0.0015102586476132274,
+            -0.006698201410472393,
+            -0.005334886722266674,
+            -0.005608365871012211,
+            0.006306350696831942,
+            -0.0022837561555206776,
+            0.0005107327597215772,
+            0.0035919665824621916,
+            0.034678805619478226,
+            -0.0040674940682947636,
+            -0.007228832691907883,
+            0.00981259997934103,
+            0.0005130287609063089,
+            0.020196648314595222,
+            -0.005518566817045212,
+            0.0037572786677628756,
+            -0.019282329827547073,
+            -0.016563864424824715,
+            0.0018429238116368651,
+            -0.01722511276602745,
+            0.023282473906874657,
+            -0.005392031744122505,
+            -0.01347803883254528,
+            -0.006906372029334307,
+            -0.020115012302994728,
+            0.012637191452085972,
+            0.02310287579894066,
+            0.007016580086201429,
+            0.01033506728708744,
+            0.012131051160395145,
+            -0.02444170042872429,
+            -0.020457882434129715,
+            0.012171869166195393,
+            0.0013571919407695532,
+            -0.008449285291135311,
+            -0.005404277238994837,
+            -0.001783737912774086,
+            0.007783954963088036,
+            -0.0032633834052830935,
+            0.007881917990744114,
+            0.06491663306951523,
+            -0.01093100756406784,
+            -0.002987863030284643,
+            -0.02966637909412384,
+            -0.004396077245473862,
+            -0.006628811359405518,
+            0.0056287748739123344,
+            0.009151351638138294,
+            -0.023135529831051826,
+            0.009232986718416214,
+            -0.030025575309991837,
+            -0.010628955438733101,
+            0.00824111420661211,
+            0.0028490826953202486,
+            -0.01220452319830656,
+            -0.0009592183632776141,
+            -0.013665799982845783,
+            0.0006107363733462989,
+            -0.009412584826350212,
+            -0.00576347392052412,
+            0.012718827463686466,
+            -0.02321716584265232,
+            0.006298186723142862,
+            0.007592111360281706,
+            -0.006514521315693855,
+            -0.013722945004701614,
+            -0.017861871048808098,
+            -0.011967780068516731,
+            -0.013314766809344292,
+            0.002996026771143079,
+            0.01190247107297182,
+            -0.02607441134750843,
+            0.012922916561365128,
+            -0.0037389106582850218,
+            0.00945340283215046,
+            0.004404240753501654,
+            0.013469874858856201,
+            0.018074123188853264,
+            0.009355440735816956,
+            0.0021429345943033695,
+            -0.008024780079722404,
+            0.012702500447630882,
+            -0.01993541419506073,
+            -0.01635161228477955,
+            0.013837235048413277,
+            0.012016761116683483,
+            0.003061335301026702,
+            0.009828926995396614,
+            0.0020959940738976,
+            0.011886144056916237,
+            0.010775899514555931,
+            -0.019739490002393723,
+            -0.0006648199632763863,
+            0.0007326795603148639,
+            0.03280118480324745,
+            -0.0005938990507274866,
+            0.008857463486492634,
+            0.004420568235218525,
+            0.013159659691154957,
+            -0.012784135527908802,
+            -0.01769859902560711,
+            0.0008087027235887945,
+            -0.01057181041687727,
+            0.021257910877466202,
+            -0.008192133158445358,
+            0.006216551177203655,
+            -0.018612917512655258,
+            -0.027086693793535233,
+            0.006392067763954401,
+            -0.016261812299489975,
+            -0.018221067264676094,
+            0.01317598670721054,
+            0.0077268099412322044,
+            0.0017572063952684402,
+            -0.005098143592476845,
+            -0.020457882434129715,
+            -0.009837090037763119,
+            0.028441844508051872,
+            0.011102442629635334,
+            -0.004120557103306055,
+            0.009584019891917706,
+            -0.009094206616282463,
+            -0.012490247376263142,
+            -0.0017541450215503573,
+            0.0036715613678097725,
+            0.008261523209512234,
+            0.005645101889967918,
+            -0.01386172603815794,
+            -0.00029873527819290757,
+            -0.012400448322296143,
+            -0.021225256845355034,
+            -0.008906444534659386,
+            0.016751626506447792,
+            0.007820690982043743,
+            -0.0038960592355579138,
+            0.015812816098332405,
+            -0.004404240753501654,
+            0.002189875114709139,
+            0.014294394291937351,
+            0.005334886722266674,
+            -0.004285869188606739,
+            -0.0095758568495512,
+            -0.004612411838024855,
+            0.004043003544211388,
+            -0.0025960123166441917,
+            0.006457376293838024,
+            0.017192458733916283,
+            0.0072900597006082535,
+            -0.013755599036812782,
+            0.013135168701410294,
+            0.0007791098323650658,
+            0.0049675265327095985,
+            -0.03699725493788719,
+            -0.003247056156396866,
+            0.015796490013599396,
+            -0.013020878657698631,
+            0.02501314878463745,
+            0.005400195252150297,
+            0.022743679583072662,
+            0.015233203768730164,
+            0.008669701404869556,
+            -0.001542912912555039,
+            0.00464098434895277,
+            0.02759283408522606,
+            -0.003151134354993701,
+            -0.008637047372758389,
+            -0.0020602785516530275,
+            0.009241150692105293,
+            0.0002063850115519017,
+            0.003000108525156975,
+            0.008281932212412357,
+            -0.03562577813863754,
+            0.011706545948982239,
+            -0.019576217979192734,
+            0.0024694770108908415,
+            0.003459308762103319,
+            -0.007183933164924383,
+            -0.014865843579173088,
+            -0.005857354495674372,
+            0.022841641679406166,
+            -0.003224606392905116,
+            0.021078312769532204,
+            0.002885818714275956,
+            0.016106704249978065,
+            -0.011575928889214993,
+            -0.009208496659994125,
+            -0.0004130251472815871,
+            -0.005351213738322258,
+            -0.007539048325270414,
+            0.010596301406621933,
+            0.0034409407526254654,
+            0.005534893833100796,
+            0.002265388146042824,
+            0.0036654386203736067,
+            -0.0036878883838653564,
+            0.02773977816104889,
+            0.003563394071534276,
+            0.007996207103133202,
+            -0.011567764915525913,
+            0.014082142151892185,
+            -0.017029186710715294,
+            0.0070778070949018,
+            0.005575711838901043,
+            -0.026156047359108925,
+            0.016849588602781296,
+            0.004775682929903269,
+            0.006628811359405518,
+            0.013812744058668613,
+            0.01618834026157856,
+            -0.010408539324998856,
+            -0.014816862531006336,
+            0.024049850180745125,
+            -0.0032041973900049925,
+            -0.015225040726363659,
+            -0.006775755435228348,
+            4.438935866346583e-05,
+            0.006559420842677355,
+            0.00373074715025723,
+            0.0013020879123359919,
+            -0.0006204306264407933,
+            -0.010743245482444763,
+            0.004645065870136023,
+            -0.01934763789176941,
+            -0.0007163524278439581,
+            -0.016449574381113052,
+            -0.015633217990398407,
+            0.018678225576877594,
+            -0.01830270327627659,
+            0.009633000940084457,
+            0.0012275953777134418,
+            0.028588788583874702,
+            -0.011706545948982239,
+            -0.012155541218817234,
+            0.0015102586476132274,
+            0.010849371552467346,
+            0.0067512644454836845,
+            -0.010033016093075275,
+            -0.007583947852253914,
+            -0.014180104248225689,
+            -0.010163633152842522,
+            0.015502601861953735,
+            -0.014498483389616013,
+            -0.014343376271426678,
+            0.008914608508348465
+          ],
+          "embedding": [
+            -0.006359657085039302,
+            -0.024594786241399234,
+            -0.02553538924848095,
+            0.005360188465088291,
+            0.012801870977589839,
+            0.021680505862506104,
+            -0.016668048966001273,
+            0.06860852474581301,
+            -0.007122272841157724,
+            0.03471775000705488,
+            -0.009004297848861675,
+            0.01059974144678391,
+            0.02565604599426148,
+            -0.022544127646041287,
+            0.010616674976540785,
+            0.013658194823359737,
+            0.016400329743363844,
+            0.02858546964917478,
+            0.0067612676434693686,
+            -0.04174737193606879,
+            0.03184155110488616,
+            -0.02223105127585514,
+            -0.05506239667974977,
+            -0.0015307551410555752,
+            -0.013169788751379287,
+            0.017681749722384007,
+            0.01682706317229353,
+            0.019652295823816928,
+            -0.05623980369271269,
+            0.030698582851533197,
+            0.02286327124296613,
+            0.016019032539689178,
+            0.005508113353872315,
+            8.58985091215149e-05,
+            -0.019206292853495906,
+            -0.018439290483495144,
+            0.03475023176950151,
+            0.01018612942141759,
+            -0.0252551158857426,
+            0.0334562130666827,
+            0.020462153537485785,
+            0.02260508249714211,
+            -0.03323524989392951,
+            -0.002094251009530707,
+            0.015177442347686773,
+            -0.00018698464350038317,
+            -0.0006788648344273403,
+            0.0007777616893581039,
+            0.013352398615059647,
+            -0.030751111528744813,
+            -0.0317733818445192,
+            -0.006729064081710058,
+            0.009413204899986514,
+            -0.005712894407943592,
+            -0.038961826880527066,
+            -0.002827692206530345,
+            0.017863300537973137,
+            0.07918455388563496,
+            0.048967158919816366,
+            0.018002566325350088,
+            0.040264962620950695,
+            -0.019715100359659787,
+            -0.02211350500879408,
+            0.004918148608475574,
+            0.01062585305866606,
+            -0.014478993521696027,
+            0.0034983131226364672,
+            0.000766228629514787,
+            0.009667268771502067,
+            -0.01558439890697323,
+            -0.04984718709870083,
+            0.029613955432821887,
+            -0.009686854114630624,
+            -0.015279319238124568,
+            0.007460629676325322,
+            -0.006235979350809034,
+            0.012514387473717085,
+            0.02800435263741497,
+            -0.008688925946672778,
+            -0.01236301543374469,
+            0.029755705020188972,
+            0.01158037219446542,
+            -0.005242392206024511,
+            0.009982679027728814,
+            0.052414361714483264,
+            -0.018788500381726227,
+            -0.020335831196027204,
+            -0.08583959943356621,
+            0.029309228216812346,
+            -0.0249285282010673,
+            -0.018929396555426307,
+            0.007710663193334574,
+            0.03096798035958051,
+            -0.017089977064591828,
+            -0.028015107605405817,
+            0.06740703271128606,
+            0.009398795412354399,
+            0.01118769022826835,
+            -0.03041417460855722,
+            0.014776002883150835,
+            0.004319194067673161,
+            0.022507636639041203,
+            0.0053562788646786845,
+            0.007548072258943124,
+            0.011123374308563128,
+            -0.006924362115585165,
+            -0.011118507147092024,
+            0.009563232643295304,
+            0.00657178473678069,
+            0.08313027447380966,
+            -0.004239370156003264,
+            -0.00930406163946907,
+            -0.039202672821898946,
+            0.009761029930556298,
+            -0.02827698465424993,
+            -0.037914533246900244,
+            -0.04496050911855815,
+            -0.017711307900236353,
+            0.01934435542287117,
+            0.004335173027873153,
+            0.03800509969714242,
+            0.031198127357716314,
+            0.031311207941680276,
+            -0.012184553894588813,
+            -0.058985863670856876,
+            0.041442017057707635,
+            -0.012456189292721711,
+            -0.0040966351720394135,
+            0.010849179519959421,
+            0.005243123049347636,
+            0.0011495799963862647,
+            0.003534183676934487,
+            -0.03139340512003631,
+            -0.023876261883905306,
+            0.008382957569752911,
+            -0.004778283217011976,
+            0.03703067589888386,
+            0.06888630688839406,
+            -0.03993532296478411,
+            -0.009472167678222372,
+            0.025246872942105057,
+            -0.01524567912700368,
+            0.01595917939878582,
+            -0.003736723164442467,
+            -0.030836797784967753,
+            0.0026246446163368533,
+            0.008894300082148884,
+            0.08342856279154155,
+            -0.0007476580521704578,
+            0.025149116238550588,
+            0.004972542633631468,
+            0.034495632546556576,
+            -0.009173329764686424,
+            0.014009381543772768,
+            0.004427261668151028,
+            -0.013426138607902904,
+            -0.022234568880321808,
+            0.01976282364689966,
+            -0.03592433789490043,
+            -0.010918278220831188,
+            0.013527180289641174,
+            -0.013023113052242677,
+            -0.008304770302495363,
+            0.003677063799167178,
+            0.0013981822663219421,
+            0.047001163242600175,
+            0.01828332714887986,
+            0.002483375662343689,
+            -0.06820498187523893,
+            0.05295312038995555,
+            0.025038753504342162,
+            0.025286870198436735,
+            -0.0029544808216316115,
+            0.029982752196635428,
+            0.0005600481523874767,
+            0.0128290258663539,
+            0.009023805227424919,
+            -0.0019219166367750596,
+            -0.04482843936478531,
+            0.04523479691409549,
+            -0.004678574027241186,
+            0.010486850804218904,
+            0.006736229865018014,
+            0.02134813656476443,
+            0.028083972673690036,
+            0.016857413440125445,
+            0.0189082447820146,
+            0.03843690613820022,
+            -0.011922027011229613,
+            0.016193433524453224,
+            0.03828207911467065,
+            -0.0013055128404331705,
+            0.043198163443700956,
+            0.024025562893260542,
+            -0.001257851946749882,
+            -0.011726713433348844,
+            -0.021907984099443396,
+            0.02752198420597673,
+            0.004431866336999851,
+            0.015798806714413912,
+            -0.012910782191382367,
+            -0.03180549180206364,
+            0.004550247372715741,
+            -0.022778272841118514,
+            0.031242218240017532,
+            -0.006106984949693483,
+            -0.006259993190497421,
+            0.02187449047033339,
+            0.009334146847221406,
+            -0.009381278960016145,
+            -0.020888853823931134,
+            -0.007502374908776156,
+            -0.010390018386728482,
+            -0.04316693435221676,
+            -0.006274513730720055,
+            0.019152443937064258,
+            -0.01668984824845791,
+            0.018573194036505578,
+            0.0035035262183249925,
+            -0.019779911381789077,
+            -0.0027495798165085993,
+            0.02425481236315913,
+            0.006857723545270935,
+            -0.014733535699087731,
+            -0.026326146471941727,
+            -0.019906660231056833,
+            -0.001033294559612807,
+            0.0159154063026895,
+            0.035649534442431415,
+            -0.026210592096240003,
+            0.014397127229517285,
+            0.005114311681443673,
+            -0.0033104236005207077,
+            0.00324988235167305,
+            0.03764004154553922,
+            -0.017822437760737605,
+            -0.008579701063708334,
+            -0.00032410984200965487,
+            0.018450351071676443,
+            0.017283379764846195,
+            0.006384411611332382,
+            -0.005115090309123598,
+            -0.030590047062583125,
+            -0.011776211359193503,
+            0.01183623914060493,
+            0.0016564847796208331,
+            -0.009530766634361575,
+            -0.004810598258154945,
+            0.03876242515526699,
+            0.027076785774401667,
+            0.005324763136165989,
+            -0.03416412480913939,
+            -0.007943016733586697,
+            0.002114307650456928,
+            -0.02961449347300628,
+            0.05142732770518828,
+            0.05566026433842563,
+            -0.0030729309664974704,
+            -0.00938186948393074,
+            -0.04547908981236149,
+            0.0651216359692231,
+            -0.01813722445944094,
+            -0.0482650853560993,
+            0.044045155892545554,
+            -0.014272653003441165,
+            0.013740652306746851,
+            0.04563732439184395,
+            0.02126513340521316,
+            -0.0054845632439384905,
+            -0.011635625101262898,
+            0.030992082069685376,
+            -0.010051935721068779,
+            -0.050967258971854425,
+            -0.009827942095033494,
+            0.013373878556654464,
+            -0.013017627321534534,
+            -0.009361001542607812,
+            -0.01925867665425919,
+            0.020329052915436022,
+            0.023020741670660188,
+            0.019387043947695172,
+            0.005628632318152202,
+            0.018451555161952665,
+            0.016226845400848455,
+            -0.01384033112836216,
+            0.004762589697426251,
+            -0.012718476561725138,
+            -0.024222765127494463,
+            -0.037487144314010595,
+            -0.019971524166659278,
+            -0.015034588610516958,
+            0.009483936496215338,
+            -0.014301924283553591,
+            -0.0024139665139412125,
+            0.013361045353529458,
+            -0.0005870798721107379,
+            -0.011192319340879789,
+            0.013927695434391159,
+            0.021337539669342293,
+            -0.027300952495815437,
+            -0.04765670695199488,
+            -0.00561706224251562,
+            0.0082776639906139,
+            -0.0030984973749633724,
+            -0.026269016057688548,
+            0.007040524859505573,
+            0.0017859008072463442,
+            0.020091542517503808,
+            -0.018716761130867725,
+            0.0035283051960672963,
+            0.008519383548064053,
+            -0.015247417050336866,
+            -0.016917626868166634,
+            -0.018580550690103854,
+            -0.024697180580294884,
+            0.034202028316737125,
+            0.0204395003463227,
+            0.030392895534267717,
+            -0.04527669147753553,
+            -0.018359428092075348,
+            0.05494107499558624,
+            0.015234754907198422,
+            -0.05066008106884831,
+            0.031139824660661085,
+            0.005016615710206006,
+            0.0019000453624929544,
+            -0.006094364589229031,
+            0.0637324238017742,
+            -0.02711381489292883,
+            -0.02187418906973867,
+            0.004601744864532875,
+            -0.04141121277496072,
+            -0.026689585608019588,
+            -0.005568100750837504,
+            0.01291019036002459,
+            -0.031853041426355644,
+            -0.02815800259372657,
+            -0.007964861641790374,
+            -0.010933515363060793,
+            0.013726639077745378,
+            -0.023956800669647334,
+            -0.015672058590455337,
+            -0.023642025374517057,
+            -0.01288435863544158,
+            -0.0042516881552192385,
+            -0.011162214160460695,
+            -0.02412733256710621,
+            0.010088528028211243,
+            0.03310175458095482,
+            0.023064360903748363,
+            0.001996159684818467,
+            0.009478229572933101,
+            0.014478772877286076,
+            0.00371783700962415,
+            0.007864103610320116,
+            -0.048511515124609,
+            -0.031398754206534776,
+            0.020227872561659397,
+            0.012159811587094092,
+            0.0172163448580163,
+            0.009578952796095119,
+            -0.04154545961112031,
+            -0.05095080393682759,
+            0.02968486144855133,
+            -0.03932395023720674,
+            -0.02063219739061335,
+            0.007512423249865194,
+            -0.011911709468952548,
+            -0.00810951734088105,
+            -0.0008079327452253181,
+            -0.01600701450676789,
+            0.023984909975645632,
+            -0.02418072541081482,
+            0.0033984265728439398,
+            -0.0104343782811971,
+            0.01588977364765743,
+            0.05109529246167992,
+            -0.027800269136366997,
+            -0.027889034481533438,
+            0.03706813210665186,
+            -0.02058120652429308,
+            0.030801396468451574,
+            0.03246127847621613,
+            0.008355517405358826,
+            -0.008605467586247932,
+            0.017693344563968573,
+            0.014499487024471321,
+            0.012431277080478217,
+            -0.040103870071440256,
+            0.017328848567062113,
+            0.02723900253727976,
+            -0.054909074068976875,
+            -0.008547596270650966,
+            -0.00777964838164648,
+            0.027134666572242665,
+            0.02765036594970255,
+            -0.002207173084941375,
+            -0.016838740117653725,
+            -0.028289233739722936,
+            -0.019433376533829132,
+            -0.011053380458946764,
+            0.03247641283550457,
+            0.0007514989009289282,
+            0.007919574219000797,
+            0.001316042436488591,
+            -0.0017934342041762042,
+            0.01924145734197619,
+            0.015260199553572459,
+            0.048259602293134723,
+            -0.03383050622747848,
+            0.0021712225589426266,
+            -0.03508567715696004,
+            -0.04578731401250471,
+            -0.008768228935292693,
+            0.010203318819548585,
+            0.024135236371743855,
+            -0.007467140460133838,
+            0.0028907962385387818,
+            -0.007582083854422456,
+            -0.005384477718912037,
+            -0.01867978482547249,
+            0.01641191108381631,
+            0.037168089880543244,
+            0.0031992495663980464,
+            -0.008533890994695162,
+            0.000778843942291118,
+            -0.004620258538699103,
+            0.010572690335163528,
+            0.033233014627100904,
+            -0.030497362482602697,
+            0.02974571152880676,
+            -0.004274442492343003,
+            0.025341347125910928,
+            -0.04307748954707389,
+            -0.018069609916024403,
+            -0.018491122727274337,
+            -0.02069307802429807,
+            0.05372967226264594,
+            -0.030768978301976772,
+            -0.03662045886993282,
+            -0.003205991669059688,
+            0.016815511161004534,
+            -0.0014152644009276566,
+            -0.0016015439520090104,
+            -0.03984716970293867,
+            -0.010828260334550217,
+            -0.05894985435251326,
+            -0.01638035185618134,
+            -0.021819442096404948,
+            -0.005315102328745982,
+            -0.06251579337157112,
+            0.0033713876515840233,
+            -0.07815974854883057,
+            0.001484860071649606,
+            0.04539427129131382,
+            -0.023530943197353476,
+            0.05142250021536681,
+            -0.024746600229063383,
+            -0.0005885740455257141,
+            -0.044698536284421976,
+            0.0026481001040031785,
+            0.02389178603305878,
+            0.051387657872255674,
+            -0.002949198236157656,
+            0.010674377196609455,
+            0.013322075519401089,
+            -0.026245090721950647,
+            -0.01036724558050635,
+            -0.005321884180696542,
+            0.005958798671856245,
+            0.013755178085972732,
+            -0.0074734329537091675,
+            0.038368016046563674,
+            0.010941421833174982,
+            0.03443115781919206,
+            0.034623617136788874,
+            -0.004361926531008214,
+            -0.03348681614127998,
+            -0.00632870804659212,
+            0.024433331840566692,
+            0.0321998807702602,
+            -0.02899983280320611,
+            0.005958471722097962,
+            -0.015602447981516634,
+            0.015633217539747047,
+            0.006321931116948646,
+            -0.011827541320878442,
+            0.02372204564999124,
+            0.015673426811623124,
+            0.006103300725621424,
+            -0.005165006670545588,
+            0.013589526056189892,
+            -0.025411864292353578,
+            0.03041791947375232,
+            -0.004456865298584878,
+            -0.04276058054583262,
+            -0.0024736078733106825,
+            -0.020059309867945227,
+            0.024883721794000956,
+            0.015578237413840115,
+            -0.00536523508503184,
+            -0.005290645798641171,
+            0.008242987948511412,
+            0.020469675635821673,
+            0.006668354647245447,
+            0.00017729517736814364,
+            0.02373604785121342,
+            0.029915687122900485,
+            -0.02432635861789771,
+            -0.029540533943782886,
+            0.008151297444985344,
+            -0.005730733861939129,
+            0.011318816579801282,
+            -0.02644855229221747,
+            0.00033932872948409137,
+            0.0035505684154028996,
+            -0.03570104343808711,
+            0.03408366016010711,
+            -0.009199700348630527,
+            0.005857598530788056,
+            -0.009538177736472163,
+            -0.0017768806148210118,
+            0.0027410153099351694,
+            0.03898108970435115,
+            -0.007367623392751247,
+            0.009827254226889936,
+            0.015465895035342481,
+            0.000981604396724216,
+            0.05143083558797524,
+            -0.027350877485564314,
+            -0.02183036962301852,
+            0.01885468378168636,
+            -0.007285239007489837,
+            -0.016846837665812908,
+            -0.020546383007536074,
+            -0.011863943814684079,
+            -0.016293192750336218,
+            0.01976516340568883,
+            -0.02387328798681798,
+            -0.011892336568452852,
+            0.007170651391639504,
+            0.0013725014050721622,
+            -0.009809669068563384,
+            0.0016617183704652932,
+            -0.006307590349798603,
+            0.02037693835615797,
+            0.031753656615094446,
+            -0.024892675105483313,
+            0.03577919637821846,
+            0.03178303188315581,
+            0.017260898395251155,
+            -0.02102289690701718,
+            -0.009661035772073294,
+            -0.00048542689442823136,
+            -0.014694631967051945,
+            -0.0326379186158631,
+            0.025443206465204822,
+            0.017194626456982255,
+            -0.011381421003142453,
+            -0.03680783761649673,
+            0.0017216115811536721,
+            0.022819859205829785,
+            -0.013600989058799166,
+            0.028141657788496834,
+            -0.01869047298692401,
+            -0.01366833755869951,
+            -0.00782959271979894,
+            0.031227914441058244,
+            -0.05252392878125891,
+            0.016957842026348195,
+            0.00020880696574616416,
+            0.001154370575234767,
+            0.010892067708194324,
+            -0.008368819756628046,
+            0.012224564683325483,
+            0.031700389750709,
+            0.018063405869147564,
+            0.0010794115307202266,
+            -0.002783668216641561,
+            0.01418234057248712,
+            0.017773269840982066,
+            -0.015653443355569053,
+            -0.000497740485657824,
+            0.019890882604100452,
+            0.022672054781919238,
+            -0.00521797370595221,
+            -0.016686942830774276,
+            -0.018443538351123584,
+            0.025270057639201107,
+            -0.010354431070738671,
+            0.03484047753830186,
+            0.04594785214094975,
+            0.0036441959528283004,
+            -0.05695476763071743,
+            -0.009412209619190693,
+            0.0010371110229966086,
+            -0.01099343927053564,
+            0.01606863921935641,
+            0.012470660511773616,
+            -0.03237187923799263,
+            0.006388672213167589,
+            0.012946570716567364,
+            0.022152554488829627,
+            -0.02540317621005867,
+            -0.023183252921885226,
+            -0.0030984019855787655,
+            -0.007890467768631811,
+            0.018648760402624687,
+            -0.008653405980661855,
+            0.0006228315253506687,
+            0.009672526897422602,
+            0.0005436494014569835,
+            0.011632326145022935,
+            0.005655942007108228,
+            0.021736827172951,
+            -0.001256608167230203,
+            0.013124413886287987,
+            -0.01986808920858981,
+            -0.0270450339809516,
+            0.006500423802582728,
+            0.04438242119211055,
+            -0.014813399694814117,
+            0.030621709833336064,
+            0.0092171224268983,
+            0.016433384739642534,
+            0.024875072913970443,
+            -0.00891470467241686,
+            -0.014124009014664287,
+            0.00845777541223868,
+            0.009019007848927453,
+            -0.04147358041382793,
+            0.012513030150663913,
+            -0.01710243399313324,
+            -0.033144939719206276,
+            0.01904674924133127,
+            -0.00534122554342279,
+            0.012367320751926314,
+            0.004270315800911569,
+            -0.002886460711688153,
+            -0.025696286516581048,
+            0.014137901708875033,
+            0.028135087538950946,
+            0.021797869775098042,
+            0.0296173052503884,
+            -0.012757078368361444,
+            0.0005460601780756187,
+            -0.008748930817524899,
+            0.005470975782187157,
+            0.04187083128854477,
+            0.0008687182830616019,
+            -0.0008032802520818105,
+            0.01507726552487313,
+            0.024945744517309208,
+            0.0376130540370386,
+            -0.015415560911023553,
+            -0.03407353556457147,
+            0.01564969972264396,
+            -0.010104443707865262,
+            0.018855792875436674,
+            0.005503259932317811,
+            0.0049634097823351596,
+            0.0038600156268000702,
+            0.019312419743287703,
+            -0.006635193789316492,
+            0.018798045806429395,
+            -0.02500983683871332,
+            -0.028785757874583755,
+            0.01829411751138161,
+            -0.005277972443258226,
+            0.03338623299668019,
+            0.016225885095995672,
+            -0.029522280454919586,
+            -0.01712097344863975,
+            0.01553706187924322,
+            -0.008465766956343768,
+            -0.024842700926275538,
+            0.01895228828512998,
+            0.007076862934523977,
+            0.008956105593078965,
+            0.0024373779389833218,
+            -0.01106503696703573,
+            0.01125466547367417,
+            -0.000818223040067914,
+            -0.015322790791698395,
+            0.014157131555941659,
+            -0.017853386731281715,
+            -0.014994640991761465,
+            0.01775991780643474,
+            -0.037498078532902435,
+            0.009570197712419421,
+            -0.007687297543935123,
+            0.009050494195378254,
+            -0.004222041469820445,
+            0.007460383234429904,
+            -0.0010804379186060264,
+            -0.014987857934522812,
+            -0.014903749199810126,
+            0.03228485430019625,
+            0.022096121825475306,
+            -0.000410816936759558,
+            0.006728733918931589,
+            -3.971636461500232e-05,
+            -0.026719848032785824,
+            -0.011253751790407687,
+            0.02245467425830765,
+            -0.005664635254398481,
+            0.005134496095445943,
+            -0.01109590855846966,
+            -0.0149616794609307,
+            0.0019726017255571673,
+            0.032668637270108894,
+            0.06066635618119076,
+            0.012001732730659747,
+            0.024422093585231484,
+            0.0189499396761026,
+            0.02119276033547528,
+            -0.008489818917377516,
+            0.006413901834269739,
+            -0.03097413287789443,
+            0.03509225573986143,
+            0.005392594635384506,
+            0.03108937318306421,
+            0.026664331248813712,
+            -0.04722525921137681,
+            -0.0058539171830594774,
+            0.017087531454837677,
+            0.022673712717885323,
+            0.0159020681868943,
+            0.0008382498574190522,
+            -0.008938174268517353,
+            0.035650940044285984,
+            -0.03315250299348084,
+            -0.023647111263155,
+            0.005059382024092534,
+            -0.01439537467317451,
+            -0.010488359956731476,
+            -0.006785295580877329,
+            0.02776449681317373,
+            -0.005584052419598926,
+            -0.004582019433008779,
+            0.009697064252638535,
+            -0.014400206558190682,
+            0.008165676294736836,
+            0.0007624778531368436,
+            0.008681263897172326,
+            0.039249866981334126,
+            0.0005918063050375113,
+            0.024275699182767528,
+            -0.004138313072843555,
+            0.011079529739650594,
+            -0.0158972266904151,
+            -0.016985058543605,
+            0.02276982363172991,
+            0.00351101120496393,
+            -0.03244126813432563,
+            -0.00768819222024939,
+            0.00928180979884261,
+            -0.0013793404539641824,
+            0.005266058566444064,
+            -0.00503433670609399,
+            0.003747448261046857,
+            0.03286230666764075,
+            0.0022383592721685153,
+            0.008606190020480513,
+            0.013593947677257625,
+            0.00030951915677929204,
+            -0.01490307680740127,
+            0.013254591138113545,
+            -2.1379308136707722e-05,
+            -0.012900073369514622,
+            0.03357519186606058,
+            0.006705902094864628,
+            0.04243347200485736,
+            -0.03302953674711988,
+            0.009031534935013593,
+            0.01822227379789576,
+            -0.008491770335368761,
+            -0.02725412856390648,
+            -0.03630788717786838,
+            0.008259145747740535,
+            -0.019003260334340084,
+            0.015781275447792516,
+            -0.010039277287105153,
+            0.029665124310477384,
+            -0.03563370794521629,
+            0.004714858110268169,
+            0.004053913355055313,
+            0.0009503673086552651,
+            0.02068360255395961,
+            0.018524093113914366,
+            -0.002785698934204024,
+            -0.031443835341186485,
+            -0.02066365433385339,
+            0.006869321907853112,
+            0.02431260052207595,
+            -0.010367048093621498,
+            -0.005612149485859478,
+            0.0006316058139530722,
+            -0.024369562712568606,
+            -0.016845124984022828,
+            0.030359419843284823,
+            -0.040974427176153246,
+            0.005252911449822132,
+            0.04272936053727608,
+            -0.04599092715016919,
+            -0.03802288716943514,
+            -0.002314988728575789,
+            -0.024124014894379466,
+            -0.006360771882090754,
+            -0.002068030981218464,
+            0.016669991113175926,
+            0.0036232707799735537,
+            -0.028763764013482795,
+            0.00943401978066451,
+            -0.005173988601101547,
+            0.003971420424839974,
+            -0.0031669592494977393,
+            -0.02216336363004636,
+            -0.01657772032717298,
+            -0.02531962163186398,
+            -0.035174102848865116,
+            -0.0015903434876311088,
+            -0.005158703150821607,
+            -0.028703075279214722,
+            -0.03941940849540492,
+            -0.021696000925262537,
+            -0.0160221488026189,
+            -0.00843213135542359,
+            0.040641882367348774,
+            0.021319060666253945,
+            0.019336844555345886,
+            0.0025252167889382917,
+            -0.020346900609125973,
+            0.049440028814233025,
+            -0.007642182460707816,
+            0.035628691886108926,
+            -0.0017882888000877158,
+            -0.013461669667950893,
+            0.00455558389906464,
+            0.03825428026120858,
+            -0.014248434654950971,
+            -0.0009902062564027718,
+            0.00027811655483232545,
+            -0.0011866253757890136,
+            0.0008535155950871286,
+            0.015433650396777696,
+            0.001307651096343185,
+            -0.007112199543314309,
+            0.030380673552683906,
+            -0.012803247194895328,
+            0.01637501752192592,
+            0.018350814869962304,
+            -0.0013359719137099667,
+            0.008064546833360553,
+            -0.021430659268086814,
+            -0.00731572118780415,
+            0.0044578576331236325,
+            0.008422210247967642,
+            -0.015096885054883586,
+            -0.009964588397149773,
+            -0.02359494467299634,
+            -0.01174145797396426,
+            0.0023855749166744366,
+            -0.004559202896782688,
+            0.004902138778592298,
+            0.034595817216889944,
+            -0.016068744500427584,
+            -0.035106033294403614,
+            0.009997207702122002,
+            -0.02663088875167486,
+            0.02222306406130687,
+            3.983384942889326e-05,
+            0.006838221769859366,
+            -0.0023434342406307616,
+            0.001765445386886839,
+            -0.00169484278667461,
+            0.002862331185439823,
+            -0.018217975204395134,
+            0.0031859734246840257,
+            -0.0032119988065673565,
+            0.0009052950782312657,
+            -0.0012459653052875638,
+            0.00687339892509406,
+            0.006332799551164718,
+            -0.0008019538898592668,
+            -0.002968451740267742,
+            -8.541086355097327e-06,
+            -0.020561816931299912,
+            -0.021642512405352372,
+            -0.028259049261410905,
+            -0.000989887278779396,
+            -0.012359733017834338,
+            -0.027804394906433664,
+            0.011443385424045703,
+            -0.006872684999639775,
+            0.008327844771213648,
+            -0.008969527138034904,
+            0.006219994886600349,
+            -0.008415106130309062,
+            -0.0004143996206881767,
+            0.013119286511794273,
+            -0.02925209282917943,
+            -0.02096907906398536,
+            -0.03096513357712367,
+            0.01762223841790055,
+            -0.00925741670612824,
+            -0.002656382260603827,
+            0.0025223527960181835,
+            0.005092659166535706,
+            -0.033606542487570154,
+            -0.006653438998497412,
+            0.007313682256676618,
+            0.009547290751154595,
+            -0.03287788897629887,
+            0.02446728061355753,
+            0.017023222076526946,
+            0.01874233066565012,
+            -0.02417566312664911,
+            0.04049827067897005,
+            0.014426879325647042,
+            -0.021740606525480655,
+            0.016712907564723732,
+            0.027385150577780248,
+            0.03177163221499527,
+            -0.03469098547555545,
+            0.004481376248622418,
+            -0.01518200063132508,
+            0.018918680020516044,
+            -0.025284921244460287,
+            0.020625170445627613,
+            -0.016707629602243063,
+            0.021278719363178687,
+            0.03520334001264928,
+            -0.005524369131116028,
+            0.03608133014180745,
+            -0.019234548780647943,
+            0.0059985801736444555,
+            0.004041778461022664,
+            -0.01645503683802124,
+            0.01024694092614158,
+            -0.0042700351372843386,
+            0.0006059159967286885,
+            0.009406090340935747,
+            0.019133229156115282,
+            0.019007219188041775,
+            -0.04442208547053466,
+            0.028588965123434923,
+            0.00804120101594811,
+            0.021472086070876265,
+            0.014177820346835521,
+            -0.02708807970982664,
+            0.004742981711172693,
+            0.002072673190279628,
+            -0.004659713164250794,
+            0.0061167904730676514,
+            -0.013796604518930328,
+            -0.034540857808309226,
+            0.006338792319814448,
+            -0.018918703778187926,
+            0.015036937364271984,
+            -0.009026670935442384,
+            0.0011667525636770179,
+            -0.01152652811400482,
+            -0.011938679695598347,
+            0.0006170592505446954,
+            -0.005495151258143897,
+            -0.0008075736908949407,
+            -0.03171721592827769,
+            -0.04427658539412448,
+            0.02036132271908089,
+            -0.0021221965958423804,
+            -0.04320396743189005,
+            0.004111179168317842,
+            -0.025352428814134187,
+            -0.01865918186335911,
+            -0.011237699157960728,
+            -0.025047923887906474,
+            -0.04219630696740606,
+            -0.011108496740166623,
+            0.007785006066683012,
+            0.016810642610918525,
+            -0.009081815651299525,
+            0.040965574143584864,
+            0.0073944364393447965,
+            0.015313576815177263,
+            -0.017731360025694955,
+            0.026137777790243826,
+            -0.019314392060123,
+            -0.03108486330987073,
+            0.02676829566273746,
+            -6.259161089416372e-05,
+            0.003053748984540415,
+            0.014358356168574803,
+            2.8186924071913628e-05,
+            -0.006181789640967465,
+            0.01947730374003864,
+            -0.028739300217145797,
+            -0.023460503178221397,
+            0.0012460072421725688,
+            -0.008850395570726385,
+            -0.016271242375881602,
+            -0.035896509136022194,
+            0.005227881053956234,
+            0.03324337105109496,
+            0.004970419987430122,
+            0.008273497228454482,
+            0.01814500594735864,
+            -0.02229593571615308,
+            -0.024230675455380678,
+            0.000639112529814716,
+            -0.010266640429960994,
+            -0.010257247303470066,
+            -0.021302483296731352,
+            0.019150757066356872,
+            0.01651375496287087,
+            -0.00390189952706486,
+            -0.006971692709239087,
+            -0.010621003777733253,
+            -0.01918206725910585,
+            -0.02020345113211036,
+            -0.013827322378214754,
+            -0.017980206764311026,
+            -0.034153741666571154,
+            -0.032195001867229935,
+            0.03225205339201753,
+            0.026354817588919596,
+            0.0014076409241492998,
+            -0.007959081015805435,
+            -0.023070419006792155,
+            -0.004483328183581872,
+            0.02879516595345542,
+            0.002549934782150634,
+            -0.0001541849716285838,
+            -0.00830090759167373,
+            0.015341095409025125,
+            0.00835695019244165,
+            0.005107844984733648,
+            -0.06676974053142512,
+            -0.006027540973808174,
+            0.01655283585235268,
+            0.01351155740684989,
+            -0.002359114145814062,
+            0.02591301169926259,
+            0.01108099264396926,
+            -0.015361038608114896,
+            -0.019765743462262668,
+            0.035195761515422364,
+            0.02408910733949848,
+            0.023468767772473746,
+            -0.0034492999385544727,
+            -0.004472311490618405,
+            0.007030833258457338,
+            -0.010684252427867153,
+            -0.013326964647936429,
+            -0.004558513842504658,
+            0.026925146859337397,
+            0.016099443021514896,
+            -0.0064331198280825575,
+            0.010848931851939124,
+            -0.0008320209125061849,
+            -0.019444491995041537,
+            -0.001841330829156697,
+            0.008730275423516445,
+            0.0012974082030866743,
+            -0.014611902493400719,
+            -0.005911807768916134,
+            -0.019130604101278263,
+            0.024719468317569562,
+            0.0209456329723249,
+            0.03381727436368338,
+            -0.013237902591972769,
+            -0.0003695142331921278,
+            0.005410360755058108,
+            0.0002841632253252816,
+            0.0023352645111704958,
+            0.004801123870325242,
+            0.02623649813035263,
+            -0.029010710171522985,
+            -0.020741774066460415,
+            0.0068426980682519505,
+            0.013445270043414486,
+            0.0022744032984779294,
+            0.033840485942696216,
+            0.035040080489222324,
+            0.004166066423508316,
+            -0.010046666684988869,
+            -0.006148765564658713,
+            0.007071740742464309,
+            0.01865897335451031,
+            -0.023267411753900076,
+            -0.0005936657905496225,
+            0.004833030519903764,
+            -0.004229582648167378,
+            0.03620610659071198,
+            -0.009961238880412731,
+            -0.007197670122096852,
+            0.022212791947347127,
+            0.09819205436926473,
+            0.00047695621444943017,
+            0.0025354521225913395,
+            -0.029358210234742176,
+            -0.023614016211567963,
+            -0.0012564198366457442,
+            0.00242680708457405,
+            0.00830167274308547,
+            0.028076260517106013,
+            0.0004852400049405906,
+            0.011579274037419117,
+            0.02773972145320728,
+            -0.017217305534161797,
+            0.006752070301141707,
+            -0.011376051474975524,
+            0.024577071515205502,
+            -0.012694034026935636,
+            -0.0019809719792138673,
+            -0.016512649783050146,
+            0.015907402258634816,
+            0.004556574687833768,
+            0.004907510649166062,
+            0.011347630984408937,
+            0.009824484201337644,
+            -0.02650731900265866,
+            0.01756773113723491,
+            -0.005671986127099093,
+            0.004449611508541289,
+            -0.027841346572384462,
+            -0.020124233970502267,
+            0.0025986391930124847,
+            -0.0014440758308989809,
+            -0.009579444248630085,
+            0.006005274872617889,
+            -0.0024984235450999127,
+            0.03765664613203687,
+            -0.03765496603611941,
+            0.01409350051861866,
+            0.0066453175608372555,
+            -0.0015390841089599882,
+            0.011806653942640485,
+            0.017306914776093522,
+            -0.008206256939290151,
+            0.02589800448884443,
+            -0.00798213351353373,
+            0.009846274212000523,
+            0.015230070803055296,
+            -0.03449531102975745,
+            0.009174739184806312,
+            0.010145762060174144,
+            -0.024165660737765746,
+            0.012146877603301336,
+            -0.013784371016298891,
+            -0.004818759788054113,
+            0.005930446700613062,
+            0.007349837554764034,
+            0.010733731974181452,
+            -0.02295571464086674,
+            0.016176424826629898,
+            -0.010947841926210105,
+            0.0037766665765835445,
+            0.013559846998199008,
+            0.024892849808979323,
+            0.0012426907454129122,
+            0.0005950515136086193,
+            -0.01379469431504576,
+            0.005534724429523084,
+            -0.015667467221535825,
+            -0.022401586867694584,
+            -0.014192041775234079,
+            -0.006366846766730035,
+            0.01029598687444803,
+            -0.003972386708834858,
+            0.016971476088808737,
+            -0.010309648640697812,
+            -0.010645860303528099,
+            0.0037818310327362103,
+            0.01040806357838676,
+            0.008316605962234849,
+            -0.02855914079113881,
+            -0.005676310556150579,
+            0.004961084138163161,
+            0.00029285971503001265,
+            0.009592497954559435,
+            -0.014556073011576487,
+            0.013689526487160009,
+            -0.013290113354287095,
+            -0.0010726537498214612,
+            0.013201860397060236,
+            -0.012443552043930505,
+            -0.004012305744087929,
+            -0.01592905426965635,
+            -0.028487066798154617,
+            0.008462143660968396,
+            -0.01365614159523485,
+            0.001076882584445294,
+            0.0019503240125505646,
+            0.004730085646042709,
+            -0.016919069570642004,
+            -0.007860046623166481,
+            -0.033515448866521046,
+            0.011065416478687504,
+            0.02314589687882036,
+            -0.006435733825418645,
+            -0.030351727858789472,
+            -0.009546333526852862,
+            0.011327489786114963,
+            -0.0071765122743172715,
+            -0.01676736383051139,
+            -0.0029390038145944096,
+            0.007617910650439855,
+            -0.008174620754097493,
+            -0.00508943241317381,
+            -0.0025173969010414098,
+            0.0217328037301181,
+            0.016094975605203832,
+            0.01769642142712967,
+            -0.021400862603034036,
+            -0.00976992221747187,
+            0.026043879119776864,
+            -0.03275550610951898,
+            0.012441465975914529,
+            0.024587446266013183,
+            0.003446527051428522,
+            0.005492228054838714,
+            0.0030377554109693714,
+            0.005459159163772825,
+            -0.006985179226831048,
+            0.019925583798328282,
+            0.010870421035240845,
+            -0.019709062313471467,
+            0.005385443078244229,
+            0.02086085539375208,
+            0.0037248737640193097,
+            0.012892299156840998,
+            -0.017546870090541213,
+            -0.0020341807891543716,
+            0.015594779287017958,
+            0.04816214672670215,
+            -0.0032851797178091685,
+            -0.010450846098750195,
+            0.029513742590677183,
+            0.023526004023637855,
+            0.005373455940309873,
+            0.0012120777808635214,
+            -0.02525630456877349,
+            -0.013075130701774785,
+            0.01962235086017781,
+            0.016944870494393874,
+            0.013484925533584295,
+            -0.0023255656480479004,
+            -0.019236081983457366,
+            0.021520156296467717,
+            -0.01701733319731161,
+            0.012422659010697477,
+            -0.023764941037007197,
+            -0.001490790833263327,
+            -0.0035419008954878724,
+            0.021998949056452454,
+            0.0021819635638621092,
+            0.008891505304514552,
+            0.024354552127458608,
+            -0.006978548952551799,
+            -0.019526615985982355,
+            -0.0051799020400150305,
+            -0.014578953088901798,
+            0.022506635912517146,
+            -0.024117708614271965,
+            -0.007008641019892289,
+            0.011379795968360019,
+            -0.012362093847309038,
+            -0.0102534051092702,
+            0.001164934138876502,
+            0.017686807394319725,
+            0.0009423993435129515,
+            -0.014784508280754078,
+            0.01944445593807992,
+            -0.0011359271135469456,
+            0.010410915597949927,
+            -0.01921419010977189,
+            0.009390638453648025,
+            -0.016682442516992777,
+            -0.007849184048209236,
+            0.010343008873053859,
+            0.0012604521199837046,
+            0.003948300703645506,
+            0.002592490121754231,
+            0.007773437307969315,
+            -0.03355658426684504,
+            0.022695505487314516,
+            -0.010751606023585792,
+            -0.003538523688786008,
+            -0.0014125330167969218,
+            -0.01678692701672328,
+            -0.0006918162525881059,
+            0.005816502852598842,
+            0.010878358093847386,
+            -0.016045147059003256,
+            -0.00873270513444376,
+            -0.04920244395446931,
+            -0.030314009649638693,
+            0.023156333024481057,
+            0.013733657599720879,
+            -0.010971506342695331,
+            0.018852159415605117,
+            -0.023760853159532587,
+            -0.010587249404421732,
+            0.0011379869499042905,
+            0.023319856329056813,
+            -0.03539290744576506,
+            0.01215369703816749,
+            -0.014493682360886549,
+            -0.02039288669534806,
+            0.009070984596830142,
+            0.011445354735283254,
+            0.0007259852045513916,
+            -0.0011514627555568814,
+            -0.029350945123925453,
+            -0.010246845821727087,
+            0.02391068177425878,
+            -0.00861718589055971,
+            -0.004723044986826333,
+            0.024716520302409263,
+            0.0033721801437248873,
+            -0.012524663490691517,
+            -0.017955758068233754,
+            0.021612671655463236,
+            -0.026077203994996058,
+            0.007645944309444457,
+            0.010032183011253845,
+            -0.0173991263874487,
+            0.021649599230588117,
+            -0.007529701797485131,
+            0.020327763440441377,
+            0.004227390938169872,
+            -0.011249233530251522,
+            0.020352283974471443,
+            0.013644443758970054,
+            0.004216958800094986,
+            -0.004081669404784771,
+            0.006869353098014179,
+            0.02212732880780841,
+            -0.011021739614266063,
+            0.02219580717345023,
+            0.028777213295471067,
+            0.00017328191188631918,
+            0.03260179554743322,
+            -0.021287655520443285,
+            -0.03159728983004563,
+            0.021740163175540124,
+            -0.01035146858119169,
+            -0.021564906188712388,
+            0.028914245422221707,
+            -0.013674731836508768,
+            -0.00786476052825481,
+            -0.0166718429993922,
+            0.00533027311342322,
+            0.016680336311088843,
+            -0.005246190451159961,
+            -0.014360934066628332,
+            0.0029628103834216345,
+            0.019809487418628386,
+            0.014587234544007958,
+            0.029780009648869975,
+            -0.010778199815920887,
+            0.035180242185168895,
+            -0.002320576872571577,
+            0.008990157317473909,
+            0.012415465232073404,
+            0.006489435152972723,
+            -0.011251531176923445,
+            0.013501894212117511,
+            -0.003446759011266526,
+            0.008230548711588447,
+            -0.001449023474280409,
+            -0.035111679577202855,
+            -0.013709382199848678,
+            -0.04709294504969336,
+            -0.0023752976043235697,
+            -0.0025098360125255215,
+            0.0030975417740913154,
+            0.09736815869389318,
+            -0.005723049647039619,
+            -0.011988817571623457,
+            0.001625059666070763,
+            0.009611697638805556,
+            -0.0009896494420056814,
+            -0.007606836952841873,
+            0.015973152482488318,
+            -0.01343292635775299,
+            -0.02140928016038583,
+            -0.010532701190806147,
+            -0.01288594454368977,
+            0.012079198190522715,
+            0.015929716343161652,
+            -0.0005947359502228821,
+            -0.015207331663761012,
+            0.006524010655124294,
+            -0.004566385899752684,
+            0.0064717704997400245,
+            0.008647099975176308,
+            -0.010414668624343924,
+            -0.0071698879102925555,
+            0.030899143952119196,
+            -0.025301269549419674,
+            -0.024361748354052457,
+            -0.0010579931222861095,
+            -0.009055796737486008,
+            -0.010335459737012047,
+            -0.002692653633719985,
+            -0.0017679501322462056,
+            -0.010519994985692081,
+            -0.010019865263564572,
+            0.006537076627706128,
+            -0.015472130358273033,
+            0.004812857776601667,
+            -0.0047457011269495634,
+            -0.013296069496526007,
+            0.017054968086538724,
+            0.022563785357424853,
+            -0.02841491200807702,
+            0.0006755428946168761,
+            -0.017422510783648978,
+            0.012941211381006863,
+            0.008866069010589792,
+            0.006049618782357251,
+            -0.022451528209097505,
+            -0.023622906582403243,
+            -0.015019272754405791,
+            -0.01927535750568903,
+            -0.024973490129133732,
+            -0.009073381957684092,
+            0.0067692803507600945,
+            -0.001267339788153158,
+            0.021918281111033645,
+            0.000816538714279555,
+            -0.01812545613159305,
+            0.0061084913570894905,
+            0.000568371220952269,
+            0.018292497018668842,
+            -0.027468677288882186,
+            0.0033806422195878524,
+            -0.02307582511848465,
+            0.033508234091571124,
+            0.0039195050746594975,
+            -0.026680111514794824,
+            -0.021558668725755275,
+            -0.003311830287529699,
+            0.018994577319066473,
+            -0.0025745912019675805,
+            -0.00724049447539589,
+            0.014775207617625293,
+            0.0155334760323838,
+            -0.004693118176196483,
+            0.002015625406296403,
+            0.027968079002437102,
+            0.01077998866180723,
+            0.0313892866933904,
+            -0.015236169297590675,
+            0.004945828495617267,
+            -0.027215016713379062,
+            0.010214419950212732,
+            -0.01503548137805401,
+            0.008859554812646242,
+            -0.01290890682557532,
+            0.03531384210182567,
+            -0.012005613861257432,
+            -0.004250698950192301,
+            0.03253288796441852,
+            -0.029150986393345252,
+            -0.0035020051536906334,
+            -0.009255437891304041,
+            -0.004142204892160768,
+            0.013240580426083011,
+            -0.001198093874819309,
+            0.008485141356876356,
+            -0.01854370462655459,
+            9.726096864073609e-05,
+            -0.003221852304410318,
+            0.005404595442127598,
+            -0.02005458688172967,
+            -0.003314118809778734,
+            0.0073427568419969814,
+            0.004259518237708634,
+            0.002408630356035373,
+            0.016420646142725346,
+            0.006662121128241274,
+            -0.0029357701493526907,
+            0.03424224243607692,
+            0.008171895850772245,
+            0.0152521424877615,
+            -0.0018638264193202728,
+            0.0030234211261443034,
+            0.019287963516760636,
+            0.019912956661192856,
+            -0.00048443399729935116,
+            -0.01058935153883094,
+            0.00195113175754497,
+            -0.001140732137433033,
+            0.011048741602327437,
+            0.024396046054912645,
+            -0.021158025221644738,
+            -0.02419562092713724,
+            0.016377661005279716,
+            -0.015076145951740961,
+            0.01903712233378994,
+            0.010541463112889985,
+            -0.010099351302095792,
+            -0.00031558616108876247,
+            -0.042316016201379245,
+            0.010964989031040692,
+            -0.02068939391415103,
+            -0.04004120028915868,
+            -0.00244383573783919,
+            0.0009542507389088937,
+            -0.029439456243505052,
+            0.02822814855547978,
+            0.0023755032182188156,
+            -0.017280221740813364,
+            0.018223709137876153,
+            -0.005548654680726505,
+            -0.0024040301239229064,
+            0.005397711787238679,
+            -0.01092026090471121,
+            -0.010648485086769927,
+            0.008235250252660972,
+            0.010957715075632535,
+            0.005425683656535705,
+            -0.005698919020608095,
+            -0.011284911899894062,
+            0.009975882651638084,
+            0.012318901214538077,
+            -0.0019922884373083155,
+            0.010050520101815593,
+            0.0031720752428247446,
+            0.007352859750323051,
+            -0.0073483049386141005,
+            0.0046483587496626965,
+            -0.005093220128411125,
+            -0.026080057085901446,
+            0.012863580653903103,
+            0.0052434400313520774,
+            -0.011863360599637747,
+            0.0103209502958491,
+            -0.01048162756158923,
+            0.005787157766911021,
+            0.001318945460232246,
+            -0.0059333230422061395,
+            0.004821320238893602,
+            -0.0038922239931512784,
+            0.011267634571983924,
+            0.013256611037442042,
+            0.004592529900805719,
+            -0.013076954459595086,
+            -0.02177318635989856,
+            0.009001291224623467,
+            0.01959767163356408,
+            0.024161754192119852,
+            -0.030416109293488064,
+            0.003784634588325551,
+            -0.0014915849060829743,
+            -0.0020550564975080885,
+            -0.015632785947151753,
+            0.01587790981601866,
+            0.01716742056401084,
+            0.012721563202528083,
+            0.01893393006967701,
+            -0.004009617703771589,
+            0.010539100665843704,
+            -0.02080068888791658,
+            0.013642451766699067,
+            0.0008609869904066794,
+            -0.013629067231663643,
+            0.0014620655123815298,
+            0.027046304278766087,
+            -0.011056783825898616,
+            0.006206969699250696,
+            0.008097527175511791,
+            0.022111870736907183,
+            -0.012509479100313492,
+            -0.003735868296238069,
+            0.01516254968167132,
+            0.0014878674796313539,
+            0.0035887079417890352,
+            0.016533149527222273,
+            0.04729052276110659,
+            0.00102751186013391,
+            0.0016713773449738153,
+            0.0192177920520243,
+            -0.005563519063161466,
+            -0.01114821419109643,
+            -0.003549999863325982,
+            -0.021807893906469106,
+            0.010998558936481785,
+            0.00431279785115251,
+            -0.02186324212032519,
+            0.01024093366826206,
+            0.0125495473550584,
+            0.005268320135120931,
+            -0.0031679212410026603,
+            0.02574575247827441,
+            -0.019807879421996258,
+            0.014935295894965128,
+            0.03329450628384787,
+            0.00493356778223407,
+            -0.011423949333937285,
+            0.007263264249871384,
+            0.0027354459678742013,
+            -0.004494406041906684,
+            -0.00606745413660583,
+            0.010497391290734088,
+            -0.008191702034391382,
+            0.009870458191510176,
+            -0.009580130479693,
+            0.039202091799313755,
+            -0.009619018331262844,
+            -0.010782138248435458,
+            -0.004558645539551881,
+            -0.006241773108460662,
+            0.003478414182938609,
+            -0.003395165694642073,
+            0.005812225135633631,
+            -0.0069045407353789945,
+            -0.014607146220271209,
+            0.0008522982757210454,
+            0.00020175973815948303,
+            0.014001922919116725,
+            0.0011804904415274088,
+            -0.004871874532755353,
+            -0.007877217154706205,
+            0.015243277056007988,
+            -0.006047906208304912,
+            0.0006531937033317192,
+            -0.0011653523616313422,
+            -0.0013662556399359335,
+            -0.007401050012179761,
+            -0.015166105795085528,
+            0.00974609244984173,
+            0.016253733908242134,
+            -0.013857834499211056,
+            -0.005142936208363415,
+            0.012274716212832219,
+            0.008540229125193836,
+            0.011742606168187876,
+            0.0007451338650642946,
+            -0.009255813682572937,
+            0.008292531177374192,
+            -0.015020523035432158,
+            -0.008781317965985481,
+            0.0026250469418136603,
+            0.002381147382977664,
+            0.005839723981046825,
+            0.0006240641828668715,
+            0.004502463508605272,
+            0.005798912705218846,
+            -0.00497936644919367,
+            0.01112666823857836,
+            0.008198424171006548,
+            -0.002452143482105151,
+            -0.01017569410837814,
+            -0.001127589984272638,
+            -0.015733298167661346,
+            -0.016758315411028353,
+            -0.0009286266498936771,
+            0.015175988113750882,
+            -0.02352862898718829,
+            0.018996352975391644,
+            -0.0014524363611708201,
+            -0.0033802546171857704,
+            -0.021392428456030312,
+            -0.0025923983911570183,
+            0.0034214188668518796,
+            0.01255768230394476,
+            0.01858853630745928,
+            -0.02729543519465384,
+            0.010575894388563116,
+            -0.007204725245442141,
+            0.030394929424014717,
+            -0.018328782140005044,
+            -0.004022941273610227,
+            0.026915618079621992,
+            -0.02200018306828758,
+            0.0008791185377513978,
+            -0.011454606955158956,
+            -0.008103032693853504,
+            0.019287674837688177,
+            0.023747720480736777,
+            0.02038287242167783,
+            0.01364489294869211,
+            0.0007758044919789085,
+            0.004768030222445397,
+            0.03572705648994403,
+            0.010568241270862248,
+            0.014828087092859878,
+            -0.007542489523110974,
+            9.457808640264811e-05,
+            -0.02554033312739386,
+            0.01635899434369978,
+            2.587383534566272e-05,
+            -0.012370727506722627,
+            0.007162422186621379,
+            -0.013370882403931942,
+            -0.011066483600541058,
+            -0.007151077450730025,
+            -0.01721910018182534,
+            0.02152239849400241,
+            -0.012322440280680164,
+            0.004587729883596212,
+            -0.009079747696669277,
+            -0.013272707286450417,
+            -0.012953682633400515,
+            -0.014202214807187545,
+            -0.01313141988034687,
+            -0.026041403779538307,
+            -0.027109731260915763,
+            -0.006573285295895077,
+            -0.023780366346334524,
+            -0.013799072648505526,
+            0.005588622594136253,
+            -0.022617902666316757,
+            0.012262325666428604,
+            -0.012689827812482627,
+            -0.01226264872059864,
+            0.012805734807835008,
+            0.020274844315616616,
+            -0.011765231803328348,
+            -0.028494489703400308,
+            -0.018583843821043854,
+            -0.019970998470704358,
+            0.012576201137505969,
+            0.002266288538944273,
+            0.013165889075444961,
+            -0.003018437854660793,
+            0.003227412667880239,
+            -0.014895831590298957,
+            0.018570624675927183,
+            -0.03442390703780963,
+            -0.03108003806963536,
+            -0.013611306765348361,
+            0.005182777066476028,
+            -0.0015782033203578553,
+            -0.011429046504256945,
+            -0.0018606648572920801,
+            -0.00433805271182878,
+            -0.025109572024446088,
+            -0.010134856632190067,
+            0.011010662054484161,
+            -0.015205629927469414,
+            0.0072933565255675036,
+            0.0009018832564038863,
+            0.013870902834260887,
+            0.009587432944616442,
+            0.013393301893492882,
+            -0.006095063698384775,
+            0.004356485794113208,
+            -0.0026476612747324386,
+            0.0004894024149759108,
+            0.009336366919474478,
+            0.00839591561027103,
+            0.010551161369079399,
+            0.002260707777260338,
+            0.007830059223001696,
+            -0.0060927446216633265,
+            0.018065002687169148,
+            0.015436944875538481,
+            -0.0002520689442496089,
+            0.013323098485251487,
+            0.0003769862591044255,
+            0.00022293304921127018,
+            -0.0029970303106340466,
+            0.007888437018607987,
+            -0.011785957933642163,
+            -0.001311704406355858,
+            0.015138190893483804,
+            -0.01707247730569978,
+            -0.013045187198628258,
+            0.03107034781283638,
+            0.0018583638488268697,
+            -0.02495564203304796,
+            0.031146554085787762,
+            0.004220047610942943,
+            -0.005835199594223787,
+            -0.013205086624848782,
+            0.0014059047822222418,
+            -0.011570624844100298,
+            0.003584232809362274,
+            0.014295032061446826,
+            -0.02231984223610487,
+            0.010521094801839404,
+            0.008446892441796151,
+            -0.009681535919980559,
+            -0.012337659795586223,
+            0.009847073709209602,
+            0.016809328445200358,
+            0.004472485504034416,
+            -0.03539093810210523,
+            -0.011957694102753411,
+            0.02145470732008716,
+            -0.029237428225009913,
+            -0.00810041956259673,
+            0.007452962400105787,
+            0.0065154832040426065,
+            -0.019353109017609053,
+            -0.008580955020412362,
+            0.005725794605897509,
+            -0.004940390374637084,
+            -0.003892210172017758,
+            0.017541623128534682,
+            0.00485047043626186,
+            0.013698341792101025,
+            -0.01689738578307474,
+            0.009771458057400557,
+            -0.022222817828602557,
+            0.004021882074676753,
+            -0.023373700500461992,
+            -0.009873403167702603,
+            0.002364143609079849,
+            0.012454271200139762,
+            0.014328556467632212,
+            0.027839363143659326,
+            0.030056404238348527,
+            0.013995696971422429,
+            0.007833677585250953,
+            -0.016265199386953717,
+            -0.018134327237616175,
+            -0.006409513840075933,
+            -0.004033173350204784,
+            -0.008714798059218596,
+            0.0066373547199279235,
+            0.01858354212009877,
+            0.004499712802472377,
+            0.02609078206670711,
+            -0.006227189596272876,
+            0.009039744622014585,
+            -3.127526911951914e-05,
+            0.0008833930131827993,
+            -0.004226083827636182,
+            -0.01971685546584527,
+            0.00481003211084742,
+            -0.0013721482162825508,
+            0.025874758880591167,
+            0.00940496860916206,
+            0.0024915013399541847,
+            0.01615978314811047,
+            -0.006378952757397484,
+            -0.01174653607178726,
+            -0.010540107733238588,
+            -0.0061926268398085176,
+            -0.021752582127981104,
+            0.029059100503124387,
+            0.021195595568253612,
+            -0.010800765264123782,
+            -0.013693582166598082,
+            -0.00903369526059487,
+            0.02795767990556847,
+            0.004080965688836953,
+            -0.0013050212744006933,
+            0.016313311850308505,
+            0.012084555958999215,
+            0.012329325712491252,
+            0.009794698207689061,
+            -0.00036157148861933123,
+            0.0015364709122308352,
+            0.03722522141440073,
+            -0.007427028510548186,
+            0.0026463292682538495,
+            -0.005239324411651515,
+            0.0025987932868461313,
+            -0.009705817520422487,
+            0.007242122655628283,
+            -0.004012676385296197,
+            -0.0158760051470016,
+            0.015003444174308607,
+            -0.022443331305169994,
+            -0.0016749787204471238,
+            -0.008150635190952799,
+            0.0016696640640947303,
+            -0.02781023337840043,
+            -0.0025731206607371845,
+            0.006695065701630532,
+            -0.015547340828489069,
+            0.011816010501093885,
+            -0.01740673229051557,
+            -0.011553604323149105,
+            0.007914961618894248,
+            -0.012034157288065447,
+            0.0016684130271324446,
+            -0.0008268957621923278,
+            0.004460864332388434,
+            0.013426055079012884,
+            -0.011008808570221245,
+            0.012166046631826273,
+            -0.005816284310138776,
+            -0.007872091214834688,
+            -0.0016055129487028885,
+            -0.0030049472973942043,
+            -0.0025600746761171543,
+            -0.0009966231170649917,
+            0.034355910234323155,
+            -0.004446570896048543,
+            0.003761242147499512,
+            -0.01362639743248326,
+            0.0019833052855181806,
+            -0.036176376394999044,
+            0.028344464817309477,
+            -0.008642835889481802,
+            0.011013502221008498,
+            -0.0017508365480865596,
+            -0.013243706268954915,
+            -0.01456517309747086,
+            0.015247299501245178,
+            0.007602596572394442,
+            -0.005854531057349665,
+            0.013558248087347848,
+            -0.019724373614448298,
+            0.008530500822910953,
+            0.004463694543554244,
+            -0.015561397976987216,
+            0.004668780190328835,
+            -0.007479912658259757,
+            0.012657201279639194,
+            -0.01774730033362932,
+            0.004314594593150765,
+            0.00958571617187387,
+            -0.01892876084898881,
+            0.009901041800918215,
+            -0.028675879564512306,
+            -0.004631920242208613,
+            -0.012323455524366223,
+            0.0053804608827285865,
+            -0.000660864620708182,
+            -0.02389088252276507,
+            -0.028944280660230762,
+            -0.0002099667627277945,
+            0.00458533068247518,
+            0.0018878351997997833,
+            0.011621407953503755,
+            0.005970807015151215,
+            0.008542958629199003,
+            0.01023896054756221,
+            -0.00205207247588841,
+            0.009402752124876478,
+            0.04169706217414899,
+            -0.022899912321317197,
+            -0.011574970753549647,
+            0.005321872977369877,
+            0.011499271505460519,
+            -0.02647653483798284,
+            -0.011039791758347888,
+            0.0033424345263339144,
+            -0.005219877979550397,
+            0.03609469691834737,
+            -0.02448746405881972,
+            0.019020618652020673,
+            -0.011392297116547224,
+            0.007625475937968714,
+            -0.021465240076078376,
+            -0.003921088264697311,
+            -0.014265745991522477,
+            -0.00011221615798127438,
+            -0.0031478236827989703,
+            -0.0076053733863387755,
+            0.007485039021296698,
+            -0.001907678661303243,
+            0.0020348246022161925,
+            0.006291436821831794,
+            -0.02173305455485444,
+            -0.0019125499969493854,
+            -0.016432460924913166,
+            0.030367263349759286,
+            -0.016242095671439496,
+            0.005072805486919414,
+            0.0053258139728101506,
+            -0.002810639419737258,
+            0.012368771814660942,
+            0.004165967191536186,
+            0.00513754440091394,
+            -0.004199767241827414,
+            -0.005142642360546972,
+            -0.0053886448467479215,
+            0.017868874549760376,
+            -0.013280048598849825,
+            0.035024701466710065,
+            -0.0100959186045929,
+            -0.007094001877261568,
+            0.0019733042325232155,
+            0.010380430710282744,
+            4.2084202709296443e-05,
+            0.01053565670076933,
+            0.0034270489770133685,
+            -0.0006311078385359693,
+            0.0019774779328212756,
+            -0.017162448473652843,
+            0.003089758682621882,
+            0.01682656561693936,
+            0.005463677441664502,
+            0.004469261499451007,
+            -0.013332130891003062,
+            -0.012050233723282738,
+            0.004538268704442613,
+            -0.007533940408255447,
+            0.0003675737140813486,
+            -0.018816953644773076,
+            0.013972696748575243,
+            -0.0011096683686631858,
+            0.0002715683052236706,
+            0.00524678369365015,
+            0.023056514997908218,
+            0.01813167860994747,
+            -0.002553352602551003,
+            0.03155164978556312,
+            0.0174738879054593,
+            0.0029268039949260673,
+            0.010637295711777793,
+            -0.033222781195984666,
+            -0.013301873512972214,
+            -0.0013721703891642354,
+            0.02238421933149517,
+            -0.01530189703335162,
+            -0.014059029663525449,
+            -0.0037393534403749947,
+            0.0010394601933309767,
+            -0.005177359933828505,
+            0.0042489306011644495,
+            0.011039889511148341,
+            -0.014697595240068863,
+            0.007597149702441915,
+            0.0026780496822907634,
+            0.02092734730183164,
+            0.0004688144432243786,
+            0.017181410417231698,
+            -0.0005083970646221274,
+            0.00749639055342427,
+            0.007319420826337891,
+            0.017541463125355265,
+            -0.01002811389296805,
+            0.023582113845081534,
+            -0.008823933397729605,
+            0.0019889258902730535,
+            -0.01257106158422629,
+            -0.028159594067544737,
+            -0.012624750220149823,
+            0.015247564545279736,
+            0.005765911899842925,
+            0.006478299226263732,
+            -0.016537090297144265,
+            0.006276997440048981,
+            0.015647122171767974,
+            -0.002251683151059454,
+            0.010295358869874306,
+            0.009730378299068289,
+            0.00012816405890877098,
+            0.010181120688239766,
+            -0.00801256937783534,
+            -0.00734257767723293,
+            -0.004497520124843868,
+            0.023982754195729983,
+            -0.005443550092842393,
+            0.022434177122972028,
+            0.012647958478747348,
+            0.00616791857573897,
+            -0.010853355357954727,
+            -0.003340111254357573,
+            0.023389634440057473,
+            0.011863246790550376,
+            0.00029642699874398923,
+            -0.009846034060495678,
+            0.0019458876579524617,
+            0.0054010373765486005,
+            0.012983913038716568,
+            -0.02049427270240938,
+            -0.004344571352918878,
+            0.008461947538684664,
+            -0.008472919299762712,
+            0.004318309005722874,
+            -0.030156444522730416,
+            -0.005812303237222271,
+            0.021354352236942434,
+            0.008392281015122372,
+            -0.010232039722494646,
+            -0.004395682831498155,
+            -0.0006527587802367882,
+            0.0040601850269714945,
+            -0.0024323167053212176,
+            -0.0023729138545912176,
+            -0.008541233856441836,
+            -0.006470933419694742,
+            0.014035696617797647,
+            -0.0052691760528871924,
+            0.008597459268477037,
+            0.00012084859681905111,
+            -0.0017255958688825231,
+            -0.007323529232751411,
+            0.03382792341957058,
+            0.008087732655845718,
+            0.005980666974986827,
+            0.0013613625176327891,
+            0.0042244330945601215,
+            0.000616791811672998,
+            -0.0028374856646453707,
+            0.018075360591135542,
+            0.0156738939725117,
+            0.005054696911814988,
+            0.009038525980341447,
+            0.014672270369917403,
+            -0.019700405219230957,
+            0.012625337070556061,
+            0.012184185186281482,
+            0.019382321278525114,
+            0.0007219498737851579,
+            0.002564970959687783,
+            -0.010601873823737926,
+            -0.0029963537053764736,
+            -0.002070567089566093,
+            0.029563028293936092,
+            0.0005903976560069006,
+            -0.010509758568838935,
+            0.010992421945232951,
+            0.020603868018231348,
+            -0.02101157589451638,
+            -0.001971974816402967,
+            0.005963911670920596,
+            -0.010704207628592966,
+            -0.014466113540150018,
+            -0.021269176952661614,
+            -0.013331247081394942,
+            0.0071597345125287174,
+            -0.008162102923896894,
+            0.02367549407145414,
+            -0.014256771777502395,
+            -0.03960796112042958,
+            -0.014803626428839152,
+            -0.00864320347698291,
+            -0.027531337207277483,
+            -0.007305441146142331,
+            0.00965056093889974,
+            0.016698589600806273,
+            -0.003330867073501157,
+            -0.025198703196464594,
+            0.030483790406960656,
+            0.008872821868559751,
+            0.023708422767352917,
+            -0.021464374508407488,
+            -0.0018617153194555344,
+            -0.00886376784514331,
+            0.014505918224188001,
+            -0.017588761818663975,
+            -0.011756112651591795,
+            0.020014632992762026,
+            0.014356419813769714,
+            0.007246557500044434,
+            -0.004355132555323843,
+            0.0067299398462013936,
+            0.007649499235136303,
+            -0.012515181210363267,
+            0.012239792251154642,
+            -0.013499277313916029,
+            -0.024858295126228433,
+            0.014430561986516884,
+            0.002229494935179828,
+            0.009563004428064055,
+            -0.0015984171085659323,
+            -0.016197659608768088,
+            0.010078159490026602,
+            0.00808729175515505,
+            -0.017508840440512225,
+            -0.004793934415922065,
+            0.007133527615305415,
+            -0.020818869260580315,
+            0.031234686259160864,
+            0.011872765152279094,
+            -0.010692949770503467,
+            -0.001989039629661815,
+            0.004431079028880923,
+            0.016511544608975148,
+            0.007893700385963683,
+            -0.01997314388229276,
+            -0.003708006454566117,
+            0.01530128306203401,
+            0.00822588305477249,
+            0.004861703074728471,
+            -0.010836328783594584,
+            0.03434874763691911,
+            0.006082873924472328,
+            4.192481378506604e-05,
+            -0.0052997095110967115,
+            -0.0076108924981838355,
+            0.004196112457289409,
+            0.018905391806961686,
+            0.013330078288599763,
+            -0.004200590783114919,
+            -0.006630107527854746,
+            0.007778448128358027,
+            -0.01706991160178345,
+            0.007329056462125937,
+            0.009390327499494826,
+            -0.012870674959022891,
+            -0.0017571189976160295,
+            0.016462608870101713,
+            0.00209846661139665,
+            0.004942714850185376,
+            -0.011305050739130339,
+            0.004252776146827917,
+            0.015841821944109317,
+            0.018727567181082646,
+            -0.008399642221255733,
+            0.021448055799259867,
+            -0.020972459569228434,
+            0.027243639465535856,
+            0.028499400850980086,
+            0.016573144403275928,
+            -0.01847096443459798,
+            -0.0022409373140116146,
+            -0.0041737366139895414,
+            -0.014761716560267328,
+            0.0034546020161337894,
+            0.005125870074458691,
+            -0.01174186608185822,
+            -0.009577921232380895,
+            -0.024331993794416026,
+            0.011075959943215117,
+            0.007918147902741738,
+            -0.0015045185909419564,
+            -0.011791155919533844,
+            0.016019077577552835,
+            0.010141868950981925,
+            -0.010938402325619172,
+            -0.005635351352447256,
+            -0.019967361901734836,
+            0.009750427567651833,
+            0.009128294017175602,
+            -0.00300556760891084,
+            -0.011496103417643132,
+            0.012977890524465716,
+            0.0014080059843484115,
+            -0.01292380658817521,
+            0.010841954636959441,
+            -0.03142011651894961,
+            -0.005510537354374977,
+            -0.011027026734432255,
+            -0.003620482919145167,
+            -0.014415606143566519,
+            0.003608572766363589,
+            0.0005034611321129689,
+            -0.011470488351158329,
+            -0.0035005774696278,
+            -0.016207453351921627,
+            0.004676327282262429,
+            0.015021513761697039,
+            0.001679614645967629,
+            -0.006917375094634678,
+            0.021994274901326064,
+            0.018518524483306403,
+            -0.021903538417919984,
+            -0.020116013379669034,
+            0.004708084983787958,
+            -0.03261690161800251,
+            0.015019348187107588,
+            0.00966124471577622,
+            0.011676937880751252,
+            -0.028768229226408037,
+            0.007114534979415504,
+            0.00900420165933673,
+            0.016398927481511824,
+            0.0050879301160439245,
+            0.009453255486450082,
+            0.01222758288108498,
+            -0.0104657614414275,
+            -0.003164502778525433,
+            -5.410382753651127e-06,
+            -0.005618225986727351,
+            0.015607620507415484,
+            0.0008527697935056332,
+            0.03603666488542239,
+            -0.005277207843139309,
+            -0.0009799879252482288,
+            -0.011735731235560851,
+            0.00812046801749312,
+            -0.011713349784345638,
+            0.006334172347602515,
+            -0.0024643499727789167,
+            0.0001240009282616789,
+            0.002965067818077541,
+            0.009442572075213767,
+            -0.01086095783353607,
+            -0.007595379345224076,
+            0.007444897256325813,
+            -0.01482814340478338,
+            -0.00019984717819268013,
+            -0.009339343730980451,
+            0.0019680804106275156,
+            -0.02241031895876346,
+            0.006629838013824249,
+            0.040935644507527846,
+            0.02140216465404127,
+            -0.005481040712269146,
+            -0.012431659209559418,
+            -0.004680230194268222,
+            -0.007229300034633229,
+            -0.006482401275991039,
+            -0.004794698160419071,
+            -0.022762405726589375,
+            -0.006928553505112732,
+            0.004320321244630314,
+            0.013300138724212552,
+            0.009741018117644266,
+            -0.008842135469729511,
+            0.004531760574809748,
+            -0.02662650543210291,
+            -0.016578081411548973,
+            -0.0060021425889846495,
+            -0.015840130031005147,
+            0.006125301421471048,
+            0.027521841635711787,
+            -0.0063432924325983525,
+            0.01789356321527744,
+            0.005142029893259461,
+            0.013034833065163448,
+            -0.005338869351330258,
+            0.00035250243782976347,
+            -0.006010938630154189,
+            0.0032958656042048292,
+            0.003074861351445954,
+            -0.0015947217852680714,
+            0.004151590621941346,
+            -0.007139285822575166,
+            -0.0026091937868967284,
+            0.0011629380311555433,
+            0.0117841884590861,
+            0.006877419488576261,
+            -0.00016252653621991807,
+            -0.01761285866765538,
+            0.022221104524808997,
+            -0.018933508559881868,
+            -0.008840161102111973,
+            0.011092684029552064,
+            -0.009660487856185668,
+            0.00702597023411946,
+            0.020387386977378873,
+            -0.006854213562767843,
+            -0.009583981919303215,
+            0.013028181087649146,
+            -0.0036507672754997286,
+            0.004464368543493108,
+            0.009867428978560914,
+            0.006036303973414791,
+            0.002482944923100156,
+            0.013244650094892797,
+            0.004530294330688855,
+            -0.0032625482538717137,
+            -0.020238533752799943,
+            -0.005186498720507101,
+            0.019015752262933585,
+            -0.02252458872825473,
+            -0.0027641215988942563,
+            0.0044506144202026456,
+            -0.0030317840657635457,
+            -0.01733962766050954,
+            0.004205861489436576,
+            -0.018770275720867473,
+            -0.0033883519277721433,
+            -0.008244668812709337,
+            0.009948090119054079,
+            -0.023700732627507225,
+            0.009109315916173436,
+            -0.011739192106106307,
+            0.02356509175177699,
+            -0.003925851077921686,
+            0.024734433836185024,
+            0.0015142645677087268,
+            -0.013482933030628261,
+            0.0055323884713736554,
+            -0.0008930661843169134,
+            -0.004477731568313757,
+            0.029338490404300612,
+            -0.005418258701133154,
+            0.002249996261871466,
+            0.013491124734151694,
+            -0.021332541202417246,
+            0.021773687217263173,
+            0.0046662696076723836,
+            0.01628697326539432,
+            -0.00986238951830334,
+            -0.013799209313714441,
+            0.025178874553620743,
+            0.03324555035648896,
+            0.008398796308070151,
+            0.011261509865513953,
+            -0.01226890254177634,
+            0.018966175575083295,
+            -0.0038328997984309597,
+            0.004344837325292706,
+            0.00878668880458693,
+            -0.007200842513742962,
+            -0.011500562922303248,
+            -0.004533014670236755,
+            0.027639894892383943,
+            -0.021838848944317424,
+            -0.008920629621611768,
+            -0.005739181836252042,
+            -0.01593272996094732,
+            -0.038167534267554606,
+            -0.004371701247332816,
+            0.019604405563120094,
+            -0.005974358141737469,
+            -0.0018341151594086288,
+            0.014334266690716505,
+            0.00402620072221496,
+            -0.002852569009185304,
+            -0.008195218707838913,
+            -0.0027944153624309856,
+            -0.0059390492900516225,
+            0.016651206007022145,
+            -0.004961864528562493,
+            -0.007929830585906699,
+            0.01145431100211762,
+            -0.005731957927601062,
+            0.012130694936714224,
+            0.0004848418056781251,
+            8.392048424584131e-05,
+            -0.0015174120840537315,
+            0.00822932723013239,
+            0.018620869354789082,
+            -0.0055803454666417465,
+            0.025682778991091174,
+            0.006956936033581638,
+            -0.003495709501259274,
+            -0.005489431577052735,
+            -0.0044154293119170445,
+            -0.024561016942073285,
+            -0.0020864636079347357,
+            0.006593060072668424,
+            0.024333813203596122,
+            -0.02515989187381735,
+            -0.004953266000463339,
+            0.020957806164031402,
+            -0.0012192724005613,
+            -0.011978722479596745,
+            0.014690559986694138,
+            0.0041346025614047875,
+            0.009881584250499358,
+            0.012176393203318002,
+            0.015764530703333238,
+            -0.004923548157246672,
+            -0.0015192775000207833,
+            0.012766482928380375,
+            0.0005745973040874939,
+            0.00997809510696241,
+            0.016495444272181515,
+            0.010467514633877887,
+            -0.021108281220544853,
+            0.0015078843789125156,
+            0.00522593579521476,
+            -0.017481430635971396,
+            0.0034362821987740755,
+            0.005397819166037461,
+            0.011626527721895694,
+            0.004048632382026344,
+            0.013207335578216437,
+            -0.015768003434275563,
+            -0.005916168437120099,
+            0.008153798523776654,
+            0.004319451296835019,
+            -0.02599474886669325,
+            -0.01485009085106089,
+            -0.0006781964116146299,
+            -0.004444144921332929,
+            0.013556265879295017,
+            0.01285768162554067,
+            0.02033626001285354,
+            0.024637349990579514,
+            -0.013833060730799217,
+            -0.00842819657792762,
+            0.009436072455719896,
+            0.015560357465711765,
+            0.005798181411384387,
+            0.007874051721372273,
+            0.009297440071118126,
+            0.0077320823375042,
+            -0.004073209576661068,
+            -0.002973928843567532,
+            0.020205003097225747,
+            -0.019725382393637726,
+            0.0006746659838642147,
+            -0.018417443693472958,
+            0.007530796896254817,
+            0.01140696723506382,
+            0.01129515042745398,
+            0.002816883643872417,
+            -0.005274757308627299,
+            0.005716917047812153,
+            -0.06254973656832136,
+            -0.009191453047502128,
+            0.003384751551032943,
+            0.027256466708746898,
+            -0.010885336140129142,
+            -0.003415707910562931,
+            0.007309577090125959,
+            0.005511970708769456,
+            0.0005572239548625122,
+            0.003186427278034852,
+            0.019698716902282903,
+            -0.02052581021967601,
+            -0.010793968505805883,
+            0.004219269922138548,
+            0.0033111837923746746,
+            -0.007269521534951044,
+            0.004805737092302134,
+            0.014143978208501908,
+            0.01986656063456352,
+            -0.0024048854610842748,
+            0.01629335842915328,
+            -0.0015498603704589015,
+            -0.014496994444493684,
+            -0.002671064325909516,
+            0.0037875833047555925,
+            -0.022466433279066646,
+            -0.005980897597195858,
+            0.0043262697442044435,
+            -0.011226121648007667,
+            0.006978495397818936,
+            0.008452177557316863,
+            -0.003018545465557218,
+            -0.0008303307644324117,
+            0.017796789945838845,
+            0.006591364879905198,
+            -0.013464486184025026,
+            0.0028668426952326957,
+            -0.023377605222213174,
+            0.004325185794796361,
+            0.001416245546006914,
+            0.007809449561523174,
+            -0.008431335139468173,
+            -0.014774499666285603,
+            -0.005933810712528226,
+            0.017746847020610673,
+            -0.01965188730040385,
+            -0.002406875192894708,
+            0.01683009662984443,
+            -0.0002518220110313279,
+            0.005456812494894182,
+            -0.02011241622618092,
+            -0.0009782974741561381,
+            -0.02933296756078644,
+            -0.01171823509606186,
+            0.001162345317872531,
+            0.00892221099991475,
+            -0.006551308022163148,
+            -0.03934356275100932,
+            0.00427623835776202,
+            -0.013111664387776261,
+            -0.008752857644809231,
+            -0.010214546496134218,
+            -0.00518087722852466,
+            0.015240297918667867,
+            -0.012498100033235827,
+            0.01086490299272505,
+            -0.00917777212602538,
+            0.005895055740265625,
+            0.0090554651297836,
+            0.014212841798768113,
+            -0.005544047915369086,
+            -0.0033589999440643385,
+            -0.007924830667307956,
+            0.0022291797354234005,
+            -0.0032976226056924502,
+            0.009615674261248298,
+            -0.0051932155872883,
+            -0.0065636389655770455,
+            0.02426106498596339,
+            0.011445591067574932,
+            0.013312596099532533,
+            0.0055118624272110965,
+            -0.007220620072349783,
+            -0.01916551187613823,
+            -0.003977616240384473,
+            -0.013204506708246246,
+            -0.006315966975649096,
+            0.01434099854765068,
+            0.012588249778961223,
+            0.006357972880218866,
+            0.0007747002406996095,
+            -0.004794439624135275,
+            0.006975593520375694,
+            0.0262335496080837,
+            -0.016481909971520166,
+            -0.0013084662210086592,
+            -0.01369313900978409,
+            0.01921748474212319,
+            0.03333761672336437,
+            -0.0008918419419301244,
+            -0.007699525186523558,
+            0.013757216065348164,
+            0.00819045303047094,
+            -0.007411380297164911,
+            0.02453611080891859,
+            -0.025045462880463647,
+            -0.015989319900023054,
+            -0.011650886878708265,
+            -0.011407253985176972,
+            -0.008386367959227559,
+            0.016331961716602287,
+            -0.00141225438918211,
+            -0.0018505754298268654,
+            0.009557291617446645,
+            0.03357924917013024,
+            0.006916860901290756,
+            0.007030308953738567,
+            -0.002739345982346728,
+            -0.009376522438071562,
+            0.011482637968411055,
+            -0.007259054391879614,
+            -0.04075661435928125,
+            0.004250195164583401,
+            -0.005160505473755424,
+            0.02405623169060588,
+            -0.02197586561375289,
+            -0.00619175761282745,
+            -0.006736129049831973,
+            -0.028200015908139763,
+            -0.005867722104196303,
+            0.02805071895018145,
+            -0.015402198567775478,
+            0.014773294766214671,
+            -0.009502185346558072,
+            -0.0013419858580438616,
+            0.004443749010136947,
+            0.013008783570355656,
+            -0.02289662952366337,
+            0.0013834888740540108,
+            -0.00048549197205067054,
+            0.027969842651517314,
+            0.010437729090975158,
+            -0.0047705656249623744,
+            0.014202218020412936,
+            0.008240861154198411,
+            -0.018637315941614406,
+            0.007314019317343154,
+            -0.005747966765101792,
+            0.014833896739762147,
+            -0.01314847940891154,
+            0.007762100494499689,
+            0.00705026393499352,
+            -0.01278023669088385,
+            0.03016342404000438,
+            -0.015204129985039077,
+            0.007776946006562982,
+            0.009278778571285689,
+            0.009656723862179986,
+            0.01257686993982689,
+            0.004362439504889069,
+            -0.02898418193036562,
+            0.0002144431781454467,
+            -0.03828100004144633,
+            -0.00833748467467171,
+            -0.002611937107022567,
+            -0.002145329666219461,
+            0.007050630951707937,
+            -0.009383645162261629,
+            -0.033976442057557676,
+            0.0006429978367640709,
+            0.01603027598131823,
+            0.017501067413108286,
+            0.0002196997337451293,
+            0.013075379412332973,
+            -0.006197537539554383,
+            -0.021148120787551232,
+            -0.016986448660412497,
+            -0.010207705210051413,
+            -0.02122194744719993,
+            -0.020550470201191853,
+            -0.00908564461857315,
+            0.012090079127547193,
+            -0.009927390005879807,
+            0.013888458967497627,
+            0.029513250508042935,
+            -0.024529175134769883,
+            0.0011144447737101862,
+            0.021977677894839922,
+            -0.00198308986575084,
+            0.014108189311291326,
+            0.001696775006906642,
+            0.014122064251288995,
+            -0.0004166140484018266,
+            -0.010556016685519523,
+            -0.009729362492661684,
+            0.017487929296614076,
+            -0.0026043257634797124,
+            0.016715798948897194,
+            0.012183487338360088,
+            0.003771721868999568,
+            -0.0030887189945795796,
+            0.010667776697461199,
+            -0.026471157050402344,
+            0.014249996582064706,
+            -0.006690961709923224,
+            -0.0181382691440242,
+            -0.008729104191096461,
+            0.003882731567105777,
+            -0.005609384377642956,
+            -0.007794812949125115,
+            0.012967089730551055,
+            -0.00944374111323783,
+            0.02088525981049842,
+            0.0023400121055514958,
+            -0.012196925240774426,
+            0.003648688547842322,
+            -0.03322041209304968,
+            0.0015191765655963997,
+            -0.018402382776013956,
+            -0.024659631472836927,
+            0.03840173429666435,
+            0.011991452465253198,
+            0.006639702644606365,
+            -0.01675838115087676,
+            0.007953102302963526,
+            0.0004257479565099406,
+            0.017484338819046517,
+            0.011068864013925452,
+            0.006466915839538296,
+            0.008364454017976605,
+            -0.023316137291380285,
+            -0.010898421318668592,
+            0.016204569093730564,
+            0.012099473359791148,
+            -0.01017229369980751,
+            0.019431669546223253,
+            0.017955119269367787,
+            0.007300372071724306,
+            -0.006898619924677706,
+            0.01647927363640854,
+            0.021012103522831126,
+            0.010291016618557653,
+            -0.00047955552103474145,
+            0.0008684297441778696,
+            -0.010534247928997246,
+            0.005037930235889528,
+            0.035417626740307205,
+            0.014712595692093886,
+            0.010650306363963653,
+            -0.03626584475133921,
+            0.021764456453493054,
+            -0.011961521673661574,
+            0.009092981192874177,
+            0.006089104887183293,
+            0.017982801101846038,
+            0.006916969397519662,
+            0.014196254958707752,
+            0.011745069030657836,
+            -0.006689305450954901,
+            0.025205280281804708,
+            -0.001601414806100461,
+            -0.011992525142592405,
+            0.003386930324498371,
+            -0.01859193251438437,
+            0.005303261319773131,
+            -0.0041710536965100615,
+            -0.004459393120257307,
+            0.0032461828144552565,
+            -0.0018650956576455287,
+            0.015678225251799887,
+            0.003894142628084471,
+            -0.02355212701026675,
+            -0.0014873755115206368,
+            -0.024284295126348117,
+            0.0027260592623277375,
+            -0.022185571998195344,
+            0.00022883605560765203,
+            -0.005557568353382136,
+            -0.009749382813201403,
+            -0.025385857431333838,
+            -0.009853073898348951,
+            -0.0035843255959309307,
+            0.007661137888845632,
+            -0.010521037433807653,
+            -0.004246698084422667,
+            -0.0019920499321964815,
+            -0.019465113363548715,
+            0.009014255846983305,
+            0.027122533756959888,
+            0.016848221859975445,
+            0.027170883580103264,
+            -0.00019554112591107156,
+            0.013614675849485926,
+            -0.002066822813103284,
+            -0.01603063005944364,
+            0.008824663429777616,
+            0.0075653522381218495,
+            -0.00531780740880296,
+            0.014643348459785112,
+            0.0072634484937344704,
+            -0.012800149015319017,
+            0.009859503660384916,
+            -0.007161712176469609,
+            -0.006437857349742842,
+            -0.009764856421740622,
+            -0.02058868349990357,
+            -0.014866137971603676,
+            0.0012450207572602557,
+            -0.002123265406747006,
+            0.006000473862248991,
+            -0.004054016790850226,
+            0.009584879012937315,
+            -0.00608816807208866,
+            -0.00925801057439376,
+            -0.013962683072792106,
+            -0.008924356118158748,
+            -0.004210064145949253,
+            -0.004381709838058668,
+            -0.011732699976353612,
+            -0.005479523097469261,
+            0.0010680258077037758,
+            -0.0019401640391035755,
+            -0.01397652201595301,
+            0.011650839243603377,
+            0.0058731838291845645,
+            -0.002462743134993977,
+            0.02219008688925881,
+            0.01097316136828334,
+            0.0018083001685932406,
+            0.01366095798812196,
+            0.0003942601242216818,
+            -0.0064063532958756175,
+            -0.03343312148366622,
+            -0.0005083355836126291,
+            -0.02815586367527057,
+            0.025770078162165123,
+            -0.0069114949102666145,
+            -0.010322087105114614,
+            -0.002519047562374146,
+            -0.006063669644021698,
+            0.005873609611264377,
+            -0.002722940652283699,
+            0.0021229525800901114,
+            -0.023673778230768945,
+            0.0028629715379660113,
+            -0.003953653437464966,
+            -0.0036366957271968823,
+            0.007469664864329332,
+            0.0346520690389124,
+            -0.004054282838167587,
+            -0.0034765253405897676,
+            0.005159466707114532,
+            0.011642913651743502,
+            0.027845571220425726,
+            -0.004445196354520585,
+            -0.0032038508043339413,
+            -0.02233838584152591,
+            0.0054112576531374175,
+            -0.00019544838850471866,
+            -0.004733192299951088,
+            0.016664217348411003,
+            -0.016719175418727594,
+            -0.01631658832047864,
+            -0.00545969677717507,
+            -0.014570848053102338,
+            -0.00022620258237138677,
+            0.014454101124666536,
+            0.011056312437637014,
+            -0.004035526363810078,
+            -0.012459378363360557,
+            -0.0022595493558020706,
+            -0.030370952502579162,
+            0.012766010284884204,
+            0.1141689619126009,
+            -0.00766197047210299,
+            -0.00896655658602649,
+            -0.009481112393590025,
+            0.001435269335646347,
+            0.001338755472877713,
+            -0.011739143028767118,
+            0.07166096084053437,
+            -0.0077855767070652395,
+            0.0002139580430342728,
+            -0.011312198831181402,
+            0.001844005266872301,
+            -0.001825412693163496,
+            0.01371932421024605,
+            -0.005025814503687022,
+            -0.018213360991353643,
+            -0.0033612481616769416,
+            -0.02941403830641238,
+            -0.0041137865872933515,
+            -0.005683630256743215,
+            -0.004288198570156578,
+            -0.014982999363445122,
+            0.0067149665791261335,
+            -0.019661973497372313,
+            -0.012688556129040908,
+            0.001124391650175233,
+            -0.0033623453063430534,
+            0.00996944383866679,
+            -0.020496336814603385,
+            -0.0044354321069609536,
+            0.0019875840782527733,
+            0.009706675240563883,
+            0.0011600364428219214,
+            -0.012755178230821602,
+            -0.0026514442584739412,
+            0.004056350166031111,
+            0.01909951327913368,
+            0.014360281326789544,
+            0.003732344102045803,
+            0.008623457695718206,
+            0.014461868085333023,
+            -0.001692149528228451,
+            0.01948035376572433,
+            0.01172550239477637,
+            0.0031966531144076173,
+            0.0004779413449255626,
+            0.0030912562273611517,
+            -0.004597633235071031,
+            0.01582547400214653,
+            -0.014811177637844808,
+            -0.02411202772171917,
+            -0.0071272238837422185,
+            0.018015541137524113,
+            -0.013081287852028074,
+            1.8837689271256874e-05,
+            0.004062803284646432,
+            0.027569726868200254,
+            0.0010564850664255057,
+            -0.01179755720247972,
+            0.0028876134800690334,
+            -0.01638215925131053,
+            0.022944183106912663,
+            0.0026043713064243836,
+            0.03279173899194346,
+            -0.005103903273968361,
+            0.00578213669463383,
+            -0.020744194169889544,
+            -0.007588793365216579,
+            -0.00819862611489168,
+            0.0017401442433921013,
+            0.010993500174794084,
+            0.005411238350013134,
+            0.01620925462604661,
+            -0.00641615069699833,
+            -0.020751730783437995,
+            0.014776878596707538,
+            -0.009552193304486048,
+            -0.03393319554458277,
+            0.008881987295131347,
+            -0.029102297123590998,
+            -0.0045267103602711,
+            0.00368971742585232,
+            -0.014269341759285646,
+            -0.016264809810970175,
+            0.012837302095591106,
+            -0.0011890334180728564,
+            -0.008937992768323412,
+            0.0014444613725248846,
+            -0.017110187133760626,
+            -0.02385668037818897,
+            0.006017405364919834,
+            0.00676670672597817,
+            -0.0077994500447595,
+            0.018312452858615755,
+            -0.027015660037024585,
+            0.006123912473734307,
+            -0.012973676426738582,
+            0.009041974762537394,
+            -0.01580183887345168,
+            0.01163516332546347,
+            0.010887662470499361,
+            -0.001503467664709027,
+            -0.0076234124970223326,
+            0.01231202541338883,
+            0.004852720134894811,
+            0.033760365210093034,
+            0.028658326381800395,
+            0.000995999783627898,
+            -0.0025605367058226737,
+            -0.0020356794893366384,
+            0.011667525096213252,
+            -0.014049700287539817,
+            0.011218246246280464,
+            0.02658113044439717,
+            0.017568631325838108,
+            -0.019133062654008885,
+            0.018050678772708717,
+            -0.010356118256625172,
+            0.012872052420999813,
+            -0.025620154415048252,
+            -0.005135768485044442,
+            0.004936490920616527,
+            0.005163008729256001,
+            0.007425278892536464,
+            0.001410212306439329,
+            0.021891387289347516,
+            0.014999616600766989,
+            0.00027613811156650413,
+            0.00558422822386443,
+            0.01477009526981092,
+            0.010497782599293994,
+            -0.0005567894749233204,
+            -0.00467776506604011,
+            -0.016998035358377343,
+            0.00957271797745408,
+            -0.006065116857898469,
+            0.005138070401846389,
+            0.01698914855850691,
+            -0.01230257809343985,
+            0.011109844802663414,
+            0.007518553660463051,
+            -0.015504773927595823,
+            -0.017169933945839305,
+            0.005552829003043926,
+            -0.02252633790984703,
+            -0.01306722099657118,
+            0.02715689072730398,
+            -0.010513941399990838,
+            -0.02790582732974509,
+            -0.0009847768069205477,
+            0.009797865241108237,
+            -0.002772952073769185,
+            -0.0027968741526000576,
+            0.0037541733651479613,
+            0.010801137722284016,
+            -0.014658156970099418,
+            0.002075415233345957,
+            0.004735264309137368,
+            0.01930804315461151,
+            -0.0014151522239496115,
+            -0.015046118414100617,
+            -0.002288495894227231,
+            0.021592829677937263,
+            0.017174609713578173,
+            0.00265560378681856,
+            -0.016499825201963983,
+            -3.965296928410772e-05,
+            -0.0074142747014567635,
+            0.00021360696629503726,
+            0.00027431174904249525,
+            -0.0056391089282546054,
+            -0.018931681700770225,
+            -0.003888248121485032,
+            0.011181675419027736,
+            -0.00247252920175839,
+            0.01977860312223846,
+            -0.0011527575128223392,
+            -0.015763587433200915,
+            0.001960734212846797,
+            -0.005280860341480037,
+            -0.010372829570478649,
+            0.01351900743232294,
+            -0.009359942374603997,
+            0.006788746813046239,
+            -0.003212015225284119,
+            0.004611143085183367,
+            -0.0003308099698088422,
+            -0.017017265064185255,
+            0.02924142709831441,
+            -0.0018238549856433466,
+            -0.004769274041546295,
+            -0.0014058363976586562,
+            -0.033393331166741945,
+            0.015017165991303838,
+            -0.014574579250749552,
+            0.006745699409061803,
+            0.002705853802154516,
+            0.01272741296341735,
+            0.003253095246601883,
+            -0.008898616461779179,
+            0.0027485606928249326,
+            0.017040360927723885,
+            0.013756402798365262,
+            -0.007292864293090161,
+            0.008694855886257706,
+            -0.0028934389922887676,
+            -0.0037483208383542015,
+            0.01363963339139356,
+            0.004170611039133969,
+            -0.006282657671540625,
+            -0.0009784372731140945
+          ],
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "Mixture-of-Experts (MoE)",
+            "fine-grained expert segmentation",
+            "shared expert isolation"
+          ],
+          "title": "Deepseekmoe: Towards Ultimate Expert Specialization In Mixture-Of-Experts Language Models"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "317d6fdf-f473-4de1-9258-7ec217a9b4ca",
+        "properties": {
+          "page_content": "3. Deepseekmoe Architecture\n\nOn top of the generic MoE architecture outlined in Section 2, we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. As illustrated in Figure 2, our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. Both of these strategies are designed to elevate the level of expert specialization.\n\n3.1. Fine-Grained Expert Segmentation\n\nIn scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts.\n\nIn pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. The finer expert segmentation enables a more flexible and adaptable combination of activated experts. To be specific, on top of a typical MoE architecture shown in Figure 2(a), we segment each expert FFN into smaller experts by reducing the FFN intermediate hidden dimension to 1 times its original size. Since each expert becomes smaller, in response, we also increase the number of activated experts to times to keep the same computation cost, as illustrated in Figure 2(b). With the fine-grained expert segmentation, the output of an MoE layer can be expressed as:\n\n$$\\mathbf{h}{t}^{i}=\\sum{i=1}^{\\text{mN}}\\left(g_{i,t}\\text{FFN}{i}\\left(\\mathbf{u}{t}^{i}\\right)\\right)+\\mathbf{u}{t}^{i},\\tag{6}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\text{Tokr}(s_{i,t}|1\\leqslant j\\leqslant m\\text{N}),m\\text{K}),\\ 0,&\\text{otherwise},\\end{cases}$$ (7) $$s_{i,t}=\\text{Softmax}{i}\\left(\\mathbf{u}{t}^{T}\\mathbf{e}_{t}^{i}\\right),\\tag{8}$$ where the total number of expert parameters is equal to $N$ times the number of parameters in a\n\n$$(6)$$\n\n(7) (8) $\\frac{1}{2}$\n\nstandard FFN, and denotes the total number of fine-grained experts. With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to .\n\nFrom a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. As an illustrative example, we consider the case where = 16. A typical top-2 routing strategy can yield 16 2 = 120 possible combinations. By contrast, if each expert is split into 4 smaller experts, the fine-grained routing strategy can yield 64 8 = 4, 426, 165, 368 potential combinations. The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition.\n\n3.2. Shared Expert Isolation\n\nWith a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. This alleviation of redundancy will contribute to a more parameterefficient model with more specialized experts.\n\nTowards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate experts to serve as shared experts. Regardless of the router module, each token will be deterministically assigned to these shared experts. In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by , as depicted in Figure 2(c). With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows:\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{K_{s}}\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)+\\sum_{i=K_{s}+1}^{mN}\\left(g_{i,t}\\;\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},\\tag{9}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\mathrm{Topk}{s_{j,t}|K_{s}+1\\leqslant j\\leqslant mN},mK-K_{s}},\\ 0,&\\text{otherwise},\\end{cases}$$ (10) $$s_{i,t}=\\mathrm{Softmax}{i}\\left(\\mathbf{u}{t}^{l,T}e_{i}^{l}\\right).\\tag{11}$$ $\\mathbf{h}\\mathbf{M}\\mathbf{E}$ the number of closed-compact is $K_{s}$ the total number of routed currents.\n\nFinally, in DeepSeekMoE, the number of shared expert is , the total number of routed experts is − , and the number of nonzero gates is − .\n\nIt is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022). The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint.\n\n3.3. Load Balance Consideration\n\nAutomatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. Firstly, there is a risk of routing collapse (Shazeer et al., 2017), i.e., the model always selects only a few experts, preventing other experts from sufficient training.\n\nSecondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.\n\nExpert-Level Balance Loss. In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. The computation of the balance loss is as follows:\n\n$$\\begin{array}{l}{{{\\mathcal L}{\\mathrm{ExpBal}}=\\alpha{1}\\sum_{i=1}^{N^{\\prime}}f_{i}P_{i},}}\\ {{f_{i}=\\frac{N^{\\prime}}{K^{T}}\\sum_{t=1}^{T}\\mathds{1}\\,(\\mathrm{Token}\\;t\\;\\mathrm{s}\\mathrm{lects}\\;\\mathrm{Expert}\\;i),}}\\ {{P_{i}=\\frac{1}{T}\\sum_{t=1}^{T}s_{i,t},}}\\end{array}$$\n\n(12) $$\\begin{array}{l}\\small\\text{(13)}\\end{array}$$ = (14) $$\\begin{array}{l}\\small\\text{(14)}\\end{array}$$ .\n\n7 where 1 is a hyper-parameter called expert-level balance factor, ′is equal to ( − ) and ′ is equal to ( − ) for brevity. 1(·) denotes the indicator function.\n\nDevice-Level Balance Loss. In addition to the expert-level balance loss, we introduce a devicelevel balance loss. When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. Instead, our primary objective is to ensure balanced computation across the devices. If we partition all routed experts into groups {E1, E2, ..., E}, and deploy each group on a single device, the device-level balance loss is computed as follows:\n\n$$\\mathcal{L}{\\text{DevBal}}=\\alpha{2}\\sum_{i=1}^{D}f_{i}^{\\prime}P_{i}^{\\prime},$$ $$f_{i}^{\\prime}=\\frac{1}{|\\mathcal{E}{i}|}\\sum{j\\in\\mathcal{E}{i}}f{j},$$ $$P_{i}^{\\prime}=\\sum_{j\\in\\mathcal{E}{i}}P{j},$$\n\n(15) $$\\begin{array}{l}\\small\\text{(16)}\\end{array}$$ = (17) .\n\nwhere 2 is a hyper-parameter called device-level balance factor. In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices.\n\n",
+          "title": "Deepseekmoe Architecture",
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "fine-grained expert segmentation",
+            "shared expert isolation",
+            "load balance consideration"
+          ],
+          "embedding": [
+            -0.004791774787008762,
+            -0.04079068824648857,
+            -0.01656515710055828,
+            -0.01955496519804001,
+            -0.006609901320189238,
+            0.034003015607595444,
+            -0.01574093848466873,
+            0.050455041229724884,
+            -0.04541277140378952,
+            0.02537297084927559,
+            -0.010747150518000126,
+            -0.015199541114270687,
+            0.008557317778468132,
+            -0.01236326340585947,
+            -0.001814086688682437,
+            0.03117481619119644,
+            0.022674063220620155,
+            0.022867996245622635,
+            0.041534099727869034,
+            -0.032096002250909805,
+            0.02517903782427311,
+            -0.04825713112950325,
+            -0.03211216256022453,
+            0.018375203013420105,
+            -0.020395344123244286,
+            0.01574093848466873,
+            0.023368991911411285,
+            0.00403220159932971,
+            -0.06251124292612076,
+            0.021995296701788902,
+            0.029623348265886307,
+            0.015635890886187553,
+            0.006242235656827688,
+            -0.0028645601123571396,
+            -0.034003015607595444,
+            -0.022593257948756218,
+            0.029300125315785408,
+            0.025227520614862442,
+            -0.026358800008893013,
+            0.03578073903918266,
+            0.03626557067036629,
+            0.021866006776690483,
+            -0.0449279360473156,
+            0.007236145436763763,
+            -0.0028686001896858215,
+            0.004626123234629631,
+            -0.015256104990839958,
+            0.008306819945573807,
+            0.01797117479145527,
+            -0.036459505558013916,
+            -0.015344991348683834,
+            -0.02821733057498932,
+            0.012444068677723408,
+            0.003559488570317626,
+            -0.028928419575095177,
+            0.009276487864553928,
+            0.031077850610017776,
+            0.05061665549874306,
+            0.02391847036778927,
+            0.005931134335696697,
+            0.03561912849545479,
+            -0.007898751646280289,
+            -0.036136284470558167,
+            0.013381414115428925,
+            0.02915467508137226,
+            -0.021332690492272377,
+            0.01580558344721794,
+            -0.009599709883332253,
+            -0.006294759456068277,
+            0.009599709883332253,
+            -0.05798612907528877,
+            0.030916238203644753,
+            -0.034746427088975906,
+            0.000505792791955173,
+            -0.006007899530231953,
+            0.00270294863730669,
+            0.03772007301449776,
+            0.03183742240071297,
+            0.013324850238859653,
+            -0.028508231043815613,
+            -0.001757522695697844,
+            0.011183501221239567,
+            -0.014771271497011185,
+            0.0015050050569698215,
+            0.01929638721048832,
+            -0.04334414750337601,
+            -0.0023595248349010944,
+            -0.06988071650266647,
+            0.048612672835588455,
+            -0.0075997705571353436,
+            -0.029332447797060013,
+            0.003993819002062082,
+            0.004078664816915989,
+            -0.0046018813736736774,
+            -0.0031049568206071854,
+            0.059246696531772614,
+            -0.00016565156693104655,
+            -0.017599469050765038,
+            -0.03529590368270874,
+            0.0033534341491758823,
+            -0.0022443768102675676,
+            0.024177048355340958,
+            -0.029865765944123268,
+            0.009147198870778084,
+            0.028863774612545967,
+            -5.586952465819195e-05,
+            0.01226629689335823,
+            0.02770017459988594,
+            -0.006545256823301315,
+            0.09709606319665909,
+            -0.008993667550384998,
+            -0.005486703012138605,
+            -0.007801784668117762,
+            0.002919103717431426,
+            -0.014003617689013481,
+            -0.02333666943013668,
+            -0.03920689597725868,
+            -0.06483844667673111,
+            0.02320738136768341,
+            -0.044055234640836716,
+            -0.0021756920032203197,
+            0.0017140896525233984,
+            0.02535681053996086,
+            -0.022754868492484093,
+            -0.06299608200788498,
+            0.04172803461551666,
+            0.0025979015044867992,
+            0.003816046519204974,
+            0.013276367448270321,
+            0.013324850238859653,
+            0.014674304984509945,
+            -0.0171954408288002,
+            -0.006525055505335331,
+            -0.04761068522930145,
+            0.021736718714237213,
+            0.036459505558013916,
+            0.04476632550358772,
+            0.05426906794309616,
+            -0.0033837363589555025,
+            0.01847217045724392,
+            0.022932641208171844,
+            -0.035522159188985825,
+            -0.010456250049173832,
+            0.002995869144797325,
+            -0.023562924936413765,
+            0.0130581920966506,
+            0.00017979255062527955,
+            0.05426906794309616,
+            -0.011749139986932278,
+            0.026843633502721786,
+            -0.017793402075767517,
+            0.06380413472652435,
+            0.0016544954851269722,
+            0.021461978554725647,
+            0.0027574924752116203,
+            0.011143097653985023,
+            -0.022609418258070946,
+            -0.019118614494800568,
+            -0.0020393324084579945,
+            0.005167520605027676,
+            0.007381595205515623,
+            -0.03267780318856239,
+            -0.01442380715161562,
+            0.04104926809668541,
+            0.0012221853248775005,
+            0.05110148712992668,
+            0.007939154282212257,
+            0.008072483353316784,
+            -0.08242175728082657,
+            0.025227520614862442,
+            -0.006226074881851673,
+            0.022124584764242172,
+            0.022221552208065987,
+            0.040952298790216446,
+            -0.012985466979444027,
+            -0.022237712517380714,
+            0.026277994737029076,
+            0.0034786828327924013,
+            -0.022738708183169365,
+            0.0346817821264267,
+            -0.031982872635126114,
+            -0.009979496710002422,
+            -0.008250256069004536,
+            -0.02094482257962227,
+            0.03316263481974602,
+            0.02181752398610115,
+            0.02005596086382866,
+            0.0104966526851058,
+            -0.004832177422940731,
+            -0.007809865288436413,
+            0.031223300844430923,
+            0.02233467996120453,
+            0.011741059832274914,
+            0.006807875353842974,
+            0.006339202634990215,
+            -0.02702140621840954,
+            0.010456250049173832,
+            -0.008452270179986954,
+            0.0032544471323490143,
+            0.01527226623147726,
+            -0.00561599200591445,
+            -0.017550986260175705,
+            -0.0024928541388362646,
+            -0.019894348457455635,
+            0.022867996245622635,
+            -0.01039968617260456,
+            0.023433636873960495,
+            0.011312790215015411,
+            -0.004403907340019941,
+            -0.042794667184352875,
+            0.006254356820136309,
+            -0.01075523067265749,
+            -0.0016181330429390073,
+            -0.04415220394730568,
+            -0.01727624610066414,
+            -0.003593830857425928,
+            0.00556750874966383,
+            0.033615145832300186,
+            0.024500271305441856,
+            -0.0036200927570462227,
+            -0.00013534944446291775,
+            -0.007656334433704615,
+            0.008395706303417683,
+            0.0056079113855957985,
+            -0.0066947476007044315,
+            0.00873508956283331,
+            0.0026787070091813803,
+            0.01797117479145527,
+            0.018197430297732353,
+            -0.03406766057014465,
+            0.010957244783639908,
+            0.026471927762031555,
+            0.0039574564434587955,
+            0.019457997754216194,
+            0.02151046134531498,
+            -0.035522159188985825,
+            -0.013098594732582569,
+            -0.00875933188945055,
+            0.0058543686755001545,
+            -0.010302719660103321,
+            -0.020847855135798454,
+            0.0041029066778719425,
+            -0.056014470756053925,
+            -0.005272568203508854,
+            0.00509075541049242,
+            0.010165349580347538,
+            -0.008371464908123016,
+            0.05213579908013344,
+            0.00807652436196804,
+            0.007874510250985622,
+            -0.017130795866250992,
+            -0.010205752216279507,
+            0.028540553525090218,
+            -0.003670596284791827,
+            -0.03355050086975098,
+            0.03626557067036629,
+            0.01229861844331026,
+            -0.034003015607595444,
+            -0.004674606490880251,
+            -0.02656889520585537,
+            0.04421684890985489,
+            -0.020459989085793495,
+            -0.018245914950966835,
+            0.044055234640836716,
+            0.0020302417688071728,
+            -0.016985345631837845,
+            0.057598263025283813,
+            0.029122352600097656,
+            0.0038483687676489353,
+            -0.00885629840195179,
+            -0.004844298120588064,
+            -0.04082300886511803,
+            -0.015288427472114563,
+            -0.0186661034822464,
+            0.011280467733740807,
+            -0.016839895397424698,
+            -0.0046584452502429485,
+            -0.008104805834591389,
+            0.02079937234520912,
+            0.02902538701891899,
+            0.02327202446758747,
+            0.012629921548068523,
+            0.02144581824541092,
+            0.004185732454061508,
+            0.002262558089569211,
+            -0.006379605270922184,
+            -0.033744435757398605,
+            -0.02320738136768341,
+            -0.035263583064079285,
+            0.006092745345085859,
+            -0.01562781073153019,
+            -0.0018423686269670725,
+            -0.020540794357657433,
+            -0.01864994317293167,
+            0.05585286021232605,
+            0.006173551082611084,
+            -0.016161128878593445,
+            0.009737079963088036,
+            0.019393354654312134,
+            -0.031918227672576904,
+            -0.023934630677103996,
+            0.0034180786460638046,
+            0.009672435000538826,
+            0.007486642803996801,
+            -0.019619610160589218,
+            -0.0017534823855385184,
+            -0.04282699152827263,
+            0.030576854944229126,
+            -0.02176903933286667,
+            -0.010698666796088219,
+            0.0290900319814682,
+            0.008201772347092628,
+            0.006771512795239687,
+            -0.04340879246592522,
+            -0.015450038947165012,
+            0.021995296701788902,
+            0.020912500098347664,
+            0.03093240037560463,
+            -0.024710364639759064,
+            -0.022237712517380714,
+            0.016548994928598404,
+            0.007874510250985622,
+            -0.04638243839144707,
+            0.002288819756358862,
+            -0.009510824456810951,
+            -0.013147078454494476,
+            0.050455041229724884,
+            0.04243912175297737,
+            -0.03555448353290558,
+            -0.0016373243415728211,
+            -0.015094494447112083,
+            -0.014819755218923092,
+            -0.03726756200194359,
+            -0.013599589467048645,
+            -0.03694434091448784,
+            -0.014294518157839775,
+            -0.01682373508810997,
+            -0.008452270179986954,
+            -0.013648073188960552,
+            -0.018197430297732353,
+            -0.0077896639704704285,
+            0.006824036594480276,
+            -0.032047517597675323,
+            0.008137128315865993,
+            0.004391786642372608,
+            0.007894711568951607,
+            -0.0403381772339344,
+            -0.006444249767810106,
+            0.023320509120821953,
+            0.03144955635070801,
+            0.001097946660593152,
+            0.012759210541844368,
+            0.01808430254459381,
+            -0.0037130191922187805,
+            0.007979556918144226,
+            -0.052523668855428696,
+            -0.006383645813912153,
+            0.005078634712845087,
+            0.0022282155696302652,
+            0.000984818791039288,
+            0.021930651739239693,
+            -0.048160161823034286,
+            -0.038107942789793015,
+            -0.005874569993466139,
+            -0.04366736859083176,
+            0.01483591552823782,
+            0.004767532926052809,
+            0.0033695953898131847,
+            -0.005712958984076977,
+            -0.006573539227247238,
+            -0.023175058886408806,
+            0.022916480898857117,
+            0.008727009408175945,
+            -0.008670445531606674,
+            0.007894711568951607,
+            0.031013205647468567,
+            0.0419866107404232,
+            -0.02086401730775833,
+            -0.025130555033683777,
+            0.054301392287015915,
+            -0.011361273005604744,
+            0.01543387770652771,
+            0.028815291821956635,
+            0.014496532268822193,
+            0.012945064343512058,
+            0.032984863966703415,
+            0.010642102919518948,
+            0.007413917686790228,
+            -0.023255864158272743,
+            0.04308556765317917,
+            -0.01871458627283573,
+            -0.028233490884304047,
+            -0.019958993420004845,
+            0.012007718905806541,
+            0.006981607526540756,
+            -0.02600325644016266,
+            -0.029558703303337097,
+            -0.027183018624782562,
+            -0.0005520035629160702,
+            -0.008613881655037403,
+            -0.016379304230213165,
+            0.020702404901385307,
+            -0.0050059095956385136,
+            0.0006893731188029051,
+            -0.036847371608018875,
+            -0.004957426339387894,
+            -0.0010787553619593382,
+            -0.006965446285903454,
+            0.0363302156329155,
+            0.0006999789038673043,
+            0.003561508608981967,
+            -0.04024121165275574,
+            -0.022738708183169365,
+            -0.004181691911071539,
+            0.022318517789244652,
+            0.056079115718603134,
+            -0.004957426339387894,
+            0.039627086371183395,
+            -0.014528854750096798,
+            0.012427907437086105,
+            -0.025082072243094444,
+            0.01738937385380268,
+            0.0272638238966465,
+            0.00945426058024168,
+            0.006266477517783642,
+            0.015046010725200176,
+            0.014924801886081696,
+            -0.019781220704317093,
+            0.02669818513095379,
+            -0.0033271722495555878,
+            0.01675909012556076,
+            0.0015726798446848989,
+            0.01154712587594986,
+            -0.02574467845261097,
+            -0.01490056049078703,
+            -0.022544773295521736,
+            -0.024629559367895126,
+            0.054301392287015915,
+            -0.02600325644016266,
+            0.013745039701461792,
+            -0.0016757070552557707,
+            0.018504492938518524,
+            -0.03180510178208351,
+            -0.01599951647222042,
+            -0.04043514281511307,
+            -0.04473400488495827,
+            -0.04686727374792099,
+            -0.019765060395002365,
+            -0.00938961561769247,
+            -5.8962868934031576e-05,
+            -0.03995031118392944,
+            0.021607428789138794,
+            -0.05627305060625076,
+            -0.008929023519158363,
+            0.03772007301449776,
+            -0.022415485233068466,
+            0.0011908732121810317,
+            -0.021365011110901833,
+            0.009421938098967075,
+            -0.04900054261088371,
+            0.002274678787216544,
+            0.027344629168510437,
+            0.0555296391248703,
+            -0.0013706657337024808,
+            -0.022981123998761177,
+            0.004375625401735306,
+            -0.013405656442046165,
+            0.0018797412049025297,
+            0.0010565337724983692,
+            0.009114876389503479,
+            0.02815268561244011,
+            -0.017114635556936264,
+            0.053105469793081284,
+            0.012177410535514355,
+            0.030463727191090584,
+            0.026455767452716827,
+            -0.007078574039041996,
+            -0.06548488885164261,
+            -0.0006020020227879286,
+            0.021720556542277336,
+            0.010165349580347538,
+            0.0018544894410297275,
+            -0.004945305176079273,
+            0.004094826057553291,
+            0.0049049025401473045,
+            -0.0015716697089374065,
+            0.01625809445977211,
+            0.012015799060463905,
+            0.00763613311573863,
+            -0.004678646568208933,
+            0.013300608843564987,
+            0.015134897083044052,
+            -0.035457514226436615,
+            0.022221552208065987,
+            0.011805704794824123,
+            -0.048612672835588455,
+            -0.008727009408175945,
+            -0.02155894599854946,
+            0.02181752398610115,
+            -0.003492823801934719,
+            -0.016338901594281197,
+            -0.004569558892399073,
+            0.033938370645046234,
+            0.01973273791372776,
+            -0.0040645236149430275,
+            -0.014650062657892704,
+            0.013332931324839592,
+            0.04250376671552658,
+            -0.022997286170721054,
+            -0.030560694634914398,
+            0.011741059832274914,
+            -0.00397159717977047,
+            -0.0015454079257324338,
+            0.0017241904279217124,
+            -0.016613639891147614,
+            0.04470168054103851,
+            -0.009316890500485897,
+            0.035134293138980865,
+            0.004115027375519276,
+            0.014633901417255402,
+            -0.019021648913621902,
+            0.009179521352052689,
+            -0.018068142235279083,
+            0.02689211815595627,
+            -0.0157813411206007,
+            0.044119879603385925,
+            0.026972923427820206,
+            0.012646082788705826,
+            0.024694204330444336,
+            -0.035134293138980865,
+            -0.012702646665275097,
+            0.014221793040633202,
+            0.01884387619793415,
+            -0.010900680907070637,
+            -0.010213833302259445,
+            -0.014609660021960735,
+            -0.008751250803470612,
+            0.03772007301449776,
+            -0.02162358909845352,
+            -0.024112403392791748,
+            0.001966607291251421,
+            -0.0013797563733533025,
+            0.0006626062677241862,
+            -0.01588638871908188,
+            -0.016468189656734467,
+            0.017631791532039642,
+            0.050519686192274094,
+            -0.02244780771434307,
+            0.022415485233068466,
+            0.040144242346286774,
+            -0.0048038954846560955,
+            -0.020670082420110703,
+            -0.025728516280651093,
+            -0.009098715148866177,
+            0.00039493758231401443,
+            -0.019247904419898987,
+            0.016379304230213165,
+            -0.008565397933125496,
+            -0.00945426058024168,
+            -0.04612386226654053,
+            -0.0010247165337204933,
+            0.02516287751495838,
+            0.0033312125597149134,
+            0.015579327940940857,
+            -0.016985345631837845,
+            -0.005684677045792341,
+            -0.004876620601862669,
+            0.024257853627204895,
+            -0.016920700669288635,
+            0.007672495674341917,
+            0.0024605318903923035,
+            0.0003421613946557045,
+            0.0009858288103714585,
+            -0.003064553951844573,
+            0.00032120241667144,
+            0.007405837066471577,
+            0.027086051180958748,
+            -0.008355303667485714,
+            -0.023449797183275223,
+            0.007983597926795483,
+            0.02079937234520912,
+            -0.02598709426820278,
+            0.004917023237794638,
+            0.049873244017362595,
+            0.023869987577199936,
+            -0.013518784195184708,
+            0.005963456351310015,
+            -0.0010444129584357142,
+            0.017049990594387054,
+            0.0040604835376143456,
+            0.01505409087985754,
+            0.03494035825133324,
+            0.0016686365706846118,
+            -0.03193439170718193,
+            -0.00795531552284956,
+            0.0017322709318250418,
+            0.010569377802312374,
+            0.016662122681736946,
+            0.0008716908632777631,
+            -0.04573599249124527,
+            0.0013757160631939769,
+            0.005797804798930883,
+            0.010884519666433334,
+            -0.014625821262598038,
+            -0.02214074693620205,
+            0.022948801517486572,
+            0.026213349774479866,
+            0.001036332338117063,
+            0.0035170656628906727,
+            0.005914973095059395,
+            0.017066150903701782,
+            0.02307809144258499,
+            0.022092262282967567,
+            -0.014367243275046349,
+            0.019183259457349777,
+            -0.004755412228405476,
+            -0.012355182319879532,
+            -0.001959536923095584,
+            -0.020459989085793495,
+            0.021348850801587105,
+            0.044378459453582764,
+            0.02094482257962227,
+            0.018423685804009438,
+            0.017825724557042122,
+            0.010270397178828716,
+            0.026229511946439743,
+            -0.02732846885919571,
+            -0.03600699454545975,
+            0.0025938611943274736,
+            -0.0003156470484100282,
+            -0.025502260774374008,
+            -0.002603961853310466,
+            -0.018439847975969315,
+            -0.039756376296281815,
+            0.009623952209949493,
+            -0.0026160827837884426,
+            0.010472411289811134,
+            0.03442320227622986,
+            -0.01077947299927473,
+            -0.03224145248532295,
+            -0.0004121087840758264,
+            0.033421214669942856,
+            0.01797117479145527,
+            0.039497796446084976,
+            -0.01877923123538494,
+            0.0062462761998176575,
+            -0.0014656123239547014,
+            0.007991678081452847,
+            0.01372887846082449,
+            -0.007676536217331886,
+            -0.013389495201408863,
+            0.030124343931674957,
+            0.023482119664549828,
+            0.04777229577302933,
+            -0.006484652869403362,
+            -0.043473437428474426,
+            -0.012233974412083626,
+            -0.01531266886740923,
+            0.015062171965837479,
+            0.008605800569057465,
+            0.0050947959534823895,
+            0.0008646203787066042,
+            0.008880539797246456,
+            -0.007583609316498041,
+            0.006254356820136309,
+            0.007098775822669268,
+            -0.01879539154469967,
+            0.03332424536347389,
+            -0.008096725679934025,
+            0.0302374716848135,
+            -0.019942833110690117,
+            -0.02548610046505928,
+            -0.01864994317293167,
+            0.0024241693317890167,
+            0.0023150816559791565,
+            -0.026229511946439743,
+            0.03049604967236519,
+            0.03471410274505615,
+            -0.0018666103715077043,
+            0.008331061340868473,
+            -0.010432008653879166,
+            0.003490803763270378,
+            0.002375685842707753,
+            -0.006213953718543053,
+            0.007935114204883575,
+            0.0003030211664736271,
+            -0.026035577058792114,
+            0.035198938101530075,
+            -0.03542519360780716,
+            0.0018777211662381887,
+            -0.020718567073345184,
+            0.021979134529829025,
+            0.008452270179986954,
+            0.008387625217437744,
+            0.011676414869725704,
+            -0.023304346948862076,
+            -0.0030059698037803173,
+            0.009777482599020004,
+            0.015603569336235523,
+            0.008710848167538643,
+            -0.015749020501971245,
+            0.012476391158998013,
+            0.0019060031045228243,
+            0.020960982888936996,
+            0.03030211664736271,
+            0.00044241087744012475,
+            0.008250256069004536,
+            -0.01638738438487053,
+            -0.019215581938624382,
+            -0.0061533497646451,
+            0.010310799814760685,
+            0.030512209981679916,
+            -0.012169329449534416,
+            0.04192196577787399,
+            0.037849362939596176,
+            -0.003840288147330284,
+            -0.006965446285903454,
+            0.010254235938191414,
+            -0.0551740936934948,
+            0.05071362107992172,
+            -0.003490803763270378,
+            -0.0007388665690086782,
+            0.020088283345103264,
+            -0.0442814938724041,
+            -0.023175058886408806,
+            0.005098836030811071,
+            0.010827955789864063,
+            0.01732472889125347,
+            0.009842127561569214,
+            0.002264578128233552,
+            0.014504612423479557,
+            -0.03136875107884407,
+            -0.039562441408634186,
+            0.02763552963733673,
+            0.00970475748181343,
+            0.004123107995837927,
+            0.012815774418413639,
+            0.02018524892628193,
+            -0.0017716636648401618,
+            0.008840137161314487,
+            0.01309051364660263,
+            -0.0021150875836610794,
+            0.01732472889125347,
+            0.02600325644016266,
+            -0.010706747882068157,
+            0.026035577058792114,
+            0.011215822771191597,
+            0.03781704232096672,
+            -0.0050543928518891335,
+            -0.00488470122218132,
+            0.005474582314491272,
+            -0.021785201504826546,
+            0.03195055201649666,
+            0.0031392991077154875,
+            -0.03846348449587822,
+            0.01682373508810997,
+            0.022367002442479134,
+            -0.017793402075767517,
+            0.021284205839037895,
+            0.0014858137583360076,
+            0.020912500098347664,
+            0.026956763118505478,
+            0.0040039196610450745,
+            0.0020585237070918083,
+            0.028896097093820572,
+            0.01886003650724888,
+            -0.008387625217437744,
+            0.050261110067367554,
+            0.010674425400793552,
+            -0.007664415054023266,
+            0.01511873584240675,
+            0.0013605649583041668,
+            0.02624567225575447,
+            -0.047028884291648865,
+            8.831804007058963e-05,
+            -0.012710727751255035,
+            -6.256124470382929e-05,
+            0.011821865104138851,
+            -0.01173297967761755,
+            0.003684737253934145,
+            -0.033744435757398605,
+            0.023643730208277702,
+            0.005102876108139753,
+            0.022560935467481613,
+            -0.008428028784692287,
+            0.006916963029652834,
+            0.016662122681736946,
+            -0.0009782533161342144,
+            0.007042211946099997,
+            0.009777482599020004,
+            -0.006193752400577068,
+            0.017486341297626495,
+            -0.008395706303417683,
+            0.016807572916150093,
+            0.046641018241643906,
+            -0.01636314205825329,
+            -0.0038039255887269974,
+            0.039691731333732605,
+            -0.009114876389503479,
+            -0.025195199996232986,
+            0.020459989085793495,
+            -0.007632093038409948,
+            -0.0006838177214376628,
+            0.04230983555316925,
+            -0.043796658515930176,
+            -0.025825483724474907,
+            0.004016040358692408,
+            -0.023562924936413765,
+            -0.019635770469903946,
+            0.009365374222397804,
+            0.016314659267663956,
+            0.003751402022317052,
+            -0.04421684890985489,
+            0.005947295110672712,
+            -0.000348221801687032,
+            0.01258143875747919,
+            -0.00016312638763338327,
+            -0.01688838005065918,
+            -0.036394860595464706,
+            0.011757221072912216,
+            -0.004436229821294546,
+            -0.025890128687024117,
+            4.217044624965638e-05,
+            -0.010294638574123383,
+            -0.05242669954895973,
+            -0.026665862649679184,
+            0.040758367627859116,
+            0.0012009738711640239,
+            0.033485859632492065,
+            -5.463850175146945e-05,
+            0.03262931853532791,
+            -0.00822601467370987,
+            -0.02075088955461979,
+            0.01517529971897602,
+            -0.0073371524922549725,
+            0.016330819576978683,
+            0.010019899345934391,
+            0.001137339393608272,
+            -0.00023951297043822706,
+            0.03490803763270378,
+            -0.033421214669942856,
+            0.004030181560665369,
+            0.0038726103957742453,
+            -0.012112765572965145,
+            -0.045186515897512436,
+            0.010213833302259445,
+            -0.00844419002532959,
+            -0.009955255314707756,
+            0.022059939801692963,
+            -0.011821865104138851,
+            0.001617122907191515,
+            0.025276005268096924,
+            0.006484652869403362,
+            -0.006488692946732044,
+            -0.0363302156329155,
+            -0.013462220318615437,
+            0.0090098287910223,
+            -0.008508834056556225,
+            -0.0008686606306582689,
+            -0.010318880900740623,
+            -0.04176035523414612,
+            0.014884399250149727,
+            -0.025502260774374008,
+            -0.006116987206041813,
+            0.014811674132943153,
+            0.019878188148140907,
+            -0.010092624463140965,
+            -0.04205125570297241,
+            -0.007038171403110027,
+            -0.03290405869483948,
+            -0.006929083727300167,
+            0.003482723142951727,
+            -0.01044008880853653,
+            0.008904782123863697,
+            0.016112644225358963,
+            -0.0186661034822464,
+            0.010165349580347538,
+            -0.02663354016840458,
+            0.021203400567173958,
+            -0.014254115521907806,
+            0.01593487337231636,
+            0.0021130675449967384,
+            0.003232225775718689,
+            -0.021962974220514297,
+            0.025954771786928177,
+            -0.0299142487347126,
+            -0.004816016182303429,
+            -0.0315626822412014,
+            -0.02624567225575447,
+            -0.030140504240989685,
+            0.006848277989774942,
+            0.0010555237531661987,
+            -0.04091997817158699,
+            -0.005284688901156187,
+            0.015320749953389168,
+            0.0020373123697936535,
+            -0.028928419575095177,
+            -0.020136766135692596,
+            0.0009827986359596252,
+            0.007349273189902306,
+            0.0017928752349689603,
+            -0.031724296510219574,
+            -0.008678525686264038,
+            -0.032483868300914764,
+            0.020201411098241806,
+            -0.0017706536455079913,
+            0.027344629168510437,
+            0.0011262285988777876,
+            0.003840288147330284,
+            -0.02029837667942047,
+            0.002234275918453932,
+            0.0036907976027578115,
+            0.022302357479929924,
+            -0.03484339267015457,
+            0.016548994928598404,
+            0.035392872989177704,
+            0.022221552208065987,
+            -0.049356088042259216,
+            0.0016342941671609879,
+            0.01385008729994297,
+            -0.011021889746189117,
+            0.01871458627283573,
+            -0.005733160302042961,
+            0.025082072243094444,
+            -0.01277537178248167,
+            0.0005641243769787252,
+            0.005684677045792341,
+            0.006327081937342882,
+            -0.0022564975079149008,
+            0.004335222765803337,
+            -0.024435626342892647,
+            0.02320738136768341,
+            0.02687595598399639,
+            0.007551287300884724,
+            0.010634022764861584,
+            -0.0031069768592715263,
+            -0.012387504801154137,
+            0.011805704794824123,
+            -0.026488089933991432,
+            -0.0005691747646778822,
+            0.011021889746189117,
+            -0.022544773295521736,
+            -0.006815955974161625,
+            0.03959476575255394,
+            0.024952782317996025,
+            -0.05313779041171074,
+            0.017680274322628975,
+            0.018423685804009438,
+            -0.005058433394879103,
+            0.01747017912566662,
+            -0.0403381772339344,
+            -0.00844419002532959,
+            -0.0254376158118248,
+            0.0005666495417244732,
+            0.016807572916150093,
+            0.011005728505551815,
+            -0.015377313829958439,
+            0.024936622008681297,
+            0.0003035262052435428,
+            0.03201519697904587,
+            -0.002511035418137908,
+            -0.020411504432559013,
+            -0.02650425024330616,
+            -0.020605439320206642,
+            -0.00743007892742753,
+            0.009858288802206516,
+            -0.0005403877585195005,
+            -0.03161116689443588,
+            -0.03746149688959122,
+            0.007385635748505592,
+            0.00307667488232255,
+            -0.04790158569812775,
+            -0.013171319849789143,
+            -0.02845974639058113,
+            -0.03060917742550373,
+            -0.009163360111415386,
+            -0.009058312512934208,
+            -0.03775239735841751,
+            0.0006009919452480972,
+            0.003133238758891821,
+            0.021672073751688004,
+            -0.010868359357118607,
+            0.013720798306167126,
+            -0.02333666943013668,
+            -0.007838147692382336,
+            0.0016706566093489528,
+            0.03985334187746048,
+            -0.009615871123969555,
+            -0.035069648176431656,
+            0.011021889746189117,
+            0.019377192482352257,
+            -0.015902550891041756,
+            0.012217813171446323,
+            0.005082674790173769,
+            -0.006678586360067129,
+            0.017793402075767517,
+            -0.013955134898424149,
+            -0.03762310743331909,
+            0.004549357574433088,
+            -0.013147078454494476,
+            -0.01442380715161562,
+            -0.017179278656840324,
+            0.010367363691329956,
+            0.02214074693620205,
+            -0.021930651739239693,
+            0.008880539797246456,
+            0.04486329108476639,
+            -0.02227003499865532,
+            -0.029364770278334618,
+            0.01053705532103777,
+            -0.018375203013420105,
+            0.01401169877499342,
+            -0.004214014392346144,
+            0.03062533773481846,
+            -0.009163360111415386,
+            0.01053705532103777,
+            -0.015425797551870346,
+            0.010092624463140965,
+            -0.03384140133857727,
+            -0.019005486741662025,
+            -0.0034079779870808125,
+            -0.007672495674341917,
+            -0.03778471797704697,
+            -0.03616860508918762,
+            0.022916480898857117,
+            0.019150936976075172,
+            -0.009906771592795849,
+            0.0058947717770934105,
+            -0.010836036875844002,
+            9.204267553286627e-05,
+            0.026520412415266037,
+            0.016613639891147614,
+            -0.008177530951797962,
+            0.0012292558094486594,
+            -0.0047594523057341576,
+            0.0065088942646980286,
+            0.019312549382448196,
+            -0.06493541598320007,
+            -0.029203159734606743,
+            0.02504974976181984,
+            0.01789036951959133,
+            0.012330940924584866,
+            0.015207622200250626,
+            0.02535681053996086,
+            -0.02383766509592533,
+            -0.020912500098347664,
+            0.037849362939596176,
+            -0.003381716087460518,
+            0.01024615578353405,
+            -0.007947235368192196,
+            0.0019302447326481342,
+            0.00030453625367954373,
+            0.002240336500108242,
+            -0.0010797653812915087,
+            0.005757402162998915,
+            0.036394860595464706,
+            0.02239932492375374,
+            0.013122836127877235,
+            0.015078333206474781,
+            -0.0013625851133838296,
+            -0.004973587114363909,
+            -0.007951275445520878,
+            -0.0037796839606016874,
+            0.025405293330550194,
+            -0.017292408272624016,
+            0.0017150997882708907,
+            -0.026277994737029076,
+            0.02492045983672142,
+            0.025130555033683777,
+            0.01708231307566166,
+            -0.005369534716010094,
+            -0.007034131325781345,
+            0.013421816751360893,
+            0.0034766627941280603,
+            -0.009276487864553928,
+            0.016807572916150093,
+            0.0068442379124462605,
+            -0.020379183813929558,
+            -0.011789543554186821,
+            0.008205813355743885,
+            -0.007207863498479128,
+            -0.022948801517486572,
+            0.031158655881881714,
+            0.012831935659050941,
+            0.012662244029343128,
+            0.004795814864337444,
+            -0.0005444280104711652,
+            0.011522884480655193,
+            0.03484339267015457,
+            -0.036588795483112335,
+            -0.01771259680390358,
+            0.005434179678559303,
+            -0.02138117328286171,
+            0.00929264910519123,
+            -0.014908640645444393,
+            0.006193752400577068,
+            0.015983356162905693,
+            0.07408261299133301,
+            -0.027247661724686623,
+            -4.744175021187402e-05,
+            -0.016274256631731987,
+            -0.017098473384976387,
+            0.018326720222830772,
+            -0.004230175167322159,
+            0.0051109567284584045,
+            0.02765168994665146,
+            0.008395706303417683,
+            0.03636253997683525,
+            -0.007013929542154074,
+            -0.005373575259000063,
+            0.023385152220726013,
+            -0.0018645902164280415,
+            -0.010852198116481304,
+            -0.006723029538989067,
+            0.001137339393608272,
+            0.012815774418413639,
+            0.00790279172360897,
+            0.03258083388209343,
+            0.017567146569490433,
+            0.0012019838904961944,
+            -0.00039493758231401443,
+            -0.0044564311392605305,
+            0.028556713834404945,
+            0.012848096899688244,
+            0.0017423717072233558,
+            -0.01953880488872528,
+            -0.003040312323719263,
+            0.027473919093608856,
+            -0.021672073751688004,
+            0.009922932833433151,
+            0.015328830108046532,
+            0.005034191533923149,
+            0.04809551686048508,
+            -0.03351818025112152,
+            -0.008298739790916443,
+            0.012064282782375813,
+            0.0018625700613483787,
+            -0.008068443275988102,
+            0.01480359397828579,
+            0.006581619381904602,
+            -0.01056129764765501,
+            -0.00926840677857399,
+            0.012613760307431221,
+            0.013575348071753979,
+            -0.0063917264342308044,
+            -0.00901790987700224,
+            0.0066543444991111755,
+            -0.010771391913294792,
+            0.019958993420004845,
+            -0.00035781748010776937,
+            0.008254296146333218,
+            0.004193812608718872,
+            0.008718929253518581,
+            -0.0007681586430408061,
+            -0.04111390933394432,
+            -0.00789067056030035,
+            0.016177289187908173,
+            -0.003333232831209898,
+            0.021348850801587105,
+            -0.001368645578622818,
+            -0.008565397933125496,
+            -0.014876319095492363,
+            -0.004876620601862669,
+            -0.0036019114777445793,
+            -0.00519984308630228,
+            0.0005641243769787252,
+            -0.00471500912681222,
+            -0.01738937385380268,
+            0.011126937344670296,
+            -0.028766809031367302,
+            -0.007680576294660568,
+            -0.0033473738003522158,
+            -0.014246034435927868,
+            -0.004525116179138422,
+            -0.007094735279679298,
+            0.005102876108139753,
+            -0.007595730479806662,
+            0.019134776666760445,
+            0.0050543928518891335,
+            -0.029057709500193596,
+            -0.008710848167538643,
+            -0.027296146377921104,
+            0.017631791532039642,
+            -0.027215341106057167,
+            0.01192691270262003,
+            0.025340650230646133,
+            0.002365585183724761,
+            -0.0012100645108148456,
+            0.004131188616156578,
+            -0.029817281290888786,
+            0.0036685762461274862,
+            0.010609780438244343,
+            -0.009728998877108097,
+            0.00901790987700224,
+            0.023627569898962975,
+            -0.01816510781645775,
+            -0.025841644033789635,
+            -0.03207983821630478,
+            0.01565205305814743,
+            0.011110776104032993,
+            -0.014157148078083992,
+            -0.011692576110363007,
+            -0.0012080443557351828,
+            0.004929144401103258,
+            -0.01378544233739376,
+            -0.00010245903104078025,
+            -0.00785834901034832,
+            -0.010326961055397987,
+            -0.008088644593954086,
+            -0.008985587395727634,
+            -0.002513055456802249,
+            -0.0038907916750758886,
+            0.008355303667485714,
+            0.006933124270290136,
+            -0.03494035825133324,
+            -0.025841644033789635,
+            0.012856177985668182,
+            0.007288668770343065,
+            -0.01331677008420229,
+            0.009567388333380222,
+            0.014989446848630905,
+            0.004965506494045258,
+            -0.002377706114202738,
+            0.009890610352158546,
+            -0.018456008285284042,
+            0.0127188079059124,
+            0.0046665258705616,
+            0.004084725398570299,
+            0.016920700669288635,
+            0.015151058323681355,
+            0.012339022010564804,
+            0.01293698325753212,
+            -0.03290405869483948,
+            0.0007151299505494535,
+            0.027991075068712234,
+            0.048580352216959,
+            0.01636314205825329,
+            -0.022754868492484093,
+            0.021962974220514297,
+            0.04625314846634865,
+            0.02024989388883114,
+            -0.00060301210032776,
+            -0.016985345631837845,
+            0.0050543928518891335,
+            0.026261834427714348,
+            0.010351202450692654,
+            0.00418977253139019,
+            -0.006415967829525471,
+            -0.013551106676459312,
+            0.014173309318721294,
+            0.009462340734899044,
+            0.00678363349288702,
+            -0.037914007902145386,
+            0.020023638382554054,
+            -0.023579087108373642,
+            0.020653922110795975,
+            0.0027756737545132637,
+            -0.012427907437086105,
+            0.016314659267663956,
+            0.022156907245516777,
+            -0.0048725800588727,
+            0.020960982888936996,
+            0.0008767411927692592,
+            0.011296628974378109,
+            0.00837954506278038,
+            -0.011934993788599968,
+            0.003896852023899555,
+            -0.005975577048957348,
+            0.003793824929744005,
+            0.006189712323248386,
+            0.02302960865199566,
+            0.00397159717977047,
+            0.0186661034822464,
+            0.017696436494588852,
+            0.0010045150993391871,
+            0.008104805834591389,
+            -0.019748898223042488,
+            -0.00022486694797407836,
+            -0.009696677327156067,
+            0.007349273189902306,
+            0.009462340734899044,
+            0.0072927093133330345,
+            0.02479117177426815,
+            -0.012597599998116493,
+            -0.010585539042949677,
+            -0.01816510781645775,
+            0.014561176300048828,
+            -0.014149067923426628,
+            0.01634698174893856,
+            0.017761079594492912,
+            0.0012363262940198183,
+            0.008565397933125496,
+            0.02446794882416725,
+            0.002816076623275876,
+            -0.01656515710055828,
+            0.011078453622758389,
+            -0.023740697652101517,
+            -0.017308568581938744,
+            0.013478381559252739,
+            -0.01315515860915184,
+            -0.010270397178828716,
+            0.020524634048342705,
+            -0.015967195853590965,
+            0.0017565126763656735,
+            0.011151178739964962,
+            0.03062533773481846,
+            -0.017373213544487953,
+            0.022318517789244652,
+            -0.01075523067265749,
+            -0.023110413923859596,
+            -0.002626183442771435,
+            0.016662122681736946,
+            0.0269406009465456,
+            1.085036728909472e-06,
+            -0.005729120224714279,
+            -0.005207923706620932,
+            0.02309425175189972,
+            -0.008573479019105434,
+            0.022867996245622635,
+            0.007034131325781345,
+            0.012419827282428741,
+            0.00020618064445443451,
+            -0.018003497272729874,
+            0.017437858507037163,
+            0.0020544833969324827,
+            0.02214074693620205,
+            0.014407645910978317,
+            -0.015749020501971245,
+            0.02904154732823372,
+            -0.0030241510830819607,
+            0.028556713834404945,
+            0.0007206853479146957,
+            -0.009583549574017525,
+            0.02845974639058113,
+            -0.0016807573847472668,
+            -0.0004017555620521307,
+            0.006165470462292433,
+            0.01505409087985754,
+            0.016443947330117226,
+            0.00392311392351985,
+            0.020605439320206642,
+            0.016306579113006592,
+            -0.00015100554446689785,
+            0.014318759553134441,
+            -0.00879165343940258,
+            -0.022819513455033302,
+            0.022043779492378235,
+            -0.005034191533923149,
+            -0.01745401881635189,
+            0.01734089106321335,
+            -0.007696737535297871,
+            -0.004058463498950005,
+            -0.0007075543981045485,
+            0.023320509120821953,
+            0.016209611669182777,
+            -0.007575529161840677,
+            0.003365555079653859,
+            -0.004282698966562748,
+            -0.001307031256146729,
+            0.02789410762488842,
+            0.008613881655037403,
+            -0.0034968641120940447,
+            0.01511873584240675,
+            0.005236205644905567,
+            0.01024615578353405,
+            -0.00025580034707672894,
+            -0.007466441486030817,
+            -0.01955496519804001,
+            -0.0035796898882836103,
+            0.004763492848724127,
+            0.005494783632457256,
+            0.0005767502589151263,
+            -0.0060563827864825726,
+            -0.020734727382659912,
+            -0.04476632550358772,
+            -0.00019128210260532796,
+            0.0014726828085258603,
+            0.025922449305653572,
+            0.04169571027159691,
+            -0.005902852397412062,
+            -0.004747331608086824,
+            -0.012452149763703346,
+            0.0014009678270667791,
+            -0.008533075451850891,
+            -0.019700415432453156,
+            -0.006064463406801224,
+            -0.013817764818668365,
+            -0.01916709914803505,
+            0.01615304872393608,
+            -0.012654163874685764,
+            0.006048302166163921,
+            0.002583760302513838,
+            0.019215581938624382,
+            -0.016742929816246033,
+            0.02744159661233425,
+            -0.01634698174893856,
+            0.004747331608086824,
+            0.001796915428712964,
+            -0.002872640499845147,
+            -0.03322727978229523,
+            0.02068624459207058,
+            -0.003304950660094619,
+            -0.036685761064291,
+            -0.0037716033402830362,
+            -0.007219984196126461,
+            -0.009955255314707756,
+            0.03429391607642174,
+            0.0014171289512887597,
+            -0.007805825211107731,
+            -0.012096604332327843,
+            0.01119158137589693,
+            0.011328951455652714,
+            0.008197732269763947,
+            -0.006020020227879286,
+            -0.015029849484562874,
+            0.007405837066471577,
+            0.03267780318856239,
+            -0.019765060395002365,
+            -0.015207622200250626,
+            -0.0014858137583360076,
+            0.011692576110363007,
+            0.002739311195909977,
+            0.005927093792706728,
+            -0.030156666412949562,
+            -0.014302598312497139,
+            -0.009607790969312191,
+            -0.01135319285094738,
+            -0.022674063220620155,
+            -0.007979556918144226,
+            0.00907447375357151,
+            -0.02125188335776329,
+            0.02138117328286171,
+            0.029881926253437996,
+            -0.027991075068712234,
+            0.0340353362262249,
+            -0.011345111764967442,
+            0.009599709883332253,
+            -0.01034312229603529,
+            0.0007449270342476666,
+            -0.011102695018053055,
+            0.0007964406395331025,
+            0.011304709129035473,
+            -0.03581305965781212,
+            -0.04237448051571846,
+            -0.008945184759795666,
+            0.0010959265055134892,
+            -0.01795501448214054,
+            -0.006925043649971485,
+            0.03959476575255394,
+            0.0100845443084836,
+            -0.0031837422866374254,
+            -0.016225773841142654,
+            0.01864994317293167,
+            0.02233467996120453,
+            0.03262931853532791,
+            0.01555508654564619,
+            0.00450895493850112,
+            0.0008186621707864106,
+            0.011789543554186821,
+            -0.027490079402923584,
+            0.014795512892305851,
+            -0.0013696555979549885,
+            0.02676282823085785,
+            -0.0015746999997645617,
+            -0.0003719584783539176,
+            0.019748898223042488,
+            -0.024257853627204895,
+            -0.0024847735185176134,
+            0.00732099125161767,
+            -0.014722787775099277,
+            0.004965506494045258,
+            0.019021648913621902,
+            0.007757341489195824,
+            -0.036588795483112335,
+            0.0041655306704342365,
+            -0.010771391913294792,
+            0.01986202597618103,
+            0.00917144026607275,
+            0.010423927567899227,
+            -0.0031291984487324953,
+            -0.011102695018053055,
+            -0.011910751461982727,
+            0.007817945443093777,
+            -0.0025979015044867992,
+            -0.008003799244761467,
+            0.03041524440050125,
+            -0.003412018297240138,
+            -0.0032382861245423555,
+            0.00837954506278038,
+            0.006032141391187906,
+            0.006036181468516588,
+            0.0085977204144001,
+            -0.009163360111415386,
+            -0.024435626342892647,
+            -0.01490056049078703,
+            -0.018746908754110336,
+            0.005353373941034079,
+            0.010262317024171352,
+            -0.003717059502378106,
+            -0.02416088618338108,
+            0.005114997271448374,
+            -0.007203822955489159,
+            0.036782726645469666,
+            -0.008347222581505775,
+            -0.035457514226436615,
+            -0.02377302013337612,
+            -0.04667333886027336,
+            0.009211842902004719,
+            6.436042895074934e-05,
+            -0.021979134529829025,
+            -0.0039614965207874775,
+            -0.011490561999380589,
+            -0.028007235378026962,
+            0.012056201696395874,
+            0.019651932641863823,
+            -0.026019416749477386,
+            0.021461978554725647,
+            0.00763613311573863,
+            -0.015102574601769447,
+            0.00567659642547369,
+            -0.008743170648813248,
+            -0.0269406009465456,
+            0.002250437159091234,
+            -0.006278598215430975,
+            0.017114635556936264,
+            -0.01173297967761755,
+            -0.0001382534101139754,
+            -0.020912500098347664,
+            0.01860145851969719,
+            0.021219560876488686,
+            -0.011070373468101025,
+            0.0014433907344937325,
+            -0.0037210998125374317,
+            -0.02568003349006176,
+            0.005555388052016497,
+            0.0007545226835645735,
+            -0.004614002071321011,
+            -0.0004100886289961636,
+            0.006791714113205671,
+            -0.009122957475483418,
+            0.00086159014608711,
+            -0.0051513598300516605,
+            0.008274497464299202,
+            0.0017393414163962007,
+            -0.01034312229603529,
+            -0.002296900376677513,
+            -0.003636253997683525,
+            0.0024484109599143267,
+            0.019134776666760445,
+            -0.015183379873633385,
+            0.007777543272823095,
+            -0.03157884627580643,
+            0.006832117214798927,
+            0.009623952209949493,
+            -0.0006408897461369634,
+            -0.02813652530312538,
+            0.0010327971540391445,
+            -0.0007090694853104651,
+            0.005531146191060543,
+            -0.026342639699578285,
+            0.029267802834510803,
+            0.0216397512704134,
+            0.011458240449428558,
+            0.015862148255109787,
+            0.001985798589885235,
+            -0.013712717220187187,
+            -0.016678284853696823,
+            0.003785744309425354,
+            -0.016936862841248512,
+            -0.009284568019211292,
+            0.00036589804221875966,
+            0.027490079402923584,
+            -0.024257853627204895,
+            -0.009720918722450733,
+            -0.00028054710128344595,
+            0.0005009949672967196,
+            -0.005680636502802372,
+            -0.010019899345934391,
+            0.004836217500269413,
+            0.013971295207738876,
+            0.017179278656840324,
+            -0.010698666796088219,
+            0.07395332306623459,
+            0.002173671731725335,
+            -0.006787674035876989,
+            0.0013817765284329653,
+            -0.011918832547962666,
+            -0.019910510629415512,
+            0.02086401730775833,
+            -0.022124584764242172,
+            0.013736959546804428,
+            0.007288668770343065,
+            -0.01567629538476467,
+            -0.00758764985948801,
+            -0.015991436317563057,
+            0.024904299527406693,
+            0.0017201501177623868,
+            0.0034867634531110525,
+            0.00689272116869688,
+            0.020734727382659912,
+            0.027296146377921104,
+            0.012274377048015594,
+            0.0041069467552006245,
+            -0.0027978953439742327,
+            -0.004840258043259382,
+            0.016726767644286156,
+            -0.01910245418548584,
+            0.002250437159091234,
+            0.005886691156774759,
+            0.022415485233068466,
+            -0.004969547037035227,
+            0.034261591732501984,
+            0.002882741391658783,
+            0.003143339417874813,
+            -0.006415967829525471,
+            0.003878670744597912,
+            0.020589277148246765,
+            -0.002850419143214822,
+            0.01664596237242222,
+            -0.006367484573274851,
+            -0.0033069709315896034,
+            0.0004800360184162855,
+            0.0010196662042289972,
+            0.002692847978323698,
+            0.0072967493906617165,
+            -0.019344870001077652,
+            -0.0002853449259418994,
+            0.00017448968719691038,
+            0.007264427375048399,
+            0.0029898087959736586,
+            0.0034180786460638046,
+            -0.004375625401735306,
+            0.00046437993296422064,
+            0.006613941863179207,
+            0.010003738105297089,
+            0.013074353337287903,
+            -0.0013696555979549885,
+            -0.005171561148017645,
+            0.006876560393720865,
+            -0.008169450797140598,
+            0.0013272326905280352,
+            0.025793161243200302,
+            0.0013504642993211746,
+            0.0062018330208957195,
+            -0.024871977046132088,
+            0.001361575094051659,
+            0.008686606772243977,
+            0.007595730479806662,
+            0.010674425400793552,
+            4.075003380421549e-05,
+            -0.0029372849967330694,
+            0.0012050141813233495,
+            -0.014019778929650784,
+            0.009632032364606857,
+            0.021785201504826546,
+            -0.0037675630301237106,
+            0.009042151272296906,
+            0.01050473377108574,
+            0.0023494239430874586,
+            -0.029687993228435516,
+            -0.015110654756426811,
+            0.013639992102980614,
+            0.001351474318653345,
+            0.008775493130087852,
+            4.4506232370622456e-05,
+            0.009543146006762981,
+            -0.02214074693620205,
+            -0.011175420135259628,
+            -0.0049089426174759865,
+            0.012807694263756275,
+            0.018229752779006958,
+            -0.021720556542277336,
+            0.013882409781217575,
+            0.006504854187369347,
+            0.004331182222813368,
+            -0.027279984205961227,
+            -0.005454380996525288,
+            0.013187481090426445,
+            -0.01128854788839817,
+            0.002939305268228054,
+            -0.01072290912270546,
+            0.005684677045792341,
+            0.003739281091839075,
+            0.02307809144258499,
+            -0.008840137161314487,
+            0.007761382032185793,
+            0.0037352407816797495,
+            -0.023045768961310387,
+            0.011078453622758389,
+            0.020411504432559013,
+            -0.00525236688554287,
+            -0.0013262225547805429,
+            -0.00577356293797493,
+            -0.00828257855027914,
+            0.015797503292560577,
+            -0.0021433697547763586,
+            -0.011660254560410976,
+            0.01658131740987301,
+            -0.006908882409334183,
+            -0.011369354091584682,
+            -0.022609418258070946,
+            -0.006177591159939766,
+            -0.006658385042101145,
+            -0.0026968882884830236,
+            0.009753241203725338,
+            -0.011442079208791256,
+            0.0034322196152061224,
+            -0.0017565126763656735,
+            -0.0078462278470397,
+            -0.004605921451002359,
+            -0.03358282521367073,
+            -0.018423685804009438,
+            -0.00811692699790001,
+            -0.02012060582637787,
+            0.002057513687759638,
+            0.003706958843395114,
+            -0.00716342031955719,
+            0.017922692000865936,
+            -0.010569377802312374,
+            -0.007575529161840677,
+            0.003414038335904479,
+            0.012912741862237453,
+            -0.00810884591192007,
+            -0.009890610352158546,
+            -0.019651932641863823,
+            0.0003696858184412122,
+            0.01591063104569912,
+            0.0183428805321455,
+            0.02537297084927559,
+            -0.016144966706633568,
+            0.0032948500011116266,
+            -0.024177048355340958,
+            0.008125007152557373,
+            -0.030382921919226646,
+            -0.012751130387187004,
+            -0.004840258043259382,
+            -0.0007378565496765077,
+            -0.008888620883226395,
+            -0.012040040455758572,
+            -0.04124319925904274,
+            -0.009785563684999943,
+            -0.015102574601769447,
+            -0.02322354167699814,
+            0.015134897083044052,
+            -0.005818006116896868,
+            0.017049990594387054,
+            -0.009567388333380222,
+            0.016856057569384575,
+            0.016904540359973907,
+            0.014553096145391464,
+            -0.007341192569583654,
+            0.00732099125161767,
+            -0.004387746565043926,
+            0.014407645910978317,
+            0.004123107995837927,
+            -0.0038928117137402296,
+            0.014973285607993603,
+            0.009995657950639725,
+            0.0050947959534823895,
+            0.00045478425454348326,
+            0.0017191400984302163,
+            0.010941083543002605,
+            0.0007924003293737769,
+            0.015635890886187553,
+            0.007975516840815544,
+            -0.0016837875591591,
+            0.034003015607595444,
+            -0.021785201504826546,
+            -0.010634022764861584,
+            0.010060302913188934,
+            0.01258143875747919,
+            -0.01873074844479561,
+            -0.0017837845953181386,
+            0.010512813925743103,
+            -0.0013201622059568763,
+            -0.030964722856879234,
+            0.013171319849789143,
+            0.011361273005604744,
+            0.0029999094549566507,
+            -0.005114997271448374,
+            -0.016791412606835365,
+            -0.005062473472207785,
+            0.007062413264065981,
+            0.020314538851380348,
+            -0.018245914950966835,
+            0.009939094074070454,
+            -0.007401796989142895,
+            -0.0061452691443264484,
+            -0.02689211815595627,
+            0.012532955035567284,
+            0.022948801517486572,
+            0.009195681661367416,
+            -0.059052761644124985,
+            -0.010375444777309895,
+            0.017858047038316727,
+            -0.0064927334897220135,
+            -0.03755846247076988,
+            -0.002294880338013172,
+            -0.01517529971897602,
+            -0.02307809144258499,
+            0.008767412044107914,
+            0.015732858330011368,
+            -0.006832117214798927,
+            0.015724778175354004,
+            0.006609901320189238,
+            -0.007482602261006832,
+            0.0003903922624886036,
+            -0.004973587114363909,
+            -0.03458481654524803,
+            -0.006678586360067129,
+            0.016548994928598404,
+            -0.03561912849545479,
+            -0.00721190357580781,
+            -0.00827045738697052,
+            -0.014553096145391464,
+            0.006052342709153891,
+            0.011248145252466202,
+            0.03972405195236206,
+            0.007405837066471577,
+            -0.01229861844331026,
+            -0.014003617689013481,
+            -0.014528854750096798,
+            -0.013389495201408863,
+            0.0017100494587793946,
+            -0.01176530122756958,
+            0.011240065097808838,
+            0.008331061340868473,
+            -0.009373454377055168,
+            0.023417474702000618,
+            -0.003403937676921487,
+            0.009559307247400284,
+            -3.494212796795182e-05,
+            -0.004710969049483538,
+            -0.005830127280205488,
+            -0.014722787775099277,
+            0.0008090665214695036,
+            0.003913013264536858,
+            -0.0025191158056259155,
+            -0.0005232165567576885,
+            0.0038221068680286407,
+            0.02706989087164402,
+            -0.010472411289811134,
+            -0.005442259833216667,
+            -0.010286558419466019,
+            -0.028362780809402466,
+            0.0003393836959730834,
+            0.02845974639058113,
+            0.030948560684919357,
+            -0.015684375539422035,
+            -0.01309051364660263,
+            -0.01789036951959133,
+            0.0067351502366364,
+            -0.0038604894652962685,
+            -0.0033392931800335646,
+            0.029397092759609222,
+            0.020977145060896873,
+            0.004137248732149601,
+            0.01732472889125347,
+            -0.009357293136417866,
+            0.006169511005282402,
+            0.028443586081266403,
+            -0.0013302628649398685,
+            0.009939094074070454,
+            0.010149188339710236,
+            0.013963215053081512,
+            -0.00986636895686388,
+            -0.000141536133014597,
+            0.005288729444146156,
+            -0.004646324552595615,
+            0.014084423892199993,
+            0.010666345246136189,
+            -0.008864378556609154,
+            -0.021203400567173958,
+            -0.005066513549536467,
+            -0.020896339789032936,
+            -0.0052483263425529,
+            0.006112946663051844,
+            -0.010836036875844002,
+            -0.001267638523131609,
+            -0.029397092759609222,
+            -0.004383706022053957,
+            0.002311041345819831,
+            -0.013211722485721111,
+            0.01027847733348608,
+            -0.011078453622758389,
+            0.0025251763872802258,
+            0.006290719378739595,
+            -0.02088017761707306,
+            -0.003717059502378106,
+            -0.007377555128186941,
+            0.003652415005490184,
+            0.005325092002749443,
+            0.00614930922165513,
+            -0.0016534854657948017,
+            0.005874569993466139,
+            0.020395344123244286,
+            0.0020787252578884363,
+            0.01486015785485506,
+            -0.02079937234520912,
+            -0.028508231043815613,
+            -0.029073869809508324,
+            0.015611650422215462,
+            -0.0031251581385731697,
+            0.0031837422866374254,
+            -0.0033291925210505724,
+            -0.010844117030501366,
+            0.004828137345612049,
+            0.003913013264536858,
+            0.0014322800561785698,
+            -0.002852439181879163,
+            0.00493318447843194,
+            -0.0031716213561594486,
+            0.010585539042949677,
+            0.006036181468516588,
+            -0.02036302164196968,
+            -0.006024060770869255,
+            -0.009623952209949493,
+            0.0073533132672309875,
+            -0.018375203013420105,
+            -0.009357293136417866,
+            0.00031968732946552336,
+            -0.0018777211662381887,
+            0.0033493938390165567,
+            -0.04812784120440483,
+            -0.019522642716765404,
+            -0.014334920793771744,
+            0.008516915142536163,
+            -0.0019443858182057738,
+            -0.03128794580698013,
+            -0.020039798691868782,
+            0.008133088238537312,
+            0.01580558344721794,
+            0.002410028362646699,
+            0.008920943364501,
+            0.010189591906964779,
+            0.00128480966668576,
+            -0.006436169613152742,
+            -0.0020251914393156767,
+            0.02081553265452385,
+            0.01714695803821087,
+            -0.030593017116189003,
+            -0.019150936976075172,
+            0.010642102919518948,
+            -0.008331061340868473,
+            -0.013615750707685947,
+            -0.01002798043191433,
+            0.007199782878160477,
+            0.0006338192615658045,
+            0.021041790023446083,
+            -0.0015514683909714222,
+            -0.019183259457349777,
+            0.004084725398570299,
+            0.00578568410128355,
+            -0.018892358988523483,
+            0.013219802640378475,
+            -0.007975516840815544,
+            -0.008743170648813248,
+            0.0058988118544220924,
+            -0.007264427375048399,
+            0.0032362660858780146,
+            -0.005183681845664978,
+            0.02580932155251503,
+            -0.020039798691868782,
+            0.0011969335610046983,
+            0.005535186734050512,
+            -0.014092504046857357,
+            0.016969185322523117,
+            -0.008605800569057465,
+            -0.0047594523057341576,
+            0.005013990215957165,
+            0.0007782593602314591,
+            0.009575468488037586,
+            0.009971416555345058,
+            0.008238134905695915,
+            -0.02023373357951641,
+            -0.0012837996473535895,
+            0.005381655879318714,
+            0.007357353810220957,
+            -0.008702768012881279,
+            0.03004353865981102,
+            -0.0054018571972846985,
+            -0.022722546011209488,
+            -0.0009408806799910963,
+            -0.008165409788489342,
+            0.0004906417452730238,
+            -0.00450491439551115,
+            0.007276548072695732,
+            0.015635890886187553,
+            0.006169511005282402,
+            -0.017599469050765038,
+            -0.012831935659050941,
+            0.003323131939396262,
+            0.005147319287061691,
+            0.012767291627824306,
+            0.015167218632996082,
+            -0.011555206961929798,
+            0.010068383067846298,
+            -0.020152926445007324,
+            0.0213165283203125,
+            -0.009753241203725338,
+            0.0038019055500626564,
+            -0.005591750610619783,
+            0.003121117828413844,
+            0.008670445531606674,
+            0.0175186637789011,
+            0.0019575166516005993,
+            -0.0012433967785909772,
+            -0.006133148446679115,
+            0.0350373275578022,
+            -0.00604022154584527,
+            0.016936862841248512,
+            -0.020734727382659912,
+            -0.010060302913188934,
+            -0.024063920602202415,
+            0.03662111610174179,
+            -0.004117047414183617,
+            -0.004589760676026344,
+            0.012007718905806541,
+            0.021930651739239693,
+            -0.012896580621600151,
+            0.009753241203725338,
+            0.003682717215269804,
+            -0.006230114959180355,
+            0.005591750610619783,
+            0.0013767260825261474,
+            0.0008519944967702031,
+            0.02079937234520912,
+            0.017163118347525597,
+            0.008141168393194675,
+            0.007805825211107731,
+            0.011377434246242046,
+            -0.0005098331021144986,
+            -0.012104685418307781,
+            0.01182994619011879,
+            -0.011999637819826603,
+            -0.004094826057553291,
+            -0.0032302055042237043,
+            -0.014415726996958256,
+            -0.009947174228727818,
+            -0.00828257855027914,
+            0.012128926813602448,
+            -0.0035453476011753082,
+            -0.01966809295117855,
+            0.0037251401226967573,
+            0.020266054198145866,
+            0.003032231703400612,
+            0.0059957788325846195,
+            0.014052101410925388,
+            0.01886003650724888,
+            0.01986202597618103,
+            0.0070462520234286785,
+            -0.02941325306892395,
+            -0.0009024980245158076,
+            -0.007070493884384632,
+            -0.006036181468516588,
+            0.009219923987984657,
+            0.0011949134059250355,
+            0.00810884591192007,
+            -0.0035776698496192694,
+            0.012799614109098911,
+            0.03713827207684517,
+            0.013583428226411343,
+            -0.0011353192385286093,
+            0.002395887393504381,
+            -0.002739311195909977,
+            -0.01986202597618103,
+            0.018116625025868416,
+            -0.01623385399580002,
+            -0.027183018624782562,
+            0.021268045529723167,
+            -0.0002911528281401843,
+            -0.006185671780258417,
+            -0.021962974220514297,
+            -0.006884640548378229,
+            0.016080323606729507,
+            0.018116625025868416,
+            -0.0008767411927692592,
+            -0.0018676203908398747,
+            -0.0033190918620675802,
+            0.0031473797280341387,
+            -0.006161430384963751,
+            0.005583669990301132,
+            -0.0020221611484885216,
+            -0.023708375170826912,
+            -0.006815955974161625,
+            -0.011296628974378109,
+            0.006803835276514292,
+            0.0033069709315896034,
+            0.013696555979549885,
+            -0.004117047414183617,
+            0.020346861332654953,
+            0.005325092002749443,
+            -0.004234215710312128,
+            0.007013929542154074,
+            -0.009858288802206516,
+            -0.0030504129827022552,
+            -0.01997515559196472,
+            0.015223783440887928,
+            0.0055028642527759075,
+            0.0032847493421286345,
+            -0.001361575094051659,
+            0.02910619229078293,
+            -0.0005600841250270605,
+            0.006864439230412245,
+            -0.003783724270761013,
+            0.011248145252466202,
+            -0.002197913359850645,
+            -0.008751250803470612,
+            -0.03183742240071297,
+            -0.01039968617260456,
+            -0.02682747319340706,
+            0.023320509120821953,
+            -0.009219923987984657,
+            0.0012353162746876478,
+            0.008589639328420162,
+            0.003672616556286812,
+            -0.004315021447837353,
+            0.010924923233687878,
+            -0.013591509312391281,
+            -0.004234215710312128,
+            0.012896580621600151,
+            -0.0004992273752577603,
+            0.009720918722450733,
+            -0.0073937163688242435,
+            -0.003852409077808261,
+            -0.011369354091584682,
+            -0.008500753901898861,
+            -0.03010818175971508,
+            -0.013308688998222351,
+            -0.01452077366411686,
+            -0.013551106676459312,
+            -0.024500271305441856,
+            -0.0008459340897388756,
+            0.004197853151708841,
+            -0.003761502681300044,
+            -0.010900680907070637,
+            0.017486341297626495,
+            0.01416522916406393,
+            0.02770017459988594,
+            -0.026714345440268517,
+            0.0006358394166454673,
+            -0.007648254279047251,
+            0.018520653247833252,
+            -0.01072290912270546,
+            0.0050947959534823895,
+            0.0015726798446848989,
+            0.01677525043487549,
+            0.0033837363589555025,
+            0.0012464270694181323,
+            0.004517035558819771,
+            0.0018858016701415181,
+            -0.01027847733348608,
+            0.009179521352052689,
+            0.0008439139346592128,
+            -0.011644093319773674,
+            -0.003921093884855509,
+            0.016872217878699303,
+            0.0201690886169672,
+            0.01293698325753212,
+            -0.007219984196126461,
+            0.0034403002355247736,
+            -0.009381535463035107,
+            -0.011846107430756092,
+            -0.011789543554186821,
+            0.008314901031553745,
+            -0.025259843096137047,
+            0.01814894750714302,
+            0.008937103673815727,
+            -0.0008227024227380753,
+            0.007785623427480459,
+            0.0023514442145824432,
+            0.015967195853590965,
+            0.005523065570741892,
+            -0.019393354654312134,
+            -0.014448048546910286,
+            0.0027978953439742327,
+            0.012104685418307781,
+            0.0010252215433865786,
+            -0.0023817464243620634,
+            0.00710281589999795,
+            0.0002638809382915497,
+            -0.011474400758743286,
+            -0.017534824088215828,
+            0.0003237276105210185,
+            0.00938961561769247,
+            0.02290031872689724,
+            0.005757402162998915,
+            0.013446059077978134,
+            0.0022282155696302652,
+            -0.0057493215426802635,
+            -0.007143219001591206,
+            -0.0161045640707016,
+            0.015959113836288452,
+            0.0038928117137402296,
+            -0.025453777983784676,
+            0.024888137355446815,
+            0.002057513687759638,
+            0.01688838005065918,
+            -0.004686727188527584,
+            -0.005838207434862852,
+            0.023756857961416245,
+            0.027780979871749878,
+            0.0043109809048473835,
+            0.024839654564857483,
+            -0.013494541868567467,
+            0.0072523062117397785,
+            0.02516287751495838,
+            0.020459989085793495,
+            -0.009688596241176128,
+            0.0012908701319247484,
+            0.011967315338551998,
+            0.001570659689605236,
+            -0.0005828107241541147,
+            0.009971416555345058,
+            -0.010593620128929615,
+            0.010690586641430855,
+            -0.02175287902355194,
+            0.017437858507037163,
+            -0.0014777331380173564,
+            -0.005183681845664978,
+            -0.006395766511559486,
+            0.0050947959534823895,
+            0.006011939607560635,
+            0.011102695018053055,
+            -0.009203762747347355,
+            -0.021542783826589584,
+            -0.0015292468015104532,
+            -0.003304950660094619,
+            -0.012395585887134075,
+            -0.028120363131165504,
+            0.012177410535514355,
+            -0.013575348071753979,
+            -0.008112886920571327,
+            0.009882530197501183,
+            -0.01721160113811493,
+            0.00241204840131104,
+            -0.02335282973945141,
+            0.010108785703778267,
+            -0.0216397512704134,
+            0.0052564069628715515,
+            0.011854187585413456,
+            -0.005846288055181503,
+            0.004282698966562748,
+            0.006165470462292433,
+            0.010415847413241863,
+            8.724484359845519e-05,
+            -0.013195561245083809,
+            -0.007450280245393515,
+            0.028605196624994278,
+            0.006290719378739595,
+            -0.003737261053174734,
+            -0.01821359246969223,
+            -0.0008454290218651295,
+            -0.028492068871855736,
+            0.018310558050870895,
+            -0.00041185625013895333,
+            0.0008888620650395751,
+            -0.019651932641863823,
+            -0.011636012233793736,
+            -0.0008120966958813369,
+            0.014391484670341015,
+            -0.00019607994181569666,
+            0.003793824929744005,
+            0.0317404568195343,
+            -0.013599589467048645,
+            0.001599951763637364,
+            -6.699608547933167e-06,
+            -0.01303394977003336,
+            0.02005596086382866,
+            -0.005757402162998915,
+            0.006925043649971485,
+            -0.0007333112298510969,
+            -0.008165409788489342,
+            -0.014666223898530006,
+            0.022108424454927444,
+            -0.0033190918620675802,
+            -0.012258215807378292,
+            -0.006763432174921036,
+            0.002161551034078002,
+            0.013551106676459312,
+            -0.012379424646496773,
+            -0.008621961809694767,
+            0.010924923233687878,
+            0.00488470122218132,
+            -0.011110776104032993,
+            -0.010044141672551632,
+            -0.013478381559252739,
+            0.0037433214019984007,
+            -0.03494035825133324,
+            -0.006929083727300167,
+            0.04586528241634369,
+            0.0056038713082671165,
+            -0.008395706303417683,
+            -0.0058503285981714725,
+            0.0009530015522614121,
+            0.006973526906222105,
+            -0.01847217045724392,
+            -0.007619971875101328,
+            -0.027780979871749878,
+            0.005983657669275999,
+            0.002139329444617033,
+            0.03243538364768028,
+            0.0003406462783459574,
+            -0.018488330766558647,
+            0.003549387911334634,
+            -0.021300368010997772,
+            -0.0074421996250748634,
+            -0.018391363322734833,
+            -0.016209611669182777,
+            -0.008920943364501,
+            0.016904540359973907,
+            -0.015167218632996082,
+            0.02663354016840458,
+            0.016597479581832886,
+            0.007603811100125313,
+            0.003117077751085162,
+            -0.0037998852785676718,
+            -0.006415967829525471,
+            0.006432129070162773,
+            -0.0019383253529667854,
+            0.004953385796397924,
+            0.011296628974378109,
+            -0.01075523067265749,
+            0.009922932833433151,
+            0.013688475824892521,
+            -0.007624012418091297,
+            0.01840752549469471,
+            -0.00837954506278038,
+            -0.020459989085793495,
+            0.01452077366411686,
+            -0.014706626534461975,
+            0.00152621662709862,
+            0.0037574623711407185,
+            -0.00832298118621111,
+            0.00472713029012084,
+            0.018989326432347298,
+            0.0073977564461529255,
+            -0.00572507968172431,
+            0.017373213544487953,
+            0.020670082420110703,
+            0.012629921548068523,
+            -0.01050473377108574,
+            0.009591629728674889,
+            0.014439968392252922,
+            -0.0001660303387325257,
+            -0.0003701908572111279,
+            0.004407947883009911,
+            -0.014407645910978317,
+            -0.0006307890289463103,
+            0.003212024224922061,
+            -0.011094614863395691,
+            -0.0115713682025671,
+            0.016839895397424698,
+            0.005244286265224218,
+            -0.004517035558819771,
+            0.011700657196342945,
+            -0.01284001674503088,
+            -0.012751130387187004,
+            -0.009825966320931911,
+            0.0066987876780331135,
+            -0.040758367627859116,
+            0.004840258043259382,
+            -0.006464451551437378,
+            0.018100464716553688,
+            -0.014456129632890224,
+            0.005236205644905567,
+            -0.005919013172388077,
+            -0.010949164628982544,
+            0.015070252120494843,
+            -0.0009762331610545516,
+            -0.002218114910647273,
+            0.03188590705394745,
+            0.011934993788599968,
+            0.01077947299927473,
+            0.004646324552595615,
+            -0.016371222212910652,
+            0.034261591732501984,
+            -0.009882530197501183,
+            0.00012966780923306942,
+            -0.012856177985668182,
+            -0.0017342910869047046,
+            0.03186974674463272,
+            0.029332447797060013,
+            -0.005046312231570482,
+            0.004480673000216484,
+            -0.00795531552284956,
+            0.01363191194832325,
+            0.015991436317563057,
+            0.008133088238537312,
+            0.00599173828959465,
+            0.0032605077140033245,
+            0.004383706022053957,
+            0.0011009768350049853,
+            0.038495808839797974,
+            -0.012613760307431221,
+            -0.011943073943257332,
+            -0.013955134898424149,
+            -0.016662122681736946,
+            -0.022738708183169365,
+            0.02138117328286171,
+            -0.00747856218367815,
+            -0.014916721731424332,
+            -0.007769462652504444,
+            0.018989326432347298,
+            0.00705433264374733,
+            0.006714948918670416,
+            -0.0012928902870044112,
+            0.005539226811379194,
+            -0.024451786652207375,
+            -0.00499378889799118,
+            -0.01738937385380268,
+            0.002729210536926985,
+            -0.0007913902518339455,
+            0.015951033681631088,
+            0.018456008285284042,
+            0.017437858507037163,
+            0.019005486741662025,
+            -0.007426038384437561,
+            0.006169511005282402,
+            -0.0055069043301045895,
+            -0.01226629689335823,
+            0.019199419766664505,
+            0.02086401730775833,
+            -0.01404402032494545,
+            -0.004274618346244097,
+            -0.016710607334971428,
+            -0.021171078085899353,
+            0.002983748447149992,
+            0.009543146006762981,
+            0.010884519666433334,
+            -0.013219802640378475,
+            0.002816076623275876,
+            0.015264186076819897,
+            -0.007042211946099997,
+            0.002769613405689597,
+            0.0048483386635780334,
+            0.017373213544487953,
+            0.01224205456674099,
+            0.0031675812788307667,
+            0.01803581975400448,
+            -0.018957003951072693,
+            -0.02776481956243515,
+            0.006589700002223253,
+            -0.014650062657892704,
+            0.017502501606941223,
+            0.009025990031659603,
+            0.005672555882483721,
+            -0.03149804100394249,
+            -0.010827955789864063,
+            0.01814894750714302,
+            -0.00046715763164684176,
+            -0.001248447224497795,
+            0.001733281067572534,
+            0.01077947299927473,
+            -0.009228004142642021,
+            -0.012670325115323067,
+            -0.01886003650724888,
+            0.0010595639469102025,
+            -0.0051513598300516605,
+            -0.0030584936030209064,
+            -0.01293698325753212,
+            -0.00844419002532959,
+            -0.013486461713910103,
+            -0.004420068580657244,
+            0.0008504794095642865,
+            0.0234659593552351,
+            0.01871458627283573,
+            0.01936103217303753,
+            0.002983748447149992,
+            0.004500874318182468,
+            0.017744919285178185,
+            0.013025869615375996,
+            0.007922993041574955,
+            0.00160500209312886,
+            -0.0019474159926176071,
+            0.028572876006364822,
+            -0.019199419766664505,
+            -0.012031960301101208,
+            0.018569136038422585,
+            -0.01331677008420229,
+            0.017761079594492912,
+            -0.02118724025785923,
+            0.001272688852623105,
+            0.005692757666110992,
+            0.003387776669114828,
+            -0.009809805080294609,
+            -0.008686606772243977,
+            0.019247904419898987,
+            0.009122957475483418,
+            -0.009413857012987137,
+            0.01979738287627697,
+            0.005769522860646248,
+            -0.006658385042101145,
+            -0.009696677327156067,
+            -0.0019383253529667854,
+            -0.0041089667938649654,
+            -0.015732858330011368,
+            0.016856057569384575,
+            0.018132785335183144,
+            -0.015353072434663773,
+            0.0042665377259254456,
+            0.01879539154469967,
+            -0.01973273791372776,
+            0.011248145252466202,
+            0.01378544233739376,
+            -0.010092624463140965,
+            0.007809865288436413,
+            -0.0010186561848968267,
+            0.013446059077978134,
+            0.003761502681300044,
+            -0.004145329352468252,
+            -0.0005929113831371069,
+            0.011967315338551998,
+            -0.017243923619389534,
+            -0.012112765572965145,
+            -0.012427907437086105,
+            -0.008157329633831978,
+            0.007123017217963934,
+            0.002535277046263218,
+            -0.0056483144871890545,
+            0.009025990031659603,
+            -0.011337031610310078,
+            -0.015991436317563057,
+            -0.028378941118717194,
+            0.0005499834078364074,
+            -0.00971283856779337,
+            -0.001211074530147016,
+            0.014698546379804611,
+            0.015749020501971245,
+            -0.0009449209901504219,
+            -0.016969185322523117,
+            0.01816510781645775,
+            0.04408755898475647,
+            -0.007809865288436413,
+            0.003777663689106703,
+            0.010601700283586979,
+            0.002488813828676939,
+            -0.0026302237529307604,
+            -0.0022908400278538465,
+            -0.016427787020802498,
+            -0.03765542805194855,
+            -0.012912741862237453,
+            0.01077947299927473,
+            -0.0012787493178620934,
+            -0.004638243932276964,
+            -0.021914489567279816,
+            -0.005842247977852821,
+            -0.016201531514525414,
+            -0.004921063780784607,
+            -0.019651932641863823,
+            -0.004799854941666126,
+            0.013979376293718815,
+            -0.015959113836288452,
+            0.019878188148140907,
+            -0.02024989388883114,
+            0.020540794357657433,
+            0.01385008729994297,
+            0.02277103066444397,
+            -0.01641162671148777,
+            0.009341131895780563,
+            -0.0014120786217972636,
+            0.014666223898530006,
+            0.01034312229603529,
+            0.007308870553970337,
+            -0.010617861524224281,
+            -0.006634143181145191,
+            0.01928022690117359,
+            0.009470420889556408,
+            0.013082433491945267,
+            0.009567388333380222,
+            -0.002262558089569211,
+            -0.0073371524922549725,
+            -0.014464209787547588,
+            -0.022156907245516777,
+            0.003163540968671441,
+            -0.0021676113829016685,
+            0.01688838005065918,
+            -0.020379183813929558,
+            -0.021607428789138794,
+            0.0002142612065654248,
+            0.014722787775099277,
+            0.035069648176431656,
+            -0.008573479019105434,
+            0.004371585324406624,
+            -0.012330940924584866,
+            0.01202387921512127,
+            0.018310558050870895,
+            0.005236205644905567,
+            0.0012221853248775005,
+            -0.020346861332654953,
+            0.011579448357224464,
+            0.011086533777415752,
+            -0.002422149060294032,
+            -0.0035089850425720215,
+            -0.0029029427096247673,
+            -0.02290031872689724,
+            0.0064078872092068195,
+            0.004399867262691259,
+            0.038043297827243805,
+            0.013761200942099094,
+            -0.0014807634288445115,
+            0.0072523062117397785,
+            0.029073869809508324,
+            -0.0032362660858780146,
+            0.009276487864553928,
+            0.0007363414042629302,
+            -0.003854429116472602,
+            0.008060363121330738,
+            -0.00811692699790001,
+            -0.0459299273788929,
+            0.010577458888292313,
+            -0.004492793697863817,
+            0.038043297827243805,
+            -0.0032746486831456423,
+            0.03043140470981598,
+            -0.006533136125653982,
+            -0.00810884591192007,
+            -0.005531146191060543,
+            0.017373213544487953,
+            -0.012508713640272617,
+            0.012258215807378292,
+            -0.008044201880693436,
+            -0.010625941678881645,
+            -0.0016342941671609879,
+            0.006779593415558338,
+            -0.012371343560516834,
+            -0.007333111949265003,
+            0.022156907245516777,
+            0.024387143552303314,
+            0.013761200942099094,
+            -0.009834046475589275,
+            0.0315626822412014,
+            -0.007510884664952755,
+            -0.014472290873527527,
+            0.019005486741662025,
+            -0.0037109991535544395,
+            -0.006428088992834091,
+            -0.001322182361036539,
+            -0.0023029607255011797,
+            -0.0024322497192770243,
+            0.017357051372528076,
+            0.0175186637789011,
+            -0.00429886020720005,
+            0.0015373273054137826,
+            0.00429886020720005,
+            -0.002468612277880311,
+            0.019150936976075172,
+            0.011749139986932278,
+            -0.03636253997683525,
+            0.014221793040633202,
+            -0.029364770278334618,
+            -0.01821359246969223,
+            -0.007579569239169359,
+            -0.004157450515776873,
+            0.011708737351000309,
+            -0.02390230819582939,
+            -0.011490561999380589,
+            -0.023417474702000618,
+            0.009753241203725338,
+            0.016969185322523117,
+            0.0065290960483253,
+            0.012137007899582386,
+            0.014302598312497139,
+            0.0012252154992893338,
+            -0.017421696335077286,
+            -0.013680395670235157,
+            -0.02765168994665146,
+            -0.014706626534461975,
+            0.012185490690171719,
+            0.0037493817508220673,
+            -0.013446059077978134,
+            -0.0023453838657587767,
+            0.026649700477719307,
+            -0.0037918046582490206,
+            -0.00313121872022748,
+            0.009753241203725338,
+            0.003694837912917137,
+            0.005220044404268265,
+            -0.005886691156774759,
+            0.014246034435927868,
+            0.00716342031955719,
+            0.01012494694441557,
+            -0.02984960377216339,
+            0.01738937385380268,
+            -0.003737261053174734,
+            -0.0008519944967702031,
+            0.0013363233301788568,
+            -0.004262497648596764,
+            -0.011660254560410976,
+            -0.0017757039749994874,
+            -0.02548610046505928,
+            -0.00689272116869688,
+            -0.0031918229069560766,
+            -0.011716818436980247,
+            -0.002658505691215396,
+            -0.00741795776411891,
+            -0.016807572916150093,
+            -0.010593620128929615,
+            0.01847217045724392,
+            -0.011062292382121086,
+            0.01293698325753212,
+            0.0011060272809118032,
+            -0.017550986260175705,
+            0.007361393887549639,
+            -0.012654163874685764,
+            0.0016403545159846544,
+            -0.009591629728674889,
+            0.0001431775017408654,
+            0.019134776666760445,
+            -0.008767412044107914,
+            0.006581619381904602,
+            -0.004278658889234066,
+            0.011151178739964962,
+            -0.00102168635930866,
+            0.016710607334971428,
+            0.015951033681631088,
+            0.011046131141483784,
+            0.00010163834667764604,
+            -0.015732858330011368,
+            -0.019053971394896507,
+            0.022108424454927444,
+            0.004214014392346144,
+            -0.01046433113515377,
+            0.02404775843024254,
+            0.002684767358005047,
+            0.01992667093873024,
+            -0.011652173474431038,
+            0.003864529775455594,
+            0.024516431614756584,
+            0.010060302913188934,
+            0.008993667550384998,
+            -0.0035776698496192694,
+            -0.0003694332845043391,
+            0.00482409680262208,
+            0.03416462615132332,
+            0.019247904419898987,
+            -0.005927093792706728,
+            -0.024306336417794228,
+            0.018262075260281563,
+            0.0019191340543329716,
+            0.001435310230590403,
+            0.02024989388883114,
+            0.024322498589754105,
+            0.01808430254459381,
+            0.007143219001591206,
+            0.008654284290969372,
+            -0.012452149763703346,
+            0.010747150518000126,
+            -0.0007651284104213119,
+            -0.0055473074316978455,
+            0.0003335757937747985,
+            -0.029591025784611702,
+            0.012258215807378292,
+            -0.00270294863730669,
+            -0.014229873195290565,
+            0.0021817523520439863,
+            -0.013130917213857174,
+            0.022867996245622635,
+            -0.0011575408279895782,
+            -0.01645202934741974,
+            -0.004694807808846235,
+            -0.024063920602202415,
+            0.0028201169334352016,
+            -0.01923174224793911,
+            -0.005369534716010094,
+            -0.012015799060463905,
+            -0.0013918771874159575,
+            -0.022237712517380714,
+            -0.009591629728674889,
+            0.011539045721292496,
+            -0.002353464253246784,
+            -0.007450280245393515,
+            -0.005914973095059395,
+            0.00763613311573863,
+            -0.018569136038422585,
+            -0.0028423385228961706,
+            0.005716999061405659,
+            0.019150936976075172,
+            0.0025191158056259155,
+            0.003535246942192316,
+            0.01871458627283573,
+            0.013963215053081512,
+            -0.03445552662014961,
+            0.003650394966825843,
+            0.00254537770524621,
+            0.00223023584112525,
+            0.025195199996232986,
+            0.007462400943040848,
+            -0.00732099125161767,
+            0.014795512892305851,
+            -0.0003237276105210185,
+            0.01871458627283573,
+            0.002375685842707753,
+            -0.014682385139167309,
+            -0.00716342031955719,
+            0.005672555882483721,
+            -0.007737140171229839,
+            0.021106433123350143,
+            -0.002543357666581869,
+            0.007813905365765095,
+            -0.009995657950639725,
+            -0.011078453622758389,
+            -0.00678363349288702,
+            0.00013863218191545457,
+            -0.006011939607560635,
+            0.00885629840195179,
+            -0.009809805080294609,
+            0.005813966039568186,
+            0.006133148446679115,
+            0.004022100940346718,
+            -0.013938973657786846,
+            0.016056081280112267,
+            -0.003199903527274728,
+            0.008573479019105434,
+            0.02605173923075199,
+            0.00572507968172431,
+            -0.009438099339604378,
+            0.017550986260175705,
+            -8.59191277413629e-05,
+            -0.01128854788839817,
+            -0.02244780771434307,
+            -0.003878670744597912,
+            -0.03907760977745056,
+            0.028378941118717194,
+            0.00023193744709715247,
+            -0.018520653247833252,
+            0.02517903782427311,
+            -0.00816136971116066,
+            -0.008274497464299202,
+            -0.00018143391935154796,
+            -0.0100845443084836,
+            -0.02340131439268589,
+            0.002715069567784667,
+            -0.010569377802312374,
+            -0.006108906585723162,
+            0.011934993788599968,
+            0.030334437265992165,
+            -0.008815895766019821,
+            -0.0008913872297853231,
+            0.006937164347618818,
+            0.003607971826568246,
+            0.01879539154469967,
+            -0.028071880340576172,
+            -0.0032948500011116266,
+            -0.018326720222830772,
+            0.019199419766664505,
+            -0.0007105846307240427,
+            -0.01682373508810997,
+            0.012339022010564804,
+            -0.014949044212698936,
+            -0.02175287902355194,
+            -0.0003398887347429991,
+            -0.0060563827864825726,
+            0.005664475727826357,
+            0.0063594039529562,
+            0.02498510479927063,
+            0.0033756557386368513,
+            0.007963395677506924,
+            -0.02580932155251503,
+            -0.023708375170826912,
+            0.02383766509592533,
+            0.01103805098682642,
+            -0.014932882972061634,
+            -0.0011555206729099154,
+            -0.005106916651129723,
+            -0.007219984196126461,
+            -0.019910510629415512,
+            0.007175541017204523,
+            0.06651920825242996,
+            -0.007361393887549639,
+            0.003096876200288534,
+            -0.015643972903490067,
+            -0.005632153246551752,
+            -0.007098775822669268,
+            -0.01929638721048832,
+            -0.006363444495946169,
+            0.00024822482373565435,
+            0.005733160302042961,
+            -0.025405293330550194,
+            0.004054422955960035,
+            -0.006706868298351765,
+            0.007001808844506741,
+            -0.012137007899582386,
+            0.010270397178828716,
+            -0.015490441583096981,
+            -0.006860399153083563,
+            0.0028686001896858215,
+            -0.009817885234951973,
+            -0.0009388605831190944,
+            -0.017243923619389534,
+            0.001279759337194264,
+            0.016112644225358963,
+            0.003616052446886897,
+            -0.012767291627824306,
+            -0.016460109502077103,
+            -0.01947415992617607,
+            -0.01214508805423975,
+            -0.005070554092526436,
+            -0.008339142426848412,
+            -0.023320509120821953,
+            0.012169329449534416,
+            -0.007741180714219809,
+            0.016856057569384575,
+            0.011054212227463722,
+            0.013429897837340832,
+            0.020573116838932037,
+            -0.003389796707779169,
+            0.009308810345828533,
+            -0.013421816751360893,
+            0.03067382238805294,
+            -0.0022787190973758698,
+            -0.023449797183275223,
+            -0.0007312910747714341,
+            0.00033913119114004076,
+            -0.007931074127554893,
+            -0.021849846467375755,
+            0.007555327378213406,
+            -0.002583760302513838,
+            0.011934993788599968,
+            -0.005959416273981333,
+            0.003995839040726423,
+            0.0015686395345255733,
+            0.006294759456068277,
+            0.004343303386121988,
+            0.017841886729002,
+            -0.0059957788325846195,
+            0.01997515559196472,
+            -0.02815268561244011,
+            -0.01738937385380268,
+            -0.008230054751038551,
+            -0.007490682881325483,
+            0.012613760307431221,
+            0.006044262088835239,
+            -0.0032685883343219757,
+            -0.016492431983351707,
+            -0.010076463222503662,
+            0.008880539797246456,
+            -0.016807572916150093,
+            -0.026601217687129974,
+            -0.0016080322675406933,
+            -0.0325646735727787,
+            0.007110896520316601,
+            0.004214014392346144,
+            -0.005700838286429644,
+            -0.005575589369982481,
+            0.02010444365441799,
+            0.00948658213019371,
+            0.006646263878792524,
+            0.005692757666110992,
+            -0.0017070191679522395,
+            -0.011312790215015411,
+            -0.0009595670271664858,
+            0.0015565187204629183,
+            0.011749139986932278,
+            -0.002377706114202738,
+            -0.01571669802069664,
+            0.015353072434663773,
+            0.009034071117639542,
+            -0.0108198756352067,
+            0.0004469561972655356,
+            0.019700415432453156,
+            0.0039574564434587955,
+            -0.0180196575820446,
+            0.004597840830683708,
+            0.018197430297732353,
+            -0.005979617591947317,
+            0.013163238763809204,
+            0.026552734896540642,
+            -1.8986169379786588e-05,
+            -0.00078280468005687,
+            -0.017599469050765038,
+            -0.01255719643086195,
+            -0.0035574682988226414,
+            0.023611409589648247,
+            0.029251642525196075,
+            0.002240336500108242,
+            -0.027344629168510437,
+            0.009567388333380222,
+            -0.016969185322523117,
+            0.012961224652826786,
+            -0.015231863595545292,
+            0.0021474098321050406,
+            0.02092866227030754,
+            -0.013761200942099094,
+            0.004989748355001211,
+            0.009898691438138485,
+            0.028750646859407425,
+            0.004561478737741709,
+            -0.015264186076819897,
+            0.0006994738359935582,
+            -0.02081553265452385,
+            0.002008020179346204,
+            -0.006395766511559486,
+            0.005733160302042961,
+            -0.011369354091584682,
+            0.00646849162876606,
+            -0.011110776104032993,
+            0.0006439199787564576,
+            0.0012141048209741712,
+            -0.02118724025785923,
+            0.007583609316498041,
+            -0.018003497272729874,
+            0.0037453414406627417,
+            0.0010767352068796754,
+            -0.004048362839967012,
+            -0.023643730208277702,
+            -0.002511035418137908,
+            0.028896097093820572,
+            -0.007805825211107731,
+            0.05788916349411011,
+            -0.006290719378739595,
+            0.0014403605600818992,
+            -0.009034071117639542,
+            0.0063068801537156105,
+            -0.0009706777636893094,
+            -0.009349212981760502,
+            -0.005413977894932032,
+            0.002723150188103318,
+            0.0029130433686077595,
+            0.004492793697863817,
+            0.009914852678775787,
+            0.004836217500269413,
+            0.0005153884994797409,
+            0.02984960377216339,
+            0.0008681556209921837,
+            0.0085977204144001,
+            -0.004484713077545166,
+            0.0022908400278538465,
+            -0.016524754464626312,
+            0.01803581975400448,
+            0.005801845341920853,
+            -0.022609418258070946,
+            -0.016662122681736946,
+            -0.017049990594387054,
+            0.013462220318615437,
+            -0.0031009165104478598,
+            0.013898570090532303,
+            -0.005846288055181503,
+            -0.014553096145391464,
+            0.017664114013314247,
+            0.008557317778468132,
+            0.009405776858329773,
+            -0.0002760017814580351,
+            0.005874569993466139,
+            0.005462461616843939,
+            0.008823975920677185,
+            -0.005102876108139753,
+            -0.005317011382430792,
+            -0.0026766869705170393,
+            0.014981365762650967,
+            -0.011029969900846481,
+            -0.0011454200139269233,
+            -0.016112644225358963,
+            -0.006791714113205671,
+            0.022932641208171844,
+            0.000245573406573385,
+            -0.003708978882059455,
+            0.02650425024330616,
+            0.011692576110363007,
+            -0.00010833006672328338,
+            -0.009696677327156067,
+            -0.0013646052684634924,
+            0.007725019473582506,
+            0.020718567073345184,
+            -0.004028161056339741,
+            -0.008500753901898861,
+            -0.017615629360079765,
+            -0.0011454200139269233,
+            0.030059698969125748,
+            -0.007535126060247421,
+            -0.019894348457455635,
+            0.01284001674503088
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "7754c53d-3aa4-47bb-93fa-1de98e8c3ac1",
+      "type": "child",
+      "source": {
+        "id": "1e687808-8db8-4320-805d-0282a8d679bc",
+        "properties": {
+          "page_content": "Deepseekmoe: Towards Ultimate Expert Specialization In\n\nMixture-Of-Experts Language Models\n\nDamai Dai∗1,2, Chengqi Deng1, Chenggang Zhao∗1,3, R.X. Xu1, Huazuo Gao1, Deli Chen1, Jiashi Li1, Wangding Zeng1, Xingkai Yu∗1,4, Y. Wu1, Zhenda Xie1, Y.K. Li1, Panpan Huang1, Fuli Luo1, Chong Ruan1, Zhifang Sui2, Wenfeng Liang1 1DeepSeek-AI 2National Key Laboratory for Multimedia Information Processing, Peking University 3Institute for Interdisciplinary Information Sciences, Tsinghua University 4National Key Laboratory for Novel Software Technology, Nanjing University {daidamai, szf}@pku.edu.cn, {wenfeng.liang}@deepseek.com https://github.com/deepseek-ai/DeepSeek-MoE\n\nAbstract\n\nIn the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top- out of experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into ones and activating from them, allowing for a more flexible combination of activated experts; (2) isolating experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5× expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations.\n\nFurther, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\n1. Introduction\n\nRecent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023a). It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017) has emerged as a popular solution. It can\n\nenable parameter scaling, while concurrently keeping computational costs at a modest level.\n\nRecent applications of MoE architectures in Transformers (Vaswani et al., 2017) have yielded successful attempts at scaling language models to a substantial size (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022), accompanied with remarkable performance. These achievements underscore the considerable potential and promise of MoE language models.\n\nDespite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts.\n\nThis architecture manifests two potential issues: (1) Knowledge Hybridity: existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. (2) Knowledge Redundancy: tokens assigned to different experts may require common knowledge. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters.\n\nThese issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models.\n\nIn response to the aforementioned issues, we introduce DeepSeekMoE, an innovative MoE architecture specifically designed towards ultimate expert specialization. Our architecture involves two principal strategies: (1) Fine-Grained Expert Segmentation: while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. (2) Shared Expert Isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.\n\nThrough compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized.\n\nStarting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B (Lepikhin et al., 2021) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5× expert parameters and computation. Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE.\n\nThese studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization.\n\nLeveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B (DeepSeek-AI, 2024), a dense model trained on the same 2T corpus. We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B (Touvron et al., 2023b), which has approximately 2.5 times the activated parameters. Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard1. Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. The experimental results still validate its substantial advantages over the GShard architecture consistently. In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nOur contributions are summarized as follows: - Architectural Innovation. We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation.\n\nEmpirical Validation. We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. Experimental results validate the high 1https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models\n\nScalability. We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B.\n\nAlignment for MoE. We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B.\n\nPublic Release. In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization.\n\n2. Preliminaries: Mixture-Of-Experts For Transformers\n\nWe first introduce a generic MoE architecture commonly used in Transformer language models. A standard Transformer language model is constructed by stacking layers of standard Transformer blocks, where each block can be represented as follows:\n\n$$\\mathbf{u}{1:T}^{l}=\\text{Self-Att}\\left(\\mathbf{h}{1:T}^{l-1}\\right)+\\mathbf{h}{1:T}^{l-1},\\tag{1}$$ $$\\mathbf{h}{t}^{l}=\\text{FFN}\\left(\\mathbf{u}{t}^{l}\\right)+\\mathbf{u}{t}^{l},\\tag{2}$$\n\nwhere denotes the sequence length, Self-Att(·) denotes the self-attention module, FFN(·) denotes the Feed-Forward Network (FFN), u 1: ∈ R× are the hidden states of all tokens after the -th attention module, and h\n\n∈ Ris the output hidden state of the -th token after the -th Transformer block. For brevity, we omit the layer normalization in the above formulations.\n\nA typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022). An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. Then, each token will be assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts. If the -th FFN is substituted with an MoE layer, the computation for its output hidden state h\n\nis expressed as: where denotes the total number of experts, FFN(·) is the -th expert FFN, , denotes the gate value for the -th expert, , denotes the token-to-expert affinity, Topk(·, ) denotes the set comprising highest affinity scores among those calculated for the -th token and all experts, and e\n\nis the centroid of the -th expert in the -th layer. Note that ,is sparse, indicating that only out of gate values are nonzero. This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only experts. Also, in the above formulations, we omit the layer normalization operation for brevity.\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{N}\\left(g_{i,t}\\operatorname{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\operatorname{Topk}({s_{j,t}|1\\leqslant j\\leqslant N},K),\\ 0,&\\text{otherwise},\\end{cases}$$ $$s_{i,t}=\\operatorname{Softmax}{i}\\left(\\mathbf{u}{t}^{l}\\,\\mathbf{e}_{i}^{l}\\right),$$\n\n$$(3)$$\n\n(4) $\\binom{4}{5}$ .\n\n3. Deepseekmoe Architecture\n\nOn top of the generic MoE architecture outlined in Section 2, we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. As illustrated in Figure 2, our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. Both of these strategies are designed to elevate the level of expert specialization.\n\n3.1. Fine-Grained Expert Segmentation\n\nIn scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts.\n\nIn pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. The finer expert segmentation enables a more flexible and adaptable combination of activated experts. To be specific, on top of a typical MoE architecture shown in Figure 2(a), we segment each expert FFN into smaller experts by reducing the FFN intermediate hidden dimension to 1 times its original size. Since each expert becomes smaller, in response, we also increase the number of activated experts to times to keep the same computation cost, as illustrated in Figure 2(b). With the fine-grained expert segmentation, the output of an MoE layer can be expressed as:\n\n$$\\mathbf{h}{t}^{i}=\\sum{i=1}^{\\text{mN}}\\left(g_{i,t}\\text{FFN}{i}\\left(\\mathbf{u}{t}^{i}\\right)\\right)+\\mathbf{u}{t}^{i},\\tag{6}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\text{Tokr}(s_{i,t}|1\\leqslant j\\leqslant m\\text{N}),m\\text{K}),\\ 0,&\\text{otherwise},\\end{cases}$$ (7) $$s_{i,t}=\\text{Softmax}{i}\\left(\\mathbf{u}{t}^{T}\\mathbf{e}_{t}^{i}\\right),\\tag{8}$$ where the total number of expert parameters is equal to $N$ times the number of parameters in a\n\n$$(6)$$\n\n(7) (8) $\\frac{1}{2}$\n\nstandard FFN, and denotes the total number of fine-grained experts. With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to .\n\nFrom a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. As an illustrative example, we consider the case where = 16. A typical top-2 routing strategy can yield 16 2 = 120 possible combinations. By contrast, if each expert is split into 4 smaller experts, the fine-grained routing strategy can yield 64 8 = 4, 426, 165, 368 potential combinations. The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition.\n\n3.2. Shared Expert Isolation\n\nWith a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. This alleviation of redundancy will contribute to a more parameterefficient model with more specialized experts.\n\nTowards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate experts to serve as shared experts. Regardless of the router module, each token will be deterministically assigned to these shared experts. In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by , as depicted in Figure 2(c). With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows:\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{K_{s}}\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)+\\sum_{i=K_{s}+1}^{mN}\\left(g_{i,t}\\;\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},\\tag{9}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\mathrm{Topk}{s_{j,t}|K_{s}+1\\leqslant j\\leqslant mN},mK-K_{s}},\\ 0,&\\text{otherwise},\\end{cases}$$ (10) $$s_{i,t}=\\mathrm{Softmax}{i}\\left(\\mathbf{u}{t}^{l,T}e_{i}^{l}\\right).\\tag{11}$$ $\\mathbf{h}\\mathbf{M}\\mathbf{E}$ the number of closed-compact is $K_{s}$ the total number of routed currents.\n\nFinally, in DeepSeekMoE, the number of shared expert is , the total number of routed experts is − , and the number of nonzero gates is − .\n\nIt is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022). The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint.\n\n3.3. Load Balance Consideration\n\nAutomatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. Firstly, there is a risk of routing collapse (Shazeer et al., 2017), i.e., the model always selects only a few experts, preventing other experts from sufficient training.\n\nSecondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.\n\nExpert-Level Balance Loss. In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. The computation of the balance loss is as follows:\n\n$$\\begin{array}{l}{{{\\mathcal L}{\\mathrm{ExpBal}}=\\alpha{1}\\sum_{i=1}^{N^{\\prime}}f_{i}P_{i},}}\\ {{f_{i}=\\frac{N^{\\prime}}{K^{T}}\\sum_{t=1}^{T}\\mathds{1}\\,(\\mathrm{Token}\\;t\\;\\mathrm{s}\\mathrm{lects}\\;\\mathrm{Expert}\\;i),}}\\ {{P_{i}=\\frac{1}{T}\\sum_{t=1}^{T}s_{i,t},}}\\end{array}$$\n\n(12) $$\\begin{array}{l}\\small\\text{(13)}\\end{array}$$ = (14) $$\\begin{array}{l}\\small\\text{(14)}\\end{array}$$ .\n\n7 where 1 is a hyper-parameter called expert-level balance factor, ′is equal to ( − ) and ′ is equal to ( − ) for brevity. 1(·) denotes the indicator function.\n\nDevice-Level Balance Loss. In addition to the expert-level balance loss, we introduce a devicelevel balance loss. When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. Instead, our primary objective is to ensure balanced computation across the devices. If we partition all routed experts into groups {E1, E2, ..., E}, and deploy each group on a single device, the device-level balance loss is computed as follows:\n\n$$\\mathcal{L}{\\text{DevBal}}=\\alpha{2}\\sum_{i=1}^{D}f_{i}^{\\prime}P_{i}^{\\prime},$$ $$f_{i}^{\\prime}=\\frac{1}{|\\mathcal{E}{i}|}\\sum{j\\in\\mathcal{E}{i}}f{j},$$ $$P_{i}^{\\prime}=\\sum_{j\\in\\mathcal{E}{i}}P{j},$$\n\n(15) $$\\begin{array}{l}\\small\\text{(16)}\\end{array}$$ = (17) .\n\nwhere 2 is a hyper-parameter called device-level balance factor. In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices.\n\n4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n6. Alignment For Deepseekmoe 16B\n\nPrevious research indicates that MoE models typically do not emerge significant gains from fine-tuning (Artetxe et al., 2022; Fedus et al., 2021). However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B.\n\n6.1. Experimental Setup\n\nTraining Data. For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios.\n\nHyper-Parameters. During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).\n\nWe employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10−5 without incorporating any learning rate scheduling strategy.\n\nEvaluation Benchmarks. For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3, with the following adjustments: (1) We exclude Pile (Gao et al., 2020) since chat models are seldom employed for pure language modeling. (2) We exclude\n\nMetric # Shot LLaMA2 DeepSeek DeepSeekMoE SFT 7B Chat 7B Chat 16B # Total Params N/A 6.7B 6.9B 16.4B # Activated Params N/A 6.7B 6.9B 2.8B FLOPs per 4K Tokens N/A 187.9T 183.5T 74.4T HellaSwag (Acc.) 0-shot 67.9 71.0 72.2 PIQA (Acc.) 0-shot 76.9 78.4 79.7 ARC-easy (Acc.) 0-shot 69.7 70.2 69.9 ARC-challenge (Acc.) 0-shot 50.8 50.2 50.0 BBH (EM) 3-shot 39.3 43.1 42.2 RACE-middle (Acc.) 5-shot 63.9 66.1 64.8 RACE-high (Acc.) 5-shot 49.6 50.8 50.6 DROP (EM) 1-shot 40.0 41.7 33.8 GSM8K (EM) 0-shot 63.4 62.6 62.2 MATH (EM) 4-shot 13.5 14.7 15.2 HumanEval (Pass@1) 0-shot 35.4 45.1 45.7 MBPP (Pass@1) 3-shot 27.8 39.0 46.2 TriviaQA (EM) 5-shot 60.1 59.5 63.3 NaturalQuestions (EM) 0-shot 35.2 32.7 35.1 MMLU (Acc.) 0-shot 50.0 49.7 47.2 WinoGrande (Acc.) 0-shot 65.1 68.4 69.0 CLUEWSC (EM) 5-shot 48.4 66.2 68.2 CEval (Acc.) 0-shot 35.1 44.7 40.0 CMMLU (Acc.) 0-shot 36.9 51.2 49.3\n\nCHID (Zheng et al., 2019) due to the observed instability of results, hindering the derivation of solid conclusions. (3) We additionally include BBH (Suzgun et al., 2022) to provide a more comprehensive assessment of the reasoning ability of the chat models.\n\nTable 5 | Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations.\n\n6.2. Evaluations\n\nBaselines. In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. Correspondingly, we construct three chat models, including LLaMA2 SFT 7B3, DeepSeek Chat 7B, and DeepSeekMoE Chat 16B.\n\nSubsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks.\n\nResults. The evaluation results are presented in Table 5. Our key observations include: (1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intensive tasks (TriviaQA, NaturalQuestions). (2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. In addition, it also surpasses DeepSeek Chat 7B. (3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1). However, it is worth noting that, after supervised finetuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. (4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations.\n\n7. Deepseekmoe 145B Ongoing\n\nEncouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available.\n\n7.1. Experimental Setup\n\nTraining Data and Tokenization. For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study.\n\nModel Settings. For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps.\n\nSubsequently, the learning rate keeps constant during the remaining training process. The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. Also, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse.\n\nEvaluation Benchmarks. We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3).\n\n7.2. Evaluations\n\nBaselines. Apart from DeepSeekMoE 145B, we consider three additional models for comparison. DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters.\n\nResults. From the evaluation results presented in Table 6, we have the following observations: (1) Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. (2) On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. (3) At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5.\n\n8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\nAppendices A. Overview Of Hyper-Parameters\n\nWe present the overview of hyper-parameters for DeepSeekMoE across various sizes in Table 7.\n\n# Params # Layers Hidden # Attn # Shared # Routed Relative Sequence Batch Size Learning Size Heads Experts Experts Expert Size Length (Sequence) Rate 2.0B 9 1280 10 1 63 (7 activated) 0.25 2048 2048 1.08e-3 16.4B 28 2048 16 2 64 (6 activated) 0.25 4096 4608 4.2e-4 144.6B 62 4096 32 4 128 (12 activated) 0.125 4096 4608 3.0e-4\n\nTable 7 | Overview of hyper-parameters for DeepSeekMoE across various sizes. The relative expert size is in comparison to a standard FFN.\n\nB. Comparing Deepseekmoe With Larger Models\n\nComparisons among DeepSeekMoE, GShard×1.2, and GShard×1.5 are shown in Table 8. Comparisons among DeepSeekMoE, Dense×4, and Dense×16 are shown in Table 9.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 2.3B 2.8B 1.9B # Activated Expert Params N/A 0.28B 0.35B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.824 1.808 1.808 HellaSwag (Acc.) 0-shot 53.7 54.4 54.8 PIQA (Acc.) 0-shot 71.8 71.1 72.3 ARC-easy (Acc.) 0-shot 46.8 47.3 49.4 ARC-challenge (Acc.) 0-shot 31.7 34.1 34.3 RACE-middle (Acc.) 5-shot 43.7 46.4 44.0 RACE-high (Acc.) 5-shot 31.9 32.4 31.7 HumanEval (Pass@1) 0-shot 3.7 3.0 4.9 MBPP (Pass@1) 3-shot 2.4 2.6 2.2 TriviaQA (EM) 5-shot 15.2 15.7 16.6 NaturalQuestions (EM) 5-shot 4.5 4.7 5.7\n\nTable 8 | Comparison between DeepSeekMoE and larger GShard models.\n\nAt a larger scale of 13B total parameters, we also compare DeepSeekMoE with GShard×1.2 and GShard×1.5, and show results in Table 10. At a larger scale, DeepSeekMoE even outperforms GShard×1.5 distinctly.\n\nMetric # Shot Dense×4 Dense×16 DeepSeekMoE Relative Expert Size N/A 1 1 0.25 # Experts N/A 4 + 0 16 + 0 1 + 63 # Activated Experts N/A 4 + 0 16 + 0 1 + 7 # Total Expert Params N/A 0.47B 1.89B 1.89B # Activated Expert Params N/A 0.47B 1.89B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.908 1.806 1.808 HellaSwag (Acc.) 0-shot 47.6 55.1 54.8 PIQA (Acc.) 0-shot 70.0 71.9 72.3 ARC-easy (Acc.) 0-shot 43.9 51.9 49.4 ARC-challenge (Acc.) 0-shot 30.5 33.8 34.3 RACE-middle (Acc.) 5-shot 42.4 46.3 44.0 RACE-high (Acc.) 5-shot 30.7 33.0 31.7 HumanEval (Pass@1) 0-shot 1.8 4.3 4.9 MBPP (Pass@1) 3-shot 0.2 2.2 2.2 TriviaQA (EM) 5-shot 9.9 16.5 16.6 NaturalQuestions (EM) 5-shot 3.0 6.3 5.7\n\nTable 9 | Comparison between DeepSeekMoE and larger dense baselines.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 15.9B 19.8B 13.3B # Activated Expert Params N/A 2.37B 2.82B 2.05B # Training Tokens N/A 100B 100B 100B HellaSwag (Acc.) 0-shot 66.6 67.7 69.1 PIQA (Acc.) 0-shot 75.6 76.0 75.7 ARC-easy (Acc.) 0-shot 56.8 56.8 58.8 ARC-challenge (Acc.) 0-shot 39.9 37.6 38.5 RACE-middle (Acc.) 5-shot 51.6 50.6 52.4 RACE-high (Acc.) 5-shot 37.4 36.3 38.5 HumanEval (Pass@1) 0-shot 6.1 6.1 9.8 MBPP (Pass@1) 3-shot 7.0 11.6 10.6 TriviaQA (EM) 5-shot 36.5 36.7 38.2 NaturalQuestions (EM) 5-shot 12.6 12.1 13.7\n\nTable 10 | Comparison between DeepSeekMoE and larger GShard models at a larger scale.\n\nC. Training Benchmark Curves Of Deepseekmoe 16B\n\nWe present the benchmark curves during training of DeepSeekMoE 16B and DeepSeek 7B (Dense) in Figure 7 for reference.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/DeepSeekMoE_2.md"
+          },
+          "summary": "DeepSeekMoE is an innovative Mixture-of-Experts (MoE) architecture designed to enhance expert specialization in language models. It employs fine-grained expert segmentation and shared expert isolation to improve performance and reduce computational costs. Starting with a model of 2B parameters, DeepSeekMoE demonstrates comparable performance to larger models like GShard 2.9B and LLaMA2 7B, using significantly fewer computations. The architecture scales up to 16B and 145B parameters, maintaining advantages over traditional MoE architectures. DeepSeekMoE 16B achieves similar results to dense models with only 40% of the computations, and its chat model version shows adaptability in bilingual scenarios. The architecture's public release aims to contribute to advancements in large-scale language models.",
+          "headlines": {
+            "1. Introduction": [],
+            "2. Preliminaries: Mixture-Of-Experts For Transformers": [],
+            "3. Deepseekmoe Architecture": [
+              "3.1. Fine-Grained Expert Segmentation",
+              "3.2. Shared Expert Isolation",
+              "3.3. Load Balance Consideration"
+            ],
+            "4. Validation Experiments": [
+              "4.1. Experimental Setup",
+              "4.1.1. Training Data And Tokenization",
+              "4.1.2. Infrastructures",
+              "4.1.3. Hyper-Parameters",
+              "4.1.4. Evaluation Benchmarks",
+              "4.2. Evaluations",
+              "4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models",
+              "4.4. Ablation Studies",
+              "4.5. Analysis On Expert Specialization"
+            ],
+            "5. Scaling Up To Deepseekmoe 16B": [
+              "5.1. Experimental Setup",
+              "5.1.1. Training Data And Tokenization",
+              "5.1.2. Hyper-Parameters",
+              "5.1.3. Evaluation Benchmarks",
+              "5.2. Evaluations",
+              "5.2.1. Internal Comparison With Deepseek 7B",
+              "5.2.2. Comparison With Open Source Models"
+            ],
+            "6. Alignment For Deepseekmoe 16B": [
+              "6.1. Experimental Setup",
+              "6.2. Evaluations"
+            ],
+            "7. Deepseekmoe 145B Ongoing": [
+              "7.1. Experimental Setup",
+              "7.2. Evaluations"
+            ],
+            "8. Related Work": [],
+            "9. Conclusion": [],
+            "Appendices": [
+              "A. Overview Of Hyper-Parameters",
+              "B. Comparing Deepseekmoe With Larger Models",
+              "C. Training Benchmark Curves Of Deepseekmoe 16B"
+            ]
+          },
+          "summary_embedding": [
+            -0.012465757317841053,
+            -0.022808987647294998,
+            -0.018792515620589256,
+            -0.02308654971420765,
+            0.01797616109251976,
+            0.007200260180979967,
+            0.006490030791610479,
+            0.042613785713911057,
+            -0.014049487188458443,
+            0.03187054023146629,
+            -0.021486490964889526,
+            -0.023592690005898476,
+            0.009045224636793137,
+            -0.014441338367760181,
+            0.005865518003702164,
+            -0.007363531738519669,
+            -0.008743173442780972,
+            -0.005163452122360468,
+            0.018841497600078583,
+            -0.03100520372390747,
+            0.011110605672001839,
+            -0.046075135469436646,
+            -0.020017050206661224,
+            -0.021845687180757523,
+            0.016425084322690964,
+            0.037389107048511505,
+            0.022547753527760506,
+            -0.0028062239289283752,
+            -0.06844329088926315,
+            0.034678805619478226,
+            0.035168617963790894,
+            0.02052319049835205,
+            0.0005438972148112953,
+            -0.00031302153365686536,
+            -0.024833550676703453,
+            -0.028556134551763535,
+            0.03347059711813927,
+            -0.004379750229418278,
+            -0.01453930139541626,
+            0.015976088121533394,
+            0.0414055772125721,
+            0.012106560170650482,
+            -0.02754385396838188,
+            0.0014245413476601243,
+            0.015151568688452244,
+            0.0055961208418011665,
+            -0.01678428053855896,
+            -0.0022960014175623655,
+            -0.002424577483907342,
+            -0.021225256845355034,
+            -0.01400050614029169,
+            0.018825169652700424,
+            0.001296985661610961,
+            -0.008865626528859138,
+            -0.0542386956512928,
+            0.007330877240747213,
+            0.03840138763189316,
+            0.05623060464859009,
+            0.044148534536361694,
+            -0.020180322229862213,
+            0.03369917720556259,
+            -0.030025575309991837,
+            -0.02501314878463745,
+            0.011028970591723919,
+            0.003983817528933287,
+            -0.018531281501054764,
+            0.01784554310142994,
+            -0.005636938381940126,
+            -0.0006556359585374594,
+            0.01751900091767311,
+            -0.03941367194056511,
+            0.020751770585775375,
+            -0.01887415163218975,
+            0.031266435980796814,
+            0.02173139713704586,
+            0.0073798587545752525,
+            0.01872720755636692,
+            0.01678428053855896,
+            0.0036858476232737303,
+            -0.04107903689146042,
+            0.009118696674704552,
+            0.016523046419024467,
+            0.014196431264281273,
+            0.016269976273179054,
+            0.06553706526756287,
+            -0.05391215533018112,
+            -0.004485876299440861,
+            -0.05169166624546051,
+            0.0012561678886413574,
+            -0.004971608519554138,
+            -0.03670336678624153,
+            0.01048201136291027,
+            0.028425518423318863,
+            0.021372200921177864,
+            -0.01063711941242218,
+            0.024392718449234962,
+            -0.022743679583072662,
+            0.0032001156359910965,
+            -0.04062187671661377,
+            0.0019653772469609976,
+            0.004518530797213316,
+            0.016531210392713547,
+            0.0070778070949018,
+            0.004273623693734407,
+            -0.016702644526958466,
+            0.006045116577297449,
+            -0.01905374974012375,
+            0.03291547670960426,
+            -0.008465612307190895,
+            0.08653374016284943,
+            0.008751336485147476,
+            0.004738946910947561,
+            0.004281787667423487,
+            0.02398454025387764,
+            0.012131051160395145,
+            -0.00810233410447836,
+            -0.04718538001179695,
+            -0.015690363943576813,
+            0.011061624623835087,
+            -0.0383034273982048,
+            0.018335357308387756,
+            -0.008759500458836555,
+            0.01282495353370905,
+            -0.04107903689146042,
+            -0.06344719231128693,
+            0.01498013362288475,
+            0.001784758409485221,
+            -0.004065453074872494,
+            -0.0002903166168835014,
+            -0.018923133611679077,
+            0.003930754493921995,
+            -0.004200151655822992,
+            -0.010645282454788685,
+            -0.021502818912267685,
+            0.03159298002719879,
+            0.03657275065779686,
+            0.06527583301067352,
+            0.036801330745220184,
+            -0.0030164355412125587,
+            0.005775718949735165,
+            0.019690508022904396,
+            -0.020784424617886543,
+            0.003481758525595069,
+            -0.01214737817645073,
+            -0.021992631256580353,
+            -0.03820546343922615,
+            0.010971825569868088,
+            0.07379858940839767,
+            -0.029029620811343193,
+            0.013437220826745033,
+            -0.017616963014006615,
+            0.04205866530537605,
+            -0.004567512311041355,
+            0.026727497577667236,
+            -0.016669990494847298,
+            0.004059330560266972,
+            -0.025388672947883606,
+            0.01620466820895672,
+            -0.024964168667793274,
+            0.012571883387863636,
+            -0.000885746325366199,
+            -0.04222193360328674,
+            0.00011116473615402356,
+            0.026890767738223076,
+            -0.010971825569868088,
+            0.02261306345462799,
+            0.0004949158756062388,
+            -0.006302268709987402,
+            -0.05097327008843422,
+            0.038727931678295135,
+            -0.003912386484444141,
+            0.018645571544766426,
+            -0.0018959869630634785,
+            0.0163107942789793,
+            -0.0016408757073804736,
+            0.013216804713010788,
+            0.009543201886117458,
+            0.007114543113857508,
+            -0.033503253012895584,
+            0.02697240374982357,
+            -0.02504580281674862,
+            0.028049994260072708,
+            0.03435226157307625,
+            -0.0030184765346348286,
+            0.03255628049373627,
+            0.019249675795435905,
+            0.03840138763189316,
+            0.0327032245695591,
+            0.030335791409015656,
+            -0.00824927818030119,
+            0.013878053054213524,
+            0.008200296200811863,
+            0.03608293831348419,
+            0.013429056853055954,
+            0.019723162055015564,
+            -0.009730963967740536,
+            -0.014041324146091938,
+            0.048785436898469925,
+            0.007318632211536169,
+            -0.0179598331451416,
+            -0.005441013257950544,
+            -0.03510330989956856,
+            -0.002832755446434021,
+            -0.014425011351704597,
+            0.0323440283536911,
+            0.02158445306122303,
+            -0.01647406443953514,
+            0.035135965794324875,
+            0.024980494752526283,
+            -0.03751972317695618,
+            -0.005232842173427343,
+            -0.02026195637881756,
+            -0.023298801854252815,
+            -0.04215662553906441,
+            0.011624909937381744,
+            0.012833117507398129,
+            -0.013976015150547028,
+            -0.007469657808542252,
+            0.020457882434129715,
+            -0.023625344038009644,
+            -0.0018265966791659594,
+            -0.01356783788651228,
+            0.006547175347805023,
+            -0.020898714661598206,
+            -0.03667071461677551,
+            -0.01048201136291027,
+            0.025862159207463264,
+            0.012637191452085972,
+            0.04362606629729271,
+            -0.013518855907022953,
+            -0.005367541220039129,
+            0.033193036913871765,
+            0.012759645469486713,
+            0.010767736472189426,
+            0.03699725493788719,
+            -0.03027048148214817,
+            0.024621298536658287,
+            -0.020294612273573875,
+            -0.0032878739293664694,
+            0.01707816869020462,
+            -0.0030062312725931406,
+            0.01962519995868206,
+            -0.04535674303770065,
+            -0.024196794256567955,
+            0.04395261034369469,
+            -0.0007107399869710207,
+            0.016302630305290222,
+            -0.005959399044513702,
+            -0.012106560170650482,
+            0.0035899258218705654,
+            0.0027715289033949375,
+            -0.01438419334590435,
+            -0.008620720356702805,
+            0.007473739795386791,
+            -0.01962519995868206,
+            0.03660540655255318,
+            0.03794422745704651,
+            -0.004132802598178387,
+            -0.014131123200058937,
+            -0.03324201703071594,
+            0.010016689077019691,
+            -0.017878197133541107,
+            -0.023951886221766472,
+            0.04078514873981476,
+            -0.032115448266267776,
+            -0.00029312283731997013,
+            0.039478980004787445,
+            0.018808843567967415,
+            -0.014898497611284256,
+            0.00868602842092514,
+            -0.01709449477493763,
+            -0.026302991434931755,
+            -0.017159804701805115,
+            0.03356856107711792,
+            0.024800896644592285,
+            -0.010743245482444763,
+            0.010245268233120441,
+            0.005330805201083422,
+            0.02893165871500969,
+            0.024376392364501953,
+            0.00648594880476594,
+            -0.013812744058668613,
+            0.015029114671051502,
+            0.027641816064715385,
+            -0.0028021421749144793,
+            -7.66609373386018e-05,
+            -0.004010349046438932,
+            -0.007796200457960367,
+            -0.006453294772654772,
+            -0.013069860637187958,
+            -0.005089980084449053,
+            0.0009184005321003497,
+            -0.032295044511556625,
+            0.015967924147844315,
+            -0.005890008993446827,
+            0.010359558276832104,
+            -0.011535110883414745,
+            0.01288209855556488,
+            0.01722511276602745,
+            -0.013494365848600864,
+            -0.0716434046626091,
+            0.014612773433327675,
+            0.02458864450454712,
+            -0.006706364918500185,
+            -0.017943505197763443,
+            0.008481939323246479,
+            0.00016148542636074126,
+            0.038597315549850464,
+            -0.014180104248225689,
+            -0.031103165820240974,
+            0.02680913358926773,
+            -0.017437364906072617,
+            -0.015241367742419243,
+            -0.010661610402166843,
+            -0.00580837344750762,
+            0.04649963974952698,
+            0.01889047957956791,
+            0.011420820839703083,
+            -0.04042595252394676,
+            -0.007449248805642128,
+            0.015078096650540829,
+            0.014506647363305092,
+            -0.04705476388335228,
+            0.0052083516493439674,
+            0.015486273914575577,
+            0.019723162055015564,
+            0.024800896644592285,
+            0.04274440184235573,
+            -0.0021531390957534313,
+            0.01661284640431404,
+            -0.015053605660796165,
+            -0.05136512219905853,
+            -0.03340528905391693,
+            0.029943939298391342,
+            -0.017731253057718277,
+            -0.02906227484345436,
+            -0.01693122461438179,
+            -0.006151242647320032,
+            0.0017939424142241478,
+            0.007126788143068552,
+            -0.01916803978383541,
+            -0.03464614972472191,
+            -0.03970756009221077,
+            0.02277633361518383,
+            0.007983962073922157,
+            0.004269542172551155,
+            -0.042777057737112045,
+            0.0013459670590236783,
+            0.009184005670249462,
+            0.012073906138539314,
+            0.0009954441338777542,
+            -0.019984396174550056,
+            0.015494437888264656,
+            0.0032001156359910965,
+            -0.031070511788129807,
+            -0.035593122243881226,
+            -0.03595232218503952,
+            0.0023919232189655304,
+            0.003781769424676895,
+            0.06279410421848297,
+            0.012988224625587463,
+            -0.03840138763189316,
+            -0.04522612690925598,
+            0.029176564887166023,
+            -0.02052319049835205,
+            -0.016678154468536377,
+            0.0059430720284581184,
+            0.012898425571620464,
+            -0.009877908043563366,
+            0.013192313723266125,
+            0.023560035973787308,
+            0.041601505130529404,
+            -0.005987971555441618,
+            0.012914752587676048,
+            0.023037567734718323,
+            0.012955570593476295,
+            0.0362788625061512,
+            -0.00936360377818346,
+            0.0035409443080425262,
+            0.039936136454343796,
+            -0.012269831262528896,
+            0.03175625205039978,
+            -0.02189466916024685,
+            0.0020398697815835476,
+            0.02249877341091633,
+            0.027184655889868736,
+            -0.0006479826406575739,
+            0.02594379521906376,
+            -0.031543996185064316,
+            0.018368011340498924,
+            -0.014498483389616013,
+            -0.04888340085744858,
+            -0.01557607389986515,
+            0.013102514669299126,
+            0.030384771525859833,
+            0.02756018005311489,
+            -0.0041348435916006565,
+            -0.019723162055015564,
+            -0.01155143789947033,
+            -0.010596301406621933,
+            0.0018419033149257302,
+            0.03285016864538193,
+            0.010506502352654934,
+            0.010890189558267593,
+            -0.011935125105082989,
+            -0.004104230087250471,
+            0.03025415539741516,
+            -0.021192602813243866,
+            0.01650671847164631,
+            0.0041511706076562405,
+            -0.002029665280133486,
+            -0.04320156201720238,
+            -0.035266581922769547,
+            0.007845181971788406,
+            0.011747363954782486,
+            0.014874007552862167,
+            0.0368339866399765,
+            0.025094784796237946,
+            -0.01662100851535797,
+            0.030629679560661316,
+            -0.01905374974012375,
+            0.027984686195850372,
+            0.039054471999406815,
+            0.011624909937381744,
+            -0.018351683393120766,
+            0.012237177230417728,
+            0.002424577483907342,
+            -0.025127438828349113,
+            0.01680060662329197,
+            -0.032735876739025116,
+            0.01738838478922844,
+            -0.02354370802640915,
+            0.025143766775727272,
+            -0.02579685114324093,
+            -0.009290131740272045,
+            -0.004428731743246317,
+            -0.006130834110081196,
+            0.04888340085744858,
+            -0.026727497577667236,
+            -0.0142454132437706,
+            -0.0024694770108908415,
+            -0.0024531499948352575,
+            -0.008718682453036308,
+            0.010000362060964108,
+            -0.045781247317790985,
+            -0.01844964735209942,
+            -0.04016471654176712,
+            0.007355368230491877,
+            0.005412440747022629,
+            -0.030907239764928818,
+            -0.05694899708032608,
+            0.025421326979994774,
+            -0.06331657618284225,
+            -0.019886434078216553,
+            0.051626358181238174,
+            -0.018400665372610092,
+            -0.008212542161345482,
+            0.009641164913773537,
+            0.006375740747898817,
+            -0.029290854930877686,
+            0.0041348435916006565,
+            0.02504580281674862,
+            0.04418119043111801,
+            -0.0045797573402523994,
+            -0.02865409664809704,
+            0.0028674507047981024,
+            -0.027788760140538216,
+            -0.013461710885167122,
+            0.005261414684355259,
+            0.016449574381113052,
+            0.024539662525057793,
+            0.007628847379237413,
+            0.03585435822606087,
+            0.007294141221791506,
+            0.006318595726042986,
+            0.044279150664806366,
+            -0.025666235014796257,
+            -0.027380581945180893,
+            0.02519274689257145,
+            0.04045860469341278,
+            0.023151857778429985,
+            -0.018253721296787262,
+            -0.0142454132437706,
+            0.0053063142113387585,
+            0.0009296254720538855,
+            -0.022139575332403183,
+            0.002808264922350645,
+            -0.013902543112635612,
+            0.005143043119460344,
+            0.03892385587096214,
+            -0.001205145614221692,
+            0.014196431264281273,
+            0.007302304729819298,
+            0.006845145486295223,
+            0.004334850702434778,
+            -0.011731036007404327,
+            -0.023592690005898476,
+            0.00032220553839579225,
+            0.034972693771123886,
+            0.015404638834297657,
+            -0.030515389516949654,
+            -0.00995954405516386,
+            0.034711457788944244,
+            -0.00795130804181099,
+            -0.009118696674704552,
+            -0.009771781973540783,
+            0.003385836724191904,
+            0.02638462744653225,
+            -0.01606588624417782,
+            0.006649219896644354,
+            0.03716052696108818,
+            -0.019870106130838394,
+            0.005543057806789875,
+            -0.025405000895261765,
+            0.0018061877926811576,
+            0.02442537248134613,
+            -0.006216551177203655,
+            0.028098974376916885,
+            5.6315813708351925e-05,
+            0.03025415539741516,
+            -0.004677720367908478,
+            0.008432958275079727,
+            -0.020327266305685043,
+            0.03987082839012146,
+            -0.017633290961384773,
+            0.03402572125196457,
+            0.02067013457417488,
+            -0.01258004643023014,
+            0.04783846437931061,
+            -0.036801330745220184,
+            -0.020784424617886543,
+            0.00584919098764658,
+            0.012743317522108555,
+            0.0028919412288814783,
+            -0.0191517136991024,
+            -0.018400665372610092,
+            -0.010996315628290176,
+            0.02380494214594364,
+            -0.022547753527760506,
+            -0.016865916550159454,
+            -0.007441085297614336,
+            -0.01890680566430092,
+            -0.009673818945884705,
+            -0.007988044060766697,
+            -0.01709449477493763,
+            -0.016547536477446556,
+            0.044573038816452026,
+            -0.06334923207759857,
+            0.016996532678604126,
+            0.017159804701805115,
+            0.011477965861558914,
+            -0.034678805619478226,
+            -0.019510909914970398,
+            -0.017633290961384773,
+            -0.002347023691982031,
+            -0.030743969604372978,
+            0.001702102366834879,
+            0.01725776679813862,
+            0.01138000376522541,
+            -0.05022222548723221,
+            0.012335140258073807,
+            0.02772345207631588,
+            -0.012065742164850235,
+            0.0014949520118534565,
+            -0.023135529831051826,
+            -0.017176130786538124,
+            0.0031776658724993467,
+            0.022123249247670174,
+            0.005975726060569286,
+            -0.008188051171600819,
+            -0.016865916550159454,
+            -0.005241005681455135,
+            -0.006045116577297449,
+            -0.002359268954023719,
+            0.005094061605632305,
+            0.011780017986893654,
+            0.019690508022904396,
+            -0.0024633544962853193,
+            -0.004436895251274109,
+            -0.006396149750798941,
+            0.027364253997802734,
+            -0.001450052484869957,
+            -0.008416631259024143,
+            0.050907962024211884,
+            0.03556047007441521,
+            0.006094098091125488,
+            -0.01108611561357975,
+            -0.03990348428487778,
+            0.010661610402166843,
+            -0.022711025550961494,
+            0.0401320643723011,
+            0.01905374974012375,
+            -0.008298259228467941,
+            -0.028866350650787354,
+            0.010294250212609768,
+            -0.02290695160627365,
+            -0.009975871071219444,
+            0.009469729848206043,
+            0.00726556871086359,
+            -0.03263791650533676,
+            -0.013543346896767616,
+            0.03510330989956856,
+            0.011437147855758667,
+            -0.006841063499450684,
+            -0.036050282418727875,
+            0.008588065393269062,
+            0.01646590232849121,
+            0.011869817040860653,
+            -0.019233347848057747,
+            0.004449140280485153,
+            -0.012041252106428146,
+            0.017470018938183784,
+            0.008849299512803555,
+            -0.017796561121940613,
+            0.018237393349409103,
+            0.004710374400019646,
+            -0.008490103296935558,
+            -0.003912386484444141,
+            -0.0163107942789793,
+            -0.02006603218615055,
+            0.03840138763189316,
+            0.024490682408213615,
+            0.026335645467042923,
+            0.005457340274006128,
+            0.02008235827088356,
+            0.03227871656417847,
+            0.011282040737569332,
+            -0.03134807199239731,
+            0.027935704216361046,
+            0.016996532678604126,
+            -0.04604247957468033,
+            0.021551799029111862,
+            -0.02218855731189251,
+            -0.028670424595475197,
+            0.009649328887462616,
+            -0.0017367975087836385,
+            0.011192241683602333,
+            0.0034837995190173388,
+            -0.003642988856881857,
+            -0.03307874873280525,
+            -0.006236960180103779,
+            0.015771998092532158,
+            0.015502601861953735,
+            -0.0048695639707148075,
+            -0.003296037670224905,
+            -0.011886144056916237,
+            0.00487364549189806,
+            0.009796272963285446,
+            0.039054471999406815,
+            0.001381682581268251,
+            -0.004751192405819893,
+            -0.0018765984568744898,
+            0.031854212284088135,
+            0.0271193478256464,
+            -0.017274094745516777,
+            -0.026286665350198746,
+            0.00580837344750762,
+            -0.014253576286137104,
+            0.026156047359108925,
+            -0.00487364549189806,
+            0.011877980083227158,
+            0.008718682453036308,
+            -0.0007908449042588472,
+            -0.001373519073240459,
+            0.025584599003195763,
+            -0.02710302174091339,
+            -0.021764053031802177,
+            0.031233781948685646,
+            -0.025731543079018593,
+            0.028735732659697533,
+            0.013061696663498878,
+            -0.01557607389986515,
+            -0.015225040726363659,
+            -0.010302413254976273,
+            0.009943217039108276,
+            -0.012277995236217976,
+            0.023853924125432968,
+            0.003608293831348419,
+            0.006392067763954401,
+            0.0003584313380997628,
+            0.02770712412893772,
+            0.024833550676703453,
+            0.007820690982043743,
+            -0.018074123188853264,
+            -0.0016816934803500772,
+            -0.02320083975791931,
+            -0.01680060662329197,
+            0.023772288113832474,
+            -0.018090449273586273,
+            -0.03268689662218094,
+            -0.028556134551763535,
+            0.012114724144339561,
+            0.007392104249447584,
+            0.010816717520356178,
+            0.03461349755525589,
+            -0.030384771525859833,
+            -0.0344502255320549,
+            0.01766594499349594,
+            0.026270337402820587,
+            -0.005551221314817667,
+            -0.021535472944378853,
+            0.03085825778543949,
+            -0.005828781984746456,
+            0.017600636929273605,
+            0.0404912605881691,
+            -0.0016225076979026198,
+            0.010228941217064857,
+            -0.042026009410619736,
+            -0.014057651162147522,
+            -0.0066614653915166855,
+            0.006261450704187155,
+            0.034417569637298584,
+            0.01048201136291027,
+            0.01535565685480833,
+            0.03448288142681122,
+            -0.021192602813243866,
+            -0.0002903166168835014,
+            -0.0022449791431427,
+            -0.02818061038851738,
+            0.03347059711813927,
+            0.025715215131640434,
+            0.011918798089027405,
+            0.019706835970282555,
+            -0.03794422745704651,
+            -0.013037205673754215,
+            0.027070365846157074,
+            0.015053605660796165,
+            0.01273515447974205,
+            0.016694480553269386,
+            0.011249386705458164,
+            0.00868602842092514,
+            -0.021633435040712357,
+            -0.05169166624546051,
+            0.01214737817645073,
+            0.01993541419506073,
+            -0.016449574381113052,
+            0.01057181041687727,
+            0.024980494752526283,
+            0.007620683871209621,
+            -0.007045152597129345,
+            0.012808626517653465,
+            -0.02682545967400074,
+            0.002259265398606658,
+            0.0019010890973731875,
+            -0.014482156373560429,
+            0.038270771503448486,
+            -0.016367938369512558,
+            0.011363675817847252,
+            -0.018204739317297935,
+            0.023870250210165977,
+            0.00011531029304023832,
+            0.0007964573451317847,
+            0.014963806606829166,
+            -0.011347348801791668,
+            -0.04235255345702171,
+            0.0021613026037812233,
+            0.03342161700129509,
+            -0.015200549736618996,
+            0.026449935510754585,
+            -0.0020674217958003283,
+            0.032458316534757614,
+            0.042777057737112045,
+            0.017731253057718277,
+            0.0003497575526125729,
+            0.025682561099529266,
+            0.0036144163459539413,
+            0.0073880222626030445,
+            0.012400448322296143,
+            0.009322785772383213,
+            0.007245160173624754,
+            0.030809277668595314,
+            -0.010261595249176025,
+            0.02173139713704586,
+            -0.04760988429188728,
+            -0.014629100449383259,
+            0.012024925090372562,
+            -0.0013316808035597205,
+            -0.012449429370462894,
+            -0.033029764890670776,
+            0.012563719414174557,
+            -0.03503800183534622,
+            0.022074267268180847,
+            -0.0004959363141097128,
+            0.04333217814564705,
+            -0.02187834307551384,
+            -0.005518566817045212,
+            0.016694480553269386,
+            0.0016510800924152136,
+            0.023282473906874657,
+            -0.0012153501156717539,
+            -0.016416920349001884,
+            -0.0067798369564116,
+            -0.009257477708160877,
+            0.004138925112783909,
+            0.04375668615102768,
+            -0.003722583642229438,
+            -0.0001682458823779598,
+            0.01409030519425869,
+            -0.011959616094827652,
+            -0.03250729665160179,
+            0.045193471014499664,
+            -0.0030837850645184517,
+            0.02484987862408161,
+            0.04979771748185158,
+            -0.06410028040409088,
+            -0.019543563947081566,
+            0.013061696663498878,
+            -0.010441194288432598,
+            0.011363675817847252,
+            0.005547139327973127,
+            0.016114868223667145,
+            0.017894525080919266,
+            -0.005485912784934044,
+            0.01619650423526764,
+            0.005106307100504637,
+            0.028098974376916885,
+            -0.02336410991847515,
+            -0.03296445682644844,
+            -0.02636830136179924,
+            -0.017747581005096436,
+            0.0014480114914476871,
+            -0.015796490013599396,
+            -3.563585551091819e-06,
+            -0.0022204883862286806,
+            -0.04388730227947235,
+            -0.025274382904171944,
+            0.040687184780836105,
+            0.006706364918500185,
+            0.02458864450454712,
+            0.0077268099412322044,
+            0.03418898954987526,
+            0.005885927006602287,
+            -0.009037061594426632,
+            0.030058229342103004,
+            -0.020049704238772392,
+            0.03833607956767082,
+            0.022106921300292015,
+            -0.002832755446434021,
+            0.005865518003702164,
+            0.031086837872862816,
+            -0.033797141164541245,
+            0.008086007088422775,
+            -0.034384917467832565,
+            0.0046695563942193985,
+            -0.008620720356702805,
+            -0.015094423666596413,
+            -0.009526874870061874,
+            -0.02504580281674862,
+            0.011216731742024422,
+            -0.021535472944378853,
+            0.01590261608362198,
+            0.010416703298687935,
+            -0.002916431985795498,
+            0.03448288142681122,
+            -0.014204595237970352,
+            -0.0027796924114227295,
+            0.0024531499948352575,
+            -0.027070365846157074,
+            -0.006343086715787649,
+            -0.003377673216164112,
+            -0.03689929470419884,
+            -0.004502203781157732,
+            0.010988152585923672,
+            -0.00024184548237826675,
+            0.016898570582270622,
+            0.02787039615213871,
+            0.0005092020728625357,
+            -0.022302847355604172,
+            0.002340900944545865,
+            -0.04199335351586342,
+            0.0011877981014549732,
+            0.01483318954706192,
+            -0.017600636929273605,
+            -0.008612556383013725,
+            0.033046092838048935,
+            0.023298801854252815,
+            -0.009396257810294628,
+            -0.035005345940589905,
+            0.017796561121940613,
+            -0.011853490024805069,
+            0.027511198073625565,
+            -0.0002630197268445045,
+            0.019298657774925232,
+            -0.0076941559091210365,
+            0.04493223875761032,
+            0.001369437319226563,
+            0.008645210415124893,
+            -0.021959977224469185,
+            0.007020662073045969,
+            -0.020278284326195717,
+            -0.002406209474429488,
+            -0.006171651650220156,
+            -0.012751481495797634,
+            -0.010669773444533348,
+            0.0032735876739025116,
+            0.007730891928076744,
+            -0.0176822729408741,
+            0.0055961208418011665,
+            -0.015959760174155235,
+            -0.02741323597729206,
+            0.007543129846453667,
+            -0.05965929850935936,
+            -0.0179598331451416,
+            -0.03373183310031891,
+            -0.009477893821895123,
+            0.02008235827088356,
+            0.03510330989956856,
+            -0.012759645469486713,
+            0.00497977202758193,
+            -0.014898497611284256,
+            0.004477712791413069,
+            0.03490738570690155,
+            0.044442422688007355,
+            -0.014759717509150505,
+            0.003381754970178008,
+            -0.010955498553812504,
+            0.033029764890670776,
+            -0.03347059711813927,
+            0.0030062312725931406,
+            0.031086837872862816,
+            -0.005057325586676598,
+            0.007959471084177494,
+            0.011624909937381744,
+            0.020474210381507874,
+            -0.019396619871258736,
+            0.00877582747489214,
+            0.002100075827911496,
+            0.019233347848057747,
+            0.02171507105231285,
+            0.020343592390418053,
+            -0.03539719805121422,
+            0.028409190475940704,
+            0.018384337425231934,
+            0.02444170042872429,
+            0.044442422688007355,
+            -0.03974021226167679,
+            -0.004147088620811701,
+            -0.0032225653994828463,
+            -0.02997659333050251,
+            0.02008235827088356,
+            -0.007530884351581335,
+            -0.01841699331998825,
+            -0.007069643586874008,
+            0.03327467292547226,
+            0.014571955427527428,
+            -0.05051611363887787,
+            0.01980479806661606,
+            0.01740471087396145,
+            0.02055584453046322,
+            0.024474354460835457,
+            -0.01123305968940258,
+            -0.0019072118448093534,
+            0.008996243588626385,
+            0.006804327480494976,
+            0.019266001880168915,
+            0.015257694758474827,
+            0.002012317767366767,
+            -0.0022082431241869926,
+            -0.011837163008749485,
+            0.005224678665399551,
+            -0.009077879600226879,
+            0.010808554477989674,
+            -0.02458864450454712,
+            -0.009698309935629368,
+            0.021388528868556023,
+            0.0031164393294602633,
+            -0.013665799982845783,
+            -0.03729114308953285,
+            -0.045944519340991974,
+            0.0033837957307696342,
+            0.004543021321296692,
+            -0.05365091934800148,
+            0.005322641227394342,
+            -0.026547899469733238,
+            -0.002108239568769932,
+            -0.005502239800989628,
+            0.0019214981002733111,
+            0.005636938381940126,
+            0.0167679525911808,
+            -0.013494365848600864,
+            0.02248244546353817,
+            -0.015502601861953735,
+            0.03165828809142113,
+            0.0055961208418011665,
+            0.007375776767730713,
+            -0.0027796924114227295,
+            0.05590406060218811,
+            -0.003712379140779376,
+            -0.03699725493788719,
+            0.023282473906874657,
+            -0.01311067771166563,
+            -0.0001747512142173946,
+            0.030090883374214172,
+            0.00884113647043705,
+            -0.015257694758474827,
+            0.013437220826745033,
+            -0.030613351613283157,
+            -0.03249097242951393,
+            0.010996315628290176,
+            -0.013755599036812782,
+            -0.0015776080545037985,
+            -0.04568328335881233,
+            -0.009339112788438797,
+            0.004269542172551155,
+            -0.018335357308387756,
+            0.010775899514555931,
+            0.01631895825266838,
+            0.013992343097925186,
+            -0.024245774373412132,
+            0.009053388610482216,
+            0.006865554489195347,
+            0.013453547842800617,
+            0.0007331897504627705,
+            0.014049487188458443,
+            -0.008677864447236061,
+            -0.011306531727313995,
+            -0.018319029361009598,
+            0.0032450153958052397,
+            -0.026792805641889572,
+            0.005155288614332676,
+            -0.003155216109007597,
+            -0.03771565109491348,
+            -0.06521052122116089,
+            -0.0344502255320549,
+            0.016294466331601143,
+            0.03396041318774223,
+            -0.009045224636793137,
+            0.0035144127905368805,
+            -0.00023648813657928258,
+            -0.01093917153775692,
+            0.024523336440324783,
+            -0.005930826533585787,
+            -0.0004829256213270128,
+            0.0035389035474509,
+            0.02050686441361904,
+            -0.009722800925374031,
+            0.005845109466463327,
+            -0.051952898502349854,
+            -0.01586179807782173,
+            0.01812310516834259,
+            0.006306350696831942,
+            0.014629100449383259,
+            -0.009690145961940289,
+            0.01693122461438179,
+            -0.030074557289481163,
+            -0.028147956356406212,
+            0.04130761697888374,
+            0.008277850225567818,
+            0.011486129835247993,
+            -0.003055212553590536,
+            -0.01680060662329197,
+            0.02504580281674862,
+            -0.0020633398089557886,
+            -0.006461458280682564,
+            -0.00817988719791174,
+            0.015820980072021484,
+            0.044279150664806366,
+            0.003255219664424658,
+            0.0009821783751249313,
+            0.017045514658093452,
+            -0.002663361607119441,
+            -0.011575928889214993,
+            0.007677828427404165,
+            0.013592327944934368,
+            -0.016384266316890717,
+            0.010375885292887688,
+            -0.025862159207463264,
+            0.0332583449780941,
+            0.03131541982293129,
+            0.011575928889214993,
+            -0.015045441687107086,
+            0.006130834110081196,
+            0.005330805201083422,
+            0.008086007088422775,
+            -0.001369437319226563,
+            -0.0017133272485807538,
+            0.020719116553664207,
+            -0.03276853263378143,
+            -0.01693122461438179,
+            -0.005347132217139006,
+            0.003932795487344265,
+            0.005277741700410843,
+            0.03396041318774223,
+            -0.0033266509417444468,
+            0.017927179113030434,
+            -0.011657563969492912,
+            0.010579974390566349,
+            0.021764053031802177,
+            0.037095218896865845,
+            -0.024996822699904442,
+            -0.029862305149435997,
+            0.012914752587676048,
+            -0.001296985661610961,
+            0.014065815135836601,
+            0.0010173837654292583,
+            0.014139287173748016,
+            0.0029409227427095175,
+            0.060475654900074005,
+            -0.017763907089829445,
+            0.004628738854080439,
+            0.0004035860183648765,
+            -0.01858026348054409,
+            -0.0013531101867556572,
+            0.004261378664523363,
+            -0.010979988612234592,
+            0.03608293831348419,
+            0.002344982698559761,
+            0.039185091853141785,
+            0.013918871060013771,
+            0.009290131740272045,
+            0.0015143405180424452,
+            -0.013592327944934368,
+            0.008355404250323772,
+            0.0015592400450259447,
+            -0.001132694073021412,
+            0.009461566805839539,
+            0.019298657774925232,
+            0.009714636951684952,
+            0.025274382904171944,
+            0.009788108989596367,
+            0.012857607565820217,
+            -0.029470453038811684,
+            0.03510330989956856,
+            -0.0006224715034477413,
+            0.012996388599276543,
+            -0.02040890045464039,
+            -0.003136848099529743,
+            -0.0008240094175562263,
+            0.015649545937776566,
+            -0.015616890974342823,
+            -0.001201063860207796,
+            -0.01297189760953188,
+            0.040099408477544785,
+            -0.018988441675901413,
+            0.013976015150547028,
+            0.009673818945884705,
+            -0.02865409664809704,
+            -0.007596192881464958,
+            0.01995174214243889,
+            0.014294394291937351,
+            -0.0053144777193665504,
+            -0.020490536466240883,
+            0.005832863971590996,
+            0.015380147844552994,
+            -0.028572462499141693,
+            1.3584674888988957e-05,
+            0.014751553535461426,
+            -0.0164903923869133,
+            -0.00025409081717953086,
+            -0.004702210891991854,
+            0.003975654020905495,
+            -0.009265640750527382,
+            0.007028825581073761,
+            0.016057724133133888,
+            -0.024507008492946625,
+            -0.020049704238772392,
+            0.005722655914723873,
+            0.006571666337549686,
+            0.01444950234144926,
+            0.007739055436104536,
+            0.004355259705334902,
+            -0.020033378154039383,
+            -0.01799248717725277,
+            0.0032674651592969894,
+            -0.01649855636060238,
+            -0.0048614004626870155,
+            0.0020531355403363705,
+            -0.019363965839147568,
+            0.01875986158847809,
+            -0.010743245482444763,
+            0.00017538899555802345,
+            -0.003226647386327386,
+            -0.020800752565264702,
+            0.006204306147992611,
+            -0.004865481983870268,
+            0.019870106130838394,
+            0.003555230563506484,
+            0.02785406820476055,
+            0.02310287579894066,
+            -0.004428731743246317,
+            0.017600636929273605,
+            -0.016155686229467392,
+            0.007983962073922157,
+            -0.031135819852352142,
+            -0.001782717532478273,
+            -0.010408539324998856,
+            0.0036103345919400454,
+            -0.015943434089422226,
+            -0.009126860648393631,
+            -0.0048777274787425995,
+            -0.0006678812787868083,
+            -0.004820582456886768,
+            0.0004275664687156677,
+            0.0027817331720143557,
+            0.02231917530298233,
+            -0.013208640739321709,
+            -0.02803366631269455,
+            -0.017437364906072617,
+            0.006045116577297449,
+            0.019396619871258736,
+            -0.026025431230664253,
+            -0.009184005670249462,
+            -0.005012426059693098,
+            -0.004326687194406986,
+            -0.021404854953289032,
+            -0.015167895704507828,
+            -0.005632856860756874,
+            -0.008045189082622528,
+            -0.010522829368710518,
+            0.0046532293781638145,
+            -0.0035715578123927116,
+            -0.011739199981093407,
+            0.021666089072823524,
+            0.018057795241475105,
+            -0.019837452098727226,
+            -0.019837452098727226,
+            0.013641309924423695,
+            -0.006159406621009111,
+            0.0018878233386203647,
+            0.02008235827088356,
+            0.00029286774224601686,
+            0.005824700463563204,
+            0.020882388576865196,
+            -0.0035225762985646725,
+            -0.01797616109251976,
+            0.003577680326998234,
+            -0.0018888438353314996,
+            -0.012122887186706066,
+            0.003212361130863428,
+            0.02773977816104889,
+            0.0008076822850853205,
+            0.023429417982697487,
+            -0.03056436963379383,
+            -0.011412657797336578,
+            0.006061443593353033,
+            0.02369065210223198,
+            0.00211844383738935,
+            -0.0021694661118090153,
+            0.020131340250372887,
+            0.03941367194056511,
+            0.005775718949735165,
+            0.0015755671774968505,
+            -0.04003410041332245,
+            0.01027792226523161,
+            0.009208496659994125,
+            0.024621298536658287,
+            0.008339077234268188,
+            0.0014653591206297278,
+            -0.012906589545309544,
+            -0.009755454957485199,
+            -0.0017939424142241478,
+            0.014702572487294674,
+            -0.023870250210165977,
+            0.0024755997583270073,
+            -0.03157665207982063,
+            0.005094061605632305,
+            0.0021959978621453047,
+            -0.009559528902173042,
+            0.0335359051823616,
+            0.00980443600565195,
+            -0.02065380848944187,
+            0.015151568688452244,
+            0.005767555441707373,
+            0.007330877240747213,
+            0.022449791431427002,
+            -0.008228869177401066,
+            0.030450081452727318,
+            -0.005049162078648806,
+            0.01213921420276165,
+            0.02742956392467022,
+            0.032588932663202286,
+            -0.013102514669299126,
+            -0.003959327004849911,
+            0.0006097159348428249,
+            -0.0002561317232903093,
+            0.0007995187188498676,
+            -0.008204378187656403,
+            0.0037389106582850218,
+            -0.014800535514950752,
+            0.012710663489997387,
+            -0.019543563947081566,
+            -0.0034286954905837774,
+            0.0038532004691660404,
+            0.01991908811032772,
+            0.0016265894519165158,
+            -0.033633869141340256,
+            0.02055584453046322,
+            -0.015420965850353241,
+            0.0135596739128232,
+            0.001290863030590117,
+            -0.011110605672001839,
+            -0.00029210239881649613,
+            0.020588500425219536,
+            0.010947334580123425,
+            -0.007845181971788406,
+            -0.0135596739128232,
+            -0.028376536443829536,
+            -0.021535472944378853,
+            0.00823703221976757,
+            0.009445239789783955,
+            -0.0028837777208536863,
+            0.0021306893322616816,
+            -0.011869817040860653,
+            0.0006520643946714699,
+            -0.0008122742874547839,
+            0.01740471087396145,
+            -0.0335359051823616,
+            0.021943651139736176,
+            -0.019527235999703407,
+            -0.019543563947081566,
+            0.017045514658093452,
+            0.008049271069467068,
+            0.009445239789783955,
+            0.0033552234526723623,
+            -0.01995174214243889,
+            -0.012326976284384727,
+            0.0338951013982296,
+            -0.01138000376522541,
+            0.006298186723142862,
+            0.004408322740346193,
+            0.012914752587676048,
+            -0.008914608508348465,
+            -0.011926962062716484,
+            0.01784554310142994,
+            -0.009861581027507782,
+            0.013951525092124939,
+            0.0010010566329583526,
+            -0.0254703089594841,
+            0.02739690989255905,
+            -0.019233347848057747,
+            0.03487472981214523,
+            -0.021617108955979347,
+            0.00711862463504076,
+            0.0066614653915166855,
+            0.008792154490947723,
+            -0.006690037902444601,
+            -0.013224967755377293,
+            0.0012786176521331072,
+            0.03807484731078148,
+            -0.02623768337070942,
+            0.024180466309189796,
+            0.022400809451937675,
+            -0.00040766780148260295,
+            0.02065380848944187,
+            -0.008367649279534817,
+            -0.026711169630289078,
+            0.0179598331451416,
+            -0.012596373446285725,
+            -0.026123393326997757,
+            0.013257622718811035,
+            -0.022678371518850327,
+            -0.020278284326195717,
+            -0.026302991434931755,
+            0.004947117529809475,
+            -0.0025694805663079023,
+            -0.005151206627488136,
+            0.009004407562315464,
+            0.02561725303530693,
+            0.020915042608976364,
+            0.019576217979192734,
+            0.02111096680164337,
+            0.01161674689501524,
+            0.022449791431427002,
+            -0.007559456862509251,
+            0.04401791840791702,
+            -0.004947117529809475,
+            0.004632820375263691,
+            -0.0015980169409886003,
+            -0.015657709911465645,
+            0.009771781973540783,
+            0.0119106350466609,
+            0.0029572497587651014,
+            -0.029176564887166023,
+            -0.009061552584171295,
+            -0.03660540655255318,
+            0.005053244065493345,
+            0.0020664012990891933,
+            0.0176822729408741,
+            0.032115448266267776,
+            -0.005865518003702164,
+            0.004592002835124731,
+            -0.005102225113660097,
+            -0.013927034102380276,
+            -0.009420748800039291,
+            -0.02739690989255905,
+            -0.025437654927372932,
+            -0.007653337903320789,
+            -0.013739272020757198,
+            -0.030368445441126823,
+            -0.011722872965037823,
+            0.01551892887800932,
+            -0.007645174395292997,
+            0.011918798089027405,
+            -0.016702644526958466,
+            0.01385356206446886,
+            -0.004110352601855993,
+            -0.004763437435030937,
+            -0.004530776292085648,
+            -0.017600636929273605,
+            -0.006420640274882317,
+            0.014367866329848766,
+            -0.009477893821895123,
+            -0.01237595733255148,
+            -0.02215590327978134,
+            -0.0015459742862731218,
+            -0.01574750803411007,
+            0.003489922033622861,
+            -0.014596446417272091,
+            -0.006771673448383808,
+            -0.003908304497599602,
+            -0.018514955416321754,
+            0.01766594499349594,
+            0.014816862531006336,
+            -0.019380291923880577,
+            -0.0090697156265378,
+            0.02638462744653225,
+            0.042777057737112045,
+            -0.03218075633049011,
+            -0.008200296200811863,
+            0.0020378287881612778,
+            0.018090449273586273,
+            0.010171796195209026,
+            0.016702644526958466,
+            -0.024653952568769455,
+            0.0022286521270871162,
+            -0.021649762988090515,
+            -0.001220452249981463,
+            -0.023739634081721306,
+            0.010775899514555931,
+            0.025715215131640434,
+            -0.010971825569868088,
+            0.025976449251174927,
+            0.01634344831109047,
+            -0.01812310516834259,
+            0.012016761116683483,
+            0.007183933164924383,
+            0.014645427465438843,
+            -0.019755816087126732,
+            0.0167679525911808,
+            -0.015796490013599396,
+            0.031233781948685646,
+            0.00892277155071497,
+            -0.04829562455415726,
+            -0.03503800183534622,
+            -0.013388238847255707,
+            0.0013388239312916994,
+            -0.010702427476644516,
+            0.015037278644740582,
+            0.015967924147844315,
+            0.013959688134491444,
+            -0.019380291923880577,
+            -0.015820980072021484,
+            0.011984107084572315,
+            0.010065670125186443,
+            0.027462217956781387,
+            -0.006649219896644354,
+            -0.004212397150695324,
+            -0.00960034690797329,
+            -0.02249877341091633,
+            -0.016098542138934135,
+            0.0209803506731987,
+            -0.018041469156742096,
+            0.015935270115733147,
+            -0.026139721274375916,
+            -0.012449429370462894,
+            0.017453692853450775,
+            -0.05211617052555084,
+            -0.01453113742172718,
+            0.0007612520130351186,
+            0.03012353740632534,
+            0.004751192405819893,
+            0.0035184945445507765,
+            0.01841699331998825,
+            -0.01993541419506073,
+            -0.0039776950143277645,
+            -0.02099667675793171,
+            0.006832899991422892,
+            -0.011853490024805069,
+            -0.013984179124236107,
+            0.014482156373560429,
+            -0.013845399022102356,
+            -0.01601690612733364,
+            0.0034368589986115694,
+            0.007183933164924383,
+            -0.021094640716910362,
+            0.011951452121138573,
+            0.003945040516555309,
+            0.012171869166195393,
+            0.012171869166195393,
+            0.01199227012693882,
+            0.021551799029111862,
+            0.02295593172311783,
+            -0.007334959227591753,
+            -0.004767519421875477,
+            -0.01004117913544178,
+            -0.0015398516552522779,
+            0.008898280560970306,
+            0.0033797139767557383,
+            -0.002989904023706913,
+            0.007837017998099327,
+            0.010033016093075275,
+            -0.01498013362288475,
+            0.033193036913871765,
+            -0.0035878848284482956,
+            -0.04134026914834976,
+            -0.003238892648369074,
+            -0.03800953924655914,
+            0.020033378154039383,
+            -0.023298801854252815,
+            -0.016996532678604126,
+            0.0032776694279164076,
+            -0.00629410520195961,
+            -0.019902760162949562,
+            0.013608654960989952,
+            0.011404493823647499,
+            0.00258580781519413,
+            0.02021297626197338,
+            0.012269831262528896,
+            -0.005134879611432552,
+            0.026172375306487083,
+            -0.014718899503350258,
+            -0.008751336485147476,
+            0.015567909926176071,
+            0.006853308994323015,
+            0.0022347746416926384,
+            0.0011031010653823614,
+            -0.018825169652700424,
+            0.005098143592476845,
+            0.0164903923869133,
+            0.007216587662696838,
+            -0.000566347036510706,
+            0.014049487188458443,
+            0.012514738366007805,
+            -0.015135240741074085,
+            -0.0002213090192526579,
+            -0.026564225554466248,
+            0.0032817514147609472,
+            0.004567512311041355,
+            0.007506393827497959,
+            -0.006771673448383808,
+            0.006281859707087278,
+            0.00658799335360527,
+            0.014759717509150505,
+            -0.010155469179153442,
+            -0.016980206593871117,
+            -0.002510294783860445,
+            0.004285869188606739,
+            0.016694480553269386,
+            0.03157665207982063,
+            -0.010310577228665352,
+            -0.005734901409596205,
+            -0.0386626236140728,
+            0.02070278860628605,
+            0.03332365304231644,
+            0.011175914667546749,
+            -0.028213264420628548,
+            0.012302485294640064,
+            0.0006515541463159025,
+            0.0019276207312941551,
+            -0.026302991434931755,
+            0.02787039615213871,
+            0.00974729098379612,
+            0.0066614653915166855,
+            0.017861871048808098,
+            0.0022306928876787424,
+            0.0029266364872455597,
+            -0.016302630305290222,
+            0.01018812321126461,
+            0.009086042642593384,
+            -0.017306748777627945,
+            0.0009745250572450459,
+            0.02324981987476349,
+            -0.01423724927008152,
+            -0.003208279376849532,
+            0.0018133309204131365,
+            0.02651524543762207,
+            -0.013820908032357693,
+            -0.014294394291937351,
+            0.0005714492290280759,
+            0.01751900091767311,
+            -0.0007979880319908261,
+            0.012098397128283978,
+            0.04898136481642723,
+            -0.012424939312040806,
+            0.019657853990793228,
+            0.005461421795189381,
+            -0.015290348790585995,
+            -0.032605260610580444,
+            -0.004922627005726099,
+            -0.01830270327627659,
+            0.011200404725968838,
+            0.014343376271426678,
+            -0.015763835981488228,
+            0.0007525782566517591,
+            -0.03144603595137596,
+            0.007743136957287788,
+            0.014735226519405842,
+            0.018776189535856247,
+            -0.002440904499962926,
+            0.006134915631264448,
+            0.027674470096826553,
+            0.02458864450454712,
+            -0.007392104249447584,
+            0.013804581016302109,
+            0.02320083975791931,
+            -0.025062130764126778,
+            -0.003069498809054494,
+            0.006820654962211847,
+            0.004104230087250471,
+            0.0053063142113387585,
+            -0.016963878646492958,
+            0.03915243595838547,
+            -0.005922663025557995,
+            -0.01506176870316267,
+            0.0030266400426626205,
+            -0.009028897620737553,
+            0.0063390047289431095,
+            -0.012563719414174557,
+            0.012637191452085972,
+            -0.008114579133689404,
+            0.00022883480414748192,
+            -0.0059430720284581184,
+            -0.011110605672001839,
+            -0.012653518468141556,
+            -0.0007949267164804041,
+            -0.016531210392713547,
+            0.007432921789586544,
+            0.00981259997934103,
+            0.005045080557465553,
+            0.01025343220680952,
+            -0.0036797248758375645,
+            -0.003136848099529743,
+            -0.016751626506447792,
+            -0.009984034113585949,
+            0.0073716952465474606,
+            0.01872720755636692,
+            -0.028588788583874702,
+            -0.018547609448432922,
+            0.009290131740272045,
+            0.0022898786701261997,
+            0.018776189535856247,
+            0.0158944521099329,
+            -0.0002595757250674069,
+            0.012318813242018223,
+            -0.01964152604341507,
+            -0.009184005670249462,
+            0.009690145961940289,
+            -0.003569516818970442,
+            0.01048201136291027,
+            -0.014759717509150505,
+            0.010033016093075275,
+            -0.011159587651491165,
+            -0.011053460650146008,
+            0.01385356206446886,
+            0.01738838478922844,
+            0.008873790502548218,
+            -0.001294944784604013,
+            3.383413059054874e-05,
+            -0.010506502352654934,
+            -0.024033522233366966,
+            0.004771600943058729,
+            0.027021385729312897,
+            -0.013273949734866619,
+            0.003883813973516226,
+            0.008702355436980724,
+            -0.017502672970294952,
+            -0.019821124151349068,
+            -0.012653518468141556,
+            -0.022106921300292015,
+            0.020572172477841377,
+            0.023413091897964478,
+            -0.02026195637881756,
+            0.00877582747489214,
+            -0.004620575346052647,
+            0.014049487188458443,
+            -0.01400050614029169,
+            0.010612628422677517,
+            0.0317889042198658,
+            -0.020000724121928215,
+            -0.0028654097113758326,
+            0.0036633978597819805,
+            -0.015347493812441826,
+            0.018629245460033417,
+            0.03268689662218094,
+            -0.00176945177372545,
+            0.014041324146091938,
+            0.0008138049161061645,
+            -0.009118696674704552,
+            0.01905374974012375,
+            0.011094278655946255,
+            0.005347132217139006,
+            -0.0042491331696510315,
+            -0.012212686240673065,
+            -0.03010721132159233,
+            0.016702644526958466,
+            0.0129637336358428,
+            -0.0037674831692129374,
+            -0.0011551438365131617,
+            -0.006343086715787649,
+            -0.015404638834297657,
+            0.0035123717971146107,
+            -0.013078023679554462,
+            -0.00803294312208891,
+            -0.008759500458836555,
+            0.004314441699534655,
+            -0.0034960447810590267,
+            -0.009053388610482216,
+            0.007236996199935675,
+            0.0009510547970421612,
+            -0.0010602424154058099,
+            -0.01902109570801258,
+            -0.025078458711504936,
+            -0.00995954405516386,
+            -0.016849588602781296,
+            0.0038287099450826645,
+            0.0005234883283264935,
+            -0.027184655889868736,
+            0.0053961132653057575,
+            -0.008139070123434067,
+            -0.008134988136589527,
+            0.007126788143068552,
+            0.006881881505250931,
+            -0.007437003776431084,
+            -0.010816717520356178,
+            -0.00786150898784399,
+            -0.006865554489195347,
+            0.020033378154039383,
+            0.0034348180051892996,
+            0.012082069180905819,
+            -0.010498338378965855,
+            -0.018025141209363937,
+            -0.016963878646492958,
+            0.016718972474336624,
+            -0.008971752598881721,
+            -0.02083340659737587,
+            -0.01444950234144926,
+            -0.008424794301390648,
+            -0.007665583398193121,
+            -0.01527402177453041,
+            -0.021159948781132698,
+            -0.0032103201374411583,
+            -0.011347348801791668,
+            -0.0067431009374558926,
+            -0.0028960229828953743,
+            0.0013571919407695532,
+            0.00951871182769537,
+            0.00546958576887846,
+            0.00438791373744607,
+            0.021666089072823524,
+            0.010122815147042274,
+            -0.005461421795189381,
+            0.016441410407423973,
+            -0.006183897145092487,
+            -0.004440976772457361,
+            -0.01498013362288475,
+            -0.009918726049363613,
+            0.018547609448432922,
+            -0.0055226488038897514,
+            0.008457448333501816,
+            -0.0009536059224046767,
+            -0.00576347392052412,
+            0.015674035996198654,
+            0.0129637336358428,
+            0.016718972474336624,
+            0.004771600943058729,
+            -0.00012168807734269649,
+            0.012416775338351727,
+            -0.0015745466807857156,
+            0.012555556371808052,
+            -0.0038429962005466223,
+            0.014506647363305092,
+            -0.009722800925374031,
+            -0.018988441675901413,
+            0.004157293122261763,
+            0.006265532691031694,
+            -0.012531065382063389,
+            0.009404421783983707,
+            0.007743136957287788,
+            -0.013747435994446278,
+            -0.02261306345462799,
+            -0.000106317616882734,
+            0.00014362763613462448,
+            0.008122743107378483,
+            0.008971752598881721,
+            -0.008073761127889156,
+            0.012890261597931385,
+            0.004616493359208107,
+            -0.007179851643741131,
+            -0.019657853990793228,
+            0.026009103283286095,
+            0.015608727931976318,
+            0.012269831262528896,
+            -0.052508022636175156,
+            -0.0030980713199824095,
+            0.010743245482444763,
+            -0.015951598063111305,
+            -0.026792805641889572,
+            -0.006404313258826733,
+            -0.008759500458836555,
+            -0.04290767386555672,
+            -0.009624837897717953,
+            -0.0005724696675315499,
+            -0.011388166807591915,
+            0.010400376282632351,
+            0.01573934406042099,
+            0.014612773433327675,
+            0.005489994306117296,
+            -0.007232914678752422,
+            -0.017633290961384773,
+            -0.02788672223687172,
+            0.010057506151497364,
+            -0.013380075804889202,
+            -0.006412476766854525,
+            -0.002263347152620554,
+            -0.007690073922276497,
+            0.0016112826997414231,
+            0.01691489666700363,
+            0.042483169585466385,
+            0.023037567734718323,
+            -0.012335140258073807,
+            -0.020131340250372887,
+            -0.012800462543964386,
+            -0.009192168712615967,
+            0.00633492274209857,
+            0.011592255905270576,
+            -0.004738946910947561,
+            0.0096656559035182,
+            -0.004294032696634531,
+            0.012302485294640064,
+            0.012767808511853218,
+            0.01680060662329197,
+            -2.3023791072773747e-05,
+            0.005579793825745583,
+            -0.0024694770108908415,
+            -0.01738838478922844,
+            -0.004379750229418278,
+            0.008261523209512234,
+            -0.0008872769540175796,
+            0.0019317024853080511,
+            0.014204595237970352,
+            0.019102731719613075,
+            -0.009094206616282463,
+            -0.014612773433327675,
+            -0.016098542138934135,
+            -0.005277741700410843,
+            -0.0012622905196622014,
+            0.029927613213658333,
+            0.02037624642252922,
+            -0.019266001880168915,
+            -0.018237393349409103,
+            -0.018351683393120766,
+            0.00922482367604971,
+            -0.003563394071534276,
+            -0.0064369672909379005,
+            0.01905374974012375,
+            0.023021239787340164,
+            0.017910851165652275,
+            -0.003642988856881857,
+            -0.014865843579173088,
+            0.003804219188168645,
+            0.02847449854016304,
+            0.008367649279534817,
+            0.01078406348824501,
+            0.008726846426725388,
+            -0.0011908593587577343,
+            -0.032441988587379456,
+            -0.0113146947696805,
+            0.00951871182769537,
+            -0.008481939323246479,
+            0.026564225554466248,
+            -0.016531210392713547,
+            0.009388094767928123,
+            -0.005077734589576721,
+            0.0019286411115899682,
+            -0.02365799807012081,
+            0.0014929111348465085,
+            0.010122815147042274,
+            -0.024947840720415115,
+            0.00336950970813632,
+            -0.023592690005898476,
+            -0.00013763252354692668,
+            0.008367649279534817,
+            0.0022837561555206776,
+            0.01650671847164631,
+            -0.0113146947696805,
+            0.0051838611252605915,
+            0.02099667675793171,
+            -0.002832755446434021,
+            -0.01512707769870758,
+            -0.0007842120248824358,
+            0.0005092020728625357,
+            -0.001286781276576221,
+            0.02563357912003994,
+            0.015812816098332405,
+            -0.0036756431218236685,
+            0.020882388576865196,
+            0.008677864447236061,
+            0.0019133344758301973,
+            -0.021764053031802177,
+            0.0023796779569238424,
+            -0.032899148762226105,
+            0.017437364906072617,
+            -0.011126932688057423,
+            0.012490247376263142,
+            0.00868602842092514,
+            -0.01341272983700037,
+            -0.0007388022495433688,
+            -0.0107187544927001,
+            -0.0015725058037787676,
+            -0.007934981025755405,
+            0.009779945947229862,
+            -0.004363423213362694,
+            0.008339077234268188,
+            -0.00024579971795901656,
+            -0.01476788055151701,
+            0.018694553524255753,
+            9.678283277025912e-06,
+            -0.00010925139940809458,
+            0.005441013257950544,
+            0.009935053065419197,
+            0.00786559097468853,
+            0.002009256277233362,
+            0.023870250210165977,
+            -0.031119491904973984,
+            -0.016082214191555977,
+            -0.010204450227320194,
+            0.005543057806789875,
+            -0.007922735065221786,
+            -0.018531281501054764,
+            -0.03257260471582413,
+            0.01318415068089962,
+            0.001293924287892878,
+            -0.0009806477464735508,
+            0.01258004643023014,
+            -0.00942891277372837,
+            -0.0013878053287044168,
+            -0.01606588624417782,
+            -0.014808698557317257,
+            0.006416558753699064,
+            0.0033633869607001543,
+            -0.01222901325672865,
+            -0.023739634081721306,
+            0.014759717509150505,
+            0.0007418635650537908,
+            -0.016980206593871117,
+            -0.017502672970294952,
+            0.00018457300029695034,
+            -0.004804255440831184,
+            0.02204161323606968,
+            -0.016678154468536377,
+            0.003373591462150216,
+            0.0007219648687168956,
+            0.009232986718416214,
+            -0.02889900468289852,
+            0.00716352416202426,
+            0.0006107363733462989,
+            0.0035164537839591503,
+            -0.006518602836877108,
+            -0.021013004705309868,
+            0.011371839791536331,
+            -0.0020674217958003283,
+            -0.014751553535461426,
+            -0.020751770585775375,
+            -0.0020796670578420162,
+            -0.007420676294714212,
+            -0.02724996581673622,
+            0.017910851165652275,
+            0.0009985055075958371,
+            -0.01738838478922844,
+            0.008077843114733696,
+            0.006596156861633062,
+            0.009355440735816956,
+            0.02455599047243595,
+            0.010147306136786938,
+            -0.018776189535856247,
+            -0.0008087027235887945,
+            0.006979844067245722,
+            0.009306458756327629,
+            0.0036450298503041267,
+            0.04222193360328674,
+            -0.008130906149744987,
+            -0.006579829845577478,
+            0.0066614653915166855,
+            -0.008449285291135311,
+            -0.013747435994446278,
+            0.026727497577667236,
+            0.005796127952635288,
+            0.015984252095222473,
+            0.016123032197356224,
+            -0.00945340283215046,
+            -0.005008344538509846,
+            -0.0018786393338814378,
+            0.02065380848944187,
+            0.0073798587545752525,
+            -0.005502239800989628,
+            -0.009714636951684952,
+            0.005881845485419035,
+            -0.00898808054625988,
+            0.0021653843577951193,
+            -0.009526874870061874,
+            -0.005824700463563204,
+            -0.008069679141044617,
+            0.016400592401623726,
+            -0.0003523086488712579,
+            0.027315273880958557,
+            0.020751770585775375,
+            -0.008563575334846973,
+            -0.0005683879135176539,
+            0.017208784818649292,
+            -0.005820618476718664,
+            0.015608727931976318,
+            -0.011192241683602333,
+            -0.007706400938332081,
+            0.006408394780009985,
+            0.024017194285988808,
+            0.012237177230417728,
+            -0.0056042843498289585,
+            0.0002885308349505067,
+            -0.0030348035506904125,
+            -0.010612628422677517,
+            0.013388238847255707,
+            0.015323002822697163,
+            -0.01947825588285923,
+            -0.004722619894891977,
+            -0.0016592437168583274,
+            -0.003300119424238801,
+            0.00945340283215046,
+            0.02305389568209648,
+            0.000171817431692034,
+            0.009184005670249462,
+            0.0012765767751261592,
+            0.03862996771931648,
+            -0.0053144777193665504,
+            0.03102152980864048,
+            -0.0010423846542835236,
+            -0.0059430720284581184,
+            -0.009094206616282463,
+            -0.020441556349396706,
+            -0.011486129835247993,
+            -0.00041481092921458185,
+            -0.005040998570621014,
+            -0.0004956811899319291,
+            -0.020327266305685043,
+            0.0026960158720612526,
+            0.014637264423072338,
+            0.007661501411348581,
+            0.017927179113030434,
+            -0.002889900468289852,
+            0.017437364906072617,
+            0.019690508022904396,
+            -0.01400050614029169,
+            -0.029862305149435997,
+            -0.0006515541463159025,
+            0.01498013362288475,
+            -0.01661284640431404,
+            0.019445601850748062,
+            0.004085862077772617,
+            0.0029531680047512054,
+            0.004779764451086521,
+            -0.006049198564141989,
+            0.018482301384210587,
+            0.02039257436990738,
+            0.01453930139541626,
+            -0.013167822733521461,
+            0.014057651162147522,
+            0.009771781973540783,
+            0.006600238848477602,
+            -0.01646590232849121,
+            -0.013535182923078537,
+            0.002555194543674588,
+            0.0009245232213288546,
+            0.024392718449234962,
+            -0.020588500425219536,
+            0.0016602640971541405,
+            0.010588138364255428,
+            0.01769859902560711,
+            -0.004302196204662323,
+            0.008906444534659386,
+            0.0008740111952647567,
+            0.007134951651096344,
+            -0.0035919665824621916,
+            -0.006457376293838024,
+            -0.006673710886389017,
+            -0.010392212308943272,
+            -0.0020959940738976,
+            -0.02070278860628605,
+            0.002198038622736931,
+            -0.002830714685842395,
+            0.018237393349409103,
+            -0.01018812321126461,
+            0.038140155375003815,
+            0.03399306535720825,
+            -0.0007025764207355678,
+            0.011102442629635334,
+            -0.01709449477493763,
+            -0.003945040516555309,
+            -0.021959977224469185,
+            0.01844964735209942,
+            0.016392430290579796,
+            0.012269831262528896,
+            0.005302232690155506,
+            0.017796561121940613,
+            -0.0006301248213276267,
+            0.014490320347249508,
+            0.0029654132667928934,
+            0.007971717044711113,
+            -0.002265388146042824,
+            0.003938918001949787,
+            -0.031103165820240974,
+            -0.017029186710715294,
+            -0.01138000376522541,
+            0.004796091932803392,
+            -0.006375740747898817,
+            -0.0035205355379730463,
+            0.0084737753495574,
+            -0.005902254022657871,
+            0.0004270562494639307,
+            -0.025992777198553085,
+            -0.0157148540019989,
+            -0.010318740271031857,
+            -0.004040962550789118,
+            -0.01169838197529316,
+            -0.0027694879099726677,
+            -0.0022082431241869926,
+            0.0029552089981734753,
+            0.020947696641087532,
+            -0.012808626517653465,
+            -0.016294466331601143,
+            0.004379750229418278,
+            -0.002263347152620554,
+            -0.014016833156347275,
+            -0.016898570582270622,
+            0.006665547378361225,
+            0.012098397128283978,
+            -0.0017470018938183784,
+            -0.013608654960989952,
+            0.011739199981093407,
+            -0.006628811359405518,
+            0.04663025587797165,
+            -0.013584164902567863,
+            -0.0078002819791436195,
+            0.003720542648807168,
+            0.004698128905147314,
+            -0.009396257810294628,
+            0.007330877240747213,
+            0.013020878657698631,
+            0.01830270327627659,
+            0.0095758568495512,
+            -0.008465612307190895,
+            -0.0024388637393712997,
+            0.007796200457960367,
+            -0.016147522255778313,
+            0.006404313258826733,
+            0.0008525818702764809,
+            -0.014400520361959934,
+            0.006110425107181072,
+            0.028980640694499016,
+            0.013151495717465878,
+            0.01288209855556488,
+            -0.022351829335093498,
+            -0.004094025585800409,
+            -0.009355440735816956,
+            -0.004943036008626223,
+            -0.010090161114931107,
+            0.027070365846157074,
+            -0.018155759200453758,
+            0.02651524543762207,
+            0.00898808054625988,
+            0.0017684312770143151,
+            -0.01176369097083807,
+            0.006343086715787649,
+            0.02832755446434021,
+            0.0028592869639396667,
+            -0.020000724121928215,
+            -0.013592327944934368,
+            0.006314514204859734,
+            0.01707816869020462,
+            -0.002647034591063857,
+            -0.005428767763078213,
+            0.019657853990793228,
+            0.01784554310142994,
+            -0.004334850702434778,
+            -0.00936360377818346,
+            0.0008337036124430597,
+            0.00928196869790554,
+            0.016963878646492958,
+            0.015020951628684998,
+            0.009086042642593384,
+            -0.002647034591063857,
+            0.008037025108933449,
+            -0.008800318464636803,
+            -0.016449574381113052,
+            0.007220669183880091,
+            -0.008432958275079727,
+            -0.016963878646492958,
+            0.01872720755636692,
+            0.015257694758474827,
+            0.0181394312530756,
+            0.0024592725094407797,
+            -0.016735298559069633,
+            0.022302847355604172,
+            0.005755309946835041,
+            0.002998067531734705,
+            0.02160078100860119,
+            -0.022221211344003677,
+            0.013012715615332127,
+            0.02037624642252922,
+            0.027609162032604218,
+            -0.01737205684185028,
+            -0.010906516574323177,
+            -0.004857318475842476,
+            -0.006853308994323015,
+            -0.00176945177372545,
+            -0.008253359235823154,
+            -0.012620864436030388,
+            0.0028735732194036245,
+            -0.030466407537460327,
+            0.018106777220964432,
+            -0.005102225113660097,
+            0.010669773444533348,
+            -0.019266001880168915,
+            0.011363675817847252,
+            -0.0035838030744343996,
+            0.011135096661746502,
+            0.0019612954929471016,
+            -0.030041903257369995,
+            -0.008571738377213478,
+            0.015388311818242073,
+            -0.0030368445441126823,
+            -0.013641309924423695,
+            0.02006603218615055,
+            0.00018942011229228228,
+            -0.022270193323493004,
+            -0.004314441699534655,
+            -0.006934944540262222,
+            0.002193956868723035,
+            -0.0029613315127789974,
+            -0.0052981507033109665,
+            -0.030188847333192825,
+            0.020033378154039383,
+            0.009559528902173042,
+            -0.03072764165699482,
+            -0.003255219664424658,
+            -0.014996460638940334,
+            -0.0025184585247188807,
+            0.008881953544914722,
+            -0.004000144544988871,
+            -0.015551582910120487,
+            0.01784554310142994,
+            0.022711025550961494,
+            -0.0023164101876318455,
+            -0.016057724133133888,
+            -0.0090697156265378,
+            -0.027494871988892555,
+            0.030825603753328323,
+            -0.0066696288995444775,
+            0.0022878379095345736,
+            -0.005322641227394342,
+            0.009388094767928123,
+            0.004228724166750908,
+            0.0157148540019989,
+            0.006081852596253157,
+            -0.002647034591063857,
+            0.026596879586577415,
+            -0.01964152604341507,
+            0.009567692875862122,
+            -1.588864870427642e-05,
+            -0.003308282932266593,
+            0.01890680566430092,
+            0.0007939062779769301,
+            0.021649762988090515,
+            -0.008571738377213478,
+            -0.0001475818717153743,
+            -0.0034388997592031956,
+            0.010898353531956673,
+            -0.0028450009413063526,
+            -0.003465431509539485,
+            -0.001531688030809164,
+            0.011118769645690918,
+            0.01013097818940878,
+            0.003804219188168645,
+            -0.01737205684185028,
+            0.003142970846965909,
+            -0.008547247387468815,
+            -0.007028825581073761,
+            0.01302904263138771,
+            -0.011192241683602333,
+            -0.0018990482203662395,
+            -0.03160930797457695,
+            0.016286302357912064,
+            0.0533243790268898,
+            0.008547247387468815,
+            -0.014433175325393677,
+            -0.0018806802108883858,
+            -0.019755816087126732,
+            0.01535565685480833,
+            0.011004479601979256,
+            -0.016816934570670128,
+            -0.011984107084572315,
+            -0.012310649268329144,
+            0.007649256382137537,
+            0.021306892856955528,
+            -0.006522684823721647,
+            -0.008579902350902557,
+            0.025372346863150597,
+            -0.011788181029260159,
+            -0.010882026515901089,
+            -0.01661284640431404,
+            -0.01635977439582348,
+            0.00241641397587955,
+            0.019902760162949562,
+            -0.0169475506991148,
+            0.021176276728510857,
+            0.00936360377818346,
+            0.0010133018950000405,
+            0.003624620847404003,
+            0.008588065393269062,
+            0.0024878450203686953,
+            0.017878197133541107,
+            0.003883813973516226,
+            -0.01439235731959343,
+            0.0021408938337117434,
+            -0.0030286810360848904,
+            0.004624656867235899,
+            3.4320437407586724e-06,
+            -0.002749078907072544,
+            -0.0065635028295218945,
+            0.003710338147357106,
+            -0.021306892856955528,
+            0.021502818912267685,
+            -0.022678371518850327,
+            -0.004245051648467779,
+            0.005555302836000919,
+            -0.005118552595376968,
+            0.013469874858856201,
+            0.018792515620589256,
+            0.01423724927008152,
+            0.008702355436980724,
+            0.01462093647569418,
+            -0.011331021785736084,
+            0.02323349379003048,
+            -0.013282112777233124,
+            0.0005816536722704768,
+            0.012939243577420712,
+            0.01755165494978428,
+            0.018792515620589256,
+            -0.006245123688131571,
+            -0.010384049266576767,
+            -0.0002946535241790116,
+            0.0129637336358428,
+            -0.022286519408226013,
+            -0.012269831262528896,
+            -0.01386172603815794,
+            0.011486129835247993,
+            -0.004743028432130814,
+            0.025453981012105942,
+            -0.00633492274209857,
+            -0.014653591439127922,
+            -0.015918942168354988,
+            0.014196431264281273,
+            -0.048752784729003906,
+            0.0030348035506904125,
+            -0.0024919267743825912,
+            0.03043375350534916,
+            -0.027021385729312897,
+            0.01738838478922844,
+            -0.0028490826953202486,
+            -0.009355440735816956,
+            0.015429128892719746,
+            -0.015992414206266403,
+            -0.0046613928861916065,
+            -0.0016806729836389422,
+            -0.007539048325270414,
+            -0.00112248957157135,
+            0.017616963014006615,
+            -0.015845470130443573,
+            0.01282495353370905,
+            -0.00989423505961895,
+            0.0027021386194974184,
+            -0.004526694305241108,
+            -0.007751300930976868,
+            0.0196088720113039,
+            0.034548189491033554,
+            -0.007053316105157137,
+            0.0014990337658673525,
+            -0.022547753527760506,
+            0.013192313723266125,
+            -0.006730855442583561,
+            0.02351105399429798,
+            -0.005432849284261465,
+            -0.016898570582270622,
+            -0.013976015150547028,
+            -0.009739127941429615,
+            0.00839214026927948,
+            -0.022433463484048843,
+            -0.011110605672001839,
+            -0.00922482367604971,
+            -0.002573562553152442,
+            -0.018694553524255753,
+            0.004865481983870268,
+            -0.002426618244498968,
+            -0.012506574392318726,
+            -0.008816645480692387,
+            0.01048201136291027,
+            -0.009347276762127876,
+            0.003077662317082286,
+            -0.0011929002357646823,
+            -0.011453475803136826,
+            -0.02369065210223198,
+            0.0158944521099329,
+            0.0001974561164388433,
+            -0.010359558276832104,
+            0.008408467285335064,
+            -0.0018276170594617724,
+            0.001211268245242536,
+            0.016963878646492958,
+            -0.0020664012990891933,
+            0.003467472270131111,
+            0.023478399962186813,
+            0.004057289566844702,
+            -0.010971825569868088,
+            -0.00025906547671183944,
+            0.013535182923078537,
+            -0.005118552595376968,
+            -0.010008525103330612,
+            1.6646010408294387e-05,
+            -0.02984597720205784,
+            -0.008351322263479233,
+            0.006094098091125488,
+            0.018629245460033417,
+            -0.010620792396366596,
+            -0.015853634104132652,
+            0.026319319382309914,
+            -0.006359413731843233,
+            0.019070077687501907,
+            0.002889900468289852,
+            0.0025694805663079023,
+            0.01724144071340561,
+            0.013461710885167122,
+            0.00786150898784399,
+            -0.013927034102380276,
+            -0.02429475635290146,
+            0.0016398552106693387,
+            -0.018090449273586273,
+            0.010579974390566349,
+            -0.011028970591723919,
+            0.0032756286673247814,
+            -0.02398454025387764,
+            -0.012237177230417728,
+            0.004934872500598431,
+            -0.020343592390418053,
+            -0.00040588201954960823,
+            -0.0056940834037959576,
+            0.022237539291381836,
+            -0.005310396198183298,
+            0.008669701404869556,
+            0.0018255761824548244,
+            -0.012237177230417728,
+            0.0009877908742055297,
+            0.009469729848206043,
+            -0.029192892834544182,
+            -0.0007127808639779687,
+            -0.003138889092952013,
+            0.0011418780777603388,
+            0.005351213738322258,
+            0.009755454957485199,
+            0.007322713732719421,
+            0.0027062203735113144,
+            0.007253323681652546,
+            0.010449357330799103,
+            -0.00022041612828616053,
+            0.0197884701192379,
+            0.018172085285186768,
+            -0.007351286243647337,
+            0.008710519410669804,
+            0.024817224591970444,
+            -0.013159659691154957,
+            -0.008718682453036308,
+            0.023266147822141647,
+            -0.006461458280682564,
+            -0.0011122850701212883,
+            -0.01483318954706192,
+            0.00815947912633419,
+            0.0017367975087836385,
+            0.028131630271673203,
+            0.019657853990793228,
+            0.005820618476718664,
+            0.000639819074422121,
+            0.015788326039910316,
+            -0.013020878657698631,
+            0.0035286990460008383,
+            0.015461783856153488,
+            -0.003940958995372057,
+            -0.019821124151349068,
+            0.013878053054213524,
+            0.009502384811639786,
+            -0.003228688146919012,
+            0.01551892887800932,
+            0.011412657797336578,
+            -0.030841931700706482,
+            -0.0011898389784619212,
+            0.010498338378965855,
+            -0.0021449755877256393,
+            -0.008751336485147476,
+            -0.00014273475972004235,
+            -0.012384121306240559,
+            0.021388528868556023,
+            -0.004408322740346193,
+            0.012277995236217976,
+            -0.00815947912633419,
+            -0.017127150669693947,
+            -0.011559601873159409,
+            0.007914572022855282,
+            -0.010547320358455181,
+            -0.0006714528426527977,
+            -0.004022594541311264,
+            -0.005779800936579704,
+            0.017747581005096436,
+            0.004706292413175106,
+            -0.0008964610169641674,
+            -0.0018398624379187822,
+            0.007983962073922157,
+            -0.006979844067245722,
+            -0.008726846426725388,
+            0.004816500470042229,
+            -0.012833117507398129,
+            0.003981776535511017,
+            0.005543057806789875,
+            0.01506176870316267,
+            -0.014857680536806583,
+            -0.021159948781132698,
+            0.007020662073045969,
+            0.02695607766509056,
+            -0.02516009286046028,
+            -0.005232842173427343,
+            0.006245123688131571,
+            0.002093953313305974,
+            0.006914535537362099,
+            -0.006934944540262222,
+            -0.007208424154669046,
+            -0.03657275065779686,
+            -0.0020745648071169853,
+            0.0009005427709780633,
+            0.008963589556515217,
+            -0.008049271069467068,
+            -0.0136494729667902,
+            -0.011135096661746502,
+            0.002410291228443384,
+            0.007134951651096344,
+            -0.02204161323606968,
+            0.0049593630246818066,
+            0.009608510881662369,
+            -0.019559891894459724,
+            0.025127438828349113,
+            -0.010890189558267593,
+            -6.964537897147238e-05,
+            -0.00995954405516386,
+            0.021633435040712357,
+            -0.013486201874911785,
+            -0.018368011340498924,
+            -0.0006898208521306515,
+            0.008355404250323772,
+            0.011926962062716484,
+            -0.0017663904000073671,
+            -0.011371839791536331,
+            0.0003752686607185751,
+            0.01646590232849121,
+            0.019135385751724243,
+            0.002744997153058648,
+            -0.0024837632663547993,
+            -0.01019628718495369,
+            -0.009592183865606785,
+            -0.0073961857706308365,
+            -0.0007449248805642128,
+            0.002091912319883704,
+            0.02035992033779621,
+            0.013469874858856201,
+            0.0024837632663547993,
+            -0.015135240741074085,
+            -0.005257333163172007,
+            0.008939098566770554,
+            0.009412584826350212,
+            -0.011690218932926655,
+            -0.00898808054625988,
+            0.002659279853105545,
+            0.02249877341091633,
+            0.017633290961384773,
+            -0.0014000505907461047,
+            0.0032878739293664694,
+            -0.0016592437168583274,
+            0.012506574392318726,
+            -0.007555375341325998,
+            -0.010269759222865105,
+            -0.006943108048290014,
+            -0.014645427465438843,
+            -0.010800390504300594,
+            -0.0023062059190124273,
+            -0.010179960168898106,
+            0.022629389539361,
+            0.007988044060766697,
+            0.009779945947229862,
+            0.008265605196356773,
+            0.010269759222865105,
+            0.002087830565869808,
+            0.025715215131640434,
+            -0.004424649756401777,
+            -0.011984107084572315,
+            0.007236996199935675,
+            -0.022368155419826508,
+            -0.021535472944378853,
+            0.008257441222667694,
+            0.006367577239871025,
+            0.03595232218503952,
+            -0.026890767738223076,
+            0.0052981507033109665,
+            -0.009779945947229862,
+            -0.010400376282632351,
+            0.007996207103133202,
+            -0.0015439334092661738,
+            -0.004245051648467779,
+            0.022368155419826508,
+            0.00726556871086359,
+            0.0035797213204205036,
+            0.0032817514147609472,
+            0.019837452098727226,
+            -0.005685919895768166,
+            0.01573934406042099,
+            0.0259274672716856,
+            0.024621298536658287,
+            0.009388094767928123,
+            -0.009469729848206043,
+            0.02785406820476055,
+            0.0002895512734539807,
+            -0.0034327772445976734,
+            0.007220669183880091,
+            -0.00305113079957664,
+            -0.0009556467994116247,
+            -0.010033016093075275,
+            0.021764053031802177,
+            -0.001455154619179666,
+            0.004359341226518154,
+            0.02081707864999771,
+            -7.774515688652173e-05,
+            0.006220633164048195,
+            0.01057181041687727,
+            0.002181711606681347,
+            0.003396041225641966,
+            0.010775899514555931,
+            -0.039315707981586456,
+            -0.0006571666453965008,
+            -0.025600925087928772,
+            -0.0018286375561729074,
+            0.0007357408758252859,
+            -0.01266984548419714,
+            0.013600491918623447,
+            -0.002015379024669528,
+            -0.010751409456133842,
+            -0.00029337796149775386,
+            0.021649762988090515,
+            0.005416522268205881,
+            0.004071575589478016,
+            -0.004469549283385277,
+            0.018792515620589256,
+            -0.006955353543162346,
+            -0.017861871048808098,
+            -0.012759645469486713,
+            -0.012367794290184975,
+            -0.01918436773121357,
+            -0.01214737817645073,
+            -0.0024551907554268837,
+            -0.002667443361133337,
+            0.004738946910947561,
+            0.002481722505763173,
+            -0.003228688146919012,
+            -0.02070278860628605,
+            0.015037278644740582,
+            -0.01430255826562643,
+            0.012873934581875801,
+            -0.011053460650146008,
+            0.005963481031358242,
+            -0.004122598096728325,
+            0.012726990506052971,
+            -0.021404854953289032,
+            0.02070278860628605,
+            -0.01536382082849741,
+            0.0038532004691660404,
+            0.019592545926570892,
+            0.006551257334649563,
+            -0.008118661120533943,
+            -0.013404566794633865,
+            -0.022074267268180847,
+            0.021633435040712357,
+            -0.011633073911070824,
+            -0.018237393349409103,
+            -0.004608329851180315,
+            0.009649328887462616,
+            -0.017470018938183784,
+            -0.007367613259702921,
+            0.01707816869020462,
+            -0.0018949664663523436,
+            0.012980061583220959,
+            0.023004913702607155,
+            -0.0052981507033109665,
+            0.0036572751123458147,
+            -0.009167678654193878,
+            0.0010286086471751332,
+            -0.02470293454825878,
+            0.013527019880712032,
+            0.008971752598881721,
+            0.006269614677876234,
+            -0.013935198076069355,
+            -0.013306603766977787,
+            0.0036531933583319187,
+            0.004498121794313192,
+            0.011437147855758667,
+            0.009420748800039291,
+            0.004220560658723116,
+            -0.013918871060013771,
+            -0.027788760140538216,
+            -0.027070365846157074,
+            0.03595232218503952,
+            -0.0005002731923013926,
+            -0.020947696641087532,
+            0.03399306535720825,
+            0.016114868223667145,
+            0.0022980421781539917,
+            -0.012449429370462894,
+            -0.004314441699534655,
+            0.024621298536658287,
+            0.016400592401623726,
+            0.010465684346854687,
+            -0.0004742518358398229,
+            0.005571629852056503,
+            0.0016469983384013176,
+            0.03012353740632534,
+            0.015323002822697163,
+            0.0009459525463171303,
+            -0.038597315549850464,
+            0.011943289078772068,
+            -0.007547211833298206,
+            0.0013408648082986474,
+            0.006702283397316933,
+            0.017780235037207603,
+            0.011935125105082989,
+            0.015567909926176071,
+            0.014865843579173088,
+            -0.003065417055040598,
+            0.0029511270113289356,
+            0.0006163488142192364,
+            -0.009624837897717953,
+            -0.001767410896718502,
+            -0.008881953544914722,
+            -0.005714492406696081,
+            -0.002920513739809394,
+            0.005592038854956627,
+            0.007657419890165329,
+            -0.003983817528933287,
+            0.028049994260072708,
+            -0.0006433906382881105,
+            -0.017600636929273605,
+            -0.009061552584171295,
+            -0.03611559048295021,
+            0.00434301421046257,
+            -0.030939893797039986,
+            -0.011257549747824669,
+            0.0033674687147140503,
+            -0.020457882434129715,
+            -0.024768242612481117,
+            0.0010306495241820812,
+            0.014335212297737598,
+            0.007392104249447584,
+            0.0033327736891806126,
+            -0.01004934310913086,
+            0.003816464450210333,
+            -0.001855169073678553,
+            -0.006461458280682564,
+            0.028833696618676186,
+            0.019559891894459724,
+            0.020457882434129715,
+            -0.01190247107297182,
+            0.019298657774925232,
+            0.005845109466463327,
+            0.02035992033779621,
+            0.010465684346854687,
+            -0.0048695639707148075,
+            0.0074859848245978355,
+            0.009567692875862122,
+            -0.006155324634164572,
+            -0.021519144997000694,
+            0.013192313723266125,
+            -0.0010837126756086946,
+            -0.0052083516493439674,
+            -0.016269976273179054,
+            -0.003534821793437004,
+            -0.015812816098332405,
+            0.010547320358455181,
+            -0.017159804701805115,
+            0.012498411349952221,
+            -0.000802069844212383,
+            0.00168475485406816,
+            -0.015469946898519993,
+            -0.013951525092124939,
+            -0.0012439225101843476,
+            -0.01244126632809639,
+            -0.008971752598881721,
+            0.006163488142192364,
+            -0.007539048325270414,
+            -0.008318668231368065,
+            -0.00535937724635005,
+            -0.011437147855758667,
+            -0.010882026515901089,
+            0.004367504734545946,
+            -0.0010653446661308408,
+            0.020931368693709373,
+            0.00011671340325847268,
+            -0.005824700463563204,
+            0.01190247107297182,
+            0.02787039615213871,
+            0.0016704685986042023,
+            0.004583839327096939,
+            -0.014261740259826183,
+            -0.011118769645690918,
+            -0.02279266156256199,
+            0.016572028398513794,
+            0.0049675265327095985,
+            -0.0344502255320549,
+            0.01438419334590435,
+            0.00824927818030119,
+            0.0015102586476132274,
+            -0.006698201410472393,
+            -0.005334886722266674,
+            -0.005608365871012211,
+            0.006306350696831942,
+            -0.0022837561555206776,
+            0.0005107327597215772,
+            0.0035919665824621916,
+            0.034678805619478226,
+            -0.0040674940682947636,
+            -0.007228832691907883,
+            0.00981259997934103,
+            0.0005130287609063089,
+            0.020196648314595222,
+            -0.005518566817045212,
+            0.0037572786677628756,
+            -0.019282329827547073,
+            -0.016563864424824715,
+            0.0018429238116368651,
+            -0.01722511276602745,
+            0.023282473906874657,
+            -0.005392031744122505,
+            -0.01347803883254528,
+            -0.006906372029334307,
+            -0.020115012302994728,
+            0.012637191452085972,
+            0.02310287579894066,
+            0.007016580086201429,
+            0.01033506728708744,
+            0.012131051160395145,
+            -0.02444170042872429,
+            -0.020457882434129715,
+            0.012171869166195393,
+            0.0013571919407695532,
+            -0.008449285291135311,
+            -0.005404277238994837,
+            -0.001783737912774086,
+            0.007783954963088036,
+            -0.0032633834052830935,
+            0.007881917990744114,
+            0.06491663306951523,
+            -0.01093100756406784,
+            -0.002987863030284643,
+            -0.02966637909412384,
+            -0.004396077245473862,
+            -0.006628811359405518,
+            0.0056287748739123344,
+            0.009151351638138294,
+            -0.023135529831051826,
+            0.009232986718416214,
+            -0.030025575309991837,
+            -0.010628955438733101,
+            0.00824111420661211,
+            0.0028490826953202486,
+            -0.01220452319830656,
+            -0.0009592183632776141,
+            -0.013665799982845783,
+            0.0006107363733462989,
+            -0.009412584826350212,
+            -0.00576347392052412,
+            0.012718827463686466,
+            -0.02321716584265232,
+            0.006298186723142862,
+            0.007592111360281706,
+            -0.006514521315693855,
+            -0.013722945004701614,
+            -0.017861871048808098,
+            -0.011967780068516731,
+            -0.013314766809344292,
+            0.002996026771143079,
+            0.01190247107297182,
+            -0.02607441134750843,
+            0.012922916561365128,
+            -0.0037389106582850218,
+            0.00945340283215046,
+            0.004404240753501654,
+            0.013469874858856201,
+            0.018074123188853264,
+            0.009355440735816956,
+            0.0021429345943033695,
+            -0.008024780079722404,
+            0.012702500447630882,
+            -0.01993541419506073,
+            -0.01635161228477955,
+            0.013837235048413277,
+            0.012016761116683483,
+            0.003061335301026702,
+            0.009828926995396614,
+            0.0020959940738976,
+            0.011886144056916237,
+            0.010775899514555931,
+            -0.019739490002393723,
+            -0.0006648199632763863,
+            0.0007326795603148639,
+            0.03280118480324745,
+            -0.0005938990507274866,
+            0.008857463486492634,
+            0.004420568235218525,
+            0.013159659691154957,
+            -0.012784135527908802,
+            -0.01769859902560711,
+            0.0008087027235887945,
+            -0.01057181041687727,
+            0.021257910877466202,
+            -0.008192133158445358,
+            0.006216551177203655,
+            -0.018612917512655258,
+            -0.027086693793535233,
+            0.006392067763954401,
+            -0.016261812299489975,
+            -0.018221067264676094,
+            0.01317598670721054,
+            0.0077268099412322044,
+            0.0017572063952684402,
+            -0.005098143592476845,
+            -0.020457882434129715,
+            -0.009837090037763119,
+            0.028441844508051872,
+            0.011102442629635334,
+            -0.004120557103306055,
+            0.009584019891917706,
+            -0.009094206616282463,
+            -0.012490247376263142,
+            -0.0017541450215503573,
+            0.0036715613678097725,
+            0.008261523209512234,
+            0.005645101889967918,
+            -0.01386172603815794,
+            -0.00029873527819290757,
+            -0.012400448322296143,
+            -0.021225256845355034,
+            -0.008906444534659386,
+            0.016751626506447792,
+            0.007820690982043743,
+            -0.0038960592355579138,
+            0.015812816098332405,
+            -0.004404240753501654,
+            0.002189875114709139,
+            0.014294394291937351,
+            0.005334886722266674,
+            -0.004285869188606739,
+            -0.0095758568495512,
+            -0.004612411838024855,
+            0.004043003544211388,
+            -0.0025960123166441917,
+            0.006457376293838024,
+            0.017192458733916283,
+            0.0072900597006082535,
+            -0.013755599036812782,
+            0.013135168701410294,
+            0.0007791098323650658,
+            0.0049675265327095985,
+            -0.03699725493788719,
+            -0.003247056156396866,
+            0.015796490013599396,
+            -0.013020878657698631,
+            0.02501314878463745,
+            0.005400195252150297,
+            0.022743679583072662,
+            0.015233203768730164,
+            0.008669701404869556,
+            -0.001542912912555039,
+            0.00464098434895277,
+            0.02759283408522606,
+            -0.003151134354993701,
+            -0.008637047372758389,
+            -0.0020602785516530275,
+            0.009241150692105293,
+            0.0002063850115519017,
+            0.003000108525156975,
+            0.008281932212412357,
+            -0.03562577813863754,
+            0.011706545948982239,
+            -0.019576217979192734,
+            0.0024694770108908415,
+            0.003459308762103319,
+            -0.007183933164924383,
+            -0.014865843579173088,
+            -0.005857354495674372,
+            0.022841641679406166,
+            -0.003224606392905116,
+            0.021078312769532204,
+            0.002885818714275956,
+            0.016106704249978065,
+            -0.011575928889214993,
+            -0.009208496659994125,
+            -0.0004130251472815871,
+            -0.005351213738322258,
+            -0.007539048325270414,
+            0.010596301406621933,
+            0.0034409407526254654,
+            0.005534893833100796,
+            0.002265388146042824,
+            0.0036654386203736067,
+            -0.0036878883838653564,
+            0.02773977816104889,
+            0.003563394071534276,
+            0.007996207103133202,
+            -0.011567764915525913,
+            0.014082142151892185,
+            -0.017029186710715294,
+            0.0070778070949018,
+            0.005575711838901043,
+            -0.026156047359108925,
+            0.016849588602781296,
+            0.004775682929903269,
+            0.006628811359405518,
+            0.013812744058668613,
+            0.01618834026157856,
+            -0.010408539324998856,
+            -0.014816862531006336,
+            0.024049850180745125,
+            -0.0032041973900049925,
+            -0.015225040726363659,
+            -0.006775755435228348,
+            4.438935866346583e-05,
+            0.006559420842677355,
+            0.00373074715025723,
+            0.0013020879123359919,
+            -0.0006204306264407933,
+            -0.010743245482444763,
+            0.004645065870136023,
+            -0.01934763789176941,
+            -0.0007163524278439581,
+            -0.016449574381113052,
+            -0.015633217990398407,
+            0.018678225576877594,
+            -0.01830270327627659,
+            0.009633000940084457,
+            0.0012275953777134418,
+            0.028588788583874702,
+            -0.011706545948982239,
+            -0.012155541218817234,
+            0.0015102586476132274,
+            0.010849371552467346,
+            0.0067512644454836845,
+            -0.010033016093075275,
+            -0.007583947852253914,
+            -0.014180104248225689,
+            -0.010163633152842522,
+            0.015502601861953735,
+            -0.014498483389616013,
+            -0.014343376271426678,
+            0.008914608508348465
+          ],
+          "embedding": [
+            -0.006359657085039302,
+            -0.024594786241399234,
+            -0.02553538924848095,
+            0.005360188465088291,
+            0.012801870977589839,
+            0.021680505862506104,
+            -0.016668048966001273,
+            0.06860852474581301,
+            -0.007122272841157724,
+            0.03471775000705488,
+            -0.009004297848861675,
+            0.01059974144678391,
+            0.02565604599426148,
+            -0.022544127646041287,
+            0.010616674976540785,
+            0.013658194823359737,
+            0.016400329743363844,
+            0.02858546964917478,
+            0.0067612676434693686,
+            -0.04174737193606879,
+            0.03184155110488616,
+            -0.02223105127585514,
+            -0.05506239667974977,
+            -0.0015307551410555752,
+            -0.013169788751379287,
+            0.017681749722384007,
+            0.01682706317229353,
+            0.019652295823816928,
+            -0.05623980369271269,
+            0.030698582851533197,
+            0.02286327124296613,
+            0.016019032539689178,
+            0.005508113353872315,
+            8.58985091215149e-05,
+            -0.019206292853495906,
+            -0.018439290483495144,
+            0.03475023176950151,
+            0.01018612942141759,
+            -0.0252551158857426,
+            0.0334562130666827,
+            0.020462153537485785,
+            0.02260508249714211,
+            -0.03323524989392951,
+            -0.002094251009530707,
+            0.015177442347686773,
+            -0.00018698464350038317,
+            -0.0006788648344273403,
+            0.0007777616893581039,
+            0.013352398615059647,
+            -0.030751111528744813,
+            -0.0317733818445192,
+            -0.006729064081710058,
+            0.009413204899986514,
+            -0.005712894407943592,
+            -0.038961826880527066,
+            -0.002827692206530345,
+            0.017863300537973137,
+            0.07918455388563496,
+            0.048967158919816366,
+            0.018002566325350088,
+            0.040264962620950695,
+            -0.019715100359659787,
+            -0.02211350500879408,
+            0.004918148608475574,
+            0.01062585305866606,
+            -0.014478993521696027,
+            0.0034983131226364672,
+            0.000766228629514787,
+            0.009667268771502067,
+            -0.01558439890697323,
+            -0.04984718709870083,
+            0.029613955432821887,
+            -0.009686854114630624,
+            -0.015279319238124568,
+            0.007460629676325322,
+            -0.006235979350809034,
+            0.012514387473717085,
+            0.02800435263741497,
+            -0.008688925946672778,
+            -0.01236301543374469,
+            0.029755705020188972,
+            0.01158037219446542,
+            -0.005242392206024511,
+            0.009982679027728814,
+            0.052414361714483264,
+            -0.018788500381726227,
+            -0.020335831196027204,
+            -0.08583959943356621,
+            0.029309228216812346,
+            -0.0249285282010673,
+            -0.018929396555426307,
+            0.007710663193334574,
+            0.03096798035958051,
+            -0.017089977064591828,
+            -0.028015107605405817,
+            0.06740703271128606,
+            0.009398795412354399,
+            0.01118769022826835,
+            -0.03041417460855722,
+            0.014776002883150835,
+            0.004319194067673161,
+            0.022507636639041203,
+            0.0053562788646786845,
+            0.007548072258943124,
+            0.011123374308563128,
+            -0.006924362115585165,
+            -0.011118507147092024,
+            0.009563232643295304,
+            0.00657178473678069,
+            0.08313027447380966,
+            -0.004239370156003264,
+            -0.00930406163946907,
+            -0.039202672821898946,
+            0.009761029930556298,
+            -0.02827698465424993,
+            -0.037914533246900244,
+            -0.04496050911855815,
+            -0.017711307900236353,
+            0.01934435542287117,
+            0.004335173027873153,
+            0.03800509969714242,
+            0.031198127357716314,
+            0.031311207941680276,
+            -0.012184553894588813,
+            -0.058985863670856876,
+            0.041442017057707635,
+            -0.012456189292721711,
+            -0.0040966351720394135,
+            0.010849179519959421,
+            0.005243123049347636,
+            0.0011495799963862647,
+            0.003534183676934487,
+            -0.03139340512003631,
+            -0.023876261883905306,
+            0.008382957569752911,
+            -0.004778283217011976,
+            0.03703067589888386,
+            0.06888630688839406,
+            -0.03993532296478411,
+            -0.009472167678222372,
+            0.025246872942105057,
+            -0.01524567912700368,
+            0.01595917939878582,
+            -0.003736723164442467,
+            -0.030836797784967753,
+            0.0026246446163368533,
+            0.008894300082148884,
+            0.08342856279154155,
+            -0.0007476580521704578,
+            0.025149116238550588,
+            0.004972542633631468,
+            0.034495632546556576,
+            -0.009173329764686424,
+            0.014009381543772768,
+            0.004427261668151028,
+            -0.013426138607902904,
+            -0.022234568880321808,
+            0.01976282364689966,
+            -0.03592433789490043,
+            -0.010918278220831188,
+            0.013527180289641174,
+            -0.013023113052242677,
+            -0.008304770302495363,
+            0.003677063799167178,
+            0.0013981822663219421,
+            0.047001163242600175,
+            0.01828332714887986,
+            0.002483375662343689,
+            -0.06820498187523893,
+            0.05295312038995555,
+            0.025038753504342162,
+            0.025286870198436735,
+            -0.0029544808216316115,
+            0.029982752196635428,
+            0.0005600481523874767,
+            0.0128290258663539,
+            0.009023805227424919,
+            -0.0019219166367750596,
+            -0.04482843936478531,
+            0.04523479691409549,
+            -0.004678574027241186,
+            0.010486850804218904,
+            0.006736229865018014,
+            0.02134813656476443,
+            0.028083972673690036,
+            0.016857413440125445,
+            0.0189082447820146,
+            0.03843690613820022,
+            -0.011922027011229613,
+            0.016193433524453224,
+            0.03828207911467065,
+            -0.0013055128404331705,
+            0.043198163443700956,
+            0.024025562893260542,
+            -0.001257851946749882,
+            -0.011726713433348844,
+            -0.021907984099443396,
+            0.02752198420597673,
+            0.004431866336999851,
+            0.015798806714413912,
+            -0.012910782191382367,
+            -0.03180549180206364,
+            0.004550247372715741,
+            -0.022778272841118514,
+            0.031242218240017532,
+            -0.006106984949693483,
+            -0.006259993190497421,
+            0.02187449047033339,
+            0.009334146847221406,
+            -0.009381278960016145,
+            -0.020888853823931134,
+            -0.007502374908776156,
+            -0.010390018386728482,
+            -0.04316693435221676,
+            -0.006274513730720055,
+            0.019152443937064258,
+            -0.01668984824845791,
+            0.018573194036505578,
+            0.0035035262183249925,
+            -0.019779911381789077,
+            -0.0027495798165085993,
+            0.02425481236315913,
+            0.006857723545270935,
+            -0.014733535699087731,
+            -0.026326146471941727,
+            -0.019906660231056833,
+            -0.001033294559612807,
+            0.0159154063026895,
+            0.035649534442431415,
+            -0.026210592096240003,
+            0.014397127229517285,
+            0.005114311681443673,
+            -0.0033104236005207077,
+            0.00324988235167305,
+            0.03764004154553922,
+            -0.017822437760737605,
+            -0.008579701063708334,
+            -0.00032410984200965487,
+            0.018450351071676443,
+            0.017283379764846195,
+            0.006384411611332382,
+            -0.005115090309123598,
+            -0.030590047062583125,
+            -0.011776211359193503,
+            0.01183623914060493,
+            0.0016564847796208331,
+            -0.009530766634361575,
+            -0.004810598258154945,
+            0.03876242515526699,
+            0.027076785774401667,
+            0.005324763136165989,
+            -0.03416412480913939,
+            -0.007943016733586697,
+            0.002114307650456928,
+            -0.02961449347300628,
+            0.05142732770518828,
+            0.05566026433842563,
+            -0.0030729309664974704,
+            -0.00938186948393074,
+            -0.04547908981236149,
+            0.0651216359692231,
+            -0.01813722445944094,
+            -0.0482650853560993,
+            0.044045155892545554,
+            -0.014272653003441165,
+            0.013740652306746851,
+            0.04563732439184395,
+            0.02126513340521316,
+            -0.0054845632439384905,
+            -0.011635625101262898,
+            0.030992082069685376,
+            -0.010051935721068779,
+            -0.050967258971854425,
+            -0.009827942095033494,
+            0.013373878556654464,
+            -0.013017627321534534,
+            -0.009361001542607812,
+            -0.01925867665425919,
+            0.020329052915436022,
+            0.023020741670660188,
+            0.019387043947695172,
+            0.005628632318152202,
+            0.018451555161952665,
+            0.016226845400848455,
+            -0.01384033112836216,
+            0.004762589697426251,
+            -0.012718476561725138,
+            -0.024222765127494463,
+            -0.037487144314010595,
+            -0.019971524166659278,
+            -0.015034588610516958,
+            0.009483936496215338,
+            -0.014301924283553591,
+            -0.0024139665139412125,
+            0.013361045353529458,
+            -0.0005870798721107379,
+            -0.011192319340879789,
+            0.013927695434391159,
+            0.021337539669342293,
+            -0.027300952495815437,
+            -0.04765670695199488,
+            -0.00561706224251562,
+            0.0082776639906139,
+            -0.0030984973749633724,
+            -0.026269016057688548,
+            0.007040524859505573,
+            0.0017859008072463442,
+            0.020091542517503808,
+            -0.018716761130867725,
+            0.0035283051960672963,
+            0.008519383548064053,
+            -0.015247417050336866,
+            -0.016917626868166634,
+            -0.018580550690103854,
+            -0.024697180580294884,
+            0.034202028316737125,
+            0.0204395003463227,
+            0.030392895534267717,
+            -0.04527669147753553,
+            -0.018359428092075348,
+            0.05494107499558624,
+            0.015234754907198422,
+            -0.05066008106884831,
+            0.031139824660661085,
+            0.005016615710206006,
+            0.0019000453624929544,
+            -0.006094364589229031,
+            0.0637324238017742,
+            -0.02711381489292883,
+            -0.02187418906973867,
+            0.004601744864532875,
+            -0.04141121277496072,
+            -0.026689585608019588,
+            -0.005568100750837504,
+            0.01291019036002459,
+            -0.031853041426355644,
+            -0.02815800259372657,
+            -0.007964861641790374,
+            -0.010933515363060793,
+            0.013726639077745378,
+            -0.023956800669647334,
+            -0.015672058590455337,
+            -0.023642025374517057,
+            -0.01288435863544158,
+            -0.0042516881552192385,
+            -0.011162214160460695,
+            -0.02412733256710621,
+            0.010088528028211243,
+            0.03310175458095482,
+            0.023064360903748363,
+            0.001996159684818467,
+            0.009478229572933101,
+            0.014478772877286076,
+            0.00371783700962415,
+            0.007864103610320116,
+            -0.048511515124609,
+            -0.031398754206534776,
+            0.020227872561659397,
+            0.012159811587094092,
+            0.0172163448580163,
+            0.009578952796095119,
+            -0.04154545961112031,
+            -0.05095080393682759,
+            0.02968486144855133,
+            -0.03932395023720674,
+            -0.02063219739061335,
+            0.007512423249865194,
+            -0.011911709468952548,
+            -0.00810951734088105,
+            -0.0008079327452253181,
+            -0.01600701450676789,
+            0.023984909975645632,
+            -0.02418072541081482,
+            0.0033984265728439398,
+            -0.0104343782811971,
+            0.01588977364765743,
+            0.05109529246167992,
+            -0.027800269136366997,
+            -0.027889034481533438,
+            0.03706813210665186,
+            -0.02058120652429308,
+            0.030801396468451574,
+            0.03246127847621613,
+            0.008355517405358826,
+            -0.008605467586247932,
+            0.017693344563968573,
+            0.014499487024471321,
+            0.012431277080478217,
+            -0.040103870071440256,
+            0.017328848567062113,
+            0.02723900253727976,
+            -0.054909074068976875,
+            -0.008547596270650966,
+            -0.00777964838164648,
+            0.027134666572242665,
+            0.02765036594970255,
+            -0.002207173084941375,
+            -0.016838740117653725,
+            -0.028289233739722936,
+            -0.019433376533829132,
+            -0.011053380458946764,
+            0.03247641283550457,
+            0.0007514989009289282,
+            0.007919574219000797,
+            0.001316042436488591,
+            -0.0017934342041762042,
+            0.01924145734197619,
+            0.015260199553572459,
+            0.048259602293134723,
+            -0.03383050622747848,
+            0.0021712225589426266,
+            -0.03508567715696004,
+            -0.04578731401250471,
+            -0.008768228935292693,
+            0.010203318819548585,
+            0.024135236371743855,
+            -0.007467140460133838,
+            0.0028907962385387818,
+            -0.007582083854422456,
+            -0.005384477718912037,
+            -0.01867978482547249,
+            0.01641191108381631,
+            0.037168089880543244,
+            0.0031992495663980464,
+            -0.008533890994695162,
+            0.000778843942291118,
+            -0.004620258538699103,
+            0.010572690335163528,
+            0.033233014627100904,
+            -0.030497362482602697,
+            0.02974571152880676,
+            -0.004274442492343003,
+            0.025341347125910928,
+            -0.04307748954707389,
+            -0.018069609916024403,
+            -0.018491122727274337,
+            -0.02069307802429807,
+            0.05372967226264594,
+            -0.030768978301976772,
+            -0.03662045886993282,
+            -0.003205991669059688,
+            0.016815511161004534,
+            -0.0014152644009276566,
+            -0.0016015439520090104,
+            -0.03984716970293867,
+            -0.010828260334550217,
+            -0.05894985435251326,
+            -0.01638035185618134,
+            -0.021819442096404948,
+            -0.005315102328745982,
+            -0.06251579337157112,
+            0.0033713876515840233,
+            -0.07815974854883057,
+            0.001484860071649606,
+            0.04539427129131382,
+            -0.023530943197353476,
+            0.05142250021536681,
+            -0.024746600229063383,
+            -0.0005885740455257141,
+            -0.044698536284421976,
+            0.0026481001040031785,
+            0.02389178603305878,
+            0.051387657872255674,
+            -0.002949198236157656,
+            0.010674377196609455,
+            0.013322075519401089,
+            -0.026245090721950647,
+            -0.01036724558050635,
+            -0.005321884180696542,
+            0.005958798671856245,
+            0.013755178085972732,
+            -0.0074734329537091675,
+            0.038368016046563674,
+            0.010941421833174982,
+            0.03443115781919206,
+            0.034623617136788874,
+            -0.004361926531008214,
+            -0.03348681614127998,
+            -0.00632870804659212,
+            0.024433331840566692,
+            0.0321998807702602,
+            -0.02899983280320611,
+            0.005958471722097962,
+            -0.015602447981516634,
+            0.015633217539747047,
+            0.006321931116948646,
+            -0.011827541320878442,
+            0.02372204564999124,
+            0.015673426811623124,
+            0.006103300725621424,
+            -0.005165006670545588,
+            0.013589526056189892,
+            -0.025411864292353578,
+            0.03041791947375232,
+            -0.004456865298584878,
+            -0.04276058054583262,
+            -0.0024736078733106825,
+            -0.020059309867945227,
+            0.024883721794000956,
+            0.015578237413840115,
+            -0.00536523508503184,
+            -0.005290645798641171,
+            0.008242987948511412,
+            0.020469675635821673,
+            0.006668354647245447,
+            0.00017729517736814364,
+            0.02373604785121342,
+            0.029915687122900485,
+            -0.02432635861789771,
+            -0.029540533943782886,
+            0.008151297444985344,
+            -0.005730733861939129,
+            0.011318816579801282,
+            -0.02644855229221747,
+            0.00033932872948409137,
+            0.0035505684154028996,
+            -0.03570104343808711,
+            0.03408366016010711,
+            -0.009199700348630527,
+            0.005857598530788056,
+            -0.009538177736472163,
+            -0.0017768806148210118,
+            0.0027410153099351694,
+            0.03898108970435115,
+            -0.007367623392751247,
+            0.009827254226889936,
+            0.015465895035342481,
+            0.000981604396724216,
+            0.05143083558797524,
+            -0.027350877485564314,
+            -0.02183036962301852,
+            0.01885468378168636,
+            -0.007285239007489837,
+            -0.016846837665812908,
+            -0.020546383007536074,
+            -0.011863943814684079,
+            -0.016293192750336218,
+            0.01976516340568883,
+            -0.02387328798681798,
+            -0.011892336568452852,
+            0.007170651391639504,
+            0.0013725014050721622,
+            -0.009809669068563384,
+            0.0016617183704652932,
+            -0.006307590349798603,
+            0.02037693835615797,
+            0.031753656615094446,
+            -0.024892675105483313,
+            0.03577919637821846,
+            0.03178303188315581,
+            0.017260898395251155,
+            -0.02102289690701718,
+            -0.009661035772073294,
+            -0.00048542689442823136,
+            -0.014694631967051945,
+            -0.0326379186158631,
+            0.025443206465204822,
+            0.017194626456982255,
+            -0.011381421003142453,
+            -0.03680783761649673,
+            0.0017216115811536721,
+            0.022819859205829785,
+            -0.013600989058799166,
+            0.028141657788496834,
+            -0.01869047298692401,
+            -0.01366833755869951,
+            -0.00782959271979894,
+            0.031227914441058244,
+            -0.05252392878125891,
+            0.016957842026348195,
+            0.00020880696574616416,
+            0.001154370575234767,
+            0.010892067708194324,
+            -0.008368819756628046,
+            0.012224564683325483,
+            0.031700389750709,
+            0.018063405869147564,
+            0.0010794115307202266,
+            -0.002783668216641561,
+            0.01418234057248712,
+            0.017773269840982066,
+            -0.015653443355569053,
+            -0.000497740485657824,
+            0.019890882604100452,
+            0.022672054781919238,
+            -0.00521797370595221,
+            -0.016686942830774276,
+            -0.018443538351123584,
+            0.025270057639201107,
+            -0.010354431070738671,
+            0.03484047753830186,
+            0.04594785214094975,
+            0.0036441959528283004,
+            -0.05695476763071743,
+            -0.009412209619190693,
+            0.0010371110229966086,
+            -0.01099343927053564,
+            0.01606863921935641,
+            0.012470660511773616,
+            -0.03237187923799263,
+            0.006388672213167589,
+            0.012946570716567364,
+            0.022152554488829627,
+            -0.02540317621005867,
+            -0.023183252921885226,
+            -0.0030984019855787655,
+            -0.007890467768631811,
+            0.018648760402624687,
+            -0.008653405980661855,
+            0.0006228315253506687,
+            0.009672526897422602,
+            0.0005436494014569835,
+            0.011632326145022935,
+            0.005655942007108228,
+            0.021736827172951,
+            -0.001256608167230203,
+            0.013124413886287987,
+            -0.01986808920858981,
+            -0.0270450339809516,
+            0.006500423802582728,
+            0.04438242119211055,
+            -0.014813399694814117,
+            0.030621709833336064,
+            0.0092171224268983,
+            0.016433384739642534,
+            0.024875072913970443,
+            -0.00891470467241686,
+            -0.014124009014664287,
+            0.00845777541223868,
+            0.009019007848927453,
+            -0.04147358041382793,
+            0.012513030150663913,
+            -0.01710243399313324,
+            -0.033144939719206276,
+            0.01904674924133127,
+            -0.00534122554342279,
+            0.012367320751926314,
+            0.004270315800911569,
+            -0.002886460711688153,
+            -0.025696286516581048,
+            0.014137901708875033,
+            0.028135087538950946,
+            0.021797869775098042,
+            0.0296173052503884,
+            -0.012757078368361444,
+            0.0005460601780756187,
+            -0.008748930817524899,
+            0.005470975782187157,
+            0.04187083128854477,
+            0.0008687182830616019,
+            -0.0008032802520818105,
+            0.01507726552487313,
+            0.024945744517309208,
+            0.0376130540370386,
+            -0.015415560911023553,
+            -0.03407353556457147,
+            0.01564969972264396,
+            -0.010104443707865262,
+            0.018855792875436674,
+            0.005503259932317811,
+            0.0049634097823351596,
+            0.0038600156268000702,
+            0.019312419743287703,
+            -0.006635193789316492,
+            0.018798045806429395,
+            -0.02500983683871332,
+            -0.028785757874583755,
+            0.01829411751138161,
+            -0.005277972443258226,
+            0.03338623299668019,
+            0.016225885095995672,
+            -0.029522280454919586,
+            -0.01712097344863975,
+            0.01553706187924322,
+            -0.008465766956343768,
+            -0.024842700926275538,
+            0.01895228828512998,
+            0.007076862934523977,
+            0.008956105593078965,
+            0.0024373779389833218,
+            -0.01106503696703573,
+            0.01125466547367417,
+            -0.000818223040067914,
+            -0.015322790791698395,
+            0.014157131555941659,
+            -0.017853386731281715,
+            -0.014994640991761465,
+            0.01775991780643474,
+            -0.037498078532902435,
+            0.009570197712419421,
+            -0.007687297543935123,
+            0.009050494195378254,
+            -0.004222041469820445,
+            0.007460383234429904,
+            -0.0010804379186060264,
+            -0.014987857934522812,
+            -0.014903749199810126,
+            0.03228485430019625,
+            0.022096121825475306,
+            -0.000410816936759558,
+            0.006728733918931589,
+            -3.971636461500232e-05,
+            -0.026719848032785824,
+            -0.011253751790407687,
+            0.02245467425830765,
+            -0.005664635254398481,
+            0.005134496095445943,
+            -0.01109590855846966,
+            -0.0149616794609307,
+            0.0019726017255571673,
+            0.032668637270108894,
+            0.06066635618119076,
+            0.012001732730659747,
+            0.024422093585231484,
+            0.0189499396761026,
+            0.02119276033547528,
+            -0.008489818917377516,
+            0.006413901834269739,
+            -0.03097413287789443,
+            0.03509225573986143,
+            0.005392594635384506,
+            0.03108937318306421,
+            0.026664331248813712,
+            -0.04722525921137681,
+            -0.0058539171830594774,
+            0.017087531454837677,
+            0.022673712717885323,
+            0.0159020681868943,
+            0.0008382498574190522,
+            -0.008938174268517353,
+            0.035650940044285984,
+            -0.03315250299348084,
+            -0.023647111263155,
+            0.005059382024092534,
+            -0.01439537467317451,
+            -0.010488359956731476,
+            -0.006785295580877329,
+            0.02776449681317373,
+            -0.005584052419598926,
+            -0.004582019433008779,
+            0.009697064252638535,
+            -0.014400206558190682,
+            0.008165676294736836,
+            0.0007624778531368436,
+            0.008681263897172326,
+            0.039249866981334126,
+            0.0005918063050375113,
+            0.024275699182767528,
+            -0.004138313072843555,
+            0.011079529739650594,
+            -0.0158972266904151,
+            -0.016985058543605,
+            0.02276982363172991,
+            0.00351101120496393,
+            -0.03244126813432563,
+            -0.00768819222024939,
+            0.00928180979884261,
+            -0.0013793404539641824,
+            0.005266058566444064,
+            -0.00503433670609399,
+            0.003747448261046857,
+            0.03286230666764075,
+            0.0022383592721685153,
+            0.008606190020480513,
+            0.013593947677257625,
+            0.00030951915677929204,
+            -0.01490307680740127,
+            0.013254591138113545,
+            -2.1379308136707722e-05,
+            -0.012900073369514622,
+            0.03357519186606058,
+            0.006705902094864628,
+            0.04243347200485736,
+            -0.03302953674711988,
+            0.009031534935013593,
+            0.01822227379789576,
+            -0.008491770335368761,
+            -0.02725412856390648,
+            -0.03630788717786838,
+            0.008259145747740535,
+            -0.019003260334340084,
+            0.015781275447792516,
+            -0.010039277287105153,
+            0.029665124310477384,
+            -0.03563370794521629,
+            0.004714858110268169,
+            0.004053913355055313,
+            0.0009503673086552651,
+            0.02068360255395961,
+            0.018524093113914366,
+            -0.002785698934204024,
+            -0.031443835341186485,
+            -0.02066365433385339,
+            0.006869321907853112,
+            0.02431260052207595,
+            -0.010367048093621498,
+            -0.005612149485859478,
+            0.0006316058139530722,
+            -0.024369562712568606,
+            -0.016845124984022828,
+            0.030359419843284823,
+            -0.040974427176153246,
+            0.005252911449822132,
+            0.04272936053727608,
+            -0.04599092715016919,
+            -0.03802288716943514,
+            -0.002314988728575789,
+            -0.024124014894379466,
+            -0.006360771882090754,
+            -0.002068030981218464,
+            0.016669991113175926,
+            0.0036232707799735537,
+            -0.028763764013482795,
+            0.00943401978066451,
+            -0.005173988601101547,
+            0.003971420424839974,
+            -0.0031669592494977393,
+            -0.02216336363004636,
+            -0.01657772032717298,
+            -0.02531962163186398,
+            -0.035174102848865116,
+            -0.0015903434876311088,
+            -0.005158703150821607,
+            -0.028703075279214722,
+            -0.03941940849540492,
+            -0.021696000925262537,
+            -0.0160221488026189,
+            -0.00843213135542359,
+            0.040641882367348774,
+            0.021319060666253945,
+            0.019336844555345886,
+            0.0025252167889382917,
+            -0.020346900609125973,
+            0.049440028814233025,
+            -0.007642182460707816,
+            0.035628691886108926,
+            -0.0017882888000877158,
+            -0.013461669667950893,
+            0.00455558389906464,
+            0.03825428026120858,
+            -0.014248434654950971,
+            -0.0009902062564027718,
+            0.00027811655483232545,
+            -0.0011866253757890136,
+            0.0008535155950871286,
+            0.015433650396777696,
+            0.001307651096343185,
+            -0.007112199543314309,
+            0.030380673552683906,
+            -0.012803247194895328,
+            0.01637501752192592,
+            0.018350814869962304,
+            -0.0013359719137099667,
+            0.008064546833360553,
+            -0.021430659268086814,
+            -0.00731572118780415,
+            0.0044578576331236325,
+            0.008422210247967642,
+            -0.015096885054883586,
+            -0.009964588397149773,
+            -0.02359494467299634,
+            -0.01174145797396426,
+            0.0023855749166744366,
+            -0.004559202896782688,
+            0.004902138778592298,
+            0.034595817216889944,
+            -0.016068744500427584,
+            -0.035106033294403614,
+            0.009997207702122002,
+            -0.02663088875167486,
+            0.02222306406130687,
+            3.983384942889326e-05,
+            0.006838221769859366,
+            -0.0023434342406307616,
+            0.001765445386886839,
+            -0.00169484278667461,
+            0.002862331185439823,
+            -0.018217975204395134,
+            0.0031859734246840257,
+            -0.0032119988065673565,
+            0.0009052950782312657,
+            -0.0012459653052875638,
+            0.00687339892509406,
+            0.006332799551164718,
+            -0.0008019538898592668,
+            -0.002968451740267742,
+            -8.541086355097327e-06,
+            -0.020561816931299912,
+            -0.021642512405352372,
+            -0.028259049261410905,
+            -0.000989887278779396,
+            -0.012359733017834338,
+            -0.027804394906433664,
+            0.011443385424045703,
+            -0.006872684999639775,
+            0.008327844771213648,
+            -0.008969527138034904,
+            0.006219994886600349,
+            -0.008415106130309062,
+            -0.0004143996206881767,
+            0.013119286511794273,
+            -0.02925209282917943,
+            -0.02096907906398536,
+            -0.03096513357712367,
+            0.01762223841790055,
+            -0.00925741670612824,
+            -0.002656382260603827,
+            0.0025223527960181835,
+            0.005092659166535706,
+            -0.033606542487570154,
+            -0.006653438998497412,
+            0.007313682256676618,
+            0.009547290751154595,
+            -0.03287788897629887,
+            0.02446728061355753,
+            0.017023222076526946,
+            0.01874233066565012,
+            -0.02417566312664911,
+            0.04049827067897005,
+            0.014426879325647042,
+            -0.021740606525480655,
+            0.016712907564723732,
+            0.027385150577780248,
+            0.03177163221499527,
+            -0.03469098547555545,
+            0.004481376248622418,
+            -0.01518200063132508,
+            0.018918680020516044,
+            -0.025284921244460287,
+            0.020625170445627613,
+            -0.016707629602243063,
+            0.021278719363178687,
+            0.03520334001264928,
+            -0.005524369131116028,
+            0.03608133014180745,
+            -0.019234548780647943,
+            0.0059985801736444555,
+            0.004041778461022664,
+            -0.01645503683802124,
+            0.01024694092614158,
+            -0.0042700351372843386,
+            0.0006059159967286885,
+            0.009406090340935747,
+            0.019133229156115282,
+            0.019007219188041775,
+            -0.04442208547053466,
+            0.028588965123434923,
+            0.00804120101594811,
+            0.021472086070876265,
+            0.014177820346835521,
+            -0.02708807970982664,
+            0.004742981711172693,
+            0.002072673190279628,
+            -0.004659713164250794,
+            0.0061167904730676514,
+            -0.013796604518930328,
+            -0.034540857808309226,
+            0.006338792319814448,
+            -0.018918703778187926,
+            0.015036937364271984,
+            -0.009026670935442384,
+            0.0011667525636770179,
+            -0.01152652811400482,
+            -0.011938679695598347,
+            0.0006170592505446954,
+            -0.005495151258143897,
+            -0.0008075736908949407,
+            -0.03171721592827769,
+            -0.04427658539412448,
+            0.02036132271908089,
+            -0.0021221965958423804,
+            -0.04320396743189005,
+            0.004111179168317842,
+            -0.025352428814134187,
+            -0.01865918186335911,
+            -0.011237699157960728,
+            -0.025047923887906474,
+            -0.04219630696740606,
+            -0.011108496740166623,
+            0.007785006066683012,
+            0.016810642610918525,
+            -0.009081815651299525,
+            0.040965574143584864,
+            0.0073944364393447965,
+            0.015313576815177263,
+            -0.017731360025694955,
+            0.026137777790243826,
+            -0.019314392060123,
+            -0.03108486330987073,
+            0.02676829566273746,
+            -6.259161089416372e-05,
+            0.003053748984540415,
+            0.014358356168574803,
+            2.8186924071913628e-05,
+            -0.006181789640967465,
+            0.01947730374003864,
+            -0.028739300217145797,
+            -0.023460503178221397,
+            0.0012460072421725688,
+            -0.008850395570726385,
+            -0.016271242375881602,
+            -0.035896509136022194,
+            0.005227881053956234,
+            0.03324337105109496,
+            0.004970419987430122,
+            0.008273497228454482,
+            0.01814500594735864,
+            -0.02229593571615308,
+            -0.024230675455380678,
+            0.000639112529814716,
+            -0.010266640429960994,
+            -0.010257247303470066,
+            -0.021302483296731352,
+            0.019150757066356872,
+            0.01651375496287087,
+            -0.00390189952706486,
+            -0.006971692709239087,
+            -0.010621003777733253,
+            -0.01918206725910585,
+            -0.02020345113211036,
+            -0.013827322378214754,
+            -0.017980206764311026,
+            -0.034153741666571154,
+            -0.032195001867229935,
+            0.03225205339201753,
+            0.026354817588919596,
+            0.0014076409241492998,
+            -0.007959081015805435,
+            -0.023070419006792155,
+            -0.004483328183581872,
+            0.02879516595345542,
+            0.002549934782150634,
+            -0.0001541849716285838,
+            -0.00830090759167373,
+            0.015341095409025125,
+            0.00835695019244165,
+            0.005107844984733648,
+            -0.06676974053142512,
+            -0.006027540973808174,
+            0.01655283585235268,
+            0.01351155740684989,
+            -0.002359114145814062,
+            0.02591301169926259,
+            0.01108099264396926,
+            -0.015361038608114896,
+            -0.019765743462262668,
+            0.035195761515422364,
+            0.02408910733949848,
+            0.023468767772473746,
+            -0.0034492999385544727,
+            -0.004472311490618405,
+            0.007030833258457338,
+            -0.010684252427867153,
+            -0.013326964647936429,
+            -0.004558513842504658,
+            0.026925146859337397,
+            0.016099443021514896,
+            -0.0064331198280825575,
+            0.010848931851939124,
+            -0.0008320209125061849,
+            -0.019444491995041537,
+            -0.001841330829156697,
+            0.008730275423516445,
+            0.0012974082030866743,
+            -0.014611902493400719,
+            -0.005911807768916134,
+            -0.019130604101278263,
+            0.024719468317569562,
+            0.0209456329723249,
+            0.03381727436368338,
+            -0.013237902591972769,
+            -0.0003695142331921278,
+            0.005410360755058108,
+            0.0002841632253252816,
+            0.0023352645111704958,
+            0.004801123870325242,
+            0.02623649813035263,
+            -0.029010710171522985,
+            -0.020741774066460415,
+            0.0068426980682519505,
+            0.013445270043414486,
+            0.0022744032984779294,
+            0.033840485942696216,
+            0.035040080489222324,
+            0.004166066423508316,
+            -0.010046666684988869,
+            -0.006148765564658713,
+            0.007071740742464309,
+            0.01865897335451031,
+            -0.023267411753900076,
+            -0.0005936657905496225,
+            0.004833030519903764,
+            -0.004229582648167378,
+            0.03620610659071198,
+            -0.009961238880412731,
+            -0.007197670122096852,
+            0.022212791947347127,
+            0.09819205436926473,
+            0.00047695621444943017,
+            0.0025354521225913395,
+            -0.029358210234742176,
+            -0.023614016211567963,
+            -0.0012564198366457442,
+            0.00242680708457405,
+            0.00830167274308547,
+            0.028076260517106013,
+            0.0004852400049405906,
+            0.011579274037419117,
+            0.02773972145320728,
+            -0.017217305534161797,
+            0.006752070301141707,
+            -0.011376051474975524,
+            0.024577071515205502,
+            -0.012694034026935636,
+            -0.0019809719792138673,
+            -0.016512649783050146,
+            0.015907402258634816,
+            0.004556574687833768,
+            0.004907510649166062,
+            0.011347630984408937,
+            0.009824484201337644,
+            -0.02650731900265866,
+            0.01756773113723491,
+            -0.005671986127099093,
+            0.004449611508541289,
+            -0.027841346572384462,
+            -0.020124233970502267,
+            0.0025986391930124847,
+            -0.0014440758308989809,
+            -0.009579444248630085,
+            0.006005274872617889,
+            -0.0024984235450999127,
+            0.03765664613203687,
+            -0.03765496603611941,
+            0.01409350051861866,
+            0.0066453175608372555,
+            -0.0015390841089599882,
+            0.011806653942640485,
+            0.017306914776093522,
+            -0.008206256939290151,
+            0.02589800448884443,
+            -0.00798213351353373,
+            0.009846274212000523,
+            0.015230070803055296,
+            -0.03449531102975745,
+            0.009174739184806312,
+            0.010145762060174144,
+            -0.024165660737765746,
+            0.012146877603301336,
+            -0.013784371016298891,
+            -0.004818759788054113,
+            0.005930446700613062,
+            0.007349837554764034,
+            0.010733731974181452,
+            -0.02295571464086674,
+            0.016176424826629898,
+            -0.010947841926210105,
+            0.0037766665765835445,
+            0.013559846998199008,
+            0.024892849808979323,
+            0.0012426907454129122,
+            0.0005950515136086193,
+            -0.01379469431504576,
+            0.005534724429523084,
+            -0.015667467221535825,
+            -0.022401586867694584,
+            -0.014192041775234079,
+            -0.006366846766730035,
+            0.01029598687444803,
+            -0.003972386708834858,
+            0.016971476088808737,
+            -0.010309648640697812,
+            -0.010645860303528099,
+            0.0037818310327362103,
+            0.01040806357838676,
+            0.008316605962234849,
+            -0.02855914079113881,
+            -0.005676310556150579,
+            0.004961084138163161,
+            0.00029285971503001265,
+            0.009592497954559435,
+            -0.014556073011576487,
+            0.013689526487160009,
+            -0.013290113354287095,
+            -0.0010726537498214612,
+            0.013201860397060236,
+            -0.012443552043930505,
+            -0.004012305744087929,
+            -0.01592905426965635,
+            -0.028487066798154617,
+            0.008462143660968396,
+            -0.01365614159523485,
+            0.001076882584445294,
+            0.0019503240125505646,
+            0.004730085646042709,
+            -0.016919069570642004,
+            -0.007860046623166481,
+            -0.033515448866521046,
+            0.011065416478687504,
+            0.02314589687882036,
+            -0.006435733825418645,
+            -0.030351727858789472,
+            -0.009546333526852862,
+            0.011327489786114963,
+            -0.0071765122743172715,
+            -0.01676736383051139,
+            -0.0029390038145944096,
+            0.007617910650439855,
+            -0.008174620754097493,
+            -0.00508943241317381,
+            -0.0025173969010414098,
+            0.0217328037301181,
+            0.016094975605203832,
+            0.01769642142712967,
+            -0.021400862603034036,
+            -0.00976992221747187,
+            0.026043879119776864,
+            -0.03275550610951898,
+            0.012441465975914529,
+            0.024587446266013183,
+            0.003446527051428522,
+            0.005492228054838714,
+            0.0030377554109693714,
+            0.005459159163772825,
+            -0.006985179226831048,
+            0.019925583798328282,
+            0.010870421035240845,
+            -0.019709062313471467,
+            0.005385443078244229,
+            0.02086085539375208,
+            0.0037248737640193097,
+            0.012892299156840998,
+            -0.017546870090541213,
+            -0.0020341807891543716,
+            0.015594779287017958,
+            0.04816214672670215,
+            -0.0032851797178091685,
+            -0.010450846098750195,
+            0.029513742590677183,
+            0.023526004023637855,
+            0.005373455940309873,
+            0.0012120777808635214,
+            -0.02525630456877349,
+            -0.013075130701774785,
+            0.01962235086017781,
+            0.016944870494393874,
+            0.013484925533584295,
+            -0.0023255656480479004,
+            -0.019236081983457366,
+            0.021520156296467717,
+            -0.01701733319731161,
+            0.012422659010697477,
+            -0.023764941037007197,
+            -0.001490790833263327,
+            -0.0035419008954878724,
+            0.021998949056452454,
+            0.0021819635638621092,
+            0.008891505304514552,
+            0.024354552127458608,
+            -0.006978548952551799,
+            -0.019526615985982355,
+            -0.0051799020400150305,
+            -0.014578953088901798,
+            0.022506635912517146,
+            -0.024117708614271965,
+            -0.007008641019892289,
+            0.011379795968360019,
+            -0.012362093847309038,
+            -0.0102534051092702,
+            0.001164934138876502,
+            0.017686807394319725,
+            0.0009423993435129515,
+            -0.014784508280754078,
+            0.01944445593807992,
+            -0.0011359271135469456,
+            0.010410915597949927,
+            -0.01921419010977189,
+            0.009390638453648025,
+            -0.016682442516992777,
+            -0.007849184048209236,
+            0.010343008873053859,
+            0.0012604521199837046,
+            0.003948300703645506,
+            0.002592490121754231,
+            0.007773437307969315,
+            -0.03355658426684504,
+            0.022695505487314516,
+            -0.010751606023585792,
+            -0.003538523688786008,
+            -0.0014125330167969218,
+            -0.01678692701672328,
+            -0.0006918162525881059,
+            0.005816502852598842,
+            0.010878358093847386,
+            -0.016045147059003256,
+            -0.00873270513444376,
+            -0.04920244395446931,
+            -0.030314009649638693,
+            0.023156333024481057,
+            0.013733657599720879,
+            -0.010971506342695331,
+            0.018852159415605117,
+            -0.023760853159532587,
+            -0.010587249404421732,
+            0.0011379869499042905,
+            0.023319856329056813,
+            -0.03539290744576506,
+            0.01215369703816749,
+            -0.014493682360886549,
+            -0.02039288669534806,
+            0.009070984596830142,
+            0.011445354735283254,
+            0.0007259852045513916,
+            -0.0011514627555568814,
+            -0.029350945123925453,
+            -0.010246845821727087,
+            0.02391068177425878,
+            -0.00861718589055971,
+            -0.004723044986826333,
+            0.024716520302409263,
+            0.0033721801437248873,
+            -0.012524663490691517,
+            -0.017955758068233754,
+            0.021612671655463236,
+            -0.026077203994996058,
+            0.007645944309444457,
+            0.010032183011253845,
+            -0.0173991263874487,
+            0.021649599230588117,
+            -0.007529701797485131,
+            0.020327763440441377,
+            0.004227390938169872,
+            -0.011249233530251522,
+            0.020352283974471443,
+            0.013644443758970054,
+            0.004216958800094986,
+            -0.004081669404784771,
+            0.006869353098014179,
+            0.02212732880780841,
+            -0.011021739614266063,
+            0.02219580717345023,
+            0.028777213295471067,
+            0.00017328191188631918,
+            0.03260179554743322,
+            -0.021287655520443285,
+            -0.03159728983004563,
+            0.021740163175540124,
+            -0.01035146858119169,
+            -0.021564906188712388,
+            0.028914245422221707,
+            -0.013674731836508768,
+            -0.00786476052825481,
+            -0.0166718429993922,
+            0.00533027311342322,
+            0.016680336311088843,
+            -0.005246190451159961,
+            -0.014360934066628332,
+            0.0029628103834216345,
+            0.019809487418628386,
+            0.014587234544007958,
+            0.029780009648869975,
+            -0.010778199815920887,
+            0.035180242185168895,
+            -0.002320576872571577,
+            0.008990157317473909,
+            0.012415465232073404,
+            0.006489435152972723,
+            -0.011251531176923445,
+            0.013501894212117511,
+            -0.003446759011266526,
+            0.008230548711588447,
+            -0.001449023474280409,
+            -0.035111679577202855,
+            -0.013709382199848678,
+            -0.04709294504969336,
+            -0.0023752976043235697,
+            -0.0025098360125255215,
+            0.0030975417740913154,
+            0.09736815869389318,
+            -0.005723049647039619,
+            -0.011988817571623457,
+            0.001625059666070763,
+            0.009611697638805556,
+            -0.0009896494420056814,
+            -0.007606836952841873,
+            0.015973152482488318,
+            -0.01343292635775299,
+            -0.02140928016038583,
+            -0.010532701190806147,
+            -0.01288594454368977,
+            0.012079198190522715,
+            0.015929716343161652,
+            -0.0005947359502228821,
+            -0.015207331663761012,
+            0.006524010655124294,
+            -0.004566385899752684,
+            0.0064717704997400245,
+            0.008647099975176308,
+            -0.010414668624343924,
+            -0.0071698879102925555,
+            0.030899143952119196,
+            -0.025301269549419674,
+            -0.024361748354052457,
+            -0.0010579931222861095,
+            -0.009055796737486008,
+            -0.010335459737012047,
+            -0.002692653633719985,
+            -0.0017679501322462056,
+            -0.010519994985692081,
+            -0.010019865263564572,
+            0.006537076627706128,
+            -0.015472130358273033,
+            0.004812857776601667,
+            -0.0047457011269495634,
+            -0.013296069496526007,
+            0.017054968086538724,
+            0.022563785357424853,
+            -0.02841491200807702,
+            0.0006755428946168761,
+            -0.017422510783648978,
+            0.012941211381006863,
+            0.008866069010589792,
+            0.006049618782357251,
+            -0.022451528209097505,
+            -0.023622906582403243,
+            -0.015019272754405791,
+            -0.01927535750568903,
+            -0.024973490129133732,
+            -0.009073381957684092,
+            0.0067692803507600945,
+            -0.001267339788153158,
+            0.021918281111033645,
+            0.000816538714279555,
+            -0.01812545613159305,
+            0.0061084913570894905,
+            0.000568371220952269,
+            0.018292497018668842,
+            -0.027468677288882186,
+            0.0033806422195878524,
+            -0.02307582511848465,
+            0.033508234091571124,
+            0.0039195050746594975,
+            -0.026680111514794824,
+            -0.021558668725755275,
+            -0.003311830287529699,
+            0.018994577319066473,
+            -0.0025745912019675805,
+            -0.00724049447539589,
+            0.014775207617625293,
+            0.0155334760323838,
+            -0.004693118176196483,
+            0.002015625406296403,
+            0.027968079002437102,
+            0.01077998866180723,
+            0.0313892866933904,
+            -0.015236169297590675,
+            0.004945828495617267,
+            -0.027215016713379062,
+            0.010214419950212732,
+            -0.01503548137805401,
+            0.008859554812646242,
+            -0.01290890682557532,
+            0.03531384210182567,
+            -0.012005613861257432,
+            -0.004250698950192301,
+            0.03253288796441852,
+            -0.029150986393345252,
+            -0.0035020051536906334,
+            -0.009255437891304041,
+            -0.004142204892160768,
+            0.013240580426083011,
+            -0.001198093874819309,
+            0.008485141356876356,
+            -0.01854370462655459,
+            9.726096864073609e-05,
+            -0.003221852304410318,
+            0.005404595442127598,
+            -0.02005458688172967,
+            -0.003314118809778734,
+            0.0073427568419969814,
+            0.004259518237708634,
+            0.002408630356035373,
+            0.016420646142725346,
+            0.006662121128241274,
+            -0.0029357701493526907,
+            0.03424224243607692,
+            0.008171895850772245,
+            0.0152521424877615,
+            -0.0018638264193202728,
+            0.0030234211261443034,
+            0.019287963516760636,
+            0.019912956661192856,
+            -0.00048443399729935116,
+            -0.01058935153883094,
+            0.00195113175754497,
+            -0.001140732137433033,
+            0.011048741602327437,
+            0.024396046054912645,
+            -0.021158025221644738,
+            -0.02419562092713724,
+            0.016377661005279716,
+            -0.015076145951740961,
+            0.01903712233378994,
+            0.010541463112889985,
+            -0.010099351302095792,
+            -0.00031558616108876247,
+            -0.042316016201379245,
+            0.010964989031040692,
+            -0.02068939391415103,
+            -0.04004120028915868,
+            -0.00244383573783919,
+            0.0009542507389088937,
+            -0.029439456243505052,
+            0.02822814855547978,
+            0.0023755032182188156,
+            -0.017280221740813364,
+            0.018223709137876153,
+            -0.005548654680726505,
+            -0.0024040301239229064,
+            0.005397711787238679,
+            -0.01092026090471121,
+            -0.010648485086769927,
+            0.008235250252660972,
+            0.010957715075632535,
+            0.005425683656535705,
+            -0.005698919020608095,
+            -0.011284911899894062,
+            0.009975882651638084,
+            0.012318901214538077,
+            -0.0019922884373083155,
+            0.010050520101815593,
+            0.0031720752428247446,
+            0.007352859750323051,
+            -0.0073483049386141005,
+            0.0046483587496626965,
+            -0.005093220128411125,
+            -0.026080057085901446,
+            0.012863580653903103,
+            0.0052434400313520774,
+            -0.011863360599637747,
+            0.0103209502958491,
+            -0.01048162756158923,
+            0.005787157766911021,
+            0.001318945460232246,
+            -0.0059333230422061395,
+            0.004821320238893602,
+            -0.0038922239931512784,
+            0.011267634571983924,
+            0.013256611037442042,
+            0.004592529900805719,
+            -0.013076954459595086,
+            -0.02177318635989856,
+            0.009001291224623467,
+            0.01959767163356408,
+            0.024161754192119852,
+            -0.030416109293488064,
+            0.003784634588325551,
+            -0.0014915849060829743,
+            -0.0020550564975080885,
+            -0.015632785947151753,
+            0.01587790981601866,
+            0.01716742056401084,
+            0.012721563202528083,
+            0.01893393006967701,
+            -0.004009617703771589,
+            0.010539100665843704,
+            -0.02080068888791658,
+            0.013642451766699067,
+            0.0008609869904066794,
+            -0.013629067231663643,
+            0.0014620655123815298,
+            0.027046304278766087,
+            -0.011056783825898616,
+            0.006206969699250696,
+            0.008097527175511791,
+            0.022111870736907183,
+            -0.012509479100313492,
+            -0.003735868296238069,
+            0.01516254968167132,
+            0.0014878674796313539,
+            0.0035887079417890352,
+            0.016533149527222273,
+            0.04729052276110659,
+            0.00102751186013391,
+            0.0016713773449738153,
+            0.0192177920520243,
+            -0.005563519063161466,
+            -0.01114821419109643,
+            -0.003549999863325982,
+            -0.021807893906469106,
+            0.010998558936481785,
+            0.00431279785115251,
+            -0.02186324212032519,
+            0.01024093366826206,
+            0.0125495473550584,
+            0.005268320135120931,
+            -0.0031679212410026603,
+            0.02574575247827441,
+            -0.019807879421996258,
+            0.014935295894965128,
+            0.03329450628384787,
+            0.00493356778223407,
+            -0.011423949333937285,
+            0.007263264249871384,
+            0.0027354459678742013,
+            -0.004494406041906684,
+            -0.00606745413660583,
+            0.010497391290734088,
+            -0.008191702034391382,
+            0.009870458191510176,
+            -0.009580130479693,
+            0.039202091799313755,
+            -0.009619018331262844,
+            -0.010782138248435458,
+            -0.004558645539551881,
+            -0.006241773108460662,
+            0.003478414182938609,
+            -0.003395165694642073,
+            0.005812225135633631,
+            -0.0069045407353789945,
+            -0.014607146220271209,
+            0.0008522982757210454,
+            0.00020175973815948303,
+            0.014001922919116725,
+            0.0011804904415274088,
+            -0.004871874532755353,
+            -0.007877217154706205,
+            0.015243277056007988,
+            -0.006047906208304912,
+            0.0006531937033317192,
+            -0.0011653523616313422,
+            -0.0013662556399359335,
+            -0.007401050012179761,
+            -0.015166105795085528,
+            0.00974609244984173,
+            0.016253733908242134,
+            -0.013857834499211056,
+            -0.005142936208363415,
+            0.012274716212832219,
+            0.008540229125193836,
+            0.011742606168187876,
+            0.0007451338650642946,
+            -0.009255813682572937,
+            0.008292531177374192,
+            -0.015020523035432158,
+            -0.008781317965985481,
+            0.0026250469418136603,
+            0.002381147382977664,
+            0.005839723981046825,
+            0.0006240641828668715,
+            0.004502463508605272,
+            0.005798912705218846,
+            -0.00497936644919367,
+            0.01112666823857836,
+            0.008198424171006548,
+            -0.002452143482105151,
+            -0.01017569410837814,
+            -0.001127589984272638,
+            -0.015733298167661346,
+            -0.016758315411028353,
+            -0.0009286266498936771,
+            0.015175988113750882,
+            -0.02352862898718829,
+            0.018996352975391644,
+            -0.0014524363611708201,
+            -0.0033802546171857704,
+            -0.021392428456030312,
+            -0.0025923983911570183,
+            0.0034214188668518796,
+            0.01255768230394476,
+            0.01858853630745928,
+            -0.02729543519465384,
+            0.010575894388563116,
+            -0.007204725245442141,
+            0.030394929424014717,
+            -0.018328782140005044,
+            -0.004022941273610227,
+            0.026915618079621992,
+            -0.02200018306828758,
+            0.0008791185377513978,
+            -0.011454606955158956,
+            -0.008103032693853504,
+            0.019287674837688177,
+            0.023747720480736777,
+            0.02038287242167783,
+            0.01364489294869211,
+            0.0007758044919789085,
+            0.004768030222445397,
+            0.03572705648994403,
+            0.010568241270862248,
+            0.014828087092859878,
+            -0.007542489523110974,
+            9.457808640264811e-05,
+            -0.02554033312739386,
+            0.01635899434369978,
+            2.587383534566272e-05,
+            -0.012370727506722627,
+            0.007162422186621379,
+            -0.013370882403931942,
+            -0.011066483600541058,
+            -0.007151077450730025,
+            -0.01721910018182534,
+            0.02152239849400241,
+            -0.012322440280680164,
+            0.004587729883596212,
+            -0.009079747696669277,
+            -0.013272707286450417,
+            -0.012953682633400515,
+            -0.014202214807187545,
+            -0.01313141988034687,
+            -0.026041403779538307,
+            -0.027109731260915763,
+            -0.006573285295895077,
+            -0.023780366346334524,
+            -0.013799072648505526,
+            0.005588622594136253,
+            -0.022617902666316757,
+            0.012262325666428604,
+            -0.012689827812482627,
+            -0.01226264872059864,
+            0.012805734807835008,
+            0.020274844315616616,
+            -0.011765231803328348,
+            -0.028494489703400308,
+            -0.018583843821043854,
+            -0.019970998470704358,
+            0.012576201137505969,
+            0.002266288538944273,
+            0.013165889075444961,
+            -0.003018437854660793,
+            0.003227412667880239,
+            -0.014895831590298957,
+            0.018570624675927183,
+            -0.03442390703780963,
+            -0.03108003806963536,
+            -0.013611306765348361,
+            0.005182777066476028,
+            -0.0015782033203578553,
+            -0.011429046504256945,
+            -0.0018606648572920801,
+            -0.00433805271182878,
+            -0.025109572024446088,
+            -0.010134856632190067,
+            0.011010662054484161,
+            -0.015205629927469414,
+            0.0072933565255675036,
+            0.0009018832564038863,
+            0.013870902834260887,
+            0.009587432944616442,
+            0.013393301893492882,
+            -0.006095063698384775,
+            0.004356485794113208,
+            -0.0026476612747324386,
+            0.0004894024149759108,
+            0.009336366919474478,
+            0.00839591561027103,
+            0.010551161369079399,
+            0.002260707777260338,
+            0.007830059223001696,
+            -0.0060927446216633265,
+            0.018065002687169148,
+            0.015436944875538481,
+            -0.0002520689442496089,
+            0.013323098485251487,
+            0.0003769862591044255,
+            0.00022293304921127018,
+            -0.0029970303106340466,
+            0.007888437018607987,
+            -0.011785957933642163,
+            -0.001311704406355858,
+            0.015138190893483804,
+            -0.01707247730569978,
+            -0.013045187198628258,
+            0.03107034781283638,
+            0.0018583638488268697,
+            -0.02495564203304796,
+            0.031146554085787762,
+            0.004220047610942943,
+            -0.005835199594223787,
+            -0.013205086624848782,
+            0.0014059047822222418,
+            -0.011570624844100298,
+            0.003584232809362274,
+            0.014295032061446826,
+            -0.02231984223610487,
+            0.010521094801839404,
+            0.008446892441796151,
+            -0.009681535919980559,
+            -0.012337659795586223,
+            0.009847073709209602,
+            0.016809328445200358,
+            0.004472485504034416,
+            -0.03539093810210523,
+            -0.011957694102753411,
+            0.02145470732008716,
+            -0.029237428225009913,
+            -0.00810041956259673,
+            0.007452962400105787,
+            0.0065154832040426065,
+            -0.019353109017609053,
+            -0.008580955020412362,
+            0.005725794605897509,
+            -0.004940390374637084,
+            -0.003892210172017758,
+            0.017541623128534682,
+            0.00485047043626186,
+            0.013698341792101025,
+            -0.01689738578307474,
+            0.009771458057400557,
+            -0.022222817828602557,
+            0.004021882074676753,
+            -0.023373700500461992,
+            -0.009873403167702603,
+            0.002364143609079849,
+            0.012454271200139762,
+            0.014328556467632212,
+            0.027839363143659326,
+            0.030056404238348527,
+            0.013995696971422429,
+            0.007833677585250953,
+            -0.016265199386953717,
+            -0.018134327237616175,
+            -0.006409513840075933,
+            -0.004033173350204784,
+            -0.008714798059218596,
+            0.0066373547199279235,
+            0.01858354212009877,
+            0.004499712802472377,
+            0.02609078206670711,
+            -0.006227189596272876,
+            0.009039744622014585,
+            -3.127526911951914e-05,
+            0.0008833930131827993,
+            -0.004226083827636182,
+            -0.01971685546584527,
+            0.00481003211084742,
+            -0.0013721482162825508,
+            0.025874758880591167,
+            0.00940496860916206,
+            0.0024915013399541847,
+            0.01615978314811047,
+            -0.006378952757397484,
+            -0.01174653607178726,
+            -0.010540107733238588,
+            -0.0061926268398085176,
+            -0.021752582127981104,
+            0.029059100503124387,
+            0.021195595568253612,
+            -0.010800765264123782,
+            -0.013693582166598082,
+            -0.00903369526059487,
+            0.02795767990556847,
+            0.004080965688836953,
+            -0.0013050212744006933,
+            0.016313311850308505,
+            0.012084555958999215,
+            0.012329325712491252,
+            0.009794698207689061,
+            -0.00036157148861933123,
+            0.0015364709122308352,
+            0.03722522141440073,
+            -0.007427028510548186,
+            0.0026463292682538495,
+            -0.005239324411651515,
+            0.0025987932868461313,
+            -0.009705817520422487,
+            0.007242122655628283,
+            -0.004012676385296197,
+            -0.0158760051470016,
+            0.015003444174308607,
+            -0.022443331305169994,
+            -0.0016749787204471238,
+            -0.008150635190952799,
+            0.0016696640640947303,
+            -0.02781023337840043,
+            -0.0025731206607371845,
+            0.006695065701630532,
+            -0.015547340828489069,
+            0.011816010501093885,
+            -0.01740673229051557,
+            -0.011553604323149105,
+            0.007914961618894248,
+            -0.012034157288065447,
+            0.0016684130271324446,
+            -0.0008268957621923278,
+            0.004460864332388434,
+            0.013426055079012884,
+            -0.011008808570221245,
+            0.012166046631826273,
+            -0.005816284310138776,
+            -0.007872091214834688,
+            -0.0016055129487028885,
+            -0.0030049472973942043,
+            -0.0025600746761171543,
+            -0.0009966231170649917,
+            0.034355910234323155,
+            -0.004446570896048543,
+            0.003761242147499512,
+            -0.01362639743248326,
+            0.0019833052855181806,
+            -0.036176376394999044,
+            0.028344464817309477,
+            -0.008642835889481802,
+            0.011013502221008498,
+            -0.0017508365480865596,
+            -0.013243706268954915,
+            -0.01456517309747086,
+            0.015247299501245178,
+            0.007602596572394442,
+            -0.005854531057349665,
+            0.013558248087347848,
+            -0.019724373614448298,
+            0.008530500822910953,
+            0.004463694543554244,
+            -0.015561397976987216,
+            0.004668780190328835,
+            -0.007479912658259757,
+            0.012657201279639194,
+            -0.01774730033362932,
+            0.004314594593150765,
+            0.00958571617187387,
+            -0.01892876084898881,
+            0.009901041800918215,
+            -0.028675879564512306,
+            -0.004631920242208613,
+            -0.012323455524366223,
+            0.0053804608827285865,
+            -0.000660864620708182,
+            -0.02389088252276507,
+            -0.028944280660230762,
+            -0.0002099667627277945,
+            0.00458533068247518,
+            0.0018878351997997833,
+            0.011621407953503755,
+            0.005970807015151215,
+            0.008542958629199003,
+            0.01023896054756221,
+            -0.00205207247588841,
+            0.009402752124876478,
+            0.04169706217414899,
+            -0.022899912321317197,
+            -0.011574970753549647,
+            0.005321872977369877,
+            0.011499271505460519,
+            -0.02647653483798284,
+            -0.011039791758347888,
+            0.0033424345263339144,
+            -0.005219877979550397,
+            0.03609469691834737,
+            -0.02448746405881972,
+            0.019020618652020673,
+            -0.011392297116547224,
+            0.007625475937968714,
+            -0.021465240076078376,
+            -0.003921088264697311,
+            -0.014265745991522477,
+            -0.00011221615798127438,
+            -0.0031478236827989703,
+            -0.0076053733863387755,
+            0.007485039021296698,
+            -0.001907678661303243,
+            0.0020348246022161925,
+            0.006291436821831794,
+            -0.02173305455485444,
+            -0.0019125499969493854,
+            -0.016432460924913166,
+            0.030367263349759286,
+            -0.016242095671439496,
+            0.005072805486919414,
+            0.0053258139728101506,
+            -0.002810639419737258,
+            0.012368771814660942,
+            0.004165967191536186,
+            0.00513754440091394,
+            -0.004199767241827414,
+            -0.005142642360546972,
+            -0.0053886448467479215,
+            0.017868874549760376,
+            -0.013280048598849825,
+            0.035024701466710065,
+            -0.0100959186045929,
+            -0.007094001877261568,
+            0.0019733042325232155,
+            0.010380430710282744,
+            4.2084202709296443e-05,
+            0.01053565670076933,
+            0.0034270489770133685,
+            -0.0006311078385359693,
+            0.0019774779328212756,
+            -0.017162448473652843,
+            0.003089758682621882,
+            0.01682656561693936,
+            0.005463677441664502,
+            0.004469261499451007,
+            -0.013332130891003062,
+            -0.012050233723282738,
+            0.004538268704442613,
+            -0.007533940408255447,
+            0.0003675737140813486,
+            -0.018816953644773076,
+            0.013972696748575243,
+            -0.0011096683686631858,
+            0.0002715683052236706,
+            0.00524678369365015,
+            0.023056514997908218,
+            0.01813167860994747,
+            -0.002553352602551003,
+            0.03155164978556312,
+            0.0174738879054593,
+            0.0029268039949260673,
+            0.010637295711777793,
+            -0.033222781195984666,
+            -0.013301873512972214,
+            -0.0013721703891642354,
+            0.02238421933149517,
+            -0.01530189703335162,
+            -0.014059029663525449,
+            -0.0037393534403749947,
+            0.0010394601933309767,
+            -0.005177359933828505,
+            0.0042489306011644495,
+            0.011039889511148341,
+            -0.014697595240068863,
+            0.007597149702441915,
+            0.0026780496822907634,
+            0.02092734730183164,
+            0.0004688144432243786,
+            0.017181410417231698,
+            -0.0005083970646221274,
+            0.00749639055342427,
+            0.007319420826337891,
+            0.017541463125355265,
+            -0.01002811389296805,
+            0.023582113845081534,
+            -0.008823933397729605,
+            0.0019889258902730535,
+            -0.01257106158422629,
+            -0.028159594067544737,
+            -0.012624750220149823,
+            0.015247564545279736,
+            0.005765911899842925,
+            0.006478299226263732,
+            -0.016537090297144265,
+            0.006276997440048981,
+            0.015647122171767974,
+            -0.002251683151059454,
+            0.010295358869874306,
+            0.009730378299068289,
+            0.00012816405890877098,
+            0.010181120688239766,
+            -0.00801256937783534,
+            -0.00734257767723293,
+            -0.004497520124843868,
+            0.023982754195729983,
+            -0.005443550092842393,
+            0.022434177122972028,
+            0.012647958478747348,
+            0.00616791857573897,
+            -0.010853355357954727,
+            -0.003340111254357573,
+            0.023389634440057473,
+            0.011863246790550376,
+            0.00029642699874398923,
+            -0.009846034060495678,
+            0.0019458876579524617,
+            0.0054010373765486005,
+            0.012983913038716568,
+            -0.02049427270240938,
+            -0.004344571352918878,
+            0.008461947538684664,
+            -0.008472919299762712,
+            0.004318309005722874,
+            -0.030156444522730416,
+            -0.005812303237222271,
+            0.021354352236942434,
+            0.008392281015122372,
+            -0.010232039722494646,
+            -0.004395682831498155,
+            -0.0006527587802367882,
+            0.0040601850269714945,
+            -0.0024323167053212176,
+            -0.0023729138545912176,
+            -0.008541233856441836,
+            -0.006470933419694742,
+            0.014035696617797647,
+            -0.0052691760528871924,
+            0.008597459268477037,
+            0.00012084859681905111,
+            -0.0017255958688825231,
+            -0.007323529232751411,
+            0.03382792341957058,
+            0.008087732655845718,
+            0.005980666974986827,
+            0.0013613625176327891,
+            0.0042244330945601215,
+            0.000616791811672998,
+            -0.0028374856646453707,
+            0.018075360591135542,
+            0.0156738939725117,
+            0.005054696911814988,
+            0.009038525980341447,
+            0.014672270369917403,
+            -0.019700405219230957,
+            0.012625337070556061,
+            0.012184185186281482,
+            0.019382321278525114,
+            0.0007219498737851579,
+            0.002564970959687783,
+            -0.010601873823737926,
+            -0.0029963537053764736,
+            -0.002070567089566093,
+            0.029563028293936092,
+            0.0005903976560069006,
+            -0.010509758568838935,
+            0.010992421945232951,
+            0.020603868018231348,
+            -0.02101157589451638,
+            -0.001971974816402967,
+            0.005963911670920596,
+            -0.010704207628592966,
+            -0.014466113540150018,
+            -0.021269176952661614,
+            -0.013331247081394942,
+            0.0071597345125287174,
+            -0.008162102923896894,
+            0.02367549407145414,
+            -0.014256771777502395,
+            -0.03960796112042958,
+            -0.014803626428839152,
+            -0.00864320347698291,
+            -0.027531337207277483,
+            -0.007305441146142331,
+            0.00965056093889974,
+            0.016698589600806273,
+            -0.003330867073501157,
+            -0.025198703196464594,
+            0.030483790406960656,
+            0.008872821868559751,
+            0.023708422767352917,
+            -0.021464374508407488,
+            -0.0018617153194555344,
+            -0.00886376784514331,
+            0.014505918224188001,
+            -0.017588761818663975,
+            -0.011756112651591795,
+            0.020014632992762026,
+            0.014356419813769714,
+            0.007246557500044434,
+            -0.004355132555323843,
+            0.0067299398462013936,
+            0.007649499235136303,
+            -0.012515181210363267,
+            0.012239792251154642,
+            -0.013499277313916029,
+            -0.024858295126228433,
+            0.014430561986516884,
+            0.002229494935179828,
+            0.009563004428064055,
+            -0.0015984171085659323,
+            -0.016197659608768088,
+            0.010078159490026602,
+            0.00808729175515505,
+            -0.017508840440512225,
+            -0.004793934415922065,
+            0.007133527615305415,
+            -0.020818869260580315,
+            0.031234686259160864,
+            0.011872765152279094,
+            -0.010692949770503467,
+            -0.001989039629661815,
+            0.004431079028880923,
+            0.016511544608975148,
+            0.007893700385963683,
+            -0.01997314388229276,
+            -0.003708006454566117,
+            0.01530128306203401,
+            0.00822588305477249,
+            0.004861703074728471,
+            -0.010836328783594584,
+            0.03434874763691911,
+            0.006082873924472328,
+            4.192481378506604e-05,
+            -0.0052997095110967115,
+            -0.0076108924981838355,
+            0.004196112457289409,
+            0.018905391806961686,
+            0.013330078288599763,
+            -0.004200590783114919,
+            -0.006630107527854746,
+            0.007778448128358027,
+            -0.01706991160178345,
+            0.007329056462125937,
+            0.009390327499494826,
+            -0.012870674959022891,
+            -0.0017571189976160295,
+            0.016462608870101713,
+            0.00209846661139665,
+            0.004942714850185376,
+            -0.011305050739130339,
+            0.004252776146827917,
+            0.015841821944109317,
+            0.018727567181082646,
+            -0.008399642221255733,
+            0.021448055799259867,
+            -0.020972459569228434,
+            0.027243639465535856,
+            0.028499400850980086,
+            0.016573144403275928,
+            -0.01847096443459798,
+            -0.0022409373140116146,
+            -0.0041737366139895414,
+            -0.014761716560267328,
+            0.0034546020161337894,
+            0.005125870074458691,
+            -0.01174186608185822,
+            -0.009577921232380895,
+            -0.024331993794416026,
+            0.011075959943215117,
+            0.007918147902741738,
+            -0.0015045185909419564,
+            -0.011791155919533844,
+            0.016019077577552835,
+            0.010141868950981925,
+            -0.010938402325619172,
+            -0.005635351352447256,
+            -0.019967361901734836,
+            0.009750427567651833,
+            0.009128294017175602,
+            -0.00300556760891084,
+            -0.011496103417643132,
+            0.012977890524465716,
+            0.0014080059843484115,
+            -0.01292380658817521,
+            0.010841954636959441,
+            -0.03142011651894961,
+            -0.005510537354374977,
+            -0.011027026734432255,
+            -0.003620482919145167,
+            -0.014415606143566519,
+            0.003608572766363589,
+            0.0005034611321129689,
+            -0.011470488351158329,
+            -0.0035005774696278,
+            -0.016207453351921627,
+            0.004676327282262429,
+            0.015021513761697039,
+            0.001679614645967629,
+            -0.006917375094634678,
+            0.021994274901326064,
+            0.018518524483306403,
+            -0.021903538417919984,
+            -0.020116013379669034,
+            0.004708084983787958,
+            -0.03261690161800251,
+            0.015019348187107588,
+            0.00966124471577622,
+            0.011676937880751252,
+            -0.028768229226408037,
+            0.007114534979415504,
+            0.00900420165933673,
+            0.016398927481511824,
+            0.0050879301160439245,
+            0.009453255486450082,
+            0.01222758288108498,
+            -0.0104657614414275,
+            -0.003164502778525433,
+            -5.410382753651127e-06,
+            -0.005618225986727351,
+            0.015607620507415484,
+            0.0008527697935056332,
+            0.03603666488542239,
+            -0.005277207843139309,
+            -0.0009799879252482288,
+            -0.011735731235560851,
+            0.00812046801749312,
+            -0.011713349784345638,
+            0.006334172347602515,
+            -0.0024643499727789167,
+            0.0001240009282616789,
+            0.002965067818077541,
+            0.009442572075213767,
+            -0.01086095783353607,
+            -0.007595379345224076,
+            0.007444897256325813,
+            -0.01482814340478338,
+            -0.00019984717819268013,
+            -0.009339343730980451,
+            0.0019680804106275156,
+            -0.02241031895876346,
+            0.006629838013824249,
+            0.040935644507527846,
+            0.02140216465404127,
+            -0.005481040712269146,
+            -0.012431659209559418,
+            -0.004680230194268222,
+            -0.007229300034633229,
+            -0.006482401275991039,
+            -0.004794698160419071,
+            -0.022762405726589375,
+            -0.006928553505112732,
+            0.004320321244630314,
+            0.013300138724212552,
+            0.009741018117644266,
+            -0.008842135469729511,
+            0.004531760574809748,
+            -0.02662650543210291,
+            -0.016578081411548973,
+            -0.0060021425889846495,
+            -0.015840130031005147,
+            0.006125301421471048,
+            0.027521841635711787,
+            -0.0063432924325983525,
+            0.01789356321527744,
+            0.005142029893259461,
+            0.013034833065163448,
+            -0.005338869351330258,
+            0.00035250243782976347,
+            -0.006010938630154189,
+            0.0032958656042048292,
+            0.003074861351445954,
+            -0.0015947217852680714,
+            0.004151590621941346,
+            -0.007139285822575166,
+            -0.0026091937868967284,
+            0.0011629380311555433,
+            0.0117841884590861,
+            0.006877419488576261,
+            -0.00016252653621991807,
+            -0.01761285866765538,
+            0.022221104524808997,
+            -0.018933508559881868,
+            -0.008840161102111973,
+            0.011092684029552064,
+            -0.009660487856185668,
+            0.00702597023411946,
+            0.020387386977378873,
+            -0.006854213562767843,
+            -0.009583981919303215,
+            0.013028181087649146,
+            -0.0036507672754997286,
+            0.004464368543493108,
+            0.009867428978560914,
+            0.006036303973414791,
+            0.002482944923100156,
+            0.013244650094892797,
+            0.004530294330688855,
+            -0.0032625482538717137,
+            -0.020238533752799943,
+            -0.005186498720507101,
+            0.019015752262933585,
+            -0.02252458872825473,
+            -0.0027641215988942563,
+            0.0044506144202026456,
+            -0.0030317840657635457,
+            -0.01733962766050954,
+            0.004205861489436576,
+            -0.018770275720867473,
+            -0.0033883519277721433,
+            -0.008244668812709337,
+            0.009948090119054079,
+            -0.023700732627507225,
+            0.009109315916173436,
+            -0.011739192106106307,
+            0.02356509175177699,
+            -0.003925851077921686,
+            0.024734433836185024,
+            0.0015142645677087268,
+            -0.013482933030628261,
+            0.0055323884713736554,
+            -0.0008930661843169134,
+            -0.004477731568313757,
+            0.029338490404300612,
+            -0.005418258701133154,
+            0.002249996261871466,
+            0.013491124734151694,
+            -0.021332541202417246,
+            0.021773687217263173,
+            0.0046662696076723836,
+            0.01628697326539432,
+            -0.00986238951830334,
+            -0.013799209313714441,
+            0.025178874553620743,
+            0.03324555035648896,
+            0.008398796308070151,
+            0.011261509865513953,
+            -0.01226890254177634,
+            0.018966175575083295,
+            -0.0038328997984309597,
+            0.004344837325292706,
+            0.00878668880458693,
+            -0.007200842513742962,
+            -0.011500562922303248,
+            -0.004533014670236755,
+            0.027639894892383943,
+            -0.021838848944317424,
+            -0.008920629621611768,
+            -0.005739181836252042,
+            -0.01593272996094732,
+            -0.038167534267554606,
+            -0.004371701247332816,
+            0.019604405563120094,
+            -0.005974358141737469,
+            -0.0018341151594086288,
+            0.014334266690716505,
+            0.00402620072221496,
+            -0.002852569009185304,
+            -0.008195218707838913,
+            -0.0027944153624309856,
+            -0.0059390492900516225,
+            0.016651206007022145,
+            -0.004961864528562493,
+            -0.007929830585906699,
+            0.01145431100211762,
+            -0.005731957927601062,
+            0.012130694936714224,
+            0.0004848418056781251,
+            8.392048424584131e-05,
+            -0.0015174120840537315,
+            0.00822932723013239,
+            0.018620869354789082,
+            -0.0055803454666417465,
+            0.025682778991091174,
+            0.006956936033581638,
+            -0.003495709501259274,
+            -0.005489431577052735,
+            -0.0044154293119170445,
+            -0.024561016942073285,
+            -0.0020864636079347357,
+            0.006593060072668424,
+            0.024333813203596122,
+            -0.02515989187381735,
+            -0.004953266000463339,
+            0.020957806164031402,
+            -0.0012192724005613,
+            -0.011978722479596745,
+            0.014690559986694138,
+            0.0041346025614047875,
+            0.009881584250499358,
+            0.012176393203318002,
+            0.015764530703333238,
+            -0.004923548157246672,
+            -0.0015192775000207833,
+            0.012766482928380375,
+            0.0005745973040874939,
+            0.00997809510696241,
+            0.016495444272181515,
+            0.010467514633877887,
+            -0.021108281220544853,
+            0.0015078843789125156,
+            0.00522593579521476,
+            -0.017481430635971396,
+            0.0034362821987740755,
+            0.005397819166037461,
+            0.011626527721895694,
+            0.004048632382026344,
+            0.013207335578216437,
+            -0.015768003434275563,
+            -0.005916168437120099,
+            0.008153798523776654,
+            0.004319451296835019,
+            -0.02599474886669325,
+            -0.01485009085106089,
+            -0.0006781964116146299,
+            -0.004444144921332929,
+            0.013556265879295017,
+            0.01285768162554067,
+            0.02033626001285354,
+            0.024637349990579514,
+            -0.013833060730799217,
+            -0.00842819657792762,
+            0.009436072455719896,
+            0.015560357465711765,
+            0.005798181411384387,
+            0.007874051721372273,
+            0.009297440071118126,
+            0.0077320823375042,
+            -0.004073209576661068,
+            -0.002973928843567532,
+            0.020205003097225747,
+            -0.019725382393637726,
+            0.0006746659838642147,
+            -0.018417443693472958,
+            0.007530796896254817,
+            0.01140696723506382,
+            0.01129515042745398,
+            0.002816883643872417,
+            -0.005274757308627299,
+            0.005716917047812153,
+            -0.06254973656832136,
+            -0.009191453047502128,
+            0.003384751551032943,
+            0.027256466708746898,
+            -0.010885336140129142,
+            -0.003415707910562931,
+            0.007309577090125959,
+            0.005511970708769456,
+            0.0005572239548625122,
+            0.003186427278034852,
+            0.019698716902282903,
+            -0.02052581021967601,
+            -0.010793968505805883,
+            0.004219269922138548,
+            0.0033111837923746746,
+            -0.007269521534951044,
+            0.004805737092302134,
+            0.014143978208501908,
+            0.01986656063456352,
+            -0.0024048854610842748,
+            0.01629335842915328,
+            -0.0015498603704589015,
+            -0.014496994444493684,
+            -0.002671064325909516,
+            0.0037875833047555925,
+            -0.022466433279066646,
+            -0.005980897597195858,
+            0.0043262697442044435,
+            -0.011226121648007667,
+            0.006978495397818936,
+            0.008452177557316863,
+            -0.003018545465557218,
+            -0.0008303307644324117,
+            0.017796789945838845,
+            0.006591364879905198,
+            -0.013464486184025026,
+            0.0028668426952326957,
+            -0.023377605222213174,
+            0.004325185794796361,
+            0.001416245546006914,
+            0.007809449561523174,
+            -0.008431335139468173,
+            -0.014774499666285603,
+            -0.005933810712528226,
+            0.017746847020610673,
+            -0.01965188730040385,
+            -0.002406875192894708,
+            0.01683009662984443,
+            -0.0002518220110313279,
+            0.005456812494894182,
+            -0.02011241622618092,
+            -0.0009782974741561381,
+            -0.02933296756078644,
+            -0.01171823509606186,
+            0.001162345317872531,
+            0.00892221099991475,
+            -0.006551308022163148,
+            -0.03934356275100932,
+            0.00427623835776202,
+            -0.013111664387776261,
+            -0.008752857644809231,
+            -0.010214546496134218,
+            -0.00518087722852466,
+            0.015240297918667867,
+            -0.012498100033235827,
+            0.01086490299272505,
+            -0.00917777212602538,
+            0.005895055740265625,
+            0.0090554651297836,
+            0.014212841798768113,
+            -0.005544047915369086,
+            -0.0033589999440643385,
+            -0.007924830667307956,
+            0.0022291797354234005,
+            -0.0032976226056924502,
+            0.009615674261248298,
+            -0.0051932155872883,
+            -0.0065636389655770455,
+            0.02426106498596339,
+            0.011445591067574932,
+            0.013312596099532533,
+            0.0055118624272110965,
+            -0.007220620072349783,
+            -0.01916551187613823,
+            -0.003977616240384473,
+            -0.013204506708246246,
+            -0.006315966975649096,
+            0.01434099854765068,
+            0.012588249778961223,
+            0.006357972880218866,
+            0.0007747002406996095,
+            -0.004794439624135275,
+            0.006975593520375694,
+            0.0262335496080837,
+            -0.016481909971520166,
+            -0.0013084662210086592,
+            -0.01369313900978409,
+            0.01921748474212319,
+            0.03333761672336437,
+            -0.0008918419419301244,
+            -0.007699525186523558,
+            0.013757216065348164,
+            0.00819045303047094,
+            -0.007411380297164911,
+            0.02453611080891859,
+            -0.025045462880463647,
+            -0.015989319900023054,
+            -0.011650886878708265,
+            -0.011407253985176972,
+            -0.008386367959227559,
+            0.016331961716602287,
+            -0.00141225438918211,
+            -0.0018505754298268654,
+            0.009557291617446645,
+            0.03357924917013024,
+            0.006916860901290756,
+            0.007030308953738567,
+            -0.002739345982346728,
+            -0.009376522438071562,
+            0.011482637968411055,
+            -0.007259054391879614,
+            -0.04075661435928125,
+            0.004250195164583401,
+            -0.005160505473755424,
+            0.02405623169060588,
+            -0.02197586561375289,
+            -0.00619175761282745,
+            -0.006736129049831973,
+            -0.028200015908139763,
+            -0.005867722104196303,
+            0.02805071895018145,
+            -0.015402198567775478,
+            0.014773294766214671,
+            -0.009502185346558072,
+            -0.0013419858580438616,
+            0.004443749010136947,
+            0.013008783570355656,
+            -0.02289662952366337,
+            0.0013834888740540108,
+            -0.00048549197205067054,
+            0.027969842651517314,
+            0.010437729090975158,
+            -0.0047705656249623744,
+            0.014202218020412936,
+            0.008240861154198411,
+            -0.018637315941614406,
+            0.007314019317343154,
+            -0.005747966765101792,
+            0.014833896739762147,
+            -0.01314847940891154,
+            0.007762100494499689,
+            0.00705026393499352,
+            -0.01278023669088385,
+            0.03016342404000438,
+            -0.015204129985039077,
+            0.007776946006562982,
+            0.009278778571285689,
+            0.009656723862179986,
+            0.01257686993982689,
+            0.004362439504889069,
+            -0.02898418193036562,
+            0.0002144431781454467,
+            -0.03828100004144633,
+            -0.00833748467467171,
+            -0.002611937107022567,
+            -0.002145329666219461,
+            0.007050630951707937,
+            -0.009383645162261629,
+            -0.033976442057557676,
+            0.0006429978367640709,
+            0.01603027598131823,
+            0.017501067413108286,
+            0.0002196997337451293,
+            0.013075379412332973,
+            -0.006197537539554383,
+            -0.021148120787551232,
+            -0.016986448660412497,
+            -0.010207705210051413,
+            -0.02122194744719993,
+            -0.020550470201191853,
+            -0.00908564461857315,
+            0.012090079127547193,
+            -0.009927390005879807,
+            0.013888458967497627,
+            0.029513250508042935,
+            -0.024529175134769883,
+            0.0011144447737101862,
+            0.021977677894839922,
+            -0.00198308986575084,
+            0.014108189311291326,
+            0.001696775006906642,
+            0.014122064251288995,
+            -0.0004166140484018266,
+            -0.010556016685519523,
+            -0.009729362492661684,
+            0.017487929296614076,
+            -0.0026043257634797124,
+            0.016715798948897194,
+            0.012183487338360088,
+            0.003771721868999568,
+            -0.0030887189945795796,
+            0.010667776697461199,
+            -0.026471157050402344,
+            0.014249996582064706,
+            -0.006690961709923224,
+            -0.0181382691440242,
+            -0.008729104191096461,
+            0.003882731567105777,
+            -0.005609384377642956,
+            -0.007794812949125115,
+            0.012967089730551055,
+            -0.00944374111323783,
+            0.02088525981049842,
+            0.0023400121055514958,
+            -0.012196925240774426,
+            0.003648688547842322,
+            -0.03322041209304968,
+            0.0015191765655963997,
+            -0.018402382776013956,
+            -0.024659631472836927,
+            0.03840173429666435,
+            0.011991452465253198,
+            0.006639702644606365,
+            -0.01675838115087676,
+            0.007953102302963526,
+            0.0004257479565099406,
+            0.017484338819046517,
+            0.011068864013925452,
+            0.006466915839538296,
+            0.008364454017976605,
+            -0.023316137291380285,
+            -0.010898421318668592,
+            0.016204569093730564,
+            0.012099473359791148,
+            -0.01017229369980751,
+            0.019431669546223253,
+            0.017955119269367787,
+            0.007300372071724306,
+            -0.006898619924677706,
+            0.01647927363640854,
+            0.021012103522831126,
+            0.010291016618557653,
+            -0.00047955552103474145,
+            0.0008684297441778696,
+            -0.010534247928997246,
+            0.005037930235889528,
+            0.035417626740307205,
+            0.014712595692093886,
+            0.010650306363963653,
+            -0.03626584475133921,
+            0.021764456453493054,
+            -0.011961521673661574,
+            0.009092981192874177,
+            0.006089104887183293,
+            0.017982801101846038,
+            0.006916969397519662,
+            0.014196254958707752,
+            0.011745069030657836,
+            -0.006689305450954901,
+            0.025205280281804708,
+            -0.001601414806100461,
+            -0.011992525142592405,
+            0.003386930324498371,
+            -0.01859193251438437,
+            0.005303261319773131,
+            -0.0041710536965100615,
+            -0.004459393120257307,
+            0.0032461828144552565,
+            -0.0018650956576455287,
+            0.015678225251799887,
+            0.003894142628084471,
+            -0.02355212701026675,
+            -0.0014873755115206368,
+            -0.024284295126348117,
+            0.0027260592623277375,
+            -0.022185571998195344,
+            0.00022883605560765203,
+            -0.005557568353382136,
+            -0.009749382813201403,
+            -0.025385857431333838,
+            -0.009853073898348951,
+            -0.0035843255959309307,
+            0.007661137888845632,
+            -0.010521037433807653,
+            -0.004246698084422667,
+            -0.0019920499321964815,
+            -0.019465113363548715,
+            0.009014255846983305,
+            0.027122533756959888,
+            0.016848221859975445,
+            0.027170883580103264,
+            -0.00019554112591107156,
+            0.013614675849485926,
+            -0.002066822813103284,
+            -0.01603063005944364,
+            0.008824663429777616,
+            0.0075653522381218495,
+            -0.00531780740880296,
+            0.014643348459785112,
+            0.0072634484937344704,
+            -0.012800149015319017,
+            0.009859503660384916,
+            -0.007161712176469609,
+            -0.006437857349742842,
+            -0.009764856421740622,
+            -0.02058868349990357,
+            -0.014866137971603676,
+            0.0012450207572602557,
+            -0.002123265406747006,
+            0.006000473862248991,
+            -0.004054016790850226,
+            0.009584879012937315,
+            -0.00608816807208866,
+            -0.00925801057439376,
+            -0.013962683072792106,
+            -0.008924356118158748,
+            -0.004210064145949253,
+            -0.004381709838058668,
+            -0.011732699976353612,
+            -0.005479523097469261,
+            0.0010680258077037758,
+            -0.0019401640391035755,
+            -0.01397652201595301,
+            0.011650839243603377,
+            0.0058731838291845645,
+            -0.002462743134993977,
+            0.02219008688925881,
+            0.01097316136828334,
+            0.0018083001685932406,
+            0.01366095798812196,
+            0.0003942601242216818,
+            -0.0064063532958756175,
+            -0.03343312148366622,
+            -0.0005083355836126291,
+            -0.02815586367527057,
+            0.025770078162165123,
+            -0.0069114949102666145,
+            -0.010322087105114614,
+            -0.002519047562374146,
+            -0.006063669644021698,
+            0.005873609611264377,
+            -0.002722940652283699,
+            0.0021229525800901114,
+            -0.023673778230768945,
+            0.0028629715379660113,
+            -0.003953653437464966,
+            -0.0036366957271968823,
+            0.007469664864329332,
+            0.0346520690389124,
+            -0.004054282838167587,
+            -0.0034765253405897676,
+            0.005159466707114532,
+            0.011642913651743502,
+            0.027845571220425726,
+            -0.004445196354520585,
+            -0.0032038508043339413,
+            -0.02233838584152591,
+            0.0054112576531374175,
+            -0.00019544838850471866,
+            -0.004733192299951088,
+            0.016664217348411003,
+            -0.016719175418727594,
+            -0.01631658832047864,
+            -0.00545969677717507,
+            -0.014570848053102338,
+            -0.00022620258237138677,
+            0.014454101124666536,
+            0.011056312437637014,
+            -0.004035526363810078,
+            -0.012459378363360557,
+            -0.0022595493558020706,
+            -0.030370952502579162,
+            0.012766010284884204,
+            0.1141689619126009,
+            -0.00766197047210299,
+            -0.00896655658602649,
+            -0.009481112393590025,
+            0.001435269335646347,
+            0.001338755472877713,
+            -0.011739143028767118,
+            0.07166096084053437,
+            -0.0077855767070652395,
+            0.0002139580430342728,
+            -0.011312198831181402,
+            0.001844005266872301,
+            -0.001825412693163496,
+            0.01371932421024605,
+            -0.005025814503687022,
+            -0.018213360991353643,
+            -0.0033612481616769416,
+            -0.02941403830641238,
+            -0.0041137865872933515,
+            -0.005683630256743215,
+            -0.004288198570156578,
+            -0.014982999363445122,
+            0.0067149665791261335,
+            -0.019661973497372313,
+            -0.012688556129040908,
+            0.001124391650175233,
+            -0.0033623453063430534,
+            0.00996944383866679,
+            -0.020496336814603385,
+            -0.0044354321069609536,
+            0.0019875840782527733,
+            0.009706675240563883,
+            0.0011600364428219214,
+            -0.012755178230821602,
+            -0.0026514442584739412,
+            0.004056350166031111,
+            0.01909951327913368,
+            0.014360281326789544,
+            0.003732344102045803,
+            0.008623457695718206,
+            0.014461868085333023,
+            -0.001692149528228451,
+            0.01948035376572433,
+            0.01172550239477637,
+            0.0031966531144076173,
+            0.0004779413449255626,
+            0.0030912562273611517,
+            -0.004597633235071031,
+            0.01582547400214653,
+            -0.014811177637844808,
+            -0.02411202772171917,
+            -0.0071272238837422185,
+            0.018015541137524113,
+            -0.013081287852028074,
+            1.8837689271256874e-05,
+            0.004062803284646432,
+            0.027569726868200254,
+            0.0010564850664255057,
+            -0.01179755720247972,
+            0.0028876134800690334,
+            -0.01638215925131053,
+            0.022944183106912663,
+            0.0026043713064243836,
+            0.03279173899194346,
+            -0.005103903273968361,
+            0.00578213669463383,
+            -0.020744194169889544,
+            -0.007588793365216579,
+            -0.00819862611489168,
+            0.0017401442433921013,
+            0.010993500174794084,
+            0.005411238350013134,
+            0.01620925462604661,
+            -0.00641615069699833,
+            -0.020751730783437995,
+            0.014776878596707538,
+            -0.009552193304486048,
+            -0.03393319554458277,
+            0.008881987295131347,
+            -0.029102297123590998,
+            -0.0045267103602711,
+            0.00368971742585232,
+            -0.014269341759285646,
+            -0.016264809810970175,
+            0.012837302095591106,
+            -0.0011890334180728564,
+            -0.008937992768323412,
+            0.0014444613725248846,
+            -0.017110187133760626,
+            -0.02385668037818897,
+            0.006017405364919834,
+            0.00676670672597817,
+            -0.0077994500447595,
+            0.018312452858615755,
+            -0.027015660037024585,
+            0.006123912473734307,
+            -0.012973676426738582,
+            0.009041974762537394,
+            -0.01580183887345168,
+            0.01163516332546347,
+            0.010887662470499361,
+            -0.001503467664709027,
+            -0.0076234124970223326,
+            0.01231202541338883,
+            0.004852720134894811,
+            0.033760365210093034,
+            0.028658326381800395,
+            0.000995999783627898,
+            -0.0025605367058226737,
+            -0.0020356794893366384,
+            0.011667525096213252,
+            -0.014049700287539817,
+            0.011218246246280464,
+            0.02658113044439717,
+            0.017568631325838108,
+            -0.019133062654008885,
+            0.018050678772708717,
+            -0.010356118256625172,
+            0.012872052420999813,
+            -0.025620154415048252,
+            -0.005135768485044442,
+            0.004936490920616527,
+            0.005163008729256001,
+            0.007425278892536464,
+            0.001410212306439329,
+            0.021891387289347516,
+            0.014999616600766989,
+            0.00027613811156650413,
+            0.00558422822386443,
+            0.01477009526981092,
+            0.010497782599293994,
+            -0.0005567894749233204,
+            -0.00467776506604011,
+            -0.016998035358377343,
+            0.00957271797745408,
+            -0.006065116857898469,
+            0.005138070401846389,
+            0.01698914855850691,
+            -0.01230257809343985,
+            0.011109844802663414,
+            0.007518553660463051,
+            -0.015504773927595823,
+            -0.017169933945839305,
+            0.005552829003043926,
+            -0.02252633790984703,
+            -0.01306722099657118,
+            0.02715689072730398,
+            -0.010513941399990838,
+            -0.02790582732974509,
+            -0.0009847768069205477,
+            0.009797865241108237,
+            -0.002772952073769185,
+            -0.0027968741526000576,
+            0.0037541733651479613,
+            0.010801137722284016,
+            -0.014658156970099418,
+            0.002075415233345957,
+            0.004735264309137368,
+            0.01930804315461151,
+            -0.0014151522239496115,
+            -0.015046118414100617,
+            -0.002288495894227231,
+            0.021592829677937263,
+            0.017174609713578173,
+            0.00265560378681856,
+            -0.016499825201963983,
+            -3.965296928410772e-05,
+            -0.0074142747014567635,
+            0.00021360696629503726,
+            0.00027431174904249525,
+            -0.0056391089282546054,
+            -0.018931681700770225,
+            -0.003888248121485032,
+            0.011181675419027736,
+            -0.00247252920175839,
+            0.01977860312223846,
+            -0.0011527575128223392,
+            -0.015763587433200915,
+            0.001960734212846797,
+            -0.005280860341480037,
+            -0.010372829570478649,
+            0.01351900743232294,
+            -0.009359942374603997,
+            0.006788746813046239,
+            -0.003212015225284119,
+            0.004611143085183367,
+            -0.0003308099698088422,
+            -0.017017265064185255,
+            0.02924142709831441,
+            -0.0018238549856433466,
+            -0.004769274041546295,
+            -0.0014058363976586562,
+            -0.033393331166741945,
+            0.015017165991303838,
+            -0.014574579250749552,
+            0.006745699409061803,
+            0.002705853802154516,
+            0.01272741296341735,
+            0.003253095246601883,
+            -0.008898616461779179,
+            0.0027485606928249326,
+            0.017040360927723885,
+            0.013756402798365262,
+            -0.007292864293090161,
+            0.008694855886257706,
+            -0.0028934389922887676,
+            -0.0037483208383542015,
+            0.01363963339139356,
+            0.004170611039133969,
+            -0.006282657671540625,
+            -0.0009784372731140945
+          ],
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "Mixture-of-Experts (MoE)",
+            "fine-grained expert segmentation",
+            "shared expert isolation"
+          ],
+          "title": "Deepseekmoe: Towards Ultimate Expert Specialization In Mixture-Of-Experts Language Models"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "1ea6fc21-365a-41fe-9b07-1e5411ca0bc9",
+        "properties": {
+          "page_content": "4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n",
+          "title": "Validation Experiments",
+          "embedding": [
+            -0.010754532180726528,
+            -0.023697586730122566,
+            -0.02424471639096737,
+            -0.004748920910060406,
+            0.0029044931288808584,
+            0.014695581048727036,
+            -0.013233717530965805,
+            0.03884626179933548,
+            -0.0167387705296278,
+            0.032195206731557846,
+            -0.028296900913119316,
+            0.003022040706127882,
+            0.033374954015016556,
+            -0.032622650265693665,
+            -0.007941939868032932,
+            0.013165325857698917,
+            0.035289913415908813,
+            0.03330656513571739,
+            0.011327309533953667,
+            -0.037751998752355576,
+            0.04247099906206131,
+            -0.036520954221487045,
+            -0.060047563165426254,
+            -0.011079391464591026,
+            -0.019577009603381157,
+            0.02089354395866394,
+            0.00764272827655077,
+            0.016584891825914383,
+            -0.0636381059885025,
+            0.03330656513571739,
+            0.00491989916190505,
+            0.023150455206632614,
+            0.0002941360289696604,
+            0.008754086680710316,
+            -0.011498288251459599,
+            -0.03205842152237892,
+            0.022381054237484932,
+            -0.0009312971960753202,
+            -0.015362396836280823,
+            0.042607780545949936,
+            0.03381950035691261,
+            0.01822628267109394,
+            -0.023372728377580643,
+            -0.004340710584074259,
+            0.016832809895277023,
+            0.00703148078173399,
+            -0.013173874467611313,
+            -0.010062070563435555,
+            0.015704352408647537,
+            -0.04240260645747185,
+            -0.03546088933944702,
+            -0.011600874364376068,
+            0.008172760717570782,
+            -0.01200267393141985,
+            -0.03761521726846695,
+            0.007856450974941254,
+            0.016140347346663475,
+            0.026159673929214478,
+            0.04093219339847565,
+            0.015439336188137531,
+            0.027305228635668755,
+            -0.018602434545755386,
+            -0.019765086472034454,
+            0.010164657607674599,
+            0.0010231980122625828,
+            -0.03443502262234688,
+            0.0016926847165450454,
+            -0.011720559559762478,
+            0.017046531662344933,
+            -0.034383729100227356,
+            -0.04520665109157562,
+            0.02380017377436161,
+            -0.009292668662965298,
+            -0.010993901640176773,
+            -0.0009270227164961398,
+            -0.010198852978646755,
+            0.031203532591462135,
+            0.027544597163796425,
+            -0.0026907704304903746,
+            0.000909390626475215,
+            0.014806716702878475,
+            0.02405664138495922,
+            -0.023714683949947357,
+            0.0028040434699505568,
+            0.04373623803257942,
+            -0.0030434129294008017,
+            -0.02814302034676075,
+            -0.06527949869632721,
+            0.06415104120969772,
+            -0.0167815163731575,
+            -0.013644064776599407,
+            0.0002978761913254857,
+            0.023646293208003044,
+            -0.019577009603381157,
+            -0.029237281531095505,
+            0.05844036862254143,
+            -0.007394809741526842,
+            0.0007074225577525795,
+            -0.018619531765580177,
+            0.014601543545722961,
+            -0.009771407581865788,
+            0.010865667834877968,
+            -0.008617304265499115,
+            0.014422016218304634,
+            0.032622650265693665,
+            -0.024877335876226425,
+            -0.014447662979364395,
+            0.012831917963922024,
+            0.013712456449866295,
+            0.10628008842468262,
+            -0.016217287629842758,
+            -0.005920122377574444,
+            -0.03257135674357414,
+            0.0011006725253537297,
+            -0.009745760820806026,
+            -0.05006243288516998,
+            -0.026826487854123116,
+            -0.025373173877596855,
+            0.031237727031111717,
+            -0.008035978302359581,
+            0.033836595714092255,
+            0.032434575259685516,
+            0.045035671442747116,
+            -0.0132422661408782,
+            -0.04804489016532898,
+            0.046950630843639374,
+            -0.0051549943163990974,
+            0.000823367154225707,
+            -0.0025347527116537094,
+            -0.003041275776922703,
+            -0.0017194000538438559,
+            -0.007373437285423279,
+            -0.01822628267109394,
+            -0.01443056482821703,
+            0.011831695213913918,
+            0.0043449848890304565,
+            0.034298237413167953,
+            0.053003259003162384,
+            -0.04373623803257942,
+            -0.009685918688774109,
+            0.026792293414473534,
+            -0.029938293620944023,
+            0.010985353030264378,
+            0.006091100629419088,
+            -0.02403954230248928,
+            0.015011890791356564,
+            0.0010638053063303232,
+            0.055465348064899445,
+            0.0003606572572607547,
+            0.02374888025224209,
+            0.0011113586369901896,
+            0.04274456575512886,
+            -0.008360836654901505,
+            0.04363365098834038,
+            0.010592103004455566,
+            0.0010258695110678673,
+            -0.020927738398313522,
+            0.018602434545755386,
+            -0.039427585899829865,
+            -0.000900841667316854,
+            0.03171646595001221,
+            -0.0211500097066164,
+            -0.009899641387164593,
+            0.012660940177738667,
+            0.004398415796458721,
+            0.06869906187057495,
+            -0.01395182590931654,
+            -0.0074888477101922035,
+            -0.0781712606549263,
+            0.04982306435704231,
+            0.023971151560544968,
+            0.00814711395651102,
+            0.005907298997044563,
+            0.04182128235697746,
+            0.003832050133496523,
+            0.01309693418443203,
+            0.012438667938113213,
+            -0.0098397983238101,
+            -0.05026760697364807,
+            0.03207552060484886,
+            -0.0026587119791656733,
+            0.012908858247101307,
+            -0.009130238555371761,
+            0.026860684156417847,
+            0.05505499988794327,
+            0.022021999582648277,
+            0.01165216788649559,
+            0.027476206421852112,
+            -0.020671270787715912,
+            0.01749962382018566,
+            0.037033889442682266,
+            -0.0011412798194214702,
+            0.03744423761963844,
+            0.026980368420481682,
+            -0.005979964509606361,
+            -0.013105483725667,
+            -0.028348194435238838,
+            0.013797945342957973,
+            0.006086825858801603,
+            0.03405886888504028,
+            -0.02034641243517399,
+            -0.03525571525096893,
+            0.007664100266993046,
+            -0.0034281141124665737,
+            0.04079541191458702,
+            -0.00934396218508482,
+            0.007822255603969097,
+            0.023099161684513092,
+            0.010942608118057251,
+            -0.004847233649343252,
+            -0.024210521951317787,
+            -0.008540363982319832,
+            -0.01188298873603344,
+            -0.0402824766933918,
+            -0.011891537345945835,
+            0.03157968446612358,
+            -0.015353847295045853,
+            0.01938893459737301,
+            -0.005915847606956959,
+            -0.007800883147865534,
+            -0.023543706163764,
+            0.018072402104735374,
+            0.0032464496325701475,
+            -0.008027428761124611,
+            -0.0069075217470526695,
+            0.003357585519552231,
+            -0.0019352601375430822,
+            0.003267821855843067,
+            0.048181671649217606,
+            -0.02402244508266449,
+            0.012113809585571289,
+            0.008527540601789951,
+            0.0011316622840240598,
+            0.02380017377436161,
+            0.03346044570207596,
+            -0.035563476383686066,
+            -0.012934505008161068,
+            -0.0014618640998378396,
+            0.009514939971268177,
+            0.020876444876194,
+            -0.009873994626104832,
+            -0.0097030159085989,
+            -0.03378530219197273,
+            -0.0005765173118561506,
+            0.00013043770741205662,
+            -0.015712901949882507,
+            -0.008467698469758034,
+            0.017166217789053917,
+            0.042641978710889816,
+            0.032434575259685516,
+            0.02034641243517399,
+            -0.035289913415908813,
+            -0.00907894503325224,
+            0.027082955464720726,
+            -0.014610092155635357,
+            0.034315336495637894,
+            0.0504043884575367,
+            -0.029459552839398384,
+            -0.018773412331938744,
+            -0.03980373963713646,
+            0.07112695276737213,
+            -0.023731783032417297,
+            -0.02964762970805168,
+            0.042368412017822266,
+            -0.01408860832452774,
+            0.0021778354421257973,
+            0.03187034651637077,
+            0.035324107855558395,
+            -0.0035307009238749743,
+            0.012053967453539371,
+            0.03499924764037132,
+            -0.0194060318171978,
+            -0.04691643267869949,
+            -0.02523639053106308,
+            -0.025116706266999245,
+            -0.016619086265563965,
+            -0.010275793261826038,
+            -0.017311548814177513,
+            0.010087717324495316,
+            0.020278021693229675,
+            0.013635516166687012,
+            -0.010882765986025333,
+            0.007022931706160307,
+            0.009985130280256271,
+            -0.009976581670343876,
+            -0.000917939527425915,
+            -0.027561694383621216,
+            -0.0071939099580049515,
+            -0.028467880561947823,
+            -0.012053967453539371,
+            -0.014977695420384407,
+            -0.006685249973088503,
+            -0.025783522054553032,
+            0.01889309659600258,
+            0.023167554289102554,
+            -0.002190658822655678,
+            -0.008459148928523064,
+            0.01238737441599369,
+            0.03472568467259407,
+            -0.028108825907111168,
+            -0.03812815248966217,
+            0.008916515856981277,
+            -0.0034558980260044336,
+            0.009762858040630817,
+            -0.015037537552416325,
+            0.010677591897547245,
+            -0.011395701207220554,
+            0.010540809482336044,
+            -0.010703238658607006,
+            -0.010002228431403637,
+            -0.0006646779947914183,
+            0.021953608840703964,
+            0.00801460538059473,
+            -0.014592994004487991,
+            -0.017593663185834885,
+            0.027715574949979782,
+            0.014293782413005829,
+            0.0334775410592556,
+            -0.029545042663812637,
+            -0.023099161684513092,
+            0.042573586106300354,
+            0.004015851765871048,
+            -0.043086521327495575,
+            0.016148896887898445,
+            0.01671312376856804,
+            0.011797499842941761,
+            0.01942313089966774,
+            0.048181671649217606,
+            -0.033870793879032135,
+            -0.038196541368961334,
+            0.005420010536909103,
+            -0.031938739120960236,
+            -0.030929965898394585,
+            0.014883656986057758,
+            -0.006227883044630289,
+            -0.05125928297638893,
+            -0.031203532591462135,
+            -0.014020216651260853,
+            -0.019337641075253487,
+            0.0022098938934504986,
+            -0.015046087093651295,
+            0.004941271618008614,
+            -0.03741004317998886,
+            0.0025689483154565096,
+            0.0004648471367545426,
+            -0.018328869715332985,
+            -0.028724348172545433,
+            0.0027548871003091335,
+            0.004488179460167885,
+            0.03542669489979744,
+            0.0017097825184464455,
+            0.01128456462174654,
+            -0.012951603159308434,
+            -0.000579188868869096,
+            0.003056236309930682,
+            -0.05016501992940903,
+            -0.019474424421787262,
+            0.038983043283224106,
+            0.009027652442455292,
+            0.006048355717211962,
+            0.011874440126121044,
+            -0.028211412951350212,
+            -0.0592268668115139,
+            0.029716020449995995,
+            -0.04643769562244415,
+            0.015268358401954174,
+            0.004415513481944799,
+            -0.0010344184702262282,
+            -0.001160514890216291,
+            -0.0007758138235658407,
+            -0.0067707388661801815,
+            0.023167554289102554,
+            -0.007407633122056723,
+            0.012498511001467705,
+            -0.0028788463678210974,
+            0.03669193387031555,
+            0.0358712375164032,
+            -0.00673226872459054,
+            -0.038948848843574524,
+            0.02956213988363743,
+            -0.008668597787618637,
+            0.025612542405724525,
+            0.01889309659600258,
+            0.02961343340575695,
+            -0.017918521538376808,
+            0.032417476177215576,
+            0.002891669748350978,
+            0.024125032126903534,
+            -0.03877786919474602,
+            0.023167554289102554,
+            0.0052191112190485,
+            -0.054268497973680496,
+            -0.00860020611435175,
+            -0.0073777115903794765,
+            0.022244270890951157,
+            0.008822478353977203,
+            -0.02477474883198738,
+            -0.021970706060528755,
+            0.0009409147314727306,
+            -0.02817721664905548,
+            -0.007574336603283882,
+            0.04568539187312126,
+            -0.012558353133499622,
+            0.0014875108608976007,
+            -0.017414135858416557,
+            0.014174097217619419,
+            0.02886112965643406,
+            0.027510400861501694,
+            0.03354593366384506,
+            -0.013986021280288696,
+            0.020021554082632065,
+            -0.022278467193245888,
+            -0.037033889442682266,
+            -0.02595449984073639,
+            0.0073050460778176785,
+            0.03939339146018028,
+            0.009728662669658661,
+            0.006039807107299566,
+            -0.0013913356233388186,
+            0.00790346972644329,
+            -0.015370945446193218,
+            0.027989141643047333,
+            0.044796302914619446,
+            -0.002494145417585969,
+            -0.011464091949164867,
+            0.007505945395678282,
+            0.015046087093651295,
+            -0.018619531765580177,
+            0.027373619377613068,
+            -0.025424467399716377,
+            0.02255203202366829,
+            0.004744646605104208,
+            0.02065417356789112,
+            -0.047702934592962265,
+            -0.004556570667773485,
+            -0.0023082063999027014,
+            -0.03445211797952652,
+            0.05946623906493187,
+            -0.028690151870250702,
+            -0.013524380512535572,
+            0.0029386887326836586,
+            -0.0011316622840240598,
+            -0.01313113048672676,
+            -0.026980368420481682,
+            -0.014849461615085602,
+            -0.0061894129030406475,
+            -0.04554860666394234,
+            -0.029801510274410248,
+            -0.014815266244113445,
+            -0.01213945634663105,
+            -0.06240706518292427,
+            0.004058596212416887,
+            -0.0741361752152443,
+            0.007206733338534832,
+            0.027715574949979782,
+            -0.016661832109093666,
+            0.03963275998830795,
+            -0.022637521848082542,
+            0.02374888025224209,
+            -0.04076121747493744,
+            -0.0157299991697073,
+            0.03258845582604408,
+            0.047463566064834595,
+            0.010848570615053177,
+            0.011498288251459599,
+            0.021047422662377357,
+            -0.03199003264307976,
+            -0.0026758096646517515,
+            -0.00860875565558672,
+            -0.0018219870980829,
+            0.03332366421818733,
+            -0.008553187362849712,
+            0.021765531972050667,
+            0.0051549943163990974,
+            0.024142129346728325,
+            0.031442902982234955,
+            -0.002782671246677637,
+            -0.023868564516305923,
+            0.012857564724981785,
+            0.025544151663780212,
+            0.013054190203547478,
+            -0.014251037500798702,
+            0.012686586938798428,
+            -0.01579839177429676,
+            0.007719668094068766,
+            0.008484795689582825,
+            0.009908189997076988,
+            0.034315336495637894,
+            0.024142129346728325,
+            -0.013105483725667,
+            0.0029814334120601416,
+            0.009865445084869862,
+            -0.03645256534218788,
+            0.042641978710889816,
+            0.0051849158480763435,
+            -0.03953017294406891,
+            -0.0005487333401106298,
+            -0.007617081515491009,
+            0.020551586523652077,
+            0.02156035788357258,
+            0.00936105940490961,
+            -0.022671716287732124,
+            -0.00020944836433045566,
+            0.016841357573866844,
+            0.008660049177706242,
+            -0.012182201258838177,
+            0.02450118400156498,
+            0.03833332657814026,
+            -0.020260922610759735,
+            -0.030639303848147392,
+            0.00922427698969841,
+            -0.010489515960216522,
+            0.0007736766128800809,
+            -0.02715134806931019,
+            -0.004394141025841236,
+            0.018072402104735374,
+            -0.044762108474969864,
+            0.05177221819758415,
+            -0.013575674034655094,
+            0.008685695938766003,
+            -0.012943054549396038,
+            -0.002703593811020255,
+            -0.02138938009738922,
+            0.034588903188705444,
+            0.014131353236734867,
+            0.02867305465042591,
+            0.016106151044368744,
+            0.010318538174033165,
+            0.028040433302521706,
+            -0.020278021693229675,
+            -0.023099161684513092,
+            0.014422016218304634,
+            -0.007980410009622574,
+            -0.00467198109254241,
+            -0.02959633618593216,
+            -0.024107934907078743,
+            -0.01892729289829731,
+            0.02527058683335781,
+            -0.01530255377292633,
+            -0.008566010743379593,
+            0.010951157659292221,
+            0.0014981969725340605,
+            -0.002131885150447488,
+            0.013943277299404144,
+            -0.007638453505933285,
+            0.008707067929208279,
+            0.04192386940121651,
+            -0.013934727758169174,
+            0.02159455418586731,
+            0.03611060976982117,
+            0.011472641490399837,
+            -0.018516944721341133,
+            -0.0064800758846104145,
+            -0.0007293291273526847,
+            -0.01066904328763485,
+            -0.03570026159286499,
+            0.035289913415908813,
+            0.020996129140257835,
+            -0.0021842471323907375,
+            -0.037239063531160355,
+            -0.012583999894559383,
+            0.014635738916695118,
+            -0.014037314802408218,
+            0.027732674032449722,
+            -0.015943722799420357,
+            -0.015653058886528015,
+            0.00474037230014801,
+            0.021970706060528755,
+            -0.049857258796691895,
+            -0.012977249920368195,
+            0.010899864137172699,
+            0.002707868115976453,
+            0.008057350292801857,
+            -0.006779287941753864,
+            0.013455988839268684,
+            0.020568683743476868,
+            0.004945545922964811,
+            0.0036439741961658,
+            -0.0038085407577455044,
+            0.01647375524044037,
+            0.013455988839268684,
+            -0.017593663185834885,
+            -0.009241375140845776,
+            0.02210748940706253,
+            0.0230820644646883,
+            -0.030211858451366425,
+            -0.023919858038425446,
+            -0.00583035871386528,
+            0.021902315318584442,
+            -0.004753195680677891,
+            0.03689710795879364,
+            0.0388120636343956,
+            0.011438445188105106,
+            -0.04318910837173462,
+            -0.014516053721308708,
+            0.0017290175892412663,
+            -0.010660494677722454,
+            0.016431011259555817,
+            -0.0012438668636605144,
+            -0.042812954634428024,
+            0.002143639838322997,
+            -0.0036055040545761585,
+            0.028450781479477882,
+            -0.040214087814092636,
+            -0.011959929019212723,
+            0.009617527015507221,
+            -0.012951603159308434,
+            0.003302017692476511,
+            0.012182201258838177,
+            0.008484795689582825,
+            0.021269695833325386,
+            0.012703684158623219,
+            -0.005526872351765633,
+            -0.0014607955235987902,
+            0.016550695523619652,
+            0.004535198211669922,
+            0.026604216545820236,
+            -0.018328869715332985,
+            -0.017345745116472244,
+            0.01844855397939682,
+            0.03860688954591751,
+            -0.012327532283961773,
+            0.04076121747493744,
+            0.007514494471251965,
+            0.012062516063451767,
+            0.012088162824511528,
+            -0.02453538030385971,
+            -0.01598646678030491,
+            -0.011951380409300327,
+            0.006257804110646248,
+            -0.037512630224227905,
+            0.014858010224997997,
+            -0.02380017377436161,
+            -0.027134248986840248,
+            0.02619386836886406,
+            -0.006014160346239805,
+            0.010686141438782215,
+            0.022244270890951157,
+            -0.012891761027276516,
+            -0.027271032333374023,
+            0.03369981423020363,
+            0.034349530935287476,
+            0.008433502167463303,
+            0.04677965119481087,
+            -0.010395478457212448,
+            0.006779287941753864,
+            0.005702124908566475,
+            0.0024920080322772264,
+            0.044762108474969864,
+            -0.008236877620220184,
+            -0.01200267393141985,
+            0.007535866927355528,
+            0.009779956191778183,
+            0.04206065088510513,
+            -0.015413690358400345,
+            -0.049412716180086136,
+            0.019508618861436844,
+            -0.014507505111396313,
+            0.009856896474957466,
+            -0.006655328441411257,
+            0.008198407478630543,
+            -0.005890200845897198,
+            0.010053521953523159,
+            -0.004522374831140041,
+            0.010113364085555077,
+            -0.005843182094395161,
+            -0.014327977783977985,
+            0.015174319967627525,
+            -0.01018175482749939,
+            0.030280249193310738,
+            0.009557684883475304,
+            -0.02503121644258499,
+            -0.026501629501581192,
+            0.01746542938053608,
+            0.0019277798710390925,
+            -0.028724348172545433,
+            0.009882543236017227,
+            0.0023872838355600834,
+            0.010104815475642681,
+            0.004381317645311356,
+            -0.012242043390870094,
+            0.015926625579595566,
+            0.01676441729068756,
+            -0.022210076451301575,
+            0.03911982476711273,
+            -0.014285233803093433,
+            -0.014644287526607513,
+            0.019235054031014442,
+            -0.03730745613574982,
+            0.008424953557550907,
+            -0.012524157762527466,
+            0.010831472463905811,
+            0.00491989916190505,
+            0.004667706321924925,
+            -0.011352956295013428,
+            -0.0026715353596955538,
+            -0.02352660894393921,
+            0.03281072899699211,
+            0.0272197388112545,
+            -0.0012834055814892054,
+            0.005958592053502798,
+            -0.007535866927355528,
+            -0.012900309637188911,
+            0.0024984197225421667,
+            0.026142574846744537,
+            -0.024381499737501144,
+            -0.0017899286467581987,
+            -0.0006310166209004819,
+            0.007694021798670292,
+            0.00551832327619195,
+            0.022432347759604454,
+            0.04900236800312996,
+            -0.0038192267529666424,
+            0.028980813920497894,
+            0.035050541162490845,
+            0.024142129346728325,
+            -0.003259273013100028,
+            0.004761744290590286,
+            -0.025150902569293976,
+            0.03645256534218788,
+            0.008950712159276009,
+            0.016670379787683487,
+            0.017798837274312973,
+            -0.04312071576714516,
+            -0.009634625166654587,
+            0.019525716081261635,
+            0.02205619588494301,
+            0.020089944824576378,
+            -0.003881206503137946,
+            -0.0035884061362594366,
+            0.03298170492053032,
+            -0.025920303538441658,
+            -0.019354738295078278,
+            0.004154771566390991,
+            -0.0053174239583313465,
+            -0.005732045974582434,
+            -0.0069032469764351845,
+            0.01865372806787491,
+            -0.014584445394575596,
+            -0.004458257928490639,
+            0.016867004334926605,
+            0.001722605898976326,
+            0.012113809585571289,
+            0.016165994107723236,
+            -0.00024137321452144533,
+            0.022415248677134514,
+            -0.004932722542434931,
+            0.021509064361453056,
+            0.005330247338861227,
+            -0.003663209266960621,
+            -0.006911796052008867,
+            -0.010275793261826038,
+            0.029015010222792625,
+            -0.006980187259614468,
+            -0.034640196710824966,
+            -0.01360986940562725,
+            -0.009284119121730328,
+            0.004156908951699734,
+            0.0011391426669433713,
+            0.0050524077378213406,
+            0.01744833029806614,
+            0.026501629501581192,
+            -0.00046751866466365755,
+            0.004280867986381054,
+            0.014336527325212955,
+            -0.0024193422868847847,
+            -0.010506614111363888,
+            0.019559912383556366,
+            0.008779733441770077,
+            -0.01624293439090252,
+            0.018756315112113953,
+            0.0030861576087772846,
+            0.04688223823904991,
+            -0.044522739946842194,
+            0.018209183588624,
+            0.007805157452821732,
+            0.0006956678116694093,
+            -0.014960597269237041,
+            -0.03327237069606781,
+            0.0036097783595323563,
+            -0.014122803695499897,
+            -0.0033597226720303297,
+            -0.024689260870218277,
+            0.00786499958485365,
+            -0.018089499324560165,
+            0.014045863412320614,
+            0.011600874364376068,
+            -0.0016349796205759048,
+            -0.002028229646384716,
+            0.017542369663715363,
+            -0.001940603251568973,
+            0.00787782296538353,
+            -0.017362842336297035,
+            0.010942608118057251,
+            0.022381054237484932,
+            -0.010677591897547245,
+            0.008638676255941391,
+            0.0073306928388774395,
+            -0.02793784812092781,
+            -0.028536271303892136,
+            0.013336303643882275,
+            -0.03546088933944702,
+            0.007681198418140411,
+            0.031203532591462135,
+            -0.032434575259685516,
+            -0.028519174084067345,
+            0.0019085448002442718,
+            -0.021406477317214012,
+            -0.017867228016257286,
+            0.011446994729340076,
+            0.018021108582615852,
+            -0.004216751083731651,
+            -0.030451226979494095,
+            0.013173874467611313,
+            -0.007809431757777929,
+            0.00698873633518815,
+            0.00015388043539132923,
+            -0.02569803223013878,
+            -0.02569803223013878,
+            0.004984016064554453,
+            -0.028211412951350212,
+            -0.007138342130929232,
+            -0.0005297654424794018,
+            -0.024466989561915398,
+            -0.043804630637168884,
+            -0.015242711640894413,
+            -0.013507282361388206,
+            -0.010156108066439629,
+            0.0480106957256794,
+            0.0196795966476202,
+            0.026894880458712578,
+            -0.0003211185394320637,
+            -0.018106596544384956,
+            0.04773712903261185,
+            0.0009954140987247229,
+            0.041752889752388,
+            0.013387597166001797,
+            -0.008335189893841743,
+            -0.006693798583000898,
+            0.03392208740115166,
+            -0.015584668144583702,
+            -0.007775236386805773,
+            -0.010651945136487484,
+            -0.0015836860984563828,
+            0.0025091059505939484,
+            0.022449444979429245,
+            0.0019085448002442718,
+            0.004037224221974611,
+            0.027100054547190666,
+            -0.023201748728752136,
+            0.002739926567301154,
+            0.01238737441599369,
+            -0.006937442813068628,
+            0.008215505629777908,
+            -0.02838239073753357,
+            -0.008300994522869587,
+            0.019081173464655876,
+            0.006044081412255764,
+            -0.007437554188072681,
+            -0.0022440897300839424,
+            -0.027134248986840248,
+            -0.0014052275801077485,
+            0.0038106779102236032,
+            -0.005826083943247795,
+            0.005744869355112314,
+            0.04459112882614136,
+            -0.004817312583327293,
+            -0.039940521121025085,
+            -0.00029493749025277793,
+            -0.03634997829794884,
+            0.0001610935723874718,
+            0.00010799681331263855,
+            -0.0033191153779625893,
+            0.005488402210175991,
+            0.0018422907451167703,
+            -0.012472864240407944,
+            0.012130907736718655,
+            -0.028741445392370224,
+            0.019782183691859245,
+            0.004817312583327293,
+            -0.007574336603283882,
+            -0.005056682042777538,
+            0.01988477073609829,
+            0.008151388727128506,
+            -0.013148227706551552,
+            -0.0014960597036406398,
+            0.0028852580580860376,
+            -0.026125477626919746,
+            -0.020773857831954956,
+            -0.03369981423020363,
+            -0.0022505014203488827,
+            -0.009583331644535065,
+            -0.03010927140712738,
+            -0.006937442813068628,
+            -0.0003617258626036346,
+            0.007775236386805773,
+            -0.020295118913054466,
+            0.0006956678116694093,
+            0.005552519112825394,
+            0.012199298478662968,
+            0.00624070642516017,
+            -0.007339241914451122,
+            -0.004263770300894976,
+            -0.03033154271543026,
+            0.003584131831303239,
+            -0.0015345298452302814,
+            0.0038619714323431253,
+            -0.016063407063484192,
+            0.006351842079311609,
+            -0.016183091327548027,
+            -0.0035264266189187765,
+            0.011335858143866062,
+            0.01800401136279106,
+            -0.028741445392370224,
+            0.036726128309965134,
+            0.02889532595872879,
+            0.017320098355412483,
+            -0.038914650678634644,
+            0.028587564826011658,
+            -0.0012331806356087327,
+            0.001522775157354772,
+            0.0030177661683410406,
+            0.021731335669755936,
+            0.026039987802505493,
+            -0.016431011259555817,
+            -0.006531369406729937,
+            0.010224499739706516,
+            0.028194313868880272,
+            -0.031545490026474,
+            -0.007266575936228037,
+            1.9886239897459745e-05,
+            0.02378307469189167,
+            0.027783967554569244,
+            0.023116260766983032,
+            0.04271036759018898,
+            -0.0053131491877138615,
+            0.002891669748350978,
+            -0.005082328803837299,
+            -0.013926179148256779,
+            0.0024727729614824057,
+            -0.0028767092153429985,
+            -0.0017952717607840896,
+            0.011489738710224628,
+            0.012695135548710823,
+            0.017593663185834885,
+            -0.07126373797655106,
+            0.038743674755096436,
+            0.026570022106170654,
+            0.012105260975658894,
+            0.008399306796491146,
+            -0.024706358090043068,
+            -0.0017909972229972482,
+            -0.0022205801215022802,
+            0.0018604571232572198,
+            0.013438890688121319,
+            -0.010874217376112938,
+            -0.02475765161216259,
+            0.016567792743444443,
+            -0.019457325339317322,
+            0.017149118706583977,
+            0.0050267609767615795,
+            0.00503958435729146,
+            -0.024603771045804024,
+            -0.014721227809786797,
+            0.006830581463873386,
+            0.008065898902714252,
+            0.008100095205008984,
+            -0.03857269510626793,
+            -0.044727910310029984,
+            0.01750817336142063,
+            0.008035978302359581,
+            -0.04862621799111366,
+            -0.006586937233805656,
+            -0.017388489097356796,
+            -0.004210339393466711,
+            -0.006326195318251848,
+            -0.01767915114760399,
+            -0.040897998958826065,
+            0.002064562402665615,
+            0.017302999272942543,
+            0.018534043803811073,
+            5.349749335437082e-05,
+            0.028826933354139328,
+            0.0017183314776048064,
+            -0.0022355406545102596,
+            -0.030434129759669304,
+            0.026860684156417847,
+            -0.007326418533921242,
+            -0.022278467193245888,
+            0.024142129346728325,
+            0.01628567837178707,
+            -0.019970260560512543,
+            -0.0038769319653511047,
+            0.0014928538585081697,
+            -0.008446325547993183,
+            0.034554705023765564,
+            -0.0017129883635789156,
+            -0.008933614008128643,
+            0.0026907704304903746,
+            -0.0020998266991227865,
+            -0.00959188025444746,
+            -0.02479184791445732,
+            0.00145438383333385,
+            0.025869010016322136,
+            -0.007749589625746012,
+            0.006437331438064575,
+            0.04202645644545555,
+            -0.012353179045021534,
+            -0.019850576296448708,
+            0.00043145293602719903,
+            -0.02839948795735836,
+            0.010318538174033165,
+            -0.006685249973088503,
+            0.030280249193310738,
+            0.004206065088510513,
+            0.013002896681427956,
+            -0.016644733026623726,
+            -0.01066904328763485,
+            -0.021970706060528755,
+            -0.0641852393746376,
+            -0.012122358195483685,
+            -0.005561067722737789,
+            -0.024894434958696365,
+            -0.02398824878036976,
+            0.03234908729791641,
+            0.022398151457309723,
+            0.010609201155602932,
+            -0.002107307082042098,
+            -0.01891019567847252,
+            -0.004458257928490639,
+            0.03009217418730259,
+            -0.005458480678498745,
+            0.017371391877532005,
+            -0.011617972515523434,
+            0.01627713069319725,
+            -0.006578388623893261,
+            0.0051336223259568214,
+            -0.053960736840963364,
+            -0.005167817696928978,
+            0.02983570657670498,
+            0.01993606425821781,
+            0.007240929175168276,
+            0.010019325651228428,
+            0.013293559662997723,
+            -0.007501671090722084,
+            -0.02764718420803547,
+            0.02525348775088787,
+            0.017132021486759186,
+            0.01991896703839302,
+            -0.015935173258185387,
+            0.007587159983813763,
+            0.011788951233029366,
+            0.00288098375312984,
+            0.000775279535446316,
+            0.0008281759219244123,
+            0.025133803486824036,
+            0.003549935994669795,
+            -0.009275570511817932,
+            0.015721451491117477,
+            -0.007163988891988993,
+            -0.021269695833325386,
+            -0.004233849234879017,
+            0.008959260769188404,
+            0.004035086836665869,
+            0.001731154858134687,
+            -0.0015922350576147437,
+            -0.008984907530248165,
+            0.02886112965643406,
+            0.03539250046014786,
+            0.03183615207672119,
+            -0.0057876138016581535,
+            -0.005137896630913019,
+            0.0016456657322123647,
+            0.003513603238388896,
+            -0.007659825962036848,
+            -0.0018294673645868897,
+            0.019799282774329185,
+            -0.030006684362888336,
+            -0.011276016011834145,
+            -0.009318315424025059,
+            0.016362618654966354,
+            0.003522152081131935,
+            0.04411239176988602,
+            0.02111581526696682,
+            0.006522820331156254,
+            -0.00557389110326767,
+            0.0037209142465144396,
+            0.004389866720885038,
+            0.029237281531095505,
+            -0.02282559685409069,
+            9.023110033012927e-05,
+            -0.0034217024222016335,
+            -0.017901424318552017,
+            0.019046977162361145,
+            -0.007300771772861481,
+            0.009540586732327938,
+            0.02205619588494301,
+            0.09212308377027512,
+            -0.0065527418628335,
+            0.022449444979429245,
+            -0.027852358296513557,
+            -0.024090835824608803,
+            0.00291517935693264,
+            0.006313371937721968,
+            0.002265461953356862,
+            0.022039096802473068,
+            0.0054499320685863495,
+            0.02525348775088787,
+            0.0272197388112545,
+            0.0009067190694622695,
+            0.006698073353618383,
+            -0.010292891412973404,
+            0.004118438810110092,
+            -0.013182424008846283,
+            -0.009549135342240334,
+            -0.009437999688088894,
+            0.008779733441770077,
+            0.019012782722711563,
+            0.00911314133554697,
+            0.007531592156738043,
+            -0.004084242973476648,
+            -0.015482081100344658,
+            0.026518728584051132,
+            0.010292891412973404,
+            -0.004475356079638004,
+            -0.015601766295731068,
+            -0.015114477835595608,
+            0.012404472567141056,
+            -0.006886149290949106,
+            -0.005894475616514683,
+            0.01721751131117344,
+            0.005069505423307419,
+            0.02865595556795597,
+            -0.03369981423020363,
+            0.008890869095921516,
+            0.014259587042033672,
+            -0.0010857119923457503,
+            0.007245203480124474,
+            0.021423576399683952,
+            -0.0022633245680481195,
+            0.0242789126932621,
+            -0.01468703243881464,
+            0.00874553807079792,
+            0.010959706269204617,
+            -0.021201303228735924,
+            0.0036055040545761585,
+            0.01990186981856823,
+            -0.003381095128133893,
+            0.015413690358400345,
+            -0.0052191112190485,
+            -0.001976936124265194,
+            0.0025903205387294292,
+            0.006582662928849459,
+            0.00363756250590086,
+            -0.01988477073609829,
+            0.01188298873603344,
+            -0.01872211880981922,
+            0.009172983467578888,
+            -0.004565119277685881,
+            0.012190749868750572,
+            -0.008424953557550907,
+            0.011370054446160793,
+            -0.006121021695435047,
+            0.006847679149359465,
+            -0.0105750048533082,
+            -0.016559245064854622,
+            -0.012515608221292496,
+            0.012960151769220829,
+            0.007249478250741959,
+            -0.013558575883507729,
+            0.008309543132781982,
+            -0.008228329010307789,
+            -0.002067768247798085,
+            0.014960597269237041,
+            -0.011746206320822239,
+            0.00407355697825551,
+            -0.039461780339479446,
+            -0.004535198211669922,
+            0.018089499324560165,
+            -0.001959838205948472,
+            0.0027719850186258554,
+            -0.007352065294981003,
+            0.011199075728654861,
+            -0.008288171142339706,
+            -0.0024086562916636467,
+            0.0254757609218359,
+            -0.019782183691859245,
+            -0.017328646034002304,
+            -0.032913316041231155,
+            -0.05009663105010986,
+            -0.002131885150447488,
+            -0.009326864033937454,
+            0.005864554084837437,
+            -0.00790346972644329,
+            0.017815934494137764,
+            -0.0040179891511797905,
+            -0.007095597684383392,
+            -0.024125032126903534,
+            0.010395478457212448,
+            0.028211412951350212,
+            0.0008682489860802889,
+            -0.03079318441450596,
+            -0.012344630435109138,
+            0.010463869199156761,
+            0.0006646779947914183,
+            -0.029049206525087357,
+            0.0083693852648139,
+            0.0006181932403706014,
+            -0.014473309740424156,
+            -0.008882320486009121,
+            -0.005971415434032679,
+            0.021440673619508743,
+            0.024193422868847847,
+            0.005753418430685997,
+            -0.019782183691859245,
+            -0.004225300159305334,
+            0.03258845582604408,
+            -0.018294673413038254,
+            0.003994479309767485,
+            0.023868564516305923,
+            -0.00264375121332705,
+            0.00982270110398531,
+            0.01054935809224844,
+            0.018072402104735374,
+            -0.008352288044989109,
+            0.012660940177738667,
+            0.007561513222754002,
+            -0.015328200533986092,
+            0.008548912592232227,
+            0.006668151821941137,
+            0.016405364498496056,
+            0.006890423595905304,
+            -0.022244270890951157,
+            -0.004411239176988602,
+            0.015832586213946342,
+            0.053276825696229935,
+            -0.006706621963530779,
+            -0.008506168611347675,
+            0.03617899864912033,
+            0.024415696039795876,
+            0.007416181731969118,
+            0.005937220063060522,
+            -0.03231488913297653,
+            -0.004646334331482649,
+            0.0008709204848855734,
+            0.005236208904534578,
+            0.0132422661408782,
+            0.0035029170103371143,
+            -0.007770961616188288,
+            0.018978586420416832,
+            -0.001043501659296453,
+            0.014293782413005829,
+            -0.020107043907046318,
+            0.009514939971268177,
+            -0.00618513859808445,
+            0.03161387890577316,
+            0.01042112521827221,
+            -0.0010632710764184594,
+            0.03318687900900841,
+            0.006719445344060659,
+            -0.03344334661960602,
+            0.006125296000391245,
+            -0.007638453505933285,
+            0.011352956295013428,
+            -0.02260332554578781,
+            -0.007894921116530895,
+            0.009899641387164593,
+            0.0012289062142372131,
+            -0.018123695626854897,
+            0.011900086887180805,
+            0.007142616901546717,
+            0.009378157556056976,
+            -0.0005217508296482265,
+            0.006356116849929094,
+            0.008553187362849712,
+            -0.004522374831140041,
+            -0.008168485946953297,
+            0.005599537864327431,
+            -0.016336971893906593,
+            -0.0040179891511797905,
+            0.029425358399748802,
+            -0.0006726926076225936,
+            0.010309988632798195,
+            0.0004135536728426814,
+            -0.0004116835771128535,
+            -0.018978586420416832,
+            0.009207178838551044,
+            -0.024569574743509293,
+            -0.006415958981961012,
+            0.005595263559371233,
+            -0.01225059200078249,
+            -0.006142393685877323,
+            0.014413466677069664,
+            0.0022633245680481195,
+            -0.021047422662377357,
+            -0.008061625063419342,
+            -0.030485423281788826,
+            -0.021218402311205864,
+            0.031904544681310654,
+            0.006518546026200056,
+            -0.005646557081490755,
+            0.015379494056105614,
+            -0.022876890376210213,
+            -0.0054969508200883865,
+            -0.005035309586673975,
+            0.016636185348033905,
+            -0.020021554082632065,
+            0.004398415796458721,
+            -0.010634847916662693,
+            -0.016431011259555817,
+            0.023099161684513092,
+            0.008326641283929348,
+            0.0011220448650419712,
+            0.006133845075964928,
+            -0.02497992292046547,
+            -0.014285233803093433,
+            0.01798691228032112,
+            0.005702124908566475,
+            -0.008228329010307789,
+            0.01795271784067154,
+            -0.0012609646655619144,
+            -0.007215282414108515,
+            -0.013780847191810608,
+            0.01841435767710209,
+            -0.02183392271399498,
+            0.026535825803875923,
+            0.012156554497778416,
+            -0.010472417809069157,
+            0.008822478353977203,
+            0.004992565140128136,
+            0.01287466287612915,
+            -0.0003889755462296307,
+            -0.015370945446193218,
+            0.013421793468296528,
+            0.013712456449866295,
+            0.002194933360442519,
+            -0.0021778354421257973,
+            0.0037572472356259823,
+            0.005826083943247795,
+            -0.0054456572979688644,
+            0.02333853207528591,
+            0.023594999685883522,
+            0.0002186651690863073,
+            0.015080282464623451,
+            -0.02084225043654442,
+            -0.019611205905675888,
+            0.02378307469189167,
+            -0.004595040809363127,
+            -0.0314599983394146,
+            0.038709476590156555,
+            -0.010292891412973404,
+            -0.007236654870212078,
+            -0.01993606425821781,
+            0.00771111948415637,
+            0.020448999479413033,
+            -0.021885216236114502,
+            -0.010002228431403637,
+            -0.00110708421561867,
+            0.008848125115036964,
+            0.027852358296513557,
+            0.020568683743476868,
+            -0.015507727861404419,
+            0.02788655459880829,
+            0.009668820537626743,
+            0.008044526912271976,
+            0.011874440126121044,
+            0.005561067722737789,
+            -0.010840021073818207,
+            0.010044972412288189,
+            -0.0005241552134975791,
+            0.005296051502227783,
+            -0.00516354339197278,
+            -0.02231266349554062,
+            -0.015644511207938194,
+            -0.04178708791732788,
+            -0.0036589347291737795,
+            -0.008604480884969234,
+            0.01553337462246418,
+            0.08624143153429031,
+            -0.014037314802408218,
+            -0.009309765882790089,
+            -0.0020271609537303448,
+            0.009976581670343876,
+            -0.01720896176993847,
+            0.012566901743412018,
+            0.024466989561915398,
+            -0.013626967556774616,
+            -0.0329304113984108,
+            0.0025518503971397877,
+            -0.012994347140192986,
+            0.007805157452821732,
+            0.017542369663715363,
+            0.0019245740259066224,
+            -0.023680489510297775,
+            -0.0018433593213558197,
+            -0.020004456862807274,
+            0.001239592325873673,
+            0.010369831696152687,
+            -0.016131797805428505,
+            -0.010771630331873894,
+            0.023902760818600655,
+            -0.02017543464899063,
+            -0.025167999789118767,
+            -0.012925956398248672,
+            -0.007745314855128527,
+            -0.011891537345945835,
+            0.017815934494137764,
+            0.008647225797176361,
+            -0.0186708252876997,
+            -0.008595932275056839,
+            0.005915847606956959,
+            -0.011378603056073189,
+            8.963001164374873e-05,
+            0.011925733648240566,
+            -0.0033169782254844904,
+            0.008788282051682472,
+            0.01482381485402584,
+            -0.03400757536292076,
+            -0.003128902055323124,
+            -0.014704130589962006,
+            0.01360986940562725,
+            0.006557016167789698,
+            0.028057532384991646,
+            -0.028279803693294525,
+            -0.021765531972050667,
+            -0.00981415156275034,
+            -0.01892729289829731,
+            -0.03082738071680069,
+            -0.02692907489836216,
+            0.0058175353333354,
+            0.00636466545984149,
+            0.021526163443922997,
+            0.014729776419699192,
+            -0.014396369457244873,
+            0.010096265934407711,
+            -0.0033981928136199713,
+            0.008984907530248165,
+            -0.014404918067157269,
+            -0.0018454965902492404,
+            -0.015550472773611546,
+            0.015131575986742973,
+            0.017662053927779198,
+            -0.002583908848464489,
+            -0.018380163237452507,
+            -0.019730890169739723,
+            0.010429673828184605,
+            -0.007604258134961128,
+            -0.004637785255908966,
+            0.02668970637023449,
+            0.003182332729920745,
+            0.0019096133764833212,
+            0.007527317851781845,
+            0.023406922817230225,
+            0.008664323017001152,
+            0.03546088933944702,
+            -0.023406922817230225,
+            0.001999377040192485,
+            -0.0189443901181221,
+            0.0031866072677075863,
+            -0.023218847811222076,
+            0.005112249869853258,
+            -0.01942313089966774,
+            0.03417855501174927,
+            -0.006133845075964928,
+            -0.007852176204323769,
+            0.030468326061964035,
+            -0.02624516189098358,
+            0.002979296026751399,
+            -0.009754309430718422,
+            0.004650608636438847,
+            0.022141683846712112,
+            -0.0038256384432315826,
+            -0.017285902053117752,
+            -0.03925660625100136,
+            0.003522152081131935,
+            -0.009899641387164593,
+            -0.006091100629419088,
+            -0.011318760924041271,
+            0.01773044466972351,
+            0.005065230652689934,
+            0.005569616798311472,
+            -0.0015633824514225125,
+            0.016892651095986366,
+            0.0013913356233388186,
+            0.0035456616897135973,
+            0.029955390840768814,
+            -0.001200053608044982,
+            0.012558353133499622,
+            0.0033533109817653894,
+            0.005689301528036594,
+            0.023492412641644478,
+            -0.007745314855128527,
+            -0.0015206378884613514,
+            -0.017362842336297035,
+            0.0067237201146781445,
+            -0.0007555101765319705,
+            0.02520219422876835,
+            0.01539659220725298,
+            -0.01771334744989872,
+            -0.040658630430698395,
+            0.02159455418586731,
+            -0.011352956295013428,
+            0.027373619377613068,
+            -0.009497841820120811,
+            -0.006569839548319578,
+            -0.0003144397051073611,
+            -0.03754682466387749,
+            0.003182332729920745,
+            -0.009720114059746265,
+            -0.052798084914684296,
+            -0.018089499324560165,
+            -0.007548689842224121,
+            -0.026296455413103104,
+            0.03342624753713608,
+            -0.0014650699449703097,
+            -0.026775194332003593,
+            0.024706358090043068,
+            -0.0034516234882175922,
+            -0.00698873633518815,
+            0.012233494780957699,
+            -0.016704576089978218,
+            -0.009378157556056976,
+            -0.0032443124800920486,
+            -0.005116524174809456,
+            0.0006069728406146169,
+            -0.012421570718288422,
+            -0.019235054031014442,
+            0.015866782516241074,
+            -0.006980187259614468,
+            0.0008538226829841733,
+            0.015866782516241074,
+            -0.0020496018696576357,
+            0.025065412744879723,
+            -0.014122803695499897,
+            0.005385815165936947,
+            0.007612806744873524,
+            -0.013883434236049652,
+            0.012301885522902012,
+            0.003847010899335146,
+            -0.015430787578225136,
+            0.01580693945288658,
+            0.009215728379786015,
+            0.00848907046020031,
+            0.0041611832566559315,
+            -0.014533151872456074,
+            0.0022889713291078806,
+            -0.011421347968280315,
+            0.006646779831498861,
+            0.013909080997109413,
+            0.005304600577801466,
+            -0.011276016011834145,
+            -0.016653282567858696,
+            0.013857787474989891,
+            0.007501671090722084,
+            0.020004456862807274,
+            -0.03180195763707161,
+            0.007386260665953159,
+            -0.004676255397498608,
+            0.006941717118024826,
+            -0.01227623876184225,
+            0.025287684053182602,
+            0.022620422765612602,
+            0.02111581526696682,
+            0.006356116849929094,
+            0.0028767092153429985,
+            0.010515162721276283,
+            -0.01966249942779541,
+            -0.0033618600573390722,
+            -0.0204148031771183,
+            -0.0006417027907446027,
+            -0.0008324504015035927,
+            0.01743123307824135,
+            -0.012318983674049377,
+            0.013105483725667,
+            0.006522820331156254,
+            0.015447885729372501,
+            -0.02887822687625885,
+            0.0039047158788889647,
+            0.0070400298573076725,
+            0.0013421793701127172,
+            0.0066211330704391,
+            0.0031224903650581837,
+            0.08371095359325409,
+            0.010472417809069157,
+            -0.0070870486088097095,
+            0.003968833014369011,
+            -0.009437999688088894,
+            -0.0077538639307022095,
+            0.015251260250806808,
+            -0.01081437524408102,
+            0.012233494780957699,
+            0.007347790524363518,
+            -0.013148227706551552,
+            0.004582217428833246,
+            0.0073563395999372005,
+            0.008108643814921379,
+            -0.01623438484966755,
+            0.011190527118742466,
+            -0.005253307055681944,
+            0.0038961670361459255,
+            0.025133803486824036,
+            0.01652504876255989,
+            -0.005723497364670038,
+            0.00971156544983387,
+            0.006467252504080534,
+            0.0009863307932391763,
+            -0.012404472567141056,
+            0.006390312220901251,
+            -0.004152634646743536,
+            0.023013673722743988,
+            -0.016208738088607788,
+            0.0344863161444664,
+            -0.0005706399097107351,
+            -0.003855559742078185,
+            -0.010882765986025333,
+            -0.003383232280611992,
+            0.007501671090722084,
+            -0.008322366513311863,
+            0.01114778220653534,
+            -0.013712456449866295,
+            0.0005973553052172065,
+            -0.013045640662312508,
+            -0.008189858868718147,
+            0.004058596212416887,
+            -0.007540141232311726,
+            -0.0073777115903794765,
+            -0.0009211453725583851,
+            0.001756801619194448,
+            -4.190970867057331e-05,
+            -0.0012331806356087327,
+            -0.0012919544242322445,
+            -0.009437999688088894,
+            -0.004513826221227646,
+            -0.013601320795714855,
+            -1.5010688912298065e-05,
+            0.02137228287756443,
+            -0.003682444104924798,
+            -0.006809209007769823,
+            0.01044677197933197,
+            0.003128902055323124,
+            0.02429600991308689,
+            -0.006095374934375286,
+            -0.014558798633515835,
+            -0.0021564632188528776,
+            -0.007005834020674229,
+            -0.007390534970909357,
+            0.00033928497578017414,
+            0.0038790691178292036,
+            0.006180863827466965,
+            0.003287056926637888,
+            0.012438667938113213,
+            0.006018434651196003,
+            -0.008831026963889599,
+            0.011763304471969604,
+            0.003120353212580085,
+            -0.0009190081618726254,
+            -0.01891019567847252,
+            -0.003242175094783306,
+            -0.013550027273595333,
+            -0.01199412439018488,
+            -0.011455543339252472,
+            0.017345745116472244,
+            -0.01723460853099823,
+            0.019765086472034454,
+            -0.006339018698781729,
+            0.00765555165708065,
+            -0.02549285814166069,
+            0.002449263585731387,
+            0.008899418637156487,
+            0.011250369250774384,
+            0.006702347658574581,
+            -0.012857564724981785,
+            -0.002013268880546093,
+            -0.0010862462222576141,
+            0.02963053248822689,
+            -0.008702793158590794,
+            -0.010010777041316032,
+            0.02816011942923069,
+            -0.014413466677069664,
+            -0.01019030436873436,
+            0.0014362173387780786,
+            -0.0013090522261336446,
+            0.02552705444395542,
+            0.011455543339252472,
+            0.017328646034002304,
+            0.0054456572979688644,
+            0.002840376226231456,
+            0.0013379048323258758,
+            0.023936955258250237,
+            0.016071956604719162,
+            0.010908412747085094,
+            -0.01418264675885439,
+            -0.008446325547993183,
+            -0.01942313089966774,
+            0.014216842129826546,
+            0.0033746834378689528,
+            -0.01530255377292633,
+            0.011224722489714622,
+            -0.021953608840703964,
+            -0.019064076244831085,
+            -0.023697586730122566,
+            -0.010386928915977478,
+            0.013438890688121319,
+            -0.011686364188790321,
+            -0.005505499895662069,
+            -0.017089277505874634,
+            -0.005454206373542547,
+            -0.010455320589244366,
+            -0.015148673206567764,
+            -0.01673022285103798,
+            -0.02598869428038597,
+            -0.022688815370202065,
+            -0.0098397983238101,
+            -0.017294451594352722,
+            -0.008300994522869587,
+            0.012797722592949867,
+            -0.020448999479413033,
+            0.017662053927779198,
+            -0.005065230652689934,
+            0.0037893056869506836,
+            0.01188298873603344,
+            0.0216116514056921,
+            -0.01748252660036087,
+            -0.031904544681310654,
+            -0.02402244508266449,
+            -0.02135518379509449,
+            0.022705912590026855,
+            0.0021927962079644203,
+            0.024142129346728325,
+            -0.007125518750399351,
+            0.016157444566488266,
+            -0.018363064154982567,
+            0.019303444772958755,
+            -0.04202645644545555,
+            -0.03222940117120743,
+            -0.007168263662606478,
+            0.00034836819395422935,
+            -0.004362082574516535,
+            -0.0010199921671301126,
+            -0.015661608427762985,
+            -0.010993901640176773,
+            -0.007168263662606478,
+            -0.006210785359144211,
+            -0.009685918688774109,
+            -0.01516577135771513,
+            0.01007916871458292,
+            0.004370631650090218,
+            0.016165994107723236,
+            0.001985484967008233,
+            0.015559021383523941,
+            -0.01516577135771513,
+            0.006945991888642311,
+            -0.0022269918117672205,
+            -0.005753418430685997,
+            0.004646334331482649,
+            0.003804266219958663,
+            0.0044924537651240826,
+            0.0002427089784760028,
+            0.0011017411015927792,
+            -0.00029386888490989804,
+            0.017362842336297035,
+            0.013361950404942036,
+            -0.007146891206502914,
+            0.01773044466972351,
+            6.485151970991865e-05,
+            0.0016787927597761154,
+            -0.010429673828184605,
+            -0.003214391181245446,
+            -0.00911314133554697,
+            -0.008882320486009121,
+            0.015362396836280823,
+            -0.019730890169739723,
+            -0.010626298375427723,
+            0.026843586936593056,
+            -0.0018348104786127806,
+            -0.03443502262234688,
+            0.02742491289973259,
+            7.86015789344674e-06,
+            -0.0024364402052015066,
+            -0.01369535829871893,
+            -0.007078499998897314,
+            -0.0006363596767187119,
+            -0.0005978895933367312,
+            0.017337195575237274,
+            -0.018038205802440643,
+            -0.005249032285064459,
+            0.00630909763276577,
+            -0.0040436359122395515,
+            -0.0021254734601825476,
+            0.006497173570096493,
+            0.014028766192495823,
+            -0.00047820480540394783,
+            -0.03501634672284126,
+            -0.012190749868750572,
+            0.020568683743476868,
+            -0.017106374725699425,
+            -0.014994793571531773,
+            0.011233271099627018,
+            -0.001976936124265194,
+            -0.012601097114384174,
+            -0.003911127801984549,
+            -0.005223385989665985,
+            -0.010993901640176773,
+            0.008442051708698273,
+            0.010224499739706516,
+            0.008339464664459229,
+            0.02210748940706253,
+            -0.014447662979364395,
+            -0.005757692735642195,
+            -0.015251260250806808,
+            0.0031075298320502043,
+            -0.022962380200624466,
+            -0.00015214394079521298,
+            -0.009745760820806026,
+            0.0016766554908826947,
+            0.012575451284646988,
+            0.01749962382018566,
+            0.028724348172545433,
+            0.0089678093791008,
+            0.0072879483923316,
+            -0.010848570615053177,
+            -0.01456734724342823,
+            0.004680529702454805,
+            0.0031417254358530045,
+            -0.00558671448379755,
+            0.006655328441411257,
+            0.019286347553133965,
+            0.0013261501444503665,
+            0.019303444772958755,
+            0.0025133804883807898,
+            0.0037978545296937227,
+            -2.8652215405600145e-05,
+            -0.010455320589244366,
+            -0.006706621963530779,
+            -0.02374888025224209,
+            0.0007453583530150354,
+            0.005779065191745758,
+            0.01889309659600258,
+            0.0005540763959288597,
+            0.0030370012391358614,
+            0.026416141539812088,
+            -0.0029771588742733,
+            -0.014345075935125351,
+            -0.012592548504471779,
+            -0.011942830868065357,
+            -0.00874553807079792,
+            0.034332435578107834,
+            0.01454170048236847,
+            -0.01454170048236847,
+            -0.015234163030982018,
+            -0.011566678993403912,
+            0.03033154271543026,
+            0.00442833686247468,
+            -0.0004928982816636562,
+            0.021902315318584442,
+            0.018294673413038254,
+            0.017345745116472244,
+            0.016362618654966354,
+            0.009420901536941528,
+            0.0027912200894206762,
+            0.020466096699237823,
+            0.0019480835180729628,
+            0.015088831074535847,
+            -0.008172760717570782,
+            0.008642951026558876,
+            -0.005864554084837437,
+            0.012712233699858189,
+            -0.010677591897547245,
+            -0.011797499842941761,
+            0.008433502167463303,
+            -0.014003119431436062,
+            0.002117993077263236,
+            -0.007373437285423279,
+            0.00948074460029602,
+            -0.02017543464899063,
+            -0.007599983364343643,
+            0.012190749868750572,
+            -0.014533151872456074,
+            0.0024043817538768053,
+            -0.022893989458680153,
+            -0.0009761789697222412,
+            0.014695581048727036,
+            -0.0025368898641318083,
+            0.0066125839948654175,
+            0.005321698263287544,
+            0.00012549536768347025,
+            0.010104815475642681,
+            -0.014789619483053684,
+            0.004030812531709671,
+            -0.005377266090363264,
+            -0.004834410268813372,
+            -0.00637748884037137,
+            -0.0011712011182680726,
+            0.0009542724001221359,
+            -0.005556793417781591,
+            0.01991896703839302,
+            0.008091545663774014,
+            0.00435994565486908,
+            -0.005270404741168022,
+            -0.012831917963922024,
+            -0.026159673929214478,
+            0.0145758967846632,
+            -0.015447885729372501,
+            0.01528545655310154,
+            0.0013550026342272758,
+            -0.003092569299042225,
+            -0.014370722696185112,
+            0.010643396526575089,
+            0.0056337337009608746,
+            0.00474037230014801,
+            0.015114477835595608,
+            -0.011481190100312233,
+            0.009583331644535065,
+            0.009147336706519127,
+            -0.009267021901905537,
+            -0.0067493668757379055,
+            0.0005936151137575507,
+            0.006279176566749811,
+            -0.0232701413333416,
+            -0.0016862730262801051,
+            0.008292445912957191,
+            -0.01962830312550068,
+            0.008762635290622711,
+            -0.03730745613574982,
+            0.0020068571902811527,
+            -0.012088162824511528,
+            0.011258917860686779,
+            -0.002776259556412697,
+            -0.033374954015016556,
+            -0.026518728584051132,
+            0.0016745183384045959,
+            0.0040179891511797905,
+            -0.009993678890168667,
+            0.009010554291307926,
+            0.009275570511817932,
+            0.011532483622431755,
+            0.01443056482821703,
+            0.0098397983238101,
+            0.010113364085555077,
+            0.04223163053393364,
+            -0.04151352122426033,
+            -0.007146891206502914,
+            0.0070656766183674335,
+            0.02209039032459259,
+            -0.010267244651913643,
+            -0.011361504904925823,
+            0.008685695938766003,
+            -0.003859834047034383,
+            0.034349530935287476,
+            -0.024877335876226425,
+            0.014994793571531773,
+            -0.0040543219074606895,
+            -0.00010605995339574292,
+            -0.024159228429198265,
+            -0.0020325039513409138,
+            -0.01969669573009014,
+            0.0032357636373490095,
+            -0.002079523168504238,
+            -0.017191864550113678,
+            0.018021108582615852,
+            -0.004595040809363127,
+            0.012789173983037472,
+            0.007813706062734127,
+            -0.01625148393213749,
+            0.004526649136096239,
+            -0.015593216754496098,
+            0.032896216958761215,
+            -0.012105260975658894,
+            0.005911573302000761,
+            0.00873698852956295,
+            -0.006779287941753864,
+            0.016439558938145638,
+            0.009352510794997215,
+            -0.006044081412255764,
+            -0.0051592690870165825,
+            -0.002759161638095975,
+            -0.0023680487647652626,
+            0.008493345230817795,
+            -0.021474869921803474,
+            0.027493303641676903,
+            -0.006894698366522789,
+            -0.014037314802408218,
+            0.006266353186219931,
+            -0.001722605898976326,
+            0.002626653527840972,
+            0.004406964406371117,
+            -0.0018059578724205494,
+            -0.0007939802599139512,
+            -0.004958369303494692,
+            -0.017798837274312973,
+            -0.010130461305379868,
+            0.0010467075044289231,
+            -0.0016114701284095645,
+            0.013105483725667,
+            -0.010540809482336044,
+            -0.016601989045739174,
+            0.005112249869853258,
+            0.00026929075829684734,
+            -0.003975244704633951,
+            -0.016807163134217262,
+            0.007215282414108515,
+            -0.007544415537267923,
+            0.005372991785407066,
+            -0.008168485946953297,
+            0.016448108479380608,
+            0.008313817903399467,
+            -0.0003630616411101073,
+            0.01767915114760399,
+            0.024706358090043068,
+            0.02061997726559639,
+            0.01152393501251936,
+            -0.040214087814092636,
+            -0.0189443901181221,
+            -0.0025091059505939484,
+            0.018807608634233475,
+            -0.0008981701685115695,
+            -0.006420233752578497,
+            -0.0017632133094593883,
+            0.0029835705645382404,
+            -0.003084020223468542,
+            -0.0007474955637007952,
+            0.016918297857046127,
+            -0.020192531868815422,
+            0.0032742335461080074,
+            -0.012720782309770584,
+            0.014404918067157269,
+            0.0006860502762719989,
+            -0.0006737611838616431,
+            5.767176844528876e-05,
+            0.00945509783923626,
+            0.014900755137205124,
+            0.014661385677754879,
+            0.0014159136917442083,
+            0.019064076244831085,
+            -0.003663209266960621,
+            -0.0016510088462382555,
+            -0.018636630848050117,
+            -0.01696104370057583,
+            -0.0034494863357394934,
+            0.01298579853028059,
+            -0.0015601766062900424,
+            0.011575227603316307,
+            -0.014191195368766785,
+            0.01649085246026516,
+            0.025680935010313988,
+            -0.012840467505156994,
+            0.005069505423307419,
+            0.002564673777669668,
+            -0.001211808412335813,
+            0.0064800758846104145,
+            -0.007381986360996962,
+            -0.004030812531709671,
+            -0.012438667938113213,
+            0.01868792437016964,
+            -0.0032208028715103865,
+            0.018277576193213463,
+            0.013635516166687012,
+            0.011968477629125118,
+            -0.00032993461354635656,
+            0.008305269293487072,
+            0.018636630848050117,
+            0.014413466677069664,
+            -0.0005158734857104719,
+            0.0011615835828706622,
+            -0.01284901611506939,
+            0.0023594999220222235,
+            0.016311325132846832,
+            -0.02715134806931019,
+            -0.007685472723096609,
+            0.011840244755148888,
+            -0.016567792743444443,
+            0.0025603994727134705,
+            -0.02597159706056118,
+            -0.011472641490399837,
+            0.012053967453539371,
+            0.010233048349618912,
+            -0.011053744703531265,
+            0.001883966731838882,
+            0.001705508097074926,
+            -0.0008377934573218226,
+            -0.00936105940490961,
+            0.0167815163731575,
+            -0.005868828855454922,
+            -0.011258917860686779,
+            0.011686364188790321,
+            -0.00981415156275034,
+            0.012695135548710823,
+            0.004603589419275522,
+            0.011746206320822239,
+            -0.0024578124284744263,
+            0.026877781376242638,
+            0.01287466287612915,
+            0.007236654870212078,
+            0.0004386660875752568,
+            0.005937220063060522,
+            -0.004847233649343252,
+            0.007407633122056723,
+            0.026570022106170654,
+            -0.00048728802357800305,
+            -0.0116350706666708,
+            -0.0013197384541854262,
+            0.010951157659292221,
+            -0.01988477073609829,
+            0.0023124809376895428,
+            0.005676478147506714,
+            0.028211412951350212,
+            0.014327977783977985,
+            0.0025796345435082912,
+            -0.0204148031771183,
+            -0.00023709874949418008,
+            -0.00041595802758820355,
+            0.04106897860765457,
+            0.0026116929948329926,
+            -0.012532706372439861,
+            0.0021489830687642097,
+            0.013849238865077496,
+            -0.005590988788753748,
+            -0.004834410268813372,
+            -0.014310880564153194,
+            -0.008027428761124611,
+            -0.006018434651196003,
+            -0.0198676735162735,
+            0.001279131043702364,
+            0.005800437182188034,
+            -0.013319206424057484,
+            0.007471750024706125,
+            -0.006403135601431131,
+            -0.03031444549560547,
+            -0.005932945758104324,
+            -0.0034131535794585943,
+            -0.023731783032417297,
+            -0.007125518750399351,
+            0.0089678093791008,
+            0.005915847606956959,
+            -0.014122803695499897,
+            -0.025338977575302124,
+            0.020996129140257835,
+            0.007668375037610531,
+            0.03392208740115166,
+            -0.012669488787651062,
+            -0.007185361348092556,
+            -0.00948074460029602,
+            0.008557462133467197,
+            -0.023150455206632614,
+            -0.008715616539120674,
+            0.009061847813427448,
+            0.01767915114760399,
+            0.012404472567141056,
+            -0.011370054446160793,
+            0.0015163634670898318,
+            0.012122358195483685,
+            -0.00010085045505547896,
+            0.02764718420803547,
+            -0.009104591794312,
+            -0.014994793571531773,
+            0.010754532180726528,
+            -0.003259273013100028,
+            0.01916666328907013,
+            -0.0006203305092640221,
+            -0.0025924579240381718,
+            0.012917407788336277,
+            0.0038341875188052654,
+            -0.012934505008161068,
+            -0.005240483675152063,
+            0.008647225797176361,
+            -0.028553368523716927,
+            0.02792074903845787,
+            0.015909526497125626,
+            -0.02816011942923069,
+            -0.008305269293487072,
+            -0.0025091059505939484,
+            0.017140571027994156,
+            0.020089944824576378,
+            -0.011446994729340076,
+            -0.009728662669658661,
+            0.014960597269237041,
+            0.0034217024222016335,
+            0.0077538639307022095,
+            -0.0026565745938569307,
+            0.01896148920059204,
+            -0.004370631650090218,
+            -0.007800883147865534,
+            -0.003408879041671753,
+            -0.012789173983037472,
+            -0.0003505054337438196,
+            0.015618863515555859,
+            0.012977249920368195,
+            -0.005984238814562559,
+            -0.008771184831857681,
+            0.015208516269922256,
+            -0.022432347759604454,
+            0.0010841090697795153,
+            0.010908412747085094,
+            -0.01745687983930111,
+            -0.007561513222754002,
+            0.01767915114760399,
+            -0.0031395882833749056,
+            0.011258917860686779,
+            -0.007381986360996962,
+            0.007856450974941254,
+            0.0067237201146781445,
+            0.031135139986872673,
+            -0.007476024329662323,
+            0.03525571525096893,
+            -0.009309765882790089,
+            0.01991896703839302,
+            0.037991367280483246,
+            0.01455025002360344,
+            -0.0031118043698370457,
+            -0.008335189893841743,
+            0.009198630228638649,
+            -0.010096265934407711,
+            0.002415067981928587,
+            0.013524380512535572,
+            -0.026809390634298325,
+            -0.01418264675885439,
+            -0.011378603056073189,
+            0.013233717530965805,
+            0.0034580351784825325,
+            -0.013661162927746773,
+            -0.011575227603316307,
+            0.019508618861436844,
+            0.010352733545005322,
+            -0.0038705202750861645,
+            -0.007869274355471134,
+            -0.02157745696604252,
+            0.0020688369404524565,
+            0.012088162824511528,
+            -0.009164434857666492,
+            -0.011532483622431755,
+            0.016670379787683487,
+            -0.01261819526553154,
+            0.0027228286489844322,
+            -0.008300994522869587,
+            -0.027989141643047333,
+            -0.005800437182188034,
+            -0.00491989916190505,
+            -0.0019844165071845055,
+            -0.005565342493355274,
+            0.006471527274698019,
+            0.005227660294622183,
+            -0.01824337989091873,
+            0.0024278913624584675,
+            -0.008788282051682472,
+            0.00861302949488163,
+            0.00436635734513402,
+            0.006249255500733852,
+            -0.0019384659826755524,
+            0.014225390739738941,
+            0.013661162927746773,
+            -0.005377266090363264,
+            -0.020773857831954956,
+            0.012772075831890106,
+            -0.03634997829794884,
+            0.020722564309835434,
+            0.008694244548678398,
+            0.0025411644019186497,
+            -0.03086157515645027,
+            0.01018175482749939,
+            0.01115633174777031,
+            0.0160035640001297,
+            0.011370054446160793,
+            0.0016766554908826947,
+            0.013190972618758678,
+            -0.01114778220653534,
+            0.006766464561223984,
+            -7.192273642431246e-06,
+            -0.01482381485402584,
+            0.020688369870185852,
+            0.011814597994089127,
+            0.02913469448685646,
+            0.002951512113213539,
+            -0.0011209761723876,
+            0.001963044051080942,
+            0.00899345614016056,
+            -0.007685472723096609,
+            0.0030904319137334824,
+            0.004954094998538494,
+            0.0058175353333354,
+            0.013190972618758678,
+            0.007745314855128527,
+            -0.00860875565558672,
+            0.007839352823793888,
+            0.0026736725121736526,
+            -0.014020216651260853,
+            -0.000775279535446316,
+            -0.006997285410761833,
+            0.007552964612841606,
+            -0.01505463570356369,
+            0.004560844972729683,
+            0.04342847689986229,
+            0.012413021177053452,
+            -0.0034323884174227715,
+            -0.017354292795062065,
+            -0.011079391464591026,
+            -0.01273788046091795,
+            -0.01272933091968298,
+            -0.0053644427098333836,
+            -0.02303077094256878,
+            -0.003434525802731514,
+            0.0065912120044231415,
+            0.02453538030385971,
+            0.01092551089823246,
+            -0.017191864550113678,
+            -0.005813260562717915,
+            -0.02380017377436161,
+            -0.010566456243395805,
+            -0.016191640868782997,
+            -0.016319874674081802,
+            -0.008061625063419342,
+            0.018978586420416832,
+            -0.008839575573801994,
+            0.01408860832452774,
+            0.002968610031530261,
+            -0.0002393695613136515,
+            -0.0020431901793926954,
+            -0.000852754048537463,
+            -0.0051549943163990974,
+            0.0015911663649603724,
+            0.003152411663904786,
+            -0.0005375128821469843,
+            0.020722564309835434,
+            -0.007800883147865534,
+            0.003866245737299323,
+            -0.0026929075829684734,
+            0.01542223896831274,
+            0.007108421064913273,
+            0.0028681603725999594,
+            -0.0238172709941864,
+            0.02304787002503872,
+            -0.024381499737501144,
+            -0.014156999997794628,
+            0.02284269593656063,
+            -0.008108643814921379,
+            0.004804489202797413,
+            0.024107934907078743,
+            0.00029493749025277793,
+            -0.0009697672794573009,
+            0.015123026445508003,
+            0.004045772831887007,
+            0.011258917860686779,
+            0.015242711640894413,
+            0.007082774303853512,
+            -0.0052875024266541,
+            0.019098270684480667,
+            0.0073306928388774395,
+            -0.006403135601431131,
+            -0.01626003161072731,
+            0.0014244626509025693,
+            0.004238123539835215,
+            -0.010044972412288189,
+            -0.004137673880904913,
+            0.008630127646028996,
+            0.004453983623534441,
+            -0.007843627594411373,
+            -0.0012972975382581353,
+            -0.024706358090043068,
+            -0.005821809638291597,
+            -0.01530255377292633,
+            -0.006039807107299566,
+            -0.023406922817230225,
+            0.005676478147506714,
+            -0.009053298272192478,
+            0.01672167330980301,
+            0.00016403225890826434,
+            0.024347303435206413,
+            0.0020154062658548355,
+            -0.0007266575703397393,
+            -0.0051592690870165825,
+            0.009318315424025059,
+            -0.0007266575703397393,
+            0.027236836031079292,
+            -0.00011026761785615236,
+            -0.005911573302000761,
+            0.01032708678394556,
+            -0.01769625023007393,
+            0.020876444876194,
+            0.006356116849929094,
+            0.01454170048236847,
+            -0.015610314905643463,
+            -0.0073563395999372005,
+            0.035289913415908813,
+            0.024843141436576843,
+            0.0035029170103371143,
+            0.019816379994153976,
+            -0.010455320589244366,
+            0.016311325132846832,
+            0.003013491863384843,
+            -0.002389421220868826,
+            0.011010999791324139,
+            -0.005420010536909103,
+            -0.014627190306782722,
+            -0.006997285410761833,
+            0.03911982476711273,
+            -0.014832363463938236,
+            -6.972707342356443e-05,
+            -0.005060956347733736,
+            -0.010104815475642681,
+            -0.030929965898394585,
+            0.011609423905611038,
+            0.011951380409300327,
+            -0.024706358090043068,
+            -0.011566678993403912,
+            0.0032699592411518097,
+            -0.00026448199059814215,
+            -0.0030348640866577625,
+            -0.005706399213522673,
+            0.006509996950626373,
+            -0.0017375665483996272,
+            0.009275570511817932,
+            0.005569616798311472,
+            -0.0009617527248337865,
+            0.00013798479631077498,
+            0.005539695732295513,
+            0.011780401691794395,
+            0.006206510588526726,
+            0.0005498019745573401,
+            -0.007381986360996962,
+            0.00558671448379755,
+            0.010745983570814133,
+            -0.015088831074535847,
+            0.020739663392305374,
+            0.013823592104017735,
+            -0.016576342284679413,
+            0.0038149524480104446,
+            -0.019132466986775398,
+            -0.0329304113984108,
+            0.004787391051650047,
+            -0.0009029789362102747,
+            0.021748434752225876,
+            -0.018311770632863045,
+            -0.00995948351919651,
+            0.00442833686247468,
+            0.003908990416675806,
+            -0.011421347968280315,
+            0.019371837377548218,
+            0.007215282414108515,
+            -0.0017664191545918584,
+            0.01745687983930111,
+            0.016593439504504204,
+            -0.006403135601431131,
+            -0.00907894503325224,
+            0.018824705854058266,
+            -0.00437276903539896,
+            0.02744201011955738,
+            0.026621313765645027,
+            0.009207178838551044,
+            -0.018602434545755386,
+            0.0018262615194544196,
+            0.0080231549218297,
+            -0.02036350965499878,
+            -0.005590988788753748,
+            0.015670157968997955,
+            0.005402912851423025,
+            -0.0003841667785309255,
+            -0.0050737797282636166,
+            -0.022175880149006844,
+            0.007407633122056723,
+            0.011840244755148888,
+            -0.005189190153032541,
+            -0.028809836134314537,
+            -0.0011199075961485505,
+            -0.01431942917406559,
+            -0.0007694021333009005,
+            0.0010648739989846945,
+            0.01236172765493393,
+            0.009036201052367687,
+            0.028074629604816437,
+            -0.009865445084869862,
+            -0.00885667372494936,
+            0.010173206217586994,
+            0.009797054342925549,
+            0.012583999894559383,
+            0.012327532283961773,
+            0.013986021280288696,
+            0.007292222697287798,
+            -0.015960820019245148,
+            -0.004966918379068375,
+            0.007527317851781845,
+            -0.00899345614016056,
+            0.003985930699855089,
+            -0.01358422264456749,
+            0.020278021693229675,
+            0.015259809792041779,
+            0.0008260387112386525,
+            0.004804489202797413,
+            -0.004753195680677891,
+            -0.0012684449320659041,
+            -0.06483495235443115,
+            -0.013532929122447968,
+            0.00861302949488163,
+            0.021252596750855446,
+            -0.016396814957261086,
+            0.00046244275290519,
+            0.0014287370722740889,
+            0.008788282051682472,
+            -0.011481190100312233,
+            0.01020740158855915,
+            0.014524603262543678,
+            -0.005223385989665985,
+            -0.0035349754616618156,
+            0.005099426489323378,
+            -0.0016403226181864738,
+            -0.00013371034583542496,
+            0.004958369303494692,
+            0.014900755137205124,
+            0.021406477317214012,
+            -0.004385592415928841,
+            0.018585337325930595,
+            0.004112027119845152,
+            -0.015080282464623451,
+            -0.0027014564257115126,
+            0.013575674034655094,
+            -0.02328723855316639,
+            -0.004607864189893007,
+            0.007480298634618521,
+            -0.006980187259614468,
+            0.010908412747085094,
+            -0.004069282673299313,
+            -0.004406964406371117,
+            0.008677146397531033,
+            0.01420829351991415,
+            0.002276147948578,
+            -0.026775194332003593,
+            0.0034665842540562153,
+            -0.011233271099627018,
+            0.0035093287006020546,
+            0.01020740158855915,
+            -0.008219779469072819,
+            1.5344630810432136e-05,
+            -0.015328200533986092,
+            -0.002739926567301154,
+            0.024860238656401634,
+            -0.004289417061954737,
+            0.0067707388661801815,
+            0.017302999272942543,
+            -0.007018657401204109,
+            0.002389421220868826,
+            -0.02862176112830639,
+            0.007796608377248049,
+            -0.0369655005633831,
+            -0.0004579011583700776,
+            0.004984016064554453,
+            0.004097066354006529,
+            -0.010104815475642681,
+            -0.030211858451366425,
+            0.008412130177021027,
+            -0.00474037230014801,
+            -0.006753641180694103,
+            -0.022723009809851646,
+            -0.007937666028738022,
+            0.008429228328168392,
+            -0.006941717118024826,
+            0.010147559456527233,
+            -0.017662053927779198,
+            0.017815934494137764,
+            0.015439336188137531,
+            0.013302108272910118,
+            0.003968833014369011,
+            0.0032379007898271084,
+            0.005129347555339336,
+            0.007946214638650417,
+            0.004701902158558369,
+            0.004697627853602171,
+            -0.01019030436873436,
+            -0.011464091949164867,
+            0.01575564593076706,
+            0.004255221225321293,
+            0.00860875565558672,
+            -0.0018604571232572198,
+            0.002479184651747346,
+            -0.0073050460778176785,
+            -0.00045629823580384254,
+            -0.014396369457244873,
+            0.0021596690639853477,
+            0.01199412439018488,
+            0.015499179251492023,
+            0.0014650699449703097,
+            -0.00625352980569005,
+            -0.007634179200977087,
+            -0.00667670089751482,
+            0.020209630951285362,
+            -0.017149118706583977,
+            -0.006698073353618383,
+            -0.012635293416678905,
+            0.021526163443922997,
+            0.01599501632153988,
+            0.0035563476849347353,
+            -0.006826307158917189,
+            -0.004092792049050331,
+            0.0097030159085989,
+            0.008570285513997078,
+            0.006766464561223984,
+            -0.023663390427827835,
+            -0.005377266090363264,
+            -0.019081173464655876,
+            -0.01539659220725298,
+            -0.01406296156346798,
+            0.021218402311205864,
+            -0.004924173932522535,
+            -0.007471750024706125,
+            0.008454875089228153,
+            0.024655064567923546,
+            0.007916293106973171,
+            0.007715393789112568,
+            0.00503958435729146,
+            -0.01726025529205799,
+            0.012071064673364162,
+            -0.010139010846614838,
+            -0.03665773943066597,
+            0.0065912120044231415,
+            -0.00971156544983387,
+            0.02598869428038597,
+            -0.017183315008878708,
+            -0.0011167017510160804,
+            -0.002635202370584011,
+            -0.030878672376275063,
+            -0.005638008005917072,
+            0.02961343340575695,
+            -0.010737434960901737,
+            0.011669266037642956,
+            -0.010062070563435555,
+            -0.009950934909284115,
+            0.01019030436873436,
+            0.013062738813459873,
+            -0.011720559559762478,
+            -0.002940826117992401,
+            0.014721227809786797,
+            0.02453538030385971,
+            0.0035093287006020546,
+            -0.004548021592199802,
+            0.009429451078176498,
+            0.0123702771961689,
+            -0.006804934702813625,
+            0.019559912383556366,
+            -0.007561513222754002,
+            -0.0015078145079314709,
+            -0.00656556524336338,
+            0.0012235631002113223,
+            0.006193687207996845,
+            -0.019577009603381157,
+            0.017320098355412483,
+            -0.009916738606989384,
+            0.008873771876096725,
+            0.0028702975250780582,
+            0.01726025529205799,
+            0.011267467401921749,
+            0.01896148920059204,
+            -0.02891242317855358,
+            -0.001095329411327839,
+            -0.0276813805103302,
+            -0.015772745013237,
+            -0.010703238658607006,
+            -0.00873698852956295,
+            -0.002485596342012286,
+            -0.007796608377248049,
+            -0.023868564516305923,
+            -0.005625184625387192,
+            0.0007939802599139512,
+            0.0036781697999686003,
+            -0.017362842336297035,
+            0.01007061917334795,
+            1.5286190318875015e-05,
+            -0.016593439504504204,
+            -0.020825151354074478,
+            -0.002087003318592906,
+            -0.022021999582648277,
+            -0.012464314699172974,
+            -0.005005388520658016,
+            -0.0022248546592891216,
+            -0.0012139456812292337,
+            -0.0005172092351131141,
+            0.027544597163796425,
+            -0.019457325339317322,
+            0.004381317645311356,
+            0.02744201011955738,
+            -0.0019352601375430822,
+            0.005035309586673975,
+            -0.006668151821941137,
+            0.012532706372439861,
+            0.0032913314644247293,
+            -0.003973107319325209,
+            -0.022688815370202065,
+            0.02521929331123829,
+            0.0005311011918820441,
+            0.013669711537659168,
+            0.010139010846614838,
+            -0.008065898902714252,
+            -0.005783339496701956,
+            0.00485150795429945,
+            -0.03253716230392456,
+            0.0028531996067613363,
+            -0.013011445291340351,
+            -0.01865372806787491,
+            -0.005539695732295513,
+            -0.004586491733789444,
+            -0.006561290472745895,
+            -0.0071939099580049515,
+            0.017285902053117752,
+            -0.004377043340355158,
+            0.02376597747206688,
+            -0.007146891206502914,
+            -0.00453947251662612,
+            0.010908412747085094,
+            -0.04301812872290611,
+            -0.007240929175168276,
+            -0.006129570305347443,
+            -0.01843145675957203,
+            0.019799282774329185,
+            0.012173651717603207,
+            -0.001380649395287037,
+            -0.020243825390934944,
+            0.019816379994153976,
+            -0.0030028056353330612,
+            0.02552705444395542,
+            0.010634847916662693,
+            0.010173206217586994,
+            0.004364219959825277,
+            -0.010840021073818207,
+            -0.010113364085555077,
+            0.018311770632863045,
+            0.006304823327809572,
+            -0.011609423905611038,
+            0.01873921789228916,
+            0.004445434547960758,
+            -0.00186686881352216,
+            -0.003041275776922703,
+            0.018756315112113953,
+            0.024603771045804024,
+            0.013002896681427956,
+            0.002274010796099901,
+            -0.016174543648958206,
+            -0.006475801579654217,
+            0.0037038165610283613,
+            0.036418367177248,
+            0.012036869302392006,
+            0.012994347140192986,
+            -0.008202682249248028,
+            0.027373619377613068,
+            -0.009181532077491283,
+            0.009044749662280083,
+            0.00863440241664648,
+            0.017174765467643738,
+            0.0035328383091837168,
+            0.020329315215349197,
+            0.010857119224965572,
+            -0.00764272827655077,
+            0.02405664138495922,
+            0.007625630125403404,
+            0.003767933463677764,
+            0.006625407375395298,
+            -0.01844855397939682,
+            0.012626743875443935,
+            -0.011609423905611038,
+            -0.005950043443590403,
+            0.004505277145653963,
+            0.002058150712400675,
+            0.02374888025224209,
+            0.008027428761124611,
+            -0.03135741129517555,
+            -0.0017867228016257286,
+            -0.023612096905708313,
+            0.004340710584074259,
+            -0.0167387705296278,
+            0.012447217479348183,
+            0.004092792049050331,
+            -0.01774754375219345,
+            -0.018790511414408684,
+            -0.010728885419666767,
+            -0.01942313089966774,
+            -0.004924173932522535,
+            -0.012173651717603207,
+            -0.010840021073818207,
+            0.006954540498554707,
+            -0.0025069687981158495,
+            0.0024813220370560884,
+            0.006963089574128389,
+            0.019046977162361145,
+            0.020260922610759735,
+            0.0035819944459944963,
+            0.018824705854058266,
+            -0.0009019103017635643,
+            -0.055431149899959564,
+            0.0036589347291737795,
+            0.009309765882790089,
+            -0.0002863885893020779,
+            0.026108380407094955,
+            -0.004041498526930809,
+            0.00042557556298561394,
+            0.007223831489682198,
+            0.006937442813068628,
+            0.013276461511850357,
+            -0.004210339393466711,
+            -0.025886109098792076,
+            -0.019337641075253487,
+            0.0040393611416220665,
+            -0.006155217066407204,
+            0.008100095205008984,
+            -0.006685249973088503,
+            0.004761744290590286,
+            0.002353088231757283,
+            -0.009745760820806026,
+            -0.006595486309379339,
+            0.009617527015507221,
+            0.0014469034504145384,
+            -0.010386928915977478,
+            -0.013173874467611313,
+            -0.0018690060824155807,
+            0.01824337989091873,
+            -0.009019102901220322,
+            -0.024364402517676353,
+            0.009660271927714348,
+            -0.010010777041316032,
+            -0.0014821677468717098,
+            0.025869010016322136,
+            0.02429600991308689,
+            -0.003970969934016466,
+            0.019371837377548218,
+            -0.0021190617699176073,
+            -0.016405364498496056,
+            -0.03029734641313553,
+            -0.006873325910419226,
+            -0.028997913002967834,
+            0.027253935113549232,
+            -0.006424508057534695,
+            -0.0013176011852920055,
+            0.0035285637713968754,
+            -0.00413553649559617,
+            0.003434525802731514,
+            -0.009788504801690578,
+            -0.007672649342566729,
+            -0.02716844528913498,
+            0.007527317851781845,
+            -0.0069075217470526695,
+            -0.008172760717570782,
+            0.00031550831045024097,
+            0.039701152592897415,
+            -0.004556570667773485,
+            0.00850189384073019,
+            0.006232157349586487,
+            0.011865891516208649,
+            0.024193422868847847,
+            -0.008463423699140549,
+            -0.012096711434423923,
+            -0.015046087093651295,
+            -0.0020154062658548355,
+            0.00014052275219000876,
+            -0.003975244704633951,
+            0.022876890376210213,
+            -0.013019993901252747,
+            -0.015191418118774891,
+            -0.00034489520476199687,
+            -0.023389825597405434,
+            -0.011737657710909843,
+            0.009968032129108906,
+            0.012908858247101307,
+            0.007535866927355528,
+            -0.014003119431436062,
+            -0.014268135651946068,
+            -0.022945282980799675,
+            0.01800401136279106,
+            0.0976627841591835,
+            0.0012545529752969742,
+            -0.009070396423339844,
+            0.0026758096646517515,
+            0.010771630331873894,
+            -0.0008372591692022979,
+            -0.017559466883540154,
+            0.05919267237186432,
+            -0.013567124493420124,
+            0.005856005474925041,
+            -0.009010554291307926,
+            0.01007916871458292,
+            0.005031035281717777,
+            -0.00013257493264973164,
+            0.005928670987486839,
+            -0.008796831592917442,
+            0.006129570305347443,
+            -0.03645256534218788,
+            -0.00186686881352216,
+            -0.011344407685101032,
+            -0.006381763610988855,
+            -0.016824260354042053,
+            0.009275570511817932,
+            -0.02911759726703167,
+            -0.00922427698969841,
+            0.0020880720112472773,
+            -0.0059543177485466,
+            0.012336080893874168,
+            -0.01628567837178707,
+            -0.0017995461821556091,
+            0.005368717480450869,
+            0.005120798945426941,
+            -0.0036867186427116394,
+            -0.0074632009491324425,
+            -0.013310656882822514,
+            0.005065230652689934,
+            0.015576119534671307,
+            0.005548244342207909,
+            -0.008228329010307789,
+            0.00959188025444746,
+            0.016388265416026115,
+            0.0009510665549896657,
+            0.013661162927746773,
+            0.009412352927029133,
+            0.0007950488943606615,
+            0.0007875685696490109,
+            0.019611205905675888,
+            -0.016345521435141563,
+            0.02137228287756443,
+            -0.015883879736065865,
+            -0.02843368425965309,
+            -0.006236432120203972,
+            0.006411684677004814,
+            -0.014413466677069664,
+            0.0010910549899563193,
+            0.006582662928849459,
+            0.04100058600306511,
+            0.00863440241664648,
+            -0.0034281141124665737,
+            -0.0009259541402570903,
+            -0.013669711537659168,
+            0.023954054340720177,
+            0.01091696135699749,
+            0.030502520501613617,
+            0.007604258134961128,
+            0.00373373762704432,
+            -0.007001559715718031,
+            -0.009557684883475304,
+            -0.003487956477329135,
+            0.00474037230014801,
+            0.0025069687981158495,
+            0.004415513481944799,
+            0.008523265831172466,
+            -0.014498956501483917,
+            -0.02036350965499878,
+            0.017610760405659676,
+            -0.00474037230014801,
+            -0.023663390427827835,
+            0.012609646655619144,
+            -0.0316309779882431,
+            0.0047831167466938496,
+            0.0010344184702262282,
+            -0.005877377465367317,
+            -0.013960374519228935,
+            0.006300548557192087,
+            0.01151538547128439,
+            -0.009378157556056976,
+            -0.015482081100344658,
+            -0.008454875089228153,
+            -0.010592103004455566,
+            0.0001227570464834571,
+            0.0037657960783690214,
+            -0.011583777144551277,
+            -0.00394318625330925,
+            -0.01796981506049633,
+            0.007142616901546717,
+            -0.003013491863384843,
+            0.004654882941395044,
+            -0.011122135445475578,
+            0.011558130383491516,
+            0.020226728171110153,
+            -0.009788504801690578,
+            -0.007069950923323631,
+            0.013105483725667,
+            0.003400330198928714,
+            0.016089053824543953,
+            0.03368271514773369,
+            -0.0033704089000821114,
+            0.0033511738292872906,
+            0.003849148051813245,
+            0.0058731031604111195,
+            0.0008874839986674488,
+            0.015909526497125626,
+            0.03621319308876991,
+            0.01942313089966774,
+            -0.0250141192227602,
+            0.02186811901628971,
+            -0.019713792949914932,
+            0.021936509758234024,
+            -0.02231266349554062,
+            -0.0030797459185123444,
+            0.007005834020674229,
+            -0.007394809741526842,
+            0.004146222956478596,
+            0.00624070642516017,
+            0.017627857625484467,
+            0.022363955155014992,
+            0.000617124664131552,
+            -0.0003163097717333585,
+            0.004860057029873133,
+            0.021953608840703964,
+            -0.0025411644019186497,
+            0.007125518750399351,
+            -0.013421793468296528,
+            0.004411239176988602,
+            -0.007202459033578634,
+            0.0021767669823020697,
+            0.008395032025873661,
+            -0.014952048659324646,
+            0.013344853185117245,
+            0.012891761027276516,
+            -0.017302999272942543,
+            0.0043342988938093185,
+            0.0025945950765162706,
+            -0.018568238243460655,
+            -0.018978586420416832,
+            0.023372728377580643,
+            -0.013635516166687012,
+            -0.012660940177738667,
+            -0.009891091845929623,
+            -0.01152393501251936,
+            -0.0201412383466959,
+            -0.005056682042777538,
+            -0.0017471840837970376,
+            0.005753418430685997,
+            -0.0024449890479445457,
+            0.006467252504080534,
+            0.01573854871094227,
+            0.005612361244857311,
+            -0.006437331438064575,
+            -0.005561067722737789,
+            -0.0015163634670898318,
+            0.020107043907046318,
+            0.005013937596231699,
+            0.0026458885986357927,
+            -0.04031667485833168,
+            0.0013143953401595354,
+            -0.010540809482336044,
+            0.006428782362490892,
+            0.0031417254358530045,
+            -0.004971192684024572,
+            -0.01942313089966774,
+            -0.014755423180758953,
+            0.01872211880981922,
+            -0.0103014400228858,
+            0.024637967348098755,
+            4.564986011246219e-05,
+            -0.012053967453539371,
+            -0.0018999958410859108,
+            0.003960283938795328,
+            0.0016029211692512035,
+            0.004415513481944799,
+            -0.00800178200006485,
+            0.016174543648958206,
+            -0.011694912798702717,
+            0.010908412747085094,
+            -0.00401157746091485,
+            -0.024706358090043068,
+            0.010891314595937729,
+            -0.01744833029806614,
+            0.0004752661334350705,
+            -0.00666387751698494,
+            -0.030040880665183067,
+            0.018551141023635864,
+            -0.023731783032417297,
+            -0.0033233899157494307,
+            0.009643173776566982,
+            0.018841803073883057,
+            0.0017461155075579882,
+            -0.010976804420351982,
+            0.00630909763276577,
+            0.017362842336297035,
+            0.013481635600328445,
+            -0.004062870983034372,
+            0.012438667938113213,
+            -0.004971192684024572,
+            0.0025603994727134705,
+            0.005172092467546463,
+            0.008399306796491146,
+            -0.006804934702813625,
+            0.007805157452821732
+          ],
+          "keyphrases": [
+            "DeepSeekMoE",
+            "multilingual corpus",
+            "HuggingFace Tokenizer",
+            "MoE architectures",
+            "evaluation benchmarks"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "3c9c713a-3f56-4404-9b66-6c4c2b7c3822",
+      "type": "child",
+      "source": {
+        "id": "1e687808-8db8-4320-805d-0282a8d679bc",
+        "properties": {
+          "page_content": "Deepseekmoe: Towards Ultimate Expert Specialization In\n\nMixture-Of-Experts Language Models\n\nDamai Dai∗1,2, Chengqi Deng1, Chenggang Zhao∗1,3, R.X. Xu1, Huazuo Gao1, Deli Chen1, Jiashi Li1, Wangding Zeng1, Xingkai Yu∗1,4, Y. Wu1, Zhenda Xie1, Y.K. Li1, Panpan Huang1, Fuli Luo1, Chong Ruan1, Zhifang Sui2, Wenfeng Liang1 1DeepSeek-AI 2National Key Laboratory for Multimedia Information Processing, Peking University 3Institute for Interdisciplinary Information Sciences, Tsinghua University 4National Key Laboratory for Novel Software Technology, Nanjing University {daidamai, szf}@pku.edu.cn, {wenfeng.liang}@deepseek.com https://github.com/deepseek-ai/DeepSeek-MoE\n\nAbstract\n\nIn the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top- out of experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into ones and activating from them, allowing for a more flexible combination of activated experts; (2) isolating experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5× expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations.\n\nFurther, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\n1. Introduction\n\nRecent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023a). It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017) has emerged as a popular solution. It can\n\nenable parameter scaling, while concurrently keeping computational costs at a modest level.\n\nRecent applications of MoE architectures in Transformers (Vaswani et al., 2017) have yielded successful attempts at scaling language models to a substantial size (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022), accompanied with remarkable performance. These achievements underscore the considerable potential and promise of MoE language models.\n\nDespite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts.\n\nThis architecture manifests two potential issues: (1) Knowledge Hybridity: existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. (2) Knowledge Redundancy: tokens assigned to different experts may require common knowledge. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters.\n\nThese issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models.\n\nIn response to the aforementioned issues, we introduce DeepSeekMoE, an innovative MoE architecture specifically designed towards ultimate expert specialization. Our architecture involves two principal strategies: (1) Fine-Grained Expert Segmentation: while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. (2) Shared Expert Isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.\n\nThrough compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized.\n\nStarting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B (Lepikhin et al., 2021) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5× expert parameters and computation. Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE.\n\nThese studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization.\n\nLeveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B (DeepSeek-AI, 2024), a dense model trained on the same 2T corpus. We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B (Touvron et al., 2023b), which has approximately 2.5 times the activated parameters. Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard1. Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. The experimental results still validate its substantial advantages over the GShard architecture consistently. In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nOur contributions are summarized as follows: - Architectural Innovation. We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation.\n\nEmpirical Validation. We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. Experimental results validate the high 1https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models\n\nScalability. We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B.\n\nAlignment for MoE. We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B.\n\nPublic Release. In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization.\n\n2. Preliminaries: Mixture-Of-Experts For Transformers\n\nWe first introduce a generic MoE architecture commonly used in Transformer language models. A standard Transformer language model is constructed by stacking layers of standard Transformer blocks, where each block can be represented as follows:\n\n$$\\mathbf{u}{1:T}^{l}=\\text{Self-Att}\\left(\\mathbf{h}{1:T}^{l-1}\\right)+\\mathbf{h}{1:T}^{l-1},\\tag{1}$$ $$\\mathbf{h}{t}^{l}=\\text{FFN}\\left(\\mathbf{u}{t}^{l}\\right)+\\mathbf{u}{t}^{l},\\tag{2}$$\n\nwhere denotes the sequence length, Self-Att(·) denotes the self-attention module, FFN(·) denotes the Feed-Forward Network (FFN), u 1: ∈ R× are the hidden states of all tokens after the -th attention module, and h\n\n∈ Ris the output hidden state of the -th token after the -th Transformer block. For brevity, we omit the layer normalization in the above formulations.\n\nA typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022). An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. Then, each token will be assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts. If the -th FFN is substituted with an MoE layer, the computation for its output hidden state h\n\nis expressed as: where denotes the total number of experts, FFN(·) is the -th expert FFN, , denotes the gate value for the -th expert, , denotes the token-to-expert affinity, Topk(·, ) denotes the set comprising highest affinity scores among those calculated for the -th token and all experts, and e\n\nis the centroid of the -th expert in the -th layer. Note that ,is sparse, indicating that only out of gate values are nonzero. This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only experts. Also, in the above formulations, we omit the layer normalization operation for brevity.\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{N}\\left(g_{i,t}\\operatorname{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\operatorname{Topk}({s_{j,t}|1\\leqslant j\\leqslant N},K),\\ 0,&\\text{otherwise},\\end{cases}$$ $$s_{i,t}=\\operatorname{Softmax}{i}\\left(\\mathbf{u}{t}^{l}\\,\\mathbf{e}_{i}^{l}\\right),$$\n\n$$(3)$$\n\n(4) $\\binom{4}{5}$ .\n\n3. Deepseekmoe Architecture\n\nOn top of the generic MoE architecture outlined in Section 2, we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. As illustrated in Figure 2, our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. Both of these strategies are designed to elevate the level of expert specialization.\n\n3.1. Fine-Grained Expert Segmentation\n\nIn scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts.\n\nIn pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. The finer expert segmentation enables a more flexible and adaptable combination of activated experts. To be specific, on top of a typical MoE architecture shown in Figure 2(a), we segment each expert FFN into smaller experts by reducing the FFN intermediate hidden dimension to 1 times its original size. Since each expert becomes smaller, in response, we also increase the number of activated experts to times to keep the same computation cost, as illustrated in Figure 2(b). With the fine-grained expert segmentation, the output of an MoE layer can be expressed as:\n\n$$\\mathbf{h}{t}^{i}=\\sum{i=1}^{\\text{mN}}\\left(g_{i,t}\\text{FFN}{i}\\left(\\mathbf{u}{t}^{i}\\right)\\right)+\\mathbf{u}{t}^{i},\\tag{6}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\text{Tokr}(s_{i,t}|1\\leqslant j\\leqslant m\\text{N}),m\\text{K}),\\ 0,&\\text{otherwise},\\end{cases}$$ (7) $$s_{i,t}=\\text{Softmax}{i}\\left(\\mathbf{u}{t}^{T}\\mathbf{e}_{t}^{i}\\right),\\tag{8}$$ where the total number of expert parameters is equal to $N$ times the number of parameters in a\n\n$$(6)$$\n\n(7) (8) $\\frac{1}{2}$\n\nstandard FFN, and denotes the total number of fine-grained experts. With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to .\n\nFrom a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. As an illustrative example, we consider the case where = 16. A typical top-2 routing strategy can yield 16 2 = 120 possible combinations. By contrast, if each expert is split into 4 smaller experts, the fine-grained routing strategy can yield 64 8 = 4, 426, 165, 368 potential combinations. The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition.\n\n3.2. Shared Expert Isolation\n\nWith a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. This alleviation of redundancy will contribute to a more parameterefficient model with more specialized experts.\n\nTowards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate experts to serve as shared experts. Regardless of the router module, each token will be deterministically assigned to these shared experts. In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by , as depicted in Figure 2(c). With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows:\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{K_{s}}\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)+\\sum_{i=K_{s}+1}^{mN}\\left(g_{i,t}\\;\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},\\tag{9}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\mathrm{Topk}{s_{j,t}|K_{s}+1\\leqslant j\\leqslant mN},mK-K_{s}},\\ 0,&\\text{otherwise},\\end{cases}$$ (10) $$s_{i,t}=\\mathrm{Softmax}{i}\\left(\\mathbf{u}{t}^{l,T}e_{i}^{l}\\right).\\tag{11}$$ $\\mathbf{h}\\mathbf{M}\\mathbf{E}$ the number of closed-compact is $K_{s}$ the total number of routed currents.\n\nFinally, in DeepSeekMoE, the number of shared expert is , the total number of routed experts is − , and the number of nonzero gates is − .\n\nIt is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022). The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint.\n\n3.3. Load Balance Consideration\n\nAutomatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. Firstly, there is a risk of routing collapse (Shazeer et al., 2017), i.e., the model always selects only a few experts, preventing other experts from sufficient training.\n\nSecondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.\n\nExpert-Level Balance Loss. In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. The computation of the balance loss is as follows:\n\n$$\\begin{array}{l}{{{\\mathcal L}{\\mathrm{ExpBal}}=\\alpha{1}\\sum_{i=1}^{N^{\\prime}}f_{i}P_{i},}}\\ {{f_{i}=\\frac{N^{\\prime}}{K^{T}}\\sum_{t=1}^{T}\\mathds{1}\\,(\\mathrm{Token}\\;t\\;\\mathrm{s}\\mathrm{lects}\\;\\mathrm{Expert}\\;i),}}\\ {{P_{i}=\\frac{1}{T}\\sum_{t=1}^{T}s_{i,t},}}\\end{array}$$\n\n(12) $$\\begin{array}{l}\\small\\text{(13)}\\end{array}$$ = (14) $$\\begin{array}{l}\\small\\text{(14)}\\end{array}$$ .\n\n7 where 1 is a hyper-parameter called expert-level balance factor, ′is equal to ( − ) and ′ is equal to ( − ) for brevity. 1(·) denotes the indicator function.\n\nDevice-Level Balance Loss. In addition to the expert-level balance loss, we introduce a devicelevel balance loss. When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. Instead, our primary objective is to ensure balanced computation across the devices. If we partition all routed experts into groups {E1, E2, ..., E}, and deploy each group on a single device, the device-level balance loss is computed as follows:\n\n$$\\mathcal{L}{\\text{DevBal}}=\\alpha{2}\\sum_{i=1}^{D}f_{i}^{\\prime}P_{i}^{\\prime},$$ $$f_{i}^{\\prime}=\\frac{1}{|\\mathcal{E}{i}|}\\sum{j\\in\\mathcal{E}{i}}f{j},$$ $$P_{i}^{\\prime}=\\sum_{j\\in\\mathcal{E}{i}}P{j},$$\n\n(15) $$\\begin{array}{l}\\small\\text{(16)}\\end{array}$$ = (17) .\n\nwhere 2 is a hyper-parameter called device-level balance factor. In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices.\n\n4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n6. Alignment For Deepseekmoe 16B\n\nPrevious research indicates that MoE models typically do not emerge significant gains from fine-tuning (Artetxe et al., 2022; Fedus et al., 2021). However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B.\n\n6.1. Experimental Setup\n\nTraining Data. For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios.\n\nHyper-Parameters. During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).\n\nWe employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10−5 without incorporating any learning rate scheduling strategy.\n\nEvaluation Benchmarks. For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3, with the following adjustments: (1) We exclude Pile (Gao et al., 2020) since chat models are seldom employed for pure language modeling. (2) We exclude\n\nMetric # Shot LLaMA2 DeepSeek DeepSeekMoE SFT 7B Chat 7B Chat 16B # Total Params N/A 6.7B 6.9B 16.4B # Activated Params N/A 6.7B 6.9B 2.8B FLOPs per 4K Tokens N/A 187.9T 183.5T 74.4T HellaSwag (Acc.) 0-shot 67.9 71.0 72.2 PIQA (Acc.) 0-shot 76.9 78.4 79.7 ARC-easy (Acc.) 0-shot 69.7 70.2 69.9 ARC-challenge (Acc.) 0-shot 50.8 50.2 50.0 BBH (EM) 3-shot 39.3 43.1 42.2 RACE-middle (Acc.) 5-shot 63.9 66.1 64.8 RACE-high (Acc.) 5-shot 49.6 50.8 50.6 DROP (EM) 1-shot 40.0 41.7 33.8 GSM8K (EM) 0-shot 63.4 62.6 62.2 MATH (EM) 4-shot 13.5 14.7 15.2 HumanEval (Pass@1) 0-shot 35.4 45.1 45.7 MBPP (Pass@1) 3-shot 27.8 39.0 46.2 TriviaQA (EM) 5-shot 60.1 59.5 63.3 NaturalQuestions (EM) 0-shot 35.2 32.7 35.1 MMLU (Acc.) 0-shot 50.0 49.7 47.2 WinoGrande (Acc.) 0-shot 65.1 68.4 69.0 CLUEWSC (EM) 5-shot 48.4 66.2 68.2 CEval (Acc.) 0-shot 35.1 44.7 40.0 CMMLU (Acc.) 0-shot 36.9 51.2 49.3\n\nCHID (Zheng et al., 2019) due to the observed instability of results, hindering the derivation of solid conclusions. (3) We additionally include BBH (Suzgun et al., 2022) to provide a more comprehensive assessment of the reasoning ability of the chat models.\n\nTable 5 | Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations.\n\n6.2. Evaluations\n\nBaselines. In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. Correspondingly, we construct three chat models, including LLaMA2 SFT 7B3, DeepSeek Chat 7B, and DeepSeekMoE Chat 16B.\n\nSubsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks.\n\nResults. The evaluation results are presented in Table 5. Our key observations include: (1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intensive tasks (TriviaQA, NaturalQuestions). (2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. In addition, it also surpasses DeepSeek Chat 7B. (3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1). However, it is worth noting that, after supervised finetuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. (4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations.\n\n7. Deepseekmoe 145B Ongoing\n\nEncouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available.\n\n7.1. Experimental Setup\n\nTraining Data and Tokenization. For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study.\n\nModel Settings. For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps.\n\nSubsequently, the learning rate keeps constant during the remaining training process. The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. Also, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse.\n\nEvaluation Benchmarks. We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3).\n\n7.2. Evaluations\n\nBaselines. Apart from DeepSeekMoE 145B, we consider three additional models for comparison. DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters.\n\nResults. From the evaluation results presented in Table 6, we have the following observations: (1) Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. (2) On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. (3) At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5.\n\n8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\nAppendices A. Overview Of Hyper-Parameters\n\nWe present the overview of hyper-parameters for DeepSeekMoE across various sizes in Table 7.\n\n# Params # Layers Hidden # Attn # Shared # Routed Relative Sequence Batch Size Learning Size Heads Experts Experts Expert Size Length (Sequence) Rate 2.0B 9 1280 10 1 63 (7 activated) 0.25 2048 2048 1.08e-3 16.4B 28 2048 16 2 64 (6 activated) 0.25 4096 4608 4.2e-4 144.6B 62 4096 32 4 128 (12 activated) 0.125 4096 4608 3.0e-4\n\nTable 7 | Overview of hyper-parameters for DeepSeekMoE across various sizes. The relative expert size is in comparison to a standard FFN.\n\nB. Comparing Deepseekmoe With Larger Models\n\nComparisons among DeepSeekMoE, GShard×1.2, and GShard×1.5 are shown in Table 8. Comparisons among DeepSeekMoE, Dense×4, and Dense×16 are shown in Table 9.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 2.3B 2.8B 1.9B # Activated Expert Params N/A 0.28B 0.35B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.824 1.808 1.808 HellaSwag (Acc.) 0-shot 53.7 54.4 54.8 PIQA (Acc.) 0-shot 71.8 71.1 72.3 ARC-easy (Acc.) 0-shot 46.8 47.3 49.4 ARC-challenge (Acc.) 0-shot 31.7 34.1 34.3 RACE-middle (Acc.) 5-shot 43.7 46.4 44.0 RACE-high (Acc.) 5-shot 31.9 32.4 31.7 HumanEval (Pass@1) 0-shot 3.7 3.0 4.9 MBPP (Pass@1) 3-shot 2.4 2.6 2.2 TriviaQA (EM) 5-shot 15.2 15.7 16.6 NaturalQuestions (EM) 5-shot 4.5 4.7 5.7\n\nTable 8 | Comparison between DeepSeekMoE and larger GShard models.\n\nAt a larger scale of 13B total parameters, we also compare DeepSeekMoE with GShard×1.2 and GShard×1.5, and show results in Table 10. At a larger scale, DeepSeekMoE even outperforms GShard×1.5 distinctly.\n\nMetric # Shot Dense×4 Dense×16 DeepSeekMoE Relative Expert Size N/A 1 1 0.25 # Experts N/A 4 + 0 16 + 0 1 + 63 # Activated Experts N/A 4 + 0 16 + 0 1 + 7 # Total Expert Params N/A 0.47B 1.89B 1.89B # Activated Expert Params N/A 0.47B 1.89B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.908 1.806 1.808 HellaSwag (Acc.) 0-shot 47.6 55.1 54.8 PIQA (Acc.) 0-shot 70.0 71.9 72.3 ARC-easy (Acc.) 0-shot 43.9 51.9 49.4 ARC-challenge (Acc.) 0-shot 30.5 33.8 34.3 RACE-middle (Acc.) 5-shot 42.4 46.3 44.0 RACE-high (Acc.) 5-shot 30.7 33.0 31.7 HumanEval (Pass@1) 0-shot 1.8 4.3 4.9 MBPP (Pass@1) 3-shot 0.2 2.2 2.2 TriviaQA (EM) 5-shot 9.9 16.5 16.6 NaturalQuestions (EM) 5-shot 3.0 6.3 5.7\n\nTable 9 | Comparison between DeepSeekMoE and larger dense baselines.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 15.9B 19.8B 13.3B # Activated Expert Params N/A 2.37B 2.82B 2.05B # Training Tokens N/A 100B 100B 100B HellaSwag (Acc.) 0-shot 66.6 67.7 69.1 PIQA (Acc.) 0-shot 75.6 76.0 75.7 ARC-easy (Acc.) 0-shot 56.8 56.8 58.8 ARC-challenge (Acc.) 0-shot 39.9 37.6 38.5 RACE-middle (Acc.) 5-shot 51.6 50.6 52.4 RACE-high (Acc.) 5-shot 37.4 36.3 38.5 HumanEval (Pass@1) 0-shot 6.1 6.1 9.8 MBPP (Pass@1) 3-shot 7.0 11.6 10.6 TriviaQA (EM) 5-shot 36.5 36.7 38.2 NaturalQuestions (EM) 5-shot 12.6 12.1 13.7\n\nTable 10 | Comparison between DeepSeekMoE and larger GShard models at a larger scale.\n\nC. Training Benchmark Curves Of Deepseekmoe 16B\n\nWe present the benchmark curves during training of DeepSeekMoE 16B and DeepSeek 7B (Dense) in Figure 7 for reference.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/DeepSeekMoE_2.md"
+          },
+          "summary": "DeepSeekMoE is an innovative Mixture-of-Experts (MoE) architecture designed to enhance expert specialization in language models. It employs fine-grained expert segmentation and shared expert isolation to improve performance and reduce computational costs. Starting with a model of 2B parameters, DeepSeekMoE demonstrates comparable performance to larger models like GShard 2.9B and LLaMA2 7B, using significantly fewer computations. The architecture scales up to 16B and 145B parameters, maintaining advantages over traditional MoE architectures. DeepSeekMoE 16B achieves similar results to dense models with only 40% of the computations, and its chat model version shows adaptability in bilingual scenarios. The architecture's public release aims to contribute to advancements in large-scale language models.",
+          "headlines": {
+            "1. Introduction": [],
+            "2. Preliminaries: Mixture-Of-Experts For Transformers": [],
+            "3. Deepseekmoe Architecture": [
+              "3.1. Fine-Grained Expert Segmentation",
+              "3.2. Shared Expert Isolation",
+              "3.3. Load Balance Consideration"
+            ],
+            "4. Validation Experiments": [
+              "4.1. Experimental Setup",
+              "4.1.1. Training Data And Tokenization",
+              "4.1.2. Infrastructures",
+              "4.1.3. Hyper-Parameters",
+              "4.1.4. Evaluation Benchmarks",
+              "4.2. Evaluations",
+              "4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models",
+              "4.4. Ablation Studies",
+              "4.5. Analysis On Expert Specialization"
+            ],
+            "5. Scaling Up To Deepseekmoe 16B": [
+              "5.1. Experimental Setup",
+              "5.1.1. Training Data And Tokenization",
+              "5.1.2. Hyper-Parameters",
+              "5.1.3. Evaluation Benchmarks",
+              "5.2. Evaluations",
+              "5.2.1. Internal Comparison With Deepseek 7B",
+              "5.2.2. Comparison With Open Source Models"
+            ],
+            "6. Alignment For Deepseekmoe 16B": [
+              "6.1. Experimental Setup",
+              "6.2. Evaluations"
+            ],
+            "7. Deepseekmoe 145B Ongoing": [
+              "7.1. Experimental Setup",
+              "7.2. Evaluations"
+            ],
+            "8. Related Work": [],
+            "9. Conclusion": [],
+            "Appendices": [
+              "A. Overview Of Hyper-Parameters",
+              "B. Comparing Deepseekmoe With Larger Models",
+              "C. Training Benchmark Curves Of Deepseekmoe 16B"
+            ]
+          },
+          "summary_embedding": [
+            -0.012465757317841053,
+            -0.022808987647294998,
+            -0.018792515620589256,
+            -0.02308654971420765,
+            0.01797616109251976,
+            0.007200260180979967,
+            0.006490030791610479,
+            0.042613785713911057,
+            -0.014049487188458443,
+            0.03187054023146629,
+            -0.021486490964889526,
+            -0.023592690005898476,
+            0.009045224636793137,
+            -0.014441338367760181,
+            0.005865518003702164,
+            -0.007363531738519669,
+            -0.008743173442780972,
+            -0.005163452122360468,
+            0.018841497600078583,
+            -0.03100520372390747,
+            0.011110605672001839,
+            -0.046075135469436646,
+            -0.020017050206661224,
+            -0.021845687180757523,
+            0.016425084322690964,
+            0.037389107048511505,
+            0.022547753527760506,
+            -0.0028062239289283752,
+            -0.06844329088926315,
+            0.034678805619478226,
+            0.035168617963790894,
+            0.02052319049835205,
+            0.0005438972148112953,
+            -0.00031302153365686536,
+            -0.024833550676703453,
+            -0.028556134551763535,
+            0.03347059711813927,
+            -0.004379750229418278,
+            -0.01453930139541626,
+            0.015976088121533394,
+            0.0414055772125721,
+            0.012106560170650482,
+            -0.02754385396838188,
+            0.0014245413476601243,
+            0.015151568688452244,
+            0.0055961208418011665,
+            -0.01678428053855896,
+            -0.0022960014175623655,
+            -0.002424577483907342,
+            -0.021225256845355034,
+            -0.01400050614029169,
+            0.018825169652700424,
+            0.001296985661610961,
+            -0.008865626528859138,
+            -0.0542386956512928,
+            0.007330877240747213,
+            0.03840138763189316,
+            0.05623060464859009,
+            0.044148534536361694,
+            -0.020180322229862213,
+            0.03369917720556259,
+            -0.030025575309991837,
+            -0.02501314878463745,
+            0.011028970591723919,
+            0.003983817528933287,
+            -0.018531281501054764,
+            0.01784554310142994,
+            -0.005636938381940126,
+            -0.0006556359585374594,
+            0.01751900091767311,
+            -0.03941367194056511,
+            0.020751770585775375,
+            -0.01887415163218975,
+            0.031266435980796814,
+            0.02173139713704586,
+            0.0073798587545752525,
+            0.01872720755636692,
+            0.01678428053855896,
+            0.0036858476232737303,
+            -0.04107903689146042,
+            0.009118696674704552,
+            0.016523046419024467,
+            0.014196431264281273,
+            0.016269976273179054,
+            0.06553706526756287,
+            -0.05391215533018112,
+            -0.004485876299440861,
+            -0.05169166624546051,
+            0.0012561678886413574,
+            -0.004971608519554138,
+            -0.03670336678624153,
+            0.01048201136291027,
+            0.028425518423318863,
+            0.021372200921177864,
+            -0.01063711941242218,
+            0.024392718449234962,
+            -0.022743679583072662,
+            0.0032001156359910965,
+            -0.04062187671661377,
+            0.0019653772469609976,
+            0.004518530797213316,
+            0.016531210392713547,
+            0.0070778070949018,
+            0.004273623693734407,
+            -0.016702644526958466,
+            0.006045116577297449,
+            -0.01905374974012375,
+            0.03291547670960426,
+            -0.008465612307190895,
+            0.08653374016284943,
+            0.008751336485147476,
+            0.004738946910947561,
+            0.004281787667423487,
+            0.02398454025387764,
+            0.012131051160395145,
+            -0.00810233410447836,
+            -0.04718538001179695,
+            -0.015690363943576813,
+            0.011061624623835087,
+            -0.0383034273982048,
+            0.018335357308387756,
+            -0.008759500458836555,
+            0.01282495353370905,
+            -0.04107903689146042,
+            -0.06344719231128693,
+            0.01498013362288475,
+            0.001784758409485221,
+            -0.004065453074872494,
+            -0.0002903166168835014,
+            -0.018923133611679077,
+            0.003930754493921995,
+            -0.004200151655822992,
+            -0.010645282454788685,
+            -0.021502818912267685,
+            0.03159298002719879,
+            0.03657275065779686,
+            0.06527583301067352,
+            0.036801330745220184,
+            -0.0030164355412125587,
+            0.005775718949735165,
+            0.019690508022904396,
+            -0.020784424617886543,
+            0.003481758525595069,
+            -0.01214737817645073,
+            -0.021992631256580353,
+            -0.03820546343922615,
+            0.010971825569868088,
+            0.07379858940839767,
+            -0.029029620811343193,
+            0.013437220826745033,
+            -0.017616963014006615,
+            0.04205866530537605,
+            -0.004567512311041355,
+            0.026727497577667236,
+            -0.016669990494847298,
+            0.004059330560266972,
+            -0.025388672947883606,
+            0.01620466820895672,
+            -0.024964168667793274,
+            0.012571883387863636,
+            -0.000885746325366199,
+            -0.04222193360328674,
+            0.00011116473615402356,
+            0.026890767738223076,
+            -0.010971825569868088,
+            0.02261306345462799,
+            0.0004949158756062388,
+            -0.006302268709987402,
+            -0.05097327008843422,
+            0.038727931678295135,
+            -0.003912386484444141,
+            0.018645571544766426,
+            -0.0018959869630634785,
+            0.0163107942789793,
+            -0.0016408757073804736,
+            0.013216804713010788,
+            0.009543201886117458,
+            0.007114543113857508,
+            -0.033503253012895584,
+            0.02697240374982357,
+            -0.02504580281674862,
+            0.028049994260072708,
+            0.03435226157307625,
+            -0.0030184765346348286,
+            0.03255628049373627,
+            0.019249675795435905,
+            0.03840138763189316,
+            0.0327032245695591,
+            0.030335791409015656,
+            -0.00824927818030119,
+            0.013878053054213524,
+            0.008200296200811863,
+            0.03608293831348419,
+            0.013429056853055954,
+            0.019723162055015564,
+            -0.009730963967740536,
+            -0.014041324146091938,
+            0.048785436898469925,
+            0.007318632211536169,
+            -0.0179598331451416,
+            -0.005441013257950544,
+            -0.03510330989956856,
+            -0.002832755446434021,
+            -0.014425011351704597,
+            0.0323440283536911,
+            0.02158445306122303,
+            -0.01647406443953514,
+            0.035135965794324875,
+            0.024980494752526283,
+            -0.03751972317695618,
+            -0.005232842173427343,
+            -0.02026195637881756,
+            -0.023298801854252815,
+            -0.04215662553906441,
+            0.011624909937381744,
+            0.012833117507398129,
+            -0.013976015150547028,
+            -0.007469657808542252,
+            0.020457882434129715,
+            -0.023625344038009644,
+            -0.0018265966791659594,
+            -0.01356783788651228,
+            0.006547175347805023,
+            -0.020898714661598206,
+            -0.03667071461677551,
+            -0.01048201136291027,
+            0.025862159207463264,
+            0.012637191452085972,
+            0.04362606629729271,
+            -0.013518855907022953,
+            -0.005367541220039129,
+            0.033193036913871765,
+            0.012759645469486713,
+            0.010767736472189426,
+            0.03699725493788719,
+            -0.03027048148214817,
+            0.024621298536658287,
+            -0.020294612273573875,
+            -0.0032878739293664694,
+            0.01707816869020462,
+            -0.0030062312725931406,
+            0.01962519995868206,
+            -0.04535674303770065,
+            -0.024196794256567955,
+            0.04395261034369469,
+            -0.0007107399869710207,
+            0.016302630305290222,
+            -0.005959399044513702,
+            -0.012106560170650482,
+            0.0035899258218705654,
+            0.0027715289033949375,
+            -0.01438419334590435,
+            -0.008620720356702805,
+            0.007473739795386791,
+            -0.01962519995868206,
+            0.03660540655255318,
+            0.03794422745704651,
+            -0.004132802598178387,
+            -0.014131123200058937,
+            -0.03324201703071594,
+            0.010016689077019691,
+            -0.017878197133541107,
+            -0.023951886221766472,
+            0.04078514873981476,
+            -0.032115448266267776,
+            -0.00029312283731997013,
+            0.039478980004787445,
+            0.018808843567967415,
+            -0.014898497611284256,
+            0.00868602842092514,
+            -0.01709449477493763,
+            -0.026302991434931755,
+            -0.017159804701805115,
+            0.03356856107711792,
+            0.024800896644592285,
+            -0.010743245482444763,
+            0.010245268233120441,
+            0.005330805201083422,
+            0.02893165871500969,
+            0.024376392364501953,
+            0.00648594880476594,
+            -0.013812744058668613,
+            0.015029114671051502,
+            0.027641816064715385,
+            -0.0028021421749144793,
+            -7.66609373386018e-05,
+            -0.004010349046438932,
+            -0.007796200457960367,
+            -0.006453294772654772,
+            -0.013069860637187958,
+            -0.005089980084449053,
+            0.0009184005321003497,
+            -0.032295044511556625,
+            0.015967924147844315,
+            -0.005890008993446827,
+            0.010359558276832104,
+            -0.011535110883414745,
+            0.01288209855556488,
+            0.01722511276602745,
+            -0.013494365848600864,
+            -0.0716434046626091,
+            0.014612773433327675,
+            0.02458864450454712,
+            -0.006706364918500185,
+            -0.017943505197763443,
+            0.008481939323246479,
+            0.00016148542636074126,
+            0.038597315549850464,
+            -0.014180104248225689,
+            -0.031103165820240974,
+            0.02680913358926773,
+            -0.017437364906072617,
+            -0.015241367742419243,
+            -0.010661610402166843,
+            -0.00580837344750762,
+            0.04649963974952698,
+            0.01889047957956791,
+            0.011420820839703083,
+            -0.04042595252394676,
+            -0.007449248805642128,
+            0.015078096650540829,
+            0.014506647363305092,
+            -0.04705476388335228,
+            0.0052083516493439674,
+            0.015486273914575577,
+            0.019723162055015564,
+            0.024800896644592285,
+            0.04274440184235573,
+            -0.0021531390957534313,
+            0.01661284640431404,
+            -0.015053605660796165,
+            -0.05136512219905853,
+            -0.03340528905391693,
+            0.029943939298391342,
+            -0.017731253057718277,
+            -0.02906227484345436,
+            -0.01693122461438179,
+            -0.006151242647320032,
+            0.0017939424142241478,
+            0.007126788143068552,
+            -0.01916803978383541,
+            -0.03464614972472191,
+            -0.03970756009221077,
+            0.02277633361518383,
+            0.007983962073922157,
+            0.004269542172551155,
+            -0.042777057737112045,
+            0.0013459670590236783,
+            0.009184005670249462,
+            0.012073906138539314,
+            0.0009954441338777542,
+            -0.019984396174550056,
+            0.015494437888264656,
+            0.0032001156359910965,
+            -0.031070511788129807,
+            -0.035593122243881226,
+            -0.03595232218503952,
+            0.0023919232189655304,
+            0.003781769424676895,
+            0.06279410421848297,
+            0.012988224625587463,
+            -0.03840138763189316,
+            -0.04522612690925598,
+            0.029176564887166023,
+            -0.02052319049835205,
+            -0.016678154468536377,
+            0.0059430720284581184,
+            0.012898425571620464,
+            -0.009877908043563366,
+            0.013192313723266125,
+            0.023560035973787308,
+            0.041601505130529404,
+            -0.005987971555441618,
+            0.012914752587676048,
+            0.023037567734718323,
+            0.012955570593476295,
+            0.0362788625061512,
+            -0.00936360377818346,
+            0.0035409443080425262,
+            0.039936136454343796,
+            -0.012269831262528896,
+            0.03175625205039978,
+            -0.02189466916024685,
+            0.0020398697815835476,
+            0.02249877341091633,
+            0.027184655889868736,
+            -0.0006479826406575739,
+            0.02594379521906376,
+            -0.031543996185064316,
+            0.018368011340498924,
+            -0.014498483389616013,
+            -0.04888340085744858,
+            -0.01557607389986515,
+            0.013102514669299126,
+            0.030384771525859833,
+            0.02756018005311489,
+            -0.0041348435916006565,
+            -0.019723162055015564,
+            -0.01155143789947033,
+            -0.010596301406621933,
+            0.0018419033149257302,
+            0.03285016864538193,
+            0.010506502352654934,
+            0.010890189558267593,
+            -0.011935125105082989,
+            -0.004104230087250471,
+            0.03025415539741516,
+            -0.021192602813243866,
+            0.01650671847164631,
+            0.0041511706076562405,
+            -0.002029665280133486,
+            -0.04320156201720238,
+            -0.035266581922769547,
+            0.007845181971788406,
+            0.011747363954782486,
+            0.014874007552862167,
+            0.0368339866399765,
+            0.025094784796237946,
+            -0.01662100851535797,
+            0.030629679560661316,
+            -0.01905374974012375,
+            0.027984686195850372,
+            0.039054471999406815,
+            0.011624909937381744,
+            -0.018351683393120766,
+            0.012237177230417728,
+            0.002424577483907342,
+            -0.025127438828349113,
+            0.01680060662329197,
+            -0.032735876739025116,
+            0.01738838478922844,
+            -0.02354370802640915,
+            0.025143766775727272,
+            -0.02579685114324093,
+            -0.009290131740272045,
+            -0.004428731743246317,
+            -0.006130834110081196,
+            0.04888340085744858,
+            -0.026727497577667236,
+            -0.0142454132437706,
+            -0.0024694770108908415,
+            -0.0024531499948352575,
+            -0.008718682453036308,
+            0.010000362060964108,
+            -0.045781247317790985,
+            -0.01844964735209942,
+            -0.04016471654176712,
+            0.007355368230491877,
+            0.005412440747022629,
+            -0.030907239764928818,
+            -0.05694899708032608,
+            0.025421326979994774,
+            -0.06331657618284225,
+            -0.019886434078216553,
+            0.051626358181238174,
+            -0.018400665372610092,
+            -0.008212542161345482,
+            0.009641164913773537,
+            0.006375740747898817,
+            -0.029290854930877686,
+            0.0041348435916006565,
+            0.02504580281674862,
+            0.04418119043111801,
+            -0.0045797573402523994,
+            -0.02865409664809704,
+            0.0028674507047981024,
+            -0.027788760140538216,
+            -0.013461710885167122,
+            0.005261414684355259,
+            0.016449574381113052,
+            0.024539662525057793,
+            0.007628847379237413,
+            0.03585435822606087,
+            0.007294141221791506,
+            0.006318595726042986,
+            0.044279150664806366,
+            -0.025666235014796257,
+            -0.027380581945180893,
+            0.02519274689257145,
+            0.04045860469341278,
+            0.023151857778429985,
+            -0.018253721296787262,
+            -0.0142454132437706,
+            0.0053063142113387585,
+            0.0009296254720538855,
+            -0.022139575332403183,
+            0.002808264922350645,
+            -0.013902543112635612,
+            0.005143043119460344,
+            0.03892385587096214,
+            -0.001205145614221692,
+            0.014196431264281273,
+            0.007302304729819298,
+            0.006845145486295223,
+            0.004334850702434778,
+            -0.011731036007404327,
+            -0.023592690005898476,
+            0.00032220553839579225,
+            0.034972693771123886,
+            0.015404638834297657,
+            -0.030515389516949654,
+            -0.00995954405516386,
+            0.034711457788944244,
+            -0.00795130804181099,
+            -0.009118696674704552,
+            -0.009771781973540783,
+            0.003385836724191904,
+            0.02638462744653225,
+            -0.01606588624417782,
+            0.006649219896644354,
+            0.03716052696108818,
+            -0.019870106130838394,
+            0.005543057806789875,
+            -0.025405000895261765,
+            0.0018061877926811576,
+            0.02442537248134613,
+            -0.006216551177203655,
+            0.028098974376916885,
+            5.6315813708351925e-05,
+            0.03025415539741516,
+            -0.004677720367908478,
+            0.008432958275079727,
+            -0.020327266305685043,
+            0.03987082839012146,
+            -0.017633290961384773,
+            0.03402572125196457,
+            0.02067013457417488,
+            -0.01258004643023014,
+            0.04783846437931061,
+            -0.036801330745220184,
+            -0.020784424617886543,
+            0.00584919098764658,
+            0.012743317522108555,
+            0.0028919412288814783,
+            -0.0191517136991024,
+            -0.018400665372610092,
+            -0.010996315628290176,
+            0.02380494214594364,
+            -0.022547753527760506,
+            -0.016865916550159454,
+            -0.007441085297614336,
+            -0.01890680566430092,
+            -0.009673818945884705,
+            -0.007988044060766697,
+            -0.01709449477493763,
+            -0.016547536477446556,
+            0.044573038816452026,
+            -0.06334923207759857,
+            0.016996532678604126,
+            0.017159804701805115,
+            0.011477965861558914,
+            -0.034678805619478226,
+            -0.019510909914970398,
+            -0.017633290961384773,
+            -0.002347023691982031,
+            -0.030743969604372978,
+            0.001702102366834879,
+            0.01725776679813862,
+            0.01138000376522541,
+            -0.05022222548723221,
+            0.012335140258073807,
+            0.02772345207631588,
+            -0.012065742164850235,
+            0.0014949520118534565,
+            -0.023135529831051826,
+            -0.017176130786538124,
+            0.0031776658724993467,
+            0.022123249247670174,
+            0.005975726060569286,
+            -0.008188051171600819,
+            -0.016865916550159454,
+            -0.005241005681455135,
+            -0.006045116577297449,
+            -0.002359268954023719,
+            0.005094061605632305,
+            0.011780017986893654,
+            0.019690508022904396,
+            -0.0024633544962853193,
+            -0.004436895251274109,
+            -0.006396149750798941,
+            0.027364253997802734,
+            -0.001450052484869957,
+            -0.008416631259024143,
+            0.050907962024211884,
+            0.03556047007441521,
+            0.006094098091125488,
+            -0.01108611561357975,
+            -0.03990348428487778,
+            0.010661610402166843,
+            -0.022711025550961494,
+            0.0401320643723011,
+            0.01905374974012375,
+            -0.008298259228467941,
+            -0.028866350650787354,
+            0.010294250212609768,
+            -0.02290695160627365,
+            -0.009975871071219444,
+            0.009469729848206043,
+            0.00726556871086359,
+            -0.03263791650533676,
+            -0.013543346896767616,
+            0.03510330989956856,
+            0.011437147855758667,
+            -0.006841063499450684,
+            -0.036050282418727875,
+            0.008588065393269062,
+            0.01646590232849121,
+            0.011869817040860653,
+            -0.019233347848057747,
+            0.004449140280485153,
+            -0.012041252106428146,
+            0.017470018938183784,
+            0.008849299512803555,
+            -0.017796561121940613,
+            0.018237393349409103,
+            0.004710374400019646,
+            -0.008490103296935558,
+            -0.003912386484444141,
+            -0.0163107942789793,
+            -0.02006603218615055,
+            0.03840138763189316,
+            0.024490682408213615,
+            0.026335645467042923,
+            0.005457340274006128,
+            0.02008235827088356,
+            0.03227871656417847,
+            0.011282040737569332,
+            -0.03134807199239731,
+            0.027935704216361046,
+            0.016996532678604126,
+            -0.04604247957468033,
+            0.021551799029111862,
+            -0.02218855731189251,
+            -0.028670424595475197,
+            0.009649328887462616,
+            -0.0017367975087836385,
+            0.011192241683602333,
+            0.0034837995190173388,
+            -0.003642988856881857,
+            -0.03307874873280525,
+            -0.006236960180103779,
+            0.015771998092532158,
+            0.015502601861953735,
+            -0.0048695639707148075,
+            -0.003296037670224905,
+            -0.011886144056916237,
+            0.00487364549189806,
+            0.009796272963285446,
+            0.039054471999406815,
+            0.001381682581268251,
+            -0.004751192405819893,
+            -0.0018765984568744898,
+            0.031854212284088135,
+            0.0271193478256464,
+            -0.017274094745516777,
+            -0.026286665350198746,
+            0.00580837344750762,
+            -0.014253576286137104,
+            0.026156047359108925,
+            -0.00487364549189806,
+            0.011877980083227158,
+            0.008718682453036308,
+            -0.0007908449042588472,
+            -0.001373519073240459,
+            0.025584599003195763,
+            -0.02710302174091339,
+            -0.021764053031802177,
+            0.031233781948685646,
+            -0.025731543079018593,
+            0.028735732659697533,
+            0.013061696663498878,
+            -0.01557607389986515,
+            -0.015225040726363659,
+            -0.010302413254976273,
+            0.009943217039108276,
+            -0.012277995236217976,
+            0.023853924125432968,
+            0.003608293831348419,
+            0.006392067763954401,
+            0.0003584313380997628,
+            0.02770712412893772,
+            0.024833550676703453,
+            0.007820690982043743,
+            -0.018074123188853264,
+            -0.0016816934803500772,
+            -0.02320083975791931,
+            -0.01680060662329197,
+            0.023772288113832474,
+            -0.018090449273586273,
+            -0.03268689662218094,
+            -0.028556134551763535,
+            0.012114724144339561,
+            0.007392104249447584,
+            0.010816717520356178,
+            0.03461349755525589,
+            -0.030384771525859833,
+            -0.0344502255320549,
+            0.01766594499349594,
+            0.026270337402820587,
+            -0.005551221314817667,
+            -0.021535472944378853,
+            0.03085825778543949,
+            -0.005828781984746456,
+            0.017600636929273605,
+            0.0404912605881691,
+            -0.0016225076979026198,
+            0.010228941217064857,
+            -0.042026009410619736,
+            -0.014057651162147522,
+            -0.0066614653915166855,
+            0.006261450704187155,
+            0.034417569637298584,
+            0.01048201136291027,
+            0.01535565685480833,
+            0.03448288142681122,
+            -0.021192602813243866,
+            -0.0002903166168835014,
+            -0.0022449791431427,
+            -0.02818061038851738,
+            0.03347059711813927,
+            0.025715215131640434,
+            0.011918798089027405,
+            0.019706835970282555,
+            -0.03794422745704651,
+            -0.013037205673754215,
+            0.027070365846157074,
+            0.015053605660796165,
+            0.01273515447974205,
+            0.016694480553269386,
+            0.011249386705458164,
+            0.00868602842092514,
+            -0.021633435040712357,
+            -0.05169166624546051,
+            0.01214737817645073,
+            0.01993541419506073,
+            -0.016449574381113052,
+            0.01057181041687727,
+            0.024980494752526283,
+            0.007620683871209621,
+            -0.007045152597129345,
+            0.012808626517653465,
+            -0.02682545967400074,
+            0.002259265398606658,
+            0.0019010890973731875,
+            -0.014482156373560429,
+            0.038270771503448486,
+            -0.016367938369512558,
+            0.011363675817847252,
+            -0.018204739317297935,
+            0.023870250210165977,
+            0.00011531029304023832,
+            0.0007964573451317847,
+            0.014963806606829166,
+            -0.011347348801791668,
+            -0.04235255345702171,
+            0.0021613026037812233,
+            0.03342161700129509,
+            -0.015200549736618996,
+            0.026449935510754585,
+            -0.0020674217958003283,
+            0.032458316534757614,
+            0.042777057737112045,
+            0.017731253057718277,
+            0.0003497575526125729,
+            0.025682561099529266,
+            0.0036144163459539413,
+            0.0073880222626030445,
+            0.012400448322296143,
+            0.009322785772383213,
+            0.007245160173624754,
+            0.030809277668595314,
+            -0.010261595249176025,
+            0.02173139713704586,
+            -0.04760988429188728,
+            -0.014629100449383259,
+            0.012024925090372562,
+            -0.0013316808035597205,
+            -0.012449429370462894,
+            -0.033029764890670776,
+            0.012563719414174557,
+            -0.03503800183534622,
+            0.022074267268180847,
+            -0.0004959363141097128,
+            0.04333217814564705,
+            -0.02187834307551384,
+            -0.005518566817045212,
+            0.016694480553269386,
+            0.0016510800924152136,
+            0.023282473906874657,
+            -0.0012153501156717539,
+            -0.016416920349001884,
+            -0.0067798369564116,
+            -0.009257477708160877,
+            0.004138925112783909,
+            0.04375668615102768,
+            -0.003722583642229438,
+            -0.0001682458823779598,
+            0.01409030519425869,
+            -0.011959616094827652,
+            -0.03250729665160179,
+            0.045193471014499664,
+            -0.0030837850645184517,
+            0.02484987862408161,
+            0.04979771748185158,
+            -0.06410028040409088,
+            -0.019543563947081566,
+            0.013061696663498878,
+            -0.010441194288432598,
+            0.011363675817847252,
+            0.005547139327973127,
+            0.016114868223667145,
+            0.017894525080919266,
+            -0.005485912784934044,
+            0.01619650423526764,
+            0.005106307100504637,
+            0.028098974376916885,
+            -0.02336410991847515,
+            -0.03296445682644844,
+            -0.02636830136179924,
+            -0.017747581005096436,
+            0.0014480114914476871,
+            -0.015796490013599396,
+            -3.563585551091819e-06,
+            -0.0022204883862286806,
+            -0.04388730227947235,
+            -0.025274382904171944,
+            0.040687184780836105,
+            0.006706364918500185,
+            0.02458864450454712,
+            0.0077268099412322044,
+            0.03418898954987526,
+            0.005885927006602287,
+            -0.009037061594426632,
+            0.030058229342103004,
+            -0.020049704238772392,
+            0.03833607956767082,
+            0.022106921300292015,
+            -0.002832755446434021,
+            0.005865518003702164,
+            0.031086837872862816,
+            -0.033797141164541245,
+            0.008086007088422775,
+            -0.034384917467832565,
+            0.0046695563942193985,
+            -0.008620720356702805,
+            -0.015094423666596413,
+            -0.009526874870061874,
+            -0.02504580281674862,
+            0.011216731742024422,
+            -0.021535472944378853,
+            0.01590261608362198,
+            0.010416703298687935,
+            -0.002916431985795498,
+            0.03448288142681122,
+            -0.014204595237970352,
+            -0.0027796924114227295,
+            0.0024531499948352575,
+            -0.027070365846157074,
+            -0.006343086715787649,
+            -0.003377673216164112,
+            -0.03689929470419884,
+            -0.004502203781157732,
+            0.010988152585923672,
+            -0.00024184548237826675,
+            0.016898570582270622,
+            0.02787039615213871,
+            0.0005092020728625357,
+            -0.022302847355604172,
+            0.002340900944545865,
+            -0.04199335351586342,
+            0.0011877981014549732,
+            0.01483318954706192,
+            -0.017600636929273605,
+            -0.008612556383013725,
+            0.033046092838048935,
+            0.023298801854252815,
+            -0.009396257810294628,
+            -0.035005345940589905,
+            0.017796561121940613,
+            -0.011853490024805069,
+            0.027511198073625565,
+            -0.0002630197268445045,
+            0.019298657774925232,
+            -0.0076941559091210365,
+            0.04493223875761032,
+            0.001369437319226563,
+            0.008645210415124893,
+            -0.021959977224469185,
+            0.007020662073045969,
+            -0.020278284326195717,
+            -0.002406209474429488,
+            -0.006171651650220156,
+            -0.012751481495797634,
+            -0.010669773444533348,
+            0.0032735876739025116,
+            0.007730891928076744,
+            -0.0176822729408741,
+            0.0055961208418011665,
+            -0.015959760174155235,
+            -0.02741323597729206,
+            0.007543129846453667,
+            -0.05965929850935936,
+            -0.0179598331451416,
+            -0.03373183310031891,
+            -0.009477893821895123,
+            0.02008235827088356,
+            0.03510330989956856,
+            -0.012759645469486713,
+            0.00497977202758193,
+            -0.014898497611284256,
+            0.004477712791413069,
+            0.03490738570690155,
+            0.044442422688007355,
+            -0.014759717509150505,
+            0.003381754970178008,
+            -0.010955498553812504,
+            0.033029764890670776,
+            -0.03347059711813927,
+            0.0030062312725931406,
+            0.031086837872862816,
+            -0.005057325586676598,
+            0.007959471084177494,
+            0.011624909937381744,
+            0.020474210381507874,
+            -0.019396619871258736,
+            0.00877582747489214,
+            0.002100075827911496,
+            0.019233347848057747,
+            0.02171507105231285,
+            0.020343592390418053,
+            -0.03539719805121422,
+            0.028409190475940704,
+            0.018384337425231934,
+            0.02444170042872429,
+            0.044442422688007355,
+            -0.03974021226167679,
+            -0.004147088620811701,
+            -0.0032225653994828463,
+            -0.02997659333050251,
+            0.02008235827088356,
+            -0.007530884351581335,
+            -0.01841699331998825,
+            -0.007069643586874008,
+            0.03327467292547226,
+            0.014571955427527428,
+            -0.05051611363887787,
+            0.01980479806661606,
+            0.01740471087396145,
+            0.02055584453046322,
+            0.024474354460835457,
+            -0.01123305968940258,
+            -0.0019072118448093534,
+            0.008996243588626385,
+            0.006804327480494976,
+            0.019266001880168915,
+            0.015257694758474827,
+            0.002012317767366767,
+            -0.0022082431241869926,
+            -0.011837163008749485,
+            0.005224678665399551,
+            -0.009077879600226879,
+            0.010808554477989674,
+            -0.02458864450454712,
+            -0.009698309935629368,
+            0.021388528868556023,
+            0.0031164393294602633,
+            -0.013665799982845783,
+            -0.03729114308953285,
+            -0.045944519340991974,
+            0.0033837957307696342,
+            0.004543021321296692,
+            -0.05365091934800148,
+            0.005322641227394342,
+            -0.026547899469733238,
+            -0.002108239568769932,
+            -0.005502239800989628,
+            0.0019214981002733111,
+            0.005636938381940126,
+            0.0167679525911808,
+            -0.013494365848600864,
+            0.02248244546353817,
+            -0.015502601861953735,
+            0.03165828809142113,
+            0.0055961208418011665,
+            0.007375776767730713,
+            -0.0027796924114227295,
+            0.05590406060218811,
+            -0.003712379140779376,
+            -0.03699725493788719,
+            0.023282473906874657,
+            -0.01311067771166563,
+            -0.0001747512142173946,
+            0.030090883374214172,
+            0.00884113647043705,
+            -0.015257694758474827,
+            0.013437220826745033,
+            -0.030613351613283157,
+            -0.03249097242951393,
+            0.010996315628290176,
+            -0.013755599036812782,
+            -0.0015776080545037985,
+            -0.04568328335881233,
+            -0.009339112788438797,
+            0.004269542172551155,
+            -0.018335357308387756,
+            0.010775899514555931,
+            0.01631895825266838,
+            0.013992343097925186,
+            -0.024245774373412132,
+            0.009053388610482216,
+            0.006865554489195347,
+            0.013453547842800617,
+            0.0007331897504627705,
+            0.014049487188458443,
+            -0.008677864447236061,
+            -0.011306531727313995,
+            -0.018319029361009598,
+            0.0032450153958052397,
+            -0.026792805641889572,
+            0.005155288614332676,
+            -0.003155216109007597,
+            -0.03771565109491348,
+            -0.06521052122116089,
+            -0.0344502255320549,
+            0.016294466331601143,
+            0.03396041318774223,
+            -0.009045224636793137,
+            0.0035144127905368805,
+            -0.00023648813657928258,
+            -0.01093917153775692,
+            0.024523336440324783,
+            -0.005930826533585787,
+            -0.0004829256213270128,
+            0.0035389035474509,
+            0.02050686441361904,
+            -0.009722800925374031,
+            0.005845109466463327,
+            -0.051952898502349854,
+            -0.01586179807782173,
+            0.01812310516834259,
+            0.006306350696831942,
+            0.014629100449383259,
+            -0.009690145961940289,
+            0.01693122461438179,
+            -0.030074557289481163,
+            -0.028147956356406212,
+            0.04130761697888374,
+            0.008277850225567818,
+            0.011486129835247993,
+            -0.003055212553590536,
+            -0.01680060662329197,
+            0.02504580281674862,
+            -0.0020633398089557886,
+            -0.006461458280682564,
+            -0.00817988719791174,
+            0.015820980072021484,
+            0.044279150664806366,
+            0.003255219664424658,
+            0.0009821783751249313,
+            0.017045514658093452,
+            -0.002663361607119441,
+            -0.011575928889214993,
+            0.007677828427404165,
+            0.013592327944934368,
+            -0.016384266316890717,
+            0.010375885292887688,
+            -0.025862159207463264,
+            0.0332583449780941,
+            0.03131541982293129,
+            0.011575928889214993,
+            -0.015045441687107086,
+            0.006130834110081196,
+            0.005330805201083422,
+            0.008086007088422775,
+            -0.001369437319226563,
+            -0.0017133272485807538,
+            0.020719116553664207,
+            -0.03276853263378143,
+            -0.01693122461438179,
+            -0.005347132217139006,
+            0.003932795487344265,
+            0.005277741700410843,
+            0.03396041318774223,
+            -0.0033266509417444468,
+            0.017927179113030434,
+            -0.011657563969492912,
+            0.010579974390566349,
+            0.021764053031802177,
+            0.037095218896865845,
+            -0.024996822699904442,
+            -0.029862305149435997,
+            0.012914752587676048,
+            -0.001296985661610961,
+            0.014065815135836601,
+            0.0010173837654292583,
+            0.014139287173748016,
+            0.0029409227427095175,
+            0.060475654900074005,
+            -0.017763907089829445,
+            0.004628738854080439,
+            0.0004035860183648765,
+            -0.01858026348054409,
+            -0.0013531101867556572,
+            0.004261378664523363,
+            -0.010979988612234592,
+            0.03608293831348419,
+            0.002344982698559761,
+            0.039185091853141785,
+            0.013918871060013771,
+            0.009290131740272045,
+            0.0015143405180424452,
+            -0.013592327944934368,
+            0.008355404250323772,
+            0.0015592400450259447,
+            -0.001132694073021412,
+            0.009461566805839539,
+            0.019298657774925232,
+            0.009714636951684952,
+            0.025274382904171944,
+            0.009788108989596367,
+            0.012857607565820217,
+            -0.029470453038811684,
+            0.03510330989956856,
+            -0.0006224715034477413,
+            0.012996388599276543,
+            -0.02040890045464039,
+            -0.003136848099529743,
+            -0.0008240094175562263,
+            0.015649545937776566,
+            -0.015616890974342823,
+            -0.001201063860207796,
+            -0.01297189760953188,
+            0.040099408477544785,
+            -0.018988441675901413,
+            0.013976015150547028,
+            0.009673818945884705,
+            -0.02865409664809704,
+            -0.007596192881464958,
+            0.01995174214243889,
+            0.014294394291937351,
+            -0.0053144777193665504,
+            -0.020490536466240883,
+            0.005832863971590996,
+            0.015380147844552994,
+            -0.028572462499141693,
+            1.3584674888988957e-05,
+            0.014751553535461426,
+            -0.0164903923869133,
+            -0.00025409081717953086,
+            -0.004702210891991854,
+            0.003975654020905495,
+            -0.009265640750527382,
+            0.007028825581073761,
+            0.016057724133133888,
+            -0.024507008492946625,
+            -0.020049704238772392,
+            0.005722655914723873,
+            0.006571666337549686,
+            0.01444950234144926,
+            0.007739055436104536,
+            0.004355259705334902,
+            -0.020033378154039383,
+            -0.01799248717725277,
+            0.0032674651592969894,
+            -0.01649855636060238,
+            -0.0048614004626870155,
+            0.0020531355403363705,
+            -0.019363965839147568,
+            0.01875986158847809,
+            -0.010743245482444763,
+            0.00017538899555802345,
+            -0.003226647386327386,
+            -0.020800752565264702,
+            0.006204306147992611,
+            -0.004865481983870268,
+            0.019870106130838394,
+            0.003555230563506484,
+            0.02785406820476055,
+            0.02310287579894066,
+            -0.004428731743246317,
+            0.017600636929273605,
+            -0.016155686229467392,
+            0.007983962073922157,
+            -0.031135819852352142,
+            -0.001782717532478273,
+            -0.010408539324998856,
+            0.0036103345919400454,
+            -0.015943434089422226,
+            -0.009126860648393631,
+            -0.0048777274787425995,
+            -0.0006678812787868083,
+            -0.004820582456886768,
+            0.0004275664687156677,
+            0.0027817331720143557,
+            0.02231917530298233,
+            -0.013208640739321709,
+            -0.02803366631269455,
+            -0.017437364906072617,
+            0.006045116577297449,
+            0.019396619871258736,
+            -0.026025431230664253,
+            -0.009184005670249462,
+            -0.005012426059693098,
+            -0.004326687194406986,
+            -0.021404854953289032,
+            -0.015167895704507828,
+            -0.005632856860756874,
+            -0.008045189082622528,
+            -0.010522829368710518,
+            0.0046532293781638145,
+            -0.0035715578123927116,
+            -0.011739199981093407,
+            0.021666089072823524,
+            0.018057795241475105,
+            -0.019837452098727226,
+            -0.019837452098727226,
+            0.013641309924423695,
+            -0.006159406621009111,
+            0.0018878233386203647,
+            0.02008235827088356,
+            0.00029286774224601686,
+            0.005824700463563204,
+            0.020882388576865196,
+            -0.0035225762985646725,
+            -0.01797616109251976,
+            0.003577680326998234,
+            -0.0018888438353314996,
+            -0.012122887186706066,
+            0.003212361130863428,
+            0.02773977816104889,
+            0.0008076822850853205,
+            0.023429417982697487,
+            -0.03056436963379383,
+            -0.011412657797336578,
+            0.006061443593353033,
+            0.02369065210223198,
+            0.00211844383738935,
+            -0.0021694661118090153,
+            0.020131340250372887,
+            0.03941367194056511,
+            0.005775718949735165,
+            0.0015755671774968505,
+            -0.04003410041332245,
+            0.01027792226523161,
+            0.009208496659994125,
+            0.024621298536658287,
+            0.008339077234268188,
+            0.0014653591206297278,
+            -0.012906589545309544,
+            -0.009755454957485199,
+            -0.0017939424142241478,
+            0.014702572487294674,
+            -0.023870250210165977,
+            0.0024755997583270073,
+            -0.03157665207982063,
+            0.005094061605632305,
+            0.0021959978621453047,
+            -0.009559528902173042,
+            0.0335359051823616,
+            0.00980443600565195,
+            -0.02065380848944187,
+            0.015151568688452244,
+            0.005767555441707373,
+            0.007330877240747213,
+            0.022449791431427002,
+            -0.008228869177401066,
+            0.030450081452727318,
+            -0.005049162078648806,
+            0.01213921420276165,
+            0.02742956392467022,
+            0.032588932663202286,
+            -0.013102514669299126,
+            -0.003959327004849911,
+            0.0006097159348428249,
+            -0.0002561317232903093,
+            0.0007995187188498676,
+            -0.008204378187656403,
+            0.0037389106582850218,
+            -0.014800535514950752,
+            0.012710663489997387,
+            -0.019543563947081566,
+            -0.0034286954905837774,
+            0.0038532004691660404,
+            0.01991908811032772,
+            0.0016265894519165158,
+            -0.033633869141340256,
+            0.02055584453046322,
+            -0.015420965850353241,
+            0.0135596739128232,
+            0.001290863030590117,
+            -0.011110605672001839,
+            -0.00029210239881649613,
+            0.020588500425219536,
+            0.010947334580123425,
+            -0.007845181971788406,
+            -0.0135596739128232,
+            -0.028376536443829536,
+            -0.021535472944378853,
+            0.00823703221976757,
+            0.009445239789783955,
+            -0.0028837777208536863,
+            0.0021306893322616816,
+            -0.011869817040860653,
+            0.0006520643946714699,
+            -0.0008122742874547839,
+            0.01740471087396145,
+            -0.0335359051823616,
+            0.021943651139736176,
+            -0.019527235999703407,
+            -0.019543563947081566,
+            0.017045514658093452,
+            0.008049271069467068,
+            0.009445239789783955,
+            0.0033552234526723623,
+            -0.01995174214243889,
+            -0.012326976284384727,
+            0.0338951013982296,
+            -0.01138000376522541,
+            0.006298186723142862,
+            0.004408322740346193,
+            0.012914752587676048,
+            -0.008914608508348465,
+            -0.011926962062716484,
+            0.01784554310142994,
+            -0.009861581027507782,
+            0.013951525092124939,
+            0.0010010566329583526,
+            -0.0254703089594841,
+            0.02739690989255905,
+            -0.019233347848057747,
+            0.03487472981214523,
+            -0.021617108955979347,
+            0.00711862463504076,
+            0.0066614653915166855,
+            0.008792154490947723,
+            -0.006690037902444601,
+            -0.013224967755377293,
+            0.0012786176521331072,
+            0.03807484731078148,
+            -0.02623768337070942,
+            0.024180466309189796,
+            0.022400809451937675,
+            -0.00040766780148260295,
+            0.02065380848944187,
+            -0.008367649279534817,
+            -0.026711169630289078,
+            0.0179598331451416,
+            -0.012596373446285725,
+            -0.026123393326997757,
+            0.013257622718811035,
+            -0.022678371518850327,
+            -0.020278284326195717,
+            -0.026302991434931755,
+            0.004947117529809475,
+            -0.0025694805663079023,
+            -0.005151206627488136,
+            0.009004407562315464,
+            0.02561725303530693,
+            0.020915042608976364,
+            0.019576217979192734,
+            0.02111096680164337,
+            0.01161674689501524,
+            0.022449791431427002,
+            -0.007559456862509251,
+            0.04401791840791702,
+            -0.004947117529809475,
+            0.004632820375263691,
+            -0.0015980169409886003,
+            -0.015657709911465645,
+            0.009771781973540783,
+            0.0119106350466609,
+            0.0029572497587651014,
+            -0.029176564887166023,
+            -0.009061552584171295,
+            -0.03660540655255318,
+            0.005053244065493345,
+            0.0020664012990891933,
+            0.0176822729408741,
+            0.032115448266267776,
+            -0.005865518003702164,
+            0.004592002835124731,
+            -0.005102225113660097,
+            -0.013927034102380276,
+            -0.009420748800039291,
+            -0.02739690989255905,
+            -0.025437654927372932,
+            -0.007653337903320789,
+            -0.013739272020757198,
+            -0.030368445441126823,
+            -0.011722872965037823,
+            0.01551892887800932,
+            -0.007645174395292997,
+            0.011918798089027405,
+            -0.016702644526958466,
+            0.01385356206446886,
+            -0.004110352601855993,
+            -0.004763437435030937,
+            -0.004530776292085648,
+            -0.017600636929273605,
+            -0.006420640274882317,
+            0.014367866329848766,
+            -0.009477893821895123,
+            -0.01237595733255148,
+            -0.02215590327978134,
+            -0.0015459742862731218,
+            -0.01574750803411007,
+            0.003489922033622861,
+            -0.014596446417272091,
+            -0.006771673448383808,
+            -0.003908304497599602,
+            -0.018514955416321754,
+            0.01766594499349594,
+            0.014816862531006336,
+            -0.019380291923880577,
+            -0.0090697156265378,
+            0.02638462744653225,
+            0.042777057737112045,
+            -0.03218075633049011,
+            -0.008200296200811863,
+            0.0020378287881612778,
+            0.018090449273586273,
+            0.010171796195209026,
+            0.016702644526958466,
+            -0.024653952568769455,
+            0.0022286521270871162,
+            -0.021649762988090515,
+            -0.001220452249981463,
+            -0.023739634081721306,
+            0.010775899514555931,
+            0.025715215131640434,
+            -0.010971825569868088,
+            0.025976449251174927,
+            0.01634344831109047,
+            -0.01812310516834259,
+            0.012016761116683483,
+            0.007183933164924383,
+            0.014645427465438843,
+            -0.019755816087126732,
+            0.0167679525911808,
+            -0.015796490013599396,
+            0.031233781948685646,
+            0.00892277155071497,
+            -0.04829562455415726,
+            -0.03503800183534622,
+            -0.013388238847255707,
+            0.0013388239312916994,
+            -0.010702427476644516,
+            0.015037278644740582,
+            0.015967924147844315,
+            0.013959688134491444,
+            -0.019380291923880577,
+            -0.015820980072021484,
+            0.011984107084572315,
+            0.010065670125186443,
+            0.027462217956781387,
+            -0.006649219896644354,
+            -0.004212397150695324,
+            -0.00960034690797329,
+            -0.02249877341091633,
+            -0.016098542138934135,
+            0.0209803506731987,
+            -0.018041469156742096,
+            0.015935270115733147,
+            -0.026139721274375916,
+            -0.012449429370462894,
+            0.017453692853450775,
+            -0.05211617052555084,
+            -0.01453113742172718,
+            0.0007612520130351186,
+            0.03012353740632534,
+            0.004751192405819893,
+            0.0035184945445507765,
+            0.01841699331998825,
+            -0.01993541419506073,
+            -0.0039776950143277645,
+            -0.02099667675793171,
+            0.006832899991422892,
+            -0.011853490024805069,
+            -0.013984179124236107,
+            0.014482156373560429,
+            -0.013845399022102356,
+            -0.01601690612733364,
+            0.0034368589986115694,
+            0.007183933164924383,
+            -0.021094640716910362,
+            0.011951452121138573,
+            0.003945040516555309,
+            0.012171869166195393,
+            0.012171869166195393,
+            0.01199227012693882,
+            0.021551799029111862,
+            0.02295593172311783,
+            -0.007334959227591753,
+            -0.004767519421875477,
+            -0.01004117913544178,
+            -0.0015398516552522779,
+            0.008898280560970306,
+            0.0033797139767557383,
+            -0.002989904023706913,
+            0.007837017998099327,
+            0.010033016093075275,
+            -0.01498013362288475,
+            0.033193036913871765,
+            -0.0035878848284482956,
+            -0.04134026914834976,
+            -0.003238892648369074,
+            -0.03800953924655914,
+            0.020033378154039383,
+            -0.023298801854252815,
+            -0.016996532678604126,
+            0.0032776694279164076,
+            -0.00629410520195961,
+            -0.019902760162949562,
+            0.013608654960989952,
+            0.011404493823647499,
+            0.00258580781519413,
+            0.02021297626197338,
+            0.012269831262528896,
+            -0.005134879611432552,
+            0.026172375306487083,
+            -0.014718899503350258,
+            -0.008751336485147476,
+            0.015567909926176071,
+            0.006853308994323015,
+            0.0022347746416926384,
+            0.0011031010653823614,
+            -0.018825169652700424,
+            0.005098143592476845,
+            0.0164903923869133,
+            0.007216587662696838,
+            -0.000566347036510706,
+            0.014049487188458443,
+            0.012514738366007805,
+            -0.015135240741074085,
+            -0.0002213090192526579,
+            -0.026564225554466248,
+            0.0032817514147609472,
+            0.004567512311041355,
+            0.007506393827497959,
+            -0.006771673448383808,
+            0.006281859707087278,
+            0.00658799335360527,
+            0.014759717509150505,
+            -0.010155469179153442,
+            -0.016980206593871117,
+            -0.002510294783860445,
+            0.004285869188606739,
+            0.016694480553269386,
+            0.03157665207982063,
+            -0.010310577228665352,
+            -0.005734901409596205,
+            -0.0386626236140728,
+            0.02070278860628605,
+            0.03332365304231644,
+            0.011175914667546749,
+            -0.028213264420628548,
+            0.012302485294640064,
+            0.0006515541463159025,
+            0.0019276207312941551,
+            -0.026302991434931755,
+            0.02787039615213871,
+            0.00974729098379612,
+            0.0066614653915166855,
+            0.017861871048808098,
+            0.0022306928876787424,
+            0.0029266364872455597,
+            -0.016302630305290222,
+            0.01018812321126461,
+            0.009086042642593384,
+            -0.017306748777627945,
+            0.0009745250572450459,
+            0.02324981987476349,
+            -0.01423724927008152,
+            -0.003208279376849532,
+            0.0018133309204131365,
+            0.02651524543762207,
+            -0.013820908032357693,
+            -0.014294394291937351,
+            0.0005714492290280759,
+            0.01751900091767311,
+            -0.0007979880319908261,
+            0.012098397128283978,
+            0.04898136481642723,
+            -0.012424939312040806,
+            0.019657853990793228,
+            0.005461421795189381,
+            -0.015290348790585995,
+            -0.032605260610580444,
+            -0.004922627005726099,
+            -0.01830270327627659,
+            0.011200404725968838,
+            0.014343376271426678,
+            -0.015763835981488228,
+            0.0007525782566517591,
+            -0.03144603595137596,
+            0.007743136957287788,
+            0.014735226519405842,
+            0.018776189535856247,
+            -0.002440904499962926,
+            0.006134915631264448,
+            0.027674470096826553,
+            0.02458864450454712,
+            -0.007392104249447584,
+            0.013804581016302109,
+            0.02320083975791931,
+            -0.025062130764126778,
+            -0.003069498809054494,
+            0.006820654962211847,
+            0.004104230087250471,
+            0.0053063142113387585,
+            -0.016963878646492958,
+            0.03915243595838547,
+            -0.005922663025557995,
+            -0.01506176870316267,
+            0.0030266400426626205,
+            -0.009028897620737553,
+            0.0063390047289431095,
+            -0.012563719414174557,
+            0.012637191452085972,
+            -0.008114579133689404,
+            0.00022883480414748192,
+            -0.0059430720284581184,
+            -0.011110605672001839,
+            -0.012653518468141556,
+            -0.0007949267164804041,
+            -0.016531210392713547,
+            0.007432921789586544,
+            0.00981259997934103,
+            0.005045080557465553,
+            0.01025343220680952,
+            -0.0036797248758375645,
+            -0.003136848099529743,
+            -0.016751626506447792,
+            -0.009984034113585949,
+            0.0073716952465474606,
+            0.01872720755636692,
+            -0.028588788583874702,
+            -0.018547609448432922,
+            0.009290131740272045,
+            0.0022898786701261997,
+            0.018776189535856247,
+            0.0158944521099329,
+            -0.0002595757250674069,
+            0.012318813242018223,
+            -0.01964152604341507,
+            -0.009184005670249462,
+            0.009690145961940289,
+            -0.003569516818970442,
+            0.01048201136291027,
+            -0.014759717509150505,
+            0.010033016093075275,
+            -0.011159587651491165,
+            -0.011053460650146008,
+            0.01385356206446886,
+            0.01738838478922844,
+            0.008873790502548218,
+            -0.001294944784604013,
+            3.383413059054874e-05,
+            -0.010506502352654934,
+            -0.024033522233366966,
+            0.004771600943058729,
+            0.027021385729312897,
+            -0.013273949734866619,
+            0.003883813973516226,
+            0.008702355436980724,
+            -0.017502672970294952,
+            -0.019821124151349068,
+            -0.012653518468141556,
+            -0.022106921300292015,
+            0.020572172477841377,
+            0.023413091897964478,
+            -0.02026195637881756,
+            0.00877582747489214,
+            -0.004620575346052647,
+            0.014049487188458443,
+            -0.01400050614029169,
+            0.010612628422677517,
+            0.0317889042198658,
+            -0.020000724121928215,
+            -0.0028654097113758326,
+            0.0036633978597819805,
+            -0.015347493812441826,
+            0.018629245460033417,
+            0.03268689662218094,
+            -0.00176945177372545,
+            0.014041324146091938,
+            0.0008138049161061645,
+            -0.009118696674704552,
+            0.01905374974012375,
+            0.011094278655946255,
+            0.005347132217139006,
+            -0.0042491331696510315,
+            -0.012212686240673065,
+            -0.03010721132159233,
+            0.016702644526958466,
+            0.0129637336358428,
+            -0.0037674831692129374,
+            -0.0011551438365131617,
+            -0.006343086715787649,
+            -0.015404638834297657,
+            0.0035123717971146107,
+            -0.013078023679554462,
+            -0.00803294312208891,
+            -0.008759500458836555,
+            0.004314441699534655,
+            -0.0034960447810590267,
+            -0.009053388610482216,
+            0.007236996199935675,
+            0.0009510547970421612,
+            -0.0010602424154058099,
+            -0.01902109570801258,
+            -0.025078458711504936,
+            -0.00995954405516386,
+            -0.016849588602781296,
+            0.0038287099450826645,
+            0.0005234883283264935,
+            -0.027184655889868736,
+            0.0053961132653057575,
+            -0.008139070123434067,
+            -0.008134988136589527,
+            0.007126788143068552,
+            0.006881881505250931,
+            -0.007437003776431084,
+            -0.010816717520356178,
+            -0.00786150898784399,
+            -0.006865554489195347,
+            0.020033378154039383,
+            0.0034348180051892996,
+            0.012082069180905819,
+            -0.010498338378965855,
+            -0.018025141209363937,
+            -0.016963878646492958,
+            0.016718972474336624,
+            -0.008971752598881721,
+            -0.02083340659737587,
+            -0.01444950234144926,
+            -0.008424794301390648,
+            -0.007665583398193121,
+            -0.01527402177453041,
+            -0.021159948781132698,
+            -0.0032103201374411583,
+            -0.011347348801791668,
+            -0.0067431009374558926,
+            -0.0028960229828953743,
+            0.0013571919407695532,
+            0.00951871182769537,
+            0.00546958576887846,
+            0.00438791373744607,
+            0.021666089072823524,
+            0.010122815147042274,
+            -0.005461421795189381,
+            0.016441410407423973,
+            -0.006183897145092487,
+            -0.004440976772457361,
+            -0.01498013362288475,
+            -0.009918726049363613,
+            0.018547609448432922,
+            -0.0055226488038897514,
+            0.008457448333501816,
+            -0.0009536059224046767,
+            -0.00576347392052412,
+            0.015674035996198654,
+            0.0129637336358428,
+            0.016718972474336624,
+            0.004771600943058729,
+            -0.00012168807734269649,
+            0.012416775338351727,
+            -0.0015745466807857156,
+            0.012555556371808052,
+            -0.0038429962005466223,
+            0.014506647363305092,
+            -0.009722800925374031,
+            -0.018988441675901413,
+            0.004157293122261763,
+            0.006265532691031694,
+            -0.012531065382063389,
+            0.009404421783983707,
+            0.007743136957287788,
+            -0.013747435994446278,
+            -0.02261306345462799,
+            -0.000106317616882734,
+            0.00014362763613462448,
+            0.008122743107378483,
+            0.008971752598881721,
+            -0.008073761127889156,
+            0.012890261597931385,
+            0.004616493359208107,
+            -0.007179851643741131,
+            -0.019657853990793228,
+            0.026009103283286095,
+            0.015608727931976318,
+            0.012269831262528896,
+            -0.052508022636175156,
+            -0.0030980713199824095,
+            0.010743245482444763,
+            -0.015951598063111305,
+            -0.026792805641889572,
+            -0.006404313258826733,
+            -0.008759500458836555,
+            -0.04290767386555672,
+            -0.009624837897717953,
+            -0.0005724696675315499,
+            -0.011388166807591915,
+            0.010400376282632351,
+            0.01573934406042099,
+            0.014612773433327675,
+            0.005489994306117296,
+            -0.007232914678752422,
+            -0.017633290961384773,
+            -0.02788672223687172,
+            0.010057506151497364,
+            -0.013380075804889202,
+            -0.006412476766854525,
+            -0.002263347152620554,
+            -0.007690073922276497,
+            0.0016112826997414231,
+            0.01691489666700363,
+            0.042483169585466385,
+            0.023037567734718323,
+            -0.012335140258073807,
+            -0.020131340250372887,
+            -0.012800462543964386,
+            -0.009192168712615967,
+            0.00633492274209857,
+            0.011592255905270576,
+            -0.004738946910947561,
+            0.0096656559035182,
+            -0.004294032696634531,
+            0.012302485294640064,
+            0.012767808511853218,
+            0.01680060662329197,
+            -2.3023791072773747e-05,
+            0.005579793825745583,
+            -0.0024694770108908415,
+            -0.01738838478922844,
+            -0.004379750229418278,
+            0.008261523209512234,
+            -0.0008872769540175796,
+            0.0019317024853080511,
+            0.014204595237970352,
+            0.019102731719613075,
+            -0.009094206616282463,
+            -0.014612773433327675,
+            -0.016098542138934135,
+            -0.005277741700410843,
+            -0.0012622905196622014,
+            0.029927613213658333,
+            0.02037624642252922,
+            -0.019266001880168915,
+            -0.018237393349409103,
+            -0.018351683393120766,
+            0.00922482367604971,
+            -0.003563394071534276,
+            -0.0064369672909379005,
+            0.01905374974012375,
+            0.023021239787340164,
+            0.017910851165652275,
+            -0.003642988856881857,
+            -0.014865843579173088,
+            0.003804219188168645,
+            0.02847449854016304,
+            0.008367649279534817,
+            0.01078406348824501,
+            0.008726846426725388,
+            -0.0011908593587577343,
+            -0.032441988587379456,
+            -0.0113146947696805,
+            0.00951871182769537,
+            -0.008481939323246479,
+            0.026564225554466248,
+            -0.016531210392713547,
+            0.009388094767928123,
+            -0.005077734589576721,
+            0.0019286411115899682,
+            -0.02365799807012081,
+            0.0014929111348465085,
+            0.010122815147042274,
+            -0.024947840720415115,
+            0.00336950970813632,
+            -0.023592690005898476,
+            -0.00013763252354692668,
+            0.008367649279534817,
+            0.0022837561555206776,
+            0.01650671847164631,
+            -0.0113146947696805,
+            0.0051838611252605915,
+            0.02099667675793171,
+            -0.002832755446434021,
+            -0.01512707769870758,
+            -0.0007842120248824358,
+            0.0005092020728625357,
+            -0.001286781276576221,
+            0.02563357912003994,
+            0.015812816098332405,
+            -0.0036756431218236685,
+            0.020882388576865196,
+            0.008677864447236061,
+            0.0019133344758301973,
+            -0.021764053031802177,
+            0.0023796779569238424,
+            -0.032899148762226105,
+            0.017437364906072617,
+            -0.011126932688057423,
+            0.012490247376263142,
+            0.00868602842092514,
+            -0.01341272983700037,
+            -0.0007388022495433688,
+            -0.0107187544927001,
+            -0.0015725058037787676,
+            -0.007934981025755405,
+            0.009779945947229862,
+            -0.004363423213362694,
+            0.008339077234268188,
+            -0.00024579971795901656,
+            -0.01476788055151701,
+            0.018694553524255753,
+            9.678283277025912e-06,
+            -0.00010925139940809458,
+            0.005441013257950544,
+            0.009935053065419197,
+            0.00786559097468853,
+            0.002009256277233362,
+            0.023870250210165977,
+            -0.031119491904973984,
+            -0.016082214191555977,
+            -0.010204450227320194,
+            0.005543057806789875,
+            -0.007922735065221786,
+            -0.018531281501054764,
+            -0.03257260471582413,
+            0.01318415068089962,
+            0.001293924287892878,
+            -0.0009806477464735508,
+            0.01258004643023014,
+            -0.00942891277372837,
+            -0.0013878053287044168,
+            -0.01606588624417782,
+            -0.014808698557317257,
+            0.006416558753699064,
+            0.0033633869607001543,
+            -0.01222901325672865,
+            -0.023739634081721306,
+            0.014759717509150505,
+            0.0007418635650537908,
+            -0.016980206593871117,
+            -0.017502672970294952,
+            0.00018457300029695034,
+            -0.004804255440831184,
+            0.02204161323606968,
+            -0.016678154468536377,
+            0.003373591462150216,
+            0.0007219648687168956,
+            0.009232986718416214,
+            -0.02889900468289852,
+            0.00716352416202426,
+            0.0006107363733462989,
+            0.0035164537839591503,
+            -0.006518602836877108,
+            -0.021013004705309868,
+            0.011371839791536331,
+            -0.0020674217958003283,
+            -0.014751553535461426,
+            -0.020751770585775375,
+            -0.0020796670578420162,
+            -0.007420676294714212,
+            -0.02724996581673622,
+            0.017910851165652275,
+            0.0009985055075958371,
+            -0.01738838478922844,
+            0.008077843114733696,
+            0.006596156861633062,
+            0.009355440735816956,
+            0.02455599047243595,
+            0.010147306136786938,
+            -0.018776189535856247,
+            -0.0008087027235887945,
+            0.006979844067245722,
+            0.009306458756327629,
+            0.0036450298503041267,
+            0.04222193360328674,
+            -0.008130906149744987,
+            -0.006579829845577478,
+            0.0066614653915166855,
+            -0.008449285291135311,
+            -0.013747435994446278,
+            0.026727497577667236,
+            0.005796127952635288,
+            0.015984252095222473,
+            0.016123032197356224,
+            -0.00945340283215046,
+            -0.005008344538509846,
+            -0.0018786393338814378,
+            0.02065380848944187,
+            0.0073798587545752525,
+            -0.005502239800989628,
+            -0.009714636951684952,
+            0.005881845485419035,
+            -0.00898808054625988,
+            0.0021653843577951193,
+            -0.009526874870061874,
+            -0.005824700463563204,
+            -0.008069679141044617,
+            0.016400592401623726,
+            -0.0003523086488712579,
+            0.027315273880958557,
+            0.020751770585775375,
+            -0.008563575334846973,
+            -0.0005683879135176539,
+            0.017208784818649292,
+            -0.005820618476718664,
+            0.015608727931976318,
+            -0.011192241683602333,
+            -0.007706400938332081,
+            0.006408394780009985,
+            0.024017194285988808,
+            0.012237177230417728,
+            -0.0056042843498289585,
+            0.0002885308349505067,
+            -0.0030348035506904125,
+            -0.010612628422677517,
+            0.013388238847255707,
+            0.015323002822697163,
+            -0.01947825588285923,
+            -0.004722619894891977,
+            -0.0016592437168583274,
+            -0.003300119424238801,
+            0.00945340283215046,
+            0.02305389568209648,
+            0.000171817431692034,
+            0.009184005670249462,
+            0.0012765767751261592,
+            0.03862996771931648,
+            -0.0053144777193665504,
+            0.03102152980864048,
+            -0.0010423846542835236,
+            -0.0059430720284581184,
+            -0.009094206616282463,
+            -0.020441556349396706,
+            -0.011486129835247993,
+            -0.00041481092921458185,
+            -0.005040998570621014,
+            -0.0004956811899319291,
+            -0.020327266305685043,
+            0.0026960158720612526,
+            0.014637264423072338,
+            0.007661501411348581,
+            0.017927179113030434,
+            -0.002889900468289852,
+            0.017437364906072617,
+            0.019690508022904396,
+            -0.01400050614029169,
+            -0.029862305149435997,
+            -0.0006515541463159025,
+            0.01498013362288475,
+            -0.01661284640431404,
+            0.019445601850748062,
+            0.004085862077772617,
+            0.0029531680047512054,
+            0.004779764451086521,
+            -0.006049198564141989,
+            0.018482301384210587,
+            0.02039257436990738,
+            0.01453930139541626,
+            -0.013167822733521461,
+            0.014057651162147522,
+            0.009771781973540783,
+            0.006600238848477602,
+            -0.01646590232849121,
+            -0.013535182923078537,
+            0.002555194543674588,
+            0.0009245232213288546,
+            0.024392718449234962,
+            -0.020588500425219536,
+            0.0016602640971541405,
+            0.010588138364255428,
+            0.01769859902560711,
+            -0.004302196204662323,
+            0.008906444534659386,
+            0.0008740111952647567,
+            0.007134951651096344,
+            -0.0035919665824621916,
+            -0.006457376293838024,
+            -0.006673710886389017,
+            -0.010392212308943272,
+            -0.0020959940738976,
+            -0.02070278860628605,
+            0.002198038622736931,
+            -0.002830714685842395,
+            0.018237393349409103,
+            -0.01018812321126461,
+            0.038140155375003815,
+            0.03399306535720825,
+            -0.0007025764207355678,
+            0.011102442629635334,
+            -0.01709449477493763,
+            -0.003945040516555309,
+            -0.021959977224469185,
+            0.01844964735209942,
+            0.016392430290579796,
+            0.012269831262528896,
+            0.005302232690155506,
+            0.017796561121940613,
+            -0.0006301248213276267,
+            0.014490320347249508,
+            0.0029654132667928934,
+            0.007971717044711113,
+            -0.002265388146042824,
+            0.003938918001949787,
+            -0.031103165820240974,
+            -0.017029186710715294,
+            -0.01138000376522541,
+            0.004796091932803392,
+            -0.006375740747898817,
+            -0.0035205355379730463,
+            0.0084737753495574,
+            -0.005902254022657871,
+            0.0004270562494639307,
+            -0.025992777198553085,
+            -0.0157148540019989,
+            -0.010318740271031857,
+            -0.004040962550789118,
+            -0.01169838197529316,
+            -0.0027694879099726677,
+            -0.0022082431241869926,
+            0.0029552089981734753,
+            0.020947696641087532,
+            -0.012808626517653465,
+            -0.016294466331601143,
+            0.004379750229418278,
+            -0.002263347152620554,
+            -0.014016833156347275,
+            -0.016898570582270622,
+            0.006665547378361225,
+            0.012098397128283978,
+            -0.0017470018938183784,
+            -0.013608654960989952,
+            0.011739199981093407,
+            -0.006628811359405518,
+            0.04663025587797165,
+            -0.013584164902567863,
+            -0.0078002819791436195,
+            0.003720542648807168,
+            0.004698128905147314,
+            -0.009396257810294628,
+            0.007330877240747213,
+            0.013020878657698631,
+            0.01830270327627659,
+            0.0095758568495512,
+            -0.008465612307190895,
+            -0.0024388637393712997,
+            0.007796200457960367,
+            -0.016147522255778313,
+            0.006404313258826733,
+            0.0008525818702764809,
+            -0.014400520361959934,
+            0.006110425107181072,
+            0.028980640694499016,
+            0.013151495717465878,
+            0.01288209855556488,
+            -0.022351829335093498,
+            -0.004094025585800409,
+            -0.009355440735816956,
+            -0.004943036008626223,
+            -0.010090161114931107,
+            0.027070365846157074,
+            -0.018155759200453758,
+            0.02651524543762207,
+            0.00898808054625988,
+            0.0017684312770143151,
+            -0.01176369097083807,
+            0.006343086715787649,
+            0.02832755446434021,
+            0.0028592869639396667,
+            -0.020000724121928215,
+            -0.013592327944934368,
+            0.006314514204859734,
+            0.01707816869020462,
+            -0.002647034591063857,
+            -0.005428767763078213,
+            0.019657853990793228,
+            0.01784554310142994,
+            -0.004334850702434778,
+            -0.00936360377818346,
+            0.0008337036124430597,
+            0.00928196869790554,
+            0.016963878646492958,
+            0.015020951628684998,
+            0.009086042642593384,
+            -0.002647034591063857,
+            0.008037025108933449,
+            -0.008800318464636803,
+            -0.016449574381113052,
+            0.007220669183880091,
+            -0.008432958275079727,
+            -0.016963878646492958,
+            0.01872720755636692,
+            0.015257694758474827,
+            0.0181394312530756,
+            0.0024592725094407797,
+            -0.016735298559069633,
+            0.022302847355604172,
+            0.005755309946835041,
+            0.002998067531734705,
+            0.02160078100860119,
+            -0.022221211344003677,
+            0.013012715615332127,
+            0.02037624642252922,
+            0.027609162032604218,
+            -0.01737205684185028,
+            -0.010906516574323177,
+            -0.004857318475842476,
+            -0.006853308994323015,
+            -0.00176945177372545,
+            -0.008253359235823154,
+            -0.012620864436030388,
+            0.0028735732194036245,
+            -0.030466407537460327,
+            0.018106777220964432,
+            -0.005102225113660097,
+            0.010669773444533348,
+            -0.019266001880168915,
+            0.011363675817847252,
+            -0.0035838030744343996,
+            0.011135096661746502,
+            0.0019612954929471016,
+            -0.030041903257369995,
+            -0.008571738377213478,
+            0.015388311818242073,
+            -0.0030368445441126823,
+            -0.013641309924423695,
+            0.02006603218615055,
+            0.00018942011229228228,
+            -0.022270193323493004,
+            -0.004314441699534655,
+            -0.006934944540262222,
+            0.002193956868723035,
+            -0.0029613315127789974,
+            -0.0052981507033109665,
+            -0.030188847333192825,
+            0.020033378154039383,
+            0.009559528902173042,
+            -0.03072764165699482,
+            -0.003255219664424658,
+            -0.014996460638940334,
+            -0.0025184585247188807,
+            0.008881953544914722,
+            -0.004000144544988871,
+            -0.015551582910120487,
+            0.01784554310142994,
+            0.022711025550961494,
+            -0.0023164101876318455,
+            -0.016057724133133888,
+            -0.0090697156265378,
+            -0.027494871988892555,
+            0.030825603753328323,
+            -0.0066696288995444775,
+            0.0022878379095345736,
+            -0.005322641227394342,
+            0.009388094767928123,
+            0.004228724166750908,
+            0.0157148540019989,
+            0.006081852596253157,
+            -0.002647034591063857,
+            0.026596879586577415,
+            -0.01964152604341507,
+            0.009567692875862122,
+            -1.588864870427642e-05,
+            -0.003308282932266593,
+            0.01890680566430092,
+            0.0007939062779769301,
+            0.021649762988090515,
+            -0.008571738377213478,
+            -0.0001475818717153743,
+            -0.0034388997592031956,
+            0.010898353531956673,
+            -0.0028450009413063526,
+            -0.003465431509539485,
+            -0.001531688030809164,
+            0.011118769645690918,
+            0.01013097818940878,
+            0.003804219188168645,
+            -0.01737205684185028,
+            0.003142970846965909,
+            -0.008547247387468815,
+            -0.007028825581073761,
+            0.01302904263138771,
+            -0.011192241683602333,
+            -0.0018990482203662395,
+            -0.03160930797457695,
+            0.016286302357912064,
+            0.0533243790268898,
+            0.008547247387468815,
+            -0.014433175325393677,
+            -0.0018806802108883858,
+            -0.019755816087126732,
+            0.01535565685480833,
+            0.011004479601979256,
+            -0.016816934570670128,
+            -0.011984107084572315,
+            -0.012310649268329144,
+            0.007649256382137537,
+            0.021306892856955528,
+            -0.006522684823721647,
+            -0.008579902350902557,
+            0.025372346863150597,
+            -0.011788181029260159,
+            -0.010882026515901089,
+            -0.01661284640431404,
+            -0.01635977439582348,
+            0.00241641397587955,
+            0.019902760162949562,
+            -0.0169475506991148,
+            0.021176276728510857,
+            0.00936360377818346,
+            0.0010133018950000405,
+            0.003624620847404003,
+            0.008588065393269062,
+            0.0024878450203686953,
+            0.017878197133541107,
+            0.003883813973516226,
+            -0.01439235731959343,
+            0.0021408938337117434,
+            -0.0030286810360848904,
+            0.004624656867235899,
+            3.4320437407586724e-06,
+            -0.002749078907072544,
+            -0.0065635028295218945,
+            0.003710338147357106,
+            -0.021306892856955528,
+            0.021502818912267685,
+            -0.022678371518850327,
+            -0.004245051648467779,
+            0.005555302836000919,
+            -0.005118552595376968,
+            0.013469874858856201,
+            0.018792515620589256,
+            0.01423724927008152,
+            0.008702355436980724,
+            0.01462093647569418,
+            -0.011331021785736084,
+            0.02323349379003048,
+            -0.013282112777233124,
+            0.0005816536722704768,
+            0.012939243577420712,
+            0.01755165494978428,
+            0.018792515620589256,
+            -0.006245123688131571,
+            -0.010384049266576767,
+            -0.0002946535241790116,
+            0.0129637336358428,
+            -0.022286519408226013,
+            -0.012269831262528896,
+            -0.01386172603815794,
+            0.011486129835247993,
+            -0.004743028432130814,
+            0.025453981012105942,
+            -0.00633492274209857,
+            -0.014653591439127922,
+            -0.015918942168354988,
+            0.014196431264281273,
+            -0.048752784729003906,
+            0.0030348035506904125,
+            -0.0024919267743825912,
+            0.03043375350534916,
+            -0.027021385729312897,
+            0.01738838478922844,
+            -0.0028490826953202486,
+            -0.009355440735816956,
+            0.015429128892719746,
+            -0.015992414206266403,
+            -0.0046613928861916065,
+            -0.0016806729836389422,
+            -0.007539048325270414,
+            -0.00112248957157135,
+            0.017616963014006615,
+            -0.015845470130443573,
+            0.01282495353370905,
+            -0.00989423505961895,
+            0.0027021386194974184,
+            -0.004526694305241108,
+            -0.007751300930976868,
+            0.0196088720113039,
+            0.034548189491033554,
+            -0.007053316105157137,
+            0.0014990337658673525,
+            -0.022547753527760506,
+            0.013192313723266125,
+            -0.006730855442583561,
+            0.02351105399429798,
+            -0.005432849284261465,
+            -0.016898570582270622,
+            -0.013976015150547028,
+            -0.009739127941429615,
+            0.00839214026927948,
+            -0.022433463484048843,
+            -0.011110605672001839,
+            -0.00922482367604971,
+            -0.002573562553152442,
+            -0.018694553524255753,
+            0.004865481983870268,
+            -0.002426618244498968,
+            -0.012506574392318726,
+            -0.008816645480692387,
+            0.01048201136291027,
+            -0.009347276762127876,
+            0.003077662317082286,
+            -0.0011929002357646823,
+            -0.011453475803136826,
+            -0.02369065210223198,
+            0.0158944521099329,
+            0.0001974561164388433,
+            -0.010359558276832104,
+            0.008408467285335064,
+            -0.0018276170594617724,
+            0.001211268245242536,
+            0.016963878646492958,
+            -0.0020664012990891933,
+            0.003467472270131111,
+            0.023478399962186813,
+            0.004057289566844702,
+            -0.010971825569868088,
+            -0.00025906547671183944,
+            0.013535182923078537,
+            -0.005118552595376968,
+            -0.010008525103330612,
+            1.6646010408294387e-05,
+            -0.02984597720205784,
+            -0.008351322263479233,
+            0.006094098091125488,
+            0.018629245460033417,
+            -0.010620792396366596,
+            -0.015853634104132652,
+            0.026319319382309914,
+            -0.006359413731843233,
+            0.019070077687501907,
+            0.002889900468289852,
+            0.0025694805663079023,
+            0.01724144071340561,
+            0.013461710885167122,
+            0.00786150898784399,
+            -0.013927034102380276,
+            -0.02429475635290146,
+            0.0016398552106693387,
+            -0.018090449273586273,
+            0.010579974390566349,
+            -0.011028970591723919,
+            0.0032756286673247814,
+            -0.02398454025387764,
+            -0.012237177230417728,
+            0.004934872500598431,
+            -0.020343592390418053,
+            -0.00040588201954960823,
+            -0.0056940834037959576,
+            0.022237539291381836,
+            -0.005310396198183298,
+            0.008669701404869556,
+            0.0018255761824548244,
+            -0.012237177230417728,
+            0.0009877908742055297,
+            0.009469729848206043,
+            -0.029192892834544182,
+            -0.0007127808639779687,
+            -0.003138889092952013,
+            0.0011418780777603388,
+            0.005351213738322258,
+            0.009755454957485199,
+            0.007322713732719421,
+            0.0027062203735113144,
+            0.007253323681652546,
+            0.010449357330799103,
+            -0.00022041612828616053,
+            0.0197884701192379,
+            0.018172085285186768,
+            -0.007351286243647337,
+            0.008710519410669804,
+            0.024817224591970444,
+            -0.013159659691154957,
+            -0.008718682453036308,
+            0.023266147822141647,
+            -0.006461458280682564,
+            -0.0011122850701212883,
+            -0.01483318954706192,
+            0.00815947912633419,
+            0.0017367975087836385,
+            0.028131630271673203,
+            0.019657853990793228,
+            0.005820618476718664,
+            0.000639819074422121,
+            0.015788326039910316,
+            -0.013020878657698631,
+            0.0035286990460008383,
+            0.015461783856153488,
+            -0.003940958995372057,
+            -0.019821124151349068,
+            0.013878053054213524,
+            0.009502384811639786,
+            -0.003228688146919012,
+            0.01551892887800932,
+            0.011412657797336578,
+            -0.030841931700706482,
+            -0.0011898389784619212,
+            0.010498338378965855,
+            -0.0021449755877256393,
+            -0.008751336485147476,
+            -0.00014273475972004235,
+            -0.012384121306240559,
+            0.021388528868556023,
+            -0.004408322740346193,
+            0.012277995236217976,
+            -0.00815947912633419,
+            -0.017127150669693947,
+            -0.011559601873159409,
+            0.007914572022855282,
+            -0.010547320358455181,
+            -0.0006714528426527977,
+            -0.004022594541311264,
+            -0.005779800936579704,
+            0.017747581005096436,
+            0.004706292413175106,
+            -0.0008964610169641674,
+            -0.0018398624379187822,
+            0.007983962073922157,
+            -0.006979844067245722,
+            -0.008726846426725388,
+            0.004816500470042229,
+            -0.012833117507398129,
+            0.003981776535511017,
+            0.005543057806789875,
+            0.01506176870316267,
+            -0.014857680536806583,
+            -0.021159948781132698,
+            0.007020662073045969,
+            0.02695607766509056,
+            -0.02516009286046028,
+            -0.005232842173427343,
+            0.006245123688131571,
+            0.002093953313305974,
+            0.006914535537362099,
+            -0.006934944540262222,
+            -0.007208424154669046,
+            -0.03657275065779686,
+            -0.0020745648071169853,
+            0.0009005427709780633,
+            0.008963589556515217,
+            -0.008049271069467068,
+            -0.0136494729667902,
+            -0.011135096661746502,
+            0.002410291228443384,
+            0.007134951651096344,
+            -0.02204161323606968,
+            0.0049593630246818066,
+            0.009608510881662369,
+            -0.019559891894459724,
+            0.025127438828349113,
+            -0.010890189558267593,
+            -6.964537897147238e-05,
+            -0.00995954405516386,
+            0.021633435040712357,
+            -0.013486201874911785,
+            -0.018368011340498924,
+            -0.0006898208521306515,
+            0.008355404250323772,
+            0.011926962062716484,
+            -0.0017663904000073671,
+            -0.011371839791536331,
+            0.0003752686607185751,
+            0.01646590232849121,
+            0.019135385751724243,
+            0.002744997153058648,
+            -0.0024837632663547993,
+            -0.01019628718495369,
+            -0.009592183865606785,
+            -0.0073961857706308365,
+            -0.0007449248805642128,
+            0.002091912319883704,
+            0.02035992033779621,
+            0.013469874858856201,
+            0.0024837632663547993,
+            -0.015135240741074085,
+            -0.005257333163172007,
+            0.008939098566770554,
+            0.009412584826350212,
+            -0.011690218932926655,
+            -0.00898808054625988,
+            0.002659279853105545,
+            0.02249877341091633,
+            0.017633290961384773,
+            -0.0014000505907461047,
+            0.0032878739293664694,
+            -0.0016592437168583274,
+            0.012506574392318726,
+            -0.007555375341325998,
+            -0.010269759222865105,
+            -0.006943108048290014,
+            -0.014645427465438843,
+            -0.010800390504300594,
+            -0.0023062059190124273,
+            -0.010179960168898106,
+            0.022629389539361,
+            0.007988044060766697,
+            0.009779945947229862,
+            0.008265605196356773,
+            0.010269759222865105,
+            0.002087830565869808,
+            0.025715215131640434,
+            -0.004424649756401777,
+            -0.011984107084572315,
+            0.007236996199935675,
+            -0.022368155419826508,
+            -0.021535472944378853,
+            0.008257441222667694,
+            0.006367577239871025,
+            0.03595232218503952,
+            -0.026890767738223076,
+            0.0052981507033109665,
+            -0.009779945947229862,
+            -0.010400376282632351,
+            0.007996207103133202,
+            -0.0015439334092661738,
+            -0.004245051648467779,
+            0.022368155419826508,
+            0.00726556871086359,
+            0.0035797213204205036,
+            0.0032817514147609472,
+            0.019837452098727226,
+            -0.005685919895768166,
+            0.01573934406042099,
+            0.0259274672716856,
+            0.024621298536658287,
+            0.009388094767928123,
+            -0.009469729848206043,
+            0.02785406820476055,
+            0.0002895512734539807,
+            -0.0034327772445976734,
+            0.007220669183880091,
+            -0.00305113079957664,
+            -0.0009556467994116247,
+            -0.010033016093075275,
+            0.021764053031802177,
+            -0.001455154619179666,
+            0.004359341226518154,
+            0.02081707864999771,
+            -7.774515688652173e-05,
+            0.006220633164048195,
+            0.01057181041687727,
+            0.002181711606681347,
+            0.003396041225641966,
+            0.010775899514555931,
+            -0.039315707981586456,
+            -0.0006571666453965008,
+            -0.025600925087928772,
+            -0.0018286375561729074,
+            0.0007357408758252859,
+            -0.01266984548419714,
+            0.013600491918623447,
+            -0.002015379024669528,
+            -0.010751409456133842,
+            -0.00029337796149775386,
+            0.021649762988090515,
+            0.005416522268205881,
+            0.004071575589478016,
+            -0.004469549283385277,
+            0.018792515620589256,
+            -0.006955353543162346,
+            -0.017861871048808098,
+            -0.012759645469486713,
+            -0.012367794290184975,
+            -0.01918436773121357,
+            -0.01214737817645073,
+            -0.0024551907554268837,
+            -0.002667443361133337,
+            0.004738946910947561,
+            0.002481722505763173,
+            -0.003228688146919012,
+            -0.02070278860628605,
+            0.015037278644740582,
+            -0.01430255826562643,
+            0.012873934581875801,
+            -0.011053460650146008,
+            0.005963481031358242,
+            -0.004122598096728325,
+            0.012726990506052971,
+            -0.021404854953289032,
+            0.02070278860628605,
+            -0.01536382082849741,
+            0.0038532004691660404,
+            0.019592545926570892,
+            0.006551257334649563,
+            -0.008118661120533943,
+            -0.013404566794633865,
+            -0.022074267268180847,
+            0.021633435040712357,
+            -0.011633073911070824,
+            -0.018237393349409103,
+            -0.004608329851180315,
+            0.009649328887462616,
+            -0.017470018938183784,
+            -0.007367613259702921,
+            0.01707816869020462,
+            -0.0018949664663523436,
+            0.012980061583220959,
+            0.023004913702607155,
+            -0.0052981507033109665,
+            0.0036572751123458147,
+            -0.009167678654193878,
+            0.0010286086471751332,
+            -0.02470293454825878,
+            0.013527019880712032,
+            0.008971752598881721,
+            0.006269614677876234,
+            -0.013935198076069355,
+            -0.013306603766977787,
+            0.0036531933583319187,
+            0.004498121794313192,
+            0.011437147855758667,
+            0.009420748800039291,
+            0.004220560658723116,
+            -0.013918871060013771,
+            -0.027788760140538216,
+            -0.027070365846157074,
+            0.03595232218503952,
+            -0.0005002731923013926,
+            -0.020947696641087532,
+            0.03399306535720825,
+            0.016114868223667145,
+            0.0022980421781539917,
+            -0.012449429370462894,
+            -0.004314441699534655,
+            0.024621298536658287,
+            0.016400592401623726,
+            0.010465684346854687,
+            -0.0004742518358398229,
+            0.005571629852056503,
+            0.0016469983384013176,
+            0.03012353740632534,
+            0.015323002822697163,
+            0.0009459525463171303,
+            -0.038597315549850464,
+            0.011943289078772068,
+            -0.007547211833298206,
+            0.0013408648082986474,
+            0.006702283397316933,
+            0.017780235037207603,
+            0.011935125105082989,
+            0.015567909926176071,
+            0.014865843579173088,
+            -0.003065417055040598,
+            0.0029511270113289356,
+            0.0006163488142192364,
+            -0.009624837897717953,
+            -0.001767410896718502,
+            -0.008881953544914722,
+            -0.005714492406696081,
+            -0.002920513739809394,
+            0.005592038854956627,
+            0.007657419890165329,
+            -0.003983817528933287,
+            0.028049994260072708,
+            -0.0006433906382881105,
+            -0.017600636929273605,
+            -0.009061552584171295,
+            -0.03611559048295021,
+            0.00434301421046257,
+            -0.030939893797039986,
+            -0.011257549747824669,
+            0.0033674687147140503,
+            -0.020457882434129715,
+            -0.024768242612481117,
+            0.0010306495241820812,
+            0.014335212297737598,
+            0.007392104249447584,
+            0.0033327736891806126,
+            -0.01004934310913086,
+            0.003816464450210333,
+            -0.001855169073678553,
+            -0.006461458280682564,
+            0.028833696618676186,
+            0.019559891894459724,
+            0.020457882434129715,
+            -0.01190247107297182,
+            0.019298657774925232,
+            0.005845109466463327,
+            0.02035992033779621,
+            0.010465684346854687,
+            -0.0048695639707148075,
+            0.0074859848245978355,
+            0.009567692875862122,
+            -0.006155324634164572,
+            -0.021519144997000694,
+            0.013192313723266125,
+            -0.0010837126756086946,
+            -0.0052083516493439674,
+            -0.016269976273179054,
+            -0.003534821793437004,
+            -0.015812816098332405,
+            0.010547320358455181,
+            -0.017159804701805115,
+            0.012498411349952221,
+            -0.000802069844212383,
+            0.00168475485406816,
+            -0.015469946898519993,
+            -0.013951525092124939,
+            -0.0012439225101843476,
+            -0.01244126632809639,
+            -0.008971752598881721,
+            0.006163488142192364,
+            -0.007539048325270414,
+            -0.008318668231368065,
+            -0.00535937724635005,
+            -0.011437147855758667,
+            -0.010882026515901089,
+            0.004367504734545946,
+            -0.0010653446661308408,
+            0.020931368693709373,
+            0.00011671340325847268,
+            -0.005824700463563204,
+            0.01190247107297182,
+            0.02787039615213871,
+            0.0016704685986042023,
+            0.004583839327096939,
+            -0.014261740259826183,
+            -0.011118769645690918,
+            -0.02279266156256199,
+            0.016572028398513794,
+            0.0049675265327095985,
+            -0.0344502255320549,
+            0.01438419334590435,
+            0.00824927818030119,
+            0.0015102586476132274,
+            -0.006698201410472393,
+            -0.005334886722266674,
+            -0.005608365871012211,
+            0.006306350696831942,
+            -0.0022837561555206776,
+            0.0005107327597215772,
+            0.0035919665824621916,
+            0.034678805619478226,
+            -0.0040674940682947636,
+            -0.007228832691907883,
+            0.00981259997934103,
+            0.0005130287609063089,
+            0.020196648314595222,
+            -0.005518566817045212,
+            0.0037572786677628756,
+            -0.019282329827547073,
+            -0.016563864424824715,
+            0.0018429238116368651,
+            -0.01722511276602745,
+            0.023282473906874657,
+            -0.005392031744122505,
+            -0.01347803883254528,
+            -0.006906372029334307,
+            -0.020115012302994728,
+            0.012637191452085972,
+            0.02310287579894066,
+            0.007016580086201429,
+            0.01033506728708744,
+            0.012131051160395145,
+            -0.02444170042872429,
+            -0.020457882434129715,
+            0.012171869166195393,
+            0.0013571919407695532,
+            -0.008449285291135311,
+            -0.005404277238994837,
+            -0.001783737912774086,
+            0.007783954963088036,
+            -0.0032633834052830935,
+            0.007881917990744114,
+            0.06491663306951523,
+            -0.01093100756406784,
+            -0.002987863030284643,
+            -0.02966637909412384,
+            -0.004396077245473862,
+            -0.006628811359405518,
+            0.0056287748739123344,
+            0.009151351638138294,
+            -0.023135529831051826,
+            0.009232986718416214,
+            -0.030025575309991837,
+            -0.010628955438733101,
+            0.00824111420661211,
+            0.0028490826953202486,
+            -0.01220452319830656,
+            -0.0009592183632776141,
+            -0.013665799982845783,
+            0.0006107363733462989,
+            -0.009412584826350212,
+            -0.00576347392052412,
+            0.012718827463686466,
+            -0.02321716584265232,
+            0.006298186723142862,
+            0.007592111360281706,
+            -0.006514521315693855,
+            -0.013722945004701614,
+            -0.017861871048808098,
+            -0.011967780068516731,
+            -0.013314766809344292,
+            0.002996026771143079,
+            0.01190247107297182,
+            -0.02607441134750843,
+            0.012922916561365128,
+            -0.0037389106582850218,
+            0.00945340283215046,
+            0.004404240753501654,
+            0.013469874858856201,
+            0.018074123188853264,
+            0.009355440735816956,
+            0.0021429345943033695,
+            -0.008024780079722404,
+            0.012702500447630882,
+            -0.01993541419506073,
+            -0.01635161228477955,
+            0.013837235048413277,
+            0.012016761116683483,
+            0.003061335301026702,
+            0.009828926995396614,
+            0.0020959940738976,
+            0.011886144056916237,
+            0.010775899514555931,
+            -0.019739490002393723,
+            -0.0006648199632763863,
+            0.0007326795603148639,
+            0.03280118480324745,
+            -0.0005938990507274866,
+            0.008857463486492634,
+            0.004420568235218525,
+            0.013159659691154957,
+            -0.012784135527908802,
+            -0.01769859902560711,
+            0.0008087027235887945,
+            -0.01057181041687727,
+            0.021257910877466202,
+            -0.008192133158445358,
+            0.006216551177203655,
+            -0.018612917512655258,
+            -0.027086693793535233,
+            0.006392067763954401,
+            -0.016261812299489975,
+            -0.018221067264676094,
+            0.01317598670721054,
+            0.0077268099412322044,
+            0.0017572063952684402,
+            -0.005098143592476845,
+            -0.020457882434129715,
+            -0.009837090037763119,
+            0.028441844508051872,
+            0.011102442629635334,
+            -0.004120557103306055,
+            0.009584019891917706,
+            -0.009094206616282463,
+            -0.012490247376263142,
+            -0.0017541450215503573,
+            0.0036715613678097725,
+            0.008261523209512234,
+            0.005645101889967918,
+            -0.01386172603815794,
+            -0.00029873527819290757,
+            -0.012400448322296143,
+            -0.021225256845355034,
+            -0.008906444534659386,
+            0.016751626506447792,
+            0.007820690982043743,
+            -0.0038960592355579138,
+            0.015812816098332405,
+            -0.004404240753501654,
+            0.002189875114709139,
+            0.014294394291937351,
+            0.005334886722266674,
+            -0.004285869188606739,
+            -0.0095758568495512,
+            -0.004612411838024855,
+            0.004043003544211388,
+            -0.0025960123166441917,
+            0.006457376293838024,
+            0.017192458733916283,
+            0.0072900597006082535,
+            -0.013755599036812782,
+            0.013135168701410294,
+            0.0007791098323650658,
+            0.0049675265327095985,
+            -0.03699725493788719,
+            -0.003247056156396866,
+            0.015796490013599396,
+            -0.013020878657698631,
+            0.02501314878463745,
+            0.005400195252150297,
+            0.022743679583072662,
+            0.015233203768730164,
+            0.008669701404869556,
+            -0.001542912912555039,
+            0.00464098434895277,
+            0.02759283408522606,
+            -0.003151134354993701,
+            -0.008637047372758389,
+            -0.0020602785516530275,
+            0.009241150692105293,
+            0.0002063850115519017,
+            0.003000108525156975,
+            0.008281932212412357,
+            -0.03562577813863754,
+            0.011706545948982239,
+            -0.019576217979192734,
+            0.0024694770108908415,
+            0.003459308762103319,
+            -0.007183933164924383,
+            -0.014865843579173088,
+            -0.005857354495674372,
+            0.022841641679406166,
+            -0.003224606392905116,
+            0.021078312769532204,
+            0.002885818714275956,
+            0.016106704249978065,
+            -0.011575928889214993,
+            -0.009208496659994125,
+            -0.0004130251472815871,
+            -0.005351213738322258,
+            -0.007539048325270414,
+            0.010596301406621933,
+            0.0034409407526254654,
+            0.005534893833100796,
+            0.002265388146042824,
+            0.0036654386203736067,
+            -0.0036878883838653564,
+            0.02773977816104889,
+            0.003563394071534276,
+            0.007996207103133202,
+            -0.011567764915525913,
+            0.014082142151892185,
+            -0.017029186710715294,
+            0.0070778070949018,
+            0.005575711838901043,
+            -0.026156047359108925,
+            0.016849588602781296,
+            0.004775682929903269,
+            0.006628811359405518,
+            0.013812744058668613,
+            0.01618834026157856,
+            -0.010408539324998856,
+            -0.014816862531006336,
+            0.024049850180745125,
+            -0.0032041973900049925,
+            -0.015225040726363659,
+            -0.006775755435228348,
+            4.438935866346583e-05,
+            0.006559420842677355,
+            0.00373074715025723,
+            0.0013020879123359919,
+            -0.0006204306264407933,
+            -0.010743245482444763,
+            0.004645065870136023,
+            -0.01934763789176941,
+            -0.0007163524278439581,
+            -0.016449574381113052,
+            -0.015633217990398407,
+            0.018678225576877594,
+            -0.01830270327627659,
+            0.009633000940084457,
+            0.0012275953777134418,
+            0.028588788583874702,
+            -0.011706545948982239,
+            -0.012155541218817234,
+            0.0015102586476132274,
+            0.010849371552467346,
+            0.0067512644454836845,
+            -0.010033016093075275,
+            -0.007583947852253914,
+            -0.014180104248225689,
+            -0.010163633152842522,
+            0.015502601861953735,
+            -0.014498483389616013,
+            -0.014343376271426678,
+            0.008914608508348465
+          ],
+          "embedding": [
+            -0.006359657085039302,
+            -0.024594786241399234,
+            -0.02553538924848095,
+            0.005360188465088291,
+            0.012801870977589839,
+            0.021680505862506104,
+            -0.016668048966001273,
+            0.06860852474581301,
+            -0.007122272841157724,
+            0.03471775000705488,
+            -0.009004297848861675,
+            0.01059974144678391,
+            0.02565604599426148,
+            -0.022544127646041287,
+            0.010616674976540785,
+            0.013658194823359737,
+            0.016400329743363844,
+            0.02858546964917478,
+            0.0067612676434693686,
+            -0.04174737193606879,
+            0.03184155110488616,
+            -0.02223105127585514,
+            -0.05506239667974977,
+            -0.0015307551410555752,
+            -0.013169788751379287,
+            0.017681749722384007,
+            0.01682706317229353,
+            0.019652295823816928,
+            -0.05623980369271269,
+            0.030698582851533197,
+            0.02286327124296613,
+            0.016019032539689178,
+            0.005508113353872315,
+            8.58985091215149e-05,
+            -0.019206292853495906,
+            -0.018439290483495144,
+            0.03475023176950151,
+            0.01018612942141759,
+            -0.0252551158857426,
+            0.0334562130666827,
+            0.020462153537485785,
+            0.02260508249714211,
+            -0.03323524989392951,
+            -0.002094251009530707,
+            0.015177442347686773,
+            -0.00018698464350038317,
+            -0.0006788648344273403,
+            0.0007777616893581039,
+            0.013352398615059647,
+            -0.030751111528744813,
+            -0.0317733818445192,
+            -0.006729064081710058,
+            0.009413204899986514,
+            -0.005712894407943592,
+            -0.038961826880527066,
+            -0.002827692206530345,
+            0.017863300537973137,
+            0.07918455388563496,
+            0.048967158919816366,
+            0.018002566325350088,
+            0.040264962620950695,
+            -0.019715100359659787,
+            -0.02211350500879408,
+            0.004918148608475574,
+            0.01062585305866606,
+            -0.014478993521696027,
+            0.0034983131226364672,
+            0.000766228629514787,
+            0.009667268771502067,
+            -0.01558439890697323,
+            -0.04984718709870083,
+            0.029613955432821887,
+            -0.009686854114630624,
+            -0.015279319238124568,
+            0.007460629676325322,
+            -0.006235979350809034,
+            0.012514387473717085,
+            0.02800435263741497,
+            -0.008688925946672778,
+            -0.01236301543374469,
+            0.029755705020188972,
+            0.01158037219446542,
+            -0.005242392206024511,
+            0.009982679027728814,
+            0.052414361714483264,
+            -0.018788500381726227,
+            -0.020335831196027204,
+            -0.08583959943356621,
+            0.029309228216812346,
+            -0.0249285282010673,
+            -0.018929396555426307,
+            0.007710663193334574,
+            0.03096798035958051,
+            -0.017089977064591828,
+            -0.028015107605405817,
+            0.06740703271128606,
+            0.009398795412354399,
+            0.01118769022826835,
+            -0.03041417460855722,
+            0.014776002883150835,
+            0.004319194067673161,
+            0.022507636639041203,
+            0.0053562788646786845,
+            0.007548072258943124,
+            0.011123374308563128,
+            -0.006924362115585165,
+            -0.011118507147092024,
+            0.009563232643295304,
+            0.00657178473678069,
+            0.08313027447380966,
+            -0.004239370156003264,
+            -0.00930406163946907,
+            -0.039202672821898946,
+            0.009761029930556298,
+            -0.02827698465424993,
+            -0.037914533246900244,
+            -0.04496050911855815,
+            -0.017711307900236353,
+            0.01934435542287117,
+            0.004335173027873153,
+            0.03800509969714242,
+            0.031198127357716314,
+            0.031311207941680276,
+            -0.012184553894588813,
+            -0.058985863670856876,
+            0.041442017057707635,
+            -0.012456189292721711,
+            -0.0040966351720394135,
+            0.010849179519959421,
+            0.005243123049347636,
+            0.0011495799963862647,
+            0.003534183676934487,
+            -0.03139340512003631,
+            -0.023876261883905306,
+            0.008382957569752911,
+            -0.004778283217011976,
+            0.03703067589888386,
+            0.06888630688839406,
+            -0.03993532296478411,
+            -0.009472167678222372,
+            0.025246872942105057,
+            -0.01524567912700368,
+            0.01595917939878582,
+            -0.003736723164442467,
+            -0.030836797784967753,
+            0.0026246446163368533,
+            0.008894300082148884,
+            0.08342856279154155,
+            -0.0007476580521704578,
+            0.025149116238550588,
+            0.004972542633631468,
+            0.034495632546556576,
+            -0.009173329764686424,
+            0.014009381543772768,
+            0.004427261668151028,
+            -0.013426138607902904,
+            -0.022234568880321808,
+            0.01976282364689966,
+            -0.03592433789490043,
+            -0.010918278220831188,
+            0.013527180289641174,
+            -0.013023113052242677,
+            -0.008304770302495363,
+            0.003677063799167178,
+            0.0013981822663219421,
+            0.047001163242600175,
+            0.01828332714887986,
+            0.002483375662343689,
+            -0.06820498187523893,
+            0.05295312038995555,
+            0.025038753504342162,
+            0.025286870198436735,
+            -0.0029544808216316115,
+            0.029982752196635428,
+            0.0005600481523874767,
+            0.0128290258663539,
+            0.009023805227424919,
+            -0.0019219166367750596,
+            -0.04482843936478531,
+            0.04523479691409549,
+            -0.004678574027241186,
+            0.010486850804218904,
+            0.006736229865018014,
+            0.02134813656476443,
+            0.028083972673690036,
+            0.016857413440125445,
+            0.0189082447820146,
+            0.03843690613820022,
+            -0.011922027011229613,
+            0.016193433524453224,
+            0.03828207911467065,
+            -0.0013055128404331705,
+            0.043198163443700956,
+            0.024025562893260542,
+            -0.001257851946749882,
+            -0.011726713433348844,
+            -0.021907984099443396,
+            0.02752198420597673,
+            0.004431866336999851,
+            0.015798806714413912,
+            -0.012910782191382367,
+            -0.03180549180206364,
+            0.004550247372715741,
+            -0.022778272841118514,
+            0.031242218240017532,
+            -0.006106984949693483,
+            -0.006259993190497421,
+            0.02187449047033339,
+            0.009334146847221406,
+            -0.009381278960016145,
+            -0.020888853823931134,
+            -0.007502374908776156,
+            -0.010390018386728482,
+            -0.04316693435221676,
+            -0.006274513730720055,
+            0.019152443937064258,
+            -0.01668984824845791,
+            0.018573194036505578,
+            0.0035035262183249925,
+            -0.019779911381789077,
+            -0.0027495798165085993,
+            0.02425481236315913,
+            0.006857723545270935,
+            -0.014733535699087731,
+            -0.026326146471941727,
+            -0.019906660231056833,
+            -0.001033294559612807,
+            0.0159154063026895,
+            0.035649534442431415,
+            -0.026210592096240003,
+            0.014397127229517285,
+            0.005114311681443673,
+            -0.0033104236005207077,
+            0.00324988235167305,
+            0.03764004154553922,
+            -0.017822437760737605,
+            -0.008579701063708334,
+            -0.00032410984200965487,
+            0.018450351071676443,
+            0.017283379764846195,
+            0.006384411611332382,
+            -0.005115090309123598,
+            -0.030590047062583125,
+            -0.011776211359193503,
+            0.01183623914060493,
+            0.0016564847796208331,
+            -0.009530766634361575,
+            -0.004810598258154945,
+            0.03876242515526699,
+            0.027076785774401667,
+            0.005324763136165989,
+            -0.03416412480913939,
+            -0.007943016733586697,
+            0.002114307650456928,
+            -0.02961449347300628,
+            0.05142732770518828,
+            0.05566026433842563,
+            -0.0030729309664974704,
+            -0.00938186948393074,
+            -0.04547908981236149,
+            0.0651216359692231,
+            -0.01813722445944094,
+            -0.0482650853560993,
+            0.044045155892545554,
+            -0.014272653003441165,
+            0.013740652306746851,
+            0.04563732439184395,
+            0.02126513340521316,
+            -0.0054845632439384905,
+            -0.011635625101262898,
+            0.030992082069685376,
+            -0.010051935721068779,
+            -0.050967258971854425,
+            -0.009827942095033494,
+            0.013373878556654464,
+            -0.013017627321534534,
+            -0.009361001542607812,
+            -0.01925867665425919,
+            0.020329052915436022,
+            0.023020741670660188,
+            0.019387043947695172,
+            0.005628632318152202,
+            0.018451555161952665,
+            0.016226845400848455,
+            -0.01384033112836216,
+            0.004762589697426251,
+            -0.012718476561725138,
+            -0.024222765127494463,
+            -0.037487144314010595,
+            -0.019971524166659278,
+            -0.015034588610516958,
+            0.009483936496215338,
+            -0.014301924283553591,
+            -0.0024139665139412125,
+            0.013361045353529458,
+            -0.0005870798721107379,
+            -0.011192319340879789,
+            0.013927695434391159,
+            0.021337539669342293,
+            -0.027300952495815437,
+            -0.04765670695199488,
+            -0.00561706224251562,
+            0.0082776639906139,
+            -0.0030984973749633724,
+            -0.026269016057688548,
+            0.007040524859505573,
+            0.0017859008072463442,
+            0.020091542517503808,
+            -0.018716761130867725,
+            0.0035283051960672963,
+            0.008519383548064053,
+            -0.015247417050336866,
+            -0.016917626868166634,
+            -0.018580550690103854,
+            -0.024697180580294884,
+            0.034202028316737125,
+            0.0204395003463227,
+            0.030392895534267717,
+            -0.04527669147753553,
+            -0.018359428092075348,
+            0.05494107499558624,
+            0.015234754907198422,
+            -0.05066008106884831,
+            0.031139824660661085,
+            0.005016615710206006,
+            0.0019000453624929544,
+            -0.006094364589229031,
+            0.0637324238017742,
+            -0.02711381489292883,
+            -0.02187418906973867,
+            0.004601744864532875,
+            -0.04141121277496072,
+            -0.026689585608019588,
+            -0.005568100750837504,
+            0.01291019036002459,
+            -0.031853041426355644,
+            -0.02815800259372657,
+            -0.007964861641790374,
+            -0.010933515363060793,
+            0.013726639077745378,
+            -0.023956800669647334,
+            -0.015672058590455337,
+            -0.023642025374517057,
+            -0.01288435863544158,
+            -0.0042516881552192385,
+            -0.011162214160460695,
+            -0.02412733256710621,
+            0.010088528028211243,
+            0.03310175458095482,
+            0.023064360903748363,
+            0.001996159684818467,
+            0.009478229572933101,
+            0.014478772877286076,
+            0.00371783700962415,
+            0.007864103610320116,
+            -0.048511515124609,
+            -0.031398754206534776,
+            0.020227872561659397,
+            0.012159811587094092,
+            0.0172163448580163,
+            0.009578952796095119,
+            -0.04154545961112031,
+            -0.05095080393682759,
+            0.02968486144855133,
+            -0.03932395023720674,
+            -0.02063219739061335,
+            0.007512423249865194,
+            -0.011911709468952548,
+            -0.00810951734088105,
+            -0.0008079327452253181,
+            -0.01600701450676789,
+            0.023984909975645632,
+            -0.02418072541081482,
+            0.0033984265728439398,
+            -0.0104343782811971,
+            0.01588977364765743,
+            0.05109529246167992,
+            -0.027800269136366997,
+            -0.027889034481533438,
+            0.03706813210665186,
+            -0.02058120652429308,
+            0.030801396468451574,
+            0.03246127847621613,
+            0.008355517405358826,
+            -0.008605467586247932,
+            0.017693344563968573,
+            0.014499487024471321,
+            0.012431277080478217,
+            -0.040103870071440256,
+            0.017328848567062113,
+            0.02723900253727976,
+            -0.054909074068976875,
+            -0.008547596270650966,
+            -0.00777964838164648,
+            0.027134666572242665,
+            0.02765036594970255,
+            -0.002207173084941375,
+            -0.016838740117653725,
+            -0.028289233739722936,
+            -0.019433376533829132,
+            -0.011053380458946764,
+            0.03247641283550457,
+            0.0007514989009289282,
+            0.007919574219000797,
+            0.001316042436488591,
+            -0.0017934342041762042,
+            0.01924145734197619,
+            0.015260199553572459,
+            0.048259602293134723,
+            -0.03383050622747848,
+            0.0021712225589426266,
+            -0.03508567715696004,
+            -0.04578731401250471,
+            -0.008768228935292693,
+            0.010203318819548585,
+            0.024135236371743855,
+            -0.007467140460133838,
+            0.0028907962385387818,
+            -0.007582083854422456,
+            -0.005384477718912037,
+            -0.01867978482547249,
+            0.01641191108381631,
+            0.037168089880543244,
+            0.0031992495663980464,
+            -0.008533890994695162,
+            0.000778843942291118,
+            -0.004620258538699103,
+            0.010572690335163528,
+            0.033233014627100904,
+            -0.030497362482602697,
+            0.02974571152880676,
+            -0.004274442492343003,
+            0.025341347125910928,
+            -0.04307748954707389,
+            -0.018069609916024403,
+            -0.018491122727274337,
+            -0.02069307802429807,
+            0.05372967226264594,
+            -0.030768978301976772,
+            -0.03662045886993282,
+            -0.003205991669059688,
+            0.016815511161004534,
+            -0.0014152644009276566,
+            -0.0016015439520090104,
+            -0.03984716970293867,
+            -0.010828260334550217,
+            -0.05894985435251326,
+            -0.01638035185618134,
+            -0.021819442096404948,
+            -0.005315102328745982,
+            -0.06251579337157112,
+            0.0033713876515840233,
+            -0.07815974854883057,
+            0.001484860071649606,
+            0.04539427129131382,
+            -0.023530943197353476,
+            0.05142250021536681,
+            -0.024746600229063383,
+            -0.0005885740455257141,
+            -0.044698536284421976,
+            0.0026481001040031785,
+            0.02389178603305878,
+            0.051387657872255674,
+            -0.002949198236157656,
+            0.010674377196609455,
+            0.013322075519401089,
+            -0.026245090721950647,
+            -0.01036724558050635,
+            -0.005321884180696542,
+            0.005958798671856245,
+            0.013755178085972732,
+            -0.0074734329537091675,
+            0.038368016046563674,
+            0.010941421833174982,
+            0.03443115781919206,
+            0.034623617136788874,
+            -0.004361926531008214,
+            -0.03348681614127998,
+            -0.00632870804659212,
+            0.024433331840566692,
+            0.0321998807702602,
+            -0.02899983280320611,
+            0.005958471722097962,
+            -0.015602447981516634,
+            0.015633217539747047,
+            0.006321931116948646,
+            -0.011827541320878442,
+            0.02372204564999124,
+            0.015673426811623124,
+            0.006103300725621424,
+            -0.005165006670545588,
+            0.013589526056189892,
+            -0.025411864292353578,
+            0.03041791947375232,
+            -0.004456865298584878,
+            -0.04276058054583262,
+            -0.0024736078733106825,
+            -0.020059309867945227,
+            0.024883721794000956,
+            0.015578237413840115,
+            -0.00536523508503184,
+            -0.005290645798641171,
+            0.008242987948511412,
+            0.020469675635821673,
+            0.006668354647245447,
+            0.00017729517736814364,
+            0.02373604785121342,
+            0.029915687122900485,
+            -0.02432635861789771,
+            -0.029540533943782886,
+            0.008151297444985344,
+            -0.005730733861939129,
+            0.011318816579801282,
+            -0.02644855229221747,
+            0.00033932872948409137,
+            0.0035505684154028996,
+            -0.03570104343808711,
+            0.03408366016010711,
+            -0.009199700348630527,
+            0.005857598530788056,
+            -0.009538177736472163,
+            -0.0017768806148210118,
+            0.0027410153099351694,
+            0.03898108970435115,
+            -0.007367623392751247,
+            0.009827254226889936,
+            0.015465895035342481,
+            0.000981604396724216,
+            0.05143083558797524,
+            -0.027350877485564314,
+            -0.02183036962301852,
+            0.01885468378168636,
+            -0.007285239007489837,
+            -0.016846837665812908,
+            -0.020546383007536074,
+            -0.011863943814684079,
+            -0.016293192750336218,
+            0.01976516340568883,
+            -0.02387328798681798,
+            -0.011892336568452852,
+            0.007170651391639504,
+            0.0013725014050721622,
+            -0.009809669068563384,
+            0.0016617183704652932,
+            -0.006307590349798603,
+            0.02037693835615797,
+            0.031753656615094446,
+            -0.024892675105483313,
+            0.03577919637821846,
+            0.03178303188315581,
+            0.017260898395251155,
+            -0.02102289690701718,
+            -0.009661035772073294,
+            -0.00048542689442823136,
+            -0.014694631967051945,
+            -0.0326379186158631,
+            0.025443206465204822,
+            0.017194626456982255,
+            -0.011381421003142453,
+            -0.03680783761649673,
+            0.0017216115811536721,
+            0.022819859205829785,
+            -0.013600989058799166,
+            0.028141657788496834,
+            -0.01869047298692401,
+            -0.01366833755869951,
+            -0.00782959271979894,
+            0.031227914441058244,
+            -0.05252392878125891,
+            0.016957842026348195,
+            0.00020880696574616416,
+            0.001154370575234767,
+            0.010892067708194324,
+            -0.008368819756628046,
+            0.012224564683325483,
+            0.031700389750709,
+            0.018063405869147564,
+            0.0010794115307202266,
+            -0.002783668216641561,
+            0.01418234057248712,
+            0.017773269840982066,
+            -0.015653443355569053,
+            -0.000497740485657824,
+            0.019890882604100452,
+            0.022672054781919238,
+            -0.00521797370595221,
+            -0.016686942830774276,
+            -0.018443538351123584,
+            0.025270057639201107,
+            -0.010354431070738671,
+            0.03484047753830186,
+            0.04594785214094975,
+            0.0036441959528283004,
+            -0.05695476763071743,
+            -0.009412209619190693,
+            0.0010371110229966086,
+            -0.01099343927053564,
+            0.01606863921935641,
+            0.012470660511773616,
+            -0.03237187923799263,
+            0.006388672213167589,
+            0.012946570716567364,
+            0.022152554488829627,
+            -0.02540317621005867,
+            -0.023183252921885226,
+            -0.0030984019855787655,
+            -0.007890467768631811,
+            0.018648760402624687,
+            -0.008653405980661855,
+            0.0006228315253506687,
+            0.009672526897422602,
+            0.0005436494014569835,
+            0.011632326145022935,
+            0.005655942007108228,
+            0.021736827172951,
+            -0.001256608167230203,
+            0.013124413886287987,
+            -0.01986808920858981,
+            -0.0270450339809516,
+            0.006500423802582728,
+            0.04438242119211055,
+            -0.014813399694814117,
+            0.030621709833336064,
+            0.0092171224268983,
+            0.016433384739642534,
+            0.024875072913970443,
+            -0.00891470467241686,
+            -0.014124009014664287,
+            0.00845777541223868,
+            0.009019007848927453,
+            -0.04147358041382793,
+            0.012513030150663913,
+            -0.01710243399313324,
+            -0.033144939719206276,
+            0.01904674924133127,
+            -0.00534122554342279,
+            0.012367320751926314,
+            0.004270315800911569,
+            -0.002886460711688153,
+            -0.025696286516581048,
+            0.014137901708875033,
+            0.028135087538950946,
+            0.021797869775098042,
+            0.0296173052503884,
+            -0.012757078368361444,
+            0.0005460601780756187,
+            -0.008748930817524899,
+            0.005470975782187157,
+            0.04187083128854477,
+            0.0008687182830616019,
+            -0.0008032802520818105,
+            0.01507726552487313,
+            0.024945744517309208,
+            0.0376130540370386,
+            -0.015415560911023553,
+            -0.03407353556457147,
+            0.01564969972264396,
+            -0.010104443707865262,
+            0.018855792875436674,
+            0.005503259932317811,
+            0.0049634097823351596,
+            0.0038600156268000702,
+            0.019312419743287703,
+            -0.006635193789316492,
+            0.018798045806429395,
+            -0.02500983683871332,
+            -0.028785757874583755,
+            0.01829411751138161,
+            -0.005277972443258226,
+            0.03338623299668019,
+            0.016225885095995672,
+            -0.029522280454919586,
+            -0.01712097344863975,
+            0.01553706187924322,
+            -0.008465766956343768,
+            -0.024842700926275538,
+            0.01895228828512998,
+            0.007076862934523977,
+            0.008956105593078965,
+            0.0024373779389833218,
+            -0.01106503696703573,
+            0.01125466547367417,
+            -0.000818223040067914,
+            -0.015322790791698395,
+            0.014157131555941659,
+            -0.017853386731281715,
+            -0.014994640991761465,
+            0.01775991780643474,
+            -0.037498078532902435,
+            0.009570197712419421,
+            -0.007687297543935123,
+            0.009050494195378254,
+            -0.004222041469820445,
+            0.007460383234429904,
+            -0.0010804379186060264,
+            -0.014987857934522812,
+            -0.014903749199810126,
+            0.03228485430019625,
+            0.022096121825475306,
+            -0.000410816936759558,
+            0.006728733918931589,
+            -3.971636461500232e-05,
+            -0.026719848032785824,
+            -0.011253751790407687,
+            0.02245467425830765,
+            -0.005664635254398481,
+            0.005134496095445943,
+            -0.01109590855846966,
+            -0.0149616794609307,
+            0.0019726017255571673,
+            0.032668637270108894,
+            0.06066635618119076,
+            0.012001732730659747,
+            0.024422093585231484,
+            0.0189499396761026,
+            0.02119276033547528,
+            -0.008489818917377516,
+            0.006413901834269739,
+            -0.03097413287789443,
+            0.03509225573986143,
+            0.005392594635384506,
+            0.03108937318306421,
+            0.026664331248813712,
+            -0.04722525921137681,
+            -0.0058539171830594774,
+            0.017087531454837677,
+            0.022673712717885323,
+            0.0159020681868943,
+            0.0008382498574190522,
+            -0.008938174268517353,
+            0.035650940044285984,
+            -0.03315250299348084,
+            -0.023647111263155,
+            0.005059382024092534,
+            -0.01439537467317451,
+            -0.010488359956731476,
+            -0.006785295580877329,
+            0.02776449681317373,
+            -0.005584052419598926,
+            -0.004582019433008779,
+            0.009697064252638535,
+            -0.014400206558190682,
+            0.008165676294736836,
+            0.0007624778531368436,
+            0.008681263897172326,
+            0.039249866981334126,
+            0.0005918063050375113,
+            0.024275699182767528,
+            -0.004138313072843555,
+            0.011079529739650594,
+            -0.0158972266904151,
+            -0.016985058543605,
+            0.02276982363172991,
+            0.00351101120496393,
+            -0.03244126813432563,
+            -0.00768819222024939,
+            0.00928180979884261,
+            -0.0013793404539641824,
+            0.005266058566444064,
+            -0.00503433670609399,
+            0.003747448261046857,
+            0.03286230666764075,
+            0.0022383592721685153,
+            0.008606190020480513,
+            0.013593947677257625,
+            0.00030951915677929204,
+            -0.01490307680740127,
+            0.013254591138113545,
+            -2.1379308136707722e-05,
+            -0.012900073369514622,
+            0.03357519186606058,
+            0.006705902094864628,
+            0.04243347200485736,
+            -0.03302953674711988,
+            0.009031534935013593,
+            0.01822227379789576,
+            -0.008491770335368761,
+            -0.02725412856390648,
+            -0.03630788717786838,
+            0.008259145747740535,
+            -0.019003260334340084,
+            0.015781275447792516,
+            -0.010039277287105153,
+            0.029665124310477384,
+            -0.03563370794521629,
+            0.004714858110268169,
+            0.004053913355055313,
+            0.0009503673086552651,
+            0.02068360255395961,
+            0.018524093113914366,
+            -0.002785698934204024,
+            -0.031443835341186485,
+            -0.02066365433385339,
+            0.006869321907853112,
+            0.02431260052207595,
+            -0.010367048093621498,
+            -0.005612149485859478,
+            0.0006316058139530722,
+            -0.024369562712568606,
+            -0.016845124984022828,
+            0.030359419843284823,
+            -0.040974427176153246,
+            0.005252911449822132,
+            0.04272936053727608,
+            -0.04599092715016919,
+            -0.03802288716943514,
+            -0.002314988728575789,
+            -0.024124014894379466,
+            -0.006360771882090754,
+            -0.002068030981218464,
+            0.016669991113175926,
+            0.0036232707799735537,
+            -0.028763764013482795,
+            0.00943401978066451,
+            -0.005173988601101547,
+            0.003971420424839974,
+            -0.0031669592494977393,
+            -0.02216336363004636,
+            -0.01657772032717298,
+            -0.02531962163186398,
+            -0.035174102848865116,
+            -0.0015903434876311088,
+            -0.005158703150821607,
+            -0.028703075279214722,
+            -0.03941940849540492,
+            -0.021696000925262537,
+            -0.0160221488026189,
+            -0.00843213135542359,
+            0.040641882367348774,
+            0.021319060666253945,
+            0.019336844555345886,
+            0.0025252167889382917,
+            -0.020346900609125973,
+            0.049440028814233025,
+            -0.007642182460707816,
+            0.035628691886108926,
+            -0.0017882888000877158,
+            -0.013461669667950893,
+            0.00455558389906464,
+            0.03825428026120858,
+            -0.014248434654950971,
+            -0.0009902062564027718,
+            0.00027811655483232545,
+            -0.0011866253757890136,
+            0.0008535155950871286,
+            0.015433650396777696,
+            0.001307651096343185,
+            -0.007112199543314309,
+            0.030380673552683906,
+            -0.012803247194895328,
+            0.01637501752192592,
+            0.018350814869962304,
+            -0.0013359719137099667,
+            0.008064546833360553,
+            -0.021430659268086814,
+            -0.00731572118780415,
+            0.0044578576331236325,
+            0.008422210247967642,
+            -0.015096885054883586,
+            -0.009964588397149773,
+            -0.02359494467299634,
+            -0.01174145797396426,
+            0.0023855749166744366,
+            -0.004559202896782688,
+            0.004902138778592298,
+            0.034595817216889944,
+            -0.016068744500427584,
+            -0.035106033294403614,
+            0.009997207702122002,
+            -0.02663088875167486,
+            0.02222306406130687,
+            3.983384942889326e-05,
+            0.006838221769859366,
+            -0.0023434342406307616,
+            0.001765445386886839,
+            -0.00169484278667461,
+            0.002862331185439823,
+            -0.018217975204395134,
+            0.0031859734246840257,
+            -0.0032119988065673565,
+            0.0009052950782312657,
+            -0.0012459653052875638,
+            0.00687339892509406,
+            0.006332799551164718,
+            -0.0008019538898592668,
+            -0.002968451740267742,
+            -8.541086355097327e-06,
+            -0.020561816931299912,
+            -0.021642512405352372,
+            -0.028259049261410905,
+            -0.000989887278779396,
+            -0.012359733017834338,
+            -0.027804394906433664,
+            0.011443385424045703,
+            -0.006872684999639775,
+            0.008327844771213648,
+            -0.008969527138034904,
+            0.006219994886600349,
+            -0.008415106130309062,
+            -0.0004143996206881767,
+            0.013119286511794273,
+            -0.02925209282917943,
+            -0.02096907906398536,
+            -0.03096513357712367,
+            0.01762223841790055,
+            -0.00925741670612824,
+            -0.002656382260603827,
+            0.0025223527960181835,
+            0.005092659166535706,
+            -0.033606542487570154,
+            -0.006653438998497412,
+            0.007313682256676618,
+            0.009547290751154595,
+            -0.03287788897629887,
+            0.02446728061355753,
+            0.017023222076526946,
+            0.01874233066565012,
+            -0.02417566312664911,
+            0.04049827067897005,
+            0.014426879325647042,
+            -0.021740606525480655,
+            0.016712907564723732,
+            0.027385150577780248,
+            0.03177163221499527,
+            -0.03469098547555545,
+            0.004481376248622418,
+            -0.01518200063132508,
+            0.018918680020516044,
+            -0.025284921244460287,
+            0.020625170445627613,
+            -0.016707629602243063,
+            0.021278719363178687,
+            0.03520334001264928,
+            -0.005524369131116028,
+            0.03608133014180745,
+            -0.019234548780647943,
+            0.0059985801736444555,
+            0.004041778461022664,
+            -0.01645503683802124,
+            0.01024694092614158,
+            -0.0042700351372843386,
+            0.0006059159967286885,
+            0.009406090340935747,
+            0.019133229156115282,
+            0.019007219188041775,
+            -0.04442208547053466,
+            0.028588965123434923,
+            0.00804120101594811,
+            0.021472086070876265,
+            0.014177820346835521,
+            -0.02708807970982664,
+            0.004742981711172693,
+            0.002072673190279628,
+            -0.004659713164250794,
+            0.0061167904730676514,
+            -0.013796604518930328,
+            -0.034540857808309226,
+            0.006338792319814448,
+            -0.018918703778187926,
+            0.015036937364271984,
+            -0.009026670935442384,
+            0.0011667525636770179,
+            -0.01152652811400482,
+            -0.011938679695598347,
+            0.0006170592505446954,
+            -0.005495151258143897,
+            -0.0008075736908949407,
+            -0.03171721592827769,
+            -0.04427658539412448,
+            0.02036132271908089,
+            -0.0021221965958423804,
+            -0.04320396743189005,
+            0.004111179168317842,
+            -0.025352428814134187,
+            -0.01865918186335911,
+            -0.011237699157960728,
+            -0.025047923887906474,
+            -0.04219630696740606,
+            -0.011108496740166623,
+            0.007785006066683012,
+            0.016810642610918525,
+            -0.009081815651299525,
+            0.040965574143584864,
+            0.0073944364393447965,
+            0.015313576815177263,
+            -0.017731360025694955,
+            0.026137777790243826,
+            -0.019314392060123,
+            -0.03108486330987073,
+            0.02676829566273746,
+            -6.259161089416372e-05,
+            0.003053748984540415,
+            0.014358356168574803,
+            2.8186924071913628e-05,
+            -0.006181789640967465,
+            0.01947730374003864,
+            -0.028739300217145797,
+            -0.023460503178221397,
+            0.0012460072421725688,
+            -0.008850395570726385,
+            -0.016271242375881602,
+            -0.035896509136022194,
+            0.005227881053956234,
+            0.03324337105109496,
+            0.004970419987430122,
+            0.008273497228454482,
+            0.01814500594735864,
+            -0.02229593571615308,
+            -0.024230675455380678,
+            0.000639112529814716,
+            -0.010266640429960994,
+            -0.010257247303470066,
+            -0.021302483296731352,
+            0.019150757066356872,
+            0.01651375496287087,
+            -0.00390189952706486,
+            -0.006971692709239087,
+            -0.010621003777733253,
+            -0.01918206725910585,
+            -0.02020345113211036,
+            -0.013827322378214754,
+            -0.017980206764311026,
+            -0.034153741666571154,
+            -0.032195001867229935,
+            0.03225205339201753,
+            0.026354817588919596,
+            0.0014076409241492998,
+            -0.007959081015805435,
+            -0.023070419006792155,
+            -0.004483328183581872,
+            0.02879516595345542,
+            0.002549934782150634,
+            -0.0001541849716285838,
+            -0.00830090759167373,
+            0.015341095409025125,
+            0.00835695019244165,
+            0.005107844984733648,
+            -0.06676974053142512,
+            -0.006027540973808174,
+            0.01655283585235268,
+            0.01351155740684989,
+            -0.002359114145814062,
+            0.02591301169926259,
+            0.01108099264396926,
+            -0.015361038608114896,
+            -0.019765743462262668,
+            0.035195761515422364,
+            0.02408910733949848,
+            0.023468767772473746,
+            -0.0034492999385544727,
+            -0.004472311490618405,
+            0.007030833258457338,
+            -0.010684252427867153,
+            -0.013326964647936429,
+            -0.004558513842504658,
+            0.026925146859337397,
+            0.016099443021514896,
+            -0.0064331198280825575,
+            0.010848931851939124,
+            -0.0008320209125061849,
+            -0.019444491995041537,
+            -0.001841330829156697,
+            0.008730275423516445,
+            0.0012974082030866743,
+            -0.014611902493400719,
+            -0.005911807768916134,
+            -0.019130604101278263,
+            0.024719468317569562,
+            0.0209456329723249,
+            0.03381727436368338,
+            -0.013237902591972769,
+            -0.0003695142331921278,
+            0.005410360755058108,
+            0.0002841632253252816,
+            0.0023352645111704958,
+            0.004801123870325242,
+            0.02623649813035263,
+            -0.029010710171522985,
+            -0.020741774066460415,
+            0.0068426980682519505,
+            0.013445270043414486,
+            0.0022744032984779294,
+            0.033840485942696216,
+            0.035040080489222324,
+            0.004166066423508316,
+            -0.010046666684988869,
+            -0.006148765564658713,
+            0.007071740742464309,
+            0.01865897335451031,
+            -0.023267411753900076,
+            -0.0005936657905496225,
+            0.004833030519903764,
+            -0.004229582648167378,
+            0.03620610659071198,
+            -0.009961238880412731,
+            -0.007197670122096852,
+            0.022212791947347127,
+            0.09819205436926473,
+            0.00047695621444943017,
+            0.0025354521225913395,
+            -0.029358210234742176,
+            -0.023614016211567963,
+            -0.0012564198366457442,
+            0.00242680708457405,
+            0.00830167274308547,
+            0.028076260517106013,
+            0.0004852400049405906,
+            0.011579274037419117,
+            0.02773972145320728,
+            -0.017217305534161797,
+            0.006752070301141707,
+            -0.011376051474975524,
+            0.024577071515205502,
+            -0.012694034026935636,
+            -0.0019809719792138673,
+            -0.016512649783050146,
+            0.015907402258634816,
+            0.004556574687833768,
+            0.004907510649166062,
+            0.011347630984408937,
+            0.009824484201337644,
+            -0.02650731900265866,
+            0.01756773113723491,
+            -0.005671986127099093,
+            0.004449611508541289,
+            -0.027841346572384462,
+            -0.020124233970502267,
+            0.0025986391930124847,
+            -0.0014440758308989809,
+            -0.009579444248630085,
+            0.006005274872617889,
+            -0.0024984235450999127,
+            0.03765664613203687,
+            -0.03765496603611941,
+            0.01409350051861866,
+            0.0066453175608372555,
+            -0.0015390841089599882,
+            0.011806653942640485,
+            0.017306914776093522,
+            -0.008206256939290151,
+            0.02589800448884443,
+            -0.00798213351353373,
+            0.009846274212000523,
+            0.015230070803055296,
+            -0.03449531102975745,
+            0.009174739184806312,
+            0.010145762060174144,
+            -0.024165660737765746,
+            0.012146877603301336,
+            -0.013784371016298891,
+            -0.004818759788054113,
+            0.005930446700613062,
+            0.007349837554764034,
+            0.010733731974181452,
+            -0.02295571464086674,
+            0.016176424826629898,
+            -0.010947841926210105,
+            0.0037766665765835445,
+            0.013559846998199008,
+            0.024892849808979323,
+            0.0012426907454129122,
+            0.0005950515136086193,
+            -0.01379469431504576,
+            0.005534724429523084,
+            -0.015667467221535825,
+            -0.022401586867694584,
+            -0.014192041775234079,
+            -0.006366846766730035,
+            0.01029598687444803,
+            -0.003972386708834858,
+            0.016971476088808737,
+            -0.010309648640697812,
+            -0.010645860303528099,
+            0.0037818310327362103,
+            0.01040806357838676,
+            0.008316605962234849,
+            -0.02855914079113881,
+            -0.005676310556150579,
+            0.004961084138163161,
+            0.00029285971503001265,
+            0.009592497954559435,
+            -0.014556073011576487,
+            0.013689526487160009,
+            -0.013290113354287095,
+            -0.0010726537498214612,
+            0.013201860397060236,
+            -0.012443552043930505,
+            -0.004012305744087929,
+            -0.01592905426965635,
+            -0.028487066798154617,
+            0.008462143660968396,
+            -0.01365614159523485,
+            0.001076882584445294,
+            0.0019503240125505646,
+            0.004730085646042709,
+            -0.016919069570642004,
+            -0.007860046623166481,
+            -0.033515448866521046,
+            0.011065416478687504,
+            0.02314589687882036,
+            -0.006435733825418645,
+            -0.030351727858789472,
+            -0.009546333526852862,
+            0.011327489786114963,
+            -0.0071765122743172715,
+            -0.01676736383051139,
+            -0.0029390038145944096,
+            0.007617910650439855,
+            -0.008174620754097493,
+            -0.00508943241317381,
+            -0.0025173969010414098,
+            0.0217328037301181,
+            0.016094975605203832,
+            0.01769642142712967,
+            -0.021400862603034036,
+            -0.00976992221747187,
+            0.026043879119776864,
+            -0.03275550610951898,
+            0.012441465975914529,
+            0.024587446266013183,
+            0.003446527051428522,
+            0.005492228054838714,
+            0.0030377554109693714,
+            0.005459159163772825,
+            -0.006985179226831048,
+            0.019925583798328282,
+            0.010870421035240845,
+            -0.019709062313471467,
+            0.005385443078244229,
+            0.02086085539375208,
+            0.0037248737640193097,
+            0.012892299156840998,
+            -0.017546870090541213,
+            -0.0020341807891543716,
+            0.015594779287017958,
+            0.04816214672670215,
+            -0.0032851797178091685,
+            -0.010450846098750195,
+            0.029513742590677183,
+            0.023526004023637855,
+            0.005373455940309873,
+            0.0012120777808635214,
+            -0.02525630456877349,
+            -0.013075130701774785,
+            0.01962235086017781,
+            0.016944870494393874,
+            0.013484925533584295,
+            -0.0023255656480479004,
+            -0.019236081983457366,
+            0.021520156296467717,
+            -0.01701733319731161,
+            0.012422659010697477,
+            -0.023764941037007197,
+            -0.001490790833263327,
+            -0.0035419008954878724,
+            0.021998949056452454,
+            0.0021819635638621092,
+            0.008891505304514552,
+            0.024354552127458608,
+            -0.006978548952551799,
+            -0.019526615985982355,
+            -0.0051799020400150305,
+            -0.014578953088901798,
+            0.022506635912517146,
+            -0.024117708614271965,
+            -0.007008641019892289,
+            0.011379795968360019,
+            -0.012362093847309038,
+            -0.0102534051092702,
+            0.001164934138876502,
+            0.017686807394319725,
+            0.0009423993435129515,
+            -0.014784508280754078,
+            0.01944445593807992,
+            -0.0011359271135469456,
+            0.010410915597949927,
+            -0.01921419010977189,
+            0.009390638453648025,
+            -0.016682442516992777,
+            -0.007849184048209236,
+            0.010343008873053859,
+            0.0012604521199837046,
+            0.003948300703645506,
+            0.002592490121754231,
+            0.007773437307969315,
+            -0.03355658426684504,
+            0.022695505487314516,
+            -0.010751606023585792,
+            -0.003538523688786008,
+            -0.0014125330167969218,
+            -0.01678692701672328,
+            -0.0006918162525881059,
+            0.005816502852598842,
+            0.010878358093847386,
+            -0.016045147059003256,
+            -0.00873270513444376,
+            -0.04920244395446931,
+            -0.030314009649638693,
+            0.023156333024481057,
+            0.013733657599720879,
+            -0.010971506342695331,
+            0.018852159415605117,
+            -0.023760853159532587,
+            -0.010587249404421732,
+            0.0011379869499042905,
+            0.023319856329056813,
+            -0.03539290744576506,
+            0.01215369703816749,
+            -0.014493682360886549,
+            -0.02039288669534806,
+            0.009070984596830142,
+            0.011445354735283254,
+            0.0007259852045513916,
+            -0.0011514627555568814,
+            -0.029350945123925453,
+            -0.010246845821727087,
+            0.02391068177425878,
+            -0.00861718589055971,
+            -0.004723044986826333,
+            0.024716520302409263,
+            0.0033721801437248873,
+            -0.012524663490691517,
+            -0.017955758068233754,
+            0.021612671655463236,
+            -0.026077203994996058,
+            0.007645944309444457,
+            0.010032183011253845,
+            -0.0173991263874487,
+            0.021649599230588117,
+            -0.007529701797485131,
+            0.020327763440441377,
+            0.004227390938169872,
+            -0.011249233530251522,
+            0.020352283974471443,
+            0.013644443758970054,
+            0.004216958800094986,
+            -0.004081669404784771,
+            0.006869353098014179,
+            0.02212732880780841,
+            -0.011021739614266063,
+            0.02219580717345023,
+            0.028777213295471067,
+            0.00017328191188631918,
+            0.03260179554743322,
+            -0.021287655520443285,
+            -0.03159728983004563,
+            0.021740163175540124,
+            -0.01035146858119169,
+            -0.021564906188712388,
+            0.028914245422221707,
+            -0.013674731836508768,
+            -0.00786476052825481,
+            -0.0166718429993922,
+            0.00533027311342322,
+            0.016680336311088843,
+            -0.005246190451159961,
+            -0.014360934066628332,
+            0.0029628103834216345,
+            0.019809487418628386,
+            0.014587234544007958,
+            0.029780009648869975,
+            -0.010778199815920887,
+            0.035180242185168895,
+            -0.002320576872571577,
+            0.008990157317473909,
+            0.012415465232073404,
+            0.006489435152972723,
+            -0.011251531176923445,
+            0.013501894212117511,
+            -0.003446759011266526,
+            0.008230548711588447,
+            -0.001449023474280409,
+            -0.035111679577202855,
+            -0.013709382199848678,
+            -0.04709294504969336,
+            -0.0023752976043235697,
+            -0.0025098360125255215,
+            0.0030975417740913154,
+            0.09736815869389318,
+            -0.005723049647039619,
+            -0.011988817571623457,
+            0.001625059666070763,
+            0.009611697638805556,
+            -0.0009896494420056814,
+            -0.007606836952841873,
+            0.015973152482488318,
+            -0.01343292635775299,
+            -0.02140928016038583,
+            -0.010532701190806147,
+            -0.01288594454368977,
+            0.012079198190522715,
+            0.015929716343161652,
+            -0.0005947359502228821,
+            -0.015207331663761012,
+            0.006524010655124294,
+            -0.004566385899752684,
+            0.0064717704997400245,
+            0.008647099975176308,
+            -0.010414668624343924,
+            -0.0071698879102925555,
+            0.030899143952119196,
+            -0.025301269549419674,
+            -0.024361748354052457,
+            -0.0010579931222861095,
+            -0.009055796737486008,
+            -0.010335459737012047,
+            -0.002692653633719985,
+            -0.0017679501322462056,
+            -0.010519994985692081,
+            -0.010019865263564572,
+            0.006537076627706128,
+            -0.015472130358273033,
+            0.004812857776601667,
+            -0.0047457011269495634,
+            -0.013296069496526007,
+            0.017054968086538724,
+            0.022563785357424853,
+            -0.02841491200807702,
+            0.0006755428946168761,
+            -0.017422510783648978,
+            0.012941211381006863,
+            0.008866069010589792,
+            0.006049618782357251,
+            -0.022451528209097505,
+            -0.023622906582403243,
+            -0.015019272754405791,
+            -0.01927535750568903,
+            -0.024973490129133732,
+            -0.009073381957684092,
+            0.0067692803507600945,
+            -0.001267339788153158,
+            0.021918281111033645,
+            0.000816538714279555,
+            -0.01812545613159305,
+            0.0061084913570894905,
+            0.000568371220952269,
+            0.018292497018668842,
+            -0.027468677288882186,
+            0.0033806422195878524,
+            -0.02307582511848465,
+            0.033508234091571124,
+            0.0039195050746594975,
+            -0.026680111514794824,
+            -0.021558668725755275,
+            -0.003311830287529699,
+            0.018994577319066473,
+            -0.0025745912019675805,
+            -0.00724049447539589,
+            0.014775207617625293,
+            0.0155334760323838,
+            -0.004693118176196483,
+            0.002015625406296403,
+            0.027968079002437102,
+            0.01077998866180723,
+            0.0313892866933904,
+            -0.015236169297590675,
+            0.004945828495617267,
+            -0.027215016713379062,
+            0.010214419950212732,
+            -0.01503548137805401,
+            0.008859554812646242,
+            -0.01290890682557532,
+            0.03531384210182567,
+            -0.012005613861257432,
+            -0.004250698950192301,
+            0.03253288796441852,
+            -0.029150986393345252,
+            -0.0035020051536906334,
+            -0.009255437891304041,
+            -0.004142204892160768,
+            0.013240580426083011,
+            -0.001198093874819309,
+            0.008485141356876356,
+            -0.01854370462655459,
+            9.726096864073609e-05,
+            -0.003221852304410318,
+            0.005404595442127598,
+            -0.02005458688172967,
+            -0.003314118809778734,
+            0.0073427568419969814,
+            0.004259518237708634,
+            0.002408630356035373,
+            0.016420646142725346,
+            0.006662121128241274,
+            -0.0029357701493526907,
+            0.03424224243607692,
+            0.008171895850772245,
+            0.0152521424877615,
+            -0.0018638264193202728,
+            0.0030234211261443034,
+            0.019287963516760636,
+            0.019912956661192856,
+            -0.00048443399729935116,
+            -0.01058935153883094,
+            0.00195113175754497,
+            -0.001140732137433033,
+            0.011048741602327437,
+            0.024396046054912645,
+            -0.021158025221644738,
+            -0.02419562092713724,
+            0.016377661005279716,
+            -0.015076145951740961,
+            0.01903712233378994,
+            0.010541463112889985,
+            -0.010099351302095792,
+            -0.00031558616108876247,
+            -0.042316016201379245,
+            0.010964989031040692,
+            -0.02068939391415103,
+            -0.04004120028915868,
+            -0.00244383573783919,
+            0.0009542507389088937,
+            -0.029439456243505052,
+            0.02822814855547978,
+            0.0023755032182188156,
+            -0.017280221740813364,
+            0.018223709137876153,
+            -0.005548654680726505,
+            -0.0024040301239229064,
+            0.005397711787238679,
+            -0.01092026090471121,
+            -0.010648485086769927,
+            0.008235250252660972,
+            0.010957715075632535,
+            0.005425683656535705,
+            -0.005698919020608095,
+            -0.011284911899894062,
+            0.009975882651638084,
+            0.012318901214538077,
+            -0.0019922884373083155,
+            0.010050520101815593,
+            0.0031720752428247446,
+            0.007352859750323051,
+            -0.0073483049386141005,
+            0.0046483587496626965,
+            -0.005093220128411125,
+            -0.026080057085901446,
+            0.012863580653903103,
+            0.0052434400313520774,
+            -0.011863360599637747,
+            0.0103209502958491,
+            -0.01048162756158923,
+            0.005787157766911021,
+            0.001318945460232246,
+            -0.0059333230422061395,
+            0.004821320238893602,
+            -0.0038922239931512784,
+            0.011267634571983924,
+            0.013256611037442042,
+            0.004592529900805719,
+            -0.013076954459595086,
+            -0.02177318635989856,
+            0.009001291224623467,
+            0.01959767163356408,
+            0.024161754192119852,
+            -0.030416109293488064,
+            0.003784634588325551,
+            -0.0014915849060829743,
+            -0.0020550564975080885,
+            -0.015632785947151753,
+            0.01587790981601866,
+            0.01716742056401084,
+            0.012721563202528083,
+            0.01893393006967701,
+            -0.004009617703771589,
+            0.010539100665843704,
+            -0.02080068888791658,
+            0.013642451766699067,
+            0.0008609869904066794,
+            -0.013629067231663643,
+            0.0014620655123815298,
+            0.027046304278766087,
+            -0.011056783825898616,
+            0.006206969699250696,
+            0.008097527175511791,
+            0.022111870736907183,
+            -0.012509479100313492,
+            -0.003735868296238069,
+            0.01516254968167132,
+            0.0014878674796313539,
+            0.0035887079417890352,
+            0.016533149527222273,
+            0.04729052276110659,
+            0.00102751186013391,
+            0.0016713773449738153,
+            0.0192177920520243,
+            -0.005563519063161466,
+            -0.01114821419109643,
+            -0.003549999863325982,
+            -0.021807893906469106,
+            0.010998558936481785,
+            0.00431279785115251,
+            -0.02186324212032519,
+            0.01024093366826206,
+            0.0125495473550584,
+            0.005268320135120931,
+            -0.0031679212410026603,
+            0.02574575247827441,
+            -0.019807879421996258,
+            0.014935295894965128,
+            0.03329450628384787,
+            0.00493356778223407,
+            -0.011423949333937285,
+            0.007263264249871384,
+            0.0027354459678742013,
+            -0.004494406041906684,
+            -0.00606745413660583,
+            0.010497391290734088,
+            -0.008191702034391382,
+            0.009870458191510176,
+            -0.009580130479693,
+            0.039202091799313755,
+            -0.009619018331262844,
+            -0.010782138248435458,
+            -0.004558645539551881,
+            -0.006241773108460662,
+            0.003478414182938609,
+            -0.003395165694642073,
+            0.005812225135633631,
+            -0.0069045407353789945,
+            -0.014607146220271209,
+            0.0008522982757210454,
+            0.00020175973815948303,
+            0.014001922919116725,
+            0.0011804904415274088,
+            -0.004871874532755353,
+            -0.007877217154706205,
+            0.015243277056007988,
+            -0.006047906208304912,
+            0.0006531937033317192,
+            -0.0011653523616313422,
+            -0.0013662556399359335,
+            -0.007401050012179761,
+            -0.015166105795085528,
+            0.00974609244984173,
+            0.016253733908242134,
+            -0.013857834499211056,
+            -0.005142936208363415,
+            0.012274716212832219,
+            0.008540229125193836,
+            0.011742606168187876,
+            0.0007451338650642946,
+            -0.009255813682572937,
+            0.008292531177374192,
+            -0.015020523035432158,
+            -0.008781317965985481,
+            0.0026250469418136603,
+            0.002381147382977664,
+            0.005839723981046825,
+            0.0006240641828668715,
+            0.004502463508605272,
+            0.005798912705218846,
+            -0.00497936644919367,
+            0.01112666823857836,
+            0.008198424171006548,
+            -0.002452143482105151,
+            -0.01017569410837814,
+            -0.001127589984272638,
+            -0.015733298167661346,
+            -0.016758315411028353,
+            -0.0009286266498936771,
+            0.015175988113750882,
+            -0.02352862898718829,
+            0.018996352975391644,
+            -0.0014524363611708201,
+            -0.0033802546171857704,
+            -0.021392428456030312,
+            -0.0025923983911570183,
+            0.0034214188668518796,
+            0.01255768230394476,
+            0.01858853630745928,
+            -0.02729543519465384,
+            0.010575894388563116,
+            -0.007204725245442141,
+            0.030394929424014717,
+            -0.018328782140005044,
+            -0.004022941273610227,
+            0.026915618079621992,
+            -0.02200018306828758,
+            0.0008791185377513978,
+            -0.011454606955158956,
+            -0.008103032693853504,
+            0.019287674837688177,
+            0.023747720480736777,
+            0.02038287242167783,
+            0.01364489294869211,
+            0.0007758044919789085,
+            0.004768030222445397,
+            0.03572705648994403,
+            0.010568241270862248,
+            0.014828087092859878,
+            -0.007542489523110974,
+            9.457808640264811e-05,
+            -0.02554033312739386,
+            0.01635899434369978,
+            2.587383534566272e-05,
+            -0.012370727506722627,
+            0.007162422186621379,
+            -0.013370882403931942,
+            -0.011066483600541058,
+            -0.007151077450730025,
+            -0.01721910018182534,
+            0.02152239849400241,
+            -0.012322440280680164,
+            0.004587729883596212,
+            -0.009079747696669277,
+            -0.013272707286450417,
+            -0.012953682633400515,
+            -0.014202214807187545,
+            -0.01313141988034687,
+            -0.026041403779538307,
+            -0.027109731260915763,
+            -0.006573285295895077,
+            -0.023780366346334524,
+            -0.013799072648505526,
+            0.005588622594136253,
+            -0.022617902666316757,
+            0.012262325666428604,
+            -0.012689827812482627,
+            -0.01226264872059864,
+            0.012805734807835008,
+            0.020274844315616616,
+            -0.011765231803328348,
+            -0.028494489703400308,
+            -0.018583843821043854,
+            -0.019970998470704358,
+            0.012576201137505969,
+            0.002266288538944273,
+            0.013165889075444961,
+            -0.003018437854660793,
+            0.003227412667880239,
+            -0.014895831590298957,
+            0.018570624675927183,
+            -0.03442390703780963,
+            -0.03108003806963536,
+            -0.013611306765348361,
+            0.005182777066476028,
+            -0.0015782033203578553,
+            -0.011429046504256945,
+            -0.0018606648572920801,
+            -0.00433805271182878,
+            -0.025109572024446088,
+            -0.010134856632190067,
+            0.011010662054484161,
+            -0.015205629927469414,
+            0.0072933565255675036,
+            0.0009018832564038863,
+            0.013870902834260887,
+            0.009587432944616442,
+            0.013393301893492882,
+            -0.006095063698384775,
+            0.004356485794113208,
+            -0.0026476612747324386,
+            0.0004894024149759108,
+            0.009336366919474478,
+            0.00839591561027103,
+            0.010551161369079399,
+            0.002260707777260338,
+            0.007830059223001696,
+            -0.0060927446216633265,
+            0.018065002687169148,
+            0.015436944875538481,
+            -0.0002520689442496089,
+            0.013323098485251487,
+            0.0003769862591044255,
+            0.00022293304921127018,
+            -0.0029970303106340466,
+            0.007888437018607987,
+            -0.011785957933642163,
+            -0.001311704406355858,
+            0.015138190893483804,
+            -0.01707247730569978,
+            -0.013045187198628258,
+            0.03107034781283638,
+            0.0018583638488268697,
+            -0.02495564203304796,
+            0.031146554085787762,
+            0.004220047610942943,
+            -0.005835199594223787,
+            -0.013205086624848782,
+            0.0014059047822222418,
+            -0.011570624844100298,
+            0.003584232809362274,
+            0.014295032061446826,
+            -0.02231984223610487,
+            0.010521094801839404,
+            0.008446892441796151,
+            -0.009681535919980559,
+            -0.012337659795586223,
+            0.009847073709209602,
+            0.016809328445200358,
+            0.004472485504034416,
+            -0.03539093810210523,
+            -0.011957694102753411,
+            0.02145470732008716,
+            -0.029237428225009913,
+            -0.00810041956259673,
+            0.007452962400105787,
+            0.0065154832040426065,
+            -0.019353109017609053,
+            -0.008580955020412362,
+            0.005725794605897509,
+            -0.004940390374637084,
+            -0.003892210172017758,
+            0.017541623128534682,
+            0.00485047043626186,
+            0.013698341792101025,
+            -0.01689738578307474,
+            0.009771458057400557,
+            -0.022222817828602557,
+            0.004021882074676753,
+            -0.023373700500461992,
+            -0.009873403167702603,
+            0.002364143609079849,
+            0.012454271200139762,
+            0.014328556467632212,
+            0.027839363143659326,
+            0.030056404238348527,
+            0.013995696971422429,
+            0.007833677585250953,
+            -0.016265199386953717,
+            -0.018134327237616175,
+            -0.006409513840075933,
+            -0.004033173350204784,
+            -0.008714798059218596,
+            0.0066373547199279235,
+            0.01858354212009877,
+            0.004499712802472377,
+            0.02609078206670711,
+            -0.006227189596272876,
+            0.009039744622014585,
+            -3.127526911951914e-05,
+            0.0008833930131827993,
+            -0.004226083827636182,
+            -0.01971685546584527,
+            0.00481003211084742,
+            -0.0013721482162825508,
+            0.025874758880591167,
+            0.00940496860916206,
+            0.0024915013399541847,
+            0.01615978314811047,
+            -0.006378952757397484,
+            -0.01174653607178726,
+            -0.010540107733238588,
+            -0.0061926268398085176,
+            -0.021752582127981104,
+            0.029059100503124387,
+            0.021195595568253612,
+            -0.010800765264123782,
+            -0.013693582166598082,
+            -0.00903369526059487,
+            0.02795767990556847,
+            0.004080965688836953,
+            -0.0013050212744006933,
+            0.016313311850308505,
+            0.012084555958999215,
+            0.012329325712491252,
+            0.009794698207689061,
+            -0.00036157148861933123,
+            0.0015364709122308352,
+            0.03722522141440073,
+            -0.007427028510548186,
+            0.0026463292682538495,
+            -0.005239324411651515,
+            0.0025987932868461313,
+            -0.009705817520422487,
+            0.007242122655628283,
+            -0.004012676385296197,
+            -0.0158760051470016,
+            0.015003444174308607,
+            -0.022443331305169994,
+            -0.0016749787204471238,
+            -0.008150635190952799,
+            0.0016696640640947303,
+            -0.02781023337840043,
+            -0.0025731206607371845,
+            0.006695065701630532,
+            -0.015547340828489069,
+            0.011816010501093885,
+            -0.01740673229051557,
+            -0.011553604323149105,
+            0.007914961618894248,
+            -0.012034157288065447,
+            0.0016684130271324446,
+            -0.0008268957621923278,
+            0.004460864332388434,
+            0.013426055079012884,
+            -0.011008808570221245,
+            0.012166046631826273,
+            -0.005816284310138776,
+            -0.007872091214834688,
+            -0.0016055129487028885,
+            -0.0030049472973942043,
+            -0.0025600746761171543,
+            -0.0009966231170649917,
+            0.034355910234323155,
+            -0.004446570896048543,
+            0.003761242147499512,
+            -0.01362639743248326,
+            0.0019833052855181806,
+            -0.036176376394999044,
+            0.028344464817309477,
+            -0.008642835889481802,
+            0.011013502221008498,
+            -0.0017508365480865596,
+            -0.013243706268954915,
+            -0.01456517309747086,
+            0.015247299501245178,
+            0.007602596572394442,
+            -0.005854531057349665,
+            0.013558248087347848,
+            -0.019724373614448298,
+            0.008530500822910953,
+            0.004463694543554244,
+            -0.015561397976987216,
+            0.004668780190328835,
+            -0.007479912658259757,
+            0.012657201279639194,
+            -0.01774730033362932,
+            0.004314594593150765,
+            0.00958571617187387,
+            -0.01892876084898881,
+            0.009901041800918215,
+            -0.028675879564512306,
+            -0.004631920242208613,
+            -0.012323455524366223,
+            0.0053804608827285865,
+            -0.000660864620708182,
+            -0.02389088252276507,
+            -0.028944280660230762,
+            -0.0002099667627277945,
+            0.00458533068247518,
+            0.0018878351997997833,
+            0.011621407953503755,
+            0.005970807015151215,
+            0.008542958629199003,
+            0.01023896054756221,
+            -0.00205207247588841,
+            0.009402752124876478,
+            0.04169706217414899,
+            -0.022899912321317197,
+            -0.011574970753549647,
+            0.005321872977369877,
+            0.011499271505460519,
+            -0.02647653483798284,
+            -0.011039791758347888,
+            0.0033424345263339144,
+            -0.005219877979550397,
+            0.03609469691834737,
+            -0.02448746405881972,
+            0.019020618652020673,
+            -0.011392297116547224,
+            0.007625475937968714,
+            -0.021465240076078376,
+            -0.003921088264697311,
+            -0.014265745991522477,
+            -0.00011221615798127438,
+            -0.0031478236827989703,
+            -0.0076053733863387755,
+            0.007485039021296698,
+            -0.001907678661303243,
+            0.0020348246022161925,
+            0.006291436821831794,
+            -0.02173305455485444,
+            -0.0019125499969493854,
+            -0.016432460924913166,
+            0.030367263349759286,
+            -0.016242095671439496,
+            0.005072805486919414,
+            0.0053258139728101506,
+            -0.002810639419737258,
+            0.012368771814660942,
+            0.004165967191536186,
+            0.00513754440091394,
+            -0.004199767241827414,
+            -0.005142642360546972,
+            -0.0053886448467479215,
+            0.017868874549760376,
+            -0.013280048598849825,
+            0.035024701466710065,
+            -0.0100959186045929,
+            -0.007094001877261568,
+            0.0019733042325232155,
+            0.010380430710282744,
+            4.2084202709296443e-05,
+            0.01053565670076933,
+            0.0034270489770133685,
+            -0.0006311078385359693,
+            0.0019774779328212756,
+            -0.017162448473652843,
+            0.003089758682621882,
+            0.01682656561693936,
+            0.005463677441664502,
+            0.004469261499451007,
+            -0.013332130891003062,
+            -0.012050233723282738,
+            0.004538268704442613,
+            -0.007533940408255447,
+            0.0003675737140813486,
+            -0.018816953644773076,
+            0.013972696748575243,
+            -0.0011096683686631858,
+            0.0002715683052236706,
+            0.00524678369365015,
+            0.023056514997908218,
+            0.01813167860994747,
+            -0.002553352602551003,
+            0.03155164978556312,
+            0.0174738879054593,
+            0.0029268039949260673,
+            0.010637295711777793,
+            -0.033222781195984666,
+            -0.013301873512972214,
+            -0.0013721703891642354,
+            0.02238421933149517,
+            -0.01530189703335162,
+            -0.014059029663525449,
+            -0.0037393534403749947,
+            0.0010394601933309767,
+            -0.005177359933828505,
+            0.0042489306011644495,
+            0.011039889511148341,
+            -0.014697595240068863,
+            0.007597149702441915,
+            0.0026780496822907634,
+            0.02092734730183164,
+            0.0004688144432243786,
+            0.017181410417231698,
+            -0.0005083970646221274,
+            0.00749639055342427,
+            0.007319420826337891,
+            0.017541463125355265,
+            -0.01002811389296805,
+            0.023582113845081534,
+            -0.008823933397729605,
+            0.0019889258902730535,
+            -0.01257106158422629,
+            -0.028159594067544737,
+            -0.012624750220149823,
+            0.015247564545279736,
+            0.005765911899842925,
+            0.006478299226263732,
+            -0.016537090297144265,
+            0.006276997440048981,
+            0.015647122171767974,
+            -0.002251683151059454,
+            0.010295358869874306,
+            0.009730378299068289,
+            0.00012816405890877098,
+            0.010181120688239766,
+            -0.00801256937783534,
+            -0.00734257767723293,
+            -0.004497520124843868,
+            0.023982754195729983,
+            -0.005443550092842393,
+            0.022434177122972028,
+            0.012647958478747348,
+            0.00616791857573897,
+            -0.010853355357954727,
+            -0.003340111254357573,
+            0.023389634440057473,
+            0.011863246790550376,
+            0.00029642699874398923,
+            -0.009846034060495678,
+            0.0019458876579524617,
+            0.0054010373765486005,
+            0.012983913038716568,
+            -0.02049427270240938,
+            -0.004344571352918878,
+            0.008461947538684664,
+            -0.008472919299762712,
+            0.004318309005722874,
+            -0.030156444522730416,
+            -0.005812303237222271,
+            0.021354352236942434,
+            0.008392281015122372,
+            -0.010232039722494646,
+            -0.004395682831498155,
+            -0.0006527587802367882,
+            0.0040601850269714945,
+            -0.0024323167053212176,
+            -0.0023729138545912176,
+            -0.008541233856441836,
+            -0.006470933419694742,
+            0.014035696617797647,
+            -0.0052691760528871924,
+            0.008597459268477037,
+            0.00012084859681905111,
+            -0.0017255958688825231,
+            -0.007323529232751411,
+            0.03382792341957058,
+            0.008087732655845718,
+            0.005980666974986827,
+            0.0013613625176327891,
+            0.0042244330945601215,
+            0.000616791811672998,
+            -0.0028374856646453707,
+            0.018075360591135542,
+            0.0156738939725117,
+            0.005054696911814988,
+            0.009038525980341447,
+            0.014672270369917403,
+            -0.019700405219230957,
+            0.012625337070556061,
+            0.012184185186281482,
+            0.019382321278525114,
+            0.0007219498737851579,
+            0.002564970959687783,
+            -0.010601873823737926,
+            -0.0029963537053764736,
+            -0.002070567089566093,
+            0.029563028293936092,
+            0.0005903976560069006,
+            -0.010509758568838935,
+            0.010992421945232951,
+            0.020603868018231348,
+            -0.02101157589451638,
+            -0.001971974816402967,
+            0.005963911670920596,
+            -0.010704207628592966,
+            -0.014466113540150018,
+            -0.021269176952661614,
+            -0.013331247081394942,
+            0.0071597345125287174,
+            -0.008162102923896894,
+            0.02367549407145414,
+            -0.014256771777502395,
+            -0.03960796112042958,
+            -0.014803626428839152,
+            -0.00864320347698291,
+            -0.027531337207277483,
+            -0.007305441146142331,
+            0.00965056093889974,
+            0.016698589600806273,
+            -0.003330867073501157,
+            -0.025198703196464594,
+            0.030483790406960656,
+            0.008872821868559751,
+            0.023708422767352917,
+            -0.021464374508407488,
+            -0.0018617153194555344,
+            -0.00886376784514331,
+            0.014505918224188001,
+            -0.017588761818663975,
+            -0.011756112651591795,
+            0.020014632992762026,
+            0.014356419813769714,
+            0.007246557500044434,
+            -0.004355132555323843,
+            0.0067299398462013936,
+            0.007649499235136303,
+            -0.012515181210363267,
+            0.012239792251154642,
+            -0.013499277313916029,
+            -0.024858295126228433,
+            0.014430561986516884,
+            0.002229494935179828,
+            0.009563004428064055,
+            -0.0015984171085659323,
+            -0.016197659608768088,
+            0.010078159490026602,
+            0.00808729175515505,
+            -0.017508840440512225,
+            -0.004793934415922065,
+            0.007133527615305415,
+            -0.020818869260580315,
+            0.031234686259160864,
+            0.011872765152279094,
+            -0.010692949770503467,
+            -0.001989039629661815,
+            0.004431079028880923,
+            0.016511544608975148,
+            0.007893700385963683,
+            -0.01997314388229276,
+            -0.003708006454566117,
+            0.01530128306203401,
+            0.00822588305477249,
+            0.004861703074728471,
+            -0.010836328783594584,
+            0.03434874763691911,
+            0.006082873924472328,
+            4.192481378506604e-05,
+            -0.0052997095110967115,
+            -0.0076108924981838355,
+            0.004196112457289409,
+            0.018905391806961686,
+            0.013330078288599763,
+            -0.004200590783114919,
+            -0.006630107527854746,
+            0.007778448128358027,
+            -0.01706991160178345,
+            0.007329056462125937,
+            0.009390327499494826,
+            -0.012870674959022891,
+            -0.0017571189976160295,
+            0.016462608870101713,
+            0.00209846661139665,
+            0.004942714850185376,
+            -0.011305050739130339,
+            0.004252776146827917,
+            0.015841821944109317,
+            0.018727567181082646,
+            -0.008399642221255733,
+            0.021448055799259867,
+            -0.020972459569228434,
+            0.027243639465535856,
+            0.028499400850980086,
+            0.016573144403275928,
+            -0.01847096443459798,
+            -0.0022409373140116146,
+            -0.0041737366139895414,
+            -0.014761716560267328,
+            0.0034546020161337894,
+            0.005125870074458691,
+            -0.01174186608185822,
+            -0.009577921232380895,
+            -0.024331993794416026,
+            0.011075959943215117,
+            0.007918147902741738,
+            -0.0015045185909419564,
+            -0.011791155919533844,
+            0.016019077577552835,
+            0.010141868950981925,
+            -0.010938402325619172,
+            -0.005635351352447256,
+            -0.019967361901734836,
+            0.009750427567651833,
+            0.009128294017175602,
+            -0.00300556760891084,
+            -0.011496103417643132,
+            0.012977890524465716,
+            0.0014080059843484115,
+            -0.01292380658817521,
+            0.010841954636959441,
+            -0.03142011651894961,
+            -0.005510537354374977,
+            -0.011027026734432255,
+            -0.003620482919145167,
+            -0.014415606143566519,
+            0.003608572766363589,
+            0.0005034611321129689,
+            -0.011470488351158329,
+            -0.0035005774696278,
+            -0.016207453351921627,
+            0.004676327282262429,
+            0.015021513761697039,
+            0.001679614645967629,
+            -0.006917375094634678,
+            0.021994274901326064,
+            0.018518524483306403,
+            -0.021903538417919984,
+            -0.020116013379669034,
+            0.004708084983787958,
+            -0.03261690161800251,
+            0.015019348187107588,
+            0.00966124471577622,
+            0.011676937880751252,
+            -0.028768229226408037,
+            0.007114534979415504,
+            0.00900420165933673,
+            0.016398927481511824,
+            0.0050879301160439245,
+            0.009453255486450082,
+            0.01222758288108498,
+            -0.0104657614414275,
+            -0.003164502778525433,
+            -5.410382753651127e-06,
+            -0.005618225986727351,
+            0.015607620507415484,
+            0.0008527697935056332,
+            0.03603666488542239,
+            -0.005277207843139309,
+            -0.0009799879252482288,
+            -0.011735731235560851,
+            0.00812046801749312,
+            -0.011713349784345638,
+            0.006334172347602515,
+            -0.0024643499727789167,
+            0.0001240009282616789,
+            0.002965067818077541,
+            0.009442572075213767,
+            -0.01086095783353607,
+            -0.007595379345224076,
+            0.007444897256325813,
+            -0.01482814340478338,
+            -0.00019984717819268013,
+            -0.009339343730980451,
+            0.0019680804106275156,
+            -0.02241031895876346,
+            0.006629838013824249,
+            0.040935644507527846,
+            0.02140216465404127,
+            -0.005481040712269146,
+            -0.012431659209559418,
+            -0.004680230194268222,
+            -0.007229300034633229,
+            -0.006482401275991039,
+            -0.004794698160419071,
+            -0.022762405726589375,
+            -0.006928553505112732,
+            0.004320321244630314,
+            0.013300138724212552,
+            0.009741018117644266,
+            -0.008842135469729511,
+            0.004531760574809748,
+            -0.02662650543210291,
+            -0.016578081411548973,
+            -0.0060021425889846495,
+            -0.015840130031005147,
+            0.006125301421471048,
+            0.027521841635711787,
+            -0.0063432924325983525,
+            0.01789356321527744,
+            0.005142029893259461,
+            0.013034833065163448,
+            -0.005338869351330258,
+            0.00035250243782976347,
+            -0.006010938630154189,
+            0.0032958656042048292,
+            0.003074861351445954,
+            -0.0015947217852680714,
+            0.004151590621941346,
+            -0.007139285822575166,
+            -0.0026091937868967284,
+            0.0011629380311555433,
+            0.0117841884590861,
+            0.006877419488576261,
+            -0.00016252653621991807,
+            -0.01761285866765538,
+            0.022221104524808997,
+            -0.018933508559881868,
+            -0.008840161102111973,
+            0.011092684029552064,
+            -0.009660487856185668,
+            0.00702597023411946,
+            0.020387386977378873,
+            -0.006854213562767843,
+            -0.009583981919303215,
+            0.013028181087649146,
+            -0.0036507672754997286,
+            0.004464368543493108,
+            0.009867428978560914,
+            0.006036303973414791,
+            0.002482944923100156,
+            0.013244650094892797,
+            0.004530294330688855,
+            -0.0032625482538717137,
+            -0.020238533752799943,
+            -0.005186498720507101,
+            0.019015752262933585,
+            -0.02252458872825473,
+            -0.0027641215988942563,
+            0.0044506144202026456,
+            -0.0030317840657635457,
+            -0.01733962766050954,
+            0.004205861489436576,
+            -0.018770275720867473,
+            -0.0033883519277721433,
+            -0.008244668812709337,
+            0.009948090119054079,
+            -0.023700732627507225,
+            0.009109315916173436,
+            -0.011739192106106307,
+            0.02356509175177699,
+            -0.003925851077921686,
+            0.024734433836185024,
+            0.0015142645677087268,
+            -0.013482933030628261,
+            0.0055323884713736554,
+            -0.0008930661843169134,
+            -0.004477731568313757,
+            0.029338490404300612,
+            -0.005418258701133154,
+            0.002249996261871466,
+            0.013491124734151694,
+            -0.021332541202417246,
+            0.021773687217263173,
+            0.0046662696076723836,
+            0.01628697326539432,
+            -0.00986238951830334,
+            -0.013799209313714441,
+            0.025178874553620743,
+            0.03324555035648896,
+            0.008398796308070151,
+            0.011261509865513953,
+            -0.01226890254177634,
+            0.018966175575083295,
+            -0.0038328997984309597,
+            0.004344837325292706,
+            0.00878668880458693,
+            -0.007200842513742962,
+            -0.011500562922303248,
+            -0.004533014670236755,
+            0.027639894892383943,
+            -0.021838848944317424,
+            -0.008920629621611768,
+            -0.005739181836252042,
+            -0.01593272996094732,
+            -0.038167534267554606,
+            -0.004371701247332816,
+            0.019604405563120094,
+            -0.005974358141737469,
+            -0.0018341151594086288,
+            0.014334266690716505,
+            0.00402620072221496,
+            -0.002852569009185304,
+            -0.008195218707838913,
+            -0.0027944153624309856,
+            -0.0059390492900516225,
+            0.016651206007022145,
+            -0.004961864528562493,
+            -0.007929830585906699,
+            0.01145431100211762,
+            -0.005731957927601062,
+            0.012130694936714224,
+            0.0004848418056781251,
+            8.392048424584131e-05,
+            -0.0015174120840537315,
+            0.00822932723013239,
+            0.018620869354789082,
+            -0.0055803454666417465,
+            0.025682778991091174,
+            0.006956936033581638,
+            -0.003495709501259274,
+            -0.005489431577052735,
+            -0.0044154293119170445,
+            -0.024561016942073285,
+            -0.0020864636079347357,
+            0.006593060072668424,
+            0.024333813203596122,
+            -0.02515989187381735,
+            -0.004953266000463339,
+            0.020957806164031402,
+            -0.0012192724005613,
+            -0.011978722479596745,
+            0.014690559986694138,
+            0.0041346025614047875,
+            0.009881584250499358,
+            0.012176393203318002,
+            0.015764530703333238,
+            -0.004923548157246672,
+            -0.0015192775000207833,
+            0.012766482928380375,
+            0.0005745973040874939,
+            0.00997809510696241,
+            0.016495444272181515,
+            0.010467514633877887,
+            -0.021108281220544853,
+            0.0015078843789125156,
+            0.00522593579521476,
+            -0.017481430635971396,
+            0.0034362821987740755,
+            0.005397819166037461,
+            0.011626527721895694,
+            0.004048632382026344,
+            0.013207335578216437,
+            -0.015768003434275563,
+            -0.005916168437120099,
+            0.008153798523776654,
+            0.004319451296835019,
+            -0.02599474886669325,
+            -0.01485009085106089,
+            -0.0006781964116146299,
+            -0.004444144921332929,
+            0.013556265879295017,
+            0.01285768162554067,
+            0.02033626001285354,
+            0.024637349990579514,
+            -0.013833060730799217,
+            -0.00842819657792762,
+            0.009436072455719896,
+            0.015560357465711765,
+            0.005798181411384387,
+            0.007874051721372273,
+            0.009297440071118126,
+            0.0077320823375042,
+            -0.004073209576661068,
+            -0.002973928843567532,
+            0.020205003097225747,
+            -0.019725382393637726,
+            0.0006746659838642147,
+            -0.018417443693472958,
+            0.007530796896254817,
+            0.01140696723506382,
+            0.01129515042745398,
+            0.002816883643872417,
+            -0.005274757308627299,
+            0.005716917047812153,
+            -0.06254973656832136,
+            -0.009191453047502128,
+            0.003384751551032943,
+            0.027256466708746898,
+            -0.010885336140129142,
+            -0.003415707910562931,
+            0.007309577090125959,
+            0.005511970708769456,
+            0.0005572239548625122,
+            0.003186427278034852,
+            0.019698716902282903,
+            -0.02052581021967601,
+            -0.010793968505805883,
+            0.004219269922138548,
+            0.0033111837923746746,
+            -0.007269521534951044,
+            0.004805737092302134,
+            0.014143978208501908,
+            0.01986656063456352,
+            -0.0024048854610842748,
+            0.01629335842915328,
+            -0.0015498603704589015,
+            -0.014496994444493684,
+            -0.002671064325909516,
+            0.0037875833047555925,
+            -0.022466433279066646,
+            -0.005980897597195858,
+            0.0043262697442044435,
+            -0.011226121648007667,
+            0.006978495397818936,
+            0.008452177557316863,
+            -0.003018545465557218,
+            -0.0008303307644324117,
+            0.017796789945838845,
+            0.006591364879905198,
+            -0.013464486184025026,
+            0.0028668426952326957,
+            -0.023377605222213174,
+            0.004325185794796361,
+            0.001416245546006914,
+            0.007809449561523174,
+            -0.008431335139468173,
+            -0.014774499666285603,
+            -0.005933810712528226,
+            0.017746847020610673,
+            -0.01965188730040385,
+            -0.002406875192894708,
+            0.01683009662984443,
+            -0.0002518220110313279,
+            0.005456812494894182,
+            -0.02011241622618092,
+            -0.0009782974741561381,
+            -0.02933296756078644,
+            -0.01171823509606186,
+            0.001162345317872531,
+            0.00892221099991475,
+            -0.006551308022163148,
+            -0.03934356275100932,
+            0.00427623835776202,
+            -0.013111664387776261,
+            -0.008752857644809231,
+            -0.010214546496134218,
+            -0.00518087722852466,
+            0.015240297918667867,
+            -0.012498100033235827,
+            0.01086490299272505,
+            -0.00917777212602538,
+            0.005895055740265625,
+            0.0090554651297836,
+            0.014212841798768113,
+            -0.005544047915369086,
+            -0.0033589999440643385,
+            -0.007924830667307956,
+            0.0022291797354234005,
+            -0.0032976226056924502,
+            0.009615674261248298,
+            -0.0051932155872883,
+            -0.0065636389655770455,
+            0.02426106498596339,
+            0.011445591067574932,
+            0.013312596099532533,
+            0.0055118624272110965,
+            -0.007220620072349783,
+            -0.01916551187613823,
+            -0.003977616240384473,
+            -0.013204506708246246,
+            -0.006315966975649096,
+            0.01434099854765068,
+            0.012588249778961223,
+            0.006357972880218866,
+            0.0007747002406996095,
+            -0.004794439624135275,
+            0.006975593520375694,
+            0.0262335496080837,
+            -0.016481909971520166,
+            -0.0013084662210086592,
+            -0.01369313900978409,
+            0.01921748474212319,
+            0.03333761672336437,
+            -0.0008918419419301244,
+            -0.007699525186523558,
+            0.013757216065348164,
+            0.00819045303047094,
+            -0.007411380297164911,
+            0.02453611080891859,
+            -0.025045462880463647,
+            -0.015989319900023054,
+            -0.011650886878708265,
+            -0.011407253985176972,
+            -0.008386367959227559,
+            0.016331961716602287,
+            -0.00141225438918211,
+            -0.0018505754298268654,
+            0.009557291617446645,
+            0.03357924917013024,
+            0.006916860901290756,
+            0.007030308953738567,
+            -0.002739345982346728,
+            -0.009376522438071562,
+            0.011482637968411055,
+            -0.007259054391879614,
+            -0.04075661435928125,
+            0.004250195164583401,
+            -0.005160505473755424,
+            0.02405623169060588,
+            -0.02197586561375289,
+            -0.00619175761282745,
+            -0.006736129049831973,
+            -0.028200015908139763,
+            -0.005867722104196303,
+            0.02805071895018145,
+            -0.015402198567775478,
+            0.014773294766214671,
+            -0.009502185346558072,
+            -0.0013419858580438616,
+            0.004443749010136947,
+            0.013008783570355656,
+            -0.02289662952366337,
+            0.0013834888740540108,
+            -0.00048549197205067054,
+            0.027969842651517314,
+            0.010437729090975158,
+            -0.0047705656249623744,
+            0.014202218020412936,
+            0.008240861154198411,
+            -0.018637315941614406,
+            0.007314019317343154,
+            -0.005747966765101792,
+            0.014833896739762147,
+            -0.01314847940891154,
+            0.007762100494499689,
+            0.00705026393499352,
+            -0.01278023669088385,
+            0.03016342404000438,
+            -0.015204129985039077,
+            0.007776946006562982,
+            0.009278778571285689,
+            0.009656723862179986,
+            0.01257686993982689,
+            0.004362439504889069,
+            -0.02898418193036562,
+            0.0002144431781454467,
+            -0.03828100004144633,
+            -0.00833748467467171,
+            -0.002611937107022567,
+            -0.002145329666219461,
+            0.007050630951707937,
+            -0.009383645162261629,
+            -0.033976442057557676,
+            0.0006429978367640709,
+            0.01603027598131823,
+            0.017501067413108286,
+            0.0002196997337451293,
+            0.013075379412332973,
+            -0.006197537539554383,
+            -0.021148120787551232,
+            -0.016986448660412497,
+            -0.010207705210051413,
+            -0.02122194744719993,
+            -0.020550470201191853,
+            -0.00908564461857315,
+            0.012090079127547193,
+            -0.009927390005879807,
+            0.013888458967497627,
+            0.029513250508042935,
+            -0.024529175134769883,
+            0.0011144447737101862,
+            0.021977677894839922,
+            -0.00198308986575084,
+            0.014108189311291326,
+            0.001696775006906642,
+            0.014122064251288995,
+            -0.0004166140484018266,
+            -0.010556016685519523,
+            -0.009729362492661684,
+            0.017487929296614076,
+            -0.0026043257634797124,
+            0.016715798948897194,
+            0.012183487338360088,
+            0.003771721868999568,
+            -0.0030887189945795796,
+            0.010667776697461199,
+            -0.026471157050402344,
+            0.014249996582064706,
+            -0.006690961709923224,
+            -0.0181382691440242,
+            -0.008729104191096461,
+            0.003882731567105777,
+            -0.005609384377642956,
+            -0.007794812949125115,
+            0.012967089730551055,
+            -0.00944374111323783,
+            0.02088525981049842,
+            0.0023400121055514958,
+            -0.012196925240774426,
+            0.003648688547842322,
+            -0.03322041209304968,
+            0.0015191765655963997,
+            -0.018402382776013956,
+            -0.024659631472836927,
+            0.03840173429666435,
+            0.011991452465253198,
+            0.006639702644606365,
+            -0.01675838115087676,
+            0.007953102302963526,
+            0.0004257479565099406,
+            0.017484338819046517,
+            0.011068864013925452,
+            0.006466915839538296,
+            0.008364454017976605,
+            -0.023316137291380285,
+            -0.010898421318668592,
+            0.016204569093730564,
+            0.012099473359791148,
+            -0.01017229369980751,
+            0.019431669546223253,
+            0.017955119269367787,
+            0.007300372071724306,
+            -0.006898619924677706,
+            0.01647927363640854,
+            0.021012103522831126,
+            0.010291016618557653,
+            -0.00047955552103474145,
+            0.0008684297441778696,
+            -0.010534247928997246,
+            0.005037930235889528,
+            0.035417626740307205,
+            0.014712595692093886,
+            0.010650306363963653,
+            -0.03626584475133921,
+            0.021764456453493054,
+            -0.011961521673661574,
+            0.009092981192874177,
+            0.006089104887183293,
+            0.017982801101846038,
+            0.006916969397519662,
+            0.014196254958707752,
+            0.011745069030657836,
+            -0.006689305450954901,
+            0.025205280281804708,
+            -0.001601414806100461,
+            -0.011992525142592405,
+            0.003386930324498371,
+            -0.01859193251438437,
+            0.005303261319773131,
+            -0.0041710536965100615,
+            -0.004459393120257307,
+            0.0032461828144552565,
+            -0.0018650956576455287,
+            0.015678225251799887,
+            0.003894142628084471,
+            -0.02355212701026675,
+            -0.0014873755115206368,
+            -0.024284295126348117,
+            0.0027260592623277375,
+            -0.022185571998195344,
+            0.00022883605560765203,
+            -0.005557568353382136,
+            -0.009749382813201403,
+            -0.025385857431333838,
+            -0.009853073898348951,
+            -0.0035843255959309307,
+            0.007661137888845632,
+            -0.010521037433807653,
+            -0.004246698084422667,
+            -0.0019920499321964815,
+            -0.019465113363548715,
+            0.009014255846983305,
+            0.027122533756959888,
+            0.016848221859975445,
+            0.027170883580103264,
+            -0.00019554112591107156,
+            0.013614675849485926,
+            -0.002066822813103284,
+            -0.01603063005944364,
+            0.008824663429777616,
+            0.0075653522381218495,
+            -0.00531780740880296,
+            0.014643348459785112,
+            0.0072634484937344704,
+            -0.012800149015319017,
+            0.009859503660384916,
+            -0.007161712176469609,
+            -0.006437857349742842,
+            -0.009764856421740622,
+            -0.02058868349990357,
+            -0.014866137971603676,
+            0.0012450207572602557,
+            -0.002123265406747006,
+            0.006000473862248991,
+            -0.004054016790850226,
+            0.009584879012937315,
+            -0.00608816807208866,
+            -0.00925801057439376,
+            -0.013962683072792106,
+            -0.008924356118158748,
+            -0.004210064145949253,
+            -0.004381709838058668,
+            -0.011732699976353612,
+            -0.005479523097469261,
+            0.0010680258077037758,
+            -0.0019401640391035755,
+            -0.01397652201595301,
+            0.011650839243603377,
+            0.0058731838291845645,
+            -0.002462743134993977,
+            0.02219008688925881,
+            0.01097316136828334,
+            0.0018083001685932406,
+            0.01366095798812196,
+            0.0003942601242216818,
+            -0.0064063532958756175,
+            -0.03343312148366622,
+            -0.0005083355836126291,
+            -0.02815586367527057,
+            0.025770078162165123,
+            -0.0069114949102666145,
+            -0.010322087105114614,
+            -0.002519047562374146,
+            -0.006063669644021698,
+            0.005873609611264377,
+            -0.002722940652283699,
+            0.0021229525800901114,
+            -0.023673778230768945,
+            0.0028629715379660113,
+            -0.003953653437464966,
+            -0.0036366957271968823,
+            0.007469664864329332,
+            0.0346520690389124,
+            -0.004054282838167587,
+            -0.0034765253405897676,
+            0.005159466707114532,
+            0.011642913651743502,
+            0.027845571220425726,
+            -0.004445196354520585,
+            -0.0032038508043339413,
+            -0.02233838584152591,
+            0.0054112576531374175,
+            -0.00019544838850471866,
+            -0.004733192299951088,
+            0.016664217348411003,
+            -0.016719175418727594,
+            -0.01631658832047864,
+            -0.00545969677717507,
+            -0.014570848053102338,
+            -0.00022620258237138677,
+            0.014454101124666536,
+            0.011056312437637014,
+            -0.004035526363810078,
+            -0.012459378363360557,
+            -0.0022595493558020706,
+            -0.030370952502579162,
+            0.012766010284884204,
+            0.1141689619126009,
+            -0.00766197047210299,
+            -0.00896655658602649,
+            -0.009481112393590025,
+            0.001435269335646347,
+            0.001338755472877713,
+            -0.011739143028767118,
+            0.07166096084053437,
+            -0.0077855767070652395,
+            0.0002139580430342728,
+            -0.011312198831181402,
+            0.001844005266872301,
+            -0.001825412693163496,
+            0.01371932421024605,
+            -0.005025814503687022,
+            -0.018213360991353643,
+            -0.0033612481616769416,
+            -0.02941403830641238,
+            -0.0041137865872933515,
+            -0.005683630256743215,
+            -0.004288198570156578,
+            -0.014982999363445122,
+            0.0067149665791261335,
+            -0.019661973497372313,
+            -0.012688556129040908,
+            0.001124391650175233,
+            -0.0033623453063430534,
+            0.00996944383866679,
+            -0.020496336814603385,
+            -0.0044354321069609536,
+            0.0019875840782527733,
+            0.009706675240563883,
+            0.0011600364428219214,
+            -0.012755178230821602,
+            -0.0026514442584739412,
+            0.004056350166031111,
+            0.01909951327913368,
+            0.014360281326789544,
+            0.003732344102045803,
+            0.008623457695718206,
+            0.014461868085333023,
+            -0.001692149528228451,
+            0.01948035376572433,
+            0.01172550239477637,
+            0.0031966531144076173,
+            0.0004779413449255626,
+            0.0030912562273611517,
+            -0.004597633235071031,
+            0.01582547400214653,
+            -0.014811177637844808,
+            -0.02411202772171917,
+            -0.0071272238837422185,
+            0.018015541137524113,
+            -0.013081287852028074,
+            1.8837689271256874e-05,
+            0.004062803284646432,
+            0.027569726868200254,
+            0.0010564850664255057,
+            -0.01179755720247972,
+            0.0028876134800690334,
+            -0.01638215925131053,
+            0.022944183106912663,
+            0.0026043713064243836,
+            0.03279173899194346,
+            -0.005103903273968361,
+            0.00578213669463383,
+            -0.020744194169889544,
+            -0.007588793365216579,
+            -0.00819862611489168,
+            0.0017401442433921013,
+            0.010993500174794084,
+            0.005411238350013134,
+            0.01620925462604661,
+            -0.00641615069699833,
+            -0.020751730783437995,
+            0.014776878596707538,
+            -0.009552193304486048,
+            -0.03393319554458277,
+            0.008881987295131347,
+            -0.029102297123590998,
+            -0.0045267103602711,
+            0.00368971742585232,
+            -0.014269341759285646,
+            -0.016264809810970175,
+            0.012837302095591106,
+            -0.0011890334180728564,
+            -0.008937992768323412,
+            0.0014444613725248846,
+            -0.017110187133760626,
+            -0.02385668037818897,
+            0.006017405364919834,
+            0.00676670672597817,
+            -0.0077994500447595,
+            0.018312452858615755,
+            -0.027015660037024585,
+            0.006123912473734307,
+            -0.012973676426738582,
+            0.009041974762537394,
+            -0.01580183887345168,
+            0.01163516332546347,
+            0.010887662470499361,
+            -0.001503467664709027,
+            -0.0076234124970223326,
+            0.01231202541338883,
+            0.004852720134894811,
+            0.033760365210093034,
+            0.028658326381800395,
+            0.000995999783627898,
+            -0.0025605367058226737,
+            -0.0020356794893366384,
+            0.011667525096213252,
+            -0.014049700287539817,
+            0.011218246246280464,
+            0.02658113044439717,
+            0.017568631325838108,
+            -0.019133062654008885,
+            0.018050678772708717,
+            -0.010356118256625172,
+            0.012872052420999813,
+            -0.025620154415048252,
+            -0.005135768485044442,
+            0.004936490920616527,
+            0.005163008729256001,
+            0.007425278892536464,
+            0.001410212306439329,
+            0.021891387289347516,
+            0.014999616600766989,
+            0.00027613811156650413,
+            0.00558422822386443,
+            0.01477009526981092,
+            0.010497782599293994,
+            -0.0005567894749233204,
+            -0.00467776506604011,
+            -0.016998035358377343,
+            0.00957271797745408,
+            -0.006065116857898469,
+            0.005138070401846389,
+            0.01698914855850691,
+            -0.01230257809343985,
+            0.011109844802663414,
+            0.007518553660463051,
+            -0.015504773927595823,
+            -0.017169933945839305,
+            0.005552829003043926,
+            -0.02252633790984703,
+            -0.01306722099657118,
+            0.02715689072730398,
+            -0.010513941399990838,
+            -0.02790582732974509,
+            -0.0009847768069205477,
+            0.009797865241108237,
+            -0.002772952073769185,
+            -0.0027968741526000576,
+            0.0037541733651479613,
+            0.010801137722284016,
+            -0.014658156970099418,
+            0.002075415233345957,
+            0.004735264309137368,
+            0.01930804315461151,
+            -0.0014151522239496115,
+            -0.015046118414100617,
+            -0.002288495894227231,
+            0.021592829677937263,
+            0.017174609713578173,
+            0.00265560378681856,
+            -0.016499825201963983,
+            -3.965296928410772e-05,
+            -0.0074142747014567635,
+            0.00021360696629503726,
+            0.00027431174904249525,
+            -0.0056391089282546054,
+            -0.018931681700770225,
+            -0.003888248121485032,
+            0.011181675419027736,
+            -0.00247252920175839,
+            0.01977860312223846,
+            -0.0011527575128223392,
+            -0.015763587433200915,
+            0.001960734212846797,
+            -0.005280860341480037,
+            -0.010372829570478649,
+            0.01351900743232294,
+            -0.009359942374603997,
+            0.006788746813046239,
+            -0.003212015225284119,
+            0.004611143085183367,
+            -0.0003308099698088422,
+            -0.017017265064185255,
+            0.02924142709831441,
+            -0.0018238549856433466,
+            -0.004769274041546295,
+            -0.0014058363976586562,
+            -0.033393331166741945,
+            0.015017165991303838,
+            -0.014574579250749552,
+            0.006745699409061803,
+            0.002705853802154516,
+            0.01272741296341735,
+            0.003253095246601883,
+            -0.008898616461779179,
+            0.0027485606928249326,
+            0.017040360927723885,
+            0.013756402798365262,
+            -0.007292864293090161,
+            0.008694855886257706,
+            -0.0028934389922887676,
+            -0.0037483208383542015,
+            0.01363963339139356,
+            0.004170611039133969,
+            -0.006282657671540625,
+            -0.0009784372731140945
+          ],
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "Mixture-of-Experts (MoE)",
+            "fine-grained expert segmentation",
+            "shared expert isolation"
+          ],
+          "title": "Deepseekmoe: Towards Ultimate Expert Specialization In Mixture-Of-Experts Language Models"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "95f27e84-0fb9-44ee-a783-45f42551401c",
+        "properties": {
+          "page_content": "5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n",
+          "title": "Scaling Up To Deepseekmoe 16B",
+          "keyphrases": [
+            "DeepSeekMoE 16B",
+            "MoE model",
+            "language modeling",
+            "evaluation benchmarks",
+            "Open LLM Leaderboard"
+          ],
+          "embedding": [
+            0.005968157202005386,
+            -0.015490485355257988,
+            -0.027613861486315727,
+            0.012274806387722492,
+            0.008261889219284058,
+            -0.01577553153038025,
+            -0.016158562153577805,
+            0.06238951161503792,
+            -0.002393943490460515,
+            0.048564765602350235,
+            0.008756265975534916,
+            -0.009914266876876354,
+            0.03532792627811432,
+            -0.018332039937376976,
+            -0.006921280641108751,
+            0.023961706086993217,
+            0.018617086112499237,
+            0.02068367227911949,
+            -0.00015922509192023426,
+            -0.02930632419884205,
+            -0.021538810804486275,
+            -0.03865940496325493,
+            -0.037447959184646606,
+            -0.023801365867257118,
+            -0.021129056811332703,
+            0.012862714938819408,
+            0.01521434634923935,
+            0.015909146517515182,
+            -0.041616760194301605,
+            0.008319789543747902,
+            0.016042763367295265,
+            0.021449735388159752,
+            0.004554060287773609,
+            0.009063581936061382,
+            0.012675653211772442,
+            -0.015294515527784824,
+            0.07097653299570084,
+            -0.0016189739108085632,
+            -0.009361989796161652,
+            0.010226036421954632,
+            0.018777426332235336,
+            0.010083513334393501,
+            -0.009317451156675816,
+            -0.010787221603095531,
+            0.019899794831871986,
+            0.012176821939647198,
+            -0.00043619878124445677,
+            -0.0028460093308240175,
+            0.00021837776876054704,
+            -0.021592257544398308,
+            -0.027720753103494644,
+            -0.013682222925126553,
+            0.0055272262543439865,
+            -0.006075049284845591,
+            -0.017797578126192093,
+            -0.005594033747911453,
+            0.03819620609283447,
+            0.0277029387652874,
+            0.03762611374258995,
+            -0.01240842230618,
+            0.03308318555355072,
+            -0.017494717612862587,
+            -0.006288834381848574,
+            -2.058165227936115e-05,
+            -0.006248749326914549,
+            -0.034597497433423996,
+            -0.005117471795529127,
+            0.001308317994698882,
+            0.005865718703716993,
+            -0.024068597704172134,
+            -0.04464537650346756,
+            0.02754260040819645,
+            -0.003171140095219016,
+            0.00048101565334945917,
+            0.02171696536242962,
+            0.0010444275103509426,
+            0.01798463985323906,
+            0.007918942719697952,
+            -0.02321345917880535,
+            -0.014279038645327091,
+            0.020505519583821297,
+            0.010386374779045582,
+            -0.006529341917484999,
+            0.025565091520547867,
+            0.06046544760465622,
+            -0.01011914387345314,
+            -0.024959366768598557,
+            -0.08444496989250183,
+            7.453237049048766e-05,
+            -0.0283442921936512,
+            -0.011490928940474987,
+            0.007825411856174469,
+            -0.004527336917817593,
+            -0.019062472507357597,
+            -0.0022169030271470547,
+            0.025547275319695473,
+            -0.005602941382676363,
+            -0.006342280190438032,
+            -0.017957916483283043,
+            0.03791115805506706,
+            -0.002776974579319358,
+            -0.004046321380883455,
+            0.030054569244384766,
+            0.0013561969390138984,
+            0.016888992860913277,
+            -0.0027858822140842676,
+            -0.02565416693687439,
+            0.029965493828058243,
+            -0.0067653958685696125,
+            0.08929075300693512,
+            -0.014564084820449352,
+            -0.009691574610769749,
+            -0.050453197211027145,
+            -0.011063359677791595,
+            -0.014083068817853928,
+            -0.003999555949121714,
+            -0.03424118831753731,
+            0.001817170181311667,
+            0.011588914319872856,
+            -0.014368114992976189,
+            0.013682222925126553,
+            0.021859489381313324,
+            0.03887319192290306,
+            -0.026990322396159172,
+            -0.04343393072485924,
+            0.06506182253360748,
+            0.021503180265426636,
+            -0.014804592356085777,
+            0.013824746012687683,
+            0.003099878551438451,
+            -0.01950785703957081,
+            0.011366221122443676,
+            -0.022393951192498207,
+            -0.013334822840988636,
+            0.017254209145903587,
+            -0.00778978131711483,
+            0.026152998208999634,
+            0.0797417014837265,
+            -0.018332039937376976,
+            0.014973838813602924,
+            0.022108905017375946,
+            -0.029217246919870377,
+            0.012372790835797787,
+            0.0028148323763161898,
+            -0.034989435225725174,
+            0.0058924416080117226,
+            0.008894335478544235,
+            0.05825633928179741,
+            0.003538582706823945,
+            0.010929744690656662,
+            0.010974282398819923,
+            0.011098990216851234,
+            -0.0009859707206487656,
+            0.023997336626052856,
+            0.008497943170368671,
+            0.0018372124759480357,
+            -0.026527121663093567,
+            0.01851019449532032,
+            -0.018813056871294975,
+            0.00869391206651926,
+            0.012167913839221,
+            -0.022661181166768074,
+            -0.012417329475283623,
+            -0.002204654971137643,
+            0.004934864118695259,
+            0.05184279754757881,
+            0.020024502649903297,
+            -0.010555621236562729,
+            -0.0979846715927124,
+            0.050132520496845245,
+            0.014145422726869583,
+            0.024923736229538918,
+            -0.04197306931018829,
+            0.017476901412010193,
+            0.009647035971283913,
+            0.028878754004836082,
+            0.0057677337899804115,
+            0.0020532242488116026,
+            -0.06816169619560242,
+            0.03545263409614563,
+            0.011419667862355709,
+            0.024994997307658195,
+            0.02485247515141964,
+            0.008840889669954777,
+            0.014065253548324108,
+            0.0004027949180454016,
+            0.04058346897363663,
+            0.029573554173111916,
+            -0.005389156751334667,
+            0.0357198640704155,
+            0.036236513406038284,
+            -0.0020799473859369755,
+            0.04428907111287117,
+            0.003500724909827113,
+            0.015089638531208038,
+            -0.006609511096030474,
+            -0.027328815311193466,
+            0.04963368922472,
+            -0.01895557902753353,
+            0.024157674983143806,
+            0.012069929391145706,
+            -0.03119475580751896,
+            -0.011125713586807251,
+            -0.01026166696101427,
+            0.028611524030566216,
+            -0.01782430149614811,
+            0.0019407644867897034,
+            0.02168133482336998,
+            0.009451067075133324,
+            -0.008163904771208763,
+            -0.026919061318039894,
+            -0.0066406880505383015,
+            -0.028967831283807755,
+            -0.04175928235054016,
+            0.0026010475121438503,
+            0.02702595293521881,
+            -0.012755822390317917,
+            -0.00048185072955675423,
+            -0.010742682963609695,
+            -0.030820632353425026,
+            -0.009932082146406174,
+            0.03388487920165062,
+            0.007901127450168133,
+            -0.011802698485553265,
+            -0.02971607819199562,
+            -0.014403746463358402,
+            0.01066251378506422,
+            -0.0017403413075953722,
+            0.05105891823768616,
+            -0.01533905416727066,
+            0.020648041740059853,
+            0.011437483131885529,
+            0.0065605188719928265,
+            0.02424675226211548,
+            0.02437146008014679,
+            -0.016559408977627754,
+            0.005843449383974075,
+            -0.003349294187501073,
+            -0.0055183181539177895,
+            0.043505191802978516,
+            -0.003075382439419627,
+            -0.029128169640898705,
+            -0.031586695462465286,
+            -0.012203545309603214,
+            0.010181497782468796,
+            -0.005081841256469488,
+            -0.01433248445391655,
+            0.02123595029115677,
+            0.02882530726492405,
+            0.04842223972082138,
+            0.01276472955942154,
+            -0.05725867673754692,
+            -0.013896007090806961,
+            0.0020732665434479713,
+            -0.02260773442685604,
+            0.032424017786979675,
+            0.060322925448417664,
+            -0.00506847957149148,
+            -0.020612411201000214,
+            -0.027150660753250122,
+            0.0659882202744484,
+            0.012328253127634525,
+            -0.04464537650346756,
+            0.04938427358865738,
+            -0.013655499555170536,
+            0.022340504452586174,
+            0.04517984017729759,
+            0.021859489381313324,
+            -0.0304286926984787,
+            0.017735224217176437,
+            0.024157674983143806,
+            -0.004836879670619965,
+            -0.06196194142103195,
+            -0.02260773442685604,
+            -0.002491928171366453,
+            -0.002779201604425907,
+            -0.013397176750004292,
+            -0.024585243314504623,
+            0.02485247515141964,
+            0.014537361450493336,
+            0.019098103046417236,
+            0.02120031975209713,
+            0.02324908971786499,
+            0.0164970550686121,
+            0.005259995348751545,
+            -0.007331034634262323,
+            -0.005153102800250053,
+            -0.023355981335043907,
+            -0.0269012451171875,
+            -0.02184167318046093,
+            -0.01982853375375271,
+            0.017574885860085487,
+            -0.04371897876262665,
+            0.016568316146731377,
+            0.0024251204449683428,
+            0.024389274418354034,
+            -0.023712290450930595,
+            0.004099767655134201,
+            0.028682785108685493,
+            -0.010591251775622368,
+            -0.04079725220799446,
+            0.025084074586629868,
+            0.011677990667521954,
+            -0.007241957820951939,
+            -0.007504734676331282,
+            0.0006981407641433179,
+            -0.008867612108588219,
+            0.031622324138879776,
+            -0.025280045345425606,
+            -0.01902684196829796,
+            -0.005375795066356659,
+            -0.02011357992887497,
+            0.0058924416080117226,
+            -0.03436589613556862,
+            -0.007464650087058544,
+            0.048600394278764725,
+            0.02132502757012844,
+            0.015098546631634235,
+            -0.04240063950419426,
+            -0.006631780415773392,
+            0.06142747774720192,
+            0.015392500907182693,
+            -0.07354194670915604,
+            0.044787902384996414,
+            0.0010861823102459311,
+            0.006698588375002146,
+            0.004685448482632637,
+            0.030054569244384766,
+            -0.015454854816198349,
+            -0.0159358698874712,
+            0.007673981133848429,
+            -0.030250540003180504,
+            -0.022091088816523552,
+            -0.005175372119992971,
+            0.021449735388159752,
+            -0.04521546885371208,
+            -0.03887319192290306,
+            -0.016791008412837982,
+            -0.0053802491165697575,
+            0.017129501327872276,
+            -0.011758159846067429,
+            -0.024923736229538918,
+            -0.030054569244384766,
+            -0.0005670305690728128,
+            -0.00188175099901855,
+            -0.04101103916764259,
+            -0.03067810833454132,
+            0.0068455650471150875,
+            0.021930750459432602,
+            0.02886093780398369,
+            0.0017392279114574194,
+            0.009459974244236946,
+            0.02504844404757023,
+            -0.00419552531093359,
+            0.029520107433199883,
+            -0.02766730636358261,
+            -0.032584354281425476,
+            0.012782545760273933,
+            -0.010911929421126842,
+            0.02116468921303749,
+            0.001672420185059309,
+            -0.021022165194153786,
+            -0.04689011722803116,
+            0.02706158347427845,
+            -0.0457855649292469,
+            -0.012087744660675526,
+            0.010974282398819923,
+            -0.0022937320172786713,
+            0.010288390330970287,
+            -0.00252533215098083,
+            0.0066451421007514,
+            0.02526222914457321,
+            -0.017601609230041504,
+            0.011909591034054756,
+            0.008012473583221436,
+            0.006306649651378393,
+            0.033991772681474686,
+            -0.008444496430456638,
+            -0.020505519583821297,
+            0.026242075487971306,
+            -0.03416992723941803,
+            0.01401180773973465,
+            0.0360405407845974,
+            0.013726761564612389,
+            0.010181497782468796,
+            0.011998668313026428,
+            0.011820513755083084,
+            0.0013862603809684515,
+            -0.02376573532819748,
+            0.01915154792368412,
+            0.01711168698966503,
+            -0.06866052746772766,
+            -0.020327365025877953,
+            -0.012702375650405884,
+            0.024068597704172134,
+            0.014412653632462025,
+            0.03340386599302292,
+            0.0033092093653976917,
+            -0.032459646463394165,
+            -0.025636352598667145,
+            -0.0075002810917794704,
+            0.04624876379966736,
+            -0.007437927182763815,
+            0.015962593257427216,
+            -0.00821289699524641,
+            0.021859489381313324,
+            0.012675653211772442,
+            -0.0016623989213258028,
+            0.026847798377275467,
+            -0.02597484551370144,
+            0.01240842230618,
+            -0.018688349053263664,
+            -0.04375460743904114,
+            -0.009771743789315224,
+            0.02360539697110653,
+            0.01886650174856186,
+            0.00418661767616868,
+            0.0078075965866446495,
+            -0.01634562388062477,
+            -0.0024986090138554573,
+            -0.03823183476924896,
+            0.010448728688061237,
+            0.027649492025375366,
+            0.0016223143320530653,
+            -0.005638572387397289,
+            -0.0038815289735794067,
+            0.0159358698874712,
+            0.0004142079269513488,
+            0.007469104137271643,
+            -0.034918174147605896,
+            0.040334053337574005,
+            0.0054158796556293964,
+            0.04582119360566139,
+            -0.04724642634391785,
+            -0.017450178042054176,
+            -0.0038904366083443165,
+            -0.007380026858299971,
+            0.042863838374614716,
+            -0.03573768213391304,
+            -0.02830866165459156,
+            -0.011464206501841545,
+            0.01947222650051117,
+            0.015668639913201332,
+            -0.02155662700533867,
+            -0.03976396098732948,
+            -0.002783655421808362,
+            -0.035274479538202286,
+            -0.010947559960186481,
+            -0.014412653632462025,
+            -0.0032802594359964132,
+            -0.049847472459077835,
+            -0.0017592702060937881,
+            -0.0835898295044899,
+            0.005081841256469488,
+            0.03755485266447067,
+            -0.009807374328374863,
+            0.04111792892217636,
+            -0.002641132101416588,
+            0.002984078601002693,
+            -0.047959040850400925,
+            0.01689790189266205,
+            0.026242075487971306,
+            0.04678322374820709,
+            -0.022180166095495224,
+            0.005242179613560438,
+            0.014679884538054466,
+            -0.011214790865778923,
+            -0.017610516399145126,
+            -0.010511082597076893,
+            0.02004231885075569,
+            -0.0051976414397358894,
+            0.0012392833596095443,
+            0.04528673365712166,
+            0.029537923634052277,
+            0.014697699807584286,
+            0.031943000853061676,
+            0.00030648044776171446,
+            -0.01814497821033001,
+            0.0025320129934698343,
+            0.043148886412382126,
+            0.04093977436423302,
+            -0.04375460743904114,
+            0.015543931163847446,
+            -0.03905134275555611,
+            0.025636352598667145,
+            0.013985084369778633,
+            0.007442380767315626,
+            0.03343949466943741,
+            0.006466988008469343,
+            -0.005803364794701338,
+            0.00020348520774859935,
+            0.024674320593476295,
+            -0.01662176288664341,
+            0.017922285944223404,
+            -0.004836879670619965,
+            -0.022661181166768074,
+            0.000857365841511637,
+            -0.015232161618769169,
+            0.017191855236887932,
+            0.02187730371952057,
+            -0.007188511546701193,
+            -0.0010806149803102016,
+            -0.0003092641127295792,
+            0.021948564797639847,
+            0.004395948722958565,
+            -0.0005812272429466248,
+            0.011428575031459332,
+            0.03500724956393242,
+            -0.03215678781270981,
+            -0.010644698515534401,
+            0.03212115541100502,
+            -0.033101003617048264,
+            0.009566866792738438,
+            -0.04104666784405708,
+            0.010609067045152187,
+            0.0022324915044009686,
+            -0.020986534655094147,
+            0.04076162353157997,
+            -0.010911929421126842,
+            0.001672420185059309,
+            -0.013317007571458817,
+            0.00856920424848795,
+            0.015561746433377266,
+            0.03762611374258995,
+            -0.0039015712682157755,
+            0.02882530726492405,
+            0.01489366963505745,
+            -0.020220471546053886,
+            0.04008463770151138,
+            -0.04268568381667137,
+            -0.010208221152424812,
+            0.015249977819621563,
+            -0.005981518421322107,
+            -0.012461868114769459,
+            -0.020576780661940575,
+            -0.021663518622517586,
+            -0.007001449819654226,
+            0.015143085271120071,
+            -0.02686561457812786,
+            0.006760942284017801,
+            0.006801026873290539,
+            -0.017842117697000504,
+            -0.007651711814105511,
+            0.006609511096030474,
+            -0.008016927167773247,
+            -0.0056296647526323795,
+            0.028201768174767494,
+            -0.017975732684135437,
+            0.01260439120233059,
+            0.02321345917880535,
+            0.028843123465776443,
+            -0.018162794411182404,
+            -0.0016189739108085632,
+            -0.0032490824814885855,
+            -0.0058479029685258865,
+            -0.03285158798098564,
+            0.026509307324886322,
+            0.009620312601327896,
+            -0.010056789964437485,
+            -0.024709951132535934,
+            -0.008836435154080391,
+            0.01879524067044258,
+            -0.0035185404121875763,
+            -0.005549495108425617,
+            -0.021984197199344635,
+            -0.02878967672586441,
+            0.0257788747549057,
+            0.01863490231335163,
+            -0.046711962670087814,
+            -0.0032112249173223972,
+            0.005411426071077585,
+            0.004783433396369219,
+            0.0027213015127927065,
+            -0.014849130995571613,
+            0.007388934958726168,
+            0.026972506195306778,
+            0.0054826876148581505,
+            -0.015543931163847446,
+            -0.012622206471860409,
+            -0.005099656525999308,
+            0.008613742887973785,
+            0.0069925421848893166,
+            0.009932082146406174,
+            0.02975170873105526,
+            0.04072599112987518,
+            0.0018628222169354558,
+            -0.01288943737745285,
+            -0.02561853639781475,
+            0.025315675884485245,
+            -0.0019630338065326214,
+            0.04165239259600639,
+            0.04564303904771805,
+            0.0025876860599964857,
+            -0.060572341084480286,
+            0.012479683384299278,
+            -0.02517315186560154,
+            -0.009843004867434502,
+            0.0066406880505383015,
+            0.0036209789104759693,
+            -0.04660507291555405,
+            -0.0031956364400684834,
+            0.0022536474280059338,
+            0.034152109175920486,
+            -0.007299857679754496,
+            -0.04436033219099045,
+            -0.008929966017603874,
+            -0.007406750228255987,
+            0.02574324421584606,
+            0.010110235773026943,
+            -0.007892219349741936,
+            -0.015294515527784824,
+            0.013361545279622078,
+            -0.0026567205786705017,
+            -0.006190849468111992,
+            0.017013700678944588,
+            -0.007691796403378248,
+            0.011170252226293087,
+            -0.012096652761101723,
+            -0.019739456474781036,
+            -0.017655055969953537,
+            0.03393832594156265,
+            -0.02706158347427845,
+            0.025565091520547867,
+            0.014947115443646908,
+            0.0026923513505607843,
+            0.039407651871442795,
+            -0.01535686943680048,
+            -0.027364445850253105,
+            -0.002897228579968214,
+            -0.0037078289315104485,
+            -0.03766174241900444,
+            0.01815388724207878,
+            -0.023516319692134857,
+            -0.022091088816523552,
+            0.013495161198079586,
+            0.00951342098414898,
+            0.0002751643187366426,
+            -0.009308543056249619,
+            -0.00851130485534668,
+            -0.0015465988544747233,
+            0.01956130377948284,
+            0.019258441403508186,
+            -0.007451288867741823,
+            0.043505191802978516,
+            -0.0004167132137808949,
+            0.0038369905669242144,
+            0.0005995993851684034,
+            0.018100440502166748,
+            0.060287293046712875,
+            0.020452072843909264,
+            0.0012481911107897758,
+            0.030482139438390732,
+            0.022500842809677124,
+            0.037162911146879196,
+            -0.03019709326326847,
+            -0.04564303904771805,
+            0.017895564436912537,
+            -0.007250865455716848,
+            0.009522328153252602,
+            0.022429581731557846,
+            0.0011813833843916655,
+            0.01418996136635542,
+            0.007063803728669882,
+            -0.010243851691484451,
+            0.023801365867257118,
+            -0.032905034720897675,
+            -0.02789890766143799,
+            0.027115030214190483,
+            0.010056789964437485,
+            0.019971057772636414,
+            0.0360405407845974,
+            -0.024870289489626884,
+            -0.033671095967292786,
+            0.004881417844444513,
+            0.006680772639811039,
+            -0.0283442921936512,
+            0.012942884117364883,
+            -0.013521884568035603,
+            0.015160900540649891,
+            0.0064001805149018764,
+            -0.012648929841816425,
+            -0.0027279823552817106,
+            0.013780207373201847,
+            -0.0141721460968256,
+            0.008698366582393646,
+            -0.015160900540649891,
+            -0.012452960945665836,
+            0.010626882314682007,
+            -0.030642477795481682,
+            0.0008445610292255878,
+            0.00507738720625639,
+            -0.0015131949912756681,
+            0.006230934057384729,
+            -0.0067297653295099735,
+            -0.004431579262018204,
+            -0.005865718703716993,
+            -0.03185392543673515,
+            0.02795235440135002,
+            0.02766730636358261,
+            -0.025119705125689507,
+            -0.0030553401447832584,
+            -0.0036900134291499853,
+            -0.014528454281389713,
+            -0.006346734240651131,
+            0.010876297950744629,
+            -0.014448285102844238,
+            0.0021411876659840345,
+            -0.016087301075458527,
+            -0.018367672339081764,
+            0.023961706086993217,
+            0.022518659010529518,
+            0.042863838374614716,
+            -0.0010037861065939069,
+            0.022750258445739746,
+            0.021307211369276047,
+            0.0011028843000531197,
+            0.0017982413992285728,
+            0.017494717612862587,
+            -0.017557071521878242,
+            0.028985645622015,
+            0.025707613676786423,
+            0.03099878504872322,
+            0.010849575512111187,
+            -0.03221023455262184,
+            -0.0024674320593476295,
+            0.018848687410354614,
+            0.017076054587960243,
+            0.0008317562169395387,
+            -0.013486253097653389,
+            -0.009771743789315224,
+            0.012533130124211311,
+            -0.02152099646627903,
+            -0.0065248883329331875,
+            0.013512976467609406,
+            -0.016514871269464493,
+            0.0032802594359964132,
+            -0.007825411856174469,
+            0.02472776733338833,
+            0.0028838668949902058,
+            -0.005193187389522791,
+            0.012773637659847736,
+            -0.00048185072955675423,
+            0.005237726029008627,
+            -0.005037302616983652,
+            0.02324908971786499,
+            0.03493598848581314,
+            0.012684560380876064,
+            0.04407528415322304,
+            0.01998887211084366,
+            -0.013423899188637733,
+            -0.009531236253678799,
+            -0.020612411201000214,
+            0.022358320653438568,
+            -0.030286170542240143,
+            -0.024709951132535934,
+            -0.022073272615671158,
+            0.01535686943680048,
+            0.00679657282307744,
+            0.015606285072863102,
+            -0.0033893787767738104,
+            -0.004447167739272118,
+            0.031372908502817154,
+            0.004179936833679676,
+            0.025511644780635834,
+            0.006907918956130743,
+            -0.007905581034719944,
+            -0.004012917634099722,
+            0.0024139860179275274,
+            0.008150543086230755,
+            -0.015241069719195366,
+            0.03826746717095375,
+            -0.0035630788188427687,
+            0.04813719540834427,
+            -0.052733566612005234,
+            -0.002003118395805359,
+            0.00635564187541604,
+            -0.01918718032538891,
+            -0.005669749341905117,
+            -0.03915823623538017,
+            0.02030954882502556,
+            -0.03377798572182655,
+            0.026669645681977272,
+            -0.02027391828596592,
+            0.026491491124033928,
+            -0.029929861426353455,
+            0.003576440503820777,
+            -0.0015243296511471272,
+            -0.0038035865873098373,
+            0.0164970550686121,
+            0.024282382801175117,
+            -0.002409531967714429,
+            -0.02590358257293701,
+            -0.0008083735010586679,
+            0.0037635019980371,
+            0.0066273268312215805,
+            -0.013717853464186192,
+            0.0054158796556293964,
+            -0.0012582122581079602,
+            -0.029680447652935982,
+            -0.033101003617048264,
+            0.012132283300161362,
+            -0.010911929421126842,
+            0.0337958037853241,
+            0.030375247821211815,
+            -0.026794353500008583,
+            -0.027043769136071205,
+            0.0034561865031719208,
+            -0.008676096796989441,
+            -0.009090305306017399,
+            0.020737119019031525,
+            0.04532236233353615,
+            0.00925509724766016,
+            -0.020790565758943558,
+            0.023943889886140823,
+            0.004734441172331572,
+            0.008582565933465958,
+            0.004422671627253294,
+            -0.021734781563282013,
+            -0.018492378294467926,
+            -0.01689790189266205,
+            -0.01726311631500721,
+            0.008177265524864197,
+            -0.01972164213657379,
+            -0.04012026637792587,
+            -0.04742458090186119,
+            -0.011179160326719284,
+            0.019383149221539497,
+            0.004144306294620037,
+            0.05052445828914642,
+            0.012452960945665836,
+            0.02478121407330036,
+            -0.0017191855004057288,
+            -0.005469325929880142,
+            0.04646254703402519,
+            -0.019899794831871986,
+            0.051700275391340256,
+            0.009700481779873371,
+            -0.0021634569857269526,
+            -0.01578443869948387,
+            0.03188955411314964,
+            -0.024905921891331673,
+            -0.0019240626133978367,
+            0.006342280190438032,
+            -0.0034984981175512075,
+            0.017298748716711998,
+            0.008644919842481613,
+            0.010128051973879337,
+            -0.02360539697110653,
+            0.028130507096648216,
+            -0.023641027510166168,
+            0.014590808190405369,
+            0.015926962718367577,
+            -0.002009799238294363,
+            0.0036098442506045103,
+            -0.0054826876148581505,
+            0.004574102349579334,
+            -0.002970716916024685,
+            0.011597821488976479,
+            -0.02510189078748226,
+            0.015009469352662563,
+            -0.02228705771267414,
+            -0.013317007571458817,
+            0.0004178266681265086,
+            -0.030945340171456337,
+            0.012399514205753803,
+            0.04108230024576187,
+            0.01037746760994196,
+            -0.02232268825173378,
+            0.005371341481804848,
+            -0.051450859755277634,
+            -0.008199535310268402,
+            0.0014809046406298876,
+            0.01271128375083208,
+            0.01346843782812357,
+            0.0007504734676331282,
+            0.009727205149829388,
+            -0.00955795869231224,
+            -0.03055340051651001,
+            0.033421680331230164,
+            -0.001038860180415213,
+            -0.0020298415329307318,
+            0.003928294405341148,
+            0.01851019449532032,
+            0.0025854590348899364,
+            0.004188844468444586,
+            0.012773637659847736,
+            -0.007709611672908068,
+            -0.03224586322903633,
+            -0.021449735388159752,
+            -0.0038926636334508657,
+            0.0019318568520247936,
+            -0.003694467479363084,
+            0.003318117232993245,
+            0.0055851261131465435,
+            -0.015535023994743824,
+            0.01082285214215517,
+            -0.011980853043496609,
+            0.010707051493227482,
+            -0.019169364124536514,
+            -0.004329140763729811,
+            0.006658503785729408,
+            -0.020808380097150803,
+            -0.018296409398317337,
+            -0.051450859755277634,
+            -0.010466543957591057,
+            0.002101103076711297,
+            0.021146873012185097,
+            0.006676319055259228,
+            0.008489035069942474,
+            -0.014394838362932205,
+            0.0068322038277983665,
+            0.003656609682366252,
+            0.02520878240466118,
+            -0.026829984039068222,
+            0.03119475580751896,
+            0.018723979592323303,
+            0.016951346769928932,
+            -0.03862377628684044,
+            0.03249527886509895,
+            0.016443608328700066,
+            -0.003759048180654645,
+            0.017521440982818604,
+            0.014002899639308453,
+            0.04753147065639496,
+            -0.030606847256422043,
+            0.004810156300663948,
+            -0.02754260040819645,
+            0.025084074586629868,
+            -0.03805368393659592,
+            0.028237400576472282,
+            0.0009163793292827904,
+            0.023943889886140823,
+            0.04428907111287117,
+            9.728875738801435e-05,
+            0.03851688280701637,
+            -0.022019827738404274,
+            -0.009045766666531563,
+            0.025636352598667145,
+            -0.017476901412010193,
+            0.015882423147559166,
+            0.00778978131711483,
+            -0.024282382801175117,
+            0.01838548667728901,
+            0.007709611672908068,
+            -0.0031956364400684834,
+            -0.04788777977228165,
+            0.02795235440135002,
+            0.02260773442685604,
+            0.02120031975209713,
+            0.0067297653295099735,
+            -0.012150098569691181,
+            -0.010234943591058254,
+            0.005611849017441273,
+            -0.009246189147233963,
+            0.016755377873778343,
+            -0.015846792608499527,
+            -0.030909709632396698,
+            0.00849348958581686,
+            -0.01927625574171543,
+            0.012648929841816425,
+            -0.0038904366083443165,
+            0.029092539101839066,
+            -0.017770856618881226,
+            -0.01068032905459404,
+            0.01101882103830576,
+            -0.0028237400110810995,
+            -0.016069484874606133,
+            -0.04842223972082138,
+            -0.038801927119493484,
+            0.008453404530882835,
+            -0.0013105449033901095,
+            -0.020184841006994247,
+            0.011909591034054756,
+            -0.010591251775622368,
+            -0.004792341031134129,
+            -0.01972164213657379,
+            -0.010742682963609695,
+            -0.023035304620862007,
+            -0.006760942284017801,
+            0.010074605233967304,
+            0.026277706027030945,
+            -0.018848687410354614,
+            0.03798241913318634,
+            0.029413215816020966,
+            0.013985084369778633,
+            -0.012506406754255295,
+            0.0257788747549057,
+            -0.009353081695735455,
+            -0.01489366963505745,
+            0.033671095967292786,
+            -0.0031466439832001925,
+            -0.009192743338644505,
+            0.00017439600196667016,
+            -0.024799028411507607,
+            -0.023070935159921646,
+            0.019579118117690086,
+            -0.022393951192498207,
+            -0.025280045345425606,
+            -0.004921502433717251,
+            -0.006141857244074345,
+            -0.02975170873105526,
+            -0.04753147065639496,
+            -0.011214790865778923,
+            0.03645029664039612,
+            0.0057632802054286,
+            0.018492378294467926,
+            0.01711168698966503,
+            -0.012230267748236656,
+            -0.03039306215941906,
+            0.004721079487353563,
+            -0.012105559930205345,
+            -0.008738450706005096,
+            -0.01287162210792303,
+            0.007700704038143158,
+            0.016951346769928932,
+            0.01101882103830576,
+            0.0011669083032757044,
+            -0.0052778106182813644,
+            -0.02123595029115677,
+            -0.007861042395234108,
+            -0.0176995936781168,
+            -0.028985645622015,
+            -0.0031466439832001925,
+            -0.033029742538928986,
+            0.03823183476924896,
+            0.018777426332235336,
+            0.005273356568068266,
+            -0.01982853375375271,
+            -0.012720191851258278,
+            -0.020790565758943558,
+            0.00927291251718998,
+            -0.00707716541364789,
+            0.009718297980725765,
+            -0.004614186938852072,
+            0.01726311631500721,
+            -0.006057234015315771,
+            0.00047684015589766204,
+            -0.04436033219099045,
+            0.009023496881127357,
+            0.01934751868247986,
+            0.003930521197617054,
+            -0.009878636337816715,
+            0.008880973793566227,
+            0.011606729589402676,
+            -0.021022165194153786,
+            -0.0236944742500782,
+            0.019222810864448547,
+            0.020576780661940575,
+            0.015909146517515182,
+            -0.009299635887145996,
+            -0.009299635887145996,
+            0.007130611687898636,
+            0.0007682888535782695,
+            -0.006315557286143303,
+            0.016550501808524132,
+            0.023623213171958923,
+            0.01375348400324583,
+            -0.00018177268793806434,
+            0.026206444948911667,
+            0.009174928069114685,
+            -0.012310437858104706,
+            -0.012114468030631542,
+            -0.007620534859597683,
+            0.0023694473784416914,
+            -0.01670193299651146,
+            -0.011989760212600231,
+            -0.010083513334393501,
+            0.025226598605513573,
+            0.009370896965265274,
+            0.011294960044324398,
+            -0.003090970916673541,
+            -0.006173034198582172,
+            0.005411426071077585,
+            0.009949897415935993,
+            0.0001479512866353616,
+            -0.001145752496086061,
+            0.007219688501209021,
+            -0.038766298443078995,
+            -0.028682785108685493,
+            0.0014196642441675067,
+            -0.002750251442193985,
+            0.0033515209797769785,
+            0.03506069630384445,
+            0.027596045285463333,
+            -0.007014811504632235,
+            -0.01918718032538891,
+            -0.0018227375112473965,
+            0.024282382801175117,
+            0.0001636789384065196,
+            -0.02308875136077404,
+            -0.009477789513766766,
+            0.009090305306017399,
+            0.00032874970929697156,
+            0.02927069365978241,
+            0.005700926296412945,
+            -0.008235165849328041,
+            0.019383149221539497,
+            0.10076387226581573,
+            0.007776419632136822,
+            0.006885650102049112,
+            -0.012150098569691181,
+            -0.031016601249575615,
+            0.0021033298689872026,
+            0.021699151024222374,
+            -0.019080286845564842,
+            0.04485916346311569,
+            0.011553282849490643,
+            0.013263560831546783,
+            0.018973395228385925,
+            -0.0021144645288586617,
+            -0.010974282398819923,
+            -0.016131838783621788,
+            0.030606847256422043,
+            0.0016323354793712497,
+            0.02184167318046093,
+            -0.020897457376122475,
+            0.0017414548201486468,
+            0.004591918084770441,
+            0.012800361029803753,
+            0.00606168806552887,
+            -0.005055117886513472,
+            -0.02510189078748226,
+            0.018777426332235336,
+            -0.018180610612034798,
+            0.007224142551422119,
+            -0.008858704939484596,
+            -0.022874966263771057,
+            0.008987866342067719,
+            -0.00014502844715025276,
+            -0.01244405284523964,
+            0.005513864569365978,
+            -0.014109792187809944,
+            0.0326734334230423,
+            -0.012738007120788097,
+            0.0015354643110185862,
+            0.003663290524855256,
+            -0.016390163451433182,
+            0.01202539075165987,
+            0.01746799424290657,
+            -0.01200757548213005,
+            0.030090201646089554,
+            -0.016710840165615082,
+            -0.004318006336688995,
+            0.012061022222042084,
+            -0.018002456054091454,
+            0.011214790865778923,
+            0.028486816212534904,
+            -0.009335266426205635,
+            0.005683110561221838,
+            -0.0075581809505820274,
+            0.009905358776450157,
+            0.009335266426205635,
+            0.026402413845062256,
+            0.023142196238040924,
+            -0.0005219354061409831,
+            0.004251198377460241,
+            -0.008880973793566227,
+            -0.0019708280451595783,
+            -0.0009464428294450045,
+            0.034312449395656586,
+            -0.018162794411182404,
+            -0.0043246871791779995,
+            -0.009326359257102013,
+            0.0006680772639811039,
+            -0.014724423177540302,
+            -0.002120031975209713,
+            -0.007437927182763815,
+            -0.0019351971568539739,
+            0.0076606194488704205,
+            -0.00036493720836006105,
+            0.016158562153577805,
+            -0.008257435634732246,
+            -0.004556287080049515,
+            0.004997218027710915,
+            -0.005224364344030619,
+            0.01657722517848015,
+            -0.02574324421584606,
+            0.0037612752057611942,
+            0.018456747755408287,
+            0.00869391206651926,
+            0.011508745141327381,
+            -0.01642579399049282,
+            0.0025854590348899364,
+            -0.002992986235767603,
+            -0.012577668763697147,
+            0.012577668763697147,
+            -0.01831422559916973,
+            -0.011499837040901184,
+            -0.021129056811332703,
+            -0.0297873392701149,
+            -0.002863824600353837,
+            -0.01851019449532032,
+            0.008146088570356369,
+            0.01918718032538891,
+            0.028647154569625854,
+            0.00018024168093688786,
+            -0.004970495123416185,
+            -0.01743236370384693,
+            0.015962593257427216,
+            0.045037318021059036,
+            -0.016292177140712738,
+            -0.02520878240466118,
+            -0.009914266876876354,
+            0.0127290990203619,
+            -0.008622650988399982,
+            -0.026046106591820717,
+            -0.003943882882595062,
+            0.00577218784019351,
+            -0.02911035344004631,
+            0.009379805065691471,
+            0.0005712060956284404,
+            0.024442721158266068,
+            0.004672087263315916,
+            0.020665857940912247,
+            -0.017521440982818604,
+            -0.0019708280451595783,
+            0.025476014241576195,
+            -0.029520107433199883,
+            0.0045874640345573425,
+            0.01802917942404747,
+            -0.01346843782812357,
+            0.010769405402243137,
+            0.030820632353425026,
+            0.025226598605513573,
+            -0.01963256485760212,
+            0.024335827678442,
+            0.014626438729465008,
+            -0.008720635436475277,
+            0.003284713253378868,
+            0.011660175397992134,
+            0.012622206471860409,
+            0.006698588375002146,
+            -0.0034584132954478264,
+            -0.005861264653503895,
+            0.009045766666531563,
+            0.031337279826402664,
+            0.0016623989213258028,
+            -0.012657837942242622,
+            0.028415553271770477,
+            0.02497718296945095,
+            -0.005429241340607405,
+            0.008426681160926819,
+            -0.02039862610399723,
+            -0.016443608328700066,
+            0.01730765588581562,
+            0.022465212270617485,
+            0.005910256877541542,
+            0.0032401748467236757,
+            -0.011366221122443676,
+            0.022839335724711418,
+            -0.008377688936889172,
+            0.01154437568038702,
+            -0.027168476954102516,
+            -0.002518651308491826,
+            -0.0036900134291499853,
+            0.02068367227911949,
+            -0.0008106004097498953,
+            -0.005032849032431841,
+            0.019062472507357597,
+            -0.01228371448814869,
+            -0.019775087013840675,
+            -0.006462534423917532,
+            -0.018118256703019142,
+            0.017637239769101143,
+            -0.004774525761604309,
+            -0.02030954882502556,
+            0.007549273315817118,
+            0.0005678656743839383,
+            -0.009041312150657177,
+            0.006801026873290539,
+            0.004144306294620037,
+            0.008057012222707272,
+            -0.02264336682856083,
+            0.010751590132713318,
+            -0.005464872345328331,
+            0.007901127450168133,
+            -0.008382143452763557,
+            0.014368114992976189,
+            -0.006738672964274883,
+            0.0015599604230374098,
+            0.0006012695375829935,
+            -0.006208664737641811,
+            0.000636900367680937,
+            0.013210115022957325,
+            0.0032624441664665937,
+            -0.016550501808524132,
+            0.015312331728637218,
+            -0.008662735112011433,
+            -0.004925956483930349,
+            -0.005028394982218742,
+            -0.004106448497623205,
+            -0.0011357313487678766,
+            0.012738007120788097,
+            0.00865382794290781,
+            -0.031123492866754532,
+            -0.006689680740237236,
+            -0.024959366768598557,
+            -0.009326359257102013,
+            0.010145867243409157,
+            0.012052114121615887,
+            -0.020576780661940575,
+            0.009001228027045727,
+            -0.027079399675130844,
+            -0.01226589921861887,
+            0.0269012451171875,
+            0.023961706086993217,
+            -0.02068367227911949,
+            0.012738007120788097,
+            -0.014804592356085777,
+            -0.02014921046793461,
+            0.015009469352662563,
+            0.021467549726366997,
+            0.01224808394908905,
+            0.005028394982218742,
+            -0.04400402307510376,
+            -0.010920836590230465,
+            0.019864164292812347,
+            -0.03140854090452194,
+            -0.009504512883722782,
+            0.027417892590165138,
+            -0.006827749777585268,
+            -0.01726311631500721,
+            -0.003516313387081027,
+            0.03522103279829025,
+            -0.02123595029115677,
+            0.015526115894317627,
+            0.011428575031459332,
+            -0.02152099646627903,
+            0.014760053716599941,
+            -0.012853806838393211,
+            0.008618197403848171,
+            0.0019296299433335662,
+            -0.022536473348736763,
+            -0.0006502618780359626,
+            0.018011363223195076,
+            -0.0005444829585030675,
+            -0.004050775431096554,
+            0.011455298401415348,
+            0.016639579087495804,
+            -0.018421117216348648,
+            0.013922730460762978,
+            0.006088410969823599,
+            9.90285407169722e-05,
+            0.016283269971609116,
+            -0.02052333392202854,
+            -0.01810934767127037,
+            0.01979290321469307,
+            -0.004179936833679676,
+            -0.02421111986041069,
+            0.02344505861401558,
+            -0.02376573532819748,
+            -0.020950904116034508,
+            -0.02340942807495594,
+            0.00032652277150191367,
+            0.00849348958581686,
+            -0.01734328642487526,
+            -0.026562752202153206,
+            0.007233050186187029,
+            0.01950785703957081,
+            0.04689011722803116,
+            0.018038086593151093,
+            -0.0023048666771501303,
+            0.023819182068109512,
+            -0.016595039516687393,
+            0.013263560831546783,
+            0.012105559930205345,
+            -0.0036543826572597027,
+            -0.015757715329527855,
+            0.0025743243750184774,
+            0.009602497331798077,
+            0.007152881007641554,
+            0.0030263899825513363,
+            -0.022251427173614502,
+            -0.02120031975209713,
+            -0.04101103916764259,
+            0.00957577396184206,
+            -0.007771965581923723,
+            0.0008968937327153981,
+            0.05790003016591072,
+            0.00126711989287287,
+            -0.033350419253110886,
+            0.01754816249012947,
+            0.00047767526120878756,
+            0.0007610513712279499,
+            -0.011633452028036118,
+            0.002986305393278599,
+            -0.02389044314622879,
+            -0.013949453830718994,
+            -0.0353991873562336,
+            -0.0036877866368740797,
+            0.01027948223054409,
+            0.005162010435014963,
+            0.005700926296412945,
+            -0.02805924601852894,
+            -0.002245853189378977,
+            -0.025867952033877373,
+            0.022304873913526535,
+            0.006475895643234253,
+            -0.0001588075392646715,
+            0.0023649935610592365,
+            0.027400076389312744,
+            -0.008622650988399982,
+            -0.01854582503437996,
+            -0.020790565758943558,
+            0.0011902910191565752,
+            -0.004248971585184336,
+            0.001448614289984107,
+            0.0009458860731683671,
+            -0.025832321494817734,
+            0.009486697614192963,
+            0.00028532466967590153,
+            -0.012595484033226967,
+            0.014100884087383747,
+            -0.013352638110518456,
+            -0.007001449819654226,
+            0.009157112799584866,
+            0.023160012438893318,
+            -0.027524784207344055,
+            -0.008506850339472294,
+            -0.024870289489626884,
+            0.009985528886318207,
+            0.012675653211772442,
+            0.0029484478291124105,
+            -0.019775087013840675,
+            -0.012925068847835064,
+            -0.02574324421584606,
+            -0.009851912967860699,
+            -0.02059459500014782,
+            -0.008012473583221436,
+            0.005041756667196751,
+            0.01657722517848015,
+            0.03244183212518692,
+            0.016799917444586754,
+            -0.005157556850463152,
+            0.0017548163887113333,
+            -0.002420666627585888,
+            0.008751812390983105,
+            -0.020060133188962936,
+            2.0077113731531426e-05,
+            -0.0159358698874712,
+            0.017957916483283043,
+            0.00999443605542183,
+            -0.013887099921703339,
+            -0.017761947587132454,
+            -0.02276807464659214,
+            0.019525671377778053,
+            -0.022429581731557846,
+            -0.017886655405163765,
+            0.01918718032538891,
+            -0.011036636307835579,
+            0.006409088149666786,
+            -0.013691130094230175,
+            0.037198543548583984,
+            0.010021159425377846,
+            0.025476014241576195,
+            -0.016612855717539787,
+            -0.005567310843616724,
+            -0.0272219218313694,
+            -0.004400402307510376,
+            -0.01682663895189762,
+            -0.00434695603325963,
+            -0.0012337160296738148,
+            0.009753928519785404,
+            -0.02526222914457321,
+            -0.005567310843616724,
+            0.02321345917880535,
+            -0.027560414746403694,
+            -0.0022124492097646,
+            -0.0021757050417363644,
+            0.006337826605886221,
+            0.007121704053133726,
+            0.0014397065388038754,
+            -0.002219130052253604,
+            -0.022180166095495224,
+            -0.008448950946331024,
+            -0.028807492926716805,
+            0.022091088816523552,
+            -0.010315113700926304,
+            -0.014653162099421024,
+            0.0001746743801049888,
+            0.021182503551244736,
+            0.0046899025328457355,
+            -0.010172589682042599,
+            0.0017403413075953722,
+            -0.007598265539854765,
+            0.0021645703818649054,
+            0.0038147212471812963,
+            0.012096652761101723,
+            -0.0020988760516047478,
+            0.014083068817853928,
+            0.02020265720784664,
+            -0.008560297079384327,
+            -0.0006975840078666806,
+            -0.0058122724294662476,
+            0.0037790904752910137,
+            0.011375129222869873,
+            0.01037746760994196,
+            0.016203101724386215,
+            0.003641021205112338,
+            -0.009486697614192963,
+            0.01670193299651146,
+            -0.014083068817853928,
+            0.02184167318046093,
+            -0.001927402918227017,
+            -0.01943659596145153,
+            0.002552055288106203,
+            -0.047923412173986435,
+            0.0009069149382412434,
+            -0.012310437858104706,
+            -0.022750258445739746,
+            0.006293287966400385,
+            -0.011838329024612904,
+            -0.008190627209842205,
+            0.019098103046417236,
+            -0.009103666059672832,
+            -0.02219798043370247,
+            0.011571098119020462,
+            0.0027591590769588947,
+            -0.009036858566105366,
+            0.006613965146243572,
+            -0.0016679662512615323,
+            0.008658281527459621,
+            0.004482798743993044,
+            0.005366887431591749,
+            0.001175816054455936,
+            -0.01565973088145256,
+            -0.018563641235232353,
+            0.020095765590667725,
+            0.008333150297403336,
+            -0.0016278816619887948,
+            0.010003344155848026,
+            0.002652266761288047,
+            0.011428575031459332,
+            -0.003694467479363084,
+            -0.004810156300663948,
+            0.00019415996212046593,
+            -0.014679884538054466,
+            0.014145422726869583,
+            -0.013085407204926014,
+            -0.032228048890829086,
+            0.013664407655596733,
+            0.006008241791278124,
+            0.00681438809260726,
+            -0.008119366131722927,
+            -0.011891775764524937,
+            -0.007161788642406464,
+            -0.012078837491571903,
+            0.007455742452293634,
+            0.015294515527784824,
+            -0.0038147212471812963,
+            -0.0008879860397428274,
+            -0.01418996136635542,
+            0.007575996220111847,
+            0.020184841006994247,
+            0.029965493828058243,
+            -0.02488810569047928,
+            0.00724641140550375,
+            -0.005344618111848831,
+            -0.011873960494995117,
+            -0.020291734486818314,
+            0.024264566600322723,
+            0.01457299292087555,
+            0.01626545563340187,
+            0.004206659737974405,
+            0.0036165250930935144,
+            0.000802249473053962,
+            -0.019365333020687103,
+            0.003191182389855385,
+            0.008137181401252747,
+            -0.013317007571458817,
+            -0.0020688127260655165,
+            0.02478121407330036,
+            -0.013780207373201847,
+            -0.0013484027003869414,
+            0.001965260598808527,
+            0.024032967165112495,
+            -0.0138514693826437,
+            0.001873956760391593,
+            0.004729987122118473,
+            0.0006508186343125999,
+            0.005522772204130888,
+            0.013219022192060947,
+            0.07496717572212219,
+            0.019579118117690086,
+            0.005393610801547766,
+            0.007446834817528725,
+            -0.006311103235930204,
+            -0.006008241791278124,
+            -0.009264005348086357,
+            -0.011553282849490643,
+            0.002538693603128195,
+            -0.0017325470689684153,
+            -0.02308875136077404,
+            -0.002113351132720709,
+            -0.005825634114444256,
+            0.00045429254532791674,
+            -0.0012982968473806977,
+            0.01734328642487526,
+            -0.012666745111346245,
+            0.013913823291659355,
+            0.03204989433288574,
+            0.007188511546701193,
+            -0.021770412102341652,
+            0.013504069298505783,
+            0.0025854590348899364,
+            -0.006484803277999163,
+            -0.003402740228921175,
+            -0.006609511096030474,
+            0.006996996235102415,
+            0.011669083498418331,
+            -0.010600159876048565,
+            0.03851688280701637,
+            0.0013951680157333612,
+            0.0014764508232474327,
+            -0.01298742275685072,
+            -0.008008019998669624,
+            -0.0038035865873098373,
+            0.0024852475617080927,
+            0.012354975566267967,
+            -0.0010978736681863666,
+            -0.01851019449532032,
+            0.003068701596930623,
+            0.004166575148701668,
+            0.021182503551244736,
+            0.0044182175770401955,
+            0.002165683778002858,
+            -0.009246189147233963,
+            0.01069814432412386,
+            -0.00923728197813034,
+            -0.004021825268864632,
+            0.009860821068286896,
+            -0.0033760173246264458,
+            -0.010457636788487434,
+            -0.016452517360448837,
+            -0.0003234607574995607,
+            0.008742904290556908,
+            -0.013691130094230175,
+            -0.014742238447070122,
+            0.022429581731557846,
+            -0.0014385931426659226,
+            0.023035304620862007,
+            -0.005406972020864487,
+            0.008622650988399982,
+            0.011473113670945168,
+            -0.0012159006437286735,
+            -0.00782986544072628,
+            0.019703825935721397,
+            -0.00027196313021704555,
+            0.004277921747416258,
+            -0.014679884538054466,
+            0.009531236253678799,
+            -0.01114352885633707,
+            -0.0023605397436767817,
+            0.016363440081477165,
+            0.0001629830221645534,
+            0.007037080824375153,
+            -0.002538693603128195,
+            -0.009032404981553555,
+            -0.031782664358615875,
+            -0.02353413589298725,
+            -0.0022057686001062393,
+            0.013094314374029636,
+            -0.008578112348914146,
+            0.0022592146415263414,
+            -0.0027480246499180794,
+            -0.004053002223372459,
+            -0.016194192692637444,
+            0.006507072597742081,
+            5.518596663023345e-05,
+            0.015971500426530838,
+            0.0011435255873948336,
+            -0.015276700258255005,
+            0.003774636657908559,
+            -0.0031399631407111883,
+            0.018937764689326286,
+            -0.013922730460762978,
+            -0.0029573554638773203,
+            0.0236944742500782,
+            -0.01618528552353382,
+            -0.0011691352119669318,
+            0.003202317049726844,
+            -0.013682222925126553,
+            0.031016601249575615,
+            0.013094314374029636,
+            0.023943889886140823,
+            0.015401408076286316,
+            0.01212337613105774,
+            0.001612293184734881,
+            0.01630108617246151,
+            0.013619869016110897,
+            0.012925068847835064,
+            -0.015989316627383232,
+            -0.007348849903792143,
+            -0.017280932515859604,
+            -0.004551833029836416,
+            0.012782545760273933,
+            -0.017031516879796982,
+            0.0053579797968268394,
+            -0.01559737790375948,
+            -0.013014145195484161,
+            -0.0167375635355711,
+            -0.006079503335058689,
+            0.005148648750036955,
+            -0.02216234989464283,
+            -0.012942884117364883,
+            -0.012862714938819408,
+            -0.009860821068286896,
+            -0.01879524067044258,
+            -0.011339498683810234,
+            -0.012016483582556248,
+            -0.019204994663596153,
+            -0.035559527575969696,
+            -0.014626438729465008,
+            -0.0037456867285072803,
+            -0.011615636758506298,
+            -0.0038414443843066692,
+            -0.03634340316057205,
+            0.012916160747408867,
+            6.475478585343808e-05,
+            -0.004371452145278454,
+            0.003963925410062075,
+            0.013397176750004292,
+            -0.00856920424848795,
+            -0.03237057104706764,
+            -0.028077062219381332,
+            -0.01972164213657379,
+            -0.00024579677847214043,
+            -0.00031984198722057045,
+            0.011259329505264759,
+            0.0004286829207558185,
+            0.017058240249753,
+            -0.01053780596703291,
+            0.017672870308160782,
+            -0.03586238995194435,
+            -0.03196081891655922,
+            -0.003759048180654645,
+            -0.008729543536901474,
+            -0.012043206952512264,
+            0.006409088149666786,
+            0.007037080824375153,
+            -0.0055851261131465435,
+            -0.012354975566267967,
+            -0.007117250002920628,
+            -0.016407977789640427,
+            -0.008925512433052063,
+            0.009210558608174324,
+            0.003333705710247159,
+            0.012809268198907375,
+            0.0150540079921484,
+            0.00868945848196745,
+            -0.01657722517848015,
+            0.01975727267563343,
+            -0.005362433847039938,
+            -0.003008574713021517,
+            0.013512976467609406,
+            0.010110235773026943,
+            0.0167375635355711,
+            0.01130386721342802,
+            0.004306871443986893,
+            -0.014964930713176727,
+            0.02911035344004631,
+            0.015846792608499527,
+            0.009174928069114685,
+            0.01543703954666853,
+            0.015151992440223694,
+            0.009081397205591202,
+            -0.01202539075165987,
+            -0.0007176263607107103,
+            -0.017655055969953537,
+            0.0008055898360908031,
+            0.007353303954005241,
+            -0.010992098599672318,
+            -0.006070595700293779,
+            0.033350419253110886,
+            0.0022703493013978004,
+            -0.015730993822216988,
+            0.013138853013515472,
+            -0.011250421404838562,
+            -0.006720857694745064,
+            -0.011214790865778923,
+            -0.0118472371250391,
+            -0.003019709372892976,
+            0.015303423628211021,
+            0.011366221122443676,
+            -0.02103998139500618,
+            0.011437483131885529,
+            0.01303196046501398,
+            -0.0057098339311778545,
+            -0.005335710477083921,
+            0.014528454281389713,
+            0.01870616339147091,
+            0.01433248445391655,
+            -0.03628995642066002,
+            -0.0023271357640624046,
+            0.012078837491571903,
+            -0.026527121663093567,
+            -0.019383149221539497,
+            0.013940545730292797,
+            0.0012548718368634582,
+            -0.019044656306505203,
+            0.01642579399049282,
+            -0.002663401421159506,
+            -0.022874966263771057,
+            -0.008542481809854507,
+            0.0034851364325731993,
+            0.019935425370931625,
+            0.021129056811332703,
+            -0.012542037293314934,
+            0.008217350579798222,
+            -0.014697699807584286,
+            0.004286829382181168,
+            -0.023355981335043907,
+            -0.0037456867285072803,
+            -0.008560297079384327,
+            -0.0021345068234950304,
+            0.017672870308160782,
+            0.008729543536901474,
+            0.02312438189983368,
+            0.019971057772636414,
+            0.0006380138220265508,
+            -0.016167471185326576,
+            -0.003306982573121786,
+            0.011232606135308743,
+            -0.008208442479372025,
+            -0.0002967376494780183,
+            -0.001585570047609508,
+            0.012907253578305244,
+            0.005393610801547766,
+            0.030642477795481682,
+            -0.002772520761936903,
+            0.0012994103599339724,
+            -3.0011284252395853e-05,
+            0.002788109239190817,
+            0.0012392833596095443,
+            -0.005972610786557198,
+            -0.0018917721463367343,
+            0.005153102800250053,
+            0.02552945911884308,
+            0.004939318168908358,
+            -0.0069123730063438416,
+            0.007976843044161797,
+            -0.0022169030271470547,
+            -0.011642360128462315,
+            -0.0038102674297988415,
+            -0.014840223826467991,
+            -0.01746799424290657,
+            0.01979290321469307,
+            0.0277029387652874,
+            -0.013762392103672028,
+            -0.026954691857099533,
+            -0.00011412987078074366,
+            0.04044094309210777,
+            0.002268122276291251,
+            0.0028727322351187468,
+            0.022946227341890335,
+            -0.0020465434063225985,
+            0.018563641235232353,
+            0.021823858842253685,
+            0.011027729138731956,
+            0.009442158974707127,
+            0.03221023455262184,
+            0.0017158451955765486,
+            -0.004694356583058834,
+            -0.0030286170076578856,
+            -0.0025320129934698343,
+            -0.012497498653829098,
+            0.0009325245628133416,
+            0.002859370782971382,
+            -0.02068367227911949,
+            0.00792785082012415,
+            -0.01959693431854248,
+            -0.0006819955888204277,
+            -0.003652155864983797,
+            0.0065382495522499084,
+            -0.025476014241576195,
+            -0.010047882795333862,
+            0.009967712685465813,
+            -0.01043982058763504,
+            0.0023739011958241463,
+            -0.01915154792368412,
+            -0.004184390883892775,
+            0.022055458277463913,
+            -0.004451621789485216,
+            0.013887099921703339,
+            -0.00841331947594881,
+            -0.0011318342294543982,
+            -0.008404412306845188,
+            -0.0066451421007514,
+            0.011196975596249104,
+            -0.0058122724294662476,
+            0.001120699685998261,
+            -0.010876297950744629,
+            0.008065919391810894,
+            -0.005634118337184191,
+            0.004805702716112137,
+            0.023712290450930595,
+            -0.0030442054849117994,
+            -0.0068901036866009235,
+            0.007424565497785807,
+            0.010038974694907665,
+            -0.022411765530705452,
+            0.02702595293521881,
+            -0.011348405852913857,
+            0.02059459500014782,
+            0.00296626309864223,
+            -0.019703825935721397,
+            -0.0027168476954102516,
+            -0.0010488813277333975,
+            0.010867390781641006,
+            -0.006649595685303211,
+            0.002970716916024685,
+            -0.01831422559916973,
+            0.015276700258255005,
+            0.005945887882262468,
+            -0.015330146998167038,
+            0.01979290321469307,
+            -0.006988088600337505,
+            0.0019663742277771235,
+            -0.017485808581113815,
+            0.00578109547495842,
+            0.007130611687898636,
+            -0.013664407655596733,
+            -0.003739005886018276,
+            -0.019703825935721397,
+            -0.0005723195499740541,
+            -0.01565973088145256,
+            0.0036098442506045103,
+            -0.009718297980725765,
+            -0.013646592386066914,
+            -0.036913495510816574,
+            -0.0001862265489762649,
+            0.004685448482632637,
+            -0.007197419181466103,
+            -0.001399621949531138,
+            0.00012637795589398593,
+            0.013958360999822617,
+            0.009290727786719799,
+            -0.0039572445675730705,
+            0.004979402758181095,
+            0.034348081797361374,
+            -0.04421781003475189,
+            -0.01138403732329607,
+            -0.00033654391882009804,
+            0.004008463583886623,
+            -0.020737119019031525,
+            -0.008275250904262066,
+            0.013121037743985653,
+            -0.0007081619114615023,
+            0.016274362802505493,
+            -0.016033854335546494,
+            0.010947559960186481,
+            0.003079836256802082,
+            0.008177265524864197,
+            -0.007135065272450447,
+            -0.0053579797968268394,
+            -0.012034298852086067,
+            -0.000576773367356509,
+            -0.001016590977087617,
+            -0.02827303111553192,
+            0.01734328642487526,
+            0.0016033854335546494,
+            -0.0056296647526323795,
+            0.0069480035454034805,
+            -0.02453179843723774,
+            -0.009148204699158669,
+            -0.012292621657252312,
+            0.023658843711018562,
+            -0.01738782413303852,
+            -0.00026625662576407194,
+            -0.004698810167610645,
+            -0.006444718688726425,
+            0.014724423177540302,
+            -0.0037234174087643623,
+            0.007322126999497414,
+            0.00037635021726600826,
+            -0.011339498683810234,
+            -0.002014253055676818,
+            0.0024785667192190886,
+            -0.010190405882894993,
+            0.031693585216999054,
+            -0.005959249567240477,
+            -0.0027702937368303537,
+            0.007433473132550716,
+            0.0010666967136785388,
+            -0.002318228129297495,
+            0.010270575061440468,
+            0.0017882202519103885,
+            0.0039572445675730705,
+            0.008894335478544235,
+            -0.0036989212967455387,
+            -0.006021603476256132,
+            0.016007130965590477,
+            0.020541150122880936,
+            0.005616303067654371,
+            -0.016033854335546494,
+            -0.011116806417703629,
+            0.011758159846067429,
+            0.0015733219916000962,
+            0.0006396839744411409,
+            -0.008057012222707272,
+            0.006324464920908213,
+            -0.002256987849250436,
+            0.012043206952512264,
+            0.011838329024612904,
+            0.017298748716711998,
+            0.012898345477879047,
+            -0.005723195616155863,
+            0.03144416958093643,
+            0.01614074781537056,
+            0.0009007908520288765,
+            0.015134177170693874,
+            -0.04368334636092186,
+            -0.01027948223054409,
+            -0.012078837491571903,
+            0.01754816249012947,
+            -0.014751146547496319,
+            -0.016488147899508476,
+            -0.013183391653001308,
+            0.010653605684638023,
+            -0.0032446286641061306,
+            0.005473779980093241,
+            0.0251375213265419,
+            -0.013414992019534111,
+            0.004393721465021372,
+            -0.002739116782322526,
+            0.023623213171958923,
+            0.0030263899825513363,
+            0.00907694362103939,
+            0.007495827041566372,
+            0.005104110576212406,
+            -0.0027057130355387926,
+            0.01963256485760212,
+            -0.009219466708600521,
+            0.036485929042100906,
+            -0.01053780596703291,
+            -0.009317451156675816,
+            -0.01582006923854351,
+            -0.017637239769101143,
+            -0.012354975566267967,
+            0.004752256441861391,
+            -0.000690346525516361,
+            0.0012570987455546856,
+            -0.00821289699524641,
+            0.013637684285640717,
+            0.03140854090452194,
+            -0.0013417218578979373,
+            0.015160900540649891,
+            0.007856588810682297,
+            -0.0006513753323815763,
+            0.01543703954666853,
+            0.003883755998685956,
+            -0.004489479120820761,
+            0.004729987122118473,
+            0.024513982236385345,
+            0.0004924286040477455,
+            0.00767843471840024,
+            -0.0010744909523054957,
+            -0.009602497331798077,
+            -0.01156219094991684,
+            0.010831759311258793,
+            0.027115030214190483,
+            0.017334379255771637,
+            0.001331700710579753,
+            -0.0118472371250391,
+            -0.00017815394676290452,
+            -0.0009375351364724338,
+            0.016791008412837982,
+            -0.010965375229716301,
+            -0.006689680740237236,
+            0.013121037743985653,
+            -0.009085850790143013,
+            0.007611627224832773,
+            -0.01662176288664341,
+            0.0013907141983509064,
+            0.00722859613597393,
+            0.014270130544900894,
+            -0.0045206560753285885,
+            0.006070595700293779,
+            -0.00707716541364789,
+            0.010555621236562729,
+            -0.0127290990203619,
+            0.011277144774794579,
+            -0.005723195616155863,
+            0.00046709735761396587,
+            0.028807492926716805,
+            -0.006475895643234253,
+            0.008195081725716591,
+            0.00011621761223068461,
+            -0.0034717749804258347,
+            -0.017494717612862587,
+            0.025761060416698456,
+            0.004084179177880287,
+            0.005914710927754641,
+            0.00020014481560792774,
+            0.010288390330970287,
+            -0.008301973342895508,
+            -0.001823851023800671,
+            0.022304873913526535,
+            0.011927406303584576,
+            0.0038703943137079477,
+            -0.0005873512709513307,
+            0.008003565482795238,
+            -0.012221360579133034,
+            0.015134177170693874,
+            0.01899120956659317,
+            0.008644919842481613,
+            0.0036543826572597027,
+            0.014599715359508991,
+            -0.013094314374029636,
+            0.008466766215860844,
+            -0.009072489105165005,
+            0.029163800179958344,
+            0.012640022672712803,
+            -0.016523778438568115,
+            0.019294071942567825,
+            0.030767185613512993,
+            -0.004930410534143448,
+            -0.010430913418531418,
+            0.0015844566514715552,
+            -0.017414547502994537,
+            -0.0021567761432379484,
+            -0.014920393005013466,
+            -0.014768961817026138,
+            0.005126379895955324,
+            -0.015312331728637218,
+            0.02497718296945095,
+            -0.017280932515859604,
+            -0.031657956540584564,
+            -0.004309098701924086,
+            -0.00825298111885786,
+            -0.02818395383656025,
+            -0.008301973342895508,
+            0.006190849468111992,
+            0.014849130995571613,
+            -0.0019930971320718527,
+            -0.0144928228110075,
+            0.013664407655596733,
+            0.009085850790143013,
+            0.01359314564615488,
+            -0.017316563054919243,
+            -0.01606057770550251,
+            -0.015615193173289299,
+            0.017316563054919243,
+            -0.010956467129290104,
+            -0.016470331698656082,
+            0.014270130544900894,
+            0.003759048180654645,
+            0.005464872345328331,
+            0.0017503624549135566,
+            -0.014519546180963516,
+            -0.016354531049728394,
+            -0.027845460921525955,
+            0.010626882314682007,
+            -0.0027970168739557266,
+            -0.010974282398819923,
+            0.0029194976668804884,
+            0.011437483131885529,
+            0.00939762033522129,
+            0.005749918520450592,
+            -0.018242964521050453,
+            0.011268236674368382,
+            0.01311213057488203,
+            -0.018528010696172714,
+            -0.007429019547998905,
+            0.005451510660350323,
+            -0.009727205149829388,
+            0.03798241913318634,
+            0.01345062255859375,
+            -0.015205439180135727,
+            -0.0030397516675293446,
+            0.004930410534143448,
+            0.01686227135360241,
+            0.010056789964437485,
+            -0.018848687410354614,
+            -0.001327246893197298,
+            0.021895119920372963,
+            -0.004549606237560511,
+            0.004366998560726643,
+            -3.483918590063695e-06,
+            0.042614422738552094,
+            0.0033426133450120687,
+            -0.0056875646114349365,
+            -0.013138853013515472,
+            0.002210222417488694,
+            0.00997662078589201,
+            0.013023053295910358,
+            0.012239175848662853,
+            -0.007326581049710512,
+            0.0025008360389620066,
+            0.02020265720784664,
+            -0.010395282879471779,
+            0.0020955358631908894,
+            0.0030352978501468897,
+            0.005936980247497559,
+            -0.016880085691809654,
+            0.026242075487971306,
+            -0.0009653716697357595,
+            0.00692573469132185,
+            0.0004620867839548737,
+            -0.009923174977302551,
+            0.014670977368950844,
+            0.024122044444084167,
+            0.0008417773642577231,
+            0.015624100342392921,
+            -0.012435144744813442,
+            0.01831422559916973,
+            0.007901127450168133,
+            0.00782986544072628,
+            -0.015330146998167038,
+            -0.010092420503497124,
+            0.006952457595616579,
+            -0.0028816401027143,
+            -0.002538693603128195,
+            -0.0039839674718678,
+            -0.012372790835797787,
+            -0.011936314404010773,
+            -0.021129056811332703,
+            0.004373679403215647,
+            0.00836878176778555,
+            -0.006132949609309435,
+            0.006850019097328186,
+            0.004309098701924086,
+            -0.0033671094570308924,
+            0.0005984858726151288,
+            -0.013682222925126553,
+            -0.008707273751497269,
+            0.0033470671623945236,
+            0.002696805400773883,
+            0.004191071726381779,
+            -0.007856588810682297,
+            0.020184841006994247,
+            0.006609511096030474,
+            -0.006110680289566517,
+            0.007874404080212116,
+            -0.019062472507357597,
+            -0.0028660516254603863,
+            -0.012105559930205345,
+            0.00011127662583021447,
+            0.0042200214229524136,
+            0.01638125441968441,
+            0.0031243746634572744,
+            -0.023747920989990234,
+            0.0068322038277983665,
+            -0.01202539075165987,
+            0.008965597487986088,
+            -0.003741232678294182,
+            -0.0005160897271707654,
+            -0.012693468481302261,
+            0.01594477705657482,
+            0.01535686943680048,
+            -0.011419667862355709,
+            -0.021093426272273064,
+            0.017637239769101143,
+            -0.03103441745042801,
+            0.022108905017375946,
+            0.010671420954167843,
+            0.0036855596117675304,
+            -0.019810717552900314,
+            0.017120594158768654,
+            0.012399514205753803,
+            0.01101882103830576,
+            -0.0032936211209744215,
+            0.011294960044324398,
+            0.021627888083457947,
+            -0.007941211573779583,
+            -0.0010410870891064405,
+            -9.525320820102934e-06,
+            -0.010430913418531418,
+            -0.0002900568943005055,
+            -0.007175149861723185,
+            0.014421561732888222,
+            0.00836878176778555,
+            -0.008836435154080391,
+            -0.013548607006669044,
+            0.01228371448814869,
+            -0.011482021771371365,
+            0.007117250002920628,
+            -0.005985972471535206,
+            0.007219688501209021,
+            0.004342502448707819,
+            0.01770850270986557,
+            -0.021057795733213425,
+            0.0005670305690728128,
+            0.005237726029008627,
+            -0.022946227341890335,
+            0.007041534408926964,
+            -0.024585243314504623,
+            0.014902577735483646,
+            -0.025351306423544884,
+            -0.005549495108425617,
+            0.02046988718211651,
+            0.014599715359508991,
+            0.0033092093653976917,
+            -0.018011363223195076,
+            -0.007059350144118071,
+            -0.013619869016110897,
+            0.002101103076711297,
+            -0.022429581731557846,
+            -0.013388268649578094,
+            -0.015303423628211021,
+            0.009531236253678799,
+            0.011027729138731956,
+            0.0024028513580560684,
+            -0.010582344606518745,
+            0.007535911630839109,
+            -0.02168133482336998,
+            -0.010617975145578384,
+            -0.014653162099421024,
+            -0.017681779339909554,
+            0.001418550731614232,
+            0.02661619894206524,
+            -0.0027413438074290752,
+            0.01879524067044258,
+            0.011954129673540592,
+            0.01471551600843668,
+            -0.0031733671203255653,
+            -0.010938651859760284,
+            -0.00923728197813034,
+            -0.0027925630565732718,
+            -0.003554171184077859,
+            -0.018777426332235336,
+            -0.012559852562844753,
+            0.00489923357963562,
+            -0.0019853028934448957,
+            0.003981740679591894,
+            0.007874404080212116,
+            0.00841331947594881,
+            -0.01100100576877594,
+            -0.010751590132713318,
+            0.016630670055747032,
+            -0.025671983137726784,
+            -0.017405640333890915,
+            0.018278595060110092,
+            -0.011232606135308743,
+            -0.0012348295422270894,
+            0.019864164292812347,
+            -0.010956467129290104,
+            -0.007491373457014561,
+            0.010493267327547073,
+            -0.003865940496325493,
+            0.010528897866606712,
+            0.022180166095495224,
+            0.006484803277999163,
+            -0.016087301075458527,
+            0.008595927618443966,
+            0.010609067045152187,
+            -0.014163237996399403,
+            -0.02734662964940071,
+            -0.0026010475121438503,
+            0.004560741130262613,
+            -0.020701488479971886,
+            -0.0026945783756673336,
+            0.0004078055208083242,
+            0.00722859613597393,
+            -0.0003855362592730671,
+            0.004925956483930349,
+            -0.018100440502166748,
+            -0.007139519322663546,
+            0.003429463366046548,
+            0.00868945848196745,
+            -0.022215796634554863,
+            0.0013873737771064043,
+            -0.013227930292487144,
+            0.02216234989464283,
+            -0.003257990116253495,
+            0.007415657863020897,
+            -0.0012548718368634582,
+            -0.0074201119132339954,
+            0.008769627660512924,
+            0.012069929391145706,
+            0.005972610786557198,
+            0.03295847773551941,
+            -0.003894890658557415,
+            -0.015009469352662563,
+            0.014234500005841255,
+            -0.025885768234729767,
+            -0.0007337715360336006,
+            -0.0038325367495417595,
+            0.010626882314682007,
+            -0.011036636307835579,
+            -0.011188067495822906,
+            0.02203764207661152,
+            0.02100434899330139,
+            -0.006930188275873661,
+            -0.0002297907485626638,
+            -0.01750362478196621,
+            0.01662176288664341,
+            -0.00824852753430605,
+            0.0015833431389182806,
+            0.007976843044161797,
+            -0.023320350795984268,
+            -0.012631114572286606,
+            -0.006021603476256132,
+            0.024870289489626884,
+            -0.01934751868247986,
+            -0.016559408977627754,
+            -0.012515314854681492,
+            0.00012282878742553294,
+            -0.02709721401333809,
+            -0.0075225504115223885,
+            0.007063803728669882,
+            -0.012925068847835064,
+            -0.007553727366030216,
+            0.007954573258757591,
+            0.009477789513766766,
+            -0.0021033298689872026,
+            -0.02520878240466118,
+            -0.011188067495822906,
+            0.013210115022957325,
+            0.003939428832381964,
+            0.006190849468111992,
+            -0.004364771768450737,
+            0.006168580148369074,
+            -0.006279926281422377,
+            0.0032513095065951347,
+            0.00899231992661953,
+            -0.0016490374691784382,
+            -0.006342280190438032,
+            0.02219798043370247,
+            0.005522772204130888,
+            -0.018180610612034798,
+            0.01433248445391655,
+            0.01570427045226097,
+            -0.012185730040073395,
+            -0.008489035069942474,
+            3.824046507361345e-05,
+            -0.013931638561189175,
+            -0.0008918831590563059,
+            -0.006475895643234253,
+            0.03196081891655922,
+            -0.022874966263771057,
+            0.003857032861560583,
+            0.008685004897415638,
+            -0.0002842112153302878,
+            -0.004903687164187431,
+            0.002912817057222128,
+            0.0054826876148581505,
+            -0.0012793679488822818,
+            0.00577218784019351,
+            0.00812827330082655,
+            -0.014216684736311436,
+            -0.006337826605886221,
+            0.013691130094230175,
+            0.002291504992172122,
+            0.017806487157940865,
+            0.016452517360448837,
+            1.9311608411953785e-05,
+            -0.03313663229346275,
+            0.004193298518657684,
+            0.0007877744501456618,
+            -0.021627888083457947,
+            -0.0038169482722878456,
+            -0.006106226239353418,
+            0.00967375934123993,
+            0.010324020870029926,
+            0.01614074781537056,
+            0.0029194976668804884,
+            0.007143972907215357,
+            0.006141857244074345,
+            0.010288390330970287,
+            -0.030927523970603943,
+            -0.006444718688726425,
+            -0.008333150297403336,
+            -0.006453626789152622,
+            0.012622206471860409,
+            0.012631114572286606,
+            0.029983308166265488,
+            0.028736229985952377,
+            -0.014751146547496319,
+            -0.019204994663596153,
+            0.004242290742695332,
+            0.011027729138731956,
+            0.008640466257929802,
+            -0.007290950044989586,
+            0.006462534423917532,
+            8.504067227477208e-05,
+            -0.02526222914457321,
+            0.0017392279114574194,
+            0.01375348400324583,
+            -0.017530348151922226,
+            -0.010617975145578384,
+            -0.018652718514204025,
+            0.014626438729465008,
+            0.02116468921303749,
+            -0.00392384035512805,
+            0.007188511546701193,
+            -0.010591251775622368,
+            -0.009317451156675816,
+            -0.04500168561935425,
+            0.0053000799380242825,
+            0.009054673835635185,
+            0.022215796634554863,
+            0.001312771812081337,
+            -0.004001782741397619,
+            -6.158837641123682e-05,
+            0.007001449819654226,
+            -0.0033960596192628145,
+            0.011464206501841545,
+            0.013210115022957325,
+            -0.018759610131382942,
+            -0.016042763367295265,
+            0.004150986671447754,
+            0.0023115472868084908,
+            -0.010172589682042599,
+            0.005968157202005386,
+            0.021253764629364014,
+            0.01586460880935192,
+            -0.007241957820951939,
+            0.004431579262018204,
+            0.005625210702419281,
+            -0.014225591905415058,
+            -0.012034298852086067,
+            -0.0045429253950715065,
+            -0.011517652310431004,
+            -0.024193305522203445,
+            0.017191855236887932,
+            -0.013824746012687683,
+            0.006395726464688778,
+            0.009869728237390518,
+            0.011223698034882545,
+            0.0018450067145749927,
+            0.023177828639745712,
+            0.00811045803129673,
+            -0.008738450706005096,
+            0.011036636307835579,
+            -0.026170814409852028,
+            0.012684560380876064,
+            -0.00023368786787614226,
+            0.011259329505264759,
+            0.0001613128260942176,
+            -0.03301192447543144,
+            -0.013512976467609406,
+            0.008172811940312386,
+            -0.006622872781008482,
+            0.009896451607346535,
+            0.014626438729465008,
+            -0.008288612589240074,
+            -0.00100879673846066,
+            -0.022340504452586174,
+            0.00445830263197422,
+            -0.03147980198264122,
+            0.00011357313633197919,
+            -0.004716625437140465,
+            0.010929744690656662,
+            -0.005362433847039938,
+            -0.0240151509642601,
+            0.0026789898984134197,
+            -0.012720191851258278,
+            -0.005825634114444256,
+            0.0010616861982271075,
+            -0.0005828974535688758,
+            0.010876297950744629,
+            -0.00593252619728446,
+            0.01375348400324583,
+            -0.007348849903792143,
+            0.005215456709265709,
+            -0.0007688456098549068,
+            0.006792119238525629,
+            -0.006658503785729408,
+            -0.012907253578305244,
+            0.00238948967307806,
+            -0.004845787305384874,
+            0.021342841908335686,
+            0.010243851691484451,
+            -0.0006380138220265508,
+            0.004362544510513544,
+            0.020291734486818314,
+            0.004672087263315916,
+            0.013165576383471489,
+            0.006676319055259228,
+            0.0032000902574509382,
+            -0.015285608358681202,
+            0.009736113250255585,
+            -0.007371119223535061,
+            0.009477789513766766,
+            0.002870505442842841,
+            -0.0010310659417882562,
+            0.014599715359508991,
+            0.008364327251911163,
+            -0.009905358776450157,
+            0.011767067946493626,
+            0.018581455573439598,
+            -0.008408865891397,
+            -0.003220132552087307,
+            -0.0127290990203619,
+            0.011294960044324398,
+            0.027204107493162155,
+            0.004260106012225151,
+            -0.01831422559916973,
+            0.014145422726869583,
+            0.00010097710037371144,
+            -0.011713622137904167,
+            0.026046106591820717,
+            -0.014751146547496319,
+            -0.025689799338579178,
+            -0.013379361480474472,
+            -0.007273134775459766,
+            -0.01750362478196621,
+            0.021271580830216408,
+            -0.011473113670945168,
+            0.012212452478706837,
+            0.004193298518657684,
+            0.03119475580751896,
+            0.010689236223697662,
+            0.01212337613105774,
+            -0.003015255555510521,
+            -0.025351306423544884,
+            0.009379805065691471,
+            0.0045874640345573425,
+            -0.04044094309210777,
+            0.0005979291745461524,
+            -0.003004120895639062,
+            0.029377585276961327,
+            0.0010550053557381034,
+            -0.0014898123918101192,
+            -0.0038971174508333206,
+            -0.020380811765789986,
+            -0.015686454251408577,
+            0.007322126999497414,
+            -0.028682785108685493,
+            0.013397176750004292,
+            -0.00507738720625639,
+            -0.012150098569691181,
+            0.002214676234871149,
+            0.010831759311258793,
+            -0.019044656306505203,
+            0.01879524067044258,
+            0.015846792608499527,
+            0.025030629709362984,
+            0.01069814432412386,
+            -0.008885428309440613,
+            0.020006688311696053,
+            0.0010210447944700718,
+            -0.006591695826500654,
+            0.0020688127260655165,
+            -1.8998447558260523e-05,
+            0.0015410316409543157,
+            -0.0033671094570308924,
+            0.003015255555510521,
+            0.01598040945827961,
+            -0.01738782413303852,
+            0.019418779760599136,
+            -0.01702260971069336,
+            0.016007130965590477,
+            0.0005347402184270322,
+            0.012925068847835064,
+            4.071095827384852e-05,
+            0.0011958583490923047,
+            -0.029840786010026932,
+            -0.0009653716697357595,
+            -0.03915823623538017,
+            -0.011312775313854218,
+            -0.0016679662512615323,
+            -0.018091533333063126,
+            -0.0037679558154195547,
+            0.009032404981553555,
+            -0.01487585436552763,
+            -0.011588914319872856,
+            0.016069484874606133,
+            0.007598265539854765,
+            0.007375573273748159,
+            0.001965260598808527,
+            -8.03084549261257e-05,
+            -0.005162010435014963,
+            -0.020808380097150803,
+            -0.012951791286468506,
+            -0.025155337527394295,
+            -0.025190968066453934,
+            -0.0035742134787142277,
+            0.007584904320538044,
+            -0.004832425620406866,
+            -0.008564750663936138,
+            0.017761947587132454,
+            -0.019044656306505203,
+            -0.00040140311466529965,
+            0.004242290742695332,
+            0.008172811940312386,
+            0.013103222474455833,
+            -0.007063803728669882,
+            0.007384480908513069,
+            0.0030709286220371723,
+            -0.0013361545279622078,
+            -0.0005183166358619928,
+            0.024104228243231773,
+            -0.008836435154080391,
+            0.011009913869202137,
+            0.007722973357886076,
+            0.011775976046919823,
+            -0.006217572372406721,
+            0.009045766666531563,
+            -0.024104228243231773,
+            0.007687342818826437,
+            -0.011045544408261776,
+            -0.026794353500008583,
+            -0.007148426957428455,
+            0.02071930281817913,
+            -0.0006536022992804646,
+            0.0008662735344842076,
+            -0.0018193970900028944,
+            -0.0004406526277307421,
+            0.023676658049225807,
+            0.01042200531810522,
+            0.0012147872475907207,
+            -0.007700704038143158,
+            -0.029769523069262505,
+            0.005985972471535206,
+            -0.021271580830216408,
+            -0.021699151024222374,
+            0.03288721665740013,
+            0.010368559509515762,
+            0.0027213015127927065,
+            -0.013691130094230175,
+            0.010038974694907665,
+            0.009442158974707127,
+            6.134480645414442e-05,
+            0.014804592356085777,
+            0.003088743891566992,
+            0.001969714416190982,
+            -0.009932082146406174,
+            -0.01918718032538891,
+            0.01614074781537056,
+            0.014581900089979172,
+            -0.01559737790375948,
+            0.036877866834402084,
+            0.016773194074630737,
+            0.012809268198907375,
+            -0.010288390330970287,
+            0.007714065723121166,
+            0.016995886340737343,
+            0.020060133188962936,
+            -0.0070860730484128,
+            0.0034339171834290028,
+            -0.010742682963609695,
+            -0.00023466214770451188,
+            0.03894445300102234,
+            0.03516758978366852,
+            0.008444496430456638,
+            -0.022785888984799385,
+            0.014581900089979172,
+            -0.009335266426205635,
+            -0.01287162210792303,
+            0.010306205600500107,
+            0.01606057770550251,
+            -0.0005923618446104228,
+            0.0014686565846204758,
+            0.009451067075133324,
+            0.0076606194488704205,
+            0.017441270872950554,
+            0.005353525746613741,
+            -0.002378355246037245,
+            0.003402740228921175,
+            -0.0034339171834290028,
+            0.00965594407171011,
+            -0.016808824613690376,
+            -0.005126379895955324,
+            -0.00339828641153872,
+            -0.0016969162970781326,
+            0.005251087248325348,
+            0.009540143422782421,
+            -0.036361221224069595,
+            -0.007415657863020897,
+            -0.033350419253110886,
+            0.004021825268864632,
+            -0.01995324157178402,
+            -0.0035430365242064,
+            -0.010965375229716301,
+            -0.01598040945827961,
+            -0.026562752202153206,
+            -0.016933532431721687,
+            -0.0031266016885638237,
+            0.008903243578970432,
+            -0.003888209816068411,
+            0.003215678734704852,
+            0.003436144208535552,
+            -0.021378472447395325,
+            0.005660841707140207,
+            0.013174484483897686,
+            0.009370896965265274,
+            0.021057795733213425,
+            0.0001496214681537822,
+            0.014368114992976189,
+            -0.014822407625615597,
+            -0.04029842093586922,
+            -0.0007649484905414283,
+            0.00031009921804070473,
+            -0.005175372119992971,
+            -0.00424674479290843,
+            0.004859148990362883,
+            -0.009379805065691471,
+            0.011098990216851234,
+            -0.0021745916455984116,
+            0.0009681553347036242,
+            -0.015722084790468216,
+            -0.010707051493227482,
+            -0.023587582632899284,
+            -0.00505066430196166,
+            -0.013121037743985653,
+            -2.176470479753334e-05,
+            -0.01288943737745285,
+            -0.004952679388225079,
+            -0.005398064386099577,
+            -0.0013461756752803922,
+            -0.006426903419196606,
+            -0.002723528305068612,
+            -0.001202539075165987,
+            -0.013263560831546783,
+            -0.012559852562844753,
+            -0.0013172257458791137,
+            0.004832425620406866,
+            0.0006942436448298395,
+            -0.01098319049924612,
+            0.013414992019534111,
+            -0.014911484904587269,
+            0.0018038086127489805,
+            0.02574324421584606,
+            0.013325914740562439,
+            -0.010484359227120876,
+            0.016479238867759705,
+            0.002199087757617235,
+            0.010128051973879337,
+            -0.02152099646627903,
+            -0.015722084790468216,
+            -0.010110235773026943,
+            0.026990322396159172,
+            0.0012682334054261446,
+            -0.013557515107095242,
+            -0.007206326816231012,
+            -0.01706714741885662,
+            0.004680994898080826,
+            -0.0077407886274158955,
+            0.011375129222869873,
+            -0.020630227401852608,
+            -0.007215234450995922,
+            -0.011473113670945168,
+            -0.007531458046287298,
+            0.0065827881917357445,
+            0.03794679045677185,
+            -0.00289054773747921,
+            -0.007852135226130486,
+            0.005963703151792288,
+            0.010849575512111187,
+            0.028237400576472282,
+            0.0010516649344936013,
+            -0.0014307989040389657,
+            -0.012043206952512264,
+            0.0034717749804258347,
+            0.004551833029836416,
+            -0.01051999069750309,
+            0.017031516879796982,
+            -0.009148204699158669,
+            -0.01934751868247986,
+            -0.004500614013522863,
+            -0.00868945848196745,
+            -0.007353303954005241,
+            0.01753925532102585,
+            0.0023226819466799498,
+            0.003208997892215848,
+            0.0008540254784747958,
+            -0.007250865455716848,
+            -0.008333150297403336,
+            0.011009913869202137,
+            0.08451622724533081,
+            0.00943325087428093,
+            -0.02228705771267414,
+            0.0012526449281722307,
+            0.004943771753460169,
+            0.001479791128076613,
+            -0.014795685186982155,
+            0.06591695547103882,
+            -0.014368114992976189,
+            0.005451510660350323,
+            0.0004428795655258,
+            0.004810156300663948,
+            -0.004447167739272118,
+            0.02136065810918808,
+            0.010199313051998615,
+            -0.01244405284523964,
+            0.002025387715548277,
+            -0.03541700541973114,
+            -0.017280932515859604,
+            0.0034984981175512075,
+            -0.0044404868967831135,
+            -0.004393721465021372,
+            0.013121037743985653,
+            -0.023106565698981285,
+            -0.008163904771208763,
+            0.010956467129290104,
+            -0.012052114121615887,
+            0.011473113670945168,
+            -0.019383149221539497,
+            -0.001312771812081337,
+            -0.0020198202691972256,
+            0.0014519545948132873,
+            -0.009727205149829388,
+            0.0036766519770026207,
+            -0.0027123938780277967,
+            0.0010705938329920173,
+            0.009477789513766766,
+            0.0077853272669017315,
+            -0.003741232678294182,
+            0.017877748236060143,
+            0.013441715389490128,
+            0.0035563979763537645,
+            0.017485808581113815,
+            0.0027480246499180794,
+            0.007651711814105511,
+            0.0003485136548988521,
+            0.01785993203520775,
+            -0.014483915641903877,
+            0.021022165194153786,
+            -0.01834985613822937,
+            -0.02059459500014782,
+            -0.013691130094230175,
+            0.009063581936061382,
+            -0.015561746433377266,
+            0.001760383602231741,
+            -0.011571098119020462,
+            0.029858600348234177,
+            0.010885206051170826,
+            -0.003088743891566992,
+            0.004380360245704651,
+            -0.007045988459140062,
+            0.03930075839161873,
+            0.007856588810682297,
+            0.033225711435079575,
+            0.003155551617965102,
+            0.017886655405163765,
+            -0.002788109239190817,
+            0.004001782741397619,
+            -0.013254653662443161,
+            -0.005821180064231157,
+            0.018278595060110092,
+            -0.009905358776450157,
+            0.019062472507357597,
+            0.004863602574914694,
+            -0.014439377002418041,
+            0.01770850270986557,
+            -0.015410316176712513,
+            -0.02292841300368309,
+            0.013842561282217503,
+            -0.02408641390502453,
+            0.0002925621811300516,
+            0.006230934057384729,
+            -0.0036276597529649734,
+            -0.01051999069750309,
+            0.021752595901489258,
+            -0.0005116358515806496,
+            -0.006907918956130743,
+            -0.010644698515534401,
+            -0.013245745562016964,
+            -0.014279038645327091,
+            0.00895668938755989,
+            -0.004574102349579334,
+            -0.01298742275685072,
+            0.011009913869202137,
+            -0.02526222914457321,
+            0.012862714938819408,
+            -0.012043206952512264,
+            0.023017490282654762,
+            -0.00795011967420578,
+            0.02075493521988392,
+            0.0022825973574072123,
+            -0.0002908919705078006,
+            0.0019853028934448957,
+            0.008863158524036407,
+            -0.003224586369469762,
+            0.016327809542417526,
+            0.004819063935428858,
+            0.004997218027710915,
+            0.004583009984344244,
+            -0.005041756667196751,
+            0.015472670085728168,
+            -0.011749252676963806,
+            -0.0009202764485962689,
+            0.01418996136635542,
+            0.009290727786719799,
+            -0.018207332119345665,
+            0.025850137695670128,
+            0.0038748481310904026,
+            0.015303423628211021,
+            -0.023266904056072235,
+            0.0058924416080117226,
+            0.012622206471860409,
+            0.002320455154404044,
+            0.01288943737745285,
+            -0.004729987122118473,
+            0.022946227341890335,
+            0.018902134150266647,
+            0.004734441172331572,
+            -0.0025854590348899364,
+            0.013040868565440178,
+            -0.002086627995595336,
+            -0.0016456970479339361,
+            0.011891775764524937,
+            -0.0034940443001687527,
+            0.004077498335391283,
+            0.001429685391485691,
+            -0.0006591695710085332,
+            0.016755377873778343,
+            -0.01399399247020483,
+            0.014216684736311436,
+            0.006030511111021042,
+            -0.011063359677791595,
+            -0.013896007090806961,
+            0.007099434733390808,
+            -0.015080731362104416,
+            0.0025097436737269163,
+            0.023177828639745712,
+            -0.0014775642193853855,
+            -0.00865382794290781,
+            0.0048502408899366856,
+            0.012381698936223984,
+            -0.0058479029685258865,
+            -0.010484359227120876,
+            0.006181941833347082,
+            0.008221804164350033,
+            0.00027766963467001915,
+            0.006596149876713753,
+            0.0068099345080554485,
+            0.007028173189610243,
+            -0.0007922283257357776,
+            -0.008195081725716591,
+            0.0030508863274008036,
+            0.020933087915182114,
+            -0.0008629331714473665,
+            -0.009210558608174324,
+            -0.03643248230218887,
+            -0.003191182389855385,
+            -0.014679884538054466,
+            0.003086517099291086,
+            0.00030870738555677235,
+            -0.0008350965799763799,
+            -0.004493933171033859,
+            -0.009638127870857716,
+            0.009099212475121021,
+            0.0008907697047106922,
+            0.024745581671595573,
+            -0.012693468481302261,
+            -0.015365777537226677,
+            0.003257990116253495,
+            -0.0055405874736607075,
+            -0.010769405402243137,
+            0.007451288867741823,
+            -0.004213340580463409,
+            0.03269124776124954,
+            0.00901013519614935,
+            0.0025052898563444614,
+            -0.014786777086555958,
+            -0.018813056871294975,
+            0.010208221152424812,
+            -0.00245852442458272,
+            -0.005598487798124552,
+            -0.003939428832381964,
+            -0.02994767762720585,
+            0.010324020870029926,
+            -0.00967375934123993,
+            0.0036744249518960714,
+            -0.0044182175770401955,
+            0.009522328153252602,
+            0.003327024867758155,
+            -0.01487585436552763,
+            -0.0030375246424227953,
+            0.013566422276198864,
+            0.015258884988725185,
+            -0.012435144744813442,
+            0.004451621789485216,
+            -0.005999334156513214,
+            -0.012185730040073395,
+            0.009326359257102013,
+            -0.00967375934123993,
+            -0.017655055969953537,
+            0.002447389764711261
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "b2385c01-68d3-4e39-ad76-ab89feac1f0a",
+      "type": "child",
+      "source": {
+        "id": "1e687808-8db8-4320-805d-0282a8d679bc",
+        "properties": {
+          "page_content": "Deepseekmoe: Towards Ultimate Expert Specialization In\n\nMixture-Of-Experts Language Models\n\nDamai Dai∗1,2, Chengqi Deng1, Chenggang Zhao∗1,3, R.X. Xu1, Huazuo Gao1, Deli Chen1, Jiashi Li1, Wangding Zeng1, Xingkai Yu∗1,4, Y. Wu1, Zhenda Xie1, Y.K. Li1, Panpan Huang1, Fuli Luo1, Chong Ruan1, Zhifang Sui2, Wenfeng Liang1 1DeepSeek-AI 2National Key Laboratory for Multimedia Information Processing, Peking University 3Institute for Interdisciplinary Information Sciences, Tsinghua University 4National Key Laboratory for Novel Software Technology, Nanjing University {daidamai, szf}@pku.edu.cn, {wenfeng.liang}@deepseek.com https://github.com/deepseek-ai/DeepSeek-MoE\n\nAbstract\n\nIn the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top- out of experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into ones and activating from them, allowing for a more flexible combination of activated experts; (2) isolating experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5× expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations.\n\nFurther, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\n1. Introduction\n\nRecent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023a). It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017) has emerged as a popular solution. It can\n\nenable parameter scaling, while concurrently keeping computational costs at a modest level.\n\nRecent applications of MoE architectures in Transformers (Vaswani et al., 2017) have yielded successful attempts at scaling language models to a substantial size (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022), accompanied with remarkable performance. These achievements underscore the considerable potential and promise of MoE language models.\n\nDespite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts.\n\nThis architecture manifests two potential issues: (1) Knowledge Hybridity: existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. (2) Knowledge Redundancy: tokens assigned to different experts may require common knowledge. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters.\n\nThese issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models.\n\nIn response to the aforementioned issues, we introduce DeepSeekMoE, an innovative MoE architecture specifically designed towards ultimate expert specialization. Our architecture involves two principal strategies: (1) Fine-Grained Expert Segmentation: while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. (2) Shared Expert Isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.\n\nThrough compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized.\n\nStarting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B (Lepikhin et al., 2021) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5× expert parameters and computation. Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE.\n\nThese studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization.\n\nLeveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B (DeepSeek-AI, 2024), a dense model trained on the same 2T corpus. We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B (Touvron et al., 2023b), which has approximately 2.5 times the activated parameters. Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard1. Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. The experimental results still validate its substantial advantages over the GShard architecture consistently. In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nOur contributions are summarized as follows: - Architectural Innovation. We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation.\n\nEmpirical Validation. We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. Experimental results validate the high 1https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models\n\nScalability. We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B.\n\nAlignment for MoE. We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B.\n\nPublic Release. In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization.\n\n2. Preliminaries: Mixture-Of-Experts For Transformers\n\nWe first introduce a generic MoE architecture commonly used in Transformer language models. A standard Transformer language model is constructed by stacking layers of standard Transformer blocks, where each block can be represented as follows:\n\n$$\\mathbf{u}{1:T}^{l}=\\text{Self-Att}\\left(\\mathbf{h}{1:T}^{l-1}\\right)+\\mathbf{h}{1:T}^{l-1},\\tag{1}$$ $$\\mathbf{h}{t}^{l}=\\text{FFN}\\left(\\mathbf{u}{t}^{l}\\right)+\\mathbf{u}{t}^{l},\\tag{2}$$\n\nwhere denotes the sequence length, Self-Att(·) denotes the self-attention module, FFN(·) denotes the Feed-Forward Network (FFN), u 1: ∈ R× are the hidden states of all tokens after the -th attention module, and h\n\n∈ Ris the output hidden state of the -th token after the -th Transformer block. For brevity, we omit the layer normalization in the above formulations.\n\nA typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022). An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. Then, each token will be assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts. If the -th FFN is substituted with an MoE layer, the computation for its output hidden state h\n\nis expressed as: where denotes the total number of experts, FFN(·) is the -th expert FFN, , denotes the gate value for the -th expert, , denotes the token-to-expert affinity, Topk(·, ) denotes the set comprising highest affinity scores among those calculated for the -th token and all experts, and e\n\nis the centroid of the -th expert in the -th layer. Note that ,is sparse, indicating that only out of gate values are nonzero. This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only experts. Also, in the above formulations, we omit the layer normalization operation for brevity.\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{N}\\left(g_{i,t}\\operatorname{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\operatorname{Topk}({s_{j,t}|1\\leqslant j\\leqslant N},K),\\ 0,&\\text{otherwise},\\end{cases}$$ $$s_{i,t}=\\operatorname{Softmax}{i}\\left(\\mathbf{u}{t}^{l}\\,\\mathbf{e}_{i}^{l}\\right),$$\n\n$$(3)$$\n\n(4) $\\binom{4}{5}$ .\n\n3. Deepseekmoe Architecture\n\nOn top of the generic MoE architecture outlined in Section 2, we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. As illustrated in Figure 2, our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. Both of these strategies are designed to elevate the level of expert specialization.\n\n3.1. Fine-Grained Expert Segmentation\n\nIn scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts.\n\nIn pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. The finer expert segmentation enables a more flexible and adaptable combination of activated experts. To be specific, on top of a typical MoE architecture shown in Figure 2(a), we segment each expert FFN into smaller experts by reducing the FFN intermediate hidden dimension to 1 times its original size. Since each expert becomes smaller, in response, we also increase the number of activated experts to times to keep the same computation cost, as illustrated in Figure 2(b). With the fine-grained expert segmentation, the output of an MoE layer can be expressed as:\n\n$$\\mathbf{h}{t}^{i}=\\sum{i=1}^{\\text{mN}}\\left(g_{i,t}\\text{FFN}{i}\\left(\\mathbf{u}{t}^{i}\\right)\\right)+\\mathbf{u}{t}^{i},\\tag{6}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\text{Tokr}(s_{i,t}|1\\leqslant j\\leqslant m\\text{N}),m\\text{K}),\\ 0,&\\text{otherwise},\\end{cases}$$ (7) $$s_{i,t}=\\text{Softmax}{i}\\left(\\mathbf{u}{t}^{T}\\mathbf{e}_{t}^{i}\\right),\\tag{8}$$ where the total number of expert parameters is equal to $N$ times the number of parameters in a\n\n$$(6)$$\n\n(7) (8) $\\frac{1}{2}$\n\nstandard FFN, and denotes the total number of fine-grained experts. With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to .\n\nFrom a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. As an illustrative example, we consider the case where = 16. A typical top-2 routing strategy can yield 16 2 = 120 possible combinations. By contrast, if each expert is split into 4 smaller experts, the fine-grained routing strategy can yield 64 8 = 4, 426, 165, 368 potential combinations. The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition.\n\n3.2. Shared Expert Isolation\n\nWith a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. This alleviation of redundancy will contribute to a more parameterefficient model with more specialized experts.\n\nTowards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate experts to serve as shared experts. Regardless of the router module, each token will be deterministically assigned to these shared experts. In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by , as depicted in Figure 2(c). With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows:\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{K_{s}}\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)+\\sum_{i=K_{s}+1}^{mN}\\left(g_{i,t}\\;\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},\\tag{9}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\mathrm{Topk}{s_{j,t}|K_{s}+1\\leqslant j\\leqslant mN},mK-K_{s}},\\ 0,&\\text{otherwise},\\end{cases}$$ (10) $$s_{i,t}=\\mathrm{Softmax}{i}\\left(\\mathbf{u}{t}^{l,T}e_{i}^{l}\\right).\\tag{11}$$ $\\mathbf{h}\\mathbf{M}\\mathbf{E}$ the number of closed-compact is $K_{s}$ the total number of routed currents.\n\nFinally, in DeepSeekMoE, the number of shared expert is , the total number of routed experts is − , and the number of nonzero gates is − .\n\nIt is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022). The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint.\n\n3.3. Load Balance Consideration\n\nAutomatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. Firstly, there is a risk of routing collapse (Shazeer et al., 2017), i.e., the model always selects only a few experts, preventing other experts from sufficient training.\n\nSecondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.\n\nExpert-Level Balance Loss. In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. The computation of the balance loss is as follows:\n\n$$\\begin{array}{l}{{{\\mathcal L}{\\mathrm{ExpBal}}=\\alpha{1}\\sum_{i=1}^{N^{\\prime}}f_{i}P_{i},}}\\ {{f_{i}=\\frac{N^{\\prime}}{K^{T}}\\sum_{t=1}^{T}\\mathds{1}\\,(\\mathrm{Token}\\;t\\;\\mathrm{s}\\mathrm{lects}\\;\\mathrm{Expert}\\;i),}}\\ {{P_{i}=\\frac{1}{T}\\sum_{t=1}^{T}s_{i,t},}}\\end{array}$$\n\n(12) $$\\begin{array}{l}\\small\\text{(13)}\\end{array}$$ = (14) $$\\begin{array}{l}\\small\\text{(14)}\\end{array}$$ .\n\n7 where 1 is a hyper-parameter called expert-level balance factor, ′is equal to ( − ) and ′ is equal to ( − ) for brevity. 1(·) denotes the indicator function.\n\nDevice-Level Balance Loss. In addition to the expert-level balance loss, we introduce a devicelevel balance loss. When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. Instead, our primary objective is to ensure balanced computation across the devices. If we partition all routed experts into groups {E1, E2, ..., E}, and deploy each group on a single device, the device-level balance loss is computed as follows:\n\n$$\\mathcal{L}{\\text{DevBal}}=\\alpha{2}\\sum_{i=1}^{D}f_{i}^{\\prime}P_{i}^{\\prime},$$ $$f_{i}^{\\prime}=\\frac{1}{|\\mathcal{E}{i}|}\\sum{j\\in\\mathcal{E}{i}}f{j},$$ $$P_{i}^{\\prime}=\\sum_{j\\in\\mathcal{E}{i}}P{j},$$\n\n(15) $$\\begin{array}{l}\\small\\text{(16)}\\end{array}$$ = (17) .\n\nwhere 2 is a hyper-parameter called device-level balance factor. In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices.\n\n4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n6. Alignment For Deepseekmoe 16B\n\nPrevious research indicates that MoE models typically do not emerge significant gains from fine-tuning (Artetxe et al., 2022; Fedus et al., 2021). However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B.\n\n6.1. Experimental Setup\n\nTraining Data. For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios.\n\nHyper-Parameters. During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).\n\nWe employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10−5 without incorporating any learning rate scheduling strategy.\n\nEvaluation Benchmarks. For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3, with the following adjustments: (1) We exclude Pile (Gao et al., 2020) since chat models are seldom employed for pure language modeling. (2) We exclude\n\nMetric # Shot LLaMA2 DeepSeek DeepSeekMoE SFT 7B Chat 7B Chat 16B # Total Params N/A 6.7B 6.9B 16.4B # Activated Params N/A 6.7B 6.9B 2.8B FLOPs per 4K Tokens N/A 187.9T 183.5T 74.4T HellaSwag (Acc.) 0-shot 67.9 71.0 72.2 PIQA (Acc.) 0-shot 76.9 78.4 79.7 ARC-easy (Acc.) 0-shot 69.7 70.2 69.9 ARC-challenge (Acc.) 0-shot 50.8 50.2 50.0 BBH (EM) 3-shot 39.3 43.1 42.2 RACE-middle (Acc.) 5-shot 63.9 66.1 64.8 RACE-high (Acc.) 5-shot 49.6 50.8 50.6 DROP (EM) 1-shot 40.0 41.7 33.8 GSM8K (EM) 0-shot 63.4 62.6 62.2 MATH (EM) 4-shot 13.5 14.7 15.2 HumanEval (Pass@1) 0-shot 35.4 45.1 45.7 MBPP (Pass@1) 3-shot 27.8 39.0 46.2 TriviaQA (EM) 5-shot 60.1 59.5 63.3 NaturalQuestions (EM) 0-shot 35.2 32.7 35.1 MMLU (Acc.) 0-shot 50.0 49.7 47.2 WinoGrande (Acc.) 0-shot 65.1 68.4 69.0 CLUEWSC (EM) 5-shot 48.4 66.2 68.2 CEval (Acc.) 0-shot 35.1 44.7 40.0 CMMLU (Acc.) 0-shot 36.9 51.2 49.3\n\nCHID (Zheng et al., 2019) due to the observed instability of results, hindering the derivation of solid conclusions. (3) We additionally include BBH (Suzgun et al., 2022) to provide a more comprehensive assessment of the reasoning ability of the chat models.\n\nTable 5 | Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations.\n\n6.2. Evaluations\n\nBaselines. In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. Correspondingly, we construct three chat models, including LLaMA2 SFT 7B3, DeepSeek Chat 7B, and DeepSeekMoE Chat 16B.\n\nSubsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks.\n\nResults. The evaluation results are presented in Table 5. Our key observations include: (1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intensive tasks (TriviaQA, NaturalQuestions). (2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. In addition, it also surpasses DeepSeek Chat 7B. (3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1). However, it is worth noting that, after supervised finetuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. (4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations.\n\n7. Deepseekmoe 145B Ongoing\n\nEncouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available.\n\n7.1. Experimental Setup\n\nTraining Data and Tokenization. For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study.\n\nModel Settings. For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps.\n\nSubsequently, the learning rate keeps constant during the remaining training process. The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. Also, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse.\n\nEvaluation Benchmarks. We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3).\n\n7.2. Evaluations\n\nBaselines. Apart from DeepSeekMoE 145B, we consider three additional models for comparison. DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters.\n\nResults. From the evaluation results presented in Table 6, we have the following observations: (1) Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. (2) On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. (3) At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5.\n\n8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\nAppendices A. Overview Of Hyper-Parameters\n\nWe present the overview of hyper-parameters for DeepSeekMoE across various sizes in Table 7.\n\n# Params # Layers Hidden # Attn # Shared # Routed Relative Sequence Batch Size Learning Size Heads Experts Experts Expert Size Length (Sequence) Rate 2.0B 9 1280 10 1 63 (7 activated) 0.25 2048 2048 1.08e-3 16.4B 28 2048 16 2 64 (6 activated) 0.25 4096 4608 4.2e-4 144.6B 62 4096 32 4 128 (12 activated) 0.125 4096 4608 3.0e-4\n\nTable 7 | Overview of hyper-parameters for DeepSeekMoE across various sizes. The relative expert size is in comparison to a standard FFN.\n\nB. Comparing Deepseekmoe With Larger Models\n\nComparisons among DeepSeekMoE, GShard×1.2, and GShard×1.5 are shown in Table 8. Comparisons among DeepSeekMoE, Dense×4, and Dense×16 are shown in Table 9.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 2.3B 2.8B 1.9B # Activated Expert Params N/A 0.28B 0.35B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.824 1.808 1.808 HellaSwag (Acc.) 0-shot 53.7 54.4 54.8 PIQA (Acc.) 0-shot 71.8 71.1 72.3 ARC-easy (Acc.) 0-shot 46.8 47.3 49.4 ARC-challenge (Acc.) 0-shot 31.7 34.1 34.3 RACE-middle (Acc.) 5-shot 43.7 46.4 44.0 RACE-high (Acc.) 5-shot 31.9 32.4 31.7 HumanEval (Pass@1) 0-shot 3.7 3.0 4.9 MBPP (Pass@1) 3-shot 2.4 2.6 2.2 TriviaQA (EM) 5-shot 15.2 15.7 16.6 NaturalQuestions (EM) 5-shot 4.5 4.7 5.7\n\nTable 8 | Comparison between DeepSeekMoE and larger GShard models.\n\nAt a larger scale of 13B total parameters, we also compare DeepSeekMoE with GShard×1.2 and GShard×1.5, and show results in Table 10. At a larger scale, DeepSeekMoE even outperforms GShard×1.5 distinctly.\n\nMetric # Shot Dense×4 Dense×16 DeepSeekMoE Relative Expert Size N/A 1 1 0.25 # Experts N/A 4 + 0 16 + 0 1 + 63 # Activated Experts N/A 4 + 0 16 + 0 1 + 7 # Total Expert Params N/A 0.47B 1.89B 1.89B # Activated Expert Params N/A 0.47B 1.89B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.908 1.806 1.808 HellaSwag (Acc.) 0-shot 47.6 55.1 54.8 PIQA (Acc.) 0-shot 70.0 71.9 72.3 ARC-easy (Acc.) 0-shot 43.9 51.9 49.4 ARC-challenge (Acc.) 0-shot 30.5 33.8 34.3 RACE-middle (Acc.) 5-shot 42.4 46.3 44.0 RACE-high (Acc.) 5-shot 30.7 33.0 31.7 HumanEval (Pass@1) 0-shot 1.8 4.3 4.9 MBPP (Pass@1) 3-shot 0.2 2.2 2.2 TriviaQA (EM) 5-shot 9.9 16.5 16.6 NaturalQuestions (EM) 5-shot 3.0 6.3 5.7\n\nTable 9 | Comparison between DeepSeekMoE and larger dense baselines.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 15.9B 19.8B 13.3B # Activated Expert Params N/A 2.37B 2.82B 2.05B # Training Tokens N/A 100B 100B 100B HellaSwag (Acc.) 0-shot 66.6 67.7 69.1 PIQA (Acc.) 0-shot 75.6 76.0 75.7 ARC-easy (Acc.) 0-shot 56.8 56.8 58.8 ARC-challenge (Acc.) 0-shot 39.9 37.6 38.5 RACE-middle (Acc.) 5-shot 51.6 50.6 52.4 RACE-high (Acc.) 5-shot 37.4 36.3 38.5 HumanEval (Pass@1) 0-shot 6.1 6.1 9.8 MBPP (Pass@1) 3-shot 7.0 11.6 10.6 TriviaQA (EM) 5-shot 36.5 36.7 38.2 NaturalQuestions (EM) 5-shot 12.6 12.1 13.7\n\nTable 10 | Comparison between DeepSeekMoE and larger GShard models at a larger scale.\n\nC. Training Benchmark Curves Of Deepseekmoe 16B\n\nWe present the benchmark curves during training of DeepSeekMoE 16B and DeepSeek 7B (Dense) in Figure 7 for reference.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/DeepSeekMoE_2.md"
+          },
+          "summary": "DeepSeekMoE is an innovative Mixture-of-Experts (MoE) architecture designed to enhance expert specialization in language models. It employs fine-grained expert segmentation and shared expert isolation to improve performance and reduce computational costs. Starting with a model of 2B parameters, DeepSeekMoE demonstrates comparable performance to larger models like GShard 2.9B and LLaMA2 7B, using significantly fewer computations. The architecture scales up to 16B and 145B parameters, maintaining advantages over traditional MoE architectures. DeepSeekMoE 16B achieves similar results to dense models with only 40% of the computations, and its chat model version shows adaptability in bilingual scenarios. The architecture's public release aims to contribute to advancements in large-scale language models.",
+          "headlines": {
+            "1. Introduction": [],
+            "2. Preliminaries: Mixture-Of-Experts For Transformers": [],
+            "3. Deepseekmoe Architecture": [
+              "3.1. Fine-Grained Expert Segmentation",
+              "3.2. Shared Expert Isolation",
+              "3.3. Load Balance Consideration"
+            ],
+            "4. Validation Experiments": [
+              "4.1. Experimental Setup",
+              "4.1.1. Training Data And Tokenization",
+              "4.1.2. Infrastructures",
+              "4.1.3. Hyper-Parameters",
+              "4.1.4. Evaluation Benchmarks",
+              "4.2. Evaluations",
+              "4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models",
+              "4.4. Ablation Studies",
+              "4.5. Analysis On Expert Specialization"
+            ],
+            "5. Scaling Up To Deepseekmoe 16B": [
+              "5.1. Experimental Setup",
+              "5.1.1. Training Data And Tokenization",
+              "5.1.2. Hyper-Parameters",
+              "5.1.3. Evaluation Benchmarks",
+              "5.2. Evaluations",
+              "5.2.1. Internal Comparison With Deepseek 7B",
+              "5.2.2. Comparison With Open Source Models"
+            ],
+            "6. Alignment For Deepseekmoe 16B": [
+              "6.1. Experimental Setup",
+              "6.2. Evaluations"
+            ],
+            "7. Deepseekmoe 145B Ongoing": [
+              "7.1. Experimental Setup",
+              "7.2. Evaluations"
+            ],
+            "8. Related Work": [],
+            "9. Conclusion": [],
+            "Appendices": [
+              "A. Overview Of Hyper-Parameters",
+              "B. Comparing Deepseekmoe With Larger Models",
+              "C. Training Benchmark Curves Of Deepseekmoe 16B"
+            ]
+          },
+          "summary_embedding": [
+            -0.012465757317841053,
+            -0.022808987647294998,
+            -0.018792515620589256,
+            -0.02308654971420765,
+            0.01797616109251976,
+            0.007200260180979967,
+            0.006490030791610479,
+            0.042613785713911057,
+            -0.014049487188458443,
+            0.03187054023146629,
+            -0.021486490964889526,
+            -0.023592690005898476,
+            0.009045224636793137,
+            -0.014441338367760181,
+            0.005865518003702164,
+            -0.007363531738519669,
+            -0.008743173442780972,
+            -0.005163452122360468,
+            0.018841497600078583,
+            -0.03100520372390747,
+            0.011110605672001839,
+            -0.046075135469436646,
+            -0.020017050206661224,
+            -0.021845687180757523,
+            0.016425084322690964,
+            0.037389107048511505,
+            0.022547753527760506,
+            -0.0028062239289283752,
+            -0.06844329088926315,
+            0.034678805619478226,
+            0.035168617963790894,
+            0.02052319049835205,
+            0.0005438972148112953,
+            -0.00031302153365686536,
+            -0.024833550676703453,
+            -0.028556134551763535,
+            0.03347059711813927,
+            -0.004379750229418278,
+            -0.01453930139541626,
+            0.015976088121533394,
+            0.0414055772125721,
+            0.012106560170650482,
+            -0.02754385396838188,
+            0.0014245413476601243,
+            0.015151568688452244,
+            0.0055961208418011665,
+            -0.01678428053855896,
+            -0.0022960014175623655,
+            -0.002424577483907342,
+            -0.021225256845355034,
+            -0.01400050614029169,
+            0.018825169652700424,
+            0.001296985661610961,
+            -0.008865626528859138,
+            -0.0542386956512928,
+            0.007330877240747213,
+            0.03840138763189316,
+            0.05623060464859009,
+            0.044148534536361694,
+            -0.020180322229862213,
+            0.03369917720556259,
+            -0.030025575309991837,
+            -0.02501314878463745,
+            0.011028970591723919,
+            0.003983817528933287,
+            -0.018531281501054764,
+            0.01784554310142994,
+            -0.005636938381940126,
+            -0.0006556359585374594,
+            0.01751900091767311,
+            -0.03941367194056511,
+            0.020751770585775375,
+            -0.01887415163218975,
+            0.031266435980796814,
+            0.02173139713704586,
+            0.0073798587545752525,
+            0.01872720755636692,
+            0.01678428053855896,
+            0.0036858476232737303,
+            -0.04107903689146042,
+            0.009118696674704552,
+            0.016523046419024467,
+            0.014196431264281273,
+            0.016269976273179054,
+            0.06553706526756287,
+            -0.05391215533018112,
+            -0.004485876299440861,
+            -0.05169166624546051,
+            0.0012561678886413574,
+            -0.004971608519554138,
+            -0.03670336678624153,
+            0.01048201136291027,
+            0.028425518423318863,
+            0.021372200921177864,
+            -0.01063711941242218,
+            0.024392718449234962,
+            -0.022743679583072662,
+            0.0032001156359910965,
+            -0.04062187671661377,
+            0.0019653772469609976,
+            0.004518530797213316,
+            0.016531210392713547,
+            0.0070778070949018,
+            0.004273623693734407,
+            -0.016702644526958466,
+            0.006045116577297449,
+            -0.01905374974012375,
+            0.03291547670960426,
+            -0.008465612307190895,
+            0.08653374016284943,
+            0.008751336485147476,
+            0.004738946910947561,
+            0.004281787667423487,
+            0.02398454025387764,
+            0.012131051160395145,
+            -0.00810233410447836,
+            -0.04718538001179695,
+            -0.015690363943576813,
+            0.011061624623835087,
+            -0.0383034273982048,
+            0.018335357308387756,
+            -0.008759500458836555,
+            0.01282495353370905,
+            -0.04107903689146042,
+            -0.06344719231128693,
+            0.01498013362288475,
+            0.001784758409485221,
+            -0.004065453074872494,
+            -0.0002903166168835014,
+            -0.018923133611679077,
+            0.003930754493921995,
+            -0.004200151655822992,
+            -0.010645282454788685,
+            -0.021502818912267685,
+            0.03159298002719879,
+            0.03657275065779686,
+            0.06527583301067352,
+            0.036801330745220184,
+            -0.0030164355412125587,
+            0.005775718949735165,
+            0.019690508022904396,
+            -0.020784424617886543,
+            0.003481758525595069,
+            -0.01214737817645073,
+            -0.021992631256580353,
+            -0.03820546343922615,
+            0.010971825569868088,
+            0.07379858940839767,
+            -0.029029620811343193,
+            0.013437220826745033,
+            -0.017616963014006615,
+            0.04205866530537605,
+            -0.004567512311041355,
+            0.026727497577667236,
+            -0.016669990494847298,
+            0.004059330560266972,
+            -0.025388672947883606,
+            0.01620466820895672,
+            -0.024964168667793274,
+            0.012571883387863636,
+            -0.000885746325366199,
+            -0.04222193360328674,
+            0.00011116473615402356,
+            0.026890767738223076,
+            -0.010971825569868088,
+            0.02261306345462799,
+            0.0004949158756062388,
+            -0.006302268709987402,
+            -0.05097327008843422,
+            0.038727931678295135,
+            -0.003912386484444141,
+            0.018645571544766426,
+            -0.0018959869630634785,
+            0.0163107942789793,
+            -0.0016408757073804736,
+            0.013216804713010788,
+            0.009543201886117458,
+            0.007114543113857508,
+            -0.033503253012895584,
+            0.02697240374982357,
+            -0.02504580281674862,
+            0.028049994260072708,
+            0.03435226157307625,
+            -0.0030184765346348286,
+            0.03255628049373627,
+            0.019249675795435905,
+            0.03840138763189316,
+            0.0327032245695591,
+            0.030335791409015656,
+            -0.00824927818030119,
+            0.013878053054213524,
+            0.008200296200811863,
+            0.03608293831348419,
+            0.013429056853055954,
+            0.019723162055015564,
+            -0.009730963967740536,
+            -0.014041324146091938,
+            0.048785436898469925,
+            0.007318632211536169,
+            -0.0179598331451416,
+            -0.005441013257950544,
+            -0.03510330989956856,
+            -0.002832755446434021,
+            -0.014425011351704597,
+            0.0323440283536911,
+            0.02158445306122303,
+            -0.01647406443953514,
+            0.035135965794324875,
+            0.024980494752526283,
+            -0.03751972317695618,
+            -0.005232842173427343,
+            -0.02026195637881756,
+            -0.023298801854252815,
+            -0.04215662553906441,
+            0.011624909937381744,
+            0.012833117507398129,
+            -0.013976015150547028,
+            -0.007469657808542252,
+            0.020457882434129715,
+            -0.023625344038009644,
+            -0.0018265966791659594,
+            -0.01356783788651228,
+            0.006547175347805023,
+            -0.020898714661598206,
+            -0.03667071461677551,
+            -0.01048201136291027,
+            0.025862159207463264,
+            0.012637191452085972,
+            0.04362606629729271,
+            -0.013518855907022953,
+            -0.005367541220039129,
+            0.033193036913871765,
+            0.012759645469486713,
+            0.010767736472189426,
+            0.03699725493788719,
+            -0.03027048148214817,
+            0.024621298536658287,
+            -0.020294612273573875,
+            -0.0032878739293664694,
+            0.01707816869020462,
+            -0.0030062312725931406,
+            0.01962519995868206,
+            -0.04535674303770065,
+            -0.024196794256567955,
+            0.04395261034369469,
+            -0.0007107399869710207,
+            0.016302630305290222,
+            -0.005959399044513702,
+            -0.012106560170650482,
+            0.0035899258218705654,
+            0.0027715289033949375,
+            -0.01438419334590435,
+            -0.008620720356702805,
+            0.007473739795386791,
+            -0.01962519995868206,
+            0.03660540655255318,
+            0.03794422745704651,
+            -0.004132802598178387,
+            -0.014131123200058937,
+            -0.03324201703071594,
+            0.010016689077019691,
+            -0.017878197133541107,
+            -0.023951886221766472,
+            0.04078514873981476,
+            -0.032115448266267776,
+            -0.00029312283731997013,
+            0.039478980004787445,
+            0.018808843567967415,
+            -0.014898497611284256,
+            0.00868602842092514,
+            -0.01709449477493763,
+            -0.026302991434931755,
+            -0.017159804701805115,
+            0.03356856107711792,
+            0.024800896644592285,
+            -0.010743245482444763,
+            0.010245268233120441,
+            0.005330805201083422,
+            0.02893165871500969,
+            0.024376392364501953,
+            0.00648594880476594,
+            -0.013812744058668613,
+            0.015029114671051502,
+            0.027641816064715385,
+            -0.0028021421749144793,
+            -7.66609373386018e-05,
+            -0.004010349046438932,
+            -0.007796200457960367,
+            -0.006453294772654772,
+            -0.013069860637187958,
+            -0.005089980084449053,
+            0.0009184005321003497,
+            -0.032295044511556625,
+            0.015967924147844315,
+            -0.005890008993446827,
+            0.010359558276832104,
+            -0.011535110883414745,
+            0.01288209855556488,
+            0.01722511276602745,
+            -0.013494365848600864,
+            -0.0716434046626091,
+            0.014612773433327675,
+            0.02458864450454712,
+            -0.006706364918500185,
+            -0.017943505197763443,
+            0.008481939323246479,
+            0.00016148542636074126,
+            0.038597315549850464,
+            -0.014180104248225689,
+            -0.031103165820240974,
+            0.02680913358926773,
+            -0.017437364906072617,
+            -0.015241367742419243,
+            -0.010661610402166843,
+            -0.00580837344750762,
+            0.04649963974952698,
+            0.01889047957956791,
+            0.011420820839703083,
+            -0.04042595252394676,
+            -0.007449248805642128,
+            0.015078096650540829,
+            0.014506647363305092,
+            -0.04705476388335228,
+            0.0052083516493439674,
+            0.015486273914575577,
+            0.019723162055015564,
+            0.024800896644592285,
+            0.04274440184235573,
+            -0.0021531390957534313,
+            0.01661284640431404,
+            -0.015053605660796165,
+            -0.05136512219905853,
+            -0.03340528905391693,
+            0.029943939298391342,
+            -0.017731253057718277,
+            -0.02906227484345436,
+            -0.01693122461438179,
+            -0.006151242647320032,
+            0.0017939424142241478,
+            0.007126788143068552,
+            -0.01916803978383541,
+            -0.03464614972472191,
+            -0.03970756009221077,
+            0.02277633361518383,
+            0.007983962073922157,
+            0.004269542172551155,
+            -0.042777057737112045,
+            0.0013459670590236783,
+            0.009184005670249462,
+            0.012073906138539314,
+            0.0009954441338777542,
+            -0.019984396174550056,
+            0.015494437888264656,
+            0.0032001156359910965,
+            -0.031070511788129807,
+            -0.035593122243881226,
+            -0.03595232218503952,
+            0.0023919232189655304,
+            0.003781769424676895,
+            0.06279410421848297,
+            0.012988224625587463,
+            -0.03840138763189316,
+            -0.04522612690925598,
+            0.029176564887166023,
+            -0.02052319049835205,
+            -0.016678154468536377,
+            0.0059430720284581184,
+            0.012898425571620464,
+            -0.009877908043563366,
+            0.013192313723266125,
+            0.023560035973787308,
+            0.041601505130529404,
+            -0.005987971555441618,
+            0.012914752587676048,
+            0.023037567734718323,
+            0.012955570593476295,
+            0.0362788625061512,
+            -0.00936360377818346,
+            0.0035409443080425262,
+            0.039936136454343796,
+            -0.012269831262528896,
+            0.03175625205039978,
+            -0.02189466916024685,
+            0.0020398697815835476,
+            0.02249877341091633,
+            0.027184655889868736,
+            -0.0006479826406575739,
+            0.02594379521906376,
+            -0.031543996185064316,
+            0.018368011340498924,
+            -0.014498483389616013,
+            -0.04888340085744858,
+            -0.01557607389986515,
+            0.013102514669299126,
+            0.030384771525859833,
+            0.02756018005311489,
+            -0.0041348435916006565,
+            -0.019723162055015564,
+            -0.01155143789947033,
+            -0.010596301406621933,
+            0.0018419033149257302,
+            0.03285016864538193,
+            0.010506502352654934,
+            0.010890189558267593,
+            -0.011935125105082989,
+            -0.004104230087250471,
+            0.03025415539741516,
+            -0.021192602813243866,
+            0.01650671847164631,
+            0.0041511706076562405,
+            -0.002029665280133486,
+            -0.04320156201720238,
+            -0.035266581922769547,
+            0.007845181971788406,
+            0.011747363954782486,
+            0.014874007552862167,
+            0.0368339866399765,
+            0.025094784796237946,
+            -0.01662100851535797,
+            0.030629679560661316,
+            -0.01905374974012375,
+            0.027984686195850372,
+            0.039054471999406815,
+            0.011624909937381744,
+            -0.018351683393120766,
+            0.012237177230417728,
+            0.002424577483907342,
+            -0.025127438828349113,
+            0.01680060662329197,
+            -0.032735876739025116,
+            0.01738838478922844,
+            -0.02354370802640915,
+            0.025143766775727272,
+            -0.02579685114324093,
+            -0.009290131740272045,
+            -0.004428731743246317,
+            -0.006130834110081196,
+            0.04888340085744858,
+            -0.026727497577667236,
+            -0.0142454132437706,
+            -0.0024694770108908415,
+            -0.0024531499948352575,
+            -0.008718682453036308,
+            0.010000362060964108,
+            -0.045781247317790985,
+            -0.01844964735209942,
+            -0.04016471654176712,
+            0.007355368230491877,
+            0.005412440747022629,
+            -0.030907239764928818,
+            -0.05694899708032608,
+            0.025421326979994774,
+            -0.06331657618284225,
+            -0.019886434078216553,
+            0.051626358181238174,
+            -0.018400665372610092,
+            -0.008212542161345482,
+            0.009641164913773537,
+            0.006375740747898817,
+            -0.029290854930877686,
+            0.0041348435916006565,
+            0.02504580281674862,
+            0.04418119043111801,
+            -0.0045797573402523994,
+            -0.02865409664809704,
+            0.0028674507047981024,
+            -0.027788760140538216,
+            -0.013461710885167122,
+            0.005261414684355259,
+            0.016449574381113052,
+            0.024539662525057793,
+            0.007628847379237413,
+            0.03585435822606087,
+            0.007294141221791506,
+            0.006318595726042986,
+            0.044279150664806366,
+            -0.025666235014796257,
+            -0.027380581945180893,
+            0.02519274689257145,
+            0.04045860469341278,
+            0.023151857778429985,
+            -0.018253721296787262,
+            -0.0142454132437706,
+            0.0053063142113387585,
+            0.0009296254720538855,
+            -0.022139575332403183,
+            0.002808264922350645,
+            -0.013902543112635612,
+            0.005143043119460344,
+            0.03892385587096214,
+            -0.001205145614221692,
+            0.014196431264281273,
+            0.007302304729819298,
+            0.006845145486295223,
+            0.004334850702434778,
+            -0.011731036007404327,
+            -0.023592690005898476,
+            0.00032220553839579225,
+            0.034972693771123886,
+            0.015404638834297657,
+            -0.030515389516949654,
+            -0.00995954405516386,
+            0.034711457788944244,
+            -0.00795130804181099,
+            -0.009118696674704552,
+            -0.009771781973540783,
+            0.003385836724191904,
+            0.02638462744653225,
+            -0.01606588624417782,
+            0.006649219896644354,
+            0.03716052696108818,
+            -0.019870106130838394,
+            0.005543057806789875,
+            -0.025405000895261765,
+            0.0018061877926811576,
+            0.02442537248134613,
+            -0.006216551177203655,
+            0.028098974376916885,
+            5.6315813708351925e-05,
+            0.03025415539741516,
+            -0.004677720367908478,
+            0.008432958275079727,
+            -0.020327266305685043,
+            0.03987082839012146,
+            -0.017633290961384773,
+            0.03402572125196457,
+            0.02067013457417488,
+            -0.01258004643023014,
+            0.04783846437931061,
+            -0.036801330745220184,
+            -0.020784424617886543,
+            0.00584919098764658,
+            0.012743317522108555,
+            0.0028919412288814783,
+            -0.0191517136991024,
+            -0.018400665372610092,
+            -0.010996315628290176,
+            0.02380494214594364,
+            -0.022547753527760506,
+            -0.016865916550159454,
+            -0.007441085297614336,
+            -0.01890680566430092,
+            -0.009673818945884705,
+            -0.007988044060766697,
+            -0.01709449477493763,
+            -0.016547536477446556,
+            0.044573038816452026,
+            -0.06334923207759857,
+            0.016996532678604126,
+            0.017159804701805115,
+            0.011477965861558914,
+            -0.034678805619478226,
+            -0.019510909914970398,
+            -0.017633290961384773,
+            -0.002347023691982031,
+            -0.030743969604372978,
+            0.001702102366834879,
+            0.01725776679813862,
+            0.01138000376522541,
+            -0.05022222548723221,
+            0.012335140258073807,
+            0.02772345207631588,
+            -0.012065742164850235,
+            0.0014949520118534565,
+            -0.023135529831051826,
+            -0.017176130786538124,
+            0.0031776658724993467,
+            0.022123249247670174,
+            0.005975726060569286,
+            -0.008188051171600819,
+            -0.016865916550159454,
+            -0.005241005681455135,
+            -0.006045116577297449,
+            -0.002359268954023719,
+            0.005094061605632305,
+            0.011780017986893654,
+            0.019690508022904396,
+            -0.0024633544962853193,
+            -0.004436895251274109,
+            -0.006396149750798941,
+            0.027364253997802734,
+            -0.001450052484869957,
+            -0.008416631259024143,
+            0.050907962024211884,
+            0.03556047007441521,
+            0.006094098091125488,
+            -0.01108611561357975,
+            -0.03990348428487778,
+            0.010661610402166843,
+            -0.022711025550961494,
+            0.0401320643723011,
+            0.01905374974012375,
+            -0.008298259228467941,
+            -0.028866350650787354,
+            0.010294250212609768,
+            -0.02290695160627365,
+            -0.009975871071219444,
+            0.009469729848206043,
+            0.00726556871086359,
+            -0.03263791650533676,
+            -0.013543346896767616,
+            0.03510330989956856,
+            0.011437147855758667,
+            -0.006841063499450684,
+            -0.036050282418727875,
+            0.008588065393269062,
+            0.01646590232849121,
+            0.011869817040860653,
+            -0.019233347848057747,
+            0.004449140280485153,
+            -0.012041252106428146,
+            0.017470018938183784,
+            0.008849299512803555,
+            -0.017796561121940613,
+            0.018237393349409103,
+            0.004710374400019646,
+            -0.008490103296935558,
+            -0.003912386484444141,
+            -0.0163107942789793,
+            -0.02006603218615055,
+            0.03840138763189316,
+            0.024490682408213615,
+            0.026335645467042923,
+            0.005457340274006128,
+            0.02008235827088356,
+            0.03227871656417847,
+            0.011282040737569332,
+            -0.03134807199239731,
+            0.027935704216361046,
+            0.016996532678604126,
+            -0.04604247957468033,
+            0.021551799029111862,
+            -0.02218855731189251,
+            -0.028670424595475197,
+            0.009649328887462616,
+            -0.0017367975087836385,
+            0.011192241683602333,
+            0.0034837995190173388,
+            -0.003642988856881857,
+            -0.03307874873280525,
+            -0.006236960180103779,
+            0.015771998092532158,
+            0.015502601861953735,
+            -0.0048695639707148075,
+            -0.003296037670224905,
+            -0.011886144056916237,
+            0.00487364549189806,
+            0.009796272963285446,
+            0.039054471999406815,
+            0.001381682581268251,
+            -0.004751192405819893,
+            -0.0018765984568744898,
+            0.031854212284088135,
+            0.0271193478256464,
+            -0.017274094745516777,
+            -0.026286665350198746,
+            0.00580837344750762,
+            -0.014253576286137104,
+            0.026156047359108925,
+            -0.00487364549189806,
+            0.011877980083227158,
+            0.008718682453036308,
+            -0.0007908449042588472,
+            -0.001373519073240459,
+            0.025584599003195763,
+            -0.02710302174091339,
+            -0.021764053031802177,
+            0.031233781948685646,
+            -0.025731543079018593,
+            0.028735732659697533,
+            0.013061696663498878,
+            -0.01557607389986515,
+            -0.015225040726363659,
+            -0.010302413254976273,
+            0.009943217039108276,
+            -0.012277995236217976,
+            0.023853924125432968,
+            0.003608293831348419,
+            0.006392067763954401,
+            0.0003584313380997628,
+            0.02770712412893772,
+            0.024833550676703453,
+            0.007820690982043743,
+            -0.018074123188853264,
+            -0.0016816934803500772,
+            -0.02320083975791931,
+            -0.01680060662329197,
+            0.023772288113832474,
+            -0.018090449273586273,
+            -0.03268689662218094,
+            -0.028556134551763535,
+            0.012114724144339561,
+            0.007392104249447584,
+            0.010816717520356178,
+            0.03461349755525589,
+            -0.030384771525859833,
+            -0.0344502255320549,
+            0.01766594499349594,
+            0.026270337402820587,
+            -0.005551221314817667,
+            -0.021535472944378853,
+            0.03085825778543949,
+            -0.005828781984746456,
+            0.017600636929273605,
+            0.0404912605881691,
+            -0.0016225076979026198,
+            0.010228941217064857,
+            -0.042026009410619736,
+            -0.014057651162147522,
+            -0.0066614653915166855,
+            0.006261450704187155,
+            0.034417569637298584,
+            0.01048201136291027,
+            0.01535565685480833,
+            0.03448288142681122,
+            -0.021192602813243866,
+            -0.0002903166168835014,
+            -0.0022449791431427,
+            -0.02818061038851738,
+            0.03347059711813927,
+            0.025715215131640434,
+            0.011918798089027405,
+            0.019706835970282555,
+            -0.03794422745704651,
+            -0.013037205673754215,
+            0.027070365846157074,
+            0.015053605660796165,
+            0.01273515447974205,
+            0.016694480553269386,
+            0.011249386705458164,
+            0.00868602842092514,
+            -0.021633435040712357,
+            -0.05169166624546051,
+            0.01214737817645073,
+            0.01993541419506073,
+            -0.016449574381113052,
+            0.01057181041687727,
+            0.024980494752526283,
+            0.007620683871209621,
+            -0.007045152597129345,
+            0.012808626517653465,
+            -0.02682545967400074,
+            0.002259265398606658,
+            0.0019010890973731875,
+            -0.014482156373560429,
+            0.038270771503448486,
+            -0.016367938369512558,
+            0.011363675817847252,
+            -0.018204739317297935,
+            0.023870250210165977,
+            0.00011531029304023832,
+            0.0007964573451317847,
+            0.014963806606829166,
+            -0.011347348801791668,
+            -0.04235255345702171,
+            0.0021613026037812233,
+            0.03342161700129509,
+            -0.015200549736618996,
+            0.026449935510754585,
+            -0.0020674217958003283,
+            0.032458316534757614,
+            0.042777057737112045,
+            0.017731253057718277,
+            0.0003497575526125729,
+            0.025682561099529266,
+            0.0036144163459539413,
+            0.0073880222626030445,
+            0.012400448322296143,
+            0.009322785772383213,
+            0.007245160173624754,
+            0.030809277668595314,
+            -0.010261595249176025,
+            0.02173139713704586,
+            -0.04760988429188728,
+            -0.014629100449383259,
+            0.012024925090372562,
+            -0.0013316808035597205,
+            -0.012449429370462894,
+            -0.033029764890670776,
+            0.012563719414174557,
+            -0.03503800183534622,
+            0.022074267268180847,
+            -0.0004959363141097128,
+            0.04333217814564705,
+            -0.02187834307551384,
+            -0.005518566817045212,
+            0.016694480553269386,
+            0.0016510800924152136,
+            0.023282473906874657,
+            -0.0012153501156717539,
+            -0.016416920349001884,
+            -0.0067798369564116,
+            -0.009257477708160877,
+            0.004138925112783909,
+            0.04375668615102768,
+            -0.003722583642229438,
+            -0.0001682458823779598,
+            0.01409030519425869,
+            -0.011959616094827652,
+            -0.03250729665160179,
+            0.045193471014499664,
+            -0.0030837850645184517,
+            0.02484987862408161,
+            0.04979771748185158,
+            -0.06410028040409088,
+            -0.019543563947081566,
+            0.013061696663498878,
+            -0.010441194288432598,
+            0.011363675817847252,
+            0.005547139327973127,
+            0.016114868223667145,
+            0.017894525080919266,
+            -0.005485912784934044,
+            0.01619650423526764,
+            0.005106307100504637,
+            0.028098974376916885,
+            -0.02336410991847515,
+            -0.03296445682644844,
+            -0.02636830136179924,
+            -0.017747581005096436,
+            0.0014480114914476871,
+            -0.015796490013599396,
+            -3.563585551091819e-06,
+            -0.0022204883862286806,
+            -0.04388730227947235,
+            -0.025274382904171944,
+            0.040687184780836105,
+            0.006706364918500185,
+            0.02458864450454712,
+            0.0077268099412322044,
+            0.03418898954987526,
+            0.005885927006602287,
+            -0.009037061594426632,
+            0.030058229342103004,
+            -0.020049704238772392,
+            0.03833607956767082,
+            0.022106921300292015,
+            -0.002832755446434021,
+            0.005865518003702164,
+            0.031086837872862816,
+            -0.033797141164541245,
+            0.008086007088422775,
+            -0.034384917467832565,
+            0.0046695563942193985,
+            -0.008620720356702805,
+            -0.015094423666596413,
+            -0.009526874870061874,
+            -0.02504580281674862,
+            0.011216731742024422,
+            -0.021535472944378853,
+            0.01590261608362198,
+            0.010416703298687935,
+            -0.002916431985795498,
+            0.03448288142681122,
+            -0.014204595237970352,
+            -0.0027796924114227295,
+            0.0024531499948352575,
+            -0.027070365846157074,
+            -0.006343086715787649,
+            -0.003377673216164112,
+            -0.03689929470419884,
+            -0.004502203781157732,
+            0.010988152585923672,
+            -0.00024184548237826675,
+            0.016898570582270622,
+            0.02787039615213871,
+            0.0005092020728625357,
+            -0.022302847355604172,
+            0.002340900944545865,
+            -0.04199335351586342,
+            0.0011877981014549732,
+            0.01483318954706192,
+            -0.017600636929273605,
+            -0.008612556383013725,
+            0.033046092838048935,
+            0.023298801854252815,
+            -0.009396257810294628,
+            -0.035005345940589905,
+            0.017796561121940613,
+            -0.011853490024805069,
+            0.027511198073625565,
+            -0.0002630197268445045,
+            0.019298657774925232,
+            -0.0076941559091210365,
+            0.04493223875761032,
+            0.001369437319226563,
+            0.008645210415124893,
+            -0.021959977224469185,
+            0.007020662073045969,
+            -0.020278284326195717,
+            -0.002406209474429488,
+            -0.006171651650220156,
+            -0.012751481495797634,
+            -0.010669773444533348,
+            0.0032735876739025116,
+            0.007730891928076744,
+            -0.0176822729408741,
+            0.0055961208418011665,
+            -0.015959760174155235,
+            -0.02741323597729206,
+            0.007543129846453667,
+            -0.05965929850935936,
+            -0.0179598331451416,
+            -0.03373183310031891,
+            -0.009477893821895123,
+            0.02008235827088356,
+            0.03510330989956856,
+            -0.012759645469486713,
+            0.00497977202758193,
+            -0.014898497611284256,
+            0.004477712791413069,
+            0.03490738570690155,
+            0.044442422688007355,
+            -0.014759717509150505,
+            0.003381754970178008,
+            -0.010955498553812504,
+            0.033029764890670776,
+            -0.03347059711813927,
+            0.0030062312725931406,
+            0.031086837872862816,
+            -0.005057325586676598,
+            0.007959471084177494,
+            0.011624909937381744,
+            0.020474210381507874,
+            -0.019396619871258736,
+            0.00877582747489214,
+            0.002100075827911496,
+            0.019233347848057747,
+            0.02171507105231285,
+            0.020343592390418053,
+            -0.03539719805121422,
+            0.028409190475940704,
+            0.018384337425231934,
+            0.02444170042872429,
+            0.044442422688007355,
+            -0.03974021226167679,
+            -0.004147088620811701,
+            -0.0032225653994828463,
+            -0.02997659333050251,
+            0.02008235827088356,
+            -0.007530884351581335,
+            -0.01841699331998825,
+            -0.007069643586874008,
+            0.03327467292547226,
+            0.014571955427527428,
+            -0.05051611363887787,
+            0.01980479806661606,
+            0.01740471087396145,
+            0.02055584453046322,
+            0.024474354460835457,
+            -0.01123305968940258,
+            -0.0019072118448093534,
+            0.008996243588626385,
+            0.006804327480494976,
+            0.019266001880168915,
+            0.015257694758474827,
+            0.002012317767366767,
+            -0.0022082431241869926,
+            -0.011837163008749485,
+            0.005224678665399551,
+            -0.009077879600226879,
+            0.010808554477989674,
+            -0.02458864450454712,
+            -0.009698309935629368,
+            0.021388528868556023,
+            0.0031164393294602633,
+            -0.013665799982845783,
+            -0.03729114308953285,
+            -0.045944519340991974,
+            0.0033837957307696342,
+            0.004543021321296692,
+            -0.05365091934800148,
+            0.005322641227394342,
+            -0.026547899469733238,
+            -0.002108239568769932,
+            -0.005502239800989628,
+            0.0019214981002733111,
+            0.005636938381940126,
+            0.0167679525911808,
+            -0.013494365848600864,
+            0.02248244546353817,
+            -0.015502601861953735,
+            0.03165828809142113,
+            0.0055961208418011665,
+            0.007375776767730713,
+            -0.0027796924114227295,
+            0.05590406060218811,
+            -0.003712379140779376,
+            -0.03699725493788719,
+            0.023282473906874657,
+            -0.01311067771166563,
+            -0.0001747512142173946,
+            0.030090883374214172,
+            0.00884113647043705,
+            -0.015257694758474827,
+            0.013437220826745033,
+            -0.030613351613283157,
+            -0.03249097242951393,
+            0.010996315628290176,
+            -0.013755599036812782,
+            -0.0015776080545037985,
+            -0.04568328335881233,
+            -0.009339112788438797,
+            0.004269542172551155,
+            -0.018335357308387756,
+            0.010775899514555931,
+            0.01631895825266838,
+            0.013992343097925186,
+            -0.024245774373412132,
+            0.009053388610482216,
+            0.006865554489195347,
+            0.013453547842800617,
+            0.0007331897504627705,
+            0.014049487188458443,
+            -0.008677864447236061,
+            -0.011306531727313995,
+            -0.018319029361009598,
+            0.0032450153958052397,
+            -0.026792805641889572,
+            0.005155288614332676,
+            -0.003155216109007597,
+            -0.03771565109491348,
+            -0.06521052122116089,
+            -0.0344502255320549,
+            0.016294466331601143,
+            0.03396041318774223,
+            -0.009045224636793137,
+            0.0035144127905368805,
+            -0.00023648813657928258,
+            -0.01093917153775692,
+            0.024523336440324783,
+            -0.005930826533585787,
+            -0.0004829256213270128,
+            0.0035389035474509,
+            0.02050686441361904,
+            -0.009722800925374031,
+            0.005845109466463327,
+            -0.051952898502349854,
+            -0.01586179807782173,
+            0.01812310516834259,
+            0.006306350696831942,
+            0.014629100449383259,
+            -0.009690145961940289,
+            0.01693122461438179,
+            -0.030074557289481163,
+            -0.028147956356406212,
+            0.04130761697888374,
+            0.008277850225567818,
+            0.011486129835247993,
+            -0.003055212553590536,
+            -0.01680060662329197,
+            0.02504580281674862,
+            -0.0020633398089557886,
+            -0.006461458280682564,
+            -0.00817988719791174,
+            0.015820980072021484,
+            0.044279150664806366,
+            0.003255219664424658,
+            0.0009821783751249313,
+            0.017045514658093452,
+            -0.002663361607119441,
+            -0.011575928889214993,
+            0.007677828427404165,
+            0.013592327944934368,
+            -0.016384266316890717,
+            0.010375885292887688,
+            -0.025862159207463264,
+            0.0332583449780941,
+            0.03131541982293129,
+            0.011575928889214993,
+            -0.015045441687107086,
+            0.006130834110081196,
+            0.005330805201083422,
+            0.008086007088422775,
+            -0.001369437319226563,
+            -0.0017133272485807538,
+            0.020719116553664207,
+            -0.03276853263378143,
+            -0.01693122461438179,
+            -0.005347132217139006,
+            0.003932795487344265,
+            0.005277741700410843,
+            0.03396041318774223,
+            -0.0033266509417444468,
+            0.017927179113030434,
+            -0.011657563969492912,
+            0.010579974390566349,
+            0.021764053031802177,
+            0.037095218896865845,
+            -0.024996822699904442,
+            -0.029862305149435997,
+            0.012914752587676048,
+            -0.001296985661610961,
+            0.014065815135836601,
+            0.0010173837654292583,
+            0.014139287173748016,
+            0.0029409227427095175,
+            0.060475654900074005,
+            -0.017763907089829445,
+            0.004628738854080439,
+            0.0004035860183648765,
+            -0.01858026348054409,
+            -0.0013531101867556572,
+            0.004261378664523363,
+            -0.010979988612234592,
+            0.03608293831348419,
+            0.002344982698559761,
+            0.039185091853141785,
+            0.013918871060013771,
+            0.009290131740272045,
+            0.0015143405180424452,
+            -0.013592327944934368,
+            0.008355404250323772,
+            0.0015592400450259447,
+            -0.001132694073021412,
+            0.009461566805839539,
+            0.019298657774925232,
+            0.009714636951684952,
+            0.025274382904171944,
+            0.009788108989596367,
+            0.012857607565820217,
+            -0.029470453038811684,
+            0.03510330989956856,
+            -0.0006224715034477413,
+            0.012996388599276543,
+            -0.02040890045464039,
+            -0.003136848099529743,
+            -0.0008240094175562263,
+            0.015649545937776566,
+            -0.015616890974342823,
+            -0.001201063860207796,
+            -0.01297189760953188,
+            0.040099408477544785,
+            -0.018988441675901413,
+            0.013976015150547028,
+            0.009673818945884705,
+            -0.02865409664809704,
+            -0.007596192881464958,
+            0.01995174214243889,
+            0.014294394291937351,
+            -0.0053144777193665504,
+            -0.020490536466240883,
+            0.005832863971590996,
+            0.015380147844552994,
+            -0.028572462499141693,
+            1.3584674888988957e-05,
+            0.014751553535461426,
+            -0.0164903923869133,
+            -0.00025409081717953086,
+            -0.004702210891991854,
+            0.003975654020905495,
+            -0.009265640750527382,
+            0.007028825581073761,
+            0.016057724133133888,
+            -0.024507008492946625,
+            -0.020049704238772392,
+            0.005722655914723873,
+            0.006571666337549686,
+            0.01444950234144926,
+            0.007739055436104536,
+            0.004355259705334902,
+            -0.020033378154039383,
+            -0.01799248717725277,
+            0.0032674651592969894,
+            -0.01649855636060238,
+            -0.0048614004626870155,
+            0.0020531355403363705,
+            -0.019363965839147568,
+            0.01875986158847809,
+            -0.010743245482444763,
+            0.00017538899555802345,
+            -0.003226647386327386,
+            -0.020800752565264702,
+            0.006204306147992611,
+            -0.004865481983870268,
+            0.019870106130838394,
+            0.003555230563506484,
+            0.02785406820476055,
+            0.02310287579894066,
+            -0.004428731743246317,
+            0.017600636929273605,
+            -0.016155686229467392,
+            0.007983962073922157,
+            -0.031135819852352142,
+            -0.001782717532478273,
+            -0.010408539324998856,
+            0.0036103345919400454,
+            -0.015943434089422226,
+            -0.009126860648393631,
+            -0.0048777274787425995,
+            -0.0006678812787868083,
+            -0.004820582456886768,
+            0.0004275664687156677,
+            0.0027817331720143557,
+            0.02231917530298233,
+            -0.013208640739321709,
+            -0.02803366631269455,
+            -0.017437364906072617,
+            0.006045116577297449,
+            0.019396619871258736,
+            -0.026025431230664253,
+            -0.009184005670249462,
+            -0.005012426059693098,
+            -0.004326687194406986,
+            -0.021404854953289032,
+            -0.015167895704507828,
+            -0.005632856860756874,
+            -0.008045189082622528,
+            -0.010522829368710518,
+            0.0046532293781638145,
+            -0.0035715578123927116,
+            -0.011739199981093407,
+            0.021666089072823524,
+            0.018057795241475105,
+            -0.019837452098727226,
+            -0.019837452098727226,
+            0.013641309924423695,
+            -0.006159406621009111,
+            0.0018878233386203647,
+            0.02008235827088356,
+            0.00029286774224601686,
+            0.005824700463563204,
+            0.020882388576865196,
+            -0.0035225762985646725,
+            -0.01797616109251976,
+            0.003577680326998234,
+            -0.0018888438353314996,
+            -0.012122887186706066,
+            0.003212361130863428,
+            0.02773977816104889,
+            0.0008076822850853205,
+            0.023429417982697487,
+            -0.03056436963379383,
+            -0.011412657797336578,
+            0.006061443593353033,
+            0.02369065210223198,
+            0.00211844383738935,
+            -0.0021694661118090153,
+            0.020131340250372887,
+            0.03941367194056511,
+            0.005775718949735165,
+            0.0015755671774968505,
+            -0.04003410041332245,
+            0.01027792226523161,
+            0.009208496659994125,
+            0.024621298536658287,
+            0.008339077234268188,
+            0.0014653591206297278,
+            -0.012906589545309544,
+            -0.009755454957485199,
+            -0.0017939424142241478,
+            0.014702572487294674,
+            -0.023870250210165977,
+            0.0024755997583270073,
+            -0.03157665207982063,
+            0.005094061605632305,
+            0.0021959978621453047,
+            -0.009559528902173042,
+            0.0335359051823616,
+            0.00980443600565195,
+            -0.02065380848944187,
+            0.015151568688452244,
+            0.005767555441707373,
+            0.007330877240747213,
+            0.022449791431427002,
+            -0.008228869177401066,
+            0.030450081452727318,
+            -0.005049162078648806,
+            0.01213921420276165,
+            0.02742956392467022,
+            0.032588932663202286,
+            -0.013102514669299126,
+            -0.003959327004849911,
+            0.0006097159348428249,
+            -0.0002561317232903093,
+            0.0007995187188498676,
+            -0.008204378187656403,
+            0.0037389106582850218,
+            -0.014800535514950752,
+            0.012710663489997387,
+            -0.019543563947081566,
+            -0.0034286954905837774,
+            0.0038532004691660404,
+            0.01991908811032772,
+            0.0016265894519165158,
+            -0.033633869141340256,
+            0.02055584453046322,
+            -0.015420965850353241,
+            0.0135596739128232,
+            0.001290863030590117,
+            -0.011110605672001839,
+            -0.00029210239881649613,
+            0.020588500425219536,
+            0.010947334580123425,
+            -0.007845181971788406,
+            -0.0135596739128232,
+            -0.028376536443829536,
+            -0.021535472944378853,
+            0.00823703221976757,
+            0.009445239789783955,
+            -0.0028837777208536863,
+            0.0021306893322616816,
+            -0.011869817040860653,
+            0.0006520643946714699,
+            -0.0008122742874547839,
+            0.01740471087396145,
+            -0.0335359051823616,
+            0.021943651139736176,
+            -0.019527235999703407,
+            -0.019543563947081566,
+            0.017045514658093452,
+            0.008049271069467068,
+            0.009445239789783955,
+            0.0033552234526723623,
+            -0.01995174214243889,
+            -0.012326976284384727,
+            0.0338951013982296,
+            -0.01138000376522541,
+            0.006298186723142862,
+            0.004408322740346193,
+            0.012914752587676048,
+            -0.008914608508348465,
+            -0.011926962062716484,
+            0.01784554310142994,
+            -0.009861581027507782,
+            0.013951525092124939,
+            0.0010010566329583526,
+            -0.0254703089594841,
+            0.02739690989255905,
+            -0.019233347848057747,
+            0.03487472981214523,
+            -0.021617108955979347,
+            0.00711862463504076,
+            0.0066614653915166855,
+            0.008792154490947723,
+            -0.006690037902444601,
+            -0.013224967755377293,
+            0.0012786176521331072,
+            0.03807484731078148,
+            -0.02623768337070942,
+            0.024180466309189796,
+            0.022400809451937675,
+            -0.00040766780148260295,
+            0.02065380848944187,
+            -0.008367649279534817,
+            -0.026711169630289078,
+            0.0179598331451416,
+            -0.012596373446285725,
+            -0.026123393326997757,
+            0.013257622718811035,
+            -0.022678371518850327,
+            -0.020278284326195717,
+            -0.026302991434931755,
+            0.004947117529809475,
+            -0.0025694805663079023,
+            -0.005151206627488136,
+            0.009004407562315464,
+            0.02561725303530693,
+            0.020915042608976364,
+            0.019576217979192734,
+            0.02111096680164337,
+            0.01161674689501524,
+            0.022449791431427002,
+            -0.007559456862509251,
+            0.04401791840791702,
+            -0.004947117529809475,
+            0.004632820375263691,
+            -0.0015980169409886003,
+            -0.015657709911465645,
+            0.009771781973540783,
+            0.0119106350466609,
+            0.0029572497587651014,
+            -0.029176564887166023,
+            -0.009061552584171295,
+            -0.03660540655255318,
+            0.005053244065493345,
+            0.0020664012990891933,
+            0.0176822729408741,
+            0.032115448266267776,
+            -0.005865518003702164,
+            0.004592002835124731,
+            -0.005102225113660097,
+            -0.013927034102380276,
+            -0.009420748800039291,
+            -0.02739690989255905,
+            -0.025437654927372932,
+            -0.007653337903320789,
+            -0.013739272020757198,
+            -0.030368445441126823,
+            -0.011722872965037823,
+            0.01551892887800932,
+            -0.007645174395292997,
+            0.011918798089027405,
+            -0.016702644526958466,
+            0.01385356206446886,
+            -0.004110352601855993,
+            -0.004763437435030937,
+            -0.004530776292085648,
+            -0.017600636929273605,
+            -0.006420640274882317,
+            0.014367866329848766,
+            -0.009477893821895123,
+            -0.01237595733255148,
+            -0.02215590327978134,
+            -0.0015459742862731218,
+            -0.01574750803411007,
+            0.003489922033622861,
+            -0.014596446417272091,
+            -0.006771673448383808,
+            -0.003908304497599602,
+            -0.018514955416321754,
+            0.01766594499349594,
+            0.014816862531006336,
+            -0.019380291923880577,
+            -0.0090697156265378,
+            0.02638462744653225,
+            0.042777057737112045,
+            -0.03218075633049011,
+            -0.008200296200811863,
+            0.0020378287881612778,
+            0.018090449273586273,
+            0.010171796195209026,
+            0.016702644526958466,
+            -0.024653952568769455,
+            0.0022286521270871162,
+            -0.021649762988090515,
+            -0.001220452249981463,
+            -0.023739634081721306,
+            0.010775899514555931,
+            0.025715215131640434,
+            -0.010971825569868088,
+            0.025976449251174927,
+            0.01634344831109047,
+            -0.01812310516834259,
+            0.012016761116683483,
+            0.007183933164924383,
+            0.014645427465438843,
+            -0.019755816087126732,
+            0.0167679525911808,
+            -0.015796490013599396,
+            0.031233781948685646,
+            0.00892277155071497,
+            -0.04829562455415726,
+            -0.03503800183534622,
+            -0.013388238847255707,
+            0.0013388239312916994,
+            -0.010702427476644516,
+            0.015037278644740582,
+            0.015967924147844315,
+            0.013959688134491444,
+            -0.019380291923880577,
+            -0.015820980072021484,
+            0.011984107084572315,
+            0.010065670125186443,
+            0.027462217956781387,
+            -0.006649219896644354,
+            -0.004212397150695324,
+            -0.00960034690797329,
+            -0.02249877341091633,
+            -0.016098542138934135,
+            0.0209803506731987,
+            -0.018041469156742096,
+            0.015935270115733147,
+            -0.026139721274375916,
+            -0.012449429370462894,
+            0.017453692853450775,
+            -0.05211617052555084,
+            -0.01453113742172718,
+            0.0007612520130351186,
+            0.03012353740632534,
+            0.004751192405819893,
+            0.0035184945445507765,
+            0.01841699331998825,
+            -0.01993541419506073,
+            -0.0039776950143277645,
+            -0.02099667675793171,
+            0.006832899991422892,
+            -0.011853490024805069,
+            -0.013984179124236107,
+            0.014482156373560429,
+            -0.013845399022102356,
+            -0.01601690612733364,
+            0.0034368589986115694,
+            0.007183933164924383,
+            -0.021094640716910362,
+            0.011951452121138573,
+            0.003945040516555309,
+            0.012171869166195393,
+            0.012171869166195393,
+            0.01199227012693882,
+            0.021551799029111862,
+            0.02295593172311783,
+            -0.007334959227591753,
+            -0.004767519421875477,
+            -0.01004117913544178,
+            -0.0015398516552522779,
+            0.008898280560970306,
+            0.0033797139767557383,
+            -0.002989904023706913,
+            0.007837017998099327,
+            0.010033016093075275,
+            -0.01498013362288475,
+            0.033193036913871765,
+            -0.0035878848284482956,
+            -0.04134026914834976,
+            -0.003238892648369074,
+            -0.03800953924655914,
+            0.020033378154039383,
+            -0.023298801854252815,
+            -0.016996532678604126,
+            0.0032776694279164076,
+            -0.00629410520195961,
+            -0.019902760162949562,
+            0.013608654960989952,
+            0.011404493823647499,
+            0.00258580781519413,
+            0.02021297626197338,
+            0.012269831262528896,
+            -0.005134879611432552,
+            0.026172375306487083,
+            -0.014718899503350258,
+            -0.008751336485147476,
+            0.015567909926176071,
+            0.006853308994323015,
+            0.0022347746416926384,
+            0.0011031010653823614,
+            -0.018825169652700424,
+            0.005098143592476845,
+            0.0164903923869133,
+            0.007216587662696838,
+            -0.000566347036510706,
+            0.014049487188458443,
+            0.012514738366007805,
+            -0.015135240741074085,
+            -0.0002213090192526579,
+            -0.026564225554466248,
+            0.0032817514147609472,
+            0.004567512311041355,
+            0.007506393827497959,
+            -0.006771673448383808,
+            0.006281859707087278,
+            0.00658799335360527,
+            0.014759717509150505,
+            -0.010155469179153442,
+            -0.016980206593871117,
+            -0.002510294783860445,
+            0.004285869188606739,
+            0.016694480553269386,
+            0.03157665207982063,
+            -0.010310577228665352,
+            -0.005734901409596205,
+            -0.0386626236140728,
+            0.02070278860628605,
+            0.03332365304231644,
+            0.011175914667546749,
+            -0.028213264420628548,
+            0.012302485294640064,
+            0.0006515541463159025,
+            0.0019276207312941551,
+            -0.026302991434931755,
+            0.02787039615213871,
+            0.00974729098379612,
+            0.0066614653915166855,
+            0.017861871048808098,
+            0.0022306928876787424,
+            0.0029266364872455597,
+            -0.016302630305290222,
+            0.01018812321126461,
+            0.009086042642593384,
+            -0.017306748777627945,
+            0.0009745250572450459,
+            0.02324981987476349,
+            -0.01423724927008152,
+            -0.003208279376849532,
+            0.0018133309204131365,
+            0.02651524543762207,
+            -0.013820908032357693,
+            -0.014294394291937351,
+            0.0005714492290280759,
+            0.01751900091767311,
+            -0.0007979880319908261,
+            0.012098397128283978,
+            0.04898136481642723,
+            -0.012424939312040806,
+            0.019657853990793228,
+            0.005461421795189381,
+            -0.015290348790585995,
+            -0.032605260610580444,
+            -0.004922627005726099,
+            -0.01830270327627659,
+            0.011200404725968838,
+            0.014343376271426678,
+            -0.015763835981488228,
+            0.0007525782566517591,
+            -0.03144603595137596,
+            0.007743136957287788,
+            0.014735226519405842,
+            0.018776189535856247,
+            -0.002440904499962926,
+            0.006134915631264448,
+            0.027674470096826553,
+            0.02458864450454712,
+            -0.007392104249447584,
+            0.013804581016302109,
+            0.02320083975791931,
+            -0.025062130764126778,
+            -0.003069498809054494,
+            0.006820654962211847,
+            0.004104230087250471,
+            0.0053063142113387585,
+            -0.016963878646492958,
+            0.03915243595838547,
+            -0.005922663025557995,
+            -0.01506176870316267,
+            0.0030266400426626205,
+            -0.009028897620737553,
+            0.0063390047289431095,
+            -0.012563719414174557,
+            0.012637191452085972,
+            -0.008114579133689404,
+            0.00022883480414748192,
+            -0.0059430720284581184,
+            -0.011110605672001839,
+            -0.012653518468141556,
+            -0.0007949267164804041,
+            -0.016531210392713547,
+            0.007432921789586544,
+            0.00981259997934103,
+            0.005045080557465553,
+            0.01025343220680952,
+            -0.0036797248758375645,
+            -0.003136848099529743,
+            -0.016751626506447792,
+            -0.009984034113585949,
+            0.0073716952465474606,
+            0.01872720755636692,
+            -0.028588788583874702,
+            -0.018547609448432922,
+            0.009290131740272045,
+            0.0022898786701261997,
+            0.018776189535856247,
+            0.0158944521099329,
+            -0.0002595757250674069,
+            0.012318813242018223,
+            -0.01964152604341507,
+            -0.009184005670249462,
+            0.009690145961940289,
+            -0.003569516818970442,
+            0.01048201136291027,
+            -0.014759717509150505,
+            0.010033016093075275,
+            -0.011159587651491165,
+            -0.011053460650146008,
+            0.01385356206446886,
+            0.01738838478922844,
+            0.008873790502548218,
+            -0.001294944784604013,
+            3.383413059054874e-05,
+            -0.010506502352654934,
+            -0.024033522233366966,
+            0.004771600943058729,
+            0.027021385729312897,
+            -0.013273949734866619,
+            0.003883813973516226,
+            0.008702355436980724,
+            -0.017502672970294952,
+            -0.019821124151349068,
+            -0.012653518468141556,
+            -0.022106921300292015,
+            0.020572172477841377,
+            0.023413091897964478,
+            -0.02026195637881756,
+            0.00877582747489214,
+            -0.004620575346052647,
+            0.014049487188458443,
+            -0.01400050614029169,
+            0.010612628422677517,
+            0.0317889042198658,
+            -0.020000724121928215,
+            -0.0028654097113758326,
+            0.0036633978597819805,
+            -0.015347493812441826,
+            0.018629245460033417,
+            0.03268689662218094,
+            -0.00176945177372545,
+            0.014041324146091938,
+            0.0008138049161061645,
+            -0.009118696674704552,
+            0.01905374974012375,
+            0.011094278655946255,
+            0.005347132217139006,
+            -0.0042491331696510315,
+            -0.012212686240673065,
+            -0.03010721132159233,
+            0.016702644526958466,
+            0.0129637336358428,
+            -0.0037674831692129374,
+            -0.0011551438365131617,
+            -0.006343086715787649,
+            -0.015404638834297657,
+            0.0035123717971146107,
+            -0.013078023679554462,
+            -0.00803294312208891,
+            -0.008759500458836555,
+            0.004314441699534655,
+            -0.0034960447810590267,
+            -0.009053388610482216,
+            0.007236996199935675,
+            0.0009510547970421612,
+            -0.0010602424154058099,
+            -0.01902109570801258,
+            -0.025078458711504936,
+            -0.00995954405516386,
+            -0.016849588602781296,
+            0.0038287099450826645,
+            0.0005234883283264935,
+            -0.027184655889868736,
+            0.0053961132653057575,
+            -0.008139070123434067,
+            -0.008134988136589527,
+            0.007126788143068552,
+            0.006881881505250931,
+            -0.007437003776431084,
+            -0.010816717520356178,
+            -0.00786150898784399,
+            -0.006865554489195347,
+            0.020033378154039383,
+            0.0034348180051892996,
+            0.012082069180905819,
+            -0.010498338378965855,
+            -0.018025141209363937,
+            -0.016963878646492958,
+            0.016718972474336624,
+            -0.008971752598881721,
+            -0.02083340659737587,
+            -0.01444950234144926,
+            -0.008424794301390648,
+            -0.007665583398193121,
+            -0.01527402177453041,
+            -0.021159948781132698,
+            -0.0032103201374411583,
+            -0.011347348801791668,
+            -0.0067431009374558926,
+            -0.0028960229828953743,
+            0.0013571919407695532,
+            0.00951871182769537,
+            0.00546958576887846,
+            0.00438791373744607,
+            0.021666089072823524,
+            0.010122815147042274,
+            -0.005461421795189381,
+            0.016441410407423973,
+            -0.006183897145092487,
+            -0.004440976772457361,
+            -0.01498013362288475,
+            -0.009918726049363613,
+            0.018547609448432922,
+            -0.0055226488038897514,
+            0.008457448333501816,
+            -0.0009536059224046767,
+            -0.00576347392052412,
+            0.015674035996198654,
+            0.0129637336358428,
+            0.016718972474336624,
+            0.004771600943058729,
+            -0.00012168807734269649,
+            0.012416775338351727,
+            -0.0015745466807857156,
+            0.012555556371808052,
+            -0.0038429962005466223,
+            0.014506647363305092,
+            -0.009722800925374031,
+            -0.018988441675901413,
+            0.004157293122261763,
+            0.006265532691031694,
+            -0.012531065382063389,
+            0.009404421783983707,
+            0.007743136957287788,
+            -0.013747435994446278,
+            -0.02261306345462799,
+            -0.000106317616882734,
+            0.00014362763613462448,
+            0.008122743107378483,
+            0.008971752598881721,
+            -0.008073761127889156,
+            0.012890261597931385,
+            0.004616493359208107,
+            -0.007179851643741131,
+            -0.019657853990793228,
+            0.026009103283286095,
+            0.015608727931976318,
+            0.012269831262528896,
+            -0.052508022636175156,
+            -0.0030980713199824095,
+            0.010743245482444763,
+            -0.015951598063111305,
+            -0.026792805641889572,
+            -0.006404313258826733,
+            -0.008759500458836555,
+            -0.04290767386555672,
+            -0.009624837897717953,
+            -0.0005724696675315499,
+            -0.011388166807591915,
+            0.010400376282632351,
+            0.01573934406042099,
+            0.014612773433327675,
+            0.005489994306117296,
+            -0.007232914678752422,
+            -0.017633290961384773,
+            -0.02788672223687172,
+            0.010057506151497364,
+            -0.013380075804889202,
+            -0.006412476766854525,
+            -0.002263347152620554,
+            -0.007690073922276497,
+            0.0016112826997414231,
+            0.01691489666700363,
+            0.042483169585466385,
+            0.023037567734718323,
+            -0.012335140258073807,
+            -0.020131340250372887,
+            -0.012800462543964386,
+            -0.009192168712615967,
+            0.00633492274209857,
+            0.011592255905270576,
+            -0.004738946910947561,
+            0.0096656559035182,
+            -0.004294032696634531,
+            0.012302485294640064,
+            0.012767808511853218,
+            0.01680060662329197,
+            -2.3023791072773747e-05,
+            0.005579793825745583,
+            -0.0024694770108908415,
+            -0.01738838478922844,
+            -0.004379750229418278,
+            0.008261523209512234,
+            -0.0008872769540175796,
+            0.0019317024853080511,
+            0.014204595237970352,
+            0.019102731719613075,
+            -0.009094206616282463,
+            -0.014612773433327675,
+            -0.016098542138934135,
+            -0.005277741700410843,
+            -0.0012622905196622014,
+            0.029927613213658333,
+            0.02037624642252922,
+            -0.019266001880168915,
+            -0.018237393349409103,
+            -0.018351683393120766,
+            0.00922482367604971,
+            -0.003563394071534276,
+            -0.0064369672909379005,
+            0.01905374974012375,
+            0.023021239787340164,
+            0.017910851165652275,
+            -0.003642988856881857,
+            -0.014865843579173088,
+            0.003804219188168645,
+            0.02847449854016304,
+            0.008367649279534817,
+            0.01078406348824501,
+            0.008726846426725388,
+            -0.0011908593587577343,
+            -0.032441988587379456,
+            -0.0113146947696805,
+            0.00951871182769537,
+            -0.008481939323246479,
+            0.026564225554466248,
+            -0.016531210392713547,
+            0.009388094767928123,
+            -0.005077734589576721,
+            0.0019286411115899682,
+            -0.02365799807012081,
+            0.0014929111348465085,
+            0.010122815147042274,
+            -0.024947840720415115,
+            0.00336950970813632,
+            -0.023592690005898476,
+            -0.00013763252354692668,
+            0.008367649279534817,
+            0.0022837561555206776,
+            0.01650671847164631,
+            -0.0113146947696805,
+            0.0051838611252605915,
+            0.02099667675793171,
+            -0.002832755446434021,
+            -0.01512707769870758,
+            -0.0007842120248824358,
+            0.0005092020728625357,
+            -0.001286781276576221,
+            0.02563357912003994,
+            0.015812816098332405,
+            -0.0036756431218236685,
+            0.020882388576865196,
+            0.008677864447236061,
+            0.0019133344758301973,
+            -0.021764053031802177,
+            0.0023796779569238424,
+            -0.032899148762226105,
+            0.017437364906072617,
+            -0.011126932688057423,
+            0.012490247376263142,
+            0.00868602842092514,
+            -0.01341272983700037,
+            -0.0007388022495433688,
+            -0.0107187544927001,
+            -0.0015725058037787676,
+            -0.007934981025755405,
+            0.009779945947229862,
+            -0.004363423213362694,
+            0.008339077234268188,
+            -0.00024579971795901656,
+            -0.01476788055151701,
+            0.018694553524255753,
+            9.678283277025912e-06,
+            -0.00010925139940809458,
+            0.005441013257950544,
+            0.009935053065419197,
+            0.00786559097468853,
+            0.002009256277233362,
+            0.023870250210165977,
+            -0.031119491904973984,
+            -0.016082214191555977,
+            -0.010204450227320194,
+            0.005543057806789875,
+            -0.007922735065221786,
+            -0.018531281501054764,
+            -0.03257260471582413,
+            0.01318415068089962,
+            0.001293924287892878,
+            -0.0009806477464735508,
+            0.01258004643023014,
+            -0.00942891277372837,
+            -0.0013878053287044168,
+            -0.01606588624417782,
+            -0.014808698557317257,
+            0.006416558753699064,
+            0.0033633869607001543,
+            -0.01222901325672865,
+            -0.023739634081721306,
+            0.014759717509150505,
+            0.0007418635650537908,
+            -0.016980206593871117,
+            -0.017502672970294952,
+            0.00018457300029695034,
+            -0.004804255440831184,
+            0.02204161323606968,
+            -0.016678154468536377,
+            0.003373591462150216,
+            0.0007219648687168956,
+            0.009232986718416214,
+            -0.02889900468289852,
+            0.00716352416202426,
+            0.0006107363733462989,
+            0.0035164537839591503,
+            -0.006518602836877108,
+            -0.021013004705309868,
+            0.011371839791536331,
+            -0.0020674217958003283,
+            -0.014751553535461426,
+            -0.020751770585775375,
+            -0.0020796670578420162,
+            -0.007420676294714212,
+            -0.02724996581673622,
+            0.017910851165652275,
+            0.0009985055075958371,
+            -0.01738838478922844,
+            0.008077843114733696,
+            0.006596156861633062,
+            0.009355440735816956,
+            0.02455599047243595,
+            0.010147306136786938,
+            -0.018776189535856247,
+            -0.0008087027235887945,
+            0.006979844067245722,
+            0.009306458756327629,
+            0.0036450298503041267,
+            0.04222193360328674,
+            -0.008130906149744987,
+            -0.006579829845577478,
+            0.0066614653915166855,
+            -0.008449285291135311,
+            -0.013747435994446278,
+            0.026727497577667236,
+            0.005796127952635288,
+            0.015984252095222473,
+            0.016123032197356224,
+            -0.00945340283215046,
+            -0.005008344538509846,
+            -0.0018786393338814378,
+            0.02065380848944187,
+            0.0073798587545752525,
+            -0.005502239800989628,
+            -0.009714636951684952,
+            0.005881845485419035,
+            -0.00898808054625988,
+            0.0021653843577951193,
+            -0.009526874870061874,
+            -0.005824700463563204,
+            -0.008069679141044617,
+            0.016400592401623726,
+            -0.0003523086488712579,
+            0.027315273880958557,
+            0.020751770585775375,
+            -0.008563575334846973,
+            -0.0005683879135176539,
+            0.017208784818649292,
+            -0.005820618476718664,
+            0.015608727931976318,
+            -0.011192241683602333,
+            -0.007706400938332081,
+            0.006408394780009985,
+            0.024017194285988808,
+            0.012237177230417728,
+            -0.0056042843498289585,
+            0.0002885308349505067,
+            -0.0030348035506904125,
+            -0.010612628422677517,
+            0.013388238847255707,
+            0.015323002822697163,
+            -0.01947825588285923,
+            -0.004722619894891977,
+            -0.0016592437168583274,
+            -0.003300119424238801,
+            0.00945340283215046,
+            0.02305389568209648,
+            0.000171817431692034,
+            0.009184005670249462,
+            0.0012765767751261592,
+            0.03862996771931648,
+            -0.0053144777193665504,
+            0.03102152980864048,
+            -0.0010423846542835236,
+            -0.0059430720284581184,
+            -0.009094206616282463,
+            -0.020441556349396706,
+            -0.011486129835247993,
+            -0.00041481092921458185,
+            -0.005040998570621014,
+            -0.0004956811899319291,
+            -0.020327266305685043,
+            0.0026960158720612526,
+            0.014637264423072338,
+            0.007661501411348581,
+            0.017927179113030434,
+            -0.002889900468289852,
+            0.017437364906072617,
+            0.019690508022904396,
+            -0.01400050614029169,
+            -0.029862305149435997,
+            -0.0006515541463159025,
+            0.01498013362288475,
+            -0.01661284640431404,
+            0.019445601850748062,
+            0.004085862077772617,
+            0.0029531680047512054,
+            0.004779764451086521,
+            -0.006049198564141989,
+            0.018482301384210587,
+            0.02039257436990738,
+            0.01453930139541626,
+            -0.013167822733521461,
+            0.014057651162147522,
+            0.009771781973540783,
+            0.006600238848477602,
+            -0.01646590232849121,
+            -0.013535182923078537,
+            0.002555194543674588,
+            0.0009245232213288546,
+            0.024392718449234962,
+            -0.020588500425219536,
+            0.0016602640971541405,
+            0.010588138364255428,
+            0.01769859902560711,
+            -0.004302196204662323,
+            0.008906444534659386,
+            0.0008740111952647567,
+            0.007134951651096344,
+            -0.0035919665824621916,
+            -0.006457376293838024,
+            -0.006673710886389017,
+            -0.010392212308943272,
+            -0.0020959940738976,
+            -0.02070278860628605,
+            0.002198038622736931,
+            -0.002830714685842395,
+            0.018237393349409103,
+            -0.01018812321126461,
+            0.038140155375003815,
+            0.03399306535720825,
+            -0.0007025764207355678,
+            0.011102442629635334,
+            -0.01709449477493763,
+            -0.003945040516555309,
+            -0.021959977224469185,
+            0.01844964735209942,
+            0.016392430290579796,
+            0.012269831262528896,
+            0.005302232690155506,
+            0.017796561121940613,
+            -0.0006301248213276267,
+            0.014490320347249508,
+            0.0029654132667928934,
+            0.007971717044711113,
+            -0.002265388146042824,
+            0.003938918001949787,
+            -0.031103165820240974,
+            -0.017029186710715294,
+            -0.01138000376522541,
+            0.004796091932803392,
+            -0.006375740747898817,
+            -0.0035205355379730463,
+            0.0084737753495574,
+            -0.005902254022657871,
+            0.0004270562494639307,
+            -0.025992777198553085,
+            -0.0157148540019989,
+            -0.010318740271031857,
+            -0.004040962550789118,
+            -0.01169838197529316,
+            -0.0027694879099726677,
+            -0.0022082431241869926,
+            0.0029552089981734753,
+            0.020947696641087532,
+            -0.012808626517653465,
+            -0.016294466331601143,
+            0.004379750229418278,
+            -0.002263347152620554,
+            -0.014016833156347275,
+            -0.016898570582270622,
+            0.006665547378361225,
+            0.012098397128283978,
+            -0.0017470018938183784,
+            -0.013608654960989952,
+            0.011739199981093407,
+            -0.006628811359405518,
+            0.04663025587797165,
+            -0.013584164902567863,
+            -0.0078002819791436195,
+            0.003720542648807168,
+            0.004698128905147314,
+            -0.009396257810294628,
+            0.007330877240747213,
+            0.013020878657698631,
+            0.01830270327627659,
+            0.0095758568495512,
+            -0.008465612307190895,
+            -0.0024388637393712997,
+            0.007796200457960367,
+            -0.016147522255778313,
+            0.006404313258826733,
+            0.0008525818702764809,
+            -0.014400520361959934,
+            0.006110425107181072,
+            0.028980640694499016,
+            0.013151495717465878,
+            0.01288209855556488,
+            -0.022351829335093498,
+            -0.004094025585800409,
+            -0.009355440735816956,
+            -0.004943036008626223,
+            -0.010090161114931107,
+            0.027070365846157074,
+            -0.018155759200453758,
+            0.02651524543762207,
+            0.00898808054625988,
+            0.0017684312770143151,
+            -0.01176369097083807,
+            0.006343086715787649,
+            0.02832755446434021,
+            0.0028592869639396667,
+            -0.020000724121928215,
+            -0.013592327944934368,
+            0.006314514204859734,
+            0.01707816869020462,
+            -0.002647034591063857,
+            -0.005428767763078213,
+            0.019657853990793228,
+            0.01784554310142994,
+            -0.004334850702434778,
+            -0.00936360377818346,
+            0.0008337036124430597,
+            0.00928196869790554,
+            0.016963878646492958,
+            0.015020951628684998,
+            0.009086042642593384,
+            -0.002647034591063857,
+            0.008037025108933449,
+            -0.008800318464636803,
+            -0.016449574381113052,
+            0.007220669183880091,
+            -0.008432958275079727,
+            -0.016963878646492958,
+            0.01872720755636692,
+            0.015257694758474827,
+            0.0181394312530756,
+            0.0024592725094407797,
+            -0.016735298559069633,
+            0.022302847355604172,
+            0.005755309946835041,
+            0.002998067531734705,
+            0.02160078100860119,
+            -0.022221211344003677,
+            0.013012715615332127,
+            0.02037624642252922,
+            0.027609162032604218,
+            -0.01737205684185028,
+            -0.010906516574323177,
+            -0.004857318475842476,
+            -0.006853308994323015,
+            -0.00176945177372545,
+            -0.008253359235823154,
+            -0.012620864436030388,
+            0.0028735732194036245,
+            -0.030466407537460327,
+            0.018106777220964432,
+            -0.005102225113660097,
+            0.010669773444533348,
+            -0.019266001880168915,
+            0.011363675817847252,
+            -0.0035838030744343996,
+            0.011135096661746502,
+            0.0019612954929471016,
+            -0.030041903257369995,
+            -0.008571738377213478,
+            0.015388311818242073,
+            -0.0030368445441126823,
+            -0.013641309924423695,
+            0.02006603218615055,
+            0.00018942011229228228,
+            -0.022270193323493004,
+            -0.004314441699534655,
+            -0.006934944540262222,
+            0.002193956868723035,
+            -0.0029613315127789974,
+            -0.0052981507033109665,
+            -0.030188847333192825,
+            0.020033378154039383,
+            0.009559528902173042,
+            -0.03072764165699482,
+            -0.003255219664424658,
+            -0.014996460638940334,
+            -0.0025184585247188807,
+            0.008881953544914722,
+            -0.004000144544988871,
+            -0.015551582910120487,
+            0.01784554310142994,
+            0.022711025550961494,
+            -0.0023164101876318455,
+            -0.016057724133133888,
+            -0.0090697156265378,
+            -0.027494871988892555,
+            0.030825603753328323,
+            -0.0066696288995444775,
+            0.0022878379095345736,
+            -0.005322641227394342,
+            0.009388094767928123,
+            0.004228724166750908,
+            0.0157148540019989,
+            0.006081852596253157,
+            -0.002647034591063857,
+            0.026596879586577415,
+            -0.01964152604341507,
+            0.009567692875862122,
+            -1.588864870427642e-05,
+            -0.003308282932266593,
+            0.01890680566430092,
+            0.0007939062779769301,
+            0.021649762988090515,
+            -0.008571738377213478,
+            -0.0001475818717153743,
+            -0.0034388997592031956,
+            0.010898353531956673,
+            -0.0028450009413063526,
+            -0.003465431509539485,
+            -0.001531688030809164,
+            0.011118769645690918,
+            0.01013097818940878,
+            0.003804219188168645,
+            -0.01737205684185028,
+            0.003142970846965909,
+            -0.008547247387468815,
+            -0.007028825581073761,
+            0.01302904263138771,
+            -0.011192241683602333,
+            -0.0018990482203662395,
+            -0.03160930797457695,
+            0.016286302357912064,
+            0.0533243790268898,
+            0.008547247387468815,
+            -0.014433175325393677,
+            -0.0018806802108883858,
+            -0.019755816087126732,
+            0.01535565685480833,
+            0.011004479601979256,
+            -0.016816934570670128,
+            -0.011984107084572315,
+            -0.012310649268329144,
+            0.007649256382137537,
+            0.021306892856955528,
+            -0.006522684823721647,
+            -0.008579902350902557,
+            0.025372346863150597,
+            -0.011788181029260159,
+            -0.010882026515901089,
+            -0.01661284640431404,
+            -0.01635977439582348,
+            0.00241641397587955,
+            0.019902760162949562,
+            -0.0169475506991148,
+            0.021176276728510857,
+            0.00936360377818346,
+            0.0010133018950000405,
+            0.003624620847404003,
+            0.008588065393269062,
+            0.0024878450203686953,
+            0.017878197133541107,
+            0.003883813973516226,
+            -0.01439235731959343,
+            0.0021408938337117434,
+            -0.0030286810360848904,
+            0.004624656867235899,
+            3.4320437407586724e-06,
+            -0.002749078907072544,
+            -0.0065635028295218945,
+            0.003710338147357106,
+            -0.021306892856955528,
+            0.021502818912267685,
+            -0.022678371518850327,
+            -0.004245051648467779,
+            0.005555302836000919,
+            -0.005118552595376968,
+            0.013469874858856201,
+            0.018792515620589256,
+            0.01423724927008152,
+            0.008702355436980724,
+            0.01462093647569418,
+            -0.011331021785736084,
+            0.02323349379003048,
+            -0.013282112777233124,
+            0.0005816536722704768,
+            0.012939243577420712,
+            0.01755165494978428,
+            0.018792515620589256,
+            -0.006245123688131571,
+            -0.010384049266576767,
+            -0.0002946535241790116,
+            0.0129637336358428,
+            -0.022286519408226013,
+            -0.012269831262528896,
+            -0.01386172603815794,
+            0.011486129835247993,
+            -0.004743028432130814,
+            0.025453981012105942,
+            -0.00633492274209857,
+            -0.014653591439127922,
+            -0.015918942168354988,
+            0.014196431264281273,
+            -0.048752784729003906,
+            0.0030348035506904125,
+            -0.0024919267743825912,
+            0.03043375350534916,
+            -0.027021385729312897,
+            0.01738838478922844,
+            -0.0028490826953202486,
+            -0.009355440735816956,
+            0.015429128892719746,
+            -0.015992414206266403,
+            -0.0046613928861916065,
+            -0.0016806729836389422,
+            -0.007539048325270414,
+            -0.00112248957157135,
+            0.017616963014006615,
+            -0.015845470130443573,
+            0.01282495353370905,
+            -0.00989423505961895,
+            0.0027021386194974184,
+            -0.004526694305241108,
+            -0.007751300930976868,
+            0.0196088720113039,
+            0.034548189491033554,
+            -0.007053316105157137,
+            0.0014990337658673525,
+            -0.022547753527760506,
+            0.013192313723266125,
+            -0.006730855442583561,
+            0.02351105399429798,
+            -0.005432849284261465,
+            -0.016898570582270622,
+            -0.013976015150547028,
+            -0.009739127941429615,
+            0.00839214026927948,
+            -0.022433463484048843,
+            -0.011110605672001839,
+            -0.00922482367604971,
+            -0.002573562553152442,
+            -0.018694553524255753,
+            0.004865481983870268,
+            -0.002426618244498968,
+            -0.012506574392318726,
+            -0.008816645480692387,
+            0.01048201136291027,
+            -0.009347276762127876,
+            0.003077662317082286,
+            -0.0011929002357646823,
+            -0.011453475803136826,
+            -0.02369065210223198,
+            0.0158944521099329,
+            0.0001974561164388433,
+            -0.010359558276832104,
+            0.008408467285335064,
+            -0.0018276170594617724,
+            0.001211268245242536,
+            0.016963878646492958,
+            -0.0020664012990891933,
+            0.003467472270131111,
+            0.023478399962186813,
+            0.004057289566844702,
+            -0.010971825569868088,
+            -0.00025906547671183944,
+            0.013535182923078537,
+            -0.005118552595376968,
+            -0.010008525103330612,
+            1.6646010408294387e-05,
+            -0.02984597720205784,
+            -0.008351322263479233,
+            0.006094098091125488,
+            0.018629245460033417,
+            -0.010620792396366596,
+            -0.015853634104132652,
+            0.026319319382309914,
+            -0.006359413731843233,
+            0.019070077687501907,
+            0.002889900468289852,
+            0.0025694805663079023,
+            0.01724144071340561,
+            0.013461710885167122,
+            0.00786150898784399,
+            -0.013927034102380276,
+            -0.02429475635290146,
+            0.0016398552106693387,
+            -0.018090449273586273,
+            0.010579974390566349,
+            -0.011028970591723919,
+            0.0032756286673247814,
+            -0.02398454025387764,
+            -0.012237177230417728,
+            0.004934872500598431,
+            -0.020343592390418053,
+            -0.00040588201954960823,
+            -0.0056940834037959576,
+            0.022237539291381836,
+            -0.005310396198183298,
+            0.008669701404869556,
+            0.0018255761824548244,
+            -0.012237177230417728,
+            0.0009877908742055297,
+            0.009469729848206043,
+            -0.029192892834544182,
+            -0.0007127808639779687,
+            -0.003138889092952013,
+            0.0011418780777603388,
+            0.005351213738322258,
+            0.009755454957485199,
+            0.007322713732719421,
+            0.0027062203735113144,
+            0.007253323681652546,
+            0.010449357330799103,
+            -0.00022041612828616053,
+            0.0197884701192379,
+            0.018172085285186768,
+            -0.007351286243647337,
+            0.008710519410669804,
+            0.024817224591970444,
+            -0.013159659691154957,
+            -0.008718682453036308,
+            0.023266147822141647,
+            -0.006461458280682564,
+            -0.0011122850701212883,
+            -0.01483318954706192,
+            0.00815947912633419,
+            0.0017367975087836385,
+            0.028131630271673203,
+            0.019657853990793228,
+            0.005820618476718664,
+            0.000639819074422121,
+            0.015788326039910316,
+            -0.013020878657698631,
+            0.0035286990460008383,
+            0.015461783856153488,
+            -0.003940958995372057,
+            -0.019821124151349068,
+            0.013878053054213524,
+            0.009502384811639786,
+            -0.003228688146919012,
+            0.01551892887800932,
+            0.011412657797336578,
+            -0.030841931700706482,
+            -0.0011898389784619212,
+            0.010498338378965855,
+            -0.0021449755877256393,
+            -0.008751336485147476,
+            -0.00014273475972004235,
+            -0.012384121306240559,
+            0.021388528868556023,
+            -0.004408322740346193,
+            0.012277995236217976,
+            -0.00815947912633419,
+            -0.017127150669693947,
+            -0.011559601873159409,
+            0.007914572022855282,
+            -0.010547320358455181,
+            -0.0006714528426527977,
+            -0.004022594541311264,
+            -0.005779800936579704,
+            0.017747581005096436,
+            0.004706292413175106,
+            -0.0008964610169641674,
+            -0.0018398624379187822,
+            0.007983962073922157,
+            -0.006979844067245722,
+            -0.008726846426725388,
+            0.004816500470042229,
+            -0.012833117507398129,
+            0.003981776535511017,
+            0.005543057806789875,
+            0.01506176870316267,
+            -0.014857680536806583,
+            -0.021159948781132698,
+            0.007020662073045969,
+            0.02695607766509056,
+            -0.02516009286046028,
+            -0.005232842173427343,
+            0.006245123688131571,
+            0.002093953313305974,
+            0.006914535537362099,
+            -0.006934944540262222,
+            -0.007208424154669046,
+            -0.03657275065779686,
+            -0.0020745648071169853,
+            0.0009005427709780633,
+            0.008963589556515217,
+            -0.008049271069467068,
+            -0.0136494729667902,
+            -0.011135096661746502,
+            0.002410291228443384,
+            0.007134951651096344,
+            -0.02204161323606968,
+            0.0049593630246818066,
+            0.009608510881662369,
+            -0.019559891894459724,
+            0.025127438828349113,
+            -0.010890189558267593,
+            -6.964537897147238e-05,
+            -0.00995954405516386,
+            0.021633435040712357,
+            -0.013486201874911785,
+            -0.018368011340498924,
+            -0.0006898208521306515,
+            0.008355404250323772,
+            0.011926962062716484,
+            -0.0017663904000073671,
+            -0.011371839791536331,
+            0.0003752686607185751,
+            0.01646590232849121,
+            0.019135385751724243,
+            0.002744997153058648,
+            -0.0024837632663547993,
+            -0.01019628718495369,
+            -0.009592183865606785,
+            -0.0073961857706308365,
+            -0.0007449248805642128,
+            0.002091912319883704,
+            0.02035992033779621,
+            0.013469874858856201,
+            0.0024837632663547993,
+            -0.015135240741074085,
+            -0.005257333163172007,
+            0.008939098566770554,
+            0.009412584826350212,
+            -0.011690218932926655,
+            -0.00898808054625988,
+            0.002659279853105545,
+            0.02249877341091633,
+            0.017633290961384773,
+            -0.0014000505907461047,
+            0.0032878739293664694,
+            -0.0016592437168583274,
+            0.012506574392318726,
+            -0.007555375341325998,
+            -0.010269759222865105,
+            -0.006943108048290014,
+            -0.014645427465438843,
+            -0.010800390504300594,
+            -0.0023062059190124273,
+            -0.010179960168898106,
+            0.022629389539361,
+            0.007988044060766697,
+            0.009779945947229862,
+            0.008265605196356773,
+            0.010269759222865105,
+            0.002087830565869808,
+            0.025715215131640434,
+            -0.004424649756401777,
+            -0.011984107084572315,
+            0.007236996199935675,
+            -0.022368155419826508,
+            -0.021535472944378853,
+            0.008257441222667694,
+            0.006367577239871025,
+            0.03595232218503952,
+            -0.026890767738223076,
+            0.0052981507033109665,
+            -0.009779945947229862,
+            -0.010400376282632351,
+            0.007996207103133202,
+            -0.0015439334092661738,
+            -0.004245051648467779,
+            0.022368155419826508,
+            0.00726556871086359,
+            0.0035797213204205036,
+            0.0032817514147609472,
+            0.019837452098727226,
+            -0.005685919895768166,
+            0.01573934406042099,
+            0.0259274672716856,
+            0.024621298536658287,
+            0.009388094767928123,
+            -0.009469729848206043,
+            0.02785406820476055,
+            0.0002895512734539807,
+            -0.0034327772445976734,
+            0.007220669183880091,
+            -0.00305113079957664,
+            -0.0009556467994116247,
+            -0.010033016093075275,
+            0.021764053031802177,
+            -0.001455154619179666,
+            0.004359341226518154,
+            0.02081707864999771,
+            -7.774515688652173e-05,
+            0.006220633164048195,
+            0.01057181041687727,
+            0.002181711606681347,
+            0.003396041225641966,
+            0.010775899514555931,
+            -0.039315707981586456,
+            -0.0006571666453965008,
+            -0.025600925087928772,
+            -0.0018286375561729074,
+            0.0007357408758252859,
+            -0.01266984548419714,
+            0.013600491918623447,
+            -0.002015379024669528,
+            -0.010751409456133842,
+            -0.00029337796149775386,
+            0.021649762988090515,
+            0.005416522268205881,
+            0.004071575589478016,
+            -0.004469549283385277,
+            0.018792515620589256,
+            -0.006955353543162346,
+            -0.017861871048808098,
+            -0.012759645469486713,
+            -0.012367794290184975,
+            -0.01918436773121357,
+            -0.01214737817645073,
+            -0.0024551907554268837,
+            -0.002667443361133337,
+            0.004738946910947561,
+            0.002481722505763173,
+            -0.003228688146919012,
+            -0.02070278860628605,
+            0.015037278644740582,
+            -0.01430255826562643,
+            0.012873934581875801,
+            -0.011053460650146008,
+            0.005963481031358242,
+            -0.004122598096728325,
+            0.012726990506052971,
+            -0.021404854953289032,
+            0.02070278860628605,
+            -0.01536382082849741,
+            0.0038532004691660404,
+            0.019592545926570892,
+            0.006551257334649563,
+            -0.008118661120533943,
+            -0.013404566794633865,
+            -0.022074267268180847,
+            0.021633435040712357,
+            -0.011633073911070824,
+            -0.018237393349409103,
+            -0.004608329851180315,
+            0.009649328887462616,
+            -0.017470018938183784,
+            -0.007367613259702921,
+            0.01707816869020462,
+            -0.0018949664663523436,
+            0.012980061583220959,
+            0.023004913702607155,
+            -0.0052981507033109665,
+            0.0036572751123458147,
+            -0.009167678654193878,
+            0.0010286086471751332,
+            -0.02470293454825878,
+            0.013527019880712032,
+            0.008971752598881721,
+            0.006269614677876234,
+            -0.013935198076069355,
+            -0.013306603766977787,
+            0.0036531933583319187,
+            0.004498121794313192,
+            0.011437147855758667,
+            0.009420748800039291,
+            0.004220560658723116,
+            -0.013918871060013771,
+            -0.027788760140538216,
+            -0.027070365846157074,
+            0.03595232218503952,
+            -0.0005002731923013926,
+            -0.020947696641087532,
+            0.03399306535720825,
+            0.016114868223667145,
+            0.0022980421781539917,
+            -0.012449429370462894,
+            -0.004314441699534655,
+            0.024621298536658287,
+            0.016400592401623726,
+            0.010465684346854687,
+            -0.0004742518358398229,
+            0.005571629852056503,
+            0.0016469983384013176,
+            0.03012353740632534,
+            0.015323002822697163,
+            0.0009459525463171303,
+            -0.038597315549850464,
+            0.011943289078772068,
+            -0.007547211833298206,
+            0.0013408648082986474,
+            0.006702283397316933,
+            0.017780235037207603,
+            0.011935125105082989,
+            0.015567909926176071,
+            0.014865843579173088,
+            -0.003065417055040598,
+            0.0029511270113289356,
+            0.0006163488142192364,
+            -0.009624837897717953,
+            -0.001767410896718502,
+            -0.008881953544914722,
+            -0.005714492406696081,
+            -0.002920513739809394,
+            0.005592038854956627,
+            0.007657419890165329,
+            -0.003983817528933287,
+            0.028049994260072708,
+            -0.0006433906382881105,
+            -0.017600636929273605,
+            -0.009061552584171295,
+            -0.03611559048295021,
+            0.00434301421046257,
+            -0.030939893797039986,
+            -0.011257549747824669,
+            0.0033674687147140503,
+            -0.020457882434129715,
+            -0.024768242612481117,
+            0.0010306495241820812,
+            0.014335212297737598,
+            0.007392104249447584,
+            0.0033327736891806126,
+            -0.01004934310913086,
+            0.003816464450210333,
+            -0.001855169073678553,
+            -0.006461458280682564,
+            0.028833696618676186,
+            0.019559891894459724,
+            0.020457882434129715,
+            -0.01190247107297182,
+            0.019298657774925232,
+            0.005845109466463327,
+            0.02035992033779621,
+            0.010465684346854687,
+            -0.0048695639707148075,
+            0.0074859848245978355,
+            0.009567692875862122,
+            -0.006155324634164572,
+            -0.021519144997000694,
+            0.013192313723266125,
+            -0.0010837126756086946,
+            -0.0052083516493439674,
+            -0.016269976273179054,
+            -0.003534821793437004,
+            -0.015812816098332405,
+            0.010547320358455181,
+            -0.017159804701805115,
+            0.012498411349952221,
+            -0.000802069844212383,
+            0.00168475485406816,
+            -0.015469946898519993,
+            -0.013951525092124939,
+            -0.0012439225101843476,
+            -0.01244126632809639,
+            -0.008971752598881721,
+            0.006163488142192364,
+            -0.007539048325270414,
+            -0.008318668231368065,
+            -0.00535937724635005,
+            -0.011437147855758667,
+            -0.010882026515901089,
+            0.004367504734545946,
+            -0.0010653446661308408,
+            0.020931368693709373,
+            0.00011671340325847268,
+            -0.005824700463563204,
+            0.01190247107297182,
+            0.02787039615213871,
+            0.0016704685986042023,
+            0.004583839327096939,
+            -0.014261740259826183,
+            -0.011118769645690918,
+            -0.02279266156256199,
+            0.016572028398513794,
+            0.0049675265327095985,
+            -0.0344502255320549,
+            0.01438419334590435,
+            0.00824927818030119,
+            0.0015102586476132274,
+            -0.006698201410472393,
+            -0.005334886722266674,
+            -0.005608365871012211,
+            0.006306350696831942,
+            -0.0022837561555206776,
+            0.0005107327597215772,
+            0.0035919665824621916,
+            0.034678805619478226,
+            -0.0040674940682947636,
+            -0.007228832691907883,
+            0.00981259997934103,
+            0.0005130287609063089,
+            0.020196648314595222,
+            -0.005518566817045212,
+            0.0037572786677628756,
+            -0.019282329827547073,
+            -0.016563864424824715,
+            0.0018429238116368651,
+            -0.01722511276602745,
+            0.023282473906874657,
+            -0.005392031744122505,
+            -0.01347803883254528,
+            -0.006906372029334307,
+            -0.020115012302994728,
+            0.012637191452085972,
+            0.02310287579894066,
+            0.007016580086201429,
+            0.01033506728708744,
+            0.012131051160395145,
+            -0.02444170042872429,
+            -0.020457882434129715,
+            0.012171869166195393,
+            0.0013571919407695532,
+            -0.008449285291135311,
+            -0.005404277238994837,
+            -0.001783737912774086,
+            0.007783954963088036,
+            -0.0032633834052830935,
+            0.007881917990744114,
+            0.06491663306951523,
+            -0.01093100756406784,
+            -0.002987863030284643,
+            -0.02966637909412384,
+            -0.004396077245473862,
+            -0.006628811359405518,
+            0.0056287748739123344,
+            0.009151351638138294,
+            -0.023135529831051826,
+            0.009232986718416214,
+            -0.030025575309991837,
+            -0.010628955438733101,
+            0.00824111420661211,
+            0.0028490826953202486,
+            -0.01220452319830656,
+            -0.0009592183632776141,
+            -0.013665799982845783,
+            0.0006107363733462989,
+            -0.009412584826350212,
+            -0.00576347392052412,
+            0.012718827463686466,
+            -0.02321716584265232,
+            0.006298186723142862,
+            0.007592111360281706,
+            -0.006514521315693855,
+            -0.013722945004701614,
+            -0.017861871048808098,
+            -0.011967780068516731,
+            -0.013314766809344292,
+            0.002996026771143079,
+            0.01190247107297182,
+            -0.02607441134750843,
+            0.012922916561365128,
+            -0.0037389106582850218,
+            0.00945340283215046,
+            0.004404240753501654,
+            0.013469874858856201,
+            0.018074123188853264,
+            0.009355440735816956,
+            0.0021429345943033695,
+            -0.008024780079722404,
+            0.012702500447630882,
+            -0.01993541419506073,
+            -0.01635161228477955,
+            0.013837235048413277,
+            0.012016761116683483,
+            0.003061335301026702,
+            0.009828926995396614,
+            0.0020959940738976,
+            0.011886144056916237,
+            0.010775899514555931,
+            -0.019739490002393723,
+            -0.0006648199632763863,
+            0.0007326795603148639,
+            0.03280118480324745,
+            -0.0005938990507274866,
+            0.008857463486492634,
+            0.004420568235218525,
+            0.013159659691154957,
+            -0.012784135527908802,
+            -0.01769859902560711,
+            0.0008087027235887945,
+            -0.01057181041687727,
+            0.021257910877466202,
+            -0.008192133158445358,
+            0.006216551177203655,
+            -0.018612917512655258,
+            -0.027086693793535233,
+            0.006392067763954401,
+            -0.016261812299489975,
+            -0.018221067264676094,
+            0.01317598670721054,
+            0.0077268099412322044,
+            0.0017572063952684402,
+            -0.005098143592476845,
+            -0.020457882434129715,
+            -0.009837090037763119,
+            0.028441844508051872,
+            0.011102442629635334,
+            -0.004120557103306055,
+            0.009584019891917706,
+            -0.009094206616282463,
+            -0.012490247376263142,
+            -0.0017541450215503573,
+            0.0036715613678097725,
+            0.008261523209512234,
+            0.005645101889967918,
+            -0.01386172603815794,
+            -0.00029873527819290757,
+            -0.012400448322296143,
+            -0.021225256845355034,
+            -0.008906444534659386,
+            0.016751626506447792,
+            0.007820690982043743,
+            -0.0038960592355579138,
+            0.015812816098332405,
+            -0.004404240753501654,
+            0.002189875114709139,
+            0.014294394291937351,
+            0.005334886722266674,
+            -0.004285869188606739,
+            -0.0095758568495512,
+            -0.004612411838024855,
+            0.004043003544211388,
+            -0.0025960123166441917,
+            0.006457376293838024,
+            0.017192458733916283,
+            0.0072900597006082535,
+            -0.013755599036812782,
+            0.013135168701410294,
+            0.0007791098323650658,
+            0.0049675265327095985,
+            -0.03699725493788719,
+            -0.003247056156396866,
+            0.015796490013599396,
+            -0.013020878657698631,
+            0.02501314878463745,
+            0.005400195252150297,
+            0.022743679583072662,
+            0.015233203768730164,
+            0.008669701404869556,
+            -0.001542912912555039,
+            0.00464098434895277,
+            0.02759283408522606,
+            -0.003151134354993701,
+            -0.008637047372758389,
+            -0.0020602785516530275,
+            0.009241150692105293,
+            0.0002063850115519017,
+            0.003000108525156975,
+            0.008281932212412357,
+            -0.03562577813863754,
+            0.011706545948982239,
+            -0.019576217979192734,
+            0.0024694770108908415,
+            0.003459308762103319,
+            -0.007183933164924383,
+            -0.014865843579173088,
+            -0.005857354495674372,
+            0.022841641679406166,
+            -0.003224606392905116,
+            0.021078312769532204,
+            0.002885818714275956,
+            0.016106704249978065,
+            -0.011575928889214993,
+            -0.009208496659994125,
+            -0.0004130251472815871,
+            -0.005351213738322258,
+            -0.007539048325270414,
+            0.010596301406621933,
+            0.0034409407526254654,
+            0.005534893833100796,
+            0.002265388146042824,
+            0.0036654386203736067,
+            -0.0036878883838653564,
+            0.02773977816104889,
+            0.003563394071534276,
+            0.007996207103133202,
+            -0.011567764915525913,
+            0.014082142151892185,
+            -0.017029186710715294,
+            0.0070778070949018,
+            0.005575711838901043,
+            -0.026156047359108925,
+            0.016849588602781296,
+            0.004775682929903269,
+            0.006628811359405518,
+            0.013812744058668613,
+            0.01618834026157856,
+            -0.010408539324998856,
+            -0.014816862531006336,
+            0.024049850180745125,
+            -0.0032041973900049925,
+            -0.015225040726363659,
+            -0.006775755435228348,
+            4.438935866346583e-05,
+            0.006559420842677355,
+            0.00373074715025723,
+            0.0013020879123359919,
+            -0.0006204306264407933,
+            -0.010743245482444763,
+            0.004645065870136023,
+            -0.01934763789176941,
+            -0.0007163524278439581,
+            -0.016449574381113052,
+            -0.015633217990398407,
+            0.018678225576877594,
+            -0.01830270327627659,
+            0.009633000940084457,
+            0.0012275953777134418,
+            0.028588788583874702,
+            -0.011706545948982239,
+            -0.012155541218817234,
+            0.0015102586476132274,
+            0.010849371552467346,
+            0.0067512644454836845,
+            -0.010033016093075275,
+            -0.007583947852253914,
+            -0.014180104248225689,
+            -0.010163633152842522,
+            0.015502601861953735,
+            -0.014498483389616013,
+            -0.014343376271426678,
+            0.008914608508348465
+          ],
+          "embedding": [
+            -0.006359657085039302,
+            -0.024594786241399234,
+            -0.02553538924848095,
+            0.005360188465088291,
+            0.012801870977589839,
+            0.021680505862506104,
+            -0.016668048966001273,
+            0.06860852474581301,
+            -0.007122272841157724,
+            0.03471775000705488,
+            -0.009004297848861675,
+            0.01059974144678391,
+            0.02565604599426148,
+            -0.022544127646041287,
+            0.010616674976540785,
+            0.013658194823359737,
+            0.016400329743363844,
+            0.02858546964917478,
+            0.0067612676434693686,
+            -0.04174737193606879,
+            0.03184155110488616,
+            -0.02223105127585514,
+            -0.05506239667974977,
+            -0.0015307551410555752,
+            -0.013169788751379287,
+            0.017681749722384007,
+            0.01682706317229353,
+            0.019652295823816928,
+            -0.05623980369271269,
+            0.030698582851533197,
+            0.02286327124296613,
+            0.016019032539689178,
+            0.005508113353872315,
+            8.58985091215149e-05,
+            -0.019206292853495906,
+            -0.018439290483495144,
+            0.03475023176950151,
+            0.01018612942141759,
+            -0.0252551158857426,
+            0.0334562130666827,
+            0.020462153537485785,
+            0.02260508249714211,
+            -0.03323524989392951,
+            -0.002094251009530707,
+            0.015177442347686773,
+            -0.00018698464350038317,
+            -0.0006788648344273403,
+            0.0007777616893581039,
+            0.013352398615059647,
+            -0.030751111528744813,
+            -0.0317733818445192,
+            -0.006729064081710058,
+            0.009413204899986514,
+            -0.005712894407943592,
+            -0.038961826880527066,
+            -0.002827692206530345,
+            0.017863300537973137,
+            0.07918455388563496,
+            0.048967158919816366,
+            0.018002566325350088,
+            0.040264962620950695,
+            -0.019715100359659787,
+            -0.02211350500879408,
+            0.004918148608475574,
+            0.01062585305866606,
+            -0.014478993521696027,
+            0.0034983131226364672,
+            0.000766228629514787,
+            0.009667268771502067,
+            -0.01558439890697323,
+            -0.04984718709870083,
+            0.029613955432821887,
+            -0.009686854114630624,
+            -0.015279319238124568,
+            0.007460629676325322,
+            -0.006235979350809034,
+            0.012514387473717085,
+            0.02800435263741497,
+            -0.008688925946672778,
+            -0.01236301543374469,
+            0.029755705020188972,
+            0.01158037219446542,
+            -0.005242392206024511,
+            0.009982679027728814,
+            0.052414361714483264,
+            -0.018788500381726227,
+            -0.020335831196027204,
+            -0.08583959943356621,
+            0.029309228216812346,
+            -0.0249285282010673,
+            -0.018929396555426307,
+            0.007710663193334574,
+            0.03096798035958051,
+            -0.017089977064591828,
+            -0.028015107605405817,
+            0.06740703271128606,
+            0.009398795412354399,
+            0.01118769022826835,
+            -0.03041417460855722,
+            0.014776002883150835,
+            0.004319194067673161,
+            0.022507636639041203,
+            0.0053562788646786845,
+            0.007548072258943124,
+            0.011123374308563128,
+            -0.006924362115585165,
+            -0.011118507147092024,
+            0.009563232643295304,
+            0.00657178473678069,
+            0.08313027447380966,
+            -0.004239370156003264,
+            -0.00930406163946907,
+            -0.039202672821898946,
+            0.009761029930556298,
+            -0.02827698465424993,
+            -0.037914533246900244,
+            -0.04496050911855815,
+            -0.017711307900236353,
+            0.01934435542287117,
+            0.004335173027873153,
+            0.03800509969714242,
+            0.031198127357716314,
+            0.031311207941680276,
+            -0.012184553894588813,
+            -0.058985863670856876,
+            0.041442017057707635,
+            -0.012456189292721711,
+            -0.0040966351720394135,
+            0.010849179519959421,
+            0.005243123049347636,
+            0.0011495799963862647,
+            0.003534183676934487,
+            -0.03139340512003631,
+            -0.023876261883905306,
+            0.008382957569752911,
+            -0.004778283217011976,
+            0.03703067589888386,
+            0.06888630688839406,
+            -0.03993532296478411,
+            -0.009472167678222372,
+            0.025246872942105057,
+            -0.01524567912700368,
+            0.01595917939878582,
+            -0.003736723164442467,
+            -0.030836797784967753,
+            0.0026246446163368533,
+            0.008894300082148884,
+            0.08342856279154155,
+            -0.0007476580521704578,
+            0.025149116238550588,
+            0.004972542633631468,
+            0.034495632546556576,
+            -0.009173329764686424,
+            0.014009381543772768,
+            0.004427261668151028,
+            -0.013426138607902904,
+            -0.022234568880321808,
+            0.01976282364689966,
+            -0.03592433789490043,
+            -0.010918278220831188,
+            0.013527180289641174,
+            -0.013023113052242677,
+            -0.008304770302495363,
+            0.003677063799167178,
+            0.0013981822663219421,
+            0.047001163242600175,
+            0.01828332714887986,
+            0.002483375662343689,
+            -0.06820498187523893,
+            0.05295312038995555,
+            0.025038753504342162,
+            0.025286870198436735,
+            -0.0029544808216316115,
+            0.029982752196635428,
+            0.0005600481523874767,
+            0.0128290258663539,
+            0.009023805227424919,
+            -0.0019219166367750596,
+            -0.04482843936478531,
+            0.04523479691409549,
+            -0.004678574027241186,
+            0.010486850804218904,
+            0.006736229865018014,
+            0.02134813656476443,
+            0.028083972673690036,
+            0.016857413440125445,
+            0.0189082447820146,
+            0.03843690613820022,
+            -0.011922027011229613,
+            0.016193433524453224,
+            0.03828207911467065,
+            -0.0013055128404331705,
+            0.043198163443700956,
+            0.024025562893260542,
+            -0.001257851946749882,
+            -0.011726713433348844,
+            -0.021907984099443396,
+            0.02752198420597673,
+            0.004431866336999851,
+            0.015798806714413912,
+            -0.012910782191382367,
+            -0.03180549180206364,
+            0.004550247372715741,
+            -0.022778272841118514,
+            0.031242218240017532,
+            -0.006106984949693483,
+            -0.006259993190497421,
+            0.02187449047033339,
+            0.009334146847221406,
+            -0.009381278960016145,
+            -0.020888853823931134,
+            -0.007502374908776156,
+            -0.010390018386728482,
+            -0.04316693435221676,
+            -0.006274513730720055,
+            0.019152443937064258,
+            -0.01668984824845791,
+            0.018573194036505578,
+            0.0035035262183249925,
+            -0.019779911381789077,
+            -0.0027495798165085993,
+            0.02425481236315913,
+            0.006857723545270935,
+            -0.014733535699087731,
+            -0.026326146471941727,
+            -0.019906660231056833,
+            -0.001033294559612807,
+            0.0159154063026895,
+            0.035649534442431415,
+            -0.026210592096240003,
+            0.014397127229517285,
+            0.005114311681443673,
+            -0.0033104236005207077,
+            0.00324988235167305,
+            0.03764004154553922,
+            -0.017822437760737605,
+            -0.008579701063708334,
+            -0.00032410984200965487,
+            0.018450351071676443,
+            0.017283379764846195,
+            0.006384411611332382,
+            -0.005115090309123598,
+            -0.030590047062583125,
+            -0.011776211359193503,
+            0.01183623914060493,
+            0.0016564847796208331,
+            -0.009530766634361575,
+            -0.004810598258154945,
+            0.03876242515526699,
+            0.027076785774401667,
+            0.005324763136165989,
+            -0.03416412480913939,
+            -0.007943016733586697,
+            0.002114307650456928,
+            -0.02961449347300628,
+            0.05142732770518828,
+            0.05566026433842563,
+            -0.0030729309664974704,
+            -0.00938186948393074,
+            -0.04547908981236149,
+            0.0651216359692231,
+            -0.01813722445944094,
+            -0.0482650853560993,
+            0.044045155892545554,
+            -0.014272653003441165,
+            0.013740652306746851,
+            0.04563732439184395,
+            0.02126513340521316,
+            -0.0054845632439384905,
+            -0.011635625101262898,
+            0.030992082069685376,
+            -0.010051935721068779,
+            -0.050967258971854425,
+            -0.009827942095033494,
+            0.013373878556654464,
+            -0.013017627321534534,
+            -0.009361001542607812,
+            -0.01925867665425919,
+            0.020329052915436022,
+            0.023020741670660188,
+            0.019387043947695172,
+            0.005628632318152202,
+            0.018451555161952665,
+            0.016226845400848455,
+            -0.01384033112836216,
+            0.004762589697426251,
+            -0.012718476561725138,
+            -0.024222765127494463,
+            -0.037487144314010595,
+            -0.019971524166659278,
+            -0.015034588610516958,
+            0.009483936496215338,
+            -0.014301924283553591,
+            -0.0024139665139412125,
+            0.013361045353529458,
+            -0.0005870798721107379,
+            -0.011192319340879789,
+            0.013927695434391159,
+            0.021337539669342293,
+            -0.027300952495815437,
+            -0.04765670695199488,
+            -0.00561706224251562,
+            0.0082776639906139,
+            -0.0030984973749633724,
+            -0.026269016057688548,
+            0.007040524859505573,
+            0.0017859008072463442,
+            0.020091542517503808,
+            -0.018716761130867725,
+            0.0035283051960672963,
+            0.008519383548064053,
+            -0.015247417050336866,
+            -0.016917626868166634,
+            -0.018580550690103854,
+            -0.024697180580294884,
+            0.034202028316737125,
+            0.0204395003463227,
+            0.030392895534267717,
+            -0.04527669147753553,
+            -0.018359428092075348,
+            0.05494107499558624,
+            0.015234754907198422,
+            -0.05066008106884831,
+            0.031139824660661085,
+            0.005016615710206006,
+            0.0019000453624929544,
+            -0.006094364589229031,
+            0.0637324238017742,
+            -0.02711381489292883,
+            -0.02187418906973867,
+            0.004601744864532875,
+            -0.04141121277496072,
+            -0.026689585608019588,
+            -0.005568100750837504,
+            0.01291019036002459,
+            -0.031853041426355644,
+            -0.02815800259372657,
+            -0.007964861641790374,
+            -0.010933515363060793,
+            0.013726639077745378,
+            -0.023956800669647334,
+            -0.015672058590455337,
+            -0.023642025374517057,
+            -0.01288435863544158,
+            -0.0042516881552192385,
+            -0.011162214160460695,
+            -0.02412733256710621,
+            0.010088528028211243,
+            0.03310175458095482,
+            0.023064360903748363,
+            0.001996159684818467,
+            0.009478229572933101,
+            0.014478772877286076,
+            0.00371783700962415,
+            0.007864103610320116,
+            -0.048511515124609,
+            -0.031398754206534776,
+            0.020227872561659397,
+            0.012159811587094092,
+            0.0172163448580163,
+            0.009578952796095119,
+            -0.04154545961112031,
+            -0.05095080393682759,
+            0.02968486144855133,
+            -0.03932395023720674,
+            -0.02063219739061335,
+            0.007512423249865194,
+            -0.011911709468952548,
+            -0.00810951734088105,
+            -0.0008079327452253181,
+            -0.01600701450676789,
+            0.023984909975645632,
+            -0.02418072541081482,
+            0.0033984265728439398,
+            -0.0104343782811971,
+            0.01588977364765743,
+            0.05109529246167992,
+            -0.027800269136366997,
+            -0.027889034481533438,
+            0.03706813210665186,
+            -0.02058120652429308,
+            0.030801396468451574,
+            0.03246127847621613,
+            0.008355517405358826,
+            -0.008605467586247932,
+            0.017693344563968573,
+            0.014499487024471321,
+            0.012431277080478217,
+            -0.040103870071440256,
+            0.017328848567062113,
+            0.02723900253727976,
+            -0.054909074068976875,
+            -0.008547596270650966,
+            -0.00777964838164648,
+            0.027134666572242665,
+            0.02765036594970255,
+            -0.002207173084941375,
+            -0.016838740117653725,
+            -0.028289233739722936,
+            -0.019433376533829132,
+            -0.011053380458946764,
+            0.03247641283550457,
+            0.0007514989009289282,
+            0.007919574219000797,
+            0.001316042436488591,
+            -0.0017934342041762042,
+            0.01924145734197619,
+            0.015260199553572459,
+            0.048259602293134723,
+            -0.03383050622747848,
+            0.0021712225589426266,
+            -0.03508567715696004,
+            -0.04578731401250471,
+            -0.008768228935292693,
+            0.010203318819548585,
+            0.024135236371743855,
+            -0.007467140460133838,
+            0.0028907962385387818,
+            -0.007582083854422456,
+            -0.005384477718912037,
+            -0.01867978482547249,
+            0.01641191108381631,
+            0.037168089880543244,
+            0.0031992495663980464,
+            -0.008533890994695162,
+            0.000778843942291118,
+            -0.004620258538699103,
+            0.010572690335163528,
+            0.033233014627100904,
+            -0.030497362482602697,
+            0.02974571152880676,
+            -0.004274442492343003,
+            0.025341347125910928,
+            -0.04307748954707389,
+            -0.018069609916024403,
+            -0.018491122727274337,
+            -0.02069307802429807,
+            0.05372967226264594,
+            -0.030768978301976772,
+            -0.03662045886993282,
+            -0.003205991669059688,
+            0.016815511161004534,
+            -0.0014152644009276566,
+            -0.0016015439520090104,
+            -0.03984716970293867,
+            -0.010828260334550217,
+            -0.05894985435251326,
+            -0.01638035185618134,
+            -0.021819442096404948,
+            -0.005315102328745982,
+            -0.06251579337157112,
+            0.0033713876515840233,
+            -0.07815974854883057,
+            0.001484860071649606,
+            0.04539427129131382,
+            -0.023530943197353476,
+            0.05142250021536681,
+            -0.024746600229063383,
+            -0.0005885740455257141,
+            -0.044698536284421976,
+            0.0026481001040031785,
+            0.02389178603305878,
+            0.051387657872255674,
+            -0.002949198236157656,
+            0.010674377196609455,
+            0.013322075519401089,
+            -0.026245090721950647,
+            -0.01036724558050635,
+            -0.005321884180696542,
+            0.005958798671856245,
+            0.013755178085972732,
+            -0.0074734329537091675,
+            0.038368016046563674,
+            0.010941421833174982,
+            0.03443115781919206,
+            0.034623617136788874,
+            -0.004361926531008214,
+            -0.03348681614127998,
+            -0.00632870804659212,
+            0.024433331840566692,
+            0.0321998807702602,
+            -0.02899983280320611,
+            0.005958471722097962,
+            -0.015602447981516634,
+            0.015633217539747047,
+            0.006321931116948646,
+            -0.011827541320878442,
+            0.02372204564999124,
+            0.015673426811623124,
+            0.006103300725621424,
+            -0.005165006670545588,
+            0.013589526056189892,
+            -0.025411864292353578,
+            0.03041791947375232,
+            -0.004456865298584878,
+            -0.04276058054583262,
+            -0.0024736078733106825,
+            -0.020059309867945227,
+            0.024883721794000956,
+            0.015578237413840115,
+            -0.00536523508503184,
+            -0.005290645798641171,
+            0.008242987948511412,
+            0.020469675635821673,
+            0.006668354647245447,
+            0.00017729517736814364,
+            0.02373604785121342,
+            0.029915687122900485,
+            -0.02432635861789771,
+            -0.029540533943782886,
+            0.008151297444985344,
+            -0.005730733861939129,
+            0.011318816579801282,
+            -0.02644855229221747,
+            0.00033932872948409137,
+            0.0035505684154028996,
+            -0.03570104343808711,
+            0.03408366016010711,
+            -0.009199700348630527,
+            0.005857598530788056,
+            -0.009538177736472163,
+            -0.0017768806148210118,
+            0.0027410153099351694,
+            0.03898108970435115,
+            -0.007367623392751247,
+            0.009827254226889936,
+            0.015465895035342481,
+            0.000981604396724216,
+            0.05143083558797524,
+            -0.027350877485564314,
+            -0.02183036962301852,
+            0.01885468378168636,
+            -0.007285239007489837,
+            -0.016846837665812908,
+            -0.020546383007536074,
+            -0.011863943814684079,
+            -0.016293192750336218,
+            0.01976516340568883,
+            -0.02387328798681798,
+            -0.011892336568452852,
+            0.007170651391639504,
+            0.0013725014050721622,
+            -0.009809669068563384,
+            0.0016617183704652932,
+            -0.006307590349798603,
+            0.02037693835615797,
+            0.031753656615094446,
+            -0.024892675105483313,
+            0.03577919637821846,
+            0.03178303188315581,
+            0.017260898395251155,
+            -0.02102289690701718,
+            -0.009661035772073294,
+            -0.00048542689442823136,
+            -0.014694631967051945,
+            -0.0326379186158631,
+            0.025443206465204822,
+            0.017194626456982255,
+            -0.011381421003142453,
+            -0.03680783761649673,
+            0.0017216115811536721,
+            0.022819859205829785,
+            -0.013600989058799166,
+            0.028141657788496834,
+            -0.01869047298692401,
+            -0.01366833755869951,
+            -0.00782959271979894,
+            0.031227914441058244,
+            -0.05252392878125891,
+            0.016957842026348195,
+            0.00020880696574616416,
+            0.001154370575234767,
+            0.010892067708194324,
+            -0.008368819756628046,
+            0.012224564683325483,
+            0.031700389750709,
+            0.018063405869147564,
+            0.0010794115307202266,
+            -0.002783668216641561,
+            0.01418234057248712,
+            0.017773269840982066,
+            -0.015653443355569053,
+            -0.000497740485657824,
+            0.019890882604100452,
+            0.022672054781919238,
+            -0.00521797370595221,
+            -0.016686942830774276,
+            -0.018443538351123584,
+            0.025270057639201107,
+            -0.010354431070738671,
+            0.03484047753830186,
+            0.04594785214094975,
+            0.0036441959528283004,
+            -0.05695476763071743,
+            -0.009412209619190693,
+            0.0010371110229966086,
+            -0.01099343927053564,
+            0.01606863921935641,
+            0.012470660511773616,
+            -0.03237187923799263,
+            0.006388672213167589,
+            0.012946570716567364,
+            0.022152554488829627,
+            -0.02540317621005867,
+            -0.023183252921885226,
+            -0.0030984019855787655,
+            -0.007890467768631811,
+            0.018648760402624687,
+            -0.008653405980661855,
+            0.0006228315253506687,
+            0.009672526897422602,
+            0.0005436494014569835,
+            0.011632326145022935,
+            0.005655942007108228,
+            0.021736827172951,
+            -0.001256608167230203,
+            0.013124413886287987,
+            -0.01986808920858981,
+            -0.0270450339809516,
+            0.006500423802582728,
+            0.04438242119211055,
+            -0.014813399694814117,
+            0.030621709833336064,
+            0.0092171224268983,
+            0.016433384739642534,
+            0.024875072913970443,
+            -0.00891470467241686,
+            -0.014124009014664287,
+            0.00845777541223868,
+            0.009019007848927453,
+            -0.04147358041382793,
+            0.012513030150663913,
+            -0.01710243399313324,
+            -0.033144939719206276,
+            0.01904674924133127,
+            -0.00534122554342279,
+            0.012367320751926314,
+            0.004270315800911569,
+            -0.002886460711688153,
+            -0.025696286516581048,
+            0.014137901708875033,
+            0.028135087538950946,
+            0.021797869775098042,
+            0.0296173052503884,
+            -0.012757078368361444,
+            0.0005460601780756187,
+            -0.008748930817524899,
+            0.005470975782187157,
+            0.04187083128854477,
+            0.0008687182830616019,
+            -0.0008032802520818105,
+            0.01507726552487313,
+            0.024945744517309208,
+            0.0376130540370386,
+            -0.015415560911023553,
+            -0.03407353556457147,
+            0.01564969972264396,
+            -0.010104443707865262,
+            0.018855792875436674,
+            0.005503259932317811,
+            0.0049634097823351596,
+            0.0038600156268000702,
+            0.019312419743287703,
+            -0.006635193789316492,
+            0.018798045806429395,
+            -0.02500983683871332,
+            -0.028785757874583755,
+            0.01829411751138161,
+            -0.005277972443258226,
+            0.03338623299668019,
+            0.016225885095995672,
+            -0.029522280454919586,
+            -0.01712097344863975,
+            0.01553706187924322,
+            -0.008465766956343768,
+            -0.024842700926275538,
+            0.01895228828512998,
+            0.007076862934523977,
+            0.008956105593078965,
+            0.0024373779389833218,
+            -0.01106503696703573,
+            0.01125466547367417,
+            -0.000818223040067914,
+            -0.015322790791698395,
+            0.014157131555941659,
+            -0.017853386731281715,
+            -0.014994640991761465,
+            0.01775991780643474,
+            -0.037498078532902435,
+            0.009570197712419421,
+            -0.007687297543935123,
+            0.009050494195378254,
+            -0.004222041469820445,
+            0.007460383234429904,
+            -0.0010804379186060264,
+            -0.014987857934522812,
+            -0.014903749199810126,
+            0.03228485430019625,
+            0.022096121825475306,
+            -0.000410816936759558,
+            0.006728733918931589,
+            -3.971636461500232e-05,
+            -0.026719848032785824,
+            -0.011253751790407687,
+            0.02245467425830765,
+            -0.005664635254398481,
+            0.005134496095445943,
+            -0.01109590855846966,
+            -0.0149616794609307,
+            0.0019726017255571673,
+            0.032668637270108894,
+            0.06066635618119076,
+            0.012001732730659747,
+            0.024422093585231484,
+            0.0189499396761026,
+            0.02119276033547528,
+            -0.008489818917377516,
+            0.006413901834269739,
+            -0.03097413287789443,
+            0.03509225573986143,
+            0.005392594635384506,
+            0.03108937318306421,
+            0.026664331248813712,
+            -0.04722525921137681,
+            -0.0058539171830594774,
+            0.017087531454837677,
+            0.022673712717885323,
+            0.0159020681868943,
+            0.0008382498574190522,
+            -0.008938174268517353,
+            0.035650940044285984,
+            -0.03315250299348084,
+            -0.023647111263155,
+            0.005059382024092534,
+            -0.01439537467317451,
+            -0.010488359956731476,
+            -0.006785295580877329,
+            0.02776449681317373,
+            -0.005584052419598926,
+            -0.004582019433008779,
+            0.009697064252638535,
+            -0.014400206558190682,
+            0.008165676294736836,
+            0.0007624778531368436,
+            0.008681263897172326,
+            0.039249866981334126,
+            0.0005918063050375113,
+            0.024275699182767528,
+            -0.004138313072843555,
+            0.011079529739650594,
+            -0.0158972266904151,
+            -0.016985058543605,
+            0.02276982363172991,
+            0.00351101120496393,
+            -0.03244126813432563,
+            -0.00768819222024939,
+            0.00928180979884261,
+            -0.0013793404539641824,
+            0.005266058566444064,
+            -0.00503433670609399,
+            0.003747448261046857,
+            0.03286230666764075,
+            0.0022383592721685153,
+            0.008606190020480513,
+            0.013593947677257625,
+            0.00030951915677929204,
+            -0.01490307680740127,
+            0.013254591138113545,
+            -2.1379308136707722e-05,
+            -0.012900073369514622,
+            0.03357519186606058,
+            0.006705902094864628,
+            0.04243347200485736,
+            -0.03302953674711988,
+            0.009031534935013593,
+            0.01822227379789576,
+            -0.008491770335368761,
+            -0.02725412856390648,
+            -0.03630788717786838,
+            0.008259145747740535,
+            -0.019003260334340084,
+            0.015781275447792516,
+            -0.010039277287105153,
+            0.029665124310477384,
+            -0.03563370794521629,
+            0.004714858110268169,
+            0.004053913355055313,
+            0.0009503673086552651,
+            0.02068360255395961,
+            0.018524093113914366,
+            -0.002785698934204024,
+            -0.031443835341186485,
+            -0.02066365433385339,
+            0.006869321907853112,
+            0.02431260052207595,
+            -0.010367048093621498,
+            -0.005612149485859478,
+            0.0006316058139530722,
+            -0.024369562712568606,
+            -0.016845124984022828,
+            0.030359419843284823,
+            -0.040974427176153246,
+            0.005252911449822132,
+            0.04272936053727608,
+            -0.04599092715016919,
+            -0.03802288716943514,
+            -0.002314988728575789,
+            -0.024124014894379466,
+            -0.006360771882090754,
+            -0.002068030981218464,
+            0.016669991113175926,
+            0.0036232707799735537,
+            -0.028763764013482795,
+            0.00943401978066451,
+            -0.005173988601101547,
+            0.003971420424839974,
+            -0.0031669592494977393,
+            -0.02216336363004636,
+            -0.01657772032717298,
+            -0.02531962163186398,
+            -0.035174102848865116,
+            -0.0015903434876311088,
+            -0.005158703150821607,
+            -0.028703075279214722,
+            -0.03941940849540492,
+            -0.021696000925262537,
+            -0.0160221488026189,
+            -0.00843213135542359,
+            0.040641882367348774,
+            0.021319060666253945,
+            0.019336844555345886,
+            0.0025252167889382917,
+            -0.020346900609125973,
+            0.049440028814233025,
+            -0.007642182460707816,
+            0.035628691886108926,
+            -0.0017882888000877158,
+            -0.013461669667950893,
+            0.00455558389906464,
+            0.03825428026120858,
+            -0.014248434654950971,
+            -0.0009902062564027718,
+            0.00027811655483232545,
+            -0.0011866253757890136,
+            0.0008535155950871286,
+            0.015433650396777696,
+            0.001307651096343185,
+            -0.007112199543314309,
+            0.030380673552683906,
+            -0.012803247194895328,
+            0.01637501752192592,
+            0.018350814869962304,
+            -0.0013359719137099667,
+            0.008064546833360553,
+            -0.021430659268086814,
+            -0.00731572118780415,
+            0.0044578576331236325,
+            0.008422210247967642,
+            -0.015096885054883586,
+            -0.009964588397149773,
+            -0.02359494467299634,
+            -0.01174145797396426,
+            0.0023855749166744366,
+            -0.004559202896782688,
+            0.004902138778592298,
+            0.034595817216889944,
+            -0.016068744500427584,
+            -0.035106033294403614,
+            0.009997207702122002,
+            -0.02663088875167486,
+            0.02222306406130687,
+            3.983384942889326e-05,
+            0.006838221769859366,
+            -0.0023434342406307616,
+            0.001765445386886839,
+            -0.00169484278667461,
+            0.002862331185439823,
+            -0.018217975204395134,
+            0.0031859734246840257,
+            -0.0032119988065673565,
+            0.0009052950782312657,
+            -0.0012459653052875638,
+            0.00687339892509406,
+            0.006332799551164718,
+            -0.0008019538898592668,
+            -0.002968451740267742,
+            -8.541086355097327e-06,
+            -0.020561816931299912,
+            -0.021642512405352372,
+            -0.028259049261410905,
+            -0.000989887278779396,
+            -0.012359733017834338,
+            -0.027804394906433664,
+            0.011443385424045703,
+            -0.006872684999639775,
+            0.008327844771213648,
+            -0.008969527138034904,
+            0.006219994886600349,
+            -0.008415106130309062,
+            -0.0004143996206881767,
+            0.013119286511794273,
+            -0.02925209282917943,
+            -0.02096907906398536,
+            -0.03096513357712367,
+            0.01762223841790055,
+            -0.00925741670612824,
+            -0.002656382260603827,
+            0.0025223527960181835,
+            0.005092659166535706,
+            -0.033606542487570154,
+            -0.006653438998497412,
+            0.007313682256676618,
+            0.009547290751154595,
+            -0.03287788897629887,
+            0.02446728061355753,
+            0.017023222076526946,
+            0.01874233066565012,
+            -0.02417566312664911,
+            0.04049827067897005,
+            0.014426879325647042,
+            -0.021740606525480655,
+            0.016712907564723732,
+            0.027385150577780248,
+            0.03177163221499527,
+            -0.03469098547555545,
+            0.004481376248622418,
+            -0.01518200063132508,
+            0.018918680020516044,
+            -0.025284921244460287,
+            0.020625170445627613,
+            -0.016707629602243063,
+            0.021278719363178687,
+            0.03520334001264928,
+            -0.005524369131116028,
+            0.03608133014180745,
+            -0.019234548780647943,
+            0.0059985801736444555,
+            0.004041778461022664,
+            -0.01645503683802124,
+            0.01024694092614158,
+            -0.0042700351372843386,
+            0.0006059159967286885,
+            0.009406090340935747,
+            0.019133229156115282,
+            0.019007219188041775,
+            -0.04442208547053466,
+            0.028588965123434923,
+            0.00804120101594811,
+            0.021472086070876265,
+            0.014177820346835521,
+            -0.02708807970982664,
+            0.004742981711172693,
+            0.002072673190279628,
+            -0.004659713164250794,
+            0.0061167904730676514,
+            -0.013796604518930328,
+            -0.034540857808309226,
+            0.006338792319814448,
+            -0.018918703778187926,
+            0.015036937364271984,
+            -0.009026670935442384,
+            0.0011667525636770179,
+            -0.01152652811400482,
+            -0.011938679695598347,
+            0.0006170592505446954,
+            -0.005495151258143897,
+            -0.0008075736908949407,
+            -0.03171721592827769,
+            -0.04427658539412448,
+            0.02036132271908089,
+            -0.0021221965958423804,
+            -0.04320396743189005,
+            0.004111179168317842,
+            -0.025352428814134187,
+            -0.01865918186335911,
+            -0.011237699157960728,
+            -0.025047923887906474,
+            -0.04219630696740606,
+            -0.011108496740166623,
+            0.007785006066683012,
+            0.016810642610918525,
+            -0.009081815651299525,
+            0.040965574143584864,
+            0.0073944364393447965,
+            0.015313576815177263,
+            -0.017731360025694955,
+            0.026137777790243826,
+            -0.019314392060123,
+            -0.03108486330987073,
+            0.02676829566273746,
+            -6.259161089416372e-05,
+            0.003053748984540415,
+            0.014358356168574803,
+            2.8186924071913628e-05,
+            -0.006181789640967465,
+            0.01947730374003864,
+            -0.028739300217145797,
+            -0.023460503178221397,
+            0.0012460072421725688,
+            -0.008850395570726385,
+            -0.016271242375881602,
+            -0.035896509136022194,
+            0.005227881053956234,
+            0.03324337105109496,
+            0.004970419987430122,
+            0.008273497228454482,
+            0.01814500594735864,
+            -0.02229593571615308,
+            -0.024230675455380678,
+            0.000639112529814716,
+            -0.010266640429960994,
+            -0.010257247303470066,
+            -0.021302483296731352,
+            0.019150757066356872,
+            0.01651375496287087,
+            -0.00390189952706486,
+            -0.006971692709239087,
+            -0.010621003777733253,
+            -0.01918206725910585,
+            -0.02020345113211036,
+            -0.013827322378214754,
+            -0.017980206764311026,
+            -0.034153741666571154,
+            -0.032195001867229935,
+            0.03225205339201753,
+            0.026354817588919596,
+            0.0014076409241492998,
+            -0.007959081015805435,
+            -0.023070419006792155,
+            -0.004483328183581872,
+            0.02879516595345542,
+            0.002549934782150634,
+            -0.0001541849716285838,
+            -0.00830090759167373,
+            0.015341095409025125,
+            0.00835695019244165,
+            0.005107844984733648,
+            -0.06676974053142512,
+            -0.006027540973808174,
+            0.01655283585235268,
+            0.01351155740684989,
+            -0.002359114145814062,
+            0.02591301169926259,
+            0.01108099264396926,
+            -0.015361038608114896,
+            -0.019765743462262668,
+            0.035195761515422364,
+            0.02408910733949848,
+            0.023468767772473746,
+            -0.0034492999385544727,
+            -0.004472311490618405,
+            0.007030833258457338,
+            -0.010684252427867153,
+            -0.013326964647936429,
+            -0.004558513842504658,
+            0.026925146859337397,
+            0.016099443021514896,
+            -0.0064331198280825575,
+            0.010848931851939124,
+            -0.0008320209125061849,
+            -0.019444491995041537,
+            -0.001841330829156697,
+            0.008730275423516445,
+            0.0012974082030866743,
+            -0.014611902493400719,
+            -0.005911807768916134,
+            -0.019130604101278263,
+            0.024719468317569562,
+            0.0209456329723249,
+            0.03381727436368338,
+            -0.013237902591972769,
+            -0.0003695142331921278,
+            0.005410360755058108,
+            0.0002841632253252816,
+            0.0023352645111704958,
+            0.004801123870325242,
+            0.02623649813035263,
+            -0.029010710171522985,
+            -0.020741774066460415,
+            0.0068426980682519505,
+            0.013445270043414486,
+            0.0022744032984779294,
+            0.033840485942696216,
+            0.035040080489222324,
+            0.004166066423508316,
+            -0.010046666684988869,
+            -0.006148765564658713,
+            0.007071740742464309,
+            0.01865897335451031,
+            -0.023267411753900076,
+            -0.0005936657905496225,
+            0.004833030519903764,
+            -0.004229582648167378,
+            0.03620610659071198,
+            -0.009961238880412731,
+            -0.007197670122096852,
+            0.022212791947347127,
+            0.09819205436926473,
+            0.00047695621444943017,
+            0.0025354521225913395,
+            -0.029358210234742176,
+            -0.023614016211567963,
+            -0.0012564198366457442,
+            0.00242680708457405,
+            0.00830167274308547,
+            0.028076260517106013,
+            0.0004852400049405906,
+            0.011579274037419117,
+            0.02773972145320728,
+            -0.017217305534161797,
+            0.006752070301141707,
+            -0.011376051474975524,
+            0.024577071515205502,
+            -0.012694034026935636,
+            -0.0019809719792138673,
+            -0.016512649783050146,
+            0.015907402258634816,
+            0.004556574687833768,
+            0.004907510649166062,
+            0.011347630984408937,
+            0.009824484201337644,
+            -0.02650731900265866,
+            0.01756773113723491,
+            -0.005671986127099093,
+            0.004449611508541289,
+            -0.027841346572384462,
+            -0.020124233970502267,
+            0.0025986391930124847,
+            -0.0014440758308989809,
+            -0.009579444248630085,
+            0.006005274872617889,
+            -0.0024984235450999127,
+            0.03765664613203687,
+            -0.03765496603611941,
+            0.01409350051861866,
+            0.0066453175608372555,
+            -0.0015390841089599882,
+            0.011806653942640485,
+            0.017306914776093522,
+            -0.008206256939290151,
+            0.02589800448884443,
+            -0.00798213351353373,
+            0.009846274212000523,
+            0.015230070803055296,
+            -0.03449531102975745,
+            0.009174739184806312,
+            0.010145762060174144,
+            -0.024165660737765746,
+            0.012146877603301336,
+            -0.013784371016298891,
+            -0.004818759788054113,
+            0.005930446700613062,
+            0.007349837554764034,
+            0.010733731974181452,
+            -0.02295571464086674,
+            0.016176424826629898,
+            -0.010947841926210105,
+            0.0037766665765835445,
+            0.013559846998199008,
+            0.024892849808979323,
+            0.0012426907454129122,
+            0.0005950515136086193,
+            -0.01379469431504576,
+            0.005534724429523084,
+            -0.015667467221535825,
+            -0.022401586867694584,
+            -0.014192041775234079,
+            -0.006366846766730035,
+            0.01029598687444803,
+            -0.003972386708834858,
+            0.016971476088808737,
+            -0.010309648640697812,
+            -0.010645860303528099,
+            0.0037818310327362103,
+            0.01040806357838676,
+            0.008316605962234849,
+            -0.02855914079113881,
+            -0.005676310556150579,
+            0.004961084138163161,
+            0.00029285971503001265,
+            0.009592497954559435,
+            -0.014556073011576487,
+            0.013689526487160009,
+            -0.013290113354287095,
+            -0.0010726537498214612,
+            0.013201860397060236,
+            -0.012443552043930505,
+            -0.004012305744087929,
+            -0.01592905426965635,
+            -0.028487066798154617,
+            0.008462143660968396,
+            -0.01365614159523485,
+            0.001076882584445294,
+            0.0019503240125505646,
+            0.004730085646042709,
+            -0.016919069570642004,
+            -0.007860046623166481,
+            -0.033515448866521046,
+            0.011065416478687504,
+            0.02314589687882036,
+            -0.006435733825418645,
+            -0.030351727858789472,
+            -0.009546333526852862,
+            0.011327489786114963,
+            -0.0071765122743172715,
+            -0.01676736383051139,
+            -0.0029390038145944096,
+            0.007617910650439855,
+            -0.008174620754097493,
+            -0.00508943241317381,
+            -0.0025173969010414098,
+            0.0217328037301181,
+            0.016094975605203832,
+            0.01769642142712967,
+            -0.021400862603034036,
+            -0.00976992221747187,
+            0.026043879119776864,
+            -0.03275550610951898,
+            0.012441465975914529,
+            0.024587446266013183,
+            0.003446527051428522,
+            0.005492228054838714,
+            0.0030377554109693714,
+            0.005459159163772825,
+            -0.006985179226831048,
+            0.019925583798328282,
+            0.010870421035240845,
+            -0.019709062313471467,
+            0.005385443078244229,
+            0.02086085539375208,
+            0.0037248737640193097,
+            0.012892299156840998,
+            -0.017546870090541213,
+            -0.0020341807891543716,
+            0.015594779287017958,
+            0.04816214672670215,
+            -0.0032851797178091685,
+            -0.010450846098750195,
+            0.029513742590677183,
+            0.023526004023637855,
+            0.005373455940309873,
+            0.0012120777808635214,
+            -0.02525630456877349,
+            -0.013075130701774785,
+            0.01962235086017781,
+            0.016944870494393874,
+            0.013484925533584295,
+            -0.0023255656480479004,
+            -0.019236081983457366,
+            0.021520156296467717,
+            -0.01701733319731161,
+            0.012422659010697477,
+            -0.023764941037007197,
+            -0.001490790833263327,
+            -0.0035419008954878724,
+            0.021998949056452454,
+            0.0021819635638621092,
+            0.008891505304514552,
+            0.024354552127458608,
+            -0.006978548952551799,
+            -0.019526615985982355,
+            -0.0051799020400150305,
+            -0.014578953088901798,
+            0.022506635912517146,
+            -0.024117708614271965,
+            -0.007008641019892289,
+            0.011379795968360019,
+            -0.012362093847309038,
+            -0.0102534051092702,
+            0.001164934138876502,
+            0.017686807394319725,
+            0.0009423993435129515,
+            -0.014784508280754078,
+            0.01944445593807992,
+            -0.0011359271135469456,
+            0.010410915597949927,
+            -0.01921419010977189,
+            0.009390638453648025,
+            -0.016682442516992777,
+            -0.007849184048209236,
+            0.010343008873053859,
+            0.0012604521199837046,
+            0.003948300703645506,
+            0.002592490121754231,
+            0.007773437307969315,
+            -0.03355658426684504,
+            0.022695505487314516,
+            -0.010751606023585792,
+            -0.003538523688786008,
+            -0.0014125330167969218,
+            -0.01678692701672328,
+            -0.0006918162525881059,
+            0.005816502852598842,
+            0.010878358093847386,
+            -0.016045147059003256,
+            -0.00873270513444376,
+            -0.04920244395446931,
+            -0.030314009649638693,
+            0.023156333024481057,
+            0.013733657599720879,
+            -0.010971506342695331,
+            0.018852159415605117,
+            -0.023760853159532587,
+            -0.010587249404421732,
+            0.0011379869499042905,
+            0.023319856329056813,
+            -0.03539290744576506,
+            0.01215369703816749,
+            -0.014493682360886549,
+            -0.02039288669534806,
+            0.009070984596830142,
+            0.011445354735283254,
+            0.0007259852045513916,
+            -0.0011514627555568814,
+            -0.029350945123925453,
+            -0.010246845821727087,
+            0.02391068177425878,
+            -0.00861718589055971,
+            -0.004723044986826333,
+            0.024716520302409263,
+            0.0033721801437248873,
+            -0.012524663490691517,
+            -0.017955758068233754,
+            0.021612671655463236,
+            -0.026077203994996058,
+            0.007645944309444457,
+            0.010032183011253845,
+            -0.0173991263874487,
+            0.021649599230588117,
+            -0.007529701797485131,
+            0.020327763440441377,
+            0.004227390938169872,
+            -0.011249233530251522,
+            0.020352283974471443,
+            0.013644443758970054,
+            0.004216958800094986,
+            -0.004081669404784771,
+            0.006869353098014179,
+            0.02212732880780841,
+            -0.011021739614266063,
+            0.02219580717345023,
+            0.028777213295471067,
+            0.00017328191188631918,
+            0.03260179554743322,
+            -0.021287655520443285,
+            -0.03159728983004563,
+            0.021740163175540124,
+            -0.01035146858119169,
+            -0.021564906188712388,
+            0.028914245422221707,
+            -0.013674731836508768,
+            -0.00786476052825481,
+            -0.0166718429993922,
+            0.00533027311342322,
+            0.016680336311088843,
+            -0.005246190451159961,
+            -0.014360934066628332,
+            0.0029628103834216345,
+            0.019809487418628386,
+            0.014587234544007958,
+            0.029780009648869975,
+            -0.010778199815920887,
+            0.035180242185168895,
+            -0.002320576872571577,
+            0.008990157317473909,
+            0.012415465232073404,
+            0.006489435152972723,
+            -0.011251531176923445,
+            0.013501894212117511,
+            -0.003446759011266526,
+            0.008230548711588447,
+            -0.001449023474280409,
+            -0.035111679577202855,
+            -0.013709382199848678,
+            -0.04709294504969336,
+            -0.0023752976043235697,
+            -0.0025098360125255215,
+            0.0030975417740913154,
+            0.09736815869389318,
+            -0.005723049647039619,
+            -0.011988817571623457,
+            0.001625059666070763,
+            0.009611697638805556,
+            -0.0009896494420056814,
+            -0.007606836952841873,
+            0.015973152482488318,
+            -0.01343292635775299,
+            -0.02140928016038583,
+            -0.010532701190806147,
+            -0.01288594454368977,
+            0.012079198190522715,
+            0.015929716343161652,
+            -0.0005947359502228821,
+            -0.015207331663761012,
+            0.006524010655124294,
+            -0.004566385899752684,
+            0.0064717704997400245,
+            0.008647099975176308,
+            -0.010414668624343924,
+            -0.0071698879102925555,
+            0.030899143952119196,
+            -0.025301269549419674,
+            -0.024361748354052457,
+            -0.0010579931222861095,
+            -0.009055796737486008,
+            -0.010335459737012047,
+            -0.002692653633719985,
+            -0.0017679501322462056,
+            -0.010519994985692081,
+            -0.010019865263564572,
+            0.006537076627706128,
+            -0.015472130358273033,
+            0.004812857776601667,
+            -0.0047457011269495634,
+            -0.013296069496526007,
+            0.017054968086538724,
+            0.022563785357424853,
+            -0.02841491200807702,
+            0.0006755428946168761,
+            -0.017422510783648978,
+            0.012941211381006863,
+            0.008866069010589792,
+            0.006049618782357251,
+            -0.022451528209097505,
+            -0.023622906582403243,
+            -0.015019272754405791,
+            -0.01927535750568903,
+            -0.024973490129133732,
+            -0.009073381957684092,
+            0.0067692803507600945,
+            -0.001267339788153158,
+            0.021918281111033645,
+            0.000816538714279555,
+            -0.01812545613159305,
+            0.0061084913570894905,
+            0.000568371220952269,
+            0.018292497018668842,
+            -0.027468677288882186,
+            0.0033806422195878524,
+            -0.02307582511848465,
+            0.033508234091571124,
+            0.0039195050746594975,
+            -0.026680111514794824,
+            -0.021558668725755275,
+            -0.003311830287529699,
+            0.018994577319066473,
+            -0.0025745912019675805,
+            -0.00724049447539589,
+            0.014775207617625293,
+            0.0155334760323838,
+            -0.004693118176196483,
+            0.002015625406296403,
+            0.027968079002437102,
+            0.01077998866180723,
+            0.0313892866933904,
+            -0.015236169297590675,
+            0.004945828495617267,
+            -0.027215016713379062,
+            0.010214419950212732,
+            -0.01503548137805401,
+            0.008859554812646242,
+            -0.01290890682557532,
+            0.03531384210182567,
+            -0.012005613861257432,
+            -0.004250698950192301,
+            0.03253288796441852,
+            -0.029150986393345252,
+            -0.0035020051536906334,
+            -0.009255437891304041,
+            -0.004142204892160768,
+            0.013240580426083011,
+            -0.001198093874819309,
+            0.008485141356876356,
+            -0.01854370462655459,
+            9.726096864073609e-05,
+            -0.003221852304410318,
+            0.005404595442127598,
+            -0.02005458688172967,
+            -0.003314118809778734,
+            0.0073427568419969814,
+            0.004259518237708634,
+            0.002408630356035373,
+            0.016420646142725346,
+            0.006662121128241274,
+            -0.0029357701493526907,
+            0.03424224243607692,
+            0.008171895850772245,
+            0.0152521424877615,
+            -0.0018638264193202728,
+            0.0030234211261443034,
+            0.019287963516760636,
+            0.019912956661192856,
+            -0.00048443399729935116,
+            -0.01058935153883094,
+            0.00195113175754497,
+            -0.001140732137433033,
+            0.011048741602327437,
+            0.024396046054912645,
+            -0.021158025221644738,
+            -0.02419562092713724,
+            0.016377661005279716,
+            -0.015076145951740961,
+            0.01903712233378994,
+            0.010541463112889985,
+            -0.010099351302095792,
+            -0.00031558616108876247,
+            -0.042316016201379245,
+            0.010964989031040692,
+            -0.02068939391415103,
+            -0.04004120028915868,
+            -0.00244383573783919,
+            0.0009542507389088937,
+            -0.029439456243505052,
+            0.02822814855547978,
+            0.0023755032182188156,
+            -0.017280221740813364,
+            0.018223709137876153,
+            -0.005548654680726505,
+            -0.0024040301239229064,
+            0.005397711787238679,
+            -0.01092026090471121,
+            -0.010648485086769927,
+            0.008235250252660972,
+            0.010957715075632535,
+            0.005425683656535705,
+            -0.005698919020608095,
+            -0.011284911899894062,
+            0.009975882651638084,
+            0.012318901214538077,
+            -0.0019922884373083155,
+            0.010050520101815593,
+            0.0031720752428247446,
+            0.007352859750323051,
+            -0.0073483049386141005,
+            0.0046483587496626965,
+            -0.005093220128411125,
+            -0.026080057085901446,
+            0.012863580653903103,
+            0.0052434400313520774,
+            -0.011863360599637747,
+            0.0103209502958491,
+            -0.01048162756158923,
+            0.005787157766911021,
+            0.001318945460232246,
+            -0.0059333230422061395,
+            0.004821320238893602,
+            -0.0038922239931512784,
+            0.011267634571983924,
+            0.013256611037442042,
+            0.004592529900805719,
+            -0.013076954459595086,
+            -0.02177318635989856,
+            0.009001291224623467,
+            0.01959767163356408,
+            0.024161754192119852,
+            -0.030416109293488064,
+            0.003784634588325551,
+            -0.0014915849060829743,
+            -0.0020550564975080885,
+            -0.015632785947151753,
+            0.01587790981601866,
+            0.01716742056401084,
+            0.012721563202528083,
+            0.01893393006967701,
+            -0.004009617703771589,
+            0.010539100665843704,
+            -0.02080068888791658,
+            0.013642451766699067,
+            0.0008609869904066794,
+            -0.013629067231663643,
+            0.0014620655123815298,
+            0.027046304278766087,
+            -0.011056783825898616,
+            0.006206969699250696,
+            0.008097527175511791,
+            0.022111870736907183,
+            -0.012509479100313492,
+            -0.003735868296238069,
+            0.01516254968167132,
+            0.0014878674796313539,
+            0.0035887079417890352,
+            0.016533149527222273,
+            0.04729052276110659,
+            0.00102751186013391,
+            0.0016713773449738153,
+            0.0192177920520243,
+            -0.005563519063161466,
+            -0.01114821419109643,
+            -0.003549999863325982,
+            -0.021807893906469106,
+            0.010998558936481785,
+            0.00431279785115251,
+            -0.02186324212032519,
+            0.01024093366826206,
+            0.0125495473550584,
+            0.005268320135120931,
+            -0.0031679212410026603,
+            0.02574575247827441,
+            -0.019807879421996258,
+            0.014935295894965128,
+            0.03329450628384787,
+            0.00493356778223407,
+            -0.011423949333937285,
+            0.007263264249871384,
+            0.0027354459678742013,
+            -0.004494406041906684,
+            -0.00606745413660583,
+            0.010497391290734088,
+            -0.008191702034391382,
+            0.009870458191510176,
+            -0.009580130479693,
+            0.039202091799313755,
+            -0.009619018331262844,
+            -0.010782138248435458,
+            -0.004558645539551881,
+            -0.006241773108460662,
+            0.003478414182938609,
+            -0.003395165694642073,
+            0.005812225135633631,
+            -0.0069045407353789945,
+            -0.014607146220271209,
+            0.0008522982757210454,
+            0.00020175973815948303,
+            0.014001922919116725,
+            0.0011804904415274088,
+            -0.004871874532755353,
+            -0.007877217154706205,
+            0.015243277056007988,
+            -0.006047906208304912,
+            0.0006531937033317192,
+            -0.0011653523616313422,
+            -0.0013662556399359335,
+            -0.007401050012179761,
+            -0.015166105795085528,
+            0.00974609244984173,
+            0.016253733908242134,
+            -0.013857834499211056,
+            -0.005142936208363415,
+            0.012274716212832219,
+            0.008540229125193836,
+            0.011742606168187876,
+            0.0007451338650642946,
+            -0.009255813682572937,
+            0.008292531177374192,
+            -0.015020523035432158,
+            -0.008781317965985481,
+            0.0026250469418136603,
+            0.002381147382977664,
+            0.005839723981046825,
+            0.0006240641828668715,
+            0.004502463508605272,
+            0.005798912705218846,
+            -0.00497936644919367,
+            0.01112666823857836,
+            0.008198424171006548,
+            -0.002452143482105151,
+            -0.01017569410837814,
+            -0.001127589984272638,
+            -0.015733298167661346,
+            -0.016758315411028353,
+            -0.0009286266498936771,
+            0.015175988113750882,
+            -0.02352862898718829,
+            0.018996352975391644,
+            -0.0014524363611708201,
+            -0.0033802546171857704,
+            -0.021392428456030312,
+            -0.0025923983911570183,
+            0.0034214188668518796,
+            0.01255768230394476,
+            0.01858853630745928,
+            -0.02729543519465384,
+            0.010575894388563116,
+            -0.007204725245442141,
+            0.030394929424014717,
+            -0.018328782140005044,
+            -0.004022941273610227,
+            0.026915618079621992,
+            -0.02200018306828758,
+            0.0008791185377513978,
+            -0.011454606955158956,
+            -0.008103032693853504,
+            0.019287674837688177,
+            0.023747720480736777,
+            0.02038287242167783,
+            0.01364489294869211,
+            0.0007758044919789085,
+            0.004768030222445397,
+            0.03572705648994403,
+            0.010568241270862248,
+            0.014828087092859878,
+            -0.007542489523110974,
+            9.457808640264811e-05,
+            -0.02554033312739386,
+            0.01635899434369978,
+            2.587383534566272e-05,
+            -0.012370727506722627,
+            0.007162422186621379,
+            -0.013370882403931942,
+            -0.011066483600541058,
+            -0.007151077450730025,
+            -0.01721910018182534,
+            0.02152239849400241,
+            -0.012322440280680164,
+            0.004587729883596212,
+            -0.009079747696669277,
+            -0.013272707286450417,
+            -0.012953682633400515,
+            -0.014202214807187545,
+            -0.01313141988034687,
+            -0.026041403779538307,
+            -0.027109731260915763,
+            -0.006573285295895077,
+            -0.023780366346334524,
+            -0.013799072648505526,
+            0.005588622594136253,
+            -0.022617902666316757,
+            0.012262325666428604,
+            -0.012689827812482627,
+            -0.01226264872059864,
+            0.012805734807835008,
+            0.020274844315616616,
+            -0.011765231803328348,
+            -0.028494489703400308,
+            -0.018583843821043854,
+            -0.019970998470704358,
+            0.012576201137505969,
+            0.002266288538944273,
+            0.013165889075444961,
+            -0.003018437854660793,
+            0.003227412667880239,
+            -0.014895831590298957,
+            0.018570624675927183,
+            -0.03442390703780963,
+            -0.03108003806963536,
+            -0.013611306765348361,
+            0.005182777066476028,
+            -0.0015782033203578553,
+            -0.011429046504256945,
+            -0.0018606648572920801,
+            -0.00433805271182878,
+            -0.025109572024446088,
+            -0.010134856632190067,
+            0.011010662054484161,
+            -0.015205629927469414,
+            0.0072933565255675036,
+            0.0009018832564038863,
+            0.013870902834260887,
+            0.009587432944616442,
+            0.013393301893492882,
+            -0.006095063698384775,
+            0.004356485794113208,
+            -0.0026476612747324386,
+            0.0004894024149759108,
+            0.009336366919474478,
+            0.00839591561027103,
+            0.010551161369079399,
+            0.002260707777260338,
+            0.007830059223001696,
+            -0.0060927446216633265,
+            0.018065002687169148,
+            0.015436944875538481,
+            -0.0002520689442496089,
+            0.013323098485251487,
+            0.0003769862591044255,
+            0.00022293304921127018,
+            -0.0029970303106340466,
+            0.007888437018607987,
+            -0.011785957933642163,
+            -0.001311704406355858,
+            0.015138190893483804,
+            -0.01707247730569978,
+            -0.013045187198628258,
+            0.03107034781283638,
+            0.0018583638488268697,
+            -0.02495564203304796,
+            0.031146554085787762,
+            0.004220047610942943,
+            -0.005835199594223787,
+            -0.013205086624848782,
+            0.0014059047822222418,
+            -0.011570624844100298,
+            0.003584232809362274,
+            0.014295032061446826,
+            -0.02231984223610487,
+            0.010521094801839404,
+            0.008446892441796151,
+            -0.009681535919980559,
+            -0.012337659795586223,
+            0.009847073709209602,
+            0.016809328445200358,
+            0.004472485504034416,
+            -0.03539093810210523,
+            -0.011957694102753411,
+            0.02145470732008716,
+            -0.029237428225009913,
+            -0.00810041956259673,
+            0.007452962400105787,
+            0.0065154832040426065,
+            -0.019353109017609053,
+            -0.008580955020412362,
+            0.005725794605897509,
+            -0.004940390374637084,
+            -0.003892210172017758,
+            0.017541623128534682,
+            0.00485047043626186,
+            0.013698341792101025,
+            -0.01689738578307474,
+            0.009771458057400557,
+            -0.022222817828602557,
+            0.004021882074676753,
+            -0.023373700500461992,
+            -0.009873403167702603,
+            0.002364143609079849,
+            0.012454271200139762,
+            0.014328556467632212,
+            0.027839363143659326,
+            0.030056404238348527,
+            0.013995696971422429,
+            0.007833677585250953,
+            -0.016265199386953717,
+            -0.018134327237616175,
+            -0.006409513840075933,
+            -0.004033173350204784,
+            -0.008714798059218596,
+            0.0066373547199279235,
+            0.01858354212009877,
+            0.004499712802472377,
+            0.02609078206670711,
+            -0.006227189596272876,
+            0.009039744622014585,
+            -3.127526911951914e-05,
+            0.0008833930131827993,
+            -0.004226083827636182,
+            -0.01971685546584527,
+            0.00481003211084742,
+            -0.0013721482162825508,
+            0.025874758880591167,
+            0.00940496860916206,
+            0.0024915013399541847,
+            0.01615978314811047,
+            -0.006378952757397484,
+            -0.01174653607178726,
+            -0.010540107733238588,
+            -0.0061926268398085176,
+            -0.021752582127981104,
+            0.029059100503124387,
+            0.021195595568253612,
+            -0.010800765264123782,
+            -0.013693582166598082,
+            -0.00903369526059487,
+            0.02795767990556847,
+            0.004080965688836953,
+            -0.0013050212744006933,
+            0.016313311850308505,
+            0.012084555958999215,
+            0.012329325712491252,
+            0.009794698207689061,
+            -0.00036157148861933123,
+            0.0015364709122308352,
+            0.03722522141440073,
+            -0.007427028510548186,
+            0.0026463292682538495,
+            -0.005239324411651515,
+            0.0025987932868461313,
+            -0.009705817520422487,
+            0.007242122655628283,
+            -0.004012676385296197,
+            -0.0158760051470016,
+            0.015003444174308607,
+            -0.022443331305169994,
+            -0.0016749787204471238,
+            -0.008150635190952799,
+            0.0016696640640947303,
+            -0.02781023337840043,
+            -0.0025731206607371845,
+            0.006695065701630532,
+            -0.015547340828489069,
+            0.011816010501093885,
+            -0.01740673229051557,
+            -0.011553604323149105,
+            0.007914961618894248,
+            -0.012034157288065447,
+            0.0016684130271324446,
+            -0.0008268957621923278,
+            0.004460864332388434,
+            0.013426055079012884,
+            -0.011008808570221245,
+            0.012166046631826273,
+            -0.005816284310138776,
+            -0.007872091214834688,
+            -0.0016055129487028885,
+            -0.0030049472973942043,
+            -0.0025600746761171543,
+            -0.0009966231170649917,
+            0.034355910234323155,
+            -0.004446570896048543,
+            0.003761242147499512,
+            -0.01362639743248326,
+            0.0019833052855181806,
+            -0.036176376394999044,
+            0.028344464817309477,
+            -0.008642835889481802,
+            0.011013502221008498,
+            -0.0017508365480865596,
+            -0.013243706268954915,
+            -0.01456517309747086,
+            0.015247299501245178,
+            0.007602596572394442,
+            -0.005854531057349665,
+            0.013558248087347848,
+            -0.019724373614448298,
+            0.008530500822910953,
+            0.004463694543554244,
+            -0.015561397976987216,
+            0.004668780190328835,
+            -0.007479912658259757,
+            0.012657201279639194,
+            -0.01774730033362932,
+            0.004314594593150765,
+            0.00958571617187387,
+            -0.01892876084898881,
+            0.009901041800918215,
+            -0.028675879564512306,
+            -0.004631920242208613,
+            -0.012323455524366223,
+            0.0053804608827285865,
+            -0.000660864620708182,
+            -0.02389088252276507,
+            -0.028944280660230762,
+            -0.0002099667627277945,
+            0.00458533068247518,
+            0.0018878351997997833,
+            0.011621407953503755,
+            0.005970807015151215,
+            0.008542958629199003,
+            0.01023896054756221,
+            -0.00205207247588841,
+            0.009402752124876478,
+            0.04169706217414899,
+            -0.022899912321317197,
+            -0.011574970753549647,
+            0.005321872977369877,
+            0.011499271505460519,
+            -0.02647653483798284,
+            -0.011039791758347888,
+            0.0033424345263339144,
+            -0.005219877979550397,
+            0.03609469691834737,
+            -0.02448746405881972,
+            0.019020618652020673,
+            -0.011392297116547224,
+            0.007625475937968714,
+            -0.021465240076078376,
+            -0.003921088264697311,
+            -0.014265745991522477,
+            -0.00011221615798127438,
+            -0.0031478236827989703,
+            -0.0076053733863387755,
+            0.007485039021296698,
+            -0.001907678661303243,
+            0.0020348246022161925,
+            0.006291436821831794,
+            -0.02173305455485444,
+            -0.0019125499969493854,
+            -0.016432460924913166,
+            0.030367263349759286,
+            -0.016242095671439496,
+            0.005072805486919414,
+            0.0053258139728101506,
+            -0.002810639419737258,
+            0.012368771814660942,
+            0.004165967191536186,
+            0.00513754440091394,
+            -0.004199767241827414,
+            -0.005142642360546972,
+            -0.0053886448467479215,
+            0.017868874549760376,
+            -0.013280048598849825,
+            0.035024701466710065,
+            -0.0100959186045929,
+            -0.007094001877261568,
+            0.0019733042325232155,
+            0.010380430710282744,
+            4.2084202709296443e-05,
+            0.01053565670076933,
+            0.0034270489770133685,
+            -0.0006311078385359693,
+            0.0019774779328212756,
+            -0.017162448473652843,
+            0.003089758682621882,
+            0.01682656561693936,
+            0.005463677441664502,
+            0.004469261499451007,
+            -0.013332130891003062,
+            -0.012050233723282738,
+            0.004538268704442613,
+            -0.007533940408255447,
+            0.0003675737140813486,
+            -0.018816953644773076,
+            0.013972696748575243,
+            -0.0011096683686631858,
+            0.0002715683052236706,
+            0.00524678369365015,
+            0.023056514997908218,
+            0.01813167860994747,
+            -0.002553352602551003,
+            0.03155164978556312,
+            0.0174738879054593,
+            0.0029268039949260673,
+            0.010637295711777793,
+            -0.033222781195984666,
+            -0.013301873512972214,
+            -0.0013721703891642354,
+            0.02238421933149517,
+            -0.01530189703335162,
+            -0.014059029663525449,
+            -0.0037393534403749947,
+            0.0010394601933309767,
+            -0.005177359933828505,
+            0.0042489306011644495,
+            0.011039889511148341,
+            -0.014697595240068863,
+            0.007597149702441915,
+            0.0026780496822907634,
+            0.02092734730183164,
+            0.0004688144432243786,
+            0.017181410417231698,
+            -0.0005083970646221274,
+            0.00749639055342427,
+            0.007319420826337891,
+            0.017541463125355265,
+            -0.01002811389296805,
+            0.023582113845081534,
+            -0.008823933397729605,
+            0.0019889258902730535,
+            -0.01257106158422629,
+            -0.028159594067544737,
+            -0.012624750220149823,
+            0.015247564545279736,
+            0.005765911899842925,
+            0.006478299226263732,
+            -0.016537090297144265,
+            0.006276997440048981,
+            0.015647122171767974,
+            -0.002251683151059454,
+            0.010295358869874306,
+            0.009730378299068289,
+            0.00012816405890877098,
+            0.010181120688239766,
+            -0.00801256937783534,
+            -0.00734257767723293,
+            -0.004497520124843868,
+            0.023982754195729983,
+            -0.005443550092842393,
+            0.022434177122972028,
+            0.012647958478747348,
+            0.00616791857573897,
+            -0.010853355357954727,
+            -0.003340111254357573,
+            0.023389634440057473,
+            0.011863246790550376,
+            0.00029642699874398923,
+            -0.009846034060495678,
+            0.0019458876579524617,
+            0.0054010373765486005,
+            0.012983913038716568,
+            -0.02049427270240938,
+            -0.004344571352918878,
+            0.008461947538684664,
+            -0.008472919299762712,
+            0.004318309005722874,
+            -0.030156444522730416,
+            -0.005812303237222271,
+            0.021354352236942434,
+            0.008392281015122372,
+            -0.010232039722494646,
+            -0.004395682831498155,
+            -0.0006527587802367882,
+            0.0040601850269714945,
+            -0.0024323167053212176,
+            -0.0023729138545912176,
+            -0.008541233856441836,
+            -0.006470933419694742,
+            0.014035696617797647,
+            -0.0052691760528871924,
+            0.008597459268477037,
+            0.00012084859681905111,
+            -0.0017255958688825231,
+            -0.007323529232751411,
+            0.03382792341957058,
+            0.008087732655845718,
+            0.005980666974986827,
+            0.0013613625176327891,
+            0.0042244330945601215,
+            0.000616791811672998,
+            -0.0028374856646453707,
+            0.018075360591135542,
+            0.0156738939725117,
+            0.005054696911814988,
+            0.009038525980341447,
+            0.014672270369917403,
+            -0.019700405219230957,
+            0.012625337070556061,
+            0.012184185186281482,
+            0.019382321278525114,
+            0.0007219498737851579,
+            0.002564970959687783,
+            -0.010601873823737926,
+            -0.0029963537053764736,
+            -0.002070567089566093,
+            0.029563028293936092,
+            0.0005903976560069006,
+            -0.010509758568838935,
+            0.010992421945232951,
+            0.020603868018231348,
+            -0.02101157589451638,
+            -0.001971974816402967,
+            0.005963911670920596,
+            -0.010704207628592966,
+            -0.014466113540150018,
+            -0.021269176952661614,
+            -0.013331247081394942,
+            0.0071597345125287174,
+            -0.008162102923896894,
+            0.02367549407145414,
+            -0.014256771777502395,
+            -0.03960796112042958,
+            -0.014803626428839152,
+            -0.00864320347698291,
+            -0.027531337207277483,
+            -0.007305441146142331,
+            0.00965056093889974,
+            0.016698589600806273,
+            -0.003330867073501157,
+            -0.025198703196464594,
+            0.030483790406960656,
+            0.008872821868559751,
+            0.023708422767352917,
+            -0.021464374508407488,
+            -0.0018617153194555344,
+            -0.00886376784514331,
+            0.014505918224188001,
+            -0.017588761818663975,
+            -0.011756112651591795,
+            0.020014632992762026,
+            0.014356419813769714,
+            0.007246557500044434,
+            -0.004355132555323843,
+            0.0067299398462013936,
+            0.007649499235136303,
+            -0.012515181210363267,
+            0.012239792251154642,
+            -0.013499277313916029,
+            -0.024858295126228433,
+            0.014430561986516884,
+            0.002229494935179828,
+            0.009563004428064055,
+            -0.0015984171085659323,
+            -0.016197659608768088,
+            0.010078159490026602,
+            0.00808729175515505,
+            -0.017508840440512225,
+            -0.004793934415922065,
+            0.007133527615305415,
+            -0.020818869260580315,
+            0.031234686259160864,
+            0.011872765152279094,
+            -0.010692949770503467,
+            -0.001989039629661815,
+            0.004431079028880923,
+            0.016511544608975148,
+            0.007893700385963683,
+            -0.01997314388229276,
+            -0.003708006454566117,
+            0.01530128306203401,
+            0.00822588305477249,
+            0.004861703074728471,
+            -0.010836328783594584,
+            0.03434874763691911,
+            0.006082873924472328,
+            4.192481378506604e-05,
+            -0.0052997095110967115,
+            -0.0076108924981838355,
+            0.004196112457289409,
+            0.018905391806961686,
+            0.013330078288599763,
+            -0.004200590783114919,
+            -0.006630107527854746,
+            0.007778448128358027,
+            -0.01706991160178345,
+            0.007329056462125937,
+            0.009390327499494826,
+            -0.012870674959022891,
+            -0.0017571189976160295,
+            0.016462608870101713,
+            0.00209846661139665,
+            0.004942714850185376,
+            -0.011305050739130339,
+            0.004252776146827917,
+            0.015841821944109317,
+            0.018727567181082646,
+            -0.008399642221255733,
+            0.021448055799259867,
+            -0.020972459569228434,
+            0.027243639465535856,
+            0.028499400850980086,
+            0.016573144403275928,
+            -0.01847096443459798,
+            -0.0022409373140116146,
+            -0.0041737366139895414,
+            -0.014761716560267328,
+            0.0034546020161337894,
+            0.005125870074458691,
+            -0.01174186608185822,
+            -0.009577921232380895,
+            -0.024331993794416026,
+            0.011075959943215117,
+            0.007918147902741738,
+            -0.0015045185909419564,
+            -0.011791155919533844,
+            0.016019077577552835,
+            0.010141868950981925,
+            -0.010938402325619172,
+            -0.005635351352447256,
+            -0.019967361901734836,
+            0.009750427567651833,
+            0.009128294017175602,
+            -0.00300556760891084,
+            -0.011496103417643132,
+            0.012977890524465716,
+            0.0014080059843484115,
+            -0.01292380658817521,
+            0.010841954636959441,
+            -0.03142011651894961,
+            -0.005510537354374977,
+            -0.011027026734432255,
+            -0.003620482919145167,
+            -0.014415606143566519,
+            0.003608572766363589,
+            0.0005034611321129689,
+            -0.011470488351158329,
+            -0.0035005774696278,
+            -0.016207453351921627,
+            0.004676327282262429,
+            0.015021513761697039,
+            0.001679614645967629,
+            -0.006917375094634678,
+            0.021994274901326064,
+            0.018518524483306403,
+            -0.021903538417919984,
+            -0.020116013379669034,
+            0.004708084983787958,
+            -0.03261690161800251,
+            0.015019348187107588,
+            0.00966124471577622,
+            0.011676937880751252,
+            -0.028768229226408037,
+            0.007114534979415504,
+            0.00900420165933673,
+            0.016398927481511824,
+            0.0050879301160439245,
+            0.009453255486450082,
+            0.01222758288108498,
+            -0.0104657614414275,
+            -0.003164502778525433,
+            -5.410382753651127e-06,
+            -0.005618225986727351,
+            0.015607620507415484,
+            0.0008527697935056332,
+            0.03603666488542239,
+            -0.005277207843139309,
+            -0.0009799879252482288,
+            -0.011735731235560851,
+            0.00812046801749312,
+            -0.011713349784345638,
+            0.006334172347602515,
+            -0.0024643499727789167,
+            0.0001240009282616789,
+            0.002965067818077541,
+            0.009442572075213767,
+            -0.01086095783353607,
+            -0.007595379345224076,
+            0.007444897256325813,
+            -0.01482814340478338,
+            -0.00019984717819268013,
+            -0.009339343730980451,
+            0.0019680804106275156,
+            -0.02241031895876346,
+            0.006629838013824249,
+            0.040935644507527846,
+            0.02140216465404127,
+            -0.005481040712269146,
+            -0.012431659209559418,
+            -0.004680230194268222,
+            -0.007229300034633229,
+            -0.006482401275991039,
+            -0.004794698160419071,
+            -0.022762405726589375,
+            -0.006928553505112732,
+            0.004320321244630314,
+            0.013300138724212552,
+            0.009741018117644266,
+            -0.008842135469729511,
+            0.004531760574809748,
+            -0.02662650543210291,
+            -0.016578081411548973,
+            -0.0060021425889846495,
+            -0.015840130031005147,
+            0.006125301421471048,
+            0.027521841635711787,
+            -0.0063432924325983525,
+            0.01789356321527744,
+            0.005142029893259461,
+            0.013034833065163448,
+            -0.005338869351330258,
+            0.00035250243782976347,
+            -0.006010938630154189,
+            0.0032958656042048292,
+            0.003074861351445954,
+            -0.0015947217852680714,
+            0.004151590621941346,
+            -0.007139285822575166,
+            -0.0026091937868967284,
+            0.0011629380311555433,
+            0.0117841884590861,
+            0.006877419488576261,
+            -0.00016252653621991807,
+            -0.01761285866765538,
+            0.022221104524808997,
+            -0.018933508559881868,
+            -0.008840161102111973,
+            0.011092684029552064,
+            -0.009660487856185668,
+            0.00702597023411946,
+            0.020387386977378873,
+            -0.006854213562767843,
+            -0.009583981919303215,
+            0.013028181087649146,
+            -0.0036507672754997286,
+            0.004464368543493108,
+            0.009867428978560914,
+            0.006036303973414791,
+            0.002482944923100156,
+            0.013244650094892797,
+            0.004530294330688855,
+            -0.0032625482538717137,
+            -0.020238533752799943,
+            -0.005186498720507101,
+            0.019015752262933585,
+            -0.02252458872825473,
+            -0.0027641215988942563,
+            0.0044506144202026456,
+            -0.0030317840657635457,
+            -0.01733962766050954,
+            0.004205861489436576,
+            -0.018770275720867473,
+            -0.0033883519277721433,
+            -0.008244668812709337,
+            0.009948090119054079,
+            -0.023700732627507225,
+            0.009109315916173436,
+            -0.011739192106106307,
+            0.02356509175177699,
+            -0.003925851077921686,
+            0.024734433836185024,
+            0.0015142645677087268,
+            -0.013482933030628261,
+            0.0055323884713736554,
+            -0.0008930661843169134,
+            -0.004477731568313757,
+            0.029338490404300612,
+            -0.005418258701133154,
+            0.002249996261871466,
+            0.013491124734151694,
+            -0.021332541202417246,
+            0.021773687217263173,
+            0.0046662696076723836,
+            0.01628697326539432,
+            -0.00986238951830334,
+            -0.013799209313714441,
+            0.025178874553620743,
+            0.03324555035648896,
+            0.008398796308070151,
+            0.011261509865513953,
+            -0.01226890254177634,
+            0.018966175575083295,
+            -0.0038328997984309597,
+            0.004344837325292706,
+            0.00878668880458693,
+            -0.007200842513742962,
+            -0.011500562922303248,
+            -0.004533014670236755,
+            0.027639894892383943,
+            -0.021838848944317424,
+            -0.008920629621611768,
+            -0.005739181836252042,
+            -0.01593272996094732,
+            -0.038167534267554606,
+            -0.004371701247332816,
+            0.019604405563120094,
+            -0.005974358141737469,
+            -0.0018341151594086288,
+            0.014334266690716505,
+            0.00402620072221496,
+            -0.002852569009185304,
+            -0.008195218707838913,
+            -0.0027944153624309856,
+            -0.0059390492900516225,
+            0.016651206007022145,
+            -0.004961864528562493,
+            -0.007929830585906699,
+            0.01145431100211762,
+            -0.005731957927601062,
+            0.012130694936714224,
+            0.0004848418056781251,
+            8.392048424584131e-05,
+            -0.0015174120840537315,
+            0.00822932723013239,
+            0.018620869354789082,
+            -0.0055803454666417465,
+            0.025682778991091174,
+            0.006956936033581638,
+            -0.003495709501259274,
+            -0.005489431577052735,
+            -0.0044154293119170445,
+            -0.024561016942073285,
+            -0.0020864636079347357,
+            0.006593060072668424,
+            0.024333813203596122,
+            -0.02515989187381735,
+            -0.004953266000463339,
+            0.020957806164031402,
+            -0.0012192724005613,
+            -0.011978722479596745,
+            0.014690559986694138,
+            0.0041346025614047875,
+            0.009881584250499358,
+            0.012176393203318002,
+            0.015764530703333238,
+            -0.004923548157246672,
+            -0.0015192775000207833,
+            0.012766482928380375,
+            0.0005745973040874939,
+            0.00997809510696241,
+            0.016495444272181515,
+            0.010467514633877887,
+            -0.021108281220544853,
+            0.0015078843789125156,
+            0.00522593579521476,
+            -0.017481430635971396,
+            0.0034362821987740755,
+            0.005397819166037461,
+            0.011626527721895694,
+            0.004048632382026344,
+            0.013207335578216437,
+            -0.015768003434275563,
+            -0.005916168437120099,
+            0.008153798523776654,
+            0.004319451296835019,
+            -0.02599474886669325,
+            -0.01485009085106089,
+            -0.0006781964116146299,
+            -0.004444144921332929,
+            0.013556265879295017,
+            0.01285768162554067,
+            0.02033626001285354,
+            0.024637349990579514,
+            -0.013833060730799217,
+            -0.00842819657792762,
+            0.009436072455719896,
+            0.015560357465711765,
+            0.005798181411384387,
+            0.007874051721372273,
+            0.009297440071118126,
+            0.0077320823375042,
+            -0.004073209576661068,
+            -0.002973928843567532,
+            0.020205003097225747,
+            -0.019725382393637726,
+            0.0006746659838642147,
+            -0.018417443693472958,
+            0.007530796896254817,
+            0.01140696723506382,
+            0.01129515042745398,
+            0.002816883643872417,
+            -0.005274757308627299,
+            0.005716917047812153,
+            -0.06254973656832136,
+            -0.009191453047502128,
+            0.003384751551032943,
+            0.027256466708746898,
+            -0.010885336140129142,
+            -0.003415707910562931,
+            0.007309577090125959,
+            0.005511970708769456,
+            0.0005572239548625122,
+            0.003186427278034852,
+            0.019698716902282903,
+            -0.02052581021967601,
+            -0.010793968505805883,
+            0.004219269922138548,
+            0.0033111837923746746,
+            -0.007269521534951044,
+            0.004805737092302134,
+            0.014143978208501908,
+            0.01986656063456352,
+            -0.0024048854610842748,
+            0.01629335842915328,
+            -0.0015498603704589015,
+            -0.014496994444493684,
+            -0.002671064325909516,
+            0.0037875833047555925,
+            -0.022466433279066646,
+            -0.005980897597195858,
+            0.0043262697442044435,
+            -0.011226121648007667,
+            0.006978495397818936,
+            0.008452177557316863,
+            -0.003018545465557218,
+            -0.0008303307644324117,
+            0.017796789945838845,
+            0.006591364879905198,
+            -0.013464486184025026,
+            0.0028668426952326957,
+            -0.023377605222213174,
+            0.004325185794796361,
+            0.001416245546006914,
+            0.007809449561523174,
+            -0.008431335139468173,
+            -0.014774499666285603,
+            -0.005933810712528226,
+            0.017746847020610673,
+            -0.01965188730040385,
+            -0.002406875192894708,
+            0.01683009662984443,
+            -0.0002518220110313279,
+            0.005456812494894182,
+            -0.02011241622618092,
+            -0.0009782974741561381,
+            -0.02933296756078644,
+            -0.01171823509606186,
+            0.001162345317872531,
+            0.00892221099991475,
+            -0.006551308022163148,
+            -0.03934356275100932,
+            0.00427623835776202,
+            -0.013111664387776261,
+            -0.008752857644809231,
+            -0.010214546496134218,
+            -0.00518087722852466,
+            0.015240297918667867,
+            -0.012498100033235827,
+            0.01086490299272505,
+            -0.00917777212602538,
+            0.005895055740265625,
+            0.0090554651297836,
+            0.014212841798768113,
+            -0.005544047915369086,
+            -0.0033589999440643385,
+            -0.007924830667307956,
+            0.0022291797354234005,
+            -0.0032976226056924502,
+            0.009615674261248298,
+            -0.0051932155872883,
+            -0.0065636389655770455,
+            0.02426106498596339,
+            0.011445591067574932,
+            0.013312596099532533,
+            0.0055118624272110965,
+            -0.007220620072349783,
+            -0.01916551187613823,
+            -0.003977616240384473,
+            -0.013204506708246246,
+            -0.006315966975649096,
+            0.01434099854765068,
+            0.012588249778961223,
+            0.006357972880218866,
+            0.0007747002406996095,
+            -0.004794439624135275,
+            0.006975593520375694,
+            0.0262335496080837,
+            -0.016481909971520166,
+            -0.0013084662210086592,
+            -0.01369313900978409,
+            0.01921748474212319,
+            0.03333761672336437,
+            -0.0008918419419301244,
+            -0.007699525186523558,
+            0.013757216065348164,
+            0.00819045303047094,
+            -0.007411380297164911,
+            0.02453611080891859,
+            -0.025045462880463647,
+            -0.015989319900023054,
+            -0.011650886878708265,
+            -0.011407253985176972,
+            -0.008386367959227559,
+            0.016331961716602287,
+            -0.00141225438918211,
+            -0.0018505754298268654,
+            0.009557291617446645,
+            0.03357924917013024,
+            0.006916860901290756,
+            0.007030308953738567,
+            -0.002739345982346728,
+            -0.009376522438071562,
+            0.011482637968411055,
+            -0.007259054391879614,
+            -0.04075661435928125,
+            0.004250195164583401,
+            -0.005160505473755424,
+            0.02405623169060588,
+            -0.02197586561375289,
+            -0.00619175761282745,
+            -0.006736129049831973,
+            -0.028200015908139763,
+            -0.005867722104196303,
+            0.02805071895018145,
+            -0.015402198567775478,
+            0.014773294766214671,
+            -0.009502185346558072,
+            -0.0013419858580438616,
+            0.004443749010136947,
+            0.013008783570355656,
+            -0.02289662952366337,
+            0.0013834888740540108,
+            -0.00048549197205067054,
+            0.027969842651517314,
+            0.010437729090975158,
+            -0.0047705656249623744,
+            0.014202218020412936,
+            0.008240861154198411,
+            -0.018637315941614406,
+            0.007314019317343154,
+            -0.005747966765101792,
+            0.014833896739762147,
+            -0.01314847940891154,
+            0.007762100494499689,
+            0.00705026393499352,
+            -0.01278023669088385,
+            0.03016342404000438,
+            -0.015204129985039077,
+            0.007776946006562982,
+            0.009278778571285689,
+            0.009656723862179986,
+            0.01257686993982689,
+            0.004362439504889069,
+            -0.02898418193036562,
+            0.0002144431781454467,
+            -0.03828100004144633,
+            -0.00833748467467171,
+            -0.002611937107022567,
+            -0.002145329666219461,
+            0.007050630951707937,
+            -0.009383645162261629,
+            -0.033976442057557676,
+            0.0006429978367640709,
+            0.01603027598131823,
+            0.017501067413108286,
+            0.0002196997337451293,
+            0.013075379412332973,
+            -0.006197537539554383,
+            -0.021148120787551232,
+            -0.016986448660412497,
+            -0.010207705210051413,
+            -0.02122194744719993,
+            -0.020550470201191853,
+            -0.00908564461857315,
+            0.012090079127547193,
+            -0.009927390005879807,
+            0.013888458967497627,
+            0.029513250508042935,
+            -0.024529175134769883,
+            0.0011144447737101862,
+            0.021977677894839922,
+            -0.00198308986575084,
+            0.014108189311291326,
+            0.001696775006906642,
+            0.014122064251288995,
+            -0.0004166140484018266,
+            -0.010556016685519523,
+            -0.009729362492661684,
+            0.017487929296614076,
+            -0.0026043257634797124,
+            0.016715798948897194,
+            0.012183487338360088,
+            0.003771721868999568,
+            -0.0030887189945795796,
+            0.010667776697461199,
+            -0.026471157050402344,
+            0.014249996582064706,
+            -0.006690961709923224,
+            -0.0181382691440242,
+            -0.008729104191096461,
+            0.003882731567105777,
+            -0.005609384377642956,
+            -0.007794812949125115,
+            0.012967089730551055,
+            -0.00944374111323783,
+            0.02088525981049842,
+            0.0023400121055514958,
+            -0.012196925240774426,
+            0.003648688547842322,
+            -0.03322041209304968,
+            0.0015191765655963997,
+            -0.018402382776013956,
+            -0.024659631472836927,
+            0.03840173429666435,
+            0.011991452465253198,
+            0.006639702644606365,
+            -0.01675838115087676,
+            0.007953102302963526,
+            0.0004257479565099406,
+            0.017484338819046517,
+            0.011068864013925452,
+            0.006466915839538296,
+            0.008364454017976605,
+            -0.023316137291380285,
+            -0.010898421318668592,
+            0.016204569093730564,
+            0.012099473359791148,
+            -0.01017229369980751,
+            0.019431669546223253,
+            0.017955119269367787,
+            0.007300372071724306,
+            -0.006898619924677706,
+            0.01647927363640854,
+            0.021012103522831126,
+            0.010291016618557653,
+            -0.00047955552103474145,
+            0.0008684297441778696,
+            -0.010534247928997246,
+            0.005037930235889528,
+            0.035417626740307205,
+            0.014712595692093886,
+            0.010650306363963653,
+            -0.03626584475133921,
+            0.021764456453493054,
+            -0.011961521673661574,
+            0.009092981192874177,
+            0.006089104887183293,
+            0.017982801101846038,
+            0.006916969397519662,
+            0.014196254958707752,
+            0.011745069030657836,
+            -0.006689305450954901,
+            0.025205280281804708,
+            -0.001601414806100461,
+            -0.011992525142592405,
+            0.003386930324498371,
+            -0.01859193251438437,
+            0.005303261319773131,
+            -0.0041710536965100615,
+            -0.004459393120257307,
+            0.0032461828144552565,
+            -0.0018650956576455287,
+            0.015678225251799887,
+            0.003894142628084471,
+            -0.02355212701026675,
+            -0.0014873755115206368,
+            -0.024284295126348117,
+            0.0027260592623277375,
+            -0.022185571998195344,
+            0.00022883605560765203,
+            -0.005557568353382136,
+            -0.009749382813201403,
+            -0.025385857431333838,
+            -0.009853073898348951,
+            -0.0035843255959309307,
+            0.007661137888845632,
+            -0.010521037433807653,
+            -0.004246698084422667,
+            -0.0019920499321964815,
+            -0.019465113363548715,
+            0.009014255846983305,
+            0.027122533756959888,
+            0.016848221859975445,
+            0.027170883580103264,
+            -0.00019554112591107156,
+            0.013614675849485926,
+            -0.002066822813103284,
+            -0.01603063005944364,
+            0.008824663429777616,
+            0.0075653522381218495,
+            -0.00531780740880296,
+            0.014643348459785112,
+            0.0072634484937344704,
+            -0.012800149015319017,
+            0.009859503660384916,
+            -0.007161712176469609,
+            -0.006437857349742842,
+            -0.009764856421740622,
+            -0.02058868349990357,
+            -0.014866137971603676,
+            0.0012450207572602557,
+            -0.002123265406747006,
+            0.006000473862248991,
+            -0.004054016790850226,
+            0.009584879012937315,
+            -0.00608816807208866,
+            -0.00925801057439376,
+            -0.013962683072792106,
+            -0.008924356118158748,
+            -0.004210064145949253,
+            -0.004381709838058668,
+            -0.011732699976353612,
+            -0.005479523097469261,
+            0.0010680258077037758,
+            -0.0019401640391035755,
+            -0.01397652201595301,
+            0.011650839243603377,
+            0.0058731838291845645,
+            -0.002462743134993977,
+            0.02219008688925881,
+            0.01097316136828334,
+            0.0018083001685932406,
+            0.01366095798812196,
+            0.0003942601242216818,
+            -0.0064063532958756175,
+            -0.03343312148366622,
+            -0.0005083355836126291,
+            -0.02815586367527057,
+            0.025770078162165123,
+            -0.0069114949102666145,
+            -0.010322087105114614,
+            -0.002519047562374146,
+            -0.006063669644021698,
+            0.005873609611264377,
+            -0.002722940652283699,
+            0.0021229525800901114,
+            -0.023673778230768945,
+            0.0028629715379660113,
+            -0.003953653437464966,
+            -0.0036366957271968823,
+            0.007469664864329332,
+            0.0346520690389124,
+            -0.004054282838167587,
+            -0.0034765253405897676,
+            0.005159466707114532,
+            0.011642913651743502,
+            0.027845571220425726,
+            -0.004445196354520585,
+            -0.0032038508043339413,
+            -0.02233838584152591,
+            0.0054112576531374175,
+            -0.00019544838850471866,
+            -0.004733192299951088,
+            0.016664217348411003,
+            -0.016719175418727594,
+            -0.01631658832047864,
+            -0.00545969677717507,
+            -0.014570848053102338,
+            -0.00022620258237138677,
+            0.014454101124666536,
+            0.011056312437637014,
+            -0.004035526363810078,
+            -0.012459378363360557,
+            -0.0022595493558020706,
+            -0.030370952502579162,
+            0.012766010284884204,
+            0.1141689619126009,
+            -0.00766197047210299,
+            -0.00896655658602649,
+            -0.009481112393590025,
+            0.001435269335646347,
+            0.001338755472877713,
+            -0.011739143028767118,
+            0.07166096084053437,
+            -0.0077855767070652395,
+            0.0002139580430342728,
+            -0.011312198831181402,
+            0.001844005266872301,
+            -0.001825412693163496,
+            0.01371932421024605,
+            -0.005025814503687022,
+            -0.018213360991353643,
+            -0.0033612481616769416,
+            -0.02941403830641238,
+            -0.0041137865872933515,
+            -0.005683630256743215,
+            -0.004288198570156578,
+            -0.014982999363445122,
+            0.0067149665791261335,
+            -0.019661973497372313,
+            -0.012688556129040908,
+            0.001124391650175233,
+            -0.0033623453063430534,
+            0.00996944383866679,
+            -0.020496336814603385,
+            -0.0044354321069609536,
+            0.0019875840782527733,
+            0.009706675240563883,
+            0.0011600364428219214,
+            -0.012755178230821602,
+            -0.0026514442584739412,
+            0.004056350166031111,
+            0.01909951327913368,
+            0.014360281326789544,
+            0.003732344102045803,
+            0.008623457695718206,
+            0.014461868085333023,
+            -0.001692149528228451,
+            0.01948035376572433,
+            0.01172550239477637,
+            0.0031966531144076173,
+            0.0004779413449255626,
+            0.0030912562273611517,
+            -0.004597633235071031,
+            0.01582547400214653,
+            -0.014811177637844808,
+            -0.02411202772171917,
+            -0.0071272238837422185,
+            0.018015541137524113,
+            -0.013081287852028074,
+            1.8837689271256874e-05,
+            0.004062803284646432,
+            0.027569726868200254,
+            0.0010564850664255057,
+            -0.01179755720247972,
+            0.0028876134800690334,
+            -0.01638215925131053,
+            0.022944183106912663,
+            0.0026043713064243836,
+            0.03279173899194346,
+            -0.005103903273968361,
+            0.00578213669463383,
+            -0.020744194169889544,
+            -0.007588793365216579,
+            -0.00819862611489168,
+            0.0017401442433921013,
+            0.010993500174794084,
+            0.005411238350013134,
+            0.01620925462604661,
+            -0.00641615069699833,
+            -0.020751730783437995,
+            0.014776878596707538,
+            -0.009552193304486048,
+            -0.03393319554458277,
+            0.008881987295131347,
+            -0.029102297123590998,
+            -0.0045267103602711,
+            0.00368971742585232,
+            -0.014269341759285646,
+            -0.016264809810970175,
+            0.012837302095591106,
+            -0.0011890334180728564,
+            -0.008937992768323412,
+            0.0014444613725248846,
+            -0.017110187133760626,
+            -0.02385668037818897,
+            0.006017405364919834,
+            0.00676670672597817,
+            -0.0077994500447595,
+            0.018312452858615755,
+            -0.027015660037024585,
+            0.006123912473734307,
+            -0.012973676426738582,
+            0.009041974762537394,
+            -0.01580183887345168,
+            0.01163516332546347,
+            0.010887662470499361,
+            -0.001503467664709027,
+            -0.0076234124970223326,
+            0.01231202541338883,
+            0.004852720134894811,
+            0.033760365210093034,
+            0.028658326381800395,
+            0.000995999783627898,
+            -0.0025605367058226737,
+            -0.0020356794893366384,
+            0.011667525096213252,
+            -0.014049700287539817,
+            0.011218246246280464,
+            0.02658113044439717,
+            0.017568631325838108,
+            -0.019133062654008885,
+            0.018050678772708717,
+            -0.010356118256625172,
+            0.012872052420999813,
+            -0.025620154415048252,
+            -0.005135768485044442,
+            0.004936490920616527,
+            0.005163008729256001,
+            0.007425278892536464,
+            0.001410212306439329,
+            0.021891387289347516,
+            0.014999616600766989,
+            0.00027613811156650413,
+            0.00558422822386443,
+            0.01477009526981092,
+            0.010497782599293994,
+            -0.0005567894749233204,
+            -0.00467776506604011,
+            -0.016998035358377343,
+            0.00957271797745408,
+            -0.006065116857898469,
+            0.005138070401846389,
+            0.01698914855850691,
+            -0.01230257809343985,
+            0.011109844802663414,
+            0.007518553660463051,
+            -0.015504773927595823,
+            -0.017169933945839305,
+            0.005552829003043926,
+            -0.02252633790984703,
+            -0.01306722099657118,
+            0.02715689072730398,
+            -0.010513941399990838,
+            -0.02790582732974509,
+            -0.0009847768069205477,
+            0.009797865241108237,
+            -0.002772952073769185,
+            -0.0027968741526000576,
+            0.0037541733651479613,
+            0.010801137722284016,
+            -0.014658156970099418,
+            0.002075415233345957,
+            0.004735264309137368,
+            0.01930804315461151,
+            -0.0014151522239496115,
+            -0.015046118414100617,
+            -0.002288495894227231,
+            0.021592829677937263,
+            0.017174609713578173,
+            0.00265560378681856,
+            -0.016499825201963983,
+            -3.965296928410772e-05,
+            -0.0074142747014567635,
+            0.00021360696629503726,
+            0.00027431174904249525,
+            -0.0056391089282546054,
+            -0.018931681700770225,
+            -0.003888248121485032,
+            0.011181675419027736,
+            -0.00247252920175839,
+            0.01977860312223846,
+            -0.0011527575128223392,
+            -0.015763587433200915,
+            0.001960734212846797,
+            -0.005280860341480037,
+            -0.010372829570478649,
+            0.01351900743232294,
+            -0.009359942374603997,
+            0.006788746813046239,
+            -0.003212015225284119,
+            0.004611143085183367,
+            -0.0003308099698088422,
+            -0.017017265064185255,
+            0.02924142709831441,
+            -0.0018238549856433466,
+            -0.004769274041546295,
+            -0.0014058363976586562,
+            -0.033393331166741945,
+            0.015017165991303838,
+            -0.014574579250749552,
+            0.006745699409061803,
+            0.002705853802154516,
+            0.01272741296341735,
+            0.003253095246601883,
+            -0.008898616461779179,
+            0.0027485606928249326,
+            0.017040360927723885,
+            0.013756402798365262,
+            -0.007292864293090161,
+            0.008694855886257706,
+            -0.0028934389922887676,
+            -0.0037483208383542015,
+            0.01363963339139356,
+            0.004170611039133969,
+            -0.006282657671540625,
+            -0.0009784372731140945
+          ],
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "Mixture-of-Experts (MoE)",
+            "fine-grained expert segmentation",
+            "shared expert isolation"
+          ],
+          "title": "Deepseekmoe: Towards Ultimate Expert Specialization In Mixture-Of-Experts Language Models"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "611ac443-aa6f-4416-a1f9-dade7d45888c",
+        "properties": {
+          "page_content": "6. Alignment For Deepseekmoe 16B\n\nPrevious research indicates that MoE models typically do not emerge significant gains from fine-tuning (Artetxe et al., 2022; Fedus et al., 2021). However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B.\n\n6.1. Experimental Setup\n\nTraining Data. For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios.\n\nHyper-Parameters. During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).\n\nWe employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10−5 without incorporating any learning rate scheduling strategy.\n\nEvaluation Benchmarks. For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3, with the following adjustments: (1) We exclude Pile (Gao et al., 2020) since chat models are seldom employed for pure language modeling. (2) We exclude\n\nMetric # Shot LLaMA2 DeepSeek DeepSeekMoE SFT 7B Chat 7B Chat 16B # Total Params N/A 6.7B 6.9B 16.4B # Activated Params N/A 6.7B 6.9B 2.8B FLOPs per 4K Tokens N/A 187.9T 183.5T 74.4T HellaSwag (Acc.) 0-shot 67.9 71.0 72.2 PIQA (Acc.) 0-shot 76.9 78.4 79.7 ARC-easy (Acc.) 0-shot 69.7 70.2 69.9 ARC-challenge (Acc.) 0-shot 50.8 50.2 50.0 BBH (EM) 3-shot 39.3 43.1 42.2 RACE-middle (Acc.) 5-shot 63.9 66.1 64.8 RACE-high (Acc.) 5-shot 49.6 50.8 50.6 DROP (EM) 1-shot 40.0 41.7 33.8 GSM8K (EM) 0-shot 63.4 62.6 62.2 MATH (EM) 4-shot 13.5 14.7 15.2 HumanEval (Pass@1) 0-shot 35.4 45.1 45.7 MBPP (Pass@1) 3-shot 27.8 39.0 46.2 TriviaQA (EM) 5-shot 60.1 59.5 63.3 NaturalQuestions (EM) 0-shot 35.2 32.7 35.1 MMLU (Acc.) 0-shot 50.0 49.7 47.2 WinoGrande (Acc.) 0-shot 65.1 68.4 69.0 CLUEWSC (EM) 5-shot 48.4 66.2 68.2 CEval (Acc.) 0-shot 35.1 44.7 40.0 CMMLU (Acc.) 0-shot 36.9 51.2 49.3\n\nCHID (Zheng et al., 2019) due to the observed instability of results, hindering the derivation of solid conclusions. (3) We additionally include BBH (Suzgun et al., 2022) to provide a more comprehensive assessment of the reasoning ability of the chat models.\n\nTable 5 | Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations.\n\n6.2. Evaluations\n\nBaselines. In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. Correspondingly, we construct three chat models, including LLaMA2 SFT 7B3, DeepSeek Chat 7B, and DeepSeekMoE Chat 16B.\n\nSubsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks.\n\nResults. The evaluation results are presented in Table 5. Our key observations include: (1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intensive tasks (TriviaQA, NaturalQuestions). (2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. In addition, it also surpasses DeepSeek Chat 7B. (3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1). However, it is worth noting that, after supervised finetuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. (4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations.\n\n",
+          "keyphrases": [
+            "DeepSeekMoE 16B",
+            "supervised fine-tuning",
+            "chat model",
+            "bilingual scenarios",
+            "evaluation benchmarks"
+          ],
+          "title": "Alignment For Deepseekmoe 16B",
+          "embedding": [
+            0.0034426983911544085,
+            -0.007639183662831783,
+            -0.011979537084698677,
+            0.00880228541791439,
+            0.011939010582864285,
+            0.013551952317357063,
+            -0.013365532271564007,
+            0.05112782120704651,
+            -0.01517299935221672,
+            0.02819000743329525,
+            -0.013219637796282768,
+            0.0074000791646540165,
+            -0.008145760744810104,
+            0.00818223413079977,
+            0.01559447217732668,
+            -0.004850983154028654,
+            0.02065213769674301,
+            0.04480573907494545,
+            0.0024822282139211893,
+            -0.021673398092389107,
+            -0.019825404509902,
+            -0.0646149292588234,
+            -0.037348922342061996,
+            0.011112277396023273,
+            0.0042836167849600315,
+            0.004996877163648605,
+            0.019955087453126907,
+            0.009685755707323551,
+            -0.005908716004341841,
+            0.017928779125213623,
+            -0.009920807555317879,
+            0.01728036068379879,
+            0.021122241392731667,
+            -0.012060589157044888,
+            -0.010982593521475792,
+            -0.028498005121946335,
+            0.05323518067598343,
+            0.011598590761423111,
+            -0.01838267222046852,
+            0.021511293947696686,
+            -0.009807334281504154,
+            0.016680574044585228,
+            -0.01202006358653307,
+            -0.014605632983148098,
+            0.0015673497691750526,
+            -0.021851712837815285,
+            -0.03183736279606819,
+            -0.01961466856300831,
+            -0.0014396923361346126,
+            -0.032809991389513016,
+            -0.012806271202862263,
+            -0.0026564907748252153,
+            -0.007140711881220341,
+            0.010480068624019623,
+            -0.033263884484767914,
+            0.034593142569065094,
+            0.045778367668390274,
+            0.03880786523222923,
+            0.04791814833879471,
+            -0.014200371690094471,
+            0.028481794521212578,
+            -0.0202792976051569,
+            0.006269399076700211,
+            -0.01276574470102787,
+            -0.017247939482331276,
+            -0.04185543209314346,
+            -0.018755514174699783,
+            -0.01043954212218523,
+            -0.004026275593787432,
+            0.014378686435520649,
+            -0.025904331356287003,
+            0.03783523663878441,
+            -0.018204357475042343,
+            -0.008129550144076347,
+            0.0036027769092470407,
+            0.0015926785999909043,
+            0.02387802116572857,
+            0.028433164581656456,
+            -0.003852012800052762,
+            -0.03744618594646454,
+            0.015083841979503632,
+            0.006763818673789501,
+            -0.014313844963908195,
+            0.028838425874710083,
+            0.06814881414175034,
+            -0.034366194158792496,
+            -0.024753388017416,
+            -0.07748604565858841,
+            0.006536871660500765,
+            0.0009523651096969843,
+            -0.027719903737306595,
+            -0.019079722464084625,
+            -0.01313048042356968,
+            -0.028757372871041298,
+            -0.025061385706067085,
+            0.0002973607915919274,
+            -0.033263884484767914,
+            -0.007444658316671848,
+            -0.045065104961395264,
+            0.016680574044585228,
+            0.025353174656629562,
+            0.017150677740573883,
+            0.010893435217440128,
+            -0.011955221183598042,
+            0.004008038435131311,
+            0.010763752274215221,
+            -0.020749401301145554,
+            -0.015805207192897797,
+            -0.004713193979114294,
+            0.08585064858198166,
+            -0.03151315450668335,
+            -0.012919744476675987,
+            -0.02702285349369049,
+            0.017572149634361267,
+            0.009450703859329224,
+            -0.010455752722918987,
+            -0.026390643790364265,
+            -0.03867818042635918,
+            0.0368950292468071,
+            -0.02243529073894024,
+            0.015707945451140404,
+            0.003924959804862738,
+            0.03371777758002281,
+            -0.0426335334777832,
+            -0.029713790863752365,
+            0.030427051708102226,
+            0.0018024016171693802,
+            -0.010585436597466469,
+            0.014921736903488636,
+            0.00511035043746233,
+            -0.017588360235095024,
+            0.0023201233707368374,
+            -0.011841747909784317,
+            0.0060262419283390045,
+            0.02198139764368534,
+            0.02209486998617649,
+            0.011525643989443779,
+            0.022613605484366417,
+            -0.01522973645478487,
+            -0.0017942963168025017,
+            0.0014224686892703176,
+            -0.03297209367156029,
+            0.015278367325663567,
+            0.0035744085907936096,
+            -0.012530692853033543,
+            0.008761758916079998,
+            0.025223489850759506,
+            0.06182673200964928,
+            0.020246876403689384,
+            0.004871245939284563,
+            -0.012619850225746632,
+            0.018868986517190933,
+            -0.028449373319745064,
+            0.024169810116291046,
+            0.0041985115967690945,
+            -0.03540366515517235,
+            0.01993887685239315,
+            0.0004736496484838426,
+            -0.012895428575575352,
+            -0.020311718806624413,
+            0.032647885382175446,
+            -0.031269997358322144,
+            -0.009588493034243584,
+            -0.0018652171129360795,
+            0.027266010642051697,
+            0.03141589090228081,
+            0.0100829117000103,
+            0.00386214442551136,
+            -0.04636194184422493,
+            0.037413764744997025,
+            -0.010585436597466469,
+            0.039650809019804,
+            -0.009531755931675434,
+            0.0054710335098207,
+            0.019792983308434486,
+            -0.006848923396319151,
+            0.008648285642266273,
+            0.007845867425203323,
+            -0.06597661226987839,
+            0.015432367101311684,
+            -0.029324740171432495,
+            0.00394724914804101,
+            0.02901674062013626,
+            -0.004611878655850887,
+            0.026828328147530556,
+            -0.0014569159829989076,
+            0.017491096630692482,
+            -0.0015399946132674813,
+            0.01872309297323227,
+            0.011639117263257504,
+            0.022013816982507706,
+            0.02780095487833023,
+            0.022954024374485016,
+            -0.005102245137095451,
+            0.011679643765091896,
+            -0.0150108952075243,
+            -0.012814376503229141,
+            -0.0010678647086024284,
+            0.008826600387692451,
+            0.010609752498567104,
+            0.003880381118506193,
+            -0.01966330036520958,
+            -0.01102311909198761,
+            -0.0032015678007155657,
+            0.018706882372498512,
+            0.005750664044171572,
+            -0.0019341117003932595,
+            0.021397819742560387,
+            0.011760695837438107,
+            -0.029308529570698738,
+            -0.02637443318963051,
+            -0.021738240495324135,
+            -0.011323013342916965,
+            -0.0371219739317894,
+            0.0024761492386460304,
+            0.017183097079396248,
+            0.016445521265268326,
+            0.001868256600573659,
+            0.013762688264250755,
+            -0.027006642892956734,
+            0.007493289653211832,
+            0.0007132606697268784,
+            -0.013008901849389076,
+            -0.04227690398693085,
+            -0.03841881453990936,
+            0.005268402863293886,
+            0.008648285642266273,
+            -0.0009052534005604684,
+            0.03841881453990936,
+            -0.009393966756761074,
+            0.01789635792374611,
+            0.031043048948049545,
+            0.006435556337237358,
+            0.012109220959246159,
+            0.014184161089360714,
+            -0.035144299268722534,
+            0.023829391226172447,
+            -0.007262290455400944,
+            0.021446451544761658,
+            0.033652935177087784,
+            -0.02037656120955944,
+            0.00043211033334955573,
+            -0.027930639684200287,
+            -0.00955607183277607,
+            0.026617590337991714,
+            -0.006220767740160227,
+            0.0009508453658781946,
+            0.017102045938372612,
+            0.025726014748215675,
+            0.033977143466472626,
+            0.009069757536053658,
+            -0.037640709429979324,
+            0.007237974554300308,
+            0.021187083795666695,
+            -0.00024974255939014256,
+            0.04269837588071823,
+            0.05255434289574623,
+            -0.03050810471177101,
+            -0.018739303573966026,
+            -0.054142966866493225,
+            0.01916077546775341,
+            -0.014848790131509304,
+            -0.011898485012352467,
+            0.014727211557328701,
+            0.007477079052478075,
+            0.009977543726563454,
+            0.02397528477013111,
+            0.017134467139840126,
+            0.004692931193858385,
+            0.012644166126847267,
+            -0.011306802742183208,
+            0.012003852985799313,
+            -0.049993086606264114,
+            0.007148817181587219,
+            0.005118455737829208,
+            -0.027557797729969025,
+            0.003432566998526454,
+            -0.010690804570913315,
+            0.0005987742333672941,
+            0.018804144114255905,
+            0.027557797729969025,
+            0.023780759423971176,
+            0.03151315450668335,
+            0.008931968361139297,
+            -0.046005312353372574,
+            -0.01732899248600006,
+            -0.021170873194932938,
+            -0.03909965232014656,
+            -0.005957347806543112,
+            -0.029875895008444786,
+            0.0010040359338745475,
+            -0.005418349523097277,
+            -0.036311451345682144,
+            -0.0011904563289135695,
+            -0.01503521017730236,
+            0.030443262308835983,
+            -0.017815306782722473,
+            0.04136911779642105,
+            0.056801486760377884,
+            -0.004449774045497179,
+            -0.06717618554830551,
+            0.009321019984781742,
+            -0.0016950072022154927,
+            -0.012392903678119183,
+            -0.018917618319392204,
+            0.014443528838455677,
+            0.020846663042902946,
+            0.03050810471177101,
+            -0.043541319668293,
+            -0.04386553168296814,
+            0.013949109241366386,
+            -0.0058844005689024925,
+            -0.01599973440170288,
+            -0.0034467510413378477,
+            -0.000564326997846365,
+            0.03206430748105049,
+            0.03540366515517235,
+            0.017572149634361267,
+            -0.038580916821956635,
+            -2.4299873985000886e-05,
+            0.01407068781554699,
+            0.02404012717306614,
+            -0.055407386273145676,
+            0.023343076929450035,
+            0.010544910095632076,
+            0.028660111129283905,
+            0.02919505536556244,
+            0.034204091876745224,
+            -0.015578261576592922,
+            5.2050803788006306e-05,
+            -0.017134467139840126,
+            -0.018009832128882408,
+            -0.01895003952085972,
+            0.0024700702633708715,
+            -0.027557797729969025,
+            -0.028676321730017662,
+            -0.041012488305568695,
+            -0.01684267818927765,
+            -0.03013526275753975,
+            0.006500398274511099,
+            -0.018868986517190933,
+            -0.0022836499847471714,
+            -0.010309858247637749,
+            0.01783151738345623,
+            -0.01559447217732668,
+            -0.0027334904298186302,
+            -0.023067498579621315,
+            -0.006881344132125378,
+            0.0018307699356228113,
+            -0.0031286205630749464,
+            0.0011549959890544415,
+            0.0025815172120928764,
+            0.030329788103699684,
+            0.030702630057930946,
+            -0.014362475834786892,
+            -0.02569359354674816,
+            -0.0497337207198143,
+            0.00334948836825788,
+            -0.005219771061092615,
+            0.018982460722327232,
+            0.012806271202862263,
+            -0.03747860714793205,
+            -0.039132073521614075,
+            0.010601647198200226,
+            -0.0371219739317894,
+            -0.034041985869407654,
+            0.01977677270770073,
+            0.012141642160713673,
+            -0.005207613576203585,
+            0.0029644896276295185,
+            -0.005458875559270382,
+            0.03174009919166565,
+            0.0023403866216540337,
+            -0.0022897287271916866,
+            0.024883070960640907,
+            0.02076561190187931,
+            0.008413233794271946,
+            -0.019760562106966972,
+            -0.005292718298733234,
+            0.0460701547563076,
+            -0.025077596306800842,
+            0.0008312931749969721,
+            -0.015659313648939133,
+            -0.029276108369231224,
+            0.017523517832159996,
+            0.021916555240750313,
+            0.000971615023445338,
+            -0.014005845412611961,
+            -0.03838639333844185,
+            0.027233589440584183,
+            0.015278367325663567,
+            -0.046556469053030014,
+            -0.0268121175467968,
+            0.0055723488330841064,
+            0.02869253046810627,
+            0.014848790131509304,
+            -0.010285543277859688,
+            0.006245083641260862,
+            0.02929231896996498,
+            -0.0056777168065309525,
+            -0.014905527234077454,
+            0.01545668300241232,
+            -0.02094392664730549,
+            -0.002857095329090953,
+            0.015278367325663567,
+            -6.775469955755398e-05,
+            0.008745548315346241,
+            -0.005000929813832045,
+            0.0182205680757761,
+            -0.0007907669642008841,
+            -0.006431503687053919,
+            -0.037737973034381866,
+            -0.025223489850759506,
+            -0.046102575957775116,
+            0.03439861536026001,
+            0.037089552730321884,
+            0.008737443014979362,
+            0.017491096630692482,
+            -0.01631583832204342,
+            -0.00244372827000916,
+            -0.033490829169750214,
+            0.01182553730905056,
+            0.02536938525736332,
+            0.005013087764382362,
+            0.004587562754750252,
+            0.0069948178716003895,
+            0.009726282209157944,
+            0.014500265009701252,
+            0.0004141268436796963,
+            -0.04347648099064827,
+            0.004790193866938353,
+            0.008003919385373592,
+            0.034982193261384964,
+            -0.031205154955387115,
+            -0.03125378489494324,
+            -0.011939010582864285,
+            0.010309858247637749,
+            0.036473557353019714,
+            -0.034528300166130066,
+            -0.0182205680757761,
+            -0.021835502237081528,
+            0.016048364341259003,
+            -0.006030294578522444,
+            -0.012441535480320454,
+            -0.0003432060475461185,
+            0.008429444395005703,
+            -0.012490167282521725,
+            0.01711825653910637,
+            -0.026115067303180695,
+            -8.656137651996687e-05,
+            -0.04493542015552521,
+            0.004437616094946861,
+            -0.06639808416366577,
+            -0.019760562106966972,
+            0.03543608635663986,
+            -0.024121178314089775,
+            0.0214302409440279,
+            -0.006366661749780178,
+            0.012247010134160519,
+            -0.05654211714863777,
+            0.002972594927996397,
+            -0.008510496467351913,
+            0.04182301089167595,
+            -0.0083078658208251,
+            -0.005134666338562965,
+            -0.034301355481147766,
+            0.002052650786936283,
+            -0.007704025600105524,
+            0.009985649026930332,
+            0.03146452084183693,
+            -0.014216582290828228,
+            -0.0016554942121729255,
+            0.012530692853033543,
+            0.00012430136848706752,
+            0.011955221183598042,
+            0.03073505125939846,
+            -0.017523517832159996,
+            -0.04227690398693085,
+            0.02071698009967804,
+            0.036797765642404556,
+            -0.021057400852441788,
+            -0.016080785542726517,
+            -0.02530454285442829,
+            0.0042836167849600315,
+            0.006447714287787676,
+            0.004769930616021156,
+            0.017685621976852417,
+            -0.0047739832662045956,
+            0.02065213769674301,
+            0.015059526078402996,
+            0.021900344640016556,
+            0.030054211616516113,
+            0.01636446826159954,
+            0.017636990174651146,
+            -0.017458675429224968,
+            -0.03783523663878441,
+            0.010966382920742035,
+            -0.03543608635663986,
+            0.0035926452837884426,
+            0.02747674658894539,
+            -0.03079989179968834,
+            -0.009645229205489159,
+            -0.006646292749792337,
+            4.45787918579299e-05,
+            0.013138585723936558,
+            0.0061478205025196075,
+            0.0366680808365345,
+            0.011031224392354488,
+            0.005341349635273218,
+            -0.0007411224069073796,
+            0.028887057676911354,
+            -0.015391840599477291,
+            0.035144299268722534,
+            -0.015853838995099068,
+            -0.01553773507475853,
+            0.006549029611051083,
+            -0.010520595125854015,
+            0.019258037209510803,
+            -0.008206550031900406,
+            -0.0043200901709496975,
+            -0.022662237286567688,
+            -0.008769864216446877,
+            0.0071204486303031445,
+            0.016542784869670868,
+            -0.019484983757138252,
+            0.02603401429951191,
+            -0.0005334257730282843,
+            -0.014865000732243061,
+            0.01950119435787201,
+            -0.03024873696267605,
+            -0.023294445127248764,
+            0.0031002522446215153,
+            -0.003789197187870741,
+            0.003740565851330757,
+            -0.005426454823464155,
+            -0.0024072546511888504,
+            -0.015772787854075432,
+            -0.002682832768186927,
+            -0.02120329439640045,
+            -0.014305739663541317,
+            -0.006678713485598564,
+            -0.005000929813832045,
+            -0.01509194727987051,
+            0.002642306499183178,
+            -0.023861810564994812,
+            0.007683762814849615,
+            0.0323074646294117,
+            -0.03922933712601662,
+            0.019857825711369514,
+            0.03627903014421463,
+            0.003473093034699559,
+            -0.018917618319392204,
+            -0.022581184282898903,
+            -0.010917751118540764,
+            -0.010042386129498482,
+            -0.05044697970151901,
+            0.001210719463415444,
+            0.0015308762667700648,
+            -0.0021843607537448406,
+            -0.022921603173017502,
+            0.0013393900590017438,
+            0.016583310440182686,
+            -0.012676587328314781,
+            0.012247010134160519,
+            -0.005045508500188589,
+            -0.014727211557328701,
+            0.014832579530775547,
+            0.027768533676862717,
+            -0.02679590694606304,
+            -0.0029462529346346855,
+            -0.0036088558845221996,
+            0.0033839354291558266,
+            0.0045835101045668125,
+            -0.008931968361139297,
+            -0.007704025600105524,
+            0.033750198781490326,
+            0.001635231077671051,
+            -0.034917350858449936,
+            -0.012911639176309109,
+            -0.002362675964832306,
+            0.027444325387477875,
+            -0.024769596755504608,
+            -0.004899614490568638,
+            0.03180494159460068,
+            0.026293382048606873,
+            -0.014881211332976818,
+            -0.03130241855978966,
+            -0.028060322627425194,
+            0.02021445520222187,
+            0.00838891789317131,
+            0.025888120755553246,
+            0.00861586444079876,
+            0.0009341283002868295,
+            -0.0538511797785759,
+            0.01899867132306099,
+            -0.018690671771764755,
+            -0.007529763039201498,
+            -0.003687881864607334,
+            0.02055487595498562,
+            -0.07592984288930893,
+            -0.03456072136759758,
+            0.03462556377053261,
+            0.033652935177087784,
+            -0.011104172095656395,
+            -0.00919133611023426,
+            0.012263220734894276,
+            -0.002164097735658288,
+            0.034982193261384964,
+            0.006196451839059591,
+            0.023521391674876213,
+            0.009596598334610462,
+            -0.0013880215119570494,
+            0.011841747909784317,
+            -0.01700478233397007,
+            0.0011874169576913118,
+            -0.02775232493877411,
+            0.016032153740525246,
+            -0.005941137205809355,
+            -0.004397090058773756,
+            -0.02215971238911152,
+            0.051646556705236435,
+            -0.014297634363174438,
+            0.034366194158792496,
+            -0.004624036606401205,
+            0.02083045430481434,
+            0.05394844338297844,
+            0.016972361132502556,
+            -0.03683018684387207,
+            0.0030739102512598038,
+            -0.018301621079444885,
+            -0.041174594312906265,
+            0.012368588708341122,
+            -0.022192133590579033,
+            -0.02598538249731064,
+            0.029227476567029953,
+            0.0009467927156947553,
+            0.006654397584497929,
+            -0.0027719903737306595,
+            0.0004359096637926996,
+            -0.008567233569920063,
+            -0.013673530891537666,
+            0.017523517832159996,
+            -0.006642240099608898,
+            0.04192027449607849,
+            -0.009977543726563454,
+            -0.008364601992070675,
+            0.002861147979274392,
+            0.013357426971197128,
+            0.045000262558460236,
+            0.019582247361540794,
+            -0.016923731192946434,
+            0.03146452084183693,
+            0.022062448784708977,
+            0.03715439513325691,
+            -0.018804144114255905,
+            -0.05135476961731911,
+            0.006358556915074587,
+            -0.0009148783865384758,
+            0.016080785542726517,
+            0.0018115199636667967,
+            0.022564973682165146,
+            0.001599770737811923,
+            0.01789635792374611,
+            -0.0019016907317563891,
+            0.04386553168296814,
+            -0.009167020209133625,
+            -0.009653334505856037,
+            0.027930639684200287,
+            -0.0005764848319813609,
+            0.0336853563785553,
+            0.02039276994764805,
+            -0.03511187806725502,
+            -0.034755248576402664,
+            0.01838267222046852,
+            0.026439275592565536,
+            -0.02237044833600521,
+            0.009102178737521172,
+            0.003746644826605916,
+            0.023991495370864868,
+            -0.014654264785349369,
+            0.012530692853033543,
+            0.0036919345147907734,
+            -0.013195321895182133,
+            -0.024964123964309692,
+            0.015043315477669239,
+            0.004052617587149143,
+            -0.0059006111696362495,
+            0.035533349961042404,
+            -0.03462556377053261,
+            -0.03362051397562027,
+            -0.00743655301630497,
+            -0.007071817293763161,
+            -0.0012198378099128604,
+            0.0371219739317894,
+            -0.005462928209453821,
+            0.01877172477543354,
+            -0.03118894435465336,
+            0.040850382298231125,
+            0.00765539426356554,
+            0.004688878543674946,
+            -0.0071731326170265675,
+            0.0016960203647613525,
+            -0.009977543726563454,
+            0.014257107861340046,
+            0.007736446801573038,
+            -0.007716183550655842,
+            0.0018084804760292172,
+            -0.011006908491253853,
+            -0.007290658541023731,
+            0.01642931066453457,
+            -0.027541587129235268,
+            -0.008518601767718792,
+            0.02259739488363266,
+            0.03465798497200012,
+            0.0007927932892926037,
+            0.001413350342772901,
+            -0.00983165018260479,
+            0.006362609565258026,
+            -0.02232181653380394,
+            0.04299016669392586,
+            -0.02138160914182663,
+            0.014751527458429337,
+            0.01590247079730034,
+            -0.042179640382528305,
+            -0.021446451544761658,
+            0.026050224900245667,
+            0.020198244601488113,
+            -0.018415093421936035,
+            -0.011225750669836998,
+            -0.014962263405323029,
+            0.016810256987810135,
+            -0.03068641945719719,
+            -0.04292532429099083,
+            0.010982593521475792,
+            -0.0015845734160393476,
+            -0.0047050886787474155,
+            -0.01595110259950161,
+            0.018690671771764755,
+            0.01085290964692831,
+            -0.054467178881168365,
+            -0.00502524571493268,
+            0.010009964928030968,
+            0.006107294466346502,
+            0.018706882372498512,
+            0.004672667942941189,
+            0.03362051397562027,
+            0.0006463924655690789,
+            0.01838267222046852,
+            -0.007898551411926746,
+            0.01307374332100153,
+            -0.010950172320008278,
+            -0.019371511414647102,
+            0.03279377892613411,
+            0.011420276015996933,
+            -0.03266409784555435,
+            -0.0044254581443965435,
+            0.0021742291282862425,
+            -0.016631942242383957,
+            0.002182334428653121,
+            -0.015878155827522278,
+            -0.006184294354170561,
+            0.028984319418668747,
+            0.00685297604650259,
+            0.025239700451493263,
+            0.029616527259349823,
+            -0.0035865663085132837,
+            -0.01581331342458725,
+            0.012133536860346794,
+            0.009718176908791065,
+            -0.01160669606178999,
+            0.022013816982507706,
+            -0.031091680750250816,
+            0.035371243953704834,
+            -0.06166462600231171,
+            0.030929576605558395,
+            0.027671271935105324,
+            -0.016794046387076378,
+            0.003426488023251295,
+            -0.021867923438549042,
+            0.01766941137611866,
+            -0.01883656531572342,
+            0.029827265068888664,
+            -0.020360350608825684,
+            0.02913021482527256,
+            -0.021170873194932938,
+            0.021576134487986565,
+            0.002658517099916935,
+            -0.018674461171030998,
+            0.008060656487941742,
+            0.0008348391856998205,
+            -0.010998803190886974,
+            -0.013600584119558334,
+            -0.029454423114657402,
+            0.012425324879586697,
+            0.058292847126722336,
+            -0.033490829169750214,
+            -0.0017031125025823712,
+            0.018917618319392204,
+            -0.02259739488363266,
+            -0.03152936324477196,
+            0.0100829117000103,
+            0.004534878768026829,
+            0.030864734202623367,
+            0.014240897260606289,
+            -0.04075312241911888,
+            -0.04172574728727341,
+            0.008324076421558857,
+            0.010350384749472141,
+            -0.004243090283125639,
+            0.046264681965112686,
+            0.01040712185204029,
+            0.014840684831142426,
+            0.01041522715240717,
+            0.0040566702373325825,
+            0.001815572613850236,
+            0.007890446111559868,
+            -0.007570289541035891,
+            -0.012587429955601692,
+            -0.001791256945580244,
+            -0.028498005121946335,
+            -0.005240034312009811,
+            0.017361413687467575,
+            -0.011598590761423111,
+            -0.010204490274190903,
+            -0.04503268375992775,
+            -0.01572415605187416,
+            0.017069624736905098,
+            -0.003262357087805867,
+            0.04668615385890007,
+            0.013171006925404072,
+            -0.004676720593124628,
+            -0.01933909021317959,
+            -0.02076561190187931,
+            0.04619983956217766,
+            -0.017636990174651146,
+            0.04000743851065636,
+            -0.0015734287444502115,
+            -0.013535741716623306,
+            -0.025110017508268356,
+            0.033879879862070084,
+            -0.03728407993912697,
+            -0.013957214541733265,
+            -0.008117392659187317,
+            -0.0035359086468815804,
+            -0.005458875559270382,
+            0.0018409014446660876,
+            -0.0008019116939976811,
+            -0.030167683959007263,
+            0.024396756663918495,
+            -0.025336964055895805,
+            0.031983256340026855,
+            0.02055487595498562,
+            0.010123438201844692,
+            0.00048200818127952516,
+            -0.03222641348838806,
+            -0.005969505291432142,
+            0.0005481165135279298,
+            0.008761758916079998,
+            -0.0011539828265085816,
+            0.00394724914804101,
+            -0.015934891998767853,
+            -0.01356816291809082,
+            -0.012287535704672337,
+            -0.04253627359867096,
+            -0.015480998903512955,
+            0.021089820191264153,
+            0.010374700650572777,
+            -0.03311799094080925,
+            -0.017636990174651146,
+            -0.03653839975595474,
+            0.006840818095952272,
+            -0.010625963099300861,
+            -0.008680706843733788,
+            0.012473956681787968,
+            0.001798348966985941,
+            0.015019000507891178,
+            -0.005479138810187578,
+            -0.023245813325047493,
+            0.026487907394766808,
+            -0.03219399228692055,
+            -0.029486844316124916,
+            -0.01043954212218523,
+            -0.003635197877883911,
+            -0.010042386129498482,
+            0.031059259548783302,
+            0.021251926198601723,
+            0.0007943130331113935,
+            -0.050479400902986526,
+            -0.03050810471177101,
+            -0.051873501390218735,
+            0.024396756663918495,
+            0.0009315954521298409,
+            -0.00938586238771677,
+            0.003252225462347269,
+            0.02039276994764805,
+            0.008178181946277618,
+            -0.01313048042356968,
+            -0.02603401429951191,
+            -0.024023916572332382,
+            -0.013884266838431358,
+            -0.005195455625653267,
+            -0.029065372422337532,
+            -0.033263884484767914,
+            -0.037089552730321884,
+            -0.01334932167083025,
+            0.005916821304708719,
+            0.011776906438171864,
+            -0.003000963246449828,
+            0.0008049511234275997,
+            -0.006759766023606062,
+            -0.02459128201007843,
+            0.004125564359128475,
+            0.015205420553684235,
+            -0.01998750865459442,
+            0.02708769403398037,
+            0.009629018604755402,
+            0.0141517398878932,
+            -0.029843473806977272,
+            0.013859951868653297,
+            0.009118389338254929,
+            0.014338160865008831,
+            -0.004976614378392696,
+            -0.003906723111867905,
+            0.02060350775718689,
+            -0.01972814090549946,
+            0.026017803698778152,
+            -0.006350451614707708,
+            -0.0024700702633708715,
+            -0.010066702030599117,
+            0.020360350608825684,
+            0.003473093034699559,
+            0.012725219130516052,
+            0.022224554792046547,
+            -2.4996417778311297e-05,
+            0.037089552730321884,
+            -0.005090087652206421,
+            -0.01326016429811716,
+            0.00575876934453845,
+            -0.02121950499713421,
+            -0.005702032707631588,
+            0.005993821192532778,
+            -0.029000530019402504,
+            0.0012887322809547186,
+            0.014792053960263729,
+            0.009880281053483486,
+            -0.0318535715341568,
+            0.023634864017367363,
+            0.02603401429951191,
+            0.006848923396319151,
+            0.010399016551673412,
+            -0.033815041184425354,
+            0.0006266359705477953,
+            0.011274381540715694,
+            -0.004778035916388035,
+            0.036700502038002014,
+            0.0201658234000206,
+            -0.03614934906363487,
+            0.010998803190886974,
+            -0.01043954212218523,
+            0.00396548630669713,
+            -0.004765877965837717,
+            0.029616527259349823,
+            -0.022338027134537697,
+            -0.011006908491253853,
+            -0.029324740171432495,
+            0.010180175304412842,
+            -0.021948976442217827,
+            -0.015497208572924137,
+            -0.03524155914783478,
+            0.03579271584749222,
+            0.005811453331261873,
+            -0.013551952317357063,
+            0.0029989369213581085,
+            -0.02039276994764805,
+            -0.0071690804325044155,
+            -0.02951926551759243,
+            -0.012660376727581024,
+            -0.012230799533426762,
+            -0.0007537868223153055,
+            0.016810256987810135,
+            0.02491549216210842,
+            0.006439608987420797,
+            0.029389580711722374,
+            0.03728407993912697,
+            0.00218841340392828,
+            -0.014167950488626957,
+            0.06169704720377922,
+            -0.0007922867080196738,
+            -0.024607492610812187,
+            0.04075312241911888,
+            -0.024964123964309692,
+            0.0016544810496270657,
+            0.01625099591910839,
+            0.014743422158062458,
+            -0.03211294114589691,
+            0.016923731192946434,
+            -0.006739502772688866,
+            -0.033815041184425354,
+            0.003924959804862738,
+            -0.005171139724552631,
+            -0.03195083513855934,
+            -0.04185543209314346,
+            -0.010115332901477814,
+            0.004879351239651442,
+            -0.02160855568945408,
+            0.017020992934703827,
+            0.020473822951316833,
+            -0.002723358804360032,
+            -0.014500265009701252,
+            0.02275949902832508,
+            0.001914861728437245,
+            0.006970501970499754,
+            -0.006771923508495092,
+            0.007833709008991718,
+            -0.0027253851294517517,
+            -0.0010035294108092785,
+            -0.024834439158439636,
+            -0.015140579082071781,
+            -0.041239432990550995,
+            0.0159267857670784,
+            -0.023780759423971176,
+            -0.046880677342414856,
+            0.0056412434205412865,
+            -0.03744618594646454,
+            0.03355567157268524,
+            0.017474886029958725,
+            0.0005683795898221433,
+            -0.0033434093929827213,
+            -0.007716183550655842,
+            -0.021689608693122864,
+            0.010917751118540764,
+            0.022856762632727623,
+            0.004316037520766258,
+            -0.0059897685423493385,
+            0.014192266389727592,
+            -0.022305605933070183,
+            0.000483021343825385,
+            -0.060335367918014526,
+            -0.02381318062543869,
+            0.025628753006458282,
+            0.007983656600117683,
+            0.0005856032366864383,
+            -0.014824474230408669,
+            0.006808397360146046,
+            -0.03266409784555435,
+            -0.019420143216848373,
+            0.025126228109002113,
+            0.019744351506233215,
+            0.014086898416280746,
+            -0.004960403777658939,
+            0.0007396026630885899,
+            0.0182205680757761,
+            -0.008850916288793087,
+            0.008980600163340569,
+            0.0030556735582649708,
+            0.025045175105333328,
+            0.00021225583623163402,
+            0.0077931834384799,
+            0.01210111565887928,
+            0.028822215273976326,
+            0.0034710667096078396,
+            0.010544910095632076,
+            0.006038399878889322,
+            0.0009300757083110511,
+            -0.0134060587733984,
+            -0.013276374898850918,
+            -0.008153866045176983,
+            0.024137388914823532,
+            0.018852775916457176,
+            0.00955607183277607,
+            -0.009580387733876705,
+            -0.009880281053483486,
+            0.008956284262239933,
+            0.009004916064441204,
+            -0.001745664980262518,
+            0.01223890483379364,
+            0.012117326259613037,
+            -0.029778633266687393,
+            -0.03355567157268524,
+            -0.0013799162115901709,
+            -0.004984719213098288,
+            0.005795242730528116,
+            0.00802012998610735,
+            0.021948976442217827,
+            0.0013181137619540095,
+            -0.013479005545377731,
+            -0.004980667028576136,
+            0.02530454285442829,
+            0.005742558743804693,
+            -0.01317911222577095,
+            -0.015716049820184708,
+            -0.009766807779669762,
+            0.00772834150120616,
+            0.011193329468369484,
+            -0.003473093034699559,
+            0.0033596197608858347,
+            0.012741428799927235,
+            0.08163592219352722,
+            -0.007533815689384937,
+            0.0036818028893321753,
+            -0.03854849562048912,
+            -0.01284679677337408,
+            -0.008640180341899395,
+            0.019484983757138252,
+            -0.0019867955707013607,
+            0.032323677092790604,
+            0.015051420778036118,
+            0.01849614642560482,
+            0.0059208739548921585,
+            0.015326999127864838,
+            0.005365665536373854,
+            -0.02929231896996498,
+            0.02227318473160267,
+            -0.0005982676520943642,
+            0.011817432008683681,
+            0.0019857825245708227,
+            -0.011979537084698677,
+            -0.011006908491253853,
+            0.02032792940735817,
+            0.003671671263873577,
+            0.02397528477013111,
+            -0.02159234508872032,
+            0.019744351506233215,
+            0.01849614642560482,
+            0.011979537084698677,
+            -0.010301753878593445,
+            -0.005004982464015484,
+            0.0014913631603121758,
+            0.00517924502491951,
+            -0.017037203535437584,
+            -0.0016139548970386386,
+            -0.019582247361540794,
+            0.06652776896953583,
+            -0.030556734651327133,
+            0.0007244053413160145,
+            0.0031346995383501053,
+            -0.018577197566628456,
+            0.004713193979114294,
+            0.001658533699810505,
+            0.01659141480922699,
+            0.0057263486087322235,
+            -0.006581450812518597,
+            0.005531822796911001,
+            0.024688545614480972,
+            -0.00560882268473506,
+            0.01762077957391739,
+            0.005523717496544123,
+            -0.012263220734894276,
+            -0.014638054184615612,
+            0.012076799757778645,
+            0.011939010582864285,
+            0.009142705239355564,
+            0.025904331356287003,
+            0.008012024685740471,
+            -0.009880281053483486,
+            -0.020895294845104218,
+            -0.011582380160689354,
+            0.006151873152703047,
+            0.005876295268535614,
+            0.03815944492816925,
+            0.030216315761208534,
+            -0.0006884383619762957,
+            -0.013592478819191456,
+            0.016259100288152695,
+            -0.005365665536373854,
+            0.024169810116291046,
+            -0.0029259896837174892,
+            -0.013608689419925213,
+            0.008980600163340569,
+            0.009564177133142948,
+            -0.008178181946277618,
+            -0.02747674658894539,
+            -0.011582380160689354,
+            -0.01013964880257845,
+            -0.01581331342458725,
+            0.010099122300744057,
+            -0.027071483433246613,
+            -0.002474122913554311,
+            0.0011884300038218498,
+            0.027039064094424248,
+            0.03738134354352951,
+            -0.011768801137804985,
+            -0.00539808627218008,
+            -0.0008814443135634065,
+            -0.025223489850759506,
+            -0.005000929813832045,
+            -0.020749401301145554,
+            -0.024964123964309692,
+            -0.016810256987810135,
+            -0.016105102375149727,
+            -0.005527770146727562,
+            -0.0029178846161812544,
+            0.004474089480936527,
+            0.0015045341569930315,
+            0.058130744844675064,
+            -0.01102311909198761,
+            -0.012084905058145523,
+            -0.02076561190187931,
+            0.006942133419215679,
+            0.019128354266285896,
+            -0.011387854814529419,
+            0.0027274114545434713,
+            -0.01393289864063263,
+            0.004010064993053675,
+            -0.014881211332976818,
+            -0.014362475834786892,
+            -0.023002656176686287,
+            0.002506543882191181,
+            -0.006455819588154554,
+            0.0028084637597203255,
+            -0.00627345172688365,
+            0.009726282209157944,
+            0.019857825711369514,
+            -3.7233421608107165e-05,
+            0.001175259007140994,
+            -0.008810389786958694,
+            0.015043315477669239,
+            -0.030491894111037254,
+            0.02392665296792984,
+            0.019144564867019653,
+            0.0033373304177075624,
+            -0.0053778234869241714,
+            0.02752537652850151,
+            0.00364127685315907,
+            -0.011720169335603714,
+            0.04568110406398773,
+            0.010399016551673412,
+            0.026179907843470573,
+            -0.009872175753116608,
+            0.005774979945272207,
+            -0.0181070938706398,
+            0.009231862612068653,
+            -0.006484187673777342,
+            0.0072420272044837475,
+            -0.006131610367447138,
+            0.02780095487833023,
+            -0.008777969516813755,
+            0.0013464820804074407,
+            0.002717280061915517,
+            0.04098006710410118,
+            -0.0019452563719823956,
+            0.015853838995099068,
+            -0.026617590337991714,
+            -0.00035536388168111444,
+            0.02126813679933548,
+            0.014362475834786892,
+            0.017183097079396248,
+            -0.000494925887323916,
+            -0.01950119435787201,
+            -0.0045997207053005695,
+            -0.014321950264275074,
+            0.0009179178159683943,
+            -0.030005579814314842,
+            0.002401175908744335,
+            -0.00473345723003149,
+            0.013187216594815254,
+            0.013876162469387054,
+            -0.011695853434503078,
+            0.02044140174984932,
+            -0.017701832577586174,
+            -0.02376454882323742,
+            0.008964389562606812,
+            0.0018135462887585163,
+            0.01700478233397007,
+            0.017020992934703827,
+            -0.022030027583241463,
+            -0.006982659921050072,
+            -0.01883656531572342,
+            -0.01849614642560482,
+            0.0032562781125307083,
+            0.034041985869407654,
+            -0.007096133194863796,
+            -0.010828593745827675,
+            0.018188146874308586,
+            -0.01750730723142624,
+            0.01732899248600006,
+            -0.007250132504850626,
+            0.01954982616007328,
+            -0.001856098766438663,
+            -0.006609818898141384,
+            0.005240034312009811,
+            -0.02358623407781124,
+            0.0012897454435005784,
+            0.00999375432729721,
+            0.001181337982416153,
+            -0.03229125589132309,
+            0.012036273255944252,
+            -0.0020678481087088585,
+            0.001809493638575077,
+            -0.006484187673777342,
+            0.014767738059163094,
+            0.006788134109228849,
+            0.00869691651314497,
+            -0.002474122913554311,
+            0.010074807330965996,
+            0.003671671263873577,
+            -0.015083841979503632,
+            -0.003112410195171833,
+            0.018593408167362213,
+            0.008599653840065002,
+            -0.011614801362156868,
+            0.00575876934453845,
+            -0.014946052804589272,
+            -0.016080785542726517,
+            0.019258037209510803,
+            0.019760562106966972,
+            -0.031432099640369415,
+            0.022791920229792595,
+            -0.024461599066853523,
+            -0.0035521192476153374,
+            0.017977410927414894,
+            -0.0020972294732928276,
+            -0.0008490233449265361,
+            -0.004474089480936527,
+            -0.027557797729969025,
+            -0.01428952906280756,
+            0.005924926605075598,
+            -0.011525643989443779,
+            0.00452677346765995,
+            0.025288332253694534,
+            0.015691734850406647,
+            -0.0056250328198075294,
+            -0.0016858888557180762,
+            0.030832313001155853,
+            -0.003071883926168084,
+            0.0054831914603710175,
+            0.005126561038196087,
+            -0.028870847076177597,
+            0.02995694801211357,
+            -0.024056337773799896,
+            0.007550026290118694,
+            -0.000363722414476797,
+            -0.00247817556373775,
+            -0.006358556915074587,
+            0.014411107636988163,
+            -0.0005481165135279298,
+            0.0016200337558984756,
+            -0.014475949108600616,
+            0.006269399076700211,
+            -0.019355300813913345,
+            0.030864734202623367,
+            -0.0016088890843093395,
+            -0.00017768194084055722,
+            0.015197315253317356,
+            -0.015205420553684235,
+            -0.017863936722278595,
+            0.01013964880257845,
+            0.008453760296106339,
+            -0.0230350773781538,
+            0.004778035916388035,
+            -0.01933909021317959,
+            -0.03871060162782669,
+            -0.026115067303180695,
+            0.003706118557602167,
+            0.008915758691728115,
+            0.0165022574365139,
+            -0.009426387958228588,
+            -0.0061113471165299416,
+            0.02692558988928795,
+            0.02358623407781124,
+            0.03378261998295784,
+            -0.013949109241366386,
+            0.009094073437154293,
+            0.0027963060420006514,
+            0.01317911222577095,
+            0.001355600543320179,
+            -0.004384932108223438,
+            0.0033839354291558266,
+            -0.004247142933309078,
+            -0.012587429955601692,
+            -0.0018044279422610998,
+            0.027606429532170296,
+            -0.006650345399975777,
+            -0.024380546063184738,
+            -0.03151315450668335,
+            0.003161041531711817,
+            -0.005231929011642933,
+            0.0040222229436039925,
+            0.03971565142273903,
+            -0.024510230869054794,
+            -0.02587191015481949,
+            0.02026308700442314,
+            -0.003205620450899005,
+            0.004567299969494343,
+            -0.0019847694784402847,
+            0.01027743797749281,
+            -0.014775843359529972,
+            -0.028643900528550148,
+            -0.022192133590579033,
+            -0.0027193063870072365,
+            0.0037425921764224768,
+            -0.007128553930670023,
+            -0.016972361132502556,
+            -0.007886393927037716,
+            0.007622973527759314,
+            -0.016510363668203354,
+            0.013762688264250755,
+            -0.02237044833600521,
+            0.010147754102945328,
+            -0.0012299694353714585,
+            0.030653998255729675,
+            -0.015513419173657894,
+            -0.01608889177441597,
+            -0.023894231766462326,
+            0.007254185155034065,
+            0.02099255844950676,
+            -0.0009913715766742826,
+            0.0012026141630485654,
+            0.006780028808861971,
+            0.005292718298733234,
+            0.007197448518127203,
+            -0.0069948178716003895,
+            0.019063511863350868,
+            -0.034820087254047394,
+            -0.0024356229696422815,
+            0.0019047302193939686,
+            0.039521124213933945,
+            -0.027168747037649155,
+            -0.003321120049804449,
+            -0.007817499339580536,
+            0.010228806175291538,
+            0.013519532047212124,
+            0.013162901625037193,
+            -0.0027679377235472202,
+            -0.01677783578634262,
+            -0.022451501339673996,
+            0.0005597677663899958,
+            -0.038516074419021606,
+            0.0026544644497334957,
+            0.015197315253317356,
+            -0.0013991661835461855,
+            0.03751102834939957,
+            0.0469779409468174,
+            -0.03738134354352951,
+            -0.006269399076700211,
+            0.009799228981137276,
+            0.03410682827234268,
+            -0.020911505445837975,
+            0.020408980548381805,
+            -0.011063645593822002,
+            0.01620236411690712,
+            0.0035602243151515722,
+            -0.009085968136787415,
+            -0.02786579728126526,
+            -0.014905527234077454,
+            0.021089820191264153,
+            0.008145760744810104,
+            -0.021478872746229172,
+            0.009734386578202248,
+            0.011558065190911293,
+            -0.008364601992070675,
+            -0.025061385706067085,
+            0.02598538249731064,
+            0.01883656531572342,
+            0.01157427579164505,
+            -0.018204357475042343,
+            -0.011768801137804985,
+            -0.02674727514386177,
+            0.008996810764074326,
+            -0.020846663042902946,
+            0.01723172888159752,
+            -0.023861810564994812,
+            0.0008839771617203951,
+            -0.01844751462340355,
+            -0.007225817069411278,
+            0.006532819475978613,
+            -0.04370342567563057,
+            -0.006476082839071751,
+            -0.005924926605075598,
+            0.012960270047187805,
+            0.028951898217201233,
+            0.021414030343294144,
+            -0.014394897036254406,
+            -0.02890326827764511,
+            -0.003379883011803031,
+            -0.0073960269801318645,
+            0.013446584343910217,
+            -0.0059208739548921585,
+            -0.008397023193538189,
+            -0.0022654130589216948,
+            0.008332181721925735,
+            -0.010026175528764725,
+            0.007622973527759314,
+            0.006082978565245867,
+            0.001210719463415444,
+            0.018593408167362213,
+            -0.0008186287013813853,
+            -0.013316900469362736,
+            -0.0033900144044309855,
+            -0.008542917668819427,
+            0.02083045430481434,
+            0.03355567157268524,
+            0.003979670349508524,
+            -0.007383869029581547,
+            -0.00466456264257431,
+            0.004648352041840553,
+            0.020684558898210526,
+            -0.015416156500577927,
+            -0.0041620382107794285,
+            -0.003440672066062689,
+            0.01307374332100153,
+            -0.011736379936337471,
+            0.027039064094424248,
+            -0.013235848397016525,
+            -0.023521391674876213,
+            -0.022030027583241463,
+            -0.0559261217713356,
+            0.006605766247957945,
+            -0.028400743380188942,
+            -0.03096199780702591,
+            -0.01581331342458725,
+            -0.012287535704672337,
+            -0.01760457083582878,
+            0.011849853210151196,
+            -0.005697980057448149,
+            -0.0032603307627141476,
+            0.005965452641248703,
+            0.013219637796282768,
+            -0.016380678862333298,
+            0.0008576351683586836,
+            0.00511035043746233,
+            0.0010156872449442744,
+            -0.01922561787068844,
+            0.016599521040916443,
+            0.009361546486616135,
+            -0.01223890483379364,
+            -0.013892372138798237,
+            0.0021661240607500076,
+            0.013187216594815254,
+            -0.004591615404933691,
+            0.006301820278167725,
+            0.015562050975859165,
+            0.010180175304412842,
+            -0.0214302409440279,
+            -0.0075540789403021336,
+            0.005065771751105785,
+            -0.007059659343212843,
+            0.01487310603260994,
+            -0.015667418017983437,
+            -0.006885396782308817,
+            0.0026321751065552235,
+            0.012076799757778645,
+            -0.018463725224137306,
+            -0.007343342527747154,
+            -0.00350348767824471,
+            3.1661072625865927e-06,
+            -0.018739303573966026,
+            0.0003079989110119641,
+            0.012919744476675987,
+            0.0008267339435406029,
+            0.008129550144076347,
+            -0.01317911222577095,
+            0.003142804838716984,
+            0.028384532779455185,
+            0.019760562106966972,
+            -0.037024714052677155,
+            0.0013272322248667479,
+            0.014362475834786892,
+            -0.009150810539722443,
+            -0.01989024691283703,
+            0.019063511863350868,
+            0.02298644557595253,
+            0.002540991175919771,
+            0.0017254018457606435,
+            -0.0023464655969291925,
+            -0.003270462155342102,
+            -0.01865825057029724,
+            0.011566170491278172,
+            0.01116090826690197,
+            -0.005843874532729387,
+            -0.0012887322809547186,
+            0.006366661749780178,
+            -0.01642931066453457,
+            -0.006326135713607073,
+            0.007789130788296461,
+            0.014856895431876183,
+            0.007489237003028393,
+            0.009499335661530495,
+            0.010561120696365833,
+            0.004575405269861221,
+            0.0015967312501743436,
+            0.024104967713356018,
+            0.07385490089654922,
+            0.0013819425366818905,
+            0.008380812592804432,
+            0.018852775916457176,
+            -0.006504450924694538,
+            -0.013187216594815254,
+            0.005860084667801857,
+            -0.008016076870262623,
+            -3.7233421608107165e-05,
+            0.005240034312009811,
+            -0.04072070121765137,
+            -0.02165718749165535,
+            -0.010877225548028946,
+            -0.013381742872297764,
+            0.004591615404933691,
+            0.018042253330349922,
+            -0.01799362152814865,
+            0.01590247079730034,
+            0.03793250024318695,
+            0.015148683451116085,
+            -0.009223757311701775,
+            0.01437058113515377,
+            0.0071893432177603245,
+            -0.011087961494922638,
+            -0.006646292749792337,
+            -0.01227132510393858,
+            0.008255181834101677,
+            0.015027105808258057,
+            0.0005825637490488589,
+            0.02741190418601036,
+            -0.0026200171560049057,
+            -0.00540213892236352,
+            0.003217778168618679,
+            -0.02491549216210842,
+            -0.015942996367812157,
+            0.004798299167305231,
+            0.008198444731533527,
+            -0.015083841979503632,
+            -0.013859951868653297,
+            -0.006881344132125378,
+            0.011087961494922638,
+            -0.0017872042953968048,
+            -0.006553082261234522,
+            -0.0013981530210003257,
+            -0.002670674817636609,
+            -0.004782088566571474,
+            -0.024023916572332382,
+            0.0024579125456511974,
+            0.013381742872297764,
+            0.006046505179256201,
+            0.014727211557328701,
+            0.003460935316979885,
+            0.007108291145414114,
+            -0.006739502772688866,
+            -0.012935955077409744,
+            -0.00822681374847889,
+            0.012563114054501057,
+            0.021365398541092873,
+            0.013414164073765278,
+            0.010553015395998955,
+            -0.005568296182900667,
+            0.007371711079031229,
+            -0.008113340474665165,
+            -0.015148683451116085,
+            -0.00663008214905858,
+            -0.00176896748598665,
+            0.0019006775692105293,
+            -0.004453826695680618,
+            0.002962463302537799,
+            -0.002729437779635191,
+            0.006982659921050072,
+            0.004842877853661776,
+            0.008559128269553185,
+            0.0007877275347709656,
+            -0.002368754940107465,
+            -0.012149747461080551,
+            -0.014719106256961823,
+            -0.019193196669220924,
+            0.017815306782722473,
+            0.010034280829131603,
+            0.013633005321025848,
+            0.018917618319392204,
+            -0.0024822282139211893,
+            -0.012749534100294113,
+            -0.0197119303047657,
+            0.00038297235732898116,
+            0.003321120049804449,
+            0.016056470572948456,
+            0.026228539645671844,
+            -0.01789635792374611,
+            0.00382972345687449,
+            -0.006168083753436804,
+            0.01179311703890562,
+            -0.022678447887301445,
+            -0.010293648578226566,
+            0.015878155827522278,
+            -0.030281158164143562,
+            -5.6609998864587396e-05,
+            -0.010723225772380829,
+            -0.00444572139531374,
+            0.018593408167362213,
+            0.023602444678544998,
+            0.017247939482331276,
+            0.012976480647921562,
+            -0.010471963323652744,
+            -0.005009035114198923,
+            0.01920940726995468,
+            0.008559128269553185,
+            -0.005795242730528116,
+            0.0002556948165874928,
+            -0.007732394151389599,
+            -0.02104119025170803,
+            0.005146824289113283,
+            0.00969386100769043,
+            -0.024737177416682243,
+            0.005122508388012648,
+            -0.017701832577586174,
+            -0.0039350916631519794,
+            -0.018431304022669792,
+            -0.00305972620844841,
+            0.00611945241689682,
+            0.00244372827000916,
+            0.0024457545951008797,
+            0.0004257781256455928,
+            -0.003122541820630431,
+            -0.0005324126104824245,
+            0.01284679677337408,
+            -0.007489237003028393,
+            -0.01972814090549946,
+            -0.027379482984542847,
+            -0.010844804346561432,
+            -0.0018165857763960958,
+            0.002555175218731165,
+            -0.007335237693041563,
+            -0.02254876308143139,
+            0.0029644896276295185,
+            -0.009896491654217243,
+            -0.000849529926199466,
+            -0.001954374834895134,
+            0.02630959264934063,
+            0.002792253391817212,
+            -0.015245947055518627,
+            -0.034301355481147766,
+            -0.03123757429420948,
+            0.011622906662523746,
+            -0.0008160958532243967,
+            0.004672667942941189,
+            -0.011631011962890625,
+            0.001885480247437954,
+            -0.016169942915439606,
+            0.01464615948498249,
+            -0.022532552480697632,
+            -0.008012024685740471,
+            0.009337230585515499,
+            -0.0003305416030343622,
+            0.001216798322275281,
+            0.000767970981542021,
+            -0.0074325003661215305,
+            0.003718276508152485,
+            -0.01728036068379879,
+            -0.009961334057152271,
+            -0.02087908424437046,
+            -0.00027431154740042984,
+            0.004352510906755924,
+            -0.013422268442809582,
+            0.019031090661883354,
+            0.01987403631210327,
+            0.0038884864188730717,
+            -0.012109220959246159,
+            0.017863936722278595,
+            0.01188227441161871,
+            -0.009547966532409191,
+            -0.014613738283514977,
+            -0.021786870434880257,
+            0.011266276240348816,
+            0.026325803250074387,
+            0.018415093421936035,
+            0.0075540789403021336,
+            0.01588626019656658,
+            0.012287535704672337,
+            0.016007838770747185,
+            0.001041522715240717,
+            0.0017831516452133656,
+            0.0009766807779669762,
+            0.011979537084698677,
+            -0.015910575166344643,
+            0.0035359086468815804,
+            0.0090535469353199,
+            0.00985596515238285,
+            -0.016072681173682213,
+            -0.01290353387594223,
+            0.024007705971598625,
+            -0.0010516542242839932,
+            -0.012028168886899948,
+            0.0028348059859126806,
+            0.00540213892236352,
+            -0.007177185267210007,
+            0.000767970981542021,
+            -0.005462928209453821,
+            -0.002014150843024254,
+            0.01590247079730034,
+            0.017766674980521202,
+            -0.009758702479302883,
+            -0.004401142708957195,
+            0.011995747685432434,
+            -0.0028773583471775055,
+            -0.03190220519900322,
+            0.006184294354170561,
+            0.028822215273976326,
+            0.006018136627972126,
+            -0.03790007904171944,
+            -0.0013454690342769027,
+            0.005219771061092615,
+            -0.032048098742961884,
+            -0.03449587896466255,
+            0.0021965187042951584,
+            -0.010771856643259525,
+            -0.015294577926397324,
+            -0.0035521192476153374,
+            0.0075905523262917995,
+            -0.028984319418668747,
+            0.009410177357494831,
+            0.014281423762440681,
+            -0.0006276491330936551,
+            0.022240763530135155,
+            -0.004944193176925182,
+            -0.012352378107607365,
+            -0.027266010642051697,
+            0.02237044833600521,
+            -0.013470900245010853,
+            -0.008461865596473217,
+            -0.0012380746193230152,
+            -0.009945123456418514,
+            0.005240034312009811,
+            0.003071883926168084,
+            0.02890326827764511,
+            0.018479935824871063,
+            -0.0029259896837174892,
+            -0.04201753810048103,
+            -0.022338027134537697,
+            0.003971565049141645,
+            -0.025045175105333328,
+            0.013389848172664642,
+            -0.006054610479623079,
+            -0.00321169919334352,
+            0.005969505291432142,
+            0.03164283558726311,
+            0.011274381540715694,
+            0.019355300813913345,
+            -3.3022501156665385e-05,
+            0.0042674061842262745,
+            -0.03384746238589287,
+            -0.02415359951555729,
+            0.00947501976042986,
+            -0.0024639912880957127,
+            0.032647885382175446,
+            0.0036270925775170326,
+            0.004619983956217766,
+            0.014475949108600616,
+            -0.006719239521771669,
+            -0.004054643679410219,
+            -0.018804144114255905,
+            0.011922799982130527,
+            -0.004761825315654278,
+            0.01932287961244583,
+            0.01625099591910839,
+            -0.014378686435520649,
+            -0.022354237735271454,
+            0.005223823711276054,
+            0.04986340552568436,
+            0.013803214766085148,
+            -0.0006757739465683699,
+            0.005839821882545948,
+            0.0021195188164711,
+            0.0021073610987514257,
+            0.004340353421866894,
+            0.004046538379043341,
+            0.015002789907157421,
+            0.026212329044938087,
+            -0.004745615180581808,
+            0.0036736975889652967,
+            0.00350348767824471,
+            -0.005098192486912012,
+            -0.0028307533357292414,
+            -0.013033217750489712,
+            0.004684825893491507,
+            -0.010925856418907642,
+            0.008125497959554195,
+            -0.02053866535425186,
+            -0.011995747685432434,
+            -0.011055540293455124,
+            -0.010609752498567104,
+            -0.010009964928030968,
+            -0.018787933513522148,
+            0.01927424781024456,
+            -0.029730001464486122,
+            0.002589622512459755,
+            -0.013227743096649647,
+            0.013689741492271423,
+            0.009620913304388523,
+            -0.0006119452300481498,
+            0.02353760227560997,
+            -0.023407917469739914,
+            -0.007935024797916412,
+            0.004465984180569649,
+            0.00266662216745317,
+            -0.0022654130589216948,
+            0.003124567912891507,
+            -0.013057533651590347,
+            -0.025612542405724525,
+            0.009547966532409191,
+            -0.00540213892236352,
+            0.0034305406734347343,
+            0.006747608073055744,
+            -0.0010405095526948571,
+            -0.005843874532729387,
+            -0.003625066252425313,
+            0.01826919987797737,
+            -0.020425191149115562,
+            0.04114217311143875,
+            -0.03018389455974102,
+            -0.00480235181748867,
+            0.020311718806624413,
+            -0.01723172888159752,
+            -0.0001227183238370344,
+            -0.0020648085046559572,
+            0.0073636057786643505,
+            0.0035014613531529903,
+            -0.0022188080474734306,
+            -0.014386791735887527,
+            0.03161041438579559,
+            3.1233648769557476e-05,
+            -0.016745414584875107,
+            0.00889144279062748,
+            -0.013308795168995857,
+            0.017734253779053688,
+            -0.028108954429626465,
+            -1.29652098621591e-05,
+            0.015221631154417992,
+            0.0016281390562653542,
+            0.009783018380403519,
+            -0.01121764536947012,
+            -0.010958277620375156,
+            -0.01260364055633545,
+            0.015797102823853493,
+            0.01933909021317959,
+            -0.011525643989443779,
+            -0.022840552031993866,
+            0.0040242490358650684,
+            0.01666436344385147,
+            -0.021867923438549042,
+            -0.0017132440116256475,
+            -0.004397090058773756,
+            -0.009531755931675434,
+            0.0003996893938165158,
+            -0.012587429955601692,
+            0.011801222339272499,
+            0.014467843808233738,
+            -0.0046686152927577496,
+            -0.03316662088036537,
+            0.008084971457719803,
+            0.010625963099300861,
+            -0.03169146925210953,
+            -0.013049428351223469,
+            0.012555008754134178,
+            -0.011898485012352467,
+            0.010180175304412842,
+            -0.006775976158678532,
+            0.00986407045274973,
+            -0.00928049348294735,
+            -0.001244153594598174,
+            -0.00031737060635350645,
+            -0.0007380829192698002,
+            -0.022856762632727623,
+            -0.008648285642266273,
+            -0.0009584440267644823,
+            -0.03079989179968834,
+            0.011695853434503078,
+            0.003937117755413055,
+            -0.009734386578202248,
+            -0.0006560174515470862,
+            -0.017085835337638855,
+            -0.004368721507489681,
+            0.0045105633325874805,
+            0.00685297604650259,
+            -0.011493222787976265,
+            -0.013187216594815254,
+            -0.002350518014281988,
+            -0.016007838770747185,
+            0.018236778676509857,
+            -0.0004701036086771637,
+            0.036635659635066986,
+            -0.0023829389829188585,
+            -0.0013900477206334472,
+            -0.0021418083924800158,
+            -0.00280441134236753,
+            0.005791190080344677,
+            0.03195083513855934,
+            -0.0013971398584544659,
+            0.006390977650880814,
+            0.0018054411048069596,
+            -0.003677750239148736,
+            0.006001926492899656,
+            0.01576468162238598,
+            0.0005633138353005052,
+            0.003975617699325085,
+            -0.009726282209157944,
+            -0.010893435217440128,
+            0.006942133419215679,
+            -0.002822648035362363,
+            0.02021445520222187,
+            -0.0014812316512688994,
+            -0.00932912528514862,
+            -0.009256178513169289,
+            0.0012887322809547186,
+            0.008153866045176983,
+            -0.005977610591799021,
+            -0.002030361210927367,
+            0.02243529073894024,
+            -0.005122508388012648,
+            0.014905527234077454,
+            -0.015424261800944805,
+            0.015805207192897797,
+            0.014394897036254406,
+            -0.013284480199217796,
+            0.00781344622373581,
+            0.018804144114255905,
+            -0.007667552214115858,
+            0.018706882372498512,
+            -0.016550889238715172,
+            0.007602710276842117,
+            -0.00983165018260479,
+            0.03884028643369675,
+            -0.009450703859329224,
+            -0.022013816982507706,
+            -0.0029766475781798363,
+            0.015383735299110413,
+            -0.02005235105752945,
+            0.010569225996732712,
+            0.013114269822835922,
+            -0.02536938525736332,
+            0.002944226609542966,
+            0.014751527458429337,
+            0.003997907042503357,
+            -0.004206616897135973,
+            0.006265346426516771,
+            0.005406191572546959,
+            0.009499335661530495,
+            0.005811453331261873,
+            0.020360350608825684,
+            -0.016048364341259003,
+            0.01972814090549946,
+            -0.012174062430858612,
+            -0.0032035941258072853,
+            -0.022889183834195137,
+            -0.0030131209641695023,
+            -0.009572282433509827,
+            0.011250065639615059,
+            -0.012230799533426762,
+            0.006828660145401955,
+            -0.010042386129498482,
+            -0.0001888266415335238,
+            0.030702630057930946,
+            0.018706882372498512,
+            0.008372707292437553,
+            0.003292751731351018,
+            0.01196332648396492,
+            0.03355567157268524,
+            0.012798165902495384,
+            -0.009742491878569126,
+            0.0025531488936394453,
+            0.023440338671207428,
+            0.00649634562432766,
+            0.0169561505317688,
+            0.008988705463707447,
+            3.9164748159237206e-05,
+            -0.013973425142467022,
+            0.011509433388710022,
+            0.033101778477430344,
+            0.024996543303132057,
+            0.025952961295843124,
+            0.005353507585823536,
+            0.0010577331995591521,
+            -0.01282248180359602,
+            -0.01343037374317646,
+            -0.016680574044585228,
+            -0.011736379936337471,
+            0.022678447887301445,
+            -0.009547966532409191,
+            0.002751727122813463,
+            -0.01872309297323227,
+            0.008632075041532516,
+            0.020019929856061935,
+            0.009507440030574799,
+            0.017653200775384903,
+            0.010228806175291538,
+            0.00451461598277092,
+            -0.012287535704672337,
+            0.0007613854832015932,
+            0.0060059791430830956,
+            -0.024964123964309692,
+            0.016558995470404625,
+            0.031756311655044556,
+            -0.0003996893938165158,
+            0.01539994589984417,
+            -0.0002955877862405032,
+            -0.003209673101082444,
+            -0.012190273031592369,
+            0.02392665296792984,
+            0.010625963099300861,
+            0.012449640780687332,
+            0.0023201233707368374,
+            -0.008713127113878727,
+            -0.011201434768736362,
+            -0.024218441918492317,
+            0.026568960398435593,
+            0.02092771604657173,
+            -0.00196551950648427,
+            0.00386417075060308,
+            -0.001570389256812632,
+            -0.012174062430858612,
+            0.014970368705689907,
+            0.015732260420918465,
+            0.019468773156404495,
+            -0.0028672267217189074,
+            0.012911639176309109,
+            -0.018139515072107315,
+            0.000802418275270611,
+            -0.012781955301761627,
+            0.01700478233397007,
+            0.004275511484593153,
+            -0.021462662145495415,
+            0.016696784645318985,
+            0.013770793564617634,
+            0.0005202547763474286,
+            -0.019955087453126907,
+            0.005531822796911001,
+            -0.0055196648463606834,
+            0.001069891033694148,
+            0.0007168066804297268,
+            -0.0012826534220948815,
+            -0.006747608073055744,
+            0.000229859389946796,
+            0.011703958734869957,
+            -0.0069097126834094524,
+            -0.022954024374485016,
+            -0.010447647422552109,
+            -0.001273535075597465,
+            -0.00656524021178484,
+            -0.020133404061198235,
+            0.015286472626030445,
+            0.004826667252928019,
+            -0.016907520592212677,
+            -0.020036140456795692,
+            -0.0004072880547028035,
+            0.008559128269553185,
+            0.03410682827234268,
+            -0.0284169539809227,
+            0.0032745148055255413,
+            -0.012352378107607365,
+            0.020798033103346825,
+            0.00357238226570189,
+            -0.023359287530183792,
+            0.022775709629058838,
+            0.0008277471060864627,
+            0.0056412434205412865,
+            -0.01179311703890562,
+            -0.0045835101045668125,
+            0.009766807779669762,
+            -0.022743288427591324,
+            0.005353507585823536,
+            0.01049627922475338,
+            -0.0038378287572413683,
+            0.008518601767718792,
+            9.124721691478044e-05,
+            0.005183297675102949,
+            0.0005379849462769926,
+            -0.009920807555317879,
+            0.018787933513522148,
+            -0.0014619816793128848,
+            -0.014605632983148098,
+            0.0034589089918881655,
+            0.013908582739531994,
+            -0.02564496360719204,
+            0.03595482185482979,
+            -0.0038256708066910505,
+            -0.014548896811902523,
+            -0.004433563444763422,
+            0.0010293647646903992,
+            0.019484983757138252,
+            0.030880944803357124,
+            -0.020911505445837975,
+            -0.010763752274215221,
+            0.005090087652206421,
+            0.01137974951416254,
+            -0.009815439581871033,
+            0.004206616897135973,
+            0.017572149634361267,
+            0.010553015395998955,
+            0.011209540069103241,
+            -0.010707015171647072,
+            -0.01595110259950161,
+            0.030427051708102226,
+            0.019922668114304543,
+            0.003683829214423895,
+            0.009701966308057308,
+            -0.002788200741633773,
+            0.020425191149115562,
+            0.0025531488936394453,
+            -0.01124196033924818,
+            0.00452677346765995,
+            0.016413100063800812,
+            -0.010763752274215221,
+            -0.0010749567300081253,
+            0.008247076533734798,
+            -0.011671538464725018,
+            0.008429444395005703,
+            -0.027395693585276604,
+            0.0263582244515419,
+            0.007906656712293625,
+            0.011955221183598042,
+            0.020133404061198235,
+            -0.01977677270770073,
+            -0.0011631011730059981,
+            0.014305739663541317,
+            0.012344272807240486,
+            -0.02353760227560997,
+            -0.008542917668819427,
+            0.008826600387692451,
+            -0.00020794992451556027,
+            -0.0008216681890189648,
+            -0.003635197877883911,
+            -0.013519532047212124,
+            -0.005000929813832045,
+            -0.04130427539348602,
+            0.011047434993088245,
+            -0.004786141216754913,
+            0.011922799982130527,
+            0.005985715892165899,
+            0.00911028403788805,
+            -0.02613127790391445,
+            0.003578461240977049,
+            -0.01000185962766409,
+            -0.015480998903512955,
+            0.005495349410921335,
+            0.006054610479623079,
+            -0.013162901625037193,
+            -0.0035642769653350115,
+            -0.0018358356319367886,
+            0.002589622512459755,
+            0.0021134400740265846,
+            -0.004790193866938353,
+            -0.029178844764828682,
+            -0.00823897123336792,
+            -0.013227743096649647,
+            0.006938081234693527,
+            -0.014840684831142426,
+            0.03052431344985962,
+            0.0005192416138015687,
+            -0.009207546710968018,
+            0.0036574872210621834,
+            -0.013859951868653297,
+            -0.00411948561668396,
+            -0.0038216181565076113,
+            0.01895003952085972,
+            -0.0038074341136962175,
+            0.028643900528550148,
+            0.00883470568805933,
+            -0.010958277620375156,
+            -0.018852775916457176,
+            0.003193462500348687,
+            -0.014119318686425686,
+            -0.0019422168843448162,
+            0.03133483603596687,
+            0.007720236200839281,
+            -0.008186287246644497,
+            0.013900477439165115,
+            -0.008996810764074326,
+            0.022078659385442734,
+            -0.01298458594828844,
+            -0.01027743797749281,
+            0.012887323275208473,
+            0.013738373294472694,
+            0.004384932108223438,
+            -1.1429647202021442e-05,
+            -0.005548033397644758,
+            0.00883470568805933,
+            -0.019922668114304543,
+            -0.001798348966985941,
+            0.012749534100294113,
+            -0.007079922594130039,
+            0.0036230399273335934,
+            0.017701832577586174,
+            0.013689741492271423,
+            -0.009766807779669762,
+            -0.00019199274538550526,
+            0.012076799757778645,
+            0.01583762839436531,
+            0.017020992934703827,
+            -0.01838267222046852,
+            0.0028874899726361036,
+            0.014751527458429337,
+            -0.02498033456504345,
+            -0.005803348030894995,
+            -0.012709008529782295,
+            0.0037000395823270082,
+            -0.036473557353019714,
+            0.00016045832308009267,
+            0.04493542015552521,
+            0.0035946716088801622,
+            0.0008135629468597472,
+            -0.029551686719059944,
+            -0.023359287530183792,
+            -0.005458875559270382,
+            -0.0011134566739201546,
+            -0.003997907042503357,
+            -0.019809193909168243,
+            -0.012263220734894276,
+            0.014702895656228065,
+            0.016631942242383957,
+            -0.018188146874308586,
+            -0.009620913304388523,
+            0.0013707978650927544,
+            -0.006192399188876152,
+            -0.005892505869269371,
+            0.0008166024344973266,
+            -0.004068827722221613,
+            -0.0010911672143265605,
+            0.01137974951416254,
+            0.015059526078402996,
+            0.007481131702661514,
+            0.016680574044585228,
+            0.011987642385065556,
+            -0.008048498071730137,
+            0.014176055788993835,
+            -0.003600750584155321,
+            -0.004555142018944025,
+            -0.004287669435143471,
+            -0.023570023477077484,
+            -0.0038540391251444817,
+            0.003408251330256462,
+            -0.014135529287159443,
+            0.008948178961873055,
+            0.00389861804433167,
+            0.003991828300058842,
+            -0.001961466856300831,
+            -0.010690804570913315,
+            0.0016960203647613525,
+            -0.02287297323346138,
+            -0.006504450924694538,
+            0.0027942797169089317,
+            -0.017815306782722473,
+            0.019306669011712074,
+            0.01922561787068844,
+            -0.0059532951563596725,
+            -0.016648152843117714,
+            0.014224686659872532,
+            0.001204640488140285,
+            0.015942996367812157,
+            0.015237841755151749,
+            0.000640313548501581,
+            0.0047739832662045956,
+            0.01204437855631113,
+            -0.0019857825245708227,
+            -0.021624766290187836,
+            -0.018301621079444885,
+            0.0053251395002007484,
+            -0.003165094181895256,
+            -0.018674461171030998,
+            0.0026443328242748976,
+            -0.0029118056409060955,
+            0.018155725672841072,
+            0.0022836499847471714,
+            0.006694924086332321,
+            -0.0030475682578980923,
+            -0.0218841340392828,
+            -0.0015795076033100486,
+            -0.004506510682404041,
+            -0.02853042632341385,
+            -0.00620455713942647,
+            -0.009888386353850365,
+            0.025466646999120712,
+            -0.021738240495324135,
+            0.015164894051849842,
+            -0.003852012800052762,
+            0.0042147221975028515,
+            0.02049003355205059,
+            0.008680706843733788,
+            0.002648385474458337,
+            0.012433430179953575,
+            -0.008591548539698124,
+            -0.00533324433490634,
+            0.01088532991707325,
+            -0.008461865596473217,
+            0.013495216146111488,
+            -0.024623703211545944,
+            0.01559447217732668,
+            -0.014694790355861187,
+            -0.005394033621996641,
+            0.010074807330965996,
+            0.013187216594815254,
+            0.0074325003661215305,
+            -0.01531078852713108,
+            -0.012376694008708,
+            0.022532552480697632,
+            -0.0077931834384799,
+            0.00331504107452929,
+            -0.003886460093781352,
+            -0.011250065639615059,
+            -0.01672920398414135,
+            -0.0006707081920467317,
+            0.013551952317357063,
+            -0.017020992934703827,
+            -0.00985596515238285,
+            -0.01595110259950161,
+            -0.0071690804325044155,
+            -0.04947435483336449,
+            -0.02369970642030239,
+            0.013025112450122833,
+            -0.009815439581871033,
+            -0.018706882372498512,
+            0.02614748664200306,
+            -0.006472030188888311,
+            -0.00591276865452528,
+            -0.008076866157352924,
+            -0.008283549919724464,
+            -0.014143634587526321,
+            0.01961466856300831,
+            0.010317963548004627,
+            -0.01365732029080391,
+            0.01922561787068844,
+            0.000424764963099733,
+            0.015051420778036118,
+            -0.012611744925379753,
+            -0.018204357475042343,
+            -0.0032542517874389887,
+            0.022743288427591324,
+            0.022030027583241463,
+            -0.009928912855684757,
+            0.0014619816793128848,
+            0.0021316767670214176,
+            0.018074674531817436,
+            0.01174448523670435,
+            -0.010739436373114586,
+            -0.010561120696365833,
+            0.013008901849389076,
+            -0.00985596515238285,
+            0.003858091775327921,
+            0.009661439806222916,
+            0.0042349849827587605,
+            0.002431570552289486,
+            -0.009037337265908718,
+            -0.001908782753162086,
+            0.006091083865612745,
+            0.0217706598341465,
+            -0.0007553065661340952,
+            -0.0061113471165299416,
+            -0.004867193289101124,
+            -0.009783018380403519,
+            -0.0025693594943732023,
+            -0.020862873643636703,
+            -0.013203427195549011,
+            -0.0023201233707368374,
+            0.018317829817533493,
+            0.004660509992390871,
+            -0.020798033103346825,
+            -0.02702285349369049,
+            -0.008988705463707447,
+            -0.023861810564994812,
+            -0.0055358754470944405,
+            -0.011250065639615059,
+            -0.008745548315346241,
+            0.008036340586841106,
+            0.009815439581871033,
+            -0.0017213491955772042,
+            -0.00199996680021286,
+            -0.006504450924694538,
+            0.005017140414565802,
+            -0.024947913363575935,
+            -0.0013272322248667479,
+            -0.0015896392287686467,
+            -0.021803081035614014,
+            -0.012733323499560356,
+            0.005916821304708719,
+            0.031059259548783302,
+            0.04224448278546333,
+            -0.008940073661506176,
+            0.0009270362206734717,
+            -0.0038216181565076113,
+            0.010674593970179558,
+            0.017847727984189987,
+            0.0090535469353199,
+            0.007019133307039738,
+            0.017345203086733818,
+            -0.03203188627958298,
+            0.0043200901709496975,
+            0.012392903678119183,
+            -0.0004067814734298736,
+            -0.007647288963198662,
+            -0.015489104203879833,
+            0.005000929813832045,
+            -0.008153866045176983,
+            0.005568296182900667,
+            0.012660376727581024,
+            -0.011039329692721367,
+            -0.00946691446006298,
+            0.0283034797757864,
+            0.009572282433509827,
+            0.011857958510518074,
+            0.01522973645478487,
+            -0.01407068781554699,
+            -0.022289395332336426,
+            0.0008231879328377545,
+            0.006549029611051083,
+            -0.0037446185015141964,
+            0.008575338870286942,
+            -0.010650279000401497,
+            -0.03595482185482979,
+            0.0002515155647415668,
+            -0.0006661489605903625,
+            -0.012336167506873608,
+            -0.016332048922777176,
+            0.0035055140033364296,
+            0.01932287961244583,
+            0.025904331356287003,
+            -0.01531889382749796,
+            0.017410043627023697,
+            0.007663499563932419,
+            -0.011720169335603714,
+            -0.009426387958228588,
+            -0.009572282433509827,
+            -0.015164894051849842,
+            -0.02037656120955944,
+            0.001938164234161377,
+            -0.014929842203855515,
+            -0.011833642609417439,
+            0.008234918117523193,
+            0.03263167664408684,
+            -0.0009467927156947553,
+            -0.004838825203478336,
+            -0.0011590485228225589,
+            0.0070839752443134785,
+            0.004518668632954359,
+            -0.010698909871280193,
+            0.01450837031006813,
+            0.019290458410978317,
+            0.010966382920742035,
+            0.001982743153348565,
+            -0.021235715597867966,
+            -0.008948178961873055,
+            0.017572149634361267,
+            -0.01223890483379364,
+            0.004575405269861221,
+            0.020198244601488113,
+            -0.005949242506176233,
+            -0.0069097126834094524,
+            0.0015339156379923224,
+            -0.007720236200839281,
+            -0.03268030658364296,
+            0.008315971121191978,
+            -0.007108291145414114,
+            0.011063645593822002,
+            0.0071528698317706585,
+            -0.019792983308434486,
+            0.003991828300058842,
+            -0.007233921904116869,
+            0.002139782067388296,
+            -0.0060789259150624275,
+            -0.0018024016171693802,
+            -0.00044198858086019754,
+            -0.01723172888159752,
+            0.024785807356238365,
+            -0.003612908534705639,
+            2.2194411940290593e-05,
+            0.0016372574027627707,
+            0.022532552480697632,
+            -0.0011428381549194455,
+            0.006334241013973951,
+            0.0033089620992541313,
+            -0.012725219130516052,
+            0.006654397584497929,
+            -0.009118389338254929,
+            -0.004393037408590317,
+            -0.00787423551082611,
+            0.012944060377776623,
+            0.013876162469387054,
+            0.02425086311995983,
+            0.00017109644250012934,
+            0.005527770146727562,
+            -0.019371511414647102,
+            -0.0035237509291619062,
+            -0.01553773507475853,
+            0.0031205154955387115,
+            0.010617857798933983,
+            0.0181070938706398,
+            -0.003770960494875908,
+            -0.004587562754750252,
+            -0.0010258187539875507,
+            0.009069757536053658,
+            0.016404995694756508,
+            -0.004064775072038174,
+            -0.022062448784708977,
+            0.0016382705653086305,
+            -0.004066801629960537,
+            0.009685755707323551,
+            0.022143501788377762,
+            -0.01634015329182148,
+            0.02032792940735817,
+            0.001971598481759429,
+            -0.015408051200211048,
+            0.013527637347579002,
+            -0.0074000791646540165,
+            -0.027233589440584183,
+            -0.020911505445837975,
+            0.0018753487383946776,
+            0.004879351239651442,
+            0.013033217750489712,
+            0.012433430179953575,
+            0.014840684831142426,
+            0.004725351929664612,
+            0.015610681846737862,
+            -0.0007117409259080887,
+            -0.0083078658208251,
+            -0.017085835337638855,
+            -0.0006367675377987325,
+            -0.007906656712293625,
+            -0.00822681374847889,
+            -0.038353972136974335,
+            -0.017426254227757454,
+            -0.007785078138113022,
+            0.02541801705956459,
+            0.013730267994105816,
+            0.005770927295088768,
+            -0.004571352619677782,
+            -0.01826919987797737,
+            -0.007290658541023731,
+            0.019484983757138252,
+            -0.0026564907748252153,
+            0.029308529570698738,
+            -0.006318030413240194,
+            0.00218841340392828,
+            0.003229936119168997,
+            -0.00817007664591074,
+            -0.020360350608825684,
+            0.022970234975218773,
+            0.01298458594828844,
+            0.007732394151389599,
+            0.014038266614079475,
+            -0.02010098285973072,
+            0.02404012717306614,
+            -0.014143634587526321,
+            -0.023164760321378708,
+            0.012668482027947903,
+            0.004534878768026829,
+            0.01961466856300831,
+            -0.009434493258595467,
+            0.005296770948916674,
+            0.015116263180971146,
+            -0.02287297323346138,
+            0.0263582244515419,
+            -0.005361612886190414,
+            0.023683495819568634,
+            -0.006390977650880814,
+            0.00852670706808567,
+            -0.015148683451116085,
+            0.005146824289113283,
+            -0.04454636946320534,
+            0.0020546771120280027,
+            -0.03024873696267605,
+            -8.561154390918091e-05,
+            0.017961200326681137,
+            -0.025612542405724525,
+            -0.0026504117995500565,
+            0.005211665760725737,
+            -0.011671538464725018,
+            -0.007456816267222166,
+            0.02254876308143139,
+            0.009783018380403519,
+            0.0008839771617203951,
+            -0.0003571369161363691,
+            0.011857958510518074,
+            -0.013770793564617634,
+            -0.024412967264652252,
+            -0.01838267222046852,
+            -0.017474886029958725,
+            -0.033750198781490326,
+            0.0003259824006818235,
+            0.00023669817892368883,
+            0.0045470367185771465,
+            -0.008915758691728115,
+            -0.004170143511146307,
+            -0.009710071608424187,
+            -0.0014579291455447674,
+            0.016048364341259003,
+            0.008198444731533527,
+            -0.014800158329308033,
+            -0.004506510682404041,
+            -0.006034347228705883,
+            -0.0016676520463079214,
+            0.00022276731033343822,
+            0.004769930616021156,
+            0.018852775916457176,
+            -0.011387854814529419,
+            -0.008591548539698124,
+            0.025612542405724525,
+            0.022954024374485016,
+            -0.016323942691087723,
+            0.00389861804433167,
+            -0.03751102834939957,
+            0.01865825057029724,
+            -0.02293781377375126,
+            -5.395046900957823e-05,
+            -0.012571219354867935,
+            0.022662237286567688,
+            -0.003939144313335419,
+            0.0035217246040701866,
+            0.013422268442809582,
+            -0.010034280829131603,
+            0.020895294845104218,
+            0.011922799982130527,
+            -0.0031326732132583857,
+            0.0026504117995500565,
+            -0.010674593970179558,
+            0.01356816291809082,
+            -0.02974621206521988,
+            0.00013918208424001932,
+            0.01893382892012596,
+            0.006285609677433968,
+            0.011314908042550087,
+            -0.015894364565610886,
+            0.004498405382037163,
+            0.0028814109973609447,
+            0.030491894111037254,
+            0.004680773243308067,
+            0.004103275015950203,
+            0.0035602243151515722,
+            0.0005020179669372737,
+            -0.029000530019402504,
+            0.009499335661530495,
+            0.007849919609725475,
+            -0.007699972949922085,
+            0.021576134487986565,
+            0.012133536860346794,
+            0.018366461619734764,
+            -0.006788134109228849,
+            -0.009742491878569126,
+            0.025661174207925797,
+            0.01744246482849121,
+            -0.006192399188876152,
+            0.0025146491825580597,
+            0.023683495819568634,
+            -0.0074000791646540165,
+            0.040234386920928955,
+            0.027655061334371567,
+            -0.009264283813536167,
+            -0.028611479327082634,
+            0.010836699046194553,
+            -0.01509194727987051,
+            -0.004131643567234278,
+            0.01922561787068844,
+            0.022678447887301445,
+            -0.011258170939981937,
+            0.01531889382749796,
+            0.015440472401678562,
+            0.012652271427214146,
+            0.01656709983944893,
+            -0.008996810764074326,
+            -0.0035338823217898607,
+            0.00047618255484849215,
+            -0.0013201400870457292,
+            -0.0002829233417287469,
+            -0.02608264610171318,
+            0.006151873152703047,
+            -0.0023748339153826237,
+            0.007193395867943764,
+            0.03007042035460472,
+            0.0073960269801318645,
+            -0.023359287530183792,
+            -0.0150108952075243,
+            -0.01998750865459442,
+            0.022662237286567688,
+            -0.022451501339673996,
+            -0.017345203086733818,
+            -0.00869691651314497,
+            -0.01260364055633545,
+            -0.032988306134939194,
+            0.009904596954584122,
+            0.00013740906433667988,
+            -0.005215718410909176,
+            0.0006104254862293601,
+            0.0013515478931367397,
+            -0.005491296760737896,
+            -0.0015268236165866256,
+            0.0016717046964913607,
+            0.008202497847378254,
+            0.010091017000377178,
+            0.016648152843117714,
+            -0.005365665536373854,
+            0.0166157316416502,
+            -0.012571219354867935,
+            -0.03738134354352951,
+            -0.0037000395823270082,
+            -0.003385961754247546,
+            0.017102045938372612,
+            0.0005400112713687122,
+            0.006480135023593903,
+            -0.019063511863350868,
+            0.003440672066062689,
+            0.00040146239916794,
+            0.005292718298733234,
+            0.01304132305085659,
+            -0.0013525610556825995,
+            -0.004474089480936527,
+            0.027071483433246613,
+            -0.004786141216754913,
+            0.007104238495230675,
+            -0.00986407045274973,
+            0.016648152843117714,
+            -0.016826467588543892,
+            -0.003088094526901841,
+            0.007027238607406616,
+            -0.010828593745827675,
+            -0.0022613604087382555,
+            -0.011509433388710022,
+            -0.017912568524479866,
+            -0.010212595574557781,
+            0.011395960114896297,
+            -0.020960137248039246,
+            -0.013106164522469044,
+            -0.0004678240220528096,
+            -0.007193395867943764,
+            0.008502391166985035,
+            0.01601594313979149,
+            0.005219771061092615,
+            -0.010763752274215221,
+            0.023229602724313736,
+            0.013284480199217796,
+            0.023732127621769905,
+            -0.006767870858311653,
+            -0.013989635743200779,
+            -0.004174196161329746,
+            -0.006301820278167725,
+            0.015983523800969124,
+            0.0070839752443134785,
+            0.005276507697999477,
+            -0.003495382610708475,
+            0.02160855568945408,
+            -0.009369651786983013,
+            0.0006732410402037203,
+            -0.002932068658992648,
+            0.010504384525120258,
+            -0.010974488221108913,
+            -0.009239967912435532,
+            0.015140579082071781,
+            0.026941800490021706,
+            0.0059006111696362495,
+            0.005673664156347513,
+            0.009021126665174961,
+            0.00020896308706142008,
+            0.009069757536053658,
+            0.0029158582910895348,
+            -0.006362609565258026,
+            -0.03465798497200012,
+            -0.00875365361571312,
+            -0.01005049142986536,
+            -0.019825404509902,
+            0.019582247361540794,
+            -0.0002821634989231825,
+            -0.009677650406956673,
+            -0.006638187449425459,
+            -0.00928049348294735,
+            0.004316037520766258,
+            0.0013900477206334472,
+            -0.000849529926199466,
+            -0.008429444395005703,
+            0.008003919385373592,
+            -0.023667285218834877,
+            -0.015027105808258057,
+            0.028708741068840027,
+            0.008915758691728115,
+            0.013292585499584675,
+            -0.008672601543366909,
+            -0.0021033084485679865,
+            0.006087031215429306,
+            -0.02442917786538601,
+            -0.00883470568805933,
+            0.06539303809404373,
+            -0.016242889687418938,
+            0.019695719704031944,
+            -0.024331914260983467,
+            -0.001989835174754262,
+            -0.002711201086640358,
+            0.018204357475042343,
+            0.021803081035614014,
+            -0.025515278801321983,
+            -0.0009123454801738262,
+            -0.02076561190187931,
+            0.0005531822680495679,
+            0.0067030293866992,
+            0.0026747274678200483,
+            -0.015124368481338024,
+            0.010350384749472141,
+            -0.03222641348838806,
+            -0.016794046387076378,
+            0.009726282209157944,
+            -0.0075378683395683765,
+            0.003035410540178418,
+            -0.016161838546395302,
+            -0.005811453331261873,
+            0.007541920989751816,
+            0.010690804570913315,
+            -0.006224820390343666,
+            -0.027849586680531502,
+            -0.02575843594968319,
+            -0.009013021364808083,
+            -0.02026308700442314,
+            0.017701832577586174,
+            -0.015780892223119736,
+            0.0166157316416502,
+            -0.009491230361163616,
+            -0.003108357544988394,
+            0.006982659921050072,
+            0.01689130999147892,
+            0.012538798153400421,
+            0.0017274281708523631,
+            -0.005787137895822525,
+            -0.008291655220091343,
+            0.01700478233397007,
+            -0.013633005321025848,
+            -0.0359872430562973,
+            0.0003687881981022656,
+            -0.005191402975469828,
+            0.0020789927802979946,
+            0.008769864216446877,
+            0.018285410478711128,
+            0.01590247079730034,
+            0.01531889382749796,
+            -0.015359420329332352,
+            -0.01379510946571827,
+            0.009118389338254929,
+            0.031205154955387115,
+            -0.001978690503165126,
+            0.03637629374861717,
+            0.011874169111251831,
+            0.004964456427842379,
+            -0.003852012800052762,
+            -0.008380812592804432,
+            -0.018285410478711128,
+            -0.003509566653519869,
+            0.02049003355205059,
+            -0.0155863668769598,
+            0.02381318062543869,
+            0.0010394963901489973,
+            -0.010609752498567104,
+            -0.001762888627126813,
+            -0.0011357460170984268,
+            -0.018285410478711128,
+            0.013503321446478367,
+            -0.01599973440170288,
+            0.0023768600076436996,
+            -0.016145627945661545,
+            0.001996927196159959,
+            -0.004855035804212093,
+            0.01933909021317959,
+            0.0019766641780734062,
+            -0.007371711079031229,
+            0.005357560236006975,
+            -0.01202006358653307,
+            0.002869253046810627,
+            -3.042629214178305e-05,
+            0.01638878509402275,
+            -0.0003361139679327607,
+            0.0010253122309222817,
+            -0.02026308700442314,
+            0.008259234018623829,
+            0.0022147553972899914,
+            -0.015562050975859165,
+            -0.01760457083582878,
+            0.007991761900484562,
+            0.002368754940107465,
+            0.002439675619825721,
+            0.006115399766713381,
+            0.003961433656513691,
+            -0.004393037408590317,
+            0.004676720593124628,
+            0.007464921101927757,
+            0.005389980971813202,
+            0.005081982351839542,
+            -0.013235848397016525,
+            0.002636227523908019,
+            0.0018642040668055415,
+            0.004311984870582819,
+            0.01771804317831993,
+            -0.0001176525474875234,
+            -0.004206616897135973,
+            0.00875365361571312,
+            0.009442598558962345,
+            0.008534812368452549,
+            -0.026682432740926743,
+            -0.018188146874308586,
+            0.012717113830149174,
+            -0.001926006400026381,
+            0.006168083753436804,
+            0.00018452072981745005,
+            0.018690671771764755,
+            0.005645296070724726,
+            0.004328195471316576,
+            0.006974554620683193,
+            0.008255181834101677,
+            -0.005134666338562965,
+            -0.0018571119289845228,
+            0.013511426746845245,
+            -0.012417219579219818,
+            0.010228806175291538,
+            -0.02901674062013626,
+            0.010326068848371506,
+            0.009167020209133625,
+            -0.0023647022899240255,
+            0.0014001793460920453,
+            -0.027849586680531502,
+            -0.005803348030894995,
+            -0.01572415605187416,
+            0.0059532951563596725,
+            -0.012068694457411766,
+            0.017572149634361267,
+            0.01783151738345623,
+            -0.018123304471373558,
+            0.04282806068658829,
+            0.0035602243151515722,
+            0.02674727514386177,
+            -0.009094073437154293,
+            -0.003929012455046177,
+            -0.00244372827000916,
+            0.008923863992094994,
+            -0.01063406839966774,
+            0.012571219354867935,
+            0.006804344709962606,
+            0.02065213769674301,
+            -0.00605055782943964,
+            0.011809326708316803,
+            0.00012461798905860633,
+            0.027849586680531502,
+            -0.006662502884864807,
+            0.013543847016990185,
+            -0.011663433164358139,
+            -0.024412967264652252,
+            -0.010366595350205898,
+            0.0037912235129624605,
+            0.006557134911417961,
+            0.005150876939296722,
+            0.008109287358820438,
+            0.001973624574020505,
+            0.009807334281504154,
+            0.007837762124836445,
+            0.04039648920297623,
+            -0.012700903229415417,
+            -0.010488173924386501,
+            0.011371644213795662,
+            -0.0032542517874389887,
+            0.0025572015438228846,
+            0.005037403665482998,
+            0.011120381765067577,
+            0.01143648661673069,
+            7.332704990403727e-05,
+            -0.014127423986792564,
+            -0.009150810539722443,
+            -0.009791123680770397,
+            0.002575438469648361,
+            -0.017134467139840126,
+            0.007254185155034065,
+            -0.001919927541166544,
+            -0.00815791916102171,
+            0.006727344822138548,
+            -0.006881344132125378,
+            0.0044781421311199665,
+            0.012417219579219818,
+            0.01223890483379364,
+            0.00765539426356554,
+            -0.014240897260606289,
+            0.00401614373549819,
+            0.004453826695680618,
+            0.022662237286567688,
+            -0.004830719903111458,
+            0.017766674980521202,
+            -0.020425191149115562,
+            -0.0043565635569393635,
+            0.010026175528764725,
+            -0.012660376727581024,
+            -0.012749534100294113,
+            0.0057790325954556465
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "53d2ebab-a323-4587-b152-4bf4aa8c59ce",
+      "type": "child",
+      "source": {
+        "id": "1e687808-8db8-4320-805d-0282a8d679bc",
+        "properties": {
+          "page_content": "Deepseekmoe: Towards Ultimate Expert Specialization In\n\nMixture-Of-Experts Language Models\n\nDamai Dai∗1,2, Chengqi Deng1, Chenggang Zhao∗1,3, R.X. Xu1, Huazuo Gao1, Deli Chen1, Jiashi Li1, Wangding Zeng1, Xingkai Yu∗1,4, Y. Wu1, Zhenda Xie1, Y.K. Li1, Panpan Huang1, Fuli Luo1, Chong Ruan1, Zhifang Sui2, Wenfeng Liang1 1DeepSeek-AI 2National Key Laboratory for Multimedia Information Processing, Peking University 3Institute for Interdisciplinary Information Sciences, Tsinghua University 4National Key Laboratory for Novel Software Technology, Nanjing University {daidamai, szf}@pku.edu.cn, {wenfeng.liang}@deepseek.com https://github.com/deepseek-ai/DeepSeek-MoE\n\nAbstract\n\nIn the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top- out of experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into ones and activating from them, allowing for a more flexible combination of activated experts; (2) isolating experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5× expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations.\n\nFurther, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\n1. Introduction\n\nRecent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023a). It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017) has emerged as a popular solution. It can\n\nenable parameter scaling, while concurrently keeping computational costs at a modest level.\n\nRecent applications of MoE architectures in Transformers (Vaswani et al., 2017) have yielded successful attempts at scaling language models to a substantial size (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022), accompanied with remarkable performance. These achievements underscore the considerable potential and promise of MoE language models.\n\nDespite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts.\n\nThis architecture manifests two potential issues: (1) Knowledge Hybridity: existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. (2) Knowledge Redundancy: tokens assigned to different experts may require common knowledge. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters.\n\nThese issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models.\n\nIn response to the aforementioned issues, we introduce DeepSeekMoE, an innovative MoE architecture specifically designed towards ultimate expert specialization. Our architecture involves two principal strategies: (1) Fine-Grained Expert Segmentation: while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. (2) Shared Expert Isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.\n\nThrough compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized.\n\nStarting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B (Lepikhin et al., 2021) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5× expert parameters and computation. Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE.\n\nThese studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization.\n\nLeveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B (DeepSeek-AI, 2024), a dense model trained on the same 2T corpus. We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B (Touvron et al., 2023b), which has approximately 2.5 times the activated parameters. Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard1. Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. The experimental results still validate its substantial advantages over the GShard architecture consistently. In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nOur contributions are summarized as follows: - Architectural Innovation. We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation.\n\nEmpirical Validation. We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. Experimental results validate the high 1https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models\n\nScalability. We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B.\n\nAlignment for MoE. We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B.\n\nPublic Release. In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization.\n\n2. Preliminaries: Mixture-Of-Experts For Transformers\n\nWe first introduce a generic MoE architecture commonly used in Transformer language models. A standard Transformer language model is constructed by stacking layers of standard Transformer blocks, where each block can be represented as follows:\n\n$$\\mathbf{u}{1:T}^{l}=\\text{Self-Att}\\left(\\mathbf{h}{1:T}^{l-1}\\right)+\\mathbf{h}{1:T}^{l-1},\\tag{1}$$ $$\\mathbf{h}{t}^{l}=\\text{FFN}\\left(\\mathbf{u}{t}^{l}\\right)+\\mathbf{u}{t}^{l},\\tag{2}$$\n\nwhere denotes the sequence length, Self-Att(·) denotes the self-attention module, FFN(·) denotes the Feed-Forward Network (FFN), u 1: ∈ R× are the hidden states of all tokens after the -th attention module, and h\n\n∈ Ris the output hidden state of the -th token after the -th Transformer block. For brevity, we omit the layer normalization in the above formulations.\n\nA typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022). An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. Then, each token will be assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts. If the -th FFN is substituted with an MoE layer, the computation for its output hidden state h\n\nis expressed as: where denotes the total number of experts, FFN(·) is the -th expert FFN, , denotes the gate value for the -th expert, , denotes the token-to-expert affinity, Topk(·, ) denotes the set comprising highest affinity scores among those calculated for the -th token and all experts, and e\n\nis the centroid of the -th expert in the -th layer. Note that ,is sparse, indicating that only out of gate values are nonzero. This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only experts. Also, in the above formulations, we omit the layer normalization operation for brevity.\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{N}\\left(g_{i,t}\\operatorname{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\operatorname{Topk}({s_{j,t}|1\\leqslant j\\leqslant N},K),\\ 0,&\\text{otherwise},\\end{cases}$$ $$s_{i,t}=\\operatorname{Softmax}{i}\\left(\\mathbf{u}{t}^{l}\\,\\mathbf{e}_{i}^{l}\\right),$$\n\n$$(3)$$\n\n(4) $\\binom{4}{5}$ .\n\n3. Deepseekmoe Architecture\n\nOn top of the generic MoE architecture outlined in Section 2, we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. As illustrated in Figure 2, our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. Both of these strategies are designed to elevate the level of expert specialization.\n\n3.1. Fine-Grained Expert Segmentation\n\nIn scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts.\n\nIn pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. The finer expert segmentation enables a more flexible and adaptable combination of activated experts. To be specific, on top of a typical MoE architecture shown in Figure 2(a), we segment each expert FFN into smaller experts by reducing the FFN intermediate hidden dimension to 1 times its original size. Since each expert becomes smaller, in response, we also increase the number of activated experts to times to keep the same computation cost, as illustrated in Figure 2(b). With the fine-grained expert segmentation, the output of an MoE layer can be expressed as:\n\n$$\\mathbf{h}{t}^{i}=\\sum{i=1}^{\\text{mN}}\\left(g_{i,t}\\text{FFN}{i}\\left(\\mathbf{u}{t}^{i}\\right)\\right)+\\mathbf{u}{t}^{i},\\tag{6}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\text{Tokr}(s_{i,t}|1\\leqslant j\\leqslant m\\text{N}),m\\text{K}),\\ 0,&\\text{otherwise},\\end{cases}$$ (7) $$s_{i,t}=\\text{Softmax}{i}\\left(\\mathbf{u}{t}^{T}\\mathbf{e}_{t}^{i}\\right),\\tag{8}$$ where the total number of expert parameters is equal to $N$ times the number of parameters in a\n\n$$(6)$$\n\n(7) (8) $\\frac{1}{2}$\n\nstandard FFN, and denotes the total number of fine-grained experts. With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to .\n\nFrom a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. As an illustrative example, we consider the case where = 16. A typical top-2 routing strategy can yield 16 2 = 120 possible combinations. By contrast, if each expert is split into 4 smaller experts, the fine-grained routing strategy can yield 64 8 = 4, 426, 165, 368 potential combinations. The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition.\n\n3.2. Shared Expert Isolation\n\nWith a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. This alleviation of redundancy will contribute to a more parameterefficient model with more specialized experts.\n\nTowards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate experts to serve as shared experts. Regardless of the router module, each token will be deterministically assigned to these shared experts. In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by , as depicted in Figure 2(c). With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows:\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{K_{s}}\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)+\\sum_{i=K_{s}+1}^{mN}\\left(g_{i,t}\\;\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},\\tag{9}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\mathrm{Topk}{s_{j,t}|K_{s}+1\\leqslant j\\leqslant mN},mK-K_{s}},\\ 0,&\\text{otherwise},\\end{cases}$$ (10) $$s_{i,t}=\\mathrm{Softmax}{i}\\left(\\mathbf{u}{t}^{l,T}e_{i}^{l}\\right).\\tag{11}$$ $\\mathbf{h}\\mathbf{M}\\mathbf{E}$ the number of closed-compact is $K_{s}$ the total number of routed currents.\n\nFinally, in DeepSeekMoE, the number of shared expert is , the total number of routed experts is − , and the number of nonzero gates is − .\n\nIt is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022). The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint.\n\n3.3. Load Balance Consideration\n\nAutomatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. Firstly, there is a risk of routing collapse (Shazeer et al., 2017), i.e., the model always selects only a few experts, preventing other experts from sufficient training.\n\nSecondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.\n\nExpert-Level Balance Loss. In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. The computation of the balance loss is as follows:\n\n$$\\begin{array}{l}{{{\\mathcal L}{\\mathrm{ExpBal}}=\\alpha{1}\\sum_{i=1}^{N^{\\prime}}f_{i}P_{i},}}\\ {{f_{i}=\\frac{N^{\\prime}}{K^{T}}\\sum_{t=1}^{T}\\mathds{1}\\,(\\mathrm{Token}\\;t\\;\\mathrm{s}\\mathrm{lects}\\;\\mathrm{Expert}\\;i),}}\\ {{P_{i}=\\frac{1}{T}\\sum_{t=1}^{T}s_{i,t},}}\\end{array}$$\n\n(12) $$\\begin{array}{l}\\small\\text{(13)}\\end{array}$$ = (14) $$\\begin{array}{l}\\small\\text{(14)}\\end{array}$$ .\n\n7 where 1 is a hyper-parameter called expert-level balance factor, ′is equal to ( − ) and ′ is equal to ( − ) for brevity. 1(·) denotes the indicator function.\n\nDevice-Level Balance Loss. In addition to the expert-level balance loss, we introduce a devicelevel balance loss. When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. Instead, our primary objective is to ensure balanced computation across the devices. If we partition all routed experts into groups {E1, E2, ..., E}, and deploy each group on a single device, the device-level balance loss is computed as follows:\n\n$$\\mathcal{L}{\\text{DevBal}}=\\alpha{2}\\sum_{i=1}^{D}f_{i}^{\\prime}P_{i}^{\\prime},$$ $$f_{i}^{\\prime}=\\frac{1}{|\\mathcal{E}{i}|}\\sum{j\\in\\mathcal{E}{i}}f{j},$$ $$P_{i}^{\\prime}=\\sum_{j\\in\\mathcal{E}{i}}P{j},$$\n\n(15) $$\\begin{array}{l}\\small\\text{(16)}\\end{array}$$ = (17) .\n\nwhere 2 is a hyper-parameter called device-level balance factor. In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices.\n\n4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n6. Alignment For Deepseekmoe 16B\n\nPrevious research indicates that MoE models typically do not emerge significant gains from fine-tuning (Artetxe et al., 2022; Fedus et al., 2021). However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B.\n\n6.1. Experimental Setup\n\nTraining Data. For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios.\n\nHyper-Parameters. During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).\n\nWe employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10−5 without incorporating any learning rate scheduling strategy.\n\nEvaluation Benchmarks. For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3, with the following adjustments: (1) We exclude Pile (Gao et al., 2020) since chat models are seldom employed for pure language modeling. (2) We exclude\n\nMetric # Shot LLaMA2 DeepSeek DeepSeekMoE SFT 7B Chat 7B Chat 16B # Total Params N/A 6.7B 6.9B 16.4B # Activated Params N/A 6.7B 6.9B 2.8B FLOPs per 4K Tokens N/A 187.9T 183.5T 74.4T HellaSwag (Acc.) 0-shot 67.9 71.0 72.2 PIQA (Acc.) 0-shot 76.9 78.4 79.7 ARC-easy (Acc.) 0-shot 69.7 70.2 69.9 ARC-challenge (Acc.) 0-shot 50.8 50.2 50.0 BBH (EM) 3-shot 39.3 43.1 42.2 RACE-middle (Acc.) 5-shot 63.9 66.1 64.8 RACE-high (Acc.) 5-shot 49.6 50.8 50.6 DROP (EM) 1-shot 40.0 41.7 33.8 GSM8K (EM) 0-shot 63.4 62.6 62.2 MATH (EM) 4-shot 13.5 14.7 15.2 HumanEval (Pass@1) 0-shot 35.4 45.1 45.7 MBPP (Pass@1) 3-shot 27.8 39.0 46.2 TriviaQA (EM) 5-shot 60.1 59.5 63.3 NaturalQuestions (EM) 0-shot 35.2 32.7 35.1 MMLU (Acc.) 0-shot 50.0 49.7 47.2 WinoGrande (Acc.) 0-shot 65.1 68.4 69.0 CLUEWSC (EM) 5-shot 48.4 66.2 68.2 CEval (Acc.) 0-shot 35.1 44.7 40.0 CMMLU (Acc.) 0-shot 36.9 51.2 49.3\n\nCHID (Zheng et al., 2019) due to the observed instability of results, hindering the derivation of solid conclusions. (3) We additionally include BBH (Suzgun et al., 2022) to provide a more comprehensive assessment of the reasoning ability of the chat models.\n\nTable 5 | Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations.\n\n6.2. Evaluations\n\nBaselines. In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. Correspondingly, we construct three chat models, including LLaMA2 SFT 7B3, DeepSeek Chat 7B, and DeepSeekMoE Chat 16B.\n\nSubsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks.\n\nResults. The evaluation results are presented in Table 5. Our key observations include: (1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intensive tasks (TriviaQA, NaturalQuestions). (2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. In addition, it also surpasses DeepSeek Chat 7B. (3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1). However, it is worth noting that, after supervised finetuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. (4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations.\n\n7. Deepseekmoe 145B Ongoing\n\nEncouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available.\n\n7.1. Experimental Setup\n\nTraining Data and Tokenization. For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study.\n\nModel Settings. For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps.\n\nSubsequently, the learning rate keeps constant during the remaining training process. The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. Also, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse.\n\nEvaluation Benchmarks. We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3).\n\n7.2. Evaluations\n\nBaselines. Apart from DeepSeekMoE 145B, we consider three additional models for comparison. DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters.\n\nResults. From the evaluation results presented in Table 6, we have the following observations: (1) Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. (2) On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. (3) At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5.\n\n8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\nAppendices A. Overview Of Hyper-Parameters\n\nWe present the overview of hyper-parameters for DeepSeekMoE across various sizes in Table 7.\n\n# Params # Layers Hidden # Attn # Shared # Routed Relative Sequence Batch Size Learning Size Heads Experts Experts Expert Size Length (Sequence) Rate 2.0B 9 1280 10 1 63 (7 activated) 0.25 2048 2048 1.08e-3 16.4B 28 2048 16 2 64 (6 activated) 0.25 4096 4608 4.2e-4 144.6B 62 4096 32 4 128 (12 activated) 0.125 4096 4608 3.0e-4\n\nTable 7 | Overview of hyper-parameters for DeepSeekMoE across various sizes. The relative expert size is in comparison to a standard FFN.\n\nB. Comparing Deepseekmoe With Larger Models\n\nComparisons among DeepSeekMoE, GShard×1.2, and GShard×1.5 are shown in Table 8. Comparisons among DeepSeekMoE, Dense×4, and Dense×16 are shown in Table 9.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 2.3B 2.8B 1.9B # Activated Expert Params N/A 0.28B 0.35B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.824 1.808 1.808 HellaSwag (Acc.) 0-shot 53.7 54.4 54.8 PIQA (Acc.) 0-shot 71.8 71.1 72.3 ARC-easy (Acc.) 0-shot 46.8 47.3 49.4 ARC-challenge (Acc.) 0-shot 31.7 34.1 34.3 RACE-middle (Acc.) 5-shot 43.7 46.4 44.0 RACE-high (Acc.) 5-shot 31.9 32.4 31.7 HumanEval (Pass@1) 0-shot 3.7 3.0 4.9 MBPP (Pass@1) 3-shot 2.4 2.6 2.2 TriviaQA (EM) 5-shot 15.2 15.7 16.6 NaturalQuestions (EM) 5-shot 4.5 4.7 5.7\n\nTable 8 | Comparison between DeepSeekMoE and larger GShard models.\n\nAt a larger scale of 13B total parameters, we also compare DeepSeekMoE with GShard×1.2 and GShard×1.5, and show results in Table 10. At a larger scale, DeepSeekMoE even outperforms GShard×1.5 distinctly.\n\nMetric # Shot Dense×4 Dense×16 DeepSeekMoE Relative Expert Size N/A 1 1 0.25 # Experts N/A 4 + 0 16 + 0 1 + 63 # Activated Experts N/A 4 + 0 16 + 0 1 + 7 # Total Expert Params N/A 0.47B 1.89B 1.89B # Activated Expert Params N/A 0.47B 1.89B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.908 1.806 1.808 HellaSwag (Acc.) 0-shot 47.6 55.1 54.8 PIQA (Acc.) 0-shot 70.0 71.9 72.3 ARC-easy (Acc.) 0-shot 43.9 51.9 49.4 ARC-challenge (Acc.) 0-shot 30.5 33.8 34.3 RACE-middle (Acc.) 5-shot 42.4 46.3 44.0 RACE-high (Acc.) 5-shot 30.7 33.0 31.7 HumanEval (Pass@1) 0-shot 1.8 4.3 4.9 MBPP (Pass@1) 3-shot 0.2 2.2 2.2 TriviaQA (EM) 5-shot 9.9 16.5 16.6 NaturalQuestions (EM) 5-shot 3.0 6.3 5.7\n\nTable 9 | Comparison between DeepSeekMoE and larger dense baselines.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 15.9B 19.8B 13.3B # Activated Expert Params N/A 2.37B 2.82B 2.05B # Training Tokens N/A 100B 100B 100B HellaSwag (Acc.) 0-shot 66.6 67.7 69.1 PIQA (Acc.) 0-shot 75.6 76.0 75.7 ARC-easy (Acc.) 0-shot 56.8 56.8 58.8 ARC-challenge (Acc.) 0-shot 39.9 37.6 38.5 RACE-middle (Acc.) 5-shot 51.6 50.6 52.4 RACE-high (Acc.) 5-shot 37.4 36.3 38.5 HumanEval (Pass@1) 0-shot 6.1 6.1 9.8 MBPP (Pass@1) 3-shot 7.0 11.6 10.6 TriviaQA (EM) 5-shot 36.5 36.7 38.2 NaturalQuestions (EM) 5-shot 12.6 12.1 13.7\n\nTable 10 | Comparison between DeepSeekMoE and larger GShard models at a larger scale.\n\nC. Training Benchmark Curves Of Deepseekmoe 16B\n\nWe present the benchmark curves during training of DeepSeekMoE 16B and DeepSeek 7B (Dense) in Figure 7 for reference.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/DeepSeekMoE_2.md"
+          },
+          "summary": "DeepSeekMoE is an innovative Mixture-of-Experts (MoE) architecture designed to enhance expert specialization in language models. It employs fine-grained expert segmentation and shared expert isolation to improve performance and reduce computational costs. Starting with a model of 2B parameters, DeepSeekMoE demonstrates comparable performance to larger models like GShard 2.9B and LLaMA2 7B, using significantly fewer computations. The architecture scales up to 16B and 145B parameters, maintaining advantages over traditional MoE architectures. DeepSeekMoE 16B achieves similar results to dense models with only 40% of the computations, and its chat model version shows adaptability in bilingual scenarios. The architecture's public release aims to contribute to advancements in large-scale language models.",
+          "headlines": {
+            "1. Introduction": [],
+            "2. Preliminaries: Mixture-Of-Experts For Transformers": [],
+            "3. Deepseekmoe Architecture": [
+              "3.1. Fine-Grained Expert Segmentation",
+              "3.2. Shared Expert Isolation",
+              "3.3. Load Balance Consideration"
+            ],
+            "4. Validation Experiments": [
+              "4.1. Experimental Setup",
+              "4.1.1. Training Data And Tokenization",
+              "4.1.2. Infrastructures",
+              "4.1.3. Hyper-Parameters",
+              "4.1.4. Evaluation Benchmarks",
+              "4.2. Evaluations",
+              "4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models",
+              "4.4. Ablation Studies",
+              "4.5. Analysis On Expert Specialization"
+            ],
+            "5. Scaling Up To Deepseekmoe 16B": [
+              "5.1. Experimental Setup",
+              "5.1.1. Training Data And Tokenization",
+              "5.1.2. Hyper-Parameters",
+              "5.1.3. Evaluation Benchmarks",
+              "5.2. Evaluations",
+              "5.2.1. Internal Comparison With Deepseek 7B",
+              "5.2.2. Comparison With Open Source Models"
+            ],
+            "6. Alignment For Deepseekmoe 16B": [
+              "6.1. Experimental Setup",
+              "6.2. Evaluations"
+            ],
+            "7. Deepseekmoe 145B Ongoing": [
+              "7.1. Experimental Setup",
+              "7.2. Evaluations"
+            ],
+            "8. Related Work": [],
+            "9. Conclusion": [],
+            "Appendices": [
+              "A. Overview Of Hyper-Parameters",
+              "B. Comparing Deepseekmoe With Larger Models",
+              "C. Training Benchmark Curves Of Deepseekmoe 16B"
+            ]
+          },
+          "summary_embedding": [
+            -0.012465757317841053,
+            -0.022808987647294998,
+            -0.018792515620589256,
+            -0.02308654971420765,
+            0.01797616109251976,
+            0.007200260180979967,
+            0.006490030791610479,
+            0.042613785713911057,
+            -0.014049487188458443,
+            0.03187054023146629,
+            -0.021486490964889526,
+            -0.023592690005898476,
+            0.009045224636793137,
+            -0.014441338367760181,
+            0.005865518003702164,
+            -0.007363531738519669,
+            -0.008743173442780972,
+            -0.005163452122360468,
+            0.018841497600078583,
+            -0.03100520372390747,
+            0.011110605672001839,
+            -0.046075135469436646,
+            -0.020017050206661224,
+            -0.021845687180757523,
+            0.016425084322690964,
+            0.037389107048511505,
+            0.022547753527760506,
+            -0.0028062239289283752,
+            -0.06844329088926315,
+            0.034678805619478226,
+            0.035168617963790894,
+            0.02052319049835205,
+            0.0005438972148112953,
+            -0.00031302153365686536,
+            -0.024833550676703453,
+            -0.028556134551763535,
+            0.03347059711813927,
+            -0.004379750229418278,
+            -0.01453930139541626,
+            0.015976088121533394,
+            0.0414055772125721,
+            0.012106560170650482,
+            -0.02754385396838188,
+            0.0014245413476601243,
+            0.015151568688452244,
+            0.0055961208418011665,
+            -0.01678428053855896,
+            -0.0022960014175623655,
+            -0.002424577483907342,
+            -0.021225256845355034,
+            -0.01400050614029169,
+            0.018825169652700424,
+            0.001296985661610961,
+            -0.008865626528859138,
+            -0.0542386956512928,
+            0.007330877240747213,
+            0.03840138763189316,
+            0.05623060464859009,
+            0.044148534536361694,
+            -0.020180322229862213,
+            0.03369917720556259,
+            -0.030025575309991837,
+            -0.02501314878463745,
+            0.011028970591723919,
+            0.003983817528933287,
+            -0.018531281501054764,
+            0.01784554310142994,
+            -0.005636938381940126,
+            -0.0006556359585374594,
+            0.01751900091767311,
+            -0.03941367194056511,
+            0.020751770585775375,
+            -0.01887415163218975,
+            0.031266435980796814,
+            0.02173139713704586,
+            0.0073798587545752525,
+            0.01872720755636692,
+            0.01678428053855896,
+            0.0036858476232737303,
+            -0.04107903689146042,
+            0.009118696674704552,
+            0.016523046419024467,
+            0.014196431264281273,
+            0.016269976273179054,
+            0.06553706526756287,
+            -0.05391215533018112,
+            -0.004485876299440861,
+            -0.05169166624546051,
+            0.0012561678886413574,
+            -0.004971608519554138,
+            -0.03670336678624153,
+            0.01048201136291027,
+            0.028425518423318863,
+            0.021372200921177864,
+            -0.01063711941242218,
+            0.024392718449234962,
+            -0.022743679583072662,
+            0.0032001156359910965,
+            -0.04062187671661377,
+            0.0019653772469609976,
+            0.004518530797213316,
+            0.016531210392713547,
+            0.0070778070949018,
+            0.004273623693734407,
+            -0.016702644526958466,
+            0.006045116577297449,
+            -0.01905374974012375,
+            0.03291547670960426,
+            -0.008465612307190895,
+            0.08653374016284943,
+            0.008751336485147476,
+            0.004738946910947561,
+            0.004281787667423487,
+            0.02398454025387764,
+            0.012131051160395145,
+            -0.00810233410447836,
+            -0.04718538001179695,
+            -0.015690363943576813,
+            0.011061624623835087,
+            -0.0383034273982048,
+            0.018335357308387756,
+            -0.008759500458836555,
+            0.01282495353370905,
+            -0.04107903689146042,
+            -0.06344719231128693,
+            0.01498013362288475,
+            0.001784758409485221,
+            -0.004065453074872494,
+            -0.0002903166168835014,
+            -0.018923133611679077,
+            0.003930754493921995,
+            -0.004200151655822992,
+            -0.010645282454788685,
+            -0.021502818912267685,
+            0.03159298002719879,
+            0.03657275065779686,
+            0.06527583301067352,
+            0.036801330745220184,
+            -0.0030164355412125587,
+            0.005775718949735165,
+            0.019690508022904396,
+            -0.020784424617886543,
+            0.003481758525595069,
+            -0.01214737817645073,
+            -0.021992631256580353,
+            -0.03820546343922615,
+            0.010971825569868088,
+            0.07379858940839767,
+            -0.029029620811343193,
+            0.013437220826745033,
+            -0.017616963014006615,
+            0.04205866530537605,
+            -0.004567512311041355,
+            0.026727497577667236,
+            -0.016669990494847298,
+            0.004059330560266972,
+            -0.025388672947883606,
+            0.01620466820895672,
+            -0.024964168667793274,
+            0.012571883387863636,
+            -0.000885746325366199,
+            -0.04222193360328674,
+            0.00011116473615402356,
+            0.026890767738223076,
+            -0.010971825569868088,
+            0.02261306345462799,
+            0.0004949158756062388,
+            -0.006302268709987402,
+            -0.05097327008843422,
+            0.038727931678295135,
+            -0.003912386484444141,
+            0.018645571544766426,
+            -0.0018959869630634785,
+            0.0163107942789793,
+            -0.0016408757073804736,
+            0.013216804713010788,
+            0.009543201886117458,
+            0.007114543113857508,
+            -0.033503253012895584,
+            0.02697240374982357,
+            -0.02504580281674862,
+            0.028049994260072708,
+            0.03435226157307625,
+            -0.0030184765346348286,
+            0.03255628049373627,
+            0.019249675795435905,
+            0.03840138763189316,
+            0.0327032245695591,
+            0.030335791409015656,
+            -0.00824927818030119,
+            0.013878053054213524,
+            0.008200296200811863,
+            0.03608293831348419,
+            0.013429056853055954,
+            0.019723162055015564,
+            -0.009730963967740536,
+            -0.014041324146091938,
+            0.048785436898469925,
+            0.007318632211536169,
+            -0.0179598331451416,
+            -0.005441013257950544,
+            -0.03510330989956856,
+            -0.002832755446434021,
+            -0.014425011351704597,
+            0.0323440283536911,
+            0.02158445306122303,
+            -0.01647406443953514,
+            0.035135965794324875,
+            0.024980494752526283,
+            -0.03751972317695618,
+            -0.005232842173427343,
+            -0.02026195637881756,
+            -0.023298801854252815,
+            -0.04215662553906441,
+            0.011624909937381744,
+            0.012833117507398129,
+            -0.013976015150547028,
+            -0.007469657808542252,
+            0.020457882434129715,
+            -0.023625344038009644,
+            -0.0018265966791659594,
+            -0.01356783788651228,
+            0.006547175347805023,
+            -0.020898714661598206,
+            -0.03667071461677551,
+            -0.01048201136291027,
+            0.025862159207463264,
+            0.012637191452085972,
+            0.04362606629729271,
+            -0.013518855907022953,
+            -0.005367541220039129,
+            0.033193036913871765,
+            0.012759645469486713,
+            0.010767736472189426,
+            0.03699725493788719,
+            -0.03027048148214817,
+            0.024621298536658287,
+            -0.020294612273573875,
+            -0.0032878739293664694,
+            0.01707816869020462,
+            -0.0030062312725931406,
+            0.01962519995868206,
+            -0.04535674303770065,
+            -0.024196794256567955,
+            0.04395261034369469,
+            -0.0007107399869710207,
+            0.016302630305290222,
+            -0.005959399044513702,
+            -0.012106560170650482,
+            0.0035899258218705654,
+            0.0027715289033949375,
+            -0.01438419334590435,
+            -0.008620720356702805,
+            0.007473739795386791,
+            -0.01962519995868206,
+            0.03660540655255318,
+            0.03794422745704651,
+            -0.004132802598178387,
+            -0.014131123200058937,
+            -0.03324201703071594,
+            0.010016689077019691,
+            -0.017878197133541107,
+            -0.023951886221766472,
+            0.04078514873981476,
+            -0.032115448266267776,
+            -0.00029312283731997013,
+            0.039478980004787445,
+            0.018808843567967415,
+            -0.014898497611284256,
+            0.00868602842092514,
+            -0.01709449477493763,
+            -0.026302991434931755,
+            -0.017159804701805115,
+            0.03356856107711792,
+            0.024800896644592285,
+            -0.010743245482444763,
+            0.010245268233120441,
+            0.005330805201083422,
+            0.02893165871500969,
+            0.024376392364501953,
+            0.00648594880476594,
+            -0.013812744058668613,
+            0.015029114671051502,
+            0.027641816064715385,
+            -0.0028021421749144793,
+            -7.66609373386018e-05,
+            -0.004010349046438932,
+            -0.007796200457960367,
+            -0.006453294772654772,
+            -0.013069860637187958,
+            -0.005089980084449053,
+            0.0009184005321003497,
+            -0.032295044511556625,
+            0.015967924147844315,
+            -0.005890008993446827,
+            0.010359558276832104,
+            -0.011535110883414745,
+            0.01288209855556488,
+            0.01722511276602745,
+            -0.013494365848600864,
+            -0.0716434046626091,
+            0.014612773433327675,
+            0.02458864450454712,
+            -0.006706364918500185,
+            -0.017943505197763443,
+            0.008481939323246479,
+            0.00016148542636074126,
+            0.038597315549850464,
+            -0.014180104248225689,
+            -0.031103165820240974,
+            0.02680913358926773,
+            -0.017437364906072617,
+            -0.015241367742419243,
+            -0.010661610402166843,
+            -0.00580837344750762,
+            0.04649963974952698,
+            0.01889047957956791,
+            0.011420820839703083,
+            -0.04042595252394676,
+            -0.007449248805642128,
+            0.015078096650540829,
+            0.014506647363305092,
+            -0.04705476388335228,
+            0.0052083516493439674,
+            0.015486273914575577,
+            0.019723162055015564,
+            0.024800896644592285,
+            0.04274440184235573,
+            -0.0021531390957534313,
+            0.01661284640431404,
+            -0.015053605660796165,
+            -0.05136512219905853,
+            -0.03340528905391693,
+            0.029943939298391342,
+            -0.017731253057718277,
+            -0.02906227484345436,
+            -0.01693122461438179,
+            -0.006151242647320032,
+            0.0017939424142241478,
+            0.007126788143068552,
+            -0.01916803978383541,
+            -0.03464614972472191,
+            -0.03970756009221077,
+            0.02277633361518383,
+            0.007983962073922157,
+            0.004269542172551155,
+            -0.042777057737112045,
+            0.0013459670590236783,
+            0.009184005670249462,
+            0.012073906138539314,
+            0.0009954441338777542,
+            -0.019984396174550056,
+            0.015494437888264656,
+            0.0032001156359910965,
+            -0.031070511788129807,
+            -0.035593122243881226,
+            -0.03595232218503952,
+            0.0023919232189655304,
+            0.003781769424676895,
+            0.06279410421848297,
+            0.012988224625587463,
+            -0.03840138763189316,
+            -0.04522612690925598,
+            0.029176564887166023,
+            -0.02052319049835205,
+            -0.016678154468536377,
+            0.0059430720284581184,
+            0.012898425571620464,
+            -0.009877908043563366,
+            0.013192313723266125,
+            0.023560035973787308,
+            0.041601505130529404,
+            -0.005987971555441618,
+            0.012914752587676048,
+            0.023037567734718323,
+            0.012955570593476295,
+            0.0362788625061512,
+            -0.00936360377818346,
+            0.0035409443080425262,
+            0.039936136454343796,
+            -0.012269831262528896,
+            0.03175625205039978,
+            -0.02189466916024685,
+            0.0020398697815835476,
+            0.02249877341091633,
+            0.027184655889868736,
+            -0.0006479826406575739,
+            0.02594379521906376,
+            -0.031543996185064316,
+            0.018368011340498924,
+            -0.014498483389616013,
+            -0.04888340085744858,
+            -0.01557607389986515,
+            0.013102514669299126,
+            0.030384771525859833,
+            0.02756018005311489,
+            -0.0041348435916006565,
+            -0.019723162055015564,
+            -0.01155143789947033,
+            -0.010596301406621933,
+            0.0018419033149257302,
+            0.03285016864538193,
+            0.010506502352654934,
+            0.010890189558267593,
+            -0.011935125105082989,
+            -0.004104230087250471,
+            0.03025415539741516,
+            -0.021192602813243866,
+            0.01650671847164631,
+            0.0041511706076562405,
+            -0.002029665280133486,
+            -0.04320156201720238,
+            -0.035266581922769547,
+            0.007845181971788406,
+            0.011747363954782486,
+            0.014874007552862167,
+            0.0368339866399765,
+            0.025094784796237946,
+            -0.01662100851535797,
+            0.030629679560661316,
+            -0.01905374974012375,
+            0.027984686195850372,
+            0.039054471999406815,
+            0.011624909937381744,
+            -0.018351683393120766,
+            0.012237177230417728,
+            0.002424577483907342,
+            -0.025127438828349113,
+            0.01680060662329197,
+            -0.032735876739025116,
+            0.01738838478922844,
+            -0.02354370802640915,
+            0.025143766775727272,
+            -0.02579685114324093,
+            -0.009290131740272045,
+            -0.004428731743246317,
+            -0.006130834110081196,
+            0.04888340085744858,
+            -0.026727497577667236,
+            -0.0142454132437706,
+            -0.0024694770108908415,
+            -0.0024531499948352575,
+            -0.008718682453036308,
+            0.010000362060964108,
+            -0.045781247317790985,
+            -0.01844964735209942,
+            -0.04016471654176712,
+            0.007355368230491877,
+            0.005412440747022629,
+            -0.030907239764928818,
+            -0.05694899708032608,
+            0.025421326979994774,
+            -0.06331657618284225,
+            -0.019886434078216553,
+            0.051626358181238174,
+            -0.018400665372610092,
+            -0.008212542161345482,
+            0.009641164913773537,
+            0.006375740747898817,
+            -0.029290854930877686,
+            0.0041348435916006565,
+            0.02504580281674862,
+            0.04418119043111801,
+            -0.0045797573402523994,
+            -0.02865409664809704,
+            0.0028674507047981024,
+            -0.027788760140538216,
+            -0.013461710885167122,
+            0.005261414684355259,
+            0.016449574381113052,
+            0.024539662525057793,
+            0.007628847379237413,
+            0.03585435822606087,
+            0.007294141221791506,
+            0.006318595726042986,
+            0.044279150664806366,
+            -0.025666235014796257,
+            -0.027380581945180893,
+            0.02519274689257145,
+            0.04045860469341278,
+            0.023151857778429985,
+            -0.018253721296787262,
+            -0.0142454132437706,
+            0.0053063142113387585,
+            0.0009296254720538855,
+            -0.022139575332403183,
+            0.002808264922350645,
+            -0.013902543112635612,
+            0.005143043119460344,
+            0.03892385587096214,
+            -0.001205145614221692,
+            0.014196431264281273,
+            0.007302304729819298,
+            0.006845145486295223,
+            0.004334850702434778,
+            -0.011731036007404327,
+            -0.023592690005898476,
+            0.00032220553839579225,
+            0.034972693771123886,
+            0.015404638834297657,
+            -0.030515389516949654,
+            -0.00995954405516386,
+            0.034711457788944244,
+            -0.00795130804181099,
+            -0.009118696674704552,
+            -0.009771781973540783,
+            0.003385836724191904,
+            0.02638462744653225,
+            -0.01606588624417782,
+            0.006649219896644354,
+            0.03716052696108818,
+            -0.019870106130838394,
+            0.005543057806789875,
+            -0.025405000895261765,
+            0.0018061877926811576,
+            0.02442537248134613,
+            -0.006216551177203655,
+            0.028098974376916885,
+            5.6315813708351925e-05,
+            0.03025415539741516,
+            -0.004677720367908478,
+            0.008432958275079727,
+            -0.020327266305685043,
+            0.03987082839012146,
+            -0.017633290961384773,
+            0.03402572125196457,
+            0.02067013457417488,
+            -0.01258004643023014,
+            0.04783846437931061,
+            -0.036801330745220184,
+            -0.020784424617886543,
+            0.00584919098764658,
+            0.012743317522108555,
+            0.0028919412288814783,
+            -0.0191517136991024,
+            -0.018400665372610092,
+            -0.010996315628290176,
+            0.02380494214594364,
+            -0.022547753527760506,
+            -0.016865916550159454,
+            -0.007441085297614336,
+            -0.01890680566430092,
+            -0.009673818945884705,
+            -0.007988044060766697,
+            -0.01709449477493763,
+            -0.016547536477446556,
+            0.044573038816452026,
+            -0.06334923207759857,
+            0.016996532678604126,
+            0.017159804701805115,
+            0.011477965861558914,
+            -0.034678805619478226,
+            -0.019510909914970398,
+            -0.017633290961384773,
+            -0.002347023691982031,
+            -0.030743969604372978,
+            0.001702102366834879,
+            0.01725776679813862,
+            0.01138000376522541,
+            -0.05022222548723221,
+            0.012335140258073807,
+            0.02772345207631588,
+            -0.012065742164850235,
+            0.0014949520118534565,
+            -0.023135529831051826,
+            -0.017176130786538124,
+            0.0031776658724993467,
+            0.022123249247670174,
+            0.005975726060569286,
+            -0.008188051171600819,
+            -0.016865916550159454,
+            -0.005241005681455135,
+            -0.006045116577297449,
+            -0.002359268954023719,
+            0.005094061605632305,
+            0.011780017986893654,
+            0.019690508022904396,
+            -0.0024633544962853193,
+            -0.004436895251274109,
+            -0.006396149750798941,
+            0.027364253997802734,
+            -0.001450052484869957,
+            -0.008416631259024143,
+            0.050907962024211884,
+            0.03556047007441521,
+            0.006094098091125488,
+            -0.01108611561357975,
+            -0.03990348428487778,
+            0.010661610402166843,
+            -0.022711025550961494,
+            0.0401320643723011,
+            0.01905374974012375,
+            -0.008298259228467941,
+            -0.028866350650787354,
+            0.010294250212609768,
+            -0.02290695160627365,
+            -0.009975871071219444,
+            0.009469729848206043,
+            0.00726556871086359,
+            -0.03263791650533676,
+            -0.013543346896767616,
+            0.03510330989956856,
+            0.011437147855758667,
+            -0.006841063499450684,
+            -0.036050282418727875,
+            0.008588065393269062,
+            0.01646590232849121,
+            0.011869817040860653,
+            -0.019233347848057747,
+            0.004449140280485153,
+            -0.012041252106428146,
+            0.017470018938183784,
+            0.008849299512803555,
+            -0.017796561121940613,
+            0.018237393349409103,
+            0.004710374400019646,
+            -0.008490103296935558,
+            -0.003912386484444141,
+            -0.0163107942789793,
+            -0.02006603218615055,
+            0.03840138763189316,
+            0.024490682408213615,
+            0.026335645467042923,
+            0.005457340274006128,
+            0.02008235827088356,
+            0.03227871656417847,
+            0.011282040737569332,
+            -0.03134807199239731,
+            0.027935704216361046,
+            0.016996532678604126,
+            -0.04604247957468033,
+            0.021551799029111862,
+            -0.02218855731189251,
+            -0.028670424595475197,
+            0.009649328887462616,
+            -0.0017367975087836385,
+            0.011192241683602333,
+            0.0034837995190173388,
+            -0.003642988856881857,
+            -0.03307874873280525,
+            -0.006236960180103779,
+            0.015771998092532158,
+            0.015502601861953735,
+            -0.0048695639707148075,
+            -0.003296037670224905,
+            -0.011886144056916237,
+            0.00487364549189806,
+            0.009796272963285446,
+            0.039054471999406815,
+            0.001381682581268251,
+            -0.004751192405819893,
+            -0.0018765984568744898,
+            0.031854212284088135,
+            0.0271193478256464,
+            -0.017274094745516777,
+            -0.026286665350198746,
+            0.00580837344750762,
+            -0.014253576286137104,
+            0.026156047359108925,
+            -0.00487364549189806,
+            0.011877980083227158,
+            0.008718682453036308,
+            -0.0007908449042588472,
+            -0.001373519073240459,
+            0.025584599003195763,
+            -0.02710302174091339,
+            -0.021764053031802177,
+            0.031233781948685646,
+            -0.025731543079018593,
+            0.028735732659697533,
+            0.013061696663498878,
+            -0.01557607389986515,
+            -0.015225040726363659,
+            -0.010302413254976273,
+            0.009943217039108276,
+            -0.012277995236217976,
+            0.023853924125432968,
+            0.003608293831348419,
+            0.006392067763954401,
+            0.0003584313380997628,
+            0.02770712412893772,
+            0.024833550676703453,
+            0.007820690982043743,
+            -0.018074123188853264,
+            -0.0016816934803500772,
+            -0.02320083975791931,
+            -0.01680060662329197,
+            0.023772288113832474,
+            -0.018090449273586273,
+            -0.03268689662218094,
+            -0.028556134551763535,
+            0.012114724144339561,
+            0.007392104249447584,
+            0.010816717520356178,
+            0.03461349755525589,
+            -0.030384771525859833,
+            -0.0344502255320549,
+            0.01766594499349594,
+            0.026270337402820587,
+            -0.005551221314817667,
+            -0.021535472944378853,
+            0.03085825778543949,
+            -0.005828781984746456,
+            0.017600636929273605,
+            0.0404912605881691,
+            -0.0016225076979026198,
+            0.010228941217064857,
+            -0.042026009410619736,
+            -0.014057651162147522,
+            -0.0066614653915166855,
+            0.006261450704187155,
+            0.034417569637298584,
+            0.01048201136291027,
+            0.01535565685480833,
+            0.03448288142681122,
+            -0.021192602813243866,
+            -0.0002903166168835014,
+            -0.0022449791431427,
+            -0.02818061038851738,
+            0.03347059711813927,
+            0.025715215131640434,
+            0.011918798089027405,
+            0.019706835970282555,
+            -0.03794422745704651,
+            -0.013037205673754215,
+            0.027070365846157074,
+            0.015053605660796165,
+            0.01273515447974205,
+            0.016694480553269386,
+            0.011249386705458164,
+            0.00868602842092514,
+            -0.021633435040712357,
+            -0.05169166624546051,
+            0.01214737817645073,
+            0.01993541419506073,
+            -0.016449574381113052,
+            0.01057181041687727,
+            0.024980494752526283,
+            0.007620683871209621,
+            -0.007045152597129345,
+            0.012808626517653465,
+            -0.02682545967400074,
+            0.002259265398606658,
+            0.0019010890973731875,
+            -0.014482156373560429,
+            0.038270771503448486,
+            -0.016367938369512558,
+            0.011363675817847252,
+            -0.018204739317297935,
+            0.023870250210165977,
+            0.00011531029304023832,
+            0.0007964573451317847,
+            0.014963806606829166,
+            -0.011347348801791668,
+            -0.04235255345702171,
+            0.0021613026037812233,
+            0.03342161700129509,
+            -0.015200549736618996,
+            0.026449935510754585,
+            -0.0020674217958003283,
+            0.032458316534757614,
+            0.042777057737112045,
+            0.017731253057718277,
+            0.0003497575526125729,
+            0.025682561099529266,
+            0.0036144163459539413,
+            0.0073880222626030445,
+            0.012400448322296143,
+            0.009322785772383213,
+            0.007245160173624754,
+            0.030809277668595314,
+            -0.010261595249176025,
+            0.02173139713704586,
+            -0.04760988429188728,
+            -0.014629100449383259,
+            0.012024925090372562,
+            -0.0013316808035597205,
+            -0.012449429370462894,
+            -0.033029764890670776,
+            0.012563719414174557,
+            -0.03503800183534622,
+            0.022074267268180847,
+            -0.0004959363141097128,
+            0.04333217814564705,
+            -0.02187834307551384,
+            -0.005518566817045212,
+            0.016694480553269386,
+            0.0016510800924152136,
+            0.023282473906874657,
+            -0.0012153501156717539,
+            -0.016416920349001884,
+            -0.0067798369564116,
+            -0.009257477708160877,
+            0.004138925112783909,
+            0.04375668615102768,
+            -0.003722583642229438,
+            -0.0001682458823779598,
+            0.01409030519425869,
+            -0.011959616094827652,
+            -0.03250729665160179,
+            0.045193471014499664,
+            -0.0030837850645184517,
+            0.02484987862408161,
+            0.04979771748185158,
+            -0.06410028040409088,
+            -0.019543563947081566,
+            0.013061696663498878,
+            -0.010441194288432598,
+            0.011363675817847252,
+            0.005547139327973127,
+            0.016114868223667145,
+            0.017894525080919266,
+            -0.005485912784934044,
+            0.01619650423526764,
+            0.005106307100504637,
+            0.028098974376916885,
+            -0.02336410991847515,
+            -0.03296445682644844,
+            -0.02636830136179924,
+            -0.017747581005096436,
+            0.0014480114914476871,
+            -0.015796490013599396,
+            -3.563585551091819e-06,
+            -0.0022204883862286806,
+            -0.04388730227947235,
+            -0.025274382904171944,
+            0.040687184780836105,
+            0.006706364918500185,
+            0.02458864450454712,
+            0.0077268099412322044,
+            0.03418898954987526,
+            0.005885927006602287,
+            -0.009037061594426632,
+            0.030058229342103004,
+            -0.020049704238772392,
+            0.03833607956767082,
+            0.022106921300292015,
+            -0.002832755446434021,
+            0.005865518003702164,
+            0.031086837872862816,
+            -0.033797141164541245,
+            0.008086007088422775,
+            -0.034384917467832565,
+            0.0046695563942193985,
+            -0.008620720356702805,
+            -0.015094423666596413,
+            -0.009526874870061874,
+            -0.02504580281674862,
+            0.011216731742024422,
+            -0.021535472944378853,
+            0.01590261608362198,
+            0.010416703298687935,
+            -0.002916431985795498,
+            0.03448288142681122,
+            -0.014204595237970352,
+            -0.0027796924114227295,
+            0.0024531499948352575,
+            -0.027070365846157074,
+            -0.006343086715787649,
+            -0.003377673216164112,
+            -0.03689929470419884,
+            -0.004502203781157732,
+            0.010988152585923672,
+            -0.00024184548237826675,
+            0.016898570582270622,
+            0.02787039615213871,
+            0.0005092020728625357,
+            -0.022302847355604172,
+            0.002340900944545865,
+            -0.04199335351586342,
+            0.0011877981014549732,
+            0.01483318954706192,
+            -0.017600636929273605,
+            -0.008612556383013725,
+            0.033046092838048935,
+            0.023298801854252815,
+            -0.009396257810294628,
+            -0.035005345940589905,
+            0.017796561121940613,
+            -0.011853490024805069,
+            0.027511198073625565,
+            -0.0002630197268445045,
+            0.019298657774925232,
+            -0.0076941559091210365,
+            0.04493223875761032,
+            0.001369437319226563,
+            0.008645210415124893,
+            -0.021959977224469185,
+            0.007020662073045969,
+            -0.020278284326195717,
+            -0.002406209474429488,
+            -0.006171651650220156,
+            -0.012751481495797634,
+            -0.010669773444533348,
+            0.0032735876739025116,
+            0.007730891928076744,
+            -0.0176822729408741,
+            0.0055961208418011665,
+            -0.015959760174155235,
+            -0.02741323597729206,
+            0.007543129846453667,
+            -0.05965929850935936,
+            -0.0179598331451416,
+            -0.03373183310031891,
+            -0.009477893821895123,
+            0.02008235827088356,
+            0.03510330989956856,
+            -0.012759645469486713,
+            0.00497977202758193,
+            -0.014898497611284256,
+            0.004477712791413069,
+            0.03490738570690155,
+            0.044442422688007355,
+            -0.014759717509150505,
+            0.003381754970178008,
+            -0.010955498553812504,
+            0.033029764890670776,
+            -0.03347059711813927,
+            0.0030062312725931406,
+            0.031086837872862816,
+            -0.005057325586676598,
+            0.007959471084177494,
+            0.011624909937381744,
+            0.020474210381507874,
+            -0.019396619871258736,
+            0.00877582747489214,
+            0.002100075827911496,
+            0.019233347848057747,
+            0.02171507105231285,
+            0.020343592390418053,
+            -0.03539719805121422,
+            0.028409190475940704,
+            0.018384337425231934,
+            0.02444170042872429,
+            0.044442422688007355,
+            -0.03974021226167679,
+            -0.004147088620811701,
+            -0.0032225653994828463,
+            -0.02997659333050251,
+            0.02008235827088356,
+            -0.007530884351581335,
+            -0.01841699331998825,
+            -0.007069643586874008,
+            0.03327467292547226,
+            0.014571955427527428,
+            -0.05051611363887787,
+            0.01980479806661606,
+            0.01740471087396145,
+            0.02055584453046322,
+            0.024474354460835457,
+            -0.01123305968940258,
+            -0.0019072118448093534,
+            0.008996243588626385,
+            0.006804327480494976,
+            0.019266001880168915,
+            0.015257694758474827,
+            0.002012317767366767,
+            -0.0022082431241869926,
+            -0.011837163008749485,
+            0.005224678665399551,
+            -0.009077879600226879,
+            0.010808554477989674,
+            -0.02458864450454712,
+            -0.009698309935629368,
+            0.021388528868556023,
+            0.0031164393294602633,
+            -0.013665799982845783,
+            -0.03729114308953285,
+            -0.045944519340991974,
+            0.0033837957307696342,
+            0.004543021321296692,
+            -0.05365091934800148,
+            0.005322641227394342,
+            -0.026547899469733238,
+            -0.002108239568769932,
+            -0.005502239800989628,
+            0.0019214981002733111,
+            0.005636938381940126,
+            0.0167679525911808,
+            -0.013494365848600864,
+            0.02248244546353817,
+            -0.015502601861953735,
+            0.03165828809142113,
+            0.0055961208418011665,
+            0.007375776767730713,
+            -0.0027796924114227295,
+            0.05590406060218811,
+            -0.003712379140779376,
+            -0.03699725493788719,
+            0.023282473906874657,
+            -0.01311067771166563,
+            -0.0001747512142173946,
+            0.030090883374214172,
+            0.00884113647043705,
+            -0.015257694758474827,
+            0.013437220826745033,
+            -0.030613351613283157,
+            -0.03249097242951393,
+            0.010996315628290176,
+            -0.013755599036812782,
+            -0.0015776080545037985,
+            -0.04568328335881233,
+            -0.009339112788438797,
+            0.004269542172551155,
+            -0.018335357308387756,
+            0.010775899514555931,
+            0.01631895825266838,
+            0.013992343097925186,
+            -0.024245774373412132,
+            0.009053388610482216,
+            0.006865554489195347,
+            0.013453547842800617,
+            0.0007331897504627705,
+            0.014049487188458443,
+            -0.008677864447236061,
+            -0.011306531727313995,
+            -0.018319029361009598,
+            0.0032450153958052397,
+            -0.026792805641889572,
+            0.005155288614332676,
+            -0.003155216109007597,
+            -0.03771565109491348,
+            -0.06521052122116089,
+            -0.0344502255320549,
+            0.016294466331601143,
+            0.03396041318774223,
+            -0.009045224636793137,
+            0.0035144127905368805,
+            -0.00023648813657928258,
+            -0.01093917153775692,
+            0.024523336440324783,
+            -0.005930826533585787,
+            -0.0004829256213270128,
+            0.0035389035474509,
+            0.02050686441361904,
+            -0.009722800925374031,
+            0.005845109466463327,
+            -0.051952898502349854,
+            -0.01586179807782173,
+            0.01812310516834259,
+            0.006306350696831942,
+            0.014629100449383259,
+            -0.009690145961940289,
+            0.01693122461438179,
+            -0.030074557289481163,
+            -0.028147956356406212,
+            0.04130761697888374,
+            0.008277850225567818,
+            0.011486129835247993,
+            -0.003055212553590536,
+            -0.01680060662329197,
+            0.02504580281674862,
+            -0.0020633398089557886,
+            -0.006461458280682564,
+            -0.00817988719791174,
+            0.015820980072021484,
+            0.044279150664806366,
+            0.003255219664424658,
+            0.0009821783751249313,
+            0.017045514658093452,
+            -0.002663361607119441,
+            -0.011575928889214993,
+            0.007677828427404165,
+            0.013592327944934368,
+            -0.016384266316890717,
+            0.010375885292887688,
+            -0.025862159207463264,
+            0.0332583449780941,
+            0.03131541982293129,
+            0.011575928889214993,
+            -0.015045441687107086,
+            0.006130834110081196,
+            0.005330805201083422,
+            0.008086007088422775,
+            -0.001369437319226563,
+            -0.0017133272485807538,
+            0.020719116553664207,
+            -0.03276853263378143,
+            -0.01693122461438179,
+            -0.005347132217139006,
+            0.003932795487344265,
+            0.005277741700410843,
+            0.03396041318774223,
+            -0.0033266509417444468,
+            0.017927179113030434,
+            -0.011657563969492912,
+            0.010579974390566349,
+            0.021764053031802177,
+            0.037095218896865845,
+            -0.024996822699904442,
+            -0.029862305149435997,
+            0.012914752587676048,
+            -0.001296985661610961,
+            0.014065815135836601,
+            0.0010173837654292583,
+            0.014139287173748016,
+            0.0029409227427095175,
+            0.060475654900074005,
+            -0.017763907089829445,
+            0.004628738854080439,
+            0.0004035860183648765,
+            -0.01858026348054409,
+            -0.0013531101867556572,
+            0.004261378664523363,
+            -0.010979988612234592,
+            0.03608293831348419,
+            0.002344982698559761,
+            0.039185091853141785,
+            0.013918871060013771,
+            0.009290131740272045,
+            0.0015143405180424452,
+            -0.013592327944934368,
+            0.008355404250323772,
+            0.0015592400450259447,
+            -0.001132694073021412,
+            0.009461566805839539,
+            0.019298657774925232,
+            0.009714636951684952,
+            0.025274382904171944,
+            0.009788108989596367,
+            0.012857607565820217,
+            -0.029470453038811684,
+            0.03510330989956856,
+            -0.0006224715034477413,
+            0.012996388599276543,
+            -0.02040890045464039,
+            -0.003136848099529743,
+            -0.0008240094175562263,
+            0.015649545937776566,
+            -0.015616890974342823,
+            -0.001201063860207796,
+            -0.01297189760953188,
+            0.040099408477544785,
+            -0.018988441675901413,
+            0.013976015150547028,
+            0.009673818945884705,
+            -0.02865409664809704,
+            -0.007596192881464958,
+            0.01995174214243889,
+            0.014294394291937351,
+            -0.0053144777193665504,
+            -0.020490536466240883,
+            0.005832863971590996,
+            0.015380147844552994,
+            -0.028572462499141693,
+            1.3584674888988957e-05,
+            0.014751553535461426,
+            -0.0164903923869133,
+            -0.00025409081717953086,
+            -0.004702210891991854,
+            0.003975654020905495,
+            -0.009265640750527382,
+            0.007028825581073761,
+            0.016057724133133888,
+            -0.024507008492946625,
+            -0.020049704238772392,
+            0.005722655914723873,
+            0.006571666337549686,
+            0.01444950234144926,
+            0.007739055436104536,
+            0.004355259705334902,
+            -0.020033378154039383,
+            -0.01799248717725277,
+            0.0032674651592969894,
+            -0.01649855636060238,
+            -0.0048614004626870155,
+            0.0020531355403363705,
+            -0.019363965839147568,
+            0.01875986158847809,
+            -0.010743245482444763,
+            0.00017538899555802345,
+            -0.003226647386327386,
+            -0.020800752565264702,
+            0.006204306147992611,
+            -0.004865481983870268,
+            0.019870106130838394,
+            0.003555230563506484,
+            0.02785406820476055,
+            0.02310287579894066,
+            -0.004428731743246317,
+            0.017600636929273605,
+            -0.016155686229467392,
+            0.007983962073922157,
+            -0.031135819852352142,
+            -0.001782717532478273,
+            -0.010408539324998856,
+            0.0036103345919400454,
+            -0.015943434089422226,
+            -0.009126860648393631,
+            -0.0048777274787425995,
+            -0.0006678812787868083,
+            -0.004820582456886768,
+            0.0004275664687156677,
+            0.0027817331720143557,
+            0.02231917530298233,
+            -0.013208640739321709,
+            -0.02803366631269455,
+            -0.017437364906072617,
+            0.006045116577297449,
+            0.019396619871258736,
+            -0.026025431230664253,
+            -0.009184005670249462,
+            -0.005012426059693098,
+            -0.004326687194406986,
+            -0.021404854953289032,
+            -0.015167895704507828,
+            -0.005632856860756874,
+            -0.008045189082622528,
+            -0.010522829368710518,
+            0.0046532293781638145,
+            -0.0035715578123927116,
+            -0.011739199981093407,
+            0.021666089072823524,
+            0.018057795241475105,
+            -0.019837452098727226,
+            -0.019837452098727226,
+            0.013641309924423695,
+            -0.006159406621009111,
+            0.0018878233386203647,
+            0.02008235827088356,
+            0.00029286774224601686,
+            0.005824700463563204,
+            0.020882388576865196,
+            -0.0035225762985646725,
+            -0.01797616109251976,
+            0.003577680326998234,
+            -0.0018888438353314996,
+            -0.012122887186706066,
+            0.003212361130863428,
+            0.02773977816104889,
+            0.0008076822850853205,
+            0.023429417982697487,
+            -0.03056436963379383,
+            -0.011412657797336578,
+            0.006061443593353033,
+            0.02369065210223198,
+            0.00211844383738935,
+            -0.0021694661118090153,
+            0.020131340250372887,
+            0.03941367194056511,
+            0.005775718949735165,
+            0.0015755671774968505,
+            -0.04003410041332245,
+            0.01027792226523161,
+            0.009208496659994125,
+            0.024621298536658287,
+            0.008339077234268188,
+            0.0014653591206297278,
+            -0.012906589545309544,
+            -0.009755454957485199,
+            -0.0017939424142241478,
+            0.014702572487294674,
+            -0.023870250210165977,
+            0.0024755997583270073,
+            -0.03157665207982063,
+            0.005094061605632305,
+            0.0021959978621453047,
+            -0.009559528902173042,
+            0.0335359051823616,
+            0.00980443600565195,
+            -0.02065380848944187,
+            0.015151568688452244,
+            0.005767555441707373,
+            0.007330877240747213,
+            0.022449791431427002,
+            -0.008228869177401066,
+            0.030450081452727318,
+            -0.005049162078648806,
+            0.01213921420276165,
+            0.02742956392467022,
+            0.032588932663202286,
+            -0.013102514669299126,
+            -0.003959327004849911,
+            0.0006097159348428249,
+            -0.0002561317232903093,
+            0.0007995187188498676,
+            -0.008204378187656403,
+            0.0037389106582850218,
+            -0.014800535514950752,
+            0.012710663489997387,
+            -0.019543563947081566,
+            -0.0034286954905837774,
+            0.0038532004691660404,
+            0.01991908811032772,
+            0.0016265894519165158,
+            -0.033633869141340256,
+            0.02055584453046322,
+            -0.015420965850353241,
+            0.0135596739128232,
+            0.001290863030590117,
+            -0.011110605672001839,
+            -0.00029210239881649613,
+            0.020588500425219536,
+            0.010947334580123425,
+            -0.007845181971788406,
+            -0.0135596739128232,
+            -0.028376536443829536,
+            -0.021535472944378853,
+            0.00823703221976757,
+            0.009445239789783955,
+            -0.0028837777208536863,
+            0.0021306893322616816,
+            -0.011869817040860653,
+            0.0006520643946714699,
+            -0.0008122742874547839,
+            0.01740471087396145,
+            -0.0335359051823616,
+            0.021943651139736176,
+            -0.019527235999703407,
+            -0.019543563947081566,
+            0.017045514658093452,
+            0.008049271069467068,
+            0.009445239789783955,
+            0.0033552234526723623,
+            -0.01995174214243889,
+            -0.012326976284384727,
+            0.0338951013982296,
+            -0.01138000376522541,
+            0.006298186723142862,
+            0.004408322740346193,
+            0.012914752587676048,
+            -0.008914608508348465,
+            -0.011926962062716484,
+            0.01784554310142994,
+            -0.009861581027507782,
+            0.013951525092124939,
+            0.0010010566329583526,
+            -0.0254703089594841,
+            0.02739690989255905,
+            -0.019233347848057747,
+            0.03487472981214523,
+            -0.021617108955979347,
+            0.00711862463504076,
+            0.0066614653915166855,
+            0.008792154490947723,
+            -0.006690037902444601,
+            -0.013224967755377293,
+            0.0012786176521331072,
+            0.03807484731078148,
+            -0.02623768337070942,
+            0.024180466309189796,
+            0.022400809451937675,
+            -0.00040766780148260295,
+            0.02065380848944187,
+            -0.008367649279534817,
+            -0.026711169630289078,
+            0.0179598331451416,
+            -0.012596373446285725,
+            -0.026123393326997757,
+            0.013257622718811035,
+            -0.022678371518850327,
+            -0.020278284326195717,
+            -0.026302991434931755,
+            0.004947117529809475,
+            -0.0025694805663079023,
+            -0.005151206627488136,
+            0.009004407562315464,
+            0.02561725303530693,
+            0.020915042608976364,
+            0.019576217979192734,
+            0.02111096680164337,
+            0.01161674689501524,
+            0.022449791431427002,
+            -0.007559456862509251,
+            0.04401791840791702,
+            -0.004947117529809475,
+            0.004632820375263691,
+            -0.0015980169409886003,
+            -0.015657709911465645,
+            0.009771781973540783,
+            0.0119106350466609,
+            0.0029572497587651014,
+            -0.029176564887166023,
+            -0.009061552584171295,
+            -0.03660540655255318,
+            0.005053244065493345,
+            0.0020664012990891933,
+            0.0176822729408741,
+            0.032115448266267776,
+            -0.005865518003702164,
+            0.004592002835124731,
+            -0.005102225113660097,
+            -0.013927034102380276,
+            -0.009420748800039291,
+            -0.02739690989255905,
+            -0.025437654927372932,
+            -0.007653337903320789,
+            -0.013739272020757198,
+            -0.030368445441126823,
+            -0.011722872965037823,
+            0.01551892887800932,
+            -0.007645174395292997,
+            0.011918798089027405,
+            -0.016702644526958466,
+            0.01385356206446886,
+            -0.004110352601855993,
+            -0.004763437435030937,
+            -0.004530776292085648,
+            -0.017600636929273605,
+            -0.006420640274882317,
+            0.014367866329848766,
+            -0.009477893821895123,
+            -0.01237595733255148,
+            -0.02215590327978134,
+            -0.0015459742862731218,
+            -0.01574750803411007,
+            0.003489922033622861,
+            -0.014596446417272091,
+            -0.006771673448383808,
+            -0.003908304497599602,
+            -0.018514955416321754,
+            0.01766594499349594,
+            0.014816862531006336,
+            -0.019380291923880577,
+            -0.0090697156265378,
+            0.02638462744653225,
+            0.042777057737112045,
+            -0.03218075633049011,
+            -0.008200296200811863,
+            0.0020378287881612778,
+            0.018090449273586273,
+            0.010171796195209026,
+            0.016702644526958466,
+            -0.024653952568769455,
+            0.0022286521270871162,
+            -0.021649762988090515,
+            -0.001220452249981463,
+            -0.023739634081721306,
+            0.010775899514555931,
+            0.025715215131640434,
+            -0.010971825569868088,
+            0.025976449251174927,
+            0.01634344831109047,
+            -0.01812310516834259,
+            0.012016761116683483,
+            0.007183933164924383,
+            0.014645427465438843,
+            -0.019755816087126732,
+            0.0167679525911808,
+            -0.015796490013599396,
+            0.031233781948685646,
+            0.00892277155071497,
+            -0.04829562455415726,
+            -0.03503800183534622,
+            -0.013388238847255707,
+            0.0013388239312916994,
+            -0.010702427476644516,
+            0.015037278644740582,
+            0.015967924147844315,
+            0.013959688134491444,
+            -0.019380291923880577,
+            -0.015820980072021484,
+            0.011984107084572315,
+            0.010065670125186443,
+            0.027462217956781387,
+            -0.006649219896644354,
+            -0.004212397150695324,
+            -0.00960034690797329,
+            -0.02249877341091633,
+            -0.016098542138934135,
+            0.0209803506731987,
+            -0.018041469156742096,
+            0.015935270115733147,
+            -0.026139721274375916,
+            -0.012449429370462894,
+            0.017453692853450775,
+            -0.05211617052555084,
+            -0.01453113742172718,
+            0.0007612520130351186,
+            0.03012353740632534,
+            0.004751192405819893,
+            0.0035184945445507765,
+            0.01841699331998825,
+            -0.01993541419506073,
+            -0.0039776950143277645,
+            -0.02099667675793171,
+            0.006832899991422892,
+            -0.011853490024805069,
+            -0.013984179124236107,
+            0.014482156373560429,
+            -0.013845399022102356,
+            -0.01601690612733364,
+            0.0034368589986115694,
+            0.007183933164924383,
+            -0.021094640716910362,
+            0.011951452121138573,
+            0.003945040516555309,
+            0.012171869166195393,
+            0.012171869166195393,
+            0.01199227012693882,
+            0.021551799029111862,
+            0.02295593172311783,
+            -0.007334959227591753,
+            -0.004767519421875477,
+            -0.01004117913544178,
+            -0.0015398516552522779,
+            0.008898280560970306,
+            0.0033797139767557383,
+            -0.002989904023706913,
+            0.007837017998099327,
+            0.010033016093075275,
+            -0.01498013362288475,
+            0.033193036913871765,
+            -0.0035878848284482956,
+            -0.04134026914834976,
+            -0.003238892648369074,
+            -0.03800953924655914,
+            0.020033378154039383,
+            -0.023298801854252815,
+            -0.016996532678604126,
+            0.0032776694279164076,
+            -0.00629410520195961,
+            -0.019902760162949562,
+            0.013608654960989952,
+            0.011404493823647499,
+            0.00258580781519413,
+            0.02021297626197338,
+            0.012269831262528896,
+            -0.005134879611432552,
+            0.026172375306487083,
+            -0.014718899503350258,
+            -0.008751336485147476,
+            0.015567909926176071,
+            0.006853308994323015,
+            0.0022347746416926384,
+            0.0011031010653823614,
+            -0.018825169652700424,
+            0.005098143592476845,
+            0.0164903923869133,
+            0.007216587662696838,
+            -0.000566347036510706,
+            0.014049487188458443,
+            0.012514738366007805,
+            -0.015135240741074085,
+            -0.0002213090192526579,
+            -0.026564225554466248,
+            0.0032817514147609472,
+            0.004567512311041355,
+            0.007506393827497959,
+            -0.006771673448383808,
+            0.006281859707087278,
+            0.00658799335360527,
+            0.014759717509150505,
+            -0.010155469179153442,
+            -0.016980206593871117,
+            -0.002510294783860445,
+            0.004285869188606739,
+            0.016694480553269386,
+            0.03157665207982063,
+            -0.010310577228665352,
+            -0.005734901409596205,
+            -0.0386626236140728,
+            0.02070278860628605,
+            0.03332365304231644,
+            0.011175914667546749,
+            -0.028213264420628548,
+            0.012302485294640064,
+            0.0006515541463159025,
+            0.0019276207312941551,
+            -0.026302991434931755,
+            0.02787039615213871,
+            0.00974729098379612,
+            0.0066614653915166855,
+            0.017861871048808098,
+            0.0022306928876787424,
+            0.0029266364872455597,
+            -0.016302630305290222,
+            0.01018812321126461,
+            0.009086042642593384,
+            -0.017306748777627945,
+            0.0009745250572450459,
+            0.02324981987476349,
+            -0.01423724927008152,
+            -0.003208279376849532,
+            0.0018133309204131365,
+            0.02651524543762207,
+            -0.013820908032357693,
+            -0.014294394291937351,
+            0.0005714492290280759,
+            0.01751900091767311,
+            -0.0007979880319908261,
+            0.012098397128283978,
+            0.04898136481642723,
+            -0.012424939312040806,
+            0.019657853990793228,
+            0.005461421795189381,
+            -0.015290348790585995,
+            -0.032605260610580444,
+            -0.004922627005726099,
+            -0.01830270327627659,
+            0.011200404725968838,
+            0.014343376271426678,
+            -0.015763835981488228,
+            0.0007525782566517591,
+            -0.03144603595137596,
+            0.007743136957287788,
+            0.014735226519405842,
+            0.018776189535856247,
+            -0.002440904499962926,
+            0.006134915631264448,
+            0.027674470096826553,
+            0.02458864450454712,
+            -0.007392104249447584,
+            0.013804581016302109,
+            0.02320083975791931,
+            -0.025062130764126778,
+            -0.003069498809054494,
+            0.006820654962211847,
+            0.004104230087250471,
+            0.0053063142113387585,
+            -0.016963878646492958,
+            0.03915243595838547,
+            -0.005922663025557995,
+            -0.01506176870316267,
+            0.0030266400426626205,
+            -0.009028897620737553,
+            0.0063390047289431095,
+            -0.012563719414174557,
+            0.012637191452085972,
+            -0.008114579133689404,
+            0.00022883480414748192,
+            -0.0059430720284581184,
+            -0.011110605672001839,
+            -0.012653518468141556,
+            -0.0007949267164804041,
+            -0.016531210392713547,
+            0.007432921789586544,
+            0.00981259997934103,
+            0.005045080557465553,
+            0.01025343220680952,
+            -0.0036797248758375645,
+            -0.003136848099529743,
+            -0.016751626506447792,
+            -0.009984034113585949,
+            0.0073716952465474606,
+            0.01872720755636692,
+            -0.028588788583874702,
+            -0.018547609448432922,
+            0.009290131740272045,
+            0.0022898786701261997,
+            0.018776189535856247,
+            0.0158944521099329,
+            -0.0002595757250674069,
+            0.012318813242018223,
+            -0.01964152604341507,
+            -0.009184005670249462,
+            0.009690145961940289,
+            -0.003569516818970442,
+            0.01048201136291027,
+            -0.014759717509150505,
+            0.010033016093075275,
+            -0.011159587651491165,
+            -0.011053460650146008,
+            0.01385356206446886,
+            0.01738838478922844,
+            0.008873790502548218,
+            -0.001294944784604013,
+            3.383413059054874e-05,
+            -0.010506502352654934,
+            -0.024033522233366966,
+            0.004771600943058729,
+            0.027021385729312897,
+            -0.013273949734866619,
+            0.003883813973516226,
+            0.008702355436980724,
+            -0.017502672970294952,
+            -0.019821124151349068,
+            -0.012653518468141556,
+            -0.022106921300292015,
+            0.020572172477841377,
+            0.023413091897964478,
+            -0.02026195637881756,
+            0.00877582747489214,
+            -0.004620575346052647,
+            0.014049487188458443,
+            -0.01400050614029169,
+            0.010612628422677517,
+            0.0317889042198658,
+            -0.020000724121928215,
+            -0.0028654097113758326,
+            0.0036633978597819805,
+            -0.015347493812441826,
+            0.018629245460033417,
+            0.03268689662218094,
+            -0.00176945177372545,
+            0.014041324146091938,
+            0.0008138049161061645,
+            -0.009118696674704552,
+            0.01905374974012375,
+            0.011094278655946255,
+            0.005347132217139006,
+            -0.0042491331696510315,
+            -0.012212686240673065,
+            -0.03010721132159233,
+            0.016702644526958466,
+            0.0129637336358428,
+            -0.0037674831692129374,
+            -0.0011551438365131617,
+            -0.006343086715787649,
+            -0.015404638834297657,
+            0.0035123717971146107,
+            -0.013078023679554462,
+            -0.00803294312208891,
+            -0.008759500458836555,
+            0.004314441699534655,
+            -0.0034960447810590267,
+            -0.009053388610482216,
+            0.007236996199935675,
+            0.0009510547970421612,
+            -0.0010602424154058099,
+            -0.01902109570801258,
+            -0.025078458711504936,
+            -0.00995954405516386,
+            -0.016849588602781296,
+            0.0038287099450826645,
+            0.0005234883283264935,
+            -0.027184655889868736,
+            0.0053961132653057575,
+            -0.008139070123434067,
+            -0.008134988136589527,
+            0.007126788143068552,
+            0.006881881505250931,
+            -0.007437003776431084,
+            -0.010816717520356178,
+            -0.00786150898784399,
+            -0.006865554489195347,
+            0.020033378154039383,
+            0.0034348180051892996,
+            0.012082069180905819,
+            -0.010498338378965855,
+            -0.018025141209363937,
+            -0.016963878646492958,
+            0.016718972474336624,
+            -0.008971752598881721,
+            -0.02083340659737587,
+            -0.01444950234144926,
+            -0.008424794301390648,
+            -0.007665583398193121,
+            -0.01527402177453041,
+            -0.021159948781132698,
+            -0.0032103201374411583,
+            -0.011347348801791668,
+            -0.0067431009374558926,
+            -0.0028960229828953743,
+            0.0013571919407695532,
+            0.00951871182769537,
+            0.00546958576887846,
+            0.00438791373744607,
+            0.021666089072823524,
+            0.010122815147042274,
+            -0.005461421795189381,
+            0.016441410407423973,
+            -0.006183897145092487,
+            -0.004440976772457361,
+            -0.01498013362288475,
+            -0.009918726049363613,
+            0.018547609448432922,
+            -0.0055226488038897514,
+            0.008457448333501816,
+            -0.0009536059224046767,
+            -0.00576347392052412,
+            0.015674035996198654,
+            0.0129637336358428,
+            0.016718972474336624,
+            0.004771600943058729,
+            -0.00012168807734269649,
+            0.012416775338351727,
+            -0.0015745466807857156,
+            0.012555556371808052,
+            -0.0038429962005466223,
+            0.014506647363305092,
+            -0.009722800925374031,
+            -0.018988441675901413,
+            0.004157293122261763,
+            0.006265532691031694,
+            -0.012531065382063389,
+            0.009404421783983707,
+            0.007743136957287788,
+            -0.013747435994446278,
+            -0.02261306345462799,
+            -0.000106317616882734,
+            0.00014362763613462448,
+            0.008122743107378483,
+            0.008971752598881721,
+            -0.008073761127889156,
+            0.012890261597931385,
+            0.004616493359208107,
+            -0.007179851643741131,
+            -0.019657853990793228,
+            0.026009103283286095,
+            0.015608727931976318,
+            0.012269831262528896,
+            -0.052508022636175156,
+            -0.0030980713199824095,
+            0.010743245482444763,
+            -0.015951598063111305,
+            -0.026792805641889572,
+            -0.006404313258826733,
+            -0.008759500458836555,
+            -0.04290767386555672,
+            -0.009624837897717953,
+            -0.0005724696675315499,
+            -0.011388166807591915,
+            0.010400376282632351,
+            0.01573934406042099,
+            0.014612773433327675,
+            0.005489994306117296,
+            -0.007232914678752422,
+            -0.017633290961384773,
+            -0.02788672223687172,
+            0.010057506151497364,
+            -0.013380075804889202,
+            -0.006412476766854525,
+            -0.002263347152620554,
+            -0.007690073922276497,
+            0.0016112826997414231,
+            0.01691489666700363,
+            0.042483169585466385,
+            0.023037567734718323,
+            -0.012335140258073807,
+            -0.020131340250372887,
+            -0.012800462543964386,
+            -0.009192168712615967,
+            0.00633492274209857,
+            0.011592255905270576,
+            -0.004738946910947561,
+            0.0096656559035182,
+            -0.004294032696634531,
+            0.012302485294640064,
+            0.012767808511853218,
+            0.01680060662329197,
+            -2.3023791072773747e-05,
+            0.005579793825745583,
+            -0.0024694770108908415,
+            -0.01738838478922844,
+            -0.004379750229418278,
+            0.008261523209512234,
+            -0.0008872769540175796,
+            0.0019317024853080511,
+            0.014204595237970352,
+            0.019102731719613075,
+            -0.009094206616282463,
+            -0.014612773433327675,
+            -0.016098542138934135,
+            -0.005277741700410843,
+            -0.0012622905196622014,
+            0.029927613213658333,
+            0.02037624642252922,
+            -0.019266001880168915,
+            -0.018237393349409103,
+            -0.018351683393120766,
+            0.00922482367604971,
+            -0.003563394071534276,
+            -0.0064369672909379005,
+            0.01905374974012375,
+            0.023021239787340164,
+            0.017910851165652275,
+            -0.003642988856881857,
+            -0.014865843579173088,
+            0.003804219188168645,
+            0.02847449854016304,
+            0.008367649279534817,
+            0.01078406348824501,
+            0.008726846426725388,
+            -0.0011908593587577343,
+            -0.032441988587379456,
+            -0.0113146947696805,
+            0.00951871182769537,
+            -0.008481939323246479,
+            0.026564225554466248,
+            -0.016531210392713547,
+            0.009388094767928123,
+            -0.005077734589576721,
+            0.0019286411115899682,
+            -0.02365799807012081,
+            0.0014929111348465085,
+            0.010122815147042274,
+            -0.024947840720415115,
+            0.00336950970813632,
+            -0.023592690005898476,
+            -0.00013763252354692668,
+            0.008367649279534817,
+            0.0022837561555206776,
+            0.01650671847164631,
+            -0.0113146947696805,
+            0.0051838611252605915,
+            0.02099667675793171,
+            -0.002832755446434021,
+            -0.01512707769870758,
+            -0.0007842120248824358,
+            0.0005092020728625357,
+            -0.001286781276576221,
+            0.02563357912003994,
+            0.015812816098332405,
+            -0.0036756431218236685,
+            0.020882388576865196,
+            0.008677864447236061,
+            0.0019133344758301973,
+            -0.021764053031802177,
+            0.0023796779569238424,
+            -0.032899148762226105,
+            0.017437364906072617,
+            -0.011126932688057423,
+            0.012490247376263142,
+            0.00868602842092514,
+            -0.01341272983700037,
+            -0.0007388022495433688,
+            -0.0107187544927001,
+            -0.0015725058037787676,
+            -0.007934981025755405,
+            0.009779945947229862,
+            -0.004363423213362694,
+            0.008339077234268188,
+            -0.00024579971795901656,
+            -0.01476788055151701,
+            0.018694553524255753,
+            9.678283277025912e-06,
+            -0.00010925139940809458,
+            0.005441013257950544,
+            0.009935053065419197,
+            0.00786559097468853,
+            0.002009256277233362,
+            0.023870250210165977,
+            -0.031119491904973984,
+            -0.016082214191555977,
+            -0.010204450227320194,
+            0.005543057806789875,
+            -0.007922735065221786,
+            -0.018531281501054764,
+            -0.03257260471582413,
+            0.01318415068089962,
+            0.001293924287892878,
+            -0.0009806477464735508,
+            0.01258004643023014,
+            -0.00942891277372837,
+            -0.0013878053287044168,
+            -0.01606588624417782,
+            -0.014808698557317257,
+            0.006416558753699064,
+            0.0033633869607001543,
+            -0.01222901325672865,
+            -0.023739634081721306,
+            0.014759717509150505,
+            0.0007418635650537908,
+            -0.016980206593871117,
+            -0.017502672970294952,
+            0.00018457300029695034,
+            -0.004804255440831184,
+            0.02204161323606968,
+            -0.016678154468536377,
+            0.003373591462150216,
+            0.0007219648687168956,
+            0.009232986718416214,
+            -0.02889900468289852,
+            0.00716352416202426,
+            0.0006107363733462989,
+            0.0035164537839591503,
+            -0.006518602836877108,
+            -0.021013004705309868,
+            0.011371839791536331,
+            -0.0020674217958003283,
+            -0.014751553535461426,
+            -0.020751770585775375,
+            -0.0020796670578420162,
+            -0.007420676294714212,
+            -0.02724996581673622,
+            0.017910851165652275,
+            0.0009985055075958371,
+            -0.01738838478922844,
+            0.008077843114733696,
+            0.006596156861633062,
+            0.009355440735816956,
+            0.02455599047243595,
+            0.010147306136786938,
+            -0.018776189535856247,
+            -0.0008087027235887945,
+            0.006979844067245722,
+            0.009306458756327629,
+            0.0036450298503041267,
+            0.04222193360328674,
+            -0.008130906149744987,
+            -0.006579829845577478,
+            0.0066614653915166855,
+            -0.008449285291135311,
+            -0.013747435994446278,
+            0.026727497577667236,
+            0.005796127952635288,
+            0.015984252095222473,
+            0.016123032197356224,
+            -0.00945340283215046,
+            -0.005008344538509846,
+            -0.0018786393338814378,
+            0.02065380848944187,
+            0.0073798587545752525,
+            -0.005502239800989628,
+            -0.009714636951684952,
+            0.005881845485419035,
+            -0.00898808054625988,
+            0.0021653843577951193,
+            -0.009526874870061874,
+            -0.005824700463563204,
+            -0.008069679141044617,
+            0.016400592401623726,
+            -0.0003523086488712579,
+            0.027315273880958557,
+            0.020751770585775375,
+            -0.008563575334846973,
+            -0.0005683879135176539,
+            0.017208784818649292,
+            -0.005820618476718664,
+            0.015608727931976318,
+            -0.011192241683602333,
+            -0.007706400938332081,
+            0.006408394780009985,
+            0.024017194285988808,
+            0.012237177230417728,
+            -0.0056042843498289585,
+            0.0002885308349505067,
+            -0.0030348035506904125,
+            -0.010612628422677517,
+            0.013388238847255707,
+            0.015323002822697163,
+            -0.01947825588285923,
+            -0.004722619894891977,
+            -0.0016592437168583274,
+            -0.003300119424238801,
+            0.00945340283215046,
+            0.02305389568209648,
+            0.000171817431692034,
+            0.009184005670249462,
+            0.0012765767751261592,
+            0.03862996771931648,
+            -0.0053144777193665504,
+            0.03102152980864048,
+            -0.0010423846542835236,
+            -0.0059430720284581184,
+            -0.009094206616282463,
+            -0.020441556349396706,
+            -0.011486129835247993,
+            -0.00041481092921458185,
+            -0.005040998570621014,
+            -0.0004956811899319291,
+            -0.020327266305685043,
+            0.0026960158720612526,
+            0.014637264423072338,
+            0.007661501411348581,
+            0.017927179113030434,
+            -0.002889900468289852,
+            0.017437364906072617,
+            0.019690508022904396,
+            -0.01400050614029169,
+            -0.029862305149435997,
+            -0.0006515541463159025,
+            0.01498013362288475,
+            -0.01661284640431404,
+            0.019445601850748062,
+            0.004085862077772617,
+            0.0029531680047512054,
+            0.004779764451086521,
+            -0.006049198564141989,
+            0.018482301384210587,
+            0.02039257436990738,
+            0.01453930139541626,
+            -0.013167822733521461,
+            0.014057651162147522,
+            0.009771781973540783,
+            0.006600238848477602,
+            -0.01646590232849121,
+            -0.013535182923078537,
+            0.002555194543674588,
+            0.0009245232213288546,
+            0.024392718449234962,
+            -0.020588500425219536,
+            0.0016602640971541405,
+            0.010588138364255428,
+            0.01769859902560711,
+            -0.004302196204662323,
+            0.008906444534659386,
+            0.0008740111952647567,
+            0.007134951651096344,
+            -0.0035919665824621916,
+            -0.006457376293838024,
+            -0.006673710886389017,
+            -0.010392212308943272,
+            -0.0020959940738976,
+            -0.02070278860628605,
+            0.002198038622736931,
+            -0.002830714685842395,
+            0.018237393349409103,
+            -0.01018812321126461,
+            0.038140155375003815,
+            0.03399306535720825,
+            -0.0007025764207355678,
+            0.011102442629635334,
+            -0.01709449477493763,
+            -0.003945040516555309,
+            -0.021959977224469185,
+            0.01844964735209942,
+            0.016392430290579796,
+            0.012269831262528896,
+            0.005302232690155506,
+            0.017796561121940613,
+            -0.0006301248213276267,
+            0.014490320347249508,
+            0.0029654132667928934,
+            0.007971717044711113,
+            -0.002265388146042824,
+            0.003938918001949787,
+            -0.031103165820240974,
+            -0.017029186710715294,
+            -0.01138000376522541,
+            0.004796091932803392,
+            -0.006375740747898817,
+            -0.0035205355379730463,
+            0.0084737753495574,
+            -0.005902254022657871,
+            0.0004270562494639307,
+            -0.025992777198553085,
+            -0.0157148540019989,
+            -0.010318740271031857,
+            -0.004040962550789118,
+            -0.01169838197529316,
+            -0.0027694879099726677,
+            -0.0022082431241869926,
+            0.0029552089981734753,
+            0.020947696641087532,
+            -0.012808626517653465,
+            -0.016294466331601143,
+            0.004379750229418278,
+            -0.002263347152620554,
+            -0.014016833156347275,
+            -0.016898570582270622,
+            0.006665547378361225,
+            0.012098397128283978,
+            -0.0017470018938183784,
+            -0.013608654960989952,
+            0.011739199981093407,
+            -0.006628811359405518,
+            0.04663025587797165,
+            -0.013584164902567863,
+            -0.0078002819791436195,
+            0.003720542648807168,
+            0.004698128905147314,
+            -0.009396257810294628,
+            0.007330877240747213,
+            0.013020878657698631,
+            0.01830270327627659,
+            0.0095758568495512,
+            -0.008465612307190895,
+            -0.0024388637393712997,
+            0.007796200457960367,
+            -0.016147522255778313,
+            0.006404313258826733,
+            0.0008525818702764809,
+            -0.014400520361959934,
+            0.006110425107181072,
+            0.028980640694499016,
+            0.013151495717465878,
+            0.01288209855556488,
+            -0.022351829335093498,
+            -0.004094025585800409,
+            -0.009355440735816956,
+            -0.004943036008626223,
+            -0.010090161114931107,
+            0.027070365846157074,
+            -0.018155759200453758,
+            0.02651524543762207,
+            0.00898808054625988,
+            0.0017684312770143151,
+            -0.01176369097083807,
+            0.006343086715787649,
+            0.02832755446434021,
+            0.0028592869639396667,
+            -0.020000724121928215,
+            -0.013592327944934368,
+            0.006314514204859734,
+            0.01707816869020462,
+            -0.002647034591063857,
+            -0.005428767763078213,
+            0.019657853990793228,
+            0.01784554310142994,
+            -0.004334850702434778,
+            -0.00936360377818346,
+            0.0008337036124430597,
+            0.00928196869790554,
+            0.016963878646492958,
+            0.015020951628684998,
+            0.009086042642593384,
+            -0.002647034591063857,
+            0.008037025108933449,
+            -0.008800318464636803,
+            -0.016449574381113052,
+            0.007220669183880091,
+            -0.008432958275079727,
+            -0.016963878646492958,
+            0.01872720755636692,
+            0.015257694758474827,
+            0.0181394312530756,
+            0.0024592725094407797,
+            -0.016735298559069633,
+            0.022302847355604172,
+            0.005755309946835041,
+            0.002998067531734705,
+            0.02160078100860119,
+            -0.022221211344003677,
+            0.013012715615332127,
+            0.02037624642252922,
+            0.027609162032604218,
+            -0.01737205684185028,
+            -0.010906516574323177,
+            -0.004857318475842476,
+            -0.006853308994323015,
+            -0.00176945177372545,
+            -0.008253359235823154,
+            -0.012620864436030388,
+            0.0028735732194036245,
+            -0.030466407537460327,
+            0.018106777220964432,
+            -0.005102225113660097,
+            0.010669773444533348,
+            -0.019266001880168915,
+            0.011363675817847252,
+            -0.0035838030744343996,
+            0.011135096661746502,
+            0.0019612954929471016,
+            -0.030041903257369995,
+            -0.008571738377213478,
+            0.015388311818242073,
+            -0.0030368445441126823,
+            -0.013641309924423695,
+            0.02006603218615055,
+            0.00018942011229228228,
+            -0.022270193323493004,
+            -0.004314441699534655,
+            -0.006934944540262222,
+            0.002193956868723035,
+            -0.0029613315127789974,
+            -0.0052981507033109665,
+            -0.030188847333192825,
+            0.020033378154039383,
+            0.009559528902173042,
+            -0.03072764165699482,
+            -0.003255219664424658,
+            -0.014996460638940334,
+            -0.0025184585247188807,
+            0.008881953544914722,
+            -0.004000144544988871,
+            -0.015551582910120487,
+            0.01784554310142994,
+            0.022711025550961494,
+            -0.0023164101876318455,
+            -0.016057724133133888,
+            -0.0090697156265378,
+            -0.027494871988892555,
+            0.030825603753328323,
+            -0.0066696288995444775,
+            0.0022878379095345736,
+            -0.005322641227394342,
+            0.009388094767928123,
+            0.004228724166750908,
+            0.0157148540019989,
+            0.006081852596253157,
+            -0.002647034591063857,
+            0.026596879586577415,
+            -0.01964152604341507,
+            0.009567692875862122,
+            -1.588864870427642e-05,
+            -0.003308282932266593,
+            0.01890680566430092,
+            0.0007939062779769301,
+            0.021649762988090515,
+            -0.008571738377213478,
+            -0.0001475818717153743,
+            -0.0034388997592031956,
+            0.010898353531956673,
+            -0.0028450009413063526,
+            -0.003465431509539485,
+            -0.001531688030809164,
+            0.011118769645690918,
+            0.01013097818940878,
+            0.003804219188168645,
+            -0.01737205684185028,
+            0.003142970846965909,
+            -0.008547247387468815,
+            -0.007028825581073761,
+            0.01302904263138771,
+            -0.011192241683602333,
+            -0.0018990482203662395,
+            -0.03160930797457695,
+            0.016286302357912064,
+            0.0533243790268898,
+            0.008547247387468815,
+            -0.014433175325393677,
+            -0.0018806802108883858,
+            -0.019755816087126732,
+            0.01535565685480833,
+            0.011004479601979256,
+            -0.016816934570670128,
+            -0.011984107084572315,
+            -0.012310649268329144,
+            0.007649256382137537,
+            0.021306892856955528,
+            -0.006522684823721647,
+            -0.008579902350902557,
+            0.025372346863150597,
+            -0.011788181029260159,
+            -0.010882026515901089,
+            -0.01661284640431404,
+            -0.01635977439582348,
+            0.00241641397587955,
+            0.019902760162949562,
+            -0.0169475506991148,
+            0.021176276728510857,
+            0.00936360377818346,
+            0.0010133018950000405,
+            0.003624620847404003,
+            0.008588065393269062,
+            0.0024878450203686953,
+            0.017878197133541107,
+            0.003883813973516226,
+            -0.01439235731959343,
+            0.0021408938337117434,
+            -0.0030286810360848904,
+            0.004624656867235899,
+            3.4320437407586724e-06,
+            -0.002749078907072544,
+            -0.0065635028295218945,
+            0.003710338147357106,
+            -0.021306892856955528,
+            0.021502818912267685,
+            -0.022678371518850327,
+            -0.004245051648467779,
+            0.005555302836000919,
+            -0.005118552595376968,
+            0.013469874858856201,
+            0.018792515620589256,
+            0.01423724927008152,
+            0.008702355436980724,
+            0.01462093647569418,
+            -0.011331021785736084,
+            0.02323349379003048,
+            -0.013282112777233124,
+            0.0005816536722704768,
+            0.012939243577420712,
+            0.01755165494978428,
+            0.018792515620589256,
+            -0.006245123688131571,
+            -0.010384049266576767,
+            -0.0002946535241790116,
+            0.0129637336358428,
+            -0.022286519408226013,
+            -0.012269831262528896,
+            -0.01386172603815794,
+            0.011486129835247993,
+            -0.004743028432130814,
+            0.025453981012105942,
+            -0.00633492274209857,
+            -0.014653591439127922,
+            -0.015918942168354988,
+            0.014196431264281273,
+            -0.048752784729003906,
+            0.0030348035506904125,
+            -0.0024919267743825912,
+            0.03043375350534916,
+            -0.027021385729312897,
+            0.01738838478922844,
+            -0.0028490826953202486,
+            -0.009355440735816956,
+            0.015429128892719746,
+            -0.015992414206266403,
+            -0.0046613928861916065,
+            -0.0016806729836389422,
+            -0.007539048325270414,
+            -0.00112248957157135,
+            0.017616963014006615,
+            -0.015845470130443573,
+            0.01282495353370905,
+            -0.00989423505961895,
+            0.0027021386194974184,
+            -0.004526694305241108,
+            -0.007751300930976868,
+            0.0196088720113039,
+            0.034548189491033554,
+            -0.007053316105157137,
+            0.0014990337658673525,
+            -0.022547753527760506,
+            0.013192313723266125,
+            -0.006730855442583561,
+            0.02351105399429798,
+            -0.005432849284261465,
+            -0.016898570582270622,
+            -0.013976015150547028,
+            -0.009739127941429615,
+            0.00839214026927948,
+            -0.022433463484048843,
+            -0.011110605672001839,
+            -0.00922482367604971,
+            -0.002573562553152442,
+            -0.018694553524255753,
+            0.004865481983870268,
+            -0.002426618244498968,
+            -0.012506574392318726,
+            -0.008816645480692387,
+            0.01048201136291027,
+            -0.009347276762127876,
+            0.003077662317082286,
+            -0.0011929002357646823,
+            -0.011453475803136826,
+            -0.02369065210223198,
+            0.0158944521099329,
+            0.0001974561164388433,
+            -0.010359558276832104,
+            0.008408467285335064,
+            -0.0018276170594617724,
+            0.001211268245242536,
+            0.016963878646492958,
+            -0.0020664012990891933,
+            0.003467472270131111,
+            0.023478399962186813,
+            0.004057289566844702,
+            -0.010971825569868088,
+            -0.00025906547671183944,
+            0.013535182923078537,
+            -0.005118552595376968,
+            -0.010008525103330612,
+            1.6646010408294387e-05,
+            -0.02984597720205784,
+            -0.008351322263479233,
+            0.006094098091125488,
+            0.018629245460033417,
+            -0.010620792396366596,
+            -0.015853634104132652,
+            0.026319319382309914,
+            -0.006359413731843233,
+            0.019070077687501907,
+            0.002889900468289852,
+            0.0025694805663079023,
+            0.01724144071340561,
+            0.013461710885167122,
+            0.00786150898784399,
+            -0.013927034102380276,
+            -0.02429475635290146,
+            0.0016398552106693387,
+            -0.018090449273586273,
+            0.010579974390566349,
+            -0.011028970591723919,
+            0.0032756286673247814,
+            -0.02398454025387764,
+            -0.012237177230417728,
+            0.004934872500598431,
+            -0.020343592390418053,
+            -0.00040588201954960823,
+            -0.0056940834037959576,
+            0.022237539291381836,
+            -0.005310396198183298,
+            0.008669701404869556,
+            0.0018255761824548244,
+            -0.012237177230417728,
+            0.0009877908742055297,
+            0.009469729848206043,
+            -0.029192892834544182,
+            -0.0007127808639779687,
+            -0.003138889092952013,
+            0.0011418780777603388,
+            0.005351213738322258,
+            0.009755454957485199,
+            0.007322713732719421,
+            0.0027062203735113144,
+            0.007253323681652546,
+            0.010449357330799103,
+            -0.00022041612828616053,
+            0.0197884701192379,
+            0.018172085285186768,
+            -0.007351286243647337,
+            0.008710519410669804,
+            0.024817224591970444,
+            -0.013159659691154957,
+            -0.008718682453036308,
+            0.023266147822141647,
+            -0.006461458280682564,
+            -0.0011122850701212883,
+            -0.01483318954706192,
+            0.00815947912633419,
+            0.0017367975087836385,
+            0.028131630271673203,
+            0.019657853990793228,
+            0.005820618476718664,
+            0.000639819074422121,
+            0.015788326039910316,
+            -0.013020878657698631,
+            0.0035286990460008383,
+            0.015461783856153488,
+            -0.003940958995372057,
+            -0.019821124151349068,
+            0.013878053054213524,
+            0.009502384811639786,
+            -0.003228688146919012,
+            0.01551892887800932,
+            0.011412657797336578,
+            -0.030841931700706482,
+            -0.0011898389784619212,
+            0.010498338378965855,
+            -0.0021449755877256393,
+            -0.008751336485147476,
+            -0.00014273475972004235,
+            -0.012384121306240559,
+            0.021388528868556023,
+            -0.004408322740346193,
+            0.012277995236217976,
+            -0.00815947912633419,
+            -0.017127150669693947,
+            -0.011559601873159409,
+            0.007914572022855282,
+            -0.010547320358455181,
+            -0.0006714528426527977,
+            -0.004022594541311264,
+            -0.005779800936579704,
+            0.017747581005096436,
+            0.004706292413175106,
+            -0.0008964610169641674,
+            -0.0018398624379187822,
+            0.007983962073922157,
+            -0.006979844067245722,
+            -0.008726846426725388,
+            0.004816500470042229,
+            -0.012833117507398129,
+            0.003981776535511017,
+            0.005543057806789875,
+            0.01506176870316267,
+            -0.014857680536806583,
+            -0.021159948781132698,
+            0.007020662073045969,
+            0.02695607766509056,
+            -0.02516009286046028,
+            -0.005232842173427343,
+            0.006245123688131571,
+            0.002093953313305974,
+            0.006914535537362099,
+            -0.006934944540262222,
+            -0.007208424154669046,
+            -0.03657275065779686,
+            -0.0020745648071169853,
+            0.0009005427709780633,
+            0.008963589556515217,
+            -0.008049271069467068,
+            -0.0136494729667902,
+            -0.011135096661746502,
+            0.002410291228443384,
+            0.007134951651096344,
+            -0.02204161323606968,
+            0.0049593630246818066,
+            0.009608510881662369,
+            -0.019559891894459724,
+            0.025127438828349113,
+            -0.010890189558267593,
+            -6.964537897147238e-05,
+            -0.00995954405516386,
+            0.021633435040712357,
+            -0.013486201874911785,
+            -0.018368011340498924,
+            -0.0006898208521306515,
+            0.008355404250323772,
+            0.011926962062716484,
+            -0.0017663904000073671,
+            -0.011371839791536331,
+            0.0003752686607185751,
+            0.01646590232849121,
+            0.019135385751724243,
+            0.002744997153058648,
+            -0.0024837632663547993,
+            -0.01019628718495369,
+            -0.009592183865606785,
+            -0.0073961857706308365,
+            -0.0007449248805642128,
+            0.002091912319883704,
+            0.02035992033779621,
+            0.013469874858856201,
+            0.0024837632663547993,
+            -0.015135240741074085,
+            -0.005257333163172007,
+            0.008939098566770554,
+            0.009412584826350212,
+            -0.011690218932926655,
+            -0.00898808054625988,
+            0.002659279853105545,
+            0.02249877341091633,
+            0.017633290961384773,
+            -0.0014000505907461047,
+            0.0032878739293664694,
+            -0.0016592437168583274,
+            0.012506574392318726,
+            -0.007555375341325998,
+            -0.010269759222865105,
+            -0.006943108048290014,
+            -0.014645427465438843,
+            -0.010800390504300594,
+            -0.0023062059190124273,
+            -0.010179960168898106,
+            0.022629389539361,
+            0.007988044060766697,
+            0.009779945947229862,
+            0.008265605196356773,
+            0.010269759222865105,
+            0.002087830565869808,
+            0.025715215131640434,
+            -0.004424649756401777,
+            -0.011984107084572315,
+            0.007236996199935675,
+            -0.022368155419826508,
+            -0.021535472944378853,
+            0.008257441222667694,
+            0.006367577239871025,
+            0.03595232218503952,
+            -0.026890767738223076,
+            0.0052981507033109665,
+            -0.009779945947229862,
+            -0.010400376282632351,
+            0.007996207103133202,
+            -0.0015439334092661738,
+            -0.004245051648467779,
+            0.022368155419826508,
+            0.00726556871086359,
+            0.0035797213204205036,
+            0.0032817514147609472,
+            0.019837452098727226,
+            -0.005685919895768166,
+            0.01573934406042099,
+            0.0259274672716856,
+            0.024621298536658287,
+            0.009388094767928123,
+            -0.009469729848206043,
+            0.02785406820476055,
+            0.0002895512734539807,
+            -0.0034327772445976734,
+            0.007220669183880091,
+            -0.00305113079957664,
+            -0.0009556467994116247,
+            -0.010033016093075275,
+            0.021764053031802177,
+            -0.001455154619179666,
+            0.004359341226518154,
+            0.02081707864999771,
+            -7.774515688652173e-05,
+            0.006220633164048195,
+            0.01057181041687727,
+            0.002181711606681347,
+            0.003396041225641966,
+            0.010775899514555931,
+            -0.039315707981586456,
+            -0.0006571666453965008,
+            -0.025600925087928772,
+            -0.0018286375561729074,
+            0.0007357408758252859,
+            -0.01266984548419714,
+            0.013600491918623447,
+            -0.002015379024669528,
+            -0.010751409456133842,
+            -0.00029337796149775386,
+            0.021649762988090515,
+            0.005416522268205881,
+            0.004071575589478016,
+            -0.004469549283385277,
+            0.018792515620589256,
+            -0.006955353543162346,
+            -0.017861871048808098,
+            -0.012759645469486713,
+            -0.012367794290184975,
+            -0.01918436773121357,
+            -0.01214737817645073,
+            -0.0024551907554268837,
+            -0.002667443361133337,
+            0.004738946910947561,
+            0.002481722505763173,
+            -0.003228688146919012,
+            -0.02070278860628605,
+            0.015037278644740582,
+            -0.01430255826562643,
+            0.012873934581875801,
+            -0.011053460650146008,
+            0.005963481031358242,
+            -0.004122598096728325,
+            0.012726990506052971,
+            -0.021404854953289032,
+            0.02070278860628605,
+            -0.01536382082849741,
+            0.0038532004691660404,
+            0.019592545926570892,
+            0.006551257334649563,
+            -0.008118661120533943,
+            -0.013404566794633865,
+            -0.022074267268180847,
+            0.021633435040712357,
+            -0.011633073911070824,
+            -0.018237393349409103,
+            -0.004608329851180315,
+            0.009649328887462616,
+            -0.017470018938183784,
+            -0.007367613259702921,
+            0.01707816869020462,
+            -0.0018949664663523436,
+            0.012980061583220959,
+            0.023004913702607155,
+            -0.0052981507033109665,
+            0.0036572751123458147,
+            -0.009167678654193878,
+            0.0010286086471751332,
+            -0.02470293454825878,
+            0.013527019880712032,
+            0.008971752598881721,
+            0.006269614677876234,
+            -0.013935198076069355,
+            -0.013306603766977787,
+            0.0036531933583319187,
+            0.004498121794313192,
+            0.011437147855758667,
+            0.009420748800039291,
+            0.004220560658723116,
+            -0.013918871060013771,
+            -0.027788760140538216,
+            -0.027070365846157074,
+            0.03595232218503952,
+            -0.0005002731923013926,
+            -0.020947696641087532,
+            0.03399306535720825,
+            0.016114868223667145,
+            0.0022980421781539917,
+            -0.012449429370462894,
+            -0.004314441699534655,
+            0.024621298536658287,
+            0.016400592401623726,
+            0.010465684346854687,
+            -0.0004742518358398229,
+            0.005571629852056503,
+            0.0016469983384013176,
+            0.03012353740632534,
+            0.015323002822697163,
+            0.0009459525463171303,
+            -0.038597315549850464,
+            0.011943289078772068,
+            -0.007547211833298206,
+            0.0013408648082986474,
+            0.006702283397316933,
+            0.017780235037207603,
+            0.011935125105082989,
+            0.015567909926176071,
+            0.014865843579173088,
+            -0.003065417055040598,
+            0.0029511270113289356,
+            0.0006163488142192364,
+            -0.009624837897717953,
+            -0.001767410896718502,
+            -0.008881953544914722,
+            -0.005714492406696081,
+            -0.002920513739809394,
+            0.005592038854956627,
+            0.007657419890165329,
+            -0.003983817528933287,
+            0.028049994260072708,
+            -0.0006433906382881105,
+            -0.017600636929273605,
+            -0.009061552584171295,
+            -0.03611559048295021,
+            0.00434301421046257,
+            -0.030939893797039986,
+            -0.011257549747824669,
+            0.0033674687147140503,
+            -0.020457882434129715,
+            -0.024768242612481117,
+            0.0010306495241820812,
+            0.014335212297737598,
+            0.007392104249447584,
+            0.0033327736891806126,
+            -0.01004934310913086,
+            0.003816464450210333,
+            -0.001855169073678553,
+            -0.006461458280682564,
+            0.028833696618676186,
+            0.019559891894459724,
+            0.020457882434129715,
+            -0.01190247107297182,
+            0.019298657774925232,
+            0.005845109466463327,
+            0.02035992033779621,
+            0.010465684346854687,
+            -0.0048695639707148075,
+            0.0074859848245978355,
+            0.009567692875862122,
+            -0.006155324634164572,
+            -0.021519144997000694,
+            0.013192313723266125,
+            -0.0010837126756086946,
+            -0.0052083516493439674,
+            -0.016269976273179054,
+            -0.003534821793437004,
+            -0.015812816098332405,
+            0.010547320358455181,
+            -0.017159804701805115,
+            0.012498411349952221,
+            -0.000802069844212383,
+            0.00168475485406816,
+            -0.015469946898519993,
+            -0.013951525092124939,
+            -0.0012439225101843476,
+            -0.01244126632809639,
+            -0.008971752598881721,
+            0.006163488142192364,
+            -0.007539048325270414,
+            -0.008318668231368065,
+            -0.00535937724635005,
+            -0.011437147855758667,
+            -0.010882026515901089,
+            0.004367504734545946,
+            -0.0010653446661308408,
+            0.020931368693709373,
+            0.00011671340325847268,
+            -0.005824700463563204,
+            0.01190247107297182,
+            0.02787039615213871,
+            0.0016704685986042023,
+            0.004583839327096939,
+            -0.014261740259826183,
+            -0.011118769645690918,
+            -0.02279266156256199,
+            0.016572028398513794,
+            0.0049675265327095985,
+            -0.0344502255320549,
+            0.01438419334590435,
+            0.00824927818030119,
+            0.0015102586476132274,
+            -0.006698201410472393,
+            -0.005334886722266674,
+            -0.005608365871012211,
+            0.006306350696831942,
+            -0.0022837561555206776,
+            0.0005107327597215772,
+            0.0035919665824621916,
+            0.034678805619478226,
+            -0.0040674940682947636,
+            -0.007228832691907883,
+            0.00981259997934103,
+            0.0005130287609063089,
+            0.020196648314595222,
+            -0.005518566817045212,
+            0.0037572786677628756,
+            -0.019282329827547073,
+            -0.016563864424824715,
+            0.0018429238116368651,
+            -0.01722511276602745,
+            0.023282473906874657,
+            -0.005392031744122505,
+            -0.01347803883254528,
+            -0.006906372029334307,
+            -0.020115012302994728,
+            0.012637191452085972,
+            0.02310287579894066,
+            0.007016580086201429,
+            0.01033506728708744,
+            0.012131051160395145,
+            -0.02444170042872429,
+            -0.020457882434129715,
+            0.012171869166195393,
+            0.0013571919407695532,
+            -0.008449285291135311,
+            -0.005404277238994837,
+            -0.001783737912774086,
+            0.007783954963088036,
+            -0.0032633834052830935,
+            0.007881917990744114,
+            0.06491663306951523,
+            -0.01093100756406784,
+            -0.002987863030284643,
+            -0.02966637909412384,
+            -0.004396077245473862,
+            -0.006628811359405518,
+            0.0056287748739123344,
+            0.009151351638138294,
+            -0.023135529831051826,
+            0.009232986718416214,
+            -0.030025575309991837,
+            -0.010628955438733101,
+            0.00824111420661211,
+            0.0028490826953202486,
+            -0.01220452319830656,
+            -0.0009592183632776141,
+            -0.013665799982845783,
+            0.0006107363733462989,
+            -0.009412584826350212,
+            -0.00576347392052412,
+            0.012718827463686466,
+            -0.02321716584265232,
+            0.006298186723142862,
+            0.007592111360281706,
+            -0.006514521315693855,
+            -0.013722945004701614,
+            -0.017861871048808098,
+            -0.011967780068516731,
+            -0.013314766809344292,
+            0.002996026771143079,
+            0.01190247107297182,
+            -0.02607441134750843,
+            0.012922916561365128,
+            -0.0037389106582850218,
+            0.00945340283215046,
+            0.004404240753501654,
+            0.013469874858856201,
+            0.018074123188853264,
+            0.009355440735816956,
+            0.0021429345943033695,
+            -0.008024780079722404,
+            0.012702500447630882,
+            -0.01993541419506073,
+            -0.01635161228477955,
+            0.013837235048413277,
+            0.012016761116683483,
+            0.003061335301026702,
+            0.009828926995396614,
+            0.0020959940738976,
+            0.011886144056916237,
+            0.010775899514555931,
+            -0.019739490002393723,
+            -0.0006648199632763863,
+            0.0007326795603148639,
+            0.03280118480324745,
+            -0.0005938990507274866,
+            0.008857463486492634,
+            0.004420568235218525,
+            0.013159659691154957,
+            -0.012784135527908802,
+            -0.01769859902560711,
+            0.0008087027235887945,
+            -0.01057181041687727,
+            0.021257910877466202,
+            -0.008192133158445358,
+            0.006216551177203655,
+            -0.018612917512655258,
+            -0.027086693793535233,
+            0.006392067763954401,
+            -0.016261812299489975,
+            -0.018221067264676094,
+            0.01317598670721054,
+            0.0077268099412322044,
+            0.0017572063952684402,
+            -0.005098143592476845,
+            -0.020457882434129715,
+            -0.009837090037763119,
+            0.028441844508051872,
+            0.011102442629635334,
+            -0.004120557103306055,
+            0.009584019891917706,
+            -0.009094206616282463,
+            -0.012490247376263142,
+            -0.0017541450215503573,
+            0.0036715613678097725,
+            0.008261523209512234,
+            0.005645101889967918,
+            -0.01386172603815794,
+            -0.00029873527819290757,
+            -0.012400448322296143,
+            -0.021225256845355034,
+            -0.008906444534659386,
+            0.016751626506447792,
+            0.007820690982043743,
+            -0.0038960592355579138,
+            0.015812816098332405,
+            -0.004404240753501654,
+            0.002189875114709139,
+            0.014294394291937351,
+            0.005334886722266674,
+            -0.004285869188606739,
+            -0.0095758568495512,
+            -0.004612411838024855,
+            0.004043003544211388,
+            -0.0025960123166441917,
+            0.006457376293838024,
+            0.017192458733916283,
+            0.0072900597006082535,
+            -0.013755599036812782,
+            0.013135168701410294,
+            0.0007791098323650658,
+            0.0049675265327095985,
+            -0.03699725493788719,
+            -0.003247056156396866,
+            0.015796490013599396,
+            -0.013020878657698631,
+            0.02501314878463745,
+            0.005400195252150297,
+            0.022743679583072662,
+            0.015233203768730164,
+            0.008669701404869556,
+            -0.001542912912555039,
+            0.00464098434895277,
+            0.02759283408522606,
+            -0.003151134354993701,
+            -0.008637047372758389,
+            -0.0020602785516530275,
+            0.009241150692105293,
+            0.0002063850115519017,
+            0.003000108525156975,
+            0.008281932212412357,
+            -0.03562577813863754,
+            0.011706545948982239,
+            -0.019576217979192734,
+            0.0024694770108908415,
+            0.003459308762103319,
+            -0.007183933164924383,
+            -0.014865843579173088,
+            -0.005857354495674372,
+            0.022841641679406166,
+            -0.003224606392905116,
+            0.021078312769532204,
+            0.002885818714275956,
+            0.016106704249978065,
+            -0.011575928889214993,
+            -0.009208496659994125,
+            -0.0004130251472815871,
+            -0.005351213738322258,
+            -0.007539048325270414,
+            0.010596301406621933,
+            0.0034409407526254654,
+            0.005534893833100796,
+            0.002265388146042824,
+            0.0036654386203736067,
+            -0.0036878883838653564,
+            0.02773977816104889,
+            0.003563394071534276,
+            0.007996207103133202,
+            -0.011567764915525913,
+            0.014082142151892185,
+            -0.017029186710715294,
+            0.0070778070949018,
+            0.005575711838901043,
+            -0.026156047359108925,
+            0.016849588602781296,
+            0.004775682929903269,
+            0.006628811359405518,
+            0.013812744058668613,
+            0.01618834026157856,
+            -0.010408539324998856,
+            -0.014816862531006336,
+            0.024049850180745125,
+            -0.0032041973900049925,
+            -0.015225040726363659,
+            -0.006775755435228348,
+            4.438935866346583e-05,
+            0.006559420842677355,
+            0.00373074715025723,
+            0.0013020879123359919,
+            -0.0006204306264407933,
+            -0.010743245482444763,
+            0.004645065870136023,
+            -0.01934763789176941,
+            -0.0007163524278439581,
+            -0.016449574381113052,
+            -0.015633217990398407,
+            0.018678225576877594,
+            -0.01830270327627659,
+            0.009633000940084457,
+            0.0012275953777134418,
+            0.028588788583874702,
+            -0.011706545948982239,
+            -0.012155541218817234,
+            0.0015102586476132274,
+            0.010849371552467346,
+            0.0067512644454836845,
+            -0.010033016093075275,
+            -0.007583947852253914,
+            -0.014180104248225689,
+            -0.010163633152842522,
+            0.015502601861953735,
+            -0.014498483389616013,
+            -0.014343376271426678,
+            0.008914608508348465
+          ],
+          "embedding": [
+            -0.006359657085039302,
+            -0.024594786241399234,
+            -0.02553538924848095,
+            0.005360188465088291,
+            0.012801870977589839,
+            0.021680505862506104,
+            -0.016668048966001273,
+            0.06860852474581301,
+            -0.007122272841157724,
+            0.03471775000705488,
+            -0.009004297848861675,
+            0.01059974144678391,
+            0.02565604599426148,
+            -0.022544127646041287,
+            0.010616674976540785,
+            0.013658194823359737,
+            0.016400329743363844,
+            0.02858546964917478,
+            0.0067612676434693686,
+            -0.04174737193606879,
+            0.03184155110488616,
+            -0.02223105127585514,
+            -0.05506239667974977,
+            -0.0015307551410555752,
+            -0.013169788751379287,
+            0.017681749722384007,
+            0.01682706317229353,
+            0.019652295823816928,
+            -0.05623980369271269,
+            0.030698582851533197,
+            0.02286327124296613,
+            0.016019032539689178,
+            0.005508113353872315,
+            8.58985091215149e-05,
+            -0.019206292853495906,
+            -0.018439290483495144,
+            0.03475023176950151,
+            0.01018612942141759,
+            -0.0252551158857426,
+            0.0334562130666827,
+            0.020462153537485785,
+            0.02260508249714211,
+            -0.03323524989392951,
+            -0.002094251009530707,
+            0.015177442347686773,
+            -0.00018698464350038317,
+            -0.0006788648344273403,
+            0.0007777616893581039,
+            0.013352398615059647,
+            -0.030751111528744813,
+            -0.0317733818445192,
+            -0.006729064081710058,
+            0.009413204899986514,
+            -0.005712894407943592,
+            -0.038961826880527066,
+            -0.002827692206530345,
+            0.017863300537973137,
+            0.07918455388563496,
+            0.048967158919816366,
+            0.018002566325350088,
+            0.040264962620950695,
+            -0.019715100359659787,
+            -0.02211350500879408,
+            0.004918148608475574,
+            0.01062585305866606,
+            -0.014478993521696027,
+            0.0034983131226364672,
+            0.000766228629514787,
+            0.009667268771502067,
+            -0.01558439890697323,
+            -0.04984718709870083,
+            0.029613955432821887,
+            -0.009686854114630624,
+            -0.015279319238124568,
+            0.007460629676325322,
+            -0.006235979350809034,
+            0.012514387473717085,
+            0.02800435263741497,
+            -0.008688925946672778,
+            -0.01236301543374469,
+            0.029755705020188972,
+            0.01158037219446542,
+            -0.005242392206024511,
+            0.009982679027728814,
+            0.052414361714483264,
+            -0.018788500381726227,
+            -0.020335831196027204,
+            -0.08583959943356621,
+            0.029309228216812346,
+            -0.0249285282010673,
+            -0.018929396555426307,
+            0.007710663193334574,
+            0.03096798035958051,
+            -0.017089977064591828,
+            -0.028015107605405817,
+            0.06740703271128606,
+            0.009398795412354399,
+            0.01118769022826835,
+            -0.03041417460855722,
+            0.014776002883150835,
+            0.004319194067673161,
+            0.022507636639041203,
+            0.0053562788646786845,
+            0.007548072258943124,
+            0.011123374308563128,
+            -0.006924362115585165,
+            -0.011118507147092024,
+            0.009563232643295304,
+            0.00657178473678069,
+            0.08313027447380966,
+            -0.004239370156003264,
+            -0.00930406163946907,
+            -0.039202672821898946,
+            0.009761029930556298,
+            -0.02827698465424993,
+            -0.037914533246900244,
+            -0.04496050911855815,
+            -0.017711307900236353,
+            0.01934435542287117,
+            0.004335173027873153,
+            0.03800509969714242,
+            0.031198127357716314,
+            0.031311207941680276,
+            -0.012184553894588813,
+            -0.058985863670856876,
+            0.041442017057707635,
+            -0.012456189292721711,
+            -0.0040966351720394135,
+            0.010849179519959421,
+            0.005243123049347636,
+            0.0011495799963862647,
+            0.003534183676934487,
+            -0.03139340512003631,
+            -0.023876261883905306,
+            0.008382957569752911,
+            -0.004778283217011976,
+            0.03703067589888386,
+            0.06888630688839406,
+            -0.03993532296478411,
+            -0.009472167678222372,
+            0.025246872942105057,
+            -0.01524567912700368,
+            0.01595917939878582,
+            -0.003736723164442467,
+            -0.030836797784967753,
+            0.0026246446163368533,
+            0.008894300082148884,
+            0.08342856279154155,
+            -0.0007476580521704578,
+            0.025149116238550588,
+            0.004972542633631468,
+            0.034495632546556576,
+            -0.009173329764686424,
+            0.014009381543772768,
+            0.004427261668151028,
+            -0.013426138607902904,
+            -0.022234568880321808,
+            0.01976282364689966,
+            -0.03592433789490043,
+            -0.010918278220831188,
+            0.013527180289641174,
+            -0.013023113052242677,
+            -0.008304770302495363,
+            0.003677063799167178,
+            0.0013981822663219421,
+            0.047001163242600175,
+            0.01828332714887986,
+            0.002483375662343689,
+            -0.06820498187523893,
+            0.05295312038995555,
+            0.025038753504342162,
+            0.025286870198436735,
+            -0.0029544808216316115,
+            0.029982752196635428,
+            0.0005600481523874767,
+            0.0128290258663539,
+            0.009023805227424919,
+            -0.0019219166367750596,
+            -0.04482843936478531,
+            0.04523479691409549,
+            -0.004678574027241186,
+            0.010486850804218904,
+            0.006736229865018014,
+            0.02134813656476443,
+            0.028083972673690036,
+            0.016857413440125445,
+            0.0189082447820146,
+            0.03843690613820022,
+            -0.011922027011229613,
+            0.016193433524453224,
+            0.03828207911467065,
+            -0.0013055128404331705,
+            0.043198163443700956,
+            0.024025562893260542,
+            -0.001257851946749882,
+            -0.011726713433348844,
+            -0.021907984099443396,
+            0.02752198420597673,
+            0.004431866336999851,
+            0.015798806714413912,
+            -0.012910782191382367,
+            -0.03180549180206364,
+            0.004550247372715741,
+            -0.022778272841118514,
+            0.031242218240017532,
+            -0.006106984949693483,
+            -0.006259993190497421,
+            0.02187449047033339,
+            0.009334146847221406,
+            -0.009381278960016145,
+            -0.020888853823931134,
+            -0.007502374908776156,
+            -0.010390018386728482,
+            -0.04316693435221676,
+            -0.006274513730720055,
+            0.019152443937064258,
+            -0.01668984824845791,
+            0.018573194036505578,
+            0.0035035262183249925,
+            -0.019779911381789077,
+            -0.0027495798165085993,
+            0.02425481236315913,
+            0.006857723545270935,
+            -0.014733535699087731,
+            -0.026326146471941727,
+            -0.019906660231056833,
+            -0.001033294559612807,
+            0.0159154063026895,
+            0.035649534442431415,
+            -0.026210592096240003,
+            0.014397127229517285,
+            0.005114311681443673,
+            -0.0033104236005207077,
+            0.00324988235167305,
+            0.03764004154553922,
+            -0.017822437760737605,
+            -0.008579701063708334,
+            -0.00032410984200965487,
+            0.018450351071676443,
+            0.017283379764846195,
+            0.006384411611332382,
+            -0.005115090309123598,
+            -0.030590047062583125,
+            -0.011776211359193503,
+            0.01183623914060493,
+            0.0016564847796208331,
+            -0.009530766634361575,
+            -0.004810598258154945,
+            0.03876242515526699,
+            0.027076785774401667,
+            0.005324763136165989,
+            -0.03416412480913939,
+            -0.007943016733586697,
+            0.002114307650456928,
+            -0.02961449347300628,
+            0.05142732770518828,
+            0.05566026433842563,
+            -0.0030729309664974704,
+            -0.00938186948393074,
+            -0.04547908981236149,
+            0.0651216359692231,
+            -0.01813722445944094,
+            -0.0482650853560993,
+            0.044045155892545554,
+            -0.014272653003441165,
+            0.013740652306746851,
+            0.04563732439184395,
+            0.02126513340521316,
+            -0.0054845632439384905,
+            -0.011635625101262898,
+            0.030992082069685376,
+            -0.010051935721068779,
+            -0.050967258971854425,
+            -0.009827942095033494,
+            0.013373878556654464,
+            -0.013017627321534534,
+            -0.009361001542607812,
+            -0.01925867665425919,
+            0.020329052915436022,
+            0.023020741670660188,
+            0.019387043947695172,
+            0.005628632318152202,
+            0.018451555161952665,
+            0.016226845400848455,
+            -0.01384033112836216,
+            0.004762589697426251,
+            -0.012718476561725138,
+            -0.024222765127494463,
+            -0.037487144314010595,
+            -0.019971524166659278,
+            -0.015034588610516958,
+            0.009483936496215338,
+            -0.014301924283553591,
+            -0.0024139665139412125,
+            0.013361045353529458,
+            -0.0005870798721107379,
+            -0.011192319340879789,
+            0.013927695434391159,
+            0.021337539669342293,
+            -0.027300952495815437,
+            -0.04765670695199488,
+            -0.00561706224251562,
+            0.0082776639906139,
+            -0.0030984973749633724,
+            -0.026269016057688548,
+            0.007040524859505573,
+            0.0017859008072463442,
+            0.020091542517503808,
+            -0.018716761130867725,
+            0.0035283051960672963,
+            0.008519383548064053,
+            -0.015247417050336866,
+            -0.016917626868166634,
+            -0.018580550690103854,
+            -0.024697180580294884,
+            0.034202028316737125,
+            0.0204395003463227,
+            0.030392895534267717,
+            -0.04527669147753553,
+            -0.018359428092075348,
+            0.05494107499558624,
+            0.015234754907198422,
+            -0.05066008106884831,
+            0.031139824660661085,
+            0.005016615710206006,
+            0.0019000453624929544,
+            -0.006094364589229031,
+            0.0637324238017742,
+            -0.02711381489292883,
+            -0.02187418906973867,
+            0.004601744864532875,
+            -0.04141121277496072,
+            -0.026689585608019588,
+            -0.005568100750837504,
+            0.01291019036002459,
+            -0.031853041426355644,
+            -0.02815800259372657,
+            -0.007964861641790374,
+            -0.010933515363060793,
+            0.013726639077745378,
+            -0.023956800669647334,
+            -0.015672058590455337,
+            -0.023642025374517057,
+            -0.01288435863544158,
+            -0.0042516881552192385,
+            -0.011162214160460695,
+            -0.02412733256710621,
+            0.010088528028211243,
+            0.03310175458095482,
+            0.023064360903748363,
+            0.001996159684818467,
+            0.009478229572933101,
+            0.014478772877286076,
+            0.00371783700962415,
+            0.007864103610320116,
+            -0.048511515124609,
+            -0.031398754206534776,
+            0.020227872561659397,
+            0.012159811587094092,
+            0.0172163448580163,
+            0.009578952796095119,
+            -0.04154545961112031,
+            -0.05095080393682759,
+            0.02968486144855133,
+            -0.03932395023720674,
+            -0.02063219739061335,
+            0.007512423249865194,
+            -0.011911709468952548,
+            -0.00810951734088105,
+            -0.0008079327452253181,
+            -0.01600701450676789,
+            0.023984909975645632,
+            -0.02418072541081482,
+            0.0033984265728439398,
+            -0.0104343782811971,
+            0.01588977364765743,
+            0.05109529246167992,
+            -0.027800269136366997,
+            -0.027889034481533438,
+            0.03706813210665186,
+            -0.02058120652429308,
+            0.030801396468451574,
+            0.03246127847621613,
+            0.008355517405358826,
+            -0.008605467586247932,
+            0.017693344563968573,
+            0.014499487024471321,
+            0.012431277080478217,
+            -0.040103870071440256,
+            0.017328848567062113,
+            0.02723900253727976,
+            -0.054909074068976875,
+            -0.008547596270650966,
+            -0.00777964838164648,
+            0.027134666572242665,
+            0.02765036594970255,
+            -0.002207173084941375,
+            -0.016838740117653725,
+            -0.028289233739722936,
+            -0.019433376533829132,
+            -0.011053380458946764,
+            0.03247641283550457,
+            0.0007514989009289282,
+            0.007919574219000797,
+            0.001316042436488591,
+            -0.0017934342041762042,
+            0.01924145734197619,
+            0.015260199553572459,
+            0.048259602293134723,
+            -0.03383050622747848,
+            0.0021712225589426266,
+            -0.03508567715696004,
+            -0.04578731401250471,
+            -0.008768228935292693,
+            0.010203318819548585,
+            0.024135236371743855,
+            -0.007467140460133838,
+            0.0028907962385387818,
+            -0.007582083854422456,
+            -0.005384477718912037,
+            -0.01867978482547249,
+            0.01641191108381631,
+            0.037168089880543244,
+            0.0031992495663980464,
+            -0.008533890994695162,
+            0.000778843942291118,
+            -0.004620258538699103,
+            0.010572690335163528,
+            0.033233014627100904,
+            -0.030497362482602697,
+            0.02974571152880676,
+            -0.004274442492343003,
+            0.025341347125910928,
+            -0.04307748954707389,
+            -0.018069609916024403,
+            -0.018491122727274337,
+            -0.02069307802429807,
+            0.05372967226264594,
+            -0.030768978301976772,
+            -0.03662045886993282,
+            -0.003205991669059688,
+            0.016815511161004534,
+            -0.0014152644009276566,
+            -0.0016015439520090104,
+            -0.03984716970293867,
+            -0.010828260334550217,
+            -0.05894985435251326,
+            -0.01638035185618134,
+            -0.021819442096404948,
+            -0.005315102328745982,
+            -0.06251579337157112,
+            0.0033713876515840233,
+            -0.07815974854883057,
+            0.001484860071649606,
+            0.04539427129131382,
+            -0.023530943197353476,
+            0.05142250021536681,
+            -0.024746600229063383,
+            -0.0005885740455257141,
+            -0.044698536284421976,
+            0.0026481001040031785,
+            0.02389178603305878,
+            0.051387657872255674,
+            -0.002949198236157656,
+            0.010674377196609455,
+            0.013322075519401089,
+            -0.026245090721950647,
+            -0.01036724558050635,
+            -0.005321884180696542,
+            0.005958798671856245,
+            0.013755178085972732,
+            -0.0074734329537091675,
+            0.038368016046563674,
+            0.010941421833174982,
+            0.03443115781919206,
+            0.034623617136788874,
+            -0.004361926531008214,
+            -0.03348681614127998,
+            -0.00632870804659212,
+            0.024433331840566692,
+            0.0321998807702602,
+            -0.02899983280320611,
+            0.005958471722097962,
+            -0.015602447981516634,
+            0.015633217539747047,
+            0.006321931116948646,
+            -0.011827541320878442,
+            0.02372204564999124,
+            0.015673426811623124,
+            0.006103300725621424,
+            -0.005165006670545588,
+            0.013589526056189892,
+            -0.025411864292353578,
+            0.03041791947375232,
+            -0.004456865298584878,
+            -0.04276058054583262,
+            -0.0024736078733106825,
+            -0.020059309867945227,
+            0.024883721794000956,
+            0.015578237413840115,
+            -0.00536523508503184,
+            -0.005290645798641171,
+            0.008242987948511412,
+            0.020469675635821673,
+            0.006668354647245447,
+            0.00017729517736814364,
+            0.02373604785121342,
+            0.029915687122900485,
+            -0.02432635861789771,
+            -0.029540533943782886,
+            0.008151297444985344,
+            -0.005730733861939129,
+            0.011318816579801282,
+            -0.02644855229221747,
+            0.00033932872948409137,
+            0.0035505684154028996,
+            -0.03570104343808711,
+            0.03408366016010711,
+            -0.009199700348630527,
+            0.005857598530788056,
+            -0.009538177736472163,
+            -0.0017768806148210118,
+            0.0027410153099351694,
+            0.03898108970435115,
+            -0.007367623392751247,
+            0.009827254226889936,
+            0.015465895035342481,
+            0.000981604396724216,
+            0.05143083558797524,
+            -0.027350877485564314,
+            -0.02183036962301852,
+            0.01885468378168636,
+            -0.007285239007489837,
+            -0.016846837665812908,
+            -0.020546383007536074,
+            -0.011863943814684079,
+            -0.016293192750336218,
+            0.01976516340568883,
+            -0.02387328798681798,
+            -0.011892336568452852,
+            0.007170651391639504,
+            0.0013725014050721622,
+            -0.009809669068563384,
+            0.0016617183704652932,
+            -0.006307590349798603,
+            0.02037693835615797,
+            0.031753656615094446,
+            -0.024892675105483313,
+            0.03577919637821846,
+            0.03178303188315581,
+            0.017260898395251155,
+            -0.02102289690701718,
+            -0.009661035772073294,
+            -0.00048542689442823136,
+            -0.014694631967051945,
+            -0.0326379186158631,
+            0.025443206465204822,
+            0.017194626456982255,
+            -0.011381421003142453,
+            -0.03680783761649673,
+            0.0017216115811536721,
+            0.022819859205829785,
+            -0.013600989058799166,
+            0.028141657788496834,
+            -0.01869047298692401,
+            -0.01366833755869951,
+            -0.00782959271979894,
+            0.031227914441058244,
+            -0.05252392878125891,
+            0.016957842026348195,
+            0.00020880696574616416,
+            0.001154370575234767,
+            0.010892067708194324,
+            -0.008368819756628046,
+            0.012224564683325483,
+            0.031700389750709,
+            0.018063405869147564,
+            0.0010794115307202266,
+            -0.002783668216641561,
+            0.01418234057248712,
+            0.017773269840982066,
+            -0.015653443355569053,
+            -0.000497740485657824,
+            0.019890882604100452,
+            0.022672054781919238,
+            -0.00521797370595221,
+            -0.016686942830774276,
+            -0.018443538351123584,
+            0.025270057639201107,
+            -0.010354431070738671,
+            0.03484047753830186,
+            0.04594785214094975,
+            0.0036441959528283004,
+            -0.05695476763071743,
+            -0.009412209619190693,
+            0.0010371110229966086,
+            -0.01099343927053564,
+            0.01606863921935641,
+            0.012470660511773616,
+            -0.03237187923799263,
+            0.006388672213167589,
+            0.012946570716567364,
+            0.022152554488829627,
+            -0.02540317621005867,
+            -0.023183252921885226,
+            -0.0030984019855787655,
+            -0.007890467768631811,
+            0.018648760402624687,
+            -0.008653405980661855,
+            0.0006228315253506687,
+            0.009672526897422602,
+            0.0005436494014569835,
+            0.011632326145022935,
+            0.005655942007108228,
+            0.021736827172951,
+            -0.001256608167230203,
+            0.013124413886287987,
+            -0.01986808920858981,
+            -0.0270450339809516,
+            0.006500423802582728,
+            0.04438242119211055,
+            -0.014813399694814117,
+            0.030621709833336064,
+            0.0092171224268983,
+            0.016433384739642534,
+            0.024875072913970443,
+            -0.00891470467241686,
+            -0.014124009014664287,
+            0.00845777541223868,
+            0.009019007848927453,
+            -0.04147358041382793,
+            0.012513030150663913,
+            -0.01710243399313324,
+            -0.033144939719206276,
+            0.01904674924133127,
+            -0.00534122554342279,
+            0.012367320751926314,
+            0.004270315800911569,
+            -0.002886460711688153,
+            -0.025696286516581048,
+            0.014137901708875033,
+            0.028135087538950946,
+            0.021797869775098042,
+            0.0296173052503884,
+            -0.012757078368361444,
+            0.0005460601780756187,
+            -0.008748930817524899,
+            0.005470975782187157,
+            0.04187083128854477,
+            0.0008687182830616019,
+            -0.0008032802520818105,
+            0.01507726552487313,
+            0.024945744517309208,
+            0.0376130540370386,
+            -0.015415560911023553,
+            -0.03407353556457147,
+            0.01564969972264396,
+            -0.010104443707865262,
+            0.018855792875436674,
+            0.005503259932317811,
+            0.0049634097823351596,
+            0.0038600156268000702,
+            0.019312419743287703,
+            -0.006635193789316492,
+            0.018798045806429395,
+            -0.02500983683871332,
+            -0.028785757874583755,
+            0.01829411751138161,
+            -0.005277972443258226,
+            0.03338623299668019,
+            0.016225885095995672,
+            -0.029522280454919586,
+            -0.01712097344863975,
+            0.01553706187924322,
+            -0.008465766956343768,
+            -0.024842700926275538,
+            0.01895228828512998,
+            0.007076862934523977,
+            0.008956105593078965,
+            0.0024373779389833218,
+            -0.01106503696703573,
+            0.01125466547367417,
+            -0.000818223040067914,
+            -0.015322790791698395,
+            0.014157131555941659,
+            -0.017853386731281715,
+            -0.014994640991761465,
+            0.01775991780643474,
+            -0.037498078532902435,
+            0.009570197712419421,
+            -0.007687297543935123,
+            0.009050494195378254,
+            -0.004222041469820445,
+            0.007460383234429904,
+            -0.0010804379186060264,
+            -0.014987857934522812,
+            -0.014903749199810126,
+            0.03228485430019625,
+            0.022096121825475306,
+            -0.000410816936759558,
+            0.006728733918931589,
+            -3.971636461500232e-05,
+            -0.026719848032785824,
+            -0.011253751790407687,
+            0.02245467425830765,
+            -0.005664635254398481,
+            0.005134496095445943,
+            -0.01109590855846966,
+            -0.0149616794609307,
+            0.0019726017255571673,
+            0.032668637270108894,
+            0.06066635618119076,
+            0.012001732730659747,
+            0.024422093585231484,
+            0.0189499396761026,
+            0.02119276033547528,
+            -0.008489818917377516,
+            0.006413901834269739,
+            -0.03097413287789443,
+            0.03509225573986143,
+            0.005392594635384506,
+            0.03108937318306421,
+            0.026664331248813712,
+            -0.04722525921137681,
+            -0.0058539171830594774,
+            0.017087531454837677,
+            0.022673712717885323,
+            0.0159020681868943,
+            0.0008382498574190522,
+            -0.008938174268517353,
+            0.035650940044285984,
+            -0.03315250299348084,
+            -0.023647111263155,
+            0.005059382024092534,
+            -0.01439537467317451,
+            -0.010488359956731476,
+            -0.006785295580877329,
+            0.02776449681317373,
+            -0.005584052419598926,
+            -0.004582019433008779,
+            0.009697064252638535,
+            -0.014400206558190682,
+            0.008165676294736836,
+            0.0007624778531368436,
+            0.008681263897172326,
+            0.039249866981334126,
+            0.0005918063050375113,
+            0.024275699182767528,
+            -0.004138313072843555,
+            0.011079529739650594,
+            -0.0158972266904151,
+            -0.016985058543605,
+            0.02276982363172991,
+            0.00351101120496393,
+            -0.03244126813432563,
+            -0.00768819222024939,
+            0.00928180979884261,
+            -0.0013793404539641824,
+            0.005266058566444064,
+            -0.00503433670609399,
+            0.003747448261046857,
+            0.03286230666764075,
+            0.0022383592721685153,
+            0.008606190020480513,
+            0.013593947677257625,
+            0.00030951915677929204,
+            -0.01490307680740127,
+            0.013254591138113545,
+            -2.1379308136707722e-05,
+            -0.012900073369514622,
+            0.03357519186606058,
+            0.006705902094864628,
+            0.04243347200485736,
+            -0.03302953674711988,
+            0.009031534935013593,
+            0.01822227379789576,
+            -0.008491770335368761,
+            -0.02725412856390648,
+            -0.03630788717786838,
+            0.008259145747740535,
+            -0.019003260334340084,
+            0.015781275447792516,
+            -0.010039277287105153,
+            0.029665124310477384,
+            -0.03563370794521629,
+            0.004714858110268169,
+            0.004053913355055313,
+            0.0009503673086552651,
+            0.02068360255395961,
+            0.018524093113914366,
+            -0.002785698934204024,
+            -0.031443835341186485,
+            -0.02066365433385339,
+            0.006869321907853112,
+            0.02431260052207595,
+            -0.010367048093621498,
+            -0.005612149485859478,
+            0.0006316058139530722,
+            -0.024369562712568606,
+            -0.016845124984022828,
+            0.030359419843284823,
+            -0.040974427176153246,
+            0.005252911449822132,
+            0.04272936053727608,
+            -0.04599092715016919,
+            -0.03802288716943514,
+            -0.002314988728575789,
+            -0.024124014894379466,
+            -0.006360771882090754,
+            -0.002068030981218464,
+            0.016669991113175926,
+            0.0036232707799735537,
+            -0.028763764013482795,
+            0.00943401978066451,
+            -0.005173988601101547,
+            0.003971420424839974,
+            -0.0031669592494977393,
+            -0.02216336363004636,
+            -0.01657772032717298,
+            -0.02531962163186398,
+            -0.035174102848865116,
+            -0.0015903434876311088,
+            -0.005158703150821607,
+            -0.028703075279214722,
+            -0.03941940849540492,
+            -0.021696000925262537,
+            -0.0160221488026189,
+            -0.00843213135542359,
+            0.040641882367348774,
+            0.021319060666253945,
+            0.019336844555345886,
+            0.0025252167889382917,
+            -0.020346900609125973,
+            0.049440028814233025,
+            -0.007642182460707816,
+            0.035628691886108926,
+            -0.0017882888000877158,
+            -0.013461669667950893,
+            0.00455558389906464,
+            0.03825428026120858,
+            -0.014248434654950971,
+            -0.0009902062564027718,
+            0.00027811655483232545,
+            -0.0011866253757890136,
+            0.0008535155950871286,
+            0.015433650396777696,
+            0.001307651096343185,
+            -0.007112199543314309,
+            0.030380673552683906,
+            -0.012803247194895328,
+            0.01637501752192592,
+            0.018350814869962304,
+            -0.0013359719137099667,
+            0.008064546833360553,
+            -0.021430659268086814,
+            -0.00731572118780415,
+            0.0044578576331236325,
+            0.008422210247967642,
+            -0.015096885054883586,
+            -0.009964588397149773,
+            -0.02359494467299634,
+            -0.01174145797396426,
+            0.0023855749166744366,
+            -0.004559202896782688,
+            0.004902138778592298,
+            0.034595817216889944,
+            -0.016068744500427584,
+            -0.035106033294403614,
+            0.009997207702122002,
+            -0.02663088875167486,
+            0.02222306406130687,
+            3.983384942889326e-05,
+            0.006838221769859366,
+            -0.0023434342406307616,
+            0.001765445386886839,
+            -0.00169484278667461,
+            0.002862331185439823,
+            -0.018217975204395134,
+            0.0031859734246840257,
+            -0.0032119988065673565,
+            0.0009052950782312657,
+            -0.0012459653052875638,
+            0.00687339892509406,
+            0.006332799551164718,
+            -0.0008019538898592668,
+            -0.002968451740267742,
+            -8.541086355097327e-06,
+            -0.020561816931299912,
+            -0.021642512405352372,
+            -0.028259049261410905,
+            -0.000989887278779396,
+            -0.012359733017834338,
+            -0.027804394906433664,
+            0.011443385424045703,
+            -0.006872684999639775,
+            0.008327844771213648,
+            -0.008969527138034904,
+            0.006219994886600349,
+            -0.008415106130309062,
+            -0.0004143996206881767,
+            0.013119286511794273,
+            -0.02925209282917943,
+            -0.02096907906398536,
+            -0.03096513357712367,
+            0.01762223841790055,
+            -0.00925741670612824,
+            -0.002656382260603827,
+            0.0025223527960181835,
+            0.005092659166535706,
+            -0.033606542487570154,
+            -0.006653438998497412,
+            0.007313682256676618,
+            0.009547290751154595,
+            -0.03287788897629887,
+            0.02446728061355753,
+            0.017023222076526946,
+            0.01874233066565012,
+            -0.02417566312664911,
+            0.04049827067897005,
+            0.014426879325647042,
+            -0.021740606525480655,
+            0.016712907564723732,
+            0.027385150577780248,
+            0.03177163221499527,
+            -0.03469098547555545,
+            0.004481376248622418,
+            -0.01518200063132508,
+            0.018918680020516044,
+            -0.025284921244460287,
+            0.020625170445627613,
+            -0.016707629602243063,
+            0.021278719363178687,
+            0.03520334001264928,
+            -0.005524369131116028,
+            0.03608133014180745,
+            -0.019234548780647943,
+            0.0059985801736444555,
+            0.004041778461022664,
+            -0.01645503683802124,
+            0.01024694092614158,
+            -0.0042700351372843386,
+            0.0006059159967286885,
+            0.009406090340935747,
+            0.019133229156115282,
+            0.019007219188041775,
+            -0.04442208547053466,
+            0.028588965123434923,
+            0.00804120101594811,
+            0.021472086070876265,
+            0.014177820346835521,
+            -0.02708807970982664,
+            0.004742981711172693,
+            0.002072673190279628,
+            -0.004659713164250794,
+            0.0061167904730676514,
+            -0.013796604518930328,
+            -0.034540857808309226,
+            0.006338792319814448,
+            -0.018918703778187926,
+            0.015036937364271984,
+            -0.009026670935442384,
+            0.0011667525636770179,
+            -0.01152652811400482,
+            -0.011938679695598347,
+            0.0006170592505446954,
+            -0.005495151258143897,
+            -0.0008075736908949407,
+            -0.03171721592827769,
+            -0.04427658539412448,
+            0.02036132271908089,
+            -0.0021221965958423804,
+            -0.04320396743189005,
+            0.004111179168317842,
+            -0.025352428814134187,
+            -0.01865918186335911,
+            -0.011237699157960728,
+            -0.025047923887906474,
+            -0.04219630696740606,
+            -0.011108496740166623,
+            0.007785006066683012,
+            0.016810642610918525,
+            -0.009081815651299525,
+            0.040965574143584864,
+            0.0073944364393447965,
+            0.015313576815177263,
+            -0.017731360025694955,
+            0.026137777790243826,
+            -0.019314392060123,
+            -0.03108486330987073,
+            0.02676829566273746,
+            -6.259161089416372e-05,
+            0.003053748984540415,
+            0.014358356168574803,
+            2.8186924071913628e-05,
+            -0.006181789640967465,
+            0.01947730374003864,
+            -0.028739300217145797,
+            -0.023460503178221397,
+            0.0012460072421725688,
+            -0.008850395570726385,
+            -0.016271242375881602,
+            -0.035896509136022194,
+            0.005227881053956234,
+            0.03324337105109496,
+            0.004970419987430122,
+            0.008273497228454482,
+            0.01814500594735864,
+            -0.02229593571615308,
+            -0.024230675455380678,
+            0.000639112529814716,
+            -0.010266640429960994,
+            -0.010257247303470066,
+            -0.021302483296731352,
+            0.019150757066356872,
+            0.01651375496287087,
+            -0.00390189952706486,
+            -0.006971692709239087,
+            -0.010621003777733253,
+            -0.01918206725910585,
+            -0.02020345113211036,
+            -0.013827322378214754,
+            -0.017980206764311026,
+            -0.034153741666571154,
+            -0.032195001867229935,
+            0.03225205339201753,
+            0.026354817588919596,
+            0.0014076409241492998,
+            -0.007959081015805435,
+            -0.023070419006792155,
+            -0.004483328183581872,
+            0.02879516595345542,
+            0.002549934782150634,
+            -0.0001541849716285838,
+            -0.00830090759167373,
+            0.015341095409025125,
+            0.00835695019244165,
+            0.005107844984733648,
+            -0.06676974053142512,
+            -0.006027540973808174,
+            0.01655283585235268,
+            0.01351155740684989,
+            -0.002359114145814062,
+            0.02591301169926259,
+            0.01108099264396926,
+            -0.015361038608114896,
+            -0.019765743462262668,
+            0.035195761515422364,
+            0.02408910733949848,
+            0.023468767772473746,
+            -0.0034492999385544727,
+            -0.004472311490618405,
+            0.007030833258457338,
+            -0.010684252427867153,
+            -0.013326964647936429,
+            -0.004558513842504658,
+            0.026925146859337397,
+            0.016099443021514896,
+            -0.0064331198280825575,
+            0.010848931851939124,
+            -0.0008320209125061849,
+            -0.019444491995041537,
+            -0.001841330829156697,
+            0.008730275423516445,
+            0.0012974082030866743,
+            -0.014611902493400719,
+            -0.005911807768916134,
+            -0.019130604101278263,
+            0.024719468317569562,
+            0.0209456329723249,
+            0.03381727436368338,
+            -0.013237902591972769,
+            -0.0003695142331921278,
+            0.005410360755058108,
+            0.0002841632253252816,
+            0.0023352645111704958,
+            0.004801123870325242,
+            0.02623649813035263,
+            -0.029010710171522985,
+            -0.020741774066460415,
+            0.0068426980682519505,
+            0.013445270043414486,
+            0.0022744032984779294,
+            0.033840485942696216,
+            0.035040080489222324,
+            0.004166066423508316,
+            -0.010046666684988869,
+            -0.006148765564658713,
+            0.007071740742464309,
+            0.01865897335451031,
+            -0.023267411753900076,
+            -0.0005936657905496225,
+            0.004833030519903764,
+            -0.004229582648167378,
+            0.03620610659071198,
+            -0.009961238880412731,
+            -0.007197670122096852,
+            0.022212791947347127,
+            0.09819205436926473,
+            0.00047695621444943017,
+            0.0025354521225913395,
+            -0.029358210234742176,
+            -0.023614016211567963,
+            -0.0012564198366457442,
+            0.00242680708457405,
+            0.00830167274308547,
+            0.028076260517106013,
+            0.0004852400049405906,
+            0.011579274037419117,
+            0.02773972145320728,
+            -0.017217305534161797,
+            0.006752070301141707,
+            -0.011376051474975524,
+            0.024577071515205502,
+            -0.012694034026935636,
+            -0.0019809719792138673,
+            -0.016512649783050146,
+            0.015907402258634816,
+            0.004556574687833768,
+            0.004907510649166062,
+            0.011347630984408937,
+            0.009824484201337644,
+            -0.02650731900265866,
+            0.01756773113723491,
+            -0.005671986127099093,
+            0.004449611508541289,
+            -0.027841346572384462,
+            -0.020124233970502267,
+            0.0025986391930124847,
+            -0.0014440758308989809,
+            -0.009579444248630085,
+            0.006005274872617889,
+            -0.0024984235450999127,
+            0.03765664613203687,
+            -0.03765496603611941,
+            0.01409350051861866,
+            0.0066453175608372555,
+            -0.0015390841089599882,
+            0.011806653942640485,
+            0.017306914776093522,
+            -0.008206256939290151,
+            0.02589800448884443,
+            -0.00798213351353373,
+            0.009846274212000523,
+            0.015230070803055296,
+            -0.03449531102975745,
+            0.009174739184806312,
+            0.010145762060174144,
+            -0.024165660737765746,
+            0.012146877603301336,
+            -0.013784371016298891,
+            -0.004818759788054113,
+            0.005930446700613062,
+            0.007349837554764034,
+            0.010733731974181452,
+            -0.02295571464086674,
+            0.016176424826629898,
+            -0.010947841926210105,
+            0.0037766665765835445,
+            0.013559846998199008,
+            0.024892849808979323,
+            0.0012426907454129122,
+            0.0005950515136086193,
+            -0.01379469431504576,
+            0.005534724429523084,
+            -0.015667467221535825,
+            -0.022401586867694584,
+            -0.014192041775234079,
+            -0.006366846766730035,
+            0.01029598687444803,
+            -0.003972386708834858,
+            0.016971476088808737,
+            -0.010309648640697812,
+            -0.010645860303528099,
+            0.0037818310327362103,
+            0.01040806357838676,
+            0.008316605962234849,
+            -0.02855914079113881,
+            -0.005676310556150579,
+            0.004961084138163161,
+            0.00029285971503001265,
+            0.009592497954559435,
+            -0.014556073011576487,
+            0.013689526487160009,
+            -0.013290113354287095,
+            -0.0010726537498214612,
+            0.013201860397060236,
+            -0.012443552043930505,
+            -0.004012305744087929,
+            -0.01592905426965635,
+            -0.028487066798154617,
+            0.008462143660968396,
+            -0.01365614159523485,
+            0.001076882584445294,
+            0.0019503240125505646,
+            0.004730085646042709,
+            -0.016919069570642004,
+            -0.007860046623166481,
+            -0.033515448866521046,
+            0.011065416478687504,
+            0.02314589687882036,
+            -0.006435733825418645,
+            -0.030351727858789472,
+            -0.009546333526852862,
+            0.011327489786114963,
+            -0.0071765122743172715,
+            -0.01676736383051139,
+            -0.0029390038145944096,
+            0.007617910650439855,
+            -0.008174620754097493,
+            -0.00508943241317381,
+            -0.0025173969010414098,
+            0.0217328037301181,
+            0.016094975605203832,
+            0.01769642142712967,
+            -0.021400862603034036,
+            -0.00976992221747187,
+            0.026043879119776864,
+            -0.03275550610951898,
+            0.012441465975914529,
+            0.024587446266013183,
+            0.003446527051428522,
+            0.005492228054838714,
+            0.0030377554109693714,
+            0.005459159163772825,
+            -0.006985179226831048,
+            0.019925583798328282,
+            0.010870421035240845,
+            -0.019709062313471467,
+            0.005385443078244229,
+            0.02086085539375208,
+            0.0037248737640193097,
+            0.012892299156840998,
+            -0.017546870090541213,
+            -0.0020341807891543716,
+            0.015594779287017958,
+            0.04816214672670215,
+            -0.0032851797178091685,
+            -0.010450846098750195,
+            0.029513742590677183,
+            0.023526004023637855,
+            0.005373455940309873,
+            0.0012120777808635214,
+            -0.02525630456877349,
+            -0.013075130701774785,
+            0.01962235086017781,
+            0.016944870494393874,
+            0.013484925533584295,
+            -0.0023255656480479004,
+            -0.019236081983457366,
+            0.021520156296467717,
+            -0.01701733319731161,
+            0.012422659010697477,
+            -0.023764941037007197,
+            -0.001490790833263327,
+            -0.0035419008954878724,
+            0.021998949056452454,
+            0.0021819635638621092,
+            0.008891505304514552,
+            0.024354552127458608,
+            -0.006978548952551799,
+            -0.019526615985982355,
+            -0.0051799020400150305,
+            -0.014578953088901798,
+            0.022506635912517146,
+            -0.024117708614271965,
+            -0.007008641019892289,
+            0.011379795968360019,
+            -0.012362093847309038,
+            -0.0102534051092702,
+            0.001164934138876502,
+            0.017686807394319725,
+            0.0009423993435129515,
+            -0.014784508280754078,
+            0.01944445593807992,
+            -0.0011359271135469456,
+            0.010410915597949927,
+            -0.01921419010977189,
+            0.009390638453648025,
+            -0.016682442516992777,
+            -0.007849184048209236,
+            0.010343008873053859,
+            0.0012604521199837046,
+            0.003948300703645506,
+            0.002592490121754231,
+            0.007773437307969315,
+            -0.03355658426684504,
+            0.022695505487314516,
+            -0.010751606023585792,
+            -0.003538523688786008,
+            -0.0014125330167969218,
+            -0.01678692701672328,
+            -0.0006918162525881059,
+            0.005816502852598842,
+            0.010878358093847386,
+            -0.016045147059003256,
+            -0.00873270513444376,
+            -0.04920244395446931,
+            -0.030314009649638693,
+            0.023156333024481057,
+            0.013733657599720879,
+            -0.010971506342695331,
+            0.018852159415605117,
+            -0.023760853159532587,
+            -0.010587249404421732,
+            0.0011379869499042905,
+            0.023319856329056813,
+            -0.03539290744576506,
+            0.01215369703816749,
+            -0.014493682360886549,
+            -0.02039288669534806,
+            0.009070984596830142,
+            0.011445354735283254,
+            0.0007259852045513916,
+            -0.0011514627555568814,
+            -0.029350945123925453,
+            -0.010246845821727087,
+            0.02391068177425878,
+            -0.00861718589055971,
+            -0.004723044986826333,
+            0.024716520302409263,
+            0.0033721801437248873,
+            -0.012524663490691517,
+            -0.017955758068233754,
+            0.021612671655463236,
+            -0.026077203994996058,
+            0.007645944309444457,
+            0.010032183011253845,
+            -0.0173991263874487,
+            0.021649599230588117,
+            -0.007529701797485131,
+            0.020327763440441377,
+            0.004227390938169872,
+            -0.011249233530251522,
+            0.020352283974471443,
+            0.013644443758970054,
+            0.004216958800094986,
+            -0.004081669404784771,
+            0.006869353098014179,
+            0.02212732880780841,
+            -0.011021739614266063,
+            0.02219580717345023,
+            0.028777213295471067,
+            0.00017328191188631918,
+            0.03260179554743322,
+            -0.021287655520443285,
+            -0.03159728983004563,
+            0.021740163175540124,
+            -0.01035146858119169,
+            -0.021564906188712388,
+            0.028914245422221707,
+            -0.013674731836508768,
+            -0.00786476052825481,
+            -0.0166718429993922,
+            0.00533027311342322,
+            0.016680336311088843,
+            -0.005246190451159961,
+            -0.014360934066628332,
+            0.0029628103834216345,
+            0.019809487418628386,
+            0.014587234544007958,
+            0.029780009648869975,
+            -0.010778199815920887,
+            0.035180242185168895,
+            -0.002320576872571577,
+            0.008990157317473909,
+            0.012415465232073404,
+            0.006489435152972723,
+            -0.011251531176923445,
+            0.013501894212117511,
+            -0.003446759011266526,
+            0.008230548711588447,
+            -0.001449023474280409,
+            -0.035111679577202855,
+            -0.013709382199848678,
+            -0.04709294504969336,
+            -0.0023752976043235697,
+            -0.0025098360125255215,
+            0.0030975417740913154,
+            0.09736815869389318,
+            -0.005723049647039619,
+            -0.011988817571623457,
+            0.001625059666070763,
+            0.009611697638805556,
+            -0.0009896494420056814,
+            -0.007606836952841873,
+            0.015973152482488318,
+            -0.01343292635775299,
+            -0.02140928016038583,
+            -0.010532701190806147,
+            -0.01288594454368977,
+            0.012079198190522715,
+            0.015929716343161652,
+            -0.0005947359502228821,
+            -0.015207331663761012,
+            0.006524010655124294,
+            -0.004566385899752684,
+            0.0064717704997400245,
+            0.008647099975176308,
+            -0.010414668624343924,
+            -0.0071698879102925555,
+            0.030899143952119196,
+            -0.025301269549419674,
+            -0.024361748354052457,
+            -0.0010579931222861095,
+            -0.009055796737486008,
+            -0.010335459737012047,
+            -0.002692653633719985,
+            -0.0017679501322462056,
+            -0.010519994985692081,
+            -0.010019865263564572,
+            0.006537076627706128,
+            -0.015472130358273033,
+            0.004812857776601667,
+            -0.0047457011269495634,
+            -0.013296069496526007,
+            0.017054968086538724,
+            0.022563785357424853,
+            -0.02841491200807702,
+            0.0006755428946168761,
+            -0.017422510783648978,
+            0.012941211381006863,
+            0.008866069010589792,
+            0.006049618782357251,
+            -0.022451528209097505,
+            -0.023622906582403243,
+            -0.015019272754405791,
+            -0.01927535750568903,
+            -0.024973490129133732,
+            -0.009073381957684092,
+            0.0067692803507600945,
+            -0.001267339788153158,
+            0.021918281111033645,
+            0.000816538714279555,
+            -0.01812545613159305,
+            0.0061084913570894905,
+            0.000568371220952269,
+            0.018292497018668842,
+            -0.027468677288882186,
+            0.0033806422195878524,
+            -0.02307582511848465,
+            0.033508234091571124,
+            0.0039195050746594975,
+            -0.026680111514794824,
+            -0.021558668725755275,
+            -0.003311830287529699,
+            0.018994577319066473,
+            -0.0025745912019675805,
+            -0.00724049447539589,
+            0.014775207617625293,
+            0.0155334760323838,
+            -0.004693118176196483,
+            0.002015625406296403,
+            0.027968079002437102,
+            0.01077998866180723,
+            0.0313892866933904,
+            -0.015236169297590675,
+            0.004945828495617267,
+            -0.027215016713379062,
+            0.010214419950212732,
+            -0.01503548137805401,
+            0.008859554812646242,
+            -0.01290890682557532,
+            0.03531384210182567,
+            -0.012005613861257432,
+            -0.004250698950192301,
+            0.03253288796441852,
+            -0.029150986393345252,
+            -0.0035020051536906334,
+            -0.009255437891304041,
+            -0.004142204892160768,
+            0.013240580426083011,
+            -0.001198093874819309,
+            0.008485141356876356,
+            -0.01854370462655459,
+            9.726096864073609e-05,
+            -0.003221852304410318,
+            0.005404595442127598,
+            -0.02005458688172967,
+            -0.003314118809778734,
+            0.0073427568419969814,
+            0.004259518237708634,
+            0.002408630356035373,
+            0.016420646142725346,
+            0.006662121128241274,
+            -0.0029357701493526907,
+            0.03424224243607692,
+            0.008171895850772245,
+            0.0152521424877615,
+            -0.0018638264193202728,
+            0.0030234211261443034,
+            0.019287963516760636,
+            0.019912956661192856,
+            -0.00048443399729935116,
+            -0.01058935153883094,
+            0.00195113175754497,
+            -0.001140732137433033,
+            0.011048741602327437,
+            0.024396046054912645,
+            -0.021158025221644738,
+            -0.02419562092713724,
+            0.016377661005279716,
+            -0.015076145951740961,
+            0.01903712233378994,
+            0.010541463112889985,
+            -0.010099351302095792,
+            -0.00031558616108876247,
+            -0.042316016201379245,
+            0.010964989031040692,
+            -0.02068939391415103,
+            -0.04004120028915868,
+            -0.00244383573783919,
+            0.0009542507389088937,
+            -0.029439456243505052,
+            0.02822814855547978,
+            0.0023755032182188156,
+            -0.017280221740813364,
+            0.018223709137876153,
+            -0.005548654680726505,
+            -0.0024040301239229064,
+            0.005397711787238679,
+            -0.01092026090471121,
+            -0.010648485086769927,
+            0.008235250252660972,
+            0.010957715075632535,
+            0.005425683656535705,
+            -0.005698919020608095,
+            -0.011284911899894062,
+            0.009975882651638084,
+            0.012318901214538077,
+            -0.0019922884373083155,
+            0.010050520101815593,
+            0.0031720752428247446,
+            0.007352859750323051,
+            -0.0073483049386141005,
+            0.0046483587496626965,
+            -0.005093220128411125,
+            -0.026080057085901446,
+            0.012863580653903103,
+            0.0052434400313520774,
+            -0.011863360599637747,
+            0.0103209502958491,
+            -0.01048162756158923,
+            0.005787157766911021,
+            0.001318945460232246,
+            -0.0059333230422061395,
+            0.004821320238893602,
+            -0.0038922239931512784,
+            0.011267634571983924,
+            0.013256611037442042,
+            0.004592529900805719,
+            -0.013076954459595086,
+            -0.02177318635989856,
+            0.009001291224623467,
+            0.01959767163356408,
+            0.024161754192119852,
+            -0.030416109293488064,
+            0.003784634588325551,
+            -0.0014915849060829743,
+            -0.0020550564975080885,
+            -0.015632785947151753,
+            0.01587790981601866,
+            0.01716742056401084,
+            0.012721563202528083,
+            0.01893393006967701,
+            -0.004009617703771589,
+            0.010539100665843704,
+            -0.02080068888791658,
+            0.013642451766699067,
+            0.0008609869904066794,
+            -0.013629067231663643,
+            0.0014620655123815298,
+            0.027046304278766087,
+            -0.011056783825898616,
+            0.006206969699250696,
+            0.008097527175511791,
+            0.022111870736907183,
+            -0.012509479100313492,
+            -0.003735868296238069,
+            0.01516254968167132,
+            0.0014878674796313539,
+            0.0035887079417890352,
+            0.016533149527222273,
+            0.04729052276110659,
+            0.00102751186013391,
+            0.0016713773449738153,
+            0.0192177920520243,
+            -0.005563519063161466,
+            -0.01114821419109643,
+            -0.003549999863325982,
+            -0.021807893906469106,
+            0.010998558936481785,
+            0.00431279785115251,
+            -0.02186324212032519,
+            0.01024093366826206,
+            0.0125495473550584,
+            0.005268320135120931,
+            -0.0031679212410026603,
+            0.02574575247827441,
+            -0.019807879421996258,
+            0.014935295894965128,
+            0.03329450628384787,
+            0.00493356778223407,
+            -0.011423949333937285,
+            0.007263264249871384,
+            0.0027354459678742013,
+            -0.004494406041906684,
+            -0.00606745413660583,
+            0.010497391290734088,
+            -0.008191702034391382,
+            0.009870458191510176,
+            -0.009580130479693,
+            0.039202091799313755,
+            -0.009619018331262844,
+            -0.010782138248435458,
+            -0.004558645539551881,
+            -0.006241773108460662,
+            0.003478414182938609,
+            -0.003395165694642073,
+            0.005812225135633631,
+            -0.0069045407353789945,
+            -0.014607146220271209,
+            0.0008522982757210454,
+            0.00020175973815948303,
+            0.014001922919116725,
+            0.0011804904415274088,
+            -0.004871874532755353,
+            -0.007877217154706205,
+            0.015243277056007988,
+            -0.006047906208304912,
+            0.0006531937033317192,
+            -0.0011653523616313422,
+            -0.0013662556399359335,
+            -0.007401050012179761,
+            -0.015166105795085528,
+            0.00974609244984173,
+            0.016253733908242134,
+            -0.013857834499211056,
+            -0.005142936208363415,
+            0.012274716212832219,
+            0.008540229125193836,
+            0.011742606168187876,
+            0.0007451338650642946,
+            -0.009255813682572937,
+            0.008292531177374192,
+            -0.015020523035432158,
+            -0.008781317965985481,
+            0.0026250469418136603,
+            0.002381147382977664,
+            0.005839723981046825,
+            0.0006240641828668715,
+            0.004502463508605272,
+            0.005798912705218846,
+            -0.00497936644919367,
+            0.01112666823857836,
+            0.008198424171006548,
+            -0.002452143482105151,
+            -0.01017569410837814,
+            -0.001127589984272638,
+            -0.015733298167661346,
+            -0.016758315411028353,
+            -0.0009286266498936771,
+            0.015175988113750882,
+            -0.02352862898718829,
+            0.018996352975391644,
+            -0.0014524363611708201,
+            -0.0033802546171857704,
+            -0.021392428456030312,
+            -0.0025923983911570183,
+            0.0034214188668518796,
+            0.01255768230394476,
+            0.01858853630745928,
+            -0.02729543519465384,
+            0.010575894388563116,
+            -0.007204725245442141,
+            0.030394929424014717,
+            -0.018328782140005044,
+            -0.004022941273610227,
+            0.026915618079621992,
+            -0.02200018306828758,
+            0.0008791185377513978,
+            -0.011454606955158956,
+            -0.008103032693853504,
+            0.019287674837688177,
+            0.023747720480736777,
+            0.02038287242167783,
+            0.01364489294869211,
+            0.0007758044919789085,
+            0.004768030222445397,
+            0.03572705648994403,
+            0.010568241270862248,
+            0.014828087092859878,
+            -0.007542489523110974,
+            9.457808640264811e-05,
+            -0.02554033312739386,
+            0.01635899434369978,
+            2.587383534566272e-05,
+            -0.012370727506722627,
+            0.007162422186621379,
+            -0.013370882403931942,
+            -0.011066483600541058,
+            -0.007151077450730025,
+            -0.01721910018182534,
+            0.02152239849400241,
+            -0.012322440280680164,
+            0.004587729883596212,
+            -0.009079747696669277,
+            -0.013272707286450417,
+            -0.012953682633400515,
+            -0.014202214807187545,
+            -0.01313141988034687,
+            -0.026041403779538307,
+            -0.027109731260915763,
+            -0.006573285295895077,
+            -0.023780366346334524,
+            -0.013799072648505526,
+            0.005588622594136253,
+            -0.022617902666316757,
+            0.012262325666428604,
+            -0.012689827812482627,
+            -0.01226264872059864,
+            0.012805734807835008,
+            0.020274844315616616,
+            -0.011765231803328348,
+            -0.028494489703400308,
+            -0.018583843821043854,
+            -0.019970998470704358,
+            0.012576201137505969,
+            0.002266288538944273,
+            0.013165889075444961,
+            -0.003018437854660793,
+            0.003227412667880239,
+            -0.014895831590298957,
+            0.018570624675927183,
+            -0.03442390703780963,
+            -0.03108003806963536,
+            -0.013611306765348361,
+            0.005182777066476028,
+            -0.0015782033203578553,
+            -0.011429046504256945,
+            -0.0018606648572920801,
+            -0.00433805271182878,
+            -0.025109572024446088,
+            -0.010134856632190067,
+            0.011010662054484161,
+            -0.015205629927469414,
+            0.0072933565255675036,
+            0.0009018832564038863,
+            0.013870902834260887,
+            0.009587432944616442,
+            0.013393301893492882,
+            -0.006095063698384775,
+            0.004356485794113208,
+            -0.0026476612747324386,
+            0.0004894024149759108,
+            0.009336366919474478,
+            0.00839591561027103,
+            0.010551161369079399,
+            0.002260707777260338,
+            0.007830059223001696,
+            -0.0060927446216633265,
+            0.018065002687169148,
+            0.015436944875538481,
+            -0.0002520689442496089,
+            0.013323098485251487,
+            0.0003769862591044255,
+            0.00022293304921127018,
+            -0.0029970303106340466,
+            0.007888437018607987,
+            -0.011785957933642163,
+            -0.001311704406355858,
+            0.015138190893483804,
+            -0.01707247730569978,
+            -0.013045187198628258,
+            0.03107034781283638,
+            0.0018583638488268697,
+            -0.02495564203304796,
+            0.031146554085787762,
+            0.004220047610942943,
+            -0.005835199594223787,
+            -0.013205086624848782,
+            0.0014059047822222418,
+            -0.011570624844100298,
+            0.003584232809362274,
+            0.014295032061446826,
+            -0.02231984223610487,
+            0.010521094801839404,
+            0.008446892441796151,
+            -0.009681535919980559,
+            -0.012337659795586223,
+            0.009847073709209602,
+            0.016809328445200358,
+            0.004472485504034416,
+            -0.03539093810210523,
+            -0.011957694102753411,
+            0.02145470732008716,
+            -0.029237428225009913,
+            -0.00810041956259673,
+            0.007452962400105787,
+            0.0065154832040426065,
+            -0.019353109017609053,
+            -0.008580955020412362,
+            0.005725794605897509,
+            -0.004940390374637084,
+            -0.003892210172017758,
+            0.017541623128534682,
+            0.00485047043626186,
+            0.013698341792101025,
+            -0.01689738578307474,
+            0.009771458057400557,
+            -0.022222817828602557,
+            0.004021882074676753,
+            -0.023373700500461992,
+            -0.009873403167702603,
+            0.002364143609079849,
+            0.012454271200139762,
+            0.014328556467632212,
+            0.027839363143659326,
+            0.030056404238348527,
+            0.013995696971422429,
+            0.007833677585250953,
+            -0.016265199386953717,
+            -0.018134327237616175,
+            -0.006409513840075933,
+            -0.004033173350204784,
+            -0.008714798059218596,
+            0.0066373547199279235,
+            0.01858354212009877,
+            0.004499712802472377,
+            0.02609078206670711,
+            -0.006227189596272876,
+            0.009039744622014585,
+            -3.127526911951914e-05,
+            0.0008833930131827993,
+            -0.004226083827636182,
+            -0.01971685546584527,
+            0.00481003211084742,
+            -0.0013721482162825508,
+            0.025874758880591167,
+            0.00940496860916206,
+            0.0024915013399541847,
+            0.01615978314811047,
+            -0.006378952757397484,
+            -0.01174653607178726,
+            -0.010540107733238588,
+            -0.0061926268398085176,
+            -0.021752582127981104,
+            0.029059100503124387,
+            0.021195595568253612,
+            -0.010800765264123782,
+            -0.013693582166598082,
+            -0.00903369526059487,
+            0.02795767990556847,
+            0.004080965688836953,
+            -0.0013050212744006933,
+            0.016313311850308505,
+            0.012084555958999215,
+            0.012329325712491252,
+            0.009794698207689061,
+            -0.00036157148861933123,
+            0.0015364709122308352,
+            0.03722522141440073,
+            -0.007427028510548186,
+            0.0026463292682538495,
+            -0.005239324411651515,
+            0.0025987932868461313,
+            -0.009705817520422487,
+            0.007242122655628283,
+            -0.004012676385296197,
+            -0.0158760051470016,
+            0.015003444174308607,
+            -0.022443331305169994,
+            -0.0016749787204471238,
+            -0.008150635190952799,
+            0.0016696640640947303,
+            -0.02781023337840043,
+            -0.0025731206607371845,
+            0.006695065701630532,
+            -0.015547340828489069,
+            0.011816010501093885,
+            -0.01740673229051557,
+            -0.011553604323149105,
+            0.007914961618894248,
+            -0.012034157288065447,
+            0.0016684130271324446,
+            -0.0008268957621923278,
+            0.004460864332388434,
+            0.013426055079012884,
+            -0.011008808570221245,
+            0.012166046631826273,
+            -0.005816284310138776,
+            -0.007872091214834688,
+            -0.0016055129487028885,
+            -0.0030049472973942043,
+            -0.0025600746761171543,
+            -0.0009966231170649917,
+            0.034355910234323155,
+            -0.004446570896048543,
+            0.003761242147499512,
+            -0.01362639743248326,
+            0.0019833052855181806,
+            -0.036176376394999044,
+            0.028344464817309477,
+            -0.008642835889481802,
+            0.011013502221008498,
+            -0.0017508365480865596,
+            -0.013243706268954915,
+            -0.01456517309747086,
+            0.015247299501245178,
+            0.007602596572394442,
+            -0.005854531057349665,
+            0.013558248087347848,
+            -0.019724373614448298,
+            0.008530500822910953,
+            0.004463694543554244,
+            -0.015561397976987216,
+            0.004668780190328835,
+            -0.007479912658259757,
+            0.012657201279639194,
+            -0.01774730033362932,
+            0.004314594593150765,
+            0.00958571617187387,
+            -0.01892876084898881,
+            0.009901041800918215,
+            -0.028675879564512306,
+            -0.004631920242208613,
+            -0.012323455524366223,
+            0.0053804608827285865,
+            -0.000660864620708182,
+            -0.02389088252276507,
+            -0.028944280660230762,
+            -0.0002099667627277945,
+            0.00458533068247518,
+            0.0018878351997997833,
+            0.011621407953503755,
+            0.005970807015151215,
+            0.008542958629199003,
+            0.01023896054756221,
+            -0.00205207247588841,
+            0.009402752124876478,
+            0.04169706217414899,
+            -0.022899912321317197,
+            -0.011574970753549647,
+            0.005321872977369877,
+            0.011499271505460519,
+            -0.02647653483798284,
+            -0.011039791758347888,
+            0.0033424345263339144,
+            -0.005219877979550397,
+            0.03609469691834737,
+            -0.02448746405881972,
+            0.019020618652020673,
+            -0.011392297116547224,
+            0.007625475937968714,
+            -0.021465240076078376,
+            -0.003921088264697311,
+            -0.014265745991522477,
+            -0.00011221615798127438,
+            -0.0031478236827989703,
+            -0.0076053733863387755,
+            0.007485039021296698,
+            -0.001907678661303243,
+            0.0020348246022161925,
+            0.006291436821831794,
+            -0.02173305455485444,
+            -0.0019125499969493854,
+            -0.016432460924913166,
+            0.030367263349759286,
+            -0.016242095671439496,
+            0.005072805486919414,
+            0.0053258139728101506,
+            -0.002810639419737258,
+            0.012368771814660942,
+            0.004165967191536186,
+            0.00513754440091394,
+            -0.004199767241827414,
+            -0.005142642360546972,
+            -0.0053886448467479215,
+            0.017868874549760376,
+            -0.013280048598849825,
+            0.035024701466710065,
+            -0.0100959186045929,
+            -0.007094001877261568,
+            0.0019733042325232155,
+            0.010380430710282744,
+            4.2084202709296443e-05,
+            0.01053565670076933,
+            0.0034270489770133685,
+            -0.0006311078385359693,
+            0.0019774779328212756,
+            -0.017162448473652843,
+            0.003089758682621882,
+            0.01682656561693936,
+            0.005463677441664502,
+            0.004469261499451007,
+            -0.013332130891003062,
+            -0.012050233723282738,
+            0.004538268704442613,
+            -0.007533940408255447,
+            0.0003675737140813486,
+            -0.018816953644773076,
+            0.013972696748575243,
+            -0.0011096683686631858,
+            0.0002715683052236706,
+            0.00524678369365015,
+            0.023056514997908218,
+            0.01813167860994747,
+            -0.002553352602551003,
+            0.03155164978556312,
+            0.0174738879054593,
+            0.0029268039949260673,
+            0.010637295711777793,
+            -0.033222781195984666,
+            -0.013301873512972214,
+            -0.0013721703891642354,
+            0.02238421933149517,
+            -0.01530189703335162,
+            -0.014059029663525449,
+            -0.0037393534403749947,
+            0.0010394601933309767,
+            -0.005177359933828505,
+            0.0042489306011644495,
+            0.011039889511148341,
+            -0.014697595240068863,
+            0.007597149702441915,
+            0.0026780496822907634,
+            0.02092734730183164,
+            0.0004688144432243786,
+            0.017181410417231698,
+            -0.0005083970646221274,
+            0.00749639055342427,
+            0.007319420826337891,
+            0.017541463125355265,
+            -0.01002811389296805,
+            0.023582113845081534,
+            -0.008823933397729605,
+            0.0019889258902730535,
+            -0.01257106158422629,
+            -0.028159594067544737,
+            -0.012624750220149823,
+            0.015247564545279736,
+            0.005765911899842925,
+            0.006478299226263732,
+            -0.016537090297144265,
+            0.006276997440048981,
+            0.015647122171767974,
+            -0.002251683151059454,
+            0.010295358869874306,
+            0.009730378299068289,
+            0.00012816405890877098,
+            0.010181120688239766,
+            -0.00801256937783534,
+            -0.00734257767723293,
+            -0.004497520124843868,
+            0.023982754195729983,
+            -0.005443550092842393,
+            0.022434177122972028,
+            0.012647958478747348,
+            0.00616791857573897,
+            -0.010853355357954727,
+            -0.003340111254357573,
+            0.023389634440057473,
+            0.011863246790550376,
+            0.00029642699874398923,
+            -0.009846034060495678,
+            0.0019458876579524617,
+            0.0054010373765486005,
+            0.012983913038716568,
+            -0.02049427270240938,
+            -0.004344571352918878,
+            0.008461947538684664,
+            -0.008472919299762712,
+            0.004318309005722874,
+            -0.030156444522730416,
+            -0.005812303237222271,
+            0.021354352236942434,
+            0.008392281015122372,
+            -0.010232039722494646,
+            -0.004395682831498155,
+            -0.0006527587802367882,
+            0.0040601850269714945,
+            -0.0024323167053212176,
+            -0.0023729138545912176,
+            -0.008541233856441836,
+            -0.006470933419694742,
+            0.014035696617797647,
+            -0.0052691760528871924,
+            0.008597459268477037,
+            0.00012084859681905111,
+            -0.0017255958688825231,
+            -0.007323529232751411,
+            0.03382792341957058,
+            0.008087732655845718,
+            0.005980666974986827,
+            0.0013613625176327891,
+            0.0042244330945601215,
+            0.000616791811672998,
+            -0.0028374856646453707,
+            0.018075360591135542,
+            0.0156738939725117,
+            0.005054696911814988,
+            0.009038525980341447,
+            0.014672270369917403,
+            -0.019700405219230957,
+            0.012625337070556061,
+            0.012184185186281482,
+            0.019382321278525114,
+            0.0007219498737851579,
+            0.002564970959687783,
+            -0.010601873823737926,
+            -0.0029963537053764736,
+            -0.002070567089566093,
+            0.029563028293936092,
+            0.0005903976560069006,
+            -0.010509758568838935,
+            0.010992421945232951,
+            0.020603868018231348,
+            -0.02101157589451638,
+            -0.001971974816402967,
+            0.005963911670920596,
+            -0.010704207628592966,
+            -0.014466113540150018,
+            -0.021269176952661614,
+            -0.013331247081394942,
+            0.0071597345125287174,
+            -0.008162102923896894,
+            0.02367549407145414,
+            -0.014256771777502395,
+            -0.03960796112042958,
+            -0.014803626428839152,
+            -0.00864320347698291,
+            -0.027531337207277483,
+            -0.007305441146142331,
+            0.00965056093889974,
+            0.016698589600806273,
+            -0.003330867073501157,
+            -0.025198703196464594,
+            0.030483790406960656,
+            0.008872821868559751,
+            0.023708422767352917,
+            -0.021464374508407488,
+            -0.0018617153194555344,
+            -0.00886376784514331,
+            0.014505918224188001,
+            -0.017588761818663975,
+            -0.011756112651591795,
+            0.020014632992762026,
+            0.014356419813769714,
+            0.007246557500044434,
+            -0.004355132555323843,
+            0.0067299398462013936,
+            0.007649499235136303,
+            -0.012515181210363267,
+            0.012239792251154642,
+            -0.013499277313916029,
+            -0.024858295126228433,
+            0.014430561986516884,
+            0.002229494935179828,
+            0.009563004428064055,
+            -0.0015984171085659323,
+            -0.016197659608768088,
+            0.010078159490026602,
+            0.00808729175515505,
+            -0.017508840440512225,
+            -0.004793934415922065,
+            0.007133527615305415,
+            -0.020818869260580315,
+            0.031234686259160864,
+            0.011872765152279094,
+            -0.010692949770503467,
+            -0.001989039629661815,
+            0.004431079028880923,
+            0.016511544608975148,
+            0.007893700385963683,
+            -0.01997314388229276,
+            -0.003708006454566117,
+            0.01530128306203401,
+            0.00822588305477249,
+            0.004861703074728471,
+            -0.010836328783594584,
+            0.03434874763691911,
+            0.006082873924472328,
+            4.192481378506604e-05,
+            -0.0052997095110967115,
+            -0.0076108924981838355,
+            0.004196112457289409,
+            0.018905391806961686,
+            0.013330078288599763,
+            -0.004200590783114919,
+            -0.006630107527854746,
+            0.007778448128358027,
+            -0.01706991160178345,
+            0.007329056462125937,
+            0.009390327499494826,
+            -0.012870674959022891,
+            -0.0017571189976160295,
+            0.016462608870101713,
+            0.00209846661139665,
+            0.004942714850185376,
+            -0.011305050739130339,
+            0.004252776146827917,
+            0.015841821944109317,
+            0.018727567181082646,
+            -0.008399642221255733,
+            0.021448055799259867,
+            -0.020972459569228434,
+            0.027243639465535856,
+            0.028499400850980086,
+            0.016573144403275928,
+            -0.01847096443459798,
+            -0.0022409373140116146,
+            -0.0041737366139895414,
+            -0.014761716560267328,
+            0.0034546020161337894,
+            0.005125870074458691,
+            -0.01174186608185822,
+            -0.009577921232380895,
+            -0.024331993794416026,
+            0.011075959943215117,
+            0.007918147902741738,
+            -0.0015045185909419564,
+            -0.011791155919533844,
+            0.016019077577552835,
+            0.010141868950981925,
+            -0.010938402325619172,
+            -0.005635351352447256,
+            -0.019967361901734836,
+            0.009750427567651833,
+            0.009128294017175602,
+            -0.00300556760891084,
+            -0.011496103417643132,
+            0.012977890524465716,
+            0.0014080059843484115,
+            -0.01292380658817521,
+            0.010841954636959441,
+            -0.03142011651894961,
+            -0.005510537354374977,
+            -0.011027026734432255,
+            -0.003620482919145167,
+            -0.014415606143566519,
+            0.003608572766363589,
+            0.0005034611321129689,
+            -0.011470488351158329,
+            -0.0035005774696278,
+            -0.016207453351921627,
+            0.004676327282262429,
+            0.015021513761697039,
+            0.001679614645967629,
+            -0.006917375094634678,
+            0.021994274901326064,
+            0.018518524483306403,
+            -0.021903538417919984,
+            -0.020116013379669034,
+            0.004708084983787958,
+            -0.03261690161800251,
+            0.015019348187107588,
+            0.00966124471577622,
+            0.011676937880751252,
+            -0.028768229226408037,
+            0.007114534979415504,
+            0.00900420165933673,
+            0.016398927481511824,
+            0.0050879301160439245,
+            0.009453255486450082,
+            0.01222758288108498,
+            -0.0104657614414275,
+            -0.003164502778525433,
+            -5.410382753651127e-06,
+            -0.005618225986727351,
+            0.015607620507415484,
+            0.0008527697935056332,
+            0.03603666488542239,
+            -0.005277207843139309,
+            -0.0009799879252482288,
+            -0.011735731235560851,
+            0.00812046801749312,
+            -0.011713349784345638,
+            0.006334172347602515,
+            -0.0024643499727789167,
+            0.0001240009282616789,
+            0.002965067818077541,
+            0.009442572075213767,
+            -0.01086095783353607,
+            -0.007595379345224076,
+            0.007444897256325813,
+            -0.01482814340478338,
+            -0.00019984717819268013,
+            -0.009339343730980451,
+            0.0019680804106275156,
+            -0.02241031895876346,
+            0.006629838013824249,
+            0.040935644507527846,
+            0.02140216465404127,
+            -0.005481040712269146,
+            -0.012431659209559418,
+            -0.004680230194268222,
+            -0.007229300034633229,
+            -0.006482401275991039,
+            -0.004794698160419071,
+            -0.022762405726589375,
+            -0.006928553505112732,
+            0.004320321244630314,
+            0.013300138724212552,
+            0.009741018117644266,
+            -0.008842135469729511,
+            0.004531760574809748,
+            -0.02662650543210291,
+            -0.016578081411548973,
+            -0.0060021425889846495,
+            -0.015840130031005147,
+            0.006125301421471048,
+            0.027521841635711787,
+            -0.0063432924325983525,
+            0.01789356321527744,
+            0.005142029893259461,
+            0.013034833065163448,
+            -0.005338869351330258,
+            0.00035250243782976347,
+            -0.006010938630154189,
+            0.0032958656042048292,
+            0.003074861351445954,
+            -0.0015947217852680714,
+            0.004151590621941346,
+            -0.007139285822575166,
+            -0.0026091937868967284,
+            0.0011629380311555433,
+            0.0117841884590861,
+            0.006877419488576261,
+            -0.00016252653621991807,
+            -0.01761285866765538,
+            0.022221104524808997,
+            -0.018933508559881868,
+            -0.008840161102111973,
+            0.011092684029552064,
+            -0.009660487856185668,
+            0.00702597023411946,
+            0.020387386977378873,
+            -0.006854213562767843,
+            -0.009583981919303215,
+            0.013028181087649146,
+            -0.0036507672754997286,
+            0.004464368543493108,
+            0.009867428978560914,
+            0.006036303973414791,
+            0.002482944923100156,
+            0.013244650094892797,
+            0.004530294330688855,
+            -0.0032625482538717137,
+            -0.020238533752799943,
+            -0.005186498720507101,
+            0.019015752262933585,
+            -0.02252458872825473,
+            -0.0027641215988942563,
+            0.0044506144202026456,
+            -0.0030317840657635457,
+            -0.01733962766050954,
+            0.004205861489436576,
+            -0.018770275720867473,
+            -0.0033883519277721433,
+            -0.008244668812709337,
+            0.009948090119054079,
+            -0.023700732627507225,
+            0.009109315916173436,
+            -0.011739192106106307,
+            0.02356509175177699,
+            -0.003925851077921686,
+            0.024734433836185024,
+            0.0015142645677087268,
+            -0.013482933030628261,
+            0.0055323884713736554,
+            -0.0008930661843169134,
+            -0.004477731568313757,
+            0.029338490404300612,
+            -0.005418258701133154,
+            0.002249996261871466,
+            0.013491124734151694,
+            -0.021332541202417246,
+            0.021773687217263173,
+            0.0046662696076723836,
+            0.01628697326539432,
+            -0.00986238951830334,
+            -0.013799209313714441,
+            0.025178874553620743,
+            0.03324555035648896,
+            0.008398796308070151,
+            0.011261509865513953,
+            -0.01226890254177634,
+            0.018966175575083295,
+            -0.0038328997984309597,
+            0.004344837325292706,
+            0.00878668880458693,
+            -0.007200842513742962,
+            -0.011500562922303248,
+            -0.004533014670236755,
+            0.027639894892383943,
+            -0.021838848944317424,
+            -0.008920629621611768,
+            -0.005739181836252042,
+            -0.01593272996094732,
+            -0.038167534267554606,
+            -0.004371701247332816,
+            0.019604405563120094,
+            -0.005974358141737469,
+            -0.0018341151594086288,
+            0.014334266690716505,
+            0.00402620072221496,
+            -0.002852569009185304,
+            -0.008195218707838913,
+            -0.0027944153624309856,
+            -0.0059390492900516225,
+            0.016651206007022145,
+            -0.004961864528562493,
+            -0.007929830585906699,
+            0.01145431100211762,
+            -0.005731957927601062,
+            0.012130694936714224,
+            0.0004848418056781251,
+            8.392048424584131e-05,
+            -0.0015174120840537315,
+            0.00822932723013239,
+            0.018620869354789082,
+            -0.0055803454666417465,
+            0.025682778991091174,
+            0.006956936033581638,
+            -0.003495709501259274,
+            -0.005489431577052735,
+            -0.0044154293119170445,
+            -0.024561016942073285,
+            -0.0020864636079347357,
+            0.006593060072668424,
+            0.024333813203596122,
+            -0.02515989187381735,
+            -0.004953266000463339,
+            0.020957806164031402,
+            -0.0012192724005613,
+            -0.011978722479596745,
+            0.014690559986694138,
+            0.0041346025614047875,
+            0.009881584250499358,
+            0.012176393203318002,
+            0.015764530703333238,
+            -0.004923548157246672,
+            -0.0015192775000207833,
+            0.012766482928380375,
+            0.0005745973040874939,
+            0.00997809510696241,
+            0.016495444272181515,
+            0.010467514633877887,
+            -0.021108281220544853,
+            0.0015078843789125156,
+            0.00522593579521476,
+            -0.017481430635971396,
+            0.0034362821987740755,
+            0.005397819166037461,
+            0.011626527721895694,
+            0.004048632382026344,
+            0.013207335578216437,
+            -0.015768003434275563,
+            -0.005916168437120099,
+            0.008153798523776654,
+            0.004319451296835019,
+            -0.02599474886669325,
+            -0.01485009085106089,
+            -0.0006781964116146299,
+            -0.004444144921332929,
+            0.013556265879295017,
+            0.01285768162554067,
+            0.02033626001285354,
+            0.024637349990579514,
+            -0.013833060730799217,
+            -0.00842819657792762,
+            0.009436072455719896,
+            0.015560357465711765,
+            0.005798181411384387,
+            0.007874051721372273,
+            0.009297440071118126,
+            0.0077320823375042,
+            -0.004073209576661068,
+            -0.002973928843567532,
+            0.020205003097225747,
+            -0.019725382393637726,
+            0.0006746659838642147,
+            -0.018417443693472958,
+            0.007530796896254817,
+            0.01140696723506382,
+            0.01129515042745398,
+            0.002816883643872417,
+            -0.005274757308627299,
+            0.005716917047812153,
+            -0.06254973656832136,
+            -0.009191453047502128,
+            0.003384751551032943,
+            0.027256466708746898,
+            -0.010885336140129142,
+            -0.003415707910562931,
+            0.007309577090125959,
+            0.005511970708769456,
+            0.0005572239548625122,
+            0.003186427278034852,
+            0.019698716902282903,
+            -0.02052581021967601,
+            -0.010793968505805883,
+            0.004219269922138548,
+            0.0033111837923746746,
+            -0.007269521534951044,
+            0.004805737092302134,
+            0.014143978208501908,
+            0.01986656063456352,
+            -0.0024048854610842748,
+            0.01629335842915328,
+            -0.0015498603704589015,
+            -0.014496994444493684,
+            -0.002671064325909516,
+            0.0037875833047555925,
+            -0.022466433279066646,
+            -0.005980897597195858,
+            0.0043262697442044435,
+            -0.011226121648007667,
+            0.006978495397818936,
+            0.008452177557316863,
+            -0.003018545465557218,
+            -0.0008303307644324117,
+            0.017796789945838845,
+            0.006591364879905198,
+            -0.013464486184025026,
+            0.0028668426952326957,
+            -0.023377605222213174,
+            0.004325185794796361,
+            0.001416245546006914,
+            0.007809449561523174,
+            -0.008431335139468173,
+            -0.014774499666285603,
+            -0.005933810712528226,
+            0.017746847020610673,
+            -0.01965188730040385,
+            -0.002406875192894708,
+            0.01683009662984443,
+            -0.0002518220110313279,
+            0.005456812494894182,
+            -0.02011241622618092,
+            -0.0009782974741561381,
+            -0.02933296756078644,
+            -0.01171823509606186,
+            0.001162345317872531,
+            0.00892221099991475,
+            -0.006551308022163148,
+            -0.03934356275100932,
+            0.00427623835776202,
+            -0.013111664387776261,
+            -0.008752857644809231,
+            -0.010214546496134218,
+            -0.00518087722852466,
+            0.015240297918667867,
+            -0.012498100033235827,
+            0.01086490299272505,
+            -0.00917777212602538,
+            0.005895055740265625,
+            0.0090554651297836,
+            0.014212841798768113,
+            -0.005544047915369086,
+            -0.0033589999440643385,
+            -0.007924830667307956,
+            0.0022291797354234005,
+            -0.0032976226056924502,
+            0.009615674261248298,
+            -0.0051932155872883,
+            -0.0065636389655770455,
+            0.02426106498596339,
+            0.011445591067574932,
+            0.013312596099532533,
+            0.0055118624272110965,
+            -0.007220620072349783,
+            -0.01916551187613823,
+            -0.003977616240384473,
+            -0.013204506708246246,
+            -0.006315966975649096,
+            0.01434099854765068,
+            0.012588249778961223,
+            0.006357972880218866,
+            0.0007747002406996095,
+            -0.004794439624135275,
+            0.006975593520375694,
+            0.0262335496080837,
+            -0.016481909971520166,
+            -0.0013084662210086592,
+            -0.01369313900978409,
+            0.01921748474212319,
+            0.03333761672336437,
+            -0.0008918419419301244,
+            -0.007699525186523558,
+            0.013757216065348164,
+            0.00819045303047094,
+            -0.007411380297164911,
+            0.02453611080891859,
+            -0.025045462880463647,
+            -0.015989319900023054,
+            -0.011650886878708265,
+            -0.011407253985176972,
+            -0.008386367959227559,
+            0.016331961716602287,
+            -0.00141225438918211,
+            -0.0018505754298268654,
+            0.009557291617446645,
+            0.03357924917013024,
+            0.006916860901290756,
+            0.007030308953738567,
+            -0.002739345982346728,
+            -0.009376522438071562,
+            0.011482637968411055,
+            -0.007259054391879614,
+            -0.04075661435928125,
+            0.004250195164583401,
+            -0.005160505473755424,
+            0.02405623169060588,
+            -0.02197586561375289,
+            -0.00619175761282745,
+            -0.006736129049831973,
+            -0.028200015908139763,
+            -0.005867722104196303,
+            0.02805071895018145,
+            -0.015402198567775478,
+            0.014773294766214671,
+            -0.009502185346558072,
+            -0.0013419858580438616,
+            0.004443749010136947,
+            0.013008783570355656,
+            -0.02289662952366337,
+            0.0013834888740540108,
+            -0.00048549197205067054,
+            0.027969842651517314,
+            0.010437729090975158,
+            -0.0047705656249623744,
+            0.014202218020412936,
+            0.008240861154198411,
+            -0.018637315941614406,
+            0.007314019317343154,
+            -0.005747966765101792,
+            0.014833896739762147,
+            -0.01314847940891154,
+            0.007762100494499689,
+            0.00705026393499352,
+            -0.01278023669088385,
+            0.03016342404000438,
+            -0.015204129985039077,
+            0.007776946006562982,
+            0.009278778571285689,
+            0.009656723862179986,
+            0.01257686993982689,
+            0.004362439504889069,
+            -0.02898418193036562,
+            0.0002144431781454467,
+            -0.03828100004144633,
+            -0.00833748467467171,
+            -0.002611937107022567,
+            -0.002145329666219461,
+            0.007050630951707937,
+            -0.009383645162261629,
+            -0.033976442057557676,
+            0.0006429978367640709,
+            0.01603027598131823,
+            0.017501067413108286,
+            0.0002196997337451293,
+            0.013075379412332973,
+            -0.006197537539554383,
+            -0.021148120787551232,
+            -0.016986448660412497,
+            -0.010207705210051413,
+            -0.02122194744719993,
+            -0.020550470201191853,
+            -0.00908564461857315,
+            0.012090079127547193,
+            -0.009927390005879807,
+            0.013888458967497627,
+            0.029513250508042935,
+            -0.024529175134769883,
+            0.0011144447737101862,
+            0.021977677894839922,
+            -0.00198308986575084,
+            0.014108189311291326,
+            0.001696775006906642,
+            0.014122064251288995,
+            -0.0004166140484018266,
+            -0.010556016685519523,
+            -0.009729362492661684,
+            0.017487929296614076,
+            -0.0026043257634797124,
+            0.016715798948897194,
+            0.012183487338360088,
+            0.003771721868999568,
+            -0.0030887189945795796,
+            0.010667776697461199,
+            -0.026471157050402344,
+            0.014249996582064706,
+            -0.006690961709923224,
+            -0.0181382691440242,
+            -0.008729104191096461,
+            0.003882731567105777,
+            -0.005609384377642956,
+            -0.007794812949125115,
+            0.012967089730551055,
+            -0.00944374111323783,
+            0.02088525981049842,
+            0.0023400121055514958,
+            -0.012196925240774426,
+            0.003648688547842322,
+            -0.03322041209304968,
+            0.0015191765655963997,
+            -0.018402382776013956,
+            -0.024659631472836927,
+            0.03840173429666435,
+            0.011991452465253198,
+            0.006639702644606365,
+            -0.01675838115087676,
+            0.007953102302963526,
+            0.0004257479565099406,
+            0.017484338819046517,
+            0.011068864013925452,
+            0.006466915839538296,
+            0.008364454017976605,
+            -0.023316137291380285,
+            -0.010898421318668592,
+            0.016204569093730564,
+            0.012099473359791148,
+            -0.01017229369980751,
+            0.019431669546223253,
+            0.017955119269367787,
+            0.007300372071724306,
+            -0.006898619924677706,
+            0.01647927363640854,
+            0.021012103522831126,
+            0.010291016618557653,
+            -0.00047955552103474145,
+            0.0008684297441778696,
+            -0.010534247928997246,
+            0.005037930235889528,
+            0.035417626740307205,
+            0.014712595692093886,
+            0.010650306363963653,
+            -0.03626584475133921,
+            0.021764456453493054,
+            -0.011961521673661574,
+            0.009092981192874177,
+            0.006089104887183293,
+            0.017982801101846038,
+            0.006916969397519662,
+            0.014196254958707752,
+            0.011745069030657836,
+            -0.006689305450954901,
+            0.025205280281804708,
+            -0.001601414806100461,
+            -0.011992525142592405,
+            0.003386930324498371,
+            -0.01859193251438437,
+            0.005303261319773131,
+            -0.0041710536965100615,
+            -0.004459393120257307,
+            0.0032461828144552565,
+            -0.0018650956576455287,
+            0.015678225251799887,
+            0.003894142628084471,
+            -0.02355212701026675,
+            -0.0014873755115206368,
+            -0.024284295126348117,
+            0.0027260592623277375,
+            -0.022185571998195344,
+            0.00022883605560765203,
+            -0.005557568353382136,
+            -0.009749382813201403,
+            -0.025385857431333838,
+            -0.009853073898348951,
+            -0.0035843255959309307,
+            0.007661137888845632,
+            -0.010521037433807653,
+            -0.004246698084422667,
+            -0.0019920499321964815,
+            -0.019465113363548715,
+            0.009014255846983305,
+            0.027122533756959888,
+            0.016848221859975445,
+            0.027170883580103264,
+            -0.00019554112591107156,
+            0.013614675849485926,
+            -0.002066822813103284,
+            -0.01603063005944364,
+            0.008824663429777616,
+            0.0075653522381218495,
+            -0.00531780740880296,
+            0.014643348459785112,
+            0.0072634484937344704,
+            -0.012800149015319017,
+            0.009859503660384916,
+            -0.007161712176469609,
+            -0.006437857349742842,
+            -0.009764856421740622,
+            -0.02058868349990357,
+            -0.014866137971603676,
+            0.0012450207572602557,
+            -0.002123265406747006,
+            0.006000473862248991,
+            -0.004054016790850226,
+            0.009584879012937315,
+            -0.00608816807208866,
+            -0.00925801057439376,
+            -0.013962683072792106,
+            -0.008924356118158748,
+            -0.004210064145949253,
+            -0.004381709838058668,
+            -0.011732699976353612,
+            -0.005479523097469261,
+            0.0010680258077037758,
+            -0.0019401640391035755,
+            -0.01397652201595301,
+            0.011650839243603377,
+            0.0058731838291845645,
+            -0.002462743134993977,
+            0.02219008688925881,
+            0.01097316136828334,
+            0.0018083001685932406,
+            0.01366095798812196,
+            0.0003942601242216818,
+            -0.0064063532958756175,
+            -0.03343312148366622,
+            -0.0005083355836126291,
+            -0.02815586367527057,
+            0.025770078162165123,
+            -0.0069114949102666145,
+            -0.010322087105114614,
+            -0.002519047562374146,
+            -0.006063669644021698,
+            0.005873609611264377,
+            -0.002722940652283699,
+            0.0021229525800901114,
+            -0.023673778230768945,
+            0.0028629715379660113,
+            -0.003953653437464966,
+            -0.0036366957271968823,
+            0.007469664864329332,
+            0.0346520690389124,
+            -0.004054282838167587,
+            -0.0034765253405897676,
+            0.005159466707114532,
+            0.011642913651743502,
+            0.027845571220425726,
+            -0.004445196354520585,
+            -0.0032038508043339413,
+            -0.02233838584152591,
+            0.0054112576531374175,
+            -0.00019544838850471866,
+            -0.004733192299951088,
+            0.016664217348411003,
+            -0.016719175418727594,
+            -0.01631658832047864,
+            -0.00545969677717507,
+            -0.014570848053102338,
+            -0.00022620258237138677,
+            0.014454101124666536,
+            0.011056312437637014,
+            -0.004035526363810078,
+            -0.012459378363360557,
+            -0.0022595493558020706,
+            -0.030370952502579162,
+            0.012766010284884204,
+            0.1141689619126009,
+            -0.00766197047210299,
+            -0.00896655658602649,
+            -0.009481112393590025,
+            0.001435269335646347,
+            0.001338755472877713,
+            -0.011739143028767118,
+            0.07166096084053437,
+            -0.0077855767070652395,
+            0.0002139580430342728,
+            -0.011312198831181402,
+            0.001844005266872301,
+            -0.001825412693163496,
+            0.01371932421024605,
+            -0.005025814503687022,
+            -0.018213360991353643,
+            -0.0033612481616769416,
+            -0.02941403830641238,
+            -0.0041137865872933515,
+            -0.005683630256743215,
+            -0.004288198570156578,
+            -0.014982999363445122,
+            0.0067149665791261335,
+            -0.019661973497372313,
+            -0.012688556129040908,
+            0.001124391650175233,
+            -0.0033623453063430534,
+            0.00996944383866679,
+            -0.020496336814603385,
+            -0.0044354321069609536,
+            0.0019875840782527733,
+            0.009706675240563883,
+            0.0011600364428219214,
+            -0.012755178230821602,
+            -0.0026514442584739412,
+            0.004056350166031111,
+            0.01909951327913368,
+            0.014360281326789544,
+            0.003732344102045803,
+            0.008623457695718206,
+            0.014461868085333023,
+            -0.001692149528228451,
+            0.01948035376572433,
+            0.01172550239477637,
+            0.0031966531144076173,
+            0.0004779413449255626,
+            0.0030912562273611517,
+            -0.004597633235071031,
+            0.01582547400214653,
+            -0.014811177637844808,
+            -0.02411202772171917,
+            -0.0071272238837422185,
+            0.018015541137524113,
+            -0.013081287852028074,
+            1.8837689271256874e-05,
+            0.004062803284646432,
+            0.027569726868200254,
+            0.0010564850664255057,
+            -0.01179755720247972,
+            0.0028876134800690334,
+            -0.01638215925131053,
+            0.022944183106912663,
+            0.0026043713064243836,
+            0.03279173899194346,
+            -0.005103903273968361,
+            0.00578213669463383,
+            -0.020744194169889544,
+            -0.007588793365216579,
+            -0.00819862611489168,
+            0.0017401442433921013,
+            0.010993500174794084,
+            0.005411238350013134,
+            0.01620925462604661,
+            -0.00641615069699833,
+            -0.020751730783437995,
+            0.014776878596707538,
+            -0.009552193304486048,
+            -0.03393319554458277,
+            0.008881987295131347,
+            -0.029102297123590998,
+            -0.0045267103602711,
+            0.00368971742585232,
+            -0.014269341759285646,
+            -0.016264809810970175,
+            0.012837302095591106,
+            -0.0011890334180728564,
+            -0.008937992768323412,
+            0.0014444613725248846,
+            -0.017110187133760626,
+            -0.02385668037818897,
+            0.006017405364919834,
+            0.00676670672597817,
+            -0.0077994500447595,
+            0.018312452858615755,
+            -0.027015660037024585,
+            0.006123912473734307,
+            -0.012973676426738582,
+            0.009041974762537394,
+            -0.01580183887345168,
+            0.01163516332546347,
+            0.010887662470499361,
+            -0.001503467664709027,
+            -0.0076234124970223326,
+            0.01231202541338883,
+            0.004852720134894811,
+            0.033760365210093034,
+            0.028658326381800395,
+            0.000995999783627898,
+            -0.0025605367058226737,
+            -0.0020356794893366384,
+            0.011667525096213252,
+            -0.014049700287539817,
+            0.011218246246280464,
+            0.02658113044439717,
+            0.017568631325838108,
+            -0.019133062654008885,
+            0.018050678772708717,
+            -0.010356118256625172,
+            0.012872052420999813,
+            -0.025620154415048252,
+            -0.005135768485044442,
+            0.004936490920616527,
+            0.005163008729256001,
+            0.007425278892536464,
+            0.001410212306439329,
+            0.021891387289347516,
+            0.014999616600766989,
+            0.00027613811156650413,
+            0.00558422822386443,
+            0.01477009526981092,
+            0.010497782599293994,
+            -0.0005567894749233204,
+            -0.00467776506604011,
+            -0.016998035358377343,
+            0.00957271797745408,
+            -0.006065116857898469,
+            0.005138070401846389,
+            0.01698914855850691,
+            -0.01230257809343985,
+            0.011109844802663414,
+            0.007518553660463051,
+            -0.015504773927595823,
+            -0.017169933945839305,
+            0.005552829003043926,
+            -0.02252633790984703,
+            -0.01306722099657118,
+            0.02715689072730398,
+            -0.010513941399990838,
+            -0.02790582732974509,
+            -0.0009847768069205477,
+            0.009797865241108237,
+            -0.002772952073769185,
+            -0.0027968741526000576,
+            0.0037541733651479613,
+            0.010801137722284016,
+            -0.014658156970099418,
+            0.002075415233345957,
+            0.004735264309137368,
+            0.01930804315461151,
+            -0.0014151522239496115,
+            -0.015046118414100617,
+            -0.002288495894227231,
+            0.021592829677937263,
+            0.017174609713578173,
+            0.00265560378681856,
+            -0.016499825201963983,
+            -3.965296928410772e-05,
+            -0.0074142747014567635,
+            0.00021360696629503726,
+            0.00027431174904249525,
+            -0.0056391089282546054,
+            -0.018931681700770225,
+            -0.003888248121485032,
+            0.011181675419027736,
+            -0.00247252920175839,
+            0.01977860312223846,
+            -0.0011527575128223392,
+            -0.015763587433200915,
+            0.001960734212846797,
+            -0.005280860341480037,
+            -0.010372829570478649,
+            0.01351900743232294,
+            -0.009359942374603997,
+            0.006788746813046239,
+            -0.003212015225284119,
+            0.004611143085183367,
+            -0.0003308099698088422,
+            -0.017017265064185255,
+            0.02924142709831441,
+            -0.0018238549856433466,
+            -0.004769274041546295,
+            -0.0014058363976586562,
+            -0.033393331166741945,
+            0.015017165991303838,
+            -0.014574579250749552,
+            0.006745699409061803,
+            0.002705853802154516,
+            0.01272741296341735,
+            0.003253095246601883,
+            -0.008898616461779179,
+            0.0027485606928249326,
+            0.017040360927723885,
+            0.013756402798365262,
+            -0.007292864293090161,
+            0.008694855886257706,
+            -0.0028934389922887676,
+            -0.0037483208383542015,
+            0.01363963339139356,
+            0.004170611039133969,
+            -0.006282657671540625,
+            -0.0009784372731140945
+          ],
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "Mixture-of-Experts (MoE)",
+            "fine-grained expert segmentation",
+            "shared expert isolation"
+          ],
+          "title": "Deepseekmoe: Towards Ultimate Expert Specialization In Mixture-Of-Experts Language Models"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "3080f374-964f-4b6f-bb07-d9220ba99e16",
+        "properties": {
+          "page_content": "7. Deepseekmoe 145B Ongoing\n\nEncouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available.\n\n7.1. Experimental Setup\n\nTraining Data and Tokenization. For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study.\n\nModel Settings. For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps.\n\nSubsequently, the learning rate keeps constant during the remaining training process. The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. Also, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse.\n\nEvaluation Benchmarks. We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3).\n\n7.2. Evaluations\n\nBaselines. Apart from DeepSeekMoE 145B, we consider three additional models for comparison. DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters.\n\nResults. From the evaluation results presented in Table 6, we have the following observations: (1) Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. (2) On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. (3) At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5.\n\n",
+          "keyphrases": [
+            "DeepSeekMoE 145B",
+            "GShard architecture",
+            "Transformer layers",
+            "multi-head attention mechanism",
+            "expert parallelism"
+          ],
+          "title": "Deepseekmoe 145B Ongoing",
+          "embedding": [
+            0.011145060881972313,
+            0.0028371023945510387,
+            -0.01864549145102501,
+            -0.008063549175858498,
+            0.004066969733685255,
+            0.007203228771686554,
+            0.0034158634953200817,
+            0.061411239206790924,
+            -0.013921549543738365,
+            0.03988758474588394,
+            0.002668948844075203,
+            -0.009103755466639996,
+            0.020194068551063538,
+            -0.01502432394772768,
+            -0.007441772148013115,
+            0.021367233246564865,
+            0.03610217571258545,
+            0.012208729982376099,
+            0.007637299597263336,
+            -0.028171585872769356,
+            -0.0256375502794981,
+            -0.05227620154619217,
+            -0.012240014038980007,
+            -0.0378853864967823,
+            0.011450083926320076,
+            0.01580643467605114,
+            0.029986079782247543,
+            0.0068551902659237385,
+            -0.014132719486951828,
+            0.018520353361964226,
+            -0.006624467670917511,
+            0.03178493306040764,
+            -0.013851160183548927,
+            -0.0013999760849401355,
+            0.0002678725286386907,
+            -0.01470365934073925,
+            0.05349629372358322,
+            0.010644510388374329,
+            -0.018989618867635727,
+            0.0074926093220710754,
+            0.04495565593242645,
+            0.005001590587198734,
+            -0.02932910807430744,
+            0.016236593946814537,
+            0.029907869175076485,
+            -0.009471346624195576,
+            -0.025668835267424583,
+            6.886230403324589e-05,
+            -0.0020589034538716078,
+            -0.024151543155312538,
+            -0.01281095389276743,
+            -0.013452284038066864,
+            0.004950753413140774,
+            -0.008759627118706703,
+            -0.03494465351104736,
+            -0.008282540366053581,
+            0.038104377686977386,
+            0.06757426261901855,
+            0.05612418055534363,
+            -0.026216311380267143,
+            0.031925711780786514,
+            -0.015743864700198174,
+            -0.030079932883381844,
+            0.011356230825185776,
+            0.0009101799805648625,
+            -0.01856728084385395,
+            -0.017018703743815422,
+            0.0004423806967679411,
+            -0.00746132479980588,
+            0.02050691284239292,
+            -0.046363454312086105,
+            0.026998421177268028,
+            -0.016596363857388496,
+            0.01866113394498825,
+            0.030392777174711227,
+            -0.00615520216524601,
+            0.03419382870197296,
+            0.007711600046604872,
+            -0.0013344744220376015,
+            -0.026623008772730827,
+            0.02580961398780346,
+            0.04971088096499443,
+            -0.015008682385087013,
+            0.010143960826098919,
+            0.06569720059633255,
+            -0.01866113394498825,
+            -0.027326907962560654,
+            -0.06444582343101501,
+            0.011090313084423542,
+            -0.012138339690864086,
+            -0.02261860854923725,
+            -0.0006212882581166923,
+            0.012756206095218658,
+            -0.022383974865078926,
+            -0.00839985627681017,
+            0.0512751005589962,
+            -0.031206170096993446,
+            -0.009651231579482555,
+            -0.017691317945718765,
+            0.019365031272172928,
+            0.008118296973407269,
+            -0.015853360295295715,
+            0.015962855890393257,
+            -0.0033376524224877357,
+            0.01507125049829483,
+            -0.005983137991279364,
+            -0.030721262097358704,
+            0.032942455261945724,
+            -0.006495419889688492,
+            0.08321645110845566,
+            -0.020741544663906097,
+            -0.001445924979634583,
+            -0.0033435183577239513,
+            0.01900526136159897,
+            -0.008524994365870953,
+            -0.007121107541024685,
+            -0.03472566232085228,
+            -0.01020652987062931,
+            0.021836498752236366,
+            -0.021429801359772682,
+            0.05143152177333832,
+            -0.00858756247907877,
+            0.016690216958522797,
+            -0.036352451890707016,
+            -0.054278403520584106,
+            0.050023727118968964,
+            0.008141760714352131,
+            0.002275938866659999,
+            0.000169253398780711,
+            -0.0024245395325124264,
+            -0.016393015161156654,
+            0.0008818284841254354,
+            -0.03838593512773514,
+            -0.00615520216524601,
+            0.003464745357632637,
+            0.013491389341652393,
+            0.06172408536076546,
+            0.062068212777376175,
+            -0.01295173354446888,
+            0.01028474047780037,
+            -0.019724803045392036,
+            -0.020991820842027664,
+            0.019803013652563095,
+            -0.002023708540946245,
+            -0.01618966832756996,
+            0.011457905173301697,
+            0.0026865461841225624,
+            0.0422339141368866,
+            -0.011700358241796494,
+            0.003171454183757305,
+            -0.00040083113708533347,
+            0.02399512007832527,
+            0.01683099754154682,
+            0.04063841328024864,
+            -0.00606525968760252,
+            0.00877526868134737,
+            -0.019224252551794052,
+            -0.006995969917625189,
+            -0.052057210355997086,
+            0.015915928408503532,
+            -0.003816694486886263,
+            -0.008337287232279778,
+            -0.01634608954191208,
+            0.002739338669925928,
+            -0.004301602486521006,
+            0.035977039486169815,
+            0.02426103875041008,
+            0.012505931779742241,
+            -0.0752076506614685,
+            0.023228652775287628,
+            0.020553838461637497,
+            0.017347188666462898,
+            -0.048177946358919144,
+            0.03757254034280777,
+            0.0016443852800875902,
+            0.015681296586990356,
+            0.008329466916620731,
+            0.0012396436650305986,
+            -0.044236116111278534,
+            0.04088868573307991,
+            -0.009948433376848698,
+            0.006557988468557596,
+            0.06360115110874176,
+            0.009455704130232334,
+            0.021648792549967766,
+            0.0022270570043474436,
+            0.052057210355997086,
+            0.012826596386730671,
+            -0.0038694869726896286,
+            0.033755846321582794,
+            -0.0005205916822887957,
+            0.005908837541937828,
+            0.028234153985977173,
+            0.0011770748533308506,
+            0.009181966073811054,
+            -0.02388562448322773,
+            0.007046807091683149,
+            0.014625448733568192,
+            -0.00258291675709188,
+            -0.018535995855927467,
+            -0.01011267676949501,
+            -0.031378235667943954,
+            -0.01828572154045105,
+            0.009486989118158817,
+            0.0248867254704237,
+            0.00891604833304882,
+            0.003511671908199787,
+            0.029157044366002083,
+            0.024511313065886497,
+            -0.017613107338547707,
+            0.009033365175127983,
+            -0.0015779059613123536,
+            -0.02821851149201393,
+            -0.048553358763456345,
+            0.00512672821059823,
+            0.022290121763944626,
+            -0.010988638736307621,
+            0.015423200093209743,
+            0.019505811855196953,
+            -0.015321525745093822,
+            -0.020194068551063538,
+            0.016142740845680237,
+            0.010542836971580982,
+            -0.02518392726778984,
+            -0.033411718904972076,
+            -0.014359530992805958,
+            0.014758407138288021,
+            -0.003525358624756336,
+            0.05446610972285271,
+            -0.025574982166290283,
+            -0.006084812339395285,
+            0.026795072481036186,
+            0.00516583351418376,
+            0.026138100773096085,
+            0.023197369650006294,
+            -0.04229648411273956,
+            0.024495670571923256,
+            0.0007200295804068446,
+            0.002708054380491376,
+            0.009463525377213955,
+            0.003918368835002184,
+            -0.01892705075442791,
+            -0.03281731531023979,
+            -0.015368452295660973,
+            0.02729562297463417,
+            -0.03203520551323891,
+            -0.025105716660618782,
+            0.02580961398780346,
+            0.010190887376666069,
+            0.014633269049227238,
+            0.006882563699036837,
+            -0.03963731229305267,
+            -0.007930590771138668,
+            0.002023708540946245,
+            -0.007789811119437218,
+            0.05969059839844704,
+            0.04886620491743088,
+            0.0029661504086107016,
+            -0.01457070093601942,
+            -0.030580483376979828,
+            0.00900990143418312,
+            -0.016095813363790512,
+            -0.03538263589143753,
+            0.029454244300723076,
+            -0.026341449469327927,
+            0.04339143633842468,
+            0.025668835267424583,
+            0.017049988731741905,
+            -0.017706960439682007,
+            -0.013577421195805073,
+            -0.012983018532395363,
+            -0.03191006928682327,
+            -0.03363071009516716,
+            0.008008801378309727,
+            0.001456678961403668,
+            -0.022947093471884727,
+            -0.011410977691411972,
+            -0.018332647159695625,
+            0.012099234387278557,
+            0.03813566267490387,
+            0.024667734280228615,
+            0.009948433376848698,
+            0.004305513110011816,
+            0.024307964369654655,
+            -0.035883184522390366,
+            -0.0030443614814430475,
+            -0.0018663088558241725,
+            -0.009846759028732777,
+            -0.015032145194709301,
+            -0.017534896731376648,
+            0.001207381603308022,
+            -0.011551758274435997,
+            -0.035069793462753296,
+            0.008712700568139553,
+            -0.0028762079309672117,
+            0.010229992680251598,
+            -0.012709279544651508,
+            0.015274599194526672,
+            0.022743744775652885,
+            -0.023823056370019913,
+            -0.049022626131772995,
+            0.014437741599977016,
+            0.020710259675979614,
+            -0.02784309908747673,
+            -0.015751685947179794,
+            -0.009862401522696018,
+            -0.0019249670440331101,
+            0.02407333068549633,
+            -0.04539363831281662,
+            -0.021257737651467323,
+            0.02471466176211834,
+            -0.019646592438220978,
+            -0.011465725488960743,
+            -0.026310164481401443,
+            -0.005298791918903589,
+            0.04192107170820236,
+            0.033755846321582794,
+            0.03907419368624687,
+            -0.06350729614496231,
+            0.002970061032101512,
+            0.019521454349160194,
+            0.03366199508309364,
+            -0.058752067387104034,
+            0.027702320367097855,
+            -0.00732445577159524,
+            -0.011583042331039906,
+            0.021914709359407425,
+            0.043766848742961884,
+            -0.023948194459080696,
+            -0.0025340348947793245,
+            0.017941592261195183,
+            -0.03485080227255821,
+            -0.02701406367123127,
+            -0.0019709160551428795,
+            0.005580351687967777,
+            -0.044236116111278534,
+            -0.03519492968916893,
+            0.00030037894612178206,
+            0.014343888498842716,
+            0.0040161325596272945,
+            -0.008650131523609161,
+            -0.03538263589143753,
+            -0.050899688154459,
+            0.02390126697719097,
+            0.0010548701975494623,
+            -0.004340707790106535,
+            -0.04160822555422783,
+            -0.014828796498477459,
+            -0.0064132981933653355,
+            0.01023781392723322,
+            0.0010734453098848462,
+            0.023275580257177353,
+            0.013155082240700722,
+            0.0010812664404511452,
+            -0.00877526868134737,
+            -0.04132666811347008,
+            -0.032754749059677124,
+            -0.01065233163535595,
+            0.007742884568870068,
+            0.029907869175076485,
+            0.024229753762483597,
+            -0.048021525144577026,
+            -0.018989618867635727,
+            0.03123745508491993,
+            -0.03869878128170967,
+            -0.014547237195074558,
+            0.04045070707798004,
+            0.018520353361964226,
+            0.0262006688863039,
+            0.00544739281758666,
+            0.00310106435790658,
+            0.043891988694667816,
+            0.0001796407887013629,
+            0.006757426541298628,
+            0.021523654460906982,
+            0.0043133338913321495,
+            0.01830136403441429,
+            -0.0014752540737390518,
+            -0.0040943436324596405,
+            0.0452372170984745,
+            -0.0049585746601223946,
+            0.03050227276980877,
+            -0.000946841377299279,
+            -0.020084572955965996,
+            0.020929250866174698,
+            0.025309065356850624,
+            0.00022534531308338046,
+            0.009064649231731892,
+            -0.011833317577838898,
+            0.0026200669817626476,
+            -0.004583161789923906,
+            -0.03566419333219528,
+            -0.013913728296756744,
+            -0.011794212274253368,
+            0.013561779633164406,
+            0.02444874495267868,
+            -0.0017939637182280421,
+            -0.015196388587355614,
+            -0.034318964928388596,
+            -0.0126779954880476,
+            -0.007817184552550316,
+            0.04564391076564789,
+            -0.004066969733685255,
+            0.012803133577108383,
+            -0.008978617377579212,
+            0.02424539625644684,
+            0.01015178207308054,
+            -0.003208604408428073,
+            0.03181621432304382,
+            -0.02047562785446644,
+            0.008110475726425648,
+            -0.020632049068808556,
+            -0.026716861873865128,
+            -0.021007461473345757,
+            -0.0005797386984340847,
+            0.03747868910431862,
+            0.023213010281324387,
+            0.009494809433817863,
+            -0.048303086310625076,
+            0.02471466176211834,
+            -0.047646112740039825,
+            0.01894269324839115,
+            0.01892705075442791,
+            -0.00530661316588521,
+            -0.013381893746554852,
+            0.008462425321340561,
+            -0.016127098351716995,
+            0.006890384946018457,
+            0.01928682066500187,
+            -0.025121359154582024,
+            0.00408652238547802,
+            0.000701943296007812,
+            0.03628988191485405,
+            -0.014719301834702492,
+            -0.04070097953081131,
+            -0.012107055634260178,
+            0.003024808596819639,
+            0.029188327491283417,
+            0.009635589085519314,
+            -0.026388375088572502,
+            0.0011487234150990844,
+            0.002221191069111228,
+            -0.009135039523243904,
+            -0.019599664956331253,
+            -0.026044247671961784,
+            -0.009995359927415848,
+            -0.054528675973415375,
+            0.010276919230818748,
+            -0.016220951452851295,
+            -0.011082491837441921,
+            -0.05396555736660957,
+            0.03325529769062996,
+            -0.06601004302501678,
+            -0.01033166702836752,
+            0.04949188977479935,
+            -0.004141270183026791,
+            -0.006733963266015053,
+            -0.007735063321888447,
+            -0.014351709745824337,
+            -0.039293184876441956,
+            0.022649891674518585,
+            0.0009991449769586325,
+            0.04877234995365143,
+            -0.02178957127034664,
+            -0.003578151110559702,
+            0.032473187893629074,
+            -0.018348289653658867,
+            -0.016009781509637833,
+            0.0018839063122868538,
+            0.028046447783708572,
+            -0.016877923160791397,
+            0.025496771559119225,
+            0.0018457785481587052,
+            0.018864482641220093,
+            -0.001967005431652069,
+            0.025293422862887383,
+            -0.0033298314083367586,
+            -0.026591723784804344,
+            -0.0008060616673901677,
+            0.04273446649312973,
+            0.018066730350255966,
+            -0.023228652775287628,
+            -0.018082372844219208,
+            -0.022915810346603394,
+            -0.02546548657119274,
+            -0.00638201367110014,
+            -0.008861301466822624,
+            -0.0019112802110612392,
+            0.0251682847738266,
+            0.030705619603395462,
+            -0.011066850274801254,
+            0.007918858900666237,
+            0.00011420021473895758,
+            0.022853240370750427,
+            0.0001724307076074183,
+            -0.017315905541181564,
+            -0.009666874073445797,
+            -0.007879753597080708,
+            0.016971776261925697,
+            0.0254498440772295,
+            -0.039199329912662506,
+            -0.021727003157138824,
+            0.017753886058926582,
+            -0.012153982184827328,
+            0.0022505202796310186,
+            0.02277502976357937,
+            0.011684716679155827,
+            0.02279067225754261,
+            -0.019177325069904327,
+            -0.01700306124985218,
+            0.024323606863617897,
+            -0.018129298463463783,
+            0.00502896448597312,
+            -0.019224252551794052,
+            -0.017128199338912964,
+            0.023744845762848854,
+            -0.004293781239539385,
+            0.0253559909760952,
+            0.01231822557747364,
+            0.016299162060022354,
+            -0.016408657655119896,
+            0.006925580091774464,
+            -0.015344989486038685,
+            0.029172684997320175,
+            -0.04179593548178673,
+            0.014085792936384678,
+            0.029626309871673584,
+            -0.00041891742148436606,
+            0.049585744738578796,
+            -0.043766848742961884,
+            -0.012513753026723862,
+            0.0074339513666927814,
+            0.019161684438586235,
+            -0.012419898994266987,
+            0.0007356717833317816,
+            -0.006999880541115999,
+            0.007672494743019342,
+            -0.00431724451482296,
+            -0.03679043427109718,
+            -0.004954664036631584,
+            0.01919296756386757,
+            -0.008376393467187881,
+            -0.0028586103580892086,
+            -0.0051971180364489555,
+            -0.007871932350099087,
+            0.002614201046526432,
+            0.05578005313873291,
+            -0.04630088433623314,
+            0.03391227126121521,
+            0.00905682798475027,
+            0.013186367228627205,
+            -0.031206170096993446,
+            -0.011246735230088234,
+            -0.003703288733959198,
+            0.00615911278873682,
+            -0.028797272592782974,
+            0.011614326387643814,
+            0.01033948827534914,
+            0.00864231027662754,
+            -0.01892705075442791,
+            -0.03253575786948204,
+            0.006624467670917511,
+            -0.028969338163733482,
+            0.0020784561056643724,
+            -0.0056742047891020775,
+            -0.007390934973955154,
+            -0.0034354161471128464,
+            0.025011863559484482,
+            -0.010667974129319191,
+            0.0014820975484326482,
+            -0.004305513110011816,
+            0.008955154567956924,
+            -0.00822779256850481,
+            -0.002438226481899619,
+            0.00794623326510191,
+            0.005959674715995789,
+            0.016862282529473305,
+            -0.007109375670552254,
+            -0.018129298463463783,
+            -0.004399366211146116,
+            0.01466455403715372,
+            -0.022540396079421043,
+            0.0036817805375903845,
+            0.03391227126121521,
+            0.0363837368786335,
+            0.0018242704682052135,
+            -0.010441162623465061,
+            -0.04254676029086113,
+            0.03409997746348381,
+            -0.013765127398073673,
+            0.01883319765329361,
+            0.027514614164829254,
+            -0.009776369668543339,
+            -0.04201492294669151,
+            -0.0014908963348716497,
+            -0.01442210003733635,
+            -0.011199808679521084,
+            0.017034346237778664,
+            -0.00836857222020626,
+            -0.02435489185154438,
+            -0.03162850812077522,
+            0.026341449469327927,
+            0.02727998048067093,
+            -0.00629207119345665,
+            -0.00733227701857686,
+            0.013452284038066864,
+            0.022118058055639267,
+            0.017112556844949722,
+            -0.016205308958888054,
+            0.01300648134201765,
+            -0.01231040433049202,
+            0.014304783195257187,
+            0.007840648293495178,
+            -0.009971896186470985,
+            0.004379813559353352,
+            -0.0043915449641644955,
+            0.006726142019033432,
+            -0.0004186730075161904,
+            -0.012232192791998386,
+            -0.01474276464432478,
+            0.03510107472538948,
+            0.004325065761804581,
+            0.006601004395633936,
+            0.005240133963525295,
+            0.028093375265598297,
+            0.035977039486169815,
+            -0.023635350167751312,
+            -0.0032281570602208376,
+            0.01866113394498825,
+            0.009189787320792675,
+            -0.03209777548909187,
+            -0.0107931112870574,
+            -0.020428700372576714,
+            -0.03014250099658966,
+            0.006796531844884157,
+            -0.007692047394812107,
+            0.013264577835798264,
+            0.009619947522878647,
+            0.026263238862156868,
+            -0.004149090964347124,
+            -0.0174410417675972,
+            0.03076818957924843,
+            -0.002999390009790659,
+            0.03907419368624687,
+            -0.003232067683711648,
+            -0.0027647572569549084,
+            -0.01093389093875885,
+            0.020147141069173813,
+            0.04066969454288483,
+            0.009041186422109604,
+            0.0033884895965456963,
+            0.011403157375752926,
+            0.022947093471884727,
+            0.044611528515815735,
+            -0.03632116690278053,
+            -0.016330447047948837,
+            -0.0011741418857127428,
+            -0.018410857766866684,
+            0.029172684997320175,
+            0.009158502332866192,
+            0.015032145194709301,
+            0.00863448902964592,
+            0.001260173972696066,
+            -0.0147505858913064,
+            0.03503850847482681,
+            -0.008783089928328991,
+            -0.015462305396795273,
+            0.02987658418715,
+            -0.02030356414616108,
+            0.03325529769062996,
+            0.009682515636086464,
+            -0.024558238685131073,
+            -0.024386174976825714,
+            0.0011389469727873802,
+            -0.0038851292338222265,
+            -0.027983879670500755,
+            0.031096674501895905,
+            0.029250897467136383,
+            -0.0049155582673847675,
+            0.006808263715356588,
+            0.009095934219658375,
+            0.02114824205636978,
+            -0.020240994170308113,
+            -0.016518153250217438,
+            0.017128199338912964,
+            -0.007171944249421358,
+            -0.022196268662810326,
+            0.03976244851946831,
+            -0.01856728084385395,
+            -0.011723821982741356,
+            -0.012302583083510399,
+            0.0032692179083824158,
+            0.011465725488960743,
+            0.017800813540816307,
+            0.03494465351104736,
+            -0.001723573892377317,
+            -0.028703419491648674,
+            0.021617507562041283,
+            0.026435302570462227,
+            -0.013170724734663963,
+            -0.019349390640854836,
+            0.025387275964021683,
+            -0.01855163834989071,
+            0.013358430936932564,
+            0.05161922797560692,
+            0.008251255378127098,
+            0.008955154567956924,
+            -0.024745944887399673,
+            -0.02178957127034664,
+            0.025387275964021683,
+            -0.017300263047218323,
+            0.04307859390974045,
+            0.013131619431078434,
+            0.02379177138209343,
+            0.007864111103117466,
+            -0.023040946573019028,
+            0.0011037520598620176,
+            0.0037756338715553284,
+            0.004927290137857199,
+            0.02369791828095913,
+            0.016846640035510063,
+            0.0032203360460698605,
+            0.005208849441260099,
+            -0.04514336213469505,
+            -0.014117076992988586,
+            -0.014062329195439816,
+            0.009768548421561718,
+            0.01747232675552368,
+            0.00155053217895329,
+            0.0033337417989969254,
+            0.005451303441077471,
+            -0.01506342925131321,
+            -0.033505573868751526,
+            0.01698741875588894,
+            -0.0031284382566809654,
+            -0.003015032270923257,
+            0.0047317626886069775,
+            0.011770748533308506,
+            -0.006909937597811222,
+            -0.0012777714291587472,
+            0.005025053862482309,
+            -0.011278019286692142,
+            -0.02269681915640831,
+            0.0009082247270271182,
+            -0.009698158130049706,
+            0.0358206182718277,
+            0.012552858330309391,
+            0.024964936077594757,
+            -0.002391299931332469,
+            0.01024563517421484,
+            -0.003580106422305107,
+            -0.016017602756619453,
+            0.012709279544651508,
+            -0.0076764049008488655,
+            -0.007246244698762894,
+            0.0022954915184527636,
+            0.027233054861426353,
+            -0.016393015161156654,
+            0.007261887192726135,
+            -0.013233293779194355,
+            0.020632049068808556,
+            0.046895287930965424,
+            0.002715875394642353,
+            0.0034667004365473986,
+            0.0422339141368866,
+            -0.009221071377396584,
+            0.0026591725181788206,
+            0.025793973356485367,
+            0.0017382384976372123,
+            -0.016705859452486038,
+            0.04282831773161888,
+            -0.0063350871205329895,
+            0.042390335351228714,
+            -0.044517673552036285,
+            -0.0013246979797258973,
+            0.0358206182718277,
+            -0.01025345642119646,
+            -0.008235613815486431,
+            -0.02599732019007206,
+            0.002567274495959282,
+            -0.020538195967674255,
+            0.021977277472615242,
+            -0.007359650917351246,
+            0.038730062544345856,
+            -0.03944960609078407,
+            0.004434560891240835,
+            0.015399736352264881,
+            -0.01216962467879057,
+            0.036164745688438416,
+            0.01737847365438938,
+            -0.031096674501895905,
+            -0.034600526094436646,
+            0.006663573440164328,
+            0.010175244882702827,
+            0.05446610972285271,
+            -0.01272492203861475,
+            0.008306003175675869,
+            0.027154842391610146,
+            -0.009979717433452606,
+            -0.00019626061839517206,
+            0.043516576290130615,
+            -0.010511551983654499,
+            0.02637273445725441,
+            0.03916804492473602,
+            -0.053621429949998856,
+            -0.02756153978407383,
+            0.019552739337086678,
+            -0.0008725409279577434,
+            0.011301483027637005,
+            0.010941712185740471,
+            0.014085792936384678,
+            -0.0028801183216273785,
+            -0.027342548593878746,
+            0.01938067376613617,
+            0.013585242442786694,
+            0.0015671519795432687,
+            -0.004997679963707924,
+            -0.016752786934375763,
+            -0.012380793690681458,
+            -0.01939631626009941,
+            -0.009979717433452606,
+            0.005025053862482309,
+            0.0025281691923737526,
+            -0.017425401136279106,
+            -0.04326630011200905,
+            -0.02929782308638096,
+            0.027045348659157753,
+            -0.0074691460467875,
+            0.03300502151250839,
+            0.023400716483592987,
+            0.023025304079055786,
+            0.0026024694088846445,
+            -0.025856541469693184,
+            0.03841722011566162,
+            -0.011817675083875656,
+            0.03372456505894661,
+            -0.0022251016926020384,
+            -0.015962855890393257,
+            -0.027968237176537514,
+            0.04310987889766693,
+            -0.01254503708332777,
+            0.006952953990548849,
+            -0.011739464476704597,
+            -0.0036153013352304697,
+            -0.004387634340673685,
+            -0.01066015288233757,
+            0.011184166185557842,
+            0.004974216688424349,
+            0.016877923160791397,
+            0.0008544546435587108,
+            0.021163884550333023,
+            0.017206409946084023,
+            -0.0030033006332814693,
+            0.018144940957427025,
+            -0.027858741581439972,
+            -0.02452695555984974,
+            0.014304783195257187,
+            -0.02684199996292591,
+            -0.027717962861061096,
+            -0.008783089928328991,
+            -0.04051327332854271,
+            -0.006941222120076418,
+            -0.0010157647775486112,
+            -0.010503730736672878,
+            0.035069793462753296,
+            0.03907419368624687,
+            0.015642190352082253,
+            -0.03575804829597473,
+            0.022446542978286743,
+            -0.04407969489693642,
+            -0.008798732422292233,
+            0.0030912880320101976,
+            0.011184166185557842,
+            -0.022305764257907867,
+            0.03278603032231331,
+            0.00886912178248167,
+            -0.008259076625108719,
+            -0.028500070795416832,
+            0.004899916239082813,
+            -0.01030038297176361,
+            -0.0003243310493417084,
+            -0.00799707043915987,
+            0.011794212274253368,
+            0.0257314033806324,
+            0.008118296973407269,
+            0.0005758281331509352,
+            0.002829281147569418,
+            -0.03779153153300285,
+            0.00258291675709188,
+            -0.00422339141368866,
+            0.0023639260325580835,
+            -0.009878043085336685,
+            -0.020147141069173813,
+            -0.005146280862390995,
+            0.002923134481534362,
+            0.003040450857952237,
+            -0.004020043183118105,
+            0.01873934455215931,
+            -0.015657832846045494,
+            0.009838937781751156,
+            0.005220581311732531,
+            -0.04154565930366516,
+            -0.01994379237294197,
+            -0.04526849836111069,
+            -0.0022075041197240353,
+            0.0002268117677886039,
+            0.022008562460541725,
+            -0.007723331451416016,
+            0.009596483781933784,
+            -0.020882325246930122,
+            -0.010832217521965504,
+            -0.0006887452327646315,
+            0.017315905541181564,
+            -0.028922410681843758,
+            0.00019540518405847251,
+            0.006601004395633936,
+            0.02986094169318676,
+            -0.04066969454288483,
+            0.007336187642067671,
+            0.029078831896185875,
+            -0.017988519743084908,
+            -0.004544056486338377,
+            -0.0060965437442064285,
+            0.028875483199954033,
+            -0.024745944887399673,
+            0.0034803873859345913,
+            -0.0065384358167648315,
+            0.006511061917990446,
+            0.01836393214762211,
+            0.01501650270074606,
+            -0.0008935601217672229,
+            0.015524874441325665,
+            0.02288452535867691,
+            -0.0020765007939189672,
+            0.03832336887717247,
+            -0.03963731229305267,
+            -0.03832336887717247,
+            0.028984978795051575,
+            -0.017597464844584465,
+            -0.0031382145825773478,
+            0.009408777579665184,
+            -0.052964456379413605,
+            -0.0026806804817169905,
+            0.011786391027271748,
+            -0.002029574243351817,
+            -0.05671858415007591,
+            0.027968237176537514,
+            0.03406869247555733,
+            0.02527778036892414,
+            0.0013012347044423223,
+            -0.012388614937663078,
+            -0.0006960774771869183,
+            0.02444874495267868,
+            0.012568499892950058,
+            0.0294073186814785,
+            0.014250035397708416,
+            -0.013749485835433006,
+            -0.0008559211273677647,
+            -0.014860081486403942,
+            -0.014719301834702492,
+            -0.017128199338912964,
+            0.016956135630607605,
+            -0.009893685579299927,
+            -0.01563436910510063,
+            0.006186486221849918,
+            -0.005009411368519068,
+            -0.0148444389924407,
+            -0.03935575112700462,
+            -0.03613346070051193,
+            0.018317004665732384,
+            -0.015313704498112202,
+            -0.04861592873930931,
+            0.0065384358167648315,
+            -0.009932790882885456,
+            -0.004446292761713266,
+            -0.007062449119985104,
+            0.010941712185740471,
+            -0.01598631963133812,
+            0.013436641544103622,
+            0.005990959238260984,
+            0.0004746427293866873,
+            0.0003773678618017584,
+            0.018410857766866684,
+            0.01873934455215931,
+            0.013843338936567307,
+            -0.02480851486325264,
+            0.03106539137661457,
+            -0.016612006351351738,
+            -0.05155666172504425,
+            0.020647691562771797,
+            -0.015352809801697731,
+            0.003449103096500039,
+            0.027702320367097855,
+            -0.010511551983654499,
+            -0.02914140187203884,
+            0.028155943378806114,
+            -0.031925711780786514,
+            -0.02343200147151947,
+            -0.022853240370750427,
+            -0.00033044128213077784,
+            -0.03876134753227234,
+            -0.0450495108962059,
+            -0.007042896468192339,
+            0.011262377724051476,
+            -0.00849370937794447,
+            0.0004577784857247025,
+            0.03463181108236313,
+            -0.00357228540815413,
+            -0.03475694730877876,
+            -0.007488698698580265,
+            -0.005251865368336439,
+            -0.009885864332318306,
+            -0.0026865461841225624,
+            0.00029206901672296226,
+            0.004720031283795834,
+            -0.006663573440164328,
+            -0.0262006688863039,
+            -0.022352689877152443,
+            -0.016064530238509178,
+            -0.007344008423388004,
+            -0.01956837996840477,
+            -0.02003764547407627,
+            -0.011098134331405163,
+            -0.03306759148836136,
+            0.03076818957924843,
+            0.026075532659888268,
+            -0.006542346440255642,
+            -0.009174144826829433,
+            -0.009526094421744347,
+            0.0020706350915133953,
+            0.02993915230035782,
+            0.006053527817130089,
+            0.005658562295138836,
+            -0.020272279158234596,
+            0.004829526413232088,
+            -0.011317124590277672,
+            0.011567399837076664,
+            -0.051775652915239334,
+            -0.017534896731376648,
+            0.039199329912662506,
+            0.02271246165037155,
+            -0.0005782722146250308,
+            -0.006409387569874525,
+            0.018535995855927467,
+            -0.029907869175076485,
+            -0.028343649581074715,
+            0.03876134753227234,
+            0.015376273542642593,
+            0.013905907981097698,
+            -0.005506051238626242,
+            0.003539045574143529,
+            0.012787491083145142,
+            0.0019386539934203029,
+            -0.01042552012950182,
+            -0.006022243294864893,
+            0.03363071009516716,
+            0.0017558359540998936,
+            0.0022172804456204176,
+            -0.00822779256850481,
+            0.023478928953409195,
+            0.01023781392723322,
+            -0.0019738490227609873,
+            0.01773824356496334,
+            0.01043334137648344,
+            -0.01709691435098648,
+            -0.011137239634990692,
+            -0.02473030425608158,
+            0.0035527325235307217,
+            0.02260296605527401,
+            0.010034465231001377,
+            -0.02379177138209343,
+            -0.0061982180923223495,
+            -0.006718320772051811,
+            0.00267676985822618,
+            -0.012052307836711407,
+            0.012302583083510399,
+            0.025105716660618782,
+            -0.030940253287553787,
+            -0.014867902733385563,
+            -0.008259076625108719,
+            0.005032874643802643,
+            0.01033948827534914,
+            0.02086668275296688,
+            0.019693518057465553,
+            -0.0020765007939189672,
+            -0.004626177717000246,
+            0.0063116238452494144,
+            0.02756153978407383,
+            0.016580721363425255,
+            -0.02555933967232704,
+            -0.020694619044661522,
+            0.012333867140114307,
+            0.009111575782299042,
+            0.034788232296705246,
+            -0.012279119342565536,
+            -0.00044995738426223397,
+            -0.01496957615017891,
+            0.09547993540763855,
+            -0.01636173203587532,
+            0.026998421177268028,
+            -0.016048887744545937,
+            -0.02857828326523304,
+            0.00028815848054364324,
+            0.003701333422213793,
+            0.011223271489143372,
+            0.05559234693646431,
+            -0.00151631492190063,
+            0.03076818957924843,
+            0.012044486589729786,
+            0.0022075041197240353,
+            0.0029837479814887047,
+            -0.00746132479980588,
+            0.016205308958888054,
+            -0.006421119440346956,
+            0.02351021207869053,
+            0.0110746705904603,
+            0.012920449487864971,
+            -0.009784189984202385,
+            0.02335379086434841,
+            -0.0007136749336495996,
+            0.020725902169942856,
+            -0.023213010281324387,
+            0.04016914591193199,
+            0.00014676773571409285,
+            0.0036368092987686396,
+            -0.004501040559262037,
+            -0.025215212255716324,
+            -0.0044150082394480705,
+            0.011090313084423542,
+            -0.014914829283952713,
+            0.011317124590277672,
+            -0.019318105652928352,
+            0.05318344756960869,
+            -0.02793695218861103,
+            0.022164983674883842,
+            -0.005373092368245125,
+            -0.03603960573673248,
+            0.014406457543373108,
+            0.028187228366732597,
+            0.008016622625291348,
+            0.0028957605827599764,
+            -0.013467926532030106,
+            0.012599784880876541,
+            0.022305764257907867,
+            -0.02030356414616108,
+            2.63045440078713e-05,
+            0.025934752076864243,
+            -0.016877923160791397,
+            0.02307223156094551,
+            0.009674695320427418,
+            -0.02279067225754261,
+            -0.005322255194187164,
+            -0.006843458395451307,
+            0.009095934219658375,
+            -0.005658562295138836,
+            -0.008618847467005253,
+            -0.01038641482591629,
+            0.010456804186105728,
+            0.022759387269616127,
+            0.027702320367097855,
+            0.019271178171038628,
+            -0.01037859357893467,
+            -0.0015642191283404827,
+            0.0007234513177536428,
+            -0.014711480587720871,
+            -0.002647440880537033,
+            -0.012240014038980007,
+            -0.01819186843931675,
+            0.02682635746896267,
+            0.0035058059729635715,
+            -0.003580106422305107,
+            -0.015251136384904385,
+            -0.013413178734481335,
+            -0.02105438895523548,
+            -0.0006515950080938637,
+            -0.0029641950968652964,
+            -0.015696939080953598,
+            -0.003108885372057557,
+            0.015501410700380802,
+            -0.015337168239057064,
+            0.018410857766866684,
+            -0.02482415735721588,
+            0.013718200847506523,
+            -0.027326907962560654,
+            -0.0035234035458415747,
+            0.012599784880876541,
+            -0.012466826476156712,
+            -0.0032574862707406282,
+            -0.01884884014725685,
+            -0.011723821982741356,
+            0.010190887376666069,
+            -0.01836393214762211,
+            -0.007449593394994736,
+            0.007738973945379257,
+            0.005408287514001131,
+            -0.017972877249121666,
+            -0.027170484885573387,
+            -0.022118058055639267,
+            0.005865821614861488,
+            0.04070097953081131,
+            -0.02020971104502678,
+            0.00018428456678520888,
+            -0.007211050018668175,
+            0.006569720339030027,
+            0.0013706469908356667,
+            -0.013772948645055294,
+            -0.0007038986077532172,
+            0.0034354161471128464,
+            -0.004211660008877516,
+            0.024057690054178238,
+            0.006565809715539217,
+            0.023948194459080696,
+            0.005728952586650848,
+            0.0012005381286144257,
+            -0.02066333405673504,
+            -0.016565080732107162,
+            0.016752786934375763,
+            -0.010097034275531769,
+            0.014351709745824337,
+            0.02435489185154438,
+            -0.021429801359772682,
+            0.0012924360344186425,
+            0.01424221508204937,
+            0.014038866385817528,
+            -0.02663865126669407,
+            0.024949293583631516,
+            0.006421119440346956,
+            0.014414278790354729,
+            -0.0021195169538259506,
+            0.02325993776321411,
+            -0.007848469540476799,
+            0.033505573868751526,
+            -0.011504831723868847,
+            -0.00804008636623621,
+            0.0010754006216302514,
+            0.030189428478479385,
+            -0.006370282266288996,
+            -0.00020762563508469611,
+            0.019224252551794052,
+            0.043516576290130615,
+            0.0035664194729179144,
+            -0.0016551392618566751,
+            -0.013452284038066864,
+            0.010362951084971428,
+            0.008259076625108719,
+            0.021382873877882957,
+            0.00450495071709156,
+            0.001804717699997127,
+            -0.008243434131145477,
+            0.017597464844584465,
+            0.006671394221484661,
+            0.011512652039527893,
+            -0.026325806975364685,
+            0.0015642191283404827,
+            0.0070389858447015285,
+            0.010073570534586906,
+            0.002948553068563342,
+            -0.0026885014958679676,
+            0.020632049068808556,
+            -0.010769648477435112,
+            -0.0360083244740963,
+            0.024464385583996773,
+            0.002373702358454466,
+            0.024182826280593872,
+            -0.007054627873003483,
+            -0.015869002789258957,
+            0.02103874646127224,
+            0.002291580894961953,
+            0.006503240671008825,
+            0.025574982166290283,
+            0.0017206409247592092,
+            -0.00269827782176435,
+            0.005623367615044117,
+            0.01005792897194624,
+            0.0031049747485667467,
+            0.0009292439208365977,
+            -0.003840157762169838,
+            0.007257976569235325,
+            0.010448983870446682,
+            0.006080901715904474,
+            -0.002289625583216548,
+            -0.028328007087111473,
+            -0.0009043141617439687,
+            0.009768548421561718,
+            0.0063585503958165646,
+            -0.039825018495321274,
+            0.01497739739716053,
+            -0.006827816367149353,
+            0.004035685211420059,
+            0.0074378615245223045,
+            -0.008478067815303802,
+            -0.012083591893315315,
+            0.01245900522917509,
+            0.0018506667111068964,
+            -0.006898206193000078,
+            0.0043602604418993,
+            -0.03660272806882858,
+            -0.01745668426156044,
+            0.023275580257177353,
+            0.003564464161172509,
+            -0.021429801359772682,
+            0.014117076992988586,
+            -0.007707689423114061,
+            -0.010143960826098919,
+            0.005322255194187164,
+            0.02757718227803707,
+            -0.044705379754304886,
+            0.013694738037884235,
+            -0.02352585457265377,
+            -0.012552858330309391,
+            0.021617507562041283,
+            0.010003181174397469,
+            0.0041295383125543594,
+            -0.00101087661460042,
+            -0.011426620185375214,
+            0.005904926918447018,
+            0.012889165431261063,
+            -0.011895885691046715,
+            -0.020162783563137054,
+            0.0217739287763834,
+            0.012928270734846592,
+            0.005944032687693834,
+            -0.0074339513666927814,
+            0.0215862225741148,
+            -0.010644510388374329,
+            1.3480696452461416e-06,
+            0.009275819174945354,
+            -0.0454249233007431,
+            0.030721262097358704,
+            -0.0043328870087862015,
+            0.02360406517982483,
+            -0.008024443872272968,
+            -0.028328007087111473,
+            -0.008978617377579212,
+            0.01645558513700962,
+            0.018692417070269585,
+            0.005717220716178417,
+            0.0061982180923223495,
+            0.032019563019275665,
+            -0.0129595547914505,
+            0.013819875195622444,
+            0.0057993424125015736,
+            -0.00016289875202346593,
+            0.0261068157851696,
+            -0.016752786934375763,
+            -0.03130002319812775,
+            0.01428914163261652,
+            -0.012513753026723862,
+            -0.03663400933146477,
+            0.015337168239057064,
+            -0.02103874646127224,
+            -0.03397483751177788,
+            -0.029986079782247543,
+            -0.0035136269871145487,
+            -0.007344008423388004,
+            -0.0014830752043053508,
+            -0.003116706619039178,
+            0.016924850642681122,
+            0.023932551965117455,
+            0.012005381286144257,
+            0.01525895670056343,
+            -0.006511061917990446,
+            0.01419528853148222,
+            -0.00397116132080555,
+            0.033224012702703476,
+            0.0012435541721060872,
+            0.004325065761804581,
+            0.001327630947344005,
+            0.017957234755158424,
+            0.013389714993536472,
+            0.020084572955965996,
+            0.009612126275897026,
+            -0.015368452295660973,
+            -0.01900526136159897,
+            -0.046738866716623306,
+            0.009760727174580097,
+            -0.004692657385021448,
+            0.006256876513361931,
+            0.03813566267490387,
+            -0.013186367228627205,
+            -0.010816575028002262,
+            0.009228892624378204,
+            0.004966395441442728,
+            0.0032437993213534355,
+            -0.013108155690133572,
+            -0.006100454367697239,
+            -0.0031499462202191353,
+            -0.02233704924583435,
+            -0.0253559909760952,
+            -0.018442142754793167,
+            0.020819755271077156,
+            -0.0017597464611753821,
+            0.013108155690133572,
+            -0.02077282965183258,
+            -0.0031577672343701124,
+            -0.028093375265598297,
+            -0.011191987432539463,
+            0.0013872667914256454,
+            0.013084692880511284,
+            0.004821705166250467,
+            0.016971776261925697,
+            -0.005330076441168785,
+            -0.020256636664271355,
+            -0.017800813540816307,
+            0.012075771577656269,
+            -0.010761827230453491,
+            -0.005947942845523357,
+            -0.004978127311915159,
+            0.01938067376613617,
+            0.007954053580760956,
+            0.004278139211237431,
+            -0.001422461704351008,
+            0.021899066865444183,
+            -0.021085673943161964,
+            -0.015955034643411636,
+            0.01747232675552368,
+            0.03262960910797119,
+            -0.03170672059059143,
+            -0.005552977789193392,
+            0.0009018700802698731,
+            0.014688016846776009,
+            0.015861181542277336,
+            0.012693637982010841,
+            -0.006557988468557596,
+            -0.008055727928876877,
+            -0.028046447783708572,
+            -0.01240425743162632,
+            -0.04095125570893288,
+            0.0029114028438925743,
+            0.029720162972807884,
+            -0.005025053862482309,
+            0.01911475695669651,
+            0.017988519743084908,
+            -0.027170484885573387,
+            6.660151848336682e-05,
+            0.004106075037270784,
+            0.017628749832510948,
+            -0.033849701285362244,
+            0.0145628796890378,
+            -0.009361851029098034,
+            0.02169571816921234,
+            0.0005733840516768396,
+            -0.01909911446273327,
+            -0.017753886058926582,
+            -0.011888065375387669,
+            0.021742645651102066,
+            -0.00253990083001554,
+            -0.029751446098089218,
+            0.021633150056004524,
+            0.009565199725329876,
+            -0.007136749569326639,
+            -0.024949293583631516,
+            0.012013202533125877,
+            0.0129595547914505,
+            0.017613107338547707,
+            -0.01074618473649025,
+            -0.002436271170154214,
+            -0.012568499892950058,
+            -0.010214350186288357,
+            -0.022211911156773567,
+            -0.00043211551383137703,
+            -0.017628749832510948,
+            0.0015945257619023323,
+            -0.011512652039527893,
+            -0.0022778939455747604,
+            0.014351709745824337,
+            -0.041483089327812195,
+            -0.0018457785481587052,
+            -0.008955154567956924,
+            0.028531355783343315,
+            0.02473030425608158,
+            0.008478067815303802,
+            0.0027960415463894606,
+            -0.02113259956240654,
+            -0.010816575028002262,
+            -0.011786391027271748,
+            0.01820751093327999,
+            -0.02261860854923725,
+            0.0003284859994892031,
+            0.010018822737038136,
+            0.008611026220023632,
+            0.0003871442168019712,
+            0.005114996340125799,
+            -0.002453868743032217,
+            -0.02094489336013794,
+            0.014641090296208858,
+            -0.0005171699449419975,
+            0.007003790698945522,
+            -0.012138339690864086,
+            0.0032262017484754324,
+            -0.0036778701469302177,
+            0.025668835267424583,
+            -0.014774048700928688,
+            -0.004387634340673685,
+            -0.010355129837989807,
+            0.0013110111467540264,
+            0.0070937336422502995,
+            -1.234266619576374e-05,
+            -0.0006423074519261718,
+            -0.010777469724416733,
+            0.01240425743162632,
+            -0.011160703375935555,
+            0.030439702793955803,
+            9.531959949526936e-05,
+            -0.03172236308455467,
+            -0.0036074803210794926,
+            -0.03766639530658722,
+            -0.007168034091591835,
+            -0.017800813540816307,
+            -0.033943552523851395,
+            -0.007430040743201971,
+            0.0017089092871174216,
+            -0.01493047084659338,
+            0.016283521428704262,
+            0.004559698514640331,
+            -0.02473030425608158,
+            0.013905907981097698,
+            0.006831726990640163,
+            -0.017894666641950607,
+            -0.0038440683856606483,
+            -0.01920861005783081,
+            0.0049116481095552444,
+            0.005142370238900185,
+            0.009987538680434227,
+            0.018348289653658867,
+            -0.007394845597445965,
+            -0.013022123835980892,
+            0.003024808596819639,
+            0.02096053585410118,
+            0.01483661774545908,
+            0.010824396274983883,
+            0.0068356371484696865,
+            0.00377758895047009,
+            -0.025043146684765816,
+            -0.003355249995365739,
+            -0.008149581030011177,
+            -0.0070389858447015285,
+            -0.0009810586925595999,
+            0.0034295502118766308,
+            -0.008337287232279778,
+            -0.0006525726639665663,
+            0.000982525059953332,
+            0.0018956379499286413,
+            0.0030072112567722797,
+            -0.013546137139201164,
+            0.0012865702155977488,
+            -0.008517173118889332,
+            0.002373702358454466,
+            0.024026405066251755,
+            -0.012185266241431236,
+            0.008650131523609161,
+            -0.0027510703075677156,
+            0.00634290836751461,
+            0.038636211305856705,
+            0.015352809801697731,
+            -0.044423822313547134,
+            0.003670048899948597,
+            0.007926680147647858,
+            0.0008823173120617867,
+            -0.017972877249121666,
+            0.017675675451755524,
+            0.004837347660213709,
+            -0.008650131523609161,
+            -0.002266162307932973,
+            -0.007844558916985989,
+            0.009385314770042896,
+            -0.015251136384904385,
+            0.012302583083510399,
+            0.013420999981462955,
+            -0.01438299473375082,
+            -0.004923379514366388,
+            0.038730062544345856,
+            -0.010941712185740471,
+            -0.011285840533673763,
+            0.0009121352341026068,
+            0.013319325633347034,
+            0.011262377724051476,
+            -0.004798241890966892,
+            0.014500310644507408,
+            0.0037365283351391554,
+            0.006186486221849918,
+            0.007218870799988508,
+            0.0597531683743,
+            0.0067769791930913925,
+            0.008251255378127098,
+            0.011364051140844822,
+            -0.007445682771503925,
+            -0.014500310644507408,
+            -0.018958335742354393,
+            -0.026357091963291168,
+            -0.004207749385386705,
+            0.01590810716152191,
+            -0.018270079046487808,
+            -0.0032653072848916054,
+            -0.02252475544810295,
+            0.01249811053276062,
+            0.013061229139566422,
+            0.005709399469196796,
+            -0.01717512495815754,
+            0.02066333405673504,
+            0.02793695218861103,
+            0.010308203287422657,
+            -0.0016746920300647616,
+            0.017034346237778664,
+            0.005036785267293453,
+            -0.015548337250947952,
+            -0.0023404627572745085,
+            -0.0023717472795397043,
+            0.004720031283795834,
+            0.021382873877882957,
+            -0.008251255378127098,
+            0.01221655122935772,
+            -0.0045166825875639915,
+            0.009221071377396584,
+            0.002291580894961953,
+            -0.013115976937115192,
+            -0.008306003175675869,
+            -0.0008432118338532746,
+            0.01792594976723194,
+            0.00013894664880353957,
+            -0.025011863559484482,
+            0.0030365402344614267,
+            0.0045948936603963375,
+            0.002862520981580019,
+            0.003812784096226096,
+            -0.014727122150361538,
+            -0.007938412018120289,
+            0.021914709359407425,
+            -0.016111455857753754,
+            0.0032066490966826677,
+            -0.012568499892950058,
+            0.0011369917774572968,
+            -0.01285788044333458,
+            0.0006750582833774388,
+            0.00397116132080555,
+            0.016612006351351738,
+            -0.027608467265963554,
+            -0.015352809801697731,
+            0.0217739287763834,
+            0.00836075097322464,
+            0.009049007669091225,
+            -0.0027256517205387354,
+            -0.005658562295138836,
+            0.016424300149083138,
+            -0.01783209666609764,
+            -0.013827696442604065,
+            0.005904926918447018,
+            -0.008024443872272968,
+            0.0052323127165436745,
+            -0.02296273596584797,
+            0.007969696074724197,
+            -0.008071370422840118,
+            -0.008165223523974419,
+            0.008212150074541569,
+            0.015829896554350853,
+            0.003789320820942521,
+            0.007918858900666237,
+            -0.0008104610024020076,
+            -0.01801980286836624,
+            -0.029516814276576042,
+            0.027311265468597412,
+            0.0043915449641644955,
+            -0.00404350645840168,
+            0.004219480790197849,
+            -0.0038049628492444754,
+            -0.004888184834271669,
+            -0.0021449353080242872,
+            -0.014641090296208858,
+            -0.02271246165037155,
+            0.013647811487317085,
+            0.012247835285961628,
+            -0.012294761836528778,
+            0.021289020776748657,
+            -0.008532814681529999,
+            0.007715510670095682,
+            -0.01791030913591385,
+            0.00850153062492609,
+            0.028703419491648674,
+            -0.006823905743658543,
+            -0.014359530992805958,
+            -0.0016248325118795037,
+            -0.008000980131328106,
+            0.0035370904952287674,
+            0.027264337986707687,
+            0.0062451446428895,
+            0.014531595632433891,
+            -0.01728462055325508,
+            -0.011176344938576221,
+            0.02243090234696865,
+            0.004434560891240835,
+            0.011207629926502705,
+            -0.0010001225164160132,
+            -0.009095934219658375,
+            -0.012099234387278557,
+            -0.00790321733802557,
+            0.010581942275166512,
+            0.012326045893132687,
+            -0.013288040645420551,
+            -0.0005117929540574551,
+            -0.007871932350099087,
+            0.005588172469288111,
+            -0.012505931779742241,
+            -0.011434441432356834,
+            -0.022008562460541725,
+            -0.008509351871907711,
+            -0.015532695688307285,
+            -0.026513513177633286,
+            -0.007168034091591835,
+            -0.0008265920332632959,
+            -0.012021023780107498,
+            -0.0369468554854393,
+            -0.022477827966213226,
+            -0.007547357119619846,
+            -0.019740445539355278,
+            0.008532814681529999,
+            0.0005748505354858935,
+            -0.021070031449198723,
+            0.006831726990640163,
+            -0.012349509634077549,
+            -0.009354029782116413,
+            0.00251448224298656,
+            0.01244336273521185,
+            -0.010065749287605286,
+            -0.014774048700928688,
+            -0.01240425743162632,
+            -0.01994379237294197,
+            0.0146567327901721,
+            -0.0027667125687003136,
+            0.003470611060038209,
+            -0.006299892440438271,
+            -0.004297691863030195,
+            -0.015454484149813652,
+            0.01994379237294197,
+            -0.02473030425608158,
+            0.0013139439979568124,
+            -0.0015867047477513552,
+            0.006929490715265274,
+            -0.003302457509562373,
+            -0.01745668426156044,
+            -0.009088112972676754,
+            -0.0010773559333756566,
+            -0.015313704498112202,
+            -0.02003764547407627,
+            -0.013014302588999271,
+            -0.005795431789010763,
+            0.015337168239057064,
+            -0.008564099669456482,
+            0.01273274328559637,
+            0.007922769524157047,
+            0.0035194929223507643,
+            -0.004422829486429691,
+            0.0035527325235307217,
+            -0.010738363489508629,
+            -0.0034803873859345913,
+            0.0017558359540998936,
+            0.005474766716361046,
+            0.006022243294864893,
+            0.0014263722114264965,
+            0.015939392149448395,
+            -0.009080291725695133,
+            0.012013202533125877,
+            0.020287921652197838,
+            0.02150801196694374,
+            0.006194307468831539,
+            0.00629207119345665,
+            0.004141270183026791,
+            -9.611393033992499e-05,
+            0.002846878720447421,
+            0.0023873893078416586,
+            -0.010980817489326,
+            0.012185266241431236,
+            -0.015587443485856056,
+            -0.008603204973042011,
+            0.013890265487134457,
+            -0.002537945518270135,
+            -0.015603085048496723,
+            0.015923749655485153,
+            0.0016746920300647616,
+            -0.02369791828095913,
+            -0.01062104757875204,
+            -0.007156302221119404,
+            0.004262497182935476,
+            0.020553838461637497,
+            0.007293171249330044,
+            -0.027811815962195396,
+            0.01708127185702324,
+            -0.0002189906663261354,
+            -0.013147260993719101,
+            -0.009103755466639996,
+            0.01837957464158535,
+            0.024042047560214996,
+            -0.0006647931295447052,
+            -0.04633216932415962,
+            -0.014226572588086128,
+            0.00808701291680336,
+            -0.026685576885938644,
+            -0.0285626407712698,
+            0.011684716679155827,
+            -0.00808701291680336,
+            -0.031565941870212555,
+            -0.005279239267110825,
+            -0.0007855312433093786,
+            -0.03078383207321167,
+            0.009494809433817863,
+            -0.0008705856744199991,
+            0.0032868152484297752,
+            0.007770258467644453,
+            -0.013647811487317085,
+            -0.00544348219409585,
+            -0.02471466176211834,
+            0.012286940589547157,
+            -0.007035075221210718,
+            -0.0038597106467932463,
+            -0.0036368092987686396,
+            0.009815474972128868,
+            -0.002762801945209503,
+            0.02279067225754261,
+            0.026701219379901886,
+            0.01966223306953907,
+            -0.003132348647341132,
+            -0.023588424548506737,
+            -0.009377493523061275,
+            0.002107785316184163,
+            0.012889165431261063,
+            0.006831726990640163,
+            -0.017018703743815422,
+            0.005678115412592888,
+            0.0014605895848944783,
+            0.013436641544103622,
+            0.000748869904782623,
+            0.02609117329120636,
+            -3.1039973691804335e-05,
+            0.002481242408975959,
+            0.002082366729155183,
+            -0.02580961398780346,
+            0.004680925514549017,
+            -0.0058110738173127174,
+            0.022477827966213226,
+            0.016784070059657097,
+            -0.0003084444615524262,
+            0.007011611945927143,
+            -0.0127718485891819,
+            -0.004727852065116167,
+            -0.028234153985977173,
+            -0.01709691435098648,
+            -0.00015947702922858298,
+            0.031002821400761604,
+            0.033130161464214325,
+            -0.02673250436782837,
+            -0.010136139579117298,
+            -0.0009385314187966287,
+            0.029735805466771126,
+            -0.006022243294864893,
+            -0.00218013022094965,
+            0.029000621289014816,
+            0.004485398065298796,
+            0.0030717351473867893,
+            0.012200908735394478,
+            0.0012767937732860446,
+            0.008861301466822624,
+            0.033568140119314194,
+            0.015915928408503532,
+            0.009627768769860268,
+            0.008861301466822624,
+            -0.002152756555005908,
+            -0.015837717801332474,
+            -0.021992919966578484,
+            -0.0020589034538716078,
+            -0.00652279332280159,
+            0.009486989118158817,
+            -0.018144940957427025,
+            0.006022243294864893,
+            -0.011786391027271748,
+            0.004125627689063549,
+            -0.011817675083875656,
+            -0.0022876705043017864,
+            0.0017450818559154868,
+            -0.01700306124985218,
+            -0.0025985590182244778,
+            -0.024980578571558,
+            -0.0027510703075677156,
+            0.0108869643881917,
+            -0.001176097197458148,
+            0.016127098351716995,
+            -0.016471227630972862,
+            0.0051541016437113285,
+            0.019177325069904327,
+            0.021289020776748657,
+            -0.01474276464432478,
+            6.71514353598468e-05,
+            0.003977026790380478,
+            -0.025981679558753967,
+            0.01579861342906952,
+            0.005158012267202139,
+            0.010175244882702827,
+            0.025684477761387825,
+            -0.0009443972376175225,
+            -0.0031831858213990927,
+            -0.012826596386730671,
+            -0.004532324615865946,
+            -0.03491336852312088,
+            0.024182826280593872,
+            -0.00034339496050961316,
+            0.009111575782299042,
+            0.004931200761348009,
+            -0.009627768769860268,
+            0.007574731018394232,
+            -0.010808753781020641,
+            0.019834298640489578,
+            0.005904926918447018,
+            0.003060003509745002,
+            -0.01492264959961176,
+            0.01653379574418068,
+            0.0025731404311954975,
+            -0.01894269324839115,
+            0.02463645115494728,
+            -0.01055847853422165,
+            0.0044932193122804165,
+            -0.0030580481979995966,
+            0.011160703375935555,
+            0.009690336883068085,
+            -0.006968596018850803,
+            0.004149090964347124,
+            -0.0296888779848814,
+            -0.02307223156094551,
+            -0.013037766329944134,
+            -0.0011389469727873802,
+            -0.007742884568870068,
+            -0.026435302570462227,
+            -0.022493470460176468,
+            0.004743494559079409,
+            -0.005294881761074066,
+            0.004688746761530638,
+            0.012208729982376099,
+            0.012654532678425312,
+            0.005728952586650848,
+            0.006331176497042179,
+            -0.009354029782116413,
+            0.006565809715539217,
+            0.026904568076133728,
+            -0.006565809715539217,
+            -0.03197263926267624,
+            -0.006933400873094797,
+            0.003554687835276127,
+            -0.018582923337817192,
+            -0.009948433376848698,
+            0.009486989118158817,
+            -0.004027863964438438,
+            0.0288442000746727,
+            -0.02380741387605667,
+            0.0038460236974060535,
+            -0.0038831739220768213,
+            0.004950753413140774,
+            -0.004806063137948513,
+            1.4840223229839467e-05,
+            -0.01911475695669651,
+            0.00017340834892820567,
+            0.004946842789649963,
+            -0.016846640035510063,
+            0.015696939080953598,
+            -0.007230602670460939,
+            -0.011019923724234104,
+            -0.007699868176132441,
+            0.0012846149038523436,
+            -0.007211050018668175,
+            -0.001168276066891849,
+            0.02627888135612011,
+            -0.007973606698215008,
+            -0.017394116148352623,
+            -0.004547967109829187,
+            0.0053574503399431705,
+            0.015149462036788464,
+            0.011473546735942364,
+            0.024057690054178238,
+            -0.013389714993536472,
+            0.003515582298859954,
+            -0.004059148486703634,
+            -0.0031734094955027103,
+            -4.744594116345979e-05,
+            0.039105478674173355,
+            -0.011434441432356834,
+            -0.0004949286812916398,
+            -0.00844678282737732,
+            0.009479167871177197,
+            0.006495419889688492,
+            0.020616406574845314,
+            0.006346818991005421,
+            0.002668948844075203,
+            0.007160212844610214,
+            -0.0006931445677764714,
+            0.0006760359392501414,
+            0.01460198499262333,
+            0.011184166185557842,
+            0.00723842391744256,
+            -0.0072814398445189,
+            -0.023025304079055786,
+            0.014234393835067749,
+            -0.0010841992916539311,
+            -0.0031675435602664948,
+            0.010261276736855507,
+            0.01212269812822342,
+            -0.009346208535134792,
+            0.012435541488230228,
+            -0.007703778799623251,
+            0.024761587381362915,
+            0.013702559284865856,
+            -0.01005792897194624,
+            0.009346208535134792,
+            0.021633150056004524,
+            -0.00032139813993126154,
+            0.015235493890941143,
+            -0.015423200093209743,
+            -0.021945994347333908,
+            0.0012210685526952147,
+            0.021539296954870224,
+            0.0005015277420170605,
+            -0.015344989486038685,
+            0.009322745725512505,
+            -0.007856289856135845,
+            -0.007805453147739172,
+            0.017847739160060883,
+            0.0125841423869133,
+            -0.009463525377213955,
+            -0.009682515636086464,
+            0.0010314069222658873,
+            0.004872542340308428,
+            0.012943913228809834,
+            0.02066333405673504,
+            -0.003447147784754634,
+            0.004239033907651901,
+            -0.00251448224298656,
+            0.03860492631793022,
+            -0.005748505238443613,
+            0.03125309571623802,
+            -0.015642190352082253,
+            -0.011528294533491135,
+            -0.009205428883433342,
+            -0.0063389977440238,
+            -0.005509961862117052,
+            -0.0024421371053904295,
+            0.0012347553856670856,
+            -0.005482587963342667,
+            -0.023103516548871994,
+            0.012717100791633129,
+            0.022634249180555344,
+            0.002792131155729294,
+            0.007031164597719908,
+            0.005013321992009878,
+            0.013710380531847477,
+            0.030924610793590546,
+            -0.005615546368062496,
+            -0.0147505858913064,
+            0.009002081118524075,
+            0.010034465231001377,
+            -0.016424300149083138,
+            0.021179527044296265,
+            0.006765247322618961,
+            0.006710499990731478,
+            -0.014320425689220428,
+            -0.0017147751059383154,
+            0.02022535353899002,
+            0.0071758548729121685,
+            0.009792011231184006,
+            -0.012294761836528778,
+            -0.0025966037064790726,
+            0.0012679951032623649,
+            0.0072814398445189,
+            -0.0034764769952744246,
+            -0.015008682385087013,
+            0.0159784983843565,
+            -0.014390815049409866,
+            0.02261860854923725,
+            -0.021335948258638382,
+            0.012513753026723862,
+            -0.009252355434000492,
+            0.020694619044661522,
+            -0.0011545892339199781,
+            -0.007727242074906826,
+            -0.00534962909296155,
+            0.013147260993719101,
+            -0.005955764092504978,
+            -0.00256140879355371,
+            -0.02250911295413971,
+            -0.013499210588634014,
+            0.017879024147987366,
+            -0.0216800756752491,
+            -0.0012826595921069384,
+            0.0019005262292921543,
+            0.010714900679886341,
+            -0.028687776997685432,
+            0.022821955382823944,
+            0.031174886971712112,
+            0.012294761836528778,
+            0.012474646791815758,
+            -0.0045166825875639915,
+            -0.016674574464559555,
+            -0.009432241320610046,
+            0.03344300389289856,
+            0.008345108479261398,
+            -0.01801980286836624,
+            0.006952953990548849,
+            0.01830136403441429,
+            -0.009119397029280663,
+            0.004661372862756252,
+            -0.0017685452476143837,
+            0.016064530238509178,
+            0.0020178426057100296,
+            0.0035898827482014894,
+            -0.01673714444041252,
+            -0.0365714430809021,
+            -0.012302583083510399,
+            0.02060076594352722,
+            -0.005666383542120457,
+            -0.008861301466822624,
+            0.009033365175127983,
+            0.0020002450328320265,
+            -0.003146035596728325,
+            -0.011387514881789684,
+            0.003810828784480691,
+            0.0074339513666927814,
+            0.006080901715904474,
+            -0.01967787556350231,
+            0.012021023780107498,
+            -0.009361851029098034,
+            -0.010276919230818748,
+            0.01075400598347187,
+            -0.013858981430530548,
+            -0.026216311380267143,
+            -0.0022172804456204176,
+            -0.015563979744911194,
+            -0.015689117833971977,
+            -0.0144690265879035,
+            0.006816084496676922,
+            -0.0012279120273888111,
+            -0.017347188666462898,
+            -0.028593923896551132,
+            0.014077971689403057,
+            0.0010186976287513971,
+            0.028265438973903656,
+            -0.029078831896185875,
+            0.022649891674518585,
+            0.00035341575858183205,
+            0.012920449487864971,
+            0.0017216185806319118,
+            -0.011848959140479565,
+            0.0019220341928303242,
+            0.005021143238991499,
+            0.003394355298951268,
+            0.001639497117139399,
+            -0.010456804186105728,
+            -0.0010890875710174441,
+            -0.018160583451390266,
+            0.006773068569600582,
+            -0.005236223340034485,
+            -0.014187467284500599,
+            -0.0026415749453008175,
+            0.015431021340191364,
+            0.00606134906411171,
+            0.022290121763944626,
+            -0.019599664956331253,
+            -0.002825370756909251,
+            -0.007817184552550316,
+            -0.012161803431808949,
+            -0.006788710597902536,
+            0.02031920664012432,
+            -0.007222781423479319,
+            0.03363071009516716,
+            0.008923869580030441,
+            -0.01024563517421484,
+            -0.011450083926320076,
+            0.0012562634656205773,
+            0.02066333405673504,
+            0.018504710868000984,
+            -0.02333814837038517,
+            -0.012114876881241798,
+            -0.0013892221031710505,
+            0.013311504386365414,
+            0.006690946873277426,
+            -0.0016717591788619757,
+            0.01736283116042614,
+            0.0066166468895971775,
+            -0.01679971255362034,
+            -0.01254503708332777,
+            -0.002807773184031248,
+            0.015650011599063873,
+            0.013217651285231113,
+            0.013960654847323895,
+            0.005337897688150406,
+            0.013248935341835022,
+            0.01902090385556221,
+            -0.01745668426156044,
+            -0.005369181744754314,
+            0.019255537539720535,
+            0.0029270448721945286,
+            -0.017879024147987366,
+            0.0028195048216730356,
+            0.0023033125326037407,
+            0.00807919166982174,
+            0.003292681183665991,
+            -0.004196017514914274,
+            0.03155029937624931,
+            -0.0009263110114261508,
+            0.005025053862482309,
+            0.018051087856292725,
+            -0.006655752193182707,
+            0.022305764257907867,
+            -0.0017421490047127008,
+            0.028906768187880516,
+            -0.02003764547407627,
+            0.003496029647067189,
+            0.0004572896577883512,
+            -0.0027979968581348658,
+            -0.0026357092428952456,
+            0.0016150561859831214,
+            -0.018317004665732384,
+            0.009275819174945354,
+            -0.04567519575357437,
+            0.008345108479261398,
+            -0.007602104917168617,
+            -0.0021410249173641205,
+            -0.0248867254704237,
+            -0.01015960332006216,
+            -0.009150682017207146,
+            -0.010581942275166512,
+            -0.018786270171403885,
+            -0.021899066865444183,
+            -0.00789930671453476,
+            0.004751315340399742,
+            0.01470365934073925,
+            -0.019896866753697395,
+            0.01939631626009941,
+            0.007070270366966724,
+            -0.026982778683304787,
+            -0.016877923160791397,
+            -0.019959434866905212,
+            -0.0010538926580920815,
+            -0.010065749287605286,
+            0.0028781630098819733,
+            -0.027952594682574272,
+            0.022024204954504967,
+            0.016674574464559555,
+            -0.022258836776018143,
+            0.0058580003678798676,
+            -0.01428914163261652,
+            -0.023776130750775337,
+            0.0017499701352789998,
+            0.0006931445677764714,
+            -0.00864231027662754,
+            0.04273446649312973,
+            0.021101314574480057,
+            8.737629832467064e-05,
+            -0.028640851378440857,
+            0.005728952586650848,
+            -0.02727998048067093,
+            0.013921549543738365,
+            0.004747404716908932,
+            0.00015752174658700824,
+            -0.0045166825875639915,
+            0.016236593946814537,
+            0.0019738490227609873,
+            0.01698741875588894,
+            0.006057438440620899,
+            0.0007400711765512824,
+            0.02473030425608158,
+            -0.011301483027637005,
+            0.006792621221393347,
+            -8.447394066024572e-06,
+            -0.0005313456640578806,
+            0.015696939080953598,
+            -0.004966395441442728,
+            -0.002283759880810976,
+            0.012279119342565536,
+            0.000982525059953332,
+            -0.009721621870994568,
+            -0.004371992312371731,
+            -0.0030424061696976423,
+            0.009135039523243904,
+            -0.015493590384721756,
+            0.01001100242137909,
+            0.0026259326841682196,
+            0.01928682066500187,
+            -0.018254436552524567,
+            -0.005803252570331097,
+            0.002670904155820608,
+            -0.009940612129867077,
+            -0.004606625065207481,
+            -0.015032145194709301,
+            -0.009338388219475746,
+            -0.02020971104502678,
+            0.01617402583360672,
+            0.04320373013615608,
+            0.02252475544810295,
+            0.000653550261631608,
+            -0.004055237863212824,
+            -0.01433606818318367,
+            0.003934010863304138,
+            -0.003478432074189186,
+            -0.011927170678973198,
+            -0.022540396079421043,
+            -0.005189296789467335,
+            0.012036665342748165,
+            -0.004078701138496399,
+            0.0019132354063913226,
+            -0.011653431691229343,
+            0.021101314574480057,
+            -0.01273274328559637,
+            -0.0004919957718811929,
+            -0.018723702058196068,
+            -0.015470126643776894,
+            0.009393135085701942,
+            0.032848600298166275,
+            -0.0023326417431235313,
+            0.013420999981462955,
+            0.013022123835980892,
+            0.004023953340947628,
+            -0.01478969119489193,
+            0.010675795376300812,
+            -0.006010511890053749,
+            0.004106075037270784,
+            -0.0025477218441665173,
+            -0.02380741387605667,
+            -0.009267997927963734,
+            -0.011176344938576221,
+            0.006909937597811222,
+            -0.009768548421561718,
+            -0.0006555055733770132,
+            -0.0015495545230805874,
+            -0.009909328073263168,
+            -0.018160583451390266,
+            0.007586462423205376,
+            -0.007774168625473976,
+            0.0032437993213534355,
+            0.00014090191689319909,
+            -0.0018741299863904715,
+            0.0087205208837986,
+            0.02020971104502678,
+            -0.0067535159178078175,
+            0.0030130769591778517,
+            0.004931200761348009,
+            -0.016846640035510063,
+            0.016393015161156654,
+            0.0019533184822648764,
+            -0.0006476844428107142,
+            0.03237933665513992,
+            0.016236593946814537,
+            0.012795312330126762,
+            -0.0017841873923316598,
+            -0.00305413780733943,
+            -0.001330563798546791,
+            0.014672375284135342,
+            -0.03388098627328873,
+            -0.009111575782299042,
+            0.014985218644142151,
+            0.02067897655069828,
+            -0.015829896554350853,
+            0.0127718485891819,
+            -0.015032145194709301,
+            -0.009690336883068085,
+            -0.007297081872820854,
+            0.00303262984380126,
+            -0.05340243875980377,
+            0.011043386533856392,
+            -0.006366371642798185,
+            0.02712355926632881,
+            0.0006178665207698941,
+            0.0108869643881917,
+            0.010120497085154057,
+            -0.006100454367697239,
+            0.025606267154216766,
+            0.0015045831678435206,
+            -0.0041764648631215096,
+            0.012850060127675533,
+            -0.011778569780290127,
+            -0.011661252938210964,
+            0.008564099669456482,
+            -0.0288442000746727,
+            0.008994259871542454,
+            -0.024933652952313423,
+            0.0022270570043474436,
+            -0.016220951452851295,
+            -0.012396436184644699,
+            0.022649891674518585,
+            0.03287988528609276,
+            0.00202175322920084,
+            0.0022329227067530155,
+            -0.010816575028002262,
+            0.025700120255351067,
+            -0.010550657287240028,
+            0.019443243741989136,
+            -0.008876943029463291,
+            -0.011207629926502705,
+            -0.017112556844949722,
+            -0.010808753781020641,
+            0.008783089928328991,
+            -0.003986803349107504,
+            -0.004375902935862541,
+            -0.021461086347699165,
+            0.005287060514092445,
+            -0.022211911156773567,
+            0.008837837725877762,
+            0.002567274495959282,
+            -0.006518883164972067,
+            0.001965050119906664,
+            0.0029876583721488714,
+            -0.0010167424334213138,
+            -0.00016729811613913625,
+            -0.024088973179459572,
+            -0.008743984624743462,
+            -0.008118296973407269,
+            0.009049007669091225,
+            0.001364781172014773,
+            -0.020538195967674255,
+            0.010581942275166512,
+            -0.009095934219658375,
+            -0.0036153013352304697,
+            -0.008657952770590782,
+            0.0028038627933710814,
+            0.0013872667914256454,
+            0.02067897655069828,
+            0.003855800023302436,
+            -0.020194068551063538,
+            0.009494809433817863,
+            0.017394116148352623,
+            -0.00253990083001554,
+            0.006331176497042179,
+            -0.00019063919899053872,
+            -0.01028474047780037,
+            -0.010589763522148132,
+            -0.0012865702155977488,
+            0.006299892440438271,
+            -0.009831116534769535,
+            0.005744594614952803,
+            0.006992059294134378,
+            0.0058345370925962925,
+            0.021070031449198723,
+            0.008900406770408154,
+            0.004242944065481424,
+            -0.0015603085048496723,
+            0.006733963266015053,
+            -0.0011614327086135745,
+            -0.0014928515302017331,
+            -0.012748385779559612,
+            0.009862401522696018,
+            -0.019630949944257736,
+            0.009651231579482555,
+            0.017675675451755524,
+            0.008986438624560833,
+            -0.016971776261925697,
+            -0.01755053736269474,
+            -0.007363561075180769,
+            -0.023478928953409195,
+            0.009095934219658375,
+            0.0007459369953721762,
+            0.018535995855927467,
+            -0.0046535516157746315,
+            0.008196507580578327,
+            0.006147380918264389,
+            -0.013804233632981777,
+            -0.0020549928303807974,
+            0.009502630680799484,
+            -0.029657592996954918,
+            -0.012521573342382908,
+            -0.000881339656189084,
+            -0.0010548701975494623,
+            0.018176225945353508,
+            0.01919296756386757,
+            0.019146041944622993,
+            0.03206649050116539,
+            -0.009705979377031326,
+            0.01023781392723322,
+            -0.0028371023945510387,
+            0.018864482641220093,
+            0.010042286477982998,
+            0.00733227701857686,
+            0.020632049068808556,
+            0.006972506642341614,
+            -0.02103874646127224,
+            -0.010832217521965504,
+            0.019818656146526337,
+            -0.010417698882520199,
+            -0.005959674715995789,
+            -0.012552858330309391,
+            0.0037756338715553284,
+            0.0033904449082911015,
+            0.011958454735577106,
+            0.018708059564232826,
+            0.008884764276444912,
+            -0.017315905541181564,
+            0.032754749059677124,
+            0.0007058538612909615,
+            0.00647977739572525,
+            0.004199928138405085,
+            -0.003887084312736988,
+            -0.005502140615135431,
+            0.013968476094305515,
+            0.012466826476156712,
+            -0.0014938291860744357,
+            0.0034745216835290194,
+            0.012623247690498829,
+            -0.01447684783488512,
+            -0.018598563969135284,
+            0.007527804467827082,
+            0.0017558359540998936,
+            0.00037883431650698185,
+            0.006589272990822792,
+            0.0025496771559119225,
+            0.017613107338547707,
+            -0.006831726990640163,
+            0.028187228366732597,
+            -0.005521693266928196,
+            -0.016142740845680237,
+            -0.013139440678060055,
+            0.0011555668897926807,
+            -0.011903706938028336,
+            -0.018144940957427025,
+            0.007504341192543507,
+            -0.0015104489866644144,
+            0.011934991925954819,
+            -0.002289625583216548,
+            0.004250765312463045,
+            0.013381893746554852,
+            0.017706960439682007,
+            0.014492489397525787,
+            0.002645485568791628,
+            -0.004508861340582371,
+            -0.015438842587172985,
+            0.008032265119254589,
+            -0.009698158130049706,
+            0.02132030576467514,
+            -0.014085792936384678,
+            -0.021977277472615242,
+            0.0031362592708319426,
+            0.01474276464432478,
+            -0.0020589034538716078,
+            0.0001868508552433923,
+            0.012748385779559612,
+            -0.0062451446428895,
+            0.00031675436184741557,
+            0.012568499892950058,
+            0.00445802416652441,
+            -0.02810901589691639,
+            -0.009612126275897026,
+            0.007273618597537279,
+            -0.0021625328809022903,
+            -0.003701333422213793,
+            -0.02388562448322773,
+            -0.004430650733411312,
+            -0.011324945837259293,
+            -0.005944032687693834,
+            -0.004371992312371731,
+            -8.847613935358822e-05,
+            0.009361851029098034,
+            -0.02269681915640831,
+            0.014461205340921879,
+            -0.013327146880328655,
+            -0.004555787891149521,
+            -0.017660032957792282,
+            0.022290121763944626,
+            0.00530661316588521,
+            0.0007527804118581116,
+            0.00507589103654027,
+            0.0012396436650305986,
+            0.008665774017572403,
+            0.01452377438545227,
+            -0.002537945518270135,
+            0.004778689239174128,
+            0.015665654093027115,
+            0.005345718469470739,
+            0.005760236643254757,
+            0.01938067376613617,
+            -0.003339607734233141,
+            -0.017769528552889824,
+            -0.003564464161172509,
+            0.016424300149083138,
+            0.011231092736124992,
+            0.015493590384721756,
+            0.004645730834454298,
+            0.01033166702836752,
+            -0.003564464161172509,
+            -0.006909937597811222,
+            0.018129298463463783,
+            0.0125841423869133,
+            -0.016205308958888054,
+            -0.011332767084240913,
+            0.012294761836528778,
+            0.01033166702836752,
+            0.011786391027271748,
+            0.003580106422305107,
+            -0.0027060990687459707,
+            0.014578522183001041,
+            0.017018703743815422,
+            0.006647930946201086,
+            0.020162783563137054,
+            -0.001665893360041082,
+            -0.03572676330804825,
+            -0.004684836138039827,
+            0.0011809853604063392,
+            -0.00016595386841800064,
+            0.009526094421744347,
+            0.00445802416652441,
+            0.008290360681712627,
+            0.005795431789010763,
+            0.01958402246236801,
+            -0.006628378294408321,
+            0.010589763522148132,
+            0.00020762563508469611,
+            -0.02056948095560074,
+            -0.0020862771198153496,
+            -0.0040708803571760654,
+            -0.032191626727581024,
+            -0.00019369431538507342,
+            0.006080901715904474,
+            0.023087874054908752,
+            -0.009479167871177197,
+            0.0029055369086563587,
+            -0.006057438440620899,
+            -0.013162903487682343,
+            -0.00220554880797863,
+            0.014101434499025345,
+            -0.0012249790597707033,
+            0.01037859357893467,
+            0.0030678247567266226,
+            0.0005034829955548048,
+            0.0036387646105140448,
+            0.01966223306953907,
+            -0.00021385807485785335,
+            0.00868923682719469,
+            0.016877923160791397,
+            0.020256636664271355,
+            8.627645729575306e-05,
+            -0.01222437247633934,
+            0.027983879670500755,
+            -0.01428914163261652,
+            -0.008697058074176311,
+            -0.005603814963251352,
+            -0.005290971137583256,
+            0.015305883251130581,
+            -0.008478067815303802,
+            0.010589763522148132,
+            0.006628378294408321,
+            -0.018035445362329483,
+            0.01102774403989315,
+            -0.011176344938576221,
+            0.02361970767378807,
+            0.01689356565475464,
+            -0.0017186857294291258,
+            0.004282049834728241,
+            0.007293171249330044,
+            -0.03466309607028961,
+            0.009792011231184006,
+            -0.010957354679703712,
+            -0.0010734453098848462,
+            -0.0030111216474324465,
+            -0.004939021542668343,
+            0.016768427565693855,
+            -0.00014383482630364597,
+            -0.022086773067712784,
+            -0.00312257232144475,
+            0.021429801359772682,
+            -0.0013638035161420703,
+            0.01010485552251339,
+            0.006448493339121342,
+            -0.0020256636198610067,
+            -0.01618966832756996,
+            -0.020788472145795822,
+            -0.018129298463463783,
+            -0.020240994170308113,
+            -0.014367352239787579,
+            -0.00849370937794447,
+            -0.004794331267476082,
+            0.0045948936603963375,
+            0.0026572172064334154,
+            -0.008923869580030441,
+            -0.020929250866174698,
+            -0.014320425689220428,
+            0.003934010863304138,
+            0.0020354401785880327,
+            0.007723331451416016,
+            -0.0071993181481957436,
+            -0.007954053580760956,
+            -0.002113651018589735,
+            -0.0005504096043296158,
+            -0.006972506642341614,
+            0.01295173354446888,
+            -0.02463645115494728,
+            -0.005658562295138836,
+            0.01947452686727047,
+            0.011434441432356834,
+            -0.004536235239356756,
+            0.010316024534404278,
+            -0.026263238862156868,
+            0.02286888286471367,
+            -0.018410857766866684,
+            -0.015470126643776894,
+            -0.0079423226416111,
+            0.013632168993353844,
+            -0.015313704498112202,
+            0.005427840165793896,
+            0.006769157946109772,
+            -0.004395455587655306,
+            0.016408657655119896,
+            0.009181966073811054,
+            -0.011066850274801254,
+            0.009987538680434227,
+            -0.009416598826646805,
+            0.0015475992113351822,
+            -0.01645558513700962,
+            0.0009624835220165551,
+            0.015423200093209743,
+            0.010495910421013832,
+            -0.005267507862299681,
+            -0.010988638736307621,
+            -0.005705488845705986,
+            0.00047635359806008637,
+            0.0004220947448629886,
+            0.007500430569052696,
+            0.004008311312645674,
+            -0.002915313234552741,
+            -0.016064530238509178,
+            -0.017706960439682007,
+            0.0262006688863039,
+            -0.0051541016437113285,
+            -0.009103755466639996,
+            0.02047562785446644,
+            0.02150801196694374,
+            0.01615838333964348,
+            -0.017534896731376648,
+            -0.0005655629793182015,
+            0.009612126275897026,
+            0.014031045138835907,
+            0.007351829670369625,
+            -0.003501895349472761,
+            -0.00719149736687541,
+            -0.021523654460906982,
+            0.030095575377345085,
+            0.007261887192726135,
+            0.0075942836701869965,
+            -0.031284380704164505,
+            0.023385075852274895,
+            0.00021129178639966995,
+            -0.010409877635538578,
+            0.0026787251699715853,
+            0.01966223306953907,
+            0.005388734862208366,
+            0.0008539658738300204,
+            0.02297837845981121,
+            0.015086892992258072,
+            0.004931200761348009,
+            -0.007680315524339676,
+            -0.013217651285231113,
+            0.004551877733319998,
+            -0.005392645485699177,
+            0.009557378478348255,
+            -0.012028845027089119,
+            -0.009033365175127983,
+            0.0037658573128283024,
+            -0.003056093119084835,
+            0.008212150074541569,
+            -0.0002551632351242006,
+            -0.0211169570684433,
+            -0.006487598642706871,
+            -0.0363837368786335,
+            0.0050172326155006886,
+            -0.024464385583996773,
+            -0.005693757440894842,
+            -0.014234393835067749,
+            -0.013069050386548042,
+            -0.03932446613907814,
+            -0.010667974129319191,
+            0.008282540366053581,
+            0.015548337250947952,
+            0.0036446305457502604,
+            -0.011543937027454376,
+            0.0037287073209881783,
+            -0.040763549506664276,
+            0.0003023342287633568,
+            0.02848443016409874,
+            0.014492489397525787,
+            0.023494569584727287,
+            -0.015321525745093822,
+            0.022665534168481827,
+            0.0005333009175956249,
+            -0.02848443016409874,
+            -0.0029896136838942766,
+            -0.0006017355481162667,
+            0.011950633488595486,
+            0.00093902024673298,
+            -0.009721621870994568,
+            -0.017675675451755524,
+            0.020741544663906097,
+            0.006264697294682264,
+            0.009502630680799484,
+            -0.005185386165976524,
+            0.001369669334962964,
+            0.00380300753749907,
+            0.005204938817769289,
+            -0.014953934587538242,
+            0.0005670294049195945,
+            0.010581942275166512,
+            0.016095813363790512,
+            -0.006808263715356588,
+            -0.01263889018446207,
+            -0.008955154567956924,
+            -0.008196507580578327,
+            -0.00213711429387331,
+            -0.012576321139931679,
+            -0.010370772331953049,
+            -0.011606505140662193,
+            -0.0028957605827599764,
+            -0.01469583809375763,
+            -0.01618966832756996,
+            0.008188687264919281,
+            -0.0009399979026056826,
+            0.02316608466207981,
+            0.013984118588268757,
+            0.018676776438951492,
+            0.008141760714352131,
+            0.01681535504758358,
+            0.005142370238900185,
+            0.004305513110011816,
+            -0.014343888498842716,
+            -0.002422584220767021,
+            -0.030533555895090103,
+            0.013647811487317085,
+            0.02984529919922352,
+            -0.007003790698945522,
+            -0.00305413780733943,
+            0.010050107724964619,
+            0.0022466096561402082,
+            -0.002281804569065571,
+            0.0019415869610384107,
+            -0.01645558513700962,
+            -0.005236223340034485,
+            -0.01010485552251339,
+            0.006483688019216061,
+            0.009346208535134792,
+            0.028906768187880516,
+            -0.005552977789193392,
+            -0.0019083472434431314,
+            0.02241525985300541,
+            0.014359530992805958,
+            0.02537163347005844,
+            0.009471346624195576,
+            -0.006604915019124746,
+            0.00042062829015776515,
+            -0.0065384358167648315,
+            -0.009095934219658375,
+            -0.026122458279132843,
+            0.031034106388688087,
+            -0.004477577283978462,
+            -0.00877526868134737,
+            -0.009267997927963734,
+            -0.011590863578021526,
+            0.0025105716194957495,
+            0.02020971104502678,
+            -0.015947213396430016,
+            0.0011233048280701041,
+            0.0001070512444130145,
+            -0.016064530238509178,
+            -0.02712355926632881,
+            0.03189442679286003,
+            0.005979227367788553,
+            -0.002801907481625676,
+            -3.3239655749639496e-05,
+            -0.007426130119711161,
+            -0.011387514881789684,
+            -0.009346208535134792,
+            -0.0056976680643856525,
+            0.06563463062047958,
+            -0.01466455403715372,
+            0.002569229807704687,
+            -0.02305658906698227,
+            0.0031499462202191353,
+            -0.007766347844153643,
+            -0.00024954183027148247,
+            0.0075395358726382256,
+            -0.011004281230270863,
+            -0.0030736904591321945,
+            -0.01310033444315195,
+            -0.002369791967794299,
+            0.004160822834819555,
+            -0.019771728664636612,
+            -0.009236713871359825,
+            -0.0034745216835290194,
+            -0.011254556477069855,
+            -0.005283149890601635,
+            0.012153982184827328,
+            0.011309304274618626,
+            0.013890265487134457,
+            -0.02537163347005844,
+            -0.003361115697771311,
+            0.004852989688515663,
+            -0.008900406770408154,
+            -0.004012221936136484,
+            -0.017769528552889824,
+            -0.010065749287605286,
+            -0.009158502332866192,
+            -0.011066850274801254,
+            0.014774048700928688,
+            -0.009526094421744347,
+            0.012200908735394478,
+            0.006956864148378372,
+            0.0065970937721431255,
+            0.01958402246236801,
+            0.014492489397525787,
+            0.011692537926137447,
+            0.013037766329944134,
+            0.005658562295138836,
+            -0.01442992128431797,
+            0.019709160551428795,
+            -0.02213370054960251,
+            -0.022383974865078926,
+            -0.010018822737038136,
+            0.0033063681330531836,
+            -0.009893685579299927,
+            -0.007926680147647858,
+            -0.011332767084240913,
+            0.015650011599063873,
+            0.0026161563582718372,
+            -0.007649031467735767,
+            -0.012388614937663078,
+            -0.0024890636559575796,
+            0.0291257593780756,
+            -0.007649031467735767,
+            0.02039741724729538,
+            -0.012153982184827328,
+            0.009533915668725967,
+            -0.015086892992258072,
+            -0.00813393946737051,
+            -0.009713800624012947,
+            -0.008282540366053581,
+            0.014907008036971092,
+            -0.006319445092231035,
+            0.005298791918903589,
+            -0.017394116148352623,
+            -0.02692021057009697,
+            -0.0003842113073915243,
+            0.00380300753749907,
+            -0.028140300884842873,
+            0.007578641641885042,
+            -0.02316608466207981,
+            0.004802152514457703,
+            -0.011692537926137447,
+            -0.014672375284135342,
+            -0.01654943823814392,
+            0.024057690054178238,
+            0.002991568995639682,
+            -0.004188196733593941,
+            0.00126995041500777,
+            -0.00521667068824172,
+            -0.006354639772325754,
+            0.009330566972494125,
+            0.006964685395359993,
+            0.0049585746601223946,
+            0.015000861138105392,
+            -0.006980327423661947,
+            -0.011450083926320076,
+            -0.00256140879355371,
+            -0.0035879274364560843,
+            -0.017597464844584465,
+            0.0015700849471613765,
+            0.002289625583216548,
+            -0.006741784047335386,
+            0.005201028194278479,
+            0.013530494645237923,
+            -0.0049703060649335384,
+            0.006530614569783211,
+            0.018692417070269585,
+            0.0022270570043474436,
+            -0.005728952586650848,
+            -0.0055177826434373856,
+            -0.003934010863304138,
+            -0.002438226481899619,
+            0.014062329195439816,
+            0.010464625433087349,
+            0.015266777947545052,
+            -0.007273618597537279,
+            0.009385314770042896,
+            -0.00436417106539011,
+            0.001194672309793532,
+            -0.028140300884842873,
+            0.00799315981566906,
+            0.005052427761256695,
+            -0.0028351470828056335,
+            0.00530661316588521,
+            0.0017069540917873383,
+            0.04270318150520325,
+            0.008814374916255474,
+            0.013327146880328655,
+            -0.003323965473100543,
+            0.009674695320427418,
+            0.0027354280464351177,
+            0.012615426443517208,
+            0.0006882564048282802,
+            -0.02105438895523548,
+            0.011567399837076664,
+            -0.015047787688672543,
+            0.0027021884452551603,
+            0.017034346237778664,
+            -0.019177325069904327,
+            0.015438842587172985,
+            -0.01025345642119646,
+            -0.0001107173811760731,
+            -0.004002445377409458,
+            -0.022743744775652885,
+            -0.01001100242137909,
+            -0.00019137242634315044,
+            0.023932551965117455,
+            -0.0017978743417188525,
+            0.025747045874595642,
+            0.006417208816856146,
+            0.02491801045835018,
+            -0.00804008636623621,
+            -0.017253335565328598,
+            -0.005607725586742163,
+            0.005103264935314655,
+            -0.003548822132870555,
+            0.0057524158619344234,
+            0.008282540366053581,
+            0.0060183326713740826,
+            -0.001099841552786529,
+            -0.0007962852832861245,
+            -0.000347061111824587,
+            0.022900167852640152,
+            0.009291461668908596,
+            0.01056629978120327,
+            -0.01037859357893467,
+            0.013084692880511284,
+            -0.018051087856292725,
+            0.00021336926147341728,
+            0.0010089213028550148,
+            -0.022649891674518585,
+            0.0014077972155064344,
+            -0.004852989688515663,
+            -0.004371992312371731,
+            0.0026669935323297977,
+            0.019505811855196953,
+            -0.019427601248025894,
+            -0.014555058442056179,
+            0.02590346708893776,
+            0.007426130119711161,
+            0.0025555428583174944,
+            -0.002262251917272806,
+            0.009831116534769535,
+            0.00873616337776184,
+            0.011614326387643814,
+            -0.004720031283795834,
+            -0.005846268963068724,
+            -0.017753886058926582,
+            0.014164003543555737,
+            -0.006534525193274021,
+            -0.010308203287422657,
+            -0.008923869580030441,
+            -0.008345108479261398,
+            0.01617402583360672,
+            0.0007835759897716343,
+            0.007668584119528532,
+            0.002807773184031248,
+            0.02637273445725441,
+            -0.01281877513974905,
+            -0.011332767084240913,
+            -0.009408777579665184,
+            0.008556278422474861,
+            0.024464385583996773,
+            0.007957964204251766,
+            0.013202008791267872,
+            -0.01618966832756996,
+            -0.0036993781104683876,
+            0.019271178171038628,
+            -0.011872422881424427,
+            -0.033224012702703476,
+            -0.007500430569052696
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "bc2283ff-904c-4516-9ad1-f43bcecf7394",
+      "type": "child",
+      "source": {
+        "id": "1e687808-8db8-4320-805d-0282a8d679bc",
+        "properties": {
+          "page_content": "Deepseekmoe: Towards Ultimate Expert Specialization In\n\nMixture-Of-Experts Language Models\n\nDamai Dai∗1,2, Chengqi Deng1, Chenggang Zhao∗1,3, R.X. Xu1, Huazuo Gao1, Deli Chen1, Jiashi Li1, Wangding Zeng1, Xingkai Yu∗1,4, Y. Wu1, Zhenda Xie1, Y.K. Li1, Panpan Huang1, Fuli Luo1, Chong Ruan1, Zhifang Sui2, Wenfeng Liang1 1DeepSeek-AI 2National Key Laboratory for Multimedia Information Processing, Peking University 3Institute for Interdisciplinary Information Sciences, Tsinghua University 4National Key Laboratory for Novel Software Technology, Nanjing University {daidamai, szf}@pku.edu.cn, {wenfeng.liang}@deepseek.com https://github.com/deepseek-ai/DeepSeek-MoE\n\nAbstract\n\nIn the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top- out of experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into ones and activating from them, allowing for a more flexible combination of activated experts; (2) isolating experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5× expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations.\n\nFurther, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\n1. Introduction\n\nRecent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023a). It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017) has emerged as a popular solution. It can\n\nenable parameter scaling, while concurrently keeping computational costs at a modest level.\n\nRecent applications of MoE architectures in Transformers (Vaswani et al., 2017) have yielded successful attempts at scaling language models to a substantial size (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022), accompanied with remarkable performance. These achievements underscore the considerable potential and promise of MoE language models.\n\nDespite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts.\n\nThis architecture manifests two potential issues: (1) Knowledge Hybridity: existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. (2) Knowledge Redundancy: tokens assigned to different experts may require common knowledge. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters.\n\nThese issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models.\n\nIn response to the aforementioned issues, we introduce DeepSeekMoE, an innovative MoE architecture specifically designed towards ultimate expert specialization. Our architecture involves two principal strategies: (1) Fine-Grained Expert Segmentation: while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. (2) Shared Expert Isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.\n\nThrough compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized.\n\nStarting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B (Lepikhin et al., 2021) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5× expert parameters and computation. Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE.\n\nThese studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization.\n\nLeveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B (DeepSeek-AI, 2024), a dense model trained on the same 2T corpus. We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B (Touvron et al., 2023b), which has approximately 2.5 times the activated parameters. Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard1. Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. The experimental results still validate its substantial advantages over the GShard architecture consistently. In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nOur contributions are summarized as follows: - Architectural Innovation. We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation.\n\nEmpirical Validation. We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. Experimental results validate the high 1https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models\n\nScalability. We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B.\n\nAlignment for MoE. We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B.\n\nPublic Release. In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization.\n\n2. Preliminaries: Mixture-Of-Experts For Transformers\n\nWe first introduce a generic MoE architecture commonly used in Transformer language models. A standard Transformer language model is constructed by stacking layers of standard Transformer blocks, where each block can be represented as follows:\n\n$$\\mathbf{u}{1:T}^{l}=\\text{Self-Att}\\left(\\mathbf{h}{1:T}^{l-1}\\right)+\\mathbf{h}{1:T}^{l-1},\\tag{1}$$ $$\\mathbf{h}{t}^{l}=\\text{FFN}\\left(\\mathbf{u}{t}^{l}\\right)+\\mathbf{u}{t}^{l},\\tag{2}$$\n\nwhere denotes the sequence length, Self-Att(·) denotes the self-attention module, FFN(·) denotes the Feed-Forward Network (FFN), u 1: ∈ R× are the hidden states of all tokens after the -th attention module, and h\n\n∈ Ris the output hidden state of the -th token after the -th Transformer block. For brevity, we omit the layer normalization in the above formulations.\n\nA typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022). An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. Then, each token will be assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts. If the -th FFN is substituted with an MoE layer, the computation for its output hidden state h\n\nis expressed as: where denotes the total number of experts, FFN(·) is the -th expert FFN, , denotes the gate value for the -th expert, , denotes the token-to-expert affinity, Topk(·, ) denotes the set comprising highest affinity scores among those calculated for the -th token and all experts, and e\n\nis the centroid of the -th expert in the -th layer. Note that ,is sparse, indicating that only out of gate values are nonzero. This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only experts. Also, in the above formulations, we omit the layer normalization operation for brevity.\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{N}\\left(g_{i,t}\\operatorname{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\operatorname{Topk}({s_{j,t}|1\\leqslant j\\leqslant N},K),\\ 0,&\\text{otherwise},\\end{cases}$$ $$s_{i,t}=\\operatorname{Softmax}{i}\\left(\\mathbf{u}{t}^{l}\\,\\mathbf{e}_{i}^{l}\\right),$$\n\n$$(3)$$\n\n(4) $\\binom{4}{5}$ .\n\n3. Deepseekmoe Architecture\n\nOn top of the generic MoE architecture outlined in Section 2, we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. As illustrated in Figure 2, our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. Both of these strategies are designed to elevate the level of expert specialization.\n\n3.1. Fine-Grained Expert Segmentation\n\nIn scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts.\n\nIn pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. The finer expert segmentation enables a more flexible and adaptable combination of activated experts. To be specific, on top of a typical MoE architecture shown in Figure 2(a), we segment each expert FFN into smaller experts by reducing the FFN intermediate hidden dimension to 1 times its original size. Since each expert becomes smaller, in response, we also increase the number of activated experts to times to keep the same computation cost, as illustrated in Figure 2(b). With the fine-grained expert segmentation, the output of an MoE layer can be expressed as:\n\n$$\\mathbf{h}{t}^{i}=\\sum{i=1}^{\\text{mN}}\\left(g_{i,t}\\text{FFN}{i}\\left(\\mathbf{u}{t}^{i}\\right)\\right)+\\mathbf{u}{t}^{i},\\tag{6}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\text{Tokr}(s_{i,t}|1\\leqslant j\\leqslant m\\text{N}),m\\text{K}),\\ 0,&\\text{otherwise},\\end{cases}$$ (7) $$s_{i,t}=\\text{Softmax}{i}\\left(\\mathbf{u}{t}^{T}\\mathbf{e}_{t}^{i}\\right),\\tag{8}$$ where the total number of expert parameters is equal to $N$ times the number of parameters in a\n\n$$(6)$$\n\n(7) (8) $\\frac{1}{2}$\n\nstandard FFN, and denotes the total number of fine-grained experts. With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to .\n\nFrom a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. As an illustrative example, we consider the case where = 16. A typical top-2 routing strategy can yield 16 2 = 120 possible combinations. By contrast, if each expert is split into 4 smaller experts, the fine-grained routing strategy can yield 64 8 = 4, 426, 165, 368 potential combinations. The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition.\n\n3.2. Shared Expert Isolation\n\nWith a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. This alleviation of redundancy will contribute to a more parameterefficient model with more specialized experts.\n\nTowards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate experts to serve as shared experts. Regardless of the router module, each token will be deterministically assigned to these shared experts. In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by , as depicted in Figure 2(c). With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows:\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{K_{s}}\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)+\\sum_{i=K_{s}+1}^{mN}\\left(g_{i,t}\\;\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},\\tag{9}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\mathrm{Topk}{s_{j,t}|K_{s}+1\\leqslant j\\leqslant mN},mK-K_{s}},\\ 0,&\\text{otherwise},\\end{cases}$$ (10) $$s_{i,t}=\\mathrm{Softmax}{i}\\left(\\mathbf{u}{t}^{l,T}e_{i}^{l}\\right).\\tag{11}$$ $\\mathbf{h}\\mathbf{M}\\mathbf{E}$ the number of closed-compact is $K_{s}$ the total number of routed currents.\n\nFinally, in DeepSeekMoE, the number of shared expert is , the total number of routed experts is − , and the number of nonzero gates is − .\n\nIt is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022). The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint.\n\n3.3. Load Balance Consideration\n\nAutomatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. Firstly, there is a risk of routing collapse (Shazeer et al., 2017), i.e., the model always selects only a few experts, preventing other experts from sufficient training.\n\nSecondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.\n\nExpert-Level Balance Loss. In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. The computation of the balance loss is as follows:\n\n$$\\begin{array}{l}{{{\\mathcal L}{\\mathrm{ExpBal}}=\\alpha{1}\\sum_{i=1}^{N^{\\prime}}f_{i}P_{i},}}\\ {{f_{i}=\\frac{N^{\\prime}}{K^{T}}\\sum_{t=1}^{T}\\mathds{1}\\,(\\mathrm{Token}\\;t\\;\\mathrm{s}\\mathrm{lects}\\;\\mathrm{Expert}\\;i),}}\\ {{P_{i}=\\frac{1}{T}\\sum_{t=1}^{T}s_{i,t},}}\\end{array}$$\n\n(12) $$\\begin{array}{l}\\small\\text{(13)}\\end{array}$$ = (14) $$\\begin{array}{l}\\small\\text{(14)}\\end{array}$$ .\n\n7 where 1 is a hyper-parameter called expert-level balance factor, ′is equal to ( − ) and ′ is equal to ( − ) for brevity. 1(·) denotes the indicator function.\n\nDevice-Level Balance Loss. In addition to the expert-level balance loss, we introduce a devicelevel balance loss. When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. Instead, our primary objective is to ensure balanced computation across the devices. If we partition all routed experts into groups {E1, E2, ..., E}, and deploy each group on a single device, the device-level balance loss is computed as follows:\n\n$$\\mathcal{L}{\\text{DevBal}}=\\alpha{2}\\sum_{i=1}^{D}f_{i}^{\\prime}P_{i}^{\\prime},$$ $$f_{i}^{\\prime}=\\frac{1}{|\\mathcal{E}{i}|}\\sum{j\\in\\mathcal{E}{i}}f{j},$$ $$P_{i}^{\\prime}=\\sum_{j\\in\\mathcal{E}{i}}P{j},$$\n\n(15) $$\\begin{array}{l}\\small\\text{(16)}\\end{array}$$ = (17) .\n\nwhere 2 is a hyper-parameter called device-level balance factor. In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices.\n\n4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n6. Alignment For Deepseekmoe 16B\n\nPrevious research indicates that MoE models typically do not emerge significant gains from fine-tuning (Artetxe et al., 2022; Fedus et al., 2021). However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B.\n\n6.1. Experimental Setup\n\nTraining Data. For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios.\n\nHyper-Parameters. During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).\n\nWe employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10−5 without incorporating any learning rate scheduling strategy.\n\nEvaluation Benchmarks. For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3, with the following adjustments: (1) We exclude Pile (Gao et al., 2020) since chat models are seldom employed for pure language modeling. (2) We exclude\n\nMetric # Shot LLaMA2 DeepSeek DeepSeekMoE SFT 7B Chat 7B Chat 16B # Total Params N/A 6.7B 6.9B 16.4B # Activated Params N/A 6.7B 6.9B 2.8B FLOPs per 4K Tokens N/A 187.9T 183.5T 74.4T HellaSwag (Acc.) 0-shot 67.9 71.0 72.2 PIQA (Acc.) 0-shot 76.9 78.4 79.7 ARC-easy (Acc.) 0-shot 69.7 70.2 69.9 ARC-challenge (Acc.) 0-shot 50.8 50.2 50.0 BBH (EM) 3-shot 39.3 43.1 42.2 RACE-middle (Acc.) 5-shot 63.9 66.1 64.8 RACE-high (Acc.) 5-shot 49.6 50.8 50.6 DROP (EM) 1-shot 40.0 41.7 33.8 GSM8K (EM) 0-shot 63.4 62.6 62.2 MATH (EM) 4-shot 13.5 14.7 15.2 HumanEval (Pass@1) 0-shot 35.4 45.1 45.7 MBPP (Pass@1) 3-shot 27.8 39.0 46.2 TriviaQA (EM) 5-shot 60.1 59.5 63.3 NaturalQuestions (EM) 0-shot 35.2 32.7 35.1 MMLU (Acc.) 0-shot 50.0 49.7 47.2 WinoGrande (Acc.) 0-shot 65.1 68.4 69.0 CLUEWSC (EM) 5-shot 48.4 66.2 68.2 CEval (Acc.) 0-shot 35.1 44.7 40.0 CMMLU (Acc.) 0-shot 36.9 51.2 49.3\n\nCHID (Zheng et al., 2019) due to the observed instability of results, hindering the derivation of solid conclusions. (3) We additionally include BBH (Suzgun et al., 2022) to provide a more comprehensive assessment of the reasoning ability of the chat models.\n\nTable 5 | Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations.\n\n6.2. Evaluations\n\nBaselines. In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. Correspondingly, we construct three chat models, including LLaMA2 SFT 7B3, DeepSeek Chat 7B, and DeepSeekMoE Chat 16B.\n\nSubsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks.\n\nResults. The evaluation results are presented in Table 5. Our key observations include: (1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intensive tasks (TriviaQA, NaturalQuestions). (2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. In addition, it also surpasses DeepSeek Chat 7B. (3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1). However, it is worth noting that, after supervised finetuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. (4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations.\n\n7. Deepseekmoe 145B Ongoing\n\nEncouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available.\n\n7.1. Experimental Setup\n\nTraining Data and Tokenization. For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study.\n\nModel Settings. For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps.\n\nSubsequently, the learning rate keeps constant during the remaining training process. The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. Also, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse.\n\nEvaluation Benchmarks. We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3).\n\n7.2. Evaluations\n\nBaselines. Apart from DeepSeekMoE 145B, we consider three additional models for comparison. DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters.\n\nResults. From the evaluation results presented in Table 6, we have the following observations: (1) Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. (2) On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. (3) At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5.\n\n8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\nAppendices A. Overview Of Hyper-Parameters\n\nWe present the overview of hyper-parameters for DeepSeekMoE across various sizes in Table 7.\n\n# Params # Layers Hidden # Attn # Shared # Routed Relative Sequence Batch Size Learning Size Heads Experts Experts Expert Size Length (Sequence) Rate 2.0B 9 1280 10 1 63 (7 activated) 0.25 2048 2048 1.08e-3 16.4B 28 2048 16 2 64 (6 activated) 0.25 4096 4608 4.2e-4 144.6B 62 4096 32 4 128 (12 activated) 0.125 4096 4608 3.0e-4\n\nTable 7 | Overview of hyper-parameters for DeepSeekMoE across various sizes. The relative expert size is in comparison to a standard FFN.\n\nB. Comparing Deepseekmoe With Larger Models\n\nComparisons among DeepSeekMoE, GShard×1.2, and GShard×1.5 are shown in Table 8. Comparisons among DeepSeekMoE, Dense×4, and Dense×16 are shown in Table 9.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 2.3B 2.8B 1.9B # Activated Expert Params N/A 0.28B 0.35B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.824 1.808 1.808 HellaSwag (Acc.) 0-shot 53.7 54.4 54.8 PIQA (Acc.) 0-shot 71.8 71.1 72.3 ARC-easy (Acc.) 0-shot 46.8 47.3 49.4 ARC-challenge (Acc.) 0-shot 31.7 34.1 34.3 RACE-middle (Acc.) 5-shot 43.7 46.4 44.0 RACE-high (Acc.) 5-shot 31.9 32.4 31.7 HumanEval (Pass@1) 0-shot 3.7 3.0 4.9 MBPP (Pass@1) 3-shot 2.4 2.6 2.2 TriviaQA (EM) 5-shot 15.2 15.7 16.6 NaturalQuestions (EM) 5-shot 4.5 4.7 5.7\n\nTable 8 | Comparison between DeepSeekMoE and larger GShard models.\n\nAt a larger scale of 13B total parameters, we also compare DeepSeekMoE with GShard×1.2 and GShard×1.5, and show results in Table 10. At a larger scale, DeepSeekMoE even outperforms GShard×1.5 distinctly.\n\nMetric # Shot Dense×4 Dense×16 DeepSeekMoE Relative Expert Size N/A 1 1 0.25 # Experts N/A 4 + 0 16 + 0 1 + 63 # Activated Experts N/A 4 + 0 16 + 0 1 + 7 # Total Expert Params N/A 0.47B 1.89B 1.89B # Activated Expert Params N/A 0.47B 1.89B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.908 1.806 1.808 HellaSwag (Acc.) 0-shot 47.6 55.1 54.8 PIQA (Acc.) 0-shot 70.0 71.9 72.3 ARC-easy (Acc.) 0-shot 43.9 51.9 49.4 ARC-challenge (Acc.) 0-shot 30.5 33.8 34.3 RACE-middle (Acc.) 5-shot 42.4 46.3 44.0 RACE-high (Acc.) 5-shot 30.7 33.0 31.7 HumanEval (Pass@1) 0-shot 1.8 4.3 4.9 MBPP (Pass@1) 3-shot 0.2 2.2 2.2 TriviaQA (EM) 5-shot 9.9 16.5 16.6 NaturalQuestions (EM) 5-shot 3.0 6.3 5.7\n\nTable 9 | Comparison between DeepSeekMoE and larger dense baselines.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 15.9B 19.8B 13.3B # Activated Expert Params N/A 2.37B 2.82B 2.05B # Training Tokens N/A 100B 100B 100B HellaSwag (Acc.) 0-shot 66.6 67.7 69.1 PIQA (Acc.) 0-shot 75.6 76.0 75.7 ARC-easy (Acc.) 0-shot 56.8 56.8 58.8 ARC-challenge (Acc.) 0-shot 39.9 37.6 38.5 RACE-middle (Acc.) 5-shot 51.6 50.6 52.4 RACE-high (Acc.) 5-shot 37.4 36.3 38.5 HumanEval (Pass@1) 0-shot 6.1 6.1 9.8 MBPP (Pass@1) 3-shot 7.0 11.6 10.6 TriviaQA (EM) 5-shot 36.5 36.7 38.2 NaturalQuestions (EM) 5-shot 12.6 12.1 13.7\n\nTable 10 | Comparison between DeepSeekMoE and larger GShard models at a larger scale.\n\nC. Training Benchmark Curves Of Deepseekmoe 16B\n\nWe present the benchmark curves during training of DeepSeekMoE 16B and DeepSeek 7B (Dense) in Figure 7 for reference.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/DeepSeekMoE_2.md"
+          },
+          "summary": "DeepSeekMoE is an innovative Mixture-of-Experts (MoE) architecture designed to enhance expert specialization in language models. It employs fine-grained expert segmentation and shared expert isolation to improve performance and reduce computational costs. Starting with a model of 2B parameters, DeepSeekMoE demonstrates comparable performance to larger models like GShard 2.9B and LLaMA2 7B, using significantly fewer computations. The architecture scales up to 16B and 145B parameters, maintaining advantages over traditional MoE architectures. DeepSeekMoE 16B achieves similar results to dense models with only 40% of the computations, and its chat model version shows adaptability in bilingual scenarios. The architecture's public release aims to contribute to advancements in large-scale language models.",
+          "headlines": {
+            "1. Introduction": [],
+            "2. Preliminaries: Mixture-Of-Experts For Transformers": [],
+            "3. Deepseekmoe Architecture": [
+              "3.1. Fine-Grained Expert Segmentation",
+              "3.2. Shared Expert Isolation",
+              "3.3. Load Balance Consideration"
+            ],
+            "4. Validation Experiments": [
+              "4.1. Experimental Setup",
+              "4.1.1. Training Data And Tokenization",
+              "4.1.2. Infrastructures",
+              "4.1.3. Hyper-Parameters",
+              "4.1.4. Evaluation Benchmarks",
+              "4.2. Evaluations",
+              "4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models",
+              "4.4. Ablation Studies",
+              "4.5. Analysis On Expert Specialization"
+            ],
+            "5. Scaling Up To Deepseekmoe 16B": [
+              "5.1. Experimental Setup",
+              "5.1.1. Training Data And Tokenization",
+              "5.1.2. Hyper-Parameters",
+              "5.1.3. Evaluation Benchmarks",
+              "5.2. Evaluations",
+              "5.2.1. Internal Comparison With Deepseek 7B",
+              "5.2.2. Comparison With Open Source Models"
+            ],
+            "6. Alignment For Deepseekmoe 16B": [
+              "6.1. Experimental Setup",
+              "6.2. Evaluations"
+            ],
+            "7. Deepseekmoe 145B Ongoing": [
+              "7.1. Experimental Setup",
+              "7.2. Evaluations"
+            ],
+            "8. Related Work": [],
+            "9. Conclusion": [],
+            "Appendices": [
+              "A. Overview Of Hyper-Parameters",
+              "B. Comparing Deepseekmoe With Larger Models",
+              "C. Training Benchmark Curves Of Deepseekmoe 16B"
+            ]
+          },
+          "summary_embedding": [
+            -0.012465757317841053,
+            -0.022808987647294998,
+            -0.018792515620589256,
+            -0.02308654971420765,
+            0.01797616109251976,
+            0.007200260180979967,
+            0.006490030791610479,
+            0.042613785713911057,
+            -0.014049487188458443,
+            0.03187054023146629,
+            -0.021486490964889526,
+            -0.023592690005898476,
+            0.009045224636793137,
+            -0.014441338367760181,
+            0.005865518003702164,
+            -0.007363531738519669,
+            -0.008743173442780972,
+            -0.005163452122360468,
+            0.018841497600078583,
+            -0.03100520372390747,
+            0.011110605672001839,
+            -0.046075135469436646,
+            -0.020017050206661224,
+            -0.021845687180757523,
+            0.016425084322690964,
+            0.037389107048511505,
+            0.022547753527760506,
+            -0.0028062239289283752,
+            -0.06844329088926315,
+            0.034678805619478226,
+            0.035168617963790894,
+            0.02052319049835205,
+            0.0005438972148112953,
+            -0.00031302153365686536,
+            -0.024833550676703453,
+            -0.028556134551763535,
+            0.03347059711813927,
+            -0.004379750229418278,
+            -0.01453930139541626,
+            0.015976088121533394,
+            0.0414055772125721,
+            0.012106560170650482,
+            -0.02754385396838188,
+            0.0014245413476601243,
+            0.015151568688452244,
+            0.0055961208418011665,
+            -0.01678428053855896,
+            -0.0022960014175623655,
+            -0.002424577483907342,
+            -0.021225256845355034,
+            -0.01400050614029169,
+            0.018825169652700424,
+            0.001296985661610961,
+            -0.008865626528859138,
+            -0.0542386956512928,
+            0.007330877240747213,
+            0.03840138763189316,
+            0.05623060464859009,
+            0.044148534536361694,
+            -0.020180322229862213,
+            0.03369917720556259,
+            -0.030025575309991837,
+            -0.02501314878463745,
+            0.011028970591723919,
+            0.003983817528933287,
+            -0.018531281501054764,
+            0.01784554310142994,
+            -0.005636938381940126,
+            -0.0006556359585374594,
+            0.01751900091767311,
+            -0.03941367194056511,
+            0.020751770585775375,
+            -0.01887415163218975,
+            0.031266435980796814,
+            0.02173139713704586,
+            0.0073798587545752525,
+            0.01872720755636692,
+            0.01678428053855896,
+            0.0036858476232737303,
+            -0.04107903689146042,
+            0.009118696674704552,
+            0.016523046419024467,
+            0.014196431264281273,
+            0.016269976273179054,
+            0.06553706526756287,
+            -0.05391215533018112,
+            -0.004485876299440861,
+            -0.05169166624546051,
+            0.0012561678886413574,
+            -0.004971608519554138,
+            -0.03670336678624153,
+            0.01048201136291027,
+            0.028425518423318863,
+            0.021372200921177864,
+            -0.01063711941242218,
+            0.024392718449234962,
+            -0.022743679583072662,
+            0.0032001156359910965,
+            -0.04062187671661377,
+            0.0019653772469609976,
+            0.004518530797213316,
+            0.016531210392713547,
+            0.0070778070949018,
+            0.004273623693734407,
+            -0.016702644526958466,
+            0.006045116577297449,
+            -0.01905374974012375,
+            0.03291547670960426,
+            -0.008465612307190895,
+            0.08653374016284943,
+            0.008751336485147476,
+            0.004738946910947561,
+            0.004281787667423487,
+            0.02398454025387764,
+            0.012131051160395145,
+            -0.00810233410447836,
+            -0.04718538001179695,
+            -0.015690363943576813,
+            0.011061624623835087,
+            -0.0383034273982048,
+            0.018335357308387756,
+            -0.008759500458836555,
+            0.01282495353370905,
+            -0.04107903689146042,
+            -0.06344719231128693,
+            0.01498013362288475,
+            0.001784758409485221,
+            -0.004065453074872494,
+            -0.0002903166168835014,
+            -0.018923133611679077,
+            0.003930754493921995,
+            -0.004200151655822992,
+            -0.010645282454788685,
+            -0.021502818912267685,
+            0.03159298002719879,
+            0.03657275065779686,
+            0.06527583301067352,
+            0.036801330745220184,
+            -0.0030164355412125587,
+            0.005775718949735165,
+            0.019690508022904396,
+            -0.020784424617886543,
+            0.003481758525595069,
+            -0.01214737817645073,
+            -0.021992631256580353,
+            -0.03820546343922615,
+            0.010971825569868088,
+            0.07379858940839767,
+            -0.029029620811343193,
+            0.013437220826745033,
+            -0.017616963014006615,
+            0.04205866530537605,
+            -0.004567512311041355,
+            0.026727497577667236,
+            -0.016669990494847298,
+            0.004059330560266972,
+            -0.025388672947883606,
+            0.01620466820895672,
+            -0.024964168667793274,
+            0.012571883387863636,
+            -0.000885746325366199,
+            -0.04222193360328674,
+            0.00011116473615402356,
+            0.026890767738223076,
+            -0.010971825569868088,
+            0.02261306345462799,
+            0.0004949158756062388,
+            -0.006302268709987402,
+            -0.05097327008843422,
+            0.038727931678295135,
+            -0.003912386484444141,
+            0.018645571544766426,
+            -0.0018959869630634785,
+            0.0163107942789793,
+            -0.0016408757073804736,
+            0.013216804713010788,
+            0.009543201886117458,
+            0.007114543113857508,
+            -0.033503253012895584,
+            0.02697240374982357,
+            -0.02504580281674862,
+            0.028049994260072708,
+            0.03435226157307625,
+            -0.0030184765346348286,
+            0.03255628049373627,
+            0.019249675795435905,
+            0.03840138763189316,
+            0.0327032245695591,
+            0.030335791409015656,
+            -0.00824927818030119,
+            0.013878053054213524,
+            0.008200296200811863,
+            0.03608293831348419,
+            0.013429056853055954,
+            0.019723162055015564,
+            -0.009730963967740536,
+            -0.014041324146091938,
+            0.048785436898469925,
+            0.007318632211536169,
+            -0.0179598331451416,
+            -0.005441013257950544,
+            -0.03510330989956856,
+            -0.002832755446434021,
+            -0.014425011351704597,
+            0.0323440283536911,
+            0.02158445306122303,
+            -0.01647406443953514,
+            0.035135965794324875,
+            0.024980494752526283,
+            -0.03751972317695618,
+            -0.005232842173427343,
+            -0.02026195637881756,
+            -0.023298801854252815,
+            -0.04215662553906441,
+            0.011624909937381744,
+            0.012833117507398129,
+            -0.013976015150547028,
+            -0.007469657808542252,
+            0.020457882434129715,
+            -0.023625344038009644,
+            -0.0018265966791659594,
+            -0.01356783788651228,
+            0.006547175347805023,
+            -0.020898714661598206,
+            -0.03667071461677551,
+            -0.01048201136291027,
+            0.025862159207463264,
+            0.012637191452085972,
+            0.04362606629729271,
+            -0.013518855907022953,
+            -0.005367541220039129,
+            0.033193036913871765,
+            0.012759645469486713,
+            0.010767736472189426,
+            0.03699725493788719,
+            -0.03027048148214817,
+            0.024621298536658287,
+            -0.020294612273573875,
+            -0.0032878739293664694,
+            0.01707816869020462,
+            -0.0030062312725931406,
+            0.01962519995868206,
+            -0.04535674303770065,
+            -0.024196794256567955,
+            0.04395261034369469,
+            -0.0007107399869710207,
+            0.016302630305290222,
+            -0.005959399044513702,
+            -0.012106560170650482,
+            0.0035899258218705654,
+            0.0027715289033949375,
+            -0.01438419334590435,
+            -0.008620720356702805,
+            0.007473739795386791,
+            -0.01962519995868206,
+            0.03660540655255318,
+            0.03794422745704651,
+            -0.004132802598178387,
+            -0.014131123200058937,
+            -0.03324201703071594,
+            0.010016689077019691,
+            -0.017878197133541107,
+            -0.023951886221766472,
+            0.04078514873981476,
+            -0.032115448266267776,
+            -0.00029312283731997013,
+            0.039478980004787445,
+            0.018808843567967415,
+            -0.014898497611284256,
+            0.00868602842092514,
+            -0.01709449477493763,
+            -0.026302991434931755,
+            -0.017159804701805115,
+            0.03356856107711792,
+            0.024800896644592285,
+            -0.010743245482444763,
+            0.010245268233120441,
+            0.005330805201083422,
+            0.02893165871500969,
+            0.024376392364501953,
+            0.00648594880476594,
+            -0.013812744058668613,
+            0.015029114671051502,
+            0.027641816064715385,
+            -0.0028021421749144793,
+            -7.66609373386018e-05,
+            -0.004010349046438932,
+            -0.007796200457960367,
+            -0.006453294772654772,
+            -0.013069860637187958,
+            -0.005089980084449053,
+            0.0009184005321003497,
+            -0.032295044511556625,
+            0.015967924147844315,
+            -0.005890008993446827,
+            0.010359558276832104,
+            -0.011535110883414745,
+            0.01288209855556488,
+            0.01722511276602745,
+            -0.013494365848600864,
+            -0.0716434046626091,
+            0.014612773433327675,
+            0.02458864450454712,
+            -0.006706364918500185,
+            -0.017943505197763443,
+            0.008481939323246479,
+            0.00016148542636074126,
+            0.038597315549850464,
+            -0.014180104248225689,
+            -0.031103165820240974,
+            0.02680913358926773,
+            -0.017437364906072617,
+            -0.015241367742419243,
+            -0.010661610402166843,
+            -0.00580837344750762,
+            0.04649963974952698,
+            0.01889047957956791,
+            0.011420820839703083,
+            -0.04042595252394676,
+            -0.007449248805642128,
+            0.015078096650540829,
+            0.014506647363305092,
+            -0.04705476388335228,
+            0.0052083516493439674,
+            0.015486273914575577,
+            0.019723162055015564,
+            0.024800896644592285,
+            0.04274440184235573,
+            -0.0021531390957534313,
+            0.01661284640431404,
+            -0.015053605660796165,
+            -0.05136512219905853,
+            -0.03340528905391693,
+            0.029943939298391342,
+            -0.017731253057718277,
+            -0.02906227484345436,
+            -0.01693122461438179,
+            -0.006151242647320032,
+            0.0017939424142241478,
+            0.007126788143068552,
+            -0.01916803978383541,
+            -0.03464614972472191,
+            -0.03970756009221077,
+            0.02277633361518383,
+            0.007983962073922157,
+            0.004269542172551155,
+            -0.042777057737112045,
+            0.0013459670590236783,
+            0.009184005670249462,
+            0.012073906138539314,
+            0.0009954441338777542,
+            -0.019984396174550056,
+            0.015494437888264656,
+            0.0032001156359910965,
+            -0.031070511788129807,
+            -0.035593122243881226,
+            -0.03595232218503952,
+            0.0023919232189655304,
+            0.003781769424676895,
+            0.06279410421848297,
+            0.012988224625587463,
+            -0.03840138763189316,
+            -0.04522612690925598,
+            0.029176564887166023,
+            -0.02052319049835205,
+            -0.016678154468536377,
+            0.0059430720284581184,
+            0.012898425571620464,
+            -0.009877908043563366,
+            0.013192313723266125,
+            0.023560035973787308,
+            0.041601505130529404,
+            -0.005987971555441618,
+            0.012914752587676048,
+            0.023037567734718323,
+            0.012955570593476295,
+            0.0362788625061512,
+            -0.00936360377818346,
+            0.0035409443080425262,
+            0.039936136454343796,
+            -0.012269831262528896,
+            0.03175625205039978,
+            -0.02189466916024685,
+            0.0020398697815835476,
+            0.02249877341091633,
+            0.027184655889868736,
+            -0.0006479826406575739,
+            0.02594379521906376,
+            -0.031543996185064316,
+            0.018368011340498924,
+            -0.014498483389616013,
+            -0.04888340085744858,
+            -0.01557607389986515,
+            0.013102514669299126,
+            0.030384771525859833,
+            0.02756018005311489,
+            -0.0041348435916006565,
+            -0.019723162055015564,
+            -0.01155143789947033,
+            -0.010596301406621933,
+            0.0018419033149257302,
+            0.03285016864538193,
+            0.010506502352654934,
+            0.010890189558267593,
+            -0.011935125105082989,
+            -0.004104230087250471,
+            0.03025415539741516,
+            -0.021192602813243866,
+            0.01650671847164631,
+            0.0041511706076562405,
+            -0.002029665280133486,
+            -0.04320156201720238,
+            -0.035266581922769547,
+            0.007845181971788406,
+            0.011747363954782486,
+            0.014874007552862167,
+            0.0368339866399765,
+            0.025094784796237946,
+            -0.01662100851535797,
+            0.030629679560661316,
+            -0.01905374974012375,
+            0.027984686195850372,
+            0.039054471999406815,
+            0.011624909937381744,
+            -0.018351683393120766,
+            0.012237177230417728,
+            0.002424577483907342,
+            -0.025127438828349113,
+            0.01680060662329197,
+            -0.032735876739025116,
+            0.01738838478922844,
+            -0.02354370802640915,
+            0.025143766775727272,
+            -0.02579685114324093,
+            -0.009290131740272045,
+            -0.004428731743246317,
+            -0.006130834110081196,
+            0.04888340085744858,
+            -0.026727497577667236,
+            -0.0142454132437706,
+            -0.0024694770108908415,
+            -0.0024531499948352575,
+            -0.008718682453036308,
+            0.010000362060964108,
+            -0.045781247317790985,
+            -0.01844964735209942,
+            -0.04016471654176712,
+            0.007355368230491877,
+            0.005412440747022629,
+            -0.030907239764928818,
+            -0.05694899708032608,
+            0.025421326979994774,
+            -0.06331657618284225,
+            -0.019886434078216553,
+            0.051626358181238174,
+            -0.018400665372610092,
+            -0.008212542161345482,
+            0.009641164913773537,
+            0.006375740747898817,
+            -0.029290854930877686,
+            0.0041348435916006565,
+            0.02504580281674862,
+            0.04418119043111801,
+            -0.0045797573402523994,
+            -0.02865409664809704,
+            0.0028674507047981024,
+            -0.027788760140538216,
+            -0.013461710885167122,
+            0.005261414684355259,
+            0.016449574381113052,
+            0.024539662525057793,
+            0.007628847379237413,
+            0.03585435822606087,
+            0.007294141221791506,
+            0.006318595726042986,
+            0.044279150664806366,
+            -0.025666235014796257,
+            -0.027380581945180893,
+            0.02519274689257145,
+            0.04045860469341278,
+            0.023151857778429985,
+            -0.018253721296787262,
+            -0.0142454132437706,
+            0.0053063142113387585,
+            0.0009296254720538855,
+            -0.022139575332403183,
+            0.002808264922350645,
+            -0.013902543112635612,
+            0.005143043119460344,
+            0.03892385587096214,
+            -0.001205145614221692,
+            0.014196431264281273,
+            0.007302304729819298,
+            0.006845145486295223,
+            0.004334850702434778,
+            -0.011731036007404327,
+            -0.023592690005898476,
+            0.00032220553839579225,
+            0.034972693771123886,
+            0.015404638834297657,
+            -0.030515389516949654,
+            -0.00995954405516386,
+            0.034711457788944244,
+            -0.00795130804181099,
+            -0.009118696674704552,
+            -0.009771781973540783,
+            0.003385836724191904,
+            0.02638462744653225,
+            -0.01606588624417782,
+            0.006649219896644354,
+            0.03716052696108818,
+            -0.019870106130838394,
+            0.005543057806789875,
+            -0.025405000895261765,
+            0.0018061877926811576,
+            0.02442537248134613,
+            -0.006216551177203655,
+            0.028098974376916885,
+            5.6315813708351925e-05,
+            0.03025415539741516,
+            -0.004677720367908478,
+            0.008432958275079727,
+            -0.020327266305685043,
+            0.03987082839012146,
+            -0.017633290961384773,
+            0.03402572125196457,
+            0.02067013457417488,
+            -0.01258004643023014,
+            0.04783846437931061,
+            -0.036801330745220184,
+            -0.020784424617886543,
+            0.00584919098764658,
+            0.012743317522108555,
+            0.0028919412288814783,
+            -0.0191517136991024,
+            -0.018400665372610092,
+            -0.010996315628290176,
+            0.02380494214594364,
+            -0.022547753527760506,
+            -0.016865916550159454,
+            -0.007441085297614336,
+            -0.01890680566430092,
+            -0.009673818945884705,
+            -0.007988044060766697,
+            -0.01709449477493763,
+            -0.016547536477446556,
+            0.044573038816452026,
+            -0.06334923207759857,
+            0.016996532678604126,
+            0.017159804701805115,
+            0.011477965861558914,
+            -0.034678805619478226,
+            -0.019510909914970398,
+            -0.017633290961384773,
+            -0.002347023691982031,
+            -0.030743969604372978,
+            0.001702102366834879,
+            0.01725776679813862,
+            0.01138000376522541,
+            -0.05022222548723221,
+            0.012335140258073807,
+            0.02772345207631588,
+            -0.012065742164850235,
+            0.0014949520118534565,
+            -0.023135529831051826,
+            -0.017176130786538124,
+            0.0031776658724993467,
+            0.022123249247670174,
+            0.005975726060569286,
+            -0.008188051171600819,
+            -0.016865916550159454,
+            -0.005241005681455135,
+            -0.006045116577297449,
+            -0.002359268954023719,
+            0.005094061605632305,
+            0.011780017986893654,
+            0.019690508022904396,
+            -0.0024633544962853193,
+            -0.004436895251274109,
+            -0.006396149750798941,
+            0.027364253997802734,
+            -0.001450052484869957,
+            -0.008416631259024143,
+            0.050907962024211884,
+            0.03556047007441521,
+            0.006094098091125488,
+            -0.01108611561357975,
+            -0.03990348428487778,
+            0.010661610402166843,
+            -0.022711025550961494,
+            0.0401320643723011,
+            0.01905374974012375,
+            -0.008298259228467941,
+            -0.028866350650787354,
+            0.010294250212609768,
+            -0.02290695160627365,
+            -0.009975871071219444,
+            0.009469729848206043,
+            0.00726556871086359,
+            -0.03263791650533676,
+            -0.013543346896767616,
+            0.03510330989956856,
+            0.011437147855758667,
+            -0.006841063499450684,
+            -0.036050282418727875,
+            0.008588065393269062,
+            0.01646590232849121,
+            0.011869817040860653,
+            -0.019233347848057747,
+            0.004449140280485153,
+            -0.012041252106428146,
+            0.017470018938183784,
+            0.008849299512803555,
+            -0.017796561121940613,
+            0.018237393349409103,
+            0.004710374400019646,
+            -0.008490103296935558,
+            -0.003912386484444141,
+            -0.0163107942789793,
+            -0.02006603218615055,
+            0.03840138763189316,
+            0.024490682408213615,
+            0.026335645467042923,
+            0.005457340274006128,
+            0.02008235827088356,
+            0.03227871656417847,
+            0.011282040737569332,
+            -0.03134807199239731,
+            0.027935704216361046,
+            0.016996532678604126,
+            -0.04604247957468033,
+            0.021551799029111862,
+            -0.02218855731189251,
+            -0.028670424595475197,
+            0.009649328887462616,
+            -0.0017367975087836385,
+            0.011192241683602333,
+            0.0034837995190173388,
+            -0.003642988856881857,
+            -0.03307874873280525,
+            -0.006236960180103779,
+            0.015771998092532158,
+            0.015502601861953735,
+            -0.0048695639707148075,
+            -0.003296037670224905,
+            -0.011886144056916237,
+            0.00487364549189806,
+            0.009796272963285446,
+            0.039054471999406815,
+            0.001381682581268251,
+            -0.004751192405819893,
+            -0.0018765984568744898,
+            0.031854212284088135,
+            0.0271193478256464,
+            -0.017274094745516777,
+            -0.026286665350198746,
+            0.00580837344750762,
+            -0.014253576286137104,
+            0.026156047359108925,
+            -0.00487364549189806,
+            0.011877980083227158,
+            0.008718682453036308,
+            -0.0007908449042588472,
+            -0.001373519073240459,
+            0.025584599003195763,
+            -0.02710302174091339,
+            -0.021764053031802177,
+            0.031233781948685646,
+            -0.025731543079018593,
+            0.028735732659697533,
+            0.013061696663498878,
+            -0.01557607389986515,
+            -0.015225040726363659,
+            -0.010302413254976273,
+            0.009943217039108276,
+            -0.012277995236217976,
+            0.023853924125432968,
+            0.003608293831348419,
+            0.006392067763954401,
+            0.0003584313380997628,
+            0.02770712412893772,
+            0.024833550676703453,
+            0.007820690982043743,
+            -0.018074123188853264,
+            -0.0016816934803500772,
+            -0.02320083975791931,
+            -0.01680060662329197,
+            0.023772288113832474,
+            -0.018090449273586273,
+            -0.03268689662218094,
+            -0.028556134551763535,
+            0.012114724144339561,
+            0.007392104249447584,
+            0.010816717520356178,
+            0.03461349755525589,
+            -0.030384771525859833,
+            -0.0344502255320549,
+            0.01766594499349594,
+            0.026270337402820587,
+            -0.005551221314817667,
+            -0.021535472944378853,
+            0.03085825778543949,
+            -0.005828781984746456,
+            0.017600636929273605,
+            0.0404912605881691,
+            -0.0016225076979026198,
+            0.010228941217064857,
+            -0.042026009410619736,
+            -0.014057651162147522,
+            -0.0066614653915166855,
+            0.006261450704187155,
+            0.034417569637298584,
+            0.01048201136291027,
+            0.01535565685480833,
+            0.03448288142681122,
+            -0.021192602813243866,
+            -0.0002903166168835014,
+            -0.0022449791431427,
+            -0.02818061038851738,
+            0.03347059711813927,
+            0.025715215131640434,
+            0.011918798089027405,
+            0.019706835970282555,
+            -0.03794422745704651,
+            -0.013037205673754215,
+            0.027070365846157074,
+            0.015053605660796165,
+            0.01273515447974205,
+            0.016694480553269386,
+            0.011249386705458164,
+            0.00868602842092514,
+            -0.021633435040712357,
+            -0.05169166624546051,
+            0.01214737817645073,
+            0.01993541419506073,
+            -0.016449574381113052,
+            0.01057181041687727,
+            0.024980494752526283,
+            0.007620683871209621,
+            -0.007045152597129345,
+            0.012808626517653465,
+            -0.02682545967400074,
+            0.002259265398606658,
+            0.0019010890973731875,
+            -0.014482156373560429,
+            0.038270771503448486,
+            -0.016367938369512558,
+            0.011363675817847252,
+            -0.018204739317297935,
+            0.023870250210165977,
+            0.00011531029304023832,
+            0.0007964573451317847,
+            0.014963806606829166,
+            -0.011347348801791668,
+            -0.04235255345702171,
+            0.0021613026037812233,
+            0.03342161700129509,
+            -0.015200549736618996,
+            0.026449935510754585,
+            -0.0020674217958003283,
+            0.032458316534757614,
+            0.042777057737112045,
+            0.017731253057718277,
+            0.0003497575526125729,
+            0.025682561099529266,
+            0.0036144163459539413,
+            0.0073880222626030445,
+            0.012400448322296143,
+            0.009322785772383213,
+            0.007245160173624754,
+            0.030809277668595314,
+            -0.010261595249176025,
+            0.02173139713704586,
+            -0.04760988429188728,
+            -0.014629100449383259,
+            0.012024925090372562,
+            -0.0013316808035597205,
+            -0.012449429370462894,
+            -0.033029764890670776,
+            0.012563719414174557,
+            -0.03503800183534622,
+            0.022074267268180847,
+            -0.0004959363141097128,
+            0.04333217814564705,
+            -0.02187834307551384,
+            -0.005518566817045212,
+            0.016694480553269386,
+            0.0016510800924152136,
+            0.023282473906874657,
+            -0.0012153501156717539,
+            -0.016416920349001884,
+            -0.0067798369564116,
+            -0.009257477708160877,
+            0.004138925112783909,
+            0.04375668615102768,
+            -0.003722583642229438,
+            -0.0001682458823779598,
+            0.01409030519425869,
+            -0.011959616094827652,
+            -0.03250729665160179,
+            0.045193471014499664,
+            -0.0030837850645184517,
+            0.02484987862408161,
+            0.04979771748185158,
+            -0.06410028040409088,
+            -0.019543563947081566,
+            0.013061696663498878,
+            -0.010441194288432598,
+            0.011363675817847252,
+            0.005547139327973127,
+            0.016114868223667145,
+            0.017894525080919266,
+            -0.005485912784934044,
+            0.01619650423526764,
+            0.005106307100504637,
+            0.028098974376916885,
+            -0.02336410991847515,
+            -0.03296445682644844,
+            -0.02636830136179924,
+            -0.017747581005096436,
+            0.0014480114914476871,
+            -0.015796490013599396,
+            -3.563585551091819e-06,
+            -0.0022204883862286806,
+            -0.04388730227947235,
+            -0.025274382904171944,
+            0.040687184780836105,
+            0.006706364918500185,
+            0.02458864450454712,
+            0.0077268099412322044,
+            0.03418898954987526,
+            0.005885927006602287,
+            -0.009037061594426632,
+            0.030058229342103004,
+            -0.020049704238772392,
+            0.03833607956767082,
+            0.022106921300292015,
+            -0.002832755446434021,
+            0.005865518003702164,
+            0.031086837872862816,
+            -0.033797141164541245,
+            0.008086007088422775,
+            -0.034384917467832565,
+            0.0046695563942193985,
+            -0.008620720356702805,
+            -0.015094423666596413,
+            -0.009526874870061874,
+            -0.02504580281674862,
+            0.011216731742024422,
+            -0.021535472944378853,
+            0.01590261608362198,
+            0.010416703298687935,
+            -0.002916431985795498,
+            0.03448288142681122,
+            -0.014204595237970352,
+            -0.0027796924114227295,
+            0.0024531499948352575,
+            -0.027070365846157074,
+            -0.006343086715787649,
+            -0.003377673216164112,
+            -0.03689929470419884,
+            -0.004502203781157732,
+            0.010988152585923672,
+            -0.00024184548237826675,
+            0.016898570582270622,
+            0.02787039615213871,
+            0.0005092020728625357,
+            -0.022302847355604172,
+            0.002340900944545865,
+            -0.04199335351586342,
+            0.0011877981014549732,
+            0.01483318954706192,
+            -0.017600636929273605,
+            -0.008612556383013725,
+            0.033046092838048935,
+            0.023298801854252815,
+            -0.009396257810294628,
+            -0.035005345940589905,
+            0.017796561121940613,
+            -0.011853490024805069,
+            0.027511198073625565,
+            -0.0002630197268445045,
+            0.019298657774925232,
+            -0.0076941559091210365,
+            0.04493223875761032,
+            0.001369437319226563,
+            0.008645210415124893,
+            -0.021959977224469185,
+            0.007020662073045969,
+            -0.020278284326195717,
+            -0.002406209474429488,
+            -0.006171651650220156,
+            -0.012751481495797634,
+            -0.010669773444533348,
+            0.0032735876739025116,
+            0.007730891928076744,
+            -0.0176822729408741,
+            0.0055961208418011665,
+            -0.015959760174155235,
+            -0.02741323597729206,
+            0.007543129846453667,
+            -0.05965929850935936,
+            -0.0179598331451416,
+            -0.03373183310031891,
+            -0.009477893821895123,
+            0.02008235827088356,
+            0.03510330989956856,
+            -0.012759645469486713,
+            0.00497977202758193,
+            -0.014898497611284256,
+            0.004477712791413069,
+            0.03490738570690155,
+            0.044442422688007355,
+            -0.014759717509150505,
+            0.003381754970178008,
+            -0.010955498553812504,
+            0.033029764890670776,
+            -0.03347059711813927,
+            0.0030062312725931406,
+            0.031086837872862816,
+            -0.005057325586676598,
+            0.007959471084177494,
+            0.011624909937381744,
+            0.020474210381507874,
+            -0.019396619871258736,
+            0.00877582747489214,
+            0.002100075827911496,
+            0.019233347848057747,
+            0.02171507105231285,
+            0.020343592390418053,
+            -0.03539719805121422,
+            0.028409190475940704,
+            0.018384337425231934,
+            0.02444170042872429,
+            0.044442422688007355,
+            -0.03974021226167679,
+            -0.004147088620811701,
+            -0.0032225653994828463,
+            -0.02997659333050251,
+            0.02008235827088356,
+            -0.007530884351581335,
+            -0.01841699331998825,
+            -0.007069643586874008,
+            0.03327467292547226,
+            0.014571955427527428,
+            -0.05051611363887787,
+            0.01980479806661606,
+            0.01740471087396145,
+            0.02055584453046322,
+            0.024474354460835457,
+            -0.01123305968940258,
+            -0.0019072118448093534,
+            0.008996243588626385,
+            0.006804327480494976,
+            0.019266001880168915,
+            0.015257694758474827,
+            0.002012317767366767,
+            -0.0022082431241869926,
+            -0.011837163008749485,
+            0.005224678665399551,
+            -0.009077879600226879,
+            0.010808554477989674,
+            -0.02458864450454712,
+            -0.009698309935629368,
+            0.021388528868556023,
+            0.0031164393294602633,
+            -0.013665799982845783,
+            -0.03729114308953285,
+            -0.045944519340991974,
+            0.0033837957307696342,
+            0.004543021321296692,
+            -0.05365091934800148,
+            0.005322641227394342,
+            -0.026547899469733238,
+            -0.002108239568769932,
+            -0.005502239800989628,
+            0.0019214981002733111,
+            0.005636938381940126,
+            0.0167679525911808,
+            -0.013494365848600864,
+            0.02248244546353817,
+            -0.015502601861953735,
+            0.03165828809142113,
+            0.0055961208418011665,
+            0.007375776767730713,
+            -0.0027796924114227295,
+            0.05590406060218811,
+            -0.003712379140779376,
+            -0.03699725493788719,
+            0.023282473906874657,
+            -0.01311067771166563,
+            -0.0001747512142173946,
+            0.030090883374214172,
+            0.00884113647043705,
+            -0.015257694758474827,
+            0.013437220826745033,
+            -0.030613351613283157,
+            -0.03249097242951393,
+            0.010996315628290176,
+            -0.013755599036812782,
+            -0.0015776080545037985,
+            -0.04568328335881233,
+            -0.009339112788438797,
+            0.004269542172551155,
+            -0.018335357308387756,
+            0.010775899514555931,
+            0.01631895825266838,
+            0.013992343097925186,
+            -0.024245774373412132,
+            0.009053388610482216,
+            0.006865554489195347,
+            0.013453547842800617,
+            0.0007331897504627705,
+            0.014049487188458443,
+            -0.008677864447236061,
+            -0.011306531727313995,
+            -0.018319029361009598,
+            0.0032450153958052397,
+            -0.026792805641889572,
+            0.005155288614332676,
+            -0.003155216109007597,
+            -0.03771565109491348,
+            -0.06521052122116089,
+            -0.0344502255320549,
+            0.016294466331601143,
+            0.03396041318774223,
+            -0.009045224636793137,
+            0.0035144127905368805,
+            -0.00023648813657928258,
+            -0.01093917153775692,
+            0.024523336440324783,
+            -0.005930826533585787,
+            -0.0004829256213270128,
+            0.0035389035474509,
+            0.02050686441361904,
+            -0.009722800925374031,
+            0.005845109466463327,
+            -0.051952898502349854,
+            -0.01586179807782173,
+            0.01812310516834259,
+            0.006306350696831942,
+            0.014629100449383259,
+            -0.009690145961940289,
+            0.01693122461438179,
+            -0.030074557289481163,
+            -0.028147956356406212,
+            0.04130761697888374,
+            0.008277850225567818,
+            0.011486129835247993,
+            -0.003055212553590536,
+            -0.01680060662329197,
+            0.02504580281674862,
+            -0.0020633398089557886,
+            -0.006461458280682564,
+            -0.00817988719791174,
+            0.015820980072021484,
+            0.044279150664806366,
+            0.003255219664424658,
+            0.0009821783751249313,
+            0.017045514658093452,
+            -0.002663361607119441,
+            -0.011575928889214993,
+            0.007677828427404165,
+            0.013592327944934368,
+            -0.016384266316890717,
+            0.010375885292887688,
+            -0.025862159207463264,
+            0.0332583449780941,
+            0.03131541982293129,
+            0.011575928889214993,
+            -0.015045441687107086,
+            0.006130834110081196,
+            0.005330805201083422,
+            0.008086007088422775,
+            -0.001369437319226563,
+            -0.0017133272485807538,
+            0.020719116553664207,
+            -0.03276853263378143,
+            -0.01693122461438179,
+            -0.005347132217139006,
+            0.003932795487344265,
+            0.005277741700410843,
+            0.03396041318774223,
+            -0.0033266509417444468,
+            0.017927179113030434,
+            -0.011657563969492912,
+            0.010579974390566349,
+            0.021764053031802177,
+            0.037095218896865845,
+            -0.024996822699904442,
+            -0.029862305149435997,
+            0.012914752587676048,
+            -0.001296985661610961,
+            0.014065815135836601,
+            0.0010173837654292583,
+            0.014139287173748016,
+            0.0029409227427095175,
+            0.060475654900074005,
+            -0.017763907089829445,
+            0.004628738854080439,
+            0.0004035860183648765,
+            -0.01858026348054409,
+            -0.0013531101867556572,
+            0.004261378664523363,
+            -0.010979988612234592,
+            0.03608293831348419,
+            0.002344982698559761,
+            0.039185091853141785,
+            0.013918871060013771,
+            0.009290131740272045,
+            0.0015143405180424452,
+            -0.013592327944934368,
+            0.008355404250323772,
+            0.0015592400450259447,
+            -0.001132694073021412,
+            0.009461566805839539,
+            0.019298657774925232,
+            0.009714636951684952,
+            0.025274382904171944,
+            0.009788108989596367,
+            0.012857607565820217,
+            -0.029470453038811684,
+            0.03510330989956856,
+            -0.0006224715034477413,
+            0.012996388599276543,
+            -0.02040890045464039,
+            -0.003136848099529743,
+            -0.0008240094175562263,
+            0.015649545937776566,
+            -0.015616890974342823,
+            -0.001201063860207796,
+            -0.01297189760953188,
+            0.040099408477544785,
+            -0.018988441675901413,
+            0.013976015150547028,
+            0.009673818945884705,
+            -0.02865409664809704,
+            -0.007596192881464958,
+            0.01995174214243889,
+            0.014294394291937351,
+            -0.0053144777193665504,
+            -0.020490536466240883,
+            0.005832863971590996,
+            0.015380147844552994,
+            -0.028572462499141693,
+            1.3584674888988957e-05,
+            0.014751553535461426,
+            -0.0164903923869133,
+            -0.00025409081717953086,
+            -0.004702210891991854,
+            0.003975654020905495,
+            -0.009265640750527382,
+            0.007028825581073761,
+            0.016057724133133888,
+            -0.024507008492946625,
+            -0.020049704238772392,
+            0.005722655914723873,
+            0.006571666337549686,
+            0.01444950234144926,
+            0.007739055436104536,
+            0.004355259705334902,
+            -0.020033378154039383,
+            -0.01799248717725277,
+            0.0032674651592969894,
+            -0.01649855636060238,
+            -0.0048614004626870155,
+            0.0020531355403363705,
+            -0.019363965839147568,
+            0.01875986158847809,
+            -0.010743245482444763,
+            0.00017538899555802345,
+            -0.003226647386327386,
+            -0.020800752565264702,
+            0.006204306147992611,
+            -0.004865481983870268,
+            0.019870106130838394,
+            0.003555230563506484,
+            0.02785406820476055,
+            0.02310287579894066,
+            -0.004428731743246317,
+            0.017600636929273605,
+            -0.016155686229467392,
+            0.007983962073922157,
+            -0.031135819852352142,
+            -0.001782717532478273,
+            -0.010408539324998856,
+            0.0036103345919400454,
+            -0.015943434089422226,
+            -0.009126860648393631,
+            -0.0048777274787425995,
+            -0.0006678812787868083,
+            -0.004820582456886768,
+            0.0004275664687156677,
+            0.0027817331720143557,
+            0.02231917530298233,
+            -0.013208640739321709,
+            -0.02803366631269455,
+            -0.017437364906072617,
+            0.006045116577297449,
+            0.019396619871258736,
+            -0.026025431230664253,
+            -0.009184005670249462,
+            -0.005012426059693098,
+            -0.004326687194406986,
+            -0.021404854953289032,
+            -0.015167895704507828,
+            -0.005632856860756874,
+            -0.008045189082622528,
+            -0.010522829368710518,
+            0.0046532293781638145,
+            -0.0035715578123927116,
+            -0.011739199981093407,
+            0.021666089072823524,
+            0.018057795241475105,
+            -0.019837452098727226,
+            -0.019837452098727226,
+            0.013641309924423695,
+            -0.006159406621009111,
+            0.0018878233386203647,
+            0.02008235827088356,
+            0.00029286774224601686,
+            0.005824700463563204,
+            0.020882388576865196,
+            -0.0035225762985646725,
+            -0.01797616109251976,
+            0.003577680326998234,
+            -0.0018888438353314996,
+            -0.012122887186706066,
+            0.003212361130863428,
+            0.02773977816104889,
+            0.0008076822850853205,
+            0.023429417982697487,
+            -0.03056436963379383,
+            -0.011412657797336578,
+            0.006061443593353033,
+            0.02369065210223198,
+            0.00211844383738935,
+            -0.0021694661118090153,
+            0.020131340250372887,
+            0.03941367194056511,
+            0.005775718949735165,
+            0.0015755671774968505,
+            -0.04003410041332245,
+            0.01027792226523161,
+            0.009208496659994125,
+            0.024621298536658287,
+            0.008339077234268188,
+            0.0014653591206297278,
+            -0.012906589545309544,
+            -0.009755454957485199,
+            -0.0017939424142241478,
+            0.014702572487294674,
+            -0.023870250210165977,
+            0.0024755997583270073,
+            -0.03157665207982063,
+            0.005094061605632305,
+            0.0021959978621453047,
+            -0.009559528902173042,
+            0.0335359051823616,
+            0.00980443600565195,
+            -0.02065380848944187,
+            0.015151568688452244,
+            0.005767555441707373,
+            0.007330877240747213,
+            0.022449791431427002,
+            -0.008228869177401066,
+            0.030450081452727318,
+            -0.005049162078648806,
+            0.01213921420276165,
+            0.02742956392467022,
+            0.032588932663202286,
+            -0.013102514669299126,
+            -0.003959327004849911,
+            0.0006097159348428249,
+            -0.0002561317232903093,
+            0.0007995187188498676,
+            -0.008204378187656403,
+            0.0037389106582850218,
+            -0.014800535514950752,
+            0.012710663489997387,
+            -0.019543563947081566,
+            -0.0034286954905837774,
+            0.0038532004691660404,
+            0.01991908811032772,
+            0.0016265894519165158,
+            -0.033633869141340256,
+            0.02055584453046322,
+            -0.015420965850353241,
+            0.0135596739128232,
+            0.001290863030590117,
+            -0.011110605672001839,
+            -0.00029210239881649613,
+            0.020588500425219536,
+            0.010947334580123425,
+            -0.007845181971788406,
+            -0.0135596739128232,
+            -0.028376536443829536,
+            -0.021535472944378853,
+            0.00823703221976757,
+            0.009445239789783955,
+            -0.0028837777208536863,
+            0.0021306893322616816,
+            -0.011869817040860653,
+            0.0006520643946714699,
+            -0.0008122742874547839,
+            0.01740471087396145,
+            -0.0335359051823616,
+            0.021943651139736176,
+            -0.019527235999703407,
+            -0.019543563947081566,
+            0.017045514658093452,
+            0.008049271069467068,
+            0.009445239789783955,
+            0.0033552234526723623,
+            -0.01995174214243889,
+            -0.012326976284384727,
+            0.0338951013982296,
+            -0.01138000376522541,
+            0.006298186723142862,
+            0.004408322740346193,
+            0.012914752587676048,
+            -0.008914608508348465,
+            -0.011926962062716484,
+            0.01784554310142994,
+            -0.009861581027507782,
+            0.013951525092124939,
+            0.0010010566329583526,
+            -0.0254703089594841,
+            0.02739690989255905,
+            -0.019233347848057747,
+            0.03487472981214523,
+            -0.021617108955979347,
+            0.00711862463504076,
+            0.0066614653915166855,
+            0.008792154490947723,
+            -0.006690037902444601,
+            -0.013224967755377293,
+            0.0012786176521331072,
+            0.03807484731078148,
+            -0.02623768337070942,
+            0.024180466309189796,
+            0.022400809451937675,
+            -0.00040766780148260295,
+            0.02065380848944187,
+            -0.008367649279534817,
+            -0.026711169630289078,
+            0.0179598331451416,
+            -0.012596373446285725,
+            -0.026123393326997757,
+            0.013257622718811035,
+            -0.022678371518850327,
+            -0.020278284326195717,
+            -0.026302991434931755,
+            0.004947117529809475,
+            -0.0025694805663079023,
+            -0.005151206627488136,
+            0.009004407562315464,
+            0.02561725303530693,
+            0.020915042608976364,
+            0.019576217979192734,
+            0.02111096680164337,
+            0.01161674689501524,
+            0.022449791431427002,
+            -0.007559456862509251,
+            0.04401791840791702,
+            -0.004947117529809475,
+            0.004632820375263691,
+            -0.0015980169409886003,
+            -0.015657709911465645,
+            0.009771781973540783,
+            0.0119106350466609,
+            0.0029572497587651014,
+            -0.029176564887166023,
+            -0.009061552584171295,
+            -0.03660540655255318,
+            0.005053244065493345,
+            0.0020664012990891933,
+            0.0176822729408741,
+            0.032115448266267776,
+            -0.005865518003702164,
+            0.004592002835124731,
+            -0.005102225113660097,
+            -0.013927034102380276,
+            -0.009420748800039291,
+            -0.02739690989255905,
+            -0.025437654927372932,
+            -0.007653337903320789,
+            -0.013739272020757198,
+            -0.030368445441126823,
+            -0.011722872965037823,
+            0.01551892887800932,
+            -0.007645174395292997,
+            0.011918798089027405,
+            -0.016702644526958466,
+            0.01385356206446886,
+            -0.004110352601855993,
+            -0.004763437435030937,
+            -0.004530776292085648,
+            -0.017600636929273605,
+            -0.006420640274882317,
+            0.014367866329848766,
+            -0.009477893821895123,
+            -0.01237595733255148,
+            -0.02215590327978134,
+            -0.0015459742862731218,
+            -0.01574750803411007,
+            0.003489922033622861,
+            -0.014596446417272091,
+            -0.006771673448383808,
+            -0.003908304497599602,
+            -0.018514955416321754,
+            0.01766594499349594,
+            0.014816862531006336,
+            -0.019380291923880577,
+            -0.0090697156265378,
+            0.02638462744653225,
+            0.042777057737112045,
+            -0.03218075633049011,
+            -0.008200296200811863,
+            0.0020378287881612778,
+            0.018090449273586273,
+            0.010171796195209026,
+            0.016702644526958466,
+            -0.024653952568769455,
+            0.0022286521270871162,
+            -0.021649762988090515,
+            -0.001220452249981463,
+            -0.023739634081721306,
+            0.010775899514555931,
+            0.025715215131640434,
+            -0.010971825569868088,
+            0.025976449251174927,
+            0.01634344831109047,
+            -0.01812310516834259,
+            0.012016761116683483,
+            0.007183933164924383,
+            0.014645427465438843,
+            -0.019755816087126732,
+            0.0167679525911808,
+            -0.015796490013599396,
+            0.031233781948685646,
+            0.00892277155071497,
+            -0.04829562455415726,
+            -0.03503800183534622,
+            -0.013388238847255707,
+            0.0013388239312916994,
+            -0.010702427476644516,
+            0.015037278644740582,
+            0.015967924147844315,
+            0.013959688134491444,
+            -0.019380291923880577,
+            -0.015820980072021484,
+            0.011984107084572315,
+            0.010065670125186443,
+            0.027462217956781387,
+            -0.006649219896644354,
+            -0.004212397150695324,
+            -0.00960034690797329,
+            -0.02249877341091633,
+            -0.016098542138934135,
+            0.0209803506731987,
+            -0.018041469156742096,
+            0.015935270115733147,
+            -0.026139721274375916,
+            -0.012449429370462894,
+            0.017453692853450775,
+            -0.05211617052555084,
+            -0.01453113742172718,
+            0.0007612520130351186,
+            0.03012353740632534,
+            0.004751192405819893,
+            0.0035184945445507765,
+            0.01841699331998825,
+            -0.01993541419506073,
+            -0.0039776950143277645,
+            -0.02099667675793171,
+            0.006832899991422892,
+            -0.011853490024805069,
+            -0.013984179124236107,
+            0.014482156373560429,
+            -0.013845399022102356,
+            -0.01601690612733364,
+            0.0034368589986115694,
+            0.007183933164924383,
+            -0.021094640716910362,
+            0.011951452121138573,
+            0.003945040516555309,
+            0.012171869166195393,
+            0.012171869166195393,
+            0.01199227012693882,
+            0.021551799029111862,
+            0.02295593172311783,
+            -0.007334959227591753,
+            -0.004767519421875477,
+            -0.01004117913544178,
+            -0.0015398516552522779,
+            0.008898280560970306,
+            0.0033797139767557383,
+            -0.002989904023706913,
+            0.007837017998099327,
+            0.010033016093075275,
+            -0.01498013362288475,
+            0.033193036913871765,
+            -0.0035878848284482956,
+            -0.04134026914834976,
+            -0.003238892648369074,
+            -0.03800953924655914,
+            0.020033378154039383,
+            -0.023298801854252815,
+            -0.016996532678604126,
+            0.0032776694279164076,
+            -0.00629410520195961,
+            -0.019902760162949562,
+            0.013608654960989952,
+            0.011404493823647499,
+            0.00258580781519413,
+            0.02021297626197338,
+            0.012269831262528896,
+            -0.005134879611432552,
+            0.026172375306487083,
+            -0.014718899503350258,
+            -0.008751336485147476,
+            0.015567909926176071,
+            0.006853308994323015,
+            0.0022347746416926384,
+            0.0011031010653823614,
+            -0.018825169652700424,
+            0.005098143592476845,
+            0.0164903923869133,
+            0.007216587662696838,
+            -0.000566347036510706,
+            0.014049487188458443,
+            0.012514738366007805,
+            -0.015135240741074085,
+            -0.0002213090192526579,
+            -0.026564225554466248,
+            0.0032817514147609472,
+            0.004567512311041355,
+            0.007506393827497959,
+            -0.006771673448383808,
+            0.006281859707087278,
+            0.00658799335360527,
+            0.014759717509150505,
+            -0.010155469179153442,
+            -0.016980206593871117,
+            -0.002510294783860445,
+            0.004285869188606739,
+            0.016694480553269386,
+            0.03157665207982063,
+            -0.010310577228665352,
+            -0.005734901409596205,
+            -0.0386626236140728,
+            0.02070278860628605,
+            0.03332365304231644,
+            0.011175914667546749,
+            -0.028213264420628548,
+            0.012302485294640064,
+            0.0006515541463159025,
+            0.0019276207312941551,
+            -0.026302991434931755,
+            0.02787039615213871,
+            0.00974729098379612,
+            0.0066614653915166855,
+            0.017861871048808098,
+            0.0022306928876787424,
+            0.0029266364872455597,
+            -0.016302630305290222,
+            0.01018812321126461,
+            0.009086042642593384,
+            -0.017306748777627945,
+            0.0009745250572450459,
+            0.02324981987476349,
+            -0.01423724927008152,
+            -0.003208279376849532,
+            0.0018133309204131365,
+            0.02651524543762207,
+            -0.013820908032357693,
+            -0.014294394291937351,
+            0.0005714492290280759,
+            0.01751900091767311,
+            -0.0007979880319908261,
+            0.012098397128283978,
+            0.04898136481642723,
+            -0.012424939312040806,
+            0.019657853990793228,
+            0.005461421795189381,
+            -0.015290348790585995,
+            -0.032605260610580444,
+            -0.004922627005726099,
+            -0.01830270327627659,
+            0.011200404725968838,
+            0.014343376271426678,
+            -0.015763835981488228,
+            0.0007525782566517591,
+            -0.03144603595137596,
+            0.007743136957287788,
+            0.014735226519405842,
+            0.018776189535856247,
+            -0.002440904499962926,
+            0.006134915631264448,
+            0.027674470096826553,
+            0.02458864450454712,
+            -0.007392104249447584,
+            0.013804581016302109,
+            0.02320083975791931,
+            -0.025062130764126778,
+            -0.003069498809054494,
+            0.006820654962211847,
+            0.004104230087250471,
+            0.0053063142113387585,
+            -0.016963878646492958,
+            0.03915243595838547,
+            -0.005922663025557995,
+            -0.01506176870316267,
+            0.0030266400426626205,
+            -0.009028897620737553,
+            0.0063390047289431095,
+            -0.012563719414174557,
+            0.012637191452085972,
+            -0.008114579133689404,
+            0.00022883480414748192,
+            -0.0059430720284581184,
+            -0.011110605672001839,
+            -0.012653518468141556,
+            -0.0007949267164804041,
+            -0.016531210392713547,
+            0.007432921789586544,
+            0.00981259997934103,
+            0.005045080557465553,
+            0.01025343220680952,
+            -0.0036797248758375645,
+            -0.003136848099529743,
+            -0.016751626506447792,
+            -0.009984034113585949,
+            0.0073716952465474606,
+            0.01872720755636692,
+            -0.028588788583874702,
+            -0.018547609448432922,
+            0.009290131740272045,
+            0.0022898786701261997,
+            0.018776189535856247,
+            0.0158944521099329,
+            -0.0002595757250674069,
+            0.012318813242018223,
+            -0.01964152604341507,
+            -0.009184005670249462,
+            0.009690145961940289,
+            -0.003569516818970442,
+            0.01048201136291027,
+            -0.014759717509150505,
+            0.010033016093075275,
+            -0.011159587651491165,
+            -0.011053460650146008,
+            0.01385356206446886,
+            0.01738838478922844,
+            0.008873790502548218,
+            -0.001294944784604013,
+            3.383413059054874e-05,
+            -0.010506502352654934,
+            -0.024033522233366966,
+            0.004771600943058729,
+            0.027021385729312897,
+            -0.013273949734866619,
+            0.003883813973516226,
+            0.008702355436980724,
+            -0.017502672970294952,
+            -0.019821124151349068,
+            -0.012653518468141556,
+            -0.022106921300292015,
+            0.020572172477841377,
+            0.023413091897964478,
+            -0.02026195637881756,
+            0.00877582747489214,
+            -0.004620575346052647,
+            0.014049487188458443,
+            -0.01400050614029169,
+            0.010612628422677517,
+            0.0317889042198658,
+            -0.020000724121928215,
+            -0.0028654097113758326,
+            0.0036633978597819805,
+            -0.015347493812441826,
+            0.018629245460033417,
+            0.03268689662218094,
+            -0.00176945177372545,
+            0.014041324146091938,
+            0.0008138049161061645,
+            -0.009118696674704552,
+            0.01905374974012375,
+            0.011094278655946255,
+            0.005347132217139006,
+            -0.0042491331696510315,
+            -0.012212686240673065,
+            -0.03010721132159233,
+            0.016702644526958466,
+            0.0129637336358428,
+            -0.0037674831692129374,
+            -0.0011551438365131617,
+            -0.006343086715787649,
+            -0.015404638834297657,
+            0.0035123717971146107,
+            -0.013078023679554462,
+            -0.00803294312208891,
+            -0.008759500458836555,
+            0.004314441699534655,
+            -0.0034960447810590267,
+            -0.009053388610482216,
+            0.007236996199935675,
+            0.0009510547970421612,
+            -0.0010602424154058099,
+            -0.01902109570801258,
+            -0.025078458711504936,
+            -0.00995954405516386,
+            -0.016849588602781296,
+            0.0038287099450826645,
+            0.0005234883283264935,
+            -0.027184655889868736,
+            0.0053961132653057575,
+            -0.008139070123434067,
+            -0.008134988136589527,
+            0.007126788143068552,
+            0.006881881505250931,
+            -0.007437003776431084,
+            -0.010816717520356178,
+            -0.00786150898784399,
+            -0.006865554489195347,
+            0.020033378154039383,
+            0.0034348180051892996,
+            0.012082069180905819,
+            -0.010498338378965855,
+            -0.018025141209363937,
+            -0.016963878646492958,
+            0.016718972474336624,
+            -0.008971752598881721,
+            -0.02083340659737587,
+            -0.01444950234144926,
+            -0.008424794301390648,
+            -0.007665583398193121,
+            -0.01527402177453041,
+            -0.021159948781132698,
+            -0.0032103201374411583,
+            -0.011347348801791668,
+            -0.0067431009374558926,
+            -0.0028960229828953743,
+            0.0013571919407695532,
+            0.00951871182769537,
+            0.00546958576887846,
+            0.00438791373744607,
+            0.021666089072823524,
+            0.010122815147042274,
+            -0.005461421795189381,
+            0.016441410407423973,
+            -0.006183897145092487,
+            -0.004440976772457361,
+            -0.01498013362288475,
+            -0.009918726049363613,
+            0.018547609448432922,
+            -0.0055226488038897514,
+            0.008457448333501816,
+            -0.0009536059224046767,
+            -0.00576347392052412,
+            0.015674035996198654,
+            0.0129637336358428,
+            0.016718972474336624,
+            0.004771600943058729,
+            -0.00012168807734269649,
+            0.012416775338351727,
+            -0.0015745466807857156,
+            0.012555556371808052,
+            -0.0038429962005466223,
+            0.014506647363305092,
+            -0.009722800925374031,
+            -0.018988441675901413,
+            0.004157293122261763,
+            0.006265532691031694,
+            -0.012531065382063389,
+            0.009404421783983707,
+            0.007743136957287788,
+            -0.013747435994446278,
+            -0.02261306345462799,
+            -0.000106317616882734,
+            0.00014362763613462448,
+            0.008122743107378483,
+            0.008971752598881721,
+            -0.008073761127889156,
+            0.012890261597931385,
+            0.004616493359208107,
+            -0.007179851643741131,
+            -0.019657853990793228,
+            0.026009103283286095,
+            0.015608727931976318,
+            0.012269831262528896,
+            -0.052508022636175156,
+            -0.0030980713199824095,
+            0.010743245482444763,
+            -0.015951598063111305,
+            -0.026792805641889572,
+            -0.006404313258826733,
+            -0.008759500458836555,
+            -0.04290767386555672,
+            -0.009624837897717953,
+            -0.0005724696675315499,
+            -0.011388166807591915,
+            0.010400376282632351,
+            0.01573934406042099,
+            0.014612773433327675,
+            0.005489994306117296,
+            -0.007232914678752422,
+            -0.017633290961384773,
+            -0.02788672223687172,
+            0.010057506151497364,
+            -0.013380075804889202,
+            -0.006412476766854525,
+            -0.002263347152620554,
+            -0.007690073922276497,
+            0.0016112826997414231,
+            0.01691489666700363,
+            0.042483169585466385,
+            0.023037567734718323,
+            -0.012335140258073807,
+            -0.020131340250372887,
+            -0.012800462543964386,
+            -0.009192168712615967,
+            0.00633492274209857,
+            0.011592255905270576,
+            -0.004738946910947561,
+            0.0096656559035182,
+            -0.004294032696634531,
+            0.012302485294640064,
+            0.012767808511853218,
+            0.01680060662329197,
+            -2.3023791072773747e-05,
+            0.005579793825745583,
+            -0.0024694770108908415,
+            -0.01738838478922844,
+            -0.004379750229418278,
+            0.008261523209512234,
+            -0.0008872769540175796,
+            0.0019317024853080511,
+            0.014204595237970352,
+            0.019102731719613075,
+            -0.009094206616282463,
+            -0.014612773433327675,
+            -0.016098542138934135,
+            -0.005277741700410843,
+            -0.0012622905196622014,
+            0.029927613213658333,
+            0.02037624642252922,
+            -0.019266001880168915,
+            -0.018237393349409103,
+            -0.018351683393120766,
+            0.00922482367604971,
+            -0.003563394071534276,
+            -0.0064369672909379005,
+            0.01905374974012375,
+            0.023021239787340164,
+            0.017910851165652275,
+            -0.003642988856881857,
+            -0.014865843579173088,
+            0.003804219188168645,
+            0.02847449854016304,
+            0.008367649279534817,
+            0.01078406348824501,
+            0.008726846426725388,
+            -0.0011908593587577343,
+            -0.032441988587379456,
+            -0.0113146947696805,
+            0.00951871182769537,
+            -0.008481939323246479,
+            0.026564225554466248,
+            -0.016531210392713547,
+            0.009388094767928123,
+            -0.005077734589576721,
+            0.0019286411115899682,
+            -0.02365799807012081,
+            0.0014929111348465085,
+            0.010122815147042274,
+            -0.024947840720415115,
+            0.00336950970813632,
+            -0.023592690005898476,
+            -0.00013763252354692668,
+            0.008367649279534817,
+            0.0022837561555206776,
+            0.01650671847164631,
+            -0.0113146947696805,
+            0.0051838611252605915,
+            0.02099667675793171,
+            -0.002832755446434021,
+            -0.01512707769870758,
+            -0.0007842120248824358,
+            0.0005092020728625357,
+            -0.001286781276576221,
+            0.02563357912003994,
+            0.015812816098332405,
+            -0.0036756431218236685,
+            0.020882388576865196,
+            0.008677864447236061,
+            0.0019133344758301973,
+            -0.021764053031802177,
+            0.0023796779569238424,
+            -0.032899148762226105,
+            0.017437364906072617,
+            -0.011126932688057423,
+            0.012490247376263142,
+            0.00868602842092514,
+            -0.01341272983700037,
+            -0.0007388022495433688,
+            -0.0107187544927001,
+            -0.0015725058037787676,
+            -0.007934981025755405,
+            0.009779945947229862,
+            -0.004363423213362694,
+            0.008339077234268188,
+            -0.00024579971795901656,
+            -0.01476788055151701,
+            0.018694553524255753,
+            9.678283277025912e-06,
+            -0.00010925139940809458,
+            0.005441013257950544,
+            0.009935053065419197,
+            0.00786559097468853,
+            0.002009256277233362,
+            0.023870250210165977,
+            -0.031119491904973984,
+            -0.016082214191555977,
+            -0.010204450227320194,
+            0.005543057806789875,
+            -0.007922735065221786,
+            -0.018531281501054764,
+            -0.03257260471582413,
+            0.01318415068089962,
+            0.001293924287892878,
+            -0.0009806477464735508,
+            0.01258004643023014,
+            -0.00942891277372837,
+            -0.0013878053287044168,
+            -0.01606588624417782,
+            -0.014808698557317257,
+            0.006416558753699064,
+            0.0033633869607001543,
+            -0.01222901325672865,
+            -0.023739634081721306,
+            0.014759717509150505,
+            0.0007418635650537908,
+            -0.016980206593871117,
+            -0.017502672970294952,
+            0.00018457300029695034,
+            -0.004804255440831184,
+            0.02204161323606968,
+            -0.016678154468536377,
+            0.003373591462150216,
+            0.0007219648687168956,
+            0.009232986718416214,
+            -0.02889900468289852,
+            0.00716352416202426,
+            0.0006107363733462989,
+            0.0035164537839591503,
+            -0.006518602836877108,
+            -0.021013004705309868,
+            0.011371839791536331,
+            -0.0020674217958003283,
+            -0.014751553535461426,
+            -0.020751770585775375,
+            -0.0020796670578420162,
+            -0.007420676294714212,
+            -0.02724996581673622,
+            0.017910851165652275,
+            0.0009985055075958371,
+            -0.01738838478922844,
+            0.008077843114733696,
+            0.006596156861633062,
+            0.009355440735816956,
+            0.02455599047243595,
+            0.010147306136786938,
+            -0.018776189535856247,
+            -0.0008087027235887945,
+            0.006979844067245722,
+            0.009306458756327629,
+            0.0036450298503041267,
+            0.04222193360328674,
+            -0.008130906149744987,
+            -0.006579829845577478,
+            0.0066614653915166855,
+            -0.008449285291135311,
+            -0.013747435994446278,
+            0.026727497577667236,
+            0.005796127952635288,
+            0.015984252095222473,
+            0.016123032197356224,
+            -0.00945340283215046,
+            -0.005008344538509846,
+            -0.0018786393338814378,
+            0.02065380848944187,
+            0.0073798587545752525,
+            -0.005502239800989628,
+            -0.009714636951684952,
+            0.005881845485419035,
+            -0.00898808054625988,
+            0.0021653843577951193,
+            -0.009526874870061874,
+            -0.005824700463563204,
+            -0.008069679141044617,
+            0.016400592401623726,
+            -0.0003523086488712579,
+            0.027315273880958557,
+            0.020751770585775375,
+            -0.008563575334846973,
+            -0.0005683879135176539,
+            0.017208784818649292,
+            -0.005820618476718664,
+            0.015608727931976318,
+            -0.011192241683602333,
+            -0.007706400938332081,
+            0.006408394780009985,
+            0.024017194285988808,
+            0.012237177230417728,
+            -0.0056042843498289585,
+            0.0002885308349505067,
+            -0.0030348035506904125,
+            -0.010612628422677517,
+            0.013388238847255707,
+            0.015323002822697163,
+            -0.01947825588285923,
+            -0.004722619894891977,
+            -0.0016592437168583274,
+            -0.003300119424238801,
+            0.00945340283215046,
+            0.02305389568209648,
+            0.000171817431692034,
+            0.009184005670249462,
+            0.0012765767751261592,
+            0.03862996771931648,
+            -0.0053144777193665504,
+            0.03102152980864048,
+            -0.0010423846542835236,
+            -0.0059430720284581184,
+            -0.009094206616282463,
+            -0.020441556349396706,
+            -0.011486129835247993,
+            -0.00041481092921458185,
+            -0.005040998570621014,
+            -0.0004956811899319291,
+            -0.020327266305685043,
+            0.0026960158720612526,
+            0.014637264423072338,
+            0.007661501411348581,
+            0.017927179113030434,
+            -0.002889900468289852,
+            0.017437364906072617,
+            0.019690508022904396,
+            -0.01400050614029169,
+            -0.029862305149435997,
+            -0.0006515541463159025,
+            0.01498013362288475,
+            -0.01661284640431404,
+            0.019445601850748062,
+            0.004085862077772617,
+            0.0029531680047512054,
+            0.004779764451086521,
+            -0.006049198564141989,
+            0.018482301384210587,
+            0.02039257436990738,
+            0.01453930139541626,
+            -0.013167822733521461,
+            0.014057651162147522,
+            0.009771781973540783,
+            0.006600238848477602,
+            -0.01646590232849121,
+            -0.013535182923078537,
+            0.002555194543674588,
+            0.0009245232213288546,
+            0.024392718449234962,
+            -0.020588500425219536,
+            0.0016602640971541405,
+            0.010588138364255428,
+            0.01769859902560711,
+            -0.004302196204662323,
+            0.008906444534659386,
+            0.0008740111952647567,
+            0.007134951651096344,
+            -0.0035919665824621916,
+            -0.006457376293838024,
+            -0.006673710886389017,
+            -0.010392212308943272,
+            -0.0020959940738976,
+            -0.02070278860628605,
+            0.002198038622736931,
+            -0.002830714685842395,
+            0.018237393349409103,
+            -0.01018812321126461,
+            0.038140155375003815,
+            0.03399306535720825,
+            -0.0007025764207355678,
+            0.011102442629635334,
+            -0.01709449477493763,
+            -0.003945040516555309,
+            -0.021959977224469185,
+            0.01844964735209942,
+            0.016392430290579796,
+            0.012269831262528896,
+            0.005302232690155506,
+            0.017796561121940613,
+            -0.0006301248213276267,
+            0.014490320347249508,
+            0.0029654132667928934,
+            0.007971717044711113,
+            -0.002265388146042824,
+            0.003938918001949787,
+            -0.031103165820240974,
+            -0.017029186710715294,
+            -0.01138000376522541,
+            0.004796091932803392,
+            -0.006375740747898817,
+            -0.0035205355379730463,
+            0.0084737753495574,
+            -0.005902254022657871,
+            0.0004270562494639307,
+            -0.025992777198553085,
+            -0.0157148540019989,
+            -0.010318740271031857,
+            -0.004040962550789118,
+            -0.01169838197529316,
+            -0.0027694879099726677,
+            -0.0022082431241869926,
+            0.0029552089981734753,
+            0.020947696641087532,
+            -0.012808626517653465,
+            -0.016294466331601143,
+            0.004379750229418278,
+            -0.002263347152620554,
+            -0.014016833156347275,
+            -0.016898570582270622,
+            0.006665547378361225,
+            0.012098397128283978,
+            -0.0017470018938183784,
+            -0.013608654960989952,
+            0.011739199981093407,
+            -0.006628811359405518,
+            0.04663025587797165,
+            -0.013584164902567863,
+            -0.0078002819791436195,
+            0.003720542648807168,
+            0.004698128905147314,
+            -0.009396257810294628,
+            0.007330877240747213,
+            0.013020878657698631,
+            0.01830270327627659,
+            0.0095758568495512,
+            -0.008465612307190895,
+            -0.0024388637393712997,
+            0.007796200457960367,
+            -0.016147522255778313,
+            0.006404313258826733,
+            0.0008525818702764809,
+            -0.014400520361959934,
+            0.006110425107181072,
+            0.028980640694499016,
+            0.013151495717465878,
+            0.01288209855556488,
+            -0.022351829335093498,
+            -0.004094025585800409,
+            -0.009355440735816956,
+            -0.004943036008626223,
+            -0.010090161114931107,
+            0.027070365846157074,
+            -0.018155759200453758,
+            0.02651524543762207,
+            0.00898808054625988,
+            0.0017684312770143151,
+            -0.01176369097083807,
+            0.006343086715787649,
+            0.02832755446434021,
+            0.0028592869639396667,
+            -0.020000724121928215,
+            -0.013592327944934368,
+            0.006314514204859734,
+            0.01707816869020462,
+            -0.002647034591063857,
+            -0.005428767763078213,
+            0.019657853990793228,
+            0.01784554310142994,
+            -0.004334850702434778,
+            -0.00936360377818346,
+            0.0008337036124430597,
+            0.00928196869790554,
+            0.016963878646492958,
+            0.015020951628684998,
+            0.009086042642593384,
+            -0.002647034591063857,
+            0.008037025108933449,
+            -0.008800318464636803,
+            -0.016449574381113052,
+            0.007220669183880091,
+            -0.008432958275079727,
+            -0.016963878646492958,
+            0.01872720755636692,
+            0.015257694758474827,
+            0.0181394312530756,
+            0.0024592725094407797,
+            -0.016735298559069633,
+            0.022302847355604172,
+            0.005755309946835041,
+            0.002998067531734705,
+            0.02160078100860119,
+            -0.022221211344003677,
+            0.013012715615332127,
+            0.02037624642252922,
+            0.027609162032604218,
+            -0.01737205684185028,
+            -0.010906516574323177,
+            -0.004857318475842476,
+            -0.006853308994323015,
+            -0.00176945177372545,
+            -0.008253359235823154,
+            -0.012620864436030388,
+            0.0028735732194036245,
+            -0.030466407537460327,
+            0.018106777220964432,
+            -0.005102225113660097,
+            0.010669773444533348,
+            -0.019266001880168915,
+            0.011363675817847252,
+            -0.0035838030744343996,
+            0.011135096661746502,
+            0.0019612954929471016,
+            -0.030041903257369995,
+            -0.008571738377213478,
+            0.015388311818242073,
+            -0.0030368445441126823,
+            -0.013641309924423695,
+            0.02006603218615055,
+            0.00018942011229228228,
+            -0.022270193323493004,
+            -0.004314441699534655,
+            -0.006934944540262222,
+            0.002193956868723035,
+            -0.0029613315127789974,
+            -0.0052981507033109665,
+            -0.030188847333192825,
+            0.020033378154039383,
+            0.009559528902173042,
+            -0.03072764165699482,
+            -0.003255219664424658,
+            -0.014996460638940334,
+            -0.0025184585247188807,
+            0.008881953544914722,
+            -0.004000144544988871,
+            -0.015551582910120487,
+            0.01784554310142994,
+            0.022711025550961494,
+            -0.0023164101876318455,
+            -0.016057724133133888,
+            -0.0090697156265378,
+            -0.027494871988892555,
+            0.030825603753328323,
+            -0.0066696288995444775,
+            0.0022878379095345736,
+            -0.005322641227394342,
+            0.009388094767928123,
+            0.004228724166750908,
+            0.0157148540019989,
+            0.006081852596253157,
+            -0.002647034591063857,
+            0.026596879586577415,
+            -0.01964152604341507,
+            0.009567692875862122,
+            -1.588864870427642e-05,
+            -0.003308282932266593,
+            0.01890680566430092,
+            0.0007939062779769301,
+            0.021649762988090515,
+            -0.008571738377213478,
+            -0.0001475818717153743,
+            -0.0034388997592031956,
+            0.010898353531956673,
+            -0.0028450009413063526,
+            -0.003465431509539485,
+            -0.001531688030809164,
+            0.011118769645690918,
+            0.01013097818940878,
+            0.003804219188168645,
+            -0.01737205684185028,
+            0.003142970846965909,
+            -0.008547247387468815,
+            -0.007028825581073761,
+            0.01302904263138771,
+            -0.011192241683602333,
+            -0.0018990482203662395,
+            -0.03160930797457695,
+            0.016286302357912064,
+            0.0533243790268898,
+            0.008547247387468815,
+            -0.014433175325393677,
+            -0.0018806802108883858,
+            -0.019755816087126732,
+            0.01535565685480833,
+            0.011004479601979256,
+            -0.016816934570670128,
+            -0.011984107084572315,
+            -0.012310649268329144,
+            0.007649256382137537,
+            0.021306892856955528,
+            -0.006522684823721647,
+            -0.008579902350902557,
+            0.025372346863150597,
+            -0.011788181029260159,
+            -0.010882026515901089,
+            -0.01661284640431404,
+            -0.01635977439582348,
+            0.00241641397587955,
+            0.019902760162949562,
+            -0.0169475506991148,
+            0.021176276728510857,
+            0.00936360377818346,
+            0.0010133018950000405,
+            0.003624620847404003,
+            0.008588065393269062,
+            0.0024878450203686953,
+            0.017878197133541107,
+            0.003883813973516226,
+            -0.01439235731959343,
+            0.0021408938337117434,
+            -0.0030286810360848904,
+            0.004624656867235899,
+            3.4320437407586724e-06,
+            -0.002749078907072544,
+            -0.0065635028295218945,
+            0.003710338147357106,
+            -0.021306892856955528,
+            0.021502818912267685,
+            -0.022678371518850327,
+            -0.004245051648467779,
+            0.005555302836000919,
+            -0.005118552595376968,
+            0.013469874858856201,
+            0.018792515620589256,
+            0.01423724927008152,
+            0.008702355436980724,
+            0.01462093647569418,
+            -0.011331021785736084,
+            0.02323349379003048,
+            -0.013282112777233124,
+            0.0005816536722704768,
+            0.012939243577420712,
+            0.01755165494978428,
+            0.018792515620589256,
+            -0.006245123688131571,
+            -0.010384049266576767,
+            -0.0002946535241790116,
+            0.0129637336358428,
+            -0.022286519408226013,
+            -0.012269831262528896,
+            -0.01386172603815794,
+            0.011486129835247993,
+            -0.004743028432130814,
+            0.025453981012105942,
+            -0.00633492274209857,
+            -0.014653591439127922,
+            -0.015918942168354988,
+            0.014196431264281273,
+            -0.048752784729003906,
+            0.0030348035506904125,
+            -0.0024919267743825912,
+            0.03043375350534916,
+            -0.027021385729312897,
+            0.01738838478922844,
+            -0.0028490826953202486,
+            -0.009355440735816956,
+            0.015429128892719746,
+            -0.015992414206266403,
+            -0.0046613928861916065,
+            -0.0016806729836389422,
+            -0.007539048325270414,
+            -0.00112248957157135,
+            0.017616963014006615,
+            -0.015845470130443573,
+            0.01282495353370905,
+            -0.00989423505961895,
+            0.0027021386194974184,
+            -0.004526694305241108,
+            -0.007751300930976868,
+            0.0196088720113039,
+            0.034548189491033554,
+            -0.007053316105157137,
+            0.0014990337658673525,
+            -0.022547753527760506,
+            0.013192313723266125,
+            -0.006730855442583561,
+            0.02351105399429798,
+            -0.005432849284261465,
+            -0.016898570582270622,
+            -0.013976015150547028,
+            -0.009739127941429615,
+            0.00839214026927948,
+            -0.022433463484048843,
+            -0.011110605672001839,
+            -0.00922482367604971,
+            -0.002573562553152442,
+            -0.018694553524255753,
+            0.004865481983870268,
+            -0.002426618244498968,
+            -0.012506574392318726,
+            -0.008816645480692387,
+            0.01048201136291027,
+            -0.009347276762127876,
+            0.003077662317082286,
+            -0.0011929002357646823,
+            -0.011453475803136826,
+            -0.02369065210223198,
+            0.0158944521099329,
+            0.0001974561164388433,
+            -0.010359558276832104,
+            0.008408467285335064,
+            -0.0018276170594617724,
+            0.001211268245242536,
+            0.016963878646492958,
+            -0.0020664012990891933,
+            0.003467472270131111,
+            0.023478399962186813,
+            0.004057289566844702,
+            -0.010971825569868088,
+            -0.00025906547671183944,
+            0.013535182923078537,
+            -0.005118552595376968,
+            -0.010008525103330612,
+            1.6646010408294387e-05,
+            -0.02984597720205784,
+            -0.008351322263479233,
+            0.006094098091125488,
+            0.018629245460033417,
+            -0.010620792396366596,
+            -0.015853634104132652,
+            0.026319319382309914,
+            -0.006359413731843233,
+            0.019070077687501907,
+            0.002889900468289852,
+            0.0025694805663079023,
+            0.01724144071340561,
+            0.013461710885167122,
+            0.00786150898784399,
+            -0.013927034102380276,
+            -0.02429475635290146,
+            0.0016398552106693387,
+            -0.018090449273586273,
+            0.010579974390566349,
+            -0.011028970591723919,
+            0.0032756286673247814,
+            -0.02398454025387764,
+            -0.012237177230417728,
+            0.004934872500598431,
+            -0.020343592390418053,
+            -0.00040588201954960823,
+            -0.0056940834037959576,
+            0.022237539291381836,
+            -0.005310396198183298,
+            0.008669701404869556,
+            0.0018255761824548244,
+            -0.012237177230417728,
+            0.0009877908742055297,
+            0.009469729848206043,
+            -0.029192892834544182,
+            -0.0007127808639779687,
+            -0.003138889092952013,
+            0.0011418780777603388,
+            0.005351213738322258,
+            0.009755454957485199,
+            0.007322713732719421,
+            0.0027062203735113144,
+            0.007253323681652546,
+            0.010449357330799103,
+            -0.00022041612828616053,
+            0.0197884701192379,
+            0.018172085285186768,
+            -0.007351286243647337,
+            0.008710519410669804,
+            0.024817224591970444,
+            -0.013159659691154957,
+            -0.008718682453036308,
+            0.023266147822141647,
+            -0.006461458280682564,
+            -0.0011122850701212883,
+            -0.01483318954706192,
+            0.00815947912633419,
+            0.0017367975087836385,
+            0.028131630271673203,
+            0.019657853990793228,
+            0.005820618476718664,
+            0.000639819074422121,
+            0.015788326039910316,
+            -0.013020878657698631,
+            0.0035286990460008383,
+            0.015461783856153488,
+            -0.003940958995372057,
+            -0.019821124151349068,
+            0.013878053054213524,
+            0.009502384811639786,
+            -0.003228688146919012,
+            0.01551892887800932,
+            0.011412657797336578,
+            -0.030841931700706482,
+            -0.0011898389784619212,
+            0.010498338378965855,
+            -0.0021449755877256393,
+            -0.008751336485147476,
+            -0.00014273475972004235,
+            -0.012384121306240559,
+            0.021388528868556023,
+            -0.004408322740346193,
+            0.012277995236217976,
+            -0.00815947912633419,
+            -0.017127150669693947,
+            -0.011559601873159409,
+            0.007914572022855282,
+            -0.010547320358455181,
+            -0.0006714528426527977,
+            -0.004022594541311264,
+            -0.005779800936579704,
+            0.017747581005096436,
+            0.004706292413175106,
+            -0.0008964610169641674,
+            -0.0018398624379187822,
+            0.007983962073922157,
+            -0.006979844067245722,
+            -0.008726846426725388,
+            0.004816500470042229,
+            -0.012833117507398129,
+            0.003981776535511017,
+            0.005543057806789875,
+            0.01506176870316267,
+            -0.014857680536806583,
+            -0.021159948781132698,
+            0.007020662073045969,
+            0.02695607766509056,
+            -0.02516009286046028,
+            -0.005232842173427343,
+            0.006245123688131571,
+            0.002093953313305974,
+            0.006914535537362099,
+            -0.006934944540262222,
+            -0.007208424154669046,
+            -0.03657275065779686,
+            -0.0020745648071169853,
+            0.0009005427709780633,
+            0.008963589556515217,
+            -0.008049271069467068,
+            -0.0136494729667902,
+            -0.011135096661746502,
+            0.002410291228443384,
+            0.007134951651096344,
+            -0.02204161323606968,
+            0.0049593630246818066,
+            0.009608510881662369,
+            -0.019559891894459724,
+            0.025127438828349113,
+            -0.010890189558267593,
+            -6.964537897147238e-05,
+            -0.00995954405516386,
+            0.021633435040712357,
+            -0.013486201874911785,
+            -0.018368011340498924,
+            -0.0006898208521306515,
+            0.008355404250323772,
+            0.011926962062716484,
+            -0.0017663904000073671,
+            -0.011371839791536331,
+            0.0003752686607185751,
+            0.01646590232849121,
+            0.019135385751724243,
+            0.002744997153058648,
+            -0.0024837632663547993,
+            -0.01019628718495369,
+            -0.009592183865606785,
+            -0.0073961857706308365,
+            -0.0007449248805642128,
+            0.002091912319883704,
+            0.02035992033779621,
+            0.013469874858856201,
+            0.0024837632663547993,
+            -0.015135240741074085,
+            -0.005257333163172007,
+            0.008939098566770554,
+            0.009412584826350212,
+            -0.011690218932926655,
+            -0.00898808054625988,
+            0.002659279853105545,
+            0.02249877341091633,
+            0.017633290961384773,
+            -0.0014000505907461047,
+            0.0032878739293664694,
+            -0.0016592437168583274,
+            0.012506574392318726,
+            -0.007555375341325998,
+            -0.010269759222865105,
+            -0.006943108048290014,
+            -0.014645427465438843,
+            -0.010800390504300594,
+            -0.0023062059190124273,
+            -0.010179960168898106,
+            0.022629389539361,
+            0.007988044060766697,
+            0.009779945947229862,
+            0.008265605196356773,
+            0.010269759222865105,
+            0.002087830565869808,
+            0.025715215131640434,
+            -0.004424649756401777,
+            -0.011984107084572315,
+            0.007236996199935675,
+            -0.022368155419826508,
+            -0.021535472944378853,
+            0.008257441222667694,
+            0.006367577239871025,
+            0.03595232218503952,
+            -0.026890767738223076,
+            0.0052981507033109665,
+            -0.009779945947229862,
+            -0.010400376282632351,
+            0.007996207103133202,
+            -0.0015439334092661738,
+            -0.004245051648467779,
+            0.022368155419826508,
+            0.00726556871086359,
+            0.0035797213204205036,
+            0.0032817514147609472,
+            0.019837452098727226,
+            -0.005685919895768166,
+            0.01573934406042099,
+            0.0259274672716856,
+            0.024621298536658287,
+            0.009388094767928123,
+            -0.009469729848206043,
+            0.02785406820476055,
+            0.0002895512734539807,
+            -0.0034327772445976734,
+            0.007220669183880091,
+            -0.00305113079957664,
+            -0.0009556467994116247,
+            -0.010033016093075275,
+            0.021764053031802177,
+            -0.001455154619179666,
+            0.004359341226518154,
+            0.02081707864999771,
+            -7.774515688652173e-05,
+            0.006220633164048195,
+            0.01057181041687727,
+            0.002181711606681347,
+            0.003396041225641966,
+            0.010775899514555931,
+            -0.039315707981586456,
+            -0.0006571666453965008,
+            -0.025600925087928772,
+            -0.0018286375561729074,
+            0.0007357408758252859,
+            -0.01266984548419714,
+            0.013600491918623447,
+            -0.002015379024669528,
+            -0.010751409456133842,
+            -0.00029337796149775386,
+            0.021649762988090515,
+            0.005416522268205881,
+            0.004071575589478016,
+            -0.004469549283385277,
+            0.018792515620589256,
+            -0.006955353543162346,
+            -0.017861871048808098,
+            -0.012759645469486713,
+            -0.012367794290184975,
+            -0.01918436773121357,
+            -0.01214737817645073,
+            -0.0024551907554268837,
+            -0.002667443361133337,
+            0.004738946910947561,
+            0.002481722505763173,
+            -0.003228688146919012,
+            -0.02070278860628605,
+            0.015037278644740582,
+            -0.01430255826562643,
+            0.012873934581875801,
+            -0.011053460650146008,
+            0.005963481031358242,
+            -0.004122598096728325,
+            0.012726990506052971,
+            -0.021404854953289032,
+            0.02070278860628605,
+            -0.01536382082849741,
+            0.0038532004691660404,
+            0.019592545926570892,
+            0.006551257334649563,
+            -0.008118661120533943,
+            -0.013404566794633865,
+            -0.022074267268180847,
+            0.021633435040712357,
+            -0.011633073911070824,
+            -0.018237393349409103,
+            -0.004608329851180315,
+            0.009649328887462616,
+            -0.017470018938183784,
+            -0.007367613259702921,
+            0.01707816869020462,
+            -0.0018949664663523436,
+            0.012980061583220959,
+            0.023004913702607155,
+            -0.0052981507033109665,
+            0.0036572751123458147,
+            -0.009167678654193878,
+            0.0010286086471751332,
+            -0.02470293454825878,
+            0.013527019880712032,
+            0.008971752598881721,
+            0.006269614677876234,
+            -0.013935198076069355,
+            -0.013306603766977787,
+            0.0036531933583319187,
+            0.004498121794313192,
+            0.011437147855758667,
+            0.009420748800039291,
+            0.004220560658723116,
+            -0.013918871060013771,
+            -0.027788760140538216,
+            -0.027070365846157074,
+            0.03595232218503952,
+            -0.0005002731923013926,
+            -0.020947696641087532,
+            0.03399306535720825,
+            0.016114868223667145,
+            0.0022980421781539917,
+            -0.012449429370462894,
+            -0.004314441699534655,
+            0.024621298536658287,
+            0.016400592401623726,
+            0.010465684346854687,
+            -0.0004742518358398229,
+            0.005571629852056503,
+            0.0016469983384013176,
+            0.03012353740632534,
+            0.015323002822697163,
+            0.0009459525463171303,
+            -0.038597315549850464,
+            0.011943289078772068,
+            -0.007547211833298206,
+            0.0013408648082986474,
+            0.006702283397316933,
+            0.017780235037207603,
+            0.011935125105082989,
+            0.015567909926176071,
+            0.014865843579173088,
+            -0.003065417055040598,
+            0.0029511270113289356,
+            0.0006163488142192364,
+            -0.009624837897717953,
+            -0.001767410896718502,
+            -0.008881953544914722,
+            -0.005714492406696081,
+            -0.002920513739809394,
+            0.005592038854956627,
+            0.007657419890165329,
+            -0.003983817528933287,
+            0.028049994260072708,
+            -0.0006433906382881105,
+            -0.017600636929273605,
+            -0.009061552584171295,
+            -0.03611559048295021,
+            0.00434301421046257,
+            -0.030939893797039986,
+            -0.011257549747824669,
+            0.0033674687147140503,
+            -0.020457882434129715,
+            -0.024768242612481117,
+            0.0010306495241820812,
+            0.014335212297737598,
+            0.007392104249447584,
+            0.0033327736891806126,
+            -0.01004934310913086,
+            0.003816464450210333,
+            -0.001855169073678553,
+            -0.006461458280682564,
+            0.028833696618676186,
+            0.019559891894459724,
+            0.020457882434129715,
+            -0.01190247107297182,
+            0.019298657774925232,
+            0.005845109466463327,
+            0.02035992033779621,
+            0.010465684346854687,
+            -0.0048695639707148075,
+            0.0074859848245978355,
+            0.009567692875862122,
+            -0.006155324634164572,
+            -0.021519144997000694,
+            0.013192313723266125,
+            -0.0010837126756086946,
+            -0.0052083516493439674,
+            -0.016269976273179054,
+            -0.003534821793437004,
+            -0.015812816098332405,
+            0.010547320358455181,
+            -0.017159804701805115,
+            0.012498411349952221,
+            -0.000802069844212383,
+            0.00168475485406816,
+            -0.015469946898519993,
+            -0.013951525092124939,
+            -0.0012439225101843476,
+            -0.01244126632809639,
+            -0.008971752598881721,
+            0.006163488142192364,
+            -0.007539048325270414,
+            -0.008318668231368065,
+            -0.00535937724635005,
+            -0.011437147855758667,
+            -0.010882026515901089,
+            0.004367504734545946,
+            -0.0010653446661308408,
+            0.020931368693709373,
+            0.00011671340325847268,
+            -0.005824700463563204,
+            0.01190247107297182,
+            0.02787039615213871,
+            0.0016704685986042023,
+            0.004583839327096939,
+            -0.014261740259826183,
+            -0.011118769645690918,
+            -0.02279266156256199,
+            0.016572028398513794,
+            0.0049675265327095985,
+            -0.0344502255320549,
+            0.01438419334590435,
+            0.00824927818030119,
+            0.0015102586476132274,
+            -0.006698201410472393,
+            -0.005334886722266674,
+            -0.005608365871012211,
+            0.006306350696831942,
+            -0.0022837561555206776,
+            0.0005107327597215772,
+            0.0035919665824621916,
+            0.034678805619478226,
+            -0.0040674940682947636,
+            -0.007228832691907883,
+            0.00981259997934103,
+            0.0005130287609063089,
+            0.020196648314595222,
+            -0.005518566817045212,
+            0.0037572786677628756,
+            -0.019282329827547073,
+            -0.016563864424824715,
+            0.0018429238116368651,
+            -0.01722511276602745,
+            0.023282473906874657,
+            -0.005392031744122505,
+            -0.01347803883254528,
+            -0.006906372029334307,
+            -0.020115012302994728,
+            0.012637191452085972,
+            0.02310287579894066,
+            0.007016580086201429,
+            0.01033506728708744,
+            0.012131051160395145,
+            -0.02444170042872429,
+            -0.020457882434129715,
+            0.012171869166195393,
+            0.0013571919407695532,
+            -0.008449285291135311,
+            -0.005404277238994837,
+            -0.001783737912774086,
+            0.007783954963088036,
+            -0.0032633834052830935,
+            0.007881917990744114,
+            0.06491663306951523,
+            -0.01093100756406784,
+            -0.002987863030284643,
+            -0.02966637909412384,
+            -0.004396077245473862,
+            -0.006628811359405518,
+            0.0056287748739123344,
+            0.009151351638138294,
+            -0.023135529831051826,
+            0.009232986718416214,
+            -0.030025575309991837,
+            -0.010628955438733101,
+            0.00824111420661211,
+            0.0028490826953202486,
+            -0.01220452319830656,
+            -0.0009592183632776141,
+            -0.013665799982845783,
+            0.0006107363733462989,
+            -0.009412584826350212,
+            -0.00576347392052412,
+            0.012718827463686466,
+            -0.02321716584265232,
+            0.006298186723142862,
+            0.007592111360281706,
+            -0.006514521315693855,
+            -0.013722945004701614,
+            -0.017861871048808098,
+            -0.011967780068516731,
+            -0.013314766809344292,
+            0.002996026771143079,
+            0.01190247107297182,
+            -0.02607441134750843,
+            0.012922916561365128,
+            -0.0037389106582850218,
+            0.00945340283215046,
+            0.004404240753501654,
+            0.013469874858856201,
+            0.018074123188853264,
+            0.009355440735816956,
+            0.0021429345943033695,
+            -0.008024780079722404,
+            0.012702500447630882,
+            -0.01993541419506073,
+            -0.01635161228477955,
+            0.013837235048413277,
+            0.012016761116683483,
+            0.003061335301026702,
+            0.009828926995396614,
+            0.0020959940738976,
+            0.011886144056916237,
+            0.010775899514555931,
+            -0.019739490002393723,
+            -0.0006648199632763863,
+            0.0007326795603148639,
+            0.03280118480324745,
+            -0.0005938990507274866,
+            0.008857463486492634,
+            0.004420568235218525,
+            0.013159659691154957,
+            -0.012784135527908802,
+            -0.01769859902560711,
+            0.0008087027235887945,
+            -0.01057181041687727,
+            0.021257910877466202,
+            -0.008192133158445358,
+            0.006216551177203655,
+            -0.018612917512655258,
+            -0.027086693793535233,
+            0.006392067763954401,
+            -0.016261812299489975,
+            -0.018221067264676094,
+            0.01317598670721054,
+            0.0077268099412322044,
+            0.0017572063952684402,
+            -0.005098143592476845,
+            -0.020457882434129715,
+            -0.009837090037763119,
+            0.028441844508051872,
+            0.011102442629635334,
+            -0.004120557103306055,
+            0.009584019891917706,
+            -0.009094206616282463,
+            -0.012490247376263142,
+            -0.0017541450215503573,
+            0.0036715613678097725,
+            0.008261523209512234,
+            0.005645101889967918,
+            -0.01386172603815794,
+            -0.00029873527819290757,
+            -0.012400448322296143,
+            -0.021225256845355034,
+            -0.008906444534659386,
+            0.016751626506447792,
+            0.007820690982043743,
+            -0.0038960592355579138,
+            0.015812816098332405,
+            -0.004404240753501654,
+            0.002189875114709139,
+            0.014294394291937351,
+            0.005334886722266674,
+            -0.004285869188606739,
+            -0.0095758568495512,
+            -0.004612411838024855,
+            0.004043003544211388,
+            -0.0025960123166441917,
+            0.006457376293838024,
+            0.017192458733916283,
+            0.0072900597006082535,
+            -0.013755599036812782,
+            0.013135168701410294,
+            0.0007791098323650658,
+            0.0049675265327095985,
+            -0.03699725493788719,
+            -0.003247056156396866,
+            0.015796490013599396,
+            -0.013020878657698631,
+            0.02501314878463745,
+            0.005400195252150297,
+            0.022743679583072662,
+            0.015233203768730164,
+            0.008669701404869556,
+            -0.001542912912555039,
+            0.00464098434895277,
+            0.02759283408522606,
+            -0.003151134354993701,
+            -0.008637047372758389,
+            -0.0020602785516530275,
+            0.009241150692105293,
+            0.0002063850115519017,
+            0.003000108525156975,
+            0.008281932212412357,
+            -0.03562577813863754,
+            0.011706545948982239,
+            -0.019576217979192734,
+            0.0024694770108908415,
+            0.003459308762103319,
+            -0.007183933164924383,
+            -0.014865843579173088,
+            -0.005857354495674372,
+            0.022841641679406166,
+            -0.003224606392905116,
+            0.021078312769532204,
+            0.002885818714275956,
+            0.016106704249978065,
+            -0.011575928889214993,
+            -0.009208496659994125,
+            -0.0004130251472815871,
+            -0.005351213738322258,
+            -0.007539048325270414,
+            0.010596301406621933,
+            0.0034409407526254654,
+            0.005534893833100796,
+            0.002265388146042824,
+            0.0036654386203736067,
+            -0.0036878883838653564,
+            0.02773977816104889,
+            0.003563394071534276,
+            0.007996207103133202,
+            -0.011567764915525913,
+            0.014082142151892185,
+            -0.017029186710715294,
+            0.0070778070949018,
+            0.005575711838901043,
+            -0.026156047359108925,
+            0.016849588602781296,
+            0.004775682929903269,
+            0.006628811359405518,
+            0.013812744058668613,
+            0.01618834026157856,
+            -0.010408539324998856,
+            -0.014816862531006336,
+            0.024049850180745125,
+            -0.0032041973900049925,
+            -0.015225040726363659,
+            -0.006775755435228348,
+            4.438935866346583e-05,
+            0.006559420842677355,
+            0.00373074715025723,
+            0.0013020879123359919,
+            -0.0006204306264407933,
+            -0.010743245482444763,
+            0.004645065870136023,
+            -0.01934763789176941,
+            -0.0007163524278439581,
+            -0.016449574381113052,
+            -0.015633217990398407,
+            0.018678225576877594,
+            -0.01830270327627659,
+            0.009633000940084457,
+            0.0012275953777134418,
+            0.028588788583874702,
+            -0.011706545948982239,
+            -0.012155541218817234,
+            0.0015102586476132274,
+            0.010849371552467346,
+            0.0067512644454836845,
+            -0.010033016093075275,
+            -0.007583947852253914,
+            -0.014180104248225689,
+            -0.010163633152842522,
+            0.015502601861953735,
+            -0.014498483389616013,
+            -0.014343376271426678,
+            0.008914608508348465
+          ],
+          "embedding": [
+            -0.006359657085039302,
+            -0.024594786241399234,
+            -0.02553538924848095,
+            0.005360188465088291,
+            0.012801870977589839,
+            0.021680505862506104,
+            -0.016668048966001273,
+            0.06860852474581301,
+            -0.007122272841157724,
+            0.03471775000705488,
+            -0.009004297848861675,
+            0.01059974144678391,
+            0.02565604599426148,
+            -0.022544127646041287,
+            0.010616674976540785,
+            0.013658194823359737,
+            0.016400329743363844,
+            0.02858546964917478,
+            0.0067612676434693686,
+            -0.04174737193606879,
+            0.03184155110488616,
+            -0.02223105127585514,
+            -0.05506239667974977,
+            -0.0015307551410555752,
+            -0.013169788751379287,
+            0.017681749722384007,
+            0.01682706317229353,
+            0.019652295823816928,
+            -0.05623980369271269,
+            0.030698582851533197,
+            0.02286327124296613,
+            0.016019032539689178,
+            0.005508113353872315,
+            8.58985091215149e-05,
+            -0.019206292853495906,
+            -0.018439290483495144,
+            0.03475023176950151,
+            0.01018612942141759,
+            -0.0252551158857426,
+            0.0334562130666827,
+            0.020462153537485785,
+            0.02260508249714211,
+            -0.03323524989392951,
+            -0.002094251009530707,
+            0.015177442347686773,
+            -0.00018698464350038317,
+            -0.0006788648344273403,
+            0.0007777616893581039,
+            0.013352398615059647,
+            -0.030751111528744813,
+            -0.0317733818445192,
+            -0.006729064081710058,
+            0.009413204899986514,
+            -0.005712894407943592,
+            -0.038961826880527066,
+            -0.002827692206530345,
+            0.017863300537973137,
+            0.07918455388563496,
+            0.048967158919816366,
+            0.018002566325350088,
+            0.040264962620950695,
+            -0.019715100359659787,
+            -0.02211350500879408,
+            0.004918148608475574,
+            0.01062585305866606,
+            -0.014478993521696027,
+            0.0034983131226364672,
+            0.000766228629514787,
+            0.009667268771502067,
+            -0.01558439890697323,
+            -0.04984718709870083,
+            0.029613955432821887,
+            -0.009686854114630624,
+            -0.015279319238124568,
+            0.007460629676325322,
+            -0.006235979350809034,
+            0.012514387473717085,
+            0.02800435263741497,
+            -0.008688925946672778,
+            -0.01236301543374469,
+            0.029755705020188972,
+            0.01158037219446542,
+            -0.005242392206024511,
+            0.009982679027728814,
+            0.052414361714483264,
+            -0.018788500381726227,
+            -0.020335831196027204,
+            -0.08583959943356621,
+            0.029309228216812346,
+            -0.0249285282010673,
+            -0.018929396555426307,
+            0.007710663193334574,
+            0.03096798035958051,
+            -0.017089977064591828,
+            -0.028015107605405817,
+            0.06740703271128606,
+            0.009398795412354399,
+            0.01118769022826835,
+            -0.03041417460855722,
+            0.014776002883150835,
+            0.004319194067673161,
+            0.022507636639041203,
+            0.0053562788646786845,
+            0.007548072258943124,
+            0.011123374308563128,
+            -0.006924362115585165,
+            -0.011118507147092024,
+            0.009563232643295304,
+            0.00657178473678069,
+            0.08313027447380966,
+            -0.004239370156003264,
+            -0.00930406163946907,
+            -0.039202672821898946,
+            0.009761029930556298,
+            -0.02827698465424993,
+            -0.037914533246900244,
+            -0.04496050911855815,
+            -0.017711307900236353,
+            0.01934435542287117,
+            0.004335173027873153,
+            0.03800509969714242,
+            0.031198127357716314,
+            0.031311207941680276,
+            -0.012184553894588813,
+            -0.058985863670856876,
+            0.041442017057707635,
+            -0.012456189292721711,
+            -0.0040966351720394135,
+            0.010849179519959421,
+            0.005243123049347636,
+            0.0011495799963862647,
+            0.003534183676934487,
+            -0.03139340512003631,
+            -0.023876261883905306,
+            0.008382957569752911,
+            -0.004778283217011976,
+            0.03703067589888386,
+            0.06888630688839406,
+            -0.03993532296478411,
+            -0.009472167678222372,
+            0.025246872942105057,
+            -0.01524567912700368,
+            0.01595917939878582,
+            -0.003736723164442467,
+            -0.030836797784967753,
+            0.0026246446163368533,
+            0.008894300082148884,
+            0.08342856279154155,
+            -0.0007476580521704578,
+            0.025149116238550588,
+            0.004972542633631468,
+            0.034495632546556576,
+            -0.009173329764686424,
+            0.014009381543772768,
+            0.004427261668151028,
+            -0.013426138607902904,
+            -0.022234568880321808,
+            0.01976282364689966,
+            -0.03592433789490043,
+            -0.010918278220831188,
+            0.013527180289641174,
+            -0.013023113052242677,
+            -0.008304770302495363,
+            0.003677063799167178,
+            0.0013981822663219421,
+            0.047001163242600175,
+            0.01828332714887986,
+            0.002483375662343689,
+            -0.06820498187523893,
+            0.05295312038995555,
+            0.025038753504342162,
+            0.025286870198436735,
+            -0.0029544808216316115,
+            0.029982752196635428,
+            0.0005600481523874767,
+            0.0128290258663539,
+            0.009023805227424919,
+            -0.0019219166367750596,
+            -0.04482843936478531,
+            0.04523479691409549,
+            -0.004678574027241186,
+            0.010486850804218904,
+            0.006736229865018014,
+            0.02134813656476443,
+            0.028083972673690036,
+            0.016857413440125445,
+            0.0189082447820146,
+            0.03843690613820022,
+            -0.011922027011229613,
+            0.016193433524453224,
+            0.03828207911467065,
+            -0.0013055128404331705,
+            0.043198163443700956,
+            0.024025562893260542,
+            -0.001257851946749882,
+            -0.011726713433348844,
+            -0.021907984099443396,
+            0.02752198420597673,
+            0.004431866336999851,
+            0.015798806714413912,
+            -0.012910782191382367,
+            -0.03180549180206364,
+            0.004550247372715741,
+            -0.022778272841118514,
+            0.031242218240017532,
+            -0.006106984949693483,
+            -0.006259993190497421,
+            0.02187449047033339,
+            0.009334146847221406,
+            -0.009381278960016145,
+            -0.020888853823931134,
+            -0.007502374908776156,
+            -0.010390018386728482,
+            -0.04316693435221676,
+            -0.006274513730720055,
+            0.019152443937064258,
+            -0.01668984824845791,
+            0.018573194036505578,
+            0.0035035262183249925,
+            -0.019779911381789077,
+            -0.0027495798165085993,
+            0.02425481236315913,
+            0.006857723545270935,
+            -0.014733535699087731,
+            -0.026326146471941727,
+            -0.019906660231056833,
+            -0.001033294559612807,
+            0.0159154063026895,
+            0.035649534442431415,
+            -0.026210592096240003,
+            0.014397127229517285,
+            0.005114311681443673,
+            -0.0033104236005207077,
+            0.00324988235167305,
+            0.03764004154553922,
+            -0.017822437760737605,
+            -0.008579701063708334,
+            -0.00032410984200965487,
+            0.018450351071676443,
+            0.017283379764846195,
+            0.006384411611332382,
+            -0.005115090309123598,
+            -0.030590047062583125,
+            -0.011776211359193503,
+            0.01183623914060493,
+            0.0016564847796208331,
+            -0.009530766634361575,
+            -0.004810598258154945,
+            0.03876242515526699,
+            0.027076785774401667,
+            0.005324763136165989,
+            -0.03416412480913939,
+            -0.007943016733586697,
+            0.002114307650456928,
+            -0.02961449347300628,
+            0.05142732770518828,
+            0.05566026433842563,
+            -0.0030729309664974704,
+            -0.00938186948393074,
+            -0.04547908981236149,
+            0.0651216359692231,
+            -0.01813722445944094,
+            -0.0482650853560993,
+            0.044045155892545554,
+            -0.014272653003441165,
+            0.013740652306746851,
+            0.04563732439184395,
+            0.02126513340521316,
+            -0.0054845632439384905,
+            -0.011635625101262898,
+            0.030992082069685376,
+            -0.010051935721068779,
+            -0.050967258971854425,
+            -0.009827942095033494,
+            0.013373878556654464,
+            -0.013017627321534534,
+            -0.009361001542607812,
+            -0.01925867665425919,
+            0.020329052915436022,
+            0.023020741670660188,
+            0.019387043947695172,
+            0.005628632318152202,
+            0.018451555161952665,
+            0.016226845400848455,
+            -0.01384033112836216,
+            0.004762589697426251,
+            -0.012718476561725138,
+            -0.024222765127494463,
+            -0.037487144314010595,
+            -0.019971524166659278,
+            -0.015034588610516958,
+            0.009483936496215338,
+            -0.014301924283553591,
+            -0.0024139665139412125,
+            0.013361045353529458,
+            -0.0005870798721107379,
+            -0.011192319340879789,
+            0.013927695434391159,
+            0.021337539669342293,
+            -0.027300952495815437,
+            -0.04765670695199488,
+            -0.00561706224251562,
+            0.0082776639906139,
+            -0.0030984973749633724,
+            -0.026269016057688548,
+            0.007040524859505573,
+            0.0017859008072463442,
+            0.020091542517503808,
+            -0.018716761130867725,
+            0.0035283051960672963,
+            0.008519383548064053,
+            -0.015247417050336866,
+            -0.016917626868166634,
+            -0.018580550690103854,
+            -0.024697180580294884,
+            0.034202028316737125,
+            0.0204395003463227,
+            0.030392895534267717,
+            -0.04527669147753553,
+            -0.018359428092075348,
+            0.05494107499558624,
+            0.015234754907198422,
+            -0.05066008106884831,
+            0.031139824660661085,
+            0.005016615710206006,
+            0.0019000453624929544,
+            -0.006094364589229031,
+            0.0637324238017742,
+            -0.02711381489292883,
+            -0.02187418906973867,
+            0.004601744864532875,
+            -0.04141121277496072,
+            -0.026689585608019588,
+            -0.005568100750837504,
+            0.01291019036002459,
+            -0.031853041426355644,
+            -0.02815800259372657,
+            -0.007964861641790374,
+            -0.010933515363060793,
+            0.013726639077745378,
+            -0.023956800669647334,
+            -0.015672058590455337,
+            -0.023642025374517057,
+            -0.01288435863544158,
+            -0.0042516881552192385,
+            -0.011162214160460695,
+            -0.02412733256710621,
+            0.010088528028211243,
+            0.03310175458095482,
+            0.023064360903748363,
+            0.001996159684818467,
+            0.009478229572933101,
+            0.014478772877286076,
+            0.00371783700962415,
+            0.007864103610320116,
+            -0.048511515124609,
+            -0.031398754206534776,
+            0.020227872561659397,
+            0.012159811587094092,
+            0.0172163448580163,
+            0.009578952796095119,
+            -0.04154545961112031,
+            -0.05095080393682759,
+            0.02968486144855133,
+            -0.03932395023720674,
+            -0.02063219739061335,
+            0.007512423249865194,
+            -0.011911709468952548,
+            -0.00810951734088105,
+            -0.0008079327452253181,
+            -0.01600701450676789,
+            0.023984909975645632,
+            -0.02418072541081482,
+            0.0033984265728439398,
+            -0.0104343782811971,
+            0.01588977364765743,
+            0.05109529246167992,
+            -0.027800269136366997,
+            -0.027889034481533438,
+            0.03706813210665186,
+            -0.02058120652429308,
+            0.030801396468451574,
+            0.03246127847621613,
+            0.008355517405358826,
+            -0.008605467586247932,
+            0.017693344563968573,
+            0.014499487024471321,
+            0.012431277080478217,
+            -0.040103870071440256,
+            0.017328848567062113,
+            0.02723900253727976,
+            -0.054909074068976875,
+            -0.008547596270650966,
+            -0.00777964838164648,
+            0.027134666572242665,
+            0.02765036594970255,
+            -0.002207173084941375,
+            -0.016838740117653725,
+            -0.028289233739722936,
+            -0.019433376533829132,
+            -0.011053380458946764,
+            0.03247641283550457,
+            0.0007514989009289282,
+            0.007919574219000797,
+            0.001316042436488591,
+            -0.0017934342041762042,
+            0.01924145734197619,
+            0.015260199553572459,
+            0.048259602293134723,
+            -0.03383050622747848,
+            0.0021712225589426266,
+            -0.03508567715696004,
+            -0.04578731401250471,
+            -0.008768228935292693,
+            0.010203318819548585,
+            0.024135236371743855,
+            -0.007467140460133838,
+            0.0028907962385387818,
+            -0.007582083854422456,
+            -0.005384477718912037,
+            -0.01867978482547249,
+            0.01641191108381631,
+            0.037168089880543244,
+            0.0031992495663980464,
+            -0.008533890994695162,
+            0.000778843942291118,
+            -0.004620258538699103,
+            0.010572690335163528,
+            0.033233014627100904,
+            -0.030497362482602697,
+            0.02974571152880676,
+            -0.004274442492343003,
+            0.025341347125910928,
+            -0.04307748954707389,
+            -0.018069609916024403,
+            -0.018491122727274337,
+            -0.02069307802429807,
+            0.05372967226264594,
+            -0.030768978301976772,
+            -0.03662045886993282,
+            -0.003205991669059688,
+            0.016815511161004534,
+            -0.0014152644009276566,
+            -0.0016015439520090104,
+            -0.03984716970293867,
+            -0.010828260334550217,
+            -0.05894985435251326,
+            -0.01638035185618134,
+            -0.021819442096404948,
+            -0.005315102328745982,
+            -0.06251579337157112,
+            0.0033713876515840233,
+            -0.07815974854883057,
+            0.001484860071649606,
+            0.04539427129131382,
+            -0.023530943197353476,
+            0.05142250021536681,
+            -0.024746600229063383,
+            -0.0005885740455257141,
+            -0.044698536284421976,
+            0.0026481001040031785,
+            0.02389178603305878,
+            0.051387657872255674,
+            -0.002949198236157656,
+            0.010674377196609455,
+            0.013322075519401089,
+            -0.026245090721950647,
+            -0.01036724558050635,
+            -0.005321884180696542,
+            0.005958798671856245,
+            0.013755178085972732,
+            -0.0074734329537091675,
+            0.038368016046563674,
+            0.010941421833174982,
+            0.03443115781919206,
+            0.034623617136788874,
+            -0.004361926531008214,
+            -0.03348681614127998,
+            -0.00632870804659212,
+            0.024433331840566692,
+            0.0321998807702602,
+            -0.02899983280320611,
+            0.005958471722097962,
+            -0.015602447981516634,
+            0.015633217539747047,
+            0.006321931116948646,
+            -0.011827541320878442,
+            0.02372204564999124,
+            0.015673426811623124,
+            0.006103300725621424,
+            -0.005165006670545588,
+            0.013589526056189892,
+            -0.025411864292353578,
+            0.03041791947375232,
+            -0.004456865298584878,
+            -0.04276058054583262,
+            -0.0024736078733106825,
+            -0.020059309867945227,
+            0.024883721794000956,
+            0.015578237413840115,
+            -0.00536523508503184,
+            -0.005290645798641171,
+            0.008242987948511412,
+            0.020469675635821673,
+            0.006668354647245447,
+            0.00017729517736814364,
+            0.02373604785121342,
+            0.029915687122900485,
+            -0.02432635861789771,
+            -0.029540533943782886,
+            0.008151297444985344,
+            -0.005730733861939129,
+            0.011318816579801282,
+            -0.02644855229221747,
+            0.00033932872948409137,
+            0.0035505684154028996,
+            -0.03570104343808711,
+            0.03408366016010711,
+            -0.009199700348630527,
+            0.005857598530788056,
+            -0.009538177736472163,
+            -0.0017768806148210118,
+            0.0027410153099351694,
+            0.03898108970435115,
+            -0.007367623392751247,
+            0.009827254226889936,
+            0.015465895035342481,
+            0.000981604396724216,
+            0.05143083558797524,
+            -0.027350877485564314,
+            -0.02183036962301852,
+            0.01885468378168636,
+            -0.007285239007489837,
+            -0.016846837665812908,
+            -0.020546383007536074,
+            -0.011863943814684079,
+            -0.016293192750336218,
+            0.01976516340568883,
+            -0.02387328798681798,
+            -0.011892336568452852,
+            0.007170651391639504,
+            0.0013725014050721622,
+            -0.009809669068563384,
+            0.0016617183704652932,
+            -0.006307590349798603,
+            0.02037693835615797,
+            0.031753656615094446,
+            -0.024892675105483313,
+            0.03577919637821846,
+            0.03178303188315581,
+            0.017260898395251155,
+            -0.02102289690701718,
+            -0.009661035772073294,
+            -0.00048542689442823136,
+            -0.014694631967051945,
+            -0.0326379186158631,
+            0.025443206465204822,
+            0.017194626456982255,
+            -0.011381421003142453,
+            -0.03680783761649673,
+            0.0017216115811536721,
+            0.022819859205829785,
+            -0.013600989058799166,
+            0.028141657788496834,
+            -0.01869047298692401,
+            -0.01366833755869951,
+            -0.00782959271979894,
+            0.031227914441058244,
+            -0.05252392878125891,
+            0.016957842026348195,
+            0.00020880696574616416,
+            0.001154370575234767,
+            0.010892067708194324,
+            -0.008368819756628046,
+            0.012224564683325483,
+            0.031700389750709,
+            0.018063405869147564,
+            0.0010794115307202266,
+            -0.002783668216641561,
+            0.01418234057248712,
+            0.017773269840982066,
+            -0.015653443355569053,
+            -0.000497740485657824,
+            0.019890882604100452,
+            0.022672054781919238,
+            -0.00521797370595221,
+            -0.016686942830774276,
+            -0.018443538351123584,
+            0.025270057639201107,
+            -0.010354431070738671,
+            0.03484047753830186,
+            0.04594785214094975,
+            0.0036441959528283004,
+            -0.05695476763071743,
+            -0.009412209619190693,
+            0.0010371110229966086,
+            -0.01099343927053564,
+            0.01606863921935641,
+            0.012470660511773616,
+            -0.03237187923799263,
+            0.006388672213167589,
+            0.012946570716567364,
+            0.022152554488829627,
+            -0.02540317621005867,
+            -0.023183252921885226,
+            -0.0030984019855787655,
+            -0.007890467768631811,
+            0.018648760402624687,
+            -0.008653405980661855,
+            0.0006228315253506687,
+            0.009672526897422602,
+            0.0005436494014569835,
+            0.011632326145022935,
+            0.005655942007108228,
+            0.021736827172951,
+            -0.001256608167230203,
+            0.013124413886287987,
+            -0.01986808920858981,
+            -0.0270450339809516,
+            0.006500423802582728,
+            0.04438242119211055,
+            -0.014813399694814117,
+            0.030621709833336064,
+            0.0092171224268983,
+            0.016433384739642534,
+            0.024875072913970443,
+            -0.00891470467241686,
+            -0.014124009014664287,
+            0.00845777541223868,
+            0.009019007848927453,
+            -0.04147358041382793,
+            0.012513030150663913,
+            -0.01710243399313324,
+            -0.033144939719206276,
+            0.01904674924133127,
+            -0.00534122554342279,
+            0.012367320751926314,
+            0.004270315800911569,
+            -0.002886460711688153,
+            -0.025696286516581048,
+            0.014137901708875033,
+            0.028135087538950946,
+            0.021797869775098042,
+            0.0296173052503884,
+            -0.012757078368361444,
+            0.0005460601780756187,
+            -0.008748930817524899,
+            0.005470975782187157,
+            0.04187083128854477,
+            0.0008687182830616019,
+            -0.0008032802520818105,
+            0.01507726552487313,
+            0.024945744517309208,
+            0.0376130540370386,
+            -0.015415560911023553,
+            -0.03407353556457147,
+            0.01564969972264396,
+            -0.010104443707865262,
+            0.018855792875436674,
+            0.005503259932317811,
+            0.0049634097823351596,
+            0.0038600156268000702,
+            0.019312419743287703,
+            -0.006635193789316492,
+            0.018798045806429395,
+            -0.02500983683871332,
+            -0.028785757874583755,
+            0.01829411751138161,
+            -0.005277972443258226,
+            0.03338623299668019,
+            0.016225885095995672,
+            -0.029522280454919586,
+            -0.01712097344863975,
+            0.01553706187924322,
+            -0.008465766956343768,
+            -0.024842700926275538,
+            0.01895228828512998,
+            0.007076862934523977,
+            0.008956105593078965,
+            0.0024373779389833218,
+            -0.01106503696703573,
+            0.01125466547367417,
+            -0.000818223040067914,
+            -0.015322790791698395,
+            0.014157131555941659,
+            -0.017853386731281715,
+            -0.014994640991761465,
+            0.01775991780643474,
+            -0.037498078532902435,
+            0.009570197712419421,
+            -0.007687297543935123,
+            0.009050494195378254,
+            -0.004222041469820445,
+            0.007460383234429904,
+            -0.0010804379186060264,
+            -0.014987857934522812,
+            -0.014903749199810126,
+            0.03228485430019625,
+            0.022096121825475306,
+            -0.000410816936759558,
+            0.006728733918931589,
+            -3.971636461500232e-05,
+            -0.026719848032785824,
+            -0.011253751790407687,
+            0.02245467425830765,
+            -0.005664635254398481,
+            0.005134496095445943,
+            -0.01109590855846966,
+            -0.0149616794609307,
+            0.0019726017255571673,
+            0.032668637270108894,
+            0.06066635618119076,
+            0.012001732730659747,
+            0.024422093585231484,
+            0.0189499396761026,
+            0.02119276033547528,
+            -0.008489818917377516,
+            0.006413901834269739,
+            -0.03097413287789443,
+            0.03509225573986143,
+            0.005392594635384506,
+            0.03108937318306421,
+            0.026664331248813712,
+            -0.04722525921137681,
+            -0.0058539171830594774,
+            0.017087531454837677,
+            0.022673712717885323,
+            0.0159020681868943,
+            0.0008382498574190522,
+            -0.008938174268517353,
+            0.035650940044285984,
+            -0.03315250299348084,
+            -0.023647111263155,
+            0.005059382024092534,
+            -0.01439537467317451,
+            -0.010488359956731476,
+            -0.006785295580877329,
+            0.02776449681317373,
+            -0.005584052419598926,
+            -0.004582019433008779,
+            0.009697064252638535,
+            -0.014400206558190682,
+            0.008165676294736836,
+            0.0007624778531368436,
+            0.008681263897172326,
+            0.039249866981334126,
+            0.0005918063050375113,
+            0.024275699182767528,
+            -0.004138313072843555,
+            0.011079529739650594,
+            -0.0158972266904151,
+            -0.016985058543605,
+            0.02276982363172991,
+            0.00351101120496393,
+            -0.03244126813432563,
+            -0.00768819222024939,
+            0.00928180979884261,
+            -0.0013793404539641824,
+            0.005266058566444064,
+            -0.00503433670609399,
+            0.003747448261046857,
+            0.03286230666764075,
+            0.0022383592721685153,
+            0.008606190020480513,
+            0.013593947677257625,
+            0.00030951915677929204,
+            -0.01490307680740127,
+            0.013254591138113545,
+            -2.1379308136707722e-05,
+            -0.012900073369514622,
+            0.03357519186606058,
+            0.006705902094864628,
+            0.04243347200485736,
+            -0.03302953674711988,
+            0.009031534935013593,
+            0.01822227379789576,
+            -0.008491770335368761,
+            -0.02725412856390648,
+            -0.03630788717786838,
+            0.008259145747740535,
+            -0.019003260334340084,
+            0.015781275447792516,
+            -0.010039277287105153,
+            0.029665124310477384,
+            -0.03563370794521629,
+            0.004714858110268169,
+            0.004053913355055313,
+            0.0009503673086552651,
+            0.02068360255395961,
+            0.018524093113914366,
+            -0.002785698934204024,
+            -0.031443835341186485,
+            -0.02066365433385339,
+            0.006869321907853112,
+            0.02431260052207595,
+            -0.010367048093621498,
+            -0.005612149485859478,
+            0.0006316058139530722,
+            -0.024369562712568606,
+            -0.016845124984022828,
+            0.030359419843284823,
+            -0.040974427176153246,
+            0.005252911449822132,
+            0.04272936053727608,
+            -0.04599092715016919,
+            -0.03802288716943514,
+            -0.002314988728575789,
+            -0.024124014894379466,
+            -0.006360771882090754,
+            -0.002068030981218464,
+            0.016669991113175926,
+            0.0036232707799735537,
+            -0.028763764013482795,
+            0.00943401978066451,
+            -0.005173988601101547,
+            0.003971420424839974,
+            -0.0031669592494977393,
+            -0.02216336363004636,
+            -0.01657772032717298,
+            -0.02531962163186398,
+            -0.035174102848865116,
+            -0.0015903434876311088,
+            -0.005158703150821607,
+            -0.028703075279214722,
+            -0.03941940849540492,
+            -0.021696000925262537,
+            -0.0160221488026189,
+            -0.00843213135542359,
+            0.040641882367348774,
+            0.021319060666253945,
+            0.019336844555345886,
+            0.0025252167889382917,
+            -0.020346900609125973,
+            0.049440028814233025,
+            -0.007642182460707816,
+            0.035628691886108926,
+            -0.0017882888000877158,
+            -0.013461669667950893,
+            0.00455558389906464,
+            0.03825428026120858,
+            -0.014248434654950971,
+            -0.0009902062564027718,
+            0.00027811655483232545,
+            -0.0011866253757890136,
+            0.0008535155950871286,
+            0.015433650396777696,
+            0.001307651096343185,
+            -0.007112199543314309,
+            0.030380673552683906,
+            -0.012803247194895328,
+            0.01637501752192592,
+            0.018350814869962304,
+            -0.0013359719137099667,
+            0.008064546833360553,
+            -0.021430659268086814,
+            -0.00731572118780415,
+            0.0044578576331236325,
+            0.008422210247967642,
+            -0.015096885054883586,
+            -0.009964588397149773,
+            -0.02359494467299634,
+            -0.01174145797396426,
+            0.0023855749166744366,
+            -0.004559202896782688,
+            0.004902138778592298,
+            0.034595817216889944,
+            -0.016068744500427584,
+            -0.035106033294403614,
+            0.009997207702122002,
+            -0.02663088875167486,
+            0.02222306406130687,
+            3.983384942889326e-05,
+            0.006838221769859366,
+            -0.0023434342406307616,
+            0.001765445386886839,
+            -0.00169484278667461,
+            0.002862331185439823,
+            -0.018217975204395134,
+            0.0031859734246840257,
+            -0.0032119988065673565,
+            0.0009052950782312657,
+            -0.0012459653052875638,
+            0.00687339892509406,
+            0.006332799551164718,
+            -0.0008019538898592668,
+            -0.002968451740267742,
+            -8.541086355097327e-06,
+            -0.020561816931299912,
+            -0.021642512405352372,
+            -0.028259049261410905,
+            -0.000989887278779396,
+            -0.012359733017834338,
+            -0.027804394906433664,
+            0.011443385424045703,
+            -0.006872684999639775,
+            0.008327844771213648,
+            -0.008969527138034904,
+            0.006219994886600349,
+            -0.008415106130309062,
+            -0.0004143996206881767,
+            0.013119286511794273,
+            -0.02925209282917943,
+            -0.02096907906398536,
+            -0.03096513357712367,
+            0.01762223841790055,
+            -0.00925741670612824,
+            -0.002656382260603827,
+            0.0025223527960181835,
+            0.005092659166535706,
+            -0.033606542487570154,
+            -0.006653438998497412,
+            0.007313682256676618,
+            0.009547290751154595,
+            -0.03287788897629887,
+            0.02446728061355753,
+            0.017023222076526946,
+            0.01874233066565012,
+            -0.02417566312664911,
+            0.04049827067897005,
+            0.014426879325647042,
+            -0.021740606525480655,
+            0.016712907564723732,
+            0.027385150577780248,
+            0.03177163221499527,
+            -0.03469098547555545,
+            0.004481376248622418,
+            -0.01518200063132508,
+            0.018918680020516044,
+            -0.025284921244460287,
+            0.020625170445627613,
+            -0.016707629602243063,
+            0.021278719363178687,
+            0.03520334001264928,
+            -0.005524369131116028,
+            0.03608133014180745,
+            -0.019234548780647943,
+            0.0059985801736444555,
+            0.004041778461022664,
+            -0.01645503683802124,
+            0.01024694092614158,
+            -0.0042700351372843386,
+            0.0006059159967286885,
+            0.009406090340935747,
+            0.019133229156115282,
+            0.019007219188041775,
+            -0.04442208547053466,
+            0.028588965123434923,
+            0.00804120101594811,
+            0.021472086070876265,
+            0.014177820346835521,
+            -0.02708807970982664,
+            0.004742981711172693,
+            0.002072673190279628,
+            -0.004659713164250794,
+            0.0061167904730676514,
+            -0.013796604518930328,
+            -0.034540857808309226,
+            0.006338792319814448,
+            -0.018918703778187926,
+            0.015036937364271984,
+            -0.009026670935442384,
+            0.0011667525636770179,
+            -0.01152652811400482,
+            -0.011938679695598347,
+            0.0006170592505446954,
+            -0.005495151258143897,
+            -0.0008075736908949407,
+            -0.03171721592827769,
+            -0.04427658539412448,
+            0.02036132271908089,
+            -0.0021221965958423804,
+            -0.04320396743189005,
+            0.004111179168317842,
+            -0.025352428814134187,
+            -0.01865918186335911,
+            -0.011237699157960728,
+            -0.025047923887906474,
+            -0.04219630696740606,
+            -0.011108496740166623,
+            0.007785006066683012,
+            0.016810642610918525,
+            -0.009081815651299525,
+            0.040965574143584864,
+            0.0073944364393447965,
+            0.015313576815177263,
+            -0.017731360025694955,
+            0.026137777790243826,
+            -0.019314392060123,
+            -0.03108486330987073,
+            0.02676829566273746,
+            -6.259161089416372e-05,
+            0.003053748984540415,
+            0.014358356168574803,
+            2.8186924071913628e-05,
+            -0.006181789640967465,
+            0.01947730374003864,
+            -0.028739300217145797,
+            -0.023460503178221397,
+            0.0012460072421725688,
+            -0.008850395570726385,
+            -0.016271242375881602,
+            -0.035896509136022194,
+            0.005227881053956234,
+            0.03324337105109496,
+            0.004970419987430122,
+            0.008273497228454482,
+            0.01814500594735864,
+            -0.02229593571615308,
+            -0.024230675455380678,
+            0.000639112529814716,
+            -0.010266640429960994,
+            -0.010257247303470066,
+            -0.021302483296731352,
+            0.019150757066356872,
+            0.01651375496287087,
+            -0.00390189952706486,
+            -0.006971692709239087,
+            -0.010621003777733253,
+            -0.01918206725910585,
+            -0.02020345113211036,
+            -0.013827322378214754,
+            -0.017980206764311026,
+            -0.034153741666571154,
+            -0.032195001867229935,
+            0.03225205339201753,
+            0.026354817588919596,
+            0.0014076409241492998,
+            -0.007959081015805435,
+            -0.023070419006792155,
+            -0.004483328183581872,
+            0.02879516595345542,
+            0.002549934782150634,
+            -0.0001541849716285838,
+            -0.00830090759167373,
+            0.015341095409025125,
+            0.00835695019244165,
+            0.005107844984733648,
+            -0.06676974053142512,
+            -0.006027540973808174,
+            0.01655283585235268,
+            0.01351155740684989,
+            -0.002359114145814062,
+            0.02591301169926259,
+            0.01108099264396926,
+            -0.015361038608114896,
+            -0.019765743462262668,
+            0.035195761515422364,
+            0.02408910733949848,
+            0.023468767772473746,
+            -0.0034492999385544727,
+            -0.004472311490618405,
+            0.007030833258457338,
+            -0.010684252427867153,
+            -0.013326964647936429,
+            -0.004558513842504658,
+            0.026925146859337397,
+            0.016099443021514896,
+            -0.0064331198280825575,
+            0.010848931851939124,
+            -0.0008320209125061849,
+            -0.019444491995041537,
+            -0.001841330829156697,
+            0.008730275423516445,
+            0.0012974082030866743,
+            -0.014611902493400719,
+            -0.005911807768916134,
+            -0.019130604101278263,
+            0.024719468317569562,
+            0.0209456329723249,
+            0.03381727436368338,
+            -0.013237902591972769,
+            -0.0003695142331921278,
+            0.005410360755058108,
+            0.0002841632253252816,
+            0.0023352645111704958,
+            0.004801123870325242,
+            0.02623649813035263,
+            -0.029010710171522985,
+            -0.020741774066460415,
+            0.0068426980682519505,
+            0.013445270043414486,
+            0.0022744032984779294,
+            0.033840485942696216,
+            0.035040080489222324,
+            0.004166066423508316,
+            -0.010046666684988869,
+            -0.006148765564658713,
+            0.007071740742464309,
+            0.01865897335451031,
+            -0.023267411753900076,
+            -0.0005936657905496225,
+            0.004833030519903764,
+            -0.004229582648167378,
+            0.03620610659071198,
+            -0.009961238880412731,
+            -0.007197670122096852,
+            0.022212791947347127,
+            0.09819205436926473,
+            0.00047695621444943017,
+            0.0025354521225913395,
+            -0.029358210234742176,
+            -0.023614016211567963,
+            -0.0012564198366457442,
+            0.00242680708457405,
+            0.00830167274308547,
+            0.028076260517106013,
+            0.0004852400049405906,
+            0.011579274037419117,
+            0.02773972145320728,
+            -0.017217305534161797,
+            0.006752070301141707,
+            -0.011376051474975524,
+            0.024577071515205502,
+            -0.012694034026935636,
+            -0.0019809719792138673,
+            -0.016512649783050146,
+            0.015907402258634816,
+            0.004556574687833768,
+            0.004907510649166062,
+            0.011347630984408937,
+            0.009824484201337644,
+            -0.02650731900265866,
+            0.01756773113723491,
+            -0.005671986127099093,
+            0.004449611508541289,
+            -0.027841346572384462,
+            -0.020124233970502267,
+            0.0025986391930124847,
+            -0.0014440758308989809,
+            -0.009579444248630085,
+            0.006005274872617889,
+            -0.0024984235450999127,
+            0.03765664613203687,
+            -0.03765496603611941,
+            0.01409350051861866,
+            0.0066453175608372555,
+            -0.0015390841089599882,
+            0.011806653942640485,
+            0.017306914776093522,
+            -0.008206256939290151,
+            0.02589800448884443,
+            -0.00798213351353373,
+            0.009846274212000523,
+            0.015230070803055296,
+            -0.03449531102975745,
+            0.009174739184806312,
+            0.010145762060174144,
+            -0.024165660737765746,
+            0.012146877603301336,
+            -0.013784371016298891,
+            -0.004818759788054113,
+            0.005930446700613062,
+            0.007349837554764034,
+            0.010733731974181452,
+            -0.02295571464086674,
+            0.016176424826629898,
+            -0.010947841926210105,
+            0.0037766665765835445,
+            0.013559846998199008,
+            0.024892849808979323,
+            0.0012426907454129122,
+            0.0005950515136086193,
+            -0.01379469431504576,
+            0.005534724429523084,
+            -0.015667467221535825,
+            -0.022401586867694584,
+            -0.014192041775234079,
+            -0.006366846766730035,
+            0.01029598687444803,
+            -0.003972386708834858,
+            0.016971476088808737,
+            -0.010309648640697812,
+            -0.010645860303528099,
+            0.0037818310327362103,
+            0.01040806357838676,
+            0.008316605962234849,
+            -0.02855914079113881,
+            -0.005676310556150579,
+            0.004961084138163161,
+            0.00029285971503001265,
+            0.009592497954559435,
+            -0.014556073011576487,
+            0.013689526487160009,
+            -0.013290113354287095,
+            -0.0010726537498214612,
+            0.013201860397060236,
+            -0.012443552043930505,
+            -0.004012305744087929,
+            -0.01592905426965635,
+            -0.028487066798154617,
+            0.008462143660968396,
+            -0.01365614159523485,
+            0.001076882584445294,
+            0.0019503240125505646,
+            0.004730085646042709,
+            -0.016919069570642004,
+            -0.007860046623166481,
+            -0.033515448866521046,
+            0.011065416478687504,
+            0.02314589687882036,
+            -0.006435733825418645,
+            -0.030351727858789472,
+            -0.009546333526852862,
+            0.011327489786114963,
+            -0.0071765122743172715,
+            -0.01676736383051139,
+            -0.0029390038145944096,
+            0.007617910650439855,
+            -0.008174620754097493,
+            -0.00508943241317381,
+            -0.0025173969010414098,
+            0.0217328037301181,
+            0.016094975605203832,
+            0.01769642142712967,
+            -0.021400862603034036,
+            -0.00976992221747187,
+            0.026043879119776864,
+            -0.03275550610951898,
+            0.012441465975914529,
+            0.024587446266013183,
+            0.003446527051428522,
+            0.005492228054838714,
+            0.0030377554109693714,
+            0.005459159163772825,
+            -0.006985179226831048,
+            0.019925583798328282,
+            0.010870421035240845,
+            -0.019709062313471467,
+            0.005385443078244229,
+            0.02086085539375208,
+            0.0037248737640193097,
+            0.012892299156840998,
+            -0.017546870090541213,
+            -0.0020341807891543716,
+            0.015594779287017958,
+            0.04816214672670215,
+            -0.0032851797178091685,
+            -0.010450846098750195,
+            0.029513742590677183,
+            0.023526004023637855,
+            0.005373455940309873,
+            0.0012120777808635214,
+            -0.02525630456877349,
+            -0.013075130701774785,
+            0.01962235086017781,
+            0.016944870494393874,
+            0.013484925533584295,
+            -0.0023255656480479004,
+            -0.019236081983457366,
+            0.021520156296467717,
+            -0.01701733319731161,
+            0.012422659010697477,
+            -0.023764941037007197,
+            -0.001490790833263327,
+            -0.0035419008954878724,
+            0.021998949056452454,
+            0.0021819635638621092,
+            0.008891505304514552,
+            0.024354552127458608,
+            -0.006978548952551799,
+            -0.019526615985982355,
+            -0.0051799020400150305,
+            -0.014578953088901798,
+            0.022506635912517146,
+            -0.024117708614271965,
+            -0.007008641019892289,
+            0.011379795968360019,
+            -0.012362093847309038,
+            -0.0102534051092702,
+            0.001164934138876502,
+            0.017686807394319725,
+            0.0009423993435129515,
+            -0.014784508280754078,
+            0.01944445593807992,
+            -0.0011359271135469456,
+            0.010410915597949927,
+            -0.01921419010977189,
+            0.009390638453648025,
+            -0.016682442516992777,
+            -0.007849184048209236,
+            0.010343008873053859,
+            0.0012604521199837046,
+            0.003948300703645506,
+            0.002592490121754231,
+            0.007773437307969315,
+            -0.03355658426684504,
+            0.022695505487314516,
+            -0.010751606023585792,
+            -0.003538523688786008,
+            -0.0014125330167969218,
+            -0.01678692701672328,
+            -0.0006918162525881059,
+            0.005816502852598842,
+            0.010878358093847386,
+            -0.016045147059003256,
+            -0.00873270513444376,
+            -0.04920244395446931,
+            -0.030314009649638693,
+            0.023156333024481057,
+            0.013733657599720879,
+            -0.010971506342695331,
+            0.018852159415605117,
+            -0.023760853159532587,
+            -0.010587249404421732,
+            0.0011379869499042905,
+            0.023319856329056813,
+            -0.03539290744576506,
+            0.01215369703816749,
+            -0.014493682360886549,
+            -0.02039288669534806,
+            0.009070984596830142,
+            0.011445354735283254,
+            0.0007259852045513916,
+            -0.0011514627555568814,
+            -0.029350945123925453,
+            -0.010246845821727087,
+            0.02391068177425878,
+            -0.00861718589055971,
+            -0.004723044986826333,
+            0.024716520302409263,
+            0.0033721801437248873,
+            -0.012524663490691517,
+            -0.017955758068233754,
+            0.021612671655463236,
+            -0.026077203994996058,
+            0.007645944309444457,
+            0.010032183011253845,
+            -0.0173991263874487,
+            0.021649599230588117,
+            -0.007529701797485131,
+            0.020327763440441377,
+            0.004227390938169872,
+            -0.011249233530251522,
+            0.020352283974471443,
+            0.013644443758970054,
+            0.004216958800094986,
+            -0.004081669404784771,
+            0.006869353098014179,
+            0.02212732880780841,
+            -0.011021739614266063,
+            0.02219580717345023,
+            0.028777213295471067,
+            0.00017328191188631918,
+            0.03260179554743322,
+            -0.021287655520443285,
+            -0.03159728983004563,
+            0.021740163175540124,
+            -0.01035146858119169,
+            -0.021564906188712388,
+            0.028914245422221707,
+            -0.013674731836508768,
+            -0.00786476052825481,
+            -0.0166718429993922,
+            0.00533027311342322,
+            0.016680336311088843,
+            -0.005246190451159961,
+            -0.014360934066628332,
+            0.0029628103834216345,
+            0.019809487418628386,
+            0.014587234544007958,
+            0.029780009648869975,
+            -0.010778199815920887,
+            0.035180242185168895,
+            -0.002320576872571577,
+            0.008990157317473909,
+            0.012415465232073404,
+            0.006489435152972723,
+            -0.011251531176923445,
+            0.013501894212117511,
+            -0.003446759011266526,
+            0.008230548711588447,
+            -0.001449023474280409,
+            -0.035111679577202855,
+            -0.013709382199848678,
+            -0.04709294504969336,
+            -0.0023752976043235697,
+            -0.0025098360125255215,
+            0.0030975417740913154,
+            0.09736815869389318,
+            -0.005723049647039619,
+            -0.011988817571623457,
+            0.001625059666070763,
+            0.009611697638805556,
+            -0.0009896494420056814,
+            -0.007606836952841873,
+            0.015973152482488318,
+            -0.01343292635775299,
+            -0.02140928016038583,
+            -0.010532701190806147,
+            -0.01288594454368977,
+            0.012079198190522715,
+            0.015929716343161652,
+            -0.0005947359502228821,
+            -0.015207331663761012,
+            0.006524010655124294,
+            -0.004566385899752684,
+            0.0064717704997400245,
+            0.008647099975176308,
+            -0.010414668624343924,
+            -0.0071698879102925555,
+            0.030899143952119196,
+            -0.025301269549419674,
+            -0.024361748354052457,
+            -0.0010579931222861095,
+            -0.009055796737486008,
+            -0.010335459737012047,
+            -0.002692653633719985,
+            -0.0017679501322462056,
+            -0.010519994985692081,
+            -0.010019865263564572,
+            0.006537076627706128,
+            -0.015472130358273033,
+            0.004812857776601667,
+            -0.0047457011269495634,
+            -0.013296069496526007,
+            0.017054968086538724,
+            0.022563785357424853,
+            -0.02841491200807702,
+            0.0006755428946168761,
+            -0.017422510783648978,
+            0.012941211381006863,
+            0.008866069010589792,
+            0.006049618782357251,
+            -0.022451528209097505,
+            -0.023622906582403243,
+            -0.015019272754405791,
+            -0.01927535750568903,
+            -0.024973490129133732,
+            -0.009073381957684092,
+            0.0067692803507600945,
+            -0.001267339788153158,
+            0.021918281111033645,
+            0.000816538714279555,
+            -0.01812545613159305,
+            0.0061084913570894905,
+            0.000568371220952269,
+            0.018292497018668842,
+            -0.027468677288882186,
+            0.0033806422195878524,
+            -0.02307582511848465,
+            0.033508234091571124,
+            0.0039195050746594975,
+            -0.026680111514794824,
+            -0.021558668725755275,
+            -0.003311830287529699,
+            0.018994577319066473,
+            -0.0025745912019675805,
+            -0.00724049447539589,
+            0.014775207617625293,
+            0.0155334760323838,
+            -0.004693118176196483,
+            0.002015625406296403,
+            0.027968079002437102,
+            0.01077998866180723,
+            0.0313892866933904,
+            -0.015236169297590675,
+            0.004945828495617267,
+            -0.027215016713379062,
+            0.010214419950212732,
+            -0.01503548137805401,
+            0.008859554812646242,
+            -0.01290890682557532,
+            0.03531384210182567,
+            -0.012005613861257432,
+            -0.004250698950192301,
+            0.03253288796441852,
+            -0.029150986393345252,
+            -0.0035020051536906334,
+            -0.009255437891304041,
+            -0.004142204892160768,
+            0.013240580426083011,
+            -0.001198093874819309,
+            0.008485141356876356,
+            -0.01854370462655459,
+            9.726096864073609e-05,
+            -0.003221852304410318,
+            0.005404595442127598,
+            -0.02005458688172967,
+            -0.003314118809778734,
+            0.0073427568419969814,
+            0.004259518237708634,
+            0.002408630356035373,
+            0.016420646142725346,
+            0.006662121128241274,
+            -0.0029357701493526907,
+            0.03424224243607692,
+            0.008171895850772245,
+            0.0152521424877615,
+            -0.0018638264193202728,
+            0.0030234211261443034,
+            0.019287963516760636,
+            0.019912956661192856,
+            -0.00048443399729935116,
+            -0.01058935153883094,
+            0.00195113175754497,
+            -0.001140732137433033,
+            0.011048741602327437,
+            0.024396046054912645,
+            -0.021158025221644738,
+            -0.02419562092713724,
+            0.016377661005279716,
+            -0.015076145951740961,
+            0.01903712233378994,
+            0.010541463112889985,
+            -0.010099351302095792,
+            -0.00031558616108876247,
+            -0.042316016201379245,
+            0.010964989031040692,
+            -0.02068939391415103,
+            -0.04004120028915868,
+            -0.00244383573783919,
+            0.0009542507389088937,
+            -0.029439456243505052,
+            0.02822814855547978,
+            0.0023755032182188156,
+            -0.017280221740813364,
+            0.018223709137876153,
+            -0.005548654680726505,
+            -0.0024040301239229064,
+            0.005397711787238679,
+            -0.01092026090471121,
+            -0.010648485086769927,
+            0.008235250252660972,
+            0.010957715075632535,
+            0.005425683656535705,
+            -0.005698919020608095,
+            -0.011284911899894062,
+            0.009975882651638084,
+            0.012318901214538077,
+            -0.0019922884373083155,
+            0.010050520101815593,
+            0.0031720752428247446,
+            0.007352859750323051,
+            -0.0073483049386141005,
+            0.0046483587496626965,
+            -0.005093220128411125,
+            -0.026080057085901446,
+            0.012863580653903103,
+            0.0052434400313520774,
+            -0.011863360599637747,
+            0.0103209502958491,
+            -0.01048162756158923,
+            0.005787157766911021,
+            0.001318945460232246,
+            -0.0059333230422061395,
+            0.004821320238893602,
+            -0.0038922239931512784,
+            0.011267634571983924,
+            0.013256611037442042,
+            0.004592529900805719,
+            -0.013076954459595086,
+            -0.02177318635989856,
+            0.009001291224623467,
+            0.01959767163356408,
+            0.024161754192119852,
+            -0.030416109293488064,
+            0.003784634588325551,
+            -0.0014915849060829743,
+            -0.0020550564975080885,
+            -0.015632785947151753,
+            0.01587790981601866,
+            0.01716742056401084,
+            0.012721563202528083,
+            0.01893393006967701,
+            -0.004009617703771589,
+            0.010539100665843704,
+            -0.02080068888791658,
+            0.013642451766699067,
+            0.0008609869904066794,
+            -0.013629067231663643,
+            0.0014620655123815298,
+            0.027046304278766087,
+            -0.011056783825898616,
+            0.006206969699250696,
+            0.008097527175511791,
+            0.022111870736907183,
+            -0.012509479100313492,
+            -0.003735868296238069,
+            0.01516254968167132,
+            0.0014878674796313539,
+            0.0035887079417890352,
+            0.016533149527222273,
+            0.04729052276110659,
+            0.00102751186013391,
+            0.0016713773449738153,
+            0.0192177920520243,
+            -0.005563519063161466,
+            -0.01114821419109643,
+            -0.003549999863325982,
+            -0.021807893906469106,
+            0.010998558936481785,
+            0.00431279785115251,
+            -0.02186324212032519,
+            0.01024093366826206,
+            0.0125495473550584,
+            0.005268320135120931,
+            -0.0031679212410026603,
+            0.02574575247827441,
+            -0.019807879421996258,
+            0.014935295894965128,
+            0.03329450628384787,
+            0.00493356778223407,
+            -0.011423949333937285,
+            0.007263264249871384,
+            0.0027354459678742013,
+            -0.004494406041906684,
+            -0.00606745413660583,
+            0.010497391290734088,
+            -0.008191702034391382,
+            0.009870458191510176,
+            -0.009580130479693,
+            0.039202091799313755,
+            -0.009619018331262844,
+            -0.010782138248435458,
+            -0.004558645539551881,
+            -0.006241773108460662,
+            0.003478414182938609,
+            -0.003395165694642073,
+            0.005812225135633631,
+            -0.0069045407353789945,
+            -0.014607146220271209,
+            0.0008522982757210454,
+            0.00020175973815948303,
+            0.014001922919116725,
+            0.0011804904415274088,
+            -0.004871874532755353,
+            -0.007877217154706205,
+            0.015243277056007988,
+            -0.006047906208304912,
+            0.0006531937033317192,
+            -0.0011653523616313422,
+            -0.0013662556399359335,
+            -0.007401050012179761,
+            -0.015166105795085528,
+            0.00974609244984173,
+            0.016253733908242134,
+            -0.013857834499211056,
+            -0.005142936208363415,
+            0.012274716212832219,
+            0.008540229125193836,
+            0.011742606168187876,
+            0.0007451338650642946,
+            -0.009255813682572937,
+            0.008292531177374192,
+            -0.015020523035432158,
+            -0.008781317965985481,
+            0.0026250469418136603,
+            0.002381147382977664,
+            0.005839723981046825,
+            0.0006240641828668715,
+            0.004502463508605272,
+            0.005798912705218846,
+            -0.00497936644919367,
+            0.01112666823857836,
+            0.008198424171006548,
+            -0.002452143482105151,
+            -0.01017569410837814,
+            -0.001127589984272638,
+            -0.015733298167661346,
+            -0.016758315411028353,
+            -0.0009286266498936771,
+            0.015175988113750882,
+            -0.02352862898718829,
+            0.018996352975391644,
+            -0.0014524363611708201,
+            -0.0033802546171857704,
+            -0.021392428456030312,
+            -0.0025923983911570183,
+            0.0034214188668518796,
+            0.01255768230394476,
+            0.01858853630745928,
+            -0.02729543519465384,
+            0.010575894388563116,
+            -0.007204725245442141,
+            0.030394929424014717,
+            -0.018328782140005044,
+            -0.004022941273610227,
+            0.026915618079621992,
+            -0.02200018306828758,
+            0.0008791185377513978,
+            -0.011454606955158956,
+            -0.008103032693853504,
+            0.019287674837688177,
+            0.023747720480736777,
+            0.02038287242167783,
+            0.01364489294869211,
+            0.0007758044919789085,
+            0.004768030222445397,
+            0.03572705648994403,
+            0.010568241270862248,
+            0.014828087092859878,
+            -0.007542489523110974,
+            9.457808640264811e-05,
+            -0.02554033312739386,
+            0.01635899434369978,
+            2.587383534566272e-05,
+            -0.012370727506722627,
+            0.007162422186621379,
+            -0.013370882403931942,
+            -0.011066483600541058,
+            -0.007151077450730025,
+            -0.01721910018182534,
+            0.02152239849400241,
+            -0.012322440280680164,
+            0.004587729883596212,
+            -0.009079747696669277,
+            -0.013272707286450417,
+            -0.012953682633400515,
+            -0.014202214807187545,
+            -0.01313141988034687,
+            -0.026041403779538307,
+            -0.027109731260915763,
+            -0.006573285295895077,
+            -0.023780366346334524,
+            -0.013799072648505526,
+            0.005588622594136253,
+            -0.022617902666316757,
+            0.012262325666428604,
+            -0.012689827812482627,
+            -0.01226264872059864,
+            0.012805734807835008,
+            0.020274844315616616,
+            -0.011765231803328348,
+            -0.028494489703400308,
+            -0.018583843821043854,
+            -0.019970998470704358,
+            0.012576201137505969,
+            0.002266288538944273,
+            0.013165889075444961,
+            -0.003018437854660793,
+            0.003227412667880239,
+            -0.014895831590298957,
+            0.018570624675927183,
+            -0.03442390703780963,
+            -0.03108003806963536,
+            -0.013611306765348361,
+            0.005182777066476028,
+            -0.0015782033203578553,
+            -0.011429046504256945,
+            -0.0018606648572920801,
+            -0.00433805271182878,
+            -0.025109572024446088,
+            -0.010134856632190067,
+            0.011010662054484161,
+            -0.015205629927469414,
+            0.0072933565255675036,
+            0.0009018832564038863,
+            0.013870902834260887,
+            0.009587432944616442,
+            0.013393301893492882,
+            -0.006095063698384775,
+            0.004356485794113208,
+            -0.0026476612747324386,
+            0.0004894024149759108,
+            0.009336366919474478,
+            0.00839591561027103,
+            0.010551161369079399,
+            0.002260707777260338,
+            0.007830059223001696,
+            -0.0060927446216633265,
+            0.018065002687169148,
+            0.015436944875538481,
+            -0.0002520689442496089,
+            0.013323098485251487,
+            0.0003769862591044255,
+            0.00022293304921127018,
+            -0.0029970303106340466,
+            0.007888437018607987,
+            -0.011785957933642163,
+            -0.001311704406355858,
+            0.015138190893483804,
+            -0.01707247730569978,
+            -0.013045187198628258,
+            0.03107034781283638,
+            0.0018583638488268697,
+            -0.02495564203304796,
+            0.031146554085787762,
+            0.004220047610942943,
+            -0.005835199594223787,
+            -0.013205086624848782,
+            0.0014059047822222418,
+            -0.011570624844100298,
+            0.003584232809362274,
+            0.014295032061446826,
+            -0.02231984223610487,
+            0.010521094801839404,
+            0.008446892441796151,
+            -0.009681535919980559,
+            -0.012337659795586223,
+            0.009847073709209602,
+            0.016809328445200358,
+            0.004472485504034416,
+            -0.03539093810210523,
+            -0.011957694102753411,
+            0.02145470732008716,
+            -0.029237428225009913,
+            -0.00810041956259673,
+            0.007452962400105787,
+            0.0065154832040426065,
+            -0.019353109017609053,
+            -0.008580955020412362,
+            0.005725794605897509,
+            -0.004940390374637084,
+            -0.003892210172017758,
+            0.017541623128534682,
+            0.00485047043626186,
+            0.013698341792101025,
+            -0.01689738578307474,
+            0.009771458057400557,
+            -0.022222817828602557,
+            0.004021882074676753,
+            -0.023373700500461992,
+            -0.009873403167702603,
+            0.002364143609079849,
+            0.012454271200139762,
+            0.014328556467632212,
+            0.027839363143659326,
+            0.030056404238348527,
+            0.013995696971422429,
+            0.007833677585250953,
+            -0.016265199386953717,
+            -0.018134327237616175,
+            -0.006409513840075933,
+            -0.004033173350204784,
+            -0.008714798059218596,
+            0.0066373547199279235,
+            0.01858354212009877,
+            0.004499712802472377,
+            0.02609078206670711,
+            -0.006227189596272876,
+            0.009039744622014585,
+            -3.127526911951914e-05,
+            0.0008833930131827993,
+            -0.004226083827636182,
+            -0.01971685546584527,
+            0.00481003211084742,
+            -0.0013721482162825508,
+            0.025874758880591167,
+            0.00940496860916206,
+            0.0024915013399541847,
+            0.01615978314811047,
+            -0.006378952757397484,
+            -0.01174653607178726,
+            -0.010540107733238588,
+            -0.0061926268398085176,
+            -0.021752582127981104,
+            0.029059100503124387,
+            0.021195595568253612,
+            -0.010800765264123782,
+            -0.013693582166598082,
+            -0.00903369526059487,
+            0.02795767990556847,
+            0.004080965688836953,
+            -0.0013050212744006933,
+            0.016313311850308505,
+            0.012084555958999215,
+            0.012329325712491252,
+            0.009794698207689061,
+            -0.00036157148861933123,
+            0.0015364709122308352,
+            0.03722522141440073,
+            -0.007427028510548186,
+            0.0026463292682538495,
+            -0.005239324411651515,
+            0.0025987932868461313,
+            -0.009705817520422487,
+            0.007242122655628283,
+            -0.004012676385296197,
+            -0.0158760051470016,
+            0.015003444174308607,
+            -0.022443331305169994,
+            -0.0016749787204471238,
+            -0.008150635190952799,
+            0.0016696640640947303,
+            -0.02781023337840043,
+            -0.0025731206607371845,
+            0.006695065701630532,
+            -0.015547340828489069,
+            0.011816010501093885,
+            -0.01740673229051557,
+            -0.011553604323149105,
+            0.007914961618894248,
+            -0.012034157288065447,
+            0.0016684130271324446,
+            -0.0008268957621923278,
+            0.004460864332388434,
+            0.013426055079012884,
+            -0.011008808570221245,
+            0.012166046631826273,
+            -0.005816284310138776,
+            -0.007872091214834688,
+            -0.0016055129487028885,
+            -0.0030049472973942043,
+            -0.0025600746761171543,
+            -0.0009966231170649917,
+            0.034355910234323155,
+            -0.004446570896048543,
+            0.003761242147499512,
+            -0.01362639743248326,
+            0.0019833052855181806,
+            -0.036176376394999044,
+            0.028344464817309477,
+            -0.008642835889481802,
+            0.011013502221008498,
+            -0.0017508365480865596,
+            -0.013243706268954915,
+            -0.01456517309747086,
+            0.015247299501245178,
+            0.007602596572394442,
+            -0.005854531057349665,
+            0.013558248087347848,
+            -0.019724373614448298,
+            0.008530500822910953,
+            0.004463694543554244,
+            -0.015561397976987216,
+            0.004668780190328835,
+            -0.007479912658259757,
+            0.012657201279639194,
+            -0.01774730033362932,
+            0.004314594593150765,
+            0.00958571617187387,
+            -0.01892876084898881,
+            0.009901041800918215,
+            -0.028675879564512306,
+            -0.004631920242208613,
+            -0.012323455524366223,
+            0.0053804608827285865,
+            -0.000660864620708182,
+            -0.02389088252276507,
+            -0.028944280660230762,
+            -0.0002099667627277945,
+            0.00458533068247518,
+            0.0018878351997997833,
+            0.011621407953503755,
+            0.005970807015151215,
+            0.008542958629199003,
+            0.01023896054756221,
+            -0.00205207247588841,
+            0.009402752124876478,
+            0.04169706217414899,
+            -0.022899912321317197,
+            -0.011574970753549647,
+            0.005321872977369877,
+            0.011499271505460519,
+            -0.02647653483798284,
+            -0.011039791758347888,
+            0.0033424345263339144,
+            -0.005219877979550397,
+            0.03609469691834737,
+            -0.02448746405881972,
+            0.019020618652020673,
+            -0.011392297116547224,
+            0.007625475937968714,
+            -0.021465240076078376,
+            -0.003921088264697311,
+            -0.014265745991522477,
+            -0.00011221615798127438,
+            -0.0031478236827989703,
+            -0.0076053733863387755,
+            0.007485039021296698,
+            -0.001907678661303243,
+            0.0020348246022161925,
+            0.006291436821831794,
+            -0.02173305455485444,
+            -0.0019125499969493854,
+            -0.016432460924913166,
+            0.030367263349759286,
+            -0.016242095671439496,
+            0.005072805486919414,
+            0.0053258139728101506,
+            -0.002810639419737258,
+            0.012368771814660942,
+            0.004165967191536186,
+            0.00513754440091394,
+            -0.004199767241827414,
+            -0.005142642360546972,
+            -0.0053886448467479215,
+            0.017868874549760376,
+            -0.013280048598849825,
+            0.035024701466710065,
+            -0.0100959186045929,
+            -0.007094001877261568,
+            0.0019733042325232155,
+            0.010380430710282744,
+            4.2084202709296443e-05,
+            0.01053565670076933,
+            0.0034270489770133685,
+            -0.0006311078385359693,
+            0.0019774779328212756,
+            -0.017162448473652843,
+            0.003089758682621882,
+            0.01682656561693936,
+            0.005463677441664502,
+            0.004469261499451007,
+            -0.013332130891003062,
+            -0.012050233723282738,
+            0.004538268704442613,
+            -0.007533940408255447,
+            0.0003675737140813486,
+            -0.018816953644773076,
+            0.013972696748575243,
+            -0.0011096683686631858,
+            0.0002715683052236706,
+            0.00524678369365015,
+            0.023056514997908218,
+            0.01813167860994747,
+            -0.002553352602551003,
+            0.03155164978556312,
+            0.0174738879054593,
+            0.0029268039949260673,
+            0.010637295711777793,
+            -0.033222781195984666,
+            -0.013301873512972214,
+            -0.0013721703891642354,
+            0.02238421933149517,
+            -0.01530189703335162,
+            -0.014059029663525449,
+            -0.0037393534403749947,
+            0.0010394601933309767,
+            -0.005177359933828505,
+            0.0042489306011644495,
+            0.011039889511148341,
+            -0.014697595240068863,
+            0.007597149702441915,
+            0.0026780496822907634,
+            0.02092734730183164,
+            0.0004688144432243786,
+            0.017181410417231698,
+            -0.0005083970646221274,
+            0.00749639055342427,
+            0.007319420826337891,
+            0.017541463125355265,
+            -0.01002811389296805,
+            0.023582113845081534,
+            -0.008823933397729605,
+            0.0019889258902730535,
+            -0.01257106158422629,
+            -0.028159594067544737,
+            -0.012624750220149823,
+            0.015247564545279736,
+            0.005765911899842925,
+            0.006478299226263732,
+            -0.016537090297144265,
+            0.006276997440048981,
+            0.015647122171767974,
+            -0.002251683151059454,
+            0.010295358869874306,
+            0.009730378299068289,
+            0.00012816405890877098,
+            0.010181120688239766,
+            -0.00801256937783534,
+            -0.00734257767723293,
+            -0.004497520124843868,
+            0.023982754195729983,
+            -0.005443550092842393,
+            0.022434177122972028,
+            0.012647958478747348,
+            0.00616791857573897,
+            -0.010853355357954727,
+            -0.003340111254357573,
+            0.023389634440057473,
+            0.011863246790550376,
+            0.00029642699874398923,
+            -0.009846034060495678,
+            0.0019458876579524617,
+            0.0054010373765486005,
+            0.012983913038716568,
+            -0.02049427270240938,
+            -0.004344571352918878,
+            0.008461947538684664,
+            -0.008472919299762712,
+            0.004318309005722874,
+            -0.030156444522730416,
+            -0.005812303237222271,
+            0.021354352236942434,
+            0.008392281015122372,
+            -0.010232039722494646,
+            -0.004395682831498155,
+            -0.0006527587802367882,
+            0.0040601850269714945,
+            -0.0024323167053212176,
+            -0.0023729138545912176,
+            -0.008541233856441836,
+            -0.006470933419694742,
+            0.014035696617797647,
+            -0.0052691760528871924,
+            0.008597459268477037,
+            0.00012084859681905111,
+            -0.0017255958688825231,
+            -0.007323529232751411,
+            0.03382792341957058,
+            0.008087732655845718,
+            0.005980666974986827,
+            0.0013613625176327891,
+            0.0042244330945601215,
+            0.000616791811672998,
+            -0.0028374856646453707,
+            0.018075360591135542,
+            0.0156738939725117,
+            0.005054696911814988,
+            0.009038525980341447,
+            0.014672270369917403,
+            -0.019700405219230957,
+            0.012625337070556061,
+            0.012184185186281482,
+            0.019382321278525114,
+            0.0007219498737851579,
+            0.002564970959687783,
+            -0.010601873823737926,
+            -0.0029963537053764736,
+            -0.002070567089566093,
+            0.029563028293936092,
+            0.0005903976560069006,
+            -0.010509758568838935,
+            0.010992421945232951,
+            0.020603868018231348,
+            -0.02101157589451638,
+            -0.001971974816402967,
+            0.005963911670920596,
+            -0.010704207628592966,
+            -0.014466113540150018,
+            -0.021269176952661614,
+            -0.013331247081394942,
+            0.0071597345125287174,
+            -0.008162102923896894,
+            0.02367549407145414,
+            -0.014256771777502395,
+            -0.03960796112042958,
+            -0.014803626428839152,
+            -0.00864320347698291,
+            -0.027531337207277483,
+            -0.007305441146142331,
+            0.00965056093889974,
+            0.016698589600806273,
+            -0.003330867073501157,
+            -0.025198703196464594,
+            0.030483790406960656,
+            0.008872821868559751,
+            0.023708422767352917,
+            -0.021464374508407488,
+            -0.0018617153194555344,
+            -0.00886376784514331,
+            0.014505918224188001,
+            -0.017588761818663975,
+            -0.011756112651591795,
+            0.020014632992762026,
+            0.014356419813769714,
+            0.007246557500044434,
+            -0.004355132555323843,
+            0.0067299398462013936,
+            0.007649499235136303,
+            -0.012515181210363267,
+            0.012239792251154642,
+            -0.013499277313916029,
+            -0.024858295126228433,
+            0.014430561986516884,
+            0.002229494935179828,
+            0.009563004428064055,
+            -0.0015984171085659323,
+            -0.016197659608768088,
+            0.010078159490026602,
+            0.00808729175515505,
+            -0.017508840440512225,
+            -0.004793934415922065,
+            0.007133527615305415,
+            -0.020818869260580315,
+            0.031234686259160864,
+            0.011872765152279094,
+            -0.010692949770503467,
+            -0.001989039629661815,
+            0.004431079028880923,
+            0.016511544608975148,
+            0.007893700385963683,
+            -0.01997314388229276,
+            -0.003708006454566117,
+            0.01530128306203401,
+            0.00822588305477249,
+            0.004861703074728471,
+            -0.010836328783594584,
+            0.03434874763691911,
+            0.006082873924472328,
+            4.192481378506604e-05,
+            -0.0052997095110967115,
+            -0.0076108924981838355,
+            0.004196112457289409,
+            0.018905391806961686,
+            0.013330078288599763,
+            -0.004200590783114919,
+            -0.006630107527854746,
+            0.007778448128358027,
+            -0.01706991160178345,
+            0.007329056462125937,
+            0.009390327499494826,
+            -0.012870674959022891,
+            -0.0017571189976160295,
+            0.016462608870101713,
+            0.00209846661139665,
+            0.004942714850185376,
+            -0.011305050739130339,
+            0.004252776146827917,
+            0.015841821944109317,
+            0.018727567181082646,
+            -0.008399642221255733,
+            0.021448055799259867,
+            -0.020972459569228434,
+            0.027243639465535856,
+            0.028499400850980086,
+            0.016573144403275928,
+            -0.01847096443459798,
+            -0.0022409373140116146,
+            -0.0041737366139895414,
+            -0.014761716560267328,
+            0.0034546020161337894,
+            0.005125870074458691,
+            -0.01174186608185822,
+            -0.009577921232380895,
+            -0.024331993794416026,
+            0.011075959943215117,
+            0.007918147902741738,
+            -0.0015045185909419564,
+            -0.011791155919533844,
+            0.016019077577552835,
+            0.010141868950981925,
+            -0.010938402325619172,
+            -0.005635351352447256,
+            -0.019967361901734836,
+            0.009750427567651833,
+            0.009128294017175602,
+            -0.00300556760891084,
+            -0.011496103417643132,
+            0.012977890524465716,
+            0.0014080059843484115,
+            -0.01292380658817521,
+            0.010841954636959441,
+            -0.03142011651894961,
+            -0.005510537354374977,
+            -0.011027026734432255,
+            -0.003620482919145167,
+            -0.014415606143566519,
+            0.003608572766363589,
+            0.0005034611321129689,
+            -0.011470488351158329,
+            -0.0035005774696278,
+            -0.016207453351921627,
+            0.004676327282262429,
+            0.015021513761697039,
+            0.001679614645967629,
+            -0.006917375094634678,
+            0.021994274901326064,
+            0.018518524483306403,
+            -0.021903538417919984,
+            -0.020116013379669034,
+            0.004708084983787958,
+            -0.03261690161800251,
+            0.015019348187107588,
+            0.00966124471577622,
+            0.011676937880751252,
+            -0.028768229226408037,
+            0.007114534979415504,
+            0.00900420165933673,
+            0.016398927481511824,
+            0.0050879301160439245,
+            0.009453255486450082,
+            0.01222758288108498,
+            -0.0104657614414275,
+            -0.003164502778525433,
+            -5.410382753651127e-06,
+            -0.005618225986727351,
+            0.015607620507415484,
+            0.0008527697935056332,
+            0.03603666488542239,
+            -0.005277207843139309,
+            -0.0009799879252482288,
+            -0.011735731235560851,
+            0.00812046801749312,
+            -0.011713349784345638,
+            0.006334172347602515,
+            -0.0024643499727789167,
+            0.0001240009282616789,
+            0.002965067818077541,
+            0.009442572075213767,
+            -0.01086095783353607,
+            -0.007595379345224076,
+            0.007444897256325813,
+            -0.01482814340478338,
+            -0.00019984717819268013,
+            -0.009339343730980451,
+            0.0019680804106275156,
+            -0.02241031895876346,
+            0.006629838013824249,
+            0.040935644507527846,
+            0.02140216465404127,
+            -0.005481040712269146,
+            -0.012431659209559418,
+            -0.004680230194268222,
+            -0.007229300034633229,
+            -0.006482401275991039,
+            -0.004794698160419071,
+            -0.022762405726589375,
+            -0.006928553505112732,
+            0.004320321244630314,
+            0.013300138724212552,
+            0.009741018117644266,
+            -0.008842135469729511,
+            0.004531760574809748,
+            -0.02662650543210291,
+            -0.016578081411548973,
+            -0.0060021425889846495,
+            -0.015840130031005147,
+            0.006125301421471048,
+            0.027521841635711787,
+            -0.0063432924325983525,
+            0.01789356321527744,
+            0.005142029893259461,
+            0.013034833065163448,
+            -0.005338869351330258,
+            0.00035250243782976347,
+            -0.006010938630154189,
+            0.0032958656042048292,
+            0.003074861351445954,
+            -0.0015947217852680714,
+            0.004151590621941346,
+            -0.007139285822575166,
+            -0.0026091937868967284,
+            0.0011629380311555433,
+            0.0117841884590861,
+            0.006877419488576261,
+            -0.00016252653621991807,
+            -0.01761285866765538,
+            0.022221104524808997,
+            -0.018933508559881868,
+            -0.008840161102111973,
+            0.011092684029552064,
+            -0.009660487856185668,
+            0.00702597023411946,
+            0.020387386977378873,
+            -0.006854213562767843,
+            -0.009583981919303215,
+            0.013028181087649146,
+            -0.0036507672754997286,
+            0.004464368543493108,
+            0.009867428978560914,
+            0.006036303973414791,
+            0.002482944923100156,
+            0.013244650094892797,
+            0.004530294330688855,
+            -0.0032625482538717137,
+            -0.020238533752799943,
+            -0.005186498720507101,
+            0.019015752262933585,
+            -0.02252458872825473,
+            -0.0027641215988942563,
+            0.0044506144202026456,
+            -0.0030317840657635457,
+            -0.01733962766050954,
+            0.004205861489436576,
+            -0.018770275720867473,
+            -0.0033883519277721433,
+            -0.008244668812709337,
+            0.009948090119054079,
+            -0.023700732627507225,
+            0.009109315916173436,
+            -0.011739192106106307,
+            0.02356509175177699,
+            -0.003925851077921686,
+            0.024734433836185024,
+            0.0015142645677087268,
+            -0.013482933030628261,
+            0.0055323884713736554,
+            -0.0008930661843169134,
+            -0.004477731568313757,
+            0.029338490404300612,
+            -0.005418258701133154,
+            0.002249996261871466,
+            0.013491124734151694,
+            -0.021332541202417246,
+            0.021773687217263173,
+            0.0046662696076723836,
+            0.01628697326539432,
+            -0.00986238951830334,
+            -0.013799209313714441,
+            0.025178874553620743,
+            0.03324555035648896,
+            0.008398796308070151,
+            0.011261509865513953,
+            -0.01226890254177634,
+            0.018966175575083295,
+            -0.0038328997984309597,
+            0.004344837325292706,
+            0.00878668880458693,
+            -0.007200842513742962,
+            -0.011500562922303248,
+            -0.004533014670236755,
+            0.027639894892383943,
+            -0.021838848944317424,
+            -0.008920629621611768,
+            -0.005739181836252042,
+            -0.01593272996094732,
+            -0.038167534267554606,
+            -0.004371701247332816,
+            0.019604405563120094,
+            -0.005974358141737469,
+            -0.0018341151594086288,
+            0.014334266690716505,
+            0.00402620072221496,
+            -0.002852569009185304,
+            -0.008195218707838913,
+            -0.0027944153624309856,
+            -0.0059390492900516225,
+            0.016651206007022145,
+            -0.004961864528562493,
+            -0.007929830585906699,
+            0.01145431100211762,
+            -0.005731957927601062,
+            0.012130694936714224,
+            0.0004848418056781251,
+            8.392048424584131e-05,
+            -0.0015174120840537315,
+            0.00822932723013239,
+            0.018620869354789082,
+            -0.0055803454666417465,
+            0.025682778991091174,
+            0.006956936033581638,
+            -0.003495709501259274,
+            -0.005489431577052735,
+            -0.0044154293119170445,
+            -0.024561016942073285,
+            -0.0020864636079347357,
+            0.006593060072668424,
+            0.024333813203596122,
+            -0.02515989187381735,
+            -0.004953266000463339,
+            0.020957806164031402,
+            -0.0012192724005613,
+            -0.011978722479596745,
+            0.014690559986694138,
+            0.0041346025614047875,
+            0.009881584250499358,
+            0.012176393203318002,
+            0.015764530703333238,
+            -0.004923548157246672,
+            -0.0015192775000207833,
+            0.012766482928380375,
+            0.0005745973040874939,
+            0.00997809510696241,
+            0.016495444272181515,
+            0.010467514633877887,
+            -0.021108281220544853,
+            0.0015078843789125156,
+            0.00522593579521476,
+            -0.017481430635971396,
+            0.0034362821987740755,
+            0.005397819166037461,
+            0.011626527721895694,
+            0.004048632382026344,
+            0.013207335578216437,
+            -0.015768003434275563,
+            -0.005916168437120099,
+            0.008153798523776654,
+            0.004319451296835019,
+            -0.02599474886669325,
+            -0.01485009085106089,
+            -0.0006781964116146299,
+            -0.004444144921332929,
+            0.013556265879295017,
+            0.01285768162554067,
+            0.02033626001285354,
+            0.024637349990579514,
+            -0.013833060730799217,
+            -0.00842819657792762,
+            0.009436072455719896,
+            0.015560357465711765,
+            0.005798181411384387,
+            0.007874051721372273,
+            0.009297440071118126,
+            0.0077320823375042,
+            -0.004073209576661068,
+            -0.002973928843567532,
+            0.020205003097225747,
+            -0.019725382393637726,
+            0.0006746659838642147,
+            -0.018417443693472958,
+            0.007530796896254817,
+            0.01140696723506382,
+            0.01129515042745398,
+            0.002816883643872417,
+            -0.005274757308627299,
+            0.005716917047812153,
+            -0.06254973656832136,
+            -0.009191453047502128,
+            0.003384751551032943,
+            0.027256466708746898,
+            -0.010885336140129142,
+            -0.003415707910562931,
+            0.007309577090125959,
+            0.005511970708769456,
+            0.0005572239548625122,
+            0.003186427278034852,
+            0.019698716902282903,
+            -0.02052581021967601,
+            -0.010793968505805883,
+            0.004219269922138548,
+            0.0033111837923746746,
+            -0.007269521534951044,
+            0.004805737092302134,
+            0.014143978208501908,
+            0.01986656063456352,
+            -0.0024048854610842748,
+            0.01629335842915328,
+            -0.0015498603704589015,
+            -0.014496994444493684,
+            -0.002671064325909516,
+            0.0037875833047555925,
+            -0.022466433279066646,
+            -0.005980897597195858,
+            0.0043262697442044435,
+            -0.011226121648007667,
+            0.006978495397818936,
+            0.008452177557316863,
+            -0.003018545465557218,
+            -0.0008303307644324117,
+            0.017796789945838845,
+            0.006591364879905198,
+            -0.013464486184025026,
+            0.0028668426952326957,
+            -0.023377605222213174,
+            0.004325185794796361,
+            0.001416245546006914,
+            0.007809449561523174,
+            -0.008431335139468173,
+            -0.014774499666285603,
+            -0.005933810712528226,
+            0.017746847020610673,
+            -0.01965188730040385,
+            -0.002406875192894708,
+            0.01683009662984443,
+            -0.0002518220110313279,
+            0.005456812494894182,
+            -0.02011241622618092,
+            -0.0009782974741561381,
+            -0.02933296756078644,
+            -0.01171823509606186,
+            0.001162345317872531,
+            0.00892221099991475,
+            -0.006551308022163148,
+            -0.03934356275100932,
+            0.00427623835776202,
+            -0.013111664387776261,
+            -0.008752857644809231,
+            -0.010214546496134218,
+            -0.00518087722852466,
+            0.015240297918667867,
+            -0.012498100033235827,
+            0.01086490299272505,
+            -0.00917777212602538,
+            0.005895055740265625,
+            0.0090554651297836,
+            0.014212841798768113,
+            -0.005544047915369086,
+            -0.0033589999440643385,
+            -0.007924830667307956,
+            0.0022291797354234005,
+            -0.0032976226056924502,
+            0.009615674261248298,
+            -0.0051932155872883,
+            -0.0065636389655770455,
+            0.02426106498596339,
+            0.011445591067574932,
+            0.013312596099532533,
+            0.0055118624272110965,
+            -0.007220620072349783,
+            -0.01916551187613823,
+            -0.003977616240384473,
+            -0.013204506708246246,
+            -0.006315966975649096,
+            0.01434099854765068,
+            0.012588249778961223,
+            0.006357972880218866,
+            0.0007747002406996095,
+            -0.004794439624135275,
+            0.006975593520375694,
+            0.0262335496080837,
+            -0.016481909971520166,
+            -0.0013084662210086592,
+            -0.01369313900978409,
+            0.01921748474212319,
+            0.03333761672336437,
+            -0.0008918419419301244,
+            -0.007699525186523558,
+            0.013757216065348164,
+            0.00819045303047094,
+            -0.007411380297164911,
+            0.02453611080891859,
+            -0.025045462880463647,
+            -0.015989319900023054,
+            -0.011650886878708265,
+            -0.011407253985176972,
+            -0.008386367959227559,
+            0.016331961716602287,
+            -0.00141225438918211,
+            -0.0018505754298268654,
+            0.009557291617446645,
+            0.03357924917013024,
+            0.006916860901290756,
+            0.007030308953738567,
+            -0.002739345982346728,
+            -0.009376522438071562,
+            0.011482637968411055,
+            -0.007259054391879614,
+            -0.04075661435928125,
+            0.004250195164583401,
+            -0.005160505473755424,
+            0.02405623169060588,
+            -0.02197586561375289,
+            -0.00619175761282745,
+            -0.006736129049831973,
+            -0.028200015908139763,
+            -0.005867722104196303,
+            0.02805071895018145,
+            -0.015402198567775478,
+            0.014773294766214671,
+            -0.009502185346558072,
+            -0.0013419858580438616,
+            0.004443749010136947,
+            0.013008783570355656,
+            -0.02289662952366337,
+            0.0013834888740540108,
+            -0.00048549197205067054,
+            0.027969842651517314,
+            0.010437729090975158,
+            -0.0047705656249623744,
+            0.014202218020412936,
+            0.008240861154198411,
+            -0.018637315941614406,
+            0.007314019317343154,
+            -0.005747966765101792,
+            0.014833896739762147,
+            -0.01314847940891154,
+            0.007762100494499689,
+            0.00705026393499352,
+            -0.01278023669088385,
+            0.03016342404000438,
+            -0.015204129985039077,
+            0.007776946006562982,
+            0.009278778571285689,
+            0.009656723862179986,
+            0.01257686993982689,
+            0.004362439504889069,
+            -0.02898418193036562,
+            0.0002144431781454467,
+            -0.03828100004144633,
+            -0.00833748467467171,
+            -0.002611937107022567,
+            -0.002145329666219461,
+            0.007050630951707937,
+            -0.009383645162261629,
+            -0.033976442057557676,
+            0.0006429978367640709,
+            0.01603027598131823,
+            0.017501067413108286,
+            0.0002196997337451293,
+            0.013075379412332973,
+            -0.006197537539554383,
+            -0.021148120787551232,
+            -0.016986448660412497,
+            -0.010207705210051413,
+            -0.02122194744719993,
+            -0.020550470201191853,
+            -0.00908564461857315,
+            0.012090079127547193,
+            -0.009927390005879807,
+            0.013888458967497627,
+            0.029513250508042935,
+            -0.024529175134769883,
+            0.0011144447737101862,
+            0.021977677894839922,
+            -0.00198308986575084,
+            0.014108189311291326,
+            0.001696775006906642,
+            0.014122064251288995,
+            -0.0004166140484018266,
+            -0.010556016685519523,
+            -0.009729362492661684,
+            0.017487929296614076,
+            -0.0026043257634797124,
+            0.016715798948897194,
+            0.012183487338360088,
+            0.003771721868999568,
+            -0.0030887189945795796,
+            0.010667776697461199,
+            -0.026471157050402344,
+            0.014249996582064706,
+            -0.006690961709923224,
+            -0.0181382691440242,
+            -0.008729104191096461,
+            0.003882731567105777,
+            -0.005609384377642956,
+            -0.007794812949125115,
+            0.012967089730551055,
+            -0.00944374111323783,
+            0.02088525981049842,
+            0.0023400121055514958,
+            -0.012196925240774426,
+            0.003648688547842322,
+            -0.03322041209304968,
+            0.0015191765655963997,
+            -0.018402382776013956,
+            -0.024659631472836927,
+            0.03840173429666435,
+            0.011991452465253198,
+            0.006639702644606365,
+            -0.01675838115087676,
+            0.007953102302963526,
+            0.0004257479565099406,
+            0.017484338819046517,
+            0.011068864013925452,
+            0.006466915839538296,
+            0.008364454017976605,
+            -0.023316137291380285,
+            -0.010898421318668592,
+            0.016204569093730564,
+            0.012099473359791148,
+            -0.01017229369980751,
+            0.019431669546223253,
+            0.017955119269367787,
+            0.007300372071724306,
+            -0.006898619924677706,
+            0.01647927363640854,
+            0.021012103522831126,
+            0.010291016618557653,
+            -0.00047955552103474145,
+            0.0008684297441778696,
+            -0.010534247928997246,
+            0.005037930235889528,
+            0.035417626740307205,
+            0.014712595692093886,
+            0.010650306363963653,
+            -0.03626584475133921,
+            0.021764456453493054,
+            -0.011961521673661574,
+            0.009092981192874177,
+            0.006089104887183293,
+            0.017982801101846038,
+            0.006916969397519662,
+            0.014196254958707752,
+            0.011745069030657836,
+            -0.006689305450954901,
+            0.025205280281804708,
+            -0.001601414806100461,
+            -0.011992525142592405,
+            0.003386930324498371,
+            -0.01859193251438437,
+            0.005303261319773131,
+            -0.0041710536965100615,
+            -0.004459393120257307,
+            0.0032461828144552565,
+            -0.0018650956576455287,
+            0.015678225251799887,
+            0.003894142628084471,
+            -0.02355212701026675,
+            -0.0014873755115206368,
+            -0.024284295126348117,
+            0.0027260592623277375,
+            -0.022185571998195344,
+            0.00022883605560765203,
+            -0.005557568353382136,
+            -0.009749382813201403,
+            -0.025385857431333838,
+            -0.009853073898348951,
+            -0.0035843255959309307,
+            0.007661137888845632,
+            -0.010521037433807653,
+            -0.004246698084422667,
+            -0.0019920499321964815,
+            -0.019465113363548715,
+            0.009014255846983305,
+            0.027122533756959888,
+            0.016848221859975445,
+            0.027170883580103264,
+            -0.00019554112591107156,
+            0.013614675849485926,
+            -0.002066822813103284,
+            -0.01603063005944364,
+            0.008824663429777616,
+            0.0075653522381218495,
+            -0.00531780740880296,
+            0.014643348459785112,
+            0.0072634484937344704,
+            -0.012800149015319017,
+            0.009859503660384916,
+            -0.007161712176469609,
+            -0.006437857349742842,
+            -0.009764856421740622,
+            -0.02058868349990357,
+            -0.014866137971603676,
+            0.0012450207572602557,
+            -0.002123265406747006,
+            0.006000473862248991,
+            -0.004054016790850226,
+            0.009584879012937315,
+            -0.00608816807208866,
+            -0.00925801057439376,
+            -0.013962683072792106,
+            -0.008924356118158748,
+            -0.004210064145949253,
+            -0.004381709838058668,
+            -0.011732699976353612,
+            -0.005479523097469261,
+            0.0010680258077037758,
+            -0.0019401640391035755,
+            -0.01397652201595301,
+            0.011650839243603377,
+            0.0058731838291845645,
+            -0.002462743134993977,
+            0.02219008688925881,
+            0.01097316136828334,
+            0.0018083001685932406,
+            0.01366095798812196,
+            0.0003942601242216818,
+            -0.0064063532958756175,
+            -0.03343312148366622,
+            -0.0005083355836126291,
+            -0.02815586367527057,
+            0.025770078162165123,
+            -0.0069114949102666145,
+            -0.010322087105114614,
+            -0.002519047562374146,
+            -0.006063669644021698,
+            0.005873609611264377,
+            -0.002722940652283699,
+            0.0021229525800901114,
+            -0.023673778230768945,
+            0.0028629715379660113,
+            -0.003953653437464966,
+            -0.0036366957271968823,
+            0.007469664864329332,
+            0.0346520690389124,
+            -0.004054282838167587,
+            -0.0034765253405897676,
+            0.005159466707114532,
+            0.011642913651743502,
+            0.027845571220425726,
+            -0.004445196354520585,
+            -0.0032038508043339413,
+            -0.02233838584152591,
+            0.0054112576531374175,
+            -0.00019544838850471866,
+            -0.004733192299951088,
+            0.016664217348411003,
+            -0.016719175418727594,
+            -0.01631658832047864,
+            -0.00545969677717507,
+            -0.014570848053102338,
+            -0.00022620258237138677,
+            0.014454101124666536,
+            0.011056312437637014,
+            -0.004035526363810078,
+            -0.012459378363360557,
+            -0.0022595493558020706,
+            -0.030370952502579162,
+            0.012766010284884204,
+            0.1141689619126009,
+            -0.00766197047210299,
+            -0.00896655658602649,
+            -0.009481112393590025,
+            0.001435269335646347,
+            0.001338755472877713,
+            -0.011739143028767118,
+            0.07166096084053437,
+            -0.0077855767070652395,
+            0.0002139580430342728,
+            -0.011312198831181402,
+            0.001844005266872301,
+            -0.001825412693163496,
+            0.01371932421024605,
+            -0.005025814503687022,
+            -0.018213360991353643,
+            -0.0033612481616769416,
+            -0.02941403830641238,
+            -0.0041137865872933515,
+            -0.005683630256743215,
+            -0.004288198570156578,
+            -0.014982999363445122,
+            0.0067149665791261335,
+            -0.019661973497372313,
+            -0.012688556129040908,
+            0.001124391650175233,
+            -0.0033623453063430534,
+            0.00996944383866679,
+            -0.020496336814603385,
+            -0.0044354321069609536,
+            0.0019875840782527733,
+            0.009706675240563883,
+            0.0011600364428219214,
+            -0.012755178230821602,
+            -0.0026514442584739412,
+            0.004056350166031111,
+            0.01909951327913368,
+            0.014360281326789544,
+            0.003732344102045803,
+            0.008623457695718206,
+            0.014461868085333023,
+            -0.001692149528228451,
+            0.01948035376572433,
+            0.01172550239477637,
+            0.0031966531144076173,
+            0.0004779413449255626,
+            0.0030912562273611517,
+            -0.004597633235071031,
+            0.01582547400214653,
+            -0.014811177637844808,
+            -0.02411202772171917,
+            -0.0071272238837422185,
+            0.018015541137524113,
+            -0.013081287852028074,
+            1.8837689271256874e-05,
+            0.004062803284646432,
+            0.027569726868200254,
+            0.0010564850664255057,
+            -0.01179755720247972,
+            0.0028876134800690334,
+            -0.01638215925131053,
+            0.022944183106912663,
+            0.0026043713064243836,
+            0.03279173899194346,
+            -0.005103903273968361,
+            0.00578213669463383,
+            -0.020744194169889544,
+            -0.007588793365216579,
+            -0.00819862611489168,
+            0.0017401442433921013,
+            0.010993500174794084,
+            0.005411238350013134,
+            0.01620925462604661,
+            -0.00641615069699833,
+            -0.020751730783437995,
+            0.014776878596707538,
+            -0.009552193304486048,
+            -0.03393319554458277,
+            0.008881987295131347,
+            -0.029102297123590998,
+            -0.0045267103602711,
+            0.00368971742585232,
+            -0.014269341759285646,
+            -0.016264809810970175,
+            0.012837302095591106,
+            -0.0011890334180728564,
+            -0.008937992768323412,
+            0.0014444613725248846,
+            -0.017110187133760626,
+            -0.02385668037818897,
+            0.006017405364919834,
+            0.00676670672597817,
+            -0.0077994500447595,
+            0.018312452858615755,
+            -0.027015660037024585,
+            0.006123912473734307,
+            -0.012973676426738582,
+            0.009041974762537394,
+            -0.01580183887345168,
+            0.01163516332546347,
+            0.010887662470499361,
+            -0.001503467664709027,
+            -0.0076234124970223326,
+            0.01231202541338883,
+            0.004852720134894811,
+            0.033760365210093034,
+            0.028658326381800395,
+            0.000995999783627898,
+            -0.0025605367058226737,
+            -0.0020356794893366384,
+            0.011667525096213252,
+            -0.014049700287539817,
+            0.011218246246280464,
+            0.02658113044439717,
+            0.017568631325838108,
+            -0.019133062654008885,
+            0.018050678772708717,
+            -0.010356118256625172,
+            0.012872052420999813,
+            -0.025620154415048252,
+            -0.005135768485044442,
+            0.004936490920616527,
+            0.005163008729256001,
+            0.007425278892536464,
+            0.001410212306439329,
+            0.021891387289347516,
+            0.014999616600766989,
+            0.00027613811156650413,
+            0.00558422822386443,
+            0.01477009526981092,
+            0.010497782599293994,
+            -0.0005567894749233204,
+            -0.00467776506604011,
+            -0.016998035358377343,
+            0.00957271797745408,
+            -0.006065116857898469,
+            0.005138070401846389,
+            0.01698914855850691,
+            -0.01230257809343985,
+            0.011109844802663414,
+            0.007518553660463051,
+            -0.015504773927595823,
+            -0.017169933945839305,
+            0.005552829003043926,
+            -0.02252633790984703,
+            -0.01306722099657118,
+            0.02715689072730398,
+            -0.010513941399990838,
+            -0.02790582732974509,
+            -0.0009847768069205477,
+            0.009797865241108237,
+            -0.002772952073769185,
+            -0.0027968741526000576,
+            0.0037541733651479613,
+            0.010801137722284016,
+            -0.014658156970099418,
+            0.002075415233345957,
+            0.004735264309137368,
+            0.01930804315461151,
+            -0.0014151522239496115,
+            -0.015046118414100617,
+            -0.002288495894227231,
+            0.021592829677937263,
+            0.017174609713578173,
+            0.00265560378681856,
+            -0.016499825201963983,
+            -3.965296928410772e-05,
+            -0.0074142747014567635,
+            0.00021360696629503726,
+            0.00027431174904249525,
+            -0.0056391089282546054,
+            -0.018931681700770225,
+            -0.003888248121485032,
+            0.011181675419027736,
+            -0.00247252920175839,
+            0.01977860312223846,
+            -0.0011527575128223392,
+            -0.015763587433200915,
+            0.001960734212846797,
+            -0.005280860341480037,
+            -0.010372829570478649,
+            0.01351900743232294,
+            -0.009359942374603997,
+            0.006788746813046239,
+            -0.003212015225284119,
+            0.004611143085183367,
+            -0.0003308099698088422,
+            -0.017017265064185255,
+            0.02924142709831441,
+            -0.0018238549856433466,
+            -0.004769274041546295,
+            -0.0014058363976586562,
+            -0.033393331166741945,
+            0.015017165991303838,
+            -0.014574579250749552,
+            0.006745699409061803,
+            0.002705853802154516,
+            0.01272741296341735,
+            0.003253095246601883,
+            -0.008898616461779179,
+            0.0027485606928249326,
+            0.017040360927723885,
+            0.013756402798365262,
+            -0.007292864293090161,
+            0.008694855886257706,
+            -0.0028934389922887676,
+            -0.0037483208383542015,
+            0.01363963339139356,
+            0.004170611039133969,
+            -0.006282657671540625,
+            -0.0009784372731140945
+          ],
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "Mixture-of-Experts (MoE)",
+            "fine-grained expert segmentation",
+            "shared expert isolation"
+          ],
+          "title": "Deepseekmoe: Towards Ultimate Expert Specialization In Mixture-Of-Experts Language Models"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "c6bf0f92-84c8-44b9-8b28-7fcab6058203",
+        "properties": {
+          "page_content": "8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n",
+          "keyphrases": [
+            "Mixture of Experts (MoE)",
+            "language model training",
+            "DeepSeekMoE architecture",
+            "expert-choice routing strategy",
+            "training instability"
+          ],
+          "title": "Related Work",
+          "embedding": [
+            -0.0004454151203390211,
+            -0.01349730882793665,
+            -0.018033159896731377,
+            0.005232766270637512,
+            0.01874188706278801,
+            0.010268663987517357,
+            -0.011331754736602306,
+            0.03266443312168121,
+            -0.018993878737092018,
+            0.018789134919643402,
+            -0.005878495052456856,
+            -0.009221323765814304,
+            0.007355009205639362,
+            -0.008567719720304012,
+            0.0011782583314925432,
+            0.02784508839249611,
+            0.018474144861102104,
+            0.015410871244966984,
+            0.002397859003394842,
+            -0.03512135148048401,
+            0.019734105095267296,
+            -0.02189178392291069,
+            -0.044067058712244034,
+            -0.013772924430668354,
+            0.014253283850848675,
+            0.01693069562315941,
+            0.03858623653650284,
+            -0.006386416032910347,
+            -0.05959605053067207,
+            0.03386139124631882,
+            0.020521579310297966,
+            0.02291550114750862,
+            -0.01582823134958744,
+            -0.014930511824786663,
+            -0.02899480052292347,
+            -0.013481559231877327,
+            0.05077634006738663,
+            0.021734289824962616,
+            -0.016410963609814644,
+            0.027687594294548035,
+            0.0467129722237587,
+            0.013268941082060337,
+            -0.02280525304377079,
+            -0.0008283244678750634,
+            0.005512319505214691,
+            0.0011831800220534205,
+            -0.019009627401828766,
+            -0.016332216560840607,
+            0.010497031733393669,
+            -0.028175828978419304,
+            -0.018883632495999336,
+            -0.027278108522295952,
+            -0.006665969267487526,
+            0.0006486818892881274,
+            -0.0642578974366188,
+            0.007378633599728346,
+            0.01752917654812336,
+            0.03389289230108261,
+            0.05058734491467476,
+            0.0015080131124705076,
+            0.012308222241699696,
+            -0.019419115036725998,
+            -0.01686769910156727,
+            -0.008260604925453663,
+            -0.00022885970247443765,
+            -0.01097739115357399,
+            0.012016857042908669,
+            -0.013324064202606678,
+            0.01982860080897808,
+            0.016221968457102776,
+            -0.040854163467884064,
+            0.027026114985346794,
+            -0.023876218125224113,
+            0.009426066651940346,
+            -0.006969146896153688,
+            0.007095142733305693,
+            0.019135624170303345,
+            0.026600880548357964,
+            -0.0019509674748405814,
+            -0.007252637762576342,
+            0.0025514166336506605,
+            0.017324432730674744,
+            -0.007122704293578863,
+            0.0022108338307589293,
+            0.05014635995030403,
+            -0.04687046632170677,
+            -0.02584490366280079,
+            -0.07049469649791718,
+            0.00882758665829897,
+            0.00037503460771404207,
+            -0.024789689108729362,
+            -0.003289673710241914,
+            0.028490817174315453,
+            0.008835460990667343,
+            -0.040602169930934906,
+            0.06312393397092819,
+            -0.014056415297091007,
+            -0.015662861987948418,
+            -0.03512135148048401,
+            -0.008355101570487022,
+            0.0005320372874848545,
+            -0.0049768369644880295,
+            0.011575871147215366,
+            0.010709649883210659,
+            0.012197976000607014,
+            -0.012812205590307713,
+            -0.033199913799762726,
+            0.04687046632170677,
+            -0.005713125690817833,
+            0.08523621410131454,
+            0.0021379925310611725,
+            -0.009945799596607685,
+            -0.003252268536016345,
+            0.03537334129214287,
+            -0.021435048431158066,
+            -0.008386600762605667,
+            -0.028349071741104126,
+            -0.04935888573527336,
+            0.024852687492966652,
+            -0.02348248101770878,
+            0.018631640821695328,
+            -0.0007800916791893542,
+            0.016395213082432747,
+            -0.019214371219277382,
+            -0.0721326395869255,
+            0.05739112198352814,
+            -0.0026774124708026648,
+            -0.015111630782485008,
+            -0.003021932439878583,
+            0.007201451808214188,
+            -0.0008194653782993555,
+            0.0037345965392887592,
+            -0.01735593192279339,
+            -0.019088376313447952,
+            0.029136545956134796,
+            0.023513980209827423,
+            0.03861773759126663,
+            0.046240486204624176,
+            -0.020301084965467453,
+            -0.01452889945358038,
+            0.023734472692012787,
+            -0.029514534398913383,
+            0.006630532909184694,
+            0.011567996814846992,
+            -0.01746617816388607,
+            0.011599495075643063,
+            0.011662493459880352,
+            0.06967572122812271,
+            -0.019860100001096725,
+            0.017009442672133446,
+            0.0022895813453942537,
+            0.0643208920955658,
+            0.0025435418356209993,
+            0.030475253239274025,
+            -0.014765141531825066,
+            0.02178153768181801,
+            -0.028160078451037407,
+            0.0025061366613954306,
+            -0.020222337916493416,
+            -0.003464886685833335,
+            -0.002510074060410261,
+            -0.04094865918159485,
+            -0.002821126487106085,
+            0.028853056952357292,
+            0.0015119505114853382,
+            0.01776541955769062,
+            0.033199913799762726,
+            0.003850748995319009,
+            -0.05962754786014557,
+            0.023198990151286125,
+            0.03452287241816521,
+            0.010985265485942364,
+            -0.009103202261030674,
+            0.021624041721224785,
+            0.011056138202548027,
+            0.010575778782367706,
+            0.02067907340824604,
+            0.008189732208848,
+            -0.022710757330060005,
+            0.03972020000219345,
+            -0.02886880561709404,
+            0.0018830477492883801,
+            0.03228644281625748,
+            0.0016743671149015427,
+            0.044791534543037415,
+            0.012906703166663647,
+            0.026852872222661972,
+            0.01693069562315941,
+            0.012316097505390644,
+            0.029939770698547363,
+            0.007166015449911356,
+            0.010536405257880688,
+            0.03540484234690666,
+            -0.015095881186425686,
+            0.008158233016729355,
+            -0.02255326136946678,
+            -0.012867328710854053,
+            0.006488787941634655,
+            0.00988280214369297,
+            -0.005815497133880854,
+            -0.0021753974724560976,
+            -0.04201962426304817,
+            0.02212802693247795,
+            -0.0055831922218203545,
+            0.021986281499266624,
+            0.0026144145522266626,
+            0.012615337036550045,
+            0.007630625274032354,
+            0.03559383377432823,
+            -0.03852323815226555,
+            -0.011788489297032356,
+            -0.007878679782152176,
+            0.005535943899303675,
+            -0.044791534543037415,
+            -0.020033344626426697,
+            0.016725953668355942,
+            -0.004776031244546175,
+            0.021135808899998665,
+            0.009276446886360645,
+            -0.02159254439175129,
+            -0.006626595743000507,
+            -0.006445476785302162,
+            0.018363898620009422,
+            -0.01533999852836132,
+            -0.030491001904010773,
+            -0.0351843498647213,
+            0.01300907414406538,
+            0.011536497622728348,
+            0.03590882569551468,
+            -0.0009178996551781893,
+            -0.020600326359272003,
+            0.014788766391575336,
+            0.001092620543204248,
+            0.0039216214790940285,
+            0.02274225652217865,
+            -0.04646097868680954,
+            0.00845747347921133,
+            0.0026537880767136812,
+            0.011520748026669025,
+            0.02085231803357601,
+            0.007394383195787668,
+            0.017182687297463417,
+            -0.035751331597566605,
+            0.004086991306394339,
+            0.03688529506325722,
+            0.011638869531452656,
+            -0.016103848814964294,
+            0.028065582737326622,
+            4.574118065647781e-05,
+            -0.012883078306913376,
+            -0.0033290472347289324,
+            -0.03222344443202019,
+            -0.012623212300240993,
+            0.0008617921266704798,
+            -0.016757450997829437,
+            0.05269777774810791,
+            0.043090589344501495,
+            -0.019324617460370064,
+            -0.014631271362304688,
+            -0.05918656289577484,
+            0.039688702672719955,
+            -0.02408096194267273,
+            -0.025262173265218735,
+            0.04416155442595482,
+            -0.016725953668355942,
+            0.020159341394901276,
+            0.032475437968969345,
+            0.012820080853998661,
+            -0.00011873635230585933,
+            -0.03527884557843208,
+            -0.004051554948091507,
+            -0.04523251950740814,
+            -0.03801925480365753,
+            0.005126457195729017,
+            0.03170371428132057,
+            -0.03559383377432823,
+            -0.0055792550556361675,
+            -0.009481189772486687,
+            0.030191762372851372,
+            0.028837306424975395,
+            0.02049008011817932,
+            -0.0016379464650526643,
+            0.007969239726662636,
+            0.016379464417696,
+            -0.03208170086145401,
+            0.013394936919212341,
+            -0.0046460977755486965,
+            -0.026522131636738777,
+            -0.010308037512004375,
+            -0.015749484300613403,
+            -0.006760466378182173,
+            -4.174228524789214e-05,
+            -0.04693346470594406,
+            0.00440985569730401,
+            0.03540484234690666,
+            -0.0037050663959234953,
+            -0.01795441284775734,
+            0.03323141112923622,
+            0.033451907336711884,
+            -0.027860838919878006,
+            -0.06167498230934143,
+            -0.001828908920288086,
+            0.021450798958539963,
+            -0.0024529823567718267,
+            -0.05616266280412674,
+            -0.003445199690759182,
+            -0.008276354521512985,
+            0.030554000288248062,
+            -0.01710394024848938,
+            -0.04201962426304817,
+            0.0032680181320756674,
+            -0.0005128426128067076,
+            -0.013481559231877327,
+            -0.03433387726545334,
+            -0.00764243770390749,
+            0.034365374594926834,
+            0.01437140442430973,
+            0.025813406333327293,
+            -0.04384656623005867,
+            -0.02929404191672802,
+            0.01270983461290598,
+            0.0394052118062973,
+            -0.06284044682979584,
+            0.010308037512004375,
+            -0.014292657375335693,
+            -0.001888953847810626,
+            0.004728782922029495,
+            0.043878063559532166,
+            -0.03291642293334007,
+            0.009016579948365688,
+            -0.00609505083411932,
+            -0.04167313501238823,
+            -0.034963857382535934,
+            -0.010827770456671715,
+            -0.00495715020224452,
+            -0.016332216560840607,
+            -0.02297849766910076,
+            0.002760097151622176,
+            -0.014269033446907997,
+            0.003340859431773424,
+            -0.02165554091334343,
+            0.004846903961151838,
+            -0.017859915271401405,
+            0.021734289824962616,
+            0.010056045837700367,
+            -0.013938293792307377,
+            -0.025703158229589462,
+            -0.004941401071846485,
+            0.02097831293940544,
+            0.012670460157096386,
+            0.00115561846178025,
+            -0.0034688240848481655,
+            0.002754191169515252,
+            -0.003376295790076256,
+            -0.009693807922303677,
+            -0.06545485556125641,
+            -0.017261436209082603,
+            -0.017607923597097397,
+            0.015552615746855736,
+            0.016056599095463753,
+            0.010725399479269981,
+            -0.046901967376470566,
+            -0.05118582397699356,
+            0.028538066893815994,
+            -0.03549933806061745,
+            -0.001561167649924755,
+            -0.0025927589740604162,
+            -0.0015926666092127562,
+            -0.004386231303215027,
+            0.016568457707762718,
+            -0.005606816615909338,
+            0.03360940143465996,
+            -0.012016857042908669,
+            0.007650312036275864,
+            0.004480728413909674,
+            0.020269587635993958,
+            0.0333574078977108,
+            -0.011103386990725994,
+            -0.011048263870179653,
+            0.049106892198324203,
+            -0.013631178997457027,
+            0.025088928639888763,
+            0.005638315342366695,
+            0.001570026739500463,
+            0.0033861391711980104,
+            0.02382897026836872,
+            0.014198160730302334,
+            0.005350887309759855,
+            -0.03713728487491608,
+            0.027089113369584084,
+            0.010363160632550716,
+            -0.060919005423784256,
+            -0.015088006854057312,
+            0.006595096550881863,
+            0.017418930307030678,
+            0.016536958515644073,
+            -0.003559383563697338,
+            -0.014662770554423332,
+            -0.013457934372127056,
+            -0.008481097407639027,
+            -0.018521394580602646,
+            0.036412809044122696,
+            -0.0033054230734705925,
+            0.010725399479269981,
+            -0.027215110138058662,
+            0.0045752255246043205,
+            0.01328469067811966,
+            0.016206219792366028,
+            0.02942003682255745,
+            -0.015922728925943375,
+            -0.015017134137451649,
+            -0.04718545451760292,
+            -0.03288492560386658,
+            -0.00928432121872902,
+            0.013245316222310066,
+            0.030128763988614082,
+            0.01600935123860836,
+            0.046429481357336044,
+            -0.012757082469761372,
+            0.00840235035866499,
+            -0.027624595910310745,
+            0.03795625641942024,
+            0.049043893814086914,
+            0.009985173121094704,
+            -0.020631825551390648,
+            0.02171853929758072,
+            0.009985173121094704,
+            -0.01995459757745266,
+            0.012308222241699696,
+            -0.020631825551390648,
+            0.040413178503513336,
+            -0.022458765655755997,
+            0.03874373435974121,
+            -0.04337408021092415,
+            -0.010709649883210659,
+            -0.03414488211274147,
+            0.003462917869910598,
+            0.02856956608593464,
+            -0.022395767271518707,
+            -0.021608293056488037,
+            -0.0017294903518632054,
+            0.0272938571870327,
+            0.012410594150424004,
+            -0.0009070718660950661,
+            -0.02293124981224537,
+            -0.021198807284235954,
+            -0.03534184396266937,
+            0.006890399381518364,
+            -0.014465902000665665,
+            -0.017371682450175285,
+            -0.0533907525241375,
+            0.014245408587157726,
+            -0.07143966108560562,
+            0.010993140749633312,
+            0.0485084131360054,
+            -0.012276723049581051,
+            0.016662955284118652,
+            0.0007077424670569599,
+            -0.004256298299878836,
+            -0.02935704030096531,
+            0.00967018399387598,
+            0.049831368029117584,
+            0.04028718173503876,
+            -0.021482296288013458,
+            -0.017670921981334686,
+            0.009150451049208641,
+            -0.028601063415408134,
+            -0.01589122973382473,
+            -0.005209141876548529,
+            0.01704094186425209,
+            0.014481651596724987,
+            -0.02589215338230133,
+            0.02638038620352745,
+            0.008465347811579704,
+            0.004953213036060333,
+            0.028396321460604668,
+            0.01331618893891573,
+            -0.0460829921066761,
+            0.014347780495882034,
+            0.029372788965702057,
+            0.001872219960205257,
+            -0.010378910228610039,
+            -0.004106678068637848,
+            0.00592968100681901,
+            0.01819065399467945,
+            -0.017576424404978752,
+            0.007114829961210489,
+            0.005591067019850016,
+            -0.01583610661327839,
+            0.020694822072982788,
+            0.011930234730243683,
+            0.01128450594842434,
+            -0.01067027635872364,
+            0.0218287855386734,
+            -0.00031375925755128264,
+            -0.019907347857952118,
+            -0.009426066651940346,
+            -0.00981980375945568,
+            0.02061607502400875,
+            0.00038364759529940784,
+            -0.03445987403392792,
+            0.011111261323094368,
+            0.04372056946158409,
+            -0.001538527780212462,
+            0.01003242190927267,
+            0.010812020860612392,
+            0.030223261564970016,
+            0.028538066893815994,
+            -0.022710757330060005,
+            -0.02825457602739334,
+            0.01691494695842266,
+            -0.009394567459821701,
+            0.004547663498669863,
+            -0.022632010281085968,
+            -0.014237534254789352,
+            0.03408188372850418,
+            -0.016725953668355942,
+            0.033136915415525436,
+            -0.002256113803014159,
+            0.015969976782798767,
+            -0.004846903961151838,
+            0.0010483250953257084,
+            -0.006433664355427027,
+            0.035215847194194794,
+            -0.026396136730909348,
+            0.014930511824786663,
+            0.007110892329365015,
+            0.002866406226530671,
+            0.05723362788558006,
+            -0.03814525157213211,
+            -0.03625531494617462,
+            0.00262228911742568,
+            0.030853239819407463,
+            -0.017796916887164116,
+            -0.005913931410759687,
+            -0.009599311277270317,
+            0.0031813958194106817,
+            0.018757635727524757,
+            -0.0351843498647213,
+            -0.01855289377272129,
+            0.01331618893891573,
+            -0.01801741123199463,
+            0.003470792667940259,
+            -0.0006501584430225194,
+            0.0018436741083860397,
+            0.00524457823485136,
+            0.0618639774620533,
+            -0.05175280570983887,
+            0.016946446150541306,
+            0.03603482246398926,
+            -0.0007520379149354994,
+            -0.03282192721962929,
+            -0.02625439129769802,
+            -0.005736749619245529,
+            -0.003931465093046427,
+            -0.03067999705672264,
+            -1.9440769392531365e-05,
+            -0.004043680150061846,
+            0.007024270016700029,
+            -0.03669629991054535,
+            0.014623397029936314,
+            0.011355378665030003,
+            -0.002449044957756996,
+            0.003744439920410514,
+            -0.01391466986387968,
+            -0.002553385216742754,
+            -0.003896028734743595,
+            0.03367239981889725,
+            -0.011064013466238976,
+            -0.002878218423575163,
+            -0.00967018399387598,
+            -0.003775938879698515,
+            -0.007197514642030001,
+            -0.0027738779317587614,
+            -0.00845747347921133,
+            0.0031873020343482494,
+            0.012237349525094032,
+            0.001938170986250043,
+            -0.012426343746483326,
+            -0.0028171890880912542,
+            0.03657030314207077,
+            -0.02510467916727066,
+            -0.021183056756854057,
+            0.051280323415994644,
+            0.040476176887750626,
+            0.004374419339001179,
+            -0.021907532587647438,
+            -0.010260789655148983,
+            0.021498046815395355,
+            -0.019749853760004044,
+            0.013670552521944046,
+            0.032176196575164795,
+            -0.007547940593212843,
+            -0.028364822268486023,
+            -0.00964655913412571,
+            -0.014410778880119324,
+            0.0019509674748405814,
+            -0.00562256621196866,
+            0.024065213277935982,
+            -0.044539544731378555,
+            0.008488972671329975,
+            0.019938847050070763,
+            0.002590790158137679,
+            -0.012394844554364681,
+            -0.011040388606488705,
+            -0.003295579692348838,
+            0.02178153768181801,
+            0.0009813897777348757,
+            -0.01893088035285473,
+            0.00861496850848198,
+            0.013418560847640038,
+            0.006158048752695322,
+            0.01893088035285473,
+            -0.005732812453061342,
+            0.02352973073720932,
+            0.009733181446790695,
+            -0.0005241625476628542,
+            0.014056415297091007,
+            -0.019970346242189407,
+            -0.00488234031945467,
+            0.061517488211393356,
+            0.011623119935393333,
+            0.025734657421708107,
+            -0.01431628130376339,
+            0.020631825551390648,
+            0.03814525157213211,
+            -0.0016999599756672978,
+            -0.020820818841457367,
+            0.008362976834177971,
+            0.00283490726724267,
+            -0.044665537774562836,
+            -0.0134894335642457,
+            -0.023923467844724655,
+            -0.037326280027627945,
+            0.0035357594024389982,
+            -0.021214555948972702,
+            0.01801741123199463,
+            0.025199174880981445,
+            0.0065832845866680145,
+            -0.030884739011526108,
+            -0.01197748351842165,
+            0.022710757330060005,
+            0.02741985209286213,
+            0.03449137136340141,
+            -0.01233972143381834,
+            0.010103294625878334,
+            -0.003295579692348838,
+            0.013694177381694317,
+            0.037987757474184036,
+            0.006925835739821196,
+            -0.0031006797216832638,
+            0.008059798739850521,
+            0.029703527688980103,
+            0.03675929829478264,
+            -0.024584945291280746,
+            -0.052130796015262604,
+            -0.0028703436255455017,
+            -0.01686769910156727,
+            0.018600141629576683,
+            -0.001023716526106,
+            0.003746408736333251,
+            0.0011428219731897116,
+            0.006654157303273678,
+            -0.008740964345633984,
+            0.02893180400133133,
+            0.008028299547731876,
+            -0.021797286346554756,
+            0.05159531161189079,
+            -0.015481743961572647,
+            0.024112461134791374,
+            -0.001423359732143581,
+            -0.02590790204703808,
+            -0.028711311519145966,
+            0.013111446052789688,
+            0.007390445563942194,
+            -0.03546784073114395,
+            0.03742077574133873,
+            0.033451907336711884,
+            0.0030731181614100933,
+            0.0012127102818340063,
+            0.0166787039488554,
+            0.016962194815278053,
+            -0.014780891127884388,
+            -0.023876218125224113,
+            0.02286825142800808,
+            -0.008677965961396694,
+            -0.021939031779766083,
+            0.01982860080897808,
+            -0.018521394580602646,
+            -0.008268479257822037,
+            0.0042484235018491745,
+            -0.001532621681690216,
+            -0.010182041674852371,
+            0.014395029284060001,
+            0.016521209850907326,
+            -0.015332123264670372,
+            -0.02527792379260063,
+            0.031215479597449303,
+            0.021513795480132103,
+            0.0038231874350458384,
+            -0.02165554091334343,
+            0.013473683968186378,
+            -0.022537512704730034,
+            -0.0010286382166668773,
+            0.043752070516347885,
+            -0.002433295361697674,
+            0.012946076691150665,
+            -0.010237164795398712,
+            -0.029719278216362,
+            0.002671506255865097,
+            0.011213633231818676,
+            0.03789326176047325,
+            0.01885213330388069,
+            0.01795441284775734,
+            0.019450614228844643,
+            -0.01631646603345871,
+            0.009528438560664654,
+            -0.014560398645699024,
+            -0.03294792026281357,
+            0.03179820999503136,
+            -0.006024178117513657,
+            0.025577163323760033,
+            0.008465347811579704,
+            -0.0321604460477829,
+            -0.004728782922029495,
+            0.011040388606488705,
+            0.013001199811697006,
+            0.023120243102312088,
+            0.013867421075701714,
+            0.003108554519712925,
+            0.031388722360134125,
+            -0.014513149857521057,
+            -0.051091328263282776,
+            0.02710486389696598,
+            0.0063076685182750225,
+            -0.006701405625790358,
+            -0.0016143221873790026,
+            0.020364083349704742,
+            -0.003027838421985507,
+            0.00017139868577942252,
+            0.01270195934921503,
+            -0.010961641557514668,
+            0.00882758665829897,
+            -0.006992771290242672,
+            -0.005469008814543486,
+            0.02414396032691002,
+            0.011827862821519375,
+            0.024065213277935982,
+            -0.019497862085700035,
+            0.006347042508423328,
+            0.017324432730674744,
+            -0.009748931042850018,
+            0.014568272978067398,
+            0.0034314189106225967,
+            -0.05011485889554024,
+            0.01607234962284565,
+            0.017576424404978752,
+            -0.005173705518245697,
+            0.007103017531335354,
+            -0.013402811251580715,
+            0.013804423622786999,
+            0.04627198725938797,
+            0.0016330246580764651,
+            0.0066856564953923225,
+            0.013205942697823048,
+            -0.005012273322790861,
+            -0.009804054163396358,
+            0.03112098202109337,
+            0.00845747347921133,
+            -0.008796087466180325,
+            0.009079578332602978,
+            0.001405641552992165,
+            0.03499535471200943,
+            -0.04828792065382004,
+            0.010717524215579033,
+            0.012520840391516685,
+            -0.018584391102194786,
+            -0.012812205590307713,
+            -0.02886880561709404,
+            0.009520563296973705,
+            -0.03044375404715538,
+            0.01814340613782406,
+            -0.022789504379034042,
+            0.021608293056488037,
+            -0.035688333213329315,
+            -0.0018732043681666255,
+            0.012623212300240993,
+            -0.0058076223358511925,
+            0.003575132926926017,
+            0.012473591603338718,
+            -0.007575502153486013,
+            -0.009323694743216038,
+            -0.018096158280968666,
+            0.023671476170420647,
+            0.05575317516922951,
+            -0.0003388600016478449,
+            0.00024116398708429188,
+            0.015930604189634323,
+            -0.01443440280854702,
+            -0.02620714344084263,
+            0.02953028306365013,
+            -0.016773201525211334,
+            0.015623488463461399,
+            0.03723178058862686,
+            -0.03468036651611328,
+            -0.027262357994914055,
+            0.01543449517339468,
+            -0.013725675642490387,
+            0.008347227238118649,
+            0.0020651512313634157,
+            0.02159254439175129,
+            -0.00283490726724267,
+            -0.026096895337104797,
+            0.028538066893815994,
+            0.006197422277182341,
+            0.009418192319571972,
+            -0.021671291440725327,
+            -0.011095511727035046,
+            -0.027325356379151344,
+            -0.023797471076250076,
+            -0.011867236346006393,
+            -0.010174167342483997,
+            0.006508474703878164,
+            -0.016820449382066727,
+            -0.057832106947898865,
+            -0.015229751355946064,
+            0.023907717317342758,
+            -0.01412728801369667,
+            0.05263477936387062,
+            0.01844264753162861,
+            0.016269218176603317,
+            0.0048035928048193455,
+            -0.04022418335080147,
+            0.030176013708114624,
+            -0.017560675740242004,
+            0.03143597021698952,
+            0.016088098287582397,
+            0.006839213892817497,
+            -0.0059611801989376545,
+            0.05058734491467476,
+            -0.02389196865260601,
+            0.007126641925424337,
+            -0.019686855375766754,
+            0.006473038345575333,
+            -0.02499443292617798,
+            -0.004508289974182844,
+            -0.00521701667457819,
+            -0.011465624906122684,
+            0.010063921101391315,
+            -0.011426251381635666,
+            0.024049462750554085,
+            0.02541966922581196,
+            0.0009026423213072121,
+            0.020301084965467453,
+            -0.030900489538908005,
+            -0.022206773981451988,
+            -0.004740594886243343,
+            0.0018899381393566728,
+            -0.02080507017672062,
+            -0.016332216560840607,
+            -0.040066689252853394,
+            -0.012788581661880016,
+            -0.0002187701902585104,
+            0.01212710328400135,
+            0.0040909284725785255,
+            0.02838057093322277,
+            -0.0017038973746821284,
+            -0.03209745138883591,
+            0.0071896398440003395,
+            -0.029514534398913383,
+            -0.0005906056612730026,
+            0.01391466986387968,
+            -0.017859915271401405,
+            -0.005945430602878332,
+            0.030648497864603996,
+            0.0016812575049698353,
+            -0.0150092588737607,
+            -0.016032975167036057,
+            0.005567443091422319,
+            -0.016237718984484673,
+            -0.0035554461646825075,
+            -0.01028441358357668,
+            0.03364089876413345,
+            -0.015332123264670372,
+            0.007780245505273342,
+            0.0005694422870874405,
+            4.2634346755221486e-05,
+            -0.03439687564969063,
+            -0.012890953570604324,
+            -0.019686855375766754,
+            0.013371312990784645,
+            0.0017777230823412538,
+            -0.027073364704847336,
+            0.007902303710579872,
+            0.0009838506812229753,
+            -0.004531914368271828,
+            -0.021324802190065384,
+            0.0032621119171380997,
+            0.005339075345546007,
+            -0.01771816983819008,
+            0.001696022693067789,
+            -0.033388908952474594,
+            -0.02831757441163063,
+            -0.029719278216362,
+            0.019419115036725998,
+            -0.0030435880180448294,
+            0.021183056756854057,
+            -0.013205942697823048,
+            0.012969700619578362,
+            -0.017261436209082603,
+            0.004590974655002356,
+            0.01570223644375801,
+            0.024663692340254784,
+            -0.03160921484231949,
+            0.013898920267820358,
+            0.01582035794854164,
+            0.021561045199632645,
+            -0.04523251950740814,
+            0.010709649883210659,
+            0.013631178997457027,
+            -0.0015227783005684614,
+            0.016332216560840607,
+            -0.010835645720362663,
+            0.03354640305042267,
+            -0.02093106508255005,
+            -0.0010749023640528321,
+            -0.005587129853665829,
+            0.013520932756364346,
+            0.01861589029431343,
+            0.012883078306913376,
+            -0.030412254855036736,
+            0.03110523149371147,
+            0.02352973073720932,
+            -0.012662585824728012,
+            0.04614599049091339,
+            -0.02656938135623932,
+            0.016458211466670036,
+            0.018600141629576683,
+            -0.02827032469213009,
+            0.0015660893404856324,
+            0.0045791626907885075,
+            -0.015631362795829773,
+            -0.014206035062670708,
+            0.025340920314192772,
+            0.00750462943688035,
+            -0.04346857964992523,
+            0.015001384541392326,
+            0.018237903714179993,
+            0.01119000930339098,
+            0.00433504581451416,
+            -0.023939216509461403,
+            0.009174074977636337,
+            0.028128579258918762,
+            0.0006875634426251054,
+            0.030475253239274025,
+            0.015064381994307041,
+            -0.02323048934340477,
+            0.011930234730243683,
+            -0.005795810371637344,
+            0.007406195160001516,
+            -0.01831665076315403,
+            0.00392752792686224,
+            -0.00747706787660718,
+            -0.014654895290732384,
+            -0.007232951000332832,
+            0.029640531167387962,
+            -0.003964932635426521,
+            -0.018883632495999336,
+            -0.03833424672484398,
+            0.015804607421159744,
+            0.009252822026610374,
+            -0.06709280610084534,
+            -0.00876458827406168,
+            -0.03386139124631882,
+            -0.02140354923903942,
+            -0.008906333707273006,
+            0.012749208137392998,
+            -0.015851857140660286,
+            -0.002389984205365181,
+            0.006154111120849848,
+            0.015599864535033703,
+            -0.01564711332321167,
+            0.02085231803357601,
+            0.006024178117513657,
+            0.005291827023029327,
+            -0.013993416912853718,
+            0.042051125317811966,
+            -0.007563690189272165,
+            -0.03814525157213211,
+            0.032176196575164795,
+            -0.012820080853998661,
+            0.003575132926926017,
+            0.024065213277935982,
+            -0.0019421083852648735,
+            -0.022002030164003372,
+            0.028963303193449974,
+            -0.026285890489816666,
+            -0.03512135148048401,
+            0.01522187702357769,
+            -0.006547848228365183,
+            -0.020285336300730705,
+            -0.02056882716715336,
+            -0.004681534133851528,
+            0.012733458541333675,
+            -0.02050582878291607,
+            0.018489895388484,
+            0.03266443312168121,
+            0.0052524530328810215,
+            -0.016206219792366028,
+            0.012457842007279396,
+            -0.0032247069757431746,
+            -0.0015030914219096303,
+            -0.005173705518245697,
+            0.024364452809095383,
+            -0.004055492114275694,
+            0.0033625150099396706,
+            -0.019119873642921448,
+            0.0030239010229706764,
+            -0.016088098287582397,
+            -0.012670460157096386,
+            -0.013457934372127056,
+            -0.025214925408363342,
+            -0.02493143454194069,
+            -0.04088566079735756,
+            0.03531034663319588,
+            0.024458950385451317,
+            -0.017324432730674744,
+            -0.010599403642117977,
+            -0.00861496850848198,
+            0.0013957980554550886,
+            0.028396321460604668,
+            -0.002578978193923831,
+            -0.0036597864236682653,
+            -0.02231702022254467,
+            0.011686117388308048,
+            -0.010583654046058655,
+            0.0014322188217192888,
+            -0.06296644359827042,
+            -0.01806465908885002,
+            0.034428372979164124,
+            0.007181765045970678,
+            -0.007996801286935806,
+            0.005917869042605162,
+            0.02376597188413143,
+            -0.020222337916493416,
+            -0.012481466867029667,
+            0.04390956461429596,
+            0.0027305670082569122,
+            0.0021458673290908337,
+            0.0070203328505158424,
+            -0.008347227238118649,
+            0.010961641557514668,
+            -0.004445292055606842,
+            -0.007729059550911188,
+            0.0005728874821215868,
+            0.010969515889883041,
+            0.009654434397816658,
+            0.00638247886672616,
+            -0.00442954245954752,
+            0.020710572600364685,
+            -0.011402626521885395,
+            -0.0043665445409715176,
+            0.006543911062180996,
+            0.019749853760004044,
+            -0.004783906042575836,
+            0.00025014611310325563,
+            -0.020301084965467453,
+            0.023986464366316795,
+            0.020773570984601974,
+            0.016158971935510635,
+            -0.0004537820350378752,
+            -0.008733089081943035,
+            -0.008059798739850521,
+            0.004401980899274349,
+            0.0030317758210003376,
+            0.017497677356004715,
+            0.028112830594182014,
+            -0.019482113420963287,
+            -0.016536958515644073,
+            -0.0012373188510537148,
+            0.0025336984544992447,
+            -0.004968962632119656,
+            0.03990919515490532,
+            0.02431720495223999,
+            0.005598941817879677,
+            -0.013969792984426022,
+            -0.0007269371417351067,
+            0.00891420803964138,
+            0.03505835309624672,
+            -0.026112645864486694,
+            -0.01571011170744896,
+            0.019907347857952118,
+            -0.0016330246580764651,
+            0.03433387726545334,
+            -0.021387800574302673,
+            0.014261158183217049,
+            -0.0026006337720900774,
+            0.08655916899442673,
+            -0.023372234776616096,
+            -0.005098895635455847,
+            -0.016174720600247383,
+            -0.014450152404606342,
+            0.004220861941576004,
+            -0.00459491228684783,
+            -0.003289673710241914,
+            0.031624965369701385,
+            -0.002029714873060584,
+            0.017182687297463417,
+            -0.0004545202828012407,
+            -0.008898458443582058,
+            0.008020425215363503,
+            -0.0035180412232875824,
+            0.018206404522061348,
+            -0.006681718863546848,
+            0.008969331160187721,
+            -0.003004214260727167,
+            0.0008957519312389195,
+            0.01322169229388237,
+            0.03291642293334007,
+            0.003939339891076088,
+            0.022537512704730034,
+            -0.02419120818376541,
+            0.018631640821695328,
+            0.007677873596549034,
+            0.016647204756736755,
+            -0.019986096769571304,
+            -0.017418930307030678,
+            -0.0007298901909962296,
+            0.013709926046431065,
+            -0.008575594052672386,
+            0.010906518436968327,
+            -0.003833030816167593,
+            0.039216216653585434,
+            -0.026396136730909348,
+            0.007047894410789013,
+            0.010835645720362663,
+            -0.004051554948091507,
+            0.009118951857089996,
+            0.02208077721297741,
+            0.017985912039875984,
+            0.005118582397699356,
+            -0.02297849766910076,
+            0.014135162346065044,
+            0.02293124981224537,
+            -0.01734018325805664,
+            0.0019470300758257508,
+            0.006539973430335522,
+            -0.0015119505114853382,
+            0.0004030883719678968,
+            0.005835183896124363,
+            -0.00024300962104462087,
+            -0.008331477642059326,
+            0.004512227606028318,
+            -0.007961364462971687,
+            -0.035562336444854736,
+            -0.0027384415734559298,
+            0.012851579114794731,
+            -0.00626435736194253,
+            0.031451720744371414,
+            0.029735026881098747,
+            0.019671106711030006,
+            -0.009811929427087307,
+            -0.006874650251120329,
+            0.004177550785243511,
+            -0.0040751793421804905,
+            -0.0011890861205756664,
+            -0.013607555069029331,
+            -0.014308406971395016,
+            0.014111538417637348,
+            -0.022474514320492744,
+            -0.00888270977884531,
+            0.0018338306108489633,
+            -0.005532006733119488,
+            -0.00459491228684783,
+            -0.0025002306792885065,
+            0.017481928691267967,
+            -0.005724937655031681,
+            0.02037983387708664,
+            0.01048915646970272,
+            -0.011528623290359974,
+            0.011308129876852036,
+            -0.026900120079517365,
+            0.009567812085151672,
+            -0.026963118463754654,
+            -0.010756897740066051,
+            0.009213448502123356,
+            -0.003992494195699692,
+            -0.017985912039875984,
+            -0.010473407804965973,
+            -0.016095973551273346,
+            -0.005220954306423664,
+            -0.00745738111436367,
+            -0.0006659079226665199,
+            0.013607555069029331,
+            0.008150358684360981,
+            -0.01601722650229931,
+            -0.016599956899881363,
+            -0.03694828972220421,
+            0.007843242958188057,
+            0.012993324548006058,
+            -0.025246424600481987,
+            -0.014119412750005722,
+            -0.007567627355456352,
+            0.008433849550783634,
+            -0.01233972143381834,
+            -0.0014351718127727509,
+            -0.0033802331890910864,
+            0.014883263036608696,
+            -0.02493143454194069,
+            0.006827401462942362,
+            -0.01929311826825142,
+            0.01558411493897438,
+            0.009583561681210995,
+            0.009260697290301323,
+            -0.03612931817770004,
+            -0.01885213330388069,
+            0.023561229929327965,
+            -0.008205481804907322,
+            -0.009504813700914383,
+            0.024049462750554085,
+            -0.012969700619578362,
+            0.011134886182844639,
+            0.019875850528478622,
+            0.01145775057375431,
+            -0.0026537880767136812,
+            0.0218287855386734,
+            -0.006213171873241663,
+            7.259527774294838e-05,
+            -0.004512227606028318,
+            0.024332953616976738,
+            0.0063588544726371765,
+            0.032790426164865494,
+            -0.015300624072551727,
+            -0.018521394580602646,
+            -0.005760374013334513,
+            0.0442875511944294,
+            0.007559752557426691,
+            -0.02723085880279541,
+            0.018174905329942703,
+            0.029136545956134796,
+            -0.0032916422933340073,
+            0.005445384420454502,
+            -0.038239747285842896,
+            0.0004163770063314587,
+            0.015505367890000343,
+            0.021923283115029335,
+            0.011292380280792713,
+            -0.009922175668179989,
+            -0.018363898620009422,
+            0.007181765045970678,
+            0.002352579263970256,
+            0.006039927247911692,
+            -0.03990919515490532,
+            0.011449875310063362,
+            -0.020946815609931946,
+            0.01970260590314865,
+            0.00861496850848198,
+            -0.009378817863762379,
+            0.022458765655755997,
+            0.0028152205049991608,
+            -0.017088191583752632,
+            0.014710018411278725,
+            -0.005980866961181164,
+            0.0006673844181932509,
+            0.03329440951347351,
+            -0.0036184440832585096,
+            0.012717708945274353,
+            -0.007815681397914886,
+            0.0061383615247905254,
+            0.0031046171206980944,
+            0.03505835309624672,
+            -0.0008091297931969166,
+            -0.014458026736974716,
+            0.0068628378212451935,
+            0.005417822860181332,
+            -0.0005571380024775863,
+            -0.005059522110968828,
+            0.0077448091469705105,
+            -0.012670460157096386,
+            0.0014400935033336282,
+            -0.003053431399166584,
+            -0.006315543316304684,
+            -0.0030396506190299988,
+            0.002159648109227419,
+            0.00047543758410029113,
+            -0.033136915415525436,
+            0.02178153768181801,
+            -0.0042484235018491745,
+            0.013843797147274017,
+            -0.005913931410759687,
+            -0.01531637366861105,
+            0.0023643914610147476,
+            0.02074207179248333,
+            -0.005492632742971182,
+            -0.011898735538125038,
+            -0.01595422811806202,
+            -0.02238001860678196,
+            -0.026301639154553413,
+            0.028522316366434097,
+            0.0024254205636680126,
+            -0.01607234962284565,
+            0.008481097407639027,
+            -0.009591436013579369,
+            -0.025545664131641388,
+            0.006520286668092012,
+            0.018395397812128067,
+            -0.041641637682914734,
+            0.016048725694417953,
+            -0.006953397300094366,
+            -0.0025573226157575846,
+            0.028648313134908676,
+            0.006732904817909002,
+            -0.003504260443150997,
+            0.014450152404606342,
+            -0.0027384415734559298,
+            0.011945984326303005,
+            0.02165554091334343,
+            0.0023072995245456696,
+            0.0066384077072143555,
+            0.014198160730302334,
+            0.015521117486059666,
+            -0.002368328860029578,
+            -0.01837964914739132,
+            0.027624595910310745,
+            -0.021545294672250748,
+            0.02025383710861206,
+            0.021261803805828094,
+            -0.026427635923027992,
+            0.029388537630438805,
+            -0.00997729878872633,
+            0.019182872027158737,
+            0.0034491370897740126,
+            -0.00592968100681901,
+            -0.01233972143381834,
+            0.004949275404214859,
+            0.007484942674636841,
+            0.009300070814788342,
+            0.010331662371754646,
+            0.044004060328006744,
+            -0.01364692859351635,
+            0.028601063415408134,
+            0.03401888534426689,
+            -0.00010723184095695615,
+            0.02255326136946678,
+            -0.018584391102194786,
+            -0.02123030461370945,
+            0.007488879840821028,
+            -0.015780983492732048,
+            -0.03025476075708866,
+            0.0032483311370015144,
+            -0.01452889945358038,
+            -0.008284228853881359,
+            -0.016269218176603317,
+            -0.0019539203494787216,
+            -0.0022324894089251757,
+            -0.0009164231596514583,
+            0.004220861941576004,
+            0.020348334684967995,
+            0.026049647480249405,
+            0.02376597188413143,
+            0.01024504005908966,
+            0.00747706787660718,
+            0.01148924883455038,
+            -0.009930050000548363,
+            0.026175644248723984,
+            -0.0011713679414242506,
+            -0.0028624688275158405,
+            -0.0172929335385561,
+            0.007055769208818674,
+            -0.00029013503808528185,
+            -0.005110707599669695,
+            0.007425881922245026,
+            -0.022175274789333344,
+            -0.0015906979097053409,
+            -0.04687046632170677,
+            -0.002504168078303337,
+            -0.0007677873945795,
+            0.010394659824669361,
+            0.03965720161795616,
+            -0.007449506316334009,
+            -0.017922913655638695,
+            0.008284228853881359,
+            -0.01685194857418537,
+            0.004098803270608187,
+            -0.01814340613782406,
+            -0.015591990202665329,
+            -0.010756897740066051,
+            -0.02414396032691002,
+            -0.009315820410847664,
+            -0.013812297955155373,
+            0.015143129974603653,
+            -0.00876458827406168,
+            0.011229382827877998,
+            -0.018710387870669365,
+            0.00041736135608516634,
+            -0.006221046671271324,
+            -0.008433849550783634,
+            -0.002559291198849678,
+            -0.020348334684967995,
+            -0.012379094958305359,
+            0.022710757330060005,
+            -0.008780337870121002,
+            -0.029609031975269318,
+            -0.006311606150120497,
+            0.004232673905789852,
+            -0.014859639108181,
+            0.0037365653552114964,
+            0.00592968100681901,
+            -0.005917869042605162,
+            -0.008985080756247044,
+            0.00961506087332964,
+            0.018726136535406113,
+            0.012457842007279396,
+            -0.0028644376434385777,
+            0.00395705783739686,
+            0.02291550114750862,
+            0.02348248101770878,
+            -0.0454845130443573,
+            -0.005102833267301321,
+            0.017686670646071434,
+            0.016347965225577354,
+            -0.0070321448147296906,
+            0.018411148339509964,
+            -0.01507225725799799,
+            0.011827862821519375,
+            -0.02250601351261139,
+            -0.00612261239439249,
+            -0.026915868744254112,
+            0.00845747347921133,
+            0.01734018325805664,
+            -0.006768341176211834,
+            0.035215847194194794,
+            0.020600326359272003,
+            -0.016788950189948082,
+            0.020994063466787338,
+            -0.0013150819577276707,
+            0.011174259707331657,
+            -0.027089113369584084,
+            0.01364692859351635,
+            -0.009229198098182678,
+            0.008236980065703392,
+            0.014206035062670708,
+            -0.020049093291163445,
+            -0.025340920314192772,
+            -0.01222159992903471,
+            0.012174352072179317,
+            0.003065243363380432,
+            -0.0027896275278180838,
+            0.022222522646188736,
+            0.015143129974603653,
+            -0.01197748351842165,
+            -0.01740317977964878,
+            0.023561229929327965,
+            0.01276495773345232,
+            0.015418745577335358,
+            -0.0032581747509539127,
+            -0.0033290472347289324,
+            -0.017009442672133446,
+            0.0005093974177725613,
+            -0.02450619824230671,
+            -0.001176289631985128,
+            -0.002449044957756996,
+            0.021529546007514,
+            -0.007681811228394508,
+            -0.006224983837455511,
+            -0.007177827879786491,
+            -0.02808133140206337,
+            -0.000330000912072137,
+            0.00023365837114397436,
+            0.01252871472388506,
+            0.007658186834305525,
+            0.009473315440118313,
+            0.017481928691267967,
+            -0.026364637538790703,
+            0.002179334871470928,
+            -0.019482113420963287,
+            0.004126364830881357,
+            -0.00928432121872902,
+            -0.014292657375335693,
+            0.01033953670412302,
+            0.003470792667940259,
+            -0.00480753043666482,
+            0.01012691855430603,
+            0.003260143334046006,
+            -0.017387431114912033,
+            0.029939770698547363,
+            -0.004006274975836277,
+            0.002279737964272499,
+            0.003663723822683096,
+            0.005736749619245529,
+            0.0019736073445528746,
+            0.02827032469213009,
+            -0.025388170033693314,
+            -0.016773201525211334,
+            -0.018600141629576683,
+            0.0024372327607125044,
+            0.004988649394363165,
+            0.005386323668062687,
+            -0.0010079670464619994,
+            -0.00013190193567425013,
+            0.002657725475728512,
+            -0.0070321448147296906,
+            0.027876587584614754,
+            0.0040909284725785255,
+            -0.03382989391684532,
+            -0.024395952001214027,
+            -0.04038167744874954,
+            0.008307852782309055,
+            -0.01982860080897808,
+            -0.029766526073217392,
+            -0.010008797980844975,
+            -0.02274225652217865,
+            -0.035940323024988174,
+            0.018474144861102104,
+            0.020757820457220078,
+            -0.02025383710861206,
+            -0.008205481804907322,
+            -0.0028447506483644247,
+            -0.013142945244908333,
+            0.010048171505331993,
+            -0.0029805900994688272,
+            -0.016269218176603317,
+            0.010355286300182343,
+            0.014765141531825066,
+            0.0068628378212451935,
+            -0.00867009162902832,
+            -0.016347965225577354,
+            0.00028521331842057407,
+            0.023813221603631973,
+            0.0064651635475456715,
+            -0.01107188779860735,
+            0.007925928570330143,
+            0.006398227997124195,
+            -0.015111630782485008,
+            0.007437694352120161,
+            -0.023088743910193443,
+            -0.000766310840845108,
+            0.0025021994952112436,
+            0.014056415297091007,
+            -0.01644246280193329,
+            0.003551508765667677,
+            0.005618628580123186,
+            -0.0036873482167720795,
+            0.003545602783560753,
+            -0.018757635727524757,
+            0.00671715522184968,
+            -0.009748931042850018,
+            0.0019913255237042904,
+            0.028128579258918762,
+            -0.007232951000332832,
+            -0.005161893554031849,
+            -0.032412439584732056,
+            0.01642671227455139,
+            0.03360940143465996,
+            0.013678427785634995,
+            -0.021513795480132103,
+            -0.015985727310180664,
+            0.004283859860152006,
+            -0.0016468055546283722,
+            -0.019198622554540634,
+            0.015473868697881699,
+            0.019450614228844643,
+            0.012197976000607014,
+            0.015505367890000343,
+            0.013166569173336029,
+            0.014379279688000679,
+            -0.025451166555285454,
+            0.001903718919493258,
+            -0.0020454642362892628,
+            -0.02170279063284397,
+            0.012024731375277042,
+            0.02061607502400875,
+            -0.02378172241151333,
+            -0.003651911858469248,
+            0.0016153064789250493,
+            0.02165554091334343,
+            -0.003827124834060669,
+            -0.002248239004984498,
+            0.01808040775358677,
+            0.010252914391458035,
+            0.015119505114853382,
+            0.004795718006789684,
+            0.061045002192258835,
+            -0.003384170588105917,
+            0.007122704293578863,
+            0.008575594052672386,
+            -0.005106770433485508,
+            -0.01965535618364811,
+            -0.0010197791270911694,
+            -0.026537882164120674,
+            0.006654157303273678,
+            0.007752683945000172,
+            -0.02778209187090397,
+            -0.005902119446545839,
+            -0.009008705615997314,
+            -0.006595096550881863,
+            0.018048910424113274,
+            0.0047484696842730045,
+            -0.0020179026760160923,
+            0.006260420195758343,
+            0.037987757474184036,
+            0.014686394482851028,
+            -0.009504813700914383,
+            0.0023072995245456696,
+            0.010150542482733727,
+            0.007760558743029833,
+            -0.008268479257822037,
+            0.009040203876793385,
+            0.0004434464208316058,
+            0.004461041651666164,
+            -0.00800467561930418,
+            0.039625704288482666,
+            -0.009040203876793385,
+            0.0009188840049318969,
+            0.0023801408242434263,
+            -0.014426528476178646,
+            0.010938017629086971,
+            0.002990433480590582,
+            0.00729201128706336,
+            -0.007095142733305693,
+            0.0013308314373716712,
+            0.002222646027803421,
+            -0.0020001844968646765,
+            -0.0012343658600002527,
+            -0.009559936821460724,
+            -0.01655270904302597,
+            -0.005587129853665829,
+            0.013016949407756329,
+            0.0005856839707121253,
+            0.0052367039024829865,
+            -0.01812765747308731,
+            -0.0072447629645466805,
+            -0.0007126641576178372,
+            -0.011134886182844639,
+            0.007221138570457697,
+            0.024899935349822044,
+            -0.027498601004481316,
+            -0.004283859860152006,
+            0.014166661538183689,
+            -0.00015171183622442186,
+            0.007327447645366192,
+            0.014607647433876991,
+            -0.013119320385158062,
+            0.008000737987458706,
+            -0.016040850430727005,
+            -0.014347780495882034,
+            0.0007456396706402302,
+            -0.006492725107818842,
+            0.0023506106808781624,
+            -0.01307207252830267,
+            -0.0012058198917657137,
+            0.01279645599424839,
+            0.0006412993534468114,
+            0.012079854495823383,
+            0.014544649049639702,
+            -0.00679196510463953,
+            -0.00038364759529940784,
+            0.012694085016846657,
+            -0.012780707329511642,
+            -0.024128209799528122,
+            0.0007146328571252525,
+            0.023876218125224113,
+            -0.00492958864197135,
+            0.00383499963209033,
+            -0.0028506566304713488,
+            0.0024825125001370907,
+            -0.012662585824728012,
+            -0.013001199811697006,
+            -0.012213725596666336,
+            0.0209153164178133,
+            0.004571287892758846,
+            -0.022758005186915398,
+            0.0008258636225946248,
+            0.0055831922218203545,
+            0.009103202261030674,
+            -0.017497677356004715,
+            0.004000368993729353,
+            0.02995551936328411,
+            -0.01479664072394371,
+            -0.0041814884170889854,
+            -0.0008224184275604784,
+            -0.013812297955155373,
+            0.008044049143791199,
+            0.017088191583752632,
+            0.005484757944941521,
+            0.016363713890314102,
+            -0.003575132926926017,
+            -0.009441816247999668,
+            0.03038075566291809,
+            0.015308499336242676,
+            0.004185425583273172,
+            -0.006536036264151335,
+            -0.012788581661880016,
+            -0.028333323076367378,
+            0.004374419339001179,
+            0.010623027570545673,
+            -0.014292657375335693,
+            -0.0005876526702195406,
+            -0.012276723049581051,
+            -0.018001660704612732,
+            -0.004488603211939335,
+            -0.006524223834276199,
+            0.012300347909331322,
+            -0.00976468063890934,
+            0.010804146528244019,
+            -0.0003120366600342095,
+            -0.0035377279855310917,
+            -0.002352579263970256,
+            0.0002889046154450625,
+            -0.009410317055881023,
+            -0.03291642293334007,
+            -0.028412070125341415,
+            -0.008898458443582058,
+            -0.008142483420670033,
+            0.006528161466121674,
+            0.0009907410712912679,
+            -0.030664246529340744,
+            0.010300163179636002,
+            -0.00012433479423634708,
+            -0.014843889512121677,
+            0.015190377831459045,
+            0.009898550808429718,
+            -0.002703005215153098,
+            -0.007528253830969334,
+            -0.007807807065546513,
+            -0.02874280884861946,
+            0.01589910499751568,
+            0.0021872096695005894,
+            0.009260697290301323,
+            -0.014237534254789352,
+            0.0010424189968034625,
+            -0.021009812131524086,
+            0.01727718487381935,
+            -0.026474883779883385,
+            -0.015095881186425686,
+            -0.01522187702357769,
+            -0.0022128026466816664,
+            -0.0060320524498820305,
+            -0.017781168222427368,
+            -0.021907532587647438,
+            -0.006161985918879509,
+            -0.025640161707997322,
+            -0.022899750620126724,
+            -0.0017609893111512065,
+            -0.0034353563096374273,
+            0.010583654046058655,
+            -0.007040019612759352,
+            0.027986833825707436,
+            0.01619047112762928,
+            0.006579347420483828,
+            -0.011560121551156044,
+            -0.013442184776067734,
+            -0.0020198714919388294,
+            0.0025179488584399223,
+            -0.009158325381577015,
+            0.002384078223258257,
+            0.019797101616859436,
+            -0.0036184440832585096,
+            0.0045240395702421665,
+            0.004106678068637848,
+            -0.004035805352032185,
+            0.016836199909448624,
+            0.007988926023244858,
+            0.011788489297032356,
+            0.006315543316304684,
+            0.0016094004968181252,
+            0.03379839286208153,
+            0.016473960131406784,
+            -0.0035869451239705086,
+            0.0022324894089251757,
+            0.020962564274668694,
+            -0.02170279063284397,
+            -0.01571011170744896,
+            0.015450244769454002,
+            0.001383001683279872,
+            -0.03546784073114395,
+            0.02067907340824604,
+            0.01661570556461811,
+            -0.009796179831027985,
+            -0.010764773003757,
+            -0.007961364462971687,
+            -0.00021323325927369297,
+            -0.005181580316275358,
+            0.008544095791876316,
+            -0.011560121551156044,
+            0.013426436111330986,
+            0.006496662274003029,
+            -0.0064651635475456715,
+            -0.014269033446907997,
+            0.01422178465873003,
+            0.027971085160970688,
+            0.0035790703259408474,
+            -0.04819342494010925,
+            -0.012276723049581051,
+            0.01704094186425209,
+            -0.02140354923903942,
+            -0.025797655805945396,
+            0.00011172290396643803,
+            -0.0075715649873018265,
+            -0.023608477786183357,
+            -0.007063644006848335,
+            -0.00392752792686224,
+            -0.016836199909448624,
+            0.012883078306913376,
+            0.01064665149897337,
+            0.012426343746483326,
+            0.008205481804907322,
+            -0.015867605805397034,
+            -0.012064104899764061,
+            -0.042366113513708115,
+            0.00788655411452055,
+            -0.024836936965584755,
+            -0.021939031779766083,
+            0.004902027081698179,
+            -0.006827401462942362,
+            0.008740964345633984,
+            0.01452889945358038,
+            0.039153218269348145,
+            0.01255233958363533,
+            -0.015080131590366364,
+            -0.025687409564852715,
+            -0.017907164990901947,
+            -0.007961364462971687,
+            0.0022758005652576685,
+            -0.0074180071242153645,
+            -0.0035003230441361666,
+            0.0033192038536071777,
+            0.002246270189061761,
+            0.029309790581464767,
+            0.001104432623833418,
+            0.018962379544973373,
+            -3.389830453670584e-05,
+            0.00928432121872902,
+            -0.0038920913357287645,
+            -0.025025932118296623,
+            0.006213171873241663,
+            0.004106678068637848,
+            0.00794167723506689,
+            0.0068628378212451935,
+            0.005106770433485508,
+            0.016584208235144615,
+            -0.015867605805397034,
+            0.005394198466092348,
+            -0.026222892105579376,
+            -0.006469100713729858,
+            -0.014418653212487698,
+            0.0294042881578207,
+            0.014733643271028996,
+            -0.02327773906290531,
+            -0.013174444437026978,
+            -0.018993878737092018,
+            0.017970161512494087,
+            -0.008685841225087643,
+            0.009756805375218391,
+            0.01355243194848299,
+            0.01040253508836031,
+            0.006205297075212002,
+            0.0053548249416053295,
+            -0.005012273322790861,
+            0.007736934348940849,
+            0.02310449443757534,
+            -0.007106955163180828,
+            0.008000737987458706,
+            -0.0013849702663719654,
+            -0.0041814884170889854,
+            -0.014812390320003033,
+            -0.01722993701696396,
+            -0.0012570057297125459,
+            -0.00373853393830359,
+            0.01935611665248871,
+            -0.017623674124479294,
+            0.020001845434308052,
+            -0.011906610801815987,
+            0.005181580316275358,
+            -0.026490632444620132,
+            -0.008481097407639027,
+            0.01644246280193329,
+            -0.015505367890000343,
+            0.0024195145815610886,
+            -0.015458119101822376,
+            -0.003754283534362912,
+            0.006665969267487526,
+            0.001975575927644968,
+            0.001650742837227881,
+            0.005575317423790693,
+            0.005201267544180155,
+            0.016662955284118652,
+            0.008110984228551388,
+            -0.017450429499149323,
+            -0.0032562059350311756,
+            0.008370851166546345,
+            -0.015135254710912704,
+            0.006961272098124027,
+            0.005862745922058821,
+            0.0009361099801026285,
+            0.04082266613841057,
+            0.0066384077072143555,
+            0.002084837993606925,
+            -0.009898550808429718,
+            0.0037798762787133455,
+            -0.036727797240018845,
+            0.02904205024242401,
+            -0.0016576332272961736,
+            0.016221968457102776,
+            0.001489310641773045,
+            -0.014465902000665665,
+            0.0045791626907885075,
+            -0.005154018756002188,
+            0.014828139916062355,
+            -0.006287981756031513,
+            0.006969146896153688,
+            -0.00131114455871284,
+            0.017387431114912033,
+            -0.007339260075241327,
+            -0.018899381160736084,
+            0.009142575785517693,
+            -0.011316005140542984,
+            0.0007323510362766683,
+            -0.009693807922303677,
+            0.0055280691012740135,
+            0.01352880708873272,
+            0.0070203328505158424,
+            0.005480820778757334,
+            -0.03808225318789482,
+            -0.008622842840850353,
+            -0.003271955531090498,
+            0.002777815330773592,
+            -0.0024687317200005054,
+            -0.023970715701580048,
+            -0.024584945291280746,
+            -0.0002655264688655734,
+            0.02170279063284397,
+            -0.006587222218513489,
+            -0.0011831800220534205,
+            0.009118951857089996,
+            0.0015503398608416319,
+            -0.011749115772545338,
+            -0.006980959326028824,
+            0.007421944756060839,
+            0.021246055141091347,
+            -0.025514164939522743,
+            -0.020883817225694656,
+            0.012048356235027313,
+            0.0029274355620145798,
+            -0.019875850528478622,
+            -0.01033953670412302,
+            -0.0020651512313634157,
+            0.008512596599757671,
+            0.03439687564969063,
+            -0.017686670646071434,
+            0.001204835600219667,
+            0.0008947675814852118,
+            0.014080039225518703,
+            -0.026301639154553413,
+            -0.005043772514909506,
+            -0.01112701091915369,
+            -0.010418283753097057,
+            0.01148924883455038,
+            -0.01631646603345871,
+            0.0070518315769732,
+            -0.002840813249349594,
+            -0.002578978193923831,
+            -0.014064289629459381,
+            0.0009021501755341887,
+            0.0013810329837724566,
+            0.0016064473893493414,
+            0.03316841647028923,
+            -0.009567812085151672,
+            -0.013505183160305023,
+            0.0018968285294249654,
+            0.00041391613194718957,
+            0.014922636561095715,
+            0.010465532541275024,
+            0.01976560428738594,
+            -0.008331477642059326,
+            0.001558214658871293,
+            0.00961506087332964,
+            0.014623397029936314,
+            0.0005468024173751473,
+            0.042240116745233536,
+            -0.005795810371637344,
+            -0.006725030019879341,
+            -0.011788489297032356,
+            -0.005390261299908161,
+            -0.01233972143381834,
+            0.01491476222872734,
+            0.00269119325093925,
+            0.010371035896241665,
+            -0.0020828694105148315,
+            -0.008945707231760025,
+            0.012111353687942028,
+            0.007445569150149822,
+            0.010560029186308384,
+            0.013205942697823048,
+            -0.004014149773865938,
+            -0.02001759596168995,
+            0.010063921101391315,
+            -0.010048171505331993,
+            0.006059614475816488,
+            -0.001052262494340539,
+            0.008866960182785988,
+            -0.007996801286935806,
+            0.018631640821695328,
+            -0.0028703436255455017,
+            0.017607923597097397,
+            0.015560491010546684,
+            -0.019545109942555428,
+            0.017072441056370735,
+            0.01782841607928276,
+            -0.010788396932184696,
+            0.012780707329511642,
+            -0.01648971065878868,
+            -0.010237164795398712,
+            -0.0052524530328810215,
+            0.02176578715443611,
+            -0.013142945244908333,
+            -0.013324064202606678,
+            -0.010922268033027649,
+            0.00395508948713541,
+            -0.007988926023244858,
+            0.017324432730674744,
+            0.00588243268430233,
+            -0.009890676476061344,
+            -0.019686855375766754,
+            0.015505367890000343,
+            0.004433480091392994,
+            0.005213079508394003,
+            0.022474514320492744,
+            -0.007898367010056973,
+            0.005878495052456856,
+            0.014717893674969673,
+            0.030349256470799446,
+            -0.008323602378368378,
+            0.02831757441163063,
+            -0.011111261323094368,
+            0.00266756908968091,
+            -0.0037601895164698362,
+            -0.016158971935510635,
+            -0.012402718886733055,
+            -8.889845776138827e-05,
+            0.0050280229188501835,
+            0.012930327095091343,
+            -0.028601063415408134,
+            0.00861496850848198,
+            0.026222892105579376,
+            0.0027502537705004215,
+            0.008937832899391651,
+            0.0026537880767136812,
+            0.006890399381518364,
+            0.005386323668062687,
+            0.0029943708796054125,
+            -0.022364268079400063,
+            -0.007126641925424337,
+            0.00323848775587976,
+            -0.010851395316421986,
+            0.02104131132364273,
+            0.014355655759572983,
+            0.01067027635872364,
+            -0.002246270189061761,
+            -0.0012619274202734232,
+            0.025151927024126053,
+            0.01959235966205597,
+            0.004051554948091507,
+            -0.001164477551355958,
+            0.006854963023215532,
+            0.0023191117215901613,
+            0.011875111609697342,
+            -0.014190285466611385,
+            -0.014166661538183689,
+            0.007331385277211666,
+            0.00262228911742568,
+            0.010890768840909004,
+            -0.01680470071732998,
+            -0.006449413951486349,
+            -0.003303454490378499,
+            0.01746617816388607,
+            -0.009993048384785652,
+            -0.0018879694398492575,
+            0.0012599588371813297,
+            0.015253376215696335,
+            0.00039250668487511575,
+            -0.0032680181320756674,
+            -0.004724845290184021,
+            -0.014119412750005722,
+            0.0047996556386351585,
+            -0.009725307114422321,
+            0.004894152283668518,
+            -0.005661939736455679,
+            0.00881971139460802,
+            -0.007311698514968157,
+            0.04378356784582138,
+            0.027876587584614754,
+            0.008512596599757671,
+            0.017371682450175285,
+            -0.003986588213592768,
+            0.0017127564642578363,
+            -0.016119597479701042,
+            0.020222337916493416,
+            0.005598941817879677,
+            4.361868923297152e-05,
+            0.003966901451349258,
+            0.024915684014558792,
+            0.004728782922029495,
+            0.009733181446790695,
+            0.0006260420195758343,
+            0.006394290830940008,
+            0.0026439446955919266,
+            0.009410317055881023,
+            -0.018474144861102104,
+            -0.01507225725799799,
+            -0.014497400261461735,
+            0.02316749282181263,
+            -0.005512319505214691,
+            -0.008567719720304012,
+            0.013733550906181335,
+            0.011111261323094368,
+            -0.002498262096196413,
+            -0.014670644886791706,
+            -0.005693438928574324,
+            -0.0008736042072996497,
+            -0.005161893554031849,
+            -0.016269218176603317,
+            0.00297862128354609,
+            0.0020316834561526775,
+            -0.012182226404547691,
+            0.021372050046920776,
+            -0.012166476808488369,
+            -0.020395582541823387,
+            -0.003490479663014412,
+            -0.006843151059001684,
+            -0.04022418335080147,
+            -0.019056877121329308,
+            0.011906610801815987,
+            0.010788396932184696,
+            -0.008591343648731709,
+            -0.03480635955929756,
+            0.021261803805828094,
+            0.0060832384042441845,
+            0.03808225318789482,
+            -0.026065398007631302,
+            0.0023762034252285957,
+            0.0006777200032956898,
+            0.00397477624937892,
+            -0.011268756352365017,
+            -4.0081209590425715e-05,
+            0.019419115036725998,
+            -0.0007800916791893542,
+            0.00812673382461071,
+            -0.00542963482439518,
+            -0.010922268033027649,
+            0.014686394482851028,
+            -0.007209326606243849,
+            0.012906703166663647,
+            -0.006161985918879509,
+            -0.014442277140915394,
+            0.0032424251548945904,
+            0.016631456092000008,
+            0.012607462704181671,
+            -0.0017993786605075002,
+            -0.014670644886791706,
+            -0.009567812085151672,
+            0.00433504581451416,
+            -0.013331938534975052,
+            -0.012890953570604324,
+            0.02378172241151333,
+            -0.0166787039488554,
+            0.018111906945705414,
+            0.013205942697823048,
+            -0.017182687297463417,
+            0.004839029163122177,
+            0.006035990081727505,
+            0.006008428521454334,
+            0.007567627355456352,
+            -0.015064381994307041,
+            -0.019434863701462746,
+            0.004476791247725487,
+            0.026128394529223442,
+            -0.00240376521833241,
+            -0.006020240485668182,
+            0.012544464319944382,
+            -0.0026951306499540806,
+            0.008512596599757671,
+            -0.023387985303997993,
+            -0.0023821096401661634,
+            0.010993140749633312,
+            0.015568365342915058,
+            0.015686485916376114,
+            0.02335648611187935,
+            0.009181949310004711,
+            0.012402718886733055,
+            -0.012323971837759018,
+            0.005177643150091171,
+            0.022726505994796753,
+            -0.009630809538066387,
+            -0.018883632495999336,
+            0.011379002593457699,
+            0.01763942278921604,
+            0.023687224835157394,
+            0.0018456426914781332,
+            -0.010268663987517357,
+            0.030916238203644753,
+            0.012284598313272,
+            -0.009063828736543655,
+            0.014135162346065044,
+            -0.010906518436968327,
+            0.024962933734059334,
+            0.010237164795398712,
+            0.016174720600247383,
+            -0.01219010166823864,
+            -0.010835645720362663,
+            0.006646282505244017,
+            -0.011567996814846992,
+            -0.007410132791846991,
+            0.01249721646308899,
+            -0.020175090059638023,
+            -0.003384170588105917,
+            -0.031325723975896835,
+            0.00638247886672616,
+            0.003254237351939082,
+            0.0035790703259408474,
+            -0.02461644448339939,
+            0.005658002570271492,
+            -0.006146236322820187,
+            0.008079485967755318,
+            -0.011048263870179653,
+            -0.031262725591659546,
+            0.0039058721158653498,
+            -0.006496662274003029,
+            0.002059245016425848,
+            -0.01433990616351366,
+            0.01589122973382473,
+            0.007114829961210489,
+            -0.021088559180498123,
+            0.0010315912077203393,
+            -0.01904112659394741,
+            -0.0072565749287605286,
+            0.0003491956158541143,
+            -0.0023880156222730875,
+            -0.016347965225577354,
+            0.011363252997398376,
+            0.005886369850486517,
+            -0.02527792379260063,
+            -0.006937648169696331,
+            -0.0024687317200005054,
+            0.0007170937024056911,
+            0.016048725694417953,
+            -0.006055676843971014,
+            0.0022324894089251757,
+            0.033388908952474594,
+            0.015867605805397034,
+            -0.008488972671329975,
+            -0.025624411180615425,
+            -0.0028644376434385777,
+            -0.02359272725880146,
+            0.015631362795829773,
+            -0.0007953489548526704,
+            0.005650127772241831,
+            -0.029971269890666008,
+            0.005150081589818001,
+            0.006709280423820019,
+            0.01746617816388607,
+            0.006488787941634655,
+            0.01033953670412302,
+            0.026837121695280075,
+            -0.005106770433485508,
+            -0.001383001683279872,
+            -1.0535543879086617e-05,
+            0.00679196510463953,
+            0.018411148339509964,
+            -0.0036184440832585096,
+            0.01812765747308731,
+            -0.009607185609638691,
+            0.0029136547818779945,
+            -0.012654710561037064,
+            0.0035554461646825075,
+            -0.003462917869910598,
+            -0.013095696456730366,
+            -9.954166307579726e-05,
+            0.012686209753155708,
+            0.003395982552319765,
+            -0.0012727552093565464,
+            -0.018836384639143944,
+            -0.003833030816167593,
+            -0.0012530683306977153,
+            -0.021261803805828094,
+            0.0029628719203174114,
+            -0.01970260590314865,
+            -0.01124513242393732,
+            -0.02778209187090397,
+            -0.0025435418356209993,
+            0.04438204690814018,
+            0.00955206248909235,
+            -0.012331846170127392,
+            0.002647882094606757,
+            -0.02067907340824604,
+            0.0009956627618521452,
+            -0.012213725596666336,
+            -0.0036479744594544172,
+            -0.006886462215334177,
+            -0.013820173218846321,
+            0.014631271362304688,
+            0.01735593192279339,
+            -0.00404761778190732,
+            -0.006701405625790358,
+            0.012481466867029667,
+            -0.01070177461951971,
+            -0.009504813700914383,
+            -0.017576424404978752,
+            -0.017922913655638695,
+            -0.00785505585372448,
+            0.018883632495999336,
+            -0.005795810371637344,
+            0.02127755433320999,
+            0.008544095791876316,
+            0.0034353563096374273,
+            -0.00692977337166667,
+            0.0015168722020462155,
+            -0.021498046815395355,
+            -0.0006855947431176901,
+            0.006287981756031513,
+            -0.01433990616351366,
+            0.016599956899881363,
+            -0.006815589498728514,
+            -0.0031361160799860954,
+            0.008725214749574661,
+            0.004413793329149485,
+            -0.008725214749574661,
+            -0.005516257137060165,
+            -0.017135439440608025,
+            0.018820634111762047,
+            -0.018332399427890778,
+            -0.0027502537705004215,
+            0.00022246147273108363,
+            -0.009969423525035381,
+            0.012158602476119995,
+            0.015591990202665329,
+            0.007481005042791367,
+            -0.010378910228610039,
+            0.006610846146941185,
+            -0.010292287915945053,
+            0.0221437755972147,
+            -0.001872219960205257,
+            0.01612747274339199,
+            0.013024823740124702,
+            0.005433572456240654,
+            0.006284044589847326,
+            0.001713740755803883,
+            -0.012544464319944382,
+            -0.006949460133910179,
+            0.011064013466238976,
+            -0.02571890875697136,
+            -0.01349730882793665,
+            0.005917869042605162,
+            0.002801439492031932,
+            -0.01148924883455038,
+            -0.0007451475248672068,
+            -0.009205574169754982,
+            0.00037798762787133455,
+            -0.008481097407639027,
+            0.016710203140974045,
+            -0.03433387726545334,
+            -0.00861496850848198,
+            0.0017580362036824226,
+            0.031136730685830116,
+            -0.008449598215520382,
+            0.01619047112762928,
+            0.007532190997153521,
+            0.006488787941634655,
+            0.023687224835157394,
+            0.001509981811977923,
+            0.007662124466150999,
+            0.02116730809211731,
+            -0.011095511727035046,
+            0.00785505585372448,
+            0.0058036851696670055,
+            -0.01995459757745266,
+            0.025120427832007408,
+            -0.0061856103129684925,
+            0.005906056612730026,
+            0.002665600273758173,
+            0.001731458934955299,
+            0.02571890875697136,
+            0.027624595910310745,
+            -0.00870158988982439,
+            -0.005665877368301153,
+            -0.013379187323153019,
+            0.027262357994914055,
+            -0.004945338238030672,
+            0.007610938511788845,
+            -0.010197791270911694,
+            0.002689224435016513,
+            0.0007668030448257923,
+            -0.01024504005908966,
+            0.021482296288013458,
+            -0.016032975167036057,
+            -0.011875111609697342,
+            -0.017560675740242004,
+            -0.002472669119015336,
+            -0.024222707375884056,
+            0.013536682352423668,
+            0.0025829155929386616,
+            -0.003352671628817916,
+            0.0023821096401661634,
+            0.006484850309789181,
+            -0.003496385645121336,
+            -0.00940244272351265,
+            -0.0020887753926217556,
+            -0.01170186698436737,
+            -0.020348334684967995,
+            0.0166787039488554,
+            -0.007248700130730867,
+            -0.002116336952894926,
+            -0.0012127102818340063,
+            0.009811929427087307,
+            0.014985634945333004,
+            0.02129330299794674,
+            0.002915623364970088,
+            0.0066344705410301685,
+            0.007311698514968157,
+            -0.0019332492956891656,
+            -0.012032606638967991,
+            0.020427081733942032,
+            0.0197183545678854,
+            0.0003445199690759182,
+            -0.0031695838551968336,
+            -0.0109143927693367,
+            -0.035562336444854736,
+            -0.009733181446790695,
+            0.0008125749882310629,
+            0.011938109062612057,
+            -0.012442093342542648,
+            -0.007103017531335354,
+            0.039027221500873566,
+            -0.003220769576728344,
+            0.02110430970788002,
+            -0.004189362749457359,
+            3.842013029498048e-05,
+            0.01674170233309269,
+            0.004606724251061678,
+            0.012142852880060673,
+            -0.01264683622866869,
+            -0.02327773906290531,
+            0.002559291198849678,
+            -0.012961826287209988,
+            0.002697099233046174,
+            0.0004523055104073137,
+            -0.0002670029934961349,
+            -0.019497862085700035,
+            -0.008504721336066723,
+            0.005189455114305019,
+            -0.01128450594842434,
+            0.010040296241641045,
+            -0.004157864023000002,
+            0.017387431114912033,
+            -0.0030179950408637524,
+            0.0037897196598351,
+            -0.008583469316363335,
+            -0.006811652332544327,
+            0.0011575871612876654,
+            -0.0017304746434092522,
+            -0.017198437824845314,
+            -8.342305227415636e-05,
+            -0.0020887753926217556,
+            -0.0003410747740417719,
+            0.008410224691033363,
+            0.008378725498914719,
+            0.01328469067811966,
+            0.01385954674333334,
+            -0.0009119936148636043,
+            0.006996708456426859,
+            0.009008705615997314,
+            0.02735685557126999,
+            -0.007185702212154865,
+            0.0068077147006988525,
+            0.002291549928486347,
+            0.008012549951672554,
+            -0.012946076691150665,
+            -0.03285342454910278,
+            0.015426619909703732,
+            -0.015780983492732048,
+            -0.0048035928048193455,
+            -0.014717893674969673,
+            0.01040253508836031,
+            -0.004445292055606842,
+            0.0010876987362280488,
+            0.006098988000303507,
+            0.00845747347921133,
+            0.004937463440001011,
+            0.019308868795633316,
+            -0.007213264238089323,
+            0.0071935770101845264,
+            0.021309053525328636,
+            -0.003295579692348838,
+            -0.002515980275347829,
+            0.006563597824424505,
+            0.009323694743216038,
+            -0.01644246280193329,
+            0.01103251427412033,
+            0.024647943675518036,
+            -0.023198990151286125,
+            0.004441354889422655,
+            0.002891999203711748,
+            -0.01070177461951971,
+            0.004126364830881357,
+            0.0065320986323058605,
+            -0.00033024701406247914,
+            0.023009996861219406,
+            0.004587037488818169,
+            0.019576609134674072,
+            0.004022024571895599,
+            -0.014670644886791706,
+            -0.015599864535033703,
+            0.006646282505244017,
+            -0.010890768840909004,
+            -0.011016764678061008,
+            -0.008063736371695995,
+            -0.010693900287151337,
+            0.0054611340165138245,
+            -0.004787843208760023,
+            -0.004022024571895599,
+            -0.0037739702966064215,
+            -0.011253006756305695,
+            0.006827401462942362,
+            -0.01148924883455038,
+            0.001115260412916541,
+            -0.00638247886672616,
+            -0.005280014593154192,
+            0.00945756584405899,
+            0.011851487681269646,
+            -0.010008797980844975,
+            -0.02674262598156929,
+            0.005555630661547184,
+            0.031199729070067406,
+            -0.006728967186063528,
+            0.007929865270853043,
+            0.012355471029877663,
+            -0.00745738111436367,
+            0.000588144815992564,
+            -0.006732904817909002,
+            -0.004650035407394171,
+            -0.03105798363685608,
+            -0.0010227321181446314,
+            -0.01067027635872364,
+            0.0003351687337271869,
+            -0.014245408587157726,
+            -0.015017134137451649,
+            -0.013166569173336029,
+            0.004968962632119656,
+            -0.013142945244908333,
+            -0.005043772514909506,
+            0.004149989224970341,
+            0.006988833658397198,
+            -0.016379464417696,
+            0.017576424404978752,
+            -0.00943394098430872,
+            -0.003933433908969164,
+            -0.00592968100681901,
+            0.033451907336711884,
+            -0.005406010430306196,
+            -0.003504260443150997,
+            -0.00492958864197135,
+            0.008874834515154362,
+            0.006776215974241495,
+            0.002053339034318924,
+            -0.0029038111679255962,
+            0.0021379925310611725,
+            0.02323048934340477,
+            0.006232858635485172,
+            0.003397951368242502,
+            0.015536867082118988,
+            -0.013473683968186378,
+            -0.008044049143791199,
+            -0.012654710561037064,
+            -0.01067027635872364,
+            0.0020454642362892628,
+            0.010985265485942364,
+            0.02365572564303875,
+            -0.005083146039396524,
+            -0.014985634945333004,
+            0.011473499238491058,
+            0.006772278342396021,
+            0.02140354923903942,
+            -0.011843612417578697,
+            5.6138298532459885e-05,
+            0.00812673382461071,
+            0.02376597188413143,
+            0.031813960522413254,
+            -0.000510381767526269,
+            7.437940075760707e-05,
+            0.0083393519744277,
+            0.010615152306854725,
+            -0.004980774596333504,
+            0.00664234533905983,
+            -0.01704094186425209,
+            -0.02795533463358879,
+            -0.008898458443582058,
+            0.006646282505244017,
+            -0.001707834773696959,
+            0.018647389486432076,
+            0.01007179543375969,
+            0.016757450997829437,
+            0.02382897026836872,
+            0.023025747388601303,
+            0.00039545970503240824,
+            0.02037983387708664,
+            -0.004425605293363333,
+            -0.004417730495333672,
+            0.0025691348128020763,
+            -0.015048632398247719,
+            -0.03471186384558678,
+            0.006118674762547016,
+            0.00442954245954752,
+            0.024773940443992615,
+            -0.015757359564304352,
+            0.023561229929327965,
+            -0.004146052058786154,
+            -0.016158971935510635,
+            -0.005898181814700365,
+            0.020789319649338722,
+            -0.008772462606430054,
+            0.020789319649338722,
+            0.0013239410473033786,
+            -0.0010945892427116632,
+            0.00988280214369297,
+            -0.0011201821034774184,
+            -0.01379654835909605,
+            0.011253006756305695,
+            0.01763942278921604,
+            0.03770426660776138,
+            0.009630809538066387,
+            -0.008118859492242336,
+            0.01752917654812336,
+            0.001776738790795207,
+            -0.008394475094974041,
+            -0.004433480091392994,
+            -0.013820173218846321,
+            0.0076936231926083565,
+            -0.0074180071242153645,
+            0.01697794534265995,
+            0.002640007296577096,
+            -0.005260327830910683,
+            0.014725768007338047,
+            -0.012481466867029667,
+            0.01219010166823864,
+            0.021246055141091347,
+            0.0011949921026825905,
+            0.005886369850486517,
+            0.020001845434308052,
+            -0.033136915415525436,
+            0.011528623290359974,
+            -0.034837860614061356,
+            -0.005138269625604153,
+            0.012064104899764061,
+            -0.01861589029431343,
+            0.007732997182756662,
+            -0.018584391102194786,
+            -0.013253191486001015,
+            -0.002553385216742754,
+            0.014599772170186043,
+            0.02497868239879608,
+            0.006555723026394844,
+            -0.017670921981334686,
+            0.01179636362940073,
+            -0.00876458827406168,
+            -0.01825365237891674,
+            -0.008874834515154362,
+            -0.019419115036725998,
+            -0.011693992651998997,
+            -0.003850748995319009,
+            0.011938109062612057,
+            -0.008811837062239647,
+            0.0037011289969086647,
+            0.002248239004984498,
+            -0.0221437755972147,
+            -0.013883170671761036,
+            0.015914853662252426,
+            0.003413700731471181,
+            -0.00664234533905983,
+            -0.007343197241425514,
+            0.013016949407756329,
+            -0.010756897740066051,
+            0.007323510479182005,
+            -0.008181856945157051,
+            0.014253283850848675,
+            -0.012142852880060673,
+            -0.00319123943336308,
+            0.007079393602907658,
+            0.0035869451239705086,
+            -0.018033159896731377,
+            -0.004732720088213682,
+            -0.02493143454194069,
+            0.015355747193098068,
+            0.0029766527004539967,
+            -0.017481928691267967,
+            -0.00692977337166667,
+            0.007732997182756662,
+            -0.017922913655638695,
+            -0.004894152283668518,
+            0.008717339485883713,
+            -0.003990525845438242,
+            0.025340920314192772,
+            0.022301269695162773,
+            -0.01248934119939804,
+            0.00016143222455866635,
+            -0.009213448502123356,
+            0.009985173121094704,
+            -0.01856864243745804,
+            -0.004484665580093861,
+            0.02650638297200203,
+            0.007315635681152344,
+            -0.00550444470718503,
+            -0.009937925264239311,
+            0.001173336640931666,
+            0.002771909348666668,
+            0.017607923597097397,
+            0.02493143454194069,
+            0.00621710903942585,
+            -0.010985265485942364,
+            -0.022458765655755997,
+            -0.027388354763388634,
+            0.027372604236006737,
+            0.0007485927199013531,
+            -0.004118490032851696,
+            0.013276815414428711,
+            0.02529367245733738,
+            0.023939216509461403,
+            -0.0062013594433665276,
+            -0.00483509199693799,
+            0.018300902098417282,
+            0.00997729878872633,
+            -0.005402073264122009,
+            -0.010166292078793049,
+            0.003392045386135578,
+            -0.0071345167234539986,
+            0.03414488211274147,
+            0.010111168958246708,
+            0.004417730495333672,
+            -0.046366482973098755,
+            0.009993048384785652,
+            -0.01619047112762928,
+            -0.003758220700547099,
+            0.008236980065703392,
+            0.021734289824962616,
+            0.010095419362187386,
+            0.013032699003815651,
+            0.007984988391399384,
+            -0.007236888166517019,
+            0.02861681394279003,
+            -0.0028703436255455017,
+            -0.010623027570545673,
+            -0.007272324524819851,
+            -0.025687409564852715,
+            0.004342920612543821,
+            -0.016410963609814644,
+            0.0021517733111977577,
+            0.005677689332515001,
+            -0.00918982457369566,
+            0.02590790204703808,
+            0.00794167723506689,
+            -0.02382897026836872,
+            0.00588243268430233,
+            -0.03682229667901993,
+            0.019482113420963287,
+            -0.02619139291346073,
+            -0.009323694743216038,
+            -0.004862653557211161,
+            -0.003758220700547099,
+            -0.025923652574419975,
+            -0.0037365653552114964,
+            -0.004783906042575836,
+            0.016883447766304016,
+            0.0024017964024096727,
+            -0.00013805407797917724,
+            -0.00013399365707300603,
+            -0.007103017531335354,
+            0.002041527070105076,
+            0.012410594150424004,
+            0.022521764039993286,
+            0.01571798510849476,
+            -0.0037897196598351,
+            0.018521394580602646,
+            -0.0005768248811364174,
+            -0.03704278916120529,
+            0.0037168783601373434,
+            -0.011662493459880352,
+            0.008103109896183014,
+            0.015332123264670372,
+            0.009591436013579369,
+            -0.01370205171406269,
+            0.013190193101763725,
+            -0.012150727212429047,
+            0.015395121648907661,
+            0.01619047112762928,
+            0.004894152283668518,
+            -0.0009976314613595605,
+            0.0157652348279953,
+            -0.02389196865260601,
+            0.00020941892580594867,
+            -0.00626435736194253,
+            0.006713218055665493,
+            -0.009930050000548363,
+            -0.0033251100685447454,
+            0.002041527070105076,
+            0.0006855947431176901,
+            -0.0034038573503494263,
+            -0.012064104899764061,
+            -0.017387431114912033,
+            -0.021498046815395355,
+            -0.01428478304296732,
+            -0.020820818841457367,
+            -0.011001015082001686,
+            0.030711494386196136,
+            -0.008622842840850353,
+            0.004559475928544998,
+            0.015025008469820023,
+            0.018710387870669365,
+            0.002596696373075247,
+            0.01856864243745804,
+            0.008260604925453663,
+            -0.007299886085093021,
+            -0.01727718487381935,
+            0.005543818697333336,
+            -0.025435417890548706,
+            0.03310541808605194,
+            0.00933944433927536,
+            -0.014032791368663311,
+            0.012969700619578362,
+            -0.01070177461951971,
+            -0.004346857778728008,
+            -0.007784182671457529,
+            0.0060163033194839954,
+            -0.025498416274785995,
+            0.01455252431333065,
+            -0.011749115772545338,
+            0.006961272098124027,
+            0.002305330941453576,
+            0.03505835309624672,
+            -0.014489525929093361,
+            -0.008929957635700703,
+            0.012032606638967991,
+            0.019119873642921448,
+            0.026553630828857422,
+            -0.009441816247999668,
+            -0.004937463440001011,
+            -0.01814340613782406,
+            -0.00357119576074183,
+            -0.005906056612730026,
+            -0.018631640821695328,
+            0.03590882569551468,
+            -0.007847180590033531,
+            -0.021387800574302673,
+            -0.00824485532939434,
+            -0.02006484381854534,
+            0.0031262726988643408,
+            0.0315462164580822,
+            0.01040253508836031,
+            0.006890399381518364,
+            -8.92060634214431e-05,
+            -0.020285336300730705,
+            -0.021324802190065384,
+            0.019797101616859436,
+            0.012166476808488369,
+            -0.019844351336359978,
+            -0.0009695776388980448,
+            -0.009394567459821701,
+            -0.00216752290725708,
+            0.0006609861738979816,
+            0.0021773662883788347,
+            0.05780060961842537,
+            -0.008087360300123692,
+            0.0013229567557573318,
+            -0.024065213277935982,
+            -0.001320003648288548,
+            -0.003332984633743763,
+            -0.0035318220034241676,
+            0.005646190140396357,
+            -0.00626435736194253,
+            0.01128450594842434,
+            -0.03480635955929756,
+            0.006784090772271156,
+            -0.010607277974486351,
+            -0.009654434397816658,
+            -0.016111722216010094,
+            0.00166747672483325,
+            -0.018285151571035385,
+            -4.6664001274621114e-05,
+            0.00021680150530301034,
+            -0.012134977616369724,
+            0.004083054140210152,
+            -0.013883170671761036,
+            -0.0003961979818996042,
+            0.024553446099162102,
+            0.011686117388308048,
+            -0.0013268940383568406,
+            -0.022222522646188736,
+            -0.022395767271518707,
+            0.0015562459593638778,
+            -0.010394659824669361,
+            0.01801741123199463,
+            -0.013229567557573318,
+            0.017182687297463417,
+            -0.02165554091334343,
+            -0.00046780891716480255,
+            0.0019765603356063366,
+            0.006232858635485172,
+            0.01861589029431343,
+            -0.00010969270078931004,
+            0.009693807922303677,
+            -0.014993509277701378,
+            0.02535667084157467,
+            -0.006949460133910179,
+            -0.02019083872437477,
+            0.0007328432402573526,
+            0.005181580316275358,
+            -0.0044216676615178585,
+            -0.0003430434735491872,
+            0.011056138202548027,
+            0.019419115036725998,
+            -0.0024766065180301666,
+            -0.007839306257665157,
+            -0.014064289629459381,
+            -0.009930050000548363,
+            0.0194033645093441,
+            0.0009297117940150201,
+            0.02055307850241661,
+            -0.011536497622728348,
+            0.00918982457369566,
+            -0.017623674124479294,
+            -0.007662124466150999,
+            -0.00907170306891203,
+            -0.004022024571895599,
+            0.016725953668355942,
+            -0.0007599126547574997,
+            0.01512738037854433,
+            -0.018001660704612732,
+            -0.021907532587647438,
+            0.0006098003359511495,
+            -0.02348248101770878,
+            -0.018411148339509964,
+            0.009205574169754982,
+            -0.01855289377272129,
+            -0.0025986649561673403,
+            0.001046356395818293,
+            -0.020883817225694656,
+            -0.013473683968186378,
+            0.018694639205932617,
+            -0.001501122722402215,
+            -0.0029628719203174114,
+            0.008496847003698349,
+            -0.006343104876577854,
+            -0.021813036873936653,
+            -0.0039688702672719955,
+            0.009512688964605331,
+            0.009748931042850018,
+            0.012993324548006058,
+            -0.022222522646188736,
+            0.004724845290184021,
+            0.0015719954390078783,
+            -0.005350887309759855,
+            -0.007095142733305693,
+            0.018174905329942703,
+            0.008047986775636673,
+            0.004531914368271828,
+            0.007768433541059494,
+            0.010008797980844975,
+            -0.0009479221189394593,
+            0.008370851166546345,
+            0.02146654762327671,
+            0.00108376145362854,
+            -0.002827032469213009,
+            -0.016347965225577354,
+            0.00612261239439249,
+            -0.012292472645640373,
+            0.015473868697881699,
+            0.02589215338230133,
+            0.008355101570487022,
+            -0.016332216560840607,
+            0.020883817225694656,
+            -0.014481651596724987,
+            0.015151004306972027,
+            -0.0282073263078928,
+            -0.00988280214369297,
+            0.018159156665205956,
+            -0.0011546340538188815,
+            0.01842689700424671,
+            -0.003108554519712925,
+            0.024065213277935982,
+            0.012481466867029667,
+            0.006417915225028992,
+            1.7518224922241643e-05,
+            -0.0007170937024056911,
+            0.010063921101391315,
+            -0.005189455114305019,
+            0.00014851271407678723,
+            -0.005740687251091003,
+            0.009134701453149319,
+            -0.006551785860210657,
+            0.00027832292835228145,
+            0.010725399479269981,
+            -0.028065582737326622,
+            0.014332030899822712,
+            -0.02274225652217865,
+            -0.0036735672038048506,
+            -0.0024431387428194284,
+            -0.007146328687667847,
+            -0.02031683549284935,
+            -0.003079024376347661,
+            0.019088376313447952,
+            -0.018489895388484,
+            0.04819342494010925,
+            0.0012028669007122517,
+            0.018300902098417282,
+            -0.007453443482518196,
+            0.002006090711802244,
+            -0.015143129974603653,
+            -0.00035928512807004154,
+            -0.02674262598156929,
+            0.00807161070406437,
+            0.003821218851953745,
+            0.0024254205636680126,
+            0.0025238548405468464,
+            0.006902211811393499,
+            -0.002173428889364004,
+            0.026648128405213356,
+            0.0002606047492008656,
+            0.01127663068473339,
+            -0.004295671824365854,
+            0.010717524215579033,
+            -0.019261619076132774,
+            0.004327171016484499,
+            0.0034491370897740126,
+            -0.00797711405903101,
+            -0.003933433908969164,
+            -0.007091205567121506,
+            0.0221437755972147,
+            0.016631456092000008,
+            0.009662308730185032,
+            -0.008020425215363503,
+            -0.0013042541686445475,
+            0.006803777534514666,
+            -0.0030396506190299988,
+            -0.011756990104913712,
+            -0.0038724045734852552,
+            0.005969054531306028,
+            0.012512966059148312,
+            -0.0009011658257804811,
+            0.011961733922362328,
+            0.005362699739634991,
+            -0.008008613251149654,
+            0.01297757588326931,
+            -0.020096343010663986,
+            -0.006390353664755821,
+            -0.009685933589935303,
+            -0.033987388014793396,
+            0.009292195551097393,
+            -0.0017304746434092522,
+            0.003090836340561509,
+            -0.007229013368487358,
+            0.010260789655148983,
+            -0.009520563296973705,
+            -0.01322169229388237,
+            -0.006512411870062351,
+            0.002208865247666836,
+            0.023639976978302002,
+            -0.006193485110998154,
+            0.00240376521833241,
+            -0.013355563394725323,
+            -0.013804423622786999,
+            0.016899196431040764,
+            -0.01170186698436737,
+            -0.007343197241425514,
+            0.009827678091824055
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "c07291e8-ab48-4102-9d7f-8ac7e396f122",
+      "type": "child",
+      "source": {
+        "id": "1e687808-8db8-4320-805d-0282a8d679bc",
+        "properties": {
+          "page_content": "Deepseekmoe: Towards Ultimate Expert Specialization In\n\nMixture-Of-Experts Language Models\n\nDamai Dai∗1,2, Chengqi Deng1, Chenggang Zhao∗1,3, R.X. Xu1, Huazuo Gao1, Deli Chen1, Jiashi Li1, Wangding Zeng1, Xingkai Yu∗1,4, Y. Wu1, Zhenda Xie1, Y.K. Li1, Panpan Huang1, Fuli Luo1, Chong Ruan1, Zhifang Sui2, Wenfeng Liang1 1DeepSeek-AI 2National Key Laboratory for Multimedia Information Processing, Peking University 3Institute for Interdisciplinary Information Sciences, Tsinghua University 4National Key Laboratory for Novel Software Technology, Nanjing University {daidamai, szf}@pku.edu.cn, {wenfeng.liang}@deepseek.com https://github.com/deepseek-ai/DeepSeek-MoE\n\nAbstract\n\nIn the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top- out of experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into ones and activating from them, allowing for a more flexible combination of activated experts; (2) isolating experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5× expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations.\n\nFurther, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\n1. Introduction\n\nRecent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023a). It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017) has emerged as a popular solution. It can\n\nenable parameter scaling, while concurrently keeping computational costs at a modest level.\n\nRecent applications of MoE architectures in Transformers (Vaswani et al., 2017) have yielded successful attempts at scaling language models to a substantial size (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022), accompanied with remarkable performance. These achievements underscore the considerable potential and promise of MoE language models.\n\nDespite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts.\n\nThis architecture manifests two potential issues: (1) Knowledge Hybridity: existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. (2) Knowledge Redundancy: tokens assigned to different experts may require common knowledge. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters.\n\nThese issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models.\n\nIn response to the aforementioned issues, we introduce DeepSeekMoE, an innovative MoE architecture specifically designed towards ultimate expert specialization. Our architecture involves two principal strategies: (1) Fine-Grained Expert Segmentation: while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. (2) Shared Expert Isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.\n\nThrough compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized.\n\nStarting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B (Lepikhin et al., 2021) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5× expert parameters and computation. Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE.\n\nThese studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization.\n\nLeveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B (DeepSeek-AI, 2024), a dense model trained on the same 2T corpus. We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B (Touvron et al., 2023b), which has approximately 2.5 times the activated parameters. Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard1. Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. The experimental results still validate its substantial advantages over the GShard architecture consistently. In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nOur contributions are summarized as follows: - Architectural Innovation. We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation.\n\nEmpirical Validation. We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. Experimental results validate the high 1https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models\n\nScalability. We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B.\n\nAlignment for MoE. We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B.\n\nPublic Release. In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization.\n\n2. Preliminaries: Mixture-Of-Experts For Transformers\n\nWe first introduce a generic MoE architecture commonly used in Transformer language models. A standard Transformer language model is constructed by stacking layers of standard Transformer blocks, where each block can be represented as follows:\n\n$$\\mathbf{u}{1:T}^{l}=\\text{Self-Att}\\left(\\mathbf{h}{1:T}^{l-1}\\right)+\\mathbf{h}{1:T}^{l-1},\\tag{1}$$ $$\\mathbf{h}{t}^{l}=\\text{FFN}\\left(\\mathbf{u}{t}^{l}\\right)+\\mathbf{u}{t}^{l},\\tag{2}$$\n\nwhere denotes the sequence length, Self-Att(·) denotes the self-attention module, FFN(·) denotes the Feed-Forward Network (FFN), u 1: ∈ R× are the hidden states of all tokens after the -th attention module, and h\n\n∈ Ris the output hidden state of the -th token after the -th Transformer block. For brevity, we omit the layer normalization in the above formulations.\n\nA typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022). An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. Then, each token will be assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts. If the -th FFN is substituted with an MoE layer, the computation for its output hidden state h\n\nis expressed as: where denotes the total number of experts, FFN(·) is the -th expert FFN, , denotes the gate value for the -th expert, , denotes the token-to-expert affinity, Topk(·, ) denotes the set comprising highest affinity scores among those calculated for the -th token and all experts, and e\n\nis the centroid of the -th expert in the -th layer. Note that ,is sparse, indicating that only out of gate values are nonzero. This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only experts. Also, in the above formulations, we omit the layer normalization operation for brevity.\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{N}\\left(g_{i,t}\\operatorname{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\operatorname{Topk}({s_{j,t}|1\\leqslant j\\leqslant N},K),\\ 0,&\\text{otherwise},\\end{cases}$$ $$s_{i,t}=\\operatorname{Softmax}{i}\\left(\\mathbf{u}{t}^{l}\\,\\mathbf{e}_{i}^{l}\\right),$$\n\n$$(3)$$\n\n(4) $\\binom{4}{5}$ .\n\n3. Deepseekmoe Architecture\n\nOn top of the generic MoE architecture outlined in Section 2, we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. As illustrated in Figure 2, our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. Both of these strategies are designed to elevate the level of expert specialization.\n\n3.1. Fine-Grained Expert Segmentation\n\nIn scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts.\n\nIn pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. The finer expert segmentation enables a more flexible and adaptable combination of activated experts. To be specific, on top of a typical MoE architecture shown in Figure 2(a), we segment each expert FFN into smaller experts by reducing the FFN intermediate hidden dimension to 1 times its original size. Since each expert becomes smaller, in response, we also increase the number of activated experts to times to keep the same computation cost, as illustrated in Figure 2(b). With the fine-grained expert segmentation, the output of an MoE layer can be expressed as:\n\n$$\\mathbf{h}{t}^{i}=\\sum{i=1}^{\\text{mN}}\\left(g_{i,t}\\text{FFN}{i}\\left(\\mathbf{u}{t}^{i}\\right)\\right)+\\mathbf{u}{t}^{i},\\tag{6}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\text{Tokr}(s_{i,t}|1\\leqslant j\\leqslant m\\text{N}),m\\text{K}),\\ 0,&\\text{otherwise},\\end{cases}$$ (7) $$s_{i,t}=\\text{Softmax}{i}\\left(\\mathbf{u}{t}^{T}\\mathbf{e}_{t}^{i}\\right),\\tag{8}$$ where the total number of expert parameters is equal to $N$ times the number of parameters in a\n\n$$(6)$$\n\n(7) (8) $\\frac{1}{2}$\n\nstandard FFN, and denotes the total number of fine-grained experts. With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to .\n\nFrom a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. As an illustrative example, we consider the case where = 16. A typical top-2 routing strategy can yield 16 2 = 120 possible combinations. By contrast, if each expert is split into 4 smaller experts, the fine-grained routing strategy can yield 64 8 = 4, 426, 165, 368 potential combinations. The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition.\n\n3.2. Shared Expert Isolation\n\nWith a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. This alleviation of redundancy will contribute to a more parameterefficient model with more specialized experts.\n\nTowards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate experts to serve as shared experts. Regardless of the router module, each token will be deterministically assigned to these shared experts. In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by , as depicted in Figure 2(c). With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows:\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{K_{s}}\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)+\\sum_{i=K_{s}+1}^{mN}\\left(g_{i,t}\\;\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},\\tag{9}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\mathrm{Topk}{s_{j,t}|K_{s}+1\\leqslant j\\leqslant mN},mK-K_{s}},\\ 0,&\\text{otherwise},\\end{cases}$$ (10) $$s_{i,t}=\\mathrm{Softmax}{i}\\left(\\mathbf{u}{t}^{l,T}e_{i}^{l}\\right).\\tag{11}$$ $\\mathbf{h}\\mathbf{M}\\mathbf{E}$ the number of closed-compact is $K_{s}$ the total number of routed currents.\n\nFinally, in DeepSeekMoE, the number of shared expert is , the total number of routed experts is − , and the number of nonzero gates is − .\n\nIt is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022). The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint.\n\n3.3. Load Balance Consideration\n\nAutomatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. Firstly, there is a risk of routing collapse (Shazeer et al., 2017), i.e., the model always selects only a few experts, preventing other experts from sufficient training.\n\nSecondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.\n\nExpert-Level Balance Loss. In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. The computation of the balance loss is as follows:\n\n$$\\begin{array}{l}{{{\\mathcal L}{\\mathrm{ExpBal}}=\\alpha{1}\\sum_{i=1}^{N^{\\prime}}f_{i}P_{i},}}\\ {{f_{i}=\\frac{N^{\\prime}}{K^{T}}\\sum_{t=1}^{T}\\mathds{1}\\,(\\mathrm{Token}\\;t\\;\\mathrm{s}\\mathrm{lects}\\;\\mathrm{Expert}\\;i),}}\\ {{P_{i}=\\frac{1}{T}\\sum_{t=1}^{T}s_{i,t},}}\\end{array}$$\n\n(12) $$\\begin{array}{l}\\small\\text{(13)}\\end{array}$$ = (14) $$\\begin{array}{l}\\small\\text{(14)}\\end{array}$$ .\n\n7 where 1 is a hyper-parameter called expert-level balance factor, ′is equal to ( − ) and ′ is equal to ( − ) for brevity. 1(·) denotes the indicator function.\n\nDevice-Level Balance Loss. In addition to the expert-level balance loss, we introduce a devicelevel balance loss. When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. Instead, our primary objective is to ensure balanced computation across the devices. If we partition all routed experts into groups {E1, E2, ..., E}, and deploy each group on a single device, the device-level balance loss is computed as follows:\n\n$$\\mathcal{L}{\\text{DevBal}}=\\alpha{2}\\sum_{i=1}^{D}f_{i}^{\\prime}P_{i}^{\\prime},$$ $$f_{i}^{\\prime}=\\frac{1}{|\\mathcal{E}{i}|}\\sum{j\\in\\mathcal{E}{i}}f{j},$$ $$P_{i}^{\\prime}=\\sum_{j\\in\\mathcal{E}{i}}P{j},$$\n\n(15) $$\\begin{array}{l}\\small\\text{(16)}\\end{array}$$ = (17) .\n\nwhere 2 is a hyper-parameter called device-level balance factor. In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices.\n\n4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n6. Alignment For Deepseekmoe 16B\n\nPrevious research indicates that MoE models typically do not emerge significant gains from fine-tuning (Artetxe et al., 2022; Fedus et al., 2021). However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B.\n\n6.1. Experimental Setup\n\nTraining Data. For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios.\n\nHyper-Parameters. During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).\n\nWe employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10−5 without incorporating any learning rate scheduling strategy.\n\nEvaluation Benchmarks. For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3, with the following adjustments: (1) We exclude Pile (Gao et al., 2020) since chat models are seldom employed for pure language modeling. (2) We exclude\n\nMetric # Shot LLaMA2 DeepSeek DeepSeekMoE SFT 7B Chat 7B Chat 16B # Total Params N/A 6.7B 6.9B 16.4B # Activated Params N/A 6.7B 6.9B 2.8B FLOPs per 4K Tokens N/A 187.9T 183.5T 74.4T HellaSwag (Acc.) 0-shot 67.9 71.0 72.2 PIQA (Acc.) 0-shot 76.9 78.4 79.7 ARC-easy (Acc.) 0-shot 69.7 70.2 69.9 ARC-challenge (Acc.) 0-shot 50.8 50.2 50.0 BBH (EM) 3-shot 39.3 43.1 42.2 RACE-middle (Acc.) 5-shot 63.9 66.1 64.8 RACE-high (Acc.) 5-shot 49.6 50.8 50.6 DROP (EM) 1-shot 40.0 41.7 33.8 GSM8K (EM) 0-shot 63.4 62.6 62.2 MATH (EM) 4-shot 13.5 14.7 15.2 HumanEval (Pass@1) 0-shot 35.4 45.1 45.7 MBPP (Pass@1) 3-shot 27.8 39.0 46.2 TriviaQA (EM) 5-shot 60.1 59.5 63.3 NaturalQuestions (EM) 0-shot 35.2 32.7 35.1 MMLU (Acc.) 0-shot 50.0 49.7 47.2 WinoGrande (Acc.) 0-shot 65.1 68.4 69.0 CLUEWSC (EM) 5-shot 48.4 66.2 68.2 CEval (Acc.) 0-shot 35.1 44.7 40.0 CMMLU (Acc.) 0-shot 36.9 51.2 49.3\n\nCHID (Zheng et al., 2019) due to the observed instability of results, hindering the derivation of solid conclusions. (3) We additionally include BBH (Suzgun et al., 2022) to provide a more comprehensive assessment of the reasoning ability of the chat models.\n\nTable 5 | Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations.\n\n6.2. Evaluations\n\nBaselines. In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. Correspondingly, we construct three chat models, including LLaMA2 SFT 7B3, DeepSeek Chat 7B, and DeepSeekMoE Chat 16B.\n\nSubsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks.\n\nResults. The evaluation results are presented in Table 5. Our key observations include: (1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intensive tasks (TriviaQA, NaturalQuestions). (2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. In addition, it also surpasses DeepSeek Chat 7B. (3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1). However, it is worth noting that, after supervised finetuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. (4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations.\n\n7. Deepseekmoe 145B Ongoing\n\nEncouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available.\n\n7.1. Experimental Setup\n\nTraining Data and Tokenization. For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study.\n\nModel Settings. For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps.\n\nSubsequently, the learning rate keeps constant during the remaining training process. The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. Also, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse.\n\nEvaluation Benchmarks. We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3).\n\n7.2. Evaluations\n\nBaselines. Apart from DeepSeekMoE 145B, we consider three additional models for comparison. DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters.\n\nResults. From the evaluation results presented in Table 6, we have the following observations: (1) Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. (2) On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. (3) At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5.\n\n8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\nAppendices A. Overview Of Hyper-Parameters\n\nWe present the overview of hyper-parameters for DeepSeekMoE across various sizes in Table 7.\n\n# Params # Layers Hidden # Attn # Shared # Routed Relative Sequence Batch Size Learning Size Heads Experts Experts Expert Size Length (Sequence) Rate 2.0B 9 1280 10 1 63 (7 activated) 0.25 2048 2048 1.08e-3 16.4B 28 2048 16 2 64 (6 activated) 0.25 4096 4608 4.2e-4 144.6B 62 4096 32 4 128 (12 activated) 0.125 4096 4608 3.0e-4\n\nTable 7 | Overview of hyper-parameters for DeepSeekMoE across various sizes. The relative expert size is in comparison to a standard FFN.\n\nB. Comparing Deepseekmoe With Larger Models\n\nComparisons among DeepSeekMoE, GShard×1.2, and GShard×1.5 are shown in Table 8. Comparisons among DeepSeekMoE, Dense×4, and Dense×16 are shown in Table 9.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 2.3B 2.8B 1.9B # Activated Expert Params N/A 0.28B 0.35B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.824 1.808 1.808 HellaSwag (Acc.) 0-shot 53.7 54.4 54.8 PIQA (Acc.) 0-shot 71.8 71.1 72.3 ARC-easy (Acc.) 0-shot 46.8 47.3 49.4 ARC-challenge (Acc.) 0-shot 31.7 34.1 34.3 RACE-middle (Acc.) 5-shot 43.7 46.4 44.0 RACE-high (Acc.) 5-shot 31.9 32.4 31.7 HumanEval (Pass@1) 0-shot 3.7 3.0 4.9 MBPP (Pass@1) 3-shot 2.4 2.6 2.2 TriviaQA (EM) 5-shot 15.2 15.7 16.6 NaturalQuestions (EM) 5-shot 4.5 4.7 5.7\n\nTable 8 | Comparison between DeepSeekMoE and larger GShard models.\n\nAt a larger scale of 13B total parameters, we also compare DeepSeekMoE with GShard×1.2 and GShard×1.5, and show results in Table 10. At a larger scale, DeepSeekMoE even outperforms GShard×1.5 distinctly.\n\nMetric # Shot Dense×4 Dense×16 DeepSeekMoE Relative Expert Size N/A 1 1 0.25 # Experts N/A 4 + 0 16 + 0 1 + 63 # Activated Experts N/A 4 + 0 16 + 0 1 + 7 # Total Expert Params N/A 0.47B 1.89B 1.89B # Activated Expert Params N/A 0.47B 1.89B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.908 1.806 1.808 HellaSwag (Acc.) 0-shot 47.6 55.1 54.8 PIQA (Acc.) 0-shot 70.0 71.9 72.3 ARC-easy (Acc.) 0-shot 43.9 51.9 49.4 ARC-challenge (Acc.) 0-shot 30.5 33.8 34.3 RACE-middle (Acc.) 5-shot 42.4 46.3 44.0 RACE-high (Acc.) 5-shot 30.7 33.0 31.7 HumanEval (Pass@1) 0-shot 1.8 4.3 4.9 MBPP (Pass@1) 3-shot 0.2 2.2 2.2 TriviaQA (EM) 5-shot 9.9 16.5 16.6 NaturalQuestions (EM) 5-shot 3.0 6.3 5.7\n\nTable 9 | Comparison between DeepSeekMoE and larger dense baselines.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 15.9B 19.8B 13.3B # Activated Expert Params N/A 2.37B 2.82B 2.05B # Training Tokens N/A 100B 100B 100B HellaSwag (Acc.) 0-shot 66.6 67.7 69.1 PIQA (Acc.) 0-shot 75.6 76.0 75.7 ARC-easy (Acc.) 0-shot 56.8 56.8 58.8 ARC-challenge (Acc.) 0-shot 39.9 37.6 38.5 RACE-middle (Acc.) 5-shot 51.6 50.6 52.4 RACE-high (Acc.) 5-shot 37.4 36.3 38.5 HumanEval (Pass@1) 0-shot 6.1 6.1 9.8 MBPP (Pass@1) 3-shot 7.0 11.6 10.6 TriviaQA (EM) 5-shot 36.5 36.7 38.2 NaturalQuestions (EM) 5-shot 12.6 12.1 13.7\n\nTable 10 | Comparison between DeepSeekMoE and larger GShard models at a larger scale.\n\nC. Training Benchmark Curves Of Deepseekmoe 16B\n\nWe present the benchmark curves during training of DeepSeekMoE 16B and DeepSeek 7B (Dense) in Figure 7 for reference.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/DeepSeekMoE_2.md"
+          },
+          "summary": "DeepSeekMoE is an innovative Mixture-of-Experts (MoE) architecture designed to enhance expert specialization in language models. It employs fine-grained expert segmentation and shared expert isolation to improve performance and reduce computational costs. Starting with a model of 2B parameters, DeepSeekMoE demonstrates comparable performance to larger models like GShard 2.9B and LLaMA2 7B, using significantly fewer computations. The architecture scales up to 16B and 145B parameters, maintaining advantages over traditional MoE architectures. DeepSeekMoE 16B achieves similar results to dense models with only 40% of the computations, and its chat model version shows adaptability in bilingual scenarios. The architecture's public release aims to contribute to advancements in large-scale language models.",
+          "headlines": {
+            "1. Introduction": [],
+            "2. Preliminaries: Mixture-Of-Experts For Transformers": [],
+            "3. Deepseekmoe Architecture": [
+              "3.1. Fine-Grained Expert Segmentation",
+              "3.2. Shared Expert Isolation",
+              "3.3. Load Balance Consideration"
+            ],
+            "4. Validation Experiments": [
+              "4.1. Experimental Setup",
+              "4.1.1. Training Data And Tokenization",
+              "4.1.2. Infrastructures",
+              "4.1.3. Hyper-Parameters",
+              "4.1.4. Evaluation Benchmarks",
+              "4.2. Evaluations",
+              "4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models",
+              "4.4. Ablation Studies",
+              "4.5. Analysis On Expert Specialization"
+            ],
+            "5. Scaling Up To Deepseekmoe 16B": [
+              "5.1. Experimental Setup",
+              "5.1.1. Training Data And Tokenization",
+              "5.1.2. Hyper-Parameters",
+              "5.1.3. Evaluation Benchmarks",
+              "5.2. Evaluations",
+              "5.2.1. Internal Comparison With Deepseek 7B",
+              "5.2.2. Comparison With Open Source Models"
+            ],
+            "6. Alignment For Deepseekmoe 16B": [
+              "6.1. Experimental Setup",
+              "6.2. Evaluations"
+            ],
+            "7. Deepseekmoe 145B Ongoing": [
+              "7.1. Experimental Setup",
+              "7.2. Evaluations"
+            ],
+            "8. Related Work": [],
+            "9. Conclusion": [],
+            "Appendices": [
+              "A. Overview Of Hyper-Parameters",
+              "B. Comparing Deepseekmoe With Larger Models",
+              "C. Training Benchmark Curves Of Deepseekmoe 16B"
+            ]
+          },
+          "summary_embedding": [
+            -0.012465757317841053,
+            -0.022808987647294998,
+            -0.018792515620589256,
+            -0.02308654971420765,
+            0.01797616109251976,
+            0.007200260180979967,
+            0.006490030791610479,
+            0.042613785713911057,
+            -0.014049487188458443,
+            0.03187054023146629,
+            -0.021486490964889526,
+            -0.023592690005898476,
+            0.009045224636793137,
+            -0.014441338367760181,
+            0.005865518003702164,
+            -0.007363531738519669,
+            -0.008743173442780972,
+            -0.005163452122360468,
+            0.018841497600078583,
+            -0.03100520372390747,
+            0.011110605672001839,
+            -0.046075135469436646,
+            -0.020017050206661224,
+            -0.021845687180757523,
+            0.016425084322690964,
+            0.037389107048511505,
+            0.022547753527760506,
+            -0.0028062239289283752,
+            -0.06844329088926315,
+            0.034678805619478226,
+            0.035168617963790894,
+            0.02052319049835205,
+            0.0005438972148112953,
+            -0.00031302153365686536,
+            -0.024833550676703453,
+            -0.028556134551763535,
+            0.03347059711813927,
+            -0.004379750229418278,
+            -0.01453930139541626,
+            0.015976088121533394,
+            0.0414055772125721,
+            0.012106560170650482,
+            -0.02754385396838188,
+            0.0014245413476601243,
+            0.015151568688452244,
+            0.0055961208418011665,
+            -0.01678428053855896,
+            -0.0022960014175623655,
+            -0.002424577483907342,
+            -0.021225256845355034,
+            -0.01400050614029169,
+            0.018825169652700424,
+            0.001296985661610961,
+            -0.008865626528859138,
+            -0.0542386956512928,
+            0.007330877240747213,
+            0.03840138763189316,
+            0.05623060464859009,
+            0.044148534536361694,
+            -0.020180322229862213,
+            0.03369917720556259,
+            -0.030025575309991837,
+            -0.02501314878463745,
+            0.011028970591723919,
+            0.003983817528933287,
+            -0.018531281501054764,
+            0.01784554310142994,
+            -0.005636938381940126,
+            -0.0006556359585374594,
+            0.01751900091767311,
+            -0.03941367194056511,
+            0.020751770585775375,
+            -0.01887415163218975,
+            0.031266435980796814,
+            0.02173139713704586,
+            0.0073798587545752525,
+            0.01872720755636692,
+            0.01678428053855896,
+            0.0036858476232737303,
+            -0.04107903689146042,
+            0.009118696674704552,
+            0.016523046419024467,
+            0.014196431264281273,
+            0.016269976273179054,
+            0.06553706526756287,
+            -0.05391215533018112,
+            -0.004485876299440861,
+            -0.05169166624546051,
+            0.0012561678886413574,
+            -0.004971608519554138,
+            -0.03670336678624153,
+            0.01048201136291027,
+            0.028425518423318863,
+            0.021372200921177864,
+            -0.01063711941242218,
+            0.024392718449234962,
+            -0.022743679583072662,
+            0.0032001156359910965,
+            -0.04062187671661377,
+            0.0019653772469609976,
+            0.004518530797213316,
+            0.016531210392713547,
+            0.0070778070949018,
+            0.004273623693734407,
+            -0.016702644526958466,
+            0.006045116577297449,
+            -0.01905374974012375,
+            0.03291547670960426,
+            -0.008465612307190895,
+            0.08653374016284943,
+            0.008751336485147476,
+            0.004738946910947561,
+            0.004281787667423487,
+            0.02398454025387764,
+            0.012131051160395145,
+            -0.00810233410447836,
+            -0.04718538001179695,
+            -0.015690363943576813,
+            0.011061624623835087,
+            -0.0383034273982048,
+            0.018335357308387756,
+            -0.008759500458836555,
+            0.01282495353370905,
+            -0.04107903689146042,
+            -0.06344719231128693,
+            0.01498013362288475,
+            0.001784758409485221,
+            -0.004065453074872494,
+            -0.0002903166168835014,
+            -0.018923133611679077,
+            0.003930754493921995,
+            -0.004200151655822992,
+            -0.010645282454788685,
+            -0.021502818912267685,
+            0.03159298002719879,
+            0.03657275065779686,
+            0.06527583301067352,
+            0.036801330745220184,
+            -0.0030164355412125587,
+            0.005775718949735165,
+            0.019690508022904396,
+            -0.020784424617886543,
+            0.003481758525595069,
+            -0.01214737817645073,
+            -0.021992631256580353,
+            -0.03820546343922615,
+            0.010971825569868088,
+            0.07379858940839767,
+            -0.029029620811343193,
+            0.013437220826745033,
+            -0.017616963014006615,
+            0.04205866530537605,
+            -0.004567512311041355,
+            0.026727497577667236,
+            -0.016669990494847298,
+            0.004059330560266972,
+            -0.025388672947883606,
+            0.01620466820895672,
+            -0.024964168667793274,
+            0.012571883387863636,
+            -0.000885746325366199,
+            -0.04222193360328674,
+            0.00011116473615402356,
+            0.026890767738223076,
+            -0.010971825569868088,
+            0.02261306345462799,
+            0.0004949158756062388,
+            -0.006302268709987402,
+            -0.05097327008843422,
+            0.038727931678295135,
+            -0.003912386484444141,
+            0.018645571544766426,
+            -0.0018959869630634785,
+            0.0163107942789793,
+            -0.0016408757073804736,
+            0.013216804713010788,
+            0.009543201886117458,
+            0.007114543113857508,
+            -0.033503253012895584,
+            0.02697240374982357,
+            -0.02504580281674862,
+            0.028049994260072708,
+            0.03435226157307625,
+            -0.0030184765346348286,
+            0.03255628049373627,
+            0.019249675795435905,
+            0.03840138763189316,
+            0.0327032245695591,
+            0.030335791409015656,
+            -0.00824927818030119,
+            0.013878053054213524,
+            0.008200296200811863,
+            0.03608293831348419,
+            0.013429056853055954,
+            0.019723162055015564,
+            -0.009730963967740536,
+            -0.014041324146091938,
+            0.048785436898469925,
+            0.007318632211536169,
+            -0.0179598331451416,
+            -0.005441013257950544,
+            -0.03510330989956856,
+            -0.002832755446434021,
+            -0.014425011351704597,
+            0.0323440283536911,
+            0.02158445306122303,
+            -0.01647406443953514,
+            0.035135965794324875,
+            0.024980494752526283,
+            -0.03751972317695618,
+            -0.005232842173427343,
+            -0.02026195637881756,
+            -0.023298801854252815,
+            -0.04215662553906441,
+            0.011624909937381744,
+            0.012833117507398129,
+            -0.013976015150547028,
+            -0.007469657808542252,
+            0.020457882434129715,
+            -0.023625344038009644,
+            -0.0018265966791659594,
+            -0.01356783788651228,
+            0.006547175347805023,
+            -0.020898714661598206,
+            -0.03667071461677551,
+            -0.01048201136291027,
+            0.025862159207463264,
+            0.012637191452085972,
+            0.04362606629729271,
+            -0.013518855907022953,
+            -0.005367541220039129,
+            0.033193036913871765,
+            0.012759645469486713,
+            0.010767736472189426,
+            0.03699725493788719,
+            -0.03027048148214817,
+            0.024621298536658287,
+            -0.020294612273573875,
+            -0.0032878739293664694,
+            0.01707816869020462,
+            -0.0030062312725931406,
+            0.01962519995868206,
+            -0.04535674303770065,
+            -0.024196794256567955,
+            0.04395261034369469,
+            -0.0007107399869710207,
+            0.016302630305290222,
+            -0.005959399044513702,
+            -0.012106560170650482,
+            0.0035899258218705654,
+            0.0027715289033949375,
+            -0.01438419334590435,
+            -0.008620720356702805,
+            0.007473739795386791,
+            -0.01962519995868206,
+            0.03660540655255318,
+            0.03794422745704651,
+            -0.004132802598178387,
+            -0.014131123200058937,
+            -0.03324201703071594,
+            0.010016689077019691,
+            -0.017878197133541107,
+            -0.023951886221766472,
+            0.04078514873981476,
+            -0.032115448266267776,
+            -0.00029312283731997013,
+            0.039478980004787445,
+            0.018808843567967415,
+            -0.014898497611284256,
+            0.00868602842092514,
+            -0.01709449477493763,
+            -0.026302991434931755,
+            -0.017159804701805115,
+            0.03356856107711792,
+            0.024800896644592285,
+            -0.010743245482444763,
+            0.010245268233120441,
+            0.005330805201083422,
+            0.02893165871500969,
+            0.024376392364501953,
+            0.00648594880476594,
+            -0.013812744058668613,
+            0.015029114671051502,
+            0.027641816064715385,
+            -0.0028021421749144793,
+            -7.66609373386018e-05,
+            -0.004010349046438932,
+            -0.007796200457960367,
+            -0.006453294772654772,
+            -0.013069860637187958,
+            -0.005089980084449053,
+            0.0009184005321003497,
+            -0.032295044511556625,
+            0.015967924147844315,
+            -0.005890008993446827,
+            0.010359558276832104,
+            -0.011535110883414745,
+            0.01288209855556488,
+            0.01722511276602745,
+            -0.013494365848600864,
+            -0.0716434046626091,
+            0.014612773433327675,
+            0.02458864450454712,
+            -0.006706364918500185,
+            -0.017943505197763443,
+            0.008481939323246479,
+            0.00016148542636074126,
+            0.038597315549850464,
+            -0.014180104248225689,
+            -0.031103165820240974,
+            0.02680913358926773,
+            -0.017437364906072617,
+            -0.015241367742419243,
+            -0.010661610402166843,
+            -0.00580837344750762,
+            0.04649963974952698,
+            0.01889047957956791,
+            0.011420820839703083,
+            -0.04042595252394676,
+            -0.007449248805642128,
+            0.015078096650540829,
+            0.014506647363305092,
+            -0.04705476388335228,
+            0.0052083516493439674,
+            0.015486273914575577,
+            0.019723162055015564,
+            0.024800896644592285,
+            0.04274440184235573,
+            -0.0021531390957534313,
+            0.01661284640431404,
+            -0.015053605660796165,
+            -0.05136512219905853,
+            -0.03340528905391693,
+            0.029943939298391342,
+            -0.017731253057718277,
+            -0.02906227484345436,
+            -0.01693122461438179,
+            -0.006151242647320032,
+            0.0017939424142241478,
+            0.007126788143068552,
+            -0.01916803978383541,
+            -0.03464614972472191,
+            -0.03970756009221077,
+            0.02277633361518383,
+            0.007983962073922157,
+            0.004269542172551155,
+            -0.042777057737112045,
+            0.0013459670590236783,
+            0.009184005670249462,
+            0.012073906138539314,
+            0.0009954441338777542,
+            -0.019984396174550056,
+            0.015494437888264656,
+            0.0032001156359910965,
+            -0.031070511788129807,
+            -0.035593122243881226,
+            -0.03595232218503952,
+            0.0023919232189655304,
+            0.003781769424676895,
+            0.06279410421848297,
+            0.012988224625587463,
+            -0.03840138763189316,
+            -0.04522612690925598,
+            0.029176564887166023,
+            -0.02052319049835205,
+            -0.016678154468536377,
+            0.0059430720284581184,
+            0.012898425571620464,
+            -0.009877908043563366,
+            0.013192313723266125,
+            0.023560035973787308,
+            0.041601505130529404,
+            -0.005987971555441618,
+            0.012914752587676048,
+            0.023037567734718323,
+            0.012955570593476295,
+            0.0362788625061512,
+            -0.00936360377818346,
+            0.0035409443080425262,
+            0.039936136454343796,
+            -0.012269831262528896,
+            0.03175625205039978,
+            -0.02189466916024685,
+            0.0020398697815835476,
+            0.02249877341091633,
+            0.027184655889868736,
+            -0.0006479826406575739,
+            0.02594379521906376,
+            -0.031543996185064316,
+            0.018368011340498924,
+            -0.014498483389616013,
+            -0.04888340085744858,
+            -0.01557607389986515,
+            0.013102514669299126,
+            0.030384771525859833,
+            0.02756018005311489,
+            -0.0041348435916006565,
+            -0.019723162055015564,
+            -0.01155143789947033,
+            -0.010596301406621933,
+            0.0018419033149257302,
+            0.03285016864538193,
+            0.010506502352654934,
+            0.010890189558267593,
+            -0.011935125105082989,
+            -0.004104230087250471,
+            0.03025415539741516,
+            -0.021192602813243866,
+            0.01650671847164631,
+            0.0041511706076562405,
+            -0.002029665280133486,
+            -0.04320156201720238,
+            -0.035266581922769547,
+            0.007845181971788406,
+            0.011747363954782486,
+            0.014874007552862167,
+            0.0368339866399765,
+            0.025094784796237946,
+            -0.01662100851535797,
+            0.030629679560661316,
+            -0.01905374974012375,
+            0.027984686195850372,
+            0.039054471999406815,
+            0.011624909937381744,
+            -0.018351683393120766,
+            0.012237177230417728,
+            0.002424577483907342,
+            -0.025127438828349113,
+            0.01680060662329197,
+            -0.032735876739025116,
+            0.01738838478922844,
+            -0.02354370802640915,
+            0.025143766775727272,
+            -0.02579685114324093,
+            -0.009290131740272045,
+            -0.004428731743246317,
+            -0.006130834110081196,
+            0.04888340085744858,
+            -0.026727497577667236,
+            -0.0142454132437706,
+            -0.0024694770108908415,
+            -0.0024531499948352575,
+            -0.008718682453036308,
+            0.010000362060964108,
+            -0.045781247317790985,
+            -0.01844964735209942,
+            -0.04016471654176712,
+            0.007355368230491877,
+            0.005412440747022629,
+            -0.030907239764928818,
+            -0.05694899708032608,
+            0.025421326979994774,
+            -0.06331657618284225,
+            -0.019886434078216553,
+            0.051626358181238174,
+            -0.018400665372610092,
+            -0.008212542161345482,
+            0.009641164913773537,
+            0.006375740747898817,
+            -0.029290854930877686,
+            0.0041348435916006565,
+            0.02504580281674862,
+            0.04418119043111801,
+            -0.0045797573402523994,
+            -0.02865409664809704,
+            0.0028674507047981024,
+            -0.027788760140538216,
+            -0.013461710885167122,
+            0.005261414684355259,
+            0.016449574381113052,
+            0.024539662525057793,
+            0.007628847379237413,
+            0.03585435822606087,
+            0.007294141221791506,
+            0.006318595726042986,
+            0.044279150664806366,
+            -0.025666235014796257,
+            -0.027380581945180893,
+            0.02519274689257145,
+            0.04045860469341278,
+            0.023151857778429985,
+            -0.018253721296787262,
+            -0.0142454132437706,
+            0.0053063142113387585,
+            0.0009296254720538855,
+            -0.022139575332403183,
+            0.002808264922350645,
+            -0.013902543112635612,
+            0.005143043119460344,
+            0.03892385587096214,
+            -0.001205145614221692,
+            0.014196431264281273,
+            0.007302304729819298,
+            0.006845145486295223,
+            0.004334850702434778,
+            -0.011731036007404327,
+            -0.023592690005898476,
+            0.00032220553839579225,
+            0.034972693771123886,
+            0.015404638834297657,
+            -0.030515389516949654,
+            -0.00995954405516386,
+            0.034711457788944244,
+            -0.00795130804181099,
+            -0.009118696674704552,
+            -0.009771781973540783,
+            0.003385836724191904,
+            0.02638462744653225,
+            -0.01606588624417782,
+            0.006649219896644354,
+            0.03716052696108818,
+            -0.019870106130838394,
+            0.005543057806789875,
+            -0.025405000895261765,
+            0.0018061877926811576,
+            0.02442537248134613,
+            -0.006216551177203655,
+            0.028098974376916885,
+            5.6315813708351925e-05,
+            0.03025415539741516,
+            -0.004677720367908478,
+            0.008432958275079727,
+            -0.020327266305685043,
+            0.03987082839012146,
+            -0.017633290961384773,
+            0.03402572125196457,
+            0.02067013457417488,
+            -0.01258004643023014,
+            0.04783846437931061,
+            -0.036801330745220184,
+            -0.020784424617886543,
+            0.00584919098764658,
+            0.012743317522108555,
+            0.0028919412288814783,
+            -0.0191517136991024,
+            -0.018400665372610092,
+            -0.010996315628290176,
+            0.02380494214594364,
+            -0.022547753527760506,
+            -0.016865916550159454,
+            -0.007441085297614336,
+            -0.01890680566430092,
+            -0.009673818945884705,
+            -0.007988044060766697,
+            -0.01709449477493763,
+            -0.016547536477446556,
+            0.044573038816452026,
+            -0.06334923207759857,
+            0.016996532678604126,
+            0.017159804701805115,
+            0.011477965861558914,
+            -0.034678805619478226,
+            -0.019510909914970398,
+            -0.017633290961384773,
+            -0.002347023691982031,
+            -0.030743969604372978,
+            0.001702102366834879,
+            0.01725776679813862,
+            0.01138000376522541,
+            -0.05022222548723221,
+            0.012335140258073807,
+            0.02772345207631588,
+            -0.012065742164850235,
+            0.0014949520118534565,
+            -0.023135529831051826,
+            -0.017176130786538124,
+            0.0031776658724993467,
+            0.022123249247670174,
+            0.005975726060569286,
+            -0.008188051171600819,
+            -0.016865916550159454,
+            -0.005241005681455135,
+            -0.006045116577297449,
+            -0.002359268954023719,
+            0.005094061605632305,
+            0.011780017986893654,
+            0.019690508022904396,
+            -0.0024633544962853193,
+            -0.004436895251274109,
+            -0.006396149750798941,
+            0.027364253997802734,
+            -0.001450052484869957,
+            -0.008416631259024143,
+            0.050907962024211884,
+            0.03556047007441521,
+            0.006094098091125488,
+            -0.01108611561357975,
+            -0.03990348428487778,
+            0.010661610402166843,
+            -0.022711025550961494,
+            0.0401320643723011,
+            0.01905374974012375,
+            -0.008298259228467941,
+            -0.028866350650787354,
+            0.010294250212609768,
+            -0.02290695160627365,
+            -0.009975871071219444,
+            0.009469729848206043,
+            0.00726556871086359,
+            -0.03263791650533676,
+            -0.013543346896767616,
+            0.03510330989956856,
+            0.011437147855758667,
+            -0.006841063499450684,
+            -0.036050282418727875,
+            0.008588065393269062,
+            0.01646590232849121,
+            0.011869817040860653,
+            -0.019233347848057747,
+            0.004449140280485153,
+            -0.012041252106428146,
+            0.017470018938183784,
+            0.008849299512803555,
+            -0.017796561121940613,
+            0.018237393349409103,
+            0.004710374400019646,
+            -0.008490103296935558,
+            -0.003912386484444141,
+            -0.0163107942789793,
+            -0.02006603218615055,
+            0.03840138763189316,
+            0.024490682408213615,
+            0.026335645467042923,
+            0.005457340274006128,
+            0.02008235827088356,
+            0.03227871656417847,
+            0.011282040737569332,
+            -0.03134807199239731,
+            0.027935704216361046,
+            0.016996532678604126,
+            -0.04604247957468033,
+            0.021551799029111862,
+            -0.02218855731189251,
+            -0.028670424595475197,
+            0.009649328887462616,
+            -0.0017367975087836385,
+            0.011192241683602333,
+            0.0034837995190173388,
+            -0.003642988856881857,
+            -0.03307874873280525,
+            -0.006236960180103779,
+            0.015771998092532158,
+            0.015502601861953735,
+            -0.0048695639707148075,
+            -0.003296037670224905,
+            -0.011886144056916237,
+            0.00487364549189806,
+            0.009796272963285446,
+            0.039054471999406815,
+            0.001381682581268251,
+            -0.004751192405819893,
+            -0.0018765984568744898,
+            0.031854212284088135,
+            0.0271193478256464,
+            -0.017274094745516777,
+            -0.026286665350198746,
+            0.00580837344750762,
+            -0.014253576286137104,
+            0.026156047359108925,
+            -0.00487364549189806,
+            0.011877980083227158,
+            0.008718682453036308,
+            -0.0007908449042588472,
+            -0.001373519073240459,
+            0.025584599003195763,
+            -0.02710302174091339,
+            -0.021764053031802177,
+            0.031233781948685646,
+            -0.025731543079018593,
+            0.028735732659697533,
+            0.013061696663498878,
+            -0.01557607389986515,
+            -0.015225040726363659,
+            -0.010302413254976273,
+            0.009943217039108276,
+            -0.012277995236217976,
+            0.023853924125432968,
+            0.003608293831348419,
+            0.006392067763954401,
+            0.0003584313380997628,
+            0.02770712412893772,
+            0.024833550676703453,
+            0.007820690982043743,
+            -0.018074123188853264,
+            -0.0016816934803500772,
+            -0.02320083975791931,
+            -0.01680060662329197,
+            0.023772288113832474,
+            -0.018090449273586273,
+            -0.03268689662218094,
+            -0.028556134551763535,
+            0.012114724144339561,
+            0.007392104249447584,
+            0.010816717520356178,
+            0.03461349755525589,
+            -0.030384771525859833,
+            -0.0344502255320549,
+            0.01766594499349594,
+            0.026270337402820587,
+            -0.005551221314817667,
+            -0.021535472944378853,
+            0.03085825778543949,
+            -0.005828781984746456,
+            0.017600636929273605,
+            0.0404912605881691,
+            -0.0016225076979026198,
+            0.010228941217064857,
+            -0.042026009410619736,
+            -0.014057651162147522,
+            -0.0066614653915166855,
+            0.006261450704187155,
+            0.034417569637298584,
+            0.01048201136291027,
+            0.01535565685480833,
+            0.03448288142681122,
+            -0.021192602813243866,
+            -0.0002903166168835014,
+            -0.0022449791431427,
+            -0.02818061038851738,
+            0.03347059711813927,
+            0.025715215131640434,
+            0.011918798089027405,
+            0.019706835970282555,
+            -0.03794422745704651,
+            -0.013037205673754215,
+            0.027070365846157074,
+            0.015053605660796165,
+            0.01273515447974205,
+            0.016694480553269386,
+            0.011249386705458164,
+            0.00868602842092514,
+            -0.021633435040712357,
+            -0.05169166624546051,
+            0.01214737817645073,
+            0.01993541419506073,
+            -0.016449574381113052,
+            0.01057181041687727,
+            0.024980494752526283,
+            0.007620683871209621,
+            -0.007045152597129345,
+            0.012808626517653465,
+            -0.02682545967400074,
+            0.002259265398606658,
+            0.0019010890973731875,
+            -0.014482156373560429,
+            0.038270771503448486,
+            -0.016367938369512558,
+            0.011363675817847252,
+            -0.018204739317297935,
+            0.023870250210165977,
+            0.00011531029304023832,
+            0.0007964573451317847,
+            0.014963806606829166,
+            -0.011347348801791668,
+            -0.04235255345702171,
+            0.0021613026037812233,
+            0.03342161700129509,
+            -0.015200549736618996,
+            0.026449935510754585,
+            -0.0020674217958003283,
+            0.032458316534757614,
+            0.042777057737112045,
+            0.017731253057718277,
+            0.0003497575526125729,
+            0.025682561099529266,
+            0.0036144163459539413,
+            0.0073880222626030445,
+            0.012400448322296143,
+            0.009322785772383213,
+            0.007245160173624754,
+            0.030809277668595314,
+            -0.010261595249176025,
+            0.02173139713704586,
+            -0.04760988429188728,
+            -0.014629100449383259,
+            0.012024925090372562,
+            -0.0013316808035597205,
+            -0.012449429370462894,
+            -0.033029764890670776,
+            0.012563719414174557,
+            -0.03503800183534622,
+            0.022074267268180847,
+            -0.0004959363141097128,
+            0.04333217814564705,
+            -0.02187834307551384,
+            -0.005518566817045212,
+            0.016694480553269386,
+            0.0016510800924152136,
+            0.023282473906874657,
+            -0.0012153501156717539,
+            -0.016416920349001884,
+            -0.0067798369564116,
+            -0.009257477708160877,
+            0.004138925112783909,
+            0.04375668615102768,
+            -0.003722583642229438,
+            -0.0001682458823779598,
+            0.01409030519425869,
+            -0.011959616094827652,
+            -0.03250729665160179,
+            0.045193471014499664,
+            -0.0030837850645184517,
+            0.02484987862408161,
+            0.04979771748185158,
+            -0.06410028040409088,
+            -0.019543563947081566,
+            0.013061696663498878,
+            -0.010441194288432598,
+            0.011363675817847252,
+            0.005547139327973127,
+            0.016114868223667145,
+            0.017894525080919266,
+            -0.005485912784934044,
+            0.01619650423526764,
+            0.005106307100504637,
+            0.028098974376916885,
+            -0.02336410991847515,
+            -0.03296445682644844,
+            -0.02636830136179924,
+            -0.017747581005096436,
+            0.0014480114914476871,
+            -0.015796490013599396,
+            -3.563585551091819e-06,
+            -0.0022204883862286806,
+            -0.04388730227947235,
+            -0.025274382904171944,
+            0.040687184780836105,
+            0.006706364918500185,
+            0.02458864450454712,
+            0.0077268099412322044,
+            0.03418898954987526,
+            0.005885927006602287,
+            -0.009037061594426632,
+            0.030058229342103004,
+            -0.020049704238772392,
+            0.03833607956767082,
+            0.022106921300292015,
+            -0.002832755446434021,
+            0.005865518003702164,
+            0.031086837872862816,
+            -0.033797141164541245,
+            0.008086007088422775,
+            -0.034384917467832565,
+            0.0046695563942193985,
+            -0.008620720356702805,
+            -0.015094423666596413,
+            -0.009526874870061874,
+            -0.02504580281674862,
+            0.011216731742024422,
+            -0.021535472944378853,
+            0.01590261608362198,
+            0.010416703298687935,
+            -0.002916431985795498,
+            0.03448288142681122,
+            -0.014204595237970352,
+            -0.0027796924114227295,
+            0.0024531499948352575,
+            -0.027070365846157074,
+            -0.006343086715787649,
+            -0.003377673216164112,
+            -0.03689929470419884,
+            -0.004502203781157732,
+            0.010988152585923672,
+            -0.00024184548237826675,
+            0.016898570582270622,
+            0.02787039615213871,
+            0.0005092020728625357,
+            -0.022302847355604172,
+            0.002340900944545865,
+            -0.04199335351586342,
+            0.0011877981014549732,
+            0.01483318954706192,
+            -0.017600636929273605,
+            -0.008612556383013725,
+            0.033046092838048935,
+            0.023298801854252815,
+            -0.009396257810294628,
+            -0.035005345940589905,
+            0.017796561121940613,
+            -0.011853490024805069,
+            0.027511198073625565,
+            -0.0002630197268445045,
+            0.019298657774925232,
+            -0.0076941559091210365,
+            0.04493223875761032,
+            0.001369437319226563,
+            0.008645210415124893,
+            -0.021959977224469185,
+            0.007020662073045969,
+            -0.020278284326195717,
+            -0.002406209474429488,
+            -0.006171651650220156,
+            -0.012751481495797634,
+            -0.010669773444533348,
+            0.0032735876739025116,
+            0.007730891928076744,
+            -0.0176822729408741,
+            0.0055961208418011665,
+            -0.015959760174155235,
+            -0.02741323597729206,
+            0.007543129846453667,
+            -0.05965929850935936,
+            -0.0179598331451416,
+            -0.03373183310031891,
+            -0.009477893821895123,
+            0.02008235827088356,
+            0.03510330989956856,
+            -0.012759645469486713,
+            0.00497977202758193,
+            -0.014898497611284256,
+            0.004477712791413069,
+            0.03490738570690155,
+            0.044442422688007355,
+            -0.014759717509150505,
+            0.003381754970178008,
+            -0.010955498553812504,
+            0.033029764890670776,
+            -0.03347059711813927,
+            0.0030062312725931406,
+            0.031086837872862816,
+            -0.005057325586676598,
+            0.007959471084177494,
+            0.011624909937381744,
+            0.020474210381507874,
+            -0.019396619871258736,
+            0.00877582747489214,
+            0.002100075827911496,
+            0.019233347848057747,
+            0.02171507105231285,
+            0.020343592390418053,
+            -0.03539719805121422,
+            0.028409190475940704,
+            0.018384337425231934,
+            0.02444170042872429,
+            0.044442422688007355,
+            -0.03974021226167679,
+            -0.004147088620811701,
+            -0.0032225653994828463,
+            -0.02997659333050251,
+            0.02008235827088356,
+            -0.007530884351581335,
+            -0.01841699331998825,
+            -0.007069643586874008,
+            0.03327467292547226,
+            0.014571955427527428,
+            -0.05051611363887787,
+            0.01980479806661606,
+            0.01740471087396145,
+            0.02055584453046322,
+            0.024474354460835457,
+            -0.01123305968940258,
+            -0.0019072118448093534,
+            0.008996243588626385,
+            0.006804327480494976,
+            0.019266001880168915,
+            0.015257694758474827,
+            0.002012317767366767,
+            -0.0022082431241869926,
+            -0.011837163008749485,
+            0.005224678665399551,
+            -0.009077879600226879,
+            0.010808554477989674,
+            -0.02458864450454712,
+            -0.009698309935629368,
+            0.021388528868556023,
+            0.0031164393294602633,
+            -0.013665799982845783,
+            -0.03729114308953285,
+            -0.045944519340991974,
+            0.0033837957307696342,
+            0.004543021321296692,
+            -0.05365091934800148,
+            0.005322641227394342,
+            -0.026547899469733238,
+            -0.002108239568769932,
+            -0.005502239800989628,
+            0.0019214981002733111,
+            0.005636938381940126,
+            0.0167679525911808,
+            -0.013494365848600864,
+            0.02248244546353817,
+            -0.015502601861953735,
+            0.03165828809142113,
+            0.0055961208418011665,
+            0.007375776767730713,
+            -0.0027796924114227295,
+            0.05590406060218811,
+            -0.003712379140779376,
+            -0.03699725493788719,
+            0.023282473906874657,
+            -0.01311067771166563,
+            -0.0001747512142173946,
+            0.030090883374214172,
+            0.00884113647043705,
+            -0.015257694758474827,
+            0.013437220826745033,
+            -0.030613351613283157,
+            -0.03249097242951393,
+            0.010996315628290176,
+            -0.013755599036812782,
+            -0.0015776080545037985,
+            -0.04568328335881233,
+            -0.009339112788438797,
+            0.004269542172551155,
+            -0.018335357308387756,
+            0.010775899514555931,
+            0.01631895825266838,
+            0.013992343097925186,
+            -0.024245774373412132,
+            0.009053388610482216,
+            0.006865554489195347,
+            0.013453547842800617,
+            0.0007331897504627705,
+            0.014049487188458443,
+            -0.008677864447236061,
+            -0.011306531727313995,
+            -0.018319029361009598,
+            0.0032450153958052397,
+            -0.026792805641889572,
+            0.005155288614332676,
+            -0.003155216109007597,
+            -0.03771565109491348,
+            -0.06521052122116089,
+            -0.0344502255320549,
+            0.016294466331601143,
+            0.03396041318774223,
+            -0.009045224636793137,
+            0.0035144127905368805,
+            -0.00023648813657928258,
+            -0.01093917153775692,
+            0.024523336440324783,
+            -0.005930826533585787,
+            -0.0004829256213270128,
+            0.0035389035474509,
+            0.02050686441361904,
+            -0.009722800925374031,
+            0.005845109466463327,
+            -0.051952898502349854,
+            -0.01586179807782173,
+            0.01812310516834259,
+            0.006306350696831942,
+            0.014629100449383259,
+            -0.009690145961940289,
+            0.01693122461438179,
+            -0.030074557289481163,
+            -0.028147956356406212,
+            0.04130761697888374,
+            0.008277850225567818,
+            0.011486129835247993,
+            -0.003055212553590536,
+            -0.01680060662329197,
+            0.02504580281674862,
+            -0.0020633398089557886,
+            -0.006461458280682564,
+            -0.00817988719791174,
+            0.015820980072021484,
+            0.044279150664806366,
+            0.003255219664424658,
+            0.0009821783751249313,
+            0.017045514658093452,
+            -0.002663361607119441,
+            -0.011575928889214993,
+            0.007677828427404165,
+            0.013592327944934368,
+            -0.016384266316890717,
+            0.010375885292887688,
+            -0.025862159207463264,
+            0.0332583449780941,
+            0.03131541982293129,
+            0.011575928889214993,
+            -0.015045441687107086,
+            0.006130834110081196,
+            0.005330805201083422,
+            0.008086007088422775,
+            -0.001369437319226563,
+            -0.0017133272485807538,
+            0.020719116553664207,
+            -0.03276853263378143,
+            -0.01693122461438179,
+            -0.005347132217139006,
+            0.003932795487344265,
+            0.005277741700410843,
+            0.03396041318774223,
+            -0.0033266509417444468,
+            0.017927179113030434,
+            -0.011657563969492912,
+            0.010579974390566349,
+            0.021764053031802177,
+            0.037095218896865845,
+            -0.024996822699904442,
+            -0.029862305149435997,
+            0.012914752587676048,
+            -0.001296985661610961,
+            0.014065815135836601,
+            0.0010173837654292583,
+            0.014139287173748016,
+            0.0029409227427095175,
+            0.060475654900074005,
+            -0.017763907089829445,
+            0.004628738854080439,
+            0.0004035860183648765,
+            -0.01858026348054409,
+            -0.0013531101867556572,
+            0.004261378664523363,
+            -0.010979988612234592,
+            0.03608293831348419,
+            0.002344982698559761,
+            0.039185091853141785,
+            0.013918871060013771,
+            0.009290131740272045,
+            0.0015143405180424452,
+            -0.013592327944934368,
+            0.008355404250323772,
+            0.0015592400450259447,
+            -0.001132694073021412,
+            0.009461566805839539,
+            0.019298657774925232,
+            0.009714636951684952,
+            0.025274382904171944,
+            0.009788108989596367,
+            0.012857607565820217,
+            -0.029470453038811684,
+            0.03510330989956856,
+            -0.0006224715034477413,
+            0.012996388599276543,
+            -0.02040890045464039,
+            -0.003136848099529743,
+            -0.0008240094175562263,
+            0.015649545937776566,
+            -0.015616890974342823,
+            -0.001201063860207796,
+            -0.01297189760953188,
+            0.040099408477544785,
+            -0.018988441675901413,
+            0.013976015150547028,
+            0.009673818945884705,
+            -0.02865409664809704,
+            -0.007596192881464958,
+            0.01995174214243889,
+            0.014294394291937351,
+            -0.0053144777193665504,
+            -0.020490536466240883,
+            0.005832863971590996,
+            0.015380147844552994,
+            -0.028572462499141693,
+            1.3584674888988957e-05,
+            0.014751553535461426,
+            -0.0164903923869133,
+            -0.00025409081717953086,
+            -0.004702210891991854,
+            0.003975654020905495,
+            -0.009265640750527382,
+            0.007028825581073761,
+            0.016057724133133888,
+            -0.024507008492946625,
+            -0.020049704238772392,
+            0.005722655914723873,
+            0.006571666337549686,
+            0.01444950234144926,
+            0.007739055436104536,
+            0.004355259705334902,
+            -0.020033378154039383,
+            -0.01799248717725277,
+            0.0032674651592969894,
+            -0.01649855636060238,
+            -0.0048614004626870155,
+            0.0020531355403363705,
+            -0.019363965839147568,
+            0.01875986158847809,
+            -0.010743245482444763,
+            0.00017538899555802345,
+            -0.003226647386327386,
+            -0.020800752565264702,
+            0.006204306147992611,
+            -0.004865481983870268,
+            0.019870106130838394,
+            0.003555230563506484,
+            0.02785406820476055,
+            0.02310287579894066,
+            -0.004428731743246317,
+            0.017600636929273605,
+            -0.016155686229467392,
+            0.007983962073922157,
+            -0.031135819852352142,
+            -0.001782717532478273,
+            -0.010408539324998856,
+            0.0036103345919400454,
+            -0.015943434089422226,
+            -0.009126860648393631,
+            -0.0048777274787425995,
+            -0.0006678812787868083,
+            -0.004820582456886768,
+            0.0004275664687156677,
+            0.0027817331720143557,
+            0.02231917530298233,
+            -0.013208640739321709,
+            -0.02803366631269455,
+            -0.017437364906072617,
+            0.006045116577297449,
+            0.019396619871258736,
+            -0.026025431230664253,
+            -0.009184005670249462,
+            -0.005012426059693098,
+            -0.004326687194406986,
+            -0.021404854953289032,
+            -0.015167895704507828,
+            -0.005632856860756874,
+            -0.008045189082622528,
+            -0.010522829368710518,
+            0.0046532293781638145,
+            -0.0035715578123927116,
+            -0.011739199981093407,
+            0.021666089072823524,
+            0.018057795241475105,
+            -0.019837452098727226,
+            -0.019837452098727226,
+            0.013641309924423695,
+            -0.006159406621009111,
+            0.0018878233386203647,
+            0.02008235827088356,
+            0.00029286774224601686,
+            0.005824700463563204,
+            0.020882388576865196,
+            -0.0035225762985646725,
+            -0.01797616109251976,
+            0.003577680326998234,
+            -0.0018888438353314996,
+            -0.012122887186706066,
+            0.003212361130863428,
+            0.02773977816104889,
+            0.0008076822850853205,
+            0.023429417982697487,
+            -0.03056436963379383,
+            -0.011412657797336578,
+            0.006061443593353033,
+            0.02369065210223198,
+            0.00211844383738935,
+            -0.0021694661118090153,
+            0.020131340250372887,
+            0.03941367194056511,
+            0.005775718949735165,
+            0.0015755671774968505,
+            -0.04003410041332245,
+            0.01027792226523161,
+            0.009208496659994125,
+            0.024621298536658287,
+            0.008339077234268188,
+            0.0014653591206297278,
+            -0.012906589545309544,
+            -0.009755454957485199,
+            -0.0017939424142241478,
+            0.014702572487294674,
+            -0.023870250210165977,
+            0.0024755997583270073,
+            -0.03157665207982063,
+            0.005094061605632305,
+            0.0021959978621453047,
+            -0.009559528902173042,
+            0.0335359051823616,
+            0.00980443600565195,
+            -0.02065380848944187,
+            0.015151568688452244,
+            0.005767555441707373,
+            0.007330877240747213,
+            0.022449791431427002,
+            -0.008228869177401066,
+            0.030450081452727318,
+            -0.005049162078648806,
+            0.01213921420276165,
+            0.02742956392467022,
+            0.032588932663202286,
+            -0.013102514669299126,
+            -0.003959327004849911,
+            0.0006097159348428249,
+            -0.0002561317232903093,
+            0.0007995187188498676,
+            -0.008204378187656403,
+            0.0037389106582850218,
+            -0.014800535514950752,
+            0.012710663489997387,
+            -0.019543563947081566,
+            -0.0034286954905837774,
+            0.0038532004691660404,
+            0.01991908811032772,
+            0.0016265894519165158,
+            -0.033633869141340256,
+            0.02055584453046322,
+            -0.015420965850353241,
+            0.0135596739128232,
+            0.001290863030590117,
+            -0.011110605672001839,
+            -0.00029210239881649613,
+            0.020588500425219536,
+            0.010947334580123425,
+            -0.007845181971788406,
+            -0.0135596739128232,
+            -0.028376536443829536,
+            -0.021535472944378853,
+            0.00823703221976757,
+            0.009445239789783955,
+            -0.0028837777208536863,
+            0.0021306893322616816,
+            -0.011869817040860653,
+            0.0006520643946714699,
+            -0.0008122742874547839,
+            0.01740471087396145,
+            -0.0335359051823616,
+            0.021943651139736176,
+            -0.019527235999703407,
+            -0.019543563947081566,
+            0.017045514658093452,
+            0.008049271069467068,
+            0.009445239789783955,
+            0.0033552234526723623,
+            -0.01995174214243889,
+            -0.012326976284384727,
+            0.0338951013982296,
+            -0.01138000376522541,
+            0.006298186723142862,
+            0.004408322740346193,
+            0.012914752587676048,
+            -0.008914608508348465,
+            -0.011926962062716484,
+            0.01784554310142994,
+            -0.009861581027507782,
+            0.013951525092124939,
+            0.0010010566329583526,
+            -0.0254703089594841,
+            0.02739690989255905,
+            -0.019233347848057747,
+            0.03487472981214523,
+            -0.021617108955979347,
+            0.00711862463504076,
+            0.0066614653915166855,
+            0.008792154490947723,
+            -0.006690037902444601,
+            -0.013224967755377293,
+            0.0012786176521331072,
+            0.03807484731078148,
+            -0.02623768337070942,
+            0.024180466309189796,
+            0.022400809451937675,
+            -0.00040766780148260295,
+            0.02065380848944187,
+            -0.008367649279534817,
+            -0.026711169630289078,
+            0.0179598331451416,
+            -0.012596373446285725,
+            -0.026123393326997757,
+            0.013257622718811035,
+            -0.022678371518850327,
+            -0.020278284326195717,
+            -0.026302991434931755,
+            0.004947117529809475,
+            -0.0025694805663079023,
+            -0.005151206627488136,
+            0.009004407562315464,
+            0.02561725303530693,
+            0.020915042608976364,
+            0.019576217979192734,
+            0.02111096680164337,
+            0.01161674689501524,
+            0.022449791431427002,
+            -0.007559456862509251,
+            0.04401791840791702,
+            -0.004947117529809475,
+            0.004632820375263691,
+            -0.0015980169409886003,
+            -0.015657709911465645,
+            0.009771781973540783,
+            0.0119106350466609,
+            0.0029572497587651014,
+            -0.029176564887166023,
+            -0.009061552584171295,
+            -0.03660540655255318,
+            0.005053244065493345,
+            0.0020664012990891933,
+            0.0176822729408741,
+            0.032115448266267776,
+            -0.005865518003702164,
+            0.004592002835124731,
+            -0.005102225113660097,
+            -0.013927034102380276,
+            -0.009420748800039291,
+            -0.02739690989255905,
+            -0.025437654927372932,
+            -0.007653337903320789,
+            -0.013739272020757198,
+            -0.030368445441126823,
+            -0.011722872965037823,
+            0.01551892887800932,
+            -0.007645174395292997,
+            0.011918798089027405,
+            -0.016702644526958466,
+            0.01385356206446886,
+            -0.004110352601855993,
+            -0.004763437435030937,
+            -0.004530776292085648,
+            -0.017600636929273605,
+            -0.006420640274882317,
+            0.014367866329848766,
+            -0.009477893821895123,
+            -0.01237595733255148,
+            -0.02215590327978134,
+            -0.0015459742862731218,
+            -0.01574750803411007,
+            0.003489922033622861,
+            -0.014596446417272091,
+            -0.006771673448383808,
+            -0.003908304497599602,
+            -0.018514955416321754,
+            0.01766594499349594,
+            0.014816862531006336,
+            -0.019380291923880577,
+            -0.0090697156265378,
+            0.02638462744653225,
+            0.042777057737112045,
+            -0.03218075633049011,
+            -0.008200296200811863,
+            0.0020378287881612778,
+            0.018090449273586273,
+            0.010171796195209026,
+            0.016702644526958466,
+            -0.024653952568769455,
+            0.0022286521270871162,
+            -0.021649762988090515,
+            -0.001220452249981463,
+            -0.023739634081721306,
+            0.010775899514555931,
+            0.025715215131640434,
+            -0.010971825569868088,
+            0.025976449251174927,
+            0.01634344831109047,
+            -0.01812310516834259,
+            0.012016761116683483,
+            0.007183933164924383,
+            0.014645427465438843,
+            -0.019755816087126732,
+            0.0167679525911808,
+            -0.015796490013599396,
+            0.031233781948685646,
+            0.00892277155071497,
+            -0.04829562455415726,
+            -0.03503800183534622,
+            -0.013388238847255707,
+            0.0013388239312916994,
+            -0.010702427476644516,
+            0.015037278644740582,
+            0.015967924147844315,
+            0.013959688134491444,
+            -0.019380291923880577,
+            -0.015820980072021484,
+            0.011984107084572315,
+            0.010065670125186443,
+            0.027462217956781387,
+            -0.006649219896644354,
+            -0.004212397150695324,
+            -0.00960034690797329,
+            -0.02249877341091633,
+            -0.016098542138934135,
+            0.0209803506731987,
+            -0.018041469156742096,
+            0.015935270115733147,
+            -0.026139721274375916,
+            -0.012449429370462894,
+            0.017453692853450775,
+            -0.05211617052555084,
+            -0.01453113742172718,
+            0.0007612520130351186,
+            0.03012353740632534,
+            0.004751192405819893,
+            0.0035184945445507765,
+            0.01841699331998825,
+            -0.01993541419506073,
+            -0.0039776950143277645,
+            -0.02099667675793171,
+            0.006832899991422892,
+            -0.011853490024805069,
+            -0.013984179124236107,
+            0.014482156373560429,
+            -0.013845399022102356,
+            -0.01601690612733364,
+            0.0034368589986115694,
+            0.007183933164924383,
+            -0.021094640716910362,
+            0.011951452121138573,
+            0.003945040516555309,
+            0.012171869166195393,
+            0.012171869166195393,
+            0.01199227012693882,
+            0.021551799029111862,
+            0.02295593172311783,
+            -0.007334959227591753,
+            -0.004767519421875477,
+            -0.01004117913544178,
+            -0.0015398516552522779,
+            0.008898280560970306,
+            0.0033797139767557383,
+            -0.002989904023706913,
+            0.007837017998099327,
+            0.010033016093075275,
+            -0.01498013362288475,
+            0.033193036913871765,
+            -0.0035878848284482956,
+            -0.04134026914834976,
+            -0.003238892648369074,
+            -0.03800953924655914,
+            0.020033378154039383,
+            -0.023298801854252815,
+            -0.016996532678604126,
+            0.0032776694279164076,
+            -0.00629410520195961,
+            -0.019902760162949562,
+            0.013608654960989952,
+            0.011404493823647499,
+            0.00258580781519413,
+            0.02021297626197338,
+            0.012269831262528896,
+            -0.005134879611432552,
+            0.026172375306487083,
+            -0.014718899503350258,
+            -0.008751336485147476,
+            0.015567909926176071,
+            0.006853308994323015,
+            0.0022347746416926384,
+            0.0011031010653823614,
+            -0.018825169652700424,
+            0.005098143592476845,
+            0.0164903923869133,
+            0.007216587662696838,
+            -0.000566347036510706,
+            0.014049487188458443,
+            0.012514738366007805,
+            -0.015135240741074085,
+            -0.0002213090192526579,
+            -0.026564225554466248,
+            0.0032817514147609472,
+            0.004567512311041355,
+            0.007506393827497959,
+            -0.006771673448383808,
+            0.006281859707087278,
+            0.00658799335360527,
+            0.014759717509150505,
+            -0.010155469179153442,
+            -0.016980206593871117,
+            -0.002510294783860445,
+            0.004285869188606739,
+            0.016694480553269386,
+            0.03157665207982063,
+            -0.010310577228665352,
+            -0.005734901409596205,
+            -0.0386626236140728,
+            0.02070278860628605,
+            0.03332365304231644,
+            0.011175914667546749,
+            -0.028213264420628548,
+            0.012302485294640064,
+            0.0006515541463159025,
+            0.0019276207312941551,
+            -0.026302991434931755,
+            0.02787039615213871,
+            0.00974729098379612,
+            0.0066614653915166855,
+            0.017861871048808098,
+            0.0022306928876787424,
+            0.0029266364872455597,
+            -0.016302630305290222,
+            0.01018812321126461,
+            0.009086042642593384,
+            -0.017306748777627945,
+            0.0009745250572450459,
+            0.02324981987476349,
+            -0.01423724927008152,
+            -0.003208279376849532,
+            0.0018133309204131365,
+            0.02651524543762207,
+            -0.013820908032357693,
+            -0.014294394291937351,
+            0.0005714492290280759,
+            0.01751900091767311,
+            -0.0007979880319908261,
+            0.012098397128283978,
+            0.04898136481642723,
+            -0.012424939312040806,
+            0.019657853990793228,
+            0.005461421795189381,
+            -0.015290348790585995,
+            -0.032605260610580444,
+            -0.004922627005726099,
+            -0.01830270327627659,
+            0.011200404725968838,
+            0.014343376271426678,
+            -0.015763835981488228,
+            0.0007525782566517591,
+            -0.03144603595137596,
+            0.007743136957287788,
+            0.014735226519405842,
+            0.018776189535856247,
+            -0.002440904499962926,
+            0.006134915631264448,
+            0.027674470096826553,
+            0.02458864450454712,
+            -0.007392104249447584,
+            0.013804581016302109,
+            0.02320083975791931,
+            -0.025062130764126778,
+            -0.003069498809054494,
+            0.006820654962211847,
+            0.004104230087250471,
+            0.0053063142113387585,
+            -0.016963878646492958,
+            0.03915243595838547,
+            -0.005922663025557995,
+            -0.01506176870316267,
+            0.0030266400426626205,
+            -0.009028897620737553,
+            0.0063390047289431095,
+            -0.012563719414174557,
+            0.012637191452085972,
+            -0.008114579133689404,
+            0.00022883480414748192,
+            -0.0059430720284581184,
+            -0.011110605672001839,
+            -0.012653518468141556,
+            -0.0007949267164804041,
+            -0.016531210392713547,
+            0.007432921789586544,
+            0.00981259997934103,
+            0.005045080557465553,
+            0.01025343220680952,
+            -0.0036797248758375645,
+            -0.003136848099529743,
+            -0.016751626506447792,
+            -0.009984034113585949,
+            0.0073716952465474606,
+            0.01872720755636692,
+            -0.028588788583874702,
+            -0.018547609448432922,
+            0.009290131740272045,
+            0.0022898786701261997,
+            0.018776189535856247,
+            0.0158944521099329,
+            -0.0002595757250674069,
+            0.012318813242018223,
+            -0.01964152604341507,
+            -0.009184005670249462,
+            0.009690145961940289,
+            -0.003569516818970442,
+            0.01048201136291027,
+            -0.014759717509150505,
+            0.010033016093075275,
+            -0.011159587651491165,
+            -0.011053460650146008,
+            0.01385356206446886,
+            0.01738838478922844,
+            0.008873790502548218,
+            -0.001294944784604013,
+            3.383413059054874e-05,
+            -0.010506502352654934,
+            -0.024033522233366966,
+            0.004771600943058729,
+            0.027021385729312897,
+            -0.013273949734866619,
+            0.003883813973516226,
+            0.008702355436980724,
+            -0.017502672970294952,
+            -0.019821124151349068,
+            -0.012653518468141556,
+            -0.022106921300292015,
+            0.020572172477841377,
+            0.023413091897964478,
+            -0.02026195637881756,
+            0.00877582747489214,
+            -0.004620575346052647,
+            0.014049487188458443,
+            -0.01400050614029169,
+            0.010612628422677517,
+            0.0317889042198658,
+            -0.020000724121928215,
+            -0.0028654097113758326,
+            0.0036633978597819805,
+            -0.015347493812441826,
+            0.018629245460033417,
+            0.03268689662218094,
+            -0.00176945177372545,
+            0.014041324146091938,
+            0.0008138049161061645,
+            -0.009118696674704552,
+            0.01905374974012375,
+            0.011094278655946255,
+            0.005347132217139006,
+            -0.0042491331696510315,
+            -0.012212686240673065,
+            -0.03010721132159233,
+            0.016702644526958466,
+            0.0129637336358428,
+            -0.0037674831692129374,
+            -0.0011551438365131617,
+            -0.006343086715787649,
+            -0.015404638834297657,
+            0.0035123717971146107,
+            -0.013078023679554462,
+            -0.00803294312208891,
+            -0.008759500458836555,
+            0.004314441699534655,
+            -0.0034960447810590267,
+            -0.009053388610482216,
+            0.007236996199935675,
+            0.0009510547970421612,
+            -0.0010602424154058099,
+            -0.01902109570801258,
+            -0.025078458711504936,
+            -0.00995954405516386,
+            -0.016849588602781296,
+            0.0038287099450826645,
+            0.0005234883283264935,
+            -0.027184655889868736,
+            0.0053961132653057575,
+            -0.008139070123434067,
+            -0.008134988136589527,
+            0.007126788143068552,
+            0.006881881505250931,
+            -0.007437003776431084,
+            -0.010816717520356178,
+            -0.00786150898784399,
+            -0.006865554489195347,
+            0.020033378154039383,
+            0.0034348180051892996,
+            0.012082069180905819,
+            -0.010498338378965855,
+            -0.018025141209363937,
+            -0.016963878646492958,
+            0.016718972474336624,
+            -0.008971752598881721,
+            -0.02083340659737587,
+            -0.01444950234144926,
+            -0.008424794301390648,
+            -0.007665583398193121,
+            -0.01527402177453041,
+            -0.021159948781132698,
+            -0.0032103201374411583,
+            -0.011347348801791668,
+            -0.0067431009374558926,
+            -0.0028960229828953743,
+            0.0013571919407695532,
+            0.00951871182769537,
+            0.00546958576887846,
+            0.00438791373744607,
+            0.021666089072823524,
+            0.010122815147042274,
+            -0.005461421795189381,
+            0.016441410407423973,
+            -0.006183897145092487,
+            -0.004440976772457361,
+            -0.01498013362288475,
+            -0.009918726049363613,
+            0.018547609448432922,
+            -0.0055226488038897514,
+            0.008457448333501816,
+            -0.0009536059224046767,
+            -0.00576347392052412,
+            0.015674035996198654,
+            0.0129637336358428,
+            0.016718972474336624,
+            0.004771600943058729,
+            -0.00012168807734269649,
+            0.012416775338351727,
+            -0.0015745466807857156,
+            0.012555556371808052,
+            -0.0038429962005466223,
+            0.014506647363305092,
+            -0.009722800925374031,
+            -0.018988441675901413,
+            0.004157293122261763,
+            0.006265532691031694,
+            -0.012531065382063389,
+            0.009404421783983707,
+            0.007743136957287788,
+            -0.013747435994446278,
+            -0.02261306345462799,
+            -0.000106317616882734,
+            0.00014362763613462448,
+            0.008122743107378483,
+            0.008971752598881721,
+            -0.008073761127889156,
+            0.012890261597931385,
+            0.004616493359208107,
+            -0.007179851643741131,
+            -0.019657853990793228,
+            0.026009103283286095,
+            0.015608727931976318,
+            0.012269831262528896,
+            -0.052508022636175156,
+            -0.0030980713199824095,
+            0.010743245482444763,
+            -0.015951598063111305,
+            -0.026792805641889572,
+            -0.006404313258826733,
+            -0.008759500458836555,
+            -0.04290767386555672,
+            -0.009624837897717953,
+            -0.0005724696675315499,
+            -0.011388166807591915,
+            0.010400376282632351,
+            0.01573934406042099,
+            0.014612773433327675,
+            0.005489994306117296,
+            -0.007232914678752422,
+            -0.017633290961384773,
+            -0.02788672223687172,
+            0.010057506151497364,
+            -0.013380075804889202,
+            -0.006412476766854525,
+            -0.002263347152620554,
+            -0.007690073922276497,
+            0.0016112826997414231,
+            0.01691489666700363,
+            0.042483169585466385,
+            0.023037567734718323,
+            -0.012335140258073807,
+            -0.020131340250372887,
+            -0.012800462543964386,
+            -0.009192168712615967,
+            0.00633492274209857,
+            0.011592255905270576,
+            -0.004738946910947561,
+            0.0096656559035182,
+            -0.004294032696634531,
+            0.012302485294640064,
+            0.012767808511853218,
+            0.01680060662329197,
+            -2.3023791072773747e-05,
+            0.005579793825745583,
+            -0.0024694770108908415,
+            -0.01738838478922844,
+            -0.004379750229418278,
+            0.008261523209512234,
+            -0.0008872769540175796,
+            0.0019317024853080511,
+            0.014204595237970352,
+            0.019102731719613075,
+            -0.009094206616282463,
+            -0.014612773433327675,
+            -0.016098542138934135,
+            -0.005277741700410843,
+            -0.0012622905196622014,
+            0.029927613213658333,
+            0.02037624642252922,
+            -0.019266001880168915,
+            -0.018237393349409103,
+            -0.018351683393120766,
+            0.00922482367604971,
+            -0.003563394071534276,
+            -0.0064369672909379005,
+            0.01905374974012375,
+            0.023021239787340164,
+            0.017910851165652275,
+            -0.003642988856881857,
+            -0.014865843579173088,
+            0.003804219188168645,
+            0.02847449854016304,
+            0.008367649279534817,
+            0.01078406348824501,
+            0.008726846426725388,
+            -0.0011908593587577343,
+            -0.032441988587379456,
+            -0.0113146947696805,
+            0.00951871182769537,
+            -0.008481939323246479,
+            0.026564225554466248,
+            -0.016531210392713547,
+            0.009388094767928123,
+            -0.005077734589576721,
+            0.0019286411115899682,
+            -0.02365799807012081,
+            0.0014929111348465085,
+            0.010122815147042274,
+            -0.024947840720415115,
+            0.00336950970813632,
+            -0.023592690005898476,
+            -0.00013763252354692668,
+            0.008367649279534817,
+            0.0022837561555206776,
+            0.01650671847164631,
+            -0.0113146947696805,
+            0.0051838611252605915,
+            0.02099667675793171,
+            -0.002832755446434021,
+            -0.01512707769870758,
+            -0.0007842120248824358,
+            0.0005092020728625357,
+            -0.001286781276576221,
+            0.02563357912003994,
+            0.015812816098332405,
+            -0.0036756431218236685,
+            0.020882388576865196,
+            0.008677864447236061,
+            0.0019133344758301973,
+            -0.021764053031802177,
+            0.0023796779569238424,
+            -0.032899148762226105,
+            0.017437364906072617,
+            -0.011126932688057423,
+            0.012490247376263142,
+            0.00868602842092514,
+            -0.01341272983700037,
+            -0.0007388022495433688,
+            -0.0107187544927001,
+            -0.0015725058037787676,
+            -0.007934981025755405,
+            0.009779945947229862,
+            -0.004363423213362694,
+            0.008339077234268188,
+            -0.00024579971795901656,
+            -0.01476788055151701,
+            0.018694553524255753,
+            9.678283277025912e-06,
+            -0.00010925139940809458,
+            0.005441013257950544,
+            0.009935053065419197,
+            0.00786559097468853,
+            0.002009256277233362,
+            0.023870250210165977,
+            -0.031119491904973984,
+            -0.016082214191555977,
+            -0.010204450227320194,
+            0.005543057806789875,
+            -0.007922735065221786,
+            -0.018531281501054764,
+            -0.03257260471582413,
+            0.01318415068089962,
+            0.001293924287892878,
+            -0.0009806477464735508,
+            0.01258004643023014,
+            -0.00942891277372837,
+            -0.0013878053287044168,
+            -0.01606588624417782,
+            -0.014808698557317257,
+            0.006416558753699064,
+            0.0033633869607001543,
+            -0.01222901325672865,
+            -0.023739634081721306,
+            0.014759717509150505,
+            0.0007418635650537908,
+            -0.016980206593871117,
+            -0.017502672970294952,
+            0.00018457300029695034,
+            -0.004804255440831184,
+            0.02204161323606968,
+            -0.016678154468536377,
+            0.003373591462150216,
+            0.0007219648687168956,
+            0.009232986718416214,
+            -0.02889900468289852,
+            0.00716352416202426,
+            0.0006107363733462989,
+            0.0035164537839591503,
+            -0.006518602836877108,
+            -0.021013004705309868,
+            0.011371839791536331,
+            -0.0020674217958003283,
+            -0.014751553535461426,
+            -0.020751770585775375,
+            -0.0020796670578420162,
+            -0.007420676294714212,
+            -0.02724996581673622,
+            0.017910851165652275,
+            0.0009985055075958371,
+            -0.01738838478922844,
+            0.008077843114733696,
+            0.006596156861633062,
+            0.009355440735816956,
+            0.02455599047243595,
+            0.010147306136786938,
+            -0.018776189535856247,
+            -0.0008087027235887945,
+            0.006979844067245722,
+            0.009306458756327629,
+            0.0036450298503041267,
+            0.04222193360328674,
+            -0.008130906149744987,
+            -0.006579829845577478,
+            0.0066614653915166855,
+            -0.008449285291135311,
+            -0.013747435994446278,
+            0.026727497577667236,
+            0.005796127952635288,
+            0.015984252095222473,
+            0.016123032197356224,
+            -0.00945340283215046,
+            -0.005008344538509846,
+            -0.0018786393338814378,
+            0.02065380848944187,
+            0.0073798587545752525,
+            -0.005502239800989628,
+            -0.009714636951684952,
+            0.005881845485419035,
+            -0.00898808054625988,
+            0.0021653843577951193,
+            -0.009526874870061874,
+            -0.005824700463563204,
+            -0.008069679141044617,
+            0.016400592401623726,
+            -0.0003523086488712579,
+            0.027315273880958557,
+            0.020751770585775375,
+            -0.008563575334846973,
+            -0.0005683879135176539,
+            0.017208784818649292,
+            -0.005820618476718664,
+            0.015608727931976318,
+            -0.011192241683602333,
+            -0.007706400938332081,
+            0.006408394780009985,
+            0.024017194285988808,
+            0.012237177230417728,
+            -0.0056042843498289585,
+            0.0002885308349505067,
+            -0.0030348035506904125,
+            -0.010612628422677517,
+            0.013388238847255707,
+            0.015323002822697163,
+            -0.01947825588285923,
+            -0.004722619894891977,
+            -0.0016592437168583274,
+            -0.003300119424238801,
+            0.00945340283215046,
+            0.02305389568209648,
+            0.000171817431692034,
+            0.009184005670249462,
+            0.0012765767751261592,
+            0.03862996771931648,
+            -0.0053144777193665504,
+            0.03102152980864048,
+            -0.0010423846542835236,
+            -0.0059430720284581184,
+            -0.009094206616282463,
+            -0.020441556349396706,
+            -0.011486129835247993,
+            -0.00041481092921458185,
+            -0.005040998570621014,
+            -0.0004956811899319291,
+            -0.020327266305685043,
+            0.0026960158720612526,
+            0.014637264423072338,
+            0.007661501411348581,
+            0.017927179113030434,
+            -0.002889900468289852,
+            0.017437364906072617,
+            0.019690508022904396,
+            -0.01400050614029169,
+            -0.029862305149435997,
+            -0.0006515541463159025,
+            0.01498013362288475,
+            -0.01661284640431404,
+            0.019445601850748062,
+            0.004085862077772617,
+            0.0029531680047512054,
+            0.004779764451086521,
+            -0.006049198564141989,
+            0.018482301384210587,
+            0.02039257436990738,
+            0.01453930139541626,
+            -0.013167822733521461,
+            0.014057651162147522,
+            0.009771781973540783,
+            0.006600238848477602,
+            -0.01646590232849121,
+            -0.013535182923078537,
+            0.002555194543674588,
+            0.0009245232213288546,
+            0.024392718449234962,
+            -0.020588500425219536,
+            0.0016602640971541405,
+            0.010588138364255428,
+            0.01769859902560711,
+            -0.004302196204662323,
+            0.008906444534659386,
+            0.0008740111952647567,
+            0.007134951651096344,
+            -0.0035919665824621916,
+            -0.006457376293838024,
+            -0.006673710886389017,
+            -0.010392212308943272,
+            -0.0020959940738976,
+            -0.02070278860628605,
+            0.002198038622736931,
+            -0.002830714685842395,
+            0.018237393349409103,
+            -0.01018812321126461,
+            0.038140155375003815,
+            0.03399306535720825,
+            -0.0007025764207355678,
+            0.011102442629635334,
+            -0.01709449477493763,
+            -0.003945040516555309,
+            -0.021959977224469185,
+            0.01844964735209942,
+            0.016392430290579796,
+            0.012269831262528896,
+            0.005302232690155506,
+            0.017796561121940613,
+            -0.0006301248213276267,
+            0.014490320347249508,
+            0.0029654132667928934,
+            0.007971717044711113,
+            -0.002265388146042824,
+            0.003938918001949787,
+            -0.031103165820240974,
+            -0.017029186710715294,
+            -0.01138000376522541,
+            0.004796091932803392,
+            -0.006375740747898817,
+            -0.0035205355379730463,
+            0.0084737753495574,
+            -0.005902254022657871,
+            0.0004270562494639307,
+            -0.025992777198553085,
+            -0.0157148540019989,
+            -0.010318740271031857,
+            -0.004040962550789118,
+            -0.01169838197529316,
+            -0.0027694879099726677,
+            -0.0022082431241869926,
+            0.0029552089981734753,
+            0.020947696641087532,
+            -0.012808626517653465,
+            -0.016294466331601143,
+            0.004379750229418278,
+            -0.002263347152620554,
+            -0.014016833156347275,
+            -0.016898570582270622,
+            0.006665547378361225,
+            0.012098397128283978,
+            -0.0017470018938183784,
+            -0.013608654960989952,
+            0.011739199981093407,
+            -0.006628811359405518,
+            0.04663025587797165,
+            -0.013584164902567863,
+            -0.0078002819791436195,
+            0.003720542648807168,
+            0.004698128905147314,
+            -0.009396257810294628,
+            0.007330877240747213,
+            0.013020878657698631,
+            0.01830270327627659,
+            0.0095758568495512,
+            -0.008465612307190895,
+            -0.0024388637393712997,
+            0.007796200457960367,
+            -0.016147522255778313,
+            0.006404313258826733,
+            0.0008525818702764809,
+            -0.014400520361959934,
+            0.006110425107181072,
+            0.028980640694499016,
+            0.013151495717465878,
+            0.01288209855556488,
+            -0.022351829335093498,
+            -0.004094025585800409,
+            -0.009355440735816956,
+            -0.004943036008626223,
+            -0.010090161114931107,
+            0.027070365846157074,
+            -0.018155759200453758,
+            0.02651524543762207,
+            0.00898808054625988,
+            0.0017684312770143151,
+            -0.01176369097083807,
+            0.006343086715787649,
+            0.02832755446434021,
+            0.0028592869639396667,
+            -0.020000724121928215,
+            -0.013592327944934368,
+            0.006314514204859734,
+            0.01707816869020462,
+            -0.002647034591063857,
+            -0.005428767763078213,
+            0.019657853990793228,
+            0.01784554310142994,
+            -0.004334850702434778,
+            -0.00936360377818346,
+            0.0008337036124430597,
+            0.00928196869790554,
+            0.016963878646492958,
+            0.015020951628684998,
+            0.009086042642593384,
+            -0.002647034591063857,
+            0.008037025108933449,
+            -0.008800318464636803,
+            -0.016449574381113052,
+            0.007220669183880091,
+            -0.008432958275079727,
+            -0.016963878646492958,
+            0.01872720755636692,
+            0.015257694758474827,
+            0.0181394312530756,
+            0.0024592725094407797,
+            -0.016735298559069633,
+            0.022302847355604172,
+            0.005755309946835041,
+            0.002998067531734705,
+            0.02160078100860119,
+            -0.022221211344003677,
+            0.013012715615332127,
+            0.02037624642252922,
+            0.027609162032604218,
+            -0.01737205684185028,
+            -0.010906516574323177,
+            -0.004857318475842476,
+            -0.006853308994323015,
+            -0.00176945177372545,
+            -0.008253359235823154,
+            -0.012620864436030388,
+            0.0028735732194036245,
+            -0.030466407537460327,
+            0.018106777220964432,
+            -0.005102225113660097,
+            0.010669773444533348,
+            -0.019266001880168915,
+            0.011363675817847252,
+            -0.0035838030744343996,
+            0.011135096661746502,
+            0.0019612954929471016,
+            -0.030041903257369995,
+            -0.008571738377213478,
+            0.015388311818242073,
+            -0.0030368445441126823,
+            -0.013641309924423695,
+            0.02006603218615055,
+            0.00018942011229228228,
+            -0.022270193323493004,
+            -0.004314441699534655,
+            -0.006934944540262222,
+            0.002193956868723035,
+            -0.0029613315127789974,
+            -0.0052981507033109665,
+            -0.030188847333192825,
+            0.020033378154039383,
+            0.009559528902173042,
+            -0.03072764165699482,
+            -0.003255219664424658,
+            -0.014996460638940334,
+            -0.0025184585247188807,
+            0.008881953544914722,
+            -0.004000144544988871,
+            -0.015551582910120487,
+            0.01784554310142994,
+            0.022711025550961494,
+            -0.0023164101876318455,
+            -0.016057724133133888,
+            -0.0090697156265378,
+            -0.027494871988892555,
+            0.030825603753328323,
+            -0.0066696288995444775,
+            0.0022878379095345736,
+            -0.005322641227394342,
+            0.009388094767928123,
+            0.004228724166750908,
+            0.0157148540019989,
+            0.006081852596253157,
+            -0.002647034591063857,
+            0.026596879586577415,
+            -0.01964152604341507,
+            0.009567692875862122,
+            -1.588864870427642e-05,
+            -0.003308282932266593,
+            0.01890680566430092,
+            0.0007939062779769301,
+            0.021649762988090515,
+            -0.008571738377213478,
+            -0.0001475818717153743,
+            -0.0034388997592031956,
+            0.010898353531956673,
+            -0.0028450009413063526,
+            -0.003465431509539485,
+            -0.001531688030809164,
+            0.011118769645690918,
+            0.01013097818940878,
+            0.003804219188168645,
+            -0.01737205684185028,
+            0.003142970846965909,
+            -0.008547247387468815,
+            -0.007028825581073761,
+            0.01302904263138771,
+            -0.011192241683602333,
+            -0.0018990482203662395,
+            -0.03160930797457695,
+            0.016286302357912064,
+            0.0533243790268898,
+            0.008547247387468815,
+            -0.014433175325393677,
+            -0.0018806802108883858,
+            -0.019755816087126732,
+            0.01535565685480833,
+            0.011004479601979256,
+            -0.016816934570670128,
+            -0.011984107084572315,
+            -0.012310649268329144,
+            0.007649256382137537,
+            0.021306892856955528,
+            -0.006522684823721647,
+            -0.008579902350902557,
+            0.025372346863150597,
+            -0.011788181029260159,
+            -0.010882026515901089,
+            -0.01661284640431404,
+            -0.01635977439582348,
+            0.00241641397587955,
+            0.019902760162949562,
+            -0.0169475506991148,
+            0.021176276728510857,
+            0.00936360377818346,
+            0.0010133018950000405,
+            0.003624620847404003,
+            0.008588065393269062,
+            0.0024878450203686953,
+            0.017878197133541107,
+            0.003883813973516226,
+            -0.01439235731959343,
+            0.0021408938337117434,
+            -0.0030286810360848904,
+            0.004624656867235899,
+            3.4320437407586724e-06,
+            -0.002749078907072544,
+            -0.0065635028295218945,
+            0.003710338147357106,
+            -0.021306892856955528,
+            0.021502818912267685,
+            -0.022678371518850327,
+            -0.004245051648467779,
+            0.005555302836000919,
+            -0.005118552595376968,
+            0.013469874858856201,
+            0.018792515620589256,
+            0.01423724927008152,
+            0.008702355436980724,
+            0.01462093647569418,
+            -0.011331021785736084,
+            0.02323349379003048,
+            -0.013282112777233124,
+            0.0005816536722704768,
+            0.012939243577420712,
+            0.01755165494978428,
+            0.018792515620589256,
+            -0.006245123688131571,
+            -0.010384049266576767,
+            -0.0002946535241790116,
+            0.0129637336358428,
+            -0.022286519408226013,
+            -0.012269831262528896,
+            -0.01386172603815794,
+            0.011486129835247993,
+            -0.004743028432130814,
+            0.025453981012105942,
+            -0.00633492274209857,
+            -0.014653591439127922,
+            -0.015918942168354988,
+            0.014196431264281273,
+            -0.048752784729003906,
+            0.0030348035506904125,
+            -0.0024919267743825912,
+            0.03043375350534916,
+            -0.027021385729312897,
+            0.01738838478922844,
+            -0.0028490826953202486,
+            -0.009355440735816956,
+            0.015429128892719746,
+            -0.015992414206266403,
+            -0.0046613928861916065,
+            -0.0016806729836389422,
+            -0.007539048325270414,
+            -0.00112248957157135,
+            0.017616963014006615,
+            -0.015845470130443573,
+            0.01282495353370905,
+            -0.00989423505961895,
+            0.0027021386194974184,
+            -0.004526694305241108,
+            -0.007751300930976868,
+            0.0196088720113039,
+            0.034548189491033554,
+            -0.007053316105157137,
+            0.0014990337658673525,
+            -0.022547753527760506,
+            0.013192313723266125,
+            -0.006730855442583561,
+            0.02351105399429798,
+            -0.005432849284261465,
+            -0.016898570582270622,
+            -0.013976015150547028,
+            -0.009739127941429615,
+            0.00839214026927948,
+            -0.022433463484048843,
+            -0.011110605672001839,
+            -0.00922482367604971,
+            -0.002573562553152442,
+            -0.018694553524255753,
+            0.004865481983870268,
+            -0.002426618244498968,
+            -0.012506574392318726,
+            -0.008816645480692387,
+            0.01048201136291027,
+            -0.009347276762127876,
+            0.003077662317082286,
+            -0.0011929002357646823,
+            -0.011453475803136826,
+            -0.02369065210223198,
+            0.0158944521099329,
+            0.0001974561164388433,
+            -0.010359558276832104,
+            0.008408467285335064,
+            -0.0018276170594617724,
+            0.001211268245242536,
+            0.016963878646492958,
+            -0.0020664012990891933,
+            0.003467472270131111,
+            0.023478399962186813,
+            0.004057289566844702,
+            -0.010971825569868088,
+            -0.00025906547671183944,
+            0.013535182923078537,
+            -0.005118552595376968,
+            -0.010008525103330612,
+            1.6646010408294387e-05,
+            -0.02984597720205784,
+            -0.008351322263479233,
+            0.006094098091125488,
+            0.018629245460033417,
+            -0.010620792396366596,
+            -0.015853634104132652,
+            0.026319319382309914,
+            -0.006359413731843233,
+            0.019070077687501907,
+            0.002889900468289852,
+            0.0025694805663079023,
+            0.01724144071340561,
+            0.013461710885167122,
+            0.00786150898784399,
+            -0.013927034102380276,
+            -0.02429475635290146,
+            0.0016398552106693387,
+            -0.018090449273586273,
+            0.010579974390566349,
+            -0.011028970591723919,
+            0.0032756286673247814,
+            -0.02398454025387764,
+            -0.012237177230417728,
+            0.004934872500598431,
+            -0.020343592390418053,
+            -0.00040588201954960823,
+            -0.0056940834037959576,
+            0.022237539291381836,
+            -0.005310396198183298,
+            0.008669701404869556,
+            0.0018255761824548244,
+            -0.012237177230417728,
+            0.0009877908742055297,
+            0.009469729848206043,
+            -0.029192892834544182,
+            -0.0007127808639779687,
+            -0.003138889092952013,
+            0.0011418780777603388,
+            0.005351213738322258,
+            0.009755454957485199,
+            0.007322713732719421,
+            0.0027062203735113144,
+            0.007253323681652546,
+            0.010449357330799103,
+            -0.00022041612828616053,
+            0.0197884701192379,
+            0.018172085285186768,
+            -0.007351286243647337,
+            0.008710519410669804,
+            0.024817224591970444,
+            -0.013159659691154957,
+            -0.008718682453036308,
+            0.023266147822141647,
+            -0.006461458280682564,
+            -0.0011122850701212883,
+            -0.01483318954706192,
+            0.00815947912633419,
+            0.0017367975087836385,
+            0.028131630271673203,
+            0.019657853990793228,
+            0.005820618476718664,
+            0.000639819074422121,
+            0.015788326039910316,
+            -0.013020878657698631,
+            0.0035286990460008383,
+            0.015461783856153488,
+            -0.003940958995372057,
+            -0.019821124151349068,
+            0.013878053054213524,
+            0.009502384811639786,
+            -0.003228688146919012,
+            0.01551892887800932,
+            0.011412657797336578,
+            -0.030841931700706482,
+            -0.0011898389784619212,
+            0.010498338378965855,
+            -0.0021449755877256393,
+            -0.008751336485147476,
+            -0.00014273475972004235,
+            -0.012384121306240559,
+            0.021388528868556023,
+            -0.004408322740346193,
+            0.012277995236217976,
+            -0.00815947912633419,
+            -0.017127150669693947,
+            -0.011559601873159409,
+            0.007914572022855282,
+            -0.010547320358455181,
+            -0.0006714528426527977,
+            -0.004022594541311264,
+            -0.005779800936579704,
+            0.017747581005096436,
+            0.004706292413175106,
+            -0.0008964610169641674,
+            -0.0018398624379187822,
+            0.007983962073922157,
+            -0.006979844067245722,
+            -0.008726846426725388,
+            0.004816500470042229,
+            -0.012833117507398129,
+            0.003981776535511017,
+            0.005543057806789875,
+            0.01506176870316267,
+            -0.014857680536806583,
+            -0.021159948781132698,
+            0.007020662073045969,
+            0.02695607766509056,
+            -0.02516009286046028,
+            -0.005232842173427343,
+            0.006245123688131571,
+            0.002093953313305974,
+            0.006914535537362099,
+            -0.006934944540262222,
+            -0.007208424154669046,
+            -0.03657275065779686,
+            -0.0020745648071169853,
+            0.0009005427709780633,
+            0.008963589556515217,
+            -0.008049271069467068,
+            -0.0136494729667902,
+            -0.011135096661746502,
+            0.002410291228443384,
+            0.007134951651096344,
+            -0.02204161323606968,
+            0.0049593630246818066,
+            0.009608510881662369,
+            -0.019559891894459724,
+            0.025127438828349113,
+            -0.010890189558267593,
+            -6.964537897147238e-05,
+            -0.00995954405516386,
+            0.021633435040712357,
+            -0.013486201874911785,
+            -0.018368011340498924,
+            -0.0006898208521306515,
+            0.008355404250323772,
+            0.011926962062716484,
+            -0.0017663904000073671,
+            -0.011371839791536331,
+            0.0003752686607185751,
+            0.01646590232849121,
+            0.019135385751724243,
+            0.002744997153058648,
+            -0.0024837632663547993,
+            -0.01019628718495369,
+            -0.009592183865606785,
+            -0.0073961857706308365,
+            -0.0007449248805642128,
+            0.002091912319883704,
+            0.02035992033779621,
+            0.013469874858856201,
+            0.0024837632663547993,
+            -0.015135240741074085,
+            -0.005257333163172007,
+            0.008939098566770554,
+            0.009412584826350212,
+            -0.011690218932926655,
+            -0.00898808054625988,
+            0.002659279853105545,
+            0.02249877341091633,
+            0.017633290961384773,
+            -0.0014000505907461047,
+            0.0032878739293664694,
+            -0.0016592437168583274,
+            0.012506574392318726,
+            -0.007555375341325998,
+            -0.010269759222865105,
+            -0.006943108048290014,
+            -0.014645427465438843,
+            -0.010800390504300594,
+            -0.0023062059190124273,
+            -0.010179960168898106,
+            0.022629389539361,
+            0.007988044060766697,
+            0.009779945947229862,
+            0.008265605196356773,
+            0.010269759222865105,
+            0.002087830565869808,
+            0.025715215131640434,
+            -0.004424649756401777,
+            -0.011984107084572315,
+            0.007236996199935675,
+            -0.022368155419826508,
+            -0.021535472944378853,
+            0.008257441222667694,
+            0.006367577239871025,
+            0.03595232218503952,
+            -0.026890767738223076,
+            0.0052981507033109665,
+            -0.009779945947229862,
+            -0.010400376282632351,
+            0.007996207103133202,
+            -0.0015439334092661738,
+            -0.004245051648467779,
+            0.022368155419826508,
+            0.00726556871086359,
+            0.0035797213204205036,
+            0.0032817514147609472,
+            0.019837452098727226,
+            -0.005685919895768166,
+            0.01573934406042099,
+            0.0259274672716856,
+            0.024621298536658287,
+            0.009388094767928123,
+            -0.009469729848206043,
+            0.02785406820476055,
+            0.0002895512734539807,
+            -0.0034327772445976734,
+            0.007220669183880091,
+            -0.00305113079957664,
+            -0.0009556467994116247,
+            -0.010033016093075275,
+            0.021764053031802177,
+            -0.001455154619179666,
+            0.004359341226518154,
+            0.02081707864999771,
+            -7.774515688652173e-05,
+            0.006220633164048195,
+            0.01057181041687727,
+            0.002181711606681347,
+            0.003396041225641966,
+            0.010775899514555931,
+            -0.039315707981586456,
+            -0.0006571666453965008,
+            -0.025600925087928772,
+            -0.0018286375561729074,
+            0.0007357408758252859,
+            -0.01266984548419714,
+            0.013600491918623447,
+            -0.002015379024669528,
+            -0.010751409456133842,
+            -0.00029337796149775386,
+            0.021649762988090515,
+            0.005416522268205881,
+            0.004071575589478016,
+            -0.004469549283385277,
+            0.018792515620589256,
+            -0.006955353543162346,
+            -0.017861871048808098,
+            -0.012759645469486713,
+            -0.012367794290184975,
+            -0.01918436773121357,
+            -0.01214737817645073,
+            -0.0024551907554268837,
+            -0.002667443361133337,
+            0.004738946910947561,
+            0.002481722505763173,
+            -0.003228688146919012,
+            -0.02070278860628605,
+            0.015037278644740582,
+            -0.01430255826562643,
+            0.012873934581875801,
+            -0.011053460650146008,
+            0.005963481031358242,
+            -0.004122598096728325,
+            0.012726990506052971,
+            -0.021404854953289032,
+            0.02070278860628605,
+            -0.01536382082849741,
+            0.0038532004691660404,
+            0.019592545926570892,
+            0.006551257334649563,
+            -0.008118661120533943,
+            -0.013404566794633865,
+            -0.022074267268180847,
+            0.021633435040712357,
+            -0.011633073911070824,
+            -0.018237393349409103,
+            -0.004608329851180315,
+            0.009649328887462616,
+            -0.017470018938183784,
+            -0.007367613259702921,
+            0.01707816869020462,
+            -0.0018949664663523436,
+            0.012980061583220959,
+            0.023004913702607155,
+            -0.0052981507033109665,
+            0.0036572751123458147,
+            -0.009167678654193878,
+            0.0010286086471751332,
+            -0.02470293454825878,
+            0.013527019880712032,
+            0.008971752598881721,
+            0.006269614677876234,
+            -0.013935198076069355,
+            -0.013306603766977787,
+            0.0036531933583319187,
+            0.004498121794313192,
+            0.011437147855758667,
+            0.009420748800039291,
+            0.004220560658723116,
+            -0.013918871060013771,
+            -0.027788760140538216,
+            -0.027070365846157074,
+            0.03595232218503952,
+            -0.0005002731923013926,
+            -0.020947696641087532,
+            0.03399306535720825,
+            0.016114868223667145,
+            0.0022980421781539917,
+            -0.012449429370462894,
+            -0.004314441699534655,
+            0.024621298536658287,
+            0.016400592401623726,
+            0.010465684346854687,
+            -0.0004742518358398229,
+            0.005571629852056503,
+            0.0016469983384013176,
+            0.03012353740632534,
+            0.015323002822697163,
+            0.0009459525463171303,
+            -0.038597315549850464,
+            0.011943289078772068,
+            -0.007547211833298206,
+            0.0013408648082986474,
+            0.006702283397316933,
+            0.017780235037207603,
+            0.011935125105082989,
+            0.015567909926176071,
+            0.014865843579173088,
+            -0.003065417055040598,
+            0.0029511270113289356,
+            0.0006163488142192364,
+            -0.009624837897717953,
+            -0.001767410896718502,
+            -0.008881953544914722,
+            -0.005714492406696081,
+            -0.002920513739809394,
+            0.005592038854956627,
+            0.007657419890165329,
+            -0.003983817528933287,
+            0.028049994260072708,
+            -0.0006433906382881105,
+            -0.017600636929273605,
+            -0.009061552584171295,
+            -0.03611559048295021,
+            0.00434301421046257,
+            -0.030939893797039986,
+            -0.011257549747824669,
+            0.0033674687147140503,
+            -0.020457882434129715,
+            -0.024768242612481117,
+            0.0010306495241820812,
+            0.014335212297737598,
+            0.007392104249447584,
+            0.0033327736891806126,
+            -0.01004934310913086,
+            0.003816464450210333,
+            -0.001855169073678553,
+            -0.006461458280682564,
+            0.028833696618676186,
+            0.019559891894459724,
+            0.020457882434129715,
+            -0.01190247107297182,
+            0.019298657774925232,
+            0.005845109466463327,
+            0.02035992033779621,
+            0.010465684346854687,
+            -0.0048695639707148075,
+            0.0074859848245978355,
+            0.009567692875862122,
+            -0.006155324634164572,
+            -0.021519144997000694,
+            0.013192313723266125,
+            -0.0010837126756086946,
+            -0.0052083516493439674,
+            -0.016269976273179054,
+            -0.003534821793437004,
+            -0.015812816098332405,
+            0.010547320358455181,
+            -0.017159804701805115,
+            0.012498411349952221,
+            -0.000802069844212383,
+            0.00168475485406816,
+            -0.015469946898519993,
+            -0.013951525092124939,
+            -0.0012439225101843476,
+            -0.01244126632809639,
+            -0.008971752598881721,
+            0.006163488142192364,
+            -0.007539048325270414,
+            -0.008318668231368065,
+            -0.00535937724635005,
+            -0.011437147855758667,
+            -0.010882026515901089,
+            0.004367504734545946,
+            -0.0010653446661308408,
+            0.020931368693709373,
+            0.00011671340325847268,
+            -0.005824700463563204,
+            0.01190247107297182,
+            0.02787039615213871,
+            0.0016704685986042023,
+            0.004583839327096939,
+            -0.014261740259826183,
+            -0.011118769645690918,
+            -0.02279266156256199,
+            0.016572028398513794,
+            0.0049675265327095985,
+            -0.0344502255320549,
+            0.01438419334590435,
+            0.00824927818030119,
+            0.0015102586476132274,
+            -0.006698201410472393,
+            -0.005334886722266674,
+            -0.005608365871012211,
+            0.006306350696831942,
+            -0.0022837561555206776,
+            0.0005107327597215772,
+            0.0035919665824621916,
+            0.034678805619478226,
+            -0.0040674940682947636,
+            -0.007228832691907883,
+            0.00981259997934103,
+            0.0005130287609063089,
+            0.020196648314595222,
+            -0.005518566817045212,
+            0.0037572786677628756,
+            -0.019282329827547073,
+            -0.016563864424824715,
+            0.0018429238116368651,
+            -0.01722511276602745,
+            0.023282473906874657,
+            -0.005392031744122505,
+            -0.01347803883254528,
+            -0.006906372029334307,
+            -0.020115012302994728,
+            0.012637191452085972,
+            0.02310287579894066,
+            0.007016580086201429,
+            0.01033506728708744,
+            0.012131051160395145,
+            -0.02444170042872429,
+            -0.020457882434129715,
+            0.012171869166195393,
+            0.0013571919407695532,
+            -0.008449285291135311,
+            -0.005404277238994837,
+            -0.001783737912774086,
+            0.007783954963088036,
+            -0.0032633834052830935,
+            0.007881917990744114,
+            0.06491663306951523,
+            -0.01093100756406784,
+            -0.002987863030284643,
+            -0.02966637909412384,
+            -0.004396077245473862,
+            -0.006628811359405518,
+            0.0056287748739123344,
+            0.009151351638138294,
+            -0.023135529831051826,
+            0.009232986718416214,
+            -0.030025575309991837,
+            -0.010628955438733101,
+            0.00824111420661211,
+            0.0028490826953202486,
+            -0.01220452319830656,
+            -0.0009592183632776141,
+            -0.013665799982845783,
+            0.0006107363733462989,
+            -0.009412584826350212,
+            -0.00576347392052412,
+            0.012718827463686466,
+            -0.02321716584265232,
+            0.006298186723142862,
+            0.007592111360281706,
+            -0.006514521315693855,
+            -0.013722945004701614,
+            -0.017861871048808098,
+            -0.011967780068516731,
+            -0.013314766809344292,
+            0.002996026771143079,
+            0.01190247107297182,
+            -0.02607441134750843,
+            0.012922916561365128,
+            -0.0037389106582850218,
+            0.00945340283215046,
+            0.004404240753501654,
+            0.013469874858856201,
+            0.018074123188853264,
+            0.009355440735816956,
+            0.0021429345943033695,
+            -0.008024780079722404,
+            0.012702500447630882,
+            -0.01993541419506073,
+            -0.01635161228477955,
+            0.013837235048413277,
+            0.012016761116683483,
+            0.003061335301026702,
+            0.009828926995396614,
+            0.0020959940738976,
+            0.011886144056916237,
+            0.010775899514555931,
+            -0.019739490002393723,
+            -0.0006648199632763863,
+            0.0007326795603148639,
+            0.03280118480324745,
+            -0.0005938990507274866,
+            0.008857463486492634,
+            0.004420568235218525,
+            0.013159659691154957,
+            -0.012784135527908802,
+            -0.01769859902560711,
+            0.0008087027235887945,
+            -0.01057181041687727,
+            0.021257910877466202,
+            -0.008192133158445358,
+            0.006216551177203655,
+            -0.018612917512655258,
+            -0.027086693793535233,
+            0.006392067763954401,
+            -0.016261812299489975,
+            -0.018221067264676094,
+            0.01317598670721054,
+            0.0077268099412322044,
+            0.0017572063952684402,
+            -0.005098143592476845,
+            -0.020457882434129715,
+            -0.009837090037763119,
+            0.028441844508051872,
+            0.011102442629635334,
+            -0.004120557103306055,
+            0.009584019891917706,
+            -0.009094206616282463,
+            -0.012490247376263142,
+            -0.0017541450215503573,
+            0.0036715613678097725,
+            0.008261523209512234,
+            0.005645101889967918,
+            -0.01386172603815794,
+            -0.00029873527819290757,
+            -0.012400448322296143,
+            -0.021225256845355034,
+            -0.008906444534659386,
+            0.016751626506447792,
+            0.007820690982043743,
+            -0.0038960592355579138,
+            0.015812816098332405,
+            -0.004404240753501654,
+            0.002189875114709139,
+            0.014294394291937351,
+            0.005334886722266674,
+            -0.004285869188606739,
+            -0.0095758568495512,
+            -0.004612411838024855,
+            0.004043003544211388,
+            -0.0025960123166441917,
+            0.006457376293838024,
+            0.017192458733916283,
+            0.0072900597006082535,
+            -0.013755599036812782,
+            0.013135168701410294,
+            0.0007791098323650658,
+            0.0049675265327095985,
+            -0.03699725493788719,
+            -0.003247056156396866,
+            0.015796490013599396,
+            -0.013020878657698631,
+            0.02501314878463745,
+            0.005400195252150297,
+            0.022743679583072662,
+            0.015233203768730164,
+            0.008669701404869556,
+            -0.001542912912555039,
+            0.00464098434895277,
+            0.02759283408522606,
+            -0.003151134354993701,
+            -0.008637047372758389,
+            -0.0020602785516530275,
+            0.009241150692105293,
+            0.0002063850115519017,
+            0.003000108525156975,
+            0.008281932212412357,
+            -0.03562577813863754,
+            0.011706545948982239,
+            -0.019576217979192734,
+            0.0024694770108908415,
+            0.003459308762103319,
+            -0.007183933164924383,
+            -0.014865843579173088,
+            -0.005857354495674372,
+            0.022841641679406166,
+            -0.003224606392905116,
+            0.021078312769532204,
+            0.002885818714275956,
+            0.016106704249978065,
+            -0.011575928889214993,
+            -0.009208496659994125,
+            -0.0004130251472815871,
+            -0.005351213738322258,
+            -0.007539048325270414,
+            0.010596301406621933,
+            0.0034409407526254654,
+            0.005534893833100796,
+            0.002265388146042824,
+            0.0036654386203736067,
+            -0.0036878883838653564,
+            0.02773977816104889,
+            0.003563394071534276,
+            0.007996207103133202,
+            -0.011567764915525913,
+            0.014082142151892185,
+            -0.017029186710715294,
+            0.0070778070949018,
+            0.005575711838901043,
+            -0.026156047359108925,
+            0.016849588602781296,
+            0.004775682929903269,
+            0.006628811359405518,
+            0.013812744058668613,
+            0.01618834026157856,
+            -0.010408539324998856,
+            -0.014816862531006336,
+            0.024049850180745125,
+            -0.0032041973900049925,
+            -0.015225040726363659,
+            -0.006775755435228348,
+            4.438935866346583e-05,
+            0.006559420842677355,
+            0.00373074715025723,
+            0.0013020879123359919,
+            -0.0006204306264407933,
+            -0.010743245482444763,
+            0.004645065870136023,
+            -0.01934763789176941,
+            -0.0007163524278439581,
+            -0.016449574381113052,
+            -0.015633217990398407,
+            0.018678225576877594,
+            -0.01830270327627659,
+            0.009633000940084457,
+            0.0012275953777134418,
+            0.028588788583874702,
+            -0.011706545948982239,
+            -0.012155541218817234,
+            0.0015102586476132274,
+            0.010849371552467346,
+            0.0067512644454836845,
+            -0.010033016093075275,
+            -0.007583947852253914,
+            -0.014180104248225689,
+            -0.010163633152842522,
+            0.015502601861953735,
+            -0.014498483389616013,
+            -0.014343376271426678,
+            0.008914608508348465
+          ],
+          "embedding": [
+            -0.006359657085039302,
+            -0.024594786241399234,
+            -0.02553538924848095,
+            0.005360188465088291,
+            0.012801870977589839,
+            0.021680505862506104,
+            -0.016668048966001273,
+            0.06860852474581301,
+            -0.007122272841157724,
+            0.03471775000705488,
+            -0.009004297848861675,
+            0.01059974144678391,
+            0.02565604599426148,
+            -0.022544127646041287,
+            0.010616674976540785,
+            0.013658194823359737,
+            0.016400329743363844,
+            0.02858546964917478,
+            0.0067612676434693686,
+            -0.04174737193606879,
+            0.03184155110488616,
+            -0.02223105127585514,
+            -0.05506239667974977,
+            -0.0015307551410555752,
+            -0.013169788751379287,
+            0.017681749722384007,
+            0.01682706317229353,
+            0.019652295823816928,
+            -0.05623980369271269,
+            0.030698582851533197,
+            0.02286327124296613,
+            0.016019032539689178,
+            0.005508113353872315,
+            8.58985091215149e-05,
+            -0.019206292853495906,
+            -0.018439290483495144,
+            0.03475023176950151,
+            0.01018612942141759,
+            -0.0252551158857426,
+            0.0334562130666827,
+            0.020462153537485785,
+            0.02260508249714211,
+            -0.03323524989392951,
+            -0.002094251009530707,
+            0.015177442347686773,
+            -0.00018698464350038317,
+            -0.0006788648344273403,
+            0.0007777616893581039,
+            0.013352398615059647,
+            -0.030751111528744813,
+            -0.0317733818445192,
+            -0.006729064081710058,
+            0.009413204899986514,
+            -0.005712894407943592,
+            -0.038961826880527066,
+            -0.002827692206530345,
+            0.017863300537973137,
+            0.07918455388563496,
+            0.048967158919816366,
+            0.018002566325350088,
+            0.040264962620950695,
+            -0.019715100359659787,
+            -0.02211350500879408,
+            0.004918148608475574,
+            0.01062585305866606,
+            -0.014478993521696027,
+            0.0034983131226364672,
+            0.000766228629514787,
+            0.009667268771502067,
+            -0.01558439890697323,
+            -0.04984718709870083,
+            0.029613955432821887,
+            -0.009686854114630624,
+            -0.015279319238124568,
+            0.007460629676325322,
+            -0.006235979350809034,
+            0.012514387473717085,
+            0.02800435263741497,
+            -0.008688925946672778,
+            -0.01236301543374469,
+            0.029755705020188972,
+            0.01158037219446542,
+            -0.005242392206024511,
+            0.009982679027728814,
+            0.052414361714483264,
+            -0.018788500381726227,
+            -0.020335831196027204,
+            -0.08583959943356621,
+            0.029309228216812346,
+            -0.0249285282010673,
+            -0.018929396555426307,
+            0.007710663193334574,
+            0.03096798035958051,
+            -0.017089977064591828,
+            -0.028015107605405817,
+            0.06740703271128606,
+            0.009398795412354399,
+            0.01118769022826835,
+            -0.03041417460855722,
+            0.014776002883150835,
+            0.004319194067673161,
+            0.022507636639041203,
+            0.0053562788646786845,
+            0.007548072258943124,
+            0.011123374308563128,
+            -0.006924362115585165,
+            -0.011118507147092024,
+            0.009563232643295304,
+            0.00657178473678069,
+            0.08313027447380966,
+            -0.004239370156003264,
+            -0.00930406163946907,
+            -0.039202672821898946,
+            0.009761029930556298,
+            -0.02827698465424993,
+            -0.037914533246900244,
+            -0.04496050911855815,
+            -0.017711307900236353,
+            0.01934435542287117,
+            0.004335173027873153,
+            0.03800509969714242,
+            0.031198127357716314,
+            0.031311207941680276,
+            -0.012184553894588813,
+            -0.058985863670856876,
+            0.041442017057707635,
+            -0.012456189292721711,
+            -0.0040966351720394135,
+            0.010849179519959421,
+            0.005243123049347636,
+            0.0011495799963862647,
+            0.003534183676934487,
+            -0.03139340512003631,
+            -0.023876261883905306,
+            0.008382957569752911,
+            -0.004778283217011976,
+            0.03703067589888386,
+            0.06888630688839406,
+            -0.03993532296478411,
+            -0.009472167678222372,
+            0.025246872942105057,
+            -0.01524567912700368,
+            0.01595917939878582,
+            -0.003736723164442467,
+            -0.030836797784967753,
+            0.0026246446163368533,
+            0.008894300082148884,
+            0.08342856279154155,
+            -0.0007476580521704578,
+            0.025149116238550588,
+            0.004972542633631468,
+            0.034495632546556576,
+            -0.009173329764686424,
+            0.014009381543772768,
+            0.004427261668151028,
+            -0.013426138607902904,
+            -0.022234568880321808,
+            0.01976282364689966,
+            -0.03592433789490043,
+            -0.010918278220831188,
+            0.013527180289641174,
+            -0.013023113052242677,
+            -0.008304770302495363,
+            0.003677063799167178,
+            0.0013981822663219421,
+            0.047001163242600175,
+            0.01828332714887986,
+            0.002483375662343689,
+            -0.06820498187523893,
+            0.05295312038995555,
+            0.025038753504342162,
+            0.025286870198436735,
+            -0.0029544808216316115,
+            0.029982752196635428,
+            0.0005600481523874767,
+            0.0128290258663539,
+            0.009023805227424919,
+            -0.0019219166367750596,
+            -0.04482843936478531,
+            0.04523479691409549,
+            -0.004678574027241186,
+            0.010486850804218904,
+            0.006736229865018014,
+            0.02134813656476443,
+            0.028083972673690036,
+            0.016857413440125445,
+            0.0189082447820146,
+            0.03843690613820022,
+            -0.011922027011229613,
+            0.016193433524453224,
+            0.03828207911467065,
+            -0.0013055128404331705,
+            0.043198163443700956,
+            0.024025562893260542,
+            -0.001257851946749882,
+            -0.011726713433348844,
+            -0.021907984099443396,
+            0.02752198420597673,
+            0.004431866336999851,
+            0.015798806714413912,
+            -0.012910782191382367,
+            -0.03180549180206364,
+            0.004550247372715741,
+            -0.022778272841118514,
+            0.031242218240017532,
+            -0.006106984949693483,
+            -0.006259993190497421,
+            0.02187449047033339,
+            0.009334146847221406,
+            -0.009381278960016145,
+            -0.020888853823931134,
+            -0.007502374908776156,
+            -0.010390018386728482,
+            -0.04316693435221676,
+            -0.006274513730720055,
+            0.019152443937064258,
+            -0.01668984824845791,
+            0.018573194036505578,
+            0.0035035262183249925,
+            -0.019779911381789077,
+            -0.0027495798165085993,
+            0.02425481236315913,
+            0.006857723545270935,
+            -0.014733535699087731,
+            -0.026326146471941727,
+            -0.019906660231056833,
+            -0.001033294559612807,
+            0.0159154063026895,
+            0.035649534442431415,
+            -0.026210592096240003,
+            0.014397127229517285,
+            0.005114311681443673,
+            -0.0033104236005207077,
+            0.00324988235167305,
+            0.03764004154553922,
+            -0.017822437760737605,
+            -0.008579701063708334,
+            -0.00032410984200965487,
+            0.018450351071676443,
+            0.017283379764846195,
+            0.006384411611332382,
+            -0.005115090309123598,
+            -0.030590047062583125,
+            -0.011776211359193503,
+            0.01183623914060493,
+            0.0016564847796208331,
+            -0.009530766634361575,
+            -0.004810598258154945,
+            0.03876242515526699,
+            0.027076785774401667,
+            0.005324763136165989,
+            -0.03416412480913939,
+            -0.007943016733586697,
+            0.002114307650456928,
+            -0.02961449347300628,
+            0.05142732770518828,
+            0.05566026433842563,
+            -0.0030729309664974704,
+            -0.00938186948393074,
+            -0.04547908981236149,
+            0.0651216359692231,
+            -0.01813722445944094,
+            -0.0482650853560993,
+            0.044045155892545554,
+            -0.014272653003441165,
+            0.013740652306746851,
+            0.04563732439184395,
+            0.02126513340521316,
+            -0.0054845632439384905,
+            -0.011635625101262898,
+            0.030992082069685376,
+            -0.010051935721068779,
+            -0.050967258971854425,
+            -0.009827942095033494,
+            0.013373878556654464,
+            -0.013017627321534534,
+            -0.009361001542607812,
+            -0.01925867665425919,
+            0.020329052915436022,
+            0.023020741670660188,
+            0.019387043947695172,
+            0.005628632318152202,
+            0.018451555161952665,
+            0.016226845400848455,
+            -0.01384033112836216,
+            0.004762589697426251,
+            -0.012718476561725138,
+            -0.024222765127494463,
+            -0.037487144314010595,
+            -0.019971524166659278,
+            -0.015034588610516958,
+            0.009483936496215338,
+            -0.014301924283553591,
+            -0.0024139665139412125,
+            0.013361045353529458,
+            -0.0005870798721107379,
+            -0.011192319340879789,
+            0.013927695434391159,
+            0.021337539669342293,
+            -0.027300952495815437,
+            -0.04765670695199488,
+            -0.00561706224251562,
+            0.0082776639906139,
+            -0.0030984973749633724,
+            -0.026269016057688548,
+            0.007040524859505573,
+            0.0017859008072463442,
+            0.020091542517503808,
+            -0.018716761130867725,
+            0.0035283051960672963,
+            0.008519383548064053,
+            -0.015247417050336866,
+            -0.016917626868166634,
+            -0.018580550690103854,
+            -0.024697180580294884,
+            0.034202028316737125,
+            0.0204395003463227,
+            0.030392895534267717,
+            -0.04527669147753553,
+            -0.018359428092075348,
+            0.05494107499558624,
+            0.015234754907198422,
+            -0.05066008106884831,
+            0.031139824660661085,
+            0.005016615710206006,
+            0.0019000453624929544,
+            -0.006094364589229031,
+            0.0637324238017742,
+            -0.02711381489292883,
+            -0.02187418906973867,
+            0.004601744864532875,
+            -0.04141121277496072,
+            -0.026689585608019588,
+            -0.005568100750837504,
+            0.01291019036002459,
+            -0.031853041426355644,
+            -0.02815800259372657,
+            -0.007964861641790374,
+            -0.010933515363060793,
+            0.013726639077745378,
+            -0.023956800669647334,
+            -0.015672058590455337,
+            -0.023642025374517057,
+            -0.01288435863544158,
+            -0.0042516881552192385,
+            -0.011162214160460695,
+            -0.02412733256710621,
+            0.010088528028211243,
+            0.03310175458095482,
+            0.023064360903748363,
+            0.001996159684818467,
+            0.009478229572933101,
+            0.014478772877286076,
+            0.00371783700962415,
+            0.007864103610320116,
+            -0.048511515124609,
+            -0.031398754206534776,
+            0.020227872561659397,
+            0.012159811587094092,
+            0.0172163448580163,
+            0.009578952796095119,
+            -0.04154545961112031,
+            -0.05095080393682759,
+            0.02968486144855133,
+            -0.03932395023720674,
+            -0.02063219739061335,
+            0.007512423249865194,
+            -0.011911709468952548,
+            -0.00810951734088105,
+            -0.0008079327452253181,
+            -0.01600701450676789,
+            0.023984909975645632,
+            -0.02418072541081482,
+            0.0033984265728439398,
+            -0.0104343782811971,
+            0.01588977364765743,
+            0.05109529246167992,
+            -0.027800269136366997,
+            -0.027889034481533438,
+            0.03706813210665186,
+            -0.02058120652429308,
+            0.030801396468451574,
+            0.03246127847621613,
+            0.008355517405358826,
+            -0.008605467586247932,
+            0.017693344563968573,
+            0.014499487024471321,
+            0.012431277080478217,
+            -0.040103870071440256,
+            0.017328848567062113,
+            0.02723900253727976,
+            -0.054909074068976875,
+            -0.008547596270650966,
+            -0.00777964838164648,
+            0.027134666572242665,
+            0.02765036594970255,
+            -0.002207173084941375,
+            -0.016838740117653725,
+            -0.028289233739722936,
+            -0.019433376533829132,
+            -0.011053380458946764,
+            0.03247641283550457,
+            0.0007514989009289282,
+            0.007919574219000797,
+            0.001316042436488591,
+            -0.0017934342041762042,
+            0.01924145734197619,
+            0.015260199553572459,
+            0.048259602293134723,
+            -0.03383050622747848,
+            0.0021712225589426266,
+            -0.03508567715696004,
+            -0.04578731401250471,
+            -0.008768228935292693,
+            0.010203318819548585,
+            0.024135236371743855,
+            -0.007467140460133838,
+            0.0028907962385387818,
+            -0.007582083854422456,
+            -0.005384477718912037,
+            -0.01867978482547249,
+            0.01641191108381631,
+            0.037168089880543244,
+            0.0031992495663980464,
+            -0.008533890994695162,
+            0.000778843942291118,
+            -0.004620258538699103,
+            0.010572690335163528,
+            0.033233014627100904,
+            -0.030497362482602697,
+            0.02974571152880676,
+            -0.004274442492343003,
+            0.025341347125910928,
+            -0.04307748954707389,
+            -0.018069609916024403,
+            -0.018491122727274337,
+            -0.02069307802429807,
+            0.05372967226264594,
+            -0.030768978301976772,
+            -0.03662045886993282,
+            -0.003205991669059688,
+            0.016815511161004534,
+            -0.0014152644009276566,
+            -0.0016015439520090104,
+            -0.03984716970293867,
+            -0.010828260334550217,
+            -0.05894985435251326,
+            -0.01638035185618134,
+            -0.021819442096404948,
+            -0.005315102328745982,
+            -0.06251579337157112,
+            0.0033713876515840233,
+            -0.07815974854883057,
+            0.001484860071649606,
+            0.04539427129131382,
+            -0.023530943197353476,
+            0.05142250021536681,
+            -0.024746600229063383,
+            -0.0005885740455257141,
+            -0.044698536284421976,
+            0.0026481001040031785,
+            0.02389178603305878,
+            0.051387657872255674,
+            -0.002949198236157656,
+            0.010674377196609455,
+            0.013322075519401089,
+            -0.026245090721950647,
+            -0.01036724558050635,
+            -0.005321884180696542,
+            0.005958798671856245,
+            0.013755178085972732,
+            -0.0074734329537091675,
+            0.038368016046563674,
+            0.010941421833174982,
+            0.03443115781919206,
+            0.034623617136788874,
+            -0.004361926531008214,
+            -0.03348681614127998,
+            -0.00632870804659212,
+            0.024433331840566692,
+            0.0321998807702602,
+            -0.02899983280320611,
+            0.005958471722097962,
+            -0.015602447981516634,
+            0.015633217539747047,
+            0.006321931116948646,
+            -0.011827541320878442,
+            0.02372204564999124,
+            0.015673426811623124,
+            0.006103300725621424,
+            -0.005165006670545588,
+            0.013589526056189892,
+            -0.025411864292353578,
+            0.03041791947375232,
+            -0.004456865298584878,
+            -0.04276058054583262,
+            -0.0024736078733106825,
+            -0.020059309867945227,
+            0.024883721794000956,
+            0.015578237413840115,
+            -0.00536523508503184,
+            -0.005290645798641171,
+            0.008242987948511412,
+            0.020469675635821673,
+            0.006668354647245447,
+            0.00017729517736814364,
+            0.02373604785121342,
+            0.029915687122900485,
+            -0.02432635861789771,
+            -0.029540533943782886,
+            0.008151297444985344,
+            -0.005730733861939129,
+            0.011318816579801282,
+            -0.02644855229221747,
+            0.00033932872948409137,
+            0.0035505684154028996,
+            -0.03570104343808711,
+            0.03408366016010711,
+            -0.009199700348630527,
+            0.005857598530788056,
+            -0.009538177736472163,
+            -0.0017768806148210118,
+            0.0027410153099351694,
+            0.03898108970435115,
+            -0.007367623392751247,
+            0.009827254226889936,
+            0.015465895035342481,
+            0.000981604396724216,
+            0.05143083558797524,
+            -0.027350877485564314,
+            -0.02183036962301852,
+            0.01885468378168636,
+            -0.007285239007489837,
+            -0.016846837665812908,
+            -0.020546383007536074,
+            -0.011863943814684079,
+            -0.016293192750336218,
+            0.01976516340568883,
+            -0.02387328798681798,
+            -0.011892336568452852,
+            0.007170651391639504,
+            0.0013725014050721622,
+            -0.009809669068563384,
+            0.0016617183704652932,
+            -0.006307590349798603,
+            0.02037693835615797,
+            0.031753656615094446,
+            -0.024892675105483313,
+            0.03577919637821846,
+            0.03178303188315581,
+            0.017260898395251155,
+            -0.02102289690701718,
+            -0.009661035772073294,
+            -0.00048542689442823136,
+            -0.014694631967051945,
+            -0.0326379186158631,
+            0.025443206465204822,
+            0.017194626456982255,
+            -0.011381421003142453,
+            -0.03680783761649673,
+            0.0017216115811536721,
+            0.022819859205829785,
+            -0.013600989058799166,
+            0.028141657788496834,
+            -0.01869047298692401,
+            -0.01366833755869951,
+            -0.00782959271979894,
+            0.031227914441058244,
+            -0.05252392878125891,
+            0.016957842026348195,
+            0.00020880696574616416,
+            0.001154370575234767,
+            0.010892067708194324,
+            -0.008368819756628046,
+            0.012224564683325483,
+            0.031700389750709,
+            0.018063405869147564,
+            0.0010794115307202266,
+            -0.002783668216641561,
+            0.01418234057248712,
+            0.017773269840982066,
+            -0.015653443355569053,
+            -0.000497740485657824,
+            0.019890882604100452,
+            0.022672054781919238,
+            -0.00521797370595221,
+            -0.016686942830774276,
+            -0.018443538351123584,
+            0.025270057639201107,
+            -0.010354431070738671,
+            0.03484047753830186,
+            0.04594785214094975,
+            0.0036441959528283004,
+            -0.05695476763071743,
+            -0.009412209619190693,
+            0.0010371110229966086,
+            -0.01099343927053564,
+            0.01606863921935641,
+            0.012470660511773616,
+            -0.03237187923799263,
+            0.006388672213167589,
+            0.012946570716567364,
+            0.022152554488829627,
+            -0.02540317621005867,
+            -0.023183252921885226,
+            -0.0030984019855787655,
+            -0.007890467768631811,
+            0.018648760402624687,
+            -0.008653405980661855,
+            0.0006228315253506687,
+            0.009672526897422602,
+            0.0005436494014569835,
+            0.011632326145022935,
+            0.005655942007108228,
+            0.021736827172951,
+            -0.001256608167230203,
+            0.013124413886287987,
+            -0.01986808920858981,
+            -0.0270450339809516,
+            0.006500423802582728,
+            0.04438242119211055,
+            -0.014813399694814117,
+            0.030621709833336064,
+            0.0092171224268983,
+            0.016433384739642534,
+            0.024875072913970443,
+            -0.00891470467241686,
+            -0.014124009014664287,
+            0.00845777541223868,
+            0.009019007848927453,
+            -0.04147358041382793,
+            0.012513030150663913,
+            -0.01710243399313324,
+            -0.033144939719206276,
+            0.01904674924133127,
+            -0.00534122554342279,
+            0.012367320751926314,
+            0.004270315800911569,
+            -0.002886460711688153,
+            -0.025696286516581048,
+            0.014137901708875033,
+            0.028135087538950946,
+            0.021797869775098042,
+            0.0296173052503884,
+            -0.012757078368361444,
+            0.0005460601780756187,
+            -0.008748930817524899,
+            0.005470975782187157,
+            0.04187083128854477,
+            0.0008687182830616019,
+            -0.0008032802520818105,
+            0.01507726552487313,
+            0.024945744517309208,
+            0.0376130540370386,
+            -0.015415560911023553,
+            -0.03407353556457147,
+            0.01564969972264396,
+            -0.010104443707865262,
+            0.018855792875436674,
+            0.005503259932317811,
+            0.0049634097823351596,
+            0.0038600156268000702,
+            0.019312419743287703,
+            -0.006635193789316492,
+            0.018798045806429395,
+            -0.02500983683871332,
+            -0.028785757874583755,
+            0.01829411751138161,
+            -0.005277972443258226,
+            0.03338623299668019,
+            0.016225885095995672,
+            -0.029522280454919586,
+            -0.01712097344863975,
+            0.01553706187924322,
+            -0.008465766956343768,
+            -0.024842700926275538,
+            0.01895228828512998,
+            0.007076862934523977,
+            0.008956105593078965,
+            0.0024373779389833218,
+            -0.01106503696703573,
+            0.01125466547367417,
+            -0.000818223040067914,
+            -0.015322790791698395,
+            0.014157131555941659,
+            -0.017853386731281715,
+            -0.014994640991761465,
+            0.01775991780643474,
+            -0.037498078532902435,
+            0.009570197712419421,
+            -0.007687297543935123,
+            0.009050494195378254,
+            -0.004222041469820445,
+            0.007460383234429904,
+            -0.0010804379186060264,
+            -0.014987857934522812,
+            -0.014903749199810126,
+            0.03228485430019625,
+            0.022096121825475306,
+            -0.000410816936759558,
+            0.006728733918931589,
+            -3.971636461500232e-05,
+            -0.026719848032785824,
+            -0.011253751790407687,
+            0.02245467425830765,
+            -0.005664635254398481,
+            0.005134496095445943,
+            -0.01109590855846966,
+            -0.0149616794609307,
+            0.0019726017255571673,
+            0.032668637270108894,
+            0.06066635618119076,
+            0.012001732730659747,
+            0.024422093585231484,
+            0.0189499396761026,
+            0.02119276033547528,
+            -0.008489818917377516,
+            0.006413901834269739,
+            -0.03097413287789443,
+            0.03509225573986143,
+            0.005392594635384506,
+            0.03108937318306421,
+            0.026664331248813712,
+            -0.04722525921137681,
+            -0.0058539171830594774,
+            0.017087531454837677,
+            0.022673712717885323,
+            0.0159020681868943,
+            0.0008382498574190522,
+            -0.008938174268517353,
+            0.035650940044285984,
+            -0.03315250299348084,
+            -0.023647111263155,
+            0.005059382024092534,
+            -0.01439537467317451,
+            -0.010488359956731476,
+            -0.006785295580877329,
+            0.02776449681317373,
+            -0.005584052419598926,
+            -0.004582019433008779,
+            0.009697064252638535,
+            -0.014400206558190682,
+            0.008165676294736836,
+            0.0007624778531368436,
+            0.008681263897172326,
+            0.039249866981334126,
+            0.0005918063050375113,
+            0.024275699182767528,
+            -0.004138313072843555,
+            0.011079529739650594,
+            -0.0158972266904151,
+            -0.016985058543605,
+            0.02276982363172991,
+            0.00351101120496393,
+            -0.03244126813432563,
+            -0.00768819222024939,
+            0.00928180979884261,
+            -0.0013793404539641824,
+            0.005266058566444064,
+            -0.00503433670609399,
+            0.003747448261046857,
+            0.03286230666764075,
+            0.0022383592721685153,
+            0.008606190020480513,
+            0.013593947677257625,
+            0.00030951915677929204,
+            -0.01490307680740127,
+            0.013254591138113545,
+            -2.1379308136707722e-05,
+            -0.012900073369514622,
+            0.03357519186606058,
+            0.006705902094864628,
+            0.04243347200485736,
+            -0.03302953674711988,
+            0.009031534935013593,
+            0.01822227379789576,
+            -0.008491770335368761,
+            -0.02725412856390648,
+            -0.03630788717786838,
+            0.008259145747740535,
+            -0.019003260334340084,
+            0.015781275447792516,
+            -0.010039277287105153,
+            0.029665124310477384,
+            -0.03563370794521629,
+            0.004714858110268169,
+            0.004053913355055313,
+            0.0009503673086552651,
+            0.02068360255395961,
+            0.018524093113914366,
+            -0.002785698934204024,
+            -0.031443835341186485,
+            -0.02066365433385339,
+            0.006869321907853112,
+            0.02431260052207595,
+            -0.010367048093621498,
+            -0.005612149485859478,
+            0.0006316058139530722,
+            -0.024369562712568606,
+            -0.016845124984022828,
+            0.030359419843284823,
+            -0.040974427176153246,
+            0.005252911449822132,
+            0.04272936053727608,
+            -0.04599092715016919,
+            -0.03802288716943514,
+            -0.002314988728575789,
+            -0.024124014894379466,
+            -0.006360771882090754,
+            -0.002068030981218464,
+            0.016669991113175926,
+            0.0036232707799735537,
+            -0.028763764013482795,
+            0.00943401978066451,
+            -0.005173988601101547,
+            0.003971420424839974,
+            -0.0031669592494977393,
+            -0.02216336363004636,
+            -0.01657772032717298,
+            -0.02531962163186398,
+            -0.035174102848865116,
+            -0.0015903434876311088,
+            -0.005158703150821607,
+            -0.028703075279214722,
+            -0.03941940849540492,
+            -0.021696000925262537,
+            -0.0160221488026189,
+            -0.00843213135542359,
+            0.040641882367348774,
+            0.021319060666253945,
+            0.019336844555345886,
+            0.0025252167889382917,
+            -0.020346900609125973,
+            0.049440028814233025,
+            -0.007642182460707816,
+            0.035628691886108926,
+            -0.0017882888000877158,
+            -0.013461669667950893,
+            0.00455558389906464,
+            0.03825428026120858,
+            -0.014248434654950971,
+            -0.0009902062564027718,
+            0.00027811655483232545,
+            -0.0011866253757890136,
+            0.0008535155950871286,
+            0.015433650396777696,
+            0.001307651096343185,
+            -0.007112199543314309,
+            0.030380673552683906,
+            -0.012803247194895328,
+            0.01637501752192592,
+            0.018350814869962304,
+            -0.0013359719137099667,
+            0.008064546833360553,
+            -0.021430659268086814,
+            -0.00731572118780415,
+            0.0044578576331236325,
+            0.008422210247967642,
+            -0.015096885054883586,
+            -0.009964588397149773,
+            -0.02359494467299634,
+            -0.01174145797396426,
+            0.0023855749166744366,
+            -0.004559202896782688,
+            0.004902138778592298,
+            0.034595817216889944,
+            -0.016068744500427584,
+            -0.035106033294403614,
+            0.009997207702122002,
+            -0.02663088875167486,
+            0.02222306406130687,
+            3.983384942889326e-05,
+            0.006838221769859366,
+            -0.0023434342406307616,
+            0.001765445386886839,
+            -0.00169484278667461,
+            0.002862331185439823,
+            -0.018217975204395134,
+            0.0031859734246840257,
+            -0.0032119988065673565,
+            0.0009052950782312657,
+            -0.0012459653052875638,
+            0.00687339892509406,
+            0.006332799551164718,
+            -0.0008019538898592668,
+            -0.002968451740267742,
+            -8.541086355097327e-06,
+            -0.020561816931299912,
+            -0.021642512405352372,
+            -0.028259049261410905,
+            -0.000989887278779396,
+            -0.012359733017834338,
+            -0.027804394906433664,
+            0.011443385424045703,
+            -0.006872684999639775,
+            0.008327844771213648,
+            -0.008969527138034904,
+            0.006219994886600349,
+            -0.008415106130309062,
+            -0.0004143996206881767,
+            0.013119286511794273,
+            -0.02925209282917943,
+            -0.02096907906398536,
+            -0.03096513357712367,
+            0.01762223841790055,
+            -0.00925741670612824,
+            -0.002656382260603827,
+            0.0025223527960181835,
+            0.005092659166535706,
+            -0.033606542487570154,
+            -0.006653438998497412,
+            0.007313682256676618,
+            0.009547290751154595,
+            -0.03287788897629887,
+            0.02446728061355753,
+            0.017023222076526946,
+            0.01874233066565012,
+            -0.02417566312664911,
+            0.04049827067897005,
+            0.014426879325647042,
+            -0.021740606525480655,
+            0.016712907564723732,
+            0.027385150577780248,
+            0.03177163221499527,
+            -0.03469098547555545,
+            0.004481376248622418,
+            -0.01518200063132508,
+            0.018918680020516044,
+            -0.025284921244460287,
+            0.020625170445627613,
+            -0.016707629602243063,
+            0.021278719363178687,
+            0.03520334001264928,
+            -0.005524369131116028,
+            0.03608133014180745,
+            -0.019234548780647943,
+            0.0059985801736444555,
+            0.004041778461022664,
+            -0.01645503683802124,
+            0.01024694092614158,
+            -0.0042700351372843386,
+            0.0006059159967286885,
+            0.009406090340935747,
+            0.019133229156115282,
+            0.019007219188041775,
+            -0.04442208547053466,
+            0.028588965123434923,
+            0.00804120101594811,
+            0.021472086070876265,
+            0.014177820346835521,
+            -0.02708807970982664,
+            0.004742981711172693,
+            0.002072673190279628,
+            -0.004659713164250794,
+            0.0061167904730676514,
+            -0.013796604518930328,
+            -0.034540857808309226,
+            0.006338792319814448,
+            -0.018918703778187926,
+            0.015036937364271984,
+            -0.009026670935442384,
+            0.0011667525636770179,
+            -0.01152652811400482,
+            -0.011938679695598347,
+            0.0006170592505446954,
+            -0.005495151258143897,
+            -0.0008075736908949407,
+            -0.03171721592827769,
+            -0.04427658539412448,
+            0.02036132271908089,
+            -0.0021221965958423804,
+            -0.04320396743189005,
+            0.004111179168317842,
+            -0.025352428814134187,
+            -0.01865918186335911,
+            -0.011237699157960728,
+            -0.025047923887906474,
+            -0.04219630696740606,
+            -0.011108496740166623,
+            0.007785006066683012,
+            0.016810642610918525,
+            -0.009081815651299525,
+            0.040965574143584864,
+            0.0073944364393447965,
+            0.015313576815177263,
+            -0.017731360025694955,
+            0.026137777790243826,
+            -0.019314392060123,
+            -0.03108486330987073,
+            0.02676829566273746,
+            -6.259161089416372e-05,
+            0.003053748984540415,
+            0.014358356168574803,
+            2.8186924071913628e-05,
+            -0.006181789640967465,
+            0.01947730374003864,
+            -0.028739300217145797,
+            -0.023460503178221397,
+            0.0012460072421725688,
+            -0.008850395570726385,
+            -0.016271242375881602,
+            -0.035896509136022194,
+            0.005227881053956234,
+            0.03324337105109496,
+            0.004970419987430122,
+            0.008273497228454482,
+            0.01814500594735864,
+            -0.02229593571615308,
+            -0.024230675455380678,
+            0.000639112529814716,
+            -0.010266640429960994,
+            -0.010257247303470066,
+            -0.021302483296731352,
+            0.019150757066356872,
+            0.01651375496287087,
+            -0.00390189952706486,
+            -0.006971692709239087,
+            -0.010621003777733253,
+            -0.01918206725910585,
+            -0.02020345113211036,
+            -0.013827322378214754,
+            -0.017980206764311026,
+            -0.034153741666571154,
+            -0.032195001867229935,
+            0.03225205339201753,
+            0.026354817588919596,
+            0.0014076409241492998,
+            -0.007959081015805435,
+            -0.023070419006792155,
+            -0.004483328183581872,
+            0.02879516595345542,
+            0.002549934782150634,
+            -0.0001541849716285838,
+            -0.00830090759167373,
+            0.015341095409025125,
+            0.00835695019244165,
+            0.005107844984733648,
+            -0.06676974053142512,
+            -0.006027540973808174,
+            0.01655283585235268,
+            0.01351155740684989,
+            -0.002359114145814062,
+            0.02591301169926259,
+            0.01108099264396926,
+            -0.015361038608114896,
+            -0.019765743462262668,
+            0.035195761515422364,
+            0.02408910733949848,
+            0.023468767772473746,
+            -0.0034492999385544727,
+            -0.004472311490618405,
+            0.007030833258457338,
+            -0.010684252427867153,
+            -0.013326964647936429,
+            -0.004558513842504658,
+            0.026925146859337397,
+            0.016099443021514896,
+            -0.0064331198280825575,
+            0.010848931851939124,
+            -0.0008320209125061849,
+            -0.019444491995041537,
+            -0.001841330829156697,
+            0.008730275423516445,
+            0.0012974082030866743,
+            -0.014611902493400719,
+            -0.005911807768916134,
+            -0.019130604101278263,
+            0.024719468317569562,
+            0.0209456329723249,
+            0.03381727436368338,
+            -0.013237902591972769,
+            -0.0003695142331921278,
+            0.005410360755058108,
+            0.0002841632253252816,
+            0.0023352645111704958,
+            0.004801123870325242,
+            0.02623649813035263,
+            -0.029010710171522985,
+            -0.020741774066460415,
+            0.0068426980682519505,
+            0.013445270043414486,
+            0.0022744032984779294,
+            0.033840485942696216,
+            0.035040080489222324,
+            0.004166066423508316,
+            -0.010046666684988869,
+            -0.006148765564658713,
+            0.007071740742464309,
+            0.01865897335451031,
+            -0.023267411753900076,
+            -0.0005936657905496225,
+            0.004833030519903764,
+            -0.004229582648167378,
+            0.03620610659071198,
+            -0.009961238880412731,
+            -0.007197670122096852,
+            0.022212791947347127,
+            0.09819205436926473,
+            0.00047695621444943017,
+            0.0025354521225913395,
+            -0.029358210234742176,
+            -0.023614016211567963,
+            -0.0012564198366457442,
+            0.00242680708457405,
+            0.00830167274308547,
+            0.028076260517106013,
+            0.0004852400049405906,
+            0.011579274037419117,
+            0.02773972145320728,
+            -0.017217305534161797,
+            0.006752070301141707,
+            -0.011376051474975524,
+            0.024577071515205502,
+            -0.012694034026935636,
+            -0.0019809719792138673,
+            -0.016512649783050146,
+            0.015907402258634816,
+            0.004556574687833768,
+            0.004907510649166062,
+            0.011347630984408937,
+            0.009824484201337644,
+            -0.02650731900265866,
+            0.01756773113723491,
+            -0.005671986127099093,
+            0.004449611508541289,
+            -0.027841346572384462,
+            -0.020124233970502267,
+            0.0025986391930124847,
+            -0.0014440758308989809,
+            -0.009579444248630085,
+            0.006005274872617889,
+            -0.0024984235450999127,
+            0.03765664613203687,
+            -0.03765496603611941,
+            0.01409350051861866,
+            0.0066453175608372555,
+            -0.0015390841089599882,
+            0.011806653942640485,
+            0.017306914776093522,
+            -0.008206256939290151,
+            0.02589800448884443,
+            -0.00798213351353373,
+            0.009846274212000523,
+            0.015230070803055296,
+            -0.03449531102975745,
+            0.009174739184806312,
+            0.010145762060174144,
+            -0.024165660737765746,
+            0.012146877603301336,
+            -0.013784371016298891,
+            -0.004818759788054113,
+            0.005930446700613062,
+            0.007349837554764034,
+            0.010733731974181452,
+            -0.02295571464086674,
+            0.016176424826629898,
+            -0.010947841926210105,
+            0.0037766665765835445,
+            0.013559846998199008,
+            0.024892849808979323,
+            0.0012426907454129122,
+            0.0005950515136086193,
+            -0.01379469431504576,
+            0.005534724429523084,
+            -0.015667467221535825,
+            -0.022401586867694584,
+            -0.014192041775234079,
+            -0.006366846766730035,
+            0.01029598687444803,
+            -0.003972386708834858,
+            0.016971476088808737,
+            -0.010309648640697812,
+            -0.010645860303528099,
+            0.0037818310327362103,
+            0.01040806357838676,
+            0.008316605962234849,
+            -0.02855914079113881,
+            -0.005676310556150579,
+            0.004961084138163161,
+            0.00029285971503001265,
+            0.009592497954559435,
+            -0.014556073011576487,
+            0.013689526487160009,
+            -0.013290113354287095,
+            -0.0010726537498214612,
+            0.013201860397060236,
+            -0.012443552043930505,
+            -0.004012305744087929,
+            -0.01592905426965635,
+            -0.028487066798154617,
+            0.008462143660968396,
+            -0.01365614159523485,
+            0.001076882584445294,
+            0.0019503240125505646,
+            0.004730085646042709,
+            -0.016919069570642004,
+            -0.007860046623166481,
+            -0.033515448866521046,
+            0.011065416478687504,
+            0.02314589687882036,
+            -0.006435733825418645,
+            -0.030351727858789472,
+            -0.009546333526852862,
+            0.011327489786114963,
+            -0.0071765122743172715,
+            -0.01676736383051139,
+            -0.0029390038145944096,
+            0.007617910650439855,
+            -0.008174620754097493,
+            -0.00508943241317381,
+            -0.0025173969010414098,
+            0.0217328037301181,
+            0.016094975605203832,
+            0.01769642142712967,
+            -0.021400862603034036,
+            -0.00976992221747187,
+            0.026043879119776864,
+            -0.03275550610951898,
+            0.012441465975914529,
+            0.024587446266013183,
+            0.003446527051428522,
+            0.005492228054838714,
+            0.0030377554109693714,
+            0.005459159163772825,
+            -0.006985179226831048,
+            0.019925583798328282,
+            0.010870421035240845,
+            -0.019709062313471467,
+            0.005385443078244229,
+            0.02086085539375208,
+            0.0037248737640193097,
+            0.012892299156840998,
+            -0.017546870090541213,
+            -0.0020341807891543716,
+            0.015594779287017958,
+            0.04816214672670215,
+            -0.0032851797178091685,
+            -0.010450846098750195,
+            0.029513742590677183,
+            0.023526004023637855,
+            0.005373455940309873,
+            0.0012120777808635214,
+            -0.02525630456877349,
+            -0.013075130701774785,
+            0.01962235086017781,
+            0.016944870494393874,
+            0.013484925533584295,
+            -0.0023255656480479004,
+            -0.019236081983457366,
+            0.021520156296467717,
+            -0.01701733319731161,
+            0.012422659010697477,
+            -0.023764941037007197,
+            -0.001490790833263327,
+            -0.0035419008954878724,
+            0.021998949056452454,
+            0.0021819635638621092,
+            0.008891505304514552,
+            0.024354552127458608,
+            -0.006978548952551799,
+            -0.019526615985982355,
+            -0.0051799020400150305,
+            -0.014578953088901798,
+            0.022506635912517146,
+            -0.024117708614271965,
+            -0.007008641019892289,
+            0.011379795968360019,
+            -0.012362093847309038,
+            -0.0102534051092702,
+            0.001164934138876502,
+            0.017686807394319725,
+            0.0009423993435129515,
+            -0.014784508280754078,
+            0.01944445593807992,
+            -0.0011359271135469456,
+            0.010410915597949927,
+            -0.01921419010977189,
+            0.009390638453648025,
+            -0.016682442516992777,
+            -0.007849184048209236,
+            0.010343008873053859,
+            0.0012604521199837046,
+            0.003948300703645506,
+            0.002592490121754231,
+            0.007773437307969315,
+            -0.03355658426684504,
+            0.022695505487314516,
+            -0.010751606023585792,
+            -0.003538523688786008,
+            -0.0014125330167969218,
+            -0.01678692701672328,
+            -0.0006918162525881059,
+            0.005816502852598842,
+            0.010878358093847386,
+            -0.016045147059003256,
+            -0.00873270513444376,
+            -0.04920244395446931,
+            -0.030314009649638693,
+            0.023156333024481057,
+            0.013733657599720879,
+            -0.010971506342695331,
+            0.018852159415605117,
+            -0.023760853159532587,
+            -0.010587249404421732,
+            0.0011379869499042905,
+            0.023319856329056813,
+            -0.03539290744576506,
+            0.01215369703816749,
+            -0.014493682360886549,
+            -0.02039288669534806,
+            0.009070984596830142,
+            0.011445354735283254,
+            0.0007259852045513916,
+            -0.0011514627555568814,
+            -0.029350945123925453,
+            -0.010246845821727087,
+            0.02391068177425878,
+            -0.00861718589055971,
+            -0.004723044986826333,
+            0.024716520302409263,
+            0.0033721801437248873,
+            -0.012524663490691517,
+            -0.017955758068233754,
+            0.021612671655463236,
+            -0.026077203994996058,
+            0.007645944309444457,
+            0.010032183011253845,
+            -0.0173991263874487,
+            0.021649599230588117,
+            -0.007529701797485131,
+            0.020327763440441377,
+            0.004227390938169872,
+            -0.011249233530251522,
+            0.020352283974471443,
+            0.013644443758970054,
+            0.004216958800094986,
+            -0.004081669404784771,
+            0.006869353098014179,
+            0.02212732880780841,
+            -0.011021739614266063,
+            0.02219580717345023,
+            0.028777213295471067,
+            0.00017328191188631918,
+            0.03260179554743322,
+            -0.021287655520443285,
+            -0.03159728983004563,
+            0.021740163175540124,
+            -0.01035146858119169,
+            -0.021564906188712388,
+            0.028914245422221707,
+            -0.013674731836508768,
+            -0.00786476052825481,
+            -0.0166718429993922,
+            0.00533027311342322,
+            0.016680336311088843,
+            -0.005246190451159961,
+            -0.014360934066628332,
+            0.0029628103834216345,
+            0.019809487418628386,
+            0.014587234544007958,
+            0.029780009648869975,
+            -0.010778199815920887,
+            0.035180242185168895,
+            -0.002320576872571577,
+            0.008990157317473909,
+            0.012415465232073404,
+            0.006489435152972723,
+            -0.011251531176923445,
+            0.013501894212117511,
+            -0.003446759011266526,
+            0.008230548711588447,
+            -0.001449023474280409,
+            -0.035111679577202855,
+            -0.013709382199848678,
+            -0.04709294504969336,
+            -0.0023752976043235697,
+            -0.0025098360125255215,
+            0.0030975417740913154,
+            0.09736815869389318,
+            -0.005723049647039619,
+            -0.011988817571623457,
+            0.001625059666070763,
+            0.009611697638805556,
+            -0.0009896494420056814,
+            -0.007606836952841873,
+            0.015973152482488318,
+            -0.01343292635775299,
+            -0.02140928016038583,
+            -0.010532701190806147,
+            -0.01288594454368977,
+            0.012079198190522715,
+            0.015929716343161652,
+            -0.0005947359502228821,
+            -0.015207331663761012,
+            0.006524010655124294,
+            -0.004566385899752684,
+            0.0064717704997400245,
+            0.008647099975176308,
+            -0.010414668624343924,
+            -0.0071698879102925555,
+            0.030899143952119196,
+            -0.025301269549419674,
+            -0.024361748354052457,
+            -0.0010579931222861095,
+            -0.009055796737486008,
+            -0.010335459737012047,
+            -0.002692653633719985,
+            -0.0017679501322462056,
+            -0.010519994985692081,
+            -0.010019865263564572,
+            0.006537076627706128,
+            -0.015472130358273033,
+            0.004812857776601667,
+            -0.0047457011269495634,
+            -0.013296069496526007,
+            0.017054968086538724,
+            0.022563785357424853,
+            -0.02841491200807702,
+            0.0006755428946168761,
+            -0.017422510783648978,
+            0.012941211381006863,
+            0.008866069010589792,
+            0.006049618782357251,
+            -0.022451528209097505,
+            -0.023622906582403243,
+            -0.015019272754405791,
+            -0.01927535750568903,
+            -0.024973490129133732,
+            -0.009073381957684092,
+            0.0067692803507600945,
+            -0.001267339788153158,
+            0.021918281111033645,
+            0.000816538714279555,
+            -0.01812545613159305,
+            0.0061084913570894905,
+            0.000568371220952269,
+            0.018292497018668842,
+            -0.027468677288882186,
+            0.0033806422195878524,
+            -0.02307582511848465,
+            0.033508234091571124,
+            0.0039195050746594975,
+            -0.026680111514794824,
+            -0.021558668725755275,
+            -0.003311830287529699,
+            0.018994577319066473,
+            -0.0025745912019675805,
+            -0.00724049447539589,
+            0.014775207617625293,
+            0.0155334760323838,
+            -0.004693118176196483,
+            0.002015625406296403,
+            0.027968079002437102,
+            0.01077998866180723,
+            0.0313892866933904,
+            -0.015236169297590675,
+            0.004945828495617267,
+            -0.027215016713379062,
+            0.010214419950212732,
+            -0.01503548137805401,
+            0.008859554812646242,
+            -0.01290890682557532,
+            0.03531384210182567,
+            -0.012005613861257432,
+            -0.004250698950192301,
+            0.03253288796441852,
+            -0.029150986393345252,
+            -0.0035020051536906334,
+            -0.009255437891304041,
+            -0.004142204892160768,
+            0.013240580426083011,
+            -0.001198093874819309,
+            0.008485141356876356,
+            -0.01854370462655459,
+            9.726096864073609e-05,
+            -0.003221852304410318,
+            0.005404595442127598,
+            -0.02005458688172967,
+            -0.003314118809778734,
+            0.0073427568419969814,
+            0.004259518237708634,
+            0.002408630356035373,
+            0.016420646142725346,
+            0.006662121128241274,
+            -0.0029357701493526907,
+            0.03424224243607692,
+            0.008171895850772245,
+            0.0152521424877615,
+            -0.0018638264193202728,
+            0.0030234211261443034,
+            0.019287963516760636,
+            0.019912956661192856,
+            -0.00048443399729935116,
+            -0.01058935153883094,
+            0.00195113175754497,
+            -0.001140732137433033,
+            0.011048741602327437,
+            0.024396046054912645,
+            -0.021158025221644738,
+            -0.02419562092713724,
+            0.016377661005279716,
+            -0.015076145951740961,
+            0.01903712233378994,
+            0.010541463112889985,
+            -0.010099351302095792,
+            -0.00031558616108876247,
+            -0.042316016201379245,
+            0.010964989031040692,
+            -0.02068939391415103,
+            -0.04004120028915868,
+            -0.00244383573783919,
+            0.0009542507389088937,
+            -0.029439456243505052,
+            0.02822814855547978,
+            0.0023755032182188156,
+            -0.017280221740813364,
+            0.018223709137876153,
+            -0.005548654680726505,
+            -0.0024040301239229064,
+            0.005397711787238679,
+            -0.01092026090471121,
+            -0.010648485086769927,
+            0.008235250252660972,
+            0.010957715075632535,
+            0.005425683656535705,
+            -0.005698919020608095,
+            -0.011284911899894062,
+            0.009975882651638084,
+            0.012318901214538077,
+            -0.0019922884373083155,
+            0.010050520101815593,
+            0.0031720752428247446,
+            0.007352859750323051,
+            -0.0073483049386141005,
+            0.0046483587496626965,
+            -0.005093220128411125,
+            -0.026080057085901446,
+            0.012863580653903103,
+            0.0052434400313520774,
+            -0.011863360599637747,
+            0.0103209502958491,
+            -0.01048162756158923,
+            0.005787157766911021,
+            0.001318945460232246,
+            -0.0059333230422061395,
+            0.004821320238893602,
+            -0.0038922239931512784,
+            0.011267634571983924,
+            0.013256611037442042,
+            0.004592529900805719,
+            -0.013076954459595086,
+            -0.02177318635989856,
+            0.009001291224623467,
+            0.01959767163356408,
+            0.024161754192119852,
+            -0.030416109293488064,
+            0.003784634588325551,
+            -0.0014915849060829743,
+            -0.0020550564975080885,
+            -0.015632785947151753,
+            0.01587790981601866,
+            0.01716742056401084,
+            0.012721563202528083,
+            0.01893393006967701,
+            -0.004009617703771589,
+            0.010539100665843704,
+            -0.02080068888791658,
+            0.013642451766699067,
+            0.0008609869904066794,
+            -0.013629067231663643,
+            0.0014620655123815298,
+            0.027046304278766087,
+            -0.011056783825898616,
+            0.006206969699250696,
+            0.008097527175511791,
+            0.022111870736907183,
+            -0.012509479100313492,
+            -0.003735868296238069,
+            0.01516254968167132,
+            0.0014878674796313539,
+            0.0035887079417890352,
+            0.016533149527222273,
+            0.04729052276110659,
+            0.00102751186013391,
+            0.0016713773449738153,
+            0.0192177920520243,
+            -0.005563519063161466,
+            -0.01114821419109643,
+            -0.003549999863325982,
+            -0.021807893906469106,
+            0.010998558936481785,
+            0.00431279785115251,
+            -0.02186324212032519,
+            0.01024093366826206,
+            0.0125495473550584,
+            0.005268320135120931,
+            -0.0031679212410026603,
+            0.02574575247827441,
+            -0.019807879421996258,
+            0.014935295894965128,
+            0.03329450628384787,
+            0.00493356778223407,
+            -0.011423949333937285,
+            0.007263264249871384,
+            0.0027354459678742013,
+            -0.004494406041906684,
+            -0.00606745413660583,
+            0.010497391290734088,
+            -0.008191702034391382,
+            0.009870458191510176,
+            -0.009580130479693,
+            0.039202091799313755,
+            -0.009619018331262844,
+            -0.010782138248435458,
+            -0.004558645539551881,
+            -0.006241773108460662,
+            0.003478414182938609,
+            -0.003395165694642073,
+            0.005812225135633631,
+            -0.0069045407353789945,
+            -0.014607146220271209,
+            0.0008522982757210454,
+            0.00020175973815948303,
+            0.014001922919116725,
+            0.0011804904415274088,
+            -0.004871874532755353,
+            -0.007877217154706205,
+            0.015243277056007988,
+            -0.006047906208304912,
+            0.0006531937033317192,
+            -0.0011653523616313422,
+            -0.0013662556399359335,
+            -0.007401050012179761,
+            -0.015166105795085528,
+            0.00974609244984173,
+            0.016253733908242134,
+            -0.013857834499211056,
+            -0.005142936208363415,
+            0.012274716212832219,
+            0.008540229125193836,
+            0.011742606168187876,
+            0.0007451338650642946,
+            -0.009255813682572937,
+            0.008292531177374192,
+            -0.015020523035432158,
+            -0.008781317965985481,
+            0.0026250469418136603,
+            0.002381147382977664,
+            0.005839723981046825,
+            0.0006240641828668715,
+            0.004502463508605272,
+            0.005798912705218846,
+            -0.00497936644919367,
+            0.01112666823857836,
+            0.008198424171006548,
+            -0.002452143482105151,
+            -0.01017569410837814,
+            -0.001127589984272638,
+            -0.015733298167661346,
+            -0.016758315411028353,
+            -0.0009286266498936771,
+            0.015175988113750882,
+            -0.02352862898718829,
+            0.018996352975391644,
+            -0.0014524363611708201,
+            -0.0033802546171857704,
+            -0.021392428456030312,
+            -0.0025923983911570183,
+            0.0034214188668518796,
+            0.01255768230394476,
+            0.01858853630745928,
+            -0.02729543519465384,
+            0.010575894388563116,
+            -0.007204725245442141,
+            0.030394929424014717,
+            -0.018328782140005044,
+            -0.004022941273610227,
+            0.026915618079621992,
+            -0.02200018306828758,
+            0.0008791185377513978,
+            -0.011454606955158956,
+            -0.008103032693853504,
+            0.019287674837688177,
+            0.023747720480736777,
+            0.02038287242167783,
+            0.01364489294869211,
+            0.0007758044919789085,
+            0.004768030222445397,
+            0.03572705648994403,
+            0.010568241270862248,
+            0.014828087092859878,
+            -0.007542489523110974,
+            9.457808640264811e-05,
+            -0.02554033312739386,
+            0.01635899434369978,
+            2.587383534566272e-05,
+            -0.012370727506722627,
+            0.007162422186621379,
+            -0.013370882403931942,
+            -0.011066483600541058,
+            -0.007151077450730025,
+            -0.01721910018182534,
+            0.02152239849400241,
+            -0.012322440280680164,
+            0.004587729883596212,
+            -0.009079747696669277,
+            -0.013272707286450417,
+            -0.012953682633400515,
+            -0.014202214807187545,
+            -0.01313141988034687,
+            -0.026041403779538307,
+            -0.027109731260915763,
+            -0.006573285295895077,
+            -0.023780366346334524,
+            -0.013799072648505526,
+            0.005588622594136253,
+            -0.022617902666316757,
+            0.012262325666428604,
+            -0.012689827812482627,
+            -0.01226264872059864,
+            0.012805734807835008,
+            0.020274844315616616,
+            -0.011765231803328348,
+            -0.028494489703400308,
+            -0.018583843821043854,
+            -0.019970998470704358,
+            0.012576201137505969,
+            0.002266288538944273,
+            0.013165889075444961,
+            -0.003018437854660793,
+            0.003227412667880239,
+            -0.014895831590298957,
+            0.018570624675927183,
+            -0.03442390703780963,
+            -0.03108003806963536,
+            -0.013611306765348361,
+            0.005182777066476028,
+            -0.0015782033203578553,
+            -0.011429046504256945,
+            -0.0018606648572920801,
+            -0.00433805271182878,
+            -0.025109572024446088,
+            -0.010134856632190067,
+            0.011010662054484161,
+            -0.015205629927469414,
+            0.0072933565255675036,
+            0.0009018832564038863,
+            0.013870902834260887,
+            0.009587432944616442,
+            0.013393301893492882,
+            -0.006095063698384775,
+            0.004356485794113208,
+            -0.0026476612747324386,
+            0.0004894024149759108,
+            0.009336366919474478,
+            0.00839591561027103,
+            0.010551161369079399,
+            0.002260707777260338,
+            0.007830059223001696,
+            -0.0060927446216633265,
+            0.018065002687169148,
+            0.015436944875538481,
+            -0.0002520689442496089,
+            0.013323098485251487,
+            0.0003769862591044255,
+            0.00022293304921127018,
+            -0.0029970303106340466,
+            0.007888437018607987,
+            -0.011785957933642163,
+            -0.001311704406355858,
+            0.015138190893483804,
+            -0.01707247730569978,
+            -0.013045187198628258,
+            0.03107034781283638,
+            0.0018583638488268697,
+            -0.02495564203304796,
+            0.031146554085787762,
+            0.004220047610942943,
+            -0.005835199594223787,
+            -0.013205086624848782,
+            0.0014059047822222418,
+            -0.011570624844100298,
+            0.003584232809362274,
+            0.014295032061446826,
+            -0.02231984223610487,
+            0.010521094801839404,
+            0.008446892441796151,
+            -0.009681535919980559,
+            -0.012337659795586223,
+            0.009847073709209602,
+            0.016809328445200358,
+            0.004472485504034416,
+            -0.03539093810210523,
+            -0.011957694102753411,
+            0.02145470732008716,
+            -0.029237428225009913,
+            -0.00810041956259673,
+            0.007452962400105787,
+            0.0065154832040426065,
+            -0.019353109017609053,
+            -0.008580955020412362,
+            0.005725794605897509,
+            -0.004940390374637084,
+            -0.003892210172017758,
+            0.017541623128534682,
+            0.00485047043626186,
+            0.013698341792101025,
+            -0.01689738578307474,
+            0.009771458057400557,
+            -0.022222817828602557,
+            0.004021882074676753,
+            -0.023373700500461992,
+            -0.009873403167702603,
+            0.002364143609079849,
+            0.012454271200139762,
+            0.014328556467632212,
+            0.027839363143659326,
+            0.030056404238348527,
+            0.013995696971422429,
+            0.007833677585250953,
+            -0.016265199386953717,
+            -0.018134327237616175,
+            -0.006409513840075933,
+            -0.004033173350204784,
+            -0.008714798059218596,
+            0.0066373547199279235,
+            0.01858354212009877,
+            0.004499712802472377,
+            0.02609078206670711,
+            -0.006227189596272876,
+            0.009039744622014585,
+            -3.127526911951914e-05,
+            0.0008833930131827993,
+            -0.004226083827636182,
+            -0.01971685546584527,
+            0.00481003211084742,
+            -0.0013721482162825508,
+            0.025874758880591167,
+            0.00940496860916206,
+            0.0024915013399541847,
+            0.01615978314811047,
+            -0.006378952757397484,
+            -0.01174653607178726,
+            -0.010540107733238588,
+            -0.0061926268398085176,
+            -0.021752582127981104,
+            0.029059100503124387,
+            0.021195595568253612,
+            -0.010800765264123782,
+            -0.013693582166598082,
+            -0.00903369526059487,
+            0.02795767990556847,
+            0.004080965688836953,
+            -0.0013050212744006933,
+            0.016313311850308505,
+            0.012084555958999215,
+            0.012329325712491252,
+            0.009794698207689061,
+            -0.00036157148861933123,
+            0.0015364709122308352,
+            0.03722522141440073,
+            -0.007427028510548186,
+            0.0026463292682538495,
+            -0.005239324411651515,
+            0.0025987932868461313,
+            -0.009705817520422487,
+            0.007242122655628283,
+            -0.004012676385296197,
+            -0.0158760051470016,
+            0.015003444174308607,
+            -0.022443331305169994,
+            -0.0016749787204471238,
+            -0.008150635190952799,
+            0.0016696640640947303,
+            -0.02781023337840043,
+            -0.0025731206607371845,
+            0.006695065701630532,
+            -0.015547340828489069,
+            0.011816010501093885,
+            -0.01740673229051557,
+            -0.011553604323149105,
+            0.007914961618894248,
+            -0.012034157288065447,
+            0.0016684130271324446,
+            -0.0008268957621923278,
+            0.004460864332388434,
+            0.013426055079012884,
+            -0.011008808570221245,
+            0.012166046631826273,
+            -0.005816284310138776,
+            -0.007872091214834688,
+            -0.0016055129487028885,
+            -0.0030049472973942043,
+            -0.0025600746761171543,
+            -0.0009966231170649917,
+            0.034355910234323155,
+            -0.004446570896048543,
+            0.003761242147499512,
+            -0.01362639743248326,
+            0.0019833052855181806,
+            -0.036176376394999044,
+            0.028344464817309477,
+            -0.008642835889481802,
+            0.011013502221008498,
+            -0.0017508365480865596,
+            -0.013243706268954915,
+            -0.01456517309747086,
+            0.015247299501245178,
+            0.007602596572394442,
+            -0.005854531057349665,
+            0.013558248087347848,
+            -0.019724373614448298,
+            0.008530500822910953,
+            0.004463694543554244,
+            -0.015561397976987216,
+            0.004668780190328835,
+            -0.007479912658259757,
+            0.012657201279639194,
+            -0.01774730033362932,
+            0.004314594593150765,
+            0.00958571617187387,
+            -0.01892876084898881,
+            0.009901041800918215,
+            -0.028675879564512306,
+            -0.004631920242208613,
+            -0.012323455524366223,
+            0.0053804608827285865,
+            -0.000660864620708182,
+            -0.02389088252276507,
+            -0.028944280660230762,
+            -0.0002099667627277945,
+            0.00458533068247518,
+            0.0018878351997997833,
+            0.011621407953503755,
+            0.005970807015151215,
+            0.008542958629199003,
+            0.01023896054756221,
+            -0.00205207247588841,
+            0.009402752124876478,
+            0.04169706217414899,
+            -0.022899912321317197,
+            -0.011574970753549647,
+            0.005321872977369877,
+            0.011499271505460519,
+            -0.02647653483798284,
+            -0.011039791758347888,
+            0.0033424345263339144,
+            -0.005219877979550397,
+            0.03609469691834737,
+            -0.02448746405881972,
+            0.019020618652020673,
+            -0.011392297116547224,
+            0.007625475937968714,
+            -0.021465240076078376,
+            -0.003921088264697311,
+            -0.014265745991522477,
+            -0.00011221615798127438,
+            -0.0031478236827989703,
+            -0.0076053733863387755,
+            0.007485039021296698,
+            -0.001907678661303243,
+            0.0020348246022161925,
+            0.006291436821831794,
+            -0.02173305455485444,
+            -0.0019125499969493854,
+            -0.016432460924913166,
+            0.030367263349759286,
+            -0.016242095671439496,
+            0.005072805486919414,
+            0.0053258139728101506,
+            -0.002810639419737258,
+            0.012368771814660942,
+            0.004165967191536186,
+            0.00513754440091394,
+            -0.004199767241827414,
+            -0.005142642360546972,
+            -0.0053886448467479215,
+            0.017868874549760376,
+            -0.013280048598849825,
+            0.035024701466710065,
+            -0.0100959186045929,
+            -0.007094001877261568,
+            0.0019733042325232155,
+            0.010380430710282744,
+            4.2084202709296443e-05,
+            0.01053565670076933,
+            0.0034270489770133685,
+            -0.0006311078385359693,
+            0.0019774779328212756,
+            -0.017162448473652843,
+            0.003089758682621882,
+            0.01682656561693936,
+            0.005463677441664502,
+            0.004469261499451007,
+            -0.013332130891003062,
+            -0.012050233723282738,
+            0.004538268704442613,
+            -0.007533940408255447,
+            0.0003675737140813486,
+            -0.018816953644773076,
+            0.013972696748575243,
+            -0.0011096683686631858,
+            0.0002715683052236706,
+            0.00524678369365015,
+            0.023056514997908218,
+            0.01813167860994747,
+            -0.002553352602551003,
+            0.03155164978556312,
+            0.0174738879054593,
+            0.0029268039949260673,
+            0.010637295711777793,
+            -0.033222781195984666,
+            -0.013301873512972214,
+            -0.0013721703891642354,
+            0.02238421933149517,
+            -0.01530189703335162,
+            -0.014059029663525449,
+            -0.0037393534403749947,
+            0.0010394601933309767,
+            -0.005177359933828505,
+            0.0042489306011644495,
+            0.011039889511148341,
+            -0.014697595240068863,
+            0.007597149702441915,
+            0.0026780496822907634,
+            0.02092734730183164,
+            0.0004688144432243786,
+            0.017181410417231698,
+            -0.0005083970646221274,
+            0.00749639055342427,
+            0.007319420826337891,
+            0.017541463125355265,
+            -0.01002811389296805,
+            0.023582113845081534,
+            -0.008823933397729605,
+            0.0019889258902730535,
+            -0.01257106158422629,
+            -0.028159594067544737,
+            -0.012624750220149823,
+            0.015247564545279736,
+            0.005765911899842925,
+            0.006478299226263732,
+            -0.016537090297144265,
+            0.006276997440048981,
+            0.015647122171767974,
+            -0.002251683151059454,
+            0.010295358869874306,
+            0.009730378299068289,
+            0.00012816405890877098,
+            0.010181120688239766,
+            -0.00801256937783534,
+            -0.00734257767723293,
+            -0.004497520124843868,
+            0.023982754195729983,
+            -0.005443550092842393,
+            0.022434177122972028,
+            0.012647958478747348,
+            0.00616791857573897,
+            -0.010853355357954727,
+            -0.003340111254357573,
+            0.023389634440057473,
+            0.011863246790550376,
+            0.00029642699874398923,
+            -0.009846034060495678,
+            0.0019458876579524617,
+            0.0054010373765486005,
+            0.012983913038716568,
+            -0.02049427270240938,
+            -0.004344571352918878,
+            0.008461947538684664,
+            -0.008472919299762712,
+            0.004318309005722874,
+            -0.030156444522730416,
+            -0.005812303237222271,
+            0.021354352236942434,
+            0.008392281015122372,
+            -0.010232039722494646,
+            -0.004395682831498155,
+            -0.0006527587802367882,
+            0.0040601850269714945,
+            -0.0024323167053212176,
+            -0.0023729138545912176,
+            -0.008541233856441836,
+            -0.006470933419694742,
+            0.014035696617797647,
+            -0.0052691760528871924,
+            0.008597459268477037,
+            0.00012084859681905111,
+            -0.0017255958688825231,
+            -0.007323529232751411,
+            0.03382792341957058,
+            0.008087732655845718,
+            0.005980666974986827,
+            0.0013613625176327891,
+            0.0042244330945601215,
+            0.000616791811672998,
+            -0.0028374856646453707,
+            0.018075360591135542,
+            0.0156738939725117,
+            0.005054696911814988,
+            0.009038525980341447,
+            0.014672270369917403,
+            -0.019700405219230957,
+            0.012625337070556061,
+            0.012184185186281482,
+            0.019382321278525114,
+            0.0007219498737851579,
+            0.002564970959687783,
+            -0.010601873823737926,
+            -0.0029963537053764736,
+            -0.002070567089566093,
+            0.029563028293936092,
+            0.0005903976560069006,
+            -0.010509758568838935,
+            0.010992421945232951,
+            0.020603868018231348,
+            -0.02101157589451638,
+            -0.001971974816402967,
+            0.005963911670920596,
+            -0.010704207628592966,
+            -0.014466113540150018,
+            -0.021269176952661614,
+            -0.013331247081394942,
+            0.0071597345125287174,
+            -0.008162102923896894,
+            0.02367549407145414,
+            -0.014256771777502395,
+            -0.03960796112042958,
+            -0.014803626428839152,
+            -0.00864320347698291,
+            -0.027531337207277483,
+            -0.007305441146142331,
+            0.00965056093889974,
+            0.016698589600806273,
+            -0.003330867073501157,
+            -0.025198703196464594,
+            0.030483790406960656,
+            0.008872821868559751,
+            0.023708422767352917,
+            -0.021464374508407488,
+            -0.0018617153194555344,
+            -0.00886376784514331,
+            0.014505918224188001,
+            -0.017588761818663975,
+            -0.011756112651591795,
+            0.020014632992762026,
+            0.014356419813769714,
+            0.007246557500044434,
+            -0.004355132555323843,
+            0.0067299398462013936,
+            0.007649499235136303,
+            -0.012515181210363267,
+            0.012239792251154642,
+            -0.013499277313916029,
+            -0.024858295126228433,
+            0.014430561986516884,
+            0.002229494935179828,
+            0.009563004428064055,
+            -0.0015984171085659323,
+            -0.016197659608768088,
+            0.010078159490026602,
+            0.00808729175515505,
+            -0.017508840440512225,
+            -0.004793934415922065,
+            0.007133527615305415,
+            -0.020818869260580315,
+            0.031234686259160864,
+            0.011872765152279094,
+            -0.010692949770503467,
+            -0.001989039629661815,
+            0.004431079028880923,
+            0.016511544608975148,
+            0.007893700385963683,
+            -0.01997314388229276,
+            -0.003708006454566117,
+            0.01530128306203401,
+            0.00822588305477249,
+            0.004861703074728471,
+            -0.010836328783594584,
+            0.03434874763691911,
+            0.006082873924472328,
+            4.192481378506604e-05,
+            -0.0052997095110967115,
+            -0.0076108924981838355,
+            0.004196112457289409,
+            0.018905391806961686,
+            0.013330078288599763,
+            -0.004200590783114919,
+            -0.006630107527854746,
+            0.007778448128358027,
+            -0.01706991160178345,
+            0.007329056462125937,
+            0.009390327499494826,
+            -0.012870674959022891,
+            -0.0017571189976160295,
+            0.016462608870101713,
+            0.00209846661139665,
+            0.004942714850185376,
+            -0.011305050739130339,
+            0.004252776146827917,
+            0.015841821944109317,
+            0.018727567181082646,
+            -0.008399642221255733,
+            0.021448055799259867,
+            -0.020972459569228434,
+            0.027243639465535856,
+            0.028499400850980086,
+            0.016573144403275928,
+            -0.01847096443459798,
+            -0.0022409373140116146,
+            -0.0041737366139895414,
+            -0.014761716560267328,
+            0.0034546020161337894,
+            0.005125870074458691,
+            -0.01174186608185822,
+            -0.009577921232380895,
+            -0.024331993794416026,
+            0.011075959943215117,
+            0.007918147902741738,
+            -0.0015045185909419564,
+            -0.011791155919533844,
+            0.016019077577552835,
+            0.010141868950981925,
+            -0.010938402325619172,
+            -0.005635351352447256,
+            -0.019967361901734836,
+            0.009750427567651833,
+            0.009128294017175602,
+            -0.00300556760891084,
+            -0.011496103417643132,
+            0.012977890524465716,
+            0.0014080059843484115,
+            -0.01292380658817521,
+            0.010841954636959441,
+            -0.03142011651894961,
+            -0.005510537354374977,
+            -0.011027026734432255,
+            -0.003620482919145167,
+            -0.014415606143566519,
+            0.003608572766363589,
+            0.0005034611321129689,
+            -0.011470488351158329,
+            -0.0035005774696278,
+            -0.016207453351921627,
+            0.004676327282262429,
+            0.015021513761697039,
+            0.001679614645967629,
+            -0.006917375094634678,
+            0.021994274901326064,
+            0.018518524483306403,
+            -0.021903538417919984,
+            -0.020116013379669034,
+            0.004708084983787958,
+            -0.03261690161800251,
+            0.015019348187107588,
+            0.00966124471577622,
+            0.011676937880751252,
+            -0.028768229226408037,
+            0.007114534979415504,
+            0.00900420165933673,
+            0.016398927481511824,
+            0.0050879301160439245,
+            0.009453255486450082,
+            0.01222758288108498,
+            -0.0104657614414275,
+            -0.003164502778525433,
+            -5.410382753651127e-06,
+            -0.005618225986727351,
+            0.015607620507415484,
+            0.0008527697935056332,
+            0.03603666488542239,
+            -0.005277207843139309,
+            -0.0009799879252482288,
+            -0.011735731235560851,
+            0.00812046801749312,
+            -0.011713349784345638,
+            0.006334172347602515,
+            -0.0024643499727789167,
+            0.0001240009282616789,
+            0.002965067818077541,
+            0.009442572075213767,
+            -0.01086095783353607,
+            -0.007595379345224076,
+            0.007444897256325813,
+            -0.01482814340478338,
+            -0.00019984717819268013,
+            -0.009339343730980451,
+            0.0019680804106275156,
+            -0.02241031895876346,
+            0.006629838013824249,
+            0.040935644507527846,
+            0.02140216465404127,
+            -0.005481040712269146,
+            -0.012431659209559418,
+            -0.004680230194268222,
+            -0.007229300034633229,
+            -0.006482401275991039,
+            -0.004794698160419071,
+            -0.022762405726589375,
+            -0.006928553505112732,
+            0.004320321244630314,
+            0.013300138724212552,
+            0.009741018117644266,
+            -0.008842135469729511,
+            0.004531760574809748,
+            -0.02662650543210291,
+            -0.016578081411548973,
+            -0.0060021425889846495,
+            -0.015840130031005147,
+            0.006125301421471048,
+            0.027521841635711787,
+            -0.0063432924325983525,
+            0.01789356321527744,
+            0.005142029893259461,
+            0.013034833065163448,
+            -0.005338869351330258,
+            0.00035250243782976347,
+            -0.006010938630154189,
+            0.0032958656042048292,
+            0.003074861351445954,
+            -0.0015947217852680714,
+            0.004151590621941346,
+            -0.007139285822575166,
+            -0.0026091937868967284,
+            0.0011629380311555433,
+            0.0117841884590861,
+            0.006877419488576261,
+            -0.00016252653621991807,
+            -0.01761285866765538,
+            0.022221104524808997,
+            -0.018933508559881868,
+            -0.008840161102111973,
+            0.011092684029552064,
+            -0.009660487856185668,
+            0.00702597023411946,
+            0.020387386977378873,
+            -0.006854213562767843,
+            -0.009583981919303215,
+            0.013028181087649146,
+            -0.0036507672754997286,
+            0.004464368543493108,
+            0.009867428978560914,
+            0.006036303973414791,
+            0.002482944923100156,
+            0.013244650094892797,
+            0.004530294330688855,
+            -0.0032625482538717137,
+            -0.020238533752799943,
+            -0.005186498720507101,
+            0.019015752262933585,
+            -0.02252458872825473,
+            -0.0027641215988942563,
+            0.0044506144202026456,
+            -0.0030317840657635457,
+            -0.01733962766050954,
+            0.004205861489436576,
+            -0.018770275720867473,
+            -0.0033883519277721433,
+            -0.008244668812709337,
+            0.009948090119054079,
+            -0.023700732627507225,
+            0.009109315916173436,
+            -0.011739192106106307,
+            0.02356509175177699,
+            -0.003925851077921686,
+            0.024734433836185024,
+            0.0015142645677087268,
+            -0.013482933030628261,
+            0.0055323884713736554,
+            -0.0008930661843169134,
+            -0.004477731568313757,
+            0.029338490404300612,
+            -0.005418258701133154,
+            0.002249996261871466,
+            0.013491124734151694,
+            -0.021332541202417246,
+            0.021773687217263173,
+            0.0046662696076723836,
+            0.01628697326539432,
+            -0.00986238951830334,
+            -0.013799209313714441,
+            0.025178874553620743,
+            0.03324555035648896,
+            0.008398796308070151,
+            0.011261509865513953,
+            -0.01226890254177634,
+            0.018966175575083295,
+            -0.0038328997984309597,
+            0.004344837325292706,
+            0.00878668880458693,
+            -0.007200842513742962,
+            -0.011500562922303248,
+            -0.004533014670236755,
+            0.027639894892383943,
+            -0.021838848944317424,
+            -0.008920629621611768,
+            -0.005739181836252042,
+            -0.01593272996094732,
+            -0.038167534267554606,
+            -0.004371701247332816,
+            0.019604405563120094,
+            -0.005974358141737469,
+            -0.0018341151594086288,
+            0.014334266690716505,
+            0.00402620072221496,
+            -0.002852569009185304,
+            -0.008195218707838913,
+            -0.0027944153624309856,
+            -0.0059390492900516225,
+            0.016651206007022145,
+            -0.004961864528562493,
+            -0.007929830585906699,
+            0.01145431100211762,
+            -0.005731957927601062,
+            0.012130694936714224,
+            0.0004848418056781251,
+            8.392048424584131e-05,
+            -0.0015174120840537315,
+            0.00822932723013239,
+            0.018620869354789082,
+            -0.0055803454666417465,
+            0.025682778991091174,
+            0.006956936033581638,
+            -0.003495709501259274,
+            -0.005489431577052735,
+            -0.0044154293119170445,
+            -0.024561016942073285,
+            -0.0020864636079347357,
+            0.006593060072668424,
+            0.024333813203596122,
+            -0.02515989187381735,
+            -0.004953266000463339,
+            0.020957806164031402,
+            -0.0012192724005613,
+            -0.011978722479596745,
+            0.014690559986694138,
+            0.0041346025614047875,
+            0.009881584250499358,
+            0.012176393203318002,
+            0.015764530703333238,
+            -0.004923548157246672,
+            -0.0015192775000207833,
+            0.012766482928380375,
+            0.0005745973040874939,
+            0.00997809510696241,
+            0.016495444272181515,
+            0.010467514633877887,
+            -0.021108281220544853,
+            0.0015078843789125156,
+            0.00522593579521476,
+            -0.017481430635971396,
+            0.0034362821987740755,
+            0.005397819166037461,
+            0.011626527721895694,
+            0.004048632382026344,
+            0.013207335578216437,
+            -0.015768003434275563,
+            -0.005916168437120099,
+            0.008153798523776654,
+            0.004319451296835019,
+            -0.02599474886669325,
+            -0.01485009085106089,
+            -0.0006781964116146299,
+            -0.004444144921332929,
+            0.013556265879295017,
+            0.01285768162554067,
+            0.02033626001285354,
+            0.024637349990579514,
+            -0.013833060730799217,
+            -0.00842819657792762,
+            0.009436072455719896,
+            0.015560357465711765,
+            0.005798181411384387,
+            0.007874051721372273,
+            0.009297440071118126,
+            0.0077320823375042,
+            -0.004073209576661068,
+            -0.002973928843567532,
+            0.020205003097225747,
+            -0.019725382393637726,
+            0.0006746659838642147,
+            -0.018417443693472958,
+            0.007530796896254817,
+            0.01140696723506382,
+            0.01129515042745398,
+            0.002816883643872417,
+            -0.005274757308627299,
+            0.005716917047812153,
+            -0.06254973656832136,
+            -0.009191453047502128,
+            0.003384751551032943,
+            0.027256466708746898,
+            -0.010885336140129142,
+            -0.003415707910562931,
+            0.007309577090125959,
+            0.005511970708769456,
+            0.0005572239548625122,
+            0.003186427278034852,
+            0.019698716902282903,
+            -0.02052581021967601,
+            -0.010793968505805883,
+            0.004219269922138548,
+            0.0033111837923746746,
+            -0.007269521534951044,
+            0.004805737092302134,
+            0.014143978208501908,
+            0.01986656063456352,
+            -0.0024048854610842748,
+            0.01629335842915328,
+            -0.0015498603704589015,
+            -0.014496994444493684,
+            -0.002671064325909516,
+            0.0037875833047555925,
+            -0.022466433279066646,
+            -0.005980897597195858,
+            0.0043262697442044435,
+            -0.011226121648007667,
+            0.006978495397818936,
+            0.008452177557316863,
+            -0.003018545465557218,
+            -0.0008303307644324117,
+            0.017796789945838845,
+            0.006591364879905198,
+            -0.013464486184025026,
+            0.0028668426952326957,
+            -0.023377605222213174,
+            0.004325185794796361,
+            0.001416245546006914,
+            0.007809449561523174,
+            -0.008431335139468173,
+            -0.014774499666285603,
+            -0.005933810712528226,
+            0.017746847020610673,
+            -0.01965188730040385,
+            -0.002406875192894708,
+            0.01683009662984443,
+            -0.0002518220110313279,
+            0.005456812494894182,
+            -0.02011241622618092,
+            -0.0009782974741561381,
+            -0.02933296756078644,
+            -0.01171823509606186,
+            0.001162345317872531,
+            0.00892221099991475,
+            -0.006551308022163148,
+            -0.03934356275100932,
+            0.00427623835776202,
+            -0.013111664387776261,
+            -0.008752857644809231,
+            -0.010214546496134218,
+            -0.00518087722852466,
+            0.015240297918667867,
+            -0.012498100033235827,
+            0.01086490299272505,
+            -0.00917777212602538,
+            0.005895055740265625,
+            0.0090554651297836,
+            0.014212841798768113,
+            -0.005544047915369086,
+            -0.0033589999440643385,
+            -0.007924830667307956,
+            0.0022291797354234005,
+            -0.0032976226056924502,
+            0.009615674261248298,
+            -0.0051932155872883,
+            -0.0065636389655770455,
+            0.02426106498596339,
+            0.011445591067574932,
+            0.013312596099532533,
+            0.0055118624272110965,
+            -0.007220620072349783,
+            -0.01916551187613823,
+            -0.003977616240384473,
+            -0.013204506708246246,
+            -0.006315966975649096,
+            0.01434099854765068,
+            0.012588249778961223,
+            0.006357972880218866,
+            0.0007747002406996095,
+            -0.004794439624135275,
+            0.006975593520375694,
+            0.0262335496080837,
+            -0.016481909971520166,
+            -0.0013084662210086592,
+            -0.01369313900978409,
+            0.01921748474212319,
+            0.03333761672336437,
+            -0.0008918419419301244,
+            -0.007699525186523558,
+            0.013757216065348164,
+            0.00819045303047094,
+            -0.007411380297164911,
+            0.02453611080891859,
+            -0.025045462880463647,
+            -0.015989319900023054,
+            -0.011650886878708265,
+            -0.011407253985176972,
+            -0.008386367959227559,
+            0.016331961716602287,
+            -0.00141225438918211,
+            -0.0018505754298268654,
+            0.009557291617446645,
+            0.03357924917013024,
+            0.006916860901290756,
+            0.007030308953738567,
+            -0.002739345982346728,
+            -0.009376522438071562,
+            0.011482637968411055,
+            -0.007259054391879614,
+            -0.04075661435928125,
+            0.004250195164583401,
+            -0.005160505473755424,
+            0.02405623169060588,
+            -0.02197586561375289,
+            -0.00619175761282745,
+            -0.006736129049831973,
+            -0.028200015908139763,
+            -0.005867722104196303,
+            0.02805071895018145,
+            -0.015402198567775478,
+            0.014773294766214671,
+            -0.009502185346558072,
+            -0.0013419858580438616,
+            0.004443749010136947,
+            0.013008783570355656,
+            -0.02289662952366337,
+            0.0013834888740540108,
+            -0.00048549197205067054,
+            0.027969842651517314,
+            0.010437729090975158,
+            -0.0047705656249623744,
+            0.014202218020412936,
+            0.008240861154198411,
+            -0.018637315941614406,
+            0.007314019317343154,
+            -0.005747966765101792,
+            0.014833896739762147,
+            -0.01314847940891154,
+            0.007762100494499689,
+            0.00705026393499352,
+            -0.01278023669088385,
+            0.03016342404000438,
+            -0.015204129985039077,
+            0.007776946006562982,
+            0.009278778571285689,
+            0.009656723862179986,
+            0.01257686993982689,
+            0.004362439504889069,
+            -0.02898418193036562,
+            0.0002144431781454467,
+            -0.03828100004144633,
+            -0.00833748467467171,
+            -0.002611937107022567,
+            -0.002145329666219461,
+            0.007050630951707937,
+            -0.009383645162261629,
+            -0.033976442057557676,
+            0.0006429978367640709,
+            0.01603027598131823,
+            0.017501067413108286,
+            0.0002196997337451293,
+            0.013075379412332973,
+            -0.006197537539554383,
+            -0.021148120787551232,
+            -0.016986448660412497,
+            -0.010207705210051413,
+            -0.02122194744719993,
+            -0.020550470201191853,
+            -0.00908564461857315,
+            0.012090079127547193,
+            -0.009927390005879807,
+            0.013888458967497627,
+            0.029513250508042935,
+            -0.024529175134769883,
+            0.0011144447737101862,
+            0.021977677894839922,
+            -0.00198308986575084,
+            0.014108189311291326,
+            0.001696775006906642,
+            0.014122064251288995,
+            -0.0004166140484018266,
+            -0.010556016685519523,
+            -0.009729362492661684,
+            0.017487929296614076,
+            -0.0026043257634797124,
+            0.016715798948897194,
+            0.012183487338360088,
+            0.003771721868999568,
+            -0.0030887189945795796,
+            0.010667776697461199,
+            -0.026471157050402344,
+            0.014249996582064706,
+            -0.006690961709923224,
+            -0.0181382691440242,
+            -0.008729104191096461,
+            0.003882731567105777,
+            -0.005609384377642956,
+            -0.007794812949125115,
+            0.012967089730551055,
+            -0.00944374111323783,
+            0.02088525981049842,
+            0.0023400121055514958,
+            -0.012196925240774426,
+            0.003648688547842322,
+            -0.03322041209304968,
+            0.0015191765655963997,
+            -0.018402382776013956,
+            -0.024659631472836927,
+            0.03840173429666435,
+            0.011991452465253198,
+            0.006639702644606365,
+            -0.01675838115087676,
+            0.007953102302963526,
+            0.0004257479565099406,
+            0.017484338819046517,
+            0.011068864013925452,
+            0.006466915839538296,
+            0.008364454017976605,
+            -0.023316137291380285,
+            -0.010898421318668592,
+            0.016204569093730564,
+            0.012099473359791148,
+            -0.01017229369980751,
+            0.019431669546223253,
+            0.017955119269367787,
+            0.007300372071724306,
+            -0.006898619924677706,
+            0.01647927363640854,
+            0.021012103522831126,
+            0.010291016618557653,
+            -0.00047955552103474145,
+            0.0008684297441778696,
+            -0.010534247928997246,
+            0.005037930235889528,
+            0.035417626740307205,
+            0.014712595692093886,
+            0.010650306363963653,
+            -0.03626584475133921,
+            0.021764456453493054,
+            -0.011961521673661574,
+            0.009092981192874177,
+            0.006089104887183293,
+            0.017982801101846038,
+            0.006916969397519662,
+            0.014196254958707752,
+            0.011745069030657836,
+            -0.006689305450954901,
+            0.025205280281804708,
+            -0.001601414806100461,
+            -0.011992525142592405,
+            0.003386930324498371,
+            -0.01859193251438437,
+            0.005303261319773131,
+            -0.0041710536965100615,
+            -0.004459393120257307,
+            0.0032461828144552565,
+            -0.0018650956576455287,
+            0.015678225251799887,
+            0.003894142628084471,
+            -0.02355212701026675,
+            -0.0014873755115206368,
+            -0.024284295126348117,
+            0.0027260592623277375,
+            -0.022185571998195344,
+            0.00022883605560765203,
+            -0.005557568353382136,
+            -0.009749382813201403,
+            -0.025385857431333838,
+            -0.009853073898348951,
+            -0.0035843255959309307,
+            0.007661137888845632,
+            -0.010521037433807653,
+            -0.004246698084422667,
+            -0.0019920499321964815,
+            -0.019465113363548715,
+            0.009014255846983305,
+            0.027122533756959888,
+            0.016848221859975445,
+            0.027170883580103264,
+            -0.00019554112591107156,
+            0.013614675849485926,
+            -0.002066822813103284,
+            -0.01603063005944364,
+            0.008824663429777616,
+            0.0075653522381218495,
+            -0.00531780740880296,
+            0.014643348459785112,
+            0.0072634484937344704,
+            -0.012800149015319017,
+            0.009859503660384916,
+            -0.007161712176469609,
+            -0.006437857349742842,
+            -0.009764856421740622,
+            -0.02058868349990357,
+            -0.014866137971603676,
+            0.0012450207572602557,
+            -0.002123265406747006,
+            0.006000473862248991,
+            -0.004054016790850226,
+            0.009584879012937315,
+            -0.00608816807208866,
+            -0.00925801057439376,
+            -0.013962683072792106,
+            -0.008924356118158748,
+            -0.004210064145949253,
+            -0.004381709838058668,
+            -0.011732699976353612,
+            -0.005479523097469261,
+            0.0010680258077037758,
+            -0.0019401640391035755,
+            -0.01397652201595301,
+            0.011650839243603377,
+            0.0058731838291845645,
+            -0.002462743134993977,
+            0.02219008688925881,
+            0.01097316136828334,
+            0.0018083001685932406,
+            0.01366095798812196,
+            0.0003942601242216818,
+            -0.0064063532958756175,
+            -0.03343312148366622,
+            -0.0005083355836126291,
+            -0.02815586367527057,
+            0.025770078162165123,
+            -0.0069114949102666145,
+            -0.010322087105114614,
+            -0.002519047562374146,
+            -0.006063669644021698,
+            0.005873609611264377,
+            -0.002722940652283699,
+            0.0021229525800901114,
+            -0.023673778230768945,
+            0.0028629715379660113,
+            -0.003953653437464966,
+            -0.0036366957271968823,
+            0.007469664864329332,
+            0.0346520690389124,
+            -0.004054282838167587,
+            -0.0034765253405897676,
+            0.005159466707114532,
+            0.011642913651743502,
+            0.027845571220425726,
+            -0.004445196354520585,
+            -0.0032038508043339413,
+            -0.02233838584152591,
+            0.0054112576531374175,
+            -0.00019544838850471866,
+            -0.004733192299951088,
+            0.016664217348411003,
+            -0.016719175418727594,
+            -0.01631658832047864,
+            -0.00545969677717507,
+            -0.014570848053102338,
+            -0.00022620258237138677,
+            0.014454101124666536,
+            0.011056312437637014,
+            -0.004035526363810078,
+            -0.012459378363360557,
+            -0.0022595493558020706,
+            -0.030370952502579162,
+            0.012766010284884204,
+            0.1141689619126009,
+            -0.00766197047210299,
+            -0.00896655658602649,
+            -0.009481112393590025,
+            0.001435269335646347,
+            0.001338755472877713,
+            -0.011739143028767118,
+            0.07166096084053437,
+            -0.0077855767070652395,
+            0.0002139580430342728,
+            -0.011312198831181402,
+            0.001844005266872301,
+            -0.001825412693163496,
+            0.01371932421024605,
+            -0.005025814503687022,
+            -0.018213360991353643,
+            -0.0033612481616769416,
+            -0.02941403830641238,
+            -0.0041137865872933515,
+            -0.005683630256743215,
+            -0.004288198570156578,
+            -0.014982999363445122,
+            0.0067149665791261335,
+            -0.019661973497372313,
+            -0.012688556129040908,
+            0.001124391650175233,
+            -0.0033623453063430534,
+            0.00996944383866679,
+            -0.020496336814603385,
+            -0.0044354321069609536,
+            0.0019875840782527733,
+            0.009706675240563883,
+            0.0011600364428219214,
+            -0.012755178230821602,
+            -0.0026514442584739412,
+            0.004056350166031111,
+            0.01909951327913368,
+            0.014360281326789544,
+            0.003732344102045803,
+            0.008623457695718206,
+            0.014461868085333023,
+            -0.001692149528228451,
+            0.01948035376572433,
+            0.01172550239477637,
+            0.0031966531144076173,
+            0.0004779413449255626,
+            0.0030912562273611517,
+            -0.004597633235071031,
+            0.01582547400214653,
+            -0.014811177637844808,
+            -0.02411202772171917,
+            -0.0071272238837422185,
+            0.018015541137524113,
+            -0.013081287852028074,
+            1.8837689271256874e-05,
+            0.004062803284646432,
+            0.027569726868200254,
+            0.0010564850664255057,
+            -0.01179755720247972,
+            0.0028876134800690334,
+            -0.01638215925131053,
+            0.022944183106912663,
+            0.0026043713064243836,
+            0.03279173899194346,
+            -0.005103903273968361,
+            0.00578213669463383,
+            -0.020744194169889544,
+            -0.007588793365216579,
+            -0.00819862611489168,
+            0.0017401442433921013,
+            0.010993500174794084,
+            0.005411238350013134,
+            0.01620925462604661,
+            -0.00641615069699833,
+            -0.020751730783437995,
+            0.014776878596707538,
+            -0.009552193304486048,
+            -0.03393319554458277,
+            0.008881987295131347,
+            -0.029102297123590998,
+            -0.0045267103602711,
+            0.00368971742585232,
+            -0.014269341759285646,
+            -0.016264809810970175,
+            0.012837302095591106,
+            -0.0011890334180728564,
+            -0.008937992768323412,
+            0.0014444613725248846,
+            -0.017110187133760626,
+            -0.02385668037818897,
+            0.006017405364919834,
+            0.00676670672597817,
+            -0.0077994500447595,
+            0.018312452858615755,
+            -0.027015660037024585,
+            0.006123912473734307,
+            -0.012973676426738582,
+            0.009041974762537394,
+            -0.01580183887345168,
+            0.01163516332546347,
+            0.010887662470499361,
+            -0.001503467664709027,
+            -0.0076234124970223326,
+            0.01231202541338883,
+            0.004852720134894811,
+            0.033760365210093034,
+            0.028658326381800395,
+            0.000995999783627898,
+            -0.0025605367058226737,
+            -0.0020356794893366384,
+            0.011667525096213252,
+            -0.014049700287539817,
+            0.011218246246280464,
+            0.02658113044439717,
+            0.017568631325838108,
+            -0.019133062654008885,
+            0.018050678772708717,
+            -0.010356118256625172,
+            0.012872052420999813,
+            -0.025620154415048252,
+            -0.005135768485044442,
+            0.004936490920616527,
+            0.005163008729256001,
+            0.007425278892536464,
+            0.001410212306439329,
+            0.021891387289347516,
+            0.014999616600766989,
+            0.00027613811156650413,
+            0.00558422822386443,
+            0.01477009526981092,
+            0.010497782599293994,
+            -0.0005567894749233204,
+            -0.00467776506604011,
+            -0.016998035358377343,
+            0.00957271797745408,
+            -0.006065116857898469,
+            0.005138070401846389,
+            0.01698914855850691,
+            -0.01230257809343985,
+            0.011109844802663414,
+            0.007518553660463051,
+            -0.015504773927595823,
+            -0.017169933945839305,
+            0.005552829003043926,
+            -0.02252633790984703,
+            -0.01306722099657118,
+            0.02715689072730398,
+            -0.010513941399990838,
+            -0.02790582732974509,
+            -0.0009847768069205477,
+            0.009797865241108237,
+            -0.002772952073769185,
+            -0.0027968741526000576,
+            0.0037541733651479613,
+            0.010801137722284016,
+            -0.014658156970099418,
+            0.002075415233345957,
+            0.004735264309137368,
+            0.01930804315461151,
+            -0.0014151522239496115,
+            -0.015046118414100617,
+            -0.002288495894227231,
+            0.021592829677937263,
+            0.017174609713578173,
+            0.00265560378681856,
+            -0.016499825201963983,
+            -3.965296928410772e-05,
+            -0.0074142747014567635,
+            0.00021360696629503726,
+            0.00027431174904249525,
+            -0.0056391089282546054,
+            -0.018931681700770225,
+            -0.003888248121485032,
+            0.011181675419027736,
+            -0.00247252920175839,
+            0.01977860312223846,
+            -0.0011527575128223392,
+            -0.015763587433200915,
+            0.001960734212846797,
+            -0.005280860341480037,
+            -0.010372829570478649,
+            0.01351900743232294,
+            -0.009359942374603997,
+            0.006788746813046239,
+            -0.003212015225284119,
+            0.004611143085183367,
+            -0.0003308099698088422,
+            -0.017017265064185255,
+            0.02924142709831441,
+            -0.0018238549856433466,
+            -0.004769274041546295,
+            -0.0014058363976586562,
+            -0.033393331166741945,
+            0.015017165991303838,
+            -0.014574579250749552,
+            0.006745699409061803,
+            0.002705853802154516,
+            0.01272741296341735,
+            0.003253095246601883,
+            -0.008898616461779179,
+            0.0027485606928249326,
+            0.017040360927723885,
+            0.013756402798365262,
+            -0.007292864293090161,
+            0.008694855886257706,
+            -0.0028934389922887676,
+            -0.0037483208383542015,
+            0.01363963339139356,
+            0.004170611039133969,
+            -0.006282657671540625,
+            -0.0009784372731140945
+          ],
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "Mixture-of-Experts (MoE)",
+            "fine-grained expert segmentation",
+            "shared expert isolation"
+          ],
+          "title": "Deepseekmoe: Towards Ultimate Expert Specialization In Mixture-Of-Experts Language Models"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "396d9412-17a9-4b7e-831f-84f4e5006484",
+        "properties": {
+          "page_content": "9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\n",
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "MoE language models",
+            "supervised fine-tuning",
+            "large-scale language models"
+          ],
+          "embedding": [
+            -0.003971853759139776,
+            -0.012639197520911694,
+            -0.032538022845983505,
+            0.01764647848904133,
+            0.028833750635385513,
+            0.017460335046052933,
+            0.0033017343375831842,
+            0.0448235459625721,
+            -0.002440816955640912,
+            0.05025895684957504,
+            -0.011708475649356842,
+            -0.010033177211880684,
+            0.026748934760689735,
+            -0.023044662550091743,
+            -0.0071293264627456665,
+            0.011140735819935799,
+            0.007008332759141922,
+            0.022765446454286575,
+            0.005472642369568348,
+            -0.03823403641581535,
+            0.014482025988399982,
+            -0.014649556018412113,
+            -0.05766750127077103,
+            -0.008972154930233955,
+            0.008334610611200333,
+            0.017088046297430992,
+            0.018679579719901085,
+            0.0209412332624197,
+            -0.06805435568094254,
+            0.020010510459542274,
+            0.045344747602939606,
+            0.012024921365082264,
+            -0.0010371727403253317,
+            -0.006017114035785198,
+            -0.006594161503016949,
+            -0.03143977001309395,
+            0.04270150139927864,
+            0.015459283255040646,
+            -0.024254601448774338,
+            0.02741905488073826,
+            0.02278406172990799,
+            -0.006980410777032375,
+            -0.02874067798256874,
+            -0.009809804148972034,
+            0.00014906085561960936,
+            0.007208437658846378,
+            -0.006026421207934618,
+            0.005379570182412863,
+            -0.0071293264627456665,
+            -0.005030549596995115,
+            -0.012173837050795555,
+            -0.014119044877588749,
+            0.02192779816687107,
+            -0.01348615437746048,
+            -0.025278395041823387,
+            0.010154170915484428,
+            0.014649556018412113,
+            0.0407283715903759,
+            0.04463740065693855,
+            0.026693090796470642,
+            0.021778881549835205,
+            -0.030192604288458824,
+            0.003183067310601473,
+            -0.000935375050175935,
+            0.02533423714339733,
+            0.013030100613832474,
+            0.004416273441165686,
+            0.03518127277493477,
+            0.023658938705921173,
+            -0.02598574385046959,
+            -0.049328237771987915,
+            0.02230008691549301,
+            -0.007487654220312834,
+            -0.012797419913113117,
+            0.004241763148456812,
+            -0.00017014751210808754,
+            0.0030667271930724382,
+            0.0407283715903759,
+            -0.023510023951530457,
+            0.003876454895362258,
+            0.010749832727015018,
+            0.013662991113960743,
+            0.010498537681996822,
+            -0.0014414548641070724,
+            0.04325993359088898,
+            -0.014845007099211216,
+            -0.010051791556179523,
+            -0.09188082069158554,
+            0.0304159764200449,
+            -0.010126249864697456,
+            -0.020475871860980988,
+            0.006919913925230503,
+            0.013616454787552357,
+            -0.011280343867838383,
+            -0.009781882166862488,
+            0.027605198323726654,
+            0.01356991846114397,
+            0.006398709956556559,
+            -0.0263208020478487,
+            0.01287187822163105,
+            -0.013030100613832474,
+            0.0022663066629320383,
+            0.012918413616716862,
+            0.003997448831796646,
+            -0.00012244803656358272,
+            -0.008799971081316471,
+            -0.029466642066836357,
+            0.030267061665654182,
+            0.014156273566186428,
+            0.10290056467056274,
+            0.00873482134193182,
+            -0.0014821739168837667,
+            -0.035088200122117996,
+            0.015133530832827091,
+            -0.028722064569592476,
+            -0.01276949793100357,
+            -0.0439300537109375,
+            -0.03203543275594711,
+            0.005868198815733194,
+            0.0019475346198305488,
+            0.03531157225370407,
+            0.01991743966937065,
+            0.02561345510184765,
+            -0.0014717032900080085,
+            -0.06310291588306427,
+            0.046982817351818085,
+            -0.015608198940753937,
+            -0.02019665576517582,
+            0.010358929634094238,
+            -0.004600090906023979,
+            -0.016101481392979622,
+            -0.018028074875473976,
+            -0.033115070313215256,
+            -0.04102620109915733,
+            0.0028037982992827892,
+            -0.010991820134222507,
+            0.014584405347704887,
+            0.06392195075750351,
+            -0.040951743721961975,
+            -0.01026585791260004,
+            0.015989795327186584,
+            0.010935977101325989,
+            0.009302561171352863,
+            -0.020866775885224342,
+            -0.029299112036824226,
+            -0.0041649783961474895,
+            0.0076691447757184505,
+            0.08391384780406952,
+            -0.007845981977880001,
+            0.02118322066962719,
+            -0.024831648916006088,
+            0.011205886490643024,
+            0.011969078332185745,
+            0.004658260848373175,
+            0.0013623435515910387,
+            -0.013523383066058159,
+            -0.030490435659885406,
+            0.005826316308230162,
+            -0.021034304052591324,
+            0.02980170026421547,
+            0.0025920593179762363,
+            -0.025632068514823914,
+            -0.0009185057133436203,
+            0.01166194025427103,
+            -0.0058030481450259686,
+            0.017702322453260422,
+            0.021127376705408096,
+            0.0077249882742762566,
+            -0.060087379068136215,
+            0.053758472204208374,
+            0.01866096444427967,
+            0.0218347255140543,
+            -0.024329058825969696,
+            0.004721084609627724,
+            0.00043831163202412426,
+            0.017972230911254883,
+            0.0070595224387943745,
+            0.011410645209252834,
+            -0.005281844176352024,
+            0.033561814576387405,
+            -0.015245217829942703,
+            0.023863697424530983,
+            0.021648580208420753,
+            -0.0014717032900080085,
+            0.030453206971287727,
+            0.013495461083948612,
+            0.018474821001291275,
+            0.02894543670117855,
+            -0.0181862972676754,
+            0.005430759862065315,
+            0.014044586569070816,
+            0.00012295703345444053,
+            0.029448026791214943,
+            -0.006738423369824886,
+            -0.007389928214251995,
+            -0.037340544164180756,
+            -0.018484128639101982,
+            0.02455243282020092,
+            -0.0004868836549576372,
+            0.009372364729642868,
+            -0.00881858542561531,
+            -0.03901584446430206,
+            -0.004251070320606232,
+            -0.012946335598826408,
+            0.028870979323983192,
+            0.005100353620946407,
+            -0.011150043457746506,
+            0.005807701963931322,
+            0.022058097645640373,
+            -0.013672297820448875,
+            -0.012434438802301884,
+            -0.007520229555666447,
+            0.002373339608311653,
+            -0.02220701426267624,
+            -0.0041021546348929405,
+            0.0037182322703301907,
+            -0.01921009086072445,
+            0.0027642427012324333,
+            0.008585905656218529,
+            -0.00825549941509962,
+            -0.01627831906080246,
+            0.015040459111332893,
+            0.03441808000206947,
+            -0.016203859820961952,
+            -0.03585138916969299,
+            -0.024310443550348282,
+            -0.022560687735676765,
+            0.018996024504303932,
+            0.028498690575361252,
+            -0.009511973708868027,
+            0.03091856651008129,
+            0.008367185480892658,
+            0.014295881614089012,
+            0.004711777437478304,
+            0.021127376705408096,
+            -0.008074008859694004,
+            -0.011354802176356316,
+            -0.018791265785694122,
+            0.014212116599082947,
+            0.026488332077860832,
+            0.00021682900842279196,
+            0.018260754644870758,
+            -0.04556812345981598,
+            -0.011540945619344711,
+            0.0020394434686750174,
+            -0.0006038055289536715,
+            -0.012406516820192337,
+            -0.016864672303199768,
+            0.01268573384732008,
+            0.029652785509824753,
+            0.008381146937608719,
+            -0.016557535156607628,
+            -0.002878256142139435,
+            0.00434181559830904,
+            -0.04084005579352379,
+            0.052455462515354156,
+            0.03990933671593666,
+            -0.012648504227399826,
+            -0.024366287514567375,
+            -0.033282600343227386,
+            0.05710906907916069,
+            -0.02380785532295704,
+            -0.025222551077604294,
+            0.06961796432733536,
+            -0.013849135488271713,
+            0.019768523052334785,
+            0.05707183852791786,
+            0.0026944386772811413,
+            0.004486077465116978,
+            -0.01794430986046791,
+            0.026227731257677078,
+            -0.021369364112615585,
+            -0.01190392766147852,
+            0.006394056603312492,
+            0.03030429035425186,
+            -0.012332059442996979,
+            -0.01941484957933426,
+            -0.002408241853117943,
+            0.0011081402190029621,
+            0.015059073455631733,
+            0.002245365409180522,
+            0.009539894759654999,
+            0.013514075428247452,
+            0.012536818161606789,
+            0.00968881044536829,
+            -0.0047303917817771435,
+            -0.01110350713133812,
+            -0.04325993359088898,
+            -0.03162591531872749,
+            -0.01740449108183384,
+            -0.035572174936532974,
+            0.0013146440032869577,
+            -0.021909182891249657,
+            0.011205886490643024,
+            0.03786174952983856,
+            -0.009027997963130474,
+            0.003348270431160927,
+            0.018884338438510895,
+            -0.0023140062112361193,
+            -0.0005747204995714128,
+            -0.031718987971544266,
+            -0.018577199429273605,
+            -0.00893958006054163,
+            0.00128555903211236,
+            -0.05126413702964783,
+            -0.01171778328716755,
+            0.008418375626206398,
+            0.03103025257587433,
+            -0.02250484563410282,
+            -0.023621710017323494,
+            0.015878107398748398,
+            -0.019805751740932465,
+            -0.028647607192397118,
+            -0.02931772544980049,
+            -0.019191475585103035,
+            0.05465196445584297,
+            0.016706449910998344,
+            0.025967128574848175,
+            -0.03147700056433678,
+            -0.03423193469643593,
+            0.05025895684957504,
+            0.008474218659102917,
+            -0.07550012320280075,
+            0.036595966666936874,
+            0.0014507620362564921,
+            0.0010063425870612264,
+            -0.005495910067111254,
+            0.05837484821677208,
+            -0.0032575251534581184,
+            -0.0010057608596980572,
+            0.012536818161606789,
+            -0.05748135596513748,
+            -0.00575651228427887,
+            -0.006412670947611332,
+            0.013253473676741123,
+            -0.04642438516020775,
+            -0.024943334981799126,
+            0.004027697257697582,
+            -0.012974257580935955,
+            -0.005700668785721064,
+            -0.042180296033620834,
+            -0.03145838528871536,
+            -0.0028526613023132086,
+            -0.0022756140679121017,
+            -0.007943707518279552,
+            -0.03078826516866684,
+            -0.025929899886250496,
+            0.021164605394005775,
+            0.045344747602939606,
+            0.007641223259270191,
+            0.0017695341957733035,
+            0.023621710017323494,
+            0.0197871383279562,
+            0.005598289426416159,
+            0.020345570519566536,
+            -0.01645515486598015,
+            -0.0375266894698143,
+            -0.005495910067111254,
+            -0.00010630584438331425,
+            0.02680477872490883,
+            -0.00926067866384983,
+            -0.049700524657964706,
+            -0.033599045127630234,
+            0.0193403922021389,
+            -0.03687518462538719,
+            -0.015487205237150192,
+            -0.004313894081860781,
+            -0.033636271953582764,
+            0.013141786679625511,
+            0.00029579491820186377,
+            -0.01730211265385151,
+            0.049328237771987915,
+            -0.023454179987311363,
+            -0.0019975609611719847,
+            -0.023752011358737946,
+            0.003443669294938445,
+            0.05312558263540268,
+            -0.018502742052078247,
+            -0.018028074875473976,
+            0.022672373801469803,
+            -0.01730211265385151,
+            0.01849343627691269,
+            0.04314824566245079,
+            0.009837726131081581,
+            0.004304586909711361,
+            0.014724013395607471,
+            0.006226526573300362,
+            -0.008483526296913624,
+            -0.0386807844042778,
+            0.021871954202651978,
+            0.012220372445881367,
+            -0.045307520776987076,
+            -0.022597916424274445,
+            -0.02479442022740841,
+            0.0007329431246034801,
+            0.011801548302173615,
+            0.016557535156607628,
+            -0.00968881044536829,
+            -0.028387004509568214,
+            0.006421978119760752,
+            -0.016017716377973557,
+            0.010163478553295135,
+            0.00786459632217884,
+            0.014845007099211216,
+            -0.005519178230315447,
+            0.0067756520584225655,
+            0.004006756003946066,
+            0.012788113206624985,
+            0.0362422950565815,
+            -0.02721429616212845,
+            -0.013216244988143444,
+            -0.020271113142371178,
+            -0.05506148189306259,
+            0.0014589058700948954,
+            0.015059073455631733,
+            0.004544247407466173,
+            -0.02455243282020092,
+            0.005709975957870483,
+            0.012546124868094921,
+            -0.04180800914764404,
+            -0.003641447750851512,
+            0.012099378742277622,
+            0.034678682684898376,
+            0.028238089755177498,
+            -0.011596789583563805,
+            0.003436689032241702,
+            0.0011959770927205682,
+            -0.0032435643952339888,
+            -0.0007631916087120771,
+            -0.023100506514310837,
+            0.027493512257933617,
+            0.014649556018412113,
+            0.038941387087106705,
+            -0.023472795262932777,
+            -0.005998499691486359,
+            -0.048211369663476944,
+            -0.003466937458142638,
+            0.030248448252677917,
+            -0.020680630579590797,
+            -0.011736397631466389,
+            -0.0005956617533229291,
+            0.005044510122388601,
+            -0.014733321033418179,
+            -0.006338213104754686,
+            -0.03194236010313034,
+            -0.026674477383494377,
+            -0.04143571853637695,
+            -0.01702289469540119,
+            -0.00934909749776125,
+            -0.01416558027267456,
+            -0.05748135596513748,
+            -0.011364108882844448,
+            -0.09463575482368469,
+            0.009218796156346798,
+            0.04117511585354805,
+            -0.0257437564432621,
+            0.028238089755177498,
+            -0.028796521946787834,
+            -0.0035576827358454466,
+            -0.03501374274492264,
+            0.012648504227399826,
+            0.040616683661937714,
+            0.033115070313215256,
+            -0.015291753225028515,
+            -0.006966450251638889,
+            0.017516177147626877,
+            -0.030732423067092896,
+            -0.03802927955985069,
+            -0.019228706136345863,
+            -0.00846025813370943,
+            0.00572859076783061,
+            0.0018474820535629988,
+            0.03789897635579109,
+            0.04504691809415817,
+            0.04631270095705986,
+            0.03592584654688835,
+            0.00934909749776125,
+            -0.035367414355278015,
+            -0.010740526020526886,
+            0.005216693971306086,
+            0.03071380779147148,
+            -0.03305922448635101,
+            0.01166194025427103,
+            -0.0311233252286911,
+            0.01853066496551037,
+            0.002405914943665266,
+            -0.023454179987311363,
+            0.021853338927030563,
+            0.005947310011833906,
+            0.0013507094699889421,
+            -0.005919388495385647,
+            0.027325982227921486,
+            -0.01828867755830288,
+            0.035125426948070526,
+            -0.028442848473787308,
+            -0.024105684831738472,
+            -0.028684835880994797,
+            -0.011122121475636959,
+            0.031104711815714836,
+            0.019154246896505356,
+            -0.0005947891622781754,
+            -0.002433836692944169,
+            -0.015273138880729675,
+            0.009172259829938412,
+            0.004853712394833565,
+            -0.02032695710659027,
+            -0.016334161162376404,
+            0.026879236102104187,
+            -0.06943181902170181,
+            -0.009763267822563648,
+            0.008306688629090786,
+            0.0005555243697017431,
+            0.00929790735244751,
+            -0.0386807844042778,
+            0.010377543978393078,
+            0.004590783733874559,
+            -0.03990933671593666,
+            0.022877132520079613,
+            0.00926067866384983,
+            0.01971268095076084,
+            -0.004362756852060556,
+            -0.008837200701236725,
+            0.007022293284535408,
+            0.06451760977506638,
+            -0.010563688352704048,
+            0.0022791041992604733,
+            0.007455078884959221,
+            -0.00812519807368517,
+            0.03845741227269173,
+            -0.025222551077604294,
+            -0.028033331036567688,
+            0.02935495413839817,
+            -0.008455604314804077,
+            -0.007650530431419611,
+            -0.015310367569327354,
+            -0.006696540862321854,
+            0.007827367633581161,
+            0.02870344929397106,
+            -0.02373339794576168,
+            -0.005212040152400732,
+            -0.02676755003631115,
+            0.006524357479065657,
+            0.0022500192280858755,
+            0.002908504568040371,
+            0.0061334543861448765,
+            -0.0009958719601854682,
+            0.039685964584350586,
+            -0.0225420743227005,
+            0.03452976793050766,
+            0.02959694154560566,
+            -0.004090521018952131,
+            -0.0045582083985209465,
+            -0.0005939166294410825,
+            0.013179016299545765,
+            -0.01289049256592989,
+            -0.01808391883969307,
+            0.01736726239323616,
+            -0.018679579719901085,
+            -0.013244166038930416,
+            -0.054949793964624405,
+            -0.002410568529739976,
+            0.002610673662275076,
+            -0.010051791556179523,
+            0.006784959230571985,
+            -0.001237859483808279,
+            0.002903850981965661,
+            0.005393530707806349,
+            0.03949981927871704,
+            -0.03827126696705818,
+            0.023714782670140266,
+            0.00015429616905748844,
+            -0.0026525561697781086,
+            0.008716206066310406,
+            0.0035018394701182842,
+            0.007106058299541473,
+            0.05554545670747757,
+            0.012899799272418022,
+            -0.007994897663593292,
+            -0.005063124932348728,
+            0.005002627614885569,
+            0.016296932473778725,
+            -0.016027024015784264,
+            0.0052073863334953785,
+            0.03668903931975365,
+            0.025297008454799652,
+            0.000632890616543591,
+            -0.0017951290355995297,
+            -0.010182092897593975,
+            0.02751212567090988,
+            -0.021034304052591324,
+            0.029820315539836884,
+            0.04307378828525543,
+            0.009470091201364994,
+            -0.0439300537109375,
+            0.0069571430794894695,
+            -0.015431362204253674,
+            -0.009251371026039124,
+            0.0031528188847005367,
+            0.019600993022322655,
+            -0.04381836578249931,
+            0.003245891071856022,
+            0.01145718153566122,
+            0.008320650085806847,
+            -0.021052919328212738,
+            -0.04675944522023201,
+            -0.008674323558807373,
+            -0.0035274343099445105,
+            0.012974257580935955,
+            -0.016846058890223503,
+            -0.006510396488010883,
+            0.005742551293224096,
+            0.016259703785181046,
+            -0.004988667089492083,
+            0.012034228071570396,
+            0.033394284546375275,
+            -0.016408618539571762,
+            0.013783984817564487,
+            -0.007054868619889021,
+            -0.02189056947827339,
+            0.0110476640984416,
+            0.03095579519867897,
+            -0.01307663694024086,
+            0.03546048700809479,
+            -0.0015345270512625575,
+            0.012415824458003044,
+            0.012964949943125248,
+            0.003639120841398835,
+            -0.014817086048424244,
+            0.002092959824949503,
+            0.018335212022066116,
+            -0.021443821489810944,
+            0.013020792976021767,
+            -0.013541997410356998,
+            -0.037992049008607864,
+            0.010051791556179523,
+            -0.0038950692396610975,
+            0.0016752986703068018,
+            0.009041959419846535,
+            0.0062591019086539745,
+            -0.027195680886507034,
+            -0.01611078903079033,
+            0.023342493921518326,
+            0.024105684831738472,
+            0.06098087131977081,
+            -0.014081815257668495,
+            -0.000535164843313396,
+            -0.007617955096065998,
+            0.009493358433246613,
+            0.04460017383098602,
+            0.024459360167384148,
+            0.0038694743998348713,
+            0.004290625918656588,
+            0.027102608233690262,
+            0.046945590525865555,
+            -0.005230654496699572,
+            -0.010377543978393078,
+            0.006394056603312492,
+            0.003197028301656246,
+            0.023789240047335625,
+            0.021574122831225395,
+            -0.0005409818259067833,
+            0.014072508551180363,
+            0.014444797299802303,
+            -0.016538919880986214,
+            0.012937027961015701,
+            -0.04556812345981598,
+            -0.040169939398765564,
+            0.007115365471690893,
+            0.00435112277045846,
+            0.019359005615115166,
+            0.002450124127790332,
+            -0.005039856769144535,
+            -0.0027851839549839497,
+            0.01750687137246132,
+            -0.010535767301917076,
+            -0.0122948307543993,
+            0.02475719153881073,
+            -0.00873482134193182,
+            0.006747730541974306,
+            0.00030801063985563815,
+            -0.009716732427477837,
+            0.0035181271377950907,
+            -0.0014972981298342347,
+            -0.005077085457742214,
+            0.0023989344481378794,
+            -0.008381146937608719,
+            -0.017516177147626877,
+            0.012229680083692074,
+            -0.04396728053689003,
+            0.013886364176869392,
+            0.001635742955841124,
+            -0.008911658078432083,
+            0.002878256142139435,
+            0.0013146440032869577,
+            0.0114944102242589,
+            -0.018381748348474503,
+            -0.005309765692800283,
+            0.012453053146600723,
+            0.026413874700665474,
+            -0.007403889205306768,
+            0.0038904156535863876,
+            0.015142838470637798,
+            -0.007534190081059933,
+            -0.00987495481967926,
+            0.028070559725165367,
+            -0.007822713814675808,
+            0.006859417073428631,
+            -0.008181042037904263,
+            -0.02540869638323784,
+            0.0053888773545622826,
+            0.03793620690703392,
+            0.04865811765193939,
+            0.007538843899965286,
+            0.01648307777941227,
+            0.03123501129448414,
+            0.014835700392723083,
+            0.00822292361408472,
+            0.015422054566442966,
+            -0.04437679797410965,
+            0.01541274692863226,
+            -0.018037382513284683,
+            0.040542226284742355,
+            0.030844109132885933,
+            -0.05044510215520859,
+            -0.02812640182673931,
+            0.019284548237919807,
+            0.023100506514310837,
+            0.017562713474035263,
+            0.013681605458259583,
+            -0.005314419511705637,
+            0.02053171582520008,
+            -0.023249421268701553,
+            -0.01235998049378395,
+            0.01316040102392435,
+            -0.0032435643952339888,
+            -0.004816483706235886,
+            0.005067778285592794,
+            0.03912752866744995,
+            -0.00436741067096591,
+            -0.016911208629608154,
+            0.02131352201104164,
+            -0.005984539166092873,
+            0.005928695667535067,
+            -0.013123172335326672,
+            0.0009958719601854682,
+            0.03916475921869278,
+            0.005393530707806349,
+            0.03761976212263107,
+            -0.021499665454030037,
+            0.012145915068686008,
+            -0.028182245790958405,
+            -0.014733321033418179,
+            0.029243268072605133,
+            -0.013216244988143444,
+            -0.021201834082603455,
+            0.016976360231637955,
+            0.020513100549578667,
+            0.0032156426459550858,
+            0.011168657802045345,
+            -0.013681605458259583,
+            -0.010600917041301727,
+            0.038159579038619995,
+            -0.0029527137521654367,
+            -0.004144037142395973,
+            0.01712527498602867,
+            -0.01812114752829075,
+            -0.011922542005777359,
+            0.016883287578821182,
+            -0.0019580053631216288,
+            -0.032742779701948166,
+            0.02241177298128605,
+            -0.0004022461944259703,
+            0.03620506450533867,
+            -0.010237935930490494,
+            -0.0056634400971233845,
+            0.010535767301917076,
+            -0.006654658354818821,
+            -0.029987845569849014,
+            -0.034176092594861984,
+            0.007455078884959221,
+            -0.027437668293714523,
+            0.010963899083435535,
+            -0.006650005001574755,
+            0.03255663812160492,
+            -0.030248448252677917,
+            0.008669670671224594,
+            0.03030429035425186,
+            -0.00825549941509962,
+            0.022690989077091217,
+            0.032947540283203125,
+            -0.019079789519309998,
+            -0.03525573015213013,
+            -0.003245891071856022,
+            -0.0003592003195080906,
+            0.029578328132629395,
+            0.002880582818761468,
+            0.0026246344204992056,
+            0.004960745573043823,
+            -0.016222475096583366,
+            -0.03510681167244911,
+            0.022430386394262314,
+            -0.04474908858537674,
+            0.007068829610943794,
+            0.026618633419275284,
+            -0.0430365614593029,
+            -0.036800727248191833,
+            -0.018446899950504303,
+            -0.0263208020478487,
+            -0.02159273810684681,
+            0.001237859483808279,
+            0.02639526128768921,
+            -0.017581328749656677,
+            -0.030602121725678444,
+            0.028089173138141632,
+            0.0034948589745908976,
+            -0.012695040553808212,
+            -0.011056970804929733,
+            -0.010126249864697456,
+            -0.027083994820713997,
+            -0.022262856364250183,
+            -0.014035279862582684,
+            0.0009807477472350001,
+            0.004881633911281824,
+            -0.010368237271904945,
+            -0.0225420743227005,
+            -0.013616454787552357,
+            -0.009032651782035828,
+            -0.004825790878385305,
+            0.050147272646427155,
+            0.021090148016810417,
+            0.023863697424530983,
+            0.0009185057133436203,
+            -0.015719885006546974,
+            0.04601486772298813,
+            -0.0007253810181282461,
+            0.02591128461062908,
+            0.005612250417470932,
+            -0.022058097645640373,
+            -0.005844930652529001,
+            0.03715439885854721,
+            -0.018279369920492172,
+            -0.004395332187414169,
+            -0.014724013395607471,
+            -0.00374150020070374,
+            -0.008609173819422722,
+            0.026693090796470642,
+            0.012024921365082264,
+            -0.004769947379827499,
+            0.013458232395350933,
+            -0.01270434819161892,
+            0.02200225554406643,
+            0.005821662489324808,
+            -0.003676349762827158,
+            -0.0010331007651984692,
+            -0.028684835880994797,
+            -0.02114599198102951,
+            0.001068002893589437,
+            0.022765446454286575,
+            -0.016334161162376404,
+            -0.013197630643844604,
+            -0.015291753225028515,
+            -0.021574122831225395,
+            -0.005198079161345959,
+            0.0004831026017200202,
+            0.005407491698861122,
+            0.015850186347961426,
+            -0.003311041509732604,
+            -0.0430365614593029,
+            -0.005272537004202604,
+            -0.03763837739825249,
+            0.00048455686192028224,
+            0.01961960829794407,
+            -0.011243115179240704,
+            0.01128965150564909,
+            0.001805599662475288,
+            -0.0038648208137601614,
+            -0.01832590624690056,
+            0.007636569440364838,
+            0.01982436701655388,
+            -0.006789613049477339,
+            -0.0022197705693542957,
+            0.018986716866493225,
+            0.0021569470409303904,
+            -0.007645876612514257,
+            0.004849058575928211,
+            -0.001855625887401402,
+            -0.006845456548035145,
+            -0.023379722610116005,
+            -0.029280496761202812,
+            -0.013607148081064224,
+            0.0018090897938236594,
+            -0.001733468729071319,
+            -0.024906106293201447,
+            0.018875030800700188,
+            -0.006082264706492424,
+            0.02336110919713974,
+            -0.011326880194246769,
+            0.01316040102392435,
+            -0.0022697970271110535,
+            -0.0090931486338377,
+            0.018214218318462372,
+            -0.010554381646215916,
+            -0.03264970704913139,
+            -0.01316040102392435,
+            0.028517305850982666,
+            0.00034872969263233244,
+            -0.008097277022898197,
+            0.010042484849691391,
+            0.00922810286283493,
+            -0.027437668293714523,
+            -0.0077017201110720634,
+            0.010619531385600567,
+            0.006626736838370562,
+            -0.032333262264728546,
+            0.026506947353482246,
+            0.03266832232475281,
+            0.01235998049378395,
+            -0.03222157806158066,
+            0.048620887100696564,
+            0.008651056326925755,
+            -0.02475719153881073,
+            0.005677401088178158,
+            0.020624786615371704,
+            0.0363725945353508,
+            -0.0299692302942276,
+            -0.007413196377456188,
+            -0.007422503549605608,
+            0.009009383618831635,
+            -0.03473452478647232,
+            0.015235910192131996,
+            -0.02537146769464016,
+            0.01961960829794407,
+            0.032333262264728546,
+            0.0032831199932843447,
+            0.040579456835985184,
+            -0.009511973708868027,
+            0.017693014815449715,
+            0.0060310750268399715,
+            -0.0076691447757184505,
+            0.006459206808358431,
+            -0.02162996679544449,
+            0.01624108850955963,
+            0.019768523052334785,
+            0.02162996679544449,
+            0.009902875870466232,
+            -0.024682732298970222,
+            0.01941484957933426,
+            0.015133530832827091,
+            0.023826468735933304,
+            0.027605198323726654,
+            -0.014658862724900246,
+            0.026842007413506508,
+            0.005812355317175388,
+            -0.013635069131851196,
+            0.004430233966559172,
+            0.009386326186358929,
+            -0.040616683661937714,
+            -0.0012948662042617798,
+            -0.022523459047079086,
+            0.01459371205419302,
+            -0.0005066614830866456,
+            -0.0028224128764122725,
+            -0.02377062663435936,
+            -0.013653683476150036,
+            -7.831148832337931e-05,
+            0.005146889481693506,
+            -0.004358103033155203,
+            -0.04087728634476662,
+            -0.045754265040159225,
+            0.0234355665743351,
+            -0.009800496511161327,
+            -0.040542226284742355,
+            -0.012648504227399826,
+            -0.009311867877840996,
+            -0.0015752461040392518,
+            -0.014109737239778042,
+            -0.01289049256592989,
+            -0.04385559633374214,
+            -0.014435489661991596,
+            0.01958237960934639,
+            0.030043689534068108,
+            -0.02203948423266411,
+            0.030248448252677917,
+            0.006873378064483404,
+            0.015198681503534317,
+            -0.014630941674113274,
+            0.013560611754655838,
+            0.00614741537719965,
+            -0.04638715833425522,
+            0.02894543670117855,
+            -0.0028898902237415314,
+            0.021276293322443962,
+            0.0384201817214489,
+            -0.005272537004202604,
+            -0.010023870505392551,
+            0.017246268689632416,
+            -0.02213255688548088,
+            -0.013551304116845131,
+            -0.008092623203992844,
+            -0.01586880162358284,
+            -0.037005484104156494,
+            -0.04884426295757294,
+            0.016129402443766594,
+            0.04080282896757126,
+            0.00911176297813654,
+            0.014426182955503464,
+            0.018679579719901085,
+            0.004311567172408104,
+            -0.033282600343227386,
+            0.0225420743227005,
+            0.0047932155430316925,
+            -0.012387902475893497,
+            -0.023826468735933304,
+            0.013653683476150036,
+            0.012778805568814278,
+            -0.006198605056852102,
+            -0.006384748965501785,
+            -0.026655862107872963,
+            -0.024477973580360413,
+            0.0021476398687809706,
+            -0.013886364176869392,
+            0.0033250025007873774,
+            -0.022765446454286575,
+            -0.02857314795255661,
+            0.032128505408763885,
+            0.03546048700809479,
+            -0.01235998049378395,
+            -0.00393229816108942,
+            -0.013579226098954678,
+            -0.008404415100812912,
+            0.008920964784920216,
+            0.005230654496699572,
+            0.017953617498278618,
+            -0.019526535645127296,
+            0.01128965150564909,
+            -0.010442694649100304,
+            0.012099378742277622,
+            -0.0762074738740921,
+            -0.01679021492600441,
+            0.025874055922031403,
+            0.022672373801469803,
+            0.0009662052034400403,
+            0.011438566260039806,
+            0.008893043734133244,
+            0.0006619756459258497,
+            -0.035088200122117996,
+            0.01921009086072445,
+            0.013793292455375195,
+            0.016222475096583366,
+            -0.01151302456855774,
+            0.005235308315604925,
+            0.023845084011554718,
+            -0.0006683743558824062,
+            -0.006319598760455847,
+            -0.01760924980044365,
+            0.011540945619344711,
+            0.020103583112359047,
+            0.0026851315051317215,
+            0.013877056539058685,
+            0.010572995990514755,
+            -0.01961960829794407,
+            -0.013011486269533634,
+            -0.0005930440966039896,
+            -0.005668093450367451,
+            -0.010963899083435535,
+            0.008706899359822273,
+            0.0007399235619232059,
+            0.03763837739825249,
+            0.024887491017580032,
+            0.02172303944826126,
+            0.015980487689375877,
+            0.0069571430794894695,
+            0.007469039876013994,
+            -0.0020917963702231646,
+            0.0018323578406125307,
+            0.01746964268386364,
+            0.03272416815161705,
+            -0.022932976484298706,
+            -0.005156196653842926,
+            0.0042533972300589085,
+            0.024831648916006088,
+            -0.008827893063426018,
+            0.025892671197652817,
+            0.030062302947044373,
+            -0.007650530431419611,
+            -0.029503870755434036,
+            0.0032924271654337645,
+            0.018502742052078247,
+            0.009302561171352863,
+            -0.01634346880018711,
+            -0.0027921644505113363,
+            0.013728141784667969,
+            0.0075434972532093525,
+            0.020922617986798286,
+            -0.004241763148456812,
+            -0.014053894206881523,
+            0.01880057342350483,
+            0.09076395630836487,
+            -0.005691361613571644,
+            -0.0012017941335216165,
+            -0.017106659710407257,
+            -0.01921009086072445,
+            0.0014670497039332986,
+            0.0098935691639781,
+            0.016613377258181572,
+            0.017209040001034737,
+            0.00374615378677845,
+            0.004586129914969206,
+            0.01455648336559534,
+            -0.011531638912856579,
+            0.009479397907853127,
+            -0.02421737276017666,
+            0.01804668828845024,
+            -0.014249345287680626,
+            -0.0067756520584225655,
+            -0.011233807541429996,
+            0.023621710017323494,
+            -0.0019568419083952904,
+            0.007720334455370903,
+            0.019154246896505356,
+            0.003871801309287548,
+            -0.0247199609875679,
+            0.006412670947611332,
+            -0.006454553455114365,
+            0.0007160738459788263,
+            -0.038866929709911346,
+            -0.018363134935498238,
+            0.01127103716135025,
+            -0.0014123697765171528,
+            -0.0011232644319534302,
+            -0.004672221839427948,
+            -0.007804099470376968,
+            -0.00025580296642147005,
+            -0.02455243282020092,
+            0.016836751252412796,
+            0.008855815045535564,
+            0.015403440222144127,
+            0.022393157705664635,
+            -0.0010522969532757998,
+            -0.0009138521272689104,
+            0.025315623730421066,
+            -0.0071293264627456665,
+            -0.004043984692543745,
+            0.027232909575104713,
+            -0.023621710017323494,
+            -0.005188771989196539,
+            0.005300458520650864,
+            -0.01692982390522957,
+            -0.002913158154115081,
+            -0.00414636405184865,
+            -0.021034304052591324,
+            -0.013206937350332737,
+            0.0014914810890331864,
+            0.020717859268188477,
+            -0.015468590892851353,
+            -0.0029178117401897907,
+            0.013048714958131313,
+            -0.005286497995257378,
+            0.007585379760712385,
+            0.0032901002559810877,
+            -0.005067778285592794,
+            -0.02114599198102951,
+            -0.026860620826482773,
+            -0.007999550551176071,
+            0.0011936502996832132,
+            2.35407078434946e-05,
+            -0.0037321930285543203,
+            -0.009372364729642868,
+            -0.004830444231629372,
+            -0.01089874841272831,
+            0.007855288684368134,
+            -0.007413196377456188,
+            -0.014928772114217281,
+            0.013988743536174297,
+            0.005226001143455505,
+            0.008436989970505238,
+            -0.031086096540093422,
+            0.013830521143972874,
+            0.020587557926774025,
+            -0.006961796432733536,
+            0.009237410500645638,
+            -0.015198681503534317,
+            -0.00025216734502464533,
+            -0.01685536466538906,
+            0.004043984692543745,
+            0.004632666241377592,
+            -0.01169916894286871,
+            0.005379570182412863,
+            0.007403889205306768,
+            -0.01913563348352909,
+            -0.006929221097379923,
+            -0.02060617320239544,
+            0.00492817023769021,
+            0.011578175239264965,
+            0.009000076912343502,
+            -0.01559889130294323,
+            -0.004886287730187178,
+            -0.03449253737926483,
+            0.016101481392979622,
+            0.017078738659620285,
+            -0.029001280665397644,
+            -0.014584405347704887,
+            0.009046612307429314,
+            0.013588533736765385,
+            -0.01771163009107113,
+            -0.012834648601710796,
+            -0.0009981987532228231,
+            0.005938002839684486,
+            -0.018744729459285736,
+            0.00533768767490983,
+            -0.0037647683639079332,
+            0.01832590624690056,
+            0.0014740300830453634,
+            0.024850262328982353,
+            -0.026581404730677605,
+            0.010107634589076042,
+            0.017004281282424927,
+            -0.027474896982312202,
+            0.0005968251498416066,
+            0.014658862724900246,
+            0.00042202402255497873,
+            0.00020286819199100137,
+            0.0004857202584389597,
+            -0.007594686932861805,
+            -0.008483526296913624,
+            0.004513998981565237,
+            -0.0014658862492069602,
+            -0.026432489976286888,
+            0.01950792223215103,
+            0.01410043053328991,
+            0.013104557991027832,
+            0.008799971081316471,
+            -0.011652632616460323,
+            0.0006020604632794857,
+            0.01573850028216839,
+            0.032742779701948166,
+            -0.0014891542959958315,
+            -0.015654735267162323,
+            0.010014562867581844,
+            0.026637248694896698,
+            0.0003402950242161751,
+            0.0008190348744392395,
+            -0.01603632979094982,
+            -0.026469718664884567,
+            0.038978613913059235,
+            0.018270062282681465,
+            0.01958237960934639,
+            -0.0018253774615004659,
+            -0.01781400851905346,
+            0.01455648336559534,
+            -0.004437214694917202,
+            0.0027595891151577234,
+            -0.015301060862839222,
+            -0.009418901056051254,
+            0.0026572097558528185,
+            0.026655862107872963,
+            -0.005374916363507509,
+            0.03382241725921631,
+            0.012052842415869236,
+            0.009423554874956608,
+            -0.035609401762485504,
+            0.0031528188847005367,
+            -0.02462689019739628,
+            0.011559560894966125,
+            -0.0094142472371459,
+            -0.0026944386772811413,
+            0.008590559475123882,
+            -0.02935495413839817,
+            -0.009032651782035828,
+            0.01011694222688675,
+            0.023063277825713158,
+            0.007366660516709089,
+            -0.023007433861494064,
+            0.020178040489554405,
+            0.0013134806649759412,
+            0.02151828072965145,
+            -0.012052842415869236,
+            0.012266908772289753,
+            -0.008744128048419952,
+            -0.017804700881242752,
+            -0.005170157644897699,
+            0.03374795988202095,
+            -0.008530061691999435,
+            -0.0036577354185283184,
+            0.006254448089748621,
+            -0.0279216431081295,
+            0.032612480223178864,
+            -0.0037182322703301907,
+            -0.0029154850635677576,
+            0.0041975537315011024,
+            -0.012006307020783424,
+            0.00991218350827694,
+            0.0094142472371459,
+            0.008115891367197037,
+            -0.018707500770688057,
+            -0.023714782670140266,
+            -0.04184523597359657,
+            -0.024533817544579506,
+            0.014733321033418179,
+            0.011503716930747032,
+            -0.015114916488528252,
+            0.03387825936079025,
+            -0.02192779816687107,
+            -0.016306240111589432,
+            -0.011447873897850513,
+            0.028684835880994797,
+            -0.03722885996103287,
+            0.00875343568623066,
+            -0.01350476872175932,
+            -0.020587557926774025,
+            -0.00148566416464746,
+            0.004900248255580664,
+            -0.002478045877069235,
+            -0.009176913648843765,
+            -0.02939218282699585,
+            0.0060636503621935844,
+            0.04999835789203644,
+            -0.0055936360731720924,
+            -0.0016415599966421723,
+            0.00418126629665494,
+            0.003774075536057353,
+            -0.014342417940497398,
+            -0.016501691192388535,
+            0.03486482426524162,
+            -0.03806650638580322,
+            0.0017520831897854805,
+            0.01845620572566986,
+            -0.003583277575671673,
+            0.00554244639351964,
+            0.004523306153714657,
+            0.003578623989596963,
+            0.012629889883100986,
+            -0.00727824168279767,
+            0.013579226098954678,
+            0.011056970804929733,
+            0.011624710634350777,
+            0.005291151348501444,
+            0.014910157769918442,
+            0.025185322389006615,
+            -0.01706012338399887,
+            0.024868877604603767,
+            0.025017792358994484,
+            0.00014222587924450636,
+            0.03581416234374046,
+            -0.019433464854955673,
+            -0.026004357263445854,
+            0.005626211408525705,
+            -0.011745704337954521,
+            -0.004711777437478304,
+            0.001653193961828947,
+            -0.0220953281968832,
+            -0.0031993549782782793,
+            -0.011401337571442127,
+            0.0011895783245563507,
+            0.014053894206881523,
+            -0.014724013395607471,
+            -0.0204944871366024,
+            0.015422054566442966,
+            0.013430310413241386,
+            0.02861037850379944,
+            0.01518937386572361,
+            -0.0034971858840435743,
+            0.02881513722240925,
+            -0.011996999382972717,
+            0.014789164066314697,
+            0.008190348744392395,
+            0.011913234367966652,
+            -0.0069571430794894695,
+            0.024329058825969696,
+            0.0019963975064456463,
+            0.009456129744648933,
+            0.001500788377597928,
+            -0.01692051626741886,
+            -0.0034297085367143154,
+            -0.034213319420814514,
+            -0.018344519659876823,
+            0.003041132353246212,
+            -0.00020417702035047114,
+            0.08748781681060791,
+            0.006980410777032375,
+            -0.02090400457382202,
+            0.003069054102525115,
+            0.0011866698041558266,
+            0.005212040152400732,
+            -0.018279369920492172,
+            0.015180067159235477,
+            -0.027456283569335938,
+            -0.020140811800956726,
+            -0.017832623794674873,
+            -0.024980563670396805,
+            0.0030969756189733744,
+            0.004339488688856363,
+            0.0102472435683012,
+            -0.027474896982312202,
+            0.0019824367482215166,
+            -0.006701194681227207,
+            -0.020717859268188477,
+            0.009027997963130474,
+            -0.01924731954932213,
+            0.004816483706235886,
+            0.011419951915740967,
+            -0.00728754885494709,
+            -0.01791638880968094,
+            0.007603994105011225,
+            0.005314419511705637,
+            0.0033878260292112827,
+            -0.01682744361460209,
+            -5.409818550106138e-05,
+            -0.00768775912001729,
+            0.004092847462743521,
+            0.009595737792551517,
+            -0.013821213506162167,
+            0.015505819581449032,
+            0.006366134621202946,
+            -0.008790664374828339,
+            0.024515202268958092,
+            0.00873482134193182,
+            -0.02241177298128605,
+            -0.0018463187152519822,
+            -0.011689861305058002,
+            0.005658786278218031,
+            -0.0016799522563815117,
+            0.004206860903650522,
+            -0.011354802176356316,
+            -0.016287624835968018,
+            -0.02693507820367813,
+            -0.0008167080814018846,
+            -0.006645351182669401,
+            -0.005374916363507509,
+            0.0036554085090756416,
+            -0.006380095612257719,
+            0.011764319613575935,
+            0.00553313922137022,
+            -0.01396082155406475,
+            0.01045200228691101,
+            -0.0005985702155157924,
+            0.013169708661735058,
+            -0.026879236102104187,
+            0.005314419511705637,
+            -0.026655862107872963,
+            0.009642274118959904,
+            0.01330001000314951,
+            -0.038866929709911346,
+            0.015533741563558578,
+            -0.007101404946297407,
+            0.015570970252156258,
+            -0.0014088796451687813,
+            -0.01828867755830288,
+            0.020550329238176346,
+            0.007887864485383034,
+            -0.008841853588819504,
+            0.012332059442996979,
+            0.025073636323213577,
+            0.00967019610106945,
+            0.03830849379301071,
+            -0.016259703785181046,
+            -0.0003152818826492876,
+            -0.0064312852919101715,
+            0.007036254275590181,
+            -0.005449374206364155,
+            -0.0021301887463778257,
+            -0.020475871860980988,
+            0.029243268072605133,
+            -0.008450950495898724,
+            0.007762216962873936,
+            0.02282129041850567,
+            -0.0051050069741904736,
+            -0.009418901056051254,
+            0.004507018718868494,
+            0.012099378742277622,
+            0.018167681992053986,
+            -0.0032342569902539253,
+            0.015096302144229412,
+            -0.015487205237150192,
+            0.012211065739393234,
+            0.008250845596194267,
+            0.0015240564243867993,
+            -0.027903029695153236,
+            -0.022970205172896385,
+            -0.0017858217470347881,
+            0.0012750884052366018,
+            0.002101103775203228,
+            0.012434438802301884,
+            0.00748300040140748,
+            -0.006822188384830952,
+            0.029652785509824753,
+            -0.005798394791781902,
+            0.012099378742277622,
+            -0.0002384973777225241,
+            0.021481052041053772,
+            0.01862373575568199,
+            0.025017792358994484,
+            -0.0010325191542506218,
+            -0.01009832788258791,
+            0.005226001143455505,
+            -0.006859417073428631,
+            0.01991743966937065,
+            0.010759140364825726,
+            -0.01328139565885067,
+            -0.023752011358737946,
+            0.015049765817821026,
+            0.00011735816224245355,
+            -0.0018963449401780963,
+            0.012695040553808212,
+            -0.02336110919713974,
+            0.0009510809904895723,
+            -0.035404644906520844,
+            0.015077687799930573,
+            -0.04240366816520691,
+            -0.01672506518661976,
+            0.005221347324550152,
+            -0.014258652925491333,
+            -0.01026585791260004,
+            0.017795395106077194,
+            0.017748858779668808,
+            -0.008213616907596588,
+            0.014975308440625668,
+            -0.008697591722011566,
+            -0.009753961116075516,
+            0.013811906799674034,
+            0.012387902475893497,
+            -0.02315635047852993,
+            0.015301060862839222,
+            0.005212040152400732,
+            0.0058914669789373875,
+            -0.021871954202651978,
+            -0.02012219838798046,
+            0.006840802729129791,
+            0.01706012338399887,
+            -0.0003562917991075665,
+            0.0016194552881643176,
+            0.006631390191614628,
+            0.003539068391546607,
+            0.00614741537719965,
+            0.003271485911682248,
+            0.0003123733913525939,
+            -0.000290705036604777,
+            0.02274683304131031,
+            0.011140735819935799,
+            -0.015524433925747871,
+            0.0071293264627456665,
+            -0.009335136041045189,
+            0.0005508707836270332,
+            0.009377018548548222,
+            -1.0870535334106535e-05,
+            0.014649556018412113,
+            -0.008176388218998909,
+            0.013653683476150036,
+            0.02043864317238331,
+            -0.007445771712809801,
+            -0.003076034365221858,
+            -0.008948886767029762,
+            0.006891992408782244,
+            0.016082866117358208,
+            0.025148093700408936,
+            -0.017748858779668808,
+            -0.0002715089067351073,
+            -0.009032651782035828,
+            -0.009437515400350094,
+            -0.026842007413506508,
+            0.017050817608833313,
+            0.020289728417992592,
+            0.0073573533445596695,
+            0.017795395106077194,
+            -0.00849748682230711,
+            0.011326880194246769,
+            -0.010191399604082108,
+            -0.0023244768381118774,
+            -0.00374150020070374,
+            -0.012648504227399826,
+            0.002643248997628689,
+            0.023826468735933304,
+            -0.013914286158978939,
+            0.00790182501077652,
+            -0.0013646703446283937,
+            0.03681934252381325,
+            -0.011233807541429996,
+            0.005612250417470932,
+            0.02019665576517582,
+            -0.004979359917342663,
+            -0.011773626320064068,
+            0.02295159175992012,
+            0.008841853588819504,
+            0.0035530291497707367,
+            0.005635518580675125,
+            0.024031227454543114,
+            0.011224500834941864,
+            0.0016043310752138495,
+            0.000546508003026247,
+            -0.04716896265745163,
+            0.00888838991522789,
+            -0.001733468729071319,
+            -0.010638146661221981,
+            0.017525484785437584,
+            0.010619531385600567,
+            0.009395632892847061,
+            -0.003269159235060215,
+            0.019191475585103035,
+            -0.012266908772289753,
+            0.012508896179497242,
+            0.033636271953582764,
+            0.0034157477784901857,
+            -0.019014639779925346,
+            0.0021848685573786497,
+            0.018009459599852562,
+            -0.0024222026113420725,
+            0.00217090779915452,
+            0.006705848034471273,
+            -0.017190424725413322,
+            0.0004281318688299507,
+            0.0036903105210512877,
+            0.029224654659628868,
+            -0.015180067159235477,
+            -0.027828572317957878,
+            0.0004447103419806808,
+            -0.005872852168977261,
+            -0.003073707688599825,
+            0.008841853588819504,
+            0.0032924271654337645,
+            -0.0016217821976169944,
+            -0.012722962535917759,
+            0.0017323052743449807,
+            0.007743602618575096,
+            0.010787061415612698,
+            0.003583277575671673,
+            -0.005965924356132746,
+            -0.008916311897337437,
+            0.021462436765432358,
+            -0.006045036017894745,
+            -0.0038834353908896446,
+            -0.0035925847478210926,
+            -0.006980410777032375,
+            -0.010172785259783268,
+            -0.012778805568814278,
+            0.011122121475636959,
+            -0.001270434819161892,
+            -0.01437033899128437,
+            -0.017004281282424927,
+            0.010461308993399143,
+            -0.0067756520584225655,
+            0.025725141167640686,
+            0.0001714563404675573,
+            -0.008032126352190971,
+            0.003369211684912443,
+            -0.015319675207138062,
+            -0.010535767301917076,
+            0.007752909790724516,
+            -0.002316333120688796,
+            -0.00495609175413847,
+            0.01091736275702715,
+            -0.0032575251534581184,
+            0.018958795815706253,
+            -0.0012913760729134083,
+            -0.008385800756514072,
+            0.002873602556064725,
+            0.0025571573060005903,
+            -0.012806727550923824,
+            -0.0006579037290066481,
+            -0.02131352201104164,
+            -0.013476846739649773,
+            -0.0014519254909828305,
+            0.014091122895479202,
+            -0.010777754709124565,
+            0.005868198815733194,
+            0.0010674211662262678,
+            0.006896646227687597,
+            -0.016371389850974083,
+            -0.010237935930490494,
+            -0.0012890491634607315,
+            0.024198757484555244,
+            0.011001127772033215,
+            -0.02319357916712761,
+            -0.010824290104210377,
+            -0.01007971353828907,
+            0.018055995926260948,
+            -0.013635069131851196,
+            0.004490730818361044,
+            0.013234859332442284,
+            -0.016194554045796394,
+            -0.0009208325645886362,
+            -0.012601968832314014,
+            -0.02220701426267624,
+            0.0003839225973933935,
+            0.032705552875995636,
+            0.012015613727271557,
+            0.017953617498278618,
+            -0.008078662678599358,
+            -0.00011510407057357952,
+            0.03168175742030144,
+            0.0032505446579307318,
+            0.0018253774615004659,
+            -0.01437033899128437,
+            0.009381672367453575,
+            -0.009493358433246613,
+            0.018875030800700188,
+            0.006612775847315788,
+            -0.00827411375939846,
+            0.003231930313631892,
+            -0.02233731560409069,
+            -0.003841552883386612,
+            -0.005244615487754345,
+            -0.009698117151856422,
+            0.012825341895222664,
+            -0.007040907628834248,
+            -0.015543048270046711,
+            0.0005316746537573636,
+            -0.007124672643840313,
+            -0.015980487689375877,
+            -0.008758088573813438,
+            -0.011373416520655155,
+            -0.030062302947044373,
+            -0.027567969635128975,
+            0.0010226302547380328,
+            -0.005868198815733194,
+            -0.028461461886763573,
+            -0.019359005615115166,
+            -0.022393157705664635,
+            0.009097802452743053,
+            -0.012369288131594658,
+            -0.02101569063961506,
+            0.007157247979193926,
+            0.005095699802041054,
+            -0.01842828467488289,
+            -0.026469718664884567,
+            -0.014435489661991596,
+            -0.016334161162376404,
+            0.025278395041823387,
+            0.012713654898107052,
+            -0.0008283421047963202,
+            -0.005174811463803053,
+            0.012220372445881367,
+            0.0032784664072096348,
+            0.01127103716135025,
+            -0.03268693760037422,
+            -0.03129085525870323,
+            -0.011187272146344185,
+            -0.00430691335350275,
+            0.0007451588753610849,
+            -0.03300338238477707,
+            0.002910831244662404,
+            0.003069054102525115,
+            -0.03827126696705818,
+            -0.008706899359822273,
+            0.015040459111332893,
+            -0.0009749307064339519,
+            0.011261729523539543,
+            0.00635217409580946,
+            0.026618633419275284,
+            0.02332388050854206,
+            0.0014437816571444273,
+            4.0937200537882745e-05,
+            0.012211065739393234,
+            -0.014705399051308632,
+            0.0014961347915232182,
+            0.005775126628577709,
+            0.008585905656218529,
+            -0.0004086449043825269,
+            0.015031151473522186,
+            -0.001642723334953189,
+            -0.015403440222144127,
+            0.017311418429017067,
+            0.01541274692863226,
+            0.00934909749776125,
+            0.002934099407866597,
+            0.0009121070033870637,
+            0.0038019970525056124,
+            0.0037880362942814827,
+            0.006770998705178499,
+            -0.01227621641010046,
+            -0.006417324300855398,
+            0.016901900991797447,
+            -0.011829469352960587,
+            -0.013942207209765911,
+            0.019805751740932465,
+            -0.00024489607312716544,
+            -0.023826468735933304,
+            0.02758658491075039,
+            0.006328905932605267,
+            -0.004479097202420235,
+            -0.02537146769464016,
+            -0.0018323578406125307,
+            -0.024664118885993958,
+            0.015543048270046711,
+            0.005407491698861122,
+            0.0015449975617229939,
+            0.022597916424274445,
+            0.01108489278703928,
+            0.0038206116296350956,
+            -0.013206937350332737,
+            0.006231179926544428,
+            0.01106627844274044,
+            0.015896722674369812,
+            -0.03704271465539932,
+            -0.02455243282020092,
+            0.012285523116588593,
+            -0.01841897703707218,
+            -0.005244615487754345,
+            0.012788113206624985,
+            0.0009766757721081376,
+            -0.02019665576517582,
+            0.0031667796429246664,
+            0.003608872415497899,
+            0.004758313298225403,
+            0.012192451395094395,
+            0.016631992533802986,
+            0.0122948307543993,
+            -0.0018521357560530305,
+            -0.022970205172896385,
+            0.006007806863635778,
+            -0.008241538889706135,
+            0.0007137469947338104,
+            -0.023342493921518326,
+            -0.0018102532485499978,
+            0.010610224679112434,
+            0.00767845194786787,
+            0.025874055922031403,
+            0.030676579102873802,
+            0.027903029695153236,
+            0.013244166038930416,
+            -0.004942130763083696,
+            -0.014416875317692757,
+            -0.020959846675395966,
+            0.0019917439203709364,
+            0.002347744768485427,
+            -0.011568867601454258,
+            -0.001557794981636107,
+            0.017488256096839905,
+            0.004011409357190132,
+            0.021816110238432884,
+            -0.005793740972876549,
+            0.017460335046052933,
+            -3.364776057424024e-05,
+            -0.005212040152400732,
+            -0.0008463748381473124,
+            -0.003173760138452053,
+            0.013244166038930416,
+            0.006650005001574755,
+            0.026004357263445854,
+            0.00022031922708265483,
+            0.003708925098180771,
+            0.005481949541717768,
+            -0.0007905315142124891,
+            -0.0008289237739518285,
+            -0.007380621042102575,
+            -0.003539068391546607,
+            -0.009847032837569714,
+            0.034678682684898376,
+            0.025278395041823387,
+            -0.02250484563410282,
+            -0.015124224126338959,
+            -0.012713654898107052,
+            0.02639526128768921,
+            -0.010833597742021084,
+            0.006524357479065657,
+            0.010796369053423405,
+            0.008660363033413887,
+            0.01065676100552082,
+            0.000908035144675523,
+            0.005719283130019903,
+            0.0028898902237415314,
+            0.022672373801469803,
+            -0.012611275538802147,
+            0.008669670671224594,
+            0.003920664079487324,
+            0.008357878774404526,
+            -0.015077687799930573,
+            -0.001618291949853301,
+            0.005872852168977261,
+            -0.014919465407729149,
+            0.006170683074742556,
+            -0.004118442535400391,
+            0.006207912229001522,
+            0.011559560894966125,
+            0.0015775728970766068,
+            -0.01607356034219265,
+            -0.017935002222657204,
+            0.012927721254527569,
+            -0.009781882166862488,
+            0.00916295312345028,
+            -0.018028074875473976,
+            -0.011606096290051937,
+            0.017897773534059525,
+            -0.006119493395090103,
+            -0.011931848712265491,
+            0.0026292880065739155,
+            0.00411146180704236,
+            -5.733389480155893e-05,
+            0.004009082447737455,
+            0.007752909790724516,
+            0.0020592212677001953,
+            0.00535630201920867,
+            0.005947310011833906,
+            -0.002478045877069235,
+            0.002440816955640912,
+            -0.0019789463840425014,
+            0.035739704966545105,
+            -0.009847032837569714,
+            0.006617429666221142,
+            -0.0022372216917574406,
+            -0.0012099378509446979,
+            -0.04154740646481514,
+            0.027772728353738785,
+            -0.006263755261898041,
+            0.015319675207138062,
+            0.00801816489547491,
+            -0.003401787020266056,
+            -0.010768447071313858,
+            0.008353224955499172,
+            0.01050784531980753,
+            -0.02135075069963932,
+            0.015142838470637798,
+            -0.024198757484555244,
+            -0.004218494985252619,
+            -0.002433836692944169,
+            -0.01108489278703928,
+            0.015496511943638325,
+            -0.01459371205419302,
+            0.008092623203992844,
+            -0.01091736275702715,
+            -0.0073713138699531555,
+            0.010144864208996296,
+            -0.0072549739852547646,
+            0.011010434478521347,
+            -0.042924873530864716,
+            0.0067756520584225655,
+            -0.004821137059479952,
+            -0.0010517152259126306,
+            -0.0006689560250379145,
+            -0.02270960435271263,
+            -0.031067483127117157,
+            -0.000392648100387305,
+            0.004853712394833565,
+            -0.005384223535656929,
+            0.00748300040140748,
+            0.005407491698861122,
+            0.004355776589363813,
+            0.0032621787395328283,
+            -0.01954515092074871,
+            0.025390081107616425,
+            0.03665181249380112,
+            -0.011829469352960587,
+            0.013383775018155575,
+            0.007161901798099279,
+            0.008148466236889362,
+            -0.03622367978096008,
+            -0.02352863922715187,
+            -0.008450950495898724,
+            0.009181567467749119,
+            0.02591128461062908,
+            -0.004679202102124691,
+            0.01559889130294323,
+            0.009437515400350094,
+            0.0050910464487969875,
+            -0.02870344929397106,
+            -0.015403440222144127,
+            -0.008046086877584457,
+            -0.015654735267162323,
+            -0.0022442021872848272,
+            -0.0033017343375831842,
+            0.014696091413497925,
+            -0.00473271869122982,
+            -0.018260754644870758,
+            -0.011941156350076199,
+            -0.006864070892333984,
+            -0.0037368466146290302,
+            -0.010154170915484428,
+            0.023547252640128136,
+            -0.010833597742021084,
+            -0.015589584596455097,
+            0.02512947842478752,
+            -0.001938227447681129,
+            0.005793740972876549,
+            0.005235308315604925,
+            0.00750161474570632,
+            -0.003304061247035861,
+            -0.004590783733874559,
+            -0.007673798594623804,
+            0.017627865076065063,
+            0.004202207550406456,
+            0.02874067798256874,
+            -0.004532613325864077,
+            0.00770637346431613,
+            0.0016764620086178184,
+            0.004355776589363813,
+            -0.002505967626348138,
+            0.008599866181612015,
+            0.006003153510391712,
+            -0.003648428013548255,
+            0.0071991304866969585,
+            -0.01350476872175932,
+            0.002410568529739976,
+            0.024943334981799126,
+            0.005984539166092873,
+            0.010414772666990757,
+            -0.012629889883100986,
+            -0.012667118571698666,
+            -0.003418074455112219,
+            -0.016203859820961952,
+            0.015924643725156784,
+            -0.020662017166614532,
+            0.018670272082090378,
+            -0.017460335046052933,
+            -0.001818397082388401,
+            0.0043930052779614925,
+            0.015198681503534317,
+            0.021090148016810417,
+            -0.004346469417214394,
+            0.026469718664884567,
+            0.013728141784667969,
+            0.0035227807238698006,
+            0.004970052745193243,
+            -0.022653760388493538,
+            -0.005514524411410093,
+            -0.011094199493527412,
+            0.003913683816790581,
+            -0.008581251837313175,
+            -0.016734370961785316,
+            -0.0010197217343375087,
+            0.007315470837056637,
+            -0.005398184526711702,
+            0.003701944602653384,
+            0.012657811865210533,
+            0.0015484878094866872,
+            0.0028293931391090155,
+            0.014230730943381786,
+            0.01651099883019924,
+            0.007510922383517027,
+            -0.007622608914971352,
+            0.0040323506109416485,
+            0.006622083019465208,
+            -0.0046280124224722385,
+            0.0009504992631264031,
+            0.017395183444023132,
+            0.01438895333558321,
+            -0.004060272127389908,
+            0.0053656091913580894,
+            -0.0027805303689092398,
+            -0.02676755003631115,
+            -0.014267959631979465,
+            -0.0003990468103438616,
+            -0.0005968251498416066,
+            -0.003506493056192994,
+            -0.015161452814936638,
+            0.004483750555664301,
+            0.012155221775174141,
+            -0.0122948307543993,
+            0.0035181271377950907,
+            0.008399761281907558,
+            -0.009949412196874619,
+            0.0021976660937070847,
+            -0.0034785715397447348,
+            -0.014249345287680626,
+            -0.009791189804673195,
+            0.027623813599348068,
+            -0.01011694222688675,
+            0.013746756128966808,
+            0.013635069131851196,
+            0.00039875597576610744,
+            -0.006440592464059591,
+            -0.0013041734928265214,
+            0.019061176106333733,
+            0.01746964268386364,
+            -0.001688096090219915,
+            -0.00036327220732346177,
+            -0.007552804425358772,
+            -0.006161375902593136,
+            0.01350476872175932,
+            -0.01634346880018711,
+            -0.004311567172408104,
+            0.009577123448252678,
+            -0.00846025813370943,
+            -0.004593110177665949,
+            -0.024533817544579506,
+            0.0021522934548556805,
+            0.009135031141340733,
+            0.0035227807238698006,
+            -0.0068314955569803715,
+            -0.001905652228742838,
+            -0.002934099407866597,
+            0.0058030481450259686,
+            -0.00118550646584481,
+            -0.00687803141772747,
+            -0.016836751252412796,
+            -0.0024175490252673626,
+            0.0070920973084867,
+            -0.006184644065797329,
+            -0.001833521295338869,
+            0.0036251600831747055,
+            -0.006082264706492424,
+            -0.015673348680138588,
+            0.03244495019316673,
+            9.452639642404392e-05,
+            0.021332135424017906,
+            0.004132403060793877,
+            0.0061334543861448765,
+            0.0027246871031820774,
+            0.007222398649901152,
+            0.02561345510184765,
+            0.009614353068172932,
+            0.006747730541974306,
+            0.01621316745877266,
+            0.014975308440625668,
+            -0.014240038581192493,
+            0.0072549739852547646,
+            0.012853262946009636,
+            -0.010163478553295135,
+            0.00417195912450552,
+            0.009251371026039124,
+            -0.0007649366743862629,
+            -0.003988141659647226,
+            -0.0037228858564049006,
+            0.016799522563815117,
+            0.003071380779147148,
+            -0.0008132178918458521,
+            0.012183143757283688,
+            0.01166194025427103,
+            -0.018754037097096443,
+            -0.00033796823117882013,
+            -0.0015135857975110412,
+            0.002343091182410717,
+            -0.005882159806787968,
+            -0.026581404730677605,
+            -0.008320650085806847,
+            0.0033854993525892496,
+            -0.0030434592626988888,
+            0.03091856651008129,
+            -0.00950266607105732,
+            -0.03354320302605629,
+            -0.01563611999154091,
+            -0.001225062063895166,
+            -0.01692982390522957,
+            -0.007966975681483746,
+            0.01276949793100357,
+            0.0295224841684103,
+            0.012415824458003044,
+            -0.013802599161863327,
+            0.024664118885993958,
+            -0.0028131057042628527,
+            0.01750687137246132,
+            -0.00870224554091692,
+            -0.009083840996026993,
+            -0.004509345628321171,
+            0.022113941609859467,
+            -0.008744128048419952,
+            -0.009605045430362225,
+            0.013579226098954678,
+            0.016985666006803513,
+            0.00842768233269453,
+            0.02323080785572529,
+            0.013644376769661903,
+            -0.007022293284535408,
+            -0.014891543425619602,
+            0.003808977548032999,
+            -0.008595212362706661,
+            -0.020252499729394913,
+            0.007324778009206057,
+            -0.014900850132107735,
+            0.005416798871010542,
+            -0.0021348423324525356,
+            -0.023696167394518852,
+            0.005537792574614286,
+            0.0053656091913580894,
+            -0.012043535709381104,
+            -0.012546124868094921,
+            0.000147243044921197,
+            -0.009507319889962673,
+            0.014463411644101143,
+            0.0068314955569803715,
+            -0.00023253493418451399,
+            -0.009251371026039124,
+            0.017832623794674873,
+            0.011001127772033215,
+            0.009968026541173458,
+            -0.00792043935507536,
+            0.0033924796152859926,
+            0.008934926241636276,
+            0.007590033579617739,
+            -0.0014495986979454756,
+            -0.0056634400971233845,
+            0.037880364805459976,
+            0.0030620736069977283,
+            0.012527510523796082,
+            -0.0017451026942580938,
+            0.009865647181868553,
+            0.01086151972413063,
+            0.015291753225028515,
+            0.010256550274789333,
+            -0.003646101336926222,
+            -0.0038275918923318386,
+            0.013458232395350933,
+            -0.012024921365082264,
+            0.0055052172392606735,
+            0.010275164619088173,
+            -0.0046675680205225945,
+            -0.005384223535656929,
+            0.010647453367710114,
+            0.00014309842663351446,
+            0.01812114752829075,
+            -0.005900774151086807,
+            0.009288599714636803,
+            0.018335212022066116,
+            0.009577123448252678,
+            -0.019098404794931412,
+            0.013728141784667969,
+            -0.01556166261434555,
+            0.029950616881251335,
+            0.01917286217212677,
+            -0.01812114752829075,
+            -0.0018975083949044347,
+            -0.011019742116332054,
+            0.009763267822563648,
+            -0.04221752658486366,
+            -0.01190392766147852,
+            0.0023919541854411364,
+            -0.004090521018952131,
+            -0.018884338438510895,
+            -0.027028150856494904,
+            0.0044558290392160416,
+            -0.004430233966559172,
+            -0.009353750385344028,
+            -0.008278767578303814,
+            0.0006509232916869223,
+            0.015924643725156784,
+            -0.005221347324550152,
+            -0.0077715241350233555,
+            -0.024310443550348282,
+            0.014993922784924507,
+            0.00886977557092905,
+            -0.004176612477749586,
+            -0.024403516203165054,
+            0.01455648336559534,
+            0.01603632979094982,
+            -0.00687803141772747,
+            0.01190392766147852,
+            -0.024049842730164528,
+            0.00197545625269413,
+            -0.005700668785721064,
+            -0.01654822751879692,
+            -0.015263832174241543,
+            0.0010999963851645589,
+            -0.012164529412984848,
+            -0.003313368419185281,
+            -0.014119044877588749,
+            -0.009083840996026993,
+            -0.0028945438098162413,
+            0.020475871860980988,
+            0.0025641375686973333,
+            -0.01541274692863226,
+            0.016390005126595497,
+            0.007869250141084194,
+            -0.0229888204485178,
+            -0.01631554774940014,
+            0.0046605877578258514,
+            -0.029876159504055977,
+            0.010805675759911537,
+            0.007869250141084194,
+            0.006826841738075018,
+            -0.02189056947827339,
+            0.007683105766773224,
+            0.02241177298128605,
+            0.007994897663593292,
+            0.0017660439480096102,
+            0.01913563348352909,
+            0.008185694925487041,
+            0.012667118571698666,
+            0.0019696392118930817,
+            -3.335691189931822e-06,
+            -0.005849584471434355,
+            -0.004148690961301327,
+            -0.004714104346930981,
+            0.03802927955985069,
+            -0.01617593877017498,
+            0.009498012252151966,
+            -0.022560687735676765,
+            0.0002395153569523245,
+            -0.01921009086072445,
+            -0.011298958212137222,
+            -0.015980487689375877,
+            0.0008010021410882473,
+            0.02151828072965145,
+            -0.007892518304288387,
+            -0.0002912867348641157,
+            -0.01583157107234001,
+            -0.0058356234803795815,
+            -0.006542971823364496,
+            0.02036418579518795,
+            -0.012741576880216599,
+            0.0018626062665134668,
+            -0.017311418429017067,
+            0.0010249570477753878,
+            0.026693090796470642,
+            0.005556406918913126,
+            -0.013225551694631577,
+            -0.026618633419275284,
+            -0.016194554045796394,
+            -0.009158299304544926,
+            -0.01316040102392435,
+            -0.005239961668848991,
+            -0.019228706136345863,
+            -0.011364108882844448,
+            -0.006473167799413204,
+            0.00654762564226985,
+            0.004039331339299679,
+            -0.015012537129223347,
+            -0.004997974261641502,
+            -0.024664118885993958,
+            -0.0202338844537735,
+            -0.010014562867581844,
+            -0.009470091201364994,
+            0.0013844481436535716,
+            0.02853591926395893,
+            0.0028549879789352417,
+            0.030397363007068634,
+            -0.00415567122399807,
+            0.008790664374828339,
+            0.009828418493270874,
+            0.007203784305602312,
+            -0.01520798821002245,
+            0.007487654220312834,
+            0.014584405347704887,
+            -0.008530061691999435,
+            0.0057890876196324825,
+            -0.0122948307543993,
+            -0.0007463222718797624,
+            0.005249268840998411,
+            0.004970052745193243,
+            0.011261729523539543,
+            -0.007548151072114706,
+            -0.01580365002155304,
+            0.013495461083948612,
+            -0.02040141448378563,
+            0.0011971404310315847,
+            0.00031295508961193264,
+            -0.003236583899706602,
+            0.003632140578702092,
+            0.018102532252669334,
+            -0.01336515974253416,
+            0.001435637823306024,
+            0.01999189704656601,
+            -0.012536818161606789,
+            0.01316040102392435,
+            0.003013210603967309,
+            0.0010214668000116944,
+            0.009511973708868027,
+            0.0011215193662792444,
+            0.005156196653842926,
+            0.0005630864761769772,
+            -0.019228706136345863,
+            -0.006966450251638889,
+            0.00831134244799614,
+            -0.024608274921774864,
+            -0.01128965150564909,
+            0.011177964508533478,
+            0.006254448089748621,
+            -0.017832623794674873,
+            -0.012788113206624985,
+            -0.023863697424530983,
+            0.01166194025427103,
+            -0.005467988550662994,
+            0.023993998765945435,
+            -0.015105608850717545,
+            0.009204835630953312,
+            -0.007738948799669743,
+            0.021164605394005775,
+            -0.0122948307543993,
+            0.005212040152400732,
+            0.0013576899655163288,
+            -0.01665060594677925,
+            0.009567816741764545,
+            -0.012285523116588593,
+            -0.0017567367758601904,
+            0.042775958776474,
+            0.006952489260584116,
+            0.0074271573685109615,
+            0.015068380162119865,
+            -0.022020868957042694,
+            0.01852135732769966,
+            0.002450124127790332,
+            0.018502742052078247,
+            0.010572995990514755,
+            -0.019265934824943542,
+            0.01695774495601654,
+            0.026823392137885094,
+            0.01982436701655388,
+            0.006221872754395008,
+            -0.01147579587996006,
+            0.027903029695153236,
+            0.012862570583820343,
+            0.0006660475628450513,
+            0.0022569994907826185,
+            0.0016846058424562216,
+            -0.011038356460630894,
+            -0.010842905379831791,
+            0.009018691256642342,
+            -0.017730243504047394,
+            -0.013272088021039963,
+            -0.0030923220328986645,
+            0.003080687951296568,
+            -0.010219321586191654,
+            -0.0036996176932007074,
+            0.006719809025526047,
+            0.007399235386401415,
+            0.011419951915740967,
+            0.0016020042821764946,
+            0.01611078903079033,
+            -0.004774601198732853,
+            -0.01893087476491928,
+            0.0002504222502466291,
+            -0.0032598518300801516,
+            0.014947386458516121,
+            -0.029299112036824226,
+            -0.007911132648587227,
+            -0.0004516907501965761,
+            0.005267883650958538,
+            0.006035728845745325,
+            0.007832021452486515,
+            -0.015096302144229412,
+            -0.005682054441422224,
+            -0.003113263286650181,
+            0.016120094805955887,
+            0.0016811155946925282,
+            0.023212192580103874,
+            0.015608198940753937,
+            -0.009968026541173458,
+            -0.006328905932605267,
+            0.0032668323256075382,
+            -0.010982513427734375,
+            0.00017320144979748875,
+            0.0025943859945982695,
+            0.011122121475636959,
+            -0.018847109749913216,
+            -0.008208963088691235,
+            0.025594839826226234,
+            0.02015942707657814,
+            0.00357629731297493,
+            0.026748934760689735,
+            -0.014174887910485268,
+            0.008227577432990074,
+            0.019842982292175293,
+            0.014407568611204624,
+            -0.010572995990514755,
+            -0.010731218382716179,
+            -0.0046675680205225945,
+            0.001287885825149715,
+            -0.0027549355290830135,
+            -0.005179464817047119,
+            0.006161375902593136,
+            -0.022467615082859993,
+            -0.016259703785181046,
+            0.0012157548917457461,
+            -0.013793292455375195,
+            0.0032342569902539253,
+            0.009358404204249382,
+            0.018502742052078247,
+            -0.011736397631466389,
+            0.01787916012108326,
+            -0.026879236102104187,
+            -0.003480898216366768,
+            0.010256550274789333,
+            0.0019545149989426136,
+            -0.016808830201625824,
+            -0.003911356907337904,
+            -0.002536216052249074,
+            -0.009218796156346798,
+            -0.006989717949181795,
+            0.01671575754880905,
+            0.012397210113704205,
+            0.015059073455631733,
+            -0.008195002563297749,
+            0.004423253703862429,
+            0.0023547252640128136,
+            0.009395632892847061,
+            -0.0018847109749913216,
+            -0.009442169219255447,
+            0.006696540862321854,
+            0.0017509197350591421,
+            -0.005863544996827841,
+            -0.005281844176352024,
+            0.023063277825713158,
+            -0.026134658604860306,
+            0.003641447750851512,
+            -0.016501691192388535,
+            0.011410645209252834,
+            0.012257601134479046,
+            0.002092959824949503,
+            -0.005598289426416159,
+            -0.006994371768087149,
+            0.013458232395350933,
+            -0.06496435403823853,
+            -0.008432336151599884,
+            -0.0015938604483380914,
+            0.027623813599348068,
+            -0.01794430986046791,
+            0.003169106552377343,
+            0.008404415100812912,
+            0.015422054566442966,
+            -0.007189823314547539,
+            0.007101404946297407,
+            0.016324853524565697,
+            -0.02261653169989586,
+            -0.011326880194246769,
+            0.00187075010035187,
+            -0.004974706098437309,
+            -0.011838776990771294,
+            0.0010470616398379207,
+            0.004500038456171751,
+            0.018940182402729988,
+            0.0015496512642130256,
+            0.011401337571442127,
+            -0.023510023951530457,
+            -0.013393081724643707,
+            -0.00436042994260788,
+            0.009060573764145374,
+            -0.005910081323236227,
+            0.0040323506109416485,
+            0.00931652169674635,
+            -0.03136531263589859,
+            0.011224500834941864,
+            0.008450950495898724,
+            -0.007241012994199991,
+            0.009605045430362225,
+            0.011326880194246769,
+            0.008474218659102917,
+            0.005267883650958538,
+            0.0011447874130681157,
+            -0.03210989013314247,
+            0.0043162209913134575,
+            0.013337238691747189,
+            -0.009200181812047958,
+            -0.013206937350332737,
+            -0.009977334178984165,
+            -0.0012366961454972625,
+            0.019973281770944595,
+            -0.004530286882072687,
+            -0.0018742403481155634,
+            0.014640248380601406,
+            0.006021767854690552,
+            0.01455648336559534,
+            -0.013337238691747189,
+            0.004942130763083696,
+            -0.010666067712008953,
+            -0.00854867696762085,
+            -0.0014100429834797978,
+            -0.0001836720621213317,
+            -0.0028968704864382744,
+            -0.038159579038619995,
+            0.00965158175677061,
+            -0.0036693692672997713,
+            -0.0197871383279562,
+            -0.01248097512871027,
+            -0.013411696068942547,
+            -0.004532613325864077,
+            -0.010870826430618763,
+            0.006854763720184565,
+            0.0031877208966761827,
+            0.009135031141340733,
+            -0.01336515974253416,
+            0.02496195025742054,
+            -0.014044586569070816,
+            -0.009977334178984165,
+            -0.009107109159231186,
+            -0.0005622139433398843,
+            0.002806125208735466,
+            0.0149846151471138,
+            -0.0038624941371381283,
+            -0.003204008564352989,
+            0.021853338927030563,
+            0.008013512007892132,
+            0.006114840041846037,
+            -0.01188531331717968,
+            -0.018875030800700188,
+            -0.023137735202908516,
+            -0.0029015240725129843,
+            -0.021946411579847336,
+            0.0007562111713923514,
+            0.004853712394833565,
+            0.02377062663435936,
+            0.007720334455370903,
+            -9.13997573661618e-05,
+            0.00436042994260788,
+            0.0038206116296350956,
+            0.019452078267931938,
+            -0.008371839299798012,
+            -0.004046311601996422,
+            -0.0071432869881391525,
+            -0.002633941825479269,
+            0.03482759743928909,
+            -0.008423029445111752,
+            0.00037955984589643776,
+            0.0013239512918516994,
+            -0.008879082277417183,
+            -0.007590033579617739,
+            0.010703296400606632,
+            -0.02881513722240925,
+            -0.014919465407729149,
+            -0.00036181797622703016,
+            0.0024873530492186546,
+            -0.010749832727015018,
+            0.02073647454380989,
+            -0.008502140641212463,
+            0.0023023721296340227,
+            0.009256024844944477,
+            0.024161528795957565,
+            0.022430386394262314,
+            0.023137735202908516,
+            0.007315470837056637,
+            -0.003671696176752448,
+            -0.002410568529739976,
+            -0.0067616915330290794,
+            -0.02496195025742054,
+            0.0161387100815773,
+            -0.0040951743721961975,
+            0.03404578939080238,
+            -0.03428777679800987,
+            0.0012413497315719724,
+            -0.007380621042102575,
+            -0.023472795262932777,
+            -0.009986640885472298,
+            0.023174963891506195,
+            -0.01045200228691101,
+            0.023900926113128662,
+            -0.0003123733913525939,
+            0.01556166261434555,
+            0.007906478829681873,
+            0.008441643789410591,
+            -0.01930316351354122,
+            0.011810855008661747,
+            -0.0023372743744403124,
+            0.01822352595627308,
+            0.009307214990258217,
+            -0.004607071168720722,
+            0.008832546882331371,
+            0.011689861305058002,
+            -0.01518937386572361,
+            -0.011196578852832317,
+            -0.0029015240725129843,
+            0.0018474820535629988,
+            -0.009847032837569714,
+            0.008553329855203629,
+            -0.002062711399048567,
+            -0.01600840874016285,
+            0.02537146769464016,
+            -0.01593395136296749,
+            0.010368237271904945,
+            0.004904902074486017,
+            0.00647782115265727,
+            0.01736726239323616,
+            -0.001187833258882165,
+            -0.027400439605116844,
+            0.0002552212681621313,
+            -0.03222157806158066,
+            -0.012341366149485111,
+            0.001407716190442443,
+            0.006566239986568689,
+            -0.0030434592626988888,
+            -0.010666067712008953,
+            -0.01883780211210251,
+            -0.0060310750268399715,
+            0.024571046233177185,
+            0.03439946472644806,
+            0.007743602618575096,
+            0.011745704337954521,
+            -0.011122121475636959,
+            -0.00967019610106945,
+            -0.006207912229001522,
+            -0.01917286217212677,
+            -0.009004729799926281,
+            -0.005863544996827841,
+            -0.00813915953040123,
+            0.011531638912856579,
+            -0.011568867601454258,
+            0.016538919880986214,
+            0.030118146911263466,
+            -0.031830672174692154,
+            -0.008530061691999435,
+            0.01570127159357071,
+            0.008651056326925755,
+            0.013253473676741123,
+            0.008804624900221825,
+            0.007631916087120771,
+            -0.011233807541429996,
+            -0.005817009136080742,
+            0.000602642132434994,
+            0.0062032584100961685,
+            -0.013802599161863327,
+            0.02162996679544449,
+            0.0010011072736233473,
+            0.002340764505788684,
+            0.004704797174781561,
+            0.01128965150564909,
+            -0.012760191224515438,
+            0.00357629731297493,
+            -0.007613301742821932,
+            -0.02537146769464016,
+            -0.026674477383494377,
+            -0.008134505711495876,
+            -0.006724462378770113,
+            -0.020680630579590797,
+            0.002073182025924325,
+            0.006333559285849333,
+            0.01950792223215103,
+            0.014724013395607471,
+            -0.009246718138456345,
+            -0.007250320166349411,
+            -0.024868877604603767,
+            0.010777754709124565,
+            -0.026376646012067795,
+            -0.02295159175992012,
+            0.027735499665141106,
+            0.013541997410356998,
+            0.014333110302686691,
+            -0.0016043310752138495,
+            0.0024315097834914923,
+            -0.010526459664106369,
+            0.013430310413241386,
+            0.02213255688548088,
+            0.004597763996571302,
+            0.007403889205306768,
+            -0.015105608850717545,
+            -0.002538542728871107,
+            0.020289728417992592,
+            -0.00022119177447166294,
+            -0.005002627614885569,
+            0.022486230358481407,
+            0.015422054566442966,
+            0.01886572316288948,
+            -0.004904902074486017,
+            0.015943259000778198,
+            0.01648307777941227,
+            0.008920964784920216,
+            -0.004106808453798294,
+            0.00020301362383179367,
+            -0.020997075363993645,
+            -0.004835098050534725,
+            0.03337567299604416,
+            0.012341366149485111,
+            0.025762369856238365,
+            -0.0263208020478487,
+            0.004825790878385305,
+            -0.004141710698604584,
+            0.013020792976021767,
+            0.01047061663120985,
+            0.019638223573565483,
+            0.028442848473787308,
+            0.009395632892847061,
+            0.009902875870466232,
+            -0.0012238987255841494,
+            0.029224654659628868,
+            -0.012611275538802147,
+            -0.009902875870466232,
+            -0.0006788449827581644,
+            -0.006622083019465208,
+            -0.004211514722555876,
+            0.00010339733853470534,
+            0.01476124208420515,
+            0.002815432380884886,
+            -0.019061176106333733,
+            0.014146965928375721,
+            0.007054868619889021,
+            -0.018819188699126244,
+            0.012127300724387169,
+            -0.02151828072965145,
+            0.011373416520655155,
+            -0.02738182619214058,
+            -0.0053656091913580894,
+            -0.01377467717975378,
+            -0.009358404204249382,
+            -0.023174963891506195,
+            -0.006933874916285276,
+            0.004034677520394325,
+            0.0082927281036973,
+            -0.0029317724984139204,
+            0.0043301815167069435,
+            -0.0030550931114703417,
+            -0.01600840874016285,
+            0.007929746992886066,
+            0.030937181785702705,
+            0.008716206066310406,
+            0.007450425066053867,
+            -0.0035530291497707367,
+            0.011224500834941864,
+            -0.008972154930233955,
+            -0.02213255688548088,
+            0.0002355888718739152,
+            -0.013467539101839066,
+            -0.008562637493014336,
+            0.013532689772546291,
+            0.006515050306916237,
+            -0.014202809892594814,
+            0.00905591994524002,
+            -0.024440744891762733,
+            -0.0002466411970090121,
+            -0.017786087468266487,
+            -0.00875343568623066,
+            -0.0053656091913580894,
+            -0.015021844767034054,
+            0.0013483826769515872,
+            0.007617955096065998,
+            -0.00890700425952673,
+            -0.006277716252952814,
+            0.00968881044536829,
+            -0.0037926898803561926,
+            -0.014333110302686691,
+            -0.0038880889769643545,
+            -0.008916311897337437,
+            -0.002316333120688796,
+            0.0018905280157923698,
+            -0.00985634047538042,
+            -0.005095699802041054,
+            -0.005407491698861122,
+            -0.026506947353482246,
+            0.006812881212681532,
+            0.019600993022322655,
+            0.004602417815476656,
+            0.020829545333981514,
+            0.0029573673382401466,
+            0.010526459664106369,
+            0.02931772544980049,
+            0.0055238320492208,
+            -0.008748781867325306,
+            -0.024887491017580032,
+            0.02159273810684681,
+            -0.010312393307685852,
+            0.024887491017580032,
+            -0.011447873897850513,
+            -0.014472718350589275,
+            0.0026548828464001417,
+            -0.00554244639351964,
+            -0.005267883650958538,
+            0.00035076565109193325,
+            0.006026421207934618,
+            -0.017497563734650612,
+            0.018009459599852562,
+            -0.011857391335070133,
+            0.008897697553038597,
+            0.010600917041301727,
+            0.041733551770448685,
+            -0.0013134806649759412,
+            -0.007375967688858509,
+            0.004462809301912785,
+            -0.001483337371610105,
+            0.01958237960934639,
+            -0.004497711546719074,
+            0.007282895501703024,
+            -0.009716732427477837,
+            -0.0007445771479979157,
+            -0.009265332482755184,
+            0.0062730624340474606,
+            0.015133530832827091,
+            -0.016771601513028145,
+            -0.01328139565885067,
+            0.0055052172392606735,
+            -0.026506947353482246,
+            0.0086138267070055,
+            0.030248448252677917,
+            0.017562713474035263,
+            -0.007324778009206057,
+            -0.014379646629095078,
+            0.007422503549605608,
+            -0.030080918222665787,
+            -0.0028480077162384987,
+            0.09649720042943954,
+            -0.009339789859950542,
+            -0.005128275137394667,
+            -0.02336110919713974,
+            -0.006524357479065657,
+            0.019228706136345863,
+            -0.014342417940497398,
+            0.07330362498760223,
+            -0.008748781867325306,
+            0.00046128881513141096,
+            -0.0073015098460018635,
+            0.001322787837125361,
+            -0.02053171582520008,
+            0.003978834021836519,
+            -0.02196502685546875,
+            -0.024868877604603767,
+            0.0021266986150294542,
+            -0.030118146911263466,
+            -0.007366660516709089,
+            -0.0034971858840435743,
+            0.008413721807301044,
+            -0.0049188630655407906,
+            0.011857391335070133,
+            -0.009865647181868553,
+            -0.012462359853088856,
+            -0.0011639834847301245,
+            0.005677401088178158,
+            0.015654735267162323,
+            -0.026264959946274757,
+            -0.01991743966937065,
+            0.010545074008405209,
+            0.01900533214211464,
+            0.004469790030270815,
+            -0.008609173819422722,
+            0.001268108026124537,
+            -0.00514223612844944,
+            0.008590559475123882,
+            0.029708629474043846,
+            0.01089874841272831,
+            0.01108489278703928,
+            0.0037531342823058367,
+            -0.011075585149228573,
+            0.015328982844948769,
+            0.007785485126078129,
+            0.013662991113960743,
+            -0.006021767854690552,
+            -0.004816483706235886,
+            -0.0021022669970989227,
+            0.009558509103953838,
+            -0.02220701426267624,
+            -0.014119044877588749,
+            -0.012238986790180206,
+            0.016967052593827248,
+            -0.006938528269529343,
+            -0.0005441812099888921,
+            -0.009791189804673195,
+            0.013579226098954678,
+            -0.008874429389834404,
+            -0.0051236217841506,
+            0.005184118635952473,
+            -0.021276293322443962,
+            0.022467615082859993,
+            0.006864070892333984,
+            0.021983640268445015,
+            -0.002318659797310829,
+            0.003573970403522253,
+            -0.014714706689119339,
+            -0.006580200511962175,
+            0.0106939896941185,
+            -0.004718757700175047,
+            0.00822292361408472,
+            0.002288411371409893,
+            0.02894543670117855,
+            0.0005770472926087677,
+            -0.013523383066058159,
+            0.008250845596194267,
+            -0.011587481945753098,
+            -0.01791638880968094,
+            0.010182092897593975,
+            -0.020382799208164215,
+            0.009060573764145374,
+            -6.11876675975509e-05,
+            -0.025445925071835518,
+            -0.010005255229771137,
+            0.004932823590934277,
+            -0.006924567744135857,
+            -0.013141786679625511,
+            0.013718834146857262,
+            -0.01853066496551037,
+            -0.01071260403841734,
+            -0.007194476667791605,
+            0.02101569063961506,
+            -0.016846058890223503,
+            0.023379722610116005,
+            -0.020680630579590797,
+            0.00634286692366004,
+            -0.00875343568623066,
+            0.02015942707657814,
+            -0.004453502129763365,
+            0.018195604905486107,
+            0.0122948307543993,
+            0.00674307718873024,
+            0.002838700544089079,
+            0.007469039876013994,
+            0.009698117151856422,
+            0.02581821382045746,
+            0.01982436701655388,
+            -0.003245891071856022,
+            -0.008534715510904789,
+            -0.011615403927862644,
+            0.013839827850461006,
+            -0.005793740972876549,
+            -0.00922810286283493,
+            0.011782933957874775,
+            0.016027024015784264,
+            -0.014947386458516121,
+            0.014854314737021923,
+            -0.0028666220605373383,
+            0.01086151972413063,
+            -0.013904978521168232,
+            0.004013736266642809,
+            0.0009057082934305072,
+            0.012369288131594658,
+            0.0075993407517671585,
+            -0.00016491221322212368,
+            0.01541274692863226,
+            0.02040141448378563,
+            -0.0004836842999793589,
+            0.01651099883019924,
+            0.0046210321597754955,
+            0.010377543978393078,
+            0.0021104109473526478,
+            -0.0037926898803561926,
+            -0.00514223612844944,
+            0.0021871954668313265,
+            -0.007892518304288387,
+            0.008786010555922985,
+            0.011056970804929733,
+            -0.010768447071313858,
+            0.002801471622660756,
+            0.012611275538802147,
+            -0.011913234367966652,
+            -0.008511447347700596,
+            0.0038275918923318386,
+            -0.026879236102104187,
+            -0.006850109901279211,
+            0.028089173138141632,
+            -0.018214218318462372,
+            -0.007273588329553604,
+            0.010033177211880684,
+            0.012425131164491177,
+            -0.0010011072736233473,
+            -0.014472718350589275,
+            0.009079188108444214,
+            0.005216693971306086,
+            -0.02557622641324997,
+            0.0024128954391926527,
+            0.011606096290051937,
+            0.018577199429273605,
+            -0.0004990993766114116,
+            -0.010768447071313858,
+            0.0013367487117648125,
+            0.025352852419018745,
+            0.016492383554577827,
+            0.013067329302430153,
+            -0.0050910464487969875,
+            -0.003990468103438616,
+            0.0051375823095440865,
+            0.0012064477195963264,
+            0.002601366490125656,
+            0.00668257987126708,
+            -0.0012983564520254731,
+            -0.0059054275043308735,
+            0.01348615437746048,
+            -0.0010458981851115823,
+            0.012322751805186272,
+            -0.013700219802558422,
+            -0.0186981949955225,
+            -0.0013134806649759412,
+            -0.003874127985909581,
+            -0.012387902475893497,
+            0.019470693543553352,
+            -0.003285446669906378,
+            0.03709855675697327,
+            0.013458232395350933,
+            0.005086392629891634,
+            -0.002601366490125656,
+            -0.008404415100812912,
+            0.016631992533802986,
+            0.0053656091913580894,
+            -0.01520798821002245,
+            0.003927644342184067,
+            -0.029168810695409775,
+            0.021071534603834152,
+            -0.009865647181868553,
+            0.010554381646215916,
+            -0.002633941825479269,
+            0.007906478829681873,
+            0.004876980558037758,
+            -0.011466488242149353,
+            -0.003373865270987153,
+            0.009567816741764545,
+            0.0022220974788069725,
+            -0.012127300724387169,
+            -0.0015612852293998003,
+            0.008678977377712727,
+            -0.006049689371138811,
+            0.012211065739393234,
+            0.005272537004202604,
+            -0.008520754985511303,
+            0.00436741067096591
+          ],
+          "title": "Conclusion"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "9ce07b80-80ac-4513-b0a8-37b1ff0cab56",
+      "type": "child",
+      "source": {
+        "id": "1e687808-8db8-4320-805d-0282a8d679bc",
+        "properties": {
+          "page_content": "Deepseekmoe: Towards Ultimate Expert Specialization In\n\nMixture-Of-Experts Language Models\n\nDamai Dai∗1,2, Chengqi Deng1, Chenggang Zhao∗1,3, R.X. Xu1, Huazuo Gao1, Deli Chen1, Jiashi Li1, Wangding Zeng1, Xingkai Yu∗1,4, Y. Wu1, Zhenda Xie1, Y.K. Li1, Panpan Huang1, Fuli Luo1, Chong Ruan1, Zhifang Sui2, Wenfeng Liang1 1DeepSeek-AI 2National Key Laboratory for Multimedia Information Processing, Peking University 3Institute for Interdisciplinary Information Sciences, Tsinghua University 4National Key Laboratory for Novel Software Technology, Nanjing University {daidamai, szf}@pku.edu.cn, {wenfeng.liang}@deepseek.com https://github.com/deepseek-ai/DeepSeek-MoE\n\nAbstract\n\nIn the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top- out of experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into ones and activating from them, allowing for a more flexible combination of activated experts; (2) isolating experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5× expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations.\n\nFurther, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\n1. Introduction\n\nRecent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023a). It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017) has emerged as a popular solution. It can\n\nenable parameter scaling, while concurrently keeping computational costs at a modest level.\n\nRecent applications of MoE architectures in Transformers (Vaswani et al., 2017) have yielded successful attempts at scaling language models to a substantial size (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022), accompanied with remarkable performance. These achievements underscore the considerable potential and promise of MoE language models.\n\nDespite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts.\n\nThis architecture manifests two potential issues: (1) Knowledge Hybridity: existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. (2) Knowledge Redundancy: tokens assigned to different experts may require common knowledge. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters.\n\nThese issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models.\n\nIn response to the aforementioned issues, we introduce DeepSeekMoE, an innovative MoE architecture specifically designed towards ultimate expert specialization. Our architecture involves two principal strategies: (1) Fine-Grained Expert Segmentation: while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. (2) Shared Expert Isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.\n\nThrough compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized.\n\nStarting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B (Lepikhin et al., 2021) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5× expert parameters and computation. Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE.\n\nThese studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization.\n\nLeveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B (DeepSeek-AI, 2024), a dense model trained on the same 2T corpus. We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B (Touvron et al., 2023b), which has approximately 2.5 times the activated parameters. Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard1. Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. The experimental results still validate its substantial advantages over the GShard architecture consistently. In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nOur contributions are summarized as follows: - Architectural Innovation. We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation.\n\nEmpirical Validation. We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. Experimental results validate the high 1https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models\n\nScalability. We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B.\n\nAlignment for MoE. We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B.\n\nPublic Release. In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization.\n\n2. Preliminaries: Mixture-Of-Experts For Transformers\n\nWe first introduce a generic MoE architecture commonly used in Transformer language models. A standard Transformer language model is constructed by stacking layers of standard Transformer blocks, where each block can be represented as follows:\n\n$$\\mathbf{u}{1:T}^{l}=\\text{Self-Att}\\left(\\mathbf{h}{1:T}^{l-1}\\right)+\\mathbf{h}{1:T}^{l-1},\\tag{1}$$ $$\\mathbf{h}{t}^{l}=\\text{FFN}\\left(\\mathbf{u}{t}^{l}\\right)+\\mathbf{u}{t}^{l},\\tag{2}$$\n\nwhere denotes the sequence length, Self-Att(·) denotes the self-attention module, FFN(·) denotes the Feed-Forward Network (FFN), u 1: ∈ R× are the hidden states of all tokens after the -th attention module, and h\n\n∈ Ris the output hidden state of the -th token after the -th Transformer block. For brevity, we omit the layer normalization in the above formulations.\n\nA typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022). An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. Then, each token will be assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts. If the -th FFN is substituted with an MoE layer, the computation for its output hidden state h\n\nis expressed as: where denotes the total number of experts, FFN(·) is the -th expert FFN, , denotes the gate value for the -th expert, , denotes the token-to-expert affinity, Topk(·, ) denotes the set comprising highest affinity scores among those calculated for the -th token and all experts, and e\n\nis the centroid of the -th expert in the -th layer. Note that ,is sparse, indicating that only out of gate values are nonzero. This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only experts. Also, in the above formulations, we omit the layer normalization operation for brevity.\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{N}\\left(g_{i,t}\\operatorname{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\operatorname{Topk}({s_{j,t}|1\\leqslant j\\leqslant N},K),\\ 0,&\\text{otherwise},\\end{cases}$$ $$s_{i,t}=\\operatorname{Softmax}{i}\\left(\\mathbf{u}{t}^{l}\\,\\mathbf{e}_{i}^{l}\\right),$$\n\n$$(3)$$\n\n(4) $\\binom{4}{5}$ .\n\n3. Deepseekmoe Architecture\n\nOn top of the generic MoE architecture outlined in Section 2, we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. As illustrated in Figure 2, our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. Both of these strategies are designed to elevate the level of expert specialization.\n\n3.1. Fine-Grained Expert Segmentation\n\nIn scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts.\n\nIn pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. The finer expert segmentation enables a more flexible and adaptable combination of activated experts. To be specific, on top of a typical MoE architecture shown in Figure 2(a), we segment each expert FFN into smaller experts by reducing the FFN intermediate hidden dimension to 1 times its original size. Since each expert becomes smaller, in response, we also increase the number of activated experts to times to keep the same computation cost, as illustrated in Figure 2(b). With the fine-grained expert segmentation, the output of an MoE layer can be expressed as:\n\n$$\\mathbf{h}{t}^{i}=\\sum{i=1}^{\\text{mN}}\\left(g_{i,t}\\text{FFN}{i}\\left(\\mathbf{u}{t}^{i}\\right)\\right)+\\mathbf{u}{t}^{i},\\tag{6}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\text{Tokr}(s_{i,t}|1\\leqslant j\\leqslant m\\text{N}),m\\text{K}),\\ 0,&\\text{otherwise},\\end{cases}$$ (7) $$s_{i,t}=\\text{Softmax}{i}\\left(\\mathbf{u}{t}^{T}\\mathbf{e}_{t}^{i}\\right),\\tag{8}$$ where the total number of expert parameters is equal to $N$ times the number of parameters in a\n\n$$(6)$$\n\n(7) (8) $\\frac{1}{2}$\n\nstandard FFN, and denotes the total number of fine-grained experts. With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to .\n\nFrom a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. As an illustrative example, we consider the case where = 16. A typical top-2 routing strategy can yield 16 2 = 120 possible combinations. By contrast, if each expert is split into 4 smaller experts, the fine-grained routing strategy can yield 64 8 = 4, 426, 165, 368 potential combinations. The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition.\n\n3.2. Shared Expert Isolation\n\nWith a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. This alleviation of redundancy will contribute to a more parameterefficient model with more specialized experts.\n\nTowards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate experts to serve as shared experts. Regardless of the router module, each token will be deterministically assigned to these shared experts. In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by , as depicted in Figure 2(c). With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows:\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{K_{s}}\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)+\\sum_{i=K_{s}+1}^{mN}\\left(g_{i,t}\\;\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},\\tag{9}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\mathrm{Topk}{s_{j,t}|K_{s}+1\\leqslant j\\leqslant mN},mK-K_{s}},\\ 0,&\\text{otherwise},\\end{cases}$$ (10) $$s_{i,t}=\\mathrm{Softmax}{i}\\left(\\mathbf{u}{t}^{l,T}e_{i}^{l}\\right).\\tag{11}$$ $\\mathbf{h}\\mathbf{M}\\mathbf{E}$ the number of closed-compact is $K_{s}$ the total number of routed currents.\n\nFinally, in DeepSeekMoE, the number of shared expert is , the total number of routed experts is − , and the number of nonzero gates is − .\n\nIt is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022). The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint.\n\n3.3. Load Balance Consideration\n\nAutomatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. Firstly, there is a risk of routing collapse (Shazeer et al., 2017), i.e., the model always selects only a few experts, preventing other experts from sufficient training.\n\nSecondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.\n\nExpert-Level Balance Loss. In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. The computation of the balance loss is as follows:\n\n$$\\begin{array}{l}{{{\\mathcal L}{\\mathrm{ExpBal}}=\\alpha{1}\\sum_{i=1}^{N^{\\prime}}f_{i}P_{i},}}\\ {{f_{i}=\\frac{N^{\\prime}}{K^{T}}\\sum_{t=1}^{T}\\mathds{1}\\,(\\mathrm{Token}\\;t\\;\\mathrm{s}\\mathrm{lects}\\;\\mathrm{Expert}\\;i),}}\\ {{P_{i}=\\frac{1}{T}\\sum_{t=1}^{T}s_{i,t},}}\\end{array}$$\n\n(12) $$\\begin{array}{l}\\small\\text{(13)}\\end{array}$$ = (14) $$\\begin{array}{l}\\small\\text{(14)}\\end{array}$$ .\n\n7 where 1 is a hyper-parameter called expert-level balance factor, ′is equal to ( − ) and ′ is equal to ( − ) for brevity. 1(·) denotes the indicator function.\n\nDevice-Level Balance Loss. In addition to the expert-level balance loss, we introduce a devicelevel balance loss. When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. Instead, our primary objective is to ensure balanced computation across the devices. If we partition all routed experts into groups {E1, E2, ..., E}, and deploy each group on a single device, the device-level balance loss is computed as follows:\n\n$$\\mathcal{L}{\\text{DevBal}}=\\alpha{2}\\sum_{i=1}^{D}f_{i}^{\\prime}P_{i}^{\\prime},$$ $$f_{i}^{\\prime}=\\frac{1}{|\\mathcal{E}{i}|}\\sum{j\\in\\mathcal{E}{i}}f{j},$$ $$P_{i}^{\\prime}=\\sum_{j\\in\\mathcal{E}{i}}P{j},$$\n\n(15) $$\\begin{array}{l}\\small\\text{(16)}\\end{array}$$ = (17) .\n\nwhere 2 is a hyper-parameter called device-level balance factor. In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices.\n\n4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n6. Alignment For Deepseekmoe 16B\n\nPrevious research indicates that MoE models typically do not emerge significant gains from fine-tuning (Artetxe et al., 2022; Fedus et al., 2021). However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B.\n\n6.1. Experimental Setup\n\nTraining Data. For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios.\n\nHyper-Parameters. During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).\n\nWe employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10−5 without incorporating any learning rate scheduling strategy.\n\nEvaluation Benchmarks. For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3, with the following adjustments: (1) We exclude Pile (Gao et al., 2020) since chat models are seldom employed for pure language modeling. (2) We exclude\n\nMetric # Shot LLaMA2 DeepSeek DeepSeekMoE SFT 7B Chat 7B Chat 16B # Total Params N/A 6.7B 6.9B 16.4B # Activated Params N/A 6.7B 6.9B 2.8B FLOPs per 4K Tokens N/A 187.9T 183.5T 74.4T HellaSwag (Acc.) 0-shot 67.9 71.0 72.2 PIQA (Acc.) 0-shot 76.9 78.4 79.7 ARC-easy (Acc.) 0-shot 69.7 70.2 69.9 ARC-challenge (Acc.) 0-shot 50.8 50.2 50.0 BBH (EM) 3-shot 39.3 43.1 42.2 RACE-middle (Acc.) 5-shot 63.9 66.1 64.8 RACE-high (Acc.) 5-shot 49.6 50.8 50.6 DROP (EM) 1-shot 40.0 41.7 33.8 GSM8K (EM) 0-shot 63.4 62.6 62.2 MATH (EM) 4-shot 13.5 14.7 15.2 HumanEval (Pass@1) 0-shot 35.4 45.1 45.7 MBPP (Pass@1) 3-shot 27.8 39.0 46.2 TriviaQA (EM) 5-shot 60.1 59.5 63.3 NaturalQuestions (EM) 0-shot 35.2 32.7 35.1 MMLU (Acc.) 0-shot 50.0 49.7 47.2 WinoGrande (Acc.) 0-shot 65.1 68.4 69.0 CLUEWSC (EM) 5-shot 48.4 66.2 68.2 CEval (Acc.) 0-shot 35.1 44.7 40.0 CMMLU (Acc.) 0-shot 36.9 51.2 49.3\n\nCHID (Zheng et al., 2019) due to the observed instability of results, hindering the derivation of solid conclusions. (3) We additionally include BBH (Suzgun et al., 2022) to provide a more comprehensive assessment of the reasoning ability of the chat models.\n\nTable 5 | Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations.\n\n6.2. Evaluations\n\nBaselines. In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. Correspondingly, we construct three chat models, including LLaMA2 SFT 7B3, DeepSeek Chat 7B, and DeepSeekMoE Chat 16B.\n\nSubsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks.\n\nResults. The evaluation results are presented in Table 5. Our key observations include: (1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intensive tasks (TriviaQA, NaturalQuestions). (2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. In addition, it also surpasses DeepSeek Chat 7B. (3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1). However, it is worth noting that, after supervised finetuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. (4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations.\n\n7. Deepseekmoe 145B Ongoing\n\nEncouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available.\n\n7.1. Experimental Setup\n\nTraining Data and Tokenization. For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study.\n\nModel Settings. For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps.\n\nSubsequently, the learning rate keeps constant during the remaining training process. The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. Also, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse.\n\nEvaluation Benchmarks. We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3).\n\n7.2. Evaluations\n\nBaselines. Apart from DeepSeekMoE 145B, we consider three additional models for comparison. DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters.\n\nResults. From the evaluation results presented in Table 6, we have the following observations: (1) Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. (2) On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. (3) At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5.\n\n8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\nAppendices A. Overview Of Hyper-Parameters\n\nWe present the overview of hyper-parameters for DeepSeekMoE across various sizes in Table 7.\n\n# Params # Layers Hidden # Attn # Shared # Routed Relative Sequence Batch Size Learning Size Heads Experts Experts Expert Size Length (Sequence) Rate 2.0B 9 1280 10 1 63 (7 activated) 0.25 2048 2048 1.08e-3 16.4B 28 2048 16 2 64 (6 activated) 0.25 4096 4608 4.2e-4 144.6B 62 4096 32 4 128 (12 activated) 0.125 4096 4608 3.0e-4\n\nTable 7 | Overview of hyper-parameters for DeepSeekMoE across various sizes. The relative expert size is in comparison to a standard FFN.\n\nB. Comparing Deepseekmoe With Larger Models\n\nComparisons among DeepSeekMoE, GShard×1.2, and GShard×1.5 are shown in Table 8. Comparisons among DeepSeekMoE, Dense×4, and Dense×16 are shown in Table 9.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 2.3B 2.8B 1.9B # Activated Expert Params N/A 0.28B 0.35B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.824 1.808 1.808 HellaSwag (Acc.) 0-shot 53.7 54.4 54.8 PIQA (Acc.) 0-shot 71.8 71.1 72.3 ARC-easy (Acc.) 0-shot 46.8 47.3 49.4 ARC-challenge (Acc.) 0-shot 31.7 34.1 34.3 RACE-middle (Acc.) 5-shot 43.7 46.4 44.0 RACE-high (Acc.) 5-shot 31.9 32.4 31.7 HumanEval (Pass@1) 0-shot 3.7 3.0 4.9 MBPP (Pass@1) 3-shot 2.4 2.6 2.2 TriviaQA (EM) 5-shot 15.2 15.7 16.6 NaturalQuestions (EM) 5-shot 4.5 4.7 5.7\n\nTable 8 | Comparison between DeepSeekMoE and larger GShard models.\n\nAt a larger scale of 13B total parameters, we also compare DeepSeekMoE with GShard×1.2 and GShard×1.5, and show results in Table 10. At a larger scale, DeepSeekMoE even outperforms GShard×1.5 distinctly.\n\nMetric # Shot Dense×4 Dense×16 DeepSeekMoE Relative Expert Size N/A 1 1 0.25 # Experts N/A 4 + 0 16 + 0 1 + 63 # Activated Experts N/A 4 + 0 16 + 0 1 + 7 # Total Expert Params N/A 0.47B 1.89B 1.89B # Activated Expert Params N/A 0.47B 1.89B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.908 1.806 1.808 HellaSwag (Acc.) 0-shot 47.6 55.1 54.8 PIQA (Acc.) 0-shot 70.0 71.9 72.3 ARC-easy (Acc.) 0-shot 43.9 51.9 49.4 ARC-challenge (Acc.) 0-shot 30.5 33.8 34.3 RACE-middle (Acc.) 5-shot 42.4 46.3 44.0 RACE-high (Acc.) 5-shot 30.7 33.0 31.7 HumanEval (Pass@1) 0-shot 1.8 4.3 4.9 MBPP (Pass@1) 3-shot 0.2 2.2 2.2 TriviaQA (EM) 5-shot 9.9 16.5 16.6 NaturalQuestions (EM) 5-shot 3.0 6.3 5.7\n\nTable 9 | Comparison between DeepSeekMoE and larger dense baselines.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 15.9B 19.8B 13.3B # Activated Expert Params N/A 2.37B 2.82B 2.05B # Training Tokens N/A 100B 100B 100B HellaSwag (Acc.) 0-shot 66.6 67.7 69.1 PIQA (Acc.) 0-shot 75.6 76.0 75.7 ARC-easy (Acc.) 0-shot 56.8 56.8 58.8 ARC-challenge (Acc.) 0-shot 39.9 37.6 38.5 RACE-middle (Acc.) 5-shot 51.6 50.6 52.4 RACE-high (Acc.) 5-shot 37.4 36.3 38.5 HumanEval (Pass@1) 0-shot 6.1 6.1 9.8 MBPP (Pass@1) 3-shot 7.0 11.6 10.6 TriviaQA (EM) 5-shot 36.5 36.7 38.2 NaturalQuestions (EM) 5-shot 12.6 12.1 13.7\n\nTable 10 | Comparison between DeepSeekMoE and larger GShard models at a larger scale.\n\nC. Training Benchmark Curves Of Deepseekmoe 16B\n\nWe present the benchmark curves during training of DeepSeekMoE 16B and DeepSeek 7B (Dense) in Figure 7 for reference.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/DeepSeekMoE_2.md"
+          },
+          "summary": "DeepSeekMoE is an innovative Mixture-of-Experts (MoE) architecture designed to enhance expert specialization in language models. It employs fine-grained expert segmentation and shared expert isolation to improve performance and reduce computational costs. Starting with a model of 2B parameters, DeepSeekMoE demonstrates comparable performance to larger models like GShard 2.9B and LLaMA2 7B, using significantly fewer computations. The architecture scales up to 16B and 145B parameters, maintaining advantages over traditional MoE architectures. DeepSeekMoE 16B achieves similar results to dense models with only 40% of the computations, and its chat model version shows adaptability in bilingual scenarios. The architecture's public release aims to contribute to advancements in large-scale language models.",
+          "headlines": {
+            "1. Introduction": [],
+            "2. Preliminaries: Mixture-Of-Experts For Transformers": [],
+            "3. Deepseekmoe Architecture": [
+              "3.1. Fine-Grained Expert Segmentation",
+              "3.2. Shared Expert Isolation",
+              "3.3. Load Balance Consideration"
+            ],
+            "4. Validation Experiments": [
+              "4.1. Experimental Setup",
+              "4.1.1. Training Data And Tokenization",
+              "4.1.2. Infrastructures",
+              "4.1.3. Hyper-Parameters",
+              "4.1.4. Evaluation Benchmarks",
+              "4.2. Evaluations",
+              "4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models",
+              "4.4. Ablation Studies",
+              "4.5. Analysis On Expert Specialization"
+            ],
+            "5. Scaling Up To Deepseekmoe 16B": [
+              "5.1. Experimental Setup",
+              "5.1.1. Training Data And Tokenization",
+              "5.1.2. Hyper-Parameters",
+              "5.1.3. Evaluation Benchmarks",
+              "5.2. Evaluations",
+              "5.2.1. Internal Comparison With Deepseek 7B",
+              "5.2.2. Comparison With Open Source Models"
+            ],
+            "6. Alignment For Deepseekmoe 16B": [
+              "6.1. Experimental Setup",
+              "6.2. Evaluations"
+            ],
+            "7. Deepseekmoe 145B Ongoing": [
+              "7.1. Experimental Setup",
+              "7.2. Evaluations"
+            ],
+            "8. Related Work": [],
+            "9. Conclusion": [],
+            "Appendices": [
+              "A. Overview Of Hyper-Parameters",
+              "B. Comparing Deepseekmoe With Larger Models",
+              "C. Training Benchmark Curves Of Deepseekmoe 16B"
+            ]
+          },
+          "summary_embedding": [
+            -0.012465757317841053,
+            -0.022808987647294998,
+            -0.018792515620589256,
+            -0.02308654971420765,
+            0.01797616109251976,
+            0.007200260180979967,
+            0.006490030791610479,
+            0.042613785713911057,
+            -0.014049487188458443,
+            0.03187054023146629,
+            -0.021486490964889526,
+            -0.023592690005898476,
+            0.009045224636793137,
+            -0.014441338367760181,
+            0.005865518003702164,
+            -0.007363531738519669,
+            -0.008743173442780972,
+            -0.005163452122360468,
+            0.018841497600078583,
+            -0.03100520372390747,
+            0.011110605672001839,
+            -0.046075135469436646,
+            -0.020017050206661224,
+            -0.021845687180757523,
+            0.016425084322690964,
+            0.037389107048511505,
+            0.022547753527760506,
+            -0.0028062239289283752,
+            -0.06844329088926315,
+            0.034678805619478226,
+            0.035168617963790894,
+            0.02052319049835205,
+            0.0005438972148112953,
+            -0.00031302153365686536,
+            -0.024833550676703453,
+            -0.028556134551763535,
+            0.03347059711813927,
+            -0.004379750229418278,
+            -0.01453930139541626,
+            0.015976088121533394,
+            0.0414055772125721,
+            0.012106560170650482,
+            -0.02754385396838188,
+            0.0014245413476601243,
+            0.015151568688452244,
+            0.0055961208418011665,
+            -0.01678428053855896,
+            -0.0022960014175623655,
+            -0.002424577483907342,
+            -0.021225256845355034,
+            -0.01400050614029169,
+            0.018825169652700424,
+            0.001296985661610961,
+            -0.008865626528859138,
+            -0.0542386956512928,
+            0.007330877240747213,
+            0.03840138763189316,
+            0.05623060464859009,
+            0.044148534536361694,
+            -0.020180322229862213,
+            0.03369917720556259,
+            -0.030025575309991837,
+            -0.02501314878463745,
+            0.011028970591723919,
+            0.003983817528933287,
+            -0.018531281501054764,
+            0.01784554310142994,
+            -0.005636938381940126,
+            -0.0006556359585374594,
+            0.01751900091767311,
+            -0.03941367194056511,
+            0.020751770585775375,
+            -0.01887415163218975,
+            0.031266435980796814,
+            0.02173139713704586,
+            0.0073798587545752525,
+            0.01872720755636692,
+            0.01678428053855896,
+            0.0036858476232737303,
+            -0.04107903689146042,
+            0.009118696674704552,
+            0.016523046419024467,
+            0.014196431264281273,
+            0.016269976273179054,
+            0.06553706526756287,
+            -0.05391215533018112,
+            -0.004485876299440861,
+            -0.05169166624546051,
+            0.0012561678886413574,
+            -0.004971608519554138,
+            -0.03670336678624153,
+            0.01048201136291027,
+            0.028425518423318863,
+            0.021372200921177864,
+            -0.01063711941242218,
+            0.024392718449234962,
+            -0.022743679583072662,
+            0.0032001156359910965,
+            -0.04062187671661377,
+            0.0019653772469609976,
+            0.004518530797213316,
+            0.016531210392713547,
+            0.0070778070949018,
+            0.004273623693734407,
+            -0.016702644526958466,
+            0.006045116577297449,
+            -0.01905374974012375,
+            0.03291547670960426,
+            -0.008465612307190895,
+            0.08653374016284943,
+            0.008751336485147476,
+            0.004738946910947561,
+            0.004281787667423487,
+            0.02398454025387764,
+            0.012131051160395145,
+            -0.00810233410447836,
+            -0.04718538001179695,
+            -0.015690363943576813,
+            0.011061624623835087,
+            -0.0383034273982048,
+            0.018335357308387756,
+            -0.008759500458836555,
+            0.01282495353370905,
+            -0.04107903689146042,
+            -0.06344719231128693,
+            0.01498013362288475,
+            0.001784758409485221,
+            -0.004065453074872494,
+            -0.0002903166168835014,
+            -0.018923133611679077,
+            0.003930754493921995,
+            -0.004200151655822992,
+            -0.010645282454788685,
+            -0.021502818912267685,
+            0.03159298002719879,
+            0.03657275065779686,
+            0.06527583301067352,
+            0.036801330745220184,
+            -0.0030164355412125587,
+            0.005775718949735165,
+            0.019690508022904396,
+            -0.020784424617886543,
+            0.003481758525595069,
+            -0.01214737817645073,
+            -0.021992631256580353,
+            -0.03820546343922615,
+            0.010971825569868088,
+            0.07379858940839767,
+            -0.029029620811343193,
+            0.013437220826745033,
+            -0.017616963014006615,
+            0.04205866530537605,
+            -0.004567512311041355,
+            0.026727497577667236,
+            -0.016669990494847298,
+            0.004059330560266972,
+            -0.025388672947883606,
+            0.01620466820895672,
+            -0.024964168667793274,
+            0.012571883387863636,
+            -0.000885746325366199,
+            -0.04222193360328674,
+            0.00011116473615402356,
+            0.026890767738223076,
+            -0.010971825569868088,
+            0.02261306345462799,
+            0.0004949158756062388,
+            -0.006302268709987402,
+            -0.05097327008843422,
+            0.038727931678295135,
+            -0.003912386484444141,
+            0.018645571544766426,
+            -0.0018959869630634785,
+            0.0163107942789793,
+            -0.0016408757073804736,
+            0.013216804713010788,
+            0.009543201886117458,
+            0.007114543113857508,
+            -0.033503253012895584,
+            0.02697240374982357,
+            -0.02504580281674862,
+            0.028049994260072708,
+            0.03435226157307625,
+            -0.0030184765346348286,
+            0.03255628049373627,
+            0.019249675795435905,
+            0.03840138763189316,
+            0.0327032245695591,
+            0.030335791409015656,
+            -0.00824927818030119,
+            0.013878053054213524,
+            0.008200296200811863,
+            0.03608293831348419,
+            0.013429056853055954,
+            0.019723162055015564,
+            -0.009730963967740536,
+            -0.014041324146091938,
+            0.048785436898469925,
+            0.007318632211536169,
+            -0.0179598331451416,
+            -0.005441013257950544,
+            -0.03510330989956856,
+            -0.002832755446434021,
+            -0.014425011351704597,
+            0.0323440283536911,
+            0.02158445306122303,
+            -0.01647406443953514,
+            0.035135965794324875,
+            0.024980494752526283,
+            -0.03751972317695618,
+            -0.005232842173427343,
+            -0.02026195637881756,
+            -0.023298801854252815,
+            -0.04215662553906441,
+            0.011624909937381744,
+            0.012833117507398129,
+            -0.013976015150547028,
+            -0.007469657808542252,
+            0.020457882434129715,
+            -0.023625344038009644,
+            -0.0018265966791659594,
+            -0.01356783788651228,
+            0.006547175347805023,
+            -0.020898714661598206,
+            -0.03667071461677551,
+            -0.01048201136291027,
+            0.025862159207463264,
+            0.012637191452085972,
+            0.04362606629729271,
+            -0.013518855907022953,
+            -0.005367541220039129,
+            0.033193036913871765,
+            0.012759645469486713,
+            0.010767736472189426,
+            0.03699725493788719,
+            -0.03027048148214817,
+            0.024621298536658287,
+            -0.020294612273573875,
+            -0.0032878739293664694,
+            0.01707816869020462,
+            -0.0030062312725931406,
+            0.01962519995868206,
+            -0.04535674303770065,
+            -0.024196794256567955,
+            0.04395261034369469,
+            -0.0007107399869710207,
+            0.016302630305290222,
+            -0.005959399044513702,
+            -0.012106560170650482,
+            0.0035899258218705654,
+            0.0027715289033949375,
+            -0.01438419334590435,
+            -0.008620720356702805,
+            0.007473739795386791,
+            -0.01962519995868206,
+            0.03660540655255318,
+            0.03794422745704651,
+            -0.004132802598178387,
+            -0.014131123200058937,
+            -0.03324201703071594,
+            0.010016689077019691,
+            -0.017878197133541107,
+            -0.023951886221766472,
+            0.04078514873981476,
+            -0.032115448266267776,
+            -0.00029312283731997013,
+            0.039478980004787445,
+            0.018808843567967415,
+            -0.014898497611284256,
+            0.00868602842092514,
+            -0.01709449477493763,
+            -0.026302991434931755,
+            -0.017159804701805115,
+            0.03356856107711792,
+            0.024800896644592285,
+            -0.010743245482444763,
+            0.010245268233120441,
+            0.005330805201083422,
+            0.02893165871500969,
+            0.024376392364501953,
+            0.00648594880476594,
+            -0.013812744058668613,
+            0.015029114671051502,
+            0.027641816064715385,
+            -0.0028021421749144793,
+            -7.66609373386018e-05,
+            -0.004010349046438932,
+            -0.007796200457960367,
+            -0.006453294772654772,
+            -0.013069860637187958,
+            -0.005089980084449053,
+            0.0009184005321003497,
+            -0.032295044511556625,
+            0.015967924147844315,
+            -0.005890008993446827,
+            0.010359558276832104,
+            -0.011535110883414745,
+            0.01288209855556488,
+            0.01722511276602745,
+            -0.013494365848600864,
+            -0.0716434046626091,
+            0.014612773433327675,
+            0.02458864450454712,
+            -0.006706364918500185,
+            -0.017943505197763443,
+            0.008481939323246479,
+            0.00016148542636074126,
+            0.038597315549850464,
+            -0.014180104248225689,
+            -0.031103165820240974,
+            0.02680913358926773,
+            -0.017437364906072617,
+            -0.015241367742419243,
+            -0.010661610402166843,
+            -0.00580837344750762,
+            0.04649963974952698,
+            0.01889047957956791,
+            0.011420820839703083,
+            -0.04042595252394676,
+            -0.007449248805642128,
+            0.015078096650540829,
+            0.014506647363305092,
+            -0.04705476388335228,
+            0.0052083516493439674,
+            0.015486273914575577,
+            0.019723162055015564,
+            0.024800896644592285,
+            0.04274440184235573,
+            -0.0021531390957534313,
+            0.01661284640431404,
+            -0.015053605660796165,
+            -0.05136512219905853,
+            -0.03340528905391693,
+            0.029943939298391342,
+            -0.017731253057718277,
+            -0.02906227484345436,
+            -0.01693122461438179,
+            -0.006151242647320032,
+            0.0017939424142241478,
+            0.007126788143068552,
+            -0.01916803978383541,
+            -0.03464614972472191,
+            -0.03970756009221077,
+            0.02277633361518383,
+            0.007983962073922157,
+            0.004269542172551155,
+            -0.042777057737112045,
+            0.0013459670590236783,
+            0.009184005670249462,
+            0.012073906138539314,
+            0.0009954441338777542,
+            -0.019984396174550056,
+            0.015494437888264656,
+            0.0032001156359910965,
+            -0.031070511788129807,
+            -0.035593122243881226,
+            -0.03595232218503952,
+            0.0023919232189655304,
+            0.003781769424676895,
+            0.06279410421848297,
+            0.012988224625587463,
+            -0.03840138763189316,
+            -0.04522612690925598,
+            0.029176564887166023,
+            -0.02052319049835205,
+            -0.016678154468536377,
+            0.0059430720284581184,
+            0.012898425571620464,
+            -0.009877908043563366,
+            0.013192313723266125,
+            0.023560035973787308,
+            0.041601505130529404,
+            -0.005987971555441618,
+            0.012914752587676048,
+            0.023037567734718323,
+            0.012955570593476295,
+            0.0362788625061512,
+            -0.00936360377818346,
+            0.0035409443080425262,
+            0.039936136454343796,
+            -0.012269831262528896,
+            0.03175625205039978,
+            -0.02189466916024685,
+            0.0020398697815835476,
+            0.02249877341091633,
+            0.027184655889868736,
+            -0.0006479826406575739,
+            0.02594379521906376,
+            -0.031543996185064316,
+            0.018368011340498924,
+            -0.014498483389616013,
+            -0.04888340085744858,
+            -0.01557607389986515,
+            0.013102514669299126,
+            0.030384771525859833,
+            0.02756018005311489,
+            -0.0041348435916006565,
+            -0.019723162055015564,
+            -0.01155143789947033,
+            -0.010596301406621933,
+            0.0018419033149257302,
+            0.03285016864538193,
+            0.010506502352654934,
+            0.010890189558267593,
+            -0.011935125105082989,
+            -0.004104230087250471,
+            0.03025415539741516,
+            -0.021192602813243866,
+            0.01650671847164631,
+            0.0041511706076562405,
+            -0.002029665280133486,
+            -0.04320156201720238,
+            -0.035266581922769547,
+            0.007845181971788406,
+            0.011747363954782486,
+            0.014874007552862167,
+            0.0368339866399765,
+            0.025094784796237946,
+            -0.01662100851535797,
+            0.030629679560661316,
+            -0.01905374974012375,
+            0.027984686195850372,
+            0.039054471999406815,
+            0.011624909937381744,
+            -0.018351683393120766,
+            0.012237177230417728,
+            0.002424577483907342,
+            -0.025127438828349113,
+            0.01680060662329197,
+            -0.032735876739025116,
+            0.01738838478922844,
+            -0.02354370802640915,
+            0.025143766775727272,
+            -0.02579685114324093,
+            -0.009290131740272045,
+            -0.004428731743246317,
+            -0.006130834110081196,
+            0.04888340085744858,
+            -0.026727497577667236,
+            -0.0142454132437706,
+            -0.0024694770108908415,
+            -0.0024531499948352575,
+            -0.008718682453036308,
+            0.010000362060964108,
+            -0.045781247317790985,
+            -0.01844964735209942,
+            -0.04016471654176712,
+            0.007355368230491877,
+            0.005412440747022629,
+            -0.030907239764928818,
+            -0.05694899708032608,
+            0.025421326979994774,
+            -0.06331657618284225,
+            -0.019886434078216553,
+            0.051626358181238174,
+            -0.018400665372610092,
+            -0.008212542161345482,
+            0.009641164913773537,
+            0.006375740747898817,
+            -0.029290854930877686,
+            0.0041348435916006565,
+            0.02504580281674862,
+            0.04418119043111801,
+            -0.0045797573402523994,
+            -0.02865409664809704,
+            0.0028674507047981024,
+            -0.027788760140538216,
+            -0.013461710885167122,
+            0.005261414684355259,
+            0.016449574381113052,
+            0.024539662525057793,
+            0.007628847379237413,
+            0.03585435822606087,
+            0.007294141221791506,
+            0.006318595726042986,
+            0.044279150664806366,
+            -0.025666235014796257,
+            -0.027380581945180893,
+            0.02519274689257145,
+            0.04045860469341278,
+            0.023151857778429985,
+            -0.018253721296787262,
+            -0.0142454132437706,
+            0.0053063142113387585,
+            0.0009296254720538855,
+            -0.022139575332403183,
+            0.002808264922350645,
+            -0.013902543112635612,
+            0.005143043119460344,
+            0.03892385587096214,
+            -0.001205145614221692,
+            0.014196431264281273,
+            0.007302304729819298,
+            0.006845145486295223,
+            0.004334850702434778,
+            -0.011731036007404327,
+            -0.023592690005898476,
+            0.00032220553839579225,
+            0.034972693771123886,
+            0.015404638834297657,
+            -0.030515389516949654,
+            -0.00995954405516386,
+            0.034711457788944244,
+            -0.00795130804181099,
+            -0.009118696674704552,
+            -0.009771781973540783,
+            0.003385836724191904,
+            0.02638462744653225,
+            -0.01606588624417782,
+            0.006649219896644354,
+            0.03716052696108818,
+            -0.019870106130838394,
+            0.005543057806789875,
+            -0.025405000895261765,
+            0.0018061877926811576,
+            0.02442537248134613,
+            -0.006216551177203655,
+            0.028098974376916885,
+            5.6315813708351925e-05,
+            0.03025415539741516,
+            -0.004677720367908478,
+            0.008432958275079727,
+            -0.020327266305685043,
+            0.03987082839012146,
+            -0.017633290961384773,
+            0.03402572125196457,
+            0.02067013457417488,
+            -0.01258004643023014,
+            0.04783846437931061,
+            -0.036801330745220184,
+            -0.020784424617886543,
+            0.00584919098764658,
+            0.012743317522108555,
+            0.0028919412288814783,
+            -0.0191517136991024,
+            -0.018400665372610092,
+            -0.010996315628290176,
+            0.02380494214594364,
+            -0.022547753527760506,
+            -0.016865916550159454,
+            -0.007441085297614336,
+            -0.01890680566430092,
+            -0.009673818945884705,
+            -0.007988044060766697,
+            -0.01709449477493763,
+            -0.016547536477446556,
+            0.044573038816452026,
+            -0.06334923207759857,
+            0.016996532678604126,
+            0.017159804701805115,
+            0.011477965861558914,
+            -0.034678805619478226,
+            -0.019510909914970398,
+            -0.017633290961384773,
+            -0.002347023691982031,
+            -0.030743969604372978,
+            0.001702102366834879,
+            0.01725776679813862,
+            0.01138000376522541,
+            -0.05022222548723221,
+            0.012335140258073807,
+            0.02772345207631588,
+            -0.012065742164850235,
+            0.0014949520118534565,
+            -0.023135529831051826,
+            -0.017176130786538124,
+            0.0031776658724993467,
+            0.022123249247670174,
+            0.005975726060569286,
+            -0.008188051171600819,
+            -0.016865916550159454,
+            -0.005241005681455135,
+            -0.006045116577297449,
+            -0.002359268954023719,
+            0.005094061605632305,
+            0.011780017986893654,
+            0.019690508022904396,
+            -0.0024633544962853193,
+            -0.004436895251274109,
+            -0.006396149750798941,
+            0.027364253997802734,
+            -0.001450052484869957,
+            -0.008416631259024143,
+            0.050907962024211884,
+            0.03556047007441521,
+            0.006094098091125488,
+            -0.01108611561357975,
+            -0.03990348428487778,
+            0.010661610402166843,
+            -0.022711025550961494,
+            0.0401320643723011,
+            0.01905374974012375,
+            -0.008298259228467941,
+            -0.028866350650787354,
+            0.010294250212609768,
+            -0.02290695160627365,
+            -0.009975871071219444,
+            0.009469729848206043,
+            0.00726556871086359,
+            -0.03263791650533676,
+            -0.013543346896767616,
+            0.03510330989956856,
+            0.011437147855758667,
+            -0.006841063499450684,
+            -0.036050282418727875,
+            0.008588065393269062,
+            0.01646590232849121,
+            0.011869817040860653,
+            -0.019233347848057747,
+            0.004449140280485153,
+            -0.012041252106428146,
+            0.017470018938183784,
+            0.008849299512803555,
+            -0.017796561121940613,
+            0.018237393349409103,
+            0.004710374400019646,
+            -0.008490103296935558,
+            -0.003912386484444141,
+            -0.0163107942789793,
+            -0.02006603218615055,
+            0.03840138763189316,
+            0.024490682408213615,
+            0.026335645467042923,
+            0.005457340274006128,
+            0.02008235827088356,
+            0.03227871656417847,
+            0.011282040737569332,
+            -0.03134807199239731,
+            0.027935704216361046,
+            0.016996532678604126,
+            -0.04604247957468033,
+            0.021551799029111862,
+            -0.02218855731189251,
+            -0.028670424595475197,
+            0.009649328887462616,
+            -0.0017367975087836385,
+            0.011192241683602333,
+            0.0034837995190173388,
+            -0.003642988856881857,
+            -0.03307874873280525,
+            -0.006236960180103779,
+            0.015771998092532158,
+            0.015502601861953735,
+            -0.0048695639707148075,
+            -0.003296037670224905,
+            -0.011886144056916237,
+            0.00487364549189806,
+            0.009796272963285446,
+            0.039054471999406815,
+            0.001381682581268251,
+            -0.004751192405819893,
+            -0.0018765984568744898,
+            0.031854212284088135,
+            0.0271193478256464,
+            -0.017274094745516777,
+            -0.026286665350198746,
+            0.00580837344750762,
+            -0.014253576286137104,
+            0.026156047359108925,
+            -0.00487364549189806,
+            0.011877980083227158,
+            0.008718682453036308,
+            -0.0007908449042588472,
+            -0.001373519073240459,
+            0.025584599003195763,
+            -0.02710302174091339,
+            -0.021764053031802177,
+            0.031233781948685646,
+            -0.025731543079018593,
+            0.028735732659697533,
+            0.013061696663498878,
+            -0.01557607389986515,
+            -0.015225040726363659,
+            -0.010302413254976273,
+            0.009943217039108276,
+            -0.012277995236217976,
+            0.023853924125432968,
+            0.003608293831348419,
+            0.006392067763954401,
+            0.0003584313380997628,
+            0.02770712412893772,
+            0.024833550676703453,
+            0.007820690982043743,
+            -0.018074123188853264,
+            -0.0016816934803500772,
+            -0.02320083975791931,
+            -0.01680060662329197,
+            0.023772288113832474,
+            -0.018090449273586273,
+            -0.03268689662218094,
+            -0.028556134551763535,
+            0.012114724144339561,
+            0.007392104249447584,
+            0.010816717520356178,
+            0.03461349755525589,
+            -0.030384771525859833,
+            -0.0344502255320549,
+            0.01766594499349594,
+            0.026270337402820587,
+            -0.005551221314817667,
+            -0.021535472944378853,
+            0.03085825778543949,
+            -0.005828781984746456,
+            0.017600636929273605,
+            0.0404912605881691,
+            -0.0016225076979026198,
+            0.010228941217064857,
+            -0.042026009410619736,
+            -0.014057651162147522,
+            -0.0066614653915166855,
+            0.006261450704187155,
+            0.034417569637298584,
+            0.01048201136291027,
+            0.01535565685480833,
+            0.03448288142681122,
+            -0.021192602813243866,
+            -0.0002903166168835014,
+            -0.0022449791431427,
+            -0.02818061038851738,
+            0.03347059711813927,
+            0.025715215131640434,
+            0.011918798089027405,
+            0.019706835970282555,
+            -0.03794422745704651,
+            -0.013037205673754215,
+            0.027070365846157074,
+            0.015053605660796165,
+            0.01273515447974205,
+            0.016694480553269386,
+            0.011249386705458164,
+            0.00868602842092514,
+            -0.021633435040712357,
+            -0.05169166624546051,
+            0.01214737817645073,
+            0.01993541419506073,
+            -0.016449574381113052,
+            0.01057181041687727,
+            0.024980494752526283,
+            0.007620683871209621,
+            -0.007045152597129345,
+            0.012808626517653465,
+            -0.02682545967400074,
+            0.002259265398606658,
+            0.0019010890973731875,
+            -0.014482156373560429,
+            0.038270771503448486,
+            -0.016367938369512558,
+            0.011363675817847252,
+            -0.018204739317297935,
+            0.023870250210165977,
+            0.00011531029304023832,
+            0.0007964573451317847,
+            0.014963806606829166,
+            -0.011347348801791668,
+            -0.04235255345702171,
+            0.0021613026037812233,
+            0.03342161700129509,
+            -0.015200549736618996,
+            0.026449935510754585,
+            -0.0020674217958003283,
+            0.032458316534757614,
+            0.042777057737112045,
+            0.017731253057718277,
+            0.0003497575526125729,
+            0.025682561099529266,
+            0.0036144163459539413,
+            0.0073880222626030445,
+            0.012400448322296143,
+            0.009322785772383213,
+            0.007245160173624754,
+            0.030809277668595314,
+            -0.010261595249176025,
+            0.02173139713704586,
+            -0.04760988429188728,
+            -0.014629100449383259,
+            0.012024925090372562,
+            -0.0013316808035597205,
+            -0.012449429370462894,
+            -0.033029764890670776,
+            0.012563719414174557,
+            -0.03503800183534622,
+            0.022074267268180847,
+            -0.0004959363141097128,
+            0.04333217814564705,
+            -0.02187834307551384,
+            -0.005518566817045212,
+            0.016694480553269386,
+            0.0016510800924152136,
+            0.023282473906874657,
+            -0.0012153501156717539,
+            -0.016416920349001884,
+            -0.0067798369564116,
+            -0.009257477708160877,
+            0.004138925112783909,
+            0.04375668615102768,
+            -0.003722583642229438,
+            -0.0001682458823779598,
+            0.01409030519425869,
+            -0.011959616094827652,
+            -0.03250729665160179,
+            0.045193471014499664,
+            -0.0030837850645184517,
+            0.02484987862408161,
+            0.04979771748185158,
+            -0.06410028040409088,
+            -0.019543563947081566,
+            0.013061696663498878,
+            -0.010441194288432598,
+            0.011363675817847252,
+            0.005547139327973127,
+            0.016114868223667145,
+            0.017894525080919266,
+            -0.005485912784934044,
+            0.01619650423526764,
+            0.005106307100504637,
+            0.028098974376916885,
+            -0.02336410991847515,
+            -0.03296445682644844,
+            -0.02636830136179924,
+            -0.017747581005096436,
+            0.0014480114914476871,
+            -0.015796490013599396,
+            -3.563585551091819e-06,
+            -0.0022204883862286806,
+            -0.04388730227947235,
+            -0.025274382904171944,
+            0.040687184780836105,
+            0.006706364918500185,
+            0.02458864450454712,
+            0.0077268099412322044,
+            0.03418898954987526,
+            0.005885927006602287,
+            -0.009037061594426632,
+            0.030058229342103004,
+            -0.020049704238772392,
+            0.03833607956767082,
+            0.022106921300292015,
+            -0.002832755446434021,
+            0.005865518003702164,
+            0.031086837872862816,
+            -0.033797141164541245,
+            0.008086007088422775,
+            -0.034384917467832565,
+            0.0046695563942193985,
+            -0.008620720356702805,
+            -0.015094423666596413,
+            -0.009526874870061874,
+            -0.02504580281674862,
+            0.011216731742024422,
+            -0.021535472944378853,
+            0.01590261608362198,
+            0.010416703298687935,
+            -0.002916431985795498,
+            0.03448288142681122,
+            -0.014204595237970352,
+            -0.0027796924114227295,
+            0.0024531499948352575,
+            -0.027070365846157074,
+            -0.006343086715787649,
+            -0.003377673216164112,
+            -0.03689929470419884,
+            -0.004502203781157732,
+            0.010988152585923672,
+            -0.00024184548237826675,
+            0.016898570582270622,
+            0.02787039615213871,
+            0.0005092020728625357,
+            -0.022302847355604172,
+            0.002340900944545865,
+            -0.04199335351586342,
+            0.0011877981014549732,
+            0.01483318954706192,
+            -0.017600636929273605,
+            -0.008612556383013725,
+            0.033046092838048935,
+            0.023298801854252815,
+            -0.009396257810294628,
+            -0.035005345940589905,
+            0.017796561121940613,
+            -0.011853490024805069,
+            0.027511198073625565,
+            -0.0002630197268445045,
+            0.019298657774925232,
+            -0.0076941559091210365,
+            0.04493223875761032,
+            0.001369437319226563,
+            0.008645210415124893,
+            -0.021959977224469185,
+            0.007020662073045969,
+            -0.020278284326195717,
+            -0.002406209474429488,
+            -0.006171651650220156,
+            -0.012751481495797634,
+            -0.010669773444533348,
+            0.0032735876739025116,
+            0.007730891928076744,
+            -0.0176822729408741,
+            0.0055961208418011665,
+            -0.015959760174155235,
+            -0.02741323597729206,
+            0.007543129846453667,
+            -0.05965929850935936,
+            -0.0179598331451416,
+            -0.03373183310031891,
+            -0.009477893821895123,
+            0.02008235827088356,
+            0.03510330989956856,
+            -0.012759645469486713,
+            0.00497977202758193,
+            -0.014898497611284256,
+            0.004477712791413069,
+            0.03490738570690155,
+            0.044442422688007355,
+            -0.014759717509150505,
+            0.003381754970178008,
+            -0.010955498553812504,
+            0.033029764890670776,
+            -0.03347059711813927,
+            0.0030062312725931406,
+            0.031086837872862816,
+            -0.005057325586676598,
+            0.007959471084177494,
+            0.011624909937381744,
+            0.020474210381507874,
+            -0.019396619871258736,
+            0.00877582747489214,
+            0.002100075827911496,
+            0.019233347848057747,
+            0.02171507105231285,
+            0.020343592390418053,
+            -0.03539719805121422,
+            0.028409190475940704,
+            0.018384337425231934,
+            0.02444170042872429,
+            0.044442422688007355,
+            -0.03974021226167679,
+            -0.004147088620811701,
+            -0.0032225653994828463,
+            -0.02997659333050251,
+            0.02008235827088356,
+            -0.007530884351581335,
+            -0.01841699331998825,
+            -0.007069643586874008,
+            0.03327467292547226,
+            0.014571955427527428,
+            -0.05051611363887787,
+            0.01980479806661606,
+            0.01740471087396145,
+            0.02055584453046322,
+            0.024474354460835457,
+            -0.01123305968940258,
+            -0.0019072118448093534,
+            0.008996243588626385,
+            0.006804327480494976,
+            0.019266001880168915,
+            0.015257694758474827,
+            0.002012317767366767,
+            -0.0022082431241869926,
+            -0.011837163008749485,
+            0.005224678665399551,
+            -0.009077879600226879,
+            0.010808554477989674,
+            -0.02458864450454712,
+            -0.009698309935629368,
+            0.021388528868556023,
+            0.0031164393294602633,
+            -0.013665799982845783,
+            -0.03729114308953285,
+            -0.045944519340991974,
+            0.0033837957307696342,
+            0.004543021321296692,
+            -0.05365091934800148,
+            0.005322641227394342,
+            -0.026547899469733238,
+            -0.002108239568769932,
+            -0.005502239800989628,
+            0.0019214981002733111,
+            0.005636938381940126,
+            0.0167679525911808,
+            -0.013494365848600864,
+            0.02248244546353817,
+            -0.015502601861953735,
+            0.03165828809142113,
+            0.0055961208418011665,
+            0.007375776767730713,
+            -0.0027796924114227295,
+            0.05590406060218811,
+            -0.003712379140779376,
+            -0.03699725493788719,
+            0.023282473906874657,
+            -0.01311067771166563,
+            -0.0001747512142173946,
+            0.030090883374214172,
+            0.00884113647043705,
+            -0.015257694758474827,
+            0.013437220826745033,
+            -0.030613351613283157,
+            -0.03249097242951393,
+            0.010996315628290176,
+            -0.013755599036812782,
+            -0.0015776080545037985,
+            -0.04568328335881233,
+            -0.009339112788438797,
+            0.004269542172551155,
+            -0.018335357308387756,
+            0.010775899514555931,
+            0.01631895825266838,
+            0.013992343097925186,
+            -0.024245774373412132,
+            0.009053388610482216,
+            0.006865554489195347,
+            0.013453547842800617,
+            0.0007331897504627705,
+            0.014049487188458443,
+            -0.008677864447236061,
+            -0.011306531727313995,
+            -0.018319029361009598,
+            0.0032450153958052397,
+            -0.026792805641889572,
+            0.005155288614332676,
+            -0.003155216109007597,
+            -0.03771565109491348,
+            -0.06521052122116089,
+            -0.0344502255320549,
+            0.016294466331601143,
+            0.03396041318774223,
+            -0.009045224636793137,
+            0.0035144127905368805,
+            -0.00023648813657928258,
+            -0.01093917153775692,
+            0.024523336440324783,
+            -0.005930826533585787,
+            -0.0004829256213270128,
+            0.0035389035474509,
+            0.02050686441361904,
+            -0.009722800925374031,
+            0.005845109466463327,
+            -0.051952898502349854,
+            -0.01586179807782173,
+            0.01812310516834259,
+            0.006306350696831942,
+            0.014629100449383259,
+            -0.009690145961940289,
+            0.01693122461438179,
+            -0.030074557289481163,
+            -0.028147956356406212,
+            0.04130761697888374,
+            0.008277850225567818,
+            0.011486129835247993,
+            -0.003055212553590536,
+            -0.01680060662329197,
+            0.02504580281674862,
+            -0.0020633398089557886,
+            -0.006461458280682564,
+            -0.00817988719791174,
+            0.015820980072021484,
+            0.044279150664806366,
+            0.003255219664424658,
+            0.0009821783751249313,
+            0.017045514658093452,
+            -0.002663361607119441,
+            -0.011575928889214993,
+            0.007677828427404165,
+            0.013592327944934368,
+            -0.016384266316890717,
+            0.010375885292887688,
+            -0.025862159207463264,
+            0.0332583449780941,
+            0.03131541982293129,
+            0.011575928889214993,
+            -0.015045441687107086,
+            0.006130834110081196,
+            0.005330805201083422,
+            0.008086007088422775,
+            -0.001369437319226563,
+            -0.0017133272485807538,
+            0.020719116553664207,
+            -0.03276853263378143,
+            -0.01693122461438179,
+            -0.005347132217139006,
+            0.003932795487344265,
+            0.005277741700410843,
+            0.03396041318774223,
+            -0.0033266509417444468,
+            0.017927179113030434,
+            -0.011657563969492912,
+            0.010579974390566349,
+            0.021764053031802177,
+            0.037095218896865845,
+            -0.024996822699904442,
+            -0.029862305149435997,
+            0.012914752587676048,
+            -0.001296985661610961,
+            0.014065815135836601,
+            0.0010173837654292583,
+            0.014139287173748016,
+            0.0029409227427095175,
+            0.060475654900074005,
+            -0.017763907089829445,
+            0.004628738854080439,
+            0.0004035860183648765,
+            -0.01858026348054409,
+            -0.0013531101867556572,
+            0.004261378664523363,
+            -0.010979988612234592,
+            0.03608293831348419,
+            0.002344982698559761,
+            0.039185091853141785,
+            0.013918871060013771,
+            0.009290131740272045,
+            0.0015143405180424452,
+            -0.013592327944934368,
+            0.008355404250323772,
+            0.0015592400450259447,
+            -0.001132694073021412,
+            0.009461566805839539,
+            0.019298657774925232,
+            0.009714636951684952,
+            0.025274382904171944,
+            0.009788108989596367,
+            0.012857607565820217,
+            -0.029470453038811684,
+            0.03510330989956856,
+            -0.0006224715034477413,
+            0.012996388599276543,
+            -0.02040890045464039,
+            -0.003136848099529743,
+            -0.0008240094175562263,
+            0.015649545937776566,
+            -0.015616890974342823,
+            -0.001201063860207796,
+            -0.01297189760953188,
+            0.040099408477544785,
+            -0.018988441675901413,
+            0.013976015150547028,
+            0.009673818945884705,
+            -0.02865409664809704,
+            -0.007596192881464958,
+            0.01995174214243889,
+            0.014294394291937351,
+            -0.0053144777193665504,
+            -0.020490536466240883,
+            0.005832863971590996,
+            0.015380147844552994,
+            -0.028572462499141693,
+            1.3584674888988957e-05,
+            0.014751553535461426,
+            -0.0164903923869133,
+            -0.00025409081717953086,
+            -0.004702210891991854,
+            0.003975654020905495,
+            -0.009265640750527382,
+            0.007028825581073761,
+            0.016057724133133888,
+            -0.024507008492946625,
+            -0.020049704238772392,
+            0.005722655914723873,
+            0.006571666337549686,
+            0.01444950234144926,
+            0.007739055436104536,
+            0.004355259705334902,
+            -0.020033378154039383,
+            -0.01799248717725277,
+            0.0032674651592969894,
+            -0.01649855636060238,
+            -0.0048614004626870155,
+            0.0020531355403363705,
+            -0.019363965839147568,
+            0.01875986158847809,
+            -0.010743245482444763,
+            0.00017538899555802345,
+            -0.003226647386327386,
+            -0.020800752565264702,
+            0.006204306147992611,
+            -0.004865481983870268,
+            0.019870106130838394,
+            0.003555230563506484,
+            0.02785406820476055,
+            0.02310287579894066,
+            -0.004428731743246317,
+            0.017600636929273605,
+            -0.016155686229467392,
+            0.007983962073922157,
+            -0.031135819852352142,
+            -0.001782717532478273,
+            -0.010408539324998856,
+            0.0036103345919400454,
+            -0.015943434089422226,
+            -0.009126860648393631,
+            -0.0048777274787425995,
+            -0.0006678812787868083,
+            -0.004820582456886768,
+            0.0004275664687156677,
+            0.0027817331720143557,
+            0.02231917530298233,
+            -0.013208640739321709,
+            -0.02803366631269455,
+            -0.017437364906072617,
+            0.006045116577297449,
+            0.019396619871258736,
+            -0.026025431230664253,
+            -0.009184005670249462,
+            -0.005012426059693098,
+            -0.004326687194406986,
+            -0.021404854953289032,
+            -0.015167895704507828,
+            -0.005632856860756874,
+            -0.008045189082622528,
+            -0.010522829368710518,
+            0.0046532293781638145,
+            -0.0035715578123927116,
+            -0.011739199981093407,
+            0.021666089072823524,
+            0.018057795241475105,
+            -0.019837452098727226,
+            -0.019837452098727226,
+            0.013641309924423695,
+            -0.006159406621009111,
+            0.0018878233386203647,
+            0.02008235827088356,
+            0.00029286774224601686,
+            0.005824700463563204,
+            0.020882388576865196,
+            -0.0035225762985646725,
+            -0.01797616109251976,
+            0.003577680326998234,
+            -0.0018888438353314996,
+            -0.012122887186706066,
+            0.003212361130863428,
+            0.02773977816104889,
+            0.0008076822850853205,
+            0.023429417982697487,
+            -0.03056436963379383,
+            -0.011412657797336578,
+            0.006061443593353033,
+            0.02369065210223198,
+            0.00211844383738935,
+            -0.0021694661118090153,
+            0.020131340250372887,
+            0.03941367194056511,
+            0.005775718949735165,
+            0.0015755671774968505,
+            -0.04003410041332245,
+            0.01027792226523161,
+            0.009208496659994125,
+            0.024621298536658287,
+            0.008339077234268188,
+            0.0014653591206297278,
+            -0.012906589545309544,
+            -0.009755454957485199,
+            -0.0017939424142241478,
+            0.014702572487294674,
+            -0.023870250210165977,
+            0.0024755997583270073,
+            -0.03157665207982063,
+            0.005094061605632305,
+            0.0021959978621453047,
+            -0.009559528902173042,
+            0.0335359051823616,
+            0.00980443600565195,
+            -0.02065380848944187,
+            0.015151568688452244,
+            0.005767555441707373,
+            0.007330877240747213,
+            0.022449791431427002,
+            -0.008228869177401066,
+            0.030450081452727318,
+            -0.005049162078648806,
+            0.01213921420276165,
+            0.02742956392467022,
+            0.032588932663202286,
+            -0.013102514669299126,
+            -0.003959327004849911,
+            0.0006097159348428249,
+            -0.0002561317232903093,
+            0.0007995187188498676,
+            -0.008204378187656403,
+            0.0037389106582850218,
+            -0.014800535514950752,
+            0.012710663489997387,
+            -0.019543563947081566,
+            -0.0034286954905837774,
+            0.0038532004691660404,
+            0.01991908811032772,
+            0.0016265894519165158,
+            -0.033633869141340256,
+            0.02055584453046322,
+            -0.015420965850353241,
+            0.0135596739128232,
+            0.001290863030590117,
+            -0.011110605672001839,
+            -0.00029210239881649613,
+            0.020588500425219536,
+            0.010947334580123425,
+            -0.007845181971788406,
+            -0.0135596739128232,
+            -0.028376536443829536,
+            -0.021535472944378853,
+            0.00823703221976757,
+            0.009445239789783955,
+            -0.0028837777208536863,
+            0.0021306893322616816,
+            -0.011869817040860653,
+            0.0006520643946714699,
+            -0.0008122742874547839,
+            0.01740471087396145,
+            -0.0335359051823616,
+            0.021943651139736176,
+            -0.019527235999703407,
+            -0.019543563947081566,
+            0.017045514658093452,
+            0.008049271069467068,
+            0.009445239789783955,
+            0.0033552234526723623,
+            -0.01995174214243889,
+            -0.012326976284384727,
+            0.0338951013982296,
+            -0.01138000376522541,
+            0.006298186723142862,
+            0.004408322740346193,
+            0.012914752587676048,
+            -0.008914608508348465,
+            -0.011926962062716484,
+            0.01784554310142994,
+            -0.009861581027507782,
+            0.013951525092124939,
+            0.0010010566329583526,
+            -0.0254703089594841,
+            0.02739690989255905,
+            -0.019233347848057747,
+            0.03487472981214523,
+            -0.021617108955979347,
+            0.00711862463504076,
+            0.0066614653915166855,
+            0.008792154490947723,
+            -0.006690037902444601,
+            -0.013224967755377293,
+            0.0012786176521331072,
+            0.03807484731078148,
+            -0.02623768337070942,
+            0.024180466309189796,
+            0.022400809451937675,
+            -0.00040766780148260295,
+            0.02065380848944187,
+            -0.008367649279534817,
+            -0.026711169630289078,
+            0.0179598331451416,
+            -0.012596373446285725,
+            -0.026123393326997757,
+            0.013257622718811035,
+            -0.022678371518850327,
+            -0.020278284326195717,
+            -0.026302991434931755,
+            0.004947117529809475,
+            -0.0025694805663079023,
+            -0.005151206627488136,
+            0.009004407562315464,
+            0.02561725303530693,
+            0.020915042608976364,
+            0.019576217979192734,
+            0.02111096680164337,
+            0.01161674689501524,
+            0.022449791431427002,
+            -0.007559456862509251,
+            0.04401791840791702,
+            -0.004947117529809475,
+            0.004632820375263691,
+            -0.0015980169409886003,
+            -0.015657709911465645,
+            0.009771781973540783,
+            0.0119106350466609,
+            0.0029572497587651014,
+            -0.029176564887166023,
+            -0.009061552584171295,
+            -0.03660540655255318,
+            0.005053244065493345,
+            0.0020664012990891933,
+            0.0176822729408741,
+            0.032115448266267776,
+            -0.005865518003702164,
+            0.004592002835124731,
+            -0.005102225113660097,
+            -0.013927034102380276,
+            -0.009420748800039291,
+            -0.02739690989255905,
+            -0.025437654927372932,
+            -0.007653337903320789,
+            -0.013739272020757198,
+            -0.030368445441126823,
+            -0.011722872965037823,
+            0.01551892887800932,
+            -0.007645174395292997,
+            0.011918798089027405,
+            -0.016702644526958466,
+            0.01385356206446886,
+            -0.004110352601855993,
+            -0.004763437435030937,
+            -0.004530776292085648,
+            -0.017600636929273605,
+            -0.006420640274882317,
+            0.014367866329848766,
+            -0.009477893821895123,
+            -0.01237595733255148,
+            -0.02215590327978134,
+            -0.0015459742862731218,
+            -0.01574750803411007,
+            0.003489922033622861,
+            -0.014596446417272091,
+            -0.006771673448383808,
+            -0.003908304497599602,
+            -0.018514955416321754,
+            0.01766594499349594,
+            0.014816862531006336,
+            -0.019380291923880577,
+            -0.0090697156265378,
+            0.02638462744653225,
+            0.042777057737112045,
+            -0.03218075633049011,
+            -0.008200296200811863,
+            0.0020378287881612778,
+            0.018090449273586273,
+            0.010171796195209026,
+            0.016702644526958466,
+            -0.024653952568769455,
+            0.0022286521270871162,
+            -0.021649762988090515,
+            -0.001220452249981463,
+            -0.023739634081721306,
+            0.010775899514555931,
+            0.025715215131640434,
+            -0.010971825569868088,
+            0.025976449251174927,
+            0.01634344831109047,
+            -0.01812310516834259,
+            0.012016761116683483,
+            0.007183933164924383,
+            0.014645427465438843,
+            -0.019755816087126732,
+            0.0167679525911808,
+            -0.015796490013599396,
+            0.031233781948685646,
+            0.00892277155071497,
+            -0.04829562455415726,
+            -0.03503800183534622,
+            -0.013388238847255707,
+            0.0013388239312916994,
+            -0.010702427476644516,
+            0.015037278644740582,
+            0.015967924147844315,
+            0.013959688134491444,
+            -0.019380291923880577,
+            -0.015820980072021484,
+            0.011984107084572315,
+            0.010065670125186443,
+            0.027462217956781387,
+            -0.006649219896644354,
+            -0.004212397150695324,
+            -0.00960034690797329,
+            -0.02249877341091633,
+            -0.016098542138934135,
+            0.0209803506731987,
+            -0.018041469156742096,
+            0.015935270115733147,
+            -0.026139721274375916,
+            -0.012449429370462894,
+            0.017453692853450775,
+            -0.05211617052555084,
+            -0.01453113742172718,
+            0.0007612520130351186,
+            0.03012353740632534,
+            0.004751192405819893,
+            0.0035184945445507765,
+            0.01841699331998825,
+            -0.01993541419506073,
+            -0.0039776950143277645,
+            -0.02099667675793171,
+            0.006832899991422892,
+            -0.011853490024805069,
+            -0.013984179124236107,
+            0.014482156373560429,
+            -0.013845399022102356,
+            -0.01601690612733364,
+            0.0034368589986115694,
+            0.007183933164924383,
+            -0.021094640716910362,
+            0.011951452121138573,
+            0.003945040516555309,
+            0.012171869166195393,
+            0.012171869166195393,
+            0.01199227012693882,
+            0.021551799029111862,
+            0.02295593172311783,
+            -0.007334959227591753,
+            -0.004767519421875477,
+            -0.01004117913544178,
+            -0.0015398516552522779,
+            0.008898280560970306,
+            0.0033797139767557383,
+            -0.002989904023706913,
+            0.007837017998099327,
+            0.010033016093075275,
+            -0.01498013362288475,
+            0.033193036913871765,
+            -0.0035878848284482956,
+            -0.04134026914834976,
+            -0.003238892648369074,
+            -0.03800953924655914,
+            0.020033378154039383,
+            -0.023298801854252815,
+            -0.016996532678604126,
+            0.0032776694279164076,
+            -0.00629410520195961,
+            -0.019902760162949562,
+            0.013608654960989952,
+            0.011404493823647499,
+            0.00258580781519413,
+            0.02021297626197338,
+            0.012269831262528896,
+            -0.005134879611432552,
+            0.026172375306487083,
+            -0.014718899503350258,
+            -0.008751336485147476,
+            0.015567909926176071,
+            0.006853308994323015,
+            0.0022347746416926384,
+            0.0011031010653823614,
+            -0.018825169652700424,
+            0.005098143592476845,
+            0.0164903923869133,
+            0.007216587662696838,
+            -0.000566347036510706,
+            0.014049487188458443,
+            0.012514738366007805,
+            -0.015135240741074085,
+            -0.0002213090192526579,
+            -0.026564225554466248,
+            0.0032817514147609472,
+            0.004567512311041355,
+            0.007506393827497959,
+            -0.006771673448383808,
+            0.006281859707087278,
+            0.00658799335360527,
+            0.014759717509150505,
+            -0.010155469179153442,
+            -0.016980206593871117,
+            -0.002510294783860445,
+            0.004285869188606739,
+            0.016694480553269386,
+            0.03157665207982063,
+            -0.010310577228665352,
+            -0.005734901409596205,
+            -0.0386626236140728,
+            0.02070278860628605,
+            0.03332365304231644,
+            0.011175914667546749,
+            -0.028213264420628548,
+            0.012302485294640064,
+            0.0006515541463159025,
+            0.0019276207312941551,
+            -0.026302991434931755,
+            0.02787039615213871,
+            0.00974729098379612,
+            0.0066614653915166855,
+            0.017861871048808098,
+            0.0022306928876787424,
+            0.0029266364872455597,
+            -0.016302630305290222,
+            0.01018812321126461,
+            0.009086042642593384,
+            -0.017306748777627945,
+            0.0009745250572450459,
+            0.02324981987476349,
+            -0.01423724927008152,
+            -0.003208279376849532,
+            0.0018133309204131365,
+            0.02651524543762207,
+            -0.013820908032357693,
+            -0.014294394291937351,
+            0.0005714492290280759,
+            0.01751900091767311,
+            -0.0007979880319908261,
+            0.012098397128283978,
+            0.04898136481642723,
+            -0.012424939312040806,
+            0.019657853990793228,
+            0.005461421795189381,
+            -0.015290348790585995,
+            -0.032605260610580444,
+            -0.004922627005726099,
+            -0.01830270327627659,
+            0.011200404725968838,
+            0.014343376271426678,
+            -0.015763835981488228,
+            0.0007525782566517591,
+            -0.03144603595137596,
+            0.007743136957287788,
+            0.014735226519405842,
+            0.018776189535856247,
+            -0.002440904499962926,
+            0.006134915631264448,
+            0.027674470096826553,
+            0.02458864450454712,
+            -0.007392104249447584,
+            0.013804581016302109,
+            0.02320083975791931,
+            -0.025062130764126778,
+            -0.003069498809054494,
+            0.006820654962211847,
+            0.004104230087250471,
+            0.0053063142113387585,
+            -0.016963878646492958,
+            0.03915243595838547,
+            -0.005922663025557995,
+            -0.01506176870316267,
+            0.0030266400426626205,
+            -0.009028897620737553,
+            0.0063390047289431095,
+            -0.012563719414174557,
+            0.012637191452085972,
+            -0.008114579133689404,
+            0.00022883480414748192,
+            -0.0059430720284581184,
+            -0.011110605672001839,
+            -0.012653518468141556,
+            -0.0007949267164804041,
+            -0.016531210392713547,
+            0.007432921789586544,
+            0.00981259997934103,
+            0.005045080557465553,
+            0.01025343220680952,
+            -0.0036797248758375645,
+            -0.003136848099529743,
+            -0.016751626506447792,
+            -0.009984034113585949,
+            0.0073716952465474606,
+            0.01872720755636692,
+            -0.028588788583874702,
+            -0.018547609448432922,
+            0.009290131740272045,
+            0.0022898786701261997,
+            0.018776189535856247,
+            0.0158944521099329,
+            -0.0002595757250674069,
+            0.012318813242018223,
+            -0.01964152604341507,
+            -0.009184005670249462,
+            0.009690145961940289,
+            -0.003569516818970442,
+            0.01048201136291027,
+            -0.014759717509150505,
+            0.010033016093075275,
+            -0.011159587651491165,
+            -0.011053460650146008,
+            0.01385356206446886,
+            0.01738838478922844,
+            0.008873790502548218,
+            -0.001294944784604013,
+            3.383413059054874e-05,
+            -0.010506502352654934,
+            -0.024033522233366966,
+            0.004771600943058729,
+            0.027021385729312897,
+            -0.013273949734866619,
+            0.003883813973516226,
+            0.008702355436980724,
+            -0.017502672970294952,
+            -0.019821124151349068,
+            -0.012653518468141556,
+            -0.022106921300292015,
+            0.020572172477841377,
+            0.023413091897964478,
+            -0.02026195637881756,
+            0.00877582747489214,
+            -0.004620575346052647,
+            0.014049487188458443,
+            -0.01400050614029169,
+            0.010612628422677517,
+            0.0317889042198658,
+            -0.020000724121928215,
+            -0.0028654097113758326,
+            0.0036633978597819805,
+            -0.015347493812441826,
+            0.018629245460033417,
+            0.03268689662218094,
+            -0.00176945177372545,
+            0.014041324146091938,
+            0.0008138049161061645,
+            -0.009118696674704552,
+            0.01905374974012375,
+            0.011094278655946255,
+            0.005347132217139006,
+            -0.0042491331696510315,
+            -0.012212686240673065,
+            -0.03010721132159233,
+            0.016702644526958466,
+            0.0129637336358428,
+            -0.0037674831692129374,
+            -0.0011551438365131617,
+            -0.006343086715787649,
+            -0.015404638834297657,
+            0.0035123717971146107,
+            -0.013078023679554462,
+            -0.00803294312208891,
+            -0.008759500458836555,
+            0.004314441699534655,
+            -0.0034960447810590267,
+            -0.009053388610482216,
+            0.007236996199935675,
+            0.0009510547970421612,
+            -0.0010602424154058099,
+            -0.01902109570801258,
+            -0.025078458711504936,
+            -0.00995954405516386,
+            -0.016849588602781296,
+            0.0038287099450826645,
+            0.0005234883283264935,
+            -0.027184655889868736,
+            0.0053961132653057575,
+            -0.008139070123434067,
+            -0.008134988136589527,
+            0.007126788143068552,
+            0.006881881505250931,
+            -0.007437003776431084,
+            -0.010816717520356178,
+            -0.00786150898784399,
+            -0.006865554489195347,
+            0.020033378154039383,
+            0.0034348180051892996,
+            0.012082069180905819,
+            -0.010498338378965855,
+            -0.018025141209363937,
+            -0.016963878646492958,
+            0.016718972474336624,
+            -0.008971752598881721,
+            -0.02083340659737587,
+            -0.01444950234144926,
+            -0.008424794301390648,
+            -0.007665583398193121,
+            -0.01527402177453041,
+            -0.021159948781132698,
+            -0.0032103201374411583,
+            -0.011347348801791668,
+            -0.0067431009374558926,
+            -0.0028960229828953743,
+            0.0013571919407695532,
+            0.00951871182769537,
+            0.00546958576887846,
+            0.00438791373744607,
+            0.021666089072823524,
+            0.010122815147042274,
+            -0.005461421795189381,
+            0.016441410407423973,
+            -0.006183897145092487,
+            -0.004440976772457361,
+            -0.01498013362288475,
+            -0.009918726049363613,
+            0.018547609448432922,
+            -0.0055226488038897514,
+            0.008457448333501816,
+            -0.0009536059224046767,
+            -0.00576347392052412,
+            0.015674035996198654,
+            0.0129637336358428,
+            0.016718972474336624,
+            0.004771600943058729,
+            -0.00012168807734269649,
+            0.012416775338351727,
+            -0.0015745466807857156,
+            0.012555556371808052,
+            -0.0038429962005466223,
+            0.014506647363305092,
+            -0.009722800925374031,
+            -0.018988441675901413,
+            0.004157293122261763,
+            0.006265532691031694,
+            -0.012531065382063389,
+            0.009404421783983707,
+            0.007743136957287788,
+            -0.013747435994446278,
+            -0.02261306345462799,
+            -0.000106317616882734,
+            0.00014362763613462448,
+            0.008122743107378483,
+            0.008971752598881721,
+            -0.008073761127889156,
+            0.012890261597931385,
+            0.004616493359208107,
+            -0.007179851643741131,
+            -0.019657853990793228,
+            0.026009103283286095,
+            0.015608727931976318,
+            0.012269831262528896,
+            -0.052508022636175156,
+            -0.0030980713199824095,
+            0.010743245482444763,
+            -0.015951598063111305,
+            -0.026792805641889572,
+            -0.006404313258826733,
+            -0.008759500458836555,
+            -0.04290767386555672,
+            -0.009624837897717953,
+            -0.0005724696675315499,
+            -0.011388166807591915,
+            0.010400376282632351,
+            0.01573934406042099,
+            0.014612773433327675,
+            0.005489994306117296,
+            -0.007232914678752422,
+            -0.017633290961384773,
+            -0.02788672223687172,
+            0.010057506151497364,
+            -0.013380075804889202,
+            -0.006412476766854525,
+            -0.002263347152620554,
+            -0.007690073922276497,
+            0.0016112826997414231,
+            0.01691489666700363,
+            0.042483169585466385,
+            0.023037567734718323,
+            -0.012335140258073807,
+            -0.020131340250372887,
+            -0.012800462543964386,
+            -0.009192168712615967,
+            0.00633492274209857,
+            0.011592255905270576,
+            -0.004738946910947561,
+            0.0096656559035182,
+            -0.004294032696634531,
+            0.012302485294640064,
+            0.012767808511853218,
+            0.01680060662329197,
+            -2.3023791072773747e-05,
+            0.005579793825745583,
+            -0.0024694770108908415,
+            -0.01738838478922844,
+            -0.004379750229418278,
+            0.008261523209512234,
+            -0.0008872769540175796,
+            0.0019317024853080511,
+            0.014204595237970352,
+            0.019102731719613075,
+            -0.009094206616282463,
+            -0.014612773433327675,
+            -0.016098542138934135,
+            -0.005277741700410843,
+            -0.0012622905196622014,
+            0.029927613213658333,
+            0.02037624642252922,
+            -0.019266001880168915,
+            -0.018237393349409103,
+            -0.018351683393120766,
+            0.00922482367604971,
+            -0.003563394071534276,
+            -0.0064369672909379005,
+            0.01905374974012375,
+            0.023021239787340164,
+            0.017910851165652275,
+            -0.003642988856881857,
+            -0.014865843579173088,
+            0.003804219188168645,
+            0.02847449854016304,
+            0.008367649279534817,
+            0.01078406348824501,
+            0.008726846426725388,
+            -0.0011908593587577343,
+            -0.032441988587379456,
+            -0.0113146947696805,
+            0.00951871182769537,
+            -0.008481939323246479,
+            0.026564225554466248,
+            -0.016531210392713547,
+            0.009388094767928123,
+            -0.005077734589576721,
+            0.0019286411115899682,
+            -0.02365799807012081,
+            0.0014929111348465085,
+            0.010122815147042274,
+            -0.024947840720415115,
+            0.00336950970813632,
+            -0.023592690005898476,
+            -0.00013763252354692668,
+            0.008367649279534817,
+            0.0022837561555206776,
+            0.01650671847164631,
+            -0.0113146947696805,
+            0.0051838611252605915,
+            0.02099667675793171,
+            -0.002832755446434021,
+            -0.01512707769870758,
+            -0.0007842120248824358,
+            0.0005092020728625357,
+            -0.001286781276576221,
+            0.02563357912003994,
+            0.015812816098332405,
+            -0.0036756431218236685,
+            0.020882388576865196,
+            0.008677864447236061,
+            0.0019133344758301973,
+            -0.021764053031802177,
+            0.0023796779569238424,
+            -0.032899148762226105,
+            0.017437364906072617,
+            -0.011126932688057423,
+            0.012490247376263142,
+            0.00868602842092514,
+            -0.01341272983700037,
+            -0.0007388022495433688,
+            -0.0107187544927001,
+            -0.0015725058037787676,
+            -0.007934981025755405,
+            0.009779945947229862,
+            -0.004363423213362694,
+            0.008339077234268188,
+            -0.00024579971795901656,
+            -0.01476788055151701,
+            0.018694553524255753,
+            9.678283277025912e-06,
+            -0.00010925139940809458,
+            0.005441013257950544,
+            0.009935053065419197,
+            0.00786559097468853,
+            0.002009256277233362,
+            0.023870250210165977,
+            -0.031119491904973984,
+            -0.016082214191555977,
+            -0.010204450227320194,
+            0.005543057806789875,
+            -0.007922735065221786,
+            -0.018531281501054764,
+            -0.03257260471582413,
+            0.01318415068089962,
+            0.001293924287892878,
+            -0.0009806477464735508,
+            0.01258004643023014,
+            -0.00942891277372837,
+            -0.0013878053287044168,
+            -0.01606588624417782,
+            -0.014808698557317257,
+            0.006416558753699064,
+            0.0033633869607001543,
+            -0.01222901325672865,
+            -0.023739634081721306,
+            0.014759717509150505,
+            0.0007418635650537908,
+            -0.016980206593871117,
+            -0.017502672970294952,
+            0.00018457300029695034,
+            -0.004804255440831184,
+            0.02204161323606968,
+            -0.016678154468536377,
+            0.003373591462150216,
+            0.0007219648687168956,
+            0.009232986718416214,
+            -0.02889900468289852,
+            0.00716352416202426,
+            0.0006107363733462989,
+            0.0035164537839591503,
+            -0.006518602836877108,
+            -0.021013004705309868,
+            0.011371839791536331,
+            -0.0020674217958003283,
+            -0.014751553535461426,
+            -0.020751770585775375,
+            -0.0020796670578420162,
+            -0.007420676294714212,
+            -0.02724996581673622,
+            0.017910851165652275,
+            0.0009985055075958371,
+            -0.01738838478922844,
+            0.008077843114733696,
+            0.006596156861633062,
+            0.009355440735816956,
+            0.02455599047243595,
+            0.010147306136786938,
+            -0.018776189535856247,
+            -0.0008087027235887945,
+            0.006979844067245722,
+            0.009306458756327629,
+            0.0036450298503041267,
+            0.04222193360328674,
+            -0.008130906149744987,
+            -0.006579829845577478,
+            0.0066614653915166855,
+            -0.008449285291135311,
+            -0.013747435994446278,
+            0.026727497577667236,
+            0.005796127952635288,
+            0.015984252095222473,
+            0.016123032197356224,
+            -0.00945340283215046,
+            -0.005008344538509846,
+            -0.0018786393338814378,
+            0.02065380848944187,
+            0.0073798587545752525,
+            -0.005502239800989628,
+            -0.009714636951684952,
+            0.005881845485419035,
+            -0.00898808054625988,
+            0.0021653843577951193,
+            -0.009526874870061874,
+            -0.005824700463563204,
+            -0.008069679141044617,
+            0.016400592401623726,
+            -0.0003523086488712579,
+            0.027315273880958557,
+            0.020751770585775375,
+            -0.008563575334846973,
+            -0.0005683879135176539,
+            0.017208784818649292,
+            -0.005820618476718664,
+            0.015608727931976318,
+            -0.011192241683602333,
+            -0.007706400938332081,
+            0.006408394780009985,
+            0.024017194285988808,
+            0.012237177230417728,
+            -0.0056042843498289585,
+            0.0002885308349505067,
+            -0.0030348035506904125,
+            -0.010612628422677517,
+            0.013388238847255707,
+            0.015323002822697163,
+            -0.01947825588285923,
+            -0.004722619894891977,
+            -0.0016592437168583274,
+            -0.003300119424238801,
+            0.00945340283215046,
+            0.02305389568209648,
+            0.000171817431692034,
+            0.009184005670249462,
+            0.0012765767751261592,
+            0.03862996771931648,
+            -0.0053144777193665504,
+            0.03102152980864048,
+            -0.0010423846542835236,
+            -0.0059430720284581184,
+            -0.009094206616282463,
+            -0.020441556349396706,
+            -0.011486129835247993,
+            -0.00041481092921458185,
+            -0.005040998570621014,
+            -0.0004956811899319291,
+            -0.020327266305685043,
+            0.0026960158720612526,
+            0.014637264423072338,
+            0.007661501411348581,
+            0.017927179113030434,
+            -0.002889900468289852,
+            0.017437364906072617,
+            0.019690508022904396,
+            -0.01400050614029169,
+            -0.029862305149435997,
+            -0.0006515541463159025,
+            0.01498013362288475,
+            -0.01661284640431404,
+            0.019445601850748062,
+            0.004085862077772617,
+            0.0029531680047512054,
+            0.004779764451086521,
+            -0.006049198564141989,
+            0.018482301384210587,
+            0.02039257436990738,
+            0.01453930139541626,
+            -0.013167822733521461,
+            0.014057651162147522,
+            0.009771781973540783,
+            0.006600238848477602,
+            -0.01646590232849121,
+            -0.013535182923078537,
+            0.002555194543674588,
+            0.0009245232213288546,
+            0.024392718449234962,
+            -0.020588500425219536,
+            0.0016602640971541405,
+            0.010588138364255428,
+            0.01769859902560711,
+            -0.004302196204662323,
+            0.008906444534659386,
+            0.0008740111952647567,
+            0.007134951651096344,
+            -0.0035919665824621916,
+            -0.006457376293838024,
+            -0.006673710886389017,
+            -0.010392212308943272,
+            -0.0020959940738976,
+            -0.02070278860628605,
+            0.002198038622736931,
+            -0.002830714685842395,
+            0.018237393349409103,
+            -0.01018812321126461,
+            0.038140155375003815,
+            0.03399306535720825,
+            -0.0007025764207355678,
+            0.011102442629635334,
+            -0.01709449477493763,
+            -0.003945040516555309,
+            -0.021959977224469185,
+            0.01844964735209942,
+            0.016392430290579796,
+            0.012269831262528896,
+            0.005302232690155506,
+            0.017796561121940613,
+            -0.0006301248213276267,
+            0.014490320347249508,
+            0.0029654132667928934,
+            0.007971717044711113,
+            -0.002265388146042824,
+            0.003938918001949787,
+            -0.031103165820240974,
+            -0.017029186710715294,
+            -0.01138000376522541,
+            0.004796091932803392,
+            -0.006375740747898817,
+            -0.0035205355379730463,
+            0.0084737753495574,
+            -0.005902254022657871,
+            0.0004270562494639307,
+            -0.025992777198553085,
+            -0.0157148540019989,
+            -0.010318740271031857,
+            -0.004040962550789118,
+            -0.01169838197529316,
+            -0.0027694879099726677,
+            -0.0022082431241869926,
+            0.0029552089981734753,
+            0.020947696641087532,
+            -0.012808626517653465,
+            -0.016294466331601143,
+            0.004379750229418278,
+            -0.002263347152620554,
+            -0.014016833156347275,
+            -0.016898570582270622,
+            0.006665547378361225,
+            0.012098397128283978,
+            -0.0017470018938183784,
+            -0.013608654960989952,
+            0.011739199981093407,
+            -0.006628811359405518,
+            0.04663025587797165,
+            -0.013584164902567863,
+            -0.0078002819791436195,
+            0.003720542648807168,
+            0.004698128905147314,
+            -0.009396257810294628,
+            0.007330877240747213,
+            0.013020878657698631,
+            0.01830270327627659,
+            0.0095758568495512,
+            -0.008465612307190895,
+            -0.0024388637393712997,
+            0.007796200457960367,
+            -0.016147522255778313,
+            0.006404313258826733,
+            0.0008525818702764809,
+            -0.014400520361959934,
+            0.006110425107181072,
+            0.028980640694499016,
+            0.013151495717465878,
+            0.01288209855556488,
+            -0.022351829335093498,
+            -0.004094025585800409,
+            -0.009355440735816956,
+            -0.004943036008626223,
+            -0.010090161114931107,
+            0.027070365846157074,
+            -0.018155759200453758,
+            0.02651524543762207,
+            0.00898808054625988,
+            0.0017684312770143151,
+            -0.01176369097083807,
+            0.006343086715787649,
+            0.02832755446434021,
+            0.0028592869639396667,
+            -0.020000724121928215,
+            -0.013592327944934368,
+            0.006314514204859734,
+            0.01707816869020462,
+            -0.002647034591063857,
+            -0.005428767763078213,
+            0.019657853990793228,
+            0.01784554310142994,
+            -0.004334850702434778,
+            -0.00936360377818346,
+            0.0008337036124430597,
+            0.00928196869790554,
+            0.016963878646492958,
+            0.015020951628684998,
+            0.009086042642593384,
+            -0.002647034591063857,
+            0.008037025108933449,
+            -0.008800318464636803,
+            -0.016449574381113052,
+            0.007220669183880091,
+            -0.008432958275079727,
+            -0.016963878646492958,
+            0.01872720755636692,
+            0.015257694758474827,
+            0.0181394312530756,
+            0.0024592725094407797,
+            -0.016735298559069633,
+            0.022302847355604172,
+            0.005755309946835041,
+            0.002998067531734705,
+            0.02160078100860119,
+            -0.022221211344003677,
+            0.013012715615332127,
+            0.02037624642252922,
+            0.027609162032604218,
+            -0.01737205684185028,
+            -0.010906516574323177,
+            -0.004857318475842476,
+            -0.006853308994323015,
+            -0.00176945177372545,
+            -0.008253359235823154,
+            -0.012620864436030388,
+            0.0028735732194036245,
+            -0.030466407537460327,
+            0.018106777220964432,
+            -0.005102225113660097,
+            0.010669773444533348,
+            -0.019266001880168915,
+            0.011363675817847252,
+            -0.0035838030744343996,
+            0.011135096661746502,
+            0.0019612954929471016,
+            -0.030041903257369995,
+            -0.008571738377213478,
+            0.015388311818242073,
+            -0.0030368445441126823,
+            -0.013641309924423695,
+            0.02006603218615055,
+            0.00018942011229228228,
+            -0.022270193323493004,
+            -0.004314441699534655,
+            -0.006934944540262222,
+            0.002193956868723035,
+            -0.0029613315127789974,
+            -0.0052981507033109665,
+            -0.030188847333192825,
+            0.020033378154039383,
+            0.009559528902173042,
+            -0.03072764165699482,
+            -0.003255219664424658,
+            -0.014996460638940334,
+            -0.0025184585247188807,
+            0.008881953544914722,
+            -0.004000144544988871,
+            -0.015551582910120487,
+            0.01784554310142994,
+            0.022711025550961494,
+            -0.0023164101876318455,
+            -0.016057724133133888,
+            -0.0090697156265378,
+            -0.027494871988892555,
+            0.030825603753328323,
+            -0.0066696288995444775,
+            0.0022878379095345736,
+            -0.005322641227394342,
+            0.009388094767928123,
+            0.004228724166750908,
+            0.0157148540019989,
+            0.006081852596253157,
+            -0.002647034591063857,
+            0.026596879586577415,
+            -0.01964152604341507,
+            0.009567692875862122,
+            -1.588864870427642e-05,
+            -0.003308282932266593,
+            0.01890680566430092,
+            0.0007939062779769301,
+            0.021649762988090515,
+            -0.008571738377213478,
+            -0.0001475818717153743,
+            -0.0034388997592031956,
+            0.010898353531956673,
+            -0.0028450009413063526,
+            -0.003465431509539485,
+            -0.001531688030809164,
+            0.011118769645690918,
+            0.01013097818940878,
+            0.003804219188168645,
+            -0.01737205684185028,
+            0.003142970846965909,
+            -0.008547247387468815,
+            -0.007028825581073761,
+            0.01302904263138771,
+            -0.011192241683602333,
+            -0.0018990482203662395,
+            -0.03160930797457695,
+            0.016286302357912064,
+            0.0533243790268898,
+            0.008547247387468815,
+            -0.014433175325393677,
+            -0.0018806802108883858,
+            -0.019755816087126732,
+            0.01535565685480833,
+            0.011004479601979256,
+            -0.016816934570670128,
+            -0.011984107084572315,
+            -0.012310649268329144,
+            0.007649256382137537,
+            0.021306892856955528,
+            -0.006522684823721647,
+            -0.008579902350902557,
+            0.025372346863150597,
+            -0.011788181029260159,
+            -0.010882026515901089,
+            -0.01661284640431404,
+            -0.01635977439582348,
+            0.00241641397587955,
+            0.019902760162949562,
+            -0.0169475506991148,
+            0.021176276728510857,
+            0.00936360377818346,
+            0.0010133018950000405,
+            0.003624620847404003,
+            0.008588065393269062,
+            0.0024878450203686953,
+            0.017878197133541107,
+            0.003883813973516226,
+            -0.01439235731959343,
+            0.0021408938337117434,
+            -0.0030286810360848904,
+            0.004624656867235899,
+            3.4320437407586724e-06,
+            -0.002749078907072544,
+            -0.0065635028295218945,
+            0.003710338147357106,
+            -0.021306892856955528,
+            0.021502818912267685,
+            -0.022678371518850327,
+            -0.004245051648467779,
+            0.005555302836000919,
+            -0.005118552595376968,
+            0.013469874858856201,
+            0.018792515620589256,
+            0.01423724927008152,
+            0.008702355436980724,
+            0.01462093647569418,
+            -0.011331021785736084,
+            0.02323349379003048,
+            -0.013282112777233124,
+            0.0005816536722704768,
+            0.012939243577420712,
+            0.01755165494978428,
+            0.018792515620589256,
+            -0.006245123688131571,
+            -0.010384049266576767,
+            -0.0002946535241790116,
+            0.0129637336358428,
+            -0.022286519408226013,
+            -0.012269831262528896,
+            -0.01386172603815794,
+            0.011486129835247993,
+            -0.004743028432130814,
+            0.025453981012105942,
+            -0.00633492274209857,
+            -0.014653591439127922,
+            -0.015918942168354988,
+            0.014196431264281273,
+            -0.048752784729003906,
+            0.0030348035506904125,
+            -0.0024919267743825912,
+            0.03043375350534916,
+            -0.027021385729312897,
+            0.01738838478922844,
+            -0.0028490826953202486,
+            -0.009355440735816956,
+            0.015429128892719746,
+            -0.015992414206266403,
+            -0.0046613928861916065,
+            -0.0016806729836389422,
+            -0.007539048325270414,
+            -0.00112248957157135,
+            0.017616963014006615,
+            -0.015845470130443573,
+            0.01282495353370905,
+            -0.00989423505961895,
+            0.0027021386194974184,
+            -0.004526694305241108,
+            -0.007751300930976868,
+            0.0196088720113039,
+            0.034548189491033554,
+            -0.007053316105157137,
+            0.0014990337658673525,
+            -0.022547753527760506,
+            0.013192313723266125,
+            -0.006730855442583561,
+            0.02351105399429798,
+            -0.005432849284261465,
+            -0.016898570582270622,
+            -0.013976015150547028,
+            -0.009739127941429615,
+            0.00839214026927948,
+            -0.022433463484048843,
+            -0.011110605672001839,
+            -0.00922482367604971,
+            -0.002573562553152442,
+            -0.018694553524255753,
+            0.004865481983870268,
+            -0.002426618244498968,
+            -0.012506574392318726,
+            -0.008816645480692387,
+            0.01048201136291027,
+            -0.009347276762127876,
+            0.003077662317082286,
+            -0.0011929002357646823,
+            -0.011453475803136826,
+            -0.02369065210223198,
+            0.0158944521099329,
+            0.0001974561164388433,
+            -0.010359558276832104,
+            0.008408467285335064,
+            -0.0018276170594617724,
+            0.001211268245242536,
+            0.016963878646492958,
+            -0.0020664012990891933,
+            0.003467472270131111,
+            0.023478399962186813,
+            0.004057289566844702,
+            -0.010971825569868088,
+            -0.00025906547671183944,
+            0.013535182923078537,
+            -0.005118552595376968,
+            -0.010008525103330612,
+            1.6646010408294387e-05,
+            -0.02984597720205784,
+            -0.008351322263479233,
+            0.006094098091125488,
+            0.018629245460033417,
+            -0.010620792396366596,
+            -0.015853634104132652,
+            0.026319319382309914,
+            -0.006359413731843233,
+            0.019070077687501907,
+            0.002889900468289852,
+            0.0025694805663079023,
+            0.01724144071340561,
+            0.013461710885167122,
+            0.00786150898784399,
+            -0.013927034102380276,
+            -0.02429475635290146,
+            0.0016398552106693387,
+            -0.018090449273586273,
+            0.010579974390566349,
+            -0.011028970591723919,
+            0.0032756286673247814,
+            -0.02398454025387764,
+            -0.012237177230417728,
+            0.004934872500598431,
+            -0.020343592390418053,
+            -0.00040588201954960823,
+            -0.0056940834037959576,
+            0.022237539291381836,
+            -0.005310396198183298,
+            0.008669701404869556,
+            0.0018255761824548244,
+            -0.012237177230417728,
+            0.0009877908742055297,
+            0.009469729848206043,
+            -0.029192892834544182,
+            -0.0007127808639779687,
+            -0.003138889092952013,
+            0.0011418780777603388,
+            0.005351213738322258,
+            0.009755454957485199,
+            0.007322713732719421,
+            0.0027062203735113144,
+            0.007253323681652546,
+            0.010449357330799103,
+            -0.00022041612828616053,
+            0.0197884701192379,
+            0.018172085285186768,
+            -0.007351286243647337,
+            0.008710519410669804,
+            0.024817224591970444,
+            -0.013159659691154957,
+            -0.008718682453036308,
+            0.023266147822141647,
+            -0.006461458280682564,
+            -0.0011122850701212883,
+            -0.01483318954706192,
+            0.00815947912633419,
+            0.0017367975087836385,
+            0.028131630271673203,
+            0.019657853990793228,
+            0.005820618476718664,
+            0.000639819074422121,
+            0.015788326039910316,
+            -0.013020878657698631,
+            0.0035286990460008383,
+            0.015461783856153488,
+            -0.003940958995372057,
+            -0.019821124151349068,
+            0.013878053054213524,
+            0.009502384811639786,
+            -0.003228688146919012,
+            0.01551892887800932,
+            0.011412657797336578,
+            -0.030841931700706482,
+            -0.0011898389784619212,
+            0.010498338378965855,
+            -0.0021449755877256393,
+            -0.008751336485147476,
+            -0.00014273475972004235,
+            -0.012384121306240559,
+            0.021388528868556023,
+            -0.004408322740346193,
+            0.012277995236217976,
+            -0.00815947912633419,
+            -0.017127150669693947,
+            -0.011559601873159409,
+            0.007914572022855282,
+            -0.010547320358455181,
+            -0.0006714528426527977,
+            -0.004022594541311264,
+            -0.005779800936579704,
+            0.017747581005096436,
+            0.004706292413175106,
+            -0.0008964610169641674,
+            -0.0018398624379187822,
+            0.007983962073922157,
+            -0.006979844067245722,
+            -0.008726846426725388,
+            0.004816500470042229,
+            -0.012833117507398129,
+            0.003981776535511017,
+            0.005543057806789875,
+            0.01506176870316267,
+            -0.014857680536806583,
+            -0.021159948781132698,
+            0.007020662073045969,
+            0.02695607766509056,
+            -0.02516009286046028,
+            -0.005232842173427343,
+            0.006245123688131571,
+            0.002093953313305974,
+            0.006914535537362099,
+            -0.006934944540262222,
+            -0.007208424154669046,
+            -0.03657275065779686,
+            -0.0020745648071169853,
+            0.0009005427709780633,
+            0.008963589556515217,
+            -0.008049271069467068,
+            -0.0136494729667902,
+            -0.011135096661746502,
+            0.002410291228443384,
+            0.007134951651096344,
+            -0.02204161323606968,
+            0.0049593630246818066,
+            0.009608510881662369,
+            -0.019559891894459724,
+            0.025127438828349113,
+            -0.010890189558267593,
+            -6.964537897147238e-05,
+            -0.00995954405516386,
+            0.021633435040712357,
+            -0.013486201874911785,
+            -0.018368011340498924,
+            -0.0006898208521306515,
+            0.008355404250323772,
+            0.011926962062716484,
+            -0.0017663904000073671,
+            -0.011371839791536331,
+            0.0003752686607185751,
+            0.01646590232849121,
+            0.019135385751724243,
+            0.002744997153058648,
+            -0.0024837632663547993,
+            -0.01019628718495369,
+            -0.009592183865606785,
+            -0.0073961857706308365,
+            -0.0007449248805642128,
+            0.002091912319883704,
+            0.02035992033779621,
+            0.013469874858856201,
+            0.0024837632663547993,
+            -0.015135240741074085,
+            -0.005257333163172007,
+            0.008939098566770554,
+            0.009412584826350212,
+            -0.011690218932926655,
+            -0.00898808054625988,
+            0.002659279853105545,
+            0.02249877341091633,
+            0.017633290961384773,
+            -0.0014000505907461047,
+            0.0032878739293664694,
+            -0.0016592437168583274,
+            0.012506574392318726,
+            -0.007555375341325998,
+            -0.010269759222865105,
+            -0.006943108048290014,
+            -0.014645427465438843,
+            -0.010800390504300594,
+            -0.0023062059190124273,
+            -0.010179960168898106,
+            0.022629389539361,
+            0.007988044060766697,
+            0.009779945947229862,
+            0.008265605196356773,
+            0.010269759222865105,
+            0.002087830565869808,
+            0.025715215131640434,
+            -0.004424649756401777,
+            -0.011984107084572315,
+            0.007236996199935675,
+            -0.022368155419826508,
+            -0.021535472944378853,
+            0.008257441222667694,
+            0.006367577239871025,
+            0.03595232218503952,
+            -0.026890767738223076,
+            0.0052981507033109665,
+            -0.009779945947229862,
+            -0.010400376282632351,
+            0.007996207103133202,
+            -0.0015439334092661738,
+            -0.004245051648467779,
+            0.022368155419826508,
+            0.00726556871086359,
+            0.0035797213204205036,
+            0.0032817514147609472,
+            0.019837452098727226,
+            -0.005685919895768166,
+            0.01573934406042099,
+            0.0259274672716856,
+            0.024621298536658287,
+            0.009388094767928123,
+            -0.009469729848206043,
+            0.02785406820476055,
+            0.0002895512734539807,
+            -0.0034327772445976734,
+            0.007220669183880091,
+            -0.00305113079957664,
+            -0.0009556467994116247,
+            -0.010033016093075275,
+            0.021764053031802177,
+            -0.001455154619179666,
+            0.004359341226518154,
+            0.02081707864999771,
+            -7.774515688652173e-05,
+            0.006220633164048195,
+            0.01057181041687727,
+            0.002181711606681347,
+            0.003396041225641966,
+            0.010775899514555931,
+            -0.039315707981586456,
+            -0.0006571666453965008,
+            -0.025600925087928772,
+            -0.0018286375561729074,
+            0.0007357408758252859,
+            -0.01266984548419714,
+            0.013600491918623447,
+            -0.002015379024669528,
+            -0.010751409456133842,
+            -0.00029337796149775386,
+            0.021649762988090515,
+            0.005416522268205881,
+            0.004071575589478016,
+            -0.004469549283385277,
+            0.018792515620589256,
+            -0.006955353543162346,
+            -0.017861871048808098,
+            -0.012759645469486713,
+            -0.012367794290184975,
+            -0.01918436773121357,
+            -0.01214737817645073,
+            -0.0024551907554268837,
+            -0.002667443361133337,
+            0.004738946910947561,
+            0.002481722505763173,
+            -0.003228688146919012,
+            -0.02070278860628605,
+            0.015037278644740582,
+            -0.01430255826562643,
+            0.012873934581875801,
+            -0.011053460650146008,
+            0.005963481031358242,
+            -0.004122598096728325,
+            0.012726990506052971,
+            -0.021404854953289032,
+            0.02070278860628605,
+            -0.01536382082849741,
+            0.0038532004691660404,
+            0.019592545926570892,
+            0.006551257334649563,
+            -0.008118661120533943,
+            -0.013404566794633865,
+            -0.022074267268180847,
+            0.021633435040712357,
+            -0.011633073911070824,
+            -0.018237393349409103,
+            -0.004608329851180315,
+            0.009649328887462616,
+            -0.017470018938183784,
+            -0.007367613259702921,
+            0.01707816869020462,
+            -0.0018949664663523436,
+            0.012980061583220959,
+            0.023004913702607155,
+            -0.0052981507033109665,
+            0.0036572751123458147,
+            -0.009167678654193878,
+            0.0010286086471751332,
+            -0.02470293454825878,
+            0.013527019880712032,
+            0.008971752598881721,
+            0.006269614677876234,
+            -0.013935198076069355,
+            -0.013306603766977787,
+            0.0036531933583319187,
+            0.004498121794313192,
+            0.011437147855758667,
+            0.009420748800039291,
+            0.004220560658723116,
+            -0.013918871060013771,
+            -0.027788760140538216,
+            -0.027070365846157074,
+            0.03595232218503952,
+            -0.0005002731923013926,
+            -0.020947696641087532,
+            0.03399306535720825,
+            0.016114868223667145,
+            0.0022980421781539917,
+            -0.012449429370462894,
+            -0.004314441699534655,
+            0.024621298536658287,
+            0.016400592401623726,
+            0.010465684346854687,
+            -0.0004742518358398229,
+            0.005571629852056503,
+            0.0016469983384013176,
+            0.03012353740632534,
+            0.015323002822697163,
+            0.0009459525463171303,
+            -0.038597315549850464,
+            0.011943289078772068,
+            -0.007547211833298206,
+            0.0013408648082986474,
+            0.006702283397316933,
+            0.017780235037207603,
+            0.011935125105082989,
+            0.015567909926176071,
+            0.014865843579173088,
+            -0.003065417055040598,
+            0.0029511270113289356,
+            0.0006163488142192364,
+            -0.009624837897717953,
+            -0.001767410896718502,
+            -0.008881953544914722,
+            -0.005714492406696081,
+            -0.002920513739809394,
+            0.005592038854956627,
+            0.007657419890165329,
+            -0.003983817528933287,
+            0.028049994260072708,
+            -0.0006433906382881105,
+            -0.017600636929273605,
+            -0.009061552584171295,
+            -0.03611559048295021,
+            0.00434301421046257,
+            -0.030939893797039986,
+            -0.011257549747824669,
+            0.0033674687147140503,
+            -0.020457882434129715,
+            -0.024768242612481117,
+            0.0010306495241820812,
+            0.014335212297737598,
+            0.007392104249447584,
+            0.0033327736891806126,
+            -0.01004934310913086,
+            0.003816464450210333,
+            -0.001855169073678553,
+            -0.006461458280682564,
+            0.028833696618676186,
+            0.019559891894459724,
+            0.020457882434129715,
+            -0.01190247107297182,
+            0.019298657774925232,
+            0.005845109466463327,
+            0.02035992033779621,
+            0.010465684346854687,
+            -0.0048695639707148075,
+            0.0074859848245978355,
+            0.009567692875862122,
+            -0.006155324634164572,
+            -0.021519144997000694,
+            0.013192313723266125,
+            -0.0010837126756086946,
+            -0.0052083516493439674,
+            -0.016269976273179054,
+            -0.003534821793437004,
+            -0.015812816098332405,
+            0.010547320358455181,
+            -0.017159804701805115,
+            0.012498411349952221,
+            -0.000802069844212383,
+            0.00168475485406816,
+            -0.015469946898519993,
+            -0.013951525092124939,
+            -0.0012439225101843476,
+            -0.01244126632809639,
+            -0.008971752598881721,
+            0.006163488142192364,
+            -0.007539048325270414,
+            -0.008318668231368065,
+            -0.00535937724635005,
+            -0.011437147855758667,
+            -0.010882026515901089,
+            0.004367504734545946,
+            -0.0010653446661308408,
+            0.020931368693709373,
+            0.00011671340325847268,
+            -0.005824700463563204,
+            0.01190247107297182,
+            0.02787039615213871,
+            0.0016704685986042023,
+            0.004583839327096939,
+            -0.014261740259826183,
+            -0.011118769645690918,
+            -0.02279266156256199,
+            0.016572028398513794,
+            0.0049675265327095985,
+            -0.0344502255320549,
+            0.01438419334590435,
+            0.00824927818030119,
+            0.0015102586476132274,
+            -0.006698201410472393,
+            -0.005334886722266674,
+            -0.005608365871012211,
+            0.006306350696831942,
+            -0.0022837561555206776,
+            0.0005107327597215772,
+            0.0035919665824621916,
+            0.034678805619478226,
+            -0.0040674940682947636,
+            -0.007228832691907883,
+            0.00981259997934103,
+            0.0005130287609063089,
+            0.020196648314595222,
+            -0.005518566817045212,
+            0.0037572786677628756,
+            -0.019282329827547073,
+            -0.016563864424824715,
+            0.0018429238116368651,
+            -0.01722511276602745,
+            0.023282473906874657,
+            -0.005392031744122505,
+            -0.01347803883254528,
+            -0.006906372029334307,
+            -0.020115012302994728,
+            0.012637191452085972,
+            0.02310287579894066,
+            0.007016580086201429,
+            0.01033506728708744,
+            0.012131051160395145,
+            -0.02444170042872429,
+            -0.020457882434129715,
+            0.012171869166195393,
+            0.0013571919407695532,
+            -0.008449285291135311,
+            -0.005404277238994837,
+            -0.001783737912774086,
+            0.007783954963088036,
+            -0.0032633834052830935,
+            0.007881917990744114,
+            0.06491663306951523,
+            -0.01093100756406784,
+            -0.002987863030284643,
+            -0.02966637909412384,
+            -0.004396077245473862,
+            -0.006628811359405518,
+            0.0056287748739123344,
+            0.009151351638138294,
+            -0.023135529831051826,
+            0.009232986718416214,
+            -0.030025575309991837,
+            -0.010628955438733101,
+            0.00824111420661211,
+            0.0028490826953202486,
+            -0.01220452319830656,
+            -0.0009592183632776141,
+            -0.013665799982845783,
+            0.0006107363733462989,
+            -0.009412584826350212,
+            -0.00576347392052412,
+            0.012718827463686466,
+            -0.02321716584265232,
+            0.006298186723142862,
+            0.007592111360281706,
+            -0.006514521315693855,
+            -0.013722945004701614,
+            -0.017861871048808098,
+            -0.011967780068516731,
+            -0.013314766809344292,
+            0.002996026771143079,
+            0.01190247107297182,
+            -0.02607441134750843,
+            0.012922916561365128,
+            -0.0037389106582850218,
+            0.00945340283215046,
+            0.004404240753501654,
+            0.013469874858856201,
+            0.018074123188853264,
+            0.009355440735816956,
+            0.0021429345943033695,
+            -0.008024780079722404,
+            0.012702500447630882,
+            -0.01993541419506073,
+            -0.01635161228477955,
+            0.013837235048413277,
+            0.012016761116683483,
+            0.003061335301026702,
+            0.009828926995396614,
+            0.0020959940738976,
+            0.011886144056916237,
+            0.010775899514555931,
+            -0.019739490002393723,
+            -0.0006648199632763863,
+            0.0007326795603148639,
+            0.03280118480324745,
+            -0.0005938990507274866,
+            0.008857463486492634,
+            0.004420568235218525,
+            0.013159659691154957,
+            -0.012784135527908802,
+            -0.01769859902560711,
+            0.0008087027235887945,
+            -0.01057181041687727,
+            0.021257910877466202,
+            -0.008192133158445358,
+            0.006216551177203655,
+            -0.018612917512655258,
+            -0.027086693793535233,
+            0.006392067763954401,
+            -0.016261812299489975,
+            -0.018221067264676094,
+            0.01317598670721054,
+            0.0077268099412322044,
+            0.0017572063952684402,
+            -0.005098143592476845,
+            -0.020457882434129715,
+            -0.009837090037763119,
+            0.028441844508051872,
+            0.011102442629635334,
+            -0.004120557103306055,
+            0.009584019891917706,
+            -0.009094206616282463,
+            -0.012490247376263142,
+            -0.0017541450215503573,
+            0.0036715613678097725,
+            0.008261523209512234,
+            0.005645101889967918,
+            -0.01386172603815794,
+            -0.00029873527819290757,
+            -0.012400448322296143,
+            -0.021225256845355034,
+            -0.008906444534659386,
+            0.016751626506447792,
+            0.007820690982043743,
+            -0.0038960592355579138,
+            0.015812816098332405,
+            -0.004404240753501654,
+            0.002189875114709139,
+            0.014294394291937351,
+            0.005334886722266674,
+            -0.004285869188606739,
+            -0.0095758568495512,
+            -0.004612411838024855,
+            0.004043003544211388,
+            -0.0025960123166441917,
+            0.006457376293838024,
+            0.017192458733916283,
+            0.0072900597006082535,
+            -0.013755599036812782,
+            0.013135168701410294,
+            0.0007791098323650658,
+            0.0049675265327095985,
+            -0.03699725493788719,
+            -0.003247056156396866,
+            0.015796490013599396,
+            -0.013020878657698631,
+            0.02501314878463745,
+            0.005400195252150297,
+            0.022743679583072662,
+            0.015233203768730164,
+            0.008669701404869556,
+            -0.001542912912555039,
+            0.00464098434895277,
+            0.02759283408522606,
+            -0.003151134354993701,
+            -0.008637047372758389,
+            -0.0020602785516530275,
+            0.009241150692105293,
+            0.0002063850115519017,
+            0.003000108525156975,
+            0.008281932212412357,
+            -0.03562577813863754,
+            0.011706545948982239,
+            -0.019576217979192734,
+            0.0024694770108908415,
+            0.003459308762103319,
+            -0.007183933164924383,
+            -0.014865843579173088,
+            -0.005857354495674372,
+            0.022841641679406166,
+            -0.003224606392905116,
+            0.021078312769532204,
+            0.002885818714275956,
+            0.016106704249978065,
+            -0.011575928889214993,
+            -0.009208496659994125,
+            -0.0004130251472815871,
+            -0.005351213738322258,
+            -0.007539048325270414,
+            0.010596301406621933,
+            0.0034409407526254654,
+            0.005534893833100796,
+            0.002265388146042824,
+            0.0036654386203736067,
+            -0.0036878883838653564,
+            0.02773977816104889,
+            0.003563394071534276,
+            0.007996207103133202,
+            -0.011567764915525913,
+            0.014082142151892185,
+            -0.017029186710715294,
+            0.0070778070949018,
+            0.005575711838901043,
+            -0.026156047359108925,
+            0.016849588602781296,
+            0.004775682929903269,
+            0.006628811359405518,
+            0.013812744058668613,
+            0.01618834026157856,
+            -0.010408539324998856,
+            -0.014816862531006336,
+            0.024049850180745125,
+            -0.0032041973900049925,
+            -0.015225040726363659,
+            -0.006775755435228348,
+            4.438935866346583e-05,
+            0.006559420842677355,
+            0.00373074715025723,
+            0.0013020879123359919,
+            -0.0006204306264407933,
+            -0.010743245482444763,
+            0.004645065870136023,
+            -0.01934763789176941,
+            -0.0007163524278439581,
+            -0.016449574381113052,
+            -0.015633217990398407,
+            0.018678225576877594,
+            -0.01830270327627659,
+            0.009633000940084457,
+            0.0012275953777134418,
+            0.028588788583874702,
+            -0.011706545948982239,
+            -0.012155541218817234,
+            0.0015102586476132274,
+            0.010849371552467346,
+            0.0067512644454836845,
+            -0.010033016093075275,
+            -0.007583947852253914,
+            -0.014180104248225689,
+            -0.010163633152842522,
+            0.015502601861953735,
+            -0.014498483389616013,
+            -0.014343376271426678,
+            0.008914608508348465
+          ],
+          "embedding": [
+            -0.006359657085039302,
+            -0.024594786241399234,
+            -0.02553538924848095,
+            0.005360188465088291,
+            0.012801870977589839,
+            0.021680505862506104,
+            -0.016668048966001273,
+            0.06860852474581301,
+            -0.007122272841157724,
+            0.03471775000705488,
+            -0.009004297848861675,
+            0.01059974144678391,
+            0.02565604599426148,
+            -0.022544127646041287,
+            0.010616674976540785,
+            0.013658194823359737,
+            0.016400329743363844,
+            0.02858546964917478,
+            0.0067612676434693686,
+            -0.04174737193606879,
+            0.03184155110488616,
+            -0.02223105127585514,
+            -0.05506239667974977,
+            -0.0015307551410555752,
+            -0.013169788751379287,
+            0.017681749722384007,
+            0.01682706317229353,
+            0.019652295823816928,
+            -0.05623980369271269,
+            0.030698582851533197,
+            0.02286327124296613,
+            0.016019032539689178,
+            0.005508113353872315,
+            8.58985091215149e-05,
+            -0.019206292853495906,
+            -0.018439290483495144,
+            0.03475023176950151,
+            0.01018612942141759,
+            -0.0252551158857426,
+            0.0334562130666827,
+            0.020462153537485785,
+            0.02260508249714211,
+            -0.03323524989392951,
+            -0.002094251009530707,
+            0.015177442347686773,
+            -0.00018698464350038317,
+            -0.0006788648344273403,
+            0.0007777616893581039,
+            0.013352398615059647,
+            -0.030751111528744813,
+            -0.0317733818445192,
+            -0.006729064081710058,
+            0.009413204899986514,
+            -0.005712894407943592,
+            -0.038961826880527066,
+            -0.002827692206530345,
+            0.017863300537973137,
+            0.07918455388563496,
+            0.048967158919816366,
+            0.018002566325350088,
+            0.040264962620950695,
+            -0.019715100359659787,
+            -0.02211350500879408,
+            0.004918148608475574,
+            0.01062585305866606,
+            -0.014478993521696027,
+            0.0034983131226364672,
+            0.000766228629514787,
+            0.009667268771502067,
+            -0.01558439890697323,
+            -0.04984718709870083,
+            0.029613955432821887,
+            -0.009686854114630624,
+            -0.015279319238124568,
+            0.007460629676325322,
+            -0.006235979350809034,
+            0.012514387473717085,
+            0.02800435263741497,
+            -0.008688925946672778,
+            -0.01236301543374469,
+            0.029755705020188972,
+            0.01158037219446542,
+            -0.005242392206024511,
+            0.009982679027728814,
+            0.052414361714483264,
+            -0.018788500381726227,
+            -0.020335831196027204,
+            -0.08583959943356621,
+            0.029309228216812346,
+            -0.0249285282010673,
+            -0.018929396555426307,
+            0.007710663193334574,
+            0.03096798035958051,
+            -0.017089977064591828,
+            -0.028015107605405817,
+            0.06740703271128606,
+            0.009398795412354399,
+            0.01118769022826835,
+            -0.03041417460855722,
+            0.014776002883150835,
+            0.004319194067673161,
+            0.022507636639041203,
+            0.0053562788646786845,
+            0.007548072258943124,
+            0.011123374308563128,
+            -0.006924362115585165,
+            -0.011118507147092024,
+            0.009563232643295304,
+            0.00657178473678069,
+            0.08313027447380966,
+            -0.004239370156003264,
+            -0.00930406163946907,
+            -0.039202672821898946,
+            0.009761029930556298,
+            -0.02827698465424993,
+            -0.037914533246900244,
+            -0.04496050911855815,
+            -0.017711307900236353,
+            0.01934435542287117,
+            0.004335173027873153,
+            0.03800509969714242,
+            0.031198127357716314,
+            0.031311207941680276,
+            -0.012184553894588813,
+            -0.058985863670856876,
+            0.041442017057707635,
+            -0.012456189292721711,
+            -0.0040966351720394135,
+            0.010849179519959421,
+            0.005243123049347636,
+            0.0011495799963862647,
+            0.003534183676934487,
+            -0.03139340512003631,
+            -0.023876261883905306,
+            0.008382957569752911,
+            -0.004778283217011976,
+            0.03703067589888386,
+            0.06888630688839406,
+            -0.03993532296478411,
+            -0.009472167678222372,
+            0.025246872942105057,
+            -0.01524567912700368,
+            0.01595917939878582,
+            -0.003736723164442467,
+            -0.030836797784967753,
+            0.0026246446163368533,
+            0.008894300082148884,
+            0.08342856279154155,
+            -0.0007476580521704578,
+            0.025149116238550588,
+            0.004972542633631468,
+            0.034495632546556576,
+            -0.009173329764686424,
+            0.014009381543772768,
+            0.004427261668151028,
+            -0.013426138607902904,
+            -0.022234568880321808,
+            0.01976282364689966,
+            -0.03592433789490043,
+            -0.010918278220831188,
+            0.013527180289641174,
+            -0.013023113052242677,
+            -0.008304770302495363,
+            0.003677063799167178,
+            0.0013981822663219421,
+            0.047001163242600175,
+            0.01828332714887986,
+            0.002483375662343689,
+            -0.06820498187523893,
+            0.05295312038995555,
+            0.025038753504342162,
+            0.025286870198436735,
+            -0.0029544808216316115,
+            0.029982752196635428,
+            0.0005600481523874767,
+            0.0128290258663539,
+            0.009023805227424919,
+            -0.0019219166367750596,
+            -0.04482843936478531,
+            0.04523479691409549,
+            -0.004678574027241186,
+            0.010486850804218904,
+            0.006736229865018014,
+            0.02134813656476443,
+            0.028083972673690036,
+            0.016857413440125445,
+            0.0189082447820146,
+            0.03843690613820022,
+            -0.011922027011229613,
+            0.016193433524453224,
+            0.03828207911467065,
+            -0.0013055128404331705,
+            0.043198163443700956,
+            0.024025562893260542,
+            -0.001257851946749882,
+            -0.011726713433348844,
+            -0.021907984099443396,
+            0.02752198420597673,
+            0.004431866336999851,
+            0.015798806714413912,
+            -0.012910782191382367,
+            -0.03180549180206364,
+            0.004550247372715741,
+            -0.022778272841118514,
+            0.031242218240017532,
+            -0.006106984949693483,
+            -0.006259993190497421,
+            0.02187449047033339,
+            0.009334146847221406,
+            -0.009381278960016145,
+            -0.020888853823931134,
+            -0.007502374908776156,
+            -0.010390018386728482,
+            -0.04316693435221676,
+            -0.006274513730720055,
+            0.019152443937064258,
+            -0.01668984824845791,
+            0.018573194036505578,
+            0.0035035262183249925,
+            -0.019779911381789077,
+            -0.0027495798165085993,
+            0.02425481236315913,
+            0.006857723545270935,
+            -0.014733535699087731,
+            -0.026326146471941727,
+            -0.019906660231056833,
+            -0.001033294559612807,
+            0.0159154063026895,
+            0.035649534442431415,
+            -0.026210592096240003,
+            0.014397127229517285,
+            0.005114311681443673,
+            -0.0033104236005207077,
+            0.00324988235167305,
+            0.03764004154553922,
+            -0.017822437760737605,
+            -0.008579701063708334,
+            -0.00032410984200965487,
+            0.018450351071676443,
+            0.017283379764846195,
+            0.006384411611332382,
+            -0.005115090309123598,
+            -0.030590047062583125,
+            -0.011776211359193503,
+            0.01183623914060493,
+            0.0016564847796208331,
+            -0.009530766634361575,
+            -0.004810598258154945,
+            0.03876242515526699,
+            0.027076785774401667,
+            0.005324763136165989,
+            -0.03416412480913939,
+            -0.007943016733586697,
+            0.002114307650456928,
+            -0.02961449347300628,
+            0.05142732770518828,
+            0.05566026433842563,
+            -0.0030729309664974704,
+            -0.00938186948393074,
+            -0.04547908981236149,
+            0.0651216359692231,
+            -0.01813722445944094,
+            -0.0482650853560993,
+            0.044045155892545554,
+            -0.014272653003441165,
+            0.013740652306746851,
+            0.04563732439184395,
+            0.02126513340521316,
+            -0.0054845632439384905,
+            -0.011635625101262898,
+            0.030992082069685376,
+            -0.010051935721068779,
+            -0.050967258971854425,
+            -0.009827942095033494,
+            0.013373878556654464,
+            -0.013017627321534534,
+            -0.009361001542607812,
+            -0.01925867665425919,
+            0.020329052915436022,
+            0.023020741670660188,
+            0.019387043947695172,
+            0.005628632318152202,
+            0.018451555161952665,
+            0.016226845400848455,
+            -0.01384033112836216,
+            0.004762589697426251,
+            -0.012718476561725138,
+            -0.024222765127494463,
+            -0.037487144314010595,
+            -0.019971524166659278,
+            -0.015034588610516958,
+            0.009483936496215338,
+            -0.014301924283553591,
+            -0.0024139665139412125,
+            0.013361045353529458,
+            -0.0005870798721107379,
+            -0.011192319340879789,
+            0.013927695434391159,
+            0.021337539669342293,
+            -0.027300952495815437,
+            -0.04765670695199488,
+            -0.00561706224251562,
+            0.0082776639906139,
+            -0.0030984973749633724,
+            -0.026269016057688548,
+            0.007040524859505573,
+            0.0017859008072463442,
+            0.020091542517503808,
+            -0.018716761130867725,
+            0.0035283051960672963,
+            0.008519383548064053,
+            -0.015247417050336866,
+            -0.016917626868166634,
+            -0.018580550690103854,
+            -0.024697180580294884,
+            0.034202028316737125,
+            0.0204395003463227,
+            0.030392895534267717,
+            -0.04527669147753553,
+            -0.018359428092075348,
+            0.05494107499558624,
+            0.015234754907198422,
+            -0.05066008106884831,
+            0.031139824660661085,
+            0.005016615710206006,
+            0.0019000453624929544,
+            -0.006094364589229031,
+            0.0637324238017742,
+            -0.02711381489292883,
+            -0.02187418906973867,
+            0.004601744864532875,
+            -0.04141121277496072,
+            -0.026689585608019588,
+            -0.005568100750837504,
+            0.01291019036002459,
+            -0.031853041426355644,
+            -0.02815800259372657,
+            -0.007964861641790374,
+            -0.010933515363060793,
+            0.013726639077745378,
+            -0.023956800669647334,
+            -0.015672058590455337,
+            -0.023642025374517057,
+            -0.01288435863544158,
+            -0.0042516881552192385,
+            -0.011162214160460695,
+            -0.02412733256710621,
+            0.010088528028211243,
+            0.03310175458095482,
+            0.023064360903748363,
+            0.001996159684818467,
+            0.009478229572933101,
+            0.014478772877286076,
+            0.00371783700962415,
+            0.007864103610320116,
+            -0.048511515124609,
+            -0.031398754206534776,
+            0.020227872561659397,
+            0.012159811587094092,
+            0.0172163448580163,
+            0.009578952796095119,
+            -0.04154545961112031,
+            -0.05095080393682759,
+            0.02968486144855133,
+            -0.03932395023720674,
+            -0.02063219739061335,
+            0.007512423249865194,
+            -0.011911709468952548,
+            -0.00810951734088105,
+            -0.0008079327452253181,
+            -0.01600701450676789,
+            0.023984909975645632,
+            -0.02418072541081482,
+            0.0033984265728439398,
+            -0.0104343782811971,
+            0.01588977364765743,
+            0.05109529246167992,
+            -0.027800269136366997,
+            -0.027889034481533438,
+            0.03706813210665186,
+            -0.02058120652429308,
+            0.030801396468451574,
+            0.03246127847621613,
+            0.008355517405358826,
+            -0.008605467586247932,
+            0.017693344563968573,
+            0.014499487024471321,
+            0.012431277080478217,
+            -0.040103870071440256,
+            0.017328848567062113,
+            0.02723900253727976,
+            -0.054909074068976875,
+            -0.008547596270650966,
+            -0.00777964838164648,
+            0.027134666572242665,
+            0.02765036594970255,
+            -0.002207173084941375,
+            -0.016838740117653725,
+            -0.028289233739722936,
+            -0.019433376533829132,
+            -0.011053380458946764,
+            0.03247641283550457,
+            0.0007514989009289282,
+            0.007919574219000797,
+            0.001316042436488591,
+            -0.0017934342041762042,
+            0.01924145734197619,
+            0.015260199553572459,
+            0.048259602293134723,
+            -0.03383050622747848,
+            0.0021712225589426266,
+            -0.03508567715696004,
+            -0.04578731401250471,
+            -0.008768228935292693,
+            0.010203318819548585,
+            0.024135236371743855,
+            -0.007467140460133838,
+            0.0028907962385387818,
+            -0.007582083854422456,
+            -0.005384477718912037,
+            -0.01867978482547249,
+            0.01641191108381631,
+            0.037168089880543244,
+            0.0031992495663980464,
+            -0.008533890994695162,
+            0.000778843942291118,
+            -0.004620258538699103,
+            0.010572690335163528,
+            0.033233014627100904,
+            -0.030497362482602697,
+            0.02974571152880676,
+            -0.004274442492343003,
+            0.025341347125910928,
+            -0.04307748954707389,
+            -0.018069609916024403,
+            -0.018491122727274337,
+            -0.02069307802429807,
+            0.05372967226264594,
+            -0.030768978301976772,
+            -0.03662045886993282,
+            -0.003205991669059688,
+            0.016815511161004534,
+            -0.0014152644009276566,
+            -0.0016015439520090104,
+            -0.03984716970293867,
+            -0.010828260334550217,
+            -0.05894985435251326,
+            -0.01638035185618134,
+            -0.021819442096404948,
+            -0.005315102328745982,
+            -0.06251579337157112,
+            0.0033713876515840233,
+            -0.07815974854883057,
+            0.001484860071649606,
+            0.04539427129131382,
+            -0.023530943197353476,
+            0.05142250021536681,
+            -0.024746600229063383,
+            -0.0005885740455257141,
+            -0.044698536284421976,
+            0.0026481001040031785,
+            0.02389178603305878,
+            0.051387657872255674,
+            -0.002949198236157656,
+            0.010674377196609455,
+            0.013322075519401089,
+            -0.026245090721950647,
+            -0.01036724558050635,
+            -0.005321884180696542,
+            0.005958798671856245,
+            0.013755178085972732,
+            -0.0074734329537091675,
+            0.038368016046563674,
+            0.010941421833174982,
+            0.03443115781919206,
+            0.034623617136788874,
+            -0.004361926531008214,
+            -0.03348681614127998,
+            -0.00632870804659212,
+            0.024433331840566692,
+            0.0321998807702602,
+            -0.02899983280320611,
+            0.005958471722097962,
+            -0.015602447981516634,
+            0.015633217539747047,
+            0.006321931116948646,
+            -0.011827541320878442,
+            0.02372204564999124,
+            0.015673426811623124,
+            0.006103300725621424,
+            -0.005165006670545588,
+            0.013589526056189892,
+            -0.025411864292353578,
+            0.03041791947375232,
+            -0.004456865298584878,
+            -0.04276058054583262,
+            -0.0024736078733106825,
+            -0.020059309867945227,
+            0.024883721794000956,
+            0.015578237413840115,
+            -0.00536523508503184,
+            -0.005290645798641171,
+            0.008242987948511412,
+            0.020469675635821673,
+            0.006668354647245447,
+            0.00017729517736814364,
+            0.02373604785121342,
+            0.029915687122900485,
+            -0.02432635861789771,
+            -0.029540533943782886,
+            0.008151297444985344,
+            -0.005730733861939129,
+            0.011318816579801282,
+            -0.02644855229221747,
+            0.00033932872948409137,
+            0.0035505684154028996,
+            -0.03570104343808711,
+            0.03408366016010711,
+            -0.009199700348630527,
+            0.005857598530788056,
+            -0.009538177736472163,
+            -0.0017768806148210118,
+            0.0027410153099351694,
+            0.03898108970435115,
+            -0.007367623392751247,
+            0.009827254226889936,
+            0.015465895035342481,
+            0.000981604396724216,
+            0.05143083558797524,
+            -0.027350877485564314,
+            -0.02183036962301852,
+            0.01885468378168636,
+            -0.007285239007489837,
+            -0.016846837665812908,
+            -0.020546383007536074,
+            -0.011863943814684079,
+            -0.016293192750336218,
+            0.01976516340568883,
+            -0.02387328798681798,
+            -0.011892336568452852,
+            0.007170651391639504,
+            0.0013725014050721622,
+            -0.009809669068563384,
+            0.0016617183704652932,
+            -0.006307590349798603,
+            0.02037693835615797,
+            0.031753656615094446,
+            -0.024892675105483313,
+            0.03577919637821846,
+            0.03178303188315581,
+            0.017260898395251155,
+            -0.02102289690701718,
+            -0.009661035772073294,
+            -0.00048542689442823136,
+            -0.014694631967051945,
+            -0.0326379186158631,
+            0.025443206465204822,
+            0.017194626456982255,
+            -0.011381421003142453,
+            -0.03680783761649673,
+            0.0017216115811536721,
+            0.022819859205829785,
+            -0.013600989058799166,
+            0.028141657788496834,
+            -0.01869047298692401,
+            -0.01366833755869951,
+            -0.00782959271979894,
+            0.031227914441058244,
+            -0.05252392878125891,
+            0.016957842026348195,
+            0.00020880696574616416,
+            0.001154370575234767,
+            0.010892067708194324,
+            -0.008368819756628046,
+            0.012224564683325483,
+            0.031700389750709,
+            0.018063405869147564,
+            0.0010794115307202266,
+            -0.002783668216641561,
+            0.01418234057248712,
+            0.017773269840982066,
+            -0.015653443355569053,
+            -0.000497740485657824,
+            0.019890882604100452,
+            0.022672054781919238,
+            -0.00521797370595221,
+            -0.016686942830774276,
+            -0.018443538351123584,
+            0.025270057639201107,
+            -0.010354431070738671,
+            0.03484047753830186,
+            0.04594785214094975,
+            0.0036441959528283004,
+            -0.05695476763071743,
+            -0.009412209619190693,
+            0.0010371110229966086,
+            -0.01099343927053564,
+            0.01606863921935641,
+            0.012470660511773616,
+            -0.03237187923799263,
+            0.006388672213167589,
+            0.012946570716567364,
+            0.022152554488829627,
+            -0.02540317621005867,
+            -0.023183252921885226,
+            -0.0030984019855787655,
+            -0.007890467768631811,
+            0.018648760402624687,
+            -0.008653405980661855,
+            0.0006228315253506687,
+            0.009672526897422602,
+            0.0005436494014569835,
+            0.011632326145022935,
+            0.005655942007108228,
+            0.021736827172951,
+            -0.001256608167230203,
+            0.013124413886287987,
+            -0.01986808920858981,
+            -0.0270450339809516,
+            0.006500423802582728,
+            0.04438242119211055,
+            -0.014813399694814117,
+            0.030621709833336064,
+            0.0092171224268983,
+            0.016433384739642534,
+            0.024875072913970443,
+            -0.00891470467241686,
+            -0.014124009014664287,
+            0.00845777541223868,
+            0.009019007848927453,
+            -0.04147358041382793,
+            0.012513030150663913,
+            -0.01710243399313324,
+            -0.033144939719206276,
+            0.01904674924133127,
+            -0.00534122554342279,
+            0.012367320751926314,
+            0.004270315800911569,
+            -0.002886460711688153,
+            -0.025696286516581048,
+            0.014137901708875033,
+            0.028135087538950946,
+            0.021797869775098042,
+            0.0296173052503884,
+            -0.012757078368361444,
+            0.0005460601780756187,
+            -0.008748930817524899,
+            0.005470975782187157,
+            0.04187083128854477,
+            0.0008687182830616019,
+            -0.0008032802520818105,
+            0.01507726552487313,
+            0.024945744517309208,
+            0.0376130540370386,
+            -0.015415560911023553,
+            -0.03407353556457147,
+            0.01564969972264396,
+            -0.010104443707865262,
+            0.018855792875436674,
+            0.005503259932317811,
+            0.0049634097823351596,
+            0.0038600156268000702,
+            0.019312419743287703,
+            -0.006635193789316492,
+            0.018798045806429395,
+            -0.02500983683871332,
+            -0.028785757874583755,
+            0.01829411751138161,
+            -0.005277972443258226,
+            0.03338623299668019,
+            0.016225885095995672,
+            -0.029522280454919586,
+            -0.01712097344863975,
+            0.01553706187924322,
+            -0.008465766956343768,
+            -0.024842700926275538,
+            0.01895228828512998,
+            0.007076862934523977,
+            0.008956105593078965,
+            0.0024373779389833218,
+            -0.01106503696703573,
+            0.01125466547367417,
+            -0.000818223040067914,
+            -0.015322790791698395,
+            0.014157131555941659,
+            -0.017853386731281715,
+            -0.014994640991761465,
+            0.01775991780643474,
+            -0.037498078532902435,
+            0.009570197712419421,
+            -0.007687297543935123,
+            0.009050494195378254,
+            -0.004222041469820445,
+            0.007460383234429904,
+            -0.0010804379186060264,
+            -0.014987857934522812,
+            -0.014903749199810126,
+            0.03228485430019625,
+            0.022096121825475306,
+            -0.000410816936759558,
+            0.006728733918931589,
+            -3.971636461500232e-05,
+            -0.026719848032785824,
+            -0.011253751790407687,
+            0.02245467425830765,
+            -0.005664635254398481,
+            0.005134496095445943,
+            -0.01109590855846966,
+            -0.0149616794609307,
+            0.0019726017255571673,
+            0.032668637270108894,
+            0.06066635618119076,
+            0.012001732730659747,
+            0.024422093585231484,
+            0.0189499396761026,
+            0.02119276033547528,
+            -0.008489818917377516,
+            0.006413901834269739,
+            -0.03097413287789443,
+            0.03509225573986143,
+            0.005392594635384506,
+            0.03108937318306421,
+            0.026664331248813712,
+            -0.04722525921137681,
+            -0.0058539171830594774,
+            0.017087531454837677,
+            0.022673712717885323,
+            0.0159020681868943,
+            0.0008382498574190522,
+            -0.008938174268517353,
+            0.035650940044285984,
+            -0.03315250299348084,
+            -0.023647111263155,
+            0.005059382024092534,
+            -0.01439537467317451,
+            -0.010488359956731476,
+            -0.006785295580877329,
+            0.02776449681317373,
+            -0.005584052419598926,
+            -0.004582019433008779,
+            0.009697064252638535,
+            -0.014400206558190682,
+            0.008165676294736836,
+            0.0007624778531368436,
+            0.008681263897172326,
+            0.039249866981334126,
+            0.0005918063050375113,
+            0.024275699182767528,
+            -0.004138313072843555,
+            0.011079529739650594,
+            -0.0158972266904151,
+            -0.016985058543605,
+            0.02276982363172991,
+            0.00351101120496393,
+            -0.03244126813432563,
+            -0.00768819222024939,
+            0.00928180979884261,
+            -0.0013793404539641824,
+            0.005266058566444064,
+            -0.00503433670609399,
+            0.003747448261046857,
+            0.03286230666764075,
+            0.0022383592721685153,
+            0.008606190020480513,
+            0.013593947677257625,
+            0.00030951915677929204,
+            -0.01490307680740127,
+            0.013254591138113545,
+            -2.1379308136707722e-05,
+            -0.012900073369514622,
+            0.03357519186606058,
+            0.006705902094864628,
+            0.04243347200485736,
+            -0.03302953674711988,
+            0.009031534935013593,
+            0.01822227379789576,
+            -0.008491770335368761,
+            -0.02725412856390648,
+            -0.03630788717786838,
+            0.008259145747740535,
+            -0.019003260334340084,
+            0.015781275447792516,
+            -0.010039277287105153,
+            0.029665124310477384,
+            -0.03563370794521629,
+            0.004714858110268169,
+            0.004053913355055313,
+            0.0009503673086552651,
+            0.02068360255395961,
+            0.018524093113914366,
+            -0.002785698934204024,
+            -0.031443835341186485,
+            -0.02066365433385339,
+            0.006869321907853112,
+            0.02431260052207595,
+            -0.010367048093621498,
+            -0.005612149485859478,
+            0.0006316058139530722,
+            -0.024369562712568606,
+            -0.016845124984022828,
+            0.030359419843284823,
+            -0.040974427176153246,
+            0.005252911449822132,
+            0.04272936053727608,
+            -0.04599092715016919,
+            -0.03802288716943514,
+            -0.002314988728575789,
+            -0.024124014894379466,
+            -0.006360771882090754,
+            -0.002068030981218464,
+            0.016669991113175926,
+            0.0036232707799735537,
+            -0.028763764013482795,
+            0.00943401978066451,
+            -0.005173988601101547,
+            0.003971420424839974,
+            -0.0031669592494977393,
+            -0.02216336363004636,
+            -0.01657772032717298,
+            -0.02531962163186398,
+            -0.035174102848865116,
+            -0.0015903434876311088,
+            -0.005158703150821607,
+            -0.028703075279214722,
+            -0.03941940849540492,
+            -0.021696000925262537,
+            -0.0160221488026189,
+            -0.00843213135542359,
+            0.040641882367348774,
+            0.021319060666253945,
+            0.019336844555345886,
+            0.0025252167889382917,
+            -0.020346900609125973,
+            0.049440028814233025,
+            -0.007642182460707816,
+            0.035628691886108926,
+            -0.0017882888000877158,
+            -0.013461669667950893,
+            0.00455558389906464,
+            0.03825428026120858,
+            -0.014248434654950971,
+            -0.0009902062564027718,
+            0.00027811655483232545,
+            -0.0011866253757890136,
+            0.0008535155950871286,
+            0.015433650396777696,
+            0.001307651096343185,
+            -0.007112199543314309,
+            0.030380673552683906,
+            -0.012803247194895328,
+            0.01637501752192592,
+            0.018350814869962304,
+            -0.0013359719137099667,
+            0.008064546833360553,
+            -0.021430659268086814,
+            -0.00731572118780415,
+            0.0044578576331236325,
+            0.008422210247967642,
+            -0.015096885054883586,
+            -0.009964588397149773,
+            -0.02359494467299634,
+            -0.01174145797396426,
+            0.0023855749166744366,
+            -0.004559202896782688,
+            0.004902138778592298,
+            0.034595817216889944,
+            -0.016068744500427584,
+            -0.035106033294403614,
+            0.009997207702122002,
+            -0.02663088875167486,
+            0.02222306406130687,
+            3.983384942889326e-05,
+            0.006838221769859366,
+            -0.0023434342406307616,
+            0.001765445386886839,
+            -0.00169484278667461,
+            0.002862331185439823,
+            -0.018217975204395134,
+            0.0031859734246840257,
+            -0.0032119988065673565,
+            0.0009052950782312657,
+            -0.0012459653052875638,
+            0.00687339892509406,
+            0.006332799551164718,
+            -0.0008019538898592668,
+            -0.002968451740267742,
+            -8.541086355097327e-06,
+            -0.020561816931299912,
+            -0.021642512405352372,
+            -0.028259049261410905,
+            -0.000989887278779396,
+            -0.012359733017834338,
+            -0.027804394906433664,
+            0.011443385424045703,
+            -0.006872684999639775,
+            0.008327844771213648,
+            -0.008969527138034904,
+            0.006219994886600349,
+            -0.008415106130309062,
+            -0.0004143996206881767,
+            0.013119286511794273,
+            -0.02925209282917943,
+            -0.02096907906398536,
+            -0.03096513357712367,
+            0.01762223841790055,
+            -0.00925741670612824,
+            -0.002656382260603827,
+            0.0025223527960181835,
+            0.005092659166535706,
+            -0.033606542487570154,
+            -0.006653438998497412,
+            0.007313682256676618,
+            0.009547290751154595,
+            -0.03287788897629887,
+            0.02446728061355753,
+            0.017023222076526946,
+            0.01874233066565012,
+            -0.02417566312664911,
+            0.04049827067897005,
+            0.014426879325647042,
+            -0.021740606525480655,
+            0.016712907564723732,
+            0.027385150577780248,
+            0.03177163221499527,
+            -0.03469098547555545,
+            0.004481376248622418,
+            -0.01518200063132508,
+            0.018918680020516044,
+            -0.025284921244460287,
+            0.020625170445627613,
+            -0.016707629602243063,
+            0.021278719363178687,
+            0.03520334001264928,
+            -0.005524369131116028,
+            0.03608133014180745,
+            -0.019234548780647943,
+            0.0059985801736444555,
+            0.004041778461022664,
+            -0.01645503683802124,
+            0.01024694092614158,
+            -0.0042700351372843386,
+            0.0006059159967286885,
+            0.009406090340935747,
+            0.019133229156115282,
+            0.019007219188041775,
+            -0.04442208547053466,
+            0.028588965123434923,
+            0.00804120101594811,
+            0.021472086070876265,
+            0.014177820346835521,
+            -0.02708807970982664,
+            0.004742981711172693,
+            0.002072673190279628,
+            -0.004659713164250794,
+            0.0061167904730676514,
+            -0.013796604518930328,
+            -0.034540857808309226,
+            0.006338792319814448,
+            -0.018918703778187926,
+            0.015036937364271984,
+            -0.009026670935442384,
+            0.0011667525636770179,
+            -0.01152652811400482,
+            -0.011938679695598347,
+            0.0006170592505446954,
+            -0.005495151258143897,
+            -0.0008075736908949407,
+            -0.03171721592827769,
+            -0.04427658539412448,
+            0.02036132271908089,
+            -0.0021221965958423804,
+            -0.04320396743189005,
+            0.004111179168317842,
+            -0.025352428814134187,
+            -0.01865918186335911,
+            -0.011237699157960728,
+            -0.025047923887906474,
+            -0.04219630696740606,
+            -0.011108496740166623,
+            0.007785006066683012,
+            0.016810642610918525,
+            -0.009081815651299525,
+            0.040965574143584864,
+            0.0073944364393447965,
+            0.015313576815177263,
+            -0.017731360025694955,
+            0.026137777790243826,
+            -0.019314392060123,
+            -0.03108486330987073,
+            0.02676829566273746,
+            -6.259161089416372e-05,
+            0.003053748984540415,
+            0.014358356168574803,
+            2.8186924071913628e-05,
+            -0.006181789640967465,
+            0.01947730374003864,
+            -0.028739300217145797,
+            -0.023460503178221397,
+            0.0012460072421725688,
+            -0.008850395570726385,
+            -0.016271242375881602,
+            -0.035896509136022194,
+            0.005227881053956234,
+            0.03324337105109496,
+            0.004970419987430122,
+            0.008273497228454482,
+            0.01814500594735864,
+            -0.02229593571615308,
+            -0.024230675455380678,
+            0.000639112529814716,
+            -0.010266640429960994,
+            -0.010257247303470066,
+            -0.021302483296731352,
+            0.019150757066356872,
+            0.01651375496287087,
+            -0.00390189952706486,
+            -0.006971692709239087,
+            -0.010621003777733253,
+            -0.01918206725910585,
+            -0.02020345113211036,
+            -0.013827322378214754,
+            -0.017980206764311026,
+            -0.034153741666571154,
+            -0.032195001867229935,
+            0.03225205339201753,
+            0.026354817588919596,
+            0.0014076409241492998,
+            -0.007959081015805435,
+            -0.023070419006792155,
+            -0.004483328183581872,
+            0.02879516595345542,
+            0.002549934782150634,
+            -0.0001541849716285838,
+            -0.00830090759167373,
+            0.015341095409025125,
+            0.00835695019244165,
+            0.005107844984733648,
+            -0.06676974053142512,
+            -0.006027540973808174,
+            0.01655283585235268,
+            0.01351155740684989,
+            -0.002359114145814062,
+            0.02591301169926259,
+            0.01108099264396926,
+            -0.015361038608114896,
+            -0.019765743462262668,
+            0.035195761515422364,
+            0.02408910733949848,
+            0.023468767772473746,
+            -0.0034492999385544727,
+            -0.004472311490618405,
+            0.007030833258457338,
+            -0.010684252427867153,
+            -0.013326964647936429,
+            -0.004558513842504658,
+            0.026925146859337397,
+            0.016099443021514896,
+            -0.0064331198280825575,
+            0.010848931851939124,
+            -0.0008320209125061849,
+            -0.019444491995041537,
+            -0.001841330829156697,
+            0.008730275423516445,
+            0.0012974082030866743,
+            -0.014611902493400719,
+            -0.005911807768916134,
+            -0.019130604101278263,
+            0.024719468317569562,
+            0.0209456329723249,
+            0.03381727436368338,
+            -0.013237902591972769,
+            -0.0003695142331921278,
+            0.005410360755058108,
+            0.0002841632253252816,
+            0.0023352645111704958,
+            0.004801123870325242,
+            0.02623649813035263,
+            -0.029010710171522985,
+            -0.020741774066460415,
+            0.0068426980682519505,
+            0.013445270043414486,
+            0.0022744032984779294,
+            0.033840485942696216,
+            0.035040080489222324,
+            0.004166066423508316,
+            -0.010046666684988869,
+            -0.006148765564658713,
+            0.007071740742464309,
+            0.01865897335451031,
+            -0.023267411753900076,
+            -0.0005936657905496225,
+            0.004833030519903764,
+            -0.004229582648167378,
+            0.03620610659071198,
+            -0.009961238880412731,
+            -0.007197670122096852,
+            0.022212791947347127,
+            0.09819205436926473,
+            0.00047695621444943017,
+            0.0025354521225913395,
+            -0.029358210234742176,
+            -0.023614016211567963,
+            -0.0012564198366457442,
+            0.00242680708457405,
+            0.00830167274308547,
+            0.028076260517106013,
+            0.0004852400049405906,
+            0.011579274037419117,
+            0.02773972145320728,
+            -0.017217305534161797,
+            0.006752070301141707,
+            -0.011376051474975524,
+            0.024577071515205502,
+            -0.012694034026935636,
+            -0.0019809719792138673,
+            -0.016512649783050146,
+            0.015907402258634816,
+            0.004556574687833768,
+            0.004907510649166062,
+            0.011347630984408937,
+            0.009824484201337644,
+            -0.02650731900265866,
+            0.01756773113723491,
+            -0.005671986127099093,
+            0.004449611508541289,
+            -0.027841346572384462,
+            -0.020124233970502267,
+            0.0025986391930124847,
+            -0.0014440758308989809,
+            -0.009579444248630085,
+            0.006005274872617889,
+            -0.0024984235450999127,
+            0.03765664613203687,
+            -0.03765496603611941,
+            0.01409350051861866,
+            0.0066453175608372555,
+            -0.0015390841089599882,
+            0.011806653942640485,
+            0.017306914776093522,
+            -0.008206256939290151,
+            0.02589800448884443,
+            -0.00798213351353373,
+            0.009846274212000523,
+            0.015230070803055296,
+            -0.03449531102975745,
+            0.009174739184806312,
+            0.010145762060174144,
+            -0.024165660737765746,
+            0.012146877603301336,
+            -0.013784371016298891,
+            -0.004818759788054113,
+            0.005930446700613062,
+            0.007349837554764034,
+            0.010733731974181452,
+            -0.02295571464086674,
+            0.016176424826629898,
+            -0.010947841926210105,
+            0.0037766665765835445,
+            0.013559846998199008,
+            0.024892849808979323,
+            0.0012426907454129122,
+            0.0005950515136086193,
+            -0.01379469431504576,
+            0.005534724429523084,
+            -0.015667467221535825,
+            -0.022401586867694584,
+            -0.014192041775234079,
+            -0.006366846766730035,
+            0.01029598687444803,
+            -0.003972386708834858,
+            0.016971476088808737,
+            -0.010309648640697812,
+            -0.010645860303528099,
+            0.0037818310327362103,
+            0.01040806357838676,
+            0.008316605962234849,
+            -0.02855914079113881,
+            -0.005676310556150579,
+            0.004961084138163161,
+            0.00029285971503001265,
+            0.009592497954559435,
+            -0.014556073011576487,
+            0.013689526487160009,
+            -0.013290113354287095,
+            -0.0010726537498214612,
+            0.013201860397060236,
+            -0.012443552043930505,
+            -0.004012305744087929,
+            -0.01592905426965635,
+            -0.028487066798154617,
+            0.008462143660968396,
+            -0.01365614159523485,
+            0.001076882584445294,
+            0.0019503240125505646,
+            0.004730085646042709,
+            -0.016919069570642004,
+            -0.007860046623166481,
+            -0.033515448866521046,
+            0.011065416478687504,
+            0.02314589687882036,
+            -0.006435733825418645,
+            -0.030351727858789472,
+            -0.009546333526852862,
+            0.011327489786114963,
+            -0.0071765122743172715,
+            -0.01676736383051139,
+            -0.0029390038145944096,
+            0.007617910650439855,
+            -0.008174620754097493,
+            -0.00508943241317381,
+            -0.0025173969010414098,
+            0.0217328037301181,
+            0.016094975605203832,
+            0.01769642142712967,
+            -0.021400862603034036,
+            -0.00976992221747187,
+            0.026043879119776864,
+            -0.03275550610951898,
+            0.012441465975914529,
+            0.024587446266013183,
+            0.003446527051428522,
+            0.005492228054838714,
+            0.0030377554109693714,
+            0.005459159163772825,
+            -0.006985179226831048,
+            0.019925583798328282,
+            0.010870421035240845,
+            -0.019709062313471467,
+            0.005385443078244229,
+            0.02086085539375208,
+            0.0037248737640193097,
+            0.012892299156840998,
+            -0.017546870090541213,
+            -0.0020341807891543716,
+            0.015594779287017958,
+            0.04816214672670215,
+            -0.0032851797178091685,
+            -0.010450846098750195,
+            0.029513742590677183,
+            0.023526004023637855,
+            0.005373455940309873,
+            0.0012120777808635214,
+            -0.02525630456877349,
+            -0.013075130701774785,
+            0.01962235086017781,
+            0.016944870494393874,
+            0.013484925533584295,
+            -0.0023255656480479004,
+            -0.019236081983457366,
+            0.021520156296467717,
+            -0.01701733319731161,
+            0.012422659010697477,
+            -0.023764941037007197,
+            -0.001490790833263327,
+            -0.0035419008954878724,
+            0.021998949056452454,
+            0.0021819635638621092,
+            0.008891505304514552,
+            0.024354552127458608,
+            -0.006978548952551799,
+            -0.019526615985982355,
+            -0.0051799020400150305,
+            -0.014578953088901798,
+            0.022506635912517146,
+            -0.024117708614271965,
+            -0.007008641019892289,
+            0.011379795968360019,
+            -0.012362093847309038,
+            -0.0102534051092702,
+            0.001164934138876502,
+            0.017686807394319725,
+            0.0009423993435129515,
+            -0.014784508280754078,
+            0.01944445593807992,
+            -0.0011359271135469456,
+            0.010410915597949927,
+            -0.01921419010977189,
+            0.009390638453648025,
+            -0.016682442516992777,
+            -0.007849184048209236,
+            0.010343008873053859,
+            0.0012604521199837046,
+            0.003948300703645506,
+            0.002592490121754231,
+            0.007773437307969315,
+            -0.03355658426684504,
+            0.022695505487314516,
+            -0.010751606023585792,
+            -0.003538523688786008,
+            -0.0014125330167969218,
+            -0.01678692701672328,
+            -0.0006918162525881059,
+            0.005816502852598842,
+            0.010878358093847386,
+            -0.016045147059003256,
+            -0.00873270513444376,
+            -0.04920244395446931,
+            -0.030314009649638693,
+            0.023156333024481057,
+            0.013733657599720879,
+            -0.010971506342695331,
+            0.018852159415605117,
+            -0.023760853159532587,
+            -0.010587249404421732,
+            0.0011379869499042905,
+            0.023319856329056813,
+            -0.03539290744576506,
+            0.01215369703816749,
+            -0.014493682360886549,
+            -0.02039288669534806,
+            0.009070984596830142,
+            0.011445354735283254,
+            0.0007259852045513916,
+            -0.0011514627555568814,
+            -0.029350945123925453,
+            -0.010246845821727087,
+            0.02391068177425878,
+            -0.00861718589055971,
+            -0.004723044986826333,
+            0.024716520302409263,
+            0.0033721801437248873,
+            -0.012524663490691517,
+            -0.017955758068233754,
+            0.021612671655463236,
+            -0.026077203994996058,
+            0.007645944309444457,
+            0.010032183011253845,
+            -0.0173991263874487,
+            0.021649599230588117,
+            -0.007529701797485131,
+            0.020327763440441377,
+            0.004227390938169872,
+            -0.011249233530251522,
+            0.020352283974471443,
+            0.013644443758970054,
+            0.004216958800094986,
+            -0.004081669404784771,
+            0.006869353098014179,
+            0.02212732880780841,
+            -0.011021739614266063,
+            0.02219580717345023,
+            0.028777213295471067,
+            0.00017328191188631918,
+            0.03260179554743322,
+            -0.021287655520443285,
+            -0.03159728983004563,
+            0.021740163175540124,
+            -0.01035146858119169,
+            -0.021564906188712388,
+            0.028914245422221707,
+            -0.013674731836508768,
+            -0.00786476052825481,
+            -0.0166718429993922,
+            0.00533027311342322,
+            0.016680336311088843,
+            -0.005246190451159961,
+            -0.014360934066628332,
+            0.0029628103834216345,
+            0.019809487418628386,
+            0.014587234544007958,
+            0.029780009648869975,
+            -0.010778199815920887,
+            0.035180242185168895,
+            -0.002320576872571577,
+            0.008990157317473909,
+            0.012415465232073404,
+            0.006489435152972723,
+            -0.011251531176923445,
+            0.013501894212117511,
+            -0.003446759011266526,
+            0.008230548711588447,
+            -0.001449023474280409,
+            -0.035111679577202855,
+            -0.013709382199848678,
+            -0.04709294504969336,
+            -0.0023752976043235697,
+            -0.0025098360125255215,
+            0.0030975417740913154,
+            0.09736815869389318,
+            -0.005723049647039619,
+            -0.011988817571623457,
+            0.001625059666070763,
+            0.009611697638805556,
+            -0.0009896494420056814,
+            -0.007606836952841873,
+            0.015973152482488318,
+            -0.01343292635775299,
+            -0.02140928016038583,
+            -0.010532701190806147,
+            -0.01288594454368977,
+            0.012079198190522715,
+            0.015929716343161652,
+            -0.0005947359502228821,
+            -0.015207331663761012,
+            0.006524010655124294,
+            -0.004566385899752684,
+            0.0064717704997400245,
+            0.008647099975176308,
+            -0.010414668624343924,
+            -0.0071698879102925555,
+            0.030899143952119196,
+            -0.025301269549419674,
+            -0.024361748354052457,
+            -0.0010579931222861095,
+            -0.009055796737486008,
+            -0.010335459737012047,
+            -0.002692653633719985,
+            -0.0017679501322462056,
+            -0.010519994985692081,
+            -0.010019865263564572,
+            0.006537076627706128,
+            -0.015472130358273033,
+            0.004812857776601667,
+            -0.0047457011269495634,
+            -0.013296069496526007,
+            0.017054968086538724,
+            0.022563785357424853,
+            -0.02841491200807702,
+            0.0006755428946168761,
+            -0.017422510783648978,
+            0.012941211381006863,
+            0.008866069010589792,
+            0.006049618782357251,
+            -0.022451528209097505,
+            -0.023622906582403243,
+            -0.015019272754405791,
+            -0.01927535750568903,
+            -0.024973490129133732,
+            -0.009073381957684092,
+            0.0067692803507600945,
+            -0.001267339788153158,
+            0.021918281111033645,
+            0.000816538714279555,
+            -0.01812545613159305,
+            0.0061084913570894905,
+            0.000568371220952269,
+            0.018292497018668842,
+            -0.027468677288882186,
+            0.0033806422195878524,
+            -0.02307582511848465,
+            0.033508234091571124,
+            0.0039195050746594975,
+            -0.026680111514794824,
+            -0.021558668725755275,
+            -0.003311830287529699,
+            0.018994577319066473,
+            -0.0025745912019675805,
+            -0.00724049447539589,
+            0.014775207617625293,
+            0.0155334760323838,
+            -0.004693118176196483,
+            0.002015625406296403,
+            0.027968079002437102,
+            0.01077998866180723,
+            0.0313892866933904,
+            -0.015236169297590675,
+            0.004945828495617267,
+            -0.027215016713379062,
+            0.010214419950212732,
+            -0.01503548137805401,
+            0.008859554812646242,
+            -0.01290890682557532,
+            0.03531384210182567,
+            -0.012005613861257432,
+            -0.004250698950192301,
+            0.03253288796441852,
+            -0.029150986393345252,
+            -0.0035020051536906334,
+            -0.009255437891304041,
+            -0.004142204892160768,
+            0.013240580426083011,
+            -0.001198093874819309,
+            0.008485141356876356,
+            -0.01854370462655459,
+            9.726096864073609e-05,
+            -0.003221852304410318,
+            0.005404595442127598,
+            -0.02005458688172967,
+            -0.003314118809778734,
+            0.0073427568419969814,
+            0.004259518237708634,
+            0.002408630356035373,
+            0.016420646142725346,
+            0.006662121128241274,
+            -0.0029357701493526907,
+            0.03424224243607692,
+            0.008171895850772245,
+            0.0152521424877615,
+            -0.0018638264193202728,
+            0.0030234211261443034,
+            0.019287963516760636,
+            0.019912956661192856,
+            -0.00048443399729935116,
+            -0.01058935153883094,
+            0.00195113175754497,
+            -0.001140732137433033,
+            0.011048741602327437,
+            0.024396046054912645,
+            -0.021158025221644738,
+            -0.02419562092713724,
+            0.016377661005279716,
+            -0.015076145951740961,
+            0.01903712233378994,
+            0.010541463112889985,
+            -0.010099351302095792,
+            -0.00031558616108876247,
+            -0.042316016201379245,
+            0.010964989031040692,
+            -0.02068939391415103,
+            -0.04004120028915868,
+            -0.00244383573783919,
+            0.0009542507389088937,
+            -0.029439456243505052,
+            0.02822814855547978,
+            0.0023755032182188156,
+            -0.017280221740813364,
+            0.018223709137876153,
+            -0.005548654680726505,
+            -0.0024040301239229064,
+            0.005397711787238679,
+            -0.01092026090471121,
+            -0.010648485086769927,
+            0.008235250252660972,
+            0.010957715075632535,
+            0.005425683656535705,
+            -0.005698919020608095,
+            -0.011284911899894062,
+            0.009975882651638084,
+            0.012318901214538077,
+            -0.0019922884373083155,
+            0.010050520101815593,
+            0.0031720752428247446,
+            0.007352859750323051,
+            -0.0073483049386141005,
+            0.0046483587496626965,
+            -0.005093220128411125,
+            -0.026080057085901446,
+            0.012863580653903103,
+            0.0052434400313520774,
+            -0.011863360599637747,
+            0.0103209502958491,
+            -0.01048162756158923,
+            0.005787157766911021,
+            0.001318945460232246,
+            -0.0059333230422061395,
+            0.004821320238893602,
+            -0.0038922239931512784,
+            0.011267634571983924,
+            0.013256611037442042,
+            0.004592529900805719,
+            -0.013076954459595086,
+            -0.02177318635989856,
+            0.009001291224623467,
+            0.01959767163356408,
+            0.024161754192119852,
+            -0.030416109293488064,
+            0.003784634588325551,
+            -0.0014915849060829743,
+            -0.0020550564975080885,
+            -0.015632785947151753,
+            0.01587790981601866,
+            0.01716742056401084,
+            0.012721563202528083,
+            0.01893393006967701,
+            -0.004009617703771589,
+            0.010539100665843704,
+            -0.02080068888791658,
+            0.013642451766699067,
+            0.0008609869904066794,
+            -0.013629067231663643,
+            0.0014620655123815298,
+            0.027046304278766087,
+            -0.011056783825898616,
+            0.006206969699250696,
+            0.008097527175511791,
+            0.022111870736907183,
+            -0.012509479100313492,
+            -0.003735868296238069,
+            0.01516254968167132,
+            0.0014878674796313539,
+            0.0035887079417890352,
+            0.016533149527222273,
+            0.04729052276110659,
+            0.00102751186013391,
+            0.0016713773449738153,
+            0.0192177920520243,
+            -0.005563519063161466,
+            -0.01114821419109643,
+            -0.003549999863325982,
+            -0.021807893906469106,
+            0.010998558936481785,
+            0.00431279785115251,
+            -0.02186324212032519,
+            0.01024093366826206,
+            0.0125495473550584,
+            0.005268320135120931,
+            -0.0031679212410026603,
+            0.02574575247827441,
+            -0.019807879421996258,
+            0.014935295894965128,
+            0.03329450628384787,
+            0.00493356778223407,
+            -0.011423949333937285,
+            0.007263264249871384,
+            0.0027354459678742013,
+            -0.004494406041906684,
+            -0.00606745413660583,
+            0.010497391290734088,
+            -0.008191702034391382,
+            0.009870458191510176,
+            -0.009580130479693,
+            0.039202091799313755,
+            -0.009619018331262844,
+            -0.010782138248435458,
+            -0.004558645539551881,
+            -0.006241773108460662,
+            0.003478414182938609,
+            -0.003395165694642073,
+            0.005812225135633631,
+            -0.0069045407353789945,
+            -0.014607146220271209,
+            0.0008522982757210454,
+            0.00020175973815948303,
+            0.014001922919116725,
+            0.0011804904415274088,
+            -0.004871874532755353,
+            -0.007877217154706205,
+            0.015243277056007988,
+            -0.006047906208304912,
+            0.0006531937033317192,
+            -0.0011653523616313422,
+            -0.0013662556399359335,
+            -0.007401050012179761,
+            -0.015166105795085528,
+            0.00974609244984173,
+            0.016253733908242134,
+            -0.013857834499211056,
+            -0.005142936208363415,
+            0.012274716212832219,
+            0.008540229125193836,
+            0.011742606168187876,
+            0.0007451338650642946,
+            -0.009255813682572937,
+            0.008292531177374192,
+            -0.015020523035432158,
+            -0.008781317965985481,
+            0.0026250469418136603,
+            0.002381147382977664,
+            0.005839723981046825,
+            0.0006240641828668715,
+            0.004502463508605272,
+            0.005798912705218846,
+            -0.00497936644919367,
+            0.01112666823857836,
+            0.008198424171006548,
+            -0.002452143482105151,
+            -0.01017569410837814,
+            -0.001127589984272638,
+            -0.015733298167661346,
+            -0.016758315411028353,
+            -0.0009286266498936771,
+            0.015175988113750882,
+            -0.02352862898718829,
+            0.018996352975391644,
+            -0.0014524363611708201,
+            -0.0033802546171857704,
+            -0.021392428456030312,
+            -0.0025923983911570183,
+            0.0034214188668518796,
+            0.01255768230394476,
+            0.01858853630745928,
+            -0.02729543519465384,
+            0.010575894388563116,
+            -0.007204725245442141,
+            0.030394929424014717,
+            -0.018328782140005044,
+            -0.004022941273610227,
+            0.026915618079621992,
+            -0.02200018306828758,
+            0.0008791185377513978,
+            -0.011454606955158956,
+            -0.008103032693853504,
+            0.019287674837688177,
+            0.023747720480736777,
+            0.02038287242167783,
+            0.01364489294869211,
+            0.0007758044919789085,
+            0.004768030222445397,
+            0.03572705648994403,
+            0.010568241270862248,
+            0.014828087092859878,
+            -0.007542489523110974,
+            9.457808640264811e-05,
+            -0.02554033312739386,
+            0.01635899434369978,
+            2.587383534566272e-05,
+            -0.012370727506722627,
+            0.007162422186621379,
+            -0.013370882403931942,
+            -0.011066483600541058,
+            -0.007151077450730025,
+            -0.01721910018182534,
+            0.02152239849400241,
+            -0.012322440280680164,
+            0.004587729883596212,
+            -0.009079747696669277,
+            -0.013272707286450417,
+            -0.012953682633400515,
+            -0.014202214807187545,
+            -0.01313141988034687,
+            -0.026041403779538307,
+            -0.027109731260915763,
+            -0.006573285295895077,
+            -0.023780366346334524,
+            -0.013799072648505526,
+            0.005588622594136253,
+            -0.022617902666316757,
+            0.012262325666428604,
+            -0.012689827812482627,
+            -0.01226264872059864,
+            0.012805734807835008,
+            0.020274844315616616,
+            -0.011765231803328348,
+            -0.028494489703400308,
+            -0.018583843821043854,
+            -0.019970998470704358,
+            0.012576201137505969,
+            0.002266288538944273,
+            0.013165889075444961,
+            -0.003018437854660793,
+            0.003227412667880239,
+            -0.014895831590298957,
+            0.018570624675927183,
+            -0.03442390703780963,
+            -0.03108003806963536,
+            -0.013611306765348361,
+            0.005182777066476028,
+            -0.0015782033203578553,
+            -0.011429046504256945,
+            -0.0018606648572920801,
+            -0.00433805271182878,
+            -0.025109572024446088,
+            -0.010134856632190067,
+            0.011010662054484161,
+            -0.015205629927469414,
+            0.0072933565255675036,
+            0.0009018832564038863,
+            0.013870902834260887,
+            0.009587432944616442,
+            0.013393301893492882,
+            -0.006095063698384775,
+            0.004356485794113208,
+            -0.0026476612747324386,
+            0.0004894024149759108,
+            0.009336366919474478,
+            0.00839591561027103,
+            0.010551161369079399,
+            0.002260707777260338,
+            0.007830059223001696,
+            -0.0060927446216633265,
+            0.018065002687169148,
+            0.015436944875538481,
+            -0.0002520689442496089,
+            0.013323098485251487,
+            0.0003769862591044255,
+            0.00022293304921127018,
+            -0.0029970303106340466,
+            0.007888437018607987,
+            -0.011785957933642163,
+            -0.001311704406355858,
+            0.015138190893483804,
+            -0.01707247730569978,
+            -0.013045187198628258,
+            0.03107034781283638,
+            0.0018583638488268697,
+            -0.02495564203304796,
+            0.031146554085787762,
+            0.004220047610942943,
+            -0.005835199594223787,
+            -0.013205086624848782,
+            0.0014059047822222418,
+            -0.011570624844100298,
+            0.003584232809362274,
+            0.014295032061446826,
+            -0.02231984223610487,
+            0.010521094801839404,
+            0.008446892441796151,
+            -0.009681535919980559,
+            -0.012337659795586223,
+            0.009847073709209602,
+            0.016809328445200358,
+            0.004472485504034416,
+            -0.03539093810210523,
+            -0.011957694102753411,
+            0.02145470732008716,
+            -0.029237428225009913,
+            -0.00810041956259673,
+            0.007452962400105787,
+            0.0065154832040426065,
+            -0.019353109017609053,
+            -0.008580955020412362,
+            0.005725794605897509,
+            -0.004940390374637084,
+            -0.003892210172017758,
+            0.017541623128534682,
+            0.00485047043626186,
+            0.013698341792101025,
+            -0.01689738578307474,
+            0.009771458057400557,
+            -0.022222817828602557,
+            0.004021882074676753,
+            -0.023373700500461992,
+            -0.009873403167702603,
+            0.002364143609079849,
+            0.012454271200139762,
+            0.014328556467632212,
+            0.027839363143659326,
+            0.030056404238348527,
+            0.013995696971422429,
+            0.007833677585250953,
+            -0.016265199386953717,
+            -0.018134327237616175,
+            -0.006409513840075933,
+            -0.004033173350204784,
+            -0.008714798059218596,
+            0.0066373547199279235,
+            0.01858354212009877,
+            0.004499712802472377,
+            0.02609078206670711,
+            -0.006227189596272876,
+            0.009039744622014585,
+            -3.127526911951914e-05,
+            0.0008833930131827993,
+            -0.004226083827636182,
+            -0.01971685546584527,
+            0.00481003211084742,
+            -0.0013721482162825508,
+            0.025874758880591167,
+            0.00940496860916206,
+            0.0024915013399541847,
+            0.01615978314811047,
+            -0.006378952757397484,
+            -0.01174653607178726,
+            -0.010540107733238588,
+            -0.0061926268398085176,
+            -0.021752582127981104,
+            0.029059100503124387,
+            0.021195595568253612,
+            -0.010800765264123782,
+            -0.013693582166598082,
+            -0.00903369526059487,
+            0.02795767990556847,
+            0.004080965688836953,
+            -0.0013050212744006933,
+            0.016313311850308505,
+            0.012084555958999215,
+            0.012329325712491252,
+            0.009794698207689061,
+            -0.00036157148861933123,
+            0.0015364709122308352,
+            0.03722522141440073,
+            -0.007427028510548186,
+            0.0026463292682538495,
+            -0.005239324411651515,
+            0.0025987932868461313,
+            -0.009705817520422487,
+            0.007242122655628283,
+            -0.004012676385296197,
+            -0.0158760051470016,
+            0.015003444174308607,
+            -0.022443331305169994,
+            -0.0016749787204471238,
+            -0.008150635190952799,
+            0.0016696640640947303,
+            -0.02781023337840043,
+            -0.0025731206607371845,
+            0.006695065701630532,
+            -0.015547340828489069,
+            0.011816010501093885,
+            -0.01740673229051557,
+            -0.011553604323149105,
+            0.007914961618894248,
+            -0.012034157288065447,
+            0.0016684130271324446,
+            -0.0008268957621923278,
+            0.004460864332388434,
+            0.013426055079012884,
+            -0.011008808570221245,
+            0.012166046631826273,
+            -0.005816284310138776,
+            -0.007872091214834688,
+            -0.0016055129487028885,
+            -0.0030049472973942043,
+            -0.0025600746761171543,
+            -0.0009966231170649917,
+            0.034355910234323155,
+            -0.004446570896048543,
+            0.003761242147499512,
+            -0.01362639743248326,
+            0.0019833052855181806,
+            -0.036176376394999044,
+            0.028344464817309477,
+            -0.008642835889481802,
+            0.011013502221008498,
+            -0.0017508365480865596,
+            -0.013243706268954915,
+            -0.01456517309747086,
+            0.015247299501245178,
+            0.007602596572394442,
+            -0.005854531057349665,
+            0.013558248087347848,
+            -0.019724373614448298,
+            0.008530500822910953,
+            0.004463694543554244,
+            -0.015561397976987216,
+            0.004668780190328835,
+            -0.007479912658259757,
+            0.012657201279639194,
+            -0.01774730033362932,
+            0.004314594593150765,
+            0.00958571617187387,
+            -0.01892876084898881,
+            0.009901041800918215,
+            -0.028675879564512306,
+            -0.004631920242208613,
+            -0.012323455524366223,
+            0.0053804608827285865,
+            -0.000660864620708182,
+            -0.02389088252276507,
+            -0.028944280660230762,
+            -0.0002099667627277945,
+            0.00458533068247518,
+            0.0018878351997997833,
+            0.011621407953503755,
+            0.005970807015151215,
+            0.008542958629199003,
+            0.01023896054756221,
+            -0.00205207247588841,
+            0.009402752124876478,
+            0.04169706217414899,
+            -0.022899912321317197,
+            -0.011574970753549647,
+            0.005321872977369877,
+            0.011499271505460519,
+            -0.02647653483798284,
+            -0.011039791758347888,
+            0.0033424345263339144,
+            -0.005219877979550397,
+            0.03609469691834737,
+            -0.02448746405881972,
+            0.019020618652020673,
+            -0.011392297116547224,
+            0.007625475937968714,
+            -0.021465240076078376,
+            -0.003921088264697311,
+            -0.014265745991522477,
+            -0.00011221615798127438,
+            -0.0031478236827989703,
+            -0.0076053733863387755,
+            0.007485039021296698,
+            -0.001907678661303243,
+            0.0020348246022161925,
+            0.006291436821831794,
+            -0.02173305455485444,
+            -0.0019125499969493854,
+            -0.016432460924913166,
+            0.030367263349759286,
+            -0.016242095671439496,
+            0.005072805486919414,
+            0.0053258139728101506,
+            -0.002810639419737258,
+            0.012368771814660942,
+            0.004165967191536186,
+            0.00513754440091394,
+            -0.004199767241827414,
+            -0.005142642360546972,
+            -0.0053886448467479215,
+            0.017868874549760376,
+            -0.013280048598849825,
+            0.035024701466710065,
+            -0.0100959186045929,
+            -0.007094001877261568,
+            0.0019733042325232155,
+            0.010380430710282744,
+            4.2084202709296443e-05,
+            0.01053565670076933,
+            0.0034270489770133685,
+            -0.0006311078385359693,
+            0.0019774779328212756,
+            -0.017162448473652843,
+            0.003089758682621882,
+            0.01682656561693936,
+            0.005463677441664502,
+            0.004469261499451007,
+            -0.013332130891003062,
+            -0.012050233723282738,
+            0.004538268704442613,
+            -0.007533940408255447,
+            0.0003675737140813486,
+            -0.018816953644773076,
+            0.013972696748575243,
+            -0.0011096683686631858,
+            0.0002715683052236706,
+            0.00524678369365015,
+            0.023056514997908218,
+            0.01813167860994747,
+            -0.002553352602551003,
+            0.03155164978556312,
+            0.0174738879054593,
+            0.0029268039949260673,
+            0.010637295711777793,
+            -0.033222781195984666,
+            -0.013301873512972214,
+            -0.0013721703891642354,
+            0.02238421933149517,
+            -0.01530189703335162,
+            -0.014059029663525449,
+            -0.0037393534403749947,
+            0.0010394601933309767,
+            -0.005177359933828505,
+            0.0042489306011644495,
+            0.011039889511148341,
+            -0.014697595240068863,
+            0.007597149702441915,
+            0.0026780496822907634,
+            0.02092734730183164,
+            0.0004688144432243786,
+            0.017181410417231698,
+            -0.0005083970646221274,
+            0.00749639055342427,
+            0.007319420826337891,
+            0.017541463125355265,
+            -0.01002811389296805,
+            0.023582113845081534,
+            -0.008823933397729605,
+            0.0019889258902730535,
+            -0.01257106158422629,
+            -0.028159594067544737,
+            -0.012624750220149823,
+            0.015247564545279736,
+            0.005765911899842925,
+            0.006478299226263732,
+            -0.016537090297144265,
+            0.006276997440048981,
+            0.015647122171767974,
+            -0.002251683151059454,
+            0.010295358869874306,
+            0.009730378299068289,
+            0.00012816405890877098,
+            0.010181120688239766,
+            -0.00801256937783534,
+            -0.00734257767723293,
+            -0.004497520124843868,
+            0.023982754195729983,
+            -0.005443550092842393,
+            0.022434177122972028,
+            0.012647958478747348,
+            0.00616791857573897,
+            -0.010853355357954727,
+            -0.003340111254357573,
+            0.023389634440057473,
+            0.011863246790550376,
+            0.00029642699874398923,
+            -0.009846034060495678,
+            0.0019458876579524617,
+            0.0054010373765486005,
+            0.012983913038716568,
+            -0.02049427270240938,
+            -0.004344571352918878,
+            0.008461947538684664,
+            -0.008472919299762712,
+            0.004318309005722874,
+            -0.030156444522730416,
+            -0.005812303237222271,
+            0.021354352236942434,
+            0.008392281015122372,
+            -0.010232039722494646,
+            -0.004395682831498155,
+            -0.0006527587802367882,
+            0.0040601850269714945,
+            -0.0024323167053212176,
+            -0.0023729138545912176,
+            -0.008541233856441836,
+            -0.006470933419694742,
+            0.014035696617797647,
+            -0.0052691760528871924,
+            0.008597459268477037,
+            0.00012084859681905111,
+            -0.0017255958688825231,
+            -0.007323529232751411,
+            0.03382792341957058,
+            0.008087732655845718,
+            0.005980666974986827,
+            0.0013613625176327891,
+            0.0042244330945601215,
+            0.000616791811672998,
+            -0.0028374856646453707,
+            0.018075360591135542,
+            0.0156738939725117,
+            0.005054696911814988,
+            0.009038525980341447,
+            0.014672270369917403,
+            -0.019700405219230957,
+            0.012625337070556061,
+            0.012184185186281482,
+            0.019382321278525114,
+            0.0007219498737851579,
+            0.002564970959687783,
+            -0.010601873823737926,
+            -0.0029963537053764736,
+            -0.002070567089566093,
+            0.029563028293936092,
+            0.0005903976560069006,
+            -0.010509758568838935,
+            0.010992421945232951,
+            0.020603868018231348,
+            -0.02101157589451638,
+            -0.001971974816402967,
+            0.005963911670920596,
+            -0.010704207628592966,
+            -0.014466113540150018,
+            -0.021269176952661614,
+            -0.013331247081394942,
+            0.0071597345125287174,
+            -0.008162102923896894,
+            0.02367549407145414,
+            -0.014256771777502395,
+            -0.03960796112042958,
+            -0.014803626428839152,
+            -0.00864320347698291,
+            -0.027531337207277483,
+            -0.007305441146142331,
+            0.00965056093889974,
+            0.016698589600806273,
+            -0.003330867073501157,
+            -0.025198703196464594,
+            0.030483790406960656,
+            0.008872821868559751,
+            0.023708422767352917,
+            -0.021464374508407488,
+            -0.0018617153194555344,
+            -0.00886376784514331,
+            0.014505918224188001,
+            -0.017588761818663975,
+            -0.011756112651591795,
+            0.020014632992762026,
+            0.014356419813769714,
+            0.007246557500044434,
+            -0.004355132555323843,
+            0.0067299398462013936,
+            0.007649499235136303,
+            -0.012515181210363267,
+            0.012239792251154642,
+            -0.013499277313916029,
+            -0.024858295126228433,
+            0.014430561986516884,
+            0.002229494935179828,
+            0.009563004428064055,
+            -0.0015984171085659323,
+            -0.016197659608768088,
+            0.010078159490026602,
+            0.00808729175515505,
+            -0.017508840440512225,
+            -0.004793934415922065,
+            0.007133527615305415,
+            -0.020818869260580315,
+            0.031234686259160864,
+            0.011872765152279094,
+            -0.010692949770503467,
+            -0.001989039629661815,
+            0.004431079028880923,
+            0.016511544608975148,
+            0.007893700385963683,
+            -0.01997314388229276,
+            -0.003708006454566117,
+            0.01530128306203401,
+            0.00822588305477249,
+            0.004861703074728471,
+            -0.010836328783594584,
+            0.03434874763691911,
+            0.006082873924472328,
+            4.192481378506604e-05,
+            -0.0052997095110967115,
+            -0.0076108924981838355,
+            0.004196112457289409,
+            0.018905391806961686,
+            0.013330078288599763,
+            -0.004200590783114919,
+            -0.006630107527854746,
+            0.007778448128358027,
+            -0.01706991160178345,
+            0.007329056462125937,
+            0.009390327499494826,
+            -0.012870674959022891,
+            -0.0017571189976160295,
+            0.016462608870101713,
+            0.00209846661139665,
+            0.004942714850185376,
+            -0.011305050739130339,
+            0.004252776146827917,
+            0.015841821944109317,
+            0.018727567181082646,
+            -0.008399642221255733,
+            0.021448055799259867,
+            -0.020972459569228434,
+            0.027243639465535856,
+            0.028499400850980086,
+            0.016573144403275928,
+            -0.01847096443459798,
+            -0.0022409373140116146,
+            -0.0041737366139895414,
+            -0.014761716560267328,
+            0.0034546020161337894,
+            0.005125870074458691,
+            -0.01174186608185822,
+            -0.009577921232380895,
+            -0.024331993794416026,
+            0.011075959943215117,
+            0.007918147902741738,
+            -0.0015045185909419564,
+            -0.011791155919533844,
+            0.016019077577552835,
+            0.010141868950981925,
+            -0.010938402325619172,
+            -0.005635351352447256,
+            -0.019967361901734836,
+            0.009750427567651833,
+            0.009128294017175602,
+            -0.00300556760891084,
+            -0.011496103417643132,
+            0.012977890524465716,
+            0.0014080059843484115,
+            -0.01292380658817521,
+            0.010841954636959441,
+            -0.03142011651894961,
+            -0.005510537354374977,
+            -0.011027026734432255,
+            -0.003620482919145167,
+            -0.014415606143566519,
+            0.003608572766363589,
+            0.0005034611321129689,
+            -0.011470488351158329,
+            -0.0035005774696278,
+            -0.016207453351921627,
+            0.004676327282262429,
+            0.015021513761697039,
+            0.001679614645967629,
+            -0.006917375094634678,
+            0.021994274901326064,
+            0.018518524483306403,
+            -0.021903538417919984,
+            -0.020116013379669034,
+            0.004708084983787958,
+            -0.03261690161800251,
+            0.015019348187107588,
+            0.00966124471577622,
+            0.011676937880751252,
+            -0.028768229226408037,
+            0.007114534979415504,
+            0.00900420165933673,
+            0.016398927481511824,
+            0.0050879301160439245,
+            0.009453255486450082,
+            0.01222758288108498,
+            -0.0104657614414275,
+            -0.003164502778525433,
+            -5.410382753651127e-06,
+            -0.005618225986727351,
+            0.015607620507415484,
+            0.0008527697935056332,
+            0.03603666488542239,
+            -0.005277207843139309,
+            -0.0009799879252482288,
+            -0.011735731235560851,
+            0.00812046801749312,
+            -0.011713349784345638,
+            0.006334172347602515,
+            -0.0024643499727789167,
+            0.0001240009282616789,
+            0.002965067818077541,
+            0.009442572075213767,
+            -0.01086095783353607,
+            -0.007595379345224076,
+            0.007444897256325813,
+            -0.01482814340478338,
+            -0.00019984717819268013,
+            -0.009339343730980451,
+            0.0019680804106275156,
+            -0.02241031895876346,
+            0.006629838013824249,
+            0.040935644507527846,
+            0.02140216465404127,
+            -0.005481040712269146,
+            -0.012431659209559418,
+            -0.004680230194268222,
+            -0.007229300034633229,
+            -0.006482401275991039,
+            -0.004794698160419071,
+            -0.022762405726589375,
+            -0.006928553505112732,
+            0.004320321244630314,
+            0.013300138724212552,
+            0.009741018117644266,
+            -0.008842135469729511,
+            0.004531760574809748,
+            -0.02662650543210291,
+            -0.016578081411548973,
+            -0.0060021425889846495,
+            -0.015840130031005147,
+            0.006125301421471048,
+            0.027521841635711787,
+            -0.0063432924325983525,
+            0.01789356321527744,
+            0.005142029893259461,
+            0.013034833065163448,
+            -0.005338869351330258,
+            0.00035250243782976347,
+            -0.006010938630154189,
+            0.0032958656042048292,
+            0.003074861351445954,
+            -0.0015947217852680714,
+            0.004151590621941346,
+            -0.007139285822575166,
+            -0.0026091937868967284,
+            0.0011629380311555433,
+            0.0117841884590861,
+            0.006877419488576261,
+            -0.00016252653621991807,
+            -0.01761285866765538,
+            0.022221104524808997,
+            -0.018933508559881868,
+            -0.008840161102111973,
+            0.011092684029552064,
+            -0.009660487856185668,
+            0.00702597023411946,
+            0.020387386977378873,
+            -0.006854213562767843,
+            -0.009583981919303215,
+            0.013028181087649146,
+            -0.0036507672754997286,
+            0.004464368543493108,
+            0.009867428978560914,
+            0.006036303973414791,
+            0.002482944923100156,
+            0.013244650094892797,
+            0.004530294330688855,
+            -0.0032625482538717137,
+            -0.020238533752799943,
+            -0.005186498720507101,
+            0.019015752262933585,
+            -0.02252458872825473,
+            -0.0027641215988942563,
+            0.0044506144202026456,
+            -0.0030317840657635457,
+            -0.01733962766050954,
+            0.004205861489436576,
+            -0.018770275720867473,
+            -0.0033883519277721433,
+            -0.008244668812709337,
+            0.009948090119054079,
+            -0.023700732627507225,
+            0.009109315916173436,
+            -0.011739192106106307,
+            0.02356509175177699,
+            -0.003925851077921686,
+            0.024734433836185024,
+            0.0015142645677087268,
+            -0.013482933030628261,
+            0.0055323884713736554,
+            -0.0008930661843169134,
+            -0.004477731568313757,
+            0.029338490404300612,
+            -0.005418258701133154,
+            0.002249996261871466,
+            0.013491124734151694,
+            -0.021332541202417246,
+            0.021773687217263173,
+            0.0046662696076723836,
+            0.01628697326539432,
+            -0.00986238951830334,
+            -0.013799209313714441,
+            0.025178874553620743,
+            0.03324555035648896,
+            0.008398796308070151,
+            0.011261509865513953,
+            -0.01226890254177634,
+            0.018966175575083295,
+            -0.0038328997984309597,
+            0.004344837325292706,
+            0.00878668880458693,
+            -0.007200842513742962,
+            -0.011500562922303248,
+            -0.004533014670236755,
+            0.027639894892383943,
+            -0.021838848944317424,
+            -0.008920629621611768,
+            -0.005739181836252042,
+            -0.01593272996094732,
+            -0.038167534267554606,
+            -0.004371701247332816,
+            0.019604405563120094,
+            -0.005974358141737469,
+            -0.0018341151594086288,
+            0.014334266690716505,
+            0.00402620072221496,
+            -0.002852569009185304,
+            -0.008195218707838913,
+            -0.0027944153624309856,
+            -0.0059390492900516225,
+            0.016651206007022145,
+            -0.004961864528562493,
+            -0.007929830585906699,
+            0.01145431100211762,
+            -0.005731957927601062,
+            0.012130694936714224,
+            0.0004848418056781251,
+            8.392048424584131e-05,
+            -0.0015174120840537315,
+            0.00822932723013239,
+            0.018620869354789082,
+            -0.0055803454666417465,
+            0.025682778991091174,
+            0.006956936033581638,
+            -0.003495709501259274,
+            -0.005489431577052735,
+            -0.0044154293119170445,
+            -0.024561016942073285,
+            -0.0020864636079347357,
+            0.006593060072668424,
+            0.024333813203596122,
+            -0.02515989187381735,
+            -0.004953266000463339,
+            0.020957806164031402,
+            -0.0012192724005613,
+            -0.011978722479596745,
+            0.014690559986694138,
+            0.0041346025614047875,
+            0.009881584250499358,
+            0.012176393203318002,
+            0.015764530703333238,
+            -0.004923548157246672,
+            -0.0015192775000207833,
+            0.012766482928380375,
+            0.0005745973040874939,
+            0.00997809510696241,
+            0.016495444272181515,
+            0.010467514633877887,
+            -0.021108281220544853,
+            0.0015078843789125156,
+            0.00522593579521476,
+            -0.017481430635971396,
+            0.0034362821987740755,
+            0.005397819166037461,
+            0.011626527721895694,
+            0.004048632382026344,
+            0.013207335578216437,
+            -0.015768003434275563,
+            -0.005916168437120099,
+            0.008153798523776654,
+            0.004319451296835019,
+            -0.02599474886669325,
+            -0.01485009085106089,
+            -0.0006781964116146299,
+            -0.004444144921332929,
+            0.013556265879295017,
+            0.01285768162554067,
+            0.02033626001285354,
+            0.024637349990579514,
+            -0.013833060730799217,
+            -0.00842819657792762,
+            0.009436072455719896,
+            0.015560357465711765,
+            0.005798181411384387,
+            0.007874051721372273,
+            0.009297440071118126,
+            0.0077320823375042,
+            -0.004073209576661068,
+            -0.002973928843567532,
+            0.020205003097225747,
+            -0.019725382393637726,
+            0.0006746659838642147,
+            -0.018417443693472958,
+            0.007530796896254817,
+            0.01140696723506382,
+            0.01129515042745398,
+            0.002816883643872417,
+            -0.005274757308627299,
+            0.005716917047812153,
+            -0.06254973656832136,
+            -0.009191453047502128,
+            0.003384751551032943,
+            0.027256466708746898,
+            -0.010885336140129142,
+            -0.003415707910562931,
+            0.007309577090125959,
+            0.005511970708769456,
+            0.0005572239548625122,
+            0.003186427278034852,
+            0.019698716902282903,
+            -0.02052581021967601,
+            -0.010793968505805883,
+            0.004219269922138548,
+            0.0033111837923746746,
+            -0.007269521534951044,
+            0.004805737092302134,
+            0.014143978208501908,
+            0.01986656063456352,
+            -0.0024048854610842748,
+            0.01629335842915328,
+            -0.0015498603704589015,
+            -0.014496994444493684,
+            -0.002671064325909516,
+            0.0037875833047555925,
+            -0.022466433279066646,
+            -0.005980897597195858,
+            0.0043262697442044435,
+            -0.011226121648007667,
+            0.006978495397818936,
+            0.008452177557316863,
+            -0.003018545465557218,
+            -0.0008303307644324117,
+            0.017796789945838845,
+            0.006591364879905198,
+            -0.013464486184025026,
+            0.0028668426952326957,
+            -0.023377605222213174,
+            0.004325185794796361,
+            0.001416245546006914,
+            0.007809449561523174,
+            -0.008431335139468173,
+            -0.014774499666285603,
+            -0.005933810712528226,
+            0.017746847020610673,
+            -0.01965188730040385,
+            -0.002406875192894708,
+            0.01683009662984443,
+            -0.0002518220110313279,
+            0.005456812494894182,
+            -0.02011241622618092,
+            -0.0009782974741561381,
+            -0.02933296756078644,
+            -0.01171823509606186,
+            0.001162345317872531,
+            0.00892221099991475,
+            -0.006551308022163148,
+            -0.03934356275100932,
+            0.00427623835776202,
+            -0.013111664387776261,
+            -0.008752857644809231,
+            -0.010214546496134218,
+            -0.00518087722852466,
+            0.015240297918667867,
+            -0.012498100033235827,
+            0.01086490299272505,
+            -0.00917777212602538,
+            0.005895055740265625,
+            0.0090554651297836,
+            0.014212841798768113,
+            -0.005544047915369086,
+            -0.0033589999440643385,
+            -0.007924830667307956,
+            0.0022291797354234005,
+            -0.0032976226056924502,
+            0.009615674261248298,
+            -0.0051932155872883,
+            -0.0065636389655770455,
+            0.02426106498596339,
+            0.011445591067574932,
+            0.013312596099532533,
+            0.0055118624272110965,
+            -0.007220620072349783,
+            -0.01916551187613823,
+            -0.003977616240384473,
+            -0.013204506708246246,
+            -0.006315966975649096,
+            0.01434099854765068,
+            0.012588249778961223,
+            0.006357972880218866,
+            0.0007747002406996095,
+            -0.004794439624135275,
+            0.006975593520375694,
+            0.0262335496080837,
+            -0.016481909971520166,
+            -0.0013084662210086592,
+            -0.01369313900978409,
+            0.01921748474212319,
+            0.03333761672336437,
+            -0.0008918419419301244,
+            -0.007699525186523558,
+            0.013757216065348164,
+            0.00819045303047094,
+            -0.007411380297164911,
+            0.02453611080891859,
+            -0.025045462880463647,
+            -0.015989319900023054,
+            -0.011650886878708265,
+            -0.011407253985176972,
+            -0.008386367959227559,
+            0.016331961716602287,
+            -0.00141225438918211,
+            -0.0018505754298268654,
+            0.009557291617446645,
+            0.03357924917013024,
+            0.006916860901290756,
+            0.007030308953738567,
+            -0.002739345982346728,
+            -0.009376522438071562,
+            0.011482637968411055,
+            -0.007259054391879614,
+            -0.04075661435928125,
+            0.004250195164583401,
+            -0.005160505473755424,
+            0.02405623169060588,
+            -0.02197586561375289,
+            -0.00619175761282745,
+            -0.006736129049831973,
+            -0.028200015908139763,
+            -0.005867722104196303,
+            0.02805071895018145,
+            -0.015402198567775478,
+            0.014773294766214671,
+            -0.009502185346558072,
+            -0.0013419858580438616,
+            0.004443749010136947,
+            0.013008783570355656,
+            -0.02289662952366337,
+            0.0013834888740540108,
+            -0.00048549197205067054,
+            0.027969842651517314,
+            0.010437729090975158,
+            -0.0047705656249623744,
+            0.014202218020412936,
+            0.008240861154198411,
+            -0.018637315941614406,
+            0.007314019317343154,
+            -0.005747966765101792,
+            0.014833896739762147,
+            -0.01314847940891154,
+            0.007762100494499689,
+            0.00705026393499352,
+            -0.01278023669088385,
+            0.03016342404000438,
+            -0.015204129985039077,
+            0.007776946006562982,
+            0.009278778571285689,
+            0.009656723862179986,
+            0.01257686993982689,
+            0.004362439504889069,
+            -0.02898418193036562,
+            0.0002144431781454467,
+            -0.03828100004144633,
+            -0.00833748467467171,
+            -0.002611937107022567,
+            -0.002145329666219461,
+            0.007050630951707937,
+            -0.009383645162261629,
+            -0.033976442057557676,
+            0.0006429978367640709,
+            0.01603027598131823,
+            0.017501067413108286,
+            0.0002196997337451293,
+            0.013075379412332973,
+            -0.006197537539554383,
+            -0.021148120787551232,
+            -0.016986448660412497,
+            -0.010207705210051413,
+            -0.02122194744719993,
+            -0.020550470201191853,
+            -0.00908564461857315,
+            0.012090079127547193,
+            -0.009927390005879807,
+            0.013888458967497627,
+            0.029513250508042935,
+            -0.024529175134769883,
+            0.0011144447737101862,
+            0.021977677894839922,
+            -0.00198308986575084,
+            0.014108189311291326,
+            0.001696775006906642,
+            0.014122064251288995,
+            -0.0004166140484018266,
+            -0.010556016685519523,
+            -0.009729362492661684,
+            0.017487929296614076,
+            -0.0026043257634797124,
+            0.016715798948897194,
+            0.012183487338360088,
+            0.003771721868999568,
+            -0.0030887189945795796,
+            0.010667776697461199,
+            -0.026471157050402344,
+            0.014249996582064706,
+            -0.006690961709923224,
+            -0.0181382691440242,
+            -0.008729104191096461,
+            0.003882731567105777,
+            -0.005609384377642956,
+            -0.007794812949125115,
+            0.012967089730551055,
+            -0.00944374111323783,
+            0.02088525981049842,
+            0.0023400121055514958,
+            -0.012196925240774426,
+            0.003648688547842322,
+            -0.03322041209304968,
+            0.0015191765655963997,
+            -0.018402382776013956,
+            -0.024659631472836927,
+            0.03840173429666435,
+            0.011991452465253198,
+            0.006639702644606365,
+            -0.01675838115087676,
+            0.007953102302963526,
+            0.0004257479565099406,
+            0.017484338819046517,
+            0.011068864013925452,
+            0.006466915839538296,
+            0.008364454017976605,
+            -0.023316137291380285,
+            -0.010898421318668592,
+            0.016204569093730564,
+            0.012099473359791148,
+            -0.01017229369980751,
+            0.019431669546223253,
+            0.017955119269367787,
+            0.007300372071724306,
+            -0.006898619924677706,
+            0.01647927363640854,
+            0.021012103522831126,
+            0.010291016618557653,
+            -0.00047955552103474145,
+            0.0008684297441778696,
+            -0.010534247928997246,
+            0.005037930235889528,
+            0.035417626740307205,
+            0.014712595692093886,
+            0.010650306363963653,
+            -0.03626584475133921,
+            0.021764456453493054,
+            -0.011961521673661574,
+            0.009092981192874177,
+            0.006089104887183293,
+            0.017982801101846038,
+            0.006916969397519662,
+            0.014196254958707752,
+            0.011745069030657836,
+            -0.006689305450954901,
+            0.025205280281804708,
+            -0.001601414806100461,
+            -0.011992525142592405,
+            0.003386930324498371,
+            -0.01859193251438437,
+            0.005303261319773131,
+            -0.0041710536965100615,
+            -0.004459393120257307,
+            0.0032461828144552565,
+            -0.0018650956576455287,
+            0.015678225251799887,
+            0.003894142628084471,
+            -0.02355212701026675,
+            -0.0014873755115206368,
+            -0.024284295126348117,
+            0.0027260592623277375,
+            -0.022185571998195344,
+            0.00022883605560765203,
+            -0.005557568353382136,
+            -0.009749382813201403,
+            -0.025385857431333838,
+            -0.009853073898348951,
+            -0.0035843255959309307,
+            0.007661137888845632,
+            -0.010521037433807653,
+            -0.004246698084422667,
+            -0.0019920499321964815,
+            -0.019465113363548715,
+            0.009014255846983305,
+            0.027122533756959888,
+            0.016848221859975445,
+            0.027170883580103264,
+            -0.00019554112591107156,
+            0.013614675849485926,
+            -0.002066822813103284,
+            -0.01603063005944364,
+            0.008824663429777616,
+            0.0075653522381218495,
+            -0.00531780740880296,
+            0.014643348459785112,
+            0.0072634484937344704,
+            -0.012800149015319017,
+            0.009859503660384916,
+            -0.007161712176469609,
+            -0.006437857349742842,
+            -0.009764856421740622,
+            -0.02058868349990357,
+            -0.014866137971603676,
+            0.0012450207572602557,
+            -0.002123265406747006,
+            0.006000473862248991,
+            -0.004054016790850226,
+            0.009584879012937315,
+            -0.00608816807208866,
+            -0.00925801057439376,
+            -0.013962683072792106,
+            -0.008924356118158748,
+            -0.004210064145949253,
+            -0.004381709838058668,
+            -0.011732699976353612,
+            -0.005479523097469261,
+            0.0010680258077037758,
+            -0.0019401640391035755,
+            -0.01397652201595301,
+            0.011650839243603377,
+            0.0058731838291845645,
+            -0.002462743134993977,
+            0.02219008688925881,
+            0.01097316136828334,
+            0.0018083001685932406,
+            0.01366095798812196,
+            0.0003942601242216818,
+            -0.0064063532958756175,
+            -0.03343312148366622,
+            -0.0005083355836126291,
+            -0.02815586367527057,
+            0.025770078162165123,
+            -0.0069114949102666145,
+            -0.010322087105114614,
+            -0.002519047562374146,
+            -0.006063669644021698,
+            0.005873609611264377,
+            -0.002722940652283699,
+            0.0021229525800901114,
+            -0.023673778230768945,
+            0.0028629715379660113,
+            -0.003953653437464966,
+            -0.0036366957271968823,
+            0.007469664864329332,
+            0.0346520690389124,
+            -0.004054282838167587,
+            -0.0034765253405897676,
+            0.005159466707114532,
+            0.011642913651743502,
+            0.027845571220425726,
+            -0.004445196354520585,
+            -0.0032038508043339413,
+            -0.02233838584152591,
+            0.0054112576531374175,
+            -0.00019544838850471866,
+            -0.004733192299951088,
+            0.016664217348411003,
+            -0.016719175418727594,
+            -0.01631658832047864,
+            -0.00545969677717507,
+            -0.014570848053102338,
+            -0.00022620258237138677,
+            0.014454101124666536,
+            0.011056312437637014,
+            -0.004035526363810078,
+            -0.012459378363360557,
+            -0.0022595493558020706,
+            -0.030370952502579162,
+            0.012766010284884204,
+            0.1141689619126009,
+            -0.00766197047210299,
+            -0.00896655658602649,
+            -0.009481112393590025,
+            0.001435269335646347,
+            0.001338755472877713,
+            -0.011739143028767118,
+            0.07166096084053437,
+            -0.0077855767070652395,
+            0.0002139580430342728,
+            -0.011312198831181402,
+            0.001844005266872301,
+            -0.001825412693163496,
+            0.01371932421024605,
+            -0.005025814503687022,
+            -0.018213360991353643,
+            -0.0033612481616769416,
+            -0.02941403830641238,
+            -0.0041137865872933515,
+            -0.005683630256743215,
+            -0.004288198570156578,
+            -0.014982999363445122,
+            0.0067149665791261335,
+            -0.019661973497372313,
+            -0.012688556129040908,
+            0.001124391650175233,
+            -0.0033623453063430534,
+            0.00996944383866679,
+            -0.020496336814603385,
+            -0.0044354321069609536,
+            0.0019875840782527733,
+            0.009706675240563883,
+            0.0011600364428219214,
+            -0.012755178230821602,
+            -0.0026514442584739412,
+            0.004056350166031111,
+            0.01909951327913368,
+            0.014360281326789544,
+            0.003732344102045803,
+            0.008623457695718206,
+            0.014461868085333023,
+            -0.001692149528228451,
+            0.01948035376572433,
+            0.01172550239477637,
+            0.0031966531144076173,
+            0.0004779413449255626,
+            0.0030912562273611517,
+            -0.004597633235071031,
+            0.01582547400214653,
+            -0.014811177637844808,
+            -0.02411202772171917,
+            -0.0071272238837422185,
+            0.018015541137524113,
+            -0.013081287852028074,
+            1.8837689271256874e-05,
+            0.004062803284646432,
+            0.027569726868200254,
+            0.0010564850664255057,
+            -0.01179755720247972,
+            0.0028876134800690334,
+            -0.01638215925131053,
+            0.022944183106912663,
+            0.0026043713064243836,
+            0.03279173899194346,
+            -0.005103903273968361,
+            0.00578213669463383,
+            -0.020744194169889544,
+            -0.007588793365216579,
+            -0.00819862611489168,
+            0.0017401442433921013,
+            0.010993500174794084,
+            0.005411238350013134,
+            0.01620925462604661,
+            -0.00641615069699833,
+            -0.020751730783437995,
+            0.014776878596707538,
+            -0.009552193304486048,
+            -0.03393319554458277,
+            0.008881987295131347,
+            -0.029102297123590998,
+            -0.0045267103602711,
+            0.00368971742585232,
+            -0.014269341759285646,
+            -0.016264809810970175,
+            0.012837302095591106,
+            -0.0011890334180728564,
+            -0.008937992768323412,
+            0.0014444613725248846,
+            -0.017110187133760626,
+            -0.02385668037818897,
+            0.006017405364919834,
+            0.00676670672597817,
+            -0.0077994500447595,
+            0.018312452858615755,
+            -0.027015660037024585,
+            0.006123912473734307,
+            -0.012973676426738582,
+            0.009041974762537394,
+            -0.01580183887345168,
+            0.01163516332546347,
+            0.010887662470499361,
+            -0.001503467664709027,
+            -0.0076234124970223326,
+            0.01231202541338883,
+            0.004852720134894811,
+            0.033760365210093034,
+            0.028658326381800395,
+            0.000995999783627898,
+            -0.0025605367058226737,
+            -0.0020356794893366384,
+            0.011667525096213252,
+            -0.014049700287539817,
+            0.011218246246280464,
+            0.02658113044439717,
+            0.017568631325838108,
+            -0.019133062654008885,
+            0.018050678772708717,
+            -0.010356118256625172,
+            0.012872052420999813,
+            -0.025620154415048252,
+            -0.005135768485044442,
+            0.004936490920616527,
+            0.005163008729256001,
+            0.007425278892536464,
+            0.001410212306439329,
+            0.021891387289347516,
+            0.014999616600766989,
+            0.00027613811156650413,
+            0.00558422822386443,
+            0.01477009526981092,
+            0.010497782599293994,
+            -0.0005567894749233204,
+            -0.00467776506604011,
+            -0.016998035358377343,
+            0.00957271797745408,
+            -0.006065116857898469,
+            0.005138070401846389,
+            0.01698914855850691,
+            -0.01230257809343985,
+            0.011109844802663414,
+            0.007518553660463051,
+            -0.015504773927595823,
+            -0.017169933945839305,
+            0.005552829003043926,
+            -0.02252633790984703,
+            -0.01306722099657118,
+            0.02715689072730398,
+            -0.010513941399990838,
+            -0.02790582732974509,
+            -0.0009847768069205477,
+            0.009797865241108237,
+            -0.002772952073769185,
+            -0.0027968741526000576,
+            0.0037541733651479613,
+            0.010801137722284016,
+            -0.014658156970099418,
+            0.002075415233345957,
+            0.004735264309137368,
+            0.01930804315461151,
+            -0.0014151522239496115,
+            -0.015046118414100617,
+            -0.002288495894227231,
+            0.021592829677937263,
+            0.017174609713578173,
+            0.00265560378681856,
+            -0.016499825201963983,
+            -3.965296928410772e-05,
+            -0.0074142747014567635,
+            0.00021360696629503726,
+            0.00027431174904249525,
+            -0.0056391089282546054,
+            -0.018931681700770225,
+            -0.003888248121485032,
+            0.011181675419027736,
+            -0.00247252920175839,
+            0.01977860312223846,
+            -0.0011527575128223392,
+            -0.015763587433200915,
+            0.001960734212846797,
+            -0.005280860341480037,
+            -0.010372829570478649,
+            0.01351900743232294,
+            -0.009359942374603997,
+            0.006788746813046239,
+            -0.003212015225284119,
+            0.004611143085183367,
+            -0.0003308099698088422,
+            -0.017017265064185255,
+            0.02924142709831441,
+            -0.0018238549856433466,
+            -0.004769274041546295,
+            -0.0014058363976586562,
+            -0.033393331166741945,
+            0.015017165991303838,
+            -0.014574579250749552,
+            0.006745699409061803,
+            0.002705853802154516,
+            0.01272741296341735,
+            0.003253095246601883,
+            -0.008898616461779179,
+            0.0027485606928249326,
+            0.017040360927723885,
+            0.013756402798365262,
+            -0.007292864293090161,
+            0.008694855886257706,
+            -0.0028934389922887676,
+            -0.0037483208383542015,
+            0.01363963339139356,
+            0.004170611039133969,
+            -0.006282657671540625,
+            -0.0009784372731140945
+          ],
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "Mixture-of-Experts (MoE)",
+            "fine-grained expert segmentation",
+            "shared expert isolation"
+          ],
+          "title": "Deepseekmoe: Towards Ultimate Expert Specialization In Mixture-Of-Experts Language Models"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "1084fc41-4c97-458c-a6c9-2140fb99d5a9",
+        "properties": {
+          "page_content": "Appendices A. Overview Of Hyper-Parameters\n\nWe present the overview of hyper-parameters for DeepSeekMoE across various sizes in Table 7.\n\n# Params # Layers Hidden # Attn # Shared # Routed Relative Sequence Batch Size Learning Size Heads Experts Experts Expert Size Length (Sequence) Rate 2.0B 9 1280 10 1 63 (7 activated) 0.25 2048 2048 1.08e-3 16.4B 28 2048 16 2 64 (6 activated) 0.25 4096 4608 4.2e-4 144.6B 62 4096 32 4 128 (12 activated) 0.125 4096 4608 3.0e-4\n\nTable 7 | Overview of hyper-parameters for DeepSeekMoE across various sizes. The relative expert size is in comparison to a standard FFN.\n\nB. Comparing Deepseekmoe With Larger Models\n\nComparisons among DeepSeekMoE, GShard×1.2, and GShard×1.5 are shown in Table 8. Comparisons among DeepSeekMoE, Dense×4, and Dense×16 are shown in Table 9.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 2.3B 2.8B 1.9B # Activated Expert Params N/A 0.28B 0.35B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.824 1.808 1.808 HellaSwag (Acc.) 0-shot 53.7 54.4 54.8 PIQA (Acc.) 0-shot 71.8 71.1 72.3 ARC-easy (Acc.) 0-shot 46.8 47.3 49.4 ARC-challenge (Acc.) 0-shot 31.7 34.1 34.3 RACE-middle (Acc.) 5-shot 43.7 46.4 44.0 RACE-high (Acc.) 5-shot 31.9 32.4 31.7 HumanEval (Pass@1) 0-shot 3.7 3.0 4.9 MBPP (Pass@1) 3-shot 2.4 2.6 2.2 TriviaQA (EM) 5-shot 15.2 15.7 16.6 NaturalQuestions (EM) 5-shot 4.5 4.7 5.7\n\nTable 8 | Comparison between DeepSeekMoE and larger GShard models.\n\nAt a larger scale of 13B total parameters, we also compare DeepSeekMoE with GShard×1.2 and GShard×1.5, and show results in Table 10. At a larger scale, DeepSeekMoE even outperforms GShard×1.5 distinctly.\n\nMetric # Shot Dense×4 Dense×16 DeepSeekMoE Relative Expert Size N/A 1 1 0.25 # Experts N/A 4 + 0 16 + 0 1 + 63 # Activated Experts N/A 4 + 0 16 + 0 1 + 7 # Total Expert Params N/A 0.47B 1.89B 1.89B # Activated Expert Params N/A 0.47B 1.89B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.908 1.806 1.808 HellaSwag (Acc.) 0-shot 47.6 55.1 54.8 PIQA (Acc.) 0-shot 70.0 71.9 72.3 ARC-easy (Acc.) 0-shot 43.9 51.9 49.4 ARC-challenge (Acc.) 0-shot 30.5 33.8 34.3 RACE-middle (Acc.) 5-shot 42.4 46.3 44.0 RACE-high (Acc.) 5-shot 30.7 33.0 31.7 HumanEval (Pass@1) 0-shot 1.8 4.3 4.9 MBPP (Pass@1) 3-shot 0.2 2.2 2.2 TriviaQA (EM) 5-shot 9.9 16.5 16.6 NaturalQuestions (EM) 5-shot 3.0 6.3 5.7\n\nTable 9 | Comparison between DeepSeekMoE and larger dense baselines.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 15.9B 19.8B 13.3B # Activated Expert Params N/A 2.37B 2.82B 2.05B # Training Tokens N/A 100B 100B 100B HellaSwag (Acc.) 0-shot 66.6 67.7 69.1 PIQA (Acc.) 0-shot 75.6 76.0 75.7 ARC-easy (Acc.) 0-shot 56.8 56.8 58.8 ARC-challenge (Acc.) 0-shot 39.9 37.6 38.5 RACE-middle (Acc.) 5-shot 51.6 50.6 52.4 RACE-high (Acc.) 5-shot 37.4 36.3 38.5 HumanEval (Pass@1) 0-shot 6.1 6.1 9.8 MBPP (Pass@1) 3-shot 7.0 11.6 10.6 TriviaQA (EM) 5-shot 36.5 36.7 38.2 NaturalQuestions (EM) 5-shot 12.6 12.1 13.7\n\nTable 10 | Comparison between DeepSeekMoE and larger GShard models at a larger scale.\n\nC. Training Benchmark Curves Of Deepseekmoe 16B\n\nWe present the benchmark curves during training of DeepSeekMoE 16B and DeepSeek 7B (Dense) in Figure 7 for reference.",
+          "keyphrases": [
+            "DeepSeekMoE",
+            "hyper-parameters",
+            "GShard models",
+            "Dense baselines",
+            "training benchmark curves"
+          ],
+          "embedding": [
+            0.020979074761271477,
+            -0.0013986049452796578,
+            -0.019327709451317787,
+            0.0138933714479208,
+            0.0006998290773481131,
+            -0.02645554021000862,
+            0.0001438888575648889,
+            0.05624751001596451,
+            -0.006693925708532333,
+            0.024147000163793564,
+            -0.016437821090221405,
+            -0.0021316087804734707,
+            0.04061009734869003,
+            0.005270044784992933,
+            0.00756173487752676,
+            0.03029749169945717,
+            0.026859955862164497,
+            0.02480417490005493,
+            0.0019567832350730896,
+            -0.041183020919561386,
+            0.017339332029223442,
+            -0.04290178790688515,
+            -0.025309694930911064,
+            0.008383204229176044,
+            -0.016185062006115913,
+            0.009588027372956276,
+            0.022377679124474525,
+            0.014769605360925198,
+            -0.037071458995342255,
+            0.022091219201683998,
+            0.01640412025153637,
+            0.023371869698166847,
+            0.0032205828465521336,
+            -0.005885093938559294,
+            -0.02074316516518593,
+            -0.004515977576375008,
+            0.06709934026002884,
+            0.027870995923876762,
+            -0.010388433001935482,
+            0.018485177308321,
+            0.029404405504465103,
+            0.011500577442348003,
+            -0.013918647542595863,
+            -0.00502149760723114,
+            0.005880881100893021,
+            -0.009082507342100143,
+            -0.018165014684200287,
+            0.014019750989973545,
+            0.011744911782443523,
+            -0.05675303190946579,
+            -0.0348808728158474,
+            -0.017726896330714226,
+            -0.008109381422400475,
+            -0.013952348381280899,
+            -0.0411493182182312,
+            -0.03166239336133003,
+            0.028376515954732895,
+            0.061841931194067,
+            0.04182334616780281,
+            0.008939276449382305,
+            0.02399534359574318,
+            -0.006129428278654814,
+            -0.03993606939911842,
+            0.0007677582907490432,
+            0.008855023421347141,
+            -0.015452057123184204,
+            0.022579887881875038,
+            -0.005105750635266304,
+            -0.006647586356848478,
+            -0.020608359947800636,
+            -0.037745483219623566,
+            0.022377679124474525,
+            -0.003243752522394061,
+            0.006015686318278313,
+            0.02106332778930664,
+            -0.01328674703836441,
+            0.027803592383861542,
+            0.02632073499262333,
+            0.00384195102378726,
+            -0.013918647542595863,
+            0.017726896330714226,
+            0.0043074507266283035,
+            -0.01196397002786398,
+            0.03747587278485298,
+            0.04842880368232727,
+            -0.026287034153938293,
+            -0.040374185889959335,
+            -0.06318998336791992,
+            0.010321030393242836,
+            -0.008244186639785767,
+            -0.018384072929620743,
+            -0.008678090758621693,
+            0.03508307784795761,
+            -0.02180475741624832,
+            -0.03354967013001442,
+            0.06864959746599197,
+            -0.012494766153395176,
+            0.007115192245692015,
+            -0.03442590311169624,
+            -0.0069087715819478035,
+            0.004235835280269384,
+            0.00045523117296397686,
+            0.008661240339279175,
+            0.0043685343116521835,
+            0.005177366081625223,
+            -0.00362710515037179,
+            -0.01878848858177662,
+            0.0002004965499509126,
+            -0.004111561458557844,
+            0.0718512237071991,
+            0.012410513125360012,
+            -0.01227570790797472,
+            -0.048327699303627014,
+            -0.013927072286605835,
+            -0.02032189816236496,
+            -0.038655418902635574,
+            -0.029775120317935944,
+            -0.021568847820162773,
+            0.024281805381178856,
+            -0.002023132750764489,
+            0.033263206481933594,
+            0.020642060786485672,
+            0.030044730752706528,
+            -0.01792910508811474,
+            -0.043070293962955475,
+            0.04421613737940788,
+            0.0006208415725268424,
+            -0.0033933022059500217,
+            -0.0019631022587418556,
+            -0.029842523857951164,
+            0.02118128351867199,
+            -0.0036713380832225084,
+            -0.03408889099955559,
+            -0.030954666435718536,
+            0.00232539139688015,
+            -0.008694942109286785,
+            0.05213594809174538,
+            0.03949795290827751,
+            -0.01417140755802393,
+            -0.0003188461414538324,
+            0.0034438541624695063,
+            -0.04034048691391945,
+            0.029337003827095032,
+            -0.017246652394533157,
+            -0.03181404992938042,
+            0.022327126935124397,
+            0.017895404249429703,
+            0.06558278203010559,
+            -0.0028098479378968477,
+            0.041351526975631714,
+            0.0015386760933324695,
+            0.04003717377781868,
+            0.02350667491555214,
+            0.015763795003294945,
+            -0.009074081666767597,
+            -0.004326407797634602,
+            -0.03693665191531181,
+            0.00613785395398736,
+            -0.03360022231936455,
+            -0.015098193660378456,
+            -0.001018938492052257,
+            -0.004907755646854639,
+            -0.017288779839873314,
+            -0.0007556469063274562,
+            -0.002022079424932599,
+            0.047788478434085846,
+            0.023355018347501755,
+            -0.013244620524346828,
+            -0.06804297864437103,
+            0.03263973444700241,
+            0.04620451480150223,
+            0.007477481383830309,
+            -0.006196831353008747,
+            0.03788029029965401,
+            0.003601829055696726,
+            0.02229342609643936,
+            0.020709464326500893,
+            0.01904124952852726,
+            -0.06531316787004471,
+            0.04522717744112015,
+            -0.015300401486456394,
+            0.003485980909317732,
+            0.01658105105161667,
+            0.011264667846262455,
+            0.0310389194637537,
+            -0.0018630513222888112,
+            0.029033690690994263,
+            0.026859955862164497,
+            -0.012604295276105404,
+            0.02136664092540741,
+            0.031932003796100616,
+            0.008248399011790752,
+            0.04553049057722092,
+            -0.001733511919155717,
+            0.0005845073610544205,
+            -0.018687384203076363,
+            0.008838172070682049,
+            0.03353281691670418,
+            -0.017423585057258606,
+            0.018215566873550415,
+            0.0006161023629829288,
+            -0.027685638517141342,
+            -0.0010357891442254186,
+            -0.013337299227714539,
+            0.027651937678456306,
+            0.0262701828032732,
+            -0.008509584702551365,
+            0.0066855004988610744,
+            0.008404267951846123,
+            -0.016067106276750565,
+            -0.031190576031804085,
+            -0.013935497961938381,
+            -0.017069721594452858,
+            -0.0532480925321579,
+            -0.0025634069461375475,
+            0.03521788492798805,
+            -0.007785005960613489,
+            0.022512484341859818,
+            0.010860252194106579,
+            -0.0323195718228817,
+            0.003799824509769678,
+            0.009933465160429478,
+            0.014179832302033901,
+            0.006390613969415426,
+            -0.017229802906513214,
+            -0.02350667491555214,
+            -0.0032016257755458355,
+            0.008753919042646885,
+            0.032841939479112625,
+            -0.010919229127466679,
+            0.001601866097189486,
+            -0.001936773071065545,
+            -0.01959731988608837,
+            0.0043348330073058605,
+            0.03257232904434204,
+            -0.01013567391782999,
+            -0.00016798004799056798,
+            0.007005663122981787,
+            0.02662404626607895,
+            0.018485177308321,
+            0.01279807835817337,
+            -0.015266700647771358,
+            -0.04192444682121277,
+            -0.016564201563596725,
+            0.01824926771223545,
+            -0.00505098607391119,
+            -0.026742001995444298,
+            -0.01732248067855835,
+            0.057629264891147614,
+            0.024197550490498543,
+            0.023304466158151627,
+            -0.06281927227973938,
+            -0.02283264696598053,
+            0.005535442847758532,
+            -0.01727192848920822,
+            0.05668563023209572,
+            0.059415437281131744,
+            0.005628121085464954,
+            -0.00452861562371254,
+            -0.04121671989560127,
+            0.04023938253521919,
+            -0.01184601616114378,
+            -0.049979064613580704,
+            0.018704235553741455,
+            -0.01627773977816105,
+            0.01627773977816105,
+            0.033633921295404434,
+            0.027685638517141342,
+            0.0044443621300160885,
+            0.008231548592448235,
+            0.04266587644815445,
+            -0.023152809590101242,
+            -0.0458001010119915,
+            -0.02374258264899254,
+            0.021349789574742317,
+            -0.018721086904406548,
+            -0.007696540094912052,
+            -0.02852817252278328,
+            0.02633758634328842,
+            0.05055198818445206,
+            0.02662404626607895,
+            0.018350372090935707,
+            0.016142934560775757,
+            0.025056935846805573,
+            -0.024096447974443436,
+            0.0011490045581012964,
+            -0.023675180971622467,
+            -0.043306201696395874,
+            -0.017710046842694283,
+            -0.015199298039078712,
+            0.0068076676689088345,
+            0.006015686318278313,
+            -0.022849498316645622,
+            0.005493315868079662,
+            0.02606797404587269,
+            0.014407316222786903,
+            -0.03140963613986969,
+            0.011702785268425941,
+            0.05119231343269348,
+            -0.035116780549287796,
+            -0.03194885700941086,
+            0.0317297987639904,
+            0.017844852060079575,
+            -0.0067992424592375755,
+            -0.01885589212179184,
+            0.01466007623821497,
+            -9.174066690320615e-06,
+            0.01113828830420971,
+            -0.009284715168178082,
+            0.0007667051395401359,
+            -0.011938693933188915,
+            -0.0016303015872836113,
+            0.007801856845617294,
+            -0.03386983275413513,
+            -0.008905574679374695,
+            0.019176054745912552,
+            0.022124920040369034,
+            0.034510158002376556,
+            -0.06474024802446365,
+            -0.01166065875440836,
+            0.06969434022903442,
+            0.01836722157895565,
+            -0.04610341414809227,
+            0.025478200986981392,
+            0.019799528643488884,
+            -0.007351101376116276,
+            0.004520190414041281,
+            0.04559789225459099,
+            -0.04684484004974365,
+            -0.014002900570631027,
+            0.029050542041659355,
+            -0.04040788859128952,
+            -0.03139278292655945,
+            0.01892329379916191,
+            0.025023233145475388,
+            -0.06376291066408157,
+            -0.03761067986488342,
+            0.005404850002378225,
+            0.012258856557309628,
+            -0.007397440727800131,
+            -0.021956413984298706,
+            0.0029320153407752514,
+            -0.036094117909669876,
+            -0.01426408626139164,
+            0.0003749272436834872,
+            -0.005135239567607641,
+            -0.028545022010803223,
+            0.002346454653888941,
+            0.020793717354536057,
+            0.01627773977816105,
+            0.00159554707352072,
+            0.012191453948616982,
+            0.03424054756760597,
+            -0.006175767630338669,
+            0.0027550833765417337,
+            -0.028443917632102966,
+            -0.02645554021000862,
+            0.01579749584197998,
+            0.0032079447992146015,
+            0.00201260088942945,
+            -0.017162399366497993,
+            -0.04377802088856697,
+            -0.03366762399673462,
+            0.028545022010803223,
+            -0.020557807758450508,
+            -0.013623761013150215,
+            0.021821608766913414,
+            -0.004482276272028685,
+            -0.011862866580486298,
+            0.012292558327317238,
+            0.002847762079909444,
+            0.030819861218333244,
+            -0.03619522228837013,
+            0.006053600460290909,
+            0.017844852060079575,
+            -0.001491283648647368,
+            0.035116780549287796,
+            -0.010422134771943092,
+            -0.02392794005572796,
+            0.021838458254933357,
+            -0.049069128930568695,
+            0.024871578440070152,
+            0.03848691284656525,
+            0.012831779196858406,
+            0.03066820651292801,
+            0.031055770814418793,
+            -0.009326841682195663,
+            0.011610106565058231,
+            -0.012385237030684948,
+            -0.0024496649857610464,
+            0.033768728375434875,
+            -0.059482838958501816,
+            -0.023034855723381042,
+            0.010801274329423904,
+            0.018872741609811783,
+            0.03943055123090744,
+            0.017221376299858093,
+            -0.010834976099431515,
+            -0.022546187043190002,
+            -0.008138869889080524,
+            0.013640611432492733,
+            0.04927133768796921,
+            -0.0011468982556834817,
+            -0.003104734467342496,
+            -0.005493315868079662,
+            0.026371287181973457,
+            0.006803454831242561,
+            0.012233581393957138,
+            0.023708881810307503,
+            0.0004207399906590581,
+            0.022074367851018906,
+            -0.0037366345059126616,
+            -0.0342910997569561,
+            -0.008581199683248997,
+            0.007616499438881874,
+            0.03515048325061798,
+            -0.0021610974799841642,
+            0.021097028627991676,
+            -0.02473677322268486,
+            0.01978267729282379,
+            -0.039767563343048096,
+            0.020490406081080437,
+            0.031679246574640274,
+            -0.0028561872895807028,
+            -0.0045328279957175255,
+            0.01227570790797472,
+            -0.015157170593738556,
+            0.0019136033952236176,
+            0.029185347259044647,
+            -0.03471236303448677,
+            0.014778031036257744,
+            0.012115626595914364,
+            0.02945495769381523,
+            -0.04320509731769562,
+            -0.022984303534030914,
+            -0.013775416649878025,
+            -0.011685934849083424,
+            0.04320509731769562,
+            -0.04023938253521919,
+            -0.02259673923254013,
+            -0.0018862209981307387,
+            0.0372062623500824,
+            0.019850080832839012,
+            -0.013177217915654182,
+            -0.04327250272035599,
+            -0.033515967428684235,
+            -0.05008016899228096,
+            -0.012284132651984692,
+            -0.03922834247350693,
+            0.002550769131630659,
+            -0.07731083780527115,
+            0.01941196247935295,
+            -0.050113871693611145,
+            0.020642060786485672,
+            0.046811141073703766,
+            -0.008572774939239025,
+            0.030836712568998337,
+            -0.0196478720754385,
+            0.014205108396708965,
+            -0.06251595914363861,
+            -0.0068076676689088345,
+            0.025747811421751976,
+            0.02945495769381523,
+            -0.003494406118988991,
+            -0.003734528087079525,
+            0.019445665180683136,
+            -0.01279807835817337,
+            -0.0065675457008183,
+            -0.008930850774049759,
+            0.02608482539653778,
+            0.008496946655213833,
+            -0.010674894787371159,
+            0.024601968005299568,
+            0.008265250362455845,
+            0.015502609312534332,
+            0.03761067986488342,
+            -0.011365772224962711,
+            -0.04023938253521919,
+            -0.0034417477436363697,
+            0.029117945581674576,
+            0.029707718640565872,
+            -0.031072622165083885,
+            0.014339913614094257,
+            -0.003869333304464817,
+            0.005906157195568085,
+            0.006512781139463186,
+            0.004033627454191446,
+            0.029758270829916,
+            -0.006394826341420412,
+            0.026708299294114113,
+            -0.014811732806265354,
+            0.019850080832839012,
+            -0.026421839371323586,
+            0.03147703781723976,
+            0.012132477015256882,
+            -0.030162686482071877,
+            0.005910370033234358,
+            -0.01628616452217102,
+            0.016918065026402473,
+            0.019361412152647972,
+            -0.013143517076969147,
+            -0.027500281110405922,
+            0.013446828350424767,
+            0.013480530120432377,
+            0.012225155718624592,
+            0.004494914319366217,
+            0.027399176731705666,
+            0.03006158210337162,
+            -0.02227657660841942,
+            -0.03545379266142845,
+            0.0174404364079237,
+            -0.021872159093618393,
+            0.015039216727018356,
+            -0.03427424654364586,
+            0.01426408626139164,
+            0.00647486699745059,
+            -0.03595931455492973,
+            0.04010457545518875,
+            0.020642060786485672,
+            0.0019451983971521258,
+            0.004128412343561649,
+            -0.005990410689264536,
+            -0.009326841682195663,
+            0.05129341781139374,
+            -0.015957577154040337,
+            0.019816379994153976,
+            0.015637414529919624,
+            0.012882331386208534,
+            0.03683554753661156,
+            -0.03322950750589371,
+            -0.014575823210179806,
+            0.01347210444509983,
+            0.014499994926154613,
+            -0.03445960581302643,
+            -0.01365746185183525,
+            -0.02148459479212761,
+            -0.011450025252997875,
+            0.040374185889959335,
+            -0.014525271020829678,
+            -0.02131608873605728,
+            0.007658625952899456,
+            0.0023359230253845453,
+            -0.016934916377067566,
+            0.008092531003057957,
+            -0.015047641471028328,
+            -0.005194216500967741,
+            0.04064379632472992,
+            -0.023978492245078087,
+            0.03622892498970032,
+            0.031561292707920074,
+            0.016859088093042374,
+            -0.042699579149484634,
+            0.0006656011682935059,
+            0.006112577859312296,
+            -0.014188257977366447,
+            -0.01597442850470543,
+            0.034442752599716187,
+            0.014887560158967972,
+            0.01450842060148716,
+            -0.023152809590101242,
+            -0.006205256562680006,
+            0.02362462878227234,
+            -0.023304466158151627,
+            0.02578151412308216,
+            -0.00396411819383502,
+            -0.02332131750881672,
+            -0.03213421255350113,
+            0.007650200743228197,
+            -0.022680992260575294,
+            0.012755951844155788,
+            -0.003985181916505098,
+            0.012292558327317238,
+            0.01236838661134243,
+            -0.025259142741560936,
+            0.0036587000358849764,
+            0.019209755584597588,
+            0.000130460990476422,
+            -0.0003262182872276753,
+            -0.009048805572092533,
+            0.004604443442076445,
+            0.018198715522885323,
+            -0.001285916194319725,
+            0.005215280223637819,
+            0.022006966173648834,
+            0.031005218625068665,
+            0.004381172358989716,
+            -0.003250071546062827,
+            -0.020793717354536057,
+            0.02797210030257702,
+            -0.02357407659292221,
+            0.02773619070649147,
+            0.028174307197332382,
+            0.014921261928975582,
+            -0.03855431452393532,
+            -0.0021800545509904623,
+            -0.016496798023581505,
+            -0.008353715762495995,
+            0.01776059903204441,
+            0.022883199155330658,
+            -0.024281805381178856,
+            -0.0024075384717434645,
+            -0.004147369414567947,
+            0.0215014461427927,
+            0.003218476427718997,
+            -0.030769309028983116,
+            -0.010843401774764061,
+            0.009866062551736832,
+            0.00896455254405737,
+            -0.01456739753484726,
+            0.0019831124227494,
+            0.011888142675161362,
+            0.006150492001324892,
+            -0.0016060788184404373,
+            -0.00011077993985963985,
+            0.01897384598851204,
+            -0.002618171740323305,
+            0.01567954197525978,
+            -0.013750140555202961,
+            -0.013446828350424767,
+            -0.011087736114859581,
+            0.05247296392917633,
+            -0.0032205828465521336,
+            0.03899243474006653,
+            0.014188257977366447,
+            0.009722832590341568,
+            0.04431724175810814,
+            -0.013253046199679375,
+            -0.009309991262853146,
+            0.01721295155584812,
+            -0.01750783808529377,
+            -0.03568970412015915,
+            0.004103136248886585,
+            -0.005025709979236126,
+            -0.028983140364289284,
+            0.01071702130138874,
+            0.005038348026573658,
+            -0.004067328758537769,
+            0.0009341586264781654,
+            0.008846597746014595,
+            -0.026843104511499405,
+            0.03247122839093208,
+            0.005944071337580681,
+            0.011879716999828815,
+            0.034746065735816956,
+            -0.016201911494135857,
+            -0.007974576205015182,
+            0.013792267069220543,
+            -0.01518244668841362,
+            0.028545022010803223,
+            -0.01347210444509983,
+            0.01047268696129322,
+            0.01707814633846283,
+            0.030381744727492332,
+            0.03622892498970032,
+            -0.02492213062942028,
+            -0.06261706352233887,
+            0.011711210012435913,
+            -0.004638144746422768,
+            0.011559554375708103,
+            0.016260888427495956,
+            0.01233468484133482,
+            -0.005640759132802486,
+            0.01811446249485016,
+            -0.007540671620517969,
+            0.028999989852309227,
+            -0.006121003068983555,
+            -0.0326734334230423,
+            0.020962223410606384,
+            0.006007261108607054,
+            0.023034855723381042,
+            0.014339913614094257,
+            -0.03895873203873634,
+            -0.009596452116966248,
+            -0.004596018232405186,
+            0.010270479135215282,
+            -0.03383613005280495,
+            0.00483614020049572,
+            0.010759147815406322,
+            0.017912253737449646,
+            0.002041036495938897,
+            -0.007595436181873083,
+            0.010556939989328384,
+            -0.005227918270975351,
+            -0.016303015872836113,
+            0.0131350914016366,
+            -0.0022664139978587627,
+            -0.013059263117611408,
+            0.026303883641958237,
+            -0.03545379266142845,
+            -0.016993893310427666,
+            -4.235703818267211e-05,
+            0.0013027668464928865,
+            0.013244620524346828,
+            0.003953586798161268,
+            0.0015492078382521868,
+            -0.02896628901362419,
+            -0.03848691284656525,
+            0.04475535824894905,
+            0.022967452183365822,
+            -0.0033153677359223366,
+            -0.0003277980431448668,
+            -0.0004360108869150281,
+            -0.005792415235191584,
+            0.023894239217042923,
+            0.03164554387331009,
+            -0.020658912137150764,
+            0.005004646722227335,
+            -0.027466580271720886,
+            -0.019698424264788628,
+            0.015283551067113876,
+            0.009031955152750015,
+            0.033701326698064804,
+            0.006217894610017538,
+            0.006912984419614077,
+            0.018384072929620743,
+            0.008610688149929047,
+            -0.004507552366703749,
+            -0.003808249719440937,
+            -0.022697841748595238,
+            0.03230271860957146,
+            0.0390598364174366,
+            0.0149381123483181,
+            0.014070303179323673,
+            -0.034746065735816956,
+            -0.003814568743109703,
+            0.010885528288781643,
+            0.02492213062942028,
+            0.02424810267984867,
+            0.006748690269887447,
+            0.001917816000059247,
+            0.015157170593738556,
+            -0.03703775629401207,
+            -0.013573208823800087,
+            0.009798659943044186,
+            -0.014129280112683773,
+            -0.027988949790596962,
+            -0.006087301764637232,
+            0.022916901856660843,
+            -0.005897731985896826,
+            0.0021526720374822617,
+            -0.004659208003431559,
+            -0.011542703956365585,
+            -0.015527885407209396,
+            -0.004541253671050072,
+            0.006693925708532333,
+            0.02380998618900776,
+            0.0044317240826785564,
+            0.024770474061369896,
+            -0.009343692101538181,
+            0.011087736114859581,
+            0.006546482443809509,
+            -0.014162981882691383,
+            0.016488373279571533,
+            -0.013059263117611408,
+            -0.026708299294114113,
+            -0.0092425886541605,
+            0.01781114935874939,
+            -0.00644537853077054,
+            -0.0016945448005571961,
+            -0.01647152192890644,
+            0.005004646722227335,
+            0.036397431045770645,
+            0.008265250362455845,
+            0.017558390274643898,
+            0.023523524403572083,
+            -0.002397006843239069,
+            -0.009318416006863117,
+            0.0131350914016366,
+            0.0055902074091136456,
+            0.0055859945714473724,
+            0.021585699170827866,
+            -0.001512346905656159,
+            0.04387912526726723,
+            -0.05058568716049194,
+            0.008383204229176044,
+            0.027803592383861542,
+            -0.0057081617414951324,
+            -0.005939858499914408,
+            -0.032841939479112625,
+            0.01653892546892166,
+            -0.01678325980901718,
+            0.009798659943044186,
+            -0.011155138723552227,
+            0.028427068144083023,
+            -0.03723996505141258,
+            -0.008378991857171059,
+            -3.804563675657846e-05,
+            0.00811359379440546,
+            0.016083957627415657,
+            0.018653683364391327,
+            -0.006580183748155832,
+            -0.02497268095612526,
+            -0.003673444502055645,
+            0.012882331386208534,
+            0.04290178790688515,
+            -0.003953586798161268,
+            -0.014685352332890034,
+            0.0215014461427927,
+            -0.011171989142894745,
+            -0.020119691267609596,
+            0.0027635085862129927,
+            -0.024821026250720024,
+            0.029471809044480324,
+            0.03612782061100006,
+            -0.041958149522542953,
+            -0.024450311437249184,
+            0.002029451774433255,
+            -0.010936079546809196,
+            0.002504429779946804,
+            0.021012775599956512,
+            0.02743287943303585,
+            -0.004170538857579231,
+            -0.007629137486219406,
+            0.0206926129758358,
+            -0.005316384136676788,
+            0.015949152410030365,
+            0.016665305942296982,
+            -0.011348920874297619,
+            -0.0069551109336316586,
+            -0.017844852060079575,
+            -0.03270713612437248,
+            -0.00037361079012043774,
+            -0.0186031311750412,
+            -0.03481346741318703,
+            -0.0346449613571167,
+            -0.01398605015128851,
+            -0.010278903879225254,
+            -0.003949373960494995,
+            0.0364985354244709,
+            0.02394479140639305,
+            0.0251074880361557,
+            0.015746943652629852,
+            -0.013278322294354439,
+            0.05405692383646965,
+            -0.006816092878580093,
+            0.0460360087454319,
+            -0.020187092944979668,
+            0.0008951914496719837,
+            -0.019816379994153976,
+            0.03226901963353157,
+            -0.014390465803444386,
+            0.011525853537023067,
+            -0.009899764321744442,
+            0.006533844396471977,
+            0.0009504826739430428,
+            0.007574372924864292,
+            -0.0019451983971521258,
+            -0.009419520385563374,
+            0.027079014107584953,
+            -0.0023274978157132864,
+            0.029471809044480324,
+            0.020406153053045273,
+            0.004600231070071459,
+            0.005076262168586254,
+            -0.029707718640565872,
+            -0.00035149429459124804,
+            0.005067836958914995,
+            -0.0066855004988610744,
+            -0.018148163333535194,
+            0.0024201765190809965,
+            -0.007001450285315514,
+            0.007999852299690247,
+            -0.0039051410276442766,
+            -0.009225737303495407,
+            0.014104004949331284,
+            0.025629857555031776,
+            0.0069845994003117085,
+            -0.04391282796859741,
+            0.0035702341701835394,
+            -0.03936314582824707,
+            0.002685574349015951,
+            0.010818125680088997,
+            0.010093546472489834,
+            -0.002889888593927026,
+            0.0057123745791614056,
+            0.009613302536308765,
+            -0.004202133975923061,
+            -0.020962223410606384,
+            0.002352773677557707,
+            -0.015047641471028328,
+            -0.006146279163658619,
+            -0.011281518265604973,
+            3.797981480602175e-05,
+            -0.003614467103034258,
+            -0.0019262413261458278,
+            -0.005986197851598263,
+            -0.0067318398505449295,
+            -0.027146417647600174,
+            -0.02822485938668251,
+            -0.006108365021646023,
+            0.0002985726750921458,
+            -0.00706885289400816,
+            -0.03024693951010704,
+            0.03134223073720932,
+            -0.008539073169231415,
+            0.007136255502700806,
+            -0.0093015655875206,
+            0.009882913902401924,
+            -0.002344348467886448,
+            -0.005960921756923199,
+            -0.004553891718387604,
+            -0.026421839371323586,
+            -0.013994474895298481,
+            -0.03895873203873634,
+            0.016741132363677025,
+            -0.008471670560538769,
+            0.007106767036020756,
+            -0.010506387799978256,
+            0.001101612113416195,
+            -0.018148163333535194,
+            -0.0009278396028093994,
+            -0.002559194341301918,
+            0.027651937678456306,
+            -0.024163849651813507,
+            0.009074081666767597,
+            0.01616821065545082,
+            0.015485758893191814,
+            -0.0346449613571167,
+            0.02008599042892456,
+            0.012191453948616982,
+            -0.019243456423282623,
+            -0.0025423436891287565,
+            0.018350372090935707,
+            0.02576466277241707,
+            -0.013901796191930771,
+            -0.014162981882691383,
+            -0.004326407797634602,
+            0.009630153886973858,
+            -0.025006383657455444,
+            0.00954589992761612,
+            -0.010203076526522636,
+            0.017356181517243385,
+            0.025174889713525772,
+            -0.011256242170929909,
+            0.04023938253521919,
+            -0.03501567617058754,
+            -0.012831779196858406,
+            0.013497380539774895,
+            -0.023675180971622467,
+            0.013177217915654182,
+            -0.0003612360742408782,
+            -0.016690582036972046,
+            0.021754205226898193,
+            -0.006428527645766735,
+            0.01841777376830578,
+            -0.03693665191531181,
+            0.030078433454036713,
+            0.03012898378074169,
+            0.006656011566519737,
+            -0.011079310439527035,
+            -0.019513066858053207,
+            -0.01056536566466093,
+            0.0057418630458414555,
+            -0.0015397293027490377,
+            0.025730961933732033,
+            -0.014331488870084286,
+            -0.02466936968266964,
+            0.004633932374417782,
+            -0.0030478634871542454,
+            0.004692909307777882,
+            -0.02643868885934353,
+            0.004105242434889078,
+            -0.009832361713051796,
+            -0.0017924892017617822,
+            0.0006161023629829288,
+            -0.005990410689264536,
+            -0.009200462140142918,
+            -0.028696678578853607,
+            -0.056416019797325134,
+            0.0014891773462295532,
+            -0.0037913990672677755,
+            -0.023725733160972595,
+            -0.017743747681379318,
+            -0.012402087450027466,
+            -0.014449442736804485,
+            0.018518878147006035,
+            -0.0017208738718181849,
+            -0.023793134838342667,
+            -0.011862866580486298,
+            0.002586576621979475,
+            0.0017630006186664104,
+            0.0024412397760897875,
+            0.02362462878227234,
+            0.011163563467562199,
+            0.031493887305259705,
+            -0.018097611144185066,
+            0.02694420889019966,
+            -0.017238227650523186,
+            -0.03831840679049492,
+            0.03980126604437828,
+            -0.0009425839525647461,
+            -0.022697841748595238,
+            0.010801274329423904,
+            0.0009594346047379076,
+            -0.006483292207121849,
+            0.044115033000707626,
+            -0.0205746591091156,
+            -0.013429977931082249,
+            0.010177800431847572,
+            -0.006441165693104267,
+            -0.016631603240966797,
+            -0.03299359604716301,
+            -0.015393080189824104,
+            0.03240382298827171,
+            -0.006373763084411621,
+            0.013783842325210571,
+            0.035487495362758636,
+            -0.023776285350322723,
+            -0.029353853315114975,
+            0.012073500081896782,
+            -0.020675763487815857,
+            -0.01615135930478573,
+            0.0027782530523836613,
+            0.018636832013726234,
+            0.012098775245249271,
+            -0.0012248324928805232,
+            0.0006171555141918361,
+            -0.004945669323205948,
+            -0.012250431813299656,
+            0.003391195787116885,
+            -0.023776285350322723,
+            -0.035656001418828964,
+            -0.024012194946408272,
+            -0.01162695698440075,
+            0.03280824050307274,
+            0.03457755967974663,
+            -0.00030146888457238674,
+            -0.022495634853839874,
+            -0.02204066701233387,
+            -0.0029256963171064854,
+            0.039329446852207184,
+            0.013758566230535507,
+            0.012410513125360012,
+            -0.005333234556019306,
+            0.010396858677268028,
+            0.0005781883373856544,
+            0.006281084381043911,
+            -0.039093535393476486,
+            -0.008197847753763199,
+            0.019058099016547203,
+            0.001322777010500431,
+            -0.01629459112882614,
+            0.018771637231111526,
+            0.00517315324395895,
+            -0.013497380539774895,
+            -0.022259725257754326,
+            0.018518878147006035,
+            0.013674312271177769,
+            0.008004064671695232,
+            0.0044570001773536205,
+            -0.01873793639242649,
+            0.0005866136634722352,
+            0.009192036464810371,
+            -0.01285705529153347,
+            -0.0011142500443384051,
+            0.021400341764092445,
+            0.012739100493490696,
+            -0.008383204229176044,
+            0.012452639639377594,
+            0.013969198800623417,
+            -0.007115192245692015,
+            -0.008172571659088135,
+            8.287103264592588e-05,
+            0.014778031036257744,
+            -0.007148893550038338,
+            -2.983422928082291e-05,
+            -0.011525853537023067,
+            0.010700170882046223,
+            0.004709760192781687,
+            0.011778613552451134,
+            -0.02443346008658409,
+            -0.006592821795493364,
+            0.0055902074091136456,
+            0.02160254865884781,
+            -0.0079324496909976,
+            -0.013708014041185379,
+            0.009908189997076988,
+            -0.024450311437249184,
+            -0.02185530960559845,
+            0.00659703416749835,
+            0.0009615409071557224,
+            -0.009402669966220856,
+            0.033751875162124634,
+            0.0323195718228817,
+            -0.0010084068635478616,
+            -0.0011511108605191112,
+            0.00995031651109457,
+            0.008180996403098106,
+            0.017608942463994026,
+            -0.022883199155330658,
+            -0.006003048736602068,
+            -0.0013817542931064963,
+            0.016067106276750565,
+            0.04856361076235771,
+            -0.005826116539537907,
+            -0.0022011178079992533,
+            0.015898600220680237,
+            0.0920720174908638,
+            -0.00039783361717127264,
+            0.0067866044119000435,
+            -0.02394479140639305,
+            -0.030769309028983116,
+            0.005017284769564867,
+            0.001989431446418166,
+            0.019058099016547203,
+            0.05786517634987831,
+            0.003043650882318616,
+            0.0014407316921278834,
+            -0.0022179684601724148,
+            0.003943054936826229,
+            -0.0085432855412364,
+            -0.022512484341859818,
+            0.015839623287320137,
+            -0.0067866044119000435,
+            0.010683320462703705,
+            -6.480264710262418e-05,
+            0.01659790240228176,
+            0.004101030062884092,
+            0.008193634450435638,
+            0.006146279163658619,
+            0.017406733706593513,
+            -0.021535146981477737,
+            0.026421839371323586,
+            -0.010219926945865154,
+            0.011525853537023067,
+            -0.014702202752232552,
+            -0.027517132461071014,
+            0.019799528643488884,
+            -4.670134876505472e-05,
+            -0.007490119431167841,
+            -0.004002032335847616,
+            -0.011348920874297619,
+            0.02606797404587269,
+            -0.021400341764092445,
+            0.022849498316645622,
+            0.01261272095143795,
+            -0.020119691267609596,
+            0.010497963055968285,
+            0.008585412986576557,
+            0.0010257840622216463,
+            0.015763795003294945,
+            -0.008821321651339531,
+            0.0161008071154356,
+            0.01242736354470253,
+            -0.027112716808915138,
+            0.009958741255104542,
+            0.014550547115504742,
+            -0.009461646899580956,
+            0.01546048279851675,
+            0.008753919042646885,
+            0.001712448545731604,
+            -0.0010089334100484848,
+            -0.01075072307139635,
+            -0.0043769595213234425,
+            -0.009748108685016632,
+            0.007641775533556938,
+            -0.005552293267101049,
+            0.00820627249777317,
+            0.016134509816765785,
+            0.04782218113541603,
+            0.008320014923810959,
+            0.003953586798161268,
+            3.060764720430598e-05,
+            0.00025091689894907176,
+            -0.015966001898050308,
+            -0.022192321717739105,
+            -0.013649036176502705,
+            -0.009832361713051796,
+            0.024821026250720024,
+            -0.005792415235191584,
+            0.006390613969415426,
+            -0.017179250717163086,
+            0.005463827401399612,
+            0.0017177143599838018,
+            0.010843401774764061,
+            0.0033722389489412308,
+            -0.042396266013383865,
+            -0.0045791673474013805,
+            0.014213534072041512,
+            -0.011104586534202099,
+            -0.001104771508835256,
+            -0.031611841171979904,
+            0.003911459818482399,
+            -0.0158648993819952,
+            -0.0044907014816999435,
+            0.016749558970332146,
+            -0.020153392106294632,
+            0.0006840315763838589,
+            0.0032816664315760136,
+            -0.015940727666020393,
+            0.005122601520270109,
+            -0.008737068623304367,
+            0.0016840130556374788,
+            0.018215566873550415,
+            0.002660298254340887,
+            -0.01298343576490879,
+            -0.002645554021000862,
+            -0.024568265303969383,
+            0.01750783808529377,
+            0.029758270829916,
+            -0.008008277043700218,
+            -0.01941196247935295,
+            -0.005177366081625223,
+            0.014247234910726547,
+            -0.004145262762904167,
+            -0.00814308226108551,
+            0.0031405421905219555,
+            0.003479661885648966,
+            -0.020052287727594376,
+            0.00356602156534791,
+            0.003753485158085823,
+            0.01585647277534008,
+            0.010489537380635738,
+            0.014769605360925198,
+            -0.029286451637744904,
+            -0.023472972214221954,
+            0.007102554198354483,
+            -0.019378261640667915,
+            0.005463827401399612,
+            0.03986866772174835,
+            -0.005404850002378225,
+            0.014348339289426804,
+            0.011011907830834389,
+            0.022006966173648834,
+            -0.023658329620957375,
+            0.017524689435958862,
+            0.0020642061717808247,
+            -0.017710046842694283,
+            -0.010321030393242836,
+            0.016673730686306953,
+            0.00954589992761612,
+            0.02766878716647625,
+            -0.019260307773947716,
+            -0.004570742137730122,
+            0.007696540094912052,
+            0.03398778662085533,
+            0.003955692984163761,
+            -0.014617949724197388,
+            0.027517132461071014,
+            0.047114454209804535,
+            0.008897149935364723,
+            0.01652207411825657,
+            -0.01678325980901718,
+            -0.005101538263261318,
+            0.007785005960613489,
+            0.009849212132394314,
+            0.013337299227714539,
+            -0.0019746869802474976,
+            -0.020962223410606384,
+            0.033027298748493195,
+            -0.009899764321744442,
+            0.002022079424932599,
+            -0.027904696762561798,
+            0.018569430336356163,
+            0.009689130820333958,
+            0.005093112587928772,
+            0.004452787339687347,
+            0.001162695698440075,
+            0.04340730607509613,
+            0.004701334983110428,
+            -0.029303302988409996,
+            0.012065074406564236,
+            0.002390687819570303,
+            0.020153392106294632,
+            -0.013615335337817669,
+            -0.013059263117611408,
+            0.004739248659461737,
+            -0.0018978058360517025,
+            -0.010877102613449097,
+            0.008998253382742405,
+            -0.0005845073610544205,
+            0.019243456423282623,
+            -0.009933465160429478,
+            -0.001282756682485342,
+            -0.0138933714479208,
+            0.0014723266940563917,
+            -0.016067106276750565,
+            0.025360247120261192,
+            -0.014407316222786903,
+            0.004398022778332233,
+            0.009630153886973858,
+            -0.00914990995079279,
+            -0.0034712364431470633,
+            0.007953512482345104,
+            0.009933465160429478,
+            -0.02369203232228756,
+            0.018822189420461655,
+            0.0015323570696637034,
+            0.016977041959762573,
+            0.003606041893362999,
+            -0.01224200613796711,
+            0.007047789636999369,
+            0.005362723488360643,
+            0.017153974622488022,
+            -0.00942794606089592,
+            0.005817691329866648,
+            -0.030904114246368408,
+            -0.02374258264899254,
+            0.038284704089164734,
+            0.015350953675806522,
+            -0.022630440071225166,
+            0.015839623287320137,
+            -0.02148459479212761,
+            -0.006656011566519737,
+            0.01285705529153347,
+            0.01365746185183525,
+            -0.03110632300376892,
+            0.00914990995079279,
+            -0.009040380828082561,
+            0.009166760370135307,
+            0.02241138182580471,
+            0.023119108751416206,
+            0.0038103561382740736,
+            -0.00668971287086606,
+            -0.021400341764092445,
+            -0.006091514602303505,
+            0.007907173596322536,
+            0.006314785685390234,
+            -0.007203658111393452,
+            0.02106332778930664,
+            0.01426408626139164,
+            -0.0055017415434122086,
+            -0.005396424792706966,
+            0.02413014881312847,
+            -0.03043229691684246,
+            0.019816379994153976,
+            0.0010763360187411308,
+            -0.03301044926047325,
+            0.01959731988608837,
+            -0.007325825747102499,
+            0.030196387320756912,
+            -0.0035428518895059824,
+            -0.008762344717979431,
+            0.0012722250539809465,
+            0.01162695698440075,
+            0.0028161669615656137,
+            -0.00014138758706394583,
+            0.007843983359634876,
+            0.020962223410606384,
+            -0.022512484341859818,
+            0.023220213130116463,
+            0.011921843513846397,
+            3.429373100516386e-05,
+            0.014676926657557487,
+            -0.023186512291431427,
+            -0.03011213429272175,
+            0.02743287943303585,
+            -0.011610106565058231,
+            -0.02123183384537697,
+            0.015359378419816494,
+            -0.013615335337817669,
+            -0.012814928777515888,
+            -0.02278209663927555,
+            -0.0092425886541605,
+            0.004014670383185148,
+            -0.006554907653480768,
+            -0.016252463683485985,
+            -0.0033427502494305372,
+            0.021518295630812645,
+            0.01806391030550003,
+            0.022310277447104454,
+            -0.00462129432708025,
+            0.012873905710875988,
+            0.001359637826681137,
+            0.019563619047403336,
+            0.019883781671524048,
+            -0.00939424429088831,
+            -0.0029467595741152763,
+            0.013488955795764923,
+            -0.007363739423453808,
+            0.022192321717739105,
+            0.0008404267719015479,
+            -0.02950550988316536,
+            -0.010969781316816807,
+            -0.03690294921398163,
+            0.007911385968327522,
+            -0.00792823638767004,
+            -0.006487505044788122,
+            0.05395582318305969,
+            0.006175767630338669,
+            -0.01248634047806263,
+            0.007704965304583311,
+            0.0054554021917283535,
+            -0.005341660231351852,
+            -0.00786925945430994,
+            0.017094997689127922,
+            -0.01380911748856306,
+            -0.015266700647771358,
+            -0.013118240982294083,
+            -0.010843401774764061,
+            0.009714406915009022,
+            0.020642060786485672,
+            -0.0033617070876061916,
+            -0.026169078424572945,
+            -0.013210919685661793,
+            -0.020305048674345016,
+            -0.013151941820979118,
+            0.007886109873652458,
+            -0.0042716427706182,
+            -0.0007840823964215815,
+            0.037071458995342255,
+            -0.019496217370033264,
+            -0.013404701836407185,
+            -0.010683320462703705,
+            -0.002651873044669628,
+            -0.01658947765827179,
+            -0.005939858499914408,
+            0.016201911494135857,
+            -0.016867512837052345,
+            -0.0034480667673051357,
+            -0.0056533971801400185,
+            -0.010245203040540218,
+            0.020187092944979668,
+            0.008522222749888897,
+            -0.00668971287086606,
+            0.01156798005104065,
+            0.04886692017316818,
+            -0.03798139467835426,
+            -0.01603340543806553,
+            -0.02515803836286068,
+            0.026489241048693657,
+            0.009259439073503017,
+            0.01670743152499199,
+            -0.02092852257192135,
+            -0.028494469821453094,
+            -0.023725733160972595,
+            -0.012292558327317238,
+            -0.019732125103473663,
+            -0.003188987961038947,
+            0.023102257400751114,
+            0.014803307130932808,
+            0.04283438250422478,
+            0.00532902218401432,
+            0.0035428518895059824,
+            -0.0028772505465894938,
+            0.0022474571596831083,
+            0.011197265237569809,
+            -0.02178790606558323,
+            0.0026834679301828146,
+            0.00024749411386437714,
+            0.026371287181973457,
+            0.01105403434485197,
+            -0.006024111993610859,
+            -0.028865184634923935,
+            0.0021610974799841642,
+            0.012418937869369984,
+            -0.007549096830189228,
+            -0.017069721594452858,
+            0.027837295085191727,
+            0.004549678880721331,
+            -0.0034017274156212807,
+            -0.011281518265604973,
+            0.028511321172118187,
+            0.0027550833765417337,
+            0.028123755007982254,
+            -0.020406153053045273,
+            -0.009040380828082561,
+            -0.022798946127295494,
+            0.009436370804905891,
+            -0.009107783436775208,
+            0.002430708147585392,
+            -0.01025362778455019,
+            0.015696391463279724,
+            -0.01508976798504591,
+            -0.010245203040540218,
+            0.030465997755527496,
+            -0.021198133006691933,
+            -0.0025739388074725866,
+            -0.015578437596559525,
+            -0.028022652491927147,
+            0.00967228040099144,
+            -0.0011005589039996266,
+            0.007852409034967422,
+            -0.006656011566519737,
+            0.003707145806401968,
+            -0.023034855723381042,
+            0.009166760370135307,
+            -0.00963857863098383,
+            -0.011222541332244873,
+            -0.001816712087020278,
+            0.013649036176502705,
+            0.0008383204694837332,
+            0.010337881743907928,
+            0.005034135654568672,
+            -0.03892502933740616,
+            0.01836722157895565,
+            0.0036776571068912745,
+            0.010278903879225254,
+            0.003707145806401968,
+            0.008155720308423042,
+            0.007924024015665054,
+            0.029337003827095032,
+            0.00982393603771925,
+            -0.009874488227069378,
+            -0.011854440905153751,
+            -0.0011216222774237394,
+            0.011163563467562199,
+            0.018653683364391327,
+            -0.004899329971522093,
+            -0.01616821065545082,
+            0.006399039179086685,
+            3.333929998916574e-05,
+            0.03491457179188728,
+            0.008509584702551365,
+            -0.020372450351715088,
+            -0.005337447393685579,
+            -0.04917023330926895,
+            0.020490406081080437,
+            -0.015376229770481586,
+            -0.0339372344315052,
+            0.01075072307139635,
+            -0.0043074507266283035,
+            -0.022428231313824654,
+            0.029353853315114975,
+            -0.004945669323205948,
+            -0.013219344429671764,
+            0.007001450285315514,
+            -0.009343692101538181,
+            -0.006879283115267754,
+            0.0014428379945456982,
+            -0.016623178496956825,
+            6.6851384872279596e-06,
+            0.00356602156534791,
+            0.019176054745912552,
+            -0.0034775554668158293,
+            -0.01549418456852436,
+            0.001500762184150517,
+            0.00623053265735507,
+            0.017861701548099518,
+            0.0021221302449703217,
+            0.003989394288510084,
+            0.0006424314924515784,
+            0.014828583225607872,
+            -0.02664089761674404,
+            0.005366935860365629,
+            -0.0012501084711402655,
+            -0.026792554184794426,
+            0.004553891718387604,
+            -0.006024111993610859,
+            -0.01983322948217392,
+            0.012494766153395176,
+            -0.0009241535444743931,
+            -0.0016408332157880068,
+            -0.00585981784388423,
+            -0.009478497318923473,
+            -0.009697556495666504,
+            -0.008972978219389915,
+            0.008956126868724823,
+            0.016016554087400436,
+            0.007304762024432421,
+            -0.010118822567164898,
+            -0.02431550621986389,
+            0.001289075706154108,
+            0.02529284358024597,
+            0.02357407659292221,
+            -0.035285286605358124,
+            0.014735904522240162,
+            0.005935645662248135,
+            -0.00851379707455635,
+            -0.027163267135620117,
+            0.0310389194637537,
+            0.02197326347231865,
+            0.0038061433006078005,
+            0.007397440727800131,
+            0.01162695698440075,
+            0.007359527051448822,
+            -0.03949795290827751,
+            0.011079310439527035,
+            -0.011020333506166935,
+            -0.009849212132394314,
+            -3.0377268558368087e-05,
+            0.009031955152750015,
+            -0.009882913902401924,
+            -0.011803888715803623,
+            0.007481694221496582,
+            0.02204066701233387,
+            -0.009368968196213245,
+            -0.003660806454718113,
+            0.01678325980901718,
+            0.002243244554847479,
+            0.001279597170650959,
+            0.016808535903692245,
+            0.05621381103992462,
+            0.003601829055696726,
+            -0.0019662617705762386,
+            0.02192271128296852,
+            -0.0018651577411219478,
+            -0.022428231313824654,
+            -0.006196831353008747,
+            -0.01792910508811474,
+            0.008105169050395489,
+            0.010304179973900318,
+            -0.01456739753484726,
+            -0.005088900215923786,
+            -0.0035786593798547983,
+            0.009048805572092533,
+            -0.0046507827937603,
+            0.017828000709414482,
+            -0.00638218829408288,
+            0.01909179985523224,
+            0.03152759000658989,
+            0.01407872885465622,
+            -0.013345724903047085,
+            0.014162981882691383,
+            -0.007473269011825323,
+            -0.0033954083919525146,
+            -0.010666469112038612,
+            0.001936773071065545,
+            0.0015576331643387675,
+            0.005000433884561062,
+            0.0009467966156080365,
+            0.025680409744381905,
+            -0.0012048223288729787,
+            -0.01978267729282379,
+            -0.005805053282529116,
+            -0.0057334378361701965,
+            -0.01224200613796711,
+            0.007755517493933439,
+            0.0014059771783649921,
+            0.0034480667673051357,
+            -0.02081056870520115,
+            0.0031005218625068665,
+            0.005261619575321674,
+            0.005927220452576876,
+            0.006112577859312296,
+            0.0030268002301454544,
+            -0.0038166751619428396,
+            0.02013654261827469,
+            -0.00623053265735507,
+            0.001733511919155717,
+            0.013151941820979118,
+            -0.006415889598429203,
+            -0.010877102613449097,
+            -0.013143517076969147,
+            -0.0007919811177998781,
+            0.019698424264788628,
+            -0.014719054102897644,
+            -0.00436432147398591,
+            0.013556358404457569,
+            0.00863596424460411,
+            0.006710776127874851,
+            -0.0019072843715548515,
+            -0.00604517525061965,
+            0.0026876807678490877,
+            -0.02343927137553692,
+            -0.020557807758450508,
+            -0.008905574679374695,
+            -0.008496946655213833,
+            0.024264954030513763,
+            -0.006681287661194801,
+            0.005956709384918213,
+            -0.00756173487752676,
+            0.006466441787779331,
+            0.01824926771223545,
+            0.005413275212049484,
+            -0.003639743197709322,
+            0.006757115479558706,
+            -0.00021734721667598933,
+            0.002392794005572796,
+            -0.009318416006863117,
+            0.0031658182851970196,
+            0.015291975811123848,
+            -0.010995057411491871,
+            0.0022263936698436737,
+            -0.007738666608929634,
+            -0.0036629128735512495,
+            -0.022866349667310715,
+            0.010430560447275639,
+            -0.001800914527848363,
+            0.009613302536308765,
+            0.007873471826314926,
+            -0.008058829233050346,
+            0.012789652682840824,
+            -0.008694942109286785,
+            0.015713242813944817,
+            -0.012250431813299656,
+            -0.0004007298266515136,
+            0.03017953597009182,
+            -0.011155138723552227,
+            -0.019007546827197075,
+            -0.0034733428619802,
+            -0.016176635399460793,
+            0.02308540791273117,
+            0.027230670675635338,
+            0.015435206703841686,
+            0.016075532883405685,
+            -0.007456418126821518,
+            0.0066517991945147514,
+            0.025680409744381905,
+            0.003513363189995289,
+            0.020962223410606384,
+            -0.01162695698440075,
+            0.011803888715803623,
+            -0.03017953597009182,
+            0.0003238486824557185,
+            -0.0029994179494678974,
+            -0.004992008674889803,
+            0.007843983359634876,
+            -0.02131608873605728,
+            -0.008711792528629303,
+            -0.004992008674889803,
+            -0.00719102006405592,
+            0.0041389442048966885,
+            -0.019732125103473663,
+            0.00952062476426363,
+            -0.018451474606990814,
+            -0.01658947765827179,
+            -0.012006097473204136,
+            -0.01806391030550003,
+            -0.013480530120432377,
+            -0.01721295155584812,
+            -0.030819861218333244,
+            -0.00356602156534791,
+            -0.02062521129846573,
+            -0.025073785334825516,
+            0.018721086904406548,
+            -0.023860538378357887,
+            0.01199767179787159,
+            -0.0008514850633218884,
+            -0.006458016578108072,
+            0.025680409744381905,
+            0.029471809044480324,
+            -0.009866062551736832,
+            -0.023708881810307503,
+            -0.009655429981648922,
+            -0.029117945581674576,
+            0.009714406915009022,
+            0.011028758250176907,
+            0.0092425886541605,
+            -0.01090237870812416,
+            0.0015397293027490377,
+            -0.014516845345497131,
+            0.01732248067855835,
+            -0.032117363065481186,
+            -0.029522361233830452,
+            -0.008029340766370296,
+            -0.006879283115267754,
+            -0.001448103808797896,
+            -0.015350953675806522,
+            0.005969346966594458,
+            -0.018080761656165123,
+            -0.02748342975974083,
+            -0.0030331192538142204,
+            -0.006163129583001137,
+            -0.01304241269826889,
+            0.005219492595642805,
+            -0.00557335652410984,
+            0.022394530475139618,
+            0.010666469112038612,
+            0.015780646353960037,
+            -0.004848778247833252,
+            -0.009192036464810371,
+            -0.014306212775409222,
+            -0.01573851890861988,
+            0.004777162801474333,
+            0.009748108685016632,
+            0.012621146626770496,
+            0.0004154741473030299,
+            0.010076696053147316,
+            0.000991029548458755,
+            0.013210919685661793,
+            0.010270479135215282,
+            0.010910804383456707,
+            0.021164432168006897,
+            -0.0023801561910659075,
+            0.014188257977366447,
+            0.00402730843052268,
+            0.005459614563733339,
+            -0.0029383343644440174,
+            -0.008357929065823555,
+            0.022360829636454582,
+            -0.007658625952899456,
+            -0.013960774056613445,
+            0.029337003827095032,
+            0.004697122145444155,
+            -0.02320336177945137,
+            0.012629571370780468,
+            0.0003151600540149957,
+            -0.018721086904406548,
+            -0.008652815595269203,
+            -0.0009815511293709278,
+            -0.004473851062357426,
+            -0.00765441358089447,
+            0.0014112429926171899,
+            -0.008686516433954239,
+            0.01621876284480095,
+            0.016875937581062317,
+            -0.008930850774049759,
+            -0.008429544046521187,
+            0.017844852060079575,
+            0.016614753752946854,
+            -0.001709289033897221,
+            -0.022209173068404198,
+            -0.008694942109286785,
+            0.02504008449614048,
+            -0.03629632666707039,
+            -0.026893656700849533,
+            0.007700752932578325,
+            0.006420102436095476,
+            -0.009756533429026604,
+            6.574062354047783e-06,
+            0.003983075264841318,
+            -0.007578585296869278,
+            0.0032479651272296906,
+            0.0016208230517804623,
+            0.012680123560130596,
+            0.01484543364495039,
+            -0.011096160858869553,
+            0.010556939989328384,
+            -0.01573851890861988,
+            0.0018672640435397625,
+            -0.020035438239574432,
+            -0.013126665726304054,
+            -0.01380069274455309,
+            0.010371582582592964,
+            0.020591510459780693,
+            0.011829164810478687,
+            0.03889133036136627,
+            0.01466007623821497,
+            0.004014670383185148,
+            -0.006917196791619062,
+            -0.017288779839873314,
+            0.004802438896149397,
+            -0.002948865992948413,
+            -0.0037766548339277506,
+            -0.0021158112213015556,
+            0.014912836253643036,
+            -0.0021779481321573257,
+            0.028123755007982254,
+            0.006158917210996151,
+            0.01236838661134243,
+            -2.6213969249511138e-05,
+            -0.007022513542324305,
+            -0.003983075264841318,
+            0.013101390562951565,
+            5.410642552305944e-05,
+            -0.0023675181437283754,
+            0.0067192018032073975,
+            -0.0018483070889487863,
+            -0.005055198911577463,
+            0.0027656150050461292,
+            -0.011365772224962711,
+            -0.0055101667530834675,
+            -0.017356181517243385,
+            -0.008787620812654495,
+            -0.009175186045467854,
+            0.02485472708940506,
+            0.028865184634923935,
+            -0.015013940632343292,
+            -0.010430560447275639,
+            -0.013455254025757313,
+            0.03589191287755966,
+            -0.004840353038161993,
+            0.0034291099291294813,
+            0.014811732806265354,
+            -0.008753919042646885,
+            0.0024222827050834894,
+            0.012418937869369984,
+            0.0042210910469293594,
+            0.003555489704012871,
+            0.029926776885986328,
+            -0.0031721373088657856,
+            -0.01013567391782999,
+            -0.0035028315614908934,
+            0.003907247446477413,
+            -0.026101676747202873,
+            0.0003712411562446505,
+            0.004840353038161993,
+            -0.010514813475310802,
+            0.013092964887619019,
+            -0.019513066858053207,
+            0.006795029621571302,
+            -0.010632768273353577,
+            -0.0008367407135665417,
+            -0.01976582780480385,
+            -0.0007946140831336379,
+            0.007633350323885679,
+            -0.012587444856762886,
+            -0.003197413170710206,
+            -0.017356181517243385,
+            -0.0020062820985913277,
+            0.014643225818872452,
+            0.007001450285315514,
+            0.006040962412953377,
+            -0.0007472215802408755,
+            0.0032016257755458355,
+            0.022478783503174782,
+            0.0043474710546433926,
+            0.004718185402452946,
+            -0.004541253671050072,
+            0.0042526861652731895,
+            -0.024264954030513763,
+            0.006453803740441799,
+            -0.005762926768511534,
+            -0.0026497666258364916,
+            0.024955831468105316,
+            -0.005842967424541712,
+            0.0009678599308244884,
+            -0.006238957867026329,
+            0.005059411283582449,
+            -0.024568265303969383,
+            0.01508976798504591,
+            -0.01762579381465912,
+            0.010700170882046223,
+            -0.008564349263906479,
+            -0.002706637606024742,
+            -0.00251074880361557,
+            0.009444796480238438,
+            0.012292558327317238,
+            -0.013084539212286472,
+            0.007397440727800131,
+            -0.015822771936655045,
+            0.021686803549528122,
+            -0.003867227118462324,
+            -0.01878848858177662,
+            0.0018314564367756248,
+            -0.008046191185712814,
+            0.008762344717979431,
+            -0.014702202752232552,
+            0.0010515867033973336,
+            0.010396858677268028,
+            -0.007216296158730984,
+            0.005809266120195389,
+            -0.023119108751416206,
+            0.006828730925917625,
+            -0.017490988597273827,
+            0.008796045556664467,
+            -0.012966584414243698,
+            -0.012966584414243698,
+            -0.02118128351867199,
+            -0.012006097473204136,
+            -0.00024551941896788776,
+            0.0051436647772789,
+            0.015713242813944817,
+            -0.006934047676622868,
+            0.005838754586875439,
+            0.021872159093618393,
+            -0.010026143863797188,
+            0.01719610020518303,
+            0.053113289177417755,
+            -0.025444500148296356,
+            -0.016934916377067566,
+            0.00136279733851552,
+            0.010784423910081387,
+            -0.03434165194630623,
+            -0.008581199683248997,
+            0.019395112991333008,
+            -0.016361992806196213,
+            0.029606614261865616,
+            -0.025663558393716812,
+            0.01156798005104065,
+            -0.010708595626056194,
+            0.00796193815767765,
+            -0.017052870243787766,
+            -0.01929400861263275,
+            -0.00991661474108696,
+            -0.003568127751350403,
+            -0.0018462006701156497,
+            -0.02394479140639305,
+            0.009495348669588566,
+            -0.014390465803444386,
+            -0.011045609600841999,
+            0.006150492001324892,
+            -0.0057882023975253105,
+            0.007460630964487791,
+            0.0016608433797955513,
+            0.013758566230535507,
+            -0.020776866003870964,
+            0.0008767610415816307,
+            -0.011045609600841999,
+            0.004747674334794283,
+            -0.0067739663645625114,
+            0.011298369616270065,
+            0.01337942574173212,
+            -0.013092964887619019,
+            -0.002439133357256651,
+            -0.002023132750764489,
+            0.015511034987866879,
+            -0.006504355929791927,
+            0.03683554753661156,
+            -0.008075679652392864,
+            -0.004339045379310846,
+            -0.008871873840689659,
+            -0.0056870984844863415,
+            0.011829164810478687,
+            0.01328674703836441,
+            0.008568561635911465,
+            0.005834541749209166,
+            0.00914990995079279,
+            -0.014112429693341255,
+            0.01536780409514904,
+            0.007949300110340118,
+            0.017659494653344154,
+            0.005307958461344242,
+            -0.013851244933903217,
+            -0.024837875738739967,
+            -0.0066855004988610744,
+            -0.007262635510414839,
+            -0.0029446533881127834,
+            -0.01665687933564186,
+            0.0035512770991772413,
+            0.0067781792022287846,
+            0.007827132940292358,
+            -0.003900928422808647,
+            0.03257232904434204,
+            0.005729225464165211,
+            0.0012753845658153296,
+            0.027635086327791214,
+            0.007793431635946035,
+            0.0015112938126549125,
+            0.017524689435958862,
+            -0.031072622165083885,
+            -0.01404502708464861,
+            -0.021889010444283485,
+            0.011761762201786041,
+            0.0004615501675289124,
+            -0.0318477526307106,
+            -0.0024833662901073694,
+            0.0011121437419205904,
+            -0.01025362778455019,
+            0.012629571370780468,
+            0.0206926129758358,
+            -0.016378844156861305,
+            -0.011947119608521461,
+            0.008037766441702843,
+            0.011070885695517063,
+            -0.005190004128962755,
+            0.027635086327791214,
+            -0.002001016167923808,
+            0.009630153886973858,
+            0.007405866403132677,
+            0.019428813830018044,
+            0.0006540163303725421,
+            0.023523524403572083,
+            -0.006643373519182205,
+            -0.00483614020049572,
+            -0.020490406081080437,
+            -0.008412693627178669,
+            -0.01175333745777607,
+            0.011610106565058231,
+            -0.005206854548305273,
+            0.003789292648434639,
+            -0.013952348381280899,
+            0.007915598340332508,
+            0.014963388442993164,
+            0.011028758250176907,
+            0.0044443621300160885,
+            0.012124051339924335,
+            0.010455836541950703,
+            0.013590059243142605,
+            0.0012353642378002405,
+            -0.020338749513030052,
+            0.008914000354707241,
+            0.017608942463994026,
+            -0.006934047676622868,
+            0.02141719125211239,
+            0.023540375754237175,
+            0.001537623000331223,
+            -0.003188987961038947,
+            0.008518009446561337,
+            0.01652207411825657,
+            0.01976582780480385,
+            -0.01298343576490879,
+            -0.0032332208938896656,
+            0.0036629128735512495,
+            -0.0028983140364289284,
+            -0.0016408332157880068,
+            -0.019327709451317787,
+            -0.000656649237498641,
+            0.008614901453256607,
+            0.0010921335779130459,
+            0.030230088159441948,
+            -0.01823241636157036,
+            -0.01428936142474413,
+            0.01915920339524746,
+            0.006622310262173414,
+            -0.01880533993244171,
+            0.012503191828727722,
+            0.00015968635852914304,
+            0.013960774056613445,
+            -0.011104586534202099,
+            0.005215280223637819,
+            -0.01004299521446228,
+            -0.004503339529037476,
+            0.017524689435958862,
+            -0.009267864748835564,
+            0.011778613552451134,
+            -0.008383204229176044,
+            -0.001989431446418166,
+            -0.02074316516518593,
+            0.02032189816236496,
+            0.02613537758588791,
+            0.01086867693811655,
+            0.008412693627178669,
+            0.004751886706799269,
+            0.007271060720086098,
+            0.00436432147398591,
+            0.01927715726196766,
+            0.020658912137150764,
+            -0.011424749158322811,
+            0.007494332268834114,
+            0.018148163333535194,
+            -0.007249997463077307,
+            0.012840204872190952,
+            0.004827714990824461,
+            0.004970945417881012,
+            -0.007094128988683224,
+            0.006015686318278313,
+            -0.017162399366497993,
+            -0.013033987022936344,
+            0.0012627465184777975,
+            0.032606031745672226,
+            0.008454820141196251,
+            -0.023304466158151627,
+            0.001874636160209775,
+            0.024349207058548927,
+            -0.020894821733236313,
+            -0.0034754490479826927,
+            0.008315801620483398,
+            -0.0019157096976414323,
+            0.012899181805551052,
+            -0.02461881749331951,
+            -0.00786925945430994,
+            0.004810864105820656,
+            -0.006348486989736557,
+            0.016193486750125885,
+            -0.012098775245249271,
+            -0.0390598364174366,
+            -0.01199767179787159,
+            -0.020153392106294632,
+            -0.017861701548099518,
+            -0.019664723426103592,
+            0.0008930850890465081,
+            0.01585647277534008,
+            -0.019024398177862167,
+            -0.018333520740270615,
+            0.016741132363677025,
+            -0.001975740073248744,
+            0.03764437884092331,
+            -0.017373032867908478,
+            0.007536458782851696,
+            -0.01727192848920822,
+            0.01646309718489647,
+            -0.016800111159682274,
+            -0.018502026796340942,
+            0.01475275494158268,
+            -0.0031679244711995125,
+            -0.003721890039741993,
+            0.0023422420490533113,
+            0.00235698651522398,
+            -0.009714406915009022,
+            -0.003024693811312318,
+            0.011652233079075813,
+            0.0006982493214309216,
+            -0.0161008071154356,
+            -0.002645554021000862,
+            0.003528107423335314,
+            0.012882331386208534,
+            0.01578907109797001,
+            -0.0069635361433029175,
+            -0.003808249719440937,
+            0.010211501270532608,
+            -0.01762579381465912,
+            -0.010363157838582993,
+            0.015300401486456394,
+            -0.01732248067855835,
+            0.03191515430808067,
+            0.006592821795493364,
+            -0.01776059903204441,
+            -0.00716153159737587,
+            0.003629211336374283,
+            0.023287614807486534,
+            0.01332044880837202,
+            -0.020726315677165985,
+            -0.008004064671695232,
+            0.007810282055288553,
+            0.012739100493490696,
+            0.005438551306724548,
+            -0.007186807692050934,
+            0.027820443734526634,
+            -0.0007530139992013574,
+            0.00015547369548585266,
+            0.001172174233943224,
+            -0.01056536566466093,
+            -0.0021737355273216963,
+            0.008804471231997013,
+            0.015839623287320137,
+            0.004292706493288279,
+            0.00985763780772686,
+            0.005333234556019306,
+            -0.01635356806218624,
+            0.015713242813944817,
+            0.0068076676689088345,
+            -0.0035449580755084753,
+            -0.016370419412851334,
+            0.011273093521595001,
+            0.007119405083358288,
+            0.016513649374246597,
+            -0.012772802263498306,
+            -0.004882479552179575,
+            -1.0696221579564735e-05,
+            0.00737637747079134,
+            0.0022221810650080442,
+            0.008067254908382893,
+            0.0010705436579883099,
+            0.02938755601644516,
+            0.03282508999109268,
+            0.03312840312719345,
+            -0.025326546281576157,
+            -0.010321030393242836,
+            0.008231548592448235,
+            -0.003157392842695117,
+            -0.004654995631426573,
+            -0.0003354334912728518,
+            -0.01720452681183815,
+            0.006971961818635464,
+            -0.02620278112590313,
+            0.01199767179787159,
+            -0.001448103808797896,
+            0.005147877614945173,
+            -0.010860252194106579,
+            0.0066517991945147514,
+            -0.0013680631527677178,
+            0.0001929927384480834,
+            -0.011424749158322811,
+            -0.019209755584597588,
+            -0.0014586354373022914,
+            0.014002900570631027,
+            -0.0012374705402180552,
+            -0.008122019469738007,
+            0.014061877503991127,
+            -0.006057813297957182,
+            -0.023237064480781555,
+            0.006087301764637232,
+            -0.028882035985589027,
+            0.0021010669879615307,
+            -0.013092964887619019,
+            0.003909353632479906,
+            -0.022680992260575294,
+            0.013421552255749702,
+            -0.0013554252218455076,
+            -0.01267169788479805,
+            0.004941456951200962,
+            -0.017288779839873314,
+            0.0025676197838038206,
+            -0.003197413170710206,
+            0.006272659171372652,
+            -0.010708595626056194,
+            0.028376515954732895,
+            0.014575823210179806,
+            -0.015140320174396038,
+            -0.017052870243787766,
+            0.008446394465863705,
+            -0.02809005416929722,
+            0.021703653037548065,
+            0.010885528288781643,
+            -0.0034733428619802,
+            -0.02406274527311325,
+            0.009748108685016632,
+            0.0036966141778975725,
+            0.02050725556910038,
+            0.010860252194106579,
+            0.009293140843510628,
+            0.018822189420461655,
+            -0.015527885407209396,
+            0.0021800545509904623,
+            -1.19962232929538e-05,
+            -0.00251074880361557,
+            0.014921261928975582,
+            -0.008071467280387878,
+            0.014584247954189777,
+            0.007359527051448822,
+            -0.0044232988730072975,
+            -0.006268446333706379,
+            -0.0030520763248205185,
+            -0.015165596269071102,
+            0.0004289020143914968,
+            -0.0020241858437657356,
+            -0.0008130444912239909,
+            0.006647586356848478,
+            0.010110397823154926,
+            -0.017305631190538406,
+            0.00914990995079279,
+            -0.0012248324928805232,
+            -0.025848915800452232,
+            0.0014038708759471774,
+            -0.016741132363677025,
+            0.002437027171254158,
+            -0.021383490413427353,
+            0.01252004224807024,
+            0.025208590552210808,
+            0.011037183925509453,
+            -0.007073065731674433,
+            -0.006234745029360056,
+            -0.011879716999828815,
+            -0.007485906593501568,
+            -0.010514813475310802,
+            -0.006125215906649828,
+            -0.019209755584597588,
+            -0.003728209063410759,
+            0.011121436953544617,
+            0.011728061363101006,
+            0.00034227909054607153,
+            -0.018350372090935707,
+            -0.005135239567607641,
+            -0.013109815306961536,
+            -0.005400637164711952,
+            -0.008547498844563961,
+            -0.013084539212286472,
+            0.0011974502122029662,
+            0.012545318342745304,
+            -0.011669083498418331,
+            0.029421256855130196,
+            0.005779777187854052,
+            0.006243170239031315,
+            -0.0022158620413392782,
+            -0.007881897501647472,
+            -0.004162113647907972,
+            -0.012216730043292046,
+            -0.0022074368316680193,
+            -0.005341660231351852,
+            -0.0020757908932864666,
+            -0.00905723124742508,
+            0.005813478492200375,
+            -0.0015913344686850905,
+            0.01555316150188446,
+            0.006062025669962168,
+            -0.008126231841742992,
+            -0.002955185016617179,
+            0.031679246574640274,
+            -0.007140468340367079,
+            -0.016614753752946854,
+            -0.00037861333112232387,
+            -0.00638218829408288,
+            0.0037303154822438955,
+            0.007414291612803936,
+            -0.01769319549202919,
+            0.0043895975686609745,
+            0.010851826518774033,
+            0.004794013686478138,
+            -0.0033827705774456263,
+            0.02081056870520115,
+            0.008779195137321949,
+            -0.002984673483297229,
+            0.010329456068575382,
+            0.0032269018702208996,
+            -0.010068271309137344,
+            -0.0044232988730072975,
+            -0.0026476604398339987,
+            0.028814632445573807,
+            -0.03484717011451721,
+            -0.0020547276362776756,
+            0.019732125103473663,
+            0.0044780634343624115,
+            -0.021147580817341805,
+            -0.002312753349542618,
+            -0.026977911591529846,
+            -0.0020673656836152077,
+            -0.009891338646411896,
+            0.013404701836407185,
+            -0.036026716232299805,
+            -0.002512854989618063,
+            -0.00905723124742508,
+            0.0044991266913712025,
+            -0.006373763084411621,
+            0.026893656700849533,
+            -0.0008936116937547922,
+            -0.005505953915417194,
+            0.011011907830834389,
+            0.009689130820333958,
+            0.011947119608521461,
+            0.026590345427393913,
+            -0.006934047676622868,
+            -0.0067529031075537205,
+            0.003083671210333705,
+            -0.028073204681277275,
+            0.02276524528861046,
+            0.00362710515037179,
+            0.0032669221982359886,
+            -0.0004115247866138816,
+            -0.006740265060216188,
+            0.02625333145260811,
+            0.02866297774016857,
+            -0.005223705433309078,
+            -0.005282682832330465,
+            -0.016387268900871277,
+            0.029101094231009483,
+            -0.011441599577665329,
+            0.0011205690680071712,
+            -0.002015760401263833,
+            -0.0121493274345994,
+            -0.01732248067855835,
+            -0.013446828350424767,
+            0.019984886050224304,
+            -0.02008599042892456,
+            -0.0017724790377542377,
+            -0.012882331386208534,
+            -0.001390179619193077,
+            -0.02497268095612526,
+            -0.008467458188533783,
+            0.008572774939239025,
+            -0.014070303179323673,
+            -0.0032416461035609245,
+            0.0003148967516608536,
+            -0.006436952855437994,
+            0.0011311006965115666,
+            -0.0067866044119000435,
+            -0.00031437017605639994,
+            -0.003799824509769678,
+            0.011129862628877163,
+            0.0002694789436645806,
+            -0.008855023421347141,
+            0.019496217370033264,
+            -0.010919229127466679,
+            0.006795029621571302,
+            0.0043221949599683285,
+            -0.004010457545518875,
+            -0.0003080511814914644,
+            0.009697556495666504,
+            0.014188257977366447,
+            -0.013708014041185379,
+            0.021939562633633614,
+            0.009377393871545792,
+            -0.01484543364495039,
+            0.010405284352600574,
+            -0.005354297813028097,
+            -0.019866930320858955,
+            -0.0026687236968427896,
+            0.015022365376353264,
+            0.019850080832839012,
+            -0.022445082664489746,
+            0.0011195158585906029,
+            0.01873793639242649,
+            -0.00021194973669480532,
+            0.0021379278041422367,
+            0.007338463328778744,
+            0.006142066325992346,
+            0.007485906593501568,
+            0.010026143863797188,
+            0.009739683009684086,
+            -0.0037576977629214525,
+            -0.0206926129758358,
+            0.015199298039078712,
+            -0.012140902690589428,
+            0.01561213843524456,
+            0.023776285350322723,
+            -0.010607492178678513,
+            -0.021467743441462517,
+            0.015106619335711002,
+            -0.007186807692050934,
+            -0.02827541157603264,
+            -0.0015039215795695782,
+            -0.0017292991979047656,
+            0.01978267729282379,
+            0.0030815647915005684,
+            -2.3564598450320773e-05,
+            -0.018282968550920486,
+            -1.969750337593723e-05,
+            0.01080970000475645,
+            -0.0017524688737466931,
+            -0.0411493182182312,
+            -0.006217894610017538,
+            0.004263217560946941,
+            -0.00486562866717577,
+            0.014255660586059093,
+            0.0017651069210842252,
+            0.005527017172425985,
+            0.026000572368502617,
+            -0.019698424264788628,
+            -0.010118822567164898,
+            0.006521206349134445,
+            0.016370419412851334,
+            0.010152524337172508,
+            0.011104586534202099,
+            0.006508568301796913,
+            0.015224573202431202,
+            -0.012039798311889172,
+            -0.007102554198354483,
+            0.018266119062900543,
+            -0.013387851417064667,
+            0.014811732806265354,
+            -0.01811446249485016,
+            0.004461213015019894,
+            -0.006310573313385248,
+            0.004739248659461737,
+            0.010969781316816807,
+            -0.00768390204757452,
+            -0.0010357891442254186,
+            -0.040744900703430176,
+            0.004570742137730122,
+            -0.0027213820721954107,
+            0.014710628427565098,
+            -0.01105403434485197,
+            0.004608656279742718,
+            -0.0001983902184292674,
+            0.005834541749209166,
+            0.009689130820333958,
+            -0.0008893990307115018,
+            0.01615135930478573,
+            -0.02601742371916771,
+            -0.025141188874840736,
+            7.082543743308634e-05,
+            -0.006917196791619062,
+            0.008762344717979431,
+            0.020423002541065216,
+            0.012444213964045048,
+            0.023489823564887047,
+            -0.01205664873123169,
+            0.008509584702551365,
+            0.011188839562237263,
+            -0.016875937581062317,
+            -0.003881971351802349,
+            0.0011953439097851515,
+            -0.008400055579841137,
+            -0.02332131750881672,
+            -0.0018177651800215244,
+            -0.004671846050769091,
+            0.008623326197266579,
+            0.0035639151465147734,
+            0.006377975922077894,
+            -0.025427650660276413,
+            0.004160007461905479,
+            0.004265323746949434,
+            -0.0298762246966362,
+            0.00018074968829751015,
+            -0.017423585057258606,
+            0.008627539500594139,
+            -0.0016650561010465026,
+            0.021282386034727097,
+            -0.013126665726304054,
+            -0.022849498316645622,
+            -0.005716587416827679,
+            0.005425913259387016,
+            -0.007389015518128872,
+            -0.01025362778455019,
+            0.015814347192645073,
+            -0.017069721594452858,
+            -0.008008277043700218,
+            -0.021585699170827866,
+            -0.006601247005164623,
+            -0.036026716232299805,
+            -0.006344274617731571,
+            0.01056536566466093,
+            0.005548080429434776,
+            -0.010068271309137344,
+            -0.029657166451215744,
+            0.005889306776225567,
+            -0.0067781792022287846,
+            -0.004549678880721331,
+            -0.01096135564148426,
+            0.0014923367416486144,
+            0.022006966173648834,
+            -0.006411677226424217,
+            0.02148459479212761,
+            -0.0174404364079237,
+            -0.004705547355115414,
+            -0.017061294987797737,
+            0.0044570001773536205,
+            -0.0022179684601724148,
+            -0.004583380185067654,
+            -0.00666022440418601,
+            -0.011736486107110977,
+            0.0024875791277736425,
+            0.002984673483297229,
+            -0.007435354869812727,
+            -0.007532245945185423,
+            0.012267282232642174,
+            0.009141484275460243,
+            0.007831345312297344,
+            0.007843983359634876,
+            0.004840353038161993,
+            -0.012621146626770496,
+            -0.00032700816518627107,
+            -0.006024111993610859,
+            0.005194216500967741,
+            0.009933465160429478,
+            0.0057207997888326645,
+            0.005004646722227335,
+            -0.008897149935364723,
+            0.012081924825906754,
+            0.01899069733917713,
+            0.033263206481933594,
+            -0.01304241269826889,
+            -0.005168940871953964,
+            -0.014011326245963573,
+            0.02166995219886303,
+            0.019024398177862167,
+            0.003317474154755473,
+            0.005088900215923786,
+            0.006925622466951609,
+            -0.0019315071403980255,
+            0.0013238302199169993,
+            0.012098775245249271,
+            -0.00872021820396185,
+            -0.009293140843510628,
+            -0.011323644779622555,
+            0.0013733289670199156,
+            -0.01781114935874939,
+            0.008058829233050346,
+            0.006441165693104267,
+            0.002065259264782071,
+            0.01025362778455019,
+            0.03936314582824707,
+            0.0026139591354876757,
+            -0.0016471522394567728,
+            -0.00238647498190403,
+            -0.01915920339524746,
+            0.02124868519604206,
+            -0.007915598340332508,
+            -0.039295744150877,
+            -0.0024960043374449015,
+            0.002679255325347185,
+            0.01347210444509983,
+            -0.0079324496909976,
+            -0.000739322800654918,
+            -0.011787038296461105,
+            -0.02074316516518593,
+            0.0014459975063800812,
+            0.015039216727018356,
+            -0.0005660769529640675,
+            0.011955545283854008,
+            0.003568127751350403,
+            0.0026350223924964666,
+            0.0034164718817919493,
+            0.009537475183606148,
+            -0.02148459479212761,
+            0.002752976957708597,
+            0.007671264000236988,
+            0.028747230768203735,
+            0.0007682848954573274,
+            -0.007233147043734789,
+            0.017828000709414482,
+            -0.0034164718817919493,
+            -0.007157318759709597,
+            0.008041978813707829,
+            -0.011256242170929909,
+            0.010893953032791615,
+            -0.013143517076969147,
+            0.011795463971793652,
+            0.02290005050599575,
+            -0.0206926129758358,
+            0.005383786745369434,
+            -0.007325825747102499,
+            -0.003957799170166254,
+            0.015696391463279724,
+            0.020102839916944504,
+            0.006255808286368847,
+            0.0010484271915629506,
+            -0.02106332778930664,
+            0.017490988597273827,
+            -0.02081056870520115,
+            -0.0102873295545578,
+            0.005581781733781099,
+            -0.014794881455600262,
+            0.010978206992149353,
+            -0.0038988220039755106,
+            -0.026539793238043785,
+            0.003875652328133583,
+            0.009360543452203274,
+            0.015115044079720974,
+            0.006340061780065298,
+            0.005560718476772308,
+            0.0012048223288729787,
+            -0.012022947892546654,
+            -0.023708881810307503,
+            -0.0058219037018716335,
+            -0.019007546827197075,
+            -0.021889010444283485,
+            -0.006424315273761749,
+            0.007110979408025742,
+            -0.008058829233050346,
+            0.0018704235553741455,
+            0.023001154884696007,
+            -0.018535729497671127,
+            -0.0065675457008183,
+            0.022748393937945366,
+            0.003507044166326523,
+            0.013017136603593826,
+            -0.007460630964487791,
+            0.017423585057258606,
+            0.00954589992761612,
+            -0.012494766153395176,
+            -0.01959731988608837,
+            0.027045313268899918,
+            -0.00495830737054348,
+            0.007321612909436226,
+            0.008846597746014595,
+            0.0012195666786283255,
+            -0.004726610612124205,
+            0.00814308226108551,
+            -0.030465997755527496,
+            0.013151941820979118,
+            -0.00734688900411129,
+            -0.013244620524346828,
+            -0.012208305299282074,
+            -0.00628950959071517,
+            0.001043687923811376,
+            -0.0015302507672458887,
+            0.01603340543806553,
+            -0.006116790696978569,
+            0.01841777376830578,
+            0.003210051218047738,
+            -0.0013785947812721133,
+            0.004608656279742718,
+            -0.0437106192111969,
+            -0.002169522689655423,
+            -0.010851826518774033,
+            -0.022798946127295494,
+            0.023422420024871826,
+            0.010952930897474289,
+            0.00039098801789805293,
+            -0.023557227104902267,
+            0.003938842564821243,
+            -0.004195814952254295,
+            -0.00025855234707705677,
+            0.020406153053045273,
+            0.0026961059775203466,
+            0.0010089334100484848,
+            -0.022731544449925423,
+            -0.0029783546924591064,
+            0.010068271309137344,
+            -0.0028182733803987503,
+            0.0010178852826356888,
+            0.01567954197525978,
+            0.010632768273353577,
+            -0.012393662706017494,
+            -0.018704235553741455,
+            0.005691311322152615,
+            0.016825387254357338,
+            0.008821321651339531,
+            0.0032416461035609245,
+            -0.008071467280387878,
+            0.0044443621300160885,
+            -0.003835632000118494,
+            0.038419511169195175,
+            0.020894821733236313,
+            0.006272659171372652,
+            -0.02504008449614048,
+            0.008475882932543755,
+            -0.019176054745912552,
+            0.004574954975396395,
+            0.010556939989328384,
+            0.00872021820396185,
+            -0.0032269018702208996,
+            -0.007587010972201824,
+            0.017052870243787766,
+            -0.005291108042001724,
+            0.018451474606990814,
+            -0.005286895204335451,
+            -0.00538799911737442,
+            0.018839040771126747,
+            -0.023540375754237175,
+            0.007650200743228197,
+            -0.018586281687021255,
+            -0.011787038296461105,
+            -0.013185643590986729,
+            -0.004248473327606916,
+            0.014087153598666191,
+            0.0013543720124289393,
+            -0.0326734334230423,
+            0.007321612909436226,
+            -0.026775702834129333,
+            0.00860647577792406,
+            -0.011323644779622555,
+            -4.169880776316859e-05,
+            -0.002578151412308216,
+            -0.008496946655213833,
+            -0.02431550621986389,
+            -0.021754205226898193,
+            0.005404850002378225,
+            -0.004570742137730122,
+            -0.0030520763248205185,
+            0.004633932374417782,
+            -0.0018082867609336972,
+            -0.01579749584197998,
+            0.008063041605055332,
+            0.028814632445573807,
+            0.02111387997865677,
+            0.008539073169231415,
+            0.0056196958757936954,
+            0.014727478846907616,
+            0.0011542703723534942,
+            -0.037745483219623566,
+            0.008404267951846123,
+            0.004684484098106623,
+            0.006529631558805704,
+            0.006175767630338669,
+            -0.005539655219763517,
+            0.0017608941998332739,
+            -0.010203076526522636,
+            -0.007553309667855501,
+            0.0029130582697689533,
+            -0.010607492178678513,
+            -0.007169956807047129,
+            -0.031241128221154213,
+            0.003083671210333705,
+            -0.010826550424098969,
+            0.007325825747102499,
+            -0.006971961818635464,
+            0.01641254499554634,
+            0.0016945448005571961,
+            0.00020286618382669985,
+            -0.013817543163895607,
+            -0.013497380539774895,
+            -0.0029320153407752514,
+            5.486338704940863e-05,
+            -0.01885589212179184,
+            -0.008416905999183655,
+            0.005105750635266304,
+            -0.002458090428262949,
+            -0.026657748967409134,
+            0.023489823564887047,
+            -0.000226299132918939,
+            0.011289943940937519,
+            0.023355018347501755,
+            0.020726315677165985,
+            4.5582357415696606e-05,
+            0.004815076943486929,
+            0.0025971082504838705,
+            -0.008404267951846123,
+            -0.01861998252570629,
+            0.007587010972201824,
+            -0.029859373345971107,
+            0.02136664092540741,
+            -0.006146279163658619,
+            -0.018636832013726234,
+            -0.006436952855437994,
+            -0.009082507342100143,
+            0.011011907830834389,
+            0.00015270912263076752,
+            0.01664845459163189,
+            -0.011348920874297619,
+            -0.003734528087079525,
+            -0.005607057828456163,
+            0.0018830614862963557,
+            0.008404267951846123,
+            0.029589762911200523,
+            -0.0032079447992146015,
+            -0.010413709096610546,
+            0.0131940683349967,
+            0.007679689209908247,
+            0.02896628901362419,
+            0.0013954454334452748,
+            -0.0008451660396531224,
+            -0.013868095353245735,
+            -0.0010378954466432333,
+            -0.01075072307139635,
+            -0.007220508996397257,
+            0.02136664092540741,
+            -0.010481111705303192,
+            -0.01995118521153927,
+            -0.00817678403109312,
+            -0.02050725556910038,
+            -0.007717603351920843,
+            0.013025562278926373,
+            -0.004954094998538494,
+            -0.004326407797634602,
+            -0.007182594854384661,
+            -0.01508976798504591,
+            -0.017962805926799774,
+            0.02606797404587269,
+            0.07434512674808502,
+            -0.021265536546707153,
+            -0.009596452116966248,
+            -0.011281518265604973,
+            0.0033954083919525146,
+            -0.004903542809188366,
+            -0.006748690269887447,
+            0.05661822482943535,
+            -0.019984886050224304,
+            0.005868243053555489,
+            -0.020406153053045273,
+            -0.006108365021646023,
+            -8.392419840674847e-05,
+            0.0068877083249390125,
+            0.002439133357256651,
+            -0.008105169050395489,
+            0.009899764321744442,
+            -0.018350372090935707,
+            -0.0057334378361701965,
+            -0.016201911494135857,
+            -0.013977624475955963,
+            -0.0169686172157526,
+            0.011643807403743267,
+            -0.019917482510209084,
+            -0.008956126868724823,
+            -0.0014091366901993752,
+            -0.009588027372956276,
+            0.015317251905798912,
+            -0.009267864748835564,
+            0.009958741255104542,
+            0.0009815511293709278,
+            0.003102628281340003,
+            -0.00939424429088831,
+            -0.010026143863797188,
+            -0.007144680712372065,
+            -0.0023822623770684004,
+            0.005703949369490147,
+            0.0007082544034346938,
+            0.0014491570182144642,
+            0.007292123977094889,
+            -0.0067739663645625114,
+            0.0054764654487371445,
+            0.020355600863695145,
+            0.0009315256611444056,
+            0.004899329971522093,
+            -0.017255079001188278,
+            0.019799528643488884,
+            -0.00517315324395895,
+            0.0289494376629591,
+            0.0057123745791614056,
+            -0.009689130820333958,
+            -0.016640029847621918,
+            0.014247234910726547,
+            -0.013573208823800087,
+            -0.0034227909054607153,
+            0.01983322948217392,
+            0.029168495908379555,
+            0.00789874792098999,
+            -0.012536892667412758,
+            0.0010415816213935614,
+            -4.456210444914177e-05,
+            0.01959731988608837,
+            -0.011348920874297619,
+            0.03230271860957146,
+            -0.011896567419171333,
+            0.004448574967682362,
+            -0.026118526235222816,
+            -0.009402669966220856,
+            -0.006323210895061493,
+            -0.003669231664389372,
+            0.022546187043190002,
+            -0.0044907014816999435,
+            0.007999852299690247,
+            -0.01267169788479805,
+            -0.021046478301286697,
+            0.004642357584089041,
+            -0.0043685343116521835,
+            -0.006723414175212383,
+            0.01579749584197998,
+            -0.029488658532500267,
+            0.004250579513609409,
+            0.009023529477417469,
+            -0.013101390562951565,
+            -0.017356181517243385,
+            0.012814928777515888,
+            0.008202060125768185,
+            -0.009461646899580956,
+            -0.008821321651339531,
+            -0.019395112991333008,
+            -0.02760138548910618,
+            0.010144098661839962,
+            0.0018925400217995048,
+            0.006192618515342474,
+            0.0022221810650080442,
+            -0.011728061363101006,
+            0.020709464326500893,
+            -0.005055198911577463,
+            0.009048805572092533,
+            -0.017726896330714226,
+            0.006281084381043911,
+            0.011155138723552227,
+            0.003974650055170059,
+            -0.0004581273824442178,
+            0.01404502708464861,
+            0.007338463328778744,
+            0.033027298748493195,
+            0.01270539965480566,
+            0.008189422078430653,
+            -0.009992443025112152,
+            -0.009882913902401924,
+            0.012140902690589428,
+            -0.021080179139971733,
+            -0.0007645988371223211,
+            0.016008129343390465,
+            0.010422134771943092,
+            -0.016260888427495956,
+            0.012680123560130596,
+            -0.01720452681183815,
+            0.010818125680088997,
+            -0.03378557786345482,
+            -0.012494766153395176,
+            0.006592821795493364,
+            0.014011326245963573,
+            0.008871873840689659,
+            0.0002376206684857607,
+            0.017592091113328934,
+            0.015376229770481586,
+            0.0031342231668531895,
+            0.006845581810921431,
+            0.022815797477960587,
+            0.004697122145444155,
+            -0.004671846050769091,
+            0.0035091505851596594,
+            -0.005244768690317869,
+            0.0016776940319687128,
+            -0.010919229127466679,
+            0.011483726091682911,
+            0.008897149935364723,
+            -0.006575970910489559,
+            0.01365746185183525,
+            -0.0009689130820333958,
+            -0.012250431813299656,
+            -0.007325825747102499,
+            -0.004587593022733927,
+            -0.008981402963399887,
+            -0.005649184808135033,
+            0.019513066858053207,
+            -0.015696391463279724,
+            -0.008939276449382305,
+            0.0024496649857610464,
+            0.014685352332890034,
+            -0.0025634069461375475,
+            -0.006571758538484573,
+            -0.008678090758621693,
+            -0.00019167628488503397,
+            -0.01205664873123169,
+            0.01450842060148716,
+            0.026742001995444298,
+            0.023894239217042923,
+            -0.004225303418934345,
+            -0.0032858792692422867,
+            -0.0013448934769257903,
+            0.013632185757160187,
+            0.0010505334939807653,
+            0.00133225554600358,
+            -0.007709178142249584,
+            -0.005725012626498938,
+            -0.018653683364391327,
+            -0.0014038708759471774,
+            0.0017956487135961652,
+            0.00014099264808464795,
+            -0.005973559804260731,
+            -0.013421552255749702,
+            0.011003483086824417,
+            0.008846597746014595,
+            0.0174404364079237,
+            -0.014668501913547516,
+            -0.02980882115662098,
+            0.0195299182087183,
+            -0.003911459818482399,
+            -0.004372747149318457,
+            0.003587084822356701,
+            -0.004574954975396395,
+            0.04333990439772606,
+            0.0051436647772789,
+            0.005939858499914408,
+            -0.011441599577665329,
+            -0.008787620812654495,
+            0.015637414529919624,
+            -0.005219492595642805,
+            -0.002031557960435748,
+            -0.019732125103473663,
+            -0.02466936968266964,
+            0.005131026729941368,
+            -0.024332357570528984,
+            -0.001645045937038958,
+            0.002439133357256651,
+            -0.0013027668464928865,
+            -0.013303598389029503,
+            -0.015949152410030365,
+            -0.003936735913157463,
+            0.011045609600841999,
+            0.01719610020518303,
+            -7.589380402350798e-05,
+            0.006116790696978569,
+            -0.00746905617415905,
+            0.003593403846025467,
+            0.01233468484133482,
+            -0.003881971351802349,
+            -0.01647152192890644,
+            -0.007511182688176632
+          ],
+          "title": "Appendices A. Overview Of Hyper-Parameters"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "84f1bb46-4b04-4565-ad60-72dcd70fc3d6",
+      "type": "next",
+      "source": {
+        "id": "4ef9096a-b51d-4ed6-b15e-5f4caff268e6",
+        "properties": {
+          "page_content": "1. Introduction\n\nRecent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023a). It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017) has emerged as a popular solution. It can\n\nenable parameter scaling, while concurrently keeping computational costs at a modest level.\n\nRecent applications of MoE architectures in Transformers (Vaswani et al., 2017) have yielded successful attempts at scaling language models to a substantial size (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022), accompanied with remarkable performance. These achievements underscore the considerable potential and promise of MoE language models.\n\nDespite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts.\n\nThis architecture manifests two potential issues: (1) Knowledge Hybridity: existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. (2) Knowledge Redundancy: tokens assigned to different experts may require common knowledge. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters.\n\nThese issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models.\n\nIn response to the aforementioned issues, we introduce DeepSeekMoE, an innovative MoE architecture specifically designed towards ultimate expert specialization. Our architecture involves two principal strategies: (1) Fine-Grained Expert Segmentation: while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. (2) Shared Expert Isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.\n\nThrough compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized.\n\nStarting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B (Lepikhin et al., 2021) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5× expert parameters and computation. Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE.\n\nThese studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization.\n\nLeveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B (DeepSeek-AI, 2024), a dense model trained on the same 2T corpus. We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B (Touvron et al., 2023b), which has approximately 2.5 times the activated parameters. Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard1. Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. The experimental results still validate its substantial advantages over the GShard architecture consistently. In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nOur contributions are summarized as follows: - Architectural Innovation. We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation.\n\nEmpirical Validation. We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. Experimental results validate the high 1https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models\n\nScalability. We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B.\n\nAlignment for MoE. We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B.\n\nPublic Release. In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization.\n\n",
+          "embedding": [
+            -0.007097613997757435,
+            -0.024122508242726326,
+            -0.01228793989866972,
+            -0.023231398314237595,
+            0.03164222836494446,
+            -0.012272306717932224,
+            -0.00579612422734499,
+            0.04649406298995018,
+            -0.012303573079407215,
+            0.0454309843480587,
+            -0.02185564860701561,
+            -0.010310300625860691,
+            0.03736409172415733,
+            -0.01593836396932602,
+            0.0027690858114510775,
+            0.013272850774228573,
+            0.008512446656823158,
+            0.021105241030454636,
+            0.017853470519185066,
+            -0.029391001909971237,
+            0.004928464535623789,
+            -0.04117866978049278,
+            -0.02241845615208149,
+            -0.022606058046221733,
+            0.011209228076040745,
+            0.016493353992700577,
+            0.025873461738228798,
+            0.010279033333063126,
+            -0.04374256357550621,
+            0.036988887935876846,
+            0.026092329993844032,
+            0.026905274018645287,
+            0.012608426623046398,
+            0.006147878244519234,
+            -0.01596963219344616,
+            -0.022027617320418358,
+            0.03842717036604881,
+            0.020776936784386635,
+            -0.013640238903462887,
+            0.023888004943728447,
+            0.03273657336831093,
+            0.01783783733844757,
+            -0.028609326109290123,
+            -0.008363928645849228,
+            0.020730035379529,
+            0.000511997495777905,
+            -0.039646584540605545,
+            0.003970911726355553,
+            0.006726318504661322,
+            -0.012373924255371094,
+            -0.017962904646992683,
+            0.0026518343947827816,
+            -0.0023176681715995073,
+            0.005772673990577459,
+            -0.04880782216787338,
+            0.008121608756482601,
+            0.03786436468362808,
+            0.007570527959614992,
+            0.06381599605083466,
+            -0.00819195993244648,
+            0.02632683329284191,
+            -0.028968896716833115,
+            -0.011349929496645927,
+            0.005518629681318998,
+            0.012342656962573528,
+            -0.03111068718135357,
+            -0.009755311533808708,
+            0.004252315033227205,
+            -0.001601457828655839,
+            0.008856384083628654,
+            -0.03999052196741104,
+            0.02268422581255436,
+            -0.021433543413877487,
+            0.01341355312615633,
+            0.022637324407696724,
+            0.0019063112558797002,
+            0.03542553633451462,
+            0.012115971185266972,
+            0.00941919069737196,
+            -0.028968896716833115,
+            0.010451002046465874,
+            0.03836463764309883,
+            0.0045532602816820145,
+            0.023387733846902847,
+            0.046181391924619675,
+            -0.033017974346876144,
+            -0.01883838139474392,
+            -0.07066347450017929,
+            0.019416820257902145,
+            -0.0036641042679548264,
+            -0.036301013082265854,
+            -0.0024915907997637987,
+            0.014390647411346436,
+            -0.017790935933589935,
+            -0.015133239328861237,
+            0.030250845476984978,
+            -0.009114337153732777,
+            -0.009606792591512203,
+            -0.04149134084582329,
+            0.0034120138734579086,
+            0.0041936892084777355,
+            0.012100338004529476,
+            0.0006917828577570617,
+            -0.01863514631986618,
+            -0.017353197559714317,
+            0.007664328906685114,
+            -0.01122486125677824,
+            0.027843283489346504,
+            -0.009442640468478203,
+            0.07210175693035126,
+            0.011662599630653858,
+            -0.013014898635447025,
+            -0.00774249667301774,
+            0.01697799377143383,
+            -0.01190491858869791,
+            -0.024028707295656204,
+            -0.042460616677999496,
+            -0.03342444822192192,
+            0.007097613997757435,
+            -0.03351824730634689,
+            0.0031052061822265387,
+            -0.007050713524222374,
+            0.022011984139680862,
+            -0.02262169122695923,
+            -0.07103867828845978,
+            0.041272468864917755,
+            -0.008317028172314167,
+            -0.01979202590882778,
+            -0.002986000617966056,
+            -0.007121064234524965,
+            0.0018261895747855306,
+            0.004967548418790102,
+            -0.03423738852143288,
+            -0.027171043679118156,
+            0.033737119287252426,
+            0.028562424704432487,
+            0.04386763274669647,
+            0.06197123974561691,
+            -0.014328112825751305,
+            0.018463177606463432,
+            0.016243217512965202,
+            -0.01963569037616253,
+            0.007547077722847462,
+            -0.013358835130929947,
+            -0.029656771570444107,
+            -0.01833810843527317,
+            0.01499253697693348,
+            0.06312812119722366,
+            -0.00027554063126444817,
+            0.005131700076162815,
+            -0.03043844737112522,
+            0.05531136319041252,
+            -0.02162114717066288,
+            0.02626429870724678,
+            -0.001939532463438809,
+            0.002087073866277933,
+            -0.018291208893060684,
+            0.014077977277338505,
+            -0.019745124503970146,
+            0.003943553194403648,
+            0.01255370955914259,
+            -0.0338309183716774,
+            0.00569450668990612,
+            0.01824430748820305,
+            0.004725228995084763,
+            0.03333064541220665,
+            0.006179145537316799,
+            0.0007611566106788814,
+            -0.06187743693590164,
+            0.02789018489420414,
+            0.007937915623188019,
+            0.02185564860701561,
+            -0.011428097262978554,
+            0.008035624399781227,
+            0.0017568158218637109,
+            0.0012887875782325864,
+            0.02079256996512413,
+            0.0060306270606815815,
+            -0.042304281145334244,
+            0.016868559643626213,
+            -0.028703127056360245,
+            0.024951083585619926,
+            0.027937084436416626,
+            0.01647772081196308,
+            0.029172131791710854,
+            0.01813487336039543,
+            0.04702560231089592,
+            0.028796928003430367,
+            0.017259396612644196,
+            0.01966695673763752,
+            0.01883838139474392,
+            0.007789397146552801,
+            0.03207996487617493,
+            0.011084159836173058,
+            0.010638604871928692,
+            -0.026076696813106537,
+            -0.0023919271770864725,
+            0.025123052299022675,
+            0.022371554747223854,
+            0.00022302180877886713,
+            -0.0031208398286253214,
+            -0.027405545115470886,
+            0.0038243476301431656,
+            -0.006437098141759634,
+            0.011725133284926414,
+            -0.0005163944442756474,
+            -0.003394426079466939,
+            0.02990690805017948,
+            0.02507615275681019,
+            -0.02055806666612625,
+            -0.013718406669795513,
+            -0.013202500529587269,
+            -0.03495652973651886,
+            -0.05943860858678818,
+            0.00731257488951087,
+            0.005628064274787903,
+            -0.002409514971077442,
+            0.012209772132337093,
+            0.024497712031006813,
+            -0.03864603862166405,
+            -0.006269038189202547,
+            -0.005651514511555433,
+            0.015563160181045532,
+            -0.02762441523373127,
+            -0.0013708635233342648,
+            -0.0062494962476193905,
+            0.012584976851940155,
+            0.0018232582369819283,
+            0.0441177673637867,
+            -0.0175720676779747,
+            -0.0032107324805110693,
+            0.018682045862078667,
+            0.00900490302592516,
+            0.02573275938630104,
+            0.03989672288298607,
+            -0.02431011013686657,
+            0.012209772132337093,
+            -0.0007171873585321009,
+            0.011154510080814362,
+            0.017368830740451813,
+            0.00593682611361146,
+            -0.002163287252187729,
+            -0.030500980094075203,
+            -0.018181772902607918,
+            0.035269200801849365,
+            0.02026103064417839,
+            0.00014326647215057164,
+            0.04396143555641174,
+            0.0065035405568778515,
+            0.021496077999472618,
+            0.013390102423727512,
+            -0.016165051609277725,
+            -0.008066891692578793,
+            0.006788852158933878,
+            -0.02623303234577179,
+            0.04271075502038002,
+            0.034831464290618896,
+            -0.0005466843722388148,
+            -0.010341567918658257,
+            -0.03958405181765556,
+            0.019682591781020164,
+            -0.028265388682484627,
+            -0.016071248799562454,
+            0.04943316429853439,
+            -0.01714996248483658,
+            0.02401307411491871,
+            0.04552478343248367,
+            0.01860387809574604,
+            -0.009575525298714638,
+            -0.0058977422304451466,
+            0.0014216724084690213,
+            -0.011396829970180988,
+            -0.02925029955804348,
+            0.013546437956392765,
+            0.003519494319334626,
+            -0.03176729381084442,
+            0.024169407784938812,
+            -0.0020089060999453068,
+            0.01830684207379818,
+            0.01532084122300148,
+            0.02158987894654274,
+            -0.009817845188081264,
+            0.02543572336435318,
+            0.010239950381219387,
+            -0.019651323556900024,
+            -0.0199014600366354,
+            -0.008270127698779106,
+            -0.04264821857213974,
+            -0.01573512889444828,
+            -0.013640238903462887,
+            -0.012413008138537407,
+            0.012037803418934345,
+            -0.05046497657895088,
+            0.004490726161748171,
+            0.00925503857433796,
+            -0.003300625132396817,
+            -0.017165595665574074,
+            0.020089061930775642,
+            0.004693961702287197,
+            -0.0008246677462011576,
+            -0.06691142916679382,
+            0.008473362773656845,
+            0.011529714800417423,
+            0.001870158826932311,
+            -0.02576402761042118,
+            0.023622235283255577,
+            -0.012209772132337093,
+            0.047901079058647156,
+            -0.01425776258111,
+            -0.03664495050907135,
+            0.0108574740588665,
+            -0.014296846464276314,
+            -0.0075079938396811485,
+            -0.007574436254799366,
+            0.0036054786760360003,
+            0.058719467371702194,
+            0.04042826220393181,
+            0.013007081113755703,
+            -0.03176729381084442,
+            -0.020683135837316513,
+            0.022699858993291855,
+            0.0027397728990763426,
+            -0.04705686867237091,
+            0.019057249650359154,
+            0.015461542643606663,
+            -0.010826206766068935,
+            0.024450812488794327,
+            0.04302342236042023,
+            -0.023794203996658325,
+            0.017024893313646317,
+            -0.011193593963980675,
+            -0.032580237835645676,
+            -0.031251389533281326,
+            0.012358291074633598,
+            -0.005334935616701841,
+            -0.04430537298321724,
+            -0.0290157962590456,
+            -0.0036191579420119524,
+            -0.007433734834194183,
+            0.009817845188081264,
+            -0.015180139802396297,
+            -0.039052512496709824,
+            -0.036394815891981125,
+            0.031517159193754196,
+            0.0017431365558877587,
+            0.0038946985732764006,
+            -0.03601961210370064,
+            0.0033651133999228477,
+            0.005702323280274868,
+            0.00542873702943325,
+            0.001134406658820808,
+            0.009090886451303959,
+            0.009294122457504272,
+            0.0013835657155141234,
+            -0.01187365222722292,
+            -0.038614772260189056,
+            -0.029203400015830994,
+            -0.0011451547034084797,
+            0.011209228076040745,
+            0.04055332764983177,
+            0.03014140948653221,
+            -0.05312267318367958,
+            -0.04149134084582329,
+            0.02957860380411148,
+            -0.03705142065882683,
+            -0.017228130251169205,
+            0.0033592507243156433,
+            0.00022241112310439348,
+            0.002487682504579425,
+            0.010724589228630066,
+            0.00532321073114872,
+            0.04427410289645195,
+            -0.0007357521099038422,
+            0.012710045091807842,
+            0.01567259430885315,
+            0.013460453599691391,
+            0.02424757555127144,
+            -0.015680411830544472,
+            -0.020354831591248512,
+            0.04755714163184166,
+            -0.023669136688113213,
+            0.01641518622636795,
+            0.011365562677383423,
+            0.011037259362637997,
+            0.013906008563935757,
+            0.017290662974119186,
+            0.001894586137495935,
+            0.011865834705531597,
+            -0.058844536542892456,
+            0.014375013299286366,
+            0.004736953880637884,
+            -0.031954895704984665,
+            -0.024482078850269318,
+            -0.014695500954985619,
+            0.02729611098766327,
+            0.03207996487617493,
+            0.00952080823481083,
+            -0.024466445669531822,
+            -0.022668590769171715,
+            -0.00555380480363965,
+            0.007304758299142122,
+            0.04596252366900444,
+            -0.0008617973071523011,
+            0.02049553394317627,
+            -0.015492809936404228,
+            0.010279033333063126,
+            0.01527394074946642,
+            0.01324940100312233,
+            0.026655137538909912,
+            -0.00941919069737196,
+            -3.502273102640174e-05,
+            -0.043930165469646454,
+            -0.027139775454998016,
+            -0.013937275856733322,
+            0.006112703122198582,
+            0.02712414227426052,
+            0.026389367878437042,
+            0.006374564487487078,
+            -0.003955278545618057,
+            0.022011984139680862,
+            -0.030500980094075203,
+            0.016165051609277725,
+            0.03608214482665062,
+            0.026670770719647408,
+            -0.016665322706103325,
+            0.0023587059695273638,
+            0.01933865435421467,
+            0.0032713122200220823,
+            0.01016178261488676,
+            -0.0252012200653553,
+            0.028859462589025497,
+            -0.022996895015239716,
+            0.029422268271446228,
+            -0.0237160362303257,
+            -0.027217943221330643,
+            -0.030954353511333466,
+            -0.003388563636690378,
+            0.03458132594823837,
+            -0.013046164996922016,
+            -0.01112324371933937,
+            -0.008449913002550602,
+            0.02380983717739582,
+            -0.014961270615458488,
+            -0.0068513862788677216,
+            -0.03702015429735184,
+            -0.017900370061397552,
+            -0.05071511119604111,
+            0.0032380910124629736,
+            0.003027038648724556,
+            -0.022199586033821106,
+            -0.044680576771497726,
+            0.014445364475250244,
+            -0.09092450141906738,
+            -0.004307032562792301,
+            0.032580237835645676,
+            -0.020683135837316513,
+            0.006538716144859791,
+            0.0038204393349587917,
+            0.011740767396986485,
+            -0.03877110779285431,
+            0.0010366971837356687,
+            0.03226756677031517,
+            0.0514029860496521,
+            -0.008637514896690845,
+            -0.01996399462223053,
+            0.00903616938740015,
+            -0.0172437634319067,
+            -0.01904161646962166,
+            0.008434279821813107,
+            0.0228874608874321,
+            0.020135963335633278,
+            0.0036289289128035307,
+            0.026358099654316902,
+            0.01718122884631157,
+            0.030860550701618195,
+            0.03336191177368164,
+            -0.01271786168217659,
+            -0.04918302595615387,
+            0.02768694795668125,
+            0.039083778858184814,
+            0.009739677421748638,
+            -0.015281757339835167,
+            -0.01543809287250042,
+            -0.0034159221686422825,
+            -0.0024661864154040813,
+            0.0010112927993759513,
+            0.0008114769589155912,
+            -0.00029141842969693244,
+            0.01521922368556261,
+            0.028109053149819374,
+            -0.009481724351644516,
+            0.026780204847455025,
+            0.003990453667938709,
+            0.02194944955408573,
+            0.004506359808146954,
+            -0.02513868547976017,
+            -0.010255583561956882,
+            -0.0037227298598736525,
+            0.02481038309633732,
+            0.005276310257613659,
+            -0.03505033254623413,
+            0.006155695300549269,
+            0.029406635090708733,
+            -0.01286637969315052,
+            0.008465546183288097,
+            0.013452637009322643,
+            -0.00037764699663966894,
+            0.016337020322680473,
+            -0.01910415105521679,
+            -0.0001468084374209866,
+            0.02330956608057022,
+            -0.01797853782773018,
+            0.008160692639648914,
+            -0.028593692928552628,
+            -0.012061254121363163,
+            0.007492360193282366,
+            -0.011396829970180988,
+            0.017556432634592056,
+            -0.001251657959073782,
+            0.015680411830544472,
+            -0.005045715719461441,
+            0.0027983984909951687,
+            -0.010654238052666187,
+            0.03501906618475914,
+            -0.00401390390470624,
+            0.03150152415037155,
+            0.021214675158262253,
+            -0.015250490047037601,
+            0.044524241238832474,
+            -0.03295544162392616,
+            -0.025873461738228798,
+            0.01159224845468998,
+            0.006386289373040199,
+            -0.011537531390786171,
+            -0.015336474403738976,
+            -0.013616788201034069,
+            -0.00037911266554147005,
+            0.01718122884631157,
+            -0.02209015190601349,
+            -0.024153774604201317,
+            -0.020010894164443016,
+            -0.010318117216229439,
+            -0.0031638317741453648,
+            0.000499783840496093,
+            -0.0059446427039802074,
+            -0.014593882486224174,
+            0.04880782216787338,
+            -0.05543643236160278,
+            0.01943245530128479,
+            0.047838546335697174,
+            0.013171233236789703,
+            -0.04252314940094948,
+            -0.023997440934181213,
+            -0.014539165422320366,
+            -0.0075509860180318356,
+            -0.041835278272628784,
+            0.014109243638813496,
+            0.002925420878455043,
+            0.005772673990577459,
+            -0.048995424062013626,
+            -0.0009712319006212056,
+            0.005784399341791868,
+            -0.022996895015239716,
+            0.014015442691743374,
+            -0.022934360429644585,
+            -0.022606058046221733,
+            0.01424212846904993,
+            0.019416820257902145,
+            -0.006104886066168547,
+            0.013741856440901756,
+            -0.0035761657636612654,
+            -0.007980907335877419,
+            -0.0021456994581967592,
+            -0.0009365450241602957,
+            0.0011207273928448558,
+            0.019229218363761902,
+            0.004365657921880484,
+            -0.00630812207236886,
+            -0.0013132150052115321,
+            -0.005002723541110754,
+            0.030688581988215446,
+            -0.008864200673997402,
+            -0.009661509655416012,
+            0.03958405181765556,
+            0.035800740122795105,
+            -0.010724589228630066,
+            -0.02218395285308361,
+            -0.023825472220778465,
+            0.014797118492424488,
+            -0.02948480285704136,
+            0.028265388682484627,
+            0.036426082253456116,
+            -0.011670416221022606,
+            -0.03162659332156181,
+            0.005577255040407181,
+            -0.022105785086750984,
+            -0.003996316343545914,
+            0.01704052835702896,
+            0.01804107241332531,
+            -0.04418030381202698,
+            -0.006003268528729677,
+            0.01688419282436371,
+            0.01238955743610859,
+            -0.009137787856161594,
+            -0.03064168244600296,
+            0.008653149008750916,
+            0.01904161646962166,
+            0.013335385359823704,
+            -0.0027808109298348427,
+            0.00702335499227047,
+            -0.0165871549397707,
+            -0.0011353837326169014,
+            -0.000332700670696795,
+            -0.013429186306893826,
+            0.03573820739984512,
+            0.0010484223021194339,
+            0.010294667445123196,
+            -0.00013398408191278577,
+            -0.02005779556930065,
+            -0.012944547459483147,
+            0.03551933914422989,
+            0.01399199292063713,
+            0.038614772260189056,
+            0.0019502805080264807,
+            0.015578794293105602,
+            0.03817703574895859,
+            -0.002221912844106555,
+            -0.0273586455732584,
+            0.021668046712875366,
+            0.0010064073139801621,
+            -0.0441177673637867,
+            0.00657389173284173,
+            -0.03128265589475632,
+            -0.04593125730752945,
+            0.023653503507375717,
+            -0.013624604791402817,
+            0.022793659940361977,
+            -0.005432645324617624,
+            -0.004381291568279266,
+            -0.0318298302590847,
+            0.0035038606729358435,
+            0.02188691683113575,
+            0.015664778649806976,
+            0.04424283653497696,
+            -0.0051629673689603806,
+            -0.005643697455525398,
+            0.011217044666409492,
+            -0.001060147536918521,
+            0.037989433854818344,
+            0.01185801811516285,
+            -0.014281212352216244,
+            0.013859108090400696,
+            0.030266478657722473,
+            0.04761967435479164,
+            -0.02928156778216362,
+            -0.03539426997303963,
+            0.008676598779857159,
+            0.0044633676297962666,
+            0.041303738951683044,
+            0.006831844337284565,
+            0.006155695300549269,
+            0.016790391877293587,
+            0.014695500954985619,
+            -0.008567164652049541,
+            0.024028707295656204,
+            -0.03014140948653221,
+            -0.034768927842378616,
+            0.03670748323202133,
+            -0.008145059458911419,
+            0.012858563102781773,
+            0.01800980605185032,
+            -0.01425776258111,
+            -0.016821658238768578,
+            0.005639789160341024,
+            0.01091219112277031,
+            -0.015344290994107723,
+            0.030798017978668213,
+            0.03389345109462738,
+            0.02398180589079857,
+            -0.0048581138253211975,
+            0.012694410979747772,
+            0.0009091863757930696,
+            0.002092936309054494,
+            -0.01744699850678444,
+            0.007003813050687313,
+            -0.0008837819332256913,
+            -0.01618068479001522,
+            0.043992701917886734,
+            -0.0290157962590456,
+            -0.012928914278745651,
+            -0.0022844469640403986,
+            -0.002614704892039299,
+            0.005502996034920216,
+            0.003972866106778383,
+            0.018619511276483536,
+            -0.012788211926817894,
+            -0.018916549161076546,
+            0.017368830740451813,
+            0.024747848510742188,
+            0.002688963897526264,
+            -0.0072461324743926525,
+            0.023966172710061073,
+            -0.007937915623188019,
+            0.009278489276766777,
+            0.02304379642009735,
+            0.004572802223265171,
+            0.012366107665002346,
+            -0.021542979404330254,
+            -0.006890470162034035,
+            -0.0059876348823308945,
+            0.0075079938396811485,
+            0.044680576771497726,
+            0.0025365371257066727,
+            0.027186676859855652,
+            0.028296655043959618,
+            -0.004799488000571728,
+            -0.009028352797031403,
+            0.0014529394684359431,
+            -0.013859108090400696,
+            0.046806734055280685,
+            0.0006092183757573366,
+            0.007265674415975809,
+            0.013468270190060139,
+            -0.05134045332670212,
+            -0.010779306292533875,
+            0.016024349257349968,
+            0.0012506808852776885,
+            0.03339318186044693,
+            -0.003781355684623122,
+            -0.0033846551086753607,
+            0.014804935082793236,
+            -0.035863276571035385,
+            -0.039677850902080536,
+            0.012366107665002346,
+            0.006018901709467173,
+            -0.011959636583924294,
+            -0.004299215506762266,
+            0.013749673031270504,
+            -0.005788307636976242,
+            -0.004346115980297327,
+            0.009184688329696655,
+            -0.006624700501561165,
+            0.008965819142758846,
+            0.009215954691171646,
+            -0.008739132434129715,
+            0.03986545279622078,
+            -0.009606792591512203,
+            0.03727029263973236,
+            0.0007645764271728694,
+            0.0010112927993759513,
+            0.006867019925266504,
+            -0.01979202590882778,
+            0.03345571458339691,
+            -0.0003559066681191325,
+            -0.04533718153834343,
+            0.0027827650774270296,
+            0.02082383632659912,
+            -0.008231043815612793,
+            0.009825661778450012,
+            -0.006233862601220608,
+            0.015711678192019463,
+            0.03952151536941528,
+            0.0055928886868059635,
+            0.01132647879421711,
+            0.017259396612644196,
+            0.0048581138253211975,
+            0.0018779755337163806,
+            0.031188854947686195,
+            0.0011090021580457687,
+            -0.009231588803231716,
+            0.029672404751181602,
+            -0.018713314086198807,
+            0.037457894533872604,
+            -0.047807276248931885,
+            -0.007414192892611027,
+            0.03561313822865486,
+            0.0005046692676842213,
+            -0.015383374877274036,
+            -0.04196034371852875,
+            0.025029251351952553,
+            -0.03229883313179016,
+            0.021183408796787262,
+            -0.003157969331368804,
+            0.033612050116062164,
+            -0.023825472220778465,
+            0.002618613187223673,
+            0.01883838139474392,
+            -0.013077432289719582,
+            0.02477911487221718,
+            0.014875286258757114,
+            -0.01064642146229744,
+            0.0021417911630123854,
+            -0.013225950300693512,
+            0.006738043390214443,
+            0.04646279662847519,
+            -0.009888196364045143,
+            0.004029537551105022,
+            0.007511902134865522,
+            -0.02224648743867874,
+            -0.018353741616010666,
+            0.04987090080976486,
+            0.0013239630497992039,
+            0.023293932899832726,
+            0.03661368414759636,
+            -0.041897810995578766,
+            -0.02168367989361286,
+            0.02135537751019001,
+            0.004912830889225006,
+            0.0037325008306652308,
+            0.014031076803803444,
+            0.016258852556347847,
+            0.014867469668388367,
+            -0.020386097952723503,
+            0.03411232307553291,
+            0.011131060309708118,
+            0.00669895950704813,
+            -0.019651323556900024,
+            -0.024169407784938812,
+            -0.020776936784386635,
+            -0.03398725390434265,
+            -0.02102707326412201,
+            -0.00927067268639803,
+            0.010200866498053074,
+            -0.008473362773656845,
+            -0.04883908852934837,
+            -0.017790935933589935,
+            0.026358099654316902,
+            -0.0069412789307534695,
+            0.04583745449781418,
+            0.0015369695611298084,
+            0.036957621574401855,
+            0.0012604518560692668,
+            -0.016993626952171326,
+            0.050590042024850845,
+            -0.027937084436416626,
+            0.04158513993024826,
+            0.022840559482574463,
+            -0.004025629255920649,
+            -0.008410829119384289,
+            0.04752587527036667,
+            -0.024122508242726326,
+            0.0032537246588617563,
+            -0.029688037931919098,
+            -0.003392471931874752,
+            -0.024951083585619926,
+            -0.0011197502026334405,
+            -0.0028746118769049644,
+            -0.008684415370225906,
+            0.024357011541724205,
+            -0.015242673456668854,
+            0.020401732996106148,
+            0.019385553896427155,
+            -0.0021437453106045723,
+            0.008739132434129715,
+            -0.020089061930775642,
+            0.012561526149511337,
+            -0.016399553045630455,
+            0.004568893928080797,
+            -0.009919462725520134,
+            0.001229184796102345,
+            -0.03279910609126091,
+            -0.00606189388781786,
+            -0.0067224097438156605,
+            -0.011193593963980675,
+            0.015367741696536541,
+            0.038552239537239075,
+            -0.0045884354040026665,
+            -0.035206668078899384,
+            0.007781580090522766,
+            -0.04008432477712631,
+            0.010169599205255508,
+            0.007144514936953783,
+            0.002470094710588455,
+            -0.009161237627267838,
+            0.01877584680914879,
+            0.022011984139680862,
+            -0.012772578746080399,
+            -0.033737119287252426,
+            0.009724044241011143,
+            -0.006038443651050329,
+            0.016524622216820717,
+            0.0029469169676303864,
+            0.0334869809448719,
+            -0.004009995609521866,
+            0.01880711503326893,
+            -0.012358291074633598,
+            -0.004693961702287197,
+            -0.017556432634592056,
+            0.0023293932899832726,
+            -0.02380983717739582,
+            -0.00526458490639925,
+            -0.00030778476502746344,
+            -0.013358835130929947,
+            -0.00802780780941248,
+            0.0027280477806925774,
+            0.011146693490445614,
+            -0.014156145043671131,
+            0.010732405818998814,
+            -0.019057249650359154,
+            -0.010044530965387821,
+            0.0072109573520720005,
+            -0.03226756677031517,
+            -0.014070160686969757,
+            -0.038552239537239075,
+            -0.003859523218125105,
+            0.013499537482857704,
+            0.03358078375458717,
+            -0.007562711369246244,
+            0.010724589228630066,
+            -0.030250845476984978,
+            -0.0007328208303079009,
+            0.023590968921780586,
+            0.023856738582253456,
+            -0.03545680269598961,
+            0.003523402614519,
+            0.025123052299022675,
+            0.0293441005051136,
+            -0.035206668078899384,
+            0.015625694766640663,
+            0.0351441353559494,
+            0.007093705702573061,
+            0.021371010690927505,
+            -0.0105291698127985,
+            0.042992155998945236,
+            -0.03786436468362808,
+            0.01160788256675005,
+            -0.004826846532523632,
+            0.024294476956129074,
+            0.007840205915272236,
+            0.024591512978076935,
+            -0.02049553394317627,
+            0.0328303724527359,
+            0.033612050116062164,
+            0.02032356522977352,
+            0.04418030381202698,
+            -0.03658241778612137,
+            -0.017415732145309448,
+            0.012538076378405094,
+            -0.01966695673763752,
+            0.01245209202170372,
+            -0.0023606601171195507,
+            -0.02481038309633732,
+            0.009755311533808708,
+            0.024325743317604065,
+            0.008637514896690845,
+            -0.059376075863838196,
+            0.023528434336185455,
+            0.02165241353213787,
+            0.0131008829921484,
+            0.02374730445444584,
+            -0.0371452234685421,
+            0.0004274788370821625,
+            0.015930548310279846,
+            0.015242673456668854,
+            0.020933272317051888,
+            0.017337564378976822,
+            -0.010349384509027004,
+            -0.0008661942556500435,
+            -0.023137597367167473,
+            0.006769310217350721,
+            -0.0035722574684768915,
+            0.010052347555756569,
+            -0.02646753564476967,
+            -0.028187220916152,
+            0.00872349925339222,
+            0.004994906950742006,
+            -0.0021535162813961506,
+            -0.03723902255296707,
+            -0.03933391347527504,
+            0.013749673031270504,
+            0.00819195993244648,
+            -0.044618040323257446,
+            -0.012921096757054329,
+            -0.03823956847190857,
+            -0.01447663176804781,
+            -0.021636780351400375,
+            0.0018261895747855306,
+            -0.010185232385993004,
+            0.0026928724255412817,
+            -0.005202050786465406,
+            0.025513891130685806,
+            -0.010451002046465874,
+            0.04552478343248367,
+            0.006566074676811695,
+            -0.003556623822078109,
+            -0.030125776305794716,
+            0.04802614822983742,
+            -0.007433734834194183,
+            -0.03933391347527504,
+            0.02454461343586445,
+            -0.017790935933589935,
+            0.011662599630653858,
+            0.03187672793865204,
+            -0.007949640974402428,
+            -0.014883102849125862,
+            0.013851291500031948,
+            -0.023262664675712585,
+            -0.042398083955049515,
+            0.007074163760989904,
+            -0.016337020322680473,
+            -0.0293441005051136,
+            -0.041460070759058,
+            -0.004834663588553667,
+            0.018713314086198807,
+            -0.025779660791158676,
+            0.004701778758317232,
+            0.01824430748820305,
+            0.019354287534952164,
+            -0.038520973175764084,
+            0.008551530539989471,
+            -0.00254044565372169,
+            -0.0017480220412835479,
+            -0.010787122882902622,
+            0.009591159410774708,
+            -0.0032380910124629736,
+            -0.004393016919493675,
+            -0.021449178457260132,
+            -0.005784399341791868,
+            -0.029203400015830994,
+            -0.010279033333063126,
+            -0.012006537057459354,
+            -0.027905818074941635,
+            -0.04589999094605446,
+            -0.04483691230416298,
+            0.025482622906565666,
+            0.03126702457666397,
+            -0.003568348940461874,
+            0.0028277114033699036,
+            -0.008379561826586723,
+            -0.012647510506212711,
+            0.019526256248354912,
+            0.00017880828818306327,
+            0.010771489702165127,
+            -0.0133510185405612,
+            0.016634056344628334,
+            -0.010279033333063126,
+            0.00540528679266572,
+            -0.04336735978722572,
+            -0.020370464771986008,
+            0.032986707985401154,
+            0.007062438875436783,
+            -0.0007914465386420488,
+            -0.004721320234239101,
+            0.013468270190060139,
+            -0.02434137649834156,
+            -0.01286637969315052,
+            0.03392472118139267,
+            0.01721249520778656,
+            0.01452353224158287,
+            -0.015000354498624802,
+            -0.008817300200462341,
+            0.003703187918290496,
+            -0.011412463150918484,
+            0.0049362811259925365,
+            -0.004615794401615858,
+            0.009075253270566463,
+            0.013804391026496887,
+            0.013569887727499008,
+            0.013898191973567009,
+            0.02235592156648636,
+            -0.009043985977768898,
+            -0.00649181567132473,
+            -0.003738363506272435,
+            0.008465546183288097,
+            -0.006272946484386921,
+            -0.006894378457218409,
+            -0.025873461738228798,
+            0.033674582839012146,
+            0.01890091598033905,
+            0.010998175479471684,
+            -0.015227040275931358,
+            -0.003757905215024948,
+            0.009356657043099403,
+            -0.009958546608686447,
+            -0.005823483224958181,
+            0.0037930808030068874,
+            0.025029251351952553,
+            -0.02194944955408573,
+            -0.013061799108982086,
+            -0.010865290649235249,
+            -0.0026479260995984077,
+            0.006870928220450878,
+            0.03773929551243782,
+            0.018431909382343292,
+            -0.008528080768883228,
+            -0.008488996885716915,
+            0.002384110586717725,
+            0.017337564378976822,
+            0.023919273167848587,
+            -0.03539426997303963,
+            -0.030751116573810577,
+            0.008105975575745106,
+            -0.0038829734548926353,
+            0.021136507391929626,
+            0.00532321073114872,
+            0.003744225949048996,
+            0.003537081880494952,
+            0.0722893550992012,
+            -0.0008554462110623717,
+            0.010067981667816639,
+            -0.017900370061397552,
+            -0.01520358957350254,
+            -0.0013493674341589212,
+            -0.005565530154854059,
+            -0.01308524888008833,
+            0.036957621574401855,
+            0.001335688168182969,
+            0.044618040323257446,
+            0.011865834705531597,
+            -0.006296396721154451,
+            -0.01190491858869791,
+            -0.0006805462762713432,
+            0.012084703892469406,
+            -0.0005608522333204746,
+            -0.0018662504153326154,
+            -0.0025658500380814075,
+            0.008973635733127594,
+            0.006499632261693478,
+            0.02570149302482605,
+            0.0005916306981816888,
+            0.017384463921189308,
+            -0.02424757555127144,
+            0.03601961210370064,
+            0.01711869426071644,
+            0.012842929922044277,
+            -0.019416820257902145,
+            -0.008207593113183975,
+            0.0028003526385873556,
+            0.0044633676297962666,
+            -0.007707321085035801,
+            -0.0006507448852062225,
+            -0.008660965599119663,
+            0.030454080551862717,
+            -0.03007887676358223,
+            0.01668095588684082,
+            0.011725133284926414,
+            -0.018197407945990562,
+            -0.005940734408795834,
+            0.015352108515799046,
+            -0.002077302895486355,
+            0.004459459334611893,
+            -0.006112703122198582,
+            0.0052450429648160934,
+            0.024357011541724205,
+            -0.028187220916152,
+            -0.006128336302936077,
+            0.010458819568157196,
+            -0.01493781991302967,
+            -0.0036445623263716698,
+            0.004221048206090927,
+            0.00956770870834589,
+            -0.014875286258757114,
+            0.011428097262978554,
+            0.02434137649834156,
+            -0.027937084436416626,
+            -0.009450457990169525,
+            0.011310845613479614,
+            -0.00010235064837615937,
+            0.01866641268134117,
+            0.023403367027640343,
+            0.008739132434129715,
+            -0.006448823492974043,
+            -0.0031423356849700212,
+            0.010341567918658257,
+            -0.009677143767476082,
+            -0.007468909956514835,
+            -0.0069842711091041565,
+            -0.010716771706938744,
+            0.01940118707716465,
+            -0.01591491512954235,
+            -0.0026928724255412817,
+            -0.003937690518796444,
+            -0.0106933219358325,
+            0.001270222826860845,
+            -0.013069615699350834,
+            0.017071794718503952,
+            0.0013249401235952973,
+            0.017931638285517693,
+            0.010951275005936623,
+            0.004846388474106789,
+            0.023590968921780586,
+            -0.012655327096581459,
+            0.012248856015503407,
+            -0.032017432153224945,
+            -0.0018027393380180001,
+            0.009145604446530342,
+            0.006022810470312834,
+            -0.01635265350341797,
+            -0.019948361441493034,
+            -0.013194683939218521,
+            0.0032244117464870214,
+            -0.009372290223836899,
+            0.0065465327352285385,
+            0.014296846464276314,
+            0.037926897406578064,
+            -0.016133783385157585,
+            -0.030469713732600212,
+            -0.015094155445694923,
+            0.012905463576316833,
+            0.016243217512965202,
+            -0.023888004943728447,
+            -0.009669327177107334,
+            -0.007269582711160183,
+            0.0034432809334248304,
+            -0.013171233236789703,
+            -0.01833810843527317,
+            -0.01668095588684082,
+            -0.003011405235156417,
+            -0.025889094918966293,
+            0.0013493674341589212,
+            -0.0043773832730948925,
+            0.0067224097438156605,
+            0.019479354843497276,
+            -0.0034725936129689217,
+            -0.02179311402142048,
+            -0.02490418404340744,
+            0.0175720676779747,
+            -0.02434137649834156,
+            0.008231043815612793,
+            0.025623325258493423,
+            -0.017853470519185066,
+            0.004928464535623789,
+            0.026670770719647408,
+            0.004948006477206945,
+            -0.009333206340670586,
+            0.012670961208641529,
+            0.009778761304914951,
+            0.006362839136272669,
+            0.008700049482285976,
+            0.0190728846937418,
+            0.012061254121363163,
+            0.020933272317051888,
+            -0.021605512127280235,
+            -0.013593338429927826,
+            -0.004025629255920649,
+            0.03626974672079086,
+            -0.0010327888885512948,
+            -0.00878603383898735,
+            0.014617333188652992,
+            0.0361446775496006,
+            0.005663239397108555,
+            -0.005585072096437216,
+            -0.01786910369992256,
+            0.0029586420860141516,
+            0.014601700007915497,
+            0.005291943438351154,
+            0.002679192926734686,
+            -0.00567105645313859,
+            -0.01048226933926344,
+            -0.00022790727962274104,
+            -0.0009438732522539794,
+            0.004697869997471571,
+            -0.026389367878437042,
+            0.018697679042816162,
+            -0.023419000208377838,
+            0.014171778224408627,
+            -0.0013063752558082342,
+            0.0007059507770463824,
+            0.03093871846795082,
+            -0.0022023709025233984,
+            -0.022340288385748863,
+            0.002501361770555377,
+            0.0019502805080264807,
+            0.013601155020296574,
+            0.022230852395296097,
+            -0.010943458415567875,
+            0.01261624414473772,
+            -0.003331892192363739,
+            0.0014382830122485757,
+            0.007461093366146088,
+            0.030469713732600212,
+            0.002923466730862856,
+            -0.011045075953006744,
+            0.020354831591248512,
+            0.009638059884309769,
+            0.0007425918010994792,
+            -0.007703412789851427,
+            0.020886370912194252,
+            -0.015602244064211845,
+            0.012491175904870033,
+            -0.012663144618272781,
+            -0.006323755253106356,
+            3.625936528806051e-07,
+            0.0010777352144941688,
+            -0.0023626144975423813,
+            -0.030688581988215446,
+            0.022965628653764725,
+            0.0005041807307861745,
+            0.0177440345287323,
+            -0.009505175054073334,
+            -0.019307386130094528,
+            0.002489636652171612,
+            0.020933272317051888,
+            -0.006855294574052095,
+            -0.01261624414473772,
+            -0.014304663054645061,
+            -0.014335930347442627,
+            -0.020073428750038147,
+            0.006695051211863756,
+            0.004060804378241301,
+            -0.015039438381791115,
+            0.0053740194998681545,
+            -0.003746180096641183,
+            -0.019119784235954285,
+            0.005280218552798033,
+            0.02410687506198883,
+            -0.027765115723013878,
+            0.03162659332156181,
+            -0.015860198065638542,
+            -0.015297390520572662,
+            0.009075253270566463,
+            0.0035175401717424393,
+            0.0005569438217207789,
+            0.005502996034920216,
+            -0.02135537751019001,
+            0.0020694860722869635,
+            0.020417366176843643,
+            -0.01704052835702896,
+            -0.0019932726863771677,
+            0.010075798258185387,
+            0.017493899911642075,
+            0.0007289124769158661,
+            -0.018025439232587814,
+            0.02490418404340744,
+            -0.016227584332227707,
+            0.02454461343586445,
+            0.02126157470047474,
+            -0.041804008185863495,
+            0.042116679251194,
+            -0.024654047563672066,
+            0.015828929841518402,
+            0.002237546257674694,
+            -0.006437098141759634,
+            0.013069615699350834,
+            0.018885280936956406,
+            -0.005256768316030502,
+            -0.0012780395336449146,
+            0.002804261166602373,
+            0.03167349472641945,
+            -0.023403367027640343,
+            0.024528980255126953,
+            0.02132410928606987,
+            -0.00010387735528638586,
+            0.02481038309633732,
+            -0.021230308338999748,
+            -0.029766205698251724,
+            0.019620057195425034,
+            -0.023465899750590324,
+            -0.034174855798482895,
+            0.020042162388563156,
+            -0.008418645709753036,
+            -0.008184143342077732,
+            -0.020339198410511017,
+            0.006386289373040199,
+            0.0054639121517539024,
+            -0.013007081113755703,
+            0.002849207492545247,
+            0.0027397728990763426,
+            0.02606106363236904,
+            0.022574789822101593,
+            0.01520358957350254,
+            -0.003320167073979974,
+            0.018166139721870422,
+            -0.0015731221064925194,
+            0.032580237835645676,
+            -0.0020304021891206503,
+            0.0004528832796495408,
+            -0.011052892543375492,
+            -0.0021456994581967592,
+            0.0034198306966573,
+            0.0039220573380589485,
+            0.017384463921189308,
+            -0.02069876901805401,
+            -0.015868013724684715,
+            -0.0351441353559494,
+            -0.0025971170980483294,
+            -0.003539036260917783,
+            0.010708955116569996,
+            0.03508159890770912,
+            -0.00903616938740015,
+            -0.013999809511005878,
+            0.009192504920065403,
+            -0.0068083941005170345,
+            -0.0002442736295051873,
+            -0.02330956608057022,
+            -0.020354831591248512,
+            -0.01493781991302967,
+            -0.013272850774228573,
+            -0.008856384083628654,
+            -0.020042162388563156,
+            0.012960180640220642,
+            -0.010599520988762379,
+            0.01048226933926344,
+            -0.02795271761715412,
+            0.014406280592083931,
+            -0.012280123308300972,
+            0.009059620089828968,
+            -0.005456095561385155,
+            -0.024200676009058952,
+            -0.004357841331511736,
+            0.02653006836771965,
+            -0.0034745479933917522,
+            -0.021449178457260132,
+            -0.01786910369992256,
+            0.005202050786465406,
+            -0.014844018965959549,
+            -0.0016180684324353933,
+            -0.015101972036063671,
+            -0.009614609181880951,
+            0.0008300417684949934,
+            0.0009849112248048186,
+            0.005475637502968311,
+            0.010443185456097126,
+            -0.0029332374688237906,
+            -0.018150506541132927,
+            0.014015442691743374,
+            0.04139753803610802,
+            -0.04386763274669647,
+            -0.017134329304099083,
+            0.012076887302100658,
+            0.010404101572930813,
+            -0.0036308830603957176,
+            0.010474452748894691,
+            -0.013554254546761513,
+            -0.003349479753524065,
+            -0.024388277903199196,
+            -0.01308524888008833,
+            -0.02513868547976017,
+            0.00749626848846674,
+            0.017931638285517693,
+            -0.008137242868542671,
+            0.03273657336831093,
+            0.030391545966267586,
+            -0.01877584680914879,
+            -0.0013884512009099126,
+            0.010583886876702309,
+            0.015281757339835167,
+            -0.029031431302428246,
+            0.016368286684155464,
+            -0.009911646135151386,
+            0.021167773753404617,
+            0.00968496035784483,
+            -0.021152140572667122,
+            -0.0146720502525568,
+            -0.016524622216820717,
+            0.007011629641056061,
+            -0.013874741271138191,
+            -3.935858694603667e-05,
+            0.027264844626188278,
+            0.015500626526772976,
+            -0.007484543602913618,
+            -0.014109243638813496,
+            0.018650779500603676,
+            0.013335385359823704,
+            0.019573155790567398,
+            -0.00540528679266572,
+            -0.0038399812765419483,
+            -0.007801122032105923,
+            -0.00946609117090702,
+            -0.018947815522551537,
+            0.00927067268639803,
+            -0.02984437346458435,
+            0.019057249650359154,
+            -0.01854134537279606,
+            -0.007437643129378557,
+            0.03273657336831093,
+            -0.04652532935142517,
+            -0.012913280166685581,
+            -0.013280668295919895,
+            0.02895326353609562,
+            0.015977447852492332,
+            0.004623610991984606,
+            0.009919462725520134,
+            -0.021449178457260132,
+            0.0033729299902915955,
+            -0.00321659492328763,
+            0.003099343739449978,
+            -0.016837291419506073,
+            -0.016399553045630455,
+            0.012413008138537407,
+            0.002137882634997368,
+            -0.004228864796459675,
+            0.0069061038084328175,
+            -0.005420919973403215,
+            -0.013538620434701443,
+            0.002679192926734686,
+            -0.0007513856398873031,
+            0.0015975494170561433,
+            0.008817300200462341,
+            0.002515041036531329,
+            0.01495345402508974,
+            0.01112324371933937,
+            -0.008660965599119663,
+            -0.005647606216371059,
+            -0.005647606216371059,
+            0.00088085065362975,
+            0.010153966024518013,
+            0.005284126847982407,
+            -0.006398014724254608,
+            -0.011631332337856293,
+            0.003376838518306613,
+            -0.007996541447937489,
+            0.026186130940914154,
+            -0.008418645709753036,
+            -0.042335547506809235,
+            0.002751498017460108,
+            -0.042366817593574524,
+            0.02354406751692295,
+            -0.02573275938630104,
+            -0.020651869475841522,
+            -0.009974180720746517,
+            -0.003695371327921748,
+            -0.019979627802968025,
+            0.009380106814205647,
+            0.007121064234524965,
+            -0.007941823452711105,
+            0.00792619027197361,
+            0.001134406658820808,
+            -0.015023804269731045,
+            0.0105291698127985,
+            -0.0101148821413517,
+            -0.010552620515227318,
+            0.005710139870643616,
+            0.0073868343606591225,
+            0.008848567493259907,
+            -0.006272946484386921,
+            -0.007472818251699209,
+            0.009505175054073334,
+            0.014734584838151932,
+            0.005229409784078598,
+            0.0053740194998681545,
+            0.020667502656579018,
+            0.013319752179086208,
+            -0.022496622055768967,
+            0.0023606601171195507,
+            -0.008019991219043732,
+            0.0038380271289497614,
+            0.003476502140983939,
+            -0.005932917352765799,
+            -0.009192504920065403,
+            0.003963095135986805,
+            -0.0003766699228435755,
+            4.7236411774065346e-05,
+            -0.0018340062815696,
+            -0.01614941656589508,
+            -0.005491270683705807,
+            -0.012006537057459354,
+            0.004221048206090927,
+            0.0300163421779871,
+            -0.006233862601220608,
+            -0.006398014724254608,
+            -0.019682591781020164,
+            0.018494443967938423,
+            0.03404978662729263,
+            0.017556432634592056,
+            -0.028671860694885254,
+            0.0005080891423858702,
+            -0.006417556665837765,
+            0.003300625132396817,
+            -0.025513891130685806,
+            0.022856194525957108,
+            0.018744580447673798,
+            0.009458274580538273,
+            0.011834568344056606,
+            -0.0025502166245132685,
+            0.005721865221858025,
+            -0.01548499334603548,
+            0.012858563102781773,
+            -0.004557168576866388,
+            -0.012014353647828102,
+            0.005006631836295128,
+            0.03942771628499031,
+            -0.015993082895874977,
+            -0.009677143767476082,
+            0.0030192218255251646,
+            0.02424757555127144,
+            -0.011889285407960415,
+            -0.007238315884023905,
+            0.011717316694557667,
+            0.009700593538582325,
+            0.004580618813633919,
+            0.00878603383898735,
+            0.06472273916006088,
+            0.0021867374889552593,
+            0.01638391986489296,
+            0.01283511333167553,
+            -0.021136507391929626,
+            -0.01919795200228691,
+            -0.005780491046607494,
+            -0.020354831591248512,
+            0.009012719616293907,
+            0.00900490302592516,
+            -0.011576615273952484,
+            -0.004068621434271336,
+            -0.020214131101965904,
+            0.0060736192390322685,
+            0.00731257488951087,
+            0.011936185881495476,
+            -0.0024192859418690205,
+            0.006870928220450878,
+            0.04274202138185501,
+            0.019448088482022285,
+            -0.0017196862027049065,
+            0.008105975575745106,
+            0.01419522799551487,
+            -0.009489541873335838,
+            -0.012420824728906155,
+            -0.002659651217982173,
+            0.0026107963640242815,
+            0.00877039972692728,
+            -0.010927824303507805,
+            0.033205579966306686,
+            -0.010419735684990883,
+            -0.015774212777614594,
+            0.011005992069840431,
+            -0.008066891692578793,
+            0.011013808660209179,
+            -0.012311390601098537,
+            0.017916003242135048,
+            -0.016665322706103325,
+            -0.0033221212215721607,
+            -0.0002916626981459558,
+            0.00019004486966878176,
+            0.017603334039449692,
+            0.0018095789710059762,
+            -0.010497902520000935,
+            0.0016815796261653304,
+            0.013851291500031948,
+            -0.008590614423155785,
+            0.003435464110225439,
+            -0.006261221133172512,
+            0.003959186840802431,
+            -0.011420279741287231,
+            -0.015555343590676785,
+            0.007191415410488844,
+            0.018822748214006424,
+            -0.026154864579439163,
+            -0.013210317119956017,
+            0.026576969772577286,
+            0.0071601481176912785,
+            0.002671376336365938,
+            0.013593338429927826,
+            -0.0058117578737437725,
+            0.0030954352114349604,
+            -0.016243217512965202,
+            -0.01777530275285244,
+            0.0061635118909180164,
+            -0.0048932889476418495,
+            0.019557522609829903,
+            -0.0024056066758930683,
+            0.004514176398515701,
+            -0.002837482374161482,
+            -0.008535897359251976,
+            0.019510623067617416,
+            0.015430275350809097,
+            -0.005170783959329128,
+            -0.006269038189202547,
+            0.011459363624453545,
+            -0.01827557571232319,
+            -0.020948905497789383,
+            0.003243953688070178,
+            0.020401732996106148,
+            -0.0065113576129078865,
+            0.004994906950742006,
+            -0.00030558629077859223,
+            -0.0012457953998818994,
+            -0.014664233662188053,
+            -0.009974180720746517,
+            -0.020902004092931747,
+            0.01919795200228691,
+            0.01336665265262127,
+            -0.02251225709915161,
+            0.0030192218255251646,
+            -0.005932917352765799,
+            0.02052680030465126,
+            -0.017587700858712196,
+            -0.003423738991841674,
+            0.033080510795116425,
+            -0.017759669572114944,
+            -0.004557168576866388,
+            -0.008231043815612793,
+            -0.014930003322660923,
+            0.016790391877293587,
+            0.03867730870842934,
+            0.0012037804117426276,
+            0.01014614850282669,
+            0.0029156499076634645,
+            -0.00452590174973011,
+            0.03558187186717987,
+            0.012647510506212711,
+            0.0004885472590103745,
+            -0.0032771748956292868,
+            -0.013835657387971878,
+            -0.02626429870724678,
+            0.0049440981820225716,
+            0.0026518343947827816,
+            -0.002016722923144698,
+            -5.20302819495555e-05,
+            -0.006014993414282799,
+            -0.007906648330390453,
+            0.006659876089543104,
+            -0.021668046712875366,
+            0.007797213736921549,
+            -0.016931092366576195,
+            -0.00276517728343606,
+            -0.00231375964358449,
+            -0.012796029448509216,
+            -0.005303668789565563,
+            0.010802756063640118,
+            -0.0016786482883617282,
+            -0.026123598217964172,
+            -0.03404978662729263,
+            -0.011670416221022606,
+            -0.012311390601098537,
+            -0.0007938892231322825,
+            0.00760179478675127,
+            -0.0267020370811224,
+            0.010224316269159317,
+            -0.0095364423468709,
+            -0.0014138557016849518,
+            0.0013679323019459844,
+            0.011983086355030537,
+            -0.006968637928366661,
+            0.0020362648647278547,
+            -0.02925029955804348,
+            -0.023434633389115334,
+            0.02135537751019001,
+            -0.003925965633243322,
+            0.012303573079407215,
+            -0.012538076378405094,
+            -0.004846388474106789,
+            -0.012592793442308903,
+            0.0177440345287323,
+            -0.017603334039449692,
+            -0.014007626101374626,
+            -0.014617333188652992,
+            -0.0005115089588798583,
+            -0.005366202909499407,
+            -0.009231588803231716,
+            -0.010083614848554134,
+            -0.010638604871928692,
+            -0.021152140572667122,
+            -0.014812751673161983,
+            -0.008442096412181854,
+            0.001930738682858646,
+            0.020151596516370773,
+            -0.006237770896404982,
+            0.026311200112104416,
+            0.02126157470047474,
+            0.00823886040598154,
+            -0.00520595908164978,
+            0.01425776258111,
+            -0.006937370635569096,
+            0.007679962553083897,
+            -0.00952080823481083,
+            0.0003239068028051406,
+            0.01671222411096096,
+            0.012850746512413025,
+            0.008410829119384289,
+            -0.004197597969323397,
+            0.002706551691517234,
+            0.01916668564081192,
+            0.015078521333634853,
+            0.01664968952536583,
+            0.005291943438351154,
+            0.008731315843760967,
+            0.010615154169499874,
+            0.0015399008989334106,
+            -0.005377927795052528,
+            0.0008955070516094565,
+            0.011701683513820171,
+            -0.011568798683583736,
+            -0.02005779556930065,
+            0.01329630147665739,
+            0.00930193904787302,
+            -0.018916549161076546,
+            0.009215954691171646,
+            0.0006575845764018595,
+            -0.009645876474678516,
+            -0.01836937665939331,
+            -0.014586065895855427,
+            -0.007805030792951584,
+            0.015430275350809097,
+            0.007918373681604862,
+            -0.022637324407696724,
+            0.01567259430885315,
+            0.010153966024518013,
+            -0.0015311070019379258,
+            -0.02712414227426052,
+            0.018619511276483536,
+            0.014054526574909687,
+            0.010638604871928692,
+            -0.04821375012397766,
+            -0.006069710943847895,
+            0.020730035379529,
+            -0.01329630147665739,
+            -0.026076696813106537,
+            -0.007625245023518801,
+            -0.008879834786057472,
+            -0.03902124613523483,
+            -0.0030758932698518038,
+            0.0029586420860141516,
+            -0.014844018965959549,
+            -0.003974820487201214,
+            0.015539710409939289,
+            0.00391228636726737,
+            0.018979083746671677,
+            -0.007781580090522766,
+            -0.01336665265262127,
+            -0.019213585183024406,
+            0.005202050786465406,
+            -0.010787122882902622,
+            -0.024263210594654083,
+            0.0027554063126444817,
+            -0.008465546183288097,
+            0.0022180043160915375,
+            0.01700926013290882,
+            0.042835820466279984,
+            0.01786910369992256,
+            -0.019917093217372894,
+            -0.025357555598020554,
+            -0.019307386130094528,
+            -0.007953548803925514,
+            0.009137787856161594,
+            0.007179690059274435,
+            -0.0021456994581967592,
+            0.003595707705244422,
+            0.006593433208763599,
+            0.031923629343509674,
+            0.0009536442230455577,
+            0.020886370912194252,
+            -3.169450064888224e-05,
+            -0.003814576892182231,
+            -0.002132020192220807,
+            -0.018979083746671677,
+            -0.0016053662402555346,
+            0.0029664586763828993,
+            0.013139965943992138,
+            0.004826846532523632,
+            0.005385744851082563,
+            0.016665322706103325,
+            -0.005628064274787903,
+            -0.0036035242956131697,
+            -0.017056161537766457,
+            -0.004428192041814327,
+            -0.004330482799559832,
+            0.03717648983001709,
+            0.020636234432458878,
+            -0.0199014600366354,
+            -0.0165871549397707,
+            -0.017431365326046944,
+            0.02194944955408573,
+            -0.00830921158194542,
+            -0.008997085504233837,
+            0.02099580504000187,
+            0.010239950381219387,
+            0.0015311070019379258,
+            0.0027280477806925774,
+            -0.006612975150346756,
+            0.011357746087014675,
+            0.02570149302482605,
+            0.012561526149511337,
+            -0.001964936964213848,
+            -0.004002179019153118,
+            -0.00016915948071982712,
+            -0.019323019310832024,
+            -0.01641518622636795,
+            0.00010302239388693124,
+            -0.012928914278745651,
+            0.025982895866036415,
+            -0.023106329143047333,
+            0.004314849153161049,
+            -0.01424212846904993,
+            -0.0024075608234852552,
+            -0.021996350958943367,
+            -0.011514080688357353,
+            0.014758034609258175,
+            -0.0237160362303257,
+            0.0054170116782188416,
+            -0.016759123653173447,
+            0.0010660100961104035,
+            0.011209228076040745,
+            -0.005159058608114719,
+            0.012561526149511337,
+            -0.00763697037473321,
+            0.004084255080670118,
+            0.004287490621209145,
+            0.00872349925339222,
+            -0.009145604446530342,
+            0.0015164506621658802,
+            0.005538171622902155,
+            -0.014578249305486679,
+            0.009239405393600464,
+            0.00925503857433796,
+            -0.004748679231852293,
+            0.03917757794260979,
+            0.015281757339835167,
+            0.00014949544856790453,
+            -0.009395739994943142,
+            -0.0038673398084938526,
+            -0.03140772506594658,
+            0.027014708146452904,
+            -0.012663144618272781,
+            0.021386643871665,
+            0.005983726587146521,
+            -0.01490655355155468,
+            -0.0016161142848432064,
+            -0.002688963897526264,
+            0.013194683939218521,
+            -0.009935096837580204,
+            0.0008969727205112576,
+            -0.013624604791402817,
+            0.011990902945399284,
+            0.00476822117343545,
+            -0.01016178261488676,
+            0.012295756489038467,
+            -0.015305208042263985,
+            0.0016571521991863847,
+            -0.007183598354458809,
+            0.0022844469640403986,
+            0.01588364690542221,
+            -0.0033416631631553173,
+            0.0028589784633368254,
+            -0.034675128757953644,
+            -0.015570977702736855,
+            -0.011529714800417423,
+            -8.244234777521342e-05,
+            0.007976999506354332,
+            -0.01754079945385456,
+            -0.028077786788344383,
+            -0.015977447852492332,
+            0.0099507300183177,
+            -0.0005926077719777822,
+            0.00028848715010099113,
+            -0.004658786579966545,
+            0.005225501023232937,
+            0.0009604838560335338,
+            -0.007468909956514835,
+            0.017525166273117065,
+            0.014875286258757114,
+            -0.026358099654316902,
+            -0.03081365115940571,
+            0.004564985167235136,
+            -0.0003722730034496635,
+            -0.027608782052993774,
+            -0.014077977277338505,
+            0.001066987169906497,
+            -0.0006072642281651497,
+            0.021199041977524757,
+            -0.020776936784386635,
+            0.0014588020276278257,
+            -0.00043260856182314456,
+            0.0030055425595492125,
+            -0.036394815891981125,
+            0.0030954352114349604,
+            -0.003144290065392852,
+            -0.0029273750260472298,
+            0.005565530154854059,
+            -0.01542245876044035,
+            0.0013278713449835777,
+            -0.008864200673997402,
+            -0.011428097262978554,
+            -0.025748392567038536,
+            -0.004592344164848328,
+            0.001534038339741528,
+            -8.647285721963271e-05,
+            0.023121964186429977,
+            -0.00622604601085186,
+            -0.019776392728090286,
+            0.002987954765558243,
+            -0.0019424638012424111,
+            -0.0030758932698518038,
+            0.018447542563080788,
+            0.020010894164443016,
+            -0.00651526590809226,
+            -0.0015105879865586758,
+            -0.0016258851392194629,
+            -0.001716754981316626,
+            -0.0004516619083005935,
+            0.039146311581134796,
+            -0.008551530539989471,
+            -0.006882653571665287,
+            0.0050261737778782845,
+            0.0016591063467785716,
+            -0.005444370210170746,
+            0.012100338004529476,
+            0.0015594427241012454,
+            0.007918373681604862,
+            0.0018388917669653893,
+            -0.010130515322089195,
+            -0.01824430748820305,
+            0.002528720535337925,
+            0.013851291500031948,
+            0.00581957446411252,
+            0.004197597969323397,
+            -0.014562616124749184,
+            0.008082525804638863,
+            -0.0075978864915668964,
+            -0.008778216317296028,
+            -0.010771489702165127,
+            0.007816756144165993,
+            -0.00444382568821311,
+            0.019526256248354912,
+            0.0003461357264313847,
+            0.021480444818735123,
+            0.022340288385748863,
+            -0.008442096412181854,
+            0.017024893313646317,
+            0.031485892832279205,
+            8.806063851807266e-05,
+            0.004744770936667919,
+            -0.011365562677383423,
+            -0.014679866842925549,
+            -0.013233767822384834,
+            0.03389345109462738,
+            0.004541534930467606,
+            -0.004267948679625988,
+            0.0018193499417975545,
+            -0.007316483184695244,
+            -0.01283511333167553,
+            0.012413008138537407,
+            0.003392471931874752,
+            -0.018556978553533554,
+            -0.004740862175822258,
+            0.009239405393600464,
+            0.006878744810819626,
+            0.012928914278745651,
+            0.02158987894654274,
+            -0.003869294188916683,
+            0.0028316196985542774,
+            0.004959731362760067,
+            0.029953807592391968,
+            -0.006374564487487078,
+            0.028593692928552628,
+            -0.0062494962476193905,
+            -0.0034823645837605,
+            -0.01107634324580431,
+            -0.015750762075185776,
+            -0.008426462300121784,
+            0.0029312833212316036,
+            0.0030602598562836647,
+            0.006186962127685547,
+            -0.01833810843527317,
+            0.006257312837988138,
+            0.02809341996908188,
+            0.0030055425595492125,
+            0.014734584838151932,
+            0.009278489276766777,
+            0.012741311453282833,
+            0.017321931198239326,
+            -0.00479167141020298,
+            -0.014851835556328297,
+            -0.003451097756624222,
+            0.020808203145861626,
+            -0.006683326326310635,
+            0.024654047563672066,
+            -0.0034745479933917522,
+            0.004350024741142988,
+            -0.00021972411195747554,
+            -0.0038810193073004484,
+            0.020479900762438774,
+            0.01500817108899355,
+            0.01664968952536583,
+            -0.001603411976248026,
+            0.00850463006645441,
+            0.0018349833553656936,
+            0.0017480220412835479,
+            -0.012913280166685581,
+            -0.016508987173438072,
+            0.01786910369992256,
+            0.0038477980997413397,
+            0.023168863728642464,
+            -0.022199586033821106,
+            -0.0007113247411325574,
+            0.006018901709467173,
+            0.016430821269750595,
+            0.004131155554205179,
+            0.010552620515227318,
+            -0.005444370210170746,
+            0.009544258937239647,
+            -0.005905558820813894,
+            0.0016913504805415869,
+            -0.01824430748820305,
+            -0.011600065045058727,
+            0.011631332337856293,
+            -0.00819195993244648,
+            0.010685505345463753,
+            0.001274131122045219,
+            0.009622426703572273,
+            -0.013843473978340626,
+            0.029672404751181602,
+            0.021402277052402496,
+            0.015234856866300106,
+            0.01499253697693348,
+            -0.0024779115337878466,
+            -0.007910557091236115,
+            -0.012420824728906155,
+            0.024560246616601944,
+            0.006800577510148287,
+            0.0029156499076634645,
+            -0.004475092515349388,
+            0.024153774604201317,
+            -0.0004384711501188576,
+            0.005151242017745972,
+            -0.0005886994185857475,
+            0.012967998161911964,
+            -0.01638391986489296,
+            0.008825116790831089,
+            -0.023559702560305595,
+            -0.013210317119956017,
+            -0.01913541741669178,
+            0.019588788971304893,
+            0.012741311453282833,
+            -0.0047721294686198235,
+            0.01543809287250042,
+            0.017603334039449692,
+            -0.0027319560758769512,
+            -0.027280477806925774,
+            -0.009786577895283699,
+            1.4351075151353143e-05,
+            -0.0032380910124629736,
+            -0.015727313235402107,
+            -0.0099507300183177,
+            0.002714368514716625,
+            -0.004885472357273102,
+            0.015547527000308037,
+            -0.0037637678906321526,
+            -0.024591512978076935,
+            -0.005686689633876085,
+            -0.007175781764090061,
+            -0.016555888578295708,
+            -0.013577704317867756,
+            0.008184143342077732,
+            0.005749223753809929,
+            -0.011506264097988605,
+            -0.018916549161076546,
+            0.018494443967938423,
+            -0.004693961702287197,
+            0.03214249759912491,
+            -0.015664778649806976,
+            -0.002808169461786747,
+            0.0053740194998681545,
+            0.01940118707716465,
+            0.0002865329443011433,
+            -0.0021183406934142113,
+            0.009638059884309769,
+            0.010615154169499874,
+            0.003232228569686413,
+            -0.007734679616987705,
+            0.0012917189160361886,
+            -0.0002904413267970085,
+            -0.019385553896427155,
+            0.005893833935260773,
+            -0.0005007609142921865,
+            -0.014070160686969757,
+            0.003351433901116252,
+            0.014335930347442627,
+            0.009622426703572273,
+            0.005452187266200781,
+            -0.02789018489420414,
+            0.012897646985948086,
+            -0.008582797832787037,
+            -0.011256128549575806,
+            -0.0016356561100110412,
+            0.01851007714867592,
+            -0.021965082734823227,
+            0.027937084436416626,
+            0.01419522799551487,
+            -0.00941919069737196,
+            0.005166875664144754,
+            0.019948361441493034,
+            0.017024893313646317,
+            0.011474997736513615,
+            -0.023794203996658325,
+            -0.004221048206090927,
+            0.01425776258111,
+            0.006304213311523199,
+            -0.0006717524374835193,
+            -0.0020206314511597157,
+            0.02463841438293457,
+            0.0034139680210500956,
+            -0.0018310750601813197,
+            -0.0070272632874548435,
+            -0.0018193499417975545,
+            0.009309755638241768,
+            0.021183408796787262,
+            0.009927279315888882,
+            0.018947815522551537,
+            -0.00131028366740793,
+            0.01578984595835209,
+            -0.004295307211577892,
+            -0.009669327177107334,
+            0.014844018965959549,
+            -0.0005882108816877007,
+            -0.009841295890510082,
+            0.01668095588684082,
+            0.008418645709753036,
+            0.012373924255371094,
+            -0.0008901330293156207,
+            -0.009872562251985073,
+            0.02871876023709774,
+            0.015109788626432419,
+            -0.0008701026090420783,
+            0.02046426571905613,
+            -0.02685837261378765,
+            0.024951083585619926,
+            0.012772578746080399,
+            0.0160868838429451,
+            -0.022340288385748863,
+            -0.004475092515349388,
+            0.00048488311585970223,
+            -0.018259940668940544,
+            -0.008700049482285976,
+            0.0037950349505990744,
+            -0.009989813901484013,
+            -0.0020264938939362764,
+            -0.037395358085632324,
+            0.01303053181618452,
+            -0.009427007287740707,
+            0.003249816130846739,
+            -0.015227040275931358,
+            -0.001274131122045219,
+            0.00532321073114872,
+            0.008543713949620724,
+            -0.007746404968202114,
+            -0.030469713732600212,
+            -0.004744770936667919,
+            0.016759123653173447,
+            -0.00835611205548048,
+            -0.019354287534952164,
+            0.024716582149267197,
+            -0.0004824404022656381,
+            -0.021699313074350357,
+            0.0041155219078063965,
+            -0.010544802993535995,
+            -0.007363383658230305,
+            -0.005518629681318998,
+            -0.008105975575745106,
+            -0.02424757555127144,
+            0.013843473978340626,
+            0.0019102196674793959,
+            -0.02046426571905613,
+            0.010232132859528065,
+            -0.01378094032406807,
+            0.004111613612622023,
+            0.008551530539989471,
+            -0.008895467966794968,
+            -0.011748583987355232,
+            0.027749482542276382,
+            0.022324655205011368,
+            -0.017618967220187187,
+            -0.015742946416139603,
+            9.386213787365705e-05,
+            -0.025060519576072693,
+            0.023700403049588203,
+            -0.00470959534868598,
+            0.0019825247582048178,
+            -0.011131060309708118,
+            0.011600065045058727,
+            0.010826206766068935,
+            0.012624060735106468,
+            -0.003554669674485922,
+            -0.0005579209537245333,
+            0.026514435186982155,
+            -0.009090886451303959,
+            0.003509723348543048,
+            -8.564843483327422e-06,
+            -0.012413008138537407,
+            0.02384110540151596,
+            -0.004486817866563797,
+            0.023950539529323578,
+            -0.006237770896404982,
+            -0.0006483022007159889,
+            -0.014961270615458488,
+            0.0026440175715833902,
+            -0.0031462442129850388,
+            -0.0069842711091041565,
+            -0.010286850854754448,
+            0.01283511333167553,
+            0.01213942188769579,
+            0.006835752632468939,
+            -0.010255583561956882,
+            0.0012184367515146732,
+            0.00038790650432929397,
+            -0.016696590930223465,
+            0.0023489349987357855,
+            -0.019713858142495155,
+            0.0009829570772126317,
+            -0.03329937905073166,
+            0.009708410128951073,
+            0.04430537298321724,
+            0.016634056344628334,
+            -0.008129426278173923,
+            -0.003537081880494952,
+            -0.01611815020442009,
+            0.005225501023232937,
+            -0.005706231575459242,
+            -0.011725133284926414,
+            -0.0136793227866292,
+            -0.01605561561882496,
+            0.0035312194377183914,
+            0.017916003242135048,
+            -0.0026518343947827816,
+            -0.020745670422911644,
+            0.02298126183450222,
+            -0.014500081539154053,
+            -0.026655137538909912,
+            -0.01754079945385456,
+            -0.02096453867852688,
+            0.007324300240725279,
+            0.020745670422911644,
+            -0.00579612422734499,
+            0.026576969772577286,
+            0.012428641319274902,
+            0.0028140321373939514,
+            -0.0010904374066740274,
+            0.01813487336039543,
+            -0.011349929496645927,
+            0.00452590174973011,
+            -0.0013816115679219365,
+            -0.020354831591248512,
+            0.0014783439692109823,
+            -0.008715682663023472,
+            -0.005600705277174711,
+            0.001242864178493619,
+            -0.005624155513942242,
+            -0.00260297954082489,
+            -0.0075978864915668964,
+            -0.017728401347994804,
+            0.02327829785645008,
+            -0.02348153479397297,
+            -0.002851161640137434,
+            -0.0005007609142921865,
+            -0.009591159410774708,
+            0.0111076096072793,
+            0.023950539529323578,
+            0.01261624414473772,
+            -0.015430275350809097,
+            0.01833810843527317,
+            -0.00037154016899876297,
+            0.026545701548457146,
+            -0.004662694875150919,
+            0.005585072096437216,
+            0.012694410979747772,
+            0.0008129425696097314,
+            0.01685292460024357,
+            -0.00710543105378747,
+            -0.011029441840946674,
+            0.0042913989163935184,
+            -0.00018686932162381709,
+            -0.021402277052402496,
+            -0.009755311533808708,
+            -0.0023782479111105204,
+            0.012217588722705841,
+            -0.017165595665574074,
+            0.015727313235402107,
+            -0.012741311453282833,
+            -0.001535992487333715,
+            -0.00387124833650887,
+            0.01397635880857706,
+            -0.04855768755078316,
+            -0.004963639657944441,
+            -0.008199776522815228,
+            0.03489399701356888,
+            -0.012444275431334972,
+            0.022434089332818985,
+            0.0040002246387302876,
+            -0.007621336728334427,
+            0.011834568344056606,
+            -0.010130515322089195,
+            -0.0004521504743024707,
+            0.015070704743266106,
+            -0.007347750477492809,
+            0.0017431365558877587,
+            0.016993626952171326,
+            -0.01836937665939331,
+            0.01674349047243595,
+            -0.019541889429092407,
+            0.0031071603298187256,
+            -0.012061254121363163,
+            -0.00640973960980773,
+            0.021386643871665,
+            0.0267020370811224,
+            -0.005815666168928146,
+            0.007398559246212244,
+            -0.009184688329696655,
+            0.007691687438637018,
+            -0.004252315033227205,
+            0.011545347981154919,
+            0.0010748038766905665,
+            -0.010232132859528065,
+            -0.0019317157566547394,
+            -0.009661509655416012,
+            0.020651869475841522,
+            -0.013382285833358765,
+            -0.012491175904870033,
+            -0.009559892117977142,
+            -0.004803396295756102,
+            -0.03351824730634689,
+            -0.005983726587146521,
+            -0.008137242868542671,
+            -0.012483359314501286,
+            -0.008426462300121784,
+            0.012733494862914085,
+            0.0010884832590818405,
+            0.0024290569126605988,
+            -0.009544258937239647,
+            -0.01144373044371605,
+            -0.013866924680769444,
+            0.015977447852492332,
+            0.004052987787872553,
+            -0.004111613612622023,
+            0.0038380271289497614,
+            0.00037056306609883904,
+            0.004795579705387354,
+            -0.00015438091941177845,
+            -0.00807470828294754,
+            -0.0032830373384058475,
+            0.012381740845739841,
+            0.006398014724254608,
+            -0.021011440083384514,
+            0.010388468392193317,
+            0.01893218234181404,
+            -0.0033572965767234564,
+            -0.006057985592633486,
+            -0.0064605483785271645,
+            -0.013382285833358765,
+            -0.004557168576866388,
+            0.0017665867926552892,
+            0.020917639136314392,
+            -0.01661842316389084,
+            -0.0013845429057255387,
+            0.019541889429092407,
+            -0.0027749482542276382,
+            0.023028161376714706,
+            0.0025384915061295033,
+            0.009935096837580204,
+            0.00988037884235382,
+            0.011740767396986485,
+            0.0062924884259700775,
+            -0.010951275005936623,
+            -0.02162114717066288,
+            0.001587778446264565,
+            -0.022434089332818985,
+            0.01588364690542221,
+            0.0023333015851676464,
+            -0.005100433249026537,
+            -0.021840015426278114,
+            -0.012506809085607529,
+            0.009512991644442081,
+            -0.02971930429339409,
+            -0.0007259811973199248,
+            -0.00737120071426034,
+            0.014265579171478748,
+            -0.007879289798438549,
+            0.0033866094890981913,
+            0.006042351946234703,
+            -0.007566619664430618,
+            0.00028799858409911394,
+            -0.0024505530018359423,
+            -0.01800980605185032,
+            -0.0045962524600327015,
+            0.00026381551288068295,
+            0.010372835211455822,
+            0.0038575688377022743,
+            0.015539710409939289,
+            0.023231398314237595,
+            0.031689126044511795,
+            -0.009653693065047264,
+            0.0018340062815696,
+            0.0009956592693924904,
+            0.016806025058031082,
+            0.005616338923573494,
+            0.0006727295112796128,
+            0.001513519324362278,
+            0.02510741911828518,
+            -0.011670416221022606,
+            -0.015899281948804855,
+            0.01596963219344616,
+            -0.020182862877845764,
+            -0.00010515979374758899,
+            -0.02029229700565338,
+            0.012084703892469406,
+            0.0011334295850247145,
+            0.013335385359823704,
+            0.013796573504805565,
+            -0.001073826802894473,
+            0.00819195993244648,
+            0.024794749915599823,
+            -0.005995451472699642,
+            0.0027358646038919687,
+            0.027233576402068138,
+            -0.00872349925339222,
+            -0.004779946058988571,
+            -0.000544730166438967,
+            0.002515041036531329,
+            0.00041257814154960215,
+            0.010599520988762379,
+            0.017822202295064926,
+            -0.027608782052993774,
+            -0.006034535355865955,
+            0.01240519154816866,
+            0.001964936964213848,
+            -0.006558258086442947,
+            0.004705687053501606,
+            0.0033553424291312695,
+            0.014851835556328297,
+            0.0003659218782559037,
+            0.01688419282436371,
+            -0.004819029942154884,
+            -0.02543572336435318,
+            -0.00930193904787302,
+            0.005835208110511303,
+            -0.012506809085607529,
+            -0.0032263658940792084,
+            -0.0034999523777514696,
+            -0.009512991644442081,
+            0.014265579171478748,
+            -0.008285760879516602,
+            0.0012340702814981341,
+            0.009395739994943142,
+            0.003802851540967822,
+            -0.007418101187795401,
+            -0.00215156190097332,
+            0.011881468817591667,
+            -0.012678777799010277,
+            0.004752587527036667,
+            0.022309020161628723,
+            0.003931828308850527,
+            -0.016821658238768578,
+            -0.018025439232587814,
+            0.006323755253106356,
+            0.03017267771065235,
+            -0.012194138951599598,
+            -0.0028765660244971514,
+            0.018916549161076546,
+            -0.0006644242093898356,
+            0.008215410634875298,
+            -0.002986000617966056,
+            -0.015023804269731045,
+            -0.03773929551243782,
+            -0.0019072884460911155,
+            -0.0018975174752995372,
+            0.005268493201583624,
+            -0.0003336777735967189,
+            -0.02374730445444584,
+            -0.007199232000857592,
+            -0.0103650176897645,
+            -0.002759314840659499,
+            -0.011021625250577927,
+            0.002632292453199625,
+            0.0008007289143279195,
+            -0.023684769868850708,
+            0.014031076803803444,
+            -0.012319207191467285,
+            0.010888740420341492,
+            -0.006100977770984173,
+            0.0190728846937418,
+            -0.0025169954169541597,
+            -0.0015457634581252933,
+            -0.008950185030698776,
+            0.006175236776471138,
+            0.013788756914436817,
+            0.0068513862788677216,
+            -0.011998720467090607,
+            0.0012467725900933146,
+            0.022856194525957108,
+            0.011818934231996536,
+            0.006179145537316799,
+            0.006683326326310635,
+            -0.011310845613479614,
+            -0.017884736880660057,
+            0.0020108604803681374,
+            -0.010451002046465874,
+            0.0033260295167565346,
+            0.007781580090522766,
+            0.014851835556328297,
+            0.005663239397108555,
+            -0.013694955967366695,
+            0.004760404117405415,
+            0.0103650176897645,
+            0.01694672554731369,
+            -0.014593882486224174,
+            -0.0034725936129689217,
+            -0.00925503857433796,
+            0.012483359314501286,
+            0.01916668564081192,
+            0.0042366813868284225,
+            0.006147878244519234,
+            0.004092071671038866,
+            0.015774212777614594,
+            -0.004228864796459675,
+            0.008332661353051662,
+            -0.013741856440901756,
+            -0.016493353992700577,
+            -0.019182318821549416,
+            0.004932372830808163,
+            0.00021899129205849022,
+            0.02745244652032852,
+            0.004557168576866388,
+            0.01976075768470764,
+            0.007457185070961714,
+            0.012749128974974155,
+            0.0020812111906707287,
+            0.022434089332818985,
+            -0.0009277511853724718,
+            -0.024153774604201317,
+            0.01206907071173191,
+            -0.021871281787753105,
+            -0.03070421703159809,
+            0.006843569688498974,
+            0.006612975150346756,
+            0.0381457693874836,
+            -0.022590424865484238,
+            0.011834568344056606,
+            -0.003156015183776617,
+            -0.023387733846902847,
+            -0.0022844469640403986,
+            -0.001389428274706006,
+            -0.0038380271289497614,
+            0.026358099654316902,
+            -0.0005198142607696354,
+            0.006315938662737608,
+            0.003521448466926813,
+            0.009810028597712517,
+            -0.00983347836881876,
+            0.021761847659945488,
+            0.01632138527929783,
+            0.018150506541132927,
+            0.005788307636976242,
+            -0.01112324371933937,
+            0.016993626952171326,
+            0.0010904374066740274,
+            -0.004955823067575693,
+            0.004568893928080797,
+            -0.005022265482693911,
+            0.0028257572557777166,
+            -0.0005564552848227322,
+            0.01474240142852068,
+            0.008707866072654724,
+            -0.015133239328861237,
+            0.010036714375019073,
+            -0.004568893928080797,
+            0.007898831740021706,
+            0.01949498802423477,
+            0.011975269764661789,
+            -0.002401698147878051,
+            0.009614609181880951,
+            -0.04430537298321724,
+            0.006140061654150486,
+            -0.03339318186044693,
+            -0.01233484037220478,
+            0.00270264339633286,
+            -0.017400098964571953,
+            0.0021105241030454636,
+            -0.01042755227535963,
+            -0.0175720676779747,
+            -0.006859203334897757,
+            0.019307386130094528,
+            0.01877584680914879,
+            0.004092071671038866,
+            0.00378917227499187,
+            0.010443185456097126,
+            -0.010873107239603996,
+            -0.013812207616865635,
+            -0.01394509244710207,
+            -0.012944547459483147,
+            -0.02049553394317627,
+            -0.0032556788064539433,
+            0.0009736746433191001,
+            -0.006441006902605295,
+            -0.002018677070736885,
+            0.009129970334470272,
+            -0.022043250501155853,
+            -0.010974724777042866,
+            0.014109243638813496,
+            -0.003392471931874752,
+            0.0006732181063853204,
+            -0.019119784235954285,
+            0.004256223328411579,
+            -0.003243953688070178,
+            0.00798481609672308,
+            -0.00581957446411252,
+            0.02431011013686657,
+            -0.01382784079760313,
+            -0.005053532775491476,
+            0.016821658238768578,
+            0.005491270683705807,
+            -0.007097613997757435,
+            -0.013546437956392765,
+            -0.03439372405409813,
+            0.025826560333371162,
+            -0.009770944714546204,
+            -0.01727502979338169,
+            -0.013288484886288643,
+            0.014984720386564732,
+            -0.009333206340670586,
+            -0.0018574565183371305,
+            0.011005992069840431,
+            0.0014988628681749105,
+            0.019713858142495155,
+            0.024826016277074814,
+            -0.010349384509027004,
+            0.0014206953346729279,
+            -0.01542245876044035,
+            0.012569342739880085,
+            -0.024700947105884552,
+            -0.009966363199055195,
+            0.013874741271138191,
+            0.012029986828565598,
+            0.002149607753381133,
+            -0.01661842316389084,
+            0.009661509655416012,
+            0.008363928645849228,
+            0.013906008563935757,
+            0.016993626952171326,
+            0.005467820446938276,
+            -0.010873107239603996,
+            -0.007914464920759201,
+            -0.024951083585619926,
+            0.0240443404763937,
+            0.005909467115998268,
+            -0.010974724777042866,
+            0.04368003085255623,
+            0.01993272639811039,
+            0.011998720467090607,
+            -0.01783783733844757,
+            -0.010552620515227318,
+            0.01664968952536583,
+            0.01218632236123085,
+            0.010802756063640118,
+            -0.00024036523245740682,
+            0.004932372830808163,
+            -0.0024603239726275206,
+            0.022168319672346115,
+            0.018213041126728058,
+            0.010185232385993004,
+            -0.05137171968817711,
+            0.014976903796195984,
+            0.002907833084464073,
+            -0.00520595908164978,
+            0.01685292460024357,
+            0.019948361441493034,
+            0.01217068824917078,
+            0.009606792591512203,
+            0.019854558631777763,
+            -0.00018735785852186382,
+            0.02194944955408573,
+            -0.009083069860935211,
+            -0.015758579596877098,
+            -0.00031950988341122866,
+            -0.013804391026496887,
+            -0.002776902401819825,
+            -0.019291752949357033,
+            0.009903829544782639,
+            0.00450245151296258,
+            -0.005237226374447346,
+            0.0240443404763937,
+            0.004209322854876518,
+            -0.029078330844640732,
+            -0.007656512316316366,
+            -0.029031431302428246,
+            0.0013796574203297496,
+            -0.019854558631777763,
+            -0.003818485187366605,
+            -0.014859652146697044,
+            -0.014421914704144001,
+            -0.026842739433050156,
+            -0.012241039425134659,
+            -0.00030363211408257484,
+            0.015023804269731045,
+            -0.001865273341536522,
+            -0.015633512288331985,
+            0.004529810044914484,
+            -0.03839590400457382,
+            -0.008582797832787037,
+            0.03645734861493111,
+            0.00819195993244648,
+            0.031954895704984665,
+            -0.008614065125584602,
+            0.01446099765598774,
+            -0.0010142240207642317,
+            -0.04261695221066475,
+            -0.0031657861545681953,
+            -0.003800897393375635,
+            0.00893455184996128,
+            0.007488451898097992,
+            0.0048932889476418495,
+            -0.012147238478064537,
+            0.016368286684155464,
+            -0.0014314433792605996,
+            0.00898926891386509,
+            0.0030524430330842733,
+            0.005151242017745972,
+            -0.010966908186674118,
+            0.010521353222429752,
+            -0.016368286684155464,
+            0.004248406738042831,
+            0.0019346470944583416,
+            0.004174147732555866,
+            -0.016868559643626213,
+            -0.016430821269750595,
+            -0.0037051422987133265,
+            -0.004576710518449545,
+            -0.012889830395579338,
+            -0.002970367204397917,
+            -0.011670416221022606,
+            -0.006655967328697443,
+            -0.008856384083628654,
+            -0.01431247964501381,
+            -0.011146693490445614,
+            0.0049362811259925365,
+            0.0027573604602366686,
+            0.008840750902891159,
+            0.014234311878681183,
+            0.003171648597344756,
+            0.007961365394294262,
+            0.02573275938630104,
+            0.001182284322567284,
+            -0.001272176974453032,
+            -0.01671222411096096,
+            0.001556511502712965,
+            -0.030157042667269707,
+            0.014453181065618992,
+            0.011748583987355232,
+            -0.025748392567038536,
+            0.006093161180615425,
+            -1.3885428415960632e-05,
+            0.0028570243157446384,
+            0.002175012370571494,
+            -0.0007220727857202291,
+            -0.014062343165278435,
+            0.003333846339955926,
+            -0.0180723387748003,
+            -0.00032488390570506454,
+            0.012467725202441216,
+            0.0401155911386013,
+            3.362579036547686e-06,
+            -0.0008657057187519968,
+            0.014429731294512749,
+            0.007707321085035801,
+            0.022074518725275993,
+            -0.01596963219344616,
+            -0.007015538401901722,
+            -0.03129829093813896,
+            -0.005342752672731876,
+            -0.005925100762397051,
+            -0.02198071777820587,
+            0.012241039425134659,
+            -0.0012545892968773842,
+            -0.010341567918658257,
+            -0.01366368867456913,
+            -0.02238718792796135,
+            0.012217588722705841,
+            0.02241845615208149,
+            0.008825116790831089,
+            0.007183598354458809,
+            0.008707866072654724,
+            -0.024450812488794327,
+            -0.020808203145861626,
+            0.029562970623373985,
+            0.013616788201034069,
+            -0.009231588803231716,
+            -0.01664968952536583,
+            -0.013585521839559078,
+            0.0031247481238096952,
+            -0.0006238748319447041,
+            0.0004951425944454968,
+            0.06219010800123215,
+            -0.018760213628411293,
+            0.0021847831085324287,
+            -0.034768927842378616,
+            -0.000998102012090385,
+            -0.009208138100802898,
+            0.013179049827158451,
+            0.005135608371347189,
+            -0.030188310891389847,
+            0.01618068479001522,
+            -0.030626049265265465,
+            -0.007257857825607061,
+            0.0015750762540847063,
+            -0.008137242868542671,
+            -0.010615154169499874,
+            -0.0023919271770864725,
+            -0.01827557571232319,
+            0.00013215202488936484,
+            -0.013327568769454956,
+            0.0021144323982298374,
+            0.002075348747894168,
+            -0.024700947105884552,
+            0.0015144963981583714,
+            0.006120519712567329,
+            -0.0046314275823533535,
+            -0.004350024741142988,
+            -0.019510623067617416,
+            -0.02019849605858326,
+            -0.014156145043671131,
+            -0.002032356569543481,
+            0.018025439232587814,
+            -0.02424757555127144,
+            0.01697799377143383,
+            -0.0032087783329188824,
+            0.004721320234239101,
+            0.003218549070879817,
+            0.008598431013524532,
+            0.018119240179657936,
+            0.01336665265262127,
+            0.0007044851081445813,
+            -0.010466636158525944,
+            0.01797853782773018,
+            -0.017947271466255188,
+            -0.02188691683113575,
+            0.001725548878312111,
+            0.0034081055782735348,
+            4.7083740355446935e-05,
+            0.0025932088028639555,
+            0.0045962524600327015,
+            0.013569887727499008,
+            0.007500177249312401,
+            -0.016196317970752716,
+            -0.002585391979664564,
+            -0.0027827650774270296,
+            0.029828740283846855,
+            0.0024603239726275206,
+            0.03276783972978592,
+            0.0012545892968773842,
+            0.015782030299305916,
+            -0.0014167869230732322,
+            -0.014570432715117931,
+            -0.017431365326046944,
+            -0.01314778346568346,
+            0.018728947266936302,
+            0.0016972130397334695,
+            0.007629153784364462,
+            -0.023700403049588203,
+            -0.01596963219344616,
+            0.003118885448202491,
+            -0.004049079492688179,
+            -0.01704052835702896,
+            0.0180723387748003,
+            -0.0177440345287323,
+            0.011521898210048676,
+            0.00032415110035799444,
+            -0.015023804269731045,
+            -0.017525166273117065,
+            0.014367196708917618,
+            0.0016923275543376803,
+            -0.000899904000107199,
+            0.012076887302100658,
+            -0.013257217593491077,
+            -0.011826751753687859,
+            0.002767131431028247,
+            0.00219259993173182,
+            -0.006824027746915817,
+            0.013616788201034069,
+            -0.016430821269750595,
+            0.005604614038020372,
+            -0.006824027746915817,
+            -0.0017802660586312413,
+            -0.010849656537175179,
+            0.013921641744673252,
+            0.012921096757054329,
+            -0.0015057026175782084,
+            0.012108154594898224,
+            0.012053437530994415,
+            0.003245907835662365,
+            0.014703317545354366,
+            0.005835208110511303,
+            -0.006378472782671452,
+            -0.010028897784650326,
+            0.0021124782506376505,
+            0.008676598779857159,
+            -0.004088163375854492,
+            -0.00383021030575037,
+            0.033080510795116425,
+            0.01668095588684082,
+            -0.017759669572114944,
+            0.021496077999472618,
+            -0.0038946985732764006,
+            0.007179690059274435,
+            -0.03186109662055969,
+            0.0004763335455209017,
+            0.013655872084200382,
+            -0.010873107239603996,
+            0.008223227225244045,
+            0.007816756144165993,
+            0.028171587735414505,
+            0.012194138951599598,
+            0.00819195993244648,
+            -0.0050769830122590065,
+            0.0021105241030454636,
+            0.013186866417527199,
+            0.0027964443434029818,
+            -0.002470094710588455,
+            -0.0026967807207256556,
+            0.01996399462223053,
+            -0.016337020322680473,
+            0.020370464771986008,
+            0.013335385359823704,
+            -0.029375368729233742,
+            0.005217684432864189,
+            -0.019010350108146667,
+            0.003961140755563974,
+            -0.006319846957921982,
+            -0.0037168674170970917,
+            -0.021386643871665,
+            0.007339933421462774,
+            0.02945353463292122,
+            -0.011772033758461475,
+            0.029734939336776733,
+            0.004326574504375458,
+            0.02016722969710827,
+            -0.009450457990169525,
+            -0.006198687478899956,
+            -0.009208138100802898,
+            0.0010054302401840687,
+            -0.008371745236217976,
+            0.011154510080814362,
+            0.0002304721565451473,
+            0.005846933461725712,
+            0.01424212846904993,
+            0.004060804378241301,
+            -0.003523402614519,
+            0.028828194364905357,
+            0.005925100762397051,
+            0.008231043815612793,
+            -0.013007081113755703,
+            0.005159058608114719,
+            -0.007863656617701054,
+            0.0038243476301431656,
+            0.0019854558631777763,
+            -0.008637514896690845,
+            9.306825086241588e-05,
+            -0.0021730579901486635,
+            0.013585521839559078,
+            0.009614609181880951,
+            0.013960725627839565,
+            -0.004678328521549702,
+            -0.008121608756482601,
+            0.025685859844088554,
+            0.001968845259398222,
+            -0.004310940857976675,
+            0.004729137290269136,
+            0.004881564062088728,
+            0.016931092366576195,
+            0.012960180640220642,
+            0.0006043329485692084,
+            -0.0021574245765805244,
+            -0.025185586884617805,
+            0.01212378777563572,
+            -0.018353741616010666,
+            -0.00039841028046794236,
+            -0.005698414985090494,
+            -0.022731125354766846,
+            0.013053981587290764,
+            -0.005248951260000467,
+            -0.003607432823628187,
+            -0.005542079918086529,
+            0.022965628653764725,
+            0.004064713139086962,
+            -0.014343746937811375,
+            -0.00399240804836154,
+            0.007468909956514835,
+            0.02765568159520626,
+            -0.0022121418733149767,
+            -0.00042381472303532064,
+            -0.010622970759868622,
+            -0.015289573930203915,
+            0.012749128974974155,
+            -0.015742946416139603,
+            -0.013624604791402817,
+            0.003101297887042165
+          ],
+          "title": "DeepSeekMoE: Achieving Ultimate Expert Specialization in Mixture-of-Experts Architectures",
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "Mixture-of-Experts (MoE)",
+            "expert specialization",
+            "fine-grained expert segmentation",
+            "shared expert isolation"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "767dcc8f-1e79-439d-92f3-8f20092ad68e",
+        "properties": {
+          "page_content": "2. Preliminaries: Mixture-Of-Experts For Transformers\n\nWe first introduce a generic MoE architecture commonly used in Transformer language models. A standard Transformer language model is constructed by stacking layers of standard Transformer blocks, where each block can be represented as follows:\n\n$$\\mathbf{u}{1:T}^{l}=\\text{Self-Att}\\left(\\mathbf{h}{1:T}^{l-1}\\right)+\\mathbf{h}{1:T}^{l-1},\\tag{1}$$ $$\\mathbf{h}{t}^{l}=\\text{FFN}\\left(\\mathbf{u}{t}^{l}\\right)+\\mathbf{u}{t}^{l},\\tag{2}$$\n\nwhere denotes the sequence length, Self-Att(·) denotes the self-attention module, FFN(·) denotes the Feed-Forward Network (FFN), u 1: ∈ R× are the hidden states of all tokens after the -th attention module, and h\n\n∈ Ris the output hidden state of the -th token after the -th Transformer block. For brevity, we omit the layer normalization in the above formulations.\n\nA typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022). An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. Then, each token will be assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts. If the -th FFN is substituted with an MoE layer, the computation for its output hidden state h\n\nis expressed as: where denotes the total number of experts, FFN(·) is the -th expert FFN, , denotes the gate value for the -th expert, , denotes the token-to-expert affinity, Topk(·, ) denotes the set comprising highest affinity scores among those calculated for the -th token and all experts, and e\n\nis the centroid of the -th expert in the -th layer. Note that ,is sparse, indicating that only out of gate values are nonzero. This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only experts. Also, in the above formulations, we omit the layer normalization operation for brevity.\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{N}\\left(g_{i,t}\\operatorname{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\operatorname{Topk}({s_{j,t}|1\\leqslant j\\leqslant N},K),\\ 0,&\\text{otherwise},\\end{cases}$$ $$s_{i,t}=\\operatorname{Softmax}{i}\\left(\\mathbf{u}{t}^{l}\\,\\mathbf{e}_{i}^{l}\\right),$$\n\n$$(3)$$\n\n(4) $\\binom{4}{5}$ .\n\n",
+          "title": "Preliminaries: Mixture-Of-Experts For Transformers",
+          "embedding": [
+            0.0012184951920062304,
+            -0.02724841982126236,
+            -0.02786005660891533,
+            -0.019281849265098572,
+            0.028028255328536034,
+            0.023196324706077576,
+            0.0037118708714842796,
+            0.03721809759736061,
+            -0.0036163025069981813,
+            0.03367060422897339,
+            -0.006009331438690424,
+            0.0015032885130494833,
+            -0.013440718874335289,
+            0.0020107559394091368,
+            -0.02350214309990406,
+            0.012936118058860302,
+            0.0041056121699512005,
+            0.036545298993587494,
+            0.003880070988088846,
+            -0.017630429938435555,
+            0.0016657545929774642,
+            -0.0020566286984831095,
+            -0.0359642431139946,
+            0.003516911529004574,
+            0.014763383194804192,
+            -0.0010483836522325873,
+            0.04584217816591263,
+            -0.011781653389334679,
+            -0.016330702230334282,
+            0.026667363941669464,
+            0.02402203530073166,
+            -0.003767300397157669,
+            0.002329953946173191,
+            -0.03529144451022148,
+            -0.05529196560382843,
+            -0.002939679427072406,
+            0.046514976769685745,
+            0.04406842961907387,
+            -0.03510795161128044,
+            0.027737729251384735,
+            0.019327722489833832,
+            0.008410005830228329,
+            -0.016453029587864876,
+            -0.02949618361890316,
+            -0.015283274464309216,
+            -0.009480370208621025,
+            -0.03651471808552742,
+            0.004227939527481794,
+            0.014755737967789173,
+            -0.027324873954057693,
+            -0.015206819400191307,
+            -0.008906960487365723,
+            0.009029287844896317,
+            0.025061817839741707,
+            -0.05318181961774826,
+            0.03244733065366745,
+            0.0012739247176796198,
+            0.0359642431139946,
+            0.053120654076337814,
+            0.006900027394294739,
+            0.025627581402659416,
+            -0.04003162682056427,
+            -0.020000522956252098,
+            -0.0015720976516604424,
+            0.040092792361974716,
+            0.0058067268691957,
+            0.019817031919956207,
+            -0.02495478093624115,
+            0.0116134537383914,
+            0.033640023320913315,
+            -0.03782973438501358,
+            0.008088896982371807,
+            -0.015275629237294197,
+            -0.018976030871272087,
+            -0.004931321833282709,
+            0.008203578181564808,
+            0.015994302928447723,
+            0.029465602710843086,
+            0.035230278968811035,
+            0.004912207834422588,
+            -0.02949618361890316,
+            0.027080219238996506,
+            -0.0006469970685429871,
+            0.021086178719997406,
+            0.028288202360272408,
+            -0.03168278560042381,
+            -0.01886899583041668,
+            -0.051530398428440094,
+            0.005971104372292757,
+            0.02536763623356819,
+            -0.023884417489171028,
+            0.01834910362958908,
+            -0.003622036660090089,
+            0.015084492042660713,
+            0.0006288390723057091,
+            0.038043808192014694,
+            0.055597785860300064,
+            -0.0010588961886242032,
+            -0.02565816417336464,
+            -0.011582871899008751,
+            0.010038488544523716,
+            0.010871844366192818,
+            0.005653817672282457,
+            0.023624470457434654,
+            0.017385775223374367,
+            -0.011712844483554363,
+            -0.006502463482320309,
+            0.042111191898584366,
+            0.034435153007507324,
+            0.08899315446615219,
+            0.007599587086588144,
+            0.0032454978208988905,
+            0.0005939566763117909,
+            0.043517958372831345,
+            -0.01177400816231966,
+            -0.012309190817177296,
+            -0.04550577700138092,
+            -0.057188041508197784,
+            0.003386938711628318,
+            -0.030214857310056686,
+            0.021223796531558037,
+            -0.003618213813751936,
+            0.009954389184713364,
+            -0.01877724938094616,
+            -0.05923702195286751,
+            0.037890899926424026,
+            -0.0064260088838636875,
+            0.012798500247299671,
+            -0.011085917241871357,
+            0.005256253760308027,
+            -0.001563496538437903,
+            0.020765069872140884,
+            -0.022125961259007454,
+            -0.03244733065366745,
+            0.036300644278526306,
+            0.02734016440808773,
+            0.023578599095344543,
+            0.020581578835844994,
+            0.0029587929602712393,
+            0.0038514004554599524,
+            0.016131920740008354,
+            -0.013822991400957108,
+            -0.0076875099912285805,
+            0.0016886909725144506,
+            -0.009725024923682213,
+            0.003293281886726618,
+            0.031147602945566177,
+            0.0689314678311348,
+            -0.0224776528775692,
+            0.011712844483554363,
+            -0.0219577606767416,
+            0.020520415157079697,
+            -0.02931269258260727,
+            0.01315783616155386,
+            -0.00795892346650362,
+            0.01703408546745777,
+            -0.007068227510899305,
+            0.009182197041809559,
+            -0.016208374872803688,
+            0.0015692306915298104,
+            -0.003788325469940901,
+            -0.030964111909270287,
+            0.0006613322766497731,
+            0.025168854743242264,
+            -0.009121033363044262,
+            0.018318522721529007,
+            0.04755476117134094,
+            0.024220816791057587,
+            -0.04868628829717636,
+            0.024419598281383514,
+            0.0011879133526235819,
+            0.03076533041894436,
+            0.040245700627565384,
+            0.0068847364746034145,
+            -0.015573801472783089,
+            0.012347417883574963,
+            0.007928341627120972,
+            0.021835433319211006,
+            0.011468189768493176,
+            0.025031235069036484,
+            -0.03929766267538071,
+            -0.00842529721558094,
+            0.032600242644548416,
+            0.007763964589685202,
+            0.03489387780427933,
+            0.018333813175559044,
+            0.037340424954891205,
+            0.019602959975600243,
+            0.010619543492794037,
+            0.015260337851941586,
+            0.03189685940742493,
+            0.011980435810983181,
+            0.03969522938132286,
+            -0.009633279405534267,
+            0.0167129747569561,
+            -0.03156045824289322,
+            -0.003220649901777506,
+            0.020994432270526886,
+            0.01137644425034523,
+            0.02050512284040451,
+            -0.0067088911309838295,
+            -0.01113943476229906,
+            0.003945057280361652,
+            -0.013287809677422047,
+            0.004407607484608889,
+            0.010000261478126049,
+            -0.0019381240708753467,
+            0.020902687683701515,
+            0.00837942399084568,
+            -0.04324272274971008,
+            0.02116263285279274,
+            -0.03422107920050621,
+            0.012278608977794647,
+            -0.009029287844896317,
+            0.006785345729440451,
+            0.0038552230689674616,
+            -0.017171703279018402,
+            0.0024656609166413546,
+            0.029664384201169014,
+            -0.017339903861284256,
+            -0.0012356974184513092,
+            0.00038083948311395943,
+            0.013058445416390896,
+            -0.0007425653166137636,
+            -0.004059739410877228,
+            -0.013196064159274101,
+            -0.017446938902139664,
+            -0.014763383194804192,
+            -0.0004931321600452065,
+            -0.006059026811271906,
+            -0.018456140533089638,
+            0.041407812386751175,
+            0.011903980746865273,
+            0.007400805130600929,
+            0.01877724938094616,
+            -0.010329016484320164,
+            0.021651942282915115,
+            0.008371778763830662,
+            0.028961002826690674,
+            -0.004357912112027407,
+            -0.02227886952459812,
+            0.02552054449915886,
+            -0.044404830783605576,
+            -0.03917533531785011,
+            0.025734618306159973,
+            0.026009853929281235,
+            -0.027798892930150032,
+            -0.0033391546458005905,
+            -0.011743426322937012,
+            0.0074275643564760685,
+            0.012714399956166744,
+            0.003622036660090089,
+            0.0030142224859446287,
+            0.005325062666088343,
+            -0.007278477773070335,
+            0.047585342079401016,
+            0.018456140533089638,
+            -0.017905667424201965,
+            -0.019924068823456764,
+            -0.01984761469066143,
+            0.028012964874505997,
+            0.018884286284446716,
+            0.001693469355814159,
+            0.017875084653496742,
+            0.02238590642809868,
+            -0.0010416939621791244,
+            0.05654582008719444,
+            -0.018945449963212013,
+            -0.01573435589671135,
+            -0.006972659379243851,
+            -0.002561229048296809,
+            -0.04853338003158569,
+            -0.02200363390147686,
+            0.007935987785458565,
+            0.05214203521609306,
+            -0.02327278070151806,
+            -0.0057073356583714485,
+            -0.0008553358493372798,
+            0.043426211923360825,
+            0.023869125172495842,
+            0.03519969806075096,
+            -2.6818839614861645e-05,
+            -0.004858689848333597,
+            0.020719196647405624,
+            0.03149929642677307,
+            -0.002643417799845338,
+            -0.016468320041894913,
+            -0.005168330855667591,
+            -0.03773799166083336,
+            -0.031346384435892105,
+            -0.017431648448109627,
+            -0.009686797857284546,
+            -0.03474096953868866,
+            0.002180867362767458,
+            0.017920957878232002,
+            0.02163665182888508,
+            -0.003543670754879713,
+            0.008333551697432995,
+            0.03177453204989433,
+            -0.02597927302122116,
+            -0.05841131508350372,
+            0.0032875477336347103,
+            0.004744007717818022,
+            0.0005552515503950417,
+            -0.04905327036976814,
+            0.0002306778624188155,
+            0.012561490759253502,
+            0.017446938902139664,
+            -0.02107088826596737,
+            -0.0047975261695683,
+            0.03090294823050499,
+            -0.007175263948738575,
+            -0.0014297009911388159,
+            -0.01163639035075903,
+            -0.03029131144285202,
+            -0.0015510726952925324,
+            0.011689907871186733,
+            -0.008494106121361256,
+            -0.026957891881465912,
+            -0.028823383152484894,
+            -0.007851887494325638,
+            0.009801479987800121,
+            -0.027645982801914215,
+            -0.015497347339987755,
+            0.006441299803555012,
+            0.012737336568534374,
+            0.021896596997976303,
+            0.02967967465519905,
+            -0.015520283952355385,
+            0.017064666375517845,
+            -0.0179821215569973,
+            -0.040704429149627686,
+            -0.028853965923190117,
+            0.0022038037423044443,
+            -0.008348842151463032,
+            -0.032508496195077896,
+            -0.004514643922448158,
+            0.0007502107764594257,
+            -0.01689646579325199,
+            -0.03498562425374985,
+            -0.023655053228139877,
+            -0.00943449791520834,
+            -0.014289364218711853,
+            0.011062980629503727,
+            -0.0020470719318836927,
+            -0.007836596108973026,
+            -0.01605546660721302,
+            0.011322925798594952,
+            0.007974214851856232,
+            0.010092006996273994,
+            0.0009609387489035726,
+            -0.0016925137024372816,
+            0.0062272269278764725,
+            -0.0016389954835176468,
+            -0.024465471506118774,
+            -0.03431282564997673,
+            -0.00899106077849865,
+            -0.015856683254241943,
+            0.007660750765353441,
+            0.0349244624376297,
+            -0.007974214851856232,
+            -0.03718751668930054,
+            -0.021988343447446823,
+            -0.0025115336757153273,
+            -0.0648946613073349,
+            -0.030214857310056686,
+            -0.0349244624376297,
+            -0.030092529952526093,
+            -0.0007473436999134719,
+            -0.0029301224276423454,
+            -0.007836596108973026,
+            0.024832453578710556,
+            -0.0006197600741870701,
+            -0.01240858156234026,
+            0.014778673648834229,
+            0.01587197557091713,
+            0.021560197696089745,
+            -0.011873398907482624,
+            -0.01793624833226204,
+            0.044404830783605576,
+            -0.022951669991016388,
+            -0.001025447272695601,
+            0.01998523250222206,
+            0.003316218266263604,
+            0.0026204814203083515,
+            0.021468451246619225,
+            0.0033601797185838223,
+            -0.007030000444501638,
+            -0.04639264941215515,
+            0.040704429149627686,
+            -0.00032756017753854394,
+            -0.02893042005598545,
+            0.009212778881192207,
+            0.007259364239871502,
+            0.011621098965406418,
+            0.02833407372236252,
+            -0.034343406558036804,
+            -0.014801610261201859,
+            -0.012018662877380848,
+            0.0012089383089914918,
+            -0.021713105961680412,
+            0.012125699780881405,
+            -0.015436183661222458,
+            -0.013134900480508804,
+            -0.023150453343987465,
+            -0.013815346173942089,
+            0.0033468001056462526,
+            0.011720489710569382,
+            0.03247791528701782,
+            -0.018379686400294304,
+            0.0033028386533260345,
+            -0.017676303163170815,
+            -0.0368817001581192,
+            -0.03029131144285202,
+            0.019113650545477867,
+            0.006926786620169878,
+            -0.02734016440808773,
+            0.024939490482211113,
+            -0.003333420492708683,
+            0.007385514210909605,
+            0.02293637953698635,
+            0.038563698530197144,
+            0.033028386533260345,
+            0.02935856580734253,
+            0.0001604352000867948,
+            0.02902216650545597,
+            -0.010665416717529297,
+            -0.010191397741436958,
+            -0.00784424226731062,
+            -0.0010264030424878001,
+            0.02425139956176281,
+            -0.007282300386577845,
+            0.05874771252274513,
+            -0.021789560094475746,
+            -0.009189843200147152,
+            -0.02541350945830345,
+            -0.006376313511282206,
+            0.01276027224957943,
+            -0.0184408500790596,
+            0.002058540005236864,
+            0.029832584783434868,
+            0.02082623355090618,
+            -0.01839497685432434,
+            0.015795519575476646,
+            -0.02223299816250801,
+            -0.02382325381040573,
+            -0.02786005660891533,
+            -0.0015548954252153635,
+            -0.011903980746865273,
+            -0.025153562426567078,
+            -0.03001607581973076,
+            -0.004839576315134764,
+            -0.09296879172325134,
+            -0.004101789090782404,
+            0.04321213811635971,
+            -0.005795258563011885,
+            0.032325003296136856,
+            -0.008937542326748371,
+            0.004151484929025173,
+            -0.03935882821679115,
+            0.00418971199542284,
+            0.03605598956346512,
+            0.039634063839912415,
+            0.0006541646434925497,
+            -0.005692044738680124,
+            -0.0009050313383340836,
+            -0.04675963148474693,
+            0.020428668707609177,
+            -0.005944345146417618,
+            -0.004289103206247091,
+            0.0002034409117186442,
+            -0.030964111909270287,
+            0.02293637953698635,
+            0.02620863728225231,
+            0.03749333694577217,
+            0.03908359259366989,
+            -0.017049375921487808,
+            -0.06532280892133713,
+            -0.0015376930823549628,
+            -0.000322303909342736,
+            0.025811072438955307,
+            0.009556825272738934,
+            -0.004442012403160334,
+            -0.009801479987800121,
+            -0.005252431146800518,
+            -0.01998523250222206,
+            0.016040174290537834,
+            0.01282908208668232,
+            0.013356618583202362,
+            0.0304289311170578,
+            -0.011491126380860806,
+            0.002372003858909011,
+            0.024312563240528107,
+            0.031071148812770844,
+            0.0031212589237838984,
+            -0.050582364201545715,
+            -0.017125830054283142,
+            -0.009319815784692764,
+            0.04883919656276703,
+            0.011361153796315193,
+            -0.021850723773241043,
+            -0.011116499081254005,
+            0.02256939746439457,
+            0.026147473603487015,
+            0.0020967673044651747,
+            -0.018853703513741493,
+            -0.015482055954635143,
+            0.011177662760019302,
+            -0.018700795248150826,
+            0.007309059612452984,
+            -0.0019916424062103033,
+            -0.010497216135263443,
+            -0.002899540588259697,
+            -0.0003841843572445214,
+            -0.018043285235762596,
+            0.01909835822880268,
+            -0.00835648737847805,
+            0.04859454184770584,
+            -0.01074187085032463,
+            0.02055099606513977,
+            -0.003369736485183239,
+            -0.0021827789023518562,
+            -0.0023758267052471638,
+            0.05113283544778824,
+            -0.00784424226731062,
+            0.01582610234618187,
+            0.04097966477274895,
+            -0.002408319851383567,
+            0.04657614231109619,
+            -0.011452899314463139,
+            -0.033120132982730865,
+            -0.0026663541793823242,
+            0.012225090526044369,
+            0.006150772329419851,
+            0.004617857746779919,
+            -0.022829342633485794,
+            -0.00800479669123888,
+            0.004526112228631973,
+            -0.029236238449811935,
+            -0.01591784693300724,
+            -0.013089027255773544,
+            -0.012125699780881405,
+            0.006842686794698238,
+            0.004338798578828573,
+            0.009212778881192207,
+            -0.021009724587202072,
+            0.044496577233076096,
+            -0.028395237401127815,
+            0.02266114391386509,
+            0.02659090980887413,
+            -0.008287678472697735,
+            -0.0008639369625598192,
+            -0.023945581167936325,
+            0.012852017767727375,
+            -0.009870288893580437,
+            -0.022309452295303345,
+            -0.01694233901798725,
+            0.006475704722106457,
+            0.0011238825973123312,
+            -0.045995086431503296,
+            0.03464922308921814,
+            0.01666710339486599,
+            -0.016300121322274208,
+            0.004560516681522131,
+            -0.0048013487830758095,
+            0.016743557527661324,
+            -0.002679733792319894,
+            0.02411377988755703,
+            0.013861218467354774,
+            0.02574990876019001,
+            -0.010458989068865776,
+            0.018884286284446716,
+            0.003998575732111931,
+            0.022997543215751648,
+            0.015053910203278065,
+            0.03253907710313797,
+            0.05663756653666496,
+            -0.005600299220532179,
+            -0.007989505305886269,
+            -0.012507972307503223,
+            0.019419468939304352,
+            -0.016315411776304245,
+            -0.00784424226731062,
+            0.02977142110466957,
+            0.015994302928447723,
+            -0.006575095467269421,
+            -0.0032990160398185253,
+            -0.0159637201577425,
+            0.013165482319891453,
+            -0.017768049612641335,
+            0.006120190490037203,
+            0.035750169306993484,
+            0.003279902273789048,
+            -0.02162136137485504,
+            0.03061242215335369,
+            -0.0040329801850020885,
+            0.034007005393505096,
+            0.00630750460550189,
+            0.02168252505362034,
+            -0.038838937878608704,
+            -0.013517173007130623,
+            0.017661012709140778,
+            0.021040305495262146,
+            -0.009052224457263947,
+            -0.01782921329140663,
+            0.028899839147925377,
+            0.018135031685233116,
+            -0.002549760974943638,
+            -0.01418232824653387,
+            -0.004430544096976519,
+            0.02851756475865841,
+            -0.020673323422670364,
+            0.023532725870609283,
+            0.005581185687333345,
+            0.02781418338418007,
+            -0.0010273586958646774,
+            -0.0013360441662371159,
+            0.004040625412017107,
+            -0.016544776037335396,
+            0.004805171396583319,
+            0.03825788199901581,
+            0.014503437094390392,
+            0.0266826543956995,
+            -4.62012758362107e-05,
+            0.011942208744585514,
+            0.03149929642677307,
+            -0.010657771490514278,
+            -0.02139199711382389,
+            0.0038838936015963554,
+            -0.0033487114124000072,
+            -0.045016467571258545,
+            0.00017799586930777878,
+            -0.007114100269973278,
+            -0.03090294823050499,
+            -0.013945318758487701,
+            0.0004625503206625581,
+            0.01500803790986538,
+            0.008654660545289516,
+            0.03685111552476883,
+            -0.022080088034272194,
+            -0.03180511295795441,
+            0.016774138435721397,
+            0.019281849265098572,
+            0.0334259495139122,
+            -0.008341196924448013,
+            0.005841131322085857,
+            -0.0055200220085680485,
+            0.007026177830994129,
+            0.024939490482211113,
+            0.0031308159232139587,
+            -0.0046255034394562244,
+            0.008585851639509201,
+            0.013899446465075016,
+            0.035750169306993484,
+            -0.024052616208791733,
+            -0.01591784693300724,
+            -0.03263082355260849,
+            -0.032233260571956635,
+            0.029389148578047752,
+            0.003199625061824918,
+            -0.014327592216432095,
+            0.0159637201577425,
+            0.020092269405722618,
+            -0.016590647399425507,
+            0.012890245765447617,
+            -0.014923937618732452,
+            -0.04293690249323845,
+            0.015130365267395973,
+            -0.02200363390147686,
+            0.007351109758019447,
+            -0.003146106842905283,
+            -0.004487885162234306,
+            -0.010504862293601036,
+            0.006452768109738827,
+            0.012248027138411999,
+            -0.03467980772256851,
+            0.012018662877380848,
+            0.03510795161128044,
+            -0.006406895350664854,
+            -0.01221744529902935,
+            -0.01402177382260561,
+            0.011598162353038788,
+            -0.019067777320742607,
+            0.00490456260740757,
+            0.00750401895493269,
+            0.004866335075348616,
+            -0.007018532138317823,
+            -0.003495886456221342,
+            -0.02050512284040451,
+            -0.007557536941021681,
+            0.013456009328365326,
+            -0.011491126380860806,
+            -0.00448406208306551,
+            0.001542471582069993,
+            0.007890114560723305,
+            0.0008448233129456639,
+            -0.008310615085065365,
+            0.013517173007130623,
+            0.019862905144691467,
+            0.016544776037335396,
+            0.0024580154567956924,
+            0.01474809180945158,
+            -0.027141382917761803,
+            -0.00588700408115983,
+            0.03168278560042381,
+            -0.0051989126950502396,
+            0.005764676723629236,
+            -0.011605808511376381,
+            -0.02752365544438362,
+            -0.003889627754688263,
+            -0.0031480181496590376,
+            0.029557347297668457,
+            0.0014134544180706143,
+            0.019740577787160873,
+            0.025015944615006447,
+            -0.0036832003388553858,
+            0.012553845532238483,
+            0.01121588982641697,
+            -0.037340424954891205,
+            0.034159913659095764,
+            -0.00941156130284071,
+            0.014151746407151222,
+            0.03431282564997673,
+            -0.015650255605578423,
+            -0.011743426322937012,
+            -0.002968349726870656,
+            0.03061242215335369,
+            0.04914501681923866,
+            0.013662436977028847,
+            0.005523844622075558,
+            0.04357912018895149,
+            -0.018716085702180862,
+            -0.04461890459060669,
+            0.03056654892861843,
+            0.020398087799549103,
+            -0.007645459845662117,
+            -0.00011969923070864752,
+            0.02659090980887413,
+            -0.017676303163170815,
+            0.003742452710866928,
+            0.04119373857975006,
+            -0.0024121426977217197,
+            0.010535444132983685,
+            -0.01909835822880268,
+            -0.024419598281383514,
+            0.04452715814113617,
+            -0.027263710275292397,
+            0.032508496195077896,
+            -0.015000392682850361,
+            0.007851887494325638,
+            0.04657614231109619,
+            -0.004671376198530197,
+            0.023471562191843987,
+            0.03113231249153614,
+            -0.03287547826766968,
+            0.0006670663715340197,
+            0.018899576738476753,
+            -0.011491126380860806,
+            -0.003715693484991789,
+            0.003880070988088846,
+            0.021667232736945152,
+            0.012064536102116108,
+            0.0036774661857634783,
+            0.021407287567853928,
+            0.0028632248286157846,
+            -0.0029836406465619802,
+            -0.014128809794783592,
+            0.018012704327702522,
+            -0.003092588623985648,
+            -0.015237401239573956,
+            0.014641055837273598,
+            0.00230510625988245,
+            0.022966962307691574,
+            -0.03437398746609688,
+            -0.0035723410546779633,
+            0.0010627189185470343,
+            0.0034404569305479527,
+            -0.044037848711013794,
+            -0.005202735308557749,
+            -0.015176237560808659,
+            -0.023762090131640434,
+            -0.013004926964640617,
+            0.006922964006662369,
+            0.016911758109927177,
+            -0.023471562191843987,
+            -0.007897759787738323,
+            0.02906803786754608,
+            -0.03376235067844391,
+            -0.004052093718200922,
+            0.00697648199275136,
+            0.005879358854144812,
+            -0.019419468939304352,
+            -0.032233260571956635,
+            0.020489832386374474,
+            0.05275367200374603,
+            0.0014975544763728976,
+            -0.0008338329498656094,
+            0.029832584783434868,
+            -0.012377999722957611,
+            -0.027370747178792953,
+            0.029939621686935425,
+            -0.01775275729596615,
+            0.022768178954720497,
+            0.03159103915095329,
+            -0.0449553057551384,
+            -0.013081382028758526,
+            -0.003134638536721468,
+            -0.03052067570388317,
+            0.005351821891963482,
+            -0.014847483485937119,
+            -0.0031289043836295605,
+            -0.01062718965113163,
+            -0.02438901737332344,
+            0.02536763623356819,
+            -0.009289233945310116,
+            0.01784450374543667,
+            -0.021239086985588074,
+            -8.111355418805033e-05,
+            -0.019327722489833832,
+            -0.022401196882128716,
+            0.010092006996273994,
+            -0.011414671316742897,
+            0.01159051712602377,
+            0.012095117941498756,
+            -0.024465471506118774,
+            -0.021009724587202072,
+            0.012270962819457054,
+            0.008708178997039795,
+            0.013066090643405914,
+            0.003167131682857871,
+            0.01639186590909958,
+            -0.010917716659605503,
+            -0.0099696796387434,
+            0.03174395114183426,
+            0.00028455446590669453,
+            0.022905798628926277,
+            -0.017446938902139664,
+            -0.030979404225945473,
+            0.0025574064347893,
+            0.01077245268970728,
+            -0.03847195580601692,
+            -0.004040625412017107,
+            -0.02986316569149494,
+            0.00022434646962210536,
+            -0.034343406558036804,
+            0.007461968809366226,
+            0.015390310436487198,
+            -0.007182909641414881,
+            -0.0007105499389581382,
+            -0.00035431928699836135,
+            0.02266114391386509,
+            0.021132051944732666,
+            0.000617370882537216,
+            0.014541665092110634,
+            -0.005787613335996866,
+            -0.01156758051365614,
+            -0.015673192217946053,
+            0.0003980417677666992,
+            0.021927179768681526,
+            -0.003945057280361652,
+            -0.04709603264927864,
+            -0.016682393848896027,
+            -0.004958080593496561,
+            -0.01769159361720085,
+            0.01261500921100378,
+            0.007599587086588144,
+            0.01102475356310606,
+            0.0032340295147150755,
+            0.009335106238722801,
+            -0.06055204197764397,
+            -0.030000785365700722,
+            0.02204950712621212,
+            -0.018135031685233116,
+            0.023563306778669357,
+            0.010420762002468109,
+            0.023089289665222168,
+            -0.01243151817470789,
+            0.00023462004901375622,
+            0.023379815742373466,
+            -0.005107167176902294,
+            0.018746668472886086,
+            0.0024866857565939426,
+            0.02434314414858818,
+            -0.0259333997964859,
+            0.030321894213557243,
+            -0.009862643666565418,
+            0.019954649731516838,
+            -0.02097914181649685,
+            -0.038838937878608704,
+            -0.033548276871442795,
+            0.018761958926916122,
+            0.00335253425873816,
+            -0.02697318233549595,
+            -0.0017575001111254096,
+            0.0068962047807872295,
+            0.02547467313706875,
+            -0.021560197696089745,
+            0.004575807601213455,
+            -0.0033104841131716967,
+            0.005833486095070839,
+            0.015887266024947166,
+            -0.015948429703712463,
+            -0.01966412365436554,
+            -0.01680472120642662,
+            0.050918761640787125,
+            0.009006352163851261,
+            0.02565816417336464,
+            -0.011468189768493176,
+            0.007091164123266935,
+            0.008868733420968056,
+            -0.020749777555465698,
+            0.005867890547960997,
+            0.02692730911076069,
+            -0.04807465150952339,
+            0.008081250824034214,
+            0.032569658011198044,
+            0.04663730412721634,
+            -0.02536763623356819,
+            0.02458779886364937,
+            0.0029702612664550543,
+            -0.003264611354097724,
+            0.031300511211156845,
+            0.02541350945830345,
+            0.030643003061413765,
+            -0.0167129747569561,
+            -0.009625634178519249,
+            -0.01957237720489502,
+            0.01975586824119091,
+            0.01998523250222206,
+            0.003597188973799348,
+            -0.044588323682546616,
+            0.022034214809536934,
+            -0.010611898265779018,
+            0.013471300713717937,
+            0.01848672144114971,
+            -0.005340353585779667,
+            0.032416749745607376,
+            0.013731245882809162,
+            -0.028762219473719597,
+            0.007549891714006662,
+            0.001983996946364641,
+            -0.0006933476543053985,
+            0.019817031919956207,
+            0.040184538811445236,
+            -0.005653817672282457,
+            -0.02163665182888508,
+            -0.010092006996273994,
+            0.014511083252727985,
+            -0.0045910985209047794,
+            0.022309452295303345,
+            -0.0043540894985198975,
+            0.019725287333130836,
+            0.004334975965321064,
+            -0.006598032079637051,
+            0.009526243433356285,
+            0.03498562425374985,
+            -0.006857977714389563,
+            0.015115073882043362,
+            -0.014251137152314186,
+            0.01821148581802845,
+            0.022814052179455757,
+            -0.02607101760804653,
+            -0.026835564523935318,
+            -0.011903980746865273,
+            -0.024450181052088737,
+            0.01058131642639637,
+            -0.003994752652943134,
+            -0.04143839329481125,
+            -0.03608657047152519,
+            -0.0056767538189888,
+            -0.018517304211854935,
+            -0.03718751668930054,
+            -0.021835433319211006,
+            -0.027508364990353584,
+            0.0008424341212958097,
+            0.004162952769547701,
+            0.011101207695901394,
+            -0.016177793964743614,
+            -0.015459120273590088,
+            0.009656216017901897,
+            0.031101731583476067,
+            -0.027508364990353584,
+            0.043884940445423126,
+            0.012492681853473186,
+            -0.02139199711382389,
+            0.009228070266544819,
+            0.033915258944034576,
+            0.0002752365544438362,
+            -0.0244348905980587,
+            0.019878195598721504,
+            0.0028288201428949833,
+            0.017294030636548996,
+            0.022645851597189903,
+            -0.011277053505182266,
+            0.0030008431058377028,
+            0.004526112228631973,
+            -0.005290658213198185,
+            -0.04269224777817726,
+            -0.0023203971795737743,
+            0.003195802215486765,
+            -0.025015944615006447,
+            0.0007052936707623303,
+            0.005164508242160082,
+            0.026804981753230095,
+            -0.0018005057936534286,
+            0.030031366273760796,
+            0.05813607573509216,
+            -0.020703906193375587,
+            -0.04452715814113617,
+            0.005088053643703461,
+            -0.010137880221009254,
+            0.0021942469757050276,
+            -0.029465602710843086,
+            0.03801322728395462,
+            -0.013868864625692368,
+            0.0033926728647202253,
+            -0.005003953352570534,
+            -0.0036239479668438435,
+            -0.029649093747138977,
+            0.021223796531558037,
+            0.0019161433447152376,
+            -0.006575095467269421,
+            -0.036117151379585266,
+            -0.0338846780359745,
+            0.02472541667521,
+            0.019511213526129723,
+            -0.022814052179455757,
+            -0.021284960210323334,
+            -0.0010426496155560017,
+            -0.011804590001702309,
+            0.026330964639782906,
+            -0.01834910362958908,
+            -0.01675884798169136,
+            0.0116134537383914,
+            0.023578599095344543,
+            0.018104448914527893,
+            -0.015451474115252495,
+            -0.06046029552817345,
+            -0.02191188745200634,
+            0.02125437930226326,
+            -0.00647188164293766,
+            -0.003503531916067004,
+            -0.016774138435721397,
+            0.007286123465746641,
+            -0.019220685586333275,
+            -0.025826362892985344,
+            0.038563698530197144,
+            -0.004239407368004322,
+            0.0227070152759552,
+            0.0004496486217249185,
+            0.0018645365489646792,
+            0.04786057770252228,
+            0.008784633129835129,
+            -0.020612159743905067,
+            -0.010619543492794037,
+            0.01376182772219181,
+            0.041744209825992584,
+            -0.01966412365436554,
+            0.005634704139083624,
+            0.01296669989824295,
+            -0.01102475356310606,
+            -0.013639500364661217,
+            0.0007310971268452704,
+            0.0099696796387434,
+            -0.002119703684002161,
+            0.004755476024001837,
+            -0.020765069872140884,
+            0.015115073882043362,
+            0.02088739722967148,
+            0.012118053622543812,
+            -0.003795970929786563,
+            -0.0037692117039114237,
+            -0.00027021922869607806,
+            0.01418232824653387,
+            0.016269538551568985,
+            0.02411377988755703,
+            0.024786580353975296,
+            -0.001051250728778541,
+            -0.01896074041724205,
+            -0.004182066768407822,
+            -0.003694668412208557,
+            0.001217539538629353,
+            0.02992432937026024,
+            0.032936640083789825,
+            -0.008738760836422443,
+            0.006357199978083372,
+            -0.018792539834976196,
+            0.012844372540712357,
+            0.021973051130771637,
+            -0.018884286284446716,
+            -0.018196195363998413,
+            0.025168854743242264,
+            -0.003528379835188389,
+            0.01441933773458004,
+            0.002433167537674308,
+            0.006364845205098391,
+            0.0206886138767004,
+            0.06813634186983109,
+            -0.015979010611772537,
+            -0.008050668984651566,
+            -0.022018924355506897,
+            0.0031136134639382362,
+            0.025444090366363525,
+            -0.026147473603487015,
+            0.013119609095156193,
+            0.00559265399351716,
+            0.02350214309990406,
+            0.016605939716100693,
+            0.004923676140606403,
+            0.0023280426394194365,
+            0.01214863546192646,
+            0.0025153562892228365,
+            0.016254248097538948,
+            0.0004979106015525758,
+            -0.00819593295454979,
+            0.0035207343753427267,
+            0.01591784693300724,
+            0.015160947106778622,
+            0.027982383966445923,
+            -0.0007554669864475727,
+            0.025734618306159973,
+            -0.03076533041894436,
+            0.022064797580242157,
+            -0.007599587086588144,
+            0.0020031104795634747,
+            -0.01802799478173256,
+            -0.008937542326748371,
+            0.011552290059626102,
+            -0.001057940535247326,
+            -0.0001701114815659821,
+            0.0025669632013887167,
+            -0.007010886911302805,
+            0.03090294823050499,
+            3.2761989132268354e-05,
+            0.011445253156125546,
+            0.008478814736008644,
+            -0.007419918663799763,
+            0.004308216739445925,
+            -0.001004422316327691,
+            0.01773746684193611,
+            -0.029832584783434868,
+            -0.010825971141457558,
+            0.012936118058860302,
+            0.016361285001039505,
+            -0.009059869684278965,
+            0.005130103789269924,
+            -0.023425688967108727,
+            0.0007463880465365946,
+            0.012951409444212914,
+            -0.010321371257305145,
+            0.006773877423256636,
+            -0.024037325754761696,
+            -0.011277053505182266,
+            0.005703513044863939,
+            -0.034526895731687546,
+            -0.006678309291601181,
+            0.008761697448790073,
+            -0.003845666302368045,
+            0.020749777555465698,
+            0.0036832003388553858,
+            0.01116237137466669,
+            -0.022783471271395683,
+            -0.011934562586247921,
+            -0.0022745244204998016,
+            0.0032951931934803724,
+            0.0010808769147843122,
+            0.006987950298935175,
+            -0.01605546660721302,
+            0.028961002826690674,
+            -0.024924200028181076,
+            -0.015252692624926567,
+            -0.004101789090782404,
+            -0.0024637493770569563,
+            0.0008410005830228329,
+            -0.01463341061025858,
+            0.004380848724395037,
+            -0.007710446137934923,
+            0.034007005393505096,
+            0.035322025418281555,
+            -0.005003953352570534,
+            -0.01077245268970728,
+            -0.03217209503054619,
+            0.015619674697518349,
+            -0.018563177436590195,
+            -0.0034939751494675875,
+            -0.0050498261116445065,
+            -0.006815927568823099,
+            0.0005284924409352243,
+            -0.02533705346286297,
+            0.0113535076379776,
+            0.002578431274741888,
+            0.0018893842352554202,
+            0.0011420405935496092,
+            0.011147080920636654,
+            0.015092138200998306,
+            -0.02003110572695732,
+            -0.040184538811445236,
+            -0.024878326803445816,
+            -0.002654885873198509,
+            0.009801479987800121,
+            -0.016254248097538948,
+            -0.008310615085065365,
+            -0.004820462316274643,
+            0.0116134537383914,
+            -0.006716536357998848,
+            0.006942077539861202,
+            0.003516911529004574,
+            0.0025841654278337955,
+            -0.010092006996273994,
+            -0.019358305260539055,
+            -0.010474280454218388,
+            0.01568848453462124,
+            0.020306341350078583,
+            0.007492550648748875,
+            -0.04672905057668686,
+            -0.0023643583990633488,
+            0.002708404092118144,
+            0.007473437115550041,
+            -0.006215759087353945,
+            0.03529144451022148,
+            0.0007846153457649052,
+            2.241374022560194e-05,
+            0.03737100958824158,
+            0.011062980629503727,
+            -0.016743557527661324,
+            0.012309190817177296,
+            -0.015459120273590088,
+            -0.002947324886918068,
+            0.00021455071691889316,
+            0.03168278560042381,
+            0.005290658213198185,
+            0.028074128553271294,
+            -0.0264532919973135,
+            -8.410005830228329e-05,
+            0.003337243339046836,
+            0.02827291004359722,
+            -0.015841392800211906,
+            -0.023762090131640434,
+            0.024006744846701622,
+            0.03929766267538071,
+            -0.007905405946075916,
+            -0.010634834878146648,
+            -0.03330362215638161,
+            -0.025918109342455864,
+            0.047829996794462204,
+            0.025872236117720604,
+            0.004598744213581085,
+            0.02102501504123211,
+            -0.021804852411150932,
+            0.015023328363895416,
+            0.015099783428013325,
+            0.0008868733420968056,
+            -0.030918240547180176,
+            0.03932824730873108,
+            -0.013478945940732956,
+            0.03526085987687111,
+            0.002372003858909011,
+            -0.003767300397157669,
+            0.020948560908436775,
+            0.023884417489171028,
+            -0.029893748462200165,
+            0.009579760953783989,
+            -0.0045643397606909275,
+            -0.013402490876615047,
+            0.015436183661222458,
+            0.007798369042575359,
+            0.02331865206360817,
+            -0.01479396503418684,
+            0.00012184951629023999,
+            -0.0013341327430680394,
+            0.03819671645760536,
+            0.008509396575391293,
+            -0.02350214309990406,
+            0.03263082355260849,
+            -0.012729690410196781,
+            0.00769133260473609,
+            -0.015046264976263046,
+            0.027049636468291283,
+            -0.011292343959212303,
+            -0.004055916331708431,
+            0.008348842151463032,
+            0.01568848453462124,
+            0.01784450374543667,
+            -0.015076846815645695,
+            -0.008318260312080383,
+            -0.03905300796031952,
+            0.01582610234618187,
+            0.012225090526044369,
+            -0.0003877681738231331,
+            0.02448076196014881,
+            -0.0072173140943050385,
+            -0.007851887494325638,
+            -0.0036832003388553858,
+            -0.007324350532144308,
+            -0.012653236277401447,
+            -0.003279902273789048,
+            -0.03510795161128044,
+            -0.03305896744132042,
+            0.011429962702095509,
+            -0.021697815507650375,
+            -0.01587197557091713,
+            0.011651680804789066,
+            -0.02672852762043476,
+            0.02214125171303749,
+            -0.007733382750302553,
+            0.03730984404683113,
+            -0.02289050631225109,
+            0.012683818116784096,
+            0.010374888777732849,
+            -0.004059739410877228,
+            -0.001281570177525282,
+            -0.012706754729151726,
+            -0.004006220959126949,
+            0.0061660632491111755,
+            0.01044369861483574,
+            0.02831878326833248,
+            0.03211093321442604,
+            -0.03458806127309799,
+            0.026346255093812943,
+            -0.007913051173090935,
+            0.010543089359998703,
+            0.00279632699675858,
+            0.001090433681383729,
+            0.03128522261977196,
+            -0.03379293158650398,
+            0.009166906587779522,
+            0.015497347339987755,
+            -0.0015682749217376113,
+            0.022584687918424606,
+            0.008043023757636547,
+            0.0003877681738231331,
+            -0.013853573240339756,
+            0.005068940110504627,
+            0.021269669756293297,
+            -0.0066591957584023476,
+            0.01503097452223301,
+            -0.01439640112221241,
+            0.026881437748670578,
+            0.03507737070322037,
+            -0.009296879172325134,
+            0.02266114391386509,
+            0.008486460894346237,
+            -0.0002391595480730757,
+            0.013249581679701805,
+            -0.007297591306269169,
+            -0.0349244624376297,
+            -0.001130572403781116,
+            -0.02715667337179184,
+            -0.010267852805554867,
+            -0.0009332239278592169,
+            -0.00590994069352746,
+            -0.023578599095344543,
+            -0.0033926728647202253,
+            0.016820011660456657,
+            -0.0007244073203764856,
+            -0.0059787495993077755,
+            -0.011536998674273491,
+            0.020076977089047432,
+            0.015749648213386536,
+            0.025061817839741707,
+            0.020245177671313286,
+            0.0020413377787917852,
+            0.014725156128406525,
+            0.0014421248342841864,
+            0.02438901737332344,
+            -0.006116367876529694,
+            -0.0017288295784965158,
+            -0.009059869684278965,
+            0.00859349686652422,
+            0.010711289010941982,
+            0.008501751348376274,
+            0.011582871899008751,
+            -0.021514324471354485,
+            -0.010902426205575466,
+            -0.012423872016370296,
+            0.004472594242542982,
+            -0.00540534034371376,
+            0.02036750502884388,
+            0.03529144451022148,
+            0.0011860019294545054,
+            -0.005305949132889509,
+            -0.016131920740008354,
+            -0.004671376198530197,
+            0.009747961536049843,
+            -0.032508496195077896,
+            0.0031919796019792557,
+            -0.015933139249682426,
+            0.0009059869917109609,
+            -0.020994432270526886,
+            0.0007736249826848507,
+            0.009579760953783989,
+            0.0030103998724371195,
+            0.016361285001039505,
+            -0.020443959161639214,
+            0.022447070106863976,
+            -0.0005710202967748046,
+            -0.008203578181564808,
+            0.013593628071248531,
+            -0.02093326859176159,
+            0.009304524399340153,
+            0.022737598046660423,
+            0.024985363706946373,
+            -0.016590647399425507,
+            -0.0008175863767974079,
+            0.00418971199542284,
+            0.006326618138700724,
+            0.00769133260473609,
+            0.0038265527691692114,
+            0.013310745358467102,
+            -0.016590647399425507,
+            0.013196064159274101,
+            0.013540109619498253,
+            0.004690489731729031,
+            5.689297177013941e-05,
+            -0.004786057863384485,
+            0.01854788511991501,
+            0.03776857256889343,
+            -0.02827291004359722,
+            -0.017095249146223068,
+            -0.0012796588707715273,
+            -0.009587407112121582,
+            0.000816152838524431,
+            0.015856683254241943,
+            -0.029847875237464905,
+            0.00437702564522624,
+            0.003228295361623168,
+            0.004445835016667843,
+            -0.021988343447446823,
+            0.017339903861284256,
+            0.008371778763830662,
+            -0.008020087145268917,
+            0.03137696906924248,
+            0.023288071155548096,
+            -0.0016514193266630173,
+            0.04088791832327843,
+            0.002578431274741888,
+            0.007087341509759426,
+            -0.026942601427435875,
+            0.031346384435892105,
+            -0.010971235111355782,
+            0.005864067934453487,
+            0.005367112811654806,
+            -0.02631567232310772,
+            -0.007068227510899305,
+            -0.010237270966172218,
+            0.004078852944076061,
+            -0.007595764473080635,
+            0.011842817068099976,
+            0.02552054449915886,
+            0.024465471506118774,
+            0.002127349143847823,
+            -0.019358305260539055,
+            0.0224776528775692,
+            0.018043285235762596,
+            0.013272518292069435,
+            -0.0011640213197097182,
+            -0.004350266885012388,
+            -0.017401067540049553,
+            -0.0032512317411601543,
+            -0.020719196647405624,
+            -0.01680472120642662,
+            -0.012278608977794647,
+            0.02279876172542572,
+            -0.010642480105161667,
+            -0.0004214559739921242,
+            -0.005183621775358915,
+            -0.02579578198492527,
+            -0.004629326052963734,
+            0.007660750765353441,
+            -0.00488162599503994,
+            0.010451343841850758,
+            0.026621490716934204,
+            0.010359598323702812,
+            -0.03269198536872864,
+            0.0068962047807872295,
+            -0.011017107404768467,
+            -0.007351109758019447,
+            0.0008854398038238287,
+            0.0025593177415430546,
+            0.01923597790300846,
+            -0.014663992449641228,
+            -0.022538816556334496,
+            0.020872104912996292,
+            0.005955813452601433,
+            -0.016590647399425507,
+            0.016376575455069542,
+            0.020765069872140884,
+            -0.008845796808600426,
+            0.010986525565385818,
+            -0.003868602681905031,
+            0.0204592514783144,
+            0.02761540189385414,
+            -0.024710126221179962,
+            9.509040683042258e-05,
+            -0.00469431234523654,
+            -0.005095698870718479,
+            0.0060016862116754055,
+            0.010734225623309612,
+            -0.008073605597019196,
+            -0.027447201311588287,
+            -0.0002871826000045985,
+            0.006991772912442684,
+            0.02561229094862938,
+            0.015275629237294197,
+            -0.03229442238807678,
+            -0.008731115609407425,
+            -0.016743557527661324,
+            0.03590308129787445,
+            -0.0404597744345665,
+            0.0001355874555883929,
+            0.0043311528861522675,
+            -0.010038488544523716,
+            -0.018471430987119675,
+            0.012316836044192314,
+            0.026483872905373573,
+            -0.008501751348376274,
+            -0.006555981934070587,
+            0.003174777142703533,
+            -0.014855128712952137,
+            0.027233127504587173,
+            0.033640023320913315,
+            -0.018471430987119675,
+            0.01775275729596615,
+            0.0037118708714842796,
+            0.006953545846045017,
+            -0.003455747850239277,
+            -0.040796175599098206,
+            0.011483481153845787,
+            0.028991583734750748,
+            0.0013828725786879659,
+            -0.004919853527098894,
+            0.00777160981670022,
+            0.009105742909014225,
+            -0.0037367185577750206,
+            0.010023198090493679,
+            -0.015933139249682426,
+            -0.0035666071344166994,
+            -0.003482507076114416,
+            0.01177400816231966,
+            -0.008417651057243347,
+            0.005936699453741312,
+            -0.008272388018667698,
+            -0.003377381945028901,
+            0.007202023174613714,
+            -0.00863937009125948,
+            -0.00014406914124265313,
+            -0.0025593177415430546,
+            0.0030428930185735226,
+            0.013563046231865883,
+            -0.009725024923682213,
+            -0.011644035577774048,
+            -0.03529144451022148,
+            0.016086047515273094,
+            0.028028255328536034,
+            0.0014908646699041128,
+            -0.030168984085321426,
+            0.006716536357998848,
+            -0.014128809794783592,
+            -0.01018375251442194,
+            -0.0060628498904407024,
+            0.020902687683701515,
+            0.03208034858107567,
+            0.020291050896048546,
+            0.01782921329140663,
+            0.007890114560723305,
+            0.019251268357038498,
+            -0.0029951089527457952,
+            0.008226514793932438,
+            -0.0038552230689674616,
+            -0.018716085702180862,
+            0.0015740090748295188,
+            0.017523394897580147,
+            -0.010351953096687794,
+            0.007056759670376778,
+            0.000946125655900687,
+            0.012561490759253502,
+            0.002589899580925703,
+            -0.0024064085446298122,
+            0.005053649190813303,
+            0.015489702112972736,
+            -0.0021827789023518562,
+            0.008853442966938019,
+            0.07370223104953766,
+            -0.01219450868666172,
+            0.010833616368472576,
+            0.008685242384672165,
+            -0.00578379025682807,
+            -0.017309321090579033,
+            -0.005535312928259373,
+            -0.022492943331599236,
+            0.027967091649770737,
+            0.00353984790854156,
+            -0.02116263285279274,
+            -0.004514643922448158,
+            0.00234715617261827,
+            0.016957629472017288,
+            0.015657901763916016,
+            -0.0027905928436666727,
+            -0.0026701767928898335,
+            -0.0023490674793720245,
+            0.017217576503753662,
+            0.0076263463124632835,
+            -0.025597000494599342,
+            -0.00450699869543314,
+            -0.0018196194432675838,
+            0.012324481271207333,
+            0.002180867362767458,
+            -0.004686667118221521,
+            0.0016428182134404778,
+            0.024603089317679405,
+            -0.005455035716295242,
+            0.029572639614343643,
+            -0.014243491925299168,
+            -0.017538685351610184,
+            0.014381109736859798,
+            -0.0012557668378576636,
+            0.021758979186415672,
+            0.00488162599503994,
+            0.018287939950823784,
+            -0.013654791750013828,
+            -0.005581185687333345,
+            0.01322664599865675,
+            -0.012622654438018799,
+            -0.0026625313330441713,
+            0.007935987785458565,
+            -0.00046756764641031623,
+            -0.002867047442123294,
+            0.006269277073442936,
+            -0.014426982961595058,
+            0.005894649773836136,
+            -0.01573435589671135,
+            -0.004537580534815788,
+            -0.007806014735251665,
+            -0.008188287727534771,
+            0.01463341061025858,
+            -0.0013886066153645515,
+            -0.03403758630156517,
+            -0.0024733063764870167,
+            0.034618642181158066,
+            0.003358268178999424,
+            0.00620429078117013,
+            0.01957237720489502,
+            0.005760854110121727,
+            0.014503437094390392,
+            -0.021132051944732666,
+            -0.0002107279869960621,
+            -0.0041056121699512005,
+            0.005554426461458206,
+            -0.006995595991611481,
+            0.0109788803383708,
+            0.005462680943310261,
+            -0.00017082825070247054,
+            -0.0028402884490787983,
+            -0.006330440752208233,
+            0.0032416749745607376,
+            0.008165351115167141,
+            0.008058315142989159,
+            0.01900661364197731,
+            -0.02266114391386509,
+            -0.01081068068742752,
+            0.009725024923682213,
+            0.009839707054197788,
+            0.01315783616155386,
+            0.008585851639509201,
+            0.0199393592774868,
+            -0.003065829398110509,
+            -0.019832322373986244,
+            -0.006127836182713509,
+            -0.0028880725149065256,
+            0.032325003296136856,
+            0.018150322139263153,
+            -0.025015944615006447,
+            -0.008647015318274498,
+            0.0018100626766681671,
+            0.002362447092309594,
+            -0.004247053060680628,
+            0.008708178997039795,
+            0.007874824106693268,
+            -0.02018401399254799,
+            -0.0008080295519903302,
+            -0.007014709524810314,
+            -0.009319815784692764,
+            0.009648570790886879,
+            0.019205395132303238,
+            -0.02223299816250801,
+            0.01435052789747715,
+            0.006819750182330608,
+            -0.02561229094862938,
+            0.024755999445915222,
+            0.008440587669610977,
+            -0.0028039724566042423,
+            -0.006804459262639284,
+            0.006173708941787481,
+            -0.03608657047152519,
+            0.01811973936855793,
+            0.003975639119744301,
+            -0.0035666071344166994,
+            0.0003598144685383886,
+            0.0019954650197178125,
+            -0.001083743991330266,
+            -0.010329016484320164,
+            0.00010416939039714634,
+            0.018425557762384415,
+            -0.008394715376198292,
+            0.01079538930207491,
+            -0.0035818980541080236,
+            0.008111832663416862,
+            -0.0066477274522185326,
+            -0.006953545846045017,
+            -0.004285280127078295,
+            -0.03324246034026146,
+            -0.01542853843420744,
+            -0.012041599489748478,
+            -0.02561229094862938,
+            0.005500908475369215,
+            0.001043605268932879,
+            -0.013234291225671768,
+            0.005669108591973782,
+            0.0013236202066764235,
+            0.0018177081365138292,
+            -0.004545225761830807,
+            0.010894780047237873,
+            -0.0040826755575835705,
+            0.016865884885191917,
+            0.008884024806320667,
+            -0.013662436977028847,
+            0.020168723538517952,
+            0.0009609387489035726,
+            0.002477128989994526,
+            -0.012660881504416466,
+            0.0002587510389275849,
+            0.003929766360670328,
+            0.009923807345330715,
+            -0.01934301294386387,
+            -0.005229494534432888,
+            -0.00601697713136673,
+            -0.011942208744585514,
+            -0.0022707015741616488,
+            -0.027645982801914215,
+            -0.007905405946075916,
+            -0.010466634295880795,
+            -0.0145798921585083,
+            -0.013700664043426514,
+            0.0022879038006067276,
+            -0.0006871357327327132,
+            0.01821148581802845,
+            -0.015130365267395973,
+            0.011108852922916412,
+            0.01280614547431469,
+            -0.0007903494406491518,
+            0.005657640285789967,
+            -0.007611055392771959,
+            -0.013004926964640617,
+            0.01854788511991501,
+            -0.017079956829547882,
+            -0.008394715376198292,
+            0.0206886138767004,
+            0.012225090526044369,
+            0.002939679427072406,
+            -0.014434628188610077,
+            -0.004201180301606655,
+            0.029052747413516045,
+            0.007733382750302553,
+            0.010864198207855225,
+            0.0032856364268809557,
+            0.003398407017812133,
+            0.02756952866911888,
+            0.0013809611555188894,
+            -0.014144101180136204,
+            0.006231050007045269,
+            0.004686667118221521,
+            -0.01102475356310606,
+            -0.013142545707523823,
+            0.017156412824988365,
+            -0.006926786620169878,
+            -0.015887266024947166,
+            0.024098489433526993,
+            0.023700926452875137,
+            0.0112388264387846,
+            -0.011926917359232903,
+            -0.020902687683701515,
+            0.0003727161674760282,
+            0.01001555286347866,
+            0.011812235228717327,
+            -0.01179694477468729,
+            -0.004224116448312998,
+            0.02457250840961933,
+            0.00901399739086628,
+            -0.022355325520038605,
+            0.008723469451069832,
+            0.030780620872974396,
+            0.015604383312165737,
+            -0.04544461518526077,
+            -0.01694233901798725,
+            -0.0007138948421925306,
+            -0.019251268357038498,
+            -0.02036750502884388,
+            -0.023609180003404617,
+            -0.011942208744585514,
+            0.0009485148475505412,
+            0.005615590140223503,
+            -0.0007291857618838549,
+            -0.017951540648937225,
+            0.01556615624576807,
+            0.017951540648937225,
+            0.011697554029524326,
+            -0.007760141976177692,
+            -0.018456140533089638,
+            -0.012530908919870853,
+            -0.01830323040485382,
+            0.01078774407505989,
+            -0.018089158460497856,
+            -0.012890245765447617,
+            0.006712713744491339,
+            -0.006854154635220766,
+            0.013983545824885368,
+            0.01554321963340044,
+            0.03076533041894436,
+            0.016040174290537834,
+            -0.021376706659793854,
+            -0.019373595714569092,
+            -0.005841131322085857,
+            -0.01074187085032463,
+            -0.004170598462224007,
+            -0.0013045065570622683,
+            0.005370935425162315,
+            0.009579760953783989,
+            0.005191267468035221,
+            0.02082623355090618,
+            -0.009610342793166637,
+            0.02776831015944481,
+            -2.6236470148433e-05,
+            -0.007362578064203262,
+            0.004201180301606655,
+            -0.018930159509181976,
+            0.009266297332942486,
+            -0.008302969858050346,
+            0.01120824459940195,
+            0.012263317592442036,
+            0.007599587086588144,
+            0.036392390727996826,
+            -0.011491126380860806,
+            -0.0013226645532995462,
+            -0.028012964874505997,
+            -0.0013417782029137015,
+            -0.003987107425928116,
+            0.021804852411150932,
+            0.005997863132506609,
+            -0.035841915756464005,
+            -0.020566286519169807,
+            -0.012959054671227932,
+            0.019281849265098572,
+            0.0035742525942623615,
+            -0.0053327083587646484,
+            0.020520415157079697,
+            0.025719327852129936,
+            -0.004679021425545216,
+            -0.0018263092497363687,
+            -0.027462491765618324,
+            -0.009686797857284546,
+            0.03287547826766968,
+            -0.005088053643703461,
+            0.001989730866625905,
+            0.0010636745719239116,
+            -0.015115073882043362,
+            -0.017309321090579033,
+            -0.012263317592442036,
+            0.003694668412208557,
+            0.007190554868429899,
+            0.027095509693026543,
+            -0.004766944330185652,
+            0.022768178954720497,
+            -0.01538266520947218,
+            0.008677597157657146,
+            -0.026850854977965355,
+            -0.012538554146885872,
+            0.018976030871272087,
+            -0.0009489927324466407,
+            0.014060000889003277,
+            -0.018058575689792633,
+            -0.006747118197381496,
+            -0.006024622358381748,
+            0.007419918663799763,
+            -0.005340353585779667,
+            -0.0017030262388288975,
+            -0.007744851056486368,
+            0.01479396503418684,
+            0.0009509040974080563,
+            0.007565182633697987,
+            -0.02275288850069046,
+            0.008318260312080383,
+            0.00087206024909392,
+            0.00670506851747632,
+            0.01882312260568142,
+            -0.00034858519211411476,
+            0.03951173648238182,
+            0.0072287824004888535,
+            0.011644035577774048,
+            -0.012003372423350811,
+            0.004560516681522131,
+            -0.008440587669610977,
+            0.012561490759253502,
+            0.002387294778600335,
+            0.010168462060391903,
+            0.007534600794315338,
+            -0.0029205656610429287,
+            -0.008975770324468613,
+            -0.0013245759764686227,
+            0.0014430806040763855,
+            -0.007706623524427414,
+            0.005703513044863939,
+            0.005493262782692909,
+            0.013570691458880901,
+            -0.01181988138705492,
+            -0.012653236277401447,
+            0.019037194550037384,
+            -0.021040305495262146,
+            -0.006479527335613966,
+            -0.00015768760931678116,
+            -0.006253986153751612,
+            0.004101789090782404,
+            0.0073664006777107716,
+            0.013066090643405914,
+            -0.03608657047152519,
+            0.005493262782692909,
+            -0.0164836123585701,
+            0.006987950298935175,
+            0.0005366157274693251,
+            -0.027829473838210106,
+            -0.024266690015792847,
+            0.0002518223482184112,
+            0.009725024923682213,
+            -0.00527918990701437,
+            0.005615590140223503,
+            -0.017920957878232002,
+            -0.0040329801850020885,
+            -0.004950435366481543,
+            -0.019878195598721504,
+            0.008348842151463032,
+            0.027584819123148918,
+            -0.018242066726088524,
+            -0.010657771490514278,
+            0.008723469451069832,
+            -0.005240962840616703,
+            -0.01989348605275154,
+            -0.006322795525193214,
+            -0.012821435928344727,
+            0.0035341137554496527,
+            0.019862905144691467,
+            -0.0028441110625863075,
+            -0.0113535076379776,
+            0.017248157411813736,
+            0.0066859545186161995,
+            -0.015367374755442142,
+            0.001213716808706522,
+            -0.0012194508453831077,
+            -0.012714399956166744,
+            0.01519152894616127,
+            -0.024266690015792847,
+            0.01578022912144661,
+            -0.0065101091749966145,
+            -0.003471038769930601,
+            -0.031621623784303665,
+            -0.006506286561489105,
+            0.008624078705906868,
+            0.0025688745081424713,
+            0.01793624833226204,
+            0.012393290176987648,
+            -0.020336924120783806,
+            0.01441933773458004,
+            -0.0058029042556881905,
+            0.005286835599690676,
+            -0.006357199978083372,
+            0.042386431246995926,
+            -0.010290789417922497,
+            -0.006517754402011633,
+            -0.00033520563738420606,
+            0.0015338703524321318,
+            0.0033850274048745632,
+            0.02761540189385414,
+            -0.004487885162234306,
+            -0.004308216739445925,
+            -0.0032493204344063997,
+            0.004434366710484028,
+            0.005611767526715994,
+            -0.0006713669281452894,
+            0.015160947106778622,
+            -0.006032268051058054,
+            0.008891670033335686,
+            -0.0224164891988039,
+            -0.0030428930185735226,
+            0.0159637201577425,
+            0.0032130044419318438,
+            -0.0012060713488608599,
+            -0.007672219071537256,
+            -0.03177453204989433,
+            -0.000630272610578686,
+            -0.023073997348546982,
+            0.006269277073442936,
+            -0.010351953096687794,
+            0.0017431648448109627,
+            -0.01415939163416624,
+            0.01081068068742752,
+            0.012744981795549393,
+            0.003799793543294072,
+            0.01460282877087593,
+            -0.0219577606767416,
+            0.0159637201577425,
+            0.007408450823277235,
+            -0.005890826694667339,
+            0.011865753680467606,
+            -0.015176237560808659,
+            -0.009717379696667194,
+            0.0017173613887280226,
+            0.02761540189385414,
+            -0.0012834816006943583,
+            -0.006047558970749378,
+            -0.00348824099637568,
+            -0.013532464392483234,
+            -0.0005409162840805948,
+            0.0010703643783926964,
+            -0.010451343841850758,
+            -0.001181223546154797,
+            -0.006655372679233551,
+            0.009396269917488098,
+            0.010543089359998703,
+            0.000875882979016751,
+            0.006093431729823351,
+            -0.009931452572345734,
+            0.00037367184995673597,
+            0.015099783428013325,
+            0.038563698530197144,
+            -0.0012959054438397288,
+            0.011659326031804085,
+            -0.01952650398015976,
+            -0.004407607484608889,
+            -0.0021789560560137033,
+            -0.029878458008170128,
+            -0.010703643783926964,
+            0.00819593295454979,
+            0.014105873182415962,
+            0.006506286561489105,
+            -0.015764938667416573,
+            0.005003953352570534,
+            0.01778334006667137,
+            0.0095950523391366,
+            0.0047898804768919945,
+            0.0071790870279073715,
+            0.003386938711628318,
+            0.006234872620552778,
+            0.00641454104334116,
+            -0.025306472554802895,
+            -0.011154726147651672,
+            -0.010084361769258976,
+            -0.021177923306822777,
+            0.013769472949206829,
+            0.01718699373304844,
+            0.01573435589671135,
+            0.013524818234145641,
+            -0.010382534936070442,
+            0.03299780562520027,
+            0.023853834718465805,
+            -0.00400239834561944,
+            0.003767300397157669,
+            0.004025334492325783,
+            0.005229494534432888,
+            0.013326036743819714,
+            -0.0028517565224319696,
+            -0.00418971199542284,
+            -0.00036602639011107385,
+            0.013165482319891453,
+            -0.0014535930240526795,
+            -0.004197357688099146,
+            -0.010199043899774551,
+            0.0032493204344063997,
+            0.0005714981234632432,
+            -0.004698134958744049,
+            -0.00477458955720067,
+            -0.006383959203958511,
+            -0.0007339641451835632,
+            -0.00025230017490684986,
+            0.013570691458880901,
+            -0.005523844622075558,
+            -0.023119870573282242,
+            -0.002480951836332679,
+            -0.0017555886879563332,
+            -0.003763477550819516,
+            -0.005088053643703461,
+            -0.005944345146417618,
+            -0.0006622879882343113,
+            0.019908778369426727,
+            0.03217209503054619,
+            0.01966412365436554,
+            0.01675884798169136,
+            0.016070757061243057,
+            0.008249451406300068,
+            -0.011070625856518745,
+            0.010803034529089928,
+            0.002651063259691,
+            -0.0030543613247573376,
+            0.005030712578445673,
+            0.008669951930642128,
+            0.006024622358381748,
+            0.009740316309034824,
+            -0.01782921329140663,
+            0.005485617555677891,
+            -0.017355194315314293,
+            0.00641454104334116,
+            -0.02481716312468052,
+            -0.007851887494325638,
+            -0.006533045321702957,
+            -0.010887134820222855,
+            -0.008914606645703316,
+            0.004407607484608889,
+            0.024327853694558144,
+            -0.006655372679233551,
+            0.01479396503418684,
+            -0.02243177965283394,
+            0.013509527780115604,
+            0.007312882225960493,
+            -0.006399250123649836,
+            -0.014694574289023876,
+            -0.006540691014379263,
+            -0.004847221542149782,
+            -0.008692887611687183,
+            0.012202153913676739,
+            -0.0036640865728259087,
+            -0.016498902812600136,
+            -4.0691170397622045e-06,
+            -0.012163926847279072,
+            -0.013249581679701805,
+            -0.005795258563011885,
+            0.020153433084487915,
+            0.013165482319891453,
+            0.005951990373432636,
+            -0.020627450197935104,
+            0.004174421075731516,
+            -0.0014937317464500666,
+            0.02827291004359722,
+            -0.004950435366481543,
+            -0.0038017048500478268,
+            0.0022343855816870928,
+            0.01402177382260561,
+            -0.007882469333708286,
+            -0.0025765199679881334,
+            0.00270267017185688,
+            0.02130025066435337,
+            0.01016081590205431,
+            0.004438189323991537,
+            -0.007966569624841213,
+            0.021514324471354485,
+            -0.012852017767727375,
+            -0.00269311317242682,
+            -0.003715693484991789,
+            -0.005867890547960997,
+            -0.013647145591676235,
+            0.015573801472783089,
+            0.010145525448024273,
+            0.0027657451573759317,
+            -0.005558249540627003,
+            0.001599812530912459,
+            -0.009327461011707783,
+            -0.010688353329896927,
+            -0.018379686400294304,
+            0.008899315260350704,
+            -0.024358434602618217,
+            0.006598032079637051,
+            0.003727161791175604,
+            0.00022745243040844798,
+            -0.00588700408115983,
+            -0.003430900163948536,
+            0.0027389859315007925,
+            0.01116237137466669,
+            -0.005879358854144812,
+            -0.017003502696752548,
+            -0.002113969763740897,
+            0.02940443903207779,
+            -0.011911626905202866,
+            -0.0015376930823549628,
+            -0.003889627754688263,
+            -0.0037443640176206827,
+            0.002712226938456297,
+            -0.01591784693300724,
+            -0.0048242853954434395,
+            0.0013618475059047341,
+            -6.134287104941905e-05,
+            0.0076034097000956535,
+            0.01675884798169136,
+            -0.007056759670376778,
+            0.016743557527661324,
+            -0.009709734469652176,
+            -0.0015262248925864697,
+            0.017920957878232002,
+            -0.0009126767981797457,
+            -0.01789037697017193,
+            0.008142414502799511,
+            -0.007905405946075916,
+            0.030367767438292503,
+            -0.009189843200147152,
+            -0.027462491765618324,
+            0.025811072438955307,
+            0.02986316569149494,
+            -0.010107298381626606,
+            0.01857846789062023,
+            -0.006590386386960745,
+            0.017997412011027336,
+            0.028915129601955414,
+            -0.0016552420565858483,
+            -0.014319946058094501,
+            0.004927498754113913,
+            -0.007943633012473583,
+            -0.014702219516038895,
+            -0.0034481023903936148,
+            0.01643773913383484,
+            -0.026376836001873016,
+            0.0071446821093559265,
+            -0.025581708177924156,
+            0.0191901046782732,
+            -0.016728267073631287,
+            -0.025994563475251198,
+            -0.008371778763830662,
+            0.012576781213283539,
+            0.007305236998945475,
+            0.029985493049025536,
+            -0.0035455820616334677,
+            -0.034098751842975616,
+            -0.0011793122394010425,
+            0.015329146757721901,
+            -0.008685242384672165,
+            -0.01095594372600317,
+            0.03165220469236374,
+            0.01081832591444254,
+            -0.032325003296136856,
+            0.008188287727534771,
+            -0.026239218190312386,
+            0.000751166429836303,
+            -0.01943475939333439,
+            0.012248027138411999,
+            -0.03984813764691353,
+            -0.0024025856982916594,
+            -0.0013006838271394372,
+            -0.0009853086667135358,
+            -0.004078852944076061,
+            0.005267722066491842,
+            0.0034328114707022905,
+            0.014495791867375374,
+            -0.017676303163170815,
+            -0.027401328086853027,
+            0.011246471665799618,
+            0.008700533770024776,
+            -0.0232269074767828,
+            -0.01737048476934433,
+            0.0035474933683872223,
+            -0.019602959975600243,
+            0.026835564523935318,
+            -0.007037645671516657,
+            -0.001758455764502287,
+            -0.018379686400294304,
+            -0.0043540894985198975,
+            -0.0019314342644065619,
+            -0.01564261130988598,
+            0.014304655604064465,
+            0.008287678472697735,
+            0.01179694477468729,
+            -0.0037214276380836964,
+            -0.01341013703495264,
+            -9.332837180409115e-06,
+            0.011911626905202866,
+            0.011429962702095509,
+            -0.005504731088876724,
+            0.009610342793166637,
+            -0.007943633012473583,
+            -0.0021464629098773003,
+            -0.015390310436487198,
+            0.013777119107544422,
+            -0.0026204814203083515,
+            -0.008914606645703316,
+            -0.0019381240708753467,
+            0.012110408395528793,
+            0.008127124048769474,
+            0.003438545623794198,
+            -0.01018375251442194,
+            0.005302126519382,
+            -0.006835041102021933,
+            -0.01540560182183981,
+            -0.01156758051365614,
+            -0.019083067774772644,
+            -0.007786900736391544,
+            -0.020306341350078583,
+            0.01119295321404934,
+            0.032844897359609604,
+            0.005347999278455973,
+            -0.009228070266544819,
+            -0.025107691064476967,
+            -0.027355454862117767,
+            0.017951540648937225,
+            0.0026529745664447546,
+            -0.002366269938647747,
+            -0.02354801632463932,
+            -0.019159521907567978,
+            0.015596738085150719,
+            0.03529144451022148,
+            -0.0109788803383708,
+            -0.016177793964743614,
+            0.004090321250259876,
+            -0.013027863577008247,
+            -0.014281718991696835,
+            -0.00896812416613102,
+            -0.013035508804023266,
+            -0.0005246697110123932,
+            0.0010464723454788327,
+            -0.007572827860713005,
+            0.028731638565659523,
+            0.009877934120595455,
+            0.009342752397060394,
+            0.011200598441064358,
+            0.0031652203761041164,
+            0.005604122299700975,
+            -0.007278477773070335,
+            -0.0009117210865952075,
+            -0.007033823058009148,
+            0.023884417489171028,
+            -0.003960348200052977,
+            -0.00041811109986156225,
+            0.001845422899350524,
+            -0.004705780651420355,
+            0.012446808628737926,
+            0.00029530588653869927,
+            -0.0329672247171402,
+            0.03501620516180992,
+            -0.006395427044481039,
+            0.0016045909142121673,
+            -0.02597927302122116,
+            -0.011605808511376381,
+            0.021177923306822777,
+            0.006280745379626751,
+            0.001812929636798799,
+            -0.006762409117072821,
+            0.0037290730979293585,
+            0.009121033363044262,
+            0.020658032968640327,
+            -0.01079538930207491,
+            0.002070008311420679,
+            0.015252692624926567,
+            0.012270962819457054,
+            0.00031131357536651194,
+            0.0043540894985198975,
+            0.0057417405769228935,
+            -0.005328885745257139,
+            0.007542246021330357,
+            -0.022492943331599236,
+            -0.01142231747508049,
+            -0.011582871899008751,
+            0.009472724981606007,
+            -0.001574964728206396,
+            0.006112545263022184,
+            0.004935144446790218,
+            0.0034595706965774298,
+            -0.010535444132983685,
+            0.008899315260350704,
+            -0.020795650780200958,
+            -0.004147661849856377,
+            0.00032182608265429735,
+            0.027875347062945366,
+            -0.013004926964640617,
+            0.005856422241777182,
+            -0.009656216017901897,
+            0.009342752397060394,
+            0.020917978137731552,
+            -0.0006183265941217542,
+            0.0026778222527354956,
+            0.0006971703842282295,
+            -0.004254698287695646,
+            0.005615590140223503,
+            -0.003937412053346634,
+            -0.017263447865843773,
+            0.017477521672844887,
+            0.00015051999071147293,
+            0.004549048840999603,
+            -0.002289815340191126,
+            0.0010311814257875085,
+            0.01802799478173256,
+            0.020352214574813843,
+            -0.00835648737847805,
+            -0.002268790267407894,
+            -0.010688353329896927,
+            0.03819671645760536,
+            -0.004491707775741816,
+            0.01793624833226204,
+            0.004938967060297728,
+            -0.004533757921308279,
+            0.0009389580227434635,
+            -0.005340353585779667,
+            0.012936118058860302,
+            -0.00842529721558094,
+            -0.015634965151548386,
+            -0.010703643783926964,
+            -0.008234160020947456,
+            -0.013517173007130623,
+            0.019786451011896133,
+            -0.023853834718465805,
+            0.016957629472017288,
+            0.0017708796076476574,
+            0.009923807345330715,
+            0.010351953096687794,
+            -0.00010811158426804468,
+            0.0049848398193717,
+            0.0003978028253186494,
+            -0.018410267308354378,
+            0.026529746130108833,
+            -0.016407156363129616,
+            0.01877724938094616,
+            -0.01175107154995203,
+            -0.0026032791938632727,
+            0.013585981912910938,
+            0.006995595991611481,
+            0.026330964639782906,
+            -0.004461125936359167,
+            -0.0050115990452468395,
+            0.015657901763916016,
+            0.008494106121361256,
+            0.029694966971874237,
+            0.015795519575476646,
+            0.0059787495993077755,
+            -0.0009169773547910154,
+            0.013257227838039398,
+            -0.01666710339486599,
+            -0.015573801472783089,
+            0.008104187436401844,
+            -0.0007229737821035087,
+            -0.004545225761830807,
+            0.0022305629681795835,
+            0.0349244624376297,
+            -0.0010416939621791244,
+            0.011942208744585514,
+            0.009258652105927467,
+            -0.019205395132303238,
+            0.04177479445934296,
+            0.009977324865758419,
+            0.014969810843467712,
+            -0.0199393592774868,
+            -0.04474123194813728,
+            -0.015466765500605106,
+            -0.008876378647983074,
+            0.005795258563011885,
+            0.0005676753935404122,
+            0.0032665226608514786,
+            -0.028716348111629486,
+            -0.019067777320742607,
+            0.006857977714389563,
+            -0.01675884798169136,
+            0.015887266024947166,
+            -0.0061813541688025,
+            0.0047287167981266975,
+            0.005799081176519394,
+            0.0017326523084193468,
+            -0.0065215774811804295,
+            -0.007569005247205496,
+            0.017079956829547882,
+            0.0033468001056462526,
+            -0.006146949715912342,
+            0.021040305495262146,
+            -0.01058131642639637,
+            0.005669108591973782,
+            -0.0009160216432064772,
+            0.007358754985034466,
+            0.016453029587864876,
+            0.01301257312297821,
+            0.0028001496102660894,
+            0.004327330272644758,
+            0.014381109736859798,
+            0.02992432937026024,
+            0.0014918203232809901,
+            -0.010244916193187237,
+            -0.003870513988658786,
+            0.016223665326833725,
+            0.001654286403208971,
+            -0.019373595714569092,
+            0.014686928130686283,
+            -0.027049636468291283,
+            -0.003406052477657795,
+            -0.01811973936855793,
+            0.002033692318946123,
+            -0.0018301319796591997,
+            -0.007630168925970793,
+            2.1741776436101645e-05,
+            0.007905405946075916,
+            -0.000541871995665133,
+            0.012959054671227932,
+            -0.003409875091165304,
+            0.009916161186993122,
+            0.021651942282915115,
+            -0.008478814736008644,
+            -0.012974345125257969,
+            0.017966831102967262,
+            0.010848907753825188,
+            -0.015038619749248028,
+            0.0008300102199427783,
+            -0.000931790447793901,
+            -0.03137696906924248,
+            -0.0037462753243744373,
+            0.012706754729151726,
+            -0.0012557668378576636,
+            0.00488162599503994,
+            -0.0092204250395298,
+            -0.018058575689792633,
+            0.024220816791057587,
+            0.008784633129835129,
+            0.01456460077315569,
+            0.009044579230248928,
+            -0.010244916193187237,
+            0.0001916143373819068,
+            0.015810811892151833,
+            -0.009885579347610474,
+            -0.0052600763738155365,
+            0.01498510129749775,
+            -0.022676434367895126,
+            0.0113535076379776,
+            -0.008769342675805092,
+            0.0023242197930812836,
+            -0.002817352069541812,
+            -0.01121588982641697,
+            -0.008868733420968056,
+            -0.010925361886620522,
+            0.013333681970834732,
+            -0.023914998397231102,
+            -0.0005160685395821929,
+            0.008081250824034214,
+            0.0029205656610429287,
+            -0.035138532519340515,
+            -0.0033391546458005905,
+            0.014946874231100082,
+            0.029572639614343643,
+            -0.01782921329140663,
+            -0.01102475356310606,
+            0.01582610234618187,
+            -0.007859532721340656,
+            -0.000128658750327304,
+            0.0019199660746380687,
+            -0.006915318313986063,
+            -0.02223299816250801,
+            -0.010566025972366333,
+            -0.021651942282915115,
+            0.007867177948355675,
+            0.003247409127652645,
+            -0.008700533770024776,
+            -0.007492550648748875,
+            -0.014319946058094501,
+            -0.011689907871186733,
+            -0.024129072204232216,
+            -0.0031919796019792557,
+            -0.0010770541848614812,
+            -0.017095249146223068,
+            0.02682027406990528,
+            -0.004036802798509598,
+            0.014908647164702415,
+            -0.020336924120783806,
+            0.04064326360821724,
+            -0.015764938667416573,
+            -0.0016514193266630173,
+            -0.010107298381626606,
+            0.01278320886194706,
+            0.0053441766649484634,
+            -2.7714791940525174e-05,
+            -0.010321371257305145,
+            -0.015979010611772537,
+            0.015451474115252495,
+            0.008631723932921886,
+            0.0029090975876897573,
+            -0.008624078705906868,
+            -0.020168723538517952,
+            -0.009977324865758419,
+            -0.009273942559957504,
+            -0.031193476170301437,
+            -0.0016017238376662135,
+            -0.005787613335996866,
+            0.014648701064288616,
+            -0.008960478939116001,
+            -0.01652948372066021,
+            0.01240858156234026,
+            0.021468451246619225,
+            0.009029287844896317,
+            -0.01639186590909958,
+            0.01118530798703432,
+            -0.010864198207855225,
+            0.004808994475752115,
+            0.017248157411813736,
+            -0.004250875674188137,
+            0.009342752397060394,
+            -0.0038380208425223827,
+            -0.0023127517197281122,
+            -0.004464948549866676,
+            -0.007882469333708286,
+            -0.01217157207429409,
+            -0.007798369042575359,
+            -0.029832584783434868,
+            0.007974214851856232,
+            -0.008792279288172722,
+            0.03351769596338272,
+            0.01240858156234026,
+            0.024496054276823997,
+            -0.01119295321404934,
+            0.023899707943201065,
+            -0.0164836123585701,
+            0.023838544264435768,
+            0.005974926985800266,
+            -0.02570403553545475,
+            0.016835302114486694,
+            -0.010642480105161667,
+            -0.03501620516180992,
+            0.0005934788496233523,
+            -0.011766362935304642,
+            0.03458806127309799,
+            -0.0359642431139946,
+            0.011414671316742897,
+            0.0043540894985198975,
+            -0.020443959161639214,
+            -0.004323507659137249,
+            0.006028444971889257,
+            0.00038275084807537496,
+            0.02139199711382389,
+            -0.009640924632549286,
+            0.004961903672665358,
+            0.014327592216432095,
+            0.022156542167067528,
+            -0.012278608977794647,
+            0.008302969858050346,
+            0.023838544264435768,
+            0.021514324471354485,
+            0.010856552980840206,
+            -0.01680472120642662,
+            0.02883867546916008,
+            -0.0037080480251461267,
+            0.0008797057089395821,
+            0.004235584754496813,
+            -0.02706492878496647,
+            -0.006311327219009399,
+            -0.002654885873198509,
+            0.011621098965406418,
+            0.008845796808600426,
+            0.0015701863449066877,
+            0.013249581679701805,
+            0.01652948372066021,
+            -0.01315783616155386,
+            0.004300571046769619,
+            0.005042180884629488,
+            0.006104899570345879,
+            0.01764572225511074,
+            -0.030750039964914322,
+            0.006028444971889257,
+            -0.021361414343118668,
+            -0.005156862549483776,
+            0.0071561504155397415,
+            0.011957499198615551,
+            0.015856683254241943,
+            -0.03694286197423935,
+            0.0011821791995316744,
+            -0.003293281886726618,
+            0.013096672482788563,
+            0.020841524004936218,
+            0.01479396503418684,
+            -0.01296669989824295,
+            0.008555269800126553,
+            -0.02186601608991623,
+            -0.00314228399656713,
+            -0.020963851362466812,
+            0.002131171990185976,
+            -0.019358305260539055,
+            0.006861800327897072,
+            0.006865622941404581,
+            -0.00559647660702467,
+            -0.0013341327430680394,
+            0.01685059443116188,
+            -0.022921089082956314,
+            -0.008302969858050346,
+            0.012377999722957611,
+            -0.0036507071927189827,
+            -0.004495530389249325,
+            -0.008624078705906868,
+            0.0012376088416203856,
+            0.012997281737625599,
+            0.0069688367657363415,
+            -0.006735650356858969,
+            0.010244916193187237,
+            0.0014268339145928621,
+            0.0025057995226234198,
+            0.006318972446024418,
+            0.004063562024384737,
+            -0.01652948372066021,
+            -0.0014956430532038212,
+            -5.632553802570328e-05,
+            0.012240380980074406,
+            0.008104187436401844,
+            -0.017309321090579033,
+            -0.02111675962805748,
+            0.019281849265098572,
+            -0.028578728437423706,
+            -0.003803616389632225,
+            -0.0008457790245302022,
+            -0.0021445516031235456,
+            0.014518728479743004,
+            0.012821435928344727,
+            -0.01058131642639637,
+            -0.0017498546512797475,
+            -0.01825735904276371,
+            0.0037080480251461267,
+            0.0009833972435444593,
+            -0.010650125332176685,
+            0.004701958037912846,
+            0.010527797974646091,
+            0.011888690292835236,
+            0.0025516722816973925,
+            -0.018242066726088524,
+            0.00994674302637577,
+            0.02152961492538452,
+            0.014411691576242447,
+            -0.004629326052963734,
+            -0.014725156128406525,
+            -0.03614773601293564,
+            -0.023012833669781685,
+            0.026193344965577126,
+            -0.009839707054197788,
+            -0.02157548815011978,
+            0.017523394897580147,
+            0.011644035577774048,
+            0.022034214809536934,
+            -0.009380979463458061,
+            -0.006796814035624266,
+            -0.0019228331511840224,
+            0.017615139484405518,
+            0.004403784871101379,
+            -0.0027389859315007925,
+            0.0017890376038849354,
+            -0.0025153562892228365,
+            0.04116315767168999,
+            0.0021005901508033276,
+            0.005336530972272158,
+            -0.0187313761562109,
+            0.008257096633315086,
+            -0.02477128989994526,
+            0.009442143142223358,
+            -0.011368799023330212,
+            0.029419729486107826,
+            0.019465340301394463,
+            0.0071676187217235565,
+            0.015657901763916016,
+            -0.00901399739086628,
+            0.03669820725917816,
+            -0.014847483485937119,
+            -0.013310745358467102,
+            -0.0024140540044754744,
+            -0.018761958926916122,
+            -0.009029287844896317,
+            -0.0037940593902021646,
+            -0.005546781234443188,
+            0.006953545846045017,
+            -0.0046255034394562244,
+            0.006563627161085606,
+            0.002496242756024003,
+            -0.017584558576345444,
+            0.011284698732197285,
+            -0.022263579070568085,
+            0.01779863052070141,
+            -0.01243151817470789,
+            -0.021269669756293297,
+            0.0014669726369902492,
+            0.01830323040485382,
+            -0.017966831102967262,
+            0.00038824600051157176,
+            -0.0021292606834322214,
+            0.02552054449915886,
+            0.008547624573111534,
+            -0.005642349366098642,
+            0.008272388018667698,
+            -0.004067384637892246,
+            -0.006173708941787481,
+            0.01905248686671257,
+            0.028945710510015488,
+            0.019419468939304352,
+            -0.007282300386577845,
+            0.014992746524512768,
+            -0.004350266885012388,
+            -0.02977142110466957,
+            0.012202153913676739,
+            -0.010803034529089928,
+            0.006586563773453236,
+            0.007301414385437965,
+            -0.009556825272738934,
+            -0.015887266024947166,
+            0.008562915027141571,
+            -0.0149545194581151,
+            0.006387781817466021,
+            0.0014077202649787068,
+            0.0035264682956039906,
+            -0.003073474857956171,
+            0.001033092732541263,
+            -0.008792279288172722,
+            0.027493074536323547,
+            0.023379815742373466,
+            0.012951409444212914,
+            -0.023807961493730545,
+            0.005435922183096409,
+            -0.02448076196014881,
+            -0.0031250817701220512,
+            -0.00686944555491209,
+            -0.0036239479668438435,
+            -0.008432942442595959,
+            -0.0071561504155397415,
+            -0.007159973029047251,
+            -0.01732461154460907,
+            -0.0010178018128499389,
+            0.013777119107544422,
+            -0.001823442173190415,
+            0.006880913861095905,
+            0.0031441953033208847,
+            -0.003029513405635953,
+            -0.0006799680995754898,
+            0.029511475935578346,
+            0.00861643347889185,
+            -0.008792279288172722,
+            -0.012018662877380848,
+            -0.01078774407505989,
+            -0.013035508804023266,
+            0.027003765106201172,
+            0.006032268051058054,
+            -0.013754182495176792,
+            0.012225090526044369,
+            -0.0071790870279073715,
+            -0.021606069058179855,
+            -0.003543670754879713,
+            0.009709734469652176,
+            -0.03403758630156517,
+            0.034007005393505096,
+            -0.004113257396966219,
+            0.0057417405769228935,
+            0.0036602639593183994,
+            0.05174447223544121,
+            -0.005890826694667339,
+            -0.007068227510899305,
+            0.017095249146223068,
+            0.014595182612538338,
+            0.011911626905202866,
+            -0.007756318897008896,
+            0.00559265399351716,
+            -0.023395108059048653,
+            0.007171441335231066,
+            -0.01764572225511074,
+            -0.012003372423350811,
+            0.02822703868150711,
+            0.0010244916193187237,
+            -0.017171703279018402,
+            -0.01117001660168171,
+            -0.015466765500605106,
+            0.007882469333708286,
+            0.029740838333964348,
+            0.022538816556334496,
+            -0.018364394083619118,
+            -0.00609725434333086,
+            -0.01458753738552332,
+            -0.016972921788692474,
+            -0.010604253038764,
+            0.00385713460855186,
+            -0.012423872016370296,
+            -0.0027313404716551304,
+            -0.006158418022096157,
+            0.0006900027510710061,
+            -0.0016848682425916195,
+            -0.000514635001309216,
+            0.06348790228366852,
+            -0.007935987785458565,
+            0.006219581700861454,
+            -0.016070757061243057,
+            0.0007707579061388969,
+            -0.0034538365434855223,
+            -0.008150060661137104,
+            0.0007822260959073901,
+            -0.01759984903037548,
+            0.0287010557949543,
+            -0.024939490482211113,
+            0.0010627189185470343,
+            -0.014992746524512768,
+            0.006877091247588396,
+            -0.029465602710843086,
+            -0.0013752271188423038,
+            -0.01943475939333439,
+            0.004598744213581085,
+            -0.0011038132943212986,
+            -0.01016081590205431,
+            0.015313856303691864,
+            -0.0311781857162714,
+            -0.014113519340753555,
+            0.007572827860713005,
+            0.010436052456498146,
+            0.015084492042660713,
+            -0.020627450197935104,
+            -0.007278477773070335,
+            -0.01877724938094616,
+            -0.023670343682169914,
+            0.008792279288172722,
+            -0.0037500981707125902,
+            0.02481716312468052,
+            -0.01923597790300846,
+            0.0037711230106651783,
+            -0.016820011660456657,
+            0.011147080920636654,
+            0.01402177382260561,
+            0.004487885162234306,
+            -0.0029129202011972666,
+            -0.010428407229483128,
+            0.023838544264435768,
+            -0.00758429616689682,
+            -0.034618642181158066,
+            0.007094986736774445,
+            0.004606389440596104,
+            -0.00779454642906785,
+            -0.011598162353038788,
+            0.0113535076379776,
+            -0.0028613132890313864,
+            0.006292213685810566,
+            -0.02050512284040451,
+            -0.0011028576409444213,
+            -0.014312300831079483,
+            0.012454453855752945,
+            0.013754182495176792,
+            0.01357833668589592,
+            3.586795719456859e-05,
+            -0.010749517008662224,
+            -0.0264532919973135,
+            -0.0004510821308940649,
+            -0.004908385220915079,
+            0.0013647145824506879,
+            0.008700533770024776,
+            0.0031652203761041164,
+            0.022599980235099792,
+            -0.016024883836507797,
+            -0.019495923072099686,
+            0.0002515834057703614,
+            -0.01853259466588497,
+            -0.02804354764521122,
+            0.005290658213198185,
+            -0.004120903089642525,
+            -0.004178243689239025,
+            0.003673643572255969,
+            -0.011108852922916412,
+            -0.010420762002468109,
+            0.007282300386577845,
+            0.0024943312164396048,
+            -0.01639186590909958,
+            0.026835564523935318,
+            -0.0076378146186470985,
+            -0.014060000889003277,
+            -0.006322795525193214,
+            0.008364133536815643,
+            -0.009725024923682213,
+            0.0057302722707390785,
+            -0.03038305789232254,
+            0.0033276863396167755,
+            -0.007159973029047251,
+            -0.012874954380095005,
+            -0.0040100435726344585,
+            0.01591784693300724,
+            0.010260207578539848,
+            -0.007030000444501638,
+            -0.004143839236348867,
+            -0.010130234062671661,
+            -0.0053212400525808334,
+            0.007446677889674902,
+            0.03052067570388317,
+            0.001286348677240312,
+            -0.010458989068865776,
+            -0.014809255488216877,
+            0.017875084653496742,
+            -0.010038488544523716,
+            -0.0287010557949543,
+            0.014281718991696835,
+            0.014732801355421543,
+            -0.03394584357738495,
+            0.0059175859205424786,
+            -0.02073448710143566,
+            -0.011666972190141678,
+            -0.01948063261806965,
+            -0.009625634178519249,
+            0.02393028885126114,
+            -0.02134612388908863,
+            0.009877934120595455,
+            0.005068940110504627,
+            0.03324246034026146,
+            0.01639186590909958,
+            -0.01938888616859913,
+            0.00936568807810545,
+            -0.0037787684705108404,
+            0.006441299803555012,
+            0.008906960487365723,
+            -0.014251137152314186,
+            -0.0002635294513311237,
+            0.016269538551568985,
+            -0.0013704487355425954,
+            0.0018177081365138292,
+            -0.003918298054486513,
+            -0.03211093321442604,
+            0.012630299665033817,
+            -0.02837994694709778,
+            -0.0037787684705108404,
+            0.010573671199381351,
+            -0.01859375834465027,
+            -0.02617805451154709,
+            -0.0024580154567956924,
+            0.012339772656559944,
+            -0.029373856261372566,
+            0.05997098609805107,
+            0.0014268339145928621,
+            0.021147342398762703,
+            -0.01456460077315569,
+            -0.008539978414773941,
+            -0.0047287167981266975,
+            0.0018224865198135376,
+            -0.001679134089499712,
+            0.00024632716667838395,
+            0.014686928130686283,
+            -0.008998706005513668,
+            0.017951540648937225,
+            0.010046134702861309,
+            -0.001892251311801374,
+            0.02477128989994526,
+            0.00678916834294796,
+            0.00739698251709342,
+            0.005034535191953182,
+            0.021055595949292183,
+            -0.0023452448658645153,
+            0.016820011660456657,
+            0.009327461011707783,
+            -0.0109788803383708,
+            0.001719272811897099,
+            -0.008792279288172722,
+            0.014281718991696835,
+            0.011040044017136097,
+            0.002257321961224079,
+            -0.008211224339902401,
+            -0.0032550545874983072,
+            0.021407287567853928,
+            -0.010757162235677242,
+            -0.022034214809536934,
+            -0.010267852805554867,
+            0.0059175859205424786,
+            0.007718091830611229,
+            0.01718699373304844,
+            0.01718699373304844,
+            -0.007458146195858717,
+            -0.011888690292835236,
+            0.022492943331599236,
+            -0.012645591050386429,
+            0.014725156128406525,
+            0.0036602639593183994,
+            -0.017706885933876038,
+            0.02200363390147686,
+            -0.011040044017136097,
+            -0.008295323699712753,
+            -0.0006183265941217542,
+            2.241374022560194e-05,
+            -0.006938254926353693,
+            -0.01587197557091713,
+            -0.020658032968640327,
+            -0.006762409117072821,
+            0.009136324748396873,
+            -0.014037064276635647,
+            0.0032187385950237513,
+            -0.0060016862116754055,
+            0.009755606763064861,
+            0.013272518292069435,
+            0.012668526731431484,
+            -0.003195802215486765,
+            -0.007297591306269169
+          ],
+          "keyphrases": [
+            "Mixture-Of-Experts",
+            "Transformer language models",
+            "self-attention module",
+            "Feed-Forward Network",
+            "computational efficiency"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "e61dc02d-94db-4be4-998f-1a0ce34ae92b",
+      "type": "next",
+      "source": {
+        "id": "767dcc8f-1e79-439d-92f3-8f20092ad68e",
+        "properties": {
+          "page_content": "2. Preliminaries: Mixture-Of-Experts For Transformers\n\nWe first introduce a generic MoE architecture commonly used in Transformer language models. A standard Transformer language model is constructed by stacking layers of standard Transformer blocks, where each block can be represented as follows:\n\n$$\\mathbf{u}{1:T}^{l}=\\text{Self-Att}\\left(\\mathbf{h}{1:T}^{l-1}\\right)+\\mathbf{h}{1:T}^{l-1},\\tag{1}$$ $$\\mathbf{h}{t}^{l}=\\text{FFN}\\left(\\mathbf{u}{t}^{l}\\right)+\\mathbf{u}{t}^{l},\\tag{2}$$\n\nwhere denotes the sequence length, Self-Att(·) denotes the self-attention module, FFN(·) denotes the Feed-Forward Network (FFN), u 1: ∈ R× are the hidden states of all tokens after the -th attention module, and h\n\n∈ Ris the output hidden state of the -th token after the -th Transformer block. For brevity, we omit the layer normalization in the above formulations.\n\nA typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022). An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. Then, each token will be assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts. If the -th FFN is substituted with an MoE layer, the computation for its output hidden state h\n\nis expressed as: where denotes the total number of experts, FFN(·) is the -th expert FFN, , denotes the gate value for the -th expert, , denotes the token-to-expert affinity, Topk(·, ) denotes the set comprising highest affinity scores among those calculated for the -th token and all experts, and e\n\nis the centroid of the -th expert in the -th layer. Note that ,is sparse, indicating that only out of gate values are nonzero. This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only experts. Also, in the above formulations, we omit the layer normalization operation for brevity.\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{N}\\left(g_{i,t}\\operatorname{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\operatorname{Topk}({s_{j,t}|1\\leqslant j\\leqslant N},K),\\ 0,&\\text{otherwise},\\end{cases}$$ $$s_{i,t}=\\operatorname{Softmax}{i}\\left(\\mathbf{u}{t}^{l}\\,\\mathbf{e}_{i}^{l}\\right),$$\n\n$$(3)$$\n\n(4) $\\binom{4}{5}$ .\n\n",
+          "title": "Preliminaries: Mixture-Of-Experts For Transformers",
+          "embedding": [
+            0.0012184951920062304,
+            -0.02724841982126236,
+            -0.02786005660891533,
+            -0.019281849265098572,
+            0.028028255328536034,
+            0.023196324706077576,
+            0.0037118708714842796,
+            0.03721809759736061,
+            -0.0036163025069981813,
+            0.03367060422897339,
+            -0.006009331438690424,
+            0.0015032885130494833,
+            -0.013440718874335289,
+            0.0020107559394091368,
+            -0.02350214309990406,
+            0.012936118058860302,
+            0.0041056121699512005,
+            0.036545298993587494,
+            0.003880070988088846,
+            -0.017630429938435555,
+            0.0016657545929774642,
+            -0.0020566286984831095,
+            -0.0359642431139946,
+            0.003516911529004574,
+            0.014763383194804192,
+            -0.0010483836522325873,
+            0.04584217816591263,
+            -0.011781653389334679,
+            -0.016330702230334282,
+            0.026667363941669464,
+            0.02402203530073166,
+            -0.003767300397157669,
+            0.002329953946173191,
+            -0.03529144451022148,
+            -0.05529196560382843,
+            -0.002939679427072406,
+            0.046514976769685745,
+            0.04406842961907387,
+            -0.03510795161128044,
+            0.027737729251384735,
+            0.019327722489833832,
+            0.008410005830228329,
+            -0.016453029587864876,
+            -0.02949618361890316,
+            -0.015283274464309216,
+            -0.009480370208621025,
+            -0.03651471808552742,
+            0.004227939527481794,
+            0.014755737967789173,
+            -0.027324873954057693,
+            -0.015206819400191307,
+            -0.008906960487365723,
+            0.009029287844896317,
+            0.025061817839741707,
+            -0.05318181961774826,
+            0.03244733065366745,
+            0.0012739247176796198,
+            0.0359642431139946,
+            0.053120654076337814,
+            0.006900027394294739,
+            0.025627581402659416,
+            -0.04003162682056427,
+            -0.020000522956252098,
+            -0.0015720976516604424,
+            0.040092792361974716,
+            0.0058067268691957,
+            0.019817031919956207,
+            -0.02495478093624115,
+            0.0116134537383914,
+            0.033640023320913315,
+            -0.03782973438501358,
+            0.008088896982371807,
+            -0.015275629237294197,
+            -0.018976030871272087,
+            -0.004931321833282709,
+            0.008203578181564808,
+            0.015994302928447723,
+            0.029465602710843086,
+            0.035230278968811035,
+            0.004912207834422588,
+            -0.02949618361890316,
+            0.027080219238996506,
+            -0.0006469970685429871,
+            0.021086178719997406,
+            0.028288202360272408,
+            -0.03168278560042381,
+            -0.01886899583041668,
+            -0.051530398428440094,
+            0.005971104372292757,
+            0.02536763623356819,
+            -0.023884417489171028,
+            0.01834910362958908,
+            -0.003622036660090089,
+            0.015084492042660713,
+            0.0006288390723057091,
+            0.038043808192014694,
+            0.055597785860300064,
+            -0.0010588961886242032,
+            -0.02565816417336464,
+            -0.011582871899008751,
+            0.010038488544523716,
+            0.010871844366192818,
+            0.005653817672282457,
+            0.023624470457434654,
+            0.017385775223374367,
+            -0.011712844483554363,
+            -0.006502463482320309,
+            0.042111191898584366,
+            0.034435153007507324,
+            0.08899315446615219,
+            0.007599587086588144,
+            0.0032454978208988905,
+            0.0005939566763117909,
+            0.043517958372831345,
+            -0.01177400816231966,
+            -0.012309190817177296,
+            -0.04550577700138092,
+            -0.057188041508197784,
+            0.003386938711628318,
+            -0.030214857310056686,
+            0.021223796531558037,
+            -0.003618213813751936,
+            0.009954389184713364,
+            -0.01877724938094616,
+            -0.05923702195286751,
+            0.037890899926424026,
+            -0.0064260088838636875,
+            0.012798500247299671,
+            -0.011085917241871357,
+            0.005256253760308027,
+            -0.001563496538437903,
+            0.020765069872140884,
+            -0.022125961259007454,
+            -0.03244733065366745,
+            0.036300644278526306,
+            0.02734016440808773,
+            0.023578599095344543,
+            0.020581578835844994,
+            0.0029587929602712393,
+            0.0038514004554599524,
+            0.016131920740008354,
+            -0.013822991400957108,
+            -0.0076875099912285805,
+            0.0016886909725144506,
+            -0.009725024923682213,
+            0.003293281886726618,
+            0.031147602945566177,
+            0.0689314678311348,
+            -0.0224776528775692,
+            0.011712844483554363,
+            -0.0219577606767416,
+            0.020520415157079697,
+            -0.02931269258260727,
+            0.01315783616155386,
+            -0.00795892346650362,
+            0.01703408546745777,
+            -0.007068227510899305,
+            0.009182197041809559,
+            -0.016208374872803688,
+            0.0015692306915298104,
+            -0.003788325469940901,
+            -0.030964111909270287,
+            0.0006613322766497731,
+            0.025168854743242264,
+            -0.009121033363044262,
+            0.018318522721529007,
+            0.04755476117134094,
+            0.024220816791057587,
+            -0.04868628829717636,
+            0.024419598281383514,
+            0.0011879133526235819,
+            0.03076533041894436,
+            0.040245700627565384,
+            0.0068847364746034145,
+            -0.015573801472783089,
+            0.012347417883574963,
+            0.007928341627120972,
+            0.021835433319211006,
+            0.011468189768493176,
+            0.025031235069036484,
+            -0.03929766267538071,
+            -0.00842529721558094,
+            0.032600242644548416,
+            0.007763964589685202,
+            0.03489387780427933,
+            0.018333813175559044,
+            0.037340424954891205,
+            0.019602959975600243,
+            0.010619543492794037,
+            0.015260337851941586,
+            0.03189685940742493,
+            0.011980435810983181,
+            0.03969522938132286,
+            -0.009633279405534267,
+            0.0167129747569561,
+            -0.03156045824289322,
+            -0.003220649901777506,
+            0.020994432270526886,
+            0.01137644425034523,
+            0.02050512284040451,
+            -0.0067088911309838295,
+            -0.01113943476229906,
+            0.003945057280361652,
+            -0.013287809677422047,
+            0.004407607484608889,
+            0.010000261478126049,
+            -0.0019381240708753467,
+            0.020902687683701515,
+            0.00837942399084568,
+            -0.04324272274971008,
+            0.02116263285279274,
+            -0.03422107920050621,
+            0.012278608977794647,
+            -0.009029287844896317,
+            0.006785345729440451,
+            0.0038552230689674616,
+            -0.017171703279018402,
+            0.0024656609166413546,
+            0.029664384201169014,
+            -0.017339903861284256,
+            -0.0012356974184513092,
+            0.00038083948311395943,
+            0.013058445416390896,
+            -0.0007425653166137636,
+            -0.004059739410877228,
+            -0.013196064159274101,
+            -0.017446938902139664,
+            -0.014763383194804192,
+            -0.0004931321600452065,
+            -0.006059026811271906,
+            -0.018456140533089638,
+            0.041407812386751175,
+            0.011903980746865273,
+            0.007400805130600929,
+            0.01877724938094616,
+            -0.010329016484320164,
+            0.021651942282915115,
+            0.008371778763830662,
+            0.028961002826690674,
+            -0.004357912112027407,
+            -0.02227886952459812,
+            0.02552054449915886,
+            -0.044404830783605576,
+            -0.03917533531785011,
+            0.025734618306159973,
+            0.026009853929281235,
+            -0.027798892930150032,
+            -0.0033391546458005905,
+            -0.011743426322937012,
+            0.0074275643564760685,
+            0.012714399956166744,
+            0.003622036660090089,
+            0.0030142224859446287,
+            0.005325062666088343,
+            -0.007278477773070335,
+            0.047585342079401016,
+            0.018456140533089638,
+            -0.017905667424201965,
+            -0.019924068823456764,
+            -0.01984761469066143,
+            0.028012964874505997,
+            0.018884286284446716,
+            0.001693469355814159,
+            0.017875084653496742,
+            0.02238590642809868,
+            -0.0010416939621791244,
+            0.05654582008719444,
+            -0.018945449963212013,
+            -0.01573435589671135,
+            -0.006972659379243851,
+            -0.002561229048296809,
+            -0.04853338003158569,
+            -0.02200363390147686,
+            0.007935987785458565,
+            0.05214203521609306,
+            -0.02327278070151806,
+            -0.0057073356583714485,
+            -0.0008553358493372798,
+            0.043426211923360825,
+            0.023869125172495842,
+            0.03519969806075096,
+            -2.6818839614861645e-05,
+            -0.004858689848333597,
+            0.020719196647405624,
+            0.03149929642677307,
+            -0.002643417799845338,
+            -0.016468320041894913,
+            -0.005168330855667591,
+            -0.03773799166083336,
+            -0.031346384435892105,
+            -0.017431648448109627,
+            -0.009686797857284546,
+            -0.03474096953868866,
+            0.002180867362767458,
+            0.017920957878232002,
+            0.02163665182888508,
+            -0.003543670754879713,
+            0.008333551697432995,
+            0.03177453204989433,
+            -0.02597927302122116,
+            -0.05841131508350372,
+            0.0032875477336347103,
+            0.004744007717818022,
+            0.0005552515503950417,
+            -0.04905327036976814,
+            0.0002306778624188155,
+            0.012561490759253502,
+            0.017446938902139664,
+            -0.02107088826596737,
+            -0.0047975261695683,
+            0.03090294823050499,
+            -0.007175263948738575,
+            -0.0014297009911388159,
+            -0.01163639035075903,
+            -0.03029131144285202,
+            -0.0015510726952925324,
+            0.011689907871186733,
+            -0.008494106121361256,
+            -0.026957891881465912,
+            -0.028823383152484894,
+            -0.007851887494325638,
+            0.009801479987800121,
+            -0.027645982801914215,
+            -0.015497347339987755,
+            0.006441299803555012,
+            0.012737336568534374,
+            0.021896596997976303,
+            0.02967967465519905,
+            -0.015520283952355385,
+            0.017064666375517845,
+            -0.0179821215569973,
+            -0.040704429149627686,
+            -0.028853965923190117,
+            0.0022038037423044443,
+            -0.008348842151463032,
+            -0.032508496195077896,
+            -0.004514643922448158,
+            0.0007502107764594257,
+            -0.01689646579325199,
+            -0.03498562425374985,
+            -0.023655053228139877,
+            -0.00943449791520834,
+            -0.014289364218711853,
+            0.011062980629503727,
+            -0.0020470719318836927,
+            -0.007836596108973026,
+            -0.01605546660721302,
+            0.011322925798594952,
+            0.007974214851856232,
+            0.010092006996273994,
+            0.0009609387489035726,
+            -0.0016925137024372816,
+            0.0062272269278764725,
+            -0.0016389954835176468,
+            -0.024465471506118774,
+            -0.03431282564997673,
+            -0.00899106077849865,
+            -0.015856683254241943,
+            0.007660750765353441,
+            0.0349244624376297,
+            -0.007974214851856232,
+            -0.03718751668930054,
+            -0.021988343447446823,
+            -0.0025115336757153273,
+            -0.0648946613073349,
+            -0.030214857310056686,
+            -0.0349244624376297,
+            -0.030092529952526093,
+            -0.0007473436999134719,
+            -0.0029301224276423454,
+            -0.007836596108973026,
+            0.024832453578710556,
+            -0.0006197600741870701,
+            -0.01240858156234026,
+            0.014778673648834229,
+            0.01587197557091713,
+            0.021560197696089745,
+            -0.011873398907482624,
+            -0.01793624833226204,
+            0.044404830783605576,
+            -0.022951669991016388,
+            -0.001025447272695601,
+            0.01998523250222206,
+            0.003316218266263604,
+            0.0026204814203083515,
+            0.021468451246619225,
+            0.0033601797185838223,
+            -0.007030000444501638,
+            -0.04639264941215515,
+            0.040704429149627686,
+            -0.00032756017753854394,
+            -0.02893042005598545,
+            0.009212778881192207,
+            0.007259364239871502,
+            0.011621098965406418,
+            0.02833407372236252,
+            -0.034343406558036804,
+            -0.014801610261201859,
+            -0.012018662877380848,
+            0.0012089383089914918,
+            -0.021713105961680412,
+            0.012125699780881405,
+            -0.015436183661222458,
+            -0.013134900480508804,
+            -0.023150453343987465,
+            -0.013815346173942089,
+            0.0033468001056462526,
+            0.011720489710569382,
+            0.03247791528701782,
+            -0.018379686400294304,
+            0.0033028386533260345,
+            -0.017676303163170815,
+            -0.0368817001581192,
+            -0.03029131144285202,
+            0.019113650545477867,
+            0.006926786620169878,
+            -0.02734016440808773,
+            0.024939490482211113,
+            -0.003333420492708683,
+            0.007385514210909605,
+            0.02293637953698635,
+            0.038563698530197144,
+            0.033028386533260345,
+            0.02935856580734253,
+            0.0001604352000867948,
+            0.02902216650545597,
+            -0.010665416717529297,
+            -0.010191397741436958,
+            -0.00784424226731062,
+            -0.0010264030424878001,
+            0.02425139956176281,
+            -0.007282300386577845,
+            0.05874771252274513,
+            -0.021789560094475746,
+            -0.009189843200147152,
+            -0.02541350945830345,
+            -0.006376313511282206,
+            0.01276027224957943,
+            -0.0184408500790596,
+            0.002058540005236864,
+            0.029832584783434868,
+            0.02082623355090618,
+            -0.01839497685432434,
+            0.015795519575476646,
+            -0.02223299816250801,
+            -0.02382325381040573,
+            -0.02786005660891533,
+            -0.0015548954252153635,
+            -0.011903980746865273,
+            -0.025153562426567078,
+            -0.03001607581973076,
+            -0.004839576315134764,
+            -0.09296879172325134,
+            -0.004101789090782404,
+            0.04321213811635971,
+            -0.005795258563011885,
+            0.032325003296136856,
+            -0.008937542326748371,
+            0.004151484929025173,
+            -0.03935882821679115,
+            0.00418971199542284,
+            0.03605598956346512,
+            0.039634063839912415,
+            0.0006541646434925497,
+            -0.005692044738680124,
+            -0.0009050313383340836,
+            -0.04675963148474693,
+            0.020428668707609177,
+            -0.005944345146417618,
+            -0.004289103206247091,
+            0.0002034409117186442,
+            -0.030964111909270287,
+            0.02293637953698635,
+            0.02620863728225231,
+            0.03749333694577217,
+            0.03908359259366989,
+            -0.017049375921487808,
+            -0.06532280892133713,
+            -0.0015376930823549628,
+            -0.000322303909342736,
+            0.025811072438955307,
+            0.009556825272738934,
+            -0.004442012403160334,
+            -0.009801479987800121,
+            -0.005252431146800518,
+            -0.01998523250222206,
+            0.016040174290537834,
+            0.01282908208668232,
+            0.013356618583202362,
+            0.0304289311170578,
+            -0.011491126380860806,
+            0.002372003858909011,
+            0.024312563240528107,
+            0.031071148812770844,
+            0.0031212589237838984,
+            -0.050582364201545715,
+            -0.017125830054283142,
+            -0.009319815784692764,
+            0.04883919656276703,
+            0.011361153796315193,
+            -0.021850723773241043,
+            -0.011116499081254005,
+            0.02256939746439457,
+            0.026147473603487015,
+            0.0020967673044651747,
+            -0.018853703513741493,
+            -0.015482055954635143,
+            0.011177662760019302,
+            -0.018700795248150826,
+            0.007309059612452984,
+            -0.0019916424062103033,
+            -0.010497216135263443,
+            -0.002899540588259697,
+            -0.0003841843572445214,
+            -0.018043285235762596,
+            0.01909835822880268,
+            -0.00835648737847805,
+            0.04859454184770584,
+            -0.01074187085032463,
+            0.02055099606513977,
+            -0.003369736485183239,
+            -0.0021827789023518562,
+            -0.0023758267052471638,
+            0.05113283544778824,
+            -0.00784424226731062,
+            0.01582610234618187,
+            0.04097966477274895,
+            -0.002408319851383567,
+            0.04657614231109619,
+            -0.011452899314463139,
+            -0.033120132982730865,
+            -0.0026663541793823242,
+            0.012225090526044369,
+            0.006150772329419851,
+            0.004617857746779919,
+            -0.022829342633485794,
+            -0.00800479669123888,
+            0.004526112228631973,
+            -0.029236238449811935,
+            -0.01591784693300724,
+            -0.013089027255773544,
+            -0.012125699780881405,
+            0.006842686794698238,
+            0.004338798578828573,
+            0.009212778881192207,
+            -0.021009724587202072,
+            0.044496577233076096,
+            -0.028395237401127815,
+            0.02266114391386509,
+            0.02659090980887413,
+            -0.008287678472697735,
+            -0.0008639369625598192,
+            -0.023945581167936325,
+            0.012852017767727375,
+            -0.009870288893580437,
+            -0.022309452295303345,
+            -0.01694233901798725,
+            0.006475704722106457,
+            0.0011238825973123312,
+            -0.045995086431503296,
+            0.03464922308921814,
+            0.01666710339486599,
+            -0.016300121322274208,
+            0.004560516681522131,
+            -0.0048013487830758095,
+            0.016743557527661324,
+            -0.002679733792319894,
+            0.02411377988755703,
+            0.013861218467354774,
+            0.02574990876019001,
+            -0.010458989068865776,
+            0.018884286284446716,
+            0.003998575732111931,
+            0.022997543215751648,
+            0.015053910203278065,
+            0.03253907710313797,
+            0.05663756653666496,
+            -0.005600299220532179,
+            -0.007989505305886269,
+            -0.012507972307503223,
+            0.019419468939304352,
+            -0.016315411776304245,
+            -0.00784424226731062,
+            0.02977142110466957,
+            0.015994302928447723,
+            -0.006575095467269421,
+            -0.0032990160398185253,
+            -0.0159637201577425,
+            0.013165482319891453,
+            -0.017768049612641335,
+            0.006120190490037203,
+            0.035750169306993484,
+            0.003279902273789048,
+            -0.02162136137485504,
+            0.03061242215335369,
+            -0.0040329801850020885,
+            0.034007005393505096,
+            0.00630750460550189,
+            0.02168252505362034,
+            -0.038838937878608704,
+            -0.013517173007130623,
+            0.017661012709140778,
+            0.021040305495262146,
+            -0.009052224457263947,
+            -0.01782921329140663,
+            0.028899839147925377,
+            0.018135031685233116,
+            -0.002549760974943638,
+            -0.01418232824653387,
+            -0.004430544096976519,
+            0.02851756475865841,
+            -0.020673323422670364,
+            0.023532725870609283,
+            0.005581185687333345,
+            0.02781418338418007,
+            -0.0010273586958646774,
+            -0.0013360441662371159,
+            0.004040625412017107,
+            -0.016544776037335396,
+            0.004805171396583319,
+            0.03825788199901581,
+            0.014503437094390392,
+            0.0266826543956995,
+            -4.62012758362107e-05,
+            0.011942208744585514,
+            0.03149929642677307,
+            -0.010657771490514278,
+            -0.02139199711382389,
+            0.0038838936015963554,
+            -0.0033487114124000072,
+            -0.045016467571258545,
+            0.00017799586930777878,
+            -0.007114100269973278,
+            -0.03090294823050499,
+            -0.013945318758487701,
+            0.0004625503206625581,
+            0.01500803790986538,
+            0.008654660545289516,
+            0.03685111552476883,
+            -0.022080088034272194,
+            -0.03180511295795441,
+            0.016774138435721397,
+            0.019281849265098572,
+            0.0334259495139122,
+            -0.008341196924448013,
+            0.005841131322085857,
+            -0.0055200220085680485,
+            0.007026177830994129,
+            0.024939490482211113,
+            0.0031308159232139587,
+            -0.0046255034394562244,
+            0.008585851639509201,
+            0.013899446465075016,
+            0.035750169306993484,
+            -0.024052616208791733,
+            -0.01591784693300724,
+            -0.03263082355260849,
+            -0.032233260571956635,
+            0.029389148578047752,
+            0.003199625061824918,
+            -0.014327592216432095,
+            0.0159637201577425,
+            0.020092269405722618,
+            -0.016590647399425507,
+            0.012890245765447617,
+            -0.014923937618732452,
+            -0.04293690249323845,
+            0.015130365267395973,
+            -0.02200363390147686,
+            0.007351109758019447,
+            -0.003146106842905283,
+            -0.004487885162234306,
+            -0.010504862293601036,
+            0.006452768109738827,
+            0.012248027138411999,
+            -0.03467980772256851,
+            0.012018662877380848,
+            0.03510795161128044,
+            -0.006406895350664854,
+            -0.01221744529902935,
+            -0.01402177382260561,
+            0.011598162353038788,
+            -0.019067777320742607,
+            0.00490456260740757,
+            0.00750401895493269,
+            0.004866335075348616,
+            -0.007018532138317823,
+            -0.003495886456221342,
+            -0.02050512284040451,
+            -0.007557536941021681,
+            0.013456009328365326,
+            -0.011491126380860806,
+            -0.00448406208306551,
+            0.001542471582069993,
+            0.007890114560723305,
+            0.0008448233129456639,
+            -0.008310615085065365,
+            0.013517173007130623,
+            0.019862905144691467,
+            0.016544776037335396,
+            0.0024580154567956924,
+            0.01474809180945158,
+            -0.027141382917761803,
+            -0.00588700408115983,
+            0.03168278560042381,
+            -0.0051989126950502396,
+            0.005764676723629236,
+            -0.011605808511376381,
+            -0.02752365544438362,
+            -0.003889627754688263,
+            -0.0031480181496590376,
+            0.029557347297668457,
+            0.0014134544180706143,
+            0.019740577787160873,
+            0.025015944615006447,
+            -0.0036832003388553858,
+            0.012553845532238483,
+            0.01121588982641697,
+            -0.037340424954891205,
+            0.034159913659095764,
+            -0.00941156130284071,
+            0.014151746407151222,
+            0.03431282564997673,
+            -0.015650255605578423,
+            -0.011743426322937012,
+            -0.002968349726870656,
+            0.03061242215335369,
+            0.04914501681923866,
+            0.013662436977028847,
+            0.005523844622075558,
+            0.04357912018895149,
+            -0.018716085702180862,
+            -0.04461890459060669,
+            0.03056654892861843,
+            0.020398087799549103,
+            -0.007645459845662117,
+            -0.00011969923070864752,
+            0.02659090980887413,
+            -0.017676303163170815,
+            0.003742452710866928,
+            0.04119373857975006,
+            -0.0024121426977217197,
+            0.010535444132983685,
+            -0.01909835822880268,
+            -0.024419598281383514,
+            0.04452715814113617,
+            -0.027263710275292397,
+            0.032508496195077896,
+            -0.015000392682850361,
+            0.007851887494325638,
+            0.04657614231109619,
+            -0.004671376198530197,
+            0.023471562191843987,
+            0.03113231249153614,
+            -0.03287547826766968,
+            0.0006670663715340197,
+            0.018899576738476753,
+            -0.011491126380860806,
+            -0.003715693484991789,
+            0.003880070988088846,
+            0.021667232736945152,
+            0.012064536102116108,
+            0.0036774661857634783,
+            0.021407287567853928,
+            0.0028632248286157846,
+            -0.0029836406465619802,
+            -0.014128809794783592,
+            0.018012704327702522,
+            -0.003092588623985648,
+            -0.015237401239573956,
+            0.014641055837273598,
+            0.00230510625988245,
+            0.022966962307691574,
+            -0.03437398746609688,
+            -0.0035723410546779633,
+            0.0010627189185470343,
+            0.0034404569305479527,
+            -0.044037848711013794,
+            -0.005202735308557749,
+            -0.015176237560808659,
+            -0.023762090131640434,
+            -0.013004926964640617,
+            0.006922964006662369,
+            0.016911758109927177,
+            -0.023471562191843987,
+            -0.007897759787738323,
+            0.02906803786754608,
+            -0.03376235067844391,
+            -0.004052093718200922,
+            0.00697648199275136,
+            0.005879358854144812,
+            -0.019419468939304352,
+            -0.032233260571956635,
+            0.020489832386374474,
+            0.05275367200374603,
+            0.0014975544763728976,
+            -0.0008338329498656094,
+            0.029832584783434868,
+            -0.012377999722957611,
+            -0.027370747178792953,
+            0.029939621686935425,
+            -0.01775275729596615,
+            0.022768178954720497,
+            0.03159103915095329,
+            -0.0449553057551384,
+            -0.013081382028758526,
+            -0.003134638536721468,
+            -0.03052067570388317,
+            0.005351821891963482,
+            -0.014847483485937119,
+            -0.0031289043836295605,
+            -0.01062718965113163,
+            -0.02438901737332344,
+            0.02536763623356819,
+            -0.009289233945310116,
+            0.01784450374543667,
+            -0.021239086985588074,
+            -8.111355418805033e-05,
+            -0.019327722489833832,
+            -0.022401196882128716,
+            0.010092006996273994,
+            -0.011414671316742897,
+            0.01159051712602377,
+            0.012095117941498756,
+            -0.024465471506118774,
+            -0.021009724587202072,
+            0.012270962819457054,
+            0.008708178997039795,
+            0.013066090643405914,
+            0.003167131682857871,
+            0.01639186590909958,
+            -0.010917716659605503,
+            -0.0099696796387434,
+            0.03174395114183426,
+            0.00028455446590669453,
+            0.022905798628926277,
+            -0.017446938902139664,
+            -0.030979404225945473,
+            0.0025574064347893,
+            0.01077245268970728,
+            -0.03847195580601692,
+            -0.004040625412017107,
+            -0.02986316569149494,
+            0.00022434646962210536,
+            -0.034343406558036804,
+            0.007461968809366226,
+            0.015390310436487198,
+            -0.007182909641414881,
+            -0.0007105499389581382,
+            -0.00035431928699836135,
+            0.02266114391386509,
+            0.021132051944732666,
+            0.000617370882537216,
+            0.014541665092110634,
+            -0.005787613335996866,
+            -0.01156758051365614,
+            -0.015673192217946053,
+            0.0003980417677666992,
+            0.021927179768681526,
+            -0.003945057280361652,
+            -0.04709603264927864,
+            -0.016682393848896027,
+            -0.004958080593496561,
+            -0.01769159361720085,
+            0.01261500921100378,
+            0.007599587086588144,
+            0.01102475356310606,
+            0.0032340295147150755,
+            0.009335106238722801,
+            -0.06055204197764397,
+            -0.030000785365700722,
+            0.02204950712621212,
+            -0.018135031685233116,
+            0.023563306778669357,
+            0.010420762002468109,
+            0.023089289665222168,
+            -0.01243151817470789,
+            0.00023462004901375622,
+            0.023379815742373466,
+            -0.005107167176902294,
+            0.018746668472886086,
+            0.0024866857565939426,
+            0.02434314414858818,
+            -0.0259333997964859,
+            0.030321894213557243,
+            -0.009862643666565418,
+            0.019954649731516838,
+            -0.02097914181649685,
+            -0.038838937878608704,
+            -0.033548276871442795,
+            0.018761958926916122,
+            0.00335253425873816,
+            -0.02697318233549595,
+            -0.0017575001111254096,
+            0.0068962047807872295,
+            0.02547467313706875,
+            -0.021560197696089745,
+            0.004575807601213455,
+            -0.0033104841131716967,
+            0.005833486095070839,
+            0.015887266024947166,
+            -0.015948429703712463,
+            -0.01966412365436554,
+            -0.01680472120642662,
+            0.050918761640787125,
+            0.009006352163851261,
+            0.02565816417336464,
+            -0.011468189768493176,
+            0.007091164123266935,
+            0.008868733420968056,
+            -0.020749777555465698,
+            0.005867890547960997,
+            0.02692730911076069,
+            -0.04807465150952339,
+            0.008081250824034214,
+            0.032569658011198044,
+            0.04663730412721634,
+            -0.02536763623356819,
+            0.02458779886364937,
+            0.0029702612664550543,
+            -0.003264611354097724,
+            0.031300511211156845,
+            0.02541350945830345,
+            0.030643003061413765,
+            -0.0167129747569561,
+            -0.009625634178519249,
+            -0.01957237720489502,
+            0.01975586824119091,
+            0.01998523250222206,
+            0.003597188973799348,
+            -0.044588323682546616,
+            0.022034214809536934,
+            -0.010611898265779018,
+            0.013471300713717937,
+            0.01848672144114971,
+            -0.005340353585779667,
+            0.032416749745607376,
+            0.013731245882809162,
+            -0.028762219473719597,
+            0.007549891714006662,
+            0.001983996946364641,
+            -0.0006933476543053985,
+            0.019817031919956207,
+            0.040184538811445236,
+            -0.005653817672282457,
+            -0.02163665182888508,
+            -0.010092006996273994,
+            0.014511083252727985,
+            -0.0045910985209047794,
+            0.022309452295303345,
+            -0.0043540894985198975,
+            0.019725287333130836,
+            0.004334975965321064,
+            -0.006598032079637051,
+            0.009526243433356285,
+            0.03498562425374985,
+            -0.006857977714389563,
+            0.015115073882043362,
+            -0.014251137152314186,
+            0.01821148581802845,
+            0.022814052179455757,
+            -0.02607101760804653,
+            -0.026835564523935318,
+            -0.011903980746865273,
+            -0.024450181052088737,
+            0.01058131642639637,
+            -0.003994752652943134,
+            -0.04143839329481125,
+            -0.03608657047152519,
+            -0.0056767538189888,
+            -0.018517304211854935,
+            -0.03718751668930054,
+            -0.021835433319211006,
+            -0.027508364990353584,
+            0.0008424341212958097,
+            0.004162952769547701,
+            0.011101207695901394,
+            -0.016177793964743614,
+            -0.015459120273590088,
+            0.009656216017901897,
+            0.031101731583476067,
+            -0.027508364990353584,
+            0.043884940445423126,
+            0.012492681853473186,
+            -0.02139199711382389,
+            0.009228070266544819,
+            0.033915258944034576,
+            0.0002752365544438362,
+            -0.0244348905980587,
+            0.019878195598721504,
+            0.0028288201428949833,
+            0.017294030636548996,
+            0.022645851597189903,
+            -0.011277053505182266,
+            0.0030008431058377028,
+            0.004526112228631973,
+            -0.005290658213198185,
+            -0.04269224777817726,
+            -0.0023203971795737743,
+            0.003195802215486765,
+            -0.025015944615006447,
+            0.0007052936707623303,
+            0.005164508242160082,
+            0.026804981753230095,
+            -0.0018005057936534286,
+            0.030031366273760796,
+            0.05813607573509216,
+            -0.020703906193375587,
+            -0.04452715814113617,
+            0.005088053643703461,
+            -0.010137880221009254,
+            0.0021942469757050276,
+            -0.029465602710843086,
+            0.03801322728395462,
+            -0.013868864625692368,
+            0.0033926728647202253,
+            -0.005003953352570534,
+            -0.0036239479668438435,
+            -0.029649093747138977,
+            0.021223796531558037,
+            0.0019161433447152376,
+            -0.006575095467269421,
+            -0.036117151379585266,
+            -0.0338846780359745,
+            0.02472541667521,
+            0.019511213526129723,
+            -0.022814052179455757,
+            -0.021284960210323334,
+            -0.0010426496155560017,
+            -0.011804590001702309,
+            0.026330964639782906,
+            -0.01834910362958908,
+            -0.01675884798169136,
+            0.0116134537383914,
+            0.023578599095344543,
+            0.018104448914527893,
+            -0.015451474115252495,
+            -0.06046029552817345,
+            -0.02191188745200634,
+            0.02125437930226326,
+            -0.00647188164293766,
+            -0.003503531916067004,
+            -0.016774138435721397,
+            0.007286123465746641,
+            -0.019220685586333275,
+            -0.025826362892985344,
+            0.038563698530197144,
+            -0.004239407368004322,
+            0.0227070152759552,
+            0.0004496486217249185,
+            0.0018645365489646792,
+            0.04786057770252228,
+            0.008784633129835129,
+            -0.020612159743905067,
+            -0.010619543492794037,
+            0.01376182772219181,
+            0.041744209825992584,
+            -0.01966412365436554,
+            0.005634704139083624,
+            0.01296669989824295,
+            -0.01102475356310606,
+            -0.013639500364661217,
+            0.0007310971268452704,
+            0.0099696796387434,
+            -0.002119703684002161,
+            0.004755476024001837,
+            -0.020765069872140884,
+            0.015115073882043362,
+            0.02088739722967148,
+            0.012118053622543812,
+            -0.003795970929786563,
+            -0.0037692117039114237,
+            -0.00027021922869607806,
+            0.01418232824653387,
+            0.016269538551568985,
+            0.02411377988755703,
+            0.024786580353975296,
+            -0.001051250728778541,
+            -0.01896074041724205,
+            -0.004182066768407822,
+            -0.003694668412208557,
+            0.001217539538629353,
+            0.02992432937026024,
+            0.032936640083789825,
+            -0.008738760836422443,
+            0.006357199978083372,
+            -0.018792539834976196,
+            0.012844372540712357,
+            0.021973051130771637,
+            -0.018884286284446716,
+            -0.018196195363998413,
+            0.025168854743242264,
+            -0.003528379835188389,
+            0.01441933773458004,
+            0.002433167537674308,
+            0.006364845205098391,
+            0.0206886138767004,
+            0.06813634186983109,
+            -0.015979010611772537,
+            -0.008050668984651566,
+            -0.022018924355506897,
+            0.0031136134639382362,
+            0.025444090366363525,
+            -0.026147473603487015,
+            0.013119609095156193,
+            0.00559265399351716,
+            0.02350214309990406,
+            0.016605939716100693,
+            0.004923676140606403,
+            0.0023280426394194365,
+            0.01214863546192646,
+            0.0025153562892228365,
+            0.016254248097538948,
+            0.0004979106015525758,
+            -0.00819593295454979,
+            0.0035207343753427267,
+            0.01591784693300724,
+            0.015160947106778622,
+            0.027982383966445923,
+            -0.0007554669864475727,
+            0.025734618306159973,
+            -0.03076533041894436,
+            0.022064797580242157,
+            -0.007599587086588144,
+            0.0020031104795634747,
+            -0.01802799478173256,
+            -0.008937542326748371,
+            0.011552290059626102,
+            -0.001057940535247326,
+            -0.0001701114815659821,
+            0.0025669632013887167,
+            -0.007010886911302805,
+            0.03090294823050499,
+            3.2761989132268354e-05,
+            0.011445253156125546,
+            0.008478814736008644,
+            -0.007419918663799763,
+            0.004308216739445925,
+            -0.001004422316327691,
+            0.01773746684193611,
+            -0.029832584783434868,
+            -0.010825971141457558,
+            0.012936118058860302,
+            0.016361285001039505,
+            -0.009059869684278965,
+            0.005130103789269924,
+            -0.023425688967108727,
+            0.0007463880465365946,
+            0.012951409444212914,
+            -0.010321371257305145,
+            0.006773877423256636,
+            -0.024037325754761696,
+            -0.011277053505182266,
+            0.005703513044863939,
+            -0.034526895731687546,
+            -0.006678309291601181,
+            0.008761697448790073,
+            -0.003845666302368045,
+            0.020749777555465698,
+            0.0036832003388553858,
+            0.01116237137466669,
+            -0.022783471271395683,
+            -0.011934562586247921,
+            -0.0022745244204998016,
+            0.0032951931934803724,
+            0.0010808769147843122,
+            0.006987950298935175,
+            -0.01605546660721302,
+            0.028961002826690674,
+            -0.024924200028181076,
+            -0.015252692624926567,
+            -0.004101789090782404,
+            -0.0024637493770569563,
+            0.0008410005830228329,
+            -0.01463341061025858,
+            0.004380848724395037,
+            -0.007710446137934923,
+            0.034007005393505096,
+            0.035322025418281555,
+            -0.005003953352570534,
+            -0.01077245268970728,
+            -0.03217209503054619,
+            0.015619674697518349,
+            -0.018563177436590195,
+            -0.0034939751494675875,
+            -0.0050498261116445065,
+            -0.006815927568823099,
+            0.0005284924409352243,
+            -0.02533705346286297,
+            0.0113535076379776,
+            0.002578431274741888,
+            0.0018893842352554202,
+            0.0011420405935496092,
+            0.011147080920636654,
+            0.015092138200998306,
+            -0.02003110572695732,
+            -0.040184538811445236,
+            -0.024878326803445816,
+            -0.002654885873198509,
+            0.009801479987800121,
+            -0.016254248097538948,
+            -0.008310615085065365,
+            -0.004820462316274643,
+            0.0116134537383914,
+            -0.006716536357998848,
+            0.006942077539861202,
+            0.003516911529004574,
+            0.0025841654278337955,
+            -0.010092006996273994,
+            -0.019358305260539055,
+            -0.010474280454218388,
+            0.01568848453462124,
+            0.020306341350078583,
+            0.007492550648748875,
+            -0.04672905057668686,
+            -0.0023643583990633488,
+            0.002708404092118144,
+            0.007473437115550041,
+            -0.006215759087353945,
+            0.03529144451022148,
+            0.0007846153457649052,
+            2.241374022560194e-05,
+            0.03737100958824158,
+            0.011062980629503727,
+            -0.016743557527661324,
+            0.012309190817177296,
+            -0.015459120273590088,
+            -0.002947324886918068,
+            0.00021455071691889316,
+            0.03168278560042381,
+            0.005290658213198185,
+            0.028074128553271294,
+            -0.0264532919973135,
+            -8.410005830228329e-05,
+            0.003337243339046836,
+            0.02827291004359722,
+            -0.015841392800211906,
+            -0.023762090131640434,
+            0.024006744846701622,
+            0.03929766267538071,
+            -0.007905405946075916,
+            -0.010634834878146648,
+            -0.03330362215638161,
+            -0.025918109342455864,
+            0.047829996794462204,
+            0.025872236117720604,
+            0.004598744213581085,
+            0.02102501504123211,
+            -0.021804852411150932,
+            0.015023328363895416,
+            0.015099783428013325,
+            0.0008868733420968056,
+            -0.030918240547180176,
+            0.03932824730873108,
+            -0.013478945940732956,
+            0.03526085987687111,
+            0.002372003858909011,
+            -0.003767300397157669,
+            0.020948560908436775,
+            0.023884417489171028,
+            -0.029893748462200165,
+            0.009579760953783989,
+            -0.0045643397606909275,
+            -0.013402490876615047,
+            0.015436183661222458,
+            0.007798369042575359,
+            0.02331865206360817,
+            -0.01479396503418684,
+            0.00012184951629023999,
+            -0.0013341327430680394,
+            0.03819671645760536,
+            0.008509396575391293,
+            -0.02350214309990406,
+            0.03263082355260849,
+            -0.012729690410196781,
+            0.00769133260473609,
+            -0.015046264976263046,
+            0.027049636468291283,
+            -0.011292343959212303,
+            -0.004055916331708431,
+            0.008348842151463032,
+            0.01568848453462124,
+            0.01784450374543667,
+            -0.015076846815645695,
+            -0.008318260312080383,
+            -0.03905300796031952,
+            0.01582610234618187,
+            0.012225090526044369,
+            -0.0003877681738231331,
+            0.02448076196014881,
+            -0.0072173140943050385,
+            -0.007851887494325638,
+            -0.0036832003388553858,
+            -0.007324350532144308,
+            -0.012653236277401447,
+            -0.003279902273789048,
+            -0.03510795161128044,
+            -0.03305896744132042,
+            0.011429962702095509,
+            -0.021697815507650375,
+            -0.01587197557091713,
+            0.011651680804789066,
+            -0.02672852762043476,
+            0.02214125171303749,
+            -0.007733382750302553,
+            0.03730984404683113,
+            -0.02289050631225109,
+            0.012683818116784096,
+            0.010374888777732849,
+            -0.004059739410877228,
+            -0.001281570177525282,
+            -0.012706754729151726,
+            -0.004006220959126949,
+            0.0061660632491111755,
+            0.01044369861483574,
+            0.02831878326833248,
+            0.03211093321442604,
+            -0.03458806127309799,
+            0.026346255093812943,
+            -0.007913051173090935,
+            0.010543089359998703,
+            0.00279632699675858,
+            0.001090433681383729,
+            0.03128522261977196,
+            -0.03379293158650398,
+            0.009166906587779522,
+            0.015497347339987755,
+            -0.0015682749217376113,
+            0.022584687918424606,
+            0.008043023757636547,
+            0.0003877681738231331,
+            -0.013853573240339756,
+            0.005068940110504627,
+            0.021269669756293297,
+            -0.0066591957584023476,
+            0.01503097452223301,
+            -0.01439640112221241,
+            0.026881437748670578,
+            0.03507737070322037,
+            -0.009296879172325134,
+            0.02266114391386509,
+            0.008486460894346237,
+            -0.0002391595480730757,
+            0.013249581679701805,
+            -0.007297591306269169,
+            -0.0349244624376297,
+            -0.001130572403781116,
+            -0.02715667337179184,
+            -0.010267852805554867,
+            -0.0009332239278592169,
+            -0.00590994069352746,
+            -0.023578599095344543,
+            -0.0033926728647202253,
+            0.016820011660456657,
+            -0.0007244073203764856,
+            -0.0059787495993077755,
+            -0.011536998674273491,
+            0.020076977089047432,
+            0.015749648213386536,
+            0.025061817839741707,
+            0.020245177671313286,
+            0.0020413377787917852,
+            0.014725156128406525,
+            0.0014421248342841864,
+            0.02438901737332344,
+            -0.006116367876529694,
+            -0.0017288295784965158,
+            -0.009059869684278965,
+            0.00859349686652422,
+            0.010711289010941982,
+            0.008501751348376274,
+            0.011582871899008751,
+            -0.021514324471354485,
+            -0.010902426205575466,
+            -0.012423872016370296,
+            0.004472594242542982,
+            -0.00540534034371376,
+            0.02036750502884388,
+            0.03529144451022148,
+            0.0011860019294545054,
+            -0.005305949132889509,
+            -0.016131920740008354,
+            -0.004671376198530197,
+            0.009747961536049843,
+            -0.032508496195077896,
+            0.0031919796019792557,
+            -0.015933139249682426,
+            0.0009059869917109609,
+            -0.020994432270526886,
+            0.0007736249826848507,
+            0.009579760953783989,
+            0.0030103998724371195,
+            0.016361285001039505,
+            -0.020443959161639214,
+            0.022447070106863976,
+            -0.0005710202967748046,
+            -0.008203578181564808,
+            0.013593628071248531,
+            -0.02093326859176159,
+            0.009304524399340153,
+            0.022737598046660423,
+            0.024985363706946373,
+            -0.016590647399425507,
+            -0.0008175863767974079,
+            0.00418971199542284,
+            0.006326618138700724,
+            0.00769133260473609,
+            0.0038265527691692114,
+            0.013310745358467102,
+            -0.016590647399425507,
+            0.013196064159274101,
+            0.013540109619498253,
+            0.004690489731729031,
+            5.689297177013941e-05,
+            -0.004786057863384485,
+            0.01854788511991501,
+            0.03776857256889343,
+            -0.02827291004359722,
+            -0.017095249146223068,
+            -0.0012796588707715273,
+            -0.009587407112121582,
+            0.000816152838524431,
+            0.015856683254241943,
+            -0.029847875237464905,
+            0.00437702564522624,
+            0.003228295361623168,
+            0.004445835016667843,
+            -0.021988343447446823,
+            0.017339903861284256,
+            0.008371778763830662,
+            -0.008020087145268917,
+            0.03137696906924248,
+            0.023288071155548096,
+            -0.0016514193266630173,
+            0.04088791832327843,
+            0.002578431274741888,
+            0.007087341509759426,
+            -0.026942601427435875,
+            0.031346384435892105,
+            -0.010971235111355782,
+            0.005864067934453487,
+            0.005367112811654806,
+            -0.02631567232310772,
+            -0.007068227510899305,
+            -0.010237270966172218,
+            0.004078852944076061,
+            -0.007595764473080635,
+            0.011842817068099976,
+            0.02552054449915886,
+            0.024465471506118774,
+            0.002127349143847823,
+            -0.019358305260539055,
+            0.0224776528775692,
+            0.018043285235762596,
+            0.013272518292069435,
+            -0.0011640213197097182,
+            -0.004350266885012388,
+            -0.017401067540049553,
+            -0.0032512317411601543,
+            -0.020719196647405624,
+            -0.01680472120642662,
+            -0.012278608977794647,
+            0.02279876172542572,
+            -0.010642480105161667,
+            -0.0004214559739921242,
+            -0.005183621775358915,
+            -0.02579578198492527,
+            -0.004629326052963734,
+            0.007660750765353441,
+            -0.00488162599503994,
+            0.010451343841850758,
+            0.026621490716934204,
+            0.010359598323702812,
+            -0.03269198536872864,
+            0.0068962047807872295,
+            -0.011017107404768467,
+            -0.007351109758019447,
+            0.0008854398038238287,
+            0.0025593177415430546,
+            0.01923597790300846,
+            -0.014663992449641228,
+            -0.022538816556334496,
+            0.020872104912996292,
+            0.005955813452601433,
+            -0.016590647399425507,
+            0.016376575455069542,
+            0.020765069872140884,
+            -0.008845796808600426,
+            0.010986525565385818,
+            -0.003868602681905031,
+            0.0204592514783144,
+            0.02761540189385414,
+            -0.024710126221179962,
+            9.509040683042258e-05,
+            -0.00469431234523654,
+            -0.005095698870718479,
+            0.0060016862116754055,
+            0.010734225623309612,
+            -0.008073605597019196,
+            -0.027447201311588287,
+            -0.0002871826000045985,
+            0.006991772912442684,
+            0.02561229094862938,
+            0.015275629237294197,
+            -0.03229442238807678,
+            -0.008731115609407425,
+            -0.016743557527661324,
+            0.03590308129787445,
+            -0.0404597744345665,
+            0.0001355874555883929,
+            0.0043311528861522675,
+            -0.010038488544523716,
+            -0.018471430987119675,
+            0.012316836044192314,
+            0.026483872905373573,
+            -0.008501751348376274,
+            -0.006555981934070587,
+            0.003174777142703533,
+            -0.014855128712952137,
+            0.027233127504587173,
+            0.033640023320913315,
+            -0.018471430987119675,
+            0.01775275729596615,
+            0.0037118708714842796,
+            0.006953545846045017,
+            -0.003455747850239277,
+            -0.040796175599098206,
+            0.011483481153845787,
+            0.028991583734750748,
+            0.0013828725786879659,
+            -0.004919853527098894,
+            0.00777160981670022,
+            0.009105742909014225,
+            -0.0037367185577750206,
+            0.010023198090493679,
+            -0.015933139249682426,
+            -0.0035666071344166994,
+            -0.003482507076114416,
+            0.01177400816231966,
+            -0.008417651057243347,
+            0.005936699453741312,
+            -0.008272388018667698,
+            -0.003377381945028901,
+            0.007202023174613714,
+            -0.00863937009125948,
+            -0.00014406914124265313,
+            -0.0025593177415430546,
+            0.0030428930185735226,
+            0.013563046231865883,
+            -0.009725024923682213,
+            -0.011644035577774048,
+            -0.03529144451022148,
+            0.016086047515273094,
+            0.028028255328536034,
+            0.0014908646699041128,
+            -0.030168984085321426,
+            0.006716536357998848,
+            -0.014128809794783592,
+            -0.01018375251442194,
+            -0.0060628498904407024,
+            0.020902687683701515,
+            0.03208034858107567,
+            0.020291050896048546,
+            0.01782921329140663,
+            0.007890114560723305,
+            0.019251268357038498,
+            -0.0029951089527457952,
+            0.008226514793932438,
+            -0.0038552230689674616,
+            -0.018716085702180862,
+            0.0015740090748295188,
+            0.017523394897580147,
+            -0.010351953096687794,
+            0.007056759670376778,
+            0.000946125655900687,
+            0.012561490759253502,
+            0.002589899580925703,
+            -0.0024064085446298122,
+            0.005053649190813303,
+            0.015489702112972736,
+            -0.0021827789023518562,
+            0.008853442966938019,
+            0.07370223104953766,
+            -0.01219450868666172,
+            0.010833616368472576,
+            0.008685242384672165,
+            -0.00578379025682807,
+            -0.017309321090579033,
+            -0.005535312928259373,
+            -0.022492943331599236,
+            0.027967091649770737,
+            0.00353984790854156,
+            -0.02116263285279274,
+            -0.004514643922448158,
+            0.00234715617261827,
+            0.016957629472017288,
+            0.015657901763916016,
+            -0.0027905928436666727,
+            -0.0026701767928898335,
+            -0.0023490674793720245,
+            0.017217576503753662,
+            0.0076263463124632835,
+            -0.025597000494599342,
+            -0.00450699869543314,
+            -0.0018196194432675838,
+            0.012324481271207333,
+            0.002180867362767458,
+            -0.004686667118221521,
+            0.0016428182134404778,
+            0.024603089317679405,
+            -0.005455035716295242,
+            0.029572639614343643,
+            -0.014243491925299168,
+            -0.017538685351610184,
+            0.014381109736859798,
+            -0.0012557668378576636,
+            0.021758979186415672,
+            0.00488162599503994,
+            0.018287939950823784,
+            -0.013654791750013828,
+            -0.005581185687333345,
+            0.01322664599865675,
+            -0.012622654438018799,
+            -0.0026625313330441713,
+            0.007935987785458565,
+            -0.00046756764641031623,
+            -0.002867047442123294,
+            0.006269277073442936,
+            -0.014426982961595058,
+            0.005894649773836136,
+            -0.01573435589671135,
+            -0.004537580534815788,
+            -0.007806014735251665,
+            -0.008188287727534771,
+            0.01463341061025858,
+            -0.0013886066153645515,
+            -0.03403758630156517,
+            -0.0024733063764870167,
+            0.034618642181158066,
+            0.003358268178999424,
+            0.00620429078117013,
+            0.01957237720489502,
+            0.005760854110121727,
+            0.014503437094390392,
+            -0.021132051944732666,
+            -0.0002107279869960621,
+            -0.0041056121699512005,
+            0.005554426461458206,
+            -0.006995595991611481,
+            0.0109788803383708,
+            0.005462680943310261,
+            -0.00017082825070247054,
+            -0.0028402884490787983,
+            -0.006330440752208233,
+            0.0032416749745607376,
+            0.008165351115167141,
+            0.008058315142989159,
+            0.01900661364197731,
+            -0.02266114391386509,
+            -0.01081068068742752,
+            0.009725024923682213,
+            0.009839707054197788,
+            0.01315783616155386,
+            0.008585851639509201,
+            0.0199393592774868,
+            -0.003065829398110509,
+            -0.019832322373986244,
+            -0.006127836182713509,
+            -0.0028880725149065256,
+            0.032325003296136856,
+            0.018150322139263153,
+            -0.025015944615006447,
+            -0.008647015318274498,
+            0.0018100626766681671,
+            0.002362447092309594,
+            -0.004247053060680628,
+            0.008708178997039795,
+            0.007874824106693268,
+            -0.02018401399254799,
+            -0.0008080295519903302,
+            -0.007014709524810314,
+            -0.009319815784692764,
+            0.009648570790886879,
+            0.019205395132303238,
+            -0.02223299816250801,
+            0.01435052789747715,
+            0.006819750182330608,
+            -0.02561229094862938,
+            0.024755999445915222,
+            0.008440587669610977,
+            -0.0028039724566042423,
+            -0.006804459262639284,
+            0.006173708941787481,
+            -0.03608657047152519,
+            0.01811973936855793,
+            0.003975639119744301,
+            -0.0035666071344166994,
+            0.0003598144685383886,
+            0.0019954650197178125,
+            -0.001083743991330266,
+            -0.010329016484320164,
+            0.00010416939039714634,
+            0.018425557762384415,
+            -0.008394715376198292,
+            0.01079538930207491,
+            -0.0035818980541080236,
+            0.008111832663416862,
+            -0.0066477274522185326,
+            -0.006953545846045017,
+            -0.004285280127078295,
+            -0.03324246034026146,
+            -0.01542853843420744,
+            -0.012041599489748478,
+            -0.02561229094862938,
+            0.005500908475369215,
+            0.001043605268932879,
+            -0.013234291225671768,
+            0.005669108591973782,
+            0.0013236202066764235,
+            0.0018177081365138292,
+            -0.004545225761830807,
+            0.010894780047237873,
+            -0.0040826755575835705,
+            0.016865884885191917,
+            0.008884024806320667,
+            -0.013662436977028847,
+            0.020168723538517952,
+            0.0009609387489035726,
+            0.002477128989994526,
+            -0.012660881504416466,
+            0.0002587510389275849,
+            0.003929766360670328,
+            0.009923807345330715,
+            -0.01934301294386387,
+            -0.005229494534432888,
+            -0.00601697713136673,
+            -0.011942208744585514,
+            -0.0022707015741616488,
+            -0.027645982801914215,
+            -0.007905405946075916,
+            -0.010466634295880795,
+            -0.0145798921585083,
+            -0.013700664043426514,
+            0.0022879038006067276,
+            -0.0006871357327327132,
+            0.01821148581802845,
+            -0.015130365267395973,
+            0.011108852922916412,
+            0.01280614547431469,
+            -0.0007903494406491518,
+            0.005657640285789967,
+            -0.007611055392771959,
+            -0.013004926964640617,
+            0.01854788511991501,
+            -0.017079956829547882,
+            -0.008394715376198292,
+            0.0206886138767004,
+            0.012225090526044369,
+            0.002939679427072406,
+            -0.014434628188610077,
+            -0.004201180301606655,
+            0.029052747413516045,
+            0.007733382750302553,
+            0.010864198207855225,
+            0.0032856364268809557,
+            0.003398407017812133,
+            0.02756952866911888,
+            0.0013809611555188894,
+            -0.014144101180136204,
+            0.006231050007045269,
+            0.004686667118221521,
+            -0.01102475356310606,
+            -0.013142545707523823,
+            0.017156412824988365,
+            -0.006926786620169878,
+            -0.015887266024947166,
+            0.024098489433526993,
+            0.023700926452875137,
+            0.0112388264387846,
+            -0.011926917359232903,
+            -0.020902687683701515,
+            0.0003727161674760282,
+            0.01001555286347866,
+            0.011812235228717327,
+            -0.01179694477468729,
+            -0.004224116448312998,
+            0.02457250840961933,
+            0.00901399739086628,
+            -0.022355325520038605,
+            0.008723469451069832,
+            0.030780620872974396,
+            0.015604383312165737,
+            -0.04544461518526077,
+            -0.01694233901798725,
+            -0.0007138948421925306,
+            -0.019251268357038498,
+            -0.02036750502884388,
+            -0.023609180003404617,
+            -0.011942208744585514,
+            0.0009485148475505412,
+            0.005615590140223503,
+            -0.0007291857618838549,
+            -0.017951540648937225,
+            0.01556615624576807,
+            0.017951540648937225,
+            0.011697554029524326,
+            -0.007760141976177692,
+            -0.018456140533089638,
+            -0.012530908919870853,
+            -0.01830323040485382,
+            0.01078774407505989,
+            -0.018089158460497856,
+            -0.012890245765447617,
+            0.006712713744491339,
+            -0.006854154635220766,
+            0.013983545824885368,
+            0.01554321963340044,
+            0.03076533041894436,
+            0.016040174290537834,
+            -0.021376706659793854,
+            -0.019373595714569092,
+            -0.005841131322085857,
+            -0.01074187085032463,
+            -0.004170598462224007,
+            -0.0013045065570622683,
+            0.005370935425162315,
+            0.009579760953783989,
+            0.005191267468035221,
+            0.02082623355090618,
+            -0.009610342793166637,
+            0.02776831015944481,
+            -2.6236470148433e-05,
+            -0.007362578064203262,
+            0.004201180301606655,
+            -0.018930159509181976,
+            0.009266297332942486,
+            -0.008302969858050346,
+            0.01120824459940195,
+            0.012263317592442036,
+            0.007599587086588144,
+            0.036392390727996826,
+            -0.011491126380860806,
+            -0.0013226645532995462,
+            -0.028012964874505997,
+            -0.0013417782029137015,
+            -0.003987107425928116,
+            0.021804852411150932,
+            0.005997863132506609,
+            -0.035841915756464005,
+            -0.020566286519169807,
+            -0.012959054671227932,
+            0.019281849265098572,
+            0.0035742525942623615,
+            -0.0053327083587646484,
+            0.020520415157079697,
+            0.025719327852129936,
+            -0.004679021425545216,
+            -0.0018263092497363687,
+            -0.027462491765618324,
+            -0.009686797857284546,
+            0.03287547826766968,
+            -0.005088053643703461,
+            0.001989730866625905,
+            0.0010636745719239116,
+            -0.015115073882043362,
+            -0.017309321090579033,
+            -0.012263317592442036,
+            0.003694668412208557,
+            0.007190554868429899,
+            0.027095509693026543,
+            -0.004766944330185652,
+            0.022768178954720497,
+            -0.01538266520947218,
+            0.008677597157657146,
+            -0.026850854977965355,
+            -0.012538554146885872,
+            0.018976030871272087,
+            -0.0009489927324466407,
+            0.014060000889003277,
+            -0.018058575689792633,
+            -0.006747118197381496,
+            -0.006024622358381748,
+            0.007419918663799763,
+            -0.005340353585779667,
+            -0.0017030262388288975,
+            -0.007744851056486368,
+            0.01479396503418684,
+            0.0009509040974080563,
+            0.007565182633697987,
+            -0.02275288850069046,
+            0.008318260312080383,
+            0.00087206024909392,
+            0.00670506851747632,
+            0.01882312260568142,
+            -0.00034858519211411476,
+            0.03951173648238182,
+            0.0072287824004888535,
+            0.011644035577774048,
+            -0.012003372423350811,
+            0.004560516681522131,
+            -0.008440587669610977,
+            0.012561490759253502,
+            0.002387294778600335,
+            0.010168462060391903,
+            0.007534600794315338,
+            -0.0029205656610429287,
+            -0.008975770324468613,
+            -0.0013245759764686227,
+            0.0014430806040763855,
+            -0.007706623524427414,
+            0.005703513044863939,
+            0.005493262782692909,
+            0.013570691458880901,
+            -0.01181988138705492,
+            -0.012653236277401447,
+            0.019037194550037384,
+            -0.021040305495262146,
+            -0.006479527335613966,
+            -0.00015768760931678116,
+            -0.006253986153751612,
+            0.004101789090782404,
+            0.0073664006777107716,
+            0.013066090643405914,
+            -0.03608657047152519,
+            0.005493262782692909,
+            -0.0164836123585701,
+            0.006987950298935175,
+            0.0005366157274693251,
+            -0.027829473838210106,
+            -0.024266690015792847,
+            0.0002518223482184112,
+            0.009725024923682213,
+            -0.00527918990701437,
+            0.005615590140223503,
+            -0.017920957878232002,
+            -0.0040329801850020885,
+            -0.004950435366481543,
+            -0.019878195598721504,
+            0.008348842151463032,
+            0.027584819123148918,
+            -0.018242066726088524,
+            -0.010657771490514278,
+            0.008723469451069832,
+            -0.005240962840616703,
+            -0.01989348605275154,
+            -0.006322795525193214,
+            -0.012821435928344727,
+            0.0035341137554496527,
+            0.019862905144691467,
+            -0.0028441110625863075,
+            -0.0113535076379776,
+            0.017248157411813736,
+            0.0066859545186161995,
+            -0.015367374755442142,
+            0.001213716808706522,
+            -0.0012194508453831077,
+            -0.012714399956166744,
+            0.01519152894616127,
+            -0.024266690015792847,
+            0.01578022912144661,
+            -0.0065101091749966145,
+            -0.003471038769930601,
+            -0.031621623784303665,
+            -0.006506286561489105,
+            0.008624078705906868,
+            0.0025688745081424713,
+            0.01793624833226204,
+            0.012393290176987648,
+            -0.020336924120783806,
+            0.01441933773458004,
+            -0.0058029042556881905,
+            0.005286835599690676,
+            -0.006357199978083372,
+            0.042386431246995926,
+            -0.010290789417922497,
+            -0.006517754402011633,
+            -0.00033520563738420606,
+            0.0015338703524321318,
+            0.0033850274048745632,
+            0.02761540189385414,
+            -0.004487885162234306,
+            -0.004308216739445925,
+            -0.0032493204344063997,
+            0.004434366710484028,
+            0.005611767526715994,
+            -0.0006713669281452894,
+            0.015160947106778622,
+            -0.006032268051058054,
+            0.008891670033335686,
+            -0.0224164891988039,
+            -0.0030428930185735226,
+            0.0159637201577425,
+            0.0032130044419318438,
+            -0.0012060713488608599,
+            -0.007672219071537256,
+            -0.03177453204989433,
+            -0.000630272610578686,
+            -0.023073997348546982,
+            0.006269277073442936,
+            -0.010351953096687794,
+            0.0017431648448109627,
+            -0.01415939163416624,
+            0.01081068068742752,
+            0.012744981795549393,
+            0.003799793543294072,
+            0.01460282877087593,
+            -0.0219577606767416,
+            0.0159637201577425,
+            0.007408450823277235,
+            -0.005890826694667339,
+            0.011865753680467606,
+            -0.015176237560808659,
+            -0.009717379696667194,
+            0.0017173613887280226,
+            0.02761540189385414,
+            -0.0012834816006943583,
+            -0.006047558970749378,
+            -0.00348824099637568,
+            -0.013532464392483234,
+            -0.0005409162840805948,
+            0.0010703643783926964,
+            -0.010451343841850758,
+            -0.001181223546154797,
+            -0.006655372679233551,
+            0.009396269917488098,
+            0.010543089359998703,
+            0.000875882979016751,
+            0.006093431729823351,
+            -0.009931452572345734,
+            0.00037367184995673597,
+            0.015099783428013325,
+            0.038563698530197144,
+            -0.0012959054438397288,
+            0.011659326031804085,
+            -0.01952650398015976,
+            -0.004407607484608889,
+            -0.0021789560560137033,
+            -0.029878458008170128,
+            -0.010703643783926964,
+            0.00819593295454979,
+            0.014105873182415962,
+            0.006506286561489105,
+            -0.015764938667416573,
+            0.005003953352570534,
+            0.01778334006667137,
+            0.0095950523391366,
+            0.0047898804768919945,
+            0.0071790870279073715,
+            0.003386938711628318,
+            0.006234872620552778,
+            0.00641454104334116,
+            -0.025306472554802895,
+            -0.011154726147651672,
+            -0.010084361769258976,
+            -0.021177923306822777,
+            0.013769472949206829,
+            0.01718699373304844,
+            0.01573435589671135,
+            0.013524818234145641,
+            -0.010382534936070442,
+            0.03299780562520027,
+            0.023853834718465805,
+            -0.00400239834561944,
+            0.003767300397157669,
+            0.004025334492325783,
+            0.005229494534432888,
+            0.013326036743819714,
+            -0.0028517565224319696,
+            -0.00418971199542284,
+            -0.00036602639011107385,
+            0.013165482319891453,
+            -0.0014535930240526795,
+            -0.004197357688099146,
+            -0.010199043899774551,
+            0.0032493204344063997,
+            0.0005714981234632432,
+            -0.004698134958744049,
+            -0.00477458955720067,
+            -0.006383959203958511,
+            -0.0007339641451835632,
+            -0.00025230017490684986,
+            0.013570691458880901,
+            -0.005523844622075558,
+            -0.023119870573282242,
+            -0.002480951836332679,
+            -0.0017555886879563332,
+            -0.003763477550819516,
+            -0.005088053643703461,
+            -0.005944345146417618,
+            -0.0006622879882343113,
+            0.019908778369426727,
+            0.03217209503054619,
+            0.01966412365436554,
+            0.01675884798169136,
+            0.016070757061243057,
+            0.008249451406300068,
+            -0.011070625856518745,
+            0.010803034529089928,
+            0.002651063259691,
+            -0.0030543613247573376,
+            0.005030712578445673,
+            0.008669951930642128,
+            0.006024622358381748,
+            0.009740316309034824,
+            -0.01782921329140663,
+            0.005485617555677891,
+            -0.017355194315314293,
+            0.00641454104334116,
+            -0.02481716312468052,
+            -0.007851887494325638,
+            -0.006533045321702957,
+            -0.010887134820222855,
+            -0.008914606645703316,
+            0.004407607484608889,
+            0.024327853694558144,
+            -0.006655372679233551,
+            0.01479396503418684,
+            -0.02243177965283394,
+            0.013509527780115604,
+            0.007312882225960493,
+            -0.006399250123649836,
+            -0.014694574289023876,
+            -0.006540691014379263,
+            -0.004847221542149782,
+            -0.008692887611687183,
+            0.012202153913676739,
+            -0.0036640865728259087,
+            -0.016498902812600136,
+            -4.0691170397622045e-06,
+            -0.012163926847279072,
+            -0.013249581679701805,
+            -0.005795258563011885,
+            0.020153433084487915,
+            0.013165482319891453,
+            0.005951990373432636,
+            -0.020627450197935104,
+            0.004174421075731516,
+            -0.0014937317464500666,
+            0.02827291004359722,
+            -0.004950435366481543,
+            -0.0038017048500478268,
+            0.0022343855816870928,
+            0.01402177382260561,
+            -0.007882469333708286,
+            -0.0025765199679881334,
+            0.00270267017185688,
+            0.02130025066435337,
+            0.01016081590205431,
+            0.004438189323991537,
+            -0.007966569624841213,
+            0.021514324471354485,
+            -0.012852017767727375,
+            -0.00269311317242682,
+            -0.003715693484991789,
+            -0.005867890547960997,
+            -0.013647145591676235,
+            0.015573801472783089,
+            0.010145525448024273,
+            0.0027657451573759317,
+            -0.005558249540627003,
+            0.001599812530912459,
+            -0.009327461011707783,
+            -0.010688353329896927,
+            -0.018379686400294304,
+            0.008899315260350704,
+            -0.024358434602618217,
+            0.006598032079637051,
+            0.003727161791175604,
+            0.00022745243040844798,
+            -0.00588700408115983,
+            -0.003430900163948536,
+            0.0027389859315007925,
+            0.01116237137466669,
+            -0.005879358854144812,
+            -0.017003502696752548,
+            -0.002113969763740897,
+            0.02940443903207779,
+            -0.011911626905202866,
+            -0.0015376930823549628,
+            -0.003889627754688263,
+            -0.0037443640176206827,
+            0.002712226938456297,
+            -0.01591784693300724,
+            -0.0048242853954434395,
+            0.0013618475059047341,
+            -6.134287104941905e-05,
+            0.0076034097000956535,
+            0.01675884798169136,
+            -0.007056759670376778,
+            0.016743557527661324,
+            -0.009709734469652176,
+            -0.0015262248925864697,
+            0.017920957878232002,
+            -0.0009126767981797457,
+            -0.01789037697017193,
+            0.008142414502799511,
+            -0.007905405946075916,
+            0.030367767438292503,
+            -0.009189843200147152,
+            -0.027462491765618324,
+            0.025811072438955307,
+            0.02986316569149494,
+            -0.010107298381626606,
+            0.01857846789062023,
+            -0.006590386386960745,
+            0.017997412011027336,
+            0.028915129601955414,
+            -0.0016552420565858483,
+            -0.014319946058094501,
+            0.004927498754113913,
+            -0.007943633012473583,
+            -0.014702219516038895,
+            -0.0034481023903936148,
+            0.01643773913383484,
+            -0.026376836001873016,
+            0.0071446821093559265,
+            -0.025581708177924156,
+            0.0191901046782732,
+            -0.016728267073631287,
+            -0.025994563475251198,
+            -0.008371778763830662,
+            0.012576781213283539,
+            0.007305236998945475,
+            0.029985493049025536,
+            -0.0035455820616334677,
+            -0.034098751842975616,
+            -0.0011793122394010425,
+            0.015329146757721901,
+            -0.008685242384672165,
+            -0.01095594372600317,
+            0.03165220469236374,
+            0.01081832591444254,
+            -0.032325003296136856,
+            0.008188287727534771,
+            -0.026239218190312386,
+            0.000751166429836303,
+            -0.01943475939333439,
+            0.012248027138411999,
+            -0.03984813764691353,
+            -0.0024025856982916594,
+            -0.0013006838271394372,
+            -0.0009853086667135358,
+            -0.004078852944076061,
+            0.005267722066491842,
+            0.0034328114707022905,
+            0.014495791867375374,
+            -0.017676303163170815,
+            -0.027401328086853027,
+            0.011246471665799618,
+            0.008700533770024776,
+            -0.0232269074767828,
+            -0.01737048476934433,
+            0.0035474933683872223,
+            -0.019602959975600243,
+            0.026835564523935318,
+            -0.007037645671516657,
+            -0.001758455764502287,
+            -0.018379686400294304,
+            -0.0043540894985198975,
+            -0.0019314342644065619,
+            -0.01564261130988598,
+            0.014304655604064465,
+            0.008287678472697735,
+            0.01179694477468729,
+            -0.0037214276380836964,
+            -0.01341013703495264,
+            -9.332837180409115e-06,
+            0.011911626905202866,
+            0.011429962702095509,
+            -0.005504731088876724,
+            0.009610342793166637,
+            -0.007943633012473583,
+            -0.0021464629098773003,
+            -0.015390310436487198,
+            0.013777119107544422,
+            -0.0026204814203083515,
+            -0.008914606645703316,
+            -0.0019381240708753467,
+            0.012110408395528793,
+            0.008127124048769474,
+            0.003438545623794198,
+            -0.01018375251442194,
+            0.005302126519382,
+            -0.006835041102021933,
+            -0.01540560182183981,
+            -0.01156758051365614,
+            -0.019083067774772644,
+            -0.007786900736391544,
+            -0.020306341350078583,
+            0.01119295321404934,
+            0.032844897359609604,
+            0.005347999278455973,
+            -0.009228070266544819,
+            -0.025107691064476967,
+            -0.027355454862117767,
+            0.017951540648937225,
+            0.0026529745664447546,
+            -0.002366269938647747,
+            -0.02354801632463932,
+            -0.019159521907567978,
+            0.015596738085150719,
+            0.03529144451022148,
+            -0.0109788803383708,
+            -0.016177793964743614,
+            0.004090321250259876,
+            -0.013027863577008247,
+            -0.014281718991696835,
+            -0.00896812416613102,
+            -0.013035508804023266,
+            -0.0005246697110123932,
+            0.0010464723454788327,
+            -0.007572827860713005,
+            0.028731638565659523,
+            0.009877934120595455,
+            0.009342752397060394,
+            0.011200598441064358,
+            0.0031652203761041164,
+            0.005604122299700975,
+            -0.007278477773070335,
+            -0.0009117210865952075,
+            -0.007033823058009148,
+            0.023884417489171028,
+            -0.003960348200052977,
+            -0.00041811109986156225,
+            0.001845422899350524,
+            -0.004705780651420355,
+            0.012446808628737926,
+            0.00029530588653869927,
+            -0.0329672247171402,
+            0.03501620516180992,
+            -0.006395427044481039,
+            0.0016045909142121673,
+            -0.02597927302122116,
+            -0.011605808511376381,
+            0.021177923306822777,
+            0.006280745379626751,
+            0.001812929636798799,
+            -0.006762409117072821,
+            0.0037290730979293585,
+            0.009121033363044262,
+            0.020658032968640327,
+            -0.01079538930207491,
+            0.002070008311420679,
+            0.015252692624926567,
+            0.012270962819457054,
+            0.00031131357536651194,
+            0.0043540894985198975,
+            0.0057417405769228935,
+            -0.005328885745257139,
+            0.007542246021330357,
+            -0.022492943331599236,
+            -0.01142231747508049,
+            -0.011582871899008751,
+            0.009472724981606007,
+            -0.001574964728206396,
+            0.006112545263022184,
+            0.004935144446790218,
+            0.0034595706965774298,
+            -0.010535444132983685,
+            0.008899315260350704,
+            -0.020795650780200958,
+            -0.004147661849856377,
+            0.00032182608265429735,
+            0.027875347062945366,
+            -0.013004926964640617,
+            0.005856422241777182,
+            -0.009656216017901897,
+            0.009342752397060394,
+            0.020917978137731552,
+            -0.0006183265941217542,
+            0.0026778222527354956,
+            0.0006971703842282295,
+            -0.004254698287695646,
+            0.005615590140223503,
+            -0.003937412053346634,
+            -0.017263447865843773,
+            0.017477521672844887,
+            0.00015051999071147293,
+            0.004549048840999603,
+            -0.002289815340191126,
+            0.0010311814257875085,
+            0.01802799478173256,
+            0.020352214574813843,
+            -0.00835648737847805,
+            -0.002268790267407894,
+            -0.010688353329896927,
+            0.03819671645760536,
+            -0.004491707775741816,
+            0.01793624833226204,
+            0.004938967060297728,
+            -0.004533757921308279,
+            0.0009389580227434635,
+            -0.005340353585779667,
+            0.012936118058860302,
+            -0.00842529721558094,
+            -0.015634965151548386,
+            -0.010703643783926964,
+            -0.008234160020947456,
+            -0.013517173007130623,
+            0.019786451011896133,
+            -0.023853834718465805,
+            0.016957629472017288,
+            0.0017708796076476574,
+            0.009923807345330715,
+            0.010351953096687794,
+            -0.00010811158426804468,
+            0.0049848398193717,
+            0.0003978028253186494,
+            -0.018410267308354378,
+            0.026529746130108833,
+            -0.016407156363129616,
+            0.01877724938094616,
+            -0.01175107154995203,
+            -0.0026032791938632727,
+            0.013585981912910938,
+            0.006995595991611481,
+            0.026330964639782906,
+            -0.004461125936359167,
+            -0.0050115990452468395,
+            0.015657901763916016,
+            0.008494106121361256,
+            0.029694966971874237,
+            0.015795519575476646,
+            0.0059787495993077755,
+            -0.0009169773547910154,
+            0.013257227838039398,
+            -0.01666710339486599,
+            -0.015573801472783089,
+            0.008104187436401844,
+            -0.0007229737821035087,
+            -0.004545225761830807,
+            0.0022305629681795835,
+            0.0349244624376297,
+            -0.0010416939621791244,
+            0.011942208744585514,
+            0.009258652105927467,
+            -0.019205395132303238,
+            0.04177479445934296,
+            0.009977324865758419,
+            0.014969810843467712,
+            -0.0199393592774868,
+            -0.04474123194813728,
+            -0.015466765500605106,
+            -0.008876378647983074,
+            0.005795258563011885,
+            0.0005676753935404122,
+            0.0032665226608514786,
+            -0.028716348111629486,
+            -0.019067777320742607,
+            0.006857977714389563,
+            -0.01675884798169136,
+            0.015887266024947166,
+            -0.0061813541688025,
+            0.0047287167981266975,
+            0.005799081176519394,
+            0.0017326523084193468,
+            -0.0065215774811804295,
+            -0.007569005247205496,
+            0.017079956829547882,
+            0.0033468001056462526,
+            -0.006146949715912342,
+            0.021040305495262146,
+            -0.01058131642639637,
+            0.005669108591973782,
+            -0.0009160216432064772,
+            0.007358754985034466,
+            0.016453029587864876,
+            0.01301257312297821,
+            0.0028001496102660894,
+            0.004327330272644758,
+            0.014381109736859798,
+            0.02992432937026024,
+            0.0014918203232809901,
+            -0.010244916193187237,
+            -0.003870513988658786,
+            0.016223665326833725,
+            0.001654286403208971,
+            -0.019373595714569092,
+            0.014686928130686283,
+            -0.027049636468291283,
+            -0.003406052477657795,
+            -0.01811973936855793,
+            0.002033692318946123,
+            -0.0018301319796591997,
+            -0.007630168925970793,
+            2.1741776436101645e-05,
+            0.007905405946075916,
+            -0.000541871995665133,
+            0.012959054671227932,
+            -0.003409875091165304,
+            0.009916161186993122,
+            0.021651942282915115,
+            -0.008478814736008644,
+            -0.012974345125257969,
+            0.017966831102967262,
+            0.010848907753825188,
+            -0.015038619749248028,
+            0.0008300102199427783,
+            -0.000931790447793901,
+            -0.03137696906924248,
+            -0.0037462753243744373,
+            0.012706754729151726,
+            -0.0012557668378576636,
+            0.00488162599503994,
+            -0.0092204250395298,
+            -0.018058575689792633,
+            0.024220816791057587,
+            0.008784633129835129,
+            0.01456460077315569,
+            0.009044579230248928,
+            -0.010244916193187237,
+            0.0001916143373819068,
+            0.015810811892151833,
+            -0.009885579347610474,
+            -0.0052600763738155365,
+            0.01498510129749775,
+            -0.022676434367895126,
+            0.0113535076379776,
+            -0.008769342675805092,
+            0.0023242197930812836,
+            -0.002817352069541812,
+            -0.01121588982641697,
+            -0.008868733420968056,
+            -0.010925361886620522,
+            0.013333681970834732,
+            -0.023914998397231102,
+            -0.0005160685395821929,
+            0.008081250824034214,
+            0.0029205656610429287,
+            -0.035138532519340515,
+            -0.0033391546458005905,
+            0.014946874231100082,
+            0.029572639614343643,
+            -0.01782921329140663,
+            -0.01102475356310606,
+            0.01582610234618187,
+            -0.007859532721340656,
+            -0.000128658750327304,
+            0.0019199660746380687,
+            -0.006915318313986063,
+            -0.02223299816250801,
+            -0.010566025972366333,
+            -0.021651942282915115,
+            0.007867177948355675,
+            0.003247409127652645,
+            -0.008700533770024776,
+            -0.007492550648748875,
+            -0.014319946058094501,
+            -0.011689907871186733,
+            -0.024129072204232216,
+            -0.0031919796019792557,
+            -0.0010770541848614812,
+            -0.017095249146223068,
+            0.02682027406990528,
+            -0.004036802798509598,
+            0.014908647164702415,
+            -0.020336924120783806,
+            0.04064326360821724,
+            -0.015764938667416573,
+            -0.0016514193266630173,
+            -0.010107298381626606,
+            0.01278320886194706,
+            0.0053441766649484634,
+            -2.7714791940525174e-05,
+            -0.010321371257305145,
+            -0.015979010611772537,
+            0.015451474115252495,
+            0.008631723932921886,
+            0.0029090975876897573,
+            -0.008624078705906868,
+            -0.020168723538517952,
+            -0.009977324865758419,
+            -0.009273942559957504,
+            -0.031193476170301437,
+            -0.0016017238376662135,
+            -0.005787613335996866,
+            0.014648701064288616,
+            -0.008960478939116001,
+            -0.01652948372066021,
+            0.01240858156234026,
+            0.021468451246619225,
+            0.009029287844896317,
+            -0.01639186590909958,
+            0.01118530798703432,
+            -0.010864198207855225,
+            0.004808994475752115,
+            0.017248157411813736,
+            -0.004250875674188137,
+            0.009342752397060394,
+            -0.0038380208425223827,
+            -0.0023127517197281122,
+            -0.004464948549866676,
+            -0.007882469333708286,
+            -0.01217157207429409,
+            -0.007798369042575359,
+            -0.029832584783434868,
+            0.007974214851856232,
+            -0.008792279288172722,
+            0.03351769596338272,
+            0.01240858156234026,
+            0.024496054276823997,
+            -0.01119295321404934,
+            0.023899707943201065,
+            -0.0164836123585701,
+            0.023838544264435768,
+            0.005974926985800266,
+            -0.02570403553545475,
+            0.016835302114486694,
+            -0.010642480105161667,
+            -0.03501620516180992,
+            0.0005934788496233523,
+            -0.011766362935304642,
+            0.03458806127309799,
+            -0.0359642431139946,
+            0.011414671316742897,
+            0.0043540894985198975,
+            -0.020443959161639214,
+            -0.004323507659137249,
+            0.006028444971889257,
+            0.00038275084807537496,
+            0.02139199711382389,
+            -0.009640924632549286,
+            0.004961903672665358,
+            0.014327592216432095,
+            0.022156542167067528,
+            -0.012278608977794647,
+            0.008302969858050346,
+            0.023838544264435768,
+            0.021514324471354485,
+            0.010856552980840206,
+            -0.01680472120642662,
+            0.02883867546916008,
+            -0.0037080480251461267,
+            0.0008797057089395821,
+            0.004235584754496813,
+            -0.02706492878496647,
+            -0.006311327219009399,
+            -0.002654885873198509,
+            0.011621098965406418,
+            0.008845796808600426,
+            0.0015701863449066877,
+            0.013249581679701805,
+            0.01652948372066021,
+            -0.01315783616155386,
+            0.004300571046769619,
+            0.005042180884629488,
+            0.006104899570345879,
+            0.01764572225511074,
+            -0.030750039964914322,
+            0.006028444971889257,
+            -0.021361414343118668,
+            -0.005156862549483776,
+            0.0071561504155397415,
+            0.011957499198615551,
+            0.015856683254241943,
+            -0.03694286197423935,
+            0.0011821791995316744,
+            -0.003293281886726618,
+            0.013096672482788563,
+            0.020841524004936218,
+            0.01479396503418684,
+            -0.01296669989824295,
+            0.008555269800126553,
+            -0.02186601608991623,
+            -0.00314228399656713,
+            -0.020963851362466812,
+            0.002131171990185976,
+            -0.019358305260539055,
+            0.006861800327897072,
+            0.006865622941404581,
+            -0.00559647660702467,
+            -0.0013341327430680394,
+            0.01685059443116188,
+            -0.022921089082956314,
+            -0.008302969858050346,
+            0.012377999722957611,
+            -0.0036507071927189827,
+            -0.004495530389249325,
+            -0.008624078705906868,
+            0.0012376088416203856,
+            0.012997281737625599,
+            0.0069688367657363415,
+            -0.006735650356858969,
+            0.010244916193187237,
+            0.0014268339145928621,
+            0.0025057995226234198,
+            0.006318972446024418,
+            0.004063562024384737,
+            -0.01652948372066021,
+            -0.0014956430532038212,
+            -5.632553802570328e-05,
+            0.012240380980074406,
+            0.008104187436401844,
+            -0.017309321090579033,
+            -0.02111675962805748,
+            0.019281849265098572,
+            -0.028578728437423706,
+            -0.003803616389632225,
+            -0.0008457790245302022,
+            -0.0021445516031235456,
+            0.014518728479743004,
+            0.012821435928344727,
+            -0.01058131642639637,
+            -0.0017498546512797475,
+            -0.01825735904276371,
+            0.0037080480251461267,
+            0.0009833972435444593,
+            -0.010650125332176685,
+            0.004701958037912846,
+            0.010527797974646091,
+            0.011888690292835236,
+            0.0025516722816973925,
+            -0.018242066726088524,
+            0.00994674302637577,
+            0.02152961492538452,
+            0.014411691576242447,
+            -0.004629326052963734,
+            -0.014725156128406525,
+            -0.03614773601293564,
+            -0.023012833669781685,
+            0.026193344965577126,
+            -0.009839707054197788,
+            -0.02157548815011978,
+            0.017523394897580147,
+            0.011644035577774048,
+            0.022034214809536934,
+            -0.009380979463458061,
+            -0.006796814035624266,
+            -0.0019228331511840224,
+            0.017615139484405518,
+            0.004403784871101379,
+            -0.0027389859315007925,
+            0.0017890376038849354,
+            -0.0025153562892228365,
+            0.04116315767168999,
+            0.0021005901508033276,
+            0.005336530972272158,
+            -0.0187313761562109,
+            0.008257096633315086,
+            -0.02477128989994526,
+            0.009442143142223358,
+            -0.011368799023330212,
+            0.029419729486107826,
+            0.019465340301394463,
+            0.0071676187217235565,
+            0.015657901763916016,
+            -0.00901399739086628,
+            0.03669820725917816,
+            -0.014847483485937119,
+            -0.013310745358467102,
+            -0.0024140540044754744,
+            -0.018761958926916122,
+            -0.009029287844896317,
+            -0.0037940593902021646,
+            -0.005546781234443188,
+            0.006953545846045017,
+            -0.0046255034394562244,
+            0.006563627161085606,
+            0.002496242756024003,
+            -0.017584558576345444,
+            0.011284698732197285,
+            -0.022263579070568085,
+            0.01779863052070141,
+            -0.01243151817470789,
+            -0.021269669756293297,
+            0.0014669726369902492,
+            0.01830323040485382,
+            -0.017966831102967262,
+            0.00038824600051157176,
+            -0.0021292606834322214,
+            0.02552054449915886,
+            0.008547624573111534,
+            -0.005642349366098642,
+            0.008272388018667698,
+            -0.004067384637892246,
+            -0.006173708941787481,
+            0.01905248686671257,
+            0.028945710510015488,
+            0.019419468939304352,
+            -0.007282300386577845,
+            0.014992746524512768,
+            -0.004350266885012388,
+            -0.02977142110466957,
+            0.012202153913676739,
+            -0.010803034529089928,
+            0.006586563773453236,
+            0.007301414385437965,
+            -0.009556825272738934,
+            -0.015887266024947166,
+            0.008562915027141571,
+            -0.0149545194581151,
+            0.006387781817466021,
+            0.0014077202649787068,
+            0.0035264682956039906,
+            -0.003073474857956171,
+            0.001033092732541263,
+            -0.008792279288172722,
+            0.027493074536323547,
+            0.023379815742373466,
+            0.012951409444212914,
+            -0.023807961493730545,
+            0.005435922183096409,
+            -0.02448076196014881,
+            -0.0031250817701220512,
+            -0.00686944555491209,
+            -0.0036239479668438435,
+            -0.008432942442595959,
+            -0.0071561504155397415,
+            -0.007159973029047251,
+            -0.01732461154460907,
+            -0.0010178018128499389,
+            0.013777119107544422,
+            -0.001823442173190415,
+            0.006880913861095905,
+            0.0031441953033208847,
+            -0.003029513405635953,
+            -0.0006799680995754898,
+            0.029511475935578346,
+            0.00861643347889185,
+            -0.008792279288172722,
+            -0.012018662877380848,
+            -0.01078774407505989,
+            -0.013035508804023266,
+            0.027003765106201172,
+            0.006032268051058054,
+            -0.013754182495176792,
+            0.012225090526044369,
+            -0.0071790870279073715,
+            -0.021606069058179855,
+            -0.003543670754879713,
+            0.009709734469652176,
+            -0.03403758630156517,
+            0.034007005393505096,
+            -0.004113257396966219,
+            0.0057417405769228935,
+            0.0036602639593183994,
+            0.05174447223544121,
+            -0.005890826694667339,
+            -0.007068227510899305,
+            0.017095249146223068,
+            0.014595182612538338,
+            0.011911626905202866,
+            -0.007756318897008896,
+            0.00559265399351716,
+            -0.023395108059048653,
+            0.007171441335231066,
+            -0.01764572225511074,
+            -0.012003372423350811,
+            0.02822703868150711,
+            0.0010244916193187237,
+            -0.017171703279018402,
+            -0.01117001660168171,
+            -0.015466765500605106,
+            0.007882469333708286,
+            0.029740838333964348,
+            0.022538816556334496,
+            -0.018364394083619118,
+            -0.00609725434333086,
+            -0.01458753738552332,
+            -0.016972921788692474,
+            -0.010604253038764,
+            0.00385713460855186,
+            -0.012423872016370296,
+            -0.0027313404716551304,
+            -0.006158418022096157,
+            0.0006900027510710061,
+            -0.0016848682425916195,
+            -0.000514635001309216,
+            0.06348790228366852,
+            -0.007935987785458565,
+            0.006219581700861454,
+            -0.016070757061243057,
+            0.0007707579061388969,
+            -0.0034538365434855223,
+            -0.008150060661137104,
+            0.0007822260959073901,
+            -0.01759984903037548,
+            0.0287010557949543,
+            -0.024939490482211113,
+            0.0010627189185470343,
+            -0.014992746524512768,
+            0.006877091247588396,
+            -0.029465602710843086,
+            -0.0013752271188423038,
+            -0.01943475939333439,
+            0.004598744213581085,
+            -0.0011038132943212986,
+            -0.01016081590205431,
+            0.015313856303691864,
+            -0.0311781857162714,
+            -0.014113519340753555,
+            0.007572827860713005,
+            0.010436052456498146,
+            0.015084492042660713,
+            -0.020627450197935104,
+            -0.007278477773070335,
+            -0.01877724938094616,
+            -0.023670343682169914,
+            0.008792279288172722,
+            -0.0037500981707125902,
+            0.02481716312468052,
+            -0.01923597790300846,
+            0.0037711230106651783,
+            -0.016820011660456657,
+            0.011147080920636654,
+            0.01402177382260561,
+            0.004487885162234306,
+            -0.0029129202011972666,
+            -0.010428407229483128,
+            0.023838544264435768,
+            -0.00758429616689682,
+            -0.034618642181158066,
+            0.007094986736774445,
+            0.004606389440596104,
+            -0.00779454642906785,
+            -0.011598162353038788,
+            0.0113535076379776,
+            -0.0028613132890313864,
+            0.006292213685810566,
+            -0.02050512284040451,
+            -0.0011028576409444213,
+            -0.014312300831079483,
+            0.012454453855752945,
+            0.013754182495176792,
+            0.01357833668589592,
+            3.586795719456859e-05,
+            -0.010749517008662224,
+            -0.0264532919973135,
+            -0.0004510821308940649,
+            -0.004908385220915079,
+            0.0013647145824506879,
+            0.008700533770024776,
+            0.0031652203761041164,
+            0.022599980235099792,
+            -0.016024883836507797,
+            -0.019495923072099686,
+            0.0002515834057703614,
+            -0.01853259466588497,
+            -0.02804354764521122,
+            0.005290658213198185,
+            -0.004120903089642525,
+            -0.004178243689239025,
+            0.003673643572255969,
+            -0.011108852922916412,
+            -0.010420762002468109,
+            0.007282300386577845,
+            0.0024943312164396048,
+            -0.01639186590909958,
+            0.026835564523935318,
+            -0.0076378146186470985,
+            -0.014060000889003277,
+            -0.006322795525193214,
+            0.008364133536815643,
+            -0.009725024923682213,
+            0.0057302722707390785,
+            -0.03038305789232254,
+            0.0033276863396167755,
+            -0.007159973029047251,
+            -0.012874954380095005,
+            -0.0040100435726344585,
+            0.01591784693300724,
+            0.010260207578539848,
+            -0.007030000444501638,
+            -0.004143839236348867,
+            -0.010130234062671661,
+            -0.0053212400525808334,
+            0.007446677889674902,
+            0.03052067570388317,
+            0.001286348677240312,
+            -0.010458989068865776,
+            -0.014809255488216877,
+            0.017875084653496742,
+            -0.010038488544523716,
+            -0.0287010557949543,
+            0.014281718991696835,
+            0.014732801355421543,
+            -0.03394584357738495,
+            0.0059175859205424786,
+            -0.02073448710143566,
+            -0.011666972190141678,
+            -0.01948063261806965,
+            -0.009625634178519249,
+            0.02393028885126114,
+            -0.02134612388908863,
+            0.009877934120595455,
+            0.005068940110504627,
+            0.03324246034026146,
+            0.01639186590909958,
+            -0.01938888616859913,
+            0.00936568807810545,
+            -0.0037787684705108404,
+            0.006441299803555012,
+            0.008906960487365723,
+            -0.014251137152314186,
+            -0.0002635294513311237,
+            0.016269538551568985,
+            -0.0013704487355425954,
+            0.0018177081365138292,
+            -0.003918298054486513,
+            -0.03211093321442604,
+            0.012630299665033817,
+            -0.02837994694709778,
+            -0.0037787684705108404,
+            0.010573671199381351,
+            -0.01859375834465027,
+            -0.02617805451154709,
+            -0.0024580154567956924,
+            0.012339772656559944,
+            -0.029373856261372566,
+            0.05997098609805107,
+            0.0014268339145928621,
+            0.021147342398762703,
+            -0.01456460077315569,
+            -0.008539978414773941,
+            -0.0047287167981266975,
+            0.0018224865198135376,
+            -0.001679134089499712,
+            0.00024632716667838395,
+            0.014686928130686283,
+            -0.008998706005513668,
+            0.017951540648937225,
+            0.010046134702861309,
+            -0.001892251311801374,
+            0.02477128989994526,
+            0.00678916834294796,
+            0.00739698251709342,
+            0.005034535191953182,
+            0.021055595949292183,
+            -0.0023452448658645153,
+            0.016820011660456657,
+            0.009327461011707783,
+            -0.0109788803383708,
+            0.001719272811897099,
+            -0.008792279288172722,
+            0.014281718991696835,
+            0.011040044017136097,
+            0.002257321961224079,
+            -0.008211224339902401,
+            -0.0032550545874983072,
+            0.021407287567853928,
+            -0.010757162235677242,
+            -0.022034214809536934,
+            -0.010267852805554867,
+            0.0059175859205424786,
+            0.007718091830611229,
+            0.01718699373304844,
+            0.01718699373304844,
+            -0.007458146195858717,
+            -0.011888690292835236,
+            0.022492943331599236,
+            -0.012645591050386429,
+            0.014725156128406525,
+            0.0036602639593183994,
+            -0.017706885933876038,
+            0.02200363390147686,
+            -0.011040044017136097,
+            -0.008295323699712753,
+            -0.0006183265941217542,
+            2.241374022560194e-05,
+            -0.006938254926353693,
+            -0.01587197557091713,
+            -0.020658032968640327,
+            -0.006762409117072821,
+            0.009136324748396873,
+            -0.014037064276635647,
+            0.0032187385950237513,
+            -0.0060016862116754055,
+            0.009755606763064861,
+            0.013272518292069435,
+            0.012668526731431484,
+            -0.003195802215486765,
+            -0.007297591306269169
+          ],
+          "keyphrases": [
+            "Mixture-Of-Experts",
+            "Transformer language models",
+            "self-attention module",
+            "Feed-Forward Network",
+            "computational efficiency"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "317d6fdf-f473-4de1-9258-7ec217a9b4ca",
+        "properties": {
+          "page_content": "3. Deepseekmoe Architecture\n\nOn top of the generic MoE architecture outlined in Section 2, we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. As illustrated in Figure 2, our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. Both of these strategies are designed to elevate the level of expert specialization.\n\n3.1. Fine-Grained Expert Segmentation\n\nIn scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts.\n\nIn pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. The finer expert segmentation enables a more flexible and adaptable combination of activated experts. To be specific, on top of a typical MoE architecture shown in Figure 2(a), we segment each expert FFN into smaller experts by reducing the FFN intermediate hidden dimension to 1 times its original size. Since each expert becomes smaller, in response, we also increase the number of activated experts to times to keep the same computation cost, as illustrated in Figure 2(b). With the fine-grained expert segmentation, the output of an MoE layer can be expressed as:\n\n$$\\mathbf{h}{t}^{i}=\\sum{i=1}^{\\text{mN}}\\left(g_{i,t}\\text{FFN}{i}\\left(\\mathbf{u}{t}^{i}\\right)\\right)+\\mathbf{u}{t}^{i},\\tag{6}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\text{Tokr}(s_{i,t}|1\\leqslant j\\leqslant m\\text{N}),m\\text{K}),\\ 0,&\\text{otherwise},\\end{cases}$$ (7) $$s_{i,t}=\\text{Softmax}{i}\\left(\\mathbf{u}{t}^{T}\\mathbf{e}_{t}^{i}\\right),\\tag{8}$$ where the total number of expert parameters is equal to $N$ times the number of parameters in a\n\n$$(6)$$\n\n(7) (8) $\\frac{1}{2}$\n\nstandard FFN, and denotes the total number of fine-grained experts. With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to .\n\nFrom a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. As an illustrative example, we consider the case where = 16. A typical top-2 routing strategy can yield 16 2 = 120 possible combinations. By contrast, if each expert is split into 4 smaller experts, the fine-grained routing strategy can yield 64 8 = 4, 426, 165, 368 potential combinations. The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition.\n\n3.2. Shared Expert Isolation\n\nWith a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. This alleviation of redundancy will contribute to a more parameterefficient model with more specialized experts.\n\nTowards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate experts to serve as shared experts. Regardless of the router module, each token will be deterministically assigned to these shared experts. In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by , as depicted in Figure 2(c). With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows:\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{K_{s}}\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)+\\sum_{i=K_{s}+1}^{mN}\\left(g_{i,t}\\;\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},\\tag{9}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\mathrm{Topk}{s_{j,t}|K_{s}+1\\leqslant j\\leqslant mN},mK-K_{s}},\\ 0,&\\text{otherwise},\\end{cases}$$ (10) $$s_{i,t}=\\mathrm{Softmax}{i}\\left(\\mathbf{u}{t}^{l,T}e_{i}^{l}\\right).\\tag{11}$$ $\\mathbf{h}\\mathbf{M}\\mathbf{E}$ the number of closed-compact is $K_{s}$ the total number of routed currents.\n\nFinally, in DeepSeekMoE, the number of shared expert is , the total number of routed experts is − , and the number of nonzero gates is − .\n\nIt is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022). The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint.\n\n3.3. Load Balance Consideration\n\nAutomatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. Firstly, there is a risk of routing collapse (Shazeer et al., 2017), i.e., the model always selects only a few experts, preventing other experts from sufficient training.\n\nSecondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.\n\nExpert-Level Balance Loss. In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. The computation of the balance loss is as follows:\n\n$$\\begin{array}{l}{{{\\mathcal L}{\\mathrm{ExpBal}}=\\alpha{1}\\sum_{i=1}^{N^{\\prime}}f_{i}P_{i},}}\\ {{f_{i}=\\frac{N^{\\prime}}{K^{T}}\\sum_{t=1}^{T}\\mathds{1}\\,(\\mathrm{Token}\\;t\\;\\mathrm{s}\\mathrm{lects}\\;\\mathrm{Expert}\\;i),}}\\ {{P_{i}=\\frac{1}{T}\\sum_{t=1}^{T}s_{i,t},}}\\end{array}$$\n\n(12) $$\\begin{array}{l}\\small\\text{(13)}\\end{array}$$ = (14) $$\\begin{array}{l}\\small\\text{(14)}\\end{array}$$ .\n\n7 where 1 is a hyper-parameter called expert-level balance factor, ′is equal to ( − ) and ′ is equal to ( − ) for brevity. 1(·) denotes the indicator function.\n\nDevice-Level Balance Loss. In addition to the expert-level balance loss, we introduce a devicelevel balance loss. When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. Instead, our primary objective is to ensure balanced computation across the devices. If we partition all routed experts into groups {E1, E2, ..., E}, and deploy each group on a single device, the device-level balance loss is computed as follows:\n\n$$\\mathcal{L}{\\text{DevBal}}=\\alpha{2}\\sum_{i=1}^{D}f_{i}^{\\prime}P_{i}^{\\prime},$$ $$f_{i}^{\\prime}=\\frac{1}{|\\mathcal{E}{i}|}\\sum{j\\in\\mathcal{E}{i}}f{j},$$ $$P_{i}^{\\prime}=\\sum_{j\\in\\mathcal{E}{i}}P{j},$$\n\n(15) $$\\begin{array}{l}\\small\\text{(16)}\\end{array}$$ = (17) .\n\nwhere 2 is a hyper-parameter called device-level balance factor. In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices.\n\n",
+          "title": "Deepseekmoe Architecture",
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "fine-grained expert segmentation",
+            "shared expert isolation",
+            "load balance consideration"
+          ],
+          "embedding": [
+            -0.004791774787008762,
+            -0.04079068824648857,
+            -0.01656515710055828,
+            -0.01955496519804001,
+            -0.006609901320189238,
+            0.034003015607595444,
+            -0.01574093848466873,
+            0.050455041229724884,
+            -0.04541277140378952,
+            0.02537297084927559,
+            -0.010747150518000126,
+            -0.015199541114270687,
+            0.008557317778468132,
+            -0.01236326340585947,
+            -0.001814086688682437,
+            0.03117481619119644,
+            0.022674063220620155,
+            0.022867996245622635,
+            0.041534099727869034,
+            -0.032096002250909805,
+            0.02517903782427311,
+            -0.04825713112950325,
+            -0.03211216256022453,
+            0.018375203013420105,
+            -0.020395344123244286,
+            0.01574093848466873,
+            0.023368991911411285,
+            0.00403220159932971,
+            -0.06251124292612076,
+            0.021995296701788902,
+            0.029623348265886307,
+            0.015635890886187553,
+            0.006242235656827688,
+            -0.0028645601123571396,
+            -0.034003015607595444,
+            -0.022593257948756218,
+            0.029300125315785408,
+            0.025227520614862442,
+            -0.026358800008893013,
+            0.03578073903918266,
+            0.03626557067036629,
+            0.021866006776690483,
+            -0.0449279360473156,
+            0.007236145436763763,
+            -0.0028686001896858215,
+            0.004626123234629631,
+            -0.015256104990839958,
+            0.008306819945573807,
+            0.01797117479145527,
+            -0.036459505558013916,
+            -0.015344991348683834,
+            -0.02821733057498932,
+            0.012444068677723408,
+            0.003559488570317626,
+            -0.028928419575095177,
+            0.009276487864553928,
+            0.031077850610017776,
+            0.05061665549874306,
+            0.02391847036778927,
+            0.005931134335696697,
+            0.03561912849545479,
+            -0.007898751646280289,
+            -0.036136284470558167,
+            0.013381414115428925,
+            0.02915467508137226,
+            -0.021332690492272377,
+            0.01580558344721794,
+            -0.009599709883332253,
+            -0.006294759456068277,
+            0.009599709883332253,
+            -0.05798612907528877,
+            0.030916238203644753,
+            -0.034746427088975906,
+            0.000505792791955173,
+            -0.006007899530231953,
+            0.00270294863730669,
+            0.03772007301449776,
+            0.03183742240071297,
+            0.013324850238859653,
+            -0.028508231043815613,
+            -0.001757522695697844,
+            0.011183501221239567,
+            -0.014771271497011185,
+            0.0015050050569698215,
+            0.01929638721048832,
+            -0.04334414750337601,
+            -0.0023595248349010944,
+            -0.06988071650266647,
+            0.048612672835588455,
+            -0.0075997705571353436,
+            -0.029332447797060013,
+            0.003993819002062082,
+            0.004078664816915989,
+            -0.0046018813736736774,
+            -0.0031049568206071854,
+            0.059246696531772614,
+            -0.00016565156693104655,
+            -0.017599469050765038,
+            -0.03529590368270874,
+            0.0033534341491758823,
+            -0.0022443768102675676,
+            0.024177048355340958,
+            -0.029865765944123268,
+            0.009147198870778084,
+            0.028863774612545967,
+            -5.586952465819195e-05,
+            0.01226629689335823,
+            0.02770017459988594,
+            -0.006545256823301315,
+            0.09709606319665909,
+            -0.008993667550384998,
+            -0.005486703012138605,
+            -0.007801784668117762,
+            0.002919103717431426,
+            -0.014003617689013481,
+            -0.02333666943013668,
+            -0.03920689597725868,
+            -0.06483844667673111,
+            0.02320738136768341,
+            -0.044055234640836716,
+            -0.0021756920032203197,
+            0.0017140896525233984,
+            0.02535681053996086,
+            -0.022754868492484093,
+            -0.06299608200788498,
+            0.04172803461551666,
+            0.0025979015044867992,
+            0.003816046519204974,
+            0.013276367448270321,
+            0.013324850238859653,
+            0.014674304984509945,
+            -0.0171954408288002,
+            -0.006525055505335331,
+            -0.04761068522930145,
+            0.021736718714237213,
+            0.036459505558013916,
+            0.04476632550358772,
+            0.05426906794309616,
+            -0.0033837363589555025,
+            0.01847217045724392,
+            0.022932641208171844,
+            -0.035522159188985825,
+            -0.010456250049173832,
+            0.002995869144797325,
+            -0.023562924936413765,
+            0.0130581920966506,
+            0.00017979255062527955,
+            0.05426906794309616,
+            -0.011749139986932278,
+            0.026843633502721786,
+            -0.017793402075767517,
+            0.06380413472652435,
+            0.0016544954851269722,
+            0.021461978554725647,
+            0.0027574924752116203,
+            0.011143097653985023,
+            -0.022609418258070946,
+            -0.019118614494800568,
+            -0.0020393324084579945,
+            0.005167520605027676,
+            0.007381595205515623,
+            -0.03267780318856239,
+            -0.01442380715161562,
+            0.04104926809668541,
+            0.0012221853248775005,
+            0.05110148712992668,
+            0.007939154282212257,
+            0.008072483353316784,
+            -0.08242175728082657,
+            0.025227520614862442,
+            -0.006226074881851673,
+            0.022124584764242172,
+            0.022221552208065987,
+            0.040952298790216446,
+            -0.012985466979444027,
+            -0.022237712517380714,
+            0.026277994737029076,
+            0.0034786828327924013,
+            -0.022738708183169365,
+            0.0346817821264267,
+            -0.031982872635126114,
+            -0.009979496710002422,
+            -0.008250256069004536,
+            -0.02094482257962227,
+            0.03316263481974602,
+            0.02181752398610115,
+            0.02005596086382866,
+            0.0104966526851058,
+            -0.004832177422940731,
+            -0.007809865288436413,
+            0.031223300844430923,
+            0.02233467996120453,
+            0.011741059832274914,
+            0.006807875353842974,
+            0.006339202634990215,
+            -0.02702140621840954,
+            0.010456250049173832,
+            -0.008452270179986954,
+            0.0032544471323490143,
+            0.01527226623147726,
+            -0.00561599200591445,
+            -0.017550986260175705,
+            -0.0024928541388362646,
+            -0.019894348457455635,
+            0.022867996245622635,
+            -0.01039968617260456,
+            0.023433636873960495,
+            0.011312790215015411,
+            -0.004403907340019941,
+            -0.042794667184352875,
+            0.006254356820136309,
+            -0.01075523067265749,
+            -0.0016181330429390073,
+            -0.04415220394730568,
+            -0.01727624610066414,
+            -0.003593830857425928,
+            0.00556750874966383,
+            0.033615145832300186,
+            0.024500271305441856,
+            -0.0036200927570462227,
+            -0.00013534944446291775,
+            -0.007656334433704615,
+            0.008395706303417683,
+            0.0056079113855957985,
+            -0.0066947476007044315,
+            0.00873508956283331,
+            0.0026787070091813803,
+            0.01797117479145527,
+            0.018197430297732353,
+            -0.03406766057014465,
+            0.010957244783639908,
+            0.026471927762031555,
+            0.0039574564434587955,
+            0.019457997754216194,
+            0.02151046134531498,
+            -0.035522159188985825,
+            -0.013098594732582569,
+            -0.00875933188945055,
+            0.0058543686755001545,
+            -0.010302719660103321,
+            -0.020847855135798454,
+            0.0041029066778719425,
+            -0.056014470756053925,
+            -0.005272568203508854,
+            0.00509075541049242,
+            0.010165349580347538,
+            -0.008371464908123016,
+            0.05213579908013344,
+            0.00807652436196804,
+            0.007874510250985622,
+            -0.017130795866250992,
+            -0.010205752216279507,
+            0.028540553525090218,
+            -0.003670596284791827,
+            -0.03355050086975098,
+            0.03626557067036629,
+            0.01229861844331026,
+            -0.034003015607595444,
+            -0.004674606490880251,
+            -0.02656889520585537,
+            0.04421684890985489,
+            -0.020459989085793495,
+            -0.018245914950966835,
+            0.044055234640836716,
+            0.0020302417688071728,
+            -0.016985345631837845,
+            0.057598263025283813,
+            0.029122352600097656,
+            0.0038483687676489353,
+            -0.00885629840195179,
+            -0.004844298120588064,
+            -0.04082300886511803,
+            -0.015288427472114563,
+            -0.0186661034822464,
+            0.011280467733740807,
+            -0.016839895397424698,
+            -0.0046584452502429485,
+            -0.008104805834591389,
+            0.02079937234520912,
+            0.02902538701891899,
+            0.02327202446758747,
+            0.012629921548068523,
+            0.02144581824541092,
+            0.004185732454061508,
+            0.002262558089569211,
+            -0.006379605270922184,
+            -0.033744435757398605,
+            -0.02320738136768341,
+            -0.035263583064079285,
+            0.006092745345085859,
+            -0.01562781073153019,
+            -0.0018423686269670725,
+            -0.020540794357657433,
+            -0.01864994317293167,
+            0.05585286021232605,
+            0.006173551082611084,
+            -0.016161128878593445,
+            0.009737079963088036,
+            0.019393354654312134,
+            -0.031918227672576904,
+            -0.023934630677103996,
+            0.0034180786460638046,
+            0.009672435000538826,
+            0.007486642803996801,
+            -0.019619610160589218,
+            -0.0017534823855385184,
+            -0.04282699152827263,
+            0.030576854944229126,
+            -0.02176903933286667,
+            -0.010698666796088219,
+            0.0290900319814682,
+            0.008201772347092628,
+            0.006771512795239687,
+            -0.04340879246592522,
+            -0.015450038947165012,
+            0.021995296701788902,
+            0.020912500098347664,
+            0.03093240037560463,
+            -0.024710364639759064,
+            -0.022237712517380714,
+            0.016548994928598404,
+            0.007874510250985622,
+            -0.04638243839144707,
+            0.002288819756358862,
+            -0.009510824456810951,
+            -0.013147078454494476,
+            0.050455041229724884,
+            0.04243912175297737,
+            -0.03555448353290558,
+            -0.0016373243415728211,
+            -0.015094494447112083,
+            -0.014819755218923092,
+            -0.03726756200194359,
+            -0.013599589467048645,
+            -0.03694434091448784,
+            -0.014294518157839775,
+            -0.01682373508810997,
+            -0.008452270179986954,
+            -0.013648073188960552,
+            -0.018197430297732353,
+            -0.0077896639704704285,
+            0.006824036594480276,
+            -0.032047517597675323,
+            0.008137128315865993,
+            0.004391786642372608,
+            0.007894711568951607,
+            -0.0403381772339344,
+            -0.006444249767810106,
+            0.023320509120821953,
+            0.03144955635070801,
+            0.001097946660593152,
+            0.012759210541844368,
+            0.01808430254459381,
+            -0.0037130191922187805,
+            0.007979556918144226,
+            -0.052523668855428696,
+            -0.006383645813912153,
+            0.005078634712845087,
+            0.0022282155696302652,
+            0.000984818791039288,
+            0.021930651739239693,
+            -0.048160161823034286,
+            -0.038107942789793015,
+            -0.005874569993466139,
+            -0.04366736859083176,
+            0.01483591552823782,
+            0.004767532926052809,
+            0.0033695953898131847,
+            -0.005712958984076977,
+            -0.006573539227247238,
+            -0.023175058886408806,
+            0.022916480898857117,
+            0.008727009408175945,
+            -0.008670445531606674,
+            0.007894711568951607,
+            0.031013205647468567,
+            0.0419866107404232,
+            -0.02086401730775833,
+            -0.025130555033683777,
+            0.054301392287015915,
+            -0.011361273005604744,
+            0.01543387770652771,
+            0.028815291821956635,
+            0.014496532268822193,
+            0.012945064343512058,
+            0.032984863966703415,
+            0.010642102919518948,
+            0.007413917686790228,
+            -0.023255864158272743,
+            0.04308556765317917,
+            -0.01871458627283573,
+            -0.028233490884304047,
+            -0.019958993420004845,
+            0.012007718905806541,
+            0.006981607526540756,
+            -0.02600325644016266,
+            -0.029558703303337097,
+            -0.027183018624782562,
+            -0.0005520035629160702,
+            -0.008613881655037403,
+            -0.016379304230213165,
+            0.020702404901385307,
+            -0.0050059095956385136,
+            0.0006893731188029051,
+            -0.036847371608018875,
+            -0.004957426339387894,
+            -0.0010787553619593382,
+            -0.006965446285903454,
+            0.0363302156329155,
+            0.0006999789038673043,
+            0.003561508608981967,
+            -0.04024121165275574,
+            -0.022738708183169365,
+            -0.004181691911071539,
+            0.022318517789244652,
+            0.056079115718603134,
+            -0.004957426339387894,
+            0.039627086371183395,
+            -0.014528854750096798,
+            0.012427907437086105,
+            -0.025082072243094444,
+            0.01738937385380268,
+            0.0272638238966465,
+            0.00945426058024168,
+            0.006266477517783642,
+            0.015046010725200176,
+            0.014924801886081696,
+            -0.019781220704317093,
+            0.02669818513095379,
+            -0.0033271722495555878,
+            0.01675909012556076,
+            0.0015726798446848989,
+            0.01154712587594986,
+            -0.02574467845261097,
+            -0.01490056049078703,
+            -0.022544773295521736,
+            -0.024629559367895126,
+            0.054301392287015915,
+            -0.02600325644016266,
+            0.013745039701461792,
+            -0.0016757070552557707,
+            0.018504492938518524,
+            -0.03180510178208351,
+            -0.01599951647222042,
+            -0.04043514281511307,
+            -0.04473400488495827,
+            -0.04686727374792099,
+            -0.019765060395002365,
+            -0.00938961561769247,
+            -5.8962868934031576e-05,
+            -0.03995031118392944,
+            0.021607428789138794,
+            -0.05627305060625076,
+            -0.008929023519158363,
+            0.03772007301449776,
+            -0.022415485233068466,
+            0.0011908732121810317,
+            -0.021365011110901833,
+            0.009421938098967075,
+            -0.04900054261088371,
+            0.002274678787216544,
+            0.027344629168510437,
+            0.0555296391248703,
+            -0.0013706657337024808,
+            -0.022981123998761177,
+            0.004375625401735306,
+            -0.013405656442046165,
+            0.0018797412049025297,
+            0.0010565337724983692,
+            0.009114876389503479,
+            0.02815268561244011,
+            -0.017114635556936264,
+            0.053105469793081284,
+            0.012177410535514355,
+            0.030463727191090584,
+            0.026455767452716827,
+            -0.007078574039041996,
+            -0.06548488885164261,
+            -0.0006020020227879286,
+            0.021720556542277336,
+            0.010165349580347538,
+            0.0018544894410297275,
+            -0.004945305176079273,
+            0.004094826057553291,
+            0.0049049025401473045,
+            -0.0015716697089374065,
+            0.01625809445977211,
+            0.012015799060463905,
+            0.00763613311573863,
+            -0.004678646568208933,
+            0.013300608843564987,
+            0.015134897083044052,
+            -0.035457514226436615,
+            0.022221552208065987,
+            0.011805704794824123,
+            -0.048612672835588455,
+            -0.008727009408175945,
+            -0.02155894599854946,
+            0.02181752398610115,
+            -0.003492823801934719,
+            -0.016338901594281197,
+            -0.004569558892399073,
+            0.033938370645046234,
+            0.01973273791372776,
+            -0.0040645236149430275,
+            -0.014650062657892704,
+            0.013332931324839592,
+            0.04250376671552658,
+            -0.022997286170721054,
+            -0.030560694634914398,
+            0.011741059832274914,
+            -0.00397159717977047,
+            -0.0015454079257324338,
+            0.0017241904279217124,
+            -0.016613639891147614,
+            0.04470168054103851,
+            -0.009316890500485897,
+            0.035134293138980865,
+            0.004115027375519276,
+            0.014633901417255402,
+            -0.019021648913621902,
+            0.009179521352052689,
+            -0.018068142235279083,
+            0.02689211815595627,
+            -0.0157813411206007,
+            0.044119879603385925,
+            0.026972923427820206,
+            0.012646082788705826,
+            0.024694204330444336,
+            -0.035134293138980865,
+            -0.012702646665275097,
+            0.014221793040633202,
+            0.01884387619793415,
+            -0.010900680907070637,
+            -0.010213833302259445,
+            -0.014609660021960735,
+            -0.008751250803470612,
+            0.03772007301449776,
+            -0.02162358909845352,
+            -0.024112403392791748,
+            0.001966607291251421,
+            -0.0013797563733533025,
+            0.0006626062677241862,
+            -0.01588638871908188,
+            -0.016468189656734467,
+            0.017631791532039642,
+            0.050519686192274094,
+            -0.02244780771434307,
+            0.022415485233068466,
+            0.040144242346286774,
+            -0.0048038954846560955,
+            -0.020670082420110703,
+            -0.025728516280651093,
+            -0.009098715148866177,
+            0.00039493758231401443,
+            -0.019247904419898987,
+            0.016379304230213165,
+            -0.008565397933125496,
+            -0.00945426058024168,
+            -0.04612386226654053,
+            -0.0010247165337204933,
+            0.02516287751495838,
+            0.0033312125597149134,
+            0.015579327940940857,
+            -0.016985345631837845,
+            -0.005684677045792341,
+            -0.004876620601862669,
+            0.024257853627204895,
+            -0.016920700669288635,
+            0.007672495674341917,
+            0.0024605318903923035,
+            0.0003421613946557045,
+            0.0009858288103714585,
+            -0.003064553951844573,
+            0.00032120241667144,
+            0.007405837066471577,
+            0.027086051180958748,
+            -0.008355303667485714,
+            -0.023449797183275223,
+            0.007983597926795483,
+            0.02079937234520912,
+            -0.02598709426820278,
+            0.004917023237794638,
+            0.049873244017362595,
+            0.023869987577199936,
+            -0.013518784195184708,
+            0.005963456351310015,
+            -0.0010444129584357142,
+            0.017049990594387054,
+            0.0040604835376143456,
+            0.01505409087985754,
+            0.03494035825133324,
+            0.0016686365706846118,
+            -0.03193439170718193,
+            -0.00795531552284956,
+            0.0017322709318250418,
+            0.010569377802312374,
+            0.016662122681736946,
+            0.0008716908632777631,
+            -0.04573599249124527,
+            0.0013757160631939769,
+            0.005797804798930883,
+            0.010884519666433334,
+            -0.014625821262598038,
+            -0.02214074693620205,
+            0.022948801517486572,
+            0.026213349774479866,
+            0.001036332338117063,
+            0.0035170656628906727,
+            0.005914973095059395,
+            0.017066150903701782,
+            0.02307809144258499,
+            0.022092262282967567,
+            -0.014367243275046349,
+            0.019183259457349777,
+            -0.004755412228405476,
+            -0.012355182319879532,
+            -0.001959536923095584,
+            -0.020459989085793495,
+            0.021348850801587105,
+            0.044378459453582764,
+            0.02094482257962227,
+            0.018423685804009438,
+            0.017825724557042122,
+            0.010270397178828716,
+            0.026229511946439743,
+            -0.02732846885919571,
+            -0.03600699454545975,
+            0.0025938611943274736,
+            -0.0003156470484100282,
+            -0.025502260774374008,
+            -0.002603961853310466,
+            -0.018439847975969315,
+            -0.039756376296281815,
+            0.009623952209949493,
+            -0.0026160827837884426,
+            0.010472411289811134,
+            0.03442320227622986,
+            -0.01077947299927473,
+            -0.03224145248532295,
+            -0.0004121087840758264,
+            0.033421214669942856,
+            0.01797117479145527,
+            0.039497796446084976,
+            -0.01877923123538494,
+            0.0062462761998176575,
+            -0.0014656123239547014,
+            0.007991678081452847,
+            0.01372887846082449,
+            -0.007676536217331886,
+            -0.013389495201408863,
+            0.030124343931674957,
+            0.023482119664549828,
+            0.04777229577302933,
+            -0.006484652869403362,
+            -0.043473437428474426,
+            -0.012233974412083626,
+            -0.01531266886740923,
+            0.015062171965837479,
+            0.008605800569057465,
+            0.0050947959534823895,
+            0.0008646203787066042,
+            0.008880539797246456,
+            -0.007583609316498041,
+            0.006254356820136309,
+            0.007098775822669268,
+            -0.01879539154469967,
+            0.03332424536347389,
+            -0.008096725679934025,
+            0.0302374716848135,
+            -0.019942833110690117,
+            -0.02548610046505928,
+            -0.01864994317293167,
+            0.0024241693317890167,
+            0.0023150816559791565,
+            -0.026229511946439743,
+            0.03049604967236519,
+            0.03471410274505615,
+            -0.0018666103715077043,
+            0.008331061340868473,
+            -0.010432008653879166,
+            0.003490803763270378,
+            0.002375685842707753,
+            -0.006213953718543053,
+            0.007935114204883575,
+            0.0003030211664736271,
+            -0.026035577058792114,
+            0.035198938101530075,
+            -0.03542519360780716,
+            0.0018777211662381887,
+            -0.020718567073345184,
+            0.021979134529829025,
+            0.008452270179986954,
+            0.008387625217437744,
+            0.011676414869725704,
+            -0.023304346948862076,
+            -0.0030059698037803173,
+            0.009777482599020004,
+            0.015603569336235523,
+            0.008710848167538643,
+            -0.015749020501971245,
+            0.012476391158998013,
+            0.0019060031045228243,
+            0.020960982888936996,
+            0.03030211664736271,
+            0.00044241087744012475,
+            0.008250256069004536,
+            -0.01638738438487053,
+            -0.019215581938624382,
+            -0.0061533497646451,
+            0.010310799814760685,
+            0.030512209981679916,
+            -0.012169329449534416,
+            0.04192196577787399,
+            0.037849362939596176,
+            -0.003840288147330284,
+            -0.006965446285903454,
+            0.010254235938191414,
+            -0.0551740936934948,
+            0.05071362107992172,
+            -0.003490803763270378,
+            -0.0007388665690086782,
+            0.020088283345103264,
+            -0.0442814938724041,
+            -0.023175058886408806,
+            0.005098836030811071,
+            0.010827955789864063,
+            0.01732472889125347,
+            0.009842127561569214,
+            0.002264578128233552,
+            0.014504612423479557,
+            -0.03136875107884407,
+            -0.039562441408634186,
+            0.02763552963733673,
+            0.00970475748181343,
+            0.004123107995837927,
+            0.012815774418413639,
+            0.02018524892628193,
+            -0.0017716636648401618,
+            0.008840137161314487,
+            0.01309051364660263,
+            -0.0021150875836610794,
+            0.01732472889125347,
+            0.02600325644016266,
+            -0.010706747882068157,
+            0.026035577058792114,
+            0.011215822771191597,
+            0.03781704232096672,
+            -0.0050543928518891335,
+            -0.00488470122218132,
+            0.005474582314491272,
+            -0.021785201504826546,
+            0.03195055201649666,
+            0.0031392991077154875,
+            -0.03846348449587822,
+            0.01682373508810997,
+            0.022367002442479134,
+            -0.017793402075767517,
+            0.021284205839037895,
+            0.0014858137583360076,
+            0.020912500098347664,
+            0.026956763118505478,
+            0.0040039196610450745,
+            0.0020585237070918083,
+            0.028896097093820572,
+            0.01886003650724888,
+            -0.008387625217437744,
+            0.050261110067367554,
+            0.010674425400793552,
+            -0.007664415054023266,
+            0.01511873584240675,
+            0.0013605649583041668,
+            0.02624567225575447,
+            -0.047028884291648865,
+            8.831804007058963e-05,
+            -0.012710727751255035,
+            -6.256124470382929e-05,
+            0.011821865104138851,
+            -0.01173297967761755,
+            0.003684737253934145,
+            -0.033744435757398605,
+            0.023643730208277702,
+            0.005102876108139753,
+            0.022560935467481613,
+            -0.008428028784692287,
+            0.006916963029652834,
+            0.016662122681736946,
+            -0.0009782533161342144,
+            0.007042211946099997,
+            0.009777482599020004,
+            -0.006193752400577068,
+            0.017486341297626495,
+            -0.008395706303417683,
+            0.016807572916150093,
+            0.046641018241643906,
+            -0.01636314205825329,
+            -0.0038039255887269974,
+            0.039691731333732605,
+            -0.009114876389503479,
+            -0.025195199996232986,
+            0.020459989085793495,
+            -0.007632093038409948,
+            -0.0006838177214376628,
+            0.04230983555316925,
+            -0.043796658515930176,
+            -0.025825483724474907,
+            0.004016040358692408,
+            -0.023562924936413765,
+            -0.019635770469903946,
+            0.009365374222397804,
+            0.016314659267663956,
+            0.003751402022317052,
+            -0.04421684890985489,
+            0.005947295110672712,
+            -0.000348221801687032,
+            0.01258143875747919,
+            -0.00016312638763338327,
+            -0.01688838005065918,
+            -0.036394860595464706,
+            0.011757221072912216,
+            -0.004436229821294546,
+            -0.025890128687024117,
+            4.217044624965638e-05,
+            -0.010294638574123383,
+            -0.05242669954895973,
+            -0.026665862649679184,
+            0.040758367627859116,
+            0.0012009738711640239,
+            0.033485859632492065,
+            -5.463850175146945e-05,
+            0.03262931853532791,
+            -0.00822601467370987,
+            -0.02075088955461979,
+            0.01517529971897602,
+            -0.0073371524922549725,
+            0.016330819576978683,
+            0.010019899345934391,
+            0.001137339393608272,
+            -0.00023951297043822706,
+            0.03490803763270378,
+            -0.033421214669942856,
+            0.004030181560665369,
+            0.0038726103957742453,
+            -0.012112765572965145,
+            -0.045186515897512436,
+            0.010213833302259445,
+            -0.00844419002532959,
+            -0.009955255314707756,
+            0.022059939801692963,
+            -0.011821865104138851,
+            0.001617122907191515,
+            0.025276005268096924,
+            0.006484652869403362,
+            -0.006488692946732044,
+            -0.0363302156329155,
+            -0.013462220318615437,
+            0.0090098287910223,
+            -0.008508834056556225,
+            -0.0008686606306582689,
+            -0.010318880900740623,
+            -0.04176035523414612,
+            0.014884399250149727,
+            -0.025502260774374008,
+            -0.006116987206041813,
+            0.014811674132943153,
+            0.019878188148140907,
+            -0.010092624463140965,
+            -0.04205125570297241,
+            -0.007038171403110027,
+            -0.03290405869483948,
+            -0.006929083727300167,
+            0.003482723142951727,
+            -0.01044008880853653,
+            0.008904782123863697,
+            0.016112644225358963,
+            -0.0186661034822464,
+            0.010165349580347538,
+            -0.02663354016840458,
+            0.021203400567173958,
+            -0.014254115521907806,
+            0.01593487337231636,
+            0.0021130675449967384,
+            0.003232225775718689,
+            -0.021962974220514297,
+            0.025954771786928177,
+            -0.0299142487347126,
+            -0.004816016182303429,
+            -0.0315626822412014,
+            -0.02624567225575447,
+            -0.030140504240989685,
+            0.006848277989774942,
+            0.0010555237531661987,
+            -0.04091997817158699,
+            -0.005284688901156187,
+            0.015320749953389168,
+            0.0020373123697936535,
+            -0.028928419575095177,
+            -0.020136766135692596,
+            0.0009827986359596252,
+            0.007349273189902306,
+            0.0017928752349689603,
+            -0.031724296510219574,
+            -0.008678525686264038,
+            -0.032483868300914764,
+            0.020201411098241806,
+            -0.0017706536455079913,
+            0.027344629168510437,
+            0.0011262285988777876,
+            0.003840288147330284,
+            -0.02029837667942047,
+            0.002234275918453932,
+            0.0036907976027578115,
+            0.022302357479929924,
+            -0.03484339267015457,
+            0.016548994928598404,
+            0.035392872989177704,
+            0.022221552208065987,
+            -0.049356088042259216,
+            0.0016342941671609879,
+            0.01385008729994297,
+            -0.011021889746189117,
+            0.01871458627283573,
+            -0.005733160302042961,
+            0.025082072243094444,
+            -0.01277537178248167,
+            0.0005641243769787252,
+            0.005684677045792341,
+            0.006327081937342882,
+            -0.0022564975079149008,
+            0.004335222765803337,
+            -0.024435626342892647,
+            0.02320738136768341,
+            0.02687595598399639,
+            0.007551287300884724,
+            0.010634022764861584,
+            -0.0031069768592715263,
+            -0.012387504801154137,
+            0.011805704794824123,
+            -0.026488089933991432,
+            -0.0005691747646778822,
+            0.011021889746189117,
+            -0.022544773295521736,
+            -0.006815955974161625,
+            0.03959476575255394,
+            0.024952782317996025,
+            -0.05313779041171074,
+            0.017680274322628975,
+            0.018423685804009438,
+            -0.005058433394879103,
+            0.01747017912566662,
+            -0.0403381772339344,
+            -0.00844419002532959,
+            -0.0254376158118248,
+            0.0005666495417244732,
+            0.016807572916150093,
+            0.011005728505551815,
+            -0.015377313829958439,
+            0.024936622008681297,
+            0.0003035262052435428,
+            0.03201519697904587,
+            -0.002511035418137908,
+            -0.020411504432559013,
+            -0.02650425024330616,
+            -0.020605439320206642,
+            -0.00743007892742753,
+            0.009858288802206516,
+            -0.0005403877585195005,
+            -0.03161116689443588,
+            -0.03746149688959122,
+            0.007385635748505592,
+            0.00307667488232255,
+            -0.04790158569812775,
+            -0.013171319849789143,
+            -0.02845974639058113,
+            -0.03060917742550373,
+            -0.009163360111415386,
+            -0.009058312512934208,
+            -0.03775239735841751,
+            0.0006009919452480972,
+            0.003133238758891821,
+            0.021672073751688004,
+            -0.010868359357118607,
+            0.013720798306167126,
+            -0.02333666943013668,
+            -0.007838147692382336,
+            0.0016706566093489528,
+            0.03985334187746048,
+            -0.009615871123969555,
+            -0.035069648176431656,
+            0.011021889746189117,
+            0.019377192482352257,
+            -0.015902550891041756,
+            0.012217813171446323,
+            0.005082674790173769,
+            -0.006678586360067129,
+            0.017793402075767517,
+            -0.013955134898424149,
+            -0.03762310743331909,
+            0.004549357574433088,
+            -0.013147078454494476,
+            -0.01442380715161562,
+            -0.017179278656840324,
+            0.010367363691329956,
+            0.02214074693620205,
+            -0.021930651739239693,
+            0.008880539797246456,
+            0.04486329108476639,
+            -0.02227003499865532,
+            -0.029364770278334618,
+            0.01053705532103777,
+            -0.018375203013420105,
+            0.01401169877499342,
+            -0.004214014392346144,
+            0.03062533773481846,
+            -0.009163360111415386,
+            0.01053705532103777,
+            -0.015425797551870346,
+            0.010092624463140965,
+            -0.03384140133857727,
+            -0.019005486741662025,
+            -0.0034079779870808125,
+            -0.007672495674341917,
+            -0.03778471797704697,
+            -0.03616860508918762,
+            0.022916480898857117,
+            0.019150936976075172,
+            -0.009906771592795849,
+            0.0058947717770934105,
+            -0.010836036875844002,
+            9.204267553286627e-05,
+            0.026520412415266037,
+            0.016613639891147614,
+            -0.008177530951797962,
+            0.0012292558094486594,
+            -0.0047594523057341576,
+            0.0065088942646980286,
+            0.019312549382448196,
+            -0.06493541598320007,
+            -0.029203159734606743,
+            0.02504974976181984,
+            0.01789036951959133,
+            0.012330940924584866,
+            0.015207622200250626,
+            0.02535681053996086,
+            -0.02383766509592533,
+            -0.020912500098347664,
+            0.037849362939596176,
+            -0.003381716087460518,
+            0.01024615578353405,
+            -0.007947235368192196,
+            0.0019302447326481342,
+            0.00030453625367954373,
+            0.002240336500108242,
+            -0.0010797653812915087,
+            0.005757402162998915,
+            0.036394860595464706,
+            0.02239932492375374,
+            0.013122836127877235,
+            0.015078333206474781,
+            -0.0013625851133838296,
+            -0.004973587114363909,
+            -0.007951275445520878,
+            -0.0037796839606016874,
+            0.025405293330550194,
+            -0.017292408272624016,
+            0.0017150997882708907,
+            -0.026277994737029076,
+            0.02492045983672142,
+            0.025130555033683777,
+            0.01708231307566166,
+            -0.005369534716010094,
+            -0.007034131325781345,
+            0.013421816751360893,
+            0.0034766627941280603,
+            -0.009276487864553928,
+            0.016807572916150093,
+            0.0068442379124462605,
+            -0.020379183813929558,
+            -0.011789543554186821,
+            0.008205813355743885,
+            -0.007207863498479128,
+            -0.022948801517486572,
+            0.031158655881881714,
+            0.012831935659050941,
+            0.012662244029343128,
+            0.004795814864337444,
+            -0.0005444280104711652,
+            0.011522884480655193,
+            0.03484339267015457,
+            -0.036588795483112335,
+            -0.01771259680390358,
+            0.005434179678559303,
+            -0.02138117328286171,
+            0.00929264910519123,
+            -0.014908640645444393,
+            0.006193752400577068,
+            0.015983356162905693,
+            0.07408261299133301,
+            -0.027247661724686623,
+            -4.744175021187402e-05,
+            -0.016274256631731987,
+            -0.017098473384976387,
+            0.018326720222830772,
+            -0.004230175167322159,
+            0.0051109567284584045,
+            0.02765168994665146,
+            0.008395706303417683,
+            0.03636253997683525,
+            -0.007013929542154074,
+            -0.005373575259000063,
+            0.023385152220726013,
+            -0.0018645902164280415,
+            -0.010852198116481304,
+            -0.006723029538989067,
+            0.001137339393608272,
+            0.012815774418413639,
+            0.00790279172360897,
+            0.03258083388209343,
+            0.017567146569490433,
+            0.0012019838904961944,
+            -0.00039493758231401443,
+            -0.0044564311392605305,
+            0.028556713834404945,
+            0.012848096899688244,
+            0.0017423717072233558,
+            -0.01953880488872528,
+            -0.003040312323719263,
+            0.027473919093608856,
+            -0.021672073751688004,
+            0.009922932833433151,
+            0.015328830108046532,
+            0.005034191533923149,
+            0.04809551686048508,
+            -0.03351818025112152,
+            -0.008298739790916443,
+            0.012064282782375813,
+            0.0018625700613483787,
+            -0.008068443275988102,
+            0.01480359397828579,
+            0.006581619381904602,
+            -0.01056129764765501,
+            -0.00926840677857399,
+            0.012613760307431221,
+            0.013575348071753979,
+            -0.0063917264342308044,
+            -0.00901790987700224,
+            0.0066543444991111755,
+            -0.010771391913294792,
+            0.019958993420004845,
+            -0.00035781748010776937,
+            0.008254296146333218,
+            0.004193812608718872,
+            0.008718929253518581,
+            -0.0007681586430408061,
+            -0.04111390933394432,
+            -0.00789067056030035,
+            0.016177289187908173,
+            -0.003333232831209898,
+            0.021348850801587105,
+            -0.001368645578622818,
+            -0.008565397933125496,
+            -0.014876319095492363,
+            -0.004876620601862669,
+            -0.0036019114777445793,
+            -0.00519984308630228,
+            0.0005641243769787252,
+            -0.00471500912681222,
+            -0.01738937385380268,
+            0.011126937344670296,
+            -0.028766809031367302,
+            -0.007680576294660568,
+            -0.0033473738003522158,
+            -0.014246034435927868,
+            -0.004525116179138422,
+            -0.007094735279679298,
+            0.005102876108139753,
+            -0.007595730479806662,
+            0.019134776666760445,
+            0.0050543928518891335,
+            -0.029057709500193596,
+            -0.008710848167538643,
+            -0.027296146377921104,
+            0.017631791532039642,
+            -0.027215341106057167,
+            0.01192691270262003,
+            0.025340650230646133,
+            0.002365585183724761,
+            -0.0012100645108148456,
+            0.004131188616156578,
+            -0.029817281290888786,
+            0.0036685762461274862,
+            0.010609780438244343,
+            -0.009728998877108097,
+            0.00901790987700224,
+            0.023627569898962975,
+            -0.01816510781645775,
+            -0.025841644033789635,
+            -0.03207983821630478,
+            0.01565205305814743,
+            0.011110776104032993,
+            -0.014157148078083992,
+            -0.011692576110363007,
+            -0.0012080443557351828,
+            0.004929144401103258,
+            -0.01378544233739376,
+            -0.00010245903104078025,
+            -0.00785834901034832,
+            -0.010326961055397987,
+            -0.008088644593954086,
+            -0.008985587395727634,
+            -0.002513055456802249,
+            -0.0038907916750758886,
+            0.008355303667485714,
+            0.006933124270290136,
+            -0.03494035825133324,
+            -0.025841644033789635,
+            0.012856177985668182,
+            0.007288668770343065,
+            -0.01331677008420229,
+            0.009567388333380222,
+            0.014989446848630905,
+            0.004965506494045258,
+            -0.002377706114202738,
+            0.009890610352158546,
+            -0.018456008285284042,
+            0.0127188079059124,
+            0.0046665258705616,
+            0.004084725398570299,
+            0.016920700669288635,
+            0.015151058323681355,
+            0.012339022010564804,
+            0.01293698325753212,
+            -0.03290405869483948,
+            0.0007151299505494535,
+            0.027991075068712234,
+            0.048580352216959,
+            0.01636314205825329,
+            -0.022754868492484093,
+            0.021962974220514297,
+            0.04625314846634865,
+            0.02024989388883114,
+            -0.00060301210032776,
+            -0.016985345631837845,
+            0.0050543928518891335,
+            0.026261834427714348,
+            0.010351202450692654,
+            0.00418977253139019,
+            -0.006415967829525471,
+            -0.013551106676459312,
+            0.014173309318721294,
+            0.009462340734899044,
+            0.00678363349288702,
+            -0.037914007902145386,
+            0.020023638382554054,
+            -0.023579087108373642,
+            0.020653922110795975,
+            0.0027756737545132637,
+            -0.012427907437086105,
+            0.016314659267663956,
+            0.022156907245516777,
+            -0.0048725800588727,
+            0.020960982888936996,
+            0.0008767411927692592,
+            0.011296628974378109,
+            0.00837954506278038,
+            -0.011934993788599968,
+            0.003896852023899555,
+            -0.005975577048957348,
+            0.003793824929744005,
+            0.006189712323248386,
+            0.02302960865199566,
+            0.00397159717977047,
+            0.0186661034822464,
+            0.017696436494588852,
+            0.0010045150993391871,
+            0.008104805834591389,
+            -0.019748898223042488,
+            -0.00022486694797407836,
+            -0.009696677327156067,
+            0.007349273189902306,
+            0.009462340734899044,
+            0.0072927093133330345,
+            0.02479117177426815,
+            -0.012597599998116493,
+            -0.010585539042949677,
+            -0.01816510781645775,
+            0.014561176300048828,
+            -0.014149067923426628,
+            0.01634698174893856,
+            0.017761079594492912,
+            0.0012363262940198183,
+            0.008565397933125496,
+            0.02446794882416725,
+            0.002816076623275876,
+            -0.01656515710055828,
+            0.011078453622758389,
+            -0.023740697652101517,
+            -0.017308568581938744,
+            0.013478381559252739,
+            -0.01315515860915184,
+            -0.010270397178828716,
+            0.020524634048342705,
+            -0.015967195853590965,
+            0.0017565126763656735,
+            0.011151178739964962,
+            0.03062533773481846,
+            -0.017373213544487953,
+            0.022318517789244652,
+            -0.01075523067265749,
+            -0.023110413923859596,
+            -0.002626183442771435,
+            0.016662122681736946,
+            0.0269406009465456,
+            1.085036728909472e-06,
+            -0.005729120224714279,
+            -0.005207923706620932,
+            0.02309425175189972,
+            -0.008573479019105434,
+            0.022867996245622635,
+            0.007034131325781345,
+            0.012419827282428741,
+            0.00020618064445443451,
+            -0.018003497272729874,
+            0.017437858507037163,
+            0.0020544833969324827,
+            0.02214074693620205,
+            0.014407645910978317,
+            -0.015749020501971245,
+            0.02904154732823372,
+            -0.0030241510830819607,
+            0.028556713834404945,
+            0.0007206853479146957,
+            -0.009583549574017525,
+            0.02845974639058113,
+            -0.0016807573847472668,
+            -0.0004017555620521307,
+            0.006165470462292433,
+            0.01505409087985754,
+            0.016443947330117226,
+            0.00392311392351985,
+            0.020605439320206642,
+            0.016306579113006592,
+            -0.00015100554446689785,
+            0.014318759553134441,
+            -0.00879165343940258,
+            -0.022819513455033302,
+            0.022043779492378235,
+            -0.005034191533923149,
+            -0.01745401881635189,
+            0.01734089106321335,
+            -0.007696737535297871,
+            -0.004058463498950005,
+            -0.0007075543981045485,
+            0.023320509120821953,
+            0.016209611669182777,
+            -0.007575529161840677,
+            0.003365555079653859,
+            -0.004282698966562748,
+            -0.001307031256146729,
+            0.02789410762488842,
+            0.008613881655037403,
+            -0.0034968641120940447,
+            0.01511873584240675,
+            0.005236205644905567,
+            0.01024615578353405,
+            -0.00025580034707672894,
+            -0.007466441486030817,
+            -0.01955496519804001,
+            -0.0035796898882836103,
+            0.004763492848724127,
+            0.005494783632457256,
+            0.0005767502589151263,
+            -0.0060563827864825726,
+            -0.020734727382659912,
+            -0.04476632550358772,
+            -0.00019128210260532796,
+            0.0014726828085258603,
+            0.025922449305653572,
+            0.04169571027159691,
+            -0.005902852397412062,
+            -0.004747331608086824,
+            -0.012452149763703346,
+            0.0014009678270667791,
+            -0.008533075451850891,
+            -0.019700415432453156,
+            -0.006064463406801224,
+            -0.013817764818668365,
+            -0.01916709914803505,
+            0.01615304872393608,
+            -0.012654163874685764,
+            0.006048302166163921,
+            0.002583760302513838,
+            0.019215581938624382,
+            -0.016742929816246033,
+            0.02744159661233425,
+            -0.01634698174893856,
+            0.004747331608086824,
+            0.001796915428712964,
+            -0.002872640499845147,
+            -0.03322727978229523,
+            0.02068624459207058,
+            -0.003304950660094619,
+            -0.036685761064291,
+            -0.0037716033402830362,
+            -0.007219984196126461,
+            -0.009955255314707756,
+            0.03429391607642174,
+            0.0014171289512887597,
+            -0.007805825211107731,
+            -0.012096604332327843,
+            0.01119158137589693,
+            0.011328951455652714,
+            0.008197732269763947,
+            -0.006020020227879286,
+            -0.015029849484562874,
+            0.007405837066471577,
+            0.03267780318856239,
+            -0.019765060395002365,
+            -0.015207622200250626,
+            -0.0014858137583360076,
+            0.011692576110363007,
+            0.002739311195909977,
+            0.005927093792706728,
+            -0.030156666412949562,
+            -0.014302598312497139,
+            -0.009607790969312191,
+            -0.01135319285094738,
+            -0.022674063220620155,
+            -0.007979556918144226,
+            0.00907447375357151,
+            -0.02125188335776329,
+            0.02138117328286171,
+            0.029881926253437996,
+            -0.027991075068712234,
+            0.0340353362262249,
+            -0.011345111764967442,
+            0.009599709883332253,
+            -0.01034312229603529,
+            0.0007449270342476666,
+            -0.011102695018053055,
+            0.0007964406395331025,
+            0.011304709129035473,
+            -0.03581305965781212,
+            -0.04237448051571846,
+            -0.008945184759795666,
+            0.0010959265055134892,
+            -0.01795501448214054,
+            -0.006925043649971485,
+            0.03959476575255394,
+            0.0100845443084836,
+            -0.0031837422866374254,
+            -0.016225773841142654,
+            0.01864994317293167,
+            0.02233467996120453,
+            0.03262931853532791,
+            0.01555508654564619,
+            0.00450895493850112,
+            0.0008186621707864106,
+            0.011789543554186821,
+            -0.027490079402923584,
+            0.014795512892305851,
+            -0.0013696555979549885,
+            0.02676282823085785,
+            -0.0015746999997645617,
+            -0.0003719584783539176,
+            0.019748898223042488,
+            -0.024257853627204895,
+            -0.0024847735185176134,
+            0.00732099125161767,
+            -0.014722787775099277,
+            0.004965506494045258,
+            0.019021648913621902,
+            0.007757341489195824,
+            -0.036588795483112335,
+            0.0041655306704342365,
+            -0.010771391913294792,
+            0.01986202597618103,
+            0.00917144026607275,
+            0.010423927567899227,
+            -0.0031291984487324953,
+            -0.011102695018053055,
+            -0.011910751461982727,
+            0.007817945443093777,
+            -0.0025979015044867992,
+            -0.008003799244761467,
+            0.03041524440050125,
+            -0.003412018297240138,
+            -0.0032382861245423555,
+            0.00837954506278038,
+            0.006032141391187906,
+            0.006036181468516588,
+            0.0085977204144001,
+            -0.009163360111415386,
+            -0.024435626342892647,
+            -0.01490056049078703,
+            -0.018746908754110336,
+            0.005353373941034079,
+            0.010262317024171352,
+            -0.003717059502378106,
+            -0.02416088618338108,
+            0.005114997271448374,
+            -0.007203822955489159,
+            0.036782726645469666,
+            -0.008347222581505775,
+            -0.035457514226436615,
+            -0.02377302013337612,
+            -0.04667333886027336,
+            0.009211842902004719,
+            6.436042895074934e-05,
+            -0.021979134529829025,
+            -0.0039614965207874775,
+            -0.011490561999380589,
+            -0.028007235378026962,
+            0.012056201696395874,
+            0.019651932641863823,
+            -0.026019416749477386,
+            0.021461978554725647,
+            0.00763613311573863,
+            -0.015102574601769447,
+            0.00567659642547369,
+            -0.008743170648813248,
+            -0.0269406009465456,
+            0.002250437159091234,
+            -0.006278598215430975,
+            0.017114635556936264,
+            -0.01173297967761755,
+            -0.0001382534101139754,
+            -0.020912500098347664,
+            0.01860145851969719,
+            0.021219560876488686,
+            -0.011070373468101025,
+            0.0014433907344937325,
+            -0.0037210998125374317,
+            -0.02568003349006176,
+            0.005555388052016497,
+            0.0007545226835645735,
+            -0.004614002071321011,
+            -0.0004100886289961636,
+            0.006791714113205671,
+            -0.009122957475483418,
+            0.00086159014608711,
+            -0.0051513598300516605,
+            0.008274497464299202,
+            0.0017393414163962007,
+            -0.01034312229603529,
+            -0.002296900376677513,
+            -0.003636253997683525,
+            0.0024484109599143267,
+            0.019134776666760445,
+            -0.015183379873633385,
+            0.007777543272823095,
+            -0.03157884627580643,
+            0.006832117214798927,
+            0.009623952209949493,
+            -0.0006408897461369634,
+            -0.02813652530312538,
+            0.0010327971540391445,
+            -0.0007090694853104651,
+            0.005531146191060543,
+            -0.026342639699578285,
+            0.029267802834510803,
+            0.0216397512704134,
+            0.011458240449428558,
+            0.015862148255109787,
+            0.001985798589885235,
+            -0.013712717220187187,
+            -0.016678284853696823,
+            0.003785744309425354,
+            -0.016936862841248512,
+            -0.009284568019211292,
+            0.00036589804221875966,
+            0.027490079402923584,
+            -0.024257853627204895,
+            -0.009720918722450733,
+            -0.00028054710128344595,
+            0.0005009949672967196,
+            -0.005680636502802372,
+            -0.010019899345934391,
+            0.004836217500269413,
+            0.013971295207738876,
+            0.017179278656840324,
+            -0.010698666796088219,
+            0.07395332306623459,
+            0.002173671731725335,
+            -0.006787674035876989,
+            0.0013817765284329653,
+            -0.011918832547962666,
+            -0.019910510629415512,
+            0.02086401730775833,
+            -0.022124584764242172,
+            0.013736959546804428,
+            0.007288668770343065,
+            -0.01567629538476467,
+            -0.00758764985948801,
+            -0.015991436317563057,
+            0.024904299527406693,
+            0.0017201501177623868,
+            0.0034867634531110525,
+            0.00689272116869688,
+            0.020734727382659912,
+            0.027296146377921104,
+            0.012274377048015594,
+            0.0041069467552006245,
+            -0.0027978953439742327,
+            -0.004840258043259382,
+            0.016726767644286156,
+            -0.01910245418548584,
+            0.002250437159091234,
+            0.005886691156774759,
+            0.022415485233068466,
+            -0.004969547037035227,
+            0.034261591732501984,
+            0.002882741391658783,
+            0.003143339417874813,
+            -0.006415967829525471,
+            0.003878670744597912,
+            0.020589277148246765,
+            -0.002850419143214822,
+            0.01664596237242222,
+            -0.006367484573274851,
+            -0.0033069709315896034,
+            0.0004800360184162855,
+            0.0010196662042289972,
+            0.002692847978323698,
+            0.0072967493906617165,
+            -0.019344870001077652,
+            -0.0002853449259418994,
+            0.00017448968719691038,
+            0.007264427375048399,
+            0.0029898087959736586,
+            0.0034180786460638046,
+            -0.004375625401735306,
+            0.00046437993296422064,
+            0.006613941863179207,
+            0.010003738105297089,
+            0.013074353337287903,
+            -0.0013696555979549885,
+            -0.005171561148017645,
+            0.006876560393720865,
+            -0.008169450797140598,
+            0.0013272326905280352,
+            0.025793161243200302,
+            0.0013504642993211746,
+            0.0062018330208957195,
+            -0.024871977046132088,
+            0.001361575094051659,
+            0.008686606772243977,
+            0.007595730479806662,
+            0.010674425400793552,
+            4.075003380421549e-05,
+            -0.0029372849967330694,
+            0.0012050141813233495,
+            -0.014019778929650784,
+            0.009632032364606857,
+            0.021785201504826546,
+            -0.0037675630301237106,
+            0.009042151272296906,
+            0.01050473377108574,
+            0.0023494239430874586,
+            -0.029687993228435516,
+            -0.015110654756426811,
+            0.013639992102980614,
+            0.001351474318653345,
+            0.008775493130087852,
+            4.4506232370622456e-05,
+            0.009543146006762981,
+            -0.02214074693620205,
+            -0.011175420135259628,
+            -0.0049089426174759865,
+            0.012807694263756275,
+            0.018229752779006958,
+            -0.021720556542277336,
+            0.013882409781217575,
+            0.006504854187369347,
+            0.004331182222813368,
+            -0.027279984205961227,
+            -0.005454380996525288,
+            0.013187481090426445,
+            -0.01128854788839817,
+            0.002939305268228054,
+            -0.01072290912270546,
+            0.005684677045792341,
+            0.003739281091839075,
+            0.02307809144258499,
+            -0.008840137161314487,
+            0.007761382032185793,
+            0.0037352407816797495,
+            -0.023045768961310387,
+            0.011078453622758389,
+            0.020411504432559013,
+            -0.00525236688554287,
+            -0.0013262225547805429,
+            -0.00577356293797493,
+            -0.00828257855027914,
+            0.015797503292560577,
+            -0.0021433697547763586,
+            -0.011660254560410976,
+            0.01658131740987301,
+            -0.006908882409334183,
+            -0.011369354091584682,
+            -0.022609418258070946,
+            -0.006177591159939766,
+            -0.006658385042101145,
+            -0.0026968882884830236,
+            0.009753241203725338,
+            -0.011442079208791256,
+            0.0034322196152061224,
+            -0.0017565126763656735,
+            -0.0078462278470397,
+            -0.004605921451002359,
+            -0.03358282521367073,
+            -0.018423685804009438,
+            -0.00811692699790001,
+            -0.02012060582637787,
+            0.002057513687759638,
+            0.003706958843395114,
+            -0.00716342031955719,
+            0.017922692000865936,
+            -0.010569377802312374,
+            -0.007575529161840677,
+            0.003414038335904479,
+            0.012912741862237453,
+            -0.00810884591192007,
+            -0.009890610352158546,
+            -0.019651932641863823,
+            0.0003696858184412122,
+            0.01591063104569912,
+            0.0183428805321455,
+            0.02537297084927559,
+            -0.016144966706633568,
+            0.0032948500011116266,
+            -0.024177048355340958,
+            0.008125007152557373,
+            -0.030382921919226646,
+            -0.012751130387187004,
+            -0.004840258043259382,
+            -0.0007378565496765077,
+            -0.008888620883226395,
+            -0.012040040455758572,
+            -0.04124319925904274,
+            -0.009785563684999943,
+            -0.015102574601769447,
+            -0.02322354167699814,
+            0.015134897083044052,
+            -0.005818006116896868,
+            0.017049990594387054,
+            -0.009567388333380222,
+            0.016856057569384575,
+            0.016904540359973907,
+            0.014553096145391464,
+            -0.007341192569583654,
+            0.00732099125161767,
+            -0.004387746565043926,
+            0.014407645910978317,
+            0.004123107995837927,
+            -0.0038928117137402296,
+            0.014973285607993603,
+            0.009995657950639725,
+            0.0050947959534823895,
+            0.00045478425454348326,
+            0.0017191400984302163,
+            0.010941083543002605,
+            0.0007924003293737769,
+            0.015635890886187553,
+            0.007975516840815544,
+            -0.0016837875591591,
+            0.034003015607595444,
+            -0.021785201504826546,
+            -0.010634022764861584,
+            0.010060302913188934,
+            0.01258143875747919,
+            -0.01873074844479561,
+            -0.0017837845953181386,
+            0.010512813925743103,
+            -0.0013201622059568763,
+            -0.030964722856879234,
+            0.013171319849789143,
+            0.011361273005604744,
+            0.0029999094549566507,
+            -0.005114997271448374,
+            -0.016791412606835365,
+            -0.005062473472207785,
+            0.007062413264065981,
+            0.020314538851380348,
+            -0.018245914950966835,
+            0.009939094074070454,
+            -0.007401796989142895,
+            -0.0061452691443264484,
+            -0.02689211815595627,
+            0.012532955035567284,
+            0.022948801517486572,
+            0.009195681661367416,
+            -0.059052761644124985,
+            -0.010375444777309895,
+            0.017858047038316727,
+            -0.0064927334897220135,
+            -0.03755846247076988,
+            -0.002294880338013172,
+            -0.01517529971897602,
+            -0.02307809144258499,
+            0.008767412044107914,
+            0.015732858330011368,
+            -0.006832117214798927,
+            0.015724778175354004,
+            0.006609901320189238,
+            -0.007482602261006832,
+            0.0003903922624886036,
+            -0.004973587114363909,
+            -0.03458481654524803,
+            -0.006678586360067129,
+            0.016548994928598404,
+            -0.03561912849545479,
+            -0.00721190357580781,
+            -0.00827045738697052,
+            -0.014553096145391464,
+            0.006052342709153891,
+            0.011248145252466202,
+            0.03972405195236206,
+            0.007405837066471577,
+            -0.01229861844331026,
+            -0.014003617689013481,
+            -0.014528854750096798,
+            -0.013389495201408863,
+            0.0017100494587793946,
+            -0.01176530122756958,
+            0.011240065097808838,
+            0.008331061340868473,
+            -0.009373454377055168,
+            0.023417474702000618,
+            -0.003403937676921487,
+            0.009559307247400284,
+            -3.494212796795182e-05,
+            -0.004710969049483538,
+            -0.005830127280205488,
+            -0.014722787775099277,
+            0.0008090665214695036,
+            0.003913013264536858,
+            -0.0025191158056259155,
+            -0.0005232165567576885,
+            0.0038221068680286407,
+            0.02706989087164402,
+            -0.010472411289811134,
+            -0.005442259833216667,
+            -0.010286558419466019,
+            -0.028362780809402466,
+            0.0003393836959730834,
+            0.02845974639058113,
+            0.030948560684919357,
+            -0.015684375539422035,
+            -0.01309051364660263,
+            -0.01789036951959133,
+            0.0067351502366364,
+            -0.0038604894652962685,
+            -0.0033392931800335646,
+            0.029397092759609222,
+            0.020977145060896873,
+            0.004137248732149601,
+            0.01732472889125347,
+            -0.009357293136417866,
+            0.006169511005282402,
+            0.028443586081266403,
+            -0.0013302628649398685,
+            0.009939094074070454,
+            0.010149188339710236,
+            0.013963215053081512,
+            -0.00986636895686388,
+            -0.000141536133014597,
+            0.005288729444146156,
+            -0.004646324552595615,
+            0.014084423892199993,
+            0.010666345246136189,
+            -0.008864378556609154,
+            -0.021203400567173958,
+            -0.005066513549536467,
+            -0.020896339789032936,
+            -0.0052483263425529,
+            0.006112946663051844,
+            -0.010836036875844002,
+            -0.001267638523131609,
+            -0.029397092759609222,
+            -0.004383706022053957,
+            0.002311041345819831,
+            -0.013211722485721111,
+            0.01027847733348608,
+            -0.011078453622758389,
+            0.0025251763872802258,
+            0.006290719378739595,
+            -0.02088017761707306,
+            -0.003717059502378106,
+            -0.007377555128186941,
+            0.003652415005490184,
+            0.005325092002749443,
+            0.00614930922165513,
+            -0.0016534854657948017,
+            0.005874569993466139,
+            0.020395344123244286,
+            0.0020787252578884363,
+            0.01486015785485506,
+            -0.02079937234520912,
+            -0.028508231043815613,
+            -0.029073869809508324,
+            0.015611650422215462,
+            -0.0031251581385731697,
+            0.0031837422866374254,
+            -0.0033291925210505724,
+            -0.010844117030501366,
+            0.004828137345612049,
+            0.003913013264536858,
+            0.0014322800561785698,
+            -0.002852439181879163,
+            0.00493318447843194,
+            -0.0031716213561594486,
+            0.010585539042949677,
+            0.006036181468516588,
+            -0.02036302164196968,
+            -0.006024060770869255,
+            -0.009623952209949493,
+            0.0073533132672309875,
+            -0.018375203013420105,
+            -0.009357293136417866,
+            0.00031968732946552336,
+            -0.0018777211662381887,
+            0.0033493938390165567,
+            -0.04812784120440483,
+            -0.019522642716765404,
+            -0.014334920793771744,
+            0.008516915142536163,
+            -0.0019443858182057738,
+            -0.03128794580698013,
+            -0.020039798691868782,
+            0.008133088238537312,
+            0.01580558344721794,
+            0.002410028362646699,
+            0.008920943364501,
+            0.010189591906964779,
+            0.00128480966668576,
+            -0.006436169613152742,
+            -0.0020251914393156767,
+            0.02081553265452385,
+            0.01714695803821087,
+            -0.030593017116189003,
+            -0.019150936976075172,
+            0.010642102919518948,
+            -0.008331061340868473,
+            -0.013615750707685947,
+            -0.01002798043191433,
+            0.007199782878160477,
+            0.0006338192615658045,
+            0.021041790023446083,
+            -0.0015514683909714222,
+            -0.019183259457349777,
+            0.004084725398570299,
+            0.00578568410128355,
+            -0.018892358988523483,
+            0.013219802640378475,
+            -0.007975516840815544,
+            -0.008743170648813248,
+            0.0058988118544220924,
+            -0.007264427375048399,
+            0.0032362660858780146,
+            -0.005183681845664978,
+            0.02580932155251503,
+            -0.020039798691868782,
+            0.0011969335610046983,
+            0.005535186734050512,
+            -0.014092504046857357,
+            0.016969185322523117,
+            -0.008605800569057465,
+            -0.0047594523057341576,
+            0.005013990215957165,
+            0.0007782593602314591,
+            0.009575468488037586,
+            0.009971416555345058,
+            0.008238134905695915,
+            -0.02023373357951641,
+            -0.0012837996473535895,
+            0.005381655879318714,
+            0.007357353810220957,
+            -0.008702768012881279,
+            0.03004353865981102,
+            -0.0054018571972846985,
+            -0.022722546011209488,
+            -0.0009408806799910963,
+            -0.008165409788489342,
+            0.0004906417452730238,
+            -0.00450491439551115,
+            0.007276548072695732,
+            0.015635890886187553,
+            0.006169511005282402,
+            -0.017599469050765038,
+            -0.012831935659050941,
+            0.003323131939396262,
+            0.005147319287061691,
+            0.012767291627824306,
+            0.015167218632996082,
+            -0.011555206961929798,
+            0.010068383067846298,
+            -0.020152926445007324,
+            0.0213165283203125,
+            -0.009753241203725338,
+            0.0038019055500626564,
+            -0.005591750610619783,
+            0.003121117828413844,
+            0.008670445531606674,
+            0.0175186637789011,
+            0.0019575166516005993,
+            -0.0012433967785909772,
+            -0.006133148446679115,
+            0.0350373275578022,
+            -0.00604022154584527,
+            0.016936862841248512,
+            -0.020734727382659912,
+            -0.010060302913188934,
+            -0.024063920602202415,
+            0.03662111610174179,
+            -0.004117047414183617,
+            -0.004589760676026344,
+            0.012007718905806541,
+            0.021930651739239693,
+            -0.012896580621600151,
+            0.009753241203725338,
+            0.003682717215269804,
+            -0.006230114959180355,
+            0.005591750610619783,
+            0.0013767260825261474,
+            0.0008519944967702031,
+            0.02079937234520912,
+            0.017163118347525597,
+            0.008141168393194675,
+            0.007805825211107731,
+            0.011377434246242046,
+            -0.0005098331021144986,
+            -0.012104685418307781,
+            0.01182994619011879,
+            -0.011999637819826603,
+            -0.004094826057553291,
+            -0.0032302055042237043,
+            -0.014415726996958256,
+            -0.009947174228727818,
+            -0.00828257855027914,
+            0.012128926813602448,
+            -0.0035453476011753082,
+            -0.01966809295117855,
+            0.0037251401226967573,
+            0.020266054198145866,
+            0.003032231703400612,
+            0.0059957788325846195,
+            0.014052101410925388,
+            0.01886003650724888,
+            0.01986202597618103,
+            0.0070462520234286785,
+            -0.02941325306892395,
+            -0.0009024980245158076,
+            -0.007070493884384632,
+            -0.006036181468516588,
+            0.009219923987984657,
+            0.0011949134059250355,
+            0.00810884591192007,
+            -0.0035776698496192694,
+            0.012799614109098911,
+            0.03713827207684517,
+            0.013583428226411343,
+            -0.0011353192385286093,
+            0.002395887393504381,
+            -0.002739311195909977,
+            -0.01986202597618103,
+            0.018116625025868416,
+            -0.01623385399580002,
+            -0.027183018624782562,
+            0.021268045529723167,
+            -0.0002911528281401843,
+            -0.006185671780258417,
+            -0.021962974220514297,
+            -0.006884640548378229,
+            0.016080323606729507,
+            0.018116625025868416,
+            -0.0008767411927692592,
+            -0.0018676203908398747,
+            -0.0033190918620675802,
+            0.0031473797280341387,
+            -0.006161430384963751,
+            0.005583669990301132,
+            -0.0020221611484885216,
+            -0.023708375170826912,
+            -0.006815955974161625,
+            -0.011296628974378109,
+            0.006803835276514292,
+            0.0033069709315896034,
+            0.013696555979549885,
+            -0.004117047414183617,
+            0.020346861332654953,
+            0.005325092002749443,
+            -0.004234215710312128,
+            0.007013929542154074,
+            -0.009858288802206516,
+            -0.0030504129827022552,
+            -0.01997515559196472,
+            0.015223783440887928,
+            0.0055028642527759075,
+            0.0032847493421286345,
+            -0.001361575094051659,
+            0.02910619229078293,
+            -0.0005600841250270605,
+            0.006864439230412245,
+            -0.003783724270761013,
+            0.011248145252466202,
+            -0.002197913359850645,
+            -0.008751250803470612,
+            -0.03183742240071297,
+            -0.01039968617260456,
+            -0.02682747319340706,
+            0.023320509120821953,
+            -0.009219923987984657,
+            0.0012353162746876478,
+            0.008589639328420162,
+            0.003672616556286812,
+            -0.004315021447837353,
+            0.010924923233687878,
+            -0.013591509312391281,
+            -0.004234215710312128,
+            0.012896580621600151,
+            -0.0004992273752577603,
+            0.009720918722450733,
+            -0.0073937163688242435,
+            -0.003852409077808261,
+            -0.011369354091584682,
+            -0.008500753901898861,
+            -0.03010818175971508,
+            -0.013308688998222351,
+            -0.01452077366411686,
+            -0.013551106676459312,
+            -0.024500271305441856,
+            -0.0008459340897388756,
+            0.004197853151708841,
+            -0.003761502681300044,
+            -0.010900680907070637,
+            0.017486341297626495,
+            0.01416522916406393,
+            0.02770017459988594,
+            -0.026714345440268517,
+            0.0006358394166454673,
+            -0.007648254279047251,
+            0.018520653247833252,
+            -0.01072290912270546,
+            0.0050947959534823895,
+            0.0015726798446848989,
+            0.01677525043487549,
+            0.0033837363589555025,
+            0.0012464270694181323,
+            0.004517035558819771,
+            0.0018858016701415181,
+            -0.01027847733348608,
+            0.009179521352052689,
+            0.0008439139346592128,
+            -0.011644093319773674,
+            -0.003921093884855509,
+            0.016872217878699303,
+            0.0201690886169672,
+            0.01293698325753212,
+            -0.007219984196126461,
+            0.0034403002355247736,
+            -0.009381535463035107,
+            -0.011846107430756092,
+            -0.011789543554186821,
+            0.008314901031553745,
+            -0.025259843096137047,
+            0.01814894750714302,
+            0.008937103673815727,
+            -0.0008227024227380753,
+            0.007785623427480459,
+            0.0023514442145824432,
+            0.015967195853590965,
+            0.005523065570741892,
+            -0.019393354654312134,
+            -0.014448048546910286,
+            0.0027978953439742327,
+            0.012104685418307781,
+            0.0010252215433865786,
+            -0.0023817464243620634,
+            0.00710281589999795,
+            0.0002638809382915497,
+            -0.011474400758743286,
+            -0.017534824088215828,
+            0.0003237276105210185,
+            0.00938961561769247,
+            0.02290031872689724,
+            0.005757402162998915,
+            0.013446059077978134,
+            0.0022282155696302652,
+            -0.0057493215426802635,
+            -0.007143219001591206,
+            -0.0161045640707016,
+            0.015959113836288452,
+            0.0038928117137402296,
+            -0.025453777983784676,
+            0.024888137355446815,
+            0.002057513687759638,
+            0.01688838005065918,
+            -0.004686727188527584,
+            -0.005838207434862852,
+            0.023756857961416245,
+            0.027780979871749878,
+            0.0043109809048473835,
+            0.024839654564857483,
+            -0.013494541868567467,
+            0.0072523062117397785,
+            0.02516287751495838,
+            0.020459989085793495,
+            -0.009688596241176128,
+            0.0012908701319247484,
+            0.011967315338551998,
+            0.001570659689605236,
+            -0.0005828107241541147,
+            0.009971416555345058,
+            -0.010593620128929615,
+            0.010690586641430855,
+            -0.02175287902355194,
+            0.017437858507037163,
+            -0.0014777331380173564,
+            -0.005183681845664978,
+            -0.006395766511559486,
+            0.0050947959534823895,
+            0.006011939607560635,
+            0.011102695018053055,
+            -0.009203762747347355,
+            -0.021542783826589584,
+            -0.0015292468015104532,
+            -0.003304950660094619,
+            -0.012395585887134075,
+            -0.028120363131165504,
+            0.012177410535514355,
+            -0.013575348071753979,
+            -0.008112886920571327,
+            0.009882530197501183,
+            -0.01721160113811493,
+            0.00241204840131104,
+            -0.02335282973945141,
+            0.010108785703778267,
+            -0.0216397512704134,
+            0.0052564069628715515,
+            0.011854187585413456,
+            -0.005846288055181503,
+            0.004282698966562748,
+            0.006165470462292433,
+            0.010415847413241863,
+            8.724484359845519e-05,
+            -0.013195561245083809,
+            -0.007450280245393515,
+            0.028605196624994278,
+            0.006290719378739595,
+            -0.003737261053174734,
+            -0.01821359246969223,
+            -0.0008454290218651295,
+            -0.028492068871855736,
+            0.018310558050870895,
+            -0.00041185625013895333,
+            0.0008888620650395751,
+            -0.019651932641863823,
+            -0.011636012233793736,
+            -0.0008120966958813369,
+            0.014391484670341015,
+            -0.00019607994181569666,
+            0.003793824929744005,
+            0.0317404568195343,
+            -0.013599589467048645,
+            0.001599951763637364,
+            -6.699608547933167e-06,
+            -0.01303394977003336,
+            0.02005596086382866,
+            -0.005757402162998915,
+            0.006925043649971485,
+            -0.0007333112298510969,
+            -0.008165409788489342,
+            -0.014666223898530006,
+            0.022108424454927444,
+            -0.0033190918620675802,
+            -0.012258215807378292,
+            -0.006763432174921036,
+            0.002161551034078002,
+            0.013551106676459312,
+            -0.012379424646496773,
+            -0.008621961809694767,
+            0.010924923233687878,
+            0.00488470122218132,
+            -0.011110776104032993,
+            -0.010044141672551632,
+            -0.013478381559252739,
+            0.0037433214019984007,
+            -0.03494035825133324,
+            -0.006929083727300167,
+            0.04586528241634369,
+            0.0056038713082671165,
+            -0.008395706303417683,
+            -0.0058503285981714725,
+            0.0009530015522614121,
+            0.006973526906222105,
+            -0.01847217045724392,
+            -0.007619971875101328,
+            -0.027780979871749878,
+            0.005983657669275999,
+            0.002139329444617033,
+            0.03243538364768028,
+            0.0003406462783459574,
+            -0.018488330766558647,
+            0.003549387911334634,
+            -0.021300368010997772,
+            -0.0074421996250748634,
+            -0.018391363322734833,
+            -0.016209611669182777,
+            -0.008920943364501,
+            0.016904540359973907,
+            -0.015167218632996082,
+            0.02663354016840458,
+            0.016597479581832886,
+            0.007603811100125313,
+            0.003117077751085162,
+            -0.0037998852785676718,
+            -0.006415967829525471,
+            0.006432129070162773,
+            -0.0019383253529667854,
+            0.004953385796397924,
+            0.011296628974378109,
+            -0.01075523067265749,
+            0.009922932833433151,
+            0.013688475824892521,
+            -0.007624012418091297,
+            0.01840752549469471,
+            -0.00837954506278038,
+            -0.020459989085793495,
+            0.01452077366411686,
+            -0.014706626534461975,
+            0.00152621662709862,
+            0.0037574623711407185,
+            -0.00832298118621111,
+            0.00472713029012084,
+            0.018989326432347298,
+            0.0073977564461529255,
+            -0.00572507968172431,
+            0.017373213544487953,
+            0.020670082420110703,
+            0.012629921548068523,
+            -0.01050473377108574,
+            0.009591629728674889,
+            0.014439968392252922,
+            -0.0001660303387325257,
+            -0.0003701908572111279,
+            0.004407947883009911,
+            -0.014407645910978317,
+            -0.0006307890289463103,
+            0.003212024224922061,
+            -0.011094614863395691,
+            -0.0115713682025671,
+            0.016839895397424698,
+            0.005244286265224218,
+            -0.004517035558819771,
+            0.011700657196342945,
+            -0.01284001674503088,
+            -0.012751130387187004,
+            -0.009825966320931911,
+            0.0066987876780331135,
+            -0.040758367627859116,
+            0.004840258043259382,
+            -0.006464451551437378,
+            0.018100464716553688,
+            -0.014456129632890224,
+            0.005236205644905567,
+            -0.005919013172388077,
+            -0.010949164628982544,
+            0.015070252120494843,
+            -0.0009762331610545516,
+            -0.002218114910647273,
+            0.03188590705394745,
+            0.011934993788599968,
+            0.01077947299927473,
+            0.004646324552595615,
+            -0.016371222212910652,
+            0.034261591732501984,
+            -0.009882530197501183,
+            0.00012966780923306942,
+            -0.012856177985668182,
+            -0.0017342910869047046,
+            0.03186974674463272,
+            0.029332447797060013,
+            -0.005046312231570482,
+            0.004480673000216484,
+            -0.00795531552284956,
+            0.01363191194832325,
+            0.015991436317563057,
+            0.008133088238537312,
+            0.00599173828959465,
+            0.0032605077140033245,
+            0.004383706022053957,
+            0.0011009768350049853,
+            0.038495808839797974,
+            -0.012613760307431221,
+            -0.011943073943257332,
+            -0.013955134898424149,
+            -0.016662122681736946,
+            -0.022738708183169365,
+            0.02138117328286171,
+            -0.00747856218367815,
+            -0.014916721731424332,
+            -0.007769462652504444,
+            0.018989326432347298,
+            0.00705433264374733,
+            0.006714948918670416,
+            -0.0012928902870044112,
+            0.005539226811379194,
+            -0.024451786652207375,
+            -0.00499378889799118,
+            -0.01738937385380268,
+            0.002729210536926985,
+            -0.0007913902518339455,
+            0.015951033681631088,
+            0.018456008285284042,
+            0.017437858507037163,
+            0.019005486741662025,
+            -0.007426038384437561,
+            0.006169511005282402,
+            -0.0055069043301045895,
+            -0.01226629689335823,
+            0.019199419766664505,
+            0.02086401730775833,
+            -0.01404402032494545,
+            -0.004274618346244097,
+            -0.016710607334971428,
+            -0.021171078085899353,
+            0.002983748447149992,
+            0.009543146006762981,
+            0.010884519666433334,
+            -0.013219802640378475,
+            0.002816076623275876,
+            0.015264186076819897,
+            -0.007042211946099997,
+            0.002769613405689597,
+            0.0048483386635780334,
+            0.017373213544487953,
+            0.01224205456674099,
+            0.0031675812788307667,
+            0.01803581975400448,
+            -0.018957003951072693,
+            -0.02776481956243515,
+            0.006589700002223253,
+            -0.014650062657892704,
+            0.017502501606941223,
+            0.009025990031659603,
+            0.005672555882483721,
+            -0.03149804100394249,
+            -0.010827955789864063,
+            0.01814894750714302,
+            -0.00046715763164684176,
+            -0.001248447224497795,
+            0.001733281067572534,
+            0.01077947299927473,
+            -0.009228004142642021,
+            -0.012670325115323067,
+            -0.01886003650724888,
+            0.0010595639469102025,
+            -0.0051513598300516605,
+            -0.0030584936030209064,
+            -0.01293698325753212,
+            -0.00844419002532959,
+            -0.013486461713910103,
+            -0.004420068580657244,
+            0.0008504794095642865,
+            0.0234659593552351,
+            0.01871458627283573,
+            0.01936103217303753,
+            0.002983748447149992,
+            0.004500874318182468,
+            0.017744919285178185,
+            0.013025869615375996,
+            0.007922993041574955,
+            0.00160500209312886,
+            -0.0019474159926176071,
+            0.028572876006364822,
+            -0.019199419766664505,
+            -0.012031960301101208,
+            0.018569136038422585,
+            -0.01331677008420229,
+            0.017761079594492912,
+            -0.02118724025785923,
+            0.001272688852623105,
+            0.005692757666110992,
+            0.003387776669114828,
+            -0.009809805080294609,
+            -0.008686606772243977,
+            0.019247904419898987,
+            0.009122957475483418,
+            -0.009413857012987137,
+            0.01979738287627697,
+            0.005769522860646248,
+            -0.006658385042101145,
+            -0.009696677327156067,
+            -0.0019383253529667854,
+            -0.0041089667938649654,
+            -0.015732858330011368,
+            0.016856057569384575,
+            0.018132785335183144,
+            -0.015353072434663773,
+            0.0042665377259254456,
+            0.01879539154469967,
+            -0.01973273791372776,
+            0.011248145252466202,
+            0.01378544233739376,
+            -0.010092624463140965,
+            0.007809865288436413,
+            -0.0010186561848968267,
+            0.013446059077978134,
+            0.003761502681300044,
+            -0.004145329352468252,
+            -0.0005929113831371069,
+            0.011967315338551998,
+            -0.017243923619389534,
+            -0.012112765572965145,
+            -0.012427907437086105,
+            -0.008157329633831978,
+            0.007123017217963934,
+            0.002535277046263218,
+            -0.0056483144871890545,
+            0.009025990031659603,
+            -0.011337031610310078,
+            -0.015991436317563057,
+            -0.028378941118717194,
+            0.0005499834078364074,
+            -0.00971283856779337,
+            -0.001211074530147016,
+            0.014698546379804611,
+            0.015749020501971245,
+            -0.0009449209901504219,
+            -0.016969185322523117,
+            0.01816510781645775,
+            0.04408755898475647,
+            -0.007809865288436413,
+            0.003777663689106703,
+            0.010601700283586979,
+            0.002488813828676939,
+            -0.0026302237529307604,
+            -0.0022908400278538465,
+            -0.016427787020802498,
+            -0.03765542805194855,
+            -0.012912741862237453,
+            0.01077947299927473,
+            -0.0012787493178620934,
+            -0.004638243932276964,
+            -0.021914489567279816,
+            -0.005842247977852821,
+            -0.016201531514525414,
+            -0.004921063780784607,
+            -0.019651932641863823,
+            -0.004799854941666126,
+            0.013979376293718815,
+            -0.015959113836288452,
+            0.019878188148140907,
+            -0.02024989388883114,
+            0.020540794357657433,
+            0.01385008729994297,
+            0.02277103066444397,
+            -0.01641162671148777,
+            0.009341131895780563,
+            -0.0014120786217972636,
+            0.014666223898530006,
+            0.01034312229603529,
+            0.007308870553970337,
+            -0.010617861524224281,
+            -0.006634143181145191,
+            0.01928022690117359,
+            0.009470420889556408,
+            0.013082433491945267,
+            0.009567388333380222,
+            -0.002262558089569211,
+            -0.0073371524922549725,
+            -0.014464209787547588,
+            -0.022156907245516777,
+            0.003163540968671441,
+            -0.0021676113829016685,
+            0.01688838005065918,
+            -0.020379183813929558,
+            -0.021607428789138794,
+            0.0002142612065654248,
+            0.014722787775099277,
+            0.035069648176431656,
+            -0.008573479019105434,
+            0.004371585324406624,
+            -0.012330940924584866,
+            0.01202387921512127,
+            0.018310558050870895,
+            0.005236205644905567,
+            0.0012221853248775005,
+            -0.020346861332654953,
+            0.011579448357224464,
+            0.011086533777415752,
+            -0.002422149060294032,
+            -0.0035089850425720215,
+            -0.0029029427096247673,
+            -0.02290031872689724,
+            0.0064078872092068195,
+            0.004399867262691259,
+            0.038043297827243805,
+            0.013761200942099094,
+            -0.0014807634288445115,
+            0.0072523062117397785,
+            0.029073869809508324,
+            -0.0032362660858780146,
+            0.009276487864553928,
+            0.0007363414042629302,
+            -0.003854429116472602,
+            0.008060363121330738,
+            -0.00811692699790001,
+            -0.0459299273788929,
+            0.010577458888292313,
+            -0.004492793697863817,
+            0.038043297827243805,
+            -0.0032746486831456423,
+            0.03043140470981598,
+            -0.006533136125653982,
+            -0.00810884591192007,
+            -0.005531146191060543,
+            0.017373213544487953,
+            -0.012508713640272617,
+            0.012258215807378292,
+            -0.008044201880693436,
+            -0.010625941678881645,
+            -0.0016342941671609879,
+            0.006779593415558338,
+            -0.012371343560516834,
+            -0.007333111949265003,
+            0.022156907245516777,
+            0.024387143552303314,
+            0.013761200942099094,
+            -0.009834046475589275,
+            0.0315626822412014,
+            -0.007510884664952755,
+            -0.014472290873527527,
+            0.019005486741662025,
+            -0.0037109991535544395,
+            -0.006428088992834091,
+            -0.001322182361036539,
+            -0.0023029607255011797,
+            -0.0024322497192770243,
+            0.017357051372528076,
+            0.0175186637789011,
+            -0.00429886020720005,
+            0.0015373273054137826,
+            0.00429886020720005,
+            -0.002468612277880311,
+            0.019150936976075172,
+            0.011749139986932278,
+            -0.03636253997683525,
+            0.014221793040633202,
+            -0.029364770278334618,
+            -0.01821359246969223,
+            -0.007579569239169359,
+            -0.004157450515776873,
+            0.011708737351000309,
+            -0.02390230819582939,
+            -0.011490561999380589,
+            -0.023417474702000618,
+            0.009753241203725338,
+            0.016969185322523117,
+            0.0065290960483253,
+            0.012137007899582386,
+            0.014302598312497139,
+            0.0012252154992893338,
+            -0.017421696335077286,
+            -0.013680395670235157,
+            -0.02765168994665146,
+            -0.014706626534461975,
+            0.012185490690171719,
+            0.0037493817508220673,
+            -0.013446059077978134,
+            -0.0023453838657587767,
+            0.026649700477719307,
+            -0.0037918046582490206,
+            -0.00313121872022748,
+            0.009753241203725338,
+            0.003694837912917137,
+            0.005220044404268265,
+            -0.005886691156774759,
+            0.014246034435927868,
+            0.00716342031955719,
+            0.01012494694441557,
+            -0.02984960377216339,
+            0.01738937385380268,
+            -0.003737261053174734,
+            -0.0008519944967702031,
+            0.0013363233301788568,
+            -0.004262497648596764,
+            -0.011660254560410976,
+            -0.0017757039749994874,
+            -0.02548610046505928,
+            -0.00689272116869688,
+            -0.0031918229069560766,
+            -0.011716818436980247,
+            -0.002658505691215396,
+            -0.00741795776411891,
+            -0.016807572916150093,
+            -0.010593620128929615,
+            0.01847217045724392,
+            -0.011062292382121086,
+            0.01293698325753212,
+            0.0011060272809118032,
+            -0.017550986260175705,
+            0.007361393887549639,
+            -0.012654163874685764,
+            0.0016403545159846544,
+            -0.009591629728674889,
+            0.0001431775017408654,
+            0.019134776666760445,
+            -0.008767412044107914,
+            0.006581619381904602,
+            -0.004278658889234066,
+            0.011151178739964962,
+            -0.00102168635930866,
+            0.016710607334971428,
+            0.015951033681631088,
+            0.011046131141483784,
+            0.00010163834667764604,
+            -0.015732858330011368,
+            -0.019053971394896507,
+            0.022108424454927444,
+            0.004214014392346144,
+            -0.01046433113515377,
+            0.02404775843024254,
+            0.002684767358005047,
+            0.01992667093873024,
+            -0.011652173474431038,
+            0.003864529775455594,
+            0.024516431614756584,
+            0.010060302913188934,
+            0.008993667550384998,
+            -0.0035776698496192694,
+            -0.0003694332845043391,
+            0.00482409680262208,
+            0.03416462615132332,
+            0.019247904419898987,
+            -0.005927093792706728,
+            -0.024306336417794228,
+            0.018262075260281563,
+            0.0019191340543329716,
+            0.001435310230590403,
+            0.02024989388883114,
+            0.024322498589754105,
+            0.01808430254459381,
+            0.007143219001591206,
+            0.008654284290969372,
+            -0.012452149763703346,
+            0.010747150518000126,
+            -0.0007651284104213119,
+            -0.0055473074316978455,
+            0.0003335757937747985,
+            -0.029591025784611702,
+            0.012258215807378292,
+            -0.00270294863730669,
+            -0.014229873195290565,
+            0.0021817523520439863,
+            -0.013130917213857174,
+            0.022867996245622635,
+            -0.0011575408279895782,
+            -0.01645202934741974,
+            -0.004694807808846235,
+            -0.024063920602202415,
+            0.0028201169334352016,
+            -0.01923174224793911,
+            -0.005369534716010094,
+            -0.012015799060463905,
+            -0.0013918771874159575,
+            -0.022237712517380714,
+            -0.009591629728674889,
+            0.011539045721292496,
+            -0.002353464253246784,
+            -0.007450280245393515,
+            -0.005914973095059395,
+            0.00763613311573863,
+            -0.018569136038422585,
+            -0.0028423385228961706,
+            0.005716999061405659,
+            0.019150936976075172,
+            0.0025191158056259155,
+            0.003535246942192316,
+            0.01871458627283573,
+            0.013963215053081512,
+            -0.03445552662014961,
+            0.003650394966825843,
+            0.00254537770524621,
+            0.00223023584112525,
+            0.025195199996232986,
+            0.007462400943040848,
+            -0.00732099125161767,
+            0.014795512892305851,
+            -0.0003237276105210185,
+            0.01871458627283573,
+            0.002375685842707753,
+            -0.014682385139167309,
+            -0.00716342031955719,
+            0.005672555882483721,
+            -0.007737140171229839,
+            0.021106433123350143,
+            -0.002543357666581869,
+            0.007813905365765095,
+            -0.009995657950639725,
+            -0.011078453622758389,
+            -0.00678363349288702,
+            0.00013863218191545457,
+            -0.006011939607560635,
+            0.00885629840195179,
+            -0.009809805080294609,
+            0.005813966039568186,
+            0.006133148446679115,
+            0.004022100940346718,
+            -0.013938973657786846,
+            0.016056081280112267,
+            -0.003199903527274728,
+            0.008573479019105434,
+            0.02605173923075199,
+            0.00572507968172431,
+            -0.009438099339604378,
+            0.017550986260175705,
+            -8.59191277413629e-05,
+            -0.01128854788839817,
+            -0.02244780771434307,
+            -0.003878670744597912,
+            -0.03907760977745056,
+            0.028378941118717194,
+            0.00023193744709715247,
+            -0.018520653247833252,
+            0.02517903782427311,
+            -0.00816136971116066,
+            -0.008274497464299202,
+            -0.00018143391935154796,
+            -0.0100845443084836,
+            -0.02340131439268589,
+            0.002715069567784667,
+            -0.010569377802312374,
+            -0.006108906585723162,
+            0.011934993788599968,
+            0.030334437265992165,
+            -0.008815895766019821,
+            -0.0008913872297853231,
+            0.006937164347618818,
+            0.003607971826568246,
+            0.01879539154469967,
+            -0.028071880340576172,
+            -0.0032948500011116266,
+            -0.018326720222830772,
+            0.019199419766664505,
+            -0.0007105846307240427,
+            -0.01682373508810997,
+            0.012339022010564804,
+            -0.014949044212698936,
+            -0.02175287902355194,
+            -0.0003398887347429991,
+            -0.0060563827864825726,
+            0.005664475727826357,
+            0.0063594039529562,
+            0.02498510479927063,
+            0.0033756557386368513,
+            0.007963395677506924,
+            -0.02580932155251503,
+            -0.023708375170826912,
+            0.02383766509592533,
+            0.01103805098682642,
+            -0.014932882972061634,
+            -0.0011555206729099154,
+            -0.005106916651129723,
+            -0.007219984196126461,
+            -0.019910510629415512,
+            0.007175541017204523,
+            0.06651920825242996,
+            -0.007361393887549639,
+            0.003096876200288534,
+            -0.015643972903490067,
+            -0.005632153246551752,
+            -0.007098775822669268,
+            -0.01929638721048832,
+            -0.006363444495946169,
+            0.00024822482373565435,
+            0.005733160302042961,
+            -0.025405293330550194,
+            0.004054422955960035,
+            -0.006706868298351765,
+            0.007001808844506741,
+            -0.012137007899582386,
+            0.010270397178828716,
+            -0.015490441583096981,
+            -0.006860399153083563,
+            0.0028686001896858215,
+            -0.009817885234951973,
+            -0.0009388605831190944,
+            -0.017243923619389534,
+            0.001279759337194264,
+            0.016112644225358963,
+            0.003616052446886897,
+            -0.012767291627824306,
+            -0.016460109502077103,
+            -0.01947415992617607,
+            -0.01214508805423975,
+            -0.005070554092526436,
+            -0.008339142426848412,
+            -0.023320509120821953,
+            0.012169329449534416,
+            -0.007741180714219809,
+            0.016856057569384575,
+            0.011054212227463722,
+            0.013429897837340832,
+            0.020573116838932037,
+            -0.003389796707779169,
+            0.009308810345828533,
+            -0.013421816751360893,
+            0.03067382238805294,
+            -0.0022787190973758698,
+            -0.023449797183275223,
+            -0.0007312910747714341,
+            0.00033913119114004076,
+            -0.007931074127554893,
+            -0.021849846467375755,
+            0.007555327378213406,
+            -0.002583760302513838,
+            0.011934993788599968,
+            -0.005959416273981333,
+            0.003995839040726423,
+            0.0015686395345255733,
+            0.006294759456068277,
+            0.004343303386121988,
+            0.017841886729002,
+            -0.0059957788325846195,
+            0.01997515559196472,
+            -0.02815268561244011,
+            -0.01738937385380268,
+            -0.008230054751038551,
+            -0.007490682881325483,
+            0.012613760307431221,
+            0.006044262088835239,
+            -0.0032685883343219757,
+            -0.016492431983351707,
+            -0.010076463222503662,
+            0.008880539797246456,
+            -0.016807572916150093,
+            -0.026601217687129974,
+            -0.0016080322675406933,
+            -0.0325646735727787,
+            0.007110896520316601,
+            0.004214014392346144,
+            -0.005700838286429644,
+            -0.005575589369982481,
+            0.02010444365441799,
+            0.00948658213019371,
+            0.006646263878792524,
+            0.005692757666110992,
+            -0.0017070191679522395,
+            -0.011312790215015411,
+            -0.0009595670271664858,
+            0.0015565187204629183,
+            0.011749139986932278,
+            -0.002377706114202738,
+            -0.01571669802069664,
+            0.015353072434663773,
+            0.009034071117639542,
+            -0.0108198756352067,
+            0.0004469561972655356,
+            0.019700415432453156,
+            0.0039574564434587955,
+            -0.0180196575820446,
+            0.004597840830683708,
+            0.018197430297732353,
+            -0.005979617591947317,
+            0.013163238763809204,
+            0.026552734896540642,
+            -1.8986169379786588e-05,
+            -0.00078280468005687,
+            -0.017599469050765038,
+            -0.01255719643086195,
+            -0.0035574682988226414,
+            0.023611409589648247,
+            0.029251642525196075,
+            0.002240336500108242,
+            -0.027344629168510437,
+            0.009567388333380222,
+            -0.016969185322523117,
+            0.012961224652826786,
+            -0.015231863595545292,
+            0.0021474098321050406,
+            0.02092866227030754,
+            -0.013761200942099094,
+            0.004989748355001211,
+            0.009898691438138485,
+            0.028750646859407425,
+            0.004561478737741709,
+            -0.015264186076819897,
+            0.0006994738359935582,
+            -0.02081553265452385,
+            0.002008020179346204,
+            -0.006395766511559486,
+            0.005733160302042961,
+            -0.011369354091584682,
+            0.00646849162876606,
+            -0.011110776104032993,
+            0.0006439199787564576,
+            0.0012141048209741712,
+            -0.02118724025785923,
+            0.007583609316498041,
+            -0.018003497272729874,
+            0.0037453414406627417,
+            0.0010767352068796754,
+            -0.004048362839967012,
+            -0.023643730208277702,
+            -0.002511035418137908,
+            0.028896097093820572,
+            -0.007805825211107731,
+            0.05788916349411011,
+            -0.006290719378739595,
+            0.0014403605600818992,
+            -0.009034071117639542,
+            0.0063068801537156105,
+            -0.0009706777636893094,
+            -0.009349212981760502,
+            -0.005413977894932032,
+            0.002723150188103318,
+            0.0029130433686077595,
+            0.004492793697863817,
+            0.009914852678775787,
+            0.004836217500269413,
+            0.0005153884994797409,
+            0.02984960377216339,
+            0.0008681556209921837,
+            0.0085977204144001,
+            -0.004484713077545166,
+            0.0022908400278538465,
+            -0.016524754464626312,
+            0.01803581975400448,
+            0.005801845341920853,
+            -0.022609418258070946,
+            -0.016662122681736946,
+            -0.017049990594387054,
+            0.013462220318615437,
+            -0.0031009165104478598,
+            0.013898570090532303,
+            -0.005846288055181503,
+            -0.014553096145391464,
+            0.017664114013314247,
+            0.008557317778468132,
+            0.009405776858329773,
+            -0.0002760017814580351,
+            0.005874569993466139,
+            0.005462461616843939,
+            0.008823975920677185,
+            -0.005102876108139753,
+            -0.005317011382430792,
+            -0.0026766869705170393,
+            0.014981365762650967,
+            -0.011029969900846481,
+            -0.0011454200139269233,
+            -0.016112644225358963,
+            -0.006791714113205671,
+            0.022932641208171844,
+            0.000245573406573385,
+            -0.003708978882059455,
+            0.02650425024330616,
+            0.011692576110363007,
+            -0.00010833006672328338,
+            -0.009696677327156067,
+            -0.0013646052684634924,
+            0.007725019473582506,
+            0.020718567073345184,
+            -0.004028161056339741,
+            -0.008500753901898861,
+            -0.017615629360079765,
+            -0.0011454200139269233,
+            0.030059698969125748,
+            -0.007535126060247421,
+            -0.019894348457455635,
+            0.01284001674503088
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "3457b5c1-3ca3-4a6c-ad56-2f51fc1ca180",
+      "type": "next",
+      "source": {
+        "id": "317d6fdf-f473-4de1-9258-7ec217a9b4ca",
+        "properties": {
+          "page_content": "3. Deepseekmoe Architecture\n\nOn top of the generic MoE architecture outlined in Section 2, we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. As illustrated in Figure 2, our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. Both of these strategies are designed to elevate the level of expert specialization.\n\n3.1. Fine-Grained Expert Segmentation\n\nIn scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts.\n\nIn pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. The finer expert segmentation enables a more flexible and adaptable combination of activated experts. To be specific, on top of a typical MoE architecture shown in Figure 2(a), we segment each expert FFN into smaller experts by reducing the FFN intermediate hidden dimension to 1 times its original size. Since each expert becomes smaller, in response, we also increase the number of activated experts to times to keep the same computation cost, as illustrated in Figure 2(b). With the fine-grained expert segmentation, the output of an MoE layer can be expressed as:\n\n$$\\mathbf{h}{t}^{i}=\\sum{i=1}^{\\text{mN}}\\left(g_{i,t}\\text{FFN}{i}\\left(\\mathbf{u}{t}^{i}\\right)\\right)+\\mathbf{u}{t}^{i},\\tag{6}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\text{Tokr}(s_{i,t}|1\\leqslant j\\leqslant m\\text{N}),m\\text{K}),\\ 0,&\\text{otherwise},\\end{cases}$$ (7) $$s_{i,t}=\\text{Softmax}{i}\\left(\\mathbf{u}{t}^{T}\\mathbf{e}_{t}^{i}\\right),\\tag{8}$$ where the total number of expert parameters is equal to $N$ times the number of parameters in a\n\n$$(6)$$\n\n(7) (8) $\\frac{1}{2}$\n\nstandard FFN, and denotes the total number of fine-grained experts. With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to .\n\nFrom a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. As an illustrative example, we consider the case where = 16. A typical top-2 routing strategy can yield 16 2 = 120 possible combinations. By contrast, if each expert is split into 4 smaller experts, the fine-grained routing strategy can yield 64 8 = 4, 426, 165, 368 potential combinations. The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition.\n\n3.2. Shared Expert Isolation\n\nWith a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. This alleviation of redundancy will contribute to a more parameterefficient model with more specialized experts.\n\nTowards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate experts to serve as shared experts. Regardless of the router module, each token will be deterministically assigned to these shared experts. In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by , as depicted in Figure 2(c). With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows:\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{K_{s}}\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)+\\sum_{i=K_{s}+1}^{mN}\\left(g_{i,t}\\;\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},\\tag{9}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\mathrm{Topk}{s_{j,t}|K_{s}+1\\leqslant j\\leqslant mN},mK-K_{s}},\\ 0,&\\text{otherwise},\\end{cases}$$ (10) $$s_{i,t}=\\mathrm{Softmax}{i}\\left(\\mathbf{u}{t}^{l,T}e_{i}^{l}\\right).\\tag{11}$$ $\\mathbf{h}\\mathbf{M}\\mathbf{E}$ the number of closed-compact is $K_{s}$ the total number of routed currents.\n\nFinally, in DeepSeekMoE, the number of shared expert is , the total number of routed experts is − , and the number of nonzero gates is − .\n\nIt is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022). The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint.\n\n3.3. Load Balance Consideration\n\nAutomatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. Firstly, there is a risk of routing collapse (Shazeer et al., 2017), i.e., the model always selects only a few experts, preventing other experts from sufficient training.\n\nSecondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.\n\nExpert-Level Balance Loss. In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. The computation of the balance loss is as follows:\n\n$$\\begin{array}{l}{{{\\mathcal L}{\\mathrm{ExpBal}}=\\alpha{1}\\sum_{i=1}^{N^{\\prime}}f_{i}P_{i},}}\\ {{f_{i}=\\frac{N^{\\prime}}{K^{T}}\\sum_{t=1}^{T}\\mathds{1}\\,(\\mathrm{Token}\\;t\\;\\mathrm{s}\\mathrm{lects}\\;\\mathrm{Expert}\\;i),}}\\ {{P_{i}=\\frac{1}{T}\\sum_{t=1}^{T}s_{i,t},}}\\end{array}$$\n\n(12) $$\\begin{array}{l}\\small\\text{(13)}\\end{array}$$ = (14) $$\\begin{array}{l}\\small\\text{(14)}\\end{array}$$ .\n\n7 where 1 is a hyper-parameter called expert-level balance factor, ′is equal to ( − ) and ′ is equal to ( − ) for brevity. 1(·) denotes the indicator function.\n\nDevice-Level Balance Loss. In addition to the expert-level balance loss, we introduce a devicelevel balance loss. When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. Instead, our primary objective is to ensure balanced computation across the devices. If we partition all routed experts into groups {E1, E2, ..., E}, and deploy each group on a single device, the device-level balance loss is computed as follows:\n\n$$\\mathcal{L}{\\text{DevBal}}=\\alpha{2}\\sum_{i=1}^{D}f_{i}^{\\prime}P_{i}^{\\prime},$$ $$f_{i}^{\\prime}=\\frac{1}{|\\mathcal{E}{i}|}\\sum{j\\in\\mathcal{E}{i}}f{j},$$ $$P_{i}^{\\prime}=\\sum_{j\\in\\mathcal{E}{i}}P{j},$$\n\n(15) $$\\begin{array}{l}\\small\\text{(16)}\\end{array}$$ = (17) .\n\nwhere 2 is a hyper-parameter called device-level balance factor. In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices.\n\n",
+          "title": "Deepseekmoe Architecture",
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "fine-grained expert segmentation",
+            "shared expert isolation",
+            "load balance consideration"
+          ],
+          "embedding": [
+            -0.004791774787008762,
+            -0.04079068824648857,
+            -0.01656515710055828,
+            -0.01955496519804001,
+            -0.006609901320189238,
+            0.034003015607595444,
+            -0.01574093848466873,
+            0.050455041229724884,
+            -0.04541277140378952,
+            0.02537297084927559,
+            -0.010747150518000126,
+            -0.015199541114270687,
+            0.008557317778468132,
+            -0.01236326340585947,
+            -0.001814086688682437,
+            0.03117481619119644,
+            0.022674063220620155,
+            0.022867996245622635,
+            0.041534099727869034,
+            -0.032096002250909805,
+            0.02517903782427311,
+            -0.04825713112950325,
+            -0.03211216256022453,
+            0.018375203013420105,
+            -0.020395344123244286,
+            0.01574093848466873,
+            0.023368991911411285,
+            0.00403220159932971,
+            -0.06251124292612076,
+            0.021995296701788902,
+            0.029623348265886307,
+            0.015635890886187553,
+            0.006242235656827688,
+            -0.0028645601123571396,
+            -0.034003015607595444,
+            -0.022593257948756218,
+            0.029300125315785408,
+            0.025227520614862442,
+            -0.026358800008893013,
+            0.03578073903918266,
+            0.03626557067036629,
+            0.021866006776690483,
+            -0.0449279360473156,
+            0.007236145436763763,
+            -0.0028686001896858215,
+            0.004626123234629631,
+            -0.015256104990839958,
+            0.008306819945573807,
+            0.01797117479145527,
+            -0.036459505558013916,
+            -0.015344991348683834,
+            -0.02821733057498932,
+            0.012444068677723408,
+            0.003559488570317626,
+            -0.028928419575095177,
+            0.009276487864553928,
+            0.031077850610017776,
+            0.05061665549874306,
+            0.02391847036778927,
+            0.005931134335696697,
+            0.03561912849545479,
+            -0.007898751646280289,
+            -0.036136284470558167,
+            0.013381414115428925,
+            0.02915467508137226,
+            -0.021332690492272377,
+            0.01580558344721794,
+            -0.009599709883332253,
+            -0.006294759456068277,
+            0.009599709883332253,
+            -0.05798612907528877,
+            0.030916238203644753,
+            -0.034746427088975906,
+            0.000505792791955173,
+            -0.006007899530231953,
+            0.00270294863730669,
+            0.03772007301449776,
+            0.03183742240071297,
+            0.013324850238859653,
+            -0.028508231043815613,
+            -0.001757522695697844,
+            0.011183501221239567,
+            -0.014771271497011185,
+            0.0015050050569698215,
+            0.01929638721048832,
+            -0.04334414750337601,
+            -0.0023595248349010944,
+            -0.06988071650266647,
+            0.048612672835588455,
+            -0.0075997705571353436,
+            -0.029332447797060013,
+            0.003993819002062082,
+            0.004078664816915989,
+            -0.0046018813736736774,
+            -0.0031049568206071854,
+            0.059246696531772614,
+            -0.00016565156693104655,
+            -0.017599469050765038,
+            -0.03529590368270874,
+            0.0033534341491758823,
+            -0.0022443768102675676,
+            0.024177048355340958,
+            -0.029865765944123268,
+            0.009147198870778084,
+            0.028863774612545967,
+            -5.586952465819195e-05,
+            0.01226629689335823,
+            0.02770017459988594,
+            -0.006545256823301315,
+            0.09709606319665909,
+            -0.008993667550384998,
+            -0.005486703012138605,
+            -0.007801784668117762,
+            0.002919103717431426,
+            -0.014003617689013481,
+            -0.02333666943013668,
+            -0.03920689597725868,
+            -0.06483844667673111,
+            0.02320738136768341,
+            -0.044055234640836716,
+            -0.0021756920032203197,
+            0.0017140896525233984,
+            0.02535681053996086,
+            -0.022754868492484093,
+            -0.06299608200788498,
+            0.04172803461551666,
+            0.0025979015044867992,
+            0.003816046519204974,
+            0.013276367448270321,
+            0.013324850238859653,
+            0.014674304984509945,
+            -0.0171954408288002,
+            -0.006525055505335331,
+            -0.04761068522930145,
+            0.021736718714237213,
+            0.036459505558013916,
+            0.04476632550358772,
+            0.05426906794309616,
+            -0.0033837363589555025,
+            0.01847217045724392,
+            0.022932641208171844,
+            -0.035522159188985825,
+            -0.010456250049173832,
+            0.002995869144797325,
+            -0.023562924936413765,
+            0.0130581920966506,
+            0.00017979255062527955,
+            0.05426906794309616,
+            -0.011749139986932278,
+            0.026843633502721786,
+            -0.017793402075767517,
+            0.06380413472652435,
+            0.0016544954851269722,
+            0.021461978554725647,
+            0.0027574924752116203,
+            0.011143097653985023,
+            -0.022609418258070946,
+            -0.019118614494800568,
+            -0.0020393324084579945,
+            0.005167520605027676,
+            0.007381595205515623,
+            -0.03267780318856239,
+            -0.01442380715161562,
+            0.04104926809668541,
+            0.0012221853248775005,
+            0.05110148712992668,
+            0.007939154282212257,
+            0.008072483353316784,
+            -0.08242175728082657,
+            0.025227520614862442,
+            -0.006226074881851673,
+            0.022124584764242172,
+            0.022221552208065987,
+            0.040952298790216446,
+            -0.012985466979444027,
+            -0.022237712517380714,
+            0.026277994737029076,
+            0.0034786828327924013,
+            -0.022738708183169365,
+            0.0346817821264267,
+            -0.031982872635126114,
+            -0.009979496710002422,
+            -0.008250256069004536,
+            -0.02094482257962227,
+            0.03316263481974602,
+            0.02181752398610115,
+            0.02005596086382866,
+            0.0104966526851058,
+            -0.004832177422940731,
+            -0.007809865288436413,
+            0.031223300844430923,
+            0.02233467996120453,
+            0.011741059832274914,
+            0.006807875353842974,
+            0.006339202634990215,
+            -0.02702140621840954,
+            0.010456250049173832,
+            -0.008452270179986954,
+            0.0032544471323490143,
+            0.01527226623147726,
+            -0.00561599200591445,
+            -0.017550986260175705,
+            -0.0024928541388362646,
+            -0.019894348457455635,
+            0.022867996245622635,
+            -0.01039968617260456,
+            0.023433636873960495,
+            0.011312790215015411,
+            -0.004403907340019941,
+            -0.042794667184352875,
+            0.006254356820136309,
+            -0.01075523067265749,
+            -0.0016181330429390073,
+            -0.04415220394730568,
+            -0.01727624610066414,
+            -0.003593830857425928,
+            0.00556750874966383,
+            0.033615145832300186,
+            0.024500271305441856,
+            -0.0036200927570462227,
+            -0.00013534944446291775,
+            -0.007656334433704615,
+            0.008395706303417683,
+            0.0056079113855957985,
+            -0.0066947476007044315,
+            0.00873508956283331,
+            0.0026787070091813803,
+            0.01797117479145527,
+            0.018197430297732353,
+            -0.03406766057014465,
+            0.010957244783639908,
+            0.026471927762031555,
+            0.0039574564434587955,
+            0.019457997754216194,
+            0.02151046134531498,
+            -0.035522159188985825,
+            -0.013098594732582569,
+            -0.00875933188945055,
+            0.0058543686755001545,
+            -0.010302719660103321,
+            -0.020847855135798454,
+            0.0041029066778719425,
+            -0.056014470756053925,
+            -0.005272568203508854,
+            0.00509075541049242,
+            0.010165349580347538,
+            -0.008371464908123016,
+            0.05213579908013344,
+            0.00807652436196804,
+            0.007874510250985622,
+            -0.017130795866250992,
+            -0.010205752216279507,
+            0.028540553525090218,
+            -0.003670596284791827,
+            -0.03355050086975098,
+            0.03626557067036629,
+            0.01229861844331026,
+            -0.034003015607595444,
+            -0.004674606490880251,
+            -0.02656889520585537,
+            0.04421684890985489,
+            -0.020459989085793495,
+            -0.018245914950966835,
+            0.044055234640836716,
+            0.0020302417688071728,
+            -0.016985345631837845,
+            0.057598263025283813,
+            0.029122352600097656,
+            0.0038483687676489353,
+            -0.00885629840195179,
+            -0.004844298120588064,
+            -0.04082300886511803,
+            -0.015288427472114563,
+            -0.0186661034822464,
+            0.011280467733740807,
+            -0.016839895397424698,
+            -0.0046584452502429485,
+            -0.008104805834591389,
+            0.02079937234520912,
+            0.02902538701891899,
+            0.02327202446758747,
+            0.012629921548068523,
+            0.02144581824541092,
+            0.004185732454061508,
+            0.002262558089569211,
+            -0.006379605270922184,
+            -0.033744435757398605,
+            -0.02320738136768341,
+            -0.035263583064079285,
+            0.006092745345085859,
+            -0.01562781073153019,
+            -0.0018423686269670725,
+            -0.020540794357657433,
+            -0.01864994317293167,
+            0.05585286021232605,
+            0.006173551082611084,
+            -0.016161128878593445,
+            0.009737079963088036,
+            0.019393354654312134,
+            -0.031918227672576904,
+            -0.023934630677103996,
+            0.0034180786460638046,
+            0.009672435000538826,
+            0.007486642803996801,
+            -0.019619610160589218,
+            -0.0017534823855385184,
+            -0.04282699152827263,
+            0.030576854944229126,
+            -0.02176903933286667,
+            -0.010698666796088219,
+            0.0290900319814682,
+            0.008201772347092628,
+            0.006771512795239687,
+            -0.04340879246592522,
+            -0.015450038947165012,
+            0.021995296701788902,
+            0.020912500098347664,
+            0.03093240037560463,
+            -0.024710364639759064,
+            -0.022237712517380714,
+            0.016548994928598404,
+            0.007874510250985622,
+            -0.04638243839144707,
+            0.002288819756358862,
+            -0.009510824456810951,
+            -0.013147078454494476,
+            0.050455041229724884,
+            0.04243912175297737,
+            -0.03555448353290558,
+            -0.0016373243415728211,
+            -0.015094494447112083,
+            -0.014819755218923092,
+            -0.03726756200194359,
+            -0.013599589467048645,
+            -0.03694434091448784,
+            -0.014294518157839775,
+            -0.01682373508810997,
+            -0.008452270179986954,
+            -0.013648073188960552,
+            -0.018197430297732353,
+            -0.0077896639704704285,
+            0.006824036594480276,
+            -0.032047517597675323,
+            0.008137128315865993,
+            0.004391786642372608,
+            0.007894711568951607,
+            -0.0403381772339344,
+            -0.006444249767810106,
+            0.023320509120821953,
+            0.03144955635070801,
+            0.001097946660593152,
+            0.012759210541844368,
+            0.01808430254459381,
+            -0.0037130191922187805,
+            0.007979556918144226,
+            -0.052523668855428696,
+            -0.006383645813912153,
+            0.005078634712845087,
+            0.0022282155696302652,
+            0.000984818791039288,
+            0.021930651739239693,
+            -0.048160161823034286,
+            -0.038107942789793015,
+            -0.005874569993466139,
+            -0.04366736859083176,
+            0.01483591552823782,
+            0.004767532926052809,
+            0.0033695953898131847,
+            -0.005712958984076977,
+            -0.006573539227247238,
+            -0.023175058886408806,
+            0.022916480898857117,
+            0.008727009408175945,
+            -0.008670445531606674,
+            0.007894711568951607,
+            0.031013205647468567,
+            0.0419866107404232,
+            -0.02086401730775833,
+            -0.025130555033683777,
+            0.054301392287015915,
+            -0.011361273005604744,
+            0.01543387770652771,
+            0.028815291821956635,
+            0.014496532268822193,
+            0.012945064343512058,
+            0.032984863966703415,
+            0.010642102919518948,
+            0.007413917686790228,
+            -0.023255864158272743,
+            0.04308556765317917,
+            -0.01871458627283573,
+            -0.028233490884304047,
+            -0.019958993420004845,
+            0.012007718905806541,
+            0.006981607526540756,
+            -0.02600325644016266,
+            -0.029558703303337097,
+            -0.027183018624782562,
+            -0.0005520035629160702,
+            -0.008613881655037403,
+            -0.016379304230213165,
+            0.020702404901385307,
+            -0.0050059095956385136,
+            0.0006893731188029051,
+            -0.036847371608018875,
+            -0.004957426339387894,
+            -0.0010787553619593382,
+            -0.006965446285903454,
+            0.0363302156329155,
+            0.0006999789038673043,
+            0.003561508608981967,
+            -0.04024121165275574,
+            -0.022738708183169365,
+            -0.004181691911071539,
+            0.022318517789244652,
+            0.056079115718603134,
+            -0.004957426339387894,
+            0.039627086371183395,
+            -0.014528854750096798,
+            0.012427907437086105,
+            -0.025082072243094444,
+            0.01738937385380268,
+            0.0272638238966465,
+            0.00945426058024168,
+            0.006266477517783642,
+            0.015046010725200176,
+            0.014924801886081696,
+            -0.019781220704317093,
+            0.02669818513095379,
+            -0.0033271722495555878,
+            0.01675909012556076,
+            0.0015726798446848989,
+            0.01154712587594986,
+            -0.02574467845261097,
+            -0.01490056049078703,
+            -0.022544773295521736,
+            -0.024629559367895126,
+            0.054301392287015915,
+            -0.02600325644016266,
+            0.013745039701461792,
+            -0.0016757070552557707,
+            0.018504492938518524,
+            -0.03180510178208351,
+            -0.01599951647222042,
+            -0.04043514281511307,
+            -0.04473400488495827,
+            -0.04686727374792099,
+            -0.019765060395002365,
+            -0.00938961561769247,
+            -5.8962868934031576e-05,
+            -0.03995031118392944,
+            0.021607428789138794,
+            -0.05627305060625076,
+            -0.008929023519158363,
+            0.03772007301449776,
+            -0.022415485233068466,
+            0.0011908732121810317,
+            -0.021365011110901833,
+            0.009421938098967075,
+            -0.04900054261088371,
+            0.002274678787216544,
+            0.027344629168510437,
+            0.0555296391248703,
+            -0.0013706657337024808,
+            -0.022981123998761177,
+            0.004375625401735306,
+            -0.013405656442046165,
+            0.0018797412049025297,
+            0.0010565337724983692,
+            0.009114876389503479,
+            0.02815268561244011,
+            -0.017114635556936264,
+            0.053105469793081284,
+            0.012177410535514355,
+            0.030463727191090584,
+            0.026455767452716827,
+            -0.007078574039041996,
+            -0.06548488885164261,
+            -0.0006020020227879286,
+            0.021720556542277336,
+            0.010165349580347538,
+            0.0018544894410297275,
+            -0.004945305176079273,
+            0.004094826057553291,
+            0.0049049025401473045,
+            -0.0015716697089374065,
+            0.01625809445977211,
+            0.012015799060463905,
+            0.00763613311573863,
+            -0.004678646568208933,
+            0.013300608843564987,
+            0.015134897083044052,
+            -0.035457514226436615,
+            0.022221552208065987,
+            0.011805704794824123,
+            -0.048612672835588455,
+            -0.008727009408175945,
+            -0.02155894599854946,
+            0.02181752398610115,
+            -0.003492823801934719,
+            -0.016338901594281197,
+            -0.004569558892399073,
+            0.033938370645046234,
+            0.01973273791372776,
+            -0.0040645236149430275,
+            -0.014650062657892704,
+            0.013332931324839592,
+            0.04250376671552658,
+            -0.022997286170721054,
+            -0.030560694634914398,
+            0.011741059832274914,
+            -0.00397159717977047,
+            -0.0015454079257324338,
+            0.0017241904279217124,
+            -0.016613639891147614,
+            0.04470168054103851,
+            -0.009316890500485897,
+            0.035134293138980865,
+            0.004115027375519276,
+            0.014633901417255402,
+            -0.019021648913621902,
+            0.009179521352052689,
+            -0.018068142235279083,
+            0.02689211815595627,
+            -0.0157813411206007,
+            0.044119879603385925,
+            0.026972923427820206,
+            0.012646082788705826,
+            0.024694204330444336,
+            -0.035134293138980865,
+            -0.012702646665275097,
+            0.014221793040633202,
+            0.01884387619793415,
+            -0.010900680907070637,
+            -0.010213833302259445,
+            -0.014609660021960735,
+            -0.008751250803470612,
+            0.03772007301449776,
+            -0.02162358909845352,
+            -0.024112403392791748,
+            0.001966607291251421,
+            -0.0013797563733533025,
+            0.0006626062677241862,
+            -0.01588638871908188,
+            -0.016468189656734467,
+            0.017631791532039642,
+            0.050519686192274094,
+            -0.02244780771434307,
+            0.022415485233068466,
+            0.040144242346286774,
+            -0.0048038954846560955,
+            -0.020670082420110703,
+            -0.025728516280651093,
+            -0.009098715148866177,
+            0.00039493758231401443,
+            -0.019247904419898987,
+            0.016379304230213165,
+            -0.008565397933125496,
+            -0.00945426058024168,
+            -0.04612386226654053,
+            -0.0010247165337204933,
+            0.02516287751495838,
+            0.0033312125597149134,
+            0.015579327940940857,
+            -0.016985345631837845,
+            -0.005684677045792341,
+            -0.004876620601862669,
+            0.024257853627204895,
+            -0.016920700669288635,
+            0.007672495674341917,
+            0.0024605318903923035,
+            0.0003421613946557045,
+            0.0009858288103714585,
+            -0.003064553951844573,
+            0.00032120241667144,
+            0.007405837066471577,
+            0.027086051180958748,
+            -0.008355303667485714,
+            -0.023449797183275223,
+            0.007983597926795483,
+            0.02079937234520912,
+            -0.02598709426820278,
+            0.004917023237794638,
+            0.049873244017362595,
+            0.023869987577199936,
+            -0.013518784195184708,
+            0.005963456351310015,
+            -0.0010444129584357142,
+            0.017049990594387054,
+            0.0040604835376143456,
+            0.01505409087985754,
+            0.03494035825133324,
+            0.0016686365706846118,
+            -0.03193439170718193,
+            -0.00795531552284956,
+            0.0017322709318250418,
+            0.010569377802312374,
+            0.016662122681736946,
+            0.0008716908632777631,
+            -0.04573599249124527,
+            0.0013757160631939769,
+            0.005797804798930883,
+            0.010884519666433334,
+            -0.014625821262598038,
+            -0.02214074693620205,
+            0.022948801517486572,
+            0.026213349774479866,
+            0.001036332338117063,
+            0.0035170656628906727,
+            0.005914973095059395,
+            0.017066150903701782,
+            0.02307809144258499,
+            0.022092262282967567,
+            -0.014367243275046349,
+            0.019183259457349777,
+            -0.004755412228405476,
+            -0.012355182319879532,
+            -0.001959536923095584,
+            -0.020459989085793495,
+            0.021348850801587105,
+            0.044378459453582764,
+            0.02094482257962227,
+            0.018423685804009438,
+            0.017825724557042122,
+            0.010270397178828716,
+            0.026229511946439743,
+            -0.02732846885919571,
+            -0.03600699454545975,
+            0.0025938611943274736,
+            -0.0003156470484100282,
+            -0.025502260774374008,
+            -0.002603961853310466,
+            -0.018439847975969315,
+            -0.039756376296281815,
+            0.009623952209949493,
+            -0.0026160827837884426,
+            0.010472411289811134,
+            0.03442320227622986,
+            -0.01077947299927473,
+            -0.03224145248532295,
+            -0.0004121087840758264,
+            0.033421214669942856,
+            0.01797117479145527,
+            0.039497796446084976,
+            -0.01877923123538494,
+            0.0062462761998176575,
+            -0.0014656123239547014,
+            0.007991678081452847,
+            0.01372887846082449,
+            -0.007676536217331886,
+            -0.013389495201408863,
+            0.030124343931674957,
+            0.023482119664549828,
+            0.04777229577302933,
+            -0.006484652869403362,
+            -0.043473437428474426,
+            -0.012233974412083626,
+            -0.01531266886740923,
+            0.015062171965837479,
+            0.008605800569057465,
+            0.0050947959534823895,
+            0.0008646203787066042,
+            0.008880539797246456,
+            -0.007583609316498041,
+            0.006254356820136309,
+            0.007098775822669268,
+            -0.01879539154469967,
+            0.03332424536347389,
+            -0.008096725679934025,
+            0.0302374716848135,
+            -0.019942833110690117,
+            -0.02548610046505928,
+            -0.01864994317293167,
+            0.0024241693317890167,
+            0.0023150816559791565,
+            -0.026229511946439743,
+            0.03049604967236519,
+            0.03471410274505615,
+            -0.0018666103715077043,
+            0.008331061340868473,
+            -0.010432008653879166,
+            0.003490803763270378,
+            0.002375685842707753,
+            -0.006213953718543053,
+            0.007935114204883575,
+            0.0003030211664736271,
+            -0.026035577058792114,
+            0.035198938101530075,
+            -0.03542519360780716,
+            0.0018777211662381887,
+            -0.020718567073345184,
+            0.021979134529829025,
+            0.008452270179986954,
+            0.008387625217437744,
+            0.011676414869725704,
+            -0.023304346948862076,
+            -0.0030059698037803173,
+            0.009777482599020004,
+            0.015603569336235523,
+            0.008710848167538643,
+            -0.015749020501971245,
+            0.012476391158998013,
+            0.0019060031045228243,
+            0.020960982888936996,
+            0.03030211664736271,
+            0.00044241087744012475,
+            0.008250256069004536,
+            -0.01638738438487053,
+            -0.019215581938624382,
+            -0.0061533497646451,
+            0.010310799814760685,
+            0.030512209981679916,
+            -0.012169329449534416,
+            0.04192196577787399,
+            0.037849362939596176,
+            -0.003840288147330284,
+            -0.006965446285903454,
+            0.010254235938191414,
+            -0.0551740936934948,
+            0.05071362107992172,
+            -0.003490803763270378,
+            -0.0007388665690086782,
+            0.020088283345103264,
+            -0.0442814938724041,
+            -0.023175058886408806,
+            0.005098836030811071,
+            0.010827955789864063,
+            0.01732472889125347,
+            0.009842127561569214,
+            0.002264578128233552,
+            0.014504612423479557,
+            -0.03136875107884407,
+            -0.039562441408634186,
+            0.02763552963733673,
+            0.00970475748181343,
+            0.004123107995837927,
+            0.012815774418413639,
+            0.02018524892628193,
+            -0.0017716636648401618,
+            0.008840137161314487,
+            0.01309051364660263,
+            -0.0021150875836610794,
+            0.01732472889125347,
+            0.02600325644016266,
+            -0.010706747882068157,
+            0.026035577058792114,
+            0.011215822771191597,
+            0.03781704232096672,
+            -0.0050543928518891335,
+            -0.00488470122218132,
+            0.005474582314491272,
+            -0.021785201504826546,
+            0.03195055201649666,
+            0.0031392991077154875,
+            -0.03846348449587822,
+            0.01682373508810997,
+            0.022367002442479134,
+            -0.017793402075767517,
+            0.021284205839037895,
+            0.0014858137583360076,
+            0.020912500098347664,
+            0.026956763118505478,
+            0.0040039196610450745,
+            0.0020585237070918083,
+            0.028896097093820572,
+            0.01886003650724888,
+            -0.008387625217437744,
+            0.050261110067367554,
+            0.010674425400793552,
+            -0.007664415054023266,
+            0.01511873584240675,
+            0.0013605649583041668,
+            0.02624567225575447,
+            -0.047028884291648865,
+            8.831804007058963e-05,
+            -0.012710727751255035,
+            -6.256124470382929e-05,
+            0.011821865104138851,
+            -0.01173297967761755,
+            0.003684737253934145,
+            -0.033744435757398605,
+            0.023643730208277702,
+            0.005102876108139753,
+            0.022560935467481613,
+            -0.008428028784692287,
+            0.006916963029652834,
+            0.016662122681736946,
+            -0.0009782533161342144,
+            0.007042211946099997,
+            0.009777482599020004,
+            -0.006193752400577068,
+            0.017486341297626495,
+            -0.008395706303417683,
+            0.016807572916150093,
+            0.046641018241643906,
+            -0.01636314205825329,
+            -0.0038039255887269974,
+            0.039691731333732605,
+            -0.009114876389503479,
+            -0.025195199996232986,
+            0.020459989085793495,
+            -0.007632093038409948,
+            -0.0006838177214376628,
+            0.04230983555316925,
+            -0.043796658515930176,
+            -0.025825483724474907,
+            0.004016040358692408,
+            -0.023562924936413765,
+            -0.019635770469903946,
+            0.009365374222397804,
+            0.016314659267663956,
+            0.003751402022317052,
+            -0.04421684890985489,
+            0.005947295110672712,
+            -0.000348221801687032,
+            0.01258143875747919,
+            -0.00016312638763338327,
+            -0.01688838005065918,
+            -0.036394860595464706,
+            0.011757221072912216,
+            -0.004436229821294546,
+            -0.025890128687024117,
+            4.217044624965638e-05,
+            -0.010294638574123383,
+            -0.05242669954895973,
+            -0.026665862649679184,
+            0.040758367627859116,
+            0.0012009738711640239,
+            0.033485859632492065,
+            -5.463850175146945e-05,
+            0.03262931853532791,
+            -0.00822601467370987,
+            -0.02075088955461979,
+            0.01517529971897602,
+            -0.0073371524922549725,
+            0.016330819576978683,
+            0.010019899345934391,
+            0.001137339393608272,
+            -0.00023951297043822706,
+            0.03490803763270378,
+            -0.033421214669942856,
+            0.004030181560665369,
+            0.0038726103957742453,
+            -0.012112765572965145,
+            -0.045186515897512436,
+            0.010213833302259445,
+            -0.00844419002532959,
+            -0.009955255314707756,
+            0.022059939801692963,
+            -0.011821865104138851,
+            0.001617122907191515,
+            0.025276005268096924,
+            0.006484652869403362,
+            -0.006488692946732044,
+            -0.0363302156329155,
+            -0.013462220318615437,
+            0.0090098287910223,
+            -0.008508834056556225,
+            -0.0008686606306582689,
+            -0.010318880900740623,
+            -0.04176035523414612,
+            0.014884399250149727,
+            -0.025502260774374008,
+            -0.006116987206041813,
+            0.014811674132943153,
+            0.019878188148140907,
+            -0.010092624463140965,
+            -0.04205125570297241,
+            -0.007038171403110027,
+            -0.03290405869483948,
+            -0.006929083727300167,
+            0.003482723142951727,
+            -0.01044008880853653,
+            0.008904782123863697,
+            0.016112644225358963,
+            -0.0186661034822464,
+            0.010165349580347538,
+            -0.02663354016840458,
+            0.021203400567173958,
+            -0.014254115521907806,
+            0.01593487337231636,
+            0.0021130675449967384,
+            0.003232225775718689,
+            -0.021962974220514297,
+            0.025954771786928177,
+            -0.0299142487347126,
+            -0.004816016182303429,
+            -0.0315626822412014,
+            -0.02624567225575447,
+            -0.030140504240989685,
+            0.006848277989774942,
+            0.0010555237531661987,
+            -0.04091997817158699,
+            -0.005284688901156187,
+            0.015320749953389168,
+            0.0020373123697936535,
+            -0.028928419575095177,
+            -0.020136766135692596,
+            0.0009827986359596252,
+            0.007349273189902306,
+            0.0017928752349689603,
+            -0.031724296510219574,
+            -0.008678525686264038,
+            -0.032483868300914764,
+            0.020201411098241806,
+            -0.0017706536455079913,
+            0.027344629168510437,
+            0.0011262285988777876,
+            0.003840288147330284,
+            -0.02029837667942047,
+            0.002234275918453932,
+            0.0036907976027578115,
+            0.022302357479929924,
+            -0.03484339267015457,
+            0.016548994928598404,
+            0.035392872989177704,
+            0.022221552208065987,
+            -0.049356088042259216,
+            0.0016342941671609879,
+            0.01385008729994297,
+            -0.011021889746189117,
+            0.01871458627283573,
+            -0.005733160302042961,
+            0.025082072243094444,
+            -0.01277537178248167,
+            0.0005641243769787252,
+            0.005684677045792341,
+            0.006327081937342882,
+            -0.0022564975079149008,
+            0.004335222765803337,
+            -0.024435626342892647,
+            0.02320738136768341,
+            0.02687595598399639,
+            0.007551287300884724,
+            0.010634022764861584,
+            -0.0031069768592715263,
+            -0.012387504801154137,
+            0.011805704794824123,
+            -0.026488089933991432,
+            -0.0005691747646778822,
+            0.011021889746189117,
+            -0.022544773295521736,
+            -0.006815955974161625,
+            0.03959476575255394,
+            0.024952782317996025,
+            -0.05313779041171074,
+            0.017680274322628975,
+            0.018423685804009438,
+            -0.005058433394879103,
+            0.01747017912566662,
+            -0.0403381772339344,
+            -0.00844419002532959,
+            -0.0254376158118248,
+            0.0005666495417244732,
+            0.016807572916150093,
+            0.011005728505551815,
+            -0.015377313829958439,
+            0.024936622008681297,
+            0.0003035262052435428,
+            0.03201519697904587,
+            -0.002511035418137908,
+            -0.020411504432559013,
+            -0.02650425024330616,
+            -0.020605439320206642,
+            -0.00743007892742753,
+            0.009858288802206516,
+            -0.0005403877585195005,
+            -0.03161116689443588,
+            -0.03746149688959122,
+            0.007385635748505592,
+            0.00307667488232255,
+            -0.04790158569812775,
+            -0.013171319849789143,
+            -0.02845974639058113,
+            -0.03060917742550373,
+            -0.009163360111415386,
+            -0.009058312512934208,
+            -0.03775239735841751,
+            0.0006009919452480972,
+            0.003133238758891821,
+            0.021672073751688004,
+            -0.010868359357118607,
+            0.013720798306167126,
+            -0.02333666943013668,
+            -0.007838147692382336,
+            0.0016706566093489528,
+            0.03985334187746048,
+            -0.009615871123969555,
+            -0.035069648176431656,
+            0.011021889746189117,
+            0.019377192482352257,
+            -0.015902550891041756,
+            0.012217813171446323,
+            0.005082674790173769,
+            -0.006678586360067129,
+            0.017793402075767517,
+            -0.013955134898424149,
+            -0.03762310743331909,
+            0.004549357574433088,
+            -0.013147078454494476,
+            -0.01442380715161562,
+            -0.017179278656840324,
+            0.010367363691329956,
+            0.02214074693620205,
+            -0.021930651739239693,
+            0.008880539797246456,
+            0.04486329108476639,
+            -0.02227003499865532,
+            -0.029364770278334618,
+            0.01053705532103777,
+            -0.018375203013420105,
+            0.01401169877499342,
+            -0.004214014392346144,
+            0.03062533773481846,
+            -0.009163360111415386,
+            0.01053705532103777,
+            -0.015425797551870346,
+            0.010092624463140965,
+            -0.03384140133857727,
+            -0.019005486741662025,
+            -0.0034079779870808125,
+            -0.007672495674341917,
+            -0.03778471797704697,
+            -0.03616860508918762,
+            0.022916480898857117,
+            0.019150936976075172,
+            -0.009906771592795849,
+            0.0058947717770934105,
+            -0.010836036875844002,
+            9.204267553286627e-05,
+            0.026520412415266037,
+            0.016613639891147614,
+            -0.008177530951797962,
+            0.0012292558094486594,
+            -0.0047594523057341576,
+            0.0065088942646980286,
+            0.019312549382448196,
+            -0.06493541598320007,
+            -0.029203159734606743,
+            0.02504974976181984,
+            0.01789036951959133,
+            0.012330940924584866,
+            0.015207622200250626,
+            0.02535681053996086,
+            -0.02383766509592533,
+            -0.020912500098347664,
+            0.037849362939596176,
+            -0.003381716087460518,
+            0.01024615578353405,
+            -0.007947235368192196,
+            0.0019302447326481342,
+            0.00030453625367954373,
+            0.002240336500108242,
+            -0.0010797653812915087,
+            0.005757402162998915,
+            0.036394860595464706,
+            0.02239932492375374,
+            0.013122836127877235,
+            0.015078333206474781,
+            -0.0013625851133838296,
+            -0.004973587114363909,
+            -0.007951275445520878,
+            -0.0037796839606016874,
+            0.025405293330550194,
+            -0.017292408272624016,
+            0.0017150997882708907,
+            -0.026277994737029076,
+            0.02492045983672142,
+            0.025130555033683777,
+            0.01708231307566166,
+            -0.005369534716010094,
+            -0.007034131325781345,
+            0.013421816751360893,
+            0.0034766627941280603,
+            -0.009276487864553928,
+            0.016807572916150093,
+            0.0068442379124462605,
+            -0.020379183813929558,
+            -0.011789543554186821,
+            0.008205813355743885,
+            -0.007207863498479128,
+            -0.022948801517486572,
+            0.031158655881881714,
+            0.012831935659050941,
+            0.012662244029343128,
+            0.004795814864337444,
+            -0.0005444280104711652,
+            0.011522884480655193,
+            0.03484339267015457,
+            -0.036588795483112335,
+            -0.01771259680390358,
+            0.005434179678559303,
+            -0.02138117328286171,
+            0.00929264910519123,
+            -0.014908640645444393,
+            0.006193752400577068,
+            0.015983356162905693,
+            0.07408261299133301,
+            -0.027247661724686623,
+            -4.744175021187402e-05,
+            -0.016274256631731987,
+            -0.017098473384976387,
+            0.018326720222830772,
+            -0.004230175167322159,
+            0.0051109567284584045,
+            0.02765168994665146,
+            0.008395706303417683,
+            0.03636253997683525,
+            -0.007013929542154074,
+            -0.005373575259000063,
+            0.023385152220726013,
+            -0.0018645902164280415,
+            -0.010852198116481304,
+            -0.006723029538989067,
+            0.001137339393608272,
+            0.012815774418413639,
+            0.00790279172360897,
+            0.03258083388209343,
+            0.017567146569490433,
+            0.0012019838904961944,
+            -0.00039493758231401443,
+            -0.0044564311392605305,
+            0.028556713834404945,
+            0.012848096899688244,
+            0.0017423717072233558,
+            -0.01953880488872528,
+            -0.003040312323719263,
+            0.027473919093608856,
+            -0.021672073751688004,
+            0.009922932833433151,
+            0.015328830108046532,
+            0.005034191533923149,
+            0.04809551686048508,
+            -0.03351818025112152,
+            -0.008298739790916443,
+            0.012064282782375813,
+            0.0018625700613483787,
+            -0.008068443275988102,
+            0.01480359397828579,
+            0.006581619381904602,
+            -0.01056129764765501,
+            -0.00926840677857399,
+            0.012613760307431221,
+            0.013575348071753979,
+            -0.0063917264342308044,
+            -0.00901790987700224,
+            0.0066543444991111755,
+            -0.010771391913294792,
+            0.019958993420004845,
+            -0.00035781748010776937,
+            0.008254296146333218,
+            0.004193812608718872,
+            0.008718929253518581,
+            -0.0007681586430408061,
+            -0.04111390933394432,
+            -0.00789067056030035,
+            0.016177289187908173,
+            -0.003333232831209898,
+            0.021348850801587105,
+            -0.001368645578622818,
+            -0.008565397933125496,
+            -0.014876319095492363,
+            -0.004876620601862669,
+            -0.0036019114777445793,
+            -0.00519984308630228,
+            0.0005641243769787252,
+            -0.00471500912681222,
+            -0.01738937385380268,
+            0.011126937344670296,
+            -0.028766809031367302,
+            -0.007680576294660568,
+            -0.0033473738003522158,
+            -0.014246034435927868,
+            -0.004525116179138422,
+            -0.007094735279679298,
+            0.005102876108139753,
+            -0.007595730479806662,
+            0.019134776666760445,
+            0.0050543928518891335,
+            -0.029057709500193596,
+            -0.008710848167538643,
+            -0.027296146377921104,
+            0.017631791532039642,
+            -0.027215341106057167,
+            0.01192691270262003,
+            0.025340650230646133,
+            0.002365585183724761,
+            -0.0012100645108148456,
+            0.004131188616156578,
+            -0.029817281290888786,
+            0.0036685762461274862,
+            0.010609780438244343,
+            -0.009728998877108097,
+            0.00901790987700224,
+            0.023627569898962975,
+            -0.01816510781645775,
+            -0.025841644033789635,
+            -0.03207983821630478,
+            0.01565205305814743,
+            0.011110776104032993,
+            -0.014157148078083992,
+            -0.011692576110363007,
+            -0.0012080443557351828,
+            0.004929144401103258,
+            -0.01378544233739376,
+            -0.00010245903104078025,
+            -0.00785834901034832,
+            -0.010326961055397987,
+            -0.008088644593954086,
+            -0.008985587395727634,
+            -0.002513055456802249,
+            -0.0038907916750758886,
+            0.008355303667485714,
+            0.006933124270290136,
+            -0.03494035825133324,
+            -0.025841644033789635,
+            0.012856177985668182,
+            0.007288668770343065,
+            -0.01331677008420229,
+            0.009567388333380222,
+            0.014989446848630905,
+            0.004965506494045258,
+            -0.002377706114202738,
+            0.009890610352158546,
+            -0.018456008285284042,
+            0.0127188079059124,
+            0.0046665258705616,
+            0.004084725398570299,
+            0.016920700669288635,
+            0.015151058323681355,
+            0.012339022010564804,
+            0.01293698325753212,
+            -0.03290405869483948,
+            0.0007151299505494535,
+            0.027991075068712234,
+            0.048580352216959,
+            0.01636314205825329,
+            -0.022754868492484093,
+            0.021962974220514297,
+            0.04625314846634865,
+            0.02024989388883114,
+            -0.00060301210032776,
+            -0.016985345631837845,
+            0.0050543928518891335,
+            0.026261834427714348,
+            0.010351202450692654,
+            0.00418977253139019,
+            -0.006415967829525471,
+            -0.013551106676459312,
+            0.014173309318721294,
+            0.009462340734899044,
+            0.00678363349288702,
+            -0.037914007902145386,
+            0.020023638382554054,
+            -0.023579087108373642,
+            0.020653922110795975,
+            0.0027756737545132637,
+            -0.012427907437086105,
+            0.016314659267663956,
+            0.022156907245516777,
+            -0.0048725800588727,
+            0.020960982888936996,
+            0.0008767411927692592,
+            0.011296628974378109,
+            0.00837954506278038,
+            -0.011934993788599968,
+            0.003896852023899555,
+            -0.005975577048957348,
+            0.003793824929744005,
+            0.006189712323248386,
+            0.02302960865199566,
+            0.00397159717977047,
+            0.0186661034822464,
+            0.017696436494588852,
+            0.0010045150993391871,
+            0.008104805834591389,
+            -0.019748898223042488,
+            -0.00022486694797407836,
+            -0.009696677327156067,
+            0.007349273189902306,
+            0.009462340734899044,
+            0.0072927093133330345,
+            0.02479117177426815,
+            -0.012597599998116493,
+            -0.010585539042949677,
+            -0.01816510781645775,
+            0.014561176300048828,
+            -0.014149067923426628,
+            0.01634698174893856,
+            0.017761079594492912,
+            0.0012363262940198183,
+            0.008565397933125496,
+            0.02446794882416725,
+            0.002816076623275876,
+            -0.01656515710055828,
+            0.011078453622758389,
+            -0.023740697652101517,
+            -0.017308568581938744,
+            0.013478381559252739,
+            -0.01315515860915184,
+            -0.010270397178828716,
+            0.020524634048342705,
+            -0.015967195853590965,
+            0.0017565126763656735,
+            0.011151178739964962,
+            0.03062533773481846,
+            -0.017373213544487953,
+            0.022318517789244652,
+            -0.01075523067265749,
+            -0.023110413923859596,
+            -0.002626183442771435,
+            0.016662122681736946,
+            0.0269406009465456,
+            1.085036728909472e-06,
+            -0.005729120224714279,
+            -0.005207923706620932,
+            0.02309425175189972,
+            -0.008573479019105434,
+            0.022867996245622635,
+            0.007034131325781345,
+            0.012419827282428741,
+            0.00020618064445443451,
+            -0.018003497272729874,
+            0.017437858507037163,
+            0.0020544833969324827,
+            0.02214074693620205,
+            0.014407645910978317,
+            -0.015749020501971245,
+            0.02904154732823372,
+            -0.0030241510830819607,
+            0.028556713834404945,
+            0.0007206853479146957,
+            -0.009583549574017525,
+            0.02845974639058113,
+            -0.0016807573847472668,
+            -0.0004017555620521307,
+            0.006165470462292433,
+            0.01505409087985754,
+            0.016443947330117226,
+            0.00392311392351985,
+            0.020605439320206642,
+            0.016306579113006592,
+            -0.00015100554446689785,
+            0.014318759553134441,
+            -0.00879165343940258,
+            -0.022819513455033302,
+            0.022043779492378235,
+            -0.005034191533923149,
+            -0.01745401881635189,
+            0.01734089106321335,
+            -0.007696737535297871,
+            -0.004058463498950005,
+            -0.0007075543981045485,
+            0.023320509120821953,
+            0.016209611669182777,
+            -0.007575529161840677,
+            0.003365555079653859,
+            -0.004282698966562748,
+            -0.001307031256146729,
+            0.02789410762488842,
+            0.008613881655037403,
+            -0.0034968641120940447,
+            0.01511873584240675,
+            0.005236205644905567,
+            0.01024615578353405,
+            -0.00025580034707672894,
+            -0.007466441486030817,
+            -0.01955496519804001,
+            -0.0035796898882836103,
+            0.004763492848724127,
+            0.005494783632457256,
+            0.0005767502589151263,
+            -0.0060563827864825726,
+            -0.020734727382659912,
+            -0.04476632550358772,
+            -0.00019128210260532796,
+            0.0014726828085258603,
+            0.025922449305653572,
+            0.04169571027159691,
+            -0.005902852397412062,
+            -0.004747331608086824,
+            -0.012452149763703346,
+            0.0014009678270667791,
+            -0.008533075451850891,
+            -0.019700415432453156,
+            -0.006064463406801224,
+            -0.013817764818668365,
+            -0.01916709914803505,
+            0.01615304872393608,
+            -0.012654163874685764,
+            0.006048302166163921,
+            0.002583760302513838,
+            0.019215581938624382,
+            -0.016742929816246033,
+            0.02744159661233425,
+            -0.01634698174893856,
+            0.004747331608086824,
+            0.001796915428712964,
+            -0.002872640499845147,
+            -0.03322727978229523,
+            0.02068624459207058,
+            -0.003304950660094619,
+            -0.036685761064291,
+            -0.0037716033402830362,
+            -0.007219984196126461,
+            -0.009955255314707756,
+            0.03429391607642174,
+            0.0014171289512887597,
+            -0.007805825211107731,
+            -0.012096604332327843,
+            0.01119158137589693,
+            0.011328951455652714,
+            0.008197732269763947,
+            -0.006020020227879286,
+            -0.015029849484562874,
+            0.007405837066471577,
+            0.03267780318856239,
+            -0.019765060395002365,
+            -0.015207622200250626,
+            -0.0014858137583360076,
+            0.011692576110363007,
+            0.002739311195909977,
+            0.005927093792706728,
+            -0.030156666412949562,
+            -0.014302598312497139,
+            -0.009607790969312191,
+            -0.01135319285094738,
+            -0.022674063220620155,
+            -0.007979556918144226,
+            0.00907447375357151,
+            -0.02125188335776329,
+            0.02138117328286171,
+            0.029881926253437996,
+            -0.027991075068712234,
+            0.0340353362262249,
+            -0.011345111764967442,
+            0.009599709883332253,
+            -0.01034312229603529,
+            0.0007449270342476666,
+            -0.011102695018053055,
+            0.0007964406395331025,
+            0.011304709129035473,
+            -0.03581305965781212,
+            -0.04237448051571846,
+            -0.008945184759795666,
+            0.0010959265055134892,
+            -0.01795501448214054,
+            -0.006925043649971485,
+            0.03959476575255394,
+            0.0100845443084836,
+            -0.0031837422866374254,
+            -0.016225773841142654,
+            0.01864994317293167,
+            0.02233467996120453,
+            0.03262931853532791,
+            0.01555508654564619,
+            0.00450895493850112,
+            0.0008186621707864106,
+            0.011789543554186821,
+            -0.027490079402923584,
+            0.014795512892305851,
+            -0.0013696555979549885,
+            0.02676282823085785,
+            -0.0015746999997645617,
+            -0.0003719584783539176,
+            0.019748898223042488,
+            -0.024257853627204895,
+            -0.0024847735185176134,
+            0.00732099125161767,
+            -0.014722787775099277,
+            0.004965506494045258,
+            0.019021648913621902,
+            0.007757341489195824,
+            -0.036588795483112335,
+            0.0041655306704342365,
+            -0.010771391913294792,
+            0.01986202597618103,
+            0.00917144026607275,
+            0.010423927567899227,
+            -0.0031291984487324953,
+            -0.011102695018053055,
+            -0.011910751461982727,
+            0.007817945443093777,
+            -0.0025979015044867992,
+            -0.008003799244761467,
+            0.03041524440050125,
+            -0.003412018297240138,
+            -0.0032382861245423555,
+            0.00837954506278038,
+            0.006032141391187906,
+            0.006036181468516588,
+            0.0085977204144001,
+            -0.009163360111415386,
+            -0.024435626342892647,
+            -0.01490056049078703,
+            -0.018746908754110336,
+            0.005353373941034079,
+            0.010262317024171352,
+            -0.003717059502378106,
+            -0.02416088618338108,
+            0.005114997271448374,
+            -0.007203822955489159,
+            0.036782726645469666,
+            -0.008347222581505775,
+            -0.035457514226436615,
+            -0.02377302013337612,
+            -0.04667333886027336,
+            0.009211842902004719,
+            6.436042895074934e-05,
+            -0.021979134529829025,
+            -0.0039614965207874775,
+            -0.011490561999380589,
+            -0.028007235378026962,
+            0.012056201696395874,
+            0.019651932641863823,
+            -0.026019416749477386,
+            0.021461978554725647,
+            0.00763613311573863,
+            -0.015102574601769447,
+            0.00567659642547369,
+            -0.008743170648813248,
+            -0.0269406009465456,
+            0.002250437159091234,
+            -0.006278598215430975,
+            0.017114635556936264,
+            -0.01173297967761755,
+            -0.0001382534101139754,
+            -0.020912500098347664,
+            0.01860145851969719,
+            0.021219560876488686,
+            -0.011070373468101025,
+            0.0014433907344937325,
+            -0.0037210998125374317,
+            -0.02568003349006176,
+            0.005555388052016497,
+            0.0007545226835645735,
+            -0.004614002071321011,
+            -0.0004100886289961636,
+            0.006791714113205671,
+            -0.009122957475483418,
+            0.00086159014608711,
+            -0.0051513598300516605,
+            0.008274497464299202,
+            0.0017393414163962007,
+            -0.01034312229603529,
+            -0.002296900376677513,
+            -0.003636253997683525,
+            0.0024484109599143267,
+            0.019134776666760445,
+            -0.015183379873633385,
+            0.007777543272823095,
+            -0.03157884627580643,
+            0.006832117214798927,
+            0.009623952209949493,
+            -0.0006408897461369634,
+            -0.02813652530312538,
+            0.0010327971540391445,
+            -0.0007090694853104651,
+            0.005531146191060543,
+            -0.026342639699578285,
+            0.029267802834510803,
+            0.0216397512704134,
+            0.011458240449428558,
+            0.015862148255109787,
+            0.001985798589885235,
+            -0.013712717220187187,
+            -0.016678284853696823,
+            0.003785744309425354,
+            -0.016936862841248512,
+            -0.009284568019211292,
+            0.00036589804221875966,
+            0.027490079402923584,
+            -0.024257853627204895,
+            -0.009720918722450733,
+            -0.00028054710128344595,
+            0.0005009949672967196,
+            -0.005680636502802372,
+            -0.010019899345934391,
+            0.004836217500269413,
+            0.013971295207738876,
+            0.017179278656840324,
+            -0.010698666796088219,
+            0.07395332306623459,
+            0.002173671731725335,
+            -0.006787674035876989,
+            0.0013817765284329653,
+            -0.011918832547962666,
+            -0.019910510629415512,
+            0.02086401730775833,
+            -0.022124584764242172,
+            0.013736959546804428,
+            0.007288668770343065,
+            -0.01567629538476467,
+            -0.00758764985948801,
+            -0.015991436317563057,
+            0.024904299527406693,
+            0.0017201501177623868,
+            0.0034867634531110525,
+            0.00689272116869688,
+            0.020734727382659912,
+            0.027296146377921104,
+            0.012274377048015594,
+            0.0041069467552006245,
+            -0.0027978953439742327,
+            -0.004840258043259382,
+            0.016726767644286156,
+            -0.01910245418548584,
+            0.002250437159091234,
+            0.005886691156774759,
+            0.022415485233068466,
+            -0.004969547037035227,
+            0.034261591732501984,
+            0.002882741391658783,
+            0.003143339417874813,
+            -0.006415967829525471,
+            0.003878670744597912,
+            0.020589277148246765,
+            -0.002850419143214822,
+            0.01664596237242222,
+            -0.006367484573274851,
+            -0.0033069709315896034,
+            0.0004800360184162855,
+            0.0010196662042289972,
+            0.002692847978323698,
+            0.0072967493906617165,
+            -0.019344870001077652,
+            -0.0002853449259418994,
+            0.00017448968719691038,
+            0.007264427375048399,
+            0.0029898087959736586,
+            0.0034180786460638046,
+            -0.004375625401735306,
+            0.00046437993296422064,
+            0.006613941863179207,
+            0.010003738105297089,
+            0.013074353337287903,
+            -0.0013696555979549885,
+            -0.005171561148017645,
+            0.006876560393720865,
+            -0.008169450797140598,
+            0.0013272326905280352,
+            0.025793161243200302,
+            0.0013504642993211746,
+            0.0062018330208957195,
+            -0.024871977046132088,
+            0.001361575094051659,
+            0.008686606772243977,
+            0.007595730479806662,
+            0.010674425400793552,
+            4.075003380421549e-05,
+            -0.0029372849967330694,
+            0.0012050141813233495,
+            -0.014019778929650784,
+            0.009632032364606857,
+            0.021785201504826546,
+            -0.0037675630301237106,
+            0.009042151272296906,
+            0.01050473377108574,
+            0.0023494239430874586,
+            -0.029687993228435516,
+            -0.015110654756426811,
+            0.013639992102980614,
+            0.001351474318653345,
+            0.008775493130087852,
+            4.4506232370622456e-05,
+            0.009543146006762981,
+            -0.02214074693620205,
+            -0.011175420135259628,
+            -0.0049089426174759865,
+            0.012807694263756275,
+            0.018229752779006958,
+            -0.021720556542277336,
+            0.013882409781217575,
+            0.006504854187369347,
+            0.004331182222813368,
+            -0.027279984205961227,
+            -0.005454380996525288,
+            0.013187481090426445,
+            -0.01128854788839817,
+            0.002939305268228054,
+            -0.01072290912270546,
+            0.005684677045792341,
+            0.003739281091839075,
+            0.02307809144258499,
+            -0.008840137161314487,
+            0.007761382032185793,
+            0.0037352407816797495,
+            -0.023045768961310387,
+            0.011078453622758389,
+            0.020411504432559013,
+            -0.00525236688554287,
+            -0.0013262225547805429,
+            -0.00577356293797493,
+            -0.00828257855027914,
+            0.015797503292560577,
+            -0.0021433697547763586,
+            -0.011660254560410976,
+            0.01658131740987301,
+            -0.006908882409334183,
+            -0.011369354091584682,
+            -0.022609418258070946,
+            -0.006177591159939766,
+            -0.006658385042101145,
+            -0.0026968882884830236,
+            0.009753241203725338,
+            -0.011442079208791256,
+            0.0034322196152061224,
+            -0.0017565126763656735,
+            -0.0078462278470397,
+            -0.004605921451002359,
+            -0.03358282521367073,
+            -0.018423685804009438,
+            -0.00811692699790001,
+            -0.02012060582637787,
+            0.002057513687759638,
+            0.003706958843395114,
+            -0.00716342031955719,
+            0.017922692000865936,
+            -0.010569377802312374,
+            -0.007575529161840677,
+            0.003414038335904479,
+            0.012912741862237453,
+            -0.00810884591192007,
+            -0.009890610352158546,
+            -0.019651932641863823,
+            0.0003696858184412122,
+            0.01591063104569912,
+            0.0183428805321455,
+            0.02537297084927559,
+            -0.016144966706633568,
+            0.0032948500011116266,
+            -0.024177048355340958,
+            0.008125007152557373,
+            -0.030382921919226646,
+            -0.012751130387187004,
+            -0.004840258043259382,
+            -0.0007378565496765077,
+            -0.008888620883226395,
+            -0.012040040455758572,
+            -0.04124319925904274,
+            -0.009785563684999943,
+            -0.015102574601769447,
+            -0.02322354167699814,
+            0.015134897083044052,
+            -0.005818006116896868,
+            0.017049990594387054,
+            -0.009567388333380222,
+            0.016856057569384575,
+            0.016904540359973907,
+            0.014553096145391464,
+            -0.007341192569583654,
+            0.00732099125161767,
+            -0.004387746565043926,
+            0.014407645910978317,
+            0.004123107995837927,
+            -0.0038928117137402296,
+            0.014973285607993603,
+            0.009995657950639725,
+            0.0050947959534823895,
+            0.00045478425454348326,
+            0.0017191400984302163,
+            0.010941083543002605,
+            0.0007924003293737769,
+            0.015635890886187553,
+            0.007975516840815544,
+            -0.0016837875591591,
+            0.034003015607595444,
+            -0.021785201504826546,
+            -0.010634022764861584,
+            0.010060302913188934,
+            0.01258143875747919,
+            -0.01873074844479561,
+            -0.0017837845953181386,
+            0.010512813925743103,
+            -0.0013201622059568763,
+            -0.030964722856879234,
+            0.013171319849789143,
+            0.011361273005604744,
+            0.0029999094549566507,
+            -0.005114997271448374,
+            -0.016791412606835365,
+            -0.005062473472207785,
+            0.007062413264065981,
+            0.020314538851380348,
+            -0.018245914950966835,
+            0.009939094074070454,
+            -0.007401796989142895,
+            -0.0061452691443264484,
+            -0.02689211815595627,
+            0.012532955035567284,
+            0.022948801517486572,
+            0.009195681661367416,
+            -0.059052761644124985,
+            -0.010375444777309895,
+            0.017858047038316727,
+            -0.0064927334897220135,
+            -0.03755846247076988,
+            -0.002294880338013172,
+            -0.01517529971897602,
+            -0.02307809144258499,
+            0.008767412044107914,
+            0.015732858330011368,
+            -0.006832117214798927,
+            0.015724778175354004,
+            0.006609901320189238,
+            -0.007482602261006832,
+            0.0003903922624886036,
+            -0.004973587114363909,
+            -0.03458481654524803,
+            -0.006678586360067129,
+            0.016548994928598404,
+            -0.03561912849545479,
+            -0.00721190357580781,
+            -0.00827045738697052,
+            -0.014553096145391464,
+            0.006052342709153891,
+            0.011248145252466202,
+            0.03972405195236206,
+            0.007405837066471577,
+            -0.01229861844331026,
+            -0.014003617689013481,
+            -0.014528854750096798,
+            -0.013389495201408863,
+            0.0017100494587793946,
+            -0.01176530122756958,
+            0.011240065097808838,
+            0.008331061340868473,
+            -0.009373454377055168,
+            0.023417474702000618,
+            -0.003403937676921487,
+            0.009559307247400284,
+            -3.494212796795182e-05,
+            -0.004710969049483538,
+            -0.005830127280205488,
+            -0.014722787775099277,
+            0.0008090665214695036,
+            0.003913013264536858,
+            -0.0025191158056259155,
+            -0.0005232165567576885,
+            0.0038221068680286407,
+            0.02706989087164402,
+            -0.010472411289811134,
+            -0.005442259833216667,
+            -0.010286558419466019,
+            -0.028362780809402466,
+            0.0003393836959730834,
+            0.02845974639058113,
+            0.030948560684919357,
+            -0.015684375539422035,
+            -0.01309051364660263,
+            -0.01789036951959133,
+            0.0067351502366364,
+            -0.0038604894652962685,
+            -0.0033392931800335646,
+            0.029397092759609222,
+            0.020977145060896873,
+            0.004137248732149601,
+            0.01732472889125347,
+            -0.009357293136417866,
+            0.006169511005282402,
+            0.028443586081266403,
+            -0.0013302628649398685,
+            0.009939094074070454,
+            0.010149188339710236,
+            0.013963215053081512,
+            -0.00986636895686388,
+            -0.000141536133014597,
+            0.005288729444146156,
+            -0.004646324552595615,
+            0.014084423892199993,
+            0.010666345246136189,
+            -0.008864378556609154,
+            -0.021203400567173958,
+            -0.005066513549536467,
+            -0.020896339789032936,
+            -0.0052483263425529,
+            0.006112946663051844,
+            -0.010836036875844002,
+            -0.001267638523131609,
+            -0.029397092759609222,
+            -0.004383706022053957,
+            0.002311041345819831,
+            -0.013211722485721111,
+            0.01027847733348608,
+            -0.011078453622758389,
+            0.0025251763872802258,
+            0.006290719378739595,
+            -0.02088017761707306,
+            -0.003717059502378106,
+            -0.007377555128186941,
+            0.003652415005490184,
+            0.005325092002749443,
+            0.00614930922165513,
+            -0.0016534854657948017,
+            0.005874569993466139,
+            0.020395344123244286,
+            0.0020787252578884363,
+            0.01486015785485506,
+            -0.02079937234520912,
+            -0.028508231043815613,
+            -0.029073869809508324,
+            0.015611650422215462,
+            -0.0031251581385731697,
+            0.0031837422866374254,
+            -0.0033291925210505724,
+            -0.010844117030501366,
+            0.004828137345612049,
+            0.003913013264536858,
+            0.0014322800561785698,
+            -0.002852439181879163,
+            0.00493318447843194,
+            -0.0031716213561594486,
+            0.010585539042949677,
+            0.006036181468516588,
+            -0.02036302164196968,
+            -0.006024060770869255,
+            -0.009623952209949493,
+            0.0073533132672309875,
+            -0.018375203013420105,
+            -0.009357293136417866,
+            0.00031968732946552336,
+            -0.0018777211662381887,
+            0.0033493938390165567,
+            -0.04812784120440483,
+            -0.019522642716765404,
+            -0.014334920793771744,
+            0.008516915142536163,
+            -0.0019443858182057738,
+            -0.03128794580698013,
+            -0.020039798691868782,
+            0.008133088238537312,
+            0.01580558344721794,
+            0.002410028362646699,
+            0.008920943364501,
+            0.010189591906964779,
+            0.00128480966668576,
+            -0.006436169613152742,
+            -0.0020251914393156767,
+            0.02081553265452385,
+            0.01714695803821087,
+            -0.030593017116189003,
+            -0.019150936976075172,
+            0.010642102919518948,
+            -0.008331061340868473,
+            -0.013615750707685947,
+            -0.01002798043191433,
+            0.007199782878160477,
+            0.0006338192615658045,
+            0.021041790023446083,
+            -0.0015514683909714222,
+            -0.019183259457349777,
+            0.004084725398570299,
+            0.00578568410128355,
+            -0.018892358988523483,
+            0.013219802640378475,
+            -0.007975516840815544,
+            -0.008743170648813248,
+            0.0058988118544220924,
+            -0.007264427375048399,
+            0.0032362660858780146,
+            -0.005183681845664978,
+            0.02580932155251503,
+            -0.020039798691868782,
+            0.0011969335610046983,
+            0.005535186734050512,
+            -0.014092504046857357,
+            0.016969185322523117,
+            -0.008605800569057465,
+            -0.0047594523057341576,
+            0.005013990215957165,
+            0.0007782593602314591,
+            0.009575468488037586,
+            0.009971416555345058,
+            0.008238134905695915,
+            -0.02023373357951641,
+            -0.0012837996473535895,
+            0.005381655879318714,
+            0.007357353810220957,
+            -0.008702768012881279,
+            0.03004353865981102,
+            -0.0054018571972846985,
+            -0.022722546011209488,
+            -0.0009408806799910963,
+            -0.008165409788489342,
+            0.0004906417452730238,
+            -0.00450491439551115,
+            0.007276548072695732,
+            0.015635890886187553,
+            0.006169511005282402,
+            -0.017599469050765038,
+            -0.012831935659050941,
+            0.003323131939396262,
+            0.005147319287061691,
+            0.012767291627824306,
+            0.015167218632996082,
+            -0.011555206961929798,
+            0.010068383067846298,
+            -0.020152926445007324,
+            0.0213165283203125,
+            -0.009753241203725338,
+            0.0038019055500626564,
+            -0.005591750610619783,
+            0.003121117828413844,
+            0.008670445531606674,
+            0.0175186637789011,
+            0.0019575166516005993,
+            -0.0012433967785909772,
+            -0.006133148446679115,
+            0.0350373275578022,
+            -0.00604022154584527,
+            0.016936862841248512,
+            -0.020734727382659912,
+            -0.010060302913188934,
+            -0.024063920602202415,
+            0.03662111610174179,
+            -0.004117047414183617,
+            -0.004589760676026344,
+            0.012007718905806541,
+            0.021930651739239693,
+            -0.012896580621600151,
+            0.009753241203725338,
+            0.003682717215269804,
+            -0.006230114959180355,
+            0.005591750610619783,
+            0.0013767260825261474,
+            0.0008519944967702031,
+            0.02079937234520912,
+            0.017163118347525597,
+            0.008141168393194675,
+            0.007805825211107731,
+            0.011377434246242046,
+            -0.0005098331021144986,
+            -0.012104685418307781,
+            0.01182994619011879,
+            -0.011999637819826603,
+            -0.004094826057553291,
+            -0.0032302055042237043,
+            -0.014415726996958256,
+            -0.009947174228727818,
+            -0.00828257855027914,
+            0.012128926813602448,
+            -0.0035453476011753082,
+            -0.01966809295117855,
+            0.0037251401226967573,
+            0.020266054198145866,
+            0.003032231703400612,
+            0.0059957788325846195,
+            0.014052101410925388,
+            0.01886003650724888,
+            0.01986202597618103,
+            0.0070462520234286785,
+            -0.02941325306892395,
+            -0.0009024980245158076,
+            -0.007070493884384632,
+            -0.006036181468516588,
+            0.009219923987984657,
+            0.0011949134059250355,
+            0.00810884591192007,
+            -0.0035776698496192694,
+            0.012799614109098911,
+            0.03713827207684517,
+            0.013583428226411343,
+            -0.0011353192385286093,
+            0.002395887393504381,
+            -0.002739311195909977,
+            -0.01986202597618103,
+            0.018116625025868416,
+            -0.01623385399580002,
+            -0.027183018624782562,
+            0.021268045529723167,
+            -0.0002911528281401843,
+            -0.006185671780258417,
+            -0.021962974220514297,
+            -0.006884640548378229,
+            0.016080323606729507,
+            0.018116625025868416,
+            -0.0008767411927692592,
+            -0.0018676203908398747,
+            -0.0033190918620675802,
+            0.0031473797280341387,
+            -0.006161430384963751,
+            0.005583669990301132,
+            -0.0020221611484885216,
+            -0.023708375170826912,
+            -0.006815955974161625,
+            -0.011296628974378109,
+            0.006803835276514292,
+            0.0033069709315896034,
+            0.013696555979549885,
+            -0.004117047414183617,
+            0.020346861332654953,
+            0.005325092002749443,
+            -0.004234215710312128,
+            0.007013929542154074,
+            -0.009858288802206516,
+            -0.0030504129827022552,
+            -0.01997515559196472,
+            0.015223783440887928,
+            0.0055028642527759075,
+            0.0032847493421286345,
+            -0.001361575094051659,
+            0.02910619229078293,
+            -0.0005600841250270605,
+            0.006864439230412245,
+            -0.003783724270761013,
+            0.011248145252466202,
+            -0.002197913359850645,
+            -0.008751250803470612,
+            -0.03183742240071297,
+            -0.01039968617260456,
+            -0.02682747319340706,
+            0.023320509120821953,
+            -0.009219923987984657,
+            0.0012353162746876478,
+            0.008589639328420162,
+            0.003672616556286812,
+            -0.004315021447837353,
+            0.010924923233687878,
+            -0.013591509312391281,
+            -0.004234215710312128,
+            0.012896580621600151,
+            -0.0004992273752577603,
+            0.009720918722450733,
+            -0.0073937163688242435,
+            -0.003852409077808261,
+            -0.011369354091584682,
+            -0.008500753901898861,
+            -0.03010818175971508,
+            -0.013308688998222351,
+            -0.01452077366411686,
+            -0.013551106676459312,
+            -0.024500271305441856,
+            -0.0008459340897388756,
+            0.004197853151708841,
+            -0.003761502681300044,
+            -0.010900680907070637,
+            0.017486341297626495,
+            0.01416522916406393,
+            0.02770017459988594,
+            -0.026714345440268517,
+            0.0006358394166454673,
+            -0.007648254279047251,
+            0.018520653247833252,
+            -0.01072290912270546,
+            0.0050947959534823895,
+            0.0015726798446848989,
+            0.01677525043487549,
+            0.0033837363589555025,
+            0.0012464270694181323,
+            0.004517035558819771,
+            0.0018858016701415181,
+            -0.01027847733348608,
+            0.009179521352052689,
+            0.0008439139346592128,
+            -0.011644093319773674,
+            -0.003921093884855509,
+            0.016872217878699303,
+            0.0201690886169672,
+            0.01293698325753212,
+            -0.007219984196126461,
+            0.0034403002355247736,
+            -0.009381535463035107,
+            -0.011846107430756092,
+            -0.011789543554186821,
+            0.008314901031553745,
+            -0.025259843096137047,
+            0.01814894750714302,
+            0.008937103673815727,
+            -0.0008227024227380753,
+            0.007785623427480459,
+            0.0023514442145824432,
+            0.015967195853590965,
+            0.005523065570741892,
+            -0.019393354654312134,
+            -0.014448048546910286,
+            0.0027978953439742327,
+            0.012104685418307781,
+            0.0010252215433865786,
+            -0.0023817464243620634,
+            0.00710281589999795,
+            0.0002638809382915497,
+            -0.011474400758743286,
+            -0.017534824088215828,
+            0.0003237276105210185,
+            0.00938961561769247,
+            0.02290031872689724,
+            0.005757402162998915,
+            0.013446059077978134,
+            0.0022282155696302652,
+            -0.0057493215426802635,
+            -0.007143219001591206,
+            -0.0161045640707016,
+            0.015959113836288452,
+            0.0038928117137402296,
+            -0.025453777983784676,
+            0.024888137355446815,
+            0.002057513687759638,
+            0.01688838005065918,
+            -0.004686727188527584,
+            -0.005838207434862852,
+            0.023756857961416245,
+            0.027780979871749878,
+            0.0043109809048473835,
+            0.024839654564857483,
+            -0.013494541868567467,
+            0.0072523062117397785,
+            0.02516287751495838,
+            0.020459989085793495,
+            -0.009688596241176128,
+            0.0012908701319247484,
+            0.011967315338551998,
+            0.001570659689605236,
+            -0.0005828107241541147,
+            0.009971416555345058,
+            -0.010593620128929615,
+            0.010690586641430855,
+            -0.02175287902355194,
+            0.017437858507037163,
+            -0.0014777331380173564,
+            -0.005183681845664978,
+            -0.006395766511559486,
+            0.0050947959534823895,
+            0.006011939607560635,
+            0.011102695018053055,
+            -0.009203762747347355,
+            -0.021542783826589584,
+            -0.0015292468015104532,
+            -0.003304950660094619,
+            -0.012395585887134075,
+            -0.028120363131165504,
+            0.012177410535514355,
+            -0.013575348071753979,
+            -0.008112886920571327,
+            0.009882530197501183,
+            -0.01721160113811493,
+            0.00241204840131104,
+            -0.02335282973945141,
+            0.010108785703778267,
+            -0.0216397512704134,
+            0.0052564069628715515,
+            0.011854187585413456,
+            -0.005846288055181503,
+            0.004282698966562748,
+            0.006165470462292433,
+            0.010415847413241863,
+            8.724484359845519e-05,
+            -0.013195561245083809,
+            -0.007450280245393515,
+            0.028605196624994278,
+            0.006290719378739595,
+            -0.003737261053174734,
+            -0.01821359246969223,
+            -0.0008454290218651295,
+            -0.028492068871855736,
+            0.018310558050870895,
+            -0.00041185625013895333,
+            0.0008888620650395751,
+            -0.019651932641863823,
+            -0.011636012233793736,
+            -0.0008120966958813369,
+            0.014391484670341015,
+            -0.00019607994181569666,
+            0.003793824929744005,
+            0.0317404568195343,
+            -0.013599589467048645,
+            0.001599951763637364,
+            -6.699608547933167e-06,
+            -0.01303394977003336,
+            0.02005596086382866,
+            -0.005757402162998915,
+            0.006925043649971485,
+            -0.0007333112298510969,
+            -0.008165409788489342,
+            -0.014666223898530006,
+            0.022108424454927444,
+            -0.0033190918620675802,
+            -0.012258215807378292,
+            -0.006763432174921036,
+            0.002161551034078002,
+            0.013551106676459312,
+            -0.012379424646496773,
+            -0.008621961809694767,
+            0.010924923233687878,
+            0.00488470122218132,
+            -0.011110776104032993,
+            -0.010044141672551632,
+            -0.013478381559252739,
+            0.0037433214019984007,
+            -0.03494035825133324,
+            -0.006929083727300167,
+            0.04586528241634369,
+            0.0056038713082671165,
+            -0.008395706303417683,
+            -0.0058503285981714725,
+            0.0009530015522614121,
+            0.006973526906222105,
+            -0.01847217045724392,
+            -0.007619971875101328,
+            -0.027780979871749878,
+            0.005983657669275999,
+            0.002139329444617033,
+            0.03243538364768028,
+            0.0003406462783459574,
+            -0.018488330766558647,
+            0.003549387911334634,
+            -0.021300368010997772,
+            -0.0074421996250748634,
+            -0.018391363322734833,
+            -0.016209611669182777,
+            -0.008920943364501,
+            0.016904540359973907,
+            -0.015167218632996082,
+            0.02663354016840458,
+            0.016597479581832886,
+            0.007603811100125313,
+            0.003117077751085162,
+            -0.0037998852785676718,
+            -0.006415967829525471,
+            0.006432129070162773,
+            -0.0019383253529667854,
+            0.004953385796397924,
+            0.011296628974378109,
+            -0.01075523067265749,
+            0.009922932833433151,
+            0.013688475824892521,
+            -0.007624012418091297,
+            0.01840752549469471,
+            -0.00837954506278038,
+            -0.020459989085793495,
+            0.01452077366411686,
+            -0.014706626534461975,
+            0.00152621662709862,
+            0.0037574623711407185,
+            -0.00832298118621111,
+            0.00472713029012084,
+            0.018989326432347298,
+            0.0073977564461529255,
+            -0.00572507968172431,
+            0.017373213544487953,
+            0.020670082420110703,
+            0.012629921548068523,
+            -0.01050473377108574,
+            0.009591629728674889,
+            0.014439968392252922,
+            -0.0001660303387325257,
+            -0.0003701908572111279,
+            0.004407947883009911,
+            -0.014407645910978317,
+            -0.0006307890289463103,
+            0.003212024224922061,
+            -0.011094614863395691,
+            -0.0115713682025671,
+            0.016839895397424698,
+            0.005244286265224218,
+            -0.004517035558819771,
+            0.011700657196342945,
+            -0.01284001674503088,
+            -0.012751130387187004,
+            -0.009825966320931911,
+            0.0066987876780331135,
+            -0.040758367627859116,
+            0.004840258043259382,
+            -0.006464451551437378,
+            0.018100464716553688,
+            -0.014456129632890224,
+            0.005236205644905567,
+            -0.005919013172388077,
+            -0.010949164628982544,
+            0.015070252120494843,
+            -0.0009762331610545516,
+            -0.002218114910647273,
+            0.03188590705394745,
+            0.011934993788599968,
+            0.01077947299927473,
+            0.004646324552595615,
+            -0.016371222212910652,
+            0.034261591732501984,
+            -0.009882530197501183,
+            0.00012966780923306942,
+            -0.012856177985668182,
+            -0.0017342910869047046,
+            0.03186974674463272,
+            0.029332447797060013,
+            -0.005046312231570482,
+            0.004480673000216484,
+            -0.00795531552284956,
+            0.01363191194832325,
+            0.015991436317563057,
+            0.008133088238537312,
+            0.00599173828959465,
+            0.0032605077140033245,
+            0.004383706022053957,
+            0.0011009768350049853,
+            0.038495808839797974,
+            -0.012613760307431221,
+            -0.011943073943257332,
+            -0.013955134898424149,
+            -0.016662122681736946,
+            -0.022738708183169365,
+            0.02138117328286171,
+            -0.00747856218367815,
+            -0.014916721731424332,
+            -0.007769462652504444,
+            0.018989326432347298,
+            0.00705433264374733,
+            0.006714948918670416,
+            -0.0012928902870044112,
+            0.005539226811379194,
+            -0.024451786652207375,
+            -0.00499378889799118,
+            -0.01738937385380268,
+            0.002729210536926985,
+            -0.0007913902518339455,
+            0.015951033681631088,
+            0.018456008285284042,
+            0.017437858507037163,
+            0.019005486741662025,
+            -0.007426038384437561,
+            0.006169511005282402,
+            -0.0055069043301045895,
+            -0.01226629689335823,
+            0.019199419766664505,
+            0.02086401730775833,
+            -0.01404402032494545,
+            -0.004274618346244097,
+            -0.016710607334971428,
+            -0.021171078085899353,
+            0.002983748447149992,
+            0.009543146006762981,
+            0.010884519666433334,
+            -0.013219802640378475,
+            0.002816076623275876,
+            0.015264186076819897,
+            -0.007042211946099997,
+            0.002769613405689597,
+            0.0048483386635780334,
+            0.017373213544487953,
+            0.01224205456674099,
+            0.0031675812788307667,
+            0.01803581975400448,
+            -0.018957003951072693,
+            -0.02776481956243515,
+            0.006589700002223253,
+            -0.014650062657892704,
+            0.017502501606941223,
+            0.009025990031659603,
+            0.005672555882483721,
+            -0.03149804100394249,
+            -0.010827955789864063,
+            0.01814894750714302,
+            -0.00046715763164684176,
+            -0.001248447224497795,
+            0.001733281067572534,
+            0.01077947299927473,
+            -0.009228004142642021,
+            -0.012670325115323067,
+            -0.01886003650724888,
+            0.0010595639469102025,
+            -0.0051513598300516605,
+            -0.0030584936030209064,
+            -0.01293698325753212,
+            -0.00844419002532959,
+            -0.013486461713910103,
+            -0.004420068580657244,
+            0.0008504794095642865,
+            0.0234659593552351,
+            0.01871458627283573,
+            0.01936103217303753,
+            0.002983748447149992,
+            0.004500874318182468,
+            0.017744919285178185,
+            0.013025869615375996,
+            0.007922993041574955,
+            0.00160500209312886,
+            -0.0019474159926176071,
+            0.028572876006364822,
+            -0.019199419766664505,
+            -0.012031960301101208,
+            0.018569136038422585,
+            -0.01331677008420229,
+            0.017761079594492912,
+            -0.02118724025785923,
+            0.001272688852623105,
+            0.005692757666110992,
+            0.003387776669114828,
+            -0.009809805080294609,
+            -0.008686606772243977,
+            0.019247904419898987,
+            0.009122957475483418,
+            -0.009413857012987137,
+            0.01979738287627697,
+            0.005769522860646248,
+            -0.006658385042101145,
+            -0.009696677327156067,
+            -0.0019383253529667854,
+            -0.0041089667938649654,
+            -0.015732858330011368,
+            0.016856057569384575,
+            0.018132785335183144,
+            -0.015353072434663773,
+            0.0042665377259254456,
+            0.01879539154469967,
+            -0.01973273791372776,
+            0.011248145252466202,
+            0.01378544233739376,
+            -0.010092624463140965,
+            0.007809865288436413,
+            -0.0010186561848968267,
+            0.013446059077978134,
+            0.003761502681300044,
+            -0.004145329352468252,
+            -0.0005929113831371069,
+            0.011967315338551998,
+            -0.017243923619389534,
+            -0.012112765572965145,
+            -0.012427907437086105,
+            -0.008157329633831978,
+            0.007123017217963934,
+            0.002535277046263218,
+            -0.0056483144871890545,
+            0.009025990031659603,
+            -0.011337031610310078,
+            -0.015991436317563057,
+            -0.028378941118717194,
+            0.0005499834078364074,
+            -0.00971283856779337,
+            -0.001211074530147016,
+            0.014698546379804611,
+            0.015749020501971245,
+            -0.0009449209901504219,
+            -0.016969185322523117,
+            0.01816510781645775,
+            0.04408755898475647,
+            -0.007809865288436413,
+            0.003777663689106703,
+            0.010601700283586979,
+            0.002488813828676939,
+            -0.0026302237529307604,
+            -0.0022908400278538465,
+            -0.016427787020802498,
+            -0.03765542805194855,
+            -0.012912741862237453,
+            0.01077947299927473,
+            -0.0012787493178620934,
+            -0.004638243932276964,
+            -0.021914489567279816,
+            -0.005842247977852821,
+            -0.016201531514525414,
+            -0.004921063780784607,
+            -0.019651932641863823,
+            -0.004799854941666126,
+            0.013979376293718815,
+            -0.015959113836288452,
+            0.019878188148140907,
+            -0.02024989388883114,
+            0.020540794357657433,
+            0.01385008729994297,
+            0.02277103066444397,
+            -0.01641162671148777,
+            0.009341131895780563,
+            -0.0014120786217972636,
+            0.014666223898530006,
+            0.01034312229603529,
+            0.007308870553970337,
+            -0.010617861524224281,
+            -0.006634143181145191,
+            0.01928022690117359,
+            0.009470420889556408,
+            0.013082433491945267,
+            0.009567388333380222,
+            -0.002262558089569211,
+            -0.0073371524922549725,
+            -0.014464209787547588,
+            -0.022156907245516777,
+            0.003163540968671441,
+            -0.0021676113829016685,
+            0.01688838005065918,
+            -0.020379183813929558,
+            -0.021607428789138794,
+            0.0002142612065654248,
+            0.014722787775099277,
+            0.035069648176431656,
+            -0.008573479019105434,
+            0.004371585324406624,
+            -0.012330940924584866,
+            0.01202387921512127,
+            0.018310558050870895,
+            0.005236205644905567,
+            0.0012221853248775005,
+            -0.020346861332654953,
+            0.011579448357224464,
+            0.011086533777415752,
+            -0.002422149060294032,
+            -0.0035089850425720215,
+            -0.0029029427096247673,
+            -0.02290031872689724,
+            0.0064078872092068195,
+            0.004399867262691259,
+            0.038043297827243805,
+            0.013761200942099094,
+            -0.0014807634288445115,
+            0.0072523062117397785,
+            0.029073869809508324,
+            -0.0032362660858780146,
+            0.009276487864553928,
+            0.0007363414042629302,
+            -0.003854429116472602,
+            0.008060363121330738,
+            -0.00811692699790001,
+            -0.0459299273788929,
+            0.010577458888292313,
+            -0.004492793697863817,
+            0.038043297827243805,
+            -0.0032746486831456423,
+            0.03043140470981598,
+            -0.006533136125653982,
+            -0.00810884591192007,
+            -0.005531146191060543,
+            0.017373213544487953,
+            -0.012508713640272617,
+            0.012258215807378292,
+            -0.008044201880693436,
+            -0.010625941678881645,
+            -0.0016342941671609879,
+            0.006779593415558338,
+            -0.012371343560516834,
+            -0.007333111949265003,
+            0.022156907245516777,
+            0.024387143552303314,
+            0.013761200942099094,
+            -0.009834046475589275,
+            0.0315626822412014,
+            -0.007510884664952755,
+            -0.014472290873527527,
+            0.019005486741662025,
+            -0.0037109991535544395,
+            -0.006428088992834091,
+            -0.001322182361036539,
+            -0.0023029607255011797,
+            -0.0024322497192770243,
+            0.017357051372528076,
+            0.0175186637789011,
+            -0.00429886020720005,
+            0.0015373273054137826,
+            0.00429886020720005,
+            -0.002468612277880311,
+            0.019150936976075172,
+            0.011749139986932278,
+            -0.03636253997683525,
+            0.014221793040633202,
+            -0.029364770278334618,
+            -0.01821359246969223,
+            -0.007579569239169359,
+            -0.004157450515776873,
+            0.011708737351000309,
+            -0.02390230819582939,
+            -0.011490561999380589,
+            -0.023417474702000618,
+            0.009753241203725338,
+            0.016969185322523117,
+            0.0065290960483253,
+            0.012137007899582386,
+            0.014302598312497139,
+            0.0012252154992893338,
+            -0.017421696335077286,
+            -0.013680395670235157,
+            -0.02765168994665146,
+            -0.014706626534461975,
+            0.012185490690171719,
+            0.0037493817508220673,
+            -0.013446059077978134,
+            -0.0023453838657587767,
+            0.026649700477719307,
+            -0.0037918046582490206,
+            -0.00313121872022748,
+            0.009753241203725338,
+            0.003694837912917137,
+            0.005220044404268265,
+            -0.005886691156774759,
+            0.014246034435927868,
+            0.00716342031955719,
+            0.01012494694441557,
+            -0.02984960377216339,
+            0.01738937385380268,
+            -0.003737261053174734,
+            -0.0008519944967702031,
+            0.0013363233301788568,
+            -0.004262497648596764,
+            -0.011660254560410976,
+            -0.0017757039749994874,
+            -0.02548610046505928,
+            -0.00689272116869688,
+            -0.0031918229069560766,
+            -0.011716818436980247,
+            -0.002658505691215396,
+            -0.00741795776411891,
+            -0.016807572916150093,
+            -0.010593620128929615,
+            0.01847217045724392,
+            -0.011062292382121086,
+            0.01293698325753212,
+            0.0011060272809118032,
+            -0.017550986260175705,
+            0.007361393887549639,
+            -0.012654163874685764,
+            0.0016403545159846544,
+            -0.009591629728674889,
+            0.0001431775017408654,
+            0.019134776666760445,
+            -0.008767412044107914,
+            0.006581619381904602,
+            -0.004278658889234066,
+            0.011151178739964962,
+            -0.00102168635930866,
+            0.016710607334971428,
+            0.015951033681631088,
+            0.011046131141483784,
+            0.00010163834667764604,
+            -0.015732858330011368,
+            -0.019053971394896507,
+            0.022108424454927444,
+            0.004214014392346144,
+            -0.01046433113515377,
+            0.02404775843024254,
+            0.002684767358005047,
+            0.01992667093873024,
+            -0.011652173474431038,
+            0.003864529775455594,
+            0.024516431614756584,
+            0.010060302913188934,
+            0.008993667550384998,
+            -0.0035776698496192694,
+            -0.0003694332845043391,
+            0.00482409680262208,
+            0.03416462615132332,
+            0.019247904419898987,
+            -0.005927093792706728,
+            -0.024306336417794228,
+            0.018262075260281563,
+            0.0019191340543329716,
+            0.001435310230590403,
+            0.02024989388883114,
+            0.024322498589754105,
+            0.01808430254459381,
+            0.007143219001591206,
+            0.008654284290969372,
+            -0.012452149763703346,
+            0.010747150518000126,
+            -0.0007651284104213119,
+            -0.0055473074316978455,
+            0.0003335757937747985,
+            -0.029591025784611702,
+            0.012258215807378292,
+            -0.00270294863730669,
+            -0.014229873195290565,
+            0.0021817523520439863,
+            -0.013130917213857174,
+            0.022867996245622635,
+            -0.0011575408279895782,
+            -0.01645202934741974,
+            -0.004694807808846235,
+            -0.024063920602202415,
+            0.0028201169334352016,
+            -0.01923174224793911,
+            -0.005369534716010094,
+            -0.012015799060463905,
+            -0.0013918771874159575,
+            -0.022237712517380714,
+            -0.009591629728674889,
+            0.011539045721292496,
+            -0.002353464253246784,
+            -0.007450280245393515,
+            -0.005914973095059395,
+            0.00763613311573863,
+            -0.018569136038422585,
+            -0.0028423385228961706,
+            0.005716999061405659,
+            0.019150936976075172,
+            0.0025191158056259155,
+            0.003535246942192316,
+            0.01871458627283573,
+            0.013963215053081512,
+            -0.03445552662014961,
+            0.003650394966825843,
+            0.00254537770524621,
+            0.00223023584112525,
+            0.025195199996232986,
+            0.007462400943040848,
+            -0.00732099125161767,
+            0.014795512892305851,
+            -0.0003237276105210185,
+            0.01871458627283573,
+            0.002375685842707753,
+            -0.014682385139167309,
+            -0.00716342031955719,
+            0.005672555882483721,
+            -0.007737140171229839,
+            0.021106433123350143,
+            -0.002543357666581869,
+            0.007813905365765095,
+            -0.009995657950639725,
+            -0.011078453622758389,
+            -0.00678363349288702,
+            0.00013863218191545457,
+            -0.006011939607560635,
+            0.00885629840195179,
+            -0.009809805080294609,
+            0.005813966039568186,
+            0.006133148446679115,
+            0.004022100940346718,
+            -0.013938973657786846,
+            0.016056081280112267,
+            -0.003199903527274728,
+            0.008573479019105434,
+            0.02605173923075199,
+            0.00572507968172431,
+            -0.009438099339604378,
+            0.017550986260175705,
+            -8.59191277413629e-05,
+            -0.01128854788839817,
+            -0.02244780771434307,
+            -0.003878670744597912,
+            -0.03907760977745056,
+            0.028378941118717194,
+            0.00023193744709715247,
+            -0.018520653247833252,
+            0.02517903782427311,
+            -0.00816136971116066,
+            -0.008274497464299202,
+            -0.00018143391935154796,
+            -0.0100845443084836,
+            -0.02340131439268589,
+            0.002715069567784667,
+            -0.010569377802312374,
+            -0.006108906585723162,
+            0.011934993788599968,
+            0.030334437265992165,
+            -0.008815895766019821,
+            -0.0008913872297853231,
+            0.006937164347618818,
+            0.003607971826568246,
+            0.01879539154469967,
+            -0.028071880340576172,
+            -0.0032948500011116266,
+            -0.018326720222830772,
+            0.019199419766664505,
+            -0.0007105846307240427,
+            -0.01682373508810997,
+            0.012339022010564804,
+            -0.014949044212698936,
+            -0.02175287902355194,
+            -0.0003398887347429991,
+            -0.0060563827864825726,
+            0.005664475727826357,
+            0.0063594039529562,
+            0.02498510479927063,
+            0.0033756557386368513,
+            0.007963395677506924,
+            -0.02580932155251503,
+            -0.023708375170826912,
+            0.02383766509592533,
+            0.01103805098682642,
+            -0.014932882972061634,
+            -0.0011555206729099154,
+            -0.005106916651129723,
+            -0.007219984196126461,
+            -0.019910510629415512,
+            0.007175541017204523,
+            0.06651920825242996,
+            -0.007361393887549639,
+            0.003096876200288534,
+            -0.015643972903490067,
+            -0.005632153246551752,
+            -0.007098775822669268,
+            -0.01929638721048832,
+            -0.006363444495946169,
+            0.00024822482373565435,
+            0.005733160302042961,
+            -0.025405293330550194,
+            0.004054422955960035,
+            -0.006706868298351765,
+            0.007001808844506741,
+            -0.012137007899582386,
+            0.010270397178828716,
+            -0.015490441583096981,
+            -0.006860399153083563,
+            0.0028686001896858215,
+            -0.009817885234951973,
+            -0.0009388605831190944,
+            -0.017243923619389534,
+            0.001279759337194264,
+            0.016112644225358963,
+            0.003616052446886897,
+            -0.012767291627824306,
+            -0.016460109502077103,
+            -0.01947415992617607,
+            -0.01214508805423975,
+            -0.005070554092526436,
+            -0.008339142426848412,
+            -0.023320509120821953,
+            0.012169329449534416,
+            -0.007741180714219809,
+            0.016856057569384575,
+            0.011054212227463722,
+            0.013429897837340832,
+            0.020573116838932037,
+            -0.003389796707779169,
+            0.009308810345828533,
+            -0.013421816751360893,
+            0.03067382238805294,
+            -0.0022787190973758698,
+            -0.023449797183275223,
+            -0.0007312910747714341,
+            0.00033913119114004076,
+            -0.007931074127554893,
+            -0.021849846467375755,
+            0.007555327378213406,
+            -0.002583760302513838,
+            0.011934993788599968,
+            -0.005959416273981333,
+            0.003995839040726423,
+            0.0015686395345255733,
+            0.006294759456068277,
+            0.004343303386121988,
+            0.017841886729002,
+            -0.0059957788325846195,
+            0.01997515559196472,
+            -0.02815268561244011,
+            -0.01738937385380268,
+            -0.008230054751038551,
+            -0.007490682881325483,
+            0.012613760307431221,
+            0.006044262088835239,
+            -0.0032685883343219757,
+            -0.016492431983351707,
+            -0.010076463222503662,
+            0.008880539797246456,
+            -0.016807572916150093,
+            -0.026601217687129974,
+            -0.0016080322675406933,
+            -0.0325646735727787,
+            0.007110896520316601,
+            0.004214014392346144,
+            -0.005700838286429644,
+            -0.005575589369982481,
+            0.02010444365441799,
+            0.00948658213019371,
+            0.006646263878792524,
+            0.005692757666110992,
+            -0.0017070191679522395,
+            -0.011312790215015411,
+            -0.0009595670271664858,
+            0.0015565187204629183,
+            0.011749139986932278,
+            -0.002377706114202738,
+            -0.01571669802069664,
+            0.015353072434663773,
+            0.009034071117639542,
+            -0.0108198756352067,
+            0.0004469561972655356,
+            0.019700415432453156,
+            0.0039574564434587955,
+            -0.0180196575820446,
+            0.004597840830683708,
+            0.018197430297732353,
+            -0.005979617591947317,
+            0.013163238763809204,
+            0.026552734896540642,
+            -1.8986169379786588e-05,
+            -0.00078280468005687,
+            -0.017599469050765038,
+            -0.01255719643086195,
+            -0.0035574682988226414,
+            0.023611409589648247,
+            0.029251642525196075,
+            0.002240336500108242,
+            -0.027344629168510437,
+            0.009567388333380222,
+            -0.016969185322523117,
+            0.012961224652826786,
+            -0.015231863595545292,
+            0.0021474098321050406,
+            0.02092866227030754,
+            -0.013761200942099094,
+            0.004989748355001211,
+            0.009898691438138485,
+            0.028750646859407425,
+            0.004561478737741709,
+            -0.015264186076819897,
+            0.0006994738359935582,
+            -0.02081553265452385,
+            0.002008020179346204,
+            -0.006395766511559486,
+            0.005733160302042961,
+            -0.011369354091584682,
+            0.00646849162876606,
+            -0.011110776104032993,
+            0.0006439199787564576,
+            0.0012141048209741712,
+            -0.02118724025785923,
+            0.007583609316498041,
+            -0.018003497272729874,
+            0.0037453414406627417,
+            0.0010767352068796754,
+            -0.004048362839967012,
+            -0.023643730208277702,
+            -0.002511035418137908,
+            0.028896097093820572,
+            -0.007805825211107731,
+            0.05788916349411011,
+            -0.006290719378739595,
+            0.0014403605600818992,
+            -0.009034071117639542,
+            0.0063068801537156105,
+            -0.0009706777636893094,
+            -0.009349212981760502,
+            -0.005413977894932032,
+            0.002723150188103318,
+            0.0029130433686077595,
+            0.004492793697863817,
+            0.009914852678775787,
+            0.004836217500269413,
+            0.0005153884994797409,
+            0.02984960377216339,
+            0.0008681556209921837,
+            0.0085977204144001,
+            -0.004484713077545166,
+            0.0022908400278538465,
+            -0.016524754464626312,
+            0.01803581975400448,
+            0.005801845341920853,
+            -0.022609418258070946,
+            -0.016662122681736946,
+            -0.017049990594387054,
+            0.013462220318615437,
+            -0.0031009165104478598,
+            0.013898570090532303,
+            -0.005846288055181503,
+            -0.014553096145391464,
+            0.017664114013314247,
+            0.008557317778468132,
+            0.009405776858329773,
+            -0.0002760017814580351,
+            0.005874569993466139,
+            0.005462461616843939,
+            0.008823975920677185,
+            -0.005102876108139753,
+            -0.005317011382430792,
+            -0.0026766869705170393,
+            0.014981365762650967,
+            -0.011029969900846481,
+            -0.0011454200139269233,
+            -0.016112644225358963,
+            -0.006791714113205671,
+            0.022932641208171844,
+            0.000245573406573385,
+            -0.003708978882059455,
+            0.02650425024330616,
+            0.011692576110363007,
+            -0.00010833006672328338,
+            -0.009696677327156067,
+            -0.0013646052684634924,
+            0.007725019473582506,
+            0.020718567073345184,
+            -0.004028161056339741,
+            -0.008500753901898861,
+            -0.017615629360079765,
+            -0.0011454200139269233,
+            0.030059698969125748,
+            -0.007535126060247421,
+            -0.019894348457455635,
+            0.01284001674503088
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "1ea6fc21-365a-41fe-9b07-1e5411ca0bc9",
+        "properties": {
+          "page_content": "4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n",
+          "title": "Validation Experiments",
+          "embedding": [
+            -0.010754532180726528,
+            -0.023697586730122566,
+            -0.02424471639096737,
+            -0.004748920910060406,
+            0.0029044931288808584,
+            0.014695581048727036,
+            -0.013233717530965805,
+            0.03884626179933548,
+            -0.0167387705296278,
+            0.032195206731557846,
+            -0.028296900913119316,
+            0.003022040706127882,
+            0.033374954015016556,
+            -0.032622650265693665,
+            -0.007941939868032932,
+            0.013165325857698917,
+            0.035289913415908813,
+            0.03330656513571739,
+            0.011327309533953667,
+            -0.037751998752355576,
+            0.04247099906206131,
+            -0.036520954221487045,
+            -0.060047563165426254,
+            -0.011079391464591026,
+            -0.019577009603381157,
+            0.02089354395866394,
+            0.00764272827655077,
+            0.016584891825914383,
+            -0.0636381059885025,
+            0.03330656513571739,
+            0.00491989916190505,
+            0.023150455206632614,
+            0.0002941360289696604,
+            0.008754086680710316,
+            -0.011498288251459599,
+            -0.03205842152237892,
+            0.022381054237484932,
+            -0.0009312971960753202,
+            -0.015362396836280823,
+            0.042607780545949936,
+            0.03381950035691261,
+            0.01822628267109394,
+            -0.023372728377580643,
+            -0.004340710584074259,
+            0.016832809895277023,
+            0.00703148078173399,
+            -0.013173874467611313,
+            -0.010062070563435555,
+            0.015704352408647537,
+            -0.04240260645747185,
+            -0.03546088933944702,
+            -0.011600874364376068,
+            0.008172760717570782,
+            -0.01200267393141985,
+            -0.03761521726846695,
+            0.007856450974941254,
+            0.016140347346663475,
+            0.026159673929214478,
+            0.04093219339847565,
+            0.015439336188137531,
+            0.027305228635668755,
+            -0.018602434545755386,
+            -0.019765086472034454,
+            0.010164657607674599,
+            0.0010231980122625828,
+            -0.03443502262234688,
+            0.0016926847165450454,
+            -0.011720559559762478,
+            0.017046531662344933,
+            -0.034383729100227356,
+            -0.04520665109157562,
+            0.02380017377436161,
+            -0.009292668662965298,
+            -0.010993901640176773,
+            -0.0009270227164961398,
+            -0.010198852978646755,
+            0.031203532591462135,
+            0.027544597163796425,
+            -0.0026907704304903746,
+            0.000909390626475215,
+            0.014806716702878475,
+            0.02405664138495922,
+            -0.023714683949947357,
+            0.0028040434699505568,
+            0.04373623803257942,
+            -0.0030434129294008017,
+            -0.02814302034676075,
+            -0.06527949869632721,
+            0.06415104120969772,
+            -0.0167815163731575,
+            -0.013644064776599407,
+            0.0002978761913254857,
+            0.023646293208003044,
+            -0.019577009603381157,
+            -0.029237281531095505,
+            0.05844036862254143,
+            -0.007394809741526842,
+            0.0007074225577525795,
+            -0.018619531765580177,
+            0.014601543545722961,
+            -0.009771407581865788,
+            0.010865667834877968,
+            -0.008617304265499115,
+            0.014422016218304634,
+            0.032622650265693665,
+            -0.024877335876226425,
+            -0.014447662979364395,
+            0.012831917963922024,
+            0.013712456449866295,
+            0.10628008842468262,
+            -0.016217287629842758,
+            -0.005920122377574444,
+            -0.03257135674357414,
+            0.0011006725253537297,
+            -0.009745760820806026,
+            -0.05006243288516998,
+            -0.026826487854123116,
+            -0.025373173877596855,
+            0.031237727031111717,
+            -0.008035978302359581,
+            0.033836595714092255,
+            0.032434575259685516,
+            0.045035671442747116,
+            -0.0132422661408782,
+            -0.04804489016532898,
+            0.046950630843639374,
+            -0.0051549943163990974,
+            0.000823367154225707,
+            -0.0025347527116537094,
+            -0.003041275776922703,
+            -0.0017194000538438559,
+            -0.007373437285423279,
+            -0.01822628267109394,
+            -0.01443056482821703,
+            0.011831695213913918,
+            0.0043449848890304565,
+            0.034298237413167953,
+            0.053003259003162384,
+            -0.04373623803257942,
+            -0.009685918688774109,
+            0.026792293414473534,
+            -0.029938293620944023,
+            0.010985353030264378,
+            0.006091100629419088,
+            -0.02403954230248928,
+            0.015011890791356564,
+            0.0010638053063303232,
+            0.055465348064899445,
+            0.0003606572572607547,
+            0.02374888025224209,
+            0.0011113586369901896,
+            0.04274456575512886,
+            -0.008360836654901505,
+            0.04363365098834038,
+            0.010592103004455566,
+            0.0010258695110678673,
+            -0.020927738398313522,
+            0.018602434545755386,
+            -0.039427585899829865,
+            -0.000900841667316854,
+            0.03171646595001221,
+            -0.0211500097066164,
+            -0.009899641387164593,
+            0.012660940177738667,
+            0.004398415796458721,
+            0.06869906187057495,
+            -0.01395182590931654,
+            -0.0074888477101922035,
+            -0.0781712606549263,
+            0.04982306435704231,
+            0.023971151560544968,
+            0.00814711395651102,
+            0.005907298997044563,
+            0.04182128235697746,
+            0.003832050133496523,
+            0.01309693418443203,
+            0.012438667938113213,
+            -0.0098397983238101,
+            -0.05026760697364807,
+            0.03207552060484886,
+            -0.0026587119791656733,
+            0.012908858247101307,
+            -0.009130238555371761,
+            0.026860684156417847,
+            0.05505499988794327,
+            0.022021999582648277,
+            0.01165216788649559,
+            0.027476206421852112,
+            -0.020671270787715912,
+            0.01749962382018566,
+            0.037033889442682266,
+            -0.0011412798194214702,
+            0.03744423761963844,
+            0.026980368420481682,
+            -0.005979964509606361,
+            -0.013105483725667,
+            -0.028348194435238838,
+            0.013797945342957973,
+            0.006086825858801603,
+            0.03405886888504028,
+            -0.02034641243517399,
+            -0.03525571525096893,
+            0.007664100266993046,
+            -0.0034281141124665737,
+            0.04079541191458702,
+            -0.00934396218508482,
+            0.007822255603969097,
+            0.023099161684513092,
+            0.010942608118057251,
+            -0.004847233649343252,
+            -0.024210521951317787,
+            -0.008540363982319832,
+            -0.01188298873603344,
+            -0.0402824766933918,
+            -0.011891537345945835,
+            0.03157968446612358,
+            -0.015353847295045853,
+            0.01938893459737301,
+            -0.005915847606956959,
+            -0.007800883147865534,
+            -0.023543706163764,
+            0.018072402104735374,
+            0.0032464496325701475,
+            -0.008027428761124611,
+            -0.0069075217470526695,
+            0.003357585519552231,
+            -0.0019352601375430822,
+            0.003267821855843067,
+            0.048181671649217606,
+            -0.02402244508266449,
+            0.012113809585571289,
+            0.008527540601789951,
+            0.0011316622840240598,
+            0.02380017377436161,
+            0.03346044570207596,
+            -0.035563476383686066,
+            -0.012934505008161068,
+            -0.0014618640998378396,
+            0.009514939971268177,
+            0.020876444876194,
+            -0.009873994626104832,
+            -0.0097030159085989,
+            -0.03378530219197273,
+            -0.0005765173118561506,
+            0.00013043770741205662,
+            -0.015712901949882507,
+            -0.008467698469758034,
+            0.017166217789053917,
+            0.042641978710889816,
+            0.032434575259685516,
+            0.02034641243517399,
+            -0.035289913415908813,
+            -0.00907894503325224,
+            0.027082955464720726,
+            -0.014610092155635357,
+            0.034315336495637894,
+            0.0504043884575367,
+            -0.029459552839398384,
+            -0.018773412331938744,
+            -0.03980373963713646,
+            0.07112695276737213,
+            -0.023731783032417297,
+            -0.02964762970805168,
+            0.042368412017822266,
+            -0.01408860832452774,
+            0.0021778354421257973,
+            0.03187034651637077,
+            0.035324107855558395,
+            -0.0035307009238749743,
+            0.012053967453539371,
+            0.03499924764037132,
+            -0.0194060318171978,
+            -0.04691643267869949,
+            -0.02523639053106308,
+            -0.025116706266999245,
+            -0.016619086265563965,
+            -0.010275793261826038,
+            -0.017311548814177513,
+            0.010087717324495316,
+            0.020278021693229675,
+            0.013635516166687012,
+            -0.010882765986025333,
+            0.007022931706160307,
+            0.009985130280256271,
+            -0.009976581670343876,
+            -0.000917939527425915,
+            -0.027561694383621216,
+            -0.0071939099580049515,
+            -0.028467880561947823,
+            -0.012053967453539371,
+            -0.014977695420384407,
+            -0.006685249973088503,
+            -0.025783522054553032,
+            0.01889309659600258,
+            0.023167554289102554,
+            -0.002190658822655678,
+            -0.008459148928523064,
+            0.01238737441599369,
+            0.03472568467259407,
+            -0.028108825907111168,
+            -0.03812815248966217,
+            0.008916515856981277,
+            -0.0034558980260044336,
+            0.009762858040630817,
+            -0.015037537552416325,
+            0.010677591897547245,
+            -0.011395701207220554,
+            0.010540809482336044,
+            -0.010703238658607006,
+            -0.010002228431403637,
+            -0.0006646779947914183,
+            0.021953608840703964,
+            0.00801460538059473,
+            -0.014592994004487991,
+            -0.017593663185834885,
+            0.027715574949979782,
+            0.014293782413005829,
+            0.0334775410592556,
+            -0.029545042663812637,
+            -0.023099161684513092,
+            0.042573586106300354,
+            0.004015851765871048,
+            -0.043086521327495575,
+            0.016148896887898445,
+            0.01671312376856804,
+            0.011797499842941761,
+            0.01942313089966774,
+            0.048181671649217606,
+            -0.033870793879032135,
+            -0.038196541368961334,
+            0.005420010536909103,
+            -0.031938739120960236,
+            -0.030929965898394585,
+            0.014883656986057758,
+            -0.006227883044630289,
+            -0.05125928297638893,
+            -0.031203532591462135,
+            -0.014020216651260853,
+            -0.019337641075253487,
+            0.0022098938934504986,
+            -0.015046087093651295,
+            0.004941271618008614,
+            -0.03741004317998886,
+            0.0025689483154565096,
+            0.0004648471367545426,
+            -0.018328869715332985,
+            -0.028724348172545433,
+            0.0027548871003091335,
+            0.004488179460167885,
+            0.03542669489979744,
+            0.0017097825184464455,
+            0.01128456462174654,
+            -0.012951603159308434,
+            -0.000579188868869096,
+            0.003056236309930682,
+            -0.05016501992940903,
+            -0.019474424421787262,
+            0.038983043283224106,
+            0.009027652442455292,
+            0.006048355717211962,
+            0.011874440126121044,
+            -0.028211412951350212,
+            -0.0592268668115139,
+            0.029716020449995995,
+            -0.04643769562244415,
+            0.015268358401954174,
+            0.004415513481944799,
+            -0.0010344184702262282,
+            -0.001160514890216291,
+            -0.0007758138235658407,
+            -0.0067707388661801815,
+            0.023167554289102554,
+            -0.007407633122056723,
+            0.012498511001467705,
+            -0.0028788463678210974,
+            0.03669193387031555,
+            0.0358712375164032,
+            -0.00673226872459054,
+            -0.038948848843574524,
+            0.02956213988363743,
+            -0.008668597787618637,
+            0.025612542405724525,
+            0.01889309659600258,
+            0.02961343340575695,
+            -0.017918521538376808,
+            0.032417476177215576,
+            0.002891669748350978,
+            0.024125032126903534,
+            -0.03877786919474602,
+            0.023167554289102554,
+            0.0052191112190485,
+            -0.054268497973680496,
+            -0.00860020611435175,
+            -0.0073777115903794765,
+            0.022244270890951157,
+            0.008822478353977203,
+            -0.02477474883198738,
+            -0.021970706060528755,
+            0.0009409147314727306,
+            -0.02817721664905548,
+            -0.007574336603283882,
+            0.04568539187312126,
+            -0.012558353133499622,
+            0.0014875108608976007,
+            -0.017414135858416557,
+            0.014174097217619419,
+            0.02886112965643406,
+            0.027510400861501694,
+            0.03354593366384506,
+            -0.013986021280288696,
+            0.020021554082632065,
+            -0.022278467193245888,
+            -0.037033889442682266,
+            -0.02595449984073639,
+            0.0073050460778176785,
+            0.03939339146018028,
+            0.009728662669658661,
+            0.006039807107299566,
+            -0.0013913356233388186,
+            0.00790346972644329,
+            -0.015370945446193218,
+            0.027989141643047333,
+            0.044796302914619446,
+            -0.002494145417585969,
+            -0.011464091949164867,
+            0.007505945395678282,
+            0.015046087093651295,
+            -0.018619531765580177,
+            0.027373619377613068,
+            -0.025424467399716377,
+            0.02255203202366829,
+            0.004744646605104208,
+            0.02065417356789112,
+            -0.047702934592962265,
+            -0.004556570667773485,
+            -0.0023082063999027014,
+            -0.03445211797952652,
+            0.05946623906493187,
+            -0.028690151870250702,
+            -0.013524380512535572,
+            0.0029386887326836586,
+            -0.0011316622840240598,
+            -0.01313113048672676,
+            -0.026980368420481682,
+            -0.014849461615085602,
+            -0.0061894129030406475,
+            -0.04554860666394234,
+            -0.029801510274410248,
+            -0.014815266244113445,
+            -0.01213945634663105,
+            -0.06240706518292427,
+            0.004058596212416887,
+            -0.0741361752152443,
+            0.007206733338534832,
+            0.027715574949979782,
+            -0.016661832109093666,
+            0.03963275998830795,
+            -0.022637521848082542,
+            0.02374888025224209,
+            -0.04076121747493744,
+            -0.0157299991697073,
+            0.03258845582604408,
+            0.047463566064834595,
+            0.010848570615053177,
+            0.011498288251459599,
+            0.021047422662377357,
+            -0.03199003264307976,
+            -0.0026758096646517515,
+            -0.00860875565558672,
+            -0.0018219870980829,
+            0.03332366421818733,
+            -0.008553187362849712,
+            0.021765531972050667,
+            0.0051549943163990974,
+            0.024142129346728325,
+            0.031442902982234955,
+            -0.002782671246677637,
+            -0.023868564516305923,
+            0.012857564724981785,
+            0.025544151663780212,
+            0.013054190203547478,
+            -0.014251037500798702,
+            0.012686586938798428,
+            -0.01579839177429676,
+            0.007719668094068766,
+            0.008484795689582825,
+            0.009908189997076988,
+            0.034315336495637894,
+            0.024142129346728325,
+            -0.013105483725667,
+            0.0029814334120601416,
+            0.009865445084869862,
+            -0.03645256534218788,
+            0.042641978710889816,
+            0.0051849158480763435,
+            -0.03953017294406891,
+            -0.0005487333401106298,
+            -0.007617081515491009,
+            0.020551586523652077,
+            0.02156035788357258,
+            0.00936105940490961,
+            -0.022671716287732124,
+            -0.00020944836433045566,
+            0.016841357573866844,
+            0.008660049177706242,
+            -0.012182201258838177,
+            0.02450118400156498,
+            0.03833332657814026,
+            -0.020260922610759735,
+            -0.030639303848147392,
+            0.00922427698969841,
+            -0.010489515960216522,
+            0.0007736766128800809,
+            -0.02715134806931019,
+            -0.004394141025841236,
+            0.018072402104735374,
+            -0.044762108474969864,
+            0.05177221819758415,
+            -0.013575674034655094,
+            0.008685695938766003,
+            -0.012943054549396038,
+            -0.002703593811020255,
+            -0.02138938009738922,
+            0.034588903188705444,
+            0.014131353236734867,
+            0.02867305465042591,
+            0.016106151044368744,
+            0.010318538174033165,
+            0.028040433302521706,
+            -0.020278021693229675,
+            -0.023099161684513092,
+            0.014422016218304634,
+            -0.007980410009622574,
+            -0.00467198109254241,
+            -0.02959633618593216,
+            -0.024107934907078743,
+            -0.01892729289829731,
+            0.02527058683335781,
+            -0.01530255377292633,
+            -0.008566010743379593,
+            0.010951157659292221,
+            0.0014981969725340605,
+            -0.002131885150447488,
+            0.013943277299404144,
+            -0.007638453505933285,
+            0.008707067929208279,
+            0.04192386940121651,
+            -0.013934727758169174,
+            0.02159455418586731,
+            0.03611060976982117,
+            0.011472641490399837,
+            -0.018516944721341133,
+            -0.0064800758846104145,
+            -0.0007293291273526847,
+            -0.01066904328763485,
+            -0.03570026159286499,
+            0.035289913415908813,
+            0.020996129140257835,
+            -0.0021842471323907375,
+            -0.037239063531160355,
+            -0.012583999894559383,
+            0.014635738916695118,
+            -0.014037314802408218,
+            0.027732674032449722,
+            -0.015943722799420357,
+            -0.015653058886528015,
+            0.00474037230014801,
+            0.021970706060528755,
+            -0.049857258796691895,
+            -0.012977249920368195,
+            0.010899864137172699,
+            0.002707868115976453,
+            0.008057350292801857,
+            -0.006779287941753864,
+            0.013455988839268684,
+            0.020568683743476868,
+            0.004945545922964811,
+            0.0036439741961658,
+            -0.0038085407577455044,
+            0.01647375524044037,
+            0.013455988839268684,
+            -0.017593663185834885,
+            -0.009241375140845776,
+            0.02210748940706253,
+            0.0230820644646883,
+            -0.030211858451366425,
+            -0.023919858038425446,
+            -0.00583035871386528,
+            0.021902315318584442,
+            -0.004753195680677891,
+            0.03689710795879364,
+            0.0388120636343956,
+            0.011438445188105106,
+            -0.04318910837173462,
+            -0.014516053721308708,
+            0.0017290175892412663,
+            -0.010660494677722454,
+            0.016431011259555817,
+            -0.0012438668636605144,
+            -0.042812954634428024,
+            0.002143639838322997,
+            -0.0036055040545761585,
+            0.028450781479477882,
+            -0.040214087814092636,
+            -0.011959929019212723,
+            0.009617527015507221,
+            -0.012951603159308434,
+            0.003302017692476511,
+            0.012182201258838177,
+            0.008484795689582825,
+            0.021269695833325386,
+            0.012703684158623219,
+            -0.005526872351765633,
+            -0.0014607955235987902,
+            0.016550695523619652,
+            0.004535198211669922,
+            0.026604216545820236,
+            -0.018328869715332985,
+            -0.017345745116472244,
+            0.01844855397939682,
+            0.03860688954591751,
+            -0.012327532283961773,
+            0.04076121747493744,
+            0.007514494471251965,
+            0.012062516063451767,
+            0.012088162824511528,
+            -0.02453538030385971,
+            -0.01598646678030491,
+            -0.011951380409300327,
+            0.006257804110646248,
+            -0.037512630224227905,
+            0.014858010224997997,
+            -0.02380017377436161,
+            -0.027134248986840248,
+            0.02619386836886406,
+            -0.006014160346239805,
+            0.010686141438782215,
+            0.022244270890951157,
+            -0.012891761027276516,
+            -0.027271032333374023,
+            0.03369981423020363,
+            0.034349530935287476,
+            0.008433502167463303,
+            0.04677965119481087,
+            -0.010395478457212448,
+            0.006779287941753864,
+            0.005702124908566475,
+            0.0024920080322772264,
+            0.044762108474969864,
+            -0.008236877620220184,
+            -0.01200267393141985,
+            0.007535866927355528,
+            0.009779956191778183,
+            0.04206065088510513,
+            -0.015413690358400345,
+            -0.049412716180086136,
+            0.019508618861436844,
+            -0.014507505111396313,
+            0.009856896474957466,
+            -0.006655328441411257,
+            0.008198407478630543,
+            -0.005890200845897198,
+            0.010053521953523159,
+            -0.004522374831140041,
+            0.010113364085555077,
+            -0.005843182094395161,
+            -0.014327977783977985,
+            0.015174319967627525,
+            -0.01018175482749939,
+            0.030280249193310738,
+            0.009557684883475304,
+            -0.02503121644258499,
+            -0.026501629501581192,
+            0.01746542938053608,
+            0.0019277798710390925,
+            -0.028724348172545433,
+            0.009882543236017227,
+            0.0023872838355600834,
+            0.010104815475642681,
+            0.004381317645311356,
+            -0.012242043390870094,
+            0.015926625579595566,
+            0.01676441729068756,
+            -0.022210076451301575,
+            0.03911982476711273,
+            -0.014285233803093433,
+            -0.014644287526607513,
+            0.019235054031014442,
+            -0.03730745613574982,
+            0.008424953557550907,
+            -0.012524157762527466,
+            0.010831472463905811,
+            0.00491989916190505,
+            0.004667706321924925,
+            -0.011352956295013428,
+            -0.0026715353596955538,
+            -0.02352660894393921,
+            0.03281072899699211,
+            0.0272197388112545,
+            -0.0012834055814892054,
+            0.005958592053502798,
+            -0.007535866927355528,
+            -0.012900309637188911,
+            0.0024984197225421667,
+            0.026142574846744537,
+            -0.024381499737501144,
+            -0.0017899286467581987,
+            -0.0006310166209004819,
+            0.007694021798670292,
+            0.00551832327619195,
+            0.022432347759604454,
+            0.04900236800312996,
+            -0.0038192267529666424,
+            0.028980813920497894,
+            0.035050541162490845,
+            0.024142129346728325,
+            -0.003259273013100028,
+            0.004761744290590286,
+            -0.025150902569293976,
+            0.03645256534218788,
+            0.008950712159276009,
+            0.016670379787683487,
+            0.017798837274312973,
+            -0.04312071576714516,
+            -0.009634625166654587,
+            0.019525716081261635,
+            0.02205619588494301,
+            0.020089944824576378,
+            -0.003881206503137946,
+            -0.0035884061362594366,
+            0.03298170492053032,
+            -0.025920303538441658,
+            -0.019354738295078278,
+            0.004154771566390991,
+            -0.0053174239583313465,
+            -0.005732045974582434,
+            -0.0069032469764351845,
+            0.01865372806787491,
+            -0.014584445394575596,
+            -0.004458257928490639,
+            0.016867004334926605,
+            0.001722605898976326,
+            0.012113809585571289,
+            0.016165994107723236,
+            -0.00024137321452144533,
+            0.022415248677134514,
+            -0.004932722542434931,
+            0.021509064361453056,
+            0.005330247338861227,
+            -0.003663209266960621,
+            -0.006911796052008867,
+            -0.010275793261826038,
+            0.029015010222792625,
+            -0.006980187259614468,
+            -0.034640196710824966,
+            -0.01360986940562725,
+            -0.009284119121730328,
+            0.004156908951699734,
+            0.0011391426669433713,
+            0.0050524077378213406,
+            0.01744833029806614,
+            0.026501629501581192,
+            -0.00046751866466365755,
+            0.004280867986381054,
+            0.014336527325212955,
+            -0.0024193422868847847,
+            -0.010506614111363888,
+            0.019559912383556366,
+            0.008779733441770077,
+            -0.01624293439090252,
+            0.018756315112113953,
+            0.0030861576087772846,
+            0.04688223823904991,
+            -0.044522739946842194,
+            0.018209183588624,
+            0.007805157452821732,
+            0.0006956678116694093,
+            -0.014960597269237041,
+            -0.03327237069606781,
+            0.0036097783595323563,
+            -0.014122803695499897,
+            -0.0033597226720303297,
+            -0.024689260870218277,
+            0.00786499958485365,
+            -0.018089499324560165,
+            0.014045863412320614,
+            0.011600874364376068,
+            -0.0016349796205759048,
+            -0.002028229646384716,
+            0.017542369663715363,
+            -0.001940603251568973,
+            0.00787782296538353,
+            -0.017362842336297035,
+            0.010942608118057251,
+            0.022381054237484932,
+            -0.010677591897547245,
+            0.008638676255941391,
+            0.0073306928388774395,
+            -0.02793784812092781,
+            -0.028536271303892136,
+            0.013336303643882275,
+            -0.03546088933944702,
+            0.007681198418140411,
+            0.031203532591462135,
+            -0.032434575259685516,
+            -0.028519174084067345,
+            0.0019085448002442718,
+            -0.021406477317214012,
+            -0.017867228016257286,
+            0.011446994729340076,
+            0.018021108582615852,
+            -0.004216751083731651,
+            -0.030451226979494095,
+            0.013173874467611313,
+            -0.007809431757777929,
+            0.00698873633518815,
+            0.00015388043539132923,
+            -0.02569803223013878,
+            -0.02569803223013878,
+            0.004984016064554453,
+            -0.028211412951350212,
+            -0.007138342130929232,
+            -0.0005297654424794018,
+            -0.024466989561915398,
+            -0.043804630637168884,
+            -0.015242711640894413,
+            -0.013507282361388206,
+            -0.010156108066439629,
+            0.0480106957256794,
+            0.0196795966476202,
+            0.026894880458712578,
+            -0.0003211185394320637,
+            -0.018106596544384956,
+            0.04773712903261185,
+            0.0009954140987247229,
+            0.041752889752388,
+            0.013387597166001797,
+            -0.008335189893841743,
+            -0.006693798583000898,
+            0.03392208740115166,
+            -0.015584668144583702,
+            -0.007775236386805773,
+            -0.010651945136487484,
+            -0.0015836860984563828,
+            0.0025091059505939484,
+            0.022449444979429245,
+            0.0019085448002442718,
+            0.004037224221974611,
+            0.027100054547190666,
+            -0.023201748728752136,
+            0.002739926567301154,
+            0.01238737441599369,
+            -0.006937442813068628,
+            0.008215505629777908,
+            -0.02838239073753357,
+            -0.008300994522869587,
+            0.019081173464655876,
+            0.006044081412255764,
+            -0.007437554188072681,
+            -0.0022440897300839424,
+            -0.027134248986840248,
+            -0.0014052275801077485,
+            0.0038106779102236032,
+            -0.005826083943247795,
+            0.005744869355112314,
+            0.04459112882614136,
+            -0.004817312583327293,
+            -0.039940521121025085,
+            -0.00029493749025277793,
+            -0.03634997829794884,
+            0.0001610935723874718,
+            0.00010799681331263855,
+            -0.0033191153779625893,
+            0.005488402210175991,
+            0.0018422907451167703,
+            -0.012472864240407944,
+            0.012130907736718655,
+            -0.028741445392370224,
+            0.019782183691859245,
+            0.004817312583327293,
+            -0.007574336603283882,
+            -0.005056682042777538,
+            0.01988477073609829,
+            0.008151388727128506,
+            -0.013148227706551552,
+            -0.0014960597036406398,
+            0.0028852580580860376,
+            -0.026125477626919746,
+            -0.020773857831954956,
+            -0.03369981423020363,
+            -0.0022505014203488827,
+            -0.009583331644535065,
+            -0.03010927140712738,
+            -0.006937442813068628,
+            -0.0003617258626036346,
+            0.007775236386805773,
+            -0.020295118913054466,
+            0.0006956678116694093,
+            0.005552519112825394,
+            0.012199298478662968,
+            0.00624070642516017,
+            -0.007339241914451122,
+            -0.004263770300894976,
+            -0.03033154271543026,
+            0.003584131831303239,
+            -0.0015345298452302814,
+            0.0038619714323431253,
+            -0.016063407063484192,
+            0.006351842079311609,
+            -0.016183091327548027,
+            -0.0035264266189187765,
+            0.011335858143866062,
+            0.01800401136279106,
+            -0.028741445392370224,
+            0.036726128309965134,
+            0.02889532595872879,
+            0.017320098355412483,
+            -0.038914650678634644,
+            0.028587564826011658,
+            -0.0012331806356087327,
+            0.001522775157354772,
+            0.0030177661683410406,
+            0.021731335669755936,
+            0.026039987802505493,
+            -0.016431011259555817,
+            -0.006531369406729937,
+            0.010224499739706516,
+            0.028194313868880272,
+            -0.031545490026474,
+            -0.007266575936228037,
+            1.9886239897459745e-05,
+            0.02378307469189167,
+            0.027783967554569244,
+            0.023116260766983032,
+            0.04271036759018898,
+            -0.0053131491877138615,
+            0.002891669748350978,
+            -0.005082328803837299,
+            -0.013926179148256779,
+            0.0024727729614824057,
+            -0.0028767092153429985,
+            -0.0017952717607840896,
+            0.011489738710224628,
+            0.012695135548710823,
+            0.017593663185834885,
+            -0.07126373797655106,
+            0.038743674755096436,
+            0.026570022106170654,
+            0.012105260975658894,
+            0.008399306796491146,
+            -0.024706358090043068,
+            -0.0017909972229972482,
+            -0.0022205801215022802,
+            0.0018604571232572198,
+            0.013438890688121319,
+            -0.010874217376112938,
+            -0.02475765161216259,
+            0.016567792743444443,
+            -0.019457325339317322,
+            0.017149118706583977,
+            0.0050267609767615795,
+            0.00503958435729146,
+            -0.024603771045804024,
+            -0.014721227809786797,
+            0.006830581463873386,
+            0.008065898902714252,
+            0.008100095205008984,
+            -0.03857269510626793,
+            -0.044727910310029984,
+            0.01750817336142063,
+            0.008035978302359581,
+            -0.04862621799111366,
+            -0.006586937233805656,
+            -0.017388489097356796,
+            -0.004210339393466711,
+            -0.006326195318251848,
+            -0.01767915114760399,
+            -0.040897998958826065,
+            0.002064562402665615,
+            0.017302999272942543,
+            0.018534043803811073,
+            5.349749335437082e-05,
+            0.028826933354139328,
+            0.0017183314776048064,
+            -0.0022355406545102596,
+            -0.030434129759669304,
+            0.026860684156417847,
+            -0.007326418533921242,
+            -0.022278467193245888,
+            0.024142129346728325,
+            0.01628567837178707,
+            -0.019970260560512543,
+            -0.0038769319653511047,
+            0.0014928538585081697,
+            -0.008446325547993183,
+            0.034554705023765564,
+            -0.0017129883635789156,
+            -0.008933614008128643,
+            0.0026907704304903746,
+            -0.0020998266991227865,
+            -0.00959188025444746,
+            -0.02479184791445732,
+            0.00145438383333385,
+            0.025869010016322136,
+            -0.007749589625746012,
+            0.006437331438064575,
+            0.04202645644545555,
+            -0.012353179045021534,
+            -0.019850576296448708,
+            0.00043145293602719903,
+            -0.02839948795735836,
+            0.010318538174033165,
+            -0.006685249973088503,
+            0.030280249193310738,
+            0.004206065088510513,
+            0.013002896681427956,
+            -0.016644733026623726,
+            -0.01066904328763485,
+            -0.021970706060528755,
+            -0.0641852393746376,
+            -0.012122358195483685,
+            -0.005561067722737789,
+            -0.024894434958696365,
+            -0.02398824878036976,
+            0.03234908729791641,
+            0.022398151457309723,
+            0.010609201155602932,
+            -0.002107307082042098,
+            -0.01891019567847252,
+            -0.004458257928490639,
+            0.03009217418730259,
+            -0.005458480678498745,
+            0.017371391877532005,
+            -0.011617972515523434,
+            0.01627713069319725,
+            -0.006578388623893261,
+            0.0051336223259568214,
+            -0.053960736840963364,
+            -0.005167817696928978,
+            0.02983570657670498,
+            0.01993606425821781,
+            0.007240929175168276,
+            0.010019325651228428,
+            0.013293559662997723,
+            -0.007501671090722084,
+            -0.02764718420803547,
+            0.02525348775088787,
+            0.017132021486759186,
+            0.01991896703839302,
+            -0.015935173258185387,
+            0.007587159983813763,
+            0.011788951233029366,
+            0.00288098375312984,
+            0.000775279535446316,
+            0.0008281759219244123,
+            0.025133803486824036,
+            0.003549935994669795,
+            -0.009275570511817932,
+            0.015721451491117477,
+            -0.007163988891988993,
+            -0.021269695833325386,
+            -0.004233849234879017,
+            0.008959260769188404,
+            0.004035086836665869,
+            0.001731154858134687,
+            -0.0015922350576147437,
+            -0.008984907530248165,
+            0.02886112965643406,
+            0.03539250046014786,
+            0.03183615207672119,
+            -0.0057876138016581535,
+            -0.005137896630913019,
+            0.0016456657322123647,
+            0.003513603238388896,
+            -0.007659825962036848,
+            -0.0018294673645868897,
+            0.019799282774329185,
+            -0.030006684362888336,
+            -0.011276016011834145,
+            -0.009318315424025059,
+            0.016362618654966354,
+            0.003522152081131935,
+            0.04411239176988602,
+            0.02111581526696682,
+            0.006522820331156254,
+            -0.00557389110326767,
+            0.0037209142465144396,
+            0.004389866720885038,
+            0.029237281531095505,
+            -0.02282559685409069,
+            9.023110033012927e-05,
+            -0.0034217024222016335,
+            -0.017901424318552017,
+            0.019046977162361145,
+            -0.007300771772861481,
+            0.009540586732327938,
+            0.02205619588494301,
+            0.09212308377027512,
+            -0.0065527418628335,
+            0.022449444979429245,
+            -0.027852358296513557,
+            -0.024090835824608803,
+            0.00291517935693264,
+            0.006313371937721968,
+            0.002265461953356862,
+            0.022039096802473068,
+            0.0054499320685863495,
+            0.02525348775088787,
+            0.0272197388112545,
+            0.0009067190694622695,
+            0.006698073353618383,
+            -0.010292891412973404,
+            0.004118438810110092,
+            -0.013182424008846283,
+            -0.009549135342240334,
+            -0.009437999688088894,
+            0.008779733441770077,
+            0.019012782722711563,
+            0.00911314133554697,
+            0.007531592156738043,
+            -0.004084242973476648,
+            -0.015482081100344658,
+            0.026518728584051132,
+            0.010292891412973404,
+            -0.004475356079638004,
+            -0.015601766295731068,
+            -0.015114477835595608,
+            0.012404472567141056,
+            -0.006886149290949106,
+            -0.005894475616514683,
+            0.01721751131117344,
+            0.005069505423307419,
+            0.02865595556795597,
+            -0.03369981423020363,
+            0.008890869095921516,
+            0.014259587042033672,
+            -0.0010857119923457503,
+            0.007245203480124474,
+            0.021423576399683952,
+            -0.0022633245680481195,
+            0.0242789126932621,
+            -0.01468703243881464,
+            0.00874553807079792,
+            0.010959706269204617,
+            -0.021201303228735924,
+            0.0036055040545761585,
+            0.01990186981856823,
+            -0.003381095128133893,
+            0.015413690358400345,
+            -0.0052191112190485,
+            -0.001976936124265194,
+            0.0025903205387294292,
+            0.006582662928849459,
+            0.00363756250590086,
+            -0.01988477073609829,
+            0.01188298873603344,
+            -0.01872211880981922,
+            0.009172983467578888,
+            -0.004565119277685881,
+            0.012190749868750572,
+            -0.008424953557550907,
+            0.011370054446160793,
+            -0.006121021695435047,
+            0.006847679149359465,
+            -0.0105750048533082,
+            -0.016559245064854622,
+            -0.012515608221292496,
+            0.012960151769220829,
+            0.007249478250741959,
+            -0.013558575883507729,
+            0.008309543132781982,
+            -0.008228329010307789,
+            -0.002067768247798085,
+            0.014960597269237041,
+            -0.011746206320822239,
+            0.00407355697825551,
+            -0.039461780339479446,
+            -0.004535198211669922,
+            0.018089499324560165,
+            -0.001959838205948472,
+            0.0027719850186258554,
+            -0.007352065294981003,
+            0.011199075728654861,
+            -0.008288171142339706,
+            -0.0024086562916636467,
+            0.0254757609218359,
+            -0.019782183691859245,
+            -0.017328646034002304,
+            -0.032913316041231155,
+            -0.05009663105010986,
+            -0.002131885150447488,
+            -0.009326864033937454,
+            0.005864554084837437,
+            -0.00790346972644329,
+            0.017815934494137764,
+            -0.0040179891511797905,
+            -0.007095597684383392,
+            -0.024125032126903534,
+            0.010395478457212448,
+            0.028211412951350212,
+            0.0008682489860802889,
+            -0.03079318441450596,
+            -0.012344630435109138,
+            0.010463869199156761,
+            0.0006646779947914183,
+            -0.029049206525087357,
+            0.0083693852648139,
+            0.0006181932403706014,
+            -0.014473309740424156,
+            -0.008882320486009121,
+            -0.005971415434032679,
+            0.021440673619508743,
+            0.024193422868847847,
+            0.005753418430685997,
+            -0.019782183691859245,
+            -0.004225300159305334,
+            0.03258845582604408,
+            -0.018294673413038254,
+            0.003994479309767485,
+            0.023868564516305923,
+            -0.00264375121332705,
+            0.00982270110398531,
+            0.01054935809224844,
+            0.018072402104735374,
+            -0.008352288044989109,
+            0.012660940177738667,
+            0.007561513222754002,
+            -0.015328200533986092,
+            0.008548912592232227,
+            0.006668151821941137,
+            0.016405364498496056,
+            0.006890423595905304,
+            -0.022244270890951157,
+            -0.004411239176988602,
+            0.015832586213946342,
+            0.053276825696229935,
+            -0.006706621963530779,
+            -0.008506168611347675,
+            0.03617899864912033,
+            0.024415696039795876,
+            0.007416181731969118,
+            0.005937220063060522,
+            -0.03231488913297653,
+            -0.004646334331482649,
+            0.0008709204848855734,
+            0.005236208904534578,
+            0.0132422661408782,
+            0.0035029170103371143,
+            -0.007770961616188288,
+            0.018978586420416832,
+            -0.001043501659296453,
+            0.014293782413005829,
+            -0.020107043907046318,
+            0.009514939971268177,
+            -0.00618513859808445,
+            0.03161387890577316,
+            0.01042112521827221,
+            -0.0010632710764184594,
+            0.03318687900900841,
+            0.006719445344060659,
+            -0.03344334661960602,
+            0.006125296000391245,
+            -0.007638453505933285,
+            0.011352956295013428,
+            -0.02260332554578781,
+            -0.007894921116530895,
+            0.009899641387164593,
+            0.0012289062142372131,
+            -0.018123695626854897,
+            0.011900086887180805,
+            0.007142616901546717,
+            0.009378157556056976,
+            -0.0005217508296482265,
+            0.006356116849929094,
+            0.008553187362849712,
+            -0.004522374831140041,
+            -0.008168485946953297,
+            0.005599537864327431,
+            -0.016336971893906593,
+            -0.0040179891511797905,
+            0.029425358399748802,
+            -0.0006726926076225936,
+            0.010309988632798195,
+            0.0004135536728426814,
+            -0.0004116835771128535,
+            -0.018978586420416832,
+            0.009207178838551044,
+            -0.024569574743509293,
+            -0.006415958981961012,
+            0.005595263559371233,
+            -0.01225059200078249,
+            -0.006142393685877323,
+            0.014413466677069664,
+            0.0022633245680481195,
+            -0.021047422662377357,
+            -0.008061625063419342,
+            -0.030485423281788826,
+            -0.021218402311205864,
+            0.031904544681310654,
+            0.006518546026200056,
+            -0.005646557081490755,
+            0.015379494056105614,
+            -0.022876890376210213,
+            -0.0054969508200883865,
+            -0.005035309586673975,
+            0.016636185348033905,
+            -0.020021554082632065,
+            0.004398415796458721,
+            -0.010634847916662693,
+            -0.016431011259555817,
+            0.023099161684513092,
+            0.008326641283929348,
+            0.0011220448650419712,
+            0.006133845075964928,
+            -0.02497992292046547,
+            -0.014285233803093433,
+            0.01798691228032112,
+            0.005702124908566475,
+            -0.008228329010307789,
+            0.01795271784067154,
+            -0.0012609646655619144,
+            -0.007215282414108515,
+            -0.013780847191810608,
+            0.01841435767710209,
+            -0.02183392271399498,
+            0.026535825803875923,
+            0.012156554497778416,
+            -0.010472417809069157,
+            0.008822478353977203,
+            0.004992565140128136,
+            0.01287466287612915,
+            -0.0003889755462296307,
+            -0.015370945446193218,
+            0.013421793468296528,
+            0.013712456449866295,
+            0.002194933360442519,
+            -0.0021778354421257973,
+            0.0037572472356259823,
+            0.005826083943247795,
+            -0.0054456572979688644,
+            0.02333853207528591,
+            0.023594999685883522,
+            0.0002186651690863073,
+            0.015080282464623451,
+            -0.02084225043654442,
+            -0.019611205905675888,
+            0.02378307469189167,
+            -0.004595040809363127,
+            -0.0314599983394146,
+            0.038709476590156555,
+            -0.010292891412973404,
+            -0.007236654870212078,
+            -0.01993606425821781,
+            0.00771111948415637,
+            0.020448999479413033,
+            -0.021885216236114502,
+            -0.010002228431403637,
+            -0.00110708421561867,
+            0.008848125115036964,
+            0.027852358296513557,
+            0.020568683743476868,
+            -0.015507727861404419,
+            0.02788655459880829,
+            0.009668820537626743,
+            0.008044526912271976,
+            0.011874440126121044,
+            0.005561067722737789,
+            -0.010840021073818207,
+            0.010044972412288189,
+            -0.0005241552134975791,
+            0.005296051502227783,
+            -0.00516354339197278,
+            -0.02231266349554062,
+            -0.015644511207938194,
+            -0.04178708791732788,
+            -0.0036589347291737795,
+            -0.008604480884969234,
+            0.01553337462246418,
+            0.08624143153429031,
+            -0.014037314802408218,
+            -0.009309765882790089,
+            -0.0020271609537303448,
+            0.009976581670343876,
+            -0.01720896176993847,
+            0.012566901743412018,
+            0.024466989561915398,
+            -0.013626967556774616,
+            -0.0329304113984108,
+            0.0025518503971397877,
+            -0.012994347140192986,
+            0.007805157452821732,
+            0.017542369663715363,
+            0.0019245740259066224,
+            -0.023680489510297775,
+            -0.0018433593213558197,
+            -0.020004456862807274,
+            0.001239592325873673,
+            0.010369831696152687,
+            -0.016131797805428505,
+            -0.010771630331873894,
+            0.023902760818600655,
+            -0.02017543464899063,
+            -0.025167999789118767,
+            -0.012925956398248672,
+            -0.007745314855128527,
+            -0.011891537345945835,
+            0.017815934494137764,
+            0.008647225797176361,
+            -0.0186708252876997,
+            -0.008595932275056839,
+            0.005915847606956959,
+            -0.011378603056073189,
+            8.963001164374873e-05,
+            0.011925733648240566,
+            -0.0033169782254844904,
+            0.008788282051682472,
+            0.01482381485402584,
+            -0.03400757536292076,
+            -0.003128902055323124,
+            -0.014704130589962006,
+            0.01360986940562725,
+            0.006557016167789698,
+            0.028057532384991646,
+            -0.028279803693294525,
+            -0.021765531972050667,
+            -0.00981415156275034,
+            -0.01892729289829731,
+            -0.03082738071680069,
+            -0.02692907489836216,
+            0.0058175353333354,
+            0.00636466545984149,
+            0.021526163443922997,
+            0.014729776419699192,
+            -0.014396369457244873,
+            0.010096265934407711,
+            -0.0033981928136199713,
+            0.008984907530248165,
+            -0.014404918067157269,
+            -0.0018454965902492404,
+            -0.015550472773611546,
+            0.015131575986742973,
+            0.017662053927779198,
+            -0.002583908848464489,
+            -0.018380163237452507,
+            -0.019730890169739723,
+            0.010429673828184605,
+            -0.007604258134961128,
+            -0.004637785255908966,
+            0.02668970637023449,
+            0.003182332729920745,
+            0.0019096133764833212,
+            0.007527317851781845,
+            0.023406922817230225,
+            0.008664323017001152,
+            0.03546088933944702,
+            -0.023406922817230225,
+            0.001999377040192485,
+            -0.0189443901181221,
+            0.0031866072677075863,
+            -0.023218847811222076,
+            0.005112249869853258,
+            -0.01942313089966774,
+            0.03417855501174927,
+            -0.006133845075964928,
+            -0.007852176204323769,
+            0.030468326061964035,
+            -0.02624516189098358,
+            0.002979296026751399,
+            -0.009754309430718422,
+            0.004650608636438847,
+            0.022141683846712112,
+            -0.0038256384432315826,
+            -0.017285902053117752,
+            -0.03925660625100136,
+            0.003522152081131935,
+            -0.009899641387164593,
+            -0.006091100629419088,
+            -0.011318760924041271,
+            0.01773044466972351,
+            0.005065230652689934,
+            0.005569616798311472,
+            -0.0015633824514225125,
+            0.016892651095986366,
+            0.0013913356233388186,
+            0.0035456616897135973,
+            0.029955390840768814,
+            -0.001200053608044982,
+            0.012558353133499622,
+            0.0033533109817653894,
+            0.005689301528036594,
+            0.023492412641644478,
+            -0.007745314855128527,
+            -0.0015206378884613514,
+            -0.017362842336297035,
+            0.0067237201146781445,
+            -0.0007555101765319705,
+            0.02520219422876835,
+            0.01539659220725298,
+            -0.01771334744989872,
+            -0.040658630430698395,
+            0.02159455418586731,
+            -0.011352956295013428,
+            0.027373619377613068,
+            -0.009497841820120811,
+            -0.006569839548319578,
+            -0.0003144397051073611,
+            -0.03754682466387749,
+            0.003182332729920745,
+            -0.009720114059746265,
+            -0.052798084914684296,
+            -0.018089499324560165,
+            -0.007548689842224121,
+            -0.026296455413103104,
+            0.03342624753713608,
+            -0.0014650699449703097,
+            -0.026775194332003593,
+            0.024706358090043068,
+            -0.0034516234882175922,
+            -0.00698873633518815,
+            0.012233494780957699,
+            -0.016704576089978218,
+            -0.009378157556056976,
+            -0.0032443124800920486,
+            -0.005116524174809456,
+            0.0006069728406146169,
+            -0.012421570718288422,
+            -0.019235054031014442,
+            0.015866782516241074,
+            -0.006980187259614468,
+            0.0008538226829841733,
+            0.015866782516241074,
+            -0.0020496018696576357,
+            0.025065412744879723,
+            -0.014122803695499897,
+            0.005385815165936947,
+            0.007612806744873524,
+            -0.013883434236049652,
+            0.012301885522902012,
+            0.003847010899335146,
+            -0.015430787578225136,
+            0.01580693945288658,
+            0.009215728379786015,
+            0.00848907046020031,
+            0.0041611832566559315,
+            -0.014533151872456074,
+            0.0022889713291078806,
+            -0.011421347968280315,
+            0.006646779831498861,
+            0.013909080997109413,
+            0.005304600577801466,
+            -0.011276016011834145,
+            -0.016653282567858696,
+            0.013857787474989891,
+            0.007501671090722084,
+            0.020004456862807274,
+            -0.03180195763707161,
+            0.007386260665953159,
+            -0.004676255397498608,
+            0.006941717118024826,
+            -0.01227623876184225,
+            0.025287684053182602,
+            0.022620422765612602,
+            0.02111581526696682,
+            0.006356116849929094,
+            0.0028767092153429985,
+            0.010515162721276283,
+            -0.01966249942779541,
+            -0.0033618600573390722,
+            -0.0204148031771183,
+            -0.0006417027907446027,
+            -0.0008324504015035927,
+            0.01743123307824135,
+            -0.012318983674049377,
+            0.013105483725667,
+            0.006522820331156254,
+            0.015447885729372501,
+            -0.02887822687625885,
+            0.0039047158788889647,
+            0.0070400298573076725,
+            0.0013421793701127172,
+            0.0066211330704391,
+            0.0031224903650581837,
+            0.08371095359325409,
+            0.010472417809069157,
+            -0.0070870486088097095,
+            0.003968833014369011,
+            -0.009437999688088894,
+            -0.0077538639307022095,
+            0.015251260250806808,
+            -0.01081437524408102,
+            0.012233494780957699,
+            0.007347790524363518,
+            -0.013148227706551552,
+            0.004582217428833246,
+            0.0073563395999372005,
+            0.008108643814921379,
+            -0.01623438484966755,
+            0.011190527118742466,
+            -0.005253307055681944,
+            0.0038961670361459255,
+            0.025133803486824036,
+            0.01652504876255989,
+            -0.005723497364670038,
+            0.00971156544983387,
+            0.006467252504080534,
+            0.0009863307932391763,
+            -0.012404472567141056,
+            0.006390312220901251,
+            -0.004152634646743536,
+            0.023013673722743988,
+            -0.016208738088607788,
+            0.0344863161444664,
+            -0.0005706399097107351,
+            -0.003855559742078185,
+            -0.010882765986025333,
+            -0.003383232280611992,
+            0.007501671090722084,
+            -0.008322366513311863,
+            0.01114778220653534,
+            -0.013712456449866295,
+            0.0005973553052172065,
+            -0.013045640662312508,
+            -0.008189858868718147,
+            0.004058596212416887,
+            -0.007540141232311726,
+            -0.0073777115903794765,
+            -0.0009211453725583851,
+            0.001756801619194448,
+            -4.190970867057331e-05,
+            -0.0012331806356087327,
+            -0.0012919544242322445,
+            -0.009437999688088894,
+            -0.004513826221227646,
+            -0.013601320795714855,
+            -1.5010688912298065e-05,
+            0.02137228287756443,
+            -0.003682444104924798,
+            -0.006809209007769823,
+            0.01044677197933197,
+            0.003128902055323124,
+            0.02429600991308689,
+            -0.006095374934375286,
+            -0.014558798633515835,
+            -0.0021564632188528776,
+            -0.007005834020674229,
+            -0.007390534970909357,
+            0.00033928497578017414,
+            0.0038790691178292036,
+            0.006180863827466965,
+            0.003287056926637888,
+            0.012438667938113213,
+            0.006018434651196003,
+            -0.008831026963889599,
+            0.011763304471969604,
+            0.003120353212580085,
+            -0.0009190081618726254,
+            -0.01891019567847252,
+            -0.003242175094783306,
+            -0.013550027273595333,
+            -0.01199412439018488,
+            -0.011455543339252472,
+            0.017345745116472244,
+            -0.01723460853099823,
+            0.019765086472034454,
+            -0.006339018698781729,
+            0.00765555165708065,
+            -0.02549285814166069,
+            0.002449263585731387,
+            0.008899418637156487,
+            0.011250369250774384,
+            0.006702347658574581,
+            -0.012857564724981785,
+            -0.002013268880546093,
+            -0.0010862462222576141,
+            0.02963053248822689,
+            -0.008702793158590794,
+            -0.010010777041316032,
+            0.02816011942923069,
+            -0.014413466677069664,
+            -0.01019030436873436,
+            0.0014362173387780786,
+            -0.0013090522261336446,
+            0.02552705444395542,
+            0.011455543339252472,
+            0.017328646034002304,
+            0.0054456572979688644,
+            0.002840376226231456,
+            0.0013379048323258758,
+            0.023936955258250237,
+            0.016071956604719162,
+            0.010908412747085094,
+            -0.01418264675885439,
+            -0.008446325547993183,
+            -0.01942313089966774,
+            0.014216842129826546,
+            0.0033746834378689528,
+            -0.01530255377292633,
+            0.011224722489714622,
+            -0.021953608840703964,
+            -0.019064076244831085,
+            -0.023697586730122566,
+            -0.010386928915977478,
+            0.013438890688121319,
+            -0.011686364188790321,
+            -0.005505499895662069,
+            -0.017089277505874634,
+            -0.005454206373542547,
+            -0.010455320589244366,
+            -0.015148673206567764,
+            -0.01673022285103798,
+            -0.02598869428038597,
+            -0.022688815370202065,
+            -0.0098397983238101,
+            -0.017294451594352722,
+            -0.008300994522869587,
+            0.012797722592949867,
+            -0.020448999479413033,
+            0.017662053927779198,
+            -0.005065230652689934,
+            0.0037893056869506836,
+            0.01188298873603344,
+            0.0216116514056921,
+            -0.01748252660036087,
+            -0.031904544681310654,
+            -0.02402244508266449,
+            -0.02135518379509449,
+            0.022705912590026855,
+            0.0021927962079644203,
+            0.024142129346728325,
+            -0.007125518750399351,
+            0.016157444566488266,
+            -0.018363064154982567,
+            0.019303444772958755,
+            -0.04202645644545555,
+            -0.03222940117120743,
+            -0.007168263662606478,
+            0.00034836819395422935,
+            -0.004362082574516535,
+            -0.0010199921671301126,
+            -0.015661608427762985,
+            -0.010993901640176773,
+            -0.007168263662606478,
+            -0.006210785359144211,
+            -0.009685918688774109,
+            -0.01516577135771513,
+            0.01007916871458292,
+            0.004370631650090218,
+            0.016165994107723236,
+            0.001985484967008233,
+            0.015559021383523941,
+            -0.01516577135771513,
+            0.006945991888642311,
+            -0.0022269918117672205,
+            -0.005753418430685997,
+            0.004646334331482649,
+            0.003804266219958663,
+            0.0044924537651240826,
+            0.0002427089784760028,
+            0.0011017411015927792,
+            -0.00029386888490989804,
+            0.017362842336297035,
+            0.013361950404942036,
+            -0.007146891206502914,
+            0.01773044466972351,
+            6.485151970991865e-05,
+            0.0016787927597761154,
+            -0.010429673828184605,
+            -0.003214391181245446,
+            -0.00911314133554697,
+            -0.008882320486009121,
+            0.015362396836280823,
+            -0.019730890169739723,
+            -0.010626298375427723,
+            0.026843586936593056,
+            -0.0018348104786127806,
+            -0.03443502262234688,
+            0.02742491289973259,
+            7.86015789344674e-06,
+            -0.0024364402052015066,
+            -0.01369535829871893,
+            -0.007078499998897314,
+            -0.0006363596767187119,
+            -0.0005978895933367312,
+            0.017337195575237274,
+            -0.018038205802440643,
+            -0.005249032285064459,
+            0.00630909763276577,
+            -0.0040436359122395515,
+            -0.0021254734601825476,
+            0.006497173570096493,
+            0.014028766192495823,
+            -0.00047820480540394783,
+            -0.03501634672284126,
+            -0.012190749868750572,
+            0.020568683743476868,
+            -0.017106374725699425,
+            -0.014994793571531773,
+            0.011233271099627018,
+            -0.001976936124265194,
+            -0.012601097114384174,
+            -0.003911127801984549,
+            -0.005223385989665985,
+            -0.010993901640176773,
+            0.008442051708698273,
+            0.010224499739706516,
+            0.008339464664459229,
+            0.02210748940706253,
+            -0.014447662979364395,
+            -0.005757692735642195,
+            -0.015251260250806808,
+            0.0031075298320502043,
+            -0.022962380200624466,
+            -0.00015214394079521298,
+            -0.009745760820806026,
+            0.0016766554908826947,
+            0.012575451284646988,
+            0.01749962382018566,
+            0.028724348172545433,
+            0.0089678093791008,
+            0.0072879483923316,
+            -0.010848570615053177,
+            -0.01456734724342823,
+            0.004680529702454805,
+            0.0031417254358530045,
+            -0.00558671448379755,
+            0.006655328441411257,
+            0.019286347553133965,
+            0.0013261501444503665,
+            0.019303444772958755,
+            0.0025133804883807898,
+            0.0037978545296937227,
+            -2.8652215405600145e-05,
+            -0.010455320589244366,
+            -0.006706621963530779,
+            -0.02374888025224209,
+            0.0007453583530150354,
+            0.005779065191745758,
+            0.01889309659600258,
+            0.0005540763959288597,
+            0.0030370012391358614,
+            0.026416141539812088,
+            -0.0029771588742733,
+            -0.014345075935125351,
+            -0.012592548504471779,
+            -0.011942830868065357,
+            -0.00874553807079792,
+            0.034332435578107834,
+            0.01454170048236847,
+            -0.01454170048236847,
+            -0.015234163030982018,
+            -0.011566678993403912,
+            0.03033154271543026,
+            0.00442833686247468,
+            -0.0004928982816636562,
+            0.021902315318584442,
+            0.018294673413038254,
+            0.017345745116472244,
+            0.016362618654966354,
+            0.009420901536941528,
+            0.0027912200894206762,
+            0.020466096699237823,
+            0.0019480835180729628,
+            0.015088831074535847,
+            -0.008172760717570782,
+            0.008642951026558876,
+            -0.005864554084837437,
+            0.012712233699858189,
+            -0.010677591897547245,
+            -0.011797499842941761,
+            0.008433502167463303,
+            -0.014003119431436062,
+            0.002117993077263236,
+            -0.007373437285423279,
+            0.00948074460029602,
+            -0.02017543464899063,
+            -0.007599983364343643,
+            0.012190749868750572,
+            -0.014533151872456074,
+            0.0024043817538768053,
+            -0.022893989458680153,
+            -0.0009761789697222412,
+            0.014695581048727036,
+            -0.0025368898641318083,
+            0.0066125839948654175,
+            0.005321698263287544,
+            0.00012549536768347025,
+            0.010104815475642681,
+            -0.014789619483053684,
+            0.004030812531709671,
+            -0.005377266090363264,
+            -0.004834410268813372,
+            -0.00637748884037137,
+            -0.0011712011182680726,
+            0.0009542724001221359,
+            -0.005556793417781591,
+            0.01991896703839302,
+            0.008091545663774014,
+            0.00435994565486908,
+            -0.005270404741168022,
+            -0.012831917963922024,
+            -0.026159673929214478,
+            0.0145758967846632,
+            -0.015447885729372501,
+            0.01528545655310154,
+            0.0013550026342272758,
+            -0.003092569299042225,
+            -0.014370722696185112,
+            0.010643396526575089,
+            0.0056337337009608746,
+            0.00474037230014801,
+            0.015114477835595608,
+            -0.011481190100312233,
+            0.009583331644535065,
+            0.009147336706519127,
+            -0.009267021901905537,
+            -0.0067493668757379055,
+            0.0005936151137575507,
+            0.006279176566749811,
+            -0.0232701413333416,
+            -0.0016862730262801051,
+            0.008292445912957191,
+            -0.01962830312550068,
+            0.008762635290622711,
+            -0.03730745613574982,
+            0.0020068571902811527,
+            -0.012088162824511528,
+            0.011258917860686779,
+            -0.002776259556412697,
+            -0.033374954015016556,
+            -0.026518728584051132,
+            0.0016745183384045959,
+            0.0040179891511797905,
+            -0.009993678890168667,
+            0.009010554291307926,
+            0.009275570511817932,
+            0.011532483622431755,
+            0.01443056482821703,
+            0.0098397983238101,
+            0.010113364085555077,
+            0.04223163053393364,
+            -0.04151352122426033,
+            -0.007146891206502914,
+            0.0070656766183674335,
+            0.02209039032459259,
+            -0.010267244651913643,
+            -0.011361504904925823,
+            0.008685695938766003,
+            -0.003859834047034383,
+            0.034349530935287476,
+            -0.024877335876226425,
+            0.014994793571531773,
+            -0.0040543219074606895,
+            -0.00010605995339574292,
+            -0.024159228429198265,
+            -0.0020325039513409138,
+            -0.01969669573009014,
+            0.0032357636373490095,
+            -0.002079523168504238,
+            -0.017191864550113678,
+            0.018021108582615852,
+            -0.004595040809363127,
+            0.012789173983037472,
+            0.007813706062734127,
+            -0.01625148393213749,
+            0.004526649136096239,
+            -0.015593216754496098,
+            0.032896216958761215,
+            -0.012105260975658894,
+            0.005911573302000761,
+            0.00873698852956295,
+            -0.006779287941753864,
+            0.016439558938145638,
+            0.009352510794997215,
+            -0.006044081412255764,
+            -0.0051592690870165825,
+            -0.002759161638095975,
+            -0.0023680487647652626,
+            0.008493345230817795,
+            -0.021474869921803474,
+            0.027493303641676903,
+            -0.006894698366522789,
+            -0.014037314802408218,
+            0.006266353186219931,
+            -0.001722605898976326,
+            0.002626653527840972,
+            0.004406964406371117,
+            -0.0018059578724205494,
+            -0.0007939802599139512,
+            -0.004958369303494692,
+            -0.017798837274312973,
+            -0.010130461305379868,
+            0.0010467075044289231,
+            -0.0016114701284095645,
+            0.013105483725667,
+            -0.010540809482336044,
+            -0.016601989045739174,
+            0.005112249869853258,
+            0.00026929075829684734,
+            -0.003975244704633951,
+            -0.016807163134217262,
+            0.007215282414108515,
+            -0.007544415537267923,
+            0.005372991785407066,
+            -0.008168485946953297,
+            0.016448108479380608,
+            0.008313817903399467,
+            -0.0003630616411101073,
+            0.01767915114760399,
+            0.024706358090043068,
+            0.02061997726559639,
+            0.01152393501251936,
+            -0.040214087814092636,
+            -0.0189443901181221,
+            -0.0025091059505939484,
+            0.018807608634233475,
+            -0.0008981701685115695,
+            -0.006420233752578497,
+            -0.0017632133094593883,
+            0.0029835705645382404,
+            -0.003084020223468542,
+            -0.0007474955637007952,
+            0.016918297857046127,
+            -0.020192531868815422,
+            0.0032742335461080074,
+            -0.012720782309770584,
+            0.014404918067157269,
+            0.0006860502762719989,
+            -0.0006737611838616431,
+            5.767176844528876e-05,
+            0.00945509783923626,
+            0.014900755137205124,
+            0.014661385677754879,
+            0.0014159136917442083,
+            0.019064076244831085,
+            -0.003663209266960621,
+            -0.0016510088462382555,
+            -0.018636630848050117,
+            -0.01696104370057583,
+            -0.0034494863357394934,
+            0.01298579853028059,
+            -0.0015601766062900424,
+            0.011575227603316307,
+            -0.014191195368766785,
+            0.01649085246026516,
+            0.025680935010313988,
+            -0.012840467505156994,
+            0.005069505423307419,
+            0.002564673777669668,
+            -0.001211808412335813,
+            0.0064800758846104145,
+            -0.007381986360996962,
+            -0.004030812531709671,
+            -0.012438667938113213,
+            0.01868792437016964,
+            -0.0032208028715103865,
+            0.018277576193213463,
+            0.013635516166687012,
+            0.011968477629125118,
+            -0.00032993461354635656,
+            0.008305269293487072,
+            0.018636630848050117,
+            0.014413466677069664,
+            -0.0005158734857104719,
+            0.0011615835828706622,
+            -0.01284901611506939,
+            0.0023594999220222235,
+            0.016311325132846832,
+            -0.02715134806931019,
+            -0.007685472723096609,
+            0.011840244755148888,
+            -0.016567792743444443,
+            0.0025603994727134705,
+            -0.02597159706056118,
+            -0.011472641490399837,
+            0.012053967453539371,
+            0.010233048349618912,
+            -0.011053744703531265,
+            0.001883966731838882,
+            0.001705508097074926,
+            -0.0008377934573218226,
+            -0.00936105940490961,
+            0.0167815163731575,
+            -0.005868828855454922,
+            -0.011258917860686779,
+            0.011686364188790321,
+            -0.00981415156275034,
+            0.012695135548710823,
+            0.004603589419275522,
+            0.011746206320822239,
+            -0.0024578124284744263,
+            0.026877781376242638,
+            0.01287466287612915,
+            0.007236654870212078,
+            0.0004386660875752568,
+            0.005937220063060522,
+            -0.004847233649343252,
+            0.007407633122056723,
+            0.026570022106170654,
+            -0.00048728802357800305,
+            -0.0116350706666708,
+            -0.0013197384541854262,
+            0.010951157659292221,
+            -0.01988477073609829,
+            0.0023124809376895428,
+            0.005676478147506714,
+            0.028211412951350212,
+            0.014327977783977985,
+            0.0025796345435082912,
+            -0.0204148031771183,
+            -0.00023709874949418008,
+            -0.00041595802758820355,
+            0.04106897860765457,
+            0.0026116929948329926,
+            -0.012532706372439861,
+            0.0021489830687642097,
+            0.013849238865077496,
+            -0.005590988788753748,
+            -0.004834410268813372,
+            -0.014310880564153194,
+            -0.008027428761124611,
+            -0.006018434651196003,
+            -0.0198676735162735,
+            0.001279131043702364,
+            0.005800437182188034,
+            -0.013319206424057484,
+            0.007471750024706125,
+            -0.006403135601431131,
+            -0.03031444549560547,
+            -0.005932945758104324,
+            -0.0034131535794585943,
+            -0.023731783032417297,
+            -0.007125518750399351,
+            0.0089678093791008,
+            0.005915847606956959,
+            -0.014122803695499897,
+            -0.025338977575302124,
+            0.020996129140257835,
+            0.007668375037610531,
+            0.03392208740115166,
+            -0.012669488787651062,
+            -0.007185361348092556,
+            -0.00948074460029602,
+            0.008557462133467197,
+            -0.023150455206632614,
+            -0.008715616539120674,
+            0.009061847813427448,
+            0.01767915114760399,
+            0.012404472567141056,
+            -0.011370054446160793,
+            0.0015163634670898318,
+            0.012122358195483685,
+            -0.00010085045505547896,
+            0.02764718420803547,
+            -0.009104591794312,
+            -0.014994793571531773,
+            0.010754532180726528,
+            -0.003259273013100028,
+            0.01916666328907013,
+            -0.0006203305092640221,
+            -0.0025924579240381718,
+            0.012917407788336277,
+            0.0038341875188052654,
+            -0.012934505008161068,
+            -0.005240483675152063,
+            0.008647225797176361,
+            -0.028553368523716927,
+            0.02792074903845787,
+            0.015909526497125626,
+            -0.02816011942923069,
+            -0.008305269293487072,
+            -0.0025091059505939484,
+            0.017140571027994156,
+            0.020089944824576378,
+            -0.011446994729340076,
+            -0.009728662669658661,
+            0.014960597269237041,
+            0.0034217024222016335,
+            0.0077538639307022095,
+            -0.0026565745938569307,
+            0.01896148920059204,
+            -0.004370631650090218,
+            -0.007800883147865534,
+            -0.003408879041671753,
+            -0.012789173983037472,
+            -0.0003505054337438196,
+            0.015618863515555859,
+            0.012977249920368195,
+            -0.005984238814562559,
+            -0.008771184831857681,
+            0.015208516269922256,
+            -0.022432347759604454,
+            0.0010841090697795153,
+            0.010908412747085094,
+            -0.01745687983930111,
+            -0.007561513222754002,
+            0.01767915114760399,
+            -0.0031395882833749056,
+            0.011258917860686779,
+            -0.007381986360996962,
+            0.007856450974941254,
+            0.0067237201146781445,
+            0.031135139986872673,
+            -0.007476024329662323,
+            0.03525571525096893,
+            -0.009309765882790089,
+            0.01991896703839302,
+            0.037991367280483246,
+            0.01455025002360344,
+            -0.0031118043698370457,
+            -0.008335189893841743,
+            0.009198630228638649,
+            -0.010096265934407711,
+            0.002415067981928587,
+            0.013524380512535572,
+            -0.026809390634298325,
+            -0.01418264675885439,
+            -0.011378603056073189,
+            0.013233717530965805,
+            0.0034580351784825325,
+            -0.013661162927746773,
+            -0.011575227603316307,
+            0.019508618861436844,
+            0.010352733545005322,
+            -0.0038705202750861645,
+            -0.007869274355471134,
+            -0.02157745696604252,
+            0.0020688369404524565,
+            0.012088162824511528,
+            -0.009164434857666492,
+            -0.011532483622431755,
+            0.016670379787683487,
+            -0.01261819526553154,
+            0.0027228286489844322,
+            -0.008300994522869587,
+            -0.027989141643047333,
+            -0.005800437182188034,
+            -0.00491989916190505,
+            -0.0019844165071845055,
+            -0.005565342493355274,
+            0.006471527274698019,
+            0.005227660294622183,
+            -0.01824337989091873,
+            0.0024278913624584675,
+            -0.008788282051682472,
+            0.00861302949488163,
+            0.00436635734513402,
+            0.006249255500733852,
+            -0.0019384659826755524,
+            0.014225390739738941,
+            0.013661162927746773,
+            -0.005377266090363264,
+            -0.020773857831954956,
+            0.012772075831890106,
+            -0.03634997829794884,
+            0.020722564309835434,
+            0.008694244548678398,
+            0.0025411644019186497,
+            -0.03086157515645027,
+            0.01018175482749939,
+            0.01115633174777031,
+            0.0160035640001297,
+            0.011370054446160793,
+            0.0016766554908826947,
+            0.013190972618758678,
+            -0.01114778220653534,
+            0.006766464561223984,
+            -7.192273642431246e-06,
+            -0.01482381485402584,
+            0.020688369870185852,
+            0.011814597994089127,
+            0.02913469448685646,
+            0.002951512113213539,
+            -0.0011209761723876,
+            0.001963044051080942,
+            0.00899345614016056,
+            -0.007685472723096609,
+            0.0030904319137334824,
+            0.004954094998538494,
+            0.0058175353333354,
+            0.013190972618758678,
+            0.007745314855128527,
+            -0.00860875565558672,
+            0.007839352823793888,
+            0.0026736725121736526,
+            -0.014020216651260853,
+            -0.000775279535446316,
+            -0.006997285410761833,
+            0.007552964612841606,
+            -0.01505463570356369,
+            0.004560844972729683,
+            0.04342847689986229,
+            0.012413021177053452,
+            -0.0034323884174227715,
+            -0.017354292795062065,
+            -0.011079391464591026,
+            -0.01273788046091795,
+            -0.01272933091968298,
+            -0.0053644427098333836,
+            -0.02303077094256878,
+            -0.003434525802731514,
+            0.0065912120044231415,
+            0.02453538030385971,
+            0.01092551089823246,
+            -0.017191864550113678,
+            -0.005813260562717915,
+            -0.02380017377436161,
+            -0.010566456243395805,
+            -0.016191640868782997,
+            -0.016319874674081802,
+            -0.008061625063419342,
+            0.018978586420416832,
+            -0.008839575573801994,
+            0.01408860832452774,
+            0.002968610031530261,
+            -0.0002393695613136515,
+            -0.0020431901793926954,
+            -0.000852754048537463,
+            -0.0051549943163990974,
+            0.0015911663649603724,
+            0.003152411663904786,
+            -0.0005375128821469843,
+            0.020722564309835434,
+            -0.007800883147865534,
+            0.003866245737299323,
+            -0.0026929075829684734,
+            0.01542223896831274,
+            0.007108421064913273,
+            0.0028681603725999594,
+            -0.0238172709941864,
+            0.02304787002503872,
+            -0.024381499737501144,
+            -0.014156999997794628,
+            0.02284269593656063,
+            -0.008108643814921379,
+            0.004804489202797413,
+            0.024107934907078743,
+            0.00029493749025277793,
+            -0.0009697672794573009,
+            0.015123026445508003,
+            0.004045772831887007,
+            0.011258917860686779,
+            0.015242711640894413,
+            0.007082774303853512,
+            -0.0052875024266541,
+            0.019098270684480667,
+            0.0073306928388774395,
+            -0.006403135601431131,
+            -0.01626003161072731,
+            0.0014244626509025693,
+            0.004238123539835215,
+            -0.010044972412288189,
+            -0.004137673880904913,
+            0.008630127646028996,
+            0.004453983623534441,
+            -0.007843627594411373,
+            -0.0012972975382581353,
+            -0.024706358090043068,
+            -0.005821809638291597,
+            -0.01530255377292633,
+            -0.006039807107299566,
+            -0.023406922817230225,
+            0.005676478147506714,
+            -0.009053298272192478,
+            0.01672167330980301,
+            0.00016403225890826434,
+            0.024347303435206413,
+            0.0020154062658548355,
+            -0.0007266575703397393,
+            -0.0051592690870165825,
+            0.009318315424025059,
+            -0.0007266575703397393,
+            0.027236836031079292,
+            -0.00011026761785615236,
+            -0.005911573302000761,
+            0.01032708678394556,
+            -0.01769625023007393,
+            0.020876444876194,
+            0.006356116849929094,
+            0.01454170048236847,
+            -0.015610314905643463,
+            -0.0073563395999372005,
+            0.035289913415908813,
+            0.024843141436576843,
+            0.0035029170103371143,
+            0.019816379994153976,
+            -0.010455320589244366,
+            0.016311325132846832,
+            0.003013491863384843,
+            -0.002389421220868826,
+            0.011010999791324139,
+            -0.005420010536909103,
+            -0.014627190306782722,
+            -0.006997285410761833,
+            0.03911982476711273,
+            -0.014832363463938236,
+            -6.972707342356443e-05,
+            -0.005060956347733736,
+            -0.010104815475642681,
+            -0.030929965898394585,
+            0.011609423905611038,
+            0.011951380409300327,
+            -0.024706358090043068,
+            -0.011566678993403912,
+            0.0032699592411518097,
+            -0.00026448199059814215,
+            -0.0030348640866577625,
+            -0.005706399213522673,
+            0.006509996950626373,
+            -0.0017375665483996272,
+            0.009275570511817932,
+            0.005569616798311472,
+            -0.0009617527248337865,
+            0.00013798479631077498,
+            0.005539695732295513,
+            0.011780401691794395,
+            0.006206510588526726,
+            0.0005498019745573401,
+            -0.007381986360996962,
+            0.00558671448379755,
+            0.010745983570814133,
+            -0.015088831074535847,
+            0.020739663392305374,
+            0.013823592104017735,
+            -0.016576342284679413,
+            0.0038149524480104446,
+            -0.019132466986775398,
+            -0.0329304113984108,
+            0.004787391051650047,
+            -0.0009029789362102747,
+            0.021748434752225876,
+            -0.018311770632863045,
+            -0.00995948351919651,
+            0.00442833686247468,
+            0.003908990416675806,
+            -0.011421347968280315,
+            0.019371837377548218,
+            0.007215282414108515,
+            -0.0017664191545918584,
+            0.01745687983930111,
+            0.016593439504504204,
+            -0.006403135601431131,
+            -0.00907894503325224,
+            0.018824705854058266,
+            -0.00437276903539896,
+            0.02744201011955738,
+            0.026621313765645027,
+            0.009207178838551044,
+            -0.018602434545755386,
+            0.0018262615194544196,
+            0.0080231549218297,
+            -0.02036350965499878,
+            -0.005590988788753748,
+            0.015670157968997955,
+            0.005402912851423025,
+            -0.0003841667785309255,
+            -0.0050737797282636166,
+            -0.022175880149006844,
+            0.007407633122056723,
+            0.011840244755148888,
+            -0.005189190153032541,
+            -0.028809836134314537,
+            -0.0011199075961485505,
+            -0.01431942917406559,
+            -0.0007694021333009005,
+            0.0010648739989846945,
+            0.01236172765493393,
+            0.009036201052367687,
+            0.028074629604816437,
+            -0.009865445084869862,
+            -0.00885667372494936,
+            0.010173206217586994,
+            0.009797054342925549,
+            0.012583999894559383,
+            0.012327532283961773,
+            0.013986021280288696,
+            0.007292222697287798,
+            -0.015960820019245148,
+            -0.004966918379068375,
+            0.007527317851781845,
+            -0.00899345614016056,
+            0.003985930699855089,
+            -0.01358422264456749,
+            0.020278021693229675,
+            0.015259809792041779,
+            0.0008260387112386525,
+            0.004804489202797413,
+            -0.004753195680677891,
+            -0.0012684449320659041,
+            -0.06483495235443115,
+            -0.013532929122447968,
+            0.00861302949488163,
+            0.021252596750855446,
+            -0.016396814957261086,
+            0.00046244275290519,
+            0.0014287370722740889,
+            0.008788282051682472,
+            -0.011481190100312233,
+            0.01020740158855915,
+            0.014524603262543678,
+            -0.005223385989665985,
+            -0.0035349754616618156,
+            0.005099426489323378,
+            -0.0016403226181864738,
+            -0.00013371034583542496,
+            0.004958369303494692,
+            0.014900755137205124,
+            0.021406477317214012,
+            -0.004385592415928841,
+            0.018585337325930595,
+            0.004112027119845152,
+            -0.015080282464623451,
+            -0.0027014564257115126,
+            0.013575674034655094,
+            -0.02328723855316639,
+            -0.004607864189893007,
+            0.007480298634618521,
+            -0.006980187259614468,
+            0.010908412747085094,
+            -0.004069282673299313,
+            -0.004406964406371117,
+            0.008677146397531033,
+            0.01420829351991415,
+            0.002276147948578,
+            -0.026775194332003593,
+            0.0034665842540562153,
+            -0.011233271099627018,
+            0.0035093287006020546,
+            0.01020740158855915,
+            -0.008219779469072819,
+            1.5344630810432136e-05,
+            -0.015328200533986092,
+            -0.002739926567301154,
+            0.024860238656401634,
+            -0.004289417061954737,
+            0.0067707388661801815,
+            0.017302999272942543,
+            -0.007018657401204109,
+            0.002389421220868826,
+            -0.02862176112830639,
+            0.007796608377248049,
+            -0.0369655005633831,
+            -0.0004579011583700776,
+            0.004984016064554453,
+            0.004097066354006529,
+            -0.010104815475642681,
+            -0.030211858451366425,
+            0.008412130177021027,
+            -0.00474037230014801,
+            -0.006753641180694103,
+            -0.022723009809851646,
+            -0.007937666028738022,
+            0.008429228328168392,
+            -0.006941717118024826,
+            0.010147559456527233,
+            -0.017662053927779198,
+            0.017815934494137764,
+            0.015439336188137531,
+            0.013302108272910118,
+            0.003968833014369011,
+            0.0032379007898271084,
+            0.005129347555339336,
+            0.007946214638650417,
+            0.004701902158558369,
+            0.004697627853602171,
+            -0.01019030436873436,
+            -0.011464091949164867,
+            0.01575564593076706,
+            0.004255221225321293,
+            0.00860875565558672,
+            -0.0018604571232572198,
+            0.002479184651747346,
+            -0.0073050460778176785,
+            -0.00045629823580384254,
+            -0.014396369457244873,
+            0.0021596690639853477,
+            0.01199412439018488,
+            0.015499179251492023,
+            0.0014650699449703097,
+            -0.00625352980569005,
+            -0.007634179200977087,
+            -0.00667670089751482,
+            0.020209630951285362,
+            -0.017149118706583977,
+            -0.006698073353618383,
+            -0.012635293416678905,
+            0.021526163443922997,
+            0.01599501632153988,
+            0.0035563476849347353,
+            -0.006826307158917189,
+            -0.004092792049050331,
+            0.0097030159085989,
+            0.008570285513997078,
+            0.006766464561223984,
+            -0.023663390427827835,
+            -0.005377266090363264,
+            -0.019081173464655876,
+            -0.01539659220725298,
+            -0.01406296156346798,
+            0.021218402311205864,
+            -0.004924173932522535,
+            -0.007471750024706125,
+            0.008454875089228153,
+            0.024655064567923546,
+            0.007916293106973171,
+            0.007715393789112568,
+            0.00503958435729146,
+            -0.01726025529205799,
+            0.012071064673364162,
+            -0.010139010846614838,
+            -0.03665773943066597,
+            0.0065912120044231415,
+            -0.00971156544983387,
+            0.02598869428038597,
+            -0.017183315008878708,
+            -0.0011167017510160804,
+            -0.002635202370584011,
+            -0.030878672376275063,
+            -0.005638008005917072,
+            0.02961343340575695,
+            -0.010737434960901737,
+            0.011669266037642956,
+            -0.010062070563435555,
+            -0.009950934909284115,
+            0.01019030436873436,
+            0.013062738813459873,
+            -0.011720559559762478,
+            -0.002940826117992401,
+            0.014721227809786797,
+            0.02453538030385971,
+            0.0035093287006020546,
+            -0.004548021592199802,
+            0.009429451078176498,
+            0.0123702771961689,
+            -0.006804934702813625,
+            0.019559912383556366,
+            -0.007561513222754002,
+            -0.0015078145079314709,
+            -0.00656556524336338,
+            0.0012235631002113223,
+            0.006193687207996845,
+            -0.019577009603381157,
+            0.017320098355412483,
+            -0.009916738606989384,
+            0.008873771876096725,
+            0.0028702975250780582,
+            0.01726025529205799,
+            0.011267467401921749,
+            0.01896148920059204,
+            -0.02891242317855358,
+            -0.001095329411327839,
+            -0.0276813805103302,
+            -0.015772745013237,
+            -0.010703238658607006,
+            -0.00873698852956295,
+            -0.002485596342012286,
+            -0.007796608377248049,
+            -0.023868564516305923,
+            -0.005625184625387192,
+            0.0007939802599139512,
+            0.0036781697999686003,
+            -0.017362842336297035,
+            0.01007061917334795,
+            1.5286190318875015e-05,
+            -0.016593439504504204,
+            -0.020825151354074478,
+            -0.002087003318592906,
+            -0.022021999582648277,
+            -0.012464314699172974,
+            -0.005005388520658016,
+            -0.0022248546592891216,
+            -0.0012139456812292337,
+            -0.0005172092351131141,
+            0.027544597163796425,
+            -0.019457325339317322,
+            0.004381317645311356,
+            0.02744201011955738,
+            -0.0019352601375430822,
+            0.005035309586673975,
+            -0.006668151821941137,
+            0.012532706372439861,
+            0.0032913314644247293,
+            -0.003973107319325209,
+            -0.022688815370202065,
+            0.02521929331123829,
+            0.0005311011918820441,
+            0.013669711537659168,
+            0.010139010846614838,
+            -0.008065898902714252,
+            -0.005783339496701956,
+            0.00485150795429945,
+            -0.03253716230392456,
+            0.0028531996067613363,
+            -0.013011445291340351,
+            -0.01865372806787491,
+            -0.005539695732295513,
+            -0.004586491733789444,
+            -0.006561290472745895,
+            -0.0071939099580049515,
+            0.017285902053117752,
+            -0.004377043340355158,
+            0.02376597747206688,
+            -0.007146891206502914,
+            -0.00453947251662612,
+            0.010908412747085094,
+            -0.04301812872290611,
+            -0.007240929175168276,
+            -0.006129570305347443,
+            -0.01843145675957203,
+            0.019799282774329185,
+            0.012173651717603207,
+            -0.001380649395287037,
+            -0.020243825390934944,
+            0.019816379994153976,
+            -0.0030028056353330612,
+            0.02552705444395542,
+            0.010634847916662693,
+            0.010173206217586994,
+            0.004364219959825277,
+            -0.010840021073818207,
+            -0.010113364085555077,
+            0.018311770632863045,
+            0.006304823327809572,
+            -0.011609423905611038,
+            0.01873921789228916,
+            0.004445434547960758,
+            -0.00186686881352216,
+            -0.003041275776922703,
+            0.018756315112113953,
+            0.024603771045804024,
+            0.013002896681427956,
+            0.002274010796099901,
+            -0.016174543648958206,
+            -0.006475801579654217,
+            0.0037038165610283613,
+            0.036418367177248,
+            0.012036869302392006,
+            0.012994347140192986,
+            -0.008202682249248028,
+            0.027373619377613068,
+            -0.009181532077491283,
+            0.009044749662280083,
+            0.00863440241664648,
+            0.017174765467643738,
+            0.0035328383091837168,
+            0.020329315215349197,
+            0.010857119224965572,
+            -0.00764272827655077,
+            0.02405664138495922,
+            0.007625630125403404,
+            0.003767933463677764,
+            0.006625407375395298,
+            -0.01844855397939682,
+            0.012626743875443935,
+            -0.011609423905611038,
+            -0.005950043443590403,
+            0.004505277145653963,
+            0.002058150712400675,
+            0.02374888025224209,
+            0.008027428761124611,
+            -0.03135741129517555,
+            -0.0017867228016257286,
+            -0.023612096905708313,
+            0.004340710584074259,
+            -0.0167387705296278,
+            0.012447217479348183,
+            0.004092792049050331,
+            -0.01774754375219345,
+            -0.018790511414408684,
+            -0.010728885419666767,
+            -0.01942313089966774,
+            -0.004924173932522535,
+            -0.012173651717603207,
+            -0.010840021073818207,
+            0.006954540498554707,
+            -0.0025069687981158495,
+            0.0024813220370560884,
+            0.006963089574128389,
+            0.019046977162361145,
+            0.020260922610759735,
+            0.0035819944459944963,
+            0.018824705854058266,
+            -0.0009019103017635643,
+            -0.055431149899959564,
+            0.0036589347291737795,
+            0.009309765882790089,
+            -0.0002863885893020779,
+            0.026108380407094955,
+            -0.004041498526930809,
+            0.00042557556298561394,
+            0.007223831489682198,
+            0.006937442813068628,
+            0.013276461511850357,
+            -0.004210339393466711,
+            -0.025886109098792076,
+            -0.019337641075253487,
+            0.0040393611416220665,
+            -0.006155217066407204,
+            0.008100095205008984,
+            -0.006685249973088503,
+            0.004761744290590286,
+            0.002353088231757283,
+            -0.009745760820806026,
+            -0.006595486309379339,
+            0.009617527015507221,
+            0.0014469034504145384,
+            -0.010386928915977478,
+            -0.013173874467611313,
+            -0.0018690060824155807,
+            0.01824337989091873,
+            -0.009019102901220322,
+            -0.024364402517676353,
+            0.009660271927714348,
+            -0.010010777041316032,
+            -0.0014821677468717098,
+            0.025869010016322136,
+            0.02429600991308689,
+            -0.003970969934016466,
+            0.019371837377548218,
+            -0.0021190617699176073,
+            -0.016405364498496056,
+            -0.03029734641313553,
+            -0.006873325910419226,
+            -0.028997913002967834,
+            0.027253935113549232,
+            -0.006424508057534695,
+            -0.0013176011852920055,
+            0.0035285637713968754,
+            -0.00413553649559617,
+            0.003434525802731514,
+            -0.009788504801690578,
+            -0.007672649342566729,
+            -0.02716844528913498,
+            0.007527317851781845,
+            -0.0069075217470526695,
+            -0.008172760717570782,
+            0.00031550831045024097,
+            0.039701152592897415,
+            -0.004556570667773485,
+            0.00850189384073019,
+            0.006232157349586487,
+            0.011865891516208649,
+            0.024193422868847847,
+            -0.008463423699140549,
+            -0.012096711434423923,
+            -0.015046087093651295,
+            -0.0020154062658548355,
+            0.00014052275219000876,
+            -0.003975244704633951,
+            0.022876890376210213,
+            -0.013019993901252747,
+            -0.015191418118774891,
+            -0.00034489520476199687,
+            -0.023389825597405434,
+            -0.011737657710909843,
+            0.009968032129108906,
+            0.012908858247101307,
+            0.007535866927355528,
+            -0.014003119431436062,
+            -0.014268135651946068,
+            -0.022945282980799675,
+            0.01800401136279106,
+            0.0976627841591835,
+            0.0012545529752969742,
+            -0.009070396423339844,
+            0.0026758096646517515,
+            0.010771630331873894,
+            -0.0008372591692022979,
+            -0.017559466883540154,
+            0.05919267237186432,
+            -0.013567124493420124,
+            0.005856005474925041,
+            -0.009010554291307926,
+            0.01007916871458292,
+            0.005031035281717777,
+            -0.00013257493264973164,
+            0.005928670987486839,
+            -0.008796831592917442,
+            0.006129570305347443,
+            -0.03645256534218788,
+            -0.00186686881352216,
+            -0.011344407685101032,
+            -0.006381763610988855,
+            -0.016824260354042053,
+            0.009275570511817932,
+            -0.02911759726703167,
+            -0.00922427698969841,
+            0.0020880720112472773,
+            -0.0059543177485466,
+            0.012336080893874168,
+            -0.01628567837178707,
+            -0.0017995461821556091,
+            0.005368717480450869,
+            0.005120798945426941,
+            -0.0036867186427116394,
+            -0.0074632009491324425,
+            -0.013310656882822514,
+            0.005065230652689934,
+            0.015576119534671307,
+            0.005548244342207909,
+            -0.008228329010307789,
+            0.00959188025444746,
+            0.016388265416026115,
+            0.0009510665549896657,
+            0.013661162927746773,
+            0.009412352927029133,
+            0.0007950488943606615,
+            0.0007875685696490109,
+            0.019611205905675888,
+            -0.016345521435141563,
+            0.02137228287756443,
+            -0.015883879736065865,
+            -0.02843368425965309,
+            -0.006236432120203972,
+            0.006411684677004814,
+            -0.014413466677069664,
+            0.0010910549899563193,
+            0.006582662928849459,
+            0.04100058600306511,
+            0.00863440241664648,
+            -0.0034281141124665737,
+            -0.0009259541402570903,
+            -0.013669711537659168,
+            0.023954054340720177,
+            0.01091696135699749,
+            0.030502520501613617,
+            0.007604258134961128,
+            0.00373373762704432,
+            -0.007001559715718031,
+            -0.009557684883475304,
+            -0.003487956477329135,
+            0.00474037230014801,
+            0.0025069687981158495,
+            0.004415513481944799,
+            0.008523265831172466,
+            -0.014498956501483917,
+            -0.02036350965499878,
+            0.017610760405659676,
+            -0.00474037230014801,
+            -0.023663390427827835,
+            0.012609646655619144,
+            -0.0316309779882431,
+            0.0047831167466938496,
+            0.0010344184702262282,
+            -0.005877377465367317,
+            -0.013960374519228935,
+            0.006300548557192087,
+            0.01151538547128439,
+            -0.009378157556056976,
+            -0.015482081100344658,
+            -0.008454875089228153,
+            -0.010592103004455566,
+            0.0001227570464834571,
+            0.0037657960783690214,
+            -0.011583777144551277,
+            -0.00394318625330925,
+            -0.01796981506049633,
+            0.007142616901546717,
+            -0.003013491863384843,
+            0.004654882941395044,
+            -0.011122135445475578,
+            0.011558130383491516,
+            0.020226728171110153,
+            -0.009788504801690578,
+            -0.007069950923323631,
+            0.013105483725667,
+            0.003400330198928714,
+            0.016089053824543953,
+            0.03368271514773369,
+            -0.0033704089000821114,
+            0.0033511738292872906,
+            0.003849148051813245,
+            0.0058731031604111195,
+            0.0008874839986674488,
+            0.015909526497125626,
+            0.03621319308876991,
+            0.01942313089966774,
+            -0.0250141192227602,
+            0.02186811901628971,
+            -0.019713792949914932,
+            0.021936509758234024,
+            -0.02231266349554062,
+            -0.0030797459185123444,
+            0.007005834020674229,
+            -0.007394809741526842,
+            0.004146222956478596,
+            0.00624070642516017,
+            0.017627857625484467,
+            0.022363955155014992,
+            0.000617124664131552,
+            -0.0003163097717333585,
+            0.004860057029873133,
+            0.021953608840703964,
+            -0.0025411644019186497,
+            0.007125518750399351,
+            -0.013421793468296528,
+            0.004411239176988602,
+            -0.007202459033578634,
+            0.0021767669823020697,
+            0.008395032025873661,
+            -0.014952048659324646,
+            0.013344853185117245,
+            0.012891761027276516,
+            -0.017302999272942543,
+            0.0043342988938093185,
+            0.0025945950765162706,
+            -0.018568238243460655,
+            -0.018978586420416832,
+            0.023372728377580643,
+            -0.013635516166687012,
+            -0.012660940177738667,
+            -0.009891091845929623,
+            -0.01152393501251936,
+            -0.0201412383466959,
+            -0.005056682042777538,
+            -0.0017471840837970376,
+            0.005753418430685997,
+            -0.0024449890479445457,
+            0.006467252504080534,
+            0.01573854871094227,
+            0.005612361244857311,
+            -0.006437331438064575,
+            -0.005561067722737789,
+            -0.0015163634670898318,
+            0.020107043907046318,
+            0.005013937596231699,
+            0.0026458885986357927,
+            -0.04031667485833168,
+            0.0013143953401595354,
+            -0.010540809482336044,
+            0.006428782362490892,
+            0.0031417254358530045,
+            -0.004971192684024572,
+            -0.01942313089966774,
+            -0.014755423180758953,
+            0.01872211880981922,
+            -0.0103014400228858,
+            0.024637967348098755,
+            4.564986011246219e-05,
+            -0.012053967453539371,
+            -0.0018999958410859108,
+            0.003960283938795328,
+            0.0016029211692512035,
+            0.004415513481944799,
+            -0.00800178200006485,
+            0.016174543648958206,
+            -0.011694912798702717,
+            0.010908412747085094,
+            -0.00401157746091485,
+            -0.024706358090043068,
+            0.010891314595937729,
+            -0.01744833029806614,
+            0.0004752661334350705,
+            -0.00666387751698494,
+            -0.030040880665183067,
+            0.018551141023635864,
+            -0.023731783032417297,
+            -0.0033233899157494307,
+            0.009643173776566982,
+            0.018841803073883057,
+            0.0017461155075579882,
+            -0.010976804420351982,
+            0.00630909763276577,
+            0.017362842336297035,
+            0.013481635600328445,
+            -0.004062870983034372,
+            0.012438667938113213,
+            -0.004971192684024572,
+            0.0025603994727134705,
+            0.005172092467546463,
+            0.008399306796491146,
+            -0.006804934702813625,
+            0.007805157452821732
+          ],
+          "keyphrases": [
+            "DeepSeekMoE",
+            "multilingual corpus",
+            "HuggingFace Tokenizer",
+            "MoE architectures",
+            "evaluation benchmarks"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "5a008db3-740d-4d73-a00b-196efaee500e",
+      "type": "next",
+      "source": {
+        "id": "1ea6fc21-365a-41fe-9b07-1e5411ca0bc9",
+        "properties": {
+          "page_content": "4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n",
+          "title": "Validation Experiments",
+          "embedding": [
+            -0.010754532180726528,
+            -0.023697586730122566,
+            -0.02424471639096737,
+            -0.004748920910060406,
+            0.0029044931288808584,
+            0.014695581048727036,
+            -0.013233717530965805,
+            0.03884626179933548,
+            -0.0167387705296278,
+            0.032195206731557846,
+            -0.028296900913119316,
+            0.003022040706127882,
+            0.033374954015016556,
+            -0.032622650265693665,
+            -0.007941939868032932,
+            0.013165325857698917,
+            0.035289913415908813,
+            0.03330656513571739,
+            0.011327309533953667,
+            -0.037751998752355576,
+            0.04247099906206131,
+            -0.036520954221487045,
+            -0.060047563165426254,
+            -0.011079391464591026,
+            -0.019577009603381157,
+            0.02089354395866394,
+            0.00764272827655077,
+            0.016584891825914383,
+            -0.0636381059885025,
+            0.03330656513571739,
+            0.00491989916190505,
+            0.023150455206632614,
+            0.0002941360289696604,
+            0.008754086680710316,
+            -0.011498288251459599,
+            -0.03205842152237892,
+            0.022381054237484932,
+            -0.0009312971960753202,
+            -0.015362396836280823,
+            0.042607780545949936,
+            0.03381950035691261,
+            0.01822628267109394,
+            -0.023372728377580643,
+            -0.004340710584074259,
+            0.016832809895277023,
+            0.00703148078173399,
+            -0.013173874467611313,
+            -0.010062070563435555,
+            0.015704352408647537,
+            -0.04240260645747185,
+            -0.03546088933944702,
+            -0.011600874364376068,
+            0.008172760717570782,
+            -0.01200267393141985,
+            -0.03761521726846695,
+            0.007856450974941254,
+            0.016140347346663475,
+            0.026159673929214478,
+            0.04093219339847565,
+            0.015439336188137531,
+            0.027305228635668755,
+            -0.018602434545755386,
+            -0.019765086472034454,
+            0.010164657607674599,
+            0.0010231980122625828,
+            -0.03443502262234688,
+            0.0016926847165450454,
+            -0.011720559559762478,
+            0.017046531662344933,
+            -0.034383729100227356,
+            -0.04520665109157562,
+            0.02380017377436161,
+            -0.009292668662965298,
+            -0.010993901640176773,
+            -0.0009270227164961398,
+            -0.010198852978646755,
+            0.031203532591462135,
+            0.027544597163796425,
+            -0.0026907704304903746,
+            0.000909390626475215,
+            0.014806716702878475,
+            0.02405664138495922,
+            -0.023714683949947357,
+            0.0028040434699505568,
+            0.04373623803257942,
+            -0.0030434129294008017,
+            -0.02814302034676075,
+            -0.06527949869632721,
+            0.06415104120969772,
+            -0.0167815163731575,
+            -0.013644064776599407,
+            0.0002978761913254857,
+            0.023646293208003044,
+            -0.019577009603381157,
+            -0.029237281531095505,
+            0.05844036862254143,
+            -0.007394809741526842,
+            0.0007074225577525795,
+            -0.018619531765580177,
+            0.014601543545722961,
+            -0.009771407581865788,
+            0.010865667834877968,
+            -0.008617304265499115,
+            0.014422016218304634,
+            0.032622650265693665,
+            -0.024877335876226425,
+            -0.014447662979364395,
+            0.012831917963922024,
+            0.013712456449866295,
+            0.10628008842468262,
+            -0.016217287629842758,
+            -0.005920122377574444,
+            -0.03257135674357414,
+            0.0011006725253537297,
+            -0.009745760820806026,
+            -0.05006243288516998,
+            -0.026826487854123116,
+            -0.025373173877596855,
+            0.031237727031111717,
+            -0.008035978302359581,
+            0.033836595714092255,
+            0.032434575259685516,
+            0.045035671442747116,
+            -0.0132422661408782,
+            -0.04804489016532898,
+            0.046950630843639374,
+            -0.0051549943163990974,
+            0.000823367154225707,
+            -0.0025347527116537094,
+            -0.003041275776922703,
+            -0.0017194000538438559,
+            -0.007373437285423279,
+            -0.01822628267109394,
+            -0.01443056482821703,
+            0.011831695213913918,
+            0.0043449848890304565,
+            0.034298237413167953,
+            0.053003259003162384,
+            -0.04373623803257942,
+            -0.009685918688774109,
+            0.026792293414473534,
+            -0.029938293620944023,
+            0.010985353030264378,
+            0.006091100629419088,
+            -0.02403954230248928,
+            0.015011890791356564,
+            0.0010638053063303232,
+            0.055465348064899445,
+            0.0003606572572607547,
+            0.02374888025224209,
+            0.0011113586369901896,
+            0.04274456575512886,
+            -0.008360836654901505,
+            0.04363365098834038,
+            0.010592103004455566,
+            0.0010258695110678673,
+            -0.020927738398313522,
+            0.018602434545755386,
+            -0.039427585899829865,
+            -0.000900841667316854,
+            0.03171646595001221,
+            -0.0211500097066164,
+            -0.009899641387164593,
+            0.012660940177738667,
+            0.004398415796458721,
+            0.06869906187057495,
+            -0.01395182590931654,
+            -0.0074888477101922035,
+            -0.0781712606549263,
+            0.04982306435704231,
+            0.023971151560544968,
+            0.00814711395651102,
+            0.005907298997044563,
+            0.04182128235697746,
+            0.003832050133496523,
+            0.01309693418443203,
+            0.012438667938113213,
+            -0.0098397983238101,
+            -0.05026760697364807,
+            0.03207552060484886,
+            -0.0026587119791656733,
+            0.012908858247101307,
+            -0.009130238555371761,
+            0.026860684156417847,
+            0.05505499988794327,
+            0.022021999582648277,
+            0.01165216788649559,
+            0.027476206421852112,
+            -0.020671270787715912,
+            0.01749962382018566,
+            0.037033889442682266,
+            -0.0011412798194214702,
+            0.03744423761963844,
+            0.026980368420481682,
+            -0.005979964509606361,
+            -0.013105483725667,
+            -0.028348194435238838,
+            0.013797945342957973,
+            0.006086825858801603,
+            0.03405886888504028,
+            -0.02034641243517399,
+            -0.03525571525096893,
+            0.007664100266993046,
+            -0.0034281141124665737,
+            0.04079541191458702,
+            -0.00934396218508482,
+            0.007822255603969097,
+            0.023099161684513092,
+            0.010942608118057251,
+            -0.004847233649343252,
+            -0.024210521951317787,
+            -0.008540363982319832,
+            -0.01188298873603344,
+            -0.0402824766933918,
+            -0.011891537345945835,
+            0.03157968446612358,
+            -0.015353847295045853,
+            0.01938893459737301,
+            -0.005915847606956959,
+            -0.007800883147865534,
+            -0.023543706163764,
+            0.018072402104735374,
+            0.0032464496325701475,
+            -0.008027428761124611,
+            -0.0069075217470526695,
+            0.003357585519552231,
+            -0.0019352601375430822,
+            0.003267821855843067,
+            0.048181671649217606,
+            -0.02402244508266449,
+            0.012113809585571289,
+            0.008527540601789951,
+            0.0011316622840240598,
+            0.02380017377436161,
+            0.03346044570207596,
+            -0.035563476383686066,
+            -0.012934505008161068,
+            -0.0014618640998378396,
+            0.009514939971268177,
+            0.020876444876194,
+            -0.009873994626104832,
+            -0.0097030159085989,
+            -0.03378530219197273,
+            -0.0005765173118561506,
+            0.00013043770741205662,
+            -0.015712901949882507,
+            -0.008467698469758034,
+            0.017166217789053917,
+            0.042641978710889816,
+            0.032434575259685516,
+            0.02034641243517399,
+            -0.035289913415908813,
+            -0.00907894503325224,
+            0.027082955464720726,
+            -0.014610092155635357,
+            0.034315336495637894,
+            0.0504043884575367,
+            -0.029459552839398384,
+            -0.018773412331938744,
+            -0.03980373963713646,
+            0.07112695276737213,
+            -0.023731783032417297,
+            -0.02964762970805168,
+            0.042368412017822266,
+            -0.01408860832452774,
+            0.0021778354421257973,
+            0.03187034651637077,
+            0.035324107855558395,
+            -0.0035307009238749743,
+            0.012053967453539371,
+            0.03499924764037132,
+            -0.0194060318171978,
+            -0.04691643267869949,
+            -0.02523639053106308,
+            -0.025116706266999245,
+            -0.016619086265563965,
+            -0.010275793261826038,
+            -0.017311548814177513,
+            0.010087717324495316,
+            0.020278021693229675,
+            0.013635516166687012,
+            -0.010882765986025333,
+            0.007022931706160307,
+            0.009985130280256271,
+            -0.009976581670343876,
+            -0.000917939527425915,
+            -0.027561694383621216,
+            -0.0071939099580049515,
+            -0.028467880561947823,
+            -0.012053967453539371,
+            -0.014977695420384407,
+            -0.006685249973088503,
+            -0.025783522054553032,
+            0.01889309659600258,
+            0.023167554289102554,
+            -0.002190658822655678,
+            -0.008459148928523064,
+            0.01238737441599369,
+            0.03472568467259407,
+            -0.028108825907111168,
+            -0.03812815248966217,
+            0.008916515856981277,
+            -0.0034558980260044336,
+            0.009762858040630817,
+            -0.015037537552416325,
+            0.010677591897547245,
+            -0.011395701207220554,
+            0.010540809482336044,
+            -0.010703238658607006,
+            -0.010002228431403637,
+            -0.0006646779947914183,
+            0.021953608840703964,
+            0.00801460538059473,
+            -0.014592994004487991,
+            -0.017593663185834885,
+            0.027715574949979782,
+            0.014293782413005829,
+            0.0334775410592556,
+            -0.029545042663812637,
+            -0.023099161684513092,
+            0.042573586106300354,
+            0.004015851765871048,
+            -0.043086521327495575,
+            0.016148896887898445,
+            0.01671312376856804,
+            0.011797499842941761,
+            0.01942313089966774,
+            0.048181671649217606,
+            -0.033870793879032135,
+            -0.038196541368961334,
+            0.005420010536909103,
+            -0.031938739120960236,
+            -0.030929965898394585,
+            0.014883656986057758,
+            -0.006227883044630289,
+            -0.05125928297638893,
+            -0.031203532591462135,
+            -0.014020216651260853,
+            -0.019337641075253487,
+            0.0022098938934504986,
+            -0.015046087093651295,
+            0.004941271618008614,
+            -0.03741004317998886,
+            0.0025689483154565096,
+            0.0004648471367545426,
+            -0.018328869715332985,
+            -0.028724348172545433,
+            0.0027548871003091335,
+            0.004488179460167885,
+            0.03542669489979744,
+            0.0017097825184464455,
+            0.01128456462174654,
+            -0.012951603159308434,
+            -0.000579188868869096,
+            0.003056236309930682,
+            -0.05016501992940903,
+            -0.019474424421787262,
+            0.038983043283224106,
+            0.009027652442455292,
+            0.006048355717211962,
+            0.011874440126121044,
+            -0.028211412951350212,
+            -0.0592268668115139,
+            0.029716020449995995,
+            -0.04643769562244415,
+            0.015268358401954174,
+            0.004415513481944799,
+            -0.0010344184702262282,
+            -0.001160514890216291,
+            -0.0007758138235658407,
+            -0.0067707388661801815,
+            0.023167554289102554,
+            -0.007407633122056723,
+            0.012498511001467705,
+            -0.0028788463678210974,
+            0.03669193387031555,
+            0.0358712375164032,
+            -0.00673226872459054,
+            -0.038948848843574524,
+            0.02956213988363743,
+            -0.008668597787618637,
+            0.025612542405724525,
+            0.01889309659600258,
+            0.02961343340575695,
+            -0.017918521538376808,
+            0.032417476177215576,
+            0.002891669748350978,
+            0.024125032126903534,
+            -0.03877786919474602,
+            0.023167554289102554,
+            0.0052191112190485,
+            -0.054268497973680496,
+            -0.00860020611435175,
+            -0.0073777115903794765,
+            0.022244270890951157,
+            0.008822478353977203,
+            -0.02477474883198738,
+            -0.021970706060528755,
+            0.0009409147314727306,
+            -0.02817721664905548,
+            -0.007574336603283882,
+            0.04568539187312126,
+            -0.012558353133499622,
+            0.0014875108608976007,
+            -0.017414135858416557,
+            0.014174097217619419,
+            0.02886112965643406,
+            0.027510400861501694,
+            0.03354593366384506,
+            -0.013986021280288696,
+            0.020021554082632065,
+            -0.022278467193245888,
+            -0.037033889442682266,
+            -0.02595449984073639,
+            0.0073050460778176785,
+            0.03939339146018028,
+            0.009728662669658661,
+            0.006039807107299566,
+            -0.0013913356233388186,
+            0.00790346972644329,
+            -0.015370945446193218,
+            0.027989141643047333,
+            0.044796302914619446,
+            -0.002494145417585969,
+            -0.011464091949164867,
+            0.007505945395678282,
+            0.015046087093651295,
+            -0.018619531765580177,
+            0.027373619377613068,
+            -0.025424467399716377,
+            0.02255203202366829,
+            0.004744646605104208,
+            0.02065417356789112,
+            -0.047702934592962265,
+            -0.004556570667773485,
+            -0.0023082063999027014,
+            -0.03445211797952652,
+            0.05946623906493187,
+            -0.028690151870250702,
+            -0.013524380512535572,
+            0.0029386887326836586,
+            -0.0011316622840240598,
+            -0.01313113048672676,
+            -0.026980368420481682,
+            -0.014849461615085602,
+            -0.0061894129030406475,
+            -0.04554860666394234,
+            -0.029801510274410248,
+            -0.014815266244113445,
+            -0.01213945634663105,
+            -0.06240706518292427,
+            0.004058596212416887,
+            -0.0741361752152443,
+            0.007206733338534832,
+            0.027715574949979782,
+            -0.016661832109093666,
+            0.03963275998830795,
+            -0.022637521848082542,
+            0.02374888025224209,
+            -0.04076121747493744,
+            -0.0157299991697073,
+            0.03258845582604408,
+            0.047463566064834595,
+            0.010848570615053177,
+            0.011498288251459599,
+            0.021047422662377357,
+            -0.03199003264307976,
+            -0.0026758096646517515,
+            -0.00860875565558672,
+            -0.0018219870980829,
+            0.03332366421818733,
+            -0.008553187362849712,
+            0.021765531972050667,
+            0.0051549943163990974,
+            0.024142129346728325,
+            0.031442902982234955,
+            -0.002782671246677637,
+            -0.023868564516305923,
+            0.012857564724981785,
+            0.025544151663780212,
+            0.013054190203547478,
+            -0.014251037500798702,
+            0.012686586938798428,
+            -0.01579839177429676,
+            0.007719668094068766,
+            0.008484795689582825,
+            0.009908189997076988,
+            0.034315336495637894,
+            0.024142129346728325,
+            -0.013105483725667,
+            0.0029814334120601416,
+            0.009865445084869862,
+            -0.03645256534218788,
+            0.042641978710889816,
+            0.0051849158480763435,
+            -0.03953017294406891,
+            -0.0005487333401106298,
+            -0.007617081515491009,
+            0.020551586523652077,
+            0.02156035788357258,
+            0.00936105940490961,
+            -0.022671716287732124,
+            -0.00020944836433045566,
+            0.016841357573866844,
+            0.008660049177706242,
+            -0.012182201258838177,
+            0.02450118400156498,
+            0.03833332657814026,
+            -0.020260922610759735,
+            -0.030639303848147392,
+            0.00922427698969841,
+            -0.010489515960216522,
+            0.0007736766128800809,
+            -0.02715134806931019,
+            -0.004394141025841236,
+            0.018072402104735374,
+            -0.044762108474969864,
+            0.05177221819758415,
+            -0.013575674034655094,
+            0.008685695938766003,
+            -0.012943054549396038,
+            -0.002703593811020255,
+            -0.02138938009738922,
+            0.034588903188705444,
+            0.014131353236734867,
+            0.02867305465042591,
+            0.016106151044368744,
+            0.010318538174033165,
+            0.028040433302521706,
+            -0.020278021693229675,
+            -0.023099161684513092,
+            0.014422016218304634,
+            -0.007980410009622574,
+            -0.00467198109254241,
+            -0.02959633618593216,
+            -0.024107934907078743,
+            -0.01892729289829731,
+            0.02527058683335781,
+            -0.01530255377292633,
+            -0.008566010743379593,
+            0.010951157659292221,
+            0.0014981969725340605,
+            -0.002131885150447488,
+            0.013943277299404144,
+            -0.007638453505933285,
+            0.008707067929208279,
+            0.04192386940121651,
+            -0.013934727758169174,
+            0.02159455418586731,
+            0.03611060976982117,
+            0.011472641490399837,
+            -0.018516944721341133,
+            -0.0064800758846104145,
+            -0.0007293291273526847,
+            -0.01066904328763485,
+            -0.03570026159286499,
+            0.035289913415908813,
+            0.020996129140257835,
+            -0.0021842471323907375,
+            -0.037239063531160355,
+            -0.012583999894559383,
+            0.014635738916695118,
+            -0.014037314802408218,
+            0.027732674032449722,
+            -0.015943722799420357,
+            -0.015653058886528015,
+            0.00474037230014801,
+            0.021970706060528755,
+            -0.049857258796691895,
+            -0.012977249920368195,
+            0.010899864137172699,
+            0.002707868115976453,
+            0.008057350292801857,
+            -0.006779287941753864,
+            0.013455988839268684,
+            0.020568683743476868,
+            0.004945545922964811,
+            0.0036439741961658,
+            -0.0038085407577455044,
+            0.01647375524044037,
+            0.013455988839268684,
+            -0.017593663185834885,
+            -0.009241375140845776,
+            0.02210748940706253,
+            0.0230820644646883,
+            -0.030211858451366425,
+            -0.023919858038425446,
+            -0.00583035871386528,
+            0.021902315318584442,
+            -0.004753195680677891,
+            0.03689710795879364,
+            0.0388120636343956,
+            0.011438445188105106,
+            -0.04318910837173462,
+            -0.014516053721308708,
+            0.0017290175892412663,
+            -0.010660494677722454,
+            0.016431011259555817,
+            -0.0012438668636605144,
+            -0.042812954634428024,
+            0.002143639838322997,
+            -0.0036055040545761585,
+            0.028450781479477882,
+            -0.040214087814092636,
+            -0.011959929019212723,
+            0.009617527015507221,
+            -0.012951603159308434,
+            0.003302017692476511,
+            0.012182201258838177,
+            0.008484795689582825,
+            0.021269695833325386,
+            0.012703684158623219,
+            -0.005526872351765633,
+            -0.0014607955235987902,
+            0.016550695523619652,
+            0.004535198211669922,
+            0.026604216545820236,
+            -0.018328869715332985,
+            -0.017345745116472244,
+            0.01844855397939682,
+            0.03860688954591751,
+            -0.012327532283961773,
+            0.04076121747493744,
+            0.007514494471251965,
+            0.012062516063451767,
+            0.012088162824511528,
+            -0.02453538030385971,
+            -0.01598646678030491,
+            -0.011951380409300327,
+            0.006257804110646248,
+            -0.037512630224227905,
+            0.014858010224997997,
+            -0.02380017377436161,
+            -0.027134248986840248,
+            0.02619386836886406,
+            -0.006014160346239805,
+            0.010686141438782215,
+            0.022244270890951157,
+            -0.012891761027276516,
+            -0.027271032333374023,
+            0.03369981423020363,
+            0.034349530935287476,
+            0.008433502167463303,
+            0.04677965119481087,
+            -0.010395478457212448,
+            0.006779287941753864,
+            0.005702124908566475,
+            0.0024920080322772264,
+            0.044762108474969864,
+            -0.008236877620220184,
+            -0.01200267393141985,
+            0.007535866927355528,
+            0.009779956191778183,
+            0.04206065088510513,
+            -0.015413690358400345,
+            -0.049412716180086136,
+            0.019508618861436844,
+            -0.014507505111396313,
+            0.009856896474957466,
+            -0.006655328441411257,
+            0.008198407478630543,
+            -0.005890200845897198,
+            0.010053521953523159,
+            -0.004522374831140041,
+            0.010113364085555077,
+            -0.005843182094395161,
+            -0.014327977783977985,
+            0.015174319967627525,
+            -0.01018175482749939,
+            0.030280249193310738,
+            0.009557684883475304,
+            -0.02503121644258499,
+            -0.026501629501581192,
+            0.01746542938053608,
+            0.0019277798710390925,
+            -0.028724348172545433,
+            0.009882543236017227,
+            0.0023872838355600834,
+            0.010104815475642681,
+            0.004381317645311356,
+            -0.012242043390870094,
+            0.015926625579595566,
+            0.01676441729068756,
+            -0.022210076451301575,
+            0.03911982476711273,
+            -0.014285233803093433,
+            -0.014644287526607513,
+            0.019235054031014442,
+            -0.03730745613574982,
+            0.008424953557550907,
+            -0.012524157762527466,
+            0.010831472463905811,
+            0.00491989916190505,
+            0.004667706321924925,
+            -0.011352956295013428,
+            -0.0026715353596955538,
+            -0.02352660894393921,
+            0.03281072899699211,
+            0.0272197388112545,
+            -0.0012834055814892054,
+            0.005958592053502798,
+            -0.007535866927355528,
+            -0.012900309637188911,
+            0.0024984197225421667,
+            0.026142574846744537,
+            -0.024381499737501144,
+            -0.0017899286467581987,
+            -0.0006310166209004819,
+            0.007694021798670292,
+            0.00551832327619195,
+            0.022432347759604454,
+            0.04900236800312996,
+            -0.0038192267529666424,
+            0.028980813920497894,
+            0.035050541162490845,
+            0.024142129346728325,
+            -0.003259273013100028,
+            0.004761744290590286,
+            -0.025150902569293976,
+            0.03645256534218788,
+            0.008950712159276009,
+            0.016670379787683487,
+            0.017798837274312973,
+            -0.04312071576714516,
+            -0.009634625166654587,
+            0.019525716081261635,
+            0.02205619588494301,
+            0.020089944824576378,
+            -0.003881206503137946,
+            -0.0035884061362594366,
+            0.03298170492053032,
+            -0.025920303538441658,
+            -0.019354738295078278,
+            0.004154771566390991,
+            -0.0053174239583313465,
+            -0.005732045974582434,
+            -0.0069032469764351845,
+            0.01865372806787491,
+            -0.014584445394575596,
+            -0.004458257928490639,
+            0.016867004334926605,
+            0.001722605898976326,
+            0.012113809585571289,
+            0.016165994107723236,
+            -0.00024137321452144533,
+            0.022415248677134514,
+            -0.004932722542434931,
+            0.021509064361453056,
+            0.005330247338861227,
+            -0.003663209266960621,
+            -0.006911796052008867,
+            -0.010275793261826038,
+            0.029015010222792625,
+            -0.006980187259614468,
+            -0.034640196710824966,
+            -0.01360986940562725,
+            -0.009284119121730328,
+            0.004156908951699734,
+            0.0011391426669433713,
+            0.0050524077378213406,
+            0.01744833029806614,
+            0.026501629501581192,
+            -0.00046751866466365755,
+            0.004280867986381054,
+            0.014336527325212955,
+            -0.0024193422868847847,
+            -0.010506614111363888,
+            0.019559912383556366,
+            0.008779733441770077,
+            -0.01624293439090252,
+            0.018756315112113953,
+            0.0030861576087772846,
+            0.04688223823904991,
+            -0.044522739946842194,
+            0.018209183588624,
+            0.007805157452821732,
+            0.0006956678116694093,
+            -0.014960597269237041,
+            -0.03327237069606781,
+            0.0036097783595323563,
+            -0.014122803695499897,
+            -0.0033597226720303297,
+            -0.024689260870218277,
+            0.00786499958485365,
+            -0.018089499324560165,
+            0.014045863412320614,
+            0.011600874364376068,
+            -0.0016349796205759048,
+            -0.002028229646384716,
+            0.017542369663715363,
+            -0.001940603251568973,
+            0.00787782296538353,
+            -0.017362842336297035,
+            0.010942608118057251,
+            0.022381054237484932,
+            -0.010677591897547245,
+            0.008638676255941391,
+            0.0073306928388774395,
+            -0.02793784812092781,
+            -0.028536271303892136,
+            0.013336303643882275,
+            -0.03546088933944702,
+            0.007681198418140411,
+            0.031203532591462135,
+            -0.032434575259685516,
+            -0.028519174084067345,
+            0.0019085448002442718,
+            -0.021406477317214012,
+            -0.017867228016257286,
+            0.011446994729340076,
+            0.018021108582615852,
+            -0.004216751083731651,
+            -0.030451226979494095,
+            0.013173874467611313,
+            -0.007809431757777929,
+            0.00698873633518815,
+            0.00015388043539132923,
+            -0.02569803223013878,
+            -0.02569803223013878,
+            0.004984016064554453,
+            -0.028211412951350212,
+            -0.007138342130929232,
+            -0.0005297654424794018,
+            -0.024466989561915398,
+            -0.043804630637168884,
+            -0.015242711640894413,
+            -0.013507282361388206,
+            -0.010156108066439629,
+            0.0480106957256794,
+            0.0196795966476202,
+            0.026894880458712578,
+            -0.0003211185394320637,
+            -0.018106596544384956,
+            0.04773712903261185,
+            0.0009954140987247229,
+            0.041752889752388,
+            0.013387597166001797,
+            -0.008335189893841743,
+            -0.006693798583000898,
+            0.03392208740115166,
+            -0.015584668144583702,
+            -0.007775236386805773,
+            -0.010651945136487484,
+            -0.0015836860984563828,
+            0.0025091059505939484,
+            0.022449444979429245,
+            0.0019085448002442718,
+            0.004037224221974611,
+            0.027100054547190666,
+            -0.023201748728752136,
+            0.002739926567301154,
+            0.01238737441599369,
+            -0.006937442813068628,
+            0.008215505629777908,
+            -0.02838239073753357,
+            -0.008300994522869587,
+            0.019081173464655876,
+            0.006044081412255764,
+            -0.007437554188072681,
+            -0.0022440897300839424,
+            -0.027134248986840248,
+            -0.0014052275801077485,
+            0.0038106779102236032,
+            -0.005826083943247795,
+            0.005744869355112314,
+            0.04459112882614136,
+            -0.004817312583327293,
+            -0.039940521121025085,
+            -0.00029493749025277793,
+            -0.03634997829794884,
+            0.0001610935723874718,
+            0.00010799681331263855,
+            -0.0033191153779625893,
+            0.005488402210175991,
+            0.0018422907451167703,
+            -0.012472864240407944,
+            0.012130907736718655,
+            -0.028741445392370224,
+            0.019782183691859245,
+            0.004817312583327293,
+            -0.007574336603283882,
+            -0.005056682042777538,
+            0.01988477073609829,
+            0.008151388727128506,
+            -0.013148227706551552,
+            -0.0014960597036406398,
+            0.0028852580580860376,
+            -0.026125477626919746,
+            -0.020773857831954956,
+            -0.03369981423020363,
+            -0.0022505014203488827,
+            -0.009583331644535065,
+            -0.03010927140712738,
+            -0.006937442813068628,
+            -0.0003617258626036346,
+            0.007775236386805773,
+            -0.020295118913054466,
+            0.0006956678116694093,
+            0.005552519112825394,
+            0.012199298478662968,
+            0.00624070642516017,
+            -0.007339241914451122,
+            -0.004263770300894976,
+            -0.03033154271543026,
+            0.003584131831303239,
+            -0.0015345298452302814,
+            0.0038619714323431253,
+            -0.016063407063484192,
+            0.006351842079311609,
+            -0.016183091327548027,
+            -0.0035264266189187765,
+            0.011335858143866062,
+            0.01800401136279106,
+            -0.028741445392370224,
+            0.036726128309965134,
+            0.02889532595872879,
+            0.017320098355412483,
+            -0.038914650678634644,
+            0.028587564826011658,
+            -0.0012331806356087327,
+            0.001522775157354772,
+            0.0030177661683410406,
+            0.021731335669755936,
+            0.026039987802505493,
+            -0.016431011259555817,
+            -0.006531369406729937,
+            0.010224499739706516,
+            0.028194313868880272,
+            -0.031545490026474,
+            -0.007266575936228037,
+            1.9886239897459745e-05,
+            0.02378307469189167,
+            0.027783967554569244,
+            0.023116260766983032,
+            0.04271036759018898,
+            -0.0053131491877138615,
+            0.002891669748350978,
+            -0.005082328803837299,
+            -0.013926179148256779,
+            0.0024727729614824057,
+            -0.0028767092153429985,
+            -0.0017952717607840896,
+            0.011489738710224628,
+            0.012695135548710823,
+            0.017593663185834885,
+            -0.07126373797655106,
+            0.038743674755096436,
+            0.026570022106170654,
+            0.012105260975658894,
+            0.008399306796491146,
+            -0.024706358090043068,
+            -0.0017909972229972482,
+            -0.0022205801215022802,
+            0.0018604571232572198,
+            0.013438890688121319,
+            -0.010874217376112938,
+            -0.02475765161216259,
+            0.016567792743444443,
+            -0.019457325339317322,
+            0.017149118706583977,
+            0.0050267609767615795,
+            0.00503958435729146,
+            -0.024603771045804024,
+            -0.014721227809786797,
+            0.006830581463873386,
+            0.008065898902714252,
+            0.008100095205008984,
+            -0.03857269510626793,
+            -0.044727910310029984,
+            0.01750817336142063,
+            0.008035978302359581,
+            -0.04862621799111366,
+            -0.006586937233805656,
+            -0.017388489097356796,
+            -0.004210339393466711,
+            -0.006326195318251848,
+            -0.01767915114760399,
+            -0.040897998958826065,
+            0.002064562402665615,
+            0.017302999272942543,
+            0.018534043803811073,
+            5.349749335437082e-05,
+            0.028826933354139328,
+            0.0017183314776048064,
+            -0.0022355406545102596,
+            -0.030434129759669304,
+            0.026860684156417847,
+            -0.007326418533921242,
+            -0.022278467193245888,
+            0.024142129346728325,
+            0.01628567837178707,
+            -0.019970260560512543,
+            -0.0038769319653511047,
+            0.0014928538585081697,
+            -0.008446325547993183,
+            0.034554705023765564,
+            -0.0017129883635789156,
+            -0.008933614008128643,
+            0.0026907704304903746,
+            -0.0020998266991227865,
+            -0.00959188025444746,
+            -0.02479184791445732,
+            0.00145438383333385,
+            0.025869010016322136,
+            -0.007749589625746012,
+            0.006437331438064575,
+            0.04202645644545555,
+            -0.012353179045021534,
+            -0.019850576296448708,
+            0.00043145293602719903,
+            -0.02839948795735836,
+            0.010318538174033165,
+            -0.006685249973088503,
+            0.030280249193310738,
+            0.004206065088510513,
+            0.013002896681427956,
+            -0.016644733026623726,
+            -0.01066904328763485,
+            -0.021970706060528755,
+            -0.0641852393746376,
+            -0.012122358195483685,
+            -0.005561067722737789,
+            -0.024894434958696365,
+            -0.02398824878036976,
+            0.03234908729791641,
+            0.022398151457309723,
+            0.010609201155602932,
+            -0.002107307082042098,
+            -0.01891019567847252,
+            -0.004458257928490639,
+            0.03009217418730259,
+            -0.005458480678498745,
+            0.017371391877532005,
+            -0.011617972515523434,
+            0.01627713069319725,
+            -0.006578388623893261,
+            0.0051336223259568214,
+            -0.053960736840963364,
+            -0.005167817696928978,
+            0.02983570657670498,
+            0.01993606425821781,
+            0.007240929175168276,
+            0.010019325651228428,
+            0.013293559662997723,
+            -0.007501671090722084,
+            -0.02764718420803547,
+            0.02525348775088787,
+            0.017132021486759186,
+            0.01991896703839302,
+            -0.015935173258185387,
+            0.007587159983813763,
+            0.011788951233029366,
+            0.00288098375312984,
+            0.000775279535446316,
+            0.0008281759219244123,
+            0.025133803486824036,
+            0.003549935994669795,
+            -0.009275570511817932,
+            0.015721451491117477,
+            -0.007163988891988993,
+            -0.021269695833325386,
+            -0.004233849234879017,
+            0.008959260769188404,
+            0.004035086836665869,
+            0.001731154858134687,
+            -0.0015922350576147437,
+            -0.008984907530248165,
+            0.02886112965643406,
+            0.03539250046014786,
+            0.03183615207672119,
+            -0.0057876138016581535,
+            -0.005137896630913019,
+            0.0016456657322123647,
+            0.003513603238388896,
+            -0.007659825962036848,
+            -0.0018294673645868897,
+            0.019799282774329185,
+            -0.030006684362888336,
+            -0.011276016011834145,
+            -0.009318315424025059,
+            0.016362618654966354,
+            0.003522152081131935,
+            0.04411239176988602,
+            0.02111581526696682,
+            0.006522820331156254,
+            -0.00557389110326767,
+            0.0037209142465144396,
+            0.004389866720885038,
+            0.029237281531095505,
+            -0.02282559685409069,
+            9.023110033012927e-05,
+            -0.0034217024222016335,
+            -0.017901424318552017,
+            0.019046977162361145,
+            -0.007300771772861481,
+            0.009540586732327938,
+            0.02205619588494301,
+            0.09212308377027512,
+            -0.0065527418628335,
+            0.022449444979429245,
+            -0.027852358296513557,
+            -0.024090835824608803,
+            0.00291517935693264,
+            0.006313371937721968,
+            0.002265461953356862,
+            0.022039096802473068,
+            0.0054499320685863495,
+            0.02525348775088787,
+            0.0272197388112545,
+            0.0009067190694622695,
+            0.006698073353618383,
+            -0.010292891412973404,
+            0.004118438810110092,
+            -0.013182424008846283,
+            -0.009549135342240334,
+            -0.009437999688088894,
+            0.008779733441770077,
+            0.019012782722711563,
+            0.00911314133554697,
+            0.007531592156738043,
+            -0.004084242973476648,
+            -0.015482081100344658,
+            0.026518728584051132,
+            0.010292891412973404,
+            -0.004475356079638004,
+            -0.015601766295731068,
+            -0.015114477835595608,
+            0.012404472567141056,
+            -0.006886149290949106,
+            -0.005894475616514683,
+            0.01721751131117344,
+            0.005069505423307419,
+            0.02865595556795597,
+            -0.03369981423020363,
+            0.008890869095921516,
+            0.014259587042033672,
+            -0.0010857119923457503,
+            0.007245203480124474,
+            0.021423576399683952,
+            -0.0022633245680481195,
+            0.0242789126932621,
+            -0.01468703243881464,
+            0.00874553807079792,
+            0.010959706269204617,
+            -0.021201303228735924,
+            0.0036055040545761585,
+            0.01990186981856823,
+            -0.003381095128133893,
+            0.015413690358400345,
+            -0.0052191112190485,
+            -0.001976936124265194,
+            0.0025903205387294292,
+            0.006582662928849459,
+            0.00363756250590086,
+            -0.01988477073609829,
+            0.01188298873603344,
+            -0.01872211880981922,
+            0.009172983467578888,
+            -0.004565119277685881,
+            0.012190749868750572,
+            -0.008424953557550907,
+            0.011370054446160793,
+            -0.006121021695435047,
+            0.006847679149359465,
+            -0.0105750048533082,
+            -0.016559245064854622,
+            -0.012515608221292496,
+            0.012960151769220829,
+            0.007249478250741959,
+            -0.013558575883507729,
+            0.008309543132781982,
+            -0.008228329010307789,
+            -0.002067768247798085,
+            0.014960597269237041,
+            -0.011746206320822239,
+            0.00407355697825551,
+            -0.039461780339479446,
+            -0.004535198211669922,
+            0.018089499324560165,
+            -0.001959838205948472,
+            0.0027719850186258554,
+            -0.007352065294981003,
+            0.011199075728654861,
+            -0.008288171142339706,
+            -0.0024086562916636467,
+            0.0254757609218359,
+            -0.019782183691859245,
+            -0.017328646034002304,
+            -0.032913316041231155,
+            -0.05009663105010986,
+            -0.002131885150447488,
+            -0.009326864033937454,
+            0.005864554084837437,
+            -0.00790346972644329,
+            0.017815934494137764,
+            -0.0040179891511797905,
+            -0.007095597684383392,
+            -0.024125032126903534,
+            0.010395478457212448,
+            0.028211412951350212,
+            0.0008682489860802889,
+            -0.03079318441450596,
+            -0.012344630435109138,
+            0.010463869199156761,
+            0.0006646779947914183,
+            -0.029049206525087357,
+            0.0083693852648139,
+            0.0006181932403706014,
+            -0.014473309740424156,
+            -0.008882320486009121,
+            -0.005971415434032679,
+            0.021440673619508743,
+            0.024193422868847847,
+            0.005753418430685997,
+            -0.019782183691859245,
+            -0.004225300159305334,
+            0.03258845582604408,
+            -0.018294673413038254,
+            0.003994479309767485,
+            0.023868564516305923,
+            -0.00264375121332705,
+            0.00982270110398531,
+            0.01054935809224844,
+            0.018072402104735374,
+            -0.008352288044989109,
+            0.012660940177738667,
+            0.007561513222754002,
+            -0.015328200533986092,
+            0.008548912592232227,
+            0.006668151821941137,
+            0.016405364498496056,
+            0.006890423595905304,
+            -0.022244270890951157,
+            -0.004411239176988602,
+            0.015832586213946342,
+            0.053276825696229935,
+            -0.006706621963530779,
+            -0.008506168611347675,
+            0.03617899864912033,
+            0.024415696039795876,
+            0.007416181731969118,
+            0.005937220063060522,
+            -0.03231488913297653,
+            -0.004646334331482649,
+            0.0008709204848855734,
+            0.005236208904534578,
+            0.0132422661408782,
+            0.0035029170103371143,
+            -0.007770961616188288,
+            0.018978586420416832,
+            -0.001043501659296453,
+            0.014293782413005829,
+            -0.020107043907046318,
+            0.009514939971268177,
+            -0.00618513859808445,
+            0.03161387890577316,
+            0.01042112521827221,
+            -0.0010632710764184594,
+            0.03318687900900841,
+            0.006719445344060659,
+            -0.03344334661960602,
+            0.006125296000391245,
+            -0.007638453505933285,
+            0.011352956295013428,
+            -0.02260332554578781,
+            -0.007894921116530895,
+            0.009899641387164593,
+            0.0012289062142372131,
+            -0.018123695626854897,
+            0.011900086887180805,
+            0.007142616901546717,
+            0.009378157556056976,
+            -0.0005217508296482265,
+            0.006356116849929094,
+            0.008553187362849712,
+            -0.004522374831140041,
+            -0.008168485946953297,
+            0.005599537864327431,
+            -0.016336971893906593,
+            -0.0040179891511797905,
+            0.029425358399748802,
+            -0.0006726926076225936,
+            0.010309988632798195,
+            0.0004135536728426814,
+            -0.0004116835771128535,
+            -0.018978586420416832,
+            0.009207178838551044,
+            -0.024569574743509293,
+            -0.006415958981961012,
+            0.005595263559371233,
+            -0.01225059200078249,
+            -0.006142393685877323,
+            0.014413466677069664,
+            0.0022633245680481195,
+            -0.021047422662377357,
+            -0.008061625063419342,
+            -0.030485423281788826,
+            -0.021218402311205864,
+            0.031904544681310654,
+            0.006518546026200056,
+            -0.005646557081490755,
+            0.015379494056105614,
+            -0.022876890376210213,
+            -0.0054969508200883865,
+            -0.005035309586673975,
+            0.016636185348033905,
+            -0.020021554082632065,
+            0.004398415796458721,
+            -0.010634847916662693,
+            -0.016431011259555817,
+            0.023099161684513092,
+            0.008326641283929348,
+            0.0011220448650419712,
+            0.006133845075964928,
+            -0.02497992292046547,
+            -0.014285233803093433,
+            0.01798691228032112,
+            0.005702124908566475,
+            -0.008228329010307789,
+            0.01795271784067154,
+            -0.0012609646655619144,
+            -0.007215282414108515,
+            -0.013780847191810608,
+            0.01841435767710209,
+            -0.02183392271399498,
+            0.026535825803875923,
+            0.012156554497778416,
+            -0.010472417809069157,
+            0.008822478353977203,
+            0.004992565140128136,
+            0.01287466287612915,
+            -0.0003889755462296307,
+            -0.015370945446193218,
+            0.013421793468296528,
+            0.013712456449866295,
+            0.002194933360442519,
+            -0.0021778354421257973,
+            0.0037572472356259823,
+            0.005826083943247795,
+            -0.0054456572979688644,
+            0.02333853207528591,
+            0.023594999685883522,
+            0.0002186651690863073,
+            0.015080282464623451,
+            -0.02084225043654442,
+            -0.019611205905675888,
+            0.02378307469189167,
+            -0.004595040809363127,
+            -0.0314599983394146,
+            0.038709476590156555,
+            -0.010292891412973404,
+            -0.007236654870212078,
+            -0.01993606425821781,
+            0.00771111948415637,
+            0.020448999479413033,
+            -0.021885216236114502,
+            -0.010002228431403637,
+            -0.00110708421561867,
+            0.008848125115036964,
+            0.027852358296513557,
+            0.020568683743476868,
+            -0.015507727861404419,
+            0.02788655459880829,
+            0.009668820537626743,
+            0.008044526912271976,
+            0.011874440126121044,
+            0.005561067722737789,
+            -0.010840021073818207,
+            0.010044972412288189,
+            -0.0005241552134975791,
+            0.005296051502227783,
+            -0.00516354339197278,
+            -0.02231266349554062,
+            -0.015644511207938194,
+            -0.04178708791732788,
+            -0.0036589347291737795,
+            -0.008604480884969234,
+            0.01553337462246418,
+            0.08624143153429031,
+            -0.014037314802408218,
+            -0.009309765882790089,
+            -0.0020271609537303448,
+            0.009976581670343876,
+            -0.01720896176993847,
+            0.012566901743412018,
+            0.024466989561915398,
+            -0.013626967556774616,
+            -0.0329304113984108,
+            0.0025518503971397877,
+            -0.012994347140192986,
+            0.007805157452821732,
+            0.017542369663715363,
+            0.0019245740259066224,
+            -0.023680489510297775,
+            -0.0018433593213558197,
+            -0.020004456862807274,
+            0.001239592325873673,
+            0.010369831696152687,
+            -0.016131797805428505,
+            -0.010771630331873894,
+            0.023902760818600655,
+            -0.02017543464899063,
+            -0.025167999789118767,
+            -0.012925956398248672,
+            -0.007745314855128527,
+            -0.011891537345945835,
+            0.017815934494137764,
+            0.008647225797176361,
+            -0.0186708252876997,
+            -0.008595932275056839,
+            0.005915847606956959,
+            -0.011378603056073189,
+            8.963001164374873e-05,
+            0.011925733648240566,
+            -0.0033169782254844904,
+            0.008788282051682472,
+            0.01482381485402584,
+            -0.03400757536292076,
+            -0.003128902055323124,
+            -0.014704130589962006,
+            0.01360986940562725,
+            0.006557016167789698,
+            0.028057532384991646,
+            -0.028279803693294525,
+            -0.021765531972050667,
+            -0.00981415156275034,
+            -0.01892729289829731,
+            -0.03082738071680069,
+            -0.02692907489836216,
+            0.0058175353333354,
+            0.00636466545984149,
+            0.021526163443922997,
+            0.014729776419699192,
+            -0.014396369457244873,
+            0.010096265934407711,
+            -0.0033981928136199713,
+            0.008984907530248165,
+            -0.014404918067157269,
+            -0.0018454965902492404,
+            -0.015550472773611546,
+            0.015131575986742973,
+            0.017662053927779198,
+            -0.002583908848464489,
+            -0.018380163237452507,
+            -0.019730890169739723,
+            0.010429673828184605,
+            -0.007604258134961128,
+            -0.004637785255908966,
+            0.02668970637023449,
+            0.003182332729920745,
+            0.0019096133764833212,
+            0.007527317851781845,
+            0.023406922817230225,
+            0.008664323017001152,
+            0.03546088933944702,
+            -0.023406922817230225,
+            0.001999377040192485,
+            -0.0189443901181221,
+            0.0031866072677075863,
+            -0.023218847811222076,
+            0.005112249869853258,
+            -0.01942313089966774,
+            0.03417855501174927,
+            -0.006133845075964928,
+            -0.007852176204323769,
+            0.030468326061964035,
+            -0.02624516189098358,
+            0.002979296026751399,
+            -0.009754309430718422,
+            0.004650608636438847,
+            0.022141683846712112,
+            -0.0038256384432315826,
+            -0.017285902053117752,
+            -0.03925660625100136,
+            0.003522152081131935,
+            -0.009899641387164593,
+            -0.006091100629419088,
+            -0.011318760924041271,
+            0.01773044466972351,
+            0.005065230652689934,
+            0.005569616798311472,
+            -0.0015633824514225125,
+            0.016892651095986366,
+            0.0013913356233388186,
+            0.0035456616897135973,
+            0.029955390840768814,
+            -0.001200053608044982,
+            0.012558353133499622,
+            0.0033533109817653894,
+            0.005689301528036594,
+            0.023492412641644478,
+            -0.007745314855128527,
+            -0.0015206378884613514,
+            -0.017362842336297035,
+            0.0067237201146781445,
+            -0.0007555101765319705,
+            0.02520219422876835,
+            0.01539659220725298,
+            -0.01771334744989872,
+            -0.040658630430698395,
+            0.02159455418586731,
+            -0.011352956295013428,
+            0.027373619377613068,
+            -0.009497841820120811,
+            -0.006569839548319578,
+            -0.0003144397051073611,
+            -0.03754682466387749,
+            0.003182332729920745,
+            -0.009720114059746265,
+            -0.052798084914684296,
+            -0.018089499324560165,
+            -0.007548689842224121,
+            -0.026296455413103104,
+            0.03342624753713608,
+            -0.0014650699449703097,
+            -0.026775194332003593,
+            0.024706358090043068,
+            -0.0034516234882175922,
+            -0.00698873633518815,
+            0.012233494780957699,
+            -0.016704576089978218,
+            -0.009378157556056976,
+            -0.0032443124800920486,
+            -0.005116524174809456,
+            0.0006069728406146169,
+            -0.012421570718288422,
+            -0.019235054031014442,
+            0.015866782516241074,
+            -0.006980187259614468,
+            0.0008538226829841733,
+            0.015866782516241074,
+            -0.0020496018696576357,
+            0.025065412744879723,
+            -0.014122803695499897,
+            0.005385815165936947,
+            0.007612806744873524,
+            -0.013883434236049652,
+            0.012301885522902012,
+            0.003847010899335146,
+            -0.015430787578225136,
+            0.01580693945288658,
+            0.009215728379786015,
+            0.00848907046020031,
+            0.0041611832566559315,
+            -0.014533151872456074,
+            0.0022889713291078806,
+            -0.011421347968280315,
+            0.006646779831498861,
+            0.013909080997109413,
+            0.005304600577801466,
+            -0.011276016011834145,
+            -0.016653282567858696,
+            0.013857787474989891,
+            0.007501671090722084,
+            0.020004456862807274,
+            -0.03180195763707161,
+            0.007386260665953159,
+            -0.004676255397498608,
+            0.006941717118024826,
+            -0.01227623876184225,
+            0.025287684053182602,
+            0.022620422765612602,
+            0.02111581526696682,
+            0.006356116849929094,
+            0.0028767092153429985,
+            0.010515162721276283,
+            -0.01966249942779541,
+            -0.0033618600573390722,
+            -0.0204148031771183,
+            -0.0006417027907446027,
+            -0.0008324504015035927,
+            0.01743123307824135,
+            -0.012318983674049377,
+            0.013105483725667,
+            0.006522820331156254,
+            0.015447885729372501,
+            -0.02887822687625885,
+            0.0039047158788889647,
+            0.0070400298573076725,
+            0.0013421793701127172,
+            0.0066211330704391,
+            0.0031224903650581837,
+            0.08371095359325409,
+            0.010472417809069157,
+            -0.0070870486088097095,
+            0.003968833014369011,
+            -0.009437999688088894,
+            -0.0077538639307022095,
+            0.015251260250806808,
+            -0.01081437524408102,
+            0.012233494780957699,
+            0.007347790524363518,
+            -0.013148227706551552,
+            0.004582217428833246,
+            0.0073563395999372005,
+            0.008108643814921379,
+            -0.01623438484966755,
+            0.011190527118742466,
+            -0.005253307055681944,
+            0.0038961670361459255,
+            0.025133803486824036,
+            0.01652504876255989,
+            -0.005723497364670038,
+            0.00971156544983387,
+            0.006467252504080534,
+            0.0009863307932391763,
+            -0.012404472567141056,
+            0.006390312220901251,
+            -0.004152634646743536,
+            0.023013673722743988,
+            -0.016208738088607788,
+            0.0344863161444664,
+            -0.0005706399097107351,
+            -0.003855559742078185,
+            -0.010882765986025333,
+            -0.003383232280611992,
+            0.007501671090722084,
+            -0.008322366513311863,
+            0.01114778220653534,
+            -0.013712456449866295,
+            0.0005973553052172065,
+            -0.013045640662312508,
+            -0.008189858868718147,
+            0.004058596212416887,
+            -0.007540141232311726,
+            -0.0073777115903794765,
+            -0.0009211453725583851,
+            0.001756801619194448,
+            -4.190970867057331e-05,
+            -0.0012331806356087327,
+            -0.0012919544242322445,
+            -0.009437999688088894,
+            -0.004513826221227646,
+            -0.013601320795714855,
+            -1.5010688912298065e-05,
+            0.02137228287756443,
+            -0.003682444104924798,
+            -0.006809209007769823,
+            0.01044677197933197,
+            0.003128902055323124,
+            0.02429600991308689,
+            -0.006095374934375286,
+            -0.014558798633515835,
+            -0.0021564632188528776,
+            -0.007005834020674229,
+            -0.007390534970909357,
+            0.00033928497578017414,
+            0.0038790691178292036,
+            0.006180863827466965,
+            0.003287056926637888,
+            0.012438667938113213,
+            0.006018434651196003,
+            -0.008831026963889599,
+            0.011763304471969604,
+            0.003120353212580085,
+            -0.0009190081618726254,
+            -0.01891019567847252,
+            -0.003242175094783306,
+            -0.013550027273595333,
+            -0.01199412439018488,
+            -0.011455543339252472,
+            0.017345745116472244,
+            -0.01723460853099823,
+            0.019765086472034454,
+            -0.006339018698781729,
+            0.00765555165708065,
+            -0.02549285814166069,
+            0.002449263585731387,
+            0.008899418637156487,
+            0.011250369250774384,
+            0.006702347658574581,
+            -0.012857564724981785,
+            -0.002013268880546093,
+            -0.0010862462222576141,
+            0.02963053248822689,
+            -0.008702793158590794,
+            -0.010010777041316032,
+            0.02816011942923069,
+            -0.014413466677069664,
+            -0.01019030436873436,
+            0.0014362173387780786,
+            -0.0013090522261336446,
+            0.02552705444395542,
+            0.011455543339252472,
+            0.017328646034002304,
+            0.0054456572979688644,
+            0.002840376226231456,
+            0.0013379048323258758,
+            0.023936955258250237,
+            0.016071956604719162,
+            0.010908412747085094,
+            -0.01418264675885439,
+            -0.008446325547993183,
+            -0.01942313089966774,
+            0.014216842129826546,
+            0.0033746834378689528,
+            -0.01530255377292633,
+            0.011224722489714622,
+            -0.021953608840703964,
+            -0.019064076244831085,
+            -0.023697586730122566,
+            -0.010386928915977478,
+            0.013438890688121319,
+            -0.011686364188790321,
+            -0.005505499895662069,
+            -0.017089277505874634,
+            -0.005454206373542547,
+            -0.010455320589244366,
+            -0.015148673206567764,
+            -0.01673022285103798,
+            -0.02598869428038597,
+            -0.022688815370202065,
+            -0.0098397983238101,
+            -0.017294451594352722,
+            -0.008300994522869587,
+            0.012797722592949867,
+            -0.020448999479413033,
+            0.017662053927779198,
+            -0.005065230652689934,
+            0.0037893056869506836,
+            0.01188298873603344,
+            0.0216116514056921,
+            -0.01748252660036087,
+            -0.031904544681310654,
+            -0.02402244508266449,
+            -0.02135518379509449,
+            0.022705912590026855,
+            0.0021927962079644203,
+            0.024142129346728325,
+            -0.007125518750399351,
+            0.016157444566488266,
+            -0.018363064154982567,
+            0.019303444772958755,
+            -0.04202645644545555,
+            -0.03222940117120743,
+            -0.007168263662606478,
+            0.00034836819395422935,
+            -0.004362082574516535,
+            -0.0010199921671301126,
+            -0.015661608427762985,
+            -0.010993901640176773,
+            -0.007168263662606478,
+            -0.006210785359144211,
+            -0.009685918688774109,
+            -0.01516577135771513,
+            0.01007916871458292,
+            0.004370631650090218,
+            0.016165994107723236,
+            0.001985484967008233,
+            0.015559021383523941,
+            -0.01516577135771513,
+            0.006945991888642311,
+            -0.0022269918117672205,
+            -0.005753418430685997,
+            0.004646334331482649,
+            0.003804266219958663,
+            0.0044924537651240826,
+            0.0002427089784760028,
+            0.0011017411015927792,
+            -0.00029386888490989804,
+            0.017362842336297035,
+            0.013361950404942036,
+            -0.007146891206502914,
+            0.01773044466972351,
+            6.485151970991865e-05,
+            0.0016787927597761154,
+            -0.010429673828184605,
+            -0.003214391181245446,
+            -0.00911314133554697,
+            -0.008882320486009121,
+            0.015362396836280823,
+            -0.019730890169739723,
+            -0.010626298375427723,
+            0.026843586936593056,
+            -0.0018348104786127806,
+            -0.03443502262234688,
+            0.02742491289973259,
+            7.86015789344674e-06,
+            -0.0024364402052015066,
+            -0.01369535829871893,
+            -0.007078499998897314,
+            -0.0006363596767187119,
+            -0.0005978895933367312,
+            0.017337195575237274,
+            -0.018038205802440643,
+            -0.005249032285064459,
+            0.00630909763276577,
+            -0.0040436359122395515,
+            -0.0021254734601825476,
+            0.006497173570096493,
+            0.014028766192495823,
+            -0.00047820480540394783,
+            -0.03501634672284126,
+            -0.012190749868750572,
+            0.020568683743476868,
+            -0.017106374725699425,
+            -0.014994793571531773,
+            0.011233271099627018,
+            -0.001976936124265194,
+            -0.012601097114384174,
+            -0.003911127801984549,
+            -0.005223385989665985,
+            -0.010993901640176773,
+            0.008442051708698273,
+            0.010224499739706516,
+            0.008339464664459229,
+            0.02210748940706253,
+            -0.014447662979364395,
+            -0.005757692735642195,
+            -0.015251260250806808,
+            0.0031075298320502043,
+            -0.022962380200624466,
+            -0.00015214394079521298,
+            -0.009745760820806026,
+            0.0016766554908826947,
+            0.012575451284646988,
+            0.01749962382018566,
+            0.028724348172545433,
+            0.0089678093791008,
+            0.0072879483923316,
+            -0.010848570615053177,
+            -0.01456734724342823,
+            0.004680529702454805,
+            0.0031417254358530045,
+            -0.00558671448379755,
+            0.006655328441411257,
+            0.019286347553133965,
+            0.0013261501444503665,
+            0.019303444772958755,
+            0.0025133804883807898,
+            0.0037978545296937227,
+            -2.8652215405600145e-05,
+            -0.010455320589244366,
+            -0.006706621963530779,
+            -0.02374888025224209,
+            0.0007453583530150354,
+            0.005779065191745758,
+            0.01889309659600258,
+            0.0005540763959288597,
+            0.0030370012391358614,
+            0.026416141539812088,
+            -0.0029771588742733,
+            -0.014345075935125351,
+            -0.012592548504471779,
+            -0.011942830868065357,
+            -0.00874553807079792,
+            0.034332435578107834,
+            0.01454170048236847,
+            -0.01454170048236847,
+            -0.015234163030982018,
+            -0.011566678993403912,
+            0.03033154271543026,
+            0.00442833686247468,
+            -0.0004928982816636562,
+            0.021902315318584442,
+            0.018294673413038254,
+            0.017345745116472244,
+            0.016362618654966354,
+            0.009420901536941528,
+            0.0027912200894206762,
+            0.020466096699237823,
+            0.0019480835180729628,
+            0.015088831074535847,
+            -0.008172760717570782,
+            0.008642951026558876,
+            -0.005864554084837437,
+            0.012712233699858189,
+            -0.010677591897547245,
+            -0.011797499842941761,
+            0.008433502167463303,
+            -0.014003119431436062,
+            0.002117993077263236,
+            -0.007373437285423279,
+            0.00948074460029602,
+            -0.02017543464899063,
+            -0.007599983364343643,
+            0.012190749868750572,
+            -0.014533151872456074,
+            0.0024043817538768053,
+            -0.022893989458680153,
+            -0.0009761789697222412,
+            0.014695581048727036,
+            -0.0025368898641318083,
+            0.0066125839948654175,
+            0.005321698263287544,
+            0.00012549536768347025,
+            0.010104815475642681,
+            -0.014789619483053684,
+            0.004030812531709671,
+            -0.005377266090363264,
+            -0.004834410268813372,
+            -0.00637748884037137,
+            -0.0011712011182680726,
+            0.0009542724001221359,
+            -0.005556793417781591,
+            0.01991896703839302,
+            0.008091545663774014,
+            0.00435994565486908,
+            -0.005270404741168022,
+            -0.012831917963922024,
+            -0.026159673929214478,
+            0.0145758967846632,
+            -0.015447885729372501,
+            0.01528545655310154,
+            0.0013550026342272758,
+            -0.003092569299042225,
+            -0.014370722696185112,
+            0.010643396526575089,
+            0.0056337337009608746,
+            0.00474037230014801,
+            0.015114477835595608,
+            -0.011481190100312233,
+            0.009583331644535065,
+            0.009147336706519127,
+            -0.009267021901905537,
+            -0.0067493668757379055,
+            0.0005936151137575507,
+            0.006279176566749811,
+            -0.0232701413333416,
+            -0.0016862730262801051,
+            0.008292445912957191,
+            -0.01962830312550068,
+            0.008762635290622711,
+            -0.03730745613574982,
+            0.0020068571902811527,
+            -0.012088162824511528,
+            0.011258917860686779,
+            -0.002776259556412697,
+            -0.033374954015016556,
+            -0.026518728584051132,
+            0.0016745183384045959,
+            0.0040179891511797905,
+            -0.009993678890168667,
+            0.009010554291307926,
+            0.009275570511817932,
+            0.011532483622431755,
+            0.01443056482821703,
+            0.0098397983238101,
+            0.010113364085555077,
+            0.04223163053393364,
+            -0.04151352122426033,
+            -0.007146891206502914,
+            0.0070656766183674335,
+            0.02209039032459259,
+            -0.010267244651913643,
+            -0.011361504904925823,
+            0.008685695938766003,
+            -0.003859834047034383,
+            0.034349530935287476,
+            -0.024877335876226425,
+            0.014994793571531773,
+            -0.0040543219074606895,
+            -0.00010605995339574292,
+            -0.024159228429198265,
+            -0.0020325039513409138,
+            -0.01969669573009014,
+            0.0032357636373490095,
+            -0.002079523168504238,
+            -0.017191864550113678,
+            0.018021108582615852,
+            -0.004595040809363127,
+            0.012789173983037472,
+            0.007813706062734127,
+            -0.01625148393213749,
+            0.004526649136096239,
+            -0.015593216754496098,
+            0.032896216958761215,
+            -0.012105260975658894,
+            0.005911573302000761,
+            0.00873698852956295,
+            -0.006779287941753864,
+            0.016439558938145638,
+            0.009352510794997215,
+            -0.006044081412255764,
+            -0.0051592690870165825,
+            -0.002759161638095975,
+            -0.0023680487647652626,
+            0.008493345230817795,
+            -0.021474869921803474,
+            0.027493303641676903,
+            -0.006894698366522789,
+            -0.014037314802408218,
+            0.006266353186219931,
+            -0.001722605898976326,
+            0.002626653527840972,
+            0.004406964406371117,
+            -0.0018059578724205494,
+            -0.0007939802599139512,
+            -0.004958369303494692,
+            -0.017798837274312973,
+            -0.010130461305379868,
+            0.0010467075044289231,
+            -0.0016114701284095645,
+            0.013105483725667,
+            -0.010540809482336044,
+            -0.016601989045739174,
+            0.005112249869853258,
+            0.00026929075829684734,
+            -0.003975244704633951,
+            -0.016807163134217262,
+            0.007215282414108515,
+            -0.007544415537267923,
+            0.005372991785407066,
+            -0.008168485946953297,
+            0.016448108479380608,
+            0.008313817903399467,
+            -0.0003630616411101073,
+            0.01767915114760399,
+            0.024706358090043068,
+            0.02061997726559639,
+            0.01152393501251936,
+            -0.040214087814092636,
+            -0.0189443901181221,
+            -0.0025091059505939484,
+            0.018807608634233475,
+            -0.0008981701685115695,
+            -0.006420233752578497,
+            -0.0017632133094593883,
+            0.0029835705645382404,
+            -0.003084020223468542,
+            -0.0007474955637007952,
+            0.016918297857046127,
+            -0.020192531868815422,
+            0.0032742335461080074,
+            -0.012720782309770584,
+            0.014404918067157269,
+            0.0006860502762719989,
+            -0.0006737611838616431,
+            5.767176844528876e-05,
+            0.00945509783923626,
+            0.014900755137205124,
+            0.014661385677754879,
+            0.0014159136917442083,
+            0.019064076244831085,
+            -0.003663209266960621,
+            -0.0016510088462382555,
+            -0.018636630848050117,
+            -0.01696104370057583,
+            -0.0034494863357394934,
+            0.01298579853028059,
+            -0.0015601766062900424,
+            0.011575227603316307,
+            -0.014191195368766785,
+            0.01649085246026516,
+            0.025680935010313988,
+            -0.012840467505156994,
+            0.005069505423307419,
+            0.002564673777669668,
+            -0.001211808412335813,
+            0.0064800758846104145,
+            -0.007381986360996962,
+            -0.004030812531709671,
+            -0.012438667938113213,
+            0.01868792437016964,
+            -0.0032208028715103865,
+            0.018277576193213463,
+            0.013635516166687012,
+            0.011968477629125118,
+            -0.00032993461354635656,
+            0.008305269293487072,
+            0.018636630848050117,
+            0.014413466677069664,
+            -0.0005158734857104719,
+            0.0011615835828706622,
+            -0.01284901611506939,
+            0.0023594999220222235,
+            0.016311325132846832,
+            -0.02715134806931019,
+            -0.007685472723096609,
+            0.011840244755148888,
+            -0.016567792743444443,
+            0.0025603994727134705,
+            -0.02597159706056118,
+            -0.011472641490399837,
+            0.012053967453539371,
+            0.010233048349618912,
+            -0.011053744703531265,
+            0.001883966731838882,
+            0.001705508097074926,
+            -0.0008377934573218226,
+            -0.00936105940490961,
+            0.0167815163731575,
+            -0.005868828855454922,
+            -0.011258917860686779,
+            0.011686364188790321,
+            -0.00981415156275034,
+            0.012695135548710823,
+            0.004603589419275522,
+            0.011746206320822239,
+            -0.0024578124284744263,
+            0.026877781376242638,
+            0.01287466287612915,
+            0.007236654870212078,
+            0.0004386660875752568,
+            0.005937220063060522,
+            -0.004847233649343252,
+            0.007407633122056723,
+            0.026570022106170654,
+            -0.00048728802357800305,
+            -0.0116350706666708,
+            -0.0013197384541854262,
+            0.010951157659292221,
+            -0.01988477073609829,
+            0.0023124809376895428,
+            0.005676478147506714,
+            0.028211412951350212,
+            0.014327977783977985,
+            0.0025796345435082912,
+            -0.0204148031771183,
+            -0.00023709874949418008,
+            -0.00041595802758820355,
+            0.04106897860765457,
+            0.0026116929948329926,
+            -0.012532706372439861,
+            0.0021489830687642097,
+            0.013849238865077496,
+            -0.005590988788753748,
+            -0.004834410268813372,
+            -0.014310880564153194,
+            -0.008027428761124611,
+            -0.006018434651196003,
+            -0.0198676735162735,
+            0.001279131043702364,
+            0.005800437182188034,
+            -0.013319206424057484,
+            0.007471750024706125,
+            -0.006403135601431131,
+            -0.03031444549560547,
+            -0.005932945758104324,
+            -0.0034131535794585943,
+            -0.023731783032417297,
+            -0.007125518750399351,
+            0.0089678093791008,
+            0.005915847606956959,
+            -0.014122803695499897,
+            -0.025338977575302124,
+            0.020996129140257835,
+            0.007668375037610531,
+            0.03392208740115166,
+            -0.012669488787651062,
+            -0.007185361348092556,
+            -0.00948074460029602,
+            0.008557462133467197,
+            -0.023150455206632614,
+            -0.008715616539120674,
+            0.009061847813427448,
+            0.01767915114760399,
+            0.012404472567141056,
+            -0.011370054446160793,
+            0.0015163634670898318,
+            0.012122358195483685,
+            -0.00010085045505547896,
+            0.02764718420803547,
+            -0.009104591794312,
+            -0.014994793571531773,
+            0.010754532180726528,
+            -0.003259273013100028,
+            0.01916666328907013,
+            -0.0006203305092640221,
+            -0.0025924579240381718,
+            0.012917407788336277,
+            0.0038341875188052654,
+            -0.012934505008161068,
+            -0.005240483675152063,
+            0.008647225797176361,
+            -0.028553368523716927,
+            0.02792074903845787,
+            0.015909526497125626,
+            -0.02816011942923069,
+            -0.008305269293487072,
+            -0.0025091059505939484,
+            0.017140571027994156,
+            0.020089944824576378,
+            -0.011446994729340076,
+            -0.009728662669658661,
+            0.014960597269237041,
+            0.0034217024222016335,
+            0.0077538639307022095,
+            -0.0026565745938569307,
+            0.01896148920059204,
+            -0.004370631650090218,
+            -0.007800883147865534,
+            -0.003408879041671753,
+            -0.012789173983037472,
+            -0.0003505054337438196,
+            0.015618863515555859,
+            0.012977249920368195,
+            -0.005984238814562559,
+            -0.008771184831857681,
+            0.015208516269922256,
+            -0.022432347759604454,
+            0.0010841090697795153,
+            0.010908412747085094,
+            -0.01745687983930111,
+            -0.007561513222754002,
+            0.01767915114760399,
+            -0.0031395882833749056,
+            0.011258917860686779,
+            -0.007381986360996962,
+            0.007856450974941254,
+            0.0067237201146781445,
+            0.031135139986872673,
+            -0.007476024329662323,
+            0.03525571525096893,
+            -0.009309765882790089,
+            0.01991896703839302,
+            0.037991367280483246,
+            0.01455025002360344,
+            -0.0031118043698370457,
+            -0.008335189893841743,
+            0.009198630228638649,
+            -0.010096265934407711,
+            0.002415067981928587,
+            0.013524380512535572,
+            -0.026809390634298325,
+            -0.01418264675885439,
+            -0.011378603056073189,
+            0.013233717530965805,
+            0.0034580351784825325,
+            -0.013661162927746773,
+            -0.011575227603316307,
+            0.019508618861436844,
+            0.010352733545005322,
+            -0.0038705202750861645,
+            -0.007869274355471134,
+            -0.02157745696604252,
+            0.0020688369404524565,
+            0.012088162824511528,
+            -0.009164434857666492,
+            -0.011532483622431755,
+            0.016670379787683487,
+            -0.01261819526553154,
+            0.0027228286489844322,
+            -0.008300994522869587,
+            -0.027989141643047333,
+            -0.005800437182188034,
+            -0.00491989916190505,
+            -0.0019844165071845055,
+            -0.005565342493355274,
+            0.006471527274698019,
+            0.005227660294622183,
+            -0.01824337989091873,
+            0.0024278913624584675,
+            -0.008788282051682472,
+            0.00861302949488163,
+            0.00436635734513402,
+            0.006249255500733852,
+            -0.0019384659826755524,
+            0.014225390739738941,
+            0.013661162927746773,
+            -0.005377266090363264,
+            -0.020773857831954956,
+            0.012772075831890106,
+            -0.03634997829794884,
+            0.020722564309835434,
+            0.008694244548678398,
+            0.0025411644019186497,
+            -0.03086157515645027,
+            0.01018175482749939,
+            0.01115633174777031,
+            0.0160035640001297,
+            0.011370054446160793,
+            0.0016766554908826947,
+            0.013190972618758678,
+            -0.01114778220653534,
+            0.006766464561223984,
+            -7.192273642431246e-06,
+            -0.01482381485402584,
+            0.020688369870185852,
+            0.011814597994089127,
+            0.02913469448685646,
+            0.002951512113213539,
+            -0.0011209761723876,
+            0.001963044051080942,
+            0.00899345614016056,
+            -0.007685472723096609,
+            0.0030904319137334824,
+            0.004954094998538494,
+            0.0058175353333354,
+            0.013190972618758678,
+            0.007745314855128527,
+            -0.00860875565558672,
+            0.007839352823793888,
+            0.0026736725121736526,
+            -0.014020216651260853,
+            -0.000775279535446316,
+            -0.006997285410761833,
+            0.007552964612841606,
+            -0.01505463570356369,
+            0.004560844972729683,
+            0.04342847689986229,
+            0.012413021177053452,
+            -0.0034323884174227715,
+            -0.017354292795062065,
+            -0.011079391464591026,
+            -0.01273788046091795,
+            -0.01272933091968298,
+            -0.0053644427098333836,
+            -0.02303077094256878,
+            -0.003434525802731514,
+            0.0065912120044231415,
+            0.02453538030385971,
+            0.01092551089823246,
+            -0.017191864550113678,
+            -0.005813260562717915,
+            -0.02380017377436161,
+            -0.010566456243395805,
+            -0.016191640868782997,
+            -0.016319874674081802,
+            -0.008061625063419342,
+            0.018978586420416832,
+            -0.008839575573801994,
+            0.01408860832452774,
+            0.002968610031530261,
+            -0.0002393695613136515,
+            -0.0020431901793926954,
+            -0.000852754048537463,
+            -0.0051549943163990974,
+            0.0015911663649603724,
+            0.003152411663904786,
+            -0.0005375128821469843,
+            0.020722564309835434,
+            -0.007800883147865534,
+            0.003866245737299323,
+            -0.0026929075829684734,
+            0.01542223896831274,
+            0.007108421064913273,
+            0.0028681603725999594,
+            -0.0238172709941864,
+            0.02304787002503872,
+            -0.024381499737501144,
+            -0.014156999997794628,
+            0.02284269593656063,
+            -0.008108643814921379,
+            0.004804489202797413,
+            0.024107934907078743,
+            0.00029493749025277793,
+            -0.0009697672794573009,
+            0.015123026445508003,
+            0.004045772831887007,
+            0.011258917860686779,
+            0.015242711640894413,
+            0.007082774303853512,
+            -0.0052875024266541,
+            0.019098270684480667,
+            0.0073306928388774395,
+            -0.006403135601431131,
+            -0.01626003161072731,
+            0.0014244626509025693,
+            0.004238123539835215,
+            -0.010044972412288189,
+            -0.004137673880904913,
+            0.008630127646028996,
+            0.004453983623534441,
+            -0.007843627594411373,
+            -0.0012972975382581353,
+            -0.024706358090043068,
+            -0.005821809638291597,
+            -0.01530255377292633,
+            -0.006039807107299566,
+            -0.023406922817230225,
+            0.005676478147506714,
+            -0.009053298272192478,
+            0.01672167330980301,
+            0.00016403225890826434,
+            0.024347303435206413,
+            0.0020154062658548355,
+            -0.0007266575703397393,
+            -0.0051592690870165825,
+            0.009318315424025059,
+            -0.0007266575703397393,
+            0.027236836031079292,
+            -0.00011026761785615236,
+            -0.005911573302000761,
+            0.01032708678394556,
+            -0.01769625023007393,
+            0.020876444876194,
+            0.006356116849929094,
+            0.01454170048236847,
+            -0.015610314905643463,
+            -0.0073563395999372005,
+            0.035289913415908813,
+            0.024843141436576843,
+            0.0035029170103371143,
+            0.019816379994153976,
+            -0.010455320589244366,
+            0.016311325132846832,
+            0.003013491863384843,
+            -0.002389421220868826,
+            0.011010999791324139,
+            -0.005420010536909103,
+            -0.014627190306782722,
+            -0.006997285410761833,
+            0.03911982476711273,
+            -0.014832363463938236,
+            -6.972707342356443e-05,
+            -0.005060956347733736,
+            -0.010104815475642681,
+            -0.030929965898394585,
+            0.011609423905611038,
+            0.011951380409300327,
+            -0.024706358090043068,
+            -0.011566678993403912,
+            0.0032699592411518097,
+            -0.00026448199059814215,
+            -0.0030348640866577625,
+            -0.005706399213522673,
+            0.006509996950626373,
+            -0.0017375665483996272,
+            0.009275570511817932,
+            0.005569616798311472,
+            -0.0009617527248337865,
+            0.00013798479631077498,
+            0.005539695732295513,
+            0.011780401691794395,
+            0.006206510588526726,
+            0.0005498019745573401,
+            -0.007381986360996962,
+            0.00558671448379755,
+            0.010745983570814133,
+            -0.015088831074535847,
+            0.020739663392305374,
+            0.013823592104017735,
+            -0.016576342284679413,
+            0.0038149524480104446,
+            -0.019132466986775398,
+            -0.0329304113984108,
+            0.004787391051650047,
+            -0.0009029789362102747,
+            0.021748434752225876,
+            -0.018311770632863045,
+            -0.00995948351919651,
+            0.00442833686247468,
+            0.003908990416675806,
+            -0.011421347968280315,
+            0.019371837377548218,
+            0.007215282414108515,
+            -0.0017664191545918584,
+            0.01745687983930111,
+            0.016593439504504204,
+            -0.006403135601431131,
+            -0.00907894503325224,
+            0.018824705854058266,
+            -0.00437276903539896,
+            0.02744201011955738,
+            0.026621313765645027,
+            0.009207178838551044,
+            -0.018602434545755386,
+            0.0018262615194544196,
+            0.0080231549218297,
+            -0.02036350965499878,
+            -0.005590988788753748,
+            0.015670157968997955,
+            0.005402912851423025,
+            -0.0003841667785309255,
+            -0.0050737797282636166,
+            -0.022175880149006844,
+            0.007407633122056723,
+            0.011840244755148888,
+            -0.005189190153032541,
+            -0.028809836134314537,
+            -0.0011199075961485505,
+            -0.01431942917406559,
+            -0.0007694021333009005,
+            0.0010648739989846945,
+            0.01236172765493393,
+            0.009036201052367687,
+            0.028074629604816437,
+            -0.009865445084869862,
+            -0.00885667372494936,
+            0.010173206217586994,
+            0.009797054342925549,
+            0.012583999894559383,
+            0.012327532283961773,
+            0.013986021280288696,
+            0.007292222697287798,
+            -0.015960820019245148,
+            -0.004966918379068375,
+            0.007527317851781845,
+            -0.00899345614016056,
+            0.003985930699855089,
+            -0.01358422264456749,
+            0.020278021693229675,
+            0.015259809792041779,
+            0.0008260387112386525,
+            0.004804489202797413,
+            -0.004753195680677891,
+            -0.0012684449320659041,
+            -0.06483495235443115,
+            -0.013532929122447968,
+            0.00861302949488163,
+            0.021252596750855446,
+            -0.016396814957261086,
+            0.00046244275290519,
+            0.0014287370722740889,
+            0.008788282051682472,
+            -0.011481190100312233,
+            0.01020740158855915,
+            0.014524603262543678,
+            -0.005223385989665985,
+            -0.0035349754616618156,
+            0.005099426489323378,
+            -0.0016403226181864738,
+            -0.00013371034583542496,
+            0.004958369303494692,
+            0.014900755137205124,
+            0.021406477317214012,
+            -0.004385592415928841,
+            0.018585337325930595,
+            0.004112027119845152,
+            -0.015080282464623451,
+            -0.0027014564257115126,
+            0.013575674034655094,
+            -0.02328723855316639,
+            -0.004607864189893007,
+            0.007480298634618521,
+            -0.006980187259614468,
+            0.010908412747085094,
+            -0.004069282673299313,
+            -0.004406964406371117,
+            0.008677146397531033,
+            0.01420829351991415,
+            0.002276147948578,
+            -0.026775194332003593,
+            0.0034665842540562153,
+            -0.011233271099627018,
+            0.0035093287006020546,
+            0.01020740158855915,
+            -0.008219779469072819,
+            1.5344630810432136e-05,
+            -0.015328200533986092,
+            -0.002739926567301154,
+            0.024860238656401634,
+            -0.004289417061954737,
+            0.0067707388661801815,
+            0.017302999272942543,
+            -0.007018657401204109,
+            0.002389421220868826,
+            -0.02862176112830639,
+            0.007796608377248049,
+            -0.0369655005633831,
+            -0.0004579011583700776,
+            0.004984016064554453,
+            0.004097066354006529,
+            -0.010104815475642681,
+            -0.030211858451366425,
+            0.008412130177021027,
+            -0.00474037230014801,
+            -0.006753641180694103,
+            -0.022723009809851646,
+            -0.007937666028738022,
+            0.008429228328168392,
+            -0.006941717118024826,
+            0.010147559456527233,
+            -0.017662053927779198,
+            0.017815934494137764,
+            0.015439336188137531,
+            0.013302108272910118,
+            0.003968833014369011,
+            0.0032379007898271084,
+            0.005129347555339336,
+            0.007946214638650417,
+            0.004701902158558369,
+            0.004697627853602171,
+            -0.01019030436873436,
+            -0.011464091949164867,
+            0.01575564593076706,
+            0.004255221225321293,
+            0.00860875565558672,
+            -0.0018604571232572198,
+            0.002479184651747346,
+            -0.0073050460778176785,
+            -0.00045629823580384254,
+            -0.014396369457244873,
+            0.0021596690639853477,
+            0.01199412439018488,
+            0.015499179251492023,
+            0.0014650699449703097,
+            -0.00625352980569005,
+            -0.007634179200977087,
+            -0.00667670089751482,
+            0.020209630951285362,
+            -0.017149118706583977,
+            -0.006698073353618383,
+            -0.012635293416678905,
+            0.021526163443922997,
+            0.01599501632153988,
+            0.0035563476849347353,
+            -0.006826307158917189,
+            -0.004092792049050331,
+            0.0097030159085989,
+            0.008570285513997078,
+            0.006766464561223984,
+            -0.023663390427827835,
+            -0.005377266090363264,
+            -0.019081173464655876,
+            -0.01539659220725298,
+            -0.01406296156346798,
+            0.021218402311205864,
+            -0.004924173932522535,
+            -0.007471750024706125,
+            0.008454875089228153,
+            0.024655064567923546,
+            0.007916293106973171,
+            0.007715393789112568,
+            0.00503958435729146,
+            -0.01726025529205799,
+            0.012071064673364162,
+            -0.010139010846614838,
+            -0.03665773943066597,
+            0.0065912120044231415,
+            -0.00971156544983387,
+            0.02598869428038597,
+            -0.017183315008878708,
+            -0.0011167017510160804,
+            -0.002635202370584011,
+            -0.030878672376275063,
+            -0.005638008005917072,
+            0.02961343340575695,
+            -0.010737434960901737,
+            0.011669266037642956,
+            -0.010062070563435555,
+            -0.009950934909284115,
+            0.01019030436873436,
+            0.013062738813459873,
+            -0.011720559559762478,
+            -0.002940826117992401,
+            0.014721227809786797,
+            0.02453538030385971,
+            0.0035093287006020546,
+            -0.004548021592199802,
+            0.009429451078176498,
+            0.0123702771961689,
+            -0.006804934702813625,
+            0.019559912383556366,
+            -0.007561513222754002,
+            -0.0015078145079314709,
+            -0.00656556524336338,
+            0.0012235631002113223,
+            0.006193687207996845,
+            -0.019577009603381157,
+            0.017320098355412483,
+            -0.009916738606989384,
+            0.008873771876096725,
+            0.0028702975250780582,
+            0.01726025529205799,
+            0.011267467401921749,
+            0.01896148920059204,
+            -0.02891242317855358,
+            -0.001095329411327839,
+            -0.0276813805103302,
+            -0.015772745013237,
+            -0.010703238658607006,
+            -0.00873698852956295,
+            -0.002485596342012286,
+            -0.007796608377248049,
+            -0.023868564516305923,
+            -0.005625184625387192,
+            0.0007939802599139512,
+            0.0036781697999686003,
+            -0.017362842336297035,
+            0.01007061917334795,
+            1.5286190318875015e-05,
+            -0.016593439504504204,
+            -0.020825151354074478,
+            -0.002087003318592906,
+            -0.022021999582648277,
+            -0.012464314699172974,
+            -0.005005388520658016,
+            -0.0022248546592891216,
+            -0.0012139456812292337,
+            -0.0005172092351131141,
+            0.027544597163796425,
+            -0.019457325339317322,
+            0.004381317645311356,
+            0.02744201011955738,
+            -0.0019352601375430822,
+            0.005035309586673975,
+            -0.006668151821941137,
+            0.012532706372439861,
+            0.0032913314644247293,
+            -0.003973107319325209,
+            -0.022688815370202065,
+            0.02521929331123829,
+            0.0005311011918820441,
+            0.013669711537659168,
+            0.010139010846614838,
+            -0.008065898902714252,
+            -0.005783339496701956,
+            0.00485150795429945,
+            -0.03253716230392456,
+            0.0028531996067613363,
+            -0.013011445291340351,
+            -0.01865372806787491,
+            -0.005539695732295513,
+            -0.004586491733789444,
+            -0.006561290472745895,
+            -0.0071939099580049515,
+            0.017285902053117752,
+            -0.004377043340355158,
+            0.02376597747206688,
+            -0.007146891206502914,
+            -0.00453947251662612,
+            0.010908412747085094,
+            -0.04301812872290611,
+            -0.007240929175168276,
+            -0.006129570305347443,
+            -0.01843145675957203,
+            0.019799282774329185,
+            0.012173651717603207,
+            -0.001380649395287037,
+            -0.020243825390934944,
+            0.019816379994153976,
+            -0.0030028056353330612,
+            0.02552705444395542,
+            0.010634847916662693,
+            0.010173206217586994,
+            0.004364219959825277,
+            -0.010840021073818207,
+            -0.010113364085555077,
+            0.018311770632863045,
+            0.006304823327809572,
+            -0.011609423905611038,
+            0.01873921789228916,
+            0.004445434547960758,
+            -0.00186686881352216,
+            -0.003041275776922703,
+            0.018756315112113953,
+            0.024603771045804024,
+            0.013002896681427956,
+            0.002274010796099901,
+            -0.016174543648958206,
+            -0.006475801579654217,
+            0.0037038165610283613,
+            0.036418367177248,
+            0.012036869302392006,
+            0.012994347140192986,
+            -0.008202682249248028,
+            0.027373619377613068,
+            -0.009181532077491283,
+            0.009044749662280083,
+            0.00863440241664648,
+            0.017174765467643738,
+            0.0035328383091837168,
+            0.020329315215349197,
+            0.010857119224965572,
+            -0.00764272827655077,
+            0.02405664138495922,
+            0.007625630125403404,
+            0.003767933463677764,
+            0.006625407375395298,
+            -0.01844855397939682,
+            0.012626743875443935,
+            -0.011609423905611038,
+            -0.005950043443590403,
+            0.004505277145653963,
+            0.002058150712400675,
+            0.02374888025224209,
+            0.008027428761124611,
+            -0.03135741129517555,
+            -0.0017867228016257286,
+            -0.023612096905708313,
+            0.004340710584074259,
+            -0.0167387705296278,
+            0.012447217479348183,
+            0.004092792049050331,
+            -0.01774754375219345,
+            -0.018790511414408684,
+            -0.010728885419666767,
+            -0.01942313089966774,
+            -0.004924173932522535,
+            -0.012173651717603207,
+            -0.010840021073818207,
+            0.006954540498554707,
+            -0.0025069687981158495,
+            0.0024813220370560884,
+            0.006963089574128389,
+            0.019046977162361145,
+            0.020260922610759735,
+            0.0035819944459944963,
+            0.018824705854058266,
+            -0.0009019103017635643,
+            -0.055431149899959564,
+            0.0036589347291737795,
+            0.009309765882790089,
+            -0.0002863885893020779,
+            0.026108380407094955,
+            -0.004041498526930809,
+            0.00042557556298561394,
+            0.007223831489682198,
+            0.006937442813068628,
+            0.013276461511850357,
+            -0.004210339393466711,
+            -0.025886109098792076,
+            -0.019337641075253487,
+            0.0040393611416220665,
+            -0.006155217066407204,
+            0.008100095205008984,
+            -0.006685249973088503,
+            0.004761744290590286,
+            0.002353088231757283,
+            -0.009745760820806026,
+            -0.006595486309379339,
+            0.009617527015507221,
+            0.0014469034504145384,
+            -0.010386928915977478,
+            -0.013173874467611313,
+            -0.0018690060824155807,
+            0.01824337989091873,
+            -0.009019102901220322,
+            -0.024364402517676353,
+            0.009660271927714348,
+            -0.010010777041316032,
+            -0.0014821677468717098,
+            0.025869010016322136,
+            0.02429600991308689,
+            -0.003970969934016466,
+            0.019371837377548218,
+            -0.0021190617699176073,
+            -0.016405364498496056,
+            -0.03029734641313553,
+            -0.006873325910419226,
+            -0.028997913002967834,
+            0.027253935113549232,
+            -0.006424508057534695,
+            -0.0013176011852920055,
+            0.0035285637713968754,
+            -0.00413553649559617,
+            0.003434525802731514,
+            -0.009788504801690578,
+            -0.007672649342566729,
+            -0.02716844528913498,
+            0.007527317851781845,
+            -0.0069075217470526695,
+            -0.008172760717570782,
+            0.00031550831045024097,
+            0.039701152592897415,
+            -0.004556570667773485,
+            0.00850189384073019,
+            0.006232157349586487,
+            0.011865891516208649,
+            0.024193422868847847,
+            -0.008463423699140549,
+            -0.012096711434423923,
+            -0.015046087093651295,
+            -0.0020154062658548355,
+            0.00014052275219000876,
+            -0.003975244704633951,
+            0.022876890376210213,
+            -0.013019993901252747,
+            -0.015191418118774891,
+            -0.00034489520476199687,
+            -0.023389825597405434,
+            -0.011737657710909843,
+            0.009968032129108906,
+            0.012908858247101307,
+            0.007535866927355528,
+            -0.014003119431436062,
+            -0.014268135651946068,
+            -0.022945282980799675,
+            0.01800401136279106,
+            0.0976627841591835,
+            0.0012545529752969742,
+            -0.009070396423339844,
+            0.0026758096646517515,
+            0.010771630331873894,
+            -0.0008372591692022979,
+            -0.017559466883540154,
+            0.05919267237186432,
+            -0.013567124493420124,
+            0.005856005474925041,
+            -0.009010554291307926,
+            0.01007916871458292,
+            0.005031035281717777,
+            -0.00013257493264973164,
+            0.005928670987486839,
+            -0.008796831592917442,
+            0.006129570305347443,
+            -0.03645256534218788,
+            -0.00186686881352216,
+            -0.011344407685101032,
+            -0.006381763610988855,
+            -0.016824260354042053,
+            0.009275570511817932,
+            -0.02911759726703167,
+            -0.00922427698969841,
+            0.0020880720112472773,
+            -0.0059543177485466,
+            0.012336080893874168,
+            -0.01628567837178707,
+            -0.0017995461821556091,
+            0.005368717480450869,
+            0.005120798945426941,
+            -0.0036867186427116394,
+            -0.0074632009491324425,
+            -0.013310656882822514,
+            0.005065230652689934,
+            0.015576119534671307,
+            0.005548244342207909,
+            -0.008228329010307789,
+            0.00959188025444746,
+            0.016388265416026115,
+            0.0009510665549896657,
+            0.013661162927746773,
+            0.009412352927029133,
+            0.0007950488943606615,
+            0.0007875685696490109,
+            0.019611205905675888,
+            -0.016345521435141563,
+            0.02137228287756443,
+            -0.015883879736065865,
+            -0.02843368425965309,
+            -0.006236432120203972,
+            0.006411684677004814,
+            -0.014413466677069664,
+            0.0010910549899563193,
+            0.006582662928849459,
+            0.04100058600306511,
+            0.00863440241664648,
+            -0.0034281141124665737,
+            -0.0009259541402570903,
+            -0.013669711537659168,
+            0.023954054340720177,
+            0.01091696135699749,
+            0.030502520501613617,
+            0.007604258134961128,
+            0.00373373762704432,
+            -0.007001559715718031,
+            -0.009557684883475304,
+            -0.003487956477329135,
+            0.00474037230014801,
+            0.0025069687981158495,
+            0.004415513481944799,
+            0.008523265831172466,
+            -0.014498956501483917,
+            -0.02036350965499878,
+            0.017610760405659676,
+            -0.00474037230014801,
+            -0.023663390427827835,
+            0.012609646655619144,
+            -0.0316309779882431,
+            0.0047831167466938496,
+            0.0010344184702262282,
+            -0.005877377465367317,
+            -0.013960374519228935,
+            0.006300548557192087,
+            0.01151538547128439,
+            -0.009378157556056976,
+            -0.015482081100344658,
+            -0.008454875089228153,
+            -0.010592103004455566,
+            0.0001227570464834571,
+            0.0037657960783690214,
+            -0.011583777144551277,
+            -0.00394318625330925,
+            -0.01796981506049633,
+            0.007142616901546717,
+            -0.003013491863384843,
+            0.004654882941395044,
+            -0.011122135445475578,
+            0.011558130383491516,
+            0.020226728171110153,
+            -0.009788504801690578,
+            -0.007069950923323631,
+            0.013105483725667,
+            0.003400330198928714,
+            0.016089053824543953,
+            0.03368271514773369,
+            -0.0033704089000821114,
+            0.0033511738292872906,
+            0.003849148051813245,
+            0.0058731031604111195,
+            0.0008874839986674488,
+            0.015909526497125626,
+            0.03621319308876991,
+            0.01942313089966774,
+            -0.0250141192227602,
+            0.02186811901628971,
+            -0.019713792949914932,
+            0.021936509758234024,
+            -0.02231266349554062,
+            -0.0030797459185123444,
+            0.007005834020674229,
+            -0.007394809741526842,
+            0.004146222956478596,
+            0.00624070642516017,
+            0.017627857625484467,
+            0.022363955155014992,
+            0.000617124664131552,
+            -0.0003163097717333585,
+            0.004860057029873133,
+            0.021953608840703964,
+            -0.0025411644019186497,
+            0.007125518750399351,
+            -0.013421793468296528,
+            0.004411239176988602,
+            -0.007202459033578634,
+            0.0021767669823020697,
+            0.008395032025873661,
+            -0.014952048659324646,
+            0.013344853185117245,
+            0.012891761027276516,
+            -0.017302999272942543,
+            0.0043342988938093185,
+            0.0025945950765162706,
+            -0.018568238243460655,
+            -0.018978586420416832,
+            0.023372728377580643,
+            -0.013635516166687012,
+            -0.012660940177738667,
+            -0.009891091845929623,
+            -0.01152393501251936,
+            -0.0201412383466959,
+            -0.005056682042777538,
+            -0.0017471840837970376,
+            0.005753418430685997,
+            -0.0024449890479445457,
+            0.006467252504080534,
+            0.01573854871094227,
+            0.005612361244857311,
+            -0.006437331438064575,
+            -0.005561067722737789,
+            -0.0015163634670898318,
+            0.020107043907046318,
+            0.005013937596231699,
+            0.0026458885986357927,
+            -0.04031667485833168,
+            0.0013143953401595354,
+            -0.010540809482336044,
+            0.006428782362490892,
+            0.0031417254358530045,
+            -0.004971192684024572,
+            -0.01942313089966774,
+            -0.014755423180758953,
+            0.01872211880981922,
+            -0.0103014400228858,
+            0.024637967348098755,
+            4.564986011246219e-05,
+            -0.012053967453539371,
+            -0.0018999958410859108,
+            0.003960283938795328,
+            0.0016029211692512035,
+            0.004415513481944799,
+            -0.00800178200006485,
+            0.016174543648958206,
+            -0.011694912798702717,
+            0.010908412747085094,
+            -0.00401157746091485,
+            -0.024706358090043068,
+            0.010891314595937729,
+            -0.01744833029806614,
+            0.0004752661334350705,
+            -0.00666387751698494,
+            -0.030040880665183067,
+            0.018551141023635864,
+            -0.023731783032417297,
+            -0.0033233899157494307,
+            0.009643173776566982,
+            0.018841803073883057,
+            0.0017461155075579882,
+            -0.010976804420351982,
+            0.00630909763276577,
+            0.017362842336297035,
+            0.013481635600328445,
+            -0.004062870983034372,
+            0.012438667938113213,
+            -0.004971192684024572,
+            0.0025603994727134705,
+            0.005172092467546463,
+            0.008399306796491146,
+            -0.006804934702813625,
+            0.007805157452821732
+          ],
+          "keyphrases": [
+            "DeepSeekMoE",
+            "multilingual corpus",
+            "HuggingFace Tokenizer",
+            "MoE architectures",
+            "evaluation benchmarks"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "95f27e84-0fb9-44ee-a783-45f42551401c",
+        "properties": {
+          "page_content": "5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n",
+          "title": "Scaling Up To Deepseekmoe 16B",
+          "keyphrases": [
+            "DeepSeekMoE 16B",
+            "MoE model",
+            "language modeling",
+            "evaluation benchmarks",
+            "Open LLM Leaderboard"
+          ],
+          "embedding": [
+            0.005968157202005386,
+            -0.015490485355257988,
+            -0.027613861486315727,
+            0.012274806387722492,
+            0.008261889219284058,
+            -0.01577553153038025,
+            -0.016158562153577805,
+            0.06238951161503792,
+            -0.002393943490460515,
+            0.048564765602350235,
+            0.008756265975534916,
+            -0.009914266876876354,
+            0.03532792627811432,
+            -0.018332039937376976,
+            -0.006921280641108751,
+            0.023961706086993217,
+            0.018617086112499237,
+            0.02068367227911949,
+            -0.00015922509192023426,
+            -0.02930632419884205,
+            -0.021538810804486275,
+            -0.03865940496325493,
+            -0.037447959184646606,
+            -0.023801365867257118,
+            -0.021129056811332703,
+            0.012862714938819408,
+            0.01521434634923935,
+            0.015909146517515182,
+            -0.041616760194301605,
+            0.008319789543747902,
+            0.016042763367295265,
+            0.021449735388159752,
+            0.004554060287773609,
+            0.009063581936061382,
+            0.012675653211772442,
+            -0.015294515527784824,
+            0.07097653299570084,
+            -0.0016189739108085632,
+            -0.009361989796161652,
+            0.010226036421954632,
+            0.018777426332235336,
+            0.010083513334393501,
+            -0.009317451156675816,
+            -0.010787221603095531,
+            0.019899794831871986,
+            0.012176821939647198,
+            -0.00043619878124445677,
+            -0.0028460093308240175,
+            0.00021837776876054704,
+            -0.021592257544398308,
+            -0.027720753103494644,
+            -0.013682222925126553,
+            0.0055272262543439865,
+            -0.006075049284845591,
+            -0.017797578126192093,
+            -0.005594033747911453,
+            0.03819620609283447,
+            0.0277029387652874,
+            0.03762611374258995,
+            -0.01240842230618,
+            0.03308318555355072,
+            -0.017494717612862587,
+            -0.006288834381848574,
+            -2.058165227936115e-05,
+            -0.006248749326914549,
+            -0.034597497433423996,
+            -0.005117471795529127,
+            0.001308317994698882,
+            0.005865718703716993,
+            -0.024068597704172134,
+            -0.04464537650346756,
+            0.02754260040819645,
+            -0.003171140095219016,
+            0.00048101565334945917,
+            0.02171696536242962,
+            0.0010444275103509426,
+            0.01798463985323906,
+            0.007918942719697952,
+            -0.02321345917880535,
+            -0.014279038645327091,
+            0.020505519583821297,
+            0.010386374779045582,
+            -0.006529341917484999,
+            0.025565091520547867,
+            0.06046544760465622,
+            -0.01011914387345314,
+            -0.024959366768598557,
+            -0.08444496989250183,
+            7.453237049048766e-05,
+            -0.0283442921936512,
+            -0.011490928940474987,
+            0.007825411856174469,
+            -0.004527336917817593,
+            -0.019062472507357597,
+            -0.0022169030271470547,
+            0.025547275319695473,
+            -0.005602941382676363,
+            -0.006342280190438032,
+            -0.017957916483283043,
+            0.03791115805506706,
+            -0.002776974579319358,
+            -0.004046321380883455,
+            0.030054569244384766,
+            0.0013561969390138984,
+            0.016888992860913277,
+            -0.0027858822140842676,
+            -0.02565416693687439,
+            0.029965493828058243,
+            -0.0067653958685696125,
+            0.08929075300693512,
+            -0.014564084820449352,
+            -0.009691574610769749,
+            -0.050453197211027145,
+            -0.011063359677791595,
+            -0.014083068817853928,
+            -0.003999555949121714,
+            -0.03424118831753731,
+            0.001817170181311667,
+            0.011588914319872856,
+            -0.014368114992976189,
+            0.013682222925126553,
+            0.021859489381313324,
+            0.03887319192290306,
+            -0.026990322396159172,
+            -0.04343393072485924,
+            0.06506182253360748,
+            0.021503180265426636,
+            -0.014804592356085777,
+            0.013824746012687683,
+            0.003099878551438451,
+            -0.01950785703957081,
+            0.011366221122443676,
+            -0.022393951192498207,
+            -0.013334822840988636,
+            0.017254209145903587,
+            -0.00778978131711483,
+            0.026152998208999634,
+            0.0797417014837265,
+            -0.018332039937376976,
+            0.014973838813602924,
+            0.022108905017375946,
+            -0.029217246919870377,
+            0.012372790835797787,
+            0.0028148323763161898,
+            -0.034989435225725174,
+            0.0058924416080117226,
+            0.008894335478544235,
+            0.05825633928179741,
+            0.003538582706823945,
+            0.010929744690656662,
+            0.010974282398819923,
+            0.011098990216851234,
+            -0.0009859707206487656,
+            0.023997336626052856,
+            0.008497943170368671,
+            0.0018372124759480357,
+            -0.026527121663093567,
+            0.01851019449532032,
+            -0.018813056871294975,
+            0.00869391206651926,
+            0.012167913839221,
+            -0.022661181166768074,
+            -0.012417329475283623,
+            -0.002204654971137643,
+            0.004934864118695259,
+            0.05184279754757881,
+            0.020024502649903297,
+            -0.010555621236562729,
+            -0.0979846715927124,
+            0.050132520496845245,
+            0.014145422726869583,
+            0.024923736229538918,
+            -0.04197306931018829,
+            0.017476901412010193,
+            0.009647035971283913,
+            0.028878754004836082,
+            0.0057677337899804115,
+            0.0020532242488116026,
+            -0.06816169619560242,
+            0.03545263409614563,
+            0.011419667862355709,
+            0.024994997307658195,
+            0.02485247515141964,
+            0.008840889669954777,
+            0.014065253548324108,
+            0.0004027949180454016,
+            0.04058346897363663,
+            0.029573554173111916,
+            -0.005389156751334667,
+            0.0357198640704155,
+            0.036236513406038284,
+            -0.0020799473859369755,
+            0.04428907111287117,
+            0.003500724909827113,
+            0.015089638531208038,
+            -0.006609511096030474,
+            -0.027328815311193466,
+            0.04963368922472,
+            -0.01895557902753353,
+            0.024157674983143806,
+            0.012069929391145706,
+            -0.03119475580751896,
+            -0.011125713586807251,
+            -0.01026166696101427,
+            0.028611524030566216,
+            -0.01782430149614811,
+            0.0019407644867897034,
+            0.02168133482336998,
+            0.009451067075133324,
+            -0.008163904771208763,
+            -0.026919061318039894,
+            -0.0066406880505383015,
+            -0.028967831283807755,
+            -0.04175928235054016,
+            0.0026010475121438503,
+            0.02702595293521881,
+            -0.012755822390317917,
+            -0.00048185072955675423,
+            -0.010742682963609695,
+            -0.030820632353425026,
+            -0.009932082146406174,
+            0.03388487920165062,
+            0.007901127450168133,
+            -0.011802698485553265,
+            -0.02971607819199562,
+            -0.014403746463358402,
+            0.01066251378506422,
+            -0.0017403413075953722,
+            0.05105891823768616,
+            -0.01533905416727066,
+            0.020648041740059853,
+            0.011437483131885529,
+            0.0065605188719928265,
+            0.02424675226211548,
+            0.02437146008014679,
+            -0.016559408977627754,
+            0.005843449383974075,
+            -0.003349294187501073,
+            -0.0055183181539177895,
+            0.043505191802978516,
+            -0.003075382439419627,
+            -0.029128169640898705,
+            -0.031586695462465286,
+            -0.012203545309603214,
+            0.010181497782468796,
+            -0.005081841256469488,
+            -0.01433248445391655,
+            0.02123595029115677,
+            0.02882530726492405,
+            0.04842223972082138,
+            0.01276472955942154,
+            -0.05725867673754692,
+            -0.013896007090806961,
+            0.0020732665434479713,
+            -0.02260773442685604,
+            0.032424017786979675,
+            0.060322925448417664,
+            -0.00506847957149148,
+            -0.020612411201000214,
+            -0.027150660753250122,
+            0.0659882202744484,
+            0.012328253127634525,
+            -0.04464537650346756,
+            0.04938427358865738,
+            -0.013655499555170536,
+            0.022340504452586174,
+            0.04517984017729759,
+            0.021859489381313324,
+            -0.0304286926984787,
+            0.017735224217176437,
+            0.024157674983143806,
+            -0.004836879670619965,
+            -0.06196194142103195,
+            -0.02260773442685604,
+            -0.002491928171366453,
+            -0.002779201604425907,
+            -0.013397176750004292,
+            -0.024585243314504623,
+            0.02485247515141964,
+            0.014537361450493336,
+            0.019098103046417236,
+            0.02120031975209713,
+            0.02324908971786499,
+            0.0164970550686121,
+            0.005259995348751545,
+            -0.007331034634262323,
+            -0.005153102800250053,
+            -0.023355981335043907,
+            -0.0269012451171875,
+            -0.02184167318046093,
+            -0.01982853375375271,
+            0.017574885860085487,
+            -0.04371897876262665,
+            0.016568316146731377,
+            0.0024251204449683428,
+            0.024389274418354034,
+            -0.023712290450930595,
+            0.004099767655134201,
+            0.028682785108685493,
+            -0.010591251775622368,
+            -0.04079725220799446,
+            0.025084074586629868,
+            0.011677990667521954,
+            -0.007241957820951939,
+            -0.007504734676331282,
+            0.0006981407641433179,
+            -0.008867612108588219,
+            0.031622324138879776,
+            -0.025280045345425606,
+            -0.01902684196829796,
+            -0.005375795066356659,
+            -0.02011357992887497,
+            0.0058924416080117226,
+            -0.03436589613556862,
+            -0.007464650087058544,
+            0.048600394278764725,
+            0.02132502757012844,
+            0.015098546631634235,
+            -0.04240063950419426,
+            -0.006631780415773392,
+            0.06142747774720192,
+            0.015392500907182693,
+            -0.07354194670915604,
+            0.044787902384996414,
+            0.0010861823102459311,
+            0.006698588375002146,
+            0.004685448482632637,
+            0.030054569244384766,
+            -0.015454854816198349,
+            -0.0159358698874712,
+            0.007673981133848429,
+            -0.030250540003180504,
+            -0.022091088816523552,
+            -0.005175372119992971,
+            0.021449735388159752,
+            -0.04521546885371208,
+            -0.03887319192290306,
+            -0.016791008412837982,
+            -0.0053802491165697575,
+            0.017129501327872276,
+            -0.011758159846067429,
+            -0.024923736229538918,
+            -0.030054569244384766,
+            -0.0005670305690728128,
+            -0.00188175099901855,
+            -0.04101103916764259,
+            -0.03067810833454132,
+            0.0068455650471150875,
+            0.021930750459432602,
+            0.02886093780398369,
+            0.0017392279114574194,
+            0.009459974244236946,
+            0.02504844404757023,
+            -0.00419552531093359,
+            0.029520107433199883,
+            -0.02766730636358261,
+            -0.032584354281425476,
+            0.012782545760273933,
+            -0.010911929421126842,
+            0.02116468921303749,
+            0.001672420185059309,
+            -0.021022165194153786,
+            -0.04689011722803116,
+            0.02706158347427845,
+            -0.0457855649292469,
+            -0.012087744660675526,
+            0.010974282398819923,
+            -0.0022937320172786713,
+            0.010288390330970287,
+            -0.00252533215098083,
+            0.0066451421007514,
+            0.02526222914457321,
+            -0.017601609230041504,
+            0.011909591034054756,
+            0.008012473583221436,
+            0.006306649651378393,
+            0.033991772681474686,
+            -0.008444496430456638,
+            -0.020505519583821297,
+            0.026242075487971306,
+            -0.03416992723941803,
+            0.01401180773973465,
+            0.0360405407845974,
+            0.013726761564612389,
+            0.010181497782468796,
+            0.011998668313026428,
+            0.011820513755083084,
+            0.0013862603809684515,
+            -0.02376573532819748,
+            0.01915154792368412,
+            0.01711168698966503,
+            -0.06866052746772766,
+            -0.020327365025877953,
+            -0.012702375650405884,
+            0.024068597704172134,
+            0.014412653632462025,
+            0.03340386599302292,
+            0.0033092093653976917,
+            -0.032459646463394165,
+            -0.025636352598667145,
+            -0.0075002810917794704,
+            0.04624876379966736,
+            -0.007437927182763815,
+            0.015962593257427216,
+            -0.00821289699524641,
+            0.021859489381313324,
+            0.012675653211772442,
+            -0.0016623989213258028,
+            0.026847798377275467,
+            -0.02597484551370144,
+            0.01240842230618,
+            -0.018688349053263664,
+            -0.04375460743904114,
+            -0.009771743789315224,
+            0.02360539697110653,
+            0.01886650174856186,
+            0.00418661767616868,
+            0.0078075965866446495,
+            -0.01634562388062477,
+            -0.0024986090138554573,
+            -0.03823183476924896,
+            0.010448728688061237,
+            0.027649492025375366,
+            0.0016223143320530653,
+            -0.005638572387397289,
+            -0.0038815289735794067,
+            0.0159358698874712,
+            0.0004142079269513488,
+            0.007469104137271643,
+            -0.034918174147605896,
+            0.040334053337574005,
+            0.0054158796556293964,
+            0.04582119360566139,
+            -0.04724642634391785,
+            -0.017450178042054176,
+            -0.0038904366083443165,
+            -0.007380026858299971,
+            0.042863838374614716,
+            -0.03573768213391304,
+            -0.02830866165459156,
+            -0.011464206501841545,
+            0.01947222650051117,
+            0.015668639913201332,
+            -0.02155662700533867,
+            -0.03976396098732948,
+            -0.002783655421808362,
+            -0.035274479538202286,
+            -0.010947559960186481,
+            -0.014412653632462025,
+            -0.0032802594359964132,
+            -0.049847472459077835,
+            -0.0017592702060937881,
+            -0.0835898295044899,
+            0.005081841256469488,
+            0.03755485266447067,
+            -0.009807374328374863,
+            0.04111792892217636,
+            -0.002641132101416588,
+            0.002984078601002693,
+            -0.047959040850400925,
+            0.01689790189266205,
+            0.026242075487971306,
+            0.04678322374820709,
+            -0.022180166095495224,
+            0.005242179613560438,
+            0.014679884538054466,
+            -0.011214790865778923,
+            -0.017610516399145126,
+            -0.010511082597076893,
+            0.02004231885075569,
+            -0.0051976414397358894,
+            0.0012392833596095443,
+            0.04528673365712166,
+            0.029537923634052277,
+            0.014697699807584286,
+            0.031943000853061676,
+            0.00030648044776171446,
+            -0.01814497821033001,
+            0.0025320129934698343,
+            0.043148886412382126,
+            0.04093977436423302,
+            -0.04375460743904114,
+            0.015543931163847446,
+            -0.03905134275555611,
+            0.025636352598667145,
+            0.013985084369778633,
+            0.007442380767315626,
+            0.03343949466943741,
+            0.006466988008469343,
+            -0.005803364794701338,
+            0.00020348520774859935,
+            0.024674320593476295,
+            -0.01662176288664341,
+            0.017922285944223404,
+            -0.004836879670619965,
+            -0.022661181166768074,
+            0.000857365841511637,
+            -0.015232161618769169,
+            0.017191855236887932,
+            0.02187730371952057,
+            -0.007188511546701193,
+            -0.0010806149803102016,
+            -0.0003092641127295792,
+            0.021948564797639847,
+            0.004395948722958565,
+            -0.0005812272429466248,
+            0.011428575031459332,
+            0.03500724956393242,
+            -0.03215678781270981,
+            -0.010644698515534401,
+            0.03212115541100502,
+            -0.033101003617048264,
+            0.009566866792738438,
+            -0.04104666784405708,
+            0.010609067045152187,
+            0.0022324915044009686,
+            -0.020986534655094147,
+            0.04076162353157997,
+            -0.010911929421126842,
+            0.001672420185059309,
+            -0.013317007571458817,
+            0.00856920424848795,
+            0.015561746433377266,
+            0.03762611374258995,
+            -0.0039015712682157755,
+            0.02882530726492405,
+            0.01489366963505745,
+            -0.020220471546053886,
+            0.04008463770151138,
+            -0.04268568381667137,
+            -0.010208221152424812,
+            0.015249977819621563,
+            -0.005981518421322107,
+            -0.012461868114769459,
+            -0.020576780661940575,
+            -0.021663518622517586,
+            -0.007001449819654226,
+            0.015143085271120071,
+            -0.02686561457812786,
+            0.006760942284017801,
+            0.006801026873290539,
+            -0.017842117697000504,
+            -0.007651711814105511,
+            0.006609511096030474,
+            -0.008016927167773247,
+            -0.0056296647526323795,
+            0.028201768174767494,
+            -0.017975732684135437,
+            0.01260439120233059,
+            0.02321345917880535,
+            0.028843123465776443,
+            -0.018162794411182404,
+            -0.0016189739108085632,
+            -0.0032490824814885855,
+            -0.0058479029685258865,
+            -0.03285158798098564,
+            0.026509307324886322,
+            0.009620312601327896,
+            -0.010056789964437485,
+            -0.024709951132535934,
+            -0.008836435154080391,
+            0.01879524067044258,
+            -0.0035185404121875763,
+            -0.005549495108425617,
+            -0.021984197199344635,
+            -0.02878967672586441,
+            0.0257788747549057,
+            0.01863490231335163,
+            -0.046711962670087814,
+            -0.0032112249173223972,
+            0.005411426071077585,
+            0.004783433396369219,
+            0.0027213015127927065,
+            -0.014849130995571613,
+            0.007388934958726168,
+            0.026972506195306778,
+            0.0054826876148581505,
+            -0.015543931163847446,
+            -0.012622206471860409,
+            -0.005099656525999308,
+            0.008613742887973785,
+            0.0069925421848893166,
+            0.009932082146406174,
+            0.02975170873105526,
+            0.04072599112987518,
+            0.0018628222169354558,
+            -0.01288943737745285,
+            -0.02561853639781475,
+            0.025315675884485245,
+            -0.0019630338065326214,
+            0.04165239259600639,
+            0.04564303904771805,
+            0.0025876860599964857,
+            -0.060572341084480286,
+            0.012479683384299278,
+            -0.02517315186560154,
+            -0.009843004867434502,
+            0.0066406880505383015,
+            0.0036209789104759693,
+            -0.04660507291555405,
+            -0.0031956364400684834,
+            0.0022536474280059338,
+            0.034152109175920486,
+            -0.007299857679754496,
+            -0.04436033219099045,
+            -0.008929966017603874,
+            -0.007406750228255987,
+            0.02574324421584606,
+            0.010110235773026943,
+            -0.007892219349741936,
+            -0.015294515527784824,
+            0.013361545279622078,
+            -0.0026567205786705017,
+            -0.006190849468111992,
+            0.017013700678944588,
+            -0.007691796403378248,
+            0.011170252226293087,
+            -0.012096652761101723,
+            -0.019739456474781036,
+            -0.017655055969953537,
+            0.03393832594156265,
+            -0.02706158347427845,
+            0.025565091520547867,
+            0.014947115443646908,
+            0.0026923513505607843,
+            0.039407651871442795,
+            -0.01535686943680048,
+            -0.027364445850253105,
+            -0.002897228579968214,
+            -0.0037078289315104485,
+            -0.03766174241900444,
+            0.01815388724207878,
+            -0.023516319692134857,
+            -0.022091088816523552,
+            0.013495161198079586,
+            0.00951342098414898,
+            0.0002751643187366426,
+            -0.009308543056249619,
+            -0.00851130485534668,
+            -0.0015465988544747233,
+            0.01956130377948284,
+            0.019258441403508186,
+            -0.007451288867741823,
+            0.043505191802978516,
+            -0.0004167132137808949,
+            0.0038369905669242144,
+            0.0005995993851684034,
+            0.018100440502166748,
+            0.060287293046712875,
+            0.020452072843909264,
+            0.0012481911107897758,
+            0.030482139438390732,
+            0.022500842809677124,
+            0.037162911146879196,
+            -0.03019709326326847,
+            -0.04564303904771805,
+            0.017895564436912537,
+            -0.007250865455716848,
+            0.009522328153252602,
+            0.022429581731557846,
+            0.0011813833843916655,
+            0.01418996136635542,
+            0.007063803728669882,
+            -0.010243851691484451,
+            0.023801365867257118,
+            -0.032905034720897675,
+            -0.02789890766143799,
+            0.027115030214190483,
+            0.010056789964437485,
+            0.019971057772636414,
+            0.0360405407845974,
+            -0.024870289489626884,
+            -0.033671095967292786,
+            0.004881417844444513,
+            0.006680772639811039,
+            -0.0283442921936512,
+            0.012942884117364883,
+            -0.013521884568035603,
+            0.015160900540649891,
+            0.0064001805149018764,
+            -0.012648929841816425,
+            -0.0027279823552817106,
+            0.013780207373201847,
+            -0.0141721460968256,
+            0.008698366582393646,
+            -0.015160900540649891,
+            -0.012452960945665836,
+            0.010626882314682007,
+            -0.030642477795481682,
+            0.0008445610292255878,
+            0.00507738720625639,
+            -0.0015131949912756681,
+            0.006230934057384729,
+            -0.0067297653295099735,
+            -0.004431579262018204,
+            -0.005865718703716993,
+            -0.03185392543673515,
+            0.02795235440135002,
+            0.02766730636358261,
+            -0.025119705125689507,
+            -0.0030553401447832584,
+            -0.0036900134291499853,
+            -0.014528454281389713,
+            -0.006346734240651131,
+            0.010876297950744629,
+            -0.014448285102844238,
+            0.0021411876659840345,
+            -0.016087301075458527,
+            -0.018367672339081764,
+            0.023961706086993217,
+            0.022518659010529518,
+            0.042863838374614716,
+            -0.0010037861065939069,
+            0.022750258445739746,
+            0.021307211369276047,
+            0.0011028843000531197,
+            0.0017982413992285728,
+            0.017494717612862587,
+            -0.017557071521878242,
+            0.028985645622015,
+            0.025707613676786423,
+            0.03099878504872322,
+            0.010849575512111187,
+            -0.03221023455262184,
+            -0.0024674320593476295,
+            0.018848687410354614,
+            0.017076054587960243,
+            0.0008317562169395387,
+            -0.013486253097653389,
+            -0.009771743789315224,
+            0.012533130124211311,
+            -0.02152099646627903,
+            -0.0065248883329331875,
+            0.013512976467609406,
+            -0.016514871269464493,
+            0.0032802594359964132,
+            -0.007825411856174469,
+            0.02472776733338833,
+            0.0028838668949902058,
+            -0.005193187389522791,
+            0.012773637659847736,
+            -0.00048185072955675423,
+            0.005237726029008627,
+            -0.005037302616983652,
+            0.02324908971786499,
+            0.03493598848581314,
+            0.012684560380876064,
+            0.04407528415322304,
+            0.01998887211084366,
+            -0.013423899188637733,
+            -0.009531236253678799,
+            -0.020612411201000214,
+            0.022358320653438568,
+            -0.030286170542240143,
+            -0.024709951132535934,
+            -0.022073272615671158,
+            0.01535686943680048,
+            0.00679657282307744,
+            0.015606285072863102,
+            -0.0033893787767738104,
+            -0.004447167739272118,
+            0.031372908502817154,
+            0.004179936833679676,
+            0.025511644780635834,
+            0.006907918956130743,
+            -0.007905581034719944,
+            -0.004012917634099722,
+            0.0024139860179275274,
+            0.008150543086230755,
+            -0.015241069719195366,
+            0.03826746717095375,
+            -0.0035630788188427687,
+            0.04813719540834427,
+            -0.052733566612005234,
+            -0.002003118395805359,
+            0.00635564187541604,
+            -0.01918718032538891,
+            -0.005669749341905117,
+            -0.03915823623538017,
+            0.02030954882502556,
+            -0.03377798572182655,
+            0.026669645681977272,
+            -0.02027391828596592,
+            0.026491491124033928,
+            -0.029929861426353455,
+            0.003576440503820777,
+            -0.0015243296511471272,
+            -0.0038035865873098373,
+            0.0164970550686121,
+            0.024282382801175117,
+            -0.002409531967714429,
+            -0.02590358257293701,
+            -0.0008083735010586679,
+            0.0037635019980371,
+            0.0066273268312215805,
+            -0.013717853464186192,
+            0.0054158796556293964,
+            -0.0012582122581079602,
+            -0.029680447652935982,
+            -0.033101003617048264,
+            0.012132283300161362,
+            -0.010911929421126842,
+            0.0337958037853241,
+            0.030375247821211815,
+            -0.026794353500008583,
+            -0.027043769136071205,
+            0.0034561865031719208,
+            -0.008676096796989441,
+            -0.009090305306017399,
+            0.020737119019031525,
+            0.04532236233353615,
+            0.00925509724766016,
+            -0.020790565758943558,
+            0.023943889886140823,
+            0.004734441172331572,
+            0.008582565933465958,
+            0.004422671627253294,
+            -0.021734781563282013,
+            -0.018492378294467926,
+            -0.01689790189266205,
+            -0.01726311631500721,
+            0.008177265524864197,
+            -0.01972164213657379,
+            -0.04012026637792587,
+            -0.04742458090186119,
+            -0.011179160326719284,
+            0.019383149221539497,
+            0.004144306294620037,
+            0.05052445828914642,
+            0.012452960945665836,
+            0.02478121407330036,
+            -0.0017191855004057288,
+            -0.005469325929880142,
+            0.04646254703402519,
+            -0.019899794831871986,
+            0.051700275391340256,
+            0.009700481779873371,
+            -0.0021634569857269526,
+            -0.01578443869948387,
+            0.03188955411314964,
+            -0.024905921891331673,
+            -0.0019240626133978367,
+            0.006342280190438032,
+            -0.0034984981175512075,
+            0.017298748716711998,
+            0.008644919842481613,
+            0.010128051973879337,
+            -0.02360539697110653,
+            0.028130507096648216,
+            -0.023641027510166168,
+            0.014590808190405369,
+            0.015926962718367577,
+            -0.002009799238294363,
+            0.0036098442506045103,
+            -0.0054826876148581505,
+            0.004574102349579334,
+            -0.002970716916024685,
+            0.011597821488976479,
+            -0.02510189078748226,
+            0.015009469352662563,
+            -0.02228705771267414,
+            -0.013317007571458817,
+            0.0004178266681265086,
+            -0.030945340171456337,
+            0.012399514205753803,
+            0.04108230024576187,
+            0.01037746760994196,
+            -0.02232268825173378,
+            0.005371341481804848,
+            -0.051450859755277634,
+            -0.008199535310268402,
+            0.0014809046406298876,
+            0.01271128375083208,
+            0.01346843782812357,
+            0.0007504734676331282,
+            0.009727205149829388,
+            -0.00955795869231224,
+            -0.03055340051651001,
+            0.033421680331230164,
+            -0.001038860180415213,
+            -0.0020298415329307318,
+            0.003928294405341148,
+            0.01851019449532032,
+            0.0025854590348899364,
+            0.004188844468444586,
+            0.012773637659847736,
+            -0.007709611672908068,
+            -0.03224586322903633,
+            -0.021449735388159752,
+            -0.0038926636334508657,
+            0.0019318568520247936,
+            -0.003694467479363084,
+            0.003318117232993245,
+            0.0055851261131465435,
+            -0.015535023994743824,
+            0.01082285214215517,
+            -0.011980853043496609,
+            0.010707051493227482,
+            -0.019169364124536514,
+            -0.004329140763729811,
+            0.006658503785729408,
+            -0.020808380097150803,
+            -0.018296409398317337,
+            -0.051450859755277634,
+            -0.010466543957591057,
+            0.002101103076711297,
+            0.021146873012185097,
+            0.006676319055259228,
+            0.008489035069942474,
+            -0.014394838362932205,
+            0.0068322038277983665,
+            0.003656609682366252,
+            0.02520878240466118,
+            -0.026829984039068222,
+            0.03119475580751896,
+            0.018723979592323303,
+            0.016951346769928932,
+            -0.03862377628684044,
+            0.03249527886509895,
+            0.016443608328700066,
+            -0.003759048180654645,
+            0.017521440982818604,
+            0.014002899639308453,
+            0.04753147065639496,
+            -0.030606847256422043,
+            0.004810156300663948,
+            -0.02754260040819645,
+            0.025084074586629868,
+            -0.03805368393659592,
+            0.028237400576472282,
+            0.0009163793292827904,
+            0.023943889886140823,
+            0.04428907111287117,
+            9.728875738801435e-05,
+            0.03851688280701637,
+            -0.022019827738404274,
+            -0.009045766666531563,
+            0.025636352598667145,
+            -0.017476901412010193,
+            0.015882423147559166,
+            0.00778978131711483,
+            -0.024282382801175117,
+            0.01838548667728901,
+            0.007709611672908068,
+            -0.0031956364400684834,
+            -0.04788777977228165,
+            0.02795235440135002,
+            0.02260773442685604,
+            0.02120031975209713,
+            0.0067297653295099735,
+            -0.012150098569691181,
+            -0.010234943591058254,
+            0.005611849017441273,
+            -0.009246189147233963,
+            0.016755377873778343,
+            -0.015846792608499527,
+            -0.030909709632396698,
+            0.00849348958581686,
+            -0.01927625574171543,
+            0.012648929841816425,
+            -0.0038904366083443165,
+            0.029092539101839066,
+            -0.017770856618881226,
+            -0.01068032905459404,
+            0.01101882103830576,
+            -0.0028237400110810995,
+            -0.016069484874606133,
+            -0.04842223972082138,
+            -0.038801927119493484,
+            0.008453404530882835,
+            -0.0013105449033901095,
+            -0.020184841006994247,
+            0.011909591034054756,
+            -0.010591251775622368,
+            -0.004792341031134129,
+            -0.01972164213657379,
+            -0.010742682963609695,
+            -0.023035304620862007,
+            -0.006760942284017801,
+            0.010074605233967304,
+            0.026277706027030945,
+            -0.018848687410354614,
+            0.03798241913318634,
+            0.029413215816020966,
+            0.013985084369778633,
+            -0.012506406754255295,
+            0.0257788747549057,
+            -0.009353081695735455,
+            -0.01489366963505745,
+            0.033671095967292786,
+            -0.0031466439832001925,
+            -0.009192743338644505,
+            0.00017439600196667016,
+            -0.024799028411507607,
+            -0.023070935159921646,
+            0.019579118117690086,
+            -0.022393951192498207,
+            -0.025280045345425606,
+            -0.004921502433717251,
+            -0.006141857244074345,
+            -0.02975170873105526,
+            -0.04753147065639496,
+            -0.011214790865778923,
+            0.03645029664039612,
+            0.0057632802054286,
+            0.018492378294467926,
+            0.01711168698966503,
+            -0.012230267748236656,
+            -0.03039306215941906,
+            0.004721079487353563,
+            -0.012105559930205345,
+            -0.008738450706005096,
+            -0.01287162210792303,
+            0.007700704038143158,
+            0.016951346769928932,
+            0.01101882103830576,
+            0.0011669083032757044,
+            -0.0052778106182813644,
+            -0.02123595029115677,
+            -0.007861042395234108,
+            -0.0176995936781168,
+            -0.028985645622015,
+            -0.0031466439832001925,
+            -0.033029742538928986,
+            0.03823183476924896,
+            0.018777426332235336,
+            0.005273356568068266,
+            -0.01982853375375271,
+            -0.012720191851258278,
+            -0.020790565758943558,
+            0.00927291251718998,
+            -0.00707716541364789,
+            0.009718297980725765,
+            -0.004614186938852072,
+            0.01726311631500721,
+            -0.006057234015315771,
+            0.00047684015589766204,
+            -0.04436033219099045,
+            0.009023496881127357,
+            0.01934751868247986,
+            0.003930521197617054,
+            -0.009878636337816715,
+            0.008880973793566227,
+            0.011606729589402676,
+            -0.021022165194153786,
+            -0.0236944742500782,
+            0.019222810864448547,
+            0.020576780661940575,
+            0.015909146517515182,
+            -0.009299635887145996,
+            -0.009299635887145996,
+            0.007130611687898636,
+            0.0007682888535782695,
+            -0.006315557286143303,
+            0.016550501808524132,
+            0.023623213171958923,
+            0.01375348400324583,
+            -0.00018177268793806434,
+            0.026206444948911667,
+            0.009174928069114685,
+            -0.012310437858104706,
+            -0.012114468030631542,
+            -0.007620534859597683,
+            0.0023694473784416914,
+            -0.01670193299651146,
+            -0.011989760212600231,
+            -0.010083513334393501,
+            0.025226598605513573,
+            0.009370896965265274,
+            0.011294960044324398,
+            -0.003090970916673541,
+            -0.006173034198582172,
+            0.005411426071077585,
+            0.009949897415935993,
+            0.0001479512866353616,
+            -0.001145752496086061,
+            0.007219688501209021,
+            -0.038766298443078995,
+            -0.028682785108685493,
+            0.0014196642441675067,
+            -0.002750251442193985,
+            0.0033515209797769785,
+            0.03506069630384445,
+            0.027596045285463333,
+            -0.007014811504632235,
+            -0.01918718032538891,
+            -0.0018227375112473965,
+            0.024282382801175117,
+            0.0001636789384065196,
+            -0.02308875136077404,
+            -0.009477789513766766,
+            0.009090305306017399,
+            0.00032874970929697156,
+            0.02927069365978241,
+            0.005700926296412945,
+            -0.008235165849328041,
+            0.019383149221539497,
+            0.10076387226581573,
+            0.007776419632136822,
+            0.006885650102049112,
+            -0.012150098569691181,
+            -0.031016601249575615,
+            0.0021033298689872026,
+            0.021699151024222374,
+            -0.019080286845564842,
+            0.04485916346311569,
+            0.011553282849490643,
+            0.013263560831546783,
+            0.018973395228385925,
+            -0.0021144645288586617,
+            -0.010974282398819923,
+            -0.016131838783621788,
+            0.030606847256422043,
+            0.0016323354793712497,
+            0.02184167318046093,
+            -0.020897457376122475,
+            0.0017414548201486468,
+            0.004591918084770441,
+            0.012800361029803753,
+            0.00606168806552887,
+            -0.005055117886513472,
+            -0.02510189078748226,
+            0.018777426332235336,
+            -0.018180610612034798,
+            0.007224142551422119,
+            -0.008858704939484596,
+            -0.022874966263771057,
+            0.008987866342067719,
+            -0.00014502844715025276,
+            -0.01244405284523964,
+            0.005513864569365978,
+            -0.014109792187809944,
+            0.0326734334230423,
+            -0.012738007120788097,
+            0.0015354643110185862,
+            0.003663290524855256,
+            -0.016390163451433182,
+            0.01202539075165987,
+            0.01746799424290657,
+            -0.01200757548213005,
+            0.030090201646089554,
+            -0.016710840165615082,
+            -0.004318006336688995,
+            0.012061022222042084,
+            -0.018002456054091454,
+            0.011214790865778923,
+            0.028486816212534904,
+            -0.009335266426205635,
+            0.005683110561221838,
+            -0.0075581809505820274,
+            0.009905358776450157,
+            0.009335266426205635,
+            0.026402413845062256,
+            0.023142196238040924,
+            -0.0005219354061409831,
+            0.004251198377460241,
+            -0.008880973793566227,
+            -0.0019708280451595783,
+            -0.0009464428294450045,
+            0.034312449395656586,
+            -0.018162794411182404,
+            -0.0043246871791779995,
+            -0.009326359257102013,
+            0.0006680772639811039,
+            -0.014724423177540302,
+            -0.002120031975209713,
+            -0.007437927182763815,
+            -0.0019351971568539739,
+            0.0076606194488704205,
+            -0.00036493720836006105,
+            0.016158562153577805,
+            -0.008257435634732246,
+            -0.004556287080049515,
+            0.004997218027710915,
+            -0.005224364344030619,
+            0.01657722517848015,
+            -0.02574324421584606,
+            0.0037612752057611942,
+            0.018456747755408287,
+            0.00869391206651926,
+            0.011508745141327381,
+            -0.01642579399049282,
+            0.0025854590348899364,
+            -0.002992986235767603,
+            -0.012577668763697147,
+            0.012577668763697147,
+            -0.01831422559916973,
+            -0.011499837040901184,
+            -0.021129056811332703,
+            -0.0297873392701149,
+            -0.002863824600353837,
+            -0.01851019449532032,
+            0.008146088570356369,
+            0.01918718032538891,
+            0.028647154569625854,
+            0.00018024168093688786,
+            -0.004970495123416185,
+            -0.01743236370384693,
+            0.015962593257427216,
+            0.045037318021059036,
+            -0.016292177140712738,
+            -0.02520878240466118,
+            -0.009914266876876354,
+            0.0127290990203619,
+            -0.008622650988399982,
+            -0.026046106591820717,
+            -0.003943882882595062,
+            0.00577218784019351,
+            -0.02911035344004631,
+            0.009379805065691471,
+            0.0005712060956284404,
+            0.024442721158266068,
+            0.004672087263315916,
+            0.020665857940912247,
+            -0.017521440982818604,
+            -0.0019708280451595783,
+            0.025476014241576195,
+            -0.029520107433199883,
+            0.0045874640345573425,
+            0.01802917942404747,
+            -0.01346843782812357,
+            0.010769405402243137,
+            0.030820632353425026,
+            0.025226598605513573,
+            -0.01963256485760212,
+            0.024335827678442,
+            0.014626438729465008,
+            -0.008720635436475277,
+            0.003284713253378868,
+            0.011660175397992134,
+            0.012622206471860409,
+            0.006698588375002146,
+            -0.0034584132954478264,
+            -0.005861264653503895,
+            0.009045766666531563,
+            0.031337279826402664,
+            0.0016623989213258028,
+            -0.012657837942242622,
+            0.028415553271770477,
+            0.02497718296945095,
+            -0.005429241340607405,
+            0.008426681160926819,
+            -0.02039862610399723,
+            -0.016443608328700066,
+            0.01730765588581562,
+            0.022465212270617485,
+            0.005910256877541542,
+            0.0032401748467236757,
+            -0.011366221122443676,
+            0.022839335724711418,
+            -0.008377688936889172,
+            0.01154437568038702,
+            -0.027168476954102516,
+            -0.002518651308491826,
+            -0.0036900134291499853,
+            0.02068367227911949,
+            -0.0008106004097498953,
+            -0.005032849032431841,
+            0.019062472507357597,
+            -0.01228371448814869,
+            -0.019775087013840675,
+            -0.006462534423917532,
+            -0.018118256703019142,
+            0.017637239769101143,
+            -0.004774525761604309,
+            -0.02030954882502556,
+            0.007549273315817118,
+            0.0005678656743839383,
+            -0.009041312150657177,
+            0.006801026873290539,
+            0.004144306294620037,
+            0.008057012222707272,
+            -0.02264336682856083,
+            0.010751590132713318,
+            -0.005464872345328331,
+            0.007901127450168133,
+            -0.008382143452763557,
+            0.014368114992976189,
+            -0.006738672964274883,
+            0.0015599604230374098,
+            0.0006012695375829935,
+            -0.006208664737641811,
+            0.000636900367680937,
+            0.013210115022957325,
+            0.0032624441664665937,
+            -0.016550501808524132,
+            0.015312331728637218,
+            -0.008662735112011433,
+            -0.004925956483930349,
+            -0.005028394982218742,
+            -0.004106448497623205,
+            -0.0011357313487678766,
+            0.012738007120788097,
+            0.00865382794290781,
+            -0.031123492866754532,
+            -0.006689680740237236,
+            -0.024959366768598557,
+            -0.009326359257102013,
+            0.010145867243409157,
+            0.012052114121615887,
+            -0.020576780661940575,
+            0.009001228027045727,
+            -0.027079399675130844,
+            -0.01226589921861887,
+            0.0269012451171875,
+            0.023961706086993217,
+            -0.02068367227911949,
+            0.012738007120788097,
+            -0.014804592356085777,
+            -0.02014921046793461,
+            0.015009469352662563,
+            0.021467549726366997,
+            0.01224808394908905,
+            0.005028394982218742,
+            -0.04400402307510376,
+            -0.010920836590230465,
+            0.019864164292812347,
+            -0.03140854090452194,
+            -0.009504512883722782,
+            0.027417892590165138,
+            -0.006827749777585268,
+            -0.01726311631500721,
+            -0.003516313387081027,
+            0.03522103279829025,
+            -0.02123595029115677,
+            0.015526115894317627,
+            0.011428575031459332,
+            -0.02152099646627903,
+            0.014760053716599941,
+            -0.012853806838393211,
+            0.008618197403848171,
+            0.0019296299433335662,
+            -0.022536473348736763,
+            -0.0006502618780359626,
+            0.018011363223195076,
+            -0.0005444829585030675,
+            -0.004050775431096554,
+            0.011455298401415348,
+            0.016639579087495804,
+            -0.018421117216348648,
+            0.013922730460762978,
+            0.006088410969823599,
+            9.90285407169722e-05,
+            0.016283269971609116,
+            -0.02052333392202854,
+            -0.01810934767127037,
+            0.01979290321469307,
+            -0.004179936833679676,
+            -0.02421111986041069,
+            0.02344505861401558,
+            -0.02376573532819748,
+            -0.020950904116034508,
+            -0.02340942807495594,
+            0.00032652277150191367,
+            0.00849348958581686,
+            -0.01734328642487526,
+            -0.026562752202153206,
+            0.007233050186187029,
+            0.01950785703957081,
+            0.04689011722803116,
+            0.018038086593151093,
+            -0.0023048666771501303,
+            0.023819182068109512,
+            -0.016595039516687393,
+            0.013263560831546783,
+            0.012105559930205345,
+            -0.0036543826572597027,
+            -0.015757715329527855,
+            0.0025743243750184774,
+            0.009602497331798077,
+            0.007152881007641554,
+            0.0030263899825513363,
+            -0.022251427173614502,
+            -0.02120031975209713,
+            -0.04101103916764259,
+            0.00957577396184206,
+            -0.007771965581923723,
+            0.0008968937327153981,
+            0.05790003016591072,
+            0.00126711989287287,
+            -0.033350419253110886,
+            0.01754816249012947,
+            0.00047767526120878756,
+            0.0007610513712279499,
+            -0.011633452028036118,
+            0.002986305393278599,
+            -0.02389044314622879,
+            -0.013949453830718994,
+            -0.0353991873562336,
+            -0.0036877866368740797,
+            0.01027948223054409,
+            0.005162010435014963,
+            0.005700926296412945,
+            -0.02805924601852894,
+            -0.002245853189378977,
+            -0.025867952033877373,
+            0.022304873913526535,
+            0.006475895643234253,
+            -0.0001588075392646715,
+            0.0023649935610592365,
+            0.027400076389312744,
+            -0.008622650988399982,
+            -0.01854582503437996,
+            -0.020790565758943558,
+            0.0011902910191565752,
+            -0.004248971585184336,
+            0.001448614289984107,
+            0.0009458860731683671,
+            -0.025832321494817734,
+            0.009486697614192963,
+            0.00028532466967590153,
+            -0.012595484033226967,
+            0.014100884087383747,
+            -0.013352638110518456,
+            -0.007001449819654226,
+            0.009157112799584866,
+            0.023160012438893318,
+            -0.027524784207344055,
+            -0.008506850339472294,
+            -0.024870289489626884,
+            0.009985528886318207,
+            0.012675653211772442,
+            0.0029484478291124105,
+            -0.019775087013840675,
+            -0.012925068847835064,
+            -0.02574324421584606,
+            -0.009851912967860699,
+            -0.02059459500014782,
+            -0.008012473583221436,
+            0.005041756667196751,
+            0.01657722517848015,
+            0.03244183212518692,
+            0.016799917444586754,
+            -0.005157556850463152,
+            0.0017548163887113333,
+            -0.002420666627585888,
+            0.008751812390983105,
+            -0.020060133188962936,
+            2.0077113731531426e-05,
+            -0.0159358698874712,
+            0.017957916483283043,
+            0.00999443605542183,
+            -0.013887099921703339,
+            -0.017761947587132454,
+            -0.02276807464659214,
+            0.019525671377778053,
+            -0.022429581731557846,
+            -0.017886655405163765,
+            0.01918718032538891,
+            -0.011036636307835579,
+            0.006409088149666786,
+            -0.013691130094230175,
+            0.037198543548583984,
+            0.010021159425377846,
+            0.025476014241576195,
+            -0.016612855717539787,
+            -0.005567310843616724,
+            -0.0272219218313694,
+            -0.004400402307510376,
+            -0.01682663895189762,
+            -0.00434695603325963,
+            -0.0012337160296738148,
+            0.009753928519785404,
+            -0.02526222914457321,
+            -0.005567310843616724,
+            0.02321345917880535,
+            -0.027560414746403694,
+            -0.0022124492097646,
+            -0.0021757050417363644,
+            0.006337826605886221,
+            0.007121704053133726,
+            0.0014397065388038754,
+            -0.002219130052253604,
+            -0.022180166095495224,
+            -0.008448950946331024,
+            -0.028807492926716805,
+            0.022091088816523552,
+            -0.010315113700926304,
+            -0.014653162099421024,
+            0.0001746743801049888,
+            0.021182503551244736,
+            0.0046899025328457355,
+            -0.010172589682042599,
+            0.0017403413075953722,
+            -0.007598265539854765,
+            0.0021645703818649054,
+            0.0038147212471812963,
+            0.012096652761101723,
+            -0.0020988760516047478,
+            0.014083068817853928,
+            0.02020265720784664,
+            -0.008560297079384327,
+            -0.0006975840078666806,
+            -0.0058122724294662476,
+            0.0037790904752910137,
+            0.011375129222869873,
+            0.01037746760994196,
+            0.016203101724386215,
+            0.003641021205112338,
+            -0.009486697614192963,
+            0.01670193299651146,
+            -0.014083068817853928,
+            0.02184167318046093,
+            -0.001927402918227017,
+            -0.01943659596145153,
+            0.002552055288106203,
+            -0.047923412173986435,
+            0.0009069149382412434,
+            -0.012310437858104706,
+            -0.022750258445739746,
+            0.006293287966400385,
+            -0.011838329024612904,
+            -0.008190627209842205,
+            0.019098103046417236,
+            -0.009103666059672832,
+            -0.02219798043370247,
+            0.011571098119020462,
+            0.0027591590769588947,
+            -0.009036858566105366,
+            0.006613965146243572,
+            -0.0016679662512615323,
+            0.008658281527459621,
+            0.004482798743993044,
+            0.005366887431591749,
+            0.001175816054455936,
+            -0.01565973088145256,
+            -0.018563641235232353,
+            0.020095765590667725,
+            0.008333150297403336,
+            -0.0016278816619887948,
+            0.010003344155848026,
+            0.002652266761288047,
+            0.011428575031459332,
+            -0.003694467479363084,
+            -0.004810156300663948,
+            0.00019415996212046593,
+            -0.014679884538054466,
+            0.014145422726869583,
+            -0.013085407204926014,
+            -0.032228048890829086,
+            0.013664407655596733,
+            0.006008241791278124,
+            0.00681438809260726,
+            -0.008119366131722927,
+            -0.011891775764524937,
+            -0.007161788642406464,
+            -0.012078837491571903,
+            0.007455742452293634,
+            0.015294515527784824,
+            -0.0038147212471812963,
+            -0.0008879860397428274,
+            -0.01418996136635542,
+            0.007575996220111847,
+            0.020184841006994247,
+            0.029965493828058243,
+            -0.02488810569047928,
+            0.00724641140550375,
+            -0.005344618111848831,
+            -0.011873960494995117,
+            -0.020291734486818314,
+            0.024264566600322723,
+            0.01457299292087555,
+            0.01626545563340187,
+            0.004206659737974405,
+            0.0036165250930935144,
+            0.000802249473053962,
+            -0.019365333020687103,
+            0.003191182389855385,
+            0.008137181401252747,
+            -0.013317007571458817,
+            -0.0020688127260655165,
+            0.02478121407330036,
+            -0.013780207373201847,
+            -0.0013484027003869414,
+            0.001965260598808527,
+            0.024032967165112495,
+            -0.0138514693826437,
+            0.001873956760391593,
+            0.004729987122118473,
+            0.0006508186343125999,
+            0.005522772204130888,
+            0.013219022192060947,
+            0.07496717572212219,
+            0.019579118117690086,
+            0.005393610801547766,
+            0.007446834817528725,
+            -0.006311103235930204,
+            -0.006008241791278124,
+            -0.009264005348086357,
+            -0.011553282849490643,
+            0.002538693603128195,
+            -0.0017325470689684153,
+            -0.02308875136077404,
+            -0.002113351132720709,
+            -0.005825634114444256,
+            0.00045429254532791674,
+            -0.0012982968473806977,
+            0.01734328642487526,
+            -0.012666745111346245,
+            0.013913823291659355,
+            0.03204989433288574,
+            0.007188511546701193,
+            -0.021770412102341652,
+            0.013504069298505783,
+            0.0025854590348899364,
+            -0.006484803277999163,
+            -0.003402740228921175,
+            -0.006609511096030474,
+            0.006996996235102415,
+            0.011669083498418331,
+            -0.010600159876048565,
+            0.03851688280701637,
+            0.0013951680157333612,
+            0.0014764508232474327,
+            -0.01298742275685072,
+            -0.008008019998669624,
+            -0.0038035865873098373,
+            0.0024852475617080927,
+            0.012354975566267967,
+            -0.0010978736681863666,
+            -0.01851019449532032,
+            0.003068701596930623,
+            0.004166575148701668,
+            0.021182503551244736,
+            0.0044182175770401955,
+            0.002165683778002858,
+            -0.009246189147233963,
+            0.01069814432412386,
+            -0.00923728197813034,
+            -0.004021825268864632,
+            0.009860821068286896,
+            -0.0033760173246264458,
+            -0.010457636788487434,
+            -0.016452517360448837,
+            -0.0003234607574995607,
+            0.008742904290556908,
+            -0.013691130094230175,
+            -0.014742238447070122,
+            0.022429581731557846,
+            -0.0014385931426659226,
+            0.023035304620862007,
+            -0.005406972020864487,
+            0.008622650988399982,
+            0.011473113670945168,
+            -0.0012159006437286735,
+            -0.00782986544072628,
+            0.019703825935721397,
+            -0.00027196313021704555,
+            0.004277921747416258,
+            -0.014679884538054466,
+            0.009531236253678799,
+            -0.01114352885633707,
+            -0.0023605397436767817,
+            0.016363440081477165,
+            0.0001629830221645534,
+            0.007037080824375153,
+            -0.002538693603128195,
+            -0.009032404981553555,
+            -0.031782664358615875,
+            -0.02353413589298725,
+            -0.0022057686001062393,
+            0.013094314374029636,
+            -0.008578112348914146,
+            0.0022592146415263414,
+            -0.0027480246499180794,
+            -0.004053002223372459,
+            -0.016194192692637444,
+            0.006507072597742081,
+            5.518596663023345e-05,
+            0.015971500426530838,
+            0.0011435255873948336,
+            -0.015276700258255005,
+            0.003774636657908559,
+            -0.0031399631407111883,
+            0.018937764689326286,
+            -0.013922730460762978,
+            -0.0029573554638773203,
+            0.0236944742500782,
+            -0.01618528552353382,
+            -0.0011691352119669318,
+            0.003202317049726844,
+            -0.013682222925126553,
+            0.031016601249575615,
+            0.013094314374029636,
+            0.023943889886140823,
+            0.015401408076286316,
+            0.01212337613105774,
+            0.001612293184734881,
+            0.01630108617246151,
+            0.013619869016110897,
+            0.012925068847835064,
+            -0.015989316627383232,
+            -0.007348849903792143,
+            -0.017280932515859604,
+            -0.004551833029836416,
+            0.012782545760273933,
+            -0.017031516879796982,
+            0.0053579797968268394,
+            -0.01559737790375948,
+            -0.013014145195484161,
+            -0.0167375635355711,
+            -0.006079503335058689,
+            0.005148648750036955,
+            -0.02216234989464283,
+            -0.012942884117364883,
+            -0.012862714938819408,
+            -0.009860821068286896,
+            -0.01879524067044258,
+            -0.011339498683810234,
+            -0.012016483582556248,
+            -0.019204994663596153,
+            -0.035559527575969696,
+            -0.014626438729465008,
+            -0.0037456867285072803,
+            -0.011615636758506298,
+            -0.0038414443843066692,
+            -0.03634340316057205,
+            0.012916160747408867,
+            6.475478585343808e-05,
+            -0.004371452145278454,
+            0.003963925410062075,
+            0.013397176750004292,
+            -0.00856920424848795,
+            -0.03237057104706764,
+            -0.028077062219381332,
+            -0.01972164213657379,
+            -0.00024579677847214043,
+            -0.00031984198722057045,
+            0.011259329505264759,
+            0.0004286829207558185,
+            0.017058240249753,
+            -0.01053780596703291,
+            0.017672870308160782,
+            -0.03586238995194435,
+            -0.03196081891655922,
+            -0.003759048180654645,
+            -0.008729543536901474,
+            -0.012043206952512264,
+            0.006409088149666786,
+            0.007037080824375153,
+            -0.0055851261131465435,
+            -0.012354975566267967,
+            -0.007117250002920628,
+            -0.016407977789640427,
+            -0.008925512433052063,
+            0.009210558608174324,
+            0.003333705710247159,
+            0.012809268198907375,
+            0.0150540079921484,
+            0.00868945848196745,
+            -0.01657722517848015,
+            0.01975727267563343,
+            -0.005362433847039938,
+            -0.003008574713021517,
+            0.013512976467609406,
+            0.010110235773026943,
+            0.0167375635355711,
+            0.01130386721342802,
+            0.004306871443986893,
+            -0.014964930713176727,
+            0.02911035344004631,
+            0.015846792608499527,
+            0.009174928069114685,
+            0.01543703954666853,
+            0.015151992440223694,
+            0.009081397205591202,
+            -0.01202539075165987,
+            -0.0007176263607107103,
+            -0.017655055969953537,
+            0.0008055898360908031,
+            0.007353303954005241,
+            -0.010992098599672318,
+            -0.006070595700293779,
+            0.033350419253110886,
+            0.0022703493013978004,
+            -0.015730993822216988,
+            0.013138853013515472,
+            -0.011250421404838562,
+            -0.006720857694745064,
+            -0.011214790865778923,
+            -0.0118472371250391,
+            -0.003019709372892976,
+            0.015303423628211021,
+            0.011366221122443676,
+            -0.02103998139500618,
+            0.011437483131885529,
+            0.01303196046501398,
+            -0.0057098339311778545,
+            -0.005335710477083921,
+            0.014528454281389713,
+            0.01870616339147091,
+            0.01433248445391655,
+            -0.03628995642066002,
+            -0.0023271357640624046,
+            0.012078837491571903,
+            -0.026527121663093567,
+            -0.019383149221539497,
+            0.013940545730292797,
+            0.0012548718368634582,
+            -0.019044656306505203,
+            0.01642579399049282,
+            -0.002663401421159506,
+            -0.022874966263771057,
+            -0.008542481809854507,
+            0.0034851364325731993,
+            0.019935425370931625,
+            0.021129056811332703,
+            -0.012542037293314934,
+            0.008217350579798222,
+            -0.014697699807584286,
+            0.004286829382181168,
+            -0.023355981335043907,
+            -0.0037456867285072803,
+            -0.008560297079384327,
+            -0.0021345068234950304,
+            0.017672870308160782,
+            0.008729543536901474,
+            0.02312438189983368,
+            0.019971057772636414,
+            0.0006380138220265508,
+            -0.016167471185326576,
+            -0.003306982573121786,
+            0.011232606135308743,
+            -0.008208442479372025,
+            -0.0002967376494780183,
+            -0.001585570047609508,
+            0.012907253578305244,
+            0.005393610801547766,
+            0.030642477795481682,
+            -0.002772520761936903,
+            0.0012994103599339724,
+            -3.0011284252395853e-05,
+            0.002788109239190817,
+            0.0012392833596095443,
+            -0.005972610786557198,
+            -0.0018917721463367343,
+            0.005153102800250053,
+            0.02552945911884308,
+            0.004939318168908358,
+            -0.0069123730063438416,
+            0.007976843044161797,
+            -0.0022169030271470547,
+            -0.011642360128462315,
+            -0.0038102674297988415,
+            -0.014840223826467991,
+            -0.01746799424290657,
+            0.01979290321469307,
+            0.0277029387652874,
+            -0.013762392103672028,
+            -0.026954691857099533,
+            -0.00011412987078074366,
+            0.04044094309210777,
+            0.002268122276291251,
+            0.0028727322351187468,
+            0.022946227341890335,
+            -0.0020465434063225985,
+            0.018563641235232353,
+            0.021823858842253685,
+            0.011027729138731956,
+            0.009442158974707127,
+            0.03221023455262184,
+            0.0017158451955765486,
+            -0.004694356583058834,
+            -0.0030286170076578856,
+            -0.0025320129934698343,
+            -0.012497498653829098,
+            0.0009325245628133416,
+            0.002859370782971382,
+            -0.02068367227911949,
+            0.00792785082012415,
+            -0.01959693431854248,
+            -0.0006819955888204277,
+            -0.003652155864983797,
+            0.0065382495522499084,
+            -0.025476014241576195,
+            -0.010047882795333862,
+            0.009967712685465813,
+            -0.01043982058763504,
+            0.0023739011958241463,
+            -0.01915154792368412,
+            -0.004184390883892775,
+            0.022055458277463913,
+            -0.004451621789485216,
+            0.013887099921703339,
+            -0.00841331947594881,
+            -0.0011318342294543982,
+            -0.008404412306845188,
+            -0.0066451421007514,
+            0.011196975596249104,
+            -0.0058122724294662476,
+            0.001120699685998261,
+            -0.010876297950744629,
+            0.008065919391810894,
+            -0.005634118337184191,
+            0.004805702716112137,
+            0.023712290450930595,
+            -0.0030442054849117994,
+            -0.0068901036866009235,
+            0.007424565497785807,
+            0.010038974694907665,
+            -0.022411765530705452,
+            0.02702595293521881,
+            -0.011348405852913857,
+            0.02059459500014782,
+            0.00296626309864223,
+            -0.019703825935721397,
+            -0.0027168476954102516,
+            -0.0010488813277333975,
+            0.010867390781641006,
+            -0.006649595685303211,
+            0.002970716916024685,
+            -0.01831422559916973,
+            0.015276700258255005,
+            0.005945887882262468,
+            -0.015330146998167038,
+            0.01979290321469307,
+            -0.006988088600337505,
+            0.0019663742277771235,
+            -0.017485808581113815,
+            0.00578109547495842,
+            0.007130611687898636,
+            -0.013664407655596733,
+            -0.003739005886018276,
+            -0.019703825935721397,
+            -0.0005723195499740541,
+            -0.01565973088145256,
+            0.0036098442506045103,
+            -0.009718297980725765,
+            -0.013646592386066914,
+            -0.036913495510816574,
+            -0.0001862265489762649,
+            0.004685448482632637,
+            -0.007197419181466103,
+            -0.001399621949531138,
+            0.00012637795589398593,
+            0.013958360999822617,
+            0.009290727786719799,
+            -0.0039572445675730705,
+            0.004979402758181095,
+            0.034348081797361374,
+            -0.04421781003475189,
+            -0.01138403732329607,
+            -0.00033654391882009804,
+            0.004008463583886623,
+            -0.020737119019031525,
+            -0.008275250904262066,
+            0.013121037743985653,
+            -0.0007081619114615023,
+            0.016274362802505493,
+            -0.016033854335546494,
+            0.010947559960186481,
+            0.003079836256802082,
+            0.008177265524864197,
+            -0.007135065272450447,
+            -0.0053579797968268394,
+            -0.012034298852086067,
+            -0.000576773367356509,
+            -0.001016590977087617,
+            -0.02827303111553192,
+            0.01734328642487526,
+            0.0016033854335546494,
+            -0.0056296647526323795,
+            0.0069480035454034805,
+            -0.02453179843723774,
+            -0.009148204699158669,
+            -0.012292621657252312,
+            0.023658843711018562,
+            -0.01738782413303852,
+            -0.00026625662576407194,
+            -0.004698810167610645,
+            -0.006444718688726425,
+            0.014724423177540302,
+            -0.0037234174087643623,
+            0.007322126999497414,
+            0.00037635021726600826,
+            -0.011339498683810234,
+            -0.002014253055676818,
+            0.0024785667192190886,
+            -0.010190405882894993,
+            0.031693585216999054,
+            -0.005959249567240477,
+            -0.0027702937368303537,
+            0.007433473132550716,
+            0.0010666967136785388,
+            -0.002318228129297495,
+            0.010270575061440468,
+            0.0017882202519103885,
+            0.0039572445675730705,
+            0.008894335478544235,
+            -0.0036989212967455387,
+            -0.006021603476256132,
+            0.016007130965590477,
+            0.020541150122880936,
+            0.005616303067654371,
+            -0.016033854335546494,
+            -0.011116806417703629,
+            0.011758159846067429,
+            0.0015733219916000962,
+            0.0006396839744411409,
+            -0.008057012222707272,
+            0.006324464920908213,
+            -0.002256987849250436,
+            0.012043206952512264,
+            0.011838329024612904,
+            0.017298748716711998,
+            0.012898345477879047,
+            -0.005723195616155863,
+            0.03144416958093643,
+            0.01614074781537056,
+            0.0009007908520288765,
+            0.015134177170693874,
+            -0.04368334636092186,
+            -0.01027948223054409,
+            -0.012078837491571903,
+            0.01754816249012947,
+            -0.014751146547496319,
+            -0.016488147899508476,
+            -0.013183391653001308,
+            0.010653605684638023,
+            -0.0032446286641061306,
+            0.005473779980093241,
+            0.0251375213265419,
+            -0.013414992019534111,
+            0.004393721465021372,
+            -0.002739116782322526,
+            0.023623213171958923,
+            0.0030263899825513363,
+            0.00907694362103939,
+            0.007495827041566372,
+            0.005104110576212406,
+            -0.0027057130355387926,
+            0.01963256485760212,
+            -0.009219466708600521,
+            0.036485929042100906,
+            -0.01053780596703291,
+            -0.009317451156675816,
+            -0.01582006923854351,
+            -0.017637239769101143,
+            -0.012354975566267967,
+            0.004752256441861391,
+            -0.000690346525516361,
+            0.0012570987455546856,
+            -0.00821289699524641,
+            0.013637684285640717,
+            0.03140854090452194,
+            -0.0013417218578979373,
+            0.015160900540649891,
+            0.007856588810682297,
+            -0.0006513753323815763,
+            0.01543703954666853,
+            0.003883755998685956,
+            -0.004489479120820761,
+            0.004729987122118473,
+            0.024513982236385345,
+            0.0004924286040477455,
+            0.00767843471840024,
+            -0.0010744909523054957,
+            -0.009602497331798077,
+            -0.01156219094991684,
+            0.010831759311258793,
+            0.027115030214190483,
+            0.017334379255771637,
+            0.001331700710579753,
+            -0.0118472371250391,
+            -0.00017815394676290452,
+            -0.0009375351364724338,
+            0.016791008412837982,
+            -0.010965375229716301,
+            -0.006689680740237236,
+            0.013121037743985653,
+            -0.009085850790143013,
+            0.007611627224832773,
+            -0.01662176288664341,
+            0.0013907141983509064,
+            0.00722859613597393,
+            0.014270130544900894,
+            -0.0045206560753285885,
+            0.006070595700293779,
+            -0.00707716541364789,
+            0.010555621236562729,
+            -0.0127290990203619,
+            0.011277144774794579,
+            -0.005723195616155863,
+            0.00046709735761396587,
+            0.028807492926716805,
+            -0.006475895643234253,
+            0.008195081725716591,
+            0.00011621761223068461,
+            -0.0034717749804258347,
+            -0.017494717612862587,
+            0.025761060416698456,
+            0.004084179177880287,
+            0.005914710927754641,
+            0.00020014481560792774,
+            0.010288390330970287,
+            -0.008301973342895508,
+            -0.001823851023800671,
+            0.022304873913526535,
+            0.011927406303584576,
+            0.0038703943137079477,
+            -0.0005873512709513307,
+            0.008003565482795238,
+            -0.012221360579133034,
+            0.015134177170693874,
+            0.01899120956659317,
+            0.008644919842481613,
+            0.0036543826572597027,
+            0.014599715359508991,
+            -0.013094314374029636,
+            0.008466766215860844,
+            -0.009072489105165005,
+            0.029163800179958344,
+            0.012640022672712803,
+            -0.016523778438568115,
+            0.019294071942567825,
+            0.030767185613512993,
+            -0.004930410534143448,
+            -0.010430913418531418,
+            0.0015844566514715552,
+            -0.017414547502994537,
+            -0.0021567761432379484,
+            -0.014920393005013466,
+            -0.014768961817026138,
+            0.005126379895955324,
+            -0.015312331728637218,
+            0.02497718296945095,
+            -0.017280932515859604,
+            -0.031657956540584564,
+            -0.004309098701924086,
+            -0.00825298111885786,
+            -0.02818395383656025,
+            -0.008301973342895508,
+            0.006190849468111992,
+            0.014849130995571613,
+            -0.0019930971320718527,
+            -0.0144928228110075,
+            0.013664407655596733,
+            0.009085850790143013,
+            0.01359314564615488,
+            -0.017316563054919243,
+            -0.01606057770550251,
+            -0.015615193173289299,
+            0.017316563054919243,
+            -0.010956467129290104,
+            -0.016470331698656082,
+            0.014270130544900894,
+            0.003759048180654645,
+            0.005464872345328331,
+            0.0017503624549135566,
+            -0.014519546180963516,
+            -0.016354531049728394,
+            -0.027845460921525955,
+            0.010626882314682007,
+            -0.0027970168739557266,
+            -0.010974282398819923,
+            0.0029194976668804884,
+            0.011437483131885529,
+            0.00939762033522129,
+            0.005749918520450592,
+            -0.018242964521050453,
+            0.011268236674368382,
+            0.01311213057488203,
+            -0.018528010696172714,
+            -0.007429019547998905,
+            0.005451510660350323,
+            -0.009727205149829388,
+            0.03798241913318634,
+            0.01345062255859375,
+            -0.015205439180135727,
+            -0.0030397516675293446,
+            0.004930410534143448,
+            0.01686227135360241,
+            0.010056789964437485,
+            -0.018848687410354614,
+            -0.001327246893197298,
+            0.021895119920372963,
+            -0.004549606237560511,
+            0.004366998560726643,
+            -3.483918590063695e-06,
+            0.042614422738552094,
+            0.0033426133450120687,
+            -0.0056875646114349365,
+            -0.013138853013515472,
+            0.002210222417488694,
+            0.00997662078589201,
+            0.013023053295910358,
+            0.012239175848662853,
+            -0.007326581049710512,
+            0.0025008360389620066,
+            0.02020265720784664,
+            -0.010395282879471779,
+            0.0020955358631908894,
+            0.0030352978501468897,
+            0.005936980247497559,
+            -0.016880085691809654,
+            0.026242075487971306,
+            -0.0009653716697357595,
+            0.00692573469132185,
+            0.0004620867839548737,
+            -0.009923174977302551,
+            0.014670977368950844,
+            0.024122044444084167,
+            0.0008417773642577231,
+            0.015624100342392921,
+            -0.012435144744813442,
+            0.01831422559916973,
+            0.007901127450168133,
+            0.00782986544072628,
+            -0.015330146998167038,
+            -0.010092420503497124,
+            0.006952457595616579,
+            -0.0028816401027143,
+            -0.002538693603128195,
+            -0.0039839674718678,
+            -0.012372790835797787,
+            -0.011936314404010773,
+            -0.021129056811332703,
+            0.004373679403215647,
+            0.00836878176778555,
+            -0.006132949609309435,
+            0.006850019097328186,
+            0.004309098701924086,
+            -0.0033671094570308924,
+            0.0005984858726151288,
+            -0.013682222925126553,
+            -0.008707273751497269,
+            0.0033470671623945236,
+            0.002696805400773883,
+            0.004191071726381779,
+            -0.007856588810682297,
+            0.020184841006994247,
+            0.006609511096030474,
+            -0.006110680289566517,
+            0.007874404080212116,
+            -0.019062472507357597,
+            -0.0028660516254603863,
+            -0.012105559930205345,
+            0.00011127662583021447,
+            0.0042200214229524136,
+            0.01638125441968441,
+            0.0031243746634572744,
+            -0.023747920989990234,
+            0.0068322038277983665,
+            -0.01202539075165987,
+            0.008965597487986088,
+            -0.003741232678294182,
+            -0.0005160897271707654,
+            -0.012693468481302261,
+            0.01594477705657482,
+            0.01535686943680048,
+            -0.011419667862355709,
+            -0.021093426272273064,
+            0.017637239769101143,
+            -0.03103441745042801,
+            0.022108905017375946,
+            0.010671420954167843,
+            0.0036855596117675304,
+            -0.019810717552900314,
+            0.017120594158768654,
+            0.012399514205753803,
+            0.01101882103830576,
+            -0.0032936211209744215,
+            0.011294960044324398,
+            0.021627888083457947,
+            -0.007941211573779583,
+            -0.0010410870891064405,
+            -9.525320820102934e-06,
+            -0.010430913418531418,
+            -0.0002900568943005055,
+            -0.007175149861723185,
+            0.014421561732888222,
+            0.00836878176778555,
+            -0.008836435154080391,
+            -0.013548607006669044,
+            0.01228371448814869,
+            -0.011482021771371365,
+            0.007117250002920628,
+            -0.005985972471535206,
+            0.007219688501209021,
+            0.004342502448707819,
+            0.01770850270986557,
+            -0.021057795733213425,
+            0.0005670305690728128,
+            0.005237726029008627,
+            -0.022946227341890335,
+            0.007041534408926964,
+            -0.024585243314504623,
+            0.014902577735483646,
+            -0.025351306423544884,
+            -0.005549495108425617,
+            0.02046988718211651,
+            0.014599715359508991,
+            0.0033092093653976917,
+            -0.018011363223195076,
+            -0.007059350144118071,
+            -0.013619869016110897,
+            0.002101103076711297,
+            -0.022429581731557846,
+            -0.013388268649578094,
+            -0.015303423628211021,
+            0.009531236253678799,
+            0.011027729138731956,
+            0.0024028513580560684,
+            -0.010582344606518745,
+            0.007535911630839109,
+            -0.02168133482336998,
+            -0.010617975145578384,
+            -0.014653162099421024,
+            -0.017681779339909554,
+            0.001418550731614232,
+            0.02661619894206524,
+            -0.0027413438074290752,
+            0.01879524067044258,
+            0.011954129673540592,
+            0.01471551600843668,
+            -0.0031733671203255653,
+            -0.010938651859760284,
+            -0.00923728197813034,
+            -0.0027925630565732718,
+            -0.003554171184077859,
+            -0.018777426332235336,
+            -0.012559852562844753,
+            0.00489923357963562,
+            -0.0019853028934448957,
+            0.003981740679591894,
+            0.007874404080212116,
+            0.00841331947594881,
+            -0.01100100576877594,
+            -0.010751590132713318,
+            0.016630670055747032,
+            -0.025671983137726784,
+            -0.017405640333890915,
+            0.018278595060110092,
+            -0.011232606135308743,
+            -0.0012348295422270894,
+            0.019864164292812347,
+            -0.010956467129290104,
+            -0.007491373457014561,
+            0.010493267327547073,
+            -0.003865940496325493,
+            0.010528897866606712,
+            0.022180166095495224,
+            0.006484803277999163,
+            -0.016087301075458527,
+            0.008595927618443966,
+            0.010609067045152187,
+            -0.014163237996399403,
+            -0.02734662964940071,
+            -0.0026010475121438503,
+            0.004560741130262613,
+            -0.020701488479971886,
+            -0.0026945783756673336,
+            0.0004078055208083242,
+            0.00722859613597393,
+            -0.0003855362592730671,
+            0.004925956483930349,
+            -0.018100440502166748,
+            -0.007139519322663546,
+            0.003429463366046548,
+            0.00868945848196745,
+            -0.022215796634554863,
+            0.0013873737771064043,
+            -0.013227930292487144,
+            0.02216234989464283,
+            -0.003257990116253495,
+            0.007415657863020897,
+            -0.0012548718368634582,
+            -0.0074201119132339954,
+            0.008769627660512924,
+            0.012069929391145706,
+            0.005972610786557198,
+            0.03295847773551941,
+            -0.003894890658557415,
+            -0.015009469352662563,
+            0.014234500005841255,
+            -0.025885768234729767,
+            -0.0007337715360336006,
+            -0.0038325367495417595,
+            0.010626882314682007,
+            -0.011036636307835579,
+            -0.011188067495822906,
+            0.02203764207661152,
+            0.02100434899330139,
+            -0.006930188275873661,
+            -0.0002297907485626638,
+            -0.01750362478196621,
+            0.01662176288664341,
+            -0.00824852753430605,
+            0.0015833431389182806,
+            0.007976843044161797,
+            -0.023320350795984268,
+            -0.012631114572286606,
+            -0.006021603476256132,
+            0.024870289489626884,
+            -0.01934751868247986,
+            -0.016559408977627754,
+            -0.012515314854681492,
+            0.00012282878742553294,
+            -0.02709721401333809,
+            -0.0075225504115223885,
+            0.007063803728669882,
+            -0.012925068847835064,
+            -0.007553727366030216,
+            0.007954573258757591,
+            0.009477789513766766,
+            -0.0021033298689872026,
+            -0.02520878240466118,
+            -0.011188067495822906,
+            0.013210115022957325,
+            0.003939428832381964,
+            0.006190849468111992,
+            -0.004364771768450737,
+            0.006168580148369074,
+            -0.006279926281422377,
+            0.0032513095065951347,
+            0.00899231992661953,
+            -0.0016490374691784382,
+            -0.006342280190438032,
+            0.02219798043370247,
+            0.005522772204130888,
+            -0.018180610612034798,
+            0.01433248445391655,
+            0.01570427045226097,
+            -0.012185730040073395,
+            -0.008489035069942474,
+            3.824046507361345e-05,
+            -0.013931638561189175,
+            -0.0008918831590563059,
+            -0.006475895643234253,
+            0.03196081891655922,
+            -0.022874966263771057,
+            0.003857032861560583,
+            0.008685004897415638,
+            -0.0002842112153302878,
+            -0.004903687164187431,
+            0.002912817057222128,
+            0.0054826876148581505,
+            -0.0012793679488822818,
+            0.00577218784019351,
+            0.00812827330082655,
+            -0.014216684736311436,
+            -0.006337826605886221,
+            0.013691130094230175,
+            0.002291504992172122,
+            0.017806487157940865,
+            0.016452517360448837,
+            1.9311608411953785e-05,
+            -0.03313663229346275,
+            0.004193298518657684,
+            0.0007877744501456618,
+            -0.021627888083457947,
+            -0.0038169482722878456,
+            -0.006106226239353418,
+            0.00967375934123993,
+            0.010324020870029926,
+            0.01614074781537056,
+            0.0029194976668804884,
+            0.007143972907215357,
+            0.006141857244074345,
+            0.010288390330970287,
+            -0.030927523970603943,
+            -0.006444718688726425,
+            -0.008333150297403336,
+            -0.006453626789152622,
+            0.012622206471860409,
+            0.012631114572286606,
+            0.029983308166265488,
+            0.028736229985952377,
+            -0.014751146547496319,
+            -0.019204994663596153,
+            0.004242290742695332,
+            0.011027729138731956,
+            0.008640466257929802,
+            -0.007290950044989586,
+            0.006462534423917532,
+            8.504067227477208e-05,
+            -0.02526222914457321,
+            0.0017392279114574194,
+            0.01375348400324583,
+            -0.017530348151922226,
+            -0.010617975145578384,
+            -0.018652718514204025,
+            0.014626438729465008,
+            0.02116468921303749,
+            -0.00392384035512805,
+            0.007188511546701193,
+            -0.010591251775622368,
+            -0.009317451156675816,
+            -0.04500168561935425,
+            0.0053000799380242825,
+            0.009054673835635185,
+            0.022215796634554863,
+            0.001312771812081337,
+            -0.004001782741397619,
+            -6.158837641123682e-05,
+            0.007001449819654226,
+            -0.0033960596192628145,
+            0.011464206501841545,
+            0.013210115022957325,
+            -0.018759610131382942,
+            -0.016042763367295265,
+            0.004150986671447754,
+            0.0023115472868084908,
+            -0.010172589682042599,
+            0.005968157202005386,
+            0.021253764629364014,
+            0.01586460880935192,
+            -0.007241957820951939,
+            0.004431579262018204,
+            0.005625210702419281,
+            -0.014225591905415058,
+            -0.012034298852086067,
+            -0.0045429253950715065,
+            -0.011517652310431004,
+            -0.024193305522203445,
+            0.017191855236887932,
+            -0.013824746012687683,
+            0.006395726464688778,
+            0.009869728237390518,
+            0.011223698034882545,
+            0.0018450067145749927,
+            0.023177828639745712,
+            0.00811045803129673,
+            -0.008738450706005096,
+            0.011036636307835579,
+            -0.026170814409852028,
+            0.012684560380876064,
+            -0.00023368786787614226,
+            0.011259329505264759,
+            0.0001613128260942176,
+            -0.03301192447543144,
+            -0.013512976467609406,
+            0.008172811940312386,
+            -0.006622872781008482,
+            0.009896451607346535,
+            0.014626438729465008,
+            -0.008288612589240074,
+            -0.00100879673846066,
+            -0.022340504452586174,
+            0.00445830263197422,
+            -0.03147980198264122,
+            0.00011357313633197919,
+            -0.004716625437140465,
+            0.010929744690656662,
+            -0.005362433847039938,
+            -0.0240151509642601,
+            0.0026789898984134197,
+            -0.012720191851258278,
+            -0.005825634114444256,
+            0.0010616861982271075,
+            -0.0005828974535688758,
+            0.010876297950744629,
+            -0.00593252619728446,
+            0.01375348400324583,
+            -0.007348849903792143,
+            0.005215456709265709,
+            -0.0007688456098549068,
+            0.006792119238525629,
+            -0.006658503785729408,
+            -0.012907253578305244,
+            0.00238948967307806,
+            -0.004845787305384874,
+            0.021342841908335686,
+            0.010243851691484451,
+            -0.0006380138220265508,
+            0.004362544510513544,
+            0.020291734486818314,
+            0.004672087263315916,
+            0.013165576383471489,
+            0.006676319055259228,
+            0.0032000902574509382,
+            -0.015285608358681202,
+            0.009736113250255585,
+            -0.007371119223535061,
+            0.009477789513766766,
+            0.002870505442842841,
+            -0.0010310659417882562,
+            0.014599715359508991,
+            0.008364327251911163,
+            -0.009905358776450157,
+            0.011767067946493626,
+            0.018581455573439598,
+            -0.008408865891397,
+            -0.003220132552087307,
+            -0.0127290990203619,
+            0.011294960044324398,
+            0.027204107493162155,
+            0.004260106012225151,
+            -0.01831422559916973,
+            0.014145422726869583,
+            0.00010097710037371144,
+            -0.011713622137904167,
+            0.026046106591820717,
+            -0.014751146547496319,
+            -0.025689799338579178,
+            -0.013379361480474472,
+            -0.007273134775459766,
+            -0.01750362478196621,
+            0.021271580830216408,
+            -0.011473113670945168,
+            0.012212452478706837,
+            0.004193298518657684,
+            0.03119475580751896,
+            0.010689236223697662,
+            0.01212337613105774,
+            -0.003015255555510521,
+            -0.025351306423544884,
+            0.009379805065691471,
+            0.0045874640345573425,
+            -0.04044094309210777,
+            0.0005979291745461524,
+            -0.003004120895639062,
+            0.029377585276961327,
+            0.0010550053557381034,
+            -0.0014898123918101192,
+            -0.0038971174508333206,
+            -0.020380811765789986,
+            -0.015686454251408577,
+            0.007322126999497414,
+            -0.028682785108685493,
+            0.013397176750004292,
+            -0.00507738720625639,
+            -0.012150098569691181,
+            0.002214676234871149,
+            0.010831759311258793,
+            -0.019044656306505203,
+            0.01879524067044258,
+            0.015846792608499527,
+            0.025030629709362984,
+            0.01069814432412386,
+            -0.008885428309440613,
+            0.020006688311696053,
+            0.0010210447944700718,
+            -0.006591695826500654,
+            0.0020688127260655165,
+            -1.8998447558260523e-05,
+            0.0015410316409543157,
+            -0.0033671094570308924,
+            0.003015255555510521,
+            0.01598040945827961,
+            -0.01738782413303852,
+            0.019418779760599136,
+            -0.01702260971069336,
+            0.016007130965590477,
+            0.0005347402184270322,
+            0.012925068847835064,
+            4.071095827384852e-05,
+            0.0011958583490923047,
+            -0.029840786010026932,
+            -0.0009653716697357595,
+            -0.03915823623538017,
+            -0.011312775313854218,
+            -0.0016679662512615323,
+            -0.018091533333063126,
+            -0.0037679558154195547,
+            0.009032404981553555,
+            -0.01487585436552763,
+            -0.011588914319872856,
+            0.016069484874606133,
+            0.007598265539854765,
+            0.007375573273748159,
+            0.001965260598808527,
+            -8.03084549261257e-05,
+            -0.005162010435014963,
+            -0.020808380097150803,
+            -0.012951791286468506,
+            -0.025155337527394295,
+            -0.025190968066453934,
+            -0.0035742134787142277,
+            0.007584904320538044,
+            -0.004832425620406866,
+            -0.008564750663936138,
+            0.017761947587132454,
+            -0.019044656306505203,
+            -0.00040140311466529965,
+            0.004242290742695332,
+            0.008172811940312386,
+            0.013103222474455833,
+            -0.007063803728669882,
+            0.007384480908513069,
+            0.0030709286220371723,
+            -0.0013361545279622078,
+            -0.0005183166358619928,
+            0.024104228243231773,
+            -0.008836435154080391,
+            0.011009913869202137,
+            0.007722973357886076,
+            0.011775976046919823,
+            -0.006217572372406721,
+            0.009045766666531563,
+            -0.024104228243231773,
+            0.007687342818826437,
+            -0.011045544408261776,
+            -0.026794353500008583,
+            -0.007148426957428455,
+            0.02071930281817913,
+            -0.0006536022992804646,
+            0.0008662735344842076,
+            -0.0018193970900028944,
+            -0.0004406526277307421,
+            0.023676658049225807,
+            0.01042200531810522,
+            0.0012147872475907207,
+            -0.007700704038143158,
+            -0.029769523069262505,
+            0.005985972471535206,
+            -0.021271580830216408,
+            -0.021699151024222374,
+            0.03288721665740013,
+            0.010368559509515762,
+            0.0027213015127927065,
+            -0.013691130094230175,
+            0.010038974694907665,
+            0.009442158974707127,
+            6.134480645414442e-05,
+            0.014804592356085777,
+            0.003088743891566992,
+            0.001969714416190982,
+            -0.009932082146406174,
+            -0.01918718032538891,
+            0.01614074781537056,
+            0.014581900089979172,
+            -0.01559737790375948,
+            0.036877866834402084,
+            0.016773194074630737,
+            0.012809268198907375,
+            -0.010288390330970287,
+            0.007714065723121166,
+            0.016995886340737343,
+            0.020060133188962936,
+            -0.0070860730484128,
+            0.0034339171834290028,
+            -0.010742682963609695,
+            -0.00023466214770451188,
+            0.03894445300102234,
+            0.03516758978366852,
+            0.008444496430456638,
+            -0.022785888984799385,
+            0.014581900089979172,
+            -0.009335266426205635,
+            -0.01287162210792303,
+            0.010306205600500107,
+            0.01606057770550251,
+            -0.0005923618446104228,
+            0.0014686565846204758,
+            0.009451067075133324,
+            0.0076606194488704205,
+            0.017441270872950554,
+            0.005353525746613741,
+            -0.002378355246037245,
+            0.003402740228921175,
+            -0.0034339171834290028,
+            0.00965594407171011,
+            -0.016808824613690376,
+            -0.005126379895955324,
+            -0.00339828641153872,
+            -0.0016969162970781326,
+            0.005251087248325348,
+            0.009540143422782421,
+            -0.036361221224069595,
+            -0.007415657863020897,
+            -0.033350419253110886,
+            0.004021825268864632,
+            -0.01995324157178402,
+            -0.0035430365242064,
+            -0.010965375229716301,
+            -0.01598040945827961,
+            -0.026562752202153206,
+            -0.016933532431721687,
+            -0.0031266016885638237,
+            0.008903243578970432,
+            -0.003888209816068411,
+            0.003215678734704852,
+            0.003436144208535552,
+            -0.021378472447395325,
+            0.005660841707140207,
+            0.013174484483897686,
+            0.009370896965265274,
+            0.021057795733213425,
+            0.0001496214681537822,
+            0.014368114992976189,
+            -0.014822407625615597,
+            -0.04029842093586922,
+            -0.0007649484905414283,
+            0.00031009921804070473,
+            -0.005175372119992971,
+            -0.00424674479290843,
+            0.004859148990362883,
+            -0.009379805065691471,
+            0.011098990216851234,
+            -0.0021745916455984116,
+            0.0009681553347036242,
+            -0.015722084790468216,
+            -0.010707051493227482,
+            -0.023587582632899284,
+            -0.00505066430196166,
+            -0.013121037743985653,
+            -2.176470479753334e-05,
+            -0.01288943737745285,
+            -0.004952679388225079,
+            -0.005398064386099577,
+            -0.0013461756752803922,
+            -0.006426903419196606,
+            -0.002723528305068612,
+            -0.001202539075165987,
+            -0.013263560831546783,
+            -0.012559852562844753,
+            -0.0013172257458791137,
+            0.004832425620406866,
+            0.0006942436448298395,
+            -0.01098319049924612,
+            0.013414992019534111,
+            -0.014911484904587269,
+            0.0018038086127489805,
+            0.02574324421584606,
+            0.013325914740562439,
+            -0.010484359227120876,
+            0.016479238867759705,
+            0.002199087757617235,
+            0.010128051973879337,
+            -0.02152099646627903,
+            -0.015722084790468216,
+            -0.010110235773026943,
+            0.026990322396159172,
+            0.0012682334054261446,
+            -0.013557515107095242,
+            -0.007206326816231012,
+            -0.01706714741885662,
+            0.004680994898080826,
+            -0.0077407886274158955,
+            0.011375129222869873,
+            -0.020630227401852608,
+            -0.007215234450995922,
+            -0.011473113670945168,
+            -0.007531458046287298,
+            0.0065827881917357445,
+            0.03794679045677185,
+            -0.00289054773747921,
+            -0.007852135226130486,
+            0.005963703151792288,
+            0.010849575512111187,
+            0.028237400576472282,
+            0.0010516649344936013,
+            -0.0014307989040389657,
+            -0.012043206952512264,
+            0.0034717749804258347,
+            0.004551833029836416,
+            -0.01051999069750309,
+            0.017031516879796982,
+            -0.009148204699158669,
+            -0.01934751868247986,
+            -0.004500614013522863,
+            -0.00868945848196745,
+            -0.007353303954005241,
+            0.01753925532102585,
+            0.0023226819466799498,
+            0.003208997892215848,
+            0.0008540254784747958,
+            -0.007250865455716848,
+            -0.008333150297403336,
+            0.011009913869202137,
+            0.08451622724533081,
+            0.00943325087428093,
+            -0.02228705771267414,
+            0.0012526449281722307,
+            0.004943771753460169,
+            0.001479791128076613,
+            -0.014795685186982155,
+            0.06591695547103882,
+            -0.014368114992976189,
+            0.005451510660350323,
+            0.0004428795655258,
+            0.004810156300663948,
+            -0.004447167739272118,
+            0.02136065810918808,
+            0.010199313051998615,
+            -0.01244405284523964,
+            0.002025387715548277,
+            -0.03541700541973114,
+            -0.017280932515859604,
+            0.0034984981175512075,
+            -0.0044404868967831135,
+            -0.004393721465021372,
+            0.013121037743985653,
+            -0.023106565698981285,
+            -0.008163904771208763,
+            0.010956467129290104,
+            -0.012052114121615887,
+            0.011473113670945168,
+            -0.019383149221539497,
+            -0.001312771812081337,
+            -0.0020198202691972256,
+            0.0014519545948132873,
+            -0.009727205149829388,
+            0.0036766519770026207,
+            -0.0027123938780277967,
+            0.0010705938329920173,
+            0.009477789513766766,
+            0.0077853272669017315,
+            -0.003741232678294182,
+            0.017877748236060143,
+            0.013441715389490128,
+            0.0035563979763537645,
+            0.017485808581113815,
+            0.0027480246499180794,
+            0.007651711814105511,
+            0.0003485136548988521,
+            0.01785993203520775,
+            -0.014483915641903877,
+            0.021022165194153786,
+            -0.01834985613822937,
+            -0.02059459500014782,
+            -0.013691130094230175,
+            0.009063581936061382,
+            -0.015561746433377266,
+            0.001760383602231741,
+            -0.011571098119020462,
+            0.029858600348234177,
+            0.010885206051170826,
+            -0.003088743891566992,
+            0.004380360245704651,
+            -0.007045988459140062,
+            0.03930075839161873,
+            0.007856588810682297,
+            0.033225711435079575,
+            0.003155551617965102,
+            0.017886655405163765,
+            -0.002788109239190817,
+            0.004001782741397619,
+            -0.013254653662443161,
+            -0.005821180064231157,
+            0.018278595060110092,
+            -0.009905358776450157,
+            0.019062472507357597,
+            0.004863602574914694,
+            -0.014439377002418041,
+            0.01770850270986557,
+            -0.015410316176712513,
+            -0.02292841300368309,
+            0.013842561282217503,
+            -0.02408641390502453,
+            0.0002925621811300516,
+            0.006230934057384729,
+            -0.0036276597529649734,
+            -0.01051999069750309,
+            0.021752595901489258,
+            -0.0005116358515806496,
+            -0.006907918956130743,
+            -0.010644698515534401,
+            -0.013245745562016964,
+            -0.014279038645327091,
+            0.00895668938755989,
+            -0.004574102349579334,
+            -0.01298742275685072,
+            0.011009913869202137,
+            -0.02526222914457321,
+            0.012862714938819408,
+            -0.012043206952512264,
+            0.023017490282654762,
+            -0.00795011967420578,
+            0.02075493521988392,
+            0.0022825973574072123,
+            -0.0002908919705078006,
+            0.0019853028934448957,
+            0.008863158524036407,
+            -0.003224586369469762,
+            0.016327809542417526,
+            0.004819063935428858,
+            0.004997218027710915,
+            0.004583009984344244,
+            -0.005041756667196751,
+            0.015472670085728168,
+            -0.011749252676963806,
+            -0.0009202764485962689,
+            0.01418996136635542,
+            0.009290727786719799,
+            -0.018207332119345665,
+            0.025850137695670128,
+            0.0038748481310904026,
+            0.015303423628211021,
+            -0.023266904056072235,
+            0.0058924416080117226,
+            0.012622206471860409,
+            0.002320455154404044,
+            0.01288943737745285,
+            -0.004729987122118473,
+            0.022946227341890335,
+            0.018902134150266647,
+            0.004734441172331572,
+            -0.0025854590348899364,
+            0.013040868565440178,
+            -0.002086627995595336,
+            -0.0016456970479339361,
+            0.011891775764524937,
+            -0.0034940443001687527,
+            0.004077498335391283,
+            0.001429685391485691,
+            -0.0006591695710085332,
+            0.016755377873778343,
+            -0.01399399247020483,
+            0.014216684736311436,
+            0.006030511111021042,
+            -0.011063359677791595,
+            -0.013896007090806961,
+            0.007099434733390808,
+            -0.015080731362104416,
+            0.0025097436737269163,
+            0.023177828639745712,
+            -0.0014775642193853855,
+            -0.00865382794290781,
+            0.0048502408899366856,
+            0.012381698936223984,
+            -0.0058479029685258865,
+            -0.010484359227120876,
+            0.006181941833347082,
+            0.008221804164350033,
+            0.00027766963467001915,
+            0.006596149876713753,
+            0.0068099345080554485,
+            0.007028173189610243,
+            -0.0007922283257357776,
+            -0.008195081725716591,
+            0.0030508863274008036,
+            0.020933087915182114,
+            -0.0008629331714473665,
+            -0.009210558608174324,
+            -0.03643248230218887,
+            -0.003191182389855385,
+            -0.014679884538054466,
+            0.003086517099291086,
+            0.00030870738555677235,
+            -0.0008350965799763799,
+            -0.004493933171033859,
+            -0.009638127870857716,
+            0.009099212475121021,
+            0.0008907697047106922,
+            0.024745581671595573,
+            -0.012693468481302261,
+            -0.015365777537226677,
+            0.003257990116253495,
+            -0.0055405874736607075,
+            -0.010769405402243137,
+            0.007451288867741823,
+            -0.004213340580463409,
+            0.03269124776124954,
+            0.00901013519614935,
+            0.0025052898563444614,
+            -0.014786777086555958,
+            -0.018813056871294975,
+            0.010208221152424812,
+            -0.00245852442458272,
+            -0.005598487798124552,
+            -0.003939428832381964,
+            -0.02994767762720585,
+            0.010324020870029926,
+            -0.00967375934123993,
+            0.0036744249518960714,
+            -0.0044182175770401955,
+            0.009522328153252602,
+            0.003327024867758155,
+            -0.01487585436552763,
+            -0.0030375246424227953,
+            0.013566422276198864,
+            0.015258884988725185,
+            -0.012435144744813442,
+            0.004451621789485216,
+            -0.005999334156513214,
+            -0.012185730040073395,
+            0.009326359257102013,
+            -0.00967375934123993,
+            -0.017655055969953537,
+            0.002447389764711261
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "ce74a089-6f7a-4fc6-9794-673c9d056b10",
+      "type": "next",
+      "source": {
+        "id": "95f27e84-0fb9-44ee-a783-45f42551401c",
+        "properties": {
+          "page_content": "5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n",
+          "title": "Scaling Up To Deepseekmoe 16B",
+          "keyphrases": [
+            "DeepSeekMoE 16B",
+            "MoE model",
+            "language modeling",
+            "evaluation benchmarks",
+            "Open LLM Leaderboard"
+          ],
+          "embedding": [
+            0.005968157202005386,
+            -0.015490485355257988,
+            -0.027613861486315727,
+            0.012274806387722492,
+            0.008261889219284058,
+            -0.01577553153038025,
+            -0.016158562153577805,
+            0.06238951161503792,
+            -0.002393943490460515,
+            0.048564765602350235,
+            0.008756265975534916,
+            -0.009914266876876354,
+            0.03532792627811432,
+            -0.018332039937376976,
+            -0.006921280641108751,
+            0.023961706086993217,
+            0.018617086112499237,
+            0.02068367227911949,
+            -0.00015922509192023426,
+            -0.02930632419884205,
+            -0.021538810804486275,
+            -0.03865940496325493,
+            -0.037447959184646606,
+            -0.023801365867257118,
+            -0.021129056811332703,
+            0.012862714938819408,
+            0.01521434634923935,
+            0.015909146517515182,
+            -0.041616760194301605,
+            0.008319789543747902,
+            0.016042763367295265,
+            0.021449735388159752,
+            0.004554060287773609,
+            0.009063581936061382,
+            0.012675653211772442,
+            -0.015294515527784824,
+            0.07097653299570084,
+            -0.0016189739108085632,
+            -0.009361989796161652,
+            0.010226036421954632,
+            0.018777426332235336,
+            0.010083513334393501,
+            -0.009317451156675816,
+            -0.010787221603095531,
+            0.019899794831871986,
+            0.012176821939647198,
+            -0.00043619878124445677,
+            -0.0028460093308240175,
+            0.00021837776876054704,
+            -0.021592257544398308,
+            -0.027720753103494644,
+            -0.013682222925126553,
+            0.0055272262543439865,
+            -0.006075049284845591,
+            -0.017797578126192093,
+            -0.005594033747911453,
+            0.03819620609283447,
+            0.0277029387652874,
+            0.03762611374258995,
+            -0.01240842230618,
+            0.03308318555355072,
+            -0.017494717612862587,
+            -0.006288834381848574,
+            -2.058165227936115e-05,
+            -0.006248749326914549,
+            -0.034597497433423996,
+            -0.005117471795529127,
+            0.001308317994698882,
+            0.005865718703716993,
+            -0.024068597704172134,
+            -0.04464537650346756,
+            0.02754260040819645,
+            -0.003171140095219016,
+            0.00048101565334945917,
+            0.02171696536242962,
+            0.0010444275103509426,
+            0.01798463985323906,
+            0.007918942719697952,
+            -0.02321345917880535,
+            -0.014279038645327091,
+            0.020505519583821297,
+            0.010386374779045582,
+            -0.006529341917484999,
+            0.025565091520547867,
+            0.06046544760465622,
+            -0.01011914387345314,
+            -0.024959366768598557,
+            -0.08444496989250183,
+            7.453237049048766e-05,
+            -0.0283442921936512,
+            -0.011490928940474987,
+            0.007825411856174469,
+            -0.004527336917817593,
+            -0.019062472507357597,
+            -0.0022169030271470547,
+            0.025547275319695473,
+            -0.005602941382676363,
+            -0.006342280190438032,
+            -0.017957916483283043,
+            0.03791115805506706,
+            -0.002776974579319358,
+            -0.004046321380883455,
+            0.030054569244384766,
+            0.0013561969390138984,
+            0.016888992860913277,
+            -0.0027858822140842676,
+            -0.02565416693687439,
+            0.029965493828058243,
+            -0.0067653958685696125,
+            0.08929075300693512,
+            -0.014564084820449352,
+            -0.009691574610769749,
+            -0.050453197211027145,
+            -0.011063359677791595,
+            -0.014083068817853928,
+            -0.003999555949121714,
+            -0.03424118831753731,
+            0.001817170181311667,
+            0.011588914319872856,
+            -0.014368114992976189,
+            0.013682222925126553,
+            0.021859489381313324,
+            0.03887319192290306,
+            -0.026990322396159172,
+            -0.04343393072485924,
+            0.06506182253360748,
+            0.021503180265426636,
+            -0.014804592356085777,
+            0.013824746012687683,
+            0.003099878551438451,
+            -0.01950785703957081,
+            0.011366221122443676,
+            -0.022393951192498207,
+            -0.013334822840988636,
+            0.017254209145903587,
+            -0.00778978131711483,
+            0.026152998208999634,
+            0.0797417014837265,
+            -0.018332039937376976,
+            0.014973838813602924,
+            0.022108905017375946,
+            -0.029217246919870377,
+            0.012372790835797787,
+            0.0028148323763161898,
+            -0.034989435225725174,
+            0.0058924416080117226,
+            0.008894335478544235,
+            0.05825633928179741,
+            0.003538582706823945,
+            0.010929744690656662,
+            0.010974282398819923,
+            0.011098990216851234,
+            -0.0009859707206487656,
+            0.023997336626052856,
+            0.008497943170368671,
+            0.0018372124759480357,
+            -0.026527121663093567,
+            0.01851019449532032,
+            -0.018813056871294975,
+            0.00869391206651926,
+            0.012167913839221,
+            -0.022661181166768074,
+            -0.012417329475283623,
+            -0.002204654971137643,
+            0.004934864118695259,
+            0.05184279754757881,
+            0.020024502649903297,
+            -0.010555621236562729,
+            -0.0979846715927124,
+            0.050132520496845245,
+            0.014145422726869583,
+            0.024923736229538918,
+            -0.04197306931018829,
+            0.017476901412010193,
+            0.009647035971283913,
+            0.028878754004836082,
+            0.0057677337899804115,
+            0.0020532242488116026,
+            -0.06816169619560242,
+            0.03545263409614563,
+            0.011419667862355709,
+            0.024994997307658195,
+            0.02485247515141964,
+            0.008840889669954777,
+            0.014065253548324108,
+            0.0004027949180454016,
+            0.04058346897363663,
+            0.029573554173111916,
+            -0.005389156751334667,
+            0.0357198640704155,
+            0.036236513406038284,
+            -0.0020799473859369755,
+            0.04428907111287117,
+            0.003500724909827113,
+            0.015089638531208038,
+            -0.006609511096030474,
+            -0.027328815311193466,
+            0.04963368922472,
+            -0.01895557902753353,
+            0.024157674983143806,
+            0.012069929391145706,
+            -0.03119475580751896,
+            -0.011125713586807251,
+            -0.01026166696101427,
+            0.028611524030566216,
+            -0.01782430149614811,
+            0.0019407644867897034,
+            0.02168133482336998,
+            0.009451067075133324,
+            -0.008163904771208763,
+            -0.026919061318039894,
+            -0.0066406880505383015,
+            -0.028967831283807755,
+            -0.04175928235054016,
+            0.0026010475121438503,
+            0.02702595293521881,
+            -0.012755822390317917,
+            -0.00048185072955675423,
+            -0.010742682963609695,
+            -0.030820632353425026,
+            -0.009932082146406174,
+            0.03388487920165062,
+            0.007901127450168133,
+            -0.011802698485553265,
+            -0.02971607819199562,
+            -0.014403746463358402,
+            0.01066251378506422,
+            -0.0017403413075953722,
+            0.05105891823768616,
+            -0.01533905416727066,
+            0.020648041740059853,
+            0.011437483131885529,
+            0.0065605188719928265,
+            0.02424675226211548,
+            0.02437146008014679,
+            -0.016559408977627754,
+            0.005843449383974075,
+            -0.003349294187501073,
+            -0.0055183181539177895,
+            0.043505191802978516,
+            -0.003075382439419627,
+            -0.029128169640898705,
+            -0.031586695462465286,
+            -0.012203545309603214,
+            0.010181497782468796,
+            -0.005081841256469488,
+            -0.01433248445391655,
+            0.02123595029115677,
+            0.02882530726492405,
+            0.04842223972082138,
+            0.01276472955942154,
+            -0.05725867673754692,
+            -0.013896007090806961,
+            0.0020732665434479713,
+            -0.02260773442685604,
+            0.032424017786979675,
+            0.060322925448417664,
+            -0.00506847957149148,
+            -0.020612411201000214,
+            -0.027150660753250122,
+            0.0659882202744484,
+            0.012328253127634525,
+            -0.04464537650346756,
+            0.04938427358865738,
+            -0.013655499555170536,
+            0.022340504452586174,
+            0.04517984017729759,
+            0.021859489381313324,
+            -0.0304286926984787,
+            0.017735224217176437,
+            0.024157674983143806,
+            -0.004836879670619965,
+            -0.06196194142103195,
+            -0.02260773442685604,
+            -0.002491928171366453,
+            -0.002779201604425907,
+            -0.013397176750004292,
+            -0.024585243314504623,
+            0.02485247515141964,
+            0.014537361450493336,
+            0.019098103046417236,
+            0.02120031975209713,
+            0.02324908971786499,
+            0.0164970550686121,
+            0.005259995348751545,
+            -0.007331034634262323,
+            -0.005153102800250053,
+            -0.023355981335043907,
+            -0.0269012451171875,
+            -0.02184167318046093,
+            -0.01982853375375271,
+            0.017574885860085487,
+            -0.04371897876262665,
+            0.016568316146731377,
+            0.0024251204449683428,
+            0.024389274418354034,
+            -0.023712290450930595,
+            0.004099767655134201,
+            0.028682785108685493,
+            -0.010591251775622368,
+            -0.04079725220799446,
+            0.025084074586629868,
+            0.011677990667521954,
+            -0.007241957820951939,
+            -0.007504734676331282,
+            0.0006981407641433179,
+            -0.008867612108588219,
+            0.031622324138879776,
+            -0.025280045345425606,
+            -0.01902684196829796,
+            -0.005375795066356659,
+            -0.02011357992887497,
+            0.0058924416080117226,
+            -0.03436589613556862,
+            -0.007464650087058544,
+            0.048600394278764725,
+            0.02132502757012844,
+            0.015098546631634235,
+            -0.04240063950419426,
+            -0.006631780415773392,
+            0.06142747774720192,
+            0.015392500907182693,
+            -0.07354194670915604,
+            0.044787902384996414,
+            0.0010861823102459311,
+            0.006698588375002146,
+            0.004685448482632637,
+            0.030054569244384766,
+            -0.015454854816198349,
+            -0.0159358698874712,
+            0.007673981133848429,
+            -0.030250540003180504,
+            -0.022091088816523552,
+            -0.005175372119992971,
+            0.021449735388159752,
+            -0.04521546885371208,
+            -0.03887319192290306,
+            -0.016791008412837982,
+            -0.0053802491165697575,
+            0.017129501327872276,
+            -0.011758159846067429,
+            -0.024923736229538918,
+            -0.030054569244384766,
+            -0.0005670305690728128,
+            -0.00188175099901855,
+            -0.04101103916764259,
+            -0.03067810833454132,
+            0.0068455650471150875,
+            0.021930750459432602,
+            0.02886093780398369,
+            0.0017392279114574194,
+            0.009459974244236946,
+            0.02504844404757023,
+            -0.00419552531093359,
+            0.029520107433199883,
+            -0.02766730636358261,
+            -0.032584354281425476,
+            0.012782545760273933,
+            -0.010911929421126842,
+            0.02116468921303749,
+            0.001672420185059309,
+            -0.021022165194153786,
+            -0.04689011722803116,
+            0.02706158347427845,
+            -0.0457855649292469,
+            -0.012087744660675526,
+            0.010974282398819923,
+            -0.0022937320172786713,
+            0.010288390330970287,
+            -0.00252533215098083,
+            0.0066451421007514,
+            0.02526222914457321,
+            -0.017601609230041504,
+            0.011909591034054756,
+            0.008012473583221436,
+            0.006306649651378393,
+            0.033991772681474686,
+            -0.008444496430456638,
+            -0.020505519583821297,
+            0.026242075487971306,
+            -0.03416992723941803,
+            0.01401180773973465,
+            0.0360405407845974,
+            0.013726761564612389,
+            0.010181497782468796,
+            0.011998668313026428,
+            0.011820513755083084,
+            0.0013862603809684515,
+            -0.02376573532819748,
+            0.01915154792368412,
+            0.01711168698966503,
+            -0.06866052746772766,
+            -0.020327365025877953,
+            -0.012702375650405884,
+            0.024068597704172134,
+            0.014412653632462025,
+            0.03340386599302292,
+            0.0033092093653976917,
+            -0.032459646463394165,
+            -0.025636352598667145,
+            -0.0075002810917794704,
+            0.04624876379966736,
+            -0.007437927182763815,
+            0.015962593257427216,
+            -0.00821289699524641,
+            0.021859489381313324,
+            0.012675653211772442,
+            -0.0016623989213258028,
+            0.026847798377275467,
+            -0.02597484551370144,
+            0.01240842230618,
+            -0.018688349053263664,
+            -0.04375460743904114,
+            -0.009771743789315224,
+            0.02360539697110653,
+            0.01886650174856186,
+            0.00418661767616868,
+            0.0078075965866446495,
+            -0.01634562388062477,
+            -0.0024986090138554573,
+            -0.03823183476924896,
+            0.010448728688061237,
+            0.027649492025375366,
+            0.0016223143320530653,
+            -0.005638572387397289,
+            -0.0038815289735794067,
+            0.0159358698874712,
+            0.0004142079269513488,
+            0.007469104137271643,
+            -0.034918174147605896,
+            0.040334053337574005,
+            0.0054158796556293964,
+            0.04582119360566139,
+            -0.04724642634391785,
+            -0.017450178042054176,
+            -0.0038904366083443165,
+            -0.007380026858299971,
+            0.042863838374614716,
+            -0.03573768213391304,
+            -0.02830866165459156,
+            -0.011464206501841545,
+            0.01947222650051117,
+            0.015668639913201332,
+            -0.02155662700533867,
+            -0.03976396098732948,
+            -0.002783655421808362,
+            -0.035274479538202286,
+            -0.010947559960186481,
+            -0.014412653632462025,
+            -0.0032802594359964132,
+            -0.049847472459077835,
+            -0.0017592702060937881,
+            -0.0835898295044899,
+            0.005081841256469488,
+            0.03755485266447067,
+            -0.009807374328374863,
+            0.04111792892217636,
+            -0.002641132101416588,
+            0.002984078601002693,
+            -0.047959040850400925,
+            0.01689790189266205,
+            0.026242075487971306,
+            0.04678322374820709,
+            -0.022180166095495224,
+            0.005242179613560438,
+            0.014679884538054466,
+            -0.011214790865778923,
+            -0.017610516399145126,
+            -0.010511082597076893,
+            0.02004231885075569,
+            -0.0051976414397358894,
+            0.0012392833596095443,
+            0.04528673365712166,
+            0.029537923634052277,
+            0.014697699807584286,
+            0.031943000853061676,
+            0.00030648044776171446,
+            -0.01814497821033001,
+            0.0025320129934698343,
+            0.043148886412382126,
+            0.04093977436423302,
+            -0.04375460743904114,
+            0.015543931163847446,
+            -0.03905134275555611,
+            0.025636352598667145,
+            0.013985084369778633,
+            0.007442380767315626,
+            0.03343949466943741,
+            0.006466988008469343,
+            -0.005803364794701338,
+            0.00020348520774859935,
+            0.024674320593476295,
+            -0.01662176288664341,
+            0.017922285944223404,
+            -0.004836879670619965,
+            -0.022661181166768074,
+            0.000857365841511637,
+            -0.015232161618769169,
+            0.017191855236887932,
+            0.02187730371952057,
+            -0.007188511546701193,
+            -0.0010806149803102016,
+            -0.0003092641127295792,
+            0.021948564797639847,
+            0.004395948722958565,
+            -0.0005812272429466248,
+            0.011428575031459332,
+            0.03500724956393242,
+            -0.03215678781270981,
+            -0.010644698515534401,
+            0.03212115541100502,
+            -0.033101003617048264,
+            0.009566866792738438,
+            -0.04104666784405708,
+            0.010609067045152187,
+            0.0022324915044009686,
+            -0.020986534655094147,
+            0.04076162353157997,
+            -0.010911929421126842,
+            0.001672420185059309,
+            -0.013317007571458817,
+            0.00856920424848795,
+            0.015561746433377266,
+            0.03762611374258995,
+            -0.0039015712682157755,
+            0.02882530726492405,
+            0.01489366963505745,
+            -0.020220471546053886,
+            0.04008463770151138,
+            -0.04268568381667137,
+            -0.010208221152424812,
+            0.015249977819621563,
+            -0.005981518421322107,
+            -0.012461868114769459,
+            -0.020576780661940575,
+            -0.021663518622517586,
+            -0.007001449819654226,
+            0.015143085271120071,
+            -0.02686561457812786,
+            0.006760942284017801,
+            0.006801026873290539,
+            -0.017842117697000504,
+            -0.007651711814105511,
+            0.006609511096030474,
+            -0.008016927167773247,
+            -0.0056296647526323795,
+            0.028201768174767494,
+            -0.017975732684135437,
+            0.01260439120233059,
+            0.02321345917880535,
+            0.028843123465776443,
+            -0.018162794411182404,
+            -0.0016189739108085632,
+            -0.0032490824814885855,
+            -0.0058479029685258865,
+            -0.03285158798098564,
+            0.026509307324886322,
+            0.009620312601327896,
+            -0.010056789964437485,
+            -0.024709951132535934,
+            -0.008836435154080391,
+            0.01879524067044258,
+            -0.0035185404121875763,
+            -0.005549495108425617,
+            -0.021984197199344635,
+            -0.02878967672586441,
+            0.0257788747549057,
+            0.01863490231335163,
+            -0.046711962670087814,
+            -0.0032112249173223972,
+            0.005411426071077585,
+            0.004783433396369219,
+            0.0027213015127927065,
+            -0.014849130995571613,
+            0.007388934958726168,
+            0.026972506195306778,
+            0.0054826876148581505,
+            -0.015543931163847446,
+            -0.012622206471860409,
+            -0.005099656525999308,
+            0.008613742887973785,
+            0.0069925421848893166,
+            0.009932082146406174,
+            0.02975170873105526,
+            0.04072599112987518,
+            0.0018628222169354558,
+            -0.01288943737745285,
+            -0.02561853639781475,
+            0.025315675884485245,
+            -0.0019630338065326214,
+            0.04165239259600639,
+            0.04564303904771805,
+            0.0025876860599964857,
+            -0.060572341084480286,
+            0.012479683384299278,
+            -0.02517315186560154,
+            -0.009843004867434502,
+            0.0066406880505383015,
+            0.0036209789104759693,
+            -0.04660507291555405,
+            -0.0031956364400684834,
+            0.0022536474280059338,
+            0.034152109175920486,
+            -0.007299857679754496,
+            -0.04436033219099045,
+            -0.008929966017603874,
+            -0.007406750228255987,
+            0.02574324421584606,
+            0.010110235773026943,
+            -0.007892219349741936,
+            -0.015294515527784824,
+            0.013361545279622078,
+            -0.0026567205786705017,
+            -0.006190849468111992,
+            0.017013700678944588,
+            -0.007691796403378248,
+            0.011170252226293087,
+            -0.012096652761101723,
+            -0.019739456474781036,
+            -0.017655055969953537,
+            0.03393832594156265,
+            -0.02706158347427845,
+            0.025565091520547867,
+            0.014947115443646908,
+            0.0026923513505607843,
+            0.039407651871442795,
+            -0.01535686943680048,
+            -0.027364445850253105,
+            -0.002897228579968214,
+            -0.0037078289315104485,
+            -0.03766174241900444,
+            0.01815388724207878,
+            -0.023516319692134857,
+            -0.022091088816523552,
+            0.013495161198079586,
+            0.00951342098414898,
+            0.0002751643187366426,
+            -0.009308543056249619,
+            -0.00851130485534668,
+            -0.0015465988544747233,
+            0.01956130377948284,
+            0.019258441403508186,
+            -0.007451288867741823,
+            0.043505191802978516,
+            -0.0004167132137808949,
+            0.0038369905669242144,
+            0.0005995993851684034,
+            0.018100440502166748,
+            0.060287293046712875,
+            0.020452072843909264,
+            0.0012481911107897758,
+            0.030482139438390732,
+            0.022500842809677124,
+            0.037162911146879196,
+            -0.03019709326326847,
+            -0.04564303904771805,
+            0.017895564436912537,
+            -0.007250865455716848,
+            0.009522328153252602,
+            0.022429581731557846,
+            0.0011813833843916655,
+            0.01418996136635542,
+            0.007063803728669882,
+            -0.010243851691484451,
+            0.023801365867257118,
+            -0.032905034720897675,
+            -0.02789890766143799,
+            0.027115030214190483,
+            0.010056789964437485,
+            0.019971057772636414,
+            0.0360405407845974,
+            -0.024870289489626884,
+            -0.033671095967292786,
+            0.004881417844444513,
+            0.006680772639811039,
+            -0.0283442921936512,
+            0.012942884117364883,
+            -0.013521884568035603,
+            0.015160900540649891,
+            0.0064001805149018764,
+            -0.012648929841816425,
+            -0.0027279823552817106,
+            0.013780207373201847,
+            -0.0141721460968256,
+            0.008698366582393646,
+            -0.015160900540649891,
+            -0.012452960945665836,
+            0.010626882314682007,
+            -0.030642477795481682,
+            0.0008445610292255878,
+            0.00507738720625639,
+            -0.0015131949912756681,
+            0.006230934057384729,
+            -0.0067297653295099735,
+            -0.004431579262018204,
+            -0.005865718703716993,
+            -0.03185392543673515,
+            0.02795235440135002,
+            0.02766730636358261,
+            -0.025119705125689507,
+            -0.0030553401447832584,
+            -0.0036900134291499853,
+            -0.014528454281389713,
+            -0.006346734240651131,
+            0.010876297950744629,
+            -0.014448285102844238,
+            0.0021411876659840345,
+            -0.016087301075458527,
+            -0.018367672339081764,
+            0.023961706086993217,
+            0.022518659010529518,
+            0.042863838374614716,
+            -0.0010037861065939069,
+            0.022750258445739746,
+            0.021307211369276047,
+            0.0011028843000531197,
+            0.0017982413992285728,
+            0.017494717612862587,
+            -0.017557071521878242,
+            0.028985645622015,
+            0.025707613676786423,
+            0.03099878504872322,
+            0.010849575512111187,
+            -0.03221023455262184,
+            -0.0024674320593476295,
+            0.018848687410354614,
+            0.017076054587960243,
+            0.0008317562169395387,
+            -0.013486253097653389,
+            -0.009771743789315224,
+            0.012533130124211311,
+            -0.02152099646627903,
+            -0.0065248883329331875,
+            0.013512976467609406,
+            -0.016514871269464493,
+            0.0032802594359964132,
+            -0.007825411856174469,
+            0.02472776733338833,
+            0.0028838668949902058,
+            -0.005193187389522791,
+            0.012773637659847736,
+            -0.00048185072955675423,
+            0.005237726029008627,
+            -0.005037302616983652,
+            0.02324908971786499,
+            0.03493598848581314,
+            0.012684560380876064,
+            0.04407528415322304,
+            0.01998887211084366,
+            -0.013423899188637733,
+            -0.009531236253678799,
+            -0.020612411201000214,
+            0.022358320653438568,
+            -0.030286170542240143,
+            -0.024709951132535934,
+            -0.022073272615671158,
+            0.01535686943680048,
+            0.00679657282307744,
+            0.015606285072863102,
+            -0.0033893787767738104,
+            -0.004447167739272118,
+            0.031372908502817154,
+            0.004179936833679676,
+            0.025511644780635834,
+            0.006907918956130743,
+            -0.007905581034719944,
+            -0.004012917634099722,
+            0.0024139860179275274,
+            0.008150543086230755,
+            -0.015241069719195366,
+            0.03826746717095375,
+            -0.0035630788188427687,
+            0.04813719540834427,
+            -0.052733566612005234,
+            -0.002003118395805359,
+            0.00635564187541604,
+            -0.01918718032538891,
+            -0.005669749341905117,
+            -0.03915823623538017,
+            0.02030954882502556,
+            -0.03377798572182655,
+            0.026669645681977272,
+            -0.02027391828596592,
+            0.026491491124033928,
+            -0.029929861426353455,
+            0.003576440503820777,
+            -0.0015243296511471272,
+            -0.0038035865873098373,
+            0.0164970550686121,
+            0.024282382801175117,
+            -0.002409531967714429,
+            -0.02590358257293701,
+            -0.0008083735010586679,
+            0.0037635019980371,
+            0.0066273268312215805,
+            -0.013717853464186192,
+            0.0054158796556293964,
+            -0.0012582122581079602,
+            -0.029680447652935982,
+            -0.033101003617048264,
+            0.012132283300161362,
+            -0.010911929421126842,
+            0.0337958037853241,
+            0.030375247821211815,
+            -0.026794353500008583,
+            -0.027043769136071205,
+            0.0034561865031719208,
+            -0.008676096796989441,
+            -0.009090305306017399,
+            0.020737119019031525,
+            0.04532236233353615,
+            0.00925509724766016,
+            -0.020790565758943558,
+            0.023943889886140823,
+            0.004734441172331572,
+            0.008582565933465958,
+            0.004422671627253294,
+            -0.021734781563282013,
+            -0.018492378294467926,
+            -0.01689790189266205,
+            -0.01726311631500721,
+            0.008177265524864197,
+            -0.01972164213657379,
+            -0.04012026637792587,
+            -0.04742458090186119,
+            -0.011179160326719284,
+            0.019383149221539497,
+            0.004144306294620037,
+            0.05052445828914642,
+            0.012452960945665836,
+            0.02478121407330036,
+            -0.0017191855004057288,
+            -0.005469325929880142,
+            0.04646254703402519,
+            -0.019899794831871986,
+            0.051700275391340256,
+            0.009700481779873371,
+            -0.0021634569857269526,
+            -0.01578443869948387,
+            0.03188955411314964,
+            -0.024905921891331673,
+            -0.0019240626133978367,
+            0.006342280190438032,
+            -0.0034984981175512075,
+            0.017298748716711998,
+            0.008644919842481613,
+            0.010128051973879337,
+            -0.02360539697110653,
+            0.028130507096648216,
+            -0.023641027510166168,
+            0.014590808190405369,
+            0.015926962718367577,
+            -0.002009799238294363,
+            0.0036098442506045103,
+            -0.0054826876148581505,
+            0.004574102349579334,
+            -0.002970716916024685,
+            0.011597821488976479,
+            -0.02510189078748226,
+            0.015009469352662563,
+            -0.02228705771267414,
+            -0.013317007571458817,
+            0.0004178266681265086,
+            -0.030945340171456337,
+            0.012399514205753803,
+            0.04108230024576187,
+            0.01037746760994196,
+            -0.02232268825173378,
+            0.005371341481804848,
+            -0.051450859755277634,
+            -0.008199535310268402,
+            0.0014809046406298876,
+            0.01271128375083208,
+            0.01346843782812357,
+            0.0007504734676331282,
+            0.009727205149829388,
+            -0.00955795869231224,
+            -0.03055340051651001,
+            0.033421680331230164,
+            -0.001038860180415213,
+            -0.0020298415329307318,
+            0.003928294405341148,
+            0.01851019449532032,
+            0.0025854590348899364,
+            0.004188844468444586,
+            0.012773637659847736,
+            -0.007709611672908068,
+            -0.03224586322903633,
+            -0.021449735388159752,
+            -0.0038926636334508657,
+            0.0019318568520247936,
+            -0.003694467479363084,
+            0.003318117232993245,
+            0.0055851261131465435,
+            -0.015535023994743824,
+            0.01082285214215517,
+            -0.011980853043496609,
+            0.010707051493227482,
+            -0.019169364124536514,
+            -0.004329140763729811,
+            0.006658503785729408,
+            -0.020808380097150803,
+            -0.018296409398317337,
+            -0.051450859755277634,
+            -0.010466543957591057,
+            0.002101103076711297,
+            0.021146873012185097,
+            0.006676319055259228,
+            0.008489035069942474,
+            -0.014394838362932205,
+            0.0068322038277983665,
+            0.003656609682366252,
+            0.02520878240466118,
+            -0.026829984039068222,
+            0.03119475580751896,
+            0.018723979592323303,
+            0.016951346769928932,
+            -0.03862377628684044,
+            0.03249527886509895,
+            0.016443608328700066,
+            -0.003759048180654645,
+            0.017521440982818604,
+            0.014002899639308453,
+            0.04753147065639496,
+            -0.030606847256422043,
+            0.004810156300663948,
+            -0.02754260040819645,
+            0.025084074586629868,
+            -0.03805368393659592,
+            0.028237400576472282,
+            0.0009163793292827904,
+            0.023943889886140823,
+            0.04428907111287117,
+            9.728875738801435e-05,
+            0.03851688280701637,
+            -0.022019827738404274,
+            -0.009045766666531563,
+            0.025636352598667145,
+            -0.017476901412010193,
+            0.015882423147559166,
+            0.00778978131711483,
+            -0.024282382801175117,
+            0.01838548667728901,
+            0.007709611672908068,
+            -0.0031956364400684834,
+            -0.04788777977228165,
+            0.02795235440135002,
+            0.02260773442685604,
+            0.02120031975209713,
+            0.0067297653295099735,
+            -0.012150098569691181,
+            -0.010234943591058254,
+            0.005611849017441273,
+            -0.009246189147233963,
+            0.016755377873778343,
+            -0.015846792608499527,
+            -0.030909709632396698,
+            0.00849348958581686,
+            -0.01927625574171543,
+            0.012648929841816425,
+            -0.0038904366083443165,
+            0.029092539101839066,
+            -0.017770856618881226,
+            -0.01068032905459404,
+            0.01101882103830576,
+            -0.0028237400110810995,
+            -0.016069484874606133,
+            -0.04842223972082138,
+            -0.038801927119493484,
+            0.008453404530882835,
+            -0.0013105449033901095,
+            -0.020184841006994247,
+            0.011909591034054756,
+            -0.010591251775622368,
+            -0.004792341031134129,
+            -0.01972164213657379,
+            -0.010742682963609695,
+            -0.023035304620862007,
+            -0.006760942284017801,
+            0.010074605233967304,
+            0.026277706027030945,
+            -0.018848687410354614,
+            0.03798241913318634,
+            0.029413215816020966,
+            0.013985084369778633,
+            -0.012506406754255295,
+            0.0257788747549057,
+            -0.009353081695735455,
+            -0.01489366963505745,
+            0.033671095967292786,
+            -0.0031466439832001925,
+            -0.009192743338644505,
+            0.00017439600196667016,
+            -0.024799028411507607,
+            -0.023070935159921646,
+            0.019579118117690086,
+            -0.022393951192498207,
+            -0.025280045345425606,
+            -0.004921502433717251,
+            -0.006141857244074345,
+            -0.02975170873105526,
+            -0.04753147065639496,
+            -0.011214790865778923,
+            0.03645029664039612,
+            0.0057632802054286,
+            0.018492378294467926,
+            0.01711168698966503,
+            -0.012230267748236656,
+            -0.03039306215941906,
+            0.004721079487353563,
+            -0.012105559930205345,
+            -0.008738450706005096,
+            -0.01287162210792303,
+            0.007700704038143158,
+            0.016951346769928932,
+            0.01101882103830576,
+            0.0011669083032757044,
+            -0.0052778106182813644,
+            -0.02123595029115677,
+            -0.007861042395234108,
+            -0.0176995936781168,
+            -0.028985645622015,
+            -0.0031466439832001925,
+            -0.033029742538928986,
+            0.03823183476924896,
+            0.018777426332235336,
+            0.005273356568068266,
+            -0.01982853375375271,
+            -0.012720191851258278,
+            -0.020790565758943558,
+            0.00927291251718998,
+            -0.00707716541364789,
+            0.009718297980725765,
+            -0.004614186938852072,
+            0.01726311631500721,
+            -0.006057234015315771,
+            0.00047684015589766204,
+            -0.04436033219099045,
+            0.009023496881127357,
+            0.01934751868247986,
+            0.003930521197617054,
+            -0.009878636337816715,
+            0.008880973793566227,
+            0.011606729589402676,
+            -0.021022165194153786,
+            -0.0236944742500782,
+            0.019222810864448547,
+            0.020576780661940575,
+            0.015909146517515182,
+            -0.009299635887145996,
+            -0.009299635887145996,
+            0.007130611687898636,
+            0.0007682888535782695,
+            -0.006315557286143303,
+            0.016550501808524132,
+            0.023623213171958923,
+            0.01375348400324583,
+            -0.00018177268793806434,
+            0.026206444948911667,
+            0.009174928069114685,
+            -0.012310437858104706,
+            -0.012114468030631542,
+            -0.007620534859597683,
+            0.0023694473784416914,
+            -0.01670193299651146,
+            -0.011989760212600231,
+            -0.010083513334393501,
+            0.025226598605513573,
+            0.009370896965265274,
+            0.011294960044324398,
+            -0.003090970916673541,
+            -0.006173034198582172,
+            0.005411426071077585,
+            0.009949897415935993,
+            0.0001479512866353616,
+            -0.001145752496086061,
+            0.007219688501209021,
+            -0.038766298443078995,
+            -0.028682785108685493,
+            0.0014196642441675067,
+            -0.002750251442193985,
+            0.0033515209797769785,
+            0.03506069630384445,
+            0.027596045285463333,
+            -0.007014811504632235,
+            -0.01918718032538891,
+            -0.0018227375112473965,
+            0.024282382801175117,
+            0.0001636789384065196,
+            -0.02308875136077404,
+            -0.009477789513766766,
+            0.009090305306017399,
+            0.00032874970929697156,
+            0.02927069365978241,
+            0.005700926296412945,
+            -0.008235165849328041,
+            0.019383149221539497,
+            0.10076387226581573,
+            0.007776419632136822,
+            0.006885650102049112,
+            -0.012150098569691181,
+            -0.031016601249575615,
+            0.0021033298689872026,
+            0.021699151024222374,
+            -0.019080286845564842,
+            0.04485916346311569,
+            0.011553282849490643,
+            0.013263560831546783,
+            0.018973395228385925,
+            -0.0021144645288586617,
+            -0.010974282398819923,
+            -0.016131838783621788,
+            0.030606847256422043,
+            0.0016323354793712497,
+            0.02184167318046093,
+            -0.020897457376122475,
+            0.0017414548201486468,
+            0.004591918084770441,
+            0.012800361029803753,
+            0.00606168806552887,
+            -0.005055117886513472,
+            -0.02510189078748226,
+            0.018777426332235336,
+            -0.018180610612034798,
+            0.007224142551422119,
+            -0.008858704939484596,
+            -0.022874966263771057,
+            0.008987866342067719,
+            -0.00014502844715025276,
+            -0.01244405284523964,
+            0.005513864569365978,
+            -0.014109792187809944,
+            0.0326734334230423,
+            -0.012738007120788097,
+            0.0015354643110185862,
+            0.003663290524855256,
+            -0.016390163451433182,
+            0.01202539075165987,
+            0.01746799424290657,
+            -0.01200757548213005,
+            0.030090201646089554,
+            -0.016710840165615082,
+            -0.004318006336688995,
+            0.012061022222042084,
+            -0.018002456054091454,
+            0.011214790865778923,
+            0.028486816212534904,
+            -0.009335266426205635,
+            0.005683110561221838,
+            -0.0075581809505820274,
+            0.009905358776450157,
+            0.009335266426205635,
+            0.026402413845062256,
+            0.023142196238040924,
+            -0.0005219354061409831,
+            0.004251198377460241,
+            -0.008880973793566227,
+            -0.0019708280451595783,
+            -0.0009464428294450045,
+            0.034312449395656586,
+            -0.018162794411182404,
+            -0.0043246871791779995,
+            -0.009326359257102013,
+            0.0006680772639811039,
+            -0.014724423177540302,
+            -0.002120031975209713,
+            -0.007437927182763815,
+            -0.0019351971568539739,
+            0.0076606194488704205,
+            -0.00036493720836006105,
+            0.016158562153577805,
+            -0.008257435634732246,
+            -0.004556287080049515,
+            0.004997218027710915,
+            -0.005224364344030619,
+            0.01657722517848015,
+            -0.02574324421584606,
+            0.0037612752057611942,
+            0.018456747755408287,
+            0.00869391206651926,
+            0.011508745141327381,
+            -0.01642579399049282,
+            0.0025854590348899364,
+            -0.002992986235767603,
+            -0.012577668763697147,
+            0.012577668763697147,
+            -0.01831422559916973,
+            -0.011499837040901184,
+            -0.021129056811332703,
+            -0.0297873392701149,
+            -0.002863824600353837,
+            -0.01851019449532032,
+            0.008146088570356369,
+            0.01918718032538891,
+            0.028647154569625854,
+            0.00018024168093688786,
+            -0.004970495123416185,
+            -0.01743236370384693,
+            0.015962593257427216,
+            0.045037318021059036,
+            -0.016292177140712738,
+            -0.02520878240466118,
+            -0.009914266876876354,
+            0.0127290990203619,
+            -0.008622650988399982,
+            -0.026046106591820717,
+            -0.003943882882595062,
+            0.00577218784019351,
+            -0.02911035344004631,
+            0.009379805065691471,
+            0.0005712060956284404,
+            0.024442721158266068,
+            0.004672087263315916,
+            0.020665857940912247,
+            -0.017521440982818604,
+            -0.0019708280451595783,
+            0.025476014241576195,
+            -0.029520107433199883,
+            0.0045874640345573425,
+            0.01802917942404747,
+            -0.01346843782812357,
+            0.010769405402243137,
+            0.030820632353425026,
+            0.025226598605513573,
+            -0.01963256485760212,
+            0.024335827678442,
+            0.014626438729465008,
+            -0.008720635436475277,
+            0.003284713253378868,
+            0.011660175397992134,
+            0.012622206471860409,
+            0.006698588375002146,
+            -0.0034584132954478264,
+            -0.005861264653503895,
+            0.009045766666531563,
+            0.031337279826402664,
+            0.0016623989213258028,
+            -0.012657837942242622,
+            0.028415553271770477,
+            0.02497718296945095,
+            -0.005429241340607405,
+            0.008426681160926819,
+            -0.02039862610399723,
+            -0.016443608328700066,
+            0.01730765588581562,
+            0.022465212270617485,
+            0.005910256877541542,
+            0.0032401748467236757,
+            -0.011366221122443676,
+            0.022839335724711418,
+            -0.008377688936889172,
+            0.01154437568038702,
+            -0.027168476954102516,
+            -0.002518651308491826,
+            -0.0036900134291499853,
+            0.02068367227911949,
+            -0.0008106004097498953,
+            -0.005032849032431841,
+            0.019062472507357597,
+            -0.01228371448814869,
+            -0.019775087013840675,
+            -0.006462534423917532,
+            -0.018118256703019142,
+            0.017637239769101143,
+            -0.004774525761604309,
+            -0.02030954882502556,
+            0.007549273315817118,
+            0.0005678656743839383,
+            -0.009041312150657177,
+            0.006801026873290539,
+            0.004144306294620037,
+            0.008057012222707272,
+            -0.02264336682856083,
+            0.010751590132713318,
+            -0.005464872345328331,
+            0.007901127450168133,
+            -0.008382143452763557,
+            0.014368114992976189,
+            -0.006738672964274883,
+            0.0015599604230374098,
+            0.0006012695375829935,
+            -0.006208664737641811,
+            0.000636900367680937,
+            0.013210115022957325,
+            0.0032624441664665937,
+            -0.016550501808524132,
+            0.015312331728637218,
+            -0.008662735112011433,
+            -0.004925956483930349,
+            -0.005028394982218742,
+            -0.004106448497623205,
+            -0.0011357313487678766,
+            0.012738007120788097,
+            0.00865382794290781,
+            -0.031123492866754532,
+            -0.006689680740237236,
+            -0.024959366768598557,
+            -0.009326359257102013,
+            0.010145867243409157,
+            0.012052114121615887,
+            -0.020576780661940575,
+            0.009001228027045727,
+            -0.027079399675130844,
+            -0.01226589921861887,
+            0.0269012451171875,
+            0.023961706086993217,
+            -0.02068367227911949,
+            0.012738007120788097,
+            -0.014804592356085777,
+            -0.02014921046793461,
+            0.015009469352662563,
+            0.021467549726366997,
+            0.01224808394908905,
+            0.005028394982218742,
+            -0.04400402307510376,
+            -0.010920836590230465,
+            0.019864164292812347,
+            -0.03140854090452194,
+            -0.009504512883722782,
+            0.027417892590165138,
+            -0.006827749777585268,
+            -0.01726311631500721,
+            -0.003516313387081027,
+            0.03522103279829025,
+            -0.02123595029115677,
+            0.015526115894317627,
+            0.011428575031459332,
+            -0.02152099646627903,
+            0.014760053716599941,
+            -0.012853806838393211,
+            0.008618197403848171,
+            0.0019296299433335662,
+            -0.022536473348736763,
+            -0.0006502618780359626,
+            0.018011363223195076,
+            -0.0005444829585030675,
+            -0.004050775431096554,
+            0.011455298401415348,
+            0.016639579087495804,
+            -0.018421117216348648,
+            0.013922730460762978,
+            0.006088410969823599,
+            9.90285407169722e-05,
+            0.016283269971609116,
+            -0.02052333392202854,
+            -0.01810934767127037,
+            0.01979290321469307,
+            -0.004179936833679676,
+            -0.02421111986041069,
+            0.02344505861401558,
+            -0.02376573532819748,
+            -0.020950904116034508,
+            -0.02340942807495594,
+            0.00032652277150191367,
+            0.00849348958581686,
+            -0.01734328642487526,
+            -0.026562752202153206,
+            0.007233050186187029,
+            0.01950785703957081,
+            0.04689011722803116,
+            0.018038086593151093,
+            -0.0023048666771501303,
+            0.023819182068109512,
+            -0.016595039516687393,
+            0.013263560831546783,
+            0.012105559930205345,
+            -0.0036543826572597027,
+            -0.015757715329527855,
+            0.0025743243750184774,
+            0.009602497331798077,
+            0.007152881007641554,
+            0.0030263899825513363,
+            -0.022251427173614502,
+            -0.02120031975209713,
+            -0.04101103916764259,
+            0.00957577396184206,
+            -0.007771965581923723,
+            0.0008968937327153981,
+            0.05790003016591072,
+            0.00126711989287287,
+            -0.033350419253110886,
+            0.01754816249012947,
+            0.00047767526120878756,
+            0.0007610513712279499,
+            -0.011633452028036118,
+            0.002986305393278599,
+            -0.02389044314622879,
+            -0.013949453830718994,
+            -0.0353991873562336,
+            -0.0036877866368740797,
+            0.01027948223054409,
+            0.005162010435014963,
+            0.005700926296412945,
+            -0.02805924601852894,
+            -0.002245853189378977,
+            -0.025867952033877373,
+            0.022304873913526535,
+            0.006475895643234253,
+            -0.0001588075392646715,
+            0.0023649935610592365,
+            0.027400076389312744,
+            -0.008622650988399982,
+            -0.01854582503437996,
+            -0.020790565758943558,
+            0.0011902910191565752,
+            -0.004248971585184336,
+            0.001448614289984107,
+            0.0009458860731683671,
+            -0.025832321494817734,
+            0.009486697614192963,
+            0.00028532466967590153,
+            -0.012595484033226967,
+            0.014100884087383747,
+            -0.013352638110518456,
+            -0.007001449819654226,
+            0.009157112799584866,
+            0.023160012438893318,
+            -0.027524784207344055,
+            -0.008506850339472294,
+            -0.024870289489626884,
+            0.009985528886318207,
+            0.012675653211772442,
+            0.0029484478291124105,
+            -0.019775087013840675,
+            -0.012925068847835064,
+            -0.02574324421584606,
+            -0.009851912967860699,
+            -0.02059459500014782,
+            -0.008012473583221436,
+            0.005041756667196751,
+            0.01657722517848015,
+            0.03244183212518692,
+            0.016799917444586754,
+            -0.005157556850463152,
+            0.0017548163887113333,
+            -0.002420666627585888,
+            0.008751812390983105,
+            -0.020060133188962936,
+            2.0077113731531426e-05,
+            -0.0159358698874712,
+            0.017957916483283043,
+            0.00999443605542183,
+            -0.013887099921703339,
+            -0.017761947587132454,
+            -0.02276807464659214,
+            0.019525671377778053,
+            -0.022429581731557846,
+            -0.017886655405163765,
+            0.01918718032538891,
+            -0.011036636307835579,
+            0.006409088149666786,
+            -0.013691130094230175,
+            0.037198543548583984,
+            0.010021159425377846,
+            0.025476014241576195,
+            -0.016612855717539787,
+            -0.005567310843616724,
+            -0.0272219218313694,
+            -0.004400402307510376,
+            -0.01682663895189762,
+            -0.00434695603325963,
+            -0.0012337160296738148,
+            0.009753928519785404,
+            -0.02526222914457321,
+            -0.005567310843616724,
+            0.02321345917880535,
+            -0.027560414746403694,
+            -0.0022124492097646,
+            -0.0021757050417363644,
+            0.006337826605886221,
+            0.007121704053133726,
+            0.0014397065388038754,
+            -0.002219130052253604,
+            -0.022180166095495224,
+            -0.008448950946331024,
+            -0.028807492926716805,
+            0.022091088816523552,
+            -0.010315113700926304,
+            -0.014653162099421024,
+            0.0001746743801049888,
+            0.021182503551244736,
+            0.0046899025328457355,
+            -0.010172589682042599,
+            0.0017403413075953722,
+            -0.007598265539854765,
+            0.0021645703818649054,
+            0.0038147212471812963,
+            0.012096652761101723,
+            -0.0020988760516047478,
+            0.014083068817853928,
+            0.02020265720784664,
+            -0.008560297079384327,
+            -0.0006975840078666806,
+            -0.0058122724294662476,
+            0.0037790904752910137,
+            0.011375129222869873,
+            0.01037746760994196,
+            0.016203101724386215,
+            0.003641021205112338,
+            -0.009486697614192963,
+            0.01670193299651146,
+            -0.014083068817853928,
+            0.02184167318046093,
+            -0.001927402918227017,
+            -0.01943659596145153,
+            0.002552055288106203,
+            -0.047923412173986435,
+            0.0009069149382412434,
+            -0.012310437858104706,
+            -0.022750258445739746,
+            0.006293287966400385,
+            -0.011838329024612904,
+            -0.008190627209842205,
+            0.019098103046417236,
+            -0.009103666059672832,
+            -0.02219798043370247,
+            0.011571098119020462,
+            0.0027591590769588947,
+            -0.009036858566105366,
+            0.006613965146243572,
+            -0.0016679662512615323,
+            0.008658281527459621,
+            0.004482798743993044,
+            0.005366887431591749,
+            0.001175816054455936,
+            -0.01565973088145256,
+            -0.018563641235232353,
+            0.020095765590667725,
+            0.008333150297403336,
+            -0.0016278816619887948,
+            0.010003344155848026,
+            0.002652266761288047,
+            0.011428575031459332,
+            -0.003694467479363084,
+            -0.004810156300663948,
+            0.00019415996212046593,
+            -0.014679884538054466,
+            0.014145422726869583,
+            -0.013085407204926014,
+            -0.032228048890829086,
+            0.013664407655596733,
+            0.006008241791278124,
+            0.00681438809260726,
+            -0.008119366131722927,
+            -0.011891775764524937,
+            -0.007161788642406464,
+            -0.012078837491571903,
+            0.007455742452293634,
+            0.015294515527784824,
+            -0.0038147212471812963,
+            -0.0008879860397428274,
+            -0.01418996136635542,
+            0.007575996220111847,
+            0.020184841006994247,
+            0.029965493828058243,
+            -0.02488810569047928,
+            0.00724641140550375,
+            -0.005344618111848831,
+            -0.011873960494995117,
+            -0.020291734486818314,
+            0.024264566600322723,
+            0.01457299292087555,
+            0.01626545563340187,
+            0.004206659737974405,
+            0.0036165250930935144,
+            0.000802249473053962,
+            -0.019365333020687103,
+            0.003191182389855385,
+            0.008137181401252747,
+            -0.013317007571458817,
+            -0.0020688127260655165,
+            0.02478121407330036,
+            -0.013780207373201847,
+            -0.0013484027003869414,
+            0.001965260598808527,
+            0.024032967165112495,
+            -0.0138514693826437,
+            0.001873956760391593,
+            0.004729987122118473,
+            0.0006508186343125999,
+            0.005522772204130888,
+            0.013219022192060947,
+            0.07496717572212219,
+            0.019579118117690086,
+            0.005393610801547766,
+            0.007446834817528725,
+            -0.006311103235930204,
+            -0.006008241791278124,
+            -0.009264005348086357,
+            -0.011553282849490643,
+            0.002538693603128195,
+            -0.0017325470689684153,
+            -0.02308875136077404,
+            -0.002113351132720709,
+            -0.005825634114444256,
+            0.00045429254532791674,
+            -0.0012982968473806977,
+            0.01734328642487526,
+            -0.012666745111346245,
+            0.013913823291659355,
+            0.03204989433288574,
+            0.007188511546701193,
+            -0.021770412102341652,
+            0.013504069298505783,
+            0.0025854590348899364,
+            -0.006484803277999163,
+            -0.003402740228921175,
+            -0.006609511096030474,
+            0.006996996235102415,
+            0.011669083498418331,
+            -0.010600159876048565,
+            0.03851688280701637,
+            0.0013951680157333612,
+            0.0014764508232474327,
+            -0.01298742275685072,
+            -0.008008019998669624,
+            -0.0038035865873098373,
+            0.0024852475617080927,
+            0.012354975566267967,
+            -0.0010978736681863666,
+            -0.01851019449532032,
+            0.003068701596930623,
+            0.004166575148701668,
+            0.021182503551244736,
+            0.0044182175770401955,
+            0.002165683778002858,
+            -0.009246189147233963,
+            0.01069814432412386,
+            -0.00923728197813034,
+            -0.004021825268864632,
+            0.009860821068286896,
+            -0.0033760173246264458,
+            -0.010457636788487434,
+            -0.016452517360448837,
+            -0.0003234607574995607,
+            0.008742904290556908,
+            -0.013691130094230175,
+            -0.014742238447070122,
+            0.022429581731557846,
+            -0.0014385931426659226,
+            0.023035304620862007,
+            -0.005406972020864487,
+            0.008622650988399982,
+            0.011473113670945168,
+            -0.0012159006437286735,
+            -0.00782986544072628,
+            0.019703825935721397,
+            -0.00027196313021704555,
+            0.004277921747416258,
+            -0.014679884538054466,
+            0.009531236253678799,
+            -0.01114352885633707,
+            -0.0023605397436767817,
+            0.016363440081477165,
+            0.0001629830221645534,
+            0.007037080824375153,
+            -0.002538693603128195,
+            -0.009032404981553555,
+            -0.031782664358615875,
+            -0.02353413589298725,
+            -0.0022057686001062393,
+            0.013094314374029636,
+            -0.008578112348914146,
+            0.0022592146415263414,
+            -0.0027480246499180794,
+            -0.004053002223372459,
+            -0.016194192692637444,
+            0.006507072597742081,
+            5.518596663023345e-05,
+            0.015971500426530838,
+            0.0011435255873948336,
+            -0.015276700258255005,
+            0.003774636657908559,
+            -0.0031399631407111883,
+            0.018937764689326286,
+            -0.013922730460762978,
+            -0.0029573554638773203,
+            0.0236944742500782,
+            -0.01618528552353382,
+            -0.0011691352119669318,
+            0.003202317049726844,
+            -0.013682222925126553,
+            0.031016601249575615,
+            0.013094314374029636,
+            0.023943889886140823,
+            0.015401408076286316,
+            0.01212337613105774,
+            0.001612293184734881,
+            0.01630108617246151,
+            0.013619869016110897,
+            0.012925068847835064,
+            -0.015989316627383232,
+            -0.007348849903792143,
+            -0.017280932515859604,
+            -0.004551833029836416,
+            0.012782545760273933,
+            -0.017031516879796982,
+            0.0053579797968268394,
+            -0.01559737790375948,
+            -0.013014145195484161,
+            -0.0167375635355711,
+            -0.006079503335058689,
+            0.005148648750036955,
+            -0.02216234989464283,
+            -0.012942884117364883,
+            -0.012862714938819408,
+            -0.009860821068286896,
+            -0.01879524067044258,
+            -0.011339498683810234,
+            -0.012016483582556248,
+            -0.019204994663596153,
+            -0.035559527575969696,
+            -0.014626438729465008,
+            -0.0037456867285072803,
+            -0.011615636758506298,
+            -0.0038414443843066692,
+            -0.03634340316057205,
+            0.012916160747408867,
+            6.475478585343808e-05,
+            -0.004371452145278454,
+            0.003963925410062075,
+            0.013397176750004292,
+            -0.00856920424848795,
+            -0.03237057104706764,
+            -0.028077062219381332,
+            -0.01972164213657379,
+            -0.00024579677847214043,
+            -0.00031984198722057045,
+            0.011259329505264759,
+            0.0004286829207558185,
+            0.017058240249753,
+            -0.01053780596703291,
+            0.017672870308160782,
+            -0.03586238995194435,
+            -0.03196081891655922,
+            -0.003759048180654645,
+            -0.008729543536901474,
+            -0.012043206952512264,
+            0.006409088149666786,
+            0.007037080824375153,
+            -0.0055851261131465435,
+            -0.012354975566267967,
+            -0.007117250002920628,
+            -0.016407977789640427,
+            -0.008925512433052063,
+            0.009210558608174324,
+            0.003333705710247159,
+            0.012809268198907375,
+            0.0150540079921484,
+            0.00868945848196745,
+            -0.01657722517848015,
+            0.01975727267563343,
+            -0.005362433847039938,
+            -0.003008574713021517,
+            0.013512976467609406,
+            0.010110235773026943,
+            0.0167375635355711,
+            0.01130386721342802,
+            0.004306871443986893,
+            -0.014964930713176727,
+            0.02911035344004631,
+            0.015846792608499527,
+            0.009174928069114685,
+            0.01543703954666853,
+            0.015151992440223694,
+            0.009081397205591202,
+            -0.01202539075165987,
+            -0.0007176263607107103,
+            -0.017655055969953537,
+            0.0008055898360908031,
+            0.007353303954005241,
+            -0.010992098599672318,
+            -0.006070595700293779,
+            0.033350419253110886,
+            0.0022703493013978004,
+            -0.015730993822216988,
+            0.013138853013515472,
+            -0.011250421404838562,
+            -0.006720857694745064,
+            -0.011214790865778923,
+            -0.0118472371250391,
+            -0.003019709372892976,
+            0.015303423628211021,
+            0.011366221122443676,
+            -0.02103998139500618,
+            0.011437483131885529,
+            0.01303196046501398,
+            -0.0057098339311778545,
+            -0.005335710477083921,
+            0.014528454281389713,
+            0.01870616339147091,
+            0.01433248445391655,
+            -0.03628995642066002,
+            -0.0023271357640624046,
+            0.012078837491571903,
+            -0.026527121663093567,
+            -0.019383149221539497,
+            0.013940545730292797,
+            0.0012548718368634582,
+            -0.019044656306505203,
+            0.01642579399049282,
+            -0.002663401421159506,
+            -0.022874966263771057,
+            -0.008542481809854507,
+            0.0034851364325731993,
+            0.019935425370931625,
+            0.021129056811332703,
+            -0.012542037293314934,
+            0.008217350579798222,
+            -0.014697699807584286,
+            0.004286829382181168,
+            -0.023355981335043907,
+            -0.0037456867285072803,
+            -0.008560297079384327,
+            -0.0021345068234950304,
+            0.017672870308160782,
+            0.008729543536901474,
+            0.02312438189983368,
+            0.019971057772636414,
+            0.0006380138220265508,
+            -0.016167471185326576,
+            -0.003306982573121786,
+            0.011232606135308743,
+            -0.008208442479372025,
+            -0.0002967376494780183,
+            -0.001585570047609508,
+            0.012907253578305244,
+            0.005393610801547766,
+            0.030642477795481682,
+            -0.002772520761936903,
+            0.0012994103599339724,
+            -3.0011284252395853e-05,
+            0.002788109239190817,
+            0.0012392833596095443,
+            -0.005972610786557198,
+            -0.0018917721463367343,
+            0.005153102800250053,
+            0.02552945911884308,
+            0.004939318168908358,
+            -0.0069123730063438416,
+            0.007976843044161797,
+            -0.0022169030271470547,
+            -0.011642360128462315,
+            -0.0038102674297988415,
+            -0.014840223826467991,
+            -0.01746799424290657,
+            0.01979290321469307,
+            0.0277029387652874,
+            -0.013762392103672028,
+            -0.026954691857099533,
+            -0.00011412987078074366,
+            0.04044094309210777,
+            0.002268122276291251,
+            0.0028727322351187468,
+            0.022946227341890335,
+            -0.0020465434063225985,
+            0.018563641235232353,
+            0.021823858842253685,
+            0.011027729138731956,
+            0.009442158974707127,
+            0.03221023455262184,
+            0.0017158451955765486,
+            -0.004694356583058834,
+            -0.0030286170076578856,
+            -0.0025320129934698343,
+            -0.012497498653829098,
+            0.0009325245628133416,
+            0.002859370782971382,
+            -0.02068367227911949,
+            0.00792785082012415,
+            -0.01959693431854248,
+            -0.0006819955888204277,
+            -0.003652155864983797,
+            0.0065382495522499084,
+            -0.025476014241576195,
+            -0.010047882795333862,
+            0.009967712685465813,
+            -0.01043982058763504,
+            0.0023739011958241463,
+            -0.01915154792368412,
+            -0.004184390883892775,
+            0.022055458277463913,
+            -0.004451621789485216,
+            0.013887099921703339,
+            -0.00841331947594881,
+            -0.0011318342294543982,
+            -0.008404412306845188,
+            -0.0066451421007514,
+            0.011196975596249104,
+            -0.0058122724294662476,
+            0.001120699685998261,
+            -0.010876297950744629,
+            0.008065919391810894,
+            -0.005634118337184191,
+            0.004805702716112137,
+            0.023712290450930595,
+            -0.0030442054849117994,
+            -0.0068901036866009235,
+            0.007424565497785807,
+            0.010038974694907665,
+            -0.022411765530705452,
+            0.02702595293521881,
+            -0.011348405852913857,
+            0.02059459500014782,
+            0.00296626309864223,
+            -0.019703825935721397,
+            -0.0027168476954102516,
+            -0.0010488813277333975,
+            0.010867390781641006,
+            -0.006649595685303211,
+            0.002970716916024685,
+            -0.01831422559916973,
+            0.015276700258255005,
+            0.005945887882262468,
+            -0.015330146998167038,
+            0.01979290321469307,
+            -0.006988088600337505,
+            0.0019663742277771235,
+            -0.017485808581113815,
+            0.00578109547495842,
+            0.007130611687898636,
+            -0.013664407655596733,
+            -0.003739005886018276,
+            -0.019703825935721397,
+            -0.0005723195499740541,
+            -0.01565973088145256,
+            0.0036098442506045103,
+            -0.009718297980725765,
+            -0.013646592386066914,
+            -0.036913495510816574,
+            -0.0001862265489762649,
+            0.004685448482632637,
+            -0.007197419181466103,
+            -0.001399621949531138,
+            0.00012637795589398593,
+            0.013958360999822617,
+            0.009290727786719799,
+            -0.0039572445675730705,
+            0.004979402758181095,
+            0.034348081797361374,
+            -0.04421781003475189,
+            -0.01138403732329607,
+            -0.00033654391882009804,
+            0.004008463583886623,
+            -0.020737119019031525,
+            -0.008275250904262066,
+            0.013121037743985653,
+            -0.0007081619114615023,
+            0.016274362802505493,
+            -0.016033854335546494,
+            0.010947559960186481,
+            0.003079836256802082,
+            0.008177265524864197,
+            -0.007135065272450447,
+            -0.0053579797968268394,
+            -0.012034298852086067,
+            -0.000576773367356509,
+            -0.001016590977087617,
+            -0.02827303111553192,
+            0.01734328642487526,
+            0.0016033854335546494,
+            -0.0056296647526323795,
+            0.0069480035454034805,
+            -0.02453179843723774,
+            -0.009148204699158669,
+            -0.012292621657252312,
+            0.023658843711018562,
+            -0.01738782413303852,
+            -0.00026625662576407194,
+            -0.004698810167610645,
+            -0.006444718688726425,
+            0.014724423177540302,
+            -0.0037234174087643623,
+            0.007322126999497414,
+            0.00037635021726600826,
+            -0.011339498683810234,
+            -0.002014253055676818,
+            0.0024785667192190886,
+            -0.010190405882894993,
+            0.031693585216999054,
+            -0.005959249567240477,
+            -0.0027702937368303537,
+            0.007433473132550716,
+            0.0010666967136785388,
+            -0.002318228129297495,
+            0.010270575061440468,
+            0.0017882202519103885,
+            0.0039572445675730705,
+            0.008894335478544235,
+            -0.0036989212967455387,
+            -0.006021603476256132,
+            0.016007130965590477,
+            0.020541150122880936,
+            0.005616303067654371,
+            -0.016033854335546494,
+            -0.011116806417703629,
+            0.011758159846067429,
+            0.0015733219916000962,
+            0.0006396839744411409,
+            -0.008057012222707272,
+            0.006324464920908213,
+            -0.002256987849250436,
+            0.012043206952512264,
+            0.011838329024612904,
+            0.017298748716711998,
+            0.012898345477879047,
+            -0.005723195616155863,
+            0.03144416958093643,
+            0.01614074781537056,
+            0.0009007908520288765,
+            0.015134177170693874,
+            -0.04368334636092186,
+            -0.01027948223054409,
+            -0.012078837491571903,
+            0.01754816249012947,
+            -0.014751146547496319,
+            -0.016488147899508476,
+            -0.013183391653001308,
+            0.010653605684638023,
+            -0.0032446286641061306,
+            0.005473779980093241,
+            0.0251375213265419,
+            -0.013414992019534111,
+            0.004393721465021372,
+            -0.002739116782322526,
+            0.023623213171958923,
+            0.0030263899825513363,
+            0.00907694362103939,
+            0.007495827041566372,
+            0.005104110576212406,
+            -0.0027057130355387926,
+            0.01963256485760212,
+            -0.009219466708600521,
+            0.036485929042100906,
+            -0.01053780596703291,
+            -0.009317451156675816,
+            -0.01582006923854351,
+            -0.017637239769101143,
+            -0.012354975566267967,
+            0.004752256441861391,
+            -0.000690346525516361,
+            0.0012570987455546856,
+            -0.00821289699524641,
+            0.013637684285640717,
+            0.03140854090452194,
+            -0.0013417218578979373,
+            0.015160900540649891,
+            0.007856588810682297,
+            -0.0006513753323815763,
+            0.01543703954666853,
+            0.003883755998685956,
+            -0.004489479120820761,
+            0.004729987122118473,
+            0.024513982236385345,
+            0.0004924286040477455,
+            0.00767843471840024,
+            -0.0010744909523054957,
+            -0.009602497331798077,
+            -0.01156219094991684,
+            0.010831759311258793,
+            0.027115030214190483,
+            0.017334379255771637,
+            0.001331700710579753,
+            -0.0118472371250391,
+            -0.00017815394676290452,
+            -0.0009375351364724338,
+            0.016791008412837982,
+            -0.010965375229716301,
+            -0.006689680740237236,
+            0.013121037743985653,
+            -0.009085850790143013,
+            0.007611627224832773,
+            -0.01662176288664341,
+            0.0013907141983509064,
+            0.00722859613597393,
+            0.014270130544900894,
+            -0.0045206560753285885,
+            0.006070595700293779,
+            -0.00707716541364789,
+            0.010555621236562729,
+            -0.0127290990203619,
+            0.011277144774794579,
+            -0.005723195616155863,
+            0.00046709735761396587,
+            0.028807492926716805,
+            -0.006475895643234253,
+            0.008195081725716591,
+            0.00011621761223068461,
+            -0.0034717749804258347,
+            -0.017494717612862587,
+            0.025761060416698456,
+            0.004084179177880287,
+            0.005914710927754641,
+            0.00020014481560792774,
+            0.010288390330970287,
+            -0.008301973342895508,
+            -0.001823851023800671,
+            0.022304873913526535,
+            0.011927406303584576,
+            0.0038703943137079477,
+            -0.0005873512709513307,
+            0.008003565482795238,
+            -0.012221360579133034,
+            0.015134177170693874,
+            0.01899120956659317,
+            0.008644919842481613,
+            0.0036543826572597027,
+            0.014599715359508991,
+            -0.013094314374029636,
+            0.008466766215860844,
+            -0.009072489105165005,
+            0.029163800179958344,
+            0.012640022672712803,
+            -0.016523778438568115,
+            0.019294071942567825,
+            0.030767185613512993,
+            -0.004930410534143448,
+            -0.010430913418531418,
+            0.0015844566514715552,
+            -0.017414547502994537,
+            -0.0021567761432379484,
+            -0.014920393005013466,
+            -0.014768961817026138,
+            0.005126379895955324,
+            -0.015312331728637218,
+            0.02497718296945095,
+            -0.017280932515859604,
+            -0.031657956540584564,
+            -0.004309098701924086,
+            -0.00825298111885786,
+            -0.02818395383656025,
+            -0.008301973342895508,
+            0.006190849468111992,
+            0.014849130995571613,
+            -0.0019930971320718527,
+            -0.0144928228110075,
+            0.013664407655596733,
+            0.009085850790143013,
+            0.01359314564615488,
+            -0.017316563054919243,
+            -0.01606057770550251,
+            -0.015615193173289299,
+            0.017316563054919243,
+            -0.010956467129290104,
+            -0.016470331698656082,
+            0.014270130544900894,
+            0.003759048180654645,
+            0.005464872345328331,
+            0.0017503624549135566,
+            -0.014519546180963516,
+            -0.016354531049728394,
+            -0.027845460921525955,
+            0.010626882314682007,
+            -0.0027970168739557266,
+            -0.010974282398819923,
+            0.0029194976668804884,
+            0.011437483131885529,
+            0.00939762033522129,
+            0.005749918520450592,
+            -0.018242964521050453,
+            0.011268236674368382,
+            0.01311213057488203,
+            -0.018528010696172714,
+            -0.007429019547998905,
+            0.005451510660350323,
+            -0.009727205149829388,
+            0.03798241913318634,
+            0.01345062255859375,
+            -0.015205439180135727,
+            -0.0030397516675293446,
+            0.004930410534143448,
+            0.01686227135360241,
+            0.010056789964437485,
+            -0.018848687410354614,
+            -0.001327246893197298,
+            0.021895119920372963,
+            -0.004549606237560511,
+            0.004366998560726643,
+            -3.483918590063695e-06,
+            0.042614422738552094,
+            0.0033426133450120687,
+            -0.0056875646114349365,
+            -0.013138853013515472,
+            0.002210222417488694,
+            0.00997662078589201,
+            0.013023053295910358,
+            0.012239175848662853,
+            -0.007326581049710512,
+            0.0025008360389620066,
+            0.02020265720784664,
+            -0.010395282879471779,
+            0.0020955358631908894,
+            0.0030352978501468897,
+            0.005936980247497559,
+            -0.016880085691809654,
+            0.026242075487971306,
+            -0.0009653716697357595,
+            0.00692573469132185,
+            0.0004620867839548737,
+            -0.009923174977302551,
+            0.014670977368950844,
+            0.024122044444084167,
+            0.0008417773642577231,
+            0.015624100342392921,
+            -0.012435144744813442,
+            0.01831422559916973,
+            0.007901127450168133,
+            0.00782986544072628,
+            -0.015330146998167038,
+            -0.010092420503497124,
+            0.006952457595616579,
+            -0.0028816401027143,
+            -0.002538693603128195,
+            -0.0039839674718678,
+            -0.012372790835797787,
+            -0.011936314404010773,
+            -0.021129056811332703,
+            0.004373679403215647,
+            0.00836878176778555,
+            -0.006132949609309435,
+            0.006850019097328186,
+            0.004309098701924086,
+            -0.0033671094570308924,
+            0.0005984858726151288,
+            -0.013682222925126553,
+            -0.008707273751497269,
+            0.0033470671623945236,
+            0.002696805400773883,
+            0.004191071726381779,
+            -0.007856588810682297,
+            0.020184841006994247,
+            0.006609511096030474,
+            -0.006110680289566517,
+            0.007874404080212116,
+            -0.019062472507357597,
+            -0.0028660516254603863,
+            -0.012105559930205345,
+            0.00011127662583021447,
+            0.0042200214229524136,
+            0.01638125441968441,
+            0.0031243746634572744,
+            -0.023747920989990234,
+            0.0068322038277983665,
+            -0.01202539075165987,
+            0.008965597487986088,
+            -0.003741232678294182,
+            -0.0005160897271707654,
+            -0.012693468481302261,
+            0.01594477705657482,
+            0.01535686943680048,
+            -0.011419667862355709,
+            -0.021093426272273064,
+            0.017637239769101143,
+            -0.03103441745042801,
+            0.022108905017375946,
+            0.010671420954167843,
+            0.0036855596117675304,
+            -0.019810717552900314,
+            0.017120594158768654,
+            0.012399514205753803,
+            0.01101882103830576,
+            -0.0032936211209744215,
+            0.011294960044324398,
+            0.021627888083457947,
+            -0.007941211573779583,
+            -0.0010410870891064405,
+            -9.525320820102934e-06,
+            -0.010430913418531418,
+            -0.0002900568943005055,
+            -0.007175149861723185,
+            0.014421561732888222,
+            0.00836878176778555,
+            -0.008836435154080391,
+            -0.013548607006669044,
+            0.01228371448814869,
+            -0.011482021771371365,
+            0.007117250002920628,
+            -0.005985972471535206,
+            0.007219688501209021,
+            0.004342502448707819,
+            0.01770850270986557,
+            -0.021057795733213425,
+            0.0005670305690728128,
+            0.005237726029008627,
+            -0.022946227341890335,
+            0.007041534408926964,
+            -0.024585243314504623,
+            0.014902577735483646,
+            -0.025351306423544884,
+            -0.005549495108425617,
+            0.02046988718211651,
+            0.014599715359508991,
+            0.0033092093653976917,
+            -0.018011363223195076,
+            -0.007059350144118071,
+            -0.013619869016110897,
+            0.002101103076711297,
+            -0.022429581731557846,
+            -0.013388268649578094,
+            -0.015303423628211021,
+            0.009531236253678799,
+            0.011027729138731956,
+            0.0024028513580560684,
+            -0.010582344606518745,
+            0.007535911630839109,
+            -0.02168133482336998,
+            -0.010617975145578384,
+            -0.014653162099421024,
+            -0.017681779339909554,
+            0.001418550731614232,
+            0.02661619894206524,
+            -0.0027413438074290752,
+            0.01879524067044258,
+            0.011954129673540592,
+            0.01471551600843668,
+            -0.0031733671203255653,
+            -0.010938651859760284,
+            -0.00923728197813034,
+            -0.0027925630565732718,
+            -0.003554171184077859,
+            -0.018777426332235336,
+            -0.012559852562844753,
+            0.00489923357963562,
+            -0.0019853028934448957,
+            0.003981740679591894,
+            0.007874404080212116,
+            0.00841331947594881,
+            -0.01100100576877594,
+            -0.010751590132713318,
+            0.016630670055747032,
+            -0.025671983137726784,
+            -0.017405640333890915,
+            0.018278595060110092,
+            -0.011232606135308743,
+            -0.0012348295422270894,
+            0.019864164292812347,
+            -0.010956467129290104,
+            -0.007491373457014561,
+            0.010493267327547073,
+            -0.003865940496325493,
+            0.010528897866606712,
+            0.022180166095495224,
+            0.006484803277999163,
+            -0.016087301075458527,
+            0.008595927618443966,
+            0.010609067045152187,
+            -0.014163237996399403,
+            -0.02734662964940071,
+            -0.0026010475121438503,
+            0.004560741130262613,
+            -0.020701488479971886,
+            -0.0026945783756673336,
+            0.0004078055208083242,
+            0.00722859613597393,
+            -0.0003855362592730671,
+            0.004925956483930349,
+            -0.018100440502166748,
+            -0.007139519322663546,
+            0.003429463366046548,
+            0.00868945848196745,
+            -0.022215796634554863,
+            0.0013873737771064043,
+            -0.013227930292487144,
+            0.02216234989464283,
+            -0.003257990116253495,
+            0.007415657863020897,
+            -0.0012548718368634582,
+            -0.0074201119132339954,
+            0.008769627660512924,
+            0.012069929391145706,
+            0.005972610786557198,
+            0.03295847773551941,
+            -0.003894890658557415,
+            -0.015009469352662563,
+            0.014234500005841255,
+            -0.025885768234729767,
+            -0.0007337715360336006,
+            -0.0038325367495417595,
+            0.010626882314682007,
+            -0.011036636307835579,
+            -0.011188067495822906,
+            0.02203764207661152,
+            0.02100434899330139,
+            -0.006930188275873661,
+            -0.0002297907485626638,
+            -0.01750362478196621,
+            0.01662176288664341,
+            -0.00824852753430605,
+            0.0015833431389182806,
+            0.007976843044161797,
+            -0.023320350795984268,
+            -0.012631114572286606,
+            -0.006021603476256132,
+            0.024870289489626884,
+            -0.01934751868247986,
+            -0.016559408977627754,
+            -0.012515314854681492,
+            0.00012282878742553294,
+            -0.02709721401333809,
+            -0.0075225504115223885,
+            0.007063803728669882,
+            -0.012925068847835064,
+            -0.007553727366030216,
+            0.007954573258757591,
+            0.009477789513766766,
+            -0.0021033298689872026,
+            -0.02520878240466118,
+            -0.011188067495822906,
+            0.013210115022957325,
+            0.003939428832381964,
+            0.006190849468111992,
+            -0.004364771768450737,
+            0.006168580148369074,
+            -0.006279926281422377,
+            0.0032513095065951347,
+            0.00899231992661953,
+            -0.0016490374691784382,
+            -0.006342280190438032,
+            0.02219798043370247,
+            0.005522772204130888,
+            -0.018180610612034798,
+            0.01433248445391655,
+            0.01570427045226097,
+            -0.012185730040073395,
+            -0.008489035069942474,
+            3.824046507361345e-05,
+            -0.013931638561189175,
+            -0.0008918831590563059,
+            -0.006475895643234253,
+            0.03196081891655922,
+            -0.022874966263771057,
+            0.003857032861560583,
+            0.008685004897415638,
+            -0.0002842112153302878,
+            -0.004903687164187431,
+            0.002912817057222128,
+            0.0054826876148581505,
+            -0.0012793679488822818,
+            0.00577218784019351,
+            0.00812827330082655,
+            -0.014216684736311436,
+            -0.006337826605886221,
+            0.013691130094230175,
+            0.002291504992172122,
+            0.017806487157940865,
+            0.016452517360448837,
+            1.9311608411953785e-05,
+            -0.03313663229346275,
+            0.004193298518657684,
+            0.0007877744501456618,
+            -0.021627888083457947,
+            -0.0038169482722878456,
+            -0.006106226239353418,
+            0.00967375934123993,
+            0.010324020870029926,
+            0.01614074781537056,
+            0.0029194976668804884,
+            0.007143972907215357,
+            0.006141857244074345,
+            0.010288390330970287,
+            -0.030927523970603943,
+            -0.006444718688726425,
+            -0.008333150297403336,
+            -0.006453626789152622,
+            0.012622206471860409,
+            0.012631114572286606,
+            0.029983308166265488,
+            0.028736229985952377,
+            -0.014751146547496319,
+            -0.019204994663596153,
+            0.004242290742695332,
+            0.011027729138731956,
+            0.008640466257929802,
+            -0.007290950044989586,
+            0.006462534423917532,
+            8.504067227477208e-05,
+            -0.02526222914457321,
+            0.0017392279114574194,
+            0.01375348400324583,
+            -0.017530348151922226,
+            -0.010617975145578384,
+            -0.018652718514204025,
+            0.014626438729465008,
+            0.02116468921303749,
+            -0.00392384035512805,
+            0.007188511546701193,
+            -0.010591251775622368,
+            -0.009317451156675816,
+            -0.04500168561935425,
+            0.0053000799380242825,
+            0.009054673835635185,
+            0.022215796634554863,
+            0.001312771812081337,
+            -0.004001782741397619,
+            -6.158837641123682e-05,
+            0.007001449819654226,
+            -0.0033960596192628145,
+            0.011464206501841545,
+            0.013210115022957325,
+            -0.018759610131382942,
+            -0.016042763367295265,
+            0.004150986671447754,
+            0.0023115472868084908,
+            -0.010172589682042599,
+            0.005968157202005386,
+            0.021253764629364014,
+            0.01586460880935192,
+            -0.007241957820951939,
+            0.004431579262018204,
+            0.005625210702419281,
+            -0.014225591905415058,
+            -0.012034298852086067,
+            -0.0045429253950715065,
+            -0.011517652310431004,
+            -0.024193305522203445,
+            0.017191855236887932,
+            -0.013824746012687683,
+            0.006395726464688778,
+            0.009869728237390518,
+            0.011223698034882545,
+            0.0018450067145749927,
+            0.023177828639745712,
+            0.00811045803129673,
+            -0.008738450706005096,
+            0.011036636307835579,
+            -0.026170814409852028,
+            0.012684560380876064,
+            -0.00023368786787614226,
+            0.011259329505264759,
+            0.0001613128260942176,
+            -0.03301192447543144,
+            -0.013512976467609406,
+            0.008172811940312386,
+            -0.006622872781008482,
+            0.009896451607346535,
+            0.014626438729465008,
+            -0.008288612589240074,
+            -0.00100879673846066,
+            -0.022340504452586174,
+            0.00445830263197422,
+            -0.03147980198264122,
+            0.00011357313633197919,
+            -0.004716625437140465,
+            0.010929744690656662,
+            -0.005362433847039938,
+            -0.0240151509642601,
+            0.0026789898984134197,
+            -0.012720191851258278,
+            -0.005825634114444256,
+            0.0010616861982271075,
+            -0.0005828974535688758,
+            0.010876297950744629,
+            -0.00593252619728446,
+            0.01375348400324583,
+            -0.007348849903792143,
+            0.005215456709265709,
+            -0.0007688456098549068,
+            0.006792119238525629,
+            -0.006658503785729408,
+            -0.012907253578305244,
+            0.00238948967307806,
+            -0.004845787305384874,
+            0.021342841908335686,
+            0.010243851691484451,
+            -0.0006380138220265508,
+            0.004362544510513544,
+            0.020291734486818314,
+            0.004672087263315916,
+            0.013165576383471489,
+            0.006676319055259228,
+            0.0032000902574509382,
+            -0.015285608358681202,
+            0.009736113250255585,
+            -0.007371119223535061,
+            0.009477789513766766,
+            0.002870505442842841,
+            -0.0010310659417882562,
+            0.014599715359508991,
+            0.008364327251911163,
+            -0.009905358776450157,
+            0.011767067946493626,
+            0.018581455573439598,
+            -0.008408865891397,
+            -0.003220132552087307,
+            -0.0127290990203619,
+            0.011294960044324398,
+            0.027204107493162155,
+            0.004260106012225151,
+            -0.01831422559916973,
+            0.014145422726869583,
+            0.00010097710037371144,
+            -0.011713622137904167,
+            0.026046106591820717,
+            -0.014751146547496319,
+            -0.025689799338579178,
+            -0.013379361480474472,
+            -0.007273134775459766,
+            -0.01750362478196621,
+            0.021271580830216408,
+            -0.011473113670945168,
+            0.012212452478706837,
+            0.004193298518657684,
+            0.03119475580751896,
+            0.010689236223697662,
+            0.01212337613105774,
+            -0.003015255555510521,
+            -0.025351306423544884,
+            0.009379805065691471,
+            0.0045874640345573425,
+            -0.04044094309210777,
+            0.0005979291745461524,
+            -0.003004120895639062,
+            0.029377585276961327,
+            0.0010550053557381034,
+            -0.0014898123918101192,
+            -0.0038971174508333206,
+            -0.020380811765789986,
+            -0.015686454251408577,
+            0.007322126999497414,
+            -0.028682785108685493,
+            0.013397176750004292,
+            -0.00507738720625639,
+            -0.012150098569691181,
+            0.002214676234871149,
+            0.010831759311258793,
+            -0.019044656306505203,
+            0.01879524067044258,
+            0.015846792608499527,
+            0.025030629709362984,
+            0.01069814432412386,
+            -0.008885428309440613,
+            0.020006688311696053,
+            0.0010210447944700718,
+            -0.006591695826500654,
+            0.0020688127260655165,
+            -1.8998447558260523e-05,
+            0.0015410316409543157,
+            -0.0033671094570308924,
+            0.003015255555510521,
+            0.01598040945827961,
+            -0.01738782413303852,
+            0.019418779760599136,
+            -0.01702260971069336,
+            0.016007130965590477,
+            0.0005347402184270322,
+            0.012925068847835064,
+            4.071095827384852e-05,
+            0.0011958583490923047,
+            -0.029840786010026932,
+            -0.0009653716697357595,
+            -0.03915823623538017,
+            -0.011312775313854218,
+            -0.0016679662512615323,
+            -0.018091533333063126,
+            -0.0037679558154195547,
+            0.009032404981553555,
+            -0.01487585436552763,
+            -0.011588914319872856,
+            0.016069484874606133,
+            0.007598265539854765,
+            0.007375573273748159,
+            0.001965260598808527,
+            -8.03084549261257e-05,
+            -0.005162010435014963,
+            -0.020808380097150803,
+            -0.012951791286468506,
+            -0.025155337527394295,
+            -0.025190968066453934,
+            -0.0035742134787142277,
+            0.007584904320538044,
+            -0.004832425620406866,
+            -0.008564750663936138,
+            0.017761947587132454,
+            -0.019044656306505203,
+            -0.00040140311466529965,
+            0.004242290742695332,
+            0.008172811940312386,
+            0.013103222474455833,
+            -0.007063803728669882,
+            0.007384480908513069,
+            0.0030709286220371723,
+            -0.0013361545279622078,
+            -0.0005183166358619928,
+            0.024104228243231773,
+            -0.008836435154080391,
+            0.011009913869202137,
+            0.007722973357886076,
+            0.011775976046919823,
+            -0.006217572372406721,
+            0.009045766666531563,
+            -0.024104228243231773,
+            0.007687342818826437,
+            -0.011045544408261776,
+            -0.026794353500008583,
+            -0.007148426957428455,
+            0.02071930281817913,
+            -0.0006536022992804646,
+            0.0008662735344842076,
+            -0.0018193970900028944,
+            -0.0004406526277307421,
+            0.023676658049225807,
+            0.01042200531810522,
+            0.0012147872475907207,
+            -0.007700704038143158,
+            -0.029769523069262505,
+            0.005985972471535206,
+            -0.021271580830216408,
+            -0.021699151024222374,
+            0.03288721665740013,
+            0.010368559509515762,
+            0.0027213015127927065,
+            -0.013691130094230175,
+            0.010038974694907665,
+            0.009442158974707127,
+            6.134480645414442e-05,
+            0.014804592356085777,
+            0.003088743891566992,
+            0.001969714416190982,
+            -0.009932082146406174,
+            -0.01918718032538891,
+            0.01614074781537056,
+            0.014581900089979172,
+            -0.01559737790375948,
+            0.036877866834402084,
+            0.016773194074630737,
+            0.012809268198907375,
+            -0.010288390330970287,
+            0.007714065723121166,
+            0.016995886340737343,
+            0.020060133188962936,
+            -0.0070860730484128,
+            0.0034339171834290028,
+            -0.010742682963609695,
+            -0.00023466214770451188,
+            0.03894445300102234,
+            0.03516758978366852,
+            0.008444496430456638,
+            -0.022785888984799385,
+            0.014581900089979172,
+            -0.009335266426205635,
+            -0.01287162210792303,
+            0.010306205600500107,
+            0.01606057770550251,
+            -0.0005923618446104228,
+            0.0014686565846204758,
+            0.009451067075133324,
+            0.0076606194488704205,
+            0.017441270872950554,
+            0.005353525746613741,
+            -0.002378355246037245,
+            0.003402740228921175,
+            -0.0034339171834290028,
+            0.00965594407171011,
+            -0.016808824613690376,
+            -0.005126379895955324,
+            -0.00339828641153872,
+            -0.0016969162970781326,
+            0.005251087248325348,
+            0.009540143422782421,
+            -0.036361221224069595,
+            -0.007415657863020897,
+            -0.033350419253110886,
+            0.004021825268864632,
+            -0.01995324157178402,
+            -0.0035430365242064,
+            -0.010965375229716301,
+            -0.01598040945827961,
+            -0.026562752202153206,
+            -0.016933532431721687,
+            -0.0031266016885638237,
+            0.008903243578970432,
+            -0.003888209816068411,
+            0.003215678734704852,
+            0.003436144208535552,
+            -0.021378472447395325,
+            0.005660841707140207,
+            0.013174484483897686,
+            0.009370896965265274,
+            0.021057795733213425,
+            0.0001496214681537822,
+            0.014368114992976189,
+            -0.014822407625615597,
+            -0.04029842093586922,
+            -0.0007649484905414283,
+            0.00031009921804070473,
+            -0.005175372119992971,
+            -0.00424674479290843,
+            0.004859148990362883,
+            -0.009379805065691471,
+            0.011098990216851234,
+            -0.0021745916455984116,
+            0.0009681553347036242,
+            -0.015722084790468216,
+            -0.010707051493227482,
+            -0.023587582632899284,
+            -0.00505066430196166,
+            -0.013121037743985653,
+            -2.176470479753334e-05,
+            -0.01288943737745285,
+            -0.004952679388225079,
+            -0.005398064386099577,
+            -0.0013461756752803922,
+            -0.006426903419196606,
+            -0.002723528305068612,
+            -0.001202539075165987,
+            -0.013263560831546783,
+            -0.012559852562844753,
+            -0.0013172257458791137,
+            0.004832425620406866,
+            0.0006942436448298395,
+            -0.01098319049924612,
+            0.013414992019534111,
+            -0.014911484904587269,
+            0.0018038086127489805,
+            0.02574324421584606,
+            0.013325914740562439,
+            -0.010484359227120876,
+            0.016479238867759705,
+            0.002199087757617235,
+            0.010128051973879337,
+            -0.02152099646627903,
+            -0.015722084790468216,
+            -0.010110235773026943,
+            0.026990322396159172,
+            0.0012682334054261446,
+            -0.013557515107095242,
+            -0.007206326816231012,
+            -0.01706714741885662,
+            0.004680994898080826,
+            -0.0077407886274158955,
+            0.011375129222869873,
+            -0.020630227401852608,
+            -0.007215234450995922,
+            -0.011473113670945168,
+            -0.007531458046287298,
+            0.0065827881917357445,
+            0.03794679045677185,
+            -0.00289054773747921,
+            -0.007852135226130486,
+            0.005963703151792288,
+            0.010849575512111187,
+            0.028237400576472282,
+            0.0010516649344936013,
+            -0.0014307989040389657,
+            -0.012043206952512264,
+            0.0034717749804258347,
+            0.004551833029836416,
+            -0.01051999069750309,
+            0.017031516879796982,
+            -0.009148204699158669,
+            -0.01934751868247986,
+            -0.004500614013522863,
+            -0.00868945848196745,
+            -0.007353303954005241,
+            0.01753925532102585,
+            0.0023226819466799498,
+            0.003208997892215848,
+            0.0008540254784747958,
+            -0.007250865455716848,
+            -0.008333150297403336,
+            0.011009913869202137,
+            0.08451622724533081,
+            0.00943325087428093,
+            -0.02228705771267414,
+            0.0012526449281722307,
+            0.004943771753460169,
+            0.001479791128076613,
+            -0.014795685186982155,
+            0.06591695547103882,
+            -0.014368114992976189,
+            0.005451510660350323,
+            0.0004428795655258,
+            0.004810156300663948,
+            -0.004447167739272118,
+            0.02136065810918808,
+            0.010199313051998615,
+            -0.01244405284523964,
+            0.002025387715548277,
+            -0.03541700541973114,
+            -0.017280932515859604,
+            0.0034984981175512075,
+            -0.0044404868967831135,
+            -0.004393721465021372,
+            0.013121037743985653,
+            -0.023106565698981285,
+            -0.008163904771208763,
+            0.010956467129290104,
+            -0.012052114121615887,
+            0.011473113670945168,
+            -0.019383149221539497,
+            -0.001312771812081337,
+            -0.0020198202691972256,
+            0.0014519545948132873,
+            -0.009727205149829388,
+            0.0036766519770026207,
+            -0.0027123938780277967,
+            0.0010705938329920173,
+            0.009477789513766766,
+            0.0077853272669017315,
+            -0.003741232678294182,
+            0.017877748236060143,
+            0.013441715389490128,
+            0.0035563979763537645,
+            0.017485808581113815,
+            0.0027480246499180794,
+            0.007651711814105511,
+            0.0003485136548988521,
+            0.01785993203520775,
+            -0.014483915641903877,
+            0.021022165194153786,
+            -0.01834985613822937,
+            -0.02059459500014782,
+            -0.013691130094230175,
+            0.009063581936061382,
+            -0.015561746433377266,
+            0.001760383602231741,
+            -0.011571098119020462,
+            0.029858600348234177,
+            0.010885206051170826,
+            -0.003088743891566992,
+            0.004380360245704651,
+            -0.007045988459140062,
+            0.03930075839161873,
+            0.007856588810682297,
+            0.033225711435079575,
+            0.003155551617965102,
+            0.017886655405163765,
+            -0.002788109239190817,
+            0.004001782741397619,
+            -0.013254653662443161,
+            -0.005821180064231157,
+            0.018278595060110092,
+            -0.009905358776450157,
+            0.019062472507357597,
+            0.004863602574914694,
+            -0.014439377002418041,
+            0.01770850270986557,
+            -0.015410316176712513,
+            -0.02292841300368309,
+            0.013842561282217503,
+            -0.02408641390502453,
+            0.0002925621811300516,
+            0.006230934057384729,
+            -0.0036276597529649734,
+            -0.01051999069750309,
+            0.021752595901489258,
+            -0.0005116358515806496,
+            -0.006907918956130743,
+            -0.010644698515534401,
+            -0.013245745562016964,
+            -0.014279038645327091,
+            0.00895668938755989,
+            -0.004574102349579334,
+            -0.01298742275685072,
+            0.011009913869202137,
+            -0.02526222914457321,
+            0.012862714938819408,
+            -0.012043206952512264,
+            0.023017490282654762,
+            -0.00795011967420578,
+            0.02075493521988392,
+            0.0022825973574072123,
+            -0.0002908919705078006,
+            0.0019853028934448957,
+            0.008863158524036407,
+            -0.003224586369469762,
+            0.016327809542417526,
+            0.004819063935428858,
+            0.004997218027710915,
+            0.004583009984344244,
+            -0.005041756667196751,
+            0.015472670085728168,
+            -0.011749252676963806,
+            -0.0009202764485962689,
+            0.01418996136635542,
+            0.009290727786719799,
+            -0.018207332119345665,
+            0.025850137695670128,
+            0.0038748481310904026,
+            0.015303423628211021,
+            -0.023266904056072235,
+            0.0058924416080117226,
+            0.012622206471860409,
+            0.002320455154404044,
+            0.01288943737745285,
+            -0.004729987122118473,
+            0.022946227341890335,
+            0.018902134150266647,
+            0.004734441172331572,
+            -0.0025854590348899364,
+            0.013040868565440178,
+            -0.002086627995595336,
+            -0.0016456970479339361,
+            0.011891775764524937,
+            -0.0034940443001687527,
+            0.004077498335391283,
+            0.001429685391485691,
+            -0.0006591695710085332,
+            0.016755377873778343,
+            -0.01399399247020483,
+            0.014216684736311436,
+            0.006030511111021042,
+            -0.011063359677791595,
+            -0.013896007090806961,
+            0.007099434733390808,
+            -0.015080731362104416,
+            0.0025097436737269163,
+            0.023177828639745712,
+            -0.0014775642193853855,
+            -0.00865382794290781,
+            0.0048502408899366856,
+            0.012381698936223984,
+            -0.0058479029685258865,
+            -0.010484359227120876,
+            0.006181941833347082,
+            0.008221804164350033,
+            0.00027766963467001915,
+            0.006596149876713753,
+            0.0068099345080554485,
+            0.007028173189610243,
+            -0.0007922283257357776,
+            -0.008195081725716591,
+            0.0030508863274008036,
+            0.020933087915182114,
+            -0.0008629331714473665,
+            -0.009210558608174324,
+            -0.03643248230218887,
+            -0.003191182389855385,
+            -0.014679884538054466,
+            0.003086517099291086,
+            0.00030870738555677235,
+            -0.0008350965799763799,
+            -0.004493933171033859,
+            -0.009638127870857716,
+            0.009099212475121021,
+            0.0008907697047106922,
+            0.024745581671595573,
+            -0.012693468481302261,
+            -0.015365777537226677,
+            0.003257990116253495,
+            -0.0055405874736607075,
+            -0.010769405402243137,
+            0.007451288867741823,
+            -0.004213340580463409,
+            0.03269124776124954,
+            0.00901013519614935,
+            0.0025052898563444614,
+            -0.014786777086555958,
+            -0.018813056871294975,
+            0.010208221152424812,
+            -0.00245852442458272,
+            -0.005598487798124552,
+            -0.003939428832381964,
+            -0.02994767762720585,
+            0.010324020870029926,
+            -0.00967375934123993,
+            0.0036744249518960714,
+            -0.0044182175770401955,
+            0.009522328153252602,
+            0.003327024867758155,
+            -0.01487585436552763,
+            -0.0030375246424227953,
+            0.013566422276198864,
+            0.015258884988725185,
+            -0.012435144744813442,
+            0.004451621789485216,
+            -0.005999334156513214,
+            -0.012185730040073395,
+            0.009326359257102013,
+            -0.00967375934123993,
+            -0.017655055969953537,
+            0.002447389764711261
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "611ac443-aa6f-4416-a1f9-dade7d45888c",
+        "properties": {
+          "page_content": "6. Alignment For Deepseekmoe 16B\n\nPrevious research indicates that MoE models typically do not emerge significant gains from fine-tuning (Artetxe et al., 2022; Fedus et al., 2021). However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B.\n\n6.1. Experimental Setup\n\nTraining Data. For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios.\n\nHyper-Parameters. During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).\n\nWe employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10−5 without incorporating any learning rate scheduling strategy.\n\nEvaluation Benchmarks. For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3, with the following adjustments: (1) We exclude Pile (Gao et al., 2020) since chat models are seldom employed for pure language modeling. (2) We exclude\n\nMetric # Shot LLaMA2 DeepSeek DeepSeekMoE SFT 7B Chat 7B Chat 16B # Total Params N/A 6.7B 6.9B 16.4B # Activated Params N/A 6.7B 6.9B 2.8B FLOPs per 4K Tokens N/A 187.9T 183.5T 74.4T HellaSwag (Acc.) 0-shot 67.9 71.0 72.2 PIQA (Acc.) 0-shot 76.9 78.4 79.7 ARC-easy (Acc.) 0-shot 69.7 70.2 69.9 ARC-challenge (Acc.) 0-shot 50.8 50.2 50.0 BBH (EM) 3-shot 39.3 43.1 42.2 RACE-middle (Acc.) 5-shot 63.9 66.1 64.8 RACE-high (Acc.) 5-shot 49.6 50.8 50.6 DROP (EM) 1-shot 40.0 41.7 33.8 GSM8K (EM) 0-shot 63.4 62.6 62.2 MATH (EM) 4-shot 13.5 14.7 15.2 HumanEval (Pass@1) 0-shot 35.4 45.1 45.7 MBPP (Pass@1) 3-shot 27.8 39.0 46.2 TriviaQA (EM) 5-shot 60.1 59.5 63.3 NaturalQuestions (EM) 0-shot 35.2 32.7 35.1 MMLU (Acc.) 0-shot 50.0 49.7 47.2 WinoGrande (Acc.) 0-shot 65.1 68.4 69.0 CLUEWSC (EM) 5-shot 48.4 66.2 68.2 CEval (Acc.) 0-shot 35.1 44.7 40.0 CMMLU (Acc.) 0-shot 36.9 51.2 49.3\n\nCHID (Zheng et al., 2019) due to the observed instability of results, hindering the derivation of solid conclusions. (3) We additionally include BBH (Suzgun et al., 2022) to provide a more comprehensive assessment of the reasoning ability of the chat models.\n\nTable 5 | Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations.\n\n6.2. Evaluations\n\nBaselines. In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. Correspondingly, we construct three chat models, including LLaMA2 SFT 7B3, DeepSeek Chat 7B, and DeepSeekMoE Chat 16B.\n\nSubsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks.\n\nResults. The evaluation results are presented in Table 5. Our key observations include: (1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intensive tasks (TriviaQA, NaturalQuestions). (2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. In addition, it also surpasses DeepSeek Chat 7B. (3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1). However, it is worth noting that, after supervised finetuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. (4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations.\n\n",
+          "keyphrases": [
+            "DeepSeekMoE 16B",
+            "supervised fine-tuning",
+            "chat model",
+            "bilingual scenarios",
+            "evaluation benchmarks"
+          ],
+          "title": "Alignment For Deepseekmoe 16B",
+          "embedding": [
+            0.0034426983911544085,
+            -0.007639183662831783,
+            -0.011979537084698677,
+            0.00880228541791439,
+            0.011939010582864285,
+            0.013551952317357063,
+            -0.013365532271564007,
+            0.05112782120704651,
+            -0.01517299935221672,
+            0.02819000743329525,
+            -0.013219637796282768,
+            0.0074000791646540165,
+            -0.008145760744810104,
+            0.00818223413079977,
+            0.01559447217732668,
+            -0.004850983154028654,
+            0.02065213769674301,
+            0.04480573907494545,
+            0.0024822282139211893,
+            -0.021673398092389107,
+            -0.019825404509902,
+            -0.0646149292588234,
+            -0.037348922342061996,
+            0.011112277396023273,
+            0.0042836167849600315,
+            0.004996877163648605,
+            0.019955087453126907,
+            0.009685755707323551,
+            -0.005908716004341841,
+            0.017928779125213623,
+            -0.009920807555317879,
+            0.01728036068379879,
+            0.021122241392731667,
+            -0.012060589157044888,
+            -0.010982593521475792,
+            -0.028498005121946335,
+            0.05323518067598343,
+            0.011598590761423111,
+            -0.01838267222046852,
+            0.021511293947696686,
+            -0.009807334281504154,
+            0.016680574044585228,
+            -0.01202006358653307,
+            -0.014605632983148098,
+            0.0015673497691750526,
+            -0.021851712837815285,
+            -0.03183736279606819,
+            -0.01961466856300831,
+            -0.0014396923361346126,
+            -0.032809991389513016,
+            -0.012806271202862263,
+            -0.0026564907748252153,
+            -0.007140711881220341,
+            0.010480068624019623,
+            -0.033263884484767914,
+            0.034593142569065094,
+            0.045778367668390274,
+            0.03880786523222923,
+            0.04791814833879471,
+            -0.014200371690094471,
+            0.028481794521212578,
+            -0.0202792976051569,
+            0.006269399076700211,
+            -0.01276574470102787,
+            -0.017247939482331276,
+            -0.04185543209314346,
+            -0.018755514174699783,
+            -0.01043954212218523,
+            -0.004026275593787432,
+            0.014378686435520649,
+            -0.025904331356287003,
+            0.03783523663878441,
+            -0.018204357475042343,
+            -0.008129550144076347,
+            0.0036027769092470407,
+            0.0015926785999909043,
+            0.02387802116572857,
+            0.028433164581656456,
+            -0.003852012800052762,
+            -0.03744618594646454,
+            0.015083841979503632,
+            0.006763818673789501,
+            -0.014313844963908195,
+            0.028838425874710083,
+            0.06814881414175034,
+            -0.034366194158792496,
+            -0.024753388017416,
+            -0.07748604565858841,
+            0.006536871660500765,
+            0.0009523651096969843,
+            -0.027719903737306595,
+            -0.019079722464084625,
+            -0.01313048042356968,
+            -0.028757372871041298,
+            -0.025061385706067085,
+            0.0002973607915919274,
+            -0.033263884484767914,
+            -0.007444658316671848,
+            -0.045065104961395264,
+            0.016680574044585228,
+            0.025353174656629562,
+            0.017150677740573883,
+            0.010893435217440128,
+            -0.011955221183598042,
+            0.004008038435131311,
+            0.010763752274215221,
+            -0.020749401301145554,
+            -0.015805207192897797,
+            -0.004713193979114294,
+            0.08585064858198166,
+            -0.03151315450668335,
+            -0.012919744476675987,
+            -0.02702285349369049,
+            0.017572149634361267,
+            0.009450703859329224,
+            -0.010455752722918987,
+            -0.026390643790364265,
+            -0.03867818042635918,
+            0.0368950292468071,
+            -0.02243529073894024,
+            0.015707945451140404,
+            0.003924959804862738,
+            0.03371777758002281,
+            -0.0426335334777832,
+            -0.029713790863752365,
+            0.030427051708102226,
+            0.0018024016171693802,
+            -0.010585436597466469,
+            0.014921736903488636,
+            0.00511035043746233,
+            -0.017588360235095024,
+            0.0023201233707368374,
+            -0.011841747909784317,
+            0.0060262419283390045,
+            0.02198139764368534,
+            0.02209486998617649,
+            0.011525643989443779,
+            0.022613605484366417,
+            -0.01522973645478487,
+            -0.0017942963168025017,
+            0.0014224686892703176,
+            -0.03297209367156029,
+            0.015278367325663567,
+            0.0035744085907936096,
+            -0.012530692853033543,
+            0.008761758916079998,
+            0.025223489850759506,
+            0.06182673200964928,
+            0.020246876403689384,
+            0.004871245939284563,
+            -0.012619850225746632,
+            0.018868986517190933,
+            -0.028449373319745064,
+            0.024169810116291046,
+            0.0041985115967690945,
+            -0.03540366515517235,
+            0.01993887685239315,
+            0.0004736496484838426,
+            -0.012895428575575352,
+            -0.020311718806624413,
+            0.032647885382175446,
+            -0.031269997358322144,
+            -0.009588493034243584,
+            -0.0018652171129360795,
+            0.027266010642051697,
+            0.03141589090228081,
+            0.0100829117000103,
+            0.00386214442551136,
+            -0.04636194184422493,
+            0.037413764744997025,
+            -0.010585436597466469,
+            0.039650809019804,
+            -0.009531755931675434,
+            0.0054710335098207,
+            0.019792983308434486,
+            -0.006848923396319151,
+            0.008648285642266273,
+            0.007845867425203323,
+            -0.06597661226987839,
+            0.015432367101311684,
+            -0.029324740171432495,
+            0.00394724914804101,
+            0.02901674062013626,
+            -0.004611878655850887,
+            0.026828328147530556,
+            -0.0014569159829989076,
+            0.017491096630692482,
+            -0.0015399946132674813,
+            0.01872309297323227,
+            0.011639117263257504,
+            0.022013816982507706,
+            0.02780095487833023,
+            0.022954024374485016,
+            -0.005102245137095451,
+            0.011679643765091896,
+            -0.0150108952075243,
+            -0.012814376503229141,
+            -0.0010678647086024284,
+            0.008826600387692451,
+            0.010609752498567104,
+            0.003880381118506193,
+            -0.01966330036520958,
+            -0.01102311909198761,
+            -0.0032015678007155657,
+            0.018706882372498512,
+            0.005750664044171572,
+            -0.0019341117003932595,
+            0.021397819742560387,
+            0.011760695837438107,
+            -0.029308529570698738,
+            -0.02637443318963051,
+            -0.021738240495324135,
+            -0.011323013342916965,
+            -0.0371219739317894,
+            0.0024761492386460304,
+            0.017183097079396248,
+            0.016445521265268326,
+            0.001868256600573659,
+            0.013762688264250755,
+            -0.027006642892956734,
+            0.007493289653211832,
+            0.0007132606697268784,
+            -0.013008901849389076,
+            -0.04227690398693085,
+            -0.03841881453990936,
+            0.005268402863293886,
+            0.008648285642266273,
+            -0.0009052534005604684,
+            0.03841881453990936,
+            -0.009393966756761074,
+            0.01789635792374611,
+            0.031043048948049545,
+            0.006435556337237358,
+            0.012109220959246159,
+            0.014184161089360714,
+            -0.035144299268722534,
+            0.023829391226172447,
+            -0.007262290455400944,
+            0.021446451544761658,
+            0.033652935177087784,
+            -0.02037656120955944,
+            0.00043211033334955573,
+            -0.027930639684200287,
+            -0.00955607183277607,
+            0.026617590337991714,
+            -0.006220767740160227,
+            0.0009508453658781946,
+            0.017102045938372612,
+            0.025726014748215675,
+            0.033977143466472626,
+            0.009069757536053658,
+            -0.037640709429979324,
+            0.007237974554300308,
+            0.021187083795666695,
+            -0.00024974255939014256,
+            0.04269837588071823,
+            0.05255434289574623,
+            -0.03050810471177101,
+            -0.018739303573966026,
+            -0.054142966866493225,
+            0.01916077546775341,
+            -0.014848790131509304,
+            -0.011898485012352467,
+            0.014727211557328701,
+            0.007477079052478075,
+            0.009977543726563454,
+            0.02397528477013111,
+            0.017134467139840126,
+            0.004692931193858385,
+            0.012644166126847267,
+            -0.011306802742183208,
+            0.012003852985799313,
+            -0.049993086606264114,
+            0.007148817181587219,
+            0.005118455737829208,
+            -0.027557797729969025,
+            0.003432566998526454,
+            -0.010690804570913315,
+            0.0005987742333672941,
+            0.018804144114255905,
+            0.027557797729969025,
+            0.023780759423971176,
+            0.03151315450668335,
+            0.008931968361139297,
+            -0.046005312353372574,
+            -0.01732899248600006,
+            -0.021170873194932938,
+            -0.03909965232014656,
+            -0.005957347806543112,
+            -0.029875895008444786,
+            0.0010040359338745475,
+            -0.005418349523097277,
+            -0.036311451345682144,
+            -0.0011904563289135695,
+            -0.01503521017730236,
+            0.030443262308835983,
+            -0.017815306782722473,
+            0.04136911779642105,
+            0.056801486760377884,
+            -0.004449774045497179,
+            -0.06717618554830551,
+            0.009321019984781742,
+            -0.0016950072022154927,
+            -0.012392903678119183,
+            -0.018917618319392204,
+            0.014443528838455677,
+            0.020846663042902946,
+            0.03050810471177101,
+            -0.043541319668293,
+            -0.04386553168296814,
+            0.013949109241366386,
+            -0.0058844005689024925,
+            -0.01599973440170288,
+            -0.0034467510413378477,
+            -0.000564326997846365,
+            0.03206430748105049,
+            0.03540366515517235,
+            0.017572149634361267,
+            -0.038580916821956635,
+            -2.4299873985000886e-05,
+            0.01407068781554699,
+            0.02404012717306614,
+            -0.055407386273145676,
+            0.023343076929450035,
+            0.010544910095632076,
+            0.028660111129283905,
+            0.02919505536556244,
+            0.034204091876745224,
+            -0.015578261576592922,
+            5.2050803788006306e-05,
+            -0.017134467139840126,
+            -0.018009832128882408,
+            -0.01895003952085972,
+            0.0024700702633708715,
+            -0.027557797729969025,
+            -0.028676321730017662,
+            -0.041012488305568695,
+            -0.01684267818927765,
+            -0.03013526275753975,
+            0.006500398274511099,
+            -0.018868986517190933,
+            -0.0022836499847471714,
+            -0.010309858247637749,
+            0.01783151738345623,
+            -0.01559447217732668,
+            -0.0027334904298186302,
+            -0.023067498579621315,
+            -0.006881344132125378,
+            0.0018307699356228113,
+            -0.0031286205630749464,
+            0.0011549959890544415,
+            0.0025815172120928764,
+            0.030329788103699684,
+            0.030702630057930946,
+            -0.014362475834786892,
+            -0.02569359354674816,
+            -0.0497337207198143,
+            0.00334948836825788,
+            -0.005219771061092615,
+            0.018982460722327232,
+            0.012806271202862263,
+            -0.03747860714793205,
+            -0.039132073521614075,
+            0.010601647198200226,
+            -0.0371219739317894,
+            -0.034041985869407654,
+            0.01977677270770073,
+            0.012141642160713673,
+            -0.005207613576203585,
+            0.0029644896276295185,
+            -0.005458875559270382,
+            0.03174009919166565,
+            0.0023403866216540337,
+            -0.0022897287271916866,
+            0.024883070960640907,
+            0.02076561190187931,
+            0.008413233794271946,
+            -0.019760562106966972,
+            -0.005292718298733234,
+            0.0460701547563076,
+            -0.025077596306800842,
+            0.0008312931749969721,
+            -0.015659313648939133,
+            -0.029276108369231224,
+            0.017523517832159996,
+            0.021916555240750313,
+            0.000971615023445338,
+            -0.014005845412611961,
+            -0.03838639333844185,
+            0.027233589440584183,
+            0.015278367325663567,
+            -0.046556469053030014,
+            -0.0268121175467968,
+            0.0055723488330841064,
+            0.02869253046810627,
+            0.014848790131509304,
+            -0.010285543277859688,
+            0.006245083641260862,
+            0.02929231896996498,
+            -0.0056777168065309525,
+            -0.014905527234077454,
+            0.01545668300241232,
+            -0.02094392664730549,
+            -0.002857095329090953,
+            0.015278367325663567,
+            -6.775469955755398e-05,
+            0.008745548315346241,
+            -0.005000929813832045,
+            0.0182205680757761,
+            -0.0007907669642008841,
+            -0.006431503687053919,
+            -0.037737973034381866,
+            -0.025223489850759506,
+            -0.046102575957775116,
+            0.03439861536026001,
+            0.037089552730321884,
+            0.008737443014979362,
+            0.017491096630692482,
+            -0.01631583832204342,
+            -0.00244372827000916,
+            -0.033490829169750214,
+            0.01182553730905056,
+            0.02536938525736332,
+            0.005013087764382362,
+            0.004587562754750252,
+            0.0069948178716003895,
+            0.009726282209157944,
+            0.014500265009701252,
+            0.0004141268436796963,
+            -0.04347648099064827,
+            0.004790193866938353,
+            0.008003919385373592,
+            0.034982193261384964,
+            -0.031205154955387115,
+            -0.03125378489494324,
+            -0.011939010582864285,
+            0.010309858247637749,
+            0.036473557353019714,
+            -0.034528300166130066,
+            -0.0182205680757761,
+            -0.021835502237081528,
+            0.016048364341259003,
+            -0.006030294578522444,
+            -0.012441535480320454,
+            -0.0003432060475461185,
+            0.008429444395005703,
+            -0.012490167282521725,
+            0.01711825653910637,
+            -0.026115067303180695,
+            -8.656137651996687e-05,
+            -0.04493542015552521,
+            0.004437616094946861,
+            -0.06639808416366577,
+            -0.019760562106966972,
+            0.03543608635663986,
+            -0.024121178314089775,
+            0.0214302409440279,
+            -0.006366661749780178,
+            0.012247010134160519,
+            -0.05654211714863777,
+            0.002972594927996397,
+            -0.008510496467351913,
+            0.04182301089167595,
+            -0.0083078658208251,
+            -0.005134666338562965,
+            -0.034301355481147766,
+            0.002052650786936283,
+            -0.007704025600105524,
+            0.009985649026930332,
+            0.03146452084183693,
+            -0.014216582290828228,
+            -0.0016554942121729255,
+            0.012530692853033543,
+            0.00012430136848706752,
+            0.011955221183598042,
+            0.03073505125939846,
+            -0.017523517832159996,
+            -0.04227690398693085,
+            0.02071698009967804,
+            0.036797765642404556,
+            -0.021057400852441788,
+            -0.016080785542726517,
+            -0.02530454285442829,
+            0.0042836167849600315,
+            0.006447714287787676,
+            0.004769930616021156,
+            0.017685621976852417,
+            -0.0047739832662045956,
+            0.02065213769674301,
+            0.015059526078402996,
+            0.021900344640016556,
+            0.030054211616516113,
+            0.01636446826159954,
+            0.017636990174651146,
+            -0.017458675429224968,
+            -0.03783523663878441,
+            0.010966382920742035,
+            -0.03543608635663986,
+            0.0035926452837884426,
+            0.02747674658894539,
+            -0.03079989179968834,
+            -0.009645229205489159,
+            -0.006646292749792337,
+            4.45787918579299e-05,
+            0.013138585723936558,
+            0.0061478205025196075,
+            0.0366680808365345,
+            0.011031224392354488,
+            0.005341349635273218,
+            -0.0007411224069073796,
+            0.028887057676911354,
+            -0.015391840599477291,
+            0.035144299268722534,
+            -0.015853838995099068,
+            -0.01553773507475853,
+            0.006549029611051083,
+            -0.010520595125854015,
+            0.019258037209510803,
+            -0.008206550031900406,
+            -0.0043200901709496975,
+            -0.022662237286567688,
+            -0.008769864216446877,
+            0.0071204486303031445,
+            0.016542784869670868,
+            -0.019484983757138252,
+            0.02603401429951191,
+            -0.0005334257730282843,
+            -0.014865000732243061,
+            0.01950119435787201,
+            -0.03024873696267605,
+            -0.023294445127248764,
+            0.0031002522446215153,
+            -0.003789197187870741,
+            0.003740565851330757,
+            -0.005426454823464155,
+            -0.0024072546511888504,
+            -0.015772787854075432,
+            -0.002682832768186927,
+            -0.02120329439640045,
+            -0.014305739663541317,
+            -0.006678713485598564,
+            -0.005000929813832045,
+            -0.01509194727987051,
+            0.002642306499183178,
+            -0.023861810564994812,
+            0.007683762814849615,
+            0.0323074646294117,
+            -0.03922933712601662,
+            0.019857825711369514,
+            0.03627903014421463,
+            0.003473093034699559,
+            -0.018917618319392204,
+            -0.022581184282898903,
+            -0.010917751118540764,
+            -0.010042386129498482,
+            -0.05044697970151901,
+            0.001210719463415444,
+            0.0015308762667700648,
+            -0.0021843607537448406,
+            -0.022921603173017502,
+            0.0013393900590017438,
+            0.016583310440182686,
+            -0.012676587328314781,
+            0.012247010134160519,
+            -0.005045508500188589,
+            -0.014727211557328701,
+            0.014832579530775547,
+            0.027768533676862717,
+            -0.02679590694606304,
+            -0.0029462529346346855,
+            -0.0036088558845221996,
+            0.0033839354291558266,
+            0.0045835101045668125,
+            -0.008931968361139297,
+            -0.007704025600105524,
+            0.033750198781490326,
+            0.001635231077671051,
+            -0.034917350858449936,
+            -0.012911639176309109,
+            -0.002362675964832306,
+            0.027444325387477875,
+            -0.024769596755504608,
+            -0.004899614490568638,
+            0.03180494159460068,
+            0.026293382048606873,
+            -0.014881211332976818,
+            -0.03130241855978966,
+            -0.028060322627425194,
+            0.02021445520222187,
+            0.00838891789317131,
+            0.025888120755553246,
+            0.00861586444079876,
+            0.0009341283002868295,
+            -0.0538511797785759,
+            0.01899867132306099,
+            -0.018690671771764755,
+            -0.007529763039201498,
+            -0.003687881864607334,
+            0.02055487595498562,
+            -0.07592984288930893,
+            -0.03456072136759758,
+            0.03462556377053261,
+            0.033652935177087784,
+            -0.011104172095656395,
+            -0.00919133611023426,
+            0.012263220734894276,
+            -0.002164097735658288,
+            0.034982193261384964,
+            0.006196451839059591,
+            0.023521391674876213,
+            0.009596598334610462,
+            -0.0013880215119570494,
+            0.011841747909784317,
+            -0.01700478233397007,
+            0.0011874169576913118,
+            -0.02775232493877411,
+            0.016032153740525246,
+            -0.005941137205809355,
+            -0.004397090058773756,
+            -0.02215971238911152,
+            0.051646556705236435,
+            -0.014297634363174438,
+            0.034366194158792496,
+            -0.004624036606401205,
+            0.02083045430481434,
+            0.05394844338297844,
+            0.016972361132502556,
+            -0.03683018684387207,
+            0.0030739102512598038,
+            -0.018301621079444885,
+            -0.041174594312906265,
+            0.012368588708341122,
+            -0.022192133590579033,
+            -0.02598538249731064,
+            0.029227476567029953,
+            0.0009467927156947553,
+            0.006654397584497929,
+            -0.0027719903737306595,
+            0.0004359096637926996,
+            -0.008567233569920063,
+            -0.013673530891537666,
+            0.017523517832159996,
+            -0.006642240099608898,
+            0.04192027449607849,
+            -0.009977543726563454,
+            -0.008364601992070675,
+            0.002861147979274392,
+            0.013357426971197128,
+            0.045000262558460236,
+            0.019582247361540794,
+            -0.016923731192946434,
+            0.03146452084183693,
+            0.022062448784708977,
+            0.03715439513325691,
+            -0.018804144114255905,
+            -0.05135476961731911,
+            0.006358556915074587,
+            -0.0009148783865384758,
+            0.016080785542726517,
+            0.0018115199636667967,
+            0.022564973682165146,
+            0.001599770737811923,
+            0.01789635792374611,
+            -0.0019016907317563891,
+            0.04386553168296814,
+            -0.009167020209133625,
+            -0.009653334505856037,
+            0.027930639684200287,
+            -0.0005764848319813609,
+            0.0336853563785553,
+            0.02039276994764805,
+            -0.03511187806725502,
+            -0.034755248576402664,
+            0.01838267222046852,
+            0.026439275592565536,
+            -0.02237044833600521,
+            0.009102178737521172,
+            0.003746644826605916,
+            0.023991495370864868,
+            -0.014654264785349369,
+            0.012530692853033543,
+            0.0036919345147907734,
+            -0.013195321895182133,
+            -0.024964123964309692,
+            0.015043315477669239,
+            0.004052617587149143,
+            -0.0059006111696362495,
+            0.035533349961042404,
+            -0.03462556377053261,
+            -0.03362051397562027,
+            -0.00743655301630497,
+            -0.007071817293763161,
+            -0.0012198378099128604,
+            0.0371219739317894,
+            -0.005462928209453821,
+            0.01877172477543354,
+            -0.03118894435465336,
+            0.040850382298231125,
+            0.00765539426356554,
+            0.004688878543674946,
+            -0.0071731326170265675,
+            0.0016960203647613525,
+            -0.009977543726563454,
+            0.014257107861340046,
+            0.007736446801573038,
+            -0.007716183550655842,
+            0.0018084804760292172,
+            -0.011006908491253853,
+            -0.007290658541023731,
+            0.01642931066453457,
+            -0.027541587129235268,
+            -0.008518601767718792,
+            0.02259739488363266,
+            0.03465798497200012,
+            0.0007927932892926037,
+            0.001413350342772901,
+            -0.00983165018260479,
+            0.006362609565258026,
+            -0.02232181653380394,
+            0.04299016669392586,
+            -0.02138160914182663,
+            0.014751527458429337,
+            0.01590247079730034,
+            -0.042179640382528305,
+            -0.021446451544761658,
+            0.026050224900245667,
+            0.020198244601488113,
+            -0.018415093421936035,
+            -0.011225750669836998,
+            -0.014962263405323029,
+            0.016810256987810135,
+            -0.03068641945719719,
+            -0.04292532429099083,
+            0.010982593521475792,
+            -0.0015845734160393476,
+            -0.0047050886787474155,
+            -0.01595110259950161,
+            0.018690671771764755,
+            0.01085290964692831,
+            -0.054467178881168365,
+            -0.00502524571493268,
+            0.010009964928030968,
+            0.006107294466346502,
+            0.018706882372498512,
+            0.004672667942941189,
+            0.03362051397562027,
+            0.0006463924655690789,
+            0.01838267222046852,
+            -0.007898551411926746,
+            0.01307374332100153,
+            -0.010950172320008278,
+            -0.019371511414647102,
+            0.03279377892613411,
+            0.011420276015996933,
+            -0.03266409784555435,
+            -0.0044254581443965435,
+            0.0021742291282862425,
+            -0.016631942242383957,
+            0.002182334428653121,
+            -0.015878155827522278,
+            -0.006184294354170561,
+            0.028984319418668747,
+            0.00685297604650259,
+            0.025239700451493263,
+            0.029616527259349823,
+            -0.0035865663085132837,
+            -0.01581331342458725,
+            0.012133536860346794,
+            0.009718176908791065,
+            -0.01160669606178999,
+            0.022013816982507706,
+            -0.031091680750250816,
+            0.035371243953704834,
+            -0.06166462600231171,
+            0.030929576605558395,
+            0.027671271935105324,
+            -0.016794046387076378,
+            0.003426488023251295,
+            -0.021867923438549042,
+            0.01766941137611866,
+            -0.01883656531572342,
+            0.029827265068888664,
+            -0.020360350608825684,
+            0.02913021482527256,
+            -0.021170873194932938,
+            0.021576134487986565,
+            0.002658517099916935,
+            -0.018674461171030998,
+            0.008060656487941742,
+            0.0008348391856998205,
+            -0.010998803190886974,
+            -0.013600584119558334,
+            -0.029454423114657402,
+            0.012425324879586697,
+            0.058292847126722336,
+            -0.033490829169750214,
+            -0.0017031125025823712,
+            0.018917618319392204,
+            -0.02259739488363266,
+            -0.03152936324477196,
+            0.0100829117000103,
+            0.004534878768026829,
+            0.030864734202623367,
+            0.014240897260606289,
+            -0.04075312241911888,
+            -0.04172574728727341,
+            0.008324076421558857,
+            0.010350384749472141,
+            -0.004243090283125639,
+            0.046264681965112686,
+            0.01040712185204029,
+            0.014840684831142426,
+            0.01041522715240717,
+            0.0040566702373325825,
+            0.001815572613850236,
+            0.007890446111559868,
+            -0.007570289541035891,
+            -0.012587429955601692,
+            -0.001791256945580244,
+            -0.028498005121946335,
+            -0.005240034312009811,
+            0.017361413687467575,
+            -0.011598590761423111,
+            -0.010204490274190903,
+            -0.04503268375992775,
+            -0.01572415605187416,
+            0.017069624736905098,
+            -0.003262357087805867,
+            0.04668615385890007,
+            0.013171006925404072,
+            -0.004676720593124628,
+            -0.01933909021317959,
+            -0.02076561190187931,
+            0.04619983956217766,
+            -0.017636990174651146,
+            0.04000743851065636,
+            -0.0015734287444502115,
+            -0.013535741716623306,
+            -0.025110017508268356,
+            0.033879879862070084,
+            -0.03728407993912697,
+            -0.013957214541733265,
+            -0.008117392659187317,
+            -0.0035359086468815804,
+            -0.005458875559270382,
+            0.0018409014446660876,
+            -0.0008019116939976811,
+            -0.030167683959007263,
+            0.024396756663918495,
+            -0.025336964055895805,
+            0.031983256340026855,
+            0.02055487595498562,
+            0.010123438201844692,
+            0.00048200818127952516,
+            -0.03222641348838806,
+            -0.005969505291432142,
+            0.0005481165135279298,
+            0.008761758916079998,
+            -0.0011539828265085816,
+            0.00394724914804101,
+            -0.015934891998767853,
+            -0.01356816291809082,
+            -0.012287535704672337,
+            -0.04253627359867096,
+            -0.015480998903512955,
+            0.021089820191264153,
+            0.010374700650572777,
+            -0.03311799094080925,
+            -0.017636990174651146,
+            -0.03653839975595474,
+            0.006840818095952272,
+            -0.010625963099300861,
+            -0.008680706843733788,
+            0.012473956681787968,
+            0.001798348966985941,
+            0.015019000507891178,
+            -0.005479138810187578,
+            -0.023245813325047493,
+            0.026487907394766808,
+            -0.03219399228692055,
+            -0.029486844316124916,
+            -0.01043954212218523,
+            -0.003635197877883911,
+            -0.010042386129498482,
+            0.031059259548783302,
+            0.021251926198601723,
+            0.0007943130331113935,
+            -0.050479400902986526,
+            -0.03050810471177101,
+            -0.051873501390218735,
+            0.024396756663918495,
+            0.0009315954521298409,
+            -0.00938586238771677,
+            0.003252225462347269,
+            0.02039276994764805,
+            0.008178181946277618,
+            -0.01313048042356968,
+            -0.02603401429951191,
+            -0.024023916572332382,
+            -0.013884266838431358,
+            -0.005195455625653267,
+            -0.029065372422337532,
+            -0.033263884484767914,
+            -0.037089552730321884,
+            -0.01334932167083025,
+            0.005916821304708719,
+            0.011776906438171864,
+            -0.003000963246449828,
+            0.0008049511234275997,
+            -0.006759766023606062,
+            -0.02459128201007843,
+            0.004125564359128475,
+            0.015205420553684235,
+            -0.01998750865459442,
+            0.02708769403398037,
+            0.009629018604755402,
+            0.0141517398878932,
+            -0.029843473806977272,
+            0.013859951868653297,
+            0.009118389338254929,
+            0.014338160865008831,
+            -0.004976614378392696,
+            -0.003906723111867905,
+            0.02060350775718689,
+            -0.01972814090549946,
+            0.026017803698778152,
+            -0.006350451614707708,
+            -0.0024700702633708715,
+            -0.010066702030599117,
+            0.020360350608825684,
+            0.003473093034699559,
+            0.012725219130516052,
+            0.022224554792046547,
+            -2.4996417778311297e-05,
+            0.037089552730321884,
+            -0.005090087652206421,
+            -0.01326016429811716,
+            0.00575876934453845,
+            -0.02121950499713421,
+            -0.005702032707631588,
+            0.005993821192532778,
+            -0.029000530019402504,
+            0.0012887322809547186,
+            0.014792053960263729,
+            0.009880281053483486,
+            -0.0318535715341568,
+            0.023634864017367363,
+            0.02603401429951191,
+            0.006848923396319151,
+            0.010399016551673412,
+            -0.033815041184425354,
+            0.0006266359705477953,
+            0.011274381540715694,
+            -0.004778035916388035,
+            0.036700502038002014,
+            0.0201658234000206,
+            -0.03614934906363487,
+            0.010998803190886974,
+            -0.01043954212218523,
+            0.00396548630669713,
+            -0.004765877965837717,
+            0.029616527259349823,
+            -0.022338027134537697,
+            -0.011006908491253853,
+            -0.029324740171432495,
+            0.010180175304412842,
+            -0.021948976442217827,
+            -0.015497208572924137,
+            -0.03524155914783478,
+            0.03579271584749222,
+            0.005811453331261873,
+            -0.013551952317357063,
+            0.0029989369213581085,
+            -0.02039276994764805,
+            -0.0071690804325044155,
+            -0.02951926551759243,
+            -0.012660376727581024,
+            -0.012230799533426762,
+            -0.0007537868223153055,
+            0.016810256987810135,
+            0.02491549216210842,
+            0.006439608987420797,
+            0.029389580711722374,
+            0.03728407993912697,
+            0.00218841340392828,
+            -0.014167950488626957,
+            0.06169704720377922,
+            -0.0007922867080196738,
+            -0.024607492610812187,
+            0.04075312241911888,
+            -0.024964123964309692,
+            0.0016544810496270657,
+            0.01625099591910839,
+            0.014743422158062458,
+            -0.03211294114589691,
+            0.016923731192946434,
+            -0.006739502772688866,
+            -0.033815041184425354,
+            0.003924959804862738,
+            -0.005171139724552631,
+            -0.03195083513855934,
+            -0.04185543209314346,
+            -0.010115332901477814,
+            0.004879351239651442,
+            -0.02160855568945408,
+            0.017020992934703827,
+            0.020473822951316833,
+            -0.002723358804360032,
+            -0.014500265009701252,
+            0.02275949902832508,
+            0.001914861728437245,
+            0.006970501970499754,
+            -0.006771923508495092,
+            0.007833709008991718,
+            -0.0027253851294517517,
+            -0.0010035294108092785,
+            -0.024834439158439636,
+            -0.015140579082071781,
+            -0.041239432990550995,
+            0.0159267857670784,
+            -0.023780759423971176,
+            -0.046880677342414856,
+            0.0056412434205412865,
+            -0.03744618594646454,
+            0.03355567157268524,
+            0.017474886029958725,
+            0.0005683795898221433,
+            -0.0033434093929827213,
+            -0.007716183550655842,
+            -0.021689608693122864,
+            0.010917751118540764,
+            0.022856762632727623,
+            0.004316037520766258,
+            -0.0059897685423493385,
+            0.014192266389727592,
+            -0.022305605933070183,
+            0.000483021343825385,
+            -0.060335367918014526,
+            -0.02381318062543869,
+            0.025628753006458282,
+            0.007983656600117683,
+            0.0005856032366864383,
+            -0.014824474230408669,
+            0.006808397360146046,
+            -0.03266409784555435,
+            -0.019420143216848373,
+            0.025126228109002113,
+            0.019744351506233215,
+            0.014086898416280746,
+            -0.004960403777658939,
+            0.0007396026630885899,
+            0.0182205680757761,
+            -0.008850916288793087,
+            0.008980600163340569,
+            0.0030556735582649708,
+            0.025045175105333328,
+            0.00021225583623163402,
+            0.0077931834384799,
+            0.01210111565887928,
+            0.028822215273976326,
+            0.0034710667096078396,
+            0.010544910095632076,
+            0.006038399878889322,
+            0.0009300757083110511,
+            -0.0134060587733984,
+            -0.013276374898850918,
+            -0.008153866045176983,
+            0.024137388914823532,
+            0.018852775916457176,
+            0.00955607183277607,
+            -0.009580387733876705,
+            -0.009880281053483486,
+            0.008956284262239933,
+            0.009004916064441204,
+            -0.001745664980262518,
+            0.01223890483379364,
+            0.012117326259613037,
+            -0.029778633266687393,
+            -0.03355567157268524,
+            -0.0013799162115901709,
+            -0.004984719213098288,
+            0.005795242730528116,
+            0.00802012998610735,
+            0.021948976442217827,
+            0.0013181137619540095,
+            -0.013479005545377731,
+            -0.004980667028576136,
+            0.02530454285442829,
+            0.005742558743804693,
+            -0.01317911222577095,
+            -0.015716049820184708,
+            -0.009766807779669762,
+            0.00772834150120616,
+            0.011193329468369484,
+            -0.003473093034699559,
+            0.0033596197608858347,
+            0.012741428799927235,
+            0.08163592219352722,
+            -0.007533815689384937,
+            0.0036818028893321753,
+            -0.03854849562048912,
+            -0.01284679677337408,
+            -0.008640180341899395,
+            0.019484983757138252,
+            -0.0019867955707013607,
+            0.032323677092790604,
+            0.015051420778036118,
+            0.01849614642560482,
+            0.0059208739548921585,
+            0.015326999127864838,
+            0.005365665536373854,
+            -0.02929231896996498,
+            0.02227318473160267,
+            -0.0005982676520943642,
+            0.011817432008683681,
+            0.0019857825245708227,
+            -0.011979537084698677,
+            -0.011006908491253853,
+            0.02032792940735817,
+            0.003671671263873577,
+            0.02397528477013111,
+            -0.02159234508872032,
+            0.019744351506233215,
+            0.01849614642560482,
+            0.011979537084698677,
+            -0.010301753878593445,
+            -0.005004982464015484,
+            0.0014913631603121758,
+            0.00517924502491951,
+            -0.017037203535437584,
+            -0.0016139548970386386,
+            -0.019582247361540794,
+            0.06652776896953583,
+            -0.030556734651327133,
+            0.0007244053413160145,
+            0.0031346995383501053,
+            -0.018577197566628456,
+            0.004713193979114294,
+            0.001658533699810505,
+            0.01659141480922699,
+            0.0057263486087322235,
+            -0.006581450812518597,
+            0.005531822796911001,
+            0.024688545614480972,
+            -0.00560882268473506,
+            0.01762077957391739,
+            0.005523717496544123,
+            -0.012263220734894276,
+            -0.014638054184615612,
+            0.012076799757778645,
+            0.011939010582864285,
+            0.009142705239355564,
+            0.025904331356287003,
+            0.008012024685740471,
+            -0.009880281053483486,
+            -0.020895294845104218,
+            -0.011582380160689354,
+            0.006151873152703047,
+            0.005876295268535614,
+            0.03815944492816925,
+            0.030216315761208534,
+            -0.0006884383619762957,
+            -0.013592478819191456,
+            0.016259100288152695,
+            -0.005365665536373854,
+            0.024169810116291046,
+            -0.0029259896837174892,
+            -0.013608689419925213,
+            0.008980600163340569,
+            0.009564177133142948,
+            -0.008178181946277618,
+            -0.02747674658894539,
+            -0.011582380160689354,
+            -0.01013964880257845,
+            -0.01581331342458725,
+            0.010099122300744057,
+            -0.027071483433246613,
+            -0.002474122913554311,
+            0.0011884300038218498,
+            0.027039064094424248,
+            0.03738134354352951,
+            -0.011768801137804985,
+            -0.00539808627218008,
+            -0.0008814443135634065,
+            -0.025223489850759506,
+            -0.005000929813832045,
+            -0.020749401301145554,
+            -0.024964123964309692,
+            -0.016810256987810135,
+            -0.016105102375149727,
+            -0.005527770146727562,
+            -0.0029178846161812544,
+            0.004474089480936527,
+            0.0015045341569930315,
+            0.058130744844675064,
+            -0.01102311909198761,
+            -0.012084905058145523,
+            -0.02076561190187931,
+            0.006942133419215679,
+            0.019128354266285896,
+            -0.011387854814529419,
+            0.0027274114545434713,
+            -0.01393289864063263,
+            0.004010064993053675,
+            -0.014881211332976818,
+            -0.014362475834786892,
+            -0.023002656176686287,
+            0.002506543882191181,
+            -0.006455819588154554,
+            0.0028084637597203255,
+            -0.00627345172688365,
+            0.009726282209157944,
+            0.019857825711369514,
+            -3.7233421608107165e-05,
+            0.001175259007140994,
+            -0.008810389786958694,
+            0.015043315477669239,
+            -0.030491894111037254,
+            0.02392665296792984,
+            0.019144564867019653,
+            0.0033373304177075624,
+            -0.0053778234869241714,
+            0.02752537652850151,
+            0.00364127685315907,
+            -0.011720169335603714,
+            0.04568110406398773,
+            0.010399016551673412,
+            0.026179907843470573,
+            -0.009872175753116608,
+            0.005774979945272207,
+            -0.0181070938706398,
+            0.009231862612068653,
+            -0.006484187673777342,
+            0.0072420272044837475,
+            -0.006131610367447138,
+            0.02780095487833023,
+            -0.008777969516813755,
+            0.0013464820804074407,
+            0.002717280061915517,
+            0.04098006710410118,
+            -0.0019452563719823956,
+            0.015853838995099068,
+            -0.026617590337991714,
+            -0.00035536388168111444,
+            0.02126813679933548,
+            0.014362475834786892,
+            0.017183097079396248,
+            -0.000494925887323916,
+            -0.01950119435787201,
+            -0.0045997207053005695,
+            -0.014321950264275074,
+            0.0009179178159683943,
+            -0.030005579814314842,
+            0.002401175908744335,
+            -0.00473345723003149,
+            0.013187216594815254,
+            0.013876162469387054,
+            -0.011695853434503078,
+            0.02044140174984932,
+            -0.017701832577586174,
+            -0.02376454882323742,
+            0.008964389562606812,
+            0.0018135462887585163,
+            0.01700478233397007,
+            0.017020992934703827,
+            -0.022030027583241463,
+            -0.006982659921050072,
+            -0.01883656531572342,
+            -0.01849614642560482,
+            0.0032562781125307083,
+            0.034041985869407654,
+            -0.007096133194863796,
+            -0.010828593745827675,
+            0.018188146874308586,
+            -0.01750730723142624,
+            0.01732899248600006,
+            -0.007250132504850626,
+            0.01954982616007328,
+            -0.001856098766438663,
+            -0.006609818898141384,
+            0.005240034312009811,
+            -0.02358623407781124,
+            0.0012897454435005784,
+            0.00999375432729721,
+            0.001181337982416153,
+            -0.03229125589132309,
+            0.012036273255944252,
+            -0.0020678481087088585,
+            0.001809493638575077,
+            -0.006484187673777342,
+            0.014767738059163094,
+            0.006788134109228849,
+            0.00869691651314497,
+            -0.002474122913554311,
+            0.010074807330965996,
+            0.003671671263873577,
+            -0.015083841979503632,
+            -0.003112410195171833,
+            0.018593408167362213,
+            0.008599653840065002,
+            -0.011614801362156868,
+            0.00575876934453845,
+            -0.014946052804589272,
+            -0.016080785542726517,
+            0.019258037209510803,
+            0.019760562106966972,
+            -0.031432099640369415,
+            0.022791920229792595,
+            -0.024461599066853523,
+            -0.0035521192476153374,
+            0.017977410927414894,
+            -0.0020972294732928276,
+            -0.0008490233449265361,
+            -0.004474089480936527,
+            -0.027557797729969025,
+            -0.01428952906280756,
+            0.005924926605075598,
+            -0.011525643989443779,
+            0.00452677346765995,
+            0.025288332253694534,
+            0.015691734850406647,
+            -0.0056250328198075294,
+            -0.0016858888557180762,
+            0.030832313001155853,
+            -0.003071883926168084,
+            0.0054831914603710175,
+            0.005126561038196087,
+            -0.028870847076177597,
+            0.02995694801211357,
+            -0.024056337773799896,
+            0.007550026290118694,
+            -0.000363722414476797,
+            -0.00247817556373775,
+            -0.006358556915074587,
+            0.014411107636988163,
+            -0.0005481165135279298,
+            0.0016200337558984756,
+            -0.014475949108600616,
+            0.006269399076700211,
+            -0.019355300813913345,
+            0.030864734202623367,
+            -0.0016088890843093395,
+            -0.00017768194084055722,
+            0.015197315253317356,
+            -0.015205420553684235,
+            -0.017863936722278595,
+            0.01013964880257845,
+            0.008453760296106339,
+            -0.0230350773781538,
+            0.004778035916388035,
+            -0.01933909021317959,
+            -0.03871060162782669,
+            -0.026115067303180695,
+            0.003706118557602167,
+            0.008915758691728115,
+            0.0165022574365139,
+            -0.009426387958228588,
+            -0.0061113471165299416,
+            0.02692558988928795,
+            0.02358623407781124,
+            0.03378261998295784,
+            -0.013949109241366386,
+            0.009094073437154293,
+            0.0027963060420006514,
+            0.01317911222577095,
+            0.001355600543320179,
+            -0.004384932108223438,
+            0.0033839354291558266,
+            -0.004247142933309078,
+            -0.012587429955601692,
+            -0.0018044279422610998,
+            0.027606429532170296,
+            -0.006650345399975777,
+            -0.024380546063184738,
+            -0.03151315450668335,
+            0.003161041531711817,
+            -0.005231929011642933,
+            0.0040222229436039925,
+            0.03971565142273903,
+            -0.024510230869054794,
+            -0.02587191015481949,
+            0.02026308700442314,
+            -0.003205620450899005,
+            0.004567299969494343,
+            -0.0019847694784402847,
+            0.01027743797749281,
+            -0.014775843359529972,
+            -0.028643900528550148,
+            -0.022192133590579033,
+            -0.0027193063870072365,
+            0.0037425921764224768,
+            -0.007128553930670023,
+            -0.016972361132502556,
+            -0.007886393927037716,
+            0.007622973527759314,
+            -0.016510363668203354,
+            0.013762688264250755,
+            -0.02237044833600521,
+            0.010147754102945328,
+            -0.0012299694353714585,
+            0.030653998255729675,
+            -0.015513419173657894,
+            -0.01608889177441597,
+            -0.023894231766462326,
+            0.007254185155034065,
+            0.02099255844950676,
+            -0.0009913715766742826,
+            0.0012026141630485654,
+            0.006780028808861971,
+            0.005292718298733234,
+            0.007197448518127203,
+            -0.0069948178716003895,
+            0.019063511863350868,
+            -0.034820087254047394,
+            -0.0024356229696422815,
+            0.0019047302193939686,
+            0.039521124213933945,
+            -0.027168747037649155,
+            -0.003321120049804449,
+            -0.007817499339580536,
+            0.010228806175291538,
+            0.013519532047212124,
+            0.013162901625037193,
+            -0.0027679377235472202,
+            -0.01677783578634262,
+            -0.022451501339673996,
+            0.0005597677663899958,
+            -0.038516074419021606,
+            0.0026544644497334957,
+            0.015197315253317356,
+            -0.0013991661835461855,
+            0.03751102834939957,
+            0.0469779409468174,
+            -0.03738134354352951,
+            -0.006269399076700211,
+            0.009799228981137276,
+            0.03410682827234268,
+            -0.020911505445837975,
+            0.020408980548381805,
+            -0.011063645593822002,
+            0.01620236411690712,
+            0.0035602243151515722,
+            -0.009085968136787415,
+            -0.02786579728126526,
+            -0.014905527234077454,
+            0.021089820191264153,
+            0.008145760744810104,
+            -0.021478872746229172,
+            0.009734386578202248,
+            0.011558065190911293,
+            -0.008364601992070675,
+            -0.025061385706067085,
+            0.02598538249731064,
+            0.01883656531572342,
+            0.01157427579164505,
+            -0.018204357475042343,
+            -0.011768801137804985,
+            -0.02674727514386177,
+            0.008996810764074326,
+            -0.020846663042902946,
+            0.01723172888159752,
+            -0.023861810564994812,
+            0.0008839771617203951,
+            -0.01844751462340355,
+            -0.007225817069411278,
+            0.006532819475978613,
+            -0.04370342567563057,
+            -0.006476082839071751,
+            -0.005924926605075598,
+            0.012960270047187805,
+            0.028951898217201233,
+            0.021414030343294144,
+            -0.014394897036254406,
+            -0.02890326827764511,
+            -0.003379883011803031,
+            -0.0073960269801318645,
+            0.013446584343910217,
+            -0.0059208739548921585,
+            -0.008397023193538189,
+            -0.0022654130589216948,
+            0.008332181721925735,
+            -0.010026175528764725,
+            0.007622973527759314,
+            0.006082978565245867,
+            0.001210719463415444,
+            0.018593408167362213,
+            -0.0008186287013813853,
+            -0.013316900469362736,
+            -0.0033900144044309855,
+            -0.008542917668819427,
+            0.02083045430481434,
+            0.03355567157268524,
+            0.003979670349508524,
+            -0.007383869029581547,
+            -0.00466456264257431,
+            0.004648352041840553,
+            0.020684558898210526,
+            -0.015416156500577927,
+            -0.0041620382107794285,
+            -0.003440672066062689,
+            0.01307374332100153,
+            -0.011736379936337471,
+            0.027039064094424248,
+            -0.013235848397016525,
+            -0.023521391674876213,
+            -0.022030027583241463,
+            -0.0559261217713356,
+            0.006605766247957945,
+            -0.028400743380188942,
+            -0.03096199780702591,
+            -0.01581331342458725,
+            -0.012287535704672337,
+            -0.01760457083582878,
+            0.011849853210151196,
+            -0.005697980057448149,
+            -0.0032603307627141476,
+            0.005965452641248703,
+            0.013219637796282768,
+            -0.016380678862333298,
+            0.0008576351683586836,
+            0.00511035043746233,
+            0.0010156872449442744,
+            -0.01922561787068844,
+            0.016599521040916443,
+            0.009361546486616135,
+            -0.01223890483379364,
+            -0.013892372138798237,
+            0.0021661240607500076,
+            0.013187216594815254,
+            -0.004591615404933691,
+            0.006301820278167725,
+            0.015562050975859165,
+            0.010180175304412842,
+            -0.0214302409440279,
+            -0.0075540789403021336,
+            0.005065771751105785,
+            -0.007059659343212843,
+            0.01487310603260994,
+            -0.015667418017983437,
+            -0.006885396782308817,
+            0.0026321751065552235,
+            0.012076799757778645,
+            -0.018463725224137306,
+            -0.007343342527747154,
+            -0.00350348767824471,
+            3.1661072625865927e-06,
+            -0.018739303573966026,
+            0.0003079989110119641,
+            0.012919744476675987,
+            0.0008267339435406029,
+            0.008129550144076347,
+            -0.01317911222577095,
+            0.003142804838716984,
+            0.028384532779455185,
+            0.019760562106966972,
+            -0.037024714052677155,
+            0.0013272322248667479,
+            0.014362475834786892,
+            -0.009150810539722443,
+            -0.01989024691283703,
+            0.019063511863350868,
+            0.02298644557595253,
+            0.002540991175919771,
+            0.0017254018457606435,
+            -0.0023464655969291925,
+            -0.003270462155342102,
+            -0.01865825057029724,
+            0.011566170491278172,
+            0.01116090826690197,
+            -0.005843874532729387,
+            -0.0012887322809547186,
+            0.006366661749780178,
+            -0.01642931066453457,
+            -0.006326135713607073,
+            0.007789130788296461,
+            0.014856895431876183,
+            0.007489237003028393,
+            0.009499335661530495,
+            0.010561120696365833,
+            0.004575405269861221,
+            0.0015967312501743436,
+            0.024104967713356018,
+            0.07385490089654922,
+            0.0013819425366818905,
+            0.008380812592804432,
+            0.018852775916457176,
+            -0.006504450924694538,
+            -0.013187216594815254,
+            0.005860084667801857,
+            -0.008016076870262623,
+            -3.7233421608107165e-05,
+            0.005240034312009811,
+            -0.04072070121765137,
+            -0.02165718749165535,
+            -0.010877225548028946,
+            -0.013381742872297764,
+            0.004591615404933691,
+            0.018042253330349922,
+            -0.01799362152814865,
+            0.01590247079730034,
+            0.03793250024318695,
+            0.015148683451116085,
+            -0.009223757311701775,
+            0.01437058113515377,
+            0.0071893432177603245,
+            -0.011087961494922638,
+            -0.006646292749792337,
+            -0.01227132510393858,
+            0.008255181834101677,
+            0.015027105808258057,
+            0.0005825637490488589,
+            0.02741190418601036,
+            -0.0026200171560049057,
+            -0.00540213892236352,
+            0.003217778168618679,
+            -0.02491549216210842,
+            -0.015942996367812157,
+            0.004798299167305231,
+            0.008198444731533527,
+            -0.015083841979503632,
+            -0.013859951868653297,
+            -0.006881344132125378,
+            0.011087961494922638,
+            -0.0017872042953968048,
+            -0.006553082261234522,
+            -0.0013981530210003257,
+            -0.002670674817636609,
+            -0.004782088566571474,
+            -0.024023916572332382,
+            0.0024579125456511974,
+            0.013381742872297764,
+            0.006046505179256201,
+            0.014727211557328701,
+            0.003460935316979885,
+            0.007108291145414114,
+            -0.006739502772688866,
+            -0.012935955077409744,
+            -0.00822681374847889,
+            0.012563114054501057,
+            0.021365398541092873,
+            0.013414164073765278,
+            0.010553015395998955,
+            -0.005568296182900667,
+            0.007371711079031229,
+            -0.008113340474665165,
+            -0.015148683451116085,
+            -0.00663008214905858,
+            -0.00176896748598665,
+            0.0019006775692105293,
+            -0.004453826695680618,
+            0.002962463302537799,
+            -0.002729437779635191,
+            0.006982659921050072,
+            0.004842877853661776,
+            0.008559128269553185,
+            0.0007877275347709656,
+            -0.002368754940107465,
+            -0.012149747461080551,
+            -0.014719106256961823,
+            -0.019193196669220924,
+            0.017815306782722473,
+            0.010034280829131603,
+            0.013633005321025848,
+            0.018917618319392204,
+            -0.0024822282139211893,
+            -0.012749534100294113,
+            -0.0197119303047657,
+            0.00038297235732898116,
+            0.003321120049804449,
+            0.016056470572948456,
+            0.026228539645671844,
+            -0.01789635792374611,
+            0.00382972345687449,
+            -0.006168083753436804,
+            0.01179311703890562,
+            -0.022678447887301445,
+            -0.010293648578226566,
+            0.015878155827522278,
+            -0.030281158164143562,
+            -5.6609998864587396e-05,
+            -0.010723225772380829,
+            -0.00444572139531374,
+            0.018593408167362213,
+            0.023602444678544998,
+            0.017247939482331276,
+            0.012976480647921562,
+            -0.010471963323652744,
+            -0.005009035114198923,
+            0.01920940726995468,
+            0.008559128269553185,
+            -0.005795242730528116,
+            0.0002556948165874928,
+            -0.007732394151389599,
+            -0.02104119025170803,
+            0.005146824289113283,
+            0.00969386100769043,
+            -0.024737177416682243,
+            0.005122508388012648,
+            -0.017701832577586174,
+            -0.0039350916631519794,
+            -0.018431304022669792,
+            -0.00305972620844841,
+            0.00611945241689682,
+            0.00244372827000916,
+            0.0024457545951008797,
+            0.0004257781256455928,
+            -0.003122541820630431,
+            -0.0005324126104824245,
+            0.01284679677337408,
+            -0.007489237003028393,
+            -0.01972814090549946,
+            -0.027379482984542847,
+            -0.010844804346561432,
+            -0.0018165857763960958,
+            0.002555175218731165,
+            -0.007335237693041563,
+            -0.02254876308143139,
+            0.0029644896276295185,
+            -0.009896491654217243,
+            -0.000849529926199466,
+            -0.001954374834895134,
+            0.02630959264934063,
+            0.002792253391817212,
+            -0.015245947055518627,
+            -0.034301355481147766,
+            -0.03123757429420948,
+            0.011622906662523746,
+            -0.0008160958532243967,
+            0.004672667942941189,
+            -0.011631011962890625,
+            0.001885480247437954,
+            -0.016169942915439606,
+            0.01464615948498249,
+            -0.022532552480697632,
+            -0.008012024685740471,
+            0.009337230585515499,
+            -0.0003305416030343622,
+            0.001216798322275281,
+            0.000767970981542021,
+            -0.0074325003661215305,
+            0.003718276508152485,
+            -0.01728036068379879,
+            -0.009961334057152271,
+            -0.02087908424437046,
+            -0.00027431154740042984,
+            0.004352510906755924,
+            -0.013422268442809582,
+            0.019031090661883354,
+            0.01987403631210327,
+            0.0038884864188730717,
+            -0.012109220959246159,
+            0.017863936722278595,
+            0.01188227441161871,
+            -0.009547966532409191,
+            -0.014613738283514977,
+            -0.021786870434880257,
+            0.011266276240348816,
+            0.026325803250074387,
+            0.018415093421936035,
+            0.0075540789403021336,
+            0.01588626019656658,
+            0.012287535704672337,
+            0.016007838770747185,
+            0.001041522715240717,
+            0.0017831516452133656,
+            0.0009766807779669762,
+            0.011979537084698677,
+            -0.015910575166344643,
+            0.0035359086468815804,
+            0.0090535469353199,
+            0.00985596515238285,
+            -0.016072681173682213,
+            -0.01290353387594223,
+            0.024007705971598625,
+            -0.0010516542242839932,
+            -0.012028168886899948,
+            0.0028348059859126806,
+            0.00540213892236352,
+            -0.007177185267210007,
+            0.000767970981542021,
+            -0.005462928209453821,
+            -0.002014150843024254,
+            0.01590247079730034,
+            0.017766674980521202,
+            -0.009758702479302883,
+            -0.004401142708957195,
+            0.011995747685432434,
+            -0.0028773583471775055,
+            -0.03190220519900322,
+            0.006184294354170561,
+            0.028822215273976326,
+            0.006018136627972126,
+            -0.03790007904171944,
+            -0.0013454690342769027,
+            0.005219771061092615,
+            -0.032048098742961884,
+            -0.03449587896466255,
+            0.0021965187042951584,
+            -0.010771856643259525,
+            -0.015294577926397324,
+            -0.0035521192476153374,
+            0.0075905523262917995,
+            -0.028984319418668747,
+            0.009410177357494831,
+            0.014281423762440681,
+            -0.0006276491330936551,
+            0.022240763530135155,
+            -0.004944193176925182,
+            -0.012352378107607365,
+            -0.027266010642051697,
+            0.02237044833600521,
+            -0.013470900245010853,
+            -0.008461865596473217,
+            -0.0012380746193230152,
+            -0.009945123456418514,
+            0.005240034312009811,
+            0.003071883926168084,
+            0.02890326827764511,
+            0.018479935824871063,
+            -0.0029259896837174892,
+            -0.04201753810048103,
+            -0.022338027134537697,
+            0.003971565049141645,
+            -0.025045175105333328,
+            0.013389848172664642,
+            -0.006054610479623079,
+            -0.00321169919334352,
+            0.005969505291432142,
+            0.03164283558726311,
+            0.011274381540715694,
+            0.019355300813913345,
+            -3.3022501156665385e-05,
+            0.0042674061842262745,
+            -0.03384746238589287,
+            -0.02415359951555729,
+            0.00947501976042986,
+            -0.0024639912880957127,
+            0.032647885382175446,
+            0.0036270925775170326,
+            0.004619983956217766,
+            0.014475949108600616,
+            -0.006719239521771669,
+            -0.004054643679410219,
+            -0.018804144114255905,
+            0.011922799982130527,
+            -0.004761825315654278,
+            0.01932287961244583,
+            0.01625099591910839,
+            -0.014378686435520649,
+            -0.022354237735271454,
+            0.005223823711276054,
+            0.04986340552568436,
+            0.013803214766085148,
+            -0.0006757739465683699,
+            0.005839821882545948,
+            0.0021195188164711,
+            0.0021073610987514257,
+            0.004340353421866894,
+            0.004046538379043341,
+            0.015002789907157421,
+            0.026212329044938087,
+            -0.004745615180581808,
+            0.0036736975889652967,
+            0.00350348767824471,
+            -0.005098192486912012,
+            -0.0028307533357292414,
+            -0.013033217750489712,
+            0.004684825893491507,
+            -0.010925856418907642,
+            0.008125497959554195,
+            -0.02053866535425186,
+            -0.011995747685432434,
+            -0.011055540293455124,
+            -0.010609752498567104,
+            -0.010009964928030968,
+            -0.018787933513522148,
+            0.01927424781024456,
+            -0.029730001464486122,
+            0.002589622512459755,
+            -0.013227743096649647,
+            0.013689741492271423,
+            0.009620913304388523,
+            -0.0006119452300481498,
+            0.02353760227560997,
+            -0.023407917469739914,
+            -0.007935024797916412,
+            0.004465984180569649,
+            0.00266662216745317,
+            -0.0022654130589216948,
+            0.003124567912891507,
+            -0.013057533651590347,
+            -0.025612542405724525,
+            0.009547966532409191,
+            -0.00540213892236352,
+            0.0034305406734347343,
+            0.006747608073055744,
+            -0.0010405095526948571,
+            -0.005843874532729387,
+            -0.003625066252425313,
+            0.01826919987797737,
+            -0.020425191149115562,
+            0.04114217311143875,
+            -0.03018389455974102,
+            -0.00480235181748867,
+            0.020311718806624413,
+            -0.01723172888159752,
+            -0.0001227183238370344,
+            -0.0020648085046559572,
+            0.0073636057786643505,
+            0.0035014613531529903,
+            -0.0022188080474734306,
+            -0.014386791735887527,
+            0.03161041438579559,
+            3.1233648769557476e-05,
+            -0.016745414584875107,
+            0.00889144279062748,
+            -0.013308795168995857,
+            0.017734253779053688,
+            -0.028108954429626465,
+            -1.29652098621591e-05,
+            0.015221631154417992,
+            0.0016281390562653542,
+            0.009783018380403519,
+            -0.01121764536947012,
+            -0.010958277620375156,
+            -0.01260364055633545,
+            0.015797102823853493,
+            0.01933909021317959,
+            -0.011525643989443779,
+            -0.022840552031993866,
+            0.0040242490358650684,
+            0.01666436344385147,
+            -0.021867923438549042,
+            -0.0017132440116256475,
+            -0.004397090058773756,
+            -0.009531755931675434,
+            0.0003996893938165158,
+            -0.012587429955601692,
+            0.011801222339272499,
+            0.014467843808233738,
+            -0.0046686152927577496,
+            -0.03316662088036537,
+            0.008084971457719803,
+            0.010625963099300861,
+            -0.03169146925210953,
+            -0.013049428351223469,
+            0.012555008754134178,
+            -0.011898485012352467,
+            0.010180175304412842,
+            -0.006775976158678532,
+            0.00986407045274973,
+            -0.00928049348294735,
+            -0.001244153594598174,
+            -0.00031737060635350645,
+            -0.0007380829192698002,
+            -0.022856762632727623,
+            -0.008648285642266273,
+            -0.0009584440267644823,
+            -0.03079989179968834,
+            0.011695853434503078,
+            0.003937117755413055,
+            -0.009734386578202248,
+            -0.0006560174515470862,
+            -0.017085835337638855,
+            -0.004368721507489681,
+            0.0045105633325874805,
+            0.00685297604650259,
+            -0.011493222787976265,
+            -0.013187216594815254,
+            -0.002350518014281988,
+            -0.016007838770747185,
+            0.018236778676509857,
+            -0.0004701036086771637,
+            0.036635659635066986,
+            -0.0023829389829188585,
+            -0.0013900477206334472,
+            -0.0021418083924800158,
+            -0.00280441134236753,
+            0.005791190080344677,
+            0.03195083513855934,
+            -0.0013971398584544659,
+            0.006390977650880814,
+            0.0018054411048069596,
+            -0.003677750239148736,
+            0.006001926492899656,
+            0.01576468162238598,
+            0.0005633138353005052,
+            0.003975617699325085,
+            -0.009726282209157944,
+            -0.010893435217440128,
+            0.006942133419215679,
+            -0.002822648035362363,
+            0.02021445520222187,
+            -0.0014812316512688994,
+            -0.00932912528514862,
+            -0.009256178513169289,
+            0.0012887322809547186,
+            0.008153866045176983,
+            -0.005977610591799021,
+            -0.002030361210927367,
+            0.02243529073894024,
+            -0.005122508388012648,
+            0.014905527234077454,
+            -0.015424261800944805,
+            0.015805207192897797,
+            0.014394897036254406,
+            -0.013284480199217796,
+            0.00781344622373581,
+            0.018804144114255905,
+            -0.007667552214115858,
+            0.018706882372498512,
+            -0.016550889238715172,
+            0.007602710276842117,
+            -0.00983165018260479,
+            0.03884028643369675,
+            -0.009450703859329224,
+            -0.022013816982507706,
+            -0.0029766475781798363,
+            0.015383735299110413,
+            -0.02005235105752945,
+            0.010569225996732712,
+            0.013114269822835922,
+            -0.02536938525736332,
+            0.002944226609542966,
+            0.014751527458429337,
+            0.003997907042503357,
+            -0.004206616897135973,
+            0.006265346426516771,
+            0.005406191572546959,
+            0.009499335661530495,
+            0.005811453331261873,
+            0.020360350608825684,
+            -0.016048364341259003,
+            0.01972814090549946,
+            -0.012174062430858612,
+            -0.0032035941258072853,
+            -0.022889183834195137,
+            -0.0030131209641695023,
+            -0.009572282433509827,
+            0.011250065639615059,
+            -0.012230799533426762,
+            0.006828660145401955,
+            -0.010042386129498482,
+            -0.0001888266415335238,
+            0.030702630057930946,
+            0.018706882372498512,
+            0.008372707292437553,
+            0.003292751731351018,
+            0.01196332648396492,
+            0.03355567157268524,
+            0.012798165902495384,
+            -0.009742491878569126,
+            0.0025531488936394453,
+            0.023440338671207428,
+            0.00649634562432766,
+            0.0169561505317688,
+            0.008988705463707447,
+            3.9164748159237206e-05,
+            -0.013973425142467022,
+            0.011509433388710022,
+            0.033101778477430344,
+            0.024996543303132057,
+            0.025952961295843124,
+            0.005353507585823536,
+            0.0010577331995591521,
+            -0.01282248180359602,
+            -0.01343037374317646,
+            -0.016680574044585228,
+            -0.011736379936337471,
+            0.022678447887301445,
+            -0.009547966532409191,
+            0.002751727122813463,
+            -0.01872309297323227,
+            0.008632075041532516,
+            0.020019929856061935,
+            0.009507440030574799,
+            0.017653200775384903,
+            0.010228806175291538,
+            0.00451461598277092,
+            -0.012287535704672337,
+            0.0007613854832015932,
+            0.0060059791430830956,
+            -0.024964123964309692,
+            0.016558995470404625,
+            0.031756311655044556,
+            -0.0003996893938165158,
+            0.01539994589984417,
+            -0.0002955877862405032,
+            -0.003209673101082444,
+            -0.012190273031592369,
+            0.02392665296792984,
+            0.010625963099300861,
+            0.012449640780687332,
+            0.0023201233707368374,
+            -0.008713127113878727,
+            -0.011201434768736362,
+            -0.024218441918492317,
+            0.026568960398435593,
+            0.02092771604657173,
+            -0.00196551950648427,
+            0.00386417075060308,
+            -0.001570389256812632,
+            -0.012174062430858612,
+            0.014970368705689907,
+            0.015732260420918465,
+            0.019468773156404495,
+            -0.0028672267217189074,
+            0.012911639176309109,
+            -0.018139515072107315,
+            0.000802418275270611,
+            -0.012781955301761627,
+            0.01700478233397007,
+            0.004275511484593153,
+            -0.021462662145495415,
+            0.016696784645318985,
+            0.013770793564617634,
+            0.0005202547763474286,
+            -0.019955087453126907,
+            0.005531822796911001,
+            -0.0055196648463606834,
+            0.001069891033694148,
+            0.0007168066804297268,
+            -0.0012826534220948815,
+            -0.006747608073055744,
+            0.000229859389946796,
+            0.011703958734869957,
+            -0.0069097126834094524,
+            -0.022954024374485016,
+            -0.010447647422552109,
+            -0.001273535075597465,
+            -0.00656524021178484,
+            -0.020133404061198235,
+            0.015286472626030445,
+            0.004826667252928019,
+            -0.016907520592212677,
+            -0.020036140456795692,
+            -0.0004072880547028035,
+            0.008559128269553185,
+            0.03410682827234268,
+            -0.0284169539809227,
+            0.0032745148055255413,
+            -0.012352378107607365,
+            0.020798033103346825,
+            0.00357238226570189,
+            -0.023359287530183792,
+            0.022775709629058838,
+            0.0008277471060864627,
+            0.0056412434205412865,
+            -0.01179311703890562,
+            -0.0045835101045668125,
+            0.009766807779669762,
+            -0.022743288427591324,
+            0.005353507585823536,
+            0.01049627922475338,
+            -0.0038378287572413683,
+            0.008518601767718792,
+            9.124721691478044e-05,
+            0.005183297675102949,
+            0.0005379849462769926,
+            -0.009920807555317879,
+            0.018787933513522148,
+            -0.0014619816793128848,
+            -0.014605632983148098,
+            0.0034589089918881655,
+            0.013908582739531994,
+            -0.02564496360719204,
+            0.03595482185482979,
+            -0.0038256708066910505,
+            -0.014548896811902523,
+            -0.004433563444763422,
+            0.0010293647646903992,
+            0.019484983757138252,
+            0.030880944803357124,
+            -0.020911505445837975,
+            -0.010763752274215221,
+            0.005090087652206421,
+            0.01137974951416254,
+            -0.009815439581871033,
+            0.004206616897135973,
+            0.017572149634361267,
+            0.010553015395998955,
+            0.011209540069103241,
+            -0.010707015171647072,
+            -0.01595110259950161,
+            0.030427051708102226,
+            0.019922668114304543,
+            0.003683829214423895,
+            0.009701966308057308,
+            -0.002788200741633773,
+            0.020425191149115562,
+            0.0025531488936394453,
+            -0.01124196033924818,
+            0.00452677346765995,
+            0.016413100063800812,
+            -0.010763752274215221,
+            -0.0010749567300081253,
+            0.008247076533734798,
+            -0.011671538464725018,
+            0.008429444395005703,
+            -0.027395693585276604,
+            0.0263582244515419,
+            0.007906656712293625,
+            0.011955221183598042,
+            0.020133404061198235,
+            -0.01977677270770073,
+            -0.0011631011730059981,
+            0.014305739663541317,
+            0.012344272807240486,
+            -0.02353760227560997,
+            -0.008542917668819427,
+            0.008826600387692451,
+            -0.00020794992451556027,
+            -0.0008216681890189648,
+            -0.003635197877883911,
+            -0.013519532047212124,
+            -0.005000929813832045,
+            -0.04130427539348602,
+            0.011047434993088245,
+            -0.004786141216754913,
+            0.011922799982130527,
+            0.005985715892165899,
+            0.00911028403788805,
+            -0.02613127790391445,
+            0.003578461240977049,
+            -0.01000185962766409,
+            -0.015480998903512955,
+            0.005495349410921335,
+            0.006054610479623079,
+            -0.013162901625037193,
+            -0.0035642769653350115,
+            -0.0018358356319367886,
+            0.002589622512459755,
+            0.0021134400740265846,
+            -0.004790193866938353,
+            -0.029178844764828682,
+            -0.00823897123336792,
+            -0.013227743096649647,
+            0.006938081234693527,
+            -0.014840684831142426,
+            0.03052431344985962,
+            0.0005192416138015687,
+            -0.009207546710968018,
+            0.0036574872210621834,
+            -0.013859951868653297,
+            -0.00411948561668396,
+            -0.0038216181565076113,
+            0.01895003952085972,
+            -0.0038074341136962175,
+            0.028643900528550148,
+            0.00883470568805933,
+            -0.010958277620375156,
+            -0.018852775916457176,
+            0.003193462500348687,
+            -0.014119318686425686,
+            -0.0019422168843448162,
+            0.03133483603596687,
+            0.007720236200839281,
+            -0.008186287246644497,
+            0.013900477439165115,
+            -0.008996810764074326,
+            0.022078659385442734,
+            -0.01298458594828844,
+            -0.01027743797749281,
+            0.012887323275208473,
+            0.013738373294472694,
+            0.004384932108223438,
+            -1.1429647202021442e-05,
+            -0.005548033397644758,
+            0.00883470568805933,
+            -0.019922668114304543,
+            -0.001798348966985941,
+            0.012749534100294113,
+            -0.007079922594130039,
+            0.0036230399273335934,
+            0.017701832577586174,
+            0.013689741492271423,
+            -0.009766807779669762,
+            -0.00019199274538550526,
+            0.012076799757778645,
+            0.01583762839436531,
+            0.017020992934703827,
+            -0.01838267222046852,
+            0.0028874899726361036,
+            0.014751527458429337,
+            -0.02498033456504345,
+            -0.005803348030894995,
+            -0.012709008529782295,
+            0.0037000395823270082,
+            -0.036473557353019714,
+            0.00016045832308009267,
+            0.04493542015552521,
+            0.0035946716088801622,
+            0.0008135629468597472,
+            -0.029551686719059944,
+            -0.023359287530183792,
+            -0.005458875559270382,
+            -0.0011134566739201546,
+            -0.003997907042503357,
+            -0.019809193909168243,
+            -0.012263220734894276,
+            0.014702895656228065,
+            0.016631942242383957,
+            -0.018188146874308586,
+            -0.009620913304388523,
+            0.0013707978650927544,
+            -0.006192399188876152,
+            -0.005892505869269371,
+            0.0008166024344973266,
+            -0.004068827722221613,
+            -0.0010911672143265605,
+            0.01137974951416254,
+            0.015059526078402996,
+            0.007481131702661514,
+            0.016680574044585228,
+            0.011987642385065556,
+            -0.008048498071730137,
+            0.014176055788993835,
+            -0.003600750584155321,
+            -0.004555142018944025,
+            -0.004287669435143471,
+            -0.023570023477077484,
+            -0.0038540391251444817,
+            0.003408251330256462,
+            -0.014135529287159443,
+            0.008948178961873055,
+            0.00389861804433167,
+            0.003991828300058842,
+            -0.001961466856300831,
+            -0.010690804570913315,
+            0.0016960203647613525,
+            -0.02287297323346138,
+            -0.006504450924694538,
+            0.0027942797169089317,
+            -0.017815306782722473,
+            0.019306669011712074,
+            0.01922561787068844,
+            -0.0059532951563596725,
+            -0.016648152843117714,
+            0.014224686659872532,
+            0.001204640488140285,
+            0.015942996367812157,
+            0.015237841755151749,
+            0.000640313548501581,
+            0.0047739832662045956,
+            0.01204437855631113,
+            -0.0019857825245708227,
+            -0.021624766290187836,
+            -0.018301621079444885,
+            0.0053251395002007484,
+            -0.003165094181895256,
+            -0.018674461171030998,
+            0.0026443328242748976,
+            -0.0029118056409060955,
+            0.018155725672841072,
+            0.0022836499847471714,
+            0.006694924086332321,
+            -0.0030475682578980923,
+            -0.0218841340392828,
+            -0.0015795076033100486,
+            -0.004506510682404041,
+            -0.02853042632341385,
+            -0.00620455713942647,
+            -0.009888386353850365,
+            0.025466646999120712,
+            -0.021738240495324135,
+            0.015164894051849842,
+            -0.003852012800052762,
+            0.0042147221975028515,
+            0.02049003355205059,
+            0.008680706843733788,
+            0.002648385474458337,
+            0.012433430179953575,
+            -0.008591548539698124,
+            -0.00533324433490634,
+            0.01088532991707325,
+            -0.008461865596473217,
+            0.013495216146111488,
+            -0.024623703211545944,
+            0.01559447217732668,
+            -0.014694790355861187,
+            -0.005394033621996641,
+            0.010074807330965996,
+            0.013187216594815254,
+            0.0074325003661215305,
+            -0.01531078852713108,
+            -0.012376694008708,
+            0.022532552480697632,
+            -0.0077931834384799,
+            0.00331504107452929,
+            -0.003886460093781352,
+            -0.011250065639615059,
+            -0.01672920398414135,
+            -0.0006707081920467317,
+            0.013551952317357063,
+            -0.017020992934703827,
+            -0.00985596515238285,
+            -0.01595110259950161,
+            -0.0071690804325044155,
+            -0.04947435483336449,
+            -0.02369970642030239,
+            0.013025112450122833,
+            -0.009815439581871033,
+            -0.018706882372498512,
+            0.02614748664200306,
+            -0.006472030188888311,
+            -0.00591276865452528,
+            -0.008076866157352924,
+            -0.008283549919724464,
+            -0.014143634587526321,
+            0.01961466856300831,
+            0.010317963548004627,
+            -0.01365732029080391,
+            0.01922561787068844,
+            0.000424764963099733,
+            0.015051420778036118,
+            -0.012611744925379753,
+            -0.018204357475042343,
+            -0.0032542517874389887,
+            0.022743288427591324,
+            0.022030027583241463,
+            -0.009928912855684757,
+            0.0014619816793128848,
+            0.0021316767670214176,
+            0.018074674531817436,
+            0.01174448523670435,
+            -0.010739436373114586,
+            -0.010561120696365833,
+            0.013008901849389076,
+            -0.00985596515238285,
+            0.003858091775327921,
+            0.009661439806222916,
+            0.0042349849827587605,
+            0.002431570552289486,
+            -0.009037337265908718,
+            -0.001908782753162086,
+            0.006091083865612745,
+            0.0217706598341465,
+            -0.0007553065661340952,
+            -0.0061113471165299416,
+            -0.004867193289101124,
+            -0.009783018380403519,
+            -0.0025693594943732023,
+            -0.020862873643636703,
+            -0.013203427195549011,
+            -0.0023201233707368374,
+            0.018317829817533493,
+            0.004660509992390871,
+            -0.020798033103346825,
+            -0.02702285349369049,
+            -0.008988705463707447,
+            -0.023861810564994812,
+            -0.0055358754470944405,
+            -0.011250065639615059,
+            -0.008745548315346241,
+            0.008036340586841106,
+            0.009815439581871033,
+            -0.0017213491955772042,
+            -0.00199996680021286,
+            -0.006504450924694538,
+            0.005017140414565802,
+            -0.024947913363575935,
+            -0.0013272322248667479,
+            -0.0015896392287686467,
+            -0.021803081035614014,
+            -0.012733323499560356,
+            0.005916821304708719,
+            0.031059259548783302,
+            0.04224448278546333,
+            -0.008940073661506176,
+            0.0009270362206734717,
+            -0.0038216181565076113,
+            0.010674593970179558,
+            0.017847727984189987,
+            0.0090535469353199,
+            0.007019133307039738,
+            0.017345203086733818,
+            -0.03203188627958298,
+            0.0043200901709496975,
+            0.012392903678119183,
+            -0.0004067814734298736,
+            -0.007647288963198662,
+            -0.015489104203879833,
+            0.005000929813832045,
+            -0.008153866045176983,
+            0.005568296182900667,
+            0.012660376727581024,
+            -0.011039329692721367,
+            -0.00946691446006298,
+            0.0283034797757864,
+            0.009572282433509827,
+            0.011857958510518074,
+            0.01522973645478487,
+            -0.01407068781554699,
+            -0.022289395332336426,
+            0.0008231879328377545,
+            0.006549029611051083,
+            -0.0037446185015141964,
+            0.008575338870286942,
+            -0.010650279000401497,
+            -0.03595482185482979,
+            0.0002515155647415668,
+            -0.0006661489605903625,
+            -0.012336167506873608,
+            -0.016332048922777176,
+            0.0035055140033364296,
+            0.01932287961244583,
+            0.025904331356287003,
+            -0.01531889382749796,
+            0.017410043627023697,
+            0.007663499563932419,
+            -0.011720169335603714,
+            -0.009426387958228588,
+            -0.009572282433509827,
+            -0.015164894051849842,
+            -0.02037656120955944,
+            0.001938164234161377,
+            -0.014929842203855515,
+            -0.011833642609417439,
+            0.008234918117523193,
+            0.03263167664408684,
+            -0.0009467927156947553,
+            -0.004838825203478336,
+            -0.0011590485228225589,
+            0.0070839752443134785,
+            0.004518668632954359,
+            -0.010698909871280193,
+            0.01450837031006813,
+            0.019290458410978317,
+            0.010966382920742035,
+            0.001982743153348565,
+            -0.021235715597867966,
+            -0.008948178961873055,
+            0.017572149634361267,
+            -0.01223890483379364,
+            0.004575405269861221,
+            0.020198244601488113,
+            -0.005949242506176233,
+            -0.0069097126834094524,
+            0.0015339156379923224,
+            -0.007720236200839281,
+            -0.03268030658364296,
+            0.008315971121191978,
+            -0.007108291145414114,
+            0.011063645593822002,
+            0.0071528698317706585,
+            -0.019792983308434486,
+            0.003991828300058842,
+            -0.007233921904116869,
+            0.002139782067388296,
+            -0.0060789259150624275,
+            -0.0018024016171693802,
+            -0.00044198858086019754,
+            -0.01723172888159752,
+            0.024785807356238365,
+            -0.003612908534705639,
+            2.2194411940290593e-05,
+            0.0016372574027627707,
+            0.022532552480697632,
+            -0.0011428381549194455,
+            0.006334241013973951,
+            0.0033089620992541313,
+            -0.012725219130516052,
+            0.006654397584497929,
+            -0.009118389338254929,
+            -0.004393037408590317,
+            -0.00787423551082611,
+            0.012944060377776623,
+            0.013876162469387054,
+            0.02425086311995983,
+            0.00017109644250012934,
+            0.005527770146727562,
+            -0.019371511414647102,
+            -0.0035237509291619062,
+            -0.01553773507475853,
+            0.0031205154955387115,
+            0.010617857798933983,
+            0.0181070938706398,
+            -0.003770960494875908,
+            -0.004587562754750252,
+            -0.0010258187539875507,
+            0.009069757536053658,
+            0.016404995694756508,
+            -0.004064775072038174,
+            -0.022062448784708977,
+            0.0016382705653086305,
+            -0.004066801629960537,
+            0.009685755707323551,
+            0.022143501788377762,
+            -0.01634015329182148,
+            0.02032792940735817,
+            0.001971598481759429,
+            -0.015408051200211048,
+            0.013527637347579002,
+            -0.0074000791646540165,
+            -0.027233589440584183,
+            -0.020911505445837975,
+            0.0018753487383946776,
+            0.004879351239651442,
+            0.013033217750489712,
+            0.012433430179953575,
+            0.014840684831142426,
+            0.004725351929664612,
+            0.015610681846737862,
+            -0.0007117409259080887,
+            -0.0083078658208251,
+            -0.017085835337638855,
+            -0.0006367675377987325,
+            -0.007906656712293625,
+            -0.00822681374847889,
+            -0.038353972136974335,
+            -0.017426254227757454,
+            -0.007785078138113022,
+            0.02541801705956459,
+            0.013730267994105816,
+            0.005770927295088768,
+            -0.004571352619677782,
+            -0.01826919987797737,
+            -0.007290658541023731,
+            0.019484983757138252,
+            -0.0026564907748252153,
+            0.029308529570698738,
+            -0.006318030413240194,
+            0.00218841340392828,
+            0.003229936119168997,
+            -0.00817007664591074,
+            -0.020360350608825684,
+            0.022970234975218773,
+            0.01298458594828844,
+            0.007732394151389599,
+            0.014038266614079475,
+            -0.02010098285973072,
+            0.02404012717306614,
+            -0.014143634587526321,
+            -0.023164760321378708,
+            0.012668482027947903,
+            0.004534878768026829,
+            0.01961466856300831,
+            -0.009434493258595467,
+            0.005296770948916674,
+            0.015116263180971146,
+            -0.02287297323346138,
+            0.0263582244515419,
+            -0.005361612886190414,
+            0.023683495819568634,
+            -0.006390977650880814,
+            0.00852670706808567,
+            -0.015148683451116085,
+            0.005146824289113283,
+            -0.04454636946320534,
+            0.0020546771120280027,
+            -0.03024873696267605,
+            -8.561154390918091e-05,
+            0.017961200326681137,
+            -0.025612542405724525,
+            -0.0026504117995500565,
+            0.005211665760725737,
+            -0.011671538464725018,
+            -0.007456816267222166,
+            0.02254876308143139,
+            0.009783018380403519,
+            0.0008839771617203951,
+            -0.0003571369161363691,
+            0.011857958510518074,
+            -0.013770793564617634,
+            -0.024412967264652252,
+            -0.01838267222046852,
+            -0.017474886029958725,
+            -0.033750198781490326,
+            0.0003259824006818235,
+            0.00023669817892368883,
+            0.0045470367185771465,
+            -0.008915758691728115,
+            -0.004170143511146307,
+            -0.009710071608424187,
+            -0.0014579291455447674,
+            0.016048364341259003,
+            0.008198444731533527,
+            -0.014800158329308033,
+            -0.004506510682404041,
+            -0.006034347228705883,
+            -0.0016676520463079214,
+            0.00022276731033343822,
+            0.004769930616021156,
+            0.018852775916457176,
+            -0.011387854814529419,
+            -0.008591548539698124,
+            0.025612542405724525,
+            0.022954024374485016,
+            -0.016323942691087723,
+            0.00389861804433167,
+            -0.03751102834939957,
+            0.01865825057029724,
+            -0.02293781377375126,
+            -5.395046900957823e-05,
+            -0.012571219354867935,
+            0.022662237286567688,
+            -0.003939144313335419,
+            0.0035217246040701866,
+            0.013422268442809582,
+            -0.010034280829131603,
+            0.020895294845104218,
+            0.011922799982130527,
+            -0.0031326732132583857,
+            0.0026504117995500565,
+            -0.010674593970179558,
+            0.01356816291809082,
+            -0.02974621206521988,
+            0.00013918208424001932,
+            0.01893382892012596,
+            0.006285609677433968,
+            0.011314908042550087,
+            -0.015894364565610886,
+            0.004498405382037163,
+            0.0028814109973609447,
+            0.030491894111037254,
+            0.004680773243308067,
+            0.004103275015950203,
+            0.0035602243151515722,
+            0.0005020179669372737,
+            -0.029000530019402504,
+            0.009499335661530495,
+            0.007849919609725475,
+            -0.007699972949922085,
+            0.021576134487986565,
+            0.012133536860346794,
+            0.018366461619734764,
+            -0.006788134109228849,
+            -0.009742491878569126,
+            0.025661174207925797,
+            0.01744246482849121,
+            -0.006192399188876152,
+            0.0025146491825580597,
+            0.023683495819568634,
+            -0.0074000791646540165,
+            0.040234386920928955,
+            0.027655061334371567,
+            -0.009264283813536167,
+            -0.028611479327082634,
+            0.010836699046194553,
+            -0.01509194727987051,
+            -0.004131643567234278,
+            0.01922561787068844,
+            0.022678447887301445,
+            -0.011258170939981937,
+            0.01531889382749796,
+            0.015440472401678562,
+            0.012652271427214146,
+            0.01656709983944893,
+            -0.008996810764074326,
+            -0.0035338823217898607,
+            0.00047618255484849215,
+            -0.0013201400870457292,
+            -0.0002829233417287469,
+            -0.02608264610171318,
+            0.006151873152703047,
+            -0.0023748339153826237,
+            0.007193395867943764,
+            0.03007042035460472,
+            0.0073960269801318645,
+            -0.023359287530183792,
+            -0.0150108952075243,
+            -0.01998750865459442,
+            0.022662237286567688,
+            -0.022451501339673996,
+            -0.017345203086733818,
+            -0.00869691651314497,
+            -0.01260364055633545,
+            -0.032988306134939194,
+            0.009904596954584122,
+            0.00013740906433667988,
+            -0.005215718410909176,
+            0.0006104254862293601,
+            0.0013515478931367397,
+            -0.005491296760737896,
+            -0.0015268236165866256,
+            0.0016717046964913607,
+            0.008202497847378254,
+            0.010091017000377178,
+            0.016648152843117714,
+            -0.005365665536373854,
+            0.0166157316416502,
+            -0.012571219354867935,
+            -0.03738134354352951,
+            -0.0037000395823270082,
+            -0.003385961754247546,
+            0.017102045938372612,
+            0.0005400112713687122,
+            0.006480135023593903,
+            -0.019063511863350868,
+            0.003440672066062689,
+            0.00040146239916794,
+            0.005292718298733234,
+            0.01304132305085659,
+            -0.0013525610556825995,
+            -0.004474089480936527,
+            0.027071483433246613,
+            -0.004786141216754913,
+            0.007104238495230675,
+            -0.00986407045274973,
+            0.016648152843117714,
+            -0.016826467588543892,
+            -0.003088094526901841,
+            0.007027238607406616,
+            -0.010828593745827675,
+            -0.0022613604087382555,
+            -0.011509433388710022,
+            -0.017912568524479866,
+            -0.010212595574557781,
+            0.011395960114896297,
+            -0.020960137248039246,
+            -0.013106164522469044,
+            -0.0004678240220528096,
+            -0.007193395867943764,
+            0.008502391166985035,
+            0.01601594313979149,
+            0.005219771061092615,
+            -0.010763752274215221,
+            0.023229602724313736,
+            0.013284480199217796,
+            0.023732127621769905,
+            -0.006767870858311653,
+            -0.013989635743200779,
+            -0.004174196161329746,
+            -0.006301820278167725,
+            0.015983523800969124,
+            0.0070839752443134785,
+            0.005276507697999477,
+            -0.003495382610708475,
+            0.02160855568945408,
+            -0.009369651786983013,
+            0.0006732410402037203,
+            -0.002932068658992648,
+            0.010504384525120258,
+            -0.010974488221108913,
+            -0.009239967912435532,
+            0.015140579082071781,
+            0.026941800490021706,
+            0.0059006111696362495,
+            0.005673664156347513,
+            0.009021126665174961,
+            0.00020896308706142008,
+            0.009069757536053658,
+            0.0029158582910895348,
+            -0.006362609565258026,
+            -0.03465798497200012,
+            -0.00875365361571312,
+            -0.01005049142986536,
+            -0.019825404509902,
+            0.019582247361540794,
+            -0.0002821634989231825,
+            -0.009677650406956673,
+            -0.006638187449425459,
+            -0.00928049348294735,
+            0.004316037520766258,
+            0.0013900477206334472,
+            -0.000849529926199466,
+            -0.008429444395005703,
+            0.008003919385373592,
+            -0.023667285218834877,
+            -0.015027105808258057,
+            0.028708741068840027,
+            0.008915758691728115,
+            0.013292585499584675,
+            -0.008672601543366909,
+            -0.0021033084485679865,
+            0.006087031215429306,
+            -0.02442917786538601,
+            -0.00883470568805933,
+            0.06539303809404373,
+            -0.016242889687418938,
+            0.019695719704031944,
+            -0.024331914260983467,
+            -0.001989835174754262,
+            -0.002711201086640358,
+            0.018204357475042343,
+            0.021803081035614014,
+            -0.025515278801321983,
+            -0.0009123454801738262,
+            -0.02076561190187931,
+            0.0005531822680495679,
+            0.0067030293866992,
+            0.0026747274678200483,
+            -0.015124368481338024,
+            0.010350384749472141,
+            -0.03222641348838806,
+            -0.016794046387076378,
+            0.009726282209157944,
+            -0.0075378683395683765,
+            0.003035410540178418,
+            -0.016161838546395302,
+            -0.005811453331261873,
+            0.007541920989751816,
+            0.010690804570913315,
+            -0.006224820390343666,
+            -0.027849586680531502,
+            -0.02575843594968319,
+            -0.009013021364808083,
+            -0.02026308700442314,
+            0.017701832577586174,
+            -0.015780892223119736,
+            0.0166157316416502,
+            -0.009491230361163616,
+            -0.003108357544988394,
+            0.006982659921050072,
+            0.01689130999147892,
+            0.012538798153400421,
+            0.0017274281708523631,
+            -0.005787137895822525,
+            -0.008291655220091343,
+            0.01700478233397007,
+            -0.013633005321025848,
+            -0.0359872430562973,
+            0.0003687881981022656,
+            -0.005191402975469828,
+            0.0020789927802979946,
+            0.008769864216446877,
+            0.018285410478711128,
+            0.01590247079730034,
+            0.01531889382749796,
+            -0.015359420329332352,
+            -0.01379510946571827,
+            0.009118389338254929,
+            0.031205154955387115,
+            -0.001978690503165126,
+            0.03637629374861717,
+            0.011874169111251831,
+            0.004964456427842379,
+            -0.003852012800052762,
+            -0.008380812592804432,
+            -0.018285410478711128,
+            -0.003509566653519869,
+            0.02049003355205059,
+            -0.0155863668769598,
+            0.02381318062543869,
+            0.0010394963901489973,
+            -0.010609752498567104,
+            -0.001762888627126813,
+            -0.0011357460170984268,
+            -0.018285410478711128,
+            0.013503321446478367,
+            -0.01599973440170288,
+            0.0023768600076436996,
+            -0.016145627945661545,
+            0.001996927196159959,
+            -0.004855035804212093,
+            0.01933909021317959,
+            0.0019766641780734062,
+            -0.007371711079031229,
+            0.005357560236006975,
+            -0.01202006358653307,
+            0.002869253046810627,
+            -3.042629214178305e-05,
+            0.01638878509402275,
+            -0.0003361139679327607,
+            0.0010253122309222817,
+            -0.02026308700442314,
+            0.008259234018623829,
+            0.0022147553972899914,
+            -0.015562050975859165,
+            -0.01760457083582878,
+            0.007991761900484562,
+            0.002368754940107465,
+            0.002439675619825721,
+            0.006115399766713381,
+            0.003961433656513691,
+            -0.004393037408590317,
+            0.004676720593124628,
+            0.007464921101927757,
+            0.005389980971813202,
+            0.005081982351839542,
+            -0.013235848397016525,
+            0.002636227523908019,
+            0.0018642040668055415,
+            0.004311984870582819,
+            0.01771804317831993,
+            -0.0001176525474875234,
+            -0.004206616897135973,
+            0.00875365361571312,
+            0.009442598558962345,
+            0.008534812368452549,
+            -0.026682432740926743,
+            -0.018188146874308586,
+            0.012717113830149174,
+            -0.001926006400026381,
+            0.006168083753436804,
+            0.00018452072981745005,
+            0.018690671771764755,
+            0.005645296070724726,
+            0.004328195471316576,
+            0.006974554620683193,
+            0.008255181834101677,
+            -0.005134666338562965,
+            -0.0018571119289845228,
+            0.013511426746845245,
+            -0.012417219579219818,
+            0.010228806175291538,
+            -0.02901674062013626,
+            0.010326068848371506,
+            0.009167020209133625,
+            -0.0023647022899240255,
+            0.0014001793460920453,
+            -0.027849586680531502,
+            -0.005803348030894995,
+            -0.01572415605187416,
+            0.0059532951563596725,
+            -0.012068694457411766,
+            0.017572149634361267,
+            0.01783151738345623,
+            -0.018123304471373558,
+            0.04282806068658829,
+            0.0035602243151515722,
+            0.02674727514386177,
+            -0.009094073437154293,
+            -0.003929012455046177,
+            -0.00244372827000916,
+            0.008923863992094994,
+            -0.01063406839966774,
+            0.012571219354867935,
+            0.006804344709962606,
+            0.02065213769674301,
+            -0.00605055782943964,
+            0.011809326708316803,
+            0.00012461798905860633,
+            0.027849586680531502,
+            -0.006662502884864807,
+            0.013543847016990185,
+            -0.011663433164358139,
+            -0.024412967264652252,
+            -0.010366595350205898,
+            0.0037912235129624605,
+            0.006557134911417961,
+            0.005150876939296722,
+            0.008109287358820438,
+            0.001973624574020505,
+            0.009807334281504154,
+            0.007837762124836445,
+            0.04039648920297623,
+            -0.012700903229415417,
+            -0.010488173924386501,
+            0.011371644213795662,
+            -0.0032542517874389887,
+            0.0025572015438228846,
+            0.005037403665482998,
+            0.011120381765067577,
+            0.01143648661673069,
+            7.332704990403727e-05,
+            -0.014127423986792564,
+            -0.009150810539722443,
+            -0.009791123680770397,
+            0.002575438469648361,
+            -0.017134467139840126,
+            0.007254185155034065,
+            -0.001919927541166544,
+            -0.00815791916102171,
+            0.006727344822138548,
+            -0.006881344132125378,
+            0.0044781421311199665,
+            0.012417219579219818,
+            0.01223890483379364,
+            0.00765539426356554,
+            -0.014240897260606289,
+            0.00401614373549819,
+            0.004453826695680618,
+            0.022662237286567688,
+            -0.004830719903111458,
+            0.017766674980521202,
+            -0.020425191149115562,
+            -0.0043565635569393635,
+            0.010026175528764725,
+            -0.012660376727581024,
+            -0.012749534100294113,
+            0.0057790325954556465
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "36d5519c-b3ca-4eef-8415-2627af35be36",
+      "type": "next",
+      "source": {
+        "id": "611ac443-aa6f-4416-a1f9-dade7d45888c",
+        "properties": {
+          "page_content": "6. Alignment For Deepseekmoe 16B\n\nPrevious research indicates that MoE models typically do not emerge significant gains from fine-tuning (Artetxe et al., 2022; Fedus et al., 2021). However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B.\n\n6.1. Experimental Setup\n\nTraining Data. For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios.\n\nHyper-Parameters. During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).\n\nWe employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10−5 without incorporating any learning rate scheduling strategy.\n\nEvaluation Benchmarks. For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3, with the following adjustments: (1) We exclude Pile (Gao et al., 2020) since chat models are seldom employed for pure language modeling. (2) We exclude\n\nMetric # Shot LLaMA2 DeepSeek DeepSeekMoE SFT 7B Chat 7B Chat 16B # Total Params N/A 6.7B 6.9B 16.4B # Activated Params N/A 6.7B 6.9B 2.8B FLOPs per 4K Tokens N/A 187.9T 183.5T 74.4T HellaSwag (Acc.) 0-shot 67.9 71.0 72.2 PIQA (Acc.) 0-shot 76.9 78.4 79.7 ARC-easy (Acc.) 0-shot 69.7 70.2 69.9 ARC-challenge (Acc.) 0-shot 50.8 50.2 50.0 BBH (EM) 3-shot 39.3 43.1 42.2 RACE-middle (Acc.) 5-shot 63.9 66.1 64.8 RACE-high (Acc.) 5-shot 49.6 50.8 50.6 DROP (EM) 1-shot 40.0 41.7 33.8 GSM8K (EM) 0-shot 63.4 62.6 62.2 MATH (EM) 4-shot 13.5 14.7 15.2 HumanEval (Pass@1) 0-shot 35.4 45.1 45.7 MBPP (Pass@1) 3-shot 27.8 39.0 46.2 TriviaQA (EM) 5-shot 60.1 59.5 63.3 NaturalQuestions (EM) 0-shot 35.2 32.7 35.1 MMLU (Acc.) 0-shot 50.0 49.7 47.2 WinoGrande (Acc.) 0-shot 65.1 68.4 69.0 CLUEWSC (EM) 5-shot 48.4 66.2 68.2 CEval (Acc.) 0-shot 35.1 44.7 40.0 CMMLU (Acc.) 0-shot 36.9 51.2 49.3\n\nCHID (Zheng et al., 2019) due to the observed instability of results, hindering the derivation of solid conclusions. (3) We additionally include BBH (Suzgun et al., 2022) to provide a more comprehensive assessment of the reasoning ability of the chat models.\n\nTable 5 | Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations.\n\n6.2. Evaluations\n\nBaselines. In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. Correspondingly, we construct three chat models, including LLaMA2 SFT 7B3, DeepSeek Chat 7B, and DeepSeekMoE Chat 16B.\n\nSubsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks.\n\nResults. The evaluation results are presented in Table 5. Our key observations include: (1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intensive tasks (TriviaQA, NaturalQuestions). (2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. In addition, it also surpasses DeepSeek Chat 7B. (3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1). However, it is worth noting that, after supervised finetuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. (4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations.\n\n",
+          "keyphrases": [
+            "DeepSeekMoE 16B",
+            "supervised fine-tuning",
+            "chat model",
+            "bilingual scenarios",
+            "evaluation benchmarks"
+          ],
+          "title": "Alignment For Deepseekmoe 16B",
+          "embedding": [
+            0.0034426983911544085,
+            -0.007639183662831783,
+            -0.011979537084698677,
+            0.00880228541791439,
+            0.011939010582864285,
+            0.013551952317357063,
+            -0.013365532271564007,
+            0.05112782120704651,
+            -0.01517299935221672,
+            0.02819000743329525,
+            -0.013219637796282768,
+            0.0074000791646540165,
+            -0.008145760744810104,
+            0.00818223413079977,
+            0.01559447217732668,
+            -0.004850983154028654,
+            0.02065213769674301,
+            0.04480573907494545,
+            0.0024822282139211893,
+            -0.021673398092389107,
+            -0.019825404509902,
+            -0.0646149292588234,
+            -0.037348922342061996,
+            0.011112277396023273,
+            0.0042836167849600315,
+            0.004996877163648605,
+            0.019955087453126907,
+            0.009685755707323551,
+            -0.005908716004341841,
+            0.017928779125213623,
+            -0.009920807555317879,
+            0.01728036068379879,
+            0.021122241392731667,
+            -0.012060589157044888,
+            -0.010982593521475792,
+            -0.028498005121946335,
+            0.05323518067598343,
+            0.011598590761423111,
+            -0.01838267222046852,
+            0.021511293947696686,
+            -0.009807334281504154,
+            0.016680574044585228,
+            -0.01202006358653307,
+            -0.014605632983148098,
+            0.0015673497691750526,
+            -0.021851712837815285,
+            -0.03183736279606819,
+            -0.01961466856300831,
+            -0.0014396923361346126,
+            -0.032809991389513016,
+            -0.012806271202862263,
+            -0.0026564907748252153,
+            -0.007140711881220341,
+            0.010480068624019623,
+            -0.033263884484767914,
+            0.034593142569065094,
+            0.045778367668390274,
+            0.03880786523222923,
+            0.04791814833879471,
+            -0.014200371690094471,
+            0.028481794521212578,
+            -0.0202792976051569,
+            0.006269399076700211,
+            -0.01276574470102787,
+            -0.017247939482331276,
+            -0.04185543209314346,
+            -0.018755514174699783,
+            -0.01043954212218523,
+            -0.004026275593787432,
+            0.014378686435520649,
+            -0.025904331356287003,
+            0.03783523663878441,
+            -0.018204357475042343,
+            -0.008129550144076347,
+            0.0036027769092470407,
+            0.0015926785999909043,
+            0.02387802116572857,
+            0.028433164581656456,
+            -0.003852012800052762,
+            -0.03744618594646454,
+            0.015083841979503632,
+            0.006763818673789501,
+            -0.014313844963908195,
+            0.028838425874710083,
+            0.06814881414175034,
+            -0.034366194158792496,
+            -0.024753388017416,
+            -0.07748604565858841,
+            0.006536871660500765,
+            0.0009523651096969843,
+            -0.027719903737306595,
+            -0.019079722464084625,
+            -0.01313048042356968,
+            -0.028757372871041298,
+            -0.025061385706067085,
+            0.0002973607915919274,
+            -0.033263884484767914,
+            -0.007444658316671848,
+            -0.045065104961395264,
+            0.016680574044585228,
+            0.025353174656629562,
+            0.017150677740573883,
+            0.010893435217440128,
+            -0.011955221183598042,
+            0.004008038435131311,
+            0.010763752274215221,
+            -0.020749401301145554,
+            -0.015805207192897797,
+            -0.004713193979114294,
+            0.08585064858198166,
+            -0.03151315450668335,
+            -0.012919744476675987,
+            -0.02702285349369049,
+            0.017572149634361267,
+            0.009450703859329224,
+            -0.010455752722918987,
+            -0.026390643790364265,
+            -0.03867818042635918,
+            0.0368950292468071,
+            -0.02243529073894024,
+            0.015707945451140404,
+            0.003924959804862738,
+            0.03371777758002281,
+            -0.0426335334777832,
+            -0.029713790863752365,
+            0.030427051708102226,
+            0.0018024016171693802,
+            -0.010585436597466469,
+            0.014921736903488636,
+            0.00511035043746233,
+            -0.017588360235095024,
+            0.0023201233707368374,
+            -0.011841747909784317,
+            0.0060262419283390045,
+            0.02198139764368534,
+            0.02209486998617649,
+            0.011525643989443779,
+            0.022613605484366417,
+            -0.01522973645478487,
+            -0.0017942963168025017,
+            0.0014224686892703176,
+            -0.03297209367156029,
+            0.015278367325663567,
+            0.0035744085907936096,
+            -0.012530692853033543,
+            0.008761758916079998,
+            0.025223489850759506,
+            0.06182673200964928,
+            0.020246876403689384,
+            0.004871245939284563,
+            -0.012619850225746632,
+            0.018868986517190933,
+            -0.028449373319745064,
+            0.024169810116291046,
+            0.0041985115967690945,
+            -0.03540366515517235,
+            0.01993887685239315,
+            0.0004736496484838426,
+            -0.012895428575575352,
+            -0.020311718806624413,
+            0.032647885382175446,
+            -0.031269997358322144,
+            -0.009588493034243584,
+            -0.0018652171129360795,
+            0.027266010642051697,
+            0.03141589090228081,
+            0.0100829117000103,
+            0.00386214442551136,
+            -0.04636194184422493,
+            0.037413764744997025,
+            -0.010585436597466469,
+            0.039650809019804,
+            -0.009531755931675434,
+            0.0054710335098207,
+            0.019792983308434486,
+            -0.006848923396319151,
+            0.008648285642266273,
+            0.007845867425203323,
+            -0.06597661226987839,
+            0.015432367101311684,
+            -0.029324740171432495,
+            0.00394724914804101,
+            0.02901674062013626,
+            -0.004611878655850887,
+            0.026828328147530556,
+            -0.0014569159829989076,
+            0.017491096630692482,
+            -0.0015399946132674813,
+            0.01872309297323227,
+            0.011639117263257504,
+            0.022013816982507706,
+            0.02780095487833023,
+            0.022954024374485016,
+            -0.005102245137095451,
+            0.011679643765091896,
+            -0.0150108952075243,
+            -0.012814376503229141,
+            -0.0010678647086024284,
+            0.008826600387692451,
+            0.010609752498567104,
+            0.003880381118506193,
+            -0.01966330036520958,
+            -0.01102311909198761,
+            -0.0032015678007155657,
+            0.018706882372498512,
+            0.005750664044171572,
+            -0.0019341117003932595,
+            0.021397819742560387,
+            0.011760695837438107,
+            -0.029308529570698738,
+            -0.02637443318963051,
+            -0.021738240495324135,
+            -0.011323013342916965,
+            -0.0371219739317894,
+            0.0024761492386460304,
+            0.017183097079396248,
+            0.016445521265268326,
+            0.001868256600573659,
+            0.013762688264250755,
+            -0.027006642892956734,
+            0.007493289653211832,
+            0.0007132606697268784,
+            -0.013008901849389076,
+            -0.04227690398693085,
+            -0.03841881453990936,
+            0.005268402863293886,
+            0.008648285642266273,
+            -0.0009052534005604684,
+            0.03841881453990936,
+            -0.009393966756761074,
+            0.01789635792374611,
+            0.031043048948049545,
+            0.006435556337237358,
+            0.012109220959246159,
+            0.014184161089360714,
+            -0.035144299268722534,
+            0.023829391226172447,
+            -0.007262290455400944,
+            0.021446451544761658,
+            0.033652935177087784,
+            -0.02037656120955944,
+            0.00043211033334955573,
+            -0.027930639684200287,
+            -0.00955607183277607,
+            0.026617590337991714,
+            -0.006220767740160227,
+            0.0009508453658781946,
+            0.017102045938372612,
+            0.025726014748215675,
+            0.033977143466472626,
+            0.009069757536053658,
+            -0.037640709429979324,
+            0.007237974554300308,
+            0.021187083795666695,
+            -0.00024974255939014256,
+            0.04269837588071823,
+            0.05255434289574623,
+            -0.03050810471177101,
+            -0.018739303573966026,
+            -0.054142966866493225,
+            0.01916077546775341,
+            -0.014848790131509304,
+            -0.011898485012352467,
+            0.014727211557328701,
+            0.007477079052478075,
+            0.009977543726563454,
+            0.02397528477013111,
+            0.017134467139840126,
+            0.004692931193858385,
+            0.012644166126847267,
+            -0.011306802742183208,
+            0.012003852985799313,
+            -0.049993086606264114,
+            0.007148817181587219,
+            0.005118455737829208,
+            -0.027557797729969025,
+            0.003432566998526454,
+            -0.010690804570913315,
+            0.0005987742333672941,
+            0.018804144114255905,
+            0.027557797729969025,
+            0.023780759423971176,
+            0.03151315450668335,
+            0.008931968361139297,
+            -0.046005312353372574,
+            -0.01732899248600006,
+            -0.021170873194932938,
+            -0.03909965232014656,
+            -0.005957347806543112,
+            -0.029875895008444786,
+            0.0010040359338745475,
+            -0.005418349523097277,
+            -0.036311451345682144,
+            -0.0011904563289135695,
+            -0.01503521017730236,
+            0.030443262308835983,
+            -0.017815306782722473,
+            0.04136911779642105,
+            0.056801486760377884,
+            -0.004449774045497179,
+            -0.06717618554830551,
+            0.009321019984781742,
+            -0.0016950072022154927,
+            -0.012392903678119183,
+            -0.018917618319392204,
+            0.014443528838455677,
+            0.020846663042902946,
+            0.03050810471177101,
+            -0.043541319668293,
+            -0.04386553168296814,
+            0.013949109241366386,
+            -0.0058844005689024925,
+            -0.01599973440170288,
+            -0.0034467510413378477,
+            -0.000564326997846365,
+            0.03206430748105049,
+            0.03540366515517235,
+            0.017572149634361267,
+            -0.038580916821956635,
+            -2.4299873985000886e-05,
+            0.01407068781554699,
+            0.02404012717306614,
+            -0.055407386273145676,
+            0.023343076929450035,
+            0.010544910095632076,
+            0.028660111129283905,
+            0.02919505536556244,
+            0.034204091876745224,
+            -0.015578261576592922,
+            5.2050803788006306e-05,
+            -0.017134467139840126,
+            -0.018009832128882408,
+            -0.01895003952085972,
+            0.0024700702633708715,
+            -0.027557797729969025,
+            -0.028676321730017662,
+            -0.041012488305568695,
+            -0.01684267818927765,
+            -0.03013526275753975,
+            0.006500398274511099,
+            -0.018868986517190933,
+            -0.0022836499847471714,
+            -0.010309858247637749,
+            0.01783151738345623,
+            -0.01559447217732668,
+            -0.0027334904298186302,
+            -0.023067498579621315,
+            -0.006881344132125378,
+            0.0018307699356228113,
+            -0.0031286205630749464,
+            0.0011549959890544415,
+            0.0025815172120928764,
+            0.030329788103699684,
+            0.030702630057930946,
+            -0.014362475834786892,
+            -0.02569359354674816,
+            -0.0497337207198143,
+            0.00334948836825788,
+            -0.005219771061092615,
+            0.018982460722327232,
+            0.012806271202862263,
+            -0.03747860714793205,
+            -0.039132073521614075,
+            0.010601647198200226,
+            -0.0371219739317894,
+            -0.034041985869407654,
+            0.01977677270770073,
+            0.012141642160713673,
+            -0.005207613576203585,
+            0.0029644896276295185,
+            -0.005458875559270382,
+            0.03174009919166565,
+            0.0023403866216540337,
+            -0.0022897287271916866,
+            0.024883070960640907,
+            0.02076561190187931,
+            0.008413233794271946,
+            -0.019760562106966972,
+            -0.005292718298733234,
+            0.0460701547563076,
+            -0.025077596306800842,
+            0.0008312931749969721,
+            -0.015659313648939133,
+            -0.029276108369231224,
+            0.017523517832159996,
+            0.021916555240750313,
+            0.000971615023445338,
+            -0.014005845412611961,
+            -0.03838639333844185,
+            0.027233589440584183,
+            0.015278367325663567,
+            -0.046556469053030014,
+            -0.0268121175467968,
+            0.0055723488330841064,
+            0.02869253046810627,
+            0.014848790131509304,
+            -0.010285543277859688,
+            0.006245083641260862,
+            0.02929231896996498,
+            -0.0056777168065309525,
+            -0.014905527234077454,
+            0.01545668300241232,
+            -0.02094392664730549,
+            -0.002857095329090953,
+            0.015278367325663567,
+            -6.775469955755398e-05,
+            0.008745548315346241,
+            -0.005000929813832045,
+            0.0182205680757761,
+            -0.0007907669642008841,
+            -0.006431503687053919,
+            -0.037737973034381866,
+            -0.025223489850759506,
+            -0.046102575957775116,
+            0.03439861536026001,
+            0.037089552730321884,
+            0.008737443014979362,
+            0.017491096630692482,
+            -0.01631583832204342,
+            -0.00244372827000916,
+            -0.033490829169750214,
+            0.01182553730905056,
+            0.02536938525736332,
+            0.005013087764382362,
+            0.004587562754750252,
+            0.0069948178716003895,
+            0.009726282209157944,
+            0.014500265009701252,
+            0.0004141268436796963,
+            -0.04347648099064827,
+            0.004790193866938353,
+            0.008003919385373592,
+            0.034982193261384964,
+            -0.031205154955387115,
+            -0.03125378489494324,
+            -0.011939010582864285,
+            0.010309858247637749,
+            0.036473557353019714,
+            -0.034528300166130066,
+            -0.0182205680757761,
+            -0.021835502237081528,
+            0.016048364341259003,
+            -0.006030294578522444,
+            -0.012441535480320454,
+            -0.0003432060475461185,
+            0.008429444395005703,
+            -0.012490167282521725,
+            0.01711825653910637,
+            -0.026115067303180695,
+            -8.656137651996687e-05,
+            -0.04493542015552521,
+            0.004437616094946861,
+            -0.06639808416366577,
+            -0.019760562106966972,
+            0.03543608635663986,
+            -0.024121178314089775,
+            0.0214302409440279,
+            -0.006366661749780178,
+            0.012247010134160519,
+            -0.05654211714863777,
+            0.002972594927996397,
+            -0.008510496467351913,
+            0.04182301089167595,
+            -0.0083078658208251,
+            -0.005134666338562965,
+            -0.034301355481147766,
+            0.002052650786936283,
+            -0.007704025600105524,
+            0.009985649026930332,
+            0.03146452084183693,
+            -0.014216582290828228,
+            -0.0016554942121729255,
+            0.012530692853033543,
+            0.00012430136848706752,
+            0.011955221183598042,
+            0.03073505125939846,
+            -0.017523517832159996,
+            -0.04227690398693085,
+            0.02071698009967804,
+            0.036797765642404556,
+            -0.021057400852441788,
+            -0.016080785542726517,
+            -0.02530454285442829,
+            0.0042836167849600315,
+            0.006447714287787676,
+            0.004769930616021156,
+            0.017685621976852417,
+            -0.0047739832662045956,
+            0.02065213769674301,
+            0.015059526078402996,
+            0.021900344640016556,
+            0.030054211616516113,
+            0.01636446826159954,
+            0.017636990174651146,
+            -0.017458675429224968,
+            -0.03783523663878441,
+            0.010966382920742035,
+            -0.03543608635663986,
+            0.0035926452837884426,
+            0.02747674658894539,
+            -0.03079989179968834,
+            -0.009645229205489159,
+            -0.006646292749792337,
+            4.45787918579299e-05,
+            0.013138585723936558,
+            0.0061478205025196075,
+            0.0366680808365345,
+            0.011031224392354488,
+            0.005341349635273218,
+            -0.0007411224069073796,
+            0.028887057676911354,
+            -0.015391840599477291,
+            0.035144299268722534,
+            -0.015853838995099068,
+            -0.01553773507475853,
+            0.006549029611051083,
+            -0.010520595125854015,
+            0.019258037209510803,
+            -0.008206550031900406,
+            -0.0043200901709496975,
+            -0.022662237286567688,
+            -0.008769864216446877,
+            0.0071204486303031445,
+            0.016542784869670868,
+            -0.019484983757138252,
+            0.02603401429951191,
+            -0.0005334257730282843,
+            -0.014865000732243061,
+            0.01950119435787201,
+            -0.03024873696267605,
+            -0.023294445127248764,
+            0.0031002522446215153,
+            -0.003789197187870741,
+            0.003740565851330757,
+            -0.005426454823464155,
+            -0.0024072546511888504,
+            -0.015772787854075432,
+            -0.002682832768186927,
+            -0.02120329439640045,
+            -0.014305739663541317,
+            -0.006678713485598564,
+            -0.005000929813832045,
+            -0.01509194727987051,
+            0.002642306499183178,
+            -0.023861810564994812,
+            0.007683762814849615,
+            0.0323074646294117,
+            -0.03922933712601662,
+            0.019857825711369514,
+            0.03627903014421463,
+            0.003473093034699559,
+            -0.018917618319392204,
+            -0.022581184282898903,
+            -0.010917751118540764,
+            -0.010042386129498482,
+            -0.05044697970151901,
+            0.001210719463415444,
+            0.0015308762667700648,
+            -0.0021843607537448406,
+            -0.022921603173017502,
+            0.0013393900590017438,
+            0.016583310440182686,
+            -0.012676587328314781,
+            0.012247010134160519,
+            -0.005045508500188589,
+            -0.014727211557328701,
+            0.014832579530775547,
+            0.027768533676862717,
+            -0.02679590694606304,
+            -0.0029462529346346855,
+            -0.0036088558845221996,
+            0.0033839354291558266,
+            0.0045835101045668125,
+            -0.008931968361139297,
+            -0.007704025600105524,
+            0.033750198781490326,
+            0.001635231077671051,
+            -0.034917350858449936,
+            -0.012911639176309109,
+            -0.002362675964832306,
+            0.027444325387477875,
+            -0.024769596755504608,
+            -0.004899614490568638,
+            0.03180494159460068,
+            0.026293382048606873,
+            -0.014881211332976818,
+            -0.03130241855978966,
+            -0.028060322627425194,
+            0.02021445520222187,
+            0.00838891789317131,
+            0.025888120755553246,
+            0.00861586444079876,
+            0.0009341283002868295,
+            -0.0538511797785759,
+            0.01899867132306099,
+            -0.018690671771764755,
+            -0.007529763039201498,
+            -0.003687881864607334,
+            0.02055487595498562,
+            -0.07592984288930893,
+            -0.03456072136759758,
+            0.03462556377053261,
+            0.033652935177087784,
+            -0.011104172095656395,
+            -0.00919133611023426,
+            0.012263220734894276,
+            -0.002164097735658288,
+            0.034982193261384964,
+            0.006196451839059591,
+            0.023521391674876213,
+            0.009596598334610462,
+            -0.0013880215119570494,
+            0.011841747909784317,
+            -0.01700478233397007,
+            0.0011874169576913118,
+            -0.02775232493877411,
+            0.016032153740525246,
+            -0.005941137205809355,
+            -0.004397090058773756,
+            -0.02215971238911152,
+            0.051646556705236435,
+            -0.014297634363174438,
+            0.034366194158792496,
+            -0.004624036606401205,
+            0.02083045430481434,
+            0.05394844338297844,
+            0.016972361132502556,
+            -0.03683018684387207,
+            0.0030739102512598038,
+            -0.018301621079444885,
+            -0.041174594312906265,
+            0.012368588708341122,
+            -0.022192133590579033,
+            -0.02598538249731064,
+            0.029227476567029953,
+            0.0009467927156947553,
+            0.006654397584497929,
+            -0.0027719903737306595,
+            0.0004359096637926996,
+            -0.008567233569920063,
+            -0.013673530891537666,
+            0.017523517832159996,
+            -0.006642240099608898,
+            0.04192027449607849,
+            -0.009977543726563454,
+            -0.008364601992070675,
+            0.002861147979274392,
+            0.013357426971197128,
+            0.045000262558460236,
+            0.019582247361540794,
+            -0.016923731192946434,
+            0.03146452084183693,
+            0.022062448784708977,
+            0.03715439513325691,
+            -0.018804144114255905,
+            -0.05135476961731911,
+            0.006358556915074587,
+            -0.0009148783865384758,
+            0.016080785542726517,
+            0.0018115199636667967,
+            0.022564973682165146,
+            0.001599770737811923,
+            0.01789635792374611,
+            -0.0019016907317563891,
+            0.04386553168296814,
+            -0.009167020209133625,
+            -0.009653334505856037,
+            0.027930639684200287,
+            -0.0005764848319813609,
+            0.0336853563785553,
+            0.02039276994764805,
+            -0.03511187806725502,
+            -0.034755248576402664,
+            0.01838267222046852,
+            0.026439275592565536,
+            -0.02237044833600521,
+            0.009102178737521172,
+            0.003746644826605916,
+            0.023991495370864868,
+            -0.014654264785349369,
+            0.012530692853033543,
+            0.0036919345147907734,
+            -0.013195321895182133,
+            -0.024964123964309692,
+            0.015043315477669239,
+            0.004052617587149143,
+            -0.0059006111696362495,
+            0.035533349961042404,
+            -0.03462556377053261,
+            -0.03362051397562027,
+            -0.00743655301630497,
+            -0.007071817293763161,
+            -0.0012198378099128604,
+            0.0371219739317894,
+            -0.005462928209453821,
+            0.01877172477543354,
+            -0.03118894435465336,
+            0.040850382298231125,
+            0.00765539426356554,
+            0.004688878543674946,
+            -0.0071731326170265675,
+            0.0016960203647613525,
+            -0.009977543726563454,
+            0.014257107861340046,
+            0.007736446801573038,
+            -0.007716183550655842,
+            0.0018084804760292172,
+            -0.011006908491253853,
+            -0.007290658541023731,
+            0.01642931066453457,
+            -0.027541587129235268,
+            -0.008518601767718792,
+            0.02259739488363266,
+            0.03465798497200012,
+            0.0007927932892926037,
+            0.001413350342772901,
+            -0.00983165018260479,
+            0.006362609565258026,
+            -0.02232181653380394,
+            0.04299016669392586,
+            -0.02138160914182663,
+            0.014751527458429337,
+            0.01590247079730034,
+            -0.042179640382528305,
+            -0.021446451544761658,
+            0.026050224900245667,
+            0.020198244601488113,
+            -0.018415093421936035,
+            -0.011225750669836998,
+            -0.014962263405323029,
+            0.016810256987810135,
+            -0.03068641945719719,
+            -0.04292532429099083,
+            0.010982593521475792,
+            -0.0015845734160393476,
+            -0.0047050886787474155,
+            -0.01595110259950161,
+            0.018690671771764755,
+            0.01085290964692831,
+            -0.054467178881168365,
+            -0.00502524571493268,
+            0.010009964928030968,
+            0.006107294466346502,
+            0.018706882372498512,
+            0.004672667942941189,
+            0.03362051397562027,
+            0.0006463924655690789,
+            0.01838267222046852,
+            -0.007898551411926746,
+            0.01307374332100153,
+            -0.010950172320008278,
+            -0.019371511414647102,
+            0.03279377892613411,
+            0.011420276015996933,
+            -0.03266409784555435,
+            -0.0044254581443965435,
+            0.0021742291282862425,
+            -0.016631942242383957,
+            0.002182334428653121,
+            -0.015878155827522278,
+            -0.006184294354170561,
+            0.028984319418668747,
+            0.00685297604650259,
+            0.025239700451493263,
+            0.029616527259349823,
+            -0.0035865663085132837,
+            -0.01581331342458725,
+            0.012133536860346794,
+            0.009718176908791065,
+            -0.01160669606178999,
+            0.022013816982507706,
+            -0.031091680750250816,
+            0.035371243953704834,
+            -0.06166462600231171,
+            0.030929576605558395,
+            0.027671271935105324,
+            -0.016794046387076378,
+            0.003426488023251295,
+            -0.021867923438549042,
+            0.01766941137611866,
+            -0.01883656531572342,
+            0.029827265068888664,
+            -0.020360350608825684,
+            0.02913021482527256,
+            -0.021170873194932938,
+            0.021576134487986565,
+            0.002658517099916935,
+            -0.018674461171030998,
+            0.008060656487941742,
+            0.0008348391856998205,
+            -0.010998803190886974,
+            -0.013600584119558334,
+            -0.029454423114657402,
+            0.012425324879586697,
+            0.058292847126722336,
+            -0.033490829169750214,
+            -0.0017031125025823712,
+            0.018917618319392204,
+            -0.02259739488363266,
+            -0.03152936324477196,
+            0.0100829117000103,
+            0.004534878768026829,
+            0.030864734202623367,
+            0.014240897260606289,
+            -0.04075312241911888,
+            -0.04172574728727341,
+            0.008324076421558857,
+            0.010350384749472141,
+            -0.004243090283125639,
+            0.046264681965112686,
+            0.01040712185204029,
+            0.014840684831142426,
+            0.01041522715240717,
+            0.0040566702373325825,
+            0.001815572613850236,
+            0.007890446111559868,
+            -0.007570289541035891,
+            -0.012587429955601692,
+            -0.001791256945580244,
+            -0.028498005121946335,
+            -0.005240034312009811,
+            0.017361413687467575,
+            -0.011598590761423111,
+            -0.010204490274190903,
+            -0.04503268375992775,
+            -0.01572415605187416,
+            0.017069624736905098,
+            -0.003262357087805867,
+            0.04668615385890007,
+            0.013171006925404072,
+            -0.004676720593124628,
+            -0.01933909021317959,
+            -0.02076561190187931,
+            0.04619983956217766,
+            -0.017636990174651146,
+            0.04000743851065636,
+            -0.0015734287444502115,
+            -0.013535741716623306,
+            -0.025110017508268356,
+            0.033879879862070084,
+            -0.03728407993912697,
+            -0.013957214541733265,
+            -0.008117392659187317,
+            -0.0035359086468815804,
+            -0.005458875559270382,
+            0.0018409014446660876,
+            -0.0008019116939976811,
+            -0.030167683959007263,
+            0.024396756663918495,
+            -0.025336964055895805,
+            0.031983256340026855,
+            0.02055487595498562,
+            0.010123438201844692,
+            0.00048200818127952516,
+            -0.03222641348838806,
+            -0.005969505291432142,
+            0.0005481165135279298,
+            0.008761758916079998,
+            -0.0011539828265085816,
+            0.00394724914804101,
+            -0.015934891998767853,
+            -0.01356816291809082,
+            -0.012287535704672337,
+            -0.04253627359867096,
+            -0.015480998903512955,
+            0.021089820191264153,
+            0.010374700650572777,
+            -0.03311799094080925,
+            -0.017636990174651146,
+            -0.03653839975595474,
+            0.006840818095952272,
+            -0.010625963099300861,
+            -0.008680706843733788,
+            0.012473956681787968,
+            0.001798348966985941,
+            0.015019000507891178,
+            -0.005479138810187578,
+            -0.023245813325047493,
+            0.026487907394766808,
+            -0.03219399228692055,
+            -0.029486844316124916,
+            -0.01043954212218523,
+            -0.003635197877883911,
+            -0.010042386129498482,
+            0.031059259548783302,
+            0.021251926198601723,
+            0.0007943130331113935,
+            -0.050479400902986526,
+            -0.03050810471177101,
+            -0.051873501390218735,
+            0.024396756663918495,
+            0.0009315954521298409,
+            -0.00938586238771677,
+            0.003252225462347269,
+            0.02039276994764805,
+            0.008178181946277618,
+            -0.01313048042356968,
+            -0.02603401429951191,
+            -0.024023916572332382,
+            -0.013884266838431358,
+            -0.005195455625653267,
+            -0.029065372422337532,
+            -0.033263884484767914,
+            -0.037089552730321884,
+            -0.01334932167083025,
+            0.005916821304708719,
+            0.011776906438171864,
+            -0.003000963246449828,
+            0.0008049511234275997,
+            -0.006759766023606062,
+            -0.02459128201007843,
+            0.004125564359128475,
+            0.015205420553684235,
+            -0.01998750865459442,
+            0.02708769403398037,
+            0.009629018604755402,
+            0.0141517398878932,
+            -0.029843473806977272,
+            0.013859951868653297,
+            0.009118389338254929,
+            0.014338160865008831,
+            -0.004976614378392696,
+            -0.003906723111867905,
+            0.02060350775718689,
+            -0.01972814090549946,
+            0.026017803698778152,
+            -0.006350451614707708,
+            -0.0024700702633708715,
+            -0.010066702030599117,
+            0.020360350608825684,
+            0.003473093034699559,
+            0.012725219130516052,
+            0.022224554792046547,
+            -2.4996417778311297e-05,
+            0.037089552730321884,
+            -0.005090087652206421,
+            -0.01326016429811716,
+            0.00575876934453845,
+            -0.02121950499713421,
+            -0.005702032707631588,
+            0.005993821192532778,
+            -0.029000530019402504,
+            0.0012887322809547186,
+            0.014792053960263729,
+            0.009880281053483486,
+            -0.0318535715341568,
+            0.023634864017367363,
+            0.02603401429951191,
+            0.006848923396319151,
+            0.010399016551673412,
+            -0.033815041184425354,
+            0.0006266359705477953,
+            0.011274381540715694,
+            -0.004778035916388035,
+            0.036700502038002014,
+            0.0201658234000206,
+            -0.03614934906363487,
+            0.010998803190886974,
+            -0.01043954212218523,
+            0.00396548630669713,
+            -0.004765877965837717,
+            0.029616527259349823,
+            -0.022338027134537697,
+            -0.011006908491253853,
+            -0.029324740171432495,
+            0.010180175304412842,
+            -0.021948976442217827,
+            -0.015497208572924137,
+            -0.03524155914783478,
+            0.03579271584749222,
+            0.005811453331261873,
+            -0.013551952317357063,
+            0.0029989369213581085,
+            -0.02039276994764805,
+            -0.0071690804325044155,
+            -0.02951926551759243,
+            -0.012660376727581024,
+            -0.012230799533426762,
+            -0.0007537868223153055,
+            0.016810256987810135,
+            0.02491549216210842,
+            0.006439608987420797,
+            0.029389580711722374,
+            0.03728407993912697,
+            0.00218841340392828,
+            -0.014167950488626957,
+            0.06169704720377922,
+            -0.0007922867080196738,
+            -0.024607492610812187,
+            0.04075312241911888,
+            -0.024964123964309692,
+            0.0016544810496270657,
+            0.01625099591910839,
+            0.014743422158062458,
+            -0.03211294114589691,
+            0.016923731192946434,
+            -0.006739502772688866,
+            -0.033815041184425354,
+            0.003924959804862738,
+            -0.005171139724552631,
+            -0.03195083513855934,
+            -0.04185543209314346,
+            -0.010115332901477814,
+            0.004879351239651442,
+            -0.02160855568945408,
+            0.017020992934703827,
+            0.020473822951316833,
+            -0.002723358804360032,
+            -0.014500265009701252,
+            0.02275949902832508,
+            0.001914861728437245,
+            0.006970501970499754,
+            -0.006771923508495092,
+            0.007833709008991718,
+            -0.0027253851294517517,
+            -0.0010035294108092785,
+            -0.024834439158439636,
+            -0.015140579082071781,
+            -0.041239432990550995,
+            0.0159267857670784,
+            -0.023780759423971176,
+            -0.046880677342414856,
+            0.0056412434205412865,
+            -0.03744618594646454,
+            0.03355567157268524,
+            0.017474886029958725,
+            0.0005683795898221433,
+            -0.0033434093929827213,
+            -0.007716183550655842,
+            -0.021689608693122864,
+            0.010917751118540764,
+            0.022856762632727623,
+            0.004316037520766258,
+            -0.0059897685423493385,
+            0.014192266389727592,
+            -0.022305605933070183,
+            0.000483021343825385,
+            -0.060335367918014526,
+            -0.02381318062543869,
+            0.025628753006458282,
+            0.007983656600117683,
+            0.0005856032366864383,
+            -0.014824474230408669,
+            0.006808397360146046,
+            -0.03266409784555435,
+            -0.019420143216848373,
+            0.025126228109002113,
+            0.019744351506233215,
+            0.014086898416280746,
+            -0.004960403777658939,
+            0.0007396026630885899,
+            0.0182205680757761,
+            -0.008850916288793087,
+            0.008980600163340569,
+            0.0030556735582649708,
+            0.025045175105333328,
+            0.00021225583623163402,
+            0.0077931834384799,
+            0.01210111565887928,
+            0.028822215273976326,
+            0.0034710667096078396,
+            0.010544910095632076,
+            0.006038399878889322,
+            0.0009300757083110511,
+            -0.0134060587733984,
+            -0.013276374898850918,
+            -0.008153866045176983,
+            0.024137388914823532,
+            0.018852775916457176,
+            0.00955607183277607,
+            -0.009580387733876705,
+            -0.009880281053483486,
+            0.008956284262239933,
+            0.009004916064441204,
+            -0.001745664980262518,
+            0.01223890483379364,
+            0.012117326259613037,
+            -0.029778633266687393,
+            -0.03355567157268524,
+            -0.0013799162115901709,
+            -0.004984719213098288,
+            0.005795242730528116,
+            0.00802012998610735,
+            0.021948976442217827,
+            0.0013181137619540095,
+            -0.013479005545377731,
+            -0.004980667028576136,
+            0.02530454285442829,
+            0.005742558743804693,
+            -0.01317911222577095,
+            -0.015716049820184708,
+            -0.009766807779669762,
+            0.00772834150120616,
+            0.011193329468369484,
+            -0.003473093034699559,
+            0.0033596197608858347,
+            0.012741428799927235,
+            0.08163592219352722,
+            -0.007533815689384937,
+            0.0036818028893321753,
+            -0.03854849562048912,
+            -0.01284679677337408,
+            -0.008640180341899395,
+            0.019484983757138252,
+            -0.0019867955707013607,
+            0.032323677092790604,
+            0.015051420778036118,
+            0.01849614642560482,
+            0.0059208739548921585,
+            0.015326999127864838,
+            0.005365665536373854,
+            -0.02929231896996498,
+            0.02227318473160267,
+            -0.0005982676520943642,
+            0.011817432008683681,
+            0.0019857825245708227,
+            -0.011979537084698677,
+            -0.011006908491253853,
+            0.02032792940735817,
+            0.003671671263873577,
+            0.02397528477013111,
+            -0.02159234508872032,
+            0.019744351506233215,
+            0.01849614642560482,
+            0.011979537084698677,
+            -0.010301753878593445,
+            -0.005004982464015484,
+            0.0014913631603121758,
+            0.00517924502491951,
+            -0.017037203535437584,
+            -0.0016139548970386386,
+            -0.019582247361540794,
+            0.06652776896953583,
+            -0.030556734651327133,
+            0.0007244053413160145,
+            0.0031346995383501053,
+            -0.018577197566628456,
+            0.004713193979114294,
+            0.001658533699810505,
+            0.01659141480922699,
+            0.0057263486087322235,
+            -0.006581450812518597,
+            0.005531822796911001,
+            0.024688545614480972,
+            -0.00560882268473506,
+            0.01762077957391739,
+            0.005523717496544123,
+            -0.012263220734894276,
+            -0.014638054184615612,
+            0.012076799757778645,
+            0.011939010582864285,
+            0.009142705239355564,
+            0.025904331356287003,
+            0.008012024685740471,
+            -0.009880281053483486,
+            -0.020895294845104218,
+            -0.011582380160689354,
+            0.006151873152703047,
+            0.005876295268535614,
+            0.03815944492816925,
+            0.030216315761208534,
+            -0.0006884383619762957,
+            -0.013592478819191456,
+            0.016259100288152695,
+            -0.005365665536373854,
+            0.024169810116291046,
+            -0.0029259896837174892,
+            -0.013608689419925213,
+            0.008980600163340569,
+            0.009564177133142948,
+            -0.008178181946277618,
+            -0.02747674658894539,
+            -0.011582380160689354,
+            -0.01013964880257845,
+            -0.01581331342458725,
+            0.010099122300744057,
+            -0.027071483433246613,
+            -0.002474122913554311,
+            0.0011884300038218498,
+            0.027039064094424248,
+            0.03738134354352951,
+            -0.011768801137804985,
+            -0.00539808627218008,
+            -0.0008814443135634065,
+            -0.025223489850759506,
+            -0.005000929813832045,
+            -0.020749401301145554,
+            -0.024964123964309692,
+            -0.016810256987810135,
+            -0.016105102375149727,
+            -0.005527770146727562,
+            -0.0029178846161812544,
+            0.004474089480936527,
+            0.0015045341569930315,
+            0.058130744844675064,
+            -0.01102311909198761,
+            -0.012084905058145523,
+            -0.02076561190187931,
+            0.006942133419215679,
+            0.019128354266285896,
+            -0.011387854814529419,
+            0.0027274114545434713,
+            -0.01393289864063263,
+            0.004010064993053675,
+            -0.014881211332976818,
+            -0.014362475834786892,
+            -0.023002656176686287,
+            0.002506543882191181,
+            -0.006455819588154554,
+            0.0028084637597203255,
+            -0.00627345172688365,
+            0.009726282209157944,
+            0.019857825711369514,
+            -3.7233421608107165e-05,
+            0.001175259007140994,
+            -0.008810389786958694,
+            0.015043315477669239,
+            -0.030491894111037254,
+            0.02392665296792984,
+            0.019144564867019653,
+            0.0033373304177075624,
+            -0.0053778234869241714,
+            0.02752537652850151,
+            0.00364127685315907,
+            -0.011720169335603714,
+            0.04568110406398773,
+            0.010399016551673412,
+            0.026179907843470573,
+            -0.009872175753116608,
+            0.005774979945272207,
+            -0.0181070938706398,
+            0.009231862612068653,
+            -0.006484187673777342,
+            0.0072420272044837475,
+            -0.006131610367447138,
+            0.02780095487833023,
+            -0.008777969516813755,
+            0.0013464820804074407,
+            0.002717280061915517,
+            0.04098006710410118,
+            -0.0019452563719823956,
+            0.015853838995099068,
+            -0.026617590337991714,
+            -0.00035536388168111444,
+            0.02126813679933548,
+            0.014362475834786892,
+            0.017183097079396248,
+            -0.000494925887323916,
+            -0.01950119435787201,
+            -0.0045997207053005695,
+            -0.014321950264275074,
+            0.0009179178159683943,
+            -0.030005579814314842,
+            0.002401175908744335,
+            -0.00473345723003149,
+            0.013187216594815254,
+            0.013876162469387054,
+            -0.011695853434503078,
+            0.02044140174984932,
+            -0.017701832577586174,
+            -0.02376454882323742,
+            0.008964389562606812,
+            0.0018135462887585163,
+            0.01700478233397007,
+            0.017020992934703827,
+            -0.022030027583241463,
+            -0.006982659921050072,
+            -0.01883656531572342,
+            -0.01849614642560482,
+            0.0032562781125307083,
+            0.034041985869407654,
+            -0.007096133194863796,
+            -0.010828593745827675,
+            0.018188146874308586,
+            -0.01750730723142624,
+            0.01732899248600006,
+            -0.007250132504850626,
+            0.01954982616007328,
+            -0.001856098766438663,
+            -0.006609818898141384,
+            0.005240034312009811,
+            -0.02358623407781124,
+            0.0012897454435005784,
+            0.00999375432729721,
+            0.001181337982416153,
+            -0.03229125589132309,
+            0.012036273255944252,
+            -0.0020678481087088585,
+            0.001809493638575077,
+            -0.006484187673777342,
+            0.014767738059163094,
+            0.006788134109228849,
+            0.00869691651314497,
+            -0.002474122913554311,
+            0.010074807330965996,
+            0.003671671263873577,
+            -0.015083841979503632,
+            -0.003112410195171833,
+            0.018593408167362213,
+            0.008599653840065002,
+            -0.011614801362156868,
+            0.00575876934453845,
+            -0.014946052804589272,
+            -0.016080785542726517,
+            0.019258037209510803,
+            0.019760562106966972,
+            -0.031432099640369415,
+            0.022791920229792595,
+            -0.024461599066853523,
+            -0.0035521192476153374,
+            0.017977410927414894,
+            -0.0020972294732928276,
+            -0.0008490233449265361,
+            -0.004474089480936527,
+            -0.027557797729969025,
+            -0.01428952906280756,
+            0.005924926605075598,
+            -0.011525643989443779,
+            0.00452677346765995,
+            0.025288332253694534,
+            0.015691734850406647,
+            -0.0056250328198075294,
+            -0.0016858888557180762,
+            0.030832313001155853,
+            -0.003071883926168084,
+            0.0054831914603710175,
+            0.005126561038196087,
+            -0.028870847076177597,
+            0.02995694801211357,
+            -0.024056337773799896,
+            0.007550026290118694,
+            -0.000363722414476797,
+            -0.00247817556373775,
+            -0.006358556915074587,
+            0.014411107636988163,
+            -0.0005481165135279298,
+            0.0016200337558984756,
+            -0.014475949108600616,
+            0.006269399076700211,
+            -0.019355300813913345,
+            0.030864734202623367,
+            -0.0016088890843093395,
+            -0.00017768194084055722,
+            0.015197315253317356,
+            -0.015205420553684235,
+            -0.017863936722278595,
+            0.01013964880257845,
+            0.008453760296106339,
+            -0.0230350773781538,
+            0.004778035916388035,
+            -0.01933909021317959,
+            -0.03871060162782669,
+            -0.026115067303180695,
+            0.003706118557602167,
+            0.008915758691728115,
+            0.0165022574365139,
+            -0.009426387958228588,
+            -0.0061113471165299416,
+            0.02692558988928795,
+            0.02358623407781124,
+            0.03378261998295784,
+            -0.013949109241366386,
+            0.009094073437154293,
+            0.0027963060420006514,
+            0.01317911222577095,
+            0.001355600543320179,
+            -0.004384932108223438,
+            0.0033839354291558266,
+            -0.004247142933309078,
+            -0.012587429955601692,
+            -0.0018044279422610998,
+            0.027606429532170296,
+            -0.006650345399975777,
+            -0.024380546063184738,
+            -0.03151315450668335,
+            0.003161041531711817,
+            -0.005231929011642933,
+            0.0040222229436039925,
+            0.03971565142273903,
+            -0.024510230869054794,
+            -0.02587191015481949,
+            0.02026308700442314,
+            -0.003205620450899005,
+            0.004567299969494343,
+            -0.0019847694784402847,
+            0.01027743797749281,
+            -0.014775843359529972,
+            -0.028643900528550148,
+            -0.022192133590579033,
+            -0.0027193063870072365,
+            0.0037425921764224768,
+            -0.007128553930670023,
+            -0.016972361132502556,
+            -0.007886393927037716,
+            0.007622973527759314,
+            -0.016510363668203354,
+            0.013762688264250755,
+            -0.02237044833600521,
+            0.010147754102945328,
+            -0.0012299694353714585,
+            0.030653998255729675,
+            -0.015513419173657894,
+            -0.01608889177441597,
+            -0.023894231766462326,
+            0.007254185155034065,
+            0.02099255844950676,
+            -0.0009913715766742826,
+            0.0012026141630485654,
+            0.006780028808861971,
+            0.005292718298733234,
+            0.007197448518127203,
+            -0.0069948178716003895,
+            0.019063511863350868,
+            -0.034820087254047394,
+            -0.0024356229696422815,
+            0.0019047302193939686,
+            0.039521124213933945,
+            -0.027168747037649155,
+            -0.003321120049804449,
+            -0.007817499339580536,
+            0.010228806175291538,
+            0.013519532047212124,
+            0.013162901625037193,
+            -0.0027679377235472202,
+            -0.01677783578634262,
+            -0.022451501339673996,
+            0.0005597677663899958,
+            -0.038516074419021606,
+            0.0026544644497334957,
+            0.015197315253317356,
+            -0.0013991661835461855,
+            0.03751102834939957,
+            0.0469779409468174,
+            -0.03738134354352951,
+            -0.006269399076700211,
+            0.009799228981137276,
+            0.03410682827234268,
+            -0.020911505445837975,
+            0.020408980548381805,
+            -0.011063645593822002,
+            0.01620236411690712,
+            0.0035602243151515722,
+            -0.009085968136787415,
+            -0.02786579728126526,
+            -0.014905527234077454,
+            0.021089820191264153,
+            0.008145760744810104,
+            -0.021478872746229172,
+            0.009734386578202248,
+            0.011558065190911293,
+            -0.008364601992070675,
+            -0.025061385706067085,
+            0.02598538249731064,
+            0.01883656531572342,
+            0.01157427579164505,
+            -0.018204357475042343,
+            -0.011768801137804985,
+            -0.02674727514386177,
+            0.008996810764074326,
+            -0.020846663042902946,
+            0.01723172888159752,
+            -0.023861810564994812,
+            0.0008839771617203951,
+            -0.01844751462340355,
+            -0.007225817069411278,
+            0.006532819475978613,
+            -0.04370342567563057,
+            -0.006476082839071751,
+            -0.005924926605075598,
+            0.012960270047187805,
+            0.028951898217201233,
+            0.021414030343294144,
+            -0.014394897036254406,
+            -0.02890326827764511,
+            -0.003379883011803031,
+            -0.0073960269801318645,
+            0.013446584343910217,
+            -0.0059208739548921585,
+            -0.008397023193538189,
+            -0.0022654130589216948,
+            0.008332181721925735,
+            -0.010026175528764725,
+            0.007622973527759314,
+            0.006082978565245867,
+            0.001210719463415444,
+            0.018593408167362213,
+            -0.0008186287013813853,
+            -0.013316900469362736,
+            -0.0033900144044309855,
+            -0.008542917668819427,
+            0.02083045430481434,
+            0.03355567157268524,
+            0.003979670349508524,
+            -0.007383869029581547,
+            -0.00466456264257431,
+            0.004648352041840553,
+            0.020684558898210526,
+            -0.015416156500577927,
+            -0.0041620382107794285,
+            -0.003440672066062689,
+            0.01307374332100153,
+            -0.011736379936337471,
+            0.027039064094424248,
+            -0.013235848397016525,
+            -0.023521391674876213,
+            -0.022030027583241463,
+            -0.0559261217713356,
+            0.006605766247957945,
+            -0.028400743380188942,
+            -0.03096199780702591,
+            -0.01581331342458725,
+            -0.012287535704672337,
+            -0.01760457083582878,
+            0.011849853210151196,
+            -0.005697980057448149,
+            -0.0032603307627141476,
+            0.005965452641248703,
+            0.013219637796282768,
+            -0.016380678862333298,
+            0.0008576351683586836,
+            0.00511035043746233,
+            0.0010156872449442744,
+            -0.01922561787068844,
+            0.016599521040916443,
+            0.009361546486616135,
+            -0.01223890483379364,
+            -0.013892372138798237,
+            0.0021661240607500076,
+            0.013187216594815254,
+            -0.004591615404933691,
+            0.006301820278167725,
+            0.015562050975859165,
+            0.010180175304412842,
+            -0.0214302409440279,
+            -0.0075540789403021336,
+            0.005065771751105785,
+            -0.007059659343212843,
+            0.01487310603260994,
+            -0.015667418017983437,
+            -0.006885396782308817,
+            0.0026321751065552235,
+            0.012076799757778645,
+            -0.018463725224137306,
+            -0.007343342527747154,
+            -0.00350348767824471,
+            3.1661072625865927e-06,
+            -0.018739303573966026,
+            0.0003079989110119641,
+            0.012919744476675987,
+            0.0008267339435406029,
+            0.008129550144076347,
+            -0.01317911222577095,
+            0.003142804838716984,
+            0.028384532779455185,
+            0.019760562106966972,
+            -0.037024714052677155,
+            0.0013272322248667479,
+            0.014362475834786892,
+            -0.009150810539722443,
+            -0.01989024691283703,
+            0.019063511863350868,
+            0.02298644557595253,
+            0.002540991175919771,
+            0.0017254018457606435,
+            -0.0023464655969291925,
+            -0.003270462155342102,
+            -0.01865825057029724,
+            0.011566170491278172,
+            0.01116090826690197,
+            -0.005843874532729387,
+            -0.0012887322809547186,
+            0.006366661749780178,
+            -0.01642931066453457,
+            -0.006326135713607073,
+            0.007789130788296461,
+            0.014856895431876183,
+            0.007489237003028393,
+            0.009499335661530495,
+            0.010561120696365833,
+            0.004575405269861221,
+            0.0015967312501743436,
+            0.024104967713356018,
+            0.07385490089654922,
+            0.0013819425366818905,
+            0.008380812592804432,
+            0.018852775916457176,
+            -0.006504450924694538,
+            -0.013187216594815254,
+            0.005860084667801857,
+            -0.008016076870262623,
+            -3.7233421608107165e-05,
+            0.005240034312009811,
+            -0.04072070121765137,
+            -0.02165718749165535,
+            -0.010877225548028946,
+            -0.013381742872297764,
+            0.004591615404933691,
+            0.018042253330349922,
+            -0.01799362152814865,
+            0.01590247079730034,
+            0.03793250024318695,
+            0.015148683451116085,
+            -0.009223757311701775,
+            0.01437058113515377,
+            0.0071893432177603245,
+            -0.011087961494922638,
+            -0.006646292749792337,
+            -0.01227132510393858,
+            0.008255181834101677,
+            0.015027105808258057,
+            0.0005825637490488589,
+            0.02741190418601036,
+            -0.0026200171560049057,
+            -0.00540213892236352,
+            0.003217778168618679,
+            -0.02491549216210842,
+            -0.015942996367812157,
+            0.004798299167305231,
+            0.008198444731533527,
+            -0.015083841979503632,
+            -0.013859951868653297,
+            -0.006881344132125378,
+            0.011087961494922638,
+            -0.0017872042953968048,
+            -0.006553082261234522,
+            -0.0013981530210003257,
+            -0.002670674817636609,
+            -0.004782088566571474,
+            -0.024023916572332382,
+            0.0024579125456511974,
+            0.013381742872297764,
+            0.006046505179256201,
+            0.014727211557328701,
+            0.003460935316979885,
+            0.007108291145414114,
+            -0.006739502772688866,
+            -0.012935955077409744,
+            -0.00822681374847889,
+            0.012563114054501057,
+            0.021365398541092873,
+            0.013414164073765278,
+            0.010553015395998955,
+            -0.005568296182900667,
+            0.007371711079031229,
+            -0.008113340474665165,
+            -0.015148683451116085,
+            -0.00663008214905858,
+            -0.00176896748598665,
+            0.0019006775692105293,
+            -0.004453826695680618,
+            0.002962463302537799,
+            -0.002729437779635191,
+            0.006982659921050072,
+            0.004842877853661776,
+            0.008559128269553185,
+            0.0007877275347709656,
+            -0.002368754940107465,
+            -0.012149747461080551,
+            -0.014719106256961823,
+            -0.019193196669220924,
+            0.017815306782722473,
+            0.010034280829131603,
+            0.013633005321025848,
+            0.018917618319392204,
+            -0.0024822282139211893,
+            -0.012749534100294113,
+            -0.0197119303047657,
+            0.00038297235732898116,
+            0.003321120049804449,
+            0.016056470572948456,
+            0.026228539645671844,
+            -0.01789635792374611,
+            0.00382972345687449,
+            -0.006168083753436804,
+            0.01179311703890562,
+            -0.022678447887301445,
+            -0.010293648578226566,
+            0.015878155827522278,
+            -0.030281158164143562,
+            -5.6609998864587396e-05,
+            -0.010723225772380829,
+            -0.00444572139531374,
+            0.018593408167362213,
+            0.023602444678544998,
+            0.017247939482331276,
+            0.012976480647921562,
+            -0.010471963323652744,
+            -0.005009035114198923,
+            0.01920940726995468,
+            0.008559128269553185,
+            -0.005795242730528116,
+            0.0002556948165874928,
+            -0.007732394151389599,
+            -0.02104119025170803,
+            0.005146824289113283,
+            0.00969386100769043,
+            -0.024737177416682243,
+            0.005122508388012648,
+            -0.017701832577586174,
+            -0.0039350916631519794,
+            -0.018431304022669792,
+            -0.00305972620844841,
+            0.00611945241689682,
+            0.00244372827000916,
+            0.0024457545951008797,
+            0.0004257781256455928,
+            -0.003122541820630431,
+            -0.0005324126104824245,
+            0.01284679677337408,
+            -0.007489237003028393,
+            -0.01972814090549946,
+            -0.027379482984542847,
+            -0.010844804346561432,
+            -0.0018165857763960958,
+            0.002555175218731165,
+            -0.007335237693041563,
+            -0.02254876308143139,
+            0.0029644896276295185,
+            -0.009896491654217243,
+            -0.000849529926199466,
+            -0.001954374834895134,
+            0.02630959264934063,
+            0.002792253391817212,
+            -0.015245947055518627,
+            -0.034301355481147766,
+            -0.03123757429420948,
+            0.011622906662523746,
+            -0.0008160958532243967,
+            0.004672667942941189,
+            -0.011631011962890625,
+            0.001885480247437954,
+            -0.016169942915439606,
+            0.01464615948498249,
+            -0.022532552480697632,
+            -0.008012024685740471,
+            0.009337230585515499,
+            -0.0003305416030343622,
+            0.001216798322275281,
+            0.000767970981542021,
+            -0.0074325003661215305,
+            0.003718276508152485,
+            -0.01728036068379879,
+            -0.009961334057152271,
+            -0.02087908424437046,
+            -0.00027431154740042984,
+            0.004352510906755924,
+            -0.013422268442809582,
+            0.019031090661883354,
+            0.01987403631210327,
+            0.0038884864188730717,
+            -0.012109220959246159,
+            0.017863936722278595,
+            0.01188227441161871,
+            -0.009547966532409191,
+            -0.014613738283514977,
+            -0.021786870434880257,
+            0.011266276240348816,
+            0.026325803250074387,
+            0.018415093421936035,
+            0.0075540789403021336,
+            0.01588626019656658,
+            0.012287535704672337,
+            0.016007838770747185,
+            0.001041522715240717,
+            0.0017831516452133656,
+            0.0009766807779669762,
+            0.011979537084698677,
+            -0.015910575166344643,
+            0.0035359086468815804,
+            0.0090535469353199,
+            0.00985596515238285,
+            -0.016072681173682213,
+            -0.01290353387594223,
+            0.024007705971598625,
+            -0.0010516542242839932,
+            -0.012028168886899948,
+            0.0028348059859126806,
+            0.00540213892236352,
+            -0.007177185267210007,
+            0.000767970981542021,
+            -0.005462928209453821,
+            -0.002014150843024254,
+            0.01590247079730034,
+            0.017766674980521202,
+            -0.009758702479302883,
+            -0.004401142708957195,
+            0.011995747685432434,
+            -0.0028773583471775055,
+            -0.03190220519900322,
+            0.006184294354170561,
+            0.028822215273976326,
+            0.006018136627972126,
+            -0.03790007904171944,
+            -0.0013454690342769027,
+            0.005219771061092615,
+            -0.032048098742961884,
+            -0.03449587896466255,
+            0.0021965187042951584,
+            -0.010771856643259525,
+            -0.015294577926397324,
+            -0.0035521192476153374,
+            0.0075905523262917995,
+            -0.028984319418668747,
+            0.009410177357494831,
+            0.014281423762440681,
+            -0.0006276491330936551,
+            0.022240763530135155,
+            -0.004944193176925182,
+            -0.012352378107607365,
+            -0.027266010642051697,
+            0.02237044833600521,
+            -0.013470900245010853,
+            -0.008461865596473217,
+            -0.0012380746193230152,
+            -0.009945123456418514,
+            0.005240034312009811,
+            0.003071883926168084,
+            0.02890326827764511,
+            0.018479935824871063,
+            -0.0029259896837174892,
+            -0.04201753810048103,
+            -0.022338027134537697,
+            0.003971565049141645,
+            -0.025045175105333328,
+            0.013389848172664642,
+            -0.006054610479623079,
+            -0.00321169919334352,
+            0.005969505291432142,
+            0.03164283558726311,
+            0.011274381540715694,
+            0.019355300813913345,
+            -3.3022501156665385e-05,
+            0.0042674061842262745,
+            -0.03384746238589287,
+            -0.02415359951555729,
+            0.00947501976042986,
+            -0.0024639912880957127,
+            0.032647885382175446,
+            0.0036270925775170326,
+            0.004619983956217766,
+            0.014475949108600616,
+            -0.006719239521771669,
+            -0.004054643679410219,
+            -0.018804144114255905,
+            0.011922799982130527,
+            -0.004761825315654278,
+            0.01932287961244583,
+            0.01625099591910839,
+            -0.014378686435520649,
+            -0.022354237735271454,
+            0.005223823711276054,
+            0.04986340552568436,
+            0.013803214766085148,
+            -0.0006757739465683699,
+            0.005839821882545948,
+            0.0021195188164711,
+            0.0021073610987514257,
+            0.004340353421866894,
+            0.004046538379043341,
+            0.015002789907157421,
+            0.026212329044938087,
+            -0.004745615180581808,
+            0.0036736975889652967,
+            0.00350348767824471,
+            -0.005098192486912012,
+            -0.0028307533357292414,
+            -0.013033217750489712,
+            0.004684825893491507,
+            -0.010925856418907642,
+            0.008125497959554195,
+            -0.02053866535425186,
+            -0.011995747685432434,
+            -0.011055540293455124,
+            -0.010609752498567104,
+            -0.010009964928030968,
+            -0.018787933513522148,
+            0.01927424781024456,
+            -0.029730001464486122,
+            0.002589622512459755,
+            -0.013227743096649647,
+            0.013689741492271423,
+            0.009620913304388523,
+            -0.0006119452300481498,
+            0.02353760227560997,
+            -0.023407917469739914,
+            -0.007935024797916412,
+            0.004465984180569649,
+            0.00266662216745317,
+            -0.0022654130589216948,
+            0.003124567912891507,
+            -0.013057533651590347,
+            -0.025612542405724525,
+            0.009547966532409191,
+            -0.00540213892236352,
+            0.0034305406734347343,
+            0.006747608073055744,
+            -0.0010405095526948571,
+            -0.005843874532729387,
+            -0.003625066252425313,
+            0.01826919987797737,
+            -0.020425191149115562,
+            0.04114217311143875,
+            -0.03018389455974102,
+            -0.00480235181748867,
+            0.020311718806624413,
+            -0.01723172888159752,
+            -0.0001227183238370344,
+            -0.0020648085046559572,
+            0.0073636057786643505,
+            0.0035014613531529903,
+            -0.0022188080474734306,
+            -0.014386791735887527,
+            0.03161041438579559,
+            3.1233648769557476e-05,
+            -0.016745414584875107,
+            0.00889144279062748,
+            -0.013308795168995857,
+            0.017734253779053688,
+            -0.028108954429626465,
+            -1.29652098621591e-05,
+            0.015221631154417992,
+            0.0016281390562653542,
+            0.009783018380403519,
+            -0.01121764536947012,
+            -0.010958277620375156,
+            -0.01260364055633545,
+            0.015797102823853493,
+            0.01933909021317959,
+            -0.011525643989443779,
+            -0.022840552031993866,
+            0.0040242490358650684,
+            0.01666436344385147,
+            -0.021867923438549042,
+            -0.0017132440116256475,
+            -0.004397090058773756,
+            -0.009531755931675434,
+            0.0003996893938165158,
+            -0.012587429955601692,
+            0.011801222339272499,
+            0.014467843808233738,
+            -0.0046686152927577496,
+            -0.03316662088036537,
+            0.008084971457719803,
+            0.010625963099300861,
+            -0.03169146925210953,
+            -0.013049428351223469,
+            0.012555008754134178,
+            -0.011898485012352467,
+            0.010180175304412842,
+            -0.006775976158678532,
+            0.00986407045274973,
+            -0.00928049348294735,
+            -0.001244153594598174,
+            -0.00031737060635350645,
+            -0.0007380829192698002,
+            -0.022856762632727623,
+            -0.008648285642266273,
+            -0.0009584440267644823,
+            -0.03079989179968834,
+            0.011695853434503078,
+            0.003937117755413055,
+            -0.009734386578202248,
+            -0.0006560174515470862,
+            -0.017085835337638855,
+            -0.004368721507489681,
+            0.0045105633325874805,
+            0.00685297604650259,
+            -0.011493222787976265,
+            -0.013187216594815254,
+            -0.002350518014281988,
+            -0.016007838770747185,
+            0.018236778676509857,
+            -0.0004701036086771637,
+            0.036635659635066986,
+            -0.0023829389829188585,
+            -0.0013900477206334472,
+            -0.0021418083924800158,
+            -0.00280441134236753,
+            0.005791190080344677,
+            0.03195083513855934,
+            -0.0013971398584544659,
+            0.006390977650880814,
+            0.0018054411048069596,
+            -0.003677750239148736,
+            0.006001926492899656,
+            0.01576468162238598,
+            0.0005633138353005052,
+            0.003975617699325085,
+            -0.009726282209157944,
+            -0.010893435217440128,
+            0.006942133419215679,
+            -0.002822648035362363,
+            0.02021445520222187,
+            -0.0014812316512688994,
+            -0.00932912528514862,
+            -0.009256178513169289,
+            0.0012887322809547186,
+            0.008153866045176983,
+            -0.005977610591799021,
+            -0.002030361210927367,
+            0.02243529073894024,
+            -0.005122508388012648,
+            0.014905527234077454,
+            -0.015424261800944805,
+            0.015805207192897797,
+            0.014394897036254406,
+            -0.013284480199217796,
+            0.00781344622373581,
+            0.018804144114255905,
+            -0.007667552214115858,
+            0.018706882372498512,
+            -0.016550889238715172,
+            0.007602710276842117,
+            -0.00983165018260479,
+            0.03884028643369675,
+            -0.009450703859329224,
+            -0.022013816982507706,
+            -0.0029766475781798363,
+            0.015383735299110413,
+            -0.02005235105752945,
+            0.010569225996732712,
+            0.013114269822835922,
+            -0.02536938525736332,
+            0.002944226609542966,
+            0.014751527458429337,
+            0.003997907042503357,
+            -0.004206616897135973,
+            0.006265346426516771,
+            0.005406191572546959,
+            0.009499335661530495,
+            0.005811453331261873,
+            0.020360350608825684,
+            -0.016048364341259003,
+            0.01972814090549946,
+            -0.012174062430858612,
+            -0.0032035941258072853,
+            -0.022889183834195137,
+            -0.0030131209641695023,
+            -0.009572282433509827,
+            0.011250065639615059,
+            -0.012230799533426762,
+            0.006828660145401955,
+            -0.010042386129498482,
+            -0.0001888266415335238,
+            0.030702630057930946,
+            0.018706882372498512,
+            0.008372707292437553,
+            0.003292751731351018,
+            0.01196332648396492,
+            0.03355567157268524,
+            0.012798165902495384,
+            -0.009742491878569126,
+            0.0025531488936394453,
+            0.023440338671207428,
+            0.00649634562432766,
+            0.0169561505317688,
+            0.008988705463707447,
+            3.9164748159237206e-05,
+            -0.013973425142467022,
+            0.011509433388710022,
+            0.033101778477430344,
+            0.024996543303132057,
+            0.025952961295843124,
+            0.005353507585823536,
+            0.0010577331995591521,
+            -0.01282248180359602,
+            -0.01343037374317646,
+            -0.016680574044585228,
+            -0.011736379936337471,
+            0.022678447887301445,
+            -0.009547966532409191,
+            0.002751727122813463,
+            -0.01872309297323227,
+            0.008632075041532516,
+            0.020019929856061935,
+            0.009507440030574799,
+            0.017653200775384903,
+            0.010228806175291538,
+            0.00451461598277092,
+            -0.012287535704672337,
+            0.0007613854832015932,
+            0.0060059791430830956,
+            -0.024964123964309692,
+            0.016558995470404625,
+            0.031756311655044556,
+            -0.0003996893938165158,
+            0.01539994589984417,
+            -0.0002955877862405032,
+            -0.003209673101082444,
+            -0.012190273031592369,
+            0.02392665296792984,
+            0.010625963099300861,
+            0.012449640780687332,
+            0.0023201233707368374,
+            -0.008713127113878727,
+            -0.011201434768736362,
+            -0.024218441918492317,
+            0.026568960398435593,
+            0.02092771604657173,
+            -0.00196551950648427,
+            0.00386417075060308,
+            -0.001570389256812632,
+            -0.012174062430858612,
+            0.014970368705689907,
+            0.015732260420918465,
+            0.019468773156404495,
+            -0.0028672267217189074,
+            0.012911639176309109,
+            -0.018139515072107315,
+            0.000802418275270611,
+            -0.012781955301761627,
+            0.01700478233397007,
+            0.004275511484593153,
+            -0.021462662145495415,
+            0.016696784645318985,
+            0.013770793564617634,
+            0.0005202547763474286,
+            -0.019955087453126907,
+            0.005531822796911001,
+            -0.0055196648463606834,
+            0.001069891033694148,
+            0.0007168066804297268,
+            -0.0012826534220948815,
+            -0.006747608073055744,
+            0.000229859389946796,
+            0.011703958734869957,
+            -0.0069097126834094524,
+            -0.022954024374485016,
+            -0.010447647422552109,
+            -0.001273535075597465,
+            -0.00656524021178484,
+            -0.020133404061198235,
+            0.015286472626030445,
+            0.004826667252928019,
+            -0.016907520592212677,
+            -0.020036140456795692,
+            -0.0004072880547028035,
+            0.008559128269553185,
+            0.03410682827234268,
+            -0.0284169539809227,
+            0.0032745148055255413,
+            -0.012352378107607365,
+            0.020798033103346825,
+            0.00357238226570189,
+            -0.023359287530183792,
+            0.022775709629058838,
+            0.0008277471060864627,
+            0.0056412434205412865,
+            -0.01179311703890562,
+            -0.0045835101045668125,
+            0.009766807779669762,
+            -0.022743288427591324,
+            0.005353507585823536,
+            0.01049627922475338,
+            -0.0038378287572413683,
+            0.008518601767718792,
+            9.124721691478044e-05,
+            0.005183297675102949,
+            0.0005379849462769926,
+            -0.009920807555317879,
+            0.018787933513522148,
+            -0.0014619816793128848,
+            -0.014605632983148098,
+            0.0034589089918881655,
+            0.013908582739531994,
+            -0.02564496360719204,
+            0.03595482185482979,
+            -0.0038256708066910505,
+            -0.014548896811902523,
+            -0.004433563444763422,
+            0.0010293647646903992,
+            0.019484983757138252,
+            0.030880944803357124,
+            -0.020911505445837975,
+            -0.010763752274215221,
+            0.005090087652206421,
+            0.01137974951416254,
+            -0.009815439581871033,
+            0.004206616897135973,
+            0.017572149634361267,
+            0.010553015395998955,
+            0.011209540069103241,
+            -0.010707015171647072,
+            -0.01595110259950161,
+            0.030427051708102226,
+            0.019922668114304543,
+            0.003683829214423895,
+            0.009701966308057308,
+            -0.002788200741633773,
+            0.020425191149115562,
+            0.0025531488936394453,
+            -0.01124196033924818,
+            0.00452677346765995,
+            0.016413100063800812,
+            -0.010763752274215221,
+            -0.0010749567300081253,
+            0.008247076533734798,
+            -0.011671538464725018,
+            0.008429444395005703,
+            -0.027395693585276604,
+            0.0263582244515419,
+            0.007906656712293625,
+            0.011955221183598042,
+            0.020133404061198235,
+            -0.01977677270770073,
+            -0.0011631011730059981,
+            0.014305739663541317,
+            0.012344272807240486,
+            -0.02353760227560997,
+            -0.008542917668819427,
+            0.008826600387692451,
+            -0.00020794992451556027,
+            -0.0008216681890189648,
+            -0.003635197877883911,
+            -0.013519532047212124,
+            -0.005000929813832045,
+            -0.04130427539348602,
+            0.011047434993088245,
+            -0.004786141216754913,
+            0.011922799982130527,
+            0.005985715892165899,
+            0.00911028403788805,
+            -0.02613127790391445,
+            0.003578461240977049,
+            -0.01000185962766409,
+            -0.015480998903512955,
+            0.005495349410921335,
+            0.006054610479623079,
+            -0.013162901625037193,
+            -0.0035642769653350115,
+            -0.0018358356319367886,
+            0.002589622512459755,
+            0.0021134400740265846,
+            -0.004790193866938353,
+            -0.029178844764828682,
+            -0.00823897123336792,
+            -0.013227743096649647,
+            0.006938081234693527,
+            -0.014840684831142426,
+            0.03052431344985962,
+            0.0005192416138015687,
+            -0.009207546710968018,
+            0.0036574872210621834,
+            -0.013859951868653297,
+            -0.00411948561668396,
+            -0.0038216181565076113,
+            0.01895003952085972,
+            -0.0038074341136962175,
+            0.028643900528550148,
+            0.00883470568805933,
+            -0.010958277620375156,
+            -0.018852775916457176,
+            0.003193462500348687,
+            -0.014119318686425686,
+            -0.0019422168843448162,
+            0.03133483603596687,
+            0.007720236200839281,
+            -0.008186287246644497,
+            0.013900477439165115,
+            -0.008996810764074326,
+            0.022078659385442734,
+            -0.01298458594828844,
+            -0.01027743797749281,
+            0.012887323275208473,
+            0.013738373294472694,
+            0.004384932108223438,
+            -1.1429647202021442e-05,
+            -0.005548033397644758,
+            0.00883470568805933,
+            -0.019922668114304543,
+            -0.001798348966985941,
+            0.012749534100294113,
+            -0.007079922594130039,
+            0.0036230399273335934,
+            0.017701832577586174,
+            0.013689741492271423,
+            -0.009766807779669762,
+            -0.00019199274538550526,
+            0.012076799757778645,
+            0.01583762839436531,
+            0.017020992934703827,
+            -0.01838267222046852,
+            0.0028874899726361036,
+            0.014751527458429337,
+            -0.02498033456504345,
+            -0.005803348030894995,
+            -0.012709008529782295,
+            0.0037000395823270082,
+            -0.036473557353019714,
+            0.00016045832308009267,
+            0.04493542015552521,
+            0.0035946716088801622,
+            0.0008135629468597472,
+            -0.029551686719059944,
+            -0.023359287530183792,
+            -0.005458875559270382,
+            -0.0011134566739201546,
+            -0.003997907042503357,
+            -0.019809193909168243,
+            -0.012263220734894276,
+            0.014702895656228065,
+            0.016631942242383957,
+            -0.018188146874308586,
+            -0.009620913304388523,
+            0.0013707978650927544,
+            -0.006192399188876152,
+            -0.005892505869269371,
+            0.0008166024344973266,
+            -0.004068827722221613,
+            -0.0010911672143265605,
+            0.01137974951416254,
+            0.015059526078402996,
+            0.007481131702661514,
+            0.016680574044585228,
+            0.011987642385065556,
+            -0.008048498071730137,
+            0.014176055788993835,
+            -0.003600750584155321,
+            -0.004555142018944025,
+            -0.004287669435143471,
+            -0.023570023477077484,
+            -0.0038540391251444817,
+            0.003408251330256462,
+            -0.014135529287159443,
+            0.008948178961873055,
+            0.00389861804433167,
+            0.003991828300058842,
+            -0.001961466856300831,
+            -0.010690804570913315,
+            0.0016960203647613525,
+            -0.02287297323346138,
+            -0.006504450924694538,
+            0.0027942797169089317,
+            -0.017815306782722473,
+            0.019306669011712074,
+            0.01922561787068844,
+            -0.0059532951563596725,
+            -0.016648152843117714,
+            0.014224686659872532,
+            0.001204640488140285,
+            0.015942996367812157,
+            0.015237841755151749,
+            0.000640313548501581,
+            0.0047739832662045956,
+            0.01204437855631113,
+            -0.0019857825245708227,
+            -0.021624766290187836,
+            -0.018301621079444885,
+            0.0053251395002007484,
+            -0.003165094181895256,
+            -0.018674461171030998,
+            0.0026443328242748976,
+            -0.0029118056409060955,
+            0.018155725672841072,
+            0.0022836499847471714,
+            0.006694924086332321,
+            -0.0030475682578980923,
+            -0.0218841340392828,
+            -0.0015795076033100486,
+            -0.004506510682404041,
+            -0.02853042632341385,
+            -0.00620455713942647,
+            -0.009888386353850365,
+            0.025466646999120712,
+            -0.021738240495324135,
+            0.015164894051849842,
+            -0.003852012800052762,
+            0.0042147221975028515,
+            0.02049003355205059,
+            0.008680706843733788,
+            0.002648385474458337,
+            0.012433430179953575,
+            -0.008591548539698124,
+            -0.00533324433490634,
+            0.01088532991707325,
+            -0.008461865596473217,
+            0.013495216146111488,
+            -0.024623703211545944,
+            0.01559447217732668,
+            -0.014694790355861187,
+            -0.005394033621996641,
+            0.010074807330965996,
+            0.013187216594815254,
+            0.0074325003661215305,
+            -0.01531078852713108,
+            -0.012376694008708,
+            0.022532552480697632,
+            -0.0077931834384799,
+            0.00331504107452929,
+            -0.003886460093781352,
+            -0.011250065639615059,
+            -0.01672920398414135,
+            -0.0006707081920467317,
+            0.013551952317357063,
+            -0.017020992934703827,
+            -0.00985596515238285,
+            -0.01595110259950161,
+            -0.0071690804325044155,
+            -0.04947435483336449,
+            -0.02369970642030239,
+            0.013025112450122833,
+            -0.009815439581871033,
+            -0.018706882372498512,
+            0.02614748664200306,
+            -0.006472030188888311,
+            -0.00591276865452528,
+            -0.008076866157352924,
+            -0.008283549919724464,
+            -0.014143634587526321,
+            0.01961466856300831,
+            0.010317963548004627,
+            -0.01365732029080391,
+            0.01922561787068844,
+            0.000424764963099733,
+            0.015051420778036118,
+            -0.012611744925379753,
+            -0.018204357475042343,
+            -0.0032542517874389887,
+            0.022743288427591324,
+            0.022030027583241463,
+            -0.009928912855684757,
+            0.0014619816793128848,
+            0.0021316767670214176,
+            0.018074674531817436,
+            0.01174448523670435,
+            -0.010739436373114586,
+            -0.010561120696365833,
+            0.013008901849389076,
+            -0.00985596515238285,
+            0.003858091775327921,
+            0.009661439806222916,
+            0.0042349849827587605,
+            0.002431570552289486,
+            -0.009037337265908718,
+            -0.001908782753162086,
+            0.006091083865612745,
+            0.0217706598341465,
+            -0.0007553065661340952,
+            -0.0061113471165299416,
+            -0.004867193289101124,
+            -0.009783018380403519,
+            -0.0025693594943732023,
+            -0.020862873643636703,
+            -0.013203427195549011,
+            -0.0023201233707368374,
+            0.018317829817533493,
+            0.004660509992390871,
+            -0.020798033103346825,
+            -0.02702285349369049,
+            -0.008988705463707447,
+            -0.023861810564994812,
+            -0.0055358754470944405,
+            -0.011250065639615059,
+            -0.008745548315346241,
+            0.008036340586841106,
+            0.009815439581871033,
+            -0.0017213491955772042,
+            -0.00199996680021286,
+            -0.006504450924694538,
+            0.005017140414565802,
+            -0.024947913363575935,
+            -0.0013272322248667479,
+            -0.0015896392287686467,
+            -0.021803081035614014,
+            -0.012733323499560356,
+            0.005916821304708719,
+            0.031059259548783302,
+            0.04224448278546333,
+            -0.008940073661506176,
+            0.0009270362206734717,
+            -0.0038216181565076113,
+            0.010674593970179558,
+            0.017847727984189987,
+            0.0090535469353199,
+            0.007019133307039738,
+            0.017345203086733818,
+            -0.03203188627958298,
+            0.0043200901709496975,
+            0.012392903678119183,
+            -0.0004067814734298736,
+            -0.007647288963198662,
+            -0.015489104203879833,
+            0.005000929813832045,
+            -0.008153866045176983,
+            0.005568296182900667,
+            0.012660376727581024,
+            -0.011039329692721367,
+            -0.00946691446006298,
+            0.0283034797757864,
+            0.009572282433509827,
+            0.011857958510518074,
+            0.01522973645478487,
+            -0.01407068781554699,
+            -0.022289395332336426,
+            0.0008231879328377545,
+            0.006549029611051083,
+            -0.0037446185015141964,
+            0.008575338870286942,
+            -0.010650279000401497,
+            -0.03595482185482979,
+            0.0002515155647415668,
+            -0.0006661489605903625,
+            -0.012336167506873608,
+            -0.016332048922777176,
+            0.0035055140033364296,
+            0.01932287961244583,
+            0.025904331356287003,
+            -0.01531889382749796,
+            0.017410043627023697,
+            0.007663499563932419,
+            -0.011720169335603714,
+            -0.009426387958228588,
+            -0.009572282433509827,
+            -0.015164894051849842,
+            -0.02037656120955944,
+            0.001938164234161377,
+            -0.014929842203855515,
+            -0.011833642609417439,
+            0.008234918117523193,
+            0.03263167664408684,
+            -0.0009467927156947553,
+            -0.004838825203478336,
+            -0.0011590485228225589,
+            0.0070839752443134785,
+            0.004518668632954359,
+            -0.010698909871280193,
+            0.01450837031006813,
+            0.019290458410978317,
+            0.010966382920742035,
+            0.001982743153348565,
+            -0.021235715597867966,
+            -0.008948178961873055,
+            0.017572149634361267,
+            -0.01223890483379364,
+            0.004575405269861221,
+            0.020198244601488113,
+            -0.005949242506176233,
+            -0.0069097126834094524,
+            0.0015339156379923224,
+            -0.007720236200839281,
+            -0.03268030658364296,
+            0.008315971121191978,
+            -0.007108291145414114,
+            0.011063645593822002,
+            0.0071528698317706585,
+            -0.019792983308434486,
+            0.003991828300058842,
+            -0.007233921904116869,
+            0.002139782067388296,
+            -0.0060789259150624275,
+            -0.0018024016171693802,
+            -0.00044198858086019754,
+            -0.01723172888159752,
+            0.024785807356238365,
+            -0.003612908534705639,
+            2.2194411940290593e-05,
+            0.0016372574027627707,
+            0.022532552480697632,
+            -0.0011428381549194455,
+            0.006334241013973951,
+            0.0033089620992541313,
+            -0.012725219130516052,
+            0.006654397584497929,
+            -0.009118389338254929,
+            -0.004393037408590317,
+            -0.00787423551082611,
+            0.012944060377776623,
+            0.013876162469387054,
+            0.02425086311995983,
+            0.00017109644250012934,
+            0.005527770146727562,
+            -0.019371511414647102,
+            -0.0035237509291619062,
+            -0.01553773507475853,
+            0.0031205154955387115,
+            0.010617857798933983,
+            0.0181070938706398,
+            -0.003770960494875908,
+            -0.004587562754750252,
+            -0.0010258187539875507,
+            0.009069757536053658,
+            0.016404995694756508,
+            -0.004064775072038174,
+            -0.022062448784708977,
+            0.0016382705653086305,
+            -0.004066801629960537,
+            0.009685755707323551,
+            0.022143501788377762,
+            -0.01634015329182148,
+            0.02032792940735817,
+            0.001971598481759429,
+            -0.015408051200211048,
+            0.013527637347579002,
+            -0.0074000791646540165,
+            -0.027233589440584183,
+            -0.020911505445837975,
+            0.0018753487383946776,
+            0.004879351239651442,
+            0.013033217750489712,
+            0.012433430179953575,
+            0.014840684831142426,
+            0.004725351929664612,
+            0.015610681846737862,
+            -0.0007117409259080887,
+            -0.0083078658208251,
+            -0.017085835337638855,
+            -0.0006367675377987325,
+            -0.007906656712293625,
+            -0.00822681374847889,
+            -0.038353972136974335,
+            -0.017426254227757454,
+            -0.007785078138113022,
+            0.02541801705956459,
+            0.013730267994105816,
+            0.005770927295088768,
+            -0.004571352619677782,
+            -0.01826919987797737,
+            -0.007290658541023731,
+            0.019484983757138252,
+            -0.0026564907748252153,
+            0.029308529570698738,
+            -0.006318030413240194,
+            0.00218841340392828,
+            0.003229936119168997,
+            -0.00817007664591074,
+            -0.020360350608825684,
+            0.022970234975218773,
+            0.01298458594828844,
+            0.007732394151389599,
+            0.014038266614079475,
+            -0.02010098285973072,
+            0.02404012717306614,
+            -0.014143634587526321,
+            -0.023164760321378708,
+            0.012668482027947903,
+            0.004534878768026829,
+            0.01961466856300831,
+            -0.009434493258595467,
+            0.005296770948916674,
+            0.015116263180971146,
+            -0.02287297323346138,
+            0.0263582244515419,
+            -0.005361612886190414,
+            0.023683495819568634,
+            -0.006390977650880814,
+            0.00852670706808567,
+            -0.015148683451116085,
+            0.005146824289113283,
+            -0.04454636946320534,
+            0.0020546771120280027,
+            -0.03024873696267605,
+            -8.561154390918091e-05,
+            0.017961200326681137,
+            -0.025612542405724525,
+            -0.0026504117995500565,
+            0.005211665760725737,
+            -0.011671538464725018,
+            -0.007456816267222166,
+            0.02254876308143139,
+            0.009783018380403519,
+            0.0008839771617203951,
+            -0.0003571369161363691,
+            0.011857958510518074,
+            -0.013770793564617634,
+            -0.024412967264652252,
+            -0.01838267222046852,
+            -0.017474886029958725,
+            -0.033750198781490326,
+            0.0003259824006818235,
+            0.00023669817892368883,
+            0.0045470367185771465,
+            -0.008915758691728115,
+            -0.004170143511146307,
+            -0.009710071608424187,
+            -0.0014579291455447674,
+            0.016048364341259003,
+            0.008198444731533527,
+            -0.014800158329308033,
+            -0.004506510682404041,
+            -0.006034347228705883,
+            -0.0016676520463079214,
+            0.00022276731033343822,
+            0.004769930616021156,
+            0.018852775916457176,
+            -0.011387854814529419,
+            -0.008591548539698124,
+            0.025612542405724525,
+            0.022954024374485016,
+            -0.016323942691087723,
+            0.00389861804433167,
+            -0.03751102834939957,
+            0.01865825057029724,
+            -0.02293781377375126,
+            -5.395046900957823e-05,
+            -0.012571219354867935,
+            0.022662237286567688,
+            -0.003939144313335419,
+            0.0035217246040701866,
+            0.013422268442809582,
+            -0.010034280829131603,
+            0.020895294845104218,
+            0.011922799982130527,
+            -0.0031326732132583857,
+            0.0026504117995500565,
+            -0.010674593970179558,
+            0.01356816291809082,
+            -0.02974621206521988,
+            0.00013918208424001932,
+            0.01893382892012596,
+            0.006285609677433968,
+            0.011314908042550087,
+            -0.015894364565610886,
+            0.004498405382037163,
+            0.0028814109973609447,
+            0.030491894111037254,
+            0.004680773243308067,
+            0.004103275015950203,
+            0.0035602243151515722,
+            0.0005020179669372737,
+            -0.029000530019402504,
+            0.009499335661530495,
+            0.007849919609725475,
+            -0.007699972949922085,
+            0.021576134487986565,
+            0.012133536860346794,
+            0.018366461619734764,
+            -0.006788134109228849,
+            -0.009742491878569126,
+            0.025661174207925797,
+            0.01744246482849121,
+            -0.006192399188876152,
+            0.0025146491825580597,
+            0.023683495819568634,
+            -0.0074000791646540165,
+            0.040234386920928955,
+            0.027655061334371567,
+            -0.009264283813536167,
+            -0.028611479327082634,
+            0.010836699046194553,
+            -0.01509194727987051,
+            -0.004131643567234278,
+            0.01922561787068844,
+            0.022678447887301445,
+            -0.011258170939981937,
+            0.01531889382749796,
+            0.015440472401678562,
+            0.012652271427214146,
+            0.01656709983944893,
+            -0.008996810764074326,
+            -0.0035338823217898607,
+            0.00047618255484849215,
+            -0.0013201400870457292,
+            -0.0002829233417287469,
+            -0.02608264610171318,
+            0.006151873152703047,
+            -0.0023748339153826237,
+            0.007193395867943764,
+            0.03007042035460472,
+            0.0073960269801318645,
+            -0.023359287530183792,
+            -0.0150108952075243,
+            -0.01998750865459442,
+            0.022662237286567688,
+            -0.022451501339673996,
+            -0.017345203086733818,
+            -0.00869691651314497,
+            -0.01260364055633545,
+            -0.032988306134939194,
+            0.009904596954584122,
+            0.00013740906433667988,
+            -0.005215718410909176,
+            0.0006104254862293601,
+            0.0013515478931367397,
+            -0.005491296760737896,
+            -0.0015268236165866256,
+            0.0016717046964913607,
+            0.008202497847378254,
+            0.010091017000377178,
+            0.016648152843117714,
+            -0.005365665536373854,
+            0.0166157316416502,
+            -0.012571219354867935,
+            -0.03738134354352951,
+            -0.0037000395823270082,
+            -0.003385961754247546,
+            0.017102045938372612,
+            0.0005400112713687122,
+            0.006480135023593903,
+            -0.019063511863350868,
+            0.003440672066062689,
+            0.00040146239916794,
+            0.005292718298733234,
+            0.01304132305085659,
+            -0.0013525610556825995,
+            -0.004474089480936527,
+            0.027071483433246613,
+            -0.004786141216754913,
+            0.007104238495230675,
+            -0.00986407045274973,
+            0.016648152843117714,
+            -0.016826467588543892,
+            -0.003088094526901841,
+            0.007027238607406616,
+            -0.010828593745827675,
+            -0.0022613604087382555,
+            -0.011509433388710022,
+            -0.017912568524479866,
+            -0.010212595574557781,
+            0.011395960114896297,
+            -0.020960137248039246,
+            -0.013106164522469044,
+            -0.0004678240220528096,
+            -0.007193395867943764,
+            0.008502391166985035,
+            0.01601594313979149,
+            0.005219771061092615,
+            -0.010763752274215221,
+            0.023229602724313736,
+            0.013284480199217796,
+            0.023732127621769905,
+            -0.006767870858311653,
+            -0.013989635743200779,
+            -0.004174196161329746,
+            -0.006301820278167725,
+            0.015983523800969124,
+            0.0070839752443134785,
+            0.005276507697999477,
+            -0.003495382610708475,
+            0.02160855568945408,
+            -0.009369651786983013,
+            0.0006732410402037203,
+            -0.002932068658992648,
+            0.010504384525120258,
+            -0.010974488221108913,
+            -0.009239967912435532,
+            0.015140579082071781,
+            0.026941800490021706,
+            0.0059006111696362495,
+            0.005673664156347513,
+            0.009021126665174961,
+            0.00020896308706142008,
+            0.009069757536053658,
+            0.0029158582910895348,
+            -0.006362609565258026,
+            -0.03465798497200012,
+            -0.00875365361571312,
+            -0.01005049142986536,
+            -0.019825404509902,
+            0.019582247361540794,
+            -0.0002821634989231825,
+            -0.009677650406956673,
+            -0.006638187449425459,
+            -0.00928049348294735,
+            0.004316037520766258,
+            0.0013900477206334472,
+            -0.000849529926199466,
+            -0.008429444395005703,
+            0.008003919385373592,
+            -0.023667285218834877,
+            -0.015027105808258057,
+            0.028708741068840027,
+            0.008915758691728115,
+            0.013292585499584675,
+            -0.008672601543366909,
+            -0.0021033084485679865,
+            0.006087031215429306,
+            -0.02442917786538601,
+            -0.00883470568805933,
+            0.06539303809404373,
+            -0.016242889687418938,
+            0.019695719704031944,
+            -0.024331914260983467,
+            -0.001989835174754262,
+            -0.002711201086640358,
+            0.018204357475042343,
+            0.021803081035614014,
+            -0.025515278801321983,
+            -0.0009123454801738262,
+            -0.02076561190187931,
+            0.0005531822680495679,
+            0.0067030293866992,
+            0.0026747274678200483,
+            -0.015124368481338024,
+            0.010350384749472141,
+            -0.03222641348838806,
+            -0.016794046387076378,
+            0.009726282209157944,
+            -0.0075378683395683765,
+            0.003035410540178418,
+            -0.016161838546395302,
+            -0.005811453331261873,
+            0.007541920989751816,
+            0.010690804570913315,
+            -0.006224820390343666,
+            -0.027849586680531502,
+            -0.02575843594968319,
+            -0.009013021364808083,
+            -0.02026308700442314,
+            0.017701832577586174,
+            -0.015780892223119736,
+            0.0166157316416502,
+            -0.009491230361163616,
+            -0.003108357544988394,
+            0.006982659921050072,
+            0.01689130999147892,
+            0.012538798153400421,
+            0.0017274281708523631,
+            -0.005787137895822525,
+            -0.008291655220091343,
+            0.01700478233397007,
+            -0.013633005321025848,
+            -0.0359872430562973,
+            0.0003687881981022656,
+            -0.005191402975469828,
+            0.0020789927802979946,
+            0.008769864216446877,
+            0.018285410478711128,
+            0.01590247079730034,
+            0.01531889382749796,
+            -0.015359420329332352,
+            -0.01379510946571827,
+            0.009118389338254929,
+            0.031205154955387115,
+            -0.001978690503165126,
+            0.03637629374861717,
+            0.011874169111251831,
+            0.004964456427842379,
+            -0.003852012800052762,
+            -0.008380812592804432,
+            -0.018285410478711128,
+            -0.003509566653519869,
+            0.02049003355205059,
+            -0.0155863668769598,
+            0.02381318062543869,
+            0.0010394963901489973,
+            -0.010609752498567104,
+            -0.001762888627126813,
+            -0.0011357460170984268,
+            -0.018285410478711128,
+            0.013503321446478367,
+            -0.01599973440170288,
+            0.0023768600076436996,
+            -0.016145627945661545,
+            0.001996927196159959,
+            -0.004855035804212093,
+            0.01933909021317959,
+            0.0019766641780734062,
+            -0.007371711079031229,
+            0.005357560236006975,
+            -0.01202006358653307,
+            0.002869253046810627,
+            -3.042629214178305e-05,
+            0.01638878509402275,
+            -0.0003361139679327607,
+            0.0010253122309222817,
+            -0.02026308700442314,
+            0.008259234018623829,
+            0.0022147553972899914,
+            -0.015562050975859165,
+            -0.01760457083582878,
+            0.007991761900484562,
+            0.002368754940107465,
+            0.002439675619825721,
+            0.006115399766713381,
+            0.003961433656513691,
+            -0.004393037408590317,
+            0.004676720593124628,
+            0.007464921101927757,
+            0.005389980971813202,
+            0.005081982351839542,
+            -0.013235848397016525,
+            0.002636227523908019,
+            0.0018642040668055415,
+            0.004311984870582819,
+            0.01771804317831993,
+            -0.0001176525474875234,
+            -0.004206616897135973,
+            0.00875365361571312,
+            0.009442598558962345,
+            0.008534812368452549,
+            -0.026682432740926743,
+            -0.018188146874308586,
+            0.012717113830149174,
+            -0.001926006400026381,
+            0.006168083753436804,
+            0.00018452072981745005,
+            0.018690671771764755,
+            0.005645296070724726,
+            0.004328195471316576,
+            0.006974554620683193,
+            0.008255181834101677,
+            -0.005134666338562965,
+            -0.0018571119289845228,
+            0.013511426746845245,
+            -0.012417219579219818,
+            0.010228806175291538,
+            -0.02901674062013626,
+            0.010326068848371506,
+            0.009167020209133625,
+            -0.0023647022899240255,
+            0.0014001793460920453,
+            -0.027849586680531502,
+            -0.005803348030894995,
+            -0.01572415605187416,
+            0.0059532951563596725,
+            -0.012068694457411766,
+            0.017572149634361267,
+            0.01783151738345623,
+            -0.018123304471373558,
+            0.04282806068658829,
+            0.0035602243151515722,
+            0.02674727514386177,
+            -0.009094073437154293,
+            -0.003929012455046177,
+            -0.00244372827000916,
+            0.008923863992094994,
+            -0.01063406839966774,
+            0.012571219354867935,
+            0.006804344709962606,
+            0.02065213769674301,
+            -0.00605055782943964,
+            0.011809326708316803,
+            0.00012461798905860633,
+            0.027849586680531502,
+            -0.006662502884864807,
+            0.013543847016990185,
+            -0.011663433164358139,
+            -0.024412967264652252,
+            -0.010366595350205898,
+            0.0037912235129624605,
+            0.006557134911417961,
+            0.005150876939296722,
+            0.008109287358820438,
+            0.001973624574020505,
+            0.009807334281504154,
+            0.007837762124836445,
+            0.04039648920297623,
+            -0.012700903229415417,
+            -0.010488173924386501,
+            0.011371644213795662,
+            -0.0032542517874389887,
+            0.0025572015438228846,
+            0.005037403665482998,
+            0.011120381765067577,
+            0.01143648661673069,
+            7.332704990403727e-05,
+            -0.014127423986792564,
+            -0.009150810539722443,
+            -0.009791123680770397,
+            0.002575438469648361,
+            -0.017134467139840126,
+            0.007254185155034065,
+            -0.001919927541166544,
+            -0.00815791916102171,
+            0.006727344822138548,
+            -0.006881344132125378,
+            0.0044781421311199665,
+            0.012417219579219818,
+            0.01223890483379364,
+            0.00765539426356554,
+            -0.014240897260606289,
+            0.00401614373549819,
+            0.004453826695680618,
+            0.022662237286567688,
+            -0.004830719903111458,
+            0.017766674980521202,
+            -0.020425191149115562,
+            -0.0043565635569393635,
+            0.010026175528764725,
+            -0.012660376727581024,
+            -0.012749534100294113,
+            0.0057790325954556465
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "3080f374-964f-4b6f-bb07-d9220ba99e16",
+        "properties": {
+          "page_content": "7. Deepseekmoe 145B Ongoing\n\nEncouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available.\n\n7.1. Experimental Setup\n\nTraining Data and Tokenization. For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study.\n\nModel Settings. For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps.\n\nSubsequently, the learning rate keeps constant during the remaining training process. The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. Also, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse.\n\nEvaluation Benchmarks. We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3).\n\n7.2. Evaluations\n\nBaselines. Apart from DeepSeekMoE 145B, we consider three additional models for comparison. DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters.\n\nResults. From the evaluation results presented in Table 6, we have the following observations: (1) Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. (2) On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. (3) At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5.\n\n",
+          "keyphrases": [
+            "DeepSeekMoE 145B",
+            "GShard architecture",
+            "Transformer layers",
+            "multi-head attention mechanism",
+            "expert parallelism"
+          ],
+          "title": "Deepseekmoe 145B Ongoing",
+          "embedding": [
+            0.011145060881972313,
+            0.0028371023945510387,
+            -0.01864549145102501,
+            -0.008063549175858498,
+            0.004066969733685255,
+            0.007203228771686554,
+            0.0034158634953200817,
+            0.061411239206790924,
+            -0.013921549543738365,
+            0.03988758474588394,
+            0.002668948844075203,
+            -0.009103755466639996,
+            0.020194068551063538,
+            -0.01502432394772768,
+            -0.007441772148013115,
+            0.021367233246564865,
+            0.03610217571258545,
+            0.012208729982376099,
+            0.007637299597263336,
+            -0.028171585872769356,
+            -0.0256375502794981,
+            -0.05227620154619217,
+            -0.012240014038980007,
+            -0.0378853864967823,
+            0.011450083926320076,
+            0.01580643467605114,
+            0.029986079782247543,
+            0.0068551902659237385,
+            -0.014132719486951828,
+            0.018520353361964226,
+            -0.006624467670917511,
+            0.03178493306040764,
+            -0.013851160183548927,
+            -0.0013999760849401355,
+            0.0002678725286386907,
+            -0.01470365934073925,
+            0.05349629372358322,
+            0.010644510388374329,
+            -0.018989618867635727,
+            0.0074926093220710754,
+            0.04495565593242645,
+            0.005001590587198734,
+            -0.02932910807430744,
+            0.016236593946814537,
+            0.029907869175076485,
+            -0.009471346624195576,
+            -0.025668835267424583,
+            6.886230403324589e-05,
+            -0.0020589034538716078,
+            -0.024151543155312538,
+            -0.01281095389276743,
+            -0.013452284038066864,
+            0.004950753413140774,
+            -0.008759627118706703,
+            -0.03494465351104736,
+            -0.008282540366053581,
+            0.038104377686977386,
+            0.06757426261901855,
+            0.05612418055534363,
+            -0.026216311380267143,
+            0.031925711780786514,
+            -0.015743864700198174,
+            -0.030079932883381844,
+            0.011356230825185776,
+            0.0009101799805648625,
+            -0.01856728084385395,
+            -0.017018703743815422,
+            0.0004423806967679411,
+            -0.00746132479980588,
+            0.02050691284239292,
+            -0.046363454312086105,
+            0.026998421177268028,
+            -0.016596363857388496,
+            0.01866113394498825,
+            0.030392777174711227,
+            -0.00615520216524601,
+            0.03419382870197296,
+            0.007711600046604872,
+            -0.0013344744220376015,
+            -0.026623008772730827,
+            0.02580961398780346,
+            0.04971088096499443,
+            -0.015008682385087013,
+            0.010143960826098919,
+            0.06569720059633255,
+            -0.01866113394498825,
+            -0.027326907962560654,
+            -0.06444582343101501,
+            0.011090313084423542,
+            -0.012138339690864086,
+            -0.02261860854923725,
+            -0.0006212882581166923,
+            0.012756206095218658,
+            -0.022383974865078926,
+            -0.00839985627681017,
+            0.0512751005589962,
+            -0.031206170096993446,
+            -0.009651231579482555,
+            -0.017691317945718765,
+            0.019365031272172928,
+            0.008118296973407269,
+            -0.015853360295295715,
+            0.015962855890393257,
+            -0.0033376524224877357,
+            0.01507125049829483,
+            -0.005983137991279364,
+            -0.030721262097358704,
+            0.032942455261945724,
+            -0.006495419889688492,
+            0.08321645110845566,
+            -0.020741544663906097,
+            -0.001445924979634583,
+            -0.0033435183577239513,
+            0.01900526136159897,
+            -0.008524994365870953,
+            -0.007121107541024685,
+            -0.03472566232085228,
+            -0.01020652987062931,
+            0.021836498752236366,
+            -0.021429801359772682,
+            0.05143152177333832,
+            -0.00858756247907877,
+            0.016690216958522797,
+            -0.036352451890707016,
+            -0.054278403520584106,
+            0.050023727118968964,
+            0.008141760714352131,
+            0.002275938866659999,
+            0.000169253398780711,
+            -0.0024245395325124264,
+            -0.016393015161156654,
+            0.0008818284841254354,
+            -0.03838593512773514,
+            -0.00615520216524601,
+            0.003464745357632637,
+            0.013491389341652393,
+            0.06172408536076546,
+            0.062068212777376175,
+            -0.01295173354446888,
+            0.01028474047780037,
+            -0.019724803045392036,
+            -0.020991820842027664,
+            0.019803013652563095,
+            -0.002023708540946245,
+            -0.01618966832756996,
+            0.011457905173301697,
+            0.0026865461841225624,
+            0.0422339141368866,
+            -0.011700358241796494,
+            0.003171454183757305,
+            -0.00040083113708533347,
+            0.02399512007832527,
+            0.01683099754154682,
+            0.04063841328024864,
+            -0.00606525968760252,
+            0.00877526868134737,
+            -0.019224252551794052,
+            -0.006995969917625189,
+            -0.052057210355997086,
+            0.015915928408503532,
+            -0.003816694486886263,
+            -0.008337287232279778,
+            -0.01634608954191208,
+            0.002739338669925928,
+            -0.004301602486521006,
+            0.035977039486169815,
+            0.02426103875041008,
+            0.012505931779742241,
+            -0.0752076506614685,
+            0.023228652775287628,
+            0.020553838461637497,
+            0.017347188666462898,
+            -0.048177946358919144,
+            0.03757254034280777,
+            0.0016443852800875902,
+            0.015681296586990356,
+            0.008329466916620731,
+            0.0012396436650305986,
+            -0.044236116111278534,
+            0.04088868573307991,
+            -0.009948433376848698,
+            0.006557988468557596,
+            0.06360115110874176,
+            0.009455704130232334,
+            0.021648792549967766,
+            0.0022270570043474436,
+            0.052057210355997086,
+            0.012826596386730671,
+            -0.0038694869726896286,
+            0.033755846321582794,
+            -0.0005205916822887957,
+            0.005908837541937828,
+            0.028234153985977173,
+            0.0011770748533308506,
+            0.009181966073811054,
+            -0.02388562448322773,
+            0.007046807091683149,
+            0.014625448733568192,
+            -0.00258291675709188,
+            -0.018535995855927467,
+            -0.01011267676949501,
+            -0.031378235667943954,
+            -0.01828572154045105,
+            0.009486989118158817,
+            0.0248867254704237,
+            0.00891604833304882,
+            0.003511671908199787,
+            0.029157044366002083,
+            0.024511313065886497,
+            -0.017613107338547707,
+            0.009033365175127983,
+            -0.0015779059613123536,
+            -0.02821851149201393,
+            -0.048553358763456345,
+            0.00512672821059823,
+            0.022290121763944626,
+            -0.010988638736307621,
+            0.015423200093209743,
+            0.019505811855196953,
+            -0.015321525745093822,
+            -0.020194068551063538,
+            0.016142740845680237,
+            0.010542836971580982,
+            -0.02518392726778984,
+            -0.033411718904972076,
+            -0.014359530992805958,
+            0.014758407138288021,
+            -0.003525358624756336,
+            0.05446610972285271,
+            -0.025574982166290283,
+            -0.006084812339395285,
+            0.026795072481036186,
+            0.00516583351418376,
+            0.026138100773096085,
+            0.023197369650006294,
+            -0.04229648411273956,
+            0.024495670571923256,
+            0.0007200295804068446,
+            0.002708054380491376,
+            0.009463525377213955,
+            0.003918368835002184,
+            -0.01892705075442791,
+            -0.03281731531023979,
+            -0.015368452295660973,
+            0.02729562297463417,
+            -0.03203520551323891,
+            -0.025105716660618782,
+            0.02580961398780346,
+            0.010190887376666069,
+            0.014633269049227238,
+            0.006882563699036837,
+            -0.03963731229305267,
+            -0.007930590771138668,
+            0.002023708540946245,
+            -0.007789811119437218,
+            0.05969059839844704,
+            0.04886620491743088,
+            0.0029661504086107016,
+            -0.01457070093601942,
+            -0.030580483376979828,
+            0.00900990143418312,
+            -0.016095813363790512,
+            -0.03538263589143753,
+            0.029454244300723076,
+            -0.026341449469327927,
+            0.04339143633842468,
+            0.025668835267424583,
+            0.017049988731741905,
+            -0.017706960439682007,
+            -0.013577421195805073,
+            -0.012983018532395363,
+            -0.03191006928682327,
+            -0.03363071009516716,
+            0.008008801378309727,
+            0.001456678961403668,
+            -0.022947093471884727,
+            -0.011410977691411972,
+            -0.018332647159695625,
+            0.012099234387278557,
+            0.03813566267490387,
+            0.024667734280228615,
+            0.009948433376848698,
+            0.004305513110011816,
+            0.024307964369654655,
+            -0.035883184522390366,
+            -0.0030443614814430475,
+            -0.0018663088558241725,
+            -0.009846759028732777,
+            -0.015032145194709301,
+            -0.017534896731376648,
+            0.001207381603308022,
+            -0.011551758274435997,
+            -0.035069793462753296,
+            0.008712700568139553,
+            -0.0028762079309672117,
+            0.010229992680251598,
+            -0.012709279544651508,
+            0.015274599194526672,
+            0.022743744775652885,
+            -0.023823056370019913,
+            -0.049022626131772995,
+            0.014437741599977016,
+            0.020710259675979614,
+            -0.02784309908747673,
+            -0.015751685947179794,
+            -0.009862401522696018,
+            -0.0019249670440331101,
+            0.02407333068549633,
+            -0.04539363831281662,
+            -0.021257737651467323,
+            0.02471466176211834,
+            -0.019646592438220978,
+            -0.011465725488960743,
+            -0.026310164481401443,
+            -0.005298791918903589,
+            0.04192107170820236,
+            0.033755846321582794,
+            0.03907419368624687,
+            -0.06350729614496231,
+            0.002970061032101512,
+            0.019521454349160194,
+            0.03366199508309364,
+            -0.058752067387104034,
+            0.027702320367097855,
+            -0.00732445577159524,
+            -0.011583042331039906,
+            0.021914709359407425,
+            0.043766848742961884,
+            -0.023948194459080696,
+            -0.0025340348947793245,
+            0.017941592261195183,
+            -0.03485080227255821,
+            -0.02701406367123127,
+            -0.0019709160551428795,
+            0.005580351687967777,
+            -0.044236116111278534,
+            -0.03519492968916893,
+            0.00030037894612178206,
+            0.014343888498842716,
+            0.0040161325596272945,
+            -0.008650131523609161,
+            -0.03538263589143753,
+            -0.050899688154459,
+            0.02390126697719097,
+            0.0010548701975494623,
+            -0.004340707790106535,
+            -0.04160822555422783,
+            -0.014828796498477459,
+            -0.0064132981933653355,
+            0.01023781392723322,
+            0.0010734453098848462,
+            0.023275580257177353,
+            0.013155082240700722,
+            0.0010812664404511452,
+            -0.00877526868134737,
+            -0.04132666811347008,
+            -0.032754749059677124,
+            -0.01065233163535595,
+            0.007742884568870068,
+            0.029907869175076485,
+            0.024229753762483597,
+            -0.048021525144577026,
+            -0.018989618867635727,
+            0.03123745508491993,
+            -0.03869878128170967,
+            -0.014547237195074558,
+            0.04045070707798004,
+            0.018520353361964226,
+            0.0262006688863039,
+            0.00544739281758666,
+            0.00310106435790658,
+            0.043891988694667816,
+            0.0001796407887013629,
+            0.006757426541298628,
+            0.021523654460906982,
+            0.0043133338913321495,
+            0.01830136403441429,
+            -0.0014752540737390518,
+            -0.0040943436324596405,
+            0.0452372170984745,
+            -0.0049585746601223946,
+            0.03050227276980877,
+            -0.000946841377299279,
+            -0.020084572955965996,
+            0.020929250866174698,
+            0.025309065356850624,
+            0.00022534531308338046,
+            0.009064649231731892,
+            -0.011833317577838898,
+            0.0026200669817626476,
+            -0.004583161789923906,
+            -0.03566419333219528,
+            -0.013913728296756744,
+            -0.011794212274253368,
+            0.013561779633164406,
+            0.02444874495267868,
+            -0.0017939637182280421,
+            -0.015196388587355614,
+            -0.034318964928388596,
+            -0.0126779954880476,
+            -0.007817184552550316,
+            0.04564391076564789,
+            -0.004066969733685255,
+            0.012803133577108383,
+            -0.008978617377579212,
+            0.02424539625644684,
+            0.01015178207308054,
+            -0.003208604408428073,
+            0.03181621432304382,
+            -0.02047562785446644,
+            0.008110475726425648,
+            -0.020632049068808556,
+            -0.026716861873865128,
+            -0.021007461473345757,
+            -0.0005797386984340847,
+            0.03747868910431862,
+            0.023213010281324387,
+            0.009494809433817863,
+            -0.048303086310625076,
+            0.02471466176211834,
+            -0.047646112740039825,
+            0.01894269324839115,
+            0.01892705075442791,
+            -0.00530661316588521,
+            -0.013381893746554852,
+            0.008462425321340561,
+            -0.016127098351716995,
+            0.006890384946018457,
+            0.01928682066500187,
+            -0.025121359154582024,
+            0.00408652238547802,
+            0.000701943296007812,
+            0.03628988191485405,
+            -0.014719301834702492,
+            -0.04070097953081131,
+            -0.012107055634260178,
+            0.003024808596819639,
+            0.029188327491283417,
+            0.009635589085519314,
+            -0.026388375088572502,
+            0.0011487234150990844,
+            0.002221191069111228,
+            -0.009135039523243904,
+            -0.019599664956331253,
+            -0.026044247671961784,
+            -0.009995359927415848,
+            -0.054528675973415375,
+            0.010276919230818748,
+            -0.016220951452851295,
+            -0.011082491837441921,
+            -0.05396555736660957,
+            0.03325529769062996,
+            -0.06601004302501678,
+            -0.01033166702836752,
+            0.04949188977479935,
+            -0.004141270183026791,
+            -0.006733963266015053,
+            -0.007735063321888447,
+            -0.014351709745824337,
+            -0.039293184876441956,
+            0.022649891674518585,
+            0.0009991449769586325,
+            0.04877234995365143,
+            -0.02178957127034664,
+            -0.003578151110559702,
+            0.032473187893629074,
+            -0.018348289653658867,
+            -0.016009781509637833,
+            0.0018839063122868538,
+            0.028046447783708572,
+            -0.016877923160791397,
+            0.025496771559119225,
+            0.0018457785481587052,
+            0.018864482641220093,
+            -0.001967005431652069,
+            0.025293422862887383,
+            -0.0033298314083367586,
+            -0.026591723784804344,
+            -0.0008060616673901677,
+            0.04273446649312973,
+            0.018066730350255966,
+            -0.023228652775287628,
+            -0.018082372844219208,
+            -0.022915810346603394,
+            -0.02546548657119274,
+            -0.00638201367110014,
+            -0.008861301466822624,
+            -0.0019112802110612392,
+            0.0251682847738266,
+            0.030705619603395462,
+            -0.011066850274801254,
+            0.007918858900666237,
+            0.00011420021473895758,
+            0.022853240370750427,
+            0.0001724307076074183,
+            -0.017315905541181564,
+            -0.009666874073445797,
+            -0.007879753597080708,
+            0.016971776261925697,
+            0.0254498440772295,
+            -0.039199329912662506,
+            -0.021727003157138824,
+            0.017753886058926582,
+            -0.012153982184827328,
+            0.0022505202796310186,
+            0.02277502976357937,
+            0.011684716679155827,
+            0.02279067225754261,
+            -0.019177325069904327,
+            -0.01700306124985218,
+            0.024323606863617897,
+            -0.018129298463463783,
+            0.00502896448597312,
+            -0.019224252551794052,
+            -0.017128199338912964,
+            0.023744845762848854,
+            -0.004293781239539385,
+            0.0253559909760952,
+            0.01231822557747364,
+            0.016299162060022354,
+            -0.016408657655119896,
+            0.006925580091774464,
+            -0.015344989486038685,
+            0.029172684997320175,
+            -0.04179593548178673,
+            0.014085792936384678,
+            0.029626309871673584,
+            -0.00041891742148436606,
+            0.049585744738578796,
+            -0.043766848742961884,
+            -0.012513753026723862,
+            0.0074339513666927814,
+            0.019161684438586235,
+            -0.012419898994266987,
+            0.0007356717833317816,
+            -0.006999880541115999,
+            0.007672494743019342,
+            -0.00431724451482296,
+            -0.03679043427109718,
+            -0.004954664036631584,
+            0.01919296756386757,
+            -0.008376393467187881,
+            -0.0028586103580892086,
+            -0.0051971180364489555,
+            -0.007871932350099087,
+            0.002614201046526432,
+            0.05578005313873291,
+            -0.04630088433623314,
+            0.03391227126121521,
+            0.00905682798475027,
+            0.013186367228627205,
+            -0.031206170096993446,
+            -0.011246735230088234,
+            -0.003703288733959198,
+            0.00615911278873682,
+            -0.028797272592782974,
+            0.011614326387643814,
+            0.01033948827534914,
+            0.00864231027662754,
+            -0.01892705075442791,
+            -0.03253575786948204,
+            0.006624467670917511,
+            -0.028969338163733482,
+            0.0020784561056643724,
+            -0.0056742047891020775,
+            -0.007390934973955154,
+            -0.0034354161471128464,
+            0.025011863559484482,
+            -0.010667974129319191,
+            0.0014820975484326482,
+            -0.004305513110011816,
+            0.008955154567956924,
+            -0.00822779256850481,
+            -0.002438226481899619,
+            0.00794623326510191,
+            0.005959674715995789,
+            0.016862282529473305,
+            -0.007109375670552254,
+            -0.018129298463463783,
+            -0.004399366211146116,
+            0.01466455403715372,
+            -0.022540396079421043,
+            0.0036817805375903845,
+            0.03391227126121521,
+            0.0363837368786335,
+            0.0018242704682052135,
+            -0.010441162623465061,
+            -0.04254676029086113,
+            0.03409997746348381,
+            -0.013765127398073673,
+            0.01883319765329361,
+            0.027514614164829254,
+            -0.009776369668543339,
+            -0.04201492294669151,
+            -0.0014908963348716497,
+            -0.01442210003733635,
+            -0.011199808679521084,
+            0.017034346237778664,
+            -0.00836857222020626,
+            -0.02435489185154438,
+            -0.03162850812077522,
+            0.026341449469327927,
+            0.02727998048067093,
+            -0.00629207119345665,
+            -0.00733227701857686,
+            0.013452284038066864,
+            0.022118058055639267,
+            0.017112556844949722,
+            -0.016205308958888054,
+            0.01300648134201765,
+            -0.01231040433049202,
+            0.014304783195257187,
+            0.007840648293495178,
+            -0.009971896186470985,
+            0.004379813559353352,
+            -0.0043915449641644955,
+            0.006726142019033432,
+            -0.0004186730075161904,
+            -0.012232192791998386,
+            -0.01474276464432478,
+            0.03510107472538948,
+            0.004325065761804581,
+            0.006601004395633936,
+            0.005240133963525295,
+            0.028093375265598297,
+            0.035977039486169815,
+            -0.023635350167751312,
+            -0.0032281570602208376,
+            0.01866113394498825,
+            0.009189787320792675,
+            -0.03209777548909187,
+            -0.0107931112870574,
+            -0.020428700372576714,
+            -0.03014250099658966,
+            0.006796531844884157,
+            -0.007692047394812107,
+            0.013264577835798264,
+            0.009619947522878647,
+            0.026263238862156868,
+            -0.004149090964347124,
+            -0.0174410417675972,
+            0.03076818957924843,
+            -0.002999390009790659,
+            0.03907419368624687,
+            -0.003232067683711648,
+            -0.0027647572569549084,
+            -0.01093389093875885,
+            0.020147141069173813,
+            0.04066969454288483,
+            0.009041186422109604,
+            0.0033884895965456963,
+            0.011403157375752926,
+            0.022947093471884727,
+            0.044611528515815735,
+            -0.03632116690278053,
+            -0.016330447047948837,
+            -0.0011741418857127428,
+            -0.018410857766866684,
+            0.029172684997320175,
+            0.009158502332866192,
+            0.015032145194709301,
+            0.00863448902964592,
+            0.001260173972696066,
+            -0.0147505858913064,
+            0.03503850847482681,
+            -0.008783089928328991,
+            -0.015462305396795273,
+            0.02987658418715,
+            -0.02030356414616108,
+            0.03325529769062996,
+            0.009682515636086464,
+            -0.024558238685131073,
+            -0.024386174976825714,
+            0.0011389469727873802,
+            -0.0038851292338222265,
+            -0.027983879670500755,
+            0.031096674501895905,
+            0.029250897467136383,
+            -0.0049155582673847675,
+            0.006808263715356588,
+            0.009095934219658375,
+            0.02114824205636978,
+            -0.020240994170308113,
+            -0.016518153250217438,
+            0.017128199338912964,
+            -0.007171944249421358,
+            -0.022196268662810326,
+            0.03976244851946831,
+            -0.01856728084385395,
+            -0.011723821982741356,
+            -0.012302583083510399,
+            0.0032692179083824158,
+            0.011465725488960743,
+            0.017800813540816307,
+            0.03494465351104736,
+            -0.001723573892377317,
+            -0.028703419491648674,
+            0.021617507562041283,
+            0.026435302570462227,
+            -0.013170724734663963,
+            -0.019349390640854836,
+            0.025387275964021683,
+            -0.01855163834989071,
+            0.013358430936932564,
+            0.05161922797560692,
+            0.008251255378127098,
+            0.008955154567956924,
+            -0.024745944887399673,
+            -0.02178957127034664,
+            0.025387275964021683,
+            -0.017300263047218323,
+            0.04307859390974045,
+            0.013131619431078434,
+            0.02379177138209343,
+            0.007864111103117466,
+            -0.023040946573019028,
+            0.0011037520598620176,
+            0.0037756338715553284,
+            0.004927290137857199,
+            0.02369791828095913,
+            0.016846640035510063,
+            0.0032203360460698605,
+            0.005208849441260099,
+            -0.04514336213469505,
+            -0.014117076992988586,
+            -0.014062329195439816,
+            0.009768548421561718,
+            0.01747232675552368,
+            0.00155053217895329,
+            0.0033337417989969254,
+            0.005451303441077471,
+            -0.01506342925131321,
+            -0.033505573868751526,
+            0.01698741875588894,
+            -0.0031284382566809654,
+            -0.003015032270923257,
+            0.0047317626886069775,
+            0.011770748533308506,
+            -0.006909937597811222,
+            -0.0012777714291587472,
+            0.005025053862482309,
+            -0.011278019286692142,
+            -0.02269681915640831,
+            0.0009082247270271182,
+            -0.009698158130049706,
+            0.0358206182718277,
+            0.012552858330309391,
+            0.024964936077594757,
+            -0.002391299931332469,
+            0.01024563517421484,
+            -0.003580106422305107,
+            -0.016017602756619453,
+            0.012709279544651508,
+            -0.0076764049008488655,
+            -0.007246244698762894,
+            0.0022954915184527636,
+            0.027233054861426353,
+            -0.016393015161156654,
+            0.007261887192726135,
+            -0.013233293779194355,
+            0.020632049068808556,
+            0.046895287930965424,
+            0.002715875394642353,
+            0.0034667004365473986,
+            0.0422339141368866,
+            -0.009221071377396584,
+            0.0026591725181788206,
+            0.025793973356485367,
+            0.0017382384976372123,
+            -0.016705859452486038,
+            0.04282831773161888,
+            -0.0063350871205329895,
+            0.042390335351228714,
+            -0.044517673552036285,
+            -0.0013246979797258973,
+            0.0358206182718277,
+            -0.01025345642119646,
+            -0.008235613815486431,
+            -0.02599732019007206,
+            0.002567274495959282,
+            -0.020538195967674255,
+            0.021977277472615242,
+            -0.007359650917351246,
+            0.038730062544345856,
+            -0.03944960609078407,
+            0.004434560891240835,
+            0.015399736352264881,
+            -0.01216962467879057,
+            0.036164745688438416,
+            0.01737847365438938,
+            -0.031096674501895905,
+            -0.034600526094436646,
+            0.006663573440164328,
+            0.010175244882702827,
+            0.05446610972285271,
+            -0.01272492203861475,
+            0.008306003175675869,
+            0.027154842391610146,
+            -0.009979717433452606,
+            -0.00019626061839517206,
+            0.043516576290130615,
+            -0.010511551983654499,
+            0.02637273445725441,
+            0.03916804492473602,
+            -0.053621429949998856,
+            -0.02756153978407383,
+            0.019552739337086678,
+            -0.0008725409279577434,
+            0.011301483027637005,
+            0.010941712185740471,
+            0.014085792936384678,
+            -0.0028801183216273785,
+            -0.027342548593878746,
+            0.01938067376613617,
+            0.013585242442786694,
+            0.0015671519795432687,
+            -0.004997679963707924,
+            -0.016752786934375763,
+            -0.012380793690681458,
+            -0.01939631626009941,
+            -0.009979717433452606,
+            0.005025053862482309,
+            0.0025281691923737526,
+            -0.017425401136279106,
+            -0.04326630011200905,
+            -0.02929782308638096,
+            0.027045348659157753,
+            -0.0074691460467875,
+            0.03300502151250839,
+            0.023400716483592987,
+            0.023025304079055786,
+            0.0026024694088846445,
+            -0.025856541469693184,
+            0.03841722011566162,
+            -0.011817675083875656,
+            0.03372456505894661,
+            -0.0022251016926020384,
+            -0.015962855890393257,
+            -0.027968237176537514,
+            0.04310987889766693,
+            -0.01254503708332777,
+            0.006952953990548849,
+            -0.011739464476704597,
+            -0.0036153013352304697,
+            -0.004387634340673685,
+            -0.01066015288233757,
+            0.011184166185557842,
+            0.004974216688424349,
+            0.016877923160791397,
+            0.0008544546435587108,
+            0.021163884550333023,
+            0.017206409946084023,
+            -0.0030033006332814693,
+            0.018144940957427025,
+            -0.027858741581439972,
+            -0.02452695555984974,
+            0.014304783195257187,
+            -0.02684199996292591,
+            -0.027717962861061096,
+            -0.008783089928328991,
+            -0.04051327332854271,
+            -0.006941222120076418,
+            -0.0010157647775486112,
+            -0.010503730736672878,
+            0.035069793462753296,
+            0.03907419368624687,
+            0.015642190352082253,
+            -0.03575804829597473,
+            0.022446542978286743,
+            -0.04407969489693642,
+            -0.008798732422292233,
+            0.0030912880320101976,
+            0.011184166185557842,
+            -0.022305764257907867,
+            0.03278603032231331,
+            0.00886912178248167,
+            -0.008259076625108719,
+            -0.028500070795416832,
+            0.004899916239082813,
+            -0.01030038297176361,
+            -0.0003243310493417084,
+            -0.00799707043915987,
+            0.011794212274253368,
+            0.0257314033806324,
+            0.008118296973407269,
+            0.0005758281331509352,
+            0.002829281147569418,
+            -0.03779153153300285,
+            0.00258291675709188,
+            -0.00422339141368866,
+            0.0023639260325580835,
+            -0.009878043085336685,
+            -0.020147141069173813,
+            -0.005146280862390995,
+            0.002923134481534362,
+            0.003040450857952237,
+            -0.004020043183118105,
+            0.01873934455215931,
+            -0.015657832846045494,
+            0.009838937781751156,
+            0.005220581311732531,
+            -0.04154565930366516,
+            -0.01994379237294197,
+            -0.04526849836111069,
+            -0.0022075041197240353,
+            0.0002268117677886039,
+            0.022008562460541725,
+            -0.007723331451416016,
+            0.009596483781933784,
+            -0.020882325246930122,
+            -0.010832217521965504,
+            -0.0006887452327646315,
+            0.017315905541181564,
+            -0.028922410681843758,
+            0.00019540518405847251,
+            0.006601004395633936,
+            0.02986094169318676,
+            -0.04066969454288483,
+            0.007336187642067671,
+            0.029078831896185875,
+            -0.017988519743084908,
+            -0.004544056486338377,
+            -0.0060965437442064285,
+            0.028875483199954033,
+            -0.024745944887399673,
+            0.0034803873859345913,
+            -0.0065384358167648315,
+            0.006511061917990446,
+            0.01836393214762211,
+            0.01501650270074606,
+            -0.0008935601217672229,
+            0.015524874441325665,
+            0.02288452535867691,
+            -0.0020765007939189672,
+            0.03832336887717247,
+            -0.03963731229305267,
+            -0.03832336887717247,
+            0.028984978795051575,
+            -0.017597464844584465,
+            -0.0031382145825773478,
+            0.009408777579665184,
+            -0.052964456379413605,
+            -0.0026806804817169905,
+            0.011786391027271748,
+            -0.002029574243351817,
+            -0.05671858415007591,
+            0.027968237176537514,
+            0.03406869247555733,
+            0.02527778036892414,
+            0.0013012347044423223,
+            -0.012388614937663078,
+            -0.0006960774771869183,
+            0.02444874495267868,
+            0.012568499892950058,
+            0.0294073186814785,
+            0.014250035397708416,
+            -0.013749485835433006,
+            -0.0008559211273677647,
+            -0.014860081486403942,
+            -0.014719301834702492,
+            -0.017128199338912964,
+            0.016956135630607605,
+            -0.009893685579299927,
+            -0.01563436910510063,
+            0.006186486221849918,
+            -0.005009411368519068,
+            -0.0148444389924407,
+            -0.03935575112700462,
+            -0.03613346070051193,
+            0.018317004665732384,
+            -0.015313704498112202,
+            -0.04861592873930931,
+            0.0065384358167648315,
+            -0.009932790882885456,
+            -0.004446292761713266,
+            -0.007062449119985104,
+            0.010941712185740471,
+            -0.01598631963133812,
+            0.013436641544103622,
+            0.005990959238260984,
+            0.0004746427293866873,
+            0.0003773678618017584,
+            0.018410857766866684,
+            0.01873934455215931,
+            0.013843338936567307,
+            -0.02480851486325264,
+            0.03106539137661457,
+            -0.016612006351351738,
+            -0.05155666172504425,
+            0.020647691562771797,
+            -0.015352809801697731,
+            0.003449103096500039,
+            0.027702320367097855,
+            -0.010511551983654499,
+            -0.02914140187203884,
+            0.028155943378806114,
+            -0.031925711780786514,
+            -0.02343200147151947,
+            -0.022853240370750427,
+            -0.00033044128213077784,
+            -0.03876134753227234,
+            -0.0450495108962059,
+            -0.007042896468192339,
+            0.011262377724051476,
+            -0.00849370937794447,
+            0.0004577784857247025,
+            0.03463181108236313,
+            -0.00357228540815413,
+            -0.03475694730877876,
+            -0.007488698698580265,
+            -0.005251865368336439,
+            -0.009885864332318306,
+            -0.0026865461841225624,
+            0.00029206901672296226,
+            0.004720031283795834,
+            -0.006663573440164328,
+            -0.0262006688863039,
+            -0.022352689877152443,
+            -0.016064530238509178,
+            -0.007344008423388004,
+            -0.01956837996840477,
+            -0.02003764547407627,
+            -0.011098134331405163,
+            -0.03306759148836136,
+            0.03076818957924843,
+            0.026075532659888268,
+            -0.006542346440255642,
+            -0.009174144826829433,
+            -0.009526094421744347,
+            0.0020706350915133953,
+            0.02993915230035782,
+            0.006053527817130089,
+            0.005658562295138836,
+            -0.020272279158234596,
+            0.004829526413232088,
+            -0.011317124590277672,
+            0.011567399837076664,
+            -0.051775652915239334,
+            -0.017534896731376648,
+            0.039199329912662506,
+            0.02271246165037155,
+            -0.0005782722146250308,
+            -0.006409387569874525,
+            0.018535995855927467,
+            -0.029907869175076485,
+            -0.028343649581074715,
+            0.03876134753227234,
+            0.015376273542642593,
+            0.013905907981097698,
+            -0.005506051238626242,
+            0.003539045574143529,
+            0.012787491083145142,
+            0.0019386539934203029,
+            -0.01042552012950182,
+            -0.006022243294864893,
+            0.03363071009516716,
+            0.0017558359540998936,
+            0.0022172804456204176,
+            -0.00822779256850481,
+            0.023478928953409195,
+            0.01023781392723322,
+            -0.0019738490227609873,
+            0.01773824356496334,
+            0.01043334137648344,
+            -0.01709691435098648,
+            -0.011137239634990692,
+            -0.02473030425608158,
+            0.0035527325235307217,
+            0.02260296605527401,
+            0.010034465231001377,
+            -0.02379177138209343,
+            -0.0061982180923223495,
+            -0.006718320772051811,
+            0.00267676985822618,
+            -0.012052307836711407,
+            0.012302583083510399,
+            0.025105716660618782,
+            -0.030940253287553787,
+            -0.014867902733385563,
+            -0.008259076625108719,
+            0.005032874643802643,
+            0.01033948827534914,
+            0.02086668275296688,
+            0.019693518057465553,
+            -0.0020765007939189672,
+            -0.004626177717000246,
+            0.0063116238452494144,
+            0.02756153978407383,
+            0.016580721363425255,
+            -0.02555933967232704,
+            -0.020694619044661522,
+            0.012333867140114307,
+            0.009111575782299042,
+            0.034788232296705246,
+            -0.012279119342565536,
+            -0.00044995738426223397,
+            -0.01496957615017891,
+            0.09547993540763855,
+            -0.01636173203587532,
+            0.026998421177268028,
+            -0.016048887744545937,
+            -0.02857828326523304,
+            0.00028815848054364324,
+            0.003701333422213793,
+            0.011223271489143372,
+            0.05559234693646431,
+            -0.00151631492190063,
+            0.03076818957924843,
+            0.012044486589729786,
+            0.0022075041197240353,
+            0.0029837479814887047,
+            -0.00746132479980588,
+            0.016205308958888054,
+            -0.006421119440346956,
+            0.02351021207869053,
+            0.0110746705904603,
+            0.012920449487864971,
+            -0.009784189984202385,
+            0.02335379086434841,
+            -0.0007136749336495996,
+            0.020725902169942856,
+            -0.023213010281324387,
+            0.04016914591193199,
+            0.00014676773571409285,
+            0.0036368092987686396,
+            -0.004501040559262037,
+            -0.025215212255716324,
+            -0.0044150082394480705,
+            0.011090313084423542,
+            -0.014914829283952713,
+            0.011317124590277672,
+            -0.019318105652928352,
+            0.05318344756960869,
+            -0.02793695218861103,
+            0.022164983674883842,
+            -0.005373092368245125,
+            -0.03603960573673248,
+            0.014406457543373108,
+            0.028187228366732597,
+            0.008016622625291348,
+            0.0028957605827599764,
+            -0.013467926532030106,
+            0.012599784880876541,
+            0.022305764257907867,
+            -0.02030356414616108,
+            2.63045440078713e-05,
+            0.025934752076864243,
+            -0.016877923160791397,
+            0.02307223156094551,
+            0.009674695320427418,
+            -0.02279067225754261,
+            -0.005322255194187164,
+            -0.006843458395451307,
+            0.009095934219658375,
+            -0.005658562295138836,
+            -0.008618847467005253,
+            -0.01038641482591629,
+            0.010456804186105728,
+            0.022759387269616127,
+            0.027702320367097855,
+            0.019271178171038628,
+            -0.01037859357893467,
+            -0.0015642191283404827,
+            0.0007234513177536428,
+            -0.014711480587720871,
+            -0.002647440880537033,
+            -0.012240014038980007,
+            -0.01819186843931675,
+            0.02682635746896267,
+            0.0035058059729635715,
+            -0.003580106422305107,
+            -0.015251136384904385,
+            -0.013413178734481335,
+            -0.02105438895523548,
+            -0.0006515950080938637,
+            -0.0029641950968652964,
+            -0.015696939080953598,
+            -0.003108885372057557,
+            0.015501410700380802,
+            -0.015337168239057064,
+            0.018410857766866684,
+            -0.02482415735721588,
+            0.013718200847506523,
+            -0.027326907962560654,
+            -0.0035234035458415747,
+            0.012599784880876541,
+            -0.012466826476156712,
+            -0.0032574862707406282,
+            -0.01884884014725685,
+            -0.011723821982741356,
+            0.010190887376666069,
+            -0.01836393214762211,
+            -0.007449593394994736,
+            0.007738973945379257,
+            0.005408287514001131,
+            -0.017972877249121666,
+            -0.027170484885573387,
+            -0.022118058055639267,
+            0.005865821614861488,
+            0.04070097953081131,
+            -0.02020971104502678,
+            0.00018428456678520888,
+            -0.007211050018668175,
+            0.006569720339030027,
+            0.0013706469908356667,
+            -0.013772948645055294,
+            -0.0007038986077532172,
+            0.0034354161471128464,
+            -0.004211660008877516,
+            0.024057690054178238,
+            0.006565809715539217,
+            0.023948194459080696,
+            0.005728952586650848,
+            0.0012005381286144257,
+            -0.02066333405673504,
+            -0.016565080732107162,
+            0.016752786934375763,
+            -0.010097034275531769,
+            0.014351709745824337,
+            0.02435489185154438,
+            -0.021429801359772682,
+            0.0012924360344186425,
+            0.01424221508204937,
+            0.014038866385817528,
+            -0.02663865126669407,
+            0.024949293583631516,
+            0.006421119440346956,
+            0.014414278790354729,
+            -0.0021195169538259506,
+            0.02325993776321411,
+            -0.007848469540476799,
+            0.033505573868751526,
+            -0.011504831723868847,
+            -0.00804008636623621,
+            0.0010754006216302514,
+            0.030189428478479385,
+            -0.006370282266288996,
+            -0.00020762563508469611,
+            0.019224252551794052,
+            0.043516576290130615,
+            0.0035664194729179144,
+            -0.0016551392618566751,
+            -0.013452284038066864,
+            0.010362951084971428,
+            0.008259076625108719,
+            0.021382873877882957,
+            0.00450495071709156,
+            0.001804717699997127,
+            -0.008243434131145477,
+            0.017597464844584465,
+            0.006671394221484661,
+            0.011512652039527893,
+            -0.026325806975364685,
+            0.0015642191283404827,
+            0.0070389858447015285,
+            0.010073570534586906,
+            0.002948553068563342,
+            -0.0026885014958679676,
+            0.020632049068808556,
+            -0.010769648477435112,
+            -0.0360083244740963,
+            0.024464385583996773,
+            0.002373702358454466,
+            0.024182826280593872,
+            -0.007054627873003483,
+            -0.015869002789258957,
+            0.02103874646127224,
+            0.002291580894961953,
+            0.006503240671008825,
+            0.025574982166290283,
+            0.0017206409247592092,
+            -0.00269827782176435,
+            0.005623367615044117,
+            0.01005792897194624,
+            0.0031049747485667467,
+            0.0009292439208365977,
+            -0.003840157762169838,
+            0.007257976569235325,
+            0.010448983870446682,
+            0.006080901715904474,
+            -0.002289625583216548,
+            -0.028328007087111473,
+            -0.0009043141617439687,
+            0.009768548421561718,
+            0.0063585503958165646,
+            -0.039825018495321274,
+            0.01497739739716053,
+            -0.006827816367149353,
+            0.004035685211420059,
+            0.0074378615245223045,
+            -0.008478067815303802,
+            -0.012083591893315315,
+            0.01245900522917509,
+            0.0018506667111068964,
+            -0.006898206193000078,
+            0.0043602604418993,
+            -0.03660272806882858,
+            -0.01745668426156044,
+            0.023275580257177353,
+            0.003564464161172509,
+            -0.021429801359772682,
+            0.014117076992988586,
+            -0.007707689423114061,
+            -0.010143960826098919,
+            0.005322255194187164,
+            0.02757718227803707,
+            -0.044705379754304886,
+            0.013694738037884235,
+            -0.02352585457265377,
+            -0.012552858330309391,
+            0.021617507562041283,
+            0.010003181174397469,
+            0.0041295383125543594,
+            -0.00101087661460042,
+            -0.011426620185375214,
+            0.005904926918447018,
+            0.012889165431261063,
+            -0.011895885691046715,
+            -0.020162783563137054,
+            0.0217739287763834,
+            0.012928270734846592,
+            0.005944032687693834,
+            -0.0074339513666927814,
+            0.0215862225741148,
+            -0.010644510388374329,
+            1.3480696452461416e-06,
+            0.009275819174945354,
+            -0.0454249233007431,
+            0.030721262097358704,
+            -0.0043328870087862015,
+            0.02360406517982483,
+            -0.008024443872272968,
+            -0.028328007087111473,
+            -0.008978617377579212,
+            0.01645558513700962,
+            0.018692417070269585,
+            0.005717220716178417,
+            0.0061982180923223495,
+            0.032019563019275665,
+            -0.0129595547914505,
+            0.013819875195622444,
+            0.0057993424125015736,
+            -0.00016289875202346593,
+            0.0261068157851696,
+            -0.016752786934375763,
+            -0.03130002319812775,
+            0.01428914163261652,
+            -0.012513753026723862,
+            -0.03663400933146477,
+            0.015337168239057064,
+            -0.02103874646127224,
+            -0.03397483751177788,
+            -0.029986079782247543,
+            -0.0035136269871145487,
+            -0.007344008423388004,
+            -0.0014830752043053508,
+            -0.003116706619039178,
+            0.016924850642681122,
+            0.023932551965117455,
+            0.012005381286144257,
+            0.01525895670056343,
+            -0.006511061917990446,
+            0.01419528853148222,
+            -0.00397116132080555,
+            0.033224012702703476,
+            0.0012435541721060872,
+            0.004325065761804581,
+            0.001327630947344005,
+            0.017957234755158424,
+            0.013389714993536472,
+            0.020084572955965996,
+            0.009612126275897026,
+            -0.015368452295660973,
+            -0.01900526136159897,
+            -0.046738866716623306,
+            0.009760727174580097,
+            -0.004692657385021448,
+            0.006256876513361931,
+            0.03813566267490387,
+            -0.013186367228627205,
+            -0.010816575028002262,
+            0.009228892624378204,
+            0.004966395441442728,
+            0.0032437993213534355,
+            -0.013108155690133572,
+            -0.006100454367697239,
+            -0.0031499462202191353,
+            -0.02233704924583435,
+            -0.0253559909760952,
+            -0.018442142754793167,
+            0.020819755271077156,
+            -0.0017597464611753821,
+            0.013108155690133572,
+            -0.02077282965183258,
+            -0.0031577672343701124,
+            -0.028093375265598297,
+            -0.011191987432539463,
+            0.0013872667914256454,
+            0.013084692880511284,
+            0.004821705166250467,
+            0.016971776261925697,
+            -0.005330076441168785,
+            -0.020256636664271355,
+            -0.017800813540816307,
+            0.012075771577656269,
+            -0.010761827230453491,
+            -0.005947942845523357,
+            -0.004978127311915159,
+            0.01938067376613617,
+            0.007954053580760956,
+            0.004278139211237431,
+            -0.001422461704351008,
+            0.021899066865444183,
+            -0.021085673943161964,
+            -0.015955034643411636,
+            0.01747232675552368,
+            0.03262960910797119,
+            -0.03170672059059143,
+            -0.005552977789193392,
+            0.0009018700802698731,
+            0.014688016846776009,
+            0.015861181542277336,
+            0.012693637982010841,
+            -0.006557988468557596,
+            -0.008055727928876877,
+            -0.028046447783708572,
+            -0.01240425743162632,
+            -0.04095125570893288,
+            0.0029114028438925743,
+            0.029720162972807884,
+            -0.005025053862482309,
+            0.01911475695669651,
+            0.017988519743084908,
+            -0.027170484885573387,
+            6.660151848336682e-05,
+            0.004106075037270784,
+            0.017628749832510948,
+            -0.033849701285362244,
+            0.0145628796890378,
+            -0.009361851029098034,
+            0.02169571816921234,
+            0.0005733840516768396,
+            -0.01909911446273327,
+            -0.017753886058926582,
+            -0.011888065375387669,
+            0.021742645651102066,
+            -0.00253990083001554,
+            -0.029751446098089218,
+            0.021633150056004524,
+            0.009565199725329876,
+            -0.007136749569326639,
+            -0.024949293583631516,
+            0.012013202533125877,
+            0.0129595547914505,
+            0.017613107338547707,
+            -0.01074618473649025,
+            -0.002436271170154214,
+            -0.012568499892950058,
+            -0.010214350186288357,
+            -0.022211911156773567,
+            -0.00043211551383137703,
+            -0.017628749832510948,
+            0.0015945257619023323,
+            -0.011512652039527893,
+            -0.0022778939455747604,
+            0.014351709745824337,
+            -0.041483089327812195,
+            -0.0018457785481587052,
+            -0.008955154567956924,
+            0.028531355783343315,
+            0.02473030425608158,
+            0.008478067815303802,
+            0.0027960415463894606,
+            -0.02113259956240654,
+            -0.010816575028002262,
+            -0.011786391027271748,
+            0.01820751093327999,
+            -0.02261860854923725,
+            0.0003284859994892031,
+            0.010018822737038136,
+            0.008611026220023632,
+            0.0003871442168019712,
+            0.005114996340125799,
+            -0.002453868743032217,
+            -0.02094489336013794,
+            0.014641090296208858,
+            -0.0005171699449419975,
+            0.007003790698945522,
+            -0.012138339690864086,
+            0.0032262017484754324,
+            -0.0036778701469302177,
+            0.025668835267424583,
+            -0.014774048700928688,
+            -0.004387634340673685,
+            -0.010355129837989807,
+            0.0013110111467540264,
+            0.0070937336422502995,
+            -1.234266619576374e-05,
+            -0.0006423074519261718,
+            -0.010777469724416733,
+            0.01240425743162632,
+            -0.011160703375935555,
+            0.030439702793955803,
+            9.531959949526936e-05,
+            -0.03172236308455467,
+            -0.0036074803210794926,
+            -0.03766639530658722,
+            -0.007168034091591835,
+            -0.017800813540816307,
+            -0.033943552523851395,
+            -0.007430040743201971,
+            0.0017089092871174216,
+            -0.01493047084659338,
+            0.016283521428704262,
+            0.004559698514640331,
+            -0.02473030425608158,
+            0.013905907981097698,
+            0.006831726990640163,
+            -0.017894666641950607,
+            -0.0038440683856606483,
+            -0.01920861005783081,
+            0.0049116481095552444,
+            0.005142370238900185,
+            0.009987538680434227,
+            0.018348289653658867,
+            -0.007394845597445965,
+            -0.013022123835980892,
+            0.003024808596819639,
+            0.02096053585410118,
+            0.01483661774545908,
+            0.010824396274983883,
+            0.0068356371484696865,
+            0.00377758895047009,
+            -0.025043146684765816,
+            -0.003355249995365739,
+            -0.008149581030011177,
+            -0.0070389858447015285,
+            -0.0009810586925595999,
+            0.0034295502118766308,
+            -0.008337287232279778,
+            -0.0006525726639665663,
+            0.000982525059953332,
+            0.0018956379499286413,
+            0.0030072112567722797,
+            -0.013546137139201164,
+            0.0012865702155977488,
+            -0.008517173118889332,
+            0.002373702358454466,
+            0.024026405066251755,
+            -0.012185266241431236,
+            0.008650131523609161,
+            -0.0027510703075677156,
+            0.00634290836751461,
+            0.038636211305856705,
+            0.015352809801697731,
+            -0.044423822313547134,
+            0.003670048899948597,
+            0.007926680147647858,
+            0.0008823173120617867,
+            -0.017972877249121666,
+            0.017675675451755524,
+            0.004837347660213709,
+            -0.008650131523609161,
+            -0.002266162307932973,
+            -0.007844558916985989,
+            0.009385314770042896,
+            -0.015251136384904385,
+            0.012302583083510399,
+            0.013420999981462955,
+            -0.01438299473375082,
+            -0.004923379514366388,
+            0.038730062544345856,
+            -0.010941712185740471,
+            -0.011285840533673763,
+            0.0009121352341026068,
+            0.013319325633347034,
+            0.011262377724051476,
+            -0.004798241890966892,
+            0.014500310644507408,
+            0.0037365283351391554,
+            0.006186486221849918,
+            0.007218870799988508,
+            0.0597531683743,
+            0.0067769791930913925,
+            0.008251255378127098,
+            0.011364051140844822,
+            -0.007445682771503925,
+            -0.014500310644507408,
+            -0.018958335742354393,
+            -0.026357091963291168,
+            -0.004207749385386705,
+            0.01590810716152191,
+            -0.018270079046487808,
+            -0.0032653072848916054,
+            -0.02252475544810295,
+            0.01249811053276062,
+            0.013061229139566422,
+            0.005709399469196796,
+            -0.01717512495815754,
+            0.02066333405673504,
+            0.02793695218861103,
+            0.010308203287422657,
+            -0.0016746920300647616,
+            0.017034346237778664,
+            0.005036785267293453,
+            -0.015548337250947952,
+            -0.0023404627572745085,
+            -0.0023717472795397043,
+            0.004720031283795834,
+            0.021382873877882957,
+            -0.008251255378127098,
+            0.01221655122935772,
+            -0.0045166825875639915,
+            0.009221071377396584,
+            0.002291580894961953,
+            -0.013115976937115192,
+            -0.008306003175675869,
+            -0.0008432118338532746,
+            0.01792594976723194,
+            0.00013894664880353957,
+            -0.025011863559484482,
+            0.0030365402344614267,
+            0.0045948936603963375,
+            0.002862520981580019,
+            0.003812784096226096,
+            -0.014727122150361538,
+            -0.007938412018120289,
+            0.021914709359407425,
+            -0.016111455857753754,
+            0.0032066490966826677,
+            -0.012568499892950058,
+            0.0011369917774572968,
+            -0.01285788044333458,
+            0.0006750582833774388,
+            0.00397116132080555,
+            0.016612006351351738,
+            -0.027608467265963554,
+            -0.015352809801697731,
+            0.0217739287763834,
+            0.00836075097322464,
+            0.009049007669091225,
+            -0.0027256517205387354,
+            -0.005658562295138836,
+            0.016424300149083138,
+            -0.01783209666609764,
+            -0.013827696442604065,
+            0.005904926918447018,
+            -0.008024443872272968,
+            0.0052323127165436745,
+            -0.02296273596584797,
+            0.007969696074724197,
+            -0.008071370422840118,
+            -0.008165223523974419,
+            0.008212150074541569,
+            0.015829896554350853,
+            0.003789320820942521,
+            0.007918858900666237,
+            -0.0008104610024020076,
+            -0.01801980286836624,
+            -0.029516814276576042,
+            0.027311265468597412,
+            0.0043915449641644955,
+            -0.00404350645840168,
+            0.004219480790197849,
+            -0.0038049628492444754,
+            -0.004888184834271669,
+            -0.0021449353080242872,
+            -0.014641090296208858,
+            -0.02271246165037155,
+            0.013647811487317085,
+            0.012247835285961628,
+            -0.012294761836528778,
+            0.021289020776748657,
+            -0.008532814681529999,
+            0.007715510670095682,
+            -0.01791030913591385,
+            0.00850153062492609,
+            0.028703419491648674,
+            -0.006823905743658543,
+            -0.014359530992805958,
+            -0.0016248325118795037,
+            -0.008000980131328106,
+            0.0035370904952287674,
+            0.027264337986707687,
+            0.0062451446428895,
+            0.014531595632433891,
+            -0.01728462055325508,
+            -0.011176344938576221,
+            0.02243090234696865,
+            0.004434560891240835,
+            0.011207629926502705,
+            -0.0010001225164160132,
+            -0.009095934219658375,
+            -0.012099234387278557,
+            -0.00790321733802557,
+            0.010581942275166512,
+            0.012326045893132687,
+            -0.013288040645420551,
+            -0.0005117929540574551,
+            -0.007871932350099087,
+            0.005588172469288111,
+            -0.012505931779742241,
+            -0.011434441432356834,
+            -0.022008562460541725,
+            -0.008509351871907711,
+            -0.015532695688307285,
+            -0.026513513177633286,
+            -0.007168034091591835,
+            -0.0008265920332632959,
+            -0.012021023780107498,
+            -0.0369468554854393,
+            -0.022477827966213226,
+            -0.007547357119619846,
+            -0.019740445539355278,
+            0.008532814681529999,
+            0.0005748505354858935,
+            -0.021070031449198723,
+            0.006831726990640163,
+            -0.012349509634077549,
+            -0.009354029782116413,
+            0.00251448224298656,
+            0.01244336273521185,
+            -0.010065749287605286,
+            -0.014774048700928688,
+            -0.01240425743162632,
+            -0.01994379237294197,
+            0.0146567327901721,
+            -0.0027667125687003136,
+            0.003470611060038209,
+            -0.006299892440438271,
+            -0.004297691863030195,
+            -0.015454484149813652,
+            0.01994379237294197,
+            -0.02473030425608158,
+            0.0013139439979568124,
+            -0.0015867047477513552,
+            0.006929490715265274,
+            -0.003302457509562373,
+            -0.01745668426156044,
+            -0.009088112972676754,
+            -0.0010773559333756566,
+            -0.015313704498112202,
+            -0.02003764547407627,
+            -0.013014302588999271,
+            -0.005795431789010763,
+            0.015337168239057064,
+            -0.008564099669456482,
+            0.01273274328559637,
+            0.007922769524157047,
+            0.0035194929223507643,
+            -0.004422829486429691,
+            0.0035527325235307217,
+            -0.010738363489508629,
+            -0.0034803873859345913,
+            0.0017558359540998936,
+            0.005474766716361046,
+            0.006022243294864893,
+            0.0014263722114264965,
+            0.015939392149448395,
+            -0.009080291725695133,
+            0.012013202533125877,
+            0.020287921652197838,
+            0.02150801196694374,
+            0.006194307468831539,
+            0.00629207119345665,
+            0.004141270183026791,
+            -9.611393033992499e-05,
+            0.002846878720447421,
+            0.0023873893078416586,
+            -0.010980817489326,
+            0.012185266241431236,
+            -0.015587443485856056,
+            -0.008603204973042011,
+            0.013890265487134457,
+            -0.002537945518270135,
+            -0.015603085048496723,
+            0.015923749655485153,
+            0.0016746920300647616,
+            -0.02369791828095913,
+            -0.01062104757875204,
+            -0.007156302221119404,
+            0.004262497182935476,
+            0.020553838461637497,
+            0.007293171249330044,
+            -0.027811815962195396,
+            0.01708127185702324,
+            -0.0002189906663261354,
+            -0.013147260993719101,
+            -0.009103755466639996,
+            0.01837957464158535,
+            0.024042047560214996,
+            -0.0006647931295447052,
+            -0.04633216932415962,
+            -0.014226572588086128,
+            0.00808701291680336,
+            -0.026685576885938644,
+            -0.0285626407712698,
+            0.011684716679155827,
+            -0.00808701291680336,
+            -0.031565941870212555,
+            -0.005279239267110825,
+            -0.0007855312433093786,
+            -0.03078383207321167,
+            0.009494809433817863,
+            -0.0008705856744199991,
+            0.0032868152484297752,
+            0.007770258467644453,
+            -0.013647811487317085,
+            -0.00544348219409585,
+            -0.02471466176211834,
+            0.012286940589547157,
+            -0.007035075221210718,
+            -0.0038597106467932463,
+            -0.0036368092987686396,
+            0.009815474972128868,
+            -0.002762801945209503,
+            0.02279067225754261,
+            0.026701219379901886,
+            0.01966223306953907,
+            -0.003132348647341132,
+            -0.023588424548506737,
+            -0.009377493523061275,
+            0.002107785316184163,
+            0.012889165431261063,
+            0.006831726990640163,
+            -0.017018703743815422,
+            0.005678115412592888,
+            0.0014605895848944783,
+            0.013436641544103622,
+            0.000748869904782623,
+            0.02609117329120636,
+            -3.1039973691804335e-05,
+            0.002481242408975959,
+            0.002082366729155183,
+            -0.02580961398780346,
+            0.004680925514549017,
+            -0.0058110738173127174,
+            0.022477827966213226,
+            0.016784070059657097,
+            -0.0003084444615524262,
+            0.007011611945927143,
+            -0.0127718485891819,
+            -0.004727852065116167,
+            -0.028234153985977173,
+            -0.01709691435098648,
+            -0.00015947702922858298,
+            0.031002821400761604,
+            0.033130161464214325,
+            -0.02673250436782837,
+            -0.010136139579117298,
+            -0.0009385314187966287,
+            0.029735805466771126,
+            -0.006022243294864893,
+            -0.00218013022094965,
+            0.029000621289014816,
+            0.004485398065298796,
+            0.0030717351473867893,
+            0.012200908735394478,
+            0.0012767937732860446,
+            0.008861301466822624,
+            0.033568140119314194,
+            0.015915928408503532,
+            0.009627768769860268,
+            0.008861301466822624,
+            -0.002152756555005908,
+            -0.015837717801332474,
+            -0.021992919966578484,
+            -0.0020589034538716078,
+            -0.00652279332280159,
+            0.009486989118158817,
+            -0.018144940957427025,
+            0.006022243294864893,
+            -0.011786391027271748,
+            0.004125627689063549,
+            -0.011817675083875656,
+            -0.0022876705043017864,
+            0.0017450818559154868,
+            -0.01700306124985218,
+            -0.0025985590182244778,
+            -0.024980578571558,
+            -0.0027510703075677156,
+            0.0108869643881917,
+            -0.001176097197458148,
+            0.016127098351716995,
+            -0.016471227630972862,
+            0.0051541016437113285,
+            0.019177325069904327,
+            0.021289020776748657,
+            -0.01474276464432478,
+            6.71514353598468e-05,
+            0.003977026790380478,
+            -0.025981679558753967,
+            0.01579861342906952,
+            0.005158012267202139,
+            0.010175244882702827,
+            0.025684477761387825,
+            -0.0009443972376175225,
+            -0.0031831858213990927,
+            -0.012826596386730671,
+            -0.004532324615865946,
+            -0.03491336852312088,
+            0.024182826280593872,
+            -0.00034339496050961316,
+            0.009111575782299042,
+            0.004931200761348009,
+            -0.009627768769860268,
+            0.007574731018394232,
+            -0.010808753781020641,
+            0.019834298640489578,
+            0.005904926918447018,
+            0.003060003509745002,
+            -0.01492264959961176,
+            0.01653379574418068,
+            0.0025731404311954975,
+            -0.01894269324839115,
+            0.02463645115494728,
+            -0.01055847853422165,
+            0.0044932193122804165,
+            -0.0030580481979995966,
+            0.011160703375935555,
+            0.009690336883068085,
+            -0.006968596018850803,
+            0.004149090964347124,
+            -0.0296888779848814,
+            -0.02307223156094551,
+            -0.013037766329944134,
+            -0.0011389469727873802,
+            -0.007742884568870068,
+            -0.026435302570462227,
+            -0.022493470460176468,
+            0.004743494559079409,
+            -0.005294881761074066,
+            0.004688746761530638,
+            0.012208729982376099,
+            0.012654532678425312,
+            0.005728952586650848,
+            0.006331176497042179,
+            -0.009354029782116413,
+            0.006565809715539217,
+            0.026904568076133728,
+            -0.006565809715539217,
+            -0.03197263926267624,
+            -0.006933400873094797,
+            0.003554687835276127,
+            -0.018582923337817192,
+            -0.009948433376848698,
+            0.009486989118158817,
+            -0.004027863964438438,
+            0.0288442000746727,
+            -0.02380741387605667,
+            0.0038460236974060535,
+            -0.0038831739220768213,
+            0.004950753413140774,
+            -0.004806063137948513,
+            1.4840223229839467e-05,
+            -0.01911475695669651,
+            0.00017340834892820567,
+            0.004946842789649963,
+            -0.016846640035510063,
+            0.015696939080953598,
+            -0.007230602670460939,
+            -0.011019923724234104,
+            -0.007699868176132441,
+            0.0012846149038523436,
+            -0.007211050018668175,
+            -0.001168276066891849,
+            0.02627888135612011,
+            -0.007973606698215008,
+            -0.017394116148352623,
+            -0.004547967109829187,
+            0.0053574503399431705,
+            0.015149462036788464,
+            0.011473546735942364,
+            0.024057690054178238,
+            -0.013389714993536472,
+            0.003515582298859954,
+            -0.004059148486703634,
+            -0.0031734094955027103,
+            -4.744594116345979e-05,
+            0.039105478674173355,
+            -0.011434441432356834,
+            -0.0004949286812916398,
+            -0.00844678282737732,
+            0.009479167871177197,
+            0.006495419889688492,
+            0.020616406574845314,
+            0.006346818991005421,
+            0.002668948844075203,
+            0.007160212844610214,
+            -0.0006931445677764714,
+            0.0006760359392501414,
+            0.01460198499262333,
+            0.011184166185557842,
+            0.00723842391744256,
+            -0.0072814398445189,
+            -0.023025304079055786,
+            0.014234393835067749,
+            -0.0010841992916539311,
+            -0.0031675435602664948,
+            0.010261276736855507,
+            0.01212269812822342,
+            -0.009346208535134792,
+            0.012435541488230228,
+            -0.007703778799623251,
+            0.024761587381362915,
+            0.013702559284865856,
+            -0.01005792897194624,
+            0.009346208535134792,
+            0.021633150056004524,
+            -0.00032139813993126154,
+            0.015235493890941143,
+            -0.015423200093209743,
+            -0.021945994347333908,
+            0.0012210685526952147,
+            0.021539296954870224,
+            0.0005015277420170605,
+            -0.015344989486038685,
+            0.009322745725512505,
+            -0.007856289856135845,
+            -0.007805453147739172,
+            0.017847739160060883,
+            0.0125841423869133,
+            -0.009463525377213955,
+            -0.009682515636086464,
+            0.0010314069222658873,
+            0.004872542340308428,
+            0.012943913228809834,
+            0.02066333405673504,
+            -0.003447147784754634,
+            0.004239033907651901,
+            -0.00251448224298656,
+            0.03860492631793022,
+            -0.005748505238443613,
+            0.03125309571623802,
+            -0.015642190352082253,
+            -0.011528294533491135,
+            -0.009205428883433342,
+            -0.0063389977440238,
+            -0.005509961862117052,
+            -0.0024421371053904295,
+            0.0012347553856670856,
+            -0.005482587963342667,
+            -0.023103516548871994,
+            0.012717100791633129,
+            0.022634249180555344,
+            0.002792131155729294,
+            0.007031164597719908,
+            0.005013321992009878,
+            0.013710380531847477,
+            0.030924610793590546,
+            -0.005615546368062496,
+            -0.0147505858913064,
+            0.009002081118524075,
+            0.010034465231001377,
+            -0.016424300149083138,
+            0.021179527044296265,
+            0.006765247322618961,
+            0.006710499990731478,
+            -0.014320425689220428,
+            -0.0017147751059383154,
+            0.02022535353899002,
+            0.0071758548729121685,
+            0.009792011231184006,
+            -0.012294761836528778,
+            -0.0025966037064790726,
+            0.0012679951032623649,
+            0.0072814398445189,
+            -0.0034764769952744246,
+            -0.015008682385087013,
+            0.0159784983843565,
+            -0.014390815049409866,
+            0.02261860854923725,
+            -0.021335948258638382,
+            0.012513753026723862,
+            -0.009252355434000492,
+            0.020694619044661522,
+            -0.0011545892339199781,
+            -0.007727242074906826,
+            -0.00534962909296155,
+            0.013147260993719101,
+            -0.005955764092504978,
+            -0.00256140879355371,
+            -0.02250911295413971,
+            -0.013499210588634014,
+            0.017879024147987366,
+            -0.0216800756752491,
+            -0.0012826595921069384,
+            0.0019005262292921543,
+            0.010714900679886341,
+            -0.028687776997685432,
+            0.022821955382823944,
+            0.031174886971712112,
+            0.012294761836528778,
+            0.012474646791815758,
+            -0.0045166825875639915,
+            -0.016674574464559555,
+            -0.009432241320610046,
+            0.03344300389289856,
+            0.008345108479261398,
+            -0.01801980286836624,
+            0.006952953990548849,
+            0.01830136403441429,
+            -0.009119397029280663,
+            0.004661372862756252,
+            -0.0017685452476143837,
+            0.016064530238509178,
+            0.0020178426057100296,
+            0.0035898827482014894,
+            -0.01673714444041252,
+            -0.0365714430809021,
+            -0.012302583083510399,
+            0.02060076594352722,
+            -0.005666383542120457,
+            -0.008861301466822624,
+            0.009033365175127983,
+            0.0020002450328320265,
+            -0.003146035596728325,
+            -0.011387514881789684,
+            0.003810828784480691,
+            0.0074339513666927814,
+            0.006080901715904474,
+            -0.01967787556350231,
+            0.012021023780107498,
+            -0.009361851029098034,
+            -0.010276919230818748,
+            0.01075400598347187,
+            -0.013858981430530548,
+            -0.026216311380267143,
+            -0.0022172804456204176,
+            -0.015563979744911194,
+            -0.015689117833971977,
+            -0.0144690265879035,
+            0.006816084496676922,
+            -0.0012279120273888111,
+            -0.017347188666462898,
+            -0.028593923896551132,
+            0.014077971689403057,
+            0.0010186976287513971,
+            0.028265438973903656,
+            -0.029078831896185875,
+            0.022649891674518585,
+            0.00035341575858183205,
+            0.012920449487864971,
+            0.0017216185806319118,
+            -0.011848959140479565,
+            0.0019220341928303242,
+            0.005021143238991499,
+            0.003394355298951268,
+            0.001639497117139399,
+            -0.010456804186105728,
+            -0.0010890875710174441,
+            -0.018160583451390266,
+            0.006773068569600582,
+            -0.005236223340034485,
+            -0.014187467284500599,
+            -0.0026415749453008175,
+            0.015431021340191364,
+            0.00606134906411171,
+            0.022290121763944626,
+            -0.019599664956331253,
+            -0.002825370756909251,
+            -0.007817184552550316,
+            -0.012161803431808949,
+            -0.006788710597902536,
+            0.02031920664012432,
+            -0.007222781423479319,
+            0.03363071009516716,
+            0.008923869580030441,
+            -0.01024563517421484,
+            -0.011450083926320076,
+            0.0012562634656205773,
+            0.02066333405673504,
+            0.018504710868000984,
+            -0.02333814837038517,
+            -0.012114876881241798,
+            -0.0013892221031710505,
+            0.013311504386365414,
+            0.006690946873277426,
+            -0.0016717591788619757,
+            0.01736283116042614,
+            0.0066166468895971775,
+            -0.01679971255362034,
+            -0.01254503708332777,
+            -0.002807773184031248,
+            0.015650011599063873,
+            0.013217651285231113,
+            0.013960654847323895,
+            0.005337897688150406,
+            0.013248935341835022,
+            0.01902090385556221,
+            -0.01745668426156044,
+            -0.005369181744754314,
+            0.019255537539720535,
+            0.0029270448721945286,
+            -0.017879024147987366,
+            0.0028195048216730356,
+            0.0023033125326037407,
+            0.00807919166982174,
+            0.003292681183665991,
+            -0.004196017514914274,
+            0.03155029937624931,
+            -0.0009263110114261508,
+            0.005025053862482309,
+            0.018051087856292725,
+            -0.006655752193182707,
+            0.022305764257907867,
+            -0.0017421490047127008,
+            0.028906768187880516,
+            -0.02003764547407627,
+            0.003496029647067189,
+            0.0004572896577883512,
+            -0.0027979968581348658,
+            -0.0026357092428952456,
+            0.0016150561859831214,
+            -0.018317004665732384,
+            0.009275819174945354,
+            -0.04567519575357437,
+            0.008345108479261398,
+            -0.007602104917168617,
+            -0.0021410249173641205,
+            -0.0248867254704237,
+            -0.01015960332006216,
+            -0.009150682017207146,
+            -0.010581942275166512,
+            -0.018786270171403885,
+            -0.021899066865444183,
+            -0.00789930671453476,
+            0.004751315340399742,
+            0.01470365934073925,
+            -0.019896866753697395,
+            0.01939631626009941,
+            0.007070270366966724,
+            -0.026982778683304787,
+            -0.016877923160791397,
+            -0.019959434866905212,
+            -0.0010538926580920815,
+            -0.010065749287605286,
+            0.0028781630098819733,
+            -0.027952594682574272,
+            0.022024204954504967,
+            0.016674574464559555,
+            -0.022258836776018143,
+            0.0058580003678798676,
+            -0.01428914163261652,
+            -0.023776130750775337,
+            0.0017499701352789998,
+            0.0006931445677764714,
+            -0.00864231027662754,
+            0.04273446649312973,
+            0.021101314574480057,
+            8.737629832467064e-05,
+            -0.028640851378440857,
+            0.005728952586650848,
+            -0.02727998048067093,
+            0.013921549543738365,
+            0.004747404716908932,
+            0.00015752174658700824,
+            -0.0045166825875639915,
+            0.016236593946814537,
+            0.0019738490227609873,
+            0.01698741875588894,
+            0.006057438440620899,
+            0.0007400711765512824,
+            0.02473030425608158,
+            -0.011301483027637005,
+            0.006792621221393347,
+            -8.447394066024572e-06,
+            -0.0005313456640578806,
+            0.015696939080953598,
+            -0.004966395441442728,
+            -0.002283759880810976,
+            0.012279119342565536,
+            0.000982525059953332,
+            -0.009721621870994568,
+            -0.004371992312371731,
+            -0.0030424061696976423,
+            0.009135039523243904,
+            -0.015493590384721756,
+            0.01001100242137909,
+            0.0026259326841682196,
+            0.01928682066500187,
+            -0.018254436552524567,
+            -0.005803252570331097,
+            0.002670904155820608,
+            -0.009940612129867077,
+            -0.004606625065207481,
+            -0.015032145194709301,
+            -0.009338388219475746,
+            -0.02020971104502678,
+            0.01617402583360672,
+            0.04320373013615608,
+            0.02252475544810295,
+            0.000653550261631608,
+            -0.004055237863212824,
+            -0.01433606818318367,
+            0.003934010863304138,
+            -0.003478432074189186,
+            -0.011927170678973198,
+            -0.022540396079421043,
+            -0.005189296789467335,
+            0.012036665342748165,
+            -0.004078701138496399,
+            0.0019132354063913226,
+            -0.011653431691229343,
+            0.021101314574480057,
+            -0.01273274328559637,
+            -0.0004919957718811929,
+            -0.018723702058196068,
+            -0.015470126643776894,
+            0.009393135085701942,
+            0.032848600298166275,
+            -0.0023326417431235313,
+            0.013420999981462955,
+            0.013022123835980892,
+            0.004023953340947628,
+            -0.01478969119489193,
+            0.010675795376300812,
+            -0.006010511890053749,
+            0.004106075037270784,
+            -0.0025477218441665173,
+            -0.02380741387605667,
+            -0.009267997927963734,
+            -0.011176344938576221,
+            0.006909937597811222,
+            -0.009768548421561718,
+            -0.0006555055733770132,
+            -0.0015495545230805874,
+            -0.009909328073263168,
+            -0.018160583451390266,
+            0.007586462423205376,
+            -0.007774168625473976,
+            0.0032437993213534355,
+            0.00014090191689319909,
+            -0.0018741299863904715,
+            0.0087205208837986,
+            0.02020971104502678,
+            -0.0067535159178078175,
+            0.0030130769591778517,
+            0.004931200761348009,
+            -0.016846640035510063,
+            0.016393015161156654,
+            0.0019533184822648764,
+            -0.0006476844428107142,
+            0.03237933665513992,
+            0.016236593946814537,
+            0.012795312330126762,
+            -0.0017841873923316598,
+            -0.00305413780733943,
+            -0.001330563798546791,
+            0.014672375284135342,
+            -0.03388098627328873,
+            -0.009111575782299042,
+            0.014985218644142151,
+            0.02067897655069828,
+            -0.015829896554350853,
+            0.0127718485891819,
+            -0.015032145194709301,
+            -0.009690336883068085,
+            -0.007297081872820854,
+            0.00303262984380126,
+            -0.05340243875980377,
+            0.011043386533856392,
+            -0.006366371642798185,
+            0.02712355926632881,
+            0.0006178665207698941,
+            0.0108869643881917,
+            0.010120497085154057,
+            -0.006100454367697239,
+            0.025606267154216766,
+            0.0015045831678435206,
+            -0.0041764648631215096,
+            0.012850060127675533,
+            -0.011778569780290127,
+            -0.011661252938210964,
+            0.008564099669456482,
+            -0.0288442000746727,
+            0.008994259871542454,
+            -0.024933652952313423,
+            0.0022270570043474436,
+            -0.016220951452851295,
+            -0.012396436184644699,
+            0.022649891674518585,
+            0.03287988528609276,
+            0.00202175322920084,
+            0.0022329227067530155,
+            -0.010816575028002262,
+            0.025700120255351067,
+            -0.010550657287240028,
+            0.019443243741989136,
+            -0.008876943029463291,
+            -0.011207629926502705,
+            -0.017112556844949722,
+            -0.010808753781020641,
+            0.008783089928328991,
+            -0.003986803349107504,
+            -0.004375902935862541,
+            -0.021461086347699165,
+            0.005287060514092445,
+            -0.022211911156773567,
+            0.008837837725877762,
+            0.002567274495959282,
+            -0.006518883164972067,
+            0.001965050119906664,
+            0.0029876583721488714,
+            -0.0010167424334213138,
+            -0.00016729811613913625,
+            -0.024088973179459572,
+            -0.008743984624743462,
+            -0.008118296973407269,
+            0.009049007669091225,
+            0.001364781172014773,
+            -0.020538195967674255,
+            0.010581942275166512,
+            -0.009095934219658375,
+            -0.0036153013352304697,
+            -0.008657952770590782,
+            0.0028038627933710814,
+            0.0013872667914256454,
+            0.02067897655069828,
+            0.003855800023302436,
+            -0.020194068551063538,
+            0.009494809433817863,
+            0.017394116148352623,
+            -0.00253990083001554,
+            0.006331176497042179,
+            -0.00019063919899053872,
+            -0.01028474047780037,
+            -0.010589763522148132,
+            -0.0012865702155977488,
+            0.006299892440438271,
+            -0.009831116534769535,
+            0.005744594614952803,
+            0.006992059294134378,
+            0.0058345370925962925,
+            0.021070031449198723,
+            0.008900406770408154,
+            0.004242944065481424,
+            -0.0015603085048496723,
+            0.006733963266015053,
+            -0.0011614327086135745,
+            -0.0014928515302017331,
+            -0.012748385779559612,
+            0.009862401522696018,
+            -0.019630949944257736,
+            0.009651231579482555,
+            0.017675675451755524,
+            0.008986438624560833,
+            -0.016971776261925697,
+            -0.01755053736269474,
+            -0.007363561075180769,
+            -0.023478928953409195,
+            0.009095934219658375,
+            0.0007459369953721762,
+            0.018535995855927467,
+            -0.0046535516157746315,
+            0.008196507580578327,
+            0.006147380918264389,
+            -0.013804233632981777,
+            -0.0020549928303807974,
+            0.009502630680799484,
+            -0.029657592996954918,
+            -0.012521573342382908,
+            -0.000881339656189084,
+            -0.0010548701975494623,
+            0.018176225945353508,
+            0.01919296756386757,
+            0.019146041944622993,
+            0.03206649050116539,
+            -0.009705979377031326,
+            0.01023781392723322,
+            -0.0028371023945510387,
+            0.018864482641220093,
+            0.010042286477982998,
+            0.00733227701857686,
+            0.020632049068808556,
+            0.006972506642341614,
+            -0.02103874646127224,
+            -0.010832217521965504,
+            0.019818656146526337,
+            -0.010417698882520199,
+            -0.005959674715995789,
+            -0.012552858330309391,
+            0.0037756338715553284,
+            0.0033904449082911015,
+            0.011958454735577106,
+            0.018708059564232826,
+            0.008884764276444912,
+            -0.017315905541181564,
+            0.032754749059677124,
+            0.0007058538612909615,
+            0.00647977739572525,
+            0.004199928138405085,
+            -0.003887084312736988,
+            -0.005502140615135431,
+            0.013968476094305515,
+            0.012466826476156712,
+            -0.0014938291860744357,
+            0.0034745216835290194,
+            0.012623247690498829,
+            -0.01447684783488512,
+            -0.018598563969135284,
+            0.007527804467827082,
+            0.0017558359540998936,
+            0.00037883431650698185,
+            0.006589272990822792,
+            0.0025496771559119225,
+            0.017613107338547707,
+            -0.006831726990640163,
+            0.028187228366732597,
+            -0.005521693266928196,
+            -0.016142740845680237,
+            -0.013139440678060055,
+            0.0011555668897926807,
+            -0.011903706938028336,
+            -0.018144940957427025,
+            0.007504341192543507,
+            -0.0015104489866644144,
+            0.011934991925954819,
+            -0.002289625583216548,
+            0.004250765312463045,
+            0.013381893746554852,
+            0.017706960439682007,
+            0.014492489397525787,
+            0.002645485568791628,
+            -0.004508861340582371,
+            -0.015438842587172985,
+            0.008032265119254589,
+            -0.009698158130049706,
+            0.02132030576467514,
+            -0.014085792936384678,
+            -0.021977277472615242,
+            0.0031362592708319426,
+            0.01474276464432478,
+            -0.0020589034538716078,
+            0.0001868508552433923,
+            0.012748385779559612,
+            -0.0062451446428895,
+            0.00031675436184741557,
+            0.012568499892950058,
+            0.00445802416652441,
+            -0.02810901589691639,
+            -0.009612126275897026,
+            0.007273618597537279,
+            -0.0021625328809022903,
+            -0.003701333422213793,
+            -0.02388562448322773,
+            -0.004430650733411312,
+            -0.011324945837259293,
+            -0.005944032687693834,
+            -0.004371992312371731,
+            -8.847613935358822e-05,
+            0.009361851029098034,
+            -0.02269681915640831,
+            0.014461205340921879,
+            -0.013327146880328655,
+            -0.004555787891149521,
+            -0.017660032957792282,
+            0.022290121763944626,
+            0.00530661316588521,
+            0.0007527804118581116,
+            0.00507589103654027,
+            0.0012396436650305986,
+            0.008665774017572403,
+            0.01452377438545227,
+            -0.002537945518270135,
+            0.004778689239174128,
+            0.015665654093027115,
+            0.005345718469470739,
+            0.005760236643254757,
+            0.01938067376613617,
+            -0.003339607734233141,
+            -0.017769528552889824,
+            -0.003564464161172509,
+            0.016424300149083138,
+            0.011231092736124992,
+            0.015493590384721756,
+            0.004645730834454298,
+            0.01033166702836752,
+            -0.003564464161172509,
+            -0.006909937597811222,
+            0.018129298463463783,
+            0.0125841423869133,
+            -0.016205308958888054,
+            -0.011332767084240913,
+            0.012294761836528778,
+            0.01033166702836752,
+            0.011786391027271748,
+            0.003580106422305107,
+            -0.0027060990687459707,
+            0.014578522183001041,
+            0.017018703743815422,
+            0.006647930946201086,
+            0.020162783563137054,
+            -0.001665893360041082,
+            -0.03572676330804825,
+            -0.004684836138039827,
+            0.0011809853604063392,
+            -0.00016595386841800064,
+            0.009526094421744347,
+            0.00445802416652441,
+            0.008290360681712627,
+            0.005795431789010763,
+            0.01958402246236801,
+            -0.006628378294408321,
+            0.010589763522148132,
+            0.00020762563508469611,
+            -0.02056948095560074,
+            -0.0020862771198153496,
+            -0.0040708803571760654,
+            -0.032191626727581024,
+            -0.00019369431538507342,
+            0.006080901715904474,
+            0.023087874054908752,
+            -0.009479167871177197,
+            0.0029055369086563587,
+            -0.006057438440620899,
+            -0.013162903487682343,
+            -0.00220554880797863,
+            0.014101434499025345,
+            -0.0012249790597707033,
+            0.01037859357893467,
+            0.0030678247567266226,
+            0.0005034829955548048,
+            0.0036387646105140448,
+            0.01966223306953907,
+            -0.00021385807485785335,
+            0.00868923682719469,
+            0.016877923160791397,
+            0.020256636664271355,
+            8.627645729575306e-05,
+            -0.01222437247633934,
+            0.027983879670500755,
+            -0.01428914163261652,
+            -0.008697058074176311,
+            -0.005603814963251352,
+            -0.005290971137583256,
+            0.015305883251130581,
+            -0.008478067815303802,
+            0.010589763522148132,
+            0.006628378294408321,
+            -0.018035445362329483,
+            0.01102774403989315,
+            -0.011176344938576221,
+            0.02361970767378807,
+            0.01689356565475464,
+            -0.0017186857294291258,
+            0.004282049834728241,
+            0.007293171249330044,
+            -0.03466309607028961,
+            0.009792011231184006,
+            -0.010957354679703712,
+            -0.0010734453098848462,
+            -0.0030111216474324465,
+            -0.004939021542668343,
+            0.016768427565693855,
+            -0.00014383482630364597,
+            -0.022086773067712784,
+            -0.00312257232144475,
+            0.021429801359772682,
+            -0.0013638035161420703,
+            0.01010485552251339,
+            0.006448493339121342,
+            -0.0020256636198610067,
+            -0.01618966832756996,
+            -0.020788472145795822,
+            -0.018129298463463783,
+            -0.020240994170308113,
+            -0.014367352239787579,
+            -0.00849370937794447,
+            -0.004794331267476082,
+            0.0045948936603963375,
+            0.0026572172064334154,
+            -0.008923869580030441,
+            -0.020929250866174698,
+            -0.014320425689220428,
+            0.003934010863304138,
+            0.0020354401785880327,
+            0.007723331451416016,
+            -0.0071993181481957436,
+            -0.007954053580760956,
+            -0.002113651018589735,
+            -0.0005504096043296158,
+            -0.006972506642341614,
+            0.01295173354446888,
+            -0.02463645115494728,
+            -0.005658562295138836,
+            0.01947452686727047,
+            0.011434441432356834,
+            -0.004536235239356756,
+            0.010316024534404278,
+            -0.026263238862156868,
+            0.02286888286471367,
+            -0.018410857766866684,
+            -0.015470126643776894,
+            -0.0079423226416111,
+            0.013632168993353844,
+            -0.015313704498112202,
+            0.005427840165793896,
+            0.006769157946109772,
+            -0.004395455587655306,
+            0.016408657655119896,
+            0.009181966073811054,
+            -0.011066850274801254,
+            0.009987538680434227,
+            -0.009416598826646805,
+            0.0015475992113351822,
+            -0.01645558513700962,
+            0.0009624835220165551,
+            0.015423200093209743,
+            0.010495910421013832,
+            -0.005267507862299681,
+            -0.010988638736307621,
+            -0.005705488845705986,
+            0.00047635359806008637,
+            0.0004220947448629886,
+            0.007500430569052696,
+            0.004008311312645674,
+            -0.002915313234552741,
+            -0.016064530238509178,
+            -0.017706960439682007,
+            0.0262006688863039,
+            -0.0051541016437113285,
+            -0.009103755466639996,
+            0.02047562785446644,
+            0.02150801196694374,
+            0.01615838333964348,
+            -0.017534896731376648,
+            -0.0005655629793182015,
+            0.009612126275897026,
+            0.014031045138835907,
+            0.007351829670369625,
+            -0.003501895349472761,
+            -0.00719149736687541,
+            -0.021523654460906982,
+            0.030095575377345085,
+            0.007261887192726135,
+            0.0075942836701869965,
+            -0.031284380704164505,
+            0.023385075852274895,
+            0.00021129178639966995,
+            -0.010409877635538578,
+            0.0026787251699715853,
+            0.01966223306953907,
+            0.005388734862208366,
+            0.0008539658738300204,
+            0.02297837845981121,
+            0.015086892992258072,
+            0.004931200761348009,
+            -0.007680315524339676,
+            -0.013217651285231113,
+            0.004551877733319998,
+            -0.005392645485699177,
+            0.009557378478348255,
+            -0.012028845027089119,
+            -0.009033365175127983,
+            0.0037658573128283024,
+            -0.003056093119084835,
+            0.008212150074541569,
+            -0.0002551632351242006,
+            -0.0211169570684433,
+            -0.006487598642706871,
+            -0.0363837368786335,
+            0.0050172326155006886,
+            -0.024464385583996773,
+            -0.005693757440894842,
+            -0.014234393835067749,
+            -0.013069050386548042,
+            -0.03932446613907814,
+            -0.010667974129319191,
+            0.008282540366053581,
+            0.015548337250947952,
+            0.0036446305457502604,
+            -0.011543937027454376,
+            0.0037287073209881783,
+            -0.040763549506664276,
+            0.0003023342287633568,
+            0.02848443016409874,
+            0.014492489397525787,
+            0.023494569584727287,
+            -0.015321525745093822,
+            0.022665534168481827,
+            0.0005333009175956249,
+            -0.02848443016409874,
+            -0.0029896136838942766,
+            -0.0006017355481162667,
+            0.011950633488595486,
+            0.00093902024673298,
+            -0.009721621870994568,
+            -0.017675675451755524,
+            0.020741544663906097,
+            0.006264697294682264,
+            0.009502630680799484,
+            -0.005185386165976524,
+            0.001369669334962964,
+            0.00380300753749907,
+            0.005204938817769289,
+            -0.014953934587538242,
+            0.0005670294049195945,
+            0.010581942275166512,
+            0.016095813363790512,
+            -0.006808263715356588,
+            -0.01263889018446207,
+            -0.008955154567956924,
+            -0.008196507580578327,
+            -0.00213711429387331,
+            -0.012576321139931679,
+            -0.010370772331953049,
+            -0.011606505140662193,
+            -0.0028957605827599764,
+            -0.01469583809375763,
+            -0.01618966832756996,
+            0.008188687264919281,
+            -0.0009399979026056826,
+            0.02316608466207981,
+            0.013984118588268757,
+            0.018676776438951492,
+            0.008141760714352131,
+            0.01681535504758358,
+            0.005142370238900185,
+            0.004305513110011816,
+            -0.014343888498842716,
+            -0.002422584220767021,
+            -0.030533555895090103,
+            0.013647811487317085,
+            0.02984529919922352,
+            -0.007003790698945522,
+            -0.00305413780733943,
+            0.010050107724964619,
+            0.0022466096561402082,
+            -0.002281804569065571,
+            0.0019415869610384107,
+            -0.01645558513700962,
+            -0.005236223340034485,
+            -0.01010485552251339,
+            0.006483688019216061,
+            0.009346208535134792,
+            0.028906768187880516,
+            -0.005552977789193392,
+            -0.0019083472434431314,
+            0.02241525985300541,
+            0.014359530992805958,
+            0.02537163347005844,
+            0.009471346624195576,
+            -0.006604915019124746,
+            0.00042062829015776515,
+            -0.0065384358167648315,
+            -0.009095934219658375,
+            -0.026122458279132843,
+            0.031034106388688087,
+            -0.004477577283978462,
+            -0.00877526868134737,
+            -0.009267997927963734,
+            -0.011590863578021526,
+            0.0025105716194957495,
+            0.02020971104502678,
+            -0.015947213396430016,
+            0.0011233048280701041,
+            0.0001070512444130145,
+            -0.016064530238509178,
+            -0.02712355926632881,
+            0.03189442679286003,
+            0.005979227367788553,
+            -0.002801907481625676,
+            -3.3239655749639496e-05,
+            -0.007426130119711161,
+            -0.011387514881789684,
+            -0.009346208535134792,
+            -0.0056976680643856525,
+            0.06563463062047958,
+            -0.01466455403715372,
+            0.002569229807704687,
+            -0.02305658906698227,
+            0.0031499462202191353,
+            -0.007766347844153643,
+            -0.00024954183027148247,
+            0.0075395358726382256,
+            -0.011004281230270863,
+            -0.0030736904591321945,
+            -0.01310033444315195,
+            -0.002369791967794299,
+            0.004160822834819555,
+            -0.019771728664636612,
+            -0.009236713871359825,
+            -0.0034745216835290194,
+            -0.011254556477069855,
+            -0.005283149890601635,
+            0.012153982184827328,
+            0.011309304274618626,
+            0.013890265487134457,
+            -0.02537163347005844,
+            -0.003361115697771311,
+            0.004852989688515663,
+            -0.008900406770408154,
+            -0.004012221936136484,
+            -0.017769528552889824,
+            -0.010065749287605286,
+            -0.009158502332866192,
+            -0.011066850274801254,
+            0.014774048700928688,
+            -0.009526094421744347,
+            0.012200908735394478,
+            0.006956864148378372,
+            0.0065970937721431255,
+            0.01958402246236801,
+            0.014492489397525787,
+            0.011692537926137447,
+            0.013037766329944134,
+            0.005658562295138836,
+            -0.01442992128431797,
+            0.019709160551428795,
+            -0.02213370054960251,
+            -0.022383974865078926,
+            -0.010018822737038136,
+            0.0033063681330531836,
+            -0.009893685579299927,
+            -0.007926680147647858,
+            -0.011332767084240913,
+            0.015650011599063873,
+            0.0026161563582718372,
+            -0.007649031467735767,
+            -0.012388614937663078,
+            -0.0024890636559575796,
+            0.0291257593780756,
+            -0.007649031467735767,
+            0.02039741724729538,
+            -0.012153982184827328,
+            0.009533915668725967,
+            -0.015086892992258072,
+            -0.00813393946737051,
+            -0.009713800624012947,
+            -0.008282540366053581,
+            0.014907008036971092,
+            -0.006319445092231035,
+            0.005298791918903589,
+            -0.017394116148352623,
+            -0.02692021057009697,
+            -0.0003842113073915243,
+            0.00380300753749907,
+            -0.028140300884842873,
+            0.007578641641885042,
+            -0.02316608466207981,
+            0.004802152514457703,
+            -0.011692537926137447,
+            -0.014672375284135342,
+            -0.01654943823814392,
+            0.024057690054178238,
+            0.002991568995639682,
+            -0.004188196733593941,
+            0.00126995041500777,
+            -0.00521667068824172,
+            -0.006354639772325754,
+            0.009330566972494125,
+            0.006964685395359993,
+            0.0049585746601223946,
+            0.015000861138105392,
+            -0.006980327423661947,
+            -0.011450083926320076,
+            -0.00256140879355371,
+            -0.0035879274364560843,
+            -0.017597464844584465,
+            0.0015700849471613765,
+            0.002289625583216548,
+            -0.006741784047335386,
+            0.005201028194278479,
+            0.013530494645237923,
+            -0.0049703060649335384,
+            0.006530614569783211,
+            0.018692417070269585,
+            0.0022270570043474436,
+            -0.005728952586650848,
+            -0.0055177826434373856,
+            -0.003934010863304138,
+            -0.002438226481899619,
+            0.014062329195439816,
+            0.010464625433087349,
+            0.015266777947545052,
+            -0.007273618597537279,
+            0.009385314770042896,
+            -0.00436417106539011,
+            0.001194672309793532,
+            -0.028140300884842873,
+            0.00799315981566906,
+            0.005052427761256695,
+            -0.0028351470828056335,
+            0.00530661316588521,
+            0.0017069540917873383,
+            0.04270318150520325,
+            0.008814374916255474,
+            0.013327146880328655,
+            -0.003323965473100543,
+            0.009674695320427418,
+            0.0027354280464351177,
+            0.012615426443517208,
+            0.0006882564048282802,
+            -0.02105438895523548,
+            0.011567399837076664,
+            -0.015047787688672543,
+            0.0027021884452551603,
+            0.017034346237778664,
+            -0.019177325069904327,
+            0.015438842587172985,
+            -0.01025345642119646,
+            -0.0001107173811760731,
+            -0.004002445377409458,
+            -0.022743744775652885,
+            -0.01001100242137909,
+            -0.00019137242634315044,
+            0.023932551965117455,
+            -0.0017978743417188525,
+            0.025747045874595642,
+            0.006417208816856146,
+            0.02491801045835018,
+            -0.00804008636623621,
+            -0.017253335565328598,
+            -0.005607725586742163,
+            0.005103264935314655,
+            -0.003548822132870555,
+            0.0057524158619344234,
+            0.008282540366053581,
+            0.0060183326713740826,
+            -0.001099841552786529,
+            -0.0007962852832861245,
+            -0.000347061111824587,
+            0.022900167852640152,
+            0.009291461668908596,
+            0.01056629978120327,
+            -0.01037859357893467,
+            0.013084692880511284,
+            -0.018051087856292725,
+            0.00021336926147341728,
+            0.0010089213028550148,
+            -0.022649891674518585,
+            0.0014077972155064344,
+            -0.004852989688515663,
+            -0.004371992312371731,
+            0.0026669935323297977,
+            0.019505811855196953,
+            -0.019427601248025894,
+            -0.014555058442056179,
+            0.02590346708893776,
+            0.007426130119711161,
+            0.0025555428583174944,
+            -0.002262251917272806,
+            0.009831116534769535,
+            0.00873616337776184,
+            0.011614326387643814,
+            -0.004720031283795834,
+            -0.005846268963068724,
+            -0.017753886058926582,
+            0.014164003543555737,
+            -0.006534525193274021,
+            -0.010308203287422657,
+            -0.008923869580030441,
+            -0.008345108479261398,
+            0.01617402583360672,
+            0.0007835759897716343,
+            0.007668584119528532,
+            0.002807773184031248,
+            0.02637273445725441,
+            -0.01281877513974905,
+            -0.011332767084240913,
+            -0.009408777579665184,
+            0.008556278422474861,
+            0.024464385583996773,
+            0.007957964204251766,
+            0.013202008791267872,
+            -0.01618966832756996,
+            -0.0036993781104683876,
+            0.019271178171038628,
+            -0.011872422881424427,
+            -0.033224012702703476,
+            -0.007500430569052696
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "f7811228-c68b-419a-91a6-f7967bab1c6c",
+      "type": "next",
+      "source": {
+        "id": "3080f374-964f-4b6f-bb07-d9220ba99e16",
+        "properties": {
+          "page_content": "7. Deepseekmoe 145B Ongoing\n\nEncouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available.\n\n7.1. Experimental Setup\n\nTraining Data and Tokenization. For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study.\n\nModel Settings. For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps.\n\nSubsequently, the learning rate keeps constant during the remaining training process. The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. Also, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse.\n\nEvaluation Benchmarks. We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3).\n\n7.2. Evaluations\n\nBaselines. Apart from DeepSeekMoE 145B, we consider three additional models for comparison. DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters.\n\nResults. From the evaluation results presented in Table 6, we have the following observations: (1) Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. (2) On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. (3) At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5.\n\n",
+          "keyphrases": [
+            "DeepSeekMoE 145B",
+            "GShard architecture",
+            "Transformer layers",
+            "multi-head attention mechanism",
+            "expert parallelism"
+          ],
+          "title": "Deepseekmoe 145B Ongoing",
+          "embedding": [
+            0.011145060881972313,
+            0.0028371023945510387,
+            -0.01864549145102501,
+            -0.008063549175858498,
+            0.004066969733685255,
+            0.007203228771686554,
+            0.0034158634953200817,
+            0.061411239206790924,
+            -0.013921549543738365,
+            0.03988758474588394,
+            0.002668948844075203,
+            -0.009103755466639996,
+            0.020194068551063538,
+            -0.01502432394772768,
+            -0.007441772148013115,
+            0.021367233246564865,
+            0.03610217571258545,
+            0.012208729982376099,
+            0.007637299597263336,
+            -0.028171585872769356,
+            -0.0256375502794981,
+            -0.05227620154619217,
+            -0.012240014038980007,
+            -0.0378853864967823,
+            0.011450083926320076,
+            0.01580643467605114,
+            0.029986079782247543,
+            0.0068551902659237385,
+            -0.014132719486951828,
+            0.018520353361964226,
+            -0.006624467670917511,
+            0.03178493306040764,
+            -0.013851160183548927,
+            -0.0013999760849401355,
+            0.0002678725286386907,
+            -0.01470365934073925,
+            0.05349629372358322,
+            0.010644510388374329,
+            -0.018989618867635727,
+            0.0074926093220710754,
+            0.04495565593242645,
+            0.005001590587198734,
+            -0.02932910807430744,
+            0.016236593946814537,
+            0.029907869175076485,
+            -0.009471346624195576,
+            -0.025668835267424583,
+            6.886230403324589e-05,
+            -0.0020589034538716078,
+            -0.024151543155312538,
+            -0.01281095389276743,
+            -0.013452284038066864,
+            0.004950753413140774,
+            -0.008759627118706703,
+            -0.03494465351104736,
+            -0.008282540366053581,
+            0.038104377686977386,
+            0.06757426261901855,
+            0.05612418055534363,
+            -0.026216311380267143,
+            0.031925711780786514,
+            -0.015743864700198174,
+            -0.030079932883381844,
+            0.011356230825185776,
+            0.0009101799805648625,
+            -0.01856728084385395,
+            -0.017018703743815422,
+            0.0004423806967679411,
+            -0.00746132479980588,
+            0.02050691284239292,
+            -0.046363454312086105,
+            0.026998421177268028,
+            -0.016596363857388496,
+            0.01866113394498825,
+            0.030392777174711227,
+            -0.00615520216524601,
+            0.03419382870197296,
+            0.007711600046604872,
+            -0.0013344744220376015,
+            -0.026623008772730827,
+            0.02580961398780346,
+            0.04971088096499443,
+            -0.015008682385087013,
+            0.010143960826098919,
+            0.06569720059633255,
+            -0.01866113394498825,
+            -0.027326907962560654,
+            -0.06444582343101501,
+            0.011090313084423542,
+            -0.012138339690864086,
+            -0.02261860854923725,
+            -0.0006212882581166923,
+            0.012756206095218658,
+            -0.022383974865078926,
+            -0.00839985627681017,
+            0.0512751005589962,
+            -0.031206170096993446,
+            -0.009651231579482555,
+            -0.017691317945718765,
+            0.019365031272172928,
+            0.008118296973407269,
+            -0.015853360295295715,
+            0.015962855890393257,
+            -0.0033376524224877357,
+            0.01507125049829483,
+            -0.005983137991279364,
+            -0.030721262097358704,
+            0.032942455261945724,
+            -0.006495419889688492,
+            0.08321645110845566,
+            -0.020741544663906097,
+            -0.001445924979634583,
+            -0.0033435183577239513,
+            0.01900526136159897,
+            -0.008524994365870953,
+            -0.007121107541024685,
+            -0.03472566232085228,
+            -0.01020652987062931,
+            0.021836498752236366,
+            -0.021429801359772682,
+            0.05143152177333832,
+            -0.00858756247907877,
+            0.016690216958522797,
+            -0.036352451890707016,
+            -0.054278403520584106,
+            0.050023727118968964,
+            0.008141760714352131,
+            0.002275938866659999,
+            0.000169253398780711,
+            -0.0024245395325124264,
+            -0.016393015161156654,
+            0.0008818284841254354,
+            -0.03838593512773514,
+            -0.00615520216524601,
+            0.003464745357632637,
+            0.013491389341652393,
+            0.06172408536076546,
+            0.062068212777376175,
+            -0.01295173354446888,
+            0.01028474047780037,
+            -0.019724803045392036,
+            -0.020991820842027664,
+            0.019803013652563095,
+            -0.002023708540946245,
+            -0.01618966832756996,
+            0.011457905173301697,
+            0.0026865461841225624,
+            0.0422339141368866,
+            -0.011700358241796494,
+            0.003171454183757305,
+            -0.00040083113708533347,
+            0.02399512007832527,
+            0.01683099754154682,
+            0.04063841328024864,
+            -0.00606525968760252,
+            0.00877526868134737,
+            -0.019224252551794052,
+            -0.006995969917625189,
+            -0.052057210355997086,
+            0.015915928408503532,
+            -0.003816694486886263,
+            -0.008337287232279778,
+            -0.01634608954191208,
+            0.002739338669925928,
+            -0.004301602486521006,
+            0.035977039486169815,
+            0.02426103875041008,
+            0.012505931779742241,
+            -0.0752076506614685,
+            0.023228652775287628,
+            0.020553838461637497,
+            0.017347188666462898,
+            -0.048177946358919144,
+            0.03757254034280777,
+            0.0016443852800875902,
+            0.015681296586990356,
+            0.008329466916620731,
+            0.0012396436650305986,
+            -0.044236116111278534,
+            0.04088868573307991,
+            -0.009948433376848698,
+            0.006557988468557596,
+            0.06360115110874176,
+            0.009455704130232334,
+            0.021648792549967766,
+            0.0022270570043474436,
+            0.052057210355997086,
+            0.012826596386730671,
+            -0.0038694869726896286,
+            0.033755846321582794,
+            -0.0005205916822887957,
+            0.005908837541937828,
+            0.028234153985977173,
+            0.0011770748533308506,
+            0.009181966073811054,
+            -0.02388562448322773,
+            0.007046807091683149,
+            0.014625448733568192,
+            -0.00258291675709188,
+            -0.018535995855927467,
+            -0.01011267676949501,
+            -0.031378235667943954,
+            -0.01828572154045105,
+            0.009486989118158817,
+            0.0248867254704237,
+            0.00891604833304882,
+            0.003511671908199787,
+            0.029157044366002083,
+            0.024511313065886497,
+            -0.017613107338547707,
+            0.009033365175127983,
+            -0.0015779059613123536,
+            -0.02821851149201393,
+            -0.048553358763456345,
+            0.00512672821059823,
+            0.022290121763944626,
+            -0.010988638736307621,
+            0.015423200093209743,
+            0.019505811855196953,
+            -0.015321525745093822,
+            -0.020194068551063538,
+            0.016142740845680237,
+            0.010542836971580982,
+            -0.02518392726778984,
+            -0.033411718904972076,
+            -0.014359530992805958,
+            0.014758407138288021,
+            -0.003525358624756336,
+            0.05446610972285271,
+            -0.025574982166290283,
+            -0.006084812339395285,
+            0.026795072481036186,
+            0.00516583351418376,
+            0.026138100773096085,
+            0.023197369650006294,
+            -0.04229648411273956,
+            0.024495670571923256,
+            0.0007200295804068446,
+            0.002708054380491376,
+            0.009463525377213955,
+            0.003918368835002184,
+            -0.01892705075442791,
+            -0.03281731531023979,
+            -0.015368452295660973,
+            0.02729562297463417,
+            -0.03203520551323891,
+            -0.025105716660618782,
+            0.02580961398780346,
+            0.010190887376666069,
+            0.014633269049227238,
+            0.006882563699036837,
+            -0.03963731229305267,
+            -0.007930590771138668,
+            0.002023708540946245,
+            -0.007789811119437218,
+            0.05969059839844704,
+            0.04886620491743088,
+            0.0029661504086107016,
+            -0.01457070093601942,
+            -0.030580483376979828,
+            0.00900990143418312,
+            -0.016095813363790512,
+            -0.03538263589143753,
+            0.029454244300723076,
+            -0.026341449469327927,
+            0.04339143633842468,
+            0.025668835267424583,
+            0.017049988731741905,
+            -0.017706960439682007,
+            -0.013577421195805073,
+            -0.012983018532395363,
+            -0.03191006928682327,
+            -0.03363071009516716,
+            0.008008801378309727,
+            0.001456678961403668,
+            -0.022947093471884727,
+            -0.011410977691411972,
+            -0.018332647159695625,
+            0.012099234387278557,
+            0.03813566267490387,
+            0.024667734280228615,
+            0.009948433376848698,
+            0.004305513110011816,
+            0.024307964369654655,
+            -0.035883184522390366,
+            -0.0030443614814430475,
+            -0.0018663088558241725,
+            -0.009846759028732777,
+            -0.015032145194709301,
+            -0.017534896731376648,
+            0.001207381603308022,
+            -0.011551758274435997,
+            -0.035069793462753296,
+            0.008712700568139553,
+            -0.0028762079309672117,
+            0.010229992680251598,
+            -0.012709279544651508,
+            0.015274599194526672,
+            0.022743744775652885,
+            -0.023823056370019913,
+            -0.049022626131772995,
+            0.014437741599977016,
+            0.020710259675979614,
+            -0.02784309908747673,
+            -0.015751685947179794,
+            -0.009862401522696018,
+            -0.0019249670440331101,
+            0.02407333068549633,
+            -0.04539363831281662,
+            -0.021257737651467323,
+            0.02471466176211834,
+            -0.019646592438220978,
+            -0.011465725488960743,
+            -0.026310164481401443,
+            -0.005298791918903589,
+            0.04192107170820236,
+            0.033755846321582794,
+            0.03907419368624687,
+            -0.06350729614496231,
+            0.002970061032101512,
+            0.019521454349160194,
+            0.03366199508309364,
+            -0.058752067387104034,
+            0.027702320367097855,
+            -0.00732445577159524,
+            -0.011583042331039906,
+            0.021914709359407425,
+            0.043766848742961884,
+            -0.023948194459080696,
+            -0.0025340348947793245,
+            0.017941592261195183,
+            -0.03485080227255821,
+            -0.02701406367123127,
+            -0.0019709160551428795,
+            0.005580351687967777,
+            -0.044236116111278534,
+            -0.03519492968916893,
+            0.00030037894612178206,
+            0.014343888498842716,
+            0.0040161325596272945,
+            -0.008650131523609161,
+            -0.03538263589143753,
+            -0.050899688154459,
+            0.02390126697719097,
+            0.0010548701975494623,
+            -0.004340707790106535,
+            -0.04160822555422783,
+            -0.014828796498477459,
+            -0.0064132981933653355,
+            0.01023781392723322,
+            0.0010734453098848462,
+            0.023275580257177353,
+            0.013155082240700722,
+            0.0010812664404511452,
+            -0.00877526868134737,
+            -0.04132666811347008,
+            -0.032754749059677124,
+            -0.01065233163535595,
+            0.007742884568870068,
+            0.029907869175076485,
+            0.024229753762483597,
+            -0.048021525144577026,
+            -0.018989618867635727,
+            0.03123745508491993,
+            -0.03869878128170967,
+            -0.014547237195074558,
+            0.04045070707798004,
+            0.018520353361964226,
+            0.0262006688863039,
+            0.00544739281758666,
+            0.00310106435790658,
+            0.043891988694667816,
+            0.0001796407887013629,
+            0.006757426541298628,
+            0.021523654460906982,
+            0.0043133338913321495,
+            0.01830136403441429,
+            -0.0014752540737390518,
+            -0.0040943436324596405,
+            0.0452372170984745,
+            -0.0049585746601223946,
+            0.03050227276980877,
+            -0.000946841377299279,
+            -0.020084572955965996,
+            0.020929250866174698,
+            0.025309065356850624,
+            0.00022534531308338046,
+            0.009064649231731892,
+            -0.011833317577838898,
+            0.0026200669817626476,
+            -0.004583161789923906,
+            -0.03566419333219528,
+            -0.013913728296756744,
+            -0.011794212274253368,
+            0.013561779633164406,
+            0.02444874495267868,
+            -0.0017939637182280421,
+            -0.015196388587355614,
+            -0.034318964928388596,
+            -0.0126779954880476,
+            -0.007817184552550316,
+            0.04564391076564789,
+            -0.004066969733685255,
+            0.012803133577108383,
+            -0.008978617377579212,
+            0.02424539625644684,
+            0.01015178207308054,
+            -0.003208604408428073,
+            0.03181621432304382,
+            -0.02047562785446644,
+            0.008110475726425648,
+            -0.020632049068808556,
+            -0.026716861873865128,
+            -0.021007461473345757,
+            -0.0005797386984340847,
+            0.03747868910431862,
+            0.023213010281324387,
+            0.009494809433817863,
+            -0.048303086310625076,
+            0.02471466176211834,
+            -0.047646112740039825,
+            0.01894269324839115,
+            0.01892705075442791,
+            -0.00530661316588521,
+            -0.013381893746554852,
+            0.008462425321340561,
+            -0.016127098351716995,
+            0.006890384946018457,
+            0.01928682066500187,
+            -0.025121359154582024,
+            0.00408652238547802,
+            0.000701943296007812,
+            0.03628988191485405,
+            -0.014719301834702492,
+            -0.04070097953081131,
+            -0.012107055634260178,
+            0.003024808596819639,
+            0.029188327491283417,
+            0.009635589085519314,
+            -0.026388375088572502,
+            0.0011487234150990844,
+            0.002221191069111228,
+            -0.009135039523243904,
+            -0.019599664956331253,
+            -0.026044247671961784,
+            -0.009995359927415848,
+            -0.054528675973415375,
+            0.010276919230818748,
+            -0.016220951452851295,
+            -0.011082491837441921,
+            -0.05396555736660957,
+            0.03325529769062996,
+            -0.06601004302501678,
+            -0.01033166702836752,
+            0.04949188977479935,
+            -0.004141270183026791,
+            -0.006733963266015053,
+            -0.007735063321888447,
+            -0.014351709745824337,
+            -0.039293184876441956,
+            0.022649891674518585,
+            0.0009991449769586325,
+            0.04877234995365143,
+            -0.02178957127034664,
+            -0.003578151110559702,
+            0.032473187893629074,
+            -0.018348289653658867,
+            -0.016009781509637833,
+            0.0018839063122868538,
+            0.028046447783708572,
+            -0.016877923160791397,
+            0.025496771559119225,
+            0.0018457785481587052,
+            0.018864482641220093,
+            -0.001967005431652069,
+            0.025293422862887383,
+            -0.0033298314083367586,
+            -0.026591723784804344,
+            -0.0008060616673901677,
+            0.04273446649312973,
+            0.018066730350255966,
+            -0.023228652775287628,
+            -0.018082372844219208,
+            -0.022915810346603394,
+            -0.02546548657119274,
+            -0.00638201367110014,
+            -0.008861301466822624,
+            -0.0019112802110612392,
+            0.0251682847738266,
+            0.030705619603395462,
+            -0.011066850274801254,
+            0.007918858900666237,
+            0.00011420021473895758,
+            0.022853240370750427,
+            0.0001724307076074183,
+            -0.017315905541181564,
+            -0.009666874073445797,
+            -0.007879753597080708,
+            0.016971776261925697,
+            0.0254498440772295,
+            -0.039199329912662506,
+            -0.021727003157138824,
+            0.017753886058926582,
+            -0.012153982184827328,
+            0.0022505202796310186,
+            0.02277502976357937,
+            0.011684716679155827,
+            0.02279067225754261,
+            -0.019177325069904327,
+            -0.01700306124985218,
+            0.024323606863617897,
+            -0.018129298463463783,
+            0.00502896448597312,
+            -0.019224252551794052,
+            -0.017128199338912964,
+            0.023744845762848854,
+            -0.004293781239539385,
+            0.0253559909760952,
+            0.01231822557747364,
+            0.016299162060022354,
+            -0.016408657655119896,
+            0.006925580091774464,
+            -0.015344989486038685,
+            0.029172684997320175,
+            -0.04179593548178673,
+            0.014085792936384678,
+            0.029626309871673584,
+            -0.00041891742148436606,
+            0.049585744738578796,
+            -0.043766848742961884,
+            -0.012513753026723862,
+            0.0074339513666927814,
+            0.019161684438586235,
+            -0.012419898994266987,
+            0.0007356717833317816,
+            -0.006999880541115999,
+            0.007672494743019342,
+            -0.00431724451482296,
+            -0.03679043427109718,
+            -0.004954664036631584,
+            0.01919296756386757,
+            -0.008376393467187881,
+            -0.0028586103580892086,
+            -0.0051971180364489555,
+            -0.007871932350099087,
+            0.002614201046526432,
+            0.05578005313873291,
+            -0.04630088433623314,
+            0.03391227126121521,
+            0.00905682798475027,
+            0.013186367228627205,
+            -0.031206170096993446,
+            -0.011246735230088234,
+            -0.003703288733959198,
+            0.00615911278873682,
+            -0.028797272592782974,
+            0.011614326387643814,
+            0.01033948827534914,
+            0.00864231027662754,
+            -0.01892705075442791,
+            -0.03253575786948204,
+            0.006624467670917511,
+            -0.028969338163733482,
+            0.0020784561056643724,
+            -0.0056742047891020775,
+            -0.007390934973955154,
+            -0.0034354161471128464,
+            0.025011863559484482,
+            -0.010667974129319191,
+            0.0014820975484326482,
+            -0.004305513110011816,
+            0.008955154567956924,
+            -0.00822779256850481,
+            -0.002438226481899619,
+            0.00794623326510191,
+            0.005959674715995789,
+            0.016862282529473305,
+            -0.007109375670552254,
+            -0.018129298463463783,
+            -0.004399366211146116,
+            0.01466455403715372,
+            -0.022540396079421043,
+            0.0036817805375903845,
+            0.03391227126121521,
+            0.0363837368786335,
+            0.0018242704682052135,
+            -0.010441162623465061,
+            -0.04254676029086113,
+            0.03409997746348381,
+            -0.013765127398073673,
+            0.01883319765329361,
+            0.027514614164829254,
+            -0.009776369668543339,
+            -0.04201492294669151,
+            -0.0014908963348716497,
+            -0.01442210003733635,
+            -0.011199808679521084,
+            0.017034346237778664,
+            -0.00836857222020626,
+            -0.02435489185154438,
+            -0.03162850812077522,
+            0.026341449469327927,
+            0.02727998048067093,
+            -0.00629207119345665,
+            -0.00733227701857686,
+            0.013452284038066864,
+            0.022118058055639267,
+            0.017112556844949722,
+            -0.016205308958888054,
+            0.01300648134201765,
+            -0.01231040433049202,
+            0.014304783195257187,
+            0.007840648293495178,
+            -0.009971896186470985,
+            0.004379813559353352,
+            -0.0043915449641644955,
+            0.006726142019033432,
+            -0.0004186730075161904,
+            -0.012232192791998386,
+            -0.01474276464432478,
+            0.03510107472538948,
+            0.004325065761804581,
+            0.006601004395633936,
+            0.005240133963525295,
+            0.028093375265598297,
+            0.035977039486169815,
+            -0.023635350167751312,
+            -0.0032281570602208376,
+            0.01866113394498825,
+            0.009189787320792675,
+            -0.03209777548909187,
+            -0.0107931112870574,
+            -0.020428700372576714,
+            -0.03014250099658966,
+            0.006796531844884157,
+            -0.007692047394812107,
+            0.013264577835798264,
+            0.009619947522878647,
+            0.026263238862156868,
+            -0.004149090964347124,
+            -0.0174410417675972,
+            0.03076818957924843,
+            -0.002999390009790659,
+            0.03907419368624687,
+            -0.003232067683711648,
+            -0.0027647572569549084,
+            -0.01093389093875885,
+            0.020147141069173813,
+            0.04066969454288483,
+            0.009041186422109604,
+            0.0033884895965456963,
+            0.011403157375752926,
+            0.022947093471884727,
+            0.044611528515815735,
+            -0.03632116690278053,
+            -0.016330447047948837,
+            -0.0011741418857127428,
+            -0.018410857766866684,
+            0.029172684997320175,
+            0.009158502332866192,
+            0.015032145194709301,
+            0.00863448902964592,
+            0.001260173972696066,
+            -0.0147505858913064,
+            0.03503850847482681,
+            -0.008783089928328991,
+            -0.015462305396795273,
+            0.02987658418715,
+            -0.02030356414616108,
+            0.03325529769062996,
+            0.009682515636086464,
+            -0.024558238685131073,
+            -0.024386174976825714,
+            0.0011389469727873802,
+            -0.0038851292338222265,
+            -0.027983879670500755,
+            0.031096674501895905,
+            0.029250897467136383,
+            -0.0049155582673847675,
+            0.006808263715356588,
+            0.009095934219658375,
+            0.02114824205636978,
+            -0.020240994170308113,
+            -0.016518153250217438,
+            0.017128199338912964,
+            -0.007171944249421358,
+            -0.022196268662810326,
+            0.03976244851946831,
+            -0.01856728084385395,
+            -0.011723821982741356,
+            -0.012302583083510399,
+            0.0032692179083824158,
+            0.011465725488960743,
+            0.017800813540816307,
+            0.03494465351104736,
+            -0.001723573892377317,
+            -0.028703419491648674,
+            0.021617507562041283,
+            0.026435302570462227,
+            -0.013170724734663963,
+            -0.019349390640854836,
+            0.025387275964021683,
+            -0.01855163834989071,
+            0.013358430936932564,
+            0.05161922797560692,
+            0.008251255378127098,
+            0.008955154567956924,
+            -0.024745944887399673,
+            -0.02178957127034664,
+            0.025387275964021683,
+            -0.017300263047218323,
+            0.04307859390974045,
+            0.013131619431078434,
+            0.02379177138209343,
+            0.007864111103117466,
+            -0.023040946573019028,
+            0.0011037520598620176,
+            0.0037756338715553284,
+            0.004927290137857199,
+            0.02369791828095913,
+            0.016846640035510063,
+            0.0032203360460698605,
+            0.005208849441260099,
+            -0.04514336213469505,
+            -0.014117076992988586,
+            -0.014062329195439816,
+            0.009768548421561718,
+            0.01747232675552368,
+            0.00155053217895329,
+            0.0033337417989969254,
+            0.005451303441077471,
+            -0.01506342925131321,
+            -0.033505573868751526,
+            0.01698741875588894,
+            -0.0031284382566809654,
+            -0.003015032270923257,
+            0.0047317626886069775,
+            0.011770748533308506,
+            -0.006909937597811222,
+            -0.0012777714291587472,
+            0.005025053862482309,
+            -0.011278019286692142,
+            -0.02269681915640831,
+            0.0009082247270271182,
+            -0.009698158130049706,
+            0.0358206182718277,
+            0.012552858330309391,
+            0.024964936077594757,
+            -0.002391299931332469,
+            0.01024563517421484,
+            -0.003580106422305107,
+            -0.016017602756619453,
+            0.012709279544651508,
+            -0.0076764049008488655,
+            -0.007246244698762894,
+            0.0022954915184527636,
+            0.027233054861426353,
+            -0.016393015161156654,
+            0.007261887192726135,
+            -0.013233293779194355,
+            0.020632049068808556,
+            0.046895287930965424,
+            0.002715875394642353,
+            0.0034667004365473986,
+            0.0422339141368866,
+            -0.009221071377396584,
+            0.0026591725181788206,
+            0.025793973356485367,
+            0.0017382384976372123,
+            -0.016705859452486038,
+            0.04282831773161888,
+            -0.0063350871205329895,
+            0.042390335351228714,
+            -0.044517673552036285,
+            -0.0013246979797258973,
+            0.0358206182718277,
+            -0.01025345642119646,
+            -0.008235613815486431,
+            -0.02599732019007206,
+            0.002567274495959282,
+            -0.020538195967674255,
+            0.021977277472615242,
+            -0.007359650917351246,
+            0.038730062544345856,
+            -0.03944960609078407,
+            0.004434560891240835,
+            0.015399736352264881,
+            -0.01216962467879057,
+            0.036164745688438416,
+            0.01737847365438938,
+            -0.031096674501895905,
+            -0.034600526094436646,
+            0.006663573440164328,
+            0.010175244882702827,
+            0.05446610972285271,
+            -0.01272492203861475,
+            0.008306003175675869,
+            0.027154842391610146,
+            -0.009979717433452606,
+            -0.00019626061839517206,
+            0.043516576290130615,
+            -0.010511551983654499,
+            0.02637273445725441,
+            0.03916804492473602,
+            -0.053621429949998856,
+            -0.02756153978407383,
+            0.019552739337086678,
+            -0.0008725409279577434,
+            0.011301483027637005,
+            0.010941712185740471,
+            0.014085792936384678,
+            -0.0028801183216273785,
+            -0.027342548593878746,
+            0.01938067376613617,
+            0.013585242442786694,
+            0.0015671519795432687,
+            -0.004997679963707924,
+            -0.016752786934375763,
+            -0.012380793690681458,
+            -0.01939631626009941,
+            -0.009979717433452606,
+            0.005025053862482309,
+            0.0025281691923737526,
+            -0.017425401136279106,
+            -0.04326630011200905,
+            -0.02929782308638096,
+            0.027045348659157753,
+            -0.0074691460467875,
+            0.03300502151250839,
+            0.023400716483592987,
+            0.023025304079055786,
+            0.0026024694088846445,
+            -0.025856541469693184,
+            0.03841722011566162,
+            -0.011817675083875656,
+            0.03372456505894661,
+            -0.0022251016926020384,
+            -0.015962855890393257,
+            -0.027968237176537514,
+            0.04310987889766693,
+            -0.01254503708332777,
+            0.006952953990548849,
+            -0.011739464476704597,
+            -0.0036153013352304697,
+            -0.004387634340673685,
+            -0.01066015288233757,
+            0.011184166185557842,
+            0.004974216688424349,
+            0.016877923160791397,
+            0.0008544546435587108,
+            0.021163884550333023,
+            0.017206409946084023,
+            -0.0030033006332814693,
+            0.018144940957427025,
+            -0.027858741581439972,
+            -0.02452695555984974,
+            0.014304783195257187,
+            -0.02684199996292591,
+            -0.027717962861061096,
+            -0.008783089928328991,
+            -0.04051327332854271,
+            -0.006941222120076418,
+            -0.0010157647775486112,
+            -0.010503730736672878,
+            0.035069793462753296,
+            0.03907419368624687,
+            0.015642190352082253,
+            -0.03575804829597473,
+            0.022446542978286743,
+            -0.04407969489693642,
+            -0.008798732422292233,
+            0.0030912880320101976,
+            0.011184166185557842,
+            -0.022305764257907867,
+            0.03278603032231331,
+            0.00886912178248167,
+            -0.008259076625108719,
+            -0.028500070795416832,
+            0.004899916239082813,
+            -0.01030038297176361,
+            -0.0003243310493417084,
+            -0.00799707043915987,
+            0.011794212274253368,
+            0.0257314033806324,
+            0.008118296973407269,
+            0.0005758281331509352,
+            0.002829281147569418,
+            -0.03779153153300285,
+            0.00258291675709188,
+            -0.00422339141368866,
+            0.0023639260325580835,
+            -0.009878043085336685,
+            -0.020147141069173813,
+            -0.005146280862390995,
+            0.002923134481534362,
+            0.003040450857952237,
+            -0.004020043183118105,
+            0.01873934455215931,
+            -0.015657832846045494,
+            0.009838937781751156,
+            0.005220581311732531,
+            -0.04154565930366516,
+            -0.01994379237294197,
+            -0.04526849836111069,
+            -0.0022075041197240353,
+            0.0002268117677886039,
+            0.022008562460541725,
+            -0.007723331451416016,
+            0.009596483781933784,
+            -0.020882325246930122,
+            -0.010832217521965504,
+            -0.0006887452327646315,
+            0.017315905541181564,
+            -0.028922410681843758,
+            0.00019540518405847251,
+            0.006601004395633936,
+            0.02986094169318676,
+            -0.04066969454288483,
+            0.007336187642067671,
+            0.029078831896185875,
+            -0.017988519743084908,
+            -0.004544056486338377,
+            -0.0060965437442064285,
+            0.028875483199954033,
+            -0.024745944887399673,
+            0.0034803873859345913,
+            -0.0065384358167648315,
+            0.006511061917990446,
+            0.01836393214762211,
+            0.01501650270074606,
+            -0.0008935601217672229,
+            0.015524874441325665,
+            0.02288452535867691,
+            -0.0020765007939189672,
+            0.03832336887717247,
+            -0.03963731229305267,
+            -0.03832336887717247,
+            0.028984978795051575,
+            -0.017597464844584465,
+            -0.0031382145825773478,
+            0.009408777579665184,
+            -0.052964456379413605,
+            -0.0026806804817169905,
+            0.011786391027271748,
+            -0.002029574243351817,
+            -0.05671858415007591,
+            0.027968237176537514,
+            0.03406869247555733,
+            0.02527778036892414,
+            0.0013012347044423223,
+            -0.012388614937663078,
+            -0.0006960774771869183,
+            0.02444874495267868,
+            0.012568499892950058,
+            0.0294073186814785,
+            0.014250035397708416,
+            -0.013749485835433006,
+            -0.0008559211273677647,
+            -0.014860081486403942,
+            -0.014719301834702492,
+            -0.017128199338912964,
+            0.016956135630607605,
+            -0.009893685579299927,
+            -0.01563436910510063,
+            0.006186486221849918,
+            -0.005009411368519068,
+            -0.0148444389924407,
+            -0.03935575112700462,
+            -0.03613346070051193,
+            0.018317004665732384,
+            -0.015313704498112202,
+            -0.04861592873930931,
+            0.0065384358167648315,
+            -0.009932790882885456,
+            -0.004446292761713266,
+            -0.007062449119985104,
+            0.010941712185740471,
+            -0.01598631963133812,
+            0.013436641544103622,
+            0.005990959238260984,
+            0.0004746427293866873,
+            0.0003773678618017584,
+            0.018410857766866684,
+            0.01873934455215931,
+            0.013843338936567307,
+            -0.02480851486325264,
+            0.03106539137661457,
+            -0.016612006351351738,
+            -0.05155666172504425,
+            0.020647691562771797,
+            -0.015352809801697731,
+            0.003449103096500039,
+            0.027702320367097855,
+            -0.010511551983654499,
+            -0.02914140187203884,
+            0.028155943378806114,
+            -0.031925711780786514,
+            -0.02343200147151947,
+            -0.022853240370750427,
+            -0.00033044128213077784,
+            -0.03876134753227234,
+            -0.0450495108962059,
+            -0.007042896468192339,
+            0.011262377724051476,
+            -0.00849370937794447,
+            0.0004577784857247025,
+            0.03463181108236313,
+            -0.00357228540815413,
+            -0.03475694730877876,
+            -0.007488698698580265,
+            -0.005251865368336439,
+            -0.009885864332318306,
+            -0.0026865461841225624,
+            0.00029206901672296226,
+            0.004720031283795834,
+            -0.006663573440164328,
+            -0.0262006688863039,
+            -0.022352689877152443,
+            -0.016064530238509178,
+            -0.007344008423388004,
+            -0.01956837996840477,
+            -0.02003764547407627,
+            -0.011098134331405163,
+            -0.03306759148836136,
+            0.03076818957924843,
+            0.026075532659888268,
+            -0.006542346440255642,
+            -0.009174144826829433,
+            -0.009526094421744347,
+            0.0020706350915133953,
+            0.02993915230035782,
+            0.006053527817130089,
+            0.005658562295138836,
+            -0.020272279158234596,
+            0.004829526413232088,
+            -0.011317124590277672,
+            0.011567399837076664,
+            -0.051775652915239334,
+            -0.017534896731376648,
+            0.039199329912662506,
+            0.02271246165037155,
+            -0.0005782722146250308,
+            -0.006409387569874525,
+            0.018535995855927467,
+            -0.029907869175076485,
+            -0.028343649581074715,
+            0.03876134753227234,
+            0.015376273542642593,
+            0.013905907981097698,
+            -0.005506051238626242,
+            0.003539045574143529,
+            0.012787491083145142,
+            0.0019386539934203029,
+            -0.01042552012950182,
+            -0.006022243294864893,
+            0.03363071009516716,
+            0.0017558359540998936,
+            0.0022172804456204176,
+            -0.00822779256850481,
+            0.023478928953409195,
+            0.01023781392723322,
+            -0.0019738490227609873,
+            0.01773824356496334,
+            0.01043334137648344,
+            -0.01709691435098648,
+            -0.011137239634990692,
+            -0.02473030425608158,
+            0.0035527325235307217,
+            0.02260296605527401,
+            0.010034465231001377,
+            -0.02379177138209343,
+            -0.0061982180923223495,
+            -0.006718320772051811,
+            0.00267676985822618,
+            -0.012052307836711407,
+            0.012302583083510399,
+            0.025105716660618782,
+            -0.030940253287553787,
+            -0.014867902733385563,
+            -0.008259076625108719,
+            0.005032874643802643,
+            0.01033948827534914,
+            0.02086668275296688,
+            0.019693518057465553,
+            -0.0020765007939189672,
+            -0.004626177717000246,
+            0.0063116238452494144,
+            0.02756153978407383,
+            0.016580721363425255,
+            -0.02555933967232704,
+            -0.020694619044661522,
+            0.012333867140114307,
+            0.009111575782299042,
+            0.034788232296705246,
+            -0.012279119342565536,
+            -0.00044995738426223397,
+            -0.01496957615017891,
+            0.09547993540763855,
+            -0.01636173203587532,
+            0.026998421177268028,
+            -0.016048887744545937,
+            -0.02857828326523304,
+            0.00028815848054364324,
+            0.003701333422213793,
+            0.011223271489143372,
+            0.05559234693646431,
+            -0.00151631492190063,
+            0.03076818957924843,
+            0.012044486589729786,
+            0.0022075041197240353,
+            0.0029837479814887047,
+            -0.00746132479980588,
+            0.016205308958888054,
+            -0.006421119440346956,
+            0.02351021207869053,
+            0.0110746705904603,
+            0.012920449487864971,
+            -0.009784189984202385,
+            0.02335379086434841,
+            -0.0007136749336495996,
+            0.020725902169942856,
+            -0.023213010281324387,
+            0.04016914591193199,
+            0.00014676773571409285,
+            0.0036368092987686396,
+            -0.004501040559262037,
+            -0.025215212255716324,
+            -0.0044150082394480705,
+            0.011090313084423542,
+            -0.014914829283952713,
+            0.011317124590277672,
+            -0.019318105652928352,
+            0.05318344756960869,
+            -0.02793695218861103,
+            0.022164983674883842,
+            -0.005373092368245125,
+            -0.03603960573673248,
+            0.014406457543373108,
+            0.028187228366732597,
+            0.008016622625291348,
+            0.0028957605827599764,
+            -0.013467926532030106,
+            0.012599784880876541,
+            0.022305764257907867,
+            -0.02030356414616108,
+            2.63045440078713e-05,
+            0.025934752076864243,
+            -0.016877923160791397,
+            0.02307223156094551,
+            0.009674695320427418,
+            -0.02279067225754261,
+            -0.005322255194187164,
+            -0.006843458395451307,
+            0.009095934219658375,
+            -0.005658562295138836,
+            -0.008618847467005253,
+            -0.01038641482591629,
+            0.010456804186105728,
+            0.022759387269616127,
+            0.027702320367097855,
+            0.019271178171038628,
+            -0.01037859357893467,
+            -0.0015642191283404827,
+            0.0007234513177536428,
+            -0.014711480587720871,
+            -0.002647440880537033,
+            -0.012240014038980007,
+            -0.01819186843931675,
+            0.02682635746896267,
+            0.0035058059729635715,
+            -0.003580106422305107,
+            -0.015251136384904385,
+            -0.013413178734481335,
+            -0.02105438895523548,
+            -0.0006515950080938637,
+            -0.0029641950968652964,
+            -0.015696939080953598,
+            -0.003108885372057557,
+            0.015501410700380802,
+            -0.015337168239057064,
+            0.018410857766866684,
+            -0.02482415735721588,
+            0.013718200847506523,
+            -0.027326907962560654,
+            -0.0035234035458415747,
+            0.012599784880876541,
+            -0.012466826476156712,
+            -0.0032574862707406282,
+            -0.01884884014725685,
+            -0.011723821982741356,
+            0.010190887376666069,
+            -0.01836393214762211,
+            -0.007449593394994736,
+            0.007738973945379257,
+            0.005408287514001131,
+            -0.017972877249121666,
+            -0.027170484885573387,
+            -0.022118058055639267,
+            0.005865821614861488,
+            0.04070097953081131,
+            -0.02020971104502678,
+            0.00018428456678520888,
+            -0.007211050018668175,
+            0.006569720339030027,
+            0.0013706469908356667,
+            -0.013772948645055294,
+            -0.0007038986077532172,
+            0.0034354161471128464,
+            -0.004211660008877516,
+            0.024057690054178238,
+            0.006565809715539217,
+            0.023948194459080696,
+            0.005728952586650848,
+            0.0012005381286144257,
+            -0.02066333405673504,
+            -0.016565080732107162,
+            0.016752786934375763,
+            -0.010097034275531769,
+            0.014351709745824337,
+            0.02435489185154438,
+            -0.021429801359772682,
+            0.0012924360344186425,
+            0.01424221508204937,
+            0.014038866385817528,
+            -0.02663865126669407,
+            0.024949293583631516,
+            0.006421119440346956,
+            0.014414278790354729,
+            -0.0021195169538259506,
+            0.02325993776321411,
+            -0.007848469540476799,
+            0.033505573868751526,
+            -0.011504831723868847,
+            -0.00804008636623621,
+            0.0010754006216302514,
+            0.030189428478479385,
+            -0.006370282266288996,
+            -0.00020762563508469611,
+            0.019224252551794052,
+            0.043516576290130615,
+            0.0035664194729179144,
+            -0.0016551392618566751,
+            -0.013452284038066864,
+            0.010362951084971428,
+            0.008259076625108719,
+            0.021382873877882957,
+            0.00450495071709156,
+            0.001804717699997127,
+            -0.008243434131145477,
+            0.017597464844584465,
+            0.006671394221484661,
+            0.011512652039527893,
+            -0.026325806975364685,
+            0.0015642191283404827,
+            0.0070389858447015285,
+            0.010073570534586906,
+            0.002948553068563342,
+            -0.0026885014958679676,
+            0.020632049068808556,
+            -0.010769648477435112,
+            -0.0360083244740963,
+            0.024464385583996773,
+            0.002373702358454466,
+            0.024182826280593872,
+            -0.007054627873003483,
+            -0.015869002789258957,
+            0.02103874646127224,
+            0.002291580894961953,
+            0.006503240671008825,
+            0.025574982166290283,
+            0.0017206409247592092,
+            -0.00269827782176435,
+            0.005623367615044117,
+            0.01005792897194624,
+            0.0031049747485667467,
+            0.0009292439208365977,
+            -0.003840157762169838,
+            0.007257976569235325,
+            0.010448983870446682,
+            0.006080901715904474,
+            -0.002289625583216548,
+            -0.028328007087111473,
+            -0.0009043141617439687,
+            0.009768548421561718,
+            0.0063585503958165646,
+            -0.039825018495321274,
+            0.01497739739716053,
+            -0.006827816367149353,
+            0.004035685211420059,
+            0.0074378615245223045,
+            -0.008478067815303802,
+            -0.012083591893315315,
+            0.01245900522917509,
+            0.0018506667111068964,
+            -0.006898206193000078,
+            0.0043602604418993,
+            -0.03660272806882858,
+            -0.01745668426156044,
+            0.023275580257177353,
+            0.003564464161172509,
+            -0.021429801359772682,
+            0.014117076992988586,
+            -0.007707689423114061,
+            -0.010143960826098919,
+            0.005322255194187164,
+            0.02757718227803707,
+            -0.044705379754304886,
+            0.013694738037884235,
+            -0.02352585457265377,
+            -0.012552858330309391,
+            0.021617507562041283,
+            0.010003181174397469,
+            0.0041295383125543594,
+            -0.00101087661460042,
+            -0.011426620185375214,
+            0.005904926918447018,
+            0.012889165431261063,
+            -0.011895885691046715,
+            -0.020162783563137054,
+            0.0217739287763834,
+            0.012928270734846592,
+            0.005944032687693834,
+            -0.0074339513666927814,
+            0.0215862225741148,
+            -0.010644510388374329,
+            1.3480696452461416e-06,
+            0.009275819174945354,
+            -0.0454249233007431,
+            0.030721262097358704,
+            -0.0043328870087862015,
+            0.02360406517982483,
+            -0.008024443872272968,
+            -0.028328007087111473,
+            -0.008978617377579212,
+            0.01645558513700962,
+            0.018692417070269585,
+            0.005717220716178417,
+            0.0061982180923223495,
+            0.032019563019275665,
+            -0.0129595547914505,
+            0.013819875195622444,
+            0.0057993424125015736,
+            -0.00016289875202346593,
+            0.0261068157851696,
+            -0.016752786934375763,
+            -0.03130002319812775,
+            0.01428914163261652,
+            -0.012513753026723862,
+            -0.03663400933146477,
+            0.015337168239057064,
+            -0.02103874646127224,
+            -0.03397483751177788,
+            -0.029986079782247543,
+            -0.0035136269871145487,
+            -0.007344008423388004,
+            -0.0014830752043053508,
+            -0.003116706619039178,
+            0.016924850642681122,
+            0.023932551965117455,
+            0.012005381286144257,
+            0.01525895670056343,
+            -0.006511061917990446,
+            0.01419528853148222,
+            -0.00397116132080555,
+            0.033224012702703476,
+            0.0012435541721060872,
+            0.004325065761804581,
+            0.001327630947344005,
+            0.017957234755158424,
+            0.013389714993536472,
+            0.020084572955965996,
+            0.009612126275897026,
+            -0.015368452295660973,
+            -0.01900526136159897,
+            -0.046738866716623306,
+            0.009760727174580097,
+            -0.004692657385021448,
+            0.006256876513361931,
+            0.03813566267490387,
+            -0.013186367228627205,
+            -0.010816575028002262,
+            0.009228892624378204,
+            0.004966395441442728,
+            0.0032437993213534355,
+            -0.013108155690133572,
+            -0.006100454367697239,
+            -0.0031499462202191353,
+            -0.02233704924583435,
+            -0.0253559909760952,
+            -0.018442142754793167,
+            0.020819755271077156,
+            -0.0017597464611753821,
+            0.013108155690133572,
+            -0.02077282965183258,
+            -0.0031577672343701124,
+            -0.028093375265598297,
+            -0.011191987432539463,
+            0.0013872667914256454,
+            0.013084692880511284,
+            0.004821705166250467,
+            0.016971776261925697,
+            -0.005330076441168785,
+            -0.020256636664271355,
+            -0.017800813540816307,
+            0.012075771577656269,
+            -0.010761827230453491,
+            -0.005947942845523357,
+            -0.004978127311915159,
+            0.01938067376613617,
+            0.007954053580760956,
+            0.004278139211237431,
+            -0.001422461704351008,
+            0.021899066865444183,
+            -0.021085673943161964,
+            -0.015955034643411636,
+            0.01747232675552368,
+            0.03262960910797119,
+            -0.03170672059059143,
+            -0.005552977789193392,
+            0.0009018700802698731,
+            0.014688016846776009,
+            0.015861181542277336,
+            0.012693637982010841,
+            -0.006557988468557596,
+            -0.008055727928876877,
+            -0.028046447783708572,
+            -0.01240425743162632,
+            -0.04095125570893288,
+            0.0029114028438925743,
+            0.029720162972807884,
+            -0.005025053862482309,
+            0.01911475695669651,
+            0.017988519743084908,
+            -0.027170484885573387,
+            6.660151848336682e-05,
+            0.004106075037270784,
+            0.017628749832510948,
+            -0.033849701285362244,
+            0.0145628796890378,
+            -0.009361851029098034,
+            0.02169571816921234,
+            0.0005733840516768396,
+            -0.01909911446273327,
+            -0.017753886058926582,
+            -0.011888065375387669,
+            0.021742645651102066,
+            -0.00253990083001554,
+            -0.029751446098089218,
+            0.021633150056004524,
+            0.009565199725329876,
+            -0.007136749569326639,
+            -0.024949293583631516,
+            0.012013202533125877,
+            0.0129595547914505,
+            0.017613107338547707,
+            -0.01074618473649025,
+            -0.002436271170154214,
+            -0.012568499892950058,
+            -0.010214350186288357,
+            -0.022211911156773567,
+            -0.00043211551383137703,
+            -0.017628749832510948,
+            0.0015945257619023323,
+            -0.011512652039527893,
+            -0.0022778939455747604,
+            0.014351709745824337,
+            -0.041483089327812195,
+            -0.0018457785481587052,
+            -0.008955154567956924,
+            0.028531355783343315,
+            0.02473030425608158,
+            0.008478067815303802,
+            0.0027960415463894606,
+            -0.02113259956240654,
+            -0.010816575028002262,
+            -0.011786391027271748,
+            0.01820751093327999,
+            -0.02261860854923725,
+            0.0003284859994892031,
+            0.010018822737038136,
+            0.008611026220023632,
+            0.0003871442168019712,
+            0.005114996340125799,
+            -0.002453868743032217,
+            -0.02094489336013794,
+            0.014641090296208858,
+            -0.0005171699449419975,
+            0.007003790698945522,
+            -0.012138339690864086,
+            0.0032262017484754324,
+            -0.0036778701469302177,
+            0.025668835267424583,
+            -0.014774048700928688,
+            -0.004387634340673685,
+            -0.010355129837989807,
+            0.0013110111467540264,
+            0.0070937336422502995,
+            -1.234266619576374e-05,
+            -0.0006423074519261718,
+            -0.010777469724416733,
+            0.01240425743162632,
+            -0.011160703375935555,
+            0.030439702793955803,
+            9.531959949526936e-05,
+            -0.03172236308455467,
+            -0.0036074803210794926,
+            -0.03766639530658722,
+            -0.007168034091591835,
+            -0.017800813540816307,
+            -0.033943552523851395,
+            -0.007430040743201971,
+            0.0017089092871174216,
+            -0.01493047084659338,
+            0.016283521428704262,
+            0.004559698514640331,
+            -0.02473030425608158,
+            0.013905907981097698,
+            0.006831726990640163,
+            -0.017894666641950607,
+            -0.0038440683856606483,
+            -0.01920861005783081,
+            0.0049116481095552444,
+            0.005142370238900185,
+            0.009987538680434227,
+            0.018348289653658867,
+            -0.007394845597445965,
+            -0.013022123835980892,
+            0.003024808596819639,
+            0.02096053585410118,
+            0.01483661774545908,
+            0.010824396274983883,
+            0.0068356371484696865,
+            0.00377758895047009,
+            -0.025043146684765816,
+            -0.003355249995365739,
+            -0.008149581030011177,
+            -0.0070389858447015285,
+            -0.0009810586925595999,
+            0.0034295502118766308,
+            -0.008337287232279778,
+            -0.0006525726639665663,
+            0.000982525059953332,
+            0.0018956379499286413,
+            0.0030072112567722797,
+            -0.013546137139201164,
+            0.0012865702155977488,
+            -0.008517173118889332,
+            0.002373702358454466,
+            0.024026405066251755,
+            -0.012185266241431236,
+            0.008650131523609161,
+            -0.0027510703075677156,
+            0.00634290836751461,
+            0.038636211305856705,
+            0.015352809801697731,
+            -0.044423822313547134,
+            0.003670048899948597,
+            0.007926680147647858,
+            0.0008823173120617867,
+            -0.017972877249121666,
+            0.017675675451755524,
+            0.004837347660213709,
+            -0.008650131523609161,
+            -0.002266162307932973,
+            -0.007844558916985989,
+            0.009385314770042896,
+            -0.015251136384904385,
+            0.012302583083510399,
+            0.013420999981462955,
+            -0.01438299473375082,
+            -0.004923379514366388,
+            0.038730062544345856,
+            -0.010941712185740471,
+            -0.011285840533673763,
+            0.0009121352341026068,
+            0.013319325633347034,
+            0.011262377724051476,
+            -0.004798241890966892,
+            0.014500310644507408,
+            0.0037365283351391554,
+            0.006186486221849918,
+            0.007218870799988508,
+            0.0597531683743,
+            0.0067769791930913925,
+            0.008251255378127098,
+            0.011364051140844822,
+            -0.007445682771503925,
+            -0.014500310644507408,
+            -0.018958335742354393,
+            -0.026357091963291168,
+            -0.004207749385386705,
+            0.01590810716152191,
+            -0.018270079046487808,
+            -0.0032653072848916054,
+            -0.02252475544810295,
+            0.01249811053276062,
+            0.013061229139566422,
+            0.005709399469196796,
+            -0.01717512495815754,
+            0.02066333405673504,
+            0.02793695218861103,
+            0.010308203287422657,
+            -0.0016746920300647616,
+            0.017034346237778664,
+            0.005036785267293453,
+            -0.015548337250947952,
+            -0.0023404627572745085,
+            -0.0023717472795397043,
+            0.004720031283795834,
+            0.021382873877882957,
+            -0.008251255378127098,
+            0.01221655122935772,
+            -0.0045166825875639915,
+            0.009221071377396584,
+            0.002291580894961953,
+            -0.013115976937115192,
+            -0.008306003175675869,
+            -0.0008432118338532746,
+            0.01792594976723194,
+            0.00013894664880353957,
+            -0.025011863559484482,
+            0.0030365402344614267,
+            0.0045948936603963375,
+            0.002862520981580019,
+            0.003812784096226096,
+            -0.014727122150361538,
+            -0.007938412018120289,
+            0.021914709359407425,
+            -0.016111455857753754,
+            0.0032066490966826677,
+            -0.012568499892950058,
+            0.0011369917774572968,
+            -0.01285788044333458,
+            0.0006750582833774388,
+            0.00397116132080555,
+            0.016612006351351738,
+            -0.027608467265963554,
+            -0.015352809801697731,
+            0.0217739287763834,
+            0.00836075097322464,
+            0.009049007669091225,
+            -0.0027256517205387354,
+            -0.005658562295138836,
+            0.016424300149083138,
+            -0.01783209666609764,
+            -0.013827696442604065,
+            0.005904926918447018,
+            -0.008024443872272968,
+            0.0052323127165436745,
+            -0.02296273596584797,
+            0.007969696074724197,
+            -0.008071370422840118,
+            -0.008165223523974419,
+            0.008212150074541569,
+            0.015829896554350853,
+            0.003789320820942521,
+            0.007918858900666237,
+            -0.0008104610024020076,
+            -0.01801980286836624,
+            -0.029516814276576042,
+            0.027311265468597412,
+            0.0043915449641644955,
+            -0.00404350645840168,
+            0.004219480790197849,
+            -0.0038049628492444754,
+            -0.004888184834271669,
+            -0.0021449353080242872,
+            -0.014641090296208858,
+            -0.02271246165037155,
+            0.013647811487317085,
+            0.012247835285961628,
+            -0.012294761836528778,
+            0.021289020776748657,
+            -0.008532814681529999,
+            0.007715510670095682,
+            -0.01791030913591385,
+            0.00850153062492609,
+            0.028703419491648674,
+            -0.006823905743658543,
+            -0.014359530992805958,
+            -0.0016248325118795037,
+            -0.008000980131328106,
+            0.0035370904952287674,
+            0.027264337986707687,
+            0.0062451446428895,
+            0.014531595632433891,
+            -0.01728462055325508,
+            -0.011176344938576221,
+            0.02243090234696865,
+            0.004434560891240835,
+            0.011207629926502705,
+            -0.0010001225164160132,
+            -0.009095934219658375,
+            -0.012099234387278557,
+            -0.00790321733802557,
+            0.010581942275166512,
+            0.012326045893132687,
+            -0.013288040645420551,
+            -0.0005117929540574551,
+            -0.007871932350099087,
+            0.005588172469288111,
+            -0.012505931779742241,
+            -0.011434441432356834,
+            -0.022008562460541725,
+            -0.008509351871907711,
+            -0.015532695688307285,
+            -0.026513513177633286,
+            -0.007168034091591835,
+            -0.0008265920332632959,
+            -0.012021023780107498,
+            -0.0369468554854393,
+            -0.022477827966213226,
+            -0.007547357119619846,
+            -0.019740445539355278,
+            0.008532814681529999,
+            0.0005748505354858935,
+            -0.021070031449198723,
+            0.006831726990640163,
+            -0.012349509634077549,
+            -0.009354029782116413,
+            0.00251448224298656,
+            0.01244336273521185,
+            -0.010065749287605286,
+            -0.014774048700928688,
+            -0.01240425743162632,
+            -0.01994379237294197,
+            0.0146567327901721,
+            -0.0027667125687003136,
+            0.003470611060038209,
+            -0.006299892440438271,
+            -0.004297691863030195,
+            -0.015454484149813652,
+            0.01994379237294197,
+            -0.02473030425608158,
+            0.0013139439979568124,
+            -0.0015867047477513552,
+            0.006929490715265274,
+            -0.003302457509562373,
+            -0.01745668426156044,
+            -0.009088112972676754,
+            -0.0010773559333756566,
+            -0.015313704498112202,
+            -0.02003764547407627,
+            -0.013014302588999271,
+            -0.005795431789010763,
+            0.015337168239057064,
+            -0.008564099669456482,
+            0.01273274328559637,
+            0.007922769524157047,
+            0.0035194929223507643,
+            -0.004422829486429691,
+            0.0035527325235307217,
+            -0.010738363489508629,
+            -0.0034803873859345913,
+            0.0017558359540998936,
+            0.005474766716361046,
+            0.006022243294864893,
+            0.0014263722114264965,
+            0.015939392149448395,
+            -0.009080291725695133,
+            0.012013202533125877,
+            0.020287921652197838,
+            0.02150801196694374,
+            0.006194307468831539,
+            0.00629207119345665,
+            0.004141270183026791,
+            -9.611393033992499e-05,
+            0.002846878720447421,
+            0.0023873893078416586,
+            -0.010980817489326,
+            0.012185266241431236,
+            -0.015587443485856056,
+            -0.008603204973042011,
+            0.013890265487134457,
+            -0.002537945518270135,
+            -0.015603085048496723,
+            0.015923749655485153,
+            0.0016746920300647616,
+            -0.02369791828095913,
+            -0.01062104757875204,
+            -0.007156302221119404,
+            0.004262497182935476,
+            0.020553838461637497,
+            0.007293171249330044,
+            -0.027811815962195396,
+            0.01708127185702324,
+            -0.0002189906663261354,
+            -0.013147260993719101,
+            -0.009103755466639996,
+            0.01837957464158535,
+            0.024042047560214996,
+            -0.0006647931295447052,
+            -0.04633216932415962,
+            -0.014226572588086128,
+            0.00808701291680336,
+            -0.026685576885938644,
+            -0.0285626407712698,
+            0.011684716679155827,
+            -0.00808701291680336,
+            -0.031565941870212555,
+            -0.005279239267110825,
+            -0.0007855312433093786,
+            -0.03078383207321167,
+            0.009494809433817863,
+            -0.0008705856744199991,
+            0.0032868152484297752,
+            0.007770258467644453,
+            -0.013647811487317085,
+            -0.00544348219409585,
+            -0.02471466176211834,
+            0.012286940589547157,
+            -0.007035075221210718,
+            -0.0038597106467932463,
+            -0.0036368092987686396,
+            0.009815474972128868,
+            -0.002762801945209503,
+            0.02279067225754261,
+            0.026701219379901886,
+            0.01966223306953907,
+            -0.003132348647341132,
+            -0.023588424548506737,
+            -0.009377493523061275,
+            0.002107785316184163,
+            0.012889165431261063,
+            0.006831726990640163,
+            -0.017018703743815422,
+            0.005678115412592888,
+            0.0014605895848944783,
+            0.013436641544103622,
+            0.000748869904782623,
+            0.02609117329120636,
+            -3.1039973691804335e-05,
+            0.002481242408975959,
+            0.002082366729155183,
+            -0.02580961398780346,
+            0.004680925514549017,
+            -0.0058110738173127174,
+            0.022477827966213226,
+            0.016784070059657097,
+            -0.0003084444615524262,
+            0.007011611945927143,
+            -0.0127718485891819,
+            -0.004727852065116167,
+            -0.028234153985977173,
+            -0.01709691435098648,
+            -0.00015947702922858298,
+            0.031002821400761604,
+            0.033130161464214325,
+            -0.02673250436782837,
+            -0.010136139579117298,
+            -0.0009385314187966287,
+            0.029735805466771126,
+            -0.006022243294864893,
+            -0.00218013022094965,
+            0.029000621289014816,
+            0.004485398065298796,
+            0.0030717351473867893,
+            0.012200908735394478,
+            0.0012767937732860446,
+            0.008861301466822624,
+            0.033568140119314194,
+            0.015915928408503532,
+            0.009627768769860268,
+            0.008861301466822624,
+            -0.002152756555005908,
+            -0.015837717801332474,
+            -0.021992919966578484,
+            -0.0020589034538716078,
+            -0.00652279332280159,
+            0.009486989118158817,
+            -0.018144940957427025,
+            0.006022243294864893,
+            -0.011786391027271748,
+            0.004125627689063549,
+            -0.011817675083875656,
+            -0.0022876705043017864,
+            0.0017450818559154868,
+            -0.01700306124985218,
+            -0.0025985590182244778,
+            -0.024980578571558,
+            -0.0027510703075677156,
+            0.0108869643881917,
+            -0.001176097197458148,
+            0.016127098351716995,
+            -0.016471227630972862,
+            0.0051541016437113285,
+            0.019177325069904327,
+            0.021289020776748657,
+            -0.01474276464432478,
+            6.71514353598468e-05,
+            0.003977026790380478,
+            -0.025981679558753967,
+            0.01579861342906952,
+            0.005158012267202139,
+            0.010175244882702827,
+            0.025684477761387825,
+            -0.0009443972376175225,
+            -0.0031831858213990927,
+            -0.012826596386730671,
+            -0.004532324615865946,
+            -0.03491336852312088,
+            0.024182826280593872,
+            -0.00034339496050961316,
+            0.009111575782299042,
+            0.004931200761348009,
+            -0.009627768769860268,
+            0.007574731018394232,
+            -0.010808753781020641,
+            0.019834298640489578,
+            0.005904926918447018,
+            0.003060003509745002,
+            -0.01492264959961176,
+            0.01653379574418068,
+            0.0025731404311954975,
+            -0.01894269324839115,
+            0.02463645115494728,
+            -0.01055847853422165,
+            0.0044932193122804165,
+            -0.0030580481979995966,
+            0.011160703375935555,
+            0.009690336883068085,
+            -0.006968596018850803,
+            0.004149090964347124,
+            -0.0296888779848814,
+            -0.02307223156094551,
+            -0.013037766329944134,
+            -0.0011389469727873802,
+            -0.007742884568870068,
+            -0.026435302570462227,
+            -0.022493470460176468,
+            0.004743494559079409,
+            -0.005294881761074066,
+            0.004688746761530638,
+            0.012208729982376099,
+            0.012654532678425312,
+            0.005728952586650848,
+            0.006331176497042179,
+            -0.009354029782116413,
+            0.006565809715539217,
+            0.026904568076133728,
+            -0.006565809715539217,
+            -0.03197263926267624,
+            -0.006933400873094797,
+            0.003554687835276127,
+            -0.018582923337817192,
+            -0.009948433376848698,
+            0.009486989118158817,
+            -0.004027863964438438,
+            0.0288442000746727,
+            -0.02380741387605667,
+            0.0038460236974060535,
+            -0.0038831739220768213,
+            0.004950753413140774,
+            -0.004806063137948513,
+            1.4840223229839467e-05,
+            -0.01911475695669651,
+            0.00017340834892820567,
+            0.004946842789649963,
+            -0.016846640035510063,
+            0.015696939080953598,
+            -0.007230602670460939,
+            -0.011019923724234104,
+            -0.007699868176132441,
+            0.0012846149038523436,
+            -0.007211050018668175,
+            -0.001168276066891849,
+            0.02627888135612011,
+            -0.007973606698215008,
+            -0.017394116148352623,
+            -0.004547967109829187,
+            0.0053574503399431705,
+            0.015149462036788464,
+            0.011473546735942364,
+            0.024057690054178238,
+            -0.013389714993536472,
+            0.003515582298859954,
+            -0.004059148486703634,
+            -0.0031734094955027103,
+            -4.744594116345979e-05,
+            0.039105478674173355,
+            -0.011434441432356834,
+            -0.0004949286812916398,
+            -0.00844678282737732,
+            0.009479167871177197,
+            0.006495419889688492,
+            0.020616406574845314,
+            0.006346818991005421,
+            0.002668948844075203,
+            0.007160212844610214,
+            -0.0006931445677764714,
+            0.0006760359392501414,
+            0.01460198499262333,
+            0.011184166185557842,
+            0.00723842391744256,
+            -0.0072814398445189,
+            -0.023025304079055786,
+            0.014234393835067749,
+            -0.0010841992916539311,
+            -0.0031675435602664948,
+            0.010261276736855507,
+            0.01212269812822342,
+            -0.009346208535134792,
+            0.012435541488230228,
+            -0.007703778799623251,
+            0.024761587381362915,
+            0.013702559284865856,
+            -0.01005792897194624,
+            0.009346208535134792,
+            0.021633150056004524,
+            -0.00032139813993126154,
+            0.015235493890941143,
+            -0.015423200093209743,
+            -0.021945994347333908,
+            0.0012210685526952147,
+            0.021539296954870224,
+            0.0005015277420170605,
+            -0.015344989486038685,
+            0.009322745725512505,
+            -0.007856289856135845,
+            -0.007805453147739172,
+            0.017847739160060883,
+            0.0125841423869133,
+            -0.009463525377213955,
+            -0.009682515636086464,
+            0.0010314069222658873,
+            0.004872542340308428,
+            0.012943913228809834,
+            0.02066333405673504,
+            -0.003447147784754634,
+            0.004239033907651901,
+            -0.00251448224298656,
+            0.03860492631793022,
+            -0.005748505238443613,
+            0.03125309571623802,
+            -0.015642190352082253,
+            -0.011528294533491135,
+            -0.009205428883433342,
+            -0.0063389977440238,
+            -0.005509961862117052,
+            -0.0024421371053904295,
+            0.0012347553856670856,
+            -0.005482587963342667,
+            -0.023103516548871994,
+            0.012717100791633129,
+            0.022634249180555344,
+            0.002792131155729294,
+            0.007031164597719908,
+            0.005013321992009878,
+            0.013710380531847477,
+            0.030924610793590546,
+            -0.005615546368062496,
+            -0.0147505858913064,
+            0.009002081118524075,
+            0.010034465231001377,
+            -0.016424300149083138,
+            0.021179527044296265,
+            0.006765247322618961,
+            0.006710499990731478,
+            -0.014320425689220428,
+            -0.0017147751059383154,
+            0.02022535353899002,
+            0.0071758548729121685,
+            0.009792011231184006,
+            -0.012294761836528778,
+            -0.0025966037064790726,
+            0.0012679951032623649,
+            0.0072814398445189,
+            -0.0034764769952744246,
+            -0.015008682385087013,
+            0.0159784983843565,
+            -0.014390815049409866,
+            0.02261860854923725,
+            -0.021335948258638382,
+            0.012513753026723862,
+            -0.009252355434000492,
+            0.020694619044661522,
+            -0.0011545892339199781,
+            -0.007727242074906826,
+            -0.00534962909296155,
+            0.013147260993719101,
+            -0.005955764092504978,
+            -0.00256140879355371,
+            -0.02250911295413971,
+            -0.013499210588634014,
+            0.017879024147987366,
+            -0.0216800756752491,
+            -0.0012826595921069384,
+            0.0019005262292921543,
+            0.010714900679886341,
+            -0.028687776997685432,
+            0.022821955382823944,
+            0.031174886971712112,
+            0.012294761836528778,
+            0.012474646791815758,
+            -0.0045166825875639915,
+            -0.016674574464559555,
+            -0.009432241320610046,
+            0.03344300389289856,
+            0.008345108479261398,
+            -0.01801980286836624,
+            0.006952953990548849,
+            0.01830136403441429,
+            -0.009119397029280663,
+            0.004661372862756252,
+            -0.0017685452476143837,
+            0.016064530238509178,
+            0.0020178426057100296,
+            0.0035898827482014894,
+            -0.01673714444041252,
+            -0.0365714430809021,
+            -0.012302583083510399,
+            0.02060076594352722,
+            -0.005666383542120457,
+            -0.008861301466822624,
+            0.009033365175127983,
+            0.0020002450328320265,
+            -0.003146035596728325,
+            -0.011387514881789684,
+            0.003810828784480691,
+            0.0074339513666927814,
+            0.006080901715904474,
+            -0.01967787556350231,
+            0.012021023780107498,
+            -0.009361851029098034,
+            -0.010276919230818748,
+            0.01075400598347187,
+            -0.013858981430530548,
+            -0.026216311380267143,
+            -0.0022172804456204176,
+            -0.015563979744911194,
+            -0.015689117833971977,
+            -0.0144690265879035,
+            0.006816084496676922,
+            -0.0012279120273888111,
+            -0.017347188666462898,
+            -0.028593923896551132,
+            0.014077971689403057,
+            0.0010186976287513971,
+            0.028265438973903656,
+            -0.029078831896185875,
+            0.022649891674518585,
+            0.00035341575858183205,
+            0.012920449487864971,
+            0.0017216185806319118,
+            -0.011848959140479565,
+            0.0019220341928303242,
+            0.005021143238991499,
+            0.003394355298951268,
+            0.001639497117139399,
+            -0.010456804186105728,
+            -0.0010890875710174441,
+            -0.018160583451390266,
+            0.006773068569600582,
+            -0.005236223340034485,
+            -0.014187467284500599,
+            -0.0026415749453008175,
+            0.015431021340191364,
+            0.00606134906411171,
+            0.022290121763944626,
+            -0.019599664956331253,
+            -0.002825370756909251,
+            -0.007817184552550316,
+            -0.012161803431808949,
+            -0.006788710597902536,
+            0.02031920664012432,
+            -0.007222781423479319,
+            0.03363071009516716,
+            0.008923869580030441,
+            -0.01024563517421484,
+            -0.011450083926320076,
+            0.0012562634656205773,
+            0.02066333405673504,
+            0.018504710868000984,
+            -0.02333814837038517,
+            -0.012114876881241798,
+            -0.0013892221031710505,
+            0.013311504386365414,
+            0.006690946873277426,
+            -0.0016717591788619757,
+            0.01736283116042614,
+            0.0066166468895971775,
+            -0.01679971255362034,
+            -0.01254503708332777,
+            -0.002807773184031248,
+            0.015650011599063873,
+            0.013217651285231113,
+            0.013960654847323895,
+            0.005337897688150406,
+            0.013248935341835022,
+            0.01902090385556221,
+            -0.01745668426156044,
+            -0.005369181744754314,
+            0.019255537539720535,
+            0.0029270448721945286,
+            -0.017879024147987366,
+            0.0028195048216730356,
+            0.0023033125326037407,
+            0.00807919166982174,
+            0.003292681183665991,
+            -0.004196017514914274,
+            0.03155029937624931,
+            -0.0009263110114261508,
+            0.005025053862482309,
+            0.018051087856292725,
+            -0.006655752193182707,
+            0.022305764257907867,
+            -0.0017421490047127008,
+            0.028906768187880516,
+            -0.02003764547407627,
+            0.003496029647067189,
+            0.0004572896577883512,
+            -0.0027979968581348658,
+            -0.0026357092428952456,
+            0.0016150561859831214,
+            -0.018317004665732384,
+            0.009275819174945354,
+            -0.04567519575357437,
+            0.008345108479261398,
+            -0.007602104917168617,
+            -0.0021410249173641205,
+            -0.0248867254704237,
+            -0.01015960332006216,
+            -0.009150682017207146,
+            -0.010581942275166512,
+            -0.018786270171403885,
+            -0.021899066865444183,
+            -0.00789930671453476,
+            0.004751315340399742,
+            0.01470365934073925,
+            -0.019896866753697395,
+            0.01939631626009941,
+            0.007070270366966724,
+            -0.026982778683304787,
+            -0.016877923160791397,
+            -0.019959434866905212,
+            -0.0010538926580920815,
+            -0.010065749287605286,
+            0.0028781630098819733,
+            -0.027952594682574272,
+            0.022024204954504967,
+            0.016674574464559555,
+            -0.022258836776018143,
+            0.0058580003678798676,
+            -0.01428914163261652,
+            -0.023776130750775337,
+            0.0017499701352789998,
+            0.0006931445677764714,
+            -0.00864231027662754,
+            0.04273446649312973,
+            0.021101314574480057,
+            8.737629832467064e-05,
+            -0.028640851378440857,
+            0.005728952586650848,
+            -0.02727998048067093,
+            0.013921549543738365,
+            0.004747404716908932,
+            0.00015752174658700824,
+            -0.0045166825875639915,
+            0.016236593946814537,
+            0.0019738490227609873,
+            0.01698741875588894,
+            0.006057438440620899,
+            0.0007400711765512824,
+            0.02473030425608158,
+            -0.011301483027637005,
+            0.006792621221393347,
+            -8.447394066024572e-06,
+            -0.0005313456640578806,
+            0.015696939080953598,
+            -0.004966395441442728,
+            -0.002283759880810976,
+            0.012279119342565536,
+            0.000982525059953332,
+            -0.009721621870994568,
+            -0.004371992312371731,
+            -0.0030424061696976423,
+            0.009135039523243904,
+            -0.015493590384721756,
+            0.01001100242137909,
+            0.0026259326841682196,
+            0.01928682066500187,
+            -0.018254436552524567,
+            -0.005803252570331097,
+            0.002670904155820608,
+            -0.009940612129867077,
+            -0.004606625065207481,
+            -0.015032145194709301,
+            -0.009338388219475746,
+            -0.02020971104502678,
+            0.01617402583360672,
+            0.04320373013615608,
+            0.02252475544810295,
+            0.000653550261631608,
+            -0.004055237863212824,
+            -0.01433606818318367,
+            0.003934010863304138,
+            -0.003478432074189186,
+            -0.011927170678973198,
+            -0.022540396079421043,
+            -0.005189296789467335,
+            0.012036665342748165,
+            -0.004078701138496399,
+            0.0019132354063913226,
+            -0.011653431691229343,
+            0.021101314574480057,
+            -0.01273274328559637,
+            -0.0004919957718811929,
+            -0.018723702058196068,
+            -0.015470126643776894,
+            0.009393135085701942,
+            0.032848600298166275,
+            -0.0023326417431235313,
+            0.013420999981462955,
+            0.013022123835980892,
+            0.004023953340947628,
+            -0.01478969119489193,
+            0.010675795376300812,
+            -0.006010511890053749,
+            0.004106075037270784,
+            -0.0025477218441665173,
+            -0.02380741387605667,
+            -0.009267997927963734,
+            -0.011176344938576221,
+            0.006909937597811222,
+            -0.009768548421561718,
+            -0.0006555055733770132,
+            -0.0015495545230805874,
+            -0.009909328073263168,
+            -0.018160583451390266,
+            0.007586462423205376,
+            -0.007774168625473976,
+            0.0032437993213534355,
+            0.00014090191689319909,
+            -0.0018741299863904715,
+            0.0087205208837986,
+            0.02020971104502678,
+            -0.0067535159178078175,
+            0.0030130769591778517,
+            0.004931200761348009,
+            -0.016846640035510063,
+            0.016393015161156654,
+            0.0019533184822648764,
+            -0.0006476844428107142,
+            0.03237933665513992,
+            0.016236593946814537,
+            0.012795312330126762,
+            -0.0017841873923316598,
+            -0.00305413780733943,
+            -0.001330563798546791,
+            0.014672375284135342,
+            -0.03388098627328873,
+            -0.009111575782299042,
+            0.014985218644142151,
+            0.02067897655069828,
+            -0.015829896554350853,
+            0.0127718485891819,
+            -0.015032145194709301,
+            -0.009690336883068085,
+            -0.007297081872820854,
+            0.00303262984380126,
+            -0.05340243875980377,
+            0.011043386533856392,
+            -0.006366371642798185,
+            0.02712355926632881,
+            0.0006178665207698941,
+            0.0108869643881917,
+            0.010120497085154057,
+            -0.006100454367697239,
+            0.025606267154216766,
+            0.0015045831678435206,
+            -0.0041764648631215096,
+            0.012850060127675533,
+            -0.011778569780290127,
+            -0.011661252938210964,
+            0.008564099669456482,
+            -0.0288442000746727,
+            0.008994259871542454,
+            -0.024933652952313423,
+            0.0022270570043474436,
+            -0.016220951452851295,
+            -0.012396436184644699,
+            0.022649891674518585,
+            0.03287988528609276,
+            0.00202175322920084,
+            0.0022329227067530155,
+            -0.010816575028002262,
+            0.025700120255351067,
+            -0.010550657287240028,
+            0.019443243741989136,
+            -0.008876943029463291,
+            -0.011207629926502705,
+            -0.017112556844949722,
+            -0.010808753781020641,
+            0.008783089928328991,
+            -0.003986803349107504,
+            -0.004375902935862541,
+            -0.021461086347699165,
+            0.005287060514092445,
+            -0.022211911156773567,
+            0.008837837725877762,
+            0.002567274495959282,
+            -0.006518883164972067,
+            0.001965050119906664,
+            0.0029876583721488714,
+            -0.0010167424334213138,
+            -0.00016729811613913625,
+            -0.024088973179459572,
+            -0.008743984624743462,
+            -0.008118296973407269,
+            0.009049007669091225,
+            0.001364781172014773,
+            -0.020538195967674255,
+            0.010581942275166512,
+            -0.009095934219658375,
+            -0.0036153013352304697,
+            -0.008657952770590782,
+            0.0028038627933710814,
+            0.0013872667914256454,
+            0.02067897655069828,
+            0.003855800023302436,
+            -0.020194068551063538,
+            0.009494809433817863,
+            0.017394116148352623,
+            -0.00253990083001554,
+            0.006331176497042179,
+            -0.00019063919899053872,
+            -0.01028474047780037,
+            -0.010589763522148132,
+            -0.0012865702155977488,
+            0.006299892440438271,
+            -0.009831116534769535,
+            0.005744594614952803,
+            0.006992059294134378,
+            0.0058345370925962925,
+            0.021070031449198723,
+            0.008900406770408154,
+            0.004242944065481424,
+            -0.0015603085048496723,
+            0.006733963266015053,
+            -0.0011614327086135745,
+            -0.0014928515302017331,
+            -0.012748385779559612,
+            0.009862401522696018,
+            -0.019630949944257736,
+            0.009651231579482555,
+            0.017675675451755524,
+            0.008986438624560833,
+            -0.016971776261925697,
+            -0.01755053736269474,
+            -0.007363561075180769,
+            -0.023478928953409195,
+            0.009095934219658375,
+            0.0007459369953721762,
+            0.018535995855927467,
+            -0.0046535516157746315,
+            0.008196507580578327,
+            0.006147380918264389,
+            -0.013804233632981777,
+            -0.0020549928303807974,
+            0.009502630680799484,
+            -0.029657592996954918,
+            -0.012521573342382908,
+            -0.000881339656189084,
+            -0.0010548701975494623,
+            0.018176225945353508,
+            0.01919296756386757,
+            0.019146041944622993,
+            0.03206649050116539,
+            -0.009705979377031326,
+            0.01023781392723322,
+            -0.0028371023945510387,
+            0.018864482641220093,
+            0.010042286477982998,
+            0.00733227701857686,
+            0.020632049068808556,
+            0.006972506642341614,
+            -0.02103874646127224,
+            -0.010832217521965504,
+            0.019818656146526337,
+            -0.010417698882520199,
+            -0.005959674715995789,
+            -0.012552858330309391,
+            0.0037756338715553284,
+            0.0033904449082911015,
+            0.011958454735577106,
+            0.018708059564232826,
+            0.008884764276444912,
+            -0.017315905541181564,
+            0.032754749059677124,
+            0.0007058538612909615,
+            0.00647977739572525,
+            0.004199928138405085,
+            -0.003887084312736988,
+            -0.005502140615135431,
+            0.013968476094305515,
+            0.012466826476156712,
+            -0.0014938291860744357,
+            0.0034745216835290194,
+            0.012623247690498829,
+            -0.01447684783488512,
+            -0.018598563969135284,
+            0.007527804467827082,
+            0.0017558359540998936,
+            0.00037883431650698185,
+            0.006589272990822792,
+            0.0025496771559119225,
+            0.017613107338547707,
+            -0.006831726990640163,
+            0.028187228366732597,
+            -0.005521693266928196,
+            -0.016142740845680237,
+            -0.013139440678060055,
+            0.0011555668897926807,
+            -0.011903706938028336,
+            -0.018144940957427025,
+            0.007504341192543507,
+            -0.0015104489866644144,
+            0.011934991925954819,
+            -0.002289625583216548,
+            0.004250765312463045,
+            0.013381893746554852,
+            0.017706960439682007,
+            0.014492489397525787,
+            0.002645485568791628,
+            -0.004508861340582371,
+            -0.015438842587172985,
+            0.008032265119254589,
+            -0.009698158130049706,
+            0.02132030576467514,
+            -0.014085792936384678,
+            -0.021977277472615242,
+            0.0031362592708319426,
+            0.01474276464432478,
+            -0.0020589034538716078,
+            0.0001868508552433923,
+            0.012748385779559612,
+            -0.0062451446428895,
+            0.00031675436184741557,
+            0.012568499892950058,
+            0.00445802416652441,
+            -0.02810901589691639,
+            -0.009612126275897026,
+            0.007273618597537279,
+            -0.0021625328809022903,
+            -0.003701333422213793,
+            -0.02388562448322773,
+            -0.004430650733411312,
+            -0.011324945837259293,
+            -0.005944032687693834,
+            -0.004371992312371731,
+            -8.847613935358822e-05,
+            0.009361851029098034,
+            -0.02269681915640831,
+            0.014461205340921879,
+            -0.013327146880328655,
+            -0.004555787891149521,
+            -0.017660032957792282,
+            0.022290121763944626,
+            0.00530661316588521,
+            0.0007527804118581116,
+            0.00507589103654027,
+            0.0012396436650305986,
+            0.008665774017572403,
+            0.01452377438545227,
+            -0.002537945518270135,
+            0.004778689239174128,
+            0.015665654093027115,
+            0.005345718469470739,
+            0.005760236643254757,
+            0.01938067376613617,
+            -0.003339607734233141,
+            -0.017769528552889824,
+            -0.003564464161172509,
+            0.016424300149083138,
+            0.011231092736124992,
+            0.015493590384721756,
+            0.004645730834454298,
+            0.01033166702836752,
+            -0.003564464161172509,
+            -0.006909937597811222,
+            0.018129298463463783,
+            0.0125841423869133,
+            -0.016205308958888054,
+            -0.011332767084240913,
+            0.012294761836528778,
+            0.01033166702836752,
+            0.011786391027271748,
+            0.003580106422305107,
+            -0.0027060990687459707,
+            0.014578522183001041,
+            0.017018703743815422,
+            0.006647930946201086,
+            0.020162783563137054,
+            -0.001665893360041082,
+            -0.03572676330804825,
+            -0.004684836138039827,
+            0.0011809853604063392,
+            -0.00016595386841800064,
+            0.009526094421744347,
+            0.00445802416652441,
+            0.008290360681712627,
+            0.005795431789010763,
+            0.01958402246236801,
+            -0.006628378294408321,
+            0.010589763522148132,
+            0.00020762563508469611,
+            -0.02056948095560074,
+            -0.0020862771198153496,
+            -0.0040708803571760654,
+            -0.032191626727581024,
+            -0.00019369431538507342,
+            0.006080901715904474,
+            0.023087874054908752,
+            -0.009479167871177197,
+            0.0029055369086563587,
+            -0.006057438440620899,
+            -0.013162903487682343,
+            -0.00220554880797863,
+            0.014101434499025345,
+            -0.0012249790597707033,
+            0.01037859357893467,
+            0.0030678247567266226,
+            0.0005034829955548048,
+            0.0036387646105140448,
+            0.01966223306953907,
+            -0.00021385807485785335,
+            0.00868923682719469,
+            0.016877923160791397,
+            0.020256636664271355,
+            8.627645729575306e-05,
+            -0.01222437247633934,
+            0.027983879670500755,
+            -0.01428914163261652,
+            -0.008697058074176311,
+            -0.005603814963251352,
+            -0.005290971137583256,
+            0.015305883251130581,
+            -0.008478067815303802,
+            0.010589763522148132,
+            0.006628378294408321,
+            -0.018035445362329483,
+            0.01102774403989315,
+            -0.011176344938576221,
+            0.02361970767378807,
+            0.01689356565475464,
+            -0.0017186857294291258,
+            0.004282049834728241,
+            0.007293171249330044,
+            -0.03466309607028961,
+            0.009792011231184006,
+            -0.010957354679703712,
+            -0.0010734453098848462,
+            -0.0030111216474324465,
+            -0.004939021542668343,
+            0.016768427565693855,
+            -0.00014383482630364597,
+            -0.022086773067712784,
+            -0.00312257232144475,
+            0.021429801359772682,
+            -0.0013638035161420703,
+            0.01010485552251339,
+            0.006448493339121342,
+            -0.0020256636198610067,
+            -0.01618966832756996,
+            -0.020788472145795822,
+            -0.018129298463463783,
+            -0.020240994170308113,
+            -0.014367352239787579,
+            -0.00849370937794447,
+            -0.004794331267476082,
+            0.0045948936603963375,
+            0.0026572172064334154,
+            -0.008923869580030441,
+            -0.020929250866174698,
+            -0.014320425689220428,
+            0.003934010863304138,
+            0.0020354401785880327,
+            0.007723331451416016,
+            -0.0071993181481957436,
+            -0.007954053580760956,
+            -0.002113651018589735,
+            -0.0005504096043296158,
+            -0.006972506642341614,
+            0.01295173354446888,
+            -0.02463645115494728,
+            -0.005658562295138836,
+            0.01947452686727047,
+            0.011434441432356834,
+            -0.004536235239356756,
+            0.010316024534404278,
+            -0.026263238862156868,
+            0.02286888286471367,
+            -0.018410857766866684,
+            -0.015470126643776894,
+            -0.0079423226416111,
+            0.013632168993353844,
+            -0.015313704498112202,
+            0.005427840165793896,
+            0.006769157946109772,
+            -0.004395455587655306,
+            0.016408657655119896,
+            0.009181966073811054,
+            -0.011066850274801254,
+            0.009987538680434227,
+            -0.009416598826646805,
+            0.0015475992113351822,
+            -0.01645558513700962,
+            0.0009624835220165551,
+            0.015423200093209743,
+            0.010495910421013832,
+            -0.005267507862299681,
+            -0.010988638736307621,
+            -0.005705488845705986,
+            0.00047635359806008637,
+            0.0004220947448629886,
+            0.007500430569052696,
+            0.004008311312645674,
+            -0.002915313234552741,
+            -0.016064530238509178,
+            -0.017706960439682007,
+            0.0262006688863039,
+            -0.0051541016437113285,
+            -0.009103755466639996,
+            0.02047562785446644,
+            0.02150801196694374,
+            0.01615838333964348,
+            -0.017534896731376648,
+            -0.0005655629793182015,
+            0.009612126275897026,
+            0.014031045138835907,
+            0.007351829670369625,
+            -0.003501895349472761,
+            -0.00719149736687541,
+            -0.021523654460906982,
+            0.030095575377345085,
+            0.007261887192726135,
+            0.0075942836701869965,
+            -0.031284380704164505,
+            0.023385075852274895,
+            0.00021129178639966995,
+            -0.010409877635538578,
+            0.0026787251699715853,
+            0.01966223306953907,
+            0.005388734862208366,
+            0.0008539658738300204,
+            0.02297837845981121,
+            0.015086892992258072,
+            0.004931200761348009,
+            -0.007680315524339676,
+            -0.013217651285231113,
+            0.004551877733319998,
+            -0.005392645485699177,
+            0.009557378478348255,
+            -0.012028845027089119,
+            -0.009033365175127983,
+            0.0037658573128283024,
+            -0.003056093119084835,
+            0.008212150074541569,
+            -0.0002551632351242006,
+            -0.0211169570684433,
+            -0.006487598642706871,
+            -0.0363837368786335,
+            0.0050172326155006886,
+            -0.024464385583996773,
+            -0.005693757440894842,
+            -0.014234393835067749,
+            -0.013069050386548042,
+            -0.03932446613907814,
+            -0.010667974129319191,
+            0.008282540366053581,
+            0.015548337250947952,
+            0.0036446305457502604,
+            -0.011543937027454376,
+            0.0037287073209881783,
+            -0.040763549506664276,
+            0.0003023342287633568,
+            0.02848443016409874,
+            0.014492489397525787,
+            0.023494569584727287,
+            -0.015321525745093822,
+            0.022665534168481827,
+            0.0005333009175956249,
+            -0.02848443016409874,
+            -0.0029896136838942766,
+            -0.0006017355481162667,
+            0.011950633488595486,
+            0.00093902024673298,
+            -0.009721621870994568,
+            -0.017675675451755524,
+            0.020741544663906097,
+            0.006264697294682264,
+            0.009502630680799484,
+            -0.005185386165976524,
+            0.001369669334962964,
+            0.00380300753749907,
+            0.005204938817769289,
+            -0.014953934587538242,
+            0.0005670294049195945,
+            0.010581942275166512,
+            0.016095813363790512,
+            -0.006808263715356588,
+            -0.01263889018446207,
+            -0.008955154567956924,
+            -0.008196507580578327,
+            -0.00213711429387331,
+            -0.012576321139931679,
+            -0.010370772331953049,
+            -0.011606505140662193,
+            -0.0028957605827599764,
+            -0.01469583809375763,
+            -0.01618966832756996,
+            0.008188687264919281,
+            -0.0009399979026056826,
+            0.02316608466207981,
+            0.013984118588268757,
+            0.018676776438951492,
+            0.008141760714352131,
+            0.01681535504758358,
+            0.005142370238900185,
+            0.004305513110011816,
+            -0.014343888498842716,
+            -0.002422584220767021,
+            -0.030533555895090103,
+            0.013647811487317085,
+            0.02984529919922352,
+            -0.007003790698945522,
+            -0.00305413780733943,
+            0.010050107724964619,
+            0.0022466096561402082,
+            -0.002281804569065571,
+            0.0019415869610384107,
+            -0.01645558513700962,
+            -0.005236223340034485,
+            -0.01010485552251339,
+            0.006483688019216061,
+            0.009346208535134792,
+            0.028906768187880516,
+            -0.005552977789193392,
+            -0.0019083472434431314,
+            0.02241525985300541,
+            0.014359530992805958,
+            0.02537163347005844,
+            0.009471346624195576,
+            -0.006604915019124746,
+            0.00042062829015776515,
+            -0.0065384358167648315,
+            -0.009095934219658375,
+            -0.026122458279132843,
+            0.031034106388688087,
+            -0.004477577283978462,
+            -0.00877526868134737,
+            -0.009267997927963734,
+            -0.011590863578021526,
+            0.0025105716194957495,
+            0.02020971104502678,
+            -0.015947213396430016,
+            0.0011233048280701041,
+            0.0001070512444130145,
+            -0.016064530238509178,
+            -0.02712355926632881,
+            0.03189442679286003,
+            0.005979227367788553,
+            -0.002801907481625676,
+            -3.3239655749639496e-05,
+            -0.007426130119711161,
+            -0.011387514881789684,
+            -0.009346208535134792,
+            -0.0056976680643856525,
+            0.06563463062047958,
+            -0.01466455403715372,
+            0.002569229807704687,
+            -0.02305658906698227,
+            0.0031499462202191353,
+            -0.007766347844153643,
+            -0.00024954183027148247,
+            0.0075395358726382256,
+            -0.011004281230270863,
+            -0.0030736904591321945,
+            -0.01310033444315195,
+            -0.002369791967794299,
+            0.004160822834819555,
+            -0.019771728664636612,
+            -0.009236713871359825,
+            -0.0034745216835290194,
+            -0.011254556477069855,
+            -0.005283149890601635,
+            0.012153982184827328,
+            0.011309304274618626,
+            0.013890265487134457,
+            -0.02537163347005844,
+            -0.003361115697771311,
+            0.004852989688515663,
+            -0.008900406770408154,
+            -0.004012221936136484,
+            -0.017769528552889824,
+            -0.010065749287605286,
+            -0.009158502332866192,
+            -0.011066850274801254,
+            0.014774048700928688,
+            -0.009526094421744347,
+            0.012200908735394478,
+            0.006956864148378372,
+            0.0065970937721431255,
+            0.01958402246236801,
+            0.014492489397525787,
+            0.011692537926137447,
+            0.013037766329944134,
+            0.005658562295138836,
+            -0.01442992128431797,
+            0.019709160551428795,
+            -0.02213370054960251,
+            -0.022383974865078926,
+            -0.010018822737038136,
+            0.0033063681330531836,
+            -0.009893685579299927,
+            -0.007926680147647858,
+            -0.011332767084240913,
+            0.015650011599063873,
+            0.0026161563582718372,
+            -0.007649031467735767,
+            -0.012388614937663078,
+            -0.0024890636559575796,
+            0.0291257593780756,
+            -0.007649031467735767,
+            0.02039741724729538,
+            -0.012153982184827328,
+            0.009533915668725967,
+            -0.015086892992258072,
+            -0.00813393946737051,
+            -0.009713800624012947,
+            -0.008282540366053581,
+            0.014907008036971092,
+            -0.006319445092231035,
+            0.005298791918903589,
+            -0.017394116148352623,
+            -0.02692021057009697,
+            -0.0003842113073915243,
+            0.00380300753749907,
+            -0.028140300884842873,
+            0.007578641641885042,
+            -0.02316608466207981,
+            0.004802152514457703,
+            -0.011692537926137447,
+            -0.014672375284135342,
+            -0.01654943823814392,
+            0.024057690054178238,
+            0.002991568995639682,
+            -0.004188196733593941,
+            0.00126995041500777,
+            -0.00521667068824172,
+            -0.006354639772325754,
+            0.009330566972494125,
+            0.006964685395359993,
+            0.0049585746601223946,
+            0.015000861138105392,
+            -0.006980327423661947,
+            -0.011450083926320076,
+            -0.00256140879355371,
+            -0.0035879274364560843,
+            -0.017597464844584465,
+            0.0015700849471613765,
+            0.002289625583216548,
+            -0.006741784047335386,
+            0.005201028194278479,
+            0.013530494645237923,
+            -0.0049703060649335384,
+            0.006530614569783211,
+            0.018692417070269585,
+            0.0022270570043474436,
+            -0.005728952586650848,
+            -0.0055177826434373856,
+            -0.003934010863304138,
+            -0.002438226481899619,
+            0.014062329195439816,
+            0.010464625433087349,
+            0.015266777947545052,
+            -0.007273618597537279,
+            0.009385314770042896,
+            -0.00436417106539011,
+            0.001194672309793532,
+            -0.028140300884842873,
+            0.00799315981566906,
+            0.005052427761256695,
+            -0.0028351470828056335,
+            0.00530661316588521,
+            0.0017069540917873383,
+            0.04270318150520325,
+            0.008814374916255474,
+            0.013327146880328655,
+            -0.003323965473100543,
+            0.009674695320427418,
+            0.0027354280464351177,
+            0.012615426443517208,
+            0.0006882564048282802,
+            -0.02105438895523548,
+            0.011567399837076664,
+            -0.015047787688672543,
+            0.0027021884452551603,
+            0.017034346237778664,
+            -0.019177325069904327,
+            0.015438842587172985,
+            -0.01025345642119646,
+            -0.0001107173811760731,
+            -0.004002445377409458,
+            -0.022743744775652885,
+            -0.01001100242137909,
+            -0.00019137242634315044,
+            0.023932551965117455,
+            -0.0017978743417188525,
+            0.025747045874595642,
+            0.006417208816856146,
+            0.02491801045835018,
+            -0.00804008636623621,
+            -0.017253335565328598,
+            -0.005607725586742163,
+            0.005103264935314655,
+            -0.003548822132870555,
+            0.0057524158619344234,
+            0.008282540366053581,
+            0.0060183326713740826,
+            -0.001099841552786529,
+            -0.0007962852832861245,
+            -0.000347061111824587,
+            0.022900167852640152,
+            0.009291461668908596,
+            0.01056629978120327,
+            -0.01037859357893467,
+            0.013084692880511284,
+            -0.018051087856292725,
+            0.00021336926147341728,
+            0.0010089213028550148,
+            -0.022649891674518585,
+            0.0014077972155064344,
+            -0.004852989688515663,
+            -0.004371992312371731,
+            0.0026669935323297977,
+            0.019505811855196953,
+            -0.019427601248025894,
+            -0.014555058442056179,
+            0.02590346708893776,
+            0.007426130119711161,
+            0.0025555428583174944,
+            -0.002262251917272806,
+            0.009831116534769535,
+            0.00873616337776184,
+            0.011614326387643814,
+            -0.004720031283795834,
+            -0.005846268963068724,
+            -0.017753886058926582,
+            0.014164003543555737,
+            -0.006534525193274021,
+            -0.010308203287422657,
+            -0.008923869580030441,
+            -0.008345108479261398,
+            0.01617402583360672,
+            0.0007835759897716343,
+            0.007668584119528532,
+            0.002807773184031248,
+            0.02637273445725441,
+            -0.01281877513974905,
+            -0.011332767084240913,
+            -0.009408777579665184,
+            0.008556278422474861,
+            0.024464385583996773,
+            0.007957964204251766,
+            0.013202008791267872,
+            -0.01618966832756996,
+            -0.0036993781104683876,
+            0.019271178171038628,
+            -0.011872422881424427,
+            -0.033224012702703476,
+            -0.007500430569052696
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "c6bf0f92-84c8-44b9-8b28-7fcab6058203",
+        "properties": {
+          "page_content": "8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n",
+          "keyphrases": [
+            "Mixture of Experts (MoE)",
+            "language model training",
+            "DeepSeekMoE architecture",
+            "expert-choice routing strategy",
+            "training instability"
+          ],
+          "title": "Related Work",
+          "embedding": [
+            -0.0004454151203390211,
+            -0.01349730882793665,
+            -0.018033159896731377,
+            0.005232766270637512,
+            0.01874188706278801,
+            0.010268663987517357,
+            -0.011331754736602306,
+            0.03266443312168121,
+            -0.018993878737092018,
+            0.018789134919643402,
+            -0.005878495052456856,
+            -0.009221323765814304,
+            0.007355009205639362,
+            -0.008567719720304012,
+            0.0011782583314925432,
+            0.02784508839249611,
+            0.018474144861102104,
+            0.015410871244966984,
+            0.002397859003394842,
+            -0.03512135148048401,
+            0.019734105095267296,
+            -0.02189178392291069,
+            -0.044067058712244034,
+            -0.013772924430668354,
+            0.014253283850848675,
+            0.01693069562315941,
+            0.03858623653650284,
+            -0.006386416032910347,
+            -0.05959605053067207,
+            0.03386139124631882,
+            0.020521579310297966,
+            0.02291550114750862,
+            -0.01582823134958744,
+            -0.014930511824786663,
+            -0.02899480052292347,
+            -0.013481559231877327,
+            0.05077634006738663,
+            0.021734289824962616,
+            -0.016410963609814644,
+            0.027687594294548035,
+            0.0467129722237587,
+            0.013268941082060337,
+            -0.02280525304377079,
+            -0.0008283244678750634,
+            0.005512319505214691,
+            0.0011831800220534205,
+            -0.019009627401828766,
+            -0.016332216560840607,
+            0.010497031733393669,
+            -0.028175828978419304,
+            -0.018883632495999336,
+            -0.027278108522295952,
+            -0.006665969267487526,
+            0.0006486818892881274,
+            -0.0642578974366188,
+            0.007378633599728346,
+            0.01752917654812336,
+            0.03389289230108261,
+            0.05058734491467476,
+            0.0015080131124705076,
+            0.012308222241699696,
+            -0.019419115036725998,
+            -0.01686769910156727,
+            -0.008260604925453663,
+            -0.00022885970247443765,
+            -0.01097739115357399,
+            0.012016857042908669,
+            -0.013324064202606678,
+            0.01982860080897808,
+            0.016221968457102776,
+            -0.040854163467884064,
+            0.027026114985346794,
+            -0.023876218125224113,
+            0.009426066651940346,
+            -0.006969146896153688,
+            0.007095142733305693,
+            0.019135624170303345,
+            0.026600880548357964,
+            -0.0019509674748405814,
+            -0.007252637762576342,
+            0.0025514166336506605,
+            0.017324432730674744,
+            -0.007122704293578863,
+            0.0022108338307589293,
+            0.05014635995030403,
+            -0.04687046632170677,
+            -0.02584490366280079,
+            -0.07049469649791718,
+            0.00882758665829897,
+            0.00037503460771404207,
+            -0.024789689108729362,
+            -0.003289673710241914,
+            0.028490817174315453,
+            0.008835460990667343,
+            -0.040602169930934906,
+            0.06312393397092819,
+            -0.014056415297091007,
+            -0.015662861987948418,
+            -0.03512135148048401,
+            -0.008355101570487022,
+            0.0005320372874848545,
+            -0.0049768369644880295,
+            0.011575871147215366,
+            0.010709649883210659,
+            0.012197976000607014,
+            -0.012812205590307713,
+            -0.033199913799762726,
+            0.04687046632170677,
+            -0.005713125690817833,
+            0.08523621410131454,
+            0.0021379925310611725,
+            -0.009945799596607685,
+            -0.003252268536016345,
+            0.03537334129214287,
+            -0.021435048431158066,
+            -0.008386600762605667,
+            -0.028349071741104126,
+            -0.04935888573527336,
+            0.024852687492966652,
+            -0.02348248101770878,
+            0.018631640821695328,
+            -0.0007800916791893542,
+            0.016395213082432747,
+            -0.019214371219277382,
+            -0.0721326395869255,
+            0.05739112198352814,
+            -0.0026774124708026648,
+            -0.015111630782485008,
+            -0.003021932439878583,
+            0.007201451808214188,
+            -0.0008194653782993555,
+            0.0037345965392887592,
+            -0.01735593192279339,
+            -0.019088376313447952,
+            0.029136545956134796,
+            0.023513980209827423,
+            0.03861773759126663,
+            0.046240486204624176,
+            -0.020301084965467453,
+            -0.01452889945358038,
+            0.023734472692012787,
+            -0.029514534398913383,
+            0.006630532909184694,
+            0.011567996814846992,
+            -0.01746617816388607,
+            0.011599495075643063,
+            0.011662493459880352,
+            0.06967572122812271,
+            -0.019860100001096725,
+            0.017009442672133446,
+            0.0022895813453942537,
+            0.0643208920955658,
+            0.0025435418356209993,
+            0.030475253239274025,
+            -0.014765141531825066,
+            0.02178153768181801,
+            -0.028160078451037407,
+            0.0025061366613954306,
+            -0.020222337916493416,
+            -0.003464886685833335,
+            -0.002510074060410261,
+            -0.04094865918159485,
+            -0.002821126487106085,
+            0.028853056952357292,
+            0.0015119505114853382,
+            0.01776541955769062,
+            0.033199913799762726,
+            0.003850748995319009,
+            -0.05962754786014557,
+            0.023198990151286125,
+            0.03452287241816521,
+            0.010985265485942364,
+            -0.009103202261030674,
+            0.021624041721224785,
+            0.011056138202548027,
+            0.010575778782367706,
+            0.02067907340824604,
+            0.008189732208848,
+            -0.022710757330060005,
+            0.03972020000219345,
+            -0.02886880561709404,
+            0.0018830477492883801,
+            0.03228644281625748,
+            0.0016743671149015427,
+            0.044791534543037415,
+            0.012906703166663647,
+            0.026852872222661972,
+            0.01693069562315941,
+            0.012316097505390644,
+            0.029939770698547363,
+            0.007166015449911356,
+            0.010536405257880688,
+            0.03540484234690666,
+            -0.015095881186425686,
+            0.008158233016729355,
+            -0.02255326136946678,
+            -0.012867328710854053,
+            0.006488787941634655,
+            0.00988280214369297,
+            -0.005815497133880854,
+            -0.0021753974724560976,
+            -0.04201962426304817,
+            0.02212802693247795,
+            -0.0055831922218203545,
+            0.021986281499266624,
+            0.0026144145522266626,
+            0.012615337036550045,
+            0.007630625274032354,
+            0.03559383377432823,
+            -0.03852323815226555,
+            -0.011788489297032356,
+            -0.007878679782152176,
+            0.005535943899303675,
+            -0.044791534543037415,
+            -0.020033344626426697,
+            0.016725953668355942,
+            -0.004776031244546175,
+            0.021135808899998665,
+            0.009276446886360645,
+            -0.02159254439175129,
+            -0.006626595743000507,
+            -0.006445476785302162,
+            0.018363898620009422,
+            -0.01533999852836132,
+            -0.030491001904010773,
+            -0.0351843498647213,
+            0.01300907414406538,
+            0.011536497622728348,
+            0.03590882569551468,
+            -0.0009178996551781893,
+            -0.020600326359272003,
+            0.014788766391575336,
+            0.001092620543204248,
+            0.0039216214790940285,
+            0.02274225652217865,
+            -0.04646097868680954,
+            0.00845747347921133,
+            0.0026537880767136812,
+            0.011520748026669025,
+            0.02085231803357601,
+            0.007394383195787668,
+            0.017182687297463417,
+            -0.035751331597566605,
+            0.004086991306394339,
+            0.03688529506325722,
+            0.011638869531452656,
+            -0.016103848814964294,
+            0.028065582737326622,
+            4.574118065647781e-05,
+            -0.012883078306913376,
+            -0.0033290472347289324,
+            -0.03222344443202019,
+            -0.012623212300240993,
+            0.0008617921266704798,
+            -0.016757450997829437,
+            0.05269777774810791,
+            0.043090589344501495,
+            -0.019324617460370064,
+            -0.014631271362304688,
+            -0.05918656289577484,
+            0.039688702672719955,
+            -0.02408096194267273,
+            -0.025262173265218735,
+            0.04416155442595482,
+            -0.016725953668355942,
+            0.020159341394901276,
+            0.032475437968969345,
+            0.012820080853998661,
+            -0.00011873635230585933,
+            -0.03527884557843208,
+            -0.004051554948091507,
+            -0.04523251950740814,
+            -0.03801925480365753,
+            0.005126457195729017,
+            0.03170371428132057,
+            -0.03559383377432823,
+            -0.0055792550556361675,
+            -0.009481189772486687,
+            0.030191762372851372,
+            0.028837306424975395,
+            0.02049008011817932,
+            -0.0016379464650526643,
+            0.007969239726662636,
+            0.016379464417696,
+            -0.03208170086145401,
+            0.013394936919212341,
+            -0.0046460977755486965,
+            -0.026522131636738777,
+            -0.010308037512004375,
+            -0.015749484300613403,
+            -0.006760466378182173,
+            -4.174228524789214e-05,
+            -0.04693346470594406,
+            0.00440985569730401,
+            0.03540484234690666,
+            -0.0037050663959234953,
+            -0.01795441284775734,
+            0.03323141112923622,
+            0.033451907336711884,
+            -0.027860838919878006,
+            -0.06167498230934143,
+            -0.001828908920288086,
+            0.021450798958539963,
+            -0.0024529823567718267,
+            -0.05616266280412674,
+            -0.003445199690759182,
+            -0.008276354521512985,
+            0.030554000288248062,
+            -0.01710394024848938,
+            -0.04201962426304817,
+            0.0032680181320756674,
+            -0.0005128426128067076,
+            -0.013481559231877327,
+            -0.03433387726545334,
+            -0.00764243770390749,
+            0.034365374594926834,
+            0.01437140442430973,
+            0.025813406333327293,
+            -0.04384656623005867,
+            -0.02929404191672802,
+            0.01270983461290598,
+            0.0394052118062973,
+            -0.06284044682979584,
+            0.010308037512004375,
+            -0.014292657375335693,
+            -0.001888953847810626,
+            0.004728782922029495,
+            0.043878063559532166,
+            -0.03291642293334007,
+            0.009016579948365688,
+            -0.00609505083411932,
+            -0.04167313501238823,
+            -0.034963857382535934,
+            -0.010827770456671715,
+            -0.00495715020224452,
+            -0.016332216560840607,
+            -0.02297849766910076,
+            0.002760097151622176,
+            -0.014269033446907997,
+            0.003340859431773424,
+            -0.02165554091334343,
+            0.004846903961151838,
+            -0.017859915271401405,
+            0.021734289824962616,
+            0.010056045837700367,
+            -0.013938293792307377,
+            -0.025703158229589462,
+            -0.004941401071846485,
+            0.02097831293940544,
+            0.012670460157096386,
+            0.00115561846178025,
+            -0.0034688240848481655,
+            0.002754191169515252,
+            -0.003376295790076256,
+            -0.009693807922303677,
+            -0.06545485556125641,
+            -0.017261436209082603,
+            -0.017607923597097397,
+            0.015552615746855736,
+            0.016056599095463753,
+            0.010725399479269981,
+            -0.046901967376470566,
+            -0.05118582397699356,
+            0.028538066893815994,
+            -0.03549933806061745,
+            -0.001561167649924755,
+            -0.0025927589740604162,
+            -0.0015926666092127562,
+            -0.004386231303215027,
+            0.016568457707762718,
+            -0.005606816615909338,
+            0.03360940143465996,
+            -0.012016857042908669,
+            0.007650312036275864,
+            0.004480728413909674,
+            0.020269587635993958,
+            0.0333574078977108,
+            -0.011103386990725994,
+            -0.011048263870179653,
+            0.049106892198324203,
+            -0.013631178997457027,
+            0.025088928639888763,
+            0.005638315342366695,
+            0.001570026739500463,
+            0.0033861391711980104,
+            0.02382897026836872,
+            0.014198160730302334,
+            0.005350887309759855,
+            -0.03713728487491608,
+            0.027089113369584084,
+            0.010363160632550716,
+            -0.060919005423784256,
+            -0.015088006854057312,
+            0.006595096550881863,
+            0.017418930307030678,
+            0.016536958515644073,
+            -0.003559383563697338,
+            -0.014662770554423332,
+            -0.013457934372127056,
+            -0.008481097407639027,
+            -0.018521394580602646,
+            0.036412809044122696,
+            -0.0033054230734705925,
+            0.010725399479269981,
+            -0.027215110138058662,
+            0.0045752255246043205,
+            0.01328469067811966,
+            0.016206219792366028,
+            0.02942003682255745,
+            -0.015922728925943375,
+            -0.015017134137451649,
+            -0.04718545451760292,
+            -0.03288492560386658,
+            -0.00928432121872902,
+            0.013245316222310066,
+            0.030128763988614082,
+            0.01600935123860836,
+            0.046429481357336044,
+            -0.012757082469761372,
+            0.00840235035866499,
+            -0.027624595910310745,
+            0.03795625641942024,
+            0.049043893814086914,
+            0.009985173121094704,
+            -0.020631825551390648,
+            0.02171853929758072,
+            0.009985173121094704,
+            -0.01995459757745266,
+            0.012308222241699696,
+            -0.020631825551390648,
+            0.040413178503513336,
+            -0.022458765655755997,
+            0.03874373435974121,
+            -0.04337408021092415,
+            -0.010709649883210659,
+            -0.03414488211274147,
+            0.003462917869910598,
+            0.02856956608593464,
+            -0.022395767271518707,
+            -0.021608293056488037,
+            -0.0017294903518632054,
+            0.0272938571870327,
+            0.012410594150424004,
+            -0.0009070718660950661,
+            -0.02293124981224537,
+            -0.021198807284235954,
+            -0.03534184396266937,
+            0.006890399381518364,
+            -0.014465902000665665,
+            -0.017371682450175285,
+            -0.0533907525241375,
+            0.014245408587157726,
+            -0.07143966108560562,
+            0.010993140749633312,
+            0.0485084131360054,
+            -0.012276723049581051,
+            0.016662955284118652,
+            0.0007077424670569599,
+            -0.004256298299878836,
+            -0.02935704030096531,
+            0.00967018399387598,
+            0.049831368029117584,
+            0.04028718173503876,
+            -0.021482296288013458,
+            -0.017670921981334686,
+            0.009150451049208641,
+            -0.028601063415408134,
+            -0.01589122973382473,
+            -0.005209141876548529,
+            0.01704094186425209,
+            0.014481651596724987,
+            -0.02589215338230133,
+            0.02638038620352745,
+            0.008465347811579704,
+            0.004953213036060333,
+            0.028396321460604668,
+            0.01331618893891573,
+            -0.0460829921066761,
+            0.014347780495882034,
+            0.029372788965702057,
+            0.001872219960205257,
+            -0.010378910228610039,
+            -0.004106678068637848,
+            0.00592968100681901,
+            0.01819065399467945,
+            -0.017576424404978752,
+            0.007114829961210489,
+            0.005591067019850016,
+            -0.01583610661327839,
+            0.020694822072982788,
+            0.011930234730243683,
+            0.01128450594842434,
+            -0.01067027635872364,
+            0.0218287855386734,
+            -0.00031375925755128264,
+            -0.019907347857952118,
+            -0.009426066651940346,
+            -0.00981980375945568,
+            0.02061607502400875,
+            0.00038364759529940784,
+            -0.03445987403392792,
+            0.011111261323094368,
+            0.04372056946158409,
+            -0.001538527780212462,
+            0.01003242190927267,
+            0.010812020860612392,
+            0.030223261564970016,
+            0.028538066893815994,
+            -0.022710757330060005,
+            -0.02825457602739334,
+            0.01691494695842266,
+            -0.009394567459821701,
+            0.004547663498669863,
+            -0.022632010281085968,
+            -0.014237534254789352,
+            0.03408188372850418,
+            -0.016725953668355942,
+            0.033136915415525436,
+            -0.002256113803014159,
+            0.015969976782798767,
+            -0.004846903961151838,
+            0.0010483250953257084,
+            -0.006433664355427027,
+            0.035215847194194794,
+            -0.026396136730909348,
+            0.014930511824786663,
+            0.007110892329365015,
+            0.002866406226530671,
+            0.05723362788558006,
+            -0.03814525157213211,
+            -0.03625531494617462,
+            0.00262228911742568,
+            0.030853239819407463,
+            -0.017796916887164116,
+            -0.005913931410759687,
+            -0.009599311277270317,
+            0.0031813958194106817,
+            0.018757635727524757,
+            -0.0351843498647213,
+            -0.01855289377272129,
+            0.01331618893891573,
+            -0.01801741123199463,
+            0.003470792667940259,
+            -0.0006501584430225194,
+            0.0018436741083860397,
+            0.00524457823485136,
+            0.0618639774620533,
+            -0.05175280570983887,
+            0.016946446150541306,
+            0.03603482246398926,
+            -0.0007520379149354994,
+            -0.03282192721962929,
+            -0.02625439129769802,
+            -0.005736749619245529,
+            -0.003931465093046427,
+            -0.03067999705672264,
+            -1.9440769392531365e-05,
+            -0.004043680150061846,
+            0.007024270016700029,
+            -0.03669629991054535,
+            0.014623397029936314,
+            0.011355378665030003,
+            -0.002449044957756996,
+            0.003744439920410514,
+            -0.01391466986387968,
+            -0.002553385216742754,
+            -0.003896028734743595,
+            0.03367239981889725,
+            -0.011064013466238976,
+            -0.002878218423575163,
+            -0.00967018399387598,
+            -0.003775938879698515,
+            -0.007197514642030001,
+            -0.0027738779317587614,
+            -0.00845747347921133,
+            0.0031873020343482494,
+            0.012237349525094032,
+            0.001938170986250043,
+            -0.012426343746483326,
+            -0.0028171890880912542,
+            0.03657030314207077,
+            -0.02510467916727066,
+            -0.021183056756854057,
+            0.051280323415994644,
+            0.040476176887750626,
+            0.004374419339001179,
+            -0.021907532587647438,
+            -0.010260789655148983,
+            0.021498046815395355,
+            -0.019749853760004044,
+            0.013670552521944046,
+            0.032176196575164795,
+            -0.007547940593212843,
+            -0.028364822268486023,
+            -0.00964655913412571,
+            -0.014410778880119324,
+            0.0019509674748405814,
+            -0.00562256621196866,
+            0.024065213277935982,
+            -0.044539544731378555,
+            0.008488972671329975,
+            0.019938847050070763,
+            0.002590790158137679,
+            -0.012394844554364681,
+            -0.011040388606488705,
+            -0.003295579692348838,
+            0.02178153768181801,
+            0.0009813897777348757,
+            -0.01893088035285473,
+            0.00861496850848198,
+            0.013418560847640038,
+            0.006158048752695322,
+            0.01893088035285473,
+            -0.005732812453061342,
+            0.02352973073720932,
+            0.009733181446790695,
+            -0.0005241625476628542,
+            0.014056415297091007,
+            -0.019970346242189407,
+            -0.00488234031945467,
+            0.061517488211393356,
+            0.011623119935393333,
+            0.025734657421708107,
+            -0.01431628130376339,
+            0.020631825551390648,
+            0.03814525157213211,
+            -0.0016999599756672978,
+            -0.020820818841457367,
+            0.008362976834177971,
+            0.00283490726724267,
+            -0.044665537774562836,
+            -0.0134894335642457,
+            -0.023923467844724655,
+            -0.037326280027627945,
+            0.0035357594024389982,
+            -0.021214555948972702,
+            0.01801741123199463,
+            0.025199174880981445,
+            0.0065832845866680145,
+            -0.030884739011526108,
+            -0.01197748351842165,
+            0.022710757330060005,
+            0.02741985209286213,
+            0.03449137136340141,
+            -0.01233972143381834,
+            0.010103294625878334,
+            -0.003295579692348838,
+            0.013694177381694317,
+            0.037987757474184036,
+            0.006925835739821196,
+            -0.0031006797216832638,
+            0.008059798739850521,
+            0.029703527688980103,
+            0.03675929829478264,
+            -0.024584945291280746,
+            -0.052130796015262604,
+            -0.0028703436255455017,
+            -0.01686769910156727,
+            0.018600141629576683,
+            -0.001023716526106,
+            0.003746408736333251,
+            0.0011428219731897116,
+            0.006654157303273678,
+            -0.008740964345633984,
+            0.02893180400133133,
+            0.008028299547731876,
+            -0.021797286346554756,
+            0.05159531161189079,
+            -0.015481743961572647,
+            0.024112461134791374,
+            -0.001423359732143581,
+            -0.02590790204703808,
+            -0.028711311519145966,
+            0.013111446052789688,
+            0.007390445563942194,
+            -0.03546784073114395,
+            0.03742077574133873,
+            0.033451907336711884,
+            0.0030731181614100933,
+            0.0012127102818340063,
+            0.0166787039488554,
+            0.016962194815278053,
+            -0.014780891127884388,
+            -0.023876218125224113,
+            0.02286825142800808,
+            -0.008677965961396694,
+            -0.021939031779766083,
+            0.01982860080897808,
+            -0.018521394580602646,
+            -0.008268479257822037,
+            0.0042484235018491745,
+            -0.001532621681690216,
+            -0.010182041674852371,
+            0.014395029284060001,
+            0.016521209850907326,
+            -0.015332123264670372,
+            -0.02527792379260063,
+            0.031215479597449303,
+            0.021513795480132103,
+            0.0038231874350458384,
+            -0.02165554091334343,
+            0.013473683968186378,
+            -0.022537512704730034,
+            -0.0010286382166668773,
+            0.043752070516347885,
+            -0.002433295361697674,
+            0.012946076691150665,
+            -0.010237164795398712,
+            -0.029719278216362,
+            0.002671506255865097,
+            0.011213633231818676,
+            0.03789326176047325,
+            0.01885213330388069,
+            0.01795441284775734,
+            0.019450614228844643,
+            -0.01631646603345871,
+            0.009528438560664654,
+            -0.014560398645699024,
+            -0.03294792026281357,
+            0.03179820999503136,
+            -0.006024178117513657,
+            0.025577163323760033,
+            0.008465347811579704,
+            -0.0321604460477829,
+            -0.004728782922029495,
+            0.011040388606488705,
+            0.013001199811697006,
+            0.023120243102312088,
+            0.013867421075701714,
+            0.003108554519712925,
+            0.031388722360134125,
+            -0.014513149857521057,
+            -0.051091328263282776,
+            0.02710486389696598,
+            0.0063076685182750225,
+            -0.006701405625790358,
+            -0.0016143221873790026,
+            0.020364083349704742,
+            -0.003027838421985507,
+            0.00017139868577942252,
+            0.01270195934921503,
+            -0.010961641557514668,
+            0.00882758665829897,
+            -0.006992771290242672,
+            -0.005469008814543486,
+            0.02414396032691002,
+            0.011827862821519375,
+            0.024065213277935982,
+            -0.019497862085700035,
+            0.006347042508423328,
+            0.017324432730674744,
+            -0.009748931042850018,
+            0.014568272978067398,
+            0.0034314189106225967,
+            -0.05011485889554024,
+            0.01607234962284565,
+            0.017576424404978752,
+            -0.005173705518245697,
+            0.007103017531335354,
+            -0.013402811251580715,
+            0.013804423622786999,
+            0.04627198725938797,
+            0.0016330246580764651,
+            0.0066856564953923225,
+            0.013205942697823048,
+            -0.005012273322790861,
+            -0.009804054163396358,
+            0.03112098202109337,
+            0.00845747347921133,
+            -0.008796087466180325,
+            0.009079578332602978,
+            0.001405641552992165,
+            0.03499535471200943,
+            -0.04828792065382004,
+            0.010717524215579033,
+            0.012520840391516685,
+            -0.018584391102194786,
+            -0.012812205590307713,
+            -0.02886880561709404,
+            0.009520563296973705,
+            -0.03044375404715538,
+            0.01814340613782406,
+            -0.022789504379034042,
+            0.021608293056488037,
+            -0.035688333213329315,
+            -0.0018732043681666255,
+            0.012623212300240993,
+            -0.0058076223358511925,
+            0.003575132926926017,
+            0.012473591603338718,
+            -0.007575502153486013,
+            -0.009323694743216038,
+            -0.018096158280968666,
+            0.023671476170420647,
+            0.05575317516922951,
+            -0.0003388600016478449,
+            0.00024116398708429188,
+            0.015930604189634323,
+            -0.01443440280854702,
+            -0.02620714344084263,
+            0.02953028306365013,
+            -0.016773201525211334,
+            0.015623488463461399,
+            0.03723178058862686,
+            -0.03468036651611328,
+            -0.027262357994914055,
+            0.01543449517339468,
+            -0.013725675642490387,
+            0.008347227238118649,
+            0.0020651512313634157,
+            0.02159254439175129,
+            -0.00283490726724267,
+            -0.026096895337104797,
+            0.028538066893815994,
+            0.006197422277182341,
+            0.009418192319571972,
+            -0.021671291440725327,
+            -0.011095511727035046,
+            -0.027325356379151344,
+            -0.023797471076250076,
+            -0.011867236346006393,
+            -0.010174167342483997,
+            0.006508474703878164,
+            -0.016820449382066727,
+            -0.057832106947898865,
+            -0.015229751355946064,
+            0.023907717317342758,
+            -0.01412728801369667,
+            0.05263477936387062,
+            0.01844264753162861,
+            0.016269218176603317,
+            0.0048035928048193455,
+            -0.04022418335080147,
+            0.030176013708114624,
+            -0.017560675740242004,
+            0.03143597021698952,
+            0.016088098287582397,
+            0.006839213892817497,
+            -0.0059611801989376545,
+            0.05058734491467476,
+            -0.02389196865260601,
+            0.007126641925424337,
+            -0.019686855375766754,
+            0.006473038345575333,
+            -0.02499443292617798,
+            -0.004508289974182844,
+            -0.00521701667457819,
+            -0.011465624906122684,
+            0.010063921101391315,
+            -0.011426251381635666,
+            0.024049462750554085,
+            0.02541966922581196,
+            0.0009026423213072121,
+            0.020301084965467453,
+            -0.030900489538908005,
+            -0.022206773981451988,
+            -0.004740594886243343,
+            0.0018899381393566728,
+            -0.02080507017672062,
+            -0.016332216560840607,
+            -0.040066689252853394,
+            -0.012788581661880016,
+            -0.0002187701902585104,
+            0.01212710328400135,
+            0.0040909284725785255,
+            0.02838057093322277,
+            -0.0017038973746821284,
+            -0.03209745138883591,
+            0.0071896398440003395,
+            -0.029514534398913383,
+            -0.0005906056612730026,
+            0.01391466986387968,
+            -0.017859915271401405,
+            -0.005945430602878332,
+            0.030648497864603996,
+            0.0016812575049698353,
+            -0.0150092588737607,
+            -0.016032975167036057,
+            0.005567443091422319,
+            -0.016237718984484673,
+            -0.0035554461646825075,
+            -0.01028441358357668,
+            0.03364089876413345,
+            -0.015332123264670372,
+            0.007780245505273342,
+            0.0005694422870874405,
+            4.2634346755221486e-05,
+            -0.03439687564969063,
+            -0.012890953570604324,
+            -0.019686855375766754,
+            0.013371312990784645,
+            0.0017777230823412538,
+            -0.027073364704847336,
+            0.007902303710579872,
+            0.0009838506812229753,
+            -0.004531914368271828,
+            -0.021324802190065384,
+            0.0032621119171380997,
+            0.005339075345546007,
+            -0.01771816983819008,
+            0.001696022693067789,
+            -0.033388908952474594,
+            -0.02831757441163063,
+            -0.029719278216362,
+            0.019419115036725998,
+            -0.0030435880180448294,
+            0.021183056756854057,
+            -0.013205942697823048,
+            0.012969700619578362,
+            -0.017261436209082603,
+            0.004590974655002356,
+            0.01570223644375801,
+            0.024663692340254784,
+            -0.03160921484231949,
+            0.013898920267820358,
+            0.01582035794854164,
+            0.021561045199632645,
+            -0.04523251950740814,
+            0.010709649883210659,
+            0.013631178997457027,
+            -0.0015227783005684614,
+            0.016332216560840607,
+            -0.010835645720362663,
+            0.03354640305042267,
+            -0.02093106508255005,
+            -0.0010749023640528321,
+            -0.005587129853665829,
+            0.013520932756364346,
+            0.01861589029431343,
+            0.012883078306913376,
+            -0.030412254855036736,
+            0.03110523149371147,
+            0.02352973073720932,
+            -0.012662585824728012,
+            0.04614599049091339,
+            -0.02656938135623932,
+            0.016458211466670036,
+            0.018600141629576683,
+            -0.02827032469213009,
+            0.0015660893404856324,
+            0.0045791626907885075,
+            -0.015631362795829773,
+            -0.014206035062670708,
+            0.025340920314192772,
+            0.00750462943688035,
+            -0.04346857964992523,
+            0.015001384541392326,
+            0.018237903714179993,
+            0.01119000930339098,
+            0.00433504581451416,
+            -0.023939216509461403,
+            0.009174074977636337,
+            0.028128579258918762,
+            0.0006875634426251054,
+            0.030475253239274025,
+            0.015064381994307041,
+            -0.02323048934340477,
+            0.011930234730243683,
+            -0.005795810371637344,
+            0.007406195160001516,
+            -0.01831665076315403,
+            0.00392752792686224,
+            -0.00747706787660718,
+            -0.014654895290732384,
+            -0.007232951000332832,
+            0.029640531167387962,
+            -0.003964932635426521,
+            -0.018883632495999336,
+            -0.03833424672484398,
+            0.015804607421159744,
+            0.009252822026610374,
+            -0.06709280610084534,
+            -0.00876458827406168,
+            -0.03386139124631882,
+            -0.02140354923903942,
+            -0.008906333707273006,
+            0.012749208137392998,
+            -0.015851857140660286,
+            -0.002389984205365181,
+            0.006154111120849848,
+            0.015599864535033703,
+            -0.01564711332321167,
+            0.02085231803357601,
+            0.006024178117513657,
+            0.005291827023029327,
+            -0.013993416912853718,
+            0.042051125317811966,
+            -0.007563690189272165,
+            -0.03814525157213211,
+            0.032176196575164795,
+            -0.012820080853998661,
+            0.003575132926926017,
+            0.024065213277935982,
+            -0.0019421083852648735,
+            -0.022002030164003372,
+            0.028963303193449974,
+            -0.026285890489816666,
+            -0.03512135148048401,
+            0.01522187702357769,
+            -0.006547848228365183,
+            -0.020285336300730705,
+            -0.02056882716715336,
+            -0.004681534133851528,
+            0.012733458541333675,
+            -0.02050582878291607,
+            0.018489895388484,
+            0.03266443312168121,
+            0.0052524530328810215,
+            -0.016206219792366028,
+            0.012457842007279396,
+            -0.0032247069757431746,
+            -0.0015030914219096303,
+            -0.005173705518245697,
+            0.024364452809095383,
+            -0.004055492114275694,
+            0.0033625150099396706,
+            -0.019119873642921448,
+            0.0030239010229706764,
+            -0.016088098287582397,
+            -0.012670460157096386,
+            -0.013457934372127056,
+            -0.025214925408363342,
+            -0.02493143454194069,
+            -0.04088566079735756,
+            0.03531034663319588,
+            0.024458950385451317,
+            -0.017324432730674744,
+            -0.010599403642117977,
+            -0.00861496850848198,
+            0.0013957980554550886,
+            0.028396321460604668,
+            -0.002578978193923831,
+            -0.0036597864236682653,
+            -0.02231702022254467,
+            0.011686117388308048,
+            -0.010583654046058655,
+            0.0014322188217192888,
+            -0.06296644359827042,
+            -0.01806465908885002,
+            0.034428372979164124,
+            0.007181765045970678,
+            -0.007996801286935806,
+            0.005917869042605162,
+            0.02376597188413143,
+            -0.020222337916493416,
+            -0.012481466867029667,
+            0.04390956461429596,
+            0.0027305670082569122,
+            0.0021458673290908337,
+            0.0070203328505158424,
+            -0.008347227238118649,
+            0.010961641557514668,
+            -0.004445292055606842,
+            -0.007729059550911188,
+            0.0005728874821215868,
+            0.010969515889883041,
+            0.009654434397816658,
+            0.00638247886672616,
+            -0.00442954245954752,
+            0.020710572600364685,
+            -0.011402626521885395,
+            -0.0043665445409715176,
+            0.006543911062180996,
+            0.019749853760004044,
+            -0.004783906042575836,
+            0.00025014611310325563,
+            -0.020301084965467453,
+            0.023986464366316795,
+            0.020773570984601974,
+            0.016158971935510635,
+            -0.0004537820350378752,
+            -0.008733089081943035,
+            -0.008059798739850521,
+            0.004401980899274349,
+            0.0030317758210003376,
+            0.017497677356004715,
+            0.028112830594182014,
+            -0.019482113420963287,
+            -0.016536958515644073,
+            -0.0012373188510537148,
+            0.0025336984544992447,
+            -0.004968962632119656,
+            0.03990919515490532,
+            0.02431720495223999,
+            0.005598941817879677,
+            -0.013969792984426022,
+            -0.0007269371417351067,
+            0.00891420803964138,
+            0.03505835309624672,
+            -0.026112645864486694,
+            -0.01571011170744896,
+            0.019907347857952118,
+            -0.0016330246580764651,
+            0.03433387726545334,
+            -0.021387800574302673,
+            0.014261158183217049,
+            -0.0026006337720900774,
+            0.08655916899442673,
+            -0.023372234776616096,
+            -0.005098895635455847,
+            -0.016174720600247383,
+            -0.014450152404606342,
+            0.004220861941576004,
+            -0.00459491228684783,
+            -0.003289673710241914,
+            0.031624965369701385,
+            -0.002029714873060584,
+            0.017182687297463417,
+            -0.0004545202828012407,
+            -0.008898458443582058,
+            0.008020425215363503,
+            -0.0035180412232875824,
+            0.018206404522061348,
+            -0.006681718863546848,
+            0.008969331160187721,
+            -0.003004214260727167,
+            0.0008957519312389195,
+            0.01322169229388237,
+            0.03291642293334007,
+            0.003939339891076088,
+            0.022537512704730034,
+            -0.02419120818376541,
+            0.018631640821695328,
+            0.007677873596549034,
+            0.016647204756736755,
+            -0.019986096769571304,
+            -0.017418930307030678,
+            -0.0007298901909962296,
+            0.013709926046431065,
+            -0.008575594052672386,
+            0.010906518436968327,
+            -0.003833030816167593,
+            0.039216216653585434,
+            -0.026396136730909348,
+            0.007047894410789013,
+            0.010835645720362663,
+            -0.004051554948091507,
+            0.009118951857089996,
+            0.02208077721297741,
+            0.017985912039875984,
+            0.005118582397699356,
+            -0.02297849766910076,
+            0.014135162346065044,
+            0.02293124981224537,
+            -0.01734018325805664,
+            0.0019470300758257508,
+            0.006539973430335522,
+            -0.0015119505114853382,
+            0.0004030883719678968,
+            0.005835183896124363,
+            -0.00024300962104462087,
+            -0.008331477642059326,
+            0.004512227606028318,
+            -0.007961364462971687,
+            -0.035562336444854736,
+            -0.0027384415734559298,
+            0.012851579114794731,
+            -0.00626435736194253,
+            0.031451720744371414,
+            0.029735026881098747,
+            0.019671106711030006,
+            -0.009811929427087307,
+            -0.006874650251120329,
+            0.004177550785243511,
+            -0.0040751793421804905,
+            -0.0011890861205756664,
+            -0.013607555069029331,
+            -0.014308406971395016,
+            0.014111538417637348,
+            -0.022474514320492744,
+            -0.00888270977884531,
+            0.0018338306108489633,
+            -0.005532006733119488,
+            -0.00459491228684783,
+            -0.0025002306792885065,
+            0.017481928691267967,
+            -0.005724937655031681,
+            0.02037983387708664,
+            0.01048915646970272,
+            -0.011528623290359974,
+            0.011308129876852036,
+            -0.026900120079517365,
+            0.009567812085151672,
+            -0.026963118463754654,
+            -0.010756897740066051,
+            0.009213448502123356,
+            -0.003992494195699692,
+            -0.017985912039875984,
+            -0.010473407804965973,
+            -0.016095973551273346,
+            -0.005220954306423664,
+            -0.00745738111436367,
+            -0.0006659079226665199,
+            0.013607555069029331,
+            0.008150358684360981,
+            -0.01601722650229931,
+            -0.016599956899881363,
+            -0.03694828972220421,
+            0.007843242958188057,
+            0.012993324548006058,
+            -0.025246424600481987,
+            -0.014119412750005722,
+            -0.007567627355456352,
+            0.008433849550783634,
+            -0.01233972143381834,
+            -0.0014351718127727509,
+            -0.0033802331890910864,
+            0.014883263036608696,
+            -0.02493143454194069,
+            0.006827401462942362,
+            -0.01929311826825142,
+            0.01558411493897438,
+            0.009583561681210995,
+            0.009260697290301323,
+            -0.03612931817770004,
+            -0.01885213330388069,
+            0.023561229929327965,
+            -0.008205481804907322,
+            -0.009504813700914383,
+            0.024049462750554085,
+            -0.012969700619578362,
+            0.011134886182844639,
+            0.019875850528478622,
+            0.01145775057375431,
+            -0.0026537880767136812,
+            0.0218287855386734,
+            -0.006213171873241663,
+            7.259527774294838e-05,
+            -0.004512227606028318,
+            0.024332953616976738,
+            0.0063588544726371765,
+            0.032790426164865494,
+            -0.015300624072551727,
+            -0.018521394580602646,
+            -0.005760374013334513,
+            0.0442875511944294,
+            0.007559752557426691,
+            -0.02723085880279541,
+            0.018174905329942703,
+            0.029136545956134796,
+            -0.0032916422933340073,
+            0.005445384420454502,
+            -0.038239747285842896,
+            0.0004163770063314587,
+            0.015505367890000343,
+            0.021923283115029335,
+            0.011292380280792713,
+            -0.009922175668179989,
+            -0.018363898620009422,
+            0.007181765045970678,
+            0.002352579263970256,
+            0.006039927247911692,
+            -0.03990919515490532,
+            0.011449875310063362,
+            -0.020946815609931946,
+            0.01970260590314865,
+            0.00861496850848198,
+            -0.009378817863762379,
+            0.022458765655755997,
+            0.0028152205049991608,
+            -0.017088191583752632,
+            0.014710018411278725,
+            -0.005980866961181164,
+            0.0006673844181932509,
+            0.03329440951347351,
+            -0.0036184440832585096,
+            0.012717708945274353,
+            -0.007815681397914886,
+            0.0061383615247905254,
+            0.0031046171206980944,
+            0.03505835309624672,
+            -0.0008091297931969166,
+            -0.014458026736974716,
+            0.0068628378212451935,
+            0.005417822860181332,
+            -0.0005571380024775863,
+            -0.005059522110968828,
+            0.0077448091469705105,
+            -0.012670460157096386,
+            0.0014400935033336282,
+            -0.003053431399166584,
+            -0.006315543316304684,
+            -0.0030396506190299988,
+            0.002159648109227419,
+            0.00047543758410029113,
+            -0.033136915415525436,
+            0.02178153768181801,
+            -0.0042484235018491745,
+            0.013843797147274017,
+            -0.005913931410759687,
+            -0.01531637366861105,
+            0.0023643914610147476,
+            0.02074207179248333,
+            -0.005492632742971182,
+            -0.011898735538125038,
+            -0.01595422811806202,
+            -0.02238001860678196,
+            -0.026301639154553413,
+            0.028522316366434097,
+            0.0024254205636680126,
+            -0.01607234962284565,
+            0.008481097407639027,
+            -0.009591436013579369,
+            -0.025545664131641388,
+            0.006520286668092012,
+            0.018395397812128067,
+            -0.041641637682914734,
+            0.016048725694417953,
+            -0.006953397300094366,
+            -0.0025573226157575846,
+            0.028648313134908676,
+            0.006732904817909002,
+            -0.003504260443150997,
+            0.014450152404606342,
+            -0.0027384415734559298,
+            0.011945984326303005,
+            0.02165554091334343,
+            0.0023072995245456696,
+            0.0066384077072143555,
+            0.014198160730302334,
+            0.015521117486059666,
+            -0.002368328860029578,
+            -0.01837964914739132,
+            0.027624595910310745,
+            -0.021545294672250748,
+            0.02025383710861206,
+            0.021261803805828094,
+            -0.026427635923027992,
+            0.029388537630438805,
+            -0.00997729878872633,
+            0.019182872027158737,
+            0.0034491370897740126,
+            -0.00592968100681901,
+            -0.01233972143381834,
+            0.004949275404214859,
+            0.007484942674636841,
+            0.009300070814788342,
+            0.010331662371754646,
+            0.044004060328006744,
+            -0.01364692859351635,
+            0.028601063415408134,
+            0.03401888534426689,
+            -0.00010723184095695615,
+            0.02255326136946678,
+            -0.018584391102194786,
+            -0.02123030461370945,
+            0.007488879840821028,
+            -0.015780983492732048,
+            -0.03025476075708866,
+            0.0032483311370015144,
+            -0.01452889945358038,
+            -0.008284228853881359,
+            -0.016269218176603317,
+            -0.0019539203494787216,
+            -0.0022324894089251757,
+            -0.0009164231596514583,
+            0.004220861941576004,
+            0.020348334684967995,
+            0.026049647480249405,
+            0.02376597188413143,
+            0.01024504005908966,
+            0.00747706787660718,
+            0.01148924883455038,
+            -0.009930050000548363,
+            0.026175644248723984,
+            -0.0011713679414242506,
+            -0.0028624688275158405,
+            -0.0172929335385561,
+            0.007055769208818674,
+            -0.00029013503808528185,
+            -0.005110707599669695,
+            0.007425881922245026,
+            -0.022175274789333344,
+            -0.0015906979097053409,
+            -0.04687046632170677,
+            -0.002504168078303337,
+            -0.0007677873945795,
+            0.010394659824669361,
+            0.03965720161795616,
+            -0.007449506316334009,
+            -0.017922913655638695,
+            0.008284228853881359,
+            -0.01685194857418537,
+            0.004098803270608187,
+            -0.01814340613782406,
+            -0.015591990202665329,
+            -0.010756897740066051,
+            -0.02414396032691002,
+            -0.009315820410847664,
+            -0.013812297955155373,
+            0.015143129974603653,
+            -0.00876458827406168,
+            0.011229382827877998,
+            -0.018710387870669365,
+            0.00041736135608516634,
+            -0.006221046671271324,
+            -0.008433849550783634,
+            -0.002559291198849678,
+            -0.020348334684967995,
+            -0.012379094958305359,
+            0.022710757330060005,
+            -0.008780337870121002,
+            -0.029609031975269318,
+            -0.006311606150120497,
+            0.004232673905789852,
+            -0.014859639108181,
+            0.0037365653552114964,
+            0.00592968100681901,
+            -0.005917869042605162,
+            -0.008985080756247044,
+            0.00961506087332964,
+            0.018726136535406113,
+            0.012457842007279396,
+            -0.0028644376434385777,
+            0.00395705783739686,
+            0.02291550114750862,
+            0.02348248101770878,
+            -0.0454845130443573,
+            -0.005102833267301321,
+            0.017686670646071434,
+            0.016347965225577354,
+            -0.0070321448147296906,
+            0.018411148339509964,
+            -0.01507225725799799,
+            0.011827862821519375,
+            -0.02250601351261139,
+            -0.00612261239439249,
+            -0.026915868744254112,
+            0.00845747347921133,
+            0.01734018325805664,
+            -0.006768341176211834,
+            0.035215847194194794,
+            0.020600326359272003,
+            -0.016788950189948082,
+            0.020994063466787338,
+            -0.0013150819577276707,
+            0.011174259707331657,
+            -0.027089113369584084,
+            0.01364692859351635,
+            -0.009229198098182678,
+            0.008236980065703392,
+            0.014206035062670708,
+            -0.020049093291163445,
+            -0.025340920314192772,
+            -0.01222159992903471,
+            0.012174352072179317,
+            0.003065243363380432,
+            -0.0027896275278180838,
+            0.022222522646188736,
+            0.015143129974603653,
+            -0.01197748351842165,
+            -0.01740317977964878,
+            0.023561229929327965,
+            0.01276495773345232,
+            0.015418745577335358,
+            -0.0032581747509539127,
+            -0.0033290472347289324,
+            -0.017009442672133446,
+            0.0005093974177725613,
+            -0.02450619824230671,
+            -0.001176289631985128,
+            -0.002449044957756996,
+            0.021529546007514,
+            -0.007681811228394508,
+            -0.006224983837455511,
+            -0.007177827879786491,
+            -0.02808133140206337,
+            -0.000330000912072137,
+            0.00023365837114397436,
+            0.01252871472388506,
+            0.007658186834305525,
+            0.009473315440118313,
+            0.017481928691267967,
+            -0.026364637538790703,
+            0.002179334871470928,
+            -0.019482113420963287,
+            0.004126364830881357,
+            -0.00928432121872902,
+            -0.014292657375335693,
+            0.01033953670412302,
+            0.003470792667940259,
+            -0.00480753043666482,
+            0.01012691855430603,
+            0.003260143334046006,
+            -0.017387431114912033,
+            0.029939770698547363,
+            -0.004006274975836277,
+            0.002279737964272499,
+            0.003663723822683096,
+            0.005736749619245529,
+            0.0019736073445528746,
+            0.02827032469213009,
+            -0.025388170033693314,
+            -0.016773201525211334,
+            -0.018600141629576683,
+            0.0024372327607125044,
+            0.004988649394363165,
+            0.005386323668062687,
+            -0.0010079670464619994,
+            -0.00013190193567425013,
+            0.002657725475728512,
+            -0.0070321448147296906,
+            0.027876587584614754,
+            0.0040909284725785255,
+            -0.03382989391684532,
+            -0.024395952001214027,
+            -0.04038167744874954,
+            0.008307852782309055,
+            -0.01982860080897808,
+            -0.029766526073217392,
+            -0.010008797980844975,
+            -0.02274225652217865,
+            -0.035940323024988174,
+            0.018474144861102104,
+            0.020757820457220078,
+            -0.02025383710861206,
+            -0.008205481804907322,
+            -0.0028447506483644247,
+            -0.013142945244908333,
+            0.010048171505331993,
+            -0.0029805900994688272,
+            -0.016269218176603317,
+            0.010355286300182343,
+            0.014765141531825066,
+            0.0068628378212451935,
+            -0.00867009162902832,
+            -0.016347965225577354,
+            0.00028521331842057407,
+            0.023813221603631973,
+            0.0064651635475456715,
+            -0.01107188779860735,
+            0.007925928570330143,
+            0.006398227997124195,
+            -0.015111630782485008,
+            0.007437694352120161,
+            -0.023088743910193443,
+            -0.000766310840845108,
+            0.0025021994952112436,
+            0.014056415297091007,
+            -0.01644246280193329,
+            0.003551508765667677,
+            0.005618628580123186,
+            -0.0036873482167720795,
+            0.003545602783560753,
+            -0.018757635727524757,
+            0.00671715522184968,
+            -0.009748931042850018,
+            0.0019913255237042904,
+            0.028128579258918762,
+            -0.007232951000332832,
+            -0.005161893554031849,
+            -0.032412439584732056,
+            0.01642671227455139,
+            0.03360940143465996,
+            0.013678427785634995,
+            -0.021513795480132103,
+            -0.015985727310180664,
+            0.004283859860152006,
+            -0.0016468055546283722,
+            -0.019198622554540634,
+            0.015473868697881699,
+            0.019450614228844643,
+            0.012197976000607014,
+            0.015505367890000343,
+            0.013166569173336029,
+            0.014379279688000679,
+            -0.025451166555285454,
+            0.001903718919493258,
+            -0.0020454642362892628,
+            -0.02170279063284397,
+            0.012024731375277042,
+            0.02061607502400875,
+            -0.02378172241151333,
+            -0.003651911858469248,
+            0.0016153064789250493,
+            0.02165554091334343,
+            -0.003827124834060669,
+            -0.002248239004984498,
+            0.01808040775358677,
+            0.010252914391458035,
+            0.015119505114853382,
+            0.004795718006789684,
+            0.061045002192258835,
+            -0.003384170588105917,
+            0.007122704293578863,
+            0.008575594052672386,
+            -0.005106770433485508,
+            -0.01965535618364811,
+            -0.0010197791270911694,
+            -0.026537882164120674,
+            0.006654157303273678,
+            0.007752683945000172,
+            -0.02778209187090397,
+            -0.005902119446545839,
+            -0.009008705615997314,
+            -0.006595096550881863,
+            0.018048910424113274,
+            0.0047484696842730045,
+            -0.0020179026760160923,
+            0.006260420195758343,
+            0.037987757474184036,
+            0.014686394482851028,
+            -0.009504813700914383,
+            0.0023072995245456696,
+            0.010150542482733727,
+            0.007760558743029833,
+            -0.008268479257822037,
+            0.009040203876793385,
+            0.0004434464208316058,
+            0.004461041651666164,
+            -0.00800467561930418,
+            0.039625704288482666,
+            -0.009040203876793385,
+            0.0009188840049318969,
+            0.0023801408242434263,
+            -0.014426528476178646,
+            0.010938017629086971,
+            0.002990433480590582,
+            0.00729201128706336,
+            -0.007095142733305693,
+            0.0013308314373716712,
+            0.002222646027803421,
+            -0.0020001844968646765,
+            -0.0012343658600002527,
+            -0.009559936821460724,
+            -0.01655270904302597,
+            -0.005587129853665829,
+            0.013016949407756329,
+            0.0005856839707121253,
+            0.0052367039024829865,
+            -0.01812765747308731,
+            -0.0072447629645466805,
+            -0.0007126641576178372,
+            -0.011134886182844639,
+            0.007221138570457697,
+            0.024899935349822044,
+            -0.027498601004481316,
+            -0.004283859860152006,
+            0.014166661538183689,
+            -0.00015171183622442186,
+            0.007327447645366192,
+            0.014607647433876991,
+            -0.013119320385158062,
+            0.008000737987458706,
+            -0.016040850430727005,
+            -0.014347780495882034,
+            0.0007456396706402302,
+            -0.006492725107818842,
+            0.0023506106808781624,
+            -0.01307207252830267,
+            -0.0012058198917657137,
+            0.01279645599424839,
+            0.0006412993534468114,
+            0.012079854495823383,
+            0.014544649049639702,
+            -0.00679196510463953,
+            -0.00038364759529940784,
+            0.012694085016846657,
+            -0.012780707329511642,
+            -0.024128209799528122,
+            0.0007146328571252525,
+            0.023876218125224113,
+            -0.00492958864197135,
+            0.00383499963209033,
+            -0.0028506566304713488,
+            0.0024825125001370907,
+            -0.012662585824728012,
+            -0.013001199811697006,
+            -0.012213725596666336,
+            0.0209153164178133,
+            0.004571287892758846,
+            -0.022758005186915398,
+            0.0008258636225946248,
+            0.0055831922218203545,
+            0.009103202261030674,
+            -0.017497677356004715,
+            0.004000368993729353,
+            0.02995551936328411,
+            -0.01479664072394371,
+            -0.0041814884170889854,
+            -0.0008224184275604784,
+            -0.013812297955155373,
+            0.008044049143791199,
+            0.017088191583752632,
+            0.005484757944941521,
+            0.016363713890314102,
+            -0.003575132926926017,
+            -0.009441816247999668,
+            0.03038075566291809,
+            0.015308499336242676,
+            0.004185425583273172,
+            -0.006536036264151335,
+            -0.012788581661880016,
+            -0.028333323076367378,
+            0.004374419339001179,
+            0.010623027570545673,
+            -0.014292657375335693,
+            -0.0005876526702195406,
+            -0.012276723049581051,
+            -0.018001660704612732,
+            -0.004488603211939335,
+            -0.006524223834276199,
+            0.012300347909331322,
+            -0.00976468063890934,
+            0.010804146528244019,
+            -0.0003120366600342095,
+            -0.0035377279855310917,
+            -0.002352579263970256,
+            0.0002889046154450625,
+            -0.009410317055881023,
+            -0.03291642293334007,
+            -0.028412070125341415,
+            -0.008898458443582058,
+            -0.008142483420670033,
+            0.006528161466121674,
+            0.0009907410712912679,
+            -0.030664246529340744,
+            0.010300163179636002,
+            -0.00012433479423634708,
+            -0.014843889512121677,
+            0.015190377831459045,
+            0.009898550808429718,
+            -0.002703005215153098,
+            -0.007528253830969334,
+            -0.007807807065546513,
+            -0.02874280884861946,
+            0.01589910499751568,
+            0.0021872096695005894,
+            0.009260697290301323,
+            -0.014237534254789352,
+            0.0010424189968034625,
+            -0.021009812131524086,
+            0.01727718487381935,
+            -0.026474883779883385,
+            -0.015095881186425686,
+            -0.01522187702357769,
+            -0.0022128026466816664,
+            -0.0060320524498820305,
+            -0.017781168222427368,
+            -0.021907532587647438,
+            -0.006161985918879509,
+            -0.025640161707997322,
+            -0.022899750620126724,
+            -0.0017609893111512065,
+            -0.0034353563096374273,
+            0.010583654046058655,
+            -0.007040019612759352,
+            0.027986833825707436,
+            0.01619047112762928,
+            0.006579347420483828,
+            -0.011560121551156044,
+            -0.013442184776067734,
+            -0.0020198714919388294,
+            0.0025179488584399223,
+            -0.009158325381577015,
+            0.002384078223258257,
+            0.019797101616859436,
+            -0.0036184440832585096,
+            0.0045240395702421665,
+            0.004106678068637848,
+            -0.004035805352032185,
+            0.016836199909448624,
+            0.007988926023244858,
+            0.011788489297032356,
+            0.006315543316304684,
+            0.0016094004968181252,
+            0.03379839286208153,
+            0.016473960131406784,
+            -0.0035869451239705086,
+            0.0022324894089251757,
+            0.020962564274668694,
+            -0.02170279063284397,
+            -0.01571011170744896,
+            0.015450244769454002,
+            0.001383001683279872,
+            -0.03546784073114395,
+            0.02067907340824604,
+            0.01661570556461811,
+            -0.009796179831027985,
+            -0.010764773003757,
+            -0.007961364462971687,
+            -0.00021323325927369297,
+            -0.005181580316275358,
+            0.008544095791876316,
+            -0.011560121551156044,
+            0.013426436111330986,
+            0.006496662274003029,
+            -0.0064651635475456715,
+            -0.014269033446907997,
+            0.01422178465873003,
+            0.027971085160970688,
+            0.0035790703259408474,
+            -0.04819342494010925,
+            -0.012276723049581051,
+            0.01704094186425209,
+            -0.02140354923903942,
+            -0.025797655805945396,
+            0.00011172290396643803,
+            -0.0075715649873018265,
+            -0.023608477786183357,
+            -0.007063644006848335,
+            -0.00392752792686224,
+            -0.016836199909448624,
+            0.012883078306913376,
+            0.01064665149897337,
+            0.012426343746483326,
+            0.008205481804907322,
+            -0.015867605805397034,
+            -0.012064104899764061,
+            -0.042366113513708115,
+            0.00788655411452055,
+            -0.024836936965584755,
+            -0.021939031779766083,
+            0.004902027081698179,
+            -0.006827401462942362,
+            0.008740964345633984,
+            0.01452889945358038,
+            0.039153218269348145,
+            0.01255233958363533,
+            -0.015080131590366364,
+            -0.025687409564852715,
+            -0.017907164990901947,
+            -0.007961364462971687,
+            0.0022758005652576685,
+            -0.0074180071242153645,
+            -0.0035003230441361666,
+            0.0033192038536071777,
+            0.002246270189061761,
+            0.029309790581464767,
+            0.001104432623833418,
+            0.018962379544973373,
+            -3.389830453670584e-05,
+            0.00928432121872902,
+            -0.0038920913357287645,
+            -0.025025932118296623,
+            0.006213171873241663,
+            0.004106678068637848,
+            0.00794167723506689,
+            0.0068628378212451935,
+            0.005106770433485508,
+            0.016584208235144615,
+            -0.015867605805397034,
+            0.005394198466092348,
+            -0.026222892105579376,
+            -0.006469100713729858,
+            -0.014418653212487698,
+            0.0294042881578207,
+            0.014733643271028996,
+            -0.02327773906290531,
+            -0.013174444437026978,
+            -0.018993878737092018,
+            0.017970161512494087,
+            -0.008685841225087643,
+            0.009756805375218391,
+            0.01355243194848299,
+            0.01040253508836031,
+            0.006205297075212002,
+            0.0053548249416053295,
+            -0.005012273322790861,
+            0.007736934348940849,
+            0.02310449443757534,
+            -0.007106955163180828,
+            0.008000737987458706,
+            -0.0013849702663719654,
+            -0.0041814884170889854,
+            -0.014812390320003033,
+            -0.01722993701696396,
+            -0.0012570057297125459,
+            -0.00373853393830359,
+            0.01935611665248871,
+            -0.017623674124479294,
+            0.020001845434308052,
+            -0.011906610801815987,
+            0.005181580316275358,
+            -0.026490632444620132,
+            -0.008481097407639027,
+            0.01644246280193329,
+            -0.015505367890000343,
+            0.0024195145815610886,
+            -0.015458119101822376,
+            -0.003754283534362912,
+            0.006665969267487526,
+            0.001975575927644968,
+            0.001650742837227881,
+            0.005575317423790693,
+            0.005201267544180155,
+            0.016662955284118652,
+            0.008110984228551388,
+            -0.017450429499149323,
+            -0.0032562059350311756,
+            0.008370851166546345,
+            -0.015135254710912704,
+            0.006961272098124027,
+            0.005862745922058821,
+            0.0009361099801026285,
+            0.04082266613841057,
+            0.0066384077072143555,
+            0.002084837993606925,
+            -0.009898550808429718,
+            0.0037798762787133455,
+            -0.036727797240018845,
+            0.02904205024242401,
+            -0.0016576332272961736,
+            0.016221968457102776,
+            0.001489310641773045,
+            -0.014465902000665665,
+            0.0045791626907885075,
+            -0.005154018756002188,
+            0.014828139916062355,
+            -0.006287981756031513,
+            0.006969146896153688,
+            -0.00131114455871284,
+            0.017387431114912033,
+            -0.007339260075241327,
+            -0.018899381160736084,
+            0.009142575785517693,
+            -0.011316005140542984,
+            0.0007323510362766683,
+            -0.009693807922303677,
+            0.0055280691012740135,
+            0.01352880708873272,
+            0.0070203328505158424,
+            0.005480820778757334,
+            -0.03808225318789482,
+            -0.008622842840850353,
+            -0.003271955531090498,
+            0.002777815330773592,
+            -0.0024687317200005054,
+            -0.023970715701580048,
+            -0.024584945291280746,
+            -0.0002655264688655734,
+            0.02170279063284397,
+            -0.006587222218513489,
+            -0.0011831800220534205,
+            0.009118951857089996,
+            0.0015503398608416319,
+            -0.011749115772545338,
+            -0.006980959326028824,
+            0.007421944756060839,
+            0.021246055141091347,
+            -0.025514164939522743,
+            -0.020883817225694656,
+            0.012048356235027313,
+            0.0029274355620145798,
+            -0.019875850528478622,
+            -0.01033953670412302,
+            -0.0020651512313634157,
+            0.008512596599757671,
+            0.03439687564969063,
+            -0.017686670646071434,
+            0.001204835600219667,
+            0.0008947675814852118,
+            0.014080039225518703,
+            -0.026301639154553413,
+            -0.005043772514909506,
+            -0.01112701091915369,
+            -0.010418283753097057,
+            0.01148924883455038,
+            -0.01631646603345871,
+            0.0070518315769732,
+            -0.002840813249349594,
+            -0.002578978193923831,
+            -0.014064289629459381,
+            0.0009021501755341887,
+            0.0013810329837724566,
+            0.0016064473893493414,
+            0.03316841647028923,
+            -0.009567812085151672,
+            -0.013505183160305023,
+            0.0018968285294249654,
+            0.00041391613194718957,
+            0.014922636561095715,
+            0.010465532541275024,
+            0.01976560428738594,
+            -0.008331477642059326,
+            0.001558214658871293,
+            0.00961506087332964,
+            0.014623397029936314,
+            0.0005468024173751473,
+            0.042240116745233536,
+            -0.005795810371637344,
+            -0.006725030019879341,
+            -0.011788489297032356,
+            -0.005390261299908161,
+            -0.01233972143381834,
+            0.01491476222872734,
+            0.00269119325093925,
+            0.010371035896241665,
+            -0.0020828694105148315,
+            -0.008945707231760025,
+            0.012111353687942028,
+            0.007445569150149822,
+            0.010560029186308384,
+            0.013205942697823048,
+            -0.004014149773865938,
+            -0.02001759596168995,
+            0.010063921101391315,
+            -0.010048171505331993,
+            0.006059614475816488,
+            -0.001052262494340539,
+            0.008866960182785988,
+            -0.007996801286935806,
+            0.018631640821695328,
+            -0.0028703436255455017,
+            0.017607923597097397,
+            0.015560491010546684,
+            -0.019545109942555428,
+            0.017072441056370735,
+            0.01782841607928276,
+            -0.010788396932184696,
+            0.012780707329511642,
+            -0.01648971065878868,
+            -0.010237164795398712,
+            -0.0052524530328810215,
+            0.02176578715443611,
+            -0.013142945244908333,
+            -0.013324064202606678,
+            -0.010922268033027649,
+            0.00395508948713541,
+            -0.007988926023244858,
+            0.017324432730674744,
+            0.00588243268430233,
+            -0.009890676476061344,
+            -0.019686855375766754,
+            0.015505367890000343,
+            0.004433480091392994,
+            0.005213079508394003,
+            0.022474514320492744,
+            -0.007898367010056973,
+            0.005878495052456856,
+            0.014717893674969673,
+            0.030349256470799446,
+            -0.008323602378368378,
+            0.02831757441163063,
+            -0.011111261323094368,
+            0.00266756908968091,
+            -0.0037601895164698362,
+            -0.016158971935510635,
+            -0.012402718886733055,
+            -8.889845776138827e-05,
+            0.0050280229188501835,
+            0.012930327095091343,
+            -0.028601063415408134,
+            0.00861496850848198,
+            0.026222892105579376,
+            0.0027502537705004215,
+            0.008937832899391651,
+            0.0026537880767136812,
+            0.006890399381518364,
+            0.005386323668062687,
+            0.0029943708796054125,
+            -0.022364268079400063,
+            -0.007126641925424337,
+            0.00323848775587976,
+            -0.010851395316421986,
+            0.02104131132364273,
+            0.014355655759572983,
+            0.01067027635872364,
+            -0.002246270189061761,
+            -0.0012619274202734232,
+            0.025151927024126053,
+            0.01959235966205597,
+            0.004051554948091507,
+            -0.001164477551355958,
+            0.006854963023215532,
+            0.0023191117215901613,
+            0.011875111609697342,
+            -0.014190285466611385,
+            -0.014166661538183689,
+            0.007331385277211666,
+            0.00262228911742568,
+            0.010890768840909004,
+            -0.01680470071732998,
+            -0.006449413951486349,
+            -0.003303454490378499,
+            0.01746617816388607,
+            -0.009993048384785652,
+            -0.0018879694398492575,
+            0.0012599588371813297,
+            0.015253376215696335,
+            0.00039250668487511575,
+            -0.0032680181320756674,
+            -0.004724845290184021,
+            -0.014119412750005722,
+            0.0047996556386351585,
+            -0.009725307114422321,
+            0.004894152283668518,
+            -0.005661939736455679,
+            0.00881971139460802,
+            -0.007311698514968157,
+            0.04378356784582138,
+            0.027876587584614754,
+            0.008512596599757671,
+            0.017371682450175285,
+            -0.003986588213592768,
+            0.0017127564642578363,
+            -0.016119597479701042,
+            0.020222337916493416,
+            0.005598941817879677,
+            4.361868923297152e-05,
+            0.003966901451349258,
+            0.024915684014558792,
+            0.004728782922029495,
+            0.009733181446790695,
+            0.0006260420195758343,
+            0.006394290830940008,
+            0.0026439446955919266,
+            0.009410317055881023,
+            -0.018474144861102104,
+            -0.01507225725799799,
+            -0.014497400261461735,
+            0.02316749282181263,
+            -0.005512319505214691,
+            -0.008567719720304012,
+            0.013733550906181335,
+            0.011111261323094368,
+            -0.002498262096196413,
+            -0.014670644886791706,
+            -0.005693438928574324,
+            -0.0008736042072996497,
+            -0.005161893554031849,
+            -0.016269218176603317,
+            0.00297862128354609,
+            0.0020316834561526775,
+            -0.012182226404547691,
+            0.021372050046920776,
+            -0.012166476808488369,
+            -0.020395582541823387,
+            -0.003490479663014412,
+            -0.006843151059001684,
+            -0.04022418335080147,
+            -0.019056877121329308,
+            0.011906610801815987,
+            0.010788396932184696,
+            -0.008591343648731709,
+            -0.03480635955929756,
+            0.021261803805828094,
+            0.0060832384042441845,
+            0.03808225318789482,
+            -0.026065398007631302,
+            0.0023762034252285957,
+            0.0006777200032956898,
+            0.00397477624937892,
+            -0.011268756352365017,
+            -4.0081209590425715e-05,
+            0.019419115036725998,
+            -0.0007800916791893542,
+            0.00812673382461071,
+            -0.00542963482439518,
+            -0.010922268033027649,
+            0.014686394482851028,
+            -0.007209326606243849,
+            0.012906703166663647,
+            -0.006161985918879509,
+            -0.014442277140915394,
+            0.0032424251548945904,
+            0.016631456092000008,
+            0.012607462704181671,
+            -0.0017993786605075002,
+            -0.014670644886791706,
+            -0.009567812085151672,
+            0.00433504581451416,
+            -0.013331938534975052,
+            -0.012890953570604324,
+            0.02378172241151333,
+            -0.0166787039488554,
+            0.018111906945705414,
+            0.013205942697823048,
+            -0.017182687297463417,
+            0.004839029163122177,
+            0.006035990081727505,
+            0.006008428521454334,
+            0.007567627355456352,
+            -0.015064381994307041,
+            -0.019434863701462746,
+            0.004476791247725487,
+            0.026128394529223442,
+            -0.00240376521833241,
+            -0.006020240485668182,
+            0.012544464319944382,
+            -0.0026951306499540806,
+            0.008512596599757671,
+            -0.023387985303997993,
+            -0.0023821096401661634,
+            0.010993140749633312,
+            0.015568365342915058,
+            0.015686485916376114,
+            0.02335648611187935,
+            0.009181949310004711,
+            0.012402718886733055,
+            -0.012323971837759018,
+            0.005177643150091171,
+            0.022726505994796753,
+            -0.009630809538066387,
+            -0.018883632495999336,
+            0.011379002593457699,
+            0.01763942278921604,
+            0.023687224835157394,
+            0.0018456426914781332,
+            -0.010268663987517357,
+            0.030916238203644753,
+            0.012284598313272,
+            -0.009063828736543655,
+            0.014135162346065044,
+            -0.010906518436968327,
+            0.024962933734059334,
+            0.010237164795398712,
+            0.016174720600247383,
+            -0.01219010166823864,
+            -0.010835645720362663,
+            0.006646282505244017,
+            -0.011567996814846992,
+            -0.007410132791846991,
+            0.01249721646308899,
+            -0.020175090059638023,
+            -0.003384170588105917,
+            -0.031325723975896835,
+            0.00638247886672616,
+            0.003254237351939082,
+            0.0035790703259408474,
+            -0.02461644448339939,
+            0.005658002570271492,
+            -0.006146236322820187,
+            0.008079485967755318,
+            -0.011048263870179653,
+            -0.031262725591659546,
+            0.0039058721158653498,
+            -0.006496662274003029,
+            0.002059245016425848,
+            -0.01433990616351366,
+            0.01589122973382473,
+            0.007114829961210489,
+            -0.021088559180498123,
+            0.0010315912077203393,
+            -0.01904112659394741,
+            -0.0072565749287605286,
+            0.0003491956158541143,
+            -0.0023880156222730875,
+            -0.016347965225577354,
+            0.011363252997398376,
+            0.005886369850486517,
+            -0.02527792379260063,
+            -0.006937648169696331,
+            -0.0024687317200005054,
+            0.0007170937024056911,
+            0.016048725694417953,
+            -0.006055676843971014,
+            0.0022324894089251757,
+            0.033388908952474594,
+            0.015867605805397034,
+            -0.008488972671329975,
+            -0.025624411180615425,
+            -0.0028644376434385777,
+            -0.02359272725880146,
+            0.015631362795829773,
+            -0.0007953489548526704,
+            0.005650127772241831,
+            -0.029971269890666008,
+            0.005150081589818001,
+            0.006709280423820019,
+            0.01746617816388607,
+            0.006488787941634655,
+            0.01033953670412302,
+            0.026837121695280075,
+            -0.005106770433485508,
+            -0.001383001683279872,
+            -1.0535543879086617e-05,
+            0.00679196510463953,
+            0.018411148339509964,
+            -0.0036184440832585096,
+            0.01812765747308731,
+            -0.009607185609638691,
+            0.0029136547818779945,
+            -0.012654710561037064,
+            0.0035554461646825075,
+            -0.003462917869910598,
+            -0.013095696456730366,
+            -9.954166307579726e-05,
+            0.012686209753155708,
+            0.003395982552319765,
+            -0.0012727552093565464,
+            -0.018836384639143944,
+            -0.003833030816167593,
+            -0.0012530683306977153,
+            -0.021261803805828094,
+            0.0029628719203174114,
+            -0.01970260590314865,
+            -0.01124513242393732,
+            -0.02778209187090397,
+            -0.0025435418356209993,
+            0.04438204690814018,
+            0.00955206248909235,
+            -0.012331846170127392,
+            0.002647882094606757,
+            -0.02067907340824604,
+            0.0009956627618521452,
+            -0.012213725596666336,
+            -0.0036479744594544172,
+            -0.006886462215334177,
+            -0.013820173218846321,
+            0.014631271362304688,
+            0.01735593192279339,
+            -0.00404761778190732,
+            -0.006701405625790358,
+            0.012481466867029667,
+            -0.01070177461951971,
+            -0.009504813700914383,
+            -0.017576424404978752,
+            -0.017922913655638695,
+            -0.00785505585372448,
+            0.018883632495999336,
+            -0.005795810371637344,
+            0.02127755433320999,
+            0.008544095791876316,
+            0.0034353563096374273,
+            -0.00692977337166667,
+            0.0015168722020462155,
+            -0.021498046815395355,
+            -0.0006855947431176901,
+            0.006287981756031513,
+            -0.01433990616351366,
+            0.016599956899881363,
+            -0.006815589498728514,
+            -0.0031361160799860954,
+            0.008725214749574661,
+            0.004413793329149485,
+            -0.008725214749574661,
+            -0.005516257137060165,
+            -0.017135439440608025,
+            0.018820634111762047,
+            -0.018332399427890778,
+            -0.0027502537705004215,
+            0.00022246147273108363,
+            -0.009969423525035381,
+            0.012158602476119995,
+            0.015591990202665329,
+            0.007481005042791367,
+            -0.010378910228610039,
+            0.006610846146941185,
+            -0.010292287915945053,
+            0.0221437755972147,
+            -0.001872219960205257,
+            0.01612747274339199,
+            0.013024823740124702,
+            0.005433572456240654,
+            0.006284044589847326,
+            0.001713740755803883,
+            -0.012544464319944382,
+            -0.006949460133910179,
+            0.011064013466238976,
+            -0.02571890875697136,
+            -0.01349730882793665,
+            0.005917869042605162,
+            0.002801439492031932,
+            -0.01148924883455038,
+            -0.0007451475248672068,
+            -0.009205574169754982,
+            0.00037798762787133455,
+            -0.008481097407639027,
+            0.016710203140974045,
+            -0.03433387726545334,
+            -0.00861496850848198,
+            0.0017580362036824226,
+            0.031136730685830116,
+            -0.008449598215520382,
+            0.01619047112762928,
+            0.007532190997153521,
+            0.006488787941634655,
+            0.023687224835157394,
+            0.001509981811977923,
+            0.007662124466150999,
+            0.02116730809211731,
+            -0.011095511727035046,
+            0.00785505585372448,
+            0.0058036851696670055,
+            -0.01995459757745266,
+            0.025120427832007408,
+            -0.0061856103129684925,
+            0.005906056612730026,
+            0.002665600273758173,
+            0.001731458934955299,
+            0.02571890875697136,
+            0.027624595910310745,
+            -0.00870158988982439,
+            -0.005665877368301153,
+            -0.013379187323153019,
+            0.027262357994914055,
+            -0.004945338238030672,
+            0.007610938511788845,
+            -0.010197791270911694,
+            0.002689224435016513,
+            0.0007668030448257923,
+            -0.01024504005908966,
+            0.021482296288013458,
+            -0.016032975167036057,
+            -0.011875111609697342,
+            -0.017560675740242004,
+            -0.002472669119015336,
+            -0.024222707375884056,
+            0.013536682352423668,
+            0.0025829155929386616,
+            -0.003352671628817916,
+            0.0023821096401661634,
+            0.006484850309789181,
+            -0.003496385645121336,
+            -0.00940244272351265,
+            -0.0020887753926217556,
+            -0.01170186698436737,
+            -0.020348334684967995,
+            0.0166787039488554,
+            -0.007248700130730867,
+            -0.002116336952894926,
+            -0.0012127102818340063,
+            0.009811929427087307,
+            0.014985634945333004,
+            0.02129330299794674,
+            0.002915623364970088,
+            0.0066344705410301685,
+            0.007311698514968157,
+            -0.0019332492956891656,
+            -0.012032606638967991,
+            0.020427081733942032,
+            0.0197183545678854,
+            0.0003445199690759182,
+            -0.0031695838551968336,
+            -0.0109143927693367,
+            -0.035562336444854736,
+            -0.009733181446790695,
+            0.0008125749882310629,
+            0.011938109062612057,
+            -0.012442093342542648,
+            -0.007103017531335354,
+            0.039027221500873566,
+            -0.003220769576728344,
+            0.02110430970788002,
+            -0.004189362749457359,
+            3.842013029498048e-05,
+            0.01674170233309269,
+            0.004606724251061678,
+            0.012142852880060673,
+            -0.01264683622866869,
+            -0.02327773906290531,
+            0.002559291198849678,
+            -0.012961826287209988,
+            0.002697099233046174,
+            0.0004523055104073137,
+            -0.0002670029934961349,
+            -0.019497862085700035,
+            -0.008504721336066723,
+            0.005189455114305019,
+            -0.01128450594842434,
+            0.010040296241641045,
+            -0.004157864023000002,
+            0.017387431114912033,
+            -0.0030179950408637524,
+            0.0037897196598351,
+            -0.008583469316363335,
+            -0.006811652332544327,
+            0.0011575871612876654,
+            -0.0017304746434092522,
+            -0.017198437824845314,
+            -8.342305227415636e-05,
+            -0.0020887753926217556,
+            -0.0003410747740417719,
+            0.008410224691033363,
+            0.008378725498914719,
+            0.01328469067811966,
+            0.01385954674333334,
+            -0.0009119936148636043,
+            0.006996708456426859,
+            0.009008705615997314,
+            0.02735685557126999,
+            -0.007185702212154865,
+            0.0068077147006988525,
+            0.002291549928486347,
+            0.008012549951672554,
+            -0.012946076691150665,
+            -0.03285342454910278,
+            0.015426619909703732,
+            -0.015780983492732048,
+            -0.0048035928048193455,
+            -0.014717893674969673,
+            0.01040253508836031,
+            -0.004445292055606842,
+            0.0010876987362280488,
+            0.006098988000303507,
+            0.00845747347921133,
+            0.004937463440001011,
+            0.019308868795633316,
+            -0.007213264238089323,
+            0.0071935770101845264,
+            0.021309053525328636,
+            -0.003295579692348838,
+            -0.002515980275347829,
+            0.006563597824424505,
+            0.009323694743216038,
+            -0.01644246280193329,
+            0.01103251427412033,
+            0.024647943675518036,
+            -0.023198990151286125,
+            0.004441354889422655,
+            0.002891999203711748,
+            -0.01070177461951971,
+            0.004126364830881357,
+            0.0065320986323058605,
+            -0.00033024701406247914,
+            0.023009996861219406,
+            0.004587037488818169,
+            0.019576609134674072,
+            0.004022024571895599,
+            -0.014670644886791706,
+            -0.015599864535033703,
+            0.006646282505244017,
+            -0.010890768840909004,
+            -0.011016764678061008,
+            -0.008063736371695995,
+            -0.010693900287151337,
+            0.0054611340165138245,
+            -0.004787843208760023,
+            -0.004022024571895599,
+            -0.0037739702966064215,
+            -0.011253006756305695,
+            0.006827401462942362,
+            -0.01148924883455038,
+            0.001115260412916541,
+            -0.00638247886672616,
+            -0.005280014593154192,
+            0.00945756584405899,
+            0.011851487681269646,
+            -0.010008797980844975,
+            -0.02674262598156929,
+            0.005555630661547184,
+            0.031199729070067406,
+            -0.006728967186063528,
+            0.007929865270853043,
+            0.012355471029877663,
+            -0.00745738111436367,
+            0.000588144815992564,
+            -0.006732904817909002,
+            -0.004650035407394171,
+            -0.03105798363685608,
+            -0.0010227321181446314,
+            -0.01067027635872364,
+            0.0003351687337271869,
+            -0.014245408587157726,
+            -0.015017134137451649,
+            -0.013166569173336029,
+            0.004968962632119656,
+            -0.013142945244908333,
+            -0.005043772514909506,
+            0.004149989224970341,
+            0.006988833658397198,
+            -0.016379464417696,
+            0.017576424404978752,
+            -0.00943394098430872,
+            -0.003933433908969164,
+            -0.00592968100681901,
+            0.033451907336711884,
+            -0.005406010430306196,
+            -0.003504260443150997,
+            -0.00492958864197135,
+            0.008874834515154362,
+            0.006776215974241495,
+            0.002053339034318924,
+            -0.0029038111679255962,
+            0.0021379925310611725,
+            0.02323048934340477,
+            0.006232858635485172,
+            0.003397951368242502,
+            0.015536867082118988,
+            -0.013473683968186378,
+            -0.008044049143791199,
+            -0.012654710561037064,
+            -0.01067027635872364,
+            0.0020454642362892628,
+            0.010985265485942364,
+            0.02365572564303875,
+            -0.005083146039396524,
+            -0.014985634945333004,
+            0.011473499238491058,
+            0.006772278342396021,
+            0.02140354923903942,
+            -0.011843612417578697,
+            5.6138298532459885e-05,
+            0.00812673382461071,
+            0.02376597188413143,
+            0.031813960522413254,
+            -0.000510381767526269,
+            7.437940075760707e-05,
+            0.0083393519744277,
+            0.010615152306854725,
+            -0.004980774596333504,
+            0.00664234533905983,
+            -0.01704094186425209,
+            -0.02795533463358879,
+            -0.008898458443582058,
+            0.006646282505244017,
+            -0.001707834773696959,
+            0.018647389486432076,
+            0.01007179543375969,
+            0.016757450997829437,
+            0.02382897026836872,
+            0.023025747388601303,
+            0.00039545970503240824,
+            0.02037983387708664,
+            -0.004425605293363333,
+            -0.004417730495333672,
+            0.0025691348128020763,
+            -0.015048632398247719,
+            -0.03471186384558678,
+            0.006118674762547016,
+            0.00442954245954752,
+            0.024773940443992615,
+            -0.015757359564304352,
+            0.023561229929327965,
+            -0.004146052058786154,
+            -0.016158971935510635,
+            -0.005898181814700365,
+            0.020789319649338722,
+            -0.008772462606430054,
+            0.020789319649338722,
+            0.0013239410473033786,
+            -0.0010945892427116632,
+            0.00988280214369297,
+            -0.0011201821034774184,
+            -0.01379654835909605,
+            0.011253006756305695,
+            0.01763942278921604,
+            0.03770426660776138,
+            0.009630809538066387,
+            -0.008118859492242336,
+            0.01752917654812336,
+            0.001776738790795207,
+            -0.008394475094974041,
+            -0.004433480091392994,
+            -0.013820173218846321,
+            0.0076936231926083565,
+            -0.0074180071242153645,
+            0.01697794534265995,
+            0.002640007296577096,
+            -0.005260327830910683,
+            0.014725768007338047,
+            -0.012481466867029667,
+            0.01219010166823864,
+            0.021246055141091347,
+            0.0011949921026825905,
+            0.005886369850486517,
+            0.020001845434308052,
+            -0.033136915415525436,
+            0.011528623290359974,
+            -0.034837860614061356,
+            -0.005138269625604153,
+            0.012064104899764061,
+            -0.01861589029431343,
+            0.007732997182756662,
+            -0.018584391102194786,
+            -0.013253191486001015,
+            -0.002553385216742754,
+            0.014599772170186043,
+            0.02497868239879608,
+            0.006555723026394844,
+            -0.017670921981334686,
+            0.01179636362940073,
+            -0.00876458827406168,
+            -0.01825365237891674,
+            -0.008874834515154362,
+            -0.019419115036725998,
+            -0.011693992651998997,
+            -0.003850748995319009,
+            0.011938109062612057,
+            -0.008811837062239647,
+            0.0037011289969086647,
+            0.002248239004984498,
+            -0.0221437755972147,
+            -0.013883170671761036,
+            0.015914853662252426,
+            0.003413700731471181,
+            -0.00664234533905983,
+            -0.007343197241425514,
+            0.013016949407756329,
+            -0.010756897740066051,
+            0.007323510479182005,
+            -0.008181856945157051,
+            0.014253283850848675,
+            -0.012142852880060673,
+            -0.00319123943336308,
+            0.007079393602907658,
+            0.0035869451239705086,
+            -0.018033159896731377,
+            -0.004732720088213682,
+            -0.02493143454194069,
+            0.015355747193098068,
+            0.0029766527004539967,
+            -0.017481928691267967,
+            -0.00692977337166667,
+            0.007732997182756662,
+            -0.017922913655638695,
+            -0.004894152283668518,
+            0.008717339485883713,
+            -0.003990525845438242,
+            0.025340920314192772,
+            0.022301269695162773,
+            -0.01248934119939804,
+            0.00016143222455866635,
+            -0.009213448502123356,
+            0.009985173121094704,
+            -0.01856864243745804,
+            -0.004484665580093861,
+            0.02650638297200203,
+            0.007315635681152344,
+            -0.00550444470718503,
+            -0.009937925264239311,
+            0.001173336640931666,
+            0.002771909348666668,
+            0.017607923597097397,
+            0.02493143454194069,
+            0.00621710903942585,
+            -0.010985265485942364,
+            -0.022458765655755997,
+            -0.027388354763388634,
+            0.027372604236006737,
+            0.0007485927199013531,
+            -0.004118490032851696,
+            0.013276815414428711,
+            0.02529367245733738,
+            0.023939216509461403,
+            -0.0062013594433665276,
+            -0.00483509199693799,
+            0.018300902098417282,
+            0.00997729878872633,
+            -0.005402073264122009,
+            -0.010166292078793049,
+            0.003392045386135578,
+            -0.0071345167234539986,
+            0.03414488211274147,
+            0.010111168958246708,
+            0.004417730495333672,
+            -0.046366482973098755,
+            0.009993048384785652,
+            -0.01619047112762928,
+            -0.003758220700547099,
+            0.008236980065703392,
+            0.021734289824962616,
+            0.010095419362187386,
+            0.013032699003815651,
+            0.007984988391399384,
+            -0.007236888166517019,
+            0.02861681394279003,
+            -0.0028703436255455017,
+            -0.010623027570545673,
+            -0.007272324524819851,
+            -0.025687409564852715,
+            0.004342920612543821,
+            -0.016410963609814644,
+            0.0021517733111977577,
+            0.005677689332515001,
+            -0.00918982457369566,
+            0.02590790204703808,
+            0.00794167723506689,
+            -0.02382897026836872,
+            0.00588243268430233,
+            -0.03682229667901993,
+            0.019482113420963287,
+            -0.02619139291346073,
+            -0.009323694743216038,
+            -0.004862653557211161,
+            -0.003758220700547099,
+            -0.025923652574419975,
+            -0.0037365653552114964,
+            -0.004783906042575836,
+            0.016883447766304016,
+            0.0024017964024096727,
+            -0.00013805407797917724,
+            -0.00013399365707300603,
+            -0.007103017531335354,
+            0.002041527070105076,
+            0.012410594150424004,
+            0.022521764039993286,
+            0.01571798510849476,
+            -0.0037897196598351,
+            0.018521394580602646,
+            -0.0005768248811364174,
+            -0.03704278916120529,
+            0.0037168783601373434,
+            -0.011662493459880352,
+            0.008103109896183014,
+            0.015332123264670372,
+            0.009591436013579369,
+            -0.01370205171406269,
+            0.013190193101763725,
+            -0.012150727212429047,
+            0.015395121648907661,
+            0.01619047112762928,
+            0.004894152283668518,
+            -0.0009976314613595605,
+            0.0157652348279953,
+            -0.02389196865260601,
+            0.00020941892580594867,
+            -0.00626435736194253,
+            0.006713218055665493,
+            -0.009930050000548363,
+            -0.0033251100685447454,
+            0.002041527070105076,
+            0.0006855947431176901,
+            -0.0034038573503494263,
+            -0.012064104899764061,
+            -0.017387431114912033,
+            -0.021498046815395355,
+            -0.01428478304296732,
+            -0.020820818841457367,
+            -0.011001015082001686,
+            0.030711494386196136,
+            -0.008622842840850353,
+            0.004559475928544998,
+            0.015025008469820023,
+            0.018710387870669365,
+            0.002596696373075247,
+            0.01856864243745804,
+            0.008260604925453663,
+            -0.007299886085093021,
+            -0.01727718487381935,
+            0.005543818697333336,
+            -0.025435417890548706,
+            0.03310541808605194,
+            0.00933944433927536,
+            -0.014032791368663311,
+            0.012969700619578362,
+            -0.01070177461951971,
+            -0.004346857778728008,
+            -0.007784182671457529,
+            0.0060163033194839954,
+            -0.025498416274785995,
+            0.01455252431333065,
+            -0.011749115772545338,
+            0.006961272098124027,
+            0.002305330941453576,
+            0.03505835309624672,
+            -0.014489525929093361,
+            -0.008929957635700703,
+            0.012032606638967991,
+            0.019119873642921448,
+            0.026553630828857422,
+            -0.009441816247999668,
+            -0.004937463440001011,
+            -0.01814340613782406,
+            -0.00357119576074183,
+            -0.005906056612730026,
+            -0.018631640821695328,
+            0.03590882569551468,
+            -0.007847180590033531,
+            -0.021387800574302673,
+            -0.00824485532939434,
+            -0.02006484381854534,
+            0.0031262726988643408,
+            0.0315462164580822,
+            0.01040253508836031,
+            0.006890399381518364,
+            -8.92060634214431e-05,
+            -0.020285336300730705,
+            -0.021324802190065384,
+            0.019797101616859436,
+            0.012166476808488369,
+            -0.019844351336359978,
+            -0.0009695776388980448,
+            -0.009394567459821701,
+            -0.00216752290725708,
+            0.0006609861738979816,
+            0.0021773662883788347,
+            0.05780060961842537,
+            -0.008087360300123692,
+            0.0013229567557573318,
+            -0.024065213277935982,
+            -0.001320003648288548,
+            -0.003332984633743763,
+            -0.0035318220034241676,
+            0.005646190140396357,
+            -0.00626435736194253,
+            0.01128450594842434,
+            -0.03480635955929756,
+            0.006784090772271156,
+            -0.010607277974486351,
+            -0.009654434397816658,
+            -0.016111722216010094,
+            0.00166747672483325,
+            -0.018285151571035385,
+            -4.6664001274621114e-05,
+            0.00021680150530301034,
+            -0.012134977616369724,
+            0.004083054140210152,
+            -0.013883170671761036,
+            -0.0003961979818996042,
+            0.024553446099162102,
+            0.011686117388308048,
+            -0.0013268940383568406,
+            -0.022222522646188736,
+            -0.022395767271518707,
+            0.0015562459593638778,
+            -0.010394659824669361,
+            0.01801741123199463,
+            -0.013229567557573318,
+            0.017182687297463417,
+            -0.02165554091334343,
+            -0.00046780891716480255,
+            0.0019765603356063366,
+            0.006232858635485172,
+            0.01861589029431343,
+            -0.00010969270078931004,
+            0.009693807922303677,
+            -0.014993509277701378,
+            0.02535667084157467,
+            -0.006949460133910179,
+            -0.02019083872437477,
+            0.0007328432402573526,
+            0.005181580316275358,
+            -0.0044216676615178585,
+            -0.0003430434735491872,
+            0.011056138202548027,
+            0.019419115036725998,
+            -0.0024766065180301666,
+            -0.007839306257665157,
+            -0.014064289629459381,
+            -0.009930050000548363,
+            0.0194033645093441,
+            0.0009297117940150201,
+            0.02055307850241661,
+            -0.011536497622728348,
+            0.00918982457369566,
+            -0.017623674124479294,
+            -0.007662124466150999,
+            -0.00907170306891203,
+            -0.004022024571895599,
+            0.016725953668355942,
+            -0.0007599126547574997,
+            0.01512738037854433,
+            -0.018001660704612732,
+            -0.021907532587647438,
+            0.0006098003359511495,
+            -0.02348248101770878,
+            -0.018411148339509964,
+            0.009205574169754982,
+            -0.01855289377272129,
+            -0.0025986649561673403,
+            0.001046356395818293,
+            -0.020883817225694656,
+            -0.013473683968186378,
+            0.018694639205932617,
+            -0.001501122722402215,
+            -0.0029628719203174114,
+            0.008496847003698349,
+            -0.006343104876577854,
+            -0.021813036873936653,
+            -0.0039688702672719955,
+            0.009512688964605331,
+            0.009748931042850018,
+            0.012993324548006058,
+            -0.022222522646188736,
+            0.004724845290184021,
+            0.0015719954390078783,
+            -0.005350887309759855,
+            -0.007095142733305693,
+            0.018174905329942703,
+            0.008047986775636673,
+            0.004531914368271828,
+            0.007768433541059494,
+            0.010008797980844975,
+            -0.0009479221189394593,
+            0.008370851166546345,
+            0.02146654762327671,
+            0.00108376145362854,
+            -0.002827032469213009,
+            -0.016347965225577354,
+            0.00612261239439249,
+            -0.012292472645640373,
+            0.015473868697881699,
+            0.02589215338230133,
+            0.008355101570487022,
+            -0.016332216560840607,
+            0.020883817225694656,
+            -0.014481651596724987,
+            0.015151004306972027,
+            -0.0282073263078928,
+            -0.00988280214369297,
+            0.018159156665205956,
+            -0.0011546340538188815,
+            0.01842689700424671,
+            -0.003108554519712925,
+            0.024065213277935982,
+            0.012481466867029667,
+            0.006417915225028992,
+            1.7518224922241643e-05,
+            -0.0007170937024056911,
+            0.010063921101391315,
+            -0.005189455114305019,
+            0.00014851271407678723,
+            -0.005740687251091003,
+            0.009134701453149319,
+            -0.006551785860210657,
+            0.00027832292835228145,
+            0.010725399479269981,
+            -0.028065582737326622,
+            0.014332030899822712,
+            -0.02274225652217865,
+            -0.0036735672038048506,
+            -0.0024431387428194284,
+            -0.007146328687667847,
+            -0.02031683549284935,
+            -0.003079024376347661,
+            0.019088376313447952,
+            -0.018489895388484,
+            0.04819342494010925,
+            0.0012028669007122517,
+            0.018300902098417282,
+            -0.007453443482518196,
+            0.002006090711802244,
+            -0.015143129974603653,
+            -0.00035928512807004154,
+            -0.02674262598156929,
+            0.00807161070406437,
+            0.003821218851953745,
+            0.0024254205636680126,
+            0.0025238548405468464,
+            0.006902211811393499,
+            -0.002173428889364004,
+            0.026648128405213356,
+            0.0002606047492008656,
+            0.01127663068473339,
+            -0.004295671824365854,
+            0.010717524215579033,
+            -0.019261619076132774,
+            0.004327171016484499,
+            0.0034491370897740126,
+            -0.00797711405903101,
+            -0.003933433908969164,
+            -0.007091205567121506,
+            0.0221437755972147,
+            0.016631456092000008,
+            0.009662308730185032,
+            -0.008020425215363503,
+            -0.0013042541686445475,
+            0.006803777534514666,
+            -0.0030396506190299988,
+            -0.011756990104913712,
+            -0.0038724045734852552,
+            0.005969054531306028,
+            0.012512966059148312,
+            -0.0009011658257804811,
+            0.011961733922362328,
+            0.005362699739634991,
+            -0.008008613251149654,
+            0.01297757588326931,
+            -0.020096343010663986,
+            -0.006390353664755821,
+            -0.009685933589935303,
+            -0.033987388014793396,
+            0.009292195551097393,
+            -0.0017304746434092522,
+            0.003090836340561509,
+            -0.007229013368487358,
+            0.010260789655148983,
+            -0.009520563296973705,
+            -0.01322169229388237,
+            -0.006512411870062351,
+            0.002208865247666836,
+            0.023639976978302002,
+            -0.006193485110998154,
+            0.00240376521833241,
+            -0.013355563394725323,
+            -0.013804423622786999,
+            0.016899196431040764,
+            -0.01170186698436737,
+            -0.007343197241425514,
+            0.009827678091824055
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "4bf7f8ae-8100-43bd-bfc1-81dbf9f911df",
+      "type": "next",
+      "source": {
+        "id": "c6bf0f92-84c8-44b9-8b28-7fcab6058203",
+        "properties": {
+          "page_content": "8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n",
+          "keyphrases": [
+            "Mixture of Experts (MoE)",
+            "language model training",
+            "DeepSeekMoE architecture",
+            "expert-choice routing strategy",
+            "training instability"
+          ],
+          "title": "Related Work",
+          "embedding": [
+            -0.0004454151203390211,
+            -0.01349730882793665,
+            -0.018033159896731377,
+            0.005232766270637512,
+            0.01874188706278801,
+            0.010268663987517357,
+            -0.011331754736602306,
+            0.03266443312168121,
+            -0.018993878737092018,
+            0.018789134919643402,
+            -0.005878495052456856,
+            -0.009221323765814304,
+            0.007355009205639362,
+            -0.008567719720304012,
+            0.0011782583314925432,
+            0.02784508839249611,
+            0.018474144861102104,
+            0.015410871244966984,
+            0.002397859003394842,
+            -0.03512135148048401,
+            0.019734105095267296,
+            -0.02189178392291069,
+            -0.044067058712244034,
+            -0.013772924430668354,
+            0.014253283850848675,
+            0.01693069562315941,
+            0.03858623653650284,
+            -0.006386416032910347,
+            -0.05959605053067207,
+            0.03386139124631882,
+            0.020521579310297966,
+            0.02291550114750862,
+            -0.01582823134958744,
+            -0.014930511824786663,
+            -0.02899480052292347,
+            -0.013481559231877327,
+            0.05077634006738663,
+            0.021734289824962616,
+            -0.016410963609814644,
+            0.027687594294548035,
+            0.0467129722237587,
+            0.013268941082060337,
+            -0.02280525304377079,
+            -0.0008283244678750634,
+            0.005512319505214691,
+            0.0011831800220534205,
+            -0.019009627401828766,
+            -0.016332216560840607,
+            0.010497031733393669,
+            -0.028175828978419304,
+            -0.018883632495999336,
+            -0.027278108522295952,
+            -0.006665969267487526,
+            0.0006486818892881274,
+            -0.0642578974366188,
+            0.007378633599728346,
+            0.01752917654812336,
+            0.03389289230108261,
+            0.05058734491467476,
+            0.0015080131124705076,
+            0.012308222241699696,
+            -0.019419115036725998,
+            -0.01686769910156727,
+            -0.008260604925453663,
+            -0.00022885970247443765,
+            -0.01097739115357399,
+            0.012016857042908669,
+            -0.013324064202606678,
+            0.01982860080897808,
+            0.016221968457102776,
+            -0.040854163467884064,
+            0.027026114985346794,
+            -0.023876218125224113,
+            0.009426066651940346,
+            -0.006969146896153688,
+            0.007095142733305693,
+            0.019135624170303345,
+            0.026600880548357964,
+            -0.0019509674748405814,
+            -0.007252637762576342,
+            0.0025514166336506605,
+            0.017324432730674744,
+            -0.007122704293578863,
+            0.0022108338307589293,
+            0.05014635995030403,
+            -0.04687046632170677,
+            -0.02584490366280079,
+            -0.07049469649791718,
+            0.00882758665829897,
+            0.00037503460771404207,
+            -0.024789689108729362,
+            -0.003289673710241914,
+            0.028490817174315453,
+            0.008835460990667343,
+            -0.040602169930934906,
+            0.06312393397092819,
+            -0.014056415297091007,
+            -0.015662861987948418,
+            -0.03512135148048401,
+            -0.008355101570487022,
+            0.0005320372874848545,
+            -0.0049768369644880295,
+            0.011575871147215366,
+            0.010709649883210659,
+            0.012197976000607014,
+            -0.012812205590307713,
+            -0.033199913799762726,
+            0.04687046632170677,
+            -0.005713125690817833,
+            0.08523621410131454,
+            0.0021379925310611725,
+            -0.009945799596607685,
+            -0.003252268536016345,
+            0.03537334129214287,
+            -0.021435048431158066,
+            -0.008386600762605667,
+            -0.028349071741104126,
+            -0.04935888573527336,
+            0.024852687492966652,
+            -0.02348248101770878,
+            0.018631640821695328,
+            -0.0007800916791893542,
+            0.016395213082432747,
+            -0.019214371219277382,
+            -0.0721326395869255,
+            0.05739112198352814,
+            -0.0026774124708026648,
+            -0.015111630782485008,
+            -0.003021932439878583,
+            0.007201451808214188,
+            -0.0008194653782993555,
+            0.0037345965392887592,
+            -0.01735593192279339,
+            -0.019088376313447952,
+            0.029136545956134796,
+            0.023513980209827423,
+            0.03861773759126663,
+            0.046240486204624176,
+            -0.020301084965467453,
+            -0.01452889945358038,
+            0.023734472692012787,
+            -0.029514534398913383,
+            0.006630532909184694,
+            0.011567996814846992,
+            -0.01746617816388607,
+            0.011599495075643063,
+            0.011662493459880352,
+            0.06967572122812271,
+            -0.019860100001096725,
+            0.017009442672133446,
+            0.0022895813453942537,
+            0.0643208920955658,
+            0.0025435418356209993,
+            0.030475253239274025,
+            -0.014765141531825066,
+            0.02178153768181801,
+            -0.028160078451037407,
+            0.0025061366613954306,
+            -0.020222337916493416,
+            -0.003464886685833335,
+            -0.002510074060410261,
+            -0.04094865918159485,
+            -0.002821126487106085,
+            0.028853056952357292,
+            0.0015119505114853382,
+            0.01776541955769062,
+            0.033199913799762726,
+            0.003850748995319009,
+            -0.05962754786014557,
+            0.023198990151286125,
+            0.03452287241816521,
+            0.010985265485942364,
+            -0.009103202261030674,
+            0.021624041721224785,
+            0.011056138202548027,
+            0.010575778782367706,
+            0.02067907340824604,
+            0.008189732208848,
+            -0.022710757330060005,
+            0.03972020000219345,
+            -0.02886880561709404,
+            0.0018830477492883801,
+            0.03228644281625748,
+            0.0016743671149015427,
+            0.044791534543037415,
+            0.012906703166663647,
+            0.026852872222661972,
+            0.01693069562315941,
+            0.012316097505390644,
+            0.029939770698547363,
+            0.007166015449911356,
+            0.010536405257880688,
+            0.03540484234690666,
+            -0.015095881186425686,
+            0.008158233016729355,
+            -0.02255326136946678,
+            -0.012867328710854053,
+            0.006488787941634655,
+            0.00988280214369297,
+            -0.005815497133880854,
+            -0.0021753974724560976,
+            -0.04201962426304817,
+            0.02212802693247795,
+            -0.0055831922218203545,
+            0.021986281499266624,
+            0.0026144145522266626,
+            0.012615337036550045,
+            0.007630625274032354,
+            0.03559383377432823,
+            -0.03852323815226555,
+            -0.011788489297032356,
+            -0.007878679782152176,
+            0.005535943899303675,
+            -0.044791534543037415,
+            -0.020033344626426697,
+            0.016725953668355942,
+            -0.004776031244546175,
+            0.021135808899998665,
+            0.009276446886360645,
+            -0.02159254439175129,
+            -0.006626595743000507,
+            -0.006445476785302162,
+            0.018363898620009422,
+            -0.01533999852836132,
+            -0.030491001904010773,
+            -0.0351843498647213,
+            0.01300907414406538,
+            0.011536497622728348,
+            0.03590882569551468,
+            -0.0009178996551781893,
+            -0.020600326359272003,
+            0.014788766391575336,
+            0.001092620543204248,
+            0.0039216214790940285,
+            0.02274225652217865,
+            -0.04646097868680954,
+            0.00845747347921133,
+            0.0026537880767136812,
+            0.011520748026669025,
+            0.02085231803357601,
+            0.007394383195787668,
+            0.017182687297463417,
+            -0.035751331597566605,
+            0.004086991306394339,
+            0.03688529506325722,
+            0.011638869531452656,
+            -0.016103848814964294,
+            0.028065582737326622,
+            4.574118065647781e-05,
+            -0.012883078306913376,
+            -0.0033290472347289324,
+            -0.03222344443202019,
+            -0.012623212300240993,
+            0.0008617921266704798,
+            -0.016757450997829437,
+            0.05269777774810791,
+            0.043090589344501495,
+            -0.019324617460370064,
+            -0.014631271362304688,
+            -0.05918656289577484,
+            0.039688702672719955,
+            -0.02408096194267273,
+            -0.025262173265218735,
+            0.04416155442595482,
+            -0.016725953668355942,
+            0.020159341394901276,
+            0.032475437968969345,
+            0.012820080853998661,
+            -0.00011873635230585933,
+            -0.03527884557843208,
+            -0.004051554948091507,
+            -0.04523251950740814,
+            -0.03801925480365753,
+            0.005126457195729017,
+            0.03170371428132057,
+            -0.03559383377432823,
+            -0.0055792550556361675,
+            -0.009481189772486687,
+            0.030191762372851372,
+            0.028837306424975395,
+            0.02049008011817932,
+            -0.0016379464650526643,
+            0.007969239726662636,
+            0.016379464417696,
+            -0.03208170086145401,
+            0.013394936919212341,
+            -0.0046460977755486965,
+            -0.026522131636738777,
+            -0.010308037512004375,
+            -0.015749484300613403,
+            -0.006760466378182173,
+            -4.174228524789214e-05,
+            -0.04693346470594406,
+            0.00440985569730401,
+            0.03540484234690666,
+            -0.0037050663959234953,
+            -0.01795441284775734,
+            0.03323141112923622,
+            0.033451907336711884,
+            -0.027860838919878006,
+            -0.06167498230934143,
+            -0.001828908920288086,
+            0.021450798958539963,
+            -0.0024529823567718267,
+            -0.05616266280412674,
+            -0.003445199690759182,
+            -0.008276354521512985,
+            0.030554000288248062,
+            -0.01710394024848938,
+            -0.04201962426304817,
+            0.0032680181320756674,
+            -0.0005128426128067076,
+            -0.013481559231877327,
+            -0.03433387726545334,
+            -0.00764243770390749,
+            0.034365374594926834,
+            0.01437140442430973,
+            0.025813406333327293,
+            -0.04384656623005867,
+            -0.02929404191672802,
+            0.01270983461290598,
+            0.0394052118062973,
+            -0.06284044682979584,
+            0.010308037512004375,
+            -0.014292657375335693,
+            -0.001888953847810626,
+            0.004728782922029495,
+            0.043878063559532166,
+            -0.03291642293334007,
+            0.009016579948365688,
+            -0.00609505083411932,
+            -0.04167313501238823,
+            -0.034963857382535934,
+            -0.010827770456671715,
+            -0.00495715020224452,
+            -0.016332216560840607,
+            -0.02297849766910076,
+            0.002760097151622176,
+            -0.014269033446907997,
+            0.003340859431773424,
+            -0.02165554091334343,
+            0.004846903961151838,
+            -0.017859915271401405,
+            0.021734289824962616,
+            0.010056045837700367,
+            -0.013938293792307377,
+            -0.025703158229589462,
+            -0.004941401071846485,
+            0.02097831293940544,
+            0.012670460157096386,
+            0.00115561846178025,
+            -0.0034688240848481655,
+            0.002754191169515252,
+            -0.003376295790076256,
+            -0.009693807922303677,
+            -0.06545485556125641,
+            -0.017261436209082603,
+            -0.017607923597097397,
+            0.015552615746855736,
+            0.016056599095463753,
+            0.010725399479269981,
+            -0.046901967376470566,
+            -0.05118582397699356,
+            0.028538066893815994,
+            -0.03549933806061745,
+            -0.001561167649924755,
+            -0.0025927589740604162,
+            -0.0015926666092127562,
+            -0.004386231303215027,
+            0.016568457707762718,
+            -0.005606816615909338,
+            0.03360940143465996,
+            -0.012016857042908669,
+            0.007650312036275864,
+            0.004480728413909674,
+            0.020269587635993958,
+            0.0333574078977108,
+            -0.011103386990725994,
+            -0.011048263870179653,
+            0.049106892198324203,
+            -0.013631178997457027,
+            0.025088928639888763,
+            0.005638315342366695,
+            0.001570026739500463,
+            0.0033861391711980104,
+            0.02382897026836872,
+            0.014198160730302334,
+            0.005350887309759855,
+            -0.03713728487491608,
+            0.027089113369584084,
+            0.010363160632550716,
+            -0.060919005423784256,
+            -0.015088006854057312,
+            0.006595096550881863,
+            0.017418930307030678,
+            0.016536958515644073,
+            -0.003559383563697338,
+            -0.014662770554423332,
+            -0.013457934372127056,
+            -0.008481097407639027,
+            -0.018521394580602646,
+            0.036412809044122696,
+            -0.0033054230734705925,
+            0.010725399479269981,
+            -0.027215110138058662,
+            0.0045752255246043205,
+            0.01328469067811966,
+            0.016206219792366028,
+            0.02942003682255745,
+            -0.015922728925943375,
+            -0.015017134137451649,
+            -0.04718545451760292,
+            -0.03288492560386658,
+            -0.00928432121872902,
+            0.013245316222310066,
+            0.030128763988614082,
+            0.01600935123860836,
+            0.046429481357336044,
+            -0.012757082469761372,
+            0.00840235035866499,
+            -0.027624595910310745,
+            0.03795625641942024,
+            0.049043893814086914,
+            0.009985173121094704,
+            -0.020631825551390648,
+            0.02171853929758072,
+            0.009985173121094704,
+            -0.01995459757745266,
+            0.012308222241699696,
+            -0.020631825551390648,
+            0.040413178503513336,
+            -0.022458765655755997,
+            0.03874373435974121,
+            -0.04337408021092415,
+            -0.010709649883210659,
+            -0.03414488211274147,
+            0.003462917869910598,
+            0.02856956608593464,
+            -0.022395767271518707,
+            -0.021608293056488037,
+            -0.0017294903518632054,
+            0.0272938571870327,
+            0.012410594150424004,
+            -0.0009070718660950661,
+            -0.02293124981224537,
+            -0.021198807284235954,
+            -0.03534184396266937,
+            0.006890399381518364,
+            -0.014465902000665665,
+            -0.017371682450175285,
+            -0.0533907525241375,
+            0.014245408587157726,
+            -0.07143966108560562,
+            0.010993140749633312,
+            0.0485084131360054,
+            -0.012276723049581051,
+            0.016662955284118652,
+            0.0007077424670569599,
+            -0.004256298299878836,
+            -0.02935704030096531,
+            0.00967018399387598,
+            0.049831368029117584,
+            0.04028718173503876,
+            -0.021482296288013458,
+            -0.017670921981334686,
+            0.009150451049208641,
+            -0.028601063415408134,
+            -0.01589122973382473,
+            -0.005209141876548529,
+            0.01704094186425209,
+            0.014481651596724987,
+            -0.02589215338230133,
+            0.02638038620352745,
+            0.008465347811579704,
+            0.004953213036060333,
+            0.028396321460604668,
+            0.01331618893891573,
+            -0.0460829921066761,
+            0.014347780495882034,
+            0.029372788965702057,
+            0.001872219960205257,
+            -0.010378910228610039,
+            -0.004106678068637848,
+            0.00592968100681901,
+            0.01819065399467945,
+            -0.017576424404978752,
+            0.007114829961210489,
+            0.005591067019850016,
+            -0.01583610661327839,
+            0.020694822072982788,
+            0.011930234730243683,
+            0.01128450594842434,
+            -0.01067027635872364,
+            0.0218287855386734,
+            -0.00031375925755128264,
+            -0.019907347857952118,
+            -0.009426066651940346,
+            -0.00981980375945568,
+            0.02061607502400875,
+            0.00038364759529940784,
+            -0.03445987403392792,
+            0.011111261323094368,
+            0.04372056946158409,
+            -0.001538527780212462,
+            0.01003242190927267,
+            0.010812020860612392,
+            0.030223261564970016,
+            0.028538066893815994,
+            -0.022710757330060005,
+            -0.02825457602739334,
+            0.01691494695842266,
+            -0.009394567459821701,
+            0.004547663498669863,
+            -0.022632010281085968,
+            -0.014237534254789352,
+            0.03408188372850418,
+            -0.016725953668355942,
+            0.033136915415525436,
+            -0.002256113803014159,
+            0.015969976782798767,
+            -0.004846903961151838,
+            0.0010483250953257084,
+            -0.006433664355427027,
+            0.035215847194194794,
+            -0.026396136730909348,
+            0.014930511824786663,
+            0.007110892329365015,
+            0.002866406226530671,
+            0.05723362788558006,
+            -0.03814525157213211,
+            -0.03625531494617462,
+            0.00262228911742568,
+            0.030853239819407463,
+            -0.017796916887164116,
+            -0.005913931410759687,
+            -0.009599311277270317,
+            0.0031813958194106817,
+            0.018757635727524757,
+            -0.0351843498647213,
+            -0.01855289377272129,
+            0.01331618893891573,
+            -0.01801741123199463,
+            0.003470792667940259,
+            -0.0006501584430225194,
+            0.0018436741083860397,
+            0.00524457823485136,
+            0.0618639774620533,
+            -0.05175280570983887,
+            0.016946446150541306,
+            0.03603482246398926,
+            -0.0007520379149354994,
+            -0.03282192721962929,
+            -0.02625439129769802,
+            -0.005736749619245529,
+            -0.003931465093046427,
+            -0.03067999705672264,
+            -1.9440769392531365e-05,
+            -0.004043680150061846,
+            0.007024270016700029,
+            -0.03669629991054535,
+            0.014623397029936314,
+            0.011355378665030003,
+            -0.002449044957756996,
+            0.003744439920410514,
+            -0.01391466986387968,
+            -0.002553385216742754,
+            -0.003896028734743595,
+            0.03367239981889725,
+            -0.011064013466238976,
+            -0.002878218423575163,
+            -0.00967018399387598,
+            -0.003775938879698515,
+            -0.007197514642030001,
+            -0.0027738779317587614,
+            -0.00845747347921133,
+            0.0031873020343482494,
+            0.012237349525094032,
+            0.001938170986250043,
+            -0.012426343746483326,
+            -0.0028171890880912542,
+            0.03657030314207077,
+            -0.02510467916727066,
+            -0.021183056756854057,
+            0.051280323415994644,
+            0.040476176887750626,
+            0.004374419339001179,
+            -0.021907532587647438,
+            -0.010260789655148983,
+            0.021498046815395355,
+            -0.019749853760004044,
+            0.013670552521944046,
+            0.032176196575164795,
+            -0.007547940593212843,
+            -0.028364822268486023,
+            -0.00964655913412571,
+            -0.014410778880119324,
+            0.0019509674748405814,
+            -0.00562256621196866,
+            0.024065213277935982,
+            -0.044539544731378555,
+            0.008488972671329975,
+            0.019938847050070763,
+            0.002590790158137679,
+            -0.012394844554364681,
+            -0.011040388606488705,
+            -0.003295579692348838,
+            0.02178153768181801,
+            0.0009813897777348757,
+            -0.01893088035285473,
+            0.00861496850848198,
+            0.013418560847640038,
+            0.006158048752695322,
+            0.01893088035285473,
+            -0.005732812453061342,
+            0.02352973073720932,
+            0.009733181446790695,
+            -0.0005241625476628542,
+            0.014056415297091007,
+            -0.019970346242189407,
+            -0.00488234031945467,
+            0.061517488211393356,
+            0.011623119935393333,
+            0.025734657421708107,
+            -0.01431628130376339,
+            0.020631825551390648,
+            0.03814525157213211,
+            -0.0016999599756672978,
+            -0.020820818841457367,
+            0.008362976834177971,
+            0.00283490726724267,
+            -0.044665537774562836,
+            -0.0134894335642457,
+            -0.023923467844724655,
+            -0.037326280027627945,
+            0.0035357594024389982,
+            -0.021214555948972702,
+            0.01801741123199463,
+            0.025199174880981445,
+            0.0065832845866680145,
+            -0.030884739011526108,
+            -0.01197748351842165,
+            0.022710757330060005,
+            0.02741985209286213,
+            0.03449137136340141,
+            -0.01233972143381834,
+            0.010103294625878334,
+            -0.003295579692348838,
+            0.013694177381694317,
+            0.037987757474184036,
+            0.006925835739821196,
+            -0.0031006797216832638,
+            0.008059798739850521,
+            0.029703527688980103,
+            0.03675929829478264,
+            -0.024584945291280746,
+            -0.052130796015262604,
+            -0.0028703436255455017,
+            -0.01686769910156727,
+            0.018600141629576683,
+            -0.001023716526106,
+            0.003746408736333251,
+            0.0011428219731897116,
+            0.006654157303273678,
+            -0.008740964345633984,
+            0.02893180400133133,
+            0.008028299547731876,
+            -0.021797286346554756,
+            0.05159531161189079,
+            -0.015481743961572647,
+            0.024112461134791374,
+            -0.001423359732143581,
+            -0.02590790204703808,
+            -0.028711311519145966,
+            0.013111446052789688,
+            0.007390445563942194,
+            -0.03546784073114395,
+            0.03742077574133873,
+            0.033451907336711884,
+            0.0030731181614100933,
+            0.0012127102818340063,
+            0.0166787039488554,
+            0.016962194815278053,
+            -0.014780891127884388,
+            -0.023876218125224113,
+            0.02286825142800808,
+            -0.008677965961396694,
+            -0.021939031779766083,
+            0.01982860080897808,
+            -0.018521394580602646,
+            -0.008268479257822037,
+            0.0042484235018491745,
+            -0.001532621681690216,
+            -0.010182041674852371,
+            0.014395029284060001,
+            0.016521209850907326,
+            -0.015332123264670372,
+            -0.02527792379260063,
+            0.031215479597449303,
+            0.021513795480132103,
+            0.0038231874350458384,
+            -0.02165554091334343,
+            0.013473683968186378,
+            -0.022537512704730034,
+            -0.0010286382166668773,
+            0.043752070516347885,
+            -0.002433295361697674,
+            0.012946076691150665,
+            -0.010237164795398712,
+            -0.029719278216362,
+            0.002671506255865097,
+            0.011213633231818676,
+            0.03789326176047325,
+            0.01885213330388069,
+            0.01795441284775734,
+            0.019450614228844643,
+            -0.01631646603345871,
+            0.009528438560664654,
+            -0.014560398645699024,
+            -0.03294792026281357,
+            0.03179820999503136,
+            -0.006024178117513657,
+            0.025577163323760033,
+            0.008465347811579704,
+            -0.0321604460477829,
+            -0.004728782922029495,
+            0.011040388606488705,
+            0.013001199811697006,
+            0.023120243102312088,
+            0.013867421075701714,
+            0.003108554519712925,
+            0.031388722360134125,
+            -0.014513149857521057,
+            -0.051091328263282776,
+            0.02710486389696598,
+            0.0063076685182750225,
+            -0.006701405625790358,
+            -0.0016143221873790026,
+            0.020364083349704742,
+            -0.003027838421985507,
+            0.00017139868577942252,
+            0.01270195934921503,
+            -0.010961641557514668,
+            0.00882758665829897,
+            -0.006992771290242672,
+            -0.005469008814543486,
+            0.02414396032691002,
+            0.011827862821519375,
+            0.024065213277935982,
+            -0.019497862085700035,
+            0.006347042508423328,
+            0.017324432730674744,
+            -0.009748931042850018,
+            0.014568272978067398,
+            0.0034314189106225967,
+            -0.05011485889554024,
+            0.01607234962284565,
+            0.017576424404978752,
+            -0.005173705518245697,
+            0.007103017531335354,
+            -0.013402811251580715,
+            0.013804423622786999,
+            0.04627198725938797,
+            0.0016330246580764651,
+            0.0066856564953923225,
+            0.013205942697823048,
+            -0.005012273322790861,
+            -0.009804054163396358,
+            0.03112098202109337,
+            0.00845747347921133,
+            -0.008796087466180325,
+            0.009079578332602978,
+            0.001405641552992165,
+            0.03499535471200943,
+            -0.04828792065382004,
+            0.010717524215579033,
+            0.012520840391516685,
+            -0.018584391102194786,
+            -0.012812205590307713,
+            -0.02886880561709404,
+            0.009520563296973705,
+            -0.03044375404715538,
+            0.01814340613782406,
+            -0.022789504379034042,
+            0.021608293056488037,
+            -0.035688333213329315,
+            -0.0018732043681666255,
+            0.012623212300240993,
+            -0.0058076223358511925,
+            0.003575132926926017,
+            0.012473591603338718,
+            -0.007575502153486013,
+            -0.009323694743216038,
+            -0.018096158280968666,
+            0.023671476170420647,
+            0.05575317516922951,
+            -0.0003388600016478449,
+            0.00024116398708429188,
+            0.015930604189634323,
+            -0.01443440280854702,
+            -0.02620714344084263,
+            0.02953028306365013,
+            -0.016773201525211334,
+            0.015623488463461399,
+            0.03723178058862686,
+            -0.03468036651611328,
+            -0.027262357994914055,
+            0.01543449517339468,
+            -0.013725675642490387,
+            0.008347227238118649,
+            0.0020651512313634157,
+            0.02159254439175129,
+            -0.00283490726724267,
+            -0.026096895337104797,
+            0.028538066893815994,
+            0.006197422277182341,
+            0.009418192319571972,
+            -0.021671291440725327,
+            -0.011095511727035046,
+            -0.027325356379151344,
+            -0.023797471076250076,
+            -0.011867236346006393,
+            -0.010174167342483997,
+            0.006508474703878164,
+            -0.016820449382066727,
+            -0.057832106947898865,
+            -0.015229751355946064,
+            0.023907717317342758,
+            -0.01412728801369667,
+            0.05263477936387062,
+            0.01844264753162861,
+            0.016269218176603317,
+            0.0048035928048193455,
+            -0.04022418335080147,
+            0.030176013708114624,
+            -0.017560675740242004,
+            0.03143597021698952,
+            0.016088098287582397,
+            0.006839213892817497,
+            -0.0059611801989376545,
+            0.05058734491467476,
+            -0.02389196865260601,
+            0.007126641925424337,
+            -0.019686855375766754,
+            0.006473038345575333,
+            -0.02499443292617798,
+            -0.004508289974182844,
+            -0.00521701667457819,
+            -0.011465624906122684,
+            0.010063921101391315,
+            -0.011426251381635666,
+            0.024049462750554085,
+            0.02541966922581196,
+            0.0009026423213072121,
+            0.020301084965467453,
+            -0.030900489538908005,
+            -0.022206773981451988,
+            -0.004740594886243343,
+            0.0018899381393566728,
+            -0.02080507017672062,
+            -0.016332216560840607,
+            -0.040066689252853394,
+            -0.012788581661880016,
+            -0.0002187701902585104,
+            0.01212710328400135,
+            0.0040909284725785255,
+            0.02838057093322277,
+            -0.0017038973746821284,
+            -0.03209745138883591,
+            0.0071896398440003395,
+            -0.029514534398913383,
+            -0.0005906056612730026,
+            0.01391466986387968,
+            -0.017859915271401405,
+            -0.005945430602878332,
+            0.030648497864603996,
+            0.0016812575049698353,
+            -0.0150092588737607,
+            -0.016032975167036057,
+            0.005567443091422319,
+            -0.016237718984484673,
+            -0.0035554461646825075,
+            -0.01028441358357668,
+            0.03364089876413345,
+            -0.015332123264670372,
+            0.007780245505273342,
+            0.0005694422870874405,
+            4.2634346755221486e-05,
+            -0.03439687564969063,
+            -0.012890953570604324,
+            -0.019686855375766754,
+            0.013371312990784645,
+            0.0017777230823412538,
+            -0.027073364704847336,
+            0.007902303710579872,
+            0.0009838506812229753,
+            -0.004531914368271828,
+            -0.021324802190065384,
+            0.0032621119171380997,
+            0.005339075345546007,
+            -0.01771816983819008,
+            0.001696022693067789,
+            -0.033388908952474594,
+            -0.02831757441163063,
+            -0.029719278216362,
+            0.019419115036725998,
+            -0.0030435880180448294,
+            0.021183056756854057,
+            -0.013205942697823048,
+            0.012969700619578362,
+            -0.017261436209082603,
+            0.004590974655002356,
+            0.01570223644375801,
+            0.024663692340254784,
+            -0.03160921484231949,
+            0.013898920267820358,
+            0.01582035794854164,
+            0.021561045199632645,
+            -0.04523251950740814,
+            0.010709649883210659,
+            0.013631178997457027,
+            -0.0015227783005684614,
+            0.016332216560840607,
+            -0.010835645720362663,
+            0.03354640305042267,
+            -0.02093106508255005,
+            -0.0010749023640528321,
+            -0.005587129853665829,
+            0.013520932756364346,
+            0.01861589029431343,
+            0.012883078306913376,
+            -0.030412254855036736,
+            0.03110523149371147,
+            0.02352973073720932,
+            -0.012662585824728012,
+            0.04614599049091339,
+            -0.02656938135623932,
+            0.016458211466670036,
+            0.018600141629576683,
+            -0.02827032469213009,
+            0.0015660893404856324,
+            0.0045791626907885075,
+            -0.015631362795829773,
+            -0.014206035062670708,
+            0.025340920314192772,
+            0.00750462943688035,
+            -0.04346857964992523,
+            0.015001384541392326,
+            0.018237903714179993,
+            0.01119000930339098,
+            0.00433504581451416,
+            -0.023939216509461403,
+            0.009174074977636337,
+            0.028128579258918762,
+            0.0006875634426251054,
+            0.030475253239274025,
+            0.015064381994307041,
+            -0.02323048934340477,
+            0.011930234730243683,
+            -0.005795810371637344,
+            0.007406195160001516,
+            -0.01831665076315403,
+            0.00392752792686224,
+            -0.00747706787660718,
+            -0.014654895290732384,
+            -0.007232951000332832,
+            0.029640531167387962,
+            -0.003964932635426521,
+            -0.018883632495999336,
+            -0.03833424672484398,
+            0.015804607421159744,
+            0.009252822026610374,
+            -0.06709280610084534,
+            -0.00876458827406168,
+            -0.03386139124631882,
+            -0.02140354923903942,
+            -0.008906333707273006,
+            0.012749208137392998,
+            -0.015851857140660286,
+            -0.002389984205365181,
+            0.006154111120849848,
+            0.015599864535033703,
+            -0.01564711332321167,
+            0.02085231803357601,
+            0.006024178117513657,
+            0.005291827023029327,
+            -0.013993416912853718,
+            0.042051125317811966,
+            -0.007563690189272165,
+            -0.03814525157213211,
+            0.032176196575164795,
+            -0.012820080853998661,
+            0.003575132926926017,
+            0.024065213277935982,
+            -0.0019421083852648735,
+            -0.022002030164003372,
+            0.028963303193449974,
+            -0.026285890489816666,
+            -0.03512135148048401,
+            0.01522187702357769,
+            -0.006547848228365183,
+            -0.020285336300730705,
+            -0.02056882716715336,
+            -0.004681534133851528,
+            0.012733458541333675,
+            -0.02050582878291607,
+            0.018489895388484,
+            0.03266443312168121,
+            0.0052524530328810215,
+            -0.016206219792366028,
+            0.012457842007279396,
+            -0.0032247069757431746,
+            -0.0015030914219096303,
+            -0.005173705518245697,
+            0.024364452809095383,
+            -0.004055492114275694,
+            0.0033625150099396706,
+            -0.019119873642921448,
+            0.0030239010229706764,
+            -0.016088098287582397,
+            -0.012670460157096386,
+            -0.013457934372127056,
+            -0.025214925408363342,
+            -0.02493143454194069,
+            -0.04088566079735756,
+            0.03531034663319588,
+            0.024458950385451317,
+            -0.017324432730674744,
+            -0.010599403642117977,
+            -0.00861496850848198,
+            0.0013957980554550886,
+            0.028396321460604668,
+            -0.002578978193923831,
+            -0.0036597864236682653,
+            -0.02231702022254467,
+            0.011686117388308048,
+            -0.010583654046058655,
+            0.0014322188217192888,
+            -0.06296644359827042,
+            -0.01806465908885002,
+            0.034428372979164124,
+            0.007181765045970678,
+            -0.007996801286935806,
+            0.005917869042605162,
+            0.02376597188413143,
+            -0.020222337916493416,
+            -0.012481466867029667,
+            0.04390956461429596,
+            0.0027305670082569122,
+            0.0021458673290908337,
+            0.0070203328505158424,
+            -0.008347227238118649,
+            0.010961641557514668,
+            -0.004445292055606842,
+            -0.007729059550911188,
+            0.0005728874821215868,
+            0.010969515889883041,
+            0.009654434397816658,
+            0.00638247886672616,
+            -0.00442954245954752,
+            0.020710572600364685,
+            -0.011402626521885395,
+            -0.0043665445409715176,
+            0.006543911062180996,
+            0.019749853760004044,
+            -0.004783906042575836,
+            0.00025014611310325563,
+            -0.020301084965467453,
+            0.023986464366316795,
+            0.020773570984601974,
+            0.016158971935510635,
+            -0.0004537820350378752,
+            -0.008733089081943035,
+            -0.008059798739850521,
+            0.004401980899274349,
+            0.0030317758210003376,
+            0.017497677356004715,
+            0.028112830594182014,
+            -0.019482113420963287,
+            -0.016536958515644073,
+            -0.0012373188510537148,
+            0.0025336984544992447,
+            -0.004968962632119656,
+            0.03990919515490532,
+            0.02431720495223999,
+            0.005598941817879677,
+            -0.013969792984426022,
+            -0.0007269371417351067,
+            0.00891420803964138,
+            0.03505835309624672,
+            -0.026112645864486694,
+            -0.01571011170744896,
+            0.019907347857952118,
+            -0.0016330246580764651,
+            0.03433387726545334,
+            -0.021387800574302673,
+            0.014261158183217049,
+            -0.0026006337720900774,
+            0.08655916899442673,
+            -0.023372234776616096,
+            -0.005098895635455847,
+            -0.016174720600247383,
+            -0.014450152404606342,
+            0.004220861941576004,
+            -0.00459491228684783,
+            -0.003289673710241914,
+            0.031624965369701385,
+            -0.002029714873060584,
+            0.017182687297463417,
+            -0.0004545202828012407,
+            -0.008898458443582058,
+            0.008020425215363503,
+            -0.0035180412232875824,
+            0.018206404522061348,
+            -0.006681718863546848,
+            0.008969331160187721,
+            -0.003004214260727167,
+            0.0008957519312389195,
+            0.01322169229388237,
+            0.03291642293334007,
+            0.003939339891076088,
+            0.022537512704730034,
+            -0.02419120818376541,
+            0.018631640821695328,
+            0.007677873596549034,
+            0.016647204756736755,
+            -0.019986096769571304,
+            -0.017418930307030678,
+            -0.0007298901909962296,
+            0.013709926046431065,
+            -0.008575594052672386,
+            0.010906518436968327,
+            -0.003833030816167593,
+            0.039216216653585434,
+            -0.026396136730909348,
+            0.007047894410789013,
+            0.010835645720362663,
+            -0.004051554948091507,
+            0.009118951857089996,
+            0.02208077721297741,
+            0.017985912039875984,
+            0.005118582397699356,
+            -0.02297849766910076,
+            0.014135162346065044,
+            0.02293124981224537,
+            -0.01734018325805664,
+            0.0019470300758257508,
+            0.006539973430335522,
+            -0.0015119505114853382,
+            0.0004030883719678968,
+            0.005835183896124363,
+            -0.00024300962104462087,
+            -0.008331477642059326,
+            0.004512227606028318,
+            -0.007961364462971687,
+            -0.035562336444854736,
+            -0.0027384415734559298,
+            0.012851579114794731,
+            -0.00626435736194253,
+            0.031451720744371414,
+            0.029735026881098747,
+            0.019671106711030006,
+            -0.009811929427087307,
+            -0.006874650251120329,
+            0.004177550785243511,
+            -0.0040751793421804905,
+            -0.0011890861205756664,
+            -0.013607555069029331,
+            -0.014308406971395016,
+            0.014111538417637348,
+            -0.022474514320492744,
+            -0.00888270977884531,
+            0.0018338306108489633,
+            -0.005532006733119488,
+            -0.00459491228684783,
+            -0.0025002306792885065,
+            0.017481928691267967,
+            -0.005724937655031681,
+            0.02037983387708664,
+            0.01048915646970272,
+            -0.011528623290359974,
+            0.011308129876852036,
+            -0.026900120079517365,
+            0.009567812085151672,
+            -0.026963118463754654,
+            -0.010756897740066051,
+            0.009213448502123356,
+            -0.003992494195699692,
+            -0.017985912039875984,
+            -0.010473407804965973,
+            -0.016095973551273346,
+            -0.005220954306423664,
+            -0.00745738111436367,
+            -0.0006659079226665199,
+            0.013607555069029331,
+            0.008150358684360981,
+            -0.01601722650229931,
+            -0.016599956899881363,
+            -0.03694828972220421,
+            0.007843242958188057,
+            0.012993324548006058,
+            -0.025246424600481987,
+            -0.014119412750005722,
+            -0.007567627355456352,
+            0.008433849550783634,
+            -0.01233972143381834,
+            -0.0014351718127727509,
+            -0.0033802331890910864,
+            0.014883263036608696,
+            -0.02493143454194069,
+            0.006827401462942362,
+            -0.01929311826825142,
+            0.01558411493897438,
+            0.009583561681210995,
+            0.009260697290301323,
+            -0.03612931817770004,
+            -0.01885213330388069,
+            0.023561229929327965,
+            -0.008205481804907322,
+            -0.009504813700914383,
+            0.024049462750554085,
+            -0.012969700619578362,
+            0.011134886182844639,
+            0.019875850528478622,
+            0.01145775057375431,
+            -0.0026537880767136812,
+            0.0218287855386734,
+            -0.006213171873241663,
+            7.259527774294838e-05,
+            -0.004512227606028318,
+            0.024332953616976738,
+            0.0063588544726371765,
+            0.032790426164865494,
+            -0.015300624072551727,
+            -0.018521394580602646,
+            -0.005760374013334513,
+            0.0442875511944294,
+            0.007559752557426691,
+            -0.02723085880279541,
+            0.018174905329942703,
+            0.029136545956134796,
+            -0.0032916422933340073,
+            0.005445384420454502,
+            -0.038239747285842896,
+            0.0004163770063314587,
+            0.015505367890000343,
+            0.021923283115029335,
+            0.011292380280792713,
+            -0.009922175668179989,
+            -0.018363898620009422,
+            0.007181765045970678,
+            0.002352579263970256,
+            0.006039927247911692,
+            -0.03990919515490532,
+            0.011449875310063362,
+            -0.020946815609931946,
+            0.01970260590314865,
+            0.00861496850848198,
+            -0.009378817863762379,
+            0.022458765655755997,
+            0.0028152205049991608,
+            -0.017088191583752632,
+            0.014710018411278725,
+            -0.005980866961181164,
+            0.0006673844181932509,
+            0.03329440951347351,
+            -0.0036184440832585096,
+            0.012717708945274353,
+            -0.007815681397914886,
+            0.0061383615247905254,
+            0.0031046171206980944,
+            0.03505835309624672,
+            -0.0008091297931969166,
+            -0.014458026736974716,
+            0.0068628378212451935,
+            0.005417822860181332,
+            -0.0005571380024775863,
+            -0.005059522110968828,
+            0.0077448091469705105,
+            -0.012670460157096386,
+            0.0014400935033336282,
+            -0.003053431399166584,
+            -0.006315543316304684,
+            -0.0030396506190299988,
+            0.002159648109227419,
+            0.00047543758410029113,
+            -0.033136915415525436,
+            0.02178153768181801,
+            -0.0042484235018491745,
+            0.013843797147274017,
+            -0.005913931410759687,
+            -0.01531637366861105,
+            0.0023643914610147476,
+            0.02074207179248333,
+            -0.005492632742971182,
+            -0.011898735538125038,
+            -0.01595422811806202,
+            -0.02238001860678196,
+            -0.026301639154553413,
+            0.028522316366434097,
+            0.0024254205636680126,
+            -0.01607234962284565,
+            0.008481097407639027,
+            -0.009591436013579369,
+            -0.025545664131641388,
+            0.006520286668092012,
+            0.018395397812128067,
+            -0.041641637682914734,
+            0.016048725694417953,
+            -0.006953397300094366,
+            -0.0025573226157575846,
+            0.028648313134908676,
+            0.006732904817909002,
+            -0.003504260443150997,
+            0.014450152404606342,
+            -0.0027384415734559298,
+            0.011945984326303005,
+            0.02165554091334343,
+            0.0023072995245456696,
+            0.0066384077072143555,
+            0.014198160730302334,
+            0.015521117486059666,
+            -0.002368328860029578,
+            -0.01837964914739132,
+            0.027624595910310745,
+            -0.021545294672250748,
+            0.02025383710861206,
+            0.021261803805828094,
+            -0.026427635923027992,
+            0.029388537630438805,
+            -0.00997729878872633,
+            0.019182872027158737,
+            0.0034491370897740126,
+            -0.00592968100681901,
+            -0.01233972143381834,
+            0.004949275404214859,
+            0.007484942674636841,
+            0.009300070814788342,
+            0.010331662371754646,
+            0.044004060328006744,
+            -0.01364692859351635,
+            0.028601063415408134,
+            0.03401888534426689,
+            -0.00010723184095695615,
+            0.02255326136946678,
+            -0.018584391102194786,
+            -0.02123030461370945,
+            0.007488879840821028,
+            -0.015780983492732048,
+            -0.03025476075708866,
+            0.0032483311370015144,
+            -0.01452889945358038,
+            -0.008284228853881359,
+            -0.016269218176603317,
+            -0.0019539203494787216,
+            -0.0022324894089251757,
+            -0.0009164231596514583,
+            0.004220861941576004,
+            0.020348334684967995,
+            0.026049647480249405,
+            0.02376597188413143,
+            0.01024504005908966,
+            0.00747706787660718,
+            0.01148924883455038,
+            -0.009930050000548363,
+            0.026175644248723984,
+            -0.0011713679414242506,
+            -0.0028624688275158405,
+            -0.0172929335385561,
+            0.007055769208818674,
+            -0.00029013503808528185,
+            -0.005110707599669695,
+            0.007425881922245026,
+            -0.022175274789333344,
+            -0.0015906979097053409,
+            -0.04687046632170677,
+            -0.002504168078303337,
+            -0.0007677873945795,
+            0.010394659824669361,
+            0.03965720161795616,
+            -0.007449506316334009,
+            -0.017922913655638695,
+            0.008284228853881359,
+            -0.01685194857418537,
+            0.004098803270608187,
+            -0.01814340613782406,
+            -0.015591990202665329,
+            -0.010756897740066051,
+            -0.02414396032691002,
+            -0.009315820410847664,
+            -0.013812297955155373,
+            0.015143129974603653,
+            -0.00876458827406168,
+            0.011229382827877998,
+            -0.018710387870669365,
+            0.00041736135608516634,
+            -0.006221046671271324,
+            -0.008433849550783634,
+            -0.002559291198849678,
+            -0.020348334684967995,
+            -0.012379094958305359,
+            0.022710757330060005,
+            -0.008780337870121002,
+            -0.029609031975269318,
+            -0.006311606150120497,
+            0.004232673905789852,
+            -0.014859639108181,
+            0.0037365653552114964,
+            0.00592968100681901,
+            -0.005917869042605162,
+            -0.008985080756247044,
+            0.00961506087332964,
+            0.018726136535406113,
+            0.012457842007279396,
+            -0.0028644376434385777,
+            0.00395705783739686,
+            0.02291550114750862,
+            0.02348248101770878,
+            -0.0454845130443573,
+            -0.005102833267301321,
+            0.017686670646071434,
+            0.016347965225577354,
+            -0.0070321448147296906,
+            0.018411148339509964,
+            -0.01507225725799799,
+            0.011827862821519375,
+            -0.02250601351261139,
+            -0.00612261239439249,
+            -0.026915868744254112,
+            0.00845747347921133,
+            0.01734018325805664,
+            -0.006768341176211834,
+            0.035215847194194794,
+            0.020600326359272003,
+            -0.016788950189948082,
+            0.020994063466787338,
+            -0.0013150819577276707,
+            0.011174259707331657,
+            -0.027089113369584084,
+            0.01364692859351635,
+            -0.009229198098182678,
+            0.008236980065703392,
+            0.014206035062670708,
+            -0.020049093291163445,
+            -0.025340920314192772,
+            -0.01222159992903471,
+            0.012174352072179317,
+            0.003065243363380432,
+            -0.0027896275278180838,
+            0.022222522646188736,
+            0.015143129974603653,
+            -0.01197748351842165,
+            -0.01740317977964878,
+            0.023561229929327965,
+            0.01276495773345232,
+            0.015418745577335358,
+            -0.0032581747509539127,
+            -0.0033290472347289324,
+            -0.017009442672133446,
+            0.0005093974177725613,
+            -0.02450619824230671,
+            -0.001176289631985128,
+            -0.002449044957756996,
+            0.021529546007514,
+            -0.007681811228394508,
+            -0.006224983837455511,
+            -0.007177827879786491,
+            -0.02808133140206337,
+            -0.000330000912072137,
+            0.00023365837114397436,
+            0.01252871472388506,
+            0.007658186834305525,
+            0.009473315440118313,
+            0.017481928691267967,
+            -0.026364637538790703,
+            0.002179334871470928,
+            -0.019482113420963287,
+            0.004126364830881357,
+            -0.00928432121872902,
+            -0.014292657375335693,
+            0.01033953670412302,
+            0.003470792667940259,
+            -0.00480753043666482,
+            0.01012691855430603,
+            0.003260143334046006,
+            -0.017387431114912033,
+            0.029939770698547363,
+            -0.004006274975836277,
+            0.002279737964272499,
+            0.003663723822683096,
+            0.005736749619245529,
+            0.0019736073445528746,
+            0.02827032469213009,
+            -0.025388170033693314,
+            -0.016773201525211334,
+            -0.018600141629576683,
+            0.0024372327607125044,
+            0.004988649394363165,
+            0.005386323668062687,
+            -0.0010079670464619994,
+            -0.00013190193567425013,
+            0.002657725475728512,
+            -0.0070321448147296906,
+            0.027876587584614754,
+            0.0040909284725785255,
+            -0.03382989391684532,
+            -0.024395952001214027,
+            -0.04038167744874954,
+            0.008307852782309055,
+            -0.01982860080897808,
+            -0.029766526073217392,
+            -0.010008797980844975,
+            -0.02274225652217865,
+            -0.035940323024988174,
+            0.018474144861102104,
+            0.020757820457220078,
+            -0.02025383710861206,
+            -0.008205481804907322,
+            -0.0028447506483644247,
+            -0.013142945244908333,
+            0.010048171505331993,
+            -0.0029805900994688272,
+            -0.016269218176603317,
+            0.010355286300182343,
+            0.014765141531825066,
+            0.0068628378212451935,
+            -0.00867009162902832,
+            -0.016347965225577354,
+            0.00028521331842057407,
+            0.023813221603631973,
+            0.0064651635475456715,
+            -0.01107188779860735,
+            0.007925928570330143,
+            0.006398227997124195,
+            -0.015111630782485008,
+            0.007437694352120161,
+            -0.023088743910193443,
+            -0.000766310840845108,
+            0.0025021994952112436,
+            0.014056415297091007,
+            -0.01644246280193329,
+            0.003551508765667677,
+            0.005618628580123186,
+            -0.0036873482167720795,
+            0.003545602783560753,
+            -0.018757635727524757,
+            0.00671715522184968,
+            -0.009748931042850018,
+            0.0019913255237042904,
+            0.028128579258918762,
+            -0.007232951000332832,
+            -0.005161893554031849,
+            -0.032412439584732056,
+            0.01642671227455139,
+            0.03360940143465996,
+            0.013678427785634995,
+            -0.021513795480132103,
+            -0.015985727310180664,
+            0.004283859860152006,
+            -0.0016468055546283722,
+            -0.019198622554540634,
+            0.015473868697881699,
+            0.019450614228844643,
+            0.012197976000607014,
+            0.015505367890000343,
+            0.013166569173336029,
+            0.014379279688000679,
+            -0.025451166555285454,
+            0.001903718919493258,
+            -0.0020454642362892628,
+            -0.02170279063284397,
+            0.012024731375277042,
+            0.02061607502400875,
+            -0.02378172241151333,
+            -0.003651911858469248,
+            0.0016153064789250493,
+            0.02165554091334343,
+            -0.003827124834060669,
+            -0.002248239004984498,
+            0.01808040775358677,
+            0.010252914391458035,
+            0.015119505114853382,
+            0.004795718006789684,
+            0.061045002192258835,
+            -0.003384170588105917,
+            0.007122704293578863,
+            0.008575594052672386,
+            -0.005106770433485508,
+            -0.01965535618364811,
+            -0.0010197791270911694,
+            -0.026537882164120674,
+            0.006654157303273678,
+            0.007752683945000172,
+            -0.02778209187090397,
+            -0.005902119446545839,
+            -0.009008705615997314,
+            -0.006595096550881863,
+            0.018048910424113274,
+            0.0047484696842730045,
+            -0.0020179026760160923,
+            0.006260420195758343,
+            0.037987757474184036,
+            0.014686394482851028,
+            -0.009504813700914383,
+            0.0023072995245456696,
+            0.010150542482733727,
+            0.007760558743029833,
+            -0.008268479257822037,
+            0.009040203876793385,
+            0.0004434464208316058,
+            0.004461041651666164,
+            -0.00800467561930418,
+            0.039625704288482666,
+            -0.009040203876793385,
+            0.0009188840049318969,
+            0.0023801408242434263,
+            -0.014426528476178646,
+            0.010938017629086971,
+            0.002990433480590582,
+            0.00729201128706336,
+            -0.007095142733305693,
+            0.0013308314373716712,
+            0.002222646027803421,
+            -0.0020001844968646765,
+            -0.0012343658600002527,
+            -0.009559936821460724,
+            -0.01655270904302597,
+            -0.005587129853665829,
+            0.013016949407756329,
+            0.0005856839707121253,
+            0.0052367039024829865,
+            -0.01812765747308731,
+            -0.0072447629645466805,
+            -0.0007126641576178372,
+            -0.011134886182844639,
+            0.007221138570457697,
+            0.024899935349822044,
+            -0.027498601004481316,
+            -0.004283859860152006,
+            0.014166661538183689,
+            -0.00015171183622442186,
+            0.007327447645366192,
+            0.014607647433876991,
+            -0.013119320385158062,
+            0.008000737987458706,
+            -0.016040850430727005,
+            -0.014347780495882034,
+            0.0007456396706402302,
+            -0.006492725107818842,
+            0.0023506106808781624,
+            -0.01307207252830267,
+            -0.0012058198917657137,
+            0.01279645599424839,
+            0.0006412993534468114,
+            0.012079854495823383,
+            0.014544649049639702,
+            -0.00679196510463953,
+            -0.00038364759529940784,
+            0.012694085016846657,
+            -0.012780707329511642,
+            -0.024128209799528122,
+            0.0007146328571252525,
+            0.023876218125224113,
+            -0.00492958864197135,
+            0.00383499963209033,
+            -0.0028506566304713488,
+            0.0024825125001370907,
+            -0.012662585824728012,
+            -0.013001199811697006,
+            -0.012213725596666336,
+            0.0209153164178133,
+            0.004571287892758846,
+            -0.022758005186915398,
+            0.0008258636225946248,
+            0.0055831922218203545,
+            0.009103202261030674,
+            -0.017497677356004715,
+            0.004000368993729353,
+            0.02995551936328411,
+            -0.01479664072394371,
+            -0.0041814884170889854,
+            -0.0008224184275604784,
+            -0.013812297955155373,
+            0.008044049143791199,
+            0.017088191583752632,
+            0.005484757944941521,
+            0.016363713890314102,
+            -0.003575132926926017,
+            -0.009441816247999668,
+            0.03038075566291809,
+            0.015308499336242676,
+            0.004185425583273172,
+            -0.006536036264151335,
+            -0.012788581661880016,
+            -0.028333323076367378,
+            0.004374419339001179,
+            0.010623027570545673,
+            -0.014292657375335693,
+            -0.0005876526702195406,
+            -0.012276723049581051,
+            -0.018001660704612732,
+            -0.004488603211939335,
+            -0.006524223834276199,
+            0.012300347909331322,
+            -0.00976468063890934,
+            0.010804146528244019,
+            -0.0003120366600342095,
+            -0.0035377279855310917,
+            -0.002352579263970256,
+            0.0002889046154450625,
+            -0.009410317055881023,
+            -0.03291642293334007,
+            -0.028412070125341415,
+            -0.008898458443582058,
+            -0.008142483420670033,
+            0.006528161466121674,
+            0.0009907410712912679,
+            -0.030664246529340744,
+            0.010300163179636002,
+            -0.00012433479423634708,
+            -0.014843889512121677,
+            0.015190377831459045,
+            0.009898550808429718,
+            -0.002703005215153098,
+            -0.007528253830969334,
+            -0.007807807065546513,
+            -0.02874280884861946,
+            0.01589910499751568,
+            0.0021872096695005894,
+            0.009260697290301323,
+            -0.014237534254789352,
+            0.0010424189968034625,
+            -0.021009812131524086,
+            0.01727718487381935,
+            -0.026474883779883385,
+            -0.015095881186425686,
+            -0.01522187702357769,
+            -0.0022128026466816664,
+            -0.0060320524498820305,
+            -0.017781168222427368,
+            -0.021907532587647438,
+            -0.006161985918879509,
+            -0.025640161707997322,
+            -0.022899750620126724,
+            -0.0017609893111512065,
+            -0.0034353563096374273,
+            0.010583654046058655,
+            -0.007040019612759352,
+            0.027986833825707436,
+            0.01619047112762928,
+            0.006579347420483828,
+            -0.011560121551156044,
+            -0.013442184776067734,
+            -0.0020198714919388294,
+            0.0025179488584399223,
+            -0.009158325381577015,
+            0.002384078223258257,
+            0.019797101616859436,
+            -0.0036184440832585096,
+            0.0045240395702421665,
+            0.004106678068637848,
+            -0.004035805352032185,
+            0.016836199909448624,
+            0.007988926023244858,
+            0.011788489297032356,
+            0.006315543316304684,
+            0.0016094004968181252,
+            0.03379839286208153,
+            0.016473960131406784,
+            -0.0035869451239705086,
+            0.0022324894089251757,
+            0.020962564274668694,
+            -0.02170279063284397,
+            -0.01571011170744896,
+            0.015450244769454002,
+            0.001383001683279872,
+            -0.03546784073114395,
+            0.02067907340824604,
+            0.01661570556461811,
+            -0.009796179831027985,
+            -0.010764773003757,
+            -0.007961364462971687,
+            -0.00021323325927369297,
+            -0.005181580316275358,
+            0.008544095791876316,
+            -0.011560121551156044,
+            0.013426436111330986,
+            0.006496662274003029,
+            -0.0064651635475456715,
+            -0.014269033446907997,
+            0.01422178465873003,
+            0.027971085160970688,
+            0.0035790703259408474,
+            -0.04819342494010925,
+            -0.012276723049581051,
+            0.01704094186425209,
+            -0.02140354923903942,
+            -0.025797655805945396,
+            0.00011172290396643803,
+            -0.0075715649873018265,
+            -0.023608477786183357,
+            -0.007063644006848335,
+            -0.00392752792686224,
+            -0.016836199909448624,
+            0.012883078306913376,
+            0.01064665149897337,
+            0.012426343746483326,
+            0.008205481804907322,
+            -0.015867605805397034,
+            -0.012064104899764061,
+            -0.042366113513708115,
+            0.00788655411452055,
+            -0.024836936965584755,
+            -0.021939031779766083,
+            0.004902027081698179,
+            -0.006827401462942362,
+            0.008740964345633984,
+            0.01452889945358038,
+            0.039153218269348145,
+            0.01255233958363533,
+            -0.015080131590366364,
+            -0.025687409564852715,
+            -0.017907164990901947,
+            -0.007961364462971687,
+            0.0022758005652576685,
+            -0.0074180071242153645,
+            -0.0035003230441361666,
+            0.0033192038536071777,
+            0.002246270189061761,
+            0.029309790581464767,
+            0.001104432623833418,
+            0.018962379544973373,
+            -3.389830453670584e-05,
+            0.00928432121872902,
+            -0.0038920913357287645,
+            -0.025025932118296623,
+            0.006213171873241663,
+            0.004106678068637848,
+            0.00794167723506689,
+            0.0068628378212451935,
+            0.005106770433485508,
+            0.016584208235144615,
+            -0.015867605805397034,
+            0.005394198466092348,
+            -0.026222892105579376,
+            -0.006469100713729858,
+            -0.014418653212487698,
+            0.0294042881578207,
+            0.014733643271028996,
+            -0.02327773906290531,
+            -0.013174444437026978,
+            -0.018993878737092018,
+            0.017970161512494087,
+            -0.008685841225087643,
+            0.009756805375218391,
+            0.01355243194848299,
+            0.01040253508836031,
+            0.006205297075212002,
+            0.0053548249416053295,
+            -0.005012273322790861,
+            0.007736934348940849,
+            0.02310449443757534,
+            -0.007106955163180828,
+            0.008000737987458706,
+            -0.0013849702663719654,
+            -0.0041814884170889854,
+            -0.014812390320003033,
+            -0.01722993701696396,
+            -0.0012570057297125459,
+            -0.00373853393830359,
+            0.01935611665248871,
+            -0.017623674124479294,
+            0.020001845434308052,
+            -0.011906610801815987,
+            0.005181580316275358,
+            -0.026490632444620132,
+            -0.008481097407639027,
+            0.01644246280193329,
+            -0.015505367890000343,
+            0.0024195145815610886,
+            -0.015458119101822376,
+            -0.003754283534362912,
+            0.006665969267487526,
+            0.001975575927644968,
+            0.001650742837227881,
+            0.005575317423790693,
+            0.005201267544180155,
+            0.016662955284118652,
+            0.008110984228551388,
+            -0.017450429499149323,
+            -0.0032562059350311756,
+            0.008370851166546345,
+            -0.015135254710912704,
+            0.006961272098124027,
+            0.005862745922058821,
+            0.0009361099801026285,
+            0.04082266613841057,
+            0.0066384077072143555,
+            0.002084837993606925,
+            -0.009898550808429718,
+            0.0037798762787133455,
+            -0.036727797240018845,
+            0.02904205024242401,
+            -0.0016576332272961736,
+            0.016221968457102776,
+            0.001489310641773045,
+            -0.014465902000665665,
+            0.0045791626907885075,
+            -0.005154018756002188,
+            0.014828139916062355,
+            -0.006287981756031513,
+            0.006969146896153688,
+            -0.00131114455871284,
+            0.017387431114912033,
+            -0.007339260075241327,
+            -0.018899381160736084,
+            0.009142575785517693,
+            -0.011316005140542984,
+            0.0007323510362766683,
+            -0.009693807922303677,
+            0.0055280691012740135,
+            0.01352880708873272,
+            0.0070203328505158424,
+            0.005480820778757334,
+            -0.03808225318789482,
+            -0.008622842840850353,
+            -0.003271955531090498,
+            0.002777815330773592,
+            -0.0024687317200005054,
+            -0.023970715701580048,
+            -0.024584945291280746,
+            -0.0002655264688655734,
+            0.02170279063284397,
+            -0.006587222218513489,
+            -0.0011831800220534205,
+            0.009118951857089996,
+            0.0015503398608416319,
+            -0.011749115772545338,
+            -0.006980959326028824,
+            0.007421944756060839,
+            0.021246055141091347,
+            -0.025514164939522743,
+            -0.020883817225694656,
+            0.012048356235027313,
+            0.0029274355620145798,
+            -0.019875850528478622,
+            -0.01033953670412302,
+            -0.0020651512313634157,
+            0.008512596599757671,
+            0.03439687564969063,
+            -0.017686670646071434,
+            0.001204835600219667,
+            0.0008947675814852118,
+            0.014080039225518703,
+            -0.026301639154553413,
+            -0.005043772514909506,
+            -0.01112701091915369,
+            -0.010418283753097057,
+            0.01148924883455038,
+            -0.01631646603345871,
+            0.0070518315769732,
+            -0.002840813249349594,
+            -0.002578978193923831,
+            -0.014064289629459381,
+            0.0009021501755341887,
+            0.0013810329837724566,
+            0.0016064473893493414,
+            0.03316841647028923,
+            -0.009567812085151672,
+            -0.013505183160305023,
+            0.0018968285294249654,
+            0.00041391613194718957,
+            0.014922636561095715,
+            0.010465532541275024,
+            0.01976560428738594,
+            -0.008331477642059326,
+            0.001558214658871293,
+            0.00961506087332964,
+            0.014623397029936314,
+            0.0005468024173751473,
+            0.042240116745233536,
+            -0.005795810371637344,
+            -0.006725030019879341,
+            -0.011788489297032356,
+            -0.005390261299908161,
+            -0.01233972143381834,
+            0.01491476222872734,
+            0.00269119325093925,
+            0.010371035896241665,
+            -0.0020828694105148315,
+            -0.008945707231760025,
+            0.012111353687942028,
+            0.007445569150149822,
+            0.010560029186308384,
+            0.013205942697823048,
+            -0.004014149773865938,
+            -0.02001759596168995,
+            0.010063921101391315,
+            -0.010048171505331993,
+            0.006059614475816488,
+            -0.001052262494340539,
+            0.008866960182785988,
+            -0.007996801286935806,
+            0.018631640821695328,
+            -0.0028703436255455017,
+            0.017607923597097397,
+            0.015560491010546684,
+            -0.019545109942555428,
+            0.017072441056370735,
+            0.01782841607928276,
+            -0.010788396932184696,
+            0.012780707329511642,
+            -0.01648971065878868,
+            -0.010237164795398712,
+            -0.0052524530328810215,
+            0.02176578715443611,
+            -0.013142945244908333,
+            -0.013324064202606678,
+            -0.010922268033027649,
+            0.00395508948713541,
+            -0.007988926023244858,
+            0.017324432730674744,
+            0.00588243268430233,
+            -0.009890676476061344,
+            -0.019686855375766754,
+            0.015505367890000343,
+            0.004433480091392994,
+            0.005213079508394003,
+            0.022474514320492744,
+            -0.007898367010056973,
+            0.005878495052456856,
+            0.014717893674969673,
+            0.030349256470799446,
+            -0.008323602378368378,
+            0.02831757441163063,
+            -0.011111261323094368,
+            0.00266756908968091,
+            -0.0037601895164698362,
+            -0.016158971935510635,
+            -0.012402718886733055,
+            -8.889845776138827e-05,
+            0.0050280229188501835,
+            0.012930327095091343,
+            -0.028601063415408134,
+            0.00861496850848198,
+            0.026222892105579376,
+            0.0027502537705004215,
+            0.008937832899391651,
+            0.0026537880767136812,
+            0.006890399381518364,
+            0.005386323668062687,
+            0.0029943708796054125,
+            -0.022364268079400063,
+            -0.007126641925424337,
+            0.00323848775587976,
+            -0.010851395316421986,
+            0.02104131132364273,
+            0.014355655759572983,
+            0.01067027635872364,
+            -0.002246270189061761,
+            -0.0012619274202734232,
+            0.025151927024126053,
+            0.01959235966205597,
+            0.004051554948091507,
+            -0.001164477551355958,
+            0.006854963023215532,
+            0.0023191117215901613,
+            0.011875111609697342,
+            -0.014190285466611385,
+            -0.014166661538183689,
+            0.007331385277211666,
+            0.00262228911742568,
+            0.010890768840909004,
+            -0.01680470071732998,
+            -0.006449413951486349,
+            -0.003303454490378499,
+            0.01746617816388607,
+            -0.009993048384785652,
+            -0.0018879694398492575,
+            0.0012599588371813297,
+            0.015253376215696335,
+            0.00039250668487511575,
+            -0.0032680181320756674,
+            -0.004724845290184021,
+            -0.014119412750005722,
+            0.0047996556386351585,
+            -0.009725307114422321,
+            0.004894152283668518,
+            -0.005661939736455679,
+            0.00881971139460802,
+            -0.007311698514968157,
+            0.04378356784582138,
+            0.027876587584614754,
+            0.008512596599757671,
+            0.017371682450175285,
+            -0.003986588213592768,
+            0.0017127564642578363,
+            -0.016119597479701042,
+            0.020222337916493416,
+            0.005598941817879677,
+            4.361868923297152e-05,
+            0.003966901451349258,
+            0.024915684014558792,
+            0.004728782922029495,
+            0.009733181446790695,
+            0.0006260420195758343,
+            0.006394290830940008,
+            0.0026439446955919266,
+            0.009410317055881023,
+            -0.018474144861102104,
+            -0.01507225725799799,
+            -0.014497400261461735,
+            0.02316749282181263,
+            -0.005512319505214691,
+            -0.008567719720304012,
+            0.013733550906181335,
+            0.011111261323094368,
+            -0.002498262096196413,
+            -0.014670644886791706,
+            -0.005693438928574324,
+            -0.0008736042072996497,
+            -0.005161893554031849,
+            -0.016269218176603317,
+            0.00297862128354609,
+            0.0020316834561526775,
+            -0.012182226404547691,
+            0.021372050046920776,
+            -0.012166476808488369,
+            -0.020395582541823387,
+            -0.003490479663014412,
+            -0.006843151059001684,
+            -0.04022418335080147,
+            -0.019056877121329308,
+            0.011906610801815987,
+            0.010788396932184696,
+            -0.008591343648731709,
+            -0.03480635955929756,
+            0.021261803805828094,
+            0.0060832384042441845,
+            0.03808225318789482,
+            -0.026065398007631302,
+            0.0023762034252285957,
+            0.0006777200032956898,
+            0.00397477624937892,
+            -0.011268756352365017,
+            -4.0081209590425715e-05,
+            0.019419115036725998,
+            -0.0007800916791893542,
+            0.00812673382461071,
+            -0.00542963482439518,
+            -0.010922268033027649,
+            0.014686394482851028,
+            -0.007209326606243849,
+            0.012906703166663647,
+            -0.006161985918879509,
+            -0.014442277140915394,
+            0.0032424251548945904,
+            0.016631456092000008,
+            0.012607462704181671,
+            -0.0017993786605075002,
+            -0.014670644886791706,
+            -0.009567812085151672,
+            0.00433504581451416,
+            -0.013331938534975052,
+            -0.012890953570604324,
+            0.02378172241151333,
+            -0.0166787039488554,
+            0.018111906945705414,
+            0.013205942697823048,
+            -0.017182687297463417,
+            0.004839029163122177,
+            0.006035990081727505,
+            0.006008428521454334,
+            0.007567627355456352,
+            -0.015064381994307041,
+            -0.019434863701462746,
+            0.004476791247725487,
+            0.026128394529223442,
+            -0.00240376521833241,
+            -0.006020240485668182,
+            0.012544464319944382,
+            -0.0026951306499540806,
+            0.008512596599757671,
+            -0.023387985303997993,
+            -0.0023821096401661634,
+            0.010993140749633312,
+            0.015568365342915058,
+            0.015686485916376114,
+            0.02335648611187935,
+            0.009181949310004711,
+            0.012402718886733055,
+            -0.012323971837759018,
+            0.005177643150091171,
+            0.022726505994796753,
+            -0.009630809538066387,
+            -0.018883632495999336,
+            0.011379002593457699,
+            0.01763942278921604,
+            0.023687224835157394,
+            0.0018456426914781332,
+            -0.010268663987517357,
+            0.030916238203644753,
+            0.012284598313272,
+            -0.009063828736543655,
+            0.014135162346065044,
+            -0.010906518436968327,
+            0.024962933734059334,
+            0.010237164795398712,
+            0.016174720600247383,
+            -0.01219010166823864,
+            -0.010835645720362663,
+            0.006646282505244017,
+            -0.011567996814846992,
+            -0.007410132791846991,
+            0.01249721646308899,
+            -0.020175090059638023,
+            -0.003384170588105917,
+            -0.031325723975896835,
+            0.00638247886672616,
+            0.003254237351939082,
+            0.0035790703259408474,
+            -0.02461644448339939,
+            0.005658002570271492,
+            -0.006146236322820187,
+            0.008079485967755318,
+            -0.011048263870179653,
+            -0.031262725591659546,
+            0.0039058721158653498,
+            -0.006496662274003029,
+            0.002059245016425848,
+            -0.01433990616351366,
+            0.01589122973382473,
+            0.007114829961210489,
+            -0.021088559180498123,
+            0.0010315912077203393,
+            -0.01904112659394741,
+            -0.0072565749287605286,
+            0.0003491956158541143,
+            -0.0023880156222730875,
+            -0.016347965225577354,
+            0.011363252997398376,
+            0.005886369850486517,
+            -0.02527792379260063,
+            -0.006937648169696331,
+            -0.0024687317200005054,
+            0.0007170937024056911,
+            0.016048725694417953,
+            -0.006055676843971014,
+            0.0022324894089251757,
+            0.033388908952474594,
+            0.015867605805397034,
+            -0.008488972671329975,
+            -0.025624411180615425,
+            -0.0028644376434385777,
+            -0.02359272725880146,
+            0.015631362795829773,
+            -0.0007953489548526704,
+            0.005650127772241831,
+            -0.029971269890666008,
+            0.005150081589818001,
+            0.006709280423820019,
+            0.01746617816388607,
+            0.006488787941634655,
+            0.01033953670412302,
+            0.026837121695280075,
+            -0.005106770433485508,
+            -0.001383001683279872,
+            -1.0535543879086617e-05,
+            0.00679196510463953,
+            0.018411148339509964,
+            -0.0036184440832585096,
+            0.01812765747308731,
+            -0.009607185609638691,
+            0.0029136547818779945,
+            -0.012654710561037064,
+            0.0035554461646825075,
+            -0.003462917869910598,
+            -0.013095696456730366,
+            -9.954166307579726e-05,
+            0.012686209753155708,
+            0.003395982552319765,
+            -0.0012727552093565464,
+            -0.018836384639143944,
+            -0.003833030816167593,
+            -0.0012530683306977153,
+            -0.021261803805828094,
+            0.0029628719203174114,
+            -0.01970260590314865,
+            -0.01124513242393732,
+            -0.02778209187090397,
+            -0.0025435418356209993,
+            0.04438204690814018,
+            0.00955206248909235,
+            -0.012331846170127392,
+            0.002647882094606757,
+            -0.02067907340824604,
+            0.0009956627618521452,
+            -0.012213725596666336,
+            -0.0036479744594544172,
+            -0.006886462215334177,
+            -0.013820173218846321,
+            0.014631271362304688,
+            0.01735593192279339,
+            -0.00404761778190732,
+            -0.006701405625790358,
+            0.012481466867029667,
+            -0.01070177461951971,
+            -0.009504813700914383,
+            -0.017576424404978752,
+            -0.017922913655638695,
+            -0.00785505585372448,
+            0.018883632495999336,
+            -0.005795810371637344,
+            0.02127755433320999,
+            0.008544095791876316,
+            0.0034353563096374273,
+            -0.00692977337166667,
+            0.0015168722020462155,
+            -0.021498046815395355,
+            -0.0006855947431176901,
+            0.006287981756031513,
+            -0.01433990616351366,
+            0.016599956899881363,
+            -0.006815589498728514,
+            -0.0031361160799860954,
+            0.008725214749574661,
+            0.004413793329149485,
+            -0.008725214749574661,
+            -0.005516257137060165,
+            -0.017135439440608025,
+            0.018820634111762047,
+            -0.018332399427890778,
+            -0.0027502537705004215,
+            0.00022246147273108363,
+            -0.009969423525035381,
+            0.012158602476119995,
+            0.015591990202665329,
+            0.007481005042791367,
+            -0.010378910228610039,
+            0.006610846146941185,
+            -0.010292287915945053,
+            0.0221437755972147,
+            -0.001872219960205257,
+            0.01612747274339199,
+            0.013024823740124702,
+            0.005433572456240654,
+            0.006284044589847326,
+            0.001713740755803883,
+            -0.012544464319944382,
+            -0.006949460133910179,
+            0.011064013466238976,
+            -0.02571890875697136,
+            -0.01349730882793665,
+            0.005917869042605162,
+            0.002801439492031932,
+            -0.01148924883455038,
+            -0.0007451475248672068,
+            -0.009205574169754982,
+            0.00037798762787133455,
+            -0.008481097407639027,
+            0.016710203140974045,
+            -0.03433387726545334,
+            -0.00861496850848198,
+            0.0017580362036824226,
+            0.031136730685830116,
+            -0.008449598215520382,
+            0.01619047112762928,
+            0.007532190997153521,
+            0.006488787941634655,
+            0.023687224835157394,
+            0.001509981811977923,
+            0.007662124466150999,
+            0.02116730809211731,
+            -0.011095511727035046,
+            0.00785505585372448,
+            0.0058036851696670055,
+            -0.01995459757745266,
+            0.025120427832007408,
+            -0.0061856103129684925,
+            0.005906056612730026,
+            0.002665600273758173,
+            0.001731458934955299,
+            0.02571890875697136,
+            0.027624595910310745,
+            -0.00870158988982439,
+            -0.005665877368301153,
+            -0.013379187323153019,
+            0.027262357994914055,
+            -0.004945338238030672,
+            0.007610938511788845,
+            -0.010197791270911694,
+            0.002689224435016513,
+            0.0007668030448257923,
+            -0.01024504005908966,
+            0.021482296288013458,
+            -0.016032975167036057,
+            -0.011875111609697342,
+            -0.017560675740242004,
+            -0.002472669119015336,
+            -0.024222707375884056,
+            0.013536682352423668,
+            0.0025829155929386616,
+            -0.003352671628817916,
+            0.0023821096401661634,
+            0.006484850309789181,
+            -0.003496385645121336,
+            -0.00940244272351265,
+            -0.0020887753926217556,
+            -0.01170186698436737,
+            -0.020348334684967995,
+            0.0166787039488554,
+            -0.007248700130730867,
+            -0.002116336952894926,
+            -0.0012127102818340063,
+            0.009811929427087307,
+            0.014985634945333004,
+            0.02129330299794674,
+            0.002915623364970088,
+            0.0066344705410301685,
+            0.007311698514968157,
+            -0.0019332492956891656,
+            -0.012032606638967991,
+            0.020427081733942032,
+            0.0197183545678854,
+            0.0003445199690759182,
+            -0.0031695838551968336,
+            -0.0109143927693367,
+            -0.035562336444854736,
+            -0.009733181446790695,
+            0.0008125749882310629,
+            0.011938109062612057,
+            -0.012442093342542648,
+            -0.007103017531335354,
+            0.039027221500873566,
+            -0.003220769576728344,
+            0.02110430970788002,
+            -0.004189362749457359,
+            3.842013029498048e-05,
+            0.01674170233309269,
+            0.004606724251061678,
+            0.012142852880060673,
+            -0.01264683622866869,
+            -0.02327773906290531,
+            0.002559291198849678,
+            -0.012961826287209988,
+            0.002697099233046174,
+            0.0004523055104073137,
+            -0.0002670029934961349,
+            -0.019497862085700035,
+            -0.008504721336066723,
+            0.005189455114305019,
+            -0.01128450594842434,
+            0.010040296241641045,
+            -0.004157864023000002,
+            0.017387431114912033,
+            -0.0030179950408637524,
+            0.0037897196598351,
+            -0.008583469316363335,
+            -0.006811652332544327,
+            0.0011575871612876654,
+            -0.0017304746434092522,
+            -0.017198437824845314,
+            -8.342305227415636e-05,
+            -0.0020887753926217556,
+            -0.0003410747740417719,
+            0.008410224691033363,
+            0.008378725498914719,
+            0.01328469067811966,
+            0.01385954674333334,
+            -0.0009119936148636043,
+            0.006996708456426859,
+            0.009008705615997314,
+            0.02735685557126999,
+            -0.007185702212154865,
+            0.0068077147006988525,
+            0.002291549928486347,
+            0.008012549951672554,
+            -0.012946076691150665,
+            -0.03285342454910278,
+            0.015426619909703732,
+            -0.015780983492732048,
+            -0.0048035928048193455,
+            -0.014717893674969673,
+            0.01040253508836031,
+            -0.004445292055606842,
+            0.0010876987362280488,
+            0.006098988000303507,
+            0.00845747347921133,
+            0.004937463440001011,
+            0.019308868795633316,
+            -0.007213264238089323,
+            0.0071935770101845264,
+            0.021309053525328636,
+            -0.003295579692348838,
+            -0.002515980275347829,
+            0.006563597824424505,
+            0.009323694743216038,
+            -0.01644246280193329,
+            0.01103251427412033,
+            0.024647943675518036,
+            -0.023198990151286125,
+            0.004441354889422655,
+            0.002891999203711748,
+            -0.01070177461951971,
+            0.004126364830881357,
+            0.0065320986323058605,
+            -0.00033024701406247914,
+            0.023009996861219406,
+            0.004587037488818169,
+            0.019576609134674072,
+            0.004022024571895599,
+            -0.014670644886791706,
+            -0.015599864535033703,
+            0.006646282505244017,
+            -0.010890768840909004,
+            -0.011016764678061008,
+            -0.008063736371695995,
+            -0.010693900287151337,
+            0.0054611340165138245,
+            -0.004787843208760023,
+            -0.004022024571895599,
+            -0.0037739702966064215,
+            -0.011253006756305695,
+            0.006827401462942362,
+            -0.01148924883455038,
+            0.001115260412916541,
+            -0.00638247886672616,
+            -0.005280014593154192,
+            0.00945756584405899,
+            0.011851487681269646,
+            -0.010008797980844975,
+            -0.02674262598156929,
+            0.005555630661547184,
+            0.031199729070067406,
+            -0.006728967186063528,
+            0.007929865270853043,
+            0.012355471029877663,
+            -0.00745738111436367,
+            0.000588144815992564,
+            -0.006732904817909002,
+            -0.004650035407394171,
+            -0.03105798363685608,
+            -0.0010227321181446314,
+            -0.01067027635872364,
+            0.0003351687337271869,
+            -0.014245408587157726,
+            -0.015017134137451649,
+            -0.013166569173336029,
+            0.004968962632119656,
+            -0.013142945244908333,
+            -0.005043772514909506,
+            0.004149989224970341,
+            0.006988833658397198,
+            -0.016379464417696,
+            0.017576424404978752,
+            -0.00943394098430872,
+            -0.003933433908969164,
+            -0.00592968100681901,
+            0.033451907336711884,
+            -0.005406010430306196,
+            -0.003504260443150997,
+            -0.00492958864197135,
+            0.008874834515154362,
+            0.006776215974241495,
+            0.002053339034318924,
+            -0.0029038111679255962,
+            0.0021379925310611725,
+            0.02323048934340477,
+            0.006232858635485172,
+            0.003397951368242502,
+            0.015536867082118988,
+            -0.013473683968186378,
+            -0.008044049143791199,
+            -0.012654710561037064,
+            -0.01067027635872364,
+            0.0020454642362892628,
+            0.010985265485942364,
+            0.02365572564303875,
+            -0.005083146039396524,
+            -0.014985634945333004,
+            0.011473499238491058,
+            0.006772278342396021,
+            0.02140354923903942,
+            -0.011843612417578697,
+            5.6138298532459885e-05,
+            0.00812673382461071,
+            0.02376597188413143,
+            0.031813960522413254,
+            -0.000510381767526269,
+            7.437940075760707e-05,
+            0.0083393519744277,
+            0.010615152306854725,
+            -0.004980774596333504,
+            0.00664234533905983,
+            -0.01704094186425209,
+            -0.02795533463358879,
+            -0.008898458443582058,
+            0.006646282505244017,
+            -0.001707834773696959,
+            0.018647389486432076,
+            0.01007179543375969,
+            0.016757450997829437,
+            0.02382897026836872,
+            0.023025747388601303,
+            0.00039545970503240824,
+            0.02037983387708664,
+            -0.004425605293363333,
+            -0.004417730495333672,
+            0.0025691348128020763,
+            -0.015048632398247719,
+            -0.03471186384558678,
+            0.006118674762547016,
+            0.00442954245954752,
+            0.024773940443992615,
+            -0.015757359564304352,
+            0.023561229929327965,
+            -0.004146052058786154,
+            -0.016158971935510635,
+            -0.005898181814700365,
+            0.020789319649338722,
+            -0.008772462606430054,
+            0.020789319649338722,
+            0.0013239410473033786,
+            -0.0010945892427116632,
+            0.00988280214369297,
+            -0.0011201821034774184,
+            -0.01379654835909605,
+            0.011253006756305695,
+            0.01763942278921604,
+            0.03770426660776138,
+            0.009630809538066387,
+            -0.008118859492242336,
+            0.01752917654812336,
+            0.001776738790795207,
+            -0.008394475094974041,
+            -0.004433480091392994,
+            -0.013820173218846321,
+            0.0076936231926083565,
+            -0.0074180071242153645,
+            0.01697794534265995,
+            0.002640007296577096,
+            -0.005260327830910683,
+            0.014725768007338047,
+            -0.012481466867029667,
+            0.01219010166823864,
+            0.021246055141091347,
+            0.0011949921026825905,
+            0.005886369850486517,
+            0.020001845434308052,
+            -0.033136915415525436,
+            0.011528623290359974,
+            -0.034837860614061356,
+            -0.005138269625604153,
+            0.012064104899764061,
+            -0.01861589029431343,
+            0.007732997182756662,
+            -0.018584391102194786,
+            -0.013253191486001015,
+            -0.002553385216742754,
+            0.014599772170186043,
+            0.02497868239879608,
+            0.006555723026394844,
+            -0.017670921981334686,
+            0.01179636362940073,
+            -0.00876458827406168,
+            -0.01825365237891674,
+            -0.008874834515154362,
+            -0.019419115036725998,
+            -0.011693992651998997,
+            -0.003850748995319009,
+            0.011938109062612057,
+            -0.008811837062239647,
+            0.0037011289969086647,
+            0.002248239004984498,
+            -0.0221437755972147,
+            -0.013883170671761036,
+            0.015914853662252426,
+            0.003413700731471181,
+            -0.00664234533905983,
+            -0.007343197241425514,
+            0.013016949407756329,
+            -0.010756897740066051,
+            0.007323510479182005,
+            -0.008181856945157051,
+            0.014253283850848675,
+            -0.012142852880060673,
+            -0.00319123943336308,
+            0.007079393602907658,
+            0.0035869451239705086,
+            -0.018033159896731377,
+            -0.004732720088213682,
+            -0.02493143454194069,
+            0.015355747193098068,
+            0.0029766527004539967,
+            -0.017481928691267967,
+            -0.00692977337166667,
+            0.007732997182756662,
+            -0.017922913655638695,
+            -0.004894152283668518,
+            0.008717339485883713,
+            -0.003990525845438242,
+            0.025340920314192772,
+            0.022301269695162773,
+            -0.01248934119939804,
+            0.00016143222455866635,
+            -0.009213448502123356,
+            0.009985173121094704,
+            -0.01856864243745804,
+            -0.004484665580093861,
+            0.02650638297200203,
+            0.007315635681152344,
+            -0.00550444470718503,
+            -0.009937925264239311,
+            0.001173336640931666,
+            0.002771909348666668,
+            0.017607923597097397,
+            0.02493143454194069,
+            0.00621710903942585,
+            -0.010985265485942364,
+            -0.022458765655755997,
+            -0.027388354763388634,
+            0.027372604236006737,
+            0.0007485927199013531,
+            -0.004118490032851696,
+            0.013276815414428711,
+            0.02529367245733738,
+            0.023939216509461403,
+            -0.0062013594433665276,
+            -0.00483509199693799,
+            0.018300902098417282,
+            0.00997729878872633,
+            -0.005402073264122009,
+            -0.010166292078793049,
+            0.003392045386135578,
+            -0.0071345167234539986,
+            0.03414488211274147,
+            0.010111168958246708,
+            0.004417730495333672,
+            -0.046366482973098755,
+            0.009993048384785652,
+            -0.01619047112762928,
+            -0.003758220700547099,
+            0.008236980065703392,
+            0.021734289824962616,
+            0.010095419362187386,
+            0.013032699003815651,
+            0.007984988391399384,
+            -0.007236888166517019,
+            0.02861681394279003,
+            -0.0028703436255455017,
+            -0.010623027570545673,
+            -0.007272324524819851,
+            -0.025687409564852715,
+            0.004342920612543821,
+            -0.016410963609814644,
+            0.0021517733111977577,
+            0.005677689332515001,
+            -0.00918982457369566,
+            0.02590790204703808,
+            0.00794167723506689,
+            -0.02382897026836872,
+            0.00588243268430233,
+            -0.03682229667901993,
+            0.019482113420963287,
+            -0.02619139291346073,
+            -0.009323694743216038,
+            -0.004862653557211161,
+            -0.003758220700547099,
+            -0.025923652574419975,
+            -0.0037365653552114964,
+            -0.004783906042575836,
+            0.016883447766304016,
+            0.0024017964024096727,
+            -0.00013805407797917724,
+            -0.00013399365707300603,
+            -0.007103017531335354,
+            0.002041527070105076,
+            0.012410594150424004,
+            0.022521764039993286,
+            0.01571798510849476,
+            -0.0037897196598351,
+            0.018521394580602646,
+            -0.0005768248811364174,
+            -0.03704278916120529,
+            0.0037168783601373434,
+            -0.011662493459880352,
+            0.008103109896183014,
+            0.015332123264670372,
+            0.009591436013579369,
+            -0.01370205171406269,
+            0.013190193101763725,
+            -0.012150727212429047,
+            0.015395121648907661,
+            0.01619047112762928,
+            0.004894152283668518,
+            -0.0009976314613595605,
+            0.0157652348279953,
+            -0.02389196865260601,
+            0.00020941892580594867,
+            -0.00626435736194253,
+            0.006713218055665493,
+            -0.009930050000548363,
+            -0.0033251100685447454,
+            0.002041527070105076,
+            0.0006855947431176901,
+            -0.0034038573503494263,
+            -0.012064104899764061,
+            -0.017387431114912033,
+            -0.021498046815395355,
+            -0.01428478304296732,
+            -0.020820818841457367,
+            -0.011001015082001686,
+            0.030711494386196136,
+            -0.008622842840850353,
+            0.004559475928544998,
+            0.015025008469820023,
+            0.018710387870669365,
+            0.002596696373075247,
+            0.01856864243745804,
+            0.008260604925453663,
+            -0.007299886085093021,
+            -0.01727718487381935,
+            0.005543818697333336,
+            -0.025435417890548706,
+            0.03310541808605194,
+            0.00933944433927536,
+            -0.014032791368663311,
+            0.012969700619578362,
+            -0.01070177461951971,
+            -0.004346857778728008,
+            -0.007784182671457529,
+            0.0060163033194839954,
+            -0.025498416274785995,
+            0.01455252431333065,
+            -0.011749115772545338,
+            0.006961272098124027,
+            0.002305330941453576,
+            0.03505835309624672,
+            -0.014489525929093361,
+            -0.008929957635700703,
+            0.012032606638967991,
+            0.019119873642921448,
+            0.026553630828857422,
+            -0.009441816247999668,
+            -0.004937463440001011,
+            -0.01814340613782406,
+            -0.00357119576074183,
+            -0.005906056612730026,
+            -0.018631640821695328,
+            0.03590882569551468,
+            -0.007847180590033531,
+            -0.021387800574302673,
+            -0.00824485532939434,
+            -0.02006484381854534,
+            0.0031262726988643408,
+            0.0315462164580822,
+            0.01040253508836031,
+            0.006890399381518364,
+            -8.92060634214431e-05,
+            -0.020285336300730705,
+            -0.021324802190065384,
+            0.019797101616859436,
+            0.012166476808488369,
+            -0.019844351336359978,
+            -0.0009695776388980448,
+            -0.009394567459821701,
+            -0.00216752290725708,
+            0.0006609861738979816,
+            0.0021773662883788347,
+            0.05780060961842537,
+            -0.008087360300123692,
+            0.0013229567557573318,
+            -0.024065213277935982,
+            -0.001320003648288548,
+            -0.003332984633743763,
+            -0.0035318220034241676,
+            0.005646190140396357,
+            -0.00626435736194253,
+            0.01128450594842434,
+            -0.03480635955929756,
+            0.006784090772271156,
+            -0.010607277974486351,
+            -0.009654434397816658,
+            -0.016111722216010094,
+            0.00166747672483325,
+            -0.018285151571035385,
+            -4.6664001274621114e-05,
+            0.00021680150530301034,
+            -0.012134977616369724,
+            0.004083054140210152,
+            -0.013883170671761036,
+            -0.0003961979818996042,
+            0.024553446099162102,
+            0.011686117388308048,
+            -0.0013268940383568406,
+            -0.022222522646188736,
+            -0.022395767271518707,
+            0.0015562459593638778,
+            -0.010394659824669361,
+            0.01801741123199463,
+            -0.013229567557573318,
+            0.017182687297463417,
+            -0.02165554091334343,
+            -0.00046780891716480255,
+            0.0019765603356063366,
+            0.006232858635485172,
+            0.01861589029431343,
+            -0.00010969270078931004,
+            0.009693807922303677,
+            -0.014993509277701378,
+            0.02535667084157467,
+            -0.006949460133910179,
+            -0.02019083872437477,
+            0.0007328432402573526,
+            0.005181580316275358,
+            -0.0044216676615178585,
+            -0.0003430434735491872,
+            0.011056138202548027,
+            0.019419115036725998,
+            -0.0024766065180301666,
+            -0.007839306257665157,
+            -0.014064289629459381,
+            -0.009930050000548363,
+            0.0194033645093441,
+            0.0009297117940150201,
+            0.02055307850241661,
+            -0.011536497622728348,
+            0.00918982457369566,
+            -0.017623674124479294,
+            -0.007662124466150999,
+            -0.00907170306891203,
+            -0.004022024571895599,
+            0.016725953668355942,
+            -0.0007599126547574997,
+            0.01512738037854433,
+            -0.018001660704612732,
+            -0.021907532587647438,
+            0.0006098003359511495,
+            -0.02348248101770878,
+            -0.018411148339509964,
+            0.009205574169754982,
+            -0.01855289377272129,
+            -0.0025986649561673403,
+            0.001046356395818293,
+            -0.020883817225694656,
+            -0.013473683968186378,
+            0.018694639205932617,
+            -0.001501122722402215,
+            -0.0029628719203174114,
+            0.008496847003698349,
+            -0.006343104876577854,
+            -0.021813036873936653,
+            -0.0039688702672719955,
+            0.009512688964605331,
+            0.009748931042850018,
+            0.012993324548006058,
+            -0.022222522646188736,
+            0.004724845290184021,
+            0.0015719954390078783,
+            -0.005350887309759855,
+            -0.007095142733305693,
+            0.018174905329942703,
+            0.008047986775636673,
+            0.004531914368271828,
+            0.007768433541059494,
+            0.010008797980844975,
+            -0.0009479221189394593,
+            0.008370851166546345,
+            0.02146654762327671,
+            0.00108376145362854,
+            -0.002827032469213009,
+            -0.016347965225577354,
+            0.00612261239439249,
+            -0.012292472645640373,
+            0.015473868697881699,
+            0.02589215338230133,
+            0.008355101570487022,
+            -0.016332216560840607,
+            0.020883817225694656,
+            -0.014481651596724987,
+            0.015151004306972027,
+            -0.0282073263078928,
+            -0.00988280214369297,
+            0.018159156665205956,
+            -0.0011546340538188815,
+            0.01842689700424671,
+            -0.003108554519712925,
+            0.024065213277935982,
+            0.012481466867029667,
+            0.006417915225028992,
+            1.7518224922241643e-05,
+            -0.0007170937024056911,
+            0.010063921101391315,
+            -0.005189455114305019,
+            0.00014851271407678723,
+            -0.005740687251091003,
+            0.009134701453149319,
+            -0.006551785860210657,
+            0.00027832292835228145,
+            0.010725399479269981,
+            -0.028065582737326622,
+            0.014332030899822712,
+            -0.02274225652217865,
+            -0.0036735672038048506,
+            -0.0024431387428194284,
+            -0.007146328687667847,
+            -0.02031683549284935,
+            -0.003079024376347661,
+            0.019088376313447952,
+            -0.018489895388484,
+            0.04819342494010925,
+            0.0012028669007122517,
+            0.018300902098417282,
+            -0.007453443482518196,
+            0.002006090711802244,
+            -0.015143129974603653,
+            -0.00035928512807004154,
+            -0.02674262598156929,
+            0.00807161070406437,
+            0.003821218851953745,
+            0.0024254205636680126,
+            0.0025238548405468464,
+            0.006902211811393499,
+            -0.002173428889364004,
+            0.026648128405213356,
+            0.0002606047492008656,
+            0.01127663068473339,
+            -0.004295671824365854,
+            0.010717524215579033,
+            -0.019261619076132774,
+            0.004327171016484499,
+            0.0034491370897740126,
+            -0.00797711405903101,
+            -0.003933433908969164,
+            -0.007091205567121506,
+            0.0221437755972147,
+            0.016631456092000008,
+            0.009662308730185032,
+            -0.008020425215363503,
+            -0.0013042541686445475,
+            0.006803777534514666,
+            -0.0030396506190299988,
+            -0.011756990104913712,
+            -0.0038724045734852552,
+            0.005969054531306028,
+            0.012512966059148312,
+            -0.0009011658257804811,
+            0.011961733922362328,
+            0.005362699739634991,
+            -0.008008613251149654,
+            0.01297757588326931,
+            -0.020096343010663986,
+            -0.006390353664755821,
+            -0.009685933589935303,
+            -0.033987388014793396,
+            0.009292195551097393,
+            -0.0017304746434092522,
+            0.003090836340561509,
+            -0.007229013368487358,
+            0.010260789655148983,
+            -0.009520563296973705,
+            -0.01322169229388237,
+            -0.006512411870062351,
+            0.002208865247666836,
+            0.023639976978302002,
+            -0.006193485110998154,
+            0.00240376521833241,
+            -0.013355563394725323,
+            -0.013804423622786999,
+            0.016899196431040764,
+            -0.01170186698436737,
+            -0.007343197241425514,
+            0.009827678091824055
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "396d9412-17a9-4b7e-831f-84f4e5006484",
+        "properties": {
+          "page_content": "9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\n",
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "MoE language models",
+            "supervised fine-tuning",
+            "large-scale language models"
+          ],
+          "embedding": [
+            -0.003971853759139776,
+            -0.012639197520911694,
+            -0.032538022845983505,
+            0.01764647848904133,
+            0.028833750635385513,
+            0.017460335046052933,
+            0.0033017343375831842,
+            0.0448235459625721,
+            -0.002440816955640912,
+            0.05025895684957504,
+            -0.011708475649356842,
+            -0.010033177211880684,
+            0.026748934760689735,
+            -0.023044662550091743,
+            -0.0071293264627456665,
+            0.011140735819935799,
+            0.007008332759141922,
+            0.022765446454286575,
+            0.005472642369568348,
+            -0.03823403641581535,
+            0.014482025988399982,
+            -0.014649556018412113,
+            -0.05766750127077103,
+            -0.008972154930233955,
+            0.008334610611200333,
+            0.017088046297430992,
+            0.018679579719901085,
+            0.0209412332624197,
+            -0.06805435568094254,
+            0.020010510459542274,
+            0.045344747602939606,
+            0.012024921365082264,
+            -0.0010371727403253317,
+            -0.006017114035785198,
+            -0.006594161503016949,
+            -0.03143977001309395,
+            0.04270150139927864,
+            0.015459283255040646,
+            -0.024254601448774338,
+            0.02741905488073826,
+            0.02278406172990799,
+            -0.006980410777032375,
+            -0.02874067798256874,
+            -0.009809804148972034,
+            0.00014906085561960936,
+            0.007208437658846378,
+            -0.006026421207934618,
+            0.005379570182412863,
+            -0.0071293264627456665,
+            -0.005030549596995115,
+            -0.012173837050795555,
+            -0.014119044877588749,
+            0.02192779816687107,
+            -0.01348615437746048,
+            -0.025278395041823387,
+            0.010154170915484428,
+            0.014649556018412113,
+            0.0407283715903759,
+            0.04463740065693855,
+            0.026693090796470642,
+            0.021778881549835205,
+            -0.030192604288458824,
+            0.003183067310601473,
+            -0.000935375050175935,
+            0.02533423714339733,
+            0.013030100613832474,
+            0.004416273441165686,
+            0.03518127277493477,
+            0.023658938705921173,
+            -0.02598574385046959,
+            -0.049328237771987915,
+            0.02230008691549301,
+            -0.007487654220312834,
+            -0.012797419913113117,
+            0.004241763148456812,
+            -0.00017014751210808754,
+            0.0030667271930724382,
+            0.0407283715903759,
+            -0.023510023951530457,
+            0.003876454895362258,
+            0.010749832727015018,
+            0.013662991113960743,
+            0.010498537681996822,
+            -0.0014414548641070724,
+            0.04325993359088898,
+            -0.014845007099211216,
+            -0.010051791556179523,
+            -0.09188082069158554,
+            0.0304159764200449,
+            -0.010126249864697456,
+            -0.020475871860980988,
+            0.006919913925230503,
+            0.013616454787552357,
+            -0.011280343867838383,
+            -0.009781882166862488,
+            0.027605198323726654,
+            0.01356991846114397,
+            0.006398709956556559,
+            -0.0263208020478487,
+            0.01287187822163105,
+            -0.013030100613832474,
+            0.0022663066629320383,
+            0.012918413616716862,
+            0.003997448831796646,
+            -0.00012244803656358272,
+            -0.008799971081316471,
+            -0.029466642066836357,
+            0.030267061665654182,
+            0.014156273566186428,
+            0.10290056467056274,
+            0.00873482134193182,
+            -0.0014821739168837667,
+            -0.035088200122117996,
+            0.015133530832827091,
+            -0.028722064569592476,
+            -0.01276949793100357,
+            -0.0439300537109375,
+            -0.03203543275594711,
+            0.005868198815733194,
+            0.0019475346198305488,
+            0.03531157225370407,
+            0.01991743966937065,
+            0.02561345510184765,
+            -0.0014717032900080085,
+            -0.06310291588306427,
+            0.046982817351818085,
+            -0.015608198940753937,
+            -0.02019665576517582,
+            0.010358929634094238,
+            -0.004600090906023979,
+            -0.016101481392979622,
+            -0.018028074875473976,
+            -0.033115070313215256,
+            -0.04102620109915733,
+            0.0028037982992827892,
+            -0.010991820134222507,
+            0.014584405347704887,
+            0.06392195075750351,
+            -0.040951743721961975,
+            -0.01026585791260004,
+            0.015989795327186584,
+            0.010935977101325989,
+            0.009302561171352863,
+            -0.020866775885224342,
+            -0.029299112036824226,
+            -0.0041649783961474895,
+            0.0076691447757184505,
+            0.08391384780406952,
+            -0.007845981977880001,
+            0.02118322066962719,
+            -0.024831648916006088,
+            0.011205886490643024,
+            0.011969078332185745,
+            0.004658260848373175,
+            0.0013623435515910387,
+            -0.013523383066058159,
+            -0.030490435659885406,
+            0.005826316308230162,
+            -0.021034304052591324,
+            0.02980170026421547,
+            0.0025920593179762363,
+            -0.025632068514823914,
+            -0.0009185057133436203,
+            0.01166194025427103,
+            -0.0058030481450259686,
+            0.017702322453260422,
+            0.021127376705408096,
+            0.0077249882742762566,
+            -0.060087379068136215,
+            0.053758472204208374,
+            0.01866096444427967,
+            0.0218347255140543,
+            -0.024329058825969696,
+            0.004721084609627724,
+            0.00043831163202412426,
+            0.017972230911254883,
+            0.0070595224387943745,
+            0.011410645209252834,
+            -0.005281844176352024,
+            0.033561814576387405,
+            -0.015245217829942703,
+            0.023863697424530983,
+            0.021648580208420753,
+            -0.0014717032900080085,
+            0.030453206971287727,
+            0.013495461083948612,
+            0.018474821001291275,
+            0.02894543670117855,
+            -0.0181862972676754,
+            0.005430759862065315,
+            0.014044586569070816,
+            0.00012295703345444053,
+            0.029448026791214943,
+            -0.006738423369824886,
+            -0.007389928214251995,
+            -0.037340544164180756,
+            -0.018484128639101982,
+            0.02455243282020092,
+            -0.0004868836549576372,
+            0.009372364729642868,
+            -0.00881858542561531,
+            -0.03901584446430206,
+            -0.004251070320606232,
+            -0.012946335598826408,
+            0.028870979323983192,
+            0.005100353620946407,
+            -0.011150043457746506,
+            0.005807701963931322,
+            0.022058097645640373,
+            -0.013672297820448875,
+            -0.012434438802301884,
+            -0.007520229555666447,
+            0.002373339608311653,
+            -0.02220701426267624,
+            -0.0041021546348929405,
+            0.0037182322703301907,
+            -0.01921009086072445,
+            0.0027642427012324333,
+            0.008585905656218529,
+            -0.00825549941509962,
+            -0.01627831906080246,
+            0.015040459111332893,
+            0.03441808000206947,
+            -0.016203859820961952,
+            -0.03585138916969299,
+            -0.024310443550348282,
+            -0.022560687735676765,
+            0.018996024504303932,
+            0.028498690575361252,
+            -0.009511973708868027,
+            0.03091856651008129,
+            0.008367185480892658,
+            0.014295881614089012,
+            0.004711777437478304,
+            0.021127376705408096,
+            -0.008074008859694004,
+            -0.011354802176356316,
+            -0.018791265785694122,
+            0.014212116599082947,
+            0.026488332077860832,
+            0.00021682900842279196,
+            0.018260754644870758,
+            -0.04556812345981598,
+            -0.011540945619344711,
+            0.0020394434686750174,
+            -0.0006038055289536715,
+            -0.012406516820192337,
+            -0.016864672303199768,
+            0.01268573384732008,
+            0.029652785509824753,
+            0.008381146937608719,
+            -0.016557535156607628,
+            -0.002878256142139435,
+            0.00434181559830904,
+            -0.04084005579352379,
+            0.052455462515354156,
+            0.03990933671593666,
+            -0.012648504227399826,
+            -0.024366287514567375,
+            -0.033282600343227386,
+            0.05710906907916069,
+            -0.02380785532295704,
+            -0.025222551077604294,
+            0.06961796432733536,
+            -0.013849135488271713,
+            0.019768523052334785,
+            0.05707183852791786,
+            0.0026944386772811413,
+            0.004486077465116978,
+            -0.01794430986046791,
+            0.026227731257677078,
+            -0.021369364112615585,
+            -0.01190392766147852,
+            0.006394056603312492,
+            0.03030429035425186,
+            -0.012332059442996979,
+            -0.01941484957933426,
+            -0.002408241853117943,
+            0.0011081402190029621,
+            0.015059073455631733,
+            0.002245365409180522,
+            0.009539894759654999,
+            0.013514075428247452,
+            0.012536818161606789,
+            0.00968881044536829,
+            -0.0047303917817771435,
+            -0.01110350713133812,
+            -0.04325993359088898,
+            -0.03162591531872749,
+            -0.01740449108183384,
+            -0.035572174936532974,
+            0.0013146440032869577,
+            -0.021909182891249657,
+            0.011205886490643024,
+            0.03786174952983856,
+            -0.009027997963130474,
+            0.003348270431160927,
+            0.018884338438510895,
+            -0.0023140062112361193,
+            -0.0005747204995714128,
+            -0.031718987971544266,
+            -0.018577199429273605,
+            -0.00893958006054163,
+            0.00128555903211236,
+            -0.05126413702964783,
+            -0.01171778328716755,
+            0.008418375626206398,
+            0.03103025257587433,
+            -0.02250484563410282,
+            -0.023621710017323494,
+            0.015878107398748398,
+            -0.019805751740932465,
+            -0.028647607192397118,
+            -0.02931772544980049,
+            -0.019191475585103035,
+            0.05465196445584297,
+            0.016706449910998344,
+            0.025967128574848175,
+            -0.03147700056433678,
+            -0.03423193469643593,
+            0.05025895684957504,
+            0.008474218659102917,
+            -0.07550012320280075,
+            0.036595966666936874,
+            0.0014507620362564921,
+            0.0010063425870612264,
+            -0.005495910067111254,
+            0.05837484821677208,
+            -0.0032575251534581184,
+            -0.0010057608596980572,
+            0.012536818161606789,
+            -0.05748135596513748,
+            -0.00575651228427887,
+            -0.006412670947611332,
+            0.013253473676741123,
+            -0.04642438516020775,
+            -0.024943334981799126,
+            0.004027697257697582,
+            -0.012974257580935955,
+            -0.005700668785721064,
+            -0.042180296033620834,
+            -0.03145838528871536,
+            -0.0028526613023132086,
+            -0.0022756140679121017,
+            -0.007943707518279552,
+            -0.03078826516866684,
+            -0.025929899886250496,
+            0.021164605394005775,
+            0.045344747602939606,
+            0.007641223259270191,
+            0.0017695341957733035,
+            0.023621710017323494,
+            0.0197871383279562,
+            0.005598289426416159,
+            0.020345570519566536,
+            -0.01645515486598015,
+            -0.0375266894698143,
+            -0.005495910067111254,
+            -0.00010630584438331425,
+            0.02680477872490883,
+            -0.00926067866384983,
+            -0.049700524657964706,
+            -0.033599045127630234,
+            0.0193403922021389,
+            -0.03687518462538719,
+            -0.015487205237150192,
+            -0.004313894081860781,
+            -0.033636271953582764,
+            0.013141786679625511,
+            0.00029579491820186377,
+            -0.01730211265385151,
+            0.049328237771987915,
+            -0.023454179987311363,
+            -0.0019975609611719847,
+            -0.023752011358737946,
+            0.003443669294938445,
+            0.05312558263540268,
+            -0.018502742052078247,
+            -0.018028074875473976,
+            0.022672373801469803,
+            -0.01730211265385151,
+            0.01849343627691269,
+            0.04314824566245079,
+            0.009837726131081581,
+            0.004304586909711361,
+            0.014724013395607471,
+            0.006226526573300362,
+            -0.008483526296913624,
+            -0.0386807844042778,
+            0.021871954202651978,
+            0.012220372445881367,
+            -0.045307520776987076,
+            -0.022597916424274445,
+            -0.02479442022740841,
+            0.0007329431246034801,
+            0.011801548302173615,
+            0.016557535156607628,
+            -0.00968881044536829,
+            -0.028387004509568214,
+            0.006421978119760752,
+            -0.016017716377973557,
+            0.010163478553295135,
+            0.00786459632217884,
+            0.014845007099211216,
+            -0.005519178230315447,
+            0.0067756520584225655,
+            0.004006756003946066,
+            0.012788113206624985,
+            0.0362422950565815,
+            -0.02721429616212845,
+            -0.013216244988143444,
+            -0.020271113142371178,
+            -0.05506148189306259,
+            0.0014589058700948954,
+            0.015059073455631733,
+            0.004544247407466173,
+            -0.02455243282020092,
+            0.005709975957870483,
+            0.012546124868094921,
+            -0.04180800914764404,
+            -0.003641447750851512,
+            0.012099378742277622,
+            0.034678682684898376,
+            0.028238089755177498,
+            -0.011596789583563805,
+            0.003436689032241702,
+            0.0011959770927205682,
+            -0.0032435643952339888,
+            -0.0007631916087120771,
+            -0.023100506514310837,
+            0.027493512257933617,
+            0.014649556018412113,
+            0.038941387087106705,
+            -0.023472795262932777,
+            -0.005998499691486359,
+            -0.048211369663476944,
+            -0.003466937458142638,
+            0.030248448252677917,
+            -0.020680630579590797,
+            -0.011736397631466389,
+            -0.0005956617533229291,
+            0.005044510122388601,
+            -0.014733321033418179,
+            -0.006338213104754686,
+            -0.03194236010313034,
+            -0.026674477383494377,
+            -0.04143571853637695,
+            -0.01702289469540119,
+            -0.00934909749776125,
+            -0.01416558027267456,
+            -0.05748135596513748,
+            -0.011364108882844448,
+            -0.09463575482368469,
+            0.009218796156346798,
+            0.04117511585354805,
+            -0.0257437564432621,
+            0.028238089755177498,
+            -0.028796521946787834,
+            -0.0035576827358454466,
+            -0.03501374274492264,
+            0.012648504227399826,
+            0.040616683661937714,
+            0.033115070313215256,
+            -0.015291753225028515,
+            -0.006966450251638889,
+            0.017516177147626877,
+            -0.030732423067092896,
+            -0.03802927955985069,
+            -0.019228706136345863,
+            -0.00846025813370943,
+            0.00572859076783061,
+            0.0018474820535629988,
+            0.03789897635579109,
+            0.04504691809415817,
+            0.04631270095705986,
+            0.03592584654688835,
+            0.00934909749776125,
+            -0.035367414355278015,
+            -0.010740526020526886,
+            0.005216693971306086,
+            0.03071380779147148,
+            -0.03305922448635101,
+            0.01166194025427103,
+            -0.0311233252286911,
+            0.01853066496551037,
+            0.002405914943665266,
+            -0.023454179987311363,
+            0.021853338927030563,
+            0.005947310011833906,
+            0.0013507094699889421,
+            -0.005919388495385647,
+            0.027325982227921486,
+            -0.01828867755830288,
+            0.035125426948070526,
+            -0.028442848473787308,
+            -0.024105684831738472,
+            -0.028684835880994797,
+            -0.011122121475636959,
+            0.031104711815714836,
+            0.019154246896505356,
+            -0.0005947891622781754,
+            -0.002433836692944169,
+            -0.015273138880729675,
+            0.009172259829938412,
+            0.004853712394833565,
+            -0.02032695710659027,
+            -0.016334161162376404,
+            0.026879236102104187,
+            -0.06943181902170181,
+            -0.009763267822563648,
+            0.008306688629090786,
+            0.0005555243697017431,
+            0.00929790735244751,
+            -0.0386807844042778,
+            0.010377543978393078,
+            0.004590783733874559,
+            -0.03990933671593666,
+            0.022877132520079613,
+            0.00926067866384983,
+            0.01971268095076084,
+            -0.004362756852060556,
+            -0.008837200701236725,
+            0.007022293284535408,
+            0.06451760977506638,
+            -0.010563688352704048,
+            0.0022791041992604733,
+            0.007455078884959221,
+            -0.00812519807368517,
+            0.03845741227269173,
+            -0.025222551077604294,
+            -0.028033331036567688,
+            0.02935495413839817,
+            -0.008455604314804077,
+            -0.007650530431419611,
+            -0.015310367569327354,
+            -0.006696540862321854,
+            0.007827367633581161,
+            0.02870344929397106,
+            -0.02373339794576168,
+            -0.005212040152400732,
+            -0.02676755003631115,
+            0.006524357479065657,
+            0.0022500192280858755,
+            0.002908504568040371,
+            0.0061334543861448765,
+            -0.0009958719601854682,
+            0.039685964584350586,
+            -0.0225420743227005,
+            0.03452976793050766,
+            0.02959694154560566,
+            -0.004090521018952131,
+            -0.0045582083985209465,
+            -0.0005939166294410825,
+            0.013179016299545765,
+            -0.01289049256592989,
+            -0.01808391883969307,
+            0.01736726239323616,
+            -0.018679579719901085,
+            -0.013244166038930416,
+            -0.054949793964624405,
+            -0.002410568529739976,
+            0.002610673662275076,
+            -0.010051791556179523,
+            0.006784959230571985,
+            -0.001237859483808279,
+            0.002903850981965661,
+            0.005393530707806349,
+            0.03949981927871704,
+            -0.03827126696705818,
+            0.023714782670140266,
+            0.00015429616905748844,
+            -0.0026525561697781086,
+            0.008716206066310406,
+            0.0035018394701182842,
+            0.007106058299541473,
+            0.05554545670747757,
+            0.012899799272418022,
+            -0.007994897663593292,
+            -0.005063124932348728,
+            0.005002627614885569,
+            0.016296932473778725,
+            -0.016027024015784264,
+            0.0052073863334953785,
+            0.03668903931975365,
+            0.025297008454799652,
+            0.000632890616543591,
+            -0.0017951290355995297,
+            -0.010182092897593975,
+            0.02751212567090988,
+            -0.021034304052591324,
+            0.029820315539836884,
+            0.04307378828525543,
+            0.009470091201364994,
+            -0.0439300537109375,
+            0.0069571430794894695,
+            -0.015431362204253674,
+            -0.009251371026039124,
+            0.0031528188847005367,
+            0.019600993022322655,
+            -0.04381836578249931,
+            0.003245891071856022,
+            0.01145718153566122,
+            0.008320650085806847,
+            -0.021052919328212738,
+            -0.04675944522023201,
+            -0.008674323558807373,
+            -0.0035274343099445105,
+            0.012974257580935955,
+            -0.016846058890223503,
+            -0.006510396488010883,
+            0.005742551293224096,
+            0.016259703785181046,
+            -0.004988667089492083,
+            0.012034228071570396,
+            0.033394284546375275,
+            -0.016408618539571762,
+            0.013783984817564487,
+            -0.007054868619889021,
+            -0.02189056947827339,
+            0.0110476640984416,
+            0.03095579519867897,
+            -0.01307663694024086,
+            0.03546048700809479,
+            -0.0015345270512625575,
+            0.012415824458003044,
+            0.012964949943125248,
+            0.003639120841398835,
+            -0.014817086048424244,
+            0.002092959824949503,
+            0.018335212022066116,
+            -0.021443821489810944,
+            0.013020792976021767,
+            -0.013541997410356998,
+            -0.037992049008607864,
+            0.010051791556179523,
+            -0.0038950692396610975,
+            0.0016752986703068018,
+            0.009041959419846535,
+            0.0062591019086539745,
+            -0.027195680886507034,
+            -0.01611078903079033,
+            0.023342493921518326,
+            0.024105684831738472,
+            0.06098087131977081,
+            -0.014081815257668495,
+            -0.000535164843313396,
+            -0.007617955096065998,
+            0.009493358433246613,
+            0.04460017383098602,
+            0.024459360167384148,
+            0.0038694743998348713,
+            0.004290625918656588,
+            0.027102608233690262,
+            0.046945590525865555,
+            -0.005230654496699572,
+            -0.010377543978393078,
+            0.006394056603312492,
+            0.003197028301656246,
+            0.023789240047335625,
+            0.021574122831225395,
+            -0.0005409818259067833,
+            0.014072508551180363,
+            0.014444797299802303,
+            -0.016538919880986214,
+            0.012937027961015701,
+            -0.04556812345981598,
+            -0.040169939398765564,
+            0.007115365471690893,
+            0.00435112277045846,
+            0.019359005615115166,
+            0.002450124127790332,
+            -0.005039856769144535,
+            -0.0027851839549839497,
+            0.01750687137246132,
+            -0.010535767301917076,
+            -0.0122948307543993,
+            0.02475719153881073,
+            -0.00873482134193182,
+            0.006747730541974306,
+            0.00030801063985563815,
+            -0.009716732427477837,
+            0.0035181271377950907,
+            -0.0014972981298342347,
+            -0.005077085457742214,
+            0.0023989344481378794,
+            -0.008381146937608719,
+            -0.017516177147626877,
+            0.012229680083692074,
+            -0.04396728053689003,
+            0.013886364176869392,
+            0.001635742955841124,
+            -0.008911658078432083,
+            0.002878256142139435,
+            0.0013146440032869577,
+            0.0114944102242589,
+            -0.018381748348474503,
+            -0.005309765692800283,
+            0.012453053146600723,
+            0.026413874700665474,
+            -0.007403889205306768,
+            0.0038904156535863876,
+            0.015142838470637798,
+            -0.007534190081059933,
+            -0.00987495481967926,
+            0.028070559725165367,
+            -0.007822713814675808,
+            0.006859417073428631,
+            -0.008181042037904263,
+            -0.02540869638323784,
+            0.0053888773545622826,
+            0.03793620690703392,
+            0.04865811765193939,
+            0.007538843899965286,
+            0.01648307777941227,
+            0.03123501129448414,
+            0.014835700392723083,
+            0.00822292361408472,
+            0.015422054566442966,
+            -0.04437679797410965,
+            0.01541274692863226,
+            -0.018037382513284683,
+            0.040542226284742355,
+            0.030844109132885933,
+            -0.05044510215520859,
+            -0.02812640182673931,
+            0.019284548237919807,
+            0.023100506514310837,
+            0.017562713474035263,
+            0.013681605458259583,
+            -0.005314419511705637,
+            0.02053171582520008,
+            -0.023249421268701553,
+            -0.01235998049378395,
+            0.01316040102392435,
+            -0.0032435643952339888,
+            -0.004816483706235886,
+            0.005067778285592794,
+            0.03912752866744995,
+            -0.00436741067096591,
+            -0.016911208629608154,
+            0.02131352201104164,
+            -0.005984539166092873,
+            0.005928695667535067,
+            -0.013123172335326672,
+            0.0009958719601854682,
+            0.03916475921869278,
+            0.005393530707806349,
+            0.03761976212263107,
+            -0.021499665454030037,
+            0.012145915068686008,
+            -0.028182245790958405,
+            -0.014733321033418179,
+            0.029243268072605133,
+            -0.013216244988143444,
+            -0.021201834082603455,
+            0.016976360231637955,
+            0.020513100549578667,
+            0.0032156426459550858,
+            0.011168657802045345,
+            -0.013681605458259583,
+            -0.010600917041301727,
+            0.038159579038619995,
+            -0.0029527137521654367,
+            -0.004144037142395973,
+            0.01712527498602867,
+            -0.01812114752829075,
+            -0.011922542005777359,
+            0.016883287578821182,
+            -0.0019580053631216288,
+            -0.032742779701948166,
+            0.02241177298128605,
+            -0.0004022461944259703,
+            0.03620506450533867,
+            -0.010237935930490494,
+            -0.0056634400971233845,
+            0.010535767301917076,
+            -0.006654658354818821,
+            -0.029987845569849014,
+            -0.034176092594861984,
+            0.007455078884959221,
+            -0.027437668293714523,
+            0.010963899083435535,
+            -0.006650005001574755,
+            0.03255663812160492,
+            -0.030248448252677917,
+            0.008669670671224594,
+            0.03030429035425186,
+            -0.00825549941509962,
+            0.022690989077091217,
+            0.032947540283203125,
+            -0.019079789519309998,
+            -0.03525573015213013,
+            -0.003245891071856022,
+            -0.0003592003195080906,
+            0.029578328132629395,
+            0.002880582818761468,
+            0.0026246344204992056,
+            0.004960745573043823,
+            -0.016222475096583366,
+            -0.03510681167244911,
+            0.022430386394262314,
+            -0.04474908858537674,
+            0.007068829610943794,
+            0.026618633419275284,
+            -0.0430365614593029,
+            -0.036800727248191833,
+            -0.018446899950504303,
+            -0.0263208020478487,
+            -0.02159273810684681,
+            0.001237859483808279,
+            0.02639526128768921,
+            -0.017581328749656677,
+            -0.030602121725678444,
+            0.028089173138141632,
+            0.0034948589745908976,
+            -0.012695040553808212,
+            -0.011056970804929733,
+            -0.010126249864697456,
+            -0.027083994820713997,
+            -0.022262856364250183,
+            -0.014035279862582684,
+            0.0009807477472350001,
+            0.004881633911281824,
+            -0.010368237271904945,
+            -0.0225420743227005,
+            -0.013616454787552357,
+            -0.009032651782035828,
+            -0.004825790878385305,
+            0.050147272646427155,
+            0.021090148016810417,
+            0.023863697424530983,
+            0.0009185057133436203,
+            -0.015719885006546974,
+            0.04601486772298813,
+            -0.0007253810181282461,
+            0.02591128461062908,
+            0.005612250417470932,
+            -0.022058097645640373,
+            -0.005844930652529001,
+            0.03715439885854721,
+            -0.018279369920492172,
+            -0.004395332187414169,
+            -0.014724013395607471,
+            -0.00374150020070374,
+            -0.008609173819422722,
+            0.026693090796470642,
+            0.012024921365082264,
+            -0.004769947379827499,
+            0.013458232395350933,
+            -0.01270434819161892,
+            0.02200225554406643,
+            0.005821662489324808,
+            -0.003676349762827158,
+            -0.0010331007651984692,
+            -0.028684835880994797,
+            -0.02114599198102951,
+            0.001068002893589437,
+            0.022765446454286575,
+            -0.016334161162376404,
+            -0.013197630643844604,
+            -0.015291753225028515,
+            -0.021574122831225395,
+            -0.005198079161345959,
+            0.0004831026017200202,
+            0.005407491698861122,
+            0.015850186347961426,
+            -0.003311041509732604,
+            -0.0430365614593029,
+            -0.005272537004202604,
+            -0.03763837739825249,
+            0.00048455686192028224,
+            0.01961960829794407,
+            -0.011243115179240704,
+            0.01128965150564909,
+            0.001805599662475288,
+            -0.0038648208137601614,
+            -0.01832590624690056,
+            0.007636569440364838,
+            0.01982436701655388,
+            -0.006789613049477339,
+            -0.0022197705693542957,
+            0.018986716866493225,
+            0.0021569470409303904,
+            -0.007645876612514257,
+            0.004849058575928211,
+            -0.001855625887401402,
+            -0.006845456548035145,
+            -0.023379722610116005,
+            -0.029280496761202812,
+            -0.013607148081064224,
+            0.0018090897938236594,
+            -0.001733468729071319,
+            -0.024906106293201447,
+            0.018875030800700188,
+            -0.006082264706492424,
+            0.02336110919713974,
+            -0.011326880194246769,
+            0.01316040102392435,
+            -0.0022697970271110535,
+            -0.0090931486338377,
+            0.018214218318462372,
+            -0.010554381646215916,
+            -0.03264970704913139,
+            -0.01316040102392435,
+            0.028517305850982666,
+            0.00034872969263233244,
+            -0.008097277022898197,
+            0.010042484849691391,
+            0.00922810286283493,
+            -0.027437668293714523,
+            -0.0077017201110720634,
+            0.010619531385600567,
+            0.006626736838370562,
+            -0.032333262264728546,
+            0.026506947353482246,
+            0.03266832232475281,
+            0.01235998049378395,
+            -0.03222157806158066,
+            0.048620887100696564,
+            0.008651056326925755,
+            -0.02475719153881073,
+            0.005677401088178158,
+            0.020624786615371704,
+            0.0363725945353508,
+            -0.0299692302942276,
+            -0.007413196377456188,
+            -0.007422503549605608,
+            0.009009383618831635,
+            -0.03473452478647232,
+            0.015235910192131996,
+            -0.02537146769464016,
+            0.01961960829794407,
+            0.032333262264728546,
+            0.0032831199932843447,
+            0.040579456835985184,
+            -0.009511973708868027,
+            0.017693014815449715,
+            0.0060310750268399715,
+            -0.0076691447757184505,
+            0.006459206808358431,
+            -0.02162996679544449,
+            0.01624108850955963,
+            0.019768523052334785,
+            0.02162996679544449,
+            0.009902875870466232,
+            -0.024682732298970222,
+            0.01941484957933426,
+            0.015133530832827091,
+            0.023826468735933304,
+            0.027605198323726654,
+            -0.014658862724900246,
+            0.026842007413506508,
+            0.005812355317175388,
+            -0.013635069131851196,
+            0.004430233966559172,
+            0.009386326186358929,
+            -0.040616683661937714,
+            -0.0012948662042617798,
+            -0.022523459047079086,
+            0.01459371205419302,
+            -0.0005066614830866456,
+            -0.0028224128764122725,
+            -0.02377062663435936,
+            -0.013653683476150036,
+            -7.831148832337931e-05,
+            0.005146889481693506,
+            -0.004358103033155203,
+            -0.04087728634476662,
+            -0.045754265040159225,
+            0.0234355665743351,
+            -0.009800496511161327,
+            -0.040542226284742355,
+            -0.012648504227399826,
+            -0.009311867877840996,
+            -0.0015752461040392518,
+            -0.014109737239778042,
+            -0.01289049256592989,
+            -0.04385559633374214,
+            -0.014435489661991596,
+            0.01958237960934639,
+            0.030043689534068108,
+            -0.02203948423266411,
+            0.030248448252677917,
+            0.006873378064483404,
+            0.015198681503534317,
+            -0.014630941674113274,
+            0.013560611754655838,
+            0.00614741537719965,
+            -0.04638715833425522,
+            0.02894543670117855,
+            -0.0028898902237415314,
+            0.021276293322443962,
+            0.0384201817214489,
+            -0.005272537004202604,
+            -0.010023870505392551,
+            0.017246268689632416,
+            -0.02213255688548088,
+            -0.013551304116845131,
+            -0.008092623203992844,
+            -0.01586880162358284,
+            -0.037005484104156494,
+            -0.04884426295757294,
+            0.016129402443766594,
+            0.04080282896757126,
+            0.00911176297813654,
+            0.014426182955503464,
+            0.018679579719901085,
+            0.004311567172408104,
+            -0.033282600343227386,
+            0.0225420743227005,
+            0.0047932155430316925,
+            -0.012387902475893497,
+            -0.023826468735933304,
+            0.013653683476150036,
+            0.012778805568814278,
+            -0.006198605056852102,
+            -0.006384748965501785,
+            -0.026655862107872963,
+            -0.024477973580360413,
+            0.0021476398687809706,
+            -0.013886364176869392,
+            0.0033250025007873774,
+            -0.022765446454286575,
+            -0.02857314795255661,
+            0.032128505408763885,
+            0.03546048700809479,
+            -0.01235998049378395,
+            -0.00393229816108942,
+            -0.013579226098954678,
+            -0.008404415100812912,
+            0.008920964784920216,
+            0.005230654496699572,
+            0.017953617498278618,
+            -0.019526535645127296,
+            0.01128965150564909,
+            -0.010442694649100304,
+            0.012099378742277622,
+            -0.0762074738740921,
+            -0.01679021492600441,
+            0.025874055922031403,
+            0.022672373801469803,
+            0.0009662052034400403,
+            0.011438566260039806,
+            0.008893043734133244,
+            0.0006619756459258497,
+            -0.035088200122117996,
+            0.01921009086072445,
+            0.013793292455375195,
+            0.016222475096583366,
+            -0.01151302456855774,
+            0.005235308315604925,
+            0.023845084011554718,
+            -0.0006683743558824062,
+            -0.006319598760455847,
+            -0.01760924980044365,
+            0.011540945619344711,
+            0.020103583112359047,
+            0.0026851315051317215,
+            0.013877056539058685,
+            0.010572995990514755,
+            -0.01961960829794407,
+            -0.013011486269533634,
+            -0.0005930440966039896,
+            -0.005668093450367451,
+            -0.010963899083435535,
+            0.008706899359822273,
+            0.0007399235619232059,
+            0.03763837739825249,
+            0.024887491017580032,
+            0.02172303944826126,
+            0.015980487689375877,
+            0.0069571430794894695,
+            0.007469039876013994,
+            -0.0020917963702231646,
+            0.0018323578406125307,
+            0.01746964268386364,
+            0.03272416815161705,
+            -0.022932976484298706,
+            -0.005156196653842926,
+            0.0042533972300589085,
+            0.024831648916006088,
+            -0.008827893063426018,
+            0.025892671197652817,
+            0.030062302947044373,
+            -0.007650530431419611,
+            -0.029503870755434036,
+            0.0032924271654337645,
+            0.018502742052078247,
+            0.009302561171352863,
+            -0.01634346880018711,
+            -0.0027921644505113363,
+            0.013728141784667969,
+            0.0075434972532093525,
+            0.020922617986798286,
+            -0.004241763148456812,
+            -0.014053894206881523,
+            0.01880057342350483,
+            0.09076395630836487,
+            -0.005691361613571644,
+            -0.0012017941335216165,
+            -0.017106659710407257,
+            -0.01921009086072445,
+            0.0014670497039332986,
+            0.0098935691639781,
+            0.016613377258181572,
+            0.017209040001034737,
+            0.00374615378677845,
+            0.004586129914969206,
+            0.01455648336559534,
+            -0.011531638912856579,
+            0.009479397907853127,
+            -0.02421737276017666,
+            0.01804668828845024,
+            -0.014249345287680626,
+            -0.0067756520584225655,
+            -0.011233807541429996,
+            0.023621710017323494,
+            -0.0019568419083952904,
+            0.007720334455370903,
+            0.019154246896505356,
+            0.003871801309287548,
+            -0.0247199609875679,
+            0.006412670947611332,
+            -0.006454553455114365,
+            0.0007160738459788263,
+            -0.038866929709911346,
+            -0.018363134935498238,
+            0.01127103716135025,
+            -0.0014123697765171528,
+            -0.0011232644319534302,
+            -0.004672221839427948,
+            -0.007804099470376968,
+            -0.00025580296642147005,
+            -0.02455243282020092,
+            0.016836751252412796,
+            0.008855815045535564,
+            0.015403440222144127,
+            0.022393157705664635,
+            -0.0010522969532757998,
+            -0.0009138521272689104,
+            0.025315623730421066,
+            -0.0071293264627456665,
+            -0.004043984692543745,
+            0.027232909575104713,
+            -0.023621710017323494,
+            -0.005188771989196539,
+            0.005300458520650864,
+            -0.01692982390522957,
+            -0.002913158154115081,
+            -0.00414636405184865,
+            -0.021034304052591324,
+            -0.013206937350332737,
+            0.0014914810890331864,
+            0.020717859268188477,
+            -0.015468590892851353,
+            -0.0029178117401897907,
+            0.013048714958131313,
+            -0.005286497995257378,
+            0.007585379760712385,
+            0.0032901002559810877,
+            -0.005067778285592794,
+            -0.02114599198102951,
+            -0.026860620826482773,
+            -0.007999550551176071,
+            0.0011936502996832132,
+            2.35407078434946e-05,
+            -0.0037321930285543203,
+            -0.009372364729642868,
+            -0.004830444231629372,
+            -0.01089874841272831,
+            0.007855288684368134,
+            -0.007413196377456188,
+            -0.014928772114217281,
+            0.013988743536174297,
+            0.005226001143455505,
+            0.008436989970505238,
+            -0.031086096540093422,
+            0.013830521143972874,
+            0.020587557926774025,
+            -0.006961796432733536,
+            0.009237410500645638,
+            -0.015198681503534317,
+            -0.00025216734502464533,
+            -0.01685536466538906,
+            0.004043984692543745,
+            0.004632666241377592,
+            -0.01169916894286871,
+            0.005379570182412863,
+            0.007403889205306768,
+            -0.01913563348352909,
+            -0.006929221097379923,
+            -0.02060617320239544,
+            0.00492817023769021,
+            0.011578175239264965,
+            0.009000076912343502,
+            -0.01559889130294323,
+            -0.004886287730187178,
+            -0.03449253737926483,
+            0.016101481392979622,
+            0.017078738659620285,
+            -0.029001280665397644,
+            -0.014584405347704887,
+            0.009046612307429314,
+            0.013588533736765385,
+            -0.01771163009107113,
+            -0.012834648601710796,
+            -0.0009981987532228231,
+            0.005938002839684486,
+            -0.018744729459285736,
+            0.00533768767490983,
+            -0.0037647683639079332,
+            0.01832590624690056,
+            0.0014740300830453634,
+            0.024850262328982353,
+            -0.026581404730677605,
+            0.010107634589076042,
+            0.017004281282424927,
+            -0.027474896982312202,
+            0.0005968251498416066,
+            0.014658862724900246,
+            0.00042202402255497873,
+            0.00020286819199100137,
+            0.0004857202584389597,
+            -0.007594686932861805,
+            -0.008483526296913624,
+            0.004513998981565237,
+            -0.0014658862492069602,
+            -0.026432489976286888,
+            0.01950792223215103,
+            0.01410043053328991,
+            0.013104557991027832,
+            0.008799971081316471,
+            -0.011652632616460323,
+            0.0006020604632794857,
+            0.01573850028216839,
+            0.032742779701948166,
+            -0.0014891542959958315,
+            -0.015654735267162323,
+            0.010014562867581844,
+            0.026637248694896698,
+            0.0003402950242161751,
+            0.0008190348744392395,
+            -0.01603632979094982,
+            -0.026469718664884567,
+            0.038978613913059235,
+            0.018270062282681465,
+            0.01958237960934639,
+            -0.0018253774615004659,
+            -0.01781400851905346,
+            0.01455648336559534,
+            -0.004437214694917202,
+            0.0027595891151577234,
+            -0.015301060862839222,
+            -0.009418901056051254,
+            0.0026572097558528185,
+            0.026655862107872963,
+            -0.005374916363507509,
+            0.03382241725921631,
+            0.012052842415869236,
+            0.009423554874956608,
+            -0.035609401762485504,
+            0.0031528188847005367,
+            -0.02462689019739628,
+            0.011559560894966125,
+            -0.0094142472371459,
+            -0.0026944386772811413,
+            0.008590559475123882,
+            -0.02935495413839817,
+            -0.009032651782035828,
+            0.01011694222688675,
+            0.023063277825713158,
+            0.007366660516709089,
+            -0.023007433861494064,
+            0.020178040489554405,
+            0.0013134806649759412,
+            0.02151828072965145,
+            -0.012052842415869236,
+            0.012266908772289753,
+            -0.008744128048419952,
+            -0.017804700881242752,
+            -0.005170157644897699,
+            0.03374795988202095,
+            -0.008530061691999435,
+            -0.0036577354185283184,
+            0.006254448089748621,
+            -0.0279216431081295,
+            0.032612480223178864,
+            -0.0037182322703301907,
+            -0.0029154850635677576,
+            0.0041975537315011024,
+            -0.012006307020783424,
+            0.00991218350827694,
+            0.0094142472371459,
+            0.008115891367197037,
+            -0.018707500770688057,
+            -0.023714782670140266,
+            -0.04184523597359657,
+            -0.024533817544579506,
+            0.014733321033418179,
+            0.011503716930747032,
+            -0.015114916488528252,
+            0.03387825936079025,
+            -0.02192779816687107,
+            -0.016306240111589432,
+            -0.011447873897850513,
+            0.028684835880994797,
+            -0.03722885996103287,
+            0.00875343568623066,
+            -0.01350476872175932,
+            -0.020587557926774025,
+            -0.00148566416464746,
+            0.004900248255580664,
+            -0.002478045877069235,
+            -0.009176913648843765,
+            -0.02939218282699585,
+            0.0060636503621935844,
+            0.04999835789203644,
+            -0.0055936360731720924,
+            -0.0016415599966421723,
+            0.00418126629665494,
+            0.003774075536057353,
+            -0.014342417940497398,
+            -0.016501691192388535,
+            0.03486482426524162,
+            -0.03806650638580322,
+            0.0017520831897854805,
+            0.01845620572566986,
+            -0.003583277575671673,
+            0.00554244639351964,
+            0.004523306153714657,
+            0.003578623989596963,
+            0.012629889883100986,
+            -0.00727824168279767,
+            0.013579226098954678,
+            0.011056970804929733,
+            0.011624710634350777,
+            0.005291151348501444,
+            0.014910157769918442,
+            0.025185322389006615,
+            -0.01706012338399887,
+            0.024868877604603767,
+            0.025017792358994484,
+            0.00014222587924450636,
+            0.03581416234374046,
+            -0.019433464854955673,
+            -0.026004357263445854,
+            0.005626211408525705,
+            -0.011745704337954521,
+            -0.004711777437478304,
+            0.001653193961828947,
+            -0.0220953281968832,
+            -0.0031993549782782793,
+            -0.011401337571442127,
+            0.0011895783245563507,
+            0.014053894206881523,
+            -0.014724013395607471,
+            -0.0204944871366024,
+            0.015422054566442966,
+            0.013430310413241386,
+            0.02861037850379944,
+            0.01518937386572361,
+            -0.0034971858840435743,
+            0.02881513722240925,
+            -0.011996999382972717,
+            0.014789164066314697,
+            0.008190348744392395,
+            0.011913234367966652,
+            -0.0069571430794894695,
+            0.024329058825969696,
+            0.0019963975064456463,
+            0.009456129744648933,
+            0.001500788377597928,
+            -0.01692051626741886,
+            -0.0034297085367143154,
+            -0.034213319420814514,
+            -0.018344519659876823,
+            0.003041132353246212,
+            -0.00020417702035047114,
+            0.08748781681060791,
+            0.006980410777032375,
+            -0.02090400457382202,
+            0.003069054102525115,
+            0.0011866698041558266,
+            0.005212040152400732,
+            -0.018279369920492172,
+            0.015180067159235477,
+            -0.027456283569335938,
+            -0.020140811800956726,
+            -0.017832623794674873,
+            -0.024980563670396805,
+            0.0030969756189733744,
+            0.004339488688856363,
+            0.0102472435683012,
+            -0.027474896982312202,
+            0.0019824367482215166,
+            -0.006701194681227207,
+            -0.020717859268188477,
+            0.009027997963130474,
+            -0.01924731954932213,
+            0.004816483706235886,
+            0.011419951915740967,
+            -0.00728754885494709,
+            -0.01791638880968094,
+            0.007603994105011225,
+            0.005314419511705637,
+            0.0033878260292112827,
+            -0.01682744361460209,
+            -5.409818550106138e-05,
+            -0.00768775912001729,
+            0.004092847462743521,
+            0.009595737792551517,
+            -0.013821213506162167,
+            0.015505819581449032,
+            0.006366134621202946,
+            -0.008790664374828339,
+            0.024515202268958092,
+            0.00873482134193182,
+            -0.02241177298128605,
+            -0.0018463187152519822,
+            -0.011689861305058002,
+            0.005658786278218031,
+            -0.0016799522563815117,
+            0.004206860903650522,
+            -0.011354802176356316,
+            -0.016287624835968018,
+            -0.02693507820367813,
+            -0.0008167080814018846,
+            -0.006645351182669401,
+            -0.005374916363507509,
+            0.0036554085090756416,
+            -0.006380095612257719,
+            0.011764319613575935,
+            0.00553313922137022,
+            -0.01396082155406475,
+            0.01045200228691101,
+            -0.0005985702155157924,
+            0.013169708661735058,
+            -0.026879236102104187,
+            0.005314419511705637,
+            -0.026655862107872963,
+            0.009642274118959904,
+            0.01330001000314951,
+            -0.038866929709911346,
+            0.015533741563558578,
+            -0.007101404946297407,
+            0.015570970252156258,
+            -0.0014088796451687813,
+            -0.01828867755830288,
+            0.020550329238176346,
+            0.007887864485383034,
+            -0.008841853588819504,
+            0.012332059442996979,
+            0.025073636323213577,
+            0.00967019610106945,
+            0.03830849379301071,
+            -0.016259703785181046,
+            -0.0003152818826492876,
+            -0.0064312852919101715,
+            0.007036254275590181,
+            -0.005449374206364155,
+            -0.0021301887463778257,
+            -0.020475871860980988,
+            0.029243268072605133,
+            -0.008450950495898724,
+            0.007762216962873936,
+            0.02282129041850567,
+            -0.0051050069741904736,
+            -0.009418901056051254,
+            0.004507018718868494,
+            0.012099378742277622,
+            0.018167681992053986,
+            -0.0032342569902539253,
+            0.015096302144229412,
+            -0.015487205237150192,
+            0.012211065739393234,
+            0.008250845596194267,
+            0.0015240564243867993,
+            -0.027903029695153236,
+            -0.022970205172896385,
+            -0.0017858217470347881,
+            0.0012750884052366018,
+            0.002101103775203228,
+            0.012434438802301884,
+            0.00748300040140748,
+            -0.006822188384830952,
+            0.029652785509824753,
+            -0.005798394791781902,
+            0.012099378742277622,
+            -0.0002384973777225241,
+            0.021481052041053772,
+            0.01862373575568199,
+            0.025017792358994484,
+            -0.0010325191542506218,
+            -0.01009832788258791,
+            0.005226001143455505,
+            -0.006859417073428631,
+            0.01991743966937065,
+            0.010759140364825726,
+            -0.01328139565885067,
+            -0.023752011358737946,
+            0.015049765817821026,
+            0.00011735816224245355,
+            -0.0018963449401780963,
+            0.012695040553808212,
+            -0.02336110919713974,
+            0.0009510809904895723,
+            -0.035404644906520844,
+            0.015077687799930573,
+            -0.04240366816520691,
+            -0.01672506518661976,
+            0.005221347324550152,
+            -0.014258652925491333,
+            -0.01026585791260004,
+            0.017795395106077194,
+            0.017748858779668808,
+            -0.008213616907596588,
+            0.014975308440625668,
+            -0.008697591722011566,
+            -0.009753961116075516,
+            0.013811906799674034,
+            0.012387902475893497,
+            -0.02315635047852993,
+            0.015301060862839222,
+            0.005212040152400732,
+            0.0058914669789373875,
+            -0.021871954202651978,
+            -0.02012219838798046,
+            0.006840802729129791,
+            0.01706012338399887,
+            -0.0003562917991075665,
+            0.0016194552881643176,
+            0.006631390191614628,
+            0.003539068391546607,
+            0.00614741537719965,
+            0.003271485911682248,
+            0.0003123733913525939,
+            -0.000290705036604777,
+            0.02274683304131031,
+            0.011140735819935799,
+            -0.015524433925747871,
+            0.0071293264627456665,
+            -0.009335136041045189,
+            0.0005508707836270332,
+            0.009377018548548222,
+            -1.0870535334106535e-05,
+            0.014649556018412113,
+            -0.008176388218998909,
+            0.013653683476150036,
+            0.02043864317238331,
+            -0.007445771712809801,
+            -0.003076034365221858,
+            -0.008948886767029762,
+            0.006891992408782244,
+            0.016082866117358208,
+            0.025148093700408936,
+            -0.017748858779668808,
+            -0.0002715089067351073,
+            -0.009032651782035828,
+            -0.009437515400350094,
+            -0.026842007413506508,
+            0.017050817608833313,
+            0.020289728417992592,
+            0.0073573533445596695,
+            0.017795395106077194,
+            -0.00849748682230711,
+            0.011326880194246769,
+            -0.010191399604082108,
+            -0.0023244768381118774,
+            -0.00374150020070374,
+            -0.012648504227399826,
+            0.002643248997628689,
+            0.023826468735933304,
+            -0.013914286158978939,
+            0.00790182501077652,
+            -0.0013646703446283937,
+            0.03681934252381325,
+            -0.011233807541429996,
+            0.005612250417470932,
+            0.02019665576517582,
+            -0.004979359917342663,
+            -0.011773626320064068,
+            0.02295159175992012,
+            0.008841853588819504,
+            0.0035530291497707367,
+            0.005635518580675125,
+            0.024031227454543114,
+            0.011224500834941864,
+            0.0016043310752138495,
+            0.000546508003026247,
+            -0.04716896265745163,
+            0.00888838991522789,
+            -0.001733468729071319,
+            -0.010638146661221981,
+            0.017525484785437584,
+            0.010619531385600567,
+            0.009395632892847061,
+            -0.003269159235060215,
+            0.019191475585103035,
+            -0.012266908772289753,
+            0.012508896179497242,
+            0.033636271953582764,
+            0.0034157477784901857,
+            -0.019014639779925346,
+            0.0021848685573786497,
+            0.018009459599852562,
+            -0.0024222026113420725,
+            0.00217090779915452,
+            0.006705848034471273,
+            -0.017190424725413322,
+            0.0004281318688299507,
+            0.0036903105210512877,
+            0.029224654659628868,
+            -0.015180067159235477,
+            -0.027828572317957878,
+            0.0004447103419806808,
+            -0.005872852168977261,
+            -0.003073707688599825,
+            0.008841853588819504,
+            0.0032924271654337645,
+            -0.0016217821976169944,
+            -0.012722962535917759,
+            0.0017323052743449807,
+            0.007743602618575096,
+            0.010787061415612698,
+            0.003583277575671673,
+            -0.005965924356132746,
+            -0.008916311897337437,
+            0.021462436765432358,
+            -0.006045036017894745,
+            -0.0038834353908896446,
+            -0.0035925847478210926,
+            -0.006980410777032375,
+            -0.010172785259783268,
+            -0.012778805568814278,
+            0.011122121475636959,
+            -0.001270434819161892,
+            -0.01437033899128437,
+            -0.017004281282424927,
+            0.010461308993399143,
+            -0.0067756520584225655,
+            0.025725141167640686,
+            0.0001714563404675573,
+            -0.008032126352190971,
+            0.003369211684912443,
+            -0.015319675207138062,
+            -0.010535767301917076,
+            0.007752909790724516,
+            -0.002316333120688796,
+            -0.00495609175413847,
+            0.01091736275702715,
+            -0.0032575251534581184,
+            0.018958795815706253,
+            -0.0012913760729134083,
+            -0.008385800756514072,
+            0.002873602556064725,
+            0.0025571573060005903,
+            -0.012806727550923824,
+            -0.0006579037290066481,
+            -0.02131352201104164,
+            -0.013476846739649773,
+            -0.0014519254909828305,
+            0.014091122895479202,
+            -0.010777754709124565,
+            0.005868198815733194,
+            0.0010674211662262678,
+            0.006896646227687597,
+            -0.016371389850974083,
+            -0.010237935930490494,
+            -0.0012890491634607315,
+            0.024198757484555244,
+            0.011001127772033215,
+            -0.02319357916712761,
+            -0.010824290104210377,
+            -0.01007971353828907,
+            0.018055995926260948,
+            -0.013635069131851196,
+            0.004490730818361044,
+            0.013234859332442284,
+            -0.016194554045796394,
+            -0.0009208325645886362,
+            -0.012601968832314014,
+            -0.02220701426267624,
+            0.0003839225973933935,
+            0.032705552875995636,
+            0.012015613727271557,
+            0.017953617498278618,
+            -0.008078662678599358,
+            -0.00011510407057357952,
+            0.03168175742030144,
+            0.0032505446579307318,
+            0.0018253774615004659,
+            -0.01437033899128437,
+            0.009381672367453575,
+            -0.009493358433246613,
+            0.018875030800700188,
+            0.006612775847315788,
+            -0.00827411375939846,
+            0.003231930313631892,
+            -0.02233731560409069,
+            -0.003841552883386612,
+            -0.005244615487754345,
+            -0.009698117151856422,
+            0.012825341895222664,
+            -0.007040907628834248,
+            -0.015543048270046711,
+            0.0005316746537573636,
+            -0.007124672643840313,
+            -0.015980487689375877,
+            -0.008758088573813438,
+            -0.011373416520655155,
+            -0.030062302947044373,
+            -0.027567969635128975,
+            0.0010226302547380328,
+            -0.005868198815733194,
+            -0.028461461886763573,
+            -0.019359005615115166,
+            -0.022393157705664635,
+            0.009097802452743053,
+            -0.012369288131594658,
+            -0.02101569063961506,
+            0.007157247979193926,
+            0.005095699802041054,
+            -0.01842828467488289,
+            -0.026469718664884567,
+            -0.014435489661991596,
+            -0.016334161162376404,
+            0.025278395041823387,
+            0.012713654898107052,
+            -0.0008283421047963202,
+            -0.005174811463803053,
+            0.012220372445881367,
+            0.0032784664072096348,
+            0.01127103716135025,
+            -0.03268693760037422,
+            -0.03129085525870323,
+            -0.011187272146344185,
+            -0.00430691335350275,
+            0.0007451588753610849,
+            -0.03300338238477707,
+            0.002910831244662404,
+            0.003069054102525115,
+            -0.03827126696705818,
+            -0.008706899359822273,
+            0.015040459111332893,
+            -0.0009749307064339519,
+            0.011261729523539543,
+            0.00635217409580946,
+            0.026618633419275284,
+            0.02332388050854206,
+            0.0014437816571444273,
+            4.0937200537882745e-05,
+            0.012211065739393234,
+            -0.014705399051308632,
+            0.0014961347915232182,
+            0.005775126628577709,
+            0.008585905656218529,
+            -0.0004086449043825269,
+            0.015031151473522186,
+            -0.001642723334953189,
+            -0.015403440222144127,
+            0.017311418429017067,
+            0.01541274692863226,
+            0.00934909749776125,
+            0.002934099407866597,
+            0.0009121070033870637,
+            0.0038019970525056124,
+            0.0037880362942814827,
+            0.006770998705178499,
+            -0.01227621641010046,
+            -0.006417324300855398,
+            0.016901900991797447,
+            -0.011829469352960587,
+            -0.013942207209765911,
+            0.019805751740932465,
+            -0.00024489607312716544,
+            -0.023826468735933304,
+            0.02758658491075039,
+            0.006328905932605267,
+            -0.004479097202420235,
+            -0.02537146769464016,
+            -0.0018323578406125307,
+            -0.024664118885993958,
+            0.015543048270046711,
+            0.005407491698861122,
+            0.0015449975617229939,
+            0.022597916424274445,
+            0.01108489278703928,
+            0.0038206116296350956,
+            -0.013206937350332737,
+            0.006231179926544428,
+            0.01106627844274044,
+            0.015896722674369812,
+            -0.03704271465539932,
+            -0.02455243282020092,
+            0.012285523116588593,
+            -0.01841897703707218,
+            -0.005244615487754345,
+            0.012788113206624985,
+            0.0009766757721081376,
+            -0.02019665576517582,
+            0.0031667796429246664,
+            0.003608872415497899,
+            0.004758313298225403,
+            0.012192451395094395,
+            0.016631992533802986,
+            0.0122948307543993,
+            -0.0018521357560530305,
+            -0.022970205172896385,
+            0.006007806863635778,
+            -0.008241538889706135,
+            0.0007137469947338104,
+            -0.023342493921518326,
+            -0.0018102532485499978,
+            0.010610224679112434,
+            0.00767845194786787,
+            0.025874055922031403,
+            0.030676579102873802,
+            0.027903029695153236,
+            0.013244166038930416,
+            -0.004942130763083696,
+            -0.014416875317692757,
+            -0.020959846675395966,
+            0.0019917439203709364,
+            0.002347744768485427,
+            -0.011568867601454258,
+            -0.001557794981636107,
+            0.017488256096839905,
+            0.004011409357190132,
+            0.021816110238432884,
+            -0.005793740972876549,
+            0.017460335046052933,
+            -3.364776057424024e-05,
+            -0.005212040152400732,
+            -0.0008463748381473124,
+            -0.003173760138452053,
+            0.013244166038930416,
+            0.006650005001574755,
+            0.026004357263445854,
+            0.00022031922708265483,
+            0.003708925098180771,
+            0.005481949541717768,
+            -0.0007905315142124891,
+            -0.0008289237739518285,
+            -0.007380621042102575,
+            -0.003539068391546607,
+            -0.009847032837569714,
+            0.034678682684898376,
+            0.025278395041823387,
+            -0.02250484563410282,
+            -0.015124224126338959,
+            -0.012713654898107052,
+            0.02639526128768921,
+            -0.010833597742021084,
+            0.006524357479065657,
+            0.010796369053423405,
+            0.008660363033413887,
+            0.01065676100552082,
+            0.000908035144675523,
+            0.005719283130019903,
+            0.0028898902237415314,
+            0.022672373801469803,
+            -0.012611275538802147,
+            0.008669670671224594,
+            0.003920664079487324,
+            0.008357878774404526,
+            -0.015077687799930573,
+            -0.001618291949853301,
+            0.005872852168977261,
+            -0.014919465407729149,
+            0.006170683074742556,
+            -0.004118442535400391,
+            0.006207912229001522,
+            0.011559560894966125,
+            0.0015775728970766068,
+            -0.01607356034219265,
+            -0.017935002222657204,
+            0.012927721254527569,
+            -0.009781882166862488,
+            0.00916295312345028,
+            -0.018028074875473976,
+            -0.011606096290051937,
+            0.017897773534059525,
+            -0.006119493395090103,
+            -0.011931848712265491,
+            0.0026292880065739155,
+            0.00411146180704236,
+            -5.733389480155893e-05,
+            0.004009082447737455,
+            0.007752909790724516,
+            0.0020592212677001953,
+            0.00535630201920867,
+            0.005947310011833906,
+            -0.002478045877069235,
+            0.002440816955640912,
+            -0.0019789463840425014,
+            0.035739704966545105,
+            -0.009847032837569714,
+            0.006617429666221142,
+            -0.0022372216917574406,
+            -0.0012099378509446979,
+            -0.04154740646481514,
+            0.027772728353738785,
+            -0.006263755261898041,
+            0.015319675207138062,
+            0.00801816489547491,
+            -0.003401787020266056,
+            -0.010768447071313858,
+            0.008353224955499172,
+            0.01050784531980753,
+            -0.02135075069963932,
+            0.015142838470637798,
+            -0.024198757484555244,
+            -0.004218494985252619,
+            -0.002433836692944169,
+            -0.01108489278703928,
+            0.015496511943638325,
+            -0.01459371205419302,
+            0.008092623203992844,
+            -0.01091736275702715,
+            -0.0073713138699531555,
+            0.010144864208996296,
+            -0.0072549739852547646,
+            0.011010434478521347,
+            -0.042924873530864716,
+            0.0067756520584225655,
+            -0.004821137059479952,
+            -0.0010517152259126306,
+            -0.0006689560250379145,
+            -0.02270960435271263,
+            -0.031067483127117157,
+            -0.000392648100387305,
+            0.004853712394833565,
+            -0.005384223535656929,
+            0.00748300040140748,
+            0.005407491698861122,
+            0.004355776589363813,
+            0.0032621787395328283,
+            -0.01954515092074871,
+            0.025390081107616425,
+            0.03665181249380112,
+            -0.011829469352960587,
+            0.013383775018155575,
+            0.007161901798099279,
+            0.008148466236889362,
+            -0.03622367978096008,
+            -0.02352863922715187,
+            -0.008450950495898724,
+            0.009181567467749119,
+            0.02591128461062908,
+            -0.004679202102124691,
+            0.01559889130294323,
+            0.009437515400350094,
+            0.0050910464487969875,
+            -0.02870344929397106,
+            -0.015403440222144127,
+            -0.008046086877584457,
+            -0.015654735267162323,
+            -0.0022442021872848272,
+            -0.0033017343375831842,
+            0.014696091413497925,
+            -0.00473271869122982,
+            -0.018260754644870758,
+            -0.011941156350076199,
+            -0.006864070892333984,
+            -0.0037368466146290302,
+            -0.010154170915484428,
+            0.023547252640128136,
+            -0.010833597742021084,
+            -0.015589584596455097,
+            0.02512947842478752,
+            -0.001938227447681129,
+            0.005793740972876549,
+            0.005235308315604925,
+            0.00750161474570632,
+            -0.003304061247035861,
+            -0.004590783733874559,
+            -0.007673798594623804,
+            0.017627865076065063,
+            0.004202207550406456,
+            0.02874067798256874,
+            -0.004532613325864077,
+            0.00770637346431613,
+            0.0016764620086178184,
+            0.004355776589363813,
+            -0.002505967626348138,
+            0.008599866181612015,
+            0.006003153510391712,
+            -0.003648428013548255,
+            0.0071991304866969585,
+            -0.01350476872175932,
+            0.002410568529739976,
+            0.024943334981799126,
+            0.005984539166092873,
+            0.010414772666990757,
+            -0.012629889883100986,
+            -0.012667118571698666,
+            -0.003418074455112219,
+            -0.016203859820961952,
+            0.015924643725156784,
+            -0.020662017166614532,
+            0.018670272082090378,
+            -0.017460335046052933,
+            -0.001818397082388401,
+            0.0043930052779614925,
+            0.015198681503534317,
+            0.021090148016810417,
+            -0.004346469417214394,
+            0.026469718664884567,
+            0.013728141784667969,
+            0.0035227807238698006,
+            0.004970052745193243,
+            -0.022653760388493538,
+            -0.005514524411410093,
+            -0.011094199493527412,
+            0.003913683816790581,
+            -0.008581251837313175,
+            -0.016734370961785316,
+            -0.0010197217343375087,
+            0.007315470837056637,
+            -0.005398184526711702,
+            0.003701944602653384,
+            0.012657811865210533,
+            0.0015484878094866872,
+            0.0028293931391090155,
+            0.014230730943381786,
+            0.01651099883019924,
+            0.007510922383517027,
+            -0.007622608914971352,
+            0.0040323506109416485,
+            0.006622083019465208,
+            -0.0046280124224722385,
+            0.0009504992631264031,
+            0.017395183444023132,
+            0.01438895333558321,
+            -0.004060272127389908,
+            0.0053656091913580894,
+            -0.0027805303689092398,
+            -0.02676755003631115,
+            -0.014267959631979465,
+            -0.0003990468103438616,
+            -0.0005968251498416066,
+            -0.003506493056192994,
+            -0.015161452814936638,
+            0.004483750555664301,
+            0.012155221775174141,
+            -0.0122948307543993,
+            0.0035181271377950907,
+            0.008399761281907558,
+            -0.009949412196874619,
+            0.0021976660937070847,
+            -0.0034785715397447348,
+            -0.014249345287680626,
+            -0.009791189804673195,
+            0.027623813599348068,
+            -0.01011694222688675,
+            0.013746756128966808,
+            0.013635069131851196,
+            0.00039875597576610744,
+            -0.006440592464059591,
+            -0.0013041734928265214,
+            0.019061176106333733,
+            0.01746964268386364,
+            -0.001688096090219915,
+            -0.00036327220732346177,
+            -0.007552804425358772,
+            -0.006161375902593136,
+            0.01350476872175932,
+            -0.01634346880018711,
+            -0.004311567172408104,
+            0.009577123448252678,
+            -0.00846025813370943,
+            -0.004593110177665949,
+            -0.024533817544579506,
+            0.0021522934548556805,
+            0.009135031141340733,
+            0.0035227807238698006,
+            -0.0068314955569803715,
+            -0.001905652228742838,
+            -0.002934099407866597,
+            0.0058030481450259686,
+            -0.00118550646584481,
+            -0.00687803141772747,
+            -0.016836751252412796,
+            -0.0024175490252673626,
+            0.0070920973084867,
+            -0.006184644065797329,
+            -0.001833521295338869,
+            0.0036251600831747055,
+            -0.006082264706492424,
+            -0.015673348680138588,
+            0.03244495019316673,
+            9.452639642404392e-05,
+            0.021332135424017906,
+            0.004132403060793877,
+            0.0061334543861448765,
+            0.0027246871031820774,
+            0.007222398649901152,
+            0.02561345510184765,
+            0.009614353068172932,
+            0.006747730541974306,
+            0.01621316745877266,
+            0.014975308440625668,
+            -0.014240038581192493,
+            0.0072549739852547646,
+            0.012853262946009636,
+            -0.010163478553295135,
+            0.00417195912450552,
+            0.009251371026039124,
+            -0.0007649366743862629,
+            -0.003988141659647226,
+            -0.0037228858564049006,
+            0.016799522563815117,
+            0.003071380779147148,
+            -0.0008132178918458521,
+            0.012183143757283688,
+            0.01166194025427103,
+            -0.018754037097096443,
+            -0.00033796823117882013,
+            -0.0015135857975110412,
+            0.002343091182410717,
+            -0.005882159806787968,
+            -0.026581404730677605,
+            -0.008320650085806847,
+            0.0033854993525892496,
+            -0.0030434592626988888,
+            0.03091856651008129,
+            -0.00950266607105732,
+            -0.03354320302605629,
+            -0.01563611999154091,
+            -0.001225062063895166,
+            -0.01692982390522957,
+            -0.007966975681483746,
+            0.01276949793100357,
+            0.0295224841684103,
+            0.012415824458003044,
+            -0.013802599161863327,
+            0.024664118885993958,
+            -0.0028131057042628527,
+            0.01750687137246132,
+            -0.00870224554091692,
+            -0.009083840996026993,
+            -0.004509345628321171,
+            0.022113941609859467,
+            -0.008744128048419952,
+            -0.009605045430362225,
+            0.013579226098954678,
+            0.016985666006803513,
+            0.00842768233269453,
+            0.02323080785572529,
+            0.013644376769661903,
+            -0.007022293284535408,
+            -0.014891543425619602,
+            0.003808977548032999,
+            -0.008595212362706661,
+            -0.020252499729394913,
+            0.007324778009206057,
+            -0.014900850132107735,
+            0.005416798871010542,
+            -0.0021348423324525356,
+            -0.023696167394518852,
+            0.005537792574614286,
+            0.0053656091913580894,
+            -0.012043535709381104,
+            -0.012546124868094921,
+            0.000147243044921197,
+            -0.009507319889962673,
+            0.014463411644101143,
+            0.0068314955569803715,
+            -0.00023253493418451399,
+            -0.009251371026039124,
+            0.017832623794674873,
+            0.011001127772033215,
+            0.009968026541173458,
+            -0.00792043935507536,
+            0.0033924796152859926,
+            0.008934926241636276,
+            0.007590033579617739,
+            -0.0014495986979454756,
+            -0.0056634400971233845,
+            0.037880364805459976,
+            0.0030620736069977283,
+            0.012527510523796082,
+            -0.0017451026942580938,
+            0.009865647181868553,
+            0.01086151972413063,
+            0.015291753225028515,
+            0.010256550274789333,
+            -0.003646101336926222,
+            -0.0038275918923318386,
+            0.013458232395350933,
+            -0.012024921365082264,
+            0.0055052172392606735,
+            0.010275164619088173,
+            -0.0046675680205225945,
+            -0.005384223535656929,
+            0.010647453367710114,
+            0.00014309842663351446,
+            0.01812114752829075,
+            -0.005900774151086807,
+            0.009288599714636803,
+            0.018335212022066116,
+            0.009577123448252678,
+            -0.019098404794931412,
+            0.013728141784667969,
+            -0.01556166261434555,
+            0.029950616881251335,
+            0.01917286217212677,
+            -0.01812114752829075,
+            -0.0018975083949044347,
+            -0.011019742116332054,
+            0.009763267822563648,
+            -0.04221752658486366,
+            -0.01190392766147852,
+            0.0023919541854411364,
+            -0.004090521018952131,
+            -0.018884338438510895,
+            -0.027028150856494904,
+            0.0044558290392160416,
+            -0.004430233966559172,
+            -0.009353750385344028,
+            -0.008278767578303814,
+            0.0006509232916869223,
+            0.015924643725156784,
+            -0.005221347324550152,
+            -0.0077715241350233555,
+            -0.024310443550348282,
+            0.014993922784924507,
+            0.00886977557092905,
+            -0.004176612477749586,
+            -0.024403516203165054,
+            0.01455648336559534,
+            0.01603632979094982,
+            -0.00687803141772747,
+            0.01190392766147852,
+            -0.024049842730164528,
+            0.00197545625269413,
+            -0.005700668785721064,
+            -0.01654822751879692,
+            -0.015263832174241543,
+            0.0010999963851645589,
+            -0.012164529412984848,
+            -0.003313368419185281,
+            -0.014119044877588749,
+            -0.009083840996026993,
+            -0.0028945438098162413,
+            0.020475871860980988,
+            0.0025641375686973333,
+            -0.01541274692863226,
+            0.016390005126595497,
+            0.007869250141084194,
+            -0.0229888204485178,
+            -0.01631554774940014,
+            0.0046605877578258514,
+            -0.029876159504055977,
+            0.010805675759911537,
+            0.007869250141084194,
+            0.006826841738075018,
+            -0.02189056947827339,
+            0.007683105766773224,
+            0.02241177298128605,
+            0.007994897663593292,
+            0.0017660439480096102,
+            0.01913563348352909,
+            0.008185694925487041,
+            0.012667118571698666,
+            0.0019696392118930817,
+            -3.335691189931822e-06,
+            -0.005849584471434355,
+            -0.004148690961301327,
+            -0.004714104346930981,
+            0.03802927955985069,
+            -0.01617593877017498,
+            0.009498012252151966,
+            -0.022560687735676765,
+            0.0002395153569523245,
+            -0.01921009086072445,
+            -0.011298958212137222,
+            -0.015980487689375877,
+            0.0008010021410882473,
+            0.02151828072965145,
+            -0.007892518304288387,
+            -0.0002912867348641157,
+            -0.01583157107234001,
+            -0.0058356234803795815,
+            -0.006542971823364496,
+            0.02036418579518795,
+            -0.012741576880216599,
+            0.0018626062665134668,
+            -0.017311418429017067,
+            0.0010249570477753878,
+            0.026693090796470642,
+            0.005556406918913126,
+            -0.013225551694631577,
+            -0.026618633419275284,
+            -0.016194554045796394,
+            -0.009158299304544926,
+            -0.01316040102392435,
+            -0.005239961668848991,
+            -0.019228706136345863,
+            -0.011364108882844448,
+            -0.006473167799413204,
+            0.00654762564226985,
+            0.004039331339299679,
+            -0.015012537129223347,
+            -0.004997974261641502,
+            -0.024664118885993958,
+            -0.0202338844537735,
+            -0.010014562867581844,
+            -0.009470091201364994,
+            0.0013844481436535716,
+            0.02853591926395893,
+            0.0028549879789352417,
+            0.030397363007068634,
+            -0.00415567122399807,
+            0.008790664374828339,
+            0.009828418493270874,
+            0.007203784305602312,
+            -0.01520798821002245,
+            0.007487654220312834,
+            0.014584405347704887,
+            -0.008530061691999435,
+            0.0057890876196324825,
+            -0.0122948307543993,
+            -0.0007463222718797624,
+            0.005249268840998411,
+            0.004970052745193243,
+            0.011261729523539543,
+            -0.007548151072114706,
+            -0.01580365002155304,
+            0.013495461083948612,
+            -0.02040141448378563,
+            0.0011971404310315847,
+            0.00031295508961193264,
+            -0.003236583899706602,
+            0.003632140578702092,
+            0.018102532252669334,
+            -0.01336515974253416,
+            0.001435637823306024,
+            0.01999189704656601,
+            -0.012536818161606789,
+            0.01316040102392435,
+            0.003013210603967309,
+            0.0010214668000116944,
+            0.009511973708868027,
+            0.0011215193662792444,
+            0.005156196653842926,
+            0.0005630864761769772,
+            -0.019228706136345863,
+            -0.006966450251638889,
+            0.00831134244799614,
+            -0.024608274921774864,
+            -0.01128965150564909,
+            0.011177964508533478,
+            0.006254448089748621,
+            -0.017832623794674873,
+            -0.012788113206624985,
+            -0.023863697424530983,
+            0.01166194025427103,
+            -0.005467988550662994,
+            0.023993998765945435,
+            -0.015105608850717545,
+            0.009204835630953312,
+            -0.007738948799669743,
+            0.021164605394005775,
+            -0.0122948307543993,
+            0.005212040152400732,
+            0.0013576899655163288,
+            -0.01665060594677925,
+            0.009567816741764545,
+            -0.012285523116588593,
+            -0.0017567367758601904,
+            0.042775958776474,
+            0.006952489260584116,
+            0.0074271573685109615,
+            0.015068380162119865,
+            -0.022020868957042694,
+            0.01852135732769966,
+            0.002450124127790332,
+            0.018502742052078247,
+            0.010572995990514755,
+            -0.019265934824943542,
+            0.01695774495601654,
+            0.026823392137885094,
+            0.01982436701655388,
+            0.006221872754395008,
+            -0.01147579587996006,
+            0.027903029695153236,
+            0.012862570583820343,
+            0.0006660475628450513,
+            0.0022569994907826185,
+            0.0016846058424562216,
+            -0.011038356460630894,
+            -0.010842905379831791,
+            0.009018691256642342,
+            -0.017730243504047394,
+            -0.013272088021039963,
+            -0.0030923220328986645,
+            0.003080687951296568,
+            -0.010219321586191654,
+            -0.0036996176932007074,
+            0.006719809025526047,
+            0.007399235386401415,
+            0.011419951915740967,
+            0.0016020042821764946,
+            0.01611078903079033,
+            -0.004774601198732853,
+            -0.01893087476491928,
+            0.0002504222502466291,
+            -0.0032598518300801516,
+            0.014947386458516121,
+            -0.029299112036824226,
+            -0.007911132648587227,
+            -0.0004516907501965761,
+            0.005267883650958538,
+            0.006035728845745325,
+            0.007832021452486515,
+            -0.015096302144229412,
+            -0.005682054441422224,
+            -0.003113263286650181,
+            0.016120094805955887,
+            0.0016811155946925282,
+            0.023212192580103874,
+            0.015608198940753937,
+            -0.009968026541173458,
+            -0.006328905932605267,
+            0.0032668323256075382,
+            -0.010982513427734375,
+            0.00017320144979748875,
+            0.0025943859945982695,
+            0.011122121475636959,
+            -0.018847109749913216,
+            -0.008208963088691235,
+            0.025594839826226234,
+            0.02015942707657814,
+            0.00357629731297493,
+            0.026748934760689735,
+            -0.014174887910485268,
+            0.008227577432990074,
+            0.019842982292175293,
+            0.014407568611204624,
+            -0.010572995990514755,
+            -0.010731218382716179,
+            -0.0046675680205225945,
+            0.001287885825149715,
+            -0.0027549355290830135,
+            -0.005179464817047119,
+            0.006161375902593136,
+            -0.022467615082859993,
+            -0.016259703785181046,
+            0.0012157548917457461,
+            -0.013793292455375195,
+            0.0032342569902539253,
+            0.009358404204249382,
+            0.018502742052078247,
+            -0.011736397631466389,
+            0.01787916012108326,
+            -0.026879236102104187,
+            -0.003480898216366768,
+            0.010256550274789333,
+            0.0019545149989426136,
+            -0.016808830201625824,
+            -0.003911356907337904,
+            -0.002536216052249074,
+            -0.009218796156346798,
+            -0.006989717949181795,
+            0.01671575754880905,
+            0.012397210113704205,
+            0.015059073455631733,
+            -0.008195002563297749,
+            0.004423253703862429,
+            0.0023547252640128136,
+            0.009395632892847061,
+            -0.0018847109749913216,
+            -0.009442169219255447,
+            0.006696540862321854,
+            0.0017509197350591421,
+            -0.005863544996827841,
+            -0.005281844176352024,
+            0.023063277825713158,
+            -0.026134658604860306,
+            0.003641447750851512,
+            -0.016501691192388535,
+            0.011410645209252834,
+            0.012257601134479046,
+            0.002092959824949503,
+            -0.005598289426416159,
+            -0.006994371768087149,
+            0.013458232395350933,
+            -0.06496435403823853,
+            -0.008432336151599884,
+            -0.0015938604483380914,
+            0.027623813599348068,
+            -0.01794430986046791,
+            0.003169106552377343,
+            0.008404415100812912,
+            0.015422054566442966,
+            -0.007189823314547539,
+            0.007101404946297407,
+            0.016324853524565697,
+            -0.02261653169989586,
+            -0.011326880194246769,
+            0.00187075010035187,
+            -0.004974706098437309,
+            -0.011838776990771294,
+            0.0010470616398379207,
+            0.004500038456171751,
+            0.018940182402729988,
+            0.0015496512642130256,
+            0.011401337571442127,
+            -0.023510023951530457,
+            -0.013393081724643707,
+            -0.00436042994260788,
+            0.009060573764145374,
+            -0.005910081323236227,
+            0.0040323506109416485,
+            0.00931652169674635,
+            -0.03136531263589859,
+            0.011224500834941864,
+            0.008450950495898724,
+            -0.007241012994199991,
+            0.009605045430362225,
+            0.011326880194246769,
+            0.008474218659102917,
+            0.005267883650958538,
+            0.0011447874130681157,
+            -0.03210989013314247,
+            0.0043162209913134575,
+            0.013337238691747189,
+            -0.009200181812047958,
+            -0.013206937350332737,
+            -0.009977334178984165,
+            -0.0012366961454972625,
+            0.019973281770944595,
+            -0.004530286882072687,
+            -0.0018742403481155634,
+            0.014640248380601406,
+            0.006021767854690552,
+            0.01455648336559534,
+            -0.013337238691747189,
+            0.004942130763083696,
+            -0.010666067712008953,
+            -0.00854867696762085,
+            -0.0014100429834797978,
+            -0.0001836720621213317,
+            -0.0028968704864382744,
+            -0.038159579038619995,
+            0.00965158175677061,
+            -0.0036693692672997713,
+            -0.0197871383279562,
+            -0.01248097512871027,
+            -0.013411696068942547,
+            -0.004532613325864077,
+            -0.010870826430618763,
+            0.006854763720184565,
+            0.0031877208966761827,
+            0.009135031141340733,
+            -0.01336515974253416,
+            0.02496195025742054,
+            -0.014044586569070816,
+            -0.009977334178984165,
+            -0.009107109159231186,
+            -0.0005622139433398843,
+            0.002806125208735466,
+            0.0149846151471138,
+            -0.0038624941371381283,
+            -0.003204008564352989,
+            0.021853338927030563,
+            0.008013512007892132,
+            0.006114840041846037,
+            -0.01188531331717968,
+            -0.018875030800700188,
+            -0.023137735202908516,
+            -0.0029015240725129843,
+            -0.021946411579847336,
+            0.0007562111713923514,
+            0.004853712394833565,
+            0.02377062663435936,
+            0.007720334455370903,
+            -9.13997573661618e-05,
+            0.00436042994260788,
+            0.0038206116296350956,
+            0.019452078267931938,
+            -0.008371839299798012,
+            -0.004046311601996422,
+            -0.0071432869881391525,
+            -0.002633941825479269,
+            0.03482759743928909,
+            -0.008423029445111752,
+            0.00037955984589643776,
+            0.0013239512918516994,
+            -0.008879082277417183,
+            -0.007590033579617739,
+            0.010703296400606632,
+            -0.02881513722240925,
+            -0.014919465407729149,
+            -0.00036181797622703016,
+            0.0024873530492186546,
+            -0.010749832727015018,
+            0.02073647454380989,
+            -0.008502140641212463,
+            0.0023023721296340227,
+            0.009256024844944477,
+            0.024161528795957565,
+            0.022430386394262314,
+            0.023137735202908516,
+            0.007315470837056637,
+            -0.003671696176752448,
+            -0.002410568529739976,
+            -0.0067616915330290794,
+            -0.02496195025742054,
+            0.0161387100815773,
+            -0.0040951743721961975,
+            0.03404578939080238,
+            -0.03428777679800987,
+            0.0012413497315719724,
+            -0.007380621042102575,
+            -0.023472795262932777,
+            -0.009986640885472298,
+            0.023174963891506195,
+            -0.01045200228691101,
+            0.023900926113128662,
+            -0.0003123733913525939,
+            0.01556166261434555,
+            0.007906478829681873,
+            0.008441643789410591,
+            -0.01930316351354122,
+            0.011810855008661747,
+            -0.0023372743744403124,
+            0.01822352595627308,
+            0.009307214990258217,
+            -0.004607071168720722,
+            0.008832546882331371,
+            0.011689861305058002,
+            -0.01518937386572361,
+            -0.011196578852832317,
+            -0.0029015240725129843,
+            0.0018474820535629988,
+            -0.009847032837569714,
+            0.008553329855203629,
+            -0.002062711399048567,
+            -0.01600840874016285,
+            0.02537146769464016,
+            -0.01593395136296749,
+            0.010368237271904945,
+            0.004904902074486017,
+            0.00647782115265727,
+            0.01736726239323616,
+            -0.001187833258882165,
+            -0.027400439605116844,
+            0.0002552212681621313,
+            -0.03222157806158066,
+            -0.012341366149485111,
+            0.001407716190442443,
+            0.006566239986568689,
+            -0.0030434592626988888,
+            -0.010666067712008953,
+            -0.01883780211210251,
+            -0.0060310750268399715,
+            0.024571046233177185,
+            0.03439946472644806,
+            0.007743602618575096,
+            0.011745704337954521,
+            -0.011122121475636959,
+            -0.00967019610106945,
+            -0.006207912229001522,
+            -0.01917286217212677,
+            -0.009004729799926281,
+            -0.005863544996827841,
+            -0.00813915953040123,
+            0.011531638912856579,
+            -0.011568867601454258,
+            0.016538919880986214,
+            0.030118146911263466,
+            -0.031830672174692154,
+            -0.008530061691999435,
+            0.01570127159357071,
+            0.008651056326925755,
+            0.013253473676741123,
+            0.008804624900221825,
+            0.007631916087120771,
+            -0.011233807541429996,
+            -0.005817009136080742,
+            0.000602642132434994,
+            0.0062032584100961685,
+            -0.013802599161863327,
+            0.02162996679544449,
+            0.0010011072736233473,
+            0.002340764505788684,
+            0.004704797174781561,
+            0.01128965150564909,
+            -0.012760191224515438,
+            0.00357629731297493,
+            -0.007613301742821932,
+            -0.02537146769464016,
+            -0.026674477383494377,
+            -0.008134505711495876,
+            -0.006724462378770113,
+            -0.020680630579590797,
+            0.002073182025924325,
+            0.006333559285849333,
+            0.01950792223215103,
+            0.014724013395607471,
+            -0.009246718138456345,
+            -0.007250320166349411,
+            -0.024868877604603767,
+            0.010777754709124565,
+            -0.026376646012067795,
+            -0.02295159175992012,
+            0.027735499665141106,
+            0.013541997410356998,
+            0.014333110302686691,
+            -0.0016043310752138495,
+            0.0024315097834914923,
+            -0.010526459664106369,
+            0.013430310413241386,
+            0.02213255688548088,
+            0.004597763996571302,
+            0.007403889205306768,
+            -0.015105608850717545,
+            -0.002538542728871107,
+            0.020289728417992592,
+            -0.00022119177447166294,
+            -0.005002627614885569,
+            0.022486230358481407,
+            0.015422054566442966,
+            0.01886572316288948,
+            -0.004904902074486017,
+            0.015943259000778198,
+            0.01648307777941227,
+            0.008920964784920216,
+            -0.004106808453798294,
+            0.00020301362383179367,
+            -0.020997075363993645,
+            -0.004835098050534725,
+            0.03337567299604416,
+            0.012341366149485111,
+            0.025762369856238365,
+            -0.0263208020478487,
+            0.004825790878385305,
+            -0.004141710698604584,
+            0.013020792976021767,
+            0.01047061663120985,
+            0.019638223573565483,
+            0.028442848473787308,
+            0.009395632892847061,
+            0.009902875870466232,
+            -0.0012238987255841494,
+            0.029224654659628868,
+            -0.012611275538802147,
+            -0.009902875870466232,
+            -0.0006788449827581644,
+            -0.006622083019465208,
+            -0.004211514722555876,
+            0.00010339733853470534,
+            0.01476124208420515,
+            0.002815432380884886,
+            -0.019061176106333733,
+            0.014146965928375721,
+            0.007054868619889021,
+            -0.018819188699126244,
+            0.012127300724387169,
+            -0.02151828072965145,
+            0.011373416520655155,
+            -0.02738182619214058,
+            -0.0053656091913580894,
+            -0.01377467717975378,
+            -0.009358404204249382,
+            -0.023174963891506195,
+            -0.006933874916285276,
+            0.004034677520394325,
+            0.0082927281036973,
+            -0.0029317724984139204,
+            0.0043301815167069435,
+            -0.0030550931114703417,
+            -0.01600840874016285,
+            0.007929746992886066,
+            0.030937181785702705,
+            0.008716206066310406,
+            0.007450425066053867,
+            -0.0035530291497707367,
+            0.011224500834941864,
+            -0.008972154930233955,
+            -0.02213255688548088,
+            0.0002355888718739152,
+            -0.013467539101839066,
+            -0.008562637493014336,
+            0.013532689772546291,
+            0.006515050306916237,
+            -0.014202809892594814,
+            0.00905591994524002,
+            -0.024440744891762733,
+            -0.0002466411970090121,
+            -0.017786087468266487,
+            -0.00875343568623066,
+            -0.0053656091913580894,
+            -0.015021844767034054,
+            0.0013483826769515872,
+            0.007617955096065998,
+            -0.00890700425952673,
+            -0.006277716252952814,
+            0.00968881044536829,
+            -0.0037926898803561926,
+            -0.014333110302686691,
+            -0.0038880889769643545,
+            -0.008916311897337437,
+            -0.002316333120688796,
+            0.0018905280157923698,
+            -0.00985634047538042,
+            -0.005095699802041054,
+            -0.005407491698861122,
+            -0.026506947353482246,
+            0.006812881212681532,
+            0.019600993022322655,
+            0.004602417815476656,
+            0.020829545333981514,
+            0.0029573673382401466,
+            0.010526459664106369,
+            0.02931772544980049,
+            0.0055238320492208,
+            -0.008748781867325306,
+            -0.024887491017580032,
+            0.02159273810684681,
+            -0.010312393307685852,
+            0.024887491017580032,
+            -0.011447873897850513,
+            -0.014472718350589275,
+            0.0026548828464001417,
+            -0.00554244639351964,
+            -0.005267883650958538,
+            0.00035076565109193325,
+            0.006026421207934618,
+            -0.017497563734650612,
+            0.018009459599852562,
+            -0.011857391335070133,
+            0.008897697553038597,
+            0.010600917041301727,
+            0.041733551770448685,
+            -0.0013134806649759412,
+            -0.007375967688858509,
+            0.004462809301912785,
+            -0.001483337371610105,
+            0.01958237960934639,
+            -0.004497711546719074,
+            0.007282895501703024,
+            -0.009716732427477837,
+            -0.0007445771479979157,
+            -0.009265332482755184,
+            0.0062730624340474606,
+            0.015133530832827091,
+            -0.016771601513028145,
+            -0.01328139565885067,
+            0.0055052172392606735,
+            -0.026506947353482246,
+            0.0086138267070055,
+            0.030248448252677917,
+            0.017562713474035263,
+            -0.007324778009206057,
+            -0.014379646629095078,
+            0.007422503549605608,
+            -0.030080918222665787,
+            -0.0028480077162384987,
+            0.09649720042943954,
+            -0.009339789859950542,
+            -0.005128275137394667,
+            -0.02336110919713974,
+            -0.006524357479065657,
+            0.019228706136345863,
+            -0.014342417940497398,
+            0.07330362498760223,
+            -0.008748781867325306,
+            0.00046128881513141096,
+            -0.0073015098460018635,
+            0.001322787837125361,
+            -0.02053171582520008,
+            0.003978834021836519,
+            -0.02196502685546875,
+            -0.024868877604603767,
+            0.0021266986150294542,
+            -0.030118146911263466,
+            -0.007366660516709089,
+            -0.0034971858840435743,
+            0.008413721807301044,
+            -0.0049188630655407906,
+            0.011857391335070133,
+            -0.009865647181868553,
+            -0.012462359853088856,
+            -0.0011639834847301245,
+            0.005677401088178158,
+            0.015654735267162323,
+            -0.026264959946274757,
+            -0.01991743966937065,
+            0.010545074008405209,
+            0.01900533214211464,
+            0.004469790030270815,
+            -0.008609173819422722,
+            0.001268108026124537,
+            -0.00514223612844944,
+            0.008590559475123882,
+            0.029708629474043846,
+            0.01089874841272831,
+            0.01108489278703928,
+            0.0037531342823058367,
+            -0.011075585149228573,
+            0.015328982844948769,
+            0.007785485126078129,
+            0.013662991113960743,
+            -0.006021767854690552,
+            -0.004816483706235886,
+            -0.0021022669970989227,
+            0.009558509103953838,
+            -0.02220701426267624,
+            -0.014119044877588749,
+            -0.012238986790180206,
+            0.016967052593827248,
+            -0.006938528269529343,
+            -0.0005441812099888921,
+            -0.009791189804673195,
+            0.013579226098954678,
+            -0.008874429389834404,
+            -0.0051236217841506,
+            0.005184118635952473,
+            -0.021276293322443962,
+            0.022467615082859993,
+            0.006864070892333984,
+            0.021983640268445015,
+            -0.002318659797310829,
+            0.003573970403522253,
+            -0.014714706689119339,
+            -0.006580200511962175,
+            0.0106939896941185,
+            -0.004718757700175047,
+            0.00822292361408472,
+            0.002288411371409893,
+            0.02894543670117855,
+            0.0005770472926087677,
+            -0.013523383066058159,
+            0.008250845596194267,
+            -0.011587481945753098,
+            -0.01791638880968094,
+            0.010182092897593975,
+            -0.020382799208164215,
+            0.009060573764145374,
+            -6.11876675975509e-05,
+            -0.025445925071835518,
+            -0.010005255229771137,
+            0.004932823590934277,
+            -0.006924567744135857,
+            -0.013141786679625511,
+            0.013718834146857262,
+            -0.01853066496551037,
+            -0.01071260403841734,
+            -0.007194476667791605,
+            0.02101569063961506,
+            -0.016846058890223503,
+            0.023379722610116005,
+            -0.020680630579590797,
+            0.00634286692366004,
+            -0.00875343568623066,
+            0.02015942707657814,
+            -0.004453502129763365,
+            0.018195604905486107,
+            0.0122948307543993,
+            0.00674307718873024,
+            0.002838700544089079,
+            0.007469039876013994,
+            0.009698117151856422,
+            0.02581821382045746,
+            0.01982436701655388,
+            -0.003245891071856022,
+            -0.008534715510904789,
+            -0.011615403927862644,
+            0.013839827850461006,
+            -0.005793740972876549,
+            -0.00922810286283493,
+            0.011782933957874775,
+            0.016027024015784264,
+            -0.014947386458516121,
+            0.014854314737021923,
+            -0.0028666220605373383,
+            0.01086151972413063,
+            -0.013904978521168232,
+            0.004013736266642809,
+            0.0009057082934305072,
+            0.012369288131594658,
+            0.0075993407517671585,
+            -0.00016491221322212368,
+            0.01541274692863226,
+            0.02040141448378563,
+            -0.0004836842999793589,
+            0.01651099883019924,
+            0.0046210321597754955,
+            0.010377543978393078,
+            0.0021104109473526478,
+            -0.0037926898803561926,
+            -0.00514223612844944,
+            0.0021871954668313265,
+            -0.007892518304288387,
+            0.008786010555922985,
+            0.011056970804929733,
+            -0.010768447071313858,
+            0.002801471622660756,
+            0.012611275538802147,
+            -0.011913234367966652,
+            -0.008511447347700596,
+            0.0038275918923318386,
+            -0.026879236102104187,
+            -0.006850109901279211,
+            0.028089173138141632,
+            -0.018214218318462372,
+            -0.007273588329553604,
+            0.010033177211880684,
+            0.012425131164491177,
+            -0.0010011072736233473,
+            -0.014472718350589275,
+            0.009079188108444214,
+            0.005216693971306086,
+            -0.02557622641324997,
+            0.0024128954391926527,
+            0.011606096290051937,
+            0.018577199429273605,
+            -0.0004990993766114116,
+            -0.010768447071313858,
+            0.0013367487117648125,
+            0.025352852419018745,
+            0.016492383554577827,
+            0.013067329302430153,
+            -0.0050910464487969875,
+            -0.003990468103438616,
+            0.0051375823095440865,
+            0.0012064477195963264,
+            0.002601366490125656,
+            0.00668257987126708,
+            -0.0012983564520254731,
+            -0.0059054275043308735,
+            0.01348615437746048,
+            -0.0010458981851115823,
+            0.012322751805186272,
+            -0.013700219802558422,
+            -0.0186981949955225,
+            -0.0013134806649759412,
+            -0.003874127985909581,
+            -0.012387902475893497,
+            0.019470693543553352,
+            -0.003285446669906378,
+            0.03709855675697327,
+            0.013458232395350933,
+            0.005086392629891634,
+            -0.002601366490125656,
+            -0.008404415100812912,
+            0.016631992533802986,
+            0.0053656091913580894,
+            -0.01520798821002245,
+            0.003927644342184067,
+            -0.029168810695409775,
+            0.021071534603834152,
+            -0.009865647181868553,
+            0.010554381646215916,
+            -0.002633941825479269,
+            0.007906478829681873,
+            0.004876980558037758,
+            -0.011466488242149353,
+            -0.003373865270987153,
+            0.009567816741764545,
+            0.0022220974788069725,
+            -0.012127300724387169,
+            -0.0015612852293998003,
+            0.008678977377712727,
+            -0.006049689371138811,
+            0.012211065739393234,
+            0.005272537004202604,
+            -0.008520754985511303,
+            0.00436741067096591
+          ],
+          "title": "Conclusion"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "4222a5ca-3cde-4141-8aa5-9e61ff049dfa",
+      "type": "next",
+      "source": {
+        "id": "396d9412-17a9-4b7e-831f-84f4e5006484",
+        "properties": {
+          "page_content": "9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\n",
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "MoE language models",
+            "supervised fine-tuning",
+            "large-scale language models"
+          ],
+          "embedding": [
+            -0.003971853759139776,
+            -0.012639197520911694,
+            -0.032538022845983505,
+            0.01764647848904133,
+            0.028833750635385513,
+            0.017460335046052933,
+            0.0033017343375831842,
+            0.0448235459625721,
+            -0.002440816955640912,
+            0.05025895684957504,
+            -0.011708475649356842,
+            -0.010033177211880684,
+            0.026748934760689735,
+            -0.023044662550091743,
+            -0.0071293264627456665,
+            0.011140735819935799,
+            0.007008332759141922,
+            0.022765446454286575,
+            0.005472642369568348,
+            -0.03823403641581535,
+            0.014482025988399982,
+            -0.014649556018412113,
+            -0.05766750127077103,
+            -0.008972154930233955,
+            0.008334610611200333,
+            0.017088046297430992,
+            0.018679579719901085,
+            0.0209412332624197,
+            -0.06805435568094254,
+            0.020010510459542274,
+            0.045344747602939606,
+            0.012024921365082264,
+            -0.0010371727403253317,
+            -0.006017114035785198,
+            -0.006594161503016949,
+            -0.03143977001309395,
+            0.04270150139927864,
+            0.015459283255040646,
+            -0.024254601448774338,
+            0.02741905488073826,
+            0.02278406172990799,
+            -0.006980410777032375,
+            -0.02874067798256874,
+            -0.009809804148972034,
+            0.00014906085561960936,
+            0.007208437658846378,
+            -0.006026421207934618,
+            0.005379570182412863,
+            -0.0071293264627456665,
+            -0.005030549596995115,
+            -0.012173837050795555,
+            -0.014119044877588749,
+            0.02192779816687107,
+            -0.01348615437746048,
+            -0.025278395041823387,
+            0.010154170915484428,
+            0.014649556018412113,
+            0.0407283715903759,
+            0.04463740065693855,
+            0.026693090796470642,
+            0.021778881549835205,
+            -0.030192604288458824,
+            0.003183067310601473,
+            -0.000935375050175935,
+            0.02533423714339733,
+            0.013030100613832474,
+            0.004416273441165686,
+            0.03518127277493477,
+            0.023658938705921173,
+            -0.02598574385046959,
+            -0.049328237771987915,
+            0.02230008691549301,
+            -0.007487654220312834,
+            -0.012797419913113117,
+            0.004241763148456812,
+            -0.00017014751210808754,
+            0.0030667271930724382,
+            0.0407283715903759,
+            -0.023510023951530457,
+            0.003876454895362258,
+            0.010749832727015018,
+            0.013662991113960743,
+            0.010498537681996822,
+            -0.0014414548641070724,
+            0.04325993359088898,
+            -0.014845007099211216,
+            -0.010051791556179523,
+            -0.09188082069158554,
+            0.0304159764200449,
+            -0.010126249864697456,
+            -0.020475871860980988,
+            0.006919913925230503,
+            0.013616454787552357,
+            -0.011280343867838383,
+            -0.009781882166862488,
+            0.027605198323726654,
+            0.01356991846114397,
+            0.006398709956556559,
+            -0.0263208020478487,
+            0.01287187822163105,
+            -0.013030100613832474,
+            0.0022663066629320383,
+            0.012918413616716862,
+            0.003997448831796646,
+            -0.00012244803656358272,
+            -0.008799971081316471,
+            -0.029466642066836357,
+            0.030267061665654182,
+            0.014156273566186428,
+            0.10290056467056274,
+            0.00873482134193182,
+            -0.0014821739168837667,
+            -0.035088200122117996,
+            0.015133530832827091,
+            -0.028722064569592476,
+            -0.01276949793100357,
+            -0.0439300537109375,
+            -0.03203543275594711,
+            0.005868198815733194,
+            0.0019475346198305488,
+            0.03531157225370407,
+            0.01991743966937065,
+            0.02561345510184765,
+            -0.0014717032900080085,
+            -0.06310291588306427,
+            0.046982817351818085,
+            -0.015608198940753937,
+            -0.02019665576517582,
+            0.010358929634094238,
+            -0.004600090906023979,
+            -0.016101481392979622,
+            -0.018028074875473976,
+            -0.033115070313215256,
+            -0.04102620109915733,
+            0.0028037982992827892,
+            -0.010991820134222507,
+            0.014584405347704887,
+            0.06392195075750351,
+            -0.040951743721961975,
+            -0.01026585791260004,
+            0.015989795327186584,
+            0.010935977101325989,
+            0.009302561171352863,
+            -0.020866775885224342,
+            -0.029299112036824226,
+            -0.0041649783961474895,
+            0.0076691447757184505,
+            0.08391384780406952,
+            -0.007845981977880001,
+            0.02118322066962719,
+            -0.024831648916006088,
+            0.011205886490643024,
+            0.011969078332185745,
+            0.004658260848373175,
+            0.0013623435515910387,
+            -0.013523383066058159,
+            -0.030490435659885406,
+            0.005826316308230162,
+            -0.021034304052591324,
+            0.02980170026421547,
+            0.0025920593179762363,
+            -0.025632068514823914,
+            -0.0009185057133436203,
+            0.01166194025427103,
+            -0.0058030481450259686,
+            0.017702322453260422,
+            0.021127376705408096,
+            0.0077249882742762566,
+            -0.060087379068136215,
+            0.053758472204208374,
+            0.01866096444427967,
+            0.0218347255140543,
+            -0.024329058825969696,
+            0.004721084609627724,
+            0.00043831163202412426,
+            0.017972230911254883,
+            0.0070595224387943745,
+            0.011410645209252834,
+            -0.005281844176352024,
+            0.033561814576387405,
+            -0.015245217829942703,
+            0.023863697424530983,
+            0.021648580208420753,
+            -0.0014717032900080085,
+            0.030453206971287727,
+            0.013495461083948612,
+            0.018474821001291275,
+            0.02894543670117855,
+            -0.0181862972676754,
+            0.005430759862065315,
+            0.014044586569070816,
+            0.00012295703345444053,
+            0.029448026791214943,
+            -0.006738423369824886,
+            -0.007389928214251995,
+            -0.037340544164180756,
+            -0.018484128639101982,
+            0.02455243282020092,
+            -0.0004868836549576372,
+            0.009372364729642868,
+            -0.00881858542561531,
+            -0.03901584446430206,
+            -0.004251070320606232,
+            -0.012946335598826408,
+            0.028870979323983192,
+            0.005100353620946407,
+            -0.011150043457746506,
+            0.005807701963931322,
+            0.022058097645640373,
+            -0.013672297820448875,
+            -0.012434438802301884,
+            -0.007520229555666447,
+            0.002373339608311653,
+            -0.02220701426267624,
+            -0.0041021546348929405,
+            0.0037182322703301907,
+            -0.01921009086072445,
+            0.0027642427012324333,
+            0.008585905656218529,
+            -0.00825549941509962,
+            -0.01627831906080246,
+            0.015040459111332893,
+            0.03441808000206947,
+            -0.016203859820961952,
+            -0.03585138916969299,
+            -0.024310443550348282,
+            -0.022560687735676765,
+            0.018996024504303932,
+            0.028498690575361252,
+            -0.009511973708868027,
+            0.03091856651008129,
+            0.008367185480892658,
+            0.014295881614089012,
+            0.004711777437478304,
+            0.021127376705408096,
+            -0.008074008859694004,
+            -0.011354802176356316,
+            -0.018791265785694122,
+            0.014212116599082947,
+            0.026488332077860832,
+            0.00021682900842279196,
+            0.018260754644870758,
+            -0.04556812345981598,
+            -0.011540945619344711,
+            0.0020394434686750174,
+            -0.0006038055289536715,
+            -0.012406516820192337,
+            -0.016864672303199768,
+            0.01268573384732008,
+            0.029652785509824753,
+            0.008381146937608719,
+            -0.016557535156607628,
+            -0.002878256142139435,
+            0.00434181559830904,
+            -0.04084005579352379,
+            0.052455462515354156,
+            0.03990933671593666,
+            -0.012648504227399826,
+            -0.024366287514567375,
+            -0.033282600343227386,
+            0.05710906907916069,
+            -0.02380785532295704,
+            -0.025222551077604294,
+            0.06961796432733536,
+            -0.013849135488271713,
+            0.019768523052334785,
+            0.05707183852791786,
+            0.0026944386772811413,
+            0.004486077465116978,
+            -0.01794430986046791,
+            0.026227731257677078,
+            -0.021369364112615585,
+            -0.01190392766147852,
+            0.006394056603312492,
+            0.03030429035425186,
+            -0.012332059442996979,
+            -0.01941484957933426,
+            -0.002408241853117943,
+            0.0011081402190029621,
+            0.015059073455631733,
+            0.002245365409180522,
+            0.009539894759654999,
+            0.013514075428247452,
+            0.012536818161606789,
+            0.00968881044536829,
+            -0.0047303917817771435,
+            -0.01110350713133812,
+            -0.04325993359088898,
+            -0.03162591531872749,
+            -0.01740449108183384,
+            -0.035572174936532974,
+            0.0013146440032869577,
+            -0.021909182891249657,
+            0.011205886490643024,
+            0.03786174952983856,
+            -0.009027997963130474,
+            0.003348270431160927,
+            0.018884338438510895,
+            -0.0023140062112361193,
+            -0.0005747204995714128,
+            -0.031718987971544266,
+            -0.018577199429273605,
+            -0.00893958006054163,
+            0.00128555903211236,
+            -0.05126413702964783,
+            -0.01171778328716755,
+            0.008418375626206398,
+            0.03103025257587433,
+            -0.02250484563410282,
+            -0.023621710017323494,
+            0.015878107398748398,
+            -0.019805751740932465,
+            -0.028647607192397118,
+            -0.02931772544980049,
+            -0.019191475585103035,
+            0.05465196445584297,
+            0.016706449910998344,
+            0.025967128574848175,
+            -0.03147700056433678,
+            -0.03423193469643593,
+            0.05025895684957504,
+            0.008474218659102917,
+            -0.07550012320280075,
+            0.036595966666936874,
+            0.0014507620362564921,
+            0.0010063425870612264,
+            -0.005495910067111254,
+            0.05837484821677208,
+            -0.0032575251534581184,
+            -0.0010057608596980572,
+            0.012536818161606789,
+            -0.05748135596513748,
+            -0.00575651228427887,
+            -0.006412670947611332,
+            0.013253473676741123,
+            -0.04642438516020775,
+            -0.024943334981799126,
+            0.004027697257697582,
+            -0.012974257580935955,
+            -0.005700668785721064,
+            -0.042180296033620834,
+            -0.03145838528871536,
+            -0.0028526613023132086,
+            -0.0022756140679121017,
+            -0.007943707518279552,
+            -0.03078826516866684,
+            -0.025929899886250496,
+            0.021164605394005775,
+            0.045344747602939606,
+            0.007641223259270191,
+            0.0017695341957733035,
+            0.023621710017323494,
+            0.0197871383279562,
+            0.005598289426416159,
+            0.020345570519566536,
+            -0.01645515486598015,
+            -0.0375266894698143,
+            -0.005495910067111254,
+            -0.00010630584438331425,
+            0.02680477872490883,
+            -0.00926067866384983,
+            -0.049700524657964706,
+            -0.033599045127630234,
+            0.0193403922021389,
+            -0.03687518462538719,
+            -0.015487205237150192,
+            -0.004313894081860781,
+            -0.033636271953582764,
+            0.013141786679625511,
+            0.00029579491820186377,
+            -0.01730211265385151,
+            0.049328237771987915,
+            -0.023454179987311363,
+            -0.0019975609611719847,
+            -0.023752011358737946,
+            0.003443669294938445,
+            0.05312558263540268,
+            -0.018502742052078247,
+            -0.018028074875473976,
+            0.022672373801469803,
+            -0.01730211265385151,
+            0.01849343627691269,
+            0.04314824566245079,
+            0.009837726131081581,
+            0.004304586909711361,
+            0.014724013395607471,
+            0.006226526573300362,
+            -0.008483526296913624,
+            -0.0386807844042778,
+            0.021871954202651978,
+            0.012220372445881367,
+            -0.045307520776987076,
+            -0.022597916424274445,
+            -0.02479442022740841,
+            0.0007329431246034801,
+            0.011801548302173615,
+            0.016557535156607628,
+            -0.00968881044536829,
+            -0.028387004509568214,
+            0.006421978119760752,
+            -0.016017716377973557,
+            0.010163478553295135,
+            0.00786459632217884,
+            0.014845007099211216,
+            -0.005519178230315447,
+            0.0067756520584225655,
+            0.004006756003946066,
+            0.012788113206624985,
+            0.0362422950565815,
+            -0.02721429616212845,
+            -0.013216244988143444,
+            -0.020271113142371178,
+            -0.05506148189306259,
+            0.0014589058700948954,
+            0.015059073455631733,
+            0.004544247407466173,
+            -0.02455243282020092,
+            0.005709975957870483,
+            0.012546124868094921,
+            -0.04180800914764404,
+            -0.003641447750851512,
+            0.012099378742277622,
+            0.034678682684898376,
+            0.028238089755177498,
+            -0.011596789583563805,
+            0.003436689032241702,
+            0.0011959770927205682,
+            -0.0032435643952339888,
+            -0.0007631916087120771,
+            -0.023100506514310837,
+            0.027493512257933617,
+            0.014649556018412113,
+            0.038941387087106705,
+            -0.023472795262932777,
+            -0.005998499691486359,
+            -0.048211369663476944,
+            -0.003466937458142638,
+            0.030248448252677917,
+            -0.020680630579590797,
+            -0.011736397631466389,
+            -0.0005956617533229291,
+            0.005044510122388601,
+            -0.014733321033418179,
+            -0.006338213104754686,
+            -0.03194236010313034,
+            -0.026674477383494377,
+            -0.04143571853637695,
+            -0.01702289469540119,
+            -0.00934909749776125,
+            -0.01416558027267456,
+            -0.05748135596513748,
+            -0.011364108882844448,
+            -0.09463575482368469,
+            0.009218796156346798,
+            0.04117511585354805,
+            -0.0257437564432621,
+            0.028238089755177498,
+            -0.028796521946787834,
+            -0.0035576827358454466,
+            -0.03501374274492264,
+            0.012648504227399826,
+            0.040616683661937714,
+            0.033115070313215256,
+            -0.015291753225028515,
+            -0.006966450251638889,
+            0.017516177147626877,
+            -0.030732423067092896,
+            -0.03802927955985069,
+            -0.019228706136345863,
+            -0.00846025813370943,
+            0.00572859076783061,
+            0.0018474820535629988,
+            0.03789897635579109,
+            0.04504691809415817,
+            0.04631270095705986,
+            0.03592584654688835,
+            0.00934909749776125,
+            -0.035367414355278015,
+            -0.010740526020526886,
+            0.005216693971306086,
+            0.03071380779147148,
+            -0.03305922448635101,
+            0.01166194025427103,
+            -0.0311233252286911,
+            0.01853066496551037,
+            0.002405914943665266,
+            -0.023454179987311363,
+            0.021853338927030563,
+            0.005947310011833906,
+            0.0013507094699889421,
+            -0.005919388495385647,
+            0.027325982227921486,
+            -0.01828867755830288,
+            0.035125426948070526,
+            -0.028442848473787308,
+            -0.024105684831738472,
+            -0.028684835880994797,
+            -0.011122121475636959,
+            0.031104711815714836,
+            0.019154246896505356,
+            -0.0005947891622781754,
+            -0.002433836692944169,
+            -0.015273138880729675,
+            0.009172259829938412,
+            0.004853712394833565,
+            -0.02032695710659027,
+            -0.016334161162376404,
+            0.026879236102104187,
+            -0.06943181902170181,
+            -0.009763267822563648,
+            0.008306688629090786,
+            0.0005555243697017431,
+            0.00929790735244751,
+            -0.0386807844042778,
+            0.010377543978393078,
+            0.004590783733874559,
+            -0.03990933671593666,
+            0.022877132520079613,
+            0.00926067866384983,
+            0.01971268095076084,
+            -0.004362756852060556,
+            -0.008837200701236725,
+            0.007022293284535408,
+            0.06451760977506638,
+            -0.010563688352704048,
+            0.0022791041992604733,
+            0.007455078884959221,
+            -0.00812519807368517,
+            0.03845741227269173,
+            -0.025222551077604294,
+            -0.028033331036567688,
+            0.02935495413839817,
+            -0.008455604314804077,
+            -0.007650530431419611,
+            -0.015310367569327354,
+            -0.006696540862321854,
+            0.007827367633581161,
+            0.02870344929397106,
+            -0.02373339794576168,
+            -0.005212040152400732,
+            -0.02676755003631115,
+            0.006524357479065657,
+            0.0022500192280858755,
+            0.002908504568040371,
+            0.0061334543861448765,
+            -0.0009958719601854682,
+            0.039685964584350586,
+            -0.0225420743227005,
+            0.03452976793050766,
+            0.02959694154560566,
+            -0.004090521018952131,
+            -0.0045582083985209465,
+            -0.0005939166294410825,
+            0.013179016299545765,
+            -0.01289049256592989,
+            -0.01808391883969307,
+            0.01736726239323616,
+            -0.018679579719901085,
+            -0.013244166038930416,
+            -0.054949793964624405,
+            -0.002410568529739976,
+            0.002610673662275076,
+            -0.010051791556179523,
+            0.006784959230571985,
+            -0.001237859483808279,
+            0.002903850981965661,
+            0.005393530707806349,
+            0.03949981927871704,
+            -0.03827126696705818,
+            0.023714782670140266,
+            0.00015429616905748844,
+            -0.0026525561697781086,
+            0.008716206066310406,
+            0.0035018394701182842,
+            0.007106058299541473,
+            0.05554545670747757,
+            0.012899799272418022,
+            -0.007994897663593292,
+            -0.005063124932348728,
+            0.005002627614885569,
+            0.016296932473778725,
+            -0.016027024015784264,
+            0.0052073863334953785,
+            0.03668903931975365,
+            0.025297008454799652,
+            0.000632890616543591,
+            -0.0017951290355995297,
+            -0.010182092897593975,
+            0.02751212567090988,
+            -0.021034304052591324,
+            0.029820315539836884,
+            0.04307378828525543,
+            0.009470091201364994,
+            -0.0439300537109375,
+            0.0069571430794894695,
+            -0.015431362204253674,
+            -0.009251371026039124,
+            0.0031528188847005367,
+            0.019600993022322655,
+            -0.04381836578249931,
+            0.003245891071856022,
+            0.01145718153566122,
+            0.008320650085806847,
+            -0.021052919328212738,
+            -0.04675944522023201,
+            -0.008674323558807373,
+            -0.0035274343099445105,
+            0.012974257580935955,
+            -0.016846058890223503,
+            -0.006510396488010883,
+            0.005742551293224096,
+            0.016259703785181046,
+            -0.004988667089492083,
+            0.012034228071570396,
+            0.033394284546375275,
+            -0.016408618539571762,
+            0.013783984817564487,
+            -0.007054868619889021,
+            -0.02189056947827339,
+            0.0110476640984416,
+            0.03095579519867897,
+            -0.01307663694024086,
+            0.03546048700809479,
+            -0.0015345270512625575,
+            0.012415824458003044,
+            0.012964949943125248,
+            0.003639120841398835,
+            -0.014817086048424244,
+            0.002092959824949503,
+            0.018335212022066116,
+            -0.021443821489810944,
+            0.013020792976021767,
+            -0.013541997410356998,
+            -0.037992049008607864,
+            0.010051791556179523,
+            -0.0038950692396610975,
+            0.0016752986703068018,
+            0.009041959419846535,
+            0.0062591019086539745,
+            -0.027195680886507034,
+            -0.01611078903079033,
+            0.023342493921518326,
+            0.024105684831738472,
+            0.06098087131977081,
+            -0.014081815257668495,
+            -0.000535164843313396,
+            -0.007617955096065998,
+            0.009493358433246613,
+            0.04460017383098602,
+            0.024459360167384148,
+            0.0038694743998348713,
+            0.004290625918656588,
+            0.027102608233690262,
+            0.046945590525865555,
+            -0.005230654496699572,
+            -0.010377543978393078,
+            0.006394056603312492,
+            0.003197028301656246,
+            0.023789240047335625,
+            0.021574122831225395,
+            -0.0005409818259067833,
+            0.014072508551180363,
+            0.014444797299802303,
+            -0.016538919880986214,
+            0.012937027961015701,
+            -0.04556812345981598,
+            -0.040169939398765564,
+            0.007115365471690893,
+            0.00435112277045846,
+            0.019359005615115166,
+            0.002450124127790332,
+            -0.005039856769144535,
+            -0.0027851839549839497,
+            0.01750687137246132,
+            -0.010535767301917076,
+            -0.0122948307543993,
+            0.02475719153881073,
+            -0.00873482134193182,
+            0.006747730541974306,
+            0.00030801063985563815,
+            -0.009716732427477837,
+            0.0035181271377950907,
+            -0.0014972981298342347,
+            -0.005077085457742214,
+            0.0023989344481378794,
+            -0.008381146937608719,
+            -0.017516177147626877,
+            0.012229680083692074,
+            -0.04396728053689003,
+            0.013886364176869392,
+            0.001635742955841124,
+            -0.008911658078432083,
+            0.002878256142139435,
+            0.0013146440032869577,
+            0.0114944102242589,
+            -0.018381748348474503,
+            -0.005309765692800283,
+            0.012453053146600723,
+            0.026413874700665474,
+            -0.007403889205306768,
+            0.0038904156535863876,
+            0.015142838470637798,
+            -0.007534190081059933,
+            -0.00987495481967926,
+            0.028070559725165367,
+            -0.007822713814675808,
+            0.006859417073428631,
+            -0.008181042037904263,
+            -0.02540869638323784,
+            0.0053888773545622826,
+            0.03793620690703392,
+            0.04865811765193939,
+            0.007538843899965286,
+            0.01648307777941227,
+            0.03123501129448414,
+            0.014835700392723083,
+            0.00822292361408472,
+            0.015422054566442966,
+            -0.04437679797410965,
+            0.01541274692863226,
+            -0.018037382513284683,
+            0.040542226284742355,
+            0.030844109132885933,
+            -0.05044510215520859,
+            -0.02812640182673931,
+            0.019284548237919807,
+            0.023100506514310837,
+            0.017562713474035263,
+            0.013681605458259583,
+            -0.005314419511705637,
+            0.02053171582520008,
+            -0.023249421268701553,
+            -0.01235998049378395,
+            0.01316040102392435,
+            -0.0032435643952339888,
+            -0.004816483706235886,
+            0.005067778285592794,
+            0.03912752866744995,
+            -0.00436741067096591,
+            -0.016911208629608154,
+            0.02131352201104164,
+            -0.005984539166092873,
+            0.005928695667535067,
+            -0.013123172335326672,
+            0.0009958719601854682,
+            0.03916475921869278,
+            0.005393530707806349,
+            0.03761976212263107,
+            -0.021499665454030037,
+            0.012145915068686008,
+            -0.028182245790958405,
+            -0.014733321033418179,
+            0.029243268072605133,
+            -0.013216244988143444,
+            -0.021201834082603455,
+            0.016976360231637955,
+            0.020513100549578667,
+            0.0032156426459550858,
+            0.011168657802045345,
+            -0.013681605458259583,
+            -0.010600917041301727,
+            0.038159579038619995,
+            -0.0029527137521654367,
+            -0.004144037142395973,
+            0.01712527498602867,
+            -0.01812114752829075,
+            -0.011922542005777359,
+            0.016883287578821182,
+            -0.0019580053631216288,
+            -0.032742779701948166,
+            0.02241177298128605,
+            -0.0004022461944259703,
+            0.03620506450533867,
+            -0.010237935930490494,
+            -0.0056634400971233845,
+            0.010535767301917076,
+            -0.006654658354818821,
+            -0.029987845569849014,
+            -0.034176092594861984,
+            0.007455078884959221,
+            -0.027437668293714523,
+            0.010963899083435535,
+            -0.006650005001574755,
+            0.03255663812160492,
+            -0.030248448252677917,
+            0.008669670671224594,
+            0.03030429035425186,
+            -0.00825549941509962,
+            0.022690989077091217,
+            0.032947540283203125,
+            -0.019079789519309998,
+            -0.03525573015213013,
+            -0.003245891071856022,
+            -0.0003592003195080906,
+            0.029578328132629395,
+            0.002880582818761468,
+            0.0026246344204992056,
+            0.004960745573043823,
+            -0.016222475096583366,
+            -0.03510681167244911,
+            0.022430386394262314,
+            -0.04474908858537674,
+            0.007068829610943794,
+            0.026618633419275284,
+            -0.0430365614593029,
+            -0.036800727248191833,
+            -0.018446899950504303,
+            -0.0263208020478487,
+            -0.02159273810684681,
+            0.001237859483808279,
+            0.02639526128768921,
+            -0.017581328749656677,
+            -0.030602121725678444,
+            0.028089173138141632,
+            0.0034948589745908976,
+            -0.012695040553808212,
+            -0.011056970804929733,
+            -0.010126249864697456,
+            -0.027083994820713997,
+            -0.022262856364250183,
+            -0.014035279862582684,
+            0.0009807477472350001,
+            0.004881633911281824,
+            -0.010368237271904945,
+            -0.0225420743227005,
+            -0.013616454787552357,
+            -0.009032651782035828,
+            -0.004825790878385305,
+            0.050147272646427155,
+            0.021090148016810417,
+            0.023863697424530983,
+            0.0009185057133436203,
+            -0.015719885006546974,
+            0.04601486772298813,
+            -0.0007253810181282461,
+            0.02591128461062908,
+            0.005612250417470932,
+            -0.022058097645640373,
+            -0.005844930652529001,
+            0.03715439885854721,
+            -0.018279369920492172,
+            -0.004395332187414169,
+            -0.014724013395607471,
+            -0.00374150020070374,
+            -0.008609173819422722,
+            0.026693090796470642,
+            0.012024921365082264,
+            -0.004769947379827499,
+            0.013458232395350933,
+            -0.01270434819161892,
+            0.02200225554406643,
+            0.005821662489324808,
+            -0.003676349762827158,
+            -0.0010331007651984692,
+            -0.028684835880994797,
+            -0.02114599198102951,
+            0.001068002893589437,
+            0.022765446454286575,
+            -0.016334161162376404,
+            -0.013197630643844604,
+            -0.015291753225028515,
+            -0.021574122831225395,
+            -0.005198079161345959,
+            0.0004831026017200202,
+            0.005407491698861122,
+            0.015850186347961426,
+            -0.003311041509732604,
+            -0.0430365614593029,
+            -0.005272537004202604,
+            -0.03763837739825249,
+            0.00048455686192028224,
+            0.01961960829794407,
+            -0.011243115179240704,
+            0.01128965150564909,
+            0.001805599662475288,
+            -0.0038648208137601614,
+            -0.01832590624690056,
+            0.007636569440364838,
+            0.01982436701655388,
+            -0.006789613049477339,
+            -0.0022197705693542957,
+            0.018986716866493225,
+            0.0021569470409303904,
+            -0.007645876612514257,
+            0.004849058575928211,
+            -0.001855625887401402,
+            -0.006845456548035145,
+            -0.023379722610116005,
+            -0.029280496761202812,
+            -0.013607148081064224,
+            0.0018090897938236594,
+            -0.001733468729071319,
+            -0.024906106293201447,
+            0.018875030800700188,
+            -0.006082264706492424,
+            0.02336110919713974,
+            -0.011326880194246769,
+            0.01316040102392435,
+            -0.0022697970271110535,
+            -0.0090931486338377,
+            0.018214218318462372,
+            -0.010554381646215916,
+            -0.03264970704913139,
+            -0.01316040102392435,
+            0.028517305850982666,
+            0.00034872969263233244,
+            -0.008097277022898197,
+            0.010042484849691391,
+            0.00922810286283493,
+            -0.027437668293714523,
+            -0.0077017201110720634,
+            0.010619531385600567,
+            0.006626736838370562,
+            -0.032333262264728546,
+            0.026506947353482246,
+            0.03266832232475281,
+            0.01235998049378395,
+            -0.03222157806158066,
+            0.048620887100696564,
+            0.008651056326925755,
+            -0.02475719153881073,
+            0.005677401088178158,
+            0.020624786615371704,
+            0.0363725945353508,
+            -0.0299692302942276,
+            -0.007413196377456188,
+            -0.007422503549605608,
+            0.009009383618831635,
+            -0.03473452478647232,
+            0.015235910192131996,
+            -0.02537146769464016,
+            0.01961960829794407,
+            0.032333262264728546,
+            0.0032831199932843447,
+            0.040579456835985184,
+            -0.009511973708868027,
+            0.017693014815449715,
+            0.0060310750268399715,
+            -0.0076691447757184505,
+            0.006459206808358431,
+            -0.02162996679544449,
+            0.01624108850955963,
+            0.019768523052334785,
+            0.02162996679544449,
+            0.009902875870466232,
+            -0.024682732298970222,
+            0.01941484957933426,
+            0.015133530832827091,
+            0.023826468735933304,
+            0.027605198323726654,
+            -0.014658862724900246,
+            0.026842007413506508,
+            0.005812355317175388,
+            -0.013635069131851196,
+            0.004430233966559172,
+            0.009386326186358929,
+            -0.040616683661937714,
+            -0.0012948662042617798,
+            -0.022523459047079086,
+            0.01459371205419302,
+            -0.0005066614830866456,
+            -0.0028224128764122725,
+            -0.02377062663435936,
+            -0.013653683476150036,
+            -7.831148832337931e-05,
+            0.005146889481693506,
+            -0.004358103033155203,
+            -0.04087728634476662,
+            -0.045754265040159225,
+            0.0234355665743351,
+            -0.009800496511161327,
+            -0.040542226284742355,
+            -0.012648504227399826,
+            -0.009311867877840996,
+            -0.0015752461040392518,
+            -0.014109737239778042,
+            -0.01289049256592989,
+            -0.04385559633374214,
+            -0.014435489661991596,
+            0.01958237960934639,
+            0.030043689534068108,
+            -0.02203948423266411,
+            0.030248448252677917,
+            0.006873378064483404,
+            0.015198681503534317,
+            -0.014630941674113274,
+            0.013560611754655838,
+            0.00614741537719965,
+            -0.04638715833425522,
+            0.02894543670117855,
+            -0.0028898902237415314,
+            0.021276293322443962,
+            0.0384201817214489,
+            -0.005272537004202604,
+            -0.010023870505392551,
+            0.017246268689632416,
+            -0.02213255688548088,
+            -0.013551304116845131,
+            -0.008092623203992844,
+            -0.01586880162358284,
+            -0.037005484104156494,
+            -0.04884426295757294,
+            0.016129402443766594,
+            0.04080282896757126,
+            0.00911176297813654,
+            0.014426182955503464,
+            0.018679579719901085,
+            0.004311567172408104,
+            -0.033282600343227386,
+            0.0225420743227005,
+            0.0047932155430316925,
+            -0.012387902475893497,
+            -0.023826468735933304,
+            0.013653683476150036,
+            0.012778805568814278,
+            -0.006198605056852102,
+            -0.006384748965501785,
+            -0.026655862107872963,
+            -0.024477973580360413,
+            0.0021476398687809706,
+            -0.013886364176869392,
+            0.0033250025007873774,
+            -0.022765446454286575,
+            -0.02857314795255661,
+            0.032128505408763885,
+            0.03546048700809479,
+            -0.01235998049378395,
+            -0.00393229816108942,
+            -0.013579226098954678,
+            -0.008404415100812912,
+            0.008920964784920216,
+            0.005230654496699572,
+            0.017953617498278618,
+            -0.019526535645127296,
+            0.01128965150564909,
+            -0.010442694649100304,
+            0.012099378742277622,
+            -0.0762074738740921,
+            -0.01679021492600441,
+            0.025874055922031403,
+            0.022672373801469803,
+            0.0009662052034400403,
+            0.011438566260039806,
+            0.008893043734133244,
+            0.0006619756459258497,
+            -0.035088200122117996,
+            0.01921009086072445,
+            0.013793292455375195,
+            0.016222475096583366,
+            -0.01151302456855774,
+            0.005235308315604925,
+            0.023845084011554718,
+            -0.0006683743558824062,
+            -0.006319598760455847,
+            -0.01760924980044365,
+            0.011540945619344711,
+            0.020103583112359047,
+            0.0026851315051317215,
+            0.013877056539058685,
+            0.010572995990514755,
+            -0.01961960829794407,
+            -0.013011486269533634,
+            -0.0005930440966039896,
+            -0.005668093450367451,
+            -0.010963899083435535,
+            0.008706899359822273,
+            0.0007399235619232059,
+            0.03763837739825249,
+            0.024887491017580032,
+            0.02172303944826126,
+            0.015980487689375877,
+            0.0069571430794894695,
+            0.007469039876013994,
+            -0.0020917963702231646,
+            0.0018323578406125307,
+            0.01746964268386364,
+            0.03272416815161705,
+            -0.022932976484298706,
+            -0.005156196653842926,
+            0.0042533972300589085,
+            0.024831648916006088,
+            -0.008827893063426018,
+            0.025892671197652817,
+            0.030062302947044373,
+            -0.007650530431419611,
+            -0.029503870755434036,
+            0.0032924271654337645,
+            0.018502742052078247,
+            0.009302561171352863,
+            -0.01634346880018711,
+            -0.0027921644505113363,
+            0.013728141784667969,
+            0.0075434972532093525,
+            0.020922617986798286,
+            -0.004241763148456812,
+            -0.014053894206881523,
+            0.01880057342350483,
+            0.09076395630836487,
+            -0.005691361613571644,
+            -0.0012017941335216165,
+            -0.017106659710407257,
+            -0.01921009086072445,
+            0.0014670497039332986,
+            0.0098935691639781,
+            0.016613377258181572,
+            0.017209040001034737,
+            0.00374615378677845,
+            0.004586129914969206,
+            0.01455648336559534,
+            -0.011531638912856579,
+            0.009479397907853127,
+            -0.02421737276017666,
+            0.01804668828845024,
+            -0.014249345287680626,
+            -0.0067756520584225655,
+            -0.011233807541429996,
+            0.023621710017323494,
+            -0.0019568419083952904,
+            0.007720334455370903,
+            0.019154246896505356,
+            0.003871801309287548,
+            -0.0247199609875679,
+            0.006412670947611332,
+            -0.006454553455114365,
+            0.0007160738459788263,
+            -0.038866929709911346,
+            -0.018363134935498238,
+            0.01127103716135025,
+            -0.0014123697765171528,
+            -0.0011232644319534302,
+            -0.004672221839427948,
+            -0.007804099470376968,
+            -0.00025580296642147005,
+            -0.02455243282020092,
+            0.016836751252412796,
+            0.008855815045535564,
+            0.015403440222144127,
+            0.022393157705664635,
+            -0.0010522969532757998,
+            -0.0009138521272689104,
+            0.025315623730421066,
+            -0.0071293264627456665,
+            -0.004043984692543745,
+            0.027232909575104713,
+            -0.023621710017323494,
+            -0.005188771989196539,
+            0.005300458520650864,
+            -0.01692982390522957,
+            -0.002913158154115081,
+            -0.00414636405184865,
+            -0.021034304052591324,
+            -0.013206937350332737,
+            0.0014914810890331864,
+            0.020717859268188477,
+            -0.015468590892851353,
+            -0.0029178117401897907,
+            0.013048714958131313,
+            -0.005286497995257378,
+            0.007585379760712385,
+            0.0032901002559810877,
+            -0.005067778285592794,
+            -0.02114599198102951,
+            -0.026860620826482773,
+            -0.007999550551176071,
+            0.0011936502996832132,
+            2.35407078434946e-05,
+            -0.0037321930285543203,
+            -0.009372364729642868,
+            -0.004830444231629372,
+            -0.01089874841272831,
+            0.007855288684368134,
+            -0.007413196377456188,
+            -0.014928772114217281,
+            0.013988743536174297,
+            0.005226001143455505,
+            0.008436989970505238,
+            -0.031086096540093422,
+            0.013830521143972874,
+            0.020587557926774025,
+            -0.006961796432733536,
+            0.009237410500645638,
+            -0.015198681503534317,
+            -0.00025216734502464533,
+            -0.01685536466538906,
+            0.004043984692543745,
+            0.004632666241377592,
+            -0.01169916894286871,
+            0.005379570182412863,
+            0.007403889205306768,
+            -0.01913563348352909,
+            -0.006929221097379923,
+            -0.02060617320239544,
+            0.00492817023769021,
+            0.011578175239264965,
+            0.009000076912343502,
+            -0.01559889130294323,
+            -0.004886287730187178,
+            -0.03449253737926483,
+            0.016101481392979622,
+            0.017078738659620285,
+            -0.029001280665397644,
+            -0.014584405347704887,
+            0.009046612307429314,
+            0.013588533736765385,
+            -0.01771163009107113,
+            -0.012834648601710796,
+            -0.0009981987532228231,
+            0.005938002839684486,
+            -0.018744729459285736,
+            0.00533768767490983,
+            -0.0037647683639079332,
+            0.01832590624690056,
+            0.0014740300830453634,
+            0.024850262328982353,
+            -0.026581404730677605,
+            0.010107634589076042,
+            0.017004281282424927,
+            -0.027474896982312202,
+            0.0005968251498416066,
+            0.014658862724900246,
+            0.00042202402255497873,
+            0.00020286819199100137,
+            0.0004857202584389597,
+            -0.007594686932861805,
+            -0.008483526296913624,
+            0.004513998981565237,
+            -0.0014658862492069602,
+            -0.026432489976286888,
+            0.01950792223215103,
+            0.01410043053328991,
+            0.013104557991027832,
+            0.008799971081316471,
+            -0.011652632616460323,
+            0.0006020604632794857,
+            0.01573850028216839,
+            0.032742779701948166,
+            -0.0014891542959958315,
+            -0.015654735267162323,
+            0.010014562867581844,
+            0.026637248694896698,
+            0.0003402950242161751,
+            0.0008190348744392395,
+            -0.01603632979094982,
+            -0.026469718664884567,
+            0.038978613913059235,
+            0.018270062282681465,
+            0.01958237960934639,
+            -0.0018253774615004659,
+            -0.01781400851905346,
+            0.01455648336559534,
+            -0.004437214694917202,
+            0.0027595891151577234,
+            -0.015301060862839222,
+            -0.009418901056051254,
+            0.0026572097558528185,
+            0.026655862107872963,
+            -0.005374916363507509,
+            0.03382241725921631,
+            0.012052842415869236,
+            0.009423554874956608,
+            -0.035609401762485504,
+            0.0031528188847005367,
+            -0.02462689019739628,
+            0.011559560894966125,
+            -0.0094142472371459,
+            -0.0026944386772811413,
+            0.008590559475123882,
+            -0.02935495413839817,
+            -0.009032651782035828,
+            0.01011694222688675,
+            0.023063277825713158,
+            0.007366660516709089,
+            -0.023007433861494064,
+            0.020178040489554405,
+            0.0013134806649759412,
+            0.02151828072965145,
+            -0.012052842415869236,
+            0.012266908772289753,
+            -0.008744128048419952,
+            -0.017804700881242752,
+            -0.005170157644897699,
+            0.03374795988202095,
+            -0.008530061691999435,
+            -0.0036577354185283184,
+            0.006254448089748621,
+            -0.0279216431081295,
+            0.032612480223178864,
+            -0.0037182322703301907,
+            -0.0029154850635677576,
+            0.0041975537315011024,
+            -0.012006307020783424,
+            0.00991218350827694,
+            0.0094142472371459,
+            0.008115891367197037,
+            -0.018707500770688057,
+            -0.023714782670140266,
+            -0.04184523597359657,
+            -0.024533817544579506,
+            0.014733321033418179,
+            0.011503716930747032,
+            -0.015114916488528252,
+            0.03387825936079025,
+            -0.02192779816687107,
+            -0.016306240111589432,
+            -0.011447873897850513,
+            0.028684835880994797,
+            -0.03722885996103287,
+            0.00875343568623066,
+            -0.01350476872175932,
+            -0.020587557926774025,
+            -0.00148566416464746,
+            0.004900248255580664,
+            -0.002478045877069235,
+            -0.009176913648843765,
+            -0.02939218282699585,
+            0.0060636503621935844,
+            0.04999835789203644,
+            -0.0055936360731720924,
+            -0.0016415599966421723,
+            0.00418126629665494,
+            0.003774075536057353,
+            -0.014342417940497398,
+            -0.016501691192388535,
+            0.03486482426524162,
+            -0.03806650638580322,
+            0.0017520831897854805,
+            0.01845620572566986,
+            -0.003583277575671673,
+            0.00554244639351964,
+            0.004523306153714657,
+            0.003578623989596963,
+            0.012629889883100986,
+            -0.00727824168279767,
+            0.013579226098954678,
+            0.011056970804929733,
+            0.011624710634350777,
+            0.005291151348501444,
+            0.014910157769918442,
+            0.025185322389006615,
+            -0.01706012338399887,
+            0.024868877604603767,
+            0.025017792358994484,
+            0.00014222587924450636,
+            0.03581416234374046,
+            -0.019433464854955673,
+            -0.026004357263445854,
+            0.005626211408525705,
+            -0.011745704337954521,
+            -0.004711777437478304,
+            0.001653193961828947,
+            -0.0220953281968832,
+            -0.0031993549782782793,
+            -0.011401337571442127,
+            0.0011895783245563507,
+            0.014053894206881523,
+            -0.014724013395607471,
+            -0.0204944871366024,
+            0.015422054566442966,
+            0.013430310413241386,
+            0.02861037850379944,
+            0.01518937386572361,
+            -0.0034971858840435743,
+            0.02881513722240925,
+            -0.011996999382972717,
+            0.014789164066314697,
+            0.008190348744392395,
+            0.011913234367966652,
+            -0.0069571430794894695,
+            0.024329058825969696,
+            0.0019963975064456463,
+            0.009456129744648933,
+            0.001500788377597928,
+            -0.01692051626741886,
+            -0.0034297085367143154,
+            -0.034213319420814514,
+            -0.018344519659876823,
+            0.003041132353246212,
+            -0.00020417702035047114,
+            0.08748781681060791,
+            0.006980410777032375,
+            -0.02090400457382202,
+            0.003069054102525115,
+            0.0011866698041558266,
+            0.005212040152400732,
+            -0.018279369920492172,
+            0.015180067159235477,
+            -0.027456283569335938,
+            -0.020140811800956726,
+            -0.017832623794674873,
+            -0.024980563670396805,
+            0.0030969756189733744,
+            0.004339488688856363,
+            0.0102472435683012,
+            -0.027474896982312202,
+            0.0019824367482215166,
+            -0.006701194681227207,
+            -0.020717859268188477,
+            0.009027997963130474,
+            -0.01924731954932213,
+            0.004816483706235886,
+            0.011419951915740967,
+            -0.00728754885494709,
+            -0.01791638880968094,
+            0.007603994105011225,
+            0.005314419511705637,
+            0.0033878260292112827,
+            -0.01682744361460209,
+            -5.409818550106138e-05,
+            -0.00768775912001729,
+            0.004092847462743521,
+            0.009595737792551517,
+            -0.013821213506162167,
+            0.015505819581449032,
+            0.006366134621202946,
+            -0.008790664374828339,
+            0.024515202268958092,
+            0.00873482134193182,
+            -0.02241177298128605,
+            -0.0018463187152519822,
+            -0.011689861305058002,
+            0.005658786278218031,
+            -0.0016799522563815117,
+            0.004206860903650522,
+            -0.011354802176356316,
+            -0.016287624835968018,
+            -0.02693507820367813,
+            -0.0008167080814018846,
+            -0.006645351182669401,
+            -0.005374916363507509,
+            0.0036554085090756416,
+            -0.006380095612257719,
+            0.011764319613575935,
+            0.00553313922137022,
+            -0.01396082155406475,
+            0.01045200228691101,
+            -0.0005985702155157924,
+            0.013169708661735058,
+            -0.026879236102104187,
+            0.005314419511705637,
+            -0.026655862107872963,
+            0.009642274118959904,
+            0.01330001000314951,
+            -0.038866929709911346,
+            0.015533741563558578,
+            -0.007101404946297407,
+            0.015570970252156258,
+            -0.0014088796451687813,
+            -0.01828867755830288,
+            0.020550329238176346,
+            0.007887864485383034,
+            -0.008841853588819504,
+            0.012332059442996979,
+            0.025073636323213577,
+            0.00967019610106945,
+            0.03830849379301071,
+            -0.016259703785181046,
+            -0.0003152818826492876,
+            -0.0064312852919101715,
+            0.007036254275590181,
+            -0.005449374206364155,
+            -0.0021301887463778257,
+            -0.020475871860980988,
+            0.029243268072605133,
+            -0.008450950495898724,
+            0.007762216962873936,
+            0.02282129041850567,
+            -0.0051050069741904736,
+            -0.009418901056051254,
+            0.004507018718868494,
+            0.012099378742277622,
+            0.018167681992053986,
+            -0.0032342569902539253,
+            0.015096302144229412,
+            -0.015487205237150192,
+            0.012211065739393234,
+            0.008250845596194267,
+            0.0015240564243867993,
+            -0.027903029695153236,
+            -0.022970205172896385,
+            -0.0017858217470347881,
+            0.0012750884052366018,
+            0.002101103775203228,
+            0.012434438802301884,
+            0.00748300040140748,
+            -0.006822188384830952,
+            0.029652785509824753,
+            -0.005798394791781902,
+            0.012099378742277622,
+            -0.0002384973777225241,
+            0.021481052041053772,
+            0.01862373575568199,
+            0.025017792358994484,
+            -0.0010325191542506218,
+            -0.01009832788258791,
+            0.005226001143455505,
+            -0.006859417073428631,
+            0.01991743966937065,
+            0.010759140364825726,
+            -0.01328139565885067,
+            -0.023752011358737946,
+            0.015049765817821026,
+            0.00011735816224245355,
+            -0.0018963449401780963,
+            0.012695040553808212,
+            -0.02336110919713974,
+            0.0009510809904895723,
+            -0.035404644906520844,
+            0.015077687799930573,
+            -0.04240366816520691,
+            -0.01672506518661976,
+            0.005221347324550152,
+            -0.014258652925491333,
+            -0.01026585791260004,
+            0.017795395106077194,
+            0.017748858779668808,
+            -0.008213616907596588,
+            0.014975308440625668,
+            -0.008697591722011566,
+            -0.009753961116075516,
+            0.013811906799674034,
+            0.012387902475893497,
+            -0.02315635047852993,
+            0.015301060862839222,
+            0.005212040152400732,
+            0.0058914669789373875,
+            -0.021871954202651978,
+            -0.02012219838798046,
+            0.006840802729129791,
+            0.01706012338399887,
+            -0.0003562917991075665,
+            0.0016194552881643176,
+            0.006631390191614628,
+            0.003539068391546607,
+            0.00614741537719965,
+            0.003271485911682248,
+            0.0003123733913525939,
+            -0.000290705036604777,
+            0.02274683304131031,
+            0.011140735819935799,
+            -0.015524433925747871,
+            0.0071293264627456665,
+            -0.009335136041045189,
+            0.0005508707836270332,
+            0.009377018548548222,
+            -1.0870535334106535e-05,
+            0.014649556018412113,
+            -0.008176388218998909,
+            0.013653683476150036,
+            0.02043864317238331,
+            -0.007445771712809801,
+            -0.003076034365221858,
+            -0.008948886767029762,
+            0.006891992408782244,
+            0.016082866117358208,
+            0.025148093700408936,
+            -0.017748858779668808,
+            -0.0002715089067351073,
+            -0.009032651782035828,
+            -0.009437515400350094,
+            -0.026842007413506508,
+            0.017050817608833313,
+            0.020289728417992592,
+            0.0073573533445596695,
+            0.017795395106077194,
+            -0.00849748682230711,
+            0.011326880194246769,
+            -0.010191399604082108,
+            -0.0023244768381118774,
+            -0.00374150020070374,
+            -0.012648504227399826,
+            0.002643248997628689,
+            0.023826468735933304,
+            -0.013914286158978939,
+            0.00790182501077652,
+            -0.0013646703446283937,
+            0.03681934252381325,
+            -0.011233807541429996,
+            0.005612250417470932,
+            0.02019665576517582,
+            -0.004979359917342663,
+            -0.011773626320064068,
+            0.02295159175992012,
+            0.008841853588819504,
+            0.0035530291497707367,
+            0.005635518580675125,
+            0.024031227454543114,
+            0.011224500834941864,
+            0.0016043310752138495,
+            0.000546508003026247,
+            -0.04716896265745163,
+            0.00888838991522789,
+            -0.001733468729071319,
+            -0.010638146661221981,
+            0.017525484785437584,
+            0.010619531385600567,
+            0.009395632892847061,
+            -0.003269159235060215,
+            0.019191475585103035,
+            -0.012266908772289753,
+            0.012508896179497242,
+            0.033636271953582764,
+            0.0034157477784901857,
+            -0.019014639779925346,
+            0.0021848685573786497,
+            0.018009459599852562,
+            -0.0024222026113420725,
+            0.00217090779915452,
+            0.006705848034471273,
+            -0.017190424725413322,
+            0.0004281318688299507,
+            0.0036903105210512877,
+            0.029224654659628868,
+            -0.015180067159235477,
+            -0.027828572317957878,
+            0.0004447103419806808,
+            -0.005872852168977261,
+            -0.003073707688599825,
+            0.008841853588819504,
+            0.0032924271654337645,
+            -0.0016217821976169944,
+            -0.012722962535917759,
+            0.0017323052743449807,
+            0.007743602618575096,
+            0.010787061415612698,
+            0.003583277575671673,
+            -0.005965924356132746,
+            -0.008916311897337437,
+            0.021462436765432358,
+            -0.006045036017894745,
+            -0.0038834353908896446,
+            -0.0035925847478210926,
+            -0.006980410777032375,
+            -0.010172785259783268,
+            -0.012778805568814278,
+            0.011122121475636959,
+            -0.001270434819161892,
+            -0.01437033899128437,
+            -0.017004281282424927,
+            0.010461308993399143,
+            -0.0067756520584225655,
+            0.025725141167640686,
+            0.0001714563404675573,
+            -0.008032126352190971,
+            0.003369211684912443,
+            -0.015319675207138062,
+            -0.010535767301917076,
+            0.007752909790724516,
+            -0.002316333120688796,
+            -0.00495609175413847,
+            0.01091736275702715,
+            -0.0032575251534581184,
+            0.018958795815706253,
+            -0.0012913760729134083,
+            -0.008385800756514072,
+            0.002873602556064725,
+            0.0025571573060005903,
+            -0.012806727550923824,
+            -0.0006579037290066481,
+            -0.02131352201104164,
+            -0.013476846739649773,
+            -0.0014519254909828305,
+            0.014091122895479202,
+            -0.010777754709124565,
+            0.005868198815733194,
+            0.0010674211662262678,
+            0.006896646227687597,
+            -0.016371389850974083,
+            -0.010237935930490494,
+            -0.0012890491634607315,
+            0.024198757484555244,
+            0.011001127772033215,
+            -0.02319357916712761,
+            -0.010824290104210377,
+            -0.01007971353828907,
+            0.018055995926260948,
+            -0.013635069131851196,
+            0.004490730818361044,
+            0.013234859332442284,
+            -0.016194554045796394,
+            -0.0009208325645886362,
+            -0.012601968832314014,
+            -0.02220701426267624,
+            0.0003839225973933935,
+            0.032705552875995636,
+            0.012015613727271557,
+            0.017953617498278618,
+            -0.008078662678599358,
+            -0.00011510407057357952,
+            0.03168175742030144,
+            0.0032505446579307318,
+            0.0018253774615004659,
+            -0.01437033899128437,
+            0.009381672367453575,
+            -0.009493358433246613,
+            0.018875030800700188,
+            0.006612775847315788,
+            -0.00827411375939846,
+            0.003231930313631892,
+            -0.02233731560409069,
+            -0.003841552883386612,
+            -0.005244615487754345,
+            -0.009698117151856422,
+            0.012825341895222664,
+            -0.007040907628834248,
+            -0.015543048270046711,
+            0.0005316746537573636,
+            -0.007124672643840313,
+            -0.015980487689375877,
+            -0.008758088573813438,
+            -0.011373416520655155,
+            -0.030062302947044373,
+            -0.027567969635128975,
+            0.0010226302547380328,
+            -0.005868198815733194,
+            -0.028461461886763573,
+            -0.019359005615115166,
+            -0.022393157705664635,
+            0.009097802452743053,
+            -0.012369288131594658,
+            -0.02101569063961506,
+            0.007157247979193926,
+            0.005095699802041054,
+            -0.01842828467488289,
+            -0.026469718664884567,
+            -0.014435489661991596,
+            -0.016334161162376404,
+            0.025278395041823387,
+            0.012713654898107052,
+            -0.0008283421047963202,
+            -0.005174811463803053,
+            0.012220372445881367,
+            0.0032784664072096348,
+            0.01127103716135025,
+            -0.03268693760037422,
+            -0.03129085525870323,
+            -0.011187272146344185,
+            -0.00430691335350275,
+            0.0007451588753610849,
+            -0.03300338238477707,
+            0.002910831244662404,
+            0.003069054102525115,
+            -0.03827126696705818,
+            -0.008706899359822273,
+            0.015040459111332893,
+            -0.0009749307064339519,
+            0.011261729523539543,
+            0.00635217409580946,
+            0.026618633419275284,
+            0.02332388050854206,
+            0.0014437816571444273,
+            4.0937200537882745e-05,
+            0.012211065739393234,
+            -0.014705399051308632,
+            0.0014961347915232182,
+            0.005775126628577709,
+            0.008585905656218529,
+            -0.0004086449043825269,
+            0.015031151473522186,
+            -0.001642723334953189,
+            -0.015403440222144127,
+            0.017311418429017067,
+            0.01541274692863226,
+            0.00934909749776125,
+            0.002934099407866597,
+            0.0009121070033870637,
+            0.0038019970525056124,
+            0.0037880362942814827,
+            0.006770998705178499,
+            -0.01227621641010046,
+            -0.006417324300855398,
+            0.016901900991797447,
+            -0.011829469352960587,
+            -0.013942207209765911,
+            0.019805751740932465,
+            -0.00024489607312716544,
+            -0.023826468735933304,
+            0.02758658491075039,
+            0.006328905932605267,
+            -0.004479097202420235,
+            -0.02537146769464016,
+            -0.0018323578406125307,
+            -0.024664118885993958,
+            0.015543048270046711,
+            0.005407491698861122,
+            0.0015449975617229939,
+            0.022597916424274445,
+            0.01108489278703928,
+            0.0038206116296350956,
+            -0.013206937350332737,
+            0.006231179926544428,
+            0.01106627844274044,
+            0.015896722674369812,
+            -0.03704271465539932,
+            -0.02455243282020092,
+            0.012285523116588593,
+            -0.01841897703707218,
+            -0.005244615487754345,
+            0.012788113206624985,
+            0.0009766757721081376,
+            -0.02019665576517582,
+            0.0031667796429246664,
+            0.003608872415497899,
+            0.004758313298225403,
+            0.012192451395094395,
+            0.016631992533802986,
+            0.0122948307543993,
+            -0.0018521357560530305,
+            -0.022970205172896385,
+            0.006007806863635778,
+            -0.008241538889706135,
+            0.0007137469947338104,
+            -0.023342493921518326,
+            -0.0018102532485499978,
+            0.010610224679112434,
+            0.00767845194786787,
+            0.025874055922031403,
+            0.030676579102873802,
+            0.027903029695153236,
+            0.013244166038930416,
+            -0.004942130763083696,
+            -0.014416875317692757,
+            -0.020959846675395966,
+            0.0019917439203709364,
+            0.002347744768485427,
+            -0.011568867601454258,
+            -0.001557794981636107,
+            0.017488256096839905,
+            0.004011409357190132,
+            0.021816110238432884,
+            -0.005793740972876549,
+            0.017460335046052933,
+            -3.364776057424024e-05,
+            -0.005212040152400732,
+            -0.0008463748381473124,
+            -0.003173760138452053,
+            0.013244166038930416,
+            0.006650005001574755,
+            0.026004357263445854,
+            0.00022031922708265483,
+            0.003708925098180771,
+            0.005481949541717768,
+            -0.0007905315142124891,
+            -0.0008289237739518285,
+            -0.007380621042102575,
+            -0.003539068391546607,
+            -0.009847032837569714,
+            0.034678682684898376,
+            0.025278395041823387,
+            -0.02250484563410282,
+            -0.015124224126338959,
+            -0.012713654898107052,
+            0.02639526128768921,
+            -0.010833597742021084,
+            0.006524357479065657,
+            0.010796369053423405,
+            0.008660363033413887,
+            0.01065676100552082,
+            0.000908035144675523,
+            0.005719283130019903,
+            0.0028898902237415314,
+            0.022672373801469803,
+            -0.012611275538802147,
+            0.008669670671224594,
+            0.003920664079487324,
+            0.008357878774404526,
+            -0.015077687799930573,
+            -0.001618291949853301,
+            0.005872852168977261,
+            -0.014919465407729149,
+            0.006170683074742556,
+            -0.004118442535400391,
+            0.006207912229001522,
+            0.011559560894966125,
+            0.0015775728970766068,
+            -0.01607356034219265,
+            -0.017935002222657204,
+            0.012927721254527569,
+            -0.009781882166862488,
+            0.00916295312345028,
+            -0.018028074875473976,
+            -0.011606096290051937,
+            0.017897773534059525,
+            -0.006119493395090103,
+            -0.011931848712265491,
+            0.0026292880065739155,
+            0.00411146180704236,
+            -5.733389480155893e-05,
+            0.004009082447737455,
+            0.007752909790724516,
+            0.0020592212677001953,
+            0.00535630201920867,
+            0.005947310011833906,
+            -0.002478045877069235,
+            0.002440816955640912,
+            -0.0019789463840425014,
+            0.035739704966545105,
+            -0.009847032837569714,
+            0.006617429666221142,
+            -0.0022372216917574406,
+            -0.0012099378509446979,
+            -0.04154740646481514,
+            0.027772728353738785,
+            -0.006263755261898041,
+            0.015319675207138062,
+            0.00801816489547491,
+            -0.003401787020266056,
+            -0.010768447071313858,
+            0.008353224955499172,
+            0.01050784531980753,
+            -0.02135075069963932,
+            0.015142838470637798,
+            -0.024198757484555244,
+            -0.004218494985252619,
+            -0.002433836692944169,
+            -0.01108489278703928,
+            0.015496511943638325,
+            -0.01459371205419302,
+            0.008092623203992844,
+            -0.01091736275702715,
+            -0.0073713138699531555,
+            0.010144864208996296,
+            -0.0072549739852547646,
+            0.011010434478521347,
+            -0.042924873530864716,
+            0.0067756520584225655,
+            -0.004821137059479952,
+            -0.0010517152259126306,
+            -0.0006689560250379145,
+            -0.02270960435271263,
+            -0.031067483127117157,
+            -0.000392648100387305,
+            0.004853712394833565,
+            -0.005384223535656929,
+            0.00748300040140748,
+            0.005407491698861122,
+            0.004355776589363813,
+            0.0032621787395328283,
+            -0.01954515092074871,
+            0.025390081107616425,
+            0.03665181249380112,
+            -0.011829469352960587,
+            0.013383775018155575,
+            0.007161901798099279,
+            0.008148466236889362,
+            -0.03622367978096008,
+            -0.02352863922715187,
+            -0.008450950495898724,
+            0.009181567467749119,
+            0.02591128461062908,
+            -0.004679202102124691,
+            0.01559889130294323,
+            0.009437515400350094,
+            0.0050910464487969875,
+            -0.02870344929397106,
+            -0.015403440222144127,
+            -0.008046086877584457,
+            -0.015654735267162323,
+            -0.0022442021872848272,
+            -0.0033017343375831842,
+            0.014696091413497925,
+            -0.00473271869122982,
+            -0.018260754644870758,
+            -0.011941156350076199,
+            -0.006864070892333984,
+            -0.0037368466146290302,
+            -0.010154170915484428,
+            0.023547252640128136,
+            -0.010833597742021084,
+            -0.015589584596455097,
+            0.02512947842478752,
+            -0.001938227447681129,
+            0.005793740972876549,
+            0.005235308315604925,
+            0.00750161474570632,
+            -0.003304061247035861,
+            -0.004590783733874559,
+            -0.007673798594623804,
+            0.017627865076065063,
+            0.004202207550406456,
+            0.02874067798256874,
+            -0.004532613325864077,
+            0.00770637346431613,
+            0.0016764620086178184,
+            0.004355776589363813,
+            -0.002505967626348138,
+            0.008599866181612015,
+            0.006003153510391712,
+            -0.003648428013548255,
+            0.0071991304866969585,
+            -0.01350476872175932,
+            0.002410568529739976,
+            0.024943334981799126,
+            0.005984539166092873,
+            0.010414772666990757,
+            -0.012629889883100986,
+            -0.012667118571698666,
+            -0.003418074455112219,
+            -0.016203859820961952,
+            0.015924643725156784,
+            -0.020662017166614532,
+            0.018670272082090378,
+            -0.017460335046052933,
+            -0.001818397082388401,
+            0.0043930052779614925,
+            0.015198681503534317,
+            0.021090148016810417,
+            -0.004346469417214394,
+            0.026469718664884567,
+            0.013728141784667969,
+            0.0035227807238698006,
+            0.004970052745193243,
+            -0.022653760388493538,
+            -0.005514524411410093,
+            -0.011094199493527412,
+            0.003913683816790581,
+            -0.008581251837313175,
+            -0.016734370961785316,
+            -0.0010197217343375087,
+            0.007315470837056637,
+            -0.005398184526711702,
+            0.003701944602653384,
+            0.012657811865210533,
+            0.0015484878094866872,
+            0.0028293931391090155,
+            0.014230730943381786,
+            0.01651099883019924,
+            0.007510922383517027,
+            -0.007622608914971352,
+            0.0040323506109416485,
+            0.006622083019465208,
+            -0.0046280124224722385,
+            0.0009504992631264031,
+            0.017395183444023132,
+            0.01438895333558321,
+            -0.004060272127389908,
+            0.0053656091913580894,
+            -0.0027805303689092398,
+            -0.02676755003631115,
+            -0.014267959631979465,
+            -0.0003990468103438616,
+            -0.0005968251498416066,
+            -0.003506493056192994,
+            -0.015161452814936638,
+            0.004483750555664301,
+            0.012155221775174141,
+            -0.0122948307543993,
+            0.0035181271377950907,
+            0.008399761281907558,
+            -0.009949412196874619,
+            0.0021976660937070847,
+            -0.0034785715397447348,
+            -0.014249345287680626,
+            -0.009791189804673195,
+            0.027623813599348068,
+            -0.01011694222688675,
+            0.013746756128966808,
+            0.013635069131851196,
+            0.00039875597576610744,
+            -0.006440592464059591,
+            -0.0013041734928265214,
+            0.019061176106333733,
+            0.01746964268386364,
+            -0.001688096090219915,
+            -0.00036327220732346177,
+            -0.007552804425358772,
+            -0.006161375902593136,
+            0.01350476872175932,
+            -0.01634346880018711,
+            -0.004311567172408104,
+            0.009577123448252678,
+            -0.00846025813370943,
+            -0.004593110177665949,
+            -0.024533817544579506,
+            0.0021522934548556805,
+            0.009135031141340733,
+            0.0035227807238698006,
+            -0.0068314955569803715,
+            -0.001905652228742838,
+            -0.002934099407866597,
+            0.0058030481450259686,
+            -0.00118550646584481,
+            -0.00687803141772747,
+            -0.016836751252412796,
+            -0.0024175490252673626,
+            0.0070920973084867,
+            -0.006184644065797329,
+            -0.001833521295338869,
+            0.0036251600831747055,
+            -0.006082264706492424,
+            -0.015673348680138588,
+            0.03244495019316673,
+            9.452639642404392e-05,
+            0.021332135424017906,
+            0.004132403060793877,
+            0.0061334543861448765,
+            0.0027246871031820774,
+            0.007222398649901152,
+            0.02561345510184765,
+            0.009614353068172932,
+            0.006747730541974306,
+            0.01621316745877266,
+            0.014975308440625668,
+            -0.014240038581192493,
+            0.0072549739852547646,
+            0.012853262946009636,
+            -0.010163478553295135,
+            0.00417195912450552,
+            0.009251371026039124,
+            -0.0007649366743862629,
+            -0.003988141659647226,
+            -0.0037228858564049006,
+            0.016799522563815117,
+            0.003071380779147148,
+            -0.0008132178918458521,
+            0.012183143757283688,
+            0.01166194025427103,
+            -0.018754037097096443,
+            -0.00033796823117882013,
+            -0.0015135857975110412,
+            0.002343091182410717,
+            -0.005882159806787968,
+            -0.026581404730677605,
+            -0.008320650085806847,
+            0.0033854993525892496,
+            -0.0030434592626988888,
+            0.03091856651008129,
+            -0.00950266607105732,
+            -0.03354320302605629,
+            -0.01563611999154091,
+            -0.001225062063895166,
+            -0.01692982390522957,
+            -0.007966975681483746,
+            0.01276949793100357,
+            0.0295224841684103,
+            0.012415824458003044,
+            -0.013802599161863327,
+            0.024664118885993958,
+            -0.0028131057042628527,
+            0.01750687137246132,
+            -0.00870224554091692,
+            -0.009083840996026993,
+            -0.004509345628321171,
+            0.022113941609859467,
+            -0.008744128048419952,
+            -0.009605045430362225,
+            0.013579226098954678,
+            0.016985666006803513,
+            0.00842768233269453,
+            0.02323080785572529,
+            0.013644376769661903,
+            -0.007022293284535408,
+            -0.014891543425619602,
+            0.003808977548032999,
+            -0.008595212362706661,
+            -0.020252499729394913,
+            0.007324778009206057,
+            -0.014900850132107735,
+            0.005416798871010542,
+            -0.0021348423324525356,
+            -0.023696167394518852,
+            0.005537792574614286,
+            0.0053656091913580894,
+            -0.012043535709381104,
+            -0.012546124868094921,
+            0.000147243044921197,
+            -0.009507319889962673,
+            0.014463411644101143,
+            0.0068314955569803715,
+            -0.00023253493418451399,
+            -0.009251371026039124,
+            0.017832623794674873,
+            0.011001127772033215,
+            0.009968026541173458,
+            -0.00792043935507536,
+            0.0033924796152859926,
+            0.008934926241636276,
+            0.007590033579617739,
+            -0.0014495986979454756,
+            -0.0056634400971233845,
+            0.037880364805459976,
+            0.0030620736069977283,
+            0.012527510523796082,
+            -0.0017451026942580938,
+            0.009865647181868553,
+            0.01086151972413063,
+            0.015291753225028515,
+            0.010256550274789333,
+            -0.003646101336926222,
+            -0.0038275918923318386,
+            0.013458232395350933,
+            -0.012024921365082264,
+            0.0055052172392606735,
+            0.010275164619088173,
+            -0.0046675680205225945,
+            -0.005384223535656929,
+            0.010647453367710114,
+            0.00014309842663351446,
+            0.01812114752829075,
+            -0.005900774151086807,
+            0.009288599714636803,
+            0.018335212022066116,
+            0.009577123448252678,
+            -0.019098404794931412,
+            0.013728141784667969,
+            -0.01556166261434555,
+            0.029950616881251335,
+            0.01917286217212677,
+            -0.01812114752829075,
+            -0.0018975083949044347,
+            -0.011019742116332054,
+            0.009763267822563648,
+            -0.04221752658486366,
+            -0.01190392766147852,
+            0.0023919541854411364,
+            -0.004090521018952131,
+            -0.018884338438510895,
+            -0.027028150856494904,
+            0.0044558290392160416,
+            -0.004430233966559172,
+            -0.009353750385344028,
+            -0.008278767578303814,
+            0.0006509232916869223,
+            0.015924643725156784,
+            -0.005221347324550152,
+            -0.0077715241350233555,
+            -0.024310443550348282,
+            0.014993922784924507,
+            0.00886977557092905,
+            -0.004176612477749586,
+            -0.024403516203165054,
+            0.01455648336559534,
+            0.01603632979094982,
+            -0.00687803141772747,
+            0.01190392766147852,
+            -0.024049842730164528,
+            0.00197545625269413,
+            -0.005700668785721064,
+            -0.01654822751879692,
+            -0.015263832174241543,
+            0.0010999963851645589,
+            -0.012164529412984848,
+            -0.003313368419185281,
+            -0.014119044877588749,
+            -0.009083840996026993,
+            -0.0028945438098162413,
+            0.020475871860980988,
+            0.0025641375686973333,
+            -0.01541274692863226,
+            0.016390005126595497,
+            0.007869250141084194,
+            -0.0229888204485178,
+            -0.01631554774940014,
+            0.0046605877578258514,
+            -0.029876159504055977,
+            0.010805675759911537,
+            0.007869250141084194,
+            0.006826841738075018,
+            -0.02189056947827339,
+            0.007683105766773224,
+            0.02241177298128605,
+            0.007994897663593292,
+            0.0017660439480096102,
+            0.01913563348352909,
+            0.008185694925487041,
+            0.012667118571698666,
+            0.0019696392118930817,
+            -3.335691189931822e-06,
+            -0.005849584471434355,
+            -0.004148690961301327,
+            -0.004714104346930981,
+            0.03802927955985069,
+            -0.01617593877017498,
+            0.009498012252151966,
+            -0.022560687735676765,
+            0.0002395153569523245,
+            -0.01921009086072445,
+            -0.011298958212137222,
+            -0.015980487689375877,
+            0.0008010021410882473,
+            0.02151828072965145,
+            -0.007892518304288387,
+            -0.0002912867348641157,
+            -0.01583157107234001,
+            -0.0058356234803795815,
+            -0.006542971823364496,
+            0.02036418579518795,
+            -0.012741576880216599,
+            0.0018626062665134668,
+            -0.017311418429017067,
+            0.0010249570477753878,
+            0.026693090796470642,
+            0.005556406918913126,
+            -0.013225551694631577,
+            -0.026618633419275284,
+            -0.016194554045796394,
+            -0.009158299304544926,
+            -0.01316040102392435,
+            -0.005239961668848991,
+            -0.019228706136345863,
+            -0.011364108882844448,
+            -0.006473167799413204,
+            0.00654762564226985,
+            0.004039331339299679,
+            -0.015012537129223347,
+            -0.004997974261641502,
+            -0.024664118885993958,
+            -0.0202338844537735,
+            -0.010014562867581844,
+            -0.009470091201364994,
+            0.0013844481436535716,
+            0.02853591926395893,
+            0.0028549879789352417,
+            0.030397363007068634,
+            -0.00415567122399807,
+            0.008790664374828339,
+            0.009828418493270874,
+            0.007203784305602312,
+            -0.01520798821002245,
+            0.007487654220312834,
+            0.014584405347704887,
+            -0.008530061691999435,
+            0.0057890876196324825,
+            -0.0122948307543993,
+            -0.0007463222718797624,
+            0.005249268840998411,
+            0.004970052745193243,
+            0.011261729523539543,
+            -0.007548151072114706,
+            -0.01580365002155304,
+            0.013495461083948612,
+            -0.02040141448378563,
+            0.0011971404310315847,
+            0.00031295508961193264,
+            -0.003236583899706602,
+            0.003632140578702092,
+            0.018102532252669334,
+            -0.01336515974253416,
+            0.001435637823306024,
+            0.01999189704656601,
+            -0.012536818161606789,
+            0.01316040102392435,
+            0.003013210603967309,
+            0.0010214668000116944,
+            0.009511973708868027,
+            0.0011215193662792444,
+            0.005156196653842926,
+            0.0005630864761769772,
+            -0.019228706136345863,
+            -0.006966450251638889,
+            0.00831134244799614,
+            -0.024608274921774864,
+            -0.01128965150564909,
+            0.011177964508533478,
+            0.006254448089748621,
+            -0.017832623794674873,
+            -0.012788113206624985,
+            -0.023863697424530983,
+            0.01166194025427103,
+            -0.005467988550662994,
+            0.023993998765945435,
+            -0.015105608850717545,
+            0.009204835630953312,
+            -0.007738948799669743,
+            0.021164605394005775,
+            -0.0122948307543993,
+            0.005212040152400732,
+            0.0013576899655163288,
+            -0.01665060594677925,
+            0.009567816741764545,
+            -0.012285523116588593,
+            -0.0017567367758601904,
+            0.042775958776474,
+            0.006952489260584116,
+            0.0074271573685109615,
+            0.015068380162119865,
+            -0.022020868957042694,
+            0.01852135732769966,
+            0.002450124127790332,
+            0.018502742052078247,
+            0.010572995990514755,
+            -0.019265934824943542,
+            0.01695774495601654,
+            0.026823392137885094,
+            0.01982436701655388,
+            0.006221872754395008,
+            -0.01147579587996006,
+            0.027903029695153236,
+            0.012862570583820343,
+            0.0006660475628450513,
+            0.0022569994907826185,
+            0.0016846058424562216,
+            -0.011038356460630894,
+            -0.010842905379831791,
+            0.009018691256642342,
+            -0.017730243504047394,
+            -0.013272088021039963,
+            -0.0030923220328986645,
+            0.003080687951296568,
+            -0.010219321586191654,
+            -0.0036996176932007074,
+            0.006719809025526047,
+            0.007399235386401415,
+            0.011419951915740967,
+            0.0016020042821764946,
+            0.01611078903079033,
+            -0.004774601198732853,
+            -0.01893087476491928,
+            0.0002504222502466291,
+            -0.0032598518300801516,
+            0.014947386458516121,
+            -0.029299112036824226,
+            -0.007911132648587227,
+            -0.0004516907501965761,
+            0.005267883650958538,
+            0.006035728845745325,
+            0.007832021452486515,
+            -0.015096302144229412,
+            -0.005682054441422224,
+            -0.003113263286650181,
+            0.016120094805955887,
+            0.0016811155946925282,
+            0.023212192580103874,
+            0.015608198940753937,
+            -0.009968026541173458,
+            -0.006328905932605267,
+            0.0032668323256075382,
+            -0.010982513427734375,
+            0.00017320144979748875,
+            0.0025943859945982695,
+            0.011122121475636959,
+            -0.018847109749913216,
+            -0.008208963088691235,
+            0.025594839826226234,
+            0.02015942707657814,
+            0.00357629731297493,
+            0.026748934760689735,
+            -0.014174887910485268,
+            0.008227577432990074,
+            0.019842982292175293,
+            0.014407568611204624,
+            -0.010572995990514755,
+            -0.010731218382716179,
+            -0.0046675680205225945,
+            0.001287885825149715,
+            -0.0027549355290830135,
+            -0.005179464817047119,
+            0.006161375902593136,
+            -0.022467615082859993,
+            -0.016259703785181046,
+            0.0012157548917457461,
+            -0.013793292455375195,
+            0.0032342569902539253,
+            0.009358404204249382,
+            0.018502742052078247,
+            -0.011736397631466389,
+            0.01787916012108326,
+            -0.026879236102104187,
+            -0.003480898216366768,
+            0.010256550274789333,
+            0.0019545149989426136,
+            -0.016808830201625824,
+            -0.003911356907337904,
+            -0.002536216052249074,
+            -0.009218796156346798,
+            -0.006989717949181795,
+            0.01671575754880905,
+            0.012397210113704205,
+            0.015059073455631733,
+            -0.008195002563297749,
+            0.004423253703862429,
+            0.0023547252640128136,
+            0.009395632892847061,
+            -0.0018847109749913216,
+            -0.009442169219255447,
+            0.006696540862321854,
+            0.0017509197350591421,
+            -0.005863544996827841,
+            -0.005281844176352024,
+            0.023063277825713158,
+            -0.026134658604860306,
+            0.003641447750851512,
+            -0.016501691192388535,
+            0.011410645209252834,
+            0.012257601134479046,
+            0.002092959824949503,
+            -0.005598289426416159,
+            -0.006994371768087149,
+            0.013458232395350933,
+            -0.06496435403823853,
+            -0.008432336151599884,
+            -0.0015938604483380914,
+            0.027623813599348068,
+            -0.01794430986046791,
+            0.003169106552377343,
+            0.008404415100812912,
+            0.015422054566442966,
+            -0.007189823314547539,
+            0.007101404946297407,
+            0.016324853524565697,
+            -0.02261653169989586,
+            -0.011326880194246769,
+            0.00187075010035187,
+            -0.004974706098437309,
+            -0.011838776990771294,
+            0.0010470616398379207,
+            0.004500038456171751,
+            0.018940182402729988,
+            0.0015496512642130256,
+            0.011401337571442127,
+            -0.023510023951530457,
+            -0.013393081724643707,
+            -0.00436042994260788,
+            0.009060573764145374,
+            -0.005910081323236227,
+            0.0040323506109416485,
+            0.00931652169674635,
+            -0.03136531263589859,
+            0.011224500834941864,
+            0.008450950495898724,
+            -0.007241012994199991,
+            0.009605045430362225,
+            0.011326880194246769,
+            0.008474218659102917,
+            0.005267883650958538,
+            0.0011447874130681157,
+            -0.03210989013314247,
+            0.0043162209913134575,
+            0.013337238691747189,
+            -0.009200181812047958,
+            -0.013206937350332737,
+            -0.009977334178984165,
+            -0.0012366961454972625,
+            0.019973281770944595,
+            -0.004530286882072687,
+            -0.0018742403481155634,
+            0.014640248380601406,
+            0.006021767854690552,
+            0.01455648336559534,
+            -0.013337238691747189,
+            0.004942130763083696,
+            -0.010666067712008953,
+            -0.00854867696762085,
+            -0.0014100429834797978,
+            -0.0001836720621213317,
+            -0.0028968704864382744,
+            -0.038159579038619995,
+            0.00965158175677061,
+            -0.0036693692672997713,
+            -0.0197871383279562,
+            -0.01248097512871027,
+            -0.013411696068942547,
+            -0.004532613325864077,
+            -0.010870826430618763,
+            0.006854763720184565,
+            0.0031877208966761827,
+            0.009135031141340733,
+            -0.01336515974253416,
+            0.02496195025742054,
+            -0.014044586569070816,
+            -0.009977334178984165,
+            -0.009107109159231186,
+            -0.0005622139433398843,
+            0.002806125208735466,
+            0.0149846151471138,
+            -0.0038624941371381283,
+            -0.003204008564352989,
+            0.021853338927030563,
+            0.008013512007892132,
+            0.006114840041846037,
+            -0.01188531331717968,
+            -0.018875030800700188,
+            -0.023137735202908516,
+            -0.0029015240725129843,
+            -0.021946411579847336,
+            0.0007562111713923514,
+            0.004853712394833565,
+            0.02377062663435936,
+            0.007720334455370903,
+            -9.13997573661618e-05,
+            0.00436042994260788,
+            0.0038206116296350956,
+            0.019452078267931938,
+            -0.008371839299798012,
+            -0.004046311601996422,
+            -0.0071432869881391525,
+            -0.002633941825479269,
+            0.03482759743928909,
+            -0.008423029445111752,
+            0.00037955984589643776,
+            0.0013239512918516994,
+            -0.008879082277417183,
+            -0.007590033579617739,
+            0.010703296400606632,
+            -0.02881513722240925,
+            -0.014919465407729149,
+            -0.00036181797622703016,
+            0.0024873530492186546,
+            -0.010749832727015018,
+            0.02073647454380989,
+            -0.008502140641212463,
+            0.0023023721296340227,
+            0.009256024844944477,
+            0.024161528795957565,
+            0.022430386394262314,
+            0.023137735202908516,
+            0.007315470837056637,
+            -0.003671696176752448,
+            -0.002410568529739976,
+            -0.0067616915330290794,
+            -0.02496195025742054,
+            0.0161387100815773,
+            -0.0040951743721961975,
+            0.03404578939080238,
+            -0.03428777679800987,
+            0.0012413497315719724,
+            -0.007380621042102575,
+            -0.023472795262932777,
+            -0.009986640885472298,
+            0.023174963891506195,
+            -0.01045200228691101,
+            0.023900926113128662,
+            -0.0003123733913525939,
+            0.01556166261434555,
+            0.007906478829681873,
+            0.008441643789410591,
+            -0.01930316351354122,
+            0.011810855008661747,
+            -0.0023372743744403124,
+            0.01822352595627308,
+            0.009307214990258217,
+            -0.004607071168720722,
+            0.008832546882331371,
+            0.011689861305058002,
+            -0.01518937386572361,
+            -0.011196578852832317,
+            -0.0029015240725129843,
+            0.0018474820535629988,
+            -0.009847032837569714,
+            0.008553329855203629,
+            -0.002062711399048567,
+            -0.01600840874016285,
+            0.02537146769464016,
+            -0.01593395136296749,
+            0.010368237271904945,
+            0.004904902074486017,
+            0.00647782115265727,
+            0.01736726239323616,
+            -0.001187833258882165,
+            -0.027400439605116844,
+            0.0002552212681621313,
+            -0.03222157806158066,
+            -0.012341366149485111,
+            0.001407716190442443,
+            0.006566239986568689,
+            -0.0030434592626988888,
+            -0.010666067712008953,
+            -0.01883780211210251,
+            -0.0060310750268399715,
+            0.024571046233177185,
+            0.03439946472644806,
+            0.007743602618575096,
+            0.011745704337954521,
+            -0.011122121475636959,
+            -0.00967019610106945,
+            -0.006207912229001522,
+            -0.01917286217212677,
+            -0.009004729799926281,
+            -0.005863544996827841,
+            -0.00813915953040123,
+            0.011531638912856579,
+            -0.011568867601454258,
+            0.016538919880986214,
+            0.030118146911263466,
+            -0.031830672174692154,
+            -0.008530061691999435,
+            0.01570127159357071,
+            0.008651056326925755,
+            0.013253473676741123,
+            0.008804624900221825,
+            0.007631916087120771,
+            -0.011233807541429996,
+            -0.005817009136080742,
+            0.000602642132434994,
+            0.0062032584100961685,
+            -0.013802599161863327,
+            0.02162996679544449,
+            0.0010011072736233473,
+            0.002340764505788684,
+            0.004704797174781561,
+            0.01128965150564909,
+            -0.012760191224515438,
+            0.00357629731297493,
+            -0.007613301742821932,
+            -0.02537146769464016,
+            -0.026674477383494377,
+            -0.008134505711495876,
+            -0.006724462378770113,
+            -0.020680630579590797,
+            0.002073182025924325,
+            0.006333559285849333,
+            0.01950792223215103,
+            0.014724013395607471,
+            -0.009246718138456345,
+            -0.007250320166349411,
+            -0.024868877604603767,
+            0.010777754709124565,
+            -0.026376646012067795,
+            -0.02295159175992012,
+            0.027735499665141106,
+            0.013541997410356998,
+            0.014333110302686691,
+            -0.0016043310752138495,
+            0.0024315097834914923,
+            -0.010526459664106369,
+            0.013430310413241386,
+            0.02213255688548088,
+            0.004597763996571302,
+            0.007403889205306768,
+            -0.015105608850717545,
+            -0.002538542728871107,
+            0.020289728417992592,
+            -0.00022119177447166294,
+            -0.005002627614885569,
+            0.022486230358481407,
+            0.015422054566442966,
+            0.01886572316288948,
+            -0.004904902074486017,
+            0.015943259000778198,
+            0.01648307777941227,
+            0.008920964784920216,
+            -0.004106808453798294,
+            0.00020301362383179367,
+            -0.020997075363993645,
+            -0.004835098050534725,
+            0.03337567299604416,
+            0.012341366149485111,
+            0.025762369856238365,
+            -0.0263208020478487,
+            0.004825790878385305,
+            -0.004141710698604584,
+            0.013020792976021767,
+            0.01047061663120985,
+            0.019638223573565483,
+            0.028442848473787308,
+            0.009395632892847061,
+            0.009902875870466232,
+            -0.0012238987255841494,
+            0.029224654659628868,
+            -0.012611275538802147,
+            -0.009902875870466232,
+            -0.0006788449827581644,
+            -0.006622083019465208,
+            -0.004211514722555876,
+            0.00010339733853470534,
+            0.01476124208420515,
+            0.002815432380884886,
+            -0.019061176106333733,
+            0.014146965928375721,
+            0.007054868619889021,
+            -0.018819188699126244,
+            0.012127300724387169,
+            -0.02151828072965145,
+            0.011373416520655155,
+            -0.02738182619214058,
+            -0.0053656091913580894,
+            -0.01377467717975378,
+            -0.009358404204249382,
+            -0.023174963891506195,
+            -0.006933874916285276,
+            0.004034677520394325,
+            0.0082927281036973,
+            -0.0029317724984139204,
+            0.0043301815167069435,
+            -0.0030550931114703417,
+            -0.01600840874016285,
+            0.007929746992886066,
+            0.030937181785702705,
+            0.008716206066310406,
+            0.007450425066053867,
+            -0.0035530291497707367,
+            0.011224500834941864,
+            -0.008972154930233955,
+            -0.02213255688548088,
+            0.0002355888718739152,
+            -0.013467539101839066,
+            -0.008562637493014336,
+            0.013532689772546291,
+            0.006515050306916237,
+            -0.014202809892594814,
+            0.00905591994524002,
+            -0.024440744891762733,
+            -0.0002466411970090121,
+            -0.017786087468266487,
+            -0.00875343568623066,
+            -0.0053656091913580894,
+            -0.015021844767034054,
+            0.0013483826769515872,
+            0.007617955096065998,
+            -0.00890700425952673,
+            -0.006277716252952814,
+            0.00968881044536829,
+            -0.0037926898803561926,
+            -0.014333110302686691,
+            -0.0038880889769643545,
+            -0.008916311897337437,
+            -0.002316333120688796,
+            0.0018905280157923698,
+            -0.00985634047538042,
+            -0.005095699802041054,
+            -0.005407491698861122,
+            -0.026506947353482246,
+            0.006812881212681532,
+            0.019600993022322655,
+            0.004602417815476656,
+            0.020829545333981514,
+            0.0029573673382401466,
+            0.010526459664106369,
+            0.02931772544980049,
+            0.0055238320492208,
+            -0.008748781867325306,
+            -0.024887491017580032,
+            0.02159273810684681,
+            -0.010312393307685852,
+            0.024887491017580032,
+            -0.011447873897850513,
+            -0.014472718350589275,
+            0.0026548828464001417,
+            -0.00554244639351964,
+            -0.005267883650958538,
+            0.00035076565109193325,
+            0.006026421207934618,
+            -0.017497563734650612,
+            0.018009459599852562,
+            -0.011857391335070133,
+            0.008897697553038597,
+            0.010600917041301727,
+            0.041733551770448685,
+            -0.0013134806649759412,
+            -0.007375967688858509,
+            0.004462809301912785,
+            -0.001483337371610105,
+            0.01958237960934639,
+            -0.004497711546719074,
+            0.007282895501703024,
+            -0.009716732427477837,
+            -0.0007445771479979157,
+            -0.009265332482755184,
+            0.0062730624340474606,
+            0.015133530832827091,
+            -0.016771601513028145,
+            -0.01328139565885067,
+            0.0055052172392606735,
+            -0.026506947353482246,
+            0.0086138267070055,
+            0.030248448252677917,
+            0.017562713474035263,
+            -0.007324778009206057,
+            -0.014379646629095078,
+            0.007422503549605608,
+            -0.030080918222665787,
+            -0.0028480077162384987,
+            0.09649720042943954,
+            -0.009339789859950542,
+            -0.005128275137394667,
+            -0.02336110919713974,
+            -0.006524357479065657,
+            0.019228706136345863,
+            -0.014342417940497398,
+            0.07330362498760223,
+            -0.008748781867325306,
+            0.00046128881513141096,
+            -0.0073015098460018635,
+            0.001322787837125361,
+            -0.02053171582520008,
+            0.003978834021836519,
+            -0.02196502685546875,
+            -0.024868877604603767,
+            0.0021266986150294542,
+            -0.030118146911263466,
+            -0.007366660516709089,
+            -0.0034971858840435743,
+            0.008413721807301044,
+            -0.0049188630655407906,
+            0.011857391335070133,
+            -0.009865647181868553,
+            -0.012462359853088856,
+            -0.0011639834847301245,
+            0.005677401088178158,
+            0.015654735267162323,
+            -0.026264959946274757,
+            -0.01991743966937065,
+            0.010545074008405209,
+            0.01900533214211464,
+            0.004469790030270815,
+            -0.008609173819422722,
+            0.001268108026124537,
+            -0.00514223612844944,
+            0.008590559475123882,
+            0.029708629474043846,
+            0.01089874841272831,
+            0.01108489278703928,
+            0.0037531342823058367,
+            -0.011075585149228573,
+            0.015328982844948769,
+            0.007785485126078129,
+            0.013662991113960743,
+            -0.006021767854690552,
+            -0.004816483706235886,
+            -0.0021022669970989227,
+            0.009558509103953838,
+            -0.02220701426267624,
+            -0.014119044877588749,
+            -0.012238986790180206,
+            0.016967052593827248,
+            -0.006938528269529343,
+            -0.0005441812099888921,
+            -0.009791189804673195,
+            0.013579226098954678,
+            -0.008874429389834404,
+            -0.0051236217841506,
+            0.005184118635952473,
+            -0.021276293322443962,
+            0.022467615082859993,
+            0.006864070892333984,
+            0.021983640268445015,
+            -0.002318659797310829,
+            0.003573970403522253,
+            -0.014714706689119339,
+            -0.006580200511962175,
+            0.0106939896941185,
+            -0.004718757700175047,
+            0.00822292361408472,
+            0.002288411371409893,
+            0.02894543670117855,
+            0.0005770472926087677,
+            -0.013523383066058159,
+            0.008250845596194267,
+            -0.011587481945753098,
+            -0.01791638880968094,
+            0.010182092897593975,
+            -0.020382799208164215,
+            0.009060573764145374,
+            -6.11876675975509e-05,
+            -0.025445925071835518,
+            -0.010005255229771137,
+            0.004932823590934277,
+            -0.006924567744135857,
+            -0.013141786679625511,
+            0.013718834146857262,
+            -0.01853066496551037,
+            -0.01071260403841734,
+            -0.007194476667791605,
+            0.02101569063961506,
+            -0.016846058890223503,
+            0.023379722610116005,
+            -0.020680630579590797,
+            0.00634286692366004,
+            -0.00875343568623066,
+            0.02015942707657814,
+            -0.004453502129763365,
+            0.018195604905486107,
+            0.0122948307543993,
+            0.00674307718873024,
+            0.002838700544089079,
+            0.007469039876013994,
+            0.009698117151856422,
+            0.02581821382045746,
+            0.01982436701655388,
+            -0.003245891071856022,
+            -0.008534715510904789,
+            -0.011615403927862644,
+            0.013839827850461006,
+            -0.005793740972876549,
+            -0.00922810286283493,
+            0.011782933957874775,
+            0.016027024015784264,
+            -0.014947386458516121,
+            0.014854314737021923,
+            -0.0028666220605373383,
+            0.01086151972413063,
+            -0.013904978521168232,
+            0.004013736266642809,
+            0.0009057082934305072,
+            0.012369288131594658,
+            0.0075993407517671585,
+            -0.00016491221322212368,
+            0.01541274692863226,
+            0.02040141448378563,
+            -0.0004836842999793589,
+            0.01651099883019924,
+            0.0046210321597754955,
+            0.010377543978393078,
+            0.0021104109473526478,
+            -0.0037926898803561926,
+            -0.00514223612844944,
+            0.0021871954668313265,
+            -0.007892518304288387,
+            0.008786010555922985,
+            0.011056970804929733,
+            -0.010768447071313858,
+            0.002801471622660756,
+            0.012611275538802147,
+            -0.011913234367966652,
+            -0.008511447347700596,
+            0.0038275918923318386,
+            -0.026879236102104187,
+            -0.006850109901279211,
+            0.028089173138141632,
+            -0.018214218318462372,
+            -0.007273588329553604,
+            0.010033177211880684,
+            0.012425131164491177,
+            -0.0010011072736233473,
+            -0.014472718350589275,
+            0.009079188108444214,
+            0.005216693971306086,
+            -0.02557622641324997,
+            0.0024128954391926527,
+            0.011606096290051937,
+            0.018577199429273605,
+            -0.0004990993766114116,
+            -0.010768447071313858,
+            0.0013367487117648125,
+            0.025352852419018745,
+            0.016492383554577827,
+            0.013067329302430153,
+            -0.0050910464487969875,
+            -0.003990468103438616,
+            0.0051375823095440865,
+            0.0012064477195963264,
+            0.002601366490125656,
+            0.00668257987126708,
+            -0.0012983564520254731,
+            -0.0059054275043308735,
+            0.01348615437746048,
+            -0.0010458981851115823,
+            0.012322751805186272,
+            -0.013700219802558422,
+            -0.0186981949955225,
+            -0.0013134806649759412,
+            -0.003874127985909581,
+            -0.012387902475893497,
+            0.019470693543553352,
+            -0.003285446669906378,
+            0.03709855675697327,
+            0.013458232395350933,
+            0.005086392629891634,
+            -0.002601366490125656,
+            -0.008404415100812912,
+            0.016631992533802986,
+            0.0053656091913580894,
+            -0.01520798821002245,
+            0.003927644342184067,
+            -0.029168810695409775,
+            0.021071534603834152,
+            -0.009865647181868553,
+            0.010554381646215916,
+            -0.002633941825479269,
+            0.007906478829681873,
+            0.004876980558037758,
+            -0.011466488242149353,
+            -0.003373865270987153,
+            0.009567816741764545,
+            0.0022220974788069725,
+            -0.012127300724387169,
+            -0.0015612852293998003,
+            0.008678977377712727,
+            -0.006049689371138811,
+            0.012211065739393234,
+            0.005272537004202604,
+            -0.008520754985511303,
+            0.00436741067096591
+          ],
+          "title": "Conclusion"
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "1084fc41-4c97-458c-a6c9-2140fb99d5a9",
+        "properties": {
+          "page_content": "Appendices A. Overview Of Hyper-Parameters\n\nWe present the overview of hyper-parameters for DeepSeekMoE across various sizes in Table 7.\n\n# Params # Layers Hidden # Attn # Shared # Routed Relative Sequence Batch Size Learning Size Heads Experts Experts Expert Size Length (Sequence) Rate 2.0B 9 1280 10 1 63 (7 activated) 0.25 2048 2048 1.08e-3 16.4B 28 2048 16 2 64 (6 activated) 0.25 4096 4608 4.2e-4 144.6B 62 4096 32 4 128 (12 activated) 0.125 4096 4608 3.0e-4\n\nTable 7 | Overview of hyper-parameters for DeepSeekMoE across various sizes. The relative expert size is in comparison to a standard FFN.\n\nB. Comparing Deepseekmoe With Larger Models\n\nComparisons among DeepSeekMoE, GShard×1.2, and GShard×1.5 are shown in Table 8. Comparisons among DeepSeekMoE, Dense×4, and Dense×16 are shown in Table 9.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 2.3B 2.8B 1.9B # Activated Expert Params N/A 0.28B 0.35B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.824 1.808 1.808 HellaSwag (Acc.) 0-shot 53.7 54.4 54.8 PIQA (Acc.) 0-shot 71.8 71.1 72.3 ARC-easy (Acc.) 0-shot 46.8 47.3 49.4 ARC-challenge (Acc.) 0-shot 31.7 34.1 34.3 RACE-middle (Acc.) 5-shot 43.7 46.4 44.0 RACE-high (Acc.) 5-shot 31.9 32.4 31.7 HumanEval (Pass@1) 0-shot 3.7 3.0 4.9 MBPP (Pass@1) 3-shot 2.4 2.6 2.2 TriviaQA (EM) 5-shot 15.2 15.7 16.6 NaturalQuestions (EM) 5-shot 4.5 4.7 5.7\n\nTable 8 | Comparison between DeepSeekMoE and larger GShard models.\n\nAt a larger scale of 13B total parameters, we also compare DeepSeekMoE with GShard×1.2 and GShard×1.5, and show results in Table 10. At a larger scale, DeepSeekMoE even outperforms GShard×1.5 distinctly.\n\nMetric # Shot Dense×4 Dense×16 DeepSeekMoE Relative Expert Size N/A 1 1 0.25 # Experts N/A 4 + 0 16 + 0 1 + 63 # Activated Experts N/A 4 + 0 16 + 0 1 + 7 # Total Expert Params N/A 0.47B 1.89B 1.89B # Activated Expert Params N/A 0.47B 1.89B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.908 1.806 1.808 HellaSwag (Acc.) 0-shot 47.6 55.1 54.8 PIQA (Acc.) 0-shot 70.0 71.9 72.3 ARC-easy (Acc.) 0-shot 43.9 51.9 49.4 ARC-challenge (Acc.) 0-shot 30.5 33.8 34.3 RACE-middle (Acc.) 5-shot 42.4 46.3 44.0 RACE-high (Acc.) 5-shot 30.7 33.0 31.7 HumanEval (Pass@1) 0-shot 1.8 4.3 4.9 MBPP (Pass@1) 3-shot 0.2 2.2 2.2 TriviaQA (EM) 5-shot 9.9 16.5 16.6 NaturalQuestions (EM) 5-shot 3.0 6.3 5.7\n\nTable 9 | Comparison between DeepSeekMoE and larger dense baselines.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 15.9B 19.8B 13.3B # Activated Expert Params N/A 2.37B 2.82B 2.05B # Training Tokens N/A 100B 100B 100B HellaSwag (Acc.) 0-shot 66.6 67.7 69.1 PIQA (Acc.) 0-shot 75.6 76.0 75.7 ARC-easy (Acc.) 0-shot 56.8 56.8 58.8 ARC-challenge (Acc.) 0-shot 39.9 37.6 38.5 RACE-middle (Acc.) 5-shot 51.6 50.6 52.4 RACE-high (Acc.) 5-shot 37.4 36.3 38.5 HumanEval (Pass@1) 0-shot 6.1 6.1 9.8 MBPP (Pass@1) 3-shot 7.0 11.6 10.6 TriviaQA (EM) 5-shot 36.5 36.7 38.2 NaturalQuestions (EM) 5-shot 12.6 12.1 13.7\n\nTable 10 | Comparison between DeepSeekMoE and larger GShard models at a larger scale.\n\nC. Training Benchmark Curves Of Deepseekmoe 16B\n\nWe present the benchmark curves during training of DeepSeekMoE 16B and DeepSeek 7B (Dense) in Figure 7 for reference.",
+          "keyphrases": [
+            "DeepSeekMoE",
+            "hyper-parameters",
+            "GShard models",
+            "Dense baselines",
+            "training benchmark curves"
+          ],
+          "embedding": [
+            0.020979074761271477,
+            -0.0013986049452796578,
+            -0.019327709451317787,
+            0.0138933714479208,
+            0.0006998290773481131,
+            -0.02645554021000862,
+            0.0001438888575648889,
+            0.05624751001596451,
+            -0.006693925708532333,
+            0.024147000163793564,
+            -0.016437821090221405,
+            -0.0021316087804734707,
+            0.04061009734869003,
+            0.005270044784992933,
+            0.00756173487752676,
+            0.03029749169945717,
+            0.026859955862164497,
+            0.02480417490005493,
+            0.0019567832350730896,
+            -0.041183020919561386,
+            0.017339332029223442,
+            -0.04290178790688515,
+            -0.025309694930911064,
+            0.008383204229176044,
+            -0.016185062006115913,
+            0.009588027372956276,
+            0.022377679124474525,
+            0.014769605360925198,
+            -0.037071458995342255,
+            0.022091219201683998,
+            0.01640412025153637,
+            0.023371869698166847,
+            0.0032205828465521336,
+            -0.005885093938559294,
+            -0.02074316516518593,
+            -0.004515977576375008,
+            0.06709934026002884,
+            0.027870995923876762,
+            -0.010388433001935482,
+            0.018485177308321,
+            0.029404405504465103,
+            0.011500577442348003,
+            -0.013918647542595863,
+            -0.00502149760723114,
+            0.005880881100893021,
+            -0.009082507342100143,
+            -0.018165014684200287,
+            0.014019750989973545,
+            0.011744911782443523,
+            -0.05675303190946579,
+            -0.0348808728158474,
+            -0.017726896330714226,
+            -0.008109381422400475,
+            -0.013952348381280899,
+            -0.0411493182182312,
+            -0.03166239336133003,
+            0.028376515954732895,
+            0.061841931194067,
+            0.04182334616780281,
+            0.008939276449382305,
+            0.02399534359574318,
+            -0.006129428278654814,
+            -0.03993606939911842,
+            0.0007677582907490432,
+            0.008855023421347141,
+            -0.015452057123184204,
+            0.022579887881875038,
+            -0.005105750635266304,
+            -0.006647586356848478,
+            -0.020608359947800636,
+            -0.037745483219623566,
+            0.022377679124474525,
+            -0.003243752522394061,
+            0.006015686318278313,
+            0.02106332778930664,
+            -0.01328674703836441,
+            0.027803592383861542,
+            0.02632073499262333,
+            0.00384195102378726,
+            -0.013918647542595863,
+            0.017726896330714226,
+            0.0043074507266283035,
+            -0.01196397002786398,
+            0.03747587278485298,
+            0.04842880368232727,
+            -0.026287034153938293,
+            -0.040374185889959335,
+            -0.06318998336791992,
+            0.010321030393242836,
+            -0.008244186639785767,
+            -0.018384072929620743,
+            -0.008678090758621693,
+            0.03508307784795761,
+            -0.02180475741624832,
+            -0.03354967013001442,
+            0.06864959746599197,
+            -0.012494766153395176,
+            0.007115192245692015,
+            -0.03442590311169624,
+            -0.0069087715819478035,
+            0.004235835280269384,
+            0.00045523117296397686,
+            0.008661240339279175,
+            0.0043685343116521835,
+            0.005177366081625223,
+            -0.00362710515037179,
+            -0.01878848858177662,
+            0.0002004965499509126,
+            -0.004111561458557844,
+            0.0718512237071991,
+            0.012410513125360012,
+            -0.01227570790797472,
+            -0.048327699303627014,
+            -0.013927072286605835,
+            -0.02032189816236496,
+            -0.038655418902635574,
+            -0.029775120317935944,
+            -0.021568847820162773,
+            0.024281805381178856,
+            -0.002023132750764489,
+            0.033263206481933594,
+            0.020642060786485672,
+            0.030044730752706528,
+            -0.01792910508811474,
+            -0.043070293962955475,
+            0.04421613737940788,
+            0.0006208415725268424,
+            -0.0033933022059500217,
+            -0.0019631022587418556,
+            -0.029842523857951164,
+            0.02118128351867199,
+            -0.0036713380832225084,
+            -0.03408889099955559,
+            -0.030954666435718536,
+            0.00232539139688015,
+            -0.008694942109286785,
+            0.05213594809174538,
+            0.03949795290827751,
+            -0.01417140755802393,
+            -0.0003188461414538324,
+            0.0034438541624695063,
+            -0.04034048691391945,
+            0.029337003827095032,
+            -0.017246652394533157,
+            -0.03181404992938042,
+            0.022327126935124397,
+            0.017895404249429703,
+            0.06558278203010559,
+            -0.0028098479378968477,
+            0.041351526975631714,
+            0.0015386760933324695,
+            0.04003717377781868,
+            0.02350667491555214,
+            0.015763795003294945,
+            -0.009074081666767597,
+            -0.004326407797634602,
+            -0.03693665191531181,
+            0.00613785395398736,
+            -0.03360022231936455,
+            -0.015098193660378456,
+            -0.001018938492052257,
+            -0.004907755646854639,
+            -0.017288779839873314,
+            -0.0007556469063274562,
+            -0.002022079424932599,
+            0.047788478434085846,
+            0.023355018347501755,
+            -0.013244620524346828,
+            -0.06804297864437103,
+            0.03263973444700241,
+            0.04620451480150223,
+            0.007477481383830309,
+            -0.006196831353008747,
+            0.03788029029965401,
+            0.003601829055696726,
+            0.02229342609643936,
+            0.020709464326500893,
+            0.01904124952852726,
+            -0.06531316787004471,
+            0.04522717744112015,
+            -0.015300401486456394,
+            0.003485980909317732,
+            0.01658105105161667,
+            0.011264667846262455,
+            0.0310389194637537,
+            -0.0018630513222888112,
+            0.029033690690994263,
+            0.026859955862164497,
+            -0.012604295276105404,
+            0.02136664092540741,
+            0.031932003796100616,
+            0.008248399011790752,
+            0.04553049057722092,
+            -0.001733511919155717,
+            0.0005845073610544205,
+            -0.018687384203076363,
+            0.008838172070682049,
+            0.03353281691670418,
+            -0.017423585057258606,
+            0.018215566873550415,
+            0.0006161023629829288,
+            -0.027685638517141342,
+            -0.0010357891442254186,
+            -0.013337299227714539,
+            0.027651937678456306,
+            0.0262701828032732,
+            -0.008509584702551365,
+            0.0066855004988610744,
+            0.008404267951846123,
+            -0.016067106276750565,
+            -0.031190576031804085,
+            -0.013935497961938381,
+            -0.017069721594452858,
+            -0.0532480925321579,
+            -0.0025634069461375475,
+            0.03521788492798805,
+            -0.007785005960613489,
+            0.022512484341859818,
+            0.010860252194106579,
+            -0.0323195718228817,
+            0.003799824509769678,
+            0.009933465160429478,
+            0.014179832302033901,
+            0.006390613969415426,
+            -0.017229802906513214,
+            -0.02350667491555214,
+            -0.0032016257755458355,
+            0.008753919042646885,
+            0.032841939479112625,
+            -0.010919229127466679,
+            0.001601866097189486,
+            -0.001936773071065545,
+            -0.01959731988608837,
+            0.0043348330073058605,
+            0.03257232904434204,
+            -0.01013567391782999,
+            -0.00016798004799056798,
+            0.007005663122981787,
+            0.02662404626607895,
+            0.018485177308321,
+            0.01279807835817337,
+            -0.015266700647771358,
+            -0.04192444682121277,
+            -0.016564201563596725,
+            0.01824926771223545,
+            -0.00505098607391119,
+            -0.026742001995444298,
+            -0.01732248067855835,
+            0.057629264891147614,
+            0.024197550490498543,
+            0.023304466158151627,
+            -0.06281927227973938,
+            -0.02283264696598053,
+            0.005535442847758532,
+            -0.01727192848920822,
+            0.05668563023209572,
+            0.059415437281131744,
+            0.005628121085464954,
+            -0.00452861562371254,
+            -0.04121671989560127,
+            0.04023938253521919,
+            -0.01184601616114378,
+            -0.049979064613580704,
+            0.018704235553741455,
+            -0.01627773977816105,
+            0.01627773977816105,
+            0.033633921295404434,
+            0.027685638517141342,
+            0.0044443621300160885,
+            0.008231548592448235,
+            0.04266587644815445,
+            -0.023152809590101242,
+            -0.0458001010119915,
+            -0.02374258264899254,
+            0.021349789574742317,
+            -0.018721086904406548,
+            -0.007696540094912052,
+            -0.02852817252278328,
+            0.02633758634328842,
+            0.05055198818445206,
+            0.02662404626607895,
+            0.018350372090935707,
+            0.016142934560775757,
+            0.025056935846805573,
+            -0.024096447974443436,
+            0.0011490045581012964,
+            -0.023675180971622467,
+            -0.043306201696395874,
+            -0.017710046842694283,
+            -0.015199298039078712,
+            0.0068076676689088345,
+            0.006015686318278313,
+            -0.022849498316645622,
+            0.005493315868079662,
+            0.02606797404587269,
+            0.014407316222786903,
+            -0.03140963613986969,
+            0.011702785268425941,
+            0.05119231343269348,
+            -0.035116780549287796,
+            -0.03194885700941086,
+            0.0317297987639904,
+            0.017844852060079575,
+            -0.0067992424592375755,
+            -0.01885589212179184,
+            0.01466007623821497,
+            -9.174066690320615e-06,
+            0.01113828830420971,
+            -0.009284715168178082,
+            0.0007667051395401359,
+            -0.011938693933188915,
+            -0.0016303015872836113,
+            0.007801856845617294,
+            -0.03386983275413513,
+            -0.008905574679374695,
+            0.019176054745912552,
+            0.022124920040369034,
+            0.034510158002376556,
+            -0.06474024802446365,
+            -0.01166065875440836,
+            0.06969434022903442,
+            0.01836722157895565,
+            -0.04610341414809227,
+            0.025478200986981392,
+            0.019799528643488884,
+            -0.007351101376116276,
+            0.004520190414041281,
+            0.04559789225459099,
+            -0.04684484004974365,
+            -0.014002900570631027,
+            0.029050542041659355,
+            -0.04040788859128952,
+            -0.03139278292655945,
+            0.01892329379916191,
+            0.025023233145475388,
+            -0.06376291066408157,
+            -0.03761067986488342,
+            0.005404850002378225,
+            0.012258856557309628,
+            -0.007397440727800131,
+            -0.021956413984298706,
+            0.0029320153407752514,
+            -0.036094117909669876,
+            -0.01426408626139164,
+            0.0003749272436834872,
+            -0.005135239567607641,
+            -0.028545022010803223,
+            0.002346454653888941,
+            0.020793717354536057,
+            0.01627773977816105,
+            0.00159554707352072,
+            0.012191453948616982,
+            0.03424054756760597,
+            -0.006175767630338669,
+            0.0027550833765417337,
+            -0.028443917632102966,
+            -0.02645554021000862,
+            0.01579749584197998,
+            0.0032079447992146015,
+            0.00201260088942945,
+            -0.017162399366497993,
+            -0.04377802088856697,
+            -0.03366762399673462,
+            0.028545022010803223,
+            -0.020557807758450508,
+            -0.013623761013150215,
+            0.021821608766913414,
+            -0.004482276272028685,
+            -0.011862866580486298,
+            0.012292558327317238,
+            0.002847762079909444,
+            0.030819861218333244,
+            -0.03619522228837013,
+            0.006053600460290909,
+            0.017844852060079575,
+            -0.001491283648647368,
+            0.035116780549287796,
+            -0.010422134771943092,
+            -0.02392794005572796,
+            0.021838458254933357,
+            -0.049069128930568695,
+            0.024871578440070152,
+            0.03848691284656525,
+            0.012831779196858406,
+            0.03066820651292801,
+            0.031055770814418793,
+            -0.009326841682195663,
+            0.011610106565058231,
+            -0.012385237030684948,
+            -0.0024496649857610464,
+            0.033768728375434875,
+            -0.059482838958501816,
+            -0.023034855723381042,
+            0.010801274329423904,
+            0.018872741609811783,
+            0.03943055123090744,
+            0.017221376299858093,
+            -0.010834976099431515,
+            -0.022546187043190002,
+            -0.008138869889080524,
+            0.013640611432492733,
+            0.04927133768796921,
+            -0.0011468982556834817,
+            -0.003104734467342496,
+            -0.005493315868079662,
+            0.026371287181973457,
+            0.006803454831242561,
+            0.012233581393957138,
+            0.023708881810307503,
+            0.0004207399906590581,
+            0.022074367851018906,
+            -0.0037366345059126616,
+            -0.0342910997569561,
+            -0.008581199683248997,
+            0.007616499438881874,
+            0.03515048325061798,
+            -0.0021610974799841642,
+            0.021097028627991676,
+            -0.02473677322268486,
+            0.01978267729282379,
+            -0.039767563343048096,
+            0.020490406081080437,
+            0.031679246574640274,
+            -0.0028561872895807028,
+            -0.0045328279957175255,
+            0.01227570790797472,
+            -0.015157170593738556,
+            0.0019136033952236176,
+            0.029185347259044647,
+            -0.03471236303448677,
+            0.014778031036257744,
+            0.012115626595914364,
+            0.02945495769381523,
+            -0.04320509731769562,
+            -0.022984303534030914,
+            -0.013775416649878025,
+            -0.011685934849083424,
+            0.04320509731769562,
+            -0.04023938253521919,
+            -0.02259673923254013,
+            -0.0018862209981307387,
+            0.0372062623500824,
+            0.019850080832839012,
+            -0.013177217915654182,
+            -0.04327250272035599,
+            -0.033515967428684235,
+            -0.05008016899228096,
+            -0.012284132651984692,
+            -0.03922834247350693,
+            0.002550769131630659,
+            -0.07731083780527115,
+            0.01941196247935295,
+            -0.050113871693611145,
+            0.020642060786485672,
+            0.046811141073703766,
+            -0.008572774939239025,
+            0.030836712568998337,
+            -0.0196478720754385,
+            0.014205108396708965,
+            -0.06251595914363861,
+            -0.0068076676689088345,
+            0.025747811421751976,
+            0.02945495769381523,
+            -0.003494406118988991,
+            -0.003734528087079525,
+            0.019445665180683136,
+            -0.01279807835817337,
+            -0.0065675457008183,
+            -0.008930850774049759,
+            0.02608482539653778,
+            0.008496946655213833,
+            -0.010674894787371159,
+            0.024601968005299568,
+            0.008265250362455845,
+            0.015502609312534332,
+            0.03761067986488342,
+            -0.011365772224962711,
+            -0.04023938253521919,
+            -0.0034417477436363697,
+            0.029117945581674576,
+            0.029707718640565872,
+            -0.031072622165083885,
+            0.014339913614094257,
+            -0.003869333304464817,
+            0.005906157195568085,
+            0.006512781139463186,
+            0.004033627454191446,
+            0.029758270829916,
+            -0.006394826341420412,
+            0.026708299294114113,
+            -0.014811732806265354,
+            0.019850080832839012,
+            -0.026421839371323586,
+            0.03147703781723976,
+            0.012132477015256882,
+            -0.030162686482071877,
+            0.005910370033234358,
+            -0.01628616452217102,
+            0.016918065026402473,
+            0.019361412152647972,
+            -0.013143517076969147,
+            -0.027500281110405922,
+            0.013446828350424767,
+            0.013480530120432377,
+            0.012225155718624592,
+            0.004494914319366217,
+            0.027399176731705666,
+            0.03006158210337162,
+            -0.02227657660841942,
+            -0.03545379266142845,
+            0.0174404364079237,
+            -0.021872159093618393,
+            0.015039216727018356,
+            -0.03427424654364586,
+            0.01426408626139164,
+            0.00647486699745059,
+            -0.03595931455492973,
+            0.04010457545518875,
+            0.020642060786485672,
+            0.0019451983971521258,
+            0.004128412343561649,
+            -0.005990410689264536,
+            -0.009326841682195663,
+            0.05129341781139374,
+            -0.015957577154040337,
+            0.019816379994153976,
+            0.015637414529919624,
+            0.012882331386208534,
+            0.03683554753661156,
+            -0.03322950750589371,
+            -0.014575823210179806,
+            0.01347210444509983,
+            0.014499994926154613,
+            -0.03445960581302643,
+            -0.01365746185183525,
+            -0.02148459479212761,
+            -0.011450025252997875,
+            0.040374185889959335,
+            -0.014525271020829678,
+            -0.02131608873605728,
+            0.007658625952899456,
+            0.0023359230253845453,
+            -0.016934916377067566,
+            0.008092531003057957,
+            -0.015047641471028328,
+            -0.005194216500967741,
+            0.04064379632472992,
+            -0.023978492245078087,
+            0.03622892498970032,
+            0.031561292707920074,
+            0.016859088093042374,
+            -0.042699579149484634,
+            0.0006656011682935059,
+            0.006112577859312296,
+            -0.014188257977366447,
+            -0.01597442850470543,
+            0.034442752599716187,
+            0.014887560158967972,
+            0.01450842060148716,
+            -0.023152809590101242,
+            -0.006205256562680006,
+            0.02362462878227234,
+            -0.023304466158151627,
+            0.02578151412308216,
+            -0.00396411819383502,
+            -0.02332131750881672,
+            -0.03213421255350113,
+            0.007650200743228197,
+            -0.022680992260575294,
+            0.012755951844155788,
+            -0.003985181916505098,
+            0.012292558327317238,
+            0.01236838661134243,
+            -0.025259142741560936,
+            0.0036587000358849764,
+            0.019209755584597588,
+            0.000130460990476422,
+            -0.0003262182872276753,
+            -0.009048805572092533,
+            0.004604443442076445,
+            0.018198715522885323,
+            -0.001285916194319725,
+            0.005215280223637819,
+            0.022006966173648834,
+            0.031005218625068665,
+            0.004381172358989716,
+            -0.003250071546062827,
+            -0.020793717354536057,
+            0.02797210030257702,
+            -0.02357407659292221,
+            0.02773619070649147,
+            0.028174307197332382,
+            0.014921261928975582,
+            -0.03855431452393532,
+            -0.0021800545509904623,
+            -0.016496798023581505,
+            -0.008353715762495995,
+            0.01776059903204441,
+            0.022883199155330658,
+            -0.024281805381178856,
+            -0.0024075384717434645,
+            -0.004147369414567947,
+            0.0215014461427927,
+            0.003218476427718997,
+            -0.030769309028983116,
+            -0.010843401774764061,
+            0.009866062551736832,
+            0.00896455254405737,
+            -0.01456739753484726,
+            0.0019831124227494,
+            0.011888142675161362,
+            0.006150492001324892,
+            -0.0016060788184404373,
+            -0.00011077993985963985,
+            0.01897384598851204,
+            -0.002618171740323305,
+            0.01567954197525978,
+            -0.013750140555202961,
+            -0.013446828350424767,
+            -0.011087736114859581,
+            0.05247296392917633,
+            -0.0032205828465521336,
+            0.03899243474006653,
+            0.014188257977366447,
+            0.009722832590341568,
+            0.04431724175810814,
+            -0.013253046199679375,
+            -0.009309991262853146,
+            0.01721295155584812,
+            -0.01750783808529377,
+            -0.03568970412015915,
+            0.004103136248886585,
+            -0.005025709979236126,
+            -0.028983140364289284,
+            0.01071702130138874,
+            0.005038348026573658,
+            -0.004067328758537769,
+            0.0009341586264781654,
+            0.008846597746014595,
+            -0.026843104511499405,
+            0.03247122839093208,
+            0.005944071337580681,
+            0.011879716999828815,
+            0.034746065735816956,
+            -0.016201911494135857,
+            -0.007974576205015182,
+            0.013792267069220543,
+            -0.01518244668841362,
+            0.028545022010803223,
+            -0.01347210444509983,
+            0.01047268696129322,
+            0.01707814633846283,
+            0.030381744727492332,
+            0.03622892498970032,
+            -0.02492213062942028,
+            -0.06261706352233887,
+            0.011711210012435913,
+            -0.004638144746422768,
+            0.011559554375708103,
+            0.016260888427495956,
+            0.01233468484133482,
+            -0.005640759132802486,
+            0.01811446249485016,
+            -0.007540671620517969,
+            0.028999989852309227,
+            -0.006121003068983555,
+            -0.0326734334230423,
+            0.020962223410606384,
+            0.006007261108607054,
+            0.023034855723381042,
+            0.014339913614094257,
+            -0.03895873203873634,
+            -0.009596452116966248,
+            -0.004596018232405186,
+            0.010270479135215282,
+            -0.03383613005280495,
+            0.00483614020049572,
+            0.010759147815406322,
+            0.017912253737449646,
+            0.002041036495938897,
+            -0.007595436181873083,
+            0.010556939989328384,
+            -0.005227918270975351,
+            -0.016303015872836113,
+            0.0131350914016366,
+            -0.0022664139978587627,
+            -0.013059263117611408,
+            0.026303883641958237,
+            -0.03545379266142845,
+            -0.016993893310427666,
+            -4.235703818267211e-05,
+            0.0013027668464928865,
+            0.013244620524346828,
+            0.003953586798161268,
+            0.0015492078382521868,
+            -0.02896628901362419,
+            -0.03848691284656525,
+            0.04475535824894905,
+            0.022967452183365822,
+            -0.0033153677359223366,
+            -0.0003277980431448668,
+            -0.0004360108869150281,
+            -0.005792415235191584,
+            0.023894239217042923,
+            0.03164554387331009,
+            -0.020658912137150764,
+            0.005004646722227335,
+            -0.027466580271720886,
+            -0.019698424264788628,
+            0.015283551067113876,
+            0.009031955152750015,
+            0.033701326698064804,
+            0.006217894610017538,
+            0.006912984419614077,
+            0.018384072929620743,
+            0.008610688149929047,
+            -0.004507552366703749,
+            -0.003808249719440937,
+            -0.022697841748595238,
+            0.03230271860957146,
+            0.0390598364174366,
+            0.0149381123483181,
+            0.014070303179323673,
+            -0.034746065735816956,
+            -0.003814568743109703,
+            0.010885528288781643,
+            0.02492213062942028,
+            0.02424810267984867,
+            0.006748690269887447,
+            0.001917816000059247,
+            0.015157170593738556,
+            -0.03703775629401207,
+            -0.013573208823800087,
+            0.009798659943044186,
+            -0.014129280112683773,
+            -0.027988949790596962,
+            -0.006087301764637232,
+            0.022916901856660843,
+            -0.005897731985896826,
+            0.0021526720374822617,
+            -0.004659208003431559,
+            -0.011542703956365585,
+            -0.015527885407209396,
+            -0.004541253671050072,
+            0.006693925708532333,
+            0.02380998618900776,
+            0.0044317240826785564,
+            0.024770474061369896,
+            -0.009343692101538181,
+            0.011087736114859581,
+            0.006546482443809509,
+            -0.014162981882691383,
+            0.016488373279571533,
+            -0.013059263117611408,
+            -0.026708299294114113,
+            -0.0092425886541605,
+            0.01781114935874939,
+            -0.00644537853077054,
+            -0.0016945448005571961,
+            -0.01647152192890644,
+            0.005004646722227335,
+            0.036397431045770645,
+            0.008265250362455845,
+            0.017558390274643898,
+            0.023523524403572083,
+            -0.002397006843239069,
+            -0.009318416006863117,
+            0.0131350914016366,
+            0.0055902074091136456,
+            0.0055859945714473724,
+            0.021585699170827866,
+            -0.001512346905656159,
+            0.04387912526726723,
+            -0.05058568716049194,
+            0.008383204229176044,
+            0.027803592383861542,
+            -0.0057081617414951324,
+            -0.005939858499914408,
+            -0.032841939479112625,
+            0.01653892546892166,
+            -0.01678325980901718,
+            0.009798659943044186,
+            -0.011155138723552227,
+            0.028427068144083023,
+            -0.03723996505141258,
+            -0.008378991857171059,
+            -3.804563675657846e-05,
+            0.00811359379440546,
+            0.016083957627415657,
+            0.018653683364391327,
+            -0.006580183748155832,
+            -0.02497268095612526,
+            -0.003673444502055645,
+            0.012882331386208534,
+            0.04290178790688515,
+            -0.003953586798161268,
+            -0.014685352332890034,
+            0.0215014461427927,
+            -0.011171989142894745,
+            -0.020119691267609596,
+            0.0027635085862129927,
+            -0.024821026250720024,
+            0.029471809044480324,
+            0.03612782061100006,
+            -0.041958149522542953,
+            -0.024450311437249184,
+            0.002029451774433255,
+            -0.010936079546809196,
+            0.002504429779946804,
+            0.021012775599956512,
+            0.02743287943303585,
+            -0.004170538857579231,
+            -0.007629137486219406,
+            0.0206926129758358,
+            -0.005316384136676788,
+            0.015949152410030365,
+            0.016665305942296982,
+            -0.011348920874297619,
+            -0.0069551109336316586,
+            -0.017844852060079575,
+            -0.03270713612437248,
+            -0.00037361079012043774,
+            -0.0186031311750412,
+            -0.03481346741318703,
+            -0.0346449613571167,
+            -0.01398605015128851,
+            -0.010278903879225254,
+            -0.003949373960494995,
+            0.0364985354244709,
+            0.02394479140639305,
+            0.0251074880361557,
+            0.015746943652629852,
+            -0.013278322294354439,
+            0.05405692383646965,
+            -0.006816092878580093,
+            0.0460360087454319,
+            -0.020187092944979668,
+            0.0008951914496719837,
+            -0.019816379994153976,
+            0.03226901963353157,
+            -0.014390465803444386,
+            0.011525853537023067,
+            -0.009899764321744442,
+            0.006533844396471977,
+            0.0009504826739430428,
+            0.007574372924864292,
+            -0.0019451983971521258,
+            -0.009419520385563374,
+            0.027079014107584953,
+            -0.0023274978157132864,
+            0.029471809044480324,
+            0.020406153053045273,
+            0.004600231070071459,
+            0.005076262168586254,
+            -0.029707718640565872,
+            -0.00035149429459124804,
+            0.005067836958914995,
+            -0.0066855004988610744,
+            -0.018148163333535194,
+            0.0024201765190809965,
+            -0.007001450285315514,
+            0.007999852299690247,
+            -0.0039051410276442766,
+            -0.009225737303495407,
+            0.014104004949331284,
+            0.025629857555031776,
+            0.0069845994003117085,
+            -0.04391282796859741,
+            0.0035702341701835394,
+            -0.03936314582824707,
+            0.002685574349015951,
+            0.010818125680088997,
+            0.010093546472489834,
+            -0.002889888593927026,
+            0.0057123745791614056,
+            0.009613302536308765,
+            -0.004202133975923061,
+            -0.020962223410606384,
+            0.002352773677557707,
+            -0.015047641471028328,
+            -0.006146279163658619,
+            -0.011281518265604973,
+            3.797981480602175e-05,
+            -0.003614467103034258,
+            -0.0019262413261458278,
+            -0.005986197851598263,
+            -0.0067318398505449295,
+            -0.027146417647600174,
+            -0.02822485938668251,
+            -0.006108365021646023,
+            0.0002985726750921458,
+            -0.00706885289400816,
+            -0.03024693951010704,
+            0.03134223073720932,
+            -0.008539073169231415,
+            0.007136255502700806,
+            -0.0093015655875206,
+            0.009882913902401924,
+            -0.002344348467886448,
+            -0.005960921756923199,
+            -0.004553891718387604,
+            -0.026421839371323586,
+            -0.013994474895298481,
+            -0.03895873203873634,
+            0.016741132363677025,
+            -0.008471670560538769,
+            0.007106767036020756,
+            -0.010506387799978256,
+            0.001101612113416195,
+            -0.018148163333535194,
+            -0.0009278396028093994,
+            -0.002559194341301918,
+            0.027651937678456306,
+            -0.024163849651813507,
+            0.009074081666767597,
+            0.01616821065545082,
+            0.015485758893191814,
+            -0.0346449613571167,
+            0.02008599042892456,
+            0.012191453948616982,
+            -0.019243456423282623,
+            -0.0025423436891287565,
+            0.018350372090935707,
+            0.02576466277241707,
+            -0.013901796191930771,
+            -0.014162981882691383,
+            -0.004326407797634602,
+            0.009630153886973858,
+            -0.025006383657455444,
+            0.00954589992761612,
+            -0.010203076526522636,
+            0.017356181517243385,
+            0.025174889713525772,
+            -0.011256242170929909,
+            0.04023938253521919,
+            -0.03501567617058754,
+            -0.012831779196858406,
+            0.013497380539774895,
+            -0.023675180971622467,
+            0.013177217915654182,
+            -0.0003612360742408782,
+            -0.016690582036972046,
+            0.021754205226898193,
+            -0.006428527645766735,
+            0.01841777376830578,
+            -0.03693665191531181,
+            0.030078433454036713,
+            0.03012898378074169,
+            0.006656011566519737,
+            -0.011079310439527035,
+            -0.019513066858053207,
+            -0.01056536566466093,
+            0.0057418630458414555,
+            -0.0015397293027490377,
+            0.025730961933732033,
+            -0.014331488870084286,
+            -0.02466936968266964,
+            0.004633932374417782,
+            -0.0030478634871542454,
+            0.004692909307777882,
+            -0.02643868885934353,
+            0.004105242434889078,
+            -0.009832361713051796,
+            -0.0017924892017617822,
+            0.0006161023629829288,
+            -0.005990410689264536,
+            -0.009200462140142918,
+            -0.028696678578853607,
+            -0.056416019797325134,
+            0.0014891773462295532,
+            -0.0037913990672677755,
+            -0.023725733160972595,
+            -0.017743747681379318,
+            -0.012402087450027466,
+            -0.014449442736804485,
+            0.018518878147006035,
+            -0.0017208738718181849,
+            -0.023793134838342667,
+            -0.011862866580486298,
+            0.002586576621979475,
+            0.0017630006186664104,
+            0.0024412397760897875,
+            0.02362462878227234,
+            0.011163563467562199,
+            0.031493887305259705,
+            -0.018097611144185066,
+            0.02694420889019966,
+            -0.017238227650523186,
+            -0.03831840679049492,
+            0.03980126604437828,
+            -0.0009425839525647461,
+            -0.022697841748595238,
+            0.010801274329423904,
+            0.0009594346047379076,
+            -0.006483292207121849,
+            0.044115033000707626,
+            -0.0205746591091156,
+            -0.013429977931082249,
+            0.010177800431847572,
+            -0.006441165693104267,
+            -0.016631603240966797,
+            -0.03299359604716301,
+            -0.015393080189824104,
+            0.03240382298827171,
+            -0.006373763084411621,
+            0.013783842325210571,
+            0.035487495362758636,
+            -0.023776285350322723,
+            -0.029353853315114975,
+            0.012073500081896782,
+            -0.020675763487815857,
+            -0.01615135930478573,
+            0.0027782530523836613,
+            0.018636832013726234,
+            0.012098775245249271,
+            -0.0012248324928805232,
+            0.0006171555141918361,
+            -0.004945669323205948,
+            -0.012250431813299656,
+            0.003391195787116885,
+            -0.023776285350322723,
+            -0.035656001418828964,
+            -0.024012194946408272,
+            -0.01162695698440075,
+            0.03280824050307274,
+            0.03457755967974663,
+            -0.00030146888457238674,
+            -0.022495634853839874,
+            -0.02204066701233387,
+            -0.0029256963171064854,
+            0.039329446852207184,
+            0.013758566230535507,
+            0.012410513125360012,
+            -0.005333234556019306,
+            0.010396858677268028,
+            0.0005781883373856544,
+            0.006281084381043911,
+            -0.039093535393476486,
+            -0.008197847753763199,
+            0.019058099016547203,
+            0.001322777010500431,
+            -0.01629459112882614,
+            0.018771637231111526,
+            0.00517315324395895,
+            -0.013497380539774895,
+            -0.022259725257754326,
+            0.018518878147006035,
+            0.013674312271177769,
+            0.008004064671695232,
+            0.0044570001773536205,
+            -0.01873793639242649,
+            0.0005866136634722352,
+            0.009192036464810371,
+            -0.01285705529153347,
+            -0.0011142500443384051,
+            0.021400341764092445,
+            0.012739100493490696,
+            -0.008383204229176044,
+            0.012452639639377594,
+            0.013969198800623417,
+            -0.007115192245692015,
+            -0.008172571659088135,
+            8.287103264592588e-05,
+            0.014778031036257744,
+            -0.007148893550038338,
+            -2.983422928082291e-05,
+            -0.011525853537023067,
+            0.010700170882046223,
+            0.004709760192781687,
+            0.011778613552451134,
+            -0.02443346008658409,
+            -0.006592821795493364,
+            0.0055902074091136456,
+            0.02160254865884781,
+            -0.0079324496909976,
+            -0.013708014041185379,
+            0.009908189997076988,
+            -0.024450311437249184,
+            -0.02185530960559845,
+            0.00659703416749835,
+            0.0009615409071557224,
+            -0.009402669966220856,
+            0.033751875162124634,
+            0.0323195718228817,
+            -0.0010084068635478616,
+            -0.0011511108605191112,
+            0.00995031651109457,
+            0.008180996403098106,
+            0.017608942463994026,
+            -0.022883199155330658,
+            -0.006003048736602068,
+            -0.0013817542931064963,
+            0.016067106276750565,
+            0.04856361076235771,
+            -0.005826116539537907,
+            -0.0022011178079992533,
+            0.015898600220680237,
+            0.0920720174908638,
+            -0.00039783361717127264,
+            0.0067866044119000435,
+            -0.02394479140639305,
+            -0.030769309028983116,
+            0.005017284769564867,
+            0.001989431446418166,
+            0.019058099016547203,
+            0.05786517634987831,
+            0.003043650882318616,
+            0.0014407316921278834,
+            -0.0022179684601724148,
+            0.003943054936826229,
+            -0.0085432855412364,
+            -0.022512484341859818,
+            0.015839623287320137,
+            -0.0067866044119000435,
+            0.010683320462703705,
+            -6.480264710262418e-05,
+            0.01659790240228176,
+            0.004101030062884092,
+            0.008193634450435638,
+            0.006146279163658619,
+            0.017406733706593513,
+            -0.021535146981477737,
+            0.026421839371323586,
+            -0.010219926945865154,
+            0.011525853537023067,
+            -0.014702202752232552,
+            -0.027517132461071014,
+            0.019799528643488884,
+            -4.670134876505472e-05,
+            -0.007490119431167841,
+            -0.004002032335847616,
+            -0.011348920874297619,
+            0.02606797404587269,
+            -0.021400341764092445,
+            0.022849498316645622,
+            0.01261272095143795,
+            -0.020119691267609596,
+            0.010497963055968285,
+            0.008585412986576557,
+            0.0010257840622216463,
+            0.015763795003294945,
+            -0.008821321651339531,
+            0.0161008071154356,
+            0.01242736354470253,
+            -0.027112716808915138,
+            0.009958741255104542,
+            0.014550547115504742,
+            -0.009461646899580956,
+            0.01546048279851675,
+            0.008753919042646885,
+            0.001712448545731604,
+            -0.0010089334100484848,
+            -0.01075072307139635,
+            -0.0043769595213234425,
+            -0.009748108685016632,
+            0.007641775533556938,
+            -0.005552293267101049,
+            0.00820627249777317,
+            0.016134509816765785,
+            0.04782218113541603,
+            0.008320014923810959,
+            0.003953586798161268,
+            3.060764720430598e-05,
+            0.00025091689894907176,
+            -0.015966001898050308,
+            -0.022192321717739105,
+            -0.013649036176502705,
+            -0.009832361713051796,
+            0.024821026250720024,
+            -0.005792415235191584,
+            0.006390613969415426,
+            -0.017179250717163086,
+            0.005463827401399612,
+            0.0017177143599838018,
+            0.010843401774764061,
+            0.0033722389489412308,
+            -0.042396266013383865,
+            -0.0045791673474013805,
+            0.014213534072041512,
+            -0.011104586534202099,
+            -0.001104771508835256,
+            -0.031611841171979904,
+            0.003911459818482399,
+            -0.0158648993819952,
+            -0.0044907014816999435,
+            0.016749558970332146,
+            -0.020153392106294632,
+            0.0006840315763838589,
+            0.0032816664315760136,
+            -0.015940727666020393,
+            0.005122601520270109,
+            -0.008737068623304367,
+            0.0016840130556374788,
+            0.018215566873550415,
+            0.002660298254340887,
+            -0.01298343576490879,
+            -0.002645554021000862,
+            -0.024568265303969383,
+            0.01750783808529377,
+            0.029758270829916,
+            -0.008008277043700218,
+            -0.01941196247935295,
+            -0.005177366081625223,
+            0.014247234910726547,
+            -0.004145262762904167,
+            -0.00814308226108551,
+            0.0031405421905219555,
+            0.003479661885648966,
+            -0.020052287727594376,
+            0.00356602156534791,
+            0.003753485158085823,
+            0.01585647277534008,
+            0.010489537380635738,
+            0.014769605360925198,
+            -0.029286451637744904,
+            -0.023472972214221954,
+            0.007102554198354483,
+            -0.019378261640667915,
+            0.005463827401399612,
+            0.03986866772174835,
+            -0.005404850002378225,
+            0.014348339289426804,
+            0.011011907830834389,
+            0.022006966173648834,
+            -0.023658329620957375,
+            0.017524689435958862,
+            0.0020642061717808247,
+            -0.017710046842694283,
+            -0.010321030393242836,
+            0.016673730686306953,
+            0.00954589992761612,
+            0.02766878716647625,
+            -0.019260307773947716,
+            -0.004570742137730122,
+            0.007696540094912052,
+            0.03398778662085533,
+            0.003955692984163761,
+            -0.014617949724197388,
+            0.027517132461071014,
+            0.047114454209804535,
+            0.008897149935364723,
+            0.01652207411825657,
+            -0.01678325980901718,
+            -0.005101538263261318,
+            0.007785005960613489,
+            0.009849212132394314,
+            0.013337299227714539,
+            -0.0019746869802474976,
+            -0.020962223410606384,
+            0.033027298748493195,
+            -0.009899764321744442,
+            0.002022079424932599,
+            -0.027904696762561798,
+            0.018569430336356163,
+            0.009689130820333958,
+            0.005093112587928772,
+            0.004452787339687347,
+            0.001162695698440075,
+            0.04340730607509613,
+            0.004701334983110428,
+            -0.029303302988409996,
+            0.012065074406564236,
+            0.002390687819570303,
+            0.020153392106294632,
+            -0.013615335337817669,
+            -0.013059263117611408,
+            0.004739248659461737,
+            -0.0018978058360517025,
+            -0.010877102613449097,
+            0.008998253382742405,
+            -0.0005845073610544205,
+            0.019243456423282623,
+            -0.009933465160429478,
+            -0.001282756682485342,
+            -0.0138933714479208,
+            0.0014723266940563917,
+            -0.016067106276750565,
+            0.025360247120261192,
+            -0.014407316222786903,
+            0.004398022778332233,
+            0.009630153886973858,
+            -0.00914990995079279,
+            -0.0034712364431470633,
+            0.007953512482345104,
+            0.009933465160429478,
+            -0.02369203232228756,
+            0.018822189420461655,
+            0.0015323570696637034,
+            0.016977041959762573,
+            0.003606041893362999,
+            -0.01224200613796711,
+            0.007047789636999369,
+            0.005362723488360643,
+            0.017153974622488022,
+            -0.00942794606089592,
+            0.005817691329866648,
+            -0.030904114246368408,
+            -0.02374258264899254,
+            0.038284704089164734,
+            0.015350953675806522,
+            -0.022630440071225166,
+            0.015839623287320137,
+            -0.02148459479212761,
+            -0.006656011566519737,
+            0.01285705529153347,
+            0.01365746185183525,
+            -0.03110632300376892,
+            0.00914990995079279,
+            -0.009040380828082561,
+            0.009166760370135307,
+            0.02241138182580471,
+            0.023119108751416206,
+            0.0038103561382740736,
+            -0.00668971287086606,
+            -0.021400341764092445,
+            -0.006091514602303505,
+            0.007907173596322536,
+            0.006314785685390234,
+            -0.007203658111393452,
+            0.02106332778930664,
+            0.01426408626139164,
+            -0.0055017415434122086,
+            -0.005396424792706966,
+            0.02413014881312847,
+            -0.03043229691684246,
+            0.019816379994153976,
+            0.0010763360187411308,
+            -0.03301044926047325,
+            0.01959731988608837,
+            -0.007325825747102499,
+            0.030196387320756912,
+            -0.0035428518895059824,
+            -0.008762344717979431,
+            0.0012722250539809465,
+            0.01162695698440075,
+            0.0028161669615656137,
+            -0.00014138758706394583,
+            0.007843983359634876,
+            0.020962223410606384,
+            -0.022512484341859818,
+            0.023220213130116463,
+            0.011921843513846397,
+            3.429373100516386e-05,
+            0.014676926657557487,
+            -0.023186512291431427,
+            -0.03011213429272175,
+            0.02743287943303585,
+            -0.011610106565058231,
+            -0.02123183384537697,
+            0.015359378419816494,
+            -0.013615335337817669,
+            -0.012814928777515888,
+            -0.02278209663927555,
+            -0.0092425886541605,
+            0.004014670383185148,
+            -0.006554907653480768,
+            -0.016252463683485985,
+            -0.0033427502494305372,
+            0.021518295630812645,
+            0.01806391030550003,
+            0.022310277447104454,
+            -0.00462129432708025,
+            0.012873905710875988,
+            0.001359637826681137,
+            0.019563619047403336,
+            0.019883781671524048,
+            -0.00939424429088831,
+            -0.0029467595741152763,
+            0.013488955795764923,
+            -0.007363739423453808,
+            0.022192321717739105,
+            0.0008404267719015479,
+            -0.02950550988316536,
+            -0.010969781316816807,
+            -0.03690294921398163,
+            0.007911385968327522,
+            -0.00792823638767004,
+            -0.006487505044788122,
+            0.05395582318305969,
+            0.006175767630338669,
+            -0.01248634047806263,
+            0.007704965304583311,
+            0.0054554021917283535,
+            -0.005341660231351852,
+            -0.00786925945430994,
+            0.017094997689127922,
+            -0.01380911748856306,
+            -0.015266700647771358,
+            -0.013118240982294083,
+            -0.010843401774764061,
+            0.009714406915009022,
+            0.020642060786485672,
+            -0.0033617070876061916,
+            -0.026169078424572945,
+            -0.013210919685661793,
+            -0.020305048674345016,
+            -0.013151941820979118,
+            0.007886109873652458,
+            -0.0042716427706182,
+            -0.0007840823964215815,
+            0.037071458995342255,
+            -0.019496217370033264,
+            -0.013404701836407185,
+            -0.010683320462703705,
+            -0.002651873044669628,
+            -0.01658947765827179,
+            -0.005939858499914408,
+            0.016201911494135857,
+            -0.016867512837052345,
+            -0.0034480667673051357,
+            -0.0056533971801400185,
+            -0.010245203040540218,
+            0.020187092944979668,
+            0.008522222749888897,
+            -0.00668971287086606,
+            0.01156798005104065,
+            0.04886692017316818,
+            -0.03798139467835426,
+            -0.01603340543806553,
+            -0.02515803836286068,
+            0.026489241048693657,
+            0.009259439073503017,
+            0.01670743152499199,
+            -0.02092852257192135,
+            -0.028494469821453094,
+            -0.023725733160972595,
+            -0.012292558327317238,
+            -0.019732125103473663,
+            -0.003188987961038947,
+            0.023102257400751114,
+            0.014803307130932808,
+            0.04283438250422478,
+            0.00532902218401432,
+            0.0035428518895059824,
+            -0.0028772505465894938,
+            0.0022474571596831083,
+            0.011197265237569809,
+            -0.02178790606558323,
+            0.0026834679301828146,
+            0.00024749411386437714,
+            0.026371287181973457,
+            0.01105403434485197,
+            -0.006024111993610859,
+            -0.028865184634923935,
+            0.0021610974799841642,
+            0.012418937869369984,
+            -0.007549096830189228,
+            -0.017069721594452858,
+            0.027837295085191727,
+            0.004549678880721331,
+            -0.0034017274156212807,
+            -0.011281518265604973,
+            0.028511321172118187,
+            0.0027550833765417337,
+            0.028123755007982254,
+            -0.020406153053045273,
+            -0.009040380828082561,
+            -0.022798946127295494,
+            0.009436370804905891,
+            -0.009107783436775208,
+            0.002430708147585392,
+            -0.01025362778455019,
+            0.015696391463279724,
+            -0.01508976798504591,
+            -0.010245203040540218,
+            0.030465997755527496,
+            -0.021198133006691933,
+            -0.0025739388074725866,
+            -0.015578437596559525,
+            -0.028022652491927147,
+            0.00967228040099144,
+            -0.0011005589039996266,
+            0.007852409034967422,
+            -0.006656011566519737,
+            0.003707145806401968,
+            -0.023034855723381042,
+            0.009166760370135307,
+            -0.00963857863098383,
+            -0.011222541332244873,
+            -0.001816712087020278,
+            0.013649036176502705,
+            0.0008383204694837332,
+            0.010337881743907928,
+            0.005034135654568672,
+            -0.03892502933740616,
+            0.01836722157895565,
+            0.0036776571068912745,
+            0.010278903879225254,
+            0.003707145806401968,
+            0.008155720308423042,
+            0.007924024015665054,
+            0.029337003827095032,
+            0.00982393603771925,
+            -0.009874488227069378,
+            -0.011854440905153751,
+            -0.0011216222774237394,
+            0.011163563467562199,
+            0.018653683364391327,
+            -0.004899329971522093,
+            -0.01616821065545082,
+            0.006399039179086685,
+            3.333929998916574e-05,
+            0.03491457179188728,
+            0.008509584702551365,
+            -0.020372450351715088,
+            -0.005337447393685579,
+            -0.04917023330926895,
+            0.020490406081080437,
+            -0.015376229770481586,
+            -0.0339372344315052,
+            0.01075072307139635,
+            -0.0043074507266283035,
+            -0.022428231313824654,
+            0.029353853315114975,
+            -0.004945669323205948,
+            -0.013219344429671764,
+            0.007001450285315514,
+            -0.009343692101538181,
+            -0.006879283115267754,
+            0.0014428379945456982,
+            -0.016623178496956825,
+            6.6851384872279596e-06,
+            0.00356602156534791,
+            0.019176054745912552,
+            -0.0034775554668158293,
+            -0.01549418456852436,
+            0.001500762184150517,
+            0.00623053265735507,
+            0.017861701548099518,
+            0.0021221302449703217,
+            0.003989394288510084,
+            0.0006424314924515784,
+            0.014828583225607872,
+            -0.02664089761674404,
+            0.005366935860365629,
+            -0.0012501084711402655,
+            -0.026792554184794426,
+            0.004553891718387604,
+            -0.006024111993610859,
+            -0.01983322948217392,
+            0.012494766153395176,
+            -0.0009241535444743931,
+            -0.0016408332157880068,
+            -0.00585981784388423,
+            -0.009478497318923473,
+            -0.009697556495666504,
+            -0.008972978219389915,
+            0.008956126868724823,
+            0.016016554087400436,
+            0.007304762024432421,
+            -0.010118822567164898,
+            -0.02431550621986389,
+            0.001289075706154108,
+            0.02529284358024597,
+            0.02357407659292221,
+            -0.035285286605358124,
+            0.014735904522240162,
+            0.005935645662248135,
+            -0.00851379707455635,
+            -0.027163267135620117,
+            0.0310389194637537,
+            0.02197326347231865,
+            0.0038061433006078005,
+            0.007397440727800131,
+            0.01162695698440075,
+            0.007359527051448822,
+            -0.03949795290827751,
+            0.011079310439527035,
+            -0.011020333506166935,
+            -0.009849212132394314,
+            -3.0377268558368087e-05,
+            0.009031955152750015,
+            -0.009882913902401924,
+            -0.011803888715803623,
+            0.007481694221496582,
+            0.02204066701233387,
+            -0.009368968196213245,
+            -0.003660806454718113,
+            0.01678325980901718,
+            0.002243244554847479,
+            0.001279597170650959,
+            0.016808535903692245,
+            0.05621381103992462,
+            0.003601829055696726,
+            -0.0019662617705762386,
+            0.02192271128296852,
+            -0.0018651577411219478,
+            -0.022428231313824654,
+            -0.006196831353008747,
+            -0.01792910508811474,
+            0.008105169050395489,
+            0.010304179973900318,
+            -0.01456739753484726,
+            -0.005088900215923786,
+            -0.0035786593798547983,
+            0.009048805572092533,
+            -0.0046507827937603,
+            0.017828000709414482,
+            -0.00638218829408288,
+            0.01909179985523224,
+            0.03152759000658989,
+            0.01407872885465622,
+            -0.013345724903047085,
+            0.014162981882691383,
+            -0.007473269011825323,
+            -0.0033954083919525146,
+            -0.010666469112038612,
+            0.001936773071065545,
+            0.0015576331643387675,
+            0.005000433884561062,
+            0.0009467966156080365,
+            0.025680409744381905,
+            -0.0012048223288729787,
+            -0.01978267729282379,
+            -0.005805053282529116,
+            -0.0057334378361701965,
+            -0.01224200613796711,
+            0.007755517493933439,
+            0.0014059771783649921,
+            0.0034480667673051357,
+            -0.02081056870520115,
+            0.0031005218625068665,
+            0.005261619575321674,
+            0.005927220452576876,
+            0.006112577859312296,
+            0.0030268002301454544,
+            -0.0038166751619428396,
+            0.02013654261827469,
+            -0.00623053265735507,
+            0.001733511919155717,
+            0.013151941820979118,
+            -0.006415889598429203,
+            -0.010877102613449097,
+            -0.013143517076969147,
+            -0.0007919811177998781,
+            0.019698424264788628,
+            -0.014719054102897644,
+            -0.00436432147398591,
+            0.013556358404457569,
+            0.00863596424460411,
+            0.006710776127874851,
+            -0.0019072843715548515,
+            -0.00604517525061965,
+            0.0026876807678490877,
+            -0.02343927137553692,
+            -0.020557807758450508,
+            -0.008905574679374695,
+            -0.008496946655213833,
+            0.024264954030513763,
+            -0.006681287661194801,
+            0.005956709384918213,
+            -0.00756173487752676,
+            0.006466441787779331,
+            0.01824926771223545,
+            0.005413275212049484,
+            -0.003639743197709322,
+            0.006757115479558706,
+            -0.00021734721667598933,
+            0.002392794005572796,
+            -0.009318416006863117,
+            0.0031658182851970196,
+            0.015291975811123848,
+            -0.010995057411491871,
+            0.0022263936698436737,
+            -0.007738666608929634,
+            -0.0036629128735512495,
+            -0.022866349667310715,
+            0.010430560447275639,
+            -0.001800914527848363,
+            0.009613302536308765,
+            0.007873471826314926,
+            -0.008058829233050346,
+            0.012789652682840824,
+            -0.008694942109286785,
+            0.015713242813944817,
+            -0.012250431813299656,
+            -0.0004007298266515136,
+            0.03017953597009182,
+            -0.011155138723552227,
+            -0.019007546827197075,
+            -0.0034733428619802,
+            -0.016176635399460793,
+            0.02308540791273117,
+            0.027230670675635338,
+            0.015435206703841686,
+            0.016075532883405685,
+            -0.007456418126821518,
+            0.0066517991945147514,
+            0.025680409744381905,
+            0.003513363189995289,
+            0.020962223410606384,
+            -0.01162695698440075,
+            0.011803888715803623,
+            -0.03017953597009182,
+            0.0003238486824557185,
+            -0.0029994179494678974,
+            -0.004992008674889803,
+            0.007843983359634876,
+            -0.02131608873605728,
+            -0.008711792528629303,
+            -0.004992008674889803,
+            -0.00719102006405592,
+            0.0041389442048966885,
+            -0.019732125103473663,
+            0.00952062476426363,
+            -0.018451474606990814,
+            -0.01658947765827179,
+            -0.012006097473204136,
+            -0.01806391030550003,
+            -0.013480530120432377,
+            -0.01721295155584812,
+            -0.030819861218333244,
+            -0.00356602156534791,
+            -0.02062521129846573,
+            -0.025073785334825516,
+            0.018721086904406548,
+            -0.023860538378357887,
+            0.01199767179787159,
+            -0.0008514850633218884,
+            -0.006458016578108072,
+            0.025680409744381905,
+            0.029471809044480324,
+            -0.009866062551736832,
+            -0.023708881810307503,
+            -0.009655429981648922,
+            -0.029117945581674576,
+            0.009714406915009022,
+            0.011028758250176907,
+            0.0092425886541605,
+            -0.01090237870812416,
+            0.0015397293027490377,
+            -0.014516845345497131,
+            0.01732248067855835,
+            -0.032117363065481186,
+            -0.029522361233830452,
+            -0.008029340766370296,
+            -0.006879283115267754,
+            -0.001448103808797896,
+            -0.015350953675806522,
+            0.005969346966594458,
+            -0.018080761656165123,
+            -0.02748342975974083,
+            -0.0030331192538142204,
+            -0.006163129583001137,
+            -0.01304241269826889,
+            0.005219492595642805,
+            -0.00557335652410984,
+            0.022394530475139618,
+            0.010666469112038612,
+            0.015780646353960037,
+            -0.004848778247833252,
+            -0.009192036464810371,
+            -0.014306212775409222,
+            -0.01573851890861988,
+            0.004777162801474333,
+            0.009748108685016632,
+            0.012621146626770496,
+            0.0004154741473030299,
+            0.010076696053147316,
+            0.000991029548458755,
+            0.013210919685661793,
+            0.010270479135215282,
+            0.010910804383456707,
+            0.021164432168006897,
+            -0.0023801561910659075,
+            0.014188257977366447,
+            0.00402730843052268,
+            0.005459614563733339,
+            -0.0029383343644440174,
+            -0.008357929065823555,
+            0.022360829636454582,
+            -0.007658625952899456,
+            -0.013960774056613445,
+            0.029337003827095032,
+            0.004697122145444155,
+            -0.02320336177945137,
+            0.012629571370780468,
+            0.0003151600540149957,
+            -0.018721086904406548,
+            -0.008652815595269203,
+            -0.0009815511293709278,
+            -0.004473851062357426,
+            -0.00765441358089447,
+            0.0014112429926171899,
+            -0.008686516433954239,
+            0.01621876284480095,
+            0.016875937581062317,
+            -0.008930850774049759,
+            -0.008429544046521187,
+            0.017844852060079575,
+            0.016614753752946854,
+            -0.001709289033897221,
+            -0.022209173068404198,
+            -0.008694942109286785,
+            0.02504008449614048,
+            -0.03629632666707039,
+            -0.026893656700849533,
+            0.007700752932578325,
+            0.006420102436095476,
+            -0.009756533429026604,
+            6.574062354047783e-06,
+            0.003983075264841318,
+            -0.007578585296869278,
+            0.0032479651272296906,
+            0.0016208230517804623,
+            0.012680123560130596,
+            0.01484543364495039,
+            -0.011096160858869553,
+            0.010556939989328384,
+            -0.01573851890861988,
+            0.0018672640435397625,
+            -0.020035438239574432,
+            -0.013126665726304054,
+            -0.01380069274455309,
+            0.010371582582592964,
+            0.020591510459780693,
+            0.011829164810478687,
+            0.03889133036136627,
+            0.01466007623821497,
+            0.004014670383185148,
+            -0.006917196791619062,
+            -0.017288779839873314,
+            0.004802438896149397,
+            -0.002948865992948413,
+            -0.0037766548339277506,
+            -0.0021158112213015556,
+            0.014912836253643036,
+            -0.0021779481321573257,
+            0.028123755007982254,
+            0.006158917210996151,
+            0.01236838661134243,
+            -2.6213969249511138e-05,
+            -0.007022513542324305,
+            -0.003983075264841318,
+            0.013101390562951565,
+            5.410642552305944e-05,
+            -0.0023675181437283754,
+            0.0067192018032073975,
+            -0.0018483070889487863,
+            -0.005055198911577463,
+            0.0027656150050461292,
+            -0.011365772224962711,
+            -0.0055101667530834675,
+            -0.017356181517243385,
+            -0.008787620812654495,
+            -0.009175186045467854,
+            0.02485472708940506,
+            0.028865184634923935,
+            -0.015013940632343292,
+            -0.010430560447275639,
+            -0.013455254025757313,
+            0.03589191287755966,
+            -0.004840353038161993,
+            0.0034291099291294813,
+            0.014811732806265354,
+            -0.008753919042646885,
+            0.0024222827050834894,
+            0.012418937869369984,
+            0.0042210910469293594,
+            0.003555489704012871,
+            0.029926776885986328,
+            -0.0031721373088657856,
+            -0.01013567391782999,
+            -0.0035028315614908934,
+            0.003907247446477413,
+            -0.026101676747202873,
+            0.0003712411562446505,
+            0.004840353038161993,
+            -0.010514813475310802,
+            0.013092964887619019,
+            -0.019513066858053207,
+            0.006795029621571302,
+            -0.010632768273353577,
+            -0.0008367407135665417,
+            -0.01976582780480385,
+            -0.0007946140831336379,
+            0.007633350323885679,
+            -0.012587444856762886,
+            -0.003197413170710206,
+            -0.017356181517243385,
+            -0.0020062820985913277,
+            0.014643225818872452,
+            0.007001450285315514,
+            0.006040962412953377,
+            -0.0007472215802408755,
+            0.0032016257755458355,
+            0.022478783503174782,
+            0.0043474710546433926,
+            0.004718185402452946,
+            -0.004541253671050072,
+            0.0042526861652731895,
+            -0.024264954030513763,
+            0.006453803740441799,
+            -0.005762926768511534,
+            -0.0026497666258364916,
+            0.024955831468105316,
+            -0.005842967424541712,
+            0.0009678599308244884,
+            -0.006238957867026329,
+            0.005059411283582449,
+            -0.024568265303969383,
+            0.01508976798504591,
+            -0.01762579381465912,
+            0.010700170882046223,
+            -0.008564349263906479,
+            -0.002706637606024742,
+            -0.00251074880361557,
+            0.009444796480238438,
+            0.012292558327317238,
+            -0.013084539212286472,
+            0.007397440727800131,
+            -0.015822771936655045,
+            0.021686803549528122,
+            -0.003867227118462324,
+            -0.01878848858177662,
+            0.0018314564367756248,
+            -0.008046191185712814,
+            0.008762344717979431,
+            -0.014702202752232552,
+            0.0010515867033973336,
+            0.010396858677268028,
+            -0.007216296158730984,
+            0.005809266120195389,
+            -0.023119108751416206,
+            0.006828730925917625,
+            -0.017490988597273827,
+            0.008796045556664467,
+            -0.012966584414243698,
+            -0.012966584414243698,
+            -0.02118128351867199,
+            -0.012006097473204136,
+            -0.00024551941896788776,
+            0.0051436647772789,
+            0.015713242813944817,
+            -0.006934047676622868,
+            0.005838754586875439,
+            0.021872159093618393,
+            -0.010026143863797188,
+            0.01719610020518303,
+            0.053113289177417755,
+            -0.025444500148296356,
+            -0.016934916377067566,
+            0.00136279733851552,
+            0.010784423910081387,
+            -0.03434165194630623,
+            -0.008581199683248997,
+            0.019395112991333008,
+            -0.016361992806196213,
+            0.029606614261865616,
+            -0.025663558393716812,
+            0.01156798005104065,
+            -0.010708595626056194,
+            0.00796193815767765,
+            -0.017052870243787766,
+            -0.01929400861263275,
+            -0.00991661474108696,
+            -0.003568127751350403,
+            -0.0018462006701156497,
+            -0.02394479140639305,
+            0.009495348669588566,
+            -0.014390465803444386,
+            -0.011045609600841999,
+            0.006150492001324892,
+            -0.0057882023975253105,
+            0.007460630964487791,
+            0.0016608433797955513,
+            0.013758566230535507,
+            -0.020776866003870964,
+            0.0008767610415816307,
+            -0.011045609600841999,
+            0.004747674334794283,
+            -0.0067739663645625114,
+            0.011298369616270065,
+            0.01337942574173212,
+            -0.013092964887619019,
+            -0.002439133357256651,
+            -0.002023132750764489,
+            0.015511034987866879,
+            -0.006504355929791927,
+            0.03683554753661156,
+            -0.008075679652392864,
+            -0.004339045379310846,
+            -0.008871873840689659,
+            -0.0056870984844863415,
+            0.011829164810478687,
+            0.01328674703836441,
+            0.008568561635911465,
+            0.005834541749209166,
+            0.00914990995079279,
+            -0.014112429693341255,
+            0.01536780409514904,
+            0.007949300110340118,
+            0.017659494653344154,
+            0.005307958461344242,
+            -0.013851244933903217,
+            -0.024837875738739967,
+            -0.0066855004988610744,
+            -0.007262635510414839,
+            -0.0029446533881127834,
+            -0.01665687933564186,
+            0.0035512770991772413,
+            0.0067781792022287846,
+            0.007827132940292358,
+            -0.003900928422808647,
+            0.03257232904434204,
+            0.005729225464165211,
+            0.0012753845658153296,
+            0.027635086327791214,
+            0.007793431635946035,
+            0.0015112938126549125,
+            0.017524689435958862,
+            -0.031072622165083885,
+            -0.01404502708464861,
+            -0.021889010444283485,
+            0.011761762201786041,
+            0.0004615501675289124,
+            -0.0318477526307106,
+            -0.0024833662901073694,
+            0.0011121437419205904,
+            -0.01025362778455019,
+            0.012629571370780468,
+            0.0206926129758358,
+            -0.016378844156861305,
+            -0.011947119608521461,
+            0.008037766441702843,
+            0.011070885695517063,
+            -0.005190004128962755,
+            0.027635086327791214,
+            -0.002001016167923808,
+            0.009630153886973858,
+            0.007405866403132677,
+            0.019428813830018044,
+            0.0006540163303725421,
+            0.023523524403572083,
+            -0.006643373519182205,
+            -0.00483614020049572,
+            -0.020490406081080437,
+            -0.008412693627178669,
+            -0.01175333745777607,
+            0.011610106565058231,
+            -0.005206854548305273,
+            0.003789292648434639,
+            -0.013952348381280899,
+            0.007915598340332508,
+            0.014963388442993164,
+            0.011028758250176907,
+            0.0044443621300160885,
+            0.012124051339924335,
+            0.010455836541950703,
+            0.013590059243142605,
+            0.0012353642378002405,
+            -0.020338749513030052,
+            0.008914000354707241,
+            0.017608942463994026,
+            -0.006934047676622868,
+            0.02141719125211239,
+            0.023540375754237175,
+            0.001537623000331223,
+            -0.003188987961038947,
+            0.008518009446561337,
+            0.01652207411825657,
+            0.01976582780480385,
+            -0.01298343576490879,
+            -0.0032332208938896656,
+            0.0036629128735512495,
+            -0.0028983140364289284,
+            -0.0016408332157880068,
+            -0.019327709451317787,
+            -0.000656649237498641,
+            0.008614901453256607,
+            0.0010921335779130459,
+            0.030230088159441948,
+            -0.01823241636157036,
+            -0.01428936142474413,
+            0.01915920339524746,
+            0.006622310262173414,
+            -0.01880533993244171,
+            0.012503191828727722,
+            0.00015968635852914304,
+            0.013960774056613445,
+            -0.011104586534202099,
+            0.005215280223637819,
+            -0.01004299521446228,
+            -0.004503339529037476,
+            0.017524689435958862,
+            -0.009267864748835564,
+            0.011778613552451134,
+            -0.008383204229176044,
+            -0.001989431446418166,
+            -0.02074316516518593,
+            0.02032189816236496,
+            0.02613537758588791,
+            0.01086867693811655,
+            0.008412693627178669,
+            0.004751886706799269,
+            0.007271060720086098,
+            0.00436432147398591,
+            0.01927715726196766,
+            0.020658912137150764,
+            -0.011424749158322811,
+            0.007494332268834114,
+            0.018148163333535194,
+            -0.007249997463077307,
+            0.012840204872190952,
+            0.004827714990824461,
+            0.004970945417881012,
+            -0.007094128988683224,
+            0.006015686318278313,
+            -0.017162399366497993,
+            -0.013033987022936344,
+            0.0012627465184777975,
+            0.032606031745672226,
+            0.008454820141196251,
+            -0.023304466158151627,
+            0.001874636160209775,
+            0.024349207058548927,
+            -0.020894821733236313,
+            -0.0034754490479826927,
+            0.008315801620483398,
+            -0.0019157096976414323,
+            0.012899181805551052,
+            -0.02461881749331951,
+            -0.00786925945430994,
+            0.004810864105820656,
+            -0.006348486989736557,
+            0.016193486750125885,
+            -0.012098775245249271,
+            -0.0390598364174366,
+            -0.01199767179787159,
+            -0.020153392106294632,
+            -0.017861701548099518,
+            -0.019664723426103592,
+            0.0008930850890465081,
+            0.01585647277534008,
+            -0.019024398177862167,
+            -0.018333520740270615,
+            0.016741132363677025,
+            -0.001975740073248744,
+            0.03764437884092331,
+            -0.017373032867908478,
+            0.007536458782851696,
+            -0.01727192848920822,
+            0.01646309718489647,
+            -0.016800111159682274,
+            -0.018502026796340942,
+            0.01475275494158268,
+            -0.0031679244711995125,
+            -0.003721890039741993,
+            0.0023422420490533113,
+            0.00235698651522398,
+            -0.009714406915009022,
+            -0.003024693811312318,
+            0.011652233079075813,
+            0.0006982493214309216,
+            -0.0161008071154356,
+            -0.002645554021000862,
+            0.003528107423335314,
+            0.012882331386208534,
+            0.01578907109797001,
+            -0.0069635361433029175,
+            -0.003808249719440937,
+            0.010211501270532608,
+            -0.01762579381465912,
+            -0.010363157838582993,
+            0.015300401486456394,
+            -0.01732248067855835,
+            0.03191515430808067,
+            0.006592821795493364,
+            -0.01776059903204441,
+            -0.00716153159737587,
+            0.003629211336374283,
+            0.023287614807486534,
+            0.01332044880837202,
+            -0.020726315677165985,
+            -0.008004064671695232,
+            0.007810282055288553,
+            0.012739100493490696,
+            0.005438551306724548,
+            -0.007186807692050934,
+            0.027820443734526634,
+            -0.0007530139992013574,
+            0.00015547369548585266,
+            0.001172174233943224,
+            -0.01056536566466093,
+            -0.0021737355273216963,
+            0.008804471231997013,
+            0.015839623287320137,
+            0.004292706493288279,
+            0.00985763780772686,
+            0.005333234556019306,
+            -0.01635356806218624,
+            0.015713242813944817,
+            0.0068076676689088345,
+            -0.0035449580755084753,
+            -0.016370419412851334,
+            0.011273093521595001,
+            0.007119405083358288,
+            0.016513649374246597,
+            -0.012772802263498306,
+            -0.004882479552179575,
+            -1.0696221579564735e-05,
+            0.00737637747079134,
+            0.0022221810650080442,
+            0.008067254908382893,
+            0.0010705436579883099,
+            0.02938755601644516,
+            0.03282508999109268,
+            0.03312840312719345,
+            -0.025326546281576157,
+            -0.010321030393242836,
+            0.008231548592448235,
+            -0.003157392842695117,
+            -0.004654995631426573,
+            -0.0003354334912728518,
+            -0.01720452681183815,
+            0.006971961818635464,
+            -0.02620278112590313,
+            0.01199767179787159,
+            -0.001448103808797896,
+            0.005147877614945173,
+            -0.010860252194106579,
+            0.0066517991945147514,
+            -0.0013680631527677178,
+            0.0001929927384480834,
+            -0.011424749158322811,
+            -0.019209755584597588,
+            -0.0014586354373022914,
+            0.014002900570631027,
+            -0.0012374705402180552,
+            -0.008122019469738007,
+            0.014061877503991127,
+            -0.006057813297957182,
+            -0.023237064480781555,
+            0.006087301764637232,
+            -0.028882035985589027,
+            0.0021010669879615307,
+            -0.013092964887619019,
+            0.003909353632479906,
+            -0.022680992260575294,
+            0.013421552255749702,
+            -0.0013554252218455076,
+            -0.01267169788479805,
+            0.004941456951200962,
+            -0.017288779839873314,
+            0.0025676197838038206,
+            -0.003197413170710206,
+            0.006272659171372652,
+            -0.010708595626056194,
+            0.028376515954732895,
+            0.014575823210179806,
+            -0.015140320174396038,
+            -0.017052870243787766,
+            0.008446394465863705,
+            -0.02809005416929722,
+            0.021703653037548065,
+            0.010885528288781643,
+            -0.0034733428619802,
+            -0.02406274527311325,
+            0.009748108685016632,
+            0.0036966141778975725,
+            0.02050725556910038,
+            0.010860252194106579,
+            0.009293140843510628,
+            0.018822189420461655,
+            -0.015527885407209396,
+            0.0021800545509904623,
+            -1.19962232929538e-05,
+            -0.00251074880361557,
+            0.014921261928975582,
+            -0.008071467280387878,
+            0.014584247954189777,
+            0.007359527051448822,
+            -0.0044232988730072975,
+            -0.006268446333706379,
+            -0.0030520763248205185,
+            -0.015165596269071102,
+            0.0004289020143914968,
+            -0.0020241858437657356,
+            -0.0008130444912239909,
+            0.006647586356848478,
+            0.010110397823154926,
+            -0.017305631190538406,
+            0.00914990995079279,
+            -0.0012248324928805232,
+            -0.025848915800452232,
+            0.0014038708759471774,
+            -0.016741132363677025,
+            0.002437027171254158,
+            -0.021383490413427353,
+            0.01252004224807024,
+            0.025208590552210808,
+            0.011037183925509453,
+            -0.007073065731674433,
+            -0.006234745029360056,
+            -0.011879716999828815,
+            -0.007485906593501568,
+            -0.010514813475310802,
+            -0.006125215906649828,
+            -0.019209755584597588,
+            -0.003728209063410759,
+            0.011121436953544617,
+            0.011728061363101006,
+            0.00034227909054607153,
+            -0.018350372090935707,
+            -0.005135239567607641,
+            -0.013109815306961536,
+            -0.005400637164711952,
+            -0.008547498844563961,
+            -0.013084539212286472,
+            0.0011974502122029662,
+            0.012545318342745304,
+            -0.011669083498418331,
+            0.029421256855130196,
+            0.005779777187854052,
+            0.006243170239031315,
+            -0.0022158620413392782,
+            -0.007881897501647472,
+            -0.004162113647907972,
+            -0.012216730043292046,
+            -0.0022074368316680193,
+            -0.005341660231351852,
+            -0.0020757908932864666,
+            -0.00905723124742508,
+            0.005813478492200375,
+            -0.0015913344686850905,
+            0.01555316150188446,
+            0.006062025669962168,
+            -0.008126231841742992,
+            -0.002955185016617179,
+            0.031679246574640274,
+            -0.007140468340367079,
+            -0.016614753752946854,
+            -0.00037861333112232387,
+            -0.00638218829408288,
+            0.0037303154822438955,
+            0.007414291612803936,
+            -0.01769319549202919,
+            0.0043895975686609745,
+            0.010851826518774033,
+            0.004794013686478138,
+            -0.0033827705774456263,
+            0.02081056870520115,
+            0.008779195137321949,
+            -0.002984673483297229,
+            0.010329456068575382,
+            0.0032269018702208996,
+            -0.010068271309137344,
+            -0.0044232988730072975,
+            -0.0026476604398339987,
+            0.028814632445573807,
+            -0.03484717011451721,
+            -0.0020547276362776756,
+            0.019732125103473663,
+            0.0044780634343624115,
+            -0.021147580817341805,
+            -0.002312753349542618,
+            -0.026977911591529846,
+            -0.0020673656836152077,
+            -0.009891338646411896,
+            0.013404701836407185,
+            -0.036026716232299805,
+            -0.002512854989618063,
+            -0.00905723124742508,
+            0.0044991266913712025,
+            -0.006373763084411621,
+            0.026893656700849533,
+            -0.0008936116937547922,
+            -0.005505953915417194,
+            0.011011907830834389,
+            0.009689130820333958,
+            0.011947119608521461,
+            0.026590345427393913,
+            -0.006934047676622868,
+            -0.0067529031075537205,
+            0.003083671210333705,
+            -0.028073204681277275,
+            0.02276524528861046,
+            0.00362710515037179,
+            0.0032669221982359886,
+            -0.0004115247866138816,
+            -0.006740265060216188,
+            0.02625333145260811,
+            0.02866297774016857,
+            -0.005223705433309078,
+            -0.005282682832330465,
+            -0.016387268900871277,
+            0.029101094231009483,
+            -0.011441599577665329,
+            0.0011205690680071712,
+            -0.002015760401263833,
+            -0.0121493274345994,
+            -0.01732248067855835,
+            -0.013446828350424767,
+            0.019984886050224304,
+            -0.02008599042892456,
+            -0.0017724790377542377,
+            -0.012882331386208534,
+            -0.001390179619193077,
+            -0.02497268095612526,
+            -0.008467458188533783,
+            0.008572774939239025,
+            -0.014070303179323673,
+            -0.0032416461035609245,
+            0.0003148967516608536,
+            -0.006436952855437994,
+            0.0011311006965115666,
+            -0.0067866044119000435,
+            -0.00031437017605639994,
+            -0.003799824509769678,
+            0.011129862628877163,
+            0.0002694789436645806,
+            -0.008855023421347141,
+            0.019496217370033264,
+            -0.010919229127466679,
+            0.006795029621571302,
+            0.0043221949599683285,
+            -0.004010457545518875,
+            -0.0003080511814914644,
+            0.009697556495666504,
+            0.014188257977366447,
+            -0.013708014041185379,
+            0.021939562633633614,
+            0.009377393871545792,
+            -0.01484543364495039,
+            0.010405284352600574,
+            -0.005354297813028097,
+            -0.019866930320858955,
+            -0.0026687236968427896,
+            0.015022365376353264,
+            0.019850080832839012,
+            -0.022445082664489746,
+            0.0011195158585906029,
+            0.01873793639242649,
+            -0.00021194973669480532,
+            0.0021379278041422367,
+            0.007338463328778744,
+            0.006142066325992346,
+            0.007485906593501568,
+            0.010026143863797188,
+            0.009739683009684086,
+            -0.0037576977629214525,
+            -0.0206926129758358,
+            0.015199298039078712,
+            -0.012140902690589428,
+            0.01561213843524456,
+            0.023776285350322723,
+            -0.010607492178678513,
+            -0.021467743441462517,
+            0.015106619335711002,
+            -0.007186807692050934,
+            -0.02827541157603264,
+            -0.0015039215795695782,
+            -0.0017292991979047656,
+            0.01978267729282379,
+            0.0030815647915005684,
+            -2.3564598450320773e-05,
+            -0.018282968550920486,
+            -1.969750337593723e-05,
+            0.01080970000475645,
+            -0.0017524688737466931,
+            -0.0411493182182312,
+            -0.006217894610017538,
+            0.004263217560946941,
+            -0.00486562866717577,
+            0.014255660586059093,
+            0.0017651069210842252,
+            0.005527017172425985,
+            0.026000572368502617,
+            -0.019698424264788628,
+            -0.010118822567164898,
+            0.006521206349134445,
+            0.016370419412851334,
+            0.010152524337172508,
+            0.011104586534202099,
+            0.006508568301796913,
+            0.015224573202431202,
+            -0.012039798311889172,
+            -0.007102554198354483,
+            0.018266119062900543,
+            -0.013387851417064667,
+            0.014811732806265354,
+            -0.01811446249485016,
+            0.004461213015019894,
+            -0.006310573313385248,
+            0.004739248659461737,
+            0.010969781316816807,
+            -0.00768390204757452,
+            -0.0010357891442254186,
+            -0.040744900703430176,
+            0.004570742137730122,
+            -0.0027213820721954107,
+            0.014710628427565098,
+            -0.01105403434485197,
+            0.004608656279742718,
+            -0.0001983902184292674,
+            0.005834541749209166,
+            0.009689130820333958,
+            -0.0008893990307115018,
+            0.01615135930478573,
+            -0.02601742371916771,
+            -0.025141188874840736,
+            7.082543743308634e-05,
+            -0.006917196791619062,
+            0.008762344717979431,
+            0.020423002541065216,
+            0.012444213964045048,
+            0.023489823564887047,
+            -0.01205664873123169,
+            0.008509584702551365,
+            0.011188839562237263,
+            -0.016875937581062317,
+            -0.003881971351802349,
+            0.0011953439097851515,
+            -0.008400055579841137,
+            -0.02332131750881672,
+            -0.0018177651800215244,
+            -0.004671846050769091,
+            0.008623326197266579,
+            0.0035639151465147734,
+            0.006377975922077894,
+            -0.025427650660276413,
+            0.004160007461905479,
+            0.004265323746949434,
+            -0.0298762246966362,
+            0.00018074968829751015,
+            -0.017423585057258606,
+            0.008627539500594139,
+            -0.0016650561010465026,
+            0.021282386034727097,
+            -0.013126665726304054,
+            -0.022849498316645622,
+            -0.005716587416827679,
+            0.005425913259387016,
+            -0.007389015518128872,
+            -0.01025362778455019,
+            0.015814347192645073,
+            -0.017069721594452858,
+            -0.008008277043700218,
+            -0.021585699170827866,
+            -0.006601247005164623,
+            -0.036026716232299805,
+            -0.006344274617731571,
+            0.01056536566466093,
+            0.005548080429434776,
+            -0.010068271309137344,
+            -0.029657166451215744,
+            0.005889306776225567,
+            -0.0067781792022287846,
+            -0.004549678880721331,
+            -0.01096135564148426,
+            0.0014923367416486144,
+            0.022006966173648834,
+            -0.006411677226424217,
+            0.02148459479212761,
+            -0.0174404364079237,
+            -0.004705547355115414,
+            -0.017061294987797737,
+            0.0044570001773536205,
+            -0.0022179684601724148,
+            -0.004583380185067654,
+            -0.00666022440418601,
+            -0.011736486107110977,
+            0.0024875791277736425,
+            0.002984673483297229,
+            -0.007435354869812727,
+            -0.007532245945185423,
+            0.012267282232642174,
+            0.009141484275460243,
+            0.007831345312297344,
+            0.007843983359634876,
+            0.004840353038161993,
+            -0.012621146626770496,
+            -0.00032700816518627107,
+            -0.006024111993610859,
+            0.005194216500967741,
+            0.009933465160429478,
+            0.0057207997888326645,
+            0.005004646722227335,
+            -0.008897149935364723,
+            0.012081924825906754,
+            0.01899069733917713,
+            0.033263206481933594,
+            -0.01304241269826889,
+            -0.005168940871953964,
+            -0.014011326245963573,
+            0.02166995219886303,
+            0.019024398177862167,
+            0.003317474154755473,
+            0.005088900215923786,
+            0.006925622466951609,
+            -0.0019315071403980255,
+            0.0013238302199169993,
+            0.012098775245249271,
+            -0.00872021820396185,
+            -0.009293140843510628,
+            -0.011323644779622555,
+            0.0013733289670199156,
+            -0.01781114935874939,
+            0.008058829233050346,
+            0.006441165693104267,
+            0.002065259264782071,
+            0.01025362778455019,
+            0.03936314582824707,
+            0.0026139591354876757,
+            -0.0016471522394567728,
+            -0.00238647498190403,
+            -0.01915920339524746,
+            0.02124868519604206,
+            -0.007915598340332508,
+            -0.039295744150877,
+            -0.0024960043374449015,
+            0.002679255325347185,
+            0.01347210444509983,
+            -0.0079324496909976,
+            -0.000739322800654918,
+            -0.011787038296461105,
+            -0.02074316516518593,
+            0.0014459975063800812,
+            0.015039216727018356,
+            -0.0005660769529640675,
+            0.011955545283854008,
+            0.003568127751350403,
+            0.0026350223924964666,
+            0.0034164718817919493,
+            0.009537475183606148,
+            -0.02148459479212761,
+            0.002752976957708597,
+            0.007671264000236988,
+            0.028747230768203735,
+            0.0007682848954573274,
+            -0.007233147043734789,
+            0.017828000709414482,
+            -0.0034164718817919493,
+            -0.007157318759709597,
+            0.008041978813707829,
+            -0.011256242170929909,
+            0.010893953032791615,
+            -0.013143517076969147,
+            0.011795463971793652,
+            0.02290005050599575,
+            -0.0206926129758358,
+            0.005383786745369434,
+            -0.007325825747102499,
+            -0.003957799170166254,
+            0.015696391463279724,
+            0.020102839916944504,
+            0.006255808286368847,
+            0.0010484271915629506,
+            -0.02106332778930664,
+            0.017490988597273827,
+            -0.02081056870520115,
+            -0.0102873295545578,
+            0.005581781733781099,
+            -0.014794881455600262,
+            0.010978206992149353,
+            -0.0038988220039755106,
+            -0.026539793238043785,
+            0.003875652328133583,
+            0.009360543452203274,
+            0.015115044079720974,
+            0.006340061780065298,
+            0.005560718476772308,
+            0.0012048223288729787,
+            -0.012022947892546654,
+            -0.023708881810307503,
+            -0.0058219037018716335,
+            -0.019007546827197075,
+            -0.021889010444283485,
+            -0.006424315273761749,
+            0.007110979408025742,
+            -0.008058829233050346,
+            0.0018704235553741455,
+            0.023001154884696007,
+            -0.018535729497671127,
+            -0.0065675457008183,
+            0.022748393937945366,
+            0.003507044166326523,
+            0.013017136603593826,
+            -0.007460630964487791,
+            0.017423585057258606,
+            0.00954589992761612,
+            -0.012494766153395176,
+            -0.01959731988608837,
+            0.027045313268899918,
+            -0.00495830737054348,
+            0.007321612909436226,
+            0.008846597746014595,
+            0.0012195666786283255,
+            -0.004726610612124205,
+            0.00814308226108551,
+            -0.030465997755527496,
+            0.013151941820979118,
+            -0.00734688900411129,
+            -0.013244620524346828,
+            -0.012208305299282074,
+            -0.00628950959071517,
+            0.001043687923811376,
+            -0.0015302507672458887,
+            0.01603340543806553,
+            -0.006116790696978569,
+            0.01841777376830578,
+            0.003210051218047738,
+            -0.0013785947812721133,
+            0.004608656279742718,
+            -0.0437106192111969,
+            -0.002169522689655423,
+            -0.010851826518774033,
+            -0.022798946127295494,
+            0.023422420024871826,
+            0.010952930897474289,
+            0.00039098801789805293,
+            -0.023557227104902267,
+            0.003938842564821243,
+            -0.004195814952254295,
+            -0.00025855234707705677,
+            0.020406153053045273,
+            0.0026961059775203466,
+            0.0010089334100484848,
+            -0.022731544449925423,
+            -0.0029783546924591064,
+            0.010068271309137344,
+            -0.0028182733803987503,
+            0.0010178852826356888,
+            0.01567954197525978,
+            0.010632768273353577,
+            -0.012393662706017494,
+            -0.018704235553741455,
+            0.005691311322152615,
+            0.016825387254357338,
+            0.008821321651339531,
+            0.0032416461035609245,
+            -0.008071467280387878,
+            0.0044443621300160885,
+            -0.003835632000118494,
+            0.038419511169195175,
+            0.020894821733236313,
+            0.006272659171372652,
+            -0.02504008449614048,
+            0.008475882932543755,
+            -0.019176054745912552,
+            0.004574954975396395,
+            0.010556939989328384,
+            0.00872021820396185,
+            -0.0032269018702208996,
+            -0.007587010972201824,
+            0.017052870243787766,
+            -0.005291108042001724,
+            0.018451474606990814,
+            -0.005286895204335451,
+            -0.00538799911737442,
+            0.018839040771126747,
+            -0.023540375754237175,
+            0.007650200743228197,
+            -0.018586281687021255,
+            -0.011787038296461105,
+            -0.013185643590986729,
+            -0.004248473327606916,
+            0.014087153598666191,
+            0.0013543720124289393,
+            -0.0326734334230423,
+            0.007321612909436226,
+            -0.026775702834129333,
+            0.00860647577792406,
+            -0.011323644779622555,
+            -4.169880776316859e-05,
+            -0.002578151412308216,
+            -0.008496946655213833,
+            -0.02431550621986389,
+            -0.021754205226898193,
+            0.005404850002378225,
+            -0.004570742137730122,
+            -0.0030520763248205185,
+            0.004633932374417782,
+            -0.0018082867609336972,
+            -0.01579749584197998,
+            0.008063041605055332,
+            0.028814632445573807,
+            0.02111387997865677,
+            0.008539073169231415,
+            0.0056196958757936954,
+            0.014727478846907616,
+            0.0011542703723534942,
+            -0.037745483219623566,
+            0.008404267951846123,
+            0.004684484098106623,
+            0.006529631558805704,
+            0.006175767630338669,
+            -0.005539655219763517,
+            0.0017608941998332739,
+            -0.010203076526522636,
+            -0.007553309667855501,
+            0.0029130582697689533,
+            -0.010607492178678513,
+            -0.007169956807047129,
+            -0.031241128221154213,
+            0.003083671210333705,
+            -0.010826550424098969,
+            0.007325825747102499,
+            -0.006971961818635464,
+            0.01641254499554634,
+            0.0016945448005571961,
+            0.00020286618382669985,
+            -0.013817543163895607,
+            -0.013497380539774895,
+            -0.0029320153407752514,
+            5.486338704940863e-05,
+            -0.01885589212179184,
+            -0.008416905999183655,
+            0.005105750635266304,
+            -0.002458090428262949,
+            -0.026657748967409134,
+            0.023489823564887047,
+            -0.000226299132918939,
+            0.011289943940937519,
+            0.023355018347501755,
+            0.020726315677165985,
+            4.5582357415696606e-05,
+            0.004815076943486929,
+            0.0025971082504838705,
+            -0.008404267951846123,
+            -0.01861998252570629,
+            0.007587010972201824,
+            -0.029859373345971107,
+            0.02136664092540741,
+            -0.006146279163658619,
+            -0.018636832013726234,
+            -0.006436952855437994,
+            -0.009082507342100143,
+            0.011011907830834389,
+            0.00015270912263076752,
+            0.01664845459163189,
+            -0.011348920874297619,
+            -0.003734528087079525,
+            -0.005607057828456163,
+            0.0018830614862963557,
+            0.008404267951846123,
+            0.029589762911200523,
+            -0.0032079447992146015,
+            -0.010413709096610546,
+            0.0131940683349967,
+            0.007679689209908247,
+            0.02896628901362419,
+            0.0013954454334452748,
+            -0.0008451660396531224,
+            -0.013868095353245735,
+            -0.0010378954466432333,
+            -0.01075072307139635,
+            -0.007220508996397257,
+            0.02136664092540741,
+            -0.010481111705303192,
+            -0.01995118521153927,
+            -0.00817678403109312,
+            -0.02050725556910038,
+            -0.007717603351920843,
+            0.013025562278926373,
+            -0.004954094998538494,
+            -0.004326407797634602,
+            -0.007182594854384661,
+            -0.01508976798504591,
+            -0.017962805926799774,
+            0.02606797404587269,
+            0.07434512674808502,
+            -0.021265536546707153,
+            -0.009596452116966248,
+            -0.011281518265604973,
+            0.0033954083919525146,
+            -0.004903542809188366,
+            -0.006748690269887447,
+            0.05661822482943535,
+            -0.019984886050224304,
+            0.005868243053555489,
+            -0.020406153053045273,
+            -0.006108365021646023,
+            -8.392419840674847e-05,
+            0.0068877083249390125,
+            0.002439133357256651,
+            -0.008105169050395489,
+            0.009899764321744442,
+            -0.018350372090935707,
+            -0.0057334378361701965,
+            -0.016201911494135857,
+            -0.013977624475955963,
+            -0.0169686172157526,
+            0.011643807403743267,
+            -0.019917482510209084,
+            -0.008956126868724823,
+            -0.0014091366901993752,
+            -0.009588027372956276,
+            0.015317251905798912,
+            -0.009267864748835564,
+            0.009958741255104542,
+            0.0009815511293709278,
+            0.003102628281340003,
+            -0.00939424429088831,
+            -0.010026143863797188,
+            -0.007144680712372065,
+            -0.0023822623770684004,
+            0.005703949369490147,
+            0.0007082544034346938,
+            0.0014491570182144642,
+            0.007292123977094889,
+            -0.0067739663645625114,
+            0.0054764654487371445,
+            0.020355600863695145,
+            0.0009315256611444056,
+            0.004899329971522093,
+            -0.017255079001188278,
+            0.019799528643488884,
+            -0.00517315324395895,
+            0.0289494376629591,
+            0.0057123745791614056,
+            -0.009689130820333958,
+            -0.016640029847621918,
+            0.014247234910726547,
+            -0.013573208823800087,
+            -0.0034227909054607153,
+            0.01983322948217392,
+            0.029168495908379555,
+            0.00789874792098999,
+            -0.012536892667412758,
+            0.0010415816213935614,
+            -4.456210444914177e-05,
+            0.01959731988608837,
+            -0.011348920874297619,
+            0.03230271860957146,
+            -0.011896567419171333,
+            0.004448574967682362,
+            -0.026118526235222816,
+            -0.009402669966220856,
+            -0.006323210895061493,
+            -0.003669231664389372,
+            0.022546187043190002,
+            -0.0044907014816999435,
+            0.007999852299690247,
+            -0.01267169788479805,
+            -0.021046478301286697,
+            0.004642357584089041,
+            -0.0043685343116521835,
+            -0.006723414175212383,
+            0.01579749584197998,
+            -0.029488658532500267,
+            0.004250579513609409,
+            0.009023529477417469,
+            -0.013101390562951565,
+            -0.017356181517243385,
+            0.012814928777515888,
+            0.008202060125768185,
+            -0.009461646899580956,
+            -0.008821321651339531,
+            -0.019395112991333008,
+            -0.02760138548910618,
+            0.010144098661839962,
+            0.0018925400217995048,
+            0.006192618515342474,
+            0.0022221810650080442,
+            -0.011728061363101006,
+            0.020709464326500893,
+            -0.005055198911577463,
+            0.009048805572092533,
+            -0.017726896330714226,
+            0.006281084381043911,
+            0.011155138723552227,
+            0.003974650055170059,
+            -0.0004581273824442178,
+            0.01404502708464861,
+            0.007338463328778744,
+            0.033027298748493195,
+            0.01270539965480566,
+            0.008189422078430653,
+            -0.009992443025112152,
+            -0.009882913902401924,
+            0.012140902690589428,
+            -0.021080179139971733,
+            -0.0007645988371223211,
+            0.016008129343390465,
+            0.010422134771943092,
+            -0.016260888427495956,
+            0.012680123560130596,
+            -0.01720452681183815,
+            0.010818125680088997,
+            -0.03378557786345482,
+            -0.012494766153395176,
+            0.006592821795493364,
+            0.014011326245963573,
+            0.008871873840689659,
+            0.0002376206684857607,
+            0.017592091113328934,
+            0.015376229770481586,
+            0.0031342231668531895,
+            0.006845581810921431,
+            0.022815797477960587,
+            0.004697122145444155,
+            -0.004671846050769091,
+            0.0035091505851596594,
+            -0.005244768690317869,
+            0.0016776940319687128,
+            -0.010919229127466679,
+            0.011483726091682911,
+            0.008897149935364723,
+            -0.006575970910489559,
+            0.01365746185183525,
+            -0.0009689130820333958,
+            -0.012250431813299656,
+            -0.007325825747102499,
+            -0.004587593022733927,
+            -0.008981402963399887,
+            -0.005649184808135033,
+            0.019513066858053207,
+            -0.015696391463279724,
+            -0.008939276449382305,
+            0.0024496649857610464,
+            0.014685352332890034,
+            -0.0025634069461375475,
+            -0.006571758538484573,
+            -0.008678090758621693,
+            -0.00019167628488503397,
+            -0.01205664873123169,
+            0.01450842060148716,
+            0.026742001995444298,
+            0.023894239217042923,
+            -0.004225303418934345,
+            -0.0032858792692422867,
+            -0.0013448934769257903,
+            0.013632185757160187,
+            0.0010505334939807653,
+            0.00133225554600358,
+            -0.007709178142249584,
+            -0.005725012626498938,
+            -0.018653683364391327,
+            -0.0014038708759471774,
+            0.0017956487135961652,
+            0.00014099264808464795,
+            -0.005973559804260731,
+            -0.013421552255749702,
+            0.011003483086824417,
+            0.008846597746014595,
+            0.0174404364079237,
+            -0.014668501913547516,
+            -0.02980882115662098,
+            0.0195299182087183,
+            -0.003911459818482399,
+            -0.004372747149318457,
+            0.003587084822356701,
+            -0.004574954975396395,
+            0.04333990439772606,
+            0.0051436647772789,
+            0.005939858499914408,
+            -0.011441599577665329,
+            -0.008787620812654495,
+            0.015637414529919624,
+            -0.005219492595642805,
+            -0.002031557960435748,
+            -0.019732125103473663,
+            -0.02466936968266964,
+            0.005131026729941368,
+            -0.024332357570528984,
+            -0.001645045937038958,
+            0.002439133357256651,
+            -0.0013027668464928865,
+            -0.013303598389029503,
+            -0.015949152410030365,
+            -0.003936735913157463,
+            0.011045609600841999,
+            0.01719610020518303,
+            -7.589380402350798e-05,
+            0.006116790696978569,
+            -0.00746905617415905,
+            0.003593403846025467,
+            0.01233468484133482,
+            -0.003881971351802349,
+            -0.01647152192890644,
+            -0.007511182688176632
+          ],
+          "title": "Appendices A. Overview Of Hyper-Parameters"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": false,
+      "properties": {}
+    },
+    {
+      "id": "cce1efd2-e327-4ef9-8a3a-acbc77487da7",
+      "type": "cosine_similarity",
+      "source": {
+        "id": "b570d71e-4f83-486e-b70f-b0c6217db455",
+        "properties": {
+          "page_content": "Mixture-Of-Experts With Expert Choice Routing\n\nYanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon Google, Mountain View, CA, USA {yanqiz, taole, hanxiaol, dunan, huangyp, vzhao, adai, zhifengc, qvl, jlaudon}@google.com\n\nAbstract\n\nSparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choice method. Instead of letting tokens select the top-k experts, we have experts selecting the top-k tokens. As a result, each token can be routed to a variable number of experts and each expert can have a fixed bucket size. We systematically study pre-training speedups using the same computational resources of the Switch Transformer top-1 and GShard top-2 gating of prior work and find that our method improves training convergence time by more than 2×. For the same computational cost, our method demonstrates higher performance in fine-tuning 11 selected tasks in the GLUE and SuperGLUE benchmarks. For a smaller activation cost, our method outperforms the T5 dense model in 7 out of the 11 tasks.\n\n1 Introduction\n\nScaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27]. The final model quality has been found to have a power-law relationship with the amount of data, model size, and compute time [16, 20]. However, training efficiency, which is defined as the total amount of computation used to achieve superior model quality than the state of the art system [21], should receive greater attention as we increase our efforts towards green AI [29].\n\nSparsely gated mixture-of-experts [31] (MoE) provides an effective way to scale model capacity given a fixed computational cost, and has recently played an important role in increasing the training efficiency of large-scale language models [10, 21]. MoE operate by adopting a number of experts, each as a sub-network, and by activating only one or a few experts for each input token. A gating network must be chosen and optimized in order to route each token to the most suited expert(s). For example, recent work has implemented sparse routing via k-means clustering [12], linear assignment to maximize token-expert affinities [22], or hashing [8, 28]. Many of the prior work use a routing strategy concerning the token choice, where each token selects the best one or two experts.\n\nWe argue that the independent token choice of prior work often leads to an imbalanced load of experts, which causes training inefficiency and sub-optimal training of the model. In order to mitigate this\n\nissue, previous sparsely gated networks introduce additional auxiliary losses as regularization to prevent too many tokens being routed to a single expert, but the effectiveness is still limited. Recent approaches [8, 22, 28] explore alternative strategies for routing, but they focus on pre-training only and do not demonstrate performance gain on downstream tasks. Moreover, none of the previous methods consider allocating a variable number of experts to each token based on importance, which can be beneficial.\n\nWe propose a very simple yet effective routing method we are calling expert choice. Unlike conventional MoE where tokens select one or two top-scoring experts, our method lets each expert pick the top-k tokens. Our method guarantees perfect load balancing, allows a variable number of experts for each token, and achieves substantial gains in training efficiency and downstream performance as demonstrated in our experiments. Our major contributions include:\n\nWe identify common pitfalls in conventional MoE such as load imbalance as described in Section 3.1. We then propose a heterogeneous, expert choice method to provide a fluid allocation of model parameters based on a learnt token-to-expert importance. This method intrinsically guarantees load balance without imposing an auxiliary loss.\n\nWe show our method provides over 2× faster training convergence in a 8B/64E (8 billion activated parameters, 64 experts) model, compared to the top-1 and top-2 gating counterparts in Switch Transformer [10] and GShard [21].\n\nWe show our method demonstrates strong scaling when increasing the number of experts from 16 to 128, evaluated in training perplexity.\n\nWe show our method demonstrates strong performance on downstream tasks selected from GLUE and SuperGLUE at all the evaluated scales. More specifically, our 8B/64E model outperforms a T5 11B dense model in 7 out of 11 tasks evaluated.\n\n2 Related Work\n\nScaling: Various approaches have been proposed to scale up neural network capacity to improve performance. Recent works have successfully scaled models to billions of parameters via various forms of model parallelism [2, 21, 26, 27, 33]. Model parallelism [30] splits weights and tensors across multiple cores while pipeline parallelism [18, 24] splits different layers across devices with micro-batches pipelined to the different layers. To enable continued scaling of neural networks, improving model training and serving efficiency has become a critical research area.\n\nConditional Computation: Computation decisions can be made dynamically based on the input [23, 25]. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation, by activating certain parameters and computation on demand, on a per-example or per-token basis [3]. Conditional convolution layers [1] with task-specific gating has been used to combat catastrophic forgetting when a sequence of learning problems are optimized. The gating decisions may be binary or sparse and continuous, stochastic or deterministic.\n\nMixture of Experts: Sparsely-gated MoE [31] is the first model to demonstrate massive improvements in model capacity, training time, or model quality with gating. Switch Transformer [10] simplifies the gating by selecting only the top expert per token using a softmax over the hidden state and demonstrates better scaling than previous work. All the prior work requires an auxiliary loss to explicitly encourage balancing. This loss term has to be carefully weighted to not overwhelm the primary loss. However, auxiliary loss does not guarantee balancing and a hard capacity factor has to be imposed. As a result, many tokens can still be unprocessed by the MoE layer. Hard MoE [12] with a single decoding layer can be efficiently trained to good effect on large scale hashtag prediction tasks.\n\nBase Layers [22] formulate a linear assignment that maximizes token-expert affinities while ensuring each expert receives an equal number of tokens. Hash layers [8, 28] devise hashing techniques on input tokens. However, the evaluations are limited to pre-training perplexity. THOR [? ] randomly activates experts during training and inference and is trained with a consistency regularization loss.\n\nTHOR has demonstrated strong performance on translation tasks. Different from these prior works, our method is a learnt method that enables heterogeneous MoE and effectively improves downstream fine-tuning performance.\n\n3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n5 Conclusion\n\nWe propose a new routing method for sparsely activated mixture-of-experts (MoE) models. This method addresses load imbalance and under-utilization of experts in conventional MoE methods, and enables selecting different numbers of experts for each token. Our model demonstrates more than 2x training efficiency improvements when compared to the state-of-the-art GShard and Switch Transformer models, and also achieves strong gains when finetuning on 11 datasets in the GLUE and SuperGLUE benchmark.\n\n6 Limitations\n\nThe expert choice method might not immediately apply to auto-regressive text generation as our current implementation takes in the past and future tokens to perform the top-k selection. One possible solution is to collect a large batch of input sequences, dispatch tokens of the same sequence into separate groups, and perform expert choice routing for each group. Another scenario where the expert choice method does not immediately apply is when the batch size becomes very small during serving or inference. A global top-k can be selected instead and we can cap the number of times each expert or token gets selected. We leave these possible improvements for future work.\n\nAnother long-standing issue with MoE has been the large memory footprint. Even though computational cost can be reduced using sparsely gated networks, the total number of parameters increases linearly or sub-linearly with the number of experts. Increasing the number of experts requires reservation of a large number of hardware devices. Therefore, dynamic (used) power is saved while static (reserved) power is not. Power saving techniques such as the ability to put hardware devices into low power states while not in use [17] can help with reducing the reserved power requirements.\n\nReferences\n\n[1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pages 3930–3939. Computer Vision Foundation / IEEE, 2020.\n\n[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. In Advances in Neural Information Processing Systems.\n\n[3] Kyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning, 2014.\n\n[4] Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems, 2021.\n\n[5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.\n\nTransformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.\n\n[6] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 933–941. JMLR.org, 2017.\n\n[7] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixtureof-experts, 2021.\n\n[8] Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan.\n\nTricks for training sparse translation models, 2021.\n\n[9] Richard L Dykstra. An iterative procedure for obtaining i-projections onto the intersection of convex sets. The annals of Probability, pages 975–984, 1985.\n\n[10] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.\n\n[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. NAS-FPN: learning scalable feature pyramid architecture for object detection. In CVPR, pages 7036–7045. Computer Vision Foundation / IEEE, 2019.\n\n[12] Sam Gross, Marc'Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision, 2017.\n\n[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.\n\n[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 630–645, Cham, 2016. Springer International Publishing.\n\n[15] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2016.\n\n[16] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.\n\n[17] Ping Huang, Zuocheng Xing, Tianran Wang, Qiang Wei, Hongyan Wang, and Guitao Fu. A brief survey on power gating design. In 2010 10th IEEE International Conference on Solid-State and Integrated Circuit Technology, pages 788–790, 2010.\n\n[18] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 103–112, 2019.\n\n[19] Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David A. Patterson. A domain-specific supercomputer for training deep neural networks. Commun. ACM, 63(7):67–78, 2020.\n\n[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.\n\n[21] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.\n\n[22] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6265–6274. PMLR, 18–24 Jul 2021.\n\n[24] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. New York, NY, USA, 2019. Association for Computing Machinery.\n\n[25] Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, Cédric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. In ICLR. OpenReview.net, 2021.\n\n[26] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.\n\n[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.\n\n[28] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models, 2021.\n\n[30] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 10435–10444, Red Hook, NY, USA, 2018. Curran Associates Inc.\n\n[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E.\n\nHinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In ICLR (Poster). OpenReview.net, 2017.\n\n[32] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 10–15 Jul 2018. [23] Min Lin, Jie Fu, and Yoshua Bengio. Conditional computation for continual learning, 2019.\n\n[29] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.\n\n[33] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.\n\n[34] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc.\n\n[35] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.\n\nGLUE: A multi-task benchmark and analysis platform for natural language understanding.\n\nIn Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, November 2018. Association for Computational Linguistics.\n\n[36] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake A. Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen. GSPMD: general and scalable parallelization for ML computation graphs. CoRR, abs/2105.04663, 2021.\n\n7 Checklist\n\n(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Yes (b) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes (c) Did you discuss any potential negative societal impacts of your work? N/A. Not any. (d) Did you describe the limitations of your work? Yes (a) Did you include the code, data, and instructions needed to reproduce the main experimental results? Yes. We include details in the experiment setup to help reproduce the main results. (b) Did you specify all the training details? Yes (c) Did you report error bars? Yes (d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? Yes (a) If your work uses existing assets, did you cite the creators? Yes (b) Did you mention the license of the assets? No. The used dataset is not released yet. (c) Did you include any new assets either in the supplemental material or as a URL? No. The dataset is not released yet.\n\n(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? No. Not using persons' data. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes. The dataset does not contain any personally identifiable information or offensive content.\n\nA Comparison On Fine-Tuning With A Dense Model\n\nOur 8B MoE model achieves stronger pre-training perplexity than its dense counterpart. However, a better perplexity does not always directly translate to downstream performance as demonstrated in Section 4.4. To this end, we compare fine-tuning performance of the 8B dense model and MoE model in Table 1. As shown in the table, our MoE model using expert choice routing consistently outperforms the dense model across the 11 tasks in GLUE and SuperGLUE.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 1: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nB Capacity Factor\n\nWe evaluate the downstream task fine-tuning performance by varying the capacity factors. Note that a capacity factor of n indicates on average how many experts each token can be received. EC-CF2 is our baseline expert choice, which matches GShard top-2 gating computational footprint. EC-CF1, however, matches Switch Transformer top-1 gating computational footprint. EC-CF0.5 further verifies that an aggressively lowered capacity factor can provide strong enough performance, that almost matches the top-2 gating baseline.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Top-2 78.1 87.0 88.3 85.0 82.6 90.1 90.7 81.6 94.7 68.2 67.2 83.0±0.3 EC-CAP2 78.2 88.0 88.5 85.7 83.0 90.8 91.1 80.0 95.4 70.4 64.1 83.2±0.4 EC-CAP3 78.5 91.7 89.3 86.3 83.5 90.9 91.1 81.8 94.9 70.0 65.6 84.0±0.4 EC-CF2 79.1 89.6 89.3 86.8 84.3 91.3 91.2 81.1 95.2 68.1 68.0 84.0±0.2 EC-CF1 77.4 90.6 88.0 85.5 83.6 90.3 91.2 79.8 95.3 66.5 64.9 83.0±0.2 EC-CF0.5 77.4 89.6 86.3 85.2 82.7 91.7 91.0 79.6 94.9 67.3 63.5 83.0 ±0.05 Hash Layers 76.1 85.2 86.7 83.4 82.5 90.0 90.3 75.7 94.0 67.4 63.3 81.3±1.0\n\nTable 2: Comparison between different routing methods in fine-tuning of 100M/64E models. We perform 3 independent fine-tuning runs for each method and report the average results. This gives more accurate difference between the variants of expert choice method, since they achieve close fine-tuning results. We do not report averaged results in other experiments.\n\nC Capped Expert Choice\n\nAs described in Section 4.5, the maximum number of experts each token is assigned can be capped by an entropy-regularized linear programming. Figure 1 compares the validation perplexity when training the 100M/64E models using the base expert choice method (EC-BASE), expert choice capped by two experts per token (EC-CAP2), expert choice capped by three experts per token (EC-CAP3), and GShard top-2 gating.\n\nAs shown in the figure, restricting the number of experts to 2 degrades the perplexity compared to the base expert choice method. This suggests that a more flexible allocation of experts (e.g. more than 2 experts for a token) can enhance model expressiveness. On the other hand, our EC-CAP2 and EC-CAP3 methods still outperform the top-2 gating method by a clear margin. We believe this confirms the effectiveness of a load balanced training, provided by our method. Finally, EC-CAP3 obtains comparable perplexity to EC-BASE. As indicated by Figure 3, only a little fraction of tokens use more than 3 experts therefore we see little or no difference between EC-BASE and EC-CAP3 variants. We present the fine-tuning results of these methods in Table 2.\n\nD Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [? ]. We use mod x to map a token ID to an expert ID. This in some way ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 2. Hashing based routing performs much worse than expert choice in terms of average scores and variance.\n\nE Fine-Tuning Details\n\nWe did a hyperparameter search for both baseline models and expert choice method. For fine-tuning of the 8B dense model, we use a constant learning rate of 0.0001 and a dropout rate of 0.1. We freeze the attention layer and feed-forward layer while leaving the embedding and layer normalization trainable. This setting has been found optimal for the 8B dense model. For MoE 8B/64E models including GShard top-2 gating and expert choice, we found continuing the learning rate from the pre-trained model while using a square root learning rate decay works better. In addition, we do not apply parameter freezing for fine-tuning MoE models. For models with 100M expert size, we use a constant learning rate of 0.0001 and no dropout is used.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/2202.09368v2.md"
+          },
+          "headlines": {
+            "1 Introduction": [],
+            "2 Related Work": [],
+            "3 Method": [
+              "3.1 Pitfalls Of Token-Choice Routing",
+              "3.2 Heterogeneous Moe Via Expert Choice",
+              "3.3 Expert Choice With Additional Constraint",
+              "3.4 Model Architecture"
+            ],
+            "4 Experiments": [
+              "4.1 Setup",
+              "4.2 Training Efficiency",
+              "4.3 Scaling The Number Of Experts",
+              "4.4 Fine-Tuning On Glue And Superglue",
+              "4.5 Heterogeneity Matters",
+              "4.6 Comparison With Hash Layer",
+              "4.7 Ablation"
+            ],
+            "5 Conclusion": [],
+            "6 Limitations": [],
+            "A Comparison On Fine-Tuning With A Dense Model": [],
+            "B Capacity Factor": [],
+            "C Capped Expert Choice": [],
+            "D Comparison With Hash Layer": [],
+            "E Fine-Tuning Details": []
+          },
+          "summary": "The paper introduces a new routing method for sparsely-activated Mixture-of-Experts (MoE) models called 'expert choice,' which addresses load imbalance and under-utilization issues in conventional MoE methods. Unlike traditional token-choice routing, expert choice allows experts to select the top-k tokens, ensuring perfect load balancing and enabling variable expert allocation per token. This method improves training efficiency, achieving over 2x faster convergence compared to existing models like Switch Transformer and GShard. It also demonstrates superior performance on downstream tasks in the GLUE and SuperGLUE benchmarks. The study highlights the benefits of flexible expert allocation and shows that expert choice outperforms hashing-based routing methods. The paper also discusses potential limitations and future improvements for the expert choice method.",
+          "summary_embedding": [
+            -0.009726986289024353,
+            -0.0337262786924839,
+            -0.023302342742681503,
+            -0.019878201186656952,
+            -0.025953780859708786,
+            0.04560471698641777,
+            -0.006874797400087118,
+            0.039665497839450836,
+            -0.006973279174417257,
+            0.034544438123703,
+            -0.025787118822336197,
+            -0.00363057479262352,
+            -0.01052999310195446,
+            0.0054657477885484695,
+            -0.024499278515577316,
+            0.036786794662475586,
+            0.010007280856370926,
+            0.003272630739957094,
+            0.006102093029767275,
+            -0.04048365354537964,
+            -0.018666116520762444,
+            -0.013522328808903694,
+            -0.03230207785964012,
+            0.008969432674348354,
+            0.02484775334596634,
+            0.0016713523073121905,
+            0.0026230288203805685,
+            0.003624893259257078,
+            -0.05311964824795723,
+            0.04339266195893288,
+            0.018681267276406288,
+            0.023711422458291054,
+            -0.016332851722836494,
+            -0.012832955457270145,
+            -0.039120059460401535,
+            -0.0197418425232172,
+            0.030786970630288124,
+            0.039968520402908325,
+            -0.019287310540676117,
+            0.03357476741075516,
+            0.018060073256492615,
+            0.02454473078250885,
+            -0.021272100508213043,
+            -0.029817303642630577,
+            -0.0061892117373645306,
+            -0.007840678095817566,
+            -0.008355814032256603,
+            0.01242387667298317,
+            0.016984347254037857,
+            -0.013007191941142082,
+            0.0212569497525692,
+            -0.01371171697974205,
+            0.03384748846292496,
+            0.008916404098272324,
+            -0.042332086712121964,
+            0.03984731063246727,
+            -0.008007340133190155,
+            0.012105703353881836,
+            0.03281721472740173,
+            0.010333028621971607,
+            -0.005287723150104284,
+            -0.021544819697737694,
+            -0.03378688171505928,
+            0.0010529992869123816,
+            0.00014014738553669304,
+            0.005950582213699818,
+            -0.003069985192269087,
+            -0.009742137044668198,
+            0.03227177634835243,
+            -0.003939277958124876,
+            -0.0740584209561348,
+            0.050089433789253235,
+            -0.026590125635266304,
+            -0.007628562860190868,
+            -0.028938541188836098,
+            -0.01736312545835972,
+            0.037362534552812576,
+            0.005662711802870035,
+            0.0098709212616086,
+            0.0106284748762846,
+            -0.0007892759749665856,
+            0.028923390433192253,
+            0.00910579226911068,
+            -0.024923507124185562,
+            0.04478655755519867,
+            -0.04790768027305603,
+            0.01801462098956108,
+            -0.04336235672235489,
+            0.022756904363632202,
+            1.6393929399782792e-05,
+            -0.020726662129163742,
+            -0.0013304532039910555,
+            -0.016666175797581673,
+            0.0218023881316185,
+            -0.01730252057313919,
+            0.026802239939570427,
+            0.013825350441038609,
+            -0.015802564099431038,
+            -0.029817303642630577,
+            0.025378039106726646,
+            0.003367325058206916,
+            0.010499690659344196,
+            -0.02075696364045143,
+            0.014332910999655724,
+            0.04393810033798218,
+            -0.02593863010406494,
+            -0.02271145209670067,
+            0.027681002393364906,
+            0.014408666640520096,
+            0.0633314698934555,
+            0.036544378846883774,
+            0.006624804809689522,
+            0.022362977266311646,
+            0.020575150847434998,
+            -0.028968842700123787,
+            -0.01963578537106514,
+            -0.046362269669771194,
+            -0.04757435619831085,
+            0.019878201186656952,
+            -0.04566532000899315,
+            0.0050869714468717575,
+            0.038544319570064545,
+            -0.013704141601920128,
+            -0.014446543529629707,
+            -0.04021093621850014,
+            0.017847958952188492,
+            0.0003908502112608403,
+            0.011976920068264008,
+            -0.0022082682698965073,
+            0.00033285003155469894,
+            -0.011825408786535263,
+            0.008878526277840137,
+            -0.03778676688671112,
+            -0.025196226313710213,
+            0.025181075558066368,
+            0.022862963378429413,
+            0.024120500311255455,
+            0.028802180662751198,
+            -0.03077181987464428,
+            -0.009984553791582584,
+            0.021378157660365105,
+            -0.04242299124598503,
+            -0.006556625012308359,
+            0.011332998983561993,
+            0.0026703758630901575,
+            0.003872992005199194,
+            0.032786913216114044,
+            0.052998438477516174,
+            -0.03172633796930313,
+            0.02065090648829937,
+            -0.010431510396301746,
+            0.04957429692149162,
+            0.00024691506405360997,
+            0.028968842700123787,
+            -0.019514575600624084,
+            -0.0134389977902174,
+            -0.029529431834816933,
+            -0.009795165620744228,
+            -0.01356020662933588,
+            -0.008484598249197006,
+            -0.002998017705976963,
+            -0.009575475007295609,
+            0.0040453351102769375,
+            0.0331808403134346,
+            0.00273855566047132,
+            0.030680913478136063,
+            0.014325335621833801,
+            0.0051475754007697105,
+            -0.08745196461677551,
+            0.026059838011860847,
+            -0.0033616432920098305,
+            0.005859675817191601,
+            0.0062460280023515224,
+            0.03013547509908676,
+            -0.0312718041241169,
+            0.020938776433467865,
+            0.026620427146553993,
+            0.0011145505122840405,
+            0.016969196498394012,
+            0.036847397685050964,
+            -0.04181694984436035,
+            0.009643655270338058,
+            0.012590537779033184,
+            -0.0033881578128784895,
+            0.04584713280200958,
+            0.011628445237874985,
+            0.019514575600624084,
+            0.018529755994677544,
+            0.003569970605894923,
+            0.00297907879576087,
+            0.006533898413181305,
+            0.03724132478237152,
+            0.003462019143626094,
+            -0.009833043441176414,
+            0.014976831153035164,
+            -0.010757259093225002,
+            -0.011219366453588009,
+            -0.037998881191015244,
+            0.014454119838774204,
+            0.008742166683077812,
+            -0.017454031854867935,
+            -0.02959003672003746,
+            0.0026949963066726923,
+            -0.031908150762319565,
+            0.026559824123978615,
+            -0.002445003716275096,
+            0.005738467443734407,
+            -0.010454237461090088,
+            -0.006757376715540886,
+            -0.035423196852207184,
+            0.028453707695007324,
+            -0.02095392718911171,
+            0.0053142374381423,
+            0.0007258309051394463,
+            -0.029438525438308716,
+            0.0060073984786868095,
+            0.017863109707832336,
+            -0.0025965142995119095,
+            0.0025302283465862274,
+            -0.02349930815398693,
+            0.024468975141644478,
+            0.025635607540607452,
+            0.004518806003034115,
+            -0.014037464745342731,
+            -0.013113250024616718,
+            -0.003605954349040985,
+            -0.009742137044668198,
+            -0.016196493059396744,
+            0.024468975141644478,
+            -0.014810169115662575,
+            0.00308892410248518,
+            0.021847840398550034,
+            0.014264730736613274,
+            -0.007045246660709381,
+            -0.004007457755506039,
+            -0.04066546633839607,
+            -0.01218903437256813,
+            -0.04760465770959854,
+            -0.011628445237874985,
+            -0.0034601252991706133,
+            -0.028696123510599136,
+            0.0061513339169323444,
+            -0.04481685906648636,
+            -0.01235569640994072,
+            0.02309022843837738,
+            0.018135828897356987,
+            0.0019488062243908644,
+            -0.009173971600830555,
+            -0.022605394944548607,
+            8.486965816700831e-05,
+            -0.01763584464788437,
+            -0.017044952139258385,
+            0.015393485315144062,
+            0.031059689819812775,
+            -0.03101423755288124,
+            0.04024123772978783,
+            0.003893824527040124,
+            -0.08066429197788239,
+            0.010393633507192135,
+            -0.026362858712673187,
+            0.07011914253234863,
+            -0.0025453795678913593,
+            0.014075342565774918,
+            0.06781618297100067,
+            -0.012431452050805092,
+            -0.02007516659796238,
+            0.03142331540584564,
+            -0.008249756880104542,
+            -1.1311515663692262e-05,
+            -0.02884763479232788,
+            -0.02739313244819641,
+            -0.053271155804395676,
+            -0.009704259224236012,
+            -0.010787560604512691,
+            0.03360506892204285,
+            -0.024317465722560883,
+            -0.0007656024536117911,
+            0.012620840221643448,
+            0.02983245439827442,
+            0.016635872423648834,
+            -0.0015681355725973845,
+            0.008272483013570309,
+            0.02881733328104019,
+            0.0274234339594841,
+            -0.0036154238041490316,
+            0.00814369972795248,
+            -0.034211114048957825,
+            -0.0068066176027059555,
+            -0.0493621826171875,
+            0.011302697472274303,
+            -0.02596893161535263,
+            -0.02596893161535263,
+            0.007643714081496,
+            -0.011476934887468815,
+            0.03748374432325363,
+            -0.024696242064237595,
+            -0.007503566797822714,
+            0.005037730094045401,
+            0.012855681590735912,
+            -0.017757052555680275,
+            -0.03612014651298523,
+            -0.0028502948116511106,
+            0.0026381798088550568,
+            0.007560383062809706,
+            -0.053240854293107986,
+            0.004499867558479309,
+            -0.018469152972102165,
+            0.0279688723385334,
+            -0.02139330841600895,
+            -0.013688990846276283,
+            0.042029064148664474,
+            0.011363301426172256,
+            0.027499189600348473,
+            -0.026468917727470398,
+            0.0017849853029474616,
+            0.012825379148125648,
+            -0.0020056227222085,
+            0.032514192163944244,
+            -0.030574856325984,
+            -0.03257479891180992,
+            0.001548249856568873,
+            -0.007840678095817566,
+            -0.0625436082482338,
+            -0.011386028490960598,
+            -0.00981789268553257,
+            0.017317671328783035,
+            0.0446956530213356,
+            0.04557441547513008,
+            -0.01068150345236063,
+            0.002174178371205926,
+            -0.02213571034371853,
+            -0.04542290419340134,
+            -0.02643861435353756,
+            0.0016486257081851363,
+            -0.019999410957098007,
+            -0.02789311669766903,
+            -0.0018313854234293103,
+            0.011211791075766087,
+            -0.013651113025844097,
+            0.01134057529270649,
+            -0.018560059368610382,
+            -0.01862066239118576,
+            -0.012332970276474953,
+            0.018575210124254227,
+            0.009901223704218864,
+            -0.0018796793883666396,
+            -0.015037435106933117,
+            -0.00980274099856615,
+            0.018969137221574783,
+            -0.011279970407485962,
+            0.0009388927719555795,
+            0.01188601367175579,
+            0.024347767233848572,
+            -0.007083124481141567,
+            0.010961798019707203,
+            -0.034756552428007126,
+            -0.01966608688235283,
+            0.007984613068401814,
+            0.002719616750255227,
+            0.020711511373519897,
+            0.008742166683077812,
+            -0.04275631532073021,
+            -0.044301725924015045,
+            0.016044981777668,
+            -0.0562104657292366,
+            0.028120383620262146,
+            -0.03327174484729767,
+            -0.012514783069491386,
+            -0.02671133354306221,
+            0.0013418165035545826,
+            -0.01828734017908573,
+            0.022438732907176018,
+            0.03778676688671112,
+            -0.0011486404109746218,
+            0.005503625608980656,
+            0.014658658765256405,
+            0.04384719207882881,
+            -0.05021063983440399,
+            -0.041180603206157684,
+            0.03754434734582901,
+            -0.01601468026638031,
+            0.0185146052390337,
+            0.00021744151308666915,
+            0.010348179377615452,
+            -0.014431392773985863,
+            0.022772056981921196,
+            0.013757170177996159,
+            -0.008764892816543579,
+            -0.030574856325984,
+            0.04945308715105057,
+            -0.023650817573070526,
+            -0.049513693898916245,
+            0.0018323323456570506,
+            0.018878230825066566,
+            0.018635815009474754,
+            -0.014870774000883102,
+            -0.02907489985227585,
+            -0.024317465722560883,
+            -0.006060427520424128,
+            -0.00947699323296547,
+            -0.03402930125594139,
+            0.011363301426172256,
+            0.0048445542342960835,
+            0.016666175797581673,
+            -0.00645056739449501,
+            -0.0073141781613230705,
+            -0.009370936080813408,
+            0.022559940814971924,
+            0.012635990977287292,
+            -0.03175663948059082,
+            -0.017484333366155624,
+            -0.02528713271021843,
+            0.004359719809144735,
+            0.010136065073311329,
+            0.022166013717651367,
+            0.024287162348628044,
+            -0.0323626808822155,
+            0.04218057543039322,
+            0.01550711877644062,
+            -0.008727015927433968,
+            -0.0019317612750455737,
+            0.02478714846074581,
+            0.04324115067720413,
+            0.0218023881316185,
+            0.026514369994401932,
+            0.002107892418280244,
+            8.705945947440341e-05,
+            -0.015620751306414604,
+            -0.0011391709558665752,
+            0.01862066239118576,
+            0.04911976307630539,
+            -0.012151156552135944,
+            0.01716616004705429,
+            -0.03578682243824005,
+            0.00396200455725193,
+            -0.02580226957798004,
+            -0.030786970630288124,
+            0.03745344281196594,
+            -0.015060162171721458,
+            0.0191964041441679,
+            0.025044716894626617,
+            -0.022469034418463707,
+            -0.011075431481003761,
+            -0.019423669204115868,
+            -0.026181045919656754,
+            -0.01702980138361454,
+            -0.03281721472740173,
+            -0.018135828897356987,
+            -0.002653330797329545,
+            -0.035635314881801605,
+            -0.04987731948494911,
+            -0.004329417832195759,
+            -0.046392571181058884,
+            -0.010507266037166119,
+            0.017378276214003563,
+            -0.011166337877511978,
+            -0.025711363181471825,
+            -0.005336964037269354,
+            0.003973367623984814,
+            -0.022196315228939056,
+            -0.0066588944755494595,
+            0.008439145050942898,
+            0.035938333719968796,
+            -0.02298417128622532,
+            -0.013060221448540688,
+            -0.0024317465722560883,
+            -0.02887793630361557,
+            -0.007878555916249752,
+            0.008999735116958618,
+            0.007723257411271334,
+            -0.0012357590021565557,
+            -0.05196816474199295,
+            -0.006916462909430265,
+            -0.012992041185498238,
+            0.029196109622716904,
+            0.026468917727470398,
+            0.0012168202083557844,
+            -0.02783251367509365,
+            0.008749742060899734,
+            -0.0072535742074251175,
+            0.02339324913918972,
+            -0.007992188446223736,
+            -0.022908415645360947,
+            -0.02034788578748703,
+            0.030786970630288124,
+            -0.0159086212515831,
+            0.009113367646932602,
+            0.009976978413760662,
+            0.0121360057964921,
+            -0.03899884968996048,
+            -0.001445980160497129,
+            0.0005648507503792644,
+            -0.010128489695489407,
+            0.00423093605786562,
+            -0.006738437805324793,
+            -0.007079336792230606,
+            -0.022090258076786995,
+            -0.0218023881316185,
+            0.03818069398403168,
+            -0.009113367646932602,
+            -0.0035661826841533184,
+            -0.01088604237884283,
+            0.03436262533068657,
+            -0.00595815759152174,
+            -0.01993880607187748,
+            -0.019347915425896645,
+            0.014431392773985863,
+            0.05105910077691078,
+            -0.028529461473226547,
+            -0.03984731063246727,
+            0.0011136035900563002,
+            0.030150625854730606,
+            0.004140029661357403,
+            -0.016908591613173485,
+            0.0035851215943694115,
+            0.056392278522253036,
+            -0.011742077767848969,
+            0.028362801298499107,
+            0.027771908789873123,
+            0.02295386977493763,
+            -0.02021152526140213,
+            -0.01678738370537758,
+            -0.020999381318688393,
+            0.011098157614469528,
+            -0.03027183562517166,
+            0.027059808373451233,
+            0.018969137221574783,
+            0.01538590993732214,
+            0.05054396390914917,
+            -0.03178694099187851,
+            -0.02515077404677868,
+            0.008499749936163425,
+            0.0021722845267504454,
+            0.011007251217961311,
+            -0.007727045100182295,
+            -0.01801462098956108,
+            -0.001950700068846345,
+            -0.014022313989698887,
+            -0.019817598164081573,
+            -0.020938776433467865,
+            0.00954517349600792,
+            0.015696506947278976,
+            0.002649543108418584,
+            0.010863316245377064,
+            0.0016590419691056013,
+            0.01686313934624195,
+            0.01321930717676878,
+            -0.0444229319691658,
+            -7.936555630294606e-05,
+            0.03378688171505928,
+            -0.0293476190418005,
+            -0.012264790013432503,
+            -0.016060132533311844,
+            -0.004439263138920069,
+            -0.01706010289490223,
+            -0.02762039750814438,
+            -0.006874797400087118,
+            -0.01889338344335556,
+            -0.00814369972795248,
+            -0.028196139261126518,
+            -0.009992130100727081,
+            0.01949942484498024,
+            0.022332675755023956,
+            0.001196934375911951,
+            -0.006783891003578901,
+            -0.005901341326534748,
+            0.02092362567782402,
+            0.032514192163944244,
+            0.01952972821891308,
+            0.0058331615291535854,
+            -0.015158643946051598,
+            -0.004295328166335821,
+            -0.012370847165584564,
+            0.016060132533311844,
+            0.012537509202957153,
+            6.344509893096983e-05,
+            0.03493836522102356,
+            0.004170331638306379,
+            -0.022726602852344513,
+            -0.001230077352374792,
+            0.024923507124185562,
+            -0.021590271964669228,
+            -0.0108330138027668,
+            0.040029123425483704,
+            0.0085679292678833,
+            -0.0024506852496415377,
+            -0.019287310540676117,
+            -0.019075196236371994,
+            0.02630225569009781,
+            -0.02133270539343357,
+            0.002083271974697709,
+            0.0482713058590889,
+            0.00035344602656550705,
+            -0.029802151024341583,
+            0.017954016104340553,
+            -0.013242034241557121,
+            -0.018560059368610382,
+            0.01848430372774601,
+            0.0012859469279646873,
+            -0.039968520402908325,
+            0.019014591351151466,
+            0.018408548086881638,
+            0.022635696455836296,
+            -0.02918095886707306,
+            0.0002127068000845611,
+            0.03233237937092781,
+            0.004037759732455015,
+            0.001595596899278462,
+            -0.008530051447451115,
+            -0.0006065162015147507,
+            0.010431510396301746,
+            0.001746160676702857,
+            0.02854461409151554,
+            -0.0030434709042310715,
+            0.027771908789873123,
+            -0.006727074272930622,
+            -0.008863375522196293,
+            0.019181253388524055,
+            -0.02109028771519661,
+            0.013166278600692749,
+            0.0685434341430664,
+            0.0021950111258774996,
+            0.025847723707556725,
+            0.007181606721132994,
+            0.024347767233848572,
+            0.02018122375011444,
+            -0.027271922677755356,
+            -0.02021152526140213,
+            -0.005397567991167307,
+            -0.0014033677289262414,
+            -0.03787767142057419,
+            0.011128460057079792,
+            -0.021302402019500732,
+            -0.013249609619379044,
+            -0.0022347825579345226,
+            -0.002316219499334693,
+            -0.006405114196240902,
+            0.034211114048957825,
+            -0.0014630251098424196,
+            -0.018560059368610382,
+            -0.035423196852207184,
+            0.037726160138845444,
+            0.04048365354537964,
+            0.0389382466673851,
+            -0.005984671879559755,
+            0.03548380360007286,
+            0.0030756669584661722,
+            0.0025302283465862274,
+            0.041453324258327484,
+            0.004791525658220053,
+            -0.011567841283977032,
+            0.02295386977493763,
+            0.012416300363838673,
+            0.03351416438817978,
+            -0.02045394293963909,
+            -0.02630225569009781,
+            -0.003299145260825753,
+            -0.026423463597893715,
+            0.012492056004703045,
+            -0.019726691767573357,
+            0.0018768386216834188,
+            0.007117214612662792,
+            0.015196521766483784,
+            -0.016044981777668,
+            0.014719262719154358,
+            -0.006102093029767275,
+            -0.021014532074332237,
+            0.011719351634383202,
+            -0.007780073676258326,
+            0.011961768381297588,
+            -0.02262054570019245,
+            0.001138224033638835,
+            -0.04063516482710838,
+            0.008280059322714806,
+            0.014022313989698887,
+            -0.01598437689244747,
+            0.03205966204404831,
+            0.020969079807400703,
+            -0.01801462098956108,
+            -0.0009682480012997985,
+            0.0034714885987341404,
+            -0.010393633507192135,
+            -0.011067855171859264,
+            -0.019923655316233635,
+            0.02786281518638134,
+            -0.015802564099431038,
+            -0.025908326730132103,
+            0.02454473078250885,
+            -0.01743887923657894,
+            0.015151068568229675,
+            -0.004299115855246782,
+            0.00888610165566206,
+            -0.019120648503303528,
+            0.02284781076014042,
+            -0.023741723969578743,
+            -0.0009128518868237734,
+            -0.020287280902266502,
+            0.0028389315120875835,
+            0.014135947450995445,
+            -0.018272189423441887,
+            -0.02139330841600895,
+            0.019150950014591217,
+            -0.005420294590294361,
+            -0.003931702114641666,
+            0.010014856234192848,
+            0.0030983935575932264,
+            -0.0029715034179389477,
+            -0.004549108445644379,
+            -0.03281721472740173,
+            -0.02915065549314022,
+            0.0033086147159337997,
+            0.011386028490960598,
+            0.015257125720381737,
+            0.027574945241212845,
+            0.03715042024850845,
+            0.006052851676940918,
+            0.0036949669010937214,
+            0.0009327376610599458,
+            -0.020726662129163742,
+            0.042635105550289154,
+            -0.020469093695282936,
+            0.040847279131412506,
+            0.032544493675231934,
+            -0.0197418425232172,
+            -0.023908386006951332,
+            0.0029052174650132656,
+            0.03115059621632099,
+            0.03151422366499901,
+            0.025211378931999207,
+            0.012310243211686611,
+            0.03690800443291664,
+            -0.004359719809144735,
+            -0.032211173325777054,
+            0.015802564099431038,
+            0.022256920114159584,
+            -0.008105821907520294,
+            0.009673957712948322,
+            0.027438584715127945,
+            0.013279912061989307,
+            -0.007757347077131271,
+            0.030150625854730606,
+            0.0012234487803652883,
+            0.02396899089217186,
+            0.02498411200940609,
+            -0.02190844528377056,
+            0.01743887923657894,
+            -0.0001154677156591788,
+            0.006727074272930622,
+            -0.009840618818998337,
+            0.004348356742411852,
+            0.011810258030891418,
+            0.018272189423441887,
+            0.018908534198999405,
+            0.003246116451919079,
+            -0.07060398161411285,
+            0.019726691767573357,
+            0.004507442936301231,
+            -0.02160542458295822,
+            0.00390140013769269,
+            -0.0024866692256182432,
+            0.012704171240329742,
+            0.02105998620390892,
+            0.012461753562092781,
+            -0.01790856383740902,
+            0.0013626492582261562,
+            0.011961768381297588,
+            -0.00595815759152174,
+            0.04730163514614105,
+            0.0019791084341704845,
+            -0.01437078882008791,
+            -0.016105586662888527,
+            0.013726867735385895,
+            0.02525683119893074,
+            -0.02355991117656231,
+            0.006325571332126856,
+            0.005621046293526888,
+            -0.004147605039179325,
+            -0.010742107406258583,
+            -0.030377892777323723,
+            -0.005280147306621075,
+            -0.03436262533068657,
+            0.023150833323597908,
+            0.015491968020796776,
+            0.014764715917408466,
+            -0.025044716894626617,
+            -0.004280176945030689,
+            0.0046400148421525955,
+            -0.021287251263856888,
+            0.003261267440393567,
+            0.01161329448223114,
+            0.0005104016163386405,
+            -0.020984230563044548,
+            0.00959820207208395,
+            0.013469300232827663,
+            0.06848283112049103,
+            0.0007442962378263474,
+            0.029771849513053894,
+            0.0422411784529686,
+            -0.0012243957025930285,
+            -0.006371024530380964,
+            0.0036286809481680393,
+            -0.019953956827521324,
+            0.003473382443189621,
+            0.025878025218844414,
+            -0.03224147483706474,
+            -0.055058982223272324,
+            -0.007545232307165861,
+            -0.04624105989933014,
+            0.001489539397880435,
+            0.014711687341332436,
+            0.023044776171445847,
+            -0.0037782976869493723,
+            -0.034817155450582504,
+            0.016908591613173485,
+            -0.003071879269555211,
+            0.007014944683760405,
+            -0.02593863010406494,
+            -0.0294839795678854,
+            -0.026014383882284164,
+            -0.02339324913918972,
+            -0.012908710166811943,
+            -0.022726602852344513,
+            0.012590537779033184,
+            -0.0015103721525520086,
+            -0.06587684899568558,
+            -0.009780014865100384,
+            0.016166189685463905,
+            0.001200722181238234,
+            0.011939042247831821,
+            0.011083006858825684,
+            -0.007083124481141567,
+            -0.0018275976181030273,
+            -0.022559940814971924,
+            0.03124150261282921,
+            0.005177877843379974,
+            0.03584742918610573,
+            -0.006503595970571041,
+            -0.009128518402576447,
+            0.01073453202843666,
+            0.04527139291167259,
+            0.0017916138749569654,
+            0.010226971469819546,
+            -0.020560000091791153,
+            0.0015065843472257257,
+            -0.02284781076014042,
+            -0.0031135445460677147,
+            -0.008863375522196293,
+            -0.0021211495622992516,
+            0.015893470495939255,
+            0.0019753205124288797,
+            -0.010204244405031204,
+            0.025438643991947174,
+            0.02251448854804039,
+            -0.009605777449905872,
+            -0.015969226136803627,
+            -0.02437806874513626,
+            -0.014545026235282421,
+            -0.0002627763315103948,
+            -0.010007280856370926,
+            -0.022453883662819862,
+            -0.020878173410892487,
+            -0.002566212322562933,
+            -0.02559015527367592,
+            0.022529639303684235,
+            0.01220418605953455,
+            -4.882668508798815e-05,
+            -0.006723286584019661,
+            -0.038544319570064545,
+            -0.003727162955328822,
+            -0.03118089959025383,
+            -0.005374841392040253,
+            0.03287781774997711,
+            -0.03745344281196594,
+            -0.0002231231628684327,
+            0.014491996727883816,
+            -0.01993880607187748,
+            0.0028938541654497385,
+            0.008961857296526432,
+            0.02769615314900875,
+            -0.027408283203840256,
+            0.0001795638381736353,
+            0.006954340264201164,
+            0.016908591613173485,
+            -0.009878496639430523,
+            0.027196168899536133,
+            -0.04275631532073021,
+            0.011037553660571575,
+            -0.037120118737220764,
+            -0.020423639565706253,
+            -0.01523439958691597,
+            0.008363389410078526,
+            -0.005018791649490595,
+            -0.02637801133096218,
+            -0.010454237461090088,
+            0.006026337388902903,
+            -0.00693918950855732,
+            -0.01902974210679531,
+            -0.010620899498462677,
+            0.027438584715127945,
+            0.00931033119559288,
+            0.007977037690579891,
+            -0.02955973520874977,
+            -0.01631770096719265,
+            0.006280118133872747,
+            0.052089374512434006,
+            -0.01584801822900772,
+            0.014938953332602978,
+            -0.006639956030994654,
+            0.02342355251312256,
+            -0.028484009206295013,
+            -0.01872672140598297,
+            0.0315445251762867,
+            0.013946558348834515,
+            0.007064185570925474,
+            0.028953691944479942,
+            0.027196168899536133,
+            0.027105262503027916,
+            -0.04654408246278763,
+            0.009886072017252445,
+            -0.002613559365272522,
+            -0.008757317438721657,
+            0.03402930125594139,
+            0.01916610263288021,
+            0.022150862962007523,
+            0.0009744031121954322,
+            -0.01730252057313919,
+            0.0139844361692667,
+            0.014885924756526947,
+            0.007681591901928186,
+            -0.014242004603147507,
+            -0.02660527639091015,
+            0.003367325058206916,
+            -0.007992188446223736,
+            0.00047276070108637214,
+            0.020726662129163742,
+            -0.009052763693034649,
+            0.030999086797237396,
+            -0.014605630189180374,
+            -0.02593863010406494,
+            0.008916404098272324,
+            -0.02268115058541298,
+            -0.0041741193272173405,
+            0.0004476667381823063,
+            0.010083036497235298,
+            -0.023847781121730804,
+            -0.04669559374451637,
+            0.019408518448472023,
+            0.028393102809786797,
+            0.005579380784183741,
+            0.01328748743981123,
+            -0.035332292318344116,
+            0.008992158807814121,
+            0.012310243211686611,
+            0.011348150670528412,
+            0.0019847899675369263,
+            0.035362593829631805,
+            -0.00028834378463216126,
+            0.012885984033346176,
+            -0.008620957843959332,
+            0.028362801298499107,
+            0.00667025800794363,
+            -0.0323626808822155,
+            -0.0378473699092865,
+            -0.01513591781258583,
+            0.015257125720381737,
+            0.006692984607070684,
+            -0.011840560473501682,
+            -0.02278720773756504,
+            -0.038817036896944046,
+            0.015787413343787193,
+            -0.012916285544633865,
+            -0.038271598517894745,
+            -0.010158791206777096,
+            -0.009742137044668198,
+            0.016635872423648834,
+            -0.006916462909430265,
+            -0.016802534461021423,
+            -0.016090434044599533,
+            0.005821797996759415,
+            7.161247049225494e-05,
+            0.0244083721190691,
+            -0.001251857029274106,
+            -0.006855858489871025,
+            -0.016393456608057022,
+            -0.01188601367175579,
+            -0.0035074723418802023,
+            0.03602924197912216,
+            -0.014219277538359165,
+            -0.02383263036608696,
+            0.014984406530857086,
+            0.02657497487962246,
+            0.011636020615696907,
+            0.03227177634835243,
+            0.007181606721132994,
+            -0.014022313989698887,
+            0.003016956616193056,
+            -0.033029329031705856,
+            -0.0046400148421525955,
+            0.005196816753596067,
+            -0.006427840795367956,
+            -0.015651054680347443,
+            -0.01815097965300083,
+            -0.015832867473363876,
+            0.030756669119000435,
+            -0.02487805485725403,
+            0.0218023881316185,
+            0.030347591266036034,
+            0.003174148965626955,
+            -0.02915065549314022,
+            0.013598084449768066,
+            -0.0072005451656877995,
+            0.01862066239118576,
+            0.0019014590652659535,
+            0.028014326468110085,
+            -0.007461901288479567,
+            -0.004818039946258068,
+            -0.02501441352069378,
+            -0.03696860745549202,
+            -0.023484155535697937,
+            -0.008817922323942184,
+            0.0027537066489458084,
+            0.020984230563044548,
+            -0.0375746488571167,
+            -0.017757052555680275,
+            0.009855770505964756,
+            0.015757111832499504,
+            -0.03027183562517166,
+            -0.0025908327661454678,
+            -0.010158791206777096,
+            -0.0004055278259329498,
+            0.005140000022947788,
+            -0.02227207086980343,
+            -0.008977008052170277,
+            -0.015726808458566666,
+            -0.0010435298318043351,
+            0.006166485138237476,
+            -0.0007409819518215954,
+            -0.05805889517068863,
+            -0.013969285413622856,
+            0.018635815009474754,
+            0.0026400736533105373,
+            0.008908828720450401,
+            0.015938924625515938,
+            -0.0111739132553339,
+            0.0031533162109553814,
+            -0.026756787672638893,
+            0.023893235251307487,
+            -0.009492144919931889,
+            0.009355784393846989,
+            -0.013742019422352314,
+            0.012514783069491386,
+            0.014014738611876965,
+            0.004795313347131014,
+            0.0010681503918021917,
+            -0.0054051438346505165,
+            0.01601468026638031,
+            0.028377952054142952,
+            -0.023756874725222588,
+            0.011454207822680473,
+            -0.0002514130319468677,
+            -0.0032158142421394587,
+            -0.029196109622716904,
+            0.00509454682469368,
+            0.003931702114641666,
+            -0.005109698045998812,
+            0.016651025041937828,
+            -0.036544378846883774,
+            0.01662072166800499,
+            0.02177208475768566,
+            0.032544493675231934,
+            0.014832896180450916,
+            -0.017287369817495346,
+            -0.011658747680485249,
+            -0.012385998852550983,
+            -0.002624922664836049,
+            0.022181164473295212,
+            0.012120855040848255,
+            -0.01815097965300083,
+            -0.024287162348628044,
+            0.011249667964875698,
+            0.0005283934879116714,
+            -0.01203752402216196,
+            0.010795135982334614,
+            -0.013166278600692749,
+            0.01349202636629343,
+            0.012711746618151665,
+            0.00866641104221344,
+            0.00888610165566206,
+            0.022862963378429413,
+            -0.011878437362611294,
+            -0.017711598426103592,
+            0.00656041270121932,
+            -0.008916404098272324,
+            -0.0034089903347194195,
+            -0.012598113156855106,
+            0.010795135982334614,
+            0.014878349378705025,
+            0.06490717828273773,
+            -0.013143551535904408,
+            0.0009526234352961183,
+            -0.020438792183995247,
+            -0.031938452273607254,
+            -0.0017944546416401863,
+            -0.020908474922180176,
+            0.0111739132553339,
+            0.008370965719223022,
+            0.01598437689244747,
+            0.012022373266518116,
+            -0.0021912232041358948,
+            0.00795431062579155,
+            0.019317612051963806,
+            -0.010423935018479824,
+            7.255941454786807e-05,
+            -0.006503595970571041,
+            0.0013683309080079198,
+            0.0035851215943694115,
+            -0.011719351634383202,
+            -0.0026949963066726923,
+            0.035695917904376984,
+            0.01068150345236063,
+            0.011651171371340752,
+            -0.03445352986454964,
+            0.0020435003098100424,
+            0.020150920376181602,
+            0.007602048572152853,
+            -0.008302785456180573,
+            -0.01563590206205845,
+            0.01706010289490223,
+            -0.015317730605602264,
+            -0.0022499337792396545,
+            0.002888172399252653,
+            -0.010151215828955173,
+            0.04581683129072189,
+            -0.019787294790148735,
+            -0.0011742077767848969,
+            0.031847547739744186,
+            0.0030207443051040173,
+            -0.0040983641520142555,
+            0.005408931523561478,
+            -0.003473382443189621,
+            -0.0069013116881251335,
+            -0.02139330841600895,
+            0.0031381649896502495,
+            0.00667025800794363,
+            -0.019287310540676117,
+            -0.0072005451656877995,
+            -0.007719469256699085,
+            0.006897523999214172,
+            0.023105379194021225,
+            -0.009492144919931889,
+            -0.00016145357221830636,
+            -0.008575504645705223,
+            0.009848194196820259,
+            -0.0011912527261301875,
+            -0.044301725924015045,
+            -0.006147546228021383,
+            0.021181194111704826,
+            0.000973456131760031,
+            0.0005757405888289213,
+            -0.007450537756085396,
+            0.005007428117096424,
+            -0.017423728480935097,
+            -0.008454295806586742,
+            0.02369627170264721,
+            0.01736312545835972,
+            -0.009113367646932602,
+            0.010863316245377064,
+            -0.0035680767614394426,
+            0.00433320552110672,
+            -0.020014561712741852,
+            -0.01095422264188528,
+            0.003462019143626094,
+            0.015272277407348156,
+            -0.0009819786064326763,
+            -0.029029447585344315,
+            -0.0012272365856915712,
+            -0.00024194362049456686,
+            0.02918095886707306,
+            0.01486319862306118,
+            -0.008961857296526432,
+            -0.021211495622992516,
+            -0.013242034241557121,
+            -0.001237652963027358,
+            -0.015726808458566666,
+            -0.009620928205549717,
+            0.011855711229145527,
+            0.0009137988090515137,
+            0.008992158807814121,
+            -0.018135828897356987,
+            -0.027347678318619728,
+            -0.016166189685463905,
+            0.0018465365283191204,
+            -0.012870832346379757,
+            0.003340810537338257,
+            0.015514694154262543,
+            -0.03430201858282089,
+            -0.015060162171721458,
+            -0.03805948421359062,
+            0.008280059322714806,
+            -0.0029658216517418623,
+            -0.02024182677268982,
+            -0.023605365306138992,
+            0.009484568610787392,
+            0.019181253388524055,
+            -0.0015548784285783768,
+            0.019847899675369263,
+            -0.000889651826582849,
+            0.020060013979673386,
+            -0.0073141781613230705,
+            0.0031703610438853502,
+            -0.013007191941142082,
+            0.0030245319940149784,
+            0.008333087898790836,
+            0.02890823967754841,
+            -0.023075077682733536,
+            -0.006503595970571041,
+            -0.004685468040406704,
+            0.004746072459965944,
+            0.012658718042075634,
+            0.014613205567002296,
+            0.020741812884807587,
+            -0.008272483013570309,
+            -0.0027612822595983744,
+            0.012484480626881123,
+            -0.003799130441620946,
+            0.007295239716768265,
+            0.002390081062912941,
+            0.0019677451346069574,
+            0.004162756260484457,
+            0.008477022871375084,
+            0.0053710537031292915,
+            0.027650700882077217,
+            -0.0028275682125240564,
+            0.0031703610438853502,
+            0.024620486423373222,
+            0.02322658896446228,
+            -0.018878230825066566,
+            -0.028074929490685463,
+            0.002024561632424593,
+            0.050695475190877914,
+            0.007245998363941908,
+            -0.006810405291616917,
+            -0.03927157074213028,
+            -0.0031760428100824356,
+            0.028635520488023758,
+            0.023650817573070526,
+            0.01584801822900772,
+            -0.010567869991064072,
+            -0.012098127976059914,
+            0.02772645466029644,
+            0.013295062817633152,
+            0.02616589516401291,
+            -0.025302285328507423,
+            0.010704229585826397,
+            -0.03869583085179329,
+            0.0449683703482151,
+            0.004439263138920069,
+            -0.014476845972239971,
+            0.0011884119594469666,
+            0.010386057198047638,
+            -0.004537744913250208,
+            0.006431628484278917,
+            -5.8059369621332735e-05,
+            -0.0008086882880888879,
+            0.011923890560865402,
+            0.012007221579551697,
+            0.004265025723725557,
+            -0.02399929240345955,
+            0.007359631359577179,
+            0.004272601567208767,
+            0.03005971945822239,
+            0.005431658122688532,
+            -0.0013863227795809507,
+            0.012385998852550983,
+            -0.005810434930026531,
+            0.008855799213051796,
+            -0.010719381272792816,
+            0.020469093695282936,
+            -0.02918095886707306,
+            1.0986003871948924e-05,
+            0.02051454596221447,
+            -0.0015785519499331713,
+            0.0038199631962925196,
+            -0.023741723969578743,
+            0.01186328660696745,
+            -0.052968136966228485,
+            0.021166043356060982,
+            -0.018741872161626816,
+            0.021741783246397972,
+            0.019590331241488457,
+            -0.009734561666846275,
+            -0.012052674777805805,
+            0.03545350208878517,
+            0.0026760573964565992,
+            -0.014264730736613274,
+            -0.010643625631928444,
+            -0.017999470233917236,
+            -0.02830219641327858,
+            0.041968461126089096,
+            -0.0006117243901826441,
+            0.0028275682125240564,
+            0.0024222771171480417,
+            -0.010666352696716785,
+            -0.008923979476094246,
+            0.001954487757757306,
+            0.020499395206570625,
+            -0.03369597718119621,
+            -0.011128460057079792,
+            -0.008689138107001781,
+            -0.02566591091454029,
+            -0.00020300064352340996,
+            -0.011961768381297588,
+            0.026211349293589592,
+            0.01240872498601675,
+            -0.03696860745549202,
+            0.003780191531404853,
+            0.023650817573070526,
+            0.001603172393515706,
+            0.018120678141713142,
+            0.002130619017407298,
+            0.0010795136913657188,
+            0.009742137044668198,
+            -0.010476963594555855,
+            0.026559824123978615,
+            -0.009204274043440819,
+            0.008658835664391518,
+            -0.001950700068846345,
+            -0.017287369817495346,
+            0.017969166859984398,
+            0.004083212930709124,
+            0.024120500311255455,
+            0.011954193003475666,
+            0.011605718173086643,
+            0.01112088467925787,
+            -0.019878201186656952,
+            0.020090317353606224,
+            -0.0018891488434746861,
+            0.009075489826500416,
+            0.024726543575525284,
+            -0.005863463506102562,
+            0.020832719281315804,
+            0.015257125720381737,
+            -0.00020229043730068952,
+            0.011295121163129807,
+            -0.020120618864893913,
+            -0.011779955588281155,
+            0.0085679292678833,
+            -0.009605777449905872,
+            -0.010325453244149685,
+            0.009901223704218864,
+            -0.010105762630701065,
+            -0.015052586793899536,
+            -0.0071247899904847145,
+            0.016635872423648834,
+            0.0028976418543606997,
+            -0.02610529027879238,
+            -1.7326075976598077e-05,
+            0.008901252411305904,
+            0.006166485138237476,
+            0.018741872161626816,
+            0.011317848227918148,
+            -0.006692984607070684,
+            0.010363331064581871,
+            0.011476934887468815,
+            0.0053104497492313385,
+            -0.005079395603388548,
+            -0.013431422412395477,
+            -0.0024241709616035223,
+            0.018090376630425453,
+            0.009810317307710648,
+            -0.01922670565545559,
+            0.004564259201288223,
+            0.0008877579239197075,
+            -0.01420412678271532,
+            -0.019347915425896645,
+            -0.0038483713287860155,
+            -0.004170331638306379,
+            0.0027915844693779945,
+            0.0509984977543354,
+            -0.0006216673064045608,
+            0.007893706671893597,
+            -0.025741664692759514,
+            -0.006299056578427553,
+            -0.00027556007262319326,
+            -0.0036097420379519463,
+            0.01272689737379551,
+            -0.01824188604950905,
+            -0.03724132478237152,
+            0.012711746618151665,
+            -0.0012584856012836099,
+            0.014923802576959133,
+            -0.00297907879576087,
+            0.014226853847503662,
+            -0.017347972840070724,
+            0.01916610263288021,
+            -0.0011117096291854978,
+            -0.016029831022024155,
+            0.0029374135192483664,
+            -0.011143610812723637,
+            -0.021317552775144577,
+            0.005916492082178593,
+            -0.006848283112049103,
+            -0.03896854817867279,
+            0.0029885482508689165,
+            -0.010423935018479824,
+            0.0011798894265666604,
+            -0.003912763670086861,
+            0.020166072994470596,
+            -0.0061513339169323444,
+            0.0008697660523466766,
+            0.02177208475768566,
+            0.0017026013229042292,
+            0.01601468026638031,
+            -0.014499573037028313,
+            0.010999675840139389,
+            0.012598113156855106,
+            0.02210540883243084,
+            -0.032180868089199066,
+            -0.01393140759319067,
+            0.014211702160537243,
+            0.0026116655208170414,
+            -0.005382417235523462,
+            0.020635755732655525,
+            -0.01294658798724413,
+            0.019650936126708984,
+            -0.0005577487172558904,
+            -0.011704200878739357,
+            -0.03524138405919075,
+            -0.016529815271496773,
+            -3.343888238305226e-05,
+            -0.008439145050942898,
+            0.025317436084151268,
+            0.01760554127395153,
+            -0.0108330138027668,
+            0.026468917727470398,
+            0.0037574649322777987,
+            0.014598054811358452,
+            -0.014257155358791351,
+            -0.005999823100864887,
+            -0.0003773563075810671,
+            -0.0119314668700099,
+            0.015264702029526234,
+            -0.03512017801403999,
+            0.0008342557121068239,
+            -0.00839369185268879,
+            0.005242269951850176,
+            -0.0254992488771677,
+            0.018257036805152893,
+            0.025817420333623886,
+            -0.0031419529113918543,
+            -0.005037730094045401,
+            -0.022802358493208885,
+            0.028968842700123787,
+            0.03049910068511963,
+            0.0024828813038766384,
+            -0.0013825349742546678,
+            -0.00352072948589921,
+            0.0033218718599528074,
+            0.0211205892264843,
+            -0.037998881191015244,
+            -0.009098216891288757,
+            -0.028696123510599136,
+            0.042332086712121964,
+            -0.008098246529698372,
+            -0.013431422412395477,
+            0.01862066239118576,
+            -0.0024715180043131113,
+            0.02355991117656231,
+            -0.004689255729317665,
+            -0.004855917301028967,
+            0.03148392215371132,
+            0.011514811776578426,
+            0.02772645466029644,
+            -0.04015033319592476,
+            0.014704111963510513,
+            -0.006867222022265196,
+            0.004867280833423138,
+            -0.013484450988471508,
+            0.005302873905748129,
+            0.02580226957798004,
+            -0.018060073256492615,
+            0.0068066176027059555,
+            0.030665762722492218,
+            0.00748084019869566,
+            0.002448791405186057,
+            0.0165904201567173,
+            0.003890036838129163,
+            0.010855740867555141,
+            0.000841831264551729,
+            0.0052574207074940205,
+            0.022423582151532173,
+            0.014454119838774204,
+            -0.016029831022024155,
+            -0.015120766125619411,
+            -0.00044790346873924136,
+            -0.004450626205652952,
+            0.005261208396404982,
+            0.013242034241557121,
+            0.0061172437854111195,
+            -0.015681356191635132,
+            0.01052999310195446,
+            0.014158673584461212,
+            0.013128400780260563,
+            -0.0039847311563789845,
+            -0.03412020578980446,
+            -0.02443867363035679,
+            -0.021438762545585632,
+            0.003104075090959668,
+            -0.0008281006012111902,
+            -0.0021211495622992516,
+            0.0017954016802832484,
+            0.017347972840070724,
+            -0.0008352026343345642,
+            0.00379155483096838,
+            0.01686313934624195,
+            -0.01415109820663929,
+            -0.022090258076786995,
+            -0.0035964848939329386,
+            -0.027529491111636162,
+            0.0053710537031292915,
+            0.001844642567448318,
+            -0.00965123064815998,
+            0.008530051447451115,
+            -0.0046059247106313705,
+            -0.007223271764814854,
+            -0.012196609750390053,
+            -0.02007516659796238,
+            0.005780132487416267,
+            0.016878290101885796,
+            0.005730891600251198,
+            -0.009272454306483269,
+            -0.003939277958124876,
+            0.011575416661798954,
+            0.017484333366155624,
+            0.017317671328783035,
+            -0.009295180439949036,
+            0.009628504514694214,
+            0.0004022135108243674,
+            0.004670316819101572,
+            -0.01834794320166111,
+            0.0045869858004152775,
+            -0.0027953721582889557,
+            -0.008658835664391518,
+            0.01105270441621542,
+            -0.00031201730598695576,
+            0.01628739945590496,
+            -0.013090522959828377,
+            0.005624833982437849,
+            0.010333028621971607,
+            -0.020590301603078842,
+            -0.0067308624275028706,
+            -0.01692374423146248,
+            0.026484068483114243,
+            -0.00504530593752861,
+            0.00045405857963487506,
+            -0.02978700026869774,
+            0.0033616432920098305,
+            0.000848933297675103,
+            -0.000845619011670351,
+            -0.012764775194227695,
+            0.010749682784080505,
+            0.03242328763008118,
+            0.0005179771687835455,
+            0.0216660276055336,
+            -0.0036154238041490316,
+            0.0007480839849449694,
+            -0.00785582885146141,
+            0.004977126140147448,
+            -0.004916521720588207,
+            -0.009901223704218864,
+            -0.010863316245377064,
+            0.021726632490754128,
+            -0.0033029329497367144,
+            0.0003356908564455807,
+            0.011332998983561993,
+            -0.012067826464772224,
+            0.0014620780711993575,
+            -0.0047687990590929985,
+            0.00971941091120243,
+            0.0061702728271484375,
+            0.009113367646932602,
+            -0.01176480483263731,
+            0.05451354384422302,
+            -0.0106284748762846,
+            0.005537715740501881,
+            -0.00439380994066596,
+            -0.022226616740226746,
+            -0.017317671328783035,
+            0.012711746618151665,
+            -0.017544938251376152,
+            0.03375658020377159,
+            0.024150803685188293,
+            0.0006467612111009657,
+            0.010393633507192135,
+            -0.006958128418773413,
+            0.023923536762595177,
+            -0.002602196065708995,
+            0.0010795136913657188,
+            -0.004855917301028967,
+            0.015802564099431038,
+            0.04511988162994385,
+            0.017393426969647408,
+            -0.001152428099885583,
+            0.00829521007835865,
+            0.020575150847434998,
+            0.00814369972795248,
+            -0.0020075165666639805,
+            0.014219277538359165,
+            -0.01024212222546339,
+            0.00900731049478054,
+            -0.004458202049136162,
+            0.027438584715127945,
+            0.00814369972795248,
+            -0.006310420110821724,
+            -0.0036836036015301943,
+            -0.0008209985098801553,
+            0.014173824340105057,
+            0.006950552575290203,
+            0.01963578537106514,
+            -0.00461350055411458,
+            -0.0012575386790558696,
+            0.013870803639292717,
+            0.014491996727883816,
+            -0.00552635220810771,
+            0.001724380999803543,
+            -0.01435563713312149,
+            -0.010204244405031204,
+            0.013575357384979725,
+            0.006086941808462143,
+            0.006310420110821724,
+            -0.014514723792672157,
+            -0.00980274099856615,
+            -0.004670316819101572,
+            0.0015435151290148497,
+            0.015052586793899536,
+            0.011060279794037342,
+            -0.004814252257347107,
+            0.006052851676940918,
+            0.013825350441038609,
+            -0.014014738611876965,
+            0.006147546228021383,
+            0.007208121009171009,
+            0.008855799213051796,
+            0.004787737503647804,
+            -0.010817863047122955,
+            -0.00428396463394165,
+            -0.0063937511295080185,
+            0.0012670081341639161,
+            0.016832837834954262,
+            0.013181429356336594,
+            0.0049581872299313545,
+            0.027529491111636162,
+            -0.00839369185268879,
+            -0.012302667833864689,
+            0.02339324913918972,
+            0.011257244274020195,
+            0.0031703610438853502,
+            0.014651083387434483,
+            -0.012635990977287292,
+            -0.012446602806448936,
+            0.006655106786638498,
+            0.004848341923207045,
+            0.006049063988029957,
+            0.009151245467364788,
+            0.00450365524739027,
+            0.007333117071539164,
+            -0.007098275702446699,
+            -0.007427811156958342,
+            0.0014810169814154506,
+            0.01642375811934471,
+            0.020393338054418564,
+            -0.024968961253762245,
+            -0.0018806264270097017,
+            0.0009052763343788683,
+            0.010878467001020908,
+            -0.003316190093755722,
+            0.012310243211686611,
+            5.65205882594455e-05,
+            -0.0074240234680473804,
+            0.010158791206777096,
+            -0.02119634486734867,
+            -0.012681444175541401,
+            0.0012717428617179394,
+            0.013037494383752346,
+            -0.019590331241488457,
+            0.018226735293865204,
+            -0.008893677033483982,
+            -0.013348091393709183,
+            0.013893529772758484,
+            0.020014561712741852,
+            0.004090788308531046,
+            -0.007738408166915178,
+            -0.021862991154193878,
+            -0.01127239502966404,
+            0.016060132533311844,
+            0.020272130146622658,
+            0.000947888707742095,
+            -0.0006472347304224968,
+            0.010173942893743515,
+            -0.013113250024616718,
+            -0.023605365306138992,
+            -0.005230906419456005,
+            0.010386057198047638,
+            0.01746918261051178,
+            0.018832778558135033,
+            -0.006685409229248762,
+            0.02443867363035679,
+            -0.0059127043932676315,
+            -0.015499543398618698,
+            2.86597842205083e-05,
+            -0.02802947722375393,
+            -0.017423728480935097,
+            0.00013778003631159663,
+            0.00246773031540215,
+            0.008598231710493565,
+            0.0008555619278922677,
+            0.01943882182240486,
+            0.000904329412151128,
+            0.017590390518307686,
+            -0.000659071491099894,
+            -0.002009410411119461,
+            -0.0013995799235999584,
+            -0.0015340456739068031,
+            0.02254479005932808,
+            -0.012166308239102364,
+            -0.01176480483263731,
+            0.022302372381091118,
+            0.006852070800960064,
+            0.014719262719154358,
+            -0.027635548263788223,
+            0.013757170177996159,
+            -0.015499543398618698,
+            0.017423728480935097,
+            -0.03408990427851677,
+            0.006420265417546034,
+            0.01489350013434887,
+            -0.010348179377615452,
+            -0.020499395206570625,
+            -0.029650641605257988,
+            -0.024150803685188293,
+            -0.0038710979279130697,
+            -0.010855740867555141,
+            -0.016847988590598106,
+            0.006121031939983368,
+            -0.005802859086543322,
+            0.018499454483389854,
+            -0.007079336792230606,
+            0.033877789974212646,
+            0.006969491485506296,
+            -0.004056698642671108,
+            0.003835114184767008,
+            0.022469034418463707,
+            -0.006102093029767275,
+            0.02109028771519661,
+            0.004408961161971092,
+            -0.001467759720981121,
+            0.013681414537131786,
+            -0.006552837323397398,
+            -0.002361672930419445,
+            0.005821797996759415,
+            -0.010484539903700352,
+            0.022014502435922623,
+            0.005924067925661802,
+            -0.007708106189966202,
+            -0.0006775368237867951,
+            -0.011454207822680473,
+            0.014726839028298855,
+            0.006530110724270344,
+            0.0022480397019535303,
+            -0.011295121163129807,
+            0.011249667964875698,
+            -0.004859705455601215,
+            0.0040642740204930305,
+            0.009484568610787392,
+            -0.01774190180003643,
+            -0.028862785547971725,
+            0.011416330002248287,
+            0.019120648503303528,
+            0.006382387597113848,
+            -0.0055755930952727795,
+            -0.0025681061670184135,
+            0.00839369185268879,
+            0.010651201009750366,
+            0.022150862962007523,
+            -0.012984465807676315,
+            -8.889415767043829e-05,
+            0.013204156421124935,
+            -0.007170243188738823,
+            -0.01056029461324215,
+            0.00487485621124506,
+            0.023438703268766403,
+            0.0068823727779090405,
+            -0.04327145218849182,
+            -0.018469152972102165,
+            0.01909034699201584,
+            0.007136153522878885,
+            -0.04154422879219055,
+            0.0026381798088550568,
+            -0.003689285134896636,
+            -0.023787178099155426,
+            0.016105586662888527,
+            -0.0011817833874374628,
+            -0.01828734017908573,
+            0.028756728395819664,
+            0.02830219641327858,
+            0.01834794320166111,
+            2.6987840101355687e-05,
+            -0.013332940638065338,
+            -0.0318172425031662,
+            -0.023590214550495148,
+            0.008280059322714806,
+            -0.02660527639091015,
+            -0.0072725131176412106,
+            0.011242092587053776,
+            -0.017454031854867935,
+            0.02972639724612236,
+            -0.008507325313985348,
+            0.026938600465655327,
+            0.011446632444858551,
+            0.001691238023340702,
+            -0.013446573168039322,
+            -0.018802475184202194,
+            -0.01350717805325985,
+            -0.008196728304028511,
+            0.014090493321418762,
+            0.017923714593052864,
+            0.011007251217961311,
+            0.011332998983561993,
+            0.02251448854804039,
+            -0.012158732861280441,
+            0.007969462312757969,
+            -2.888172457460314e-05,
+            -0.008961857296526432,
+            0.0026343921199440956,
+            -0.018393397331237793,
+            0.00868156272917986,
+            0.0062308767810463905,
+            0.005458172410726547,
+            -0.007208121009171009,
+            0.010742107406258583,
+            0.02712041325867176,
+            -0.018847929313778877,
+            0.011401179246604443,
+            -0.031362712383270264,
+            -0.014787442982196808,
+            -0.01056029461324215,
+            0.026862844824790955,
+            0.007757347077131271,
+            -0.015726808458566666,
+            -0.007514929864555597,
+            -0.01545409020036459,
+            0.012885984033346176,
+            0.0032442226074635983,
+            0.005764981731772423,
+            0.015817714855074883,
+            0.021893294528126717,
+            0.01746918261051178,
+            -0.00795431062579155,
+            0.006852070800960064,
+            0.004674104508012533,
+            0.028090080246329308,
+            -0.01777220331132412,
+            0.020802417770028114,
+            0.009954252280294895,
+            -0.003935490269213915,
+            0.005685438401997089,
+            -0.014257155358791351,
+            0.008530051447451115,
+            -0.007325541693717241,
+            0.030514251440763474,
+            0.0033067206386476755,
+            0.014075342565774918,
+            -0.02062060497701168,
+            0.006575563922524452,
+            -0.03172633796930313,
+            0.009022461250424385,
+            0.013704141601920128,
+            0.012976890429854393,
+            -0.007677804213017225,
+            -0.024181105196475983,
+            -0.0119314668700099,
+            0.0057990713976323605,
+            -0.01279507763683796,
+            -0.01046181283891201,
+            -0.0034828518982976675,
+            0.012446602806448936,
+            0.018847929313778877,
+            -0.01848430372774601,
+            -0.019923655316233635,
+            -0.009583051316440105,
+            0.02898399345576763,
+            -0.015787413343787193,
+            -0.001471547526307404,
+            0.02413565292954445,
+            0.021241798996925354,
+            0.015022284351289272,
+            0.002155239460989833,
+            0.020090317353606224,
+            -0.0007111532613635063,
+            -0.0237265732139349,
+            -0.02813553437590599,
+            0.007302815094590187,
+            -0.006912675220519304,
+            0.01943882182240486,
+            -0.0035548193845897913,
+            -0.005658924113959074,
+            -0.0034033088013529778,
+            -0.007503566797822714,
+            -0.0165904201567173,
+            -0.01787826046347618,
+            0.0021647089160978794,
+            -0.008454295806586742,
+            0.005863463506102562,
+            -0.0002795845502987504,
+            -0.013105674646794796,
+            0.013401119969785213,
+            -0.014075342565774918,
+            0.005772557109594345,
+            -0.002653330797329545,
+            0.001962063368409872,
+            0.0035926972050219774,
+            0.009113367646932602,
+            -0.007336904760450125,
+            -0.06381630152463913,
+            -0.013105674646794796,
+            -0.003704436356201768,
+            0.006071790587157011,
+            -0.00019578021601773798,
+            -0.016635872423648834,
+            -0.013196581043303013,
+            0.01706010289490223,
+            0.008863375522196293,
+            -0.013870803639292717,
+            0.016954045742750168,
+            0.009961827658116817,
+            0.003622999181970954,
+            -0.024044746533036232,
+            -0.0008148433989845216,
+            0.008757317438721657,
+            0.032756611704826355,
+            -0.008643684908747673,
+            -0.0003839849086944014,
+            0.0036684523802250624,
+            -0.0033332351595163345,
+            -0.010052734054625034,
+            -0.014582903124392033,
+            -0.011848135851323605,
+            0.011908739805221558,
+            0.01790856383740902,
+            -0.014264730736613274,
+            -0.02007516659796238,
+            -0.0007106798002496362,
+            0.012931437231600285,
+            -0.026590125635266304,
+            0.0007244104635901749,
+            -0.007787649054080248,
+            -0.01052999310195446,
+            0.018302490934729576,
+            -0.019014591351151466,
+            0.005249845329672098,
+            0.00048294031876139343,
+            0.008507325313985348,
+            -0.023544760420918465,
+            -0.024893205612897873,
+            0.0035131541080772877,
+            -0.023847781121730804,
+            0.03612014651298523,
+            -0.021590271964669228,
+            -0.012082977220416069,
+            0.034483831375837326,
+            -0.0037423139438033104,
+            0.01702980138361454,
+            0.0023995505180209875,
+            0.02739313244819641,
+            -0.011439057067036629,
+            -0.00617784820497036,
+            0.012499631382524967,
+            0.009575475007295609,
+            0.0032593735959380865,
+            0.03775646165013313,
+            -0.01257538702338934,
+            -0.013901105150580406,
+            -0.005556654185056686,
+            -0.012954163365066051,
+            0.0006775368237867951,
+            0.014431392773985863,
+            0.028393102809786797,
+            0.004257450345903635,
+            -0.00628769351169467,
+            -0.006143758539110422,
+            -0.0014649189542979002,
+            -0.004242299124598503,
+            -0.00947699323296547,
+            -0.00715130427852273,
+            0.017287369817495346,
+            -0.025241680443286896,
+            0.016711628064513206,
+            -0.006420265417546034,
+            0.021135739982128143,
+            -0.011757229454815388,
+            0.007007369305938482,
+            -0.008961857296526432,
+            0.003492321353405714,
+            0.0023635667748749256,
+            -0.005109698045998812,
+            0.009287605062127113,
+            -0.014772292226552963,
+            0.008454295806586742,
+            0.013742019422352314,
+            0.0011183382011950016,
+            -0.010590597055852413,
+            -0.019272159785032272,
+            0.0022972808219492435,
+            -0.000710206339135766,
+            0.01181783340871334,
+            -0.0047498601488769054,
+            -0.000981031684204936,
+            0.0002921315317507833,
+            -0.014476845972239971,
+            -0.009575475007295609,
+            0.0006050958181731403,
+            0.005302873905748129,
+            -0.008961857296526432,
+            0.010378481820225716,
+            -0.005693014245480299,
+            -0.00034160923678427935,
+            0.02627195231616497,
+            0.006526322569698095,
+            -0.007632351014763117,
+            -0.014098069630563259,
+            0.01563590206205845,
+            0.015370759181678295,
+            -0.007454325910657644,
+            0.01933276280760765,
+            -0.016196493059396744,
+            -0.0008366230758838356,
+            0.013916256837546825,
+            -0.009219424799084663,
+            0.0017698341980576515,
+            -0.008196728304028511,
+            0.03554440662264824,
+            -0.0008787619881331921,
+            -0.0035415622405707836,
+            -0.0006401326390914619,
+            0.012976890429854393,
+            -0.0026779514737427235,
+            0.001182730309665203,
+            0.01132542360574007,
+            0.005321812815964222,
+            0.0185146052390337,
+            0.00328020635060966,
+            -0.03378688171505928,
+            -0.007780073676258326,
+            -0.0065490491688251495,
+            -0.008090670220553875,
+            0.009022461250424385,
+            0.025196226313710213,
+            0.01990850456058979,
+            0.006408901885151863,
+            0.001881573349237442,
+            0.018423698842525482,
+            0.015757111832499504,
+            -0.01489350013434887,
+            0.016363153234124184,
+            0.02802947722375393,
+            -0.010030006989836693,
+            0.022741753607988358,
+            -0.005253633018583059,
+            -0.008893677033483982,
+            0.008946705609560013,
+            -0.0015378334792330861,
+            -0.017393426969647408,
+            -0.015681356191635132,
+            0.005022579338401556,
+            -0.008999735116958618,
+            -0.0022897052112966776,
+            -0.018605511635541916,
+            0.004670316819101572,
+            0.017105557024478912,
+            -0.004965762607753277,
+            -3.5243989259470254e-05,
+            0.019484274089336395,
+            -0.006965703796595335,
+            -0.016908591613173485,
+            0.006325571332126856,
+            -0.015007133595645428,
+            0.007946735247969627,
+            -0.013666263781487942,
+            0.010492115281522274,
+            -0.006098304875195026,
+            0.018817627802491188,
+            0.005446809343993664,
+            -0.008090670220553875,
+            -0.0008958069374784827,
+            -0.018984289839863777,
+            0.02065090648829937,
+            -0.033089932054281235,
+            0.00839369185268879,
+            -0.026226500049233437,
+            -0.0009781909175217152,
+            0.0008849170990288258,
+            0.014605630189180374,
+            -0.001218714052811265,
+            0.019696388393640518,
+            -0.002751812804490328,
+            -0.020090317353606224,
+            -0.007086912170052528,
+            -0.0010634156642481685,
+            -0.01618134044110775,
+            -0.030923331156373024,
+            -0.004272601567208767,
+            0.0036267871037125587,
+            -0.004711982328444719,
+            0.0021135739516466856,
+            0.017984317615628242,
+            -0.005613470915704966,
+            0.009173971600830555,
+            -0.02690829709172249,
+            -0.007416448090225458,
+            -0.00355103169567883,
+            0.0010776197304949164,
+            -0.011242092587053776,
+            0.0010596278589218855,
+            0.011226941831409931,
+            -0.019044892862439156,
+            0.013461724855005741,
+            -0.021378157660365105,
+            -0.004371083341538906,
+            -0.008507325313985348,
+            -0.00135412672534585,
+            -0.013446573168039322,
+            -0.028120383620262146,
+            0.025029566138982773,
+            -0.00021495578403118998,
+            -0.003528305096551776,
+            -0.011999646201729774,
+            0.023181134834885597,
+            0.002914686920121312,
+            0.027817361056804657,
+            -0.03157482668757439,
+            0.0013427634257823229,
+            0.011552689597010612,
+            0.0025984083767980337,
+            -0.017287369817495346,
+            0.004746072459965944,
+            -0.005234694108366966,
+            0.02210540883243084,
+            -0.0011694730492308736,
+            -0.0005340751959010959,
+            -0.008234606124460697,
+            0.030999086797237396,
+            0.009136094711720943,
+            0.0012944694608449936,
+            -0.013226882554590702,
+            -0.008060368709266186,
+            -0.002759388415142894,
+            -0.0011069749016314745,
+            0.012855681590735912,
+            0.01290113478899002,
+            -0.012105703353881836,
+            0.0013607552973553538,
+            -0.006878585088998079,
+            0.0011694730492308736,
+            -0.002935519441962242,
+            -0.0004966709529981017,
+            -0.01149208564311266,
+            0.003257479751482606,
+            0.016272246837615967,
+            -0.00780280027538538,
+            -0.006480869371443987,
+            0.0007750718505121768,
+            0.009181547909975052,
+            0.012461753562092781,
+            -0.012742049060761929,
+            -0.018226735293865204,
+            -0.011908739805221558,
+            0.016029831022024155,
+            0.010522416792809963,
+            -0.008204303681850433,
+            -0.013234458863735199,
+            -0.02583257108926773,
+            0.01169662456959486,
+            -0.03293842449784279,
+            -0.012242063879966736,
+            0.005821797996759415,
+            0.028559764847159386,
+            0.019999410957098007,
+            0.012226912193000317,
+            0.001332347048446536,
+            0.002950670663267374,
+            0.00023342114582192153,
+            -0.01916610263288021,
+            0.014408666640520096,
+            -0.011143610812723637,
+            -0.018166130408644676,
+            0.0075906855054199696,
+            0.011939042247831821,
+            0.03148392215371132,
+            0.006942977197468281,
+            -0.0010160685051232576,
+            0.01686313934624195,
+            -0.001218714052811265,
+            -0.013582932762801647,
+            0.01420412678271532,
+            -0.024256860837340355,
+            0.016438908874988556,
+            0.02860521711409092,
+            -0.004780162125825882,
+            0.0008531945641152561,
+            -0.020938776433467865,
+            0.00482561532407999,
+            -0.011529963463544846,
+            -0.002177966060116887,
+            -0.003287781961262226,
+            -0.03460504114627838,
+            0.007946735247969627,
+            -0.006825556512922049,
+            0.01046181283891201,
+            -0.00814369972795248,
+            0.006655106786638498,
+            -0.012431452050805092,
+            0.004253662656992674,
+            0.01513591781258583,
+            0.016105586662888527,
+            0.007060397882014513,
+            -0.025211378931999207,
+            -0.009196698665618896,
+            0.008105821907520294,
+            -0.01349202636629343,
+            -0.02295386977493763,
+            -0.006852070800960064,
+            0.009840618818998337,
+            -0.01061332318931818,
+            -0.00488243205472827,
+            -0.028180986642837524,
+            -0.013681414537131786,
+            -0.0014327229000627995,
+            -0.017817657440900803,
+            -0.005556654185056686,
+            -0.003100287402048707,
+            0.01571165770292282,
+            -0.0049733384512364864,
+            -0.006185423582792282,
+            0.004742284305393696,
+            -0.023878084495663643,
+            0.0037423139438033104,
+            -0.008280059322714806,
+            0.0008266801596619189,
+            -0.0032517979852855206,
+            -0.0014762822538614273,
+            -0.007969462312757969,
+            -0.014332910999655724,
+            0.006893736310303211,
+            -0.016469212248921394,
+            0.002954458352178335,
+            -0.015173795633018017,
+            -0.016499513760209084,
+            -0.00682934420183301,
+            0.00775355938822031,
+            0.021575121209025383,
+            0.006541473791003227,
+            0.021484214812517166,
+            0.005469535943120718,
+            0.028680972754955292,
+            0.0003953482082579285,
+            0.003041577059775591,
+            -8.958956641436089e-06,
+            0.003232859307900071,
+            0.019347915425896645,
+            0.006571775767952204,
+            0.018984289839863777,
+            -0.008742166683077812,
+            0.010514841414988041,
+            -0.010355755686759949,
+            0.010014856234192848,
+            0.006333146709948778,
+            -0.006473293993622065,
+            -0.003799130441620946,
+            -0.010348179377615452,
+            -0.0031419529113918543,
+            -0.016544966027140617,
+            0.0027366618160158396,
+            0.00693161366507411,
+            0.0071588801220059395,
+            -0.011582992039620876,
+            -0.005230906419456005,
+            -0.014704111963510513,
+            -0.0064354161731898785,
+            -0.01831764169037342,
+            0.014385939575731754,
+            0.029226411134004593,
+            -0.016332851722836494,
+            -0.01977214403450489,
+            -0.007772498298436403,
+            0.0007296186522580683,
+            -0.006636167876422405,
+            -0.01240872498601675,
+            -0.009151245467364788,
+            -0.035059571266174316,
+            -0.017666146159172058,
+            0.007586897816509008,
+            0.02484775334596634,
+            0.01387837901711464,
+            -0.013628385961055756,
+            -0.0006983695784583688,
+            -0.016984347254037857,
+            -0.010787560604512691,
+            -0.01535560842603445,
+            -0.02054484933614731,
+            -0.02498411200940609,
+            0.0077952248975634575,
+            -0.004033972043544054,
+            0.014931377954781055,
+            0.008113397285342216,
+            0.0032631612848490477,
+            -0.004371083341538906,
+            -0.0024109138175845146,
+            -0.013196581043303013,
+            0.0012925754999741912,
+            -0.004787737503647804,
+            -0.0017291157273575664,
+            0.03554440662264824,
+            -0.008098246529698372,
+            -0.005871038883924484,
+            0.011060279794037342,
+            0.011128460057079792,
+            -0.013764745555818081,
+            0.0006642796797677875,
+            -0.011249667964875698,
+            0.021847840398550034,
+            -0.021620575338602066,
+            0.019257009029388428,
+            0.0014507147716358304,
+            -0.010840589180588722,
+            0.028999146074056625,
+            0.01892368495464325,
+            0.00743159931153059,
+            0.006579351611435413,
+            0.01672677882015705,
+            0.0044316877610981464,
+            0.016196493059396744,
+            -0.0026552248746156693,
+            -0.004632438998669386,
+            0.009295180439949036,
+            0.015681356191635132,
+            -0.000702630786690861,
+            -0.006942977197468281,
+            0.016438908874988556,
+            0.004056698642671108,
+            0.0075414446182549,
+            -0.01356020662933588,
+            -0.008961857296526432,
+            0.001507531269453466,
+            0.006257391534745693,
+            -0.005939218681305647,
+            -0.0026003022212535143,
+            0.005530139897018671,
+            0.003943065647035837,
+            -0.013302638195455074,
+            0.01689344085752964,
+            -0.016166189685463905,
+            0.008742166683077812,
+            -0.0037972365971654654,
+            0.0121360057964921,
+            -0.006799042224884033,
+            0.0013153020991012454,
+            -0.01818128302693367,
+            -0.003869204083457589,
+            0.0038597346283495426,
+            0.012173883616924286,
+            -0.01262841559946537,
+            0.025484098121523857,
+            0.008166425861418247,
+            0.006602078210562468,
+            0.0059467945247888565,
+            -0.012855681590735912,
+            0.027211319655179977,
+            0.005969521123915911,
+            0.009886072017252445,
+            -0.008893677033483982,
+            0.011136035434901714,
+            0.035362593829631805,
+            0.03060515783727169,
+            0.014135947450995445,
+            0.004469565115869045,
+            -0.018999440595507622,
+            0.015120766125619411,
+            0.0018427487229928374,
+            0.0023806116078048944,
+            0.00863610953092575,
+            -0.011348150670528412,
+            0.01305264513939619,
+            -0.009484568610787392,
+            0.015241974964737892,
+            -0.021272100508213043,
+            -0.011749654076993465,
+            -0.014408666640520096,
+            -0.021075136959552765,
+            -0.01430260855704546,
+            0.034180812537670135,
+            -0.002346521709114313,
+            -0.010393633507192135,
+            0.0045112306252121925,
+            0.008477022871375084,
+            -0.006079366430640221,
+            0.0053142374381423,
+            -0.008204303681850433,
+            -0.013749594800174236,
+            -0.01889338344335556,
+            0.006628592498600483,
+            -0.01550711877644062,
+            0.007651289459317923,
+            -0.003304826794192195,
+            0.0037688282318413258,
+            0.007014944683760405,
+            0.018105527386069298,
+            0.00530666159465909,
+            -0.00135412672534585,
+            0.0014033677289262414,
+            0.0016865032957866788,
+            -0.00931033119559288,
+            0.0108330138027668,
+            0.015408637002110481,
+            -0.0011921997647732496,
+            -0.011279970407485962,
+            -0.009136094711720943,
+            -0.042877525091171265,
+            0.0006264019757509232,
+            -0.013128400780260563,
+            0.0064354161731898785,
+            -0.011196639388799667,
+            0.0022234192583709955,
+            0.015863168984651566,
+            -0.0054051438346505165,
+            -0.0010918239131569862,
+            0.02193874679505825,
+            0.013014768250286579,
+            0.021075136959552765,
+            0.02065090648829937,
+            0.014188976027071476,
+            -0.015302578918635845,
+            -0.03469594568014145,
+            0.002221525413915515,
+            -0.010030006989836693,
+            0.013772320933640003,
+            0.004238511435687542,
+            0.016060132533311844,
+            -0.023893235251307487,
+            -0.011878437362611294,
+            -0.006662682630121708,
+            0.014431392773985863,
+            0.01491622719913721,
+            -0.0032271775417029858,
+            0.02690829709172249,
+            0.009280029684305191,
+            0.0020510759204626083,
+            -0.02281750924885273,
+            0.003312402404844761,
+            0.005321812815964222,
+            -0.008370965719223022,
+            -0.016681326553225517,
+            -0.0004552422615233809,
+            -0.0012168202083557844,
+            -0.0007817004225216806,
+            -0.021650876849889755,
+            0.018832778558135033,
+            -0.004018820822238922,
+            0.0037574649322777987,
+            0.013840501196682453,
+            0.01516621932387352,
+            0.017984317615628242,
+            0.008370965719223022,
+            0.001497115008533001,
+            -0.006605865899473429,
+            -0.0007973249885253608,
+            -0.0005828426801599562,
+            -0.007389933802187443,
+            -0.005598319694399834,
+            0.006382387597113848,
+            -0.01134057529270649,
+            0.008810346014797688,
+            0.000896753859706223,
+            0.006113456096500158,
+            0.010605747811496258,
+            -0.004836978856474161,
+            -0.011628445237874985,
+            0.014628356322646141,
+            -0.0010548931313678622,
+            -0.0055755930952727795,
+            -0.011060279794037342,
+            0.003624893259257078,
+            0.013575357384979725,
+            -0.003096499713137746,
+            -0.013499601744115353,
+            0.00910579226911068,
+            0.002009410411119461,
+            -0.027438584715127945,
+            0.015499543398618698,
+            0.010295150801539421,
+            -0.013370818458497524,
+            0.004980913829058409,
+            -0.003371112747117877,
+            -0.03027183562517166,
+            0.004791525658220053,
+            0.002660906407982111,
+            -0.010643625631928444,
+            0.028938541188836098,
+            -0.003821857040748,
+            0.029256712645292282,
+            -0.004980913829058409,
+            -0.006905099377036095,
+            -0.011355726048350334,
+            0.009416389279067516,
+            -0.018681267276406288,
+            -0.0037953427527099848,
+            -0.008727015927433968,
+            -0.00915882084518671,
+            -0.0009838725673034787,
+            -0.005185453221201897,
+            0.006984642706811428,
+            0.005121061112731695,
+            -0.012704171240329742,
+            0.006757376715540886,
+            -0.010257272981107235,
+            -0.00020856392802670598,
+            0.0006718551740050316,
+            -0.008408842608332634,
+            0.013885954394936562,
+            0.005996035411953926,
+            -0.008340663276612759,
+            -0.0005032995832152665,
+            0.027817361056804657,
+            0.05051366239786148,
+            -0.018863080069422722,
+            0.0009270560112781823,
+            0.01452229917049408,
+            -0.0028824908658862114,
+            -0.0021590273827314377,
+            -0.011976920068264008,
+            0.005821797996759415,
+            -0.019423669204115868,
+            0.013317788951098919,
+            0.006473293993622065,
+            -0.008833073079586029,
+            0.003202557098120451,
+            -0.016711628064513206,
+            0.002621134975925088,
+            -0.0018058179412037134,
+            -0.018681267276406288,
+            -0.015529844909906387,
+            -0.012060250155627728,
+            -0.024181105196475983,
+            0.0004235197266098112,
+            -0.0008546149474568665,
+            -0.013870803639292717,
+            0.006579351611435413,
+            0.008431569673120975,
+            0.032483890652656555,
+            0.0011912527261301875,
+            0.010128489695489407,
+            -2.1513333194889128e-05,
+            -0.0010416359873488545,
+            -0.005988460034132004,
+            -0.01523439958691597,
+            0.00015387803432531655,
+            -0.002342734020203352,
+            0.022590242326259613,
+            0.005549078807234764,
+            0.017560089007019997,
+            0.0017887729918584228,
+            -0.016514664515852928,
+            -0.00596573343500495,
+            -0.007408872712403536,
+            -0.018060073256492615,
+            0.0062649669125676155,
+            -0.005473323632031679,
+            0.009605777449905872,
+            -0.015863168984651566,
+            -0.02731737680733204,
+            0.00792400911450386,
+            0.012234487570822239,
+            0.028514310717582703,
+            -0.0035623949952423573,
+            0.001716805505566299,
+            -0.005533927585929632,
+            0.016408607363700867,
+            0.032180868089199066,
+            -0.0008550884085707366,
+            0.006950552575290203,
+            -0.019999410957098007,
+            0.0017404790269210935,
+            0.013704141601920128,
+            -0.008401267230510712,
+            -0.01821158453822136,
+            0.003939277958124876,
+            -0.020635755732655525,
+            0.004583198111504316,
+            -0.020393338054418564,
+            0.030408194288611412,
+            -0.0023522034753113985,
+            0.008105821907520294,
+            0.004310478921979666,
+            0.01760554127395153,
+            -0.0031646795105189085,
+            0.010545143857598305,
+            0.013916256837546825,
+            0.014908651821315289,
+            0.0009980766335502267,
+            -0.011189064010977745,
+            -0.03790797293186188,
+            0.001859793672338128,
+            0.00817400123924017,
+            0.018635815009474754,
+            -0.01980244740843773,
+            0.011681473813951015,
+            -0.01296931505203247,
+            -0.020363036543130875,
+            -0.01003758329898119,
+            0.011575416661798954,
+            -0.006139970384538174,
+            0.03378688171505928,
+            -0.005874827038496733,
+            -0.008204303681850433,
+            0.005136212334036827,
+            0.010454237461090088,
+            0.0066778333857655525,
+            0.014704111963510513,
+            0.0326354019343853,
+            0.010196669027209282,
+            0.014173824340105057,
+            -0.004859705455601215,
+            0.02399929240345955,
+            -0.003931702114641666,
+            0.003954428713768721,
+            0.00509454682469368,
+            -0.022832660004496574,
+            -0.0040983641520142555,
+            -0.003931702114641666,
+            -0.016711628064513206,
+            -0.007458113599568605,
+            0.0032366469968110323,
+            0.008802770636975765,
+            -0.014045040123164654,
+            -0.00649980828166008,
+            -0.01350717805325985,
+            0.01218903437256813,
+            0.002730980049818754,
+            0.018302490934729576,
+            -0.029802151024341583,
+            0.009825468063354492,
+            -0.017484333366155624,
+            -0.017044952139258385,
+            0.006511171814054251,
+            -0.0038332203403115273,
+            0.006855858489871025,
+            -0.018272189423441887,
+            -0.01318900566548109,
+            0.012817803770303726,
+            0.0007897494360804558,
+            0.013060221448540688,
+            -0.009825468063354492,
+            0.0009706153068691492,
+            0.006064215209335089,
+            -0.009833043441176414,
+            -0.010961798019707203,
+            -0.007977037690579891,
+            -0.009681533090770245,
+            -0.008848223835229874,
+            0.005621046293526888,
+            0.01656011864542961,
+            -0.013742019422352314,
+            -0.003685497445985675,
+            0.016135888174176216,
+            0.006416477728635073,
+            -0.008166425861418247,
+            0.004075637552887201,
+            -0.003793448908254504,
+            -9.948806837201118e-05,
+            0.0013740125577896833,
+            0.012439027428627014,
+            -0.0017632056260481477,
+            0.024665940552949905,
+            -0.02654467150568962,
+            0.007598260883241892,
+            -0.0054051438346505165,
+            0.005446809343993664,
+            0.018226735293865204,
+            -0.02366596832871437,
+            -0.013082947582006454,
+            0.006276329979300499,
+            -0.01966608688235283,
+            -0.008060368709266186,
+            0.0054051438346505165,
+            -0.00016405766655225307,
+            -0.019923655316233635,
+            -0.012923861853778362,
+            -0.034786853939294815,
+            -0.02309022843837738,
+            0.024665940552949905,
+            0.008401267230510712,
+            -0.0073672072030603886,
+            0.018060073256492615,
+            -0.018438849598169327,
+            0.0172722190618515,
+            -0.002225313102826476,
+            -0.006613441277295351,
+            0.003935490269213915,
+            0.007674016058444977,
+            0.025362888351082802,
+            0.002653330797329545,
+            0.008264907635748386,
+            -0.0042082094587385654,
+            -0.014931377954781055,
+            -0.006321783177554607,
+            0.01980244740843773,
+            0.02471139281988144,
+            0.018605511635541916,
+            -0.018575210124254227,
+            0.010742107406258583,
+            -0.011749654076993465,
+            0.01818128302693367,
+            -0.006537686102092266,
+            0.00039037675014697015,
+            0.01859036087989807,
+            0.007969462312757969,
+            0.025241680443286896,
+            0.019923655316233635,
+            0.006852070800960064,
+            0.014128371141850948,
+            0.011908739805221558,
+            0.001204509986564517,
+            -0.012832955457270145,
+            -0.01154511421918869,
+            -0.02840825356543064,
+            0.042847223579883575,
+            0.0025434857234358788,
+            0.008317936211824417,
+            -0.027635548263788223,
+            -0.00021389048197306693,
+            -0.0034942151978611946,
+            -0.018832778558135033,
+            0.005939218681305647,
+            0.03260510042309761,
+            0.03284751623868942,
+            0.021438762545585632,
+            -2.888172457460314e-05,
+            0.0012234487803652883,
+            0.0329081192612648,
+            -0.013673839159309864,
+            -0.00937851145863533,
+            -0.01034060399979353,
+            -0.018393397331237793,
+            -0.006852070800960064,
+            -0.0032423287630081177,
+            -0.01031030248850584,
+            -0.0061172437854111195,
+            -0.009272454306483269,
+            0.0238629337400198,
+            0.0001374249259242788,
+            -0.0022594030015170574,
+            0.006605865899473429,
+            -0.020878173410892487,
+            0.011537538841366768,
+            -0.02813553437590599,
+            -0.013431422412395477,
+            -0.008651260286569595,
+            -0.006950552575290203,
+            -0.018802475184202194,
+            0.008643684908747673,
+            -0.006942977197468281,
+            0.021438762545585632,
+            0.002914686920121312,
+            -0.023590214550495148,
+            0.007530081085860729,
+            0.015317730605602264,
+            -0.0003825644962489605,
+            0.014037464745342731,
+            -0.003890036838129163,
+            0.009060339070856571,
+            0.0045112306252121925,
+            0.011681473813951015,
+            0.016954045742750168,
+            0.010651201009750366,
+            0.009030036628246307,
+            0.003437398700043559,
+            -0.001167579204775393,
+            0.01878732442855835,
+            -0.0007163614500313997,
+            -0.03405960276722908,
+            2.9236827685963362e-05,
+            -0.014628356322646141,
+            0.029499130323529243,
+            -0.009795165620744228,
+            0.009514871053397655,
+            0.008105821907520294,
+            0.007098275702446699,
+            -0.0067119235172867775,
+            0.007916433736681938,
+            -0.008742166683077812,
+            0.0052763596177101135,
+            0.015832867473363876,
+            -0.0042877523228526115,
+            -0.0014980619307607412,
+            0.007840678095817566,
+            0.00407184986397624,
+            -0.005454384721815586,
+            -0.019711541011929512,
+            -0.012287517078220844,
+            0.00726114958524704,
+            -0.02065090648829937,
+            -0.004367295652627945,
+            0.01161329448223114,
+            -0.005454384721815586,
+            -0.009620928205549717,
+            0.01393140759319067,
+            0.02136300690472126,
+            -0.002382505452260375,
+            0.022893264889717102,
+            -0.01349202636629343,
+            -0.009833043441176414,
+            -0.007984613068401814,
+            -0.004374871030449867,
+            -0.041120000183582306,
+            0.025984082370996475,
+            0.01068150345236063,
+            -0.006871009711176157,
+            0.01895398646593094,
+            0.0022499337792396545,
+            -0.002954458352178335,
+            -0.0011041341349482536,
+            0.00010499217023607343,
+            -0.03281721472740173,
+            0.03660498186945915,
+            -0.007007369305938482,
+            0.0005449649761430919,
+            0.010272424668073654,
+            0.042029064148664474,
+            -0.004590773954987526,
+            -0.005859675817191601,
+            0.01225721463561058,
+            0.005336964037269354,
+            0.031938452273607254,
+            -0.010416359640657902,
+            -0.014340486377477646,
+            -0.01815097965300083,
+            -0.004833190701901913,
+            -0.01939336769282818,
+            -0.0038313264958560467,
+            0.014037464745342731,
+            -0.019650936126708984,
+            -0.011242092587053776,
+            -0.001287840772420168,
+            -0.012537509202957153,
+            -0.0010747789638116956,
+            0.008598231710493565,
+            0.017832808196544647,
+            -0.011348150670528412,
+            -0.004450626205652952,
+            -2.2904154320713133e-05,
+            -0.005003640428185463,
+            0.0185146052390337,
+            0.0020056227222085,
+            -0.01702980138361454,
+            0.01321930717676878,
+            -0.00526499655097723,
+            0.006749800872057676,
+            0.0074429623782634735,
+            0.007939159870147705,
+            0.05145302787423134,
+            0.010317877866327763,
+            -0.001339922659099102,
+            -0.01607528328895569,
+            0.023196285590529442,
+            -0.01749948412179947,
+            -0.014181400649249554,
+            -0.02119634486734867,
+            -0.0029923359397798777,
+            -0.007060397882014513,
+            -0.028014326468110085,
+            -9.137987944995984e-05,
+            0.0016221113037317991,
+            -0.0024980325251817703,
+            -0.010045158676803112,
+            0.0005208179936744273,
+            -0.011529963463544846,
+            -0.006954340264201164,
+            -0.008545203134417534,
+            -0.016044981777668,
+            0.02240842953324318,
+            -0.021817538887262344,
+            -0.008855799213051796,
+            0.011514811776578426,
+            0.03118089959025383,
+            -0.005011215806007385,
+            -0.014173824340105057,
+            -0.019817598164081573,
+            0.0025643182452768087,
+            -0.01763584464788437,
+            0.012181458994746208,
+            -0.01225721463561058,
+            0.015045011416077614,
+            -0.03142331540584564,
+            -0.006079366430640221,
+            -0.008992158807814121,
+            0.012840530835092068,
+            0.019817598164081573,
+            -0.0025093958247452974,
+            0.00606800289824605,
+            -0.010295150801539421,
+            0.030211230739951134,
+            -0.00012286569108255208,
+            -0.018605511635541916,
+            0.00873459130525589,
+            0.007977037690579891,
+            0.012537509202957153,
+            -0.004435475450009108,
+            0.00257757562212646,
+            0.011567841283977032,
+            0.012742049060761929,
+            -0.01634800247848034,
+            -0.003174148965626955,
+            -0.017969166859984398,
+            0.005996035411953926,
+            0.006999793462455273,
+            0.021711481735110283,
+            -0.0013550736475735903,
+            0.018272189423441887,
+            -0.01563590206205845,
+            0.00019412307301536202,
+            -0.011749654076993465,
+            -0.0082270298153162,
+            -0.0028237802907824516,
+            -0.003958216868340969,
+            0.006685409229248762,
+            -0.028211290016770363,
+            -0.01274962443858385,
+            0.005287723150104284,
+            -0.004992276895791292,
+            -0.03405960276722908,
+            0.0053710537031292915,
+            -0.02725677192211151,
+            0.013128400780260563,
+            -0.002397656673565507,
+            -0.017847958952188492,
+            -0.00030846628942526877,
+            0.026332557201385498,
+            -0.003677921835333109,
+            -0.012537509202957153,
+            0.015544996596872807,
+            -0.006757376715540886,
+            -0.02119634486734867,
+            -0.005420294590294361,
+            0.014658658765256405,
+            -0.0010719380807131529,
+            -0.012393574230372906,
+            -0.029635490849614143,
+            0.012992041185498238,
+            0.0026874206960201263,
+            -0.007386146113276482,
+            0.010893618687987328,
+            0.025953780859708786,
+            0.008151275105774403,
+            -0.005924067925661802,
+            0.0015757111832499504,
+            0.0049581872299313545,
+            0.00300938100554049,
+            0.020287280902266502,
+            0.02728707529604435,
+            -0.008014915511012077,
+            -0.0011145505122840405,
+            -0.014954105019569397,
+            0.015802564099431038,
+            0.0013020449550822377,
+            0.010507266037166119,
+            0.021893294528126717,
+            0.007045246660709381,
+            -0.005738467443734407,
+            0.003905187826603651,
+            -0.023135682567954063,
+            0.01420412678271532,
+            -0.012272365391254425,
+            0.000622140767518431,
+            0.008749742060899734,
+            -0.007283876184374094,
+            0.01437078882008791,
+            -0.0015245762187987566,
+            0.0339989997446537,
+            0.025211378931999207,
+            0.0043218424543738365,
+            0.0007362472242675722,
+            -0.028559764847159386,
+            0.003547244006767869,
+            0.008651260286569595,
+            0.014499573037028313,
+            -0.0076664406806230545,
+            0.006261179223656654,
+            -0.004355932120233774,
+            0.00959820207208395,
+            -0.0065490491688251495,
+            -0.015893470495939255,
+            0.006113456096500158,
+            0.0012689019786193967,
+            0.006795254070311785,
+            0.0023408401757478714,
+            -0.008863375522196293,
+            -0.024362917989492416,
+            -0.025984082370996475,
+            0.021984200924634933,
+            -0.03918066248297691,
+            0.0073293293826282024,
+            0.011234517209231853,
+            0.03324144333600998,
+            -0.008158850483596325,
+            -0.004980913829058409,
+            0.006276329979300499,
+            -0.016711628064513206,
+            -0.004147605039179325,
+            0.008204303681850433,
+            0.012385998852550983,
+            -0.002420383272692561,
+            0.00974971242249012,
+            0.01318900566548109,
+            -0.012378423474729061,
+            0.03660498186945915,
+            0.016514664515852928,
+            0.01467380952090025,
+            -0.004007457755506039,
+            0.01312082540243864,
+            0.0066172294318675995,
+            0.019044892862439156,
+            0.025165924802422523,
+            -0.017454031854867935,
+            -0.002365460619330406,
+            -0.02292356640100479,
+            0.024165954440832138,
+            0.00726114958524704,
+            0.004556683823466301,
+            -0.008310360834002495,
+            0.014870774000883102,
+            0.0017954016802832484,
+            0.0028199926018714905,
+            -0.006696772295981646,
+            -0.010173942893743515,
+            0.019241856411099434,
+            0.0034317169338464737,
+            0.014870774000883102,
+            4.8501173296244815e-05,
+            -0.005503625608980656,
+            0.006052851676940918,
+            0.014832896180450916,
+            -0.023590214550495148,
+            -0.00203592493198812,
+            -0.019484274089336395,
+            -0.03202936053276062,
+            0.009022461250424385,
+            0.0016135887708514929,
+            -0.007052822504192591,
+            0.03230207785964012,
+            -0.010083036497235298,
+            -0.019787294790148735,
+            -0.008067944087088108,
+            -0.013113250024616718,
+            -0.010469388216733932,
+            0.0008271536789834499,
+            0.003908975515514612,
+            -0.00012428610352799296,
+            -0.009317907504737377,
+            -0.005749830510467291,
+            0.01990850456058979,
+            -0.0030358952935785055,
+            -0.00753386877477169,
+            0.00868156272917986
+          ],
+          "embedding": [
+            0.0040707582550429606,
+            -0.010230625151989194,
+            -0.021520242101903186,
+            -0.010115756270640508,
+            0.026738734586159035,
+            0.01796370233223267,
+            -0.012245199029994153,
+            0.06506924494147726,
+            -0.0014087587222033442,
+            0.009300068719271429,
+            0.003787396973047584,
+            0.002818772639754174,
+            0.022129077062176306,
+            -0.021526709808445683,
+            -0.0034388135367225824,
+            0.02763761747879405,
+            0.013210013805395276,
+            0.025255872672296602,
+            0.00029011480909986703,
+            -0.05438974762290585,
+            0.016671640187490865,
+            -0.016426252259893328,
+            -0.056591180651012554,
+            0.009335648425523075,
+            0.006141624459418827,
+            0.02568943956694641,
+            0.019377132054042116,
+            0.017946464337078746,
+            -0.057184776985927686,
+            0.03208527139096687,
+            0.030132392063755547,
+            -0.003846032371791839,
+            -0.008376524427531415,
+            -0.0010513963058713435,
+            -0.02734868388724371,
+            -0.014879400370545479,
+            0.04383920959748825,
+            0.02563613659516289,
+            -0.020117467074438997,
+            0.0398016699155716,
+            0.029896497884197775,
+            -0.0004195844530067313,
+            -0.023302720788253625,
+            -0.001168337916577517,
+            0.009149021369661628,
+            -0.02508145721900992,
+            -0.004307594138480351,
+            0.00550940415564303,
+            0.014016998918718476,
+            -0.028607379616884317,
+            -0.0018317656587526954,
+            -0.01973464940415756,
+            0.03762738392056255,
+            0.015358106058583114,
+            -0.03931907682104936,
+            0.011519637298832444,
+            0.004482080558226775,
+            0.0747867063383242,
+            0.05550679018263077,
+            0.023243129601931176,
+            0.02671631686665238,
+            -0.009686146566573536,
+            -0.04088582189725495,
+            -0.0065344623594924085,
+            0.019617295999325712,
+            -0.004722087402575777,
+            -0.002871844114437583,
+            0.003935087265300519,
+            0.03229890413393201,
+            -0.02813577047904649,
+            -0.05124172132597234,
+            0.03325235407841968,
+            -0.008687655557217192,
+            -0.011027174838246948,
+            -0.003653553063547375,
+            -0.02350737945182988,
+            0.016223087939183468,
+            0.03789252356462714,
+            0.0034350623378603083,
+            0.00195158657888739,
+            0.011789712862232108,
+            -0.0004435439660769551,
+            -0.0037330456918125894,
+            0.0031552713243866858,
+            0.04465432992298031,
+            -0.0032465421571915712,
+            -0.0075366743861922745,
+            -0.059958752731410256,
+            0.016784616861008125,
+            -0.028734506065772967,
+            -0.03035575736273943,
+            0.02680390892611741,
+            0.02338368719038086,
+            -0.0030658375437784277,
+            -0.03754896050794851,
+            0.07005659523248504,
+            0.0020711575553570695,
+            0.013034425083076923,
+            -0.02316832262132332,
+            0.021332142030732894,
+            -0.0318756233020585,
+            0.02270782509507388,
+            0.01570810309454529,
+            0.004788124787612736,
+            0.026727250200142456,
+            -0.022380142778058276,
+            -0.008593910379436904,
+            0.014439754382387034,
+            0.015007697617383174,
+            0.07662531963612977,
+            0.011195998137723795,
+            -0.010809283538470098,
+            -0.042227822120219305,
+            0.003824320933275175,
+            -0.03797629427607094,
+            -0.02347390684453348,
+            -0.0526876440280276,
+            -0.017799861680287913,
+            0.01701482796266917,
+            0.002730395144917155,
+            0.03789451759899825,
+            0.038338545350313946,
+            0.028463789278406586,
+            0.015494583705011921,
+            -0.05700237266350479,
+            0.047895170359075166,
+            -0.0064406894534929406,
+            -0.0007954721295065988,
+            0.012543029839729358,
+            -0.004703334868741216,
+            -0.0060569679284068,
+            0.012324901254711192,
+            -0.019883935809784068,
+            -0.04253565942154215,
+            0.02164409276211115,
+            -0.0075367555415808635,
+            0.03617159147722157,
+            0.07567745988686005,
+            -0.04225977020036783,
+            -0.022548837589622003,
+            0.029811602145850144,
+            -0.02175792196819533,
+            0.02069861458115363,
+            -0.00941331829664397,
+            -0.035479639885943264,
+            0.020352552580119105,
+            0.003678588588550461,
+            0.06776105274336729,
+            -0.02346047721661687,
+            0.02721075076349443,
+            -0.029386781707644253,
+            0.023530831921117993,
+            -0.02077660533100372,
+            0.020022475323372906,
+            0.024195118676513715,
+            -0.028378518720494666,
+            -0.012900969023913307,
+            0.019264089970719624,
+            -0.0374983624629265,
+            -0.019726171557384355,
+            0.00979636085642416,
+            -0.014479528435832063,
+            -0.014533384583777662,
+            0.005904623758767224,
+            0.010344475500263744,
+            0.05659646358026115,
+            0.04163866113482093,
+            0.007152494819169405,
+            -0.07584500130974763,
+            0.05471858597878099,
+            0.03580211013427615,
+            0.044161429444475306,
+            0.02258180566046248,
+            0.00980779221334715,
+            0.0011193505603397395,
+            0.018086566768892507,
+            0.015605066702191871,
+            -0.00954727399396313,
+            -0.027950079146919375,
+            0.03028514025485407,
+            -0.006351669761257431,
+            0.005095804801686436,
+            0.009620040085743162,
+            0.019748001227987134,
+            0.033709535059510916,
+            0.008203559956940196,
+            0.007389752602218563,
+            0.04197771878351742,
+            -0.02067363758471765,
+            0.007543739188471913,
+            0.011207941261799052,
+            0.013964827251172813,
+            0.03839562075248879,
+            0.0009775244543292637,
+            0.006469507611492216,
+            -0.022808235395746654,
+            -0.024795004398633292,
+            0.006249024864814684,
+            0.011292877109066925,
+            0.01596839619826819,
+            -0.0014105396434174883,
+            -0.04152075438277094,
+            0.008267788411079337,
+            -0.028834191212891387,
+            0.030940526405027906,
+            -0.0018271056434014425,
+            0.006185475047820603,
+            -0.007920574674727123,
+            0.012907560753043616,
+            -0.015165704767187252,
+            -0.021962244496450727,
+            0.0037071053072833147,
+            -0.013508376064336042,
+            -0.015178823540513864,
+            -0.02518557053061311,
+            0.011693256916760944,
+            -0.01716015674592331,
+            0.02048897513447166,
+            -0.003412078966361745,
+            -0.001807298511378529,
+            -0.0012575655363779904,
+            0.03116103961636643,
+            0.011668533361641842,
+            -0.011637010548030312,
+            -0.040260195687925236,
+            -0.011129376691522074,
+            -0.0034493180842871908,
+            -0.02098244789711689,
+            0.026658743116016938,
+            -0.01923736325953796,
+            -0.009723937181300192,
+            -0.013134679989565051,
+            -0.011974548914818263,
+            -0.004028712301052159,
+            0.019628329175867303,
+            -0.028192034966932684,
+            -0.012638520793012082,
+            -0.011568652932013111,
+            -0.0025919910519445737,
+            0.012497908463405111,
+            0.015113186712897145,
+            0.014880607940212641,
+            -0.03059227859452334,
+            -0.021899716217105084,
+            -0.0008513506358489523,
+            0.011985931118464679,
+            -0.01321584595585435,
+            -0.011735288366213981,
+            0.010710683541435907,
+            0.01961722172525993,
+            -0.023571946726192585,
+            -0.03364281880125354,
+            0.0014460076743756465,
+            0.016563167828467187,
+            -0.02891589059260944,
+            0.04595166331401113,
+            0.034271553243501465,
+            -0.027557971577291153,
+            -0.007138821520412525,
+            -0.039508407807651844,
+            0.06814329636830853,
+            0.00022136123564675344,
+            -0.039718016705753735,
+            0.0503457594963753,
+            -0.003580581891576583,
+            0.013820796398521455,
+            0.02054050419088757,
+            0.035682661890720495,
+            -0.0013220739026852943,
+            -0.0052759394697328675,
+            0.04733018719162478,
+            -0.0187845855131196,
+            -0.03982699468608443,
+            -0.023905460465070062,
+            0.002631233133039351,
+            -0.03090995554989513,
+            -0.012617612051488022,
+            -0.013605787969473513,
+            0.039672783793764745,
+            0.008669582242097126,
+            0.009974277556736362,
+            0.005535636914843583,
+            0.014180668493093584,
+            0.012042659947850441,
+            0.011870115226306776,
+            0.00451259454348581,
+            -0.014023048592193627,
+            -0.007769033996392623,
+            -0.05715987527668358,
+            0.003374564518384728,
+            -0.028801158523506985,
+            -0.00756331369170698,
+            -0.02163054894055572,
+            0.005367181573233089,
+            0.02308705505078254,
+            -0.005438909398653553,
+            -0.00873620909426313,
+            -0.0036530652237719077,
+            0.013920407174296543,
+            -0.01539613379172909,
+            -0.03908211213269472,
+            -0.016810726548844992,
+            0.007460173854386443,
+            -0.007069189426742819,
+            -0.041197222498539215,
+            -0.003843233633335066,
+            0.004373367480223065,
+            0.016194121660459908,
+            -0.029165150066068787,
+            -0.00735139702658342,
+            0.017257895082337448,
+            0.014153313280536095,
+            -0.002376178042741926,
+            -0.023223700836427403,
+            -0.0319877537205969,
+            0.02088060774052704,
+            0.0012829290529515294,
+            0.03251550040316258,
+            -0.04128453069131847,
+            -0.023376114412063578,
+            0.04083022735076889,
+            0.005690491480259003,
+            -0.05810403800689653,
+            0.016582434816685452,
+            0.010402038964417569,
+            0.016795644054313566,
+            -0.00908590043098552,
+            0.05003656634510042,
+            -0.037216197954986735,
+            -0.013027960261454162,
+            -0.007090943677204207,
+            -0.05175371787508879,
+            -0.0020065660448063322,
+            -0.014908489552053589,
+            0.010627472758306387,
+            -0.03275473161158777,
+            -0.017334608434133866,
+            0.025321047439048318,
+            -0.0033515195870701753,
+            0.011526068468647207,
+            -0.026783879816504975,
+            -0.0012830935419413407,
+            -0.017506908439998053,
+            -0.02114853058440477,
+            0.004878884893669383,
+            -0.020936900837115968,
+            -0.0025587027781733642,
+            0.01557850246191957,
+            0.023652335794299834,
+            0.011489428313303988,
+            0.0019089137782552768,
+            0.02054907941022243,
+            0.014361232983780595,
+            -0.0056413109308506085,
+            -0.0010300050480036684,
+            -0.03413262505887213,
+            -0.018821160485010483,
+            0.03361989907065425,
+            0.013125889701529858,
+            -0.003742167496379789,
+            0.008813100942466033,
+            -0.0383265293638935,
+            -0.035316957263713485,
+            0.03020499323039515,
+            -0.05001623971243151,
+            -0.00855479667673597,
+            -0.005939674940109814,
+            -0.026451831943555135,
+            -0.008032873049527808,
+            0.0009431571685654594,
+            -0.014387605128858954,
+            0.016382559541755014,
+            -0.024909510232038502,
+            0.003715736505444385,
+            -0.006016712902409688,
+            -0.009110197692342108,
+            0.05507263160354303,
+            -0.022103031032598353,
+            -0.0475415595313825,
+            0.021958434235402655,
+            -0.016670809659931243,
+            0.0323519363801999,
+            0.03867494951405658,
+            0.011124508964363016,
+            -0.008002445714431074,
+            0.034041838454818577,
+            0.018400123235267234,
+            -0.01161754789471199,
+            -0.025901844576206774,
+            0.021800298132870245,
+            0.00943488425697727,
+            -0.05906691123008531,
+            -0.00782549274387971,
+            -0.019357446956797447,
+            0.008682376344965976,
+            0.02863077694455766,
+            0.0024925977373894447,
+            -0.002682170968419412,
+            -0.013590235283325244,
+            -0.021511149507762284,
+            -0.03255453916518719,
+            0.009039307015853905,
+            0.008305286476500875,
+            0.009312625027668458,
+            -0.0014144165766983945,
+            -0.007853869334376857,
+            0.009067065760942163,
+            0.022485201611182105,
+            0.03966189080765722,
+            -0.041627289519629344,
+            0.002707086456239785,
+            -0.022678178050044222,
+            -0.03653753142622053,
+            0.0036574646778835588,
+            0.029996160786576367,
+            0.02901163765559891,
+            -0.01971777193183787,
+            -0.0036589934841109884,
+            0.009181234938554578,
+            -0.025979483002227535,
+            -0.00773645103995767,
+            0.004280770859737866,
+            0.04116668301017644,
+            0.006201085381597264,
+            0.01402728258112097,
+            0.001060407317110527,
+            -0.00920801784000087,
+            0.02109880889040528,
+            0.025211022083982727,
+            -0.029866678264942617,
+            0.014439702646820226,
+            0.009379206296908577,
+            0.04156435812161122,
+            -0.037309655469525846,
+            -0.011819731348299495,
+            -0.02766498020409985,
+            -0.010454143230794476,
+            0.027911532296566614,
+            -0.04127890742215716,
+            -0.006948536183086887,
+            0.010207784113909938,
+            -0.0016584043356136647,
+            0.016437495428919636,
+            -0.014607861750686952,
+            -0.03222311919805017,
+            -0.000610723203118213,
+            -0.02867445198803213,
+            -0.024772816792038133,
+            -0.016510309753766612,
+            -0.018581201775667577,
+            -0.07450654873867177,
+            0.00026832248829154255,
+            -0.06724696990816072,
+            -0.02447002541264633,
+            0.04885569249258513,
+            -0.010195604207134053,
+            0.03332905935704749,
+            -0.02815272112271314,
+            -0.005296340304911926,
+            -0.05908835042526019,
+            0.0038476172058970774,
+            0.025753576863447866,
+            0.02386003857017594,
+            0.005206447371391696,
+            0.024799057871126032,
+            0.012060738268265606,
+            -0.04146503002980051,
+            -0.024949907556862643,
+            -0.014503475223770958,
+            -0.0016340466065744286,
+            -0.0034009167013795906,
+            -0.031298432995411005,
+            0.036241272149880994,
+            0.008902644391348814,
+            0.03550223922667804,
+            0.04463301409716232,
+            -0.010385984413759852,
+            -0.03305004479734889,
+            -0.006952739338820577,
+            -0.0175981588092108,
+            0.042668533202572856,
+            -0.017112184620086308,
+            0.005149173014526686,
+            -0.03219589107997635,
+            0.024181808150698054,
+            0.0014339790422945406,
+            0.01254721190611376,
+            0.011099458649404119,
+            0.0027626836767364253,
+            -0.026883640446090158,
+            0.0024160770766463607,
+            0.023874514847956222,
+            -0.022558433617669373,
+            0.00917667715473093,
+            -0.006754091166465801,
+            -0.033686935718776144,
+            -0.01078389638118685,
+            -0.013595327486167114,
+            0.024712781935748277,
+            0.013168464807451798,
+            0.005875864604862434,
+            -0.02417814420558463,
+            -0.0023978800459714144,
+            0.028650463366555304,
+            0.020176980768301288,
+            -0.02118184201861616,
+            0.019082340279671355,
+            0.019734925740100277,
+            -0.04209645071102001,
+            -0.04217310001316092,
+            0.006553913719098956,
+            -0.003735719227468833,
+            0.0061793909220459,
+            -0.028285889794908054,
+            0.008181788076484304,
+            0.004432488437404309,
+            -0.01743905248466261,
+            0.041221269621866216,
+            -0.014892354419172107,
+            0.005087102208335201,
+            -0.022198383716186407,
+            -0.011936388003605221,
+            -0.00027487042024290816,
+            0.044599215801158845,
+            0.008068975171827777,
+            0.003496950921841073,
+            0.02238779276447619,
+            0.0054839808262270315,
+            0.04816723616939378,
+            -0.02685221260466647,
+            -0.0190356946337995,
+            0.016464136459560113,
+            -0.0176092780782677,
+            -0.012513274095432353,
+            -0.010695421445334002,
+            -0.004784348908457072,
+            -0.022203041334932165,
+            0.010788155380415583,
+            -0.018353897272137543,
+            -0.02277431189907027,
+            -0.003945465850796918,
+            0.016024639792926797,
+            -0.006248368166794934,
+            -0.0038769803892794875,
+            0.0016292556540434277,
+            0.02412864830338344,
+            0.02892955453819876,
+            -0.02408022470555968,
+            0.019190282372105023,
+            0.018941584034503536,
+            -0.006709516380602965,
+            -0.020956377933747532,
+            -0.01964885206488053,
+            0.01434623424964373,
+            -0.01352968586212152,
+            -0.013872556892707697,
+            0.004052798542613835,
+            0.009201147374299997,
+            -0.004086067985859617,
+            -0.04185829685685201,
+            -0.014668698123513235,
+            0.003179403563664022,
+            -0.0006415014901450319,
+            0.019360184344240947,
+            -0.0150817538567364,
+            -0.029771999278536083,
+            0.014221776097046751,
+            0.022944884879589678,
+            -0.04493425220030206,
+            0.02117577719533236,
+            -0.0030648751497261983,
+            -0.00016251533643103604,
+            -3.817296018601414e-05,
+            0.008727767382077138,
+            0.026914779443266063,
+            0.03456565094655899,
+            0.01974806772696219,
+            0.00450800007152441,
+            -0.008952599700759673,
+            0.025184190461155245,
+            0.009846044083064704,
+            -0.025168735183134738,
+            0.010983353565382029,
+            0.016740721743326522,
+            0.0070838930012654296,
+            0.009173879407277282,
+            -0.006078288608766152,
+            -0.018269347893097643,
+            0.013181494188066667,
+            -0.01435693747022231,
+            0.01678907949532146,
+            0.05154463552784974,
+            -0.00039632596424461964,
+            -0.02461439550772063,
+            0.008418147037164186,
+            -0.006544319842335593,
+            -0.010065129286193278,
+            0.02214628550460128,
+            0.0029515574751370417,
+            -0.030862152033847717,
+            0.007509556025223583,
+            0.00675485304374519,
+            0.01894936231559203,
+            -0.02355525295639599,
+            -0.01387885858544803,
+            -0.00431398020087544,
+            0.0021208056468763523,
+            0.01580766504149989,
+            -0.004739477121073512,
+            -0.013204823447960706,
+            0.02994098229517376,
+            0.025496728234662373,
+            -0.00026124946738213036,
+            0.015060064980768175,
+            0.022262345898533903,
+            0.009968408664790368,
+            0.007588431141088797,
+            -0.0083063413609779,
+            -0.031336272171937095,
+            0.0018957858383561667,
+            0.05800811417728381,
+            -0.014937085296865802,
+            0.04313875432235027,
+            0.02018476425923236,
+            0.015657644644987107,
+            0.017880263053253607,
+            -0.01662268082870562,
+            -0.008740053682875323,
+            -0.0017630394204179638,
+            -0.008294417990024647,
+            -0.027613464270562948,
+            -0.005953209556351086,
+            -0.02379961210881074,
+            -0.013878676554701543,
+            0.015418315245979704,
+            -0.014764728065382074,
+            0.0027420640870858457,
+            0.010395538988699663,
+            -0.006590633833835581,
+            -0.00841537209927409,
+            -0.003943772700797381,
+            0.03376338017637296,
+            0.011980354721458463,
+            0.027761929822019422,
+            -0.00011128539412582973,
+            0.009342430952107655,
+            -0.018419526564396063,
+            -0.006839752805318664,
+            0.03271190887864607,
+            0.006011520427944665,
+            0.004652835471939905,
+            0.01485920920301398,
+            0.014305626381398871,
+            0.038015677219487023,
+            -0.005102005401565164,
+            -0.018204460895989052,
+            0.02065495133597456,
+            -0.0049699013356374775,
+            0.019149036693562712,
+            0.008825981577388056,
+            -0.002660055848523568,
+            -0.004292379561953203,
+            0.03090270710860331,
+            -0.020243440437217503,
+            0.018735727616321387,
+            -0.02238236089688497,
+            -0.0276584413929402,
+            0.023028408906176946,
+            0.008320436733620496,
+            0.04047641118026783,
+            0.011463851066078041,
+            -0.02247143742166039,
+            -0.034588725413285185,
+            0.009827713930789184,
+            0.00337071450636392,
+            -0.024446280199879062,
+            0.017542777160283324,
+            0.00780836483490212,
+            0.008453886667283516,
+            0.016624116261219637,
+            -0.004175682824655864,
+            -0.0042784362933758835,
+            -0.001546656717896957,
+            -0.008819739495525477,
+            0.01406790007187259,
+            -0.015487658289541583,
+            -0.021457182907140035,
+            0.024687169583245905,
+            -0.03915911962859764,
+            0.01771290139419629,
+            -0.006510046907173026,
+            -0.00997450880725535,
+            -0.02059560682587115,
+            1.5939836949198004e-05,
+            -0.007159493752862143,
+            -0.0004782363814453761,
+            -0.017503181957042913,
+            0.016643971600145645,
+            0.024056442566869802,
+            -0.013209211443909658,
+            -0.00445521990927108,
+            0.0020313626821877526,
+            -0.029235692455820056,
+            0.006662209097255912,
+            0.016993027053110635,
+            0.00529940612845993,
+            0.0026104743851393716,
+            0.0001707934714983891,
+            -0.031289446611163674,
+            -0.005782208620284939,
+            0.03053516889069683,
+            0.05590704747438521,
+            0.021628254147289437,
+            0.02403314981042378,
+            0.02908606663381421,
+            0.02717206823892633,
+            -0.003198844932040613,
+            0.0016217508103442348,
+            -0.01620660606431983,
+            0.027098323143010743,
+            -0.011664168290676943,
+            0.04322175021589083,
+            0.008652323184522541,
+            -0.049493020736445056,
+            0.004537958157696794,
+            0.00943239454147692,
+            0.034918603888916544,
+            0.016678716235690318,
+            0.0005887414054803805,
+            -0.005894338664998101,
+            0.03279165663336323,
+            -0.02735726709032382,
+            -0.011962070182991011,
+            -0.00036014407560325284,
+            -0.019079003730607717,
+            -0.015938295169860884,
+            0.010357611943935459,
+            0.04278924678845588,
+            0.0009957483598272451,
+            -0.0071422790468850215,
+            0.01571201177743999,
+            -0.009445307674383,
+            0.02286338492075306,
+            0.007336440384050498,
+            0.0042809497342459904,
+            0.025581467690095977,
+            -0.0025007871391868105,
+            0.02300539823997411,
+            -0.000628898771317254,
+            0.004582863909134464,
+            -0.015960578501883316,
+            -0.00930649396114688,
+            0.027838455754803444,
+            -0.011112268986721,
+            -0.029253746504396393,
+            -0.008734785561831333,
+            0.00472579547270089,
+            -0.0003805825712986138,
+            0.0021854358149324227,
+            -0.01183885533661493,
+            -0.010840951201591629,
+            0.019965691788904057,
+            -0.004922943301424112,
+            0.003800048754664761,
+            0.012664049429916252,
+            -0.0030124219674414897,
+            -0.01910844996750822,
+            0.01889983984736845,
+            -0.003858283778519141,
+            -0.024168821338992813,
+            0.020083955711766945,
+            0.01850497889545421,
+            0.045156567155119454,
+            -0.01586824055785609,
+            0.004612757419198795,
+            0.017497774574602505,
+            -0.002172088240279357,
+            -0.019617327081566827,
+            -0.014944795772093997,
+            -0.018212462639041492,
+            -0.015975283672064702,
+            0.012837956594691596,
+            -0.01243789877725244,
+            0.019500571353639314,
+            -0.0380735226388175,
+            0.012590900534826689,
+            0.004832168601136412,
+            0.007505385730121269,
+            0.016011983211212992,
+            0.025994560026860358,
+            0.002157436199771672,
+            -0.033465476168065784,
+            -0.01014599891204166,
+            0.007845373376916211,
+            0.010093432683262838,
+            -0.002849185178237036,
+            -0.018147004183434554,
+            0.003633790611809425,
+            -0.0266147420637404,
+            -0.018644422659309035,
+            0.022926737299761104,
+            -0.04798772015599113,
+            0.017908415301694906,
+            0.024859125158586466,
+            -0.04343699525707025,
+            -0.04886699607556352,
+            -0.011459188432595229,
+            -0.036774381316312464,
+            0.0014128005215813578,
+            -0.0005256067584181517,
+            0.011194992852264102,
+            -0.009342140079829378,
+            -0.018598671981710183,
+            0.01584789414301952,
+            -0.0036412641242200418,
+            -0.004225840239935549,
+            -0.00616696736018045,
+            -0.0025598631463178384,
+            -0.010188803157718543,
+            -0.03244694662411977,
+            -0.03290891136780909,
+            -0.00033742479296272127,
+            -0.009743927051414455,
+            -0.02334288359945272,
+            -0.03447062782872289,
+            -0.008728968107967028,
+            -0.037753659749578956,
+            -0.00515561961442376,
+            0.04070890150714597,
+            0.027517147608335957,
+            0.009449414978133334,
+            0.008086900089929912,
+            -0.013989034604490105,
+            0.05691047488419298,
+            -0.020425883381389086,
+            0.039466620642512155,
+            0.005301493569940605,
+            -0.0037222353899918703,
+            0.013075425214878277,
+            0.027710826422351564,
+            -0.006253665106995653,
+            -0.0017412928073238833,
+            -0.01928096630475438,
+            -0.012920476859996416,
+            0.009444408011467955,
+            0.015799884767184885,
+            -0.0025083061619197524,
+            0.001274457251739847,
+            0.00930533161700852,
+            -0.0006652478540264343,
+            0.017258038491335247,
+            0.03996268817486439,
+            0.003833844612473671,
+            -0.00826404141706295,
+            -0.018822002570963614,
+            -0.003188844900317347,
+            -0.011807006674248829,
+            0.010482058734260632,
+            -0.015410635475145028,
+            -0.011540803927522013,
+            -0.01249428777269578,
+            -0.017081856786754605,
+            -0.002183034976477535,
+            0.003114834030705039,
+            0.0027475411722881747,
+            0.02261627635192778,
+            -0.015872027027080824,
+            -0.022487581027490516,
+            -0.0035823117885461506,
+            -0.02398778449664045,
+            8.13716819067491e-05,
+            -0.01050060782407259,
+            -0.015616181283381218,
+            0.013676480044963827,
+            -0.021513173575512936,
+            -0.01091399089076612,
+            -0.008699176177861256,
+            -0.0006986953850759474,
+            0.005853723710671249,
+            -0.00993355748621621,
+            -0.00010241843582159751,
+            0.01150872694850241,
+            0.005559552266827916,
+            -0.012336449703762907,
+            0.0039431718621504,
+            -0.012740550272925562,
+            -0.004432059174951158,
+            -0.03592309122711106,
+            -0.019665686718735042,
+            -0.01846795637714379,
+            0.005642094448288269,
+            -0.005108964028543505,
+            -0.020586678054151854,
+            0.00668364061048941,
+            0.0034775127050453774,
+            0.004713169526661792,
+            -0.010723667875209928,
+            0.013944735241748791,
+            0.010882420168586074,
+            -0.003518245916227596,
+            0.013827334494230662,
+            -0.001330399936559009,
+            -0.020286697504699764,
+            -0.011139922402071095,
+            0.03538258948179278,
+            -0.024961826464903997,
+            -0.0018019328351763667,
+            -0.0026423695690168066,
+            0.00597003516870726,
+            -0.03071835946501977,
+            -0.013634212232744325,
+            0.02164400349828422,
+            -0.005846552894831441,
+            -0.026367014426434226,
+            0.03399116480234033,
+            0.03473258231629753,
+            0.010041605592929843,
+            -0.013423384340589794,
+            0.04730054254696893,
+            -0.012968978954938308,
+            -0.018274800841400665,
+            0.026826644936918647,
+            0.0278389611094493,
+            0.02525164095926168,
+            -0.029782550447890466,
+            -0.003010713086487275,
+            -0.02393629084357885,
+            0.017993366287762866,
+            -0.0263476663180971,
+            0.01864011499179424,
+            -0.02332263152537849,
+            0.016014908881756034,
+            0.01868464825773604,
+            -0.003104041044182249,
+            0.026313491086684425,
+            -0.0052945248919367815,
+            0.015253731232325074,
+            0.0012371308759955573,
+            -0.02119765517013431,
+            0.007795466385020642,
+            -0.006469500272298062,
+            0.025762169721880456,
+            0.00833992768202466,
+            0.01801901639978976,
+            0.022019940825351116,
+            -0.0409522298213741,
+            0.02003698451081499,
+            -0.001367916864679543,
+            0.01782984895971748,
+            0.0038516134262842373,
+            -0.007885432294782304,
+            0.014221728513400065,
+            0.010991352188016265,
+            -0.012851099261924757,
+            0.003840539873231741,
+            0.0008923482291826014,
+            -0.028928104542716937,
+            0.007066004768771198,
+            -0.026914574065060926,
+            0.025866907400807087,
+            -0.005526774784152086,
+            -0.00998122210956923,
+            -0.020744532673097317,
+            -0.0071425207262770865,
+            0.003754468896273156,
+            -0.012747874997429746,
+            -0.011385805601999295,
+            -0.024700457669046855,
+            -0.02401291382882952,
+            0.029591582802488962,
+            -0.014054332096344003,
+            -0.04015978298687865,
+            0.001234440863385445,
+            -0.023301255402129767,
+            0.01193125007773239,
+            -0.005793124129267758,
+            -0.024440100649464773,
+            -0.042563399082257154,
+            -0.017548888269389457,
+            -0.0009394887961485922,
+            0.022739763644266633,
+            -0.00114156949394707,
+            0.041791395064755515,
+            0.0008816696205437911,
+            0.022207132857533028,
+            -0.015547580543606755,
+            0.005319731264840819,
+            -0.02071229859314312,
+            -0.041495579742912285,
+            0.03660432378985971,
+            -0.014637031447181937,
+            0.025854844497435178,
+            0.03324325296704186,
+            -0.013009347371118843,
+            0.001354504885302625,
+            0.02652570709474851,
+            -0.014618100923440427,
+            -0.010782232747863238,
+            -0.0034936699515378212,
+            0.003461670978569965,
+            -0.006468116285021298,
+            -0.027793221545159466,
+            -0.007752458229193391,
+            0.04193706715422074,
+            0.017894069367616637,
+            0.021928994684423043,
+            0.022606401934291136,
+            -0.017972404868085358,
+            -0.026402048654300563,
+            0.003656646576244765,
+            -0.013129722990153284,
+            -0.018090360146368677,
+            -0.01624598856366761,
+            0.012397150789705746,
+            0.011741002980073786,
+            -0.004052422390675261,
+            -0.01836750423615859,
+            -0.020929335029248472,
+            -0.016327412417122416,
+            -0.0042615345226698015,
+            -0.008458117095535,
+            -0.005002582649405199,
+            -0.012158173174049797,
+            -0.020409616620022143,
+            0.03706368540614453,
+            0.0285565642978937,
+            -0.0041732933884846566,
+            -0.00607154864704767,
+            -0.011142150176334957,
+            0.008834431806990914,
+            0.006021232046087652,
+            -0.005028870471297621,
+            -2.3396043292139426e-05,
+            0.005112033372713154,
+            0.022822312435625314,
+            0.007271235299348001,
+            0.0035791789459533856,
+            -0.06340305811365254,
+            -0.005392000517722054,
+            0.022560201096078862,
+            0.012581401789917604,
+            -0.0032248219865223496,
+            0.025180718735801837,
+            0.0006451724964970927,
+            -0.009714514747463593,
+            -0.018389422219956937,
+            0.024728862150150004,
+            0.008760329201960964,
+            0.020988700992999768,
+            -0.014974763313686516,
+            0.003845297486923296,
+            0.0006445983847138896,
+            0.00750394530065325,
+            -0.0019792992847026793,
+            -0.008578254004439607,
+            0.025776677078006516,
+            0.016593438298233645,
+            -0.005695578040595484,
+            0.01604018433477423,
+            -0.0017497260488899657,
+            -0.021460543443428924,
+            -0.010265213095609554,
+            0.00042017209251926397,
+            -0.010570864288876387,
+            -0.018015790888295944,
+            0.017569414447505878,
+            -0.002601036282535863,
+            0.0072399377790164435,
+            0.030281707151249355,
+            0.0423183067569767,
+            0.009566493398534713,
+            -0.005928603293907002,
+            -0.0037936911251953654,
+            0.015477873304924939,
+            0.01513127995244932,
+            0.010741256034652754,
+            0.03015858543221727,
+            -0.029247492875021843,
+            -0.006690306960497428,
+            0.0046653727410352576,
+            0.004610405633429778,
+            -0.011692179099382171,
+            0.030055897497741527,
+            0.04569730592994177,
+            0.006736813244286893,
+            -0.004799978513117371,
+            0.002121067028052434,
+            0.011299889107780231,
+            0.007768137218140627,
+            -0.018927080714961783,
+            -0.0012222962555201746,
+            -0.006009911026251445,
+            -0.014001766533604134,
+            0.026941585318261085,
+            -0.003960837039351648,
+            -0.011848232223811548,
+            0.02903357274500332,
+            0.1105267435019781,
+            -0.006011029338781496,
+            -0.0053646998281195206,
+            -0.03242321704621517,
+            -0.015194102811332289,
+            -0.008860590919147765,
+            0.0056988205877963076,
+            0.01635957585050017,
+            0.020327661913559888,
+            -0.0008943408274661007,
+            0.016820860249955754,
+            0.02822175614376368,
+            -0.013854827839129885,
+            0.007906244895327178,
+            -0.030891381690403466,
+            0.015702117924362928,
+            -0.005383974926093977,
+            -0.012032955400313954,
+            -0.0026169511828034054,
+            -0.008618307035330928,
+            -0.0037356680273096105,
+            0.0009666430093455632,
+            0.011296307527684261,
+            0.004427516814276546,
+            -0.014484198313369681,
+            0.002829529156309784,
+            -0.008595300985863424,
+            -0.005122996529848575,
+            -0.02393571073523279,
+            -0.024119245543788293,
+            0.005223147047205755,
+            0.0019389305548629142,
+            6.344308292376579e-05,
+            0.006848965054539454,
+            -0.003810712095343454,
+            0.030149081317333418,
+            -0.01700322008272963,
+            0.0012288912810468446,
+            0.01193226302726573,
+            0.012332749933413344,
+            0.01688602002694633,
+            0.0028600984823761366,
+            -1.9882692552530084e-06,
+            0.021172977047420227,
+            -0.010514036509402561,
+            0.01460334896316005,
+            0.013091417303145243,
+            -0.04103572381458182,
+            0.0089746401463863,
+            -0.0027841289484449873,
+            -0.024252447218794344,
+            0.004446472209246425,
+            -0.018987867343500102,
+            -0.00881974688349601,
+            0.0005365625696166774,
+            0.012054719659248512,
+            0.01960300421041832,
+            -0.008553943699846696,
+            0.02111201386480852,
+            -0.0003501657740808554,
+            0.004177088753111459,
+            -0.012879944670340026,
+            0.0065222126975051685,
+            -0.007563354540402343,
+            0.0008168128607271199,
+            -0.018213241937597355,
+            0.012526561612175514,
+            -0.0008285725179894544,
+            -0.015737003159436623,
+            -0.011337688275309921,
+            0.004737346137976519,
+            -0.0027993545140114705,
+            0.0030843163114700313,
+            0.010153740340626154,
+            -0.008815302484351509,
+            0.004919275910399053,
+            -0.0017541042538165297,
+            0.002562816687714991,
+            0.014285249186421434,
+            -0.029816964593980453,
+            0.005357227738099325,
+            0.009154842495811638,
+            -0.00048782303639333943,
+            -0.013074976152495576,
+            -0.004054505559072866,
+            0.0028322102449485015,
+            -0.016877328079971877,
+            -0.012128199639779175,
+            0.023421330164419888,
+            -0.012565145848820333,
+            0.008609070188894315,
+            -0.010936598722317274,
+            -0.018401801734378692,
+            -0.01066119413133089,
+            -0.005254229780560808,
+            0.01234969508270852,
+            -0.0038427767204741877,
+            -0.006947594635952202,
+            -0.01756894543880097,
+            -0.007890897373987378,
+            -0.047246698568222466,
+            0.009617876521097624,
+            0.027157555263862723,
+            -0.011355758309667893,
+            -0.04050311381480088,
+            -0.01017774338301804,
+            0.02437257208967103,
+            -0.004056329295095018,
+            -0.008345450168215274,
+            -0.001599171409135998,
+            0.018358386463243014,
+            0.0027207445181251406,
+            -0.007292967749349572,
+            -0.011267861428599221,
+            0.01657507035043879,
+            0.015105061280816214,
+            0.021419854579631276,
+            -0.02088952706825817,
+            0.006295984081511286,
+            0.020973020776936996,
+            -0.03073246543555915,
+            0.0023720435876790876,
+            0.03363966216709148,
+            0.010412588613998134,
+            0.013023311559810034,
+            -0.0010217662933572685,
+            -0.0075567381078977215,
+            -0.008248818794295818,
+            0.011848200323549865,
+            0.005066463524567732,
+            -0.023174489297821385,
+            0.021823825355858556,
+            0.020368868756448753,
+            0.0026042856858509524,
+            0.003457067317891926,
+            0.0005539745115410357,
+            0.01062670844108589,
+            0.009948889803890108,
+            0.05100741556350918,
+            -0.006493714124814164,
+            -0.014021751055930108,
+            0.015063988102064609,
+            0.02488059237697814,
+            -0.0023499941119596228,
+            0.01087282522970853,
+            -0.02612218935931224,
+            -0.021423880914461062,
+            0.023389294741183817,
+            0.008967763132667892,
+            0.009637806238448432,
+            -0.010815706635624337,
+            -0.011390999462441815,
+            0.031750675158698435,
+            -0.01307829462211914,
+            0.00759749911213318,
+            -0.02169895957209634,
+            0.01387879409130081,
+            -0.008606362268863441,
+            0.032229658009209815,
+            -0.0009066257436946125,
+            0.011062436424260079,
+            0.028010486866468023,
+            -0.007835805879241447,
+            -0.019657486763625547,
+            0.003967480939987674,
+            -0.015577952617624695,
+            0.02188429826317828,
+            -0.020502758155233298,
+            -0.0002362500702658591,
+            0.0040935406015014265,
+            -0.010056462958039452,
+            -0.0057038083589231615,
+            0.011475170331848666,
+            0.0015345076181482267,
+            0.006359262977587827,
+            -0.020293014401106,
+            0.01369706440475074,
+            -0.003989482215141903,
+            0.007887824197823247,
+            -0.015925992045349828,
+            0.014370223626942766,
+            -0.02309139937510304,
+            -0.021808736033141787,
+            0.008142621179624918,
+            0.0108996933426044,
+            -0.007589050511458233,
+            -0.025971539510030145,
+            0.02250992080022228,
+            -0.024980833805349545,
+            0.024979008145354243,
+            -0.006622339940663538,
+            0.002566378488088469,
+            0.004913874740363343,
+            -0.008102103508732738,
+            -0.0028098905427027783,
+            0.0057865587191235565,
+            -0.0073509382400023995,
+            -0.014123756810537343,
+            -0.019453380788786282,
+            -0.04083629135586275,
+            -0.03177927047231389,
+            0.022927581786088044,
+            0.011557232599801421,
+            0.0017073832658476193,
+            0.02339552660545296,
+            -0.02247471449131386,
+            -0.017890866287500053,
+            -0.009775068417716798,
+            0.025184899522229628,
+            -0.04408541669037361,
+            0.012104255392615549,
+            -0.014669458760022245,
+            -0.0245258242481677,
+            0.006274252098230249,
+            -0.004161960147481513,
+            0.009299171724026455,
+            -0.010779421730862708,
+            -0.04713627671791054,
+            -0.005192684784377356,
+            0.02330975631183273,
+            -0.0036802876364168935,
+            -0.006197548344287773,
+            0.03048284650641829,
+            -0.0019503797069087845,
+            -0.005038254331399784,
+            -0.005524875459245342,
+            0.027783596017224226,
+            -0.024950083604372984,
+            0.010988398531346805,
+            0.008304937912856357,
+            -0.008485065895193593,
+            0.006572665697403556,
+            0.0011518846966102158,
+            0.011927348472282199,
+            0.015632588644277183,
+            -0.015559413716826264,
+            0.02262990861928957,
+            0.011446850286896198,
+            0.006506075777765303,
+            0.008015225053785824,
+            0.00010054430757843335,
+            0.0022975742216194894,
+            -0.024948925449668525,
+            0.026047924967851216,
+            0.01888211557921096,
+            0.0001632181017876392,
+            0.012553094476616919,
+            -0.015042621225242777,
+            -0.02782009986290425,
+            0.02606646297992026,
+            -0.016847653820092335,
+            -0.00045272379465939274,
+            0.020053091486344633,
+            -0.016210036358455773,
+            -0.007930550661303182,
+            -0.01123514849773852,
+            -0.0162675150307978,
+            0.01803076559921804,
+            -0.012160217543572069,
+            -0.02216594536057653,
+            0.002594675636753233,
+            0.016443123828746582,
+            0.02134480656850119,
+            0.026694575953313482,
+            -0.011470930697351533,
+            0.017768927048191052,
+            0.0008653408391607816,
+            0.001865765599795092,
+            0.025745954171649543,
+            0.01702677011968492,
+            -0.008474889109168202,
+            0.025347176686722313,
+            -0.0016686223698600755,
+            -0.002421921060670558,
+            -0.012265504084322075,
+            -0.0165214436458303,
+            -0.007545471138114208,
+            -0.015333235556146196,
+            -0.0051830106590765685,
+            0.000762746782664525,
+            -0.0018118022696726346,
+            0.10876576838038368,
+            0.0034948044540582626,
+            -0.011608460689239234,
+            -0.005870263500417206,
+            0.007510774308250375,
+            0.0023525201199670765,
+            -0.006846154764389575,
+            0.016082721103476375,
+            -0.018616626204086575,
+            -0.023593744079394,
+            0.004743481974389998,
+            -0.0009262286319469741,
+            0.00870122878944883,
+            0.02029523251993269,
+            -0.008608837818169835,
+            -0.0200567711498164,
+            0.007089477722191928,
+            0.005535149644041221,
+            0.0023640746516071774,
+            0.005256431824732927,
+            -0.0006118671321824933,
+            0.008209920240361308,
+            0.02448422122961942,
+            -0.015622892878860465,
+            -0.029770836607612903,
+            0.010535371601298532,
+            -0.00015994156470228762,
+            -0.007193425789067776,
+            -0.011046316463797128,
+            0.014728841492009627,
+            -0.0011488451990980268,
+            -0.02751404133172324,
+            0.017663132712536528,
+            -0.019055335178900415,
+            0.014643246200933506,
+            0.0025348345163347464,
+            -0.00016387173410118788,
+            0.022565953069005714,
+            0.013101158251834834,
+            -0.04272085076882881,
+            -0.0034442020309914823,
+            0.01041205537746844,
+            0.011059422927946552,
+            0.0116446016862942,
+            0.0012992535381387887,
+            -0.018670801241345514,
+            -0.011788984490283379,
+            -0.009109849103208542,
+            -0.021268488414296117,
+            -0.012523033785938503,
+            -0.011393853408628214,
+            0.010743646386439633,
+            0.0012964030814325153,
+            0.023809566099779113,
+            -0.003030535542277875,
+            -0.006526351867141901,
+            -0.000430699447948508,
+            0.00880561288237257,
+            0.009694646009259077,
+            -0.02079011138439714,
+            0.008294338719512953,
+            -0.01823922789220214,
+            0.011819769685519291,
+            0.006416044253693268,
+            -0.022799568832306067,
+            -0.009911619276012527,
+            0.007152786695893951,
+            0.008393677195358034,
+            -0.013569953300860946,
+            -0.006503988731872938,
+            0.02299498754558268,
+            0.018870619152710703,
+            0.012233964644738905,
+            0.024994616203196145,
+            0.03838969722235787,
+            0.005744255912479976,
+            0.027682123067526515,
+            -0.016791471092091604,
+            0.008459745383680297,
+            -0.022493868174622695,
+            0.012902262638928338,
+            -0.00460399519266689,
+            -0.0050431343566427886,
+            -0.018188192498411616,
+            0.033765105245412236,
+            -0.009159751849801773,
+            -0.0027150644587327763,
+            0.034778224828120714,
+            -0.0066502021793471075,
+            0.004525230911930458,
+            0.005096748531775903,
+            -0.016067049494073893,
+            0.033274672789154185,
+            0.01686831040872425,
+            0.016353966592947262,
+            -0.021631941883929817,
+            0.011748317746828848,
+            -0.005675701689448899,
+            0.0001273442591116645,
+            -0.020738819037390263,
+            -0.0037528323797385733,
+            0.0027725541343763104,
+            0.0059860509282538546,
+            -0.016556134518370437,
+            0.02033623476757896,
+            0.00019647023567641008,
+            0.0052895744489259875,
+            0.03253462247073863,
+            0.01043311189804924,
+            0.019182938878855655,
+            0.010051051068017807,
+            0.019523739618278902,
+            0.0281939391326836,
+            0.025685787482623124,
+            0.0029932688462254147,
+            -0.018668945726139946,
+            -0.0009326901784812825,
+            -0.007880352863625262,
+            0.010492063055849727,
+            0.022849902214818293,
+            -0.022619228461640748,
+            -0.02772644874135741,
+            0.018566981184526675,
+            -0.0022608185345048414,
+            0.020031103834122595,
+            0.007544464625538984,
+            -0.020935181884939796,
+            0.003407355515103286,
+            -0.023178230166337178,
+            -0.0005418155871620593,
+            -0.031039187720144967,
+            -0.028591174077243604,
+            -0.0007659241904525392,
+            0.011945344534674635,
+            -0.01688153017778311,
+            0.03357788163306365,
+            0.017667778671101954,
+            -0.014289109343082812,
+            0.01604684528817524,
+            -0.011054587325539086,
+            -0.0036959091138078683,
+            -0.005819853870004682,
+            -0.00203755010992417,
+            -0.02157944474979592,
+            0.005138614763345542,
+            -0.0033509138150859593,
+            -0.0024871060745482883,
+            -0.016936575573598154,
+            -0.021901830366872595,
+            0.016375173843894617,
+            0.016742770709641697,
+            -0.006606252542838758,
+            0.039771905404651496,
+            -0.004238071340064403,
+            0.00018364315349545657,
+            0.011690521707403059,
+            0.0010018392078563795,
+            -0.013651921068215563,
+            -0.010829994512513501,
+            0.012362647379788833,
+            0.011321403447882543,
+            -0.018942043512252663,
+            0.005763773608775982,
+            -0.009131565038297889,
+            0.002089216671850269,
+            0.005438887787329238,
+            -0.008183835275629478,
+            0.006690280329947951,
+            -0.01673229763262922,
+            0.015833397876364723,
+            0.00295474085300386,
+            0.003186554859191664,
+            -0.011797859525926772,
+            -0.020991927615680524,
+            0.01576413532776261,
+            0.0032403514518527287,
+            0.018819824905201635,
+            -0.026755218746060662,
+            -0.00012946467986739473,
+            -0.009552681500884924,
+            -0.0028941357661144826,
+            -0.012533800273548772,
+            0.013975640628890588,
+            0.016864739319953755,
+            0.0063941926578529,
+            0.01781016436903586,
+            -0.0026268224264609724,
+            0.015368284773773678,
+            -0.018122416427266218,
+            0.008686009180697955,
+            -0.006801074032721299,
+            -0.018537521789181048,
+            0.0013237338264105415,
+            0.027863741654435434,
+            -0.004670012662973029,
+            0.011066853115455966,
+            0.011704884649534624,
+            0.018954056680143452,
+            -0.0012626215734593707,
+            -0.0030850559460101866,
+            0.015610404665171367,
+            -0.005838226405658572,
+            -0.006861883833171456,
+            0.020872966233916948,
+            0.032951653763679145,
+            -0.006716215648910268,
+            -0.0021292599951407863,
+            0.010886596327481677,
+            -0.004901684487088035,
+            -0.008644584974417106,
+            -0.009587037829603014,
+            -0.03158137900717491,
+            0.024369089446706894,
+            0.012092412439477474,
+            -0.009601042951586109,
+            0.022690862146548786,
+            0.020752792552278052,
+            0.011412743968296708,
+            -0.010471860708195726,
+            0.02291536038589634,
+            -0.01268035142112569,
+            0.024168869865226136,
+            0.032482831775271157,
+            0.004400973556358811,
+            0.0022312906286559167,
+            0.0021175489661843985,
+            0.01691348460546639,
+            0.006050357346231155,
+            -0.007068133214675996,
+            0.011656003476409986,
+            -0.018487923180009552,
+            0.01598530891381001,
+            -0.004432366596808033,
+            0.02808371007118948,
+            -0.01337145216499331,
+            -0.010473700033368562,
+            -0.00537325376270318,
+            0.00010726431372328729,
+            0.006445701087575359,
+            -0.003324002570256215,
+            0.005255443653813165,
+            0.0007072577303252132,
+            -0.017825649146944237,
+            0.003050128130393781,
+            0.004825967170196209,
+            0.009018799597337101,
+            0.004103653497834134,
+            0.007249957401824204,
+            -0.012360355929483,
+            0.01441885664789707,
+            0.004401820363035808,
+            -0.0026190938717927086,
+            -0.0007163225760398784,
+            0.011124538295057815,
+            -0.013232822471919533,
+            -0.004895624967029031,
+            0.005281228035321008,
+            0.0059275753353475414,
+            -0.007507924998741734,
+            -0.004484140989405051,
+            0.011908076549392969,
+            -0.0037917291321648483,
+            0.006351543504726163,
+            -0.004409464890606014,
+            -0.0037760571637986827,
+            0.005303661313434895,
+            -0.00974511544836468,
+            -0.004204167898342997,
+            -0.0007093125445559502,
+            -0.006869250680233214,
+            0.00786202375969793,
+            0.005971033227117194,
+            0.010606085272785863,
+            0.006969437776542303,
+            0.004293129058563288,
+            0.004114452739123398,
+            0.004071017374828283,
+            -0.002445681623750424,
+            -0.00592480769363354,
+            -0.004617501328052899,
+            -0.012067458701076778,
+            -0.008335239153829813,
+            0.009025028998949398,
+            0.008653779421856954,
+            -0.019363018792788574,
+            0.02048011578221327,
+            -0.001188320960771659,
+            0.00772165044232104,
+            -0.01602086976290928,
+            -0.000907553574811279,
+            0.005464880629734941,
+            0.008267702703609528,
+            0.016136762227127184,
+            -0.03654209526492064,
+            0.005179791231187316,
+            -0.0037321883493524766,
+            0.016026013709955444,
+            -0.008215703411373182,
+            -0.006979816286354862,
+            0.02564060526656963,
+            -0.01450763388985563,
+            -0.0009847758228920171,
+            -0.016142304341196283,
+            -0.01407778890808498,
+            0.010815725252959506,
+            0.02050940280233426,
+            0.020684808835848383,
+            -0.0004965802897137407,
+            -0.0005781445948759235,
+            0.00036179841583004193,
+            0.022775804864342608,
+            0.015545681427566235,
+            0.018697154644607316,
+            -0.004543500782944068,
+            0.007028436664826706,
+            -0.023057215501463973,
+            0.009912581556496658,
+            0.0011795417129232596,
+            -0.008215027850908477,
+            -0.008554137076267468,
+            -0.008703378564520125,
+            -0.010448538943182258,
+            -0.011773811327190638,
+            0.004106616839960258,
+            0.013895013678116032,
+            0.004190549543250594,
+            0.0031571190600981074,
+            -0.0225385788570314,
+            -0.006243193914542516,
+            -0.011229641377928767,
+            -0.01674214043453308,
+            -0.007367389908398082,
+            -0.02386334969150212,
+            -0.028021339684039734,
+            0.0002794207370355175,
+            -0.02411861258809571,
+            -0.015958336406924612,
+            0.005268815864773021,
+            -0.01546610225139129,
+            0.00898600137695074,
+            -0.008396481690803911,
+            -0.011873486990732417,
+            0.0074743033679134105,
+            0.015336444788692,
+            -0.017676977079590463,
+            -0.010165317702640217,
+            -0.01690524990506033,
+            -0.013460451917691695,
+            0.010918550283871637,
+            0.00595960977280955,
+            0.009250157926438213,
+            -0.012447979337850305,
+            0.0062932435120863365,
+            -0.012673905639312696,
+            0.015073013552721701,
+            -0.03769166067890086,
+            -0.029992249463405965,
+            -0.00014990754540999537,
+            0.007288036630677436,
+            -0.0020735742042505426,
+            -0.018695120921334806,
+            0.0003860116505754868,
+            -0.003608995261360232,
+            -0.014900191335144973,
+            -0.013866216738266176,
+            0.025251475305524053,
+            -0.006788483107616133,
+            0.01223227424986397,
+            0.0034937277040015626,
+            0.03438333560343413,
+            0.008803852645562129,
+            0.014666206816302367,
+            -0.005709421955160623,
+            0.003742433643393657,
+            -0.0033706258426092744,
+            0.006906464031352315,
+            0.0076578888497945185,
+            0.009276592191710524,
+            0.008025321004305927,
+            0.004576470294889512,
+            0.002887346543823031,
+            -0.001925468214331082,
+            0.02125241545487307,
+            0.025671028960818562,
+            -0.00037467906648160044,
+            0.008981063192139038,
+            0.0009387900084090577,
+            0.0002735789978680719,
+            -0.008374290340790827,
+            0.006547013284767913,
+            -0.008274637814401893,
+            0.004041066530254948,
+            0.012897549078838394,
+            -0.009150863940242148,
+            -0.010211568689323908,
+            0.016575088843292566,
+            -0.012380099999574783,
+            -0.03420455578618604,
+            0.026550654680719987,
+            0.005324512466403649,
+            -0.0035670106106754245,
+            -0.010657928468872608,
+            0.0037176027853158167,
+            -0.01103891334887315,
+            0.008096915671982816,
+            0.022587659771132487,
+            -0.0013527917944322358,
+            -0.0018085171207746743,
+            0.0038503794877980586,
+            -0.011870484733648887,
+            -0.007881489342351292,
+            0.006462772034381968,
+            0.0315696514747912,
+            0.01326417653779577,
+            -0.020400369539140532,
+            -0.015223808971473513,
+            0.020406862349006738,
+            -0.023142176712111575,
+            -0.006254959806032159,
+            0.009039506093607052,
+            0.0009087520492922078,
+            -0.022766046838415165,
+            0.0009662037093312435,
+            0.004701966191595815,
+            0.005946458676746661,
+            -0.006358986495019584,
+            0.03237532341275415,
+            0.001075992257265679,
+            0.015892020271402903,
+            -0.017189453434945788,
+            0.0017514632185789299,
+            -0.014189278671121677,
+            -0.014265447388465805,
+            -0.029809460132405256,
+            0.0005557491671071693,
+            -0.0030324198417091704,
+            0.011038984948790744,
+            0.024808067315720335,
+            0.01145103533194245,
+            0.014432246996007246,
+            0.010552852947578788,
+            0.01513689173494211,
+            -0.013653722180565311,
+            -0.029009757850432732,
+            -0.005171522291984226,
+            -0.018001048743500243,
+            -0.02297482628212251,
+            0.003962613844063587,
+            0.013645125285548414,
+            0.006278662690346082,
+            0.023786381386575035,
+            0.0021738516691483163,
+            0.0009580591750940011,
+            -2.690805459518147e-05,
+            0.008894091070322921,
+            -0.004952203441302471,
+            -0.006553986197076102,
+            0.003570189040872711,
+            -0.000652714137223501,
+            0.021228360881461853,
+            -0.005758946428212041,
+            7.897153237591187e-05,
+            0.014242175492137011,
+            -0.005622256348934008,
+            -0.0035009192529791146,
+            -0.017339926143652377,
+            0.006248842389703505,
+            -0.01622011586976866,
+            0.025783086900871728,
+            0.018450783573486916,
+            -0.009109295293546267,
+            -0.013469521573468915,
+            -0.011259555201313116,
+            0.02773097452736661,
+            0.004631282554715908,
+            0.010787055220617197,
+            0.018228322680582464,
+            0.01461254259249465,
+            0.01784830193964943,
+            0.004664725683522323,
+            -0.0034339781146649384,
+            -0.0007692550982545399,
+            0.021176646860433974,
+            -0.014700178590249292,
+            -0.002701208084390198,
+            -0.008842788249773623,
+            0.0021799005046816444,
+            -0.008431299000066363,
+            0.006086447848362669,
+            -0.010793016137040172,
+            -0.015251381840165954,
+            0.0189503457652375,
+            -0.02660767353735015,
+            0.016423753600066336,
+            -0.00593016170680532,
+            -0.00482891500342298,
+            -0.027459214027897378,
+            0.010728039528784636,
+            0.0027866316191608297,
+            0.0022519974195000687,
+            0.010770472550461752,
+            -0.021427634123401408,
+            -0.01867717541189363,
+            0.008532444554010673,
+            -0.011457117350255678,
+            -0.007152301670244725,
+            -0.00348619742388509,
+            0.0154715607296319,
+            -0.00017898955187849443,
+            -0.009925365923862246,
+            0.02085093731060963,
+            -0.007175309371250024,
+            0.01223979370120031,
+            0.005256127464980655,
+            -0.008756077893734508,
+            -0.0028438320920811362,
+            0.021022708898898203,
+            0.027557382862284808,
+            -0.008342206702778956,
+            0.014526066580760486,
+            -0.011883797998224506,
+            -0.01088335352319397,
+            -0.03479444009382274,
+            0.030104838505937447,
+            -0.013305732257226255,
+            0.026773747582411633,
+            -0.0006968265979630097,
+            -0.017037729111584094,
+            -0.016206207453228583,
+            0.01625225876515654,
+            -0.0024290584023212254,
+            -0.009366287391027166,
+            0.01465885301870365,
+            -0.02428628495473643,
+            0.0058336571592899735,
+            -0.0006627935593670669,
+            -0.011531762838191602,
+            0.013275508677482622,
+            0.0019223252930710324,
+            0.010738622785461513,
+            -0.02030405805119001,
+            0.0009661595529028194,
+            0.009263716373456188,
+            -0.016791855691574455,
+            0.006057830565306032,
+            -0.03730952786847912,
+            0.0029341721588188943,
+            -0.014833895684348656,
+            0.005725615593914644,
+            0.001498755253963699,
+            -0.01820708352937343,
+            -0.03652692232407375,
+            0.00043665701245599497,
+            0.01913883585802903,
+            -0.007975988799790905,
+            0.020538595326175857,
+            0.008243569241768276,
+            0.008421404577931517,
+            0.002314738067443065,
+            0.0016702403734608475,
+            0.008279538988821465,
+            0.03679456861213399,
+            -0.01775927634148114,
+            -0.004836606118958108,
+            -0.0012151329859941935,
+            0.010644195708319258,
+            -0.02028470571116305,
+            -0.009287515726906219,
+            -0.0036506912311261536,
+            -0.013134467570201307,
+            0.030615862647585222,
+            -0.020265555441523482,
+            0.025634449009756734,
+            -0.009300770899023007,
+            0.009274072905190077,
+            -0.02097685009125915,
+            -0.009797008690754956,
+            -0.010661540100131183,
+            0.0019532291898233895,
+            0.013465005282849381,
+            -0.009151430285942593,
+            0.0071467934925125145,
+            -0.006655608635025676,
+            0.01020327561163374,
+            -0.001472597695707201,
+            -0.02443508800166349,
+            -0.0024998811551575563,
+            -0.01107321258077183,
+            0.02456164421154801,
+            -0.0019619696941303388,
+            0.0005043179699158694,
+            0.010951580163871948,
+            -0.00853593956432799,
+            0.023927336608555987,
+            -0.008855702280766362,
+            0.0010093679746665581,
+            -0.004863269000020324,
+            -0.004005691415117686,
+            0.001698212044518236,
+            0.01822024022650613,
+            -0.005707404715849356,
+            0.03722587302228139,
+            -0.01708925706528118,
+            -0.0065500772743641925,
+            -0.00015538686795361824,
+            0.010852925330001283,
+            0.010035359503147549,
+            0.008754719437760327,
+            0.004067820167375348,
+            -0.014861417626433965,
+            0.009346669946626484,
+            -0.020136029447262147,
+            0.007501669351583119,
+            0.023178490169347605,
+            -0.006840359541885465,
+            0.0006676450965357345,
+            -0.01283320739092204,
+            -0.015610538827742276,
+            0.006308543164022682,
+            0.002329420005586303,
+            -0.002295739637504597,
+            -0.02181189272979859,
+            0.006298804075737111,
+            -0.014398470126723037,
+            0.004007419942728446,
+            -0.0007497656993578918,
+            0.013863867106289847,
+            0.016451166273826513,
+            -0.006740209707090658,
+            0.039292260347522444,
+            0.01871723939316731,
+            0.009109005310505462,
+            0.005674782595856756,
+            -0.040588444759736254,
+            -0.008993850998154852,
+            -0.0010197185016393919,
+            0.004866150984083949,
+            -0.01141354216871663,
+            -0.018944881712855702,
+            -0.002646110151132781,
+            0.000308585036997592,
+            -0.010681954707313127,
+            -0.009091585604191968,
+            0.01751599002710486,
+            -0.006034424764302979,
+            0.00872057856926618,
+            0.004146961781109887,
+            0.023758065635807138,
+            -0.009727802672285544,
+            0.01176257119851645,
+            -0.0071822083765114165,
+            -0.005456197751568526,
+            0.0006726765703478756,
+            0.011011628907288813,
+            -0.008031792262886802,
+            0.009763340867550022,
+            -0.004279891138185881,
+            0.008355720233269893,
+            -0.008552596384067825,
+            -0.023952129458400563,
+            -0.014972827087541732,
+            0.014347535404251018,
+            0.011651260496709961,
+            0.008137738462704715,
+            -0.013624435569539285,
+            0.0031728721674035352,
+            0.00675830277744976,
+            -0.009184784787581942,
+            0.009161771311910335,
+            0.017733082848622708,
+            -0.001411542568472906,
+            -0.0017830167499636113,
+            -0.0016130003349297027,
+            -0.012173528776899308,
+            0.0011196437753239162,
+            0.02284887358189995,
+            -0.003054479643937998,
+            0.014614283913976716,
+            0.010689486748036806,
+            0.009643292262102645,
+            -0.00810930837603355,
+            0.001915955511174678,
+            0.02341863048264366,
+            0.015022291783105098,
+            -0.004625715884342598,
+            -0.011422973894623295,
+            0.004259757717597968,
+            -0.010332744704689279,
+            0.007373328473086509,
+            -0.02117940113144938,
+            -0.004672698999328004,
+            0.008896311380763048,
+            -0.007028862917603513,
+            -0.01732432285099198,
+            -0.028026173555055017,
+            -0.016765805892242715,
+            0.019819329380181,
+            0.0027213760678977965,
+            -0.013291251876048186,
+            -0.0032919084721662363,
+            0.0031214883337300926,
+            0.002553005168048115,
+            -0.0071104806975682885,
+            0.005310583052470149,
+            0.007359183453835734,
+            -0.008674133980486269,
+            0.017665958608281126,
+            -0.000521269431423141,
+            -0.0008020251454616109,
+            -0.0005442374261144659,
+            0.0006127687598711624,
+            -0.016387752806126442,
+            0.01666161912459431,
+            -0.010949228376419892,
+            0.015198110119816624,
+            -0.008966726991404666,
+            0.008600499511690635,
+            -0.007617043351740469,
+            0.009458641139070436,
+            0.011597957400445055,
+            0.012755037944648426,
+            -0.004403680820063604,
+            0.008519241473418196,
+            0.004807816191718704,
+            -0.008147125240985166,
+            0.007017601015398136,
+            0.01296082929962557,
+            0.00582353932160707,
+            0.0013634237970372698,
+            -0.0008246894322793264,
+            0.0022035670958705576,
+            -0.002480976219285683,
+            0.0056293125704658135,
+            0.012714850523393719,
+            -0.002497639449055006,
+            -0.007135683201188999,
+            0.014988249771676699,
+            0.014955895630183352,
+            -0.022994948923833993,
+            0.007121634434082677,
+            0.005835971825639792,
+            -0.0030507120821557008,
+            0.002145266953387315,
+            -0.027334525318320587,
+            -0.020359474070022213,
+            0.01901995970350808,
+            -0.015260454474605018,
+            0.02850460517419317,
+            -0.011624138792780823,
+            -0.02284242513728605,
+            -0.01277489645734181,
+            -0.004209529633534193,
+            -0.022684500849916897,
+            -0.004894414098943742,
+            0.01919165657366256,
+            0.013401560972810082,
+            -0.004313549020265677,
+            -0.019089369171102138,
+            0.03863248408555474,
+            0.010497951840980463,
+            0.03508157919687289,
+            -0.008597970944278045,
+            0.0023354831464448313,
+            -0.01563401808449416,
+            0.015551176593436512,
+            0.001075882802133782,
+            -0.011575692834796734,
+            0.01505716108583101,
+            0.031836226915668404,
+            0.0034740889237207494,
+            -0.00617206855371193,
+            0.0018420207148235309,
+            0.00418433325800744,
+            -0.0008167766650900381,
+            0.011160989178055499,
+            -0.018497472318034093,
+            -0.011951262348399922,
+            0.0013093376768780122,
+            0.00216347903176653,
+            0.0059233460419514045,
+            -0.001778091863744448,
+            -0.013317639704511549,
+            0.011433308892643204,
+            0.011596810919469926,
+            -0.01518143105525228,
+            -0.007497373860338335,
+            0.0050957665991362405,
+            -0.018974405317819656,
+            0.01872472902494418,
+            0.011859522398789263,
+            0.000792413325332956,
+            -0.0006503947129906383,
+            0.00808635755624928,
+            0.003749813548995223,
+            0.0033657799335872368,
+            -0.007862215303617144,
+            -0.0028014669075116876,
+            0.0009892310856576799,
+            3.834379772674841e-06,
+            0.015402211364447726,
+            -0.015264893397161102,
+            0.02488855366696587,
+            -0.0025643639525691083,
+            0.00953993868921819,
+            -0.017460590918773188,
+            -0.0017911712931080155,
+            -0.00402374787529868,
+            0.01838051670627292,
+            0.006753072393532114,
+            -0.010130676216186492,
+            0.004921510943896829,
+            0.007975201081296845,
+            -0.012589786869771252,
+            0.00018319054705558588,
+            -0.0022606180909706688,
+            -0.014791528072476771,
+            -0.0031171076370692706,
+            0.017873825561816544,
+            0.007593688317551089,
+            0.0037375070591255375,
+            -0.015863143964928172,
+            0.007192324815779268,
+            0.007522990667369163,
+            0.024504875417877527,
+            -0.026309816223960274,
+            0.024779042752840836,
+            -0.022104087334088543,
+            0.02165781034613893,
+            0.0319051741666687,
+            0.007538146369110045,
+            -0.01690181186708103,
+            -0.006595947249169521,
+            0.006373991305923002,
+            -0.023410814842488123,
+            -0.005494017994983188,
+            0.016903736810723716,
+            -0.01792400152372258,
+            0.000330865061689199,
+            -0.014660413003063089,
+            0.0040825880154611895,
+            0.008248037865838212,
+            0.011540333451799686,
+            -0.0033578830043738085,
+            0.018327525495676525,
+            0.020171745352159914,
+            -0.01138759854371235,
+            -0.0004221149053107702,
+            -0.022321409200941777,
+            0.005046865873981027,
+            0.022509643273449936,
+            0.001111272240506322,
+            -0.009540467231190513,
+            0.018376261423745193,
+            0.008480532791922674,
+            -0.020801891594934313,
+            0.008436374227753589,
+            -0.036175982655998785,
+            -0.010454118523313122,
+            -0.009328507118928412,
+            -0.001218633166683384,
+            -0.004142773602181115,
+            -0.0002494371580772723,
+            -0.011365548842513265,
+            -7.699548642315061e-05,
+            -0.006889339420877534,
+            -0.0037173070609413474,
+            0.014953978004522174,
+            0.011678957032329096,
+            -0.003501047405385249,
+            -0.0073984932581048915,
+            0.008493128567102267,
+            0.006163937693564917,
+            -0.017034696006318337,
+            -0.013268418502042576,
+            0.0037122128689059085,
+            -0.020067656401141467,
+            0.005148329575092478,
+            0.00459729178556505,
+            0.014031669598917171,
+            -0.01849235806598133,
+            -0.009298127665365717,
+            0.019716512208448042,
+            0.0094466405158553,
+            -0.0008988916054855498,
+            0.004264260983102956,
+            -0.00032312548009733906,
+            0.002279987803645912,
+            -0.0008902345374346296,
+            -6.145528806973222e-06,
+            0.003602229465148649,
+            0.012503366799381277,
+            0.007759393855482671,
+            0.03694402922447051,
+            -0.016692148708641812,
+            0.005873765863350649,
+            -0.014187609702966002,
+            -0.0037592517335595654,
+            -0.00749939010396784,
+            -0.0004308157309481927,
+            -0.0022282234382577843,
+            0.0021999996319642854,
+            0.0052176773733353925,
+            0.009353171029508977,
+            0.002155405440765206,
+            -0.00823565584692709,
+            0.009691447250255694,
+            -0.015228173095533029,
+            0.003853858061067531,
+            -0.008207910291425473,
+            0.01017972550354623,
+            -0.009695280827515048,
+            0.0028705587799331246,
+            0.03849358164875786,
+            0.01130021300937487,
+            -0.004934175853966098,
+            -0.014300147534361459,
+            -0.0007796051618993811,
+            -0.01737877501558196,
+            -0.008579119859817895,
+            -0.005434792488602325,
+            -0.02617338653902575,
+            -0.00838916510137031,
+            0.015344260464413648,
+            0.011202567142310492,
+            0.014124853863170132,
+            -0.023894182185855824,
+            0.008897943489322673,
+            -0.03001670823127986,
+            -0.011951507895140876,
+            -0.001530967140156163,
+            -0.015467812535428609,
+            0.0010110999991167703,
+            0.023623196504289838,
+            -0.007762735863648054,
+            0.012654652822750325,
+            0.012867958170258448,
+            0.018777522196571934,
+            0.006390986057157016,
+            0.00038045736974183937,
+            -0.009444911783760569,
+            0.009534310165417939,
+            0.008806031651319704,
+            0.0080954543713255,
+            0.013161833638467553,
+            -0.0022840529693446553,
+            -0.018060162439023572,
+            0.005469288590000856,
+            0.014354475778235131,
+            -0.0033504276914600585,
+            -0.001615191819057537,
+            -0.011088493180661976,
+            0.016039040565630432,
+            -0.023559046333872156,
+            -0.009482398794557168,
+            0.009918576960319548,
+            -0.0027308309155029842,
+            0.009212063547988036,
+            0.0160181200081126,
+            -0.01881373001297218,
+            -0.0024168618343039703,
+            0.016244889977796682,
+            0.0024104546308670025,
+            -0.00022606425578303413,
+            0.016792244238218245,
+            -0.0007616857360728623,
+            -0.004279120886828465,
+            0.00997764266310123,
+            -0.0075631917138318655,
+            7.3878137053128405e-06,
+            -0.002416364915008295,
+            -0.0028253498168087964,
+            0.02054128727706496,
+            -0.021281801678871736,
+            -0.005379259148621041,
+            0.02257306103533458,
+            0.015687105622685975,
+            -0.007843745111084413,
+            6.258498042855072e-05,
+            -0.0219506172079994,
+            0.0019925886885244454,
+            -0.007768133999661569,
+            0.016524850592507592,
+            -0.007957949830060354,
+            0.0030085044673901665,
+            -0.01361024620290394,
+            0.011563781350927133,
+            -0.01277407169098418,
+            0.00873716215529134,
+            -0.002986943948333218,
+            -0.017567646684567492,
+            -0.0053109390854215125,
+            0.004561236136619266,
+            0.0014045175130801308,
+            0.044958041347040185,
+            -0.005539136232703934,
+            0.003729517522976541,
+            0.018086335927383804,
+            -0.014199453349727933,
+            0.02079719360289355,
+            0.016663293302592568,
+            0.00991445855219406,
+            -0.00033967370741635794,
+            -0.0018726748849858836,
+            0.023373354806639515,
+            0.013258226441642464,
+            0.00861986998150065,
+            -0.0014340218811281975,
+            -0.008516541227326329,
+            0.018239230461345617,
+            0.005458629943667832,
+            -0.012421896028975013,
+            0.012732721349758832,
+            0.013559877132508506,
+            -0.013720939472355347,
+            -0.011545210834395917,
+            0.01616358223872008,
+            -0.019904594319155373,
+            -0.018260016571763644,
+            -0.0036124537591368864,
+            -0.01614510591132948,
+            -0.03864699384205461,
+            0.01619401261888991,
+            0.018351607973242735,
+            0.006666313000123845,
+            0.013647410298896132,
+            0.002597680061427196,
+            0.004545530267244157,
+            -0.018531082430861854,
+            -0.0077618625909995275,
+            -0.008085144524135461,
+            0.003744735923004248,
+            0.011059457364156674,
+            -0.011335432904114519,
+            -0.009514379703581616,
+            -0.00034500702695418285,
+            -0.007433132060989644,
+            0.01971420270994951,
+            0.0020753610290475642,
+            0.0034506706065161024,
+            -0.00628437298309269,
+            0.017966485854173167,
+            0.026789678964152908,
+            -0.007532446825077476,
+            0.021886182869092073,
+            -0.002864230253056434,
+            -0.0015870360120997492,
+            -0.014393519359298496,
+            -0.0014439412842964876,
+            -0.011527182213472351,
+            0.0005453493437517964,
+            -0.0006791481533636673,
+            0.014093767997869645,
+            -0.03288824642147967,
+            -0.003957301296796813,
+            0.014444592778851293,
+            0.007538052312876599,
+            -0.005057755472988024,
+            0.011357919882949903,
+            -0.002964451028327082,
+            0.016977317604413924,
+            0.0075201055190542146,
+            0.023391646079994784,
+            -0.016588823248397513,
+            -0.004504858318770565,
+            0.006660877064954993,
+            0.0034631255931793638,
+            0.0014484203909961972,
+            0.01912983041187014,
+            0.023439941258581887,
+            -0.014715886385460541,
+            -0.0013596123366001458,
+            -0.0029020372473879614,
+            -0.021414221343490594,
+            0.014976806123253727,
+            0.0074729481304182865,
+            0.010967351226114886,
+            0.009806663105020302,
+            0.015465076881663691,
+            -0.02109587527354609,
+            0.0011671121739822202,
+            0.020567458365051382,
+            -0.008618333549105003,
+            -0.018815706906171112,
+            -0.009804829158883123,
+            -0.00966828714719191,
+            -0.006078893678150302,
+            -0.00803783313451146,
+            0.005523023167028105,
+            0.0022647506733877573,
+            0.01652997604669998,
+            -0.008204200434441975,
+            -0.008404609147866811,
+            0.008489247125986078,
+            0.007546785673285454,
+            -0.0033093451283570316,
+            0.010614050963656496,
+            0.0030633179859393587,
+            -0.006940540318011839,
+            0.00894884343338777,
+            -0.002787816848430143,
+            0.014347598920245487,
+            -0.02441769157218451,
+            0.004717742147185283,
+            -0.012096876825337124,
+            0.006405151160675699,
+            0.0041568790527989905,
+            0.0016702066682613428,
+            -0.0041159211459004,
+            0.0035217362856754353,
+            0.0008893479210743541,
+            -0.07433312151926533,
+            -0.012629681414004524,
+            -0.0035797012285864486,
+            0.033657838713146884,
+            -0.019695806809053914,
+            0.004477927056336216,
+            0.011966065644382396,
+            0.0096717896523898,
+            0.003070713232538593,
+            0.006136726101157844,
+            0.006112385569942848,
+            -0.0245258242481677,
+            -0.00015284418977092615,
+            0.004821040694679944,
+            0.0020347004045943454,
+            -0.007532408630122127,
+            0.00750565079228405,
+            0.015327710245879296,
+            0.020989926895388523,
+            -0.010970305087289488,
+            0.017227663358885748,
+            -0.003352718496981928,
+            -0.006068892126250978,
+            0.0014285750322333686,
+            -0.010153799722483554,
+            -0.017307415843726394,
+            -0.006495008086641125,
+            0.0068983699213297075,
+            -0.001982568363667046,
+            0.0044032687939897855,
+            0.003333596495330213,
+            -0.0032171586993948696,
+            0.0072365127663633925,
+            0.013920326463272646,
+            0.006415916101287133,
+            -0.013675562796300214,
+            0.003003782758807665,
+            -0.02500375035312058,
+            0.011945988879567646,
+            -0.012708975181025296,
+            -0.0009488161508767279,
+            -0.0047093877027523014,
+            -0.0035750724825159503,
+            -0.002213435677949806,
+            0.02043567538125188,
+            -0.014304009033474186,
+            -0.01802899775650995,
+            0.008187227768196673,
+            -0.011217013195839762,
+            0.008300740149622635,
+            -0.03382831122769974,
+            0.012386335491333264,
+            -0.0276029934382802,
+            -0.0002243752790856292,
+            -0.004521387639042107,
+            -0.0010631721066129095,
+            -0.009491184157278669,
+            -0.03274559774619223,
+            0.015845705808609598,
+            -0.007204751554079551,
+            -0.020338337341085228,
+            0.0007143424225187802,
+            -0.021267094083674304,
+            0.014150458196361147,
+            -0.01851165553224022,
+            0.0033450353123933058,
+            -0.018775439467431685,
+            0.002782289821140401,
+            -0.0018711350216089556,
+            0.010502195116579441,
+            -0.00557814442809019,
+            -0.0007130962242755068,
+            -0.013072988887382269,
+            -0.002121841212381393,
+            -0.006653444103331213,
+            0.009394907690516121,
+            0.0028678074167728367,
+            -0.009601577734902932,
+            0.011153571864462314,
+            0.013046116892519414,
+            0.019812860521973938,
+            -0.00539047787915588,
+            -0.00555139958411929,
+            -0.011553689090179406,
+            0.009852063270798534,
+            -0.016326212706634428,
+            -0.00827402913960903,
+            0.008982303148984887,
+            0.015216947130173639,
+            0.01026889105233137,
+            0.003352028462252994,
+            0.003150801750574548,
+            0.002531620731961765,
+            0.026744691368233237,
+            -0.0015120532859309638,
+            0.0008324701960729416,
+            -0.014508988496009931,
+            0.022127525293249926,
+            0.02772430884375744,
+            -0.0012206102581726523,
+            -0.00018859639594030037,
+            0.017437360169132967,
+            -0.004697052522250805,
+            -0.0038111184371524645,
+            0.024015246959315593,
+            -0.038396108183338655,
+            -0.008416825232533771,
+            -0.006788816882617399,
+            -0.006043428198944214,
+            -0.009587366247924467,
+            0.007294190273465812,
+            0.00041966080769004275,
+            0.0024709692753547823,
+            -0.0005250904505954556,
+            0.03452858797518386,
+            0.003403515654095309,
+            -0.0003392060101496309,
+            -0.0032980864446307116,
+            0.0019125121221204029,
+            0.0032987517853214346,
+            -0.009116761755779728,
+            -0.038030085122772624,
+            0.006006334000396066,
+            0.007932785488877407,
+            0.00975009749364395,
+            -0.008120121513849434,
+            -0.008864461019882246,
+            0.004909348738798099,
+            -0.02273129283865203,
+            -0.01372636307167243,
+            0.02625229144362647,
+            -0.008523028986127237,
+            0.029005877324974557,
+            -0.022435059061182174,
+            -0.013135836801818153,
+            -0.004126433322528265,
+            0.010283975013795537,
+            -0.019732856462444506,
+            0.004343860338939202,
+            -0.004022387065239562,
+            0.023733721026005954,
+            0.0077578144626961245,
+            -0.008371631446146718,
+            0.012386735614153616,
+            0.0064789380436815714,
+            -0.0026638642827013837,
+            0.00136333052963848,
+            -0.007209517067748103,
+            0.016916505654836485,
+            -0.011819784026452944,
+            -0.005833909403097991,
+            0.008977527508585844,
+            -0.0024556434204852946,
+            0.02509293948918158,
+            -0.0237762245071987,
+            0.005288171058938098,
+            -0.00019047961064753207,
+            0.01822542471604037,
+            0.008127347078975925,
+            0.008361588667386882,
+            -0.011835407590060994,
+            0.010004482616697591,
+            -0.03378107067882622,
+            -0.015433906677849963,
+            -0.007702910250795191,
+            -0.0013877702264578865,
+            -0.002059152251387601,
+            -0.015745218548968338,
+            -0.026393221735795527,
+            0.006294583740809379,
+            0.016267525575472445,
+            0.023291528020980265,
+            -0.004683197930892445,
+            0.012912830736650997,
+            -0.002831583991769044,
+            -0.008904271394710672,
+            -0.011441882849296418,
+            -0.012202629605208118,
+            -0.012470773094383122,
+            -0.014560694782340029,
+            -0.017045701603711442,
+            0.012139665577738337,
+            -0.020079061996873537,
+            0.013911518328924777,
+            0.0347210775522539,
+            -0.017176114991595492,
+            -0.0018941772168839354,
+            0.019702722555622765,
+            -0.004221469612316137,
+            0.01212088353057276,
+            0.0063707489618353725,
+            0.015298014532351866,
+            0.0038643174573634725,
+            -0.012181479717826009,
+            -0.007374005740809251,
+            0.019411457664902376,
+            -0.004902591844710158,
+            0.014591446725717338,
+            0.015069599884557399,
+            -0.008675672164385891,
+            0.002544198867387345,
+            0.01951865904496276,
+            -0.01762101777798757,
+            0.01665654707715257,
+            -0.00458402229480694,
+            -0.009936206685538294,
+            -0.013345740581684638,
+            -0.005336416724426228,
+            -0.019469457091169776,
+            0.0061608454109892425,
+            0.018931676738452464,
+            -0.00533013594306834,
+            0.009310871143727683,
+            -0.0037670793158045,
+            -0.005045151384034014,
+            0.0002041909886640482,
+            -0.04152751596227068,
+            -0.008340911336217609,
+            -0.0006651584370005978,
+            -0.02505748815737022,
+            0.025954102004571416,
+            0.007205602377144382,
+            0.022602344303400163,
+            -0.017168610245491406,
+            0.004459735690564842,
+            -0.005882024209102714,
+            0.013217517324383834,
+            0.02255689669623969,
+            0.0017984057469785134,
+            0.005699128497215882,
+            -0.02353849721739205,
+            -0.0034711576718386185,
+            0.015924913374553738,
+            -0.003604478141616307,
+            -0.0034993903155063817,
+            0.012592308877056896,
+            0.012864938348003884,
+            0.01567418642605425,
+            0.008801753550506258,
+            0.015606443428689292,
+            0.010589933994371628,
+            0.008928721556432368,
+            -0.005805129999077126,
+            -0.001025103064751441,
+            -0.015907755521365925,
+            0.0045874330999415985,
+            0.031701049450012876,
+            0.012594483199773746,
+            0.016783858873835704,
+            -0.01778536268185936,
+            0.00789523134933153,
+            -0.0012706411671644011,
+            0.008325108613530198,
+            -0.001677173669891956,
+            0.029540136288412976,
+            0.011739999605699826,
+            0.015670966746894652,
+            0.008732334090105053,
+            -0.016237731209426752,
+            0.045334690225148,
+            -0.016911928942220152,
+            -0.0021139311364009113,
+            -0.01058970056572469,
+            -0.021705338531028618,
+            0.007324792557651635,
+            -0.006556582104686972,
+            0.0018243188317438144,
+            0.010780292738415671,
+            -0.006471850255151019,
+            0.0006997395548596146,
+            0.003907433192252573,
+            -0.023129313269763224,
+            -0.002550875642832235,
+            -0.02237917712764273,
+            -0.003107775639788617,
+            -0.02621608419628401,
+            -0.00012482032027919036,
+            -0.016207377503696517,
+            -0.0038712739995960753,
+            -0.023315508569288476,
+            -0.014963211090562272,
+            -0.01633110043829169,
+            -0.004241081909202262,
+            -0.008295402942465353,
+            0.004976396647108597,
+            -0.007402107755970892,
+            -0.020321297548927468,
+            0.0020986169430507757,
+            0.016803273183070004,
+            0.01434729320047051,
+            0.018612970083178984,
+            0.00603868809307069,
+            0.014278001198851581,
+            -0.0026350854954985117,
+            0.017171016209107656,
+            -0.006968968261909459,
+            0.006881847166565678,
+            0.011454001418121107,
+            0.02140935294963631,
+            0.011121833270235448,
+            -0.012801756637470525,
+            0.0010870041670931706,
+            -0.009655388034828094,
+            0.002094601818569396,
+            -0.00828486034340067,
+            -0.009190975158350585,
+            0.0017762581946855365,
+            -0.007261996868906865,
+            0.004674405202069334,
+            0.0030526253833799426,
+            -0.0014082232541965316,
+            0.010419302200840906,
+            -0.0024200053377751805,
+            -0.012208685875688886,
+            -0.0033037330470612695,
+            -0.017975940046938498,
+            -0.006958042903103732,
+            -0.0014496522238488239,
+            -0.011711502944602638,
+            -0.01016130418405875,
+            0.003321436189059608,
+            -0.00538282506552848,
+            -0.009353581612407435,
+            0.013884147453136417,
+            0.0001309738696853529,
+            0.0005878330075918333,
+            0.023495280602974504,
+            0.006035028203642483,
+            0.008087594627921345,
+            0.013144288945385904,
+            -0.01023995000994454,
+            -0.022574958230635715,
+            -0.0243435727055662,
+            0.007867022215096418,
+            -0.01573478573423379,
+            0.02324699908478915,
+            -0.007929809922043611,
+            -0.00561630876431978,
+            0.0020982723335057625,
+            -0.013699317926847062,
+            0.008578431496569186,
+            -0.011271381061801356,
+            0.0014450645788554738,
+            -0.019581451799757368,
+            0.018059311800267413,
+            -0.010917911869373693,
+            -0.004302687883018328,
+            0.009466103226841532,
+            0.034049086327052606,
+            -0.0005689605174080737,
+            -0.007410970644432269,
+            -0.006594150877740108,
+            -0.00022810271313786974,
+            0.018766765773409123,
+            -0.0029041815756733333,
+            0.005758781908059541,
+            -0.01563519107805783,
+            0.006571628287171105,
+            -0.006132372252825422,
+            0.0012423261344297225,
+            0.01513074831647933,
+            -0.026306320707079913,
+            -0.015491710256868008,
+            -0.007857352164317122,
+            -0.024893430640551818,
+            0.002253574511878983,
+            0.017579477884206534,
+            0.020505606775521504,
+            -0.020324505029926956,
+            -0.02513332780748072,
+            0.005150966733930881,
+            -0.026580994694344065,
+            0.0037390577082486594,
+            0.1241031878517593,
+            -0.002138404898309707,
+            0.002720723037908521,
+            -0.011425036886011179,
+            0.0022399461009316216,
+            0.005389407121650262,
+            -0.019552543955456567,
+            0.05950746473186985,
+            -0.00638023587296763,
+            -0.004023798021748334,
+            0.002170545188776296,
+            0.001785855390676113,
+            -0.005831534307818096,
+            0.008643501093778434,
+            -0.017602787281866816,
+            -0.02184792441891879,
+            -0.0052215705748838905,
+            -0.021252169339074325,
+            -0.008509621422071373,
+            -0.013000137524907565,
+            0.002893880203681282,
+            -0.015821609145923173,
+            0.010397692426614709,
+            -0.01638916243796692,
+            -0.010283706937701179,
+            -0.01103621454761223,
+            -0.00373552043529251,
+            0.01563553822905027,
+            -0.03188533919653949,
+            -0.004873550301425776,
+            -0.005183590054152726,
+            0.018218971950313273,
+            0.006004328765748325,
+            0.004357734370247252,
+            -0.014230318971458851,
+            0.0064792070801456235,
+            0.0215589173714926,
+            0.0055272493764401305,
+            0.014643335833970546,
+            0.010618070641357003,
+            -0.0022177303813575846,
+            -0.0024402761885758007,
+            0.017541799738501007,
+            -0.005445774414228888,
+            -0.00023502936119626386,
+            -0.00537569046685066,
+            -0.005974703247446444,
+            -0.010712244203425036,
+            0.01843613166635215,
+            -0.009938564643456672,
+            -0.017388592763277214,
+            -0.0015797412875366156,
+            0.02028434083164951,
+            -0.012761534382003973,
+            0.0031906812114950982,
+            0.000788436210275669,
+            0.020589196433948156,
+            0.0007726855314648301,
+            -0.005481260996289263,
+            0.007165014826294827,
+            -0.02956334397512344,
+            0.004095715232997753,
+            0.0003568512047626187,
+            0.027290768799658912,
+            -0.0017631129386615335,
+            0.01552542586826711,
+            -0.020871286597113327,
+            0.004388125824324094,
+            -0.016926008631435403,
+            0.005099899758322398,
+            0.012560923968841436,
+            0.002781049890810361,
+            0.010889021753593997,
+            -0.008500358897706648,
+            -0.021353460542554444,
+            0.012663934009161902,
+            -0.018657094130860155,
+            -0.029156017587920958,
+            0.0018889388761887053,
+            -0.028158076315864772,
+            -0.003048854609750186,
+            0.006812072971079798,
+            -0.035217412591558014,
+            -0.02251675180883659,
+            0.007396524512996277,
+            -0.00995571600201846,
+            -0.011912753666073882,
+            0.010675240017836214,
+            -0.02645155743041798,
+            -0.018137648527851668,
+            0.0010666944556449141,
+            0.012588800077165337,
+            -0.004561825939541096,
+            0.007555558090958502,
+            -0.028558603070876605,
+            0.014816816621445352,
+            -0.01313764261744744,
+            0.016036065063831813,
+            -0.0037797584923144134,
+            0.022217516257304868,
+            0.011131597210722573,
+            0.008795376259031849,
+            -0.004950080558710245,
+            0.021409590863550987,
+            0.005574351961953293,
+            0.051358937509801736,
+            0.031972009383746405,
+            0.0019821003238211276,
+            -0.003251257665378234,
+            0.003313045083311649,
+            0.008294776423730887,
+            -0.00940948257229233,
+            0.005114631360856872,
+            0.028053227377164806,
+            0.016738013668710083,
+            -0.018755924318109216,
+            0.006606711923035118,
+            -0.02702692774986498,
+            0.017950884766781056,
+            -0.022054653845163,
+            0.0003755944703013671,
+            0.007766253634286795,
+            0.0016206993755471613,
+            0.010682347575070117,
+            0.0013842930079185415,
+            0.01705234255261425,
+            0.023975191051210525,
+            -0.002976402980885077,
+            0.017466027107477605,
+            0.012734909275073793,
+            0.018143161773162015,
+            -0.0038399373197265827,
+            0.006929440921265761,
+            -0.012003095293274786,
+            -0.0008883470597690258,
+            0.0020490527756307267,
+            -0.004231091383919565,
+            0.012128848448049833,
+            0.008340302519160299,
+            0.012989849936937843,
+            0.002974488532907779,
+            0.003198854381461613,
+            -0.005309753760578859,
+            -0.0017024982049936384,
+            -0.01763395035509857,
+            -0.01819680226734474,
+            0.028417724274372468,
+            -0.020217879775485576,
+            -0.050118184195774004,
+            0.0025509548991664425,
+            0.014185761006969567,
+            -0.0012021669304388173,
+            -0.0051260576846881515,
+            -0.0005774778872927327,
+            0.005718864494530703,
+            -0.016918749210038116,
+            -0.0015472638835434902,
+            0.008125845907627804,
+            0.020347547062899797,
+            -0.002036218493206914,
+            -0.01079739518866248,
+            0.0038112958713600906,
+            0.02169339303105242,
+            0.01667594322285835,
+            0.012338193092567552,
+            -0.009798409233733167,
+            0.0003198215558702285,
+            -1.6308740825163337e-05,
+            0.0076870827874737325,
+            0.004356644356656038,
+            0.003500023005601613,
+            -0.019968549869439228,
+            -0.008634740687755112,
+            0.012781565963545847,
+            -0.0025637758121512927,
+            0.010243567793632335,
+            -0.00042296044205132426,
+            -0.003069251553118065,
+            -0.009344199449486244,
+            -0.004694953215957636,
+            -0.009583993744231791,
+            0.013783878036521561,
+            -0.007705982042252044,
+            -0.01549567538665377,
+            0.00020666298495823114,
+            0.0005515775451348472,
+            0.001667367278489879,
+            -0.004518431526236256,
+            0.03302324454129141,
+            0.009531990824506103,
+            -0.0004937348108340977,
+            -0.004047613286672229,
+            -0.0327383112522095,
+            0.012456150771224677,
+            -0.009677533214778067,
+            0.0006455811519981003,
+            0.020012976827426605,
+            -0.00020277620481825933,
+            0.002168432791818704,
+            -0.015657551629104767,
+            0.0032924357559238502,
+            0.006161295735841883,
+            0.013814165416788555,
+            0.0005164426482972658,
+            0.01076512534105537,
+            -0.006498977875968572,
+            -0.010325144017898257,
+            0.004804158067334203,
+            0.0104064855510473,
+            -0.0012188100892986933,
+            0.005171197790296134
+          ],
+          "title": "Mixture-Of-Experts With Expert Choice Routing",
+          "keyphrases": [
+            "Mixture-Of-Experts",
+            "Expert Choice Routing",
+            "sparsely-activated models",
+            "training efficiency",
+            "downstream performance"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "14339b1f-53f8-479b-9da7-9430b174b9ee",
+        "properties": {
+          "page_content": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer\n\nNoam Shazeer1, Azalia Mirhoseini∗†1, Krzysztof Maziarz∗2, Andy Davis1, Quoc Le1, Geoffrey Hinton1and Jeff Dean1 1Google Brain, {noam,azalia,andydavis,qvl,geoffhinton,jeff}@google.com 2Jagiellonian University, Cracow, krzysztof.maziarz@student.uj.edu.pl\n\nAbstract\n\nThe capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost.\n\n1 Introduction And Related Work 1.1 Conditional Computation\n\nExploiting scale in both training data and model size has been central to the success of deep learning. When datasets are sufficiently large, increasing the capacity (number of parameters) of neural networks can give much better prediction accuracy. This has been shown in domains such as text (Sutskever et al., 2014; Bahdanau et al., 2014; Jozefowicz et al., 2016; Wu et al., 2016), images (Krizhevsky et al., 2012; Le et al., 2012), and audio (Hinton et al., 2012; Amodei et al., 2015). For typical deep learning models, where the entire model is activated for every example, this leads to a roughly quadratic blow-up in training costs, as both the model size and the number of training examples increase. Unfortunately, the advances in computing power and distributed computation fall short of meeting such demand.\n\nVarious forms of conditional computation have been proposed as a way to increase model capacity without a proportional increase in computational costs (Davis & Arel, 2013; Bengio et al., 2013; Eigen et al., 2013; Ludovic Denoyer, 2014; Cho & Bengio, 2014; Bengio et al., 2015; Almahairi et al., 2015). In these schemes, large parts of a network are active or inactive on a per-example basis. The gating decisions may be binary or sparse and continuous, stochastic or deterministic. Various forms of reinforcement learning and back-propagation are proposed for trarining the gating decisions.\n\nWhile these ideas are promising in theory, no work to date has yet demonstrated massive improvements in model capacity, training time, or model quality. We blame this on a combination of the following challenges:\n\nModern computing devices, especially GPUs, are much faster at arithmetic than at branching. Most of the works above recognize this and propose turning on/off large chunks of the network with each gating decision.\n\nLarge batch sizes are critical for performance, as they amortize the costs of parameter transfers and updates. Conditional computation reduces the batch sizes for the conditionally active chunks of the network.\n\nNetwork bandwidth can be a bottleneck. A cluster of GPUs may have computational power thousands of times greater than the aggregate inter-device network bandwidth. To be computationally efficient, the relative computational versus network demands of an algorithm must exceed this ratio. Embedding layers, which can be seen as a form of conditional computation, are handicapped by this very problem. Since the embeddings generally need to be sent across the network, the number of (example, parameter) interactions is limited by network bandwidth instead of computational capacity.\n\nDepending on the scheme, loss terms may be necessary to achieve the desired level of sparsity per-chunk and/or per example. Bengio et al. (2015) use three such terms. These issues can affect both model quality and load-balancing.\n\nModel capacity is most critical for very large data sets. The existing literature on conditional computation deals with relatively small image recognition data sets consisting of up to 600,000 images. It is hard to imagine that the labels of these images provide a sufficient signal to adequately train a model with millions, let alone billions of parameters. In this work, we for the first time address all of the above challenges and finally realize the promise of conditional computation. We obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets.\n\n1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1). All parts of the network are trained jointly by back-propagation.\n\nWhile the introduced technique is generic, in this paper we focus on language modeling and machine translation tasks, which are known to benefit from very large models. In particular, we apply a MoE convolutionally between stacked LSTM layers (Hochreiter & Schmidhuber, 1997), as in Figure 1.\n\nThe MoE is called once for each position in the text, selecting a potentially different combination of experts at each position. The different experts tend to become highly specialized based on syntax and semantics (see Appendix E Table 9). On both language modeling and machine translation benchmarks, we improve on best published results at a fraction of the computational cost.\n\n1.3 Related Work On Mixtures Of Experts\n\nSince its introduction more than two decades ago (Jacobs et al., 1991; Jordan & Jacobs, 1994), the mixture-of-experts approach has been the subject of much research. Different types of expert architectures hae been proposed such as SVMs (Collobert et al., 2002), Gaussian Processes (Tresp, 2001; Theis & Bethge, 2015; Deisenroth & Ng, 2015), Dirichlet Processes (Shahbaba & Neal, 2009), and deep networks. Other work has focused on different expert configurations such as a hierarchical structure (Yao et al., 2009), infinite numbers of experts (Rasmussen & Ghahramani, 2002), and adding experts sequentially (Aljundi et al., 2016). Garmash & Monz (2016) suggest an ensemble model in the format of mixture of experts for machine translation. The gating network is trained on a pre-trained ensemble NMT model. The works above concern top-level mixtures of experts. The mixture of experts is the whole model. Eigen et al. (2013) introduce the idea of using multiple MoEs with their own gating networks as parts of a deep model. It is intuitive that the latter approach is more powerful, since complex problems may contain many sub-problems each requiring different experts. They also allude in their conclusion to the potential to introduce sparsity, turning MoEs into a vehicle for computational computation.\n\nOur work builds on this use of MoEs as a general purpose neural network component. While Eigen et al. (2013) uses two stacked MoEs allowing for two sets of gating decisions, our convolutional application of the MoE allows for different gating decisions at each position in the text. We also realize sparse gating and demonstrate its use as a practical way to massively increase model capacity.\n\n2 The Structure Of The Mixture-Of-Experts Layer\n\nThe Mixture-of-Experts (MoE) layer consists of a set of n \"expert networks\" E1, · · · , En, and a \"gating network\" G whose output is a sparse n-dimensional vector. Figure 1 shows an overview of the MoE module. The experts are themselves neural networks, each with their own parameters.\n\nAlthough in principle we only require that the experts accept the same sized inputs and produce the same-sized outputs, in our initial investigations in this paper, we restrict ourselves to the case where the models are feed-forward networks with identical architectures, but with separate parameters.\n\nLet us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows:\n\n$$y=\\sum_{i=1}^{n}G(x){i}E{i}(x)$$ $$(\\mathbf{l})$$ G(x)iEi(x) (1) We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example. If the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network. In the following we focus on ordinary MoEs. We provide more details on hierarchical MoEs in Appendix B. Our implementation is related to other models of conditional computation. A MoE whose experts are simple weight matrices is similar to the parameterized weight matrix proposed in (Cho & Bengio, 2014). A MoE whose experts have one hidden layer is similar to the block-wise dropout described in (Bengio et al., 2015), where the dropped-out layer is sandwiched between fully-activated layers.\n\n2.1 Gating Network\n\nSoftmax Gating: A simple choice of non-sparse gating function (Jordan & Jacobs, 1994) is to multiply the input by a trainable weight matrix Wg and then apply the Sof tmax function.\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ $\\eqref{eq:walpha}$. Gσ(x) = Sof tmax(x · Wg) (2) Noisy Top-K Gating: We add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to −∞ (which causes the corresponding gate values to equal 0). The sparsity serves to save computation, as described above. While this form of sparsity creates some theoretically scary discontinuities in the output of gating function, we have not yet observed this to be a problem in practice. The noise term helps with load balancing, as will be discussed in Appendix A. The amount of noise per component is controlled by a second trainable weight matrix Wnoise.\n\n$$G(x)=S o f t m a x(K e e p T o p K(H(x),k))$$ $$(4)$$ $$(S)$$ G(x) = Sof tmax(KeepT opK(H(x), k)) (3) $$H(x){i}=(x\\cdot W{g}){i}+S t a n d a r d N o r m a l()\\cdot S o f t p l u s((x\\cdot W{n o i s e}){i})$$ $KeepTopK(v,k){i}=\\begin{cases}v_{i}&\\text{if}v_{i}\\text{is in the top}k\\text{elements of}v.\\ -\\infty&\\text{otherwise.}\\end{cases}$ Training the Gating Network We train the gating network by simple back-propagation, along with the rest of the model. If we choose k > 1, the gate values for the top k experts have nonzero derivatives with respect to the weights of the gating network. This type of occasionally-sensitive behavior is described in (Bengio et al., 2013) with respect to noisy rectifiers. Gradients also backpropagate through the gating network to its inputs. Our method differs here from (Bengio et al., 2015) who use boolean gates and a REINFORCE-style approach to train the gating network.\n\n3 Addressing Performance Challenges 3.1 The Shrinking Batch Problem\n\nOn modern CPUs and GPUs, large batch sizes are necessary for computational efficiency, so as to amortize the overhead of parameter loads and updates. If the gating network chooses k out of n experts for each example, then for a batch of b examples, each expert receives a much smaller batch of approximately kb n b examples. This causes a naive MoE implementation to become very inefficient as the number of experts increases. The solution to this shrinking batch problem is to make the original batch size as large as possible. However, batch size tends to be limited by the memory necessary to store activations between the forwards and backwards passes. We propose the following techniques for increasing the batch size: Mixing Data Parallelism and Model Parallelism: In a conventional distributed training setting, multiple copies of the model on different devices asynchronously process distinct batches of data, and parameters are synchronized through a set of parameter servers. In our technique, these different batches run synchronously so that they can be combined for the MoE layer. We distribute the standard layers of the model and the gating network according to conventional data-parallel schemes, but keep only one shared copy of each expert. Each expert in the MoE layer receives a combined batch consisting of the relevant examples from all of the data-parallel input batches. The same set of devices function as data-parallel replicas (for the standard layers and the gating networks) and as model-parallel shards (each hosting a subset of the experts). If the model is distributed over d devices, and each device processes a batch of size b, each expert receives a batch of approximately kbd nexamples. Thus, we achieve a factor of d improvement in expert batch size.\n\nIn the case of a hierarchical MoE (Section B), the primary gating network employs data parallelism, and the secondary MoEs employ model parallelism. Each secondary MoE resides on one device.\n\nThis technique allows us to increase the number of experts (and hence the number of parameters) by proportionally increasing the number of devices in the training cluster. The total batch size increases, keeping the batch size per expert constant. The memory and bandwidth requirements per device also remain constant, as do the step times, as does the amount of time necessary to process a number of training examples equal to the number of parameters in the model. It is our goal to train a trillionparameter model on a trillion-word corpus. We have not scaled our systems this far as of the writing of this paper, but it should be possible by adding more hardware.\n\nTaking Advantage of Convolutionality: In our language models, we apply the same MoE to each time step of the previous layer. If we wait for the previous layer to finish, we can apply the MoE to all the time steps together as one big batch. Doing so increases the size of the input batch to the MoE layer by a factor of the number of unrolled time steps.\n\nIncreasing Batch Size for a Recurrent MoE: We suspect that even more powerful models may involve applying a MoE recurrently. For example, the weight matrices of a LSTM or other RNN could be replaced by a MoE. Sadly, such models break the convolutional trick from the last paragraph, since the input to the MoE at one timestep depends on the output of the MoE at the previous timestep. Gruslys et al. (2016) describe a technique for drastically reducing the number of stored activations in an unrolled RNN, at the cost of recomputing forward activations. This would allow for a large increase in batch size.\n\n3.2 Network Bandwidth\n\nAnother major performance concern in distributed computing is network bandwidth. Since the experts are stationary (see above) and the number of gating parameters is small, most of the communication involves sending the inputs and outputs of the experts across the network. To maintain computational efficiency, the ratio of an expert's computation to the size of its input and output must exceed the ratio of computational to network capacity of the computing device. For GPUs, this may be thousands to one. In our experiments, we use experts with one hidden layer containing thousands of RELU-activated units. Since the weight matrices in the expert have sizes input_size×hidden_size and hidden_size × output_size, the ratio of computation to input and output is equal to the size of the hidden layer. Conveniently, we can increase computational efficiency simply by using a larger hidden layer, or more hidden layers.\n\n4 Balancing Expert Utilization\n\nWe have observed that the gating network tends to converge to a state where it always produces large weights for the same few experts. This imbalance is self-reinforcing, as the favored experts are trained more rapidly and thus are selected even more by the gating network. Eigen et al. (2013) describe the same phenomenon, and use a hard constraint at the beginning of training to avoid this local minimum. Bengio et al. (2015) include a soft constraint on the batch-wise average of each gate.1 We take a soft constraint approach. We define the importance of an expert relative to a batch of training examples to be the batchwise sum of the gate values for that expert. We define an additional loss Limportance, which is added to the overall loss function for the model. This loss is equal to the square of the coefficient of variation of the set of importance values, multiplied by a hand-tuned scaling factor wimportance. This additional loss encourages all experts to have equal importance.\n\n$$I m p o r t a n c e(X)=\\sum_{x\\in X}G(x)$$ $$L_{importance}(X)=w_{importance}\\cdot CV(Importance(X))^{2}\\tag{7}$$ $$(6)$$\n\nWhile this loss function can ensure equal importance, experts may still receive very different numbers of examples. For example, one expert may receive a few examples with large weights, and another may receive many examples with small weights. This can cause memory and performance problems on distributed hardware. To solve this problem, we introduce a second loss function, Lload , which ensures balanced loads. Appendix A contains the definition of this function, along with experimental results.\n\n5 Experiments 5.1 1 Billion Word Language Modeling Benchmark\n\nDataset: This dataset, introduced by (Chelba et al., 2013) consists of shuffled unique sentences from news articles, totaling approximately 829 million words, with a vocabulary of 793,471 words.\n\nPrevious State-of-the-Art: The best previously published results (Jozefowicz et al., 2016) use models consisting of one or more stacked Long Short-Term Memory (LSTM) layers (Hochreiter & Schmidhuber, 1997; Gers et al., 2000). The number of parameters in the LSTM layers of these models vary from 2 million to 151 million. Quality increases greatly with parameter count, as do computational costs. Results for these models form the top line of Figure 2-right. MoE Models: Our models consist of two stacked LSTM layers with a MoE layer between them (see Figure 1). We vary the sizes of the layers and the number of experts. For full details on model architecture, training regimen, additional baselines and results, see Appendix C.\n\nLow Computation, Varied Capacity: To investigate the effects of adding capacity, we trained\n\na series of MoE models all with roughly equal computational costs: about 8 million multiply-andadds per training example per timestep in the forwards pass, excluding the softmax layer. We call this metric (ops/timestep). We trained models with flat MoEs containing 4, 32, and 256 experts, and models with hierarchical MoEs containing 256, 1024, and 4096 experts. Each expert had about 1 million parameters. For all the MoE layers, 4 experts were active per input.\n\nThe results of these models are shown in Figure 2-left. The model with 4 always-active experts performed (unsurprisingly) similarly to the computationally-matched baseline models, while the largest of the models (4096 experts) achieved an impressive 24% lower perplexity on the test set. Figure 2: Model comparison on 1-Billion-Word Language-Modeling Benchmark. On the left, we plot test perplexity as a function of model capacity for models with similar computational budgets of approximately 8-million-ops-per-timestep. On the right, we plot test perplexity as a function of computational budget. The top line represents the LSTM models from (Jozefowicz et al., 2016).\n\nThe bottom line represents 4-billion parameter MoE models with different computational budgets.\n\nVaried Computation, High Capacity: In addition to the largest model from the previous section, we trained two more MoE models with similarly high capacity (4 billion parameters), but higher computation budgets. These models had larger LSTMs, and fewer but larger and experts. Details\n\nTest Test #Parameters ops/timestep Training TFLOPS Perplexity Perplexity excluding embedding Time /GPU 10 epochs 100 epochs and softmax layers 10 epochs Best Published Results 34.7 30.6 151 million 151 million 59 hours, 32 k40s 1.09 Low-Budget MoE Model 34.1 4303 million 8.9 million 15 hours, 16 k40s 0.74 Medium-Budget MoE Model 31.3 4313 million 33.8 million 17 hours, 32 k40s 1.22 High-Budget MoE Model 28.0 4371 million 142.7 million 47 hours, 32 k40s 1.56\n\nTable 1: Summary of high-capacity MoE-augmented models with varying computational budgets, vs. best previously published results (Jozefowicz et al., 2016). Details in Appendix C. can be found in Appendix C.2. Results of these three models form the bottom line of Figure 2-right.\n\nTable 1 compares the results of these models to the best previously-published result on this dataset .\n\nEven the fastest of these models beats the best published result (when controlling for the number of training epochs), despite requiring only 6% of the computation.\n\nComputational Efficiency: We trained our models using TensorFlow (Abadi et al., 2016) on clusters containing 16-32 Tesla K40 GPUs. For each of our models, we determine computational efficiency in TFLOPS/GPU by dividing the number of floating point operations required to process one training batch by the observed step time and the number of GPUs in the cluster. The operation counts used here are higher than the ones we report in our ops/timestep numbers in that we include the backwards pass, we include the importance-sampling-based training of the softmax layer, and we count a multiply-and-add as two separate operations. For all of our MoE models, the floating point operations involved in the experts represent between 37% and 46% of the total.\n\nFor our baseline models wtih no MoE, observed computational efficiency ranged from 1.07-1.29 TFLOPS/GPU. For our low-computation MoE models, computation efficiency ranged from 0.740.90 TFLOPS/GPU, except for the 4-expert model which did not make full use of the available parallelism. Our highest-computation MoE model was more efficient at 1.56 TFLOPS/GPU, likely due to the larger matrices. These numbers represent a significant fraction of the theoretical maximum of 4.29 TFLOPS/GPU claimed by NVIDIA. Detailed results are in Appendix C, Table 7.\n\n5.2 100 BILLION WORD GOOGLE NEWS CORPUS\n\nOn the 1-billion-word corpus, adding additional capacity seems to produce diminishing returns as the number of parameters in the MoE layer exceeds 1 billion, as can be seen in Figure 2-left. We hypothesized that for a larger training set, even higher capacities would produce significant quality improvements. We constructed a similar training set consisting of shuffled unique sentences from Google's internal news corpus, totalling roughly 100 billion words. Similarly to the previous section, we tested a series of models with similar computational costs of about 8 million ops/timestep. In addition to a baseline LSTM model, we trained models augmented with MoE layers containing 32, 256, 1024, 4096, 16384, 65536, and 131072 experts. This corresponds to up to 137 billion parameters in the MoE layer. Details on architecture, training, and results are given in Appendix D.\n\nResults: Figure 3 shows test perplexity as a function of capacity after training on 10 billion words (top line) and 100 billion words (bottom line). When training over the full 100 billion words, test perplexity improves significantly up to 65536 experts (68 billion parameters), dropping 39% lower than the computationally matched baseline, but degrades at 131072 experts, possibly a result of too much sparsity. The widening gap between the two lines demonstrates (unsurprisingly) that increased model capacity helps more on larger training sets. Even at 65536 experts (99.994% layer sparsity), computational efficiency for the model stays at a respectable 0.72 TFLOPS/GPU.\n\n5.3 Machine Translation (Single Language Pair)\n\nModel Architecture: Our model was a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decreased the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We inserted MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). Each MoE layer contained up to 2048 experts each with about two million parameters, adding a total of about 8 billion parameters to the models. Further details on model architecture, testing procedure and results can be found in Appendix E.\n\nDatasets: We benchmarked our method on the WMT'14 En→Fr and En→De corpora, whose training sets have 36M sentence pairs and 5M sentence pairs, respectively. The experimental protocols were also similar to those in (Wu et al., 2016): newstest2014 was used as the test set to compare against previous work (Luong et al., 2015a; Zhou et al., 2016; Wu et al., 2016), while the combination of newstest2012 and newstest2013 was used as the development set. We also tested the same model on a Google's Production English to French data.\n\nTable 2: Results on WMT'14 En→ Fr newstest2014 (bold values represent best results). Model Test Test ops/timenstep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.69 40.35 85M 8.7B 3 days/64 k40s MoE with 2048 Experts (longer training) 2.63 40.56 85M 8.7B 6 days/64 k40s GNMT (Wu et al., 2016) 2.79 39.22 214M 278M 6 days/96 k80s GNMT+RL (Wu et al., 2016) 2.96 39.92 214M 278M 6 days/96 k80s PBMT (Durrani et al., 2014) 37.0 LSTM (6-layer) (Luong et al., 2015b) 31.5 LSTM (6-layer+PosUnk) (Luong et al., 2015b) 33.1 DeepAtt (Zhou et al., 2016) 37.7 DeepAtt+PosUnk (Zhou et al., 2016) 39.2\n\nTable 3: Results on WMT'14 En → De newstest2014 (bold values represent best results). Model Test Test ops/timestep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 4.64 26.03 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 5.25 24.91 214M 278M 1 day/96 k80s GNMT +RL (Wu et al., 2016) 8.08 24.66 214M 278M 1 day/96 k80s PBMT (Durrani et al., 2014) 20.7 DeepAtt (Zhou et al., 2016) 20.6\n\nTable 4: Results on the Google Production En→ Fr dataset (bold values represent best results). Model Eval Eval Test Test ops/timestep Total Training Perplexity BLEU Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.60 37.27 2.69 36.57 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 2.78 35.80 2.87 35.56 214M 278M 6 days/96 k80s\n\nResults: Tables 2, 3, and 4 show the results of our largest models, compared with published results. Our approach achieved BLEU scores of 40.56 and 26.03 on the WMT'14 En→Fr and En→De benchmarks. As our models did not use RL refinement, these results constitute significant gains of 1.34 and 1.12 BLEU score on top of the strong baselines in (Wu et al., 2016). The perplexity scores are also better.2 On the Google Production dataset, our model achieved 1.01 higher test BLEU score even after training for only one sixth of the time.\n\n5.4 Multilingual Machine Translation\n\nDataset: (Johnson et al., 2016) train a single GNMT (Wu et al., 2016) model on a very large combined dataset of twelve language pairs. Results are somewhat worse than those for 12 separately trained single-pair GNMT models. This is not surprising, given that the twelve models have 12 times the capacity and twelve times the aggregate training of the one model. We repeat this experiment with a single MoE-augmented model. See Appendix E for details on model architecture.\n\nWe train our model on the same dataset as (Johnson et al., 2016) and process the same number of training examples (about 3 billion sentence pairs). Our training time was shorter due to the lower computational budget of our model.\n\nResults: Results for the single-pair GNMT models, the multilingual GNMT model and the multilingual MoE model are given in Table 5. The MoE model achieves 19% lower perplexity on the dev set than the multilingual GNMT model. On BLEU score, the MoE model significantly beats the multilingual GNMT model on 11 of the 12 language pairs (by as much as 5.84 points), and even beats the monolingual GNMT models on 8 of 12 language pairs. The poor performance on English → Korean seems to be a result of severe overtraining, as for the rarer language pairs a small number of real examples were highly oversampled in the training corpus.\n\nTable 5: Multilingual Machine Translation (bold values represent best results). GNMT-Mono GNMT-Multi MoE-Multi MoE-Multi vs. GNMT-Multi Parameters 278M / model 278M 8.7B ops/timestep 212M 212M 102M training time, hardware various 21 days, 96 k20s 12 days, 64 k40s Perplexity (dev) 4.14 3.35 -19% French → English Test BLEU 36.47 34.40 37.46 +3.06 German → English Test BLEU 31.77 31.17 34.80 +3.63 Japanese → English Test BLEU 23.41 21.62 25.91 +4.29 Korean → English Test BLEU 25.42 22.87 28.71 +5.84 Portuguese → English Test BLEU 44.40 42.53 46.13 +3.60 Spanish → English Test BLEU 38.00 36.04 39.39 +3.35 English → French Test BLEU 35.37 34.00 36.59 +2.59 English → German Test BLEU 26.43 23.15 24.53 +1.38 English → Japanese Test BLEU 23.66 21.10 22.78 +1.68 English → Korean Test BLEU 19.75 18.41 16.62 -1.79 English → Portuguese Test BLEU 38.40 37.35 37.90 +0.55 English → Spanish Test BLEU 34.50 34.25 36.21 +1.96\n\n6 Conclusion\n\nThis work is the first to demonstrate major wins from conditional computation in deep networks.\n\nWe carefully identified the design considerations and challenges of conditional computing and addressed them with a combination of algorithmic and engineering solutions. While we focused on text, conditional computation may help in other domains as well, provided sufficiently large training sets. We look forward to seeing many novel implementations and applications of conditional computation in the years to come.\n\nAcknowledgments\n\nWe would like to thank all of the members of the Google Brain and Google Translate teams who helped us with this project, in particular Zhifeng Chen, Yonghui Wu, and Melvin Johnson. Thanks also to our anonymous ICLR reviewers for the helpful suggestions on making this paper better.\n\n2Reported perplexities relative to the tokenization used by both our models and GNMT.\n\nReferences\n\nMartín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Gordon Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda B. Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. CoRR, abs/1603.04467, 2016. URL http://arxiv.org/abs/1603.04467.\n\nRahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. CoRR, abs/1611.06194, 2016. URL http://arxiv.org/abs/1611.\n\n06194.\n\nA. Almahairi, N. Ballas, T. Cooijmans, Y. Zheng, H. Larochelle, and A. Courville. Dynamic Capacity Networks. ArXiv e-prints, November 2015.\n\nDario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan, Christopher Fougner, Tony Han, Awni Y. Hannun, Billy Jun, Patrick LeGresley, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595, 2015. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.\n\nEmmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297, 2015.\n\nYoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.\n\nCiprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling.\n\narXiv preprint arXiv:1312.3005, 2013.\n\nK. Cho and Y. Bengio. Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning. ArXiv e-prints, June 2014. Ronan Collobert, Samy Bengio, and Yoshua Bengio. A parallel mixture of SVMs for very large scale problems. Neural Computing, 2002.\n\nAndrew Davis and Itamar Arel. Low-rank approximations for conditional feedforward computation in deep neural networks. arXiv preprint arXiv:1312.4461, 2013. Marc Peter Deisenroth and Jun Wei Ng. Distributed Gaussian processes. In ICML, 2015.\n\nJohn Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization, 2010.\n\nNadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. Edinburgh's phrase-based machine translation systems for wmt-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation, 2014.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013. Ekaterina Garmash and Christof Monz. Ensemble learning for multi-source neural machine translation. In staff.science.uva.nl/c.monz, 2016.\n\nFelix A. Gers, Jürgen A. Schmidhuber, and Fred A. Cummins. Learning to forget: Continual prediction with lstm. Neural Computation, 2000.\n\nAudrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. Memory-efficient backpropagation through time. CoRR, abs/1606.03401, 2016. URL http://arxiv.org/ abs/1606.03401. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2015.\n\nGeoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 2012. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.\n\nSergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.\n\nRobert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computing, 1991.\n\nMelvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation.\n\nCoRR, abs/1611.04558, 2016. URL http://arxiv.org/abs/1611.04558.\n\nMichael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm.\n\nNeural Computing, 1994. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.\n\nDiederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.\n\nReinhard Kneser and Hermann. Ney. Improved backingoff for m-gram language modeling., 1995.\n\nAlex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.\n\nQuoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeffrey Dean, and Andrew Y. Ng. Building high-level features using large scale unsupervised learning. In ICML, 2012. Patrick Gallinari Ludovic Denoyer. Deep sequential neural network. arXiv preprint arXiv:1410.0510, 2014.\n\nMinh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attentionbased neural machine translation. EMNLP, 2015a.\n\nMinh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the rare word problem in neural machine translation. ACL, 2015b.\n\nCarl Edward Rasmussen and Zoubin Ghahramani. Infinite mixtures of Gaussian process experts.\n\nNIPS, 2002.\n\nHasim Sak, Andrew W Senior, and Françoise Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH, pp. 338–342, 2014.\n\nMike Schuster and Kaisuke Nakajima. Japanese and Korean voice search. ICASSP, 2012.\n\nBabak Shahbaba and Radford Neal. Nonlinear models using dirichlet process mixtures. JMLR, 2009.\n\nIlya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks.\n\nIn NIPS, 2014.\n\nLucas Theis and Matthias Bethge. Generative image modeling using spatial LSTMs. In NIPS, 2015. Volker Tresp. Mixtures of Gaussian Processes. In NIPS, 2001.\n\nYonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.\n\nBangpeng Yao, Dirk Walther, Diane Beck, and Li Fei-fei. Hierarchical mixture of classification experts uncovers interactions between brain regions. In NIPS. 2009.\n\nWojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization.\n\narXiv preprint arXiv:1409.2329, 2014.\n\nJie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with fast-forward connections for neural machine translation. arXiv preprint arXiv:1606.04199, 2016.\n\nAppendices A Load-Balancing Loss\n\nAs discussed in section 4, for load-balancing purposes, we want to define an additional loss function to encourage experts to receive roughly equal numbers of training examples. Unfortunately, the number of examples received by an expert is a discrete quantity, so it can not be used in backpropagation. Instead, we define a smooth estimator Load(X) of the number of examples assigned to each expert for a batch X of inputs. The smoothness allows us to back-propagate gradients through the estimator. This is the purpose of the noise term in the gating function. We define P(x, i) as the probability that G(x)iis nonzero, given a new random choice of noise on element i, but keeping the already-sampled choices of noise on the other elements. To compute P(x, i), we note that the G(x)iis nonzero if and only if H(x)iis greater than the k th-greatest element of H(x) excluding itself. The probability works out to be:\n\n$$P(x,i)=P r\\Big((x\\cdot W_{g}){i}+S t a n d x N o r a l()\\cdot S o f t p u s((x\\cdot W{n o i s e})_{i})$$ $$>k t h_e x c l u d i n(H(x),k,i)\\Big)$$ $$({\\boldsymbol{8}})$$\n\n$$(9)$$ $$(10)$$ (8) Where kth_excluding(v, k, i) means the kth highest component of v, excluding component i. Simplifying, we get:\n\n$$P(x,i)=\\Phi{\\Big(}{\\frac{(x\\cdot W_{g}){i}-k t h_c x c l u d i n g(H(x),k,i)}{S o f t p l u s((x\\cdot W{n o i s e})_{i})}}{\\Big)}$$\n\nWhere Φ is the CDF of the standard normal distribution.\n\n$$L o a d(X){i}=\\sum{x\\in X}P(x,i)$$\n\n$$(11)$$ P(x, i) (10) We can now define the load loss to be the square of the coefficient of variation of the load vector, multiplied by a hand-tuned scaling factor wload.\n\n$$L_{l o a d}(X)=w_{l o a d}\\cdot C V(L o a d(X))^{2}$$ Lload(X) = wload · CV (Load(X))2(11) Initial Load Imbalance: To avoid out-of-memory errors, we need to initialize the network in a state of approximately equal expert load (since the soft constraints need some time to work). To accomplish this, we initialize the matrices Wg and Wnoise to all zeros, which yields no signal and some noise.\n\nExperiments: We trained a set of models with identical architecture (the MoE-256 model described in Appendix C), using different values of wimportance and wload. We trained each model for 10 epochs, then measured perplexity on the test set. We also measured the coefficients of variation in Importance and Load, as well as ratio of the load on the most overloaded expert to the average load. This last value is significant for load balancing purposes on distributed hardware. All of these metrics were averaged over several training batches.\n\nwimportance wload Test Perplexity CV (Importance(X)) CV (Load(X)) max(Load(X)) mean(Load(X)) 0.0 0.0 39.8 3.04 3.01 17.80 0.2 0.0 35.6 0.06 0.17 1.47 0.0 0.2 35.7 0.22 0.04 1.15 0.1 0.1 35.6 0.06 0.05 1.14 0.01 0.01 35.7 0.48 0.11 1.37 1.0 1.0 35.7 0.03 0.02 1.07\n\nTable 6: Experiments with different combinations of losses.\n\n13 Results: Results are reported in Table 6. All the combinations containing at least one the two losses led to very similar model quality, where having no loss was much worse. Models with higher values of wload had lower loads on the most overloaded expert.\n\nB Hierachical Mixture Of Experts\n\nIf the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network.3If the hierarchical MoE consists of a groups of b experts each, we denote the primary gating network by Gprimary, the secondary gating networks by (G1, G2..Ga), and the expert networks by (E0,0, E0,1..Ea,b). The output of the MoE is given by:\n\n$$y_{H}=\\sum_{i=1}^{a}\\sum_{j=1}^{b}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}\\cdot E{i,j}(x)$$ $$(12)$$ $$(13)$$ $$(14)$$\n\nOur metrics of expert utilization change to the following:\n\n$$I m p o r t a n c e_{H}(X){i,j}=\\sum{x\\in X}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}$$ $$L o a d{H}(X){i,j}={\\frac{L o a d{p r i m a r y}(X){i}\\cdot L o a d{i}(X^{(i)})_{j}}{|X^{(i)}|}}$$\n\nLoadprimary and Loadi deonte the Load functions for the primary gating network and i th secondary gating network respectively. X(i) denotes the subset of X for which Gprimary(x)i > 0.\n\nIt would seem simpler to let LoadH(X)i,j = Loadi(Xi)j , but this would not have a gradient with respect to the primary gating network, so we use the formulation above.\n\nC 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS C.1 8-MILLION-OPERATIONS-PER-TIMESTEP MODELS Model Architecture: Our model consists of five layers: a word embedding layer, a recurrent Long Short-Term Memory (LSTM) layer (Hochreiter & Schmidhuber, 1997; Gers et al., 2000), a MoE layer, a second LSTM layer, and a softmax layer. The dimensionality of the embedding layer, the number of units in each LSTM layer, and the input and output dimensionality of the MoE layer are all equal to 512. For every layer other than the softmax, we apply drouput (Zaremba et al., 2014) to the layer output, dropping each activation with probability DropP rob, otherwise dividing by (1 − DropP rob). After dropout, the output of the previous layer is added to the layer output.\n\nThis residual connection encourages gradient flow (He et al., 2015).\n\nMoE Layer Architecture: Each expert in the MoE layer is a feed forward network with one ReLU-activated hidden layer of size 1024 and an output layer of size 512. Thus, each expert contains [512 ∗ 1024] + [1024 ∗ 512] = 1M parameters. The output of the MoE layer is passed through a sigmoid function before dropout. We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts.\n\nWe call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers. Thus, each example is processed by exactly 4 experts for a total of 4M ops/timestep. The two LSTM layers contribute 2M ops/timestep each for the desired total of 8M.\n\n3 We have not found the need for deeper hierarchies.\n\nComputationally-Matched Baselines: The MoE-4 model does not employ sparsity, since all 4 experts are always used. In addition, we trained four more computationally-matched baseline models with no sparsity:\n\nMoE-1-Wide: The MoE layer consists of a single \"expert\" containing one ReLU-activated hidden layer of size 4096.\n\nMoE-1-Deep: The MoE layer consists of a single \"expert\" containing four ReLU-activated hidden layers, each with size 1024.\n\n4xLSTM-512: We replace the MoE layer with two additional 512-unit LSTM layers.\n\nLSTM-2048-512: The model contains one 2048-unit LSTM layer (and no MoE). The output of the LSTM is projected down to 512 dimensions (Sak et al., 2014). The next timestep of the LSTM receives the projected output. This is identical to one of the models published in (Jozefowicz et al., 2016). We re-ran it to account for differences in training regimen, and obtained results very similar to the published ones. Training: The models were trained on a cluster of 16 K40 GPUs using the synchronous method described in Section 3. Each batch consisted of a set of sentences totaling roughly 300,000 words. In the interest of time, we limited training to 10 epochs, (27,000 steps). Training took 12-16 hours for all models, except for MoE-4, which took 18 hours (since all the expert computation was performed on only 4 of 16 GPUs). We used the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 1000 training steps, and decreased after that so as to be proportional to the inverse square root of the step number. The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016). For each model, we performed a hyper-parmeter search to find the best dropout probability, in increments of 0.1.\n\nTo ensure balanced expert utilization we set wimportance = 0.1 and wload = 0.1, as described in Section 4 and Appendix A.\n\nResults: We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol. Results are reported in Table 7. For each model, we report the test perplexity, the computational budget, the parameter counts, the value of DropP rob, and the computational efficiency.\n\nModel Test Test ops/timestep #Params excluding Total Drop- TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params P rob per GPU 10 epochs (final) (millions) (billions) (observed) Kneser-Ney 5-gram* 67.6 0.00001 1.8 LSTM-512-512* 54.1 2.4 2.4 0.8 0.1 LSTM-1024-512* 48.2 4.7 4.7 0.8 0.1 LSTM-2048-512* 45.0 43.7 9.4 9.4 0.8 0.1 0.61 LSTM-2048-512 44.7 9.4 9.4 0.8 0.1 1.21 4xLSTM-512 46.0 8.4 8.4 0.8 0.1 1.07 MoE-1-Wide 46.1 8.4 8.4 0.8 0.1 1.29 MoE-1-Deep 45.7 8.4 8.4 0.8 0.1 1.29 MoE-4 45.0 8.4 8.4 0.8 0.1 0.52 MoE-32 39.7 8.4 37.8 0.9 0.1 0.87 MoE-256 35.7 8.6 272.9 1.1 0.1 0.81 MoE-256-h 36.0 8.4 272.9 1.1 0.1 0.89 MoE-1024-h 34.6 8.5 1079.0 1.9 0.2 0.90 MoE-4096-h 34.1 8.9 4303.4 5.1 0.2 0.74 2xLSTM-8192-1024* 34.7 30.6 151.0 151.0 1.8 0.25 1.09 MoE-34M 31.3 33.8 4313.9 6.0 0.3 1.22 MoE-143M 28.0 142.7 4371.1 6.0 0.4 1.56\n\nC.2 More Expensive Models\n\nWe ran two additional models (MoE-34M and MoE-143M) to investigate the effects of adding more computation in the presence of a large MoE layer. These models have computation budgets of 34M and 143M ops/timestep. Similar to the models above, these models use a MoE layer between two LSTM layers. The dimensionality of the embedding layer, and the input and output dimensionality of the MoE layer are set to 1024 instead of 512. For MoE-34M, the LSTM layers have 1024 units.\n\nFor MoE-143M, the LSTM layers have 4096 units and an output projection of size 1024 (Sak et al., 2014). MoE-34M uses a hierarchical MoE layer with 1024 experts, each with a hidden layer of size 2048. MoE-143M uses a hierarchical MoE layer with 256 experts, each with a hidden layer of size 8192. Both models have 4B parameters in the MoE layers. We searched for the best DropP rob for each model, and trained each model for 10 epochs.\n\nThe two models achieved test perplexity of 31.3 and 28.0 respectively, showing that even in the presence of a large MoE, more computation is still useful. Results are reported at the bottom of Table 7. The larger of the two models has a similar computational budget to the best published model from the literature, and training times are similar. Comparing after 10 epochs, our model has a lower test perplexity by 18%.\n\nD 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS Model Architecture: The models are similar in structure to the 8-million-operations-per-timestep models described in the previous section. We vary the number of experts between models, using an ordinary MoE layer with 32 experts and hierarchical MoE layers with 256, 1024, 4096, 16384, 65536 and 131072 experts. For the hierarchical MoE layers, the first level branching factors are 32, 32, 64, 128, 256 and 256, respectively.\n\nTraining: Models are trained on a cluster of 32 Tesla K40 GPUs, except for the last two models, which are trained on clusters of 64 and 128 GPUs so as to have enough memory for all the parameters. For all models, training batch sizes are approximately 2.5 million words. Models are trained once-through over about 100 billion words. We implement several memory optimizations in order to fit up to 1 billion parameters per GPU. First, we do not store the activations of the hidden layers of the experts, but instead recompute them on the backwards pass. Secondly, we modify the optimizer on the expert parameters to require less auxiliary storage: The Adam optimizer (Kingma & Ba, 2015) keeps first and second moment estimates of the perparameter gradients. This triples the required memory. To avoid keeping a first-moment estimator, we set β1 = 0. To reduce the size of the second moment estimator, we replace it with a factored approximation. For a matrix of parameters, instead of maintaining a full matrix of second-moment estimators, we maintain vectors of row-wise and column-wise averages of that matrix. At each step, the matrix of estimators is taken to be the outer product of those two vectors divided by the mean of either one. This technique could similarly be applied to Adagrad (Duchi et al., 2010).\n\nModel Test Test ops/timestep #Params excluding Total TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params per GPU .1 epochs 1 epoch (millions) (billions) (observed) Kneser-Ney 5-gram 67.1 45.3 0.00001 76.0 4xLSTM-512 54.5 47.0 8.4 8.4 0.1 1.23 MoE-32 48.5 40.4 8.4 37.8 0.1 0.83 MoE-256-h 42.8 35.3 8.4 272.9 0.4 1.11 MoE-1024-h 40.3 32.7 8.5 1079.0 1.2 1.14 MoE-4096-h 38.9 30.9 8.6 4303.4 4.4 1.07 MoE-16384-h 38.2 29.7 8.8 17201.0 17.3 0.96 MoE-65536-h 38.2 28.9 9.2 68791.0 68.9 0.72 MoE-131072-h 39.8 29.2 9.7 137577.6 137.7 0.30\n\nResults: We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8. Perplexity after 100 billion training words is 39% lower for the 68-billion-parameter MoE model than for the baseline model. It is notable that the measured computational efficiency of the largest model (0.30 TFLOPS/GPU) is very low compared to the other models. This is likely a result of the fact that, for purposes of comparison to the other models, we did not increase the training batch size proportionally to the number of GPUs. For comparison, we include results for a computationally matched baseline model consisting of 4 LSTMs, and for an unpruned 5-gram model with Kneser-Ney smoothing (Kneser & Ney, 1995).4\n\nE Machine Translation - Experimental Details\n\nModel Architecture for Single Language Pair MoE Models: Our model is a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decrease the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We insert MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). We use an attention mechanism between the encoder and decoder, with the first decoder LSTM receiving output from and providing input for the attention 5. All of the layers in our model have input and output dimensionality of 512. Our LSTM layers have 2048 hidden units, with a 512-dimensional output projection. We add residual connections around all LSTM and MoE layers to encourage gradient flow (He et al., 2015). Similar to GNMT, to effectively deal with rare words, we used subword units (also known as \"wordpieces\") (Schuster & Nakajima, 2012) for inputs and outputs in our system.\n\nWe use a shared source and target vocabulary of 32K wordpieces. We also used the same beam search technique as proposed in (Wu et al., 2016).\n\nWe train models with different numbers of experts in the MoE layers. In addition to a baseline model with no MoE layers, we train models with flat MoE layers containing 32 experts, and models with hierarchical MoE layers containing 512 and 2048 experts. The flat MoE layers use k = 4 and the hierarchical MoE models use k = 2 at each level of the gating network. Thus, each input is processed by exactly 4 experts in each MoE layer. Each expert in the MoE layer is a feed forward network with one hidden layer of size 2048 and ReLU activation. Thus, each expert contains [512 ∗ 2048] + [2048 ∗ 512] = 2M parameters. The output of the MoE layer is passed through a sigmoid function. We use the strictly-balanced gating function described in Appendix F.\n\nModel Architecture for Multilingual MoE Model: We used the same model architecture as for the single-language-pair models, with the following exceptions: We used noisy-top-k gating as described in Section 2.1, not the scheme from Appendix F. The MoE layers in the encoder and decoder are non-hierarchical MoEs with n = 512 experts, and k = 2. Each expert has a larger hidden layer of size 8192. This doubles the amount of computation in the MoE layers, raising the computational budget of the entire model from 85M to 102M ops/timestep. Training: We trained our networks using the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 2000 training steps, held constant for an additional 8000 steps, and decreased after that so as to be proportional to the inverse square root of the step number. For the single-language-pair models, similarly to (Wu et al., 2016), we applied dropout (Zaremba et al., 2014) to the output of all embedding, LSTM and MoE layers, using DropP rob = 0.4. Training was done synchronously on a cluster of up to 64 GPUs as described in section 3. Each training batch consisted of a set of sentence pairs containing roughly 16000 words per GPU.\n\nTo ensure balanced expert utilization we set wimportance = 0.01 and wload = 0.01, as described in Section 4 and Appendix A. Metrics: We evaluated our models using the perplexity and the standard BLEU score metric. We reported tokenized BLEU score as computed by the multi-bleu.pl script, downloaded from the public implementation of Moses (on Github), which was also used in (Luong et al., 2015a). Results: Tables 2, 3 and 4 in Section 5.3 show comparisons of our results to other published methods. Figure 4 shows test perplexity as a function of number of words in the (training data's) source sentences processed for models with different numbers of experts. As can be seen from the Figure, as we increased the number of experts to approach 2048, the test perplexity of our model continued to improve.\n\nWe found that the experts indeed become highly specialized by syntax and/or semantics, as can be seen in Table 9. For example, one expert is used when the indefinite article \"a\" introduces the direct object in a verb phrase indicating importance or leadership.\n\nExpert 381 Expert 752 Expert 2004 ... with researchers , ... ... plays a core ... ... with rapidly growing ... ... to innovation . ... plays a critical ... ... under static conditions ... ... tics researchers . ... provides a legislative ... ... to swift ly ... ... the generation of ... ... play a leading ... ... to dras tically ... ... technology innovations is ... ... assume a leadership ... ... the rapid and ... ... technological innovations , ... ... plays a central ... ... the fast est ... ... support innovation throughout ... ... taken a leading ... ... the Quick Method ... ... role innovation will ... ... established a reconciliation ... ... rec urrent ) ... ... research scienti st ... ... played a vital ... ... provides quick access ... ... promoting innovation where ... ... have a central ... ... of volatile organic ... ... ... ...\n\nDue to some peculiarities in our infrastructure which have since been fixed, at the time we ran some of the machine translation experiments, our models ran faster if every expert received exactly the same batch size. To accommodate this, we used a different gating function which we describe below. Recall that we define the softmax gating function to be:\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ Gσ(x) = Sof tmax(x · Wg) (15) Sparse Gating (alternate formulation): To obtain a sparse gating vector, we multiply Gσ(x) component-wise with a sparse mask M(Gσ(x)) and normalize the output. The mask itself is a function of Gσ(x) and specifies which experts are assigned to each input example:\n\n$$(15)$$ $$G(x){i}=\\frac{G{\\sigma}(x){i}M(G{\\sigma}(x)){i}}{\\sum{j=1}^{n}G_{\\sigma}(x){j}M(G{\\sigma}(x))_{j}}\\tag{1}$$ $$(17)$$\n\n$$(16)$$\n\nTop-K Mask: To implement top-k gating in this formulation, we would let M(v) = T opK(v, k), where:\n\n$$T o p K(v,k){i}=\\begin{cases}1&{\\mathrm{if~}}v{i}{\\mathrm{~is~in~the~top~}}k{\\mathrm{~elements~of~}}v.\\ 0&{\\mathrm{otherwise.}}\\end{cases}$$ 0 otherwise. (17) Batchwise Mask: To force each expert to receive the exact same number of examples, we introduce an alternative mask function, Mbatchwise(X, m), which operates over batches of input vectors.\n\nInstead of keeping the top k values per example, we keep the top m values per expert across the training batch, where m = k|X| n, so that each example is sent to an average of k experts.\n\n$$M_{b a t c h w i s e}(X,m){j,i}=\\begin{cases}1&\\text{if}X{j,i}\\text{is in the top}m\\text{values for to expert}i\\ 0&\\text{otherwise}\\end{cases}$$\n\nAs our experiments suggest and also observed in (Ioffe & Szegedy, 2015), using a batchwise function during training (such as Mbatchwise) requires modifications to the inference when we may not have a large batch of examples. Our solution to this is to train a vector T of per-expert threshold values to approximate the effects of the batchwise mask. We use the following mask at inference time:\n\n$$(18)$$ $$M_{threshold}(x,T){i}=\\begin{cases}1&\\text{if}x{i}>T_{i}\\ 0&\\text{otherwise}\\end{cases}\\tag{1}$$ $$(19)$$\n\nTo learn the threshold values, we apply an additional loss at training time which is minimized when the batchwise mask and the threshold mask are identical.\n\n$$L_{batchwise}(X,T,m)=\\sum_{j=1}^{|X|}\\sum_{i=1}^{n}(M_{threshold}(x,T){i}-M{batchwise}(X,m){j,i})(X{j,i}-T_{i})\\tag{20}$$\n\nG ATTENTION FUNCTION The attention mechanism described in GNMT (Wu et al., 2016) involves a learned \"Attention Function\" A(xi, yj ) which takes a \"source vector\" xi and a \"target vector\" yj , and must be computed for every source time step i and target time step j. In GNMT, the attention function is implemented as a feed forward neural network with a hidden layer of size n. It can be expressed as:\n\n$$A_{G N M T}(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}t a n h((x_{i}U){d}+(y{j}W)_{d})$$\n\nWhere U and W are trainable weight matrices and V is a trainable weight vector.\n\nFor performance reasons, in our models, we used a slightly different attention function:\n\n$$A(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}tanh((x_{i}U){d})tanh((y{j}W)_{d})\\tag{22}$$\n\nWith our attention function, we can simultaneously compute the attention function on multiple source time steps and multiple target time steps using optimized matrix multiplications. We found little difference in quality between the two functions.\n\n$$(21)$$",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/1701.06538v1.md"
+          },
+          "headlines": {
+            "1 Introduction And Related Work": [
+              "1.1 Conditional Computation",
+              "1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER",
+              "1.3 Related Work On Mixtures Of Experts"
+            ],
+            "2 The Structure Of The Mixture-Of-Experts Layer": [
+              "2.1 Gating Network"
+            ],
+            "3 Addressing Performance Challenges": [
+              "3.1 The Shrinking Batch Problem",
+              "3.2 Network Bandwidth"
+            ],
+            "4 Balancing Expert Utilization": [],
+            "5 Experiments": [
+              "5.1 1 Billion Word Language Modeling Benchmark",
+              "5.2 100 BILLION WORD GOOGLE NEWS CORPUS",
+              "5.3 Machine Translation (Single Language Pair)",
+              "5.4 Multilingual Machine Translation"
+            ],
+            "6 Conclusion": [],
+            "Appendices": [
+              "A Load-Balancing Loss",
+              "B Hierachical Mixture Of Experts",
+              "C 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS",
+              "D 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS",
+              "E Machine Translation - Experimental Details"
+            ]
+          },
+          "summary": "The paper introduces a Sparsely-Gated Mixture-of-Experts (MoE) layer to enhance neural network capacity without a proportional increase in computation. This approach uses a trainable gating network to select a sparse combination of experts for each input, significantly improving model capacity and performance in language modeling and machine translation tasks. The MoE layer, with up to 137 billion parameters, achieves better results than state-of-the-art models at lower computational costs. The paper addresses challenges like batch size and network bandwidth, proposing solutions like mixing data and model parallelism. Experiments show that MoE models outperform traditional models in various benchmarks, demonstrating the potential of conditional computation in deep learning.",
+          "summary_embedding": [
+            -0.01644345559179783,
+            -0.016995344310998917,
+            -0.016372699290513992,
+            -0.014349107630550861,
+            0.02282555401325226,
+            0.045934125781059265,
+            -0.014646278694272041,
+            0.03563220426440239,
+            -0.004829026758670807,
+            0.005617944523692131,
+            -0.00932550523430109,
+            -0.008143897168338299,
+            0.005575491581112146,
+            0.01538213063031435,
+            -0.005653322208672762,
+            -0.020306676626205444,
+            -0.013832597061991692,
+            -0.023235931992530823,
+            0.028344441205263138,
+            -0.020080260932445526,
+            0.011823155917227268,
+            -0.020943470299243927,
+            -0.025868017226457596,
+            -0.016500059515237808,
+            0.025061409920454025,
+            0.024806693196296692,
+            0.02583971433341503,
+            0.003827843815088272,
+            -0.03492465242743492,
+            0.033651065081357956,
+            0.014674580655992031,
+            0.012488252483308315,
+            -0.031500112265348434,
+            -0.009162768721580505,
+            -0.05351905897259712,
+            -0.05227377265691757,
+            0.04143410921096802,
+            0.022245362401008606,
+            -0.016231190413236618,
+            0.023717066273093224,
+            0.017264213413000107,
+            0.005816058721393347,
+            -0.020632149651646614,
+            -0.020844414830207825,
+            0.003700484987348318,
+            -0.013103820383548737,
+            -0.01983969286084175,
+            -0.013528349809348583,
+            -0.01828308403491974,
+            -0.01577835902571678,
+            0.009551920928061008,
+            0.0002856731880456209,
+            0.0029186424799263477,
+            -0.003424540627747774,
+            -0.047830358147621155,
+            0.0248208437114954,
+            0.019853845238685608,
+            0.03492465242743492,
+            0.04633035510778427,
+            0.013011839240789413,
+            0.01516986545175314,
+            -0.014271277002990246,
+            -0.04021712392568588,
+            0.005497660953551531,
+            -0.00041656987741589546,
+            -0.005380915477871895,
+            -0.005819596350193024,
+            0.008575502783060074,
+            0.008724087849259377,
+            -0.01143400277942419,
+            -0.04202845320105553,
+            0.013952880166471004,
+            -0.020561393350362778,
+            0.00010143388499272987,
+            0.01354957651346922,
+            0.008292482234537601,
+            0.02677368000149727,
+            0.03195294365286827,
+            0.023971782997250557,
+            -0.008724087849259377,
+            -0.01340806670486927,
+            0.019924599677324295,
+            -0.0022676968947052956,
+            0.007018893025815487,
+            0.04570771008729935,
+            -0.04033033177256584,
+            0.02163686975836754,
+            -0.025726506486535072,
+            -0.015339677222073078,
+            -0.005349075887352228,
+            -0.022471778094768524,
+            0.02806142158806324,
+            -0.007344366051256657,
+            0.017801949754357338,
+            0.017094401642680168,
+            0.01594817079603672,
+            0.013330236077308655,
+            -0.001894464367069304,
+            -0.022938761860132217,
+            0.0018360915128141642,
+            -0.01541043259203434,
+            0.02114158496260643,
+            0.0015230007702484727,
+            0.005469358991831541,
+            0.027594437822699547,
+            0.031160488724708557,
+            -0.023490650579333305,
+            0.009502392262220383,
+            0.02422650158405304,
+            0.06967949122190475,
+            0.015424583107233047,
+            -0.006435164716094732,
+            -0.0024640418123453856,
+            0.04143410921096802,
+            0.0015513027319684625,
+            0.0038349193055182695,
+            -0.06826439499855042,
+            -0.03022652305662632,
+            -0.0011365016689524055,
+            -0.02254253253340721,
+            0.02227366529405117,
+            0.01002597901970148,
+            -0.009750034660100937,
+            -0.022457627579569817,
+            -0.04516997188329697,
+            0.05190584436058998,
+            -0.00545874610543251,
+            -0.016259491443634033,
+            -0.005554265342652798,
+            -0.05340585112571716,
+            0.00039136342820711434,
+            0.015042506158351898,
+            -0.018481198698282242,
+            -0.03670767694711685,
+            0.036141637712717056,
+            0.0007398316520266235,
+            0.03741522878408432,
+            0.012459950521588326,
+            -0.01708024926483631,
+            -0.020094411447644234,
+            0.022259512916207314,
+            -0.011639192700386047,
+            -0.0038313816767185926,
+            -0.027339721098542213,
+            0.0065236082300543785,
+            -0.028103873133659363,
+            -0.0014566679019480944,
+            0.06730213016271591,
+            -0.04816998168826103,
+            0.02946236915886402,
+            -0.029660483822226524,
+            0.023660462349653244,
+            -0.008575502783060074,
+            0.0278916098177433,
+            -0.010238244198262691,
+            0.005840822588652372,
+            -0.013457595370709896,
+            -0.0010586712742224336,
+            -0.020745357498526573,
+            0.0034210027661174536,
+            -0.016259491443634033,
+            -0.027750099077820778,
+            -0.007054270710796118,
+            0.031273696571588516,
+            0.01698119193315506,
+            0.025655752047896385,
+            0.025429336354136467,
+            0.01246702577918768,
+            -0.04935866594314575,
+            0.02595292218029499,
+            0.0018926954362541437,
+            0.013514199294149876,
+            0.013181651011109352,
+            -0.0010418669553473592,
+            0.025599148124456406,
+            0.012049571610987186,
+            0.025061409920454025,
+            0.037443529814481735,
+            0.0027682881336659193,
+            0.04737752676010132,
+            -0.015693452209234238,
+            -0.005292471963912249,
+            0.028528403490781784,
+            0.000505455827806145,
+            0.02465103194117546,
+            -0.001686621573753655,
+            0.013330236077308655,
+            0.000776978034991771,
+            0.03834919258952141,
+            -0.006339645478874445,
+            0.029518973082304,
+            0.021608568727970123,
+            0.028316138312220573,
+            -0.014759486541152,
+            0.023122724145650864,
+            -0.019754787907004356,
+            -0.02292460948228836,
+            0.035547297447919846,
+            0.004082561936229467,
+            0.01886327564716339,
+            -0.028457649052143097,
+            -0.030679354444146156,
+            0.015452885068953037,
+            -0.01855195313692093,
+            0.016556663438677788,
+            0.03639635816216469,
+            0.0008273909334093332,
+            -0.0025719432160258293,
+            0.026292547583580017,
+            -0.04245298355817795,
+            0.012856177985668182,
+            -0.01532552670687437,
+            -0.02855670638382435,
+            -0.019528372213244438,
+            -0.00875946506857872,
+            -0.0018095583654940128,
+            -0.013691086322069168,
+            -0.0037570889107882977,
+            -0.007563706487417221,
+            -0.02459442801773548,
+            0.00935380719602108,
+            0.013627407141029835,
+            0.009849091991782188,
+            -0.018665160983800888,
+            0.0019952901639044285,
+            -0.008257105015218258,
+            -0.0160047747194767,
+            -0.03752843663096428,
+            0.015268922783434391,
+            -0.0024888061452656984,
+            0.009191070683300495,
+            0.04078316316008568,
+            0.028485950082540512,
+            -0.005055442452430725,
+            0.0329718142747879,
+            -0.033707667142152786,
+            0.017915157601237297,
+            -0.008320784196257591,
+            0.009191070683300495,
+            0.002989397384226322,
+            -0.025542544201016426,
+            0.027424626052379608,
+            -0.03792466223239899,
+            -0.038151077926158905,
+            0.022995365783572197,
+            0.011858533136546612,
+            0.0017096169758588076,
+            -0.019386861473321915,
+            -0.017717044800519943,
+            0.0008499440737068653,
+            0.002752368338406086,
+            -0.018834972754120827,
+            -0.01019579079002142,
+            0.02227366529405117,
+            -0.025500090792775154,
+            0.036085035651922226,
+            0.023391593247652054,
+            -0.03520767390727997,
+            -0.00608492735773325,
+            -0.01903308741748333,
+            0.059037946164608,
+            -0.0011506526498124003,
+            -0.010301923379302025,
+            0.022938761860132217,
+            -0.027268964797258377,
+            -0.004524780437350273,
+            0.022301966324448586,
+            0.01828308403491974,
+            -0.008278331719338894,
+            0.003983504604548216,
+            -0.022287815809249878,
+            -0.05281151086091995,
+            -0.010577867738902569,
+            -0.0003305583668407053,
+            0.054368119686841965,
+            -0.012367968447506428,
+            0.0020695829298347235,
+            -0.006413938011974096,
+            0.004100250545889139,
+            0.03704730048775673,
+            0.002140337834134698,
+            0.0025082635693252087,
+            -0.013698162510991096,
+            0.0371888130903244,
+            0.022316116839647293,
+            0.005883275531232357,
+            -0.012162779457867146,
+            0.005377377849072218,
+            -0.04899074137210846,
+            -0.0009118546731770039,
+            0.005741765722632408,
+            0.007386818993836641,
+            -0.025910470634698868,
+            0.017717044800519943,
+            0.009205221198499203,
+            -0.0024693484883755445,
+            -0.009792488068342209,
+            0.01173117384314537,
+            0.025429336354136467,
+            -0.019938750192523003,
+            -0.05700020492076874,
+            0.0034705311991274357,
+            0.012014193460345268,
+            -0.004054259974509478,
+            -0.019401011988520622,
+            0.006792477332055569,
+            0.0021934041287750006,
+            0.022698193788528442,
+            -0.011271266266703606,
+            -0.0029221803415566683,
+            0.040585052222013474,
+            0.009516543708741665,
+            -0.0016813150141388178,
+            -0.017717044800519943,
+            -0.011794853024184704,
+            0.018070818856358528,
+            0.004740582779049873,
+            0.0035289041697978973,
+            -0.01751893013715744,
+            -0.029264256358146667,
+            0.02800481766462326,
+            0.019967053085565567,
+            -0.048905834555625916,
+            -0.008200501091778278,
+            -0.019273653626441956,
+            0.0019599127117544413,
+            0.008695785887539387,
+            0.04613224044442177,
+            -0.00590803986415267,
+            0.008313708938658237,
+            -0.020179318264126778,
+            -0.05003791302442551,
+            -0.02708500251173973,
+            0.03707560524344444,
+            -0.028684064745903015,
+            -0.04276430234313011,
+            -0.010252394713461399,
+            0.016740625724196434,
+            -0.009403335861861706,
+            -0.018679311498999596,
+            -0.011738249100744724,
+            -0.04969828948378563,
+            -0.003497064346447587,
+            0.0337359681725502,
+            0.017264213413000107,
+            -0.008441068232059479,
+            -0.033877480775117874,
+            0.019641580060124397,
+            -0.007160402834415436,
+            -0.006474079564213753,
+            0.0007862646016292274,
+            0.008794843219220638,
+            0.02400008589029312,
+            0.011879759840667248,
+            -0.02763689123094082,
+            -0.03427370637655258,
+            -0.0024888061452656984,
+            0.014929298311471939,
+            -0.00878776703029871,
+            0.0245378240942955,
+            0.024679332971572876,
+            -0.022995365783572197,
+            -0.01654251106083393,
+            0.015750056132674217,
+            -0.05779265984892845,
+            -0.010287772864103317,
+            -0.029235953465104103,
+            -0.006183984223753214,
+            0.0009613831643946469,
+            0.0061096916906535625,
+            0.00012304731353651732,
+            0.03846240043640137,
+            0.008801918476819992,
+            0.006014172453433275,
+            0.009191070683300495,
+            0.01913214474916458,
+            0.028471799567341805,
+            0.009155693463981152,
+            -0.03517937287688255,
+            0.02850010246038437,
+            -0.017051948234438896,
+            0.007811348885297775,
+            -0.025754809379577637,
+            -0.013033065013587475,
+            0.00443633645772934,
+            0.02725481428205967,
+            -0.000503244751598686,
+            0.018849123269319534,
+            -0.021919889375567436,
+            0.037500135600566864,
+            -0.029915200546383858,
+            -0.05278320610523224,
+            -0.02005195803940296,
+            -0.008285406976938248,
+            0.015764206647872925,
+            0.017278363928198814,
+            -0.019924599677324295,
+            -0.0286557637155056,
+            -0.021750077605247498,
+            0.006470541935414076,
+            0.010542490519583225,
+            -0.021099131554365158,
+            -0.010627396404743195,
+            0.01502835564315319,
+            -0.00404364662244916,
+            -0.01886327564716339,
+            0.013684011064469814,
+            0.015693452209234238,
+            0.03169822692871094,
+            -0.03308502212166786,
+            0.004287751391530037,
+            -0.025967074558138847,
+            -0.04327373951673508,
+            0.004096712917089462,
+            0.03166992589831352,
+            0.027509532868862152,
+            -0.018636859953403473,
+            0.010754755698144436,
+            0.003990580327808857,
+            0.004234685096889734,
+            -0.0033803186379373074,
+            0.023603858426213264,
+            0.03291521221399307,
+            0.02763689123094082,
+            -0.011653343215584755,
+            0.004921008367091417,
+            -0.01559439580887556,
+            -0.0011108530452474952,
+            0.01865101046860218,
+            -0.017816102132201195,
+            0.033877480775117874,
+            -0.029405765235424042,
+            0.02104252763092518,
+            -0.035122767090797424,
+            0.006141531281173229,
+            -0.012099100276827812,
+            -0.00404364662244916,
+            0.04833979532122612,
+            -0.017646290361881256,
+            -0.0019546060357242823,
+            -0.008101443760097027,
+            -0.014575524255633354,
+            -0.02158026583492756,
+            0.008766540326178074,
+            -0.04033033177256584,
+            -0.012367968447506428,
+            -0.028429346159100533,
+            0.003424540627747774,
+            0.019910449162125587,
+            -0.06277380883693695,
+            -0.06237758323550224,
+            0.012275987304747105,
+            -0.04918885603547096,
+            -0.026419905945658684,
+            0.02963218092918396,
+            -0.002984090941026807,
+            0.0012780115939676762,
+            -0.0024003623984754086,
+            0.0068101659417152405,
+            -0.029207652434706688,
+            0.00425944896414876,
+            -0.000887974863871932,
+            0.03271709755063057,
+            -0.021056678146123886,
+            -0.008122670464217663,
+            0.004878554958850145,
+            -0.06860402226448059,
+            -0.028344441205263138,
+            0.011271266266703606,
+            -0.009459939785301685,
+            -0.0066615804098546505,
+            -0.017915157601237297,
+            -0.004383270163089037,
+            0.0008950503543019295,
+            0.0032830305863171816,
+            0.05801907554268837,
+            -0.01983969286084175,
+            -0.053773775696754456,
+            0.03633975237607956,
+            0.00887267291545868,
+            0.029603879898786545,
+            -0.025556694716215134,
+            -0.017533080652356148,
+            -0.01855195313692093,
+            -0.020504789426922798,
+            -0.04123599827289581,
+            0.00404364662244916,
+            -0.005918653216212988,
+            0.009212297387421131,
+            -0.0009693430620245636,
+            0.02763689123094082,
+            0.029235953465104103,
+            0.023108573630452156,
+            0.012750045396387577,
+            0.01079013291746378,
+            -0.022372720763087273,
+            -0.021594416350126266,
+            0.0012222920777276158,
+            0.04683978855609894,
+            0.024636881425976753,
+            -0.02675952948629856,
+            -0.011504758149385452,
+            0.03953787684440613,
+            0.0419718474149704,
+            -0.01974063739180565,
+            -0.02276895008981228,
+            0.00764861237257719,
+            0.03266049548983574,
+            -0.04477374628186226,
+            0.005932804197072983,
+            0.03410389646887779,
+            0.014405711553990841,
+            -0.005356151144951582,
+            -0.010294848121702671,
+            0.022570835426449776,
+            0.03152841329574585,
+            -0.008724087849259377,
+            0.01876421831548214,
+            0.016910437494516373,
+            0.029349161311984062,
+            -0.0067889392375946045,
+            0.0005766529939137399,
+            -0.024735936895012856,
+            0.055188875645399094,
+            -0.03059444949030876,
+            0.02909444272518158,
+            0.010316074825823307,
+            -0.00593634182587266,
+            0.039056744426488876,
+            -0.016641568392515182,
+            -0.0431322306394577,
+            0.01913214474916458,
+            -0.006979977712035179,
+            -0.0055613406002521515,
+            -0.03444352000951767,
+            -0.007592008449137211,
+            -0.007507102098315954,
+            -0.006714646704494953,
+            -0.022556684911251068,
+            -0.02606613002717495,
+            0.004415110219269991,
+            0.00973588414490223,
+            -0.006484692916274071,
+            0.00698705343529582,
+            0.02092931978404522,
+            -0.013507124036550522,
+            0.02525952458381653,
+            -0.046047333627939224,
+            -0.0008366775000467896,
+            -0.0038596836384385824,
+            -0.025712355971336365,
+            -0.0345001220703125,
+            -0.0005770951975136995,
+            -0.0024251264985650778,
+            -0.024622729048132896,
+            -0.00914154201745987,
+            -0.008752389810979366,
+            -0.0023119186516851187,
+            -0.004938696976751089,
+            -0.055188875645399094,
+            0.008695785887539387,
+            0.010846736840903759,
+            -0.002713453024625778,
+            0.0055507272481918335,
+            -0.010613245889544487,
+            -0.03039633482694626,
+            0.03039633482694626,
+            0.04129260033369064,
+            0.033707667142152786,
+            0.014334957115352154,
+            -0.0074646491557359695,
+            -0.017165156081318855,
+            0.006060163024812937,
+            0.021721776574850082,
+            0.00902833417057991,
+            0.007846726104617119,
+            0.03877372294664383,
+            0.00011331850691931322,
+            -0.008349086157977581,
+            -0.009764186106622219,
+            0.028740668669342995,
+            -0.001085204305127263,
+            0.023603858426213264,
+            0.050802066922187805,
+            0.030424637719988823,
+            0.020589696243405342,
+            -0.01681138016283512,
+            -0.02010856196284294,
+            -0.00851889792829752,
+            -0.02855670638382435,
+            0.0364246591925621,
+            0.012127402238547802,
+            -0.005232329946011305,
+            -0.011540135368704796,
+            0.019061388447880745,
+            -0.033339742571115494,
+            -0.0037358624394983053,
+            0.00048422932741232216,
+            0.013924578204751015,
+            -0.03379257395863533,
+            0.0002480846014805138,
+            0.031613320112228394,
+            0.014957600273191929,
+            -0.016995344310998917,
+            -0.0188774261623621,
+            0.010549565777182579,
+            0.020462337881326675,
+            0.018948180601000786,
+            0.00037743354914709926,
+            0.00897173024713993,
+            -0.00022121983056422323,
+            -0.0005319889169186354,
+            0.01590571738779545,
+            -0.007952858693897724,
+            0.036254845559597015,
+            -0.003525366308167577,
+            -0.01325240544974804,
+            -0.0037500134203583,
+            -0.021438755095005035,
+            -0.024467067793011665,
+            0.043556760996580124,
+            0.023971782997250557,
+            0.04661337286233902,
+            -0.005660397466272116,
+            -0.004245297983288765,
+            0.025825563818216324,
+            0.008221727795898914,
+            -0.03288691118359566,
+            0.000265110022155568,
+            0.005281858611851931,
+            -0.02892463095486164,
+            0.016641568392515182,
+            -0.02476423978805542,
+            -0.016896286979317665,
+            -0.006771250627934933,
+            -0.0015557248843833804,
+            -0.028188779950141907,
+            0.01703779585659504,
+            0.0031309074256569147,
+            -0.0027099153958261013,
+            -0.039764292538166046,
+            0.006880920846015215,
+            0.020419884473085403,
+            0.015311375260353088,
+            0.0037358624394983053,
+            0.011525984853506088,
+            -0.016613267362117767,
+            0.002126186853274703,
+            0.03379257395863533,
+            0.004878554958850145,
+            0.02005195803940296,
+            0.0027647505048662424,
+            0.023773670196533203,
+            0.03633975237607956,
+            -0.01681138016283512,
+            -0.02189158834517002,
+            0.01519816741347313,
+            -0.02233026921749115,
+            0.029943503439426422,
+            0.004889168310910463,
+            -0.012283062562346458,
+            0.022033097222447395,
+            -0.009827865287661552,
+            -0.0027346794959157705,
+            0.01659911684691906,
+            -0.022740647196769714,
+            -0.017717044800519943,
+            0.03183973580598831,
+            -0.039452970027923584,
+            0.012856177985668182,
+            -0.008547200821340084,
+            0.015863263979554176,
+            -0.023066120222210884,
+            0.011172209866344929,
+            0.007100261282175779,
+            -0.01573590561747551,
+            0.010797208175063133,
+            0.009693430736660957,
+            0.00545874610543251,
+            0.0018926954362541437,
+            0.02179253101348877,
+            -0.0006049550138413906,
+            -0.011257115751504898,
+            -0.018155725672841072,
+            -0.004694592207670212,
+            -0.002614396158605814,
+            -0.016556663438677788,
+            0.01865101046860218,
+            -0.041066184639930725,
+            -0.013280707411468029,
+            -0.008879749104380608,
+            -0.013988257385790348,
+            -0.010011828504502773,
+            0.024184048175811768,
+            0.025669902563095093,
+            -0.002361447084695101,
+            -0.031217092648148537,
+            0.01790100708603859,
+            0.024198200553655624,
+            0.003654494183138013,
+            -0.023646311834454536,
+            0.0302831269800663,
+            -0.008936353027820587,
+            0.014971751719713211,
+            0.026136886328458786,
+            0.014094389975070953,
+            0.004595535341650248,
+            -0.010500037111341953,
+            -0.02114158496260643,
+            0.01481609046459198,
+            0.010507113300263882,
+            0.033537857234478,
+            0.014745336025953293,
+            0.02071705460548401,
+            0.01594817079603672,
+            -0.023320838809013367,
+            -0.011214662343263626,
+            -0.003482913365587592,
+            -0.022514231503009796,
+            0.03093407303094864,
+            0.0006129149114713073,
+            0.02114158496260643,
+            0.026618018746376038,
+            -0.03319823369383812,
+            -0.01187268365174532,
+            0.032688796520233154,
+            0.02779255248606205,
+            0.05108508840203285,
+            0.025316128507256508,
+            0.015396281145513058,
+            0.027226513251662254,
+            -0.03585861995816231,
+            -0.036368053406476974,
+            0.001548649393953383,
+            0.04256619140505791,
+            -0.008426916785538197,
+            0.00967928022146225,
+            0.03900013864040375,
+            -0.006781863979995251,
+            -0.01611798256635666,
+            0.019457615911960602,
+            0.0026710000820457935,
+            0.002448122017085552,
+            -0.0033113325480371714,
+            -0.01335146278142929,
+            0.03175482898950577,
+            -0.0051474240608513355,
+            0.013570803217589855,
+            -0.019273653626441956,
+            0.027113303542137146,
+            0.0001799829478841275,
+            -0.003229964291676879,
+            0.028344441205263138,
+            -0.002660386962816119,
+            -0.051990751177072525,
+            0.024212351068854332,
+            0.019811391830444336,
+            -0.011561362072825432,
+            -0.0036332677118480206,
+            -0.003792466362938285,
+            0.04567940905690193,
+            0.034698236733675,
+            0.029009537771344185,
+            -0.0020501252729445696,
+            0.0168679840862751,
+            0.018792519345879555,
+            0.0053667644970119,
+            0.014278353191912174,
+            -0.0017025414854288101,
+            -0.007853801362216473,
+            0.021056678146123886,
+            -0.020787810906767845,
+            0.03226426616311073,
+            -0.016570813953876495,
+            -0.020122714340686798,
+            0.008738238364458084,
+            -0.011172209866344929,
+            -0.0009083169279620051,
+            -0.006311343517154455,
+            -0.0008813415770418942,
+            -0.0029664020985364914,
+            0.016938740387558937,
+            -0.004446949809789658,
+            0.04661337286233902,
+            -0.013931653462350368,
+            -0.008808993734419346,
+            0.006972902454435825,
+            -0.025981225073337555,
+            0.0013346156338229775,
+            0.0028726516757160425,
+            0.027000095695257187,
+            -0.01989629678428173,
+            -0.00970758218318224,
+            0.01606137864291668,
+            0.06039644405245781,
+            -0.008639181964099407,
+            -0.0026267783250659704,
+            0.027226513251662254,
+            -0.002787745790556073,
+            -0.03551899641752243,
+            0.012919857166707516,
+            -0.04494355618953705,
+            0.017589686438441277,
+            0.026349151507019997,
+            -0.04986810311675072,
+            -0.022089701145887375,
+            -0.02347649820148945,
+            -0.02043403498828411,
+            0.023434046655893326,
+            -0.009976450353860855,
+            0.013867974281311035,
+            -0.017801949754357338,
+            -0.009014182724058628,
+            0.020518941804766655,
+            0.009176919236779213,
+            0.03735862299799919,
+            -0.027551984414458275,
+            -0.016783079132437706,
+            -0.027820853516459465,
+            -0.04304732382297516,
+            0.009558996185660362,
+            -0.0278916098177433,
+            0.0027099153958261013,
+            -0.0016291331266984344,
+            -0.040302030742168427,
+            -0.037981268018484116,
+            0.005278320983052254,
+            -0.017985913902521133,
+            0.01516986545175314,
+            0.020915169268846512,
+            0.02731141820549965,
+            -0.0020253609400242567,
+            -0.030198222026228905,
+            0.029235953465104103,
+            -0.007344366051256657,
+            0.03223596513271332,
+            -0.00689507182687521,
+            0.005066055804491043,
+            -0.0070436573587358,
+            0.031160488724708557,
+            -0.023009516298770905,
+            0.018792519345879555,
+            -0.026801982894539833,
+            -0.004086099565029144,
+            -0.018990634009242058,
+            -0.01774534583091736,
+            0.004224071744829416,
+            0.004521242808550596,
+            7.888073741924018e-05,
+            0.0048750173300504684,
+            0.005777143407613039,
+            0.024806693196296692,
+            0.0021421066485345364,
+            0.02589631825685501,
+            -0.027877457439899445,
+            -0.025316128507256508,
+            0.01719345711171627,
+            -0.02077365852892399,
+            0.005678086541593075,
+            -0.006261814851313829,
+            -0.0302831269800663,
+            -0.01202834490686655,
+            0.023391593247652054,
+            0.02162271924316883,
+            -0.005490585695952177,
+            0.002644466934725642,
+            0.02206140011548996,
+            -0.026023678481578827,
+            0.00437973253428936,
+            -0.03781145438551903,
+            -0.018396291881799698,
+            0.029773691669106483,
+            -0.03195294365286827,
+            0.020745357498526573,
+            0.003185742534697056,
+            0.03483974561095238,
+            -0.017292514443397522,
+            -0.01947176828980446,
+            0.01654251106083393,
+            -0.01008258294314146,
+            0.004142703488469124,
+            0.01606137864291668,
+            0.01616043597459793,
+            -0.0015070808585733175,
+            0.05380208045244217,
+            0.0008167776977643371,
+            0.006909222807735205,
+            -0.049075648188591,
+            0.004496478475630283,
+            -0.044858649373054504,
+            0.0002768288250081241,
+            -0.013337312266230583,
+            -0.02514631673693657,
+            0.0014027173165231943,
+            0.001001182827167213,
+            0.001963450340554118,
+            -0.01913214474916458,
+            -0.01989629678428173,
+            0.011596739292144775,
+            -0.015863263979554176,
+            0.004928083624690771,
+            -0.06418890506029129,
+            -0.01855195313692093,
+            -0.004974074196070433,
+            0.0043337419629096985,
+            0.0007699025445617735,
+            0.03342464938759804,
+            -0.01478778850287199,
+            0.010117961093783379,
+            -0.01549533847719431,
+            -0.026363302022218704,
+            0.04010391607880592,
+            0.018750067800283432,
+            -0.023773670196533203,
+            0.024693485349416733,
+            0.012106175534427166,
+            0.03099067695438862,
+            -0.0318114347755909,
+            0.020349130034446716,
+            0.018523650243878365,
+            -0.0029823218937963247,
+            0.006551910191774368,
+            0.008575502783060074,
+            0.01832553744316101,
+            -0.015764206647872925,
+            -0.0003632825391832739,
+            0.011419852264225483,
+            0.010033054277300835,
+            0.031924642622470856,
+            0.01403071079403162,
+            -0.03183973580598831,
+            0.019047237932682037,
+            0.01616043597459793,
+            0.02298121340572834,
+            0.033820874989032745,
+            -0.027481229975819588,
+            0.038575608283281326,
+            -0.010386829264461994,
+            -0.024042539298534393,
+            0.020801961421966553,
+            -0.032688796520233154,
+            -0.006679269019514322,
+            0.030735958367586136,
+            0.04398128762841225,
+            -0.0015566092915832996,
+            -0.044632233679294586,
+            0.01535382866859436,
+            0.01316749956458807,
+            0.0027806703001260757,
+            0.028811423107981682,
+            0.011264191009104252,
+            0.021820832043886185,
+            0.025273675099015236,
+            0.011419852264225483,
+            -0.0037252490874379873,
+            0.024962352588772774,
+            -0.008101443760097027,
+            -0.004446949809789658,
+            -0.0007035697344690561,
+            0.03206615149974823,
+            0.014702882617712021,
+            -0.017603836953639984,
+            -0.036962397396564484,
+            -0.007620310410857201,
+            0.009452863596379757,
+            -0.007903330028057098,
+            -0.027127455919981003,
+            -0.03950957581400871,
+            -0.026787832379341125,
+            -0.005667473189532757,
+            0.01505665760487318,
+            -0.06724552065134048,
+            -0.007620310410857201,
+            -0.012233533896505833,
+            0.017717044800519943,
+            0.0022906921803951263,
+            0.0181132722645998,
+            0.001277127186767757,
+            -0.009403335861861706,
+            0.004209920763969421,
+            0.050802066922187805,
+            -0.03059444949030876,
+            0.014242975041270256,
+            0.011292492970824242,
+            0.0032105068676173687,
+            0.002787745790556073,
+            0.038688816130161285,
+            -0.010988246649503708,
+            -0.030424637719988823,
+            0.011575513519346714,
+            -0.017292514443397522,
+            0.003383856499567628,
+            0.05006621778011322,
+            -0.009481165558099747,
+            -0.027495380491018295,
+            0.020915169268846512,
+            -0.009148617275059223,
+            -0.015523640438914299,
+            -0.0027258351910859346,
+            -0.006035399157553911,
+            0.0006283925613388419,
+            -0.03832089155912399,
+            -0.015750056132674217,
+            0.03212275728583336,
+            0.008617955259978771,
+            0.014603826217353344,
+            0.025344429537653923,
+            0.010209942236542702,
+            -0.030481241643428802,
+            0.027481229975819588,
+            -0.0005678086308762431,
+            0.005221716593950987,
+            0.013323160819709301,
+            0.03424540534615517,
+            -0.014264201745390892,
+            -0.020264223217964172,
+            -0.040245428681373596,
+            -0.013853822834789753,
+            -0.0463869571685791,
+            0.001943992800079286,
+            0.006919836159795523,
+            -0.012919857166707516,
+            -0.04366996884346008,
+            -0.0329718142747879,
+            0.021127434447407722,
+            0.03178313374519348,
+            -0.017023645341396332,
+            0.0025294902734458447,
+            -0.008483520708978176,
+            -0.007882103323936462,
+            -0.0017520699184387922,
+            -0.013217028230428696,
+            -0.006378560326993465,
+            -0.006229975260794163,
+            0.015452885068953037,
+            -0.004652139265090227,
+            -0.0036934094969183207,
+            -0.0498398020863533,
+            -0.018778368830680847,
+            0.004082561936229467,
+            -0.008221727795898914,
+            0.007323139347136021,
+            -0.006955213379114866,
+            -0.002727604005485773,
+            -0.012474101036787033,
+            -0.03829259052872658,
+            0.024184048175811768,
+            0.017278363928198814,
+            0.0035501306410878897,
+            -0.008922201581299305,
+            -0.018367990851402283,
+            0.03294351324439049,
+            0.012523629702627659,
+            -0.017547233030200005,
+            -0.018580254167318344,
+            0.025981225073337555,
+            0.026957644149661064,
+            0.008122670464217663,
+            -0.006137993652373552,
+            0.029886899515986443,
+            0.005695775151252747,
+            -0.022938761860132217,
+            -0.003983504604548216,
+            -0.005055442452430725,
+            -0.0038844477385282516,
+            0.020561393350362778,
+            -0.010556641966104507,
+            0.0300001073628664,
+            0.038717120885849,
+            0.012870329432189465,
+            0.0039198254235088825,
+            -0.0033714743331074715,
+            -0.008922201581299305,
+            0.015976471826434135,
+            0.006481155287474394,
+            0.008525974117219448,
+            0.014759486541152,
+            -0.016358548775315285,
+            -0.020405733957886696,
+            0.006215824279934168,
+            -0.007107336539775133,
+            0.020122714340686798,
+            0.01708024926483631,
+            0.016429303213953972,
+            0.0007610581233166158,
+            -0.028287837281823158,
+            -0.0018024828750640154,
+            0.02411329373717308,
+            0.02010856196284294,
+            -0.004655676893889904,
+            -0.023264234885573387,
+            0.014398636296391487,
+            -0.015806660056114197,
+            -0.009969375096261501,
+            0.0033431723713874817,
+            0.006229975260794163,
+            -0.003083147807046771,
+            0.0571134127676487,
+            -0.020193468779325485,
+            -0.011419852264225483,
+            -0.010287772864103317,
+            -0.010867963545024395,
+            0.0012072566896677017,
+            -0.01828308403491974,
+            0.008964654989540577,
+            0.025613298639655113,
+            -0.009764186106622219,
+            0.03568880632519722,
+            0.0228114016354084,
+            0.006912760436534882,
+            0.005260631907731295,
+            -0.027070851996541023,
+            0.012304289266467094,
+            -0.016047226265072823,
+            0.005660397466272116,
+            0.018467046320438385,
+            0.017320817336440086,
+            0.0076132346875965595,
+            0.026306698098778725,
+            0.019386861473321915,
+            0.016684021800756454,
+            -0.022783100605010986,
+            0.025457637384533882,
+            -0.007832575589418411,
+            0.027877457439899445,
+            -0.01611798256635666,
+            -0.002253545681014657,
+            -0.012990612536668777,
+            0.008681634441018105,
+            0.002416282193735242,
+            -0.0019351483788341284,
+            0.00036129256477579474,
+            0.016783079132437706,
+            -0.014589674770832062,
+            0.01062032114714384,
+            0.013316085562109947,
+            -0.027919910848140717,
+            -0.010372678749263287,
+            0.016344398260116577,
+            -0.00668988237157464,
+            0.006877383217215538,
+            -0.011865608394145966,
+            -0.0017202302115038037,
+            0.011009473353624344,
+            -0.0436982698738575,
+            0.0104929618537426,
+            0.006279503460973501,
+            -0.015155714936554432,
+            0.014801939949393272,
+            -0.009799563325941563,
+            -0.008235878311097622,
+            -0.013592029921710491,
+            0.003792466362938285,
+            -0.00030225637601688504,
+            -0.0018290160223841667,
+            -0.00022995365725364536,
+            0.00389859895221889,
+            0.010047205723822117,
+            -0.0045990729704499245,
+            -0.010450509376823902,
+            -0.0028638073708862066,
+            -0.02265574224293232,
+            -0.0010047205723822117,
+            0.018735915422439575,
+            -0.004591997712850571,
+            0.0005222601466812193,
+            0.0145189194008708,
+            -0.005830209702253342,
+            0.010556641966104507,
+            -0.005221716593950987,
+            -0.008320784196257591,
+            -0.010768906213343143,
+            -0.033764272928237915,
+            0.0026710000820457935,
+            -0.015254771336913109,
+            0.008865597657859325,
+            -0.012544856406748295,
+            0.04027372971177101,
+            0.01022409275174141,
+            -0.00837738811969757,
+            -0.002796590095385909,
+            -0.0228114016354084,
+            -0.004475251771509647,
+            -0.016938740387558937,
+            -0.01184438169002533,
+            0.008342010900378227,
+            0.012629762291908264,
+            0.005614406894892454,
+            -0.018410442396998405,
+            0.003665107535198331,
+            0.0030955299735069275,
+            -0.005575491581112146,
+            -0.008370312862098217,
+            0.010853812098503113,
+            -0.0006248548161238432,
+            -0.01800006441771984,
+            -0.01957082375884056,
+            -0.027424626052379608,
+            0.0150000536814332,
+            0.014589674770832062,
+            -0.01011088490486145,
+            -0.022754797711968422,
+            -0.007471724878996611,
+            0.007107336539775133,
+            -0.02064630016684532,
+            0.0006779210525564849,
+            0.006891534198075533,
+            0.008717012591660023,
+            0.004117939155548811,
+            0.009636826813220978,
+            0.0007699025445617735,
+            0.0121698547154665,
+            0.018566103652119637,
+            0.03271709755063057,
+            -0.0396793857216835,
+            -0.009537769481539726,
+            0.0057028504088521,
+            -0.001465512323193252,
+            -0.013853822834789753,
+            0.009672204032540321,
+            0.0024375086650252342,
+            -0.0026426981203258038,
+            0.024424616247415543,
+            -0.008044839836657047,
+            -0.01505665760487318,
+            0.01184438169002533,
+            0.00887267291545868,
+            -0.019075538963079453,
+            0.02746707946062088,
+            0.006169833242893219,
+            -0.01011088490486145,
+            0.02259913645684719,
+            -0.012360893189907074,
+            0.012658064253628254,
+            -0.005522425286471844,
+            0.023660462349653244,
+            -0.019004784524440765,
+            0.0006306036957539618,
+            0.015665151178836823,
+            0.03883032873272896,
+            -0.00437973253428936,
+            -0.004567233379930258,
+            -0.03421710431575775,
+            0.005115584470331669,
+            0.028853876516222954,
+            0.02666047215461731,
+            -0.0012346742441877723,
+            -0.005596718285232782,
+            -0.0019864458590745926,
+            0.010768906213343143,
+            0.011589664034545422,
+            0.017561383545398712,
+            -0.029547275975346565,
+            0.00022011429246049374,
+            -0.016938740387558937,
+            0.025500090792775154,
+            0.014632128179073334,
+            0.006951675750315189,
+            0.04072656109929085,
+            0.0027789014857262373,
+            0.003841994795948267,
+            0.027452928945422173,
+            -0.00779719790443778,
+            0.003546592779457569,
+            -0.015566092915832996,
+            0.0003175129240844399,
+            0.022853855043649673,
+            0.0016370931407436728,
+            0.0020784272346645594,
+            0.013337312266230583,
+            0.029320860281586647,
+            -0.022471778094768524,
+            -0.023929331451654434,
+            0.003482913365587592,
+            0.007185167167335749,
+            -0.010570792481303215,
+            0.0038738346192985773,
+            0.006275965832173824,
+            -0.006007096730172634,
+            0.008900974877178669,
+            0.006332569755613804,
+            -0.0003166284877806902,
+            0.0039198254235088825,
+            -0.00308845448307693,
+            0.012962310574948788,
+            -0.02702839858829975,
+            0.036849189549684525,
+            0.004439874552190304,
+            0.010075507685542107,
+            0.015268922783434391,
+            0.001627364312298596,
+            0.004917470272630453,
+            0.005869124550372362,
+            0.010188715532422066,
+            0.018636859953403473,
+            -0.00956607237458229,
+            -0.027481229975819588,
+            -0.03166992589831352,
+            0.020363280549645424,
+            -0.01173117384314537,
+            0.0035872769076377153,
+            0.0074646491557359695,
+            -0.01828308403491974,
+            -0.007507102098315954,
+            -0.009976450353860855,
+            0.015764206647872925,
+            -0.03189634159207344,
+            0.012990612536668777,
+            -0.0043231286108493805,
+            -0.028669914230704308,
+            0.00851889792829752,
+            -0.010181640274822712,
+            -0.0045884596183896065,
+            -0.0035094465129077435,
+            -0.029264256358146667,
+            -0.007931631989777088,
+            0.03908504545688629,
+            -0.02141045406460762,
+            0.024184048175811768,
+            0.008257105015218258,
+            -0.00014449491573031992,
+            -0.0040365708991885185,
+            -0.013563727959990501,
+            0.005331386812031269,
+            -0.01778779923915863,
+            0.025457637384533882,
+            0.010294848121702671,
+            -0.013924578204751015,
+            0.0322076641023159,
+            -0.011179285123944283,
+            0.027438776567578316,
+            -0.018580254167318344,
+            0.036198243498802185,
+            0.007422196213155985,
+            0.00674648629501462,
+            0.01974063739180565,
+            -0.004025958012789488,
+            0.008228803053498268,
+            0.04559450224041939,
+            -0.018523650243878365,
+            0.02460857853293419,
+            0.020349130034446716,
+            -0.00022685811563860625,
+            0.017915157601237297,
+            0.0028390432707965374,
+            -0.023165177553892136,
+            0.009495317004621029,
+            -0.01135617308318615,
+            -0.009955224581062794,
+            0.01583496294915676,
+            -0.019316107034683228,
+            -0.018721764907240868,
+            -0.025613298639655113,
+            0.0029416377656161785,
+            -0.003350247861817479,
+            -0.0077618202194571495,
+            -0.0019616815261542797,
+            0.01454722136259079,
+            0.008724087849259377,
+            0.025669902563095093,
+            0.0008326975512318313,
+            0.011971740983426571,
+            0.01511326152831316,
+            -0.00035576484515331686,
+            0.02313687466084957,
+            -0.03229256719350815,
+            0.01246702577918768,
+            -0.004517704714089632,
+            -0.00509789539501071,
+            0.023858575150370598,
+            0.0003747802402358502,
+            -0.0004877670726273209,
+            -0.0024888061452656984,
+            0.0005496777012012899,
+            -0.014589674770832062,
+            -0.0057099261321127415,
+            0.000907432462554425,
+            -0.005175726022571325,
+            0.04299071803689003,
+            -0.00701535539701581,
+            -0.004772422835230827,
+            -0.039169952273368835,
+            -0.01918874867260456,
+            0.00623351288959384,
+            -0.01583496294915676,
+            -0.018778368830680847,
+            -0.006307805422693491,
+            0.005801907740533352,
+            -0.04893413558602333,
+            0.008342010900378227,
+            0.03336804360151291,
+            -0.012219383381307125,
+            0.00259493850171566,
+            -0.010981171391904354,
+            -0.006969364359974861,
+            -0.0029115669894963503,
+            -7.766463386360556e-05,
+            -0.002412744564935565,
+            -0.020830262452363968,
+            -0.005617944523692131,
+            0.003364398842677474,
+            -0.0008795727044343948,
+            -0.025047259405255318,
+            0.0006177793256938457,
+            0.018849123269319534,
+            -0.003979966975748539,
+            -0.0011152751976624131,
+            -0.001825478277169168,
+            0.006399787031114101,
+            -0.01508495956659317,
+            0.008009462617337704,
+            -0.0076344613917171955,
+            0.003841994795948267,
+            -0.0030495391692966223,
+            -0.002214630600064993,
+            0.02844349853694439,
+            0.021594416350126266,
+            -0.04808507859706879,
+            -0.01405901275575161,
+            0.023009516298770905,
+            0.007280686404556036,
+            -0.004740582779049873,
+            0.01178777776658535,
+            -0.031613320112228394,
+            -0.00028943203506059945,
+            -0.026787832379341125,
+            0.02200479619204998,
+            -0.009205221198499203,
+            -0.007386818993836641,
+            0.016967041417956352,
+            -0.026632171124219894,
+            0.016344398260116577,
+            0.00794578343629837,
+            -0.00030181417241692543,
+            -0.000493515923153609,
+            0.0063856360502541065,
+            0.0008972614887170494,
+            -0.01659911684691906,
+            0.006442239973694086,
+            -0.0048502529971301556,
+            0.007959933951497078,
+            -0.005087282508611679,
+            -0.05527378246188164,
+            -0.009933997876942158,
+            -0.005497660953551531,
+            -0.008936353027820587,
+            -0.012820800766348839,
+            0.008136821910738945,
+            0.02827368676662445,
+            0.01606137864291668,
+            -0.009863242506980896,
+            -4.8312376748071983e-05,
+            0.033000119030475616,
+            -0.0008300442132167518,
+            0.009955224581062794,
+            -0.011349096894264221,
+            0.00015709814033471048,
+            -0.0006911876262165606,
+            -0.014660430140793324,
+            -0.019061388447880745,
+            0.024042539298534393,
+            -0.028344441205263138,
+            0.02378782071173191,
+            -0.01843874529004097,
+            -0.0130825936794281,
+            0.01757553406059742,
+            -0.032575588673353195,
+            -0.005108508747071028,
+            0.007450498174875975,
+            0.0090566361322999,
+            0.021608568727970123,
+            0.0032794929575175047,
+            0.020589696243405342,
+            -0.03716050833463669,
+            -0.009134466759860516,
+            -0.010160413570702076,
+            -0.00794578343629837,
+            -0.010231168940663338,
+            -0.011596739292144775,
+            0.021863285452127457,
+            -0.015240620821714401,
+            -0.016952890902757645,
+            0.007054270710796118,
+            0.0030212372075766325,
+            -0.011313719674944878,
+            0.015750056132674217,
+            0.00017810352437663823,
+            0.004712280817329884,
+            0.013011839240789413,
+            -0.0008923970744945109,
+            0.04086806997656822,
+            0.0253019779920578,
+            -0.0030212372075766325,
+            -0.008568426594138145,
+            0.021438755095005035,
+            0.0004285097820684314,
+            0.0150000536814332,
+            0.009375033900141716,
+            0.023264234885573387,
+            -0.02092931978404522,
+            0.009997677057981491,
+            -0.005501199048012495,
+            0.014603826217353344,
+            -0.009495317004621029,
+            -0.015339677222073078,
+            0.00479718716815114,
+            -0.01577835902571678,
+            0.010153338313102722,
+            -0.010860888287425041,
+            0.008221727795898914,
+            0.01913214474916458,
+            0.002467579673975706,
+            -0.00941041111946106,
+            0.01146230474114418,
+            0.02075950801372528,
+            -0.025641601532697678,
+            0.008851447142660618,
+            -0.0031998935155570507,
+            -0.0021916350815445185,
+            0.021127434447407722,
+            0.010301923379302025,
+            -0.01594817079603672,
+            0.01573590561747551,
+            -0.008879749104380608,
+            -0.0015053119277581573,
+            -0.01882082223892212,
+            -0.03291521221399307,
+            -0.0015283073298633099,
+            0.012396270409226418,
+            0.004800724796950817,
+            0.0150000536814332,
+            0.011264191009104252,
+            -0.0008950503543019295,
+            -0.006601438857614994,
+            0.0008433107868768275,
+            -0.013485897332429886,
+            -0.004963461309671402,
+            0.010231168940663338,
+            0.017051948234438896,
+            -0.0037217114586383104,
+            0.015650998800992966,
+            0.0088160689920187,
+            0.006137993652373552,
+            -0.0010480580385774374,
+            0.0036138100549578667,
+            -0.002060738392174244,
+            -0.007783046457916498,
+            0.019938750192523003,
+            0.019075538963079453,
+            -0.018509499728679657,
+            -0.0024870370980352163,
+            -0.02141045406460762,
+            0.025882167741656303,
+            0.022188758477568626,
+            0.011165134608745575,
+            -0.04222656786441803,
+            0.00012050455552525818,
+            0.00752832880243659,
+            -0.0013744152383878827,
+            -0.031103884801268578,
+            0.008172199130058289,
+            0.005221716593950987,
+            0.004330204334110022,
+            0.020844414830207825,
+            -0.018424594774842262,
+            0.010853812098503113,
+            0.0070861103013157845,
+            -0.0005766529939137399,
+            0.01019579079002142,
+            -0.0073372903279960155,
+            0.008688710629940033,
+            0.038717120885849,
+            -0.018410442396998405,
+            0.005303084850311279,
+            0.0062370505183935165,
+            0.023320838809013367,
+            -0.011228813789784908,
+            -0.00031485961517319083,
+            0.00800238735973835,
+            0.021608568727970123,
+            -0.012934008613228798,
+            0.005883275531232357,
+            0.067075714468956,
+            -0.012912781909108162,
+            0.031613320112228394,
+            0.020533092319965363,
+            -0.004970536567270756,
+            -0.0104929618537426,
+            -0.012445799075067043,
+            -0.0181132722645998,
+            0.021283095702528954,
+            0.01175947580486536,
+            0.003419233951717615,
+            0.01941516436636448,
+            -0.01610383205115795,
+            0.006636816076934338,
+            0.017547233030200005,
+            0.012601460330188274,
+            -0.007379743270576,
+            -0.0036173479165881872,
+            0.019004784524440765,
+            0.018509499728679657,
+            -0.0030849166214466095,
+            0.00019424450874794275,
+            0.01508495956659317,
+            -0.022075550630688667,
+            -0.012530704960227013,
+            0.01644345559179783,
+            -0.0037358624394983053,
+            0.017773648723959923,
+            -0.02227366529405117,
+            0.024198200553655624,
+            -0.004531855694949627,
+            -0.017179306596517563,
+            0.004698129836469889,
+            -0.007514177821576595,
+            0.022033097222447395,
+            -0.001578720286488533,
+            0.004857328720390797,
+            0.000821199850179255,
+            -0.0018015984678640962,
+            0.01349297259002924,
+            0.002826661104336381,
+            -0.0005120891146361828,
+            0.010507113300263882,
+            -0.0009702275274321437,
+            -0.0013496510218828917,
+            0.024509521201252937,
+            1.9374700059415773e-05,
+            0.005989408120512962,
+            0.0032936439383774996,
+            -0.009261825121939182,
+            -0.004429261200129986,
+            -0.004443412180989981,
+            0.0011842612875625491,
+            0.003877372248098254,
+            -0.007029506377875805,
+            -0.02384442463517189,
+            0.022471778094768524,
+            -0.008540124632418156,
+            0.01175947580486536,
+            0.011349096894264221,
+            0.003665107535198331,
+            0.006824316922575235,
+            -0.009771261364221573,
+            -0.017320817336440086,
+            0.0006633278680965304,
+            -0.01617458648979664,
+            0.006912760436534882,
+            -0.001656550681218505,
+            0.0020235921256244183,
+            0.009445788338780403,
+            -0.002570174168795347,
+            0.0015123875346034765,
+            0.011306644417345524,
+            0.011497682891786098,
+            0.003254728624597192,
+            0.004577846731990576,
+            -0.018721764907240868,
+            -0.020915169268846512,
+            0.005288933869451284,
+            0.013924578204751015,
+            0.004273600410670042,
+            0.0015309606678783894,
+            0.017603836953639984,
+            -0.006166295614093542,
+            -0.022967062890529633,
+            -0.010117961093783379,
+            -0.01584911346435547,
+            0.039396367967128754,
+            0.01800006441771984,
+            -0.02525952458381653,
+            -0.000530220044311136,
+            -0.009728807955980301,
+            0.010450509376823902,
+            -0.00404364662244916,
+            0.014575524255633354,
+            0.005476434715092182,
+            -0.02426895499229431,
+            -0.016797229647636414,
+            -0.010662773624062538,
+            -0.005257094278931618,
+            0.020080260932445526,
+            0.027014248073101044,
+            -0.004956385586410761,
+            0.0061026159673929214,
+            0.019584976136684418,
+            -0.010938717983663082,
+            0.009849091991782188,
+            0.020957622677087784,
+            0.0031698227394372225,
+            -0.007570781745016575,
+            -0.004553082399070263,
+            -0.016514210030436516,
+            0.024467067793011665,
+            0.014250051230192184,
+            -0.005104971118271351,
+            -0.02465103194117546,
+            0.004110863897949457,
+            -0.02428310550749302,
+            -0.01616043597459793,
+            -0.007301913108676672,
+            -0.014143918640911579,
+            -0.004931621253490448,
+            0.0027594438288360834,
+            -0.004836102016270161,
+            0.011278342455625534,
+            -0.0031592093873769045,
+            -0.004531855694949627,
+            -0.0041851564310491085,
+            -0.025584997609257698,
+            -0.02524537406861782,
+            0.007691065315157175,
+            -0.003005317412316799,
+            0.005596718285232782,
+            0.000650945701636374,
+            -0.013400991447269917,
+            -0.011589664034545422,
+            0.00970758218318224,
+            -0.007988235913217068,
+            0.012813725508749485,
+            0.0031149876303970814,
+            -0.009183995425701141,
+            0.0017193458043038845,
+            -0.0012311364989727736,
+            -0.0059752571396529675,
+            0.019924599677324295,
+            0.00916984397917986,
+            0.011724098585546017,
+            -0.03118879161775112,
+            -0.0021633331198245287,
+            0.008349086157977581,
+            0.01828308403491974,
+            -0.02244347706437111,
+            -0.008313708938658237,
+            0.005179263651371002,
+            -0.011228813789784908,
+            -0.013507124036550522,
+            -0.028952933847904205,
+            -0.0030460015404969454,
+            8.175515540642664e-05,
+            -0.018636859953403473,
+            -0.029886899515986443,
+            0.0009038947173394263,
+            -0.007860877551138401,
+            0.016882136464118958,
+            -0.006643891800194979,
+            0.01790100708603859,
+            0.00870286114513874,
+            0.0036173479165881872,
+            0.011554286815226078,
+            0.0025082635693252087,
+            -0.013266556896269321,
+            0.008426916785538197,
+            -0.02671707607805729,
+            -0.01516986545175314,
+            0.009792488068342209,
+            0.014214673079550266,
+            -0.0026126273442059755,
+            5.074458022136241e-05,
+            0.0058372849598526955,
+            0.024254804477095604,
+            0.013584954664111137,
+            0.012474101036787033,
+            0.003042463678866625,
+            -0.005303084850311279,
+            0.005568416323512793,
+            0.0227123461663723,
+            0.008603804744780064,
+            0.004011806566268206,
+            0.007592008449137211,
+            0.00389859895221889,
+            0.0009932229295372963,
+            -0.005943417549133301,
+            -0.00843399204313755,
+            -0.021537812426686287,
+            -0.008908051066100597,
+            0.019938750192523003,
+            -0.010422207415103912,
+            -0.03005671128630638,
+            0.0006023016758263111,
+            -0.0032777239102870226,
+            0.007988235913217068,
+            0.02595292218029499,
+            -0.014172220602631569,
+            0.004970536567270756,
+            0.005536576267331839,
+            -0.007910405285656452,
+            -0.00614860700443387,
+            0.022967062890529633,
+            0.02639160305261612,
+            -0.0034404604230076075,
+            -0.04800017178058624,
+            -0.01671232469379902,
+            0.004305440001189709,
+            -0.007768895477056503,
+            -0.034047290682792664,
+            -0.015424583107233047,
+            0.002757675014436245,
+            -0.03206615149974823,
+            0.010068432427942753,
+            -0.012948159128427505,
+            -0.01214155275374651,
+            0.020264223217964172,
+            0.03223596513271332,
+            0.005154499784111977,
+            0.00014847487909719348,
+            -0.0214670579880476,
+            -0.0021686397958546877,
+            -0.021382151171565056,
+            0.010174565017223358,
+            -0.008801918476819992,
+            -0.010584943927824497,
+            0.0059611061587929726,
+            -0.012191081419587135,
+            0.013266556896269321,
+            0.010797208175063133,
+            0.024778390303254128,
+            0.01584911346435547,
+            -0.01876421831548214,
+            -0.0024746551644057035,
+            -0.01108730398118496,
+            -0.0016450530383735895,
+            -0.009820790030062199,
+            0.006360871717333794,
+            -0.0031592093873769045,
+            0.011264191009104252,
+            0.015184016898274422,
+            0.004432798828929663,
+            0.00046786724124103785,
+            0.038151077926158905,
+            -2.4487851987942122e-05,
+            -0.0003418349369894713,
+            0.006803090218454599,
+            -0.015750056132674217,
+            0.008837295696139336,
+            0.012191081419587135,
+            0.005051904823631048,
+            -0.007698140572756529,
+            0.006739411037415266,
+            0.03099067695438862,
+            -0.01516986545175314,
+            -0.009014182724058628,
+            -0.023504801094532013,
+            0.007549555506557226,
+            -0.0004656561359297484,
+            0.029971804469823837,
+            0.020533092319965363,
+            -0.027509532868862152,
+            -0.016018925234675407,
+            -0.02179253101348877,
+            0.01267929095774889,
+            -0.011497682891786098,
+            -0.003704022616147995,
+            0.012240610085427761,
+            0.02367461286485195,
+            0.0018838511314243078,
+            -0.014455240219831467,
+            -0.008950503543019295,
+            0.008943428285419941,
+            0.014617976732552052,
+            0.003497064346447587,
+            0.007570781745016575,
+            0.00743634719401598,
+            0.008886824361979961,
+            -0.014801939949393272,
+            -0.012254760600626469,
+            0.00767691433429718,
+            -0.010202866978943348,
+            0.01794346049427986,
+            -0.009403335861861706,
+            0.016768928617239,
+            -0.004645063541829586,
+            -0.011964665725827217,
+            -0.0320095494389534,
+            -0.002292460994794965,
+            0.005037753842771053,
+            0.004627374932169914,
+            0.0034705311991274357,
+            -0.0022217060904949903,
+            -0.013160424306988716,
+            0.004673365969210863,
+            0.005073131527751684,
+            1.875283123808913e-05,
+            -0.008313708938658237,
+            -0.00479718716815114,
+            0.004252373706549406,
+            -0.001031253719702363,
+            -0.0073372903279960155,
+            -0.0059858704917132854,
+            0.018481198698282242,
+            0.0066509670577943325,
+            0.00800238735973835,
+            0.03234917297959328,
+            0.02974538877606392,
+            -0.0016353242099285126,
+            -0.004606148693710566,
+            -0.00023304918431676924,
+            -0.01589156687259674,
+            -0.0057205394841730595,
+            -0.023320838809013367,
+            0.022259512916207314,
+            -0.009665128774940968,
+            0.021820832043886185,
+            0.019457615911960602,
+            0.0039304387755692005,
+            -0.008391539566218853,
+            -0.0011586126638576388,
+            -0.007308988366276026,
+            -0.003560743760317564,
+            0.016641568392515182,
+            0.006742948666214943,
+            0.01226891204714775,
+            0.00030933189555071294,
+            -0.028401045128703117,
+            0.026405755430459976,
+            0.000579748535528779,
+            0.0034015451092272997,
+            0.008908051066100597,
+            0.014129767194390297,
+            0.013209952972829342,
+            -0.01235381793230772,
+            0.01768874190747738,
+            -0.03280200436711311,
+            -0.0033484790474176407,
+            -0.016500059515237808,
+            0.002076658420264721,
+            -0.0018520113080739975,
+            -0.0008145665633492172,
+            -0.05354736000299454,
+            0.0003842879377771169,
+            0.005313698202371597,
+            -0.015339677222073078,
+            0.029349161311984062,
+            -0.020589696243405342,
+            -0.017165156081318855,
+            -0.032858606427907944,
+            -0.010238244198262691,
+            0.018099121749401093,
+            0.005617944523692131,
+            0.00035178486723452806,
+            -0.02660386823117733,
+            0.015750056132674217,
+            -0.0047476585023105145,
+            -0.00743634719401598,
+            -0.029858596622943878,
+            -0.010938717983663082,
+            0.009403335861861706,
+            0.004485865123569965,
+            -0.0130825936794281,
+            0.009898620657622814,
+            -0.002890340518206358,
+            0.010867963545024395,
+            -0.0138467475771904,
+            -0.00420284504070878,
+            0.0010020672343671322,
+            -0.005363226868212223,
+            0.012615610845386982,
+            -0.030537845566868782,
+            0.011391550302505493,
+            -0.01352127455174923,
+            0.0008548084879294038,
+            -0.017858553677797318,
+            -0.027707645669579506,
+            0.0020288988016545773,
+            -0.016132133081555367,
+            0.039990708231925964,
+            0.010188715532422066,
+            -0.01481609046459198,
+            0.016995344310998917,
+            -0.002080196049064398,
+            0.017377421259880066,
+            0.0014955832157284021,
+            0.017023645341396332,
+            -0.011971740983426571,
+            0.021056678146123886,
+            -0.0041745430789887905,
+            0.02156611531972885,
+            0.00600002147257328,
+            0.034047290682792664,
+            -0.02033497765660286,
+            0.0070825726725161076,
+            0.003792466362938285,
+            0.005798369646072388,
+            -0.0026816134341061115,
+            0.021424604579806328,
+            0.017985913902521133,
+            0.019698183983564377,
+            0.0037110981065779924,
+            -0.024084990844130516,
+            -0.006137993652373552,
+            0.004867942072451115,
+            0.023858575150370598,
+            0.011150983162224293,
+            -0.016415152698755264,
+            -0.021056678146123886,
+            0.015226469375193119,
+            -0.007860877551138401,
+            0.006771250627934933,
+            -0.00851889792829752,
+            0.00037544354563578963,
+            -0.013259481638669968,
+            0.012233533896505833,
+            -0.012523629702627659,
+            0.0032335021533071995,
+            0.01043635793030262,
+            -0.027806703001260757,
+            0.01511326152831316,
+            -0.00371817359700799,
+            -0.014313730411231518,
+            0.023320838809013367,
+            -0.015877414494752884,
+            -0.013988257385790348,
+            0.005501199048012495,
+            0.008610880002379417,
+            0.01746232621371746,
+            -0.002761212643235922,
+            0.002669231267645955,
+            -0.020504789426922798,
+            -0.012445799075067043,
+            0.004591997712850571,
+            0.02210385352373123,
+            -0.015820810571312904,
+            0.012431648559868336,
+            -0.018099121749401093,
+            0.01364863384515047,
+            0.016202887520194054,
+            0.007149789482355118,
+            0.00573115237057209,
+            -0.007726442534476519,
+            -0.0011471149045974016,
+            0.032151058316230774,
+            -0.0029876285698264837,
+            0.02704254910349846,
+            -0.014915147796273232,
+            0.009615600109100342,
+            0.00518987700343132,
+            -0.013089669868350029,
+            -0.014589674770832062,
+            0.011108530685305595,
+            0.02428310550749302,
+            -0.009530694223940372,
+            -0.023660462349653244,
+            0.0018484735628589988,
+            0.014264201745390892,
+            0.010337301529943943,
+            0.009403335861861706,
+            -0.001997058978304267,
+            -0.0037358624394983053,
+            0.01484439242631197,
+            -0.0032918748911470175,
+            -0.0286557637155056,
+            -0.01633024774491787,
+            -0.004400959238409996,
+            -0.017490629106760025,
+            0.020745357498526573,
+            0.02367461286485195,
+            0.010507113300263882,
+            0.0037570889107882977,
+            0.005540114361792803,
+            0.018297234550118446,
+            0.01843874529004097,
+            0.013330236077308655,
+            0.009304278530180454,
+            0.012700516730546951,
+            -0.004319590982049704,
+            0.02800481766462326,
+            -0.010860888287425041,
+            -0.01046465989202261,
+            -0.015707602724432945,
+            0.0016636261716485023,
+            -0.005193414632230997,
+            -0.012431648559868336,
+            -0.0069481381215155125,
+            0.004369119182229042,
+            0.013330236077308655,
+            -0.012240610085427761,
+            -0.007149789482355118,
+            0.01016748882830143,
+            0.011971740983426571,
+            0.01403071079403162,
+            0.01719345711171627,
+            -0.01633024774491787,
+            -0.011278342455625534,
+            0.0009834941010922194,
+            -0.02844349853694439,
+            -0.0160047747194767,
+            -0.01767459139227867,
+            0.009580222889780998,
+            -0.014469391666352749,
+            0.03724541515111923,
+            0.028471799567341805,
+            0.0073514413088560104,
+            0.031613320112228394,
+            -0.028401045128703117,
+            0.00970758218318224,
+            -0.014398636296391487,
+            0.007132100872695446,
+            -0.004188694059848785,
+            0.003902136581018567,
+            0.002448122017085552,
+            0.020419884473085403,
+            0.00791748147457838,
+            0.014264201745390892,
+            0.008150972425937653,
+            -0.021934039890766144,
+            -0.00662620272487402,
+            -0.012559006921947002,
+            -0.016401002183556557,
+            -0.02595292218029499,
+            0.013719388283789158,
+            0.005087282508611679,
+            -0.003074303502216935,
+            -0.0002704166399780661,
+            0.006566061172634363,
+            -0.0013195802457630634,
+            0.010443433187901974,
+            -0.027608588337898254,
+            -0.01311089564114809,
+            0.0069269114173948765,
+            0.01449061743915081,
+            -0.02114158496260643,
+            0.0014637433923780918,
+            0.008900974877178669,
+            -0.004602610599249601,
+            0.035660505294799805,
+            -0.014674580655992031,
+            -0.008610880002379417,
+            0.0027806703001260757,
+            0.005062518175691366,
+            -0.017660440877079964,
+            -0.02319347858428955,
+            0.0070648835971951485,
+            0.011575513519346714,
+            0.010867963545024395,
+            -0.028570856899023056,
+            0.02087271586060524,
+            -0.0054339817725121975,
+            0.03832089155912399,
+            0.00401534466072917,
+            -0.013337312266230583,
+            0.003962278366088867,
+            0.0011860302183777094,
+            -0.012360893189907074,
+            0.002060738392174244,
+            -0.000340508297085762,
+            0.003700484987348318,
+            0.011228813789784908,
+            -0.014801939949393272,
+            -0.016967041417956352,
+            0.016910437494516373,
+            -0.011476456187665462,
+            -0.0035395172890275717,
+            -0.00943871308118105,
+            -0.016669871285557747,
+            0.007308988366276026,
+            0.025047259405255318,
+            0.0041745430789887905,
+            0.001242634141817689,
+            -0.011632117442786694,
+            -0.0038596836384385824,
+            -0.008575502783060074,
+            0.0011718792375177145,
+            -0.0207029040902853,
+            0.027396325021982193,
+            -0.01801421493291855,
+            0.002410975517705083,
+            0.021099131554365158,
+            0.008709936402738094,
+            -0.003944589756429195,
+            0.016316095367074013,
+            0.026363302022218704,
+            0.01205664686858654,
+            0.0031539027113467455,
+            -0.00902833417057991,
+            1.648646139074117e-05,
+            0.014957600273191929,
+            0.0015123875346034765,
+            -0.0015663381200283766,
+            0.0032140444964170456,
+            -0.006583749782294035,
+            -0.001859086798503995,
+            -0.017561383545398712,
+            0.005564878229051828,
+            0.02125479280948639,
+            -0.004132090136408806,
+            0.020476488396525383,
+            0.015509488992393017,
+            0.011625041253864765,
+            0.010726453736424446,
+            -0.0012390963966026902,
+            -0.002150951186195016,
+            -0.0021668709814548492,
+            -0.007393894251435995,
+            -0.018297234550118446,
+            0.006470541935414076,
+            0.0034493047278374434,
+            0.01822648011147976,
+            0.020504789426922798,
+            -0.02308027073740959,
+            0.010146263055503368,
+            0.014985902234911919,
+            -0.013011839240789413,
+            0.0235755555331707,
+            -0.02006610855460167,
+            0.0024605041835457087,
+            0.02156611531972885,
+            0.013337312266230583,
+            -0.014271277002990246,
+            -0.009219372645020485,
+            0.004662752617150545,
+            -0.01543873455375433,
+            0.000872939417604357,
+            -0.01478778850287199,
+            -0.010783057659864426,
+            0.014533070847392082,
+            -0.01490099634975195,
+            0.01229721400886774,
+            -0.006969364359974861,
+            0.0070967236533761024,
+            -0.017165156081318855,
+            0.015014204196631908,
+            0.008632106706500053,
+            0.01757553406059742,
+            0.0021279556676745415,
+            -0.02518877014517784,
+            -0.006290116813033819,
+            0.021169887855648994,
+            0.009509467519819736,
+            -0.02497650496661663,
+            0.019613277167081833,
+            0.0020094411447644234,
+            -0.019528372213244438,
+            -0.022401023656129837,
+            -0.012686366215348244,
+            -0.005055442452430725,
+            -0.017264213413000107,
+            -0.0011276573641225696,
+            -0.024792540818452835,
+            0.003972891718149185,
+            0.009106164798140526,
+            -0.011978816241025925,
+            -0.019273653626441956,
+            -0.0002315014135092497,
+            -0.01040805596858263,
+            0.006219361908733845,
+            -0.01132787112146616,
+            -0.019853845238685608,
+            0.0070436573587358,
+            0.012127402238547802,
+            -0.0027647505048662424,
+            -0.005646246485412121,
+            -0.02562745101749897,
+            -0.006919836159795523,
+            0.020179318264126778,
+            -0.003994117956608534,
+            -0.004974074196070433,
+            -0.01703779585659504,
+            0.01108730398118496,
+            0.011165134608745575,
+            0.020901018753647804,
+            0.010931642726063728,
+            0.005674548447132111,
+            0.013634482398629189,
+            -0.0029416377656161785,
+            -0.007156865205615759,
+            -9.645892532716971e-06,
+            0.004284213297069073,
+            0.0069233737885952,
+            0.0016636261716485023,
+            0.020235922187566757,
+            -0.01692458800971508,
+            0.018934030085802078,
+            -0.012290137819945812,
+            0.008794843219220638,
+            -0.01855195313692093,
+            -0.0003155229496769607,
+            -0.01067692507058382,
+            0.00015002264990471303,
+            -0.0038455326575785875,
+            -0.009544845670461655,
+            1.9595809135353193e-05,
+            0.00143986358307302,
+            0.008511822670698166,
+            -0.016783079132437706,
+            0.009615600109100342,
+            -0.010839661583304405,
+            -0.004308977629989386,
+            -0.019542522728443146,
+            0.017448175698518753,
+            0.013344387523829937,
+            0.007308988366276026,
+            -0.008278331719338894,
+            -0.006300730165094137,
+            -0.02282555401325226,
+            0.008582578040659428,
+            -0.0014637433923780918,
+            0.002214630600064993,
+            -0.028301987797021866,
+            -0.004358506295830011,
+            0.0024587351363152266,
+            0.020306676626205444,
+            0.0033661676570773125,
+            -0.010712302289903164,
+            0.008978805504739285,
+            -0.023377442732453346,
+            -0.0030159305315464735,
+            -0.023349139839410782,
+            -0.004075486212968826,
+            -0.005112046375870705,
+            0.010103809647262096,
+            0.008101443760097027,
+            0.01549533847719431,
+            -0.009898620657622814,
+            0.01478778850287199,
+            -0.004793649073690176,
+            0.012049571610987186,
+            -0.0012152165872976184,
+            0.01337976474314928,
+            -0.0003111007681582123,
+            -0.01505665760487318,
+            0.02158026583492756,
+            -0.008738238364458084,
+            0.001741456682793796,
+            -0.0052853962406516075,
+            0.0027346794959157705,
+            -0.017702894285321236,
+            -0.004553082399070263,
+            -0.014455240219831467,
+            0.01957082375884056,
+            -0.022584985941648483,
+            0.004153316840529442,
+            -0.005055442452430725,
+            -0.016075529158115387,
+            0.029434068128466606,
+            0.022896308451890945,
+            -0.0035483618266880512,
+            0.015070809051394463,
+            0.0017989451298490167,
+            0.0015061964513733983,
+            0.016202887520194054,
+            -0.015806660056114197,
+            -0.013025989755988121,
+            0.035066161304712296,
+            0.0176321379840374,
+            0.0004313841927796602,
+            -0.0023260696325451136,
+            -0.0037323245778679848,
+            0.0032193511724472046,
+            0.008023613132536411,
+            -0.010450509376823902,
+            -0.01141277700662613,
+            -0.017051948234438896,
+            0.009997677057981491,
+            -0.006155682262033224,
+            0.004446949809789658,
+            -0.00025891896802932024,
+            -7.617214578203857e-05,
+            -0.01516986545175314,
+            0.022584985941648483,
+            -0.017717044800519943,
+            -0.00602832343429327,
+            0.004687516950070858,
+            0.0207029040902853,
+            -0.02541518583893776,
+            0.01616043597459793,
+            -0.007107336539775133,
+            -0.013429293408989906,
+            0.012573158368468285,
+            -0.015127412974834442,
+            -0.0107335289940238,
+            -0.003175129182636738,
+            -0.008009462617337704,
+            -0.022754797711968422,
+            0.017929309979081154,
+            -0.013627407141029835,
+            0.015650998800992966,
+            -0.00287972716614604,
+            0.000562502013053745,
+            0.002057200763374567,
+            0.0032582662533968687,
+            0.02250008098781109,
+            0.030735958367586136,
+            0.010415131226181984,
+            -0.005465821363031864,
+            -0.021000074222683907,
+            0.012983537279069424,
+            0.0020589695777744055,
+            0.013202877715229988,
+            -0.002253545681014657,
+            -0.002135031158104539,
+            -0.016613267362117767,
+            -0.023122724145650864,
+            -0.0013010069960728288,
+            -0.01761798746883869,
+            -0.02552839368581772,
+            -0.01774534583091736,
+            -0.007047194987535477,
+            0.011349096894264221,
+            0.020405733957886696,
+            0.0009755341452546418,
+            -0.005384453106671572,
+            0.01559439580887556,
+            -0.006941062398254871,
+            0.002080196049064398,
+            -0.006594363134354353,
+            -0.002032436430454254,
+            -0.008200501091778278,
+            -0.021863285452127457,
+            0.029660483822226524,
+            -0.010103809647262096,
+            0.01141277700662613,
+            -0.005472897086292505,
+            -0.00965097825974226,
+            0.005575491581112146,
+            0.00900003220885992,
+            0.024849144741892815,
+            0.006452853325754404,
+            0.03141520544886589,
+            0.020151015371084213,
+            -0.018891576677560806,
+            -0.0006071660900488496,
+            0.022684043273329735,
+            -0.0037570889107882977,
+            -0.004889168310910463,
+            -0.0031786670442670584,
+            -0.017985913902521133,
+            -0.012007118202745914,
+            0.007839650847017765,
+            -0.013471745885908604,
+            -0.006636816076934338,
+            -0.00837738811969757,
+            0.018070818856358528,
+            -0.017773648723959923,
+            0.025698205456137657,
+            -0.006488230545073748,
+            0.005154499784111977,
+            0.031924642622470856,
+            0.014363259077072144,
+            0.019698183983564377,
+            -0.011794853024184704,
+            -0.031075581908226013,
+            -0.00683846790343523,
+            -0.007882103323936462,
+            0.009558996185660362,
+            -0.01062032114714384,
+            0.005204027984291315,
+            -0.02006610855460167,
+            -0.018948180601000786,
+            0.012863253243267536,
+            -0.017476476728916168,
+            0.013924578204751015,
+            -0.009955224581062794,
+            0.027226513251662254,
+            -0.001228483160957694,
+            0.0121698547154665,
+            -0.002614396158605814,
+            -0.017801949754357338,
+            0.01205664686858654,
+            0.004089637193828821,
+            -0.023943481966853142,
+            0.010337301529943943,
+            -0.001864393474534154,
+            -0.014037786051630974,
+            -0.018410442396998405,
+            -0.001228483160957694,
+            -0.005200490355491638,
+            -0.009658053517341614,
+            0.025457637384533882,
+            0.014101465232670307,
+            0.016231190413236618,
+            0.0035359796602278948,
+            0.0016335553955286741,
+            -0.011073152534663677,
+            0.009573147632181644,
+            -0.006803090218454599,
+            0.000612030504271388,
+            -0.0034404604230076075,
+            0.011893910355865955,
+            -0.004252373706549406,
+            -0.01543873455375433,
+            0.002324300818145275,
+            0.008561351336538792,
+            0.0143278818577528,
+            5.947839599684812e-05,
+            0.009587298147380352,
+            0.0009790719486773014,
+            -0.009764186106622219,
+            -0.00046388726332224905,
+            -0.0060884649865329266,
+            0.0069375247694551945,
+            0.02400008589029312,
+            -0.012516554445028305,
+            -0.01681138016283512,
+            0.014504768885672092,
+            0.004033033270388842,
+            -0.017589686438441277,
+            0.018367990851402283,
+            0.004103788174688816,
+            -0.022358570247888565,
+            0.0035412863362580538,
+            0.010662773624062538,
+            -0.009523618966341019,
+            -0.014469391666352749,
+            -0.0010339069413021207,
+            -0.015509488992393017,
+            0.03130199760198593,
+            0.00025073791039176285,
+            0.016910437494516373,
+            -0.01062032114714384,
+            -0.0025967073161154985,
+            -0.013309009373188019,
+            0.013103820383548737,
+            -0.015750056132674217,
+            -0.007273610681295395,
+            0.00035620704875327647,
+            -0.02298121340572834,
+            0.01741987280547619,
+            0.0013425755314528942,
+            0.00794578343629837,
+            0.003436922561377287,
+            -0.010556641966104507,
+            0.009820790030062199,
+            0.0018237093463540077,
+            0.006668656133115292,
+            -0.014915147796273232,
+            0.023618008941411972,
+            0.006413938011974096,
+            0.011235889047384262,
+            -0.01352127455174923,
+            -0.01546703651547432,
+            0.002253545681014657,
+            0.04134920611977577,
+            -0.01306136790663004,
+            0.0002593612007331103,
+            0.01562269777059555,
+            7.081024523358792e-05,
+            0.004156854469329119,
+            -0.0023755980655550957,
+            -0.0034493047278374434,
+            -0.034585028886795044,
+            0.005370302125811577,
+            -0.0006328147719614208,
+            0.008073141798377037,
+            -0.008992956951260567,
+            -0.005430444143712521,
+            -0.0040684109553694725,
+            -0.008625030517578125,
+            -0.00314859626814723,
+            -0.01478778850287199,
+            0.008363237604498863,
+            -0.014533070847392082,
+            -0.028910480439662933,
+            0.03166992589831352,
+            -0.006887996103614569,
+            -0.00533846253529191,
+            -0.021396303549408913,
+            0.007570781745016575,
+            -0.01370523776859045,
+            0.002663924591615796,
+            0.010967019945383072,
+            0.008865597657859325,
+            0.008738238364458084,
+            0.002762981690466404,
+            0.0009516543359495699,
+            -0.004255911335349083,
+            0.010273621417582035,
+            -0.00835616234689951,
+            -0.0005943417199887335,
+            -0.0014186371117830276,
+            -0.02725481428205967,
+            -0.01405901275575161,
+            -0.010592019185423851,
+            -0.007811348885297775,
+            -0.00435143057256937,
+            0.012608535587787628,
+            0.007174553815275431,
+            -0.0017485321732237935,
+            -0.022938761860132217,
+            -0.014745336025953293,
+            -0.0034103896468877792,
+            0.008349086157977581,
+            -0.01838214136660099,
+            0.01064862310886383,
+            0.0016830838285386562,
+            0.009976450353860855,
+            0.014292503707110882,
+            0.00302831269800663,
+            0.0016742395237088203,
+            0.003458149265497923,
+            0.014886845834553242,
+            0.012601460330188274,
+            -0.005801907740533352,
+            -0.021382151171565056,
+            -0.0024605041835457087,
+            -0.007068421691656113,
+            -0.0034687623847275972,
+            -0.012332591228187084,
+            0.03039633482694626,
+            0.0029805530793964863,
+            0.0176321379840374,
+            0.007832575589418411,
+            0.009573147632181644,
+            0.010507113300263882,
+            0.029377464205026627,
+            0.006049550138413906,
+            -0.0029929352458566427,
+            -0.008603804744780064,
+            -0.02141045406460762,
+            -0.013146273791790009,
+            0.009898620657622814,
+            0.013294858857989311,
+            0.03597182780504227,
+            -0.034896351397037506,
+            0.012162779457867146,
+            -0.011101454496383667,
+            -0.008264180272817612,
+            -0.008533049374818802,
+            0.0044717141427099705,
+            0.010068432427942753,
+            0.014370334334671497,
+            -0.0023402206134051085,
+            -0.01175947580486536,
+            0.009070787578821182,
+            0.023547254502773285,
+            -0.012983537279069424,
+            0.03016991913318634,
+            0.027325568720698357,
+            0.023066120222210884,
+            0.01761798746883869,
+            -0.012537780217826366,
+            0.026462359353899956,
+            -6.484030018327758e-05,
+            0.0006261814851313829,
+            -0.018084971234202385,
+            -0.00587973790243268,
+            -0.004294826649129391,
+            -0.004362043924629688,
+            0.013316085562109947,
+            -0.0025471788831055164,
+            -0.0008468485320918262,
+            0.015099111013114452,
+            -0.0037323245778679848,
+            -3.927453508367762e-05,
+            -0.0059752571396529675,
+            0.015127412974834442,
+            0.0002699744363781065,
+            0.019457615911960602,
+            -0.04270770028233528,
+            0.019726485013961792,
+            -0.021396303549408913,
+            -0.003102605463936925,
+            -0.0020890405867248774,
+            0.009523618966341019,
+            0.00764861237257719,
+            -0.007142714224755764,
+            -0.014334957115352154,
+            0.004524780437350273,
+            0.014066088013350964,
+            -0.0006726144347339869,
+            0.005080206785351038,
+            -0.0027541371528059244,
+            0.0017228835495188832,
+            -0.010089659132063389,
+            -0.005862049292773008,
+            5.842812606715597e-05,
+            -0.010570792481303215,
+            -0.008646257221698761,
+            -0.03232087194919586,
+            0.01062032114714384,
+            0.010209942236542702,
+            0.007294837385416031,
+            0.004252373706549406,
+            -0.022018946707248688,
+            -0.012318440712988377,
+            0.012948159128427505,
+            -0.01132787112146616,
+            -0.002570174168795347,
+            -0.0020005968399345875,
+            0.005387990735471249,
+            0.0004705205501522869,
+            0.010160413570702076,
+            -0.003472300246357918,
+            0.010938717983663082,
+            -0.014603826217353344,
+            0.018084971234202385,
+            0.03229256719350815,
+            -0.00533846253529191,
+            -0.019967053085565567,
+            0.000907432462554425,
+            -0.009077862836420536,
+            0.003813692834228277,
+            0.007641536649316549,
+            -0.021962342783808708,
+            -0.02173592709004879,
+            0.005087282508611679,
+            -0.013153349049389362,
+            -0.007783046457916498,
+            0.014830241911113262,
+            0.005221716593950987,
+            0.010839661583304405,
+            0.01751893013715744,
+            0.010889190249145031,
+            0.00840569008141756,
+            -0.026533113792538643,
+            0.0006734988419339061,
+            -0.012686366215348244,
+            -0.00590803986415267,
+            0.002239394700154662,
+            -0.0017591454088687897,
+            0.004591997712850571,
+            -0.00010115749319083989,
+            -0.011200511828064919,
+            -0.0007057808106765151,
+            0.010903340764343739,
+            0.027919910848140717,
+            -0.004146241117268801,
+            -0.0022570835426449776,
+            -0.014257126487791538,
+            0.0010100272484123707,
+            0.036481261253356934,
+            -0.012275987304747105,
+            -0.017434025183320045,
+            0.01757553406059742,
+            0.027452928945422173,
+            0.005954030901193619,
+            -0.0032529598101973534,
+            -0.0027364485431462526,
+            0.00039600671152584255,
+            0.011978816241025925,
+            0.007719367276877165,
+            -0.002692226553335786,
+            -0.002131493529304862,
+            0.004761809483170509,
+            0.035773713141679764,
+            -0.0015203474322333932,
+            -0.010330225341022015,
+            -0.024240652099251747,
+            0.010974096134305,
+            0.00277182599529624,
+            0.004754733759909868,
+            0.002313687466084957,
+            0.023504801094532013,
+            0.019655730575323105,
+            0.014858543872833252,
+            0.00470520555973053,
+            -0.009771261364221573,
+            0.03005671128630638,
+            -0.008172199130058289,
+            -0.004542469047009945,
+            -0.02838689461350441,
+            -0.005087282508611679,
+            -0.015650998800992966,
+            0.008929276838898659,
+            -0.0002221042668679729,
+            -0.005016527138650417,
+            -0.013422218151390553,
+            0.019726485013961792,
+            0.01064862310886383,
+            -0.011073152534663677,
+            0.012608535587787628,
+            -0.027070851996541023,
+            0.02346234768629074,
+            -0.03826428949832916,
+            -0.0071993181481957436,
+            -0.006421013735234737,
+            -0.010514188557863235,
+            -0.02627839520573616,
+            -0.0030353881884366274,
+            0.007372668012976646,
+            0.014250051230192184,
+            0.0022570835426449776,
+            0.0022818478755652905,
+            0.02282555401325226,
+            0.0016415152931585908,
+            -0.01508495956659317,
+            0.017985913902521133,
+            0.03492465242743492,
+            0.012926933355629444,
+            0.00155837822239846,
+            0.015990622341632843,
+            0.0027789014857262373,
+            0.0021633331198245287,
+            0.00788917951285839,
+            -0.0034493047278374434,
+            -0.007938707247376442,
+            0.008582578040659428,
+            0.006990591064095497,
+            -0.02763689123094082,
+            0.003824306186288595,
+            -0.00010408719390397891,
+            -0.009898620657622814,
+            -0.022429324686527252,
+            0.011568437330424786,
+            0.01913214474916458,
+            -0.004224071744829416,
+            -0.006771250627934933,
+            -0.0043337419629096985,
+            0.0006836698739789426,
+            0.006785401608794928,
+            -0.002412744564935565,
+            -0.000515184598043561,
+            -0.0015176940942183137,
+            -0.02368876338005066,
+            -0.010316074825823307,
+            0.013570803217589855,
+            -0.005175726022571325,
+            0.010358527302742004,
+            0.005242943298071623,
+            -0.018792519345879555,
+            -0.016938740387558937,
+            0.006852618884295225,
+            -0.016188737004995346,
+            0.01364863384515047,
+            0.00027948213391937315,
+            -0.0023172253277152777,
+            0.024962352588772774,
+            0.03594352304935455,
+            -0.013188726268708706,
+            0.005119122099131346,
+            -0.019386861473321915,
+            -0.0245378240942955,
+            -0.019004784524440765,
+            0.022613288834691048,
+            0.008306633681058884,
+            -0.018792519345879555,
+            0.012304289266467094,
+            -0.002752368338406086,
+            -0.008865597657859325,
+            -0.0066721937619149685,
+            -0.006845543161034584,
+            -0.008547200821340084,
+            0.020533092319965363,
+            -0.004935159347951412,
+            0.0038950610905885696,
+            0.019047237932682037,
+            0.04075486212968826,
+            -0.008292482234537601,
+            -0.01349297259002924,
+            0.0054339817725121975,
+            0.004878554958850145,
+            0.0018626245437189937,
+            -0.004814875777810812,
+            -0.008016537874937057,
+            -0.0185944065451622,
+            -0.01740572229027748,
+            -0.009615600109100342,
+            -0.0020041344687342644,
+            0.020122714340686798,
+            -0.007768895477056503,
+            -0.01354957651346922,
+            -0.0245378240942955,
+            -0.028188779950141907,
+            -0.006580212153494358,
+            0.02319347858428955,
+            0.017179306596517563,
+            -0.00013200224202591926,
+            -4.3337418901501223e-05,
+            -0.012042495422065258,
+            -0.01968403346836567,
+            0.015509488992393017,
+            0.00037876018905080855,
+            -0.01773119531571865,
+            -0.0065448349341750145,
+            -0.015919867902994156,
+            0.005787756759673357,
+            -0.0051368107087910175,
+            0.004574308637529612,
+            0.05057565122842789,
+            0.0005271245026960969,
+            0.008441068232059479,
+            -0.026575567200779915,
+            0.009106164798140526,
+            -0.01583496294915676,
+            -0.017377421259880066,
+            -0.00012702727690339088,
+            -0.022301966324448586,
+            -0.0030955299735069275,
+            -0.018792519345879555,
+            -0.007386818993836641,
+            0.008568426594138145,
+            0.009799563325941563,
+            -0.013896276243031025,
+            -0.013669860549271107,
+            -0.024693485349416733,
+            -0.0035872769076377153,
+            -0.001539805089123547,
+            0.009311353787779808,
+            0.007825499400496483,
+            -0.03693409264087677,
+            0.002536565763875842,
+            -0.0009507699287496507,
+            0.0005779796629212797,
+            0.01208494883030653,
+            -0.008667483925819397,
+            -0.003063690150156617,
+            -0.0214670579880476,
+            0.006452853325754404,
+            0.007103798910975456,
+            -0.013500047847628593,
+            0.012375044636428356,
+            -0.012863253243267536,
+            -0.007669838611036539,
+            -0.0005426021525636315,
+            0.03379257395863533,
+            0.011794853024184704,
+            -0.0025153392925858498,
+            -0.00654129683971405,
+            -0.011150983162224293,
+            0.02654726430773735,
+            -0.011349096894264221,
+            -0.026405755430459976,
+            0.028910480439662933,
+            0.013245330192148685,
+            0.0005974372616037726,
+            0.020151015371084213,
+            -0.0017936384538188577,
+            -0.01397410687059164,
+            0.009601449593901634,
+            0.004266524687409401,
+            0.012424572370946407,
+            -0.00014460546663030982,
+            0.0214670579880476,
+            0.0018555490532889962,
+            0.0027399861719459295,
+            -0.0036226543597877026,
+            0.004687516950070858,
+            -0.03093407303094864,
+            -0.0006549257086589932,
+            -0.003102605463936925,
+            -0.008398614823818207,
+            0.011207587085664272,
+            -0.002421588869765401,
+            0.006558985915035009,
+            -0.022188758477568626,
+            -0.028485950082540512,
+            0.011342021636664867,
+            -0.016231190413236618,
+            -0.021396303549408913,
+            0.015339677222073078,
+            -0.0007641536649316549,
+            0.00614860700443387,
+            -0.007443422917276621,
+            -0.023278385400772095,
+            -0.019386861473321915,
+            0.02064630016684532,
+            0.012212308123707771,
+            -0.01190098561346531,
+            0.012750045396387577,
+            -0.021113283932209015,
+            -0.012948159128427505,
+            -0.001894464367069304,
+            0.0090566361322999,
+            -0.0019864458590745926,
+            0.01325240544974804,
+            -0.012198156677186489,
+            -0.00334494118578732,
+            0.0031592093873769045,
+            -0.011589664034545422,
+            0.008158047683537006,
+            0.01978308893740177,
+            0.008193425834178925,
+            -0.009339655749499798,
+            0.003422771580517292,
+            -0.011922212317585945,
+            0.013500047847628593,
+            0.015537790954113007,
+            0.018311386927962303,
+            -0.006498843897134066,
+            0.0020412809681147337,
+            -0.01370523776859045,
+            0.0063962494023144245,
+            -0.009827865287661552,
+            -0.004878554958850145,
+            -0.0005333155859261751,
+            0.004921008367091417,
+            -0.011179285123944283,
+            -0.0025772498920559883,
+            -0.026830283924937248,
+            -0.0013823752524331212,
+            -0.020533092319965363,
+            0.005303084850311279,
+            0.023207630962133408,
+            -0.007514177821576595,
+            0.0006045127520337701,
+            -0.0039092120714485645,
+            0.014476466923952103,
+            0.020688753575086594,
+            0.00870286114513874,
+            0.00482548912987113,
+            -0.006081389728933573,
+            0.02179253101348877,
+            0.011603815481066704,
+            -0.017816102132201195,
+            0.00164239970035851,
+            -0.001509734196588397,
+            -0.0014336726162582636,
+            0.011625041253864765,
+            0.004443412180989981,
+            -0.020844414830207825,
+            0.0028602697420865297,
+            -0.010089659132063389,
+            0.007818424142897129,
+            0.014702882617712021,
+            -0.00031906069489195943,
+            -0.02486329711973667,
+            -0.009827865287661552,
+            0.021919889375567436,
+            -0.015070809051394463,
+            -0.0008291598060168326,
+            0.00764861237257719,
+            0.023264234885573387,
+            -0.024792540818452835,
+            -0.006831392180174589,
+            -0.0023455272894352674,
+            -0.003933976404368877,
+            -0.005193414632230997,
+            0.0010489424457773566,
+            0.0214670579880476,
+            0.004415110219269991,
+            -0.004234685096889734,
+            0.005724077112972736,
+            0.006470541935414076,
+            0.02325008250772953,
+            0.0040577976033091545,
+            0.028047269210219383,
+            -0.00943871308118105,
+            -0.0023809047415852547,
+            -0.0019988277927041054,
+            -4.775960405822843e-05,
+            0.0160047747194767,
+            -0.005430444143712521,
+            0.020320827141404152,
+            0.009445788338780403,
+            0.0036934094969183207,
+            0.016514210030436516,
+            0.02319347858428955,
+            -0.006767712999135256,
+            -0.004793649073690176,
+            0.019160445779561996,
+            0.0053667644970119,
+            -0.008858522400259972,
+            -0.02194819226861,
+            0.01141277700662613,
+            0.0014124460285529494,
+            -0.0006956097786314785,
+            0.012106175534427166,
+            0.0024375086650252342,
+            0.006300730165094137,
+            0.008172199130058289,
+            -0.016684021800756454,
+            0.004241760354489088,
+            -0.004698129836469889,
+            -0.01860855706036091,
+            0.010910416021943092,
+            -0.012092024087905884,
+            -5.201595922699198e-05,
+            -0.0010002984199672937,
+            0.014165145345032215,
+            -0.01703779585659504,
+            -0.0019103841623291373,
+            -0.0070719593204557896,
+            -1.3024719009990804e-05,
+            -0.0009162768837995827,
+            0.008320784196257591,
+            -0.006254739128053188,
+            -0.010938717983663082,
+            -0.010938717983663082,
+            0.027509532868862152,
+            -0.004991763271391392,
+            0.003916287794709206,
+            0.019330257549881935
+          ],
+          "embedding": [
+            0.007204997251390758,
+            -0.03173324616100502,
+            -0.020505790884701024,
+            0.0037451608231038273,
+            0.009817085391002229,
+            0.026883177166072393,
+            -0.016477826654817607,
+            0.06949311853773875,
+            -0.01224876667472692,
+            0.031952566873802865,
+            0.013633671951171586,
+            -0.005041437355104989,
+            0.02663392750069342,
+            -0.022099132865267587,
+            0.006665702000169926,
+            0.0030467129715919033,
+            0.007931802508415542,
+            0.009984477355413895,
+            -0.010362836464183985,
+            -0.03180848181251013,
+            -0.013542377333474449,
+            -0.013572345022577053,
+            -0.03979985300694044,
+            -0.007698412130292187,
+            -0.005953452234356532,
+            0.018890228917932694,
+            0.02217446994660065,
+            0.005158357501784182,
+            -0.05468694715835175,
+            0.00789994048345594,
+            0.017130184174786147,
+            -0.0026591475669416114,
+            -0.02081141846859738,
+            -0.008403928318906045,
+            -0.030925346724945978,
+            -0.03855023691059801,
+            0.038853354785097736,
+            0.01612011653434478,
+            -0.035635702395196575,
+            0.026266623757525934,
+            0.043450373922427316,
+            -0.007254334835408145,
+            -0.015376804869420366,
+            -0.0017276202867766907,
+            0.004650414207094434,
+            -0.002365514361535186,
+            0.011906139905776183,
+            -0.0005664194302009289,
+            -0.0191723447249157,
+            -0.025546124518037998,
+            0.01025005257280377,
+            -0.014436532079436437,
+            0.029277529346694118,
+            -0.0045414983847733344,
+            -0.02260315456913697,
+            0.01717918656248626,
+            0.013666132349359846,
+            0.057804090189845364,
+            0.056600576093100735,
+            0.051208022172527734,
+            0.03553760896972431,
+            0.02094599510567085,
+            -0.02438674472832137,
+            -0.0027797634437333432,
+            0.03614248430646102,
+            -0.026729308357113124,
+            -0.002251394534452815,
+            0.018929251842788906,
+            0.00689697483318378,
+            -0.026603606434115974,
+            -0.0541443728344337,
+            0.027164630314707807,
+            -0.004335272389193875,
+            -0.013967321193992958,
+            0.003323121377857836,
+            -0.0245295956412367,
+            0.024489335129134787,
+            0.06212059558640156,
+            0.009214058207162371,
+            -0.002082293435905329,
+            0.013659934201796686,
+            -0.003344037927706573,
+            -0.00689041317919184,
+            0.020235785390418023,
+            0.0212739457082623,
+            0.0016782346585238381,
+            -0.013484221664059555,
+            -0.06465582506604173,
+            0.011882797526607326,
+            -0.032970655964890155,
+            -0.04027700439305653,
+            0.025396847912528105,
+            0.013861157517716707,
+            0.00604596157583048,
+            -0.00744611407420936,
+            0.04038084298991919,
+            0.021507532290328625,
+            0.005983822806623904,
+            -0.012711110369915742,
+            0.02410138142963901,
+            -0.018372791517880082,
+            0.021016674128298246,
+            0.013387285956415756,
+            0.02039383525127615,
+            0.010700292312227496,
+            0.0053934660497349735,
+            -0.014807678014449691,
+            0.009422805142740376,
+            0.0414357625490679,
+            0.12206398267002624,
+            -0.010877509834324155,
+            -0.011079302503786522,
+            -0.04107714853756596,
+            0.006631397986165343,
+            -0.031914267104222384,
+            -0.0179965295457466,
+            -0.057530036803978266,
+            -0.0320488164246567,
+            0.008208202363320509,
+            -0.0020203090915064162,
+            0.03502055780919601,
+            0.04464599164916666,
+            0.019332152079287082,
+            0.017133943087431935,
+            -0.039885565008919065,
+            0.04703059988373572,
+            -0.029370696964304416,
+            0.0008494086835971101,
+            0.0014082189719987074,
+            -0.030973762582071615,
+            -0.006561157852024518,
+            0.011634653995510726,
+            -0.019896826887411713,
+            -0.034275921680252924,
+            0.026274465530039866,
+            -0.007728234125236442,
+            0.04814514040690998,
+            0.06550407296280117,
+            -0.02749051703615343,
+            -0.010041099502187074,
+            0.02019601967763537,
+            -0.01871701432558228,
+            0.008532829759864928,
+            -0.020164138072967613,
+            -0.045196925901010855,
+            -0.004523517988516156,
+            -0.002113469149506399,
+            0.07929650663710812,
+            -0.01111269708042164,
+            0.03358254444720164,
+            -0.019973382759721527,
+            -0.0007145592863325417,
+            -0.0011740862928390678,
+            0.02166182480535788,
+            0.004543228023398081,
+            -0.013377688428761516,
+            -0.01758250073647576,
+            0.009548292014596834,
+            -0.04753617258004735,
+            -0.007125744019792479,
+            -0.009930358894299303,
+            -0.007931672044235996,
+            -0.024561936537429774,
+            0.012979195012881556,
+            0.009209145250061479,
+            0.04586756662401364,
+            0.04311807301838248,
+            -0.0036241105681042675,
+            -0.06579256720896123,
+            0.04967191755416882,
+            0.015475886231466817,
+            0.03157651656768371,
+            0.02515941109300714,
+            0.01773554572598468,
+            0.005824418191900522,
+            0.016317635547562007,
+            0.007650872352957098,
+            0.010015166465492963,
+            -0.033203307142867844,
+            0.040272188258443496,
+            -0.013408423788025047,
+            0.01036004294342104,
+            0.012391622448703885,
+            0.04214857862444935,
+            0.03418032388760535,
+            0.015263018079889338,
+            0.018786704580603317,
+            0.02704685343985817,
+            0.015348111464412869,
+            0.0013031600434754704,
+            0.03241427528588285,
+            0.011689924942195664,
+            0.04544406114742012,
+            0.016186615914968734,
+            0.011624500626103958,
+            -0.008272953078864385,
+            -0.02363080189189375,
+            0.02784298119317616,
+            -0.010585980875564109,
+            0.01455013604305903,
+            0.01749228557615982,
+            -0.054742056843690476,
+            -0.005876213780171779,
+            -0.010556605098730201,
+            0.003967456397644932,
+            0.014663888851767762,
+            0.0015548476559947807,
+            -0.003858367677774835,
+            0.012504286824484335,
+            -0.025806882498628832,
+            -0.014216299453397403,
+            0.0031620924516417956,
+            -0.006448390918604307,
+            -0.02785577502346934,
+            -0.00766402687921159,
+            0.005275000635711884,
+            -0.0090448751942,
+            0.019702662630533,
+            0.007009091837467867,
+            -0.01847307100043082,
+            -0.0006774105206839772,
+            0.032952656101827446,
+            0.015900033044760432,
+            -0.02170121261931376,
+            -0.017964879081469612,
+            -0.005062849745084668,
+            -0.03171847232054206,
+            -0.038177398521606544,
+            0.029426291950046455,
+            -0.025687201683643784,
+            0.030158867757101797,
+            0.015780411479509624,
+            -0.007011484049169482,
+            -0.012114023533118393,
+            0.020661407942701628,
+            -0.025067530380881376,
+            0.009777190971413269,
+            -0.004160771288889298,
+            -0.028825369485727344,
+            0.008307290529469729,
+            -0.014415903546305744,
+            -0.007561235629544945,
+            -0.03201704135271719,
+            -0.015182869388110948,
+            -0.00501561450194676,
+            -0.0003863746169672397,
+            -0.01073403445816663,
+            -0.03321103354401307,
+            -0.002304357889348034,
+            0.017001213960120047,
+            0.00028222146729658097,
+            -0.04002882852650754,
+            0.014509588052058002,
+            -0.004150541038999649,
+            -0.034333837380045466,
+            0.0564137596224324,
+            0.03584898691608747,
+            -0.006830698606679202,
+            0.006121456445207273,
+            -0.04505674213649008,
+            0.04358467937307492,
+            0.010847859073780811,
+            -0.04918983713702805,
+            0.049190442935189385,
+            -0.00528960443684769,
+            0.01419195371129309,
+            0.040316024601793406,
+            0.004600843996974165,
+            -0.006780970176765864,
+            0.005206651850992678,
+            0.03397517915308201,
+            -0.010037223444627569,
+            -0.027377913828275806,
+            -0.03328136706915221,
+            0.02562568440972934,
+            -0.03485770844248482,
+            -0.020069979230687374,
+            -0.0052041408068332524,
+            0.029818953377223505,
+            0.028789460147370508,
+            0.004071274130619537,
+            0.01928375754300958,
+            0.0014970947734202088,
+            0.010502599795027616,
+            0.016914145702515263,
+            0.011942576738188502,
+            -0.014909953528043828,
+            -0.003432489584767786,
+            -0.05733768990013957,
+            0.0030710780388681002,
+            -0.02627280695622545,
+            0.007930906480371135,
+            -0.031227917075485834,
+            0.013132797714872345,
+            0.002514351081823354,
+            -0.007152021475569949,
+            -0.0066245205132327055,
+            -0.005279224195782313,
+            0.028512607222275155,
+            -0.00921960825771865,
+            -0.0359704105589433,
+            -0.021347597388353182,
+            0.011200044645812517,
+            -0.0004171294784927543,
+            -0.016784468873218786,
+            -0.0018746544944553878,
+            0.007749103417240419,
+            0.02390272097796833,
+            -0.005128540108845805,
+            0.006397849195755672,
+            0.01214922062611769,
+            0.01163371266250103,
+            0.012307186722130844,
+            -0.004982589662882946,
+            -0.016343724553037307,
+            0.019806887143115643,
+            0.013139657463054087,
+            0.0205801609265216,
+            -0.03767676007629612,
+            -0.0249879011550972,
+            0.05547392586363707,
+            0.005903313549319051,
+            -0.05568806298774435,
+            0.018827693208840233,
+            -0.007931178941307645,
+            0.008183992600768387,
+            -0.010319022946041053,
+            0.05111352384595183,
+            -0.024897203290807753,
+            -0.02096678205188722,
+            0.006436523335212348,
+            -0.04917084523640266,
+            -0.010459284852442706,
+            -0.011138558192296276,
+            0.014789370043397917,
+            -0.03105725978752438,
+            -0.004934745004818259,
+            0.015595801375817122,
+            -4.687585727052446e-05,
+            0.0028886067775750254,
+            -0.00850352807499054,
+            -0.0034885579171087815,
+            0.016339681487281293,
+            -0.0034050317046826813,
+            -0.008309570448487812,
+            -0.003615165868022203,
+            -0.027638409919916757,
+            0.023352910161586186,
+            0.026364511529991978,
+            0.001249936082895606,
+            0.001987948967948376,
+            0.01756629246122769,
+            0.004656075485833687,
+            -0.006752579977941239,
+            -0.013681608638716132,
+            -0.03405936202918687,
+            -0.0401867651078737,
+            0.022991180183813725,
+            -0.02318908140203705,
+            0.018299512436825923,
+            0.00904081847012441,
+            0.019697091729447287,
+            -0.019236130984371835,
+            0.015159917756323988,
+            -0.0387297415849285,
+            -0.018793498297161176,
+            -0.005256288213225733,
+            -0.010414697711744128,
+            0.0007039894599589104,
+            -0.01739645696111503,
+            -0.013728845328097506,
+            0.022709522955937343,
+            -0.022789243441781803,
+            -0.0036763802589970026,
+            -0.01429671717544519,
+            0.0023674126354206932,
+            0.049448298344058644,
+            -0.009926033487593551,
+            -0.023371285193090733,
+            0.020846792916650956,
+            -0.026996519120163966,
+            0.011246908457705283,
+            0.04091365159640201,
+            0.005469162850197613,
+            0.005640919201612091,
+            0.029575753750229278,
+            -0.0007860452837141593,
+            -0.0052805237594885516,
+            -0.009782745443933828,
+            0.021178503581190618,
+            -0.0009176512672832309,
+            -0.05245650420332918,
+            -0.007698364493410058,
+            -0.039027418406703554,
+            0.005448840777995117,
+            0.017581625031764185,
+            0.025522517097715556,
+            -0.020775492472580457,
+            -0.015906320383873178,
+            -0.000643669839811819,
+            -0.0086896084429604,
+            0.0029399233482444638,
+            0.014381240985647907,
+            0.021086693776922936,
+            0.002436575081260609,
+            -0.004913514868101904,
+            -0.011582101006213432,
+            0.014400297663309155,
+            0.0467181039341492,
+            -0.01722692180137712,
+            0.0011622224560709315,
+            -0.013036540368441797,
+            -0.06208454638917413,
+            -0.019160212331432766,
+            0.0351496273293468,
+            0.02429814055116254,
+            -0.009896662169762772,
+            -0.01647101145477158,
+            -0.002915821099802562,
+            -0.03557863734177486,
+            -0.01786433999615293,
+            -0.010699364747311715,
+            0.02390901423299708,
+            0.007213809581960646,
+            0.0005601364043388203,
+            0.00916614601011734,
+            0.014737491634516238,
+            0.019303599465763203,
+            0.008864535607492504,
+            -0.0575839606069681,
+            0.018051837786406608,
+            0.006315830080225406,
+            0.02525619013082567,
+            -0.015671924543113065,
+            0.003988943108210209,
+            -0.03179878281917948,
+            -0.029440641244429775,
+            0.040586322748158446,
+            -0.03418722657593859,
+            -0.030650542761032924,
+            0.006566745611226466,
+            0.009571089937153852,
+            0.004491158558315032,
+            -0.011352813042040581,
+            -0.02535505987721779,
+            -0.013381948772812133,
+            -0.030918676056735445,
+            -0.02192708443081135,
+            -0.005232667170255381,
+            -0.03342408726436631,
+            -0.07498634115550039,
+            -0.006245638452568239,
+            -0.049946896465756174,
+            -0.03900385398225758,
+            0.04458524125677807,
+            -0.006106489377959481,
+            0.02398410968928338,
+            -0.04615183642824994,
+            -0.01165382599292015,
+            -0.04890013373371541,
+            0.013624324077017377,
+            0.01476398305411351,
+            0.020202554298470666,
+            0.00014553149242532853,
+            0.036224316512987846,
+            0.00021498496267774312,
+            -0.05055208658013767,
+            -0.057482169168904106,
+            -0.003960451575823462,
+            -0.010217078509789989,
+            -0.005593738972150311,
+            -0.034090261748104665,
+            0.05835341009022349,
+            0.0116359661127526,
+            0.04203678184748103,
+            0.03447818335186531,
+            -0.006747564131379555,
+            -0.025756962754841885,
+            -0.028123416424745064,
+            -0.008073625379614702,
+            0.047544241458556825,
+            -0.029605671083240564,
+            0.008171081332993893,
+            -0.026007452205474395,
+            0.0067084589781962805,
+            -0.0070645448617940995,
+            -0.0012298101311792947,
+            0.012372286141738867,
+            0.006377319686763927,
+            -0.015024537877377194,
+            0.007934745664790507,
+            0.026041755081605937,
+            -0.007303731324541215,
+            0.0407000170257006,
+            -0.012990703097937777,
+            -0.033189084013863494,
+            -0.022525679964848666,
+            0.01657717698191603,
+            0.03440284885162039,
+            0.010230633738702899,
+            -0.008707808754455952,
+            -0.0363777020540216,
+            0.011184076269109602,
+            0.031774012431843524,
+            0.01504703222643543,
+            -0.022509833303978505,
+            0.025230974050270107,
+            0.017502662694813216,
+            -0.018700579139537456,
+            -0.03511734747709348,
+            0.015162183119327901,
+            -0.017711638132862063,
+            0.018190335050681287,
+            -0.01590174153215615,
+            0.011103110869700727,
+            0.0025189370696524936,
+            -0.00920305384736575,
+            0.01929652151514404,
+            -0.006196715938358087,
+            0.011426611271994685,
+            0.007058186102245829,
+            -0.018191352599117545,
+            0.0027202185072220937,
+            0.057310566886380025,
+            -0.000531639834386418,
+            0.01494975681909879,
+            0.03845692752400366,
+            -0.001971953300066856,
+            0.045554909803324765,
+            -0.02532332999916258,
+            -0.042616337063845953,
+            0.02633963974172818,
+            -0.02970705577683783,
+            -0.01624154998005974,
+            -0.02620993449678864,
+            -0.021490817420853802,
+            -0.004038424045768319,
+            0.009940887982818171,
+            -0.02692933845395842,
+            -0.023801570856681796,
+            -0.001948460540662095,
+            0.02290911497874141,
+            -0.03300532378029689,
+            0.006525488380791647,
+            0.007039306647666766,
+            -0.003685585408710752,
+            0.03183735852538168,
+            -0.027197064244321705,
+            0.006861928286688311,
+            0.011209869849682412,
+            -0.001094897323854524,
+            -0.005328676992017907,
+            -0.004007474089818478,
+            0.010796946231051235,
+            -0.030543173497906445,
+            -0.012879972141706173,
+            0.00935894527375688,
+            0.0017573071057611829,
+            -0.010050882212831222,
+            -0.03102132152986303,
+            -0.002915141291985554,
+            0.007634137769426993,
+            -0.011094093612593696,
+            0.01199609810193786,
+            -0.010458016758814742,
+            -0.04929347607956551,
+            0.010973132159729983,
+            0.028148117094907874,
+            -0.025504019123994758,
+            0.012470127357818321,
+            0.009182729379865986,
+            -0.016020133756226275,
+            0.016320111225714855,
+            -0.012051271446461976,
+            0.024098195821827578,
+            0.04317620443481784,
+            0.024592886889234396,
+            0.007727787037954994,
+            0.00771746776156926,
+            -0.0003147538723468374,
+            0.022572868970938624,
+            -0.016683340264320543,
+            0.013672085891597506,
+            0.020827021901755415,
+            0.02092309219721837,
+            0.0057130785714458055,
+            -0.003901728981013356,
+            -0.0036968415923910314,
+            0.002227024825579113,
+            -0.02106678364798051,
+            0.014578546755457675,
+            0.03128101864424301,
+            -0.005234627629617124,
+            -0.022394744692442837,
+            0.002430617645746389,
+            -0.015780389432069438,
+            4.7277885648917e-05,
+            0.013357803020920043,
+            0.013777651543690907,
+            -0.027713499926132385,
+            -0.0018999112228756628,
+            0.028364630573666852,
+            0.009146793705085586,
+            -0.009891297866910313,
+            -0.01644269527910728,
+            -0.0003903768506971414,
+            0.006512217314869576,
+            0.02408785204739448,
+            0.0018406499545367782,
+            -0.007360473750031044,
+            0.01484497577088404,
+            0.016723277095327158,
+            0.003629911330658209,
+            -0.0004820567356447713,
+            0.02394406451074397,
+            0.023794019615473026,
+            0.001632266615911801,
+            -0.02064379152040184,
+            -0.02755693521684878,
+            0.027335938400284726,
+            0.02769248863677192,
+            -0.006832353240623308,
+            0.027324973360715134,
+            0.003937207909967385,
+            0.013580154030133622,
+            0.02076818768224654,
+            -0.01784151382870951,
+            -0.017617903734311997,
+            0.0017316476451235418,
+            -0.009605294810027804,
+            -0.009829196696575069,
+            0.004891780008779742,
+            -0.011655402383771472,
+            -0.01124379942906224,
+            0.043205454954628145,
+            0.004713701851783307,
+            0.003560551284131962,
+            0.007894840447501912,
+            -0.018881214526749082,
+            -0.00945538175144614,
+            0.0019040259741796817,
+            0.029505863720301916,
+            -0.001400002868254808,
+            0.03315866433970851,
+            -0.00830542045504805,
+            0.008602408964961391,
+            -0.03202191233287069,
+            -0.006978289755114341,
+            0.0326499698728755,
+            0.015314305362524091,
+            0.015104436300213667,
+            0.0021257042489266787,
+            -0.0010840650756005028,
+            0.02434348355847526,
+            -0.007174433778817674,
+            -0.04839373063710409,
+            0.022696994450018916,
+            -0.004779511451775743,
+            0.023588216956749154,
+            0.0382733980620746,
+            -0.004232286553944853,
+            0.003815507826545929,
+            0.023531223314846965,
+            -0.01887650634213847,
+            0.015055403379782985,
+            -0.01918944132635701,
+            -0.017597900721414636,
+            0.02176630010543207,
+            -0.008896608357798277,
+            0.01732387427588561,
+            0.01764363756727683,
+            -0.012180043444768312,
+            -0.019003360499321047,
+            0.01229139850966224,
+            0.0025710091163655626,
+            -0.017075658234265562,
+            0.01797412486853335,
+            0.018372010308966135,
+            0.006723179570342314,
+            0.015417677746651683,
+            -0.00047634684109332134,
+            0.017608296628666153,
+            -0.001049984114818522,
+            -0.008416990841821452,
+            0.01827827907328822,
+            -0.009568717277044265,
+            -0.01658089563620308,
+            0.009808478422454319,
+            -0.05193977747210182,
+            0.010644335931025957,
+            0.0016781339270411392,
+            -0.01562285100294917,
+            0.00405036330963539,
+            0.007102724536527262,
+            0.005195069570261247,
+            0.0028773630901681767,
+            0.004610474981005765,
+            0.042921486858787584,
+            0.013213672397112759,
+            -0.0064030584930089415,
+            -0.010711457508461704,
+            -0.017480309425763254,
+            -0.027151012983864797,
+            -0.005204425551873271,
+            0.009927884476991038,
+            0.022601230152836282,
+            0.008046297278661014,
+            -0.0031858480011267704,
+            -0.02589628103025927,
+            0.019723430185137954,
+            0.03604835658149313,
+            0.062423099201768716,
+            0.010930268386637612,
+            0.02925564269324381,
+            0.02439517070384333,
+            0.023930432096006114,
+            0.013036828091946605,
+            0.006910311087503408,
+            -0.032995841694293995,
+            0.012813650252889403,
+            0.01333714655191133,
+            0.03678966264802025,
+            0.012571138353467183,
+            -0.05610151838497368,
+            3.0777966587107344e-05,
+            0.009791774904454618,
+            0.041959488802722524,
+            0.01956296954085397,
+            0.000155217569370113,
+            -0.007285602454264675,
+            0.031547870838711856,
+            -0.026518385135879206,
+            -0.012299593280363843,
+            -0.0026551378470968594,
+            -0.006190117573235779,
+            -0.003807907001161504,
+            -0.004991957475076049,
+            0.036983035940964804,
+            -0.015648565314743963,
+            -0.019256463332683493,
+            0.0004300362961407231,
+            -0.0015686065121077472,
+            0.0011311524136286674,
+            0.0032236641835276464,
+            0.013795345278976164,
+            0.015385029265915279,
+            -0.004800448639353721,
+            0.026514926384796363,
+            -0.008699453553651681,
+            0.022186817580012775,
+            -0.013427553742493343,
+            -0.011927259975371873,
+            0.017776228476138103,
+            -0.021243797085501972,
+            -0.01667280589187595,
+            -0.003688045116580565,
+            0.006189970874536939,
+            0.0020911968266404467,
+            0.0006097761986763103,
+            -0.004685468039463817,
+            -0.01011357666327168,
+            0.022631082517271542,
+            0.012398000826637277,
+            -0.006721588925828951,
+            0.018142966317821054,
+            -0.0021330638043572654,
+            -0.011869184647489572,
+            0.025073455250141923,
+            -0.0015392945266926425,
+            -0.0337085890598252,
+            0.01267302113143573,
+            -0.013438601165482187,
+            0.02673290895281225,
+            -0.009586762706902736,
+            -0.011208527168019762,
+            0.019144928964011168,
+            -0.0017591316941248897,
+            -0.005459150067129732,
+            -0.029230079356228025,
+            0.004679497471366242,
+            -0.011723195942933286,
+            0.015899377439148873,
+            0.0022054234777254627,
+            0.029187759252864872,
+            -0.018290694895057157,
+            0.00792533824593637,
+            0.011719198071061566,
+            0.014612375997487872,
+            0.005419943379355563,
+            0.010515265723144554,
+            -0.006893314445458116,
+            -0.04424495426064596,
+            5.8241472764594054e-05,
+            -0.005719143565673274,
+            0.0038220021965914754,
+            0.013106940982933836,
+            -0.01203955335201393,
+            0.011260626096963215,
+            -0.0279896016489392,
+            -0.0295228261396555,
+            0.036474811902902415,
+            -0.04900747926861378,
+            0.019261089742811332,
+            0.00014104646541594166,
+            -0.03057197731717424,
+            -0.04331886560073797,
+            -0.03232957159604667,
+            -0.019401402365464362,
+            0.004102283356333861,
+            0.0002610070704924167,
+            0.0012280372705500212,
+            -0.014025012323929194,
+            -0.014135990809256075,
+            0.013269444858490211,
+            0.00992155580941576,
+            0.011225162609266308,
+            -0.006545044517282357,
+            0.002368862850663672,
+            -0.02063183105457681,
+            -0.026981383275694142,
+            -0.02774194980145225,
+            0.013337303400678109,
+            -0.0025902548982556788,
+            -0.02794339093668326,
+            -0.025606606546251152,
+            -0.0029852498102770587,
+            -0.01906873442473352,
+            0.006508845592362198,
+            0.03268861529625085,
+            0.0355975143313428,
+            0.010066906387952277,
+            0.01690250467284985,
+            -0.017201544758038183,
+            0.048345309114140714,
+            -0.019031171948182872,
+            0.03188329748476941,
+            -0.0005845169448621133,
+            -0.024140202991373587,
+            0.00011597263700677706,
+            0.03203623528903764,
+            -0.019285565857567896,
+            -0.01332755507465857,
+            -0.010914187315481855,
+            -0.003374912271519852,
+            0.02419326154642231,
+            0.02443825785057301,
+            -0.0037923303809811746,
+            0.00022800399604730175,
+            0.015893265651968885,
+            0.003154527829977343,
+            0.007688529734165207,
+            0.030623379985627763,
+            0.014052282574575454,
+            0.01127235345371133,
+            -0.04164707563955819,
+            -0.011811587719246128,
+            -0.003763086694358232,
+            0.005909410979841885,
+            0.002913220159138154,
+            -0.002671613084376487,
+            0.0045765194162907885,
+            -0.007241703279856166,
+            -0.005855949422953439,
+            0.0016973541436777935,
+            -0.001576326424468957,
+            0.008393873711385013,
+            -0.0037217094191171083,
+            -0.03498879678492834,
+            -0.026096985039773694,
+            -0.021280075788049788,
+            0.006437497348154639,
+            0.0070214556591898375,
+            -0.00972721128392027,
+            0.01949553762684561,
+            -0.0029404319702066146,
+            0.000271037702827666,
+            -0.010734867909488861,
+            -0.0029784341515646075,
+            -0.01206464664516104,
+            -0.019544778160755956,
+            0.00035139045159739705,
+            0.013457601448003152,
+            0.00025947376767512006,
+            -0.03165522602526639,
+            0.011979811705146604,
+            0.006642849985850598,
+            0.013181162886675883,
+            -0.024499404540711422,
+            -0.02310714308893576,
+            -0.028028912907092744,
+            0.010570122662701183,
+            -0.0066866321204157725,
+            -0.024617571284817428,
+            -0.004097807033448707,
+            -0.0074576460628928256,
+            -0.0020417256533259887,
+            -0.02470953739335365,
+            -0.011408461668188081,
+            0.023537349682260345,
+            -0.006373652338136973,
+            0.01936338676879376,
+            -0.017636435070085086,
+            -0.017036091876814843,
+            -0.002198685044404603,
+            0.015584758061011048,
+            -0.038943923931820196,
+            -0.002076529978164001,
+            -0.008452513321945455,
+            0.006551055965075869,
+            -0.028840587857625097,
+            -0.007128279504360441,
+            0.0199397499488538,
+            0.012424319366271584,
+            -0.03518875279748495,
+            0.013596027106451115,
+            0.022191403935861113,
+            0.01218507466930854,
+            -0.0038551718784437783,
+            0.03910075115429626,
+            -0.0025215641203030336,
+            -0.005422688956956598,
+            0.016759888275451905,
+            0.04462538468871483,
+            0.023708097134660096,
+            -0.03152743757744915,
+            -0.008668774531128668,
+            -0.015224825785549947,
+            0.004102648962663789,
+            -0.025867795193966987,
+            0.004676262922216584,
+            -0.011739753544166856,
+            0.01645951757031186,
+            0.011253400919230348,
+            0.013958707917471253,
+            0.0376596076542735,
+            -0.005077354977296842,
+            0.02668910096392324,
+            0.0010332284250006117,
+            -0.00856723931523911,
+            0.0014431148305642308,
+            -0.01412579094241589,
+            0.029892393553325205,
+            0.02269599036836208,
+            0.014421409701746036,
+            0.029248195108770955,
+            -0.03588867295960255,
+            0.02458655544632528,
+            -0.0010013539341895672,
+            0.008368648583431747,
+            0.018150734970019018,
+            0.02628022565029391,
+            0.015423867373844644,
+            0.012477080028047823,
+            -0.02513018750675568,
+            -0.022180001014682045,
+            -0.0027550350630288997,
+            -0.031944843677988374,
+            -0.0014668093006914267,
+            -0.009789309083948147,
+            0.026386027599666254,
+            0.0046988411391149,
+            -0.019803160322638583,
+            -0.003931806845108628,
+            0.004482097428470392,
+            -0.00244660100774939,
+            -0.010441732346892935,
+            -0.004649318716095894,
+            -0.01869464992812615,
+            -0.017838643679294407,
+            0.023677878122153204,
+            -0.023958455640817666,
+            -0.025238725541146534,
+            0.014582798102040157,
+            0.0013223590277629197,
+            -0.002295641913710489,
+            0.0008238518461567685,
+            -0.02115533275282518,
+            -0.04214275933987015,
+            -0.027817083427786927,
+            0.0037588803464689412,
+            0.03565056002940463,
+            -0.026788865866928923,
+            0.014162550819567465,
+            -0.011336583458519,
+            0.019175861037658344,
+            -0.005127229083828615,
+            0.015531116410874209,
+            -0.018251290727445846,
+            -0.044630374763679266,
+            0.021833307809708984,
+            -0.012087009688208446,
+            0.02720742725750304,
+            0.021497809277370574,
+            -0.026937653159100156,
+            -0.013572733244188407,
+            0.01358214612023832,
+            0.0020408253706382332,
+            -0.009652962552750464,
+            -0.0063411929391088626,
+            -0.010533309167065193,
+            0.003901271588567942,
+            -0.04496345296706673,
+            -0.008761188928422701,
+            0.06948704056954039,
+            0.005916180375965137,
+            0.03157905519638486,
+            0.014668818318138927,
+            -0.028448496686381258,
+            -0.04008949536631331,
+            -0.00462253589823598,
+            -0.0003701522833020314,
+            -0.00024410762276269235,
+            -0.01850585249309217,
+            0.017699615637510956,
+            -0.002712069912576265,
+            -0.009767338815933488,
+            -0.013119464222297351,
+            -0.013743610418034631,
+            -0.030375365173543157,
+            -0.0030316327877227783,
+            -0.008632689470358149,
+            -0.006565457264917239,
+            -0.0328089275944709,
+            -0.011713996290006943,
+            0.029773910045828737,
+            0.02740100904353622,
+            -0.0061926915497348915,
+            0.00638737826122608,
+            -0.010244154924811975,
+            0.0010396203272677069,
+            -0.001264454445070139,
+            -0.004059969507786729,
+            0.008715730726831364,
+            0.0014767015415289076,
+            0.031074767756160294,
+            0.004761858344125113,
+            0.00987870172044607,
+            -0.059926117793699066,
+            -0.009294021263133636,
+            0.021188432871397475,
+            0.016602826531924377,
+            -0.009125784136292983,
+            0.013946188378030194,
+            -0.0017670794602549428,
+            -0.006837334179368295,
+            -0.028290055353601393,
+            0.024556541126233806,
+            0.04543639597357669,
+            0.0031311343860416473,
+            -0.011199639773247286,
+            0.014965677855595444,
+            0.0018248533506196013,
+            0.015507721118783542,
+            -0.011409402378400194,
+            -0.01338737990904486,
+            0.03063410115541065,
+            0.009305362935335817,
+            0.0025578631174065385,
+            0.012727648343848194,
+            -0.0038680554900795763,
+            -0.01173215289303824,
+            -0.01568862495293563,
+            -0.004101703129244437,
+            -0.03649200408657064,
+            -0.0022107849490954137,
+            0.009659780679763728,
+            0.0034091693454195806,
+            0.008848380026939594,
+            0.01842435611499587,
+            0.03546926214224266,
+            0.013478278853234764,
+            0.005970293629431872,
+            -0.0069197533761813455,
+            0.023196445250565058,
+            0.010403914879205351,
+            0.010017272564548626,
+            0.015076807790462658,
+            -0.029649260409273275,
+            -0.012737152920662164,
+            0.00373276777994811,
+            0.013239909450830258,
+            0.0001682544254742693,
+            0.02030652051051447,
+            0.014143961923579586,
+            0.0010147733405793957,
+            -0.012509446388074923,
+            0.0020437769941487626,
+            0.007405578085797006,
+            0.013458276946120546,
+            0.0006779114540819997,
+            -0.004250745498730309,
+            0.0009149988500358458,
+            -0.008355906072311008,
+            0.040391939952386505,
+            -0.01831218885700067,
+            -0.016793844108223176,
+            0.035064882870542116,
+            0.11416006426189942,
+            0.0010630856112440689,
+            -0.015660014600661857,
+            -0.017142534874931403,
+            -0.0086899013413391,
+            0.016611825572132997,
+            0.01609650977226868,
+            0.009780106186629793,
+            0.009622367309076686,
+            -0.000672926017804395,
+            0.024310949465350565,
+            0.02944698490225121,
+            -0.0037169531700178868,
+            -0.010990838766452194,
+            -0.036819646475041955,
+            0.022923962979125287,
+            -0.004596366586768123,
+            -0.007108041562635549,
+            -0.002604794313552557,
+            0.012641223813290853,
+            0.013668775645501733,
+            0.005125100040230003,
+            0.0077993841846437815,
+            -0.011500851674334447,
+            -0.017334458072700107,
+            0.0056819751688881774,
+            -0.015982234644378358,
+            -0.0076533013358874935,
+            -0.013127172654362694,
+            -0.011837346554842119,
+            -0.006562312362371076,
+            0.0029877420008976425,
+            -0.008572723854050838,
+            0.00854083969164562,
+            -0.010939639783598493,
+            0.016185642759084087,
+            -0.008052282332943615,
+            -0.0011992212936609377,
+            0.0010378646897070758,
+            -0.0025399515501741718,
+            0.010712865639618352,
+            0.01451357949543628,
+            -0.02521401163091652,
+            0.020836357346214075,
+            -0.008252813367155418,
+            0.0006409807249997337,
+            0.015338809580802425,
+            -0.04672526649678653,
+            0.022728876784937357,
+            0.005129225753457052,
+            -0.018372222092794564,
+            0.0066182736895667805,
+            -0.02048375780196863,
+            -0.00896382274306482,
+            0.0021738974344880794,
+            0.004491167150658828,
+            0.0034280799511763956,
+            0.015611133773752265,
+            0.018126362497548308,
+            0.00014383671280220997,
+            0.007927708693242186,
+            -0.006973076292062532,
+            0.008118313055163714,
+            -0.006791836664375275,
+            -0.01164567445668625,
+            -0.010894577039363014,
+            0.00014731542306100506,
+            -0.008865132658571428,
+            -0.011415696755798914,
+            -0.017922318360382588,
+            0.00358538001727405,
+            0.00289327637128695,
+            0.010406054711003452,
+            -0.007757607207626101,
+            -0.010994775479337082,
+            -0.011716096880261664,
+            -0.0013333474316266438,
+            -0.01231144419953554,
+            0.009792603083441917,
+            -0.02884953407490196,
+            0.0029097409399144306,
+            0.025277274938032477,
+            0.009263530315933738,
+            -0.028037993274175277,
+            -0.02532252134134532,
+            -0.0018596440554143638,
+            -0.013847737486669237,
+            0.009910107581119272,
+            0.013952962280235523,
+            -0.011807495289410774,
+            0.003896325037406752,
+            -0.02144921729633576,
+            -0.01628545945817568,
+            -0.0040148671532777005,
+            -0.016032493124255323,
+            0.004580041957001551,
+            -0.011375180364988195,
+            -0.007789996450583554,
+            -0.022041665680945066,
+            -0.005890065942573431,
+            -0.050455058269193866,
+            0.014401045842891128,
+            0.030219548461217365,
+            -0.004580747668649223,
+            -0.028684680969792875,
+            -0.015525870112797619,
+            0.011121401958547706,
+            -0.008879749268609245,
+            -0.019454243014778435,
+            -0.007839434838464604,
+            0.018908534797107714,
+            -0.0019283067432864216,
+            0.0024769961730655835,
+            -0.006732653593696035,
+            0.027604805021030646,
+            0.01366593691107021,
+            0.03740746059041736,
+            -0.014662268336336578,
+            0.02539666271785517,
+            0.02371809783638648,
+            -0.036126848124425154,
+            -0.007627445211958312,
+            0.03657481892016626,
+            0.01092515982671624,
+            0.0011046573579274704,
+            0.012350047525582187,
+            -0.006340802546498992,
+            -0.031330519599469066,
+            -0.0013691360532301728,
+            -0.0014676630169621005,
+            -0.023111265345539388,
+            0.027359543575867225,
+            0.018091458372134273,
+            0.014237156751820934,
+            0.0030993618578507172,
+            -0.0014037082734049385,
+            0.013257269860435359,
+            0.01021315143964527,
+            0.030885417052806473,
+            -0.003225065956381245,
+            -0.012825341172770057,
+            0.013476039084668826,
+            0.01852164843904835,
+            -0.009965495459579544,
+            0.012541079912270015,
+            -0.00824122613985912,
+            -0.018124134280975132,
+            0.03192263766222747,
+            0.0001484308749572035,
+            0.010888733498667343,
+            -0.010649515858317311,
+            -0.013082335214246402,
+            0.032614733822945205,
+            -0.00999075761269065,
+            0.004257643407198955,
+            -0.030723533031261387,
+            -0.003820776505926347,
+            -0.010471792928758869,
+            0.01737608180148319,
+            0.004597677757231315,
+            0.02095148515295301,
+            0.029253627700524396,
+            -0.011604450820008121,
+            -0.01996325509217509,
+            -0.001158626158150122,
+            -0.020583793738536985,
+            0.004168757459099133,
+            -0.03632338333261702,
+            -0.019373660230711246,
+            0.022259216603413172,
+            -0.010364620831688015,
+            -0.004751193403783363,
+            0.00444187509644351,
+            0.0008572202661850934,
+            0.014943779567410408,
+            -0.025408263453313398,
+            0.017755758370543148,
+            0.004627392860417132,
+            0.007402073071525438,
+            -0.024913329140362466,
+            0.013760771254234878,
+            -0.006087346663657887,
+            -0.03780469033982951,
+            0.005509388120163784,
+            0.03756141956381493,
+            0.02381019168406692,
+            -0.024580249929652097,
+            0.01807584361080568,
+            -0.03111039245576191,
+            0.023245542501125073,
+            -0.004868090951368273,
+            5.1544460403943004e-05,
+            0.00640235757127173,
+            -0.004698373089469779,
+            -0.00621383035264185,
+            -0.0025132415458431587,
+            -0.008746634331769936,
+            -0.01611801559493664,
+            -0.01193535623883546,
+            -0.03381789115349148,
+            -0.024119754887378317,
+            0.013881102852385332,
+            0.0019232668966222687,
+            -0.0095263729740736,
+            0.018652966857927805,
+            -0.03775912497511837,
+            -0.012022352025516115,
+            -0.023289377270709736,
+            0.023579672990954228,
+            -0.028605627815260902,
+            0.019235397553197357,
+            -0.025838655372560505,
+            -0.021458526449310157,
+            0.01508157885177254,
+            -0.010064396230734452,
+            -0.0016708624560165368,
+            -0.015042071642751917,
+            -0.03670785416972153,
+            -0.006267614265977843,
+            0.02298763967146357,
+            -0.012506444728541897,
+            0.00029871598584066644,
+            0.02671390618094565,
+            -0.003059865833779805,
+            -0.011103452976820622,
+            -0.0017382684168257506,
+            0.010923055147701782,
+            -0.02938686260440486,
+            0.014300097108092281,
+            0.0038571851420598074,
+            -0.0037614499297117365,
+            0.0010991440540809893,
+            -0.010108413486243776,
+            0.02127952161226619,
+            0.009142545227414532,
+            -0.0030260119510809205,
+            0.040589877142650374,
+            0.008677453889680373,
+            0.0017339508605300814,
+            0.008732950895530873,
+            0.01063352447960556,
+            0.006618635029823979,
+            -0.014986307792626314,
+            0.016905861015681957,
+            0.013101237900847744,
+            0.0002162791516784468,
+            0.019309343995079022,
+            0.0021263376627204313,
+            -0.024516267132902416,
+            0.013339486637573757,
+            -0.02311659751478777,
+            0.0020575982953961246,
+            0.02391695586495566,
+            -0.020440172706421535,
+            -0.001656132876701879,
+            -0.006304133667169861,
+            0.000543457661904495,
+            0.022135707414160034,
+            -0.0230903871986617,
+            -0.023444763940153112,
+            0.009604600033024832,
+            0.01721387169484476,
+            0.01827905276638783,
+            0.02824773106648473,
+            -0.007100625424382998,
+            0.01348510178676837,
+            0.0002604386361122316,
+            -0.011802178189607182,
+            0.007664157334475143,
+            0.0074175377458313594,
+            -0.0044008460682833875,
+            0.025491035291032547,
+            -0.0017015687016376205,
+            -0.006134010425966364,
+            -0.005379657499404915,
+            -0.01584573519366957,
+            0.0034386456145185213,
+            -0.013860742817569676,
+            -0.013355238174583658,
+            0.0040342831827141845,
+            -0.0208160950322342,
+            0.10904592104153767,
+            -0.0037038089210379766,
+            -0.019472012964045247,
+            0.0063554319446707044,
+            0.003716719884684827,
+            -0.003692786470980806,
+            -0.003353843210051379,
+            0.007788605566526459,
+            -0.011254480693190834,
+            -0.0045751956417398095,
+            -0.013976039724381875,
+            -0.001989262807706565,
+            0.010067374586433014,
+            0.009201983771978006,
+            -0.005902980676079493,
+            -0.02029714318363332,
+            0.013031676901335606,
+            -0.016970144750162643,
+            0.008874670945038486,
+            0.0008311255074953139,
+            0.00884167159995975,
+            0.001716603195908536,
+            0.01854951250316076,
+            -0.009310881405486074,
+            -0.01325242738760595,
+            0.029667589863813217,
+            -0.005616511284128333,
+            -0.0038179509972925954,
+            -0.0004690716809300012,
+            -0.005068974756744239,
+            0.0021204231125051274,
+            -0.027958274497127258,
+            0.022127088236172387,
+            -0.017271838492631593,
+            0.010267709087809265,
+            0.01316800560257366,
+            0.005599367128954229,
+            0.01817806970002991,
+            -0.007115594036834107,
+            -0.02627643123342057,
+            0.006542742528003751,
+            -0.003433874053082844,
+            0.006631427284537971,
+            0.02316295746074422,
+            -0.014573351469966297,
+            -0.033520864104177005,
+            -0.013707022143307006,
+            -0.01877574599842729,
+            -0.005992004412448576,
+            -0.0043698881826095435,
+            -0.01481352026460129,
+            0.006424121402963946,
+            0.01382938882387991,
+            0.018136021761515388,
+            0.0012697483790911433,
+            -0.011531136974861559,
+            0.00609187274277241,
+            0.01076133059240578,
+            0.008400887242930113,
+            -0.01533504932731406,
+            0.011439227832642668,
+            -0.028718487721012004,
+            0.01522697877548034,
+            -0.012452665599305595,
+            -0.024938748080573947,
+            -0.0022082450217489366,
+            0.017589399637694264,
+            0.0037824832255756763,
+            -0.004355445812154662,
+            -0.018778668474337727,
+            0.019825087149714933,
+            0.024692878851503508,
+            0.003883497401229232,
+            0.019470881143244197,
+            0.041834635328088364,
+            -0.003544509491805351,
+            0.027139164223766474,
+            -0.02129723143323512,
+            0.01461354860170749,
+            -0.03441729954693155,
+            0.001893713808284104,
+            -0.004222657393633789,
+            0.0013677766871517654,
+            -0.003811928806805752,
+            0.020142937759360904,
+            -0.009158202577174012,
+            0.0030638058587375543,
+            0.024854053046139163,
+            -0.018665378030958756,
+            0.00805903734779081,
+            0.010625429315145887,
+            -0.012856237251902233,
+            0.028535299711292306,
+            -0.006335094045840286,
+            0.00305422126851784,
+            -0.0297883488914759,
+            -0.0029192872993071216,
+            0.0026067044379754047,
+            -0.006097749614529602,
+            -0.00968023644800927,
+            -0.007433916539502931,
+            -0.006202274026852405,
+            0.0038354607448269555,
+            -0.021471361566683354,
+            0.003195646827154078,
+            -0.004390751282171024,
+            -0.003323447423321121,
+            0.02518759723554948,
+            0.025857746688368054,
+            0.016893707217894103,
+            0.012739170140289531,
+            0.008553625305650129,
+            0.019805351588748434,
+            0.01654964927067909,
+            0.011549329129728739,
+            -0.020026013396046632,
+            0.01150644479272541,
+            -0.004159611921476011,
+            -0.0022937297602724912,
+            0.014815466072709131,
+            -0.01572171840394285,
+            -0.02276187015864222,
+            0.024693009795978448,
+            -0.019725315346836404,
+            0.00998768979020729,
+            -1.1702210281215188e-06,
+            -0.011129745730507545,
+            -0.002591293237053776,
+            -0.018034357254990988,
+            0.00012070821244376275,
+            -0.023051045150566527,
+            -0.028465270110388716,
+            -0.00048603008831669926,
+            -0.000574546300981984,
+            -0.012992623897880395,
+            0.02064747165046032,
+            0.014820638761258362,
+            -0.024820174379739057,
+            0.02666518374464728,
+            -0.017430076811109155,
+            -0.01495051200726725,
+            -0.005861181182847845,
+            0.00100881727574559,
+            -0.0055223282819321295,
+            0.0034719859452138017,
+            -0.010180752654175722,
+            -0.009961678082468036,
+            -0.012134423047419183,
+            -0.028073672819317354,
+            0.011574683159318163,
+            0.016120674453124317,
+            -0.015043147648505721,
+            0.031834693135959095,
+            -0.008802805725050891,
+            -0.0018698230152007877,
+            0.016179742266521146,
+            0.005096232332051617,
+            -0.017187077376166815,
+            -0.018999569809271876,
+            0.012251308792871571,
+            0.013279090498325945,
+            -0.009588297039279112,
+            0.006630431200804809,
+            -0.014130176783207386,
+            -0.005092662672039885,
+            0.007020236623084732,
+            -0.005049363863784624,
+            0.0055051712211635915,
+            -0.00995314895304919,
+            0.0052743456988306485,
+            0.005841370337169303,
+            -0.0029434616355351975,
+            0.004677049987338193,
+            -0.018568941264514256,
+            0.020031253265629777,
+            0.0018725390936655962,
+            0.01332532735283085,
+            -0.03518855189467458,
+            -0.0056298723325832025,
+            -0.005369475318146738,
+            -0.004650364775945294,
+            -0.020833510091412353,
+            0.020445280915606132,
+            0.0159991245532085,
+            0.013202409908533162,
+            0.023094507005426774,
+            -0.014905666713082246,
+            0.01572052456975675,
+            -0.020945766915121455,
+            0.013114000985348841,
+            0.003349101658930482,
+            -0.019829605346106578,
+            -0.001423750749031516,
+            0.04557459007627153,
+            -0.005847231973329081,
+            0.011986110554869603,
+            0.0012709842140249691,
+            0.026758922819943853,
+            -0.0001635536762342011,
+            0.0065419519929442995,
+            0.005997692702992206,
+            -0.009935128108338126,
+            -0.0008057798606777003,
+            0.00865712443881613,
+            0.046254320605750854,
+            -0.01181132866321511,
+            0.008170688314239364,
+            0.013099051984057827,
+            0.0034971494510865624,
+            -0.0008386400616584293,
+            -0.015346506644390764,
+            -0.03067678541405639,
+            0.019305962963428017,
+            0.007736916644077731,
+            -0.012756525606221703,
+            0.008734673666457216,
+            0.015137771675476126,
+            0.012891824153867332,
+            0.003895886821685762,
+            0.02155154331885366,
+            -0.012761690789748027,
+            0.018528313792219413,
+            0.024663453412919027,
+            0.009077230086690327,
+            9.184526542175535e-05,
+            -0.000751529849753638,
+            0.006791927267880182,
+            -0.0004433479107716552,
+            -0.009832230053513483,
+            0.015858232096630655,
+            -0.014861341148074698,
+            0.017790924034108806,
+            2.7216806238946785e-05,
+            0.03744841066552559,
+            -0.015216207585985075,
+            -0.010138319613387346,
+            -0.007774325714088762,
+            -0.0016997034352120362,
+            0.003971564314593291,
+            0.0018686537276849653,
+            0.006051669054361943,
+            -0.0008319624938748146,
+            -0.017326561560776794,
+            0.01667737942301944,
+            0.005719808546053257,
+            -0.0007405637872110778,
+            0.013750752065593804,
+            0.0038732323799683453,
+            -0.00367645612814479,
+            0.023580906347714314,
+            0.006238541579764762,
+            -0.005235795826199542,
+            0.0028182323816636243,
+            -0.006443210286549713,
+            -0.016255209299456314,
+            -0.003061441395209219,
+            0.004597970789372988,
+            -0.0016590920570710628,
+            -0.011138261849069419,
+            -0.0019423173609729848,
+            0.020355534503272887,
+            -0.004612410336124759,
+            0.013346750536413418,
+            -0.005400060374036387,
+            0.006252154576727367,
+            0.010601459925435238,
+            -0.012346569079682625,
+            0.0014154620309235822,
+            0.010406076751218612,
+            0.0013983417564564663,
+            -0.007140180075225315,
+            0.0010544410919327675,
+            0.007757599672903709,
+            0.006505350216372801,
+            0.00483063937962671,
+            0.00836630436955631,
+            0.013284451514874498,
+            0.0033749897293287438,
+            -0.006243293993633622,
+            0.0013949831763016042,
+            -0.013269921385800992,
+            -0.006308639321247727,
+            0.0010906169976769569,
+            -0.0010152835709869353,
+            -0.02017142197515583,
+            0.010787630901414118,
+            0.005355784839945777,
+            0.007838485470991034,
+            -0.01706932891696025,
+            0.009576030158827565,
+            -0.0008144311254998525,
+            0.02829832828245172,
+            0.007407953684956539,
+            -0.02769403281764041,
+            0.014266001222243134,
+            -0.010492199740754951,
+            0.022575782293320284,
+            -0.0016349316593207844,
+            -0.005656505606143177,
+            0.008025514398322551,
+            -0.009634281579228592,
+            -0.005991026158213869,
+            -0.022366083313133788,
+            -0.025825209755160946,
+            0.016871087849192924,
+            0.013583173973358155,
+            0.021125252303971605,
+            0.00012555627355712715,
+            0.015347535950809997,
+            0.0020128011217859444,
+            0.015164099887550086,
+            0.0024544903332736167,
+            0.012507136277293888,
+            -0.011798050869794805,
+            0.0104828761281283,
+            -0.0137805761114781,
+            0.014620357150314097,
+            -0.005246639228547017,
+            -0.006986408834884282,
+            -0.008849580196616098,
+            -0.005442165099441538,
+            -0.020891551850452786,
+            -0.009155930286949747,
+            -0.0013590151948670868,
+            0.010665872513236615,
+            -0.0005386423407222223,
+            -0.0027671778465775897,
+            -0.03237311596935737,
+            -0.004875062533405715,
+            -0.006670982068843501,
+            -0.01942244110096111,
+            -0.0045975373348006885,
+            -0.026137783023267008,
+            -0.01657173843643047,
+            -0.0007784046498232684,
+            -0.015037306676062236,
+            -0.006889680260149007,
+            -0.02009705080543122,
+            -0.027192867101205106,
+            3.0161964469866572e-05,
+            -0.009647644858034833,
+            -0.005380165957043753,
+            0.013815092649296373,
+            0.011902366689169847,
+            -0.01572592170758115,
+            -0.022321702398924333,
+            -0.01088937403483412,
+            -0.0165754067023749,
+            0.01646435743868707,
+            0.009914938007210463,
+            0.018834863207293908,
+            -0.003724245912768107,
+            0.005643505694722323,
+            0.0009581443359000331,
+            0.02327704258596552,
+            -0.02532422120223976,
+            -0.028694220894404807,
+            0.0019298778426469338,
+            0.0025970175688368323,
+            0.006086164866310174,
+            -0.009414665394188936,
+            0.001200390869257535,
+            -0.007782899559317819,
+            -0.0228264209954853,
+            -0.00898519580978142,
+            0.008156835270668449,
+            -0.010036630164522355,
+            0.02230664068547531,
+            0.012332230358869292,
+            0.013699197379189934,
+            0.010593587938184292,
+            0.01576174456497367,
+            -0.0001654241831584687,
+            0.014471905257729428,
+            -0.008675915624727887,
+            -0.0031070346099204425,
+            0.0038358404385932777,
+            0.005878417037187274,
+            -0.00625022998235275,
+            0.028077114529659795,
+            -0.0011836453716817525,
+            0.0002962695480296182,
+            0.014405782572641941,
+            0.02697667559474498,
+            -0.004710313427099278,
+            0.0210512043261309,
+            -0.002393165025068831,
+            -0.006135816286212752,
+            -0.008429776018235104,
+            0.008607617299702318,
+            -0.009529291136733401,
+            0.011133497451942684,
+            0.019864404836361926,
+            -0.014041315457579166,
+            -0.007351413712676263,
+            0.02950254974910368,
+            -0.014564388091367897,
+            -0.01825317106331622,
+            0.02325964952838705,
+            -0.007414295676409368,
+            -0.0039042587057605484,
+            0.00377144546378926,
+            0.007169856503233239,
+            -0.004036744966949995,
+            -0.012928341573802416,
+            0.016090527447479415,
+            -0.00011159893712484579,
+            0.014146137810890391,
+            -0.0004655839804281017,
+            -0.006372231942983921,
+            -0.010047382825720416,
+            0.002373224247019062,
+            0.026818813126567614,
+            0.013483627627077692,
+            -0.017895710143841333,
+            -0.013785856764479817,
+            0.010826610341182479,
+            -0.01538893818197323,
+            -0.004425206001945227,
+            0.02155595194812746,
+            0.0028768153668128553,
+            -0.020543027467698483,
+            0.008028975884363618,
+            0.0003835165718138906,
+            0.00037046947047892953,
+            -0.00029363296913641716,
+            0.030135739208689217,
+            0.001343516577762509,
+            0.00018948658255495327,
+            -0.00875053830853609,
+            0.008606573235959595,
+            -0.01280305089741873,
+            0.012786478836532904,
+            -0.02437993139337839,
+            0.007648181775166282,
+            -0.00629541989541625,
+            0.004579605315599212,
+            0.016658386927051592,
+            0.01218098627022323,
+            0.015031618813778413,
+            0.008237200519537417,
+            -0.008942965899756683,
+            0.0009093265012901892,
+            -0.025302175262520477,
+            -0.001333639155622843,
+            -0.017971876771137367,
+            -0.02036433247634014,
+            0.0002463842837918767,
+            0.022144712689630985,
+            0.018559482073124734,
+            0.011579840028464503,
+            -0.008657014440886656,
+            0.0166696911919226,
+            -2.9834620942882283e-05,
+            0.011845827764518911,
+            -0.00037901176785495324,
+            -0.006138374154553732,
+            0.004711058953619512,
+            -0.003622694588720417,
+            0.013542405259906128,
+            -0.014730567462694827,
+            0.012076044672004906,
+            0.01690223095263866,
+            -0.009108632851807137,
+            -0.008946251274440141,
+            -0.013992423874475428,
+            0.00048381889145447367,
+            -0.015909599902931202,
+            0.035489252674454214,
+            0.02252772349379229,
+            -0.01210464970923921,
+            -0.010469120693943236,
+            -0.014164999313775756,
+            0.019496048806752753,
+            0.010427539351244087,
+            0.0028018908451425276,
+            0.017091768406027965,
+            0.021395638416767112,
+            0.008817247593404623,
+            -0.006081063711583319,
+            -0.0021216503501144983,
+            -0.003781810594530787,
+            0.020143098218482827,
+            -0.003078172339607037,
+            -0.011350171884507555,
+            -0.012516379645909518,
+            0.005766102930704529,
+            -0.01626433668691071,
+            0.0019294643928566657,
+            0.008641558521764407,
+            -0.015057688533773374,
+            0.019407986696657772,
+            -0.028126197703647868,
+            0.012364374199657305,
+            -0.0044278434183731225,
+            -0.007251147925369614,
+            -0.03496220945653279,
+            0.006389489026974697,
+            0.008941259259632024,
+            0.003941752277454314,
+            0.023003838655784314,
+            -0.024528514847470928,
+            -0.018361931455105486,
+            0.0037791061321962543,
+            0.00047162060848090104,
+            -0.00711894577052754,
+            0.0010138871688747137,
+            0.004128392769213672,
+            -0.00674546985015164,
+            0.0017307097443508704,
+            0.0033354800298095546,
+            -0.002829898077850263,
+            0.013252947102333348,
+            0.0081724269619999,
+            -0.021449979263734438,
+            0.001368791856860285,
+            0.01590656734732547,
+            0.01452008402429858,
+            -0.02391847163018108,
+            0.006331397860689212,
+            -0.02064812934794864,
+            -0.000631340242718475,
+            -0.031272866000888556,
+            0.016859793097564958,
+            -0.0069008624795683276,
+            0.01899718801899956,
+            -0.0015377436942771224,
+            -0.0065810685260109004,
+            -0.012822620655323715,
+            0.02137604937338179,
+            -0.0019121137078344135,
+            -0.0025598772127157477,
+            0.015678293057066646,
+            -0.025700865215693987,
+            0.017825186616033165,
+            -0.0026230848396158204,
+            -0.017342967698019966,
+            0.017199139867162275,
+            0.004536019178633225,
+            0.007224666811309087,
+            -0.023505589665315567,
+            0.0007208041198255396,
+            0.003900164485347562,
+            -0.017250324025622685,
+            0.004470862192961212,
+            -0.030807400122398566,
+            0.0006579147990327597,
+            -0.013222907940559794,
+            -0.0020514994691204847,
+            -0.0046763235724575445,
+            -0.014050035576183385,
+            -0.039181147615786484,
+            -0.005429639279537522,
+            0.016065742005148733,
+            -0.0009029636981539927,
+            0.01800715198737081,
+            -0.002730931639143803,
+            0.002947680833843522,
+            0.004912692811020822,
+            0.00818378920043427,
+            0.006915895602313347,
+            0.02274116906914758,
+            -0.0057854922886970365,
+            0.00471999900678522,
+            0.007058014277137233,
+            0.014638081356210031,
+            -0.01719565951201012,
+            -0.015170377598372723,
+            -0.016015921063369496,
+            -0.006351213642269936,
+            0.01739665344303718,
+            -0.017488885259028475,
+            0.013303332445465686,
+            -0.00429428898520701,
+            0.014825338166443021,
+            -0.008686427208690176,
+            -0.01575945086724703,
+            0.0028564767084934386,
+            -0.007570417028925067,
+            0.001007670647768344,
+            -0.02323908008484089,
+            0.02019729564706863,
+            -0.00847581038203716,
+            0.010508144473197669,
+            0.01729824052022313,
+            -0.021364529179605813,
+            -0.009742851959432839,
+            -0.014342281441152413,
+            0.029896210049029176,
+            0.013925163184699139,
+            -0.009392252140772913,
+            0.003927230832659267,
+            0.008873561269230905,
+            0.011529536139797032,
+            -0.012209070214071445,
+            -0.0027679974125029362,
+            0.002411914349172495,
+            -0.0062056207925691356,
+            -0.002025687569076897,
+            0.03090134105002816,
+            0.0038285457265343245,
+            0.024877474647546086,
+            -0.013214997316123852,
+            -0.0033763842539861907,
+            0.00023379106197363714,
+            0.005134764372494098,
+            0.01183867975321486,
+            -0.0017966997954747142,
+            -0.008029791503028367,
+            -0.01179500476055557,
+            0.02599911330072515,
+            -0.021852083993691557,
+            0.007024505421035273,
+            0.026784024996764876,
+            0.001390202993990949,
+            0.0029956956045771046,
+            -0.013131691460546865,
+            -0.006298736864509989,
+            0.02523640664202355,
+            0.010030824059858138,
+            0.0041523782603032225,
+            -0.026465819482580182,
+            0.0032128899272377593,
+            -0.0061901409332948115,
+            -0.008681758126710213,
+            0.009866461769956222,
+            -0.0028095454857224164,
+            0.01188012041754054,
+            -0.012191996383095982,
+            0.028459939543805705,
+            0.010711502132045176,
+            0.01047154118931583,
+            0.025888652452698974,
+            -0.02079614305315134,
+            0.0042203106489994615,
+            -0.010829796140534066,
+            0.0006773493086316061,
+            0.005596640511307504,
+            -0.027510562413905437,
+            0.001969898603871003,
+            0.010390866563404346,
+            -0.005642706366801436,
+            -0.01882994179544761,
+            0.01187970346127799,
+            -0.019814385265412387,
+            0.024479579275617887,
+            0.003231341626501863,
+            0.020056914356101588,
+            -0.0053276448188067154,
+            0.009834685132275868,
+            0.0002742942778577165,
+            0.0031361331890882127,
+            -0.0008074598092128465,
+            0.009009012303347016,
+            0.005786777410144894,
+            0.013719989127068968,
+            -0.025528370602188686,
+            0.006821572597883499,
+            -0.0029197127345352626,
+            -0.030539872624857384,
+            -0.013383097680839732,
+            0.010291485622655026,
+            0.012218533188117704,
+            -0.009144425807890594,
+            -0.010378679518611277,
+            0.0030042579827305603,
+            0.005748243116238633,
+            -0.006692403885450407,
+            0.006913672151000981,
+            0.01629516912493545,
+            -0.002533517074771728,
+            -0.0019083178803585312,
+            -0.0035007340321378893,
+            -0.01154609935927537,
+            -0.010016998439072824,
+            0.023193275321990937,
+            -0.00482990342915347,
+            0.017943025772239603,
+            0.025873224580389702,
+            0.0039876567755322,
+            -0.0057577932736143506,
+            -0.005460363755206188,
+            0.01478435999841998,
+            0.014195457012279476,
+            0.004669295894868759,
+            -0.009702220679239768,
+            0.007974305230005956,
+            -0.007760086293456636,
+            0.008375077126777163,
+            -0.020260917110786147,
+            0.002199401871514877,
+            0.0026191373298767766,
+            -0.0022089262577237776,
+            -0.015213807046175918,
+            -0.01956788452420682,
+            -0.021898368819136302,
+            0.012634659463647929,
+            0.00034643102239488474,
+            -0.009241772775582378,
+            -0.013645250997151429,
+            0.0016665106990117016,
+            0.0025630500346914592,
+            0.0002095521560180936,
+            -0.007783022628694597,
+            0.009546147813008391,
+            -0.01265287317178631,
+            0.007445223311736028,
+            -0.004007029181436962,
+            0.0051846833906574515,
+            -0.0016916014415925766,
+            -0.016413938859402438,
+            -0.0210868129634149,
+            0.01920794362224444,
+            -0.012897042227841011,
+            0.02269303737601583,
+            -0.0014935878147854765,
+            3.062680855559304e-05,
+            0.0033132626755401546,
+            0.007274487651476732,
+            0.012110042834251237,
+            0.011426424896797977,
+            0.004307086698938411,
+            0.008206932254416481,
+            0.004673427176334148,
+            -0.013124571552764586,
+            0.004324963759928715,
+            0.0039216033705804084,
+            -0.010449887922072297,
+            0.005298323150690876,
+            -0.0010196552573491,
+            0.006128327172621373,
+            0.012167492341907742,
+            0.01923525204293908,
+            0.01908107550907106,
+            -0.0012031208519043073,
+            -0.0010601912677170903,
+            0.014474374858125607,
+            0.004540954805526089,
+            -0.004883537463424679,
+            -0.002772988629590886,
+            -0.013363713078069854,
+            -0.002572403646338979,
+            0.011541515690646252,
+            -0.02376161747414617,
+            -0.018215231053002656,
+            0.021831219299143226,
+            -0.004991882316041097,
+            0.033796174834149546,
+            -0.013140970283091536,
+            -0.016197177724633614,
+            -0.00464190076219332,
+            0.0026389653040644607,
+            -0.013788049032378498,
+            -0.010300859744205445,
+            0.010545541278382187,
+            0.013848738866656793,
+            0.009903590151120531,
+            -0.0005984085581408575,
+            0.02922816576443645,
+            0.0019900932217491907,
+            0.02245395592422119,
+            0.0018978407151245295,
+            -0.007029294098689854,
+            -0.01440575120349887,
+            0.010840472832356266,
+            0.005529335652955532,
+            0.009050748012161991,
+            0.017791269808484802,
+            0.014502057454389247,
+            0.006265567405788131,
+            0.013928705812292101,
+            0.0012903002372833693,
+            0.0010858377120770652,
+            -0.002355229463060164,
+            -0.01692711767136341,
+            -0.01456786846097011,
+            -0.01627640642218236,
+            0.003873864312554098,
+            0.012620350492241247,
+            0.00525890570023481,
+            0.0007393789113866752,
+            -0.007153210751899716,
+            0.009254566395058412,
+            0.0065087881512835375,
+            -0.012279152412209152,
+            -0.016530068283509593,
+            0.004060671955726967,
+            -0.02210570272789274,
+            0.007849467899009437,
+            0.01554233481902349,
+            0.00219885032769439,
+            -0.002655449524219968,
+            0.0006843524304842362,
+            0.006272944169764281,
+            0.005531927368102191,
+            -0.0036710328198476905,
+            -0.00544113514775027,
+            0.011264661763762068,
+            -0.006379654234440906,
+            0.009537150372938296,
+            -0.00290031548037575,
+            0.03718190949929003,
+            -0.011976080887998194,
+            0.012120408392610773,
+            -0.001751472085435711,
+            0.008481703962951994,
+            0.008067669857949423,
+            0.010772687289903202,
+            -0.003805089040741351,
+            -0.0022164543131735167,
+            -0.008753625032781523,
+            0.006751953442322185,
+            -0.01582177621255226,
+            0.013631301238130705,
+            -0.00030685726238985963,
+            -0.010728958775974582,
+            0.01055472059121978,
+            0.006788316621149835,
+            0.004413382316882087,
+            -0.003262024961860459,
+            -0.01972283203505511,
+            0.014937729522976615,
+            0.0018799533869990667,
+            0.019247008681353865,
+            -0.023276106189004046,
+            0.029043727846427652,
+            -0.02019024213343764,
+            0.023083643395685142,
+            0.01865875504942157,
+            0.001179436167200093,
+            -0.02070329911663634,
+            -0.007618131527214554,
+            0.014871709122115308,
+            -0.02659818889735726,
+            -0.006234719259825804,
+            0.005986829551040766,
+            -0.014639912579831785,
+            0.00037813423546252645,
+            -0.01924149909676064,
+            0.00010832498610532006,
+            0.011013130149723431,
+            -0.011782407678373089,
+            0.00438725832178584,
+            0.01608349750921335,
+            0.019347883652009713,
+            -0.009760017612545174,
+            0.003869246694835168,
+            -0.024240324890794748,
+            0.010958789926998691,
+            0.020511684636649225,
+            0.008883830392748497,
+            -0.01761944522796127,
+            0.021817973960311446,
+            0.010652131228027521,
+            -0.012575679152857703,
+            -0.0012753095453828826,
+            -0.028191274775392804,
+            -0.012421170815054556,
+            -0.02188248722244861,
+            -0.0011670175621882146,
+            -0.014436683528976771,
+            -0.0029783880907228383,
+            0.0009506986717516226,
+            -0.004719165363484719,
+            -0.01179163390107033,
+            -0.019543732196676634,
+            0.016606805875498898,
+            0.0227313413004497,
+            -0.006738735201626204,
+            -0.014547240981002825,
+            0.013866974327185123,
+            0.031694077725890536,
+            -0.016189209297529324,
+            -0.00542110554672736,
+            0.0028061798782555557,
+            -0.012964876141500577,
+            0.018946046750284537,
+            0.0039419760434645495,
+            0.000322115192590907,
+            -0.01991851369210666,
+            0.002489725351392778,
+            0.010835458511052207,
+            0.01684964045708685,
+            0.0005541314524621196,
+            0.010855852688653808,
+            -0.007549882748861744,
+            0.010096800951420057,
+            -0.011832260836468588,
+            -3.6179995603517093e-06,
+            -0.007800366796516945,
+            0.007001049318404058,
+            -0.004424776169203575,
+            0.034356203288485945,
+            -0.02817071276766299,
+            0.004989631403681216,
+            -0.014539644354369607,
+            -0.010888763939716889,
+            -0.0027322086509797843,
+            0.0004865534340981866,
+            -0.010775238981581461,
+            -0.005162220412866082,
+            -0.0011263271110752205,
+            0.00913081242511139,
+            0.005844327320514489,
+            -0.013886529519306775,
+            0.019171827605726503,
+            -0.030984919318656618,
+            0.004469458202753611,
+            -0.008553175891755315,
+            0.010511707988936336,
+            -0.0151185694875566,
+            0.013830711686954931,
+            0.01301959345025159,
+            0.008205984085944315,
+            0.007740367849216328,
+            -0.01532123052721834,
+            -0.01678314222392063,
+            0.001376468747213867,
+            0.0036953018347776244,
+            -0.00931855481896886,
+            -0.033391028922085425,
+            -0.007072913446166969,
+            0.005331350534425606,
+            0.011614453225043,
+            0.008156128603003278,
+            -0.02345206873048703,
+            0.005003870024809975,
+            -0.024178720646704597,
+            -0.012115112684614331,
+            -0.007750557665424973,
+            -0.011580152272614108,
+            -0.006117298954192534,
+            0.026449765548206453,
+            0.007989237291215073,
+            0.016691517078382376,
+            0.004954133565782356,
+            0.02742809127725913,
+            0.011670893621991404,
+            -0.011736840457474895,
+            -0.003349151747880166,
+            0.00035419034843196817,
+            0.005507651806218584,
+            -0.008706590052024251,
+            0.011793241980257321,
+            -0.0013295035978611142,
+            -0.0065061682758755625,
+            -0.004398354347597199,
+            0.009372816666628725,
+            0.014188676762501037,
+            -0.003007980559117429,
+            -0.008724255229333351,
+            0.015749776067989826,
+            -0.030899793617096834,
+            -0.015215033379100091,
+            0.0016152308157299517,
+            -0.007813175908301852,
+            0.017806770680528907,
+            0.01169056345888971,
+            -0.026835412610223673,
+            -0.004538235170332766,
+            0.010532316940606419,
+            0.004755930024691789,
+            0.0095408889025506,
+            0.00954782703036282,
+            0.012079064562163901,
+            -0.007829549014214643,
+            0.01046722658021748,
+            -0.011519369840999348,
+            -0.007092961453184232,
+            -0.007241799314569802,
+            -0.008077934089061052,
+            0.021467547268987214,
+            -0.025816030568238717,
+            -0.006558853543772633,
+            0.01134670818701549,
+            0.013027491377727365,
+            -0.002301991242932548,
+            -0.006299073927206214,
+            -0.017357989759336386,
+            0.008311464809363397,
+            -0.013922568511594478,
+            0.025633907424998367,
+            -0.004854655498817219,
+            -0.005336807899100369,
+            -0.005228056880188717,
+            0.01958682721272445,
+            -0.016774409737540904,
+            0.02072573220949266,
+            -0.014449250679704904,
+            -0.007857351393427215,
+            0.001673929749689194,
+            0.006890008607998747,
+            -0.002948835017234416,
+            0.03734128725793149,
+            -0.00556474133167756,
+            -0.0014544201137417158,
+            0.02209084868209573,
+            -0.01883610080898967,
+            0.017901421953179442,
+            0.01563604483998435,
+            0.012327159100148019,
+            0.0018161034827768648,
+            -0.010253034391420751,
+            0.019979728662282904,
+            0.020823925491027635,
+            0.0139490143219451,
+            0.009577521724445478,
+            -0.013478761809038993,
+            0.014934933784421556,
+            -0.0003725422088504494,
+            -0.0024362115214204105,
+            0.012785534912074068,
+            0.016441944143888537,
+            -0.02164522371903646,
+            -0.013152102018736346,
+            0.0063548333449661805,
+            -0.016547629104776748,
+            -0.024875443975589356,
+            -0.008991095813132973,
+            -0.021885127280977718,
+            -0.023336714382596175,
+            0.00688008599534903,
+            0.016482610587564443,
+            0.0035797836729533367,
+            0.013111012464332918,
+            0.006820471349350837,
+            0.018274678506489214,
+            -0.00980248993077149,
+            -0.01582838461385605,
+            -0.0009454586621769173,
+            -0.0004906493597104733,
+            0.009787688665732045,
+            -0.01835583040075359,
+            -0.013977939077760832,
+            0.004340902272034388,
+            -0.020118484730757877,
+            0.01421736173162507,
+            -0.006428386730058143,
+            0.016756376129345597,
+            -0.0004630315831872392,
+            0.011701948921076007,
+            0.02621773278421472,
+            -0.001391567406453007,
+            0.01262026995055899,
+            0.004696906079976175,
+            -0.0030577364576645635,
+            -0.010624066224426196,
+            0.0009420498021845824,
+            -0.0044079288776429516,
+            -0.0004845606666642234,
+            0.006481113327122954,
+            0.020269778435314,
+            -0.03321232649491074,
+            -0.009425498978427855,
+            0.01246754422779043,
+            -0.001658574527323383,
+            -0.005816985985181387,
+            0.013050062223188965,
+            0.00011195028631856881,
+            0.016573827662919836,
+            -0.0017453203044868104,
+            0.01417371165305189,
+            -0.008165910519539772,
+            -0.0007461950113024975,
+            0.0016492309267666983,
+            0.005076015595439873,
+            0.005829299318055953,
+            0.012146800191035666,
+            0.01148884639506138,
+            -0.021051327863689618,
+            -0.0026357096198729156,
+            0.0036146148489310443,
+            -0.010673282438301611,
+            0.014967444393862347,
+            0.005737933867764028,
+            0.01391411712868596,
+            0.0029556189893790237,
+            0.01167806258173173,
+            -0.024440974650111174,
+            -0.01029944007931845,
+            0.01385509328958837,
+            0.00869671728965227,
+            -0.00998315356917989,
+            -0.016580053751364735,
+            -0.007061005072623325,
+            -0.00477175200214453,
+            -0.0021182980051220428,
+            -0.009871524928312118,
+            -0.0005933941177237234,
+            0.006284920058476413,
+            5.466820734762947e-05,
+            -0.012842295882002122,
+            0.015810530885294202,
+            0.004572385791133295,
+            0.003685966629168175,
+            0.005093714133656017,
+            0.011877541830396259,
+            -0.013130170365831912,
+            0.01258190257102377,
+            0.011433121961378686,
+            0.01142665981860032,
+            -0.013710025169815697,
+            -0.001774030746825879,
+            -0.01109881340948764,
+            0.0065684584345163735,
+            0.009983753894796107,
+            0.005346443477816739,
+            -0.002062387245572686,
+            0.004950044847352647,
+            0.006105199826576752,
+            -0.06391185159544624,
+            -0.02131511418723256,
+            0.0027165737784844413,
+            0.033389688791557785,
+            -0.007167593552251769,
+            -0.0037646833460996797,
+            0.002883510089742145,
+            0.01356887065340832,
+            0.009730171586767032,
+            0.00418566245984083,
+            0.020320934773473808,
+            -0.008814826860651366,
+            -0.003014193747732543,
+            0.001559573791453607,
+            0.003499420226345028,
+            -0.008703837291821111,
+            0.0044844701302276645,
+            0.0053963630238108035,
+            0.027764369548110287,
+            0.001469968540767627,
+            0.015793086297019008,
+            -0.008505290826588508,
+            -0.004075888558568453,
+            -0.009166710762668385,
+            -0.002860047211143503,
+            -0.011127820563172684,
+            0.003937110717404271,
+            0.02054186721680431,
+            -0.006126180963243227,
+            0.009346433819831456,
+            0.01057504086030076,
+            0.0001650851447316893,
+            -0.012033423322524951,
+            0.0049587335923558,
+            -0.0032240390628760535,
+            -0.005568976687946761,
+            -0.0024667040429312737,
+            -0.03474027725033016,
+            0.026855947684394643,
+            -0.004369626674503285,
+            0.009252512430957932,
+            -0.005081834388513357,
+            -0.0037896610731020337,
+            -0.014201609270756739,
+            0.017871088029615106,
+            -0.008249444723433136,
+            -0.020092257756971717,
+            0.009532994544485572,
+            -0.004991989149115383,
+            0.006464417738670472,
+            -0.02643078108339741,
+            0.0029138922065812495,
+            -0.017232410749655366,
+            -0.003999293090834831,
+            -0.006924488971625995,
+            0.01318082244178942,
+            -0.007232695119856389,
+            -0.02660711037908311,
+            0.01979251106803667,
+            -0.019485860712345603,
+            -0.011977904020170568,
+            -0.0177530973032872,
+            -0.014524003566945937,
+            0.004874952064727108,
+            -0.015326044325348929,
+            0.002047771741041311,
+            -0.009697340332342246,
+            -0.003652304998155628,
+            -0.0008878698629674222,
+            0.0017782435347417623,
+            -0.013017776298713044,
+            -0.018272697230002275,
+            -0.01353020425912225,
+            -0.00757239449706798,
+            -0.0008250000619957538,
+            0.01299142281772714,
+            -0.001679606555140765,
+            -0.007329941421303076,
+            0.0044605462460153116,
+            0.010231192277666706,
+            0.0006558644644452857,
+            0.0024231733856020414,
+            -0.010728844680699885,
+            -0.014137434553169443,
+            0.005230053181525096,
+            -0.012086468475542818,
+            -0.009459717916037003,
+            -0.0028185227600210556,
+            0.014984527039312847,
+            0.008189464334213754,
+            -0.009139240774931914,
+            -0.005639157579163322,
+            0.014325949891859377,
+            0.009020303519690971,
+            -0.013310205617445609,
+            0.01567574314514385,
+            -0.025369802629268523,
+            0.021964786021408878,
+            0.030536905527039065,
+            0.004945288795196981,
+            -0.01168303908203322,
+            0.02281128628230144,
+            -0.004072286832145419,
+            0.0028278384348551146,
+            0.01740522162288968,
+            -0.042558465579514926,
+            -0.013146562547084278,
+            0.002903743316339479,
+            -0.015297177099602701,
+            -0.010963312697387696,
+            0.008804938803398296,
+            -0.0026727734429658817,
+            0.005273238046953793,
+            0.004257613402172282,
+            0.03512246556126453,
+            0.002258612643297096,
+            -0.0002885708728599208,
+            -0.0019720190285930487,
+            0.012312613129996533,
+            0.010451003939611476,
+            -0.015556620910136185,
+            -0.02387856865043814,
+            -0.004583090523619507,
+            -0.0037828776396525835,
+            -0.0014588238692793338,
+            -0.004361834167147271,
+            -0.008159679676329573,
+            0.012243319920992457,
+            -0.017279964797769668,
+            -0.013748032758693542,
+            0.018394195628121487,
+            -0.006411113522842359,
+            0.019002611495040334,
+            -0.014970362556522149,
+            -0.0077437581301661946,
+            0.0017311115715410025,
+            0.024439691379769795,
+            -0.025013956798520212,
+            0.007359134924497539,
+            -0.0031406470488477937,
+            0.017574824561755627,
+            0.00419409250132525,
+            -0.0046803792187546656,
+            0.010556972371044426,
+            0.015229789583480183,
+            -0.004608293492221483,
+            -0.014449487985966966,
+            -0.01172850291078532,
+            0.016823024209386785,
+            -0.012304887597699588,
+            0.0011773351576246157,
+            0.002262670226244277,
+            -0.0023810578015456716,
+            0.03477119000959688,
+            -0.03442178291363675,
+            0.004062693040714608,
+            0.0029249583507755944,
+            0.003291349443702718,
+            0.006461525089528308,
+            0.006047777261425826,
+            -0.020642802018978404,
+            0.008916088381876748,
+            -0.03316762661930299,
+            -0.006367254689957291,
+            -0.005404276207844267,
+            0.0009189006423526284,
+            0.009401330264376041,
+            -0.011106443887384533,
+            -0.02259405372588916,
+            -0.006010439054323966,
+            0.0070744551115721095,
+            0.010262607009446879,
+            -0.002856616377476421,
+            0.019719451109535182,
+            0.008426509638139982,
+            0.0015877648339862691,
+            -0.010655792508751512,
+            -0.015527851374834499,
+            -0.015460974859638219,
+            -0.017847462632188386,
+            -0.02236964311096428,
+            0.011841749854626384,
+            -0.00576688153869993,
+            0.009054902130717224,
+            0.014317707346504314,
+            -0.015366657284925976,
+            0.002910211008094564,
+            0.017761685988205086,
+            0.007974806171830457,
+            0.002692237107055498,
+            0.0032405477366987947,
+            0.02685230091638845,
+            -0.0009116884421301454,
+            -0.0032228977180251673,
+            0.003158213042682977,
+            0.015872199488167846,
+            0.0004966905858799325,
+            0.019981401695749582,
+            0.015201538351780685,
+            0.005359075814444414,
+            -0.004668738193285921,
+            0.019853988551405383,
+            -0.009844394286458195,
+            0.013476144827747423,
+            -0.00411789677405284,
+            -0.00830656434595337,
+            -0.008496132520586954,
+            0.006442581751999304,
+            -0.008138786054348571,
+            0.006977337946948795,
+            0.02419128791801387,
+            -0.0005734827808485311,
+            -0.0028842142002271044,
+            0.005503033191779872,
+            -0.01696639779775455,
+            -0.00389835761601045,
+            -0.03366077608692946,
+            -0.009942613075976742,
+            -0.01708630689497001,
+            -0.02910756562213045,
+            0.026536268897199546,
+            0.00219827495662445,
+            0.02586503862171508,
+            -0.01905122645609761,
+            0.008935637454859214,
+            -0.01385806889332683,
+            0.029747093931541325,
+            0.009163959012862627,
+            0.0008611776492791918,
+            0.003598831885031635,
+            -0.02103045281939351,
+            -0.01150941825152158,
+            0.027341774725604333,
+            -0.0006700608710615124,
+            -0.012089044078073347,
+            0.022396400579038037,
+            0.017491565611764764,
+            0.0051760367056280536,
+            -0.00559536785318233,
+            0.01979262595019399,
+            0.002306380546194741,
+            0.011973863166872444,
+            -0.0028136750606972223,
+            0.012911235736067303,
+            -0.01510421118359169,
+            -0.003338602157155174,
+            0.039758808572572274,
+            0.007290642519242007,
+            -0.001802921446521437,
+            -0.013696774990705774,
+            0.003941198181491921,
+            -0.0008308293948698925,
+            0.0200351513103389,
+            -0.00014327587644797734,
+            0.028291423013030337,
+            -0.0001085859276231927,
+            -0.010723064585989723,
+            0.020443848393732245,
+            -0.017860296930396913,
+            0.04540901629084378,
+            -0.018671578423261854,
+            0.000852031295629291,
+            -0.010639853151638835,
+            -0.019017194262730474,
+            0.003861296738239369,
+            0.0010977143012920602,
+            0.0008387482016061222,
+            0.0060155816314618,
+            -0.010780239332444652,
+            0.002152214497259421,
+            0.010556338812873054,
+            -0.027636192450621728,
+            0.0010938992111792307,
+            -0.014385202899929138,
+            -0.01353192238925814,
+            -0.026667778226211785,
+            -0.00017403398236549974,
+            -0.015357674131635848,
+            -0.013956722505869091,
+            -0.01985038526848704,
+            -0.008927903879720431,
+            -0.009580214158077466,
+            -0.0020314441852913386,
+            -0.004755584878514854,
+            -0.00018388548107372635,
+            -0.002918855567842974,
+            -0.0015941852262316128,
+            -0.014230272061190995,
+            0.02133992212037431,
+            0.015283345362445234,
+            0.004642765060980717,
+            0.012829570028541471,
+            0.01708952142208596,
+            -0.004770647543823141,
+            0.006604762788847952,
+            -0.0025369836845081078,
+            0.008368000846861623,
+            0.014785031726639733,
+            0.012074738887486398,
+            0.02124518119032885,
+            -0.015459818988572564,
+            -0.004992865135357168,
+            -0.012722391036055767,
+            -0.012210202603282509,
+            -0.017850065946244383,
+            -0.004300429148015694,
+            -0.0014387298144714717,
+            -0.023128300275416423,
+            0.004468952290723408,
+            -0.0026602931688877443,
+            -0.005768852559887139,
+            0.007815212168719318,
+            0.0005862972228454734,
+            -0.002265719947629544,
+            -0.00019323737674204774,
+            -0.02735820445266249,
+            0.00024009122317215594,
+            -0.00018701993387558173,
+            -0.014663710240203917,
+            -0.00450797561624596,
+            0.010584731989847334,
+            -0.011966527748437568,
+            -0.00602367292291343,
+            0.0111155226518017,
+            0.0015565179432852256,
+            0.006062862961835896,
+            0.016952559697728433,
+            0.0005012157272316869,
+            0.011160668242334889,
+            0.014993782560797772,
+            -0.01591303023837098,
+            -0.015356412848444082,
+            -0.0236025426434855,
+            -0.004514538154596937,
+            -0.009753334017065119,
+            0.007553951535816575,
+            -0.004616595532996391,
+            -0.013023153809236325,
+            0.006040285197946957,
+            -0.0006343451398642784,
+            0.008338564073158257,
+            -0.017608246105292328,
+            -8.346168119689692e-05,
+            -0.020944870308705715,
+            0.009230123321316848,
+            -0.00531237224627929,
+            -0.00445985089300067,
+            0.01410675485522911,
+            0.02895945580213926,
+            -0.007799830351886844,
+            -0.009455128322575888,
+            -0.005663072244883114,
+            0.012817229945992764,
+            0.012474431030896143,
+            -0.004423141749110039,
+            -0.0018294512146253298,
+            -0.012817101093394574,
+            0.0004755435377918658,
+            -0.01129485262081729,
+            0.0062747111125005155,
+            0.017199374798326412,
+            -0.021965117839794063,
+            -0.01296097993683748,
+            -0.016765996091978336,
+            -0.016882855472266523,
+            0.004372970343291638,
+            0.017290565252244258,
+            0.022670788364378462,
+            -0.008456547449078897,
+            -0.019564049850942417,
+            0.006898353991539946,
+            -0.01620125883985687,
+            0.0037108638122592746,
+            0.12887455743463638,
+            -0.0028864852421660366,
+            -0.011487804449328495,
+            -0.022683334123299446,
+            -0.005467192611880829,
+            0.01210557696712196,
+            -0.01555159294496261,
+            0.06709672179195036,
+            -0.015071089552631105,
+            0.004364902817496082,
+            -0.004899872946473657,
+            0.0006310685609937571,
+            -0.017610935588191347,
+            -0.005962991715597585,
+            -0.007768819050529025,
+            -0.02688607975562596,
+            0.000883763319535686,
+            -0.009935313912803587,
+            -0.012883104064962595,
+            0.002126368813790577,
+            -0.0013697265269800346,
+            -0.016131186838419288,
+            0.01575514418037863,
+            -0.020677990768697592,
+            -0.007251917191831732,
+            -0.0029325623780992118,
+            -0.00600710212646933,
+            0.014573093764769918,
+            -0.03253336117394778,
+            0.002973725493330224,
+            -0.013601417220439602,
+            0.011905788192118131,
+            0.001542117126832946,
+            -0.010789486980776022,
+            -0.013862139885514524,
+            0.007086571810314664,
+            0.015658913897173155,
+            0.017039624764452546,
+            0.01901374401756777,
+            0.004797954569087795,
+            0.003928999581093576,
+            -0.008046722893614655,
+            0.021974371139905104,
+            -0.00697335955527197,
+            0.0008975473383295958,
+            -0.00970589295353857,
+            -0.004346869073499034,
+            0.0031279458769925416,
+            0.02303983493541673,
+            -0.012560380384977343,
+            -0.01536378830925542,
+            0.0012035987157572355,
+            0.033313215063595594,
+            -0.010845084209743817,
+            0.013587126023659577,
+            0.0029867620644300073,
+            0.020679673435988446,
+            -0.0006640787253300063,
+            0.0025071780490108306,
+            0.022554193713122565,
+            -0.018082906254599396,
+            0.018230000220500718,
+            -0.004451773302171552,
+            0.01915083025790679,
+            -0.004183521043835777,
+            0.015019320101074747,
+            -0.027014055979204272,
+            -0.005362044589023088,
+            -0.012355559881051253,
+            0.003995309003198048,
+            0.020680455787871383,
+            0.0054391755324872375,
+            0.009865676762244514,
+            -0.01645386866657314,
+            -0.01760657627715638,
+            0.022969828924328126,
+            -0.021566687194482292,
+            -0.015826424849757402,
+            0.00012804774676298237,
+            -0.015021970627042762,
+            0.006000610438452896,
+            0.010126179023323158,
+            -0.02912121211344025,
+            -0.02046150879033127,
+            0.004881850039543051,
+            -0.0023469840669773042,
+            -0.00915786433296249,
+            0.006547377770403066,
+            -0.022329063637464074,
+            -0.015702777677743352,
+            -0.005692904224111613,
+            0.01207999389934878,
+            -0.008429160038192635,
+            0.010075061963131637,
+            -0.020106712664936504,
+            0.0036787045650556907,
+            -0.002764965946892891,
+            0.010307702251974629,
+            0.004697169247387421,
+            0.011192285826515207,
+            0.0030113901179382946,
+            0.0014774681536380047,
+            -0.02778270000997313,
+            0.010043261430740039,
+            -5.178406450222847e-05,
+            0.04216636255860709,
+            0.030977096295679525,
+            0.004479857028809617,
+            -5.798020745008712e-05,
+            -0.009996471945562616,
+            0.012476009017187798,
+            -0.019306907391548302,
+            -0.005092322343599383,
+            0.01856463597985508,
+            0.02195467141594795,
+            -0.02236232264139305,
+            0.0052217593371508145,
+            -0.016766955100332014,
+            0.016133299654723637,
+            -0.011208763137897244,
+            0.006248024409576809,
+            0.011601988603146743,
+            0.0022330878390492527,
+            0.010317781892717903,
+            -0.005895142237225119,
+            0.019670196605183973,
+            0.01186677200787666,
+            -0.002339367025714548,
+            0.02367408853110795,
+            0.006858134385159451,
+            0.004479683673962342,
+            0.004798719889930319,
+            -0.009604173975576537,
+            -0.009045602998213713,
+            0.009230969488015385,
+            0.004083616606774546,
+            -0.007944628589259331,
+            0.011836323287097144,
+            0.015100350195476968,
+            0.01885317701149666,
+            -0.0011707847499298092,
+            -0.00684548433712413,
+            -0.006550059418141426,
+            0.001899758393077906,
+            -0.024577381680573596,
+            -0.02855511854789239,
+            0.021238935710272536,
+            -0.010427135883958709,
+            -0.019187769881105637,
+            -0.0015864640188683347,
+            0.016360037306061392,
+            -0.008853972319021563,
+            -0.01627167079143546,
+            0.001270414822216867,
+            0.02255755055961612,
+            -0.005959411125391464,
+            0.006009654905468478,
+            0.005909233773418581,
+            0.005874361519568704,
+            -0.0027859463941992753,
+            -0.024599789577567636,
+            0.0036612374968223453,
+            0.03221346865585554,
+            0.016461563205582293,
+            0.016139782455492073,
+            -0.007158168059808459,
+            0.001838455290310879,
+            0.007530954381521755,
+            0.005438791523412295,
+            0.015674000912615536,
+            0.008604106101542998,
+            -0.006990462920227135,
+            0.00244736628387359,
+            0.012213048255418866,
+            -0.007329299946651028,
+            0.017424711328323798,
+            0.001973882532418776,
+            -0.0133178791530314,
+            -0.008373452013951822,
+            -0.010462150707776556,
+            -0.014575483571750984,
+            0.0027958704799249897,
+            -0.011401916623878957,
+            0.0008016041012669959,
+            0.003953958717263787,
+            0.010260753041492368,
+            0.013961649189650642,
+            -0.0019275791443041316,
+            0.017019168629298462,
+            0.011534594243860177,
+            -0.0017859096307142257,
+            -0.002970739203219076,
+            -0.03142778833370656,
+            0.010207789755749909,
+            -0.014730553506704018,
+            -0.0037611416875830746,
+            0.01825790347981639,
+            -0.007766890847082858,
+            -0.0032778433418881666,
+            -0.021003405429807113,
+            0.0024512381432602736,
+            0.008768616692363321,
+            0.01558713989712387,
+            -0.012819542714293536,
+            0.009201155584074711,
+            -0.0012254337002053017,
+            -0.0057451574939563045,
+            0.00977251392462179,
+            0.0042356831792011935,
+            -0.0033433823868435786,
+            0.011499764125503874
+          ],
+          "title": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer",
+          "keyphrases": [
+            "Sparsely-Gated Mixture-of-Experts Layer",
+            "conditional computation",
+            "neural network",
+            "language modeling",
+            "machine translation"
+          ]
+        },
+        "type": "document"
+      },
+      "bidirectional": true,
+      "properties": {
+        "cosine_similarity": 0.8503794853623671
+      }
+    },
+    {
+      "id": "4821fad7-ab83-4957-969e-7814fdc651a8",
+      "type": "cosine_similarity",
+      "source": {
+        "id": "b570d71e-4f83-486e-b70f-b0c6217db455",
+        "properties": {
+          "page_content": "Mixture-Of-Experts With Expert Choice Routing\n\nYanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon Google, Mountain View, CA, USA {yanqiz, taole, hanxiaol, dunan, huangyp, vzhao, adai, zhifengc, qvl, jlaudon}@google.com\n\nAbstract\n\nSparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choice method. Instead of letting tokens select the top-k experts, we have experts selecting the top-k tokens. As a result, each token can be routed to a variable number of experts and each expert can have a fixed bucket size. We systematically study pre-training speedups using the same computational resources of the Switch Transformer top-1 and GShard top-2 gating of prior work and find that our method improves training convergence time by more than 2×. For the same computational cost, our method demonstrates higher performance in fine-tuning 11 selected tasks in the GLUE and SuperGLUE benchmarks. For a smaller activation cost, our method outperforms the T5 dense model in 7 out of the 11 tasks.\n\n1 Introduction\n\nScaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27]. The final model quality has been found to have a power-law relationship with the amount of data, model size, and compute time [16, 20]. However, training efficiency, which is defined as the total amount of computation used to achieve superior model quality than the state of the art system [21], should receive greater attention as we increase our efforts towards green AI [29].\n\nSparsely gated mixture-of-experts [31] (MoE) provides an effective way to scale model capacity given a fixed computational cost, and has recently played an important role in increasing the training efficiency of large-scale language models [10, 21]. MoE operate by adopting a number of experts, each as a sub-network, and by activating only one or a few experts for each input token. A gating network must be chosen and optimized in order to route each token to the most suited expert(s). For example, recent work has implemented sparse routing via k-means clustering [12], linear assignment to maximize token-expert affinities [22], or hashing [8, 28]. Many of the prior work use a routing strategy concerning the token choice, where each token selects the best one or two experts.\n\nWe argue that the independent token choice of prior work often leads to an imbalanced load of experts, which causes training inefficiency and sub-optimal training of the model. In order to mitigate this\n\nissue, previous sparsely gated networks introduce additional auxiliary losses as regularization to prevent too many tokens being routed to a single expert, but the effectiveness is still limited. Recent approaches [8, 22, 28] explore alternative strategies for routing, but they focus on pre-training only and do not demonstrate performance gain on downstream tasks. Moreover, none of the previous methods consider allocating a variable number of experts to each token based on importance, which can be beneficial.\n\nWe propose a very simple yet effective routing method we are calling expert choice. Unlike conventional MoE where tokens select one or two top-scoring experts, our method lets each expert pick the top-k tokens. Our method guarantees perfect load balancing, allows a variable number of experts for each token, and achieves substantial gains in training efficiency and downstream performance as demonstrated in our experiments. Our major contributions include:\n\nWe identify common pitfalls in conventional MoE such as load imbalance as described in Section 3.1. We then propose a heterogeneous, expert choice method to provide a fluid allocation of model parameters based on a learnt token-to-expert importance. This method intrinsically guarantees load balance without imposing an auxiliary loss.\n\nWe show our method provides over 2× faster training convergence in a 8B/64E (8 billion activated parameters, 64 experts) model, compared to the top-1 and top-2 gating counterparts in Switch Transformer [10] and GShard [21].\n\nWe show our method demonstrates strong scaling when increasing the number of experts from 16 to 128, evaluated in training perplexity.\n\nWe show our method demonstrates strong performance on downstream tasks selected from GLUE and SuperGLUE at all the evaluated scales. More specifically, our 8B/64E model outperforms a T5 11B dense model in 7 out of 11 tasks evaluated.\n\n2 Related Work\n\nScaling: Various approaches have been proposed to scale up neural network capacity to improve performance. Recent works have successfully scaled models to billions of parameters via various forms of model parallelism [2, 21, 26, 27, 33]. Model parallelism [30] splits weights and tensors across multiple cores while pipeline parallelism [18, 24] splits different layers across devices with micro-batches pipelined to the different layers. To enable continued scaling of neural networks, improving model training and serving efficiency has become a critical research area.\n\nConditional Computation: Computation decisions can be made dynamically based on the input [23, 25]. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation, by activating certain parameters and computation on demand, on a per-example or per-token basis [3]. Conditional convolution layers [1] with task-specific gating has been used to combat catastrophic forgetting when a sequence of learning problems are optimized. The gating decisions may be binary or sparse and continuous, stochastic or deterministic.\n\nMixture of Experts: Sparsely-gated MoE [31] is the first model to demonstrate massive improvements in model capacity, training time, or model quality with gating. Switch Transformer [10] simplifies the gating by selecting only the top expert per token using a softmax over the hidden state and demonstrates better scaling than previous work. All the prior work requires an auxiliary loss to explicitly encourage balancing. This loss term has to be carefully weighted to not overwhelm the primary loss. However, auxiliary loss does not guarantee balancing and a hard capacity factor has to be imposed. As a result, many tokens can still be unprocessed by the MoE layer. Hard MoE [12] with a single decoding layer can be efficiently trained to good effect on large scale hashtag prediction tasks.\n\nBase Layers [22] formulate a linear assignment that maximizes token-expert affinities while ensuring each expert receives an equal number of tokens. Hash layers [8, 28] devise hashing techniques on input tokens. However, the evaluations are limited to pre-training perplexity. THOR [? ] randomly activates experts during training and inference and is trained with a consistency regularization loss.\n\nTHOR has demonstrated strong performance on translation tasks. Different from these prior works, our method is a learnt method that enables heterogeneous MoE and effectively improves downstream fine-tuning performance.\n\n3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n5 Conclusion\n\nWe propose a new routing method for sparsely activated mixture-of-experts (MoE) models. This method addresses load imbalance and under-utilization of experts in conventional MoE methods, and enables selecting different numbers of experts for each token. Our model demonstrates more than 2x training efficiency improvements when compared to the state-of-the-art GShard and Switch Transformer models, and also achieves strong gains when finetuning on 11 datasets in the GLUE and SuperGLUE benchmark.\n\n6 Limitations\n\nThe expert choice method might not immediately apply to auto-regressive text generation as our current implementation takes in the past and future tokens to perform the top-k selection. One possible solution is to collect a large batch of input sequences, dispatch tokens of the same sequence into separate groups, and perform expert choice routing for each group. Another scenario where the expert choice method does not immediately apply is when the batch size becomes very small during serving or inference. A global top-k can be selected instead and we can cap the number of times each expert or token gets selected. We leave these possible improvements for future work.\n\nAnother long-standing issue with MoE has been the large memory footprint. Even though computational cost can be reduced using sparsely gated networks, the total number of parameters increases linearly or sub-linearly with the number of experts. Increasing the number of experts requires reservation of a large number of hardware devices. Therefore, dynamic (used) power is saved while static (reserved) power is not. Power saving techniques such as the ability to put hardware devices into low power states while not in use [17] can help with reducing the reserved power requirements.\n\nReferences\n\n[1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pages 3930–3939. Computer Vision Foundation / IEEE, 2020.\n\n[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. In Advances in Neural Information Processing Systems.\n\n[3] Kyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning, 2014.\n\n[4] Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems, 2021.\n\n[5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.\n\nTransformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.\n\n[6] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 933–941. JMLR.org, 2017.\n\n[7] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixtureof-experts, 2021.\n\n[8] Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan.\n\nTricks for training sparse translation models, 2021.\n\n[9] Richard L Dykstra. An iterative procedure for obtaining i-projections onto the intersection of convex sets. The annals of Probability, pages 975–984, 1985.\n\n[10] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.\n\n[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. NAS-FPN: learning scalable feature pyramid architecture for object detection. In CVPR, pages 7036–7045. Computer Vision Foundation / IEEE, 2019.\n\n[12] Sam Gross, Marc'Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision, 2017.\n\n[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.\n\n[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 630–645, Cham, 2016. Springer International Publishing.\n\n[15] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2016.\n\n[16] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.\n\n[17] Ping Huang, Zuocheng Xing, Tianran Wang, Qiang Wei, Hongyan Wang, and Guitao Fu. A brief survey on power gating design. In 2010 10th IEEE International Conference on Solid-State and Integrated Circuit Technology, pages 788–790, 2010.\n\n[18] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 103–112, 2019.\n\n[19] Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David A. Patterson. A domain-specific supercomputer for training deep neural networks. Commun. ACM, 63(7):67–78, 2020.\n\n[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.\n\n[21] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.\n\n[22] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6265–6274. PMLR, 18–24 Jul 2021.\n\n[24] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. New York, NY, USA, 2019. Association for Computing Machinery.\n\n[25] Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, Cédric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. In ICLR. OpenReview.net, 2021.\n\n[26] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.\n\n[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.\n\n[28] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models, 2021.\n\n[30] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 10435–10444, Red Hook, NY, USA, 2018. Curran Associates Inc.\n\n[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E.\n\nHinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In ICLR (Poster). OpenReview.net, 2017.\n\n[32] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 10–15 Jul 2018. [23] Min Lin, Jie Fu, and Yoshua Bengio. Conditional computation for continual learning, 2019.\n\n[29] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.\n\n[33] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.\n\n[34] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc.\n\n[35] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.\n\nGLUE: A multi-task benchmark and analysis platform for natural language understanding.\n\nIn Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, November 2018. Association for Computational Linguistics.\n\n[36] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake A. Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen. GSPMD: general and scalable parallelization for ML computation graphs. CoRR, abs/2105.04663, 2021.\n\n7 Checklist\n\n(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Yes (b) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes (c) Did you discuss any potential negative societal impacts of your work? N/A. Not any. (d) Did you describe the limitations of your work? Yes (a) Did you include the code, data, and instructions needed to reproduce the main experimental results? Yes. We include details in the experiment setup to help reproduce the main results. (b) Did you specify all the training details? Yes (c) Did you report error bars? Yes (d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? Yes (a) If your work uses existing assets, did you cite the creators? Yes (b) Did you mention the license of the assets? No. The used dataset is not released yet. (c) Did you include any new assets either in the supplemental material or as a URL? No. The dataset is not released yet.\n\n(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? No. Not using persons' data. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes. The dataset does not contain any personally identifiable information or offensive content.\n\nA Comparison On Fine-Tuning With A Dense Model\n\nOur 8B MoE model achieves stronger pre-training perplexity than its dense counterpart. However, a better perplexity does not always directly translate to downstream performance as demonstrated in Section 4.4. To this end, we compare fine-tuning performance of the 8B dense model and MoE model in Table 1. As shown in the table, our MoE model using expert choice routing consistently outperforms the dense model across the 11 tasks in GLUE and SuperGLUE.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 1: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nB Capacity Factor\n\nWe evaluate the downstream task fine-tuning performance by varying the capacity factors. Note that a capacity factor of n indicates on average how many experts each token can be received. EC-CF2 is our baseline expert choice, which matches GShard top-2 gating computational footprint. EC-CF1, however, matches Switch Transformer top-1 gating computational footprint. EC-CF0.5 further verifies that an aggressively lowered capacity factor can provide strong enough performance, that almost matches the top-2 gating baseline.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Top-2 78.1 87.0 88.3 85.0 82.6 90.1 90.7 81.6 94.7 68.2 67.2 83.0±0.3 EC-CAP2 78.2 88.0 88.5 85.7 83.0 90.8 91.1 80.0 95.4 70.4 64.1 83.2±0.4 EC-CAP3 78.5 91.7 89.3 86.3 83.5 90.9 91.1 81.8 94.9 70.0 65.6 84.0±0.4 EC-CF2 79.1 89.6 89.3 86.8 84.3 91.3 91.2 81.1 95.2 68.1 68.0 84.0±0.2 EC-CF1 77.4 90.6 88.0 85.5 83.6 90.3 91.2 79.8 95.3 66.5 64.9 83.0±0.2 EC-CF0.5 77.4 89.6 86.3 85.2 82.7 91.7 91.0 79.6 94.9 67.3 63.5 83.0 ±0.05 Hash Layers 76.1 85.2 86.7 83.4 82.5 90.0 90.3 75.7 94.0 67.4 63.3 81.3±1.0\n\nTable 2: Comparison between different routing methods in fine-tuning of 100M/64E models. We perform 3 independent fine-tuning runs for each method and report the average results. This gives more accurate difference between the variants of expert choice method, since they achieve close fine-tuning results. We do not report averaged results in other experiments.\n\nC Capped Expert Choice\n\nAs described in Section 4.5, the maximum number of experts each token is assigned can be capped by an entropy-regularized linear programming. Figure 1 compares the validation perplexity when training the 100M/64E models using the base expert choice method (EC-BASE), expert choice capped by two experts per token (EC-CAP2), expert choice capped by three experts per token (EC-CAP3), and GShard top-2 gating.\n\nAs shown in the figure, restricting the number of experts to 2 degrades the perplexity compared to the base expert choice method. This suggests that a more flexible allocation of experts (e.g. more than 2 experts for a token) can enhance model expressiveness. On the other hand, our EC-CAP2 and EC-CAP3 methods still outperform the top-2 gating method by a clear margin. We believe this confirms the effectiveness of a load balanced training, provided by our method. Finally, EC-CAP3 obtains comparable perplexity to EC-BASE. As indicated by Figure 3, only a little fraction of tokens use more than 3 experts therefore we see little or no difference between EC-BASE and EC-CAP3 variants. We present the fine-tuning results of these methods in Table 2.\n\nD Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [? ]. We use mod x to map a token ID to an expert ID. This in some way ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 2. Hashing based routing performs much worse than expert choice in terms of average scores and variance.\n\nE Fine-Tuning Details\n\nWe did a hyperparameter search for both baseline models and expert choice method. For fine-tuning of the 8B dense model, we use a constant learning rate of 0.0001 and a dropout rate of 0.1. We freeze the attention layer and feed-forward layer while leaving the embedding and layer normalization trainable. This setting has been found optimal for the 8B dense model. For MoE 8B/64E models including GShard top-2 gating and expert choice, we found continuing the learning rate from the pre-trained model while using a square root learning rate decay works better. In addition, we do not apply parameter freezing for fine-tuning MoE models. For models with 100M expert size, we use a constant learning rate of 0.0001 and no dropout is used.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/2202.09368v2.md"
+          },
+          "headlines": {
+            "1 Introduction": [],
+            "2 Related Work": [],
+            "3 Method": [
+              "3.1 Pitfalls Of Token-Choice Routing",
+              "3.2 Heterogeneous Moe Via Expert Choice",
+              "3.3 Expert Choice With Additional Constraint",
+              "3.4 Model Architecture"
+            ],
+            "4 Experiments": [
+              "4.1 Setup",
+              "4.2 Training Efficiency",
+              "4.3 Scaling The Number Of Experts",
+              "4.4 Fine-Tuning On Glue And Superglue",
+              "4.5 Heterogeneity Matters",
+              "4.6 Comparison With Hash Layer",
+              "4.7 Ablation"
+            ],
+            "5 Conclusion": [],
+            "6 Limitations": [],
+            "A Comparison On Fine-Tuning With A Dense Model": [],
+            "B Capacity Factor": [],
+            "C Capped Expert Choice": [],
+            "D Comparison With Hash Layer": [],
+            "E Fine-Tuning Details": []
+          },
+          "summary": "The paper introduces a new routing method for sparsely-activated Mixture-of-Experts (MoE) models called 'expert choice,' which addresses load imbalance and under-utilization issues in conventional MoE methods. Unlike traditional token-choice routing, expert choice allows experts to select the top-k tokens, ensuring perfect load balancing and enabling variable expert allocation per token. This method improves training efficiency, achieving over 2x faster convergence compared to existing models like Switch Transformer and GShard. It also demonstrates superior performance on downstream tasks in the GLUE and SuperGLUE benchmarks. The study highlights the benefits of flexible expert allocation and shows that expert choice outperforms hashing-based routing methods. The paper also discusses potential limitations and future improvements for the expert choice method.",
+          "summary_embedding": [
+            -0.009726986289024353,
+            -0.0337262786924839,
+            -0.023302342742681503,
+            -0.019878201186656952,
+            -0.025953780859708786,
+            0.04560471698641777,
+            -0.006874797400087118,
+            0.039665497839450836,
+            -0.006973279174417257,
+            0.034544438123703,
+            -0.025787118822336197,
+            -0.00363057479262352,
+            -0.01052999310195446,
+            0.0054657477885484695,
+            -0.024499278515577316,
+            0.036786794662475586,
+            0.010007280856370926,
+            0.003272630739957094,
+            0.006102093029767275,
+            -0.04048365354537964,
+            -0.018666116520762444,
+            -0.013522328808903694,
+            -0.03230207785964012,
+            0.008969432674348354,
+            0.02484775334596634,
+            0.0016713523073121905,
+            0.0026230288203805685,
+            0.003624893259257078,
+            -0.05311964824795723,
+            0.04339266195893288,
+            0.018681267276406288,
+            0.023711422458291054,
+            -0.016332851722836494,
+            -0.012832955457270145,
+            -0.039120059460401535,
+            -0.0197418425232172,
+            0.030786970630288124,
+            0.039968520402908325,
+            -0.019287310540676117,
+            0.03357476741075516,
+            0.018060073256492615,
+            0.02454473078250885,
+            -0.021272100508213043,
+            -0.029817303642630577,
+            -0.0061892117373645306,
+            -0.007840678095817566,
+            -0.008355814032256603,
+            0.01242387667298317,
+            0.016984347254037857,
+            -0.013007191941142082,
+            0.0212569497525692,
+            -0.01371171697974205,
+            0.03384748846292496,
+            0.008916404098272324,
+            -0.042332086712121964,
+            0.03984731063246727,
+            -0.008007340133190155,
+            0.012105703353881836,
+            0.03281721472740173,
+            0.010333028621971607,
+            -0.005287723150104284,
+            -0.021544819697737694,
+            -0.03378688171505928,
+            0.0010529992869123816,
+            0.00014014738553669304,
+            0.005950582213699818,
+            -0.003069985192269087,
+            -0.009742137044668198,
+            0.03227177634835243,
+            -0.003939277958124876,
+            -0.0740584209561348,
+            0.050089433789253235,
+            -0.026590125635266304,
+            -0.007628562860190868,
+            -0.028938541188836098,
+            -0.01736312545835972,
+            0.037362534552812576,
+            0.005662711802870035,
+            0.0098709212616086,
+            0.0106284748762846,
+            -0.0007892759749665856,
+            0.028923390433192253,
+            0.00910579226911068,
+            -0.024923507124185562,
+            0.04478655755519867,
+            -0.04790768027305603,
+            0.01801462098956108,
+            -0.04336235672235489,
+            0.022756904363632202,
+            1.6393929399782792e-05,
+            -0.020726662129163742,
+            -0.0013304532039910555,
+            -0.016666175797581673,
+            0.0218023881316185,
+            -0.01730252057313919,
+            0.026802239939570427,
+            0.013825350441038609,
+            -0.015802564099431038,
+            -0.029817303642630577,
+            0.025378039106726646,
+            0.003367325058206916,
+            0.010499690659344196,
+            -0.02075696364045143,
+            0.014332910999655724,
+            0.04393810033798218,
+            -0.02593863010406494,
+            -0.02271145209670067,
+            0.027681002393364906,
+            0.014408666640520096,
+            0.0633314698934555,
+            0.036544378846883774,
+            0.006624804809689522,
+            0.022362977266311646,
+            0.020575150847434998,
+            -0.028968842700123787,
+            -0.01963578537106514,
+            -0.046362269669771194,
+            -0.04757435619831085,
+            0.019878201186656952,
+            -0.04566532000899315,
+            0.0050869714468717575,
+            0.038544319570064545,
+            -0.013704141601920128,
+            -0.014446543529629707,
+            -0.04021093621850014,
+            0.017847958952188492,
+            0.0003908502112608403,
+            0.011976920068264008,
+            -0.0022082682698965073,
+            0.00033285003155469894,
+            -0.011825408786535263,
+            0.008878526277840137,
+            -0.03778676688671112,
+            -0.025196226313710213,
+            0.025181075558066368,
+            0.022862963378429413,
+            0.024120500311255455,
+            0.028802180662751198,
+            -0.03077181987464428,
+            -0.009984553791582584,
+            0.021378157660365105,
+            -0.04242299124598503,
+            -0.006556625012308359,
+            0.011332998983561993,
+            0.0026703758630901575,
+            0.003872992005199194,
+            0.032786913216114044,
+            0.052998438477516174,
+            -0.03172633796930313,
+            0.02065090648829937,
+            -0.010431510396301746,
+            0.04957429692149162,
+            0.00024691506405360997,
+            0.028968842700123787,
+            -0.019514575600624084,
+            -0.0134389977902174,
+            -0.029529431834816933,
+            -0.009795165620744228,
+            -0.01356020662933588,
+            -0.008484598249197006,
+            -0.002998017705976963,
+            -0.009575475007295609,
+            0.0040453351102769375,
+            0.0331808403134346,
+            0.00273855566047132,
+            0.030680913478136063,
+            0.014325335621833801,
+            0.0051475754007697105,
+            -0.08745196461677551,
+            0.026059838011860847,
+            -0.0033616432920098305,
+            0.005859675817191601,
+            0.0062460280023515224,
+            0.03013547509908676,
+            -0.0312718041241169,
+            0.020938776433467865,
+            0.026620427146553993,
+            0.0011145505122840405,
+            0.016969196498394012,
+            0.036847397685050964,
+            -0.04181694984436035,
+            0.009643655270338058,
+            0.012590537779033184,
+            -0.0033881578128784895,
+            0.04584713280200958,
+            0.011628445237874985,
+            0.019514575600624084,
+            0.018529755994677544,
+            0.003569970605894923,
+            0.00297907879576087,
+            0.006533898413181305,
+            0.03724132478237152,
+            0.003462019143626094,
+            -0.009833043441176414,
+            0.014976831153035164,
+            -0.010757259093225002,
+            -0.011219366453588009,
+            -0.037998881191015244,
+            0.014454119838774204,
+            0.008742166683077812,
+            -0.017454031854867935,
+            -0.02959003672003746,
+            0.0026949963066726923,
+            -0.031908150762319565,
+            0.026559824123978615,
+            -0.002445003716275096,
+            0.005738467443734407,
+            -0.010454237461090088,
+            -0.006757376715540886,
+            -0.035423196852207184,
+            0.028453707695007324,
+            -0.02095392718911171,
+            0.0053142374381423,
+            0.0007258309051394463,
+            -0.029438525438308716,
+            0.0060073984786868095,
+            0.017863109707832336,
+            -0.0025965142995119095,
+            0.0025302283465862274,
+            -0.02349930815398693,
+            0.024468975141644478,
+            0.025635607540607452,
+            0.004518806003034115,
+            -0.014037464745342731,
+            -0.013113250024616718,
+            -0.003605954349040985,
+            -0.009742137044668198,
+            -0.016196493059396744,
+            0.024468975141644478,
+            -0.014810169115662575,
+            0.00308892410248518,
+            0.021847840398550034,
+            0.014264730736613274,
+            -0.007045246660709381,
+            -0.004007457755506039,
+            -0.04066546633839607,
+            -0.01218903437256813,
+            -0.04760465770959854,
+            -0.011628445237874985,
+            -0.0034601252991706133,
+            -0.028696123510599136,
+            0.0061513339169323444,
+            -0.04481685906648636,
+            -0.01235569640994072,
+            0.02309022843837738,
+            0.018135828897356987,
+            0.0019488062243908644,
+            -0.009173971600830555,
+            -0.022605394944548607,
+            8.486965816700831e-05,
+            -0.01763584464788437,
+            -0.017044952139258385,
+            0.015393485315144062,
+            0.031059689819812775,
+            -0.03101423755288124,
+            0.04024123772978783,
+            0.003893824527040124,
+            -0.08066429197788239,
+            0.010393633507192135,
+            -0.026362858712673187,
+            0.07011914253234863,
+            -0.0025453795678913593,
+            0.014075342565774918,
+            0.06781618297100067,
+            -0.012431452050805092,
+            -0.02007516659796238,
+            0.03142331540584564,
+            -0.008249756880104542,
+            -1.1311515663692262e-05,
+            -0.02884763479232788,
+            -0.02739313244819641,
+            -0.053271155804395676,
+            -0.009704259224236012,
+            -0.010787560604512691,
+            0.03360506892204285,
+            -0.024317465722560883,
+            -0.0007656024536117911,
+            0.012620840221643448,
+            0.02983245439827442,
+            0.016635872423648834,
+            -0.0015681355725973845,
+            0.008272483013570309,
+            0.02881733328104019,
+            0.0274234339594841,
+            -0.0036154238041490316,
+            0.00814369972795248,
+            -0.034211114048957825,
+            -0.0068066176027059555,
+            -0.0493621826171875,
+            0.011302697472274303,
+            -0.02596893161535263,
+            -0.02596893161535263,
+            0.007643714081496,
+            -0.011476934887468815,
+            0.03748374432325363,
+            -0.024696242064237595,
+            -0.007503566797822714,
+            0.005037730094045401,
+            0.012855681590735912,
+            -0.017757052555680275,
+            -0.03612014651298523,
+            -0.0028502948116511106,
+            0.0026381798088550568,
+            0.007560383062809706,
+            -0.053240854293107986,
+            0.004499867558479309,
+            -0.018469152972102165,
+            0.0279688723385334,
+            -0.02139330841600895,
+            -0.013688990846276283,
+            0.042029064148664474,
+            0.011363301426172256,
+            0.027499189600348473,
+            -0.026468917727470398,
+            0.0017849853029474616,
+            0.012825379148125648,
+            -0.0020056227222085,
+            0.032514192163944244,
+            -0.030574856325984,
+            -0.03257479891180992,
+            0.001548249856568873,
+            -0.007840678095817566,
+            -0.0625436082482338,
+            -0.011386028490960598,
+            -0.00981789268553257,
+            0.017317671328783035,
+            0.0446956530213356,
+            0.04557441547513008,
+            -0.01068150345236063,
+            0.002174178371205926,
+            -0.02213571034371853,
+            -0.04542290419340134,
+            -0.02643861435353756,
+            0.0016486257081851363,
+            -0.019999410957098007,
+            -0.02789311669766903,
+            -0.0018313854234293103,
+            0.011211791075766087,
+            -0.013651113025844097,
+            0.01134057529270649,
+            -0.018560059368610382,
+            -0.01862066239118576,
+            -0.012332970276474953,
+            0.018575210124254227,
+            0.009901223704218864,
+            -0.0018796793883666396,
+            -0.015037435106933117,
+            -0.00980274099856615,
+            0.018969137221574783,
+            -0.011279970407485962,
+            0.0009388927719555795,
+            0.01188601367175579,
+            0.024347767233848572,
+            -0.007083124481141567,
+            0.010961798019707203,
+            -0.034756552428007126,
+            -0.01966608688235283,
+            0.007984613068401814,
+            0.002719616750255227,
+            0.020711511373519897,
+            0.008742166683077812,
+            -0.04275631532073021,
+            -0.044301725924015045,
+            0.016044981777668,
+            -0.0562104657292366,
+            0.028120383620262146,
+            -0.03327174484729767,
+            -0.012514783069491386,
+            -0.02671133354306221,
+            0.0013418165035545826,
+            -0.01828734017908573,
+            0.022438732907176018,
+            0.03778676688671112,
+            -0.0011486404109746218,
+            0.005503625608980656,
+            0.014658658765256405,
+            0.04384719207882881,
+            -0.05021063983440399,
+            -0.041180603206157684,
+            0.03754434734582901,
+            -0.01601468026638031,
+            0.0185146052390337,
+            0.00021744151308666915,
+            0.010348179377615452,
+            -0.014431392773985863,
+            0.022772056981921196,
+            0.013757170177996159,
+            -0.008764892816543579,
+            -0.030574856325984,
+            0.04945308715105057,
+            -0.023650817573070526,
+            -0.049513693898916245,
+            0.0018323323456570506,
+            0.018878230825066566,
+            0.018635815009474754,
+            -0.014870774000883102,
+            -0.02907489985227585,
+            -0.024317465722560883,
+            -0.006060427520424128,
+            -0.00947699323296547,
+            -0.03402930125594139,
+            0.011363301426172256,
+            0.0048445542342960835,
+            0.016666175797581673,
+            -0.00645056739449501,
+            -0.0073141781613230705,
+            -0.009370936080813408,
+            0.022559940814971924,
+            0.012635990977287292,
+            -0.03175663948059082,
+            -0.017484333366155624,
+            -0.02528713271021843,
+            0.004359719809144735,
+            0.010136065073311329,
+            0.022166013717651367,
+            0.024287162348628044,
+            -0.0323626808822155,
+            0.04218057543039322,
+            0.01550711877644062,
+            -0.008727015927433968,
+            -0.0019317612750455737,
+            0.02478714846074581,
+            0.04324115067720413,
+            0.0218023881316185,
+            0.026514369994401932,
+            0.002107892418280244,
+            8.705945947440341e-05,
+            -0.015620751306414604,
+            -0.0011391709558665752,
+            0.01862066239118576,
+            0.04911976307630539,
+            -0.012151156552135944,
+            0.01716616004705429,
+            -0.03578682243824005,
+            0.00396200455725193,
+            -0.02580226957798004,
+            -0.030786970630288124,
+            0.03745344281196594,
+            -0.015060162171721458,
+            0.0191964041441679,
+            0.025044716894626617,
+            -0.022469034418463707,
+            -0.011075431481003761,
+            -0.019423669204115868,
+            -0.026181045919656754,
+            -0.01702980138361454,
+            -0.03281721472740173,
+            -0.018135828897356987,
+            -0.002653330797329545,
+            -0.035635314881801605,
+            -0.04987731948494911,
+            -0.004329417832195759,
+            -0.046392571181058884,
+            -0.010507266037166119,
+            0.017378276214003563,
+            -0.011166337877511978,
+            -0.025711363181471825,
+            -0.005336964037269354,
+            0.003973367623984814,
+            -0.022196315228939056,
+            -0.0066588944755494595,
+            0.008439145050942898,
+            0.035938333719968796,
+            -0.02298417128622532,
+            -0.013060221448540688,
+            -0.0024317465722560883,
+            -0.02887793630361557,
+            -0.007878555916249752,
+            0.008999735116958618,
+            0.007723257411271334,
+            -0.0012357590021565557,
+            -0.05196816474199295,
+            -0.006916462909430265,
+            -0.012992041185498238,
+            0.029196109622716904,
+            0.026468917727470398,
+            0.0012168202083557844,
+            -0.02783251367509365,
+            0.008749742060899734,
+            -0.0072535742074251175,
+            0.02339324913918972,
+            -0.007992188446223736,
+            -0.022908415645360947,
+            -0.02034788578748703,
+            0.030786970630288124,
+            -0.0159086212515831,
+            0.009113367646932602,
+            0.009976978413760662,
+            0.0121360057964921,
+            -0.03899884968996048,
+            -0.001445980160497129,
+            0.0005648507503792644,
+            -0.010128489695489407,
+            0.00423093605786562,
+            -0.006738437805324793,
+            -0.007079336792230606,
+            -0.022090258076786995,
+            -0.0218023881316185,
+            0.03818069398403168,
+            -0.009113367646932602,
+            -0.0035661826841533184,
+            -0.01088604237884283,
+            0.03436262533068657,
+            -0.00595815759152174,
+            -0.01993880607187748,
+            -0.019347915425896645,
+            0.014431392773985863,
+            0.05105910077691078,
+            -0.028529461473226547,
+            -0.03984731063246727,
+            0.0011136035900563002,
+            0.030150625854730606,
+            0.004140029661357403,
+            -0.016908591613173485,
+            0.0035851215943694115,
+            0.056392278522253036,
+            -0.011742077767848969,
+            0.028362801298499107,
+            0.027771908789873123,
+            0.02295386977493763,
+            -0.02021152526140213,
+            -0.01678738370537758,
+            -0.020999381318688393,
+            0.011098157614469528,
+            -0.03027183562517166,
+            0.027059808373451233,
+            0.018969137221574783,
+            0.01538590993732214,
+            0.05054396390914917,
+            -0.03178694099187851,
+            -0.02515077404677868,
+            0.008499749936163425,
+            0.0021722845267504454,
+            0.011007251217961311,
+            -0.007727045100182295,
+            -0.01801462098956108,
+            -0.001950700068846345,
+            -0.014022313989698887,
+            -0.019817598164081573,
+            -0.020938776433467865,
+            0.00954517349600792,
+            0.015696506947278976,
+            0.002649543108418584,
+            0.010863316245377064,
+            0.0016590419691056013,
+            0.01686313934624195,
+            0.01321930717676878,
+            -0.0444229319691658,
+            -7.936555630294606e-05,
+            0.03378688171505928,
+            -0.0293476190418005,
+            -0.012264790013432503,
+            -0.016060132533311844,
+            -0.004439263138920069,
+            -0.01706010289490223,
+            -0.02762039750814438,
+            -0.006874797400087118,
+            -0.01889338344335556,
+            -0.00814369972795248,
+            -0.028196139261126518,
+            -0.009992130100727081,
+            0.01949942484498024,
+            0.022332675755023956,
+            0.001196934375911951,
+            -0.006783891003578901,
+            -0.005901341326534748,
+            0.02092362567782402,
+            0.032514192163944244,
+            0.01952972821891308,
+            0.0058331615291535854,
+            -0.015158643946051598,
+            -0.004295328166335821,
+            -0.012370847165584564,
+            0.016060132533311844,
+            0.012537509202957153,
+            6.344509893096983e-05,
+            0.03493836522102356,
+            0.004170331638306379,
+            -0.022726602852344513,
+            -0.001230077352374792,
+            0.024923507124185562,
+            -0.021590271964669228,
+            -0.0108330138027668,
+            0.040029123425483704,
+            0.0085679292678833,
+            -0.0024506852496415377,
+            -0.019287310540676117,
+            -0.019075196236371994,
+            0.02630225569009781,
+            -0.02133270539343357,
+            0.002083271974697709,
+            0.0482713058590889,
+            0.00035344602656550705,
+            -0.029802151024341583,
+            0.017954016104340553,
+            -0.013242034241557121,
+            -0.018560059368610382,
+            0.01848430372774601,
+            0.0012859469279646873,
+            -0.039968520402908325,
+            0.019014591351151466,
+            0.018408548086881638,
+            0.022635696455836296,
+            -0.02918095886707306,
+            0.0002127068000845611,
+            0.03233237937092781,
+            0.004037759732455015,
+            0.001595596899278462,
+            -0.008530051447451115,
+            -0.0006065162015147507,
+            0.010431510396301746,
+            0.001746160676702857,
+            0.02854461409151554,
+            -0.0030434709042310715,
+            0.027771908789873123,
+            -0.006727074272930622,
+            -0.008863375522196293,
+            0.019181253388524055,
+            -0.02109028771519661,
+            0.013166278600692749,
+            0.0685434341430664,
+            0.0021950111258774996,
+            0.025847723707556725,
+            0.007181606721132994,
+            0.024347767233848572,
+            0.02018122375011444,
+            -0.027271922677755356,
+            -0.02021152526140213,
+            -0.005397567991167307,
+            -0.0014033677289262414,
+            -0.03787767142057419,
+            0.011128460057079792,
+            -0.021302402019500732,
+            -0.013249609619379044,
+            -0.0022347825579345226,
+            -0.002316219499334693,
+            -0.006405114196240902,
+            0.034211114048957825,
+            -0.0014630251098424196,
+            -0.018560059368610382,
+            -0.035423196852207184,
+            0.037726160138845444,
+            0.04048365354537964,
+            0.0389382466673851,
+            -0.005984671879559755,
+            0.03548380360007286,
+            0.0030756669584661722,
+            0.0025302283465862274,
+            0.041453324258327484,
+            0.004791525658220053,
+            -0.011567841283977032,
+            0.02295386977493763,
+            0.012416300363838673,
+            0.03351416438817978,
+            -0.02045394293963909,
+            -0.02630225569009781,
+            -0.003299145260825753,
+            -0.026423463597893715,
+            0.012492056004703045,
+            -0.019726691767573357,
+            0.0018768386216834188,
+            0.007117214612662792,
+            0.015196521766483784,
+            -0.016044981777668,
+            0.014719262719154358,
+            -0.006102093029767275,
+            -0.021014532074332237,
+            0.011719351634383202,
+            -0.007780073676258326,
+            0.011961768381297588,
+            -0.02262054570019245,
+            0.001138224033638835,
+            -0.04063516482710838,
+            0.008280059322714806,
+            0.014022313989698887,
+            -0.01598437689244747,
+            0.03205966204404831,
+            0.020969079807400703,
+            -0.01801462098956108,
+            -0.0009682480012997985,
+            0.0034714885987341404,
+            -0.010393633507192135,
+            -0.011067855171859264,
+            -0.019923655316233635,
+            0.02786281518638134,
+            -0.015802564099431038,
+            -0.025908326730132103,
+            0.02454473078250885,
+            -0.01743887923657894,
+            0.015151068568229675,
+            -0.004299115855246782,
+            0.00888610165566206,
+            -0.019120648503303528,
+            0.02284781076014042,
+            -0.023741723969578743,
+            -0.0009128518868237734,
+            -0.020287280902266502,
+            0.0028389315120875835,
+            0.014135947450995445,
+            -0.018272189423441887,
+            -0.02139330841600895,
+            0.019150950014591217,
+            -0.005420294590294361,
+            -0.003931702114641666,
+            0.010014856234192848,
+            0.0030983935575932264,
+            -0.0029715034179389477,
+            -0.004549108445644379,
+            -0.03281721472740173,
+            -0.02915065549314022,
+            0.0033086147159337997,
+            0.011386028490960598,
+            0.015257125720381737,
+            0.027574945241212845,
+            0.03715042024850845,
+            0.006052851676940918,
+            0.0036949669010937214,
+            0.0009327376610599458,
+            -0.020726662129163742,
+            0.042635105550289154,
+            -0.020469093695282936,
+            0.040847279131412506,
+            0.032544493675231934,
+            -0.0197418425232172,
+            -0.023908386006951332,
+            0.0029052174650132656,
+            0.03115059621632099,
+            0.03151422366499901,
+            0.025211378931999207,
+            0.012310243211686611,
+            0.03690800443291664,
+            -0.004359719809144735,
+            -0.032211173325777054,
+            0.015802564099431038,
+            0.022256920114159584,
+            -0.008105821907520294,
+            0.009673957712948322,
+            0.027438584715127945,
+            0.013279912061989307,
+            -0.007757347077131271,
+            0.030150625854730606,
+            0.0012234487803652883,
+            0.02396899089217186,
+            0.02498411200940609,
+            -0.02190844528377056,
+            0.01743887923657894,
+            -0.0001154677156591788,
+            0.006727074272930622,
+            -0.009840618818998337,
+            0.004348356742411852,
+            0.011810258030891418,
+            0.018272189423441887,
+            0.018908534198999405,
+            0.003246116451919079,
+            -0.07060398161411285,
+            0.019726691767573357,
+            0.004507442936301231,
+            -0.02160542458295822,
+            0.00390140013769269,
+            -0.0024866692256182432,
+            0.012704171240329742,
+            0.02105998620390892,
+            0.012461753562092781,
+            -0.01790856383740902,
+            0.0013626492582261562,
+            0.011961768381297588,
+            -0.00595815759152174,
+            0.04730163514614105,
+            0.0019791084341704845,
+            -0.01437078882008791,
+            -0.016105586662888527,
+            0.013726867735385895,
+            0.02525683119893074,
+            -0.02355991117656231,
+            0.006325571332126856,
+            0.005621046293526888,
+            -0.004147605039179325,
+            -0.010742107406258583,
+            -0.030377892777323723,
+            -0.005280147306621075,
+            -0.03436262533068657,
+            0.023150833323597908,
+            0.015491968020796776,
+            0.014764715917408466,
+            -0.025044716894626617,
+            -0.004280176945030689,
+            0.0046400148421525955,
+            -0.021287251263856888,
+            0.003261267440393567,
+            0.01161329448223114,
+            0.0005104016163386405,
+            -0.020984230563044548,
+            0.00959820207208395,
+            0.013469300232827663,
+            0.06848283112049103,
+            0.0007442962378263474,
+            0.029771849513053894,
+            0.0422411784529686,
+            -0.0012243957025930285,
+            -0.006371024530380964,
+            0.0036286809481680393,
+            -0.019953956827521324,
+            0.003473382443189621,
+            0.025878025218844414,
+            -0.03224147483706474,
+            -0.055058982223272324,
+            -0.007545232307165861,
+            -0.04624105989933014,
+            0.001489539397880435,
+            0.014711687341332436,
+            0.023044776171445847,
+            -0.0037782976869493723,
+            -0.034817155450582504,
+            0.016908591613173485,
+            -0.003071879269555211,
+            0.007014944683760405,
+            -0.02593863010406494,
+            -0.0294839795678854,
+            -0.026014383882284164,
+            -0.02339324913918972,
+            -0.012908710166811943,
+            -0.022726602852344513,
+            0.012590537779033184,
+            -0.0015103721525520086,
+            -0.06587684899568558,
+            -0.009780014865100384,
+            0.016166189685463905,
+            0.001200722181238234,
+            0.011939042247831821,
+            0.011083006858825684,
+            -0.007083124481141567,
+            -0.0018275976181030273,
+            -0.022559940814971924,
+            0.03124150261282921,
+            0.005177877843379974,
+            0.03584742918610573,
+            -0.006503595970571041,
+            -0.009128518402576447,
+            0.01073453202843666,
+            0.04527139291167259,
+            0.0017916138749569654,
+            0.010226971469819546,
+            -0.020560000091791153,
+            0.0015065843472257257,
+            -0.02284781076014042,
+            -0.0031135445460677147,
+            -0.008863375522196293,
+            -0.0021211495622992516,
+            0.015893470495939255,
+            0.0019753205124288797,
+            -0.010204244405031204,
+            0.025438643991947174,
+            0.02251448854804039,
+            -0.009605777449905872,
+            -0.015969226136803627,
+            -0.02437806874513626,
+            -0.014545026235282421,
+            -0.0002627763315103948,
+            -0.010007280856370926,
+            -0.022453883662819862,
+            -0.020878173410892487,
+            -0.002566212322562933,
+            -0.02559015527367592,
+            0.022529639303684235,
+            0.01220418605953455,
+            -4.882668508798815e-05,
+            -0.006723286584019661,
+            -0.038544319570064545,
+            -0.003727162955328822,
+            -0.03118089959025383,
+            -0.005374841392040253,
+            0.03287781774997711,
+            -0.03745344281196594,
+            -0.0002231231628684327,
+            0.014491996727883816,
+            -0.01993880607187748,
+            0.0028938541654497385,
+            0.008961857296526432,
+            0.02769615314900875,
+            -0.027408283203840256,
+            0.0001795638381736353,
+            0.006954340264201164,
+            0.016908591613173485,
+            -0.009878496639430523,
+            0.027196168899536133,
+            -0.04275631532073021,
+            0.011037553660571575,
+            -0.037120118737220764,
+            -0.020423639565706253,
+            -0.01523439958691597,
+            0.008363389410078526,
+            -0.005018791649490595,
+            -0.02637801133096218,
+            -0.010454237461090088,
+            0.006026337388902903,
+            -0.00693918950855732,
+            -0.01902974210679531,
+            -0.010620899498462677,
+            0.027438584715127945,
+            0.00931033119559288,
+            0.007977037690579891,
+            -0.02955973520874977,
+            -0.01631770096719265,
+            0.006280118133872747,
+            0.052089374512434006,
+            -0.01584801822900772,
+            0.014938953332602978,
+            -0.006639956030994654,
+            0.02342355251312256,
+            -0.028484009206295013,
+            -0.01872672140598297,
+            0.0315445251762867,
+            0.013946558348834515,
+            0.007064185570925474,
+            0.028953691944479942,
+            0.027196168899536133,
+            0.027105262503027916,
+            -0.04654408246278763,
+            0.009886072017252445,
+            -0.002613559365272522,
+            -0.008757317438721657,
+            0.03402930125594139,
+            0.01916610263288021,
+            0.022150862962007523,
+            0.0009744031121954322,
+            -0.01730252057313919,
+            0.0139844361692667,
+            0.014885924756526947,
+            0.007681591901928186,
+            -0.014242004603147507,
+            -0.02660527639091015,
+            0.003367325058206916,
+            -0.007992188446223736,
+            0.00047276070108637214,
+            0.020726662129163742,
+            -0.009052763693034649,
+            0.030999086797237396,
+            -0.014605630189180374,
+            -0.02593863010406494,
+            0.008916404098272324,
+            -0.02268115058541298,
+            -0.0041741193272173405,
+            0.0004476667381823063,
+            0.010083036497235298,
+            -0.023847781121730804,
+            -0.04669559374451637,
+            0.019408518448472023,
+            0.028393102809786797,
+            0.005579380784183741,
+            0.01328748743981123,
+            -0.035332292318344116,
+            0.008992158807814121,
+            0.012310243211686611,
+            0.011348150670528412,
+            0.0019847899675369263,
+            0.035362593829631805,
+            -0.00028834378463216126,
+            0.012885984033346176,
+            -0.008620957843959332,
+            0.028362801298499107,
+            0.00667025800794363,
+            -0.0323626808822155,
+            -0.0378473699092865,
+            -0.01513591781258583,
+            0.015257125720381737,
+            0.006692984607070684,
+            -0.011840560473501682,
+            -0.02278720773756504,
+            -0.038817036896944046,
+            0.015787413343787193,
+            -0.012916285544633865,
+            -0.038271598517894745,
+            -0.010158791206777096,
+            -0.009742137044668198,
+            0.016635872423648834,
+            -0.006916462909430265,
+            -0.016802534461021423,
+            -0.016090434044599533,
+            0.005821797996759415,
+            7.161247049225494e-05,
+            0.0244083721190691,
+            -0.001251857029274106,
+            -0.006855858489871025,
+            -0.016393456608057022,
+            -0.01188601367175579,
+            -0.0035074723418802023,
+            0.03602924197912216,
+            -0.014219277538359165,
+            -0.02383263036608696,
+            0.014984406530857086,
+            0.02657497487962246,
+            0.011636020615696907,
+            0.03227177634835243,
+            0.007181606721132994,
+            -0.014022313989698887,
+            0.003016956616193056,
+            -0.033029329031705856,
+            -0.0046400148421525955,
+            0.005196816753596067,
+            -0.006427840795367956,
+            -0.015651054680347443,
+            -0.01815097965300083,
+            -0.015832867473363876,
+            0.030756669119000435,
+            -0.02487805485725403,
+            0.0218023881316185,
+            0.030347591266036034,
+            0.003174148965626955,
+            -0.02915065549314022,
+            0.013598084449768066,
+            -0.0072005451656877995,
+            0.01862066239118576,
+            0.0019014590652659535,
+            0.028014326468110085,
+            -0.007461901288479567,
+            -0.004818039946258068,
+            -0.02501441352069378,
+            -0.03696860745549202,
+            -0.023484155535697937,
+            -0.008817922323942184,
+            0.0027537066489458084,
+            0.020984230563044548,
+            -0.0375746488571167,
+            -0.017757052555680275,
+            0.009855770505964756,
+            0.015757111832499504,
+            -0.03027183562517166,
+            -0.0025908327661454678,
+            -0.010158791206777096,
+            -0.0004055278259329498,
+            0.005140000022947788,
+            -0.02227207086980343,
+            -0.008977008052170277,
+            -0.015726808458566666,
+            -0.0010435298318043351,
+            0.006166485138237476,
+            -0.0007409819518215954,
+            -0.05805889517068863,
+            -0.013969285413622856,
+            0.018635815009474754,
+            0.0026400736533105373,
+            0.008908828720450401,
+            0.015938924625515938,
+            -0.0111739132553339,
+            0.0031533162109553814,
+            -0.026756787672638893,
+            0.023893235251307487,
+            -0.009492144919931889,
+            0.009355784393846989,
+            -0.013742019422352314,
+            0.012514783069491386,
+            0.014014738611876965,
+            0.004795313347131014,
+            0.0010681503918021917,
+            -0.0054051438346505165,
+            0.01601468026638031,
+            0.028377952054142952,
+            -0.023756874725222588,
+            0.011454207822680473,
+            -0.0002514130319468677,
+            -0.0032158142421394587,
+            -0.029196109622716904,
+            0.00509454682469368,
+            0.003931702114641666,
+            -0.005109698045998812,
+            0.016651025041937828,
+            -0.036544378846883774,
+            0.01662072166800499,
+            0.02177208475768566,
+            0.032544493675231934,
+            0.014832896180450916,
+            -0.017287369817495346,
+            -0.011658747680485249,
+            -0.012385998852550983,
+            -0.002624922664836049,
+            0.022181164473295212,
+            0.012120855040848255,
+            -0.01815097965300083,
+            -0.024287162348628044,
+            0.011249667964875698,
+            0.0005283934879116714,
+            -0.01203752402216196,
+            0.010795135982334614,
+            -0.013166278600692749,
+            0.01349202636629343,
+            0.012711746618151665,
+            0.00866641104221344,
+            0.00888610165566206,
+            0.022862963378429413,
+            -0.011878437362611294,
+            -0.017711598426103592,
+            0.00656041270121932,
+            -0.008916404098272324,
+            -0.0034089903347194195,
+            -0.012598113156855106,
+            0.010795135982334614,
+            0.014878349378705025,
+            0.06490717828273773,
+            -0.013143551535904408,
+            0.0009526234352961183,
+            -0.020438792183995247,
+            -0.031938452273607254,
+            -0.0017944546416401863,
+            -0.020908474922180176,
+            0.0111739132553339,
+            0.008370965719223022,
+            0.01598437689244747,
+            0.012022373266518116,
+            -0.0021912232041358948,
+            0.00795431062579155,
+            0.019317612051963806,
+            -0.010423935018479824,
+            7.255941454786807e-05,
+            -0.006503595970571041,
+            0.0013683309080079198,
+            0.0035851215943694115,
+            -0.011719351634383202,
+            -0.0026949963066726923,
+            0.035695917904376984,
+            0.01068150345236063,
+            0.011651171371340752,
+            -0.03445352986454964,
+            0.0020435003098100424,
+            0.020150920376181602,
+            0.007602048572152853,
+            -0.008302785456180573,
+            -0.01563590206205845,
+            0.01706010289490223,
+            -0.015317730605602264,
+            -0.0022499337792396545,
+            0.002888172399252653,
+            -0.010151215828955173,
+            0.04581683129072189,
+            -0.019787294790148735,
+            -0.0011742077767848969,
+            0.031847547739744186,
+            0.0030207443051040173,
+            -0.0040983641520142555,
+            0.005408931523561478,
+            -0.003473382443189621,
+            -0.0069013116881251335,
+            -0.02139330841600895,
+            0.0031381649896502495,
+            0.00667025800794363,
+            -0.019287310540676117,
+            -0.0072005451656877995,
+            -0.007719469256699085,
+            0.006897523999214172,
+            0.023105379194021225,
+            -0.009492144919931889,
+            -0.00016145357221830636,
+            -0.008575504645705223,
+            0.009848194196820259,
+            -0.0011912527261301875,
+            -0.044301725924015045,
+            -0.006147546228021383,
+            0.021181194111704826,
+            0.000973456131760031,
+            0.0005757405888289213,
+            -0.007450537756085396,
+            0.005007428117096424,
+            -0.017423728480935097,
+            -0.008454295806586742,
+            0.02369627170264721,
+            0.01736312545835972,
+            -0.009113367646932602,
+            0.010863316245377064,
+            -0.0035680767614394426,
+            0.00433320552110672,
+            -0.020014561712741852,
+            -0.01095422264188528,
+            0.003462019143626094,
+            0.015272277407348156,
+            -0.0009819786064326763,
+            -0.029029447585344315,
+            -0.0012272365856915712,
+            -0.00024194362049456686,
+            0.02918095886707306,
+            0.01486319862306118,
+            -0.008961857296526432,
+            -0.021211495622992516,
+            -0.013242034241557121,
+            -0.001237652963027358,
+            -0.015726808458566666,
+            -0.009620928205549717,
+            0.011855711229145527,
+            0.0009137988090515137,
+            0.008992158807814121,
+            -0.018135828897356987,
+            -0.027347678318619728,
+            -0.016166189685463905,
+            0.0018465365283191204,
+            -0.012870832346379757,
+            0.003340810537338257,
+            0.015514694154262543,
+            -0.03430201858282089,
+            -0.015060162171721458,
+            -0.03805948421359062,
+            0.008280059322714806,
+            -0.0029658216517418623,
+            -0.02024182677268982,
+            -0.023605365306138992,
+            0.009484568610787392,
+            0.019181253388524055,
+            -0.0015548784285783768,
+            0.019847899675369263,
+            -0.000889651826582849,
+            0.020060013979673386,
+            -0.0073141781613230705,
+            0.0031703610438853502,
+            -0.013007191941142082,
+            0.0030245319940149784,
+            0.008333087898790836,
+            0.02890823967754841,
+            -0.023075077682733536,
+            -0.006503595970571041,
+            -0.004685468040406704,
+            0.004746072459965944,
+            0.012658718042075634,
+            0.014613205567002296,
+            0.020741812884807587,
+            -0.008272483013570309,
+            -0.0027612822595983744,
+            0.012484480626881123,
+            -0.003799130441620946,
+            0.007295239716768265,
+            0.002390081062912941,
+            0.0019677451346069574,
+            0.004162756260484457,
+            0.008477022871375084,
+            0.0053710537031292915,
+            0.027650700882077217,
+            -0.0028275682125240564,
+            0.0031703610438853502,
+            0.024620486423373222,
+            0.02322658896446228,
+            -0.018878230825066566,
+            -0.028074929490685463,
+            0.002024561632424593,
+            0.050695475190877914,
+            0.007245998363941908,
+            -0.006810405291616917,
+            -0.03927157074213028,
+            -0.0031760428100824356,
+            0.028635520488023758,
+            0.023650817573070526,
+            0.01584801822900772,
+            -0.010567869991064072,
+            -0.012098127976059914,
+            0.02772645466029644,
+            0.013295062817633152,
+            0.02616589516401291,
+            -0.025302285328507423,
+            0.010704229585826397,
+            -0.03869583085179329,
+            0.0449683703482151,
+            0.004439263138920069,
+            -0.014476845972239971,
+            0.0011884119594469666,
+            0.010386057198047638,
+            -0.004537744913250208,
+            0.006431628484278917,
+            -5.8059369621332735e-05,
+            -0.0008086882880888879,
+            0.011923890560865402,
+            0.012007221579551697,
+            0.004265025723725557,
+            -0.02399929240345955,
+            0.007359631359577179,
+            0.004272601567208767,
+            0.03005971945822239,
+            0.005431658122688532,
+            -0.0013863227795809507,
+            0.012385998852550983,
+            -0.005810434930026531,
+            0.008855799213051796,
+            -0.010719381272792816,
+            0.020469093695282936,
+            -0.02918095886707306,
+            1.0986003871948924e-05,
+            0.02051454596221447,
+            -0.0015785519499331713,
+            0.0038199631962925196,
+            -0.023741723969578743,
+            0.01186328660696745,
+            -0.052968136966228485,
+            0.021166043356060982,
+            -0.018741872161626816,
+            0.021741783246397972,
+            0.019590331241488457,
+            -0.009734561666846275,
+            -0.012052674777805805,
+            0.03545350208878517,
+            0.0026760573964565992,
+            -0.014264730736613274,
+            -0.010643625631928444,
+            -0.017999470233917236,
+            -0.02830219641327858,
+            0.041968461126089096,
+            -0.0006117243901826441,
+            0.0028275682125240564,
+            0.0024222771171480417,
+            -0.010666352696716785,
+            -0.008923979476094246,
+            0.001954487757757306,
+            0.020499395206570625,
+            -0.03369597718119621,
+            -0.011128460057079792,
+            -0.008689138107001781,
+            -0.02566591091454029,
+            -0.00020300064352340996,
+            -0.011961768381297588,
+            0.026211349293589592,
+            0.01240872498601675,
+            -0.03696860745549202,
+            0.003780191531404853,
+            0.023650817573070526,
+            0.001603172393515706,
+            0.018120678141713142,
+            0.002130619017407298,
+            0.0010795136913657188,
+            0.009742137044668198,
+            -0.010476963594555855,
+            0.026559824123978615,
+            -0.009204274043440819,
+            0.008658835664391518,
+            -0.001950700068846345,
+            -0.017287369817495346,
+            0.017969166859984398,
+            0.004083212930709124,
+            0.024120500311255455,
+            0.011954193003475666,
+            0.011605718173086643,
+            0.01112088467925787,
+            -0.019878201186656952,
+            0.020090317353606224,
+            -0.0018891488434746861,
+            0.009075489826500416,
+            0.024726543575525284,
+            -0.005863463506102562,
+            0.020832719281315804,
+            0.015257125720381737,
+            -0.00020229043730068952,
+            0.011295121163129807,
+            -0.020120618864893913,
+            -0.011779955588281155,
+            0.0085679292678833,
+            -0.009605777449905872,
+            -0.010325453244149685,
+            0.009901223704218864,
+            -0.010105762630701065,
+            -0.015052586793899536,
+            -0.0071247899904847145,
+            0.016635872423648834,
+            0.0028976418543606997,
+            -0.02610529027879238,
+            -1.7326075976598077e-05,
+            0.008901252411305904,
+            0.006166485138237476,
+            0.018741872161626816,
+            0.011317848227918148,
+            -0.006692984607070684,
+            0.010363331064581871,
+            0.011476934887468815,
+            0.0053104497492313385,
+            -0.005079395603388548,
+            -0.013431422412395477,
+            -0.0024241709616035223,
+            0.018090376630425453,
+            0.009810317307710648,
+            -0.01922670565545559,
+            0.004564259201288223,
+            0.0008877579239197075,
+            -0.01420412678271532,
+            -0.019347915425896645,
+            -0.0038483713287860155,
+            -0.004170331638306379,
+            0.0027915844693779945,
+            0.0509984977543354,
+            -0.0006216673064045608,
+            0.007893706671893597,
+            -0.025741664692759514,
+            -0.006299056578427553,
+            -0.00027556007262319326,
+            -0.0036097420379519463,
+            0.01272689737379551,
+            -0.01824188604950905,
+            -0.03724132478237152,
+            0.012711746618151665,
+            -0.0012584856012836099,
+            0.014923802576959133,
+            -0.00297907879576087,
+            0.014226853847503662,
+            -0.017347972840070724,
+            0.01916610263288021,
+            -0.0011117096291854978,
+            -0.016029831022024155,
+            0.0029374135192483664,
+            -0.011143610812723637,
+            -0.021317552775144577,
+            0.005916492082178593,
+            -0.006848283112049103,
+            -0.03896854817867279,
+            0.0029885482508689165,
+            -0.010423935018479824,
+            0.0011798894265666604,
+            -0.003912763670086861,
+            0.020166072994470596,
+            -0.0061513339169323444,
+            0.0008697660523466766,
+            0.02177208475768566,
+            0.0017026013229042292,
+            0.01601468026638031,
+            -0.014499573037028313,
+            0.010999675840139389,
+            0.012598113156855106,
+            0.02210540883243084,
+            -0.032180868089199066,
+            -0.01393140759319067,
+            0.014211702160537243,
+            0.0026116655208170414,
+            -0.005382417235523462,
+            0.020635755732655525,
+            -0.01294658798724413,
+            0.019650936126708984,
+            -0.0005577487172558904,
+            -0.011704200878739357,
+            -0.03524138405919075,
+            -0.016529815271496773,
+            -3.343888238305226e-05,
+            -0.008439145050942898,
+            0.025317436084151268,
+            0.01760554127395153,
+            -0.0108330138027668,
+            0.026468917727470398,
+            0.0037574649322777987,
+            0.014598054811358452,
+            -0.014257155358791351,
+            -0.005999823100864887,
+            -0.0003773563075810671,
+            -0.0119314668700099,
+            0.015264702029526234,
+            -0.03512017801403999,
+            0.0008342557121068239,
+            -0.00839369185268879,
+            0.005242269951850176,
+            -0.0254992488771677,
+            0.018257036805152893,
+            0.025817420333623886,
+            -0.0031419529113918543,
+            -0.005037730094045401,
+            -0.022802358493208885,
+            0.028968842700123787,
+            0.03049910068511963,
+            0.0024828813038766384,
+            -0.0013825349742546678,
+            -0.00352072948589921,
+            0.0033218718599528074,
+            0.0211205892264843,
+            -0.037998881191015244,
+            -0.009098216891288757,
+            -0.028696123510599136,
+            0.042332086712121964,
+            -0.008098246529698372,
+            -0.013431422412395477,
+            0.01862066239118576,
+            -0.0024715180043131113,
+            0.02355991117656231,
+            -0.004689255729317665,
+            -0.004855917301028967,
+            0.03148392215371132,
+            0.011514811776578426,
+            0.02772645466029644,
+            -0.04015033319592476,
+            0.014704111963510513,
+            -0.006867222022265196,
+            0.004867280833423138,
+            -0.013484450988471508,
+            0.005302873905748129,
+            0.02580226957798004,
+            -0.018060073256492615,
+            0.0068066176027059555,
+            0.030665762722492218,
+            0.00748084019869566,
+            0.002448791405186057,
+            0.0165904201567173,
+            0.003890036838129163,
+            0.010855740867555141,
+            0.000841831264551729,
+            0.0052574207074940205,
+            0.022423582151532173,
+            0.014454119838774204,
+            -0.016029831022024155,
+            -0.015120766125619411,
+            -0.00044790346873924136,
+            -0.004450626205652952,
+            0.005261208396404982,
+            0.013242034241557121,
+            0.0061172437854111195,
+            -0.015681356191635132,
+            0.01052999310195446,
+            0.014158673584461212,
+            0.013128400780260563,
+            -0.0039847311563789845,
+            -0.03412020578980446,
+            -0.02443867363035679,
+            -0.021438762545585632,
+            0.003104075090959668,
+            -0.0008281006012111902,
+            -0.0021211495622992516,
+            0.0017954016802832484,
+            0.017347972840070724,
+            -0.0008352026343345642,
+            0.00379155483096838,
+            0.01686313934624195,
+            -0.01415109820663929,
+            -0.022090258076786995,
+            -0.0035964848939329386,
+            -0.027529491111636162,
+            0.0053710537031292915,
+            0.001844642567448318,
+            -0.00965123064815998,
+            0.008530051447451115,
+            -0.0046059247106313705,
+            -0.007223271764814854,
+            -0.012196609750390053,
+            -0.02007516659796238,
+            0.005780132487416267,
+            0.016878290101885796,
+            0.005730891600251198,
+            -0.009272454306483269,
+            -0.003939277958124876,
+            0.011575416661798954,
+            0.017484333366155624,
+            0.017317671328783035,
+            -0.009295180439949036,
+            0.009628504514694214,
+            0.0004022135108243674,
+            0.004670316819101572,
+            -0.01834794320166111,
+            0.0045869858004152775,
+            -0.0027953721582889557,
+            -0.008658835664391518,
+            0.01105270441621542,
+            -0.00031201730598695576,
+            0.01628739945590496,
+            -0.013090522959828377,
+            0.005624833982437849,
+            0.010333028621971607,
+            -0.020590301603078842,
+            -0.0067308624275028706,
+            -0.01692374423146248,
+            0.026484068483114243,
+            -0.00504530593752861,
+            0.00045405857963487506,
+            -0.02978700026869774,
+            0.0033616432920098305,
+            0.000848933297675103,
+            -0.000845619011670351,
+            -0.012764775194227695,
+            0.010749682784080505,
+            0.03242328763008118,
+            0.0005179771687835455,
+            0.0216660276055336,
+            -0.0036154238041490316,
+            0.0007480839849449694,
+            -0.00785582885146141,
+            0.004977126140147448,
+            -0.004916521720588207,
+            -0.009901223704218864,
+            -0.010863316245377064,
+            0.021726632490754128,
+            -0.0033029329497367144,
+            0.0003356908564455807,
+            0.011332998983561993,
+            -0.012067826464772224,
+            0.0014620780711993575,
+            -0.0047687990590929985,
+            0.00971941091120243,
+            0.0061702728271484375,
+            0.009113367646932602,
+            -0.01176480483263731,
+            0.05451354384422302,
+            -0.0106284748762846,
+            0.005537715740501881,
+            -0.00439380994066596,
+            -0.022226616740226746,
+            -0.017317671328783035,
+            0.012711746618151665,
+            -0.017544938251376152,
+            0.03375658020377159,
+            0.024150803685188293,
+            0.0006467612111009657,
+            0.010393633507192135,
+            -0.006958128418773413,
+            0.023923536762595177,
+            -0.002602196065708995,
+            0.0010795136913657188,
+            -0.004855917301028967,
+            0.015802564099431038,
+            0.04511988162994385,
+            0.017393426969647408,
+            -0.001152428099885583,
+            0.00829521007835865,
+            0.020575150847434998,
+            0.00814369972795248,
+            -0.0020075165666639805,
+            0.014219277538359165,
+            -0.01024212222546339,
+            0.00900731049478054,
+            -0.004458202049136162,
+            0.027438584715127945,
+            0.00814369972795248,
+            -0.006310420110821724,
+            -0.0036836036015301943,
+            -0.0008209985098801553,
+            0.014173824340105057,
+            0.006950552575290203,
+            0.01963578537106514,
+            -0.00461350055411458,
+            -0.0012575386790558696,
+            0.013870803639292717,
+            0.014491996727883816,
+            -0.00552635220810771,
+            0.001724380999803543,
+            -0.01435563713312149,
+            -0.010204244405031204,
+            0.013575357384979725,
+            0.006086941808462143,
+            0.006310420110821724,
+            -0.014514723792672157,
+            -0.00980274099856615,
+            -0.004670316819101572,
+            0.0015435151290148497,
+            0.015052586793899536,
+            0.011060279794037342,
+            -0.004814252257347107,
+            0.006052851676940918,
+            0.013825350441038609,
+            -0.014014738611876965,
+            0.006147546228021383,
+            0.007208121009171009,
+            0.008855799213051796,
+            0.004787737503647804,
+            -0.010817863047122955,
+            -0.00428396463394165,
+            -0.0063937511295080185,
+            0.0012670081341639161,
+            0.016832837834954262,
+            0.013181429356336594,
+            0.0049581872299313545,
+            0.027529491111636162,
+            -0.00839369185268879,
+            -0.012302667833864689,
+            0.02339324913918972,
+            0.011257244274020195,
+            0.0031703610438853502,
+            0.014651083387434483,
+            -0.012635990977287292,
+            -0.012446602806448936,
+            0.006655106786638498,
+            0.004848341923207045,
+            0.006049063988029957,
+            0.009151245467364788,
+            0.00450365524739027,
+            0.007333117071539164,
+            -0.007098275702446699,
+            -0.007427811156958342,
+            0.0014810169814154506,
+            0.01642375811934471,
+            0.020393338054418564,
+            -0.024968961253762245,
+            -0.0018806264270097017,
+            0.0009052763343788683,
+            0.010878467001020908,
+            -0.003316190093755722,
+            0.012310243211686611,
+            5.65205882594455e-05,
+            -0.0074240234680473804,
+            0.010158791206777096,
+            -0.02119634486734867,
+            -0.012681444175541401,
+            0.0012717428617179394,
+            0.013037494383752346,
+            -0.019590331241488457,
+            0.018226735293865204,
+            -0.008893677033483982,
+            -0.013348091393709183,
+            0.013893529772758484,
+            0.020014561712741852,
+            0.004090788308531046,
+            -0.007738408166915178,
+            -0.021862991154193878,
+            -0.01127239502966404,
+            0.016060132533311844,
+            0.020272130146622658,
+            0.000947888707742095,
+            -0.0006472347304224968,
+            0.010173942893743515,
+            -0.013113250024616718,
+            -0.023605365306138992,
+            -0.005230906419456005,
+            0.010386057198047638,
+            0.01746918261051178,
+            0.018832778558135033,
+            -0.006685409229248762,
+            0.02443867363035679,
+            -0.0059127043932676315,
+            -0.015499543398618698,
+            2.86597842205083e-05,
+            -0.02802947722375393,
+            -0.017423728480935097,
+            0.00013778003631159663,
+            0.00246773031540215,
+            0.008598231710493565,
+            0.0008555619278922677,
+            0.01943882182240486,
+            0.000904329412151128,
+            0.017590390518307686,
+            -0.000659071491099894,
+            -0.002009410411119461,
+            -0.0013995799235999584,
+            -0.0015340456739068031,
+            0.02254479005932808,
+            -0.012166308239102364,
+            -0.01176480483263731,
+            0.022302372381091118,
+            0.006852070800960064,
+            0.014719262719154358,
+            -0.027635548263788223,
+            0.013757170177996159,
+            -0.015499543398618698,
+            0.017423728480935097,
+            -0.03408990427851677,
+            0.006420265417546034,
+            0.01489350013434887,
+            -0.010348179377615452,
+            -0.020499395206570625,
+            -0.029650641605257988,
+            -0.024150803685188293,
+            -0.0038710979279130697,
+            -0.010855740867555141,
+            -0.016847988590598106,
+            0.006121031939983368,
+            -0.005802859086543322,
+            0.018499454483389854,
+            -0.007079336792230606,
+            0.033877789974212646,
+            0.006969491485506296,
+            -0.004056698642671108,
+            0.003835114184767008,
+            0.022469034418463707,
+            -0.006102093029767275,
+            0.02109028771519661,
+            0.004408961161971092,
+            -0.001467759720981121,
+            0.013681414537131786,
+            -0.006552837323397398,
+            -0.002361672930419445,
+            0.005821797996759415,
+            -0.010484539903700352,
+            0.022014502435922623,
+            0.005924067925661802,
+            -0.007708106189966202,
+            -0.0006775368237867951,
+            -0.011454207822680473,
+            0.014726839028298855,
+            0.006530110724270344,
+            0.0022480397019535303,
+            -0.011295121163129807,
+            0.011249667964875698,
+            -0.004859705455601215,
+            0.0040642740204930305,
+            0.009484568610787392,
+            -0.01774190180003643,
+            -0.028862785547971725,
+            0.011416330002248287,
+            0.019120648503303528,
+            0.006382387597113848,
+            -0.0055755930952727795,
+            -0.0025681061670184135,
+            0.00839369185268879,
+            0.010651201009750366,
+            0.022150862962007523,
+            -0.012984465807676315,
+            -8.889415767043829e-05,
+            0.013204156421124935,
+            -0.007170243188738823,
+            -0.01056029461324215,
+            0.00487485621124506,
+            0.023438703268766403,
+            0.0068823727779090405,
+            -0.04327145218849182,
+            -0.018469152972102165,
+            0.01909034699201584,
+            0.007136153522878885,
+            -0.04154422879219055,
+            0.0026381798088550568,
+            -0.003689285134896636,
+            -0.023787178099155426,
+            0.016105586662888527,
+            -0.0011817833874374628,
+            -0.01828734017908573,
+            0.028756728395819664,
+            0.02830219641327858,
+            0.01834794320166111,
+            2.6987840101355687e-05,
+            -0.013332940638065338,
+            -0.0318172425031662,
+            -0.023590214550495148,
+            0.008280059322714806,
+            -0.02660527639091015,
+            -0.0072725131176412106,
+            0.011242092587053776,
+            -0.017454031854867935,
+            0.02972639724612236,
+            -0.008507325313985348,
+            0.026938600465655327,
+            0.011446632444858551,
+            0.001691238023340702,
+            -0.013446573168039322,
+            -0.018802475184202194,
+            -0.01350717805325985,
+            -0.008196728304028511,
+            0.014090493321418762,
+            0.017923714593052864,
+            0.011007251217961311,
+            0.011332998983561993,
+            0.02251448854804039,
+            -0.012158732861280441,
+            0.007969462312757969,
+            -2.888172457460314e-05,
+            -0.008961857296526432,
+            0.0026343921199440956,
+            -0.018393397331237793,
+            0.00868156272917986,
+            0.0062308767810463905,
+            0.005458172410726547,
+            -0.007208121009171009,
+            0.010742107406258583,
+            0.02712041325867176,
+            -0.018847929313778877,
+            0.011401179246604443,
+            -0.031362712383270264,
+            -0.014787442982196808,
+            -0.01056029461324215,
+            0.026862844824790955,
+            0.007757347077131271,
+            -0.015726808458566666,
+            -0.007514929864555597,
+            -0.01545409020036459,
+            0.012885984033346176,
+            0.0032442226074635983,
+            0.005764981731772423,
+            0.015817714855074883,
+            0.021893294528126717,
+            0.01746918261051178,
+            -0.00795431062579155,
+            0.006852070800960064,
+            0.004674104508012533,
+            0.028090080246329308,
+            -0.01777220331132412,
+            0.020802417770028114,
+            0.009954252280294895,
+            -0.003935490269213915,
+            0.005685438401997089,
+            -0.014257155358791351,
+            0.008530051447451115,
+            -0.007325541693717241,
+            0.030514251440763474,
+            0.0033067206386476755,
+            0.014075342565774918,
+            -0.02062060497701168,
+            0.006575563922524452,
+            -0.03172633796930313,
+            0.009022461250424385,
+            0.013704141601920128,
+            0.012976890429854393,
+            -0.007677804213017225,
+            -0.024181105196475983,
+            -0.0119314668700099,
+            0.0057990713976323605,
+            -0.01279507763683796,
+            -0.01046181283891201,
+            -0.0034828518982976675,
+            0.012446602806448936,
+            0.018847929313778877,
+            -0.01848430372774601,
+            -0.019923655316233635,
+            -0.009583051316440105,
+            0.02898399345576763,
+            -0.015787413343787193,
+            -0.001471547526307404,
+            0.02413565292954445,
+            0.021241798996925354,
+            0.015022284351289272,
+            0.002155239460989833,
+            0.020090317353606224,
+            -0.0007111532613635063,
+            -0.0237265732139349,
+            -0.02813553437590599,
+            0.007302815094590187,
+            -0.006912675220519304,
+            0.01943882182240486,
+            -0.0035548193845897913,
+            -0.005658924113959074,
+            -0.0034033088013529778,
+            -0.007503566797822714,
+            -0.0165904201567173,
+            -0.01787826046347618,
+            0.0021647089160978794,
+            -0.008454295806586742,
+            0.005863463506102562,
+            -0.0002795845502987504,
+            -0.013105674646794796,
+            0.013401119969785213,
+            -0.014075342565774918,
+            0.005772557109594345,
+            -0.002653330797329545,
+            0.001962063368409872,
+            0.0035926972050219774,
+            0.009113367646932602,
+            -0.007336904760450125,
+            -0.06381630152463913,
+            -0.013105674646794796,
+            -0.003704436356201768,
+            0.006071790587157011,
+            -0.00019578021601773798,
+            -0.016635872423648834,
+            -0.013196581043303013,
+            0.01706010289490223,
+            0.008863375522196293,
+            -0.013870803639292717,
+            0.016954045742750168,
+            0.009961827658116817,
+            0.003622999181970954,
+            -0.024044746533036232,
+            -0.0008148433989845216,
+            0.008757317438721657,
+            0.032756611704826355,
+            -0.008643684908747673,
+            -0.0003839849086944014,
+            0.0036684523802250624,
+            -0.0033332351595163345,
+            -0.010052734054625034,
+            -0.014582903124392033,
+            -0.011848135851323605,
+            0.011908739805221558,
+            0.01790856383740902,
+            -0.014264730736613274,
+            -0.02007516659796238,
+            -0.0007106798002496362,
+            0.012931437231600285,
+            -0.026590125635266304,
+            0.0007244104635901749,
+            -0.007787649054080248,
+            -0.01052999310195446,
+            0.018302490934729576,
+            -0.019014591351151466,
+            0.005249845329672098,
+            0.00048294031876139343,
+            0.008507325313985348,
+            -0.023544760420918465,
+            -0.024893205612897873,
+            0.0035131541080772877,
+            -0.023847781121730804,
+            0.03612014651298523,
+            -0.021590271964669228,
+            -0.012082977220416069,
+            0.034483831375837326,
+            -0.0037423139438033104,
+            0.01702980138361454,
+            0.0023995505180209875,
+            0.02739313244819641,
+            -0.011439057067036629,
+            -0.00617784820497036,
+            0.012499631382524967,
+            0.009575475007295609,
+            0.0032593735959380865,
+            0.03775646165013313,
+            -0.01257538702338934,
+            -0.013901105150580406,
+            -0.005556654185056686,
+            -0.012954163365066051,
+            0.0006775368237867951,
+            0.014431392773985863,
+            0.028393102809786797,
+            0.004257450345903635,
+            -0.00628769351169467,
+            -0.006143758539110422,
+            -0.0014649189542979002,
+            -0.004242299124598503,
+            -0.00947699323296547,
+            -0.00715130427852273,
+            0.017287369817495346,
+            -0.025241680443286896,
+            0.016711628064513206,
+            -0.006420265417546034,
+            0.021135739982128143,
+            -0.011757229454815388,
+            0.007007369305938482,
+            -0.008961857296526432,
+            0.003492321353405714,
+            0.0023635667748749256,
+            -0.005109698045998812,
+            0.009287605062127113,
+            -0.014772292226552963,
+            0.008454295806586742,
+            0.013742019422352314,
+            0.0011183382011950016,
+            -0.010590597055852413,
+            -0.019272159785032272,
+            0.0022972808219492435,
+            -0.000710206339135766,
+            0.01181783340871334,
+            -0.0047498601488769054,
+            -0.000981031684204936,
+            0.0002921315317507833,
+            -0.014476845972239971,
+            -0.009575475007295609,
+            0.0006050958181731403,
+            0.005302873905748129,
+            -0.008961857296526432,
+            0.010378481820225716,
+            -0.005693014245480299,
+            -0.00034160923678427935,
+            0.02627195231616497,
+            0.006526322569698095,
+            -0.007632351014763117,
+            -0.014098069630563259,
+            0.01563590206205845,
+            0.015370759181678295,
+            -0.007454325910657644,
+            0.01933276280760765,
+            -0.016196493059396744,
+            -0.0008366230758838356,
+            0.013916256837546825,
+            -0.009219424799084663,
+            0.0017698341980576515,
+            -0.008196728304028511,
+            0.03554440662264824,
+            -0.0008787619881331921,
+            -0.0035415622405707836,
+            -0.0006401326390914619,
+            0.012976890429854393,
+            -0.0026779514737427235,
+            0.001182730309665203,
+            0.01132542360574007,
+            0.005321812815964222,
+            0.0185146052390337,
+            0.00328020635060966,
+            -0.03378688171505928,
+            -0.007780073676258326,
+            -0.0065490491688251495,
+            -0.008090670220553875,
+            0.009022461250424385,
+            0.025196226313710213,
+            0.01990850456058979,
+            0.006408901885151863,
+            0.001881573349237442,
+            0.018423698842525482,
+            0.015757111832499504,
+            -0.01489350013434887,
+            0.016363153234124184,
+            0.02802947722375393,
+            -0.010030006989836693,
+            0.022741753607988358,
+            -0.005253633018583059,
+            -0.008893677033483982,
+            0.008946705609560013,
+            -0.0015378334792330861,
+            -0.017393426969647408,
+            -0.015681356191635132,
+            0.005022579338401556,
+            -0.008999735116958618,
+            -0.0022897052112966776,
+            -0.018605511635541916,
+            0.004670316819101572,
+            0.017105557024478912,
+            -0.004965762607753277,
+            -3.5243989259470254e-05,
+            0.019484274089336395,
+            -0.006965703796595335,
+            -0.016908591613173485,
+            0.006325571332126856,
+            -0.015007133595645428,
+            0.007946735247969627,
+            -0.013666263781487942,
+            0.010492115281522274,
+            -0.006098304875195026,
+            0.018817627802491188,
+            0.005446809343993664,
+            -0.008090670220553875,
+            -0.0008958069374784827,
+            -0.018984289839863777,
+            0.02065090648829937,
+            -0.033089932054281235,
+            0.00839369185268879,
+            -0.026226500049233437,
+            -0.0009781909175217152,
+            0.0008849170990288258,
+            0.014605630189180374,
+            -0.001218714052811265,
+            0.019696388393640518,
+            -0.002751812804490328,
+            -0.020090317353606224,
+            -0.007086912170052528,
+            -0.0010634156642481685,
+            -0.01618134044110775,
+            -0.030923331156373024,
+            -0.004272601567208767,
+            0.0036267871037125587,
+            -0.004711982328444719,
+            0.0021135739516466856,
+            0.017984317615628242,
+            -0.005613470915704966,
+            0.009173971600830555,
+            -0.02690829709172249,
+            -0.007416448090225458,
+            -0.00355103169567883,
+            0.0010776197304949164,
+            -0.011242092587053776,
+            0.0010596278589218855,
+            0.011226941831409931,
+            -0.019044892862439156,
+            0.013461724855005741,
+            -0.021378157660365105,
+            -0.004371083341538906,
+            -0.008507325313985348,
+            -0.00135412672534585,
+            -0.013446573168039322,
+            -0.028120383620262146,
+            0.025029566138982773,
+            -0.00021495578403118998,
+            -0.003528305096551776,
+            -0.011999646201729774,
+            0.023181134834885597,
+            0.002914686920121312,
+            0.027817361056804657,
+            -0.03157482668757439,
+            0.0013427634257823229,
+            0.011552689597010612,
+            0.0025984083767980337,
+            -0.017287369817495346,
+            0.004746072459965944,
+            -0.005234694108366966,
+            0.02210540883243084,
+            -0.0011694730492308736,
+            -0.0005340751959010959,
+            -0.008234606124460697,
+            0.030999086797237396,
+            0.009136094711720943,
+            0.0012944694608449936,
+            -0.013226882554590702,
+            -0.008060368709266186,
+            -0.002759388415142894,
+            -0.0011069749016314745,
+            0.012855681590735912,
+            0.01290113478899002,
+            -0.012105703353881836,
+            0.0013607552973553538,
+            -0.006878585088998079,
+            0.0011694730492308736,
+            -0.002935519441962242,
+            -0.0004966709529981017,
+            -0.01149208564311266,
+            0.003257479751482606,
+            0.016272246837615967,
+            -0.00780280027538538,
+            -0.006480869371443987,
+            0.0007750718505121768,
+            0.009181547909975052,
+            0.012461753562092781,
+            -0.012742049060761929,
+            -0.018226735293865204,
+            -0.011908739805221558,
+            0.016029831022024155,
+            0.010522416792809963,
+            -0.008204303681850433,
+            -0.013234458863735199,
+            -0.02583257108926773,
+            0.01169662456959486,
+            -0.03293842449784279,
+            -0.012242063879966736,
+            0.005821797996759415,
+            0.028559764847159386,
+            0.019999410957098007,
+            0.012226912193000317,
+            0.001332347048446536,
+            0.002950670663267374,
+            0.00023342114582192153,
+            -0.01916610263288021,
+            0.014408666640520096,
+            -0.011143610812723637,
+            -0.018166130408644676,
+            0.0075906855054199696,
+            0.011939042247831821,
+            0.03148392215371132,
+            0.006942977197468281,
+            -0.0010160685051232576,
+            0.01686313934624195,
+            -0.001218714052811265,
+            -0.013582932762801647,
+            0.01420412678271532,
+            -0.024256860837340355,
+            0.016438908874988556,
+            0.02860521711409092,
+            -0.004780162125825882,
+            0.0008531945641152561,
+            -0.020938776433467865,
+            0.00482561532407999,
+            -0.011529963463544846,
+            -0.002177966060116887,
+            -0.003287781961262226,
+            -0.03460504114627838,
+            0.007946735247969627,
+            -0.006825556512922049,
+            0.01046181283891201,
+            -0.00814369972795248,
+            0.006655106786638498,
+            -0.012431452050805092,
+            0.004253662656992674,
+            0.01513591781258583,
+            0.016105586662888527,
+            0.007060397882014513,
+            -0.025211378931999207,
+            -0.009196698665618896,
+            0.008105821907520294,
+            -0.01349202636629343,
+            -0.02295386977493763,
+            -0.006852070800960064,
+            0.009840618818998337,
+            -0.01061332318931818,
+            -0.00488243205472827,
+            -0.028180986642837524,
+            -0.013681414537131786,
+            -0.0014327229000627995,
+            -0.017817657440900803,
+            -0.005556654185056686,
+            -0.003100287402048707,
+            0.01571165770292282,
+            -0.0049733384512364864,
+            -0.006185423582792282,
+            0.004742284305393696,
+            -0.023878084495663643,
+            0.0037423139438033104,
+            -0.008280059322714806,
+            0.0008266801596619189,
+            -0.0032517979852855206,
+            -0.0014762822538614273,
+            -0.007969462312757969,
+            -0.014332910999655724,
+            0.006893736310303211,
+            -0.016469212248921394,
+            0.002954458352178335,
+            -0.015173795633018017,
+            -0.016499513760209084,
+            -0.00682934420183301,
+            0.00775355938822031,
+            0.021575121209025383,
+            0.006541473791003227,
+            0.021484214812517166,
+            0.005469535943120718,
+            0.028680972754955292,
+            0.0003953482082579285,
+            0.003041577059775591,
+            -8.958956641436089e-06,
+            0.003232859307900071,
+            0.019347915425896645,
+            0.006571775767952204,
+            0.018984289839863777,
+            -0.008742166683077812,
+            0.010514841414988041,
+            -0.010355755686759949,
+            0.010014856234192848,
+            0.006333146709948778,
+            -0.006473293993622065,
+            -0.003799130441620946,
+            -0.010348179377615452,
+            -0.0031419529113918543,
+            -0.016544966027140617,
+            0.0027366618160158396,
+            0.00693161366507411,
+            0.0071588801220059395,
+            -0.011582992039620876,
+            -0.005230906419456005,
+            -0.014704111963510513,
+            -0.0064354161731898785,
+            -0.01831764169037342,
+            0.014385939575731754,
+            0.029226411134004593,
+            -0.016332851722836494,
+            -0.01977214403450489,
+            -0.007772498298436403,
+            0.0007296186522580683,
+            -0.006636167876422405,
+            -0.01240872498601675,
+            -0.009151245467364788,
+            -0.035059571266174316,
+            -0.017666146159172058,
+            0.007586897816509008,
+            0.02484775334596634,
+            0.01387837901711464,
+            -0.013628385961055756,
+            -0.0006983695784583688,
+            -0.016984347254037857,
+            -0.010787560604512691,
+            -0.01535560842603445,
+            -0.02054484933614731,
+            -0.02498411200940609,
+            0.0077952248975634575,
+            -0.004033972043544054,
+            0.014931377954781055,
+            0.008113397285342216,
+            0.0032631612848490477,
+            -0.004371083341538906,
+            -0.0024109138175845146,
+            -0.013196581043303013,
+            0.0012925754999741912,
+            -0.004787737503647804,
+            -0.0017291157273575664,
+            0.03554440662264824,
+            -0.008098246529698372,
+            -0.005871038883924484,
+            0.011060279794037342,
+            0.011128460057079792,
+            -0.013764745555818081,
+            0.0006642796797677875,
+            -0.011249667964875698,
+            0.021847840398550034,
+            -0.021620575338602066,
+            0.019257009029388428,
+            0.0014507147716358304,
+            -0.010840589180588722,
+            0.028999146074056625,
+            0.01892368495464325,
+            0.00743159931153059,
+            0.006579351611435413,
+            0.01672677882015705,
+            0.0044316877610981464,
+            0.016196493059396744,
+            -0.0026552248746156693,
+            -0.004632438998669386,
+            0.009295180439949036,
+            0.015681356191635132,
+            -0.000702630786690861,
+            -0.006942977197468281,
+            0.016438908874988556,
+            0.004056698642671108,
+            0.0075414446182549,
+            -0.01356020662933588,
+            -0.008961857296526432,
+            0.001507531269453466,
+            0.006257391534745693,
+            -0.005939218681305647,
+            -0.0026003022212535143,
+            0.005530139897018671,
+            0.003943065647035837,
+            -0.013302638195455074,
+            0.01689344085752964,
+            -0.016166189685463905,
+            0.008742166683077812,
+            -0.0037972365971654654,
+            0.0121360057964921,
+            -0.006799042224884033,
+            0.0013153020991012454,
+            -0.01818128302693367,
+            -0.003869204083457589,
+            0.0038597346283495426,
+            0.012173883616924286,
+            -0.01262841559946537,
+            0.025484098121523857,
+            0.008166425861418247,
+            0.006602078210562468,
+            0.0059467945247888565,
+            -0.012855681590735912,
+            0.027211319655179977,
+            0.005969521123915911,
+            0.009886072017252445,
+            -0.008893677033483982,
+            0.011136035434901714,
+            0.035362593829631805,
+            0.03060515783727169,
+            0.014135947450995445,
+            0.004469565115869045,
+            -0.018999440595507622,
+            0.015120766125619411,
+            0.0018427487229928374,
+            0.0023806116078048944,
+            0.00863610953092575,
+            -0.011348150670528412,
+            0.01305264513939619,
+            -0.009484568610787392,
+            0.015241974964737892,
+            -0.021272100508213043,
+            -0.011749654076993465,
+            -0.014408666640520096,
+            -0.021075136959552765,
+            -0.01430260855704546,
+            0.034180812537670135,
+            -0.002346521709114313,
+            -0.010393633507192135,
+            0.0045112306252121925,
+            0.008477022871375084,
+            -0.006079366430640221,
+            0.0053142374381423,
+            -0.008204303681850433,
+            -0.013749594800174236,
+            -0.01889338344335556,
+            0.006628592498600483,
+            -0.01550711877644062,
+            0.007651289459317923,
+            -0.003304826794192195,
+            0.0037688282318413258,
+            0.007014944683760405,
+            0.018105527386069298,
+            0.00530666159465909,
+            -0.00135412672534585,
+            0.0014033677289262414,
+            0.0016865032957866788,
+            -0.00931033119559288,
+            0.0108330138027668,
+            0.015408637002110481,
+            -0.0011921997647732496,
+            -0.011279970407485962,
+            -0.009136094711720943,
+            -0.042877525091171265,
+            0.0006264019757509232,
+            -0.013128400780260563,
+            0.0064354161731898785,
+            -0.011196639388799667,
+            0.0022234192583709955,
+            0.015863168984651566,
+            -0.0054051438346505165,
+            -0.0010918239131569862,
+            0.02193874679505825,
+            0.013014768250286579,
+            0.021075136959552765,
+            0.02065090648829937,
+            0.014188976027071476,
+            -0.015302578918635845,
+            -0.03469594568014145,
+            0.002221525413915515,
+            -0.010030006989836693,
+            0.013772320933640003,
+            0.004238511435687542,
+            0.016060132533311844,
+            -0.023893235251307487,
+            -0.011878437362611294,
+            -0.006662682630121708,
+            0.014431392773985863,
+            0.01491622719913721,
+            -0.0032271775417029858,
+            0.02690829709172249,
+            0.009280029684305191,
+            0.0020510759204626083,
+            -0.02281750924885273,
+            0.003312402404844761,
+            0.005321812815964222,
+            -0.008370965719223022,
+            -0.016681326553225517,
+            -0.0004552422615233809,
+            -0.0012168202083557844,
+            -0.0007817004225216806,
+            -0.021650876849889755,
+            0.018832778558135033,
+            -0.004018820822238922,
+            0.0037574649322777987,
+            0.013840501196682453,
+            0.01516621932387352,
+            0.017984317615628242,
+            0.008370965719223022,
+            0.001497115008533001,
+            -0.006605865899473429,
+            -0.0007973249885253608,
+            -0.0005828426801599562,
+            -0.007389933802187443,
+            -0.005598319694399834,
+            0.006382387597113848,
+            -0.01134057529270649,
+            0.008810346014797688,
+            0.000896753859706223,
+            0.006113456096500158,
+            0.010605747811496258,
+            -0.004836978856474161,
+            -0.011628445237874985,
+            0.014628356322646141,
+            -0.0010548931313678622,
+            -0.0055755930952727795,
+            -0.011060279794037342,
+            0.003624893259257078,
+            0.013575357384979725,
+            -0.003096499713137746,
+            -0.013499601744115353,
+            0.00910579226911068,
+            0.002009410411119461,
+            -0.027438584715127945,
+            0.015499543398618698,
+            0.010295150801539421,
+            -0.013370818458497524,
+            0.004980913829058409,
+            -0.003371112747117877,
+            -0.03027183562517166,
+            0.004791525658220053,
+            0.002660906407982111,
+            -0.010643625631928444,
+            0.028938541188836098,
+            -0.003821857040748,
+            0.029256712645292282,
+            -0.004980913829058409,
+            -0.006905099377036095,
+            -0.011355726048350334,
+            0.009416389279067516,
+            -0.018681267276406288,
+            -0.0037953427527099848,
+            -0.008727015927433968,
+            -0.00915882084518671,
+            -0.0009838725673034787,
+            -0.005185453221201897,
+            0.006984642706811428,
+            0.005121061112731695,
+            -0.012704171240329742,
+            0.006757376715540886,
+            -0.010257272981107235,
+            -0.00020856392802670598,
+            0.0006718551740050316,
+            -0.008408842608332634,
+            0.013885954394936562,
+            0.005996035411953926,
+            -0.008340663276612759,
+            -0.0005032995832152665,
+            0.027817361056804657,
+            0.05051366239786148,
+            -0.018863080069422722,
+            0.0009270560112781823,
+            0.01452229917049408,
+            -0.0028824908658862114,
+            -0.0021590273827314377,
+            -0.011976920068264008,
+            0.005821797996759415,
+            -0.019423669204115868,
+            0.013317788951098919,
+            0.006473293993622065,
+            -0.008833073079586029,
+            0.003202557098120451,
+            -0.016711628064513206,
+            0.002621134975925088,
+            -0.0018058179412037134,
+            -0.018681267276406288,
+            -0.015529844909906387,
+            -0.012060250155627728,
+            -0.024181105196475983,
+            0.0004235197266098112,
+            -0.0008546149474568665,
+            -0.013870803639292717,
+            0.006579351611435413,
+            0.008431569673120975,
+            0.032483890652656555,
+            0.0011912527261301875,
+            0.010128489695489407,
+            -2.1513333194889128e-05,
+            -0.0010416359873488545,
+            -0.005988460034132004,
+            -0.01523439958691597,
+            0.00015387803432531655,
+            -0.002342734020203352,
+            0.022590242326259613,
+            0.005549078807234764,
+            0.017560089007019997,
+            0.0017887729918584228,
+            -0.016514664515852928,
+            -0.00596573343500495,
+            -0.007408872712403536,
+            -0.018060073256492615,
+            0.0062649669125676155,
+            -0.005473323632031679,
+            0.009605777449905872,
+            -0.015863168984651566,
+            -0.02731737680733204,
+            0.00792400911450386,
+            0.012234487570822239,
+            0.028514310717582703,
+            -0.0035623949952423573,
+            0.001716805505566299,
+            -0.005533927585929632,
+            0.016408607363700867,
+            0.032180868089199066,
+            -0.0008550884085707366,
+            0.006950552575290203,
+            -0.019999410957098007,
+            0.0017404790269210935,
+            0.013704141601920128,
+            -0.008401267230510712,
+            -0.01821158453822136,
+            0.003939277958124876,
+            -0.020635755732655525,
+            0.004583198111504316,
+            -0.020393338054418564,
+            0.030408194288611412,
+            -0.0023522034753113985,
+            0.008105821907520294,
+            0.004310478921979666,
+            0.01760554127395153,
+            -0.0031646795105189085,
+            0.010545143857598305,
+            0.013916256837546825,
+            0.014908651821315289,
+            0.0009980766335502267,
+            -0.011189064010977745,
+            -0.03790797293186188,
+            0.001859793672338128,
+            0.00817400123924017,
+            0.018635815009474754,
+            -0.01980244740843773,
+            0.011681473813951015,
+            -0.01296931505203247,
+            -0.020363036543130875,
+            -0.01003758329898119,
+            0.011575416661798954,
+            -0.006139970384538174,
+            0.03378688171505928,
+            -0.005874827038496733,
+            -0.008204303681850433,
+            0.005136212334036827,
+            0.010454237461090088,
+            0.0066778333857655525,
+            0.014704111963510513,
+            0.0326354019343853,
+            0.010196669027209282,
+            0.014173824340105057,
+            -0.004859705455601215,
+            0.02399929240345955,
+            -0.003931702114641666,
+            0.003954428713768721,
+            0.00509454682469368,
+            -0.022832660004496574,
+            -0.0040983641520142555,
+            -0.003931702114641666,
+            -0.016711628064513206,
+            -0.007458113599568605,
+            0.0032366469968110323,
+            0.008802770636975765,
+            -0.014045040123164654,
+            -0.00649980828166008,
+            -0.01350717805325985,
+            0.01218903437256813,
+            0.002730980049818754,
+            0.018302490934729576,
+            -0.029802151024341583,
+            0.009825468063354492,
+            -0.017484333366155624,
+            -0.017044952139258385,
+            0.006511171814054251,
+            -0.0038332203403115273,
+            0.006855858489871025,
+            -0.018272189423441887,
+            -0.01318900566548109,
+            0.012817803770303726,
+            0.0007897494360804558,
+            0.013060221448540688,
+            -0.009825468063354492,
+            0.0009706153068691492,
+            0.006064215209335089,
+            -0.009833043441176414,
+            -0.010961798019707203,
+            -0.007977037690579891,
+            -0.009681533090770245,
+            -0.008848223835229874,
+            0.005621046293526888,
+            0.01656011864542961,
+            -0.013742019422352314,
+            -0.003685497445985675,
+            0.016135888174176216,
+            0.006416477728635073,
+            -0.008166425861418247,
+            0.004075637552887201,
+            -0.003793448908254504,
+            -9.948806837201118e-05,
+            0.0013740125577896833,
+            0.012439027428627014,
+            -0.0017632056260481477,
+            0.024665940552949905,
+            -0.02654467150568962,
+            0.007598260883241892,
+            -0.0054051438346505165,
+            0.005446809343993664,
+            0.018226735293865204,
+            -0.02366596832871437,
+            -0.013082947582006454,
+            0.006276329979300499,
+            -0.01966608688235283,
+            -0.008060368709266186,
+            0.0054051438346505165,
+            -0.00016405766655225307,
+            -0.019923655316233635,
+            -0.012923861853778362,
+            -0.034786853939294815,
+            -0.02309022843837738,
+            0.024665940552949905,
+            0.008401267230510712,
+            -0.0073672072030603886,
+            0.018060073256492615,
+            -0.018438849598169327,
+            0.0172722190618515,
+            -0.002225313102826476,
+            -0.006613441277295351,
+            0.003935490269213915,
+            0.007674016058444977,
+            0.025362888351082802,
+            0.002653330797329545,
+            0.008264907635748386,
+            -0.0042082094587385654,
+            -0.014931377954781055,
+            -0.006321783177554607,
+            0.01980244740843773,
+            0.02471139281988144,
+            0.018605511635541916,
+            -0.018575210124254227,
+            0.010742107406258583,
+            -0.011749654076993465,
+            0.01818128302693367,
+            -0.006537686102092266,
+            0.00039037675014697015,
+            0.01859036087989807,
+            0.007969462312757969,
+            0.025241680443286896,
+            0.019923655316233635,
+            0.006852070800960064,
+            0.014128371141850948,
+            0.011908739805221558,
+            0.001204509986564517,
+            -0.012832955457270145,
+            -0.01154511421918869,
+            -0.02840825356543064,
+            0.042847223579883575,
+            0.0025434857234358788,
+            0.008317936211824417,
+            -0.027635548263788223,
+            -0.00021389048197306693,
+            -0.0034942151978611946,
+            -0.018832778558135033,
+            0.005939218681305647,
+            0.03260510042309761,
+            0.03284751623868942,
+            0.021438762545585632,
+            -2.888172457460314e-05,
+            0.0012234487803652883,
+            0.0329081192612648,
+            -0.013673839159309864,
+            -0.00937851145863533,
+            -0.01034060399979353,
+            -0.018393397331237793,
+            -0.006852070800960064,
+            -0.0032423287630081177,
+            -0.01031030248850584,
+            -0.0061172437854111195,
+            -0.009272454306483269,
+            0.0238629337400198,
+            0.0001374249259242788,
+            -0.0022594030015170574,
+            0.006605865899473429,
+            -0.020878173410892487,
+            0.011537538841366768,
+            -0.02813553437590599,
+            -0.013431422412395477,
+            -0.008651260286569595,
+            -0.006950552575290203,
+            -0.018802475184202194,
+            0.008643684908747673,
+            -0.006942977197468281,
+            0.021438762545585632,
+            0.002914686920121312,
+            -0.023590214550495148,
+            0.007530081085860729,
+            0.015317730605602264,
+            -0.0003825644962489605,
+            0.014037464745342731,
+            -0.003890036838129163,
+            0.009060339070856571,
+            0.0045112306252121925,
+            0.011681473813951015,
+            0.016954045742750168,
+            0.010651201009750366,
+            0.009030036628246307,
+            0.003437398700043559,
+            -0.001167579204775393,
+            0.01878732442855835,
+            -0.0007163614500313997,
+            -0.03405960276722908,
+            2.9236827685963362e-05,
+            -0.014628356322646141,
+            0.029499130323529243,
+            -0.009795165620744228,
+            0.009514871053397655,
+            0.008105821907520294,
+            0.007098275702446699,
+            -0.0067119235172867775,
+            0.007916433736681938,
+            -0.008742166683077812,
+            0.0052763596177101135,
+            0.015832867473363876,
+            -0.0042877523228526115,
+            -0.0014980619307607412,
+            0.007840678095817566,
+            0.00407184986397624,
+            -0.005454384721815586,
+            -0.019711541011929512,
+            -0.012287517078220844,
+            0.00726114958524704,
+            -0.02065090648829937,
+            -0.004367295652627945,
+            0.01161329448223114,
+            -0.005454384721815586,
+            -0.009620928205549717,
+            0.01393140759319067,
+            0.02136300690472126,
+            -0.002382505452260375,
+            0.022893264889717102,
+            -0.01349202636629343,
+            -0.009833043441176414,
+            -0.007984613068401814,
+            -0.004374871030449867,
+            -0.041120000183582306,
+            0.025984082370996475,
+            0.01068150345236063,
+            -0.006871009711176157,
+            0.01895398646593094,
+            0.0022499337792396545,
+            -0.002954458352178335,
+            -0.0011041341349482536,
+            0.00010499217023607343,
+            -0.03281721472740173,
+            0.03660498186945915,
+            -0.007007369305938482,
+            0.0005449649761430919,
+            0.010272424668073654,
+            0.042029064148664474,
+            -0.004590773954987526,
+            -0.005859675817191601,
+            0.01225721463561058,
+            0.005336964037269354,
+            0.031938452273607254,
+            -0.010416359640657902,
+            -0.014340486377477646,
+            -0.01815097965300083,
+            -0.004833190701901913,
+            -0.01939336769282818,
+            -0.0038313264958560467,
+            0.014037464745342731,
+            -0.019650936126708984,
+            -0.011242092587053776,
+            -0.001287840772420168,
+            -0.012537509202957153,
+            -0.0010747789638116956,
+            0.008598231710493565,
+            0.017832808196544647,
+            -0.011348150670528412,
+            -0.004450626205652952,
+            -2.2904154320713133e-05,
+            -0.005003640428185463,
+            0.0185146052390337,
+            0.0020056227222085,
+            -0.01702980138361454,
+            0.01321930717676878,
+            -0.00526499655097723,
+            0.006749800872057676,
+            0.0074429623782634735,
+            0.007939159870147705,
+            0.05145302787423134,
+            0.010317877866327763,
+            -0.001339922659099102,
+            -0.01607528328895569,
+            0.023196285590529442,
+            -0.01749948412179947,
+            -0.014181400649249554,
+            -0.02119634486734867,
+            -0.0029923359397798777,
+            -0.007060397882014513,
+            -0.028014326468110085,
+            -9.137987944995984e-05,
+            0.0016221113037317991,
+            -0.0024980325251817703,
+            -0.010045158676803112,
+            0.0005208179936744273,
+            -0.011529963463544846,
+            -0.006954340264201164,
+            -0.008545203134417534,
+            -0.016044981777668,
+            0.02240842953324318,
+            -0.021817538887262344,
+            -0.008855799213051796,
+            0.011514811776578426,
+            0.03118089959025383,
+            -0.005011215806007385,
+            -0.014173824340105057,
+            -0.019817598164081573,
+            0.0025643182452768087,
+            -0.01763584464788437,
+            0.012181458994746208,
+            -0.01225721463561058,
+            0.015045011416077614,
+            -0.03142331540584564,
+            -0.006079366430640221,
+            -0.008992158807814121,
+            0.012840530835092068,
+            0.019817598164081573,
+            -0.0025093958247452974,
+            0.00606800289824605,
+            -0.010295150801539421,
+            0.030211230739951134,
+            -0.00012286569108255208,
+            -0.018605511635541916,
+            0.00873459130525589,
+            0.007977037690579891,
+            0.012537509202957153,
+            -0.004435475450009108,
+            0.00257757562212646,
+            0.011567841283977032,
+            0.012742049060761929,
+            -0.01634800247848034,
+            -0.003174148965626955,
+            -0.017969166859984398,
+            0.005996035411953926,
+            0.006999793462455273,
+            0.021711481735110283,
+            -0.0013550736475735903,
+            0.018272189423441887,
+            -0.01563590206205845,
+            0.00019412307301536202,
+            -0.011749654076993465,
+            -0.0082270298153162,
+            -0.0028237802907824516,
+            -0.003958216868340969,
+            0.006685409229248762,
+            -0.028211290016770363,
+            -0.01274962443858385,
+            0.005287723150104284,
+            -0.004992276895791292,
+            -0.03405960276722908,
+            0.0053710537031292915,
+            -0.02725677192211151,
+            0.013128400780260563,
+            -0.002397656673565507,
+            -0.017847958952188492,
+            -0.00030846628942526877,
+            0.026332557201385498,
+            -0.003677921835333109,
+            -0.012537509202957153,
+            0.015544996596872807,
+            -0.006757376715540886,
+            -0.02119634486734867,
+            -0.005420294590294361,
+            0.014658658765256405,
+            -0.0010719380807131529,
+            -0.012393574230372906,
+            -0.029635490849614143,
+            0.012992041185498238,
+            0.0026874206960201263,
+            -0.007386146113276482,
+            0.010893618687987328,
+            0.025953780859708786,
+            0.008151275105774403,
+            -0.005924067925661802,
+            0.0015757111832499504,
+            0.0049581872299313545,
+            0.00300938100554049,
+            0.020287280902266502,
+            0.02728707529604435,
+            -0.008014915511012077,
+            -0.0011145505122840405,
+            -0.014954105019569397,
+            0.015802564099431038,
+            0.0013020449550822377,
+            0.010507266037166119,
+            0.021893294528126717,
+            0.007045246660709381,
+            -0.005738467443734407,
+            0.003905187826603651,
+            -0.023135682567954063,
+            0.01420412678271532,
+            -0.012272365391254425,
+            0.000622140767518431,
+            0.008749742060899734,
+            -0.007283876184374094,
+            0.01437078882008791,
+            -0.0015245762187987566,
+            0.0339989997446537,
+            0.025211378931999207,
+            0.0043218424543738365,
+            0.0007362472242675722,
+            -0.028559764847159386,
+            0.003547244006767869,
+            0.008651260286569595,
+            0.014499573037028313,
+            -0.0076664406806230545,
+            0.006261179223656654,
+            -0.004355932120233774,
+            0.00959820207208395,
+            -0.0065490491688251495,
+            -0.015893470495939255,
+            0.006113456096500158,
+            0.0012689019786193967,
+            0.006795254070311785,
+            0.0023408401757478714,
+            -0.008863375522196293,
+            -0.024362917989492416,
+            -0.025984082370996475,
+            0.021984200924634933,
+            -0.03918066248297691,
+            0.0073293293826282024,
+            0.011234517209231853,
+            0.03324144333600998,
+            -0.008158850483596325,
+            -0.004980913829058409,
+            0.006276329979300499,
+            -0.016711628064513206,
+            -0.004147605039179325,
+            0.008204303681850433,
+            0.012385998852550983,
+            -0.002420383272692561,
+            0.00974971242249012,
+            0.01318900566548109,
+            -0.012378423474729061,
+            0.03660498186945915,
+            0.016514664515852928,
+            0.01467380952090025,
+            -0.004007457755506039,
+            0.01312082540243864,
+            0.0066172294318675995,
+            0.019044892862439156,
+            0.025165924802422523,
+            -0.017454031854867935,
+            -0.002365460619330406,
+            -0.02292356640100479,
+            0.024165954440832138,
+            0.00726114958524704,
+            0.004556683823466301,
+            -0.008310360834002495,
+            0.014870774000883102,
+            0.0017954016802832484,
+            0.0028199926018714905,
+            -0.006696772295981646,
+            -0.010173942893743515,
+            0.019241856411099434,
+            0.0034317169338464737,
+            0.014870774000883102,
+            4.8501173296244815e-05,
+            -0.005503625608980656,
+            0.006052851676940918,
+            0.014832896180450916,
+            -0.023590214550495148,
+            -0.00203592493198812,
+            -0.019484274089336395,
+            -0.03202936053276062,
+            0.009022461250424385,
+            0.0016135887708514929,
+            -0.007052822504192591,
+            0.03230207785964012,
+            -0.010083036497235298,
+            -0.019787294790148735,
+            -0.008067944087088108,
+            -0.013113250024616718,
+            -0.010469388216733932,
+            0.0008271536789834499,
+            0.003908975515514612,
+            -0.00012428610352799296,
+            -0.009317907504737377,
+            -0.005749830510467291,
+            0.01990850456058979,
+            -0.0030358952935785055,
+            -0.00753386877477169,
+            0.00868156272917986
+          ],
+          "embedding": [
+            0.0040707582550429606,
+            -0.010230625151989194,
+            -0.021520242101903186,
+            -0.010115756270640508,
+            0.026738734586159035,
+            0.01796370233223267,
+            -0.012245199029994153,
+            0.06506924494147726,
+            -0.0014087587222033442,
+            0.009300068719271429,
+            0.003787396973047584,
+            0.002818772639754174,
+            0.022129077062176306,
+            -0.021526709808445683,
+            -0.0034388135367225824,
+            0.02763761747879405,
+            0.013210013805395276,
+            0.025255872672296602,
+            0.00029011480909986703,
+            -0.05438974762290585,
+            0.016671640187490865,
+            -0.016426252259893328,
+            -0.056591180651012554,
+            0.009335648425523075,
+            0.006141624459418827,
+            0.02568943956694641,
+            0.019377132054042116,
+            0.017946464337078746,
+            -0.057184776985927686,
+            0.03208527139096687,
+            0.030132392063755547,
+            -0.003846032371791839,
+            -0.008376524427531415,
+            -0.0010513963058713435,
+            -0.02734868388724371,
+            -0.014879400370545479,
+            0.04383920959748825,
+            0.02563613659516289,
+            -0.020117467074438997,
+            0.0398016699155716,
+            0.029896497884197775,
+            -0.0004195844530067313,
+            -0.023302720788253625,
+            -0.001168337916577517,
+            0.009149021369661628,
+            -0.02508145721900992,
+            -0.004307594138480351,
+            0.00550940415564303,
+            0.014016998918718476,
+            -0.028607379616884317,
+            -0.0018317656587526954,
+            -0.01973464940415756,
+            0.03762738392056255,
+            0.015358106058583114,
+            -0.03931907682104936,
+            0.011519637298832444,
+            0.004482080558226775,
+            0.0747867063383242,
+            0.05550679018263077,
+            0.023243129601931176,
+            0.02671631686665238,
+            -0.009686146566573536,
+            -0.04088582189725495,
+            -0.0065344623594924085,
+            0.019617295999325712,
+            -0.004722087402575777,
+            -0.002871844114437583,
+            0.003935087265300519,
+            0.03229890413393201,
+            -0.02813577047904649,
+            -0.05124172132597234,
+            0.03325235407841968,
+            -0.008687655557217192,
+            -0.011027174838246948,
+            -0.003653553063547375,
+            -0.02350737945182988,
+            0.016223087939183468,
+            0.03789252356462714,
+            0.0034350623378603083,
+            0.00195158657888739,
+            0.011789712862232108,
+            -0.0004435439660769551,
+            -0.0037330456918125894,
+            0.0031552713243866858,
+            0.04465432992298031,
+            -0.0032465421571915712,
+            -0.0075366743861922745,
+            -0.059958752731410256,
+            0.016784616861008125,
+            -0.028734506065772967,
+            -0.03035575736273943,
+            0.02680390892611741,
+            0.02338368719038086,
+            -0.0030658375437784277,
+            -0.03754896050794851,
+            0.07005659523248504,
+            0.0020711575553570695,
+            0.013034425083076923,
+            -0.02316832262132332,
+            0.021332142030732894,
+            -0.0318756233020585,
+            0.02270782509507388,
+            0.01570810309454529,
+            0.004788124787612736,
+            0.026727250200142456,
+            -0.022380142778058276,
+            -0.008593910379436904,
+            0.014439754382387034,
+            0.015007697617383174,
+            0.07662531963612977,
+            0.011195998137723795,
+            -0.010809283538470098,
+            -0.042227822120219305,
+            0.003824320933275175,
+            -0.03797629427607094,
+            -0.02347390684453348,
+            -0.0526876440280276,
+            -0.017799861680287913,
+            0.01701482796266917,
+            0.002730395144917155,
+            0.03789451759899825,
+            0.038338545350313946,
+            0.028463789278406586,
+            0.015494583705011921,
+            -0.05700237266350479,
+            0.047895170359075166,
+            -0.0064406894534929406,
+            -0.0007954721295065988,
+            0.012543029839729358,
+            -0.004703334868741216,
+            -0.0060569679284068,
+            0.012324901254711192,
+            -0.019883935809784068,
+            -0.04253565942154215,
+            0.02164409276211115,
+            -0.0075367555415808635,
+            0.03617159147722157,
+            0.07567745988686005,
+            -0.04225977020036783,
+            -0.022548837589622003,
+            0.029811602145850144,
+            -0.02175792196819533,
+            0.02069861458115363,
+            -0.00941331829664397,
+            -0.035479639885943264,
+            0.020352552580119105,
+            0.003678588588550461,
+            0.06776105274336729,
+            -0.02346047721661687,
+            0.02721075076349443,
+            -0.029386781707644253,
+            0.023530831921117993,
+            -0.02077660533100372,
+            0.020022475323372906,
+            0.024195118676513715,
+            -0.028378518720494666,
+            -0.012900969023913307,
+            0.019264089970719624,
+            -0.0374983624629265,
+            -0.019726171557384355,
+            0.00979636085642416,
+            -0.014479528435832063,
+            -0.014533384583777662,
+            0.005904623758767224,
+            0.010344475500263744,
+            0.05659646358026115,
+            0.04163866113482093,
+            0.007152494819169405,
+            -0.07584500130974763,
+            0.05471858597878099,
+            0.03580211013427615,
+            0.044161429444475306,
+            0.02258180566046248,
+            0.00980779221334715,
+            0.0011193505603397395,
+            0.018086566768892507,
+            0.015605066702191871,
+            -0.00954727399396313,
+            -0.027950079146919375,
+            0.03028514025485407,
+            -0.006351669761257431,
+            0.005095804801686436,
+            0.009620040085743162,
+            0.019748001227987134,
+            0.033709535059510916,
+            0.008203559956940196,
+            0.007389752602218563,
+            0.04197771878351742,
+            -0.02067363758471765,
+            0.007543739188471913,
+            0.011207941261799052,
+            0.013964827251172813,
+            0.03839562075248879,
+            0.0009775244543292637,
+            0.006469507611492216,
+            -0.022808235395746654,
+            -0.024795004398633292,
+            0.006249024864814684,
+            0.011292877109066925,
+            0.01596839619826819,
+            -0.0014105396434174883,
+            -0.04152075438277094,
+            0.008267788411079337,
+            -0.028834191212891387,
+            0.030940526405027906,
+            -0.0018271056434014425,
+            0.006185475047820603,
+            -0.007920574674727123,
+            0.012907560753043616,
+            -0.015165704767187252,
+            -0.021962244496450727,
+            0.0037071053072833147,
+            -0.013508376064336042,
+            -0.015178823540513864,
+            -0.02518557053061311,
+            0.011693256916760944,
+            -0.01716015674592331,
+            0.02048897513447166,
+            -0.003412078966361745,
+            -0.001807298511378529,
+            -0.0012575655363779904,
+            0.03116103961636643,
+            0.011668533361641842,
+            -0.011637010548030312,
+            -0.040260195687925236,
+            -0.011129376691522074,
+            -0.0034493180842871908,
+            -0.02098244789711689,
+            0.026658743116016938,
+            -0.01923736325953796,
+            -0.009723937181300192,
+            -0.013134679989565051,
+            -0.011974548914818263,
+            -0.004028712301052159,
+            0.019628329175867303,
+            -0.028192034966932684,
+            -0.012638520793012082,
+            -0.011568652932013111,
+            -0.0025919910519445737,
+            0.012497908463405111,
+            0.015113186712897145,
+            0.014880607940212641,
+            -0.03059227859452334,
+            -0.021899716217105084,
+            -0.0008513506358489523,
+            0.011985931118464679,
+            -0.01321584595585435,
+            -0.011735288366213981,
+            0.010710683541435907,
+            0.01961722172525993,
+            -0.023571946726192585,
+            -0.03364281880125354,
+            0.0014460076743756465,
+            0.016563167828467187,
+            -0.02891589059260944,
+            0.04595166331401113,
+            0.034271553243501465,
+            -0.027557971577291153,
+            -0.007138821520412525,
+            -0.039508407807651844,
+            0.06814329636830853,
+            0.00022136123564675344,
+            -0.039718016705753735,
+            0.0503457594963753,
+            -0.003580581891576583,
+            0.013820796398521455,
+            0.02054050419088757,
+            0.035682661890720495,
+            -0.0013220739026852943,
+            -0.0052759394697328675,
+            0.04733018719162478,
+            -0.0187845855131196,
+            -0.03982699468608443,
+            -0.023905460465070062,
+            0.002631233133039351,
+            -0.03090995554989513,
+            -0.012617612051488022,
+            -0.013605787969473513,
+            0.039672783793764745,
+            0.008669582242097126,
+            0.009974277556736362,
+            0.005535636914843583,
+            0.014180668493093584,
+            0.012042659947850441,
+            0.011870115226306776,
+            0.00451259454348581,
+            -0.014023048592193627,
+            -0.007769033996392623,
+            -0.05715987527668358,
+            0.003374564518384728,
+            -0.028801158523506985,
+            -0.00756331369170698,
+            -0.02163054894055572,
+            0.005367181573233089,
+            0.02308705505078254,
+            -0.005438909398653553,
+            -0.00873620909426313,
+            -0.0036530652237719077,
+            0.013920407174296543,
+            -0.01539613379172909,
+            -0.03908211213269472,
+            -0.016810726548844992,
+            0.007460173854386443,
+            -0.007069189426742819,
+            -0.041197222498539215,
+            -0.003843233633335066,
+            0.004373367480223065,
+            0.016194121660459908,
+            -0.029165150066068787,
+            -0.00735139702658342,
+            0.017257895082337448,
+            0.014153313280536095,
+            -0.002376178042741926,
+            -0.023223700836427403,
+            -0.0319877537205969,
+            0.02088060774052704,
+            0.0012829290529515294,
+            0.03251550040316258,
+            -0.04128453069131847,
+            -0.023376114412063578,
+            0.04083022735076889,
+            0.005690491480259003,
+            -0.05810403800689653,
+            0.016582434816685452,
+            0.010402038964417569,
+            0.016795644054313566,
+            -0.00908590043098552,
+            0.05003656634510042,
+            -0.037216197954986735,
+            -0.013027960261454162,
+            -0.007090943677204207,
+            -0.05175371787508879,
+            -0.0020065660448063322,
+            -0.014908489552053589,
+            0.010627472758306387,
+            -0.03275473161158777,
+            -0.017334608434133866,
+            0.025321047439048318,
+            -0.0033515195870701753,
+            0.011526068468647207,
+            -0.026783879816504975,
+            -0.0012830935419413407,
+            -0.017506908439998053,
+            -0.02114853058440477,
+            0.004878884893669383,
+            -0.020936900837115968,
+            -0.0025587027781733642,
+            0.01557850246191957,
+            0.023652335794299834,
+            0.011489428313303988,
+            0.0019089137782552768,
+            0.02054907941022243,
+            0.014361232983780595,
+            -0.0056413109308506085,
+            -0.0010300050480036684,
+            -0.03413262505887213,
+            -0.018821160485010483,
+            0.03361989907065425,
+            0.013125889701529858,
+            -0.003742167496379789,
+            0.008813100942466033,
+            -0.0383265293638935,
+            -0.035316957263713485,
+            0.03020499323039515,
+            -0.05001623971243151,
+            -0.00855479667673597,
+            -0.005939674940109814,
+            -0.026451831943555135,
+            -0.008032873049527808,
+            0.0009431571685654594,
+            -0.014387605128858954,
+            0.016382559541755014,
+            -0.024909510232038502,
+            0.003715736505444385,
+            -0.006016712902409688,
+            -0.009110197692342108,
+            0.05507263160354303,
+            -0.022103031032598353,
+            -0.0475415595313825,
+            0.021958434235402655,
+            -0.016670809659931243,
+            0.0323519363801999,
+            0.03867494951405658,
+            0.011124508964363016,
+            -0.008002445714431074,
+            0.034041838454818577,
+            0.018400123235267234,
+            -0.01161754789471199,
+            -0.025901844576206774,
+            0.021800298132870245,
+            0.00943488425697727,
+            -0.05906691123008531,
+            -0.00782549274387971,
+            -0.019357446956797447,
+            0.008682376344965976,
+            0.02863077694455766,
+            0.0024925977373894447,
+            -0.002682170968419412,
+            -0.013590235283325244,
+            -0.021511149507762284,
+            -0.03255453916518719,
+            0.009039307015853905,
+            0.008305286476500875,
+            0.009312625027668458,
+            -0.0014144165766983945,
+            -0.007853869334376857,
+            0.009067065760942163,
+            0.022485201611182105,
+            0.03966189080765722,
+            -0.041627289519629344,
+            0.002707086456239785,
+            -0.022678178050044222,
+            -0.03653753142622053,
+            0.0036574646778835588,
+            0.029996160786576367,
+            0.02901163765559891,
+            -0.01971777193183787,
+            -0.0036589934841109884,
+            0.009181234938554578,
+            -0.025979483002227535,
+            -0.00773645103995767,
+            0.004280770859737866,
+            0.04116668301017644,
+            0.006201085381597264,
+            0.01402728258112097,
+            0.001060407317110527,
+            -0.00920801784000087,
+            0.02109880889040528,
+            0.025211022083982727,
+            -0.029866678264942617,
+            0.014439702646820226,
+            0.009379206296908577,
+            0.04156435812161122,
+            -0.037309655469525846,
+            -0.011819731348299495,
+            -0.02766498020409985,
+            -0.010454143230794476,
+            0.027911532296566614,
+            -0.04127890742215716,
+            -0.006948536183086887,
+            0.010207784113909938,
+            -0.0016584043356136647,
+            0.016437495428919636,
+            -0.014607861750686952,
+            -0.03222311919805017,
+            -0.000610723203118213,
+            -0.02867445198803213,
+            -0.024772816792038133,
+            -0.016510309753766612,
+            -0.018581201775667577,
+            -0.07450654873867177,
+            0.00026832248829154255,
+            -0.06724696990816072,
+            -0.02447002541264633,
+            0.04885569249258513,
+            -0.010195604207134053,
+            0.03332905935704749,
+            -0.02815272112271314,
+            -0.005296340304911926,
+            -0.05908835042526019,
+            0.0038476172058970774,
+            0.025753576863447866,
+            0.02386003857017594,
+            0.005206447371391696,
+            0.024799057871126032,
+            0.012060738268265606,
+            -0.04146503002980051,
+            -0.024949907556862643,
+            -0.014503475223770958,
+            -0.0016340466065744286,
+            -0.0034009167013795906,
+            -0.031298432995411005,
+            0.036241272149880994,
+            0.008902644391348814,
+            0.03550223922667804,
+            0.04463301409716232,
+            -0.010385984413759852,
+            -0.03305004479734889,
+            -0.006952739338820577,
+            -0.0175981588092108,
+            0.042668533202572856,
+            -0.017112184620086308,
+            0.005149173014526686,
+            -0.03219589107997635,
+            0.024181808150698054,
+            0.0014339790422945406,
+            0.01254721190611376,
+            0.011099458649404119,
+            0.0027626836767364253,
+            -0.026883640446090158,
+            0.0024160770766463607,
+            0.023874514847956222,
+            -0.022558433617669373,
+            0.00917667715473093,
+            -0.006754091166465801,
+            -0.033686935718776144,
+            -0.01078389638118685,
+            -0.013595327486167114,
+            0.024712781935748277,
+            0.013168464807451798,
+            0.005875864604862434,
+            -0.02417814420558463,
+            -0.0023978800459714144,
+            0.028650463366555304,
+            0.020176980768301288,
+            -0.02118184201861616,
+            0.019082340279671355,
+            0.019734925740100277,
+            -0.04209645071102001,
+            -0.04217310001316092,
+            0.006553913719098956,
+            -0.003735719227468833,
+            0.0061793909220459,
+            -0.028285889794908054,
+            0.008181788076484304,
+            0.004432488437404309,
+            -0.01743905248466261,
+            0.041221269621866216,
+            -0.014892354419172107,
+            0.005087102208335201,
+            -0.022198383716186407,
+            -0.011936388003605221,
+            -0.00027487042024290816,
+            0.044599215801158845,
+            0.008068975171827777,
+            0.003496950921841073,
+            0.02238779276447619,
+            0.0054839808262270315,
+            0.04816723616939378,
+            -0.02685221260466647,
+            -0.0190356946337995,
+            0.016464136459560113,
+            -0.0176092780782677,
+            -0.012513274095432353,
+            -0.010695421445334002,
+            -0.004784348908457072,
+            -0.022203041334932165,
+            0.010788155380415583,
+            -0.018353897272137543,
+            -0.02277431189907027,
+            -0.003945465850796918,
+            0.016024639792926797,
+            -0.006248368166794934,
+            -0.0038769803892794875,
+            0.0016292556540434277,
+            0.02412864830338344,
+            0.02892955453819876,
+            -0.02408022470555968,
+            0.019190282372105023,
+            0.018941584034503536,
+            -0.006709516380602965,
+            -0.020956377933747532,
+            -0.01964885206488053,
+            0.01434623424964373,
+            -0.01352968586212152,
+            -0.013872556892707697,
+            0.004052798542613835,
+            0.009201147374299997,
+            -0.004086067985859617,
+            -0.04185829685685201,
+            -0.014668698123513235,
+            0.003179403563664022,
+            -0.0006415014901450319,
+            0.019360184344240947,
+            -0.0150817538567364,
+            -0.029771999278536083,
+            0.014221776097046751,
+            0.022944884879589678,
+            -0.04493425220030206,
+            0.02117577719533236,
+            -0.0030648751497261983,
+            -0.00016251533643103604,
+            -3.817296018601414e-05,
+            0.008727767382077138,
+            0.026914779443266063,
+            0.03456565094655899,
+            0.01974806772696219,
+            0.00450800007152441,
+            -0.008952599700759673,
+            0.025184190461155245,
+            0.009846044083064704,
+            -0.025168735183134738,
+            0.010983353565382029,
+            0.016740721743326522,
+            0.0070838930012654296,
+            0.009173879407277282,
+            -0.006078288608766152,
+            -0.018269347893097643,
+            0.013181494188066667,
+            -0.01435693747022231,
+            0.01678907949532146,
+            0.05154463552784974,
+            -0.00039632596424461964,
+            -0.02461439550772063,
+            0.008418147037164186,
+            -0.006544319842335593,
+            -0.010065129286193278,
+            0.02214628550460128,
+            0.0029515574751370417,
+            -0.030862152033847717,
+            0.007509556025223583,
+            0.00675485304374519,
+            0.01894936231559203,
+            -0.02355525295639599,
+            -0.01387885858544803,
+            -0.00431398020087544,
+            0.0021208056468763523,
+            0.01580766504149989,
+            -0.004739477121073512,
+            -0.013204823447960706,
+            0.02994098229517376,
+            0.025496728234662373,
+            -0.00026124946738213036,
+            0.015060064980768175,
+            0.022262345898533903,
+            0.009968408664790368,
+            0.007588431141088797,
+            -0.0083063413609779,
+            -0.031336272171937095,
+            0.0018957858383561667,
+            0.05800811417728381,
+            -0.014937085296865802,
+            0.04313875432235027,
+            0.02018476425923236,
+            0.015657644644987107,
+            0.017880263053253607,
+            -0.01662268082870562,
+            -0.008740053682875323,
+            -0.0017630394204179638,
+            -0.008294417990024647,
+            -0.027613464270562948,
+            -0.005953209556351086,
+            -0.02379961210881074,
+            -0.013878676554701543,
+            0.015418315245979704,
+            -0.014764728065382074,
+            0.0027420640870858457,
+            0.010395538988699663,
+            -0.006590633833835581,
+            -0.00841537209927409,
+            -0.003943772700797381,
+            0.03376338017637296,
+            0.011980354721458463,
+            0.027761929822019422,
+            -0.00011128539412582973,
+            0.009342430952107655,
+            -0.018419526564396063,
+            -0.006839752805318664,
+            0.03271190887864607,
+            0.006011520427944665,
+            0.004652835471939905,
+            0.01485920920301398,
+            0.014305626381398871,
+            0.038015677219487023,
+            -0.005102005401565164,
+            -0.018204460895989052,
+            0.02065495133597456,
+            -0.0049699013356374775,
+            0.019149036693562712,
+            0.008825981577388056,
+            -0.002660055848523568,
+            -0.004292379561953203,
+            0.03090270710860331,
+            -0.020243440437217503,
+            0.018735727616321387,
+            -0.02238236089688497,
+            -0.0276584413929402,
+            0.023028408906176946,
+            0.008320436733620496,
+            0.04047641118026783,
+            0.011463851066078041,
+            -0.02247143742166039,
+            -0.034588725413285185,
+            0.009827713930789184,
+            0.00337071450636392,
+            -0.024446280199879062,
+            0.017542777160283324,
+            0.00780836483490212,
+            0.008453886667283516,
+            0.016624116261219637,
+            -0.004175682824655864,
+            -0.0042784362933758835,
+            -0.001546656717896957,
+            -0.008819739495525477,
+            0.01406790007187259,
+            -0.015487658289541583,
+            -0.021457182907140035,
+            0.024687169583245905,
+            -0.03915911962859764,
+            0.01771290139419629,
+            -0.006510046907173026,
+            -0.00997450880725535,
+            -0.02059560682587115,
+            1.5939836949198004e-05,
+            -0.007159493752862143,
+            -0.0004782363814453761,
+            -0.017503181957042913,
+            0.016643971600145645,
+            0.024056442566869802,
+            -0.013209211443909658,
+            -0.00445521990927108,
+            0.0020313626821877526,
+            -0.029235692455820056,
+            0.006662209097255912,
+            0.016993027053110635,
+            0.00529940612845993,
+            0.0026104743851393716,
+            0.0001707934714983891,
+            -0.031289446611163674,
+            -0.005782208620284939,
+            0.03053516889069683,
+            0.05590704747438521,
+            0.021628254147289437,
+            0.02403314981042378,
+            0.02908606663381421,
+            0.02717206823892633,
+            -0.003198844932040613,
+            0.0016217508103442348,
+            -0.01620660606431983,
+            0.027098323143010743,
+            -0.011664168290676943,
+            0.04322175021589083,
+            0.008652323184522541,
+            -0.049493020736445056,
+            0.004537958157696794,
+            0.00943239454147692,
+            0.034918603888916544,
+            0.016678716235690318,
+            0.0005887414054803805,
+            -0.005894338664998101,
+            0.03279165663336323,
+            -0.02735726709032382,
+            -0.011962070182991011,
+            -0.00036014407560325284,
+            -0.019079003730607717,
+            -0.015938295169860884,
+            0.010357611943935459,
+            0.04278924678845588,
+            0.0009957483598272451,
+            -0.0071422790468850215,
+            0.01571201177743999,
+            -0.009445307674383,
+            0.02286338492075306,
+            0.007336440384050498,
+            0.0042809497342459904,
+            0.025581467690095977,
+            -0.0025007871391868105,
+            0.02300539823997411,
+            -0.000628898771317254,
+            0.004582863909134464,
+            -0.015960578501883316,
+            -0.00930649396114688,
+            0.027838455754803444,
+            -0.011112268986721,
+            -0.029253746504396393,
+            -0.008734785561831333,
+            0.00472579547270089,
+            -0.0003805825712986138,
+            0.0021854358149324227,
+            -0.01183885533661493,
+            -0.010840951201591629,
+            0.019965691788904057,
+            -0.004922943301424112,
+            0.003800048754664761,
+            0.012664049429916252,
+            -0.0030124219674414897,
+            -0.01910844996750822,
+            0.01889983984736845,
+            -0.003858283778519141,
+            -0.024168821338992813,
+            0.020083955711766945,
+            0.01850497889545421,
+            0.045156567155119454,
+            -0.01586824055785609,
+            0.004612757419198795,
+            0.017497774574602505,
+            -0.002172088240279357,
+            -0.019617327081566827,
+            -0.014944795772093997,
+            -0.018212462639041492,
+            -0.015975283672064702,
+            0.012837956594691596,
+            -0.01243789877725244,
+            0.019500571353639314,
+            -0.0380735226388175,
+            0.012590900534826689,
+            0.004832168601136412,
+            0.007505385730121269,
+            0.016011983211212992,
+            0.025994560026860358,
+            0.002157436199771672,
+            -0.033465476168065784,
+            -0.01014599891204166,
+            0.007845373376916211,
+            0.010093432683262838,
+            -0.002849185178237036,
+            -0.018147004183434554,
+            0.003633790611809425,
+            -0.0266147420637404,
+            -0.018644422659309035,
+            0.022926737299761104,
+            -0.04798772015599113,
+            0.017908415301694906,
+            0.024859125158586466,
+            -0.04343699525707025,
+            -0.04886699607556352,
+            -0.011459188432595229,
+            -0.036774381316312464,
+            0.0014128005215813578,
+            -0.0005256067584181517,
+            0.011194992852264102,
+            -0.009342140079829378,
+            -0.018598671981710183,
+            0.01584789414301952,
+            -0.0036412641242200418,
+            -0.004225840239935549,
+            -0.00616696736018045,
+            -0.0025598631463178384,
+            -0.010188803157718543,
+            -0.03244694662411977,
+            -0.03290891136780909,
+            -0.00033742479296272127,
+            -0.009743927051414455,
+            -0.02334288359945272,
+            -0.03447062782872289,
+            -0.008728968107967028,
+            -0.037753659749578956,
+            -0.00515561961442376,
+            0.04070890150714597,
+            0.027517147608335957,
+            0.009449414978133334,
+            0.008086900089929912,
+            -0.013989034604490105,
+            0.05691047488419298,
+            -0.020425883381389086,
+            0.039466620642512155,
+            0.005301493569940605,
+            -0.0037222353899918703,
+            0.013075425214878277,
+            0.027710826422351564,
+            -0.006253665106995653,
+            -0.0017412928073238833,
+            -0.01928096630475438,
+            -0.012920476859996416,
+            0.009444408011467955,
+            0.015799884767184885,
+            -0.0025083061619197524,
+            0.001274457251739847,
+            0.00930533161700852,
+            -0.0006652478540264343,
+            0.017258038491335247,
+            0.03996268817486439,
+            0.003833844612473671,
+            -0.00826404141706295,
+            -0.018822002570963614,
+            -0.003188844900317347,
+            -0.011807006674248829,
+            0.010482058734260632,
+            -0.015410635475145028,
+            -0.011540803927522013,
+            -0.01249428777269578,
+            -0.017081856786754605,
+            -0.002183034976477535,
+            0.003114834030705039,
+            0.0027475411722881747,
+            0.02261627635192778,
+            -0.015872027027080824,
+            -0.022487581027490516,
+            -0.0035823117885461506,
+            -0.02398778449664045,
+            8.13716819067491e-05,
+            -0.01050060782407259,
+            -0.015616181283381218,
+            0.013676480044963827,
+            -0.021513173575512936,
+            -0.01091399089076612,
+            -0.008699176177861256,
+            -0.0006986953850759474,
+            0.005853723710671249,
+            -0.00993355748621621,
+            -0.00010241843582159751,
+            0.01150872694850241,
+            0.005559552266827916,
+            -0.012336449703762907,
+            0.0039431718621504,
+            -0.012740550272925562,
+            -0.004432059174951158,
+            -0.03592309122711106,
+            -0.019665686718735042,
+            -0.01846795637714379,
+            0.005642094448288269,
+            -0.005108964028543505,
+            -0.020586678054151854,
+            0.00668364061048941,
+            0.0034775127050453774,
+            0.004713169526661792,
+            -0.010723667875209928,
+            0.013944735241748791,
+            0.010882420168586074,
+            -0.003518245916227596,
+            0.013827334494230662,
+            -0.001330399936559009,
+            -0.020286697504699764,
+            -0.011139922402071095,
+            0.03538258948179278,
+            -0.024961826464903997,
+            -0.0018019328351763667,
+            -0.0026423695690168066,
+            0.00597003516870726,
+            -0.03071835946501977,
+            -0.013634212232744325,
+            0.02164400349828422,
+            -0.005846552894831441,
+            -0.026367014426434226,
+            0.03399116480234033,
+            0.03473258231629753,
+            0.010041605592929843,
+            -0.013423384340589794,
+            0.04730054254696893,
+            -0.012968978954938308,
+            -0.018274800841400665,
+            0.026826644936918647,
+            0.0278389611094493,
+            0.02525164095926168,
+            -0.029782550447890466,
+            -0.003010713086487275,
+            -0.02393629084357885,
+            0.017993366287762866,
+            -0.0263476663180971,
+            0.01864011499179424,
+            -0.02332263152537849,
+            0.016014908881756034,
+            0.01868464825773604,
+            -0.003104041044182249,
+            0.026313491086684425,
+            -0.0052945248919367815,
+            0.015253731232325074,
+            0.0012371308759955573,
+            -0.02119765517013431,
+            0.007795466385020642,
+            -0.006469500272298062,
+            0.025762169721880456,
+            0.00833992768202466,
+            0.01801901639978976,
+            0.022019940825351116,
+            -0.0409522298213741,
+            0.02003698451081499,
+            -0.001367916864679543,
+            0.01782984895971748,
+            0.0038516134262842373,
+            -0.007885432294782304,
+            0.014221728513400065,
+            0.010991352188016265,
+            -0.012851099261924757,
+            0.003840539873231741,
+            0.0008923482291826014,
+            -0.028928104542716937,
+            0.007066004768771198,
+            -0.026914574065060926,
+            0.025866907400807087,
+            -0.005526774784152086,
+            -0.00998122210956923,
+            -0.020744532673097317,
+            -0.0071425207262770865,
+            0.003754468896273156,
+            -0.012747874997429746,
+            -0.011385805601999295,
+            -0.024700457669046855,
+            -0.02401291382882952,
+            0.029591582802488962,
+            -0.014054332096344003,
+            -0.04015978298687865,
+            0.001234440863385445,
+            -0.023301255402129767,
+            0.01193125007773239,
+            -0.005793124129267758,
+            -0.024440100649464773,
+            -0.042563399082257154,
+            -0.017548888269389457,
+            -0.0009394887961485922,
+            0.022739763644266633,
+            -0.00114156949394707,
+            0.041791395064755515,
+            0.0008816696205437911,
+            0.022207132857533028,
+            -0.015547580543606755,
+            0.005319731264840819,
+            -0.02071229859314312,
+            -0.041495579742912285,
+            0.03660432378985971,
+            -0.014637031447181937,
+            0.025854844497435178,
+            0.03324325296704186,
+            -0.013009347371118843,
+            0.001354504885302625,
+            0.02652570709474851,
+            -0.014618100923440427,
+            -0.010782232747863238,
+            -0.0034936699515378212,
+            0.003461670978569965,
+            -0.006468116285021298,
+            -0.027793221545159466,
+            -0.007752458229193391,
+            0.04193706715422074,
+            0.017894069367616637,
+            0.021928994684423043,
+            0.022606401934291136,
+            -0.017972404868085358,
+            -0.026402048654300563,
+            0.003656646576244765,
+            -0.013129722990153284,
+            -0.018090360146368677,
+            -0.01624598856366761,
+            0.012397150789705746,
+            0.011741002980073786,
+            -0.004052422390675261,
+            -0.01836750423615859,
+            -0.020929335029248472,
+            -0.016327412417122416,
+            -0.0042615345226698015,
+            -0.008458117095535,
+            -0.005002582649405199,
+            -0.012158173174049797,
+            -0.020409616620022143,
+            0.03706368540614453,
+            0.0285565642978937,
+            -0.0041732933884846566,
+            -0.00607154864704767,
+            -0.011142150176334957,
+            0.008834431806990914,
+            0.006021232046087652,
+            -0.005028870471297621,
+            -2.3396043292139426e-05,
+            0.005112033372713154,
+            0.022822312435625314,
+            0.007271235299348001,
+            0.0035791789459533856,
+            -0.06340305811365254,
+            -0.005392000517722054,
+            0.022560201096078862,
+            0.012581401789917604,
+            -0.0032248219865223496,
+            0.025180718735801837,
+            0.0006451724964970927,
+            -0.009714514747463593,
+            -0.018389422219956937,
+            0.024728862150150004,
+            0.008760329201960964,
+            0.020988700992999768,
+            -0.014974763313686516,
+            0.003845297486923296,
+            0.0006445983847138896,
+            0.00750394530065325,
+            -0.0019792992847026793,
+            -0.008578254004439607,
+            0.025776677078006516,
+            0.016593438298233645,
+            -0.005695578040595484,
+            0.01604018433477423,
+            -0.0017497260488899657,
+            -0.021460543443428924,
+            -0.010265213095609554,
+            0.00042017209251926397,
+            -0.010570864288876387,
+            -0.018015790888295944,
+            0.017569414447505878,
+            -0.002601036282535863,
+            0.0072399377790164435,
+            0.030281707151249355,
+            0.0423183067569767,
+            0.009566493398534713,
+            -0.005928603293907002,
+            -0.0037936911251953654,
+            0.015477873304924939,
+            0.01513127995244932,
+            0.010741256034652754,
+            0.03015858543221727,
+            -0.029247492875021843,
+            -0.006690306960497428,
+            0.0046653727410352576,
+            0.004610405633429778,
+            -0.011692179099382171,
+            0.030055897497741527,
+            0.04569730592994177,
+            0.006736813244286893,
+            -0.004799978513117371,
+            0.002121067028052434,
+            0.011299889107780231,
+            0.007768137218140627,
+            -0.018927080714961783,
+            -0.0012222962555201746,
+            -0.006009911026251445,
+            -0.014001766533604134,
+            0.026941585318261085,
+            -0.003960837039351648,
+            -0.011848232223811548,
+            0.02903357274500332,
+            0.1105267435019781,
+            -0.006011029338781496,
+            -0.0053646998281195206,
+            -0.03242321704621517,
+            -0.015194102811332289,
+            -0.008860590919147765,
+            0.0056988205877963076,
+            0.01635957585050017,
+            0.020327661913559888,
+            -0.0008943408274661007,
+            0.016820860249955754,
+            0.02822175614376368,
+            -0.013854827839129885,
+            0.007906244895327178,
+            -0.030891381690403466,
+            0.015702117924362928,
+            -0.005383974926093977,
+            -0.012032955400313954,
+            -0.0026169511828034054,
+            -0.008618307035330928,
+            -0.0037356680273096105,
+            0.0009666430093455632,
+            0.011296307527684261,
+            0.004427516814276546,
+            -0.014484198313369681,
+            0.002829529156309784,
+            -0.008595300985863424,
+            -0.005122996529848575,
+            -0.02393571073523279,
+            -0.024119245543788293,
+            0.005223147047205755,
+            0.0019389305548629142,
+            6.344308292376579e-05,
+            0.006848965054539454,
+            -0.003810712095343454,
+            0.030149081317333418,
+            -0.01700322008272963,
+            0.0012288912810468446,
+            0.01193226302726573,
+            0.012332749933413344,
+            0.01688602002694633,
+            0.0028600984823761366,
+            -1.9882692552530084e-06,
+            0.021172977047420227,
+            -0.010514036509402561,
+            0.01460334896316005,
+            0.013091417303145243,
+            -0.04103572381458182,
+            0.0089746401463863,
+            -0.0027841289484449873,
+            -0.024252447218794344,
+            0.004446472209246425,
+            -0.018987867343500102,
+            -0.00881974688349601,
+            0.0005365625696166774,
+            0.012054719659248512,
+            0.01960300421041832,
+            -0.008553943699846696,
+            0.02111201386480852,
+            -0.0003501657740808554,
+            0.004177088753111459,
+            -0.012879944670340026,
+            0.0065222126975051685,
+            -0.007563354540402343,
+            0.0008168128607271199,
+            -0.018213241937597355,
+            0.012526561612175514,
+            -0.0008285725179894544,
+            -0.015737003159436623,
+            -0.011337688275309921,
+            0.004737346137976519,
+            -0.0027993545140114705,
+            0.0030843163114700313,
+            0.010153740340626154,
+            -0.008815302484351509,
+            0.004919275910399053,
+            -0.0017541042538165297,
+            0.002562816687714991,
+            0.014285249186421434,
+            -0.029816964593980453,
+            0.005357227738099325,
+            0.009154842495811638,
+            -0.00048782303639333943,
+            -0.013074976152495576,
+            -0.004054505559072866,
+            0.0028322102449485015,
+            -0.016877328079971877,
+            -0.012128199639779175,
+            0.023421330164419888,
+            -0.012565145848820333,
+            0.008609070188894315,
+            -0.010936598722317274,
+            -0.018401801734378692,
+            -0.01066119413133089,
+            -0.005254229780560808,
+            0.01234969508270852,
+            -0.0038427767204741877,
+            -0.006947594635952202,
+            -0.01756894543880097,
+            -0.007890897373987378,
+            -0.047246698568222466,
+            0.009617876521097624,
+            0.027157555263862723,
+            -0.011355758309667893,
+            -0.04050311381480088,
+            -0.01017774338301804,
+            0.02437257208967103,
+            -0.004056329295095018,
+            -0.008345450168215274,
+            -0.001599171409135998,
+            0.018358386463243014,
+            0.0027207445181251406,
+            -0.007292967749349572,
+            -0.011267861428599221,
+            0.01657507035043879,
+            0.015105061280816214,
+            0.021419854579631276,
+            -0.02088952706825817,
+            0.006295984081511286,
+            0.020973020776936996,
+            -0.03073246543555915,
+            0.0023720435876790876,
+            0.03363966216709148,
+            0.010412588613998134,
+            0.013023311559810034,
+            -0.0010217662933572685,
+            -0.0075567381078977215,
+            -0.008248818794295818,
+            0.011848200323549865,
+            0.005066463524567732,
+            -0.023174489297821385,
+            0.021823825355858556,
+            0.020368868756448753,
+            0.0026042856858509524,
+            0.003457067317891926,
+            0.0005539745115410357,
+            0.01062670844108589,
+            0.009948889803890108,
+            0.05100741556350918,
+            -0.006493714124814164,
+            -0.014021751055930108,
+            0.015063988102064609,
+            0.02488059237697814,
+            -0.0023499941119596228,
+            0.01087282522970853,
+            -0.02612218935931224,
+            -0.021423880914461062,
+            0.023389294741183817,
+            0.008967763132667892,
+            0.009637806238448432,
+            -0.010815706635624337,
+            -0.011390999462441815,
+            0.031750675158698435,
+            -0.01307829462211914,
+            0.00759749911213318,
+            -0.02169895957209634,
+            0.01387879409130081,
+            -0.008606362268863441,
+            0.032229658009209815,
+            -0.0009066257436946125,
+            0.011062436424260079,
+            0.028010486866468023,
+            -0.007835805879241447,
+            -0.019657486763625547,
+            0.003967480939987674,
+            -0.015577952617624695,
+            0.02188429826317828,
+            -0.020502758155233298,
+            -0.0002362500702658591,
+            0.0040935406015014265,
+            -0.010056462958039452,
+            -0.0057038083589231615,
+            0.011475170331848666,
+            0.0015345076181482267,
+            0.006359262977587827,
+            -0.020293014401106,
+            0.01369706440475074,
+            -0.003989482215141903,
+            0.007887824197823247,
+            -0.015925992045349828,
+            0.014370223626942766,
+            -0.02309139937510304,
+            -0.021808736033141787,
+            0.008142621179624918,
+            0.0108996933426044,
+            -0.007589050511458233,
+            -0.025971539510030145,
+            0.02250992080022228,
+            -0.024980833805349545,
+            0.024979008145354243,
+            -0.006622339940663538,
+            0.002566378488088469,
+            0.004913874740363343,
+            -0.008102103508732738,
+            -0.0028098905427027783,
+            0.0057865587191235565,
+            -0.0073509382400023995,
+            -0.014123756810537343,
+            -0.019453380788786282,
+            -0.04083629135586275,
+            -0.03177927047231389,
+            0.022927581786088044,
+            0.011557232599801421,
+            0.0017073832658476193,
+            0.02339552660545296,
+            -0.02247471449131386,
+            -0.017890866287500053,
+            -0.009775068417716798,
+            0.025184899522229628,
+            -0.04408541669037361,
+            0.012104255392615549,
+            -0.014669458760022245,
+            -0.0245258242481677,
+            0.006274252098230249,
+            -0.004161960147481513,
+            0.009299171724026455,
+            -0.010779421730862708,
+            -0.04713627671791054,
+            -0.005192684784377356,
+            0.02330975631183273,
+            -0.0036802876364168935,
+            -0.006197548344287773,
+            0.03048284650641829,
+            -0.0019503797069087845,
+            -0.005038254331399784,
+            -0.005524875459245342,
+            0.027783596017224226,
+            -0.024950083604372984,
+            0.010988398531346805,
+            0.008304937912856357,
+            -0.008485065895193593,
+            0.006572665697403556,
+            0.0011518846966102158,
+            0.011927348472282199,
+            0.015632588644277183,
+            -0.015559413716826264,
+            0.02262990861928957,
+            0.011446850286896198,
+            0.006506075777765303,
+            0.008015225053785824,
+            0.00010054430757843335,
+            0.0022975742216194894,
+            -0.024948925449668525,
+            0.026047924967851216,
+            0.01888211557921096,
+            0.0001632181017876392,
+            0.012553094476616919,
+            -0.015042621225242777,
+            -0.02782009986290425,
+            0.02606646297992026,
+            -0.016847653820092335,
+            -0.00045272379465939274,
+            0.020053091486344633,
+            -0.016210036358455773,
+            -0.007930550661303182,
+            -0.01123514849773852,
+            -0.0162675150307978,
+            0.01803076559921804,
+            -0.012160217543572069,
+            -0.02216594536057653,
+            0.002594675636753233,
+            0.016443123828746582,
+            0.02134480656850119,
+            0.026694575953313482,
+            -0.011470930697351533,
+            0.017768927048191052,
+            0.0008653408391607816,
+            0.001865765599795092,
+            0.025745954171649543,
+            0.01702677011968492,
+            -0.008474889109168202,
+            0.025347176686722313,
+            -0.0016686223698600755,
+            -0.002421921060670558,
+            -0.012265504084322075,
+            -0.0165214436458303,
+            -0.007545471138114208,
+            -0.015333235556146196,
+            -0.0051830106590765685,
+            0.000762746782664525,
+            -0.0018118022696726346,
+            0.10876576838038368,
+            0.0034948044540582626,
+            -0.011608460689239234,
+            -0.005870263500417206,
+            0.007510774308250375,
+            0.0023525201199670765,
+            -0.006846154764389575,
+            0.016082721103476375,
+            -0.018616626204086575,
+            -0.023593744079394,
+            0.004743481974389998,
+            -0.0009262286319469741,
+            0.00870122878944883,
+            0.02029523251993269,
+            -0.008608837818169835,
+            -0.0200567711498164,
+            0.007089477722191928,
+            0.005535149644041221,
+            0.0023640746516071774,
+            0.005256431824732927,
+            -0.0006118671321824933,
+            0.008209920240361308,
+            0.02448422122961942,
+            -0.015622892878860465,
+            -0.029770836607612903,
+            0.010535371601298532,
+            -0.00015994156470228762,
+            -0.007193425789067776,
+            -0.011046316463797128,
+            0.014728841492009627,
+            -0.0011488451990980268,
+            -0.02751404133172324,
+            0.017663132712536528,
+            -0.019055335178900415,
+            0.014643246200933506,
+            0.0025348345163347464,
+            -0.00016387173410118788,
+            0.022565953069005714,
+            0.013101158251834834,
+            -0.04272085076882881,
+            -0.0034442020309914823,
+            0.01041205537746844,
+            0.011059422927946552,
+            0.0116446016862942,
+            0.0012992535381387887,
+            -0.018670801241345514,
+            -0.011788984490283379,
+            -0.009109849103208542,
+            -0.021268488414296117,
+            -0.012523033785938503,
+            -0.011393853408628214,
+            0.010743646386439633,
+            0.0012964030814325153,
+            0.023809566099779113,
+            -0.003030535542277875,
+            -0.006526351867141901,
+            -0.000430699447948508,
+            0.00880561288237257,
+            0.009694646009259077,
+            -0.02079011138439714,
+            0.008294338719512953,
+            -0.01823922789220214,
+            0.011819769685519291,
+            0.006416044253693268,
+            -0.022799568832306067,
+            -0.009911619276012527,
+            0.007152786695893951,
+            0.008393677195358034,
+            -0.013569953300860946,
+            -0.006503988731872938,
+            0.02299498754558268,
+            0.018870619152710703,
+            0.012233964644738905,
+            0.024994616203196145,
+            0.03838969722235787,
+            0.005744255912479976,
+            0.027682123067526515,
+            -0.016791471092091604,
+            0.008459745383680297,
+            -0.022493868174622695,
+            0.012902262638928338,
+            -0.00460399519266689,
+            -0.0050431343566427886,
+            -0.018188192498411616,
+            0.033765105245412236,
+            -0.009159751849801773,
+            -0.0027150644587327763,
+            0.034778224828120714,
+            -0.0066502021793471075,
+            0.004525230911930458,
+            0.005096748531775903,
+            -0.016067049494073893,
+            0.033274672789154185,
+            0.01686831040872425,
+            0.016353966592947262,
+            -0.021631941883929817,
+            0.011748317746828848,
+            -0.005675701689448899,
+            0.0001273442591116645,
+            -0.020738819037390263,
+            -0.0037528323797385733,
+            0.0027725541343763104,
+            0.0059860509282538546,
+            -0.016556134518370437,
+            0.02033623476757896,
+            0.00019647023567641008,
+            0.0052895744489259875,
+            0.03253462247073863,
+            0.01043311189804924,
+            0.019182938878855655,
+            0.010051051068017807,
+            0.019523739618278902,
+            0.0281939391326836,
+            0.025685787482623124,
+            0.0029932688462254147,
+            -0.018668945726139946,
+            -0.0009326901784812825,
+            -0.007880352863625262,
+            0.010492063055849727,
+            0.022849902214818293,
+            -0.022619228461640748,
+            -0.02772644874135741,
+            0.018566981184526675,
+            -0.0022608185345048414,
+            0.020031103834122595,
+            0.007544464625538984,
+            -0.020935181884939796,
+            0.003407355515103286,
+            -0.023178230166337178,
+            -0.0005418155871620593,
+            -0.031039187720144967,
+            -0.028591174077243604,
+            -0.0007659241904525392,
+            0.011945344534674635,
+            -0.01688153017778311,
+            0.03357788163306365,
+            0.017667778671101954,
+            -0.014289109343082812,
+            0.01604684528817524,
+            -0.011054587325539086,
+            -0.0036959091138078683,
+            -0.005819853870004682,
+            -0.00203755010992417,
+            -0.02157944474979592,
+            0.005138614763345542,
+            -0.0033509138150859593,
+            -0.0024871060745482883,
+            -0.016936575573598154,
+            -0.021901830366872595,
+            0.016375173843894617,
+            0.016742770709641697,
+            -0.006606252542838758,
+            0.039771905404651496,
+            -0.004238071340064403,
+            0.00018364315349545657,
+            0.011690521707403059,
+            0.0010018392078563795,
+            -0.013651921068215563,
+            -0.010829994512513501,
+            0.012362647379788833,
+            0.011321403447882543,
+            -0.018942043512252663,
+            0.005763773608775982,
+            -0.009131565038297889,
+            0.002089216671850269,
+            0.005438887787329238,
+            -0.008183835275629478,
+            0.006690280329947951,
+            -0.01673229763262922,
+            0.015833397876364723,
+            0.00295474085300386,
+            0.003186554859191664,
+            -0.011797859525926772,
+            -0.020991927615680524,
+            0.01576413532776261,
+            0.0032403514518527287,
+            0.018819824905201635,
+            -0.026755218746060662,
+            -0.00012946467986739473,
+            -0.009552681500884924,
+            -0.0028941357661144826,
+            -0.012533800273548772,
+            0.013975640628890588,
+            0.016864739319953755,
+            0.0063941926578529,
+            0.01781016436903586,
+            -0.0026268224264609724,
+            0.015368284773773678,
+            -0.018122416427266218,
+            0.008686009180697955,
+            -0.006801074032721299,
+            -0.018537521789181048,
+            0.0013237338264105415,
+            0.027863741654435434,
+            -0.004670012662973029,
+            0.011066853115455966,
+            0.011704884649534624,
+            0.018954056680143452,
+            -0.0012626215734593707,
+            -0.0030850559460101866,
+            0.015610404665171367,
+            -0.005838226405658572,
+            -0.006861883833171456,
+            0.020872966233916948,
+            0.032951653763679145,
+            -0.006716215648910268,
+            -0.0021292599951407863,
+            0.010886596327481677,
+            -0.004901684487088035,
+            -0.008644584974417106,
+            -0.009587037829603014,
+            -0.03158137900717491,
+            0.024369089446706894,
+            0.012092412439477474,
+            -0.009601042951586109,
+            0.022690862146548786,
+            0.020752792552278052,
+            0.011412743968296708,
+            -0.010471860708195726,
+            0.02291536038589634,
+            -0.01268035142112569,
+            0.024168869865226136,
+            0.032482831775271157,
+            0.004400973556358811,
+            0.0022312906286559167,
+            0.0021175489661843985,
+            0.01691348460546639,
+            0.006050357346231155,
+            -0.007068133214675996,
+            0.011656003476409986,
+            -0.018487923180009552,
+            0.01598530891381001,
+            -0.004432366596808033,
+            0.02808371007118948,
+            -0.01337145216499331,
+            -0.010473700033368562,
+            -0.00537325376270318,
+            0.00010726431372328729,
+            0.006445701087575359,
+            -0.003324002570256215,
+            0.005255443653813165,
+            0.0007072577303252132,
+            -0.017825649146944237,
+            0.003050128130393781,
+            0.004825967170196209,
+            0.009018799597337101,
+            0.004103653497834134,
+            0.007249957401824204,
+            -0.012360355929483,
+            0.01441885664789707,
+            0.004401820363035808,
+            -0.0026190938717927086,
+            -0.0007163225760398784,
+            0.011124538295057815,
+            -0.013232822471919533,
+            -0.004895624967029031,
+            0.005281228035321008,
+            0.0059275753353475414,
+            -0.007507924998741734,
+            -0.004484140989405051,
+            0.011908076549392969,
+            -0.0037917291321648483,
+            0.006351543504726163,
+            -0.004409464890606014,
+            -0.0037760571637986827,
+            0.005303661313434895,
+            -0.00974511544836468,
+            -0.004204167898342997,
+            -0.0007093125445559502,
+            -0.006869250680233214,
+            0.00786202375969793,
+            0.005971033227117194,
+            0.010606085272785863,
+            0.006969437776542303,
+            0.004293129058563288,
+            0.004114452739123398,
+            0.004071017374828283,
+            -0.002445681623750424,
+            -0.00592480769363354,
+            -0.004617501328052899,
+            -0.012067458701076778,
+            -0.008335239153829813,
+            0.009025028998949398,
+            0.008653779421856954,
+            -0.019363018792788574,
+            0.02048011578221327,
+            -0.001188320960771659,
+            0.00772165044232104,
+            -0.01602086976290928,
+            -0.000907553574811279,
+            0.005464880629734941,
+            0.008267702703609528,
+            0.016136762227127184,
+            -0.03654209526492064,
+            0.005179791231187316,
+            -0.0037321883493524766,
+            0.016026013709955444,
+            -0.008215703411373182,
+            -0.006979816286354862,
+            0.02564060526656963,
+            -0.01450763388985563,
+            -0.0009847758228920171,
+            -0.016142304341196283,
+            -0.01407778890808498,
+            0.010815725252959506,
+            0.02050940280233426,
+            0.020684808835848383,
+            -0.0004965802897137407,
+            -0.0005781445948759235,
+            0.00036179841583004193,
+            0.022775804864342608,
+            0.015545681427566235,
+            0.018697154644607316,
+            -0.004543500782944068,
+            0.007028436664826706,
+            -0.023057215501463973,
+            0.009912581556496658,
+            0.0011795417129232596,
+            -0.008215027850908477,
+            -0.008554137076267468,
+            -0.008703378564520125,
+            -0.010448538943182258,
+            -0.011773811327190638,
+            0.004106616839960258,
+            0.013895013678116032,
+            0.004190549543250594,
+            0.0031571190600981074,
+            -0.0225385788570314,
+            -0.006243193914542516,
+            -0.011229641377928767,
+            -0.01674214043453308,
+            -0.007367389908398082,
+            -0.02386334969150212,
+            -0.028021339684039734,
+            0.0002794207370355175,
+            -0.02411861258809571,
+            -0.015958336406924612,
+            0.005268815864773021,
+            -0.01546610225139129,
+            0.00898600137695074,
+            -0.008396481690803911,
+            -0.011873486990732417,
+            0.0074743033679134105,
+            0.015336444788692,
+            -0.017676977079590463,
+            -0.010165317702640217,
+            -0.01690524990506033,
+            -0.013460451917691695,
+            0.010918550283871637,
+            0.00595960977280955,
+            0.009250157926438213,
+            -0.012447979337850305,
+            0.0062932435120863365,
+            -0.012673905639312696,
+            0.015073013552721701,
+            -0.03769166067890086,
+            -0.029992249463405965,
+            -0.00014990754540999537,
+            0.007288036630677436,
+            -0.0020735742042505426,
+            -0.018695120921334806,
+            0.0003860116505754868,
+            -0.003608995261360232,
+            -0.014900191335144973,
+            -0.013866216738266176,
+            0.025251475305524053,
+            -0.006788483107616133,
+            0.01223227424986397,
+            0.0034937277040015626,
+            0.03438333560343413,
+            0.008803852645562129,
+            0.014666206816302367,
+            -0.005709421955160623,
+            0.003742433643393657,
+            -0.0033706258426092744,
+            0.006906464031352315,
+            0.0076578888497945185,
+            0.009276592191710524,
+            0.008025321004305927,
+            0.004576470294889512,
+            0.002887346543823031,
+            -0.001925468214331082,
+            0.02125241545487307,
+            0.025671028960818562,
+            -0.00037467906648160044,
+            0.008981063192139038,
+            0.0009387900084090577,
+            0.0002735789978680719,
+            -0.008374290340790827,
+            0.006547013284767913,
+            -0.008274637814401893,
+            0.004041066530254948,
+            0.012897549078838394,
+            -0.009150863940242148,
+            -0.010211568689323908,
+            0.016575088843292566,
+            -0.012380099999574783,
+            -0.03420455578618604,
+            0.026550654680719987,
+            0.005324512466403649,
+            -0.0035670106106754245,
+            -0.010657928468872608,
+            0.0037176027853158167,
+            -0.01103891334887315,
+            0.008096915671982816,
+            0.022587659771132487,
+            -0.0013527917944322358,
+            -0.0018085171207746743,
+            0.0038503794877980586,
+            -0.011870484733648887,
+            -0.007881489342351292,
+            0.006462772034381968,
+            0.0315696514747912,
+            0.01326417653779577,
+            -0.020400369539140532,
+            -0.015223808971473513,
+            0.020406862349006738,
+            -0.023142176712111575,
+            -0.006254959806032159,
+            0.009039506093607052,
+            0.0009087520492922078,
+            -0.022766046838415165,
+            0.0009662037093312435,
+            0.004701966191595815,
+            0.005946458676746661,
+            -0.006358986495019584,
+            0.03237532341275415,
+            0.001075992257265679,
+            0.015892020271402903,
+            -0.017189453434945788,
+            0.0017514632185789299,
+            -0.014189278671121677,
+            -0.014265447388465805,
+            -0.029809460132405256,
+            0.0005557491671071693,
+            -0.0030324198417091704,
+            0.011038984948790744,
+            0.024808067315720335,
+            0.01145103533194245,
+            0.014432246996007246,
+            0.010552852947578788,
+            0.01513689173494211,
+            -0.013653722180565311,
+            -0.029009757850432732,
+            -0.005171522291984226,
+            -0.018001048743500243,
+            -0.02297482628212251,
+            0.003962613844063587,
+            0.013645125285548414,
+            0.006278662690346082,
+            0.023786381386575035,
+            0.0021738516691483163,
+            0.0009580591750940011,
+            -2.690805459518147e-05,
+            0.008894091070322921,
+            -0.004952203441302471,
+            -0.006553986197076102,
+            0.003570189040872711,
+            -0.000652714137223501,
+            0.021228360881461853,
+            -0.005758946428212041,
+            7.897153237591187e-05,
+            0.014242175492137011,
+            -0.005622256348934008,
+            -0.0035009192529791146,
+            -0.017339926143652377,
+            0.006248842389703505,
+            -0.01622011586976866,
+            0.025783086900871728,
+            0.018450783573486916,
+            -0.009109295293546267,
+            -0.013469521573468915,
+            -0.011259555201313116,
+            0.02773097452736661,
+            0.004631282554715908,
+            0.010787055220617197,
+            0.018228322680582464,
+            0.01461254259249465,
+            0.01784830193964943,
+            0.004664725683522323,
+            -0.0034339781146649384,
+            -0.0007692550982545399,
+            0.021176646860433974,
+            -0.014700178590249292,
+            -0.002701208084390198,
+            -0.008842788249773623,
+            0.0021799005046816444,
+            -0.008431299000066363,
+            0.006086447848362669,
+            -0.010793016137040172,
+            -0.015251381840165954,
+            0.0189503457652375,
+            -0.02660767353735015,
+            0.016423753600066336,
+            -0.00593016170680532,
+            -0.00482891500342298,
+            -0.027459214027897378,
+            0.010728039528784636,
+            0.0027866316191608297,
+            0.0022519974195000687,
+            0.010770472550461752,
+            -0.021427634123401408,
+            -0.01867717541189363,
+            0.008532444554010673,
+            -0.011457117350255678,
+            -0.007152301670244725,
+            -0.00348619742388509,
+            0.0154715607296319,
+            -0.00017898955187849443,
+            -0.009925365923862246,
+            0.02085093731060963,
+            -0.007175309371250024,
+            0.01223979370120031,
+            0.005256127464980655,
+            -0.008756077893734508,
+            -0.0028438320920811362,
+            0.021022708898898203,
+            0.027557382862284808,
+            -0.008342206702778956,
+            0.014526066580760486,
+            -0.011883797998224506,
+            -0.01088335352319397,
+            -0.03479444009382274,
+            0.030104838505937447,
+            -0.013305732257226255,
+            0.026773747582411633,
+            -0.0006968265979630097,
+            -0.017037729111584094,
+            -0.016206207453228583,
+            0.01625225876515654,
+            -0.0024290584023212254,
+            -0.009366287391027166,
+            0.01465885301870365,
+            -0.02428628495473643,
+            0.0058336571592899735,
+            -0.0006627935593670669,
+            -0.011531762838191602,
+            0.013275508677482622,
+            0.0019223252930710324,
+            0.010738622785461513,
+            -0.02030405805119001,
+            0.0009661595529028194,
+            0.009263716373456188,
+            -0.016791855691574455,
+            0.006057830565306032,
+            -0.03730952786847912,
+            0.0029341721588188943,
+            -0.014833895684348656,
+            0.005725615593914644,
+            0.001498755253963699,
+            -0.01820708352937343,
+            -0.03652692232407375,
+            0.00043665701245599497,
+            0.01913883585802903,
+            -0.007975988799790905,
+            0.020538595326175857,
+            0.008243569241768276,
+            0.008421404577931517,
+            0.002314738067443065,
+            0.0016702403734608475,
+            0.008279538988821465,
+            0.03679456861213399,
+            -0.01775927634148114,
+            -0.004836606118958108,
+            -0.0012151329859941935,
+            0.010644195708319258,
+            -0.02028470571116305,
+            -0.009287515726906219,
+            -0.0036506912311261536,
+            -0.013134467570201307,
+            0.030615862647585222,
+            -0.020265555441523482,
+            0.025634449009756734,
+            -0.009300770899023007,
+            0.009274072905190077,
+            -0.02097685009125915,
+            -0.009797008690754956,
+            -0.010661540100131183,
+            0.0019532291898233895,
+            0.013465005282849381,
+            -0.009151430285942593,
+            0.0071467934925125145,
+            -0.006655608635025676,
+            0.01020327561163374,
+            -0.001472597695707201,
+            -0.02443508800166349,
+            -0.0024998811551575563,
+            -0.01107321258077183,
+            0.02456164421154801,
+            -0.0019619696941303388,
+            0.0005043179699158694,
+            0.010951580163871948,
+            -0.00853593956432799,
+            0.023927336608555987,
+            -0.008855702280766362,
+            0.0010093679746665581,
+            -0.004863269000020324,
+            -0.004005691415117686,
+            0.001698212044518236,
+            0.01822024022650613,
+            -0.005707404715849356,
+            0.03722587302228139,
+            -0.01708925706528118,
+            -0.0065500772743641925,
+            -0.00015538686795361824,
+            0.010852925330001283,
+            0.010035359503147549,
+            0.008754719437760327,
+            0.004067820167375348,
+            -0.014861417626433965,
+            0.009346669946626484,
+            -0.020136029447262147,
+            0.007501669351583119,
+            0.023178490169347605,
+            -0.006840359541885465,
+            0.0006676450965357345,
+            -0.01283320739092204,
+            -0.015610538827742276,
+            0.006308543164022682,
+            0.002329420005586303,
+            -0.002295739637504597,
+            -0.02181189272979859,
+            0.006298804075737111,
+            -0.014398470126723037,
+            0.004007419942728446,
+            -0.0007497656993578918,
+            0.013863867106289847,
+            0.016451166273826513,
+            -0.006740209707090658,
+            0.039292260347522444,
+            0.01871723939316731,
+            0.009109005310505462,
+            0.005674782595856756,
+            -0.040588444759736254,
+            -0.008993850998154852,
+            -0.0010197185016393919,
+            0.004866150984083949,
+            -0.01141354216871663,
+            -0.018944881712855702,
+            -0.002646110151132781,
+            0.000308585036997592,
+            -0.010681954707313127,
+            -0.009091585604191968,
+            0.01751599002710486,
+            -0.006034424764302979,
+            0.00872057856926618,
+            0.004146961781109887,
+            0.023758065635807138,
+            -0.009727802672285544,
+            0.01176257119851645,
+            -0.0071822083765114165,
+            -0.005456197751568526,
+            0.0006726765703478756,
+            0.011011628907288813,
+            -0.008031792262886802,
+            0.009763340867550022,
+            -0.004279891138185881,
+            0.008355720233269893,
+            -0.008552596384067825,
+            -0.023952129458400563,
+            -0.014972827087541732,
+            0.014347535404251018,
+            0.011651260496709961,
+            0.008137738462704715,
+            -0.013624435569539285,
+            0.0031728721674035352,
+            0.00675830277744976,
+            -0.009184784787581942,
+            0.009161771311910335,
+            0.017733082848622708,
+            -0.001411542568472906,
+            -0.0017830167499636113,
+            -0.0016130003349297027,
+            -0.012173528776899308,
+            0.0011196437753239162,
+            0.02284887358189995,
+            -0.003054479643937998,
+            0.014614283913976716,
+            0.010689486748036806,
+            0.009643292262102645,
+            -0.00810930837603355,
+            0.001915955511174678,
+            0.02341863048264366,
+            0.015022291783105098,
+            -0.004625715884342598,
+            -0.011422973894623295,
+            0.004259757717597968,
+            -0.010332744704689279,
+            0.007373328473086509,
+            -0.02117940113144938,
+            -0.004672698999328004,
+            0.008896311380763048,
+            -0.007028862917603513,
+            -0.01732432285099198,
+            -0.028026173555055017,
+            -0.016765805892242715,
+            0.019819329380181,
+            0.0027213760678977965,
+            -0.013291251876048186,
+            -0.0032919084721662363,
+            0.0031214883337300926,
+            0.002553005168048115,
+            -0.0071104806975682885,
+            0.005310583052470149,
+            0.007359183453835734,
+            -0.008674133980486269,
+            0.017665958608281126,
+            -0.000521269431423141,
+            -0.0008020251454616109,
+            -0.0005442374261144659,
+            0.0006127687598711624,
+            -0.016387752806126442,
+            0.01666161912459431,
+            -0.010949228376419892,
+            0.015198110119816624,
+            -0.008966726991404666,
+            0.008600499511690635,
+            -0.007617043351740469,
+            0.009458641139070436,
+            0.011597957400445055,
+            0.012755037944648426,
+            -0.004403680820063604,
+            0.008519241473418196,
+            0.004807816191718704,
+            -0.008147125240985166,
+            0.007017601015398136,
+            0.01296082929962557,
+            0.00582353932160707,
+            0.0013634237970372698,
+            -0.0008246894322793264,
+            0.0022035670958705576,
+            -0.002480976219285683,
+            0.0056293125704658135,
+            0.012714850523393719,
+            -0.002497639449055006,
+            -0.007135683201188999,
+            0.014988249771676699,
+            0.014955895630183352,
+            -0.022994948923833993,
+            0.007121634434082677,
+            0.005835971825639792,
+            -0.0030507120821557008,
+            0.002145266953387315,
+            -0.027334525318320587,
+            -0.020359474070022213,
+            0.01901995970350808,
+            -0.015260454474605018,
+            0.02850460517419317,
+            -0.011624138792780823,
+            -0.02284242513728605,
+            -0.01277489645734181,
+            -0.004209529633534193,
+            -0.022684500849916897,
+            -0.004894414098943742,
+            0.01919165657366256,
+            0.013401560972810082,
+            -0.004313549020265677,
+            -0.019089369171102138,
+            0.03863248408555474,
+            0.010497951840980463,
+            0.03508157919687289,
+            -0.008597970944278045,
+            0.0023354831464448313,
+            -0.01563401808449416,
+            0.015551176593436512,
+            0.001075882802133782,
+            -0.011575692834796734,
+            0.01505716108583101,
+            0.031836226915668404,
+            0.0034740889237207494,
+            -0.00617206855371193,
+            0.0018420207148235309,
+            0.00418433325800744,
+            -0.0008167766650900381,
+            0.011160989178055499,
+            -0.018497472318034093,
+            -0.011951262348399922,
+            0.0013093376768780122,
+            0.00216347903176653,
+            0.0059233460419514045,
+            -0.001778091863744448,
+            -0.013317639704511549,
+            0.011433308892643204,
+            0.011596810919469926,
+            -0.01518143105525228,
+            -0.007497373860338335,
+            0.0050957665991362405,
+            -0.018974405317819656,
+            0.01872472902494418,
+            0.011859522398789263,
+            0.000792413325332956,
+            -0.0006503947129906383,
+            0.00808635755624928,
+            0.003749813548995223,
+            0.0033657799335872368,
+            -0.007862215303617144,
+            -0.0028014669075116876,
+            0.0009892310856576799,
+            3.834379772674841e-06,
+            0.015402211364447726,
+            -0.015264893397161102,
+            0.02488855366696587,
+            -0.0025643639525691083,
+            0.00953993868921819,
+            -0.017460590918773188,
+            -0.0017911712931080155,
+            -0.00402374787529868,
+            0.01838051670627292,
+            0.006753072393532114,
+            -0.010130676216186492,
+            0.004921510943896829,
+            0.007975201081296845,
+            -0.012589786869771252,
+            0.00018319054705558588,
+            -0.0022606180909706688,
+            -0.014791528072476771,
+            -0.0031171076370692706,
+            0.017873825561816544,
+            0.007593688317551089,
+            0.0037375070591255375,
+            -0.015863143964928172,
+            0.007192324815779268,
+            0.007522990667369163,
+            0.024504875417877527,
+            -0.026309816223960274,
+            0.024779042752840836,
+            -0.022104087334088543,
+            0.02165781034613893,
+            0.0319051741666687,
+            0.007538146369110045,
+            -0.01690181186708103,
+            -0.006595947249169521,
+            0.006373991305923002,
+            -0.023410814842488123,
+            -0.005494017994983188,
+            0.016903736810723716,
+            -0.01792400152372258,
+            0.000330865061689199,
+            -0.014660413003063089,
+            0.0040825880154611895,
+            0.008248037865838212,
+            0.011540333451799686,
+            -0.0033578830043738085,
+            0.018327525495676525,
+            0.020171745352159914,
+            -0.01138759854371235,
+            -0.0004221149053107702,
+            -0.022321409200941777,
+            0.005046865873981027,
+            0.022509643273449936,
+            0.001111272240506322,
+            -0.009540467231190513,
+            0.018376261423745193,
+            0.008480532791922674,
+            -0.020801891594934313,
+            0.008436374227753589,
+            -0.036175982655998785,
+            -0.010454118523313122,
+            -0.009328507118928412,
+            -0.001218633166683384,
+            -0.004142773602181115,
+            -0.0002494371580772723,
+            -0.011365548842513265,
+            -7.699548642315061e-05,
+            -0.006889339420877534,
+            -0.0037173070609413474,
+            0.014953978004522174,
+            0.011678957032329096,
+            -0.003501047405385249,
+            -0.0073984932581048915,
+            0.008493128567102267,
+            0.006163937693564917,
+            -0.017034696006318337,
+            -0.013268418502042576,
+            0.0037122128689059085,
+            -0.020067656401141467,
+            0.005148329575092478,
+            0.00459729178556505,
+            0.014031669598917171,
+            -0.01849235806598133,
+            -0.009298127665365717,
+            0.019716512208448042,
+            0.0094466405158553,
+            -0.0008988916054855498,
+            0.004264260983102956,
+            -0.00032312548009733906,
+            0.002279987803645912,
+            -0.0008902345374346296,
+            -6.145528806973222e-06,
+            0.003602229465148649,
+            0.012503366799381277,
+            0.007759393855482671,
+            0.03694402922447051,
+            -0.016692148708641812,
+            0.005873765863350649,
+            -0.014187609702966002,
+            -0.0037592517335595654,
+            -0.00749939010396784,
+            -0.0004308157309481927,
+            -0.0022282234382577843,
+            0.0021999996319642854,
+            0.0052176773733353925,
+            0.009353171029508977,
+            0.002155405440765206,
+            -0.00823565584692709,
+            0.009691447250255694,
+            -0.015228173095533029,
+            0.003853858061067531,
+            -0.008207910291425473,
+            0.01017972550354623,
+            -0.009695280827515048,
+            0.0028705587799331246,
+            0.03849358164875786,
+            0.01130021300937487,
+            -0.004934175853966098,
+            -0.014300147534361459,
+            -0.0007796051618993811,
+            -0.01737877501558196,
+            -0.008579119859817895,
+            -0.005434792488602325,
+            -0.02617338653902575,
+            -0.00838916510137031,
+            0.015344260464413648,
+            0.011202567142310492,
+            0.014124853863170132,
+            -0.023894182185855824,
+            0.008897943489322673,
+            -0.03001670823127986,
+            -0.011951507895140876,
+            -0.001530967140156163,
+            -0.015467812535428609,
+            0.0010110999991167703,
+            0.023623196504289838,
+            -0.007762735863648054,
+            0.012654652822750325,
+            0.012867958170258448,
+            0.018777522196571934,
+            0.006390986057157016,
+            0.00038045736974183937,
+            -0.009444911783760569,
+            0.009534310165417939,
+            0.008806031651319704,
+            0.0080954543713255,
+            0.013161833638467553,
+            -0.0022840529693446553,
+            -0.018060162439023572,
+            0.005469288590000856,
+            0.014354475778235131,
+            -0.0033504276914600585,
+            -0.001615191819057537,
+            -0.011088493180661976,
+            0.016039040565630432,
+            -0.023559046333872156,
+            -0.009482398794557168,
+            0.009918576960319548,
+            -0.0027308309155029842,
+            0.009212063547988036,
+            0.0160181200081126,
+            -0.01881373001297218,
+            -0.0024168618343039703,
+            0.016244889977796682,
+            0.0024104546308670025,
+            -0.00022606425578303413,
+            0.016792244238218245,
+            -0.0007616857360728623,
+            -0.004279120886828465,
+            0.00997764266310123,
+            -0.0075631917138318655,
+            7.3878137053128405e-06,
+            -0.002416364915008295,
+            -0.0028253498168087964,
+            0.02054128727706496,
+            -0.021281801678871736,
+            -0.005379259148621041,
+            0.02257306103533458,
+            0.015687105622685975,
+            -0.007843745111084413,
+            6.258498042855072e-05,
+            -0.0219506172079994,
+            0.0019925886885244454,
+            -0.007768133999661569,
+            0.016524850592507592,
+            -0.007957949830060354,
+            0.0030085044673901665,
+            -0.01361024620290394,
+            0.011563781350927133,
+            -0.01277407169098418,
+            0.00873716215529134,
+            -0.002986943948333218,
+            -0.017567646684567492,
+            -0.0053109390854215125,
+            0.004561236136619266,
+            0.0014045175130801308,
+            0.044958041347040185,
+            -0.005539136232703934,
+            0.003729517522976541,
+            0.018086335927383804,
+            -0.014199453349727933,
+            0.02079719360289355,
+            0.016663293302592568,
+            0.00991445855219406,
+            -0.00033967370741635794,
+            -0.0018726748849858836,
+            0.023373354806639515,
+            0.013258226441642464,
+            0.00861986998150065,
+            -0.0014340218811281975,
+            -0.008516541227326329,
+            0.018239230461345617,
+            0.005458629943667832,
+            -0.012421896028975013,
+            0.012732721349758832,
+            0.013559877132508506,
+            -0.013720939472355347,
+            -0.011545210834395917,
+            0.01616358223872008,
+            -0.019904594319155373,
+            -0.018260016571763644,
+            -0.0036124537591368864,
+            -0.01614510591132948,
+            -0.03864699384205461,
+            0.01619401261888991,
+            0.018351607973242735,
+            0.006666313000123845,
+            0.013647410298896132,
+            0.002597680061427196,
+            0.004545530267244157,
+            -0.018531082430861854,
+            -0.0077618625909995275,
+            -0.008085144524135461,
+            0.003744735923004248,
+            0.011059457364156674,
+            -0.011335432904114519,
+            -0.009514379703581616,
+            -0.00034500702695418285,
+            -0.007433132060989644,
+            0.01971420270994951,
+            0.0020753610290475642,
+            0.0034506706065161024,
+            -0.00628437298309269,
+            0.017966485854173167,
+            0.026789678964152908,
+            -0.007532446825077476,
+            0.021886182869092073,
+            -0.002864230253056434,
+            -0.0015870360120997492,
+            -0.014393519359298496,
+            -0.0014439412842964876,
+            -0.011527182213472351,
+            0.0005453493437517964,
+            -0.0006791481533636673,
+            0.014093767997869645,
+            -0.03288824642147967,
+            -0.003957301296796813,
+            0.014444592778851293,
+            0.007538052312876599,
+            -0.005057755472988024,
+            0.011357919882949903,
+            -0.002964451028327082,
+            0.016977317604413924,
+            0.0075201055190542146,
+            0.023391646079994784,
+            -0.016588823248397513,
+            -0.004504858318770565,
+            0.006660877064954993,
+            0.0034631255931793638,
+            0.0014484203909961972,
+            0.01912983041187014,
+            0.023439941258581887,
+            -0.014715886385460541,
+            -0.0013596123366001458,
+            -0.0029020372473879614,
+            -0.021414221343490594,
+            0.014976806123253727,
+            0.0074729481304182865,
+            0.010967351226114886,
+            0.009806663105020302,
+            0.015465076881663691,
+            -0.02109587527354609,
+            0.0011671121739822202,
+            0.020567458365051382,
+            -0.008618333549105003,
+            -0.018815706906171112,
+            -0.009804829158883123,
+            -0.00966828714719191,
+            -0.006078893678150302,
+            -0.00803783313451146,
+            0.005523023167028105,
+            0.0022647506733877573,
+            0.01652997604669998,
+            -0.008204200434441975,
+            -0.008404609147866811,
+            0.008489247125986078,
+            0.007546785673285454,
+            -0.0033093451283570316,
+            0.010614050963656496,
+            0.0030633179859393587,
+            -0.006940540318011839,
+            0.00894884343338777,
+            -0.002787816848430143,
+            0.014347598920245487,
+            -0.02441769157218451,
+            0.004717742147185283,
+            -0.012096876825337124,
+            0.006405151160675699,
+            0.0041568790527989905,
+            0.0016702066682613428,
+            -0.0041159211459004,
+            0.0035217362856754353,
+            0.0008893479210743541,
+            -0.07433312151926533,
+            -0.012629681414004524,
+            -0.0035797012285864486,
+            0.033657838713146884,
+            -0.019695806809053914,
+            0.004477927056336216,
+            0.011966065644382396,
+            0.0096717896523898,
+            0.003070713232538593,
+            0.006136726101157844,
+            0.006112385569942848,
+            -0.0245258242481677,
+            -0.00015284418977092615,
+            0.004821040694679944,
+            0.0020347004045943454,
+            -0.007532408630122127,
+            0.00750565079228405,
+            0.015327710245879296,
+            0.020989926895388523,
+            -0.010970305087289488,
+            0.017227663358885748,
+            -0.003352718496981928,
+            -0.006068892126250978,
+            0.0014285750322333686,
+            -0.010153799722483554,
+            -0.017307415843726394,
+            -0.006495008086641125,
+            0.0068983699213297075,
+            -0.001982568363667046,
+            0.0044032687939897855,
+            0.003333596495330213,
+            -0.0032171586993948696,
+            0.0072365127663633925,
+            0.013920326463272646,
+            0.006415916101287133,
+            -0.013675562796300214,
+            0.003003782758807665,
+            -0.02500375035312058,
+            0.011945988879567646,
+            -0.012708975181025296,
+            -0.0009488161508767279,
+            -0.0047093877027523014,
+            -0.0035750724825159503,
+            -0.002213435677949806,
+            0.02043567538125188,
+            -0.014304009033474186,
+            -0.01802899775650995,
+            0.008187227768196673,
+            -0.011217013195839762,
+            0.008300740149622635,
+            -0.03382831122769974,
+            0.012386335491333264,
+            -0.0276029934382802,
+            -0.0002243752790856292,
+            -0.004521387639042107,
+            -0.0010631721066129095,
+            -0.009491184157278669,
+            -0.03274559774619223,
+            0.015845705808609598,
+            -0.007204751554079551,
+            -0.020338337341085228,
+            0.0007143424225187802,
+            -0.021267094083674304,
+            0.014150458196361147,
+            -0.01851165553224022,
+            0.0033450353123933058,
+            -0.018775439467431685,
+            0.002782289821140401,
+            -0.0018711350216089556,
+            0.010502195116579441,
+            -0.00557814442809019,
+            -0.0007130962242755068,
+            -0.013072988887382269,
+            -0.002121841212381393,
+            -0.006653444103331213,
+            0.009394907690516121,
+            0.0028678074167728367,
+            -0.009601577734902932,
+            0.011153571864462314,
+            0.013046116892519414,
+            0.019812860521973938,
+            -0.00539047787915588,
+            -0.00555139958411929,
+            -0.011553689090179406,
+            0.009852063270798534,
+            -0.016326212706634428,
+            -0.00827402913960903,
+            0.008982303148984887,
+            0.015216947130173639,
+            0.01026889105233137,
+            0.003352028462252994,
+            0.003150801750574548,
+            0.002531620731961765,
+            0.026744691368233237,
+            -0.0015120532859309638,
+            0.0008324701960729416,
+            -0.014508988496009931,
+            0.022127525293249926,
+            0.02772430884375744,
+            -0.0012206102581726523,
+            -0.00018859639594030037,
+            0.017437360169132967,
+            -0.004697052522250805,
+            -0.0038111184371524645,
+            0.024015246959315593,
+            -0.038396108183338655,
+            -0.008416825232533771,
+            -0.006788816882617399,
+            -0.006043428198944214,
+            -0.009587366247924467,
+            0.007294190273465812,
+            0.00041966080769004275,
+            0.0024709692753547823,
+            -0.0005250904505954556,
+            0.03452858797518386,
+            0.003403515654095309,
+            -0.0003392060101496309,
+            -0.0032980864446307116,
+            0.0019125121221204029,
+            0.0032987517853214346,
+            -0.009116761755779728,
+            -0.038030085122772624,
+            0.006006334000396066,
+            0.007932785488877407,
+            0.00975009749364395,
+            -0.008120121513849434,
+            -0.008864461019882246,
+            0.004909348738798099,
+            -0.02273129283865203,
+            -0.01372636307167243,
+            0.02625229144362647,
+            -0.008523028986127237,
+            0.029005877324974557,
+            -0.022435059061182174,
+            -0.013135836801818153,
+            -0.004126433322528265,
+            0.010283975013795537,
+            -0.019732856462444506,
+            0.004343860338939202,
+            -0.004022387065239562,
+            0.023733721026005954,
+            0.0077578144626961245,
+            -0.008371631446146718,
+            0.012386735614153616,
+            0.0064789380436815714,
+            -0.0026638642827013837,
+            0.00136333052963848,
+            -0.007209517067748103,
+            0.016916505654836485,
+            -0.011819784026452944,
+            -0.005833909403097991,
+            0.008977527508585844,
+            -0.0024556434204852946,
+            0.02509293948918158,
+            -0.0237762245071987,
+            0.005288171058938098,
+            -0.00019047961064753207,
+            0.01822542471604037,
+            0.008127347078975925,
+            0.008361588667386882,
+            -0.011835407590060994,
+            0.010004482616697591,
+            -0.03378107067882622,
+            -0.015433906677849963,
+            -0.007702910250795191,
+            -0.0013877702264578865,
+            -0.002059152251387601,
+            -0.015745218548968338,
+            -0.026393221735795527,
+            0.006294583740809379,
+            0.016267525575472445,
+            0.023291528020980265,
+            -0.004683197930892445,
+            0.012912830736650997,
+            -0.002831583991769044,
+            -0.008904271394710672,
+            -0.011441882849296418,
+            -0.012202629605208118,
+            -0.012470773094383122,
+            -0.014560694782340029,
+            -0.017045701603711442,
+            0.012139665577738337,
+            -0.020079061996873537,
+            0.013911518328924777,
+            0.0347210775522539,
+            -0.017176114991595492,
+            -0.0018941772168839354,
+            0.019702722555622765,
+            -0.004221469612316137,
+            0.01212088353057276,
+            0.0063707489618353725,
+            0.015298014532351866,
+            0.0038643174573634725,
+            -0.012181479717826009,
+            -0.007374005740809251,
+            0.019411457664902376,
+            -0.004902591844710158,
+            0.014591446725717338,
+            0.015069599884557399,
+            -0.008675672164385891,
+            0.002544198867387345,
+            0.01951865904496276,
+            -0.01762101777798757,
+            0.01665654707715257,
+            -0.00458402229480694,
+            -0.009936206685538294,
+            -0.013345740581684638,
+            -0.005336416724426228,
+            -0.019469457091169776,
+            0.0061608454109892425,
+            0.018931676738452464,
+            -0.00533013594306834,
+            0.009310871143727683,
+            -0.0037670793158045,
+            -0.005045151384034014,
+            0.0002041909886640482,
+            -0.04152751596227068,
+            -0.008340911336217609,
+            -0.0006651584370005978,
+            -0.02505748815737022,
+            0.025954102004571416,
+            0.007205602377144382,
+            0.022602344303400163,
+            -0.017168610245491406,
+            0.004459735690564842,
+            -0.005882024209102714,
+            0.013217517324383834,
+            0.02255689669623969,
+            0.0017984057469785134,
+            0.005699128497215882,
+            -0.02353849721739205,
+            -0.0034711576718386185,
+            0.015924913374553738,
+            -0.003604478141616307,
+            -0.0034993903155063817,
+            0.012592308877056896,
+            0.012864938348003884,
+            0.01567418642605425,
+            0.008801753550506258,
+            0.015606443428689292,
+            0.010589933994371628,
+            0.008928721556432368,
+            -0.005805129999077126,
+            -0.001025103064751441,
+            -0.015907755521365925,
+            0.0045874330999415985,
+            0.031701049450012876,
+            0.012594483199773746,
+            0.016783858873835704,
+            -0.01778536268185936,
+            0.00789523134933153,
+            -0.0012706411671644011,
+            0.008325108613530198,
+            -0.001677173669891956,
+            0.029540136288412976,
+            0.011739999605699826,
+            0.015670966746894652,
+            0.008732334090105053,
+            -0.016237731209426752,
+            0.045334690225148,
+            -0.016911928942220152,
+            -0.0021139311364009113,
+            -0.01058970056572469,
+            -0.021705338531028618,
+            0.007324792557651635,
+            -0.006556582104686972,
+            0.0018243188317438144,
+            0.010780292738415671,
+            -0.006471850255151019,
+            0.0006997395548596146,
+            0.003907433192252573,
+            -0.023129313269763224,
+            -0.002550875642832235,
+            -0.02237917712764273,
+            -0.003107775639788617,
+            -0.02621608419628401,
+            -0.00012482032027919036,
+            -0.016207377503696517,
+            -0.0038712739995960753,
+            -0.023315508569288476,
+            -0.014963211090562272,
+            -0.01633110043829169,
+            -0.004241081909202262,
+            -0.008295402942465353,
+            0.004976396647108597,
+            -0.007402107755970892,
+            -0.020321297548927468,
+            0.0020986169430507757,
+            0.016803273183070004,
+            0.01434729320047051,
+            0.018612970083178984,
+            0.00603868809307069,
+            0.014278001198851581,
+            -0.0026350854954985117,
+            0.017171016209107656,
+            -0.006968968261909459,
+            0.006881847166565678,
+            0.011454001418121107,
+            0.02140935294963631,
+            0.011121833270235448,
+            -0.012801756637470525,
+            0.0010870041670931706,
+            -0.009655388034828094,
+            0.002094601818569396,
+            -0.00828486034340067,
+            -0.009190975158350585,
+            0.0017762581946855365,
+            -0.007261996868906865,
+            0.004674405202069334,
+            0.0030526253833799426,
+            -0.0014082232541965316,
+            0.010419302200840906,
+            -0.0024200053377751805,
+            -0.012208685875688886,
+            -0.0033037330470612695,
+            -0.017975940046938498,
+            -0.006958042903103732,
+            -0.0014496522238488239,
+            -0.011711502944602638,
+            -0.01016130418405875,
+            0.003321436189059608,
+            -0.00538282506552848,
+            -0.009353581612407435,
+            0.013884147453136417,
+            0.0001309738696853529,
+            0.0005878330075918333,
+            0.023495280602974504,
+            0.006035028203642483,
+            0.008087594627921345,
+            0.013144288945385904,
+            -0.01023995000994454,
+            -0.022574958230635715,
+            -0.0243435727055662,
+            0.007867022215096418,
+            -0.01573478573423379,
+            0.02324699908478915,
+            -0.007929809922043611,
+            -0.00561630876431978,
+            0.0020982723335057625,
+            -0.013699317926847062,
+            0.008578431496569186,
+            -0.011271381061801356,
+            0.0014450645788554738,
+            -0.019581451799757368,
+            0.018059311800267413,
+            -0.010917911869373693,
+            -0.004302687883018328,
+            0.009466103226841532,
+            0.034049086327052606,
+            -0.0005689605174080737,
+            -0.007410970644432269,
+            -0.006594150877740108,
+            -0.00022810271313786974,
+            0.018766765773409123,
+            -0.0029041815756733333,
+            0.005758781908059541,
+            -0.01563519107805783,
+            0.006571628287171105,
+            -0.006132372252825422,
+            0.0012423261344297225,
+            0.01513074831647933,
+            -0.026306320707079913,
+            -0.015491710256868008,
+            -0.007857352164317122,
+            -0.024893430640551818,
+            0.002253574511878983,
+            0.017579477884206534,
+            0.020505606775521504,
+            -0.020324505029926956,
+            -0.02513332780748072,
+            0.005150966733930881,
+            -0.026580994694344065,
+            0.0037390577082486594,
+            0.1241031878517593,
+            -0.002138404898309707,
+            0.002720723037908521,
+            -0.011425036886011179,
+            0.0022399461009316216,
+            0.005389407121650262,
+            -0.019552543955456567,
+            0.05950746473186985,
+            -0.00638023587296763,
+            -0.004023798021748334,
+            0.002170545188776296,
+            0.001785855390676113,
+            -0.005831534307818096,
+            0.008643501093778434,
+            -0.017602787281866816,
+            -0.02184792441891879,
+            -0.0052215705748838905,
+            -0.021252169339074325,
+            -0.008509621422071373,
+            -0.013000137524907565,
+            0.002893880203681282,
+            -0.015821609145923173,
+            0.010397692426614709,
+            -0.01638916243796692,
+            -0.010283706937701179,
+            -0.01103621454761223,
+            -0.00373552043529251,
+            0.01563553822905027,
+            -0.03188533919653949,
+            -0.004873550301425776,
+            -0.005183590054152726,
+            0.018218971950313273,
+            0.006004328765748325,
+            0.004357734370247252,
+            -0.014230318971458851,
+            0.0064792070801456235,
+            0.0215589173714926,
+            0.0055272493764401305,
+            0.014643335833970546,
+            0.010618070641357003,
+            -0.0022177303813575846,
+            -0.0024402761885758007,
+            0.017541799738501007,
+            -0.005445774414228888,
+            -0.00023502936119626386,
+            -0.00537569046685066,
+            -0.005974703247446444,
+            -0.010712244203425036,
+            0.01843613166635215,
+            -0.009938564643456672,
+            -0.017388592763277214,
+            -0.0015797412875366156,
+            0.02028434083164951,
+            -0.012761534382003973,
+            0.0031906812114950982,
+            0.000788436210275669,
+            0.020589196433948156,
+            0.0007726855314648301,
+            -0.005481260996289263,
+            0.007165014826294827,
+            -0.02956334397512344,
+            0.004095715232997753,
+            0.0003568512047626187,
+            0.027290768799658912,
+            -0.0017631129386615335,
+            0.01552542586826711,
+            -0.020871286597113327,
+            0.004388125824324094,
+            -0.016926008631435403,
+            0.005099899758322398,
+            0.012560923968841436,
+            0.002781049890810361,
+            0.010889021753593997,
+            -0.008500358897706648,
+            -0.021353460542554444,
+            0.012663934009161902,
+            -0.018657094130860155,
+            -0.029156017587920958,
+            0.0018889388761887053,
+            -0.028158076315864772,
+            -0.003048854609750186,
+            0.006812072971079798,
+            -0.035217412591558014,
+            -0.02251675180883659,
+            0.007396524512996277,
+            -0.00995571600201846,
+            -0.011912753666073882,
+            0.010675240017836214,
+            -0.02645155743041798,
+            -0.018137648527851668,
+            0.0010666944556449141,
+            0.012588800077165337,
+            -0.004561825939541096,
+            0.007555558090958502,
+            -0.028558603070876605,
+            0.014816816621445352,
+            -0.01313764261744744,
+            0.016036065063831813,
+            -0.0037797584923144134,
+            0.022217516257304868,
+            0.011131597210722573,
+            0.008795376259031849,
+            -0.004950080558710245,
+            0.021409590863550987,
+            0.005574351961953293,
+            0.051358937509801736,
+            0.031972009383746405,
+            0.0019821003238211276,
+            -0.003251257665378234,
+            0.003313045083311649,
+            0.008294776423730887,
+            -0.00940948257229233,
+            0.005114631360856872,
+            0.028053227377164806,
+            0.016738013668710083,
+            -0.018755924318109216,
+            0.006606711923035118,
+            -0.02702692774986498,
+            0.017950884766781056,
+            -0.022054653845163,
+            0.0003755944703013671,
+            0.007766253634286795,
+            0.0016206993755471613,
+            0.010682347575070117,
+            0.0013842930079185415,
+            0.01705234255261425,
+            0.023975191051210525,
+            -0.002976402980885077,
+            0.017466027107477605,
+            0.012734909275073793,
+            0.018143161773162015,
+            -0.0038399373197265827,
+            0.006929440921265761,
+            -0.012003095293274786,
+            -0.0008883470597690258,
+            0.0020490527756307267,
+            -0.004231091383919565,
+            0.012128848448049833,
+            0.008340302519160299,
+            0.012989849936937843,
+            0.002974488532907779,
+            0.003198854381461613,
+            -0.005309753760578859,
+            -0.0017024982049936384,
+            -0.01763395035509857,
+            -0.01819680226734474,
+            0.028417724274372468,
+            -0.020217879775485576,
+            -0.050118184195774004,
+            0.0025509548991664425,
+            0.014185761006969567,
+            -0.0012021669304388173,
+            -0.0051260576846881515,
+            -0.0005774778872927327,
+            0.005718864494530703,
+            -0.016918749210038116,
+            -0.0015472638835434902,
+            0.008125845907627804,
+            0.020347547062899797,
+            -0.002036218493206914,
+            -0.01079739518866248,
+            0.0038112958713600906,
+            0.02169339303105242,
+            0.01667594322285835,
+            0.012338193092567552,
+            -0.009798409233733167,
+            0.0003198215558702285,
+            -1.6308740825163337e-05,
+            0.0076870827874737325,
+            0.004356644356656038,
+            0.003500023005601613,
+            -0.019968549869439228,
+            -0.008634740687755112,
+            0.012781565963545847,
+            -0.0025637758121512927,
+            0.010243567793632335,
+            -0.00042296044205132426,
+            -0.003069251553118065,
+            -0.009344199449486244,
+            -0.004694953215957636,
+            -0.009583993744231791,
+            0.013783878036521561,
+            -0.007705982042252044,
+            -0.01549567538665377,
+            0.00020666298495823114,
+            0.0005515775451348472,
+            0.001667367278489879,
+            -0.004518431526236256,
+            0.03302324454129141,
+            0.009531990824506103,
+            -0.0004937348108340977,
+            -0.004047613286672229,
+            -0.0327383112522095,
+            0.012456150771224677,
+            -0.009677533214778067,
+            0.0006455811519981003,
+            0.020012976827426605,
+            -0.00020277620481825933,
+            0.002168432791818704,
+            -0.015657551629104767,
+            0.0032924357559238502,
+            0.006161295735841883,
+            0.013814165416788555,
+            0.0005164426482972658,
+            0.01076512534105537,
+            -0.006498977875968572,
+            -0.010325144017898257,
+            0.004804158067334203,
+            0.0104064855510473,
+            -0.0012188100892986933,
+            0.005171197790296134
+          ],
+          "title": "Mixture-Of-Experts With Expert Choice Routing",
+          "keyphrases": [
+            "Mixture-Of-Experts",
+            "Expert Choice Routing",
+            "sparsely-activated models",
+            "training efficiency",
+            "downstream performance"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "1e687808-8db8-4320-805d-0282a8d679bc",
+        "properties": {
+          "page_content": "Deepseekmoe: Towards Ultimate Expert Specialization In\n\nMixture-Of-Experts Language Models\n\nDamai Dai∗1,2, Chengqi Deng1, Chenggang Zhao∗1,3, R.X. Xu1, Huazuo Gao1, Deli Chen1, Jiashi Li1, Wangding Zeng1, Xingkai Yu∗1,4, Y. Wu1, Zhenda Xie1, Y.K. Li1, Panpan Huang1, Fuli Luo1, Chong Ruan1, Zhifang Sui2, Wenfeng Liang1 1DeepSeek-AI 2National Key Laboratory for Multimedia Information Processing, Peking University 3Institute for Interdisciplinary Information Sciences, Tsinghua University 4National Key Laboratory for Novel Software Technology, Nanjing University {daidamai, szf}@pku.edu.cn, {wenfeng.liang}@deepseek.com https://github.com/deepseek-ai/DeepSeek-MoE\n\nAbstract\n\nIn the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top- out of experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into ones and activating from them, allowing for a more flexible combination of activated experts; (2) isolating experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5× expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations.\n\nFurther, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\n1. Introduction\n\nRecent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023a). It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017) has emerged as a popular solution. It can\n\nenable parameter scaling, while concurrently keeping computational costs at a modest level.\n\nRecent applications of MoE architectures in Transformers (Vaswani et al., 2017) have yielded successful attempts at scaling language models to a substantial size (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022), accompanied with remarkable performance. These achievements underscore the considerable potential and promise of MoE language models.\n\nDespite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts.\n\nThis architecture manifests two potential issues: (1) Knowledge Hybridity: existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. (2) Knowledge Redundancy: tokens assigned to different experts may require common knowledge. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters.\n\nThese issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models.\n\nIn response to the aforementioned issues, we introduce DeepSeekMoE, an innovative MoE architecture specifically designed towards ultimate expert specialization. Our architecture involves two principal strategies: (1) Fine-Grained Expert Segmentation: while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. (2) Shared Expert Isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.\n\nThrough compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized.\n\nStarting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B (Lepikhin et al., 2021) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5× expert parameters and computation. Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE.\n\nThese studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization.\n\nLeveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B (DeepSeek-AI, 2024), a dense model trained on the same 2T corpus. We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B (Touvron et al., 2023b), which has approximately 2.5 times the activated parameters. Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard1. Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. The experimental results still validate its substantial advantages over the GShard architecture consistently. In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nOur contributions are summarized as follows: - Architectural Innovation. We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation.\n\nEmpirical Validation. We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. Experimental results validate the high 1https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models\n\nScalability. We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B.\n\nAlignment for MoE. We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B.\n\nPublic Release. In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization.\n\n2. Preliminaries: Mixture-Of-Experts For Transformers\n\nWe first introduce a generic MoE architecture commonly used in Transformer language models. A standard Transformer language model is constructed by stacking layers of standard Transformer blocks, where each block can be represented as follows:\n\n$$\\mathbf{u}{1:T}^{l}=\\text{Self-Att}\\left(\\mathbf{h}{1:T}^{l-1}\\right)+\\mathbf{h}{1:T}^{l-1},\\tag{1}$$ $$\\mathbf{h}{t}^{l}=\\text{FFN}\\left(\\mathbf{u}{t}^{l}\\right)+\\mathbf{u}{t}^{l},\\tag{2}$$\n\nwhere denotes the sequence length, Self-Att(·) denotes the self-attention module, FFN(·) denotes the Feed-Forward Network (FFN), u 1: ∈ R× are the hidden states of all tokens after the -th attention module, and h\n\n∈ Ris the output hidden state of the -th token after the -th Transformer block. For brevity, we omit the layer normalization in the above formulations.\n\nA typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022). An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. Then, each token will be assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts. If the -th FFN is substituted with an MoE layer, the computation for its output hidden state h\n\nis expressed as: where denotes the total number of experts, FFN(·) is the -th expert FFN, , denotes the gate value for the -th expert, , denotes the token-to-expert affinity, Topk(·, ) denotes the set comprising highest affinity scores among those calculated for the -th token and all experts, and e\n\nis the centroid of the -th expert in the -th layer. Note that ,is sparse, indicating that only out of gate values are nonzero. This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only experts. Also, in the above formulations, we omit the layer normalization operation for brevity.\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{N}\\left(g_{i,t}\\operatorname{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\operatorname{Topk}({s_{j,t}|1\\leqslant j\\leqslant N},K),\\ 0,&\\text{otherwise},\\end{cases}$$ $$s_{i,t}=\\operatorname{Softmax}{i}\\left(\\mathbf{u}{t}^{l}\\,\\mathbf{e}_{i}^{l}\\right),$$\n\n$$(3)$$\n\n(4) $\\binom{4}{5}$ .\n\n3. Deepseekmoe Architecture\n\nOn top of the generic MoE architecture outlined in Section 2, we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. As illustrated in Figure 2, our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. Both of these strategies are designed to elevate the level of expert specialization.\n\n3.1. Fine-Grained Expert Segmentation\n\nIn scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts.\n\nIn pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. The finer expert segmentation enables a more flexible and adaptable combination of activated experts. To be specific, on top of a typical MoE architecture shown in Figure 2(a), we segment each expert FFN into smaller experts by reducing the FFN intermediate hidden dimension to 1 times its original size. Since each expert becomes smaller, in response, we also increase the number of activated experts to times to keep the same computation cost, as illustrated in Figure 2(b). With the fine-grained expert segmentation, the output of an MoE layer can be expressed as:\n\n$$\\mathbf{h}{t}^{i}=\\sum{i=1}^{\\text{mN}}\\left(g_{i,t}\\text{FFN}{i}\\left(\\mathbf{u}{t}^{i}\\right)\\right)+\\mathbf{u}{t}^{i},\\tag{6}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\text{Tokr}(s_{i,t}|1\\leqslant j\\leqslant m\\text{N}),m\\text{K}),\\ 0,&\\text{otherwise},\\end{cases}$$ (7) $$s_{i,t}=\\text{Softmax}{i}\\left(\\mathbf{u}{t}^{T}\\mathbf{e}_{t}^{i}\\right),\\tag{8}$$ where the total number of expert parameters is equal to $N$ times the number of parameters in a\n\n$$(6)$$\n\n(7) (8) $\\frac{1}{2}$\n\nstandard FFN, and denotes the total number of fine-grained experts. With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to .\n\nFrom a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. As an illustrative example, we consider the case where = 16. A typical top-2 routing strategy can yield 16 2 = 120 possible combinations. By contrast, if each expert is split into 4 smaller experts, the fine-grained routing strategy can yield 64 8 = 4, 426, 165, 368 potential combinations. The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition.\n\n3.2. Shared Expert Isolation\n\nWith a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. This alleviation of redundancy will contribute to a more parameterefficient model with more specialized experts.\n\nTowards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate experts to serve as shared experts. Regardless of the router module, each token will be deterministically assigned to these shared experts. In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by , as depicted in Figure 2(c). With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows:\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{K_{s}}\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)+\\sum_{i=K_{s}+1}^{mN}\\left(g_{i,t}\\;\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},\\tag{9}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\mathrm{Topk}{s_{j,t}|K_{s}+1\\leqslant j\\leqslant mN},mK-K_{s}},\\ 0,&\\text{otherwise},\\end{cases}$$ (10) $$s_{i,t}=\\mathrm{Softmax}{i}\\left(\\mathbf{u}{t}^{l,T}e_{i}^{l}\\right).\\tag{11}$$ $\\mathbf{h}\\mathbf{M}\\mathbf{E}$ the number of closed-compact is $K_{s}$ the total number of routed currents.\n\nFinally, in DeepSeekMoE, the number of shared expert is , the total number of routed experts is − , and the number of nonzero gates is − .\n\nIt is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022). The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint.\n\n3.3. Load Balance Consideration\n\nAutomatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. Firstly, there is a risk of routing collapse (Shazeer et al., 2017), i.e., the model always selects only a few experts, preventing other experts from sufficient training.\n\nSecondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.\n\nExpert-Level Balance Loss. In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. The computation of the balance loss is as follows:\n\n$$\\begin{array}{l}{{{\\mathcal L}{\\mathrm{ExpBal}}=\\alpha{1}\\sum_{i=1}^{N^{\\prime}}f_{i}P_{i},}}\\ {{f_{i}=\\frac{N^{\\prime}}{K^{T}}\\sum_{t=1}^{T}\\mathds{1}\\,(\\mathrm{Token}\\;t\\;\\mathrm{s}\\mathrm{lects}\\;\\mathrm{Expert}\\;i),}}\\ {{P_{i}=\\frac{1}{T}\\sum_{t=1}^{T}s_{i,t},}}\\end{array}$$\n\n(12) $$\\begin{array}{l}\\small\\text{(13)}\\end{array}$$ = (14) $$\\begin{array}{l}\\small\\text{(14)}\\end{array}$$ .\n\n7 where 1 is a hyper-parameter called expert-level balance factor, ′is equal to ( − ) and ′ is equal to ( − ) for brevity. 1(·) denotes the indicator function.\n\nDevice-Level Balance Loss. In addition to the expert-level balance loss, we introduce a devicelevel balance loss. When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. Instead, our primary objective is to ensure balanced computation across the devices. If we partition all routed experts into groups {E1, E2, ..., E}, and deploy each group on a single device, the device-level balance loss is computed as follows:\n\n$$\\mathcal{L}{\\text{DevBal}}=\\alpha{2}\\sum_{i=1}^{D}f_{i}^{\\prime}P_{i}^{\\prime},$$ $$f_{i}^{\\prime}=\\frac{1}{|\\mathcal{E}{i}|}\\sum{j\\in\\mathcal{E}{i}}f{j},$$ $$P_{i}^{\\prime}=\\sum_{j\\in\\mathcal{E}{i}}P{j},$$\n\n(15) $$\\begin{array}{l}\\small\\text{(16)}\\end{array}$$ = (17) .\n\nwhere 2 is a hyper-parameter called device-level balance factor. In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices.\n\n4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n6. Alignment For Deepseekmoe 16B\n\nPrevious research indicates that MoE models typically do not emerge significant gains from fine-tuning (Artetxe et al., 2022; Fedus et al., 2021). However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B.\n\n6.1. Experimental Setup\n\nTraining Data. For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios.\n\nHyper-Parameters. During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).\n\nWe employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10−5 without incorporating any learning rate scheduling strategy.\n\nEvaluation Benchmarks. For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3, with the following adjustments: (1) We exclude Pile (Gao et al., 2020) since chat models are seldom employed for pure language modeling. (2) We exclude\n\nMetric # Shot LLaMA2 DeepSeek DeepSeekMoE SFT 7B Chat 7B Chat 16B # Total Params N/A 6.7B 6.9B 16.4B # Activated Params N/A 6.7B 6.9B 2.8B FLOPs per 4K Tokens N/A 187.9T 183.5T 74.4T HellaSwag (Acc.) 0-shot 67.9 71.0 72.2 PIQA (Acc.) 0-shot 76.9 78.4 79.7 ARC-easy (Acc.) 0-shot 69.7 70.2 69.9 ARC-challenge (Acc.) 0-shot 50.8 50.2 50.0 BBH (EM) 3-shot 39.3 43.1 42.2 RACE-middle (Acc.) 5-shot 63.9 66.1 64.8 RACE-high (Acc.) 5-shot 49.6 50.8 50.6 DROP (EM) 1-shot 40.0 41.7 33.8 GSM8K (EM) 0-shot 63.4 62.6 62.2 MATH (EM) 4-shot 13.5 14.7 15.2 HumanEval (Pass@1) 0-shot 35.4 45.1 45.7 MBPP (Pass@1) 3-shot 27.8 39.0 46.2 TriviaQA (EM) 5-shot 60.1 59.5 63.3 NaturalQuestions (EM) 0-shot 35.2 32.7 35.1 MMLU (Acc.) 0-shot 50.0 49.7 47.2 WinoGrande (Acc.) 0-shot 65.1 68.4 69.0 CLUEWSC (EM) 5-shot 48.4 66.2 68.2 CEval (Acc.) 0-shot 35.1 44.7 40.0 CMMLU (Acc.) 0-shot 36.9 51.2 49.3\n\nCHID (Zheng et al., 2019) due to the observed instability of results, hindering the derivation of solid conclusions. (3) We additionally include BBH (Suzgun et al., 2022) to provide a more comprehensive assessment of the reasoning ability of the chat models.\n\nTable 5 | Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations.\n\n6.2. Evaluations\n\nBaselines. In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. Correspondingly, we construct three chat models, including LLaMA2 SFT 7B3, DeepSeek Chat 7B, and DeepSeekMoE Chat 16B.\n\nSubsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks.\n\nResults. The evaluation results are presented in Table 5. Our key observations include: (1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intensive tasks (TriviaQA, NaturalQuestions). (2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. In addition, it also surpasses DeepSeek Chat 7B. (3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1). However, it is worth noting that, after supervised finetuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. (4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations.\n\n7. Deepseekmoe 145B Ongoing\n\nEncouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available.\n\n7.1. Experimental Setup\n\nTraining Data and Tokenization. For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study.\n\nModel Settings. For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps.\n\nSubsequently, the learning rate keeps constant during the remaining training process. The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. Also, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse.\n\nEvaluation Benchmarks. We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3).\n\n7.2. Evaluations\n\nBaselines. Apart from DeepSeekMoE 145B, we consider three additional models for comparison. DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters.\n\nResults. From the evaluation results presented in Table 6, we have the following observations: (1) Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. (2) On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. (3) At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5.\n\n8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\nAppendices A. Overview Of Hyper-Parameters\n\nWe present the overview of hyper-parameters for DeepSeekMoE across various sizes in Table 7.\n\n# Params # Layers Hidden # Attn # Shared # Routed Relative Sequence Batch Size Learning Size Heads Experts Experts Expert Size Length (Sequence) Rate 2.0B 9 1280 10 1 63 (7 activated) 0.25 2048 2048 1.08e-3 16.4B 28 2048 16 2 64 (6 activated) 0.25 4096 4608 4.2e-4 144.6B 62 4096 32 4 128 (12 activated) 0.125 4096 4608 3.0e-4\n\nTable 7 | Overview of hyper-parameters for DeepSeekMoE across various sizes. The relative expert size is in comparison to a standard FFN.\n\nB. Comparing Deepseekmoe With Larger Models\n\nComparisons among DeepSeekMoE, GShard×1.2, and GShard×1.5 are shown in Table 8. Comparisons among DeepSeekMoE, Dense×4, and Dense×16 are shown in Table 9.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 2.3B 2.8B 1.9B # Activated Expert Params N/A 0.28B 0.35B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.824 1.808 1.808 HellaSwag (Acc.) 0-shot 53.7 54.4 54.8 PIQA (Acc.) 0-shot 71.8 71.1 72.3 ARC-easy (Acc.) 0-shot 46.8 47.3 49.4 ARC-challenge (Acc.) 0-shot 31.7 34.1 34.3 RACE-middle (Acc.) 5-shot 43.7 46.4 44.0 RACE-high (Acc.) 5-shot 31.9 32.4 31.7 HumanEval (Pass@1) 0-shot 3.7 3.0 4.9 MBPP (Pass@1) 3-shot 2.4 2.6 2.2 TriviaQA (EM) 5-shot 15.2 15.7 16.6 NaturalQuestions (EM) 5-shot 4.5 4.7 5.7\n\nTable 8 | Comparison between DeepSeekMoE and larger GShard models.\n\nAt a larger scale of 13B total parameters, we also compare DeepSeekMoE with GShard×1.2 and GShard×1.5, and show results in Table 10. At a larger scale, DeepSeekMoE even outperforms GShard×1.5 distinctly.\n\nMetric # Shot Dense×4 Dense×16 DeepSeekMoE Relative Expert Size N/A 1 1 0.25 # Experts N/A 4 + 0 16 + 0 1 + 63 # Activated Experts N/A 4 + 0 16 + 0 1 + 7 # Total Expert Params N/A 0.47B 1.89B 1.89B # Activated Expert Params N/A 0.47B 1.89B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.908 1.806 1.808 HellaSwag (Acc.) 0-shot 47.6 55.1 54.8 PIQA (Acc.) 0-shot 70.0 71.9 72.3 ARC-easy (Acc.) 0-shot 43.9 51.9 49.4 ARC-challenge (Acc.) 0-shot 30.5 33.8 34.3 RACE-middle (Acc.) 5-shot 42.4 46.3 44.0 RACE-high (Acc.) 5-shot 30.7 33.0 31.7 HumanEval (Pass@1) 0-shot 1.8 4.3 4.9 MBPP (Pass@1) 3-shot 0.2 2.2 2.2 TriviaQA (EM) 5-shot 9.9 16.5 16.6 NaturalQuestions (EM) 5-shot 3.0 6.3 5.7\n\nTable 9 | Comparison between DeepSeekMoE and larger dense baselines.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 15.9B 19.8B 13.3B # Activated Expert Params N/A 2.37B 2.82B 2.05B # Training Tokens N/A 100B 100B 100B HellaSwag (Acc.) 0-shot 66.6 67.7 69.1 PIQA (Acc.) 0-shot 75.6 76.0 75.7 ARC-easy (Acc.) 0-shot 56.8 56.8 58.8 ARC-challenge (Acc.) 0-shot 39.9 37.6 38.5 RACE-middle (Acc.) 5-shot 51.6 50.6 52.4 RACE-high (Acc.) 5-shot 37.4 36.3 38.5 HumanEval (Pass@1) 0-shot 6.1 6.1 9.8 MBPP (Pass@1) 3-shot 7.0 11.6 10.6 TriviaQA (EM) 5-shot 36.5 36.7 38.2 NaturalQuestions (EM) 5-shot 12.6 12.1 13.7\n\nTable 10 | Comparison between DeepSeekMoE and larger GShard models at a larger scale.\n\nC. Training Benchmark Curves Of Deepseekmoe 16B\n\nWe present the benchmark curves during training of DeepSeekMoE 16B and DeepSeek 7B (Dense) in Figure 7 for reference.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/DeepSeekMoE_2.md"
+          },
+          "summary": "DeepSeekMoE is an innovative Mixture-of-Experts (MoE) architecture designed to enhance expert specialization in language models. It employs fine-grained expert segmentation and shared expert isolation to improve performance and reduce computational costs. Starting with a model of 2B parameters, DeepSeekMoE demonstrates comparable performance to larger models like GShard 2.9B and LLaMA2 7B, using significantly fewer computations. The architecture scales up to 16B and 145B parameters, maintaining advantages over traditional MoE architectures. DeepSeekMoE 16B achieves similar results to dense models with only 40% of the computations, and its chat model version shows adaptability in bilingual scenarios. The architecture's public release aims to contribute to advancements in large-scale language models.",
+          "headlines": {
+            "1. Introduction": [],
+            "2. Preliminaries: Mixture-Of-Experts For Transformers": [],
+            "3. Deepseekmoe Architecture": [
+              "3.1. Fine-Grained Expert Segmentation",
+              "3.2. Shared Expert Isolation",
+              "3.3. Load Balance Consideration"
+            ],
+            "4. Validation Experiments": [
+              "4.1. Experimental Setup",
+              "4.1.1. Training Data And Tokenization",
+              "4.1.2. Infrastructures",
+              "4.1.3. Hyper-Parameters",
+              "4.1.4. Evaluation Benchmarks",
+              "4.2. Evaluations",
+              "4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models",
+              "4.4. Ablation Studies",
+              "4.5. Analysis On Expert Specialization"
+            ],
+            "5. Scaling Up To Deepseekmoe 16B": [
+              "5.1. Experimental Setup",
+              "5.1.1. Training Data And Tokenization",
+              "5.1.2. Hyper-Parameters",
+              "5.1.3. Evaluation Benchmarks",
+              "5.2. Evaluations",
+              "5.2.1. Internal Comparison With Deepseek 7B",
+              "5.2.2. Comparison With Open Source Models"
+            ],
+            "6. Alignment For Deepseekmoe 16B": [
+              "6.1. Experimental Setup",
+              "6.2. Evaluations"
+            ],
+            "7. Deepseekmoe 145B Ongoing": [
+              "7.1. Experimental Setup",
+              "7.2. Evaluations"
+            ],
+            "8. Related Work": [],
+            "9. Conclusion": [],
+            "Appendices": [
+              "A. Overview Of Hyper-Parameters",
+              "B. Comparing Deepseekmoe With Larger Models",
+              "C. Training Benchmark Curves Of Deepseekmoe 16B"
+            ]
+          },
+          "summary_embedding": [
+            -0.012465757317841053,
+            -0.022808987647294998,
+            -0.018792515620589256,
+            -0.02308654971420765,
+            0.01797616109251976,
+            0.007200260180979967,
+            0.006490030791610479,
+            0.042613785713911057,
+            -0.014049487188458443,
+            0.03187054023146629,
+            -0.021486490964889526,
+            -0.023592690005898476,
+            0.009045224636793137,
+            -0.014441338367760181,
+            0.005865518003702164,
+            -0.007363531738519669,
+            -0.008743173442780972,
+            -0.005163452122360468,
+            0.018841497600078583,
+            -0.03100520372390747,
+            0.011110605672001839,
+            -0.046075135469436646,
+            -0.020017050206661224,
+            -0.021845687180757523,
+            0.016425084322690964,
+            0.037389107048511505,
+            0.022547753527760506,
+            -0.0028062239289283752,
+            -0.06844329088926315,
+            0.034678805619478226,
+            0.035168617963790894,
+            0.02052319049835205,
+            0.0005438972148112953,
+            -0.00031302153365686536,
+            -0.024833550676703453,
+            -0.028556134551763535,
+            0.03347059711813927,
+            -0.004379750229418278,
+            -0.01453930139541626,
+            0.015976088121533394,
+            0.0414055772125721,
+            0.012106560170650482,
+            -0.02754385396838188,
+            0.0014245413476601243,
+            0.015151568688452244,
+            0.0055961208418011665,
+            -0.01678428053855896,
+            -0.0022960014175623655,
+            -0.002424577483907342,
+            -0.021225256845355034,
+            -0.01400050614029169,
+            0.018825169652700424,
+            0.001296985661610961,
+            -0.008865626528859138,
+            -0.0542386956512928,
+            0.007330877240747213,
+            0.03840138763189316,
+            0.05623060464859009,
+            0.044148534536361694,
+            -0.020180322229862213,
+            0.03369917720556259,
+            -0.030025575309991837,
+            -0.02501314878463745,
+            0.011028970591723919,
+            0.003983817528933287,
+            -0.018531281501054764,
+            0.01784554310142994,
+            -0.005636938381940126,
+            -0.0006556359585374594,
+            0.01751900091767311,
+            -0.03941367194056511,
+            0.020751770585775375,
+            -0.01887415163218975,
+            0.031266435980796814,
+            0.02173139713704586,
+            0.0073798587545752525,
+            0.01872720755636692,
+            0.01678428053855896,
+            0.0036858476232737303,
+            -0.04107903689146042,
+            0.009118696674704552,
+            0.016523046419024467,
+            0.014196431264281273,
+            0.016269976273179054,
+            0.06553706526756287,
+            -0.05391215533018112,
+            -0.004485876299440861,
+            -0.05169166624546051,
+            0.0012561678886413574,
+            -0.004971608519554138,
+            -0.03670336678624153,
+            0.01048201136291027,
+            0.028425518423318863,
+            0.021372200921177864,
+            -0.01063711941242218,
+            0.024392718449234962,
+            -0.022743679583072662,
+            0.0032001156359910965,
+            -0.04062187671661377,
+            0.0019653772469609976,
+            0.004518530797213316,
+            0.016531210392713547,
+            0.0070778070949018,
+            0.004273623693734407,
+            -0.016702644526958466,
+            0.006045116577297449,
+            -0.01905374974012375,
+            0.03291547670960426,
+            -0.008465612307190895,
+            0.08653374016284943,
+            0.008751336485147476,
+            0.004738946910947561,
+            0.004281787667423487,
+            0.02398454025387764,
+            0.012131051160395145,
+            -0.00810233410447836,
+            -0.04718538001179695,
+            -0.015690363943576813,
+            0.011061624623835087,
+            -0.0383034273982048,
+            0.018335357308387756,
+            -0.008759500458836555,
+            0.01282495353370905,
+            -0.04107903689146042,
+            -0.06344719231128693,
+            0.01498013362288475,
+            0.001784758409485221,
+            -0.004065453074872494,
+            -0.0002903166168835014,
+            -0.018923133611679077,
+            0.003930754493921995,
+            -0.004200151655822992,
+            -0.010645282454788685,
+            -0.021502818912267685,
+            0.03159298002719879,
+            0.03657275065779686,
+            0.06527583301067352,
+            0.036801330745220184,
+            -0.0030164355412125587,
+            0.005775718949735165,
+            0.019690508022904396,
+            -0.020784424617886543,
+            0.003481758525595069,
+            -0.01214737817645073,
+            -0.021992631256580353,
+            -0.03820546343922615,
+            0.010971825569868088,
+            0.07379858940839767,
+            -0.029029620811343193,
+            0.013437220826745033,
+            -0.017616963014006615,
+            0.04205866530537605,
+            -0.004567512311041355,
+            0.026727497577667236,
+            -0.016669990494847298,
+            0.004059330560266972,
+            -0.025388672947883606,
+            0.01620466820895672,
+            -0.024964168667793274,
+            0.012571883387863636,
+            -0.000885746325366199,
+            -0.04222193360328674,
+            0.00011116473615402356,
+            0.026890767738223076,
+            -0.010971825569868088,
+            0.02261306345462799,
+            0.0004949158756062388,
+            -0.006302268709987402,
+            -0.05097327008843422,
+            0.038727931678295135,
+            -0.003912386484444141,
+            0.018645571544766426,
+            -0.0018959869630634785,
+            0.0163107942789793,
+            -0.0016408757073804736,
+            0.013216804713010788,
+            0.009543201886117458,
+            0.007114543113857508,
+            -0.033503253012895584,
+            0.02697240374982357,
+            -0.02504580281674862,
+            0.028049994260072708,
+            0.03435226157307625,
+            -0.0030184765346348286,
+            0.03255628049373627,
+            0.019249675795435905,
+            0.03840138763189316,
+            0.0327032245695591,
+            0.030335791409015656,
+            -0.00824927818030119,
+            0.013878053054213524,
+            0.008200296200811863,
+            0.03608293831348419,
+            0.013429056853055954,
+            0.019723162055015564,
+            -0.009730963967740536,
+            -0.014041324146091938,
+            0.048785436898469925,
+            0.007318632211536169,
+            -0.0179598331451416,
+            -0.005441013257950544,
+            -0.03510330989956856,
+            -0.002832755446434021,
+            -0.014425011351704597,
+            0.0323440283536911,
+            0.02158445306122303,
+            -0.01647406443953514,
+            0.035135965794324875,
+            0.024980494752526283,
+            -0.03751972317695618,
+            -0.005232842173427343,
+            -0.02026195637881756,
+            -0.023298801854252815,
+            -0.04215662553906441,
+            0.011624909937381744,
+            0.012833117507398129,
+            -0.013976015150547028,
+            -0.007469657808542252,
+            0.020457882434129715,
+            -0.023625344038009644,
+            -0.0018265966791659594,
+            -0.01356783788651228,
+            0.006547175347805023,
+            -0.020898714661598206,
+            -0.03667071461677551,
+            -0.01048201136291027,
+            0.025862159207463264,
+            0.012637191452085972,
+            0.04362606629729271,
+            -0.013518855907022953,
+            -0.005367541220039129,
+            0.033193036913871765,
+            0.012759645469486713,
+            0.010767736472189426,
+            0.03699725493788719,
+            -0.03027048148214817,
+            0.024621298536658287,
+            -0.020294612273573875,
+            -0.0032878739293664694,
+            0.01707816869020462,
+            -0.0030062312725931406,
+            0.01962519995868206,
+            -0.04535674303770065,
+            -0.024196794256567955,
+            0.04395261034369469,
+            -0.0007107399869710207,
+            0.016302630305290222,
+            -0.005959399044513702,
+            -0.012106560170650482,
+            0.0035899258218705654,
+            0.0027715289033949375,
+            -0.01438419334590435,
+            -0.008620720356702805,
+            0.007473739795386791,
+            -0.01962519995868206,
+            0.03660540655255318,
+            0.03794422745704651,
+            -0.004132802598178387,
+            -0.014131123200058937,
+            -0.03324201703071594,
+            0.010016689077019691,
+            -0.017878197133541107,
+            -0.023951886221766472,
+            0.04078514873981476,
+            -0.032115448266267776,
+            -0.00029312283731997013,
+            0.039478980004787445,
+            0.018808843567967415,
+            -0.014898497611284256,
+            0.00868602842092514,
+            -0.01709449477493763,
+            -0.026302991434931755,
+            -0.017159804701805115,
+            0.03356856107711792,
+            0.024800896644592285,
+            -0.010743245482444763,
+            0.010245268233120441,
+            0.005330805201083422,
+            0.02893165871500969,
+            0.024376392364501953,
+            0.00648594880476594,
+            -0.013812744058668613,
+            0.015029114671051502,
+            0.027641816064715385,
+            -0.0028021421749144793,
+            -7.66609373386018e-05,
+            -0.004010349046438932,
+            -0.007796200457960367,
+            -0.006453294772654772,
+            -0.013069860637187958,
+            -0.005089980084449053,
+            0.0009184005321003497,
+            -0.032295044511556625,
+            0.015967924147844315,
+            -0.005890008993446827,
+            0.010359558276832104,
+            -0.011535110883414745,
+            0.01288209855556488,
+            0.01722511276602745,
+            -0.013494365848600864,
+            -0.0716434046626091,
+            0.014612773433327675,
+            0.02458864450454712,
+            -0.006706364918500185,
+            -0.017943505197763443,
+            0.008481939323246479,
+            0.00016148542636074126,
+            0.038597315549850464,
+            -0.014180104248225689,
+            -0.031103165820240974,
+            0.02680913358926773,
+            -0.017437364906072617,
+            -0.015241367742419243,
+            -0.010661610402166843,
+            -0.00580837344750762,
+            0.04649963974952698,
+            0.01889047957956791,
+            0.011420820839703083,
+            -0.04042595252394676,
+            -0.007449248805642128,
+            0.015078096650540829,
+            0.014506647363305092,
+            -0.04705476388335228,
+            0.0052083516493439674,
+            0.015486273914575577,
+            0.019723162055015564,
+            0.024800896644592285,
+            0.04274440184235573,
+            -0.0021531390957534313,
+            0.01661284640431404,
+            -0.015053605660796165,
+            -0.05136512219905853,
+            -0.03340528905391693,
+            0.029943939298391342,
+            -0.017731253057718277,
+            -0.02906227484345436,
+            -0.01693122461438179,
+            -0.006151242647320032,
+            0.0017939424142241478,
+            0.007126788143068552,
+            -0.01916803978383541,
+            -0.03464614972472191,
+            -0.03970756009221077,
+            0.02277633361518383,
+            0.007983962073922157,
+            0.004269542172551155,
+            -0.042777057737112045,
+            0.0013459670590236783,
+            0.009184005670249462,
+            0.012073906138539314,
+            0.0009954441338777542,
+            -0.019984396174550056,
+            0.015494437888264656,
+            0.0032001156359910965,
+            -0.031070511788129807,
+            -0.035593122243881226,
+            -0.03595232218503952,
+            0.0023919232189655304,
+            0.003781769424676895,
+            0.06279410421848297,
+            0.012988224625587463,
+            -0.03840138763189316,
+            -0.04522612690925598,
+            0.029176564887166023,
+            -0.02052319049835205,
+            -0.016678154468536377,
+            0.0059430720284581184,
+            0.012898425571620464,
+            -0.009877908043563366,
+            0.013192313723266125,
+            0.023560035973787308,
+            0.041601505130529404,
+            -0.005987971555441618,
+            0.012914752587676048,
+            0.023037567734718323,
+            0.012955570593476295,
+            0.0362788625061512,
+            -0.00936360377818346,
+            0.0035409443080425262,
+            0.039936136454343796,
+            -0.012269831262528896,
+            0.03175625205039978,
+            -0.02189466916024685,
+            0.0020398697815835476,
+            0.02249877341091633,
+            0.027184655889868736,
+            -0.0006479826406575739,
+            0.02594379521906376,
+            -0.031543996185064316,
+            0.018368011340498924,
+            -0.014498483389616013,
+            -0.04888340085744858,
+            -0.01557607389986515,
+            0.013102514669299126,
+            0.030384771525859833,
+            0.02756018005311489,
+            -0.0041348435916006565,
+            -0.019723162055015564,
+            -0.01155143789947033,
+            -0.010596301406621933,
+            0.0018419033149257302,
+            0.03285016864538193,
+            0.010506502352654934,
+            0.010890189558267593,
+            -0.011935125105082989,
+            -0.004104230087250471,
+            0.03025415539741516,
+            -0.021192602813243866,
+            0.01650671847164631,
+            0.0041511706076562405,
+            -0.002029665280133486,
+            -0.04320156201720238,
+            -0.035266581922769547,
+            0.007845181971788406,
+            0.011747363954782486,
+            0.014874007552862167,
+            0.0368339866399765,
+            0.025094784796237946,
+            -0.01662100851535797,
+            0.030629679560661316,
+            -0.01905374974012375,
+            0.027984686195850372,
+            0.039054471999406815,
+            0.011624909937381744,
+            -0.018351683393120766,
+            0.012237177230417728,
+            0.002424577483907342,
+            -0.025127438828349113,
+            0.01680060662329197,
+            -0.032735876739025116,
+            0.01738838478922844,
+            -0.02354370802640915,
+            0.025143766775727272,
+            -0.02579685114324093,
+            -0.009290131740272045,
+            -0.004428731743246317,
+            -0.006130834110081196,
+            0.04888340085744858,
+            -0.026727497577667236,
+            -0.0142454132437706,
+            -0.0024694770108908415,
+            -0.0024531499948352575,
+            -0.008718682453036308,
+            0.010000362060964108,
+            -0.045781247317790985,
+            -0.01844964735209942,
+            -0.04016471654176712,
+            0.007355368230491877,
+            0.005412440747022629,
+            -0.030907239764928818,
+            -0.05694899708032608,
+            0.025421326979994774,
+            -0.06331657618284225,
+            -0.019886434078216553,
+            0.051626358181238174,
+            -0.018400665372610092,
+            -0.008212542161345482,
+            0.009641164913773537,
+            0.006375740747898817,
+            -0.029290854930877686,
+            0.0041348435916006565,
+            0.02504580281674862,
+            0.04418119043111801,
+            -0.0045797573402523994,
+            -0.02865409664809704,
+            0.0028674507047981024,
+            -0.027788760140538216,
+            -0.013461710885167122,
+            0.005261414684355259,
+            0.016449574381113052,
+            0.024539662525057793,
+            0.007628847379237413,
+            0.03585435822606087,
+            0.007294141221791506,
+            0.006318595726042986,
+            0.044279150664806366,
+            -0.025666235014796257,
+            -0.027380581945180893,
+            0.02519274689257145,
+            0.04045860469341278,
+            0.023151857778429985,
+            -0.018253721296787262,
+            -0.0142454132437706,
+            0.0053063142113387585,
+            0.0009296254720538855,
+            -0.022139575332403183,
+            0.002808264922350645,
+            -0.013902543112635612,
+            0.005143043119460344,
+            0.03892385587096214,
+            -0.001205145614221692,
+            0.014196431264281273,
+            0.007302304729819298,
+            0.006845145486295223,
+            0.004334850702434778,
+            -0.011731036007404327,
+            -0.023592690005898476,
+            0.00032220553839579225,
+            0.034972693771123886,
+            0.015404638834297657,
+            -0.030515389516949654,
+            -0.00995954405516386,
+            0.034711457788944244,
+            -0.00795130804181099,
+            -0.009118696674704552,
+            -0.009771781973540783,
+            0.003385836724191904,
+            0.02638462744653225,
+            -0.01606588624417782,
+            0.006649219896644354,
+            0.03716052696108818,
+            -0.019870106130838394,
+            0.005543057806789875,
+            -0.025405000895261765,
+            0.0018061877926811576,
+            0.02442537248134613,
+            -0.006216551177203655,
+            0.028098974376916885,
+            5.6315813708351925e-05,
+            0.03025415539741516,
+            -0.004677720367908478,
+            0.008432958275079727,
+            -0.020327266305685043,
+            0.03987082839012146,
+            -0.017633290961384773,
+            0.03402572125196457,
+            0.02067013457417488,
+            -0.01258004643023014,
+            0.04783846437931061,
+            -0.036801330745220184,
+            -0.020784424617886543,
+            0.00584919098764658,
+            0.012743317522108555,
+            0.0028919412288814783,
+            -0.0191517136991024,
+            -0.018400665372610092,
+            -0.010996315628290176,
+            0.02380494214594364,
+            -0.022547753527760506,
+            -0.016865916550159454,
+            -0.007441085297614336,
+            -0.01890680566430092,
+            -0.009673818945884705,
+            -0.007988044060766697,
+            -0.01709449477493763,
+            -0.016547536477446556,
+            0.044573038816452026,
+            -0.06334923207759857,
+            0.016996532678604126,
+            0.017159804701805115,
+            0.011477965861558914,
+            -0.034678805619478226,
+            -0.019510909914970398,
+            -0.017633290961384773,
+            -0.002347023691982031,
+            -0.030743969604372978,
+            0.001702102366834879,
+            0.01725776679813862,
+            0.01138000376522541,
+            -0.05022222548723221,
+            0.012335140258073807,
+            0.02772345207631588,
+            -0.012065742164850235,
+            0.0014949520118534565,
+            -0.023135529831051826,
+            -0.017176130786538124,
+            0.0031776658724993467,
+            0.022123249247670174,
+            0.005975726060569286,
+            -0.008188051171600819,
+            -0.016865916550159454,
+            -0.005241005681455135,
+            -0.006045116577297449,
+            -0.002359268954023719,
+            0.005094061605632305,
+            0.011780017986893654,
+            0.019690508022904396,
+            -0.0024633544962853193,
+            -0.004436895251274109,
+            -0.006396149750798941,
+            0.027364253997802734,
+            -0.001450052484869957,
+            -0.008416631259024143,
+            0.050907962024211884,
+            0.03556047007441521,
+            0.006094098091125488,
+            -0.01108611561357975,
+            -0.03990348428487778,
+            0.010661610402166843,
+            -0.022711025550961494,
+            0.0401320643723011,
+            0.01905374974012375,
+            -0.008298259228467941,
+            -0.028866350650787354,
+            0.010294250212609768,
+            -0.02290695160627365,
+            -0.009975871071219444,
+            0.009469729848206043,
+            0.00726556871086359,
+            -0.03263791650533676,
+            -0.013543346896767616,
+            0.03510330989956856,
+            0.011437147855758667,
+            -0.006841063499450684,
+            -0.036050282418727875,
+            0.008588065393269062,
+            0.01646590232849121,
+            0.011869817040860653,
+            -0.019233347848057747,
+            0.004449140280485153,
+            -0.012041252106428146,
+            0.017470018938183784,
+            0.008849299512803555,
+            -0.017796561121940613,
+            0.018237393349409103,
+            0.004710374400019646,
+            -0.008490103296935558,
+            -0.003912386484444141,
+            -0.0163107942789793,
+            -0.02006603218615055,
+            0.03840138763189316,
+            0.024490682408213615,
+            0.026335645467042923,
+            0.005457340274006128,
+            0.02008235827088356,
+            0.03227871656417847,
+            0.011282040737569332,
+            -0.03134807199239731,
+            0.027935704216361046,
+            0.016996532678604126,
+            -0.04604247957468033,
+            0.021551799029111862,
+            -0.02218855731189251,
+            -0.028670424595475197,
+            0.009649328887462616,
+            -0.0017367975087836385,
+            0.011192241683602333,
+            0.0034837995190173388,
+            -0.003642988856881857,
+            -0.03307874873280525,
+            -0.006236960180103779,
+            0.015771998092532158,
+            0.015502601861953735,
+            -0.0048695639707148075,
+            -0.003296037670224905,
+            -0.011886144056916237,
+            0.00487364549189806,
+            0.009796272963285446,
+            0.039054471999406815,
+            0.001381682581268251,
+            -0.004751192405819893,
+            -0.0018765984568744898,
+            0.031854212284088135,
+            0.0271193478256464,
+            -0.017274094745516777,
+            -0.026286665350198746,
+            0.00580837344750762,
+            -0.014253576286137104,
+            0.026156047359108925,
+            -0.00487364549189806,
+            0.011877980083227158,
+            0.008718682453036308,
+            -0.0007908449042588472,
+            -0.001373519073240459,
+            0.025584599003195763,
+            -0.02710302174091339,
+            -0.021764053031802177,
+            0.031233781948685646,
+            -0.025731543079018593,
+            0.028735732659697533,
+            0.013061696663498878,
+            -0.01557607389986515,
+            -0.015225040726363659,
+            -0.010302413254976273,
+            0.009943217039108276,
+            -0.012277995236217976,
+            0.023853924125432968,
+            0.003608293831348419,
+            0.006392067763954401,
+            0.0003584313380997628,
+            0.02770712412893772,
+            0.024833550676703453,
+            0.007820690982043743,
+            -0.018074123188853264,
+            -0.0016816934803500772,
+            -0.02320083975791931,
+            -0.01680060662329197,
+            0.023772288113832474,
+            -0.018090449273586273,
+            -0.03268689662218094,
+            -0.028556134551763535,
+            0.012114724144339561,
+            0.007392104249447584,
+            0.010816717520356178,
+            0.03461349755525589,
+            -0.030384771525859833,
+            -0.0344502255320549,
+            0.01766594499349594,
+            0.026270337402820587,
+            -0.005551221314817667,
+            -0.021535472944378853,
+            0.03085825778543949,
+            -0.005828781984746456,
+            0.017600636929273605,
+            0.0404912605881691,
+            -0.0016225076979026198,
+            0.010228941217064857,
+            -0.042026009410619736,
+            -0.014057651162147522,
+            -0.0066614653915166855,
+            0.006261450704187155,
+            0.034417569637298584,
+            0.01048201136291027,
+            0.01535565685480833,
+            0.03448288142681122,
+            -0.021192602813243866,
+            -0.0002903166168835014,
+            -0.0022449791431427,
+            -0.02818061038851738,
+            0.03347059711813927,
+            0.025715215131640434,
+            0.011918798089027405,
+            0.019706835970282555,
+            -0.03794422745704651,
+            -0.013037205673754215,
+            0.027070365846157074,
+            0.015053605660796165,
+            0.01273515447974205,
+            0.016694480553269386,
+            0.011249386705458164,
+            0.00868602842092514,
+            -0.021633435040712357,
+            -0.05169166624546051,
+            0.01214737817645073,
+            0.01993541419506073,
+            -0.016449574381113052,
+            0.01057181041687727,
+            0.024980494752526283,
+            0.007620683871209621,
+            -0.007045152597129345,
+            0.012808626517653465,
+            -0.02682545967400074,
+            0.002259265398606658,
+            0.0019010890973731875,
+            -0.014482156373560429,
+            0.038270771503448486,
+            -0.016367938369512558,
+            0.011363675817847252,
+            -0.018204739317297935,
+            0.023870250210165977,
+            0.00011531029304023832,
+            0.0007964573451317847,
+            0.014963806606829166,
+            -0.011347348801791668,
+            -0.04235255345702171,
+            0.0021613026037812233,
+            0.03342161700129509,
+            -0.015200549736618996,
+            0.026449935510754585,
+            -0.0020674217958003283,
+            0.032458316534757614,
+            0.042777057737112045,
+            0.017731253057718277,
+            0.0003497575526125729,
+            0.025682561099529266,
+            0.0036144163459539413,
+            0.0073880222626030445,
+            0.012400448322296143,
+            0.009322785772383213,
+            0.007245160173624754,
+            0.030809277668595314,
+            -0.010261595249176025,
+            0.02173139713704586,
+            -0.04760988429188728,
+            -0.014629100449383259,
+            0.012024925090372562,
+            -0.0013316808035597205,
+            -0.012449429370462894,
+            -0.033029764890670776,
+            0.012563719414174557,
+            -0.03503800183534622,
+            0.022074267268180847,
+            -0.0004959363141097128,
+            0.04333217814564705,
+            -0.02187834307551384,
+            -0.005518566817045212,
+            0.016694480553269386,
+            0.0016510800924152136,
+            0.023282473906874657,
+            -0.0012153501156717539,
+            -0.016416920349001884,
+            -0.0067798369564116,
+            -0.009257477708160877,
+            0.004138925112783909,
+            0.04375668615102768,
+            -0.003722583642229438,
+            -0.0001682458823779598,
+            0.01409030519425869,
+            -0.011959616094827652,
+            -0.03250729665160179,
+            0.045193471014499664,
+            -0.0030837850645184517,
+            0.02484987862408161,
+            0.04979771748185158,
+            -0.06410028040409088,
+            -0.019543563947081566,
+            0.013061696663498878,
+            -0.010441194288432598,
+            0.011363675817847252,
+            0.005547139327973127,
+            0.016114868223667145,
+            0.017894525080919266,
+            -0.005485912784934044,
+            0.01619650423526764,
+            0.005106307100504637,
+            0.028098974376916885,
+            -0.02336410991847515,
+            -0.03296445682644844,
+            -0.02636830136179924,
+            -0.017747581005096436,
+            0.0014480114914476871,
+            -0.015796490013599396,
+            -3.563585551091819e-06,
+            -0.0022204883862286806,
+            -0.04388730227947235,
+            -0.025274382904171944,
+            0.040687184780836105,
+            0.006706364918500185,
+            0.02458864450454712,
+            0.0077268099412322044,
+            0.03418898954987526,
+            0.005885927006602287,
+            -0.009037061594426632,
+            0.030058229342103004,
+            -0.020049704238772392,
+            0.03833607956767082,
+            0.022106921300292015,
+            -0.002832755446434021,
+            0.005865518003702164,
+            0.031086837872862816,
+            -0.033797141164541245,
+            0.008086007088422775,
+            -0.034384917467832565,
+            0.0046695563942193985,
+            -0.008620720356702805,
+            -0.015094423666596413,
+            -0.009526874870061874,
+            -0.02504580281674862,
+            0.011216731742024422,
+            -0.021535472944378853,
+            0.01590261608362198,
+            0.010416703298687935,
+            -0.002916431985795498,
+            0.03448288142681122,
+            -0.014204595237970352,
+            -0.0027796924114227295,
+            0.0024531499948352575,
+            -0.027070365846157074,
+            -0.006343086715787649,
+            -0.003377673216164112,
+            -0.03689929470419884,
+            -0.004502203781157732,
+            0.010988152585923672,
+            -0.00024184548237826675,
+            0.016898570582270622,
+            0.02787039615213871,
+            0.0005092020728625357,
+            -0.022302847355604172,
+            0.002340900944545865,
+            -0.04199335351586342,
+            0.0011877981014549732,
+            0.01483318954706192,
+            -0.017600636929273605,
+            -0.008612556383013725,
+            0.033046092838048935,
+            0.023298801854252815,
+            -0.009396257810294628,
+            -0.035005345940589905,
+            0.017796561121940613,
+            -0.011853490024805069,
+            0.027511198073625565,
+            -0.0002630197268445045,
+            0.019298657774925232,
+            -0.0076941559091210365,
+            0.04493223875761032,
+            0.001369437319226563,
+            0.008645210415124893,
+            -0.021959977224469185,
+            0.007020662073045969,
+            -0.020278284326195717,
+            -0.002406209474429488,
+            -0.006171651650220156,
+            -0.012751481495797634,
+            -0.010669773444533348,
+            0.0032735876739025116,
+            0.007730891928076744,
+            -0.0176822729408741,
+            0.0055961208418011665,
+            -0.015959760174155235,
+            -0.02741323597729206,
+            0.007543129846453667,
+            -0.05965929850935936,
+            -0.0179598331451416,
+            -0.03373183310031891,
+            -0.009477893821895123,
+            0.02008235827088356,
+            0.03510330989956856,
+            -0.012759645469486713,
+            0.00497977202758193,
+            -0.014898497611284256,
+            0.004477712791413069,
+            0.03490738570690155,
+            0.044442422688007355,
+            -0.014759717509150505,
+            0.003381754970178008,
+            -0.010955498553812504,
+            0.033029764890670776,
+            -0.03347059711813927,
+            0.0030062312725931406,
+            0.031086837872862816,
+            -0.005057325586676598,
+            0.007959471084177494,
+            0.011624909937381744,
+            0.020474210381507874,
+            -0.019396619871258736,
+            0.00877582747489214,
+            0.002100075827911496,
+            0.019233347848057747,
+            0.02171507105231285,
+            0.020343592390418053,
+            -0.03539719805121422,
+            0.028409190475940704,
+            0.018384337425231934,
+            0.02444170042872429,
+            0.044442422688007355,
+            -0.03974021226167679,
+            -0.004147088620811701,
+            -0.0032225653994828463,
+            -0.02997659333050251,
+            0.02008235827088356,
+            -0.007530884351581335,
+            -0.01841699331998825,
+            -0.007069643586874008,
+            0.03327467292547226,
+            0.014571955427527428,
+            -0.05051611363887787,
+            0.01980479806661606,
+            0.01740471087396145,
+            0.02055584453046322,
+            0.024474354460835457,
+            -0.01123305968940258,
+            -0.0019072118448093534,
+            0.008996243588626385,
+            0.006804327480494976,
+            0.019266001880168915,
+            0.015257694758474827,
+            0.002012317767366767,
+            -0.0022082431241869926,
+            -0.011837163008749485,
+            0.005224678665399551,
+            -0.009077879600226879,
+            0.010808554477989674,
+            -0.02458864450454712,
+            -0.009698309935629368,
+            0.021388528868556023,
+            0.0031164393294602633,
+            -0.013665799982845783,
+            -0.03729114308953285,
+            -0.045944519340991974,
+            0.0033837957307696342,
+            0.004543021321296692,
+            -0.05365091934800148,
+            0.005322641227394342,
+            -0.026547899469733238,
+            -0.002108239568769932,
+            -0.005502239800989628,
+            0.0019214981002733111,
+            0.005636938381940126,
+            0.0167679525911808,
+            -0.013494365848600864,
+            0.02248244546353817,
+            -0.015502601861953735,
+            0.03165828809142113,
+            0.0055961208418011665,
+            0.007375776767730713,
+            -0.0027796924114227295,
+            0.05590406060218811,
+            -0.003712379140779376,
+            -0.03699725493788719,
+            0.023282473906874657,
+            -0.01311067771166563,
+            -0.0001747512142173946,
+            0.030090883374214172,
+            0.00884113647043705,
+            -0.015257694758474827,
+            0.013437220826745033,
+            -0.030613351613283157,
+            -0.03249097242951393,
+            0.010996315628290176,
+            -0.013755599036812782,
+            -0.0015776080545037985,
+            -0.04568328335881233,
+            -0.009339112788438797,
+            0.004269542172551155,
+            -0.018335357308387756,
+            0.010775899514555931,
+            0.01631895825266838,
+            0.013992343097925186,
+            -0.024245774373412132,
+            0.009053388610482216,
+            0.006865554489195347,
+            0.013453547842800617,
+            0.0007331897504627705,
+            0.014049487188458443,
+            -0.008677864447236061,
+            -0.011306531727313995,
+            -0.018319029361009598,
+            0.0032450153958052397,
+            -0.026792805641889572,
+            0.005155288614332676,
+            -0.003155216109007597,
+            -0.03771565109491348,
+            -0.06521052122116089,
+            -0.0344502255320549,
+            0.016294466331601143,
+            0.03396041318774223,
+            -0.009045224636793137,
+            0.0035144127905368805,
+            -0.00023648813657928258,
+            -0.01093917153775692,
+            0.024523336440324783,
+            -0.005930826533585787,
+            -0.0004829256213270128,
+            0.0035389035474509,
+            0.02050686441361904,
+            -0.009722800925374031,
+            0.005845109466463327,
+            -0.051952898502349854,
+            -0.01586179807782173,
+            0.01812310516834259,
+            0.006306350696831942,
+            0.014629100449383259,
+            -0.009690145961940289,
+            0.01693122461438179,
+            -0.030074557289481163,
+            -0.028147956356406212,
+            0.04130761697888374,
+            0.008277850225567818,
+            0.011486129835247993,
+            -0.003055212553590536,
+            -0.01680060662329197,
+            0.02504580281674862,
+            -0.0020633398089557886,
+            -0.006461458280682564,
+            -0.00817988719791174,
+            0.015820980072021484,
+            0.044279150664806366,
+            0.003255219664424658,
+            0.0009821783751249313,
+            0.017045514658093452,
+            -0.002663361607119441,
+            -0.011575928889214993,
+            0.007677828427404165,
+            0.013592327944934368,
+            -0.016384266316890717,
+            0.010375885292887688,
+            -0.025862159207463264,
+            0.0332583449780941,
+            0.03131541982293129,
+            0.011575928889214993,
+            -0.015045441687107086,
+            0.006130834110081196,
+            0.005330805201083422,
+            0.008086007088422775,
+            -0.001369437319226563,
+            -0.0017133272485807538,
+            0.020719116553664207,
+            -0.03276853263378143,
+            -0.01693122461438179,
+            -0.005347132217139006,
+            0.003932795487344265,
+            0.005277741700410843,
+            0.03396041318774223,
+            -0.0033266509417444468,
+            0.017927179113030434,
+            -0.011657563969492912,
+            0.010579974390566349,
+            0.021764053031802177,
+            0.037095218896865845,
+            -0.024996822699904442,
+            -0.029862305149435997,
+            0.012914752587676048,
+            -0.001296985661610961,
+            0.014065815135836601,
+            0.0010173837654292583,
+            0.014139287173748016,
+            0.0029409227427095175,
+            0.060475654900074005,
+            -0.017763907089829445,
+            0.004628738854080439,
+            0.0004035860183648765,
+            -0.01858026348054409,
+            -0.0013531101867556572,
+            0.004261378664523363,
+            -0.010979988612234592,
+            0.03608293831348419,
+            0.002344982698559761,
+            0.039185091853141785,
+            0.013918871060013771,
+            0.009290131740272045,
+            0.0015143405180424452,
+            -0.013592327944934368,
+            0.008355404250323772,
+            0.0015592400450259447,
+            -0.001132694073021412,
+            0.009461566805839539,
+            0.019298657774925232,
+            0.009714636951684952,
+            0.025274382904171944,
+            0.009788108989596367,
+            0.012857607565820217,
+            -0.029470453038811684,
+            0.03510330989956856,
+            -0.0006224715034477413,
+            0.012996388599276543,
+            -0.02040890045464039,
+            -0.003136848099529743,
+            -0.0008240094175562263,
+            0.015649545937776566,
+            -0.015616890974342823,
+            -0.001201063860207796,
+            -0.01297189760953188,
+            0.040099408477544785,
+            -0.018988441675901413,
+            0.013976015150547028,
+            0.009673818945884705,
+            -0.02865409664809704,
+            -0.007596192881464958,
+            0.01995174214243889,
+            0.014294394291937351,
+            -0.0053144777193665504,
+            -0.020490536466240883,
+            0.005832863971590996,
+            0.015380147844552994,
+            -0.028572462499141693,
+            1.3584674888988957e-05,
+            0.014751553535461426,
+            -0.0164903923869133,
+            -0.00025409081717953086,
+            -0.004702210891991854,
+            0.003975654020905495,
+            -0.009265640750527382,
+            0.007028825581073761,
+            0.016057724133133888,
+            -0.024507008492946625,
+            -0.020049704238772392,
+            0.005722655914723873,
+            0.006571666337549686,
+            0.01444950234144926,
+            0.007739055436104536,
+            0.004355259705334902,
+            -0.020033378154039383,
+            -0.01799248717725277,
+            0.0032674651592969894,
+            -0.01649855636060238,
+            -0.0048614004626870155,
+            0.0020531355403363705,
+            -0.019363965839147568,
+            0.01875986158847809,
+            -0.010743245482444763,
+            0.00017538899555802345,
+            -0.003226647386327386,
+            -0.020800752565264702,
+            0.006204306147992611,
+            -0.004865481983870268,
+            0.019870106130838394,
+            0.003555230563506484,
+            0.02785406820476055,
+            0.02310287579894066,
+            -0.004428731743246317,
+            0.017600636929273605,
+            -0.016155686229467392,
+            0.007983962073922157,
+            -0.031135819852352142,
+            -0.001782717532478273,
+            -0.010408539324998856,
+            0.0036103345919400454,
+            -0.015943434089422226,
+            -0.009126860648393631,
+            -0.0048777274787425995,
+            -0.0006678812787868083,
+            -0.004820582456886768,
+            0.0004275664687156677,
+            0.0027817331720143557,
+            0.02231917530298233,
+            -0.013208640739321709,
+            -0.02803366631269455,
+            -0.017437364906072617,
+            0.006045116577297449,
+            0.019396619871258736,
+            -0.026025431230664253,
+            -0.009184005670249462,
+            -0.005012426059693098,
+            -0.004326687194406986,
+            -0.021404854953289032,
+            -0.015167895704507828,
+            -0.005632856860756874,
+            -0.008045189082622528,
+            -0.010522829368710518,
+            0.0046532293781638145,
+            -0.0035715578123927116,
+            -0.011739199981093407,
+            0.021666089072823524,
+            0.018057795241475105,
+            -0.019837452098727226,
+            -0.019837452098727226,
+            0.013641309924423695,
+            -0.006159406621009111,
+            0.0018878233386203647,
+            0.02008235827088356,
+            0.00029286774224601686,
+            0.005824700463563204,
+            0.020882388576865196,
+            -0.0035225762985646725,
+            -0.01797616109251976,
+            0.003577680326998234,
+            -0.0018888438353314996,
+            -0.012122887186706066,
+            0.003212361130863428,
+            0.02773977816104889,
+            0.0008076822850853205,
+            0.023429417982697487,
+            -0.03056436963379383,
+            -0.011412657797336578,
+            0.006061443593353033,
+            0.02369065210223198,
+            0.00211844383738935,
+            -0.0021694661118090153,
+            0.020131340250372887,
+            0.03941367194056511,
+            0.005775718949735165,
+            0.0015755671774968505,
+            -0.04003410041332245,
+            0.01027792226523161,
+            0.009208496659994125,
+            0.024621298536658287,
+            0.008339077234268188,
+            0.0014653591206297278,
+            -0.012906589545309544,
+            -0.009755454957485199,
+            -0.0017939424142241478,
+            0.014702572487294674,
+            -0.023870250210165977,
+            0.0024755997583270073,
+            -0.03157665207982063,
+            0.005094061605632305,
+            0.0021959978621453047,
+            -0.009559528902173042,
+            0.0335359051823616,
+            0.00980443600565195,
+            -0.02065380848944187,
+            0.015151568688452244,
+            0.005767555441707373,
+            0.007330877240747213,
+            0.022449791431427002,
+            -0.008228869177401066,
+            0.030450081452727318,
+            -0.005049162078648806,
+            0.01213921420276165,
+            0.02742956392467022,
+            0.032588932663202286,
+            -0.013102514669299126,
+            -0.003959327004849911,
+            0.0006097159348428249,
+            -0.0002561317232903093,
+            0.0007995187188498676,
+            -0.008204378187656403,
+            0.0037389106582850218,
+            -0.014800535514950752,
+            0.012710663489997387,
+            -0.019543563947081566,
+            -0.0034286954905837774,
+            0.0038532004691660404,
+            0.01991908811032772,
+            0.0016265894519165158,
+            -0.033633869141340256,
+            0.02055584453046322,
+            -0.015420965850353241,
+            0.0135596739128232,
+            0.001290863030590117,
+            -0.011110605672001839,
+            -0.00029210239881649613,
+            0.020588500425219536,
+            0.010947334580123425,
+            -0.007845181971788406,
+            -0.0135596739128232,
+            -0.028376536443829536,
+            -0.021535472944378853,
+            0.00823703221976757,
+            0.009445239789783955,
+            -0.0028837777208536863,
+            0.0021306893322616816,
+            -0.011869817040860653,
+            0.0006520643946714699,
+            -0.0008122742874547839,
+            0.01740471087396145,
+            -0.0335359051823616,
+            0.021943651139736176,
+            -0.019527235999703407,
+            -0.019543563947081566,
+            0.017045514658093452,
+            0.008049271069467068,
+            0.009445239789783955,
+            0.0033552234526723623,
+            -0.01995174214243889,
+            -0.012326976284384727,
+            0.0338951013982296,
+            -0.01138000376522541,
+            0.006298186723142862,
+            0.004408322740346193,
+            0.012914752587676048,
+            -0.008914608508348465,
+            -0.011926962062716484,
+            0.01784554310142994,
+            -0.009861581027507782,
+            0.013951525092124939,
+            0.0010010566329583526,
+            -0.0254703089594841,
+            0.02739690989255905,
+            -0.019233347848057747,
+            0.03487472981214523,
+            -0.021617108955979347,
+            0.00711862463504076,
+            0.0066614653915166855,
+            0.008792154490947723,
+            -0.006690037902444601,
+            -0.013224967755377293,
+            0.0012786176521331072,
+            0.03807484731078148,
+            -0.02623768337070942,
+            0.024180466309189796,
+            0.022400809451937675,
+            -0.00040766780148260295,
+            0.02065380848944187,
+            -0.008367649279534817,
+            -0.026711169630289078,
+            0.0179598331451416,
+            -0.012596373446285725,
+            -0.026123393326997757,
+            0.013257622718811035,
+            -0.022678371518850327,
+            -0.020278284326195717,
+            -0.026302991434931755,
+            0.004947117529809475,
+            -0.0025694805663079023,
+            -0.005151206627488136,
+            0.009004407562315464,
+            0.02561725303530693,
+            0.020915042608976364,
+            0.019576217979192734,
+            0.02111096680164337,
+            0.01161674689501524,
+            0.022449791431427002,
+            -0.007559456862509251,
+            0.04401791840791702,
+            -0.004947117529809475,
+            0.004632820375263691,
+            -0.0015980169409886003,
+            -0.015657709911465645,
+            0.009771781973540783,
+            0.0119106350466609,
+            0.0029572497587651014,
+            -0.029176564887166023,
+            -0.009061552584171295,
+            -0.03660540655255318,
+            0.005053244065493345,
+            0.0020664012990891933,
+            0.0176822729408741,
+            0.032115448266267776,
+            -0.005865518003702164,
+            0.004592002835124731,
+            -0.005102225113660097,
+            -0.013927034102380276,
+            -0.009420748800039291,
+            -0.02739690989255905,
+            -0.025437654927372932,
+            -0.007653337903320789,
+            -0.013739272020757198,
+            -0.030368445441126823,
+            -0.011722872965037823,
+            0.01551892887800932,
+            -0.007645174395292997,
+            0.011918798089027405,
+            -0.016702644526958466,
+            0.01385356206446886,
+            -0.004110352601855993,
+            -0.004763437435030937,
+            -0.004530776292085648,
+            -0.017600636929273605,
+            -0.006420640274882317,
+            0.014367866329848766,
+            -0.009477893821895123,
+            -0.01237595733255148,
+            -0.02215590327978134,
+            -0.0015459742862731218,
+            -0.01574750803411007,
+            0.003489922033622861,
+            -0.014596446417272091,
+            -0.006771673448383808,
+            -0.003908304497599602,
+            -0.018514955416321754,
+            0.01766594499349594,
+            0.014816862531006336,
+            -0.019380291923880577,
+            -0.0090697156265378,
+            0.02638462744653225,
+            0.042777057737112045,
+            -0.03218075633049011,
+            -0.008200296200811863,
+            0.0020378287881612778,
+            0.018090449273586273,
+            0.010171796195209026,
+            0.016702644526958466,
+            -0.024653952568769455,
+            0.0022286521270871162,
+            -0.021649762988090515,
+            -0.001220452249981463,
+            -0.023739634081721306,
+            0.010775899514555931,
+            0.025715215131640434,
+            -0.010971825569868088,
+            0.025976449251174927,
+            0.01634344831109047,
+            -0.01812310516834259,
+            0.012016761116683483,
+            0.007183933164924383,
+            0.014645427465438843,
+            -0.019755816087126732,
+            0.0167679525911808,
+            -0.015796490013599396,
+            0.031233781948685646,
+            0.00892277155071497,
+            -0.04829562455415726,
+            -0.03503800183534622,
+            -0.013388238847255707,
+            0.0013388239312916994,
+            -0.010702427476644516,
+            0.015037278644740582,
+            0.015967924147844315,
+            0.013959688134491444,
+            -0.019380291923880577,
+            -0.015820980072021484,
+            0.011984107084572315,
+            0.010065670125186443,
+            0.027462217956781387,
+            -0.006649219896644354,
+            -0.004212397150695324,
+            -0.00960034690797329,
+            -0.02249877341091633,
+            -0.016098542138934135,
+            0.0209803506731987,
+            -0.018041469156742096,
+            0.015935270115733147,
+            -0.026139721274375916,
+            -0.012449429370462894,
+            0.017453692853450775,
+            -0.05211617052555084,
+            -0.01453113742172718,
+            0.0007612520130351186,
+            0.03012353740632534,
+            0.004751192405819893,
+            0.0035184945445507765,
+            0.01841699331998825,
+            -0.01993541419506073,
+            -0.0039776950143277645,
+            -0.02099667675793171,
+            0.006832899991422892,
+            -0.011853490024805069,
+            -0.013984179124236107,
+            0.014482156373560429,
+            -0.013845399022102356,
+            -0.01601690612733364,
+            0.0034368589986115694,
+            0.007183933164924383,
+            -0.021094640716910362,
+            0.011951452121138573,
+            0.003945040516555309,
+            0.012171869166195393,
+            0.012171869166195393,
+            0.01199227012693882,
+            0.021551799029111862,
+            0.02295593172311783,
+            -0.007334959227591753,
+            -0.004767519421875477,
+            -0.01004117913544178,
+            -0.0015398516552522779,
+            0.008898280560970306,
+            0.0033797139767557383,
+            -0.002989904023706913,
+            0.007837017998099327,
+            0.010033016093075275,
+            -0.01498013362288475,
+            0.033193036913871765,
+            -0.0035878848284482956,
+            -0.04134026914834976,
+            -0.003238892648369074,
+            -0.03800953924655914,
+            0.020033378154039383,
+            -0.023298801854252815,
+            -0.016996532678604126,
+            0.0032776694279164076,
+            -0.00629410520195961,
+            -0.019902760162949562,
+            0.013608654960989952,
+            0.011404493823647499,
+            0.00258580781519413,
+            0.02021297626197338,
+            0.012269831262528896,
+            -0.005134879611432552,
+            0.026172375306487083,
+            -0.014718899503350258,
+            -0.008751336485147476,
+            0.015567909926176071,
+            0.006853308994323015,
+            0.0022347746416926384,
+            0.0011031010653823614,
+            -0.018825169652700424,
+            0.005098143592476845,
+            0.0164903923869133,
+            0.007216587662696838,
+            -0.000566347036510706,
+            0.014049487188458443,
+            0.012514738366007805,
+            -0.015135240741074085,
+            -0.0002213090192526579,
+            -0.026564225554466248,
+            0.0032817514147609472,
+            0.004567512311041355,
+            0.007506393827497959,
+            -0.006771673448383808,
+            0.006281859707087278,
+            0.00658799335360527,
+            0.014759717509150505,
+            -0.010155469179153442,
+            -0.016980206593871117,
+            -0.002510294783860445,
+            0.004285869188606739,
+            0.016694480553269386,
+            0.03157665207982063,
+            -0.010310577228665352,
+            -0.005734901409596205,
+            -0.0386626236140728,
+            0.02070278860628605,
+            0.03332365304231644,
+            0.011175914667546749,
+            -0.028213264420628548,
+            0.012302485294640064,
+            0.0006515541463159025,
+            0.0019276207312941551,
+            -0.026302991434931755,
+            0.02787039615213871,
+            0.00974729098379612,
+            0.0066614653915166855,
+            0.017861871048808098,
+            0.0022306928876787424,
+            0.0029266364872455597,
+            -0.016302630305290222,
+            0.01018812321126461,
+            0.009086042642593384,
+            -0.017306748777627945,
+            0.0009745250572450459,
+            0.02324981987476349,
+            -0.01423724927008152,
+            -0.003208279376849532,
+            0.0018133309204131365,
+            0.02651524543762207,
+            -0.013820908032357693,
+            -0.014294394291937351,
+            0.0005714492290280759,
+            0.01751900091767311,
+            -0.0007979880319908261,
+            0.012098397128283978,
+            0.04898136481642723,
+            -0.012424939312040806,
+            0.019657853990793228,
+            0.005461421795189381,
+            -0.015290348790585995,
+            -0.032605260610580444,
+            -0.004922627005726099,
+            -0.01830270327627659,
+            0.011200404725968838,
+            0.014343376271426678,
+            -0.015763835981488228,
+            0.0007525782566517591,
+            -0.03144603595137596,
+            0.007743136957287788,
+            0.014735226519405842,
+            0.018776189535856247,
+            -0.002440904499962926,
+            0.006134915631264448,
+            0.027674470096826553,
+            0.02458864450454712,
+            -0.007392104249447584,
+            0.013804581016302109,
+            0.02320083975791931,
+            -0.025062130764126778,
+            -0.003069498809054494,
+            0.006820654962211847,
+            0.004104230087250471,
+            0.0053063142113387585,
+            -0.016963878646492958,
+            0.03915243595838547,
+            -0.005922663025557995,
+            -0.01506176870316267,
+            0.0030266400426626205,
+            -0.009028897620737553,
+            0.0063390047289431095,
+            -0.012563719414174557,
+            0.012637191452085972,
+            -0.008114579133689404,
+            0.00022883480414748192,
+            -0.0059430720284581184,
+            -0.011110605672001839,
+            -0.012653518468141556,
+            -0.0007949267164804041,
+            -0.016531210392713547,
+            0.007432921789586544,
+            0.00981259997934103,
+            0.005045080557465553,
+            0.01025343220680952,
+            -0.0036797248758375645,
+            -0.003136848099529743,
+            -0.016751626506447792,
+            -0.009984034113585949,
+            0.0073716952465474606,
+            0.01872720755636692,
+            -0.028588788583874702,
+            -0.018547609448432922,
+            0.009290131740272045,
+            0.0022898786701261997,
+            0.018776189535856247,
+            0.0158944521099329,
+            -0.0002595757250674069,
+            0.012318813242018223,
+            -0.01964152604341507,
+            -0.009184005670249462,
+            0.009690145961940289,
+            -0.003569516818970442,
+            0.01048201136291027,
+            -0.014759717509150505,
+            0.010033016093075275,
+            -0.011159587651491165,
+            -0.011053460650146008,
+            0.01385356206446886,
+            0.01738838478922844,
+            0.008873790502548218,
+            -0.001294944784604013,
+            3.383413059054874e-05,
+            -0.010506502352654934,
+            -0.024033522233366966,
+            0.004771600943058729,
+            0.027021385729312897,
+            -0.013273949734866619,
+            0.003883813973516226,
+            0.008702355436980724,
+            -0.017502672970294952,
+            -0.019821124151349068,
+            -0.012653518468141556,
+            -0.022106921300292015,
+            0.020572172477841377,
+            0.023413091897964478,
+            -0.02026195637881756,
+            0.00877582747489214,
+            -0.004620575346052647,
+            0.014049487188458443,
+            -0.01400050614029169,
+            0.010612628422677517,
+            0.0317889042198658,
+            -0.020000724121928215,
+            -0.0028654097113758326,
+            0.0036633978597819805,
+            -0.015347493812441826,
+            0.018629245460033417,
+            0.03268689662218094,
+            -0.00176945177372545,
+            0.014041324146091938,
+            0.0008138049161061645,
+            -0.009118696674704552,
+            0.01905374974012375,
+            0.011094278655946255,
+            0.005347132217139006,
+            -0.0042491331696510315,
+            -0.012212686240673065,
+            -0.03010721132159233,
+            0.016702644526958466,
+            0.0129637336358428,
+            -0.0037674831692129374,
+            -0.0011551438365131617,
+            -0.006343086715787649,
+            -0.015404638834297657,
+            0.0035123717971146107,
+            -0.013078023679554462,
+            -0.00803294312208891,
+            -0.008759500458836555,
+            0.004314441699534655,
+            -0.0034960447810590267,
+            -0.009053388610482216,
+            0.007236996199935675,
+            0.0009510547970421612,
+            -0.0010602424154058099,
+            -0.01902109570801258,
+            -0.025078458711504936,
+            -0.00995954405516386,
+            -0.016849588602781296,
+            0.0038287099450826645,
+            0.0005234883283264935,
+            -0.027184655889868736,
+            0.0053961132653057575,
+            -0.008139070123434067,
+            -0.008134988136589527,
+            0.007126788143068552,
+            0.006881881505250931,
+            -0.007437003776431084,
+            -0.010816717520356178,
+            -0.00786150898784399,
+            -0.006865554489195347,
+            0.020033378154039383,
+            0.0034348180051892996,
+            0.012082069180905819,
+            -0.010498338378965855,
+            -0.018025141209363937,
+            -0.016963878646492958,
+            0.016718972474336624,
+            -0.008971752598881721,
+            -0.02083340659737587,
+            -0.01444950234144926,
+            -0.008424794301390648,
+            -0.007665583398193121,
+            -0.01527402177453041,
+            -0.021159948781132698,
+            -0.0032103201374411583,
+            -0.011347348801791668,
+            -0.0067431009374558926,
+            -0.0028960229828953743,
+            0.0013571919407695532,
+            0.00951871182769537,
+            0.00546958576887846,
+            0.00438791373744607,
+            0.021666089072823524,
+            0.010122815147042274,
+            -0.005461421795189381,
+            0.016441410407423973,
+            -0.006183897145092487,
+            -0.004440976772457361,
+            -0.01498013362288475,
+            -0.009918726049363613,
+            0.018547609448432922,
+            -0.0055226488038897514,
+            0.008457448333501816,
+            -0.0009536059224046767,
+            -0.00576347392052412,
+            0.015674035996198654,
+            0.0129637336358428,
+            0.016718972474336624,
+            0.004771600943058729,
+            -0.00012168807734269649,
+            0.012416775338351727,
+            -0.0015745466807857156,
+            0.012555556371808052,
+            -0.0038429962005466223,
+            0.014506647363305092,
+            -0.009722800925374031,
+            -0.018988441675901413,
+            0.004157293122261763,
+            0.006265532691031694,
+            -0.012531065382063389,
+            0.009404421783983707,
+            0.007743136957287788,
+            -0.013747435994446278,
+            -0.02261306345462799,
+            -0.000106317616882734,
+            0.00014362763613462448,
+            0.008122743107378483,
+            0.008971752598881721,
+            -0.008073761127889156,
+            0.012890261597931385,
+            0.004616493359208107,
+            -0.007179851643741131,
+            -0.019657853990793228,
+            0.026009103283286095,
+            0.015608727931976318,
+            0.012269831262528896,
+            -0.052508022636175156,
+            -0.0030980713199824095,
+            0.010743245482444763,
+            -0.015951598063111305,
+            -0.026792805641889572,
+            -0.006404313258826733,
+            -0.008759500458836555,
+            -0.04290767386555672,
+            -0.009624837897717953,
+            -0.0005724696675315499,
+            -0.011388166807591915,
+            0.010400376282632351,
+            0.01573934406042099,
+            0.014612773433327675,
+            0.005489994306117296,
+            -0.007232914678752422,
+            -0.017633290961384773,
+            -0.02788672223687172,
+            0.010057506151497364,
+            -0.013380075804889202,
+            -0.006412476766854525,
+            -0.002263347152620554,
+            -0.007690073922276497,
+            0.0016112826997414231,
+            0.01691489666700363,
+            0.042483169585466385,
+            0.023037567734718323,
+            -0.012335140258073807,
+            -0.020131340250372887,
+            -0.012800462543964386,
+            -0.009192168712615967,
+            0.00633492274209857,
+            0.011592255905270576,
+            -0.004738946910947561,
+            0.0096656559035182,
+            -0.004294032696634531,
+            0.012302485294640064,
+            0.012767808511853218,
+            0.01680060662329197,
+            -2.3023791072773747e-05,
+            0.005579793825745583,
+            -0.0024694770108908415,
+            -0.01738838478922844,
+            -0.004379750229418278,
+            0.008261523209512234,
+            -0.0008872769540175796,
+            0.0019317024853080511,
+            0.014204595237970352,
+            0.019102731719613075,
+            -0.009094206616282463,
+            -0.014612773433327675,
+            -0.016098542138934135,
+            -0.005277741700410843,
+            -0.0012622905196622014,
+            0.029927613213658333,
+            0.02037624642252922,
+            -0.019266001880168915,
+            -0.018237393349409103,
+            -0.018351683393120766,
+            0.00922482367604971,
+            -0.003563394071534276,
+            -0.0064369672909379005,
+            0.01905374974012375,
+            0.023021239787340164,
+            0.017910851165652275,
+            -0.003642988856881857,
+            -0.014865843579173088,
+            0.003804219188168645,
+            0.02847449854016304,
+            0.008367649279534817,
+            0.01078406348824501,
+            0.008726846426725388,
+            -0.0011908593587577343,
+            -0.032441988587379456,
+            -0.0113146947696805,
+            0.00951871182769537,
+            -0.008481939323246479,
+            0.026564225554466248,
+            -0.016531210392713547,
+            0.009388094767928123,
+            -0.005077734589576721,
+            0.0019286411115899682,
+            -0.02365799807012081,
+            0.0014929111348465085,
+            0.010122815147042274,
+            -0.024947840720415115,
+            0.00336950970813632,
+            -0.023592690005898476,
+            -0.00013763252354692668,
+            0.008367649279534817,
+            0.0022837561555206776,
+            0.01650671847164631,
+            -0.0113146947696805,
+            0.0051838611252605915,
+            0.02099667675793171,
+            -0.002832755446434021,
+            -0.01512707769870758,
+            -0.0007842120248824358,
+            0.0005092020728625357,
+            -0.001286781276576221,
+            0.02563357912003994,
+            0.015812816098332405,
+            -0.0036756431218236685,
+            0.020882388576865196,
+            0.008677864447236061,
+            0.0019133344758301973,
+            -0.021764053031802177,
+            0.0023796779569238424,
+            -0.032899148762226105,
+            0.017437364906072617,
+            -0.011126932688057423,
+            0.012490247376263142,
+            0.00868602842092514,
+            -0.01341272983700037,
+            -0.0007388022495433688,
+            -0.0107187544927001,
+            -0.0015725058037787676,
+            -0.007934981025755405,
+            0.009779945947229862,
+            -0.004363423213362694,
+            0.008339077234268188,
+            -0.00024579971795901656,
+            -0.01476788055151701,
+            0.018694553524255753,
+            9.678283277025912e-06,
+            -0.00010925139940809458,
+            0.005441013257950544,
+            0.009935053065419197,
+            0.00786559097468853,
+            0.002009256277233362,
+            0.023870250210165977,
+            -0.031119491904973984,
+            -0.016082214191555977,
+            -0.010204450227320194,
+            0.005543057806789875,
+            -0.007922735065221786,
+            -0.018531281501054764,
+            -0.03257260471582413,
+            0.01318415068089962,
+            0.001293924287892878,
+            -0.0009806477464735508,
+            0.01258004643023014,
+            -0.00942891277372837,
+            -0.0013878053287044168,
+            -0.01606588624417782,
+            -0.014808698557317257,
+            0.006416558753699064,
+            0.0033633869607001543,
+            -0.01222901325672865,
+            -0.023739634081721306,
+            0.014759717509150505,
+            0.0007418635650537908,
+            -0.016980206593871117,
+            -0.017502672970294952,
+            0.00018457300029695034,
+            -0.004804255440831184,
+            0.02204161323606968,
+            -0.016678154468536377,
+            0.003373591462150216,
+            0.0007219648687168956,
+            0.009232986718416214,
+            -0.02889900468289852,
+            0.00716352416202426,
+            0.0006107363733462989,
+            0.0035164537839591503,
+            -0.006518602836877108,
+            -0.021013004705309868,
+            0.011371839791536331,
+            -0.0020674217958003283,
+            -0.014751553535461426,
+            -0.020751770585775375,
+            -0.0020796670578420162,
+            -0.007420676294714212,
+            -0.02724996581673622,
+            0.017910851165652275,
+            0.0009985055075958371,
+            -0.01738838478922844,
+            0.008077843114733696,
+            0.006596156861633062,
+            0.009355440735816956,
+            0.02455599047243595,
+            0.010147306136786938,
+            -0.018776189535856247,
+            -0.0008087027235887945,
+            0.006979844067245722,
+            0.009306458756327629,
+            0.0036450298503041267,
+            0.04222193360328674,
+            -0.008130906149744987,
+            -0.006579829845577478,
+            0.0066614653915166855,
+            -0.008449285291135311,
+            -0.013747435994446278,
+            0.026727497577667236,
+            0.005796127952635288,
+            0.015984252095222473,
+            0.016123032197356224,
+            -0.00945340283215046,
+            -0.005008344538509846,
+            -0.0018786393338814378,
+            0.02065380848944187,
+            0.0073798587545752525,
+            -0.005502239800989628,
+            -0.009714636951684952,
+            0.005881845485419035,
+            -0.00898808054625988,
+            0.0021653843577951193,
+            -0.009526874870061874,
+            -0.005824700463563204,
+            -0.008069679141044617,
+            0.016400592401623726,
+            -0.0003523086488712579,
+            0.027315273880958557,
+            0.020751770585775375,
+            -0.008563575334846973,
+            -0.0005683879135176539,
+            0.017208784818649292,
+            -0.005820618476718664,
+            0.015608727931976318,
+            -0.011192241683602333,
+            -0.007706400938332081,
+            0.006408394780009985,
+            0.024017194285988808,
+            0.012237177230417728,
+            -0.0056042843498289585,
+            0.0002885308349505067,
+            -0.0030348035506904125,
+            -0.010612628422677517,
+            0.013388238847255707,
+            0.015323002822697163,
+            -0.01947825588285923,
+            -0.004722619894891977,
+            -0.0016592437168583274,
+            -0.003300119424238801,
+            0.00945340283215046,
+            0.02305389568209648,
+            0.000171817431692034,
+            0.009184005670249462,
+            0.0012765767751261592,
+            0.03862996771931648,
+            -0.0053144777193665504,
+            0.03102152980864048,
+            -0.0010423846542835236,
+            -0.0059430720284581184,
+            -0.009094206616282463,
+            -0.020441556349396706,
+            -0.011486129835247993,
+            -0.00041481092921458185,
+            -0.005040998570621014,
+            -0.0004956811899319291,
+            -0.020327266305685043,
+            0.0026960158720612526,
+            0.014637264423072338,
+            0.007661501411348581,
+            0.017927179113030434,
+            -0.002889900468289852,
+            0.017437364906072617,
+            0.019690508022904396,
+            -0.01400050614029169,
+            -0.029862305149435997,
+            -0.0006515541463159025,
+            0.01498013362288475,
+            -0.01661284640431404,
+            0.019445601850748062,
+            0.004085862077772617,
+            0.0029531680047512054,
+            0.004779764451086521,
+            -0.006049198564141989,
+            0.018482301384210587,
+            0.02039257436990738,
+            0.01453930139541626,
+            -0.013167822733521461,
+            0.014057651162147522,
+            0.009771781973540783,
+            0.006600238848477602,
+            -0.01646590232849121,
+            -0.013535182923078537,
+            0.002555194543674588,
+            0.0009245232213288546,
+            0.024392718449234962,
+            -0.020588500425219536,
+            0.0016602640971541405,
+            0.010588138364255428,
+            0.01769859902560711,
+            -0.004302196204662323,
+            0.008906444534659386,
+            0.0008740111952647567,
+            0.007134951651096344,
+            -0.0035919665824621916,
+            -0.006457376293838024,
+            -0.006673710886389017,
+            -0.010392212308943272,
+            -0.0020959940738976,
+            -0.02070278860628605,
+            0.002198038622736931,
+            -0.002830714685842395,
+            0.018237393349409103,
+            -0.01018812321126461,
+            0.038140155375003815,
+            0.03399306535720825,
+            -0.0007025764207355678,
+            0.011102442629635334,
+            -0.01709449477493763,
+            -0.003945040516555309,
+            -0.021959977224469185,
+            0.01844964735209942,
+            0.016392430290579796,
+            0.012269831262528896,
+            0.005302232690155506,
+            0.017796561121940613,
+            -0.0006301248213276267,
+            0.014490320347249508,
+            0.0029654132667928934,
+            0.007971717044711113,
+            -0.002265388146042824,
+            0.003938918001949787,
+            -0.031103165820240974,
+            -0.017029186710715294,
+            -0.01138000376522541,
+            0.004796091932803392,
+            -0.006375740747898817,
+            -0.0035205355379730463,
+            0.0084737753495574,
+            -0.005902254022657871,
+            0.0004270562494639307,
+            -0.025992777198553085,
+            -0.0157148540019989,
+            -0.010318740271031857,
+            -0.004040962550789118,
+            -0.01169838197529316,
+            -0.0027694879099726677,
+            -0.0022082431241869926,
+            0.0029552089981734753,
+            0.020947696641087532,
+            -0.012808626517653465,
+            -0.016294466331601143,
+            0.004379750229418278,
+            -0.002263347152620554,
+            -0.014016833156347275,
+            -0.016898570582270622,
+            0.006665547378361225,
+            0.012098397128283978,
+            -0.0017470018938183784,
+            -0.013608654960989952,
+            0.011739199981093407,
+            -0.006628811359405518,
+            0.04663025587797165,
+            -0.013584164902567863,
+            -0.0078002819791436195,
+            0.003720542648807168,
+            0.004698128905147314,
+            -0.009396257810294628,
+            0.007330877240747213,
+            0.013020878657698631,
+            0.01830270327627659,
+            0.0095758568495512,
+            -0.008465612307190895,
+            -0.0024388637393712997,
+            0.007796200457960367,
+            -0.016147522255778313,
+            0.006404313258826733,
+            0.0008525818702764809,
+            -0.014400520361959934,
+            0.006110425107181072,
+            0.028980640694499016,
+            0.013151495717465878,
+            0.01288209855556488,
+            -0.022351829335093498,
+            -0.004094025585800409,
+            -0.009355440735816956,
+            -0.004943036008626223,
+            -0.010090161114931107,
+            0.027070365846157074,
+            -0.018155759200453758,
+            0.02651524543762207,
+            0.00898808054625988,
+            0.0017684312770143151,
+            -0.01176369097083807,
+            0.006343086715787649,
+            0.02832755446434021,
+            0.0028592869639396667,
+            -0.020000724121928215,
+            -0.013592327944934368,
+            0.006314514204859734,
+            0.01707816869020462,
+            -0.002647034591063857,
+            -0.005428767763078213,
+            0.019657853990793228,
+            0.01784554310142994,
+            -0.004334850702434778,
+            -0.00936360377818346,
+            0.0008337036124430597,
+            0.00928196869790554,
+            0.016963878646492958,
+            0.015020951628684998,
+            0.009086042642593384,
+            -0.002647034591063857,
+            0.008037025108933449,
+            -0.008800318464636803,
+            -0.016449574381113052,
+            0.007220669183880091,
+            -0.008432958275079727,
+            -0.016963878646492958,
+            0.01872720755636692,
+            0.015257694758474827,
+            0.0181394312530756,
+            0.0024592725094407797,
+            -0.016735298559069633,
+            0.022302847355604172,
+            0.005755309946835041,
+            0.002998067531734705,
+            0.02160078100860119,
+            -0.022221211344003677,
+            0.013012715615332127,
+            0.02037624642252922,
+            0.027609162032604218,
+            -0.01737205684185028,
+            -0.010906516574323177,
+            -0.004857318475842476,
+            -0.006853308994323015,
+            -0.00176945177372545,
+            -0.008253359235823154,
+            -0.012620864436030388,
+            0.0028735732194036245,
+            -0.030466407537460327,
+            0.018106777220964432,
+            -0.005102225113660097,
+            0.010669773444533348,
+            -0.019266001880168915,
+            0.011363675817847252,
+            -0.0035838030744343996,
+            0.011135096661746502,
+            0.0019612954929471016,
+            -0.030041903257369995,
+            -0.008571738377213478,
+            0.015388311818242073,
+            -0.0030368445441126823,
+            -0.013641309924423695,
+            0.02006603218615055,
+            0.00018942011229228228,
+            -0.022270193323493004,
+            -0.004314441699534655,
+            -0.006934944540262222,
+            0.002193956868723035,
+            -0.0029613315127789974,
+            -0.0052981507033109665,
+            -0.030188847333192825,
+            0.020033378154039383,
+            0.009559528902173042,
+            -0.03072764165699482,
+            -0.003255219664424658,
+            -0.014996460638940334,
+            -0.0025184585247188807,
+            0.008881953544914722,
+            -0.004000144544988871,
+            -0.015551582910120487,
+            0.01784554310142994,
+            0.022711025550961494,
+            -0.0023164101876318455,
+            -0.016057724133133888,
+            -0.0090697156265378,
+            -0.027494871988892555,
+            0.030825603753328323,
+            -0.0066696288995444775,
+            0.0022878379095345736,
+            -0.005322641227394342,
+            0.009388094767928123,
+            0.004228724166750908,
+            0.0157148540019989,
+            0.006081852596253157,
+            -0.002647034591063857,
+            0.026596879586577415,
+            -0.01964152604341507,
+            0.009567692875862122,
+            -1.588864870427642e-05,
+            -0.003308282932266593,
+            0.01890680566430092,
+            0.0007939062779769301,
+            0.021649762988090515,
+            -0.008571738377213478,
+            -0.0001475818717153743,
+            -0.0034388997592031956,
+            0.010898353531956673,
+            -0.0028450009413063526,
+            -0.003465431509539485,
+            -0.001531688030809164,
+            0.011118769645690918,
+            0.01013097818940878,
+            0.003804219188168645,
+            -0.01737205684185028,
+            0.003142970846965909,
+            -0.008547247387468815,
+            -0.007028825581073761,
+            0.01302904263138771,
+            -0.011192241683602333,
+            -0.0018990482203662395,
+            -0.03160930797457695,
+            0.016286302357912064,
+            0.0533243790268898,
+            0.008547247387468815,
+            -0.014433175325393677,
+            -0.0018806802108883858,
+            -0.019755816087126732,
+            0.01535565685480833,
+            0.011004479601979256,
+            -0.016816934570670128,
+            -0.011984107084572315,
+            -0.012310649268329144,
+            0.007649256382137537,
+            0.021306892856955528,
+            -0.006522684823721647,
+            -0.008579902350902557,
+            0.025372346863150597,
+            -0.011788181029260159,
+            -0.010882026515901089,
+            -0.01661284640431404,
+            -0.01635977439582348,
+            0.00241641397587955,
+            0.019902760162949562,
+            -0.0169475506991148,
+            0.021176276728510857,
+            0.00936360377818346,
+            0.0010133018950000405,
+            0.003624620847404003,
+            0.008588065393269062,
+            0.0024878450203686953,
+            0.017878197133541107,
+            0.003883813973516226,
+            -0.01439235731959343,
+            0.0021408938337117434,
+            -0.0030286810360848904,
+            0.004624656867235899,
+            3.4320437407586724e-06,
+            -0.002749078907072544,
+            -0.0065635028295218945,
+            0.003710338147357106,
+            -0.021306892856955528,
+            0.021502818912267685,
+            -0.022678371518850327,
+            -0.004245051648467779,
+            0.005555302836000919,
+            -0.005118552595376968,
+            0.013469874858856201,
+            0.018792515620589256,
+            0.01423724927008152,
+            0.008702355436980724,
+            0.01462093647569418,
+            -0.011331021785736084,
+            0.02323349379003048,
+            -0.013282112777233124,
+            0.0005816536722704768,
+            0.012939243577420712,
+            0.01755165494978428,
+            0.018792515620589256,
+            -0.006245123688131571,
+            -0.010384049266576767,
+            -0.0002946535241790116,
+            0.0129637336358428,
+            -0.022286519408226013,
+            -0.012269831262528896,
+            -0.01386172603815794,
+            0.011486129835247993,
+            -0.004743028432130814,
+            0.025453981012105942,
+            -0.00633492274209857,
+            -0.014653591439127922,
+            -0.015918942168354988,
+            0.014196431264281273,
+            -0.048752784729003906,
+            0.0030348035506904125,
+            -0.0024919267743825912,
+            0.03043375350534916,
+            -0.027021385729312897,
+            0.01738838478922844,
+            -0.0028490826953202486,
+            -0.009355440735816956,
+            0.015429128892719746,
+            -0.015992414206266403,
+            -0.0046613928861916065,
+            -0.0016806729836389422,
+            -0.007539048325270414,
+            -0.00112248957157135,
+            0.017616963014006615,
+            -0.015845470130443573,
+            0.01282495353370905,
+            -0.00989423505961895,
+            0.0027021386194974184,
+            -0.004526694305241108,
+            -0.007751300930976868,
+            0.0196088720113039,
+            0.034548189491033554,
+            -0.007053316105157137,
+            0.0014990337658673525,
+            -0.022547753527760506,
+            0.013192313723266125,
+            -0.006730855442583561,
+            0.02351105399429798,
+            -0.005432849284261465,
+            -0.016898570582270622,
+            -0.013976015150547028,
+            -0.009739127941429615,
+            0.00839214026927948,
+            -0.022433463484048843,
+            -0.011110605672001839,
+            -0.00922482367604971,
+            -0.002573562553152442,
+            -0.018694553524255753,
+            0.004865481983870268,
+            -0.002426618244498968,
+            -0.012506574392318726,
+            -0.008816645480692387,
+            0.01048201136291027,
+            -0.009347276762127876,
+            0.003077662317082286,
+            -0.0011929002357646823,
+            -0.011453475803136826,
+            -0.02369065210223198,
+            0.0158944521099329,
+            0.0001974561164388433,
+            -0.010359558276832104,
+            0.008408467285335064,
+            -0.0018276170594617724,
+            0.001211268245242536,
+            0.016963878646492958,
+            -0.0020664012990891933,
+            0.003467472270131111,
+            0.023478399962186813,
+            0.004057289566844702,
+            -0.010971825569868088,
+            -0.00025906547671183944,
+            0.013535182923078537,
+            -0.005118552595376968,
+            -0.010008525103330612,
+            1.6646010408294387e-05,
+            -0.02984597720205784,
+            -0.008351322263479233,
+            0.006094098091125488,
+            0.018629245460033417,
+            -0.010620792396366596,
+            -0.015853634104132652,
+            0.026319319382309914,
+            -0.006359413731843233,
+            0.019070077687501907,
+            0.002889900468289852,
+            0.0025694805663079023,
+            0.01724144071340561,
+            0.013461710885167122,
+            0.00786150898784399,
+            -0.013927034102380276,
+            -0.02429475635290146,
+            0.0016398552106693387,
+            -0.018090449273586273,
+            0.010579974390566349,
+            -0.011028970591723919,
+            0.0032756286673247814,
+            -0.02398454025387764,
+            -0.012237177230417728,
+            0.004934872500598431,
+            -0.020343592390418053,
+            -0.00040588201954960823,
+            -0.0056940834037959576,
+            0.022237539291381836,
+            -0.005310396198183298,
+            0.008669701404869556,
+            0.0018255761824548244,
+            -0.012237177230417728,
+            0.0009877908742055297,
+            0.009469729848206043,
+            -0.029192892834544182,
+            -0.0007127808639779687,
+            -0.003138889092952013,
+            0.0011418780777603388,
+            0.005351213738322258,
+            0.009755454957485199,
+            0.007322713732719421,
+            0.0027062203735113144,
+            0.007253323681652546,
+            0.010449357330799103,
+            -0.00022041612828616053,
+            0.0197884701192379,
+            0.018172085285186768,
+            -0.007351286243647337,
+            0.008710519410669804,
+            0.024817224591970444,
+            -0.013159659691154957,
+            -0.008718682453036308,
+            0.023266147822141647,
+            -0.006461458280682564,
+            -0.0011122850701212883,
+            -0.01483318954706192,
+            0.00815947912633419,
+            0.0017367975087836385,
+            0.028131630271673203,
+            0.019657853990793228,
+            0.005820618476718664,
+            0.000639819074422121,
+            0.015788326039910316,
+            -0.013020878657698631,
+            0.0035286990460008383,
+            0.015461783856153488,
+            -0.003940958995372057,
+            -0.019821124151349068,
+            0.013878053054213524,
+            0.009502384811639786,
+            -0.003228688146919012,
+            0.01551892887800932,
+            0.011412657797336578,
+            -0.030841931700706482,
+            -0.0011898389784619212,
+            0.010498338378965855,
+            -0.0021449755877256393,
+            -0.008751336485147476,
+            -0.00014273475972004235,
+            -0.012384121306240559,
+            0.021388528868556023,
+            -0.004408322740346193,
+            0.012277995236217976,
+            -0.00815947912633419,
+            -0.017127150669693947,
+            -0.011559601873159409,
+            0.007914572022855282,
+            -0.010547320358455181,
+            -0.0006714528426527977,
+            -0.004022594541311264,
+            -0.005779800936579704,
+            0.017747581005096436,
+            0.004706292413175106,
+            -0.0008964610169641674,
+            -0.0018398624379187822,
+            0.007983962073922157,
+            -0.006979844067245722,
+            -0.008726846426725388,
+            0.004816500470042229,
+            -0.012833117507398129,
+            0.003981776535511017,
+            0.005543057806789875,
+            0.01506176870316267,
+            -0.014857680536806583,
+            -0.021159948781132698,
+            0.007020662073045969,
+            0.02695607766509056,
+            -0.02516009286046028,
+            -0.005232842173427343,
+            0.006245123688131571,
+            0.002093953313305974,
+            0.006914535537362099,
+            -0.006934944540262222,
+            -0.007208424154669046,
+            -0.03657275065779686,
+            -0.0020745648071169853,
+            0.0009005427709780633,
+            0.008963589556515217,
+            -0.008049271069467068,
+            -0.0136494729667902,
+            -0.011135096661746502,
+            0.002410291228443384,
+            0.007134951651096344,
+            -0.02204161323606968,
+            0.0049593630246818066,
+            0.009608510881662369,
+            -0.019559891894459724,
+            0.025127438828349113,
+            -0.010890189558267593,
+            -6.964537897147238e-05,
+            -0.00995954405516386,
+            0.021633435040712357,
+            -0.013486201874911785,
+            -0.018368011340498924,
+            -0.0006898208521306515,
+            0.008355404250323772,
+            0.011926962062716484,
+            -0.0017663904000073671,
+            -0.011371839791536331,
+            0.0003752686607185751,
+            0.01646590232849121,
+            0.019135385751724243,
+            0.002744997153058648,
+            -0.0024837632663547993,
+            -0.01019628718495369,
+            -0.009592183865606785,
+            -0.0073961857706308365,
+            -0.0007449248805642128,
+            0.002091912319883704,
+            0.02035992033779621,
+            0.013469874858856201,
+            0.0024837632663547993,
+            -0.015135240741074085,
+            -0.005257333163172007,
+            0.008939098566770554,
+            0.009412584826350212,
+            -0.011690218932926655,
+            -0.00898808054625988,
+            0.002659279853105545,
+            0.02249877341091633,
+            0.017633290961384773,
+            -0.0014000505907461047,
+            0.0032878739293664694,
+            -0.0016592437168583274,
+            0.012506574392318726,
+            -0.007555375341325998,
+            -0.010269759222865105,
+            -0.006943108048290014,
+            -0.014645427465438843,
+            -0.010800390504300594,
+            -0.0023062059190124273,
+            -0.010179960168898106,
+            0.022629389539361,
+            0.007988044060766697,
+            0.009779945947229862,
+            0.008265605196356773,
+            0.010269759222865105,
+            0.002087830565869808,
+            0.025715215131640434,
+            -0.004424649756401777,
+            -0.011984107084572315,
+            0.007236996199935675,
+            -0.022368155419826508,
+            -0.021535472944378853,
+            0.008257441222667694,
+            0.006367577239871025,
+            0.03595232218503952,
+            -0.026890767738223076,
+            0.0052981507033109665,
+            -0.009779945947229862,
+            -0.010400376282632351,
+            0.007996207103133202,
+            -0.0015439334092661738,
+            -0.004245051648467779,
+            0.022368155419826508,
+            0.00726556871086359,
+            0.0035797213204205036,
+            0.0032817514147609472,
+            0.019837452098727226,
+            -0.005685919895768166,
+            0.01573934406042099,
+            0.0259274672716856,
+            0.024621298536658287,
+            0.009388094767928123,
+            -0.009469729848206043,
+            0.02785406820476055,
+            0.0002895512734539807,
+            -0.0034327772445976734,
+            0.007220669183880091,
+            -0.00305113079957664,
+            -0.0009556467994116247,
+            -0.010033016093075275,
+            0.021764053031802177,
+            -0.001455154619179666,
+            0.004359341226518154,
+            0.02081707864999771,
+            -7.774515688652173e-05,
+            0.006220633164048195,
+            0.01057181041687727,
+            0.002181711606681347,
+            0.003396041225641966,
+            0.010775899514555931,
+            -0.039315707981586456,
+            -0.0006571666453965008,
+            -0.025600925087928772,
+            -0.0018286375561729074,
+            0.0007357408758252859,
+            -0.01266984548419714,
+            0.013600491918623447,
+            -0.002015379024669528,
+            -0.010751409456133842,
+            -0.00029337796149775386,
+            0.021649762988090515,
+            0.005416522268205881,
+            0.004071575589478016,
+            -0.004469549283385277,
+            0.018792515620589256,
+            -0.006955353543162346,
+            -0.017861871048808098,
+            -0.012759645469486713,
+            -0.012367794290184975,
+            -0.01918436773121357,
+            -0.01214737817645073,
+            -0.0024551907554268837,
+            -0.002667443361133337,
+            0.004738946910947561,
+            0.002481722505763173,
+            -0.003228688146919012,
+            -0.02070278860628605,
+            0.015037278644740582,
+            -0.01430255826562643,
+            0.012873934581875801,
+            -0.011053460650146008,
+            0.005963481031358242,
+            -0.004122598096728325,
+            0.012726990506052971,
+            -0.021404854953289032,
+            0.02070278860628605,
+            -0.01536382082849741,
+            0.0038532004691660404,
+            0.019592545926570892,
+            0.006551257334649563,
+            -0.008118661120533943,
+            -0.013404566794633865,
+            -0.022074267268180847,
+            0.021633435040712357,
+            -0.011633073911070824,
+            -0.018237393349409103,
+            -0.004608329851180315,
+            0.009649328887462616,
+            -0.017470018938183784,
+            -0.007367613259702921,
+            0.01707816869020462,
+            -0.0018949664663523436,
+            0.012980061583220959,
+            0.023004913702607155,
+            -0.0052981507033109665,
+            0.0036572751123458147,
+            -0.009167678654193878,
+            0.0010286086471751332,
+            -0.02470293454825878,
+            0.013527019880712032,
+            0.008971752598881721,
+            0.006269614677876234,
+            -0.013935198076069355,
+            -0.013306603766977787,
+            0.0036531933583319187,
+            0.004498121794313192,
+            0.011437147855758667,
+            0.009420748800039291,
+            0.004220560658723116,
+            -0.013918871060013771,
+            -0.027788760140538216,
+            -0.027070365846157074,
+            0.03595232218503952,
+            -0.0005002731923013926,
+            -0.020947696641087532,
+            0.03399306535720825,
+            0.016114868223667145,
+            0.0022980421781539917,
+            -0.012449429370462894,
+            -0.004314441699534655,
+            0.024621298536658287,
+            0.016400592401623726,
+            0.010465684346854687,
+            -0.0004742518358398229,
+            0.005571629852056503,
+            0.0016469983384013176,
+            0.03012353740632534,
+            0.015323002822697163,
+            0.0009459525463171303,
+            -0.038597315549850464,
+            0.011943289078772068,
+            -0.007547211833298206,
+            0.0013408648082986474,
+            0.006702283397316933,
+            0.017780235037207603,
+            0.011935125105082989,
+            0.015567909926176071,
+            0.014865843579173088,
+            -0.003065417055040598,
+            0.0029511270113289356,
+            0.0006163488142192364,
+            -0.009624837897717953,
+            -0.001767410896718502,
+            -0.008881953544914722,
+            -0.005714492406696081,
+            -0.002920513739809394,
+            0.005592038854956627,
+            0.007657419890165329,
+            -0.003983817528933287,
+            0.028049994260072708,
+            -0.0006433906382881105,
+            -0.017600636929273605,
+            -0.009061552584171295,
+            -0.03611559048295021,
+            0.00434301421046257,
+            -0.030939893797039986,
+            -0.011257549747824669,
+            0.0033674687147140503,
+            -0.020457882434129715,
+            -0.024768242612481117,
+            0.0010306495241820812,
+            0.014335212297737598,
+            0.007392104249447584,
+            0.0033327736891806126,
+            -0.01004934310913086,
+            0.003816464450210333,
+            -0.001855169073678553,
+            -0.006461458280682564,
+            0.028833696618676186,
+            0.019559891894459724,
+            0.020457882434129715,
+            -0.01190247107297182,
+            0.019298657774925232,
+            0.005845109466463327,
+            0.02035992033779621,
+            0.010465684346854687,
+            -0.0048695639707148075,
+            0.0074859848245978355,
+            0.009567692875862122,
+            -0.006155324634164572,
+            -0.021519144997000694,
+            0.013192313723266125,
+            -0.0010837126756086946,
+            -0.0052083516493439674,
+            -0.016269976273179054,
+            -0.003534821793437004,
+            -0.015812816098332405,
+            0.010547320358455181,
+            -0.017159804701805115,
+            0.012498411349952221,
+            -0.000802069844212383,
+            0.00168475485406816,
+            -0.015469946898519993,
+            -0.013951525092124939,
+            -0.0012439225101843476,
+            -0.01244126632809639,
+            -0.008971752598881721,
+            0.006163488142192364,
+            -0.007539048325270414,
+            -0.008318668231368065,
+            -0.00535937724635005,
+            -0.011437147855758667,
+            -0.010882026515901089,
+            0.004367504734545946,
+            -0.0010653446661308408,
+            0.020931368693709373,
+            0.00011671340325847268,
+            -0.005824700463563204,
+            0.01190247107297182,
+            0.02787039615213871,
+            0.0016704685986042023,
+            0.004583839327096939,
+            -0.014261740259826183,
+            -0.011118769645690918,
+            -0.02279266156256199,
+            0.016572028398513794,
+            0.0049675265327095985,
+            -0.0344502255320549,
+            0.01438419334590435,
+            0.00824927818030119,
+            0.0015102586476132274,
+            -0.006698201410472393,
+            -0.005334886722266674,
+            -0.005608365871012211,
+            0.006306350696831942,
+            -0.0022837561555206776,
+            0.0005107327597215772,
+            0.0035919665824621916,
+            0.034678805619478226,
+            -0.0040674940682947636,
+            -0.007228832691907883,
+            0.00981259997934103,
+            0.0005130287609063089,
+            0.020196648314595222,
+            -0.005518566817045212,
+            0.0037572786677628756,
+            -0.019282329827547073,
+            -0.016563864424824715,
+            0.0018429238116368651,
+            -0.01722511276602745,
+            0.023282473906874657,
+            -0.005392031744122505,
+            -0.01347803883254528,
+            -0.006906372029334307,
+            -0.020115012302994728,
+            0.012637191452085972,
+            0.02310287579894066,
+            0.007016580086201429,
+            0.01033506728708744,
+            0.012131051160395145,
+            -0.02444170042872429,
+            -0.020457882434129715,
+            0.012171869166195393,
+            0.0013571919407695532,
+            -0.008449285291135311,
+            -0.005404277238994837,
+            -0.001783737912774086,
+            0.007783954963088036,
+            -0.0032633834052830935,
+            0.007881917990744114,
+            0.06491663306951523,
+            -0.01093100756406784,
+            -0.002987863030284643,
+            -0.02966637909412384,
+            -0.004396077245473862,
+            -0.006628811359405518,
+            0.0056287748739123344,
+            0.009151351638138294,
+            -0.023135529831051826,
+            0.009232986718416214,
+            -0.030025575309991837,
+            -0.010628955438733101,
+            0.00824111420661211,
+            0.0028490826953202486,
+            -0.01220452319830656,
+            -0.0009592183632776141,
+            -0.013665799982845783,
+            0.0006107363733462989,
+            -0.009412584826350212,
+            -0.00576347392052412,
+            0.012718827463686466,
+            -0.02321716584265232,
+            0.006298186723142862,
+            0.007592111360281706,
+            -0.006514521315693855,
+            -0.013722945004701614,
+            -0.017861871048808098,
+            -0.011967780068516731,
+            -0.013314766809344292,
+            0.002996026771143079,
+            0.01190247107297182,
+            -0.02607441134750843,
+            0.012922916561365128,
+            -0.0037389106582850218,
+            0.00945340283215046,
+            0.004404240753501654,
+            0.013469874858856201,
+            0.018074123188853264,
+            0.009355440735816956,
+            0.0021429345943033695,
+            -0.008024780079722404,
+            0.012702500447630882,
+            -0.01993541419506073,
+            -0.01635161228477955,
+            0.013837235048413277,
+            0.012016761116683483,
+            0.003061335301026702,
+            0.009828926995396614,
+            0.0020959940738976,
+            0.011886144056916237,
+            0.010775899514555931,
+            -0.019739490002393723,
+            -0.0006648199632763863,
+            0.0007326795603148639,
+            0.03280118480324745,
+            -0.0005938990507274866,
+            0.008857463486492634,
+            0.004420568235218525,
+            0.013159659691154957,
+            -0.012784135527908802,
+            -0.01769859902560711,
+            0.0008087027235887945,
+            -0.01057181041687727,
+            0.021257910877466202,
+            -0.008192133158445358,
+            0.006216551177203655,
+            -0.018612917512655258,
+            -0.027086693793535233,
+            0.006392067763954401,
+            -0.016261812299489975,
+            -0.018221067264676094,
+            0.01317598670721054,
+            0.0077268099412322044,
+            0.0017572063952684402,
+            -0.005098143592476845,
+            -0.020457882434129715,
+            -0.009837090037763119,
+            0.028441844508051872,
+            0.011102442629635334,
+            -0.004120557103306055,
+            0.009584019891917706,
+            -0.009094206616282463,
+            -0.012490247376263142,
+            -0.0017541450215503573,
+            0.0036715613678097725,
+            0.008261523209512234,
+            0.005645101889967918,
+            -0.01386172603815794,
+            -0.00029873527819290757,
+            -0.012400448322296143,
+            -0.021225256845355034,
+            -0.008906444534659386,
+            0.016751626506447792,
+            0.007820690982043743,
+            -0.0038960592355579138,
+            0.015812816098332405,
+            -0.004404240753501654,
+            0.002189875114709139,
+            0.014294394291937351,
+            0.005334886722266674,
+            -0.004285869188606739,
+            -0.0095758568495512,
+            -0.004612411838024855,
+            0.004043003544211388,
+            -0.0025960123166441917,
+            0.006457376293838024,
+            0.017192458733916283,
+            0.0072900597006082535,
+            -0.013755599036812782,
+            0.013135168701410294,
+            0.0007791098323650658,
+            0.0049675265327095985,
+            -0.03699725493788719,
+            -0.003247056156396866,
+            0.015796490013599396,
+            -0.013020878657698631,
+            0.02501314878463745,
+            0.005400195252150297,
+            0.022743679583072662,
+            0.015233203768730164,
+            0.008669701404869556,
+            -0.001542912912555039,
+            0.00464098434895277,
+            0.02759283408522606,
+            -0.003151134354993701,
+            -0.008637047372758389,
+            -0.0020602785516530275,
+            0.009241150692105293,
+            0.0002063850115519017,
+            0.003000108525156975,
+            0.008281932212412357,
+            -0.03562577813863754,
+            0.011706545948982239,
+            -0.019576217979192734,
+            0.0024694770108908415,
+            0.003459308762103319,
+            -0.007183933164924383,
+            -0.014865843579173088,
+            -0.005857354495674372,
+            0.022841641679406166,
+            -0.003224606392905116,
+            0.021078312769532204,
+            0.002885818714275956,
+            0.016106704249978065,
+            -0.011575928889214993,
+            -0.009208496659994125,
+            -0.0004130251472815871,
+            -0.005351213738322258,
+            -0.007539048325270414,
+            0.010596301406621933,
+            0.0034409407526254654,
+            0.005534893833100796,
+            0.002265388146042824,
+            0.0036654386203736067,
+            -0.0036878883838653564,
+            0.02773977816104889,
+            0.003563394071534276,
+            0.007996207103133202,
+            -0.011567764915525913,
+            0.014082142151892185,
+            -0.017029186710715294,
+            0.0070778070949018,
+            0.005575711838901043,
+            -0.026156047359108925,
+            0.016849588602781296,
+            0.004775682929903269,
+            0.006628811359405518,
+            0.013812744058668613,
+            0.01618834026157856,
+            -0.010408539324998856,
+            -0.014816862531006336,
+            0.024049850180745125,
+            -0.0032041973900049925,
+            -0.015225040726363659,
+            -0.006775755435228348,
+            4.438935866346583e-05,
+            0.006559420842677355,
+            0.00373074715025723,
+            0.0013020879123359919,
+            -0.0006204306264407933,
+            -0.010743245482444763,
+            0.004645065870136023,
+            -0.01934763789176941,
+            -0.0007163524278439581,
+            -0.016449574381113052,
+            -0.015633217990398407,
+            0.018678225576877594,
+            -0.01830270327627659,
+            0.009633000940084457,
+            0.0012275953777134418,
+            0.028588788583874702,
+            -0.011706545948982239,
+            -0.012155541218817234,
+            0.0015102586476132274,
+            0.010849371552467346,
+            0.0067512644454836845,
+            -0.010033016093075275,
+            -0.007583947852253914,
+            -0.014180104248225689,
+            -0.010163633152842522,
+            0.015502601861953735,
+            -0.014498483389616013,
+            -0.014343376271426678,
+            0.008914608508348465
+          ],
+          "embedding": [
+            -0.006359657085039302,
+            -0.024594786241399234,
+            -0.02553538924848095,
+            0.005360188465088291,
+            0.012801870977589839,
+            0.021680505862506104,
+            -0.016668048966001273,
+            0.06860852474581301,
+            -0.007122272841157724,
+            0.03471775000705488,
+            -0.009004297848861675,
+            0.01059974144678391,
+            0.02565604599426148,
+            -0.022544127646041287,
+            0.010616674976540785,
+            0.013658194823359737,
+            0.016400329743363844,
+            0.02858546964917478,
+            0.0067612676434693686,
+            -0.04174737193606879,
+            0.03184155110488616,
+            -0.02223105127585514,
+            -0.05506239667974977,
+            -0.0015307551410555752,
+            -0.013169788751379287,
+            0.017681749722384007,
+            0.01682706317229353,
+            0.019652295823816928,
+            -0.05623980369271269,
+            0.030698582851533197,
+            0.02286327124296613,
+            0.016019032539689178,
+            0.005508113353872315,
+            8.58985091215149e-05,
+            -0.019206292853495906,
+            -0.018439290483495144,
+            0.03475023176950151,
+            0.01018612942141759,
+            -0.0252551158857426,
+            0.0334562130666827,
+            0.020462153537485785,
+            0.02260508249714211,
+            -0.03323524989392951,
+            -0.002094251009530707,
+            0.015177442347686773,
+            -0.00018698464350038317,
+            -0.0006788648344273403,
+            0.0007777616893581039,
+            0.013352398615059647,
+            -0.030751111528744813,
+            -0.0317733818445192,
+            -0.006729064081710058,
+            0.009413204899986514,
+            -0.005712894407943592,
+            -0.038961826880527066,
+            -0.002827692206530345,
+            0.017863300537973137,
+            0.07918455388563496,
+            0.048967158919816366,
+            0.018002566325350088,
+            0.040264962620950695,
+            -0.019715100359659787,
+            -0.02211350500879408,
+            0.004918148608475574,
+            0.01062585305866606,
+            -0.014478993521696027,
+            0.0034983131226364672,
+            0.000766228629514787,
+            0.009667268771502067,
+            -0.01558439890697323,
+            -0.04984718709870083,
+            0.029613955432821887,
+            -0.009686854114630624,
+            -0.015279319238124568,
+            0.007460629676325322,
+            -0.006235979350809034,
+            0.012514387473717085,
+            0.02800435263741497,
+            -0.008688925946672778,
+            -0.01236301543374469,
+            0.029755705020188972,
+            0.01158037219446542,
+            -0.005242392206024511,
+            0.009982679027728814,
+            0.052414361714483264,
+            -0.018788500381726227,
+            -0.020335831196027204,
+            -0.08583959943356621,
+            0.029309228216812346,
+            -0.0249285282010673,
+            -0.018929396555426307,
+            0.007710663193334574,
+            0.03096798035958051,
+            -0.017089977064591828,
+            -0.028015107605405817,
+            0.06740703271128606,
+            0.009398795412354399,
+            0.01118769022826835,
+            -0.03041417460855722,
+            0.014776002883150835,
+            0.004319194067673161,
+            0.022507636639041203,
+            0.0053562788646786845,
+            0.007548072258943124,
+            0.011123374308563128,
+            -0.006924362115585165,
+            -0.011118507147092024,
+            0.009563232643295304,
+            0.00657178473678069,
+            0.08313027447380966,
+            -0.004239370156003264,
+            -0.00930406163946907,
+            -0.039202672821898946,
+            0.009761029930556298,
+            -0.02827698465424993,
+            -0.037914533246900244,
+            -0.04496050911855815,
+            -0.017711307900236353,
+            0.01934435542287117,
+            0.004335173027873153,
+            0.03800509969714242,
+            0.031198127357716314,
+            0.031311207941680276,
+            -0.012184553894588813,
+            -0.058985863670856876,
+            0.041442017057707635,
+            -0.012456189292721711,
+            -0.0040966351720394135,
+            0.010849179519959421,
+            0.005243123049347636,
+            0.0011495799963862647,
+            0.003534183676934487,
+            -0.03139340512003631,
+            -0.023876261883905306,
+            0.008382957569752911,
+            -0.004778283217011976,
+            0.03703067589888386,
+            0.06888630688839406,
+            -0.03993532296478411,
+            -0.009472167678222372,
+            0.025246872942105057,
+            -0.01524567912700368,
+            0.01595917939878582,
+            -0.003736723164442467,
+            -0.030836797784967753,
+            0.0026246446163368533,
+            0.008894300082148884,
+            0.08342856279154155,
+            -0.0007476580521704578,
+            0.025149116238550588,
+            0.004972542633631468,
+            0.034495632546556576,
+            -0.009173329764686424,
+            0.014009381543772768,
+            0.004427261668151028,
+            -0.013426138607902904,
+            -0.022234568880321808,
+            0.01976282364689966,
+            -0.03592433789490043,
+            -0.010918278220831188,
+            0.013527180289641174,
+            -0.013023113052242677,
+            -0.008304770302495363,
+            0.003677063799167178,
+            0.0013981822663219421,
+            0.047001163242600175,
+            0.01828332714887986,
+            0.002483375662343689,
+            -0.06820498187523893,
+            0.05295312038995555,
+            0.025038753504342162,
+            0.025286870198436735,
+            -0.0029544808216316115,
+            0.029982752196635428,
+            0.0005600481523874767,
+            0.0128290258663539,
+            0.009023805227424919,
+            -0.0019219166367750596,
+            -0.04482843936478531,
+            0.04523479691409549,
+            -0.004678574027241186,
+            0.010486850804218904,
+            0.006736229865018014,
+            0.02134813656476443,
+            0.028083972673690036,
+            0.016857413440125445,
+            0.0189082447820146,
+            0.03843690613820022,
+            -0.011922027011229613,
+            0.016193433524453224,
+            0.03828207911467065,
+            -0.0013055128404331705,
+            0.043198163443700956,
+            0.024025562893260542,
+            -0.001257851946749882,
+            -0.011726713433348844,
+            -0.021907984099443396,
+            0.02752198420597673,
+            0.004431866336999851,
+            0.015798806714413912,
+            -0.012910782191382367,
+            -0.03180549180206364,
+            0.004550247372715741,
+            -0.022778272841118514,
+            0.031242218240017532,
+            -0.006106984949693483,
+            -0.006259993190497421,
+            0.02187449047033339,
+            0.009334146847221406,
+            -0.009381278960016145,
+            -0.020888853823931134,
+            -0.007502374908776156,
+            -0.010390018386728482,
+            -0.04316693435221676,
+            -0.006274513730720055,
+            0.019152443937064258,
+            -0.01668984824845791,
+            0.018573194036505578,
+            0.0035035262183249925,
+            -0.019779911381789077,
+            -0.0027495798165085993,
+            0.02425481236315913,
+            0.006857723545270935,
+            -0.014733535699087731,
+            -0.026326146471941727,
+            -0.019906660231056833,
+            -0.001033294559612807,
+            0.0159154063026895,
+            0.035649534442431415,
+            -0.026210592096240003,
+            0.014397127229517285,
+            0.005114311681443673,
+            -0.0033104236005207077,
+            0.00324988235167305,
+            0.03764004154553922,
+            -0.017822437760737605,
+            -0.008579701063708334,
+            -0.00032410984200965487,
+            0.018450351071676443,
+            0.017283379764846195,
+            0.006384411611332382,
+            -0.005115090309123598,
+            -0.030590047062583125,
+            -0.011776211359193503,
+            0.01183623914060493,
+            0.0016564847796208331,
+            -0.009530766634361575,
+            -0.004810598258154945,
+            0.03876242515526699,
+            0.027076785774401667,
+            0.005324763136165989,
+            -0.03416412480913939,
+            -0.007943016733586697,
+            0.002114307650456928,
+            -0.02961449347300628,
+            0.05142732770518828,
+            0.05566026433842563,
+            -0.0030729309664974704,
+            -0.00938186948393074,
+            -0.04547908981236149,
+            0.0651216359692231,
+            -0.01813722445944094,
+            -0.0482650853560993,
+            0.044045155892545554,
+            -0.014272653003441165,
+            0.013740652306746851,
+            0.04563732439184395,
+            0.02126513340521316,
+            -0.0054845632439384905,
+            -0.011635625101262898,
+            0.030992082069685376,
+            -0.010051935721068779,
+            -0.050967258971854425,
+            -0.009827942095033494,
+            0.013373878556654464,
+            -0.013017627321534534,
+            -0.009361001542607812,
+            -0.01925867665425919,
+            0.020329052915436022,
+            0.023020741670660188,
+            0.019387043947695172,
+            0.005628632318152202,
+            0.018451555161952665,
+            0.016226845400848455,
+            -0.01384033112836216,
+            0.004762589697426251,
+            -0.012718476561725138,
+            -0.024222765127494463,
+            -0.037487144314010595,
+            -0.019971524166659278,
+            -0.015034588610516958,
+            0.009483936496215338,
+            -0.014301924283553591,
+            -0.0024139665139412125,
+            0.013361045353529458,
+            -0.0005870798721107379,
+            -0.011192319340879789,
+            0.013927695434391159,
+            0.021337539669342293,
+            -0.027300952495815437,
+            -0.04765670695199488,
+            -0.00561706224251562,
+            0.0082776639906139,
+            -0.0030984973749633724,
+            -0.026269016057688548,
+            0.007040524859505573,
+            0.0017859008072463442,
+            0.020091542517503808,
+            -0.018716761130867725,
+            0.0035283051960672963,
+            0.008519383548064053,
+            -0.015247417050336866,
+            -0.016917626868166634,
+            -0.018580550690103854,
+            -0.024697180580294884,
+            0.034202028316737125,
+            0.0204395003463227,
+            0.030392895534267717,
+            -0.04527669147753553,
+            -0.018359428092075348,
+            0.05494107499558624,
+            0.015234754907198422,
+            -0.05066008106884831,
+            0.031139824660661085,
+            0.005016615710206006,
+            0.0019000453624929544,
+            -0.006094364589229031,
+            0.0637324238017742,
+            -0.02711381489292883,
+            -0.02187418906973867,
+            0.004601744864532875,
+            -0.04141121277496072,
+            -0.026689585608019588,
+            -0.005568100750837504,
+            0.01291019036002459,
+            -0.031853041426355644,
+            -0.02815800259372657,
+            -0.007964861641790374,
+            -0.010933515363060793,
+            0.013726639077745378,
+            -0.023956800669647334,
+            -0.015672058590455337,
+            -0.023642025374517057,
+            -0.01288435863544158,
+            -0.0042516881552192385,
+            -0.011162214160460695,
+            -0.02412733256710621,
+            0.010088528028211243,
+            0.03310175458095482,
+            0.023064360903748363,
+            0.001996159684818467,
+            0.009478229572933101,
+            0.014478772877286076,
+            0.00371783700962415,
+            0.007864103610320116,
+            -0.048511515124609,
+            -0.031398754206534776,
+            0.020227872561659397,
+            0.012159811587094092,
+            0.0172163448580163,
+            0.009578952796095119,
+            -0.04154545961112031,
+            -0.05095080393682759,
+            0.02968486144855133,
+            -0.03932395023720674,
+            -0.02063219739061335,
+            0.007512423249865194,
+            -0.011911709468952548,
+            -0.00810951734088105,
+            -0.0008079327452253181,
+            -0.01600701450676789,
+            0.023984909975645632,
+            -0.02418072541081482,
+            0.0033984265728439398,
+            -0.0104343782811971,
+            0.01588977364765743,
+            0.05109529246167992,
+            -0.027800269136366997,
+            -0.027889034481533438,
+            0.03706813210665186,
+            -0.02058120652429308,
+            0.030801396468451574,
+            0.03246127847621613,
+            0.008355517405358826,
+            -0.008605467586247932,
+            0.017693344563968573,
+            0.014499487024471321,
+            0.012431277080478217,
+            -0.040103870071440256,
+            0.017328848567062113,
+            0.02723900253727976,
+            -0.054909074068976875,
+            -0.008547596270650966,
+            -0.00777964838164648,
+            0.027134666572242665,
+            0.02765036594970255,
+            -0.002207173084941375,
+            -0.016838740117653725,
+            -0.028289233739722936,
+            -0.019433376533829132,
+            -0.011053380458946764,
+            0.03247641283550457,
+            0.0007514989009289282,
+            0.007919574219000797,
+            0.001316042436488591,
+            -0.0017934342041762042,
+            0.01924145734197619,
+            0.015260199553572459,
+            0.048259602293134723,
+            -0.03383050622747848,
+            0.0021712225589426266,
+            -0.03508567715696004,
+            -0.04578731401250471,
+            -0.008768228935292693,
+            0.010203318819548585,
+            0.024135236371743855,
+            -0.007467140460133838,
+            0.0028907962385387818,
+            -0.007582083854422456,
+            -0.005384477718912037,
+            -0.01867978482547249,
+            0.01641191108381631,
+            0.037168089880543244,
+            0.0031992495663980464,
+            -0.008533890994695162,
+            0.000778843942291118,
+            -0.004620258538699103,
+            0.010572690335163528,
+            0.033233014627100904,
+            -0.030497362482602697,
+            0.02974571152880676,
+            -0.004274442492343003,
+            0.025341347125910928,
+            -0.04307748954707389,
+            -0.018069609916024403,
+            -0.018491122727274337,
+            -0.02069307802429807,
+            0.05372967226264594,
+            -0.030768978301976772,
+            -0.03662045886993282,
+            -0.003205991669059688,
+            0.016815511161004534,
+            -0.0014152644009276566,
+            -0.0016015439520090104,
+            -0.03984716970293867,
+            -0.010828260334550217,
+            -0.05894985435251326,
+            -0.01638035185618134,
+            -0.021819442096404948,
+            -0.005315102328745982,
+            -0.06251579337157112,
+            0.0033713876515840233,
+            -0.07815974854883057,
+            0.001484860071649606,
+            0.04539427129131382,
+            -0.023530943197353476,
+            0.05142250021536681,
+            -0.024746600229063383,
+            -0.0005885740455257141,
+            -0.044698536284421976,
+            0.0026481001040031785,
+            0.02389178603305878,
+            0.051387657872255674,
+            -0.002949198236157656,
+            0.010674377196609455,
+            0.013322075519401089,
+            -0.026245090721950647,
+            -0.01036724558050635,
+            -0.005321884180696542,
+            0.005958798671856245,
+            0.013755178085972732,
+            -0.0074734329537091675,
+            0.038368016046563674,
+            0.010941421833174982,
+            0.03443115781919206,
+            0.034623617136788874,
+            -0.004361926531008214,
+            -0.03348681614127998,
+            -0.00632870804659212,
+            0.024433331840566692,
+            0.0321998807702602,
+            -0.02899983280320611,
+            0.005958471722097962,
+            -0.015602447981516634,
+            0.015633217539747047,
+            0.006321931116948646,
+            -0.011827541320878442,
+            0.02372204564999124,
+            0.015673426811623124,
+            0.006103300725621424,
+            -0.005165006670545588,
+            0.013589526056189892,
+            -0.025411864292353578,
+            0.03041791947375232,
+            -0.004456865298584878,
+            -0.04276058054583262,
+            -0.0024736078733106825,
+            -0.020059309867945227,
+            0.024883721794000956,
+            0.015578237413840115,
+            -0.00536523508503184,
+            -0.005290645798641171,
+            0.008242987948511412,
+            0.020469675635821673,
+            0.006668354647245447,
+            0.00017729517736814364,
+            0.02373604785121342,
+            0.029915687122900485,
+            -0.02432635861789771,
+            -0.029540533943782886,
+            0.008151297444985344,
+            -0.005730733861939129,
+            0.011318816579801282,
+            -0.02644855229221747,
+            0.00033932872948409137,
+            0.0035505684154028996,
+            -0.03570104343808711,
+            0.03408366016010711,
+            -0.009199700348630527,
+            0.005857598530788056,
+            -0.009538177736472163,
+            -0.0017768806148210118,
+            0.0027410153099351694,
+            0.03898108970435115,
+            -0.007367623392751247,
+            0.009827254226889936,
+            0.015465895035342481,
+            0.000981604396724216,
+            0.05143083558797524,
+            -0.027350877485564314,
+            -0.02183036962301852,
+            0.01885468378168636,
+            -0.007285239007489837,
+            -0.016846837665812908,
+            -0.020546383007536074,
+            -0.011863943814684079,
+            -0.016293192750336218,
+            0.01976516340568883,
+            -0.02387328798681798,
+            -0.011892336568452852,
+            0.007170651391639504,
+            0.0013725014050721622,
+            -0.009809669068563384,
+            0.0016617183704652932,
+            -0.006307590349798603,
+            0.02037693835615797,
+            0.031753656615094446,
+            -0.024892675105483313,
+            0.03577919637821846,
+            0.03178303188315581,
+            0.017260898395251155,
+            -0.02102289690701718,
+            -0.009661035772073294,
+            -0.00048542689442823136,
+            -0.014694631967051945,
+            -0.0326379186158631,
+            0.025443206465204822,
+            0.017194626456982255,
+            -0.011381421003142453,
+            -0.03680783761649673,
+            0.0017216115811536721,
+            0.022819859205829785,
+            -0.013600989058799166,
+            0.028141657788496834,
+            -0.01869047298692401,
+            -0.01366833755869951,
+            -0.00782959271979894,
+            0.031227914441058244,
+            -0.05252392878125891,
+            0.016957842026348195,
+            0.00020880696574616416,
+            0.001154370575234767,
+            0.010892067708194324,
+            -0.008368819756628046,
+            0.012224564683325483,
+            0.031700389750709,
+            0.018063405869147564,
+            0.0010794115307202266,
+            -0.002783668216641561,
+            0.01418234057248712,
+            0.017773269840982066,
+            -0.015653443355569053,
+            -0.000497740485657824,
+            0.019890882604100452,
+            0.022672054781919238,
+            -0.00521797370595221,
+            -0.016686942830774276,
+            -0.018443538351123584,
+            0.025270057639201107,
+            -0.010354431070738671,
+            0.03484047753830186,
+            0.04594785214094975,
+            0.0036441959528283004,
+            -0.05695476763071743,
+            -0.009412209619190693,
+            0.0010371110229966086,
+            -0.01099343927053564,
+            0.01606863921935641,
+            0.012470660511773616,
+            -0.03237187923799263,
+            0.006388672213167589,
+            0.012946570716567364,
+            0.022152554488829627,
+            -0.02540317621005867,
+            -0.023183252921885226,
+            -0.0030984019855787655,
+            -0.007890467768631811,
+            0.018648760402624687,
+            -0.008653405980661855,
+            0.0006228315253506687,
+            0.009672526897422602,
+            0.0005436494014569835,
+            0.011632326145022935,
+            0.005655942007108228,
+            0.021736827172951,
+            -0.001256608167230203,
+            0.013124413886287987,
+            -0.01986808920858981,
+            -0.0270450339809516,
+            0.006500423802582728,
+            0.04438242119211055,
+            -0.014813399694814117,
+            0.030621709833336064,
+            0.0092171224268983,
+            0.016433384739642534,
+            0.024875072913970443,
+            -0.00891470467241686,
+            -0.014124009014664287,
+            0.00845777541223868,
+            0.009019007848927453,
+            -0.04147358041382793,
+            0.012513030150663913,
+            -0.01710243399313324,
+            -0.033144939719206276,
+            0.01904674924133127,
+            -0.00534122554342279,
+            0.012367320751926314,
+            0.004270315800911569,
+            -0.002886460711688153,
+            -0.025696286516581048,
+            0.014137901708875033,
+            0.028135087538950946,
+            0.021797869775098042,
+            0.0296173052503884,
+            -0.012757078368361444,
+            0.0005460601780756187,
+            -0.008748930817524899,
+            0.005470975782187157,
+            0.04187083128854477,
+            0.0008687182830616019,
+            -0.0008032802520818105,
+            0.01507726552487313,
+            0.024945744517309208,
+            0.0376130540370386,
+            -0.015415560911023553,
+            -0.03407353556457147,
+            0.01564969972264396,
+            -0.010104443707865262,
+            0.018855792875436674,
+            0.005503259932317811,
+            0.0049634097823351596,
+            0.0038600156268000702,
+            0.019312419743287703,
+            -0.006635193789316492,
+            0.018798045806429395,
+            -0.02500983683871332,
+            -0.028785757874583755,
+            0.01829411751138161,
+            -0.005277972443258226,
+            0.03338623299668019,
+            0.016225885095995672,
+            -0.029522280454919586,
+            -0.01712097344863975,
+            0.01553706187924322,
+            -0.008465766956343768,
+            -0.024842700926275538,
+            0.01895228828512998,
+            0.007076862934523977,
+            0.008956105593078965,
+            0.0024373779389833218,
+            -0.01106503696703573,
+            0.01125466547367417,
+            -0.000818223040067914,
+            -0.015322790791698395,
+            0.014157131555941659,
+            -0.017853386731281715,
+            -0.014994640991761465,
+            0.01775991780643474,
+            -0.037498078532902435,
+            0.009570197712419421,
+            -0.007687297543935123,
+            0.009050494195378254,
+            -0.004222041469820445,
+            0.007460383234429904,
+            -0.0010804379186060264,
+            -0.014987857934522812,
+            -0.014903749199810126,
+            0.03228485430019625,
+            0.022096121825475306,
+            -0.000410816936759558,
+            0.006728733918931589,
+            -3.971636461500232e-05,
+            -0.026719848032785824,
+            -0.011253751790407687,
+            0.02245467425830765,
+            -0.005664635254398481,
+            0.005134496095445943,
+            -0.01109590855846966,
+            -0.0149616794609307,
+            0.0019726017255571673,
+            0.032668637270108894,
+            0.06066635618119076,
+            0.012001732730659747,
+            0.024422093585231484,
+            0.0189499396761026,
+            0.02119276033547528,
+            -0.008489818917377516,
+            0.006413901834269739,
+            -0.03097413287789443,
+            0.03509225573986143,
+            0.005392594635384506,
+            0.03108937318306421,
+            0.026664331248813712,
+            -0.04722525921137681,
+            -0.0058539171830594774,
+            0.017087531454837677,
+            0.022673712717885323,
+            0.0159020681868943,
+            0.0008382498574190522,
+            -0.008938174268517353,
+            0.035650940044285984,
+            -0.03315250299348084,
+            -0.023647111263155,
+            0.005059382024092534,
+            -0.01439537467317451,
+            -0.010488359956731476,
+            -0.006785295580877329,
+            0.02776449681317373,
+            -0.005584052419598926,
+            -0.004582019433008779,
+            0.009697064252638535,
+            -0.014400206558190682,
+            0.008165676294736836,
+            0.0007624778531368436,
+            0.008681263897172326,
+            0.039249866981334126,
+            0.0005918063050375113,
+            0.024275699182767528,
+            -0.004138313072843555,
+            0.011079529739650594,
+            -0.0158972266904151,
+            -0.016985058543605,
+            0.02276982363172991,
+            0.00351101120496393,
+            -0.03244126813432563,
+            -0.00768819222024939,
+            0.00928180979884261,
+            -0.0013793404539641824,
+            0.005266058566444064,
+            -0.00503433670609399,
+            0.003747448261046857,
+            0.03286230666764075,
+            0.0022383592721685153,
+            0.008606190020480513,
+            0.013593947677257625,
+            0.00030951915677929204,
+            -0.01490307680740127,
+            0.013254591138113545,
+            -2.1379308136707722e-05,
+            -0.012900073369514622,
+            0.03357519186606058,
+            0.006705902094864628,
+            0.04243347200485736,
+            -0.03302953674711988,
+            0.009031534935013593,
+            0.01822227379789576,
+            -0.008491770335368761,
+            -0.02725412856390648,
+            -0.03630788717786838,
+            0.008259145747740535,
+            -0.019003260334340084,
+            0.015781275447792516,
+            -0.010039277287105153,
+            0.029665124310477384,
+            -0.03563370794521629,
+            0.004714858110268169,
+            0.004053913355055313,
+            0.0009503673086552651,
+            0.02068360255395961,
+            0.018524093113914366,
+            -0.002785698934204024,
+            -0.031443835341186485,
+            -0.02066365433385339,
+            0.006869321907853112,
+            0.02431260052207595,
+            -0.010367048093621498,
+            -0.005612149485859478,
+            0.0006316058139530722,
+            -0.024369562712568606,
+            -0.016845124984022828,
+            0.030359419843284823,
+            -0.040974427176153246,
+            0.005252911449822132,
+            0.04272936053727608,
+            -0.04599092715016919,
+            -0.03802288716943514,
+            -0.002314988728575789,
+            -0.024124014894379466,
+            -0.006360771882090754,
+            -0.002068030981218464,
+            0.016669991113175926,
+            0.0036232707799735537,
+            -0.028763764013482795,
+            0.00943401978066451,
+            -0.005173988601101547,
+            0.003971420424839974,
+            -0.0031669592494977393,
+            -0.02216336363004636,
+            -0.01657772032717298,
+            -0.02531962163186398,
+            -0.035174102848865116,
+            -0.0015903434876311088,
+            -0.005158703150821607,
+            -0.028703075279214722,
+            -0.03941940849540492,
+            -0.021696000925262537,
+            -0.0160221488026189,
+            -0.00843213135542359,
+            0.040641882367348774,
+            0.021319060666253945,
+            0.019336844555345886,
+            0.0025252167889382917,
+            -0.020346900609125973,
+            0.049440028814233025,
+            -0.007642182460707816,
+            0.035628691886108926,
+            -0.0017882888000877158,
+            -0.013461669667950893,
+            0.00455558389906464,
+            0.03825428026120858,
+            -0.014248434654950971,
+            -0.0009902062564027718,
+            0.00027811655483232545,
+            -0.0011866253757890136,
+            0.0008535155950871286,
+            0.015433650396777696,
+            0.001307651096343185,
+            -0.007112199543314309,
+            0.030380673552683906,
+            -0.012803247194895328,
+            0.01637501752192592,
+            0.018350814869962304,
+            -0.0013359719137099667,
+            0.008064546833360553,
+            -0.021430659268086814,
+            -0.00731572118780415,
+            0.0044578576331236325,
+            0.008422210247967642,
+            -0.015096885054883586,
+            -0.009964588397149773,
+            -0.02359494467299634,
+            -0.01174145797396426,
+            0.0023855749166744366,
+            -0.004559202896782688,
+            0.004902138778592298,
+            0.034595817216889944,
+            -0.016068744500427584,
+            -0.035106033294403614,
+            0.009997207702122002,
+            -0.02663088875167486,
+            0.02222306406130687,
+            3.983384942889326e-05,
+            0.006838221769859366,
+            -0.0023434342406307616,
+            0.001765445386886839,
+            -0.00169484278667461,
+            0.002862331185439823,
+            -0.018217975204395134,
+            0.0031859734246840257,
+            -0.0032119988065673565,
+            0.0009052950782312657,
+            -0.0012459653052875638,
+            0.00687339892509406,
+            0.006332799551164718,
+            -0.0008019538898592668,
+            -0.002968451740267742,
+            -8.541086355097327e-06,
+            -0.020561816931299912,
+            -0.021642512405352372,
+            -0.028259049261410905,
+            -0.000989887278779396,
+            -0.012359733017834338,
+            -0.027804394906433664,
+            0.011443385424045703,
+            -0.006872684999639775,
+            0.008327844771213648,
+            -0.008969527138034904,
+            0.006219994886600349,
+            -0.008415106130309062,
+            -0.0004143996206881767,
+            0.013119286511794273,
+            -0.02925209282917943,
+            -0.02096907906398536,
+            -0.03096513357712367,
+            0.01762223841790055,
+            -0.00925741670612824,
+            -0.002656382260603827,
+            0.0025223527960181835,
+            0.005092659166535706,
+            -0.033606542487570154,
+            -0.006653438998497412,
+            0.007313682256676618,
+            0.009547290751154595,
+            -0.03287788897629887,
+            0.02446728061355753,
+            0.017023222076526946,
+            0.01874233066565012,
+            -0.02417566312664911,
+            0.04049827067897005,
+            0.014426879325647042,
+            -0.021740606525480655,
+            0.016712907564723732,
+            0.027385150577780248,
+            0.03177163221499527,
+            -0.03469098547555545,
+            0.004481376248622418,
+            -0.01518200063132508,
+            0.018918680020516044,
+            -0.025284921244460287,
+            0.020625170445627613,
+            -0.016707629602243063,
+            0.021278719363178687,
+            0.03520334001264928,
+            -0.005524369131116028,
+            0.03608133014180745,
+            -0.019234548780647943,
+            0.0059985801736444555,
+            0.004041778461022664,
+            -0.01645503683802124,
+            0.01024694092614158,
+            -0.0042700351372843386,
+            0.0006059159967286885,
+            0.009406090340935747,
+            0.019133229156115282,
+            0.019007219188041775,
+            -0.04442208547053466,
+            0.028588965123434923,
+            0.00804120101594811,
+            0.021472086070876265,
+            0.014177820346835521,
+            -0.02708807970982664,
+            0.004742981711172693,
+            0.002072673190279628,
+            -0.004659713164250794,
+            0.0061167904730676514,
+            -0.013796604518930328,
+            -0.034540857808309226,
+            0.006338792319814448,
+            -0.018918703778187926,
+            0.015036937364271984,
+            -0.009026670935442384,
+            0.0011667525636770179,
+            -0.01152652811400482,
+            -0.011938679695598347,
+            0.0006170592505446954,
+            -0.005495151258143897,
+            -0.0008075736908949407,
+            -0.03171721592827769,
+            -0.04427658539412448,
+            0.02036132271908089,
+            -0.0021221965958423804,
+            -0.04320396743189005,
+            0.004111179168317842,
+            -0.025352428814134187,
+            -0.01865918186335911,
+            -0.011237699157960728,
+            -0.025047923887906474,
+            -0.04219630696740606,
+            -0.011108496740166623,
+            0.007785006066683012,
+            0.016810642610918525,
+            -0.009081815651299525,
+            0.040965574143584864,
+            0.0073944364393447965,
+            0.015313576815177263,
+            -0.017731360025694955,
+            0.026137777790243826,
+            -0.019314392060123,
+            -0.03108486330987073,
+            0.02676829566273746,
+            -6.259161089416372e-05,
+            0.003053748984540415,
+            0.014358356168574803,
+            2.8186924071913628e-05,
+            -0.006181789640967465,
+            0.01947730374003864,
+            -0.028739300217145797,
+            -0.023460503178221397,
+            0.0012460072421725688,
+            -0.008850395570726385,
+            -0.016271242375881602,
+            -0.035896509136022194,
+            0.005227881053956234,
+            0.03324337105109496,
+            0.004970419987430122,
+            0.008273497228454482,
+            0.01814500594735864,
+            -0.02229593571615308,
+            -0.024230675455380678,
+            0.000639112529814716,
+            -0.010266640429960994,
+            -0.010257247303470066,
+            -0.021302483296731352,
+            0.019150757066356872,
+            0.01651375496287087,
+            -0.00390189952706486,
+            -0.006971692709239087,
+            -0.010621003777733253,
+            -0.01918206725910585,
+            -0.02020345113211036,
+            -0.013827322378214754,
+            -0.017980206764311026,
+            -0.034153741666571154,
+            -0.032195001867229935,
+            0.03225205339201753,
+            0.026354817588919596,
+            0.0014076409241492998,
+            -0.007959081015805435,
+            -0.023070419006792155,
+            -0.004483328183581872,
+            0.02879516595345542,
+            0.002549934782150634,
+            -0.0001541849716285838,
+            -0.00830090759167373,
+            0.015341095409025125,
+            0.00835695019244165,
+            0.005107844984733648,
+            -0.06676974053142512,
+            -0.006027540973808174,
+            0.01655283585235268,
+            0.01351155740684989,
+            -0.002359114145814062,
+            0.02591301169926259,
+            0.01108099264396926,
+            -0.015361038608114896,
+            -0.019765743462262668,
+            0.035195761515422364,
+            0.02408910733949848,
+            0.023468767772473746,
+            -0.0034492999385544727,
+            -0.004472311490618405,
+            0.007030833258457338,
+            -0.010684252427867153,
+            -0.013326964647936429,
+            -0.004558513842504658,
+            0.026925146859337397,
+            0.016099443021514896,
+            -0.0064331198280825575,
+            0.010848931851939124,
+            -0.0008320209125061849,
+            -0.019444491995041537,
+            -0.001841330829156697,
+            0.008730275423516445,
+            0.0012974082030866743,
+            -0.014611902493400719,
+            -0.005911807768916134,
+            -0.019130604101278263,
+            0.024719468317569562,
+            0.0209456329723249,
+            0.03381727436368338,
+            -0.013237902591972769,
+            -0.0003695142331921278,
+            0.005410360755058108,
+            0.0002841632253252816,
+            0.0023352645111704958,
+            0.004801123870325242,
+            0.02623649813035263,
+            -0.029010710171522985,
+            -0.020741774066460415,
+            0.0068426980682519505,
+            0.013445270043414486,
+            0.0022744032984779294,
+            0.033840485942696216,
+            0.035040080489222324,
+            0.004166066423508316,
+            -0.010046666684988869,
+            -0.006148765564658713,
+            0.007071740742464309,
+            0.01865897335451031,
+            -0.023267411753900076,
+            -0.0005936657905496225,
+            0.004833030519903764,
+            -0.004229582648167378,
+            0.03620610659071198,
+            -0.009961238880412731,
+            -0.007197670122096852,
+            0.022212791947347127,
+            0.09819205436926473,
+            0.00047695621444943017,
+            0.0025354521225913395,
+            -0.029358210234742176,
+            -0.023614016211567963,
+            -0.0012564198366457442,
+            0.00242680708457405,
+            0.00830167274308547,
+            0.028076260517106013,
+            0.0004852400049405906,
+            0.011579274037419117,
+            0.02773972145320728,
+            -0.017217305534161797,
+            0.006752070301141707,
+            -0.011376051474975524,
+            0.024577071515205502,
+            -0.012694034026935636,
+            -0.0019809719792138673,
+            -0.016512649783050146,
+            0.015907402258634816,
+            0.004556574687833768,
+            0.004907510649166062,
+            0.011347630984408937,
+            0.009824484201337644,
+            -0.02650731900265866,
+            0.01756773113723491,
+            -0.005671986127099093,
+            0.004449611508541289,
+            -0.027841346572384462,
+            -0.020124233970502267,
+            0.0025986391930124847,
+            -0.0014440758308989809,
+            -0.009579444248630085,
+            0.006005274872617889,
+            -0.0024984235450999127,
+            0.03765664613203687,
+            -0.03765496603611941,
+            0.01409350051861866,
+            0.0066453175608372555,
+            -0.0015390841089599882,
+            0.011806653942640485,
+            0.017306914776093522,
+            -0.008206256939290151,
+            0.02589800448884443,
+            -0.00798213351353373,
+            0.009846274212000523,
+            0.015230070803055296,
+            -0.03449531102975745,
+            0.009174739184806312,
+            0.010145762060174144,
+            -0.024165660737765746,
+            0.012146877603301336,
+            -0.013784371016298891,
+            -0.004818759788054113,
+            0.005930446700613062,
+            0.007349837554764034,
+            0.010733731974181452,
+            -0.02295571464086674,
+            0.016176424826629898,
+            -0.010947841926210105,
+            0.0037766665765835445,
+            0.013559846998199008,
+            0.024892849808979323,
+            0.0012426907454129122,
+            0.0005950515136086193,
+            -0.01379469431504576,
+            0.005534724429523084,
+            -0.015667467221535825,
+            -0.022401586867694584,
+            -0.014192041775234079,
+            -0.006366846766730035,
+            0.01029598687444803,
+            -0.003972386708834858,
+            0.016971476088808737,
+            -0.010309648640697812,
+            -0.010645860303528099,
+            0.0037818310327362103,
+            0.01040806357838676,
+            0.008316605962234849,
+            -0.02855914079113881,
+            -0.005676310556150579,
+            0.004961084138163161,
+            0.00029285971503001265,
+            0.009592497954559435,
+            -0.014556073011576487,
+            0.013689526487160009,
+            -0.013290113354287095,
+            -0.0010726537498214612,
+            0.013201860397060236,
+            -0.012443552043930505,
+            -0.004012305744087929,
+            -0.01592905426965635,
+            -0.028487066798154617,
+            0.008462143660968396,
+            -0.01365614159523485,
+            0.001076882584445294,
+            0.0019503240125505646,
+            0.004730085646042709,
+            -0.016919069570642004,
+            -0.007860046623166481,
+            -0.033515448866521046,
+            0.011065416478687504,
+            0.02314589687882036,
+            -0.006435733825418645,
+            -0.030351727858789472,
+            -0.009546333526852862,
+            0.011327489786114963,
+            -0.0071765122743172715,
+            -0.01676736383051139,
+            -0.0029390038145944096,
+            0.007617910650439855,
+            -0.008174620754097493,
+            -0.00508943241317381,
+            -0.0025173969010414098,
+            0.0217328037301181,
+            0.016094975605203832,
+            0.01769642142712967,
+            -0.021400862603034036,
+            -0.00976992221747187,
+            0.026043879119776864,
+            -0.03275550610951898,
+            0.012441465975914529,
+            0.024587446266013183,
+            0.003446527051428522,
+            0.005492228054838714,
+            0.0030377554109693714,
+            0.005459159163772825,
+            -0.006985179226831048,
+            0.019925583798328282,
+            0.010870421035240845,
+            -0.019709062313471467,
+            0.005385443078244229,
+            0.02086085539375208,
+            0.0037248737640193097,
+            0.012892299156840998,
+            -0.017546870090541213,
+            -0.0020341807891543716,
+            0.015594779287017958,
+            0.04816214672670215,
+            -0.0032851797178091685,
+            -0.010450846098750195,
+            0.029513742590677183,
+            0.023526004023637855,
+            0.005373455940309873,
+            0.0012120777808635214,
+            -0.02525630456877349,
+            -0.013075130701774785,
+            0.01962235086017781,
+            0.016944870494393874,
+            0.013484925533584295,
+            -0.0023255656480479004,
+            -0.019236081983457366,
+            0.021520156296467717,
+            -0.01701733319731161,
+            0.012422659010697477,
+            -0.023764941037007197,
+            -0.001490790833263327,
+            -0.0035419008954878724,
+            0.021998949056452454,
+            0.0021819635638621092,
+            0.008891505304514552,
+            0.024354552127458608,
+            -0.006978548952551799,
+            -0.019526615985982355,
+            -0.0051799020400150305,
+            -0.014578953088901798,
+            0.022506635912517146,
+            -0.024117708614271965,
+            -0.007008641019892289,
+            0.011379795968360019,
+            -0.012362093847309038,
+            -0.0102534051092702,
+            0.001164934138876502,
+            0.017686807394319725,
+            0.0009423993435129515,
+            -0.014784508280754078,
+            0.01944445593807992,
+            -0.0011359271135469456,
+            0.010410915597949927,
+            -0.01921419010977189,
+            0.009390638453648025,
+            -0.016682442516992777,
+            -0.007849184048209236,
+            0.010343008873053859,
+            0.0012604521199837046,
+            0.003948300703645506,
+            0.002592490121754231,
+            0.007773437307969315,
+            -0.03355658426684504,
+            0.022695505487314516,
+            -0.010751606023585792,
+            -0.003538523688786008,
+            -0.0014125330167969218,
+            -0.01678692701672328,
+            -0.0006918162525881059,
+            0.005816502852598842,
+            0.010878358093847386,
+            -0.016045147059003256,
+            -0.00873270513444376,
+            -0.04920244395446931,
+            -0.030314009649638693,
+            0.023156333024481057,
+            0.013733657599720879,
+            -0.010971506342695331,
+            0.018852159415605117,
+            -0.023760853159532587,
+            -0.010587249404421732,
+            0.0011379869499042905,
+            0.023319856329056813,
+            -0.03539290744576506,
+            0.01215369703816749,
+            -0.014493682360886549,
+            -0.02039288669534806,
+            0.009070984596830142,
+            0.011445354735283254,
+            0.0007259852045513916,
+            -0.0011514627555568814,
+            -0.029350945123925453,
+            -0.010246845821727087,
+            0.02391068177425878,
+            -0.00861718589055971,
+            -0.004723044986826333,
+            0.024716520302409263,
+            0.0033721801437248873,
+            -0.012524663490691517,
+            -0.017955758068233754,
+            0.021612671655463236,
+            -0.026077203994996058,
+            0.007645944309444457,
+            0.010032183011253845,
+            -0.0173991263874487,
+            0.021649599230588117,
+            -0.007529701797485131,
+            0.020327763440441377,
+            0.004227390938169872,
+            -0.011249233530251522,
+            0.020352283974471443,
+            0.013644443758970054,
+            0.004216958800094986,
+            -0.004081669404784771,
+            0.006869353098014179,
+            0.02212732880780841,
+            -0.011021739614266063,
+            0.02219580717345023,
+            0.028777213295471067,
+            0.00017328191188631918,
+            0.03260179554743322,
+            -0.021287655520443285,
+            -0.03159728983004563,
+            0.021740163175540124,
+            -0.01035146858119169,
+            -0.021564906188712388,
+            0.028914245422221707,
+            -0.013674731836508768,
+            -0.00786476052825481,
+            -0.0166718429993922,
+            0.00533027311342322,
+            0.016680336311088843,
+            -0.005246190451159961,
+            -0.014360934066628332,
+            0.0029628103834216345,
+            0.019809487418628386,
+            0.014587234544007958,
+            0.029780009648869975,
+            -0.010778199815920887,
+            0.035180242185168895,
+            -0.002320576872571577,
+            0.008990157317473909,
+            0.012415465232073404,
+            0.006489435152972723,
+            -0.011251531176923445,
+            0.013501894212117511,
+            -0.003446759011266526,
+            0.008230548711588447,
+            -0.001449023474280409,
+            -0.035111679577202855,
+            -0.013709382199848678,
+            -0.04709294504969336,
+            -0.0023752976043235697,
+            -0.0025098360125255215,
+            0.0030975417740913154,
+            0.09736815869389318,
+            -0.005723049647039619,
+            -0.011988817571623457,
+            0.001625059666070763,
+            0.009611697638805556,
+            -0.0009896494420056814,
+            -0.007606836952841873,
+            0.015973152482488318,
+            -0.01343292635775299,
+            -0.02140928016038583,
+            -0.010532701190806147,
+            -0.01288594454368977,
+            0.012079198190522715,
+            0.015929716343161652,
+            -0.0005947359502228821,
+            -0.015207331663761012,
+            0.006524010655124294,
+            -0.004566385899752684,
+            0.0064717704997400245,
+            0.008647099975176308,
+            -0.010414668624343924,
+            -0.0071698879102925555,
+            0.030899143952119196,
+            -0.025301269549419674,
+            -0.024361748354052457,
+            -0.0010579931222861095,
+            -0.009055796737486008,
+            -0.010335459737012047,
+            -0.002692653633719985,
+            -0.0017679501322462056,
+            -0.010519994985692081,
+            -0.010019865263564572,
+            0.006537076627706128,
+            -0.015472130358273033,
+            0.004812857776601667,
+            -0.0047457011269495634,
+            -0.013296069496526007,
+            0.017054968086538724,
+            0.022563785357424853,
+            -0.02841491200807702,
+            0.0006755428946168761,
+            -0.017422510783648978,
+            0.012941211381006863,
+            0.008866069010589792,
+            0.006049618782357251,
+            -0.022451528209097505,
+            -0.023622906582403243,
+            -0.015019272754405791,
+            -0.01927535750568903,
+            -0.024973490129133732,
+            -0.009073381957684092,
+            0.0067692803507600945,
+            -0.001267339788153158,
+            0.021918281111033645,
+            0.000816538714279555,
+            -0.01812545613159305,
+            0.0061084913570894905,
+            0.000568371220952269,
+            0.018292497018668842,
+            -0.027468677288882186,
+            0.0033806422195878524,
+            -0.02307582511848465,
+            0.033508234091571124,
+            0.0039195050746594975,
+            -0.026680111514794824,
+            -0.021558668725755275,
+            -0.003311830287529699,
+            0.018994577319066473,
+            -0.0025745912019675805,
+            -0.00724049447539589,
+            0.014775207617625293,
+            0.0155334760323838,
+            -0.004693118176196483,
+            0.002015625406296403,
+            0.027968079002437102,
+            0.01077998866180723,
+            0.0313892866933904,
+            -0.015236169297590675,
+            0.004945828495617267,
+            -0.027215016713379062,
+            0.010214419950212732,
+            -0.01503548137805401,
+            0.008859554812646242,
+            -0.01290890682557532,
+            0.03531384210182567,
+            -0.012005613861257432,
+            -0.004250698950192301,
+            0.03253288796441852,
+            -0.029150986393345252,
+            -0.0035020051536906334,
+            -0.009255437891304041,
+            -0.004142204892160768,
+            0.013240580426083011,
+            -0.001198093874819309,
+            0.008485141356876356,
+            -0.01854370462655459,
+            9.726096864073609e-05,
+            -0.003221852304410318,
+            0.005404595442127598,
+            -0.02005458688172967,
+            -0.003314118809778734,
+            0.0073427568419969814,
+            0.004259518237708634,
+            0.002408630356035373,
+            0.016420646142725346,
+            0.006662121128241274,
+            -0.0029357701493526907,
+            0.03424224243607692,
+            0.008171895850772245,
+            0.0152521424877615,
+            -0.0018638264193202728,
+            0.0030234211261443034,
+            0.019287963516760636,
+            0.019912956661192856,
+            -0.00048443399729935116,
+            -0.01058935153883094,
+            0.00195113175754497,
+            -0.001140732137433033,
+            0.011048741602327437,
+            0.024396046054912645,
+            -0.021158025221644738,
+            -0.02419562092713724,
+            0.016377661005279716,
+            -0.015076145951740961,
+            0.01903712233378994,
+            0.010541463112889985,
+            -0.010099351302095792,
+            -0.00031558616108876247,
+            -0.042316016201379245,
+            0.010964989031040692,
+            -0.02068939391415103,
+            -0.04004120028915868,
+            -0.00244383573783919,
+            0.0009542507389088937,
+            -0.029439456243505052,
+            0.02822814855547978,
+            0.0023755032182188156,
+            -0.017280221740813364,
+            0.018223709137876153,
+            -0.005548654680726505,
+            -0.0024040301239229064,
+            0.005397711787238679,
+            -0.01092026090471121,
+            -0.010648485086769927,
+            0.008235250252660972,
+            0.010957715075632535,
+            0.005425683656535705,
+            -0.005698919020608095,
+            -0.011284911899894062,
+            0.009975882651638084,
+            0.012318901214538077,
+            -0.0019922884373083155,
+            0.010050520101815593,
+            0.0031720752428247446,
+            0.007352859750323051,
+            -0.0073483049386141005,
+            0.0046483587496626965,
+            -0.005093220128411125,
+            -0.026080057085901446,
+            0.012863580653903103,
+            0.0052434400313520774,
+            -0.011863360599637747,
+            0.0103209502958491,
+            -0.01048162756158923,
+            0.005787157766911021,
+            0.001318945460232246,
+            -0.0059333230422061395,
+            0.004821320238893602,
+            -0.0038922239931512784,
+            0.011267634571983924,
+            0.013256611037442042,
+            0.004592529900805719,
+            -0.013076954459595086,
+            -0.02177318635989856,
+            0.009001291224623467,
+            0.01959767163356408,
+            0.024161754192119852,
+            -0.030416109293488064,
+            0.003784634588325551,
+            -0.0014915849060829743,
+            -0.0020550564975080885,
+            -0.015632785947151753,
+            0.01587790981601866,
+            0.01716742056401084,
+            0.012721563202528083,
+            0.01893393006967701,
+            -0.004009617703771589,
+            0.010539100665843704,
+            -0.02080068888791658,
+            0.013642451766699067,
+            0.0008609869904066794,
+            -0.013629067231663643,
+            0.0014620655123815298,
+            0.027046304278766087,
+            -0.011056783825898616,
+            0.006206969699250696,
+            0.008097527175511791,
+            0.022111870736907183,
+            -0.012509479100313492,
+            -0.003735868296238069,
+            0.01516254968167132,
+            0.0014878674796313539,
+            0.0035887079417890352,
+            0.016533149527222273,
+            0.04729052276110659,
+            0.00102751186013391,
+            0.0016713773449738153,
+            0.0192177920520243,
+            -0.005563519063161466,
+            -0.01114821419109643,
+            -0.003549999863325982,
+            -0.021807893906469106,
+            0.010998558936481785,
+            0.00431279785115251,
+            -0.02186324212032519,
+            0.01024093366826206,
+            0.0125495473550584,
+            0.005268320135120931,
+            -0.0031679212410026603,
+            0.02574575247827441,
+            -0.019807879421996258,
+            0.014935295894965128,
+            0.03329450628384787,
+            0.00493356778223407,
+            -0.011423949333937285,
+            0.007263264249871384,
+            0.0027354459678742013,
+            -0.004494406041906684,
+            -0.00606745413660583,
+            0.010497391290734088,
+            -0.008191702034391382,
+            0.009870458191510176,
+            -0.009580130479693,
+            0.039202091799313755,
+            -0.009619018331262844,
+            -0.010782138248435458,
+            -0.004558645539551881,
+            -0.006241773108460662,
+            0.003478414182938609,
+            -0.003395165694642073,
+            0.005812225135633631,
+            -0.0069045407353789945,
+            -0.014607146220271209,
+            0.0008522982757210454,
+            0.00020175973815948303,
+            0.014001922919116725,
+            0.0011804904415274088,
+            -0.004871874532755353,
+            -0.007877217154706205,
+            0.015243277056007988,
+            -0.006047906208304912,
+            0.0006531937033317192,
+            -0.0011653523616313422,
+            -0.0013662556399359335,
+            -0.007401050012179761,
+            -0.015166105795085528,
+            0.00974609244984173,
+            0.016253733908242134,
+            -0.013857834499211056,
+            -0.005142936208363415,
+            0.012274716212832219,
+            0.008540229125193836,
+            0.011742606168187876,
+            0.0007451338650642946,
+            -0.009255813682572937,
+            0.008292531177374192,
+            -0.015020523035432158,
+            -0.008781317965985481,
+            0.0026250469418136603,
+            0.002381147382977664,
+            0.005839723981046825,
+            0.0006240641828668715,
+            0.004502463508605272,
+            0.005798912705218846,
+            -0.00497936644919367,
+            0.01112666823857836,
+            0.008198424171006548,
+            -0.002452143482105151,
+            -0.01017569410837814,
+            -0.001127589984272638,
+            -0.015733298167661346,
+            -0.016758315411028353,
+            -0.0009286266498936771,
+            0.015175988113750882,
+            -0.02352862898718829,
+            0.018996352975391644,
+            -0.0014524363611708201,
+            -0.0033802546171857704,
+            -0.021392428456030312,
+            -0.0025923983911570183,
+            0.0034214188668518796,
+            0.01255768230394476,
+            0.01858853630745928,
+            -0.02729543519465384,
+            0.010575894388563116,
+            -0.007204725245442141,
+            0.030394929424014717,
+            -0.018328782140005044,
+            -0.004022941273610227,
+            0.026915618079621992,
+            -0.02200018306828758,
+            0.0008791185377513978,
+            -0.011454606955158956,
+            -0.008103032693853504,
+            0.019287674837688177,
+            0.023747720480736777,
+            0.02038287242167783,
+            0.01364489294869211,
+            0.0007758044919789085,
+            0.004768030222445397,
+            0.03572705648994403,
+            0.010568241270862248,
+            0.014828087092859878,
+            -0.007542489523110974,
+            9.457808640264811e-05,
+            -0.02554033312739386,
+            0.01635899434369978,
+            2.587383534566272e-05,
+            -0.012370727506722627,
+            0.007162422186621379,
+            -0.013370882403931942,
+            -0.011066483600541058,
+            -0.007151077450730025,
+            -0.01721910018182534,
+            0.02152239849400241,
+            -0.012322440280680164,
+            0.004587729883596212,
+            -0.009079747696669277,
+            -0.013272707286450417,
+            -0.012953682633400515,
+            -0.014202214807187545,
+            -0.01313141988034687,
+            -0.026041403779538307,
+            -0.027109731260915763,
+            -0.006573285295895077,
+            -0.023780366346334524,
+            -0.013799072648505526,
+            0.005588622594136253,
+            -0.022617902666316757,
+            0.012262325666428604,
+            -0.012689827812482627,
+            -0.01226264872059864,
+            0.012805734807835008,
+            0.020274844315616616,
+            -0.011765231803328348,
+            -0.028494489703400308,
+            -0.018583843821043854,
+            -0.019970998470704358,
+            0.012576201137505969,
+            0.002266288538944273,
+            0.013165889075444961,
+            -0.003018437854660793,
+            0.003227412667880239,
+            -0.014895831590298957,
+            0.018570624675927183,
+            -0.03442390703780963,
+            -0.03108003806963536,
+            -0.013611306765348361,
+            0.005182777066476028,
+            -0.0015782033203578553,
+            -0.011429046504256945,
+            -0.0018606648572920801,
+            -0.00433805271182878,
+            -0.025109572024446088,
+            -0.010134856632190067,
+            0.011010662054484161,
+            -0.015205629927469414,
+            0.0072933565255675036,
+            0.0009018832564038863,
+            0.013870902834260887,
+            0.009587432944616442,
+            0.013393301893492882,
+            -0.006095063698384775,
+            0.004356485794113208,
+            -0.0026476612747324386,
+            0.0004894024149759108,
+            0.009336366919474478,
+            0.00839591561027103,
+            0.010551161369079399,
+            0.002260707777260338,
+            0.007830059223001696,
+            -0.0060927446216633265,
+            0.018065002687169148,
+            0.015436944875538481,
+            -0.0002520689442496089,
+            0.013323098485251487,
+            0.0003769862591044255,
+            0.00022293304921127018,
+            -0.0029970303106340466,
+            0.007888437018607987,
+            -0.011785957933642163,
+            -0.001311704406355858,
+            0.015138190893483804,
+            -0.01707247730569978,
+            -0.013045187198628258,
+            0.03107034781283638,
+            0.0018583638488268697,
+            -0.02495564203304796,
+            0.031146554085787762,
+            0.004220047610942943,
+            -0.005835199594223787,
+            -0.013205086624848782,
+            0.0014059047822222418,
+            -0.011570624844100298,
+            0.003584232809362274,
+            0.014295032061446826,
+            -0.02231984223610487,
+            0.010521094801839404,
+            0.008446892441796151,
+            -0.009681535919980559,
+            -0.012337659795586223,
+            0.009847073709209602,
+            0.016809328445200358,
+            0.004472485504034416,
+            -0.03539093810210523,
+            -0.011957694102753411,
+            0.02145470732008716,
+            -0.029237428225009913,
+            -0.00810041956259673,
+            0.007452962400105787,
+            0.0065154832040426065,
+            -0.019353109017609053,
+            -0.008580955020412362,
+            0.005725794605897509,
+            -0.004940390374637084,
+            -0.003892210172017758,
+            0.017541623128534682,
+            0.00485047043626186,
+            0.013698341792101025,
+            -0.01689738578307474,
+            0.009771458057400557,
+            -0.022222817828602557,
+            0.004021882074676753,
+            -0.023373700500461992,
+            -0.009873403167702603,
+            0.002364143609079849,
+            0.012454271200139762,
+            0.014328556467632212,
+            0.027839363143659326,
+            0.030056404238348527,
+            0.013995696971422429,
+            0.007833677585250953,
+            -0.016265199386953717,
+            -0.018134327237616175,
+            -0.006409513840075933,
+            -0.004033173350204784,
+            -0.008714798059218596,
+            0.0066373547199279235,
+            0.01858354212009877,
+            0.004499712802472377,
+            0.02609078206670711,
+            -0.006227189596272876,
+            0.009039744622014585,
+            -3.127526911951914e-05,
+            0.0008833930131827993,
+            -0.004226083827636182,
+            -0.01971685546584527,
+            0.00481003211084742,
+            -0.0013721482162825508,
+            0.025874758880591167,
+            0.00940496860916206,
+            0.0024915013399541847,
+            0.01615978314811047,
+            -0.006378952757397484,
+            -0.01174653607178726,
+            -0.010540107733238588,
+            -0.0061926268398085176,
+            -0.021752582127981104,
+            0.029059100503124387,
+            0.021195595568253612,
+            -0.010800765264123782,
+            -0.013693582166598082,
+            -0.00903369526059487,
+            0.02795767990556847,
+            0.004080965688836953,
+            -0.0013050212744006933,
+            0.016313311850308505,
+            0.012084555958999215,
+            0.012329325712491252,
+            0.009794698207689061,
+            -0.00036157148861933123,
+            0.0015364709122308352,
+            0.03722522141440073,
+            -0.007427028510548186,
+            0.0026463292682538495,
+            -0.005239324411651515,
+            0.0025987932868461313,
+            -0.009705817520422487,
+            0.007242122655628283,
+            -0.004012676385296197,
+            -0.0158760051470016,
+            0.015003444174308607,
+            -0.022443331305169994,
+            -0.0016749787204471238,
+            -0.008150635190952799,
+            0.0016696640640947303,
+            -0.02781023337840043,
+            -0.0025731206607371845,
+            0.006695065701630532,
+            -0.015547340828489069,
+            0.011816010501093885,
+            -0.01740673229051557,
+            -0.011553604323149105,
+            0.007914961618894248,
+            -0.012034157288065447,
+            0.0016684130271324446,
+            -0.0008268957621923278,
+            0.004460864332388434,
+            0.013426055079012884,
+            -0.011008808570221245,
+            0.012166046631826273,
+            -0.005816284310138776,
+            -0.007872091214834688,
+            -0.0016055129487028885,
+            -0.0030049472973942043,
+            -0.0025600746761171543,
+            -0.0009966231170649917,
+            0.034355910234323155,
+            -0.004446570896048543,
+            0.003761242147499512,
+            -0.01362639743248326,
+            0.0019833052855181806,
+            -0.036176376394999044,
+            0.028344464817309477,
+            -0.008642835889481802,
+            0.011013502221008498,
+            -0.0017508365480865596,
+            -0.013243706268954915,
+            -0.01456517309747086,
+            0.015247299501245178,
+            0.007602596572394442,
+            -0.005854531057349665,
+            0.013558248087347848,
+            -0.019724373614448298,
+            0.008530500822910953,
+            0.004463694543554244,
+            -0.015561397976987216,
+            0.004668780190328835,
+            -0.007479912658259757,
+            0.012657201279639194,
+            -0.01774730033362932,
+            0.004314594593150765,
+            0.00958571617187387,
+            -0.01892876084898881,
+            0.009901041800918215,
+            -0.028675879564512306,
+            -0.004631920242208613,
+            -0.012323455524366223,
+            0.0053804608827285865,
+            -0.000660864620708182,
+            -0.02389088252276507,
+            -0.028944280660230762,
+            -0.0002099667627277945,
+            0.00458533068247518,
+            0.0018878351997997833,
+            0.011621407953503755,
+            0.005970807015151215,
+            0.008542958629199003,
+            0.01023896054756221,
+            -0.00205207247588841,
+            0.009402752124876478,
+            0.04169706217414899,
+            -0.022899912321317197,
+            -0.011574970753549647,
+            0.005321872977369877,
+            0.011499271505460519,
+            -0.02647653483798284,
+            -0.011039791758347888,
+            0.0033424345263339144,
+            -0.005219877979550397,
+            0.03609469691834737,
+            -0.02448746405881972,
+            0.019020618652020673,
+            -0.011392297116547224,
+            0.007625475937968714,
+            -0.021465240076078376,
+            -0.003921088264697311,
+            -0.014265745991522477,
+            -0.00011221615798127438,
+            -0.0031478236827989703,
+            -0.0076053733863387755,
+            0.007485039021296698,
+            -0.001907678661303243,
+            0.0020348246022161925,
+            0.006291436821831794,
+            -0.02173305455485444,
+            -0.0019125499969493854,
+            -0.016432460924913166,
+            0.030367263349759286,
+            -0.016242095671439496,
+            0.005072805486919414,
+            0.0053258139728101506,
+            -0.002810639419737258,
+            0.012368771814660942,
+            0.004165967191536186,
+            0.00513754440091394,
+            -0.004199767241827414,
+            -0.005142642360546972,
+            -0.0053886448467479215,
+            0.017868874549760376,
+            -0.013280048598849825,
+            0.035024701466710065,
+            -0.0100959186045929,
+            -0.007094001877261568,
+            0.0019733042325232155,
+            0.010380430710282744,
+            4.2084202709296443e-05,
+            0.01053565670076933,
+            0.0034270489770133685,
+            -0.0006311078385359693,
+            0.0019774779328212756,
+            -0.017162448473652843,
+            0.003089758682621882,
+            0.01682656561693936,
+            0.005463677441664502,
+            0.004469261499451007,
+            -0.013332130891003062,
+            -0.012050233723282738,
+            0.004538268704442613,
+            -0.007533940408255447,
+            0.0003675737140813486,
+            -0.018816953644773076,
+            0.013972696748575243,
+            -0.0011096683686631858,
+            0.0002715683052236706,
+            0.00524678369365015,
+            0.023056514997908218,
+            0.01813167860994747,
+            -0.002553352602551003,
+            0.03155164978556312,
+            0.0174738879054593,
+            0.0029268039949260673,
+            0.010637295711777793,
+            -0.033222781195984666,
+            -0.013301873512972214,
+            -0.0013721703891642354,
+            0.02238421933149517,
+            -0.01530189703335162,
+            -0.014059029663525449,
+            -0.0037393534403749947,
+            0.0010394601933309767,
+            -0.005177359933828505,
+            0.0042489306011644495,
+            0.011039889511148341,
+            -0.014697595240068863,
+            0.007597149702441915,
+            0.0026780496822907634,
+            0.02092734730183164,
+            0.0004688144432243786,
+            0.017181410417231698,
+            -0.0005083970646221274,
+            0.00749639055342427,
+            0.007319420826337891,
+            0.017541463125355265,
+            -0.01002811389296805,
+            0.023582113845081534,
+            -0.008823933397729605,
+            0.0019889258902730535,
+            -0.01257106158422629,
+            -0.028159594067544737,
+            -0.012624750220149823,
+            0.015247564545279736,
+            0.005765911899842925,
+            0.006478299226263732,
+            -0.016537090297144265,
+            0.006276997440048981,
+            0.015647122171767974,
+            -0.002251683151059454,
+            0.010295358869874306,
+            0.009730378299068289,
+            0.00012816405890877098,
+            0.010181120688239766,
+            -0.00801256937783534,
+            -0.00734257767723293,
+            -0.004497520124843868,
+            0.023982754195729983,
+            -0.005443550092842393,
+            0.022434177122972028,
+            0.012647958478747348,
+            0.00616791857573897,
+            -0.010853355357954727,
+            -0.003340111254357573,
+            0.023389634440057473,
+            0.011863246790550376,
+            0.00029642699874398923,
+            -0.009846034060495678,
+            0.0019458876579524617,
+            0.0054010373765486005,
+            0.012983913038716568,
+            -0.02049427270240938,
+            -0.004344571352918878,
+            0.008461947538684664,
+            -0.008472919299762712,
+            0.004318309005722874,
+            -0.030156444522730416,
+            -0.005812303237222271,
+            0.021354352236942434,
+            0.008392281015122372,
+            -0.010232039722494646,
+            -0.004395682831498155,
+            -0.0006527587802367882,
+            0.0040601850269714945,
+            -0.0024323167053212176,
+            -0.0023729138545912176,
+            -0.008541233856441836,
+            -0.006470933419694742,
+            0.014035696617797647,
+            -0.0052691760528871924,
+            0.008597459268477037,
+            0.00012084859681905111,
+            -0.0017255958688825231,
+            -0.007323529232751411,
+            0.03382792341957058,
+            0.008087732655845718,
+            0.005980666974986827,
+            0.0013613625176327891,
+            0.0042244330945601215,
+            0.000616791811672998,
+            -0.0028374856646453707,
+            0.018075360591135542,
+            0.0156738939725117,
+            0.005054696911814988,
+            0.009038525980341447,
+            0.014672270369917403,
+            -0.019700405219230957,
+            0.012625337070556061,
+            0.012184185186281482,
+            0.019382321278525114,
+            0.0007219498737851579,
+            0.002564970959687783,
+            -0.010601873823737926,
+            -0.0029963537053764736,
+            -0.002070567089566093,
+            0.029563028293936092,
+            0.0005903976560069006,
+            -0.010509758568838935,
+            0.010992421945232951,
+            0.020603868018231348,
+            -0.02101157589451638,
+            -0.001971974816402967,
+            0.005963911670920596,
+            -0.010704207628592966,
+            -0.014466113540150018,
+            -0.021269176952661614,
+            -0.013331247081394942,
+            0.0071597345125287174,
+            -0.008162102923896894,
+            0.02367549407145414,
+            -0.014256771777502395,
+            -0.03960796112042958,
+            -0.014803626428839152,
+            -0.00864320347698291,
+            -0.027531337207277483,
+            -0.007305441146142331,
+            0.00965056093889974,
+            0.016698589600806273,
+            -0.003330867073501157,
+            -0.025198703196464594,
+            0.030483790406960656,
+            0.008872821868559751,
+            0.023708422767352917,
+            -0.021464374508407488,
+            -0.0018617153194555344,
+            -0.00886376784514331,
+            0.014505918224188001,
+            -0.017588761818663975,
+            -0.011756112651591795,
+            0.020014632992762026,
+            0.014356419813769714,
+            0.007246557500044434,
+            -0.004355132555323843,
+            0.0067299398462013936,
+            0.007649499235136303,
+            -0.012515181210363267,
+            0.012239792251154642,
+            -0.013499277313916029,
+            -0.024858295126228433,
+            0.014430561986516884,
+            0.002229494935179828,
+            0.009563004428064055,
+            -0.0015984171085659323,
+            -0.016197659608768088,
+            0.010078159490026602,
+            0.00808729175515505,
+            -0.017508840440512225,
+            -0.004793934415922065,
+            0.007133527615305415,
+            -0.020818869260580315,
+            0.031234686259160864,
+            0.011872765152279094,
+            -0.010692949770503467,
+            -0.001989039629661815,
+            0.004431079028880923,
+            0.016511544608975148,
+            0.007893700385963683,
+            -0.01997314388229276,
+            -0.003708006454566117,
+            0.01530128306203401,
+            0.00822588305477249,
+            0.004861703074728471,
+            -0.010836328783594584,
+            0.03434874763691911,
+            0.006082873924472328,
+            4.192481378506604e-05,
+            -0.0052997095110967115,
+            -0.0076108924981838355,
+            0.004196112457289409,
+            0.018905391806961686,
+            0.013330078288599763,
+            -0.004200590783114919,
+            -0.006630107527854746,
+            0.007778448128358027,
+            -0.01706991160178345,
+            0.007329056462125937,
+            0.009390327499494826,
+            -0.012870674959022891,
+            -0.0017571189976160295,
+            0.016462608870101713,
+            0.00209846661139665,
+            0.004942714850185376,
+            -0.011305050739130339,
+            0.004252776146827917,
+            0.015841821944109317,
+            0.018727567181082646,
+            -0.008399642221255733,
+            0.021448055799259867,
+            -0.020972459569228434,
+            0.027243639465535856,
+            0.028499400850980086,
+            0.016573144403275928,
+            -0.01847096443459798,
+            -0.0022409373140116146,
+            -0.0041737366139895414,
+            -0.014761716560267328,
+            0.0034546020161337894,
+            0.005125870074458691,
+            -0.01174186608185822,
+            -0.009577921232380895,
+            -0.024331993794416026,
+            0.011075959943215117,
+            0.007918147902741738,
+            -0.0015045185909419564,
+            -0.011791155919533844,
+            0.016019077577552835,
+            0.010141868950981925,
+            -0.010938402325619172,
+            -0.005635351352447256,
+            -0.019967361901734836,
+            0.009750427567651833,
+            0.009128294017175602,
+            -0.00300556760891084,
+            -0.011496103417643132,
+            0.012977890524465716,
+            0.0014080059843484115,
+            -0.01292380658817521,
+            0.010841954636959441,
+            -0.03142011651894961,
+            -0.005510537354374977,
+            -0.011027026734432255,
+            -0.003620482919145167,
+            -0.014415606143566519,
+            0.003608572766363589,
+            0.0005034611321129689,
+            -0.011470488351158329,
+            -0.0035005774696278,
+            -0.016207453351921627,
+            0.004676327282262429,
+            0.015021513761697039,
+            0.001679614645967629,
+            -0.006917375094634678,
+            0.021994274901326064,
+            0.018518524483306403,
+            -0.021903538417919984,
+            -0.020116013379669034,
+            0.004708084983787958,
+            -0.03261690161800251,
+            0.015019348187107588,
+            0.00966124471577622,
+            0.011676937880751252,
+            -0.028768229226408037,
+            0.007114534979415504,
+            0.00900420165933673,
+            0.016398927481511824,
+            0.0050879301160439245,
+            0.009453255486450082,
+            0.01222758288108498,
+            -0.0104657614414275,
+            -0.003164502778525433,
+            -5.410382753651127e-06,
+            -0.005618225986727351,
+            0.015607620507415484,
+            0.0008527697935056332,
+            0.03603666488542239,
+            -0.005277207843139309,
+            -0.0009799879252482288,
+            -0.011735731235560851,
+            0.00812046801749312,
+            -0.011713349784345638,
+            0.006334172347602515,
+            -0.0024643499727789167,
+            0.0001240009282616789,
+            0.002965067818077541,
+            0.009442572075213767,
+            -0.01086095783353607,
+            -0.007595379345224076,
+            0.007444897256325813,
+            -0.01482814340478338,
+            -0.00019984717819268013,
+            -0.009339343730980451,
+            0.0019680804106275156,
+            -0.02241031895876346,
+            0.006629838013824249,
+            0.040935644507527846,
+            0.02140216465404127,
+            -0.005481040712269146,
+            -0.012431659209559418,
+            -0.004680230194268222,
+            -0.007229300034633229,
+            -0.006482401275991039,
+            -0.004794698160419071,
+            -0.022762405726589375,
+            -0.006928553505112732,
+            0.004320321244630314,
+            0.013300138724212552,
+            0.009741018117644266,
+            -0.008842135469729511,
+            0.004531760574809748,
+            -0.02662650543210291,
+            -0.016578081411548973,
+            -0.0060021425889846495,
+            -0.015840130031005147,
+            0.006125301421471048,
+            0.027521841635711787,
+            -0.0063432924325983525,
+            0.01789356321527744,
+            0.005142029893259461,
+            0.013034833065163448,
+            -0.005338869351330258,
+            0.00035250243782976347,
+            -0.006010938630154189,
+            0.0032958656042048292,
+            0.003074861351445954,
+            -0.0015947217852680714,
+            0.004151590621941346,
+            -0.007139285822575166,
+            -0.0026091937868967284,
+            0.0011629380311555433,
+            0.0117841884590861,
+            0.006877419488576261,
+            -0.00016252653621991807,
+            -0.01761285866765538,
+            0.022221104524808997,
+            -0.018933508559881868,
+            -0.008840161102111973,
+            0.011092684029552064,
+            -0.009660487856185668,
+            0.00702597023411946,
+            0.020387386977378873,
+            -0.006854213562767843,
+            -0.009583981919303215,
+            0.013028181087649146,
+            -0.0036507672754997286,
+            0.004464368543493108,
+            0.009867428978560914,
+            0.006036303973414791,
+            0.002482944923100156,
+            0.013244650094892797,
+            0.004530294330688855,
+            -0.0032625482538717137,
+            -0.020238533752799943,
+            -0.005186498720507101,
+            0.019015752262933585,
+            -0.02252458872825473,
+            -0.0027641215988942563,
+            0.0044506144202026456,
+            -0.0030317840657635457,
+            -0.01733962766050954,
+            0.004205861489436576,
+            -0.018770275720867473,
+            -0.0033883519277721433,
+            -0.008244668812709337,
+            0.009948090119054079,
+            -0.023700732627507225,
+            0.009109315916173436,
+            -0.011739192106106307,
+            0.02356509175177699,
+            -0.003925851077921686,
+            0.024734433836185024,
+            0.0015142645677087268,
+            -0.013482933030628261,
+            0.0055323884713736554,
+            -0.0008930661843169134,
+            -0.004477731568313757,
+            0.029338490404300612,
+            -0.005418258701133154,
+            0.002249996261871466,
+            0.013491124734151694,
+            -0.021332541202417246,
+            0.021773687217263173,
+            0.0046662696076723836,
+            0.01628697326539432,
+            -0.00986238951830334,
+            -0.013799209313714441,
+            0.025178874553620743,
+            0.03324555035648896,
+            0.008398796308070151,
+            0.011261509865513953,
+            -0.01226890254177634,
+            0.018966175575083295,
+            -0.0038328997984309597,
+            0.004344837325292706,
+            0.00878668880458693,
+            -0.007200842513742962,
+            -0.011500562922303248,
+            -0.004533014670236755,
+            0.027639894892383943,
+            -0.021838848944317424,
+            -0.008920629621611768,
+            -0.005739181836252042,
+            -0.01593272996094732,
+            -0.038167534267554606,
+            -0.004371701247332816,
+            0.019604405563120094,
+            -0.005974358141737469,
+            -0.0018341151594086288,
+            0.014334266690716505,
+            0.00402620072221496,
+            -0.002852569009185304,
+            -0.008195218707838913,
+            -0.0027944153624309856,
+            -0.0059390492900516225,
+            0.016651206007022145,
+            -0.004961864528562493,
+            -0.007929830585906699,
+            0.01145431100211762,
+            -0.005731957927601062,
+            0.012130694936714224,
+            0.0004848418056781251,
+            8.392048424584131e-05,
+            -0.0015174120840537315,
+            0.00822932723013239,
+            0.018620869354789082,
+            -0.0055803454666417465,
+            0.025682778991091174,
+            0.006956936033581638,
+            -0.003495709501259274,
+            -0.005489431577052735,
+            -0.0044154293119170445,
+            -0.024561016942073285,
+            -0.0020864636079347357,
+            0.006593060072668424,
+            0.024333813203596122,
+            -0.02515989187381735,
+            -0.004953266000463339,
+            0.020957806164031402,
+            -0.0012192724005613,
+            -0.011978722479596745,
+            0.014690559986694138,
+            0.0041346025614047875,
+            0.009881584250499358,
+            0.012176393203318002,
+            0.015764530703333238,
+            -0.004923548157246672,
+            -0.0015192775000207833,
+            0.012766482928380375,
+            0.0005745973040874939,
+            0.00997809510696241,
+            0.016495444272181515,
+            0.010467514633877887,
+            -0.021108281220544853,
+            0.0015078843789125156,
+            0.00522593579521476,
+            -0.017481430635971396,
+            0.0034362821987740755,
+            0.005397819166037461,
+            0.011626527721895694,
+            0.004048632382026344,
+            0.013207335578216437,
+            -0.015768003434275563,
+            -0.005916168437120099,
+            0.008153798523776654,
+            0.004319451296835019,
+            -0.02599474886669325,
+            -0.01485009085106089,
+            -0.0006781964116146299,
+            -0.004444144921332929,
+            0.013556265879295017,
+            0.01285768162554067,
+            0.02033626001285354,
+            0.024637349990579514,
+            -0.013833060730799217,
+            -0.00842819657792762,
+            0.009436072455719896,
+            0.015560357465711765,
+            0.005798181411384387,
+            0.007874051721372273,
+            0.009297440071118126,
+            0.0077320823375042,
+            -0.004073209576661068,
+            -0.002973928843567532,
+            0.020205003097225747,
+            -0.019725382393637726,
+            0.0006746659838642147,
+            -0.018417443693472958,
+            0.007530796896254817,
+            0.01140696723506382,
+            0.01129515042745398,
+            0.002816883643872417,
+            -0.005274757308627299,
+            0.005716917047812153,
+            -0.06254973656832136,
+            -0.009191453047502128,
+            0.003384751551032943,
+            0.027256466708746898,
+            -0.010885336140129142,
+            -0.003415707910562931,
+            0.007309577090125959,
+            0.005511970708769456,
+            0.0005572239548625122,
+            0.003186427278034852,
+            0.019698716902282903,
+            -0.02052581021967601,
+            -0.010793968505805883,
+            0.004219269922138548,
+            0.0033111837923746746,
+            -0.007269521534951044,
+            0.004805737092302134,
+            0.014143978208501908,
+            0.01986656063456352,
+            -0.0024048854610842748,
+            0.01629335842915328,
+            -0.0015498603704589015,
+            -0.014496994444493684,
+            -0.002671064325909516,
+            0.0037875833047555925,
+            -0.022466433279066646,
+            -0.005980897597195858,
+            0.0043262697442044435,
+            -0.011226121648007667,
+            0.006978495397818936,
+            0.008452177557316863,
+            -0.003018545465557218,
+            -0.0008303307644324117,
+            0.017796789945838845,
+            0.006591364879905198,
+            -0.013464486184025026,
+            0.0028668426952326957,
+            -0.023377605222213174,
+            0.004325185794796361,
+            0.001416245546006914,
+            0.007809449561523174,
+            -0.008431335139468173,
+            -0.014774499666285603,
+            -0.005933810712528226,
+            0.017746847020610673,
+            -0.01965188730040385,
+            -0.002406875192894708,
+            0.01683009662984443,
+            -0.0002518220110313279,
+            0.005456812494894182,
+            -0.02011241622618092,
+            -0.0009782974741561381,
+            -0.02933296756078644,
+            -0.01171823509606186,
+            0.001162345317872531,
+            0.00892221099991475,
+            -0.006551308022163148,
+            -0.03934356275100932,
+            0.00427623835776202,
+            -0.013111664387776261,
+            -0.008752857644809231,
+            -0.010214546496134218,
+            -0.00518087722852466,
+            0.015240297918667867,
+            -0.012498100033235827,
+            0.01086490299272505,
+            -0.00917777212602538,
+            0.005895055740265625,
+            0.0090554651297836,
+            0.014212841798768113,
+            -0.005544047915369086,
+            -0.0033589999440643385,
+            -0.007924830667307956,
+            0.0022291797354234005,
+            -0.0032976226056924502,
+            0.009615674261248298,
+            -0.0051932155872883,
+            -0.0065636389655770455,
+            0.02426106498596339,
+            0.011445591067574932,
+            0.013312596099532533,
+            0.0055118624272110965,
+            -0.007220620072349783,
+            -0.01916551187613823,
+            -0.003977616240384473,
+            -0.013204506708246246,
+            -0.006315966975649096,
+            0.01434099854765068,
+            0.012588249778961223,
+            0.006357972880218866,
+            0.0007747002406996095,
+            -0.004794439624135275,
+            0.006975593520375694,
+            0.0262335496080837,
+            -0.016481909971520166,
+            -0.0013084662210086592,
+            -0.01369313900978409,
+            0.01921748474212319,
+            0.03333761672336437,
+            -0.0008918419419301244,
+            -0.007699525186523558,
+            0.013757216065348164,
+            0.00819045303047094,
+            -0.007411380297164911,
+            0.02453611080891859,
+            -0.025045462880463647,
+            -0.015989319900023054,
+            -0.011650886878708265,
+            -0.011407253985176972,
+            -0.008386367959227559,
+            0.016331961716602287,
+            -0.00141225438918211,
+            -0.0018505754298268654,
+            0.009557291617446645,
+            0.03357924917013024,
+            0.006916860901290756,
+            0.007030308953738567,
+            -0.002739345982346728,
+            -0.009376522438071562,
+            0.011482637968411055,
+            -0.007259054391879614,
+            -0.04075661435928125,
+            0.004250195164583401,
+            -0.005160505473755424,
+            0.02405623169060588,
+            -0.02197586561375289,
+            -0.00619175761282745,
+            -0.006736129049831973,
+            -0.028200015908139763,
+            -0.005867722104196303,
+            0.02805071895018145,
+            -0.015402198567775478,
+            0.014773294766214671,
+            -0.009502185346558072,
+            -0.0013419858580438616,
+            0.004443749010136947,
+            0.013008783570355656,
+            -0.02289662952366337,
+            0.0013834888740540108,
+            -0.00048549197205067054,
+            0.027969842651517314,
+            0.010437729090975158,
+            -0.0047705656249623744,
+            0.014202218020412936,
+            0.008240861154198411,
+            -0.018637315941614406,
+            0.007314019317343154,
+            -0.005747966765101792,
+            0.014833896739762147,
+            -0.01314847940891154,
+            0.007762100494499689,
+            0.00705026393499352,
+            -0.01278023669088385,
+            0.03016342404000438,
+            -0.015204129985039077,
+            0.007776946006562982,
+            0.009278778571285689,
+            0.009656723862179986,
+            0.01257686993982689,
+            0.004362439504889069,
+            -0.02898418193036562,
+            0.0002144431781454467,
+            -0.03828100004144633,
+            -0.00833748467467171,
+            -0.002611937107022567,
+            -0.002145329666219461,
+            0.007050630951707937,
+            -0.009383645162261629,
+            -0.033976442057557676,
+            0.0006429978367640709,
+            0.01603027598131823,
+            0.017501067413108286,
+            0.0002196997337451293,
+            0.013075379412332973,
+            -0.006197537539554383,
+            -0.021148120787551232,
+            -0.016986448660412497,
+            -0.010207705210051413,
+            -0.02122194744719993,
+            -0.020550470201191853,
+            -0.00908564461857315,
+            0.012090079127547193,
+            -0.009927390005879807,
+            0.013888458967497627,
+            0.029513250508042935,
+            -0.024529175134769883,
+            0.0011144447737101862,
+            0.021977677894839922,
+            -0.00198308986575084,
+            0.014108189311291326,
+            0.001696775006906642,
+            0.014122064251288995,
+            -0.0004166140484018266,
+            -0.010556016685519523,
+            -0.009729362492661684,
+            0.017487929296614076,
+            -0.0026043257634797124,
+            0.016715798948897194,
+            0.012183487338360088,
+            0.003771721868999568,
+            -0.0030887189945795796,
+            0.010667776697461199,
+            -0.026471157050402344,
+            0.014249996582064706,
+            -0.006690961709923224,
+            -0.0181382691440242,
+            -0.008729104191096461,
+            0.003882731567105777,
+            -0.005609384377642956,
+            -0.007794812949125115,
+            0.012967089730551055,
+            -0.00944374111323783,
+            0.02088525981049842,
+            0.0023400121055514958,
+            -0.012196925240774426,
+            0.003648688547842322,
+            -0.03322041209304968,
+            0.0015191765655963997,
+            -0.018402382776013956,
+            -0.024659631472836927,
+            0.03840173429666435,
+            0.011991452465253198,
+            0.006639702644606365,
+            -0.01675838115087676,
+            0.007953102302963526,
+            0.0004257479565099406,
+            0.017484338819046517,
+            0.011068864013925452,
+            0.006466915839538296,
+            0.008364454017976605,
+            -0.023316137291380285,
+            -0.010898421318668592,
+            0.016204569093730564,
+            0.012099473359791148,
+            -0.01017229369980751,
+            0.019431669546223253,
+            0.017955119269367787,
+            0.007300372071724306,
+            -0.006898619924677706,
+            0.01647927363640854,
+            0.021012103522831126,
+            0.010291016618557653,
+            -0.00047955552103474145,
+            0.0008684297441778696,
+            -0.010534247928997246,
+            0.005037930235889528,
+            0.035417626740307205,
+            0.014712595692093886,
+            0.010650306363963653,
+            -0.03626584475133921,
+            0.021764456453493054,
+            -0.011961521673661574,
+            0.009092981192874177,
+            0.006089104887183293,
+            0.017982801101846038,
+            0.006916969397519662,
+            0.014196254958707752,
+            0.011745069030657836,
+            -0.006689305450954901,
+            0.025205280281804708,
+            -0.001601414806100461,
+            -0.011992525142592405,
+            0.003386930324498371,
+            -0.01859193251438437,
+            0.005303261319773131,
+            -0.0041710536965100615,
+            -0.004459393120257307,
+            0.0032461828144552565,
+            -0.0018650956576455287,
+            0.015678225251799887,
+            0.003894142628084471,
+            -0.02355212701026675,
+            -0.0014873755115206368,
+            -0.024284295126348117,
+            0.0027260592623277375,
+            -0.022185571998195344,
+            0.00022883605560765203,
+            -0.005557568353382136,
+            -0.009749382813201403,
+            -0.025385857431333838,
+            -0.009853073898348951,
+            -0.0035843255959309307,
+            0.007661137888845632,
+            -0.010521037433807653,
+            -0.004246698084422667,
+            -0.0019920499321964815,
+            -0.019465113363548715,
+            0.009014255846983305,
+            0.027122533756959888,
+            0.016848221859975445,
+            0.027170883580103264,
+            -0.00019554112591107156,
+            0.013614675849485926,
+            -0.002066822813103284,
+            -0.01603063005944364,
+            0.008824663429777616,
+            0.0075653522381218495,
+            -0.00531780740880296,
+            0.014643348459785112,
+            0.0072634484937344704,
+            -0.012800149015319017,
+            0.009859503660384916,
+            -0.007161712176469609,
+            -0.006437857349742842,
+            -0.009764856421740622,
+            -0.02058868349990357,
+            -0.014866137971603676,
+            0.0012450207572602557,
+            -0.002123265406747006,
+            0.006000473862248991,
+            -0.004054016790850226,
+            0.009584879012937315,
+            -0.00608816807208866,
+            -0.00925801057439376,
+            -0.013962683072792106,
+            -0.008924356118158748,
+            -0.004210064145949253,
+            -0.004381709838058668,
+            -0.011732699976353612,
+            -0.005479523097469261,
+            0.0010680258077037758,
+            -0.0019401640391035755,
+            -0.01397652201595301,
+            0.011650839243603377,
+            0.0058731838291845645,
+            -0.002462743134993977,
+            0.02219008688925881,
+            0.01097316136828334,
+            0.0018083001685932406,
+            0.01366095798812196,
+            0.0003942601242216818,
+            -0.0064063532958756175,
+            -0.03343312148366622,
+            -0.0005083355836126291,
+            -0.02815586367527057,
+            0.025770078162165123,
+            -0.0069114949102666145,
+            -0.010322087105114614,
+            -0.002519047562374146,
+            -0.006063669644021698,
+            0.005873609611264377,
+            -0.002722940652283699,
+            0.0021229525800901114,
+            -0.023673778230768945,
+            0.0028629715379660113,
+            -0.003953653437464966,
+            -0.0036366957271968823,
+            0.007469664864329332,
+            0.0346520690389124,
+            -0.004054282838167587,
+            -0.0034765253405897676,
+            0.005159466707114532,
+            0.011642913651743502,
+            0.027845571220425726,
+            -0.004445196354520585,
+            -0.0032038508043339413,
+            -0.02233838584152591,
+            0.0054112576531374175,
+            -0.00019544838850471866,
+            -0.004733192299951088,
+            0.016664217348411003,
+            -0.016719175418727594,
+            -0.01631658832047864,
+            -0.00545969677717507,
+            -0.014570848053102338,
+            -0.00022620258237138677,
+            0.014454101124666536,
+            0.011056312437637014,
+            -0.004035526363810078,
+            -0.012459378363360557,
+            -0.0022595493558020706,
+            -0.030370952502579162,
+            0.012766010284884204,
+            0.1141689619126009,
+            -0.00766197047210299,
+            -0.00896655658602649,
+            -0.009481112393590025,
+            0.001435269335646347,
+            0.001338755472877713,
+            -0.011739143028767118,
+            0.07166096084053437,
+            -0.0077855767070652395,
+            0.0002139580430342728,
+            -0.011312198831181402,
+            0.001844005266872301,
+            -0.001825412693163496,
+            0.01371932421024605,
+            -0.005025814503687022,
+            -0.018213360991353643,
+            -0.0033612481616769416,
+            -0.02941403830641238,
+            -0.0041137865872933515,
+            -0.005683630256743215,
+            -0.004288198570156578,
+            -0.014982999363445122,
+            0.0067149665791261335,
+            -0.019661973497372313,
+            -0.012688556129040908,
+            0.001124391650175233,
+            -0.0033623453063430534,
+            0.00996944383866679,
+            -0.020496336814603385,
+            -0.0044354321069609536,
+            0.0019875840782527733,
+            0.009706675240563883,
+            0.0011600364428219214,
+            -0.012755178230821602,
+            -0.0026514442584739412,
+            0.004056350166031111,
+            0.01909951327913368,
+            0.014360281326789544,
+            0.003732344102045803,
+            0.008623457695718206,
+            0.014461868085333023,
+            -0.001692149528228451,
+            0.01948035376572433,
+            0.01172550239477637,
+            0.0031966531144076173,
+            0.0004779413449255626,
+            0.0030912562273611517,
+            -0.004597633235071031,
+            0.01582547400214653,
+            -0.014811177637844808,
+            -0.02411202772171917,
+            -0.0071272238837422185,
+            0.018015541137524113,
+            -0.013081287852028074,
+            1.8837689271256874e-05,
+            0.004062803284646432,
+            0.027569726868200254,
+            0.0010564850664255057,
+            -0.01179755720247972,
+            0.0028876134800690334,
+            -0.01638215925131053,
+            0.022944183106912663,
+            0.0026043713064243836,
+            0.03279173899194346,
+            -0.005103903273968361,
+            0.00578213669463383,
+            -0.020744194169889544,
+            -0.007588793365216579,
+            -0.00819862611489168,
+            0.0017401442433921013,
+            0.010993500174794084,
+            0.005411238350013134,
+            0.01620925462604661,
+            -0.00641615069699833,
+            -0.020751730783437995,
+            0.014776878596707538,
+            -0.009552193304486048,
+            -0.03393319554458277,
+            0.008881987295131347,
+            -0.029102297123590998,
+            -0.0045267103602711,
+            0.00368971742585232,
+            -0.014269341759285646,
+            -0.016264809810970175,
+            0.012837302095591106,
+            -0.0011890334180728564,
+            -0.008937992768323412,
+            0.0014444613725248846,
+            -0.017110187133760626,
+            -0.02385668037818897,
+            0.006017405364919834,
+            0.00676670672597817,
+            -0.0077994500447595,
+            0.018312452858615755,
+            -0.027015660037024585,
+            0.006123912473734307,
+            -0.012973676426738582,
+            0.009041974762537394,
+            -0.01580183887345168,
+            0.01163516332546347,
+            0.010887662470499361,
+            -0.001503467664709027,
+            -0.0076234124970223326,
+            0.01231202541338883,
+            0.004852720134894811,
+            0.033760365210093034,
+            0.028658326381800395,
+            0.000995999783627898,
+            -0.0025605367058226737,
+            -0.0020356794893366384,
+            0.011667525096213252,
+            -0.014049700287539817,
+            0.011218246246280464,
+            0.02658113044439717,
+            0.017568631325838108,
+            -0.019133062654008885,
+            0.018050678772708717,
+            -0.010356118256625172,
+            0.012872052420999813,
+            -0.025620154415048252,
+            -0.005135768485044442,
+            0.004936490920616527,
+            0.005163008729256001,
+            0.007425278892536464,
+            0.001410212306439329,
+            0.021891387289347516,
+            0.014999616600766989,
+            0.00027613811156650413,
+            0.00558422822386443,
+            0.01477009526981092,
+            0.010497782599293994,
+            -0.0005567894749233204,
+            -0.00467776506604011,
+            -0.016998035358377343,
+            0.00957271797745408,
+            -0.006065116857898469,
+            0.005138070401846389,
+            0.01698914855850691,
+            -0.01230257809343985,
+            0.011109844802663414,
+            0.007518553660463051,
+            -0.015504773927595823,
+            -0.017169933945839305,
+            0.005552829003043926,
+            -0.02252633790984703,
+            -0.01306722099657118,
+            0.02715689072730398,
+            -0.010513941399990838,
+            -0.02790582732974509,
+            -0.0009847768069205477,
+            0.009797865241108237,
+            -0.002772952073769185,
+            -0.0027968741526000576,
+            0.0037541733651479613,
+            0.010801137722284016,
+            -0.014658156970099418,
+            0.002075415233345957,
+            0.004735264309137368,
+            0.01930804315461151,
+            -0.0014151522239496115,
+            -0.015046118414100617,
+            -0.002288495894227231,
+            0.021592829677937263,
+            0.017174609713578173,
+            0.00265560378681856,
+            -0.016499825201963983,
+            -3.965296928410772e-05,
+            -0.0074142747014567635,
+            0.00021360696629503726,
+            0.00027431174904249525,
+            -0.0056391089282546054,
+            -0.018931681700770225,
+            -0.003888248121485032,
+            0.011181675419027736,
+            -0.00247252920175839,
+            0.01977860312223846,
+            -0.0011527575128223392,
+            -0.015763587433200915,
+            0.001960734212846797,
+            -0.005280860341480037,
+            -0.010372829570478649,
+            0.01351900743232294,
+            -0.009359942374603997,
+            0.006788746813046239,
+            -0.003212015225284119,
+            0.004611143085183367,
+            -0.0003308099698088422,
+            -0.017017265064185255,
+            0.02924142709831441,
+            -0.0018238549856433466,
+            -0.004769274041546295,
+            -0.0014058363976586562,
+            -0.033393331166741945,
+            0.015017165991303838,
+            -0.014574579250749552,
+            0.006745699409061803,
+            0.002705853802154516,
+            0.01272741296341735,
+            0.003253095246601883,
+            -0.008898616461779179,
+            0.0027485606928249326,
+            0.017040360927723885,
+            0.013756402798365262,
+            -0.007292864293090161,
+            0.008694855886257706,
+            -0.0028934389922887676,
+            -0.0037483208383542015,
+            0.01363963339139356,
+            0.004170611039133969,
+            -0.006282657671540625,
+            -0.0009784372731140945
+          ],
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "Mixture-of-Experts (MoE)",
+            "fine-grained expert segmentation",
+            "shared expert isolation"
+          ],
+          "title": "Deepseekmoe: Towards Ultimate Expert Specialization In Mixture-Of-Experts Language Models"
+        },
+        "type": "document"
+      },
+      "bidirectional": true,
+      "properties": {
+        "cosine_similarity": 0.8689436720928033
+      }
+    },
+    {
+      "id": "fd315d47-6d4b-441b-8018-948f41540a89",
+      "type": "cosine_similarity",
+      "source": {
+        "id": "b570d71e-4f83-486e-b70f-b0c6217db455",
+        "properties": {
+          "page_content": "Mixture-Of-Experts With Expert Choice Routing\n\nYanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon Google, Mountain View, CA, USA {yanqiz, taole, hanxiaol, dunan, huangyp, vzhao, adai, zhifengc, qvl, jlaudon}@google.com\n\nAbstract\n\nSparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choice method. Instead of letting tokens select the top-k experts, we have experts selecting the top-k tokens. As a result, each token can be routed to a variable number of experts and each expert can have a fixed bucket size. We systematically study pre-training speedups using the same computational resources of the Switch Transformer top-1 and GShard top-2 gating of prior work and find that our method improves training convergence time by more than 2×. For the same computational cost, our method demonstrates higher performance in fine-tuning 11 selected tasks in the GLUE and SuperGLUE benchmarks. For a smaller activation cost, our method outperforms the T5 dense model in 7 out of the 11 tasks.\n\n1 Introduction\n\nScaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27]. The final model quality has been found to have a power-law relationship with the amount of data, model size, and compute time [16, 20]. However, training efficiency, which is defined as the total amount of computation used to achieve superior model quality than the state of the art system [21], should receive greater attention as we increase our efforts towards green AI [29].\n\nSparsely gated mixture-of-experts [31] (MoE) provides an effective way to scale model capacity given a fixed computational cost, and has recently played an important role in increasing the training efficiency of large-scale language models [10, 21]. MoE operate by adopting a number of experts, each as a sub-network, and by activating only one or a few experts for each input token. A gating network must be chosen and optimized in order to route each token to the most suited expert(s). For example, recent work has implemented sparse routing via k-means clustering [12], linear assignment to maximize token-expert affinities [22], or hashing [8, 28]. Many of the prior work use a routing strategy concerning the token choice, where each token selects the best one or two experts.\n\nWe argue that the independent token choice of prior work often leads to an imbalanced load of experts, which causes training inefficiency and sub-optimal training of the model. In order to mitigate this\n\nissue, previous sparsely gated networks introduce additional auxiliary losses as regularization to prevent too many tokens being routed to a single expert, but the effectiveness is still limited. Recent approaches [8, 22, 28] explore alternative strategies for routing, but they focus on pre-training only and do not demonstrate performance gain on downstream tasks. Moreover, none of the previous methods consider allocating a variable number of experts to each token based on importance, which can be beneficial.\n\nWe propose a very simple yet effective routing method we are calling expert choice. Unlike conventional MoE where tokens select one or two top-scoring experts, our method lets each expert pick the top-k tokens. Our method guarantees perfect load balancing, allows a variable number of experts for each token, and achieves substantial gains in training efficiency and downstream performance as demonstrated in our experiments. Our major contributions include:\n\nWe identify common pitfalls in conventional MoE such as load imbalance as described in Section 3.1. We then propose a heterogeneous, expert choice method to provide a fluid allocation of model parameters based on a learnt token-to-expert importance. This method intrinsically guarantees load balance without imposing an auxiliary loss.\n\nWe show our method provides over 2× faster training convergence in a 8B/64E (8 billion activated parameters, 64 experts) model, compared to the top-1 and top-2 gating counterparts in Switch Transformer [10] and GShard [21].\n\nWe show our method demonstrates strong scaling when increasing the number of experts from 16 to 128, evaluated in training perplexity.\n\nWe show our method demonstrates strong performance on downstream tasks selected from GLUE and SuperGLUE at all the evaluated scales. More specifically, our 8B/64E model outperforms a T5 11B dense model in 7 out of 11 tasks evaluated.\n\n2 Related Work\n\nScaling: Various approaches have been proposed to scale up neural network capacity to improve performance. Recent works have successfully scaled models to billions of parameters via various forms of model parallelism [2, 21, 26, 27, 33]. Model parallelism [30] splits weights and tensors across multiple cores while pipeline parallelism [18, 24] splits different layers across devices with micro-batches pipelined to the different layers. To enable continued scaling of neural networks, improving model training and serving efficiency has become a critical research area.\n\nConditional Computation: Computation decisions can be made dynamically based on the input [23, 25]. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation, by activating certain parameters and computation on demand, on a per-example or per-token basis [3]. Conditional convolution layers [1] with task-specific gating has been used to combat catastrophic forgetting when a sequence of learning problems are optimized. The gating decisions may be binary or sparse and continuous, stochastic or deterministic.\n\nMixture of Experts: Sparsely-gated MoE [31] is the first model to demonstrate massive improvements in model capacity, training time, or model quality with gating. Switch Transformer [10] simplifies the gating by selecting only the top expert per token using a softmax over the hidden state and demonstrates better scaling than previous work. All the prior work requires an auxiliary loss to explicitly encourage balancing. This loss term has to be carefully weighted to not overwhelm the primary loss. However, auxiliary loss does not guarantee balancing and a hard capacity factor has to be imposed. As a result, many tokens can still be unprocessed by the MoE layer. Hard MoE [12] with a single decoding layer can be efficiently trained to good effect on large scale hashtag prediction tasks.\n\nBase Layers [22] formulate a linear assignment that maximizes token-expert affinities while ensuring each expert receives an equal number of tokens. Hash layers [8, 28] devise hashing techniques on input tokens. However, the evaluations are limited to pre-training perplexity. THOR [? ] randomly activates experts during training and inference and is trained with a consistency regularization loss.\n\nTHOR has demonstrated strong performance on translation tasks. Different from these prior works, our method is a learnt method that enables heterogeneous MoE and effectively improves downstream fine-tuning performance.\n\n3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n5 Conclusion\n\nWe propose a new routing method for sparsely activated mixture-of-experts (MoE) models. This method addresses load imbalance and under-utilization of experts in conventional MoE methods, and enables selecting different numbers of experts for each token. Our model demonstrates more than 2x training efficiency improvements when compared to the state-of-the-art GShard and Switch Transformer models, and also achieves strong gains when finetuning on 11 datasets in the GLUE and SuperGLUE benchmark.\n\n6 Limitations\n\nThe expert choice method might not immediately apply to auto-regressive text generation as our current implementation takes in the past and future tokens to perform the top-k selection. One possible solution is to collect a large batch of input sequences, dispatch tokens of the same sequence into separate groups, and perform expert choice routing for each group. Another scenario where the expert choice method does not immediately apply is when the batch size becomes very small during serving or inference. A global top-k can be selected instead and we can cap the number of times each expert or token gets selected. We leave these possible improvements for future work.\n\nAnother long-standing issue with MoE has been the large memory footprint. Even though computational cost can be reduced using sparsely gated networks, the total number of parameters increases linearly or sub-linearly with the number of experts. Increasing the number of experts requires reservation of a large number of hardware devices. Therefore, dynamic (used) power is saved while static (reserved) power is not. Power saving techniques such as the ability to put hardware devices into low power states while not in use [17] can help with reducing the reserved power requirements.\n\nReferences\n\n[1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pages 3930–3939. Computer Vision Foundation / IEEE, 2020.\n\n[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. In Advances in Neural Information Processing Systems.\n\n[3] Kyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning, 2014.\n\n[4] Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems, 2021.\n\n[5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.\n\nTransformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.\n\n[6] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 933–941. JMLR.org, 2017.\n\n[7] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixtureof-experts, 2021.\n\n[8] Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan.\n\nTricks for training sparse translation models, 2021.\n\n[9] Richard L Dykstra. An iterative procedure for obtaining i-projections onto the intersection of convex sets. The annals of Probability, pages 975–984, 1985.\n\n[10] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.\n\n[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. NAS-FPN: learning scalable feature pyramid architecture for object detection. In CVPR, pages 7036–7045. Computer Vision Foundation / IEEE, 2019.\n\n[12] Sam Gross, Marc'Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision, 2017.\n\n[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.\n\n[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 630–645, Cham, 2016. Springer International Publishing.\n\n[15] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2016.\n\n[16] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.\n\n[17] Ping Huang, Zuocheng Xing, Tianran Wang, Qiang Wei, Hongyan Wang, and Guitao Fu. A brief survey on power gating design. In 2010 10th IEEE International Conference on Solid-State and Integrated Circuit Technology, pages 788–790, 2010.\n\n[18] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 103–112, 2019.\n\n[19] Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David A. Patterson. A domain-specific supercomputer for training deep neural networks. Commun. ACM, 63(7):67–78, 2020.\n\n[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.\n\n[21] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.\n\n[22] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6265–6274. PMLR, 18–24 Jul 2021.\n\n[24] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. New York, NY, USA, 2019. Association for Computing Machinery.\n\n[25] Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, Cédric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. In ICLR. OpenReview.net, 2021.\n\n[26] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.\n\n[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.\n\n[28] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models, 2021.\n\n[30] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 10435–10444, Red Hook, NY, USA, 2018. Curran Associates Inc.\n\n[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E.\n\nHinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In ICLR (Poster). OpenReview.net, 2017.\n\n[32] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 10–15 Jul 2018. [23] Min Lin, Jie Fu, and Yoshua Bengio. Conditional computation for continual learning, 2019.\n\n[29] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.\n\n[33] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.\n\n[34] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc.\n\n[35] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.\n\nGLUE: A multi-task benchmark and analysis platform for natural language understanding.\n\nIn Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, November 2018. Association for Computational Linguistics.\n\n[36] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake A. Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen. GSPMD: general and scalable parallelization for ML computation graphs. CoRR, abs/2105.04663, 2021.\n\n7 Checklist\n\n(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Yes (b) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes (c) Did you discuss any potential negative societal impacts of your work? N/A. Not any. (d) Did you describe the limitations of your work? Yes (a) Did you include the code, data, and instructions needed to reproduce the main experimental results? Yes. We include details in the experiment setup to help reproduce the main results. (b) Did you specify all the training details? Yes (c) Did you report error bars? Yes (d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? Yes (a) If your work uses existing assets, did you cite the creators? Yes (b) Did you mention the license of the assets? No. The used dataset is not released yet. (c) Did you include any new assets either in the supplemental material or as a URL? No. The dataset is not released yet.\n\n(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? No. Not using persons' data. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes. The dataset does not contain any personally identifiable information or offensive content.\n\nA Comparison On Fine-Tuning With A Dense Model\n\nOur 8B MoE model achieves stronger pre-training perplexity than its dense counterpart. However, a better perplexity does not always directly translate to downstream performance as demonstrated in Section 4.4. To this end, we compare fine-tuning performance of the 8B dense model and MoE model in Table 1. As shown in the table, our MoE model using expert choice routing consistently outperforms the dense model across the 11 tasks in GLUE and SuperGLUE.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 1: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nB Capacity Factor\n\nWe evaluate the downstream task fine-tuning performance by varying the capacity factors. Note that a capacity factor of n indicates on average how many experts each token can be received. EC-CF2 is our baseline expert choice, which matches GShard top-2 gating computational footprint. EC-CF1, however, matches Switch Transformer top-1 gating computational footprint. EC-CF0.5 further verifies that an aggressively lowered capacity factor can provide strong enough performance, that almost matches the top-2 gating baseline.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Top-2 78.1 87.0 88.3 85.0 82.6 90.1 90.7 81.6 94.7 68.2 67.2 83.0±0.3 EC-CAP2 78.2 88.0 88.5 85.7 83.0 90.8 91.1 80.0 95.4 70.4 64.1 83.2±0.4 EC-CAP3 78.5 91.7 89.3 86.3 83.5 90.9 91.1 81.8 94.9 70.0 65.6 84.0±0.4 EC-CF2 79.1 89.6 89.3 86.8 84.3 91.3 91.2 81.1 95.2 68.1 68.0 84.0±0.2 EC-CF1 77.4 90.6 88.0 85.5 83.6 90.3 91.2 79.8 95.3 66.5 64.9 83.0±0.2 EC-CF0.5 77.4 89.6 86.3 85.2 82.7 91.7 91.0 79.6 94.9 67.3 63.5 83.0 ±0.05 Hash Layers 76.1 85.2 86.7 83.4 82.5 90.0 90.3 75.7 94.0 67.4 63.3 81.3±1.0\n\nTable 2: Comparison between different routing methods in fine-tuning of 100M/64E models. We perform 3 independent fine-tuning runs for each method and report the average results. This gives more accurate difference between the variants of expert choice method, since they achieve close fine-tuning results. We do not report averaged results in other experiments.\n\nC Capped Expert Choice\n\nAs described in Section 4.5, the maximum number of experts each token is assigned can be capped by an entropy-regularized linear programming. Figure 1 compares the validation perplexity when training the 100M/64E models using the base expert choice method (EC-BASE), expert choice capped by two experts per token (EC-CAP2), expert choice capped by three experts per token (EC-CAP3), and GShard top-2 gating.\n\nAs shown in the figure, restricting the number of experts to 2 degrades the perplexity compared to the base expert choice method. This suggests that a more flexible allocation of experts (e.g. more than 2 experts for a token) can enhance model expressiveness. On the other hand, our EC-CAP2 and EC-CAP3 methods still outperform the top-2 gating method by a clear margin. We believe this confirms the effectiveness of a load balanced training, provided by our method. Finally, EC-CAP3 obtains comparable perplexity to EC-BASE. As indicated by Figure 3, only a little fraction of tokens use more than 3 experts therefore we see little or no difference between EC-BASE and EC-CAP3 variants. We present the fine-tuning results of these methods in Table 2.\n\nD Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [? ]. We use mod x to map a token ID to an expert ID. This in some way ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 2. Hashing based routing performs much worse than expert choice in terms of average scores and variance.\n\nE Fine-Tuning Details\n\nWe did a hyperparameter search for both baseline models and expert choice method. For fine-tuning of the 8B dense model, we use a constant learning rate of 0.0001 and a dropout rate of 0.1. We freeze the attention layer and feed-forward layer while leaving the embedding and layer normalization trainable. This setting has been found optimal for the 8B dense model. For MoE 8B/64E models including GShard top-2 gating and expert choice, we found continuing the learning rate from the pre-trained model while using a square root learning rate decay works better. In addition, we do not apply parameter freezing for fine-tuning MoE models. For models with 100M expert size, we use a constant learning rate of 0.0001 and no dropout is used.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/2202.09368v2.md"
+          },
+          "headlines": {
+            "1 Introduction": [],
+            "2 Related Work": [],
+            "3 Method": [
+              "3.1 Pitfalls Of Token-Choice Routing",
+              "3.2 Heterogeneous Moe Via Expert Choice",
+              "3.3 Expert Choice With Additional Constraint",
+              "3.4 Model Architecture"
+            ],
+            "4 Experiments": [
+              "4.1 Setup",
+              "4.2 Training Efficiency",
+              "4.3 Scaling The Number Of Experts",
+              "4.4 Fine-Tuning On Glue And Superglue",
+              "4.5 Heterogeneity Matters",
+              "4.6 Comparison With Hash Layer",
+              "4.7 Ablation"
+            ],
+            "5 Conclusion": [],
+            "6 Limitations": [],
+            "A Comparison On Fine-Tuning With A Dense Model": [],
+            "B Capacity Factor": [],
+            "C Capped Expert Choice": [],
+            "D Comparison With Hash Layer": [],
+            "E Fine-Tuning Details": []
+          },
+          "summary": "The paper introduces a new routing method for sparsely-activated Mixture-of-Experts (MoE) models called 'expert choice,' which addresses load imbalance and under-utilization issues in conventional MoE methods. Unlike traditional token-choice routing, expert choice allows experts to select the top-k tokens, ensuring perfect load balancing and enabling variable expert allocation per token. This method improves training efficiency, achieving over 2x faster convergence compared to existing models like Switch Transformer and GShard. It also demonstrates superior performance on downstream tasks in the GLUE and SuperGLUE benchmarks. The study highlights the benefits of flexible expert allocation and shows that expert choice outperforms hashing-based routing methods. The paper also discusses potential limitations and future improvements for the expert choice method.",
+          "summary_embedding": [
+            -0.009726986289024353,
+            -0.0337262786924839,
+            -0.023302342742681503,
+            -0.019878201186656952,
+            -0.025953780859708786,
+            0.04560471698641777,
+            -0.006874797400087118,
+            0.039665497839450836,
+            -0.006973279174417257,
+            0.034544438123703,
+            -0.025787118822336197,
+            -0.00363057479262352,
+            -0.01052999310195446,
+            0.0054657477885484695,
+            -0.024499278515577316,
+            0.036786794662475586,
+            0.010007280856370926,
+            0.003272630739957094,
+            0.006102093029767275,
+            -0.04048365354537964,
+            -0.018666116520762444,
+            -0.013522328808903694,
+            -0.03230207785964012,
+            0.008969432674348354,
+            0.02484775334596634,
+            0.0016713523073121905,
+            0.0026230288203805685,
+            0.003624893259257078,
+            -0.05311964824795723,
+            0.04339266195893288,
+            0.018681267276406288,
+            0.023711422458291054,
+            -0.016332851722836494,
+            -0.012832955457270145,
+            -0.039120059460401535,
+            -0.0197418425232172,
+            0.030786970630288124,
+            0.039968520402908325,
+            -0.019287310540676117,
+            0.03357476741075516,
+            0.018060073256492615,
+            0.02454473078250885,
+            -0.021272100508213043,
+            -0.029817303642630577,
+            -0.0061892117373645306,
+            -0.007840678095817566,
+            -0.008355814032256603,
+            0.01242387667298317,
+            0.016984347254037857,
+            -0.013007191941142082,
+            0.0212569497525692,
+            -0.01371171697974205,
+            0.03384748846292496,
+            0.008916404098272324,
+            -0.042332086712121964,
+            0.03984731063246727,
+            -0.008007340133190155,
+            0.012105703353881836,
+            0.03281721472740173,
+            0.010333028621971607,
+            -0.005287723150104284,
+            -0.021544819697737694,
+            -0.03378688171505928,
+            0.0010529992869123816,
+            0.00014014738553669304,
+            0.005950582213699818,
+            -0.003069985192269087,
+            -0.009742137044668198,
+            0.03227177634835243,
+            -0.003939277958124876,
+            -0.0740584209561348,
+            0.050089433789253235,
+            -0.026590125635266304,
+            -0.007628562860190868,
+            -0.028938541188836098,
+            -0.01736312545835972,
+            0.037362534552812576,
+            0.005662711802870035,
+            0.0098709212616086,
+            0.0106284748762846,
+            -0.0007892759749665856,
+            0.028923390433192253,
+            0.00910579226911068,
+            -0.024923507124185562,
+            0.04478655755519867,
+            -0.04790768027305603,
+            0.01801462098956108,
+            -0.04336235672235489,
+            0.022756904363632202,
+            1.6393929399782792e-05,
+            -0.020726662129163742,
+            -0.0013304532039910555,
+            -0.016666175797581673,
+            0.0218023881316185,
+            -0.01730252057313919,
+            0.026802239939570427,
+            0.013825350441038609,
+            -0.015802564099431038,
+            -0.029817303642630577,
+            0.025378039106726646,
+            0.003367325058206916,
+            0.010499690659344196,
+            -0.02075696364045143,
+            0.014332910999655724,
+            0.04393810033798218,
+            -0.02593863010406494,
+            -0.02271145209670067,
+            0.027681002393364906,
+            0.014408666640520096,
+            0.0633314698934555,
+            0.036544378846883774,
+            0.006624804809689522,
+            0.022362977266311646,
+            0.020575150847434998,
+            -0.028968842700123787,
+            -0.01963578537106514,
+            -0.046362269669771194,
+            -0.04757435619831085,
+            0.019878201186656952,
+            -0.04566532000899315,
+            0.0050869714468717575,
+            0.038544319570064545,
+            -0.013704141601920128,
+            -0.014446543529629707,
+            -0.04021093621850014,
+            0.017847958952188492,
+            0.0003908502112608403,
+            0.011976920068264008,
+            -0.0022082682698965073,
+            0.00033285003155469894,
+            -0.011825408786535263,
+            0.008878526277840137,
+            -0.03778676688671112,
+            -0.025196226313710213,
+            0.025181075558066368,
+            0.022862963378429413,
+            0.024120500311255455,
+            0.028802180662751198,
+            -0.03077181987464428,
+            -0.009984553791582584,
+            0.021378157660365105,
+            -0.04242299124598503,
+            -0.006556625012308359,
+            0.011332998983561993,
+            0.0026703758630901575,
+            0.003872992005199194,
+            0.032786913216114044,
+            0.052998438477516174,
+            -0.03172633796930313,
+            0.02065090648829937,
+            -0.010431510396301746,
+            0.04957429692149162,
+            0.00024691506405360997,
+            0.028968842700123787,
+            -0.019514575600624084,
+            -0.0134389977902174,
+            -0.029529431834816933,
+            -0.009795165620744228,
+            -0.01356020662933588,
+            -0.008484598249197006,
+            -0.002998017705976963,
+            -0.009575475007295609,
+            0.0040453351102769375,
+            0.0331808403134346,
+            0.00273855566047132,
+            0.030680913478136063,
+            0.014325335621833801,
+            0.0051475754007697105,
+            -0.08745196461677551,
+            0.026059838011860847,
+            -0.0033616432920098305,
+            0.005859675817191601,
+            0.0062460280023515224,
+            0.03013547509908676,
+            -0.0312718041241169,
+            0.020938776433467865,
+            0.026620427146553993,
+            0.0011145505122840405,
+            0.016969196498394012,
+            0.036847397685050964,
+            -0.04181694984436035,
+            0.009643655270338058,
+            0.012590537779033184,
+            -0.0033881578128784895,
+            0.04584713280200958,
+            0.011628445237874985,
+            0.019514575600624084,
+            0.018529755994677544,
+            0.003569970605894923,
+            0.00297907879576087,
+            0.006533898413181305,
+            0.03724132478237152,
+            0.003462019143626094,
+            -0.009833043441176414,
+            0.014976831153035164,
+            -0.010757259093225002,
+            -0.011219366453588009,
+            -0.037998881191015244,
+            0.014454119838774204,
+            0.008742166683077812,
+            -0.017454031854867935,
+            -0.02959003672003746,
+            0.0026949963066726923,
+            -0.031908150762319565,
+            0.026559824123978615,
+            -0.002445003716275096,
+            0.005738467443734407,
+            -0.010454237461090088,
+            -0.006757376715540886,
+            -0.035423196852207184,
+            0.028453707695007324,
+            -0.02095392718911171,
+            0.0053142374381423,
+            0.0007258309051394463,
+            -0.029438525438308716,
+            0.0060073984786868095,
+            0.017863109707832336,
+            -0.0025965142995119095,
+            0.0025302283465862274,
+            -0.02349930815398693,
+            0.024468975141644478,
+            0.025635607540607452,
+            0.004518806003034115,
+            -0.014037464745342731,
+            -0.013113250024616718,
+            -0.003605954349040985,
+            -0.009742137044668198,
+            -0.016196493059396744,
+            0.024468975141644478,
+            -0.014810169115662575,
+            0.00308892410248518,
+            0.021847840398550034,
+            0.014264730736613274,
+            -0.007045246660709381,
+            -0.004007457755506039,
+            -0.04066546633839607,
+            -0.01218903437256813,
+            -0.04760465770959854,
+            -0.011628445237874985,
+            -0.0034601252991706133,
+            -0.028696123510599136,
+            0.0061513339169323444,
+            -0.04481685906648636,
+            -0.01235569640994072,
+            0.02309022843837738,
+            0.018135828897356987,
+            0.0019488062243908644,
+            -0.009173971600830555,
+            -0.022605394944548607,
+            8.486965816700831e-05,
+            -0.01763584464788437,
+            -0.017044952139258385,
+            0.015393485315144062,
+            0.031059689819812775,
+            -0.03101423755288124,
+            0.04024123772978783,
+            0.003893824527040124,
+            -0.08066429197788239,
+            0.010393633507192135,
+            -0.026362858712673187,
+            0.07011914253234863,
+            -0.0025453795678913593,
+            0.014075342565774918,
+            0.06781618297100067,
+            -0.012431452050805092,
+            -0.02007516659796238,
+            0.03142331540584564,
+            -0.008249756880104542,
+            -1.1311515663692262e-05,
+            -0.02884763479232788,
+            -0.02739313244819641,
+            -0.053271155804395676,
+            -0.009704259224236012,
+            -0.010787560604512691,
+            0.03360506892204285,
+            -0.024317465722560883,
+            -0.0007656024536117911,
+            0.012620840221643448,
+            0.02983245439827442,
+            0.016635872423648834,
+            -0.0015681355725973845,
+            0.008272483013570309,
+            0.02881733328104019,
+            0.0274234339594841,
+            -0.0036154238041490316,
+            0.00814369972795248,
+            -0.034211114048957825,
+            -0.0068066176027059555,
+            -0.0493621826171875,
+            0.011302697472274303,
+            -0.02596893161535263,
+            -0.02596893161535263,
+            0.007643714081496,
+            -0.011476934887468815,
+            0.03748374432325363,
+            -0.024696242064237595,
+            -0.007503566797822714,
+            0.005037730094045401,
+            0.012855681590735912,
+            -0.017757052555680275,
+            -0.03612014651298523,
+            -0.0028502948116511106,
+            0.0026381798088550568,
+            0.007560383062809706,
+            -0.053240854293107986,
+            0.004499867558479309,
+            -0.018469152972102165,
+            0.0279688723385334,
+            -0.02139330841600895,
+            -0.013688990846276283,
+            0.042029064148664474,
+            0.011363301426172256,
+            0.027499189600348473,
+            -0.026468917727470398,
+            0.0017849853029474616,
+            0.012825379148125648,
+            -0.0020056227222085,
+            0.032514192163944244,
+            -0.030574856325984,
+            -0.03257479891180992,
+            0.001548249856568873,
+            -0.007840678095817566,
+            -0.0625436082482338,
+            -0.011386028490960598,
+            -0.00981789268553257,
+            0.017317671328783035,
+            0.0446956530213356,
+            0.04557441547513008,
+            -0.01068150345236063,
+            0.002174178371205926,
+            -0.02213571034371853,
+            -0.04542290419340134,
+            -0.02643861435353756,
+            0.0016486257081851363,
+            -0.019999410957098007,
+            -0.02789311669766903,
+            -0.0018313854234293103,
+            0.011211791075766087,
+            -0.013651113025844097,
+            0.01134057529270649,
+            -0.018560059368610382,
+            -0.01862066239118576,
+            -0.012332970276474953,
+            0.018575210124254227,
+            0.009901223704218864,
+            -0.0018796793883666396,
+            -0.015037435106933117,
+            -0.00980274099856615,
+            0.018969137221574783,
+            -0.011279970407485962,
+            0.0009388927719555795,
+            0.01188601367175579,
+            0.024347767233848572,
+            -0.007083124481141567,
+            0.010961798019707203,
+            -0.034756552428007126,
+            -0.01966608688235283,
+            0.007984613068401814,
+            0.002719616750255227,
+            0.020711511373519897,
+            0.008742166683077812,
+            -0.04275631532073021,
+            -0.044301725924015045,
+            0.016044981777668,
+            -0.0562104657292366,
+            0.028120383620262146,
+            -0.03327174484729767,
+            -0.012514783069491386,
+            -0.02671133354306221,
+            0.0013418165035545826,
+            -0.01828734017908573,
+            0.022438732907176018,
+            0.03778676688671112,
+            -0.0011486404109746218,
+            0.005503625608980656,
+            0.014658658765256405,
+            0.04384719207882881,
+            -0.05021063983440399,
+            -0.041180603206157684,
+            0.03754434734582901,
+            -0.01601468026638031,
+            0.0185146052390337,
+            0.00021744151308666915,
+            0.010348179377615452,
+            -0.014431392773985863,
+            0.022772056981921196,
+            0.013757170177996159,
+            -0.008764892816543579,
+            -0.030574856325984,
+            0.04945308715105057,
+            -0.023650817573070526,
+            -0.049513693898916245,
+            0.0018323323456570506,
+            0.018878230825066566,
+            0.018635815009474754,
+            -0.014870774000883102,
+            -0.02907489985227585,
+            -0.024317465722560883,
+            -0.006060427520424128,
+            -0.00947699323296547,
+            -0.03402930125594139,
+            0.011363301426172256,
+            0.0048445542342960835,
+            0.016666175797581673,
+            -0.00645056739449501,
+            -0.0073141781613230705,
+            -0.009370936080813408,
+            0.022559940814971924,
+            0.012635990977287292,
+            -0.03175663948059082,
+            -0.017484333366155624,
+            -0.02528713271021843,
+            0.004359719809144735,
+            0.010136065073311329,
+            0.022166013717651367,
+            0.024287162348628044,
+            -0.0323626808822155,
+            0.04218057543039322,
+            0.01550711877644062,
+            -0.008727015927433968,
+            -0.0019317612750455737,
+            0.02478714846074581,
+            0.04324115067720413,
+            0.0218023881316185,
+            0.026514369994401932,
+            0.002107892418280244,
+            8.705945947440341e-05,
+            -0.015620751306414604,
+            -0.0011391709558665752,
+            0.01862066239118576,
+            0.04911976307630539,
+            -0.012151156552135944,
+            0.01716616004705429,
+            -0.03578682243824005,
+            0.00396200455725193,
+            -0.02580226957798004,
+            -0.030786970630288124,
+            0.03745344281196594,
+            -0.015060162171721458,
+            0.0191964041441679,
+            0.025044716894626617,
+            -0.022469034418463707,
+            -0.011075431481003761,
+            -0.019423669204115868,
+            -0.026181045919656754,
+            -0.01702980138361454,
+            -0.03281721472740173,
+            -0.018135828897356987,
+            -0.002653330797329545,
+            -0.035635314881801605,
+            -0.04987731948494911,
+            -0.004329417832195759,
+            -0.046392571181058884,
+            -0.010507266037166119,
+            0.017378276214003563,
+            -0.011166337877511978,
+            -0.025711363181471825,
+            -0.005336964037269354,
+            0.003973367623984814,
+            -0.022196315228939056,
+            -0.0066588944755494595,
+            0.008439145050942898,
+            0.035938333719968796,
+            -0.02298417128622532,
+            -0.013060221448540688,
+            -0.0024317465722560883,
+            -0.02887793630361557,
+            -0.007878555916249752,
+            0.008999735116958618,
+            0.007723257411271334,
+            -0.0012357590021565557,
+            -0.05196816474199295,
+            -0.006916462909430265,
+            -0.012992041185498238,
+            0.029196109622716904,
+            0.026468917727470398,
+            0.0012168202083557844,
+            -0.02783251367509365,
+            0.008749742060899734,
+            -0.0072535742074251175,
+            0.02339324913918972,
+            -0.007992188446223736,
+            -0.022908415645360947,
+            -0.02034788578748703,
+            0.030786970630288124,
+            -0.0159086212515831,
+            0.009113367646932602,
+            0.009976978413760662,
+            0.0121360057964921,
+            -0.03899884968996048,
+            -0.001445980160497129,
+            0.0005648507503792644,
+            -0.010128489695489407,
+            0.00423093605786562,
+            -0.006738437805324793,
+            -0.007079336792230606,
+            -0.022090258076786995,
+            -0.0218023881316185,
+            0.03818069398403168,
+            -0.009113367646932602,
+            -0.0035661826841533184,
+            -0.01088604237884283,
+            0.03436262533068657,
+            -0.00595815759152174,
+            -0.01993880607187748,
+            -0.019347915425896645,
+            0.014431392773985863,
+            0.05105910077691078,
+            -0.028529461473226547,
+            -0.03984731063246727,
+            0.0011136035900563002,
+            0.030150625854730606,
+            0.004140029661357403,
+            -0.016908591613173485,
+            0.0035851215943694115,
+            0.056392278522253036,
+            -0.011742077767848969,
+            0.028362801298499107,
+            0.027771908789873123,
+            0.02295386977493763,
+            -0.02021152526140213,
+            -0.01678738370537758,
+            -0.020999381318688393,
+            0.011098157614469528,
+            -0.03027183562517166,
+            0.027059808373451233,
+            0.018969137221574783,
+            0.01538590993732214,
+            0.05054396390914917,
+            -0.03178694099187851,
+            -0.02515077404677868,
+            0.008499749936163425,
+            0.0021722845267504454,
+            0.011007251217961311,
+            -0.007727045100182295,
+            -0.01801462098956108,
+            -0.001950700068846345,
+            -0.014022313989698887,
+            -0.019817598164081573,
+            -0.020938776433467865,
+            0.00954517349600792,
+            0.015696506947278976,
+            0.002649543108418584,
+            0.010863316245377064,
+            0.0016590419691056013,
+            0.01686313934624195,
+            0.01321930717676878,
+            -0.0444229319691658,
+            -7.936555630294606e-05,
+            0.03378688171505928,
+            -0.0293476190418005,
+            -0.012264790013432503,
+            -0.016060132533311844,
+            -0.004439263138920069,
+            -0.01706010289490223,
+            -0.02762039750814438,
+            -0.006874797400087118,
+            -0.01889338344335556,
+            -0.00814369972795248,
+            -0.028196139261126518,
+            -0.009992130100727081,
+            0.01949942484498024,
+            0.022332675755023956,
+            0.001196934375911951,
+            -0.006783891003578901,
+            -0.005901341326534748,
+            0.02092362567782402,
+            0.032514192163944244,
+            0.01952972821891308,
+            0.0058331615291535854,
+            -0.015158643946051598,
+            -0.004295328166335821,
+            -0.012370847165584564,
+            0.016060132533311844,
+            0.012537509202957153,
+            6.344509893096983e-05,
+            0.03493836522102356,
+            0.004170331638306379,
+            -0.022726602852344513,
+            -0.001230077352374792,
+            0.024923507124185562,
+            -0.021590271964669228,
+            -0.0108330138027668,
+            0.040029123425483704,
+            0.0085679292678833,
+            -0.0024506852496415377,
+            -0.019287310540676117,
+            -0.019075196236371994,
+            0.02630225569009781,
+            -0.02133270539343357,
+            0.002083271974697709,
+            0.0482713058590889,
+            0.00035344602656550705,
+            -0.029802151024341583,
+            0.017954016104340553,
+            -0.013242034241557121,
+            -0.018560059368610382,
+            0.01848430372774601,
+            0.0012859469279646873,
+            -0.039968520402908325,
+            0.019014591351151466,
+            0.018408548086881638,
+            0.022635696455836296,
+            -0.02918095886707306,
+            0.0002127068000845611,
+            0.03233237937092781,
+            0.004037759732455015,
+            0.001595596899278462,
+            -0.008530051447451115,
+            -0.0006065162015147507,
+            0.010431510396301746,
+            0.001746160676702857,
+            0.02854461409151554,
+            -0.0030434709042310715,
+            0.027771908789873123,
+            -0.006727074272930622,
+            -0.008863375522196293,
+            0.019181253388524055,
+            -0.02109028771519661,
+            0.013166278600692749,
+            0.0685434341430664,
+            0.0021950111258774996,
+            0.025847723707556725,
+            0.007181606721132994,
+            0.024347767233848572,
+            0.02018122375011444,
+            -0.027271922677755356,
+            -0.02021152526140213,
+            -0.005397567991167307,
+            -0.0014033677289262414,
+            -0.03787767142057419,
+            0.011128460057079792,
+            -0.021302402019500732,
+            -0.013249609619379044,
+            -0.0022347825579345226,
+            -0.002316219499334693,
+            -0.006405114196240902,
+            0.034211114048957825,
+            -0.0014630251098424196,
+            -0.018560059368610382,
+            -0.035423196852207184,
+            0.037726160138845444,
+            0.04048365354537964,
+            0.0389382466673851,
+            -0.005984671879559755,
+            0.03548380360007286,
+            0.0030756669584661722,
+            0.0025302283465862274,
+            0.041453324258327484,
+            0.004791525658220053,
+            -0.011567841283977032,
+            0.02295386977493763,
+            0.012416300363838673,
+            0.03351416438817978,
+            -0.02045394293963909,
+            -0.02630225569009781,
+            -0.003299145260825753,
+            -0.026423463597893715,
+            0.012492056004703045,
+            -0.019726691767573357,
+            0.0018768386216834188,
+            0.007117214612662792,
+            0.015196521766483784,
+            -0.016044981777668,
+            0.014719262719154358,
+            -0.006102093029767275,
+            -0.021014532074332237,
+            0.011719351634383202,
+            -0.007780073676258326,
+            0.011961768381297588,
+            -0.02262054570019245,
+            0.001138224033638835,
+            -0.04063516482710838,
+            0.008280059322714806,
+            0.014022313989698887,
+            -0.01598437689244747,
+            0.03205966204404831,
+            0.020969079807400703,
+            -0.01801462098956108,
+            -0.0009682480012997985,
+            0.0034714885987341404,
+            -0.010393633507192135,
+            -0.011067855171859264,
+            -0.019923655316233635,
+            0.02786281518638134,
+            -0.015802564099431038,
+            -0.025908326730132103,
+            0.02454473078250885,
+            -0.01743887923657894,
+            0.015151068568229675,
+            -0.004299115855246782,
+            0.00888610165566206,
+            -0.019120648503303528,
+            0.02284781076014042,
+            -0.023741723969578743,
+            -0.0009128518868237734,
+            -0.020287280902266502,
+            0.0028389315120875835,
+            0.014135947450995445,
+            -0.018272189423441887,
+            -0.02139330841600895,
+            0.019150950014591217,
+            -0.005420294590294361,
+            -0.003931702114641666,
+            0.010014856234192848,
+            0.0030983935575932264,
+            -0.0029715034179389477,
+            -0.004549108445644379,
+            -0.03281721472740173,
+            -0.02915065549314022,
+            0.0033086147159337997,
+            0.011386028490960598,
+            0.015257125720381737,
+            0.027574945241212845,
+            0.03715042024850845,
+            0.006052851676940918,
+            0.0036949669010937214,
+            0.0009327376610599458,
+            -0.020726662129163742,
+            0.042635105550289154,
+            -0.020469093695282936,
+            0.040847279131412506,
+            0.032544493675231934,
+            -0.0197418425232172,
+            -0.023908386006951332,
+            0.0029052174650132656,
+            0.03115059621632099,
+            0.03151422366499901,
+            0.025211378931999207,
+            0.012310243211686611,
+            0.03690800443291664,
+            -0.004359719809144735,
+            -0.032211173325777054,
+            0.015802564099431038,
+            0.022256920114159584,
+            -0.008105821907520294,
+            0.009673957712948322,
+            0.027438584715127945,
+            0.013279912061989307,
+            -0.007757347077131271,
+            0.030150625854730606,
+            0.0012234487803652883,
+            0.02396899089217186,
+            0.02498411200940609,
+            -0.02190844528377056,
+            0.01743887923657894,
+            -0.0001154677156591788,
+            0.006727074272930622,
+            -0.009840618818998337,
+            0.004348356742411852,
+            0.011810258030891418,
+            0.018272189423441887,
+            0.018908534198999405,
+            0.003246116451919079,
+            -0.07060398161411285,
+            0.019726691767573357,
+            0.004507442936301231,
+            -0.02160542458295822,
+            0.00390140013769269,
+            -0.0024866692256182432,
+            0.012704171240329742,
+            0.02105998620390892,
+            0.012461753562092781,
+            -0.01790856383740902,
+            0.0013626492582261562,
+            0.011961768381297588,
+            -0.00595815759152174,
+            0.04730163514614105,
+            0.0019791084341704845,
+            -0.01437078882008791,
+            -0.016105586662888527,
+            0.013726867735385895,
+            0.02525683119893074,
+            -0.02355991117656231,
+            0.006325571332126856,
+            0.005621046293526888,
+            -0.004147605039179325,
+            -0.010742107406258583,
+            -0.030377892777323723,
+            -0.005280147306621075,
+            -0.03436262533068657,
+            0.023150833323597908,
+            0.015491968020796776,
+            0.014764715917408466,
+            -0.025044716894626617,
+            -0.004280176945030689,
+            0.0046400148421525955,
+            -0.021287251263856888,
+            0.003261267440393567,
+            0.01161329448223114,
+            0.0005104016163386405,
+            -0.020984230563044548,
+            0.00959820207208395,
+            0.013469300232827663,
+            0.06848283112049103,
+            0.0007442962378263474,
+            0.029771849513053894,
+            0.0422411784529686,
+            -0.0012243957025930285,
+            -0.006371024530380964,
+            0.0036286809481680393,
+            -0.019953956827521324,
+            0.003473382443189621,
+            0.025878025218844414,
+            -0.03224147483706474,
+            -0.055058982223272324,
+            -0.007545232307165861,
+            -0.04624105989933014,
+            0.001489539397880435,
+            0.014711687341332436,
+            0.023044776171445847,
+            -0.0037782976869493723,
+            -0.034817155450582504,
+            0.016908591613173485,
+            -0.003071879269555211,
+            0.007014944683760405,
+            -0.02593863010406494,
+            -0.0294839795678854,
+            -0.026014383882284164,
+            -0.02339324913918972,
+            -0.012908710166811943,
+            -0.022726602852344513,
+            0.012590537779033184,
+            -0.0015103721525520086,
+            -0.06587684899568558,
+            -0.009780014865100384,
+            0.016166189685463905,
+            0.001200722181238234,
+            0.011939042247831821,
+            0.011083006858825684,
+            -0.007083124481141567,
+            -0.0018275976181030273,
+            -0.022559940814971924,
+            0.03124150261282921,
+            0.005177877843379974,
+            0.03584742918610573,
+            -0.006503595970571041,
+            -0.009128518402576447,
+            0.01073453202843666,
+            0.04527139291167259,
+            0.0017916138749569654,
+            0.010226971469819546,
+            -0.020560000091791153,
+            0.0015065843472257257,
+            -0.02284781076014042,
+            -0.0031135445460677147,
+            -0.008863375522196293,
+            -0.0021211495622992516,
+            0.015893470495939255,
+            0.0019753205124288797,
+            -0.010204244405031204,
+            0.025438643991947174,
+            0.02251448854804039,
+            -0.009605777449905872,
+            -0.015969226136803627,
+            -0.02437806874513626,
+            -0.014545026235282421,
+            -0.0002627763315103948,
+            -0.010007280856370926,
+            -0.022453883662819862,
+            -0.020878173410892487,
+            -0.002566212322562933,
+            -0.02559015527367592,
+            0.022529639303684235,
+            0.01220418605953455,
+            -4.882668508798815e-05,
+            -0.006723286584019661,
+            -0.038544319570064545,
+            -0.003727162955328822,
+            -0.03118089959025383,
+            -0.005374841392040253,
+            0.03287781774997711,
+            -0.03745344281196594,
+            -0.0002231231628684327,
+            0.014491996727883816,
+            -0.01993880607187748,
+            0.0028938541654497385,
+            0.008961857296526432,
+            0.02769615314900875,
+            -0.027408283203840256,
+            0.0001795638381736353,
+            0.006954340264201164,
+            0.016908591613173485,
+            -0.009878496639430523,
+            0.027196168899536133,
+            -0.04275631532073021,
+            0.011037553660571575,
+            -0.037120118737220764,
+            -0.020423639565706253,
+            -0.01523439958691597,
+            0.008363389410078526,
+            -0.005018791649490595,
+            -0.02637801133096218,
+            -0.010454237461090088,
+            0.006026337388902903,
+            -0.00693918950855732,
+            -0.01902974210679531,
+            -0.010620899498462677,
+            0.027438584715127945,
+            0.00931033119559288,
+            0.007977037690579891,
+            -0.02955973520874977,
+            -0.01631770096719265,
+            0.006280118133872747,
+            0.052089374512434006,
+            -0.01584801822900772,
+            0.014938953332602978,
+            -0.006639956030994654,
+            0.02342355251312256,
+            -0.028484009206295013,
+            -0.01872672140598297,
+            0.0315445251762867,
+            0.013946558348834515,
+            0.007064185570925474,
+            0.028953691944479942,
+            0.027196168899536133,
+            0.027105262503027916,
+            -0.04654408246278763,
+            0.009886072017252445,
+            -0.002613559365272522,
+            -0.008757317438721657,
+            0.03402930125594139,
+            0.01916610263288021,
+            0.022150862962007523,
+            0.0009744031121954322,
+            -0.01730252057313919,
+            0.0139844361692667,
+            0.014885924756526947,
+            0.007681591901928186,
+            -0.014242004603147507,
+            -0.02660527639091015,
+            0.003367325058206916,
+            -0.007992188446223736,
+            0.00047276070108637214,
+            0.020726662129163742,
+            -0.009052763693034649,
+            0.030999086797237396,
+            -0.014605630189180374,
+            -0.02593863010406494,
+            0.008916404098272324,
+            -0.02268115058541298,
+            -0.0041741193272173405,
+            0.0004476667381823063,
+            0.010083036497235298,
+            -0.023847781121730804,
+            -0.04669559374451637,
+            0.019408518448472023,
+            0.028393102809786797,
+            0.005579380784183741,
+            0.01328748743981123,
+            -0.035332292318344116,
+            0.008992158807814121,
+            0.012310243211686611,
+            0.011348150670528412,
+            0.0019847899675369263,
+            0.035362593829631805,
+            -0.00028834378463216126,
+            0.012885984033346176,
+            -0.008620957843959332,
+            0.028362801298499107,
+            0.00667025800794363,
+            -0.0323626808822155,
+            -0.0378473699092865,
+            -0.01513591781258583,
+            0.015257125720381737,
+            0.006692984607070684,
+            -0.011840560473501682,
+            -0.02278720773756504,
+            -0.038817036896944046,
+            0.015787413343787193,
+            -0.012916285544633865,
+            -0.038271598517894745,
+            -0.010158791206777096,
+            -0.009742137044668198,
+            0.016635872423648834,
+            -0.006916462909430265,
+            -0.016802534461021423,
+            -0.016090434044599533,
+            0.005821797996759415,
+            7.161247049225494e-05,
+            0.0244083721190691,
+            -0.001251857029274106,
+            -0.006855858489871025,
+            -0.016393456608057022,
+            -0.01188601367175579,
+            -0.0035074723418802023,
+            0.03602924197912216,
+            -0.014219277538359165,
+            -0.02383263036608696,
+            0.014984406530857086,
+            0.02657497487962246,
+            0.011636020615696907,
+            0.03227177634835243,
+            0.007181606721132994,
+            -0.014022313989698887,
+            0.003016956616193056,
+            -0.033029329031705856,
+            -0.0046400148421525955,
+            0.005196816753596067,
+            -0.006427840795367956,
+            -0.015651054680347443,
+            -0.01815097965300083,
+            -0.015832867473363876,
+            0.030756669119000435,
+            -0.02487805485725403,
+            0.0218023881316185,
+            0.030347591266036034,
+            0.003174148965626955,
+            -0.02915065549314022,
+            0.013598084449768066,
+            -0.0072005451656877995,
+            0.01862066239118576,
+            0.0019014590652659535,
+            0.028014326468110085,
+            -0.007461901288479567,
+            -0.004818039946258068,
+            -0.02501441352069378,
+            -0.03696860745549202,
+            -0.023484155535697937,
+            -0.008817922323942184,
+            0.0027537066489458084,
+            0.020984230563044548,
+            -0.0375746488571167,
+            -0.017757052555680275,
+            0.009855770505964756,
+            0.015757111832499504,
+            -0.03027183562517166,
+            -0.0025908327661454678,
+            -0.010158791206777096,
+            -0.0004055278259329498,
+            0.005140000022947788,
+            -0.02227207086980343,
+            -0.008977008052170277,
+            -0.015726808458566666,
+            -0.0010435298318043351,
+            0.006166485138237476,
+            -0.0007409819518215954,
+            -0.05805889517068863,
+            -0.013969285413622856,
+            0.018635815009474754,
+            0.0026400736533105373,
+            0.008908828720450401,
+            0.015938924625515938,
+            -0.0111739132553339,
+            0.0031533162109553814,
+            -0.026756787672638893,
+            0.023893235251307487,
+            -0.009492144919931889,
+            0.009355784393846989,
+            -0.013742019422352314,
+            0.012514783069491386,
+            0.014014738611876965,
+            0.004795313347131014,
+            0.0010681503918021917,
+            -0.0054051438346505165,
+            0.01601468026638031,
+            0.028377952054142952,
+            -0.023756874725222588,
+            0.011454207822680473,
+            -0.0002514130319468677,
+            -0.0032158142421394587,
+            -0.029196109622716904,
+            0.00509454682469368,
+            0.003931702114641666,
+            -0.005109698045998812,
+            0.016651025041937828,
+            -0.036544378846883774,
+            0.01662072166800499,
+            0.02177208475768566,
+            0.032544493675231934,
+            0.014832896180450916,
+            -0.017287369817495346,
+            -0.011658747680485249,
+            -0.012385998852550983,
+            -0.002624922664836049,
+            0.022181164473295212,
+            0.012120855040848255,
+            -0.01815097965300083,
+            -0.024287162348628044,
+            0.011249667964875698,
+            0.0005283934879116714,
+            -0.01203752402216196,
+            0.010795135982334614,
+            -0.013166278600692749,
+            0.01349202636629343,
+            0.012711746618151665,
+            0.00866641104221344,
+            0.00888610165566206,
+            0.022862963378429413,
+            -0.011878437362611294,
+            -0.017711598426103592,
+            0.00656041270121932,
+            -0.008916404098272324,
+            -0.0034089903347194195,
+            -0.012598113156855106,
+            0.010795135982334614,
+            0.014878349378705025,
+            0.06490717828273773,
+            -0.013143551535904408,
+            0.0009526234352961183,
+            -0.020438792183995247,
+            -0.031938452273607254,
+            -0.0017944546416401863,
+            -0.020908474922180176,
+            0.0111739132553339,
+            0.008370965719223022,
+            0.01598437689244747,
+            0.012022373266518116,
+            -0.0021912232041358948,
+            0.00795431062579155,
+            0.019317612051963806,
+            -0.010423935018479824,
+            7.255941454786807e-05,
+            -0.006503595970571041,
+            0.0013683309080079198,
+            0.0035851215943694115,
+            -0.011719351634383202,
+            -0.0026949963066726923,
+            0.035695917904376984,
+            0.01068150345236063,
+            0.011651171371340752,
+            -0.03445352986454964,
+            0.0020435003098100424,
+            0.020150920376181602,
+            0.007602048572152853,
+            -0.008302785456180573,
+            -0.01563590206205845,
+            0.01706010289490223,
+            -0.015317730605602264,
+            -0.0022499337792396545,
+            0.002888172399252653,
+            -0.010151215828955173,
+            0.04581683129072189,
+            -0.019787294790148735,
+            -0.0011742077767848969,
+            0.031847547739744186,
+            0.0030207443051040173,
+            -0.0040983641520142555,
+            0.005408931523561478,
+            -0.003473382443189621,
+            -0.0069013116881251335,
+            -0.02139330841600895,
+            0.0031381649896502495,
+            0.00667025800794363,
+            -0.019287310540676117,
+            -0.0072005451656877995,
+            -0.007719469256699085,
+            0.006897523999214172,
+            0.023105379194021225,
+            -0.009492144919931889,
+            -0.00016145357221830636,
+            -0.008575504645705223,
+            0.009848194196820259,
+            -0.0011912527261301875,
+            -0.044301725924015045,
+            -0.006147546228021383,
+            0.021181194111704826,
+            0.000973456131760031,
+            0.0005757405888289213,
+            -0.007450537756085396,
+            0.005007428117096424,
+            -0.017423728480935097,
+            -0.008454295806586742,
+            0.02369627170264721,
+            0.01736312545835972,
+            -0.009113367646932602,
+            0.010863316245377064,
+            -0.0035680767614394426,
+            0.00433320552110672,
+            -0.020014561712741852,
+            -0.01095422264188528,
+            0.003462019143626094,
+            0.015272277407348156,
+            -0.0009819786064326763,
+            -0.029029447585344315,
+            -0.0012272365856915712,
+            -0.00024194362049456686,
+            0.02918095886707306,
+            0.01486319862306118,
+            -0.008961857296526432,
+            -0.021211495622992516,
+            -0.013242034241557121,
+            -0.001237652963027358,
+            -0.015726808458566666,
+            -0.009620928205549717,
+            0.011855711229145527,
+            0.0009137988090515137,
+            0.008992158807814121,
+            -0.018135828897356987,
+            -0.027347678318619728,
+            -0.016166189685463905,
+            0.0018465365283191204,
+            -0.012870832346379757,
+            0.003340810537338257,
+            0.015514694154262543,
+            -0.03430201858282089,
+            -0.015060162171721458,
+            -0.03805948421359062,
+            0.008280059322714806,
+            -0.0029658216517418623,
+            -0.02024182677268982,
+            -0.023605365306138992,
+            0.009484568610787392,
+            0.019181253388524055,
+            -0.0015548784285783768,
+            0.019847899675369263,
+            -0.000889651826582849,
+            0.020060013979673386,
+            -0.0073141781613230705,
+            0.0031703610438853502,
+            -0.013007191941142082,
+            0.0030245319940149784,
+            0.008333087898790836,
+            0.02890823967754841,
+            -0.023075077682733536,
+            -0.006503595970571041,
+            -0.004685468040406704,
+            0.004746072459965944,
+            0.012658718042075634,
+            0.014613205567002296,
+            0.020741812884807587,
+            -0.008272483013570309,
+            -0.0027612822595983744,
+            0.012484480626881123,
+            -0.003799130441620946,
+            0.007295239716768265,
+            0.002390081062912941,
+            0.0019677451346069574,
+            0.004162756260484457,
+            0.008477022871375084,
+            0.0053710537031292915,
+            0.027650700882077217,
+            -0.0028275682125240564,
+            0.0031703610438853502,
+            0.024620486423373222,
+            0.02322658896446228,
+            -0.018878230825066566,
+            -0.028074929490685463,
+            0.002024561632424593,
+            0.050695475190877914,
+            0.007245998363941908,
+            -0.006810405291616917,
+            -0.03927157074213028,
+            -0.0031760428100824356,
+            0.028635520488023758,
+            0.023650817573070526,
+            0.01584801822900772,
+            -0.010567869991064072,
+            -0.012098127976059914,
+            0.02772645466029644,
+            0.013295062817633152,
+            0.02616589516401291,
+            -0.025302285328507423,
+            0.010704229585826397,
+            -0.03869583085179329,
+            0.0449683703482151,
+            0.004439263138920069,
+            -0.014476845972239971,
+            0.0011884119594469666,
+            0.010386057198047638,
+            -0.004537744913250208,
+            0.006431628484278917,
+            -5.8059369621332735e-05,
+            -0.0008086882880888879,
+            0.011923890560865402,
+            0.012007221579551697,
+            0.004265025723725557,
+            -0.02399929240345955,
+            0.007359631359577179,
+            0.004272601567208767,
+            0.03005971945822239,
+            0.005431658122688532,
+            -0.0013863227795809507,
+            0.012385998852550983,
+            -0.005810434930026531,
+            0.008855799213051796,
+            -0.010719381272792816,
+            0.020469093695282936,
+            -0.02918095886707306,
+            1.0986003871948924e-05,
+            0.02051454596221447,
+            -0.0015785519499331713,
+            0.0038199631962925196,
+            -0.023741723969578743,
+            0.01186328660696745,
+            -0.052968136966228485,
+            0.021166043356060982,
+            -0.018741872161626816,
+            0.021741783246397972,
+            0.019590331241488457,
+            -0.009734561666846275,
+            -0.012052674777805805,
+            0.03545350208878517,
+            0.0026760573964565992,
+            -0.014264730736613274,
+            -0.010643625631928444,
+            -0.017999470233917236,
+            -0.02830219641327858,
+            0.041968461126089096,
+            -0.0006117243901826441,
+            0.0028275682125240564,
+            0.0024222771171480417,
+            -0.010666352696716785,
+            -0.008923979476094246,
+            0.001954487757757306,
+            0.020499395206570625,
+            -0.03369597718119621,
+            -0.011128460057079792,
+            -0.008689138107001781,
+            -0.02566591091454029,
+            -0.00020300064352340996,
+            -0.011961768381297588,
+            0.026211349293589592,
+            0.01240872498601675,
+            -0.03696860745549202,
+            0.003780191531404853,
+            0.023650817573070526,
+            0.001603172393515706,
+            0.018120678141713142,
+            0.002130619017407298,
+            0.0010795136913657188,
+            0.009742137044668198,
+            -0.010476963594555855,
+            0.026559824123978615,
+            -0.009204274043440819,
+            0.008658835664391518,
+            -0.001950700068846345,
+            -0.017287369817495346,
+            0.017969166859984398,
+            0.004083212930709124,
+            0.024120500311255455,
+            0.011954193003475666,
+            0.011605718173086643,
+            0.01112088467925787,
+            -0.019878201186656952,
+            0.020090317353606224,
+            -0.0018891488434746861,
+            0.009075489826500416,
+            0.024726543575525284,
+            -0.005863463506102562,
+            0.020832719281315804,
+            0.015257125720381737,
+            -0.00020229043730068952,
+            0.011295121163129807,
+            -0.020120618864893913,
+            -0.011779955588281155,
+            0.0085679292678833,
+            -0.009605777449905872,
+            -0.010325453244149685,
+            0.009901223704218864,
+            -0.010105762630701065,
+            -0.015052586793899536,
+            -0.0071247899904847145,
+            0.016635872423648834,
+            0.0028976418543606997,
+            -0.02610529027879238,
+            -1.7326075976598077e-05,
+            0.008901252411305904,
+            0.006166485138237476,
+            0.018741872161626816,
+            0.011317848227918148,
+            -0.006692984607070684,
+            0.010363331064581871,
+            0.011476934887468815,
+            0.0053104497492313385,
+            -0.005079395603388548,
+            -0.013431422412395477,
+            -0.0024241709616035223,
+            0.018090376630425453,
+            0.009810317307710648,
+            -0.01922670565545559,
+            0.004564259201288223,
+            0.0008877579239197075,
+            -0.01420412678271532,
+            -0.019347915425896645,
+            -0.0038483713287860155,
+            -0.004170331638306379,
+            0.0027915844693779945,
+            0.0509984977543354,
+            -0.0006216673064045608,
+            0.007893706671893597,
+            -0.025741664692759514,
+            -0.006299056578427553,
+            -0.00027556007262319326,
+            -0.0036097420379519463,
+            0.01272689737379551,
+            -0.01824188604950905,
+            -0.03724132478237152,
+            0.012711746618151665,
+            -0.0012584856012836099,
+            0.014923802576959133,
+            -0.00297907879576087,
+            0.014226853847503662,
+            -0.017347972840070724,
+            0.01916610263288021,
+            -0.0011117096291854978,
+            -0.016029831022024155,
+            0.0029374135192483664,
+            -0.011143610812723637,
+            -0.021317552775144577,
+            0.005916492082178593,
+            -0.006848283112049103,
+            -0.03896854817867279,
+            0.0029885482508689165,
+            -0.010423935018479824,
+            0.0011798894265666604,
+            -0.003912763670086861,
+            0.020166072994470596,
+            -0.0061513339169323444,
+            0.0008697660523466766,
+            0.02177208475768566,
+            0.0017026013229042292,
+            0.01601468026638031,
+            -0.014499573037028313,
+            0.010999675840139389,
+            0.012598113156855106,
+            0.02210540883243084,
+            -0.032180868089199066,
+            -0.01393140759319067,
+            0.014211702160537243,
+            0.0026116655208170414,
+            -0.005382417235523462,
+            0.020635755732655525,
+            -0.01294658798724413,
+            0.019650936126708984,
+            -0.0005577487172558904,
+            -0.011704200878739357,
+            -0.03524138405919075,
+            -0.016529815271496773,
+            -3.343888238305226e-05,
+            -0.008439145050942898,
+            0.025317436084151268,
+            0.01760554127395153,
+            -0.0108330138027668,
+            0.026468917727470398,
+            0.0037574649322777987,
+            0.014598054811358452,
+            -0.014257155358791351,
+            -0.005999823100864887,
+            -0.0003773563075810671,
+            -0.0119314668700099,
+            0.015264702029526234,
+            -0.03512017801403999,
+            0.0008342557121068239,
+            -0.00839369185268879,
+            0.005242269951850176,
+            -0.0254992488771677,
+            0.018257036805152893,
+            0.025817420333623886,
+            -0.0031419529113918543,
+            -0.005037730094045401,
+            -0.022802358493208885,
+            0.028968842700123787,
+            0.03049910068511963,
+            0.0024828813038766384,
+            -0.0013825349742546678,
+            -0.00352072948589921,
+            0.0033218718599528074,
+            0.0211205892264843,
+            -0.037998881191015244,
+            -0.009098216891288757,
+            -0.028696123510599136,
+            0.042332086712121964,
+            -0.008098246529698372,
+            -0.013431422412395477,
+            0.01862066239118576,
+            -0.0024715180043131113,
+            0.02355991117656231,
+            -0.004689255729317665,
+            -0.004855917301028967,
+            0.03148392215371132,
+            0.011514811776578426,
+            0.02772645466029644,
+            -0.04015033319592476,
+            0.014704111963510513,
+            -0.006867222022265196,
+            0.004867280833423138,
+            -0.013484450988471508,
+            0.005302873905748129,
+            0.02580226957798004,
+            -0.018060073256492615,
+            0.0068066176027059555,
+            0.030665762722492218,
+            0.00748084019869566,
+            0.002448791405186057,
+            0.0165904201567173,
+            0.003890036838129163,
+            0.010855740867555141,
+            0.000841831264551729,
+            0.0052574207074940205,
+            0.022423582151532173,
+            0.014454119838774204,
+            -0.016029831022024155,
+            -0.015120766125619411,
+            -0.00044790346873924136,
+            -0.004450626205652952,
+            0.005261208396404982,
+            0.013242034241557121,
+            0.0061172437854111195,
+            -0.015681356191635132,
+            0.01052999310195446,
+            0.014158673584461212,
+            0.013128400780260563,
+            -0.0039847311563789845,
+            -0.03412020578980446,
+            -0.02443867363035679,
+            -0.021438762545585632,
+            0.003104075090959668,
+            -0.0008281006012111902,
+            -0.0021211495622992516,
+            0.0017954016802832484,
+            0.017347972840070724,
+            -0.0008352026343345642,
+            0.00379155483096838,
+            0.01686313934624195,
+            -0.01415109820663929,
+            -0.022090258076786995,
+            -0.0035964848939329386,
+            -0.027529491111636162,
+            0.0053710537031292915,
+            0.001844642567448318,
+            -0.00965123064815998,
+            0.008530051447451115,
+            -0.0046059247106313705,
+            -0.007223271764814854,
+            -0.012196609750390053,
+            -0.02007516659796238,
+            0.005780132487416267,
+            0.016878290101885796,
+            0.005730891600251198,
+            -0.009272454306483269,
+            -0.003939277958124876,
+            0.011575416661798954,
+            0.017484333366155624,
+            0.017317671328783035,
+            -0.009295180439949036,
+            0.009628504514694214,
+            0.0004022135108243674,
+            0.004670316819101572,
+            -0.01834794320166111,
+            0.0045869858004152775,
+            -0.0027953721582889557,
+            -0.008658835664391518,
+            0.01105270441621542,
+            -0.00031201730598695576,
+            0.01628739945590496,
+            -0.013090522959828377,
+            0.005624833982437849,
+            0.010333028621971607,
+            -0.020590301603078842,
+            -0.0067308624275028706,
+            -0.01692374423146248,
+            0.026484068483114243,
+            -0.00504530593752861,
+            0.00045405857963487506,
+            -0.02978700026869774,
+            0.0033616432920098305,
+            0.000848933297675103,
+            -0.000845619011670351,
+            -0.012764775194227695,
+            0.010749682784080505,
+            0.03242328763008118,
+            0.0005179771687835455,
+            0.0216660276055336,
+            -0.0036154238041490316,
+            0.0007480839849449694,
+            -0.00785582885146141,
+            0.004977126140147448,
+            -0.004916521720588207,
+            -0.009901223704218864,
+            -0.010863316245377064,
+            0.021726632490754128,
+            -0.0033029329497367144,
+            0.0003356908564455807,
+            0.011332998983561993,
+            -0.012067826464772224,
+            0.0014620780711993575,
+            -0.0047687990590929985,
+            0.00971941091120243,
+            0.0061702728271484375,
+            0.009113367646932602,
+            -0.01176480483263731,
+            0.05451354384422302,
+            -0.0106284748762846,
+            0.005537715740501881,
+            -0.00439380994066596,
+            -0.022226616740226746,
+            -0.017317671328783035,
+            0.012711746618151665,
+            -0.017544938251376152,
+            0.03375658020377159,
+            0.024150803685188293,
+            0.0006467612111009657,
+            0.010393633507192135,
+            -0.006958128418773413,
+            0.023923536762595177,
+            -0.002602196065708995,
+            0.0010795136913657188,
+            -0.004855917301028967,
+            0.015802564099431038,
+            0.04511988162994385,
+            0.017393426969647408,
+            -0.001152428099885583,
+            0.00829521007835865,
+            0.020575150847434998,
+            0.00814369972795248,
+            -0.0020075165666639805,
+            0.014219277538359165,
+            -0.01024212222546339,
+            0.00900731049478054,
+            -0.004458202049136162,
+            0.027438584715127945,
+            0.00814369972795248,
+            -0.006310420110821724,
+            -0.0036836036015301943,
+            -0.0008209985098801553,
+            0.014173824340105057,
+            0.006950552575290203,
+            0.01963578537106514,
+            -0.00461350055411458,
+            -0.0012575386790558696,
+            0.013870803639292717,
+            0.014491996727883816,
+            -0.00552635220810771,
+            0.001724380999803543,
+            -0.01435563713312149,
+            -0.010204244405031204,
+            0.013575357384979725,
+            0.006086941808462143,
+            0.006310420110821724,
+            -0.014514723792672157,
+            -0.00980274099856615,
+            -0.004670316819101572,
+            0.0015435151290148497,
+            0.015052586793899536,
+            0.011060279794037342,
+            -0.004814252257347107,
+            0.006052851676940918,
+            0.013825350441038609,
+            -0.014014738611876965,
+            0.006147546228021383,
+            0.007208121009171009,
+            0.008855799213051796,
+            0.004787737503647804,
+            -0.010817863047122955,
+            -0.00428396463394165,
+            -0.0063937511295080185,
+            0.0012670081341639161,
+            0.016832837834954262,
+            0.013181429356336594,
+            0.0049581872299313545,
+            0.027529491111636162,
+            -0.00839369185268879,
+            -0.012302667833864689,
+            0.02339324913918972,
+            0.011257244274020195,
+            0.0031703610438853502,
+            0.014651083387434483,
+            -0.012635990977287292,
+            -0.012446602806448936,
+            0.006655106786638498,
+            0.004848341923207045,
+            0.006049063988029957,
+            0.009151245467364788,
+            0.00450365524739027,
+            0.007333117071539164,
+            -0.007098275702446699,
+            -0.007427811156958342,
+            0.0014810169814154506,
+            0.01642375811934471,
+            0.020393338054418564,
+            -0.024968961253762245,
+            -0.0018806264270097017,
+            0.0009052763343788683,
+            0.010878467001020908,
+            -0.003316190093755722,
+            0.012310243211686611,
+            5.65205882594455e-05,
+            -0.0074240234680473804,
+            0.010158791206777096,
+            -0.02119634486734867,
+            -0.012681444175541401,
+            0.0012717428617179394,
+            0.013037494383752346,
+            -0.019590331241488457,
+            0.018226735293865204,
+            -0.008893677033483982,
+            -0.013348091393709183,
+            0.013893529772758484,
+            0.020014561712741852,
+            0.004090788308531046,
+            -0.007738408166915178,
+            -0.021862991154193878,
+            -0.01127239502966404,
+            0.016060132533311844,
+            0.020272130146622658,
+            0.000947888707742095,
+            -0.0006472347304224968,
+            0.010173942893743515,
+            -0.013113250024616718,
+            -0.023605365306138992,
+            -0.005230906419456005,
+            0.010386057198047638,
+            0.01746918261051178,
+            0.018832778558135033,
+            -0.006685409229248762,
+            0.02443867363035679,
+            -0.0059127043932676315,
+            -0.015499543398618698,
+            2.86597842205083e-05,
+            -0.02802947722375393,
+            -0.017423728480935097,
+            0.00013778003631159663,
+            0.00246773031540215,
+            0.008598231710493565,
+            0.0008555619278922677,
+            0.01943882182240486,
+            0.000904329412151128,
+            0.017590390518307686,
+            -0.000659071491099894,
+            -0.002009410411119461,
+            -0.0013995799235999584,
+            -0.0015340456739068031,
+            0.02254479005932808,
+            -0.012166308239102364,
+            -0.01176480483263731,
+            0.022302372381091118,
+            0.006852070800960064,
+            0.014719262719154358,
+            -0.027635548263788223,
+            0.013757170177996159,
+            -0.015499543398618698,
+            0.017423728480935097,
+            -0.03408990427851677,
+            0.006420265417546034,
+            0.01489350013434887,
+            -0.010348179377615452,
+            -0.020499395206570625,
+            -0.029650641605257988,
+            -0.024150803685188293,
+            -0.0038710979279130697,
+            -0.010855740867555141,
+            -0.016847988590598106,
+            0.006121031939983368,
+            -0.005802859086543322,
+            0.018499454483389854,
+            -0.007079336792230606,
+            0.033877789974212646,
+            0.006969491485506296,
+            -0.004056698642671108,
+            0.003835114184767008,
+            0.022469034418463707,
+            -0.006102093029767275,
+            0.02109028771519661,
+            0.004408961161971092,
+            -0.001467759720981121,
+            0.013681414537131786,
+            -0.006552837323397398,
+            -0.002361672930419445,
+            0.005821797996759415,
+            -0.010484539903700352,
+            0.022014502435922623,
+            0.005924067925661802,
+            -0.007708106189966202,
+            -0.0006775368237867951,
+            -0.011454207822680473,
+            0.014726839028298855,
+            0.006530110724270344,
+            0.0022480397019535303,
+            -0.011295121163129807,
+            0.011249667964875698,
+            -0.004859705455601215,
+            0.0040642740204930305,
+            0.009484568610787392,
+            -0.01774190180003643,
+            -0.028862785547971725,
+            0.011416330002248287,
+            0.019120648503303528,
+            0.006382387597113848,
+            -0.0055755930952727795,
+            -0.0025681061670184135,
+            0.00839369185268879,
+            0.010651201009750366,
+            0.022150862962007523,
+            -0.012984465807676315,
+            -8.889415767043829e-05,
+            0.013204156421124935,
+            -0.007170243188738823,
+            -0.01056029461324215,
+            0.00487485621124506,
+            0.023438703268766403,
+            0.0068823727779090405,
+            -0.04327145218849182,
+            -0.018469152972102165,
+            0.01909034699201584,
+            0.007136153522878885,
+            -0.04154422879219055,
+            0.0026381798088550568,
+            -0.003689285134896636,
+            -0.023787178099155426,
+            0.016105586662888527,
+            -0.0011817833874374628,
+            -0.01828734017908573,
+            0.028756728395819664,
+            0.02830219641327858,
+            0.01834794320166111,
+            2.6987840101355687e-05,
+            -0.013332940638065338,
+            -0.0318172425031662,
+            -0.023590214550495148,
+            0.008280059322714806,
+            -0.02660527639091015,
+            -0.0072725131176412106,
+            0.011242092587053776,
+            -0.017454031854867935,
+            0.02972639724612236,
+            -0.008507325313985348,
+            0.026938600465655327,
+            0.011446632444858551,
+            0.001691238023340702,
+            -0.013446573168039322,
+            -0.018802475184202194,
+            -0.01350717805325985,
+            -0.008196728304028511,
+            0.014090493321418762,
+            0.017923714593052864,
+            0.011007251217961311,
+            0.011332998983561993,
+            0.02251448854804039,
+            -0.012158732861280441,
+            0.007969462312757969,
+            -2.888172457460314e-05,
+            -0.008961857296526432,
+            0.0026343921199440956,
+            -0.018393397331237793,
+            0.00868156272917986,
+            0.0062308767810463905,
+            0.005458172410726547,
+            -0.007208121009171009,
+            0.010742107406258583,
+            0.02712041325867176,
+            -0.018847929313778877,
+            0.011401179246604443,
+            -0.031362712383270264,
+            -0.014787442982196808,
+            -0.01056029461324215,
+            0.026862844824790955,
+            0.007757347077131271,
+            -0.015726808458566666,
+            -0.007514929864555597,
+            -0.01545409020036459,
+            0.012885984033346176,
+            0.0032442226074635983,
+            0.005764981731772423,
+            0.015817714855074883,
+            0.021893294528126717,
+            0.01746918261051178,
+            -0.00795431062579155,
+            0.006852070800960064,
+            0.004674104508012533,
+            0.028090080246329308,
+            -0.01777220331132412,
+            0.020802417770028114,
+            0.009954252280294895,
+            -0.003935490269213915,
+            0.005685438401997089,
+            -0.014257155358791351,
+            0.008530051447451115,
+            -0.007325541693717241,
+            0.030514251440763474,
+            0.0033067206386476755,
+            0.014075342565774918,
+            -0.02062060497701168,
+            0.006575563922524452,
+            -0.03172633796930313,
+            0.009022461250424385,
+            0.013704141601920128,
+            0.012976890429854393,
+            -0.007677804213017225,
+            -0.024181105196475983,
+            -0.0119314668700099,
+            0.0057990713976323605,
+            -0.01279507763683796,
+            -0.01046181283891201,
+            -0.0034828518982976675,
+            0.012446602806448936,
+            0.018847929313778877,
+            -0.01848430372774601,
+            -0.019923655316233635,
+            -0.009583051316440105,
+            0.02898399345576763,
+            -0.015787413343787193,
+            -0.001471547526307404,
+            0.02413565292954445,
+            0.021241798996925354,
+            0.015022284351289272,
+            0.002155239460989833,
+            0.020090317353606224,
+            -0.0007111532613635063,
+            -0.0237265732139349,
+            -0.02813553437590599,
+            0.007302815094590187,
+            -0.006912675220519304,
+            0.01943882182240486,
+            -0.0035548193845897913,
+            -0.005658924113959074,
+            -0.0034033088013529778,
+            -0.007503566797822714,
+            -0.0165904201567173,
+            -0.01787826046347618,
+            0.0021647089160978794,
+            -0.008454295806586742,
+            0.005863463506102562,
+            -0.0002795845502987504,
+            -0.013105674646794796,
+            0.013401119969785213,
+            -0.014075342565774918,
+            0.005772557109594345,
+            -0.002653330797329545,
+            0.001962063368409872,
+            0.0035926972050219774,
+            0.009113367646932602,
+            -0.007336904760450125,
+            -0.06381630152463913,
+            -0.013105674646794796,
+            -0.003704436356201768,
+            0.006071790587157011,
+            -0.00019578021601773798,
+            -0.016635872423648834,
+            -0.013196581043303013,
+            0.01706010289490223,
+            0.008863375522196293,
+            -0.013870803639292717,
+            0.016954045742750168,
+            0.009961827658116817,
+            0.003622999181970954,
+            -0.024044746533036232,
+            -0.0008148433989845216,
+            0.008757317438721657,
+            0.032756611704826355,
+            -0.008643684908747673,
+            -0.0003839849086944014,
+            0.0036684523802250624,
+            -0.0033332351595163345,
+            -0.010052734054625034,
+            -0.014582903124392033,
+            -0.011848135851323605,
+            0.011908739805221558,
+            0.01790856383740902,
+            -0.014264730736613274,
+            -0.02007516659796238,
+            -0.0007106798002496362,
+            0.012931437231600285,
+            -0.026590125635266304,
+            0.0007244104635901749,
+            -0.007787649054080248,
+            -0.01052999310195446,
+            0.018302490934729576,
+            -0.019014591351151466,
+            0.005249845329672098,
+            0.00048294031876139343,
+            0.008507325313985348,
+            -0.023544760420918465,
+            -0.024893205612897873,
+            0.0035131541080772877,
+            -0.023847781121730804,
+            0.03612014651298523,
+            -0.021590271964669228,
+            -0.012082977220416069,
+            0.034483831375837326,
+            -0.0037423139438033104,
+            0.01702980138361454,
+            0.0023995505180209875,
+            0.02739313244819641,
+            -0.011439057067036629,
+            -0.00617784820497036,
+            0.012499631382524967,
+            0.009575475007295609,
+            0.0032593735959380865,
+            0.03775646165013313,
+            -0.01257538702338934,
+            -0.013901105150580406,
+            -0.005556654185056686,
+            -0.012954163365066051,
+            0.0006775368237867951,
+            0.014431392773985863,
+            0.028393102809786797,
+            0.004257450345903635,
+            -0.00628769351169467,
+            -0.006143758539110422,
+            -0.0014649189542979002,
+            -0.004242299124598503,
+            -0.00947699323296547,
+            -0.00715130427852273,
+            0.017287369817495346,
+            -0.025241680443286896,
+            0.016711628064513206,
+            -0.006420265417546034,
+            0.021135739982128143,
+            -0.011757229454815388,
+            0.007007369305938482,
+            -0.008961857296526432,
+            0.003492321353405714,
+            0.0023635667748749256,
+            -0.005109698045998812,
+            0.009287605062127113,
+            -0.014772292226552963,
+            0.008454295806586742,
+            0.013742019422352314,
+            0.0011183382011950016,
+            -0.010590597055852413,
+            -0.019272159785032272,
+            0.0022972808219492435,
+            -0.000710206339135766,
+            0.01181783340871334,
+            -0.0047498601488769054,
+            -0.000981031684204936,
+            0.0002921315317507833,
+            -0.014476845972239971,
+            -0.009575475007295609,
+            0.0006050958181731403,
+            0.005302873905748129,
+            -0.008961857296526432,
+            0.010378481820225716,
+            -0.005693014245480299,
+            -0.00034160923678427935,
+            0.02627195231616497,
+            0.006526322569698095,
+            -0.007632351014763117,
+            -0.014098069630563259,
+            0.01563590206205845,
+            0.015370759181678295,
+            -0.007454325910657644,
+            0.01933276280760765,
+            -0.016196493059396744,
+            -0.0008366230758838356,
+            0.013916256837546825,
+            -0.009219424799084663,
+            0.0017698341980576515,
+            -0.008196728304028511,
+            0.03554440662264824,
+            -0.0008787619881331921,
+            -0.0035415622405707836,
+            -0.0006401326390914619,
+            0.012976890429854393,
+            -0.0026779514737427235,
+            0.001182730309665203,
+            0.01132542360574007,
+            0.005321812815964222,
+            0.0185146052390337,
+            0.00328020635060966,
+            -0.03378688171505928,
+            -0.007780073676258326,
+            -0.0065490491688251495,
+            -0.008090670220553875,
+            0.009022461250424385,
+            0.025196226313710213,
+            0.01990850456058979,
+            0.006408901885151863,
+            0.001881573349237442,
+            0.018423698842525482,
+            0.015757111832499504,
+            -0.01489350013434887,
+            0.016363153234124184,
+            0.02802947722375393,
+            -0.010030006989836693,
+            0.022741753607988358,
+            -0.005253633018583059,
+            -0.008893677033483982,
+            0.008946705609560013,
+            -0.0015378334792330861,
+            -0.017393426969647408,
+            -0.015681356191635132,
+            0.005022579338401556,
+            -0.008999735116958618,
+            -0.0022897052112966776,
+            -0.018605511635541916,
+            0.004670316819101572,
+            0.017105557024478912,
+            -0.004965762607753277,
+            -3.5243989259470254e-05,
+            0.019484274089336395,
+            -0.006965703796595335,
+            -0.016908591613173485,
+            0.006325571332126856,
+            -0.015007133595645428,
+            0.007946735247969627,
+            -0.013666263781487942,
+            0.010492115281522274,
+            -0.006098304875195026,
+            0.018817627802491188,
+            0.005446809343993664,
+            -0.008090670220553875,
+            -0.0008958069374784827,
+            -0.018984289839863777,
+            0.02065090648829937,
+            -0.033089932054281235,
+            0.00839369185268879,
+            -0.026226500049233437,
+            -0.0009781909175217152,
+            0.0008849170990288258,
+            0.014605630189180374,
+            -0.001218714052811265,
+            0.019696388393640518,
+            -0.002751812804490328,
+            -0.020090317353606224,
+            -0.007086912170052528,
+            -0.0010634156642481685,
+            -0.01618134044110775,
+            -0.030923331156373024,
+            -0.004272601567208767,
+            0.0036267871037125587,
+            -0.004711982328444719,
+            0.0021135739516466856,
+            0.017984317615628242,
+            -0.005613470915704966,
+            0.009173971600830555,
+            -0.02690829709172249,
+            -0.007416448090225458,
+            -0.00355103169567883,
+            0.0010776197304949164,
+            -0.011242092587053776,
+            0.0010596278589218855,
+            0.011226941831409931,
+            -0.019044892862439156,
+            0.013461724855005741,
+            -0.021378157660365105,
+            -0.004371083341538906,
+            -0.008507325313985348,
+            -0.00135412672534585,
+            -0.013446573168039322,
+            -0.028120383620262146,
+            0.025029566138982773,
+            -0.00021495578403118998,
+            -0.003528305096551776,
+            -0.011999646201729774,
+            0.023181134834885597,
+            0.002914686920121312,
+            0.027817361056804657,
+            -0.03157482668757439,
+            0.0013427634257823229,
+            0.011552689597010612,
+            0.0025984083767980337,
+            -0.017287369817495346,
+            0.004746072459965944,
+            -0.005234694108366966,
+            0.02210540883243084,
+            -0.0011694730492308736,
+            -0.0005340751959010959,
+            -0.008234606124460697,
+            0.030999086797237396,
+            0.009136094711720943,
+            0.0012944694608449936,
+            -0.013226882554590702,
+            -0.008060368709266186,
+            -0.002759388415142894,
+            -0.0011069749016314745,
+            0.012855681590735912,
+            0.01290113478899002,
+            -0.012105703353881836,
+            0.0013607552973553538,
+            -0.006878585088998079,
+            0.0011694730492308736,
+            -0.002935519441962242,
+            -0.0004966709529981017,
+            -0.01149208564311266,
+            0.003257479751482606,
+            0.016272246837615967,
+            -0.00780280027538538,
+            -0.006480869371443987,
+            0.0007750718505121768,
+            0.009181547909975052,
+            0.012461753562092781,
+            -0.012742049060761929,
+            -0.018226735293865204,
+            -0.011908739805221558,
+            0.016029831022024155,
+            0.010522416792809963,
+            -0.008204303681850433,
+            -0.013234458863735199,
+            -0.02583257108926773,
+            0.01169662456959486,
+            -0.03293842449784279,
+            -0.012242063879966736,
+            0.005821797996759415,
+            0.028559764847159386,
+            0.019999410957098007,
+            0.012226912193000317,
+            0.001332347048446536,
+            0.002950670663267374,
+            0.00023342114582192153,
+            -0.01916610263288021,
+            0.014408666640520096,
+            -0.011143610812723637,
+            -0.018166130408644676,
+            0.0075906855054199696,
+            0.011939042247831821,
+            0.03148392215371132,
+            0.006942977197468281,
+            -0.0010160685051232576,
+            0.01686313934624195,
+            -0.001218714052811265,
+            -0.013582932762801647,
+            0.01420412678271532,
+            -0.024256860837340355,
+            0.016438908874988556,
+            0.02860521711409092,
+            -0.004780162125825882,
+            0.0008531945641152561,
+            -0.020938776433467865,
+            0.00482561532407999,
+            -0.011529963463544846,
+            -0.002177966060116887,
+            -0.003287781961262226,
+            -0.03460504114627838,
+            0.007946735247969627,
+            -0.006825556512922049,
+            0.01046181283891201,
+            -0.00814369972795248,
+            0.006655106786638498,
+            -0.012431452050805092,
+            0.004253662656992674,
+            0.01513591781258583,
+            0.016105586662888527,
+            0.007060397882014513,
+            -0.025211378931999207,
+            -0.009196698665618896,
+            0.008105821907520294,
+            -0.01349202636629343,
+            -0.02295386977493763,
+            -0.006852070800960064,
+            0.009840618818998337,
+            -0.01061332318931818,
+            -0.00488243205472827,
+            -0.028180986642837524,
+            -0.013681414537131786,
+            -0.0014327229000627995,
+            -0.017817657440900803,
+            -0.005556654185056686,
+            -0.003100287402048707,
+            0.01571165770292282,
+            -0.0049733384512364864,
+            -0.006185423582792282,
+            0.004742284305393696,
+            -0.023878084495663643,
+            0.0037423139438033104,
+            -0.008280059322714806,
+            0.0008266801596619189,
+            -0.0032517979852855206,
+            -0.0014762822538614273,
+            -0.007969462312757969,
+            -0.014332910999655724,
+            0.006893736310303211,
+            -0.016469212248921394,
+            0.002954458352178335,
+            -0.015173795633018017,
+            -0.016499513760209084,
+            -0.00682934420183301,
+            0.00775355938822031,
+            0.021575121209025383,
+            0.006541473791003227,
+            0.021484214812517166,
+            0.005469535943120718,
+            0.028680972754955292,
+            0.0003953482082579285,
+            0.003041577059775591,
+            -8.958956641436089e-06,
+            0.003232859307900071,
+            0.019347915425896645,
+            0.006571775767952204,
+            0.018984289839863777,
+            -0.008742166683077812,
+            0.010514841414988041,
+            -0.010355755686759949,
+            0.010014856234192848,
+            0.006333146709948778,
+            -0.006473293993622065,
+            -0.003799130441620946,
+            -0.010348179377615452,
+            -0.0031419529113918543,
+            -0.016544966027140617,
+            0.0027366618160158396,
+            0.00693161366507411,
+            0.0071588801220059395,
+            -0.011582992039620876,
+            -0.005230906419456005,
+            -0.014704111963510513,
+            -0.0064354161731898785,
+            -0.01831764169037342,
+            0.014385939575731754,
+            0.029226411134004593,
+            -0.016332851722836494,
+            -0.01977214403450489,
+            -0.007772498298436403,
+            0.0007296186522580683,
+            -0.006636167876422405,
+            -0.01240872498601675,
+            -0.009151245467364788,
+            -0.035059571266174316,
+            -0.017666146159172058,
+            0.007586897816509008,
+            0.02484775334596634,
+            0.01387837901711464,
+            -0.013628385961055756,
+            -0.0006983695784583688,
+            -0.016984347254037857,
+            -0.010787560604512691,
+            -0.01535560842603445,
+            -0.02054484933614731,
+            -0.02498411200940609,
+            0.0077952248975634575,
+            -0.004033972043544054,
+            0.014931377954781055,
+            0.008113397285342216,
+            0.0032631612848490477,
+            -0.004371083341538906,
+            -0.0024109138175845146,
+            -0.013196581043303013,
+            0.0012925754999741912,
+            -0.004787737503647804,
+            -0.0017291157273575664,
+            0.03554440662264824,
+            -0.008098246529698372,
+            -0.005871038883924484,
+            0.011060279794037342,
+            0.011128460057079792,
+            -0.013764745555818081,
+            0.0006642796797677875,
+            -0.011249667964875698,
+            0.021847840398550034,
+            -0.021620575338602066,
+            0.019257009029388428,
+            0.0014507147716358304,
+            -0.010840589180588722,
+            0.028999146074056625,
+            0.01892368495464325,
+            0.00743159931153059,
+            0.006579351611435413,
+            0.01672677882015705,
+            0.0044316877610981464,
+            0.016196493059396744,
+            -0.0026552248746156693,
+            -0.004632438998669386,
+            0.009295180439949036,
+            0.015681356191635132,
+            -0.000702630786690861,
+            -0.006942977197468281,
+            0.016438908874988556,
+            0.004056698642671108,
+            0.0075414446182549,
+            -0.01356020662933588,
+            -0.008961857296526432,
+            0.001507531269453466,
+            0.006257391534745693,
+            -0.005939218681305647,
+            -0.0026003022212535143,
+            0.005530139897018671,
+            0.003943065647035837,
+            -0.013302638195455074,
+            0.01689344085752964,
+            -0.016166189685463905,
+            0.008742166683077812,
+            -0.0037972365971654654,
+            0.0121360057964921,
+            -0.006799042224884033,
+            0.0013153020991012454,
+            -0.01818128302693367,
+            -0.003869204083457589,
+            0.0038597346283495426,
+            0.012173883616924286,
+            -0.01262841559946537,
+            0.025484098121523857,
+            0.008166425861418247,
+            0.006602078210562468,
+            0.0059467945247888565,
+            -0.012855681590735912,
+            0.027211319655179977,
+            0.005969521123915911,
+            0.009886072017252445,
+            -0.008893677033483982,
+            0.011136035434901714,
+            0.035362593829631805,
+            0.03060515783727169,
+            0.014135947450995445,
+            0.004469565115869045,
+            -0.018999440595507622,
+            0.015120766125619411,
+            0.0018427487229928374,
+            0.0023806116078048944,
+            0.00863610953092575,
+            -0.011348150670528412,
+            0.01305264513939619,
+            -0.009484568610787392,
+            0.015241974964737892,
+            -0.021272100508213043,
+            -0.011749654076993465,
+            -0.014408666640520096,
+            -0.021075136959552765,
+            -0.01430260855704546,
+            0.034180812537670135,
+            -0.002346521709114313,
+            -0.010393633507192135,
+            0.0045112306252121925,
+            0.008477022871375084,
+            -0.006079366430640221,
+            0.0053142374381423,
+            -0.008204303681850433,
+            -0.013749594800174236,
+            -0.01889338344335556,
+            0.006628592498600483,
+            -0.01550711877644062,
+            0.007651289459317923,
+            -0.003304826794192195,
+            0.0037688282318413258,
+            0.007014944683760405,
+            0.018105527386069298,
+            0.00530666159465909,
+            -0.00135412672534585,
+            0.0014033677289262414,
+            0.0016865032957866788,
+            -0.00931033119559288,
+            0.0108330138027668,
+            0.015408637002110481,
+            -0.0011921997647732496,
+            -0.011279970407485962,
+            -0.009136094711720943,
+            -0.042877525091171265,
+            0.0006264019757509232,
+            -0.013128400780260563,
+            0.0064354161731898785,
+            -0.011196639388799667,
+            0.0022234192583709955,
+            0.015863168984651566,
+            -0.0054051438346505165,
+            -0.0010918239131569862,
+            0.02193874679505825,
+            0.013014768250286579,
+            0.021075136959552765,
+            0.02065090648829937,
+            0.014188976027071476,
+            -0.015302578918635845,
+            -0.03469594568014145,
+            0.002221525413915515,
+            -0.010030006989836693,
+            0.013772320933640003,
+            0.004238511435687542,
+            0.016060132533311844,
+            -0.023893235251307487,
+            -0.011878437362611294,
+            -0.006662682630121708,
+            0.014431392773985863,
+            0.01491622719913721,
+            -0.0032271775417029858,
+            0.02690829709172249,
+            0.009280029684305191,
+            0.0020510759204626083,
+            -0.02281750924885273,
+            0.003312402404844761,
+            0.005321812815964222,
+            -0.008370965719223022,
+            -0.016681326553225517,
+            -0.0004552422615233809,
+            -0.0012168202083557844,
+            -0.0007817004225216806,
+            -0.021650876849889755,
+            0.018832778558135033,
+            -0.004018820822238922,
+            0.0037574649322777987,
+            0.013840501196682453,
+            0.01516621932387352,
+            0.017984317615628242,
+            0.008370965719223022,
+            0.001497115008533001,
+            -0.006605865899473429,
+            -0.0007973249885253608,
+            -0.0005828426801599562,
+            -0.007389933802187443,
+            -0.005598319694399834,
+            0.006382387597113848,
+            -0.01134057529270649,
+            0.008810346014797688,
+            0.000896753859706223,
+            0.006113456096500158,
+            0.010605747811496258,
+            -0.004836978856474161,
+            -0.011628445237874985,
+            0.014628356322646141,
+            -0.0010548931313678622,
+            -0.0055755930952727795,
+            -0.011060279794037342,
+            0.003624893259257078,
+            0.013575357384979725,
+            -0.003096499713137746,
+            -0.013499601744115353,
+            0.00910579226911068,
+            0.002009410411119461,
+            -0.027438584715127945,
+            0.015499543398618698,
+            0.010295150801539421,
+            -0.013370818458497524,
+            0.004980913829058409,
+            -0.003371112747117877,
+            -0.03027183562517166,
+            0.004791525658220053,
+            0.002660906407982111,
+            -0.010643625631928444,
+            0.028938541188836098,
+            -0.003821857040748,
+            0.029256712645292282,
+            -0.004980913829058409,
+            -0.006905099377036095,
+            -0.011355726048350334,
+            0.009416389279067516,
+            -0.018681267276406288,
+            -0.0037953427527099848,
+            -0.008727015927433968,
+            -0.00915882084518671,
+            -0.0009838725673034787,
+            -0.005185453221201897,
+            0.006984642706811428,
+            0.005121061112731695,
+            -0.012704171240329742,
+            0.006757376715540886,
+            -0.010257272981107235,
+            -0.00020856392802670598,
+            0.0006718551740050316,
+            -0.008408842608332634,
+            0.013885954394936562,
+            0.005996035411953926,
+            -0.008340663276612759,
+            -0.0005032995832152665,
+            0.027817361056804657,
+            0.05051366239786148,
+            -0.018863080069422722,
+            0.0009270560112781823,
+            0.01452229917049408,
+            -0.0028824908658862114,
+            -0.0021590273827314377,
+            -0.011976920068264008,
+            0.005821797996759415,
+            -0.019423669204115868,
+            0.013317788951098919,
+            0.006473293993622065,
+            -0.008833073079586029,
+            0.003202557098120451,
+            -0.016711628064513206,
+            0.002621134975925088,
+            -0.0018058179412037134,
+            -0.018681267276406288,
+            -0.015529844909906387,
+            -0.012060250155627728,
+            -0.024181105196475983,
+            0.0004235197266098112,
+            -0.0008546149474568665,
+            -0.013870803639292717,
+            0.006579351611435413,
+            0.008431569673120975,
+            0.032483890652656555,
+            0.0011912527261301875,
+            0.010128489695489407,
+            -2.1513333194889128e-05,
+            -0.0010416359873488545,
+            -0.005988460034132004,
+            -0.01523439958691597,
+            0.00015387803432531655,
+            -0.002342734020203352,
+            0.022590242326259613,
+            0.005549078807234764,
+            0.017560089007019997,
+            0.0017887729918584228,
+            -0.016514664515852928,
+            -0.00596573343500495,
+            -0.007408872712403536,
+            -0.018060073256492615,
+            0.0062649669125676155,
+            -0.005473323632031679,
+            0.009605777449905872,
+            -0.015863168984651566,
+            -0.02731737680733204,
+            0.00792400911450386,
+            0.012234487570822239,
+            0.028514310717582703,
+            -0.0035623949952423573,
+            0.001716805505566299,
+            -0.005533927585929632,
+            0.016408607363700867,
+            0.032180868089199066,
+            -0.0008550884085707366,
+            0.006950552575290203,
+            -0.019999410957098007,
+            0.0017404790269210935,
+            0.013704141601920128,
+            -0.008401267230510712,
+            -0.01821158453822136,
+            0.003939277958124876,
+            -0.020635755732655525,
+            0.004583198111504316,
+            -0.020393338054418564,
+            0.030408194288611412,
+            -0.0023522034753113985,
+            0.008105821907520294,
+            0.004310478921979666,
+            0.01760554127395153,
+            -0.0031646795105189085,
+            0.010545143857598305,
+            0.013916256837546825,
+            0.014908651821315289,
+            0.0009980766335502267,
+            -0.011189064010977745,
+            -0.03790797293186188,
+            0.001859793672338128,
+            0.00817400123924017,
+            0.018635815009474754,
+            -0.01980244740843773,
+            0.011681473813951015,
+            -0.01296931505203247,
+            -0.020363036543130875,
+            -0.01003758329898119,
+            0.011575416661798954,
+            -0.006139970384538174,
+            0.03378688171505928,
+            -0.005874827038496733,
+            -0.008204303681850433,
+            0.005136212334036827,
+            0.010454237461090088,
+            0.0066778333857655525,
+            0.014704111963510513,
+            0.0326354019343853,
+            0.010196669027209282,
+            0.014173824340105057,
+            -0.004859705455601215,
+            0.02399929240345955,
+            -0.003931702114641666,
+            0.003954428713768721,
+            0.00509454682469368,
+            -0.022832660004496574,
+            -0.0040983641520142555,
+            -0.003931702114641666,
+            -0.016711628064513206,
+            -0.007458113599568605,
+            0.0032366469968110323,
+            0.008802770636975765,
+            -0.014045040123164654,
+            -0.00649980828166008,
+            -0.01350717805325985,
+            0.01218903437256813,
+            0.002730980049818754,
+            0.018302490934729576,
+            -0.029802151024341583,
+            0.009825468063354492,
+            -0.017484333366155624,
+            -0.017044952139258385,
+            0.006511171814054251,
+            -0.0038332203403115273,
+            0.006855858489871025,
+            -0.018272189423441887,
+            -0.01318900566548109,
+            0.012817803770303726,
+            0.0007897494360804558,
+            0.013060221448540688,
+            -0.009825468063354492,
+            0.0009706153068691492,
+            0.006064215209335089,
+            -0.009833043441176414,
+            -0.010961798019707203,
+            -0.007977037690579891,
+            -0.009681533090770245,
+            -0.008848223835229874,
+            0.005621046293526888,
+            0.01656011864542961,
+            -0.013742019422352314,
+            -0.003685497445985675,
+            0.016135888174176216,
+            0.006416477728635073,
+            -0.008166425861418247,
+            0.004075637552887201,
+            -0.003793448908254504,
+            -9.948806837201118e-05,
+            0.0013740125577896833,
+            0.012439027428627014,
+            -0.0017632056260481477,
+            0.024665940552949905,
+            -0.02654467150568962,
+            0.007598260883241892,
+            -0.0054051438346505165,
+            0.005446809343993664,
+            0.018226735293865204,
+            -0.02366596832871437,
+            -0.013082947582006454,
+            0.006276329979300499,
+            -0.01966608688235283,
+            -0.008060368709266186,
+            0.0054051438346505165,
+            -0.00016405766655225307,
+            -0.019923655316233635,
+            -0.012923861853778362,
+            -0.034786853939294815,
+            -0.02309022843837738,
+            0.024665940552949905,
+            0.008401267230510712,
+            -0.0073672072030603886,
+            0.018060073256492615,
+            -0.018438849598169327,
+            0.0172722190618515,
+            -0.002225313102826476,
+            -0.006613441277295351,
+            0.003935490269213915,
+            0.007674016058444977,
+            0.025362888351082802,
+            0.002653330797329545,
+            0.008264907635748386,
+            -0.0042082094587385654,
+            -0.014931377954781055,
+            -0.006321783177554607,
+            0.01980244740843773,
+            0.02471139281988144,
+            0.018605511635541916,
+            -0.018575210124254227,
+            0.010742107406258583,
+            -0.011749654076993465,
+            0.01818128302693367,
+            -0.006537686102092266,
+            0.00039037675014697015,
+            0.01859036087989807,
+            0.007969462312757969,
+            0.025241680443286896,
+            0.019923655316233635,
+            0.006852070800960064,
+            0.014128371141850948,
+            0.011908739805221558,
+            0.001204509986564517,
+            -0.012832955457270145,
+            -0.01154511421918869,
+            -0.02840825356543064,
+            0.042847223579883575,
+            0.0025434857234358788,
+            0.008317936211824417,
+            -0.027635548263788223,
+            -0.00021389048197306693,
+            -0.0034942151978611946,
+            -0.018832778558135033,
+            0.005939218681305647,
+            0.03260510042309761,
+            0.03284751623868942,
+            0.021438762545585632,
+            -2.888172457460314e-05,
+            0.0012234487803652883,
+            0.0329081192612648,
+            -0.013673839159309864,
+            -0.00937851145863533,
+            -0.01034060399979353,
+            -0.018393397331237793,
+            -0.006852070800960064,
+            -0.0032423287630081177,
+            -0.01031030248850584,
+            -0.0061172437854111195,
+            -0.009272454306483269,
+            0.0238629337400198,
+            0.0001374249259242788,
+            -0.0022594030015170574,
+            0.006605865899473429,
+            -0.020878173410892487,
+            0.011537538841366768,
+            -0.02813553437590599,
+            -0.013431422412395477,
+            -0.008651260286569595,
+            -0.006950552575290203,
+            -0.018802475184202194,
+            0.008643684908747673,
+            -0.006942977197468281,
+            0.021438762545585632,
+            0.002914686920121312,
+            -0.023590214550495148,
+            0.007530081085860729,
+            0.015317730605602264,
+            -0.0003825644962489605,
+            0.014037464745342731,
+            -0.003890036838129163,
+            0.009060339070856571,
+            0.0045112306252121925,
+            0.011681473813951015,
+            0.016954045742750168,
+            0.010651201009750366,
+            0.009030036628246307,
+            0.003437398700043559,
+            -0.001167579204775393,
+            0.01878732442855835,
+            -0.0007163614500313997,
+            -0.03405960276722908,
+            2.9236827685963362e-05,
+            -0.014628356322646141,
+            0.029499130323529243,
+            -0.009795165620744228,
+            0.009514871053397655,
+            0.008105821907520294,
+            0.007098275702446699,
+            -0.0067119235172867775,
+            0.007916433736681938,
+            -0.008742166683077812,
+            0.0052763596177101135,
+            0.015832867473363876,
+            -0.0042877523228526115,
+            -0.0014980619307607412,
+            0.007840678095817566,
+            0.00407184986397624,
+            -0.005454384721815586,
+            -0.019711541011929512,
+            -0.012287517078220844,
+            0.00726114958524704,
+            -0.02065090648829937,
+            -0.004367295652627945,
+            0.01161329448223114,
+            -0.005454384721815586,
+            -0.009620928205549717,
+            0.01393140759319067,
+            0.02136300690472126,
+            -0.002382505452260375,
+            0.022893264889717102,
+            -0.01349202636629343,
+            -0.009833043441176414,
+            -0.007984613068401814,
+            -0.004374871030449867,
+            -0.041120000183582306,
+            0.025984082370996475,
+            0.01068150345236063,
+            -0.006871009711176157,
+            0.01895398646593094,
+            0.0022499337792396545,
+            -0.002954458352178335,
+            -0.0011041341349482536,
+            0.00010499217023607343,
+            -0.03281721472740173,
+            0.03660498186945915,
+            -0.007007369305938482,
+            0.0005449649761430919,
+            0.010272424668073654,
+            0.042029064148664474,
+            -0.004590773954987526,
+            -0.005859675817191601,
+            0.01225721463561058,
+            0.005336964037269354,
+            0.031938452273607254,
+            -0.010416359640657902,
+            -0.014340486377477646,
+            -0.01815097965300083,
+            -0.004833190701901913,
+            -0.01939336769282818,
+            -0.0038313264958560467,
+            0.014037464745342731,
+            -0.019650936126708984,
+            -0.011242092587053776,
+            -0.001287840772420168,
+            -0.012537509202957153,
+            -0.0010747789638116956,
+            0.008598231710493565,
+            0.017832808196544647,
+            -0.011348150670528412,
+            -0.004450626205652952,
+            -2.2904154320713133e-05,
+            -0.005003640428185463,
+            0.0185146052390337,
+            0.0020056227222085,
+            -0.01702980138361454,
+            0.01321930717676878,
+            -0.00526499655097723,
+            0.006749800872057676,
+            0.0074429623782634735,
+            0.007939159870147705,
+            0.05145302787423134,
+            0.010317877866327763,
+            -0.001339922659099102,
+            -0.01607528328895569,
+            0.023196285590529442,
+            -0.01749948412179947,
+            -0.014181400649249554,
+            -0.02119634486734867,
+            -0.0029923359397798777,
+            -0.007060397882014513,
+            -0.028014326468110085,
+            -9.137987944995984e-05,
+            0.0016221113037317991,
+            -0.0024980325251817703,
+            -0.010045158676803112,
+            0.0005208179936744273,
+            -0.011529963463544846,
+            -0.006954340264201164,
+            -0.008545203134417534,
+            -0.016044981777668,
+            0.02240842953324318,
+            -0.021817538887262344,
+            -0.008855799213051796,
+            0.011514811776578426,
+            0.03118089959025383,
+            -0.005011215806007385,
+            -0.014173824340105057,
+            -0.019817598164081573,
+            0.0025643182452768087,
+            -0.01763584464788437,
+            0.012181458994746208,
+            -0.01225721463561058,
+            0.015045011416077614,
+            -0.03142331540584564,
+            -0.006079366430640221,
+            -0.008992158807814121,
+            0.012840530835092068,
+            0.019817598164081573,
+            -0.0025093958247452974,
+            0.00606800289824605,
+            -0.010295150801539421,
+            0.030211230739951134,
+            -0.00012286569108255208,
+            -0.018605511635541916,
+            0.00873459130525589,
+            0.007977037690579891,
+            0.012537509202957153,
+            -0.004435475450009108,
+            0.00257757562212646,
+            0.011567841283977032,
+            0.012742049060761929,
+            -0.01634800247848034,
+            -0.003174148965626955,
+            -0.017969166859984398,
+            0.005996035411953926,
+            0.006999793462455273,
+            0.021711481735110283,
+            -0.0013550736475735903,
+            0.018272189423441887,
+            -0.01563590206205845,
+            0.00019412307301536202,
+            -0.011749654076993465,
+            -0.0082270298153162,
+            -0.0028237802907824516,
+            -0.003958216868340969,
+            0.006685409229248762,
+            -0.028211290016770363,
+            -0.01274962443858385,
+            0.005287723150104284,
+            -0.004992276895791292,
+            -0.03405960276722908,
+            0.0053710537031292915,
+            -0.02725677192211151,
+            0.013128400780260563,
+            -0.002397656673565507,
+            -0.017847958952188492,
+            -0.00030846628942526877,
+            0.026332557201385498,
+            -0.003677921835333109,
+            -0.012537509202957153,
+            0.015544996596872807,
+            -0.006757376715540886,
+            -0.02119634486734867,
+            -0.005420294590294361,
+            0.014658658765256405,
+            -0.0010719380807131529,
+            -0.012393574230372906,
+            -0.029635490849614143,
+            0.012992041185498238,
+            0.0026874206960201263,
+            -0.007386146113276482,
+            0.010893618687987328,
+            0.025953780859708786,
+            0.008151275105774403,
+            -0.005924067925661802,
+            0.0015757111832499504,
+            0.0049581872299313545,
+            0.00300938100554049,
+            0.020287280902266502,
+            0.02728707529604435,
+            -0.008014915511012077,
+            -0.0011145505122840405,
+            -0.014954105019569397,
+            0.015802564099431038,
+            0.0013020449550822377,
+            0.010507266037166119,
+            0.021893294528126717,
+            0.007045246660709381,
+            -0.005738467443734407,
+            0.003905187826603651,
+            -0.023135682567954063,
+            0.01420412678271532,
+            -0.012272365391254425,
+            0.000622140767518431,
+            0.008749742060899734,
+            -0.007283876184374094,
+            0.01437078882008791,
+            -0.0015245762187987566,
+            0.0339989997446537,
+            0.025211378931999207,
+            0.0043218424543738365,
+            0.0007362472242675722,
+            -0.028559764847159386,
+            0.003547244006767869,
+            0.008651260286569595,
+            0.014499573037028313,
+            -0.0076664406806230545,
+            0.006261179223656654,
+            -0.004355932120233774,
+            0.00959820207208395,
+            -0.0065490491688251495,
+            -0.015893470495939255,
+            0.006113456096500158,
+            0.0012689019786193967,
+            0.006795254070311785,
+            0.0023408401757478714,
+            -0.008863375522196293,
+            -0.024362917989492416,
+            -0.025984082370996475,
+            0.021984200924634933,
+            -0.03918066248297691,
+            0.0073293293826282024,
+            0.011234517209231853,
+            0.03324144333600998,
+            -0.008158850483596325,
+            -0.004980913829058409,
+            0.006276329979300499,
+            -0.016711628064513206,
+            -0.004147605039179325,
+            0.008204303681850433,
+            0.012385998852550983,
+            -0.002420383272692561,
+            0.00974971242249012,
+            0.01318900566548109,
+            -0.012378423474729061,
+            0.03660498186945915,
+            0.016514664515852928,
+            0.01467380952090025,
+            -0.004007457755506039,
+            0.01312082540243864,
+            0.0066172294318675995,
+            0.019044892862439156,
+            0.025165924802422523,
+            -0.017454031854867935,
+            -0.002365460619330406,
+            -0.02292356640100479,
+            0.024165954440832138,
+            0.00726114958524704,
+            0.004556683823466301,
+            -0.008310360834002495,
+            0.014870774000883102,
+            0.0017954016802832484,
+            0.0028199926018714905,
+            -0.006696772295981646,
+            -0.010173942893743515,
+            0.019241856411099434,
+            0.0034317169338464737,
+            0.014870774000883102,
+            4.8501173296244815e-05,
+            -0.005503625608980656,
+            0.006052851676940918,
+            0.014832896180450916,
+            -0.023590214550495148,
+            -0.00203592493198812,
+            -0.019484274089336395,
+            -0.03202936053276062,
+            0.009022461250424385,
+            0.0016135887708514929,
+            -0.007052822504192591,
+            0.03230207785964012,
+            -0.010083036497235298,
+            -0.019787294790148735,
+            -0.008067944087088108,
+            -0.013113250024616718,
+            -0.010469388216733932,
+            0.0008271536789834499,
+            0.003908975515514612,
+            -0.00012428610352799296,
+            -0.009317907504737377,
+            -0.005749830510467291,
+            0.01990850456058979,
+            -0.0030358952935785055,
+            -0.00753386877477169,
+            0.00868156272917986
+          ],
+          "embedding": [
+            0.0040707582550429606,
+            -0.010230625151989194,
+            -0.021520242101903186,
+            -0.010115756270640508,
+            0.026738734586159035,
+            0.01796370233223267,
+            -0.012245199029994153,
+            0.06506924494147726,
+            -0.0014087587222033442,
+            0.009300068719271429,
+            0.003787396973047584,
+            0.002818772639754174,
+            0.022129077062176306,
+            -0.021526709808445683,
+            -0.0034388135367225824,
+            0.02763761747879405,
+            0.013210013805395276,
+            0.025255872672296602,
+            0.00029011480909986703,
+            -0.05438974762290585,
+            0.016671640187490865,
+            -0.016426252259893328,
+            -0.056591180651012554,
+            0.009335648425523075,
+            0.006141624459418827,
+            0.02568943956694641,
+            0.019377132054042116,
+            0.017946464337078746,
+            -0.057184776985927686,
+            0.03208527139096687,
+            0.030132392063755547,
+            -0.003846032371791839,
+            -0.008376524427531415,
+            -0.0010513963058713435,
+            -0.02734868388724371,
+            -0.014879400370545479,
+            0.04383920959748825,
+            0.02563613659516289,
+            -0.020117467074438997,
+            0.0398016699155716,
+            0.029896497884197775,
+            -0.0004195844530067313,
+            -0.023302720788253625,
+            -0.001168337916577517,
+            0.009149021369661628,
+            -0.02508145721900992,
+            -0.004307594138480351,
+            0.00550940415564303,
+            0.014016998918718476,
+            -0.028607379616884317,
+            -0.0018317656587526954,
+            -0.01973464940415756,
+            0.03762738392056255,
+            0.015358106058583114,
+            -0.03931907682104936,
+            0.011519637298832444,
+            0.004482080558226775,
+            0.0747867063383242,
+            0.05550679018263077,
+            0.023243129601931176,
+            0.02671631686665238,
+            -0.009686146566573536,
+            -0.04088582189725495,
+            -0.0065344623594924085,
+            0.019617295999325712,
+            -0.004722087402575777,
+            -0.002871844114437583,
+            0.003935087265300519,
+            0.03229890413393201,
+            -0.02813577047904649,
+            -0.05124172132597234,
+            0.03325235407841968,
+            -0.008687655557217192,
+            -0.011027174838246948,
+            -0.003653553063547375,
+            -0.02350737945182988,
+            0.016223087939183468,
+            0.03789252356462714,
+            0.0034350623378603083,
+            0.00195158657888739,
+            0.011789712862232108,
+            -0.0004435439660769551,
+            -0.0037330456918125894,
+            0.0031552713243866858,
+            0.04465432992298031,
+            -0.0032465421571915712,
+            -0.0075366743861922745,
+            -0.059958752731410256,
+            0.016784616861008125,
+            -0.028734506065772967,
+            -0.03035575736273943,
+            0.02680390892611741,
+            0.02338368719038086,
+            -0.0030658375437784277,
+            -0.03754896050794851,
+            0.07005659523248504,
+            0.0020711575553570695,
+            0.013034425083076923,
+            -0.02316832262132332,
+            0.021332142030732894,
+            -0.0318756233020585,
+            0.02270782509507388,
+            0.01570810309454529,
+            0.004788124787612736,
+            0.026727250200142456,
+            -0.022380142778058276,
+            -0.008593910379436904,
+            0.014439754382387034,
+            0.015007697617383174,
+            0.07662531963612977,
+            0.011195998137723795,
+            -0.010809283538470098,
+            -0.042227822120219305,
+            0.003824320933275175,
+            -0.03797629427607094,
+            -0.02347390684453348,
+            -0.0526876440280276,
+            -0.017799861680287913,
+            0.01701482796266917,
+            0.002730395144917155,
+            0.03789451759899825,
+            0.038338545350313946,
+            0.028463789278406586,
+            0.015494583705011921,
+            -0.05700237266350479,
+            0.047895170359075166,
+            -0.0064406894534929406,
+            -0.0007954721295065988,
+            0.012543029839729358,
+            -0.004703334868741216,
+            -0.0060569679284068,
+            0.012324901254711192,
+            -0.019883935809784068,
+            -0.04253565942154215,
+            0.02164409276211115,
+            -0.0075367555415808635,
+            0.03617159147722157,
+            0.07567745988686005,
+            -0.04225977020036783,
+            -0.022548837589622003,
+            0.029811602145850144,
+            -0.02175792196819533,
+            0.02069861458115363,
+            -0.00941331829664397,
+            -0.035479639885943264,
+            0.020352552580119105,
+            0.003678588588550461,
+            0.06776105274336729,
+            -0.02346047721661687,
+            0.02721075076349443,
+            -0.029386781707644253,
+            0.023530831921117993,
+            -0.02077660533100372,
+            0.020022475323372906,
+            0.024195118676513715,
+            -0.028378518720494666,
+            -0.012900969023913307,
+            0.019264089970719624,
+            -0.0374983624629265,
+            -0.019726171557384355,
+            0.00979636085642416,
+            -0.014479528435832063,
+            -0.014533384583777662,
+            0.005904623758767224,
+            0.010344475500263744,
+            0.05659646358026115,
+            0.04163866113482093,
+            0.007152494819169405,
+            -0.07584500130974763,
+            0.05471858597878099,
+            0.03580211013427615,
+            0.044161429444475306,
+            0.02258180566046248,
+            0.00980779221334715,
+            0.0011193505603397395,
+            0.018086566768892507,
+            0.015605066702191871,
+            -0.00954727399396313,
+            -0.027950079146919375,
+            0.03028514025485407,
+            -0.006351669761257431,
+            0.005095804801686436,
+            0.009620040085743162,
+            0.019748001227987134,
+            0.033709535059510916,
+            0.008203559956940196,
+            0.007389752602218563,
+            0.04197771878351742,
+            -0.02067363758471765,
+            0.007543739188471913,
+            0.011207941261799052,
+            0.013964827251172813,
+            0.03839562075248879,
+            0.0009775244543292637,
+            0.006469507611492216,
+            -0.022808235395746654,
+            -0.024795004398633292,
+            0.006249024864814684,
+            0.011292877109066925,
+            0.01596839619826819,
+            -0.0014105396434174883,
+            -0.04152075438277094,
+            0.008267788411079337,
+            -0.028834191212891387,
+            0.030940526405027906,
+            -0.0018271056434014425,
+            0.006185475047820603,
+            -0.007920574674727123,
+            0.012907560753043616,
+            -0.015165704767187252,
+            -0.021962244496450727,
+            0.0037071053072833147,
+            -0.013508376064336042,
+            -0.015178823540513864,
+            -0.02518557053061311,
+            0.011693256916760944,
+            -0.01716015674592331,
+            0.02048897513447166,
+            -0.003412078966361745,
+            -0.001807298511378529,
+            -0.0012575655363779904,
+            0.03116103961636643,
+            0.011668533361641842,
+            -0.011637010548030312,
+            -0.040260195687925236,
+            -0.011129376691522074,
+            -0.0034493180842871908,
+            -0.02098244789711689,
+            0.026658743116016938,
+            -0.01923736325953796,
+            -0.009723937181300192,
+            -0.013134679989565051,
+            -0.011974548914818263,
+            -0.004028712301052159,
+            0.019628329175867303,
+            -0.028192034966932684,
+            -0.012638520793012082,
+            -0.011568652932013111,
+            -0.0025919910519445737,
+            0.012497908463405111,
+            0.015113186712897145,
+            0.014880607940212641,
+            -0.03059227859452334,
+            -0.021899716217105084,
+            -0.0008513506358489523,
+            0.011985931118464679,
+            -0.01321584595585435,
+            -0.011735288366213981,
+            0.010710683541435907,
+            0.01961722172525993,
+            -0.023571946726192585,
+            -0.03364281880125354,
+            0.0014460076743756465,
+            0.016563167828467187,
+            -0.02891589059260944,
+            0.04595166331401113,
+            0.034271553243501465,
+            -0.027557971577291153,
+            -0.007138821520412525,
+            -0.039508407807651844,
+            0.06814329636830853,
+            0.00022136123564675344,
+            -0.039718016705753735,
+            0.0503457594963753,
+            -0.003580581891576583,
+            0.013820796398521455,
+            0.02054050419088757,
+            0.035682661890720495,
+            -0.0013220739026852943,
+            -0.0052759394697328675,
+            0.04733018719162478,
+            -0.0187845855131196,
+            -0.03982699468608443,
+            -0.023905460465070062,
+            0.002631233133039351,
+            -0.03090995554989513,
+            -0.012617612051488022,
+            -0.013605787969473513,
+            0.039672783793764745,
+            0.008669582242097126,
+            0.009974277556736362,
+            0.005535636914843583,
+            0.014180668493093584,
+            0.012042659947850441,
+            0.011870115226306776,
+            0.00451259454348581,
+            -0.014023048592193627,
+            -0.007769033996392623,
+            -0.05715987527668358,
+            0.003374564518384728,
+            -0.028801158523506985,
+            -0.00756331369170698,
+            -0.02163054894055572,
+            0.005367181573233089,
+            0.02308705505078254,
+            -0.005438909398653553,
+            -0.00873620909426313,
+            -0.0036530652237719077,
+            0.013920407174296543,
+            -0.01539613379172909,
+            -0.03908211213269472,
+            -0.016810726548844992,
+            0.007460173854386443,
+            -0.007069189426742819,
+            -0.041197222498539215,
+            -0.003843233633335066,
+            0.004373367480223065,
+            0.016194121660459908,
+            -0.029165150066068787,
+            -0.00735139702658342,
+            0.017257895082337448,
+            0.014153313280536095,
+            -0.002376178042741926,
+            -0.023223700836427403,
+            -0.0319877537205969,
+            0.02088060774052704,
+            0.0012829290529515294,
+            0.03251550040316258,
+            -0.04128453069131847,
+            -0.023376114412063578,
+            0.04083022735076889,
+            0.005690491480259003,
+            -0.05810403800689653,
+            0.016582434816685452,
+            0.010402038964417569,
+            0.016795644054313566,
+            -0.00908590043098552,
+            0.05003656634510042,
+            -0.037216197954986735,
+            -0.013027960261454162,
+            -0.007090943677204207,
+            -0.05175371787508879,
+            -0.0020065660448063322,
+            -0.014908489552053589,
+            0.010627472758306387,
+            -0.03275473161158777,
+            -0.017334608434133866,
+            0.025321047439048318,
+            -0.0033515195870701753,
+            0.011526068468647207,
+            -0.026783879816504975,
+            -0.0012830935419413407,
+            -0.017506908439998053,
+            -0.02114853058440477,
+            0.004878884893669383,
+            -0.020936900837115968,
+            -0.0025587027781733642,
+            0.01557850246191957,
+            0.023652335794299834,
+            0.011489428313303988,
+            0.0019089137782552768,
+            0.02054907941022243,
+            0.014361232983780595,
+            -0.0056413109308506085,
+            -0.0010300050480036684,
+            -0.03413262505887213,
+            -0.018821160485010483,
+            0.03361989907065425,
+            0.013125889701529858,
+            -0.003742167496379789,
+            0.008813100942466033,
+            -0.0383265293638935,
+            -0.035316957263713485,
+            0.03020499323039515,
+            -0.05001623971243151,
+            -0.00855479667673597,
+            -0.005939674940109814,
+            -0.026451831943555135,
+            -0.008032873049527808,
+            0.0009431571685654594,
+            -0.014387605128858954,
+            0.016382559541755014,
+            -0.024909510232038502,
+            0.003715736505444385,
+            -0.006016712902409688,
+            -0.009110197692342108,
+            0.05507263160354303,
+            -0.022103031032598353,
+            -0.0475415595313825,
+            0.021958434235402655,
+            -0.016670809659931243,
+            0.0323519363801999,
+            0.03867494951405658,
+            0.011124508964363016,
+            -0.008002445714431074,
+            0.034041838454818577,
+            0.018400123235267234,
+            -0.01161754789471199,
+            -0.025901844576206774,
+            0.021800298132870245,
+            0.00943488425697727,
+            -0.05906691123008531,
+            -0.00782549274387971,
+            -0.019357446956797447,
+            0.008682376344965976,
+            0.02863077694455766,
+            0.0024925977373894447,
+            -0.002682170968419412,
+            -0.013590235283325244,
+            -0.021511149507762284,
+            -0.03255453916518719,
+            0.009039307015853905,
+            0.008305286476500875,
+            0.009312625027668458,
+            -0.0014144165766983945,
+            -0.007853869334376857,
+            0.009067065760942163,
+            0.022485201611182105,
+            0.03966189080765722,
+            -0.041627289519629344,
+            0.002707086456239785,
+            -0.022678178050044222,
+            -0.03653753142622053,
+            0.0036574646778835588,
+            0.029996160786576367,
+            0.02901163765559891,
+            -0.01971777193183787,
+            -0.0036589934841109884,
+            0.009181234938554578,
+            -0.025979483002227535,
+            -0.00773645103995767,
+            0.004280770859737866,
+            0.04116668301017644,
+            0.006201085381597264,
+            0.01402728258112097,
+            0.001060407317110527,
+            -0.00920801784000087,
+            0.02109880889040528,
+            0.025211022083982727,
+            -0.029866678264942617,
+            0.014439702646820226,
+            0.009379206296908577,
+            0.04156435812161122,
+            -0.037309655469525846,
+            -0.011819731348299495,
+            -0.02766498020409985,
+            -0.010454143230794476,
+            0.027911532296566614,
+            -0.04127890742215716,
+            -0.006948536183086887,
+            0.010207784113909938,
+            -0.0016584043356136647,
+            0.016437495428919636,
+            -0.014607861750686952,
+            -0.03222311919805017,
+            -0.000610723203118213,
+            -0.02867445198803213,
+            -0.024772816792038133,
+            -0.016510309753766612,
+            -0.018581201775667577,
+            -0.07450654873867177,
+            0.00026832248829154255,
+            -0.06724696990816072,
+            -0.02447002541264633,
+            0.04885569249258513,
+            -0.010195604207134053,
+            0.03332905935704749,
+            -0.02815272112271314,
+            -0.005296340304911926,
+            -0.05908835042526019,
+            0.0038476172058970774,
+            0.025753576863447866,
+            0.02386003857017594,
+            0.005206447371391696,
+            0.024799057871126032,
+            0.012060738268265606,
+            -0.04146503002980051,
+            -0.024949907556862643,
+            -0.014503475223770958,
+            -0.0016340466065744286,
+            -0.0034009167013795906,
+            -0.031298432995411005,
+            0.036241272149880994,
+            0.008902644391348814,
+            0.03550223922667804,
+            0.04463301409716232,
+            -0.010385984413759852,
+            -0.03305004479734889,
+            -0.006952739338820577,
+            -0.0175981588092108,
+            0.042668533202572856,
+            -0.017112184620086308,
+            0.005149173014526686,
+            -0.03219589107997635,
+            0.024181808150698054,
+            0.0014339790422945406,
+            0.01254721190611376,
+            0.011099458649404119,
+            0.0027626836767364253,
+            -0.026883640446090158,
+            0.0024160770766463607,
+            0.023874514847956222,
+            -0.022558433617669373,
+            0.00917667715473093,
+            -0.006754091166465801,
+            -0.033686935718776144,
+            -0.01078389638118685,
+            -0.013595327486167114,
+            0.024712781935748277,
+            0.013168464807451798,
+            0.005875864604862434,
+            -0.02417814420558463,
+            -0.0023978800459714144,
+            0.028650463366555304,
+            0.020176980768301288,
+            -0.02118184201861616,
+            0.019082340279671355,
+            0.019734925740100277,
+            -0.04209645071102001,
+            -0.04217310001316092,
+            0.006553913719098956,
+            -0.003735719227468833,
+            0.0061793909220459,
+            -0.028285889794908054,
+            0.008181788076484304,
+            0.004432488437404309,
+            -0.01743905248466261,
+            0.041221269621866216,
+            -0.014892354419172107,
+            0.005087102208335201,
+            -0.022198383716186407,
+            -0.011936388003605221,
+            -0.00027487042024290816,
+            0.044599215801158845,
+            0.008068975171827777,
+            0.003496950921841073,
+            0.02238779276447619,
+            0.0054839808262270315,
+            0.04816723616939378,
+            -0.02685221260466647,
+            -0.0190356946337995,
+            0.016464136459560113,
+            -0.0176092780782677,
+            -0.012513274095432353,
+            -0.010695421445334002,
+            -0.004784348908457072,
+            -0.022203041334932165,
+            0.010788155380415583,
+            -0.018353897272137543,
+            -0.02277431189907027,
+            -0.003945465850796918,
+            0.016024639792926797,
+            -0.006248368166794934,
+            -0.0038769803892794875,
+            0.0016292556540434277,
+            0.02412864830338344,
+            0.02892955453819876,
+            -0.02408022470555968,
+            0.019190282372105023,
+            0.018941584034503536,
+            -0.006709516380602965,
+            -0.020956377933747532,
+            -0.01964885206488053,
+            0.01434623424964373,
+            -0.01352968586212152,
+            -0.013872556892707697,
+            0.004052798542613835,
+            0.009201147374299997,
+            -0.004086067985859617,
+            -0.04185829685685201,
+            -0.014668698123513235,
+            0.003179403563664022,
+            -0.0006415014901450319,
+            0.019360184344240947,
+            -0.0150817538567364,
+            -0.029771999278536083,
+            0.014221776097046751,
+            0.022944884879589678,
+            -0.04493425220030206,
+            0.02117577719533236,
+            -0.0030648751497261983,
+            -0.00016251533643103604,
+            -3.817296018601414e-05,
+            0.008727767382077138,
+            0.026914779443266063,
+            0.03456565094655899,
+            0.01974806772696219,
+            0.00450800007152441,
+            -0.008952599700759673,
+            0.025184190461155245,
+            0.009846044083064704,
+            -0.025168735183134738,
+            0.010983353565382029,
+            0.016740721743326522,
+            0.0070838930012654296,
+            0.009173879407277282,
+            -0.006078288608766152,
+            -0.018269347893097643,
+            0.013181494188066667,
+            -0.01435693747022231,
+            0.01678907949532146,
+            0.05154463552784974,
+            -0.00039632596424461964,
+            -0.02461439550772063,
+            0.008418147037164186,
+            -0.006544319842335593,
+            -0.010065129286193278,
+            0.02214628550460128,
+            0.0029515574751370417,
+            -0.030862152033847717,
+            0.007509556025223583,
+            0.00675485304374519,
+            0.01894936231559203,
+            -0.02355525295639599,
+            -0.01387885858544803,
+            -0.00431398020087544,
+            0.0021208056468763523,
+            0.01580766504149989,
+            -0.004739477121073512,
+            -0.013204823447960706,
+            0.02994098229517376,
+            0.025496728234662373,
+            -0.00026124946738213036,
+            0.015060064980768175,
+            0.022262345898533903,
+            0.009968408664790368,
+            0.007588431141088797,
+            -0.0083063413609779,
+            -0.031336272171937095,
+            0.0018957858383561667,
+            0.05800811417728381,
+            -0.014937085296865802,
+            0.04313875432235027,
+            0.02018476425923236,
+            0.015657644644987107,
+            0.017880263053253607,
+            -0.01662268082870562,
+            -0.008740053682875323,
+            -0.0017630394204179638,
+            -0.008294417990024647,
+            -0.027613464270562948,
+            -0.005953209556351086,
+            -0.02379961210881074,
+            -0.013878676554701543,
+            0.015418315245979704,
+            -0.014764728065382074,
+            0.0027420640870858457,
+            0.010395538988699663,
+            -0.006590633833835581,
+            -0.00841537209927409,
+            -0.003943772700797381,
+            0.03376338017637296,
+            0.011980354721458463,
+            0.027761929822019422,
+            -0.00011128539412582973,
+            0.009342430952107655,
+            -0.018419526564396063,
+            -0.006839752805318664,
+            0.03271190887864607,
+            0.006011520427944665,
+            0.004652835471939905,
+            0.01485920920301398,
+            0.014305626381398871,
+            0.038015677219487023,
+            -0.005102005401565164,
+            -0.018204460895989052,
+            0.02065495133597456,
+            -0.0049699013356374775,
+            0.019149036693562712,
+            0.008825981577388056,
+            -0.002660055848523568,
+            -0.004292379561953203,
+            0.03090270710860331,
+            -0.020243440437217503,
+            0.018735727616321387,
+            -0.02238236089688497,
+            -0.0276584413929402,
+            0.023028408906176946,
+            0.008320436733620496,
+            0.04047641118026783,
+            0.011463851066078041,
+            -0.02247143742166039,
+            -0.034588725413285185,
+            0.009827713930789184,
+            0.00337071450636392,
+            -0.024446280199879062,
+            0.017542777160283324,
+            0.00780836483490212,
+            0.008453886667283516,
+            0.016624116261219637,
+            -0.004175682824655864,
+            -0.0042784362933758835,
+            -0.001546656717896957,
+            -0.008819739495525477,
+            0.01406790007187259,
+            -0.015487658289541583,
+            -0.021457182907140035,
+            0.024687169583245905,
+            -0.03915911962859764,
+            0.01771290139419629,
+            -0.006510046907173026,
+            -0.00997450880725535,
+            -0.02059560682587115,
+            1.5939836949198004e-05,
+            -0.007159493752862143,
+            -0.0004782363814453761,
+            -0.017503181957042913,
+            0.016643971600145645,
+            0.024056442566869802,
+            -0.013209211443909658,
+            -0.00445521990927108,
+            0.0020313626821877526,
+            -0.029235692455820056,
+            0.006662209097255912,
+            0.016993027053110635,
+            0.00529940612845993,
+            0.0026104743851393716,
+            0.0001707934714983891,
+            -0.031289446611163674,
+            -0.005782208620284939,
+            0.03053516889069683,
+            0.05590704747438521,
+            0.021628254147289437,
+            0.02403314981042378,
+            0.02908606663381421,
+            0.02717206823892633,
+            -0.003198844932040613,
+            0.0016217508103442348,
+            -0.01620660606431983,
+            0.027098323143010743,
+            -0.011664168290676943,
+            0.04322175021589083,
+            0.008652323184522541,
+            -0.049493020736445056,
+            0.004537958157696794,
+            0.00943239454147692,
+            0.034918603888916544,
+            0.016678716235690318,
+            0.0005887414054803805,
+            -0.005894338664998101,
+            0.03279165663336323,
+            -0.02735726709032382,
+            -0.011962070182991011,
+            -0.00036014407560325284,
+            -0.019079003730607717,
+            -0.015938295169860884,
+            0.010357611943935459,
+            0.04278924678845588,
+            0.0009957483598272451,
+            -0.0071422790468850215,
+            0.01571201177743999,
+            -0.009445307674383,
+            0.02286338492075306,
+            0.007336440384050498,
+            0.0042809497342459904,
+            0.025581467690095977,
+            -0.0025007871391868105,
+            0.02300539823997411,
+            -0.000628898771317254,
+            0.004582863909134464,
+            -0.015960578501883316,
+            -0.00930649396114688,
+            0.027838455754803444,
+            -0.011112268986721,
+            -0.029253746504396393,
+            -0.008734785561831333,
+            0.00472579547270089,
+            -0.0003805825712986138,
+            0.0021854358149324227,
+            -0.01183885533661493,
+            -0.010840951201591629,
+            0.019965691788904057,
+            -0.004922943301424112,
+            0.003800048754664761,
+            0.012664049429916252,
+            -0.0030124219674414897,
+            -0.01910844996750822,
+            0.01889983984736845,
+            -0.003858283778519141,
+            -0.024168821338992813,
+            0.020083955711766945,
+            0.01850497889545421,
+            0.045156567155119454,
+            -0.01586824055785609,
+            0.004612757419198795,
+            0.017497774574602505,
+            -0.002172088240279357,
+            -0.019617327081566827,
+            -0.014944795772093997,
+            -0.018212462639041492,
+            -0.015975283672064702,
+            0.012837956594691596,
+            -0.01243789877725244,
+            0.019500571353639314,
+            -0.0380735226388175,
+            0.012590900534826689,
+            0.004832168601136412,
+            0.007505385730121269,
+            0.016011983211212992,
+            0.025994560026860358,
+            0.002157436199771672,
+            -0.033465476168065784,
+            -0.01014599891204166,
+            0.007845373376916211,
+            0.010093432683262838,
+            -0.002849185178237036,
+            -0.018147004183434554,
+            0.003633790611809425,
+            -0.0266147420637404,
+            -0.018644422659309035,
+            0.022926737299761104,
+            -0.04798772015599113,
+            0.017908415301694906,
+            0.024859125158586466,
+            -0.04343699525707025,
+            -0.04886699607556352,
+            -0.011459188432595229,
+            -0.036774381316312464,
+            0.0014128005215813578,
+            -0.0005256067584181517,
+            0.011194992852264102,
+            -0.009342140079829378,
+            -0.018598671981710183,
+            0.01584789414301952,
+            -0.0036412641242200418,
+            -0.004225840239935549,
+            -0.00616696736018045,
+            -0.0025598631463178384,
+            -0.010188803157718543,
+            -0.03244694662411977,
+            -0.03290891136780909,
+            -0.00033742479296272127,
+            -0.009743927051414455,
+            -0.02334288359945272,
+            -0.03447062782872289,
+            -0.008728968107967028,
+            -0.037753659749578956,
+            -0.00515561961442376,
+            0.04070890150714597,
+            0.027517147608335957,
+            0.009449414978133334,
+            0.008086900089929912,
+            -0.013989034604490105,
+            0.05691047488419298,
+            -0.020425883381389086,
+            0.039466620642512155,
+            0.005301493569940605,
+            -0.0037222353899918703,
+            0.013075425214878277,
+            0.027710826422351564,
+            -0.006253665106995653,
+            -0.0017412928073238833,
+            -0.01928096630475438,
+            -0.012920476859996416,
+            0.009444408011467955,
+            0.015799884767184885,
+            -0.0025083061619197524,
+            0.001274457251739847,
+            0.00930533161700852,
+            -0.0006652478540264343,
+            0.017258038491335247,
+            0.03996268817486439,
+            0.003833844612473671,
+            -0.00826404141706295,
+            -0.018822002570963614,
+            -0.003188844900317347,
+            -0.011807006674248829,
+            0.010482058734260632,
+            -0.015410635475145028,
+            -0.011540803927522013,
+            -0.01249428777269578,
+            -0.017081856786754605,
+            -0.002183034976477535,
+            0.003114834030705039,
+            0.0027475411722881747,
+            0.02261627635192778,
+            -0.015872027027080824,
+            -0.022487581027490516,
+            -0.0035823117885461506,
+            -0.02398778449664045,
+            8.13716819067491e-05,
+            -0.01050060782407259,
+            -0.015616181283381218,
+            0.013676480044963827,
+            -0.021513173575512936,
+            -0.01091399089076612,
+            -0.008699176177861256,
+            -0.0006986953850759474,
+            0.005853723710671249,
+            -0.00993355748621621,
+            -0.00010241843582159751,
+            0.01150872694850241,
+            0.005559552266827916,
+            -0.012336449703762907,
+            0.0039431718621504,
+            -0.012740550272925562,
+            -0.004432059174951158,
+            -0.03592309122711106,
+            -0.019665686718735042,
+            -0.01846795637714379,
+            0.005642094448288269,
+            -0.005108964028543505,
+            -0.020586678054151854,
+            0.00668364061048941,
+            0.0034775127050453774,
+            0.004713169526661792,
+            -0.010723667875209928,
+            0.013944735241748791,
+            0.010882420168586074,
+            -0.003518245916227596,
+            0.013827334494230662,
+            -0.001330399936559009,
+            -0.020286697504699764,
+            -0.011139922402071095,
+            0.03538258948179278,
+            -0.024961826464903997,
+            -0.0018019328351763667,
+            -0.0026423695690168066,
+            0.00597003516870726,
+            -0.03071835946501977,
+            -0.013634212232744325,
+            0.02164400349828422,
+            -0.005846552894831441,
+            -0.026367014426434226,
+            0.03399116480234033,
+            0.03473258231629753,
+            0.010041605592929843,
+            -0.013423384340589794,
+            0.04730054254696893,
+            -0.012968978954938308,
+            -0.018274800841400665,
+            0.026826644936918647,
+            0.0278389611094493,
+            0.02525164095926168,
+            -0.029782550447890466,
+            -0.003010713086487275,
+            -0.02393629084357885,
+            0.017993366287762866,
+            -0.0263476663180971,
+            0.01864011499179424,
+            -0.02332263152537849,
+            0.016014908881756034,
+            0.01868464825773604,
+            -0.003104041044182249,
+            0.026313491086684425,
+            -0.0052945248919367815,
+            0.015253731232325074,
+            0.0012371308759955573,
+            -0.02119765517013431,
+            0.007795466385020642,
+            -0.006469500272298062,
+            0.025762169721880456,
+            0.00833992768202466,
+            0.01801901639978976,
+            0.022019940825351116,
+            -0.0409522298213741,
+            0.02003698451081499,
+            -0.001367916864679543,
+            0.01782984895971748,
+            0.0038516134262842373,
+            -0.007885432294782304,
+            0.014221728513400065,
+            0.010991352188016265,
+            -0.012851099261924757,
+            0.003840539873231741,
+            0.0008923482291826014,
+            -0.028928104542716937,
+            0.007066004768771198,
+            -0.026914574065060926,
+            0.025866907400807087,
+            -0.005526774784152086,
+            -0.00998122210956923,
+            -0.020744532673097317,
+            -0.0071425207262770865,
+            0.003754468896273156,
+            -0.012747874997429746,
+            -0.011385805601999295,
+            -0.024700457669046855,
+            -0.02401291382882952,
+            0.029591582802488962,
+            -0.014054332096344003,
+            -0.04015978298687865,
+            0.001234440863385445,
+            -0.023301255402129767,
+            0.01193125007773239,
+            -0.005793124129267758,
+            -0.024440100649464773,
+            -0.042563399082257154,
+            -0.017548888269389457,
+            -0.0009394887961485922,
+            0.022739763644266633,
+            -0.00114156949394707,
+            0.041791395064755515,
+            0.0008816696205437911,
+            0.022207132857533028,
+            -0.015547580543606755,
+            0.005319731264840819,
+            -0.02071229859314312,
+            -0.041495579742912285,
+            0.03660432378985971,
+            -0.014637031447181937,
+            0.025854844497435178,
+            0.03324325296704186,
+            -0.013009347371118843,
+            0.001354504885302625,
+            0.02652570709474851,
+            -0.014618100923440427,
+            -0.010782232747863238,
+            -0.0034936699515378212,
+            0.003461670978569965,
+            -0.006468116285021298,
+            -0.027793221545159466,
+            -0.007752458229193391,
+            0.04193706715422074,
+            0.017894069367616637,
+            0.021928994684423043,
+            0.022606401934291136,
+            -0.017972404868085358,
+            -0.026402048654300563,
+            0.003656646576244765,
+            -0.013129722990153284,
+            -0.018090360146368677,
+            -0.01624598856366761,
+            0.012397150789705746,
+            0.011741002980073786,
+            -0.004052422390675261,
+            -0.01836750423615859,
+            -0.020929335029248472,
+            -0.016327412417122416,
+            -0.0042615345226698015,
+            -0.008458117095535,
+            -0.005002582649405199,
+            -0.012158173174049797,
+            -0.020409616620022143,
+            0.03706368540614453,
+            0.0285565642978937,
+            -0.0041732933884846566,
+            -0.00607154864704767,
+            -0.011142150176334957,
+            0.008834431806990914,
+            0.006021232046087652,
+            -0.005028870471297621,
+            -2.3396043292139426e-05,
+            0.005112033372713154,
+            0.022822312435625314,
+            0.007271235299348001,
+            0.0035791789459533856,
+            -0.06340305811365254,
+            -0.005392000517722054,
+            0.022560201096078862,
+            0.012581401789917604,
+            -0.0032248219865223496,
+            0.025180718735801837,
+            0.0006451724964970927,
+            -0.009714514747463593,
+            -0.018389422219956937,
+            0.024728862150150004,
+            0.008760329201960964,
+            0.020988700992999768,
+            -0.014974763313686516,
+            0.003845297486923296,
+            0.0006445983847138896,
+            0.00750394530065325,
+            -0.0019792992847026793,
+            -0.008578254004439607,
+            0.025776677078006516,
+            0.016593438298233645,
+            -0.005695578040595484,
+            0.01604018433477423,
+            -0.0017497260488899657,
+            -0.021460543443428924,
+            -0.010265213095609554,
+            0.00042017209251926397,
+            -0.010570864288876387,
+            -0.018015790888295944,
+            0.017569414447505878,
+            -0.002601036282535863,
+            0.0072399377790164435,
+            0.030281707151249355,
+            0.0423183067569767,
+            0.009566493398534713,
+            -0.005928603293907002,
+            -0.0037936911251953654,
+            0.015477873304924939,
+            0.01513127995244932,
+            0.010741256034652754,
+            0.03015858543221727,
+            -0.029247492875021843,
+            -0.006690306960497428,
+            0.0046653727410352576,
+            0.004610405633429778,
+            -0.011692179099382171,
+            0.030055897497741527,
+            0.04569730592994177,
+            0.006736813244286893,
+            -0.004799978513117371,
+            0.002121067028052434,
+            0.011299889107780231,
+            0.007768137218140627,
+            -0.018927080714961783,
+            -0.0012222962555201746,
+            -0.006009911026251445,
+            -0.014001766533604134,
+            0.026941585318261085,
+            -0.003960837039351648,
+            -0.011848232223811548,
+            0.02903357274500332,
+            0.1105267435019781,
+            -0.006011029338781496,
+            -0.0053646998281195206,
+            -0.03242321704621517,
+            -0.015194102811332289,
+            -0.008860590919147765,
+            0.0056988205877963076,
+            0.01635957585050017,
+            0.020327661913559888,
+            -0.0008943408274661007,
+            0.016820860249955754,
+            0.02822175614376368,
+            -0.013854827839129885,
+            0.007906244895327178,
+            -0.030891381690403466,
+            0.015702117924362928,
+            -0.005383974926093977,
+            -0.012032955400313954,
+            -0.0026169511828034054,
+            -0.008618307035330928,
+            -0.0037356680273096105,
+            0.0009666430093455632,
+            0.011296307527684261,
+            0.004427516814276546,
+            -0.014484198313369681,
+            0.002829529156309784,
+            -0.008595300985863424,
+            -0.005122996529848575,
+            -0.02393571073523279,
+            -0.024119245543788293,
+            0.005223147047205755,
+            0.0019389305548629142,
+            6.344308292376579e-05,
+            0.006848965054539454,
+            -0.003810712095343454,
+            0.030149081317333418,
+            -0.01700322008272963,
+            0.0012288912810468446,
+            0.01193226302726573,
+            0.012332749933413344,
+            0.01688602002694633,
+            0.0028600984823761366,
+            -1.9882692552530084e-06,
+            0.021172977047420227,
+            -0.010514036509402561,
+            0.01460334896316005,
+            0.013091417303145243,
+            -0.04103572381458182,
+            0.0089746401463863,
+            -0.0027841289484449873,
+            -0.024252447218794344,
+            0.004446472209246425,
+            -0.018987867343500102,
+            -0.00881974688349601,
+            0.0005365625696166774,
+            0.012054719659248512,
+            0.01960300421041832,
+            -0.008553943699846696,
+            0.02111201386480852,
+            -0.0003501657740808554,
+            0.004177088753111459,
+            -0.012879944670340026,
+            0.0065222126975051685,
+            -0.007563354540402343,
+            0.0008168128607271199,
+            -0.018213241937597355,
+            0.012526561612175514,
+            -0.0008285725179894544,
+            -0.015737003159436623,
+            -0.011337688275309921,
+            0.004737346137976519,
+            -0.0027993545140114705,
+            0.0030843163114700313,
+            0.010153740340626154,
+            -0.008815302484351509,
+            0.004919275910399053,
+            -0.0017541042538165297,
+            0.002562816687714991,
+            0.014285249186421434,
+            -0.029816964593980453,
+            0.005357227738099325,
+            0.009154842495811638,
+            -0.00048782303639333943,
+            -0.013074976152495576,
+            -0.004054505559072866,
+            0.0028322102449485015,
+            -0.016877328079971877,
+            -0.012128199639779175,
+            0.023421330164419888,
+            -0.012565145848820333,
+            0.008609070188894315,
+            -0.010936598722317274,
+            -0.018401801734378692,
+            -0.01066119413133089,
+            -0.005254229780560808,
+            0.01234969508270852,
+            -0.0038427767204741877,
+            -0.006947594635952202,
+            -0.01756894543880097,
+            -0.007890897373987378,
+            -0.047246698568222466,
+            0.009617876521097624,
+            0.027157555263862723,
+            -0.011355758309667893,
+            -0.04050311381480088,
+            -0.01017774338301804,
+            0.02437257208967103,
+            -0.004056329295095018,
+            -0.008345450168215274,
+            -0.001599171409135998,
+            0.018358386463243014,
+            0.0027207445181251406,
+            -0.007292967749349572,
+            -0.011267861428599221,
+            0.01657507035043879,
+            0.015105061280816214,
+            0.021419854579631276,
+            -0.02088952706825817,
+            0.006295984081511286,
+            0.020973020776936996,
+            -0.03073246543555915,
+            0.0023720435876790876,
+            0.03363966216709148,
+            0.010412588613998134,
+            0.013023311559810034,
+            -0.0010217662933572685,
+            -0.0075567381078977215,
+            -0.008248818794295818,
+            0.011848200323549865,
+            0.005066463524567732,
+            -0.023174489297821385,
+            0.021823825355858556,
+            0.020368868756448753,
+            0.0026042856858509524,
+            0.003457067317891926,
+            0.0005539745115410357,
+            0.01062670844108589,
+            0.009948889803890108,
+            0.05100741556350918,
+            -0.006493714124814164,
+            -0.014021751055930108,
+            0.015063988102064609,
+            0.02488059237697814,
+            -0.0023499941119596228,
+            0.01087282522970853,
+            -0.02612218935931224,
+            -0.021423880914461062,
+            0.023389294741183817,
+            0.008967763132667892,
+            0.009637806238448432,
+            -0.010815706635624337,
+            -0.011390999462441815,
+            0.031750675158698435,
+            -0.01307829462211914,
+            0.00759749911213318,
+            -0.02169895957209634,
+            0.01387879409130081,
+            -0.008606362268863441,
+            0.032229658009209815,
+            -0.0009066257436946125,
+            0.011062436424260079,
+            0.028010486866468023,
+            -0.007835805879241447,
+            -0.019657486763625547,
+            0.003967480939987674,
+            -0.015577952617624695,
+            0.02188429826317828,
+            -0.020502758155233298,
+            -0.0002362500702658591,
+            0.0040935406015014265,
+            -0.010056462958039452,
+            -0.0057038083589231615,
+            0.011475170331848666,
+            0.0015345076181482267,
+            0.006359262977587827,
+            -0.020293014401106,
+            0.01369706440475074,
+            -0.003989482215141903,
+            0.007887824197823247,
+            -0.015925992045349828,
+            0.014370223626942766,
+            -0.02309139937510304,
+            -0.021808736033141787,
+            0.008142621179624918,
+            0.0108996933426044,
+            -0.007589050511458233,
+            -0.025971539510030145,
+            0.02250992080022228,
+            -0.024980833805349545,
+            0.024979008145354243,
+            -0.006622339940663538,
+            0.002566378488088469,
+            0.004913874740363343,
+            -0.008102103508732738,
+            -0.0028098905427027783,
+            0.0057865587191235565,
+            -0.0073509382400023995,
+            -0.014123756810537343,
+            -0.019453380788786282,
+            -0.04083629135586275,
+            -0.03177927047231389,
+            0.022927581786088044,
+            0.011557232599801421,
+            0.0017073832658476193,
+            0.02339552660545296,
+            -0.02247471449131386,
+            -0.017890866287500053,
+            -0.009775068417716798,
+            0.025184899522229628,
+            -0.04408541669037361,
+            0.012104255392615549,
+            -0.014669458760022245,
+            -0.0245258242481677,
+            0.006274252098230249,
+            -0.004161960147481513,
+            0.009299171724026455,
+            -0.010779421730862708,
+            -0.04713627671791054,
+            -0.005192684784377356,
+            0.02330975631183273,
+            -0.0036802876364168935,
+            -0.006197548344287773,
+            0.03048284650641829,
+            -0.0019503797069087845,
+            -0.005038254331399784,
+            -0.005524875459245342,
+            0.027783596017224226,
+            -0.024950083604372984,
+            0.010988398531346805,
+            0.008304937912856357,
+            -0.008485065895193593,
+            0.006572665697403556,
+            0.0011518846966102158,
+            0.011927348472282199,
+            0.015632588644277183,
+            -0.015559413716826264,
+            0.02262990861928957,
+            0.011446850286896198,
+            0.006506075777765303,
+            0.008015225053785824,
+            0.00010054430757843335,
+            0.0022975742216194894,
+            -0.024948925449668525,
+            0.026047924967851216,
+            0.01888211557921096,
+            0.0001632181017876392,
+            0.012553094476616919,
+            -0.015042621225242777,
+            -0.02782009986290425,
+            0.02606646297992026,
+            -0.016847653820092335,
+            -0.00045272379465939274,
+            0.020053091486344633,
+            -0.016210036358455773,
+            -0.007930550661303182,
+            -0.01123514849773852,
+            -0.0162675150307978,
+            0.01803076559921804,
+            -0.012160217543572069,
+            -0.02216594536057653,
+            0.002594675636753233,
+            0.016443123828746582,
+            0.02134480656850119,
+            0.026694575953313482,
+            -0.011470930697351533,
+            0.017768927048191052,
+            0.0008653408391607816,
+            0.001865765599795092,
+            0.025745954171649543,
+            0.01702677011968492,
+            -0.008474889109168202,
+            0.025347176686722313,
+            -0.0016686223698600755,
+            -0.002421921060670558,
+            -0.012265504084322075,
+            -0.0165214436458303,
+            -0.007545471138114208,
+            -0.015333235556146196,
+            -0.0051830106590765685,
+            0.000762746782664525,
+            -0.0018118022696726346,
+            0.10876576838038368,
+            0.0034948044540582626,
+            -0.011608460689239234,
+            -0.005870263500417206,
+            0.007510774308250375,
+            0.0023525201199670765,
+            -0.006846154764389575,
+            0.016082721103476375,
+            -0.018616626204086575,
+            -0.023593744079394,
+            0.004743481974389998,
+            -0.0009262286319469741,
+            0.00870122878944883,
+            0.02029523251993269,
+            -0.008608837818169835,
+            -0.0200567711498164,
+            0.007089477722191928,
+            0.005535149644041221,
+            0.0023640746516071774,
+            0.005256431824732927,
+            -0.0006118671321824933,
+            0.008209920240361308,
+            0.02448422122961942,
+            -0.015622892878860465,
+            -0.029770836607612903,
+            0.010535371601298532,
+            -0.00015994156470228762,
+            -0.007193425789067776,
+            -0.011046316463797128,
+            0.014728841492009627,
+            -0.0011488451990980268,
+            -0.02751404133172324,
+            0.017663132712536528,
+            -0.019055335178900415,
+            0.014643246200933506,
+            0.0025348345163347464,
+            -0.00016387173410118788,
+            0.022565953069005714,
+            0.013101158251834834,
+            -0.04272085076882881,
+            -0.0034442020309914823,
+            0.01041205537746844,
+            0.011059422927946552,
+            0.0116446016862942,
+            0.0012992535381387887,
+            -0.018670801241345514,
+            -0.011788984490283379,
+            -0.009109849103208542,
+            -0.021268488414296117,
+            -0.012523033785938503,
+            -0.011393853408628214,
+            0.010743646386439633,
+            0.0012964030814325153,
+            0.023809566099779113,
+            -0.003030535542277875,
+            -0.006526351867141901,
+            -0.000430699447948508,
+            0.00880561288237257,
+            0.009694646009259077,
+            -0.02079011138439714,
+            0.008294338719512953,
+            -0.01823922789220214,
+            0.011819769685519291,
+            0.006416044253693268,
+            -0.022799568832306067,
+            -0.009911619276012527,
+            0.007152786695893951,
+            0.008393677195358034,
+            -0.013569953300860946,
+            -0.006503988731872938,
+            0.02299498754558268,
+            0.018870619152710703,
+            0.012233964644738905,
+            0.024994616203196145,
+            0.03838969722235787,
+            0.005744255912479976,
+            0.027682123067526515,
+            -0.016791471092091604,
+            0.008459745383680297,
+            -0.022493868174622695,
+            0.012902262638928338,
+            -0.00460399519266689,
+            -0.0050431343566427886,
+            -0.018188192498411616,
+            0.033765105245412236,
+            -0.009159751849801773,
+            -0.0027150644587327763,
+            0.034778224828120714,
+            -0.0066502021793471075,
+            0.004525230911930458,
+            0.005096748531775903,
+            -0.016067049494073893,
+            0.033274672789154185,
+            0.01686831040872425,
+            0.016353966592947262,
+            -0.021631941883929817,
+            0.011748317746828848,
+            -0.005675701689448899,
+            0.0001273442591116645,
+            -0.020738819037390263,
+            -0.0037528323797385733,
+            0.0027725541343763104,
+            0.0059860509282538546,
+            -0.016556134518370437,
+            0.02033623476757896,
+            0.00019647023567641008,
+            0.0052895744489259875,
+            0.03253462247073863,
+            0.01043311189804924,
+            0.019182938878855655,
+            0.010051051068017807,
+            0.019523739618278902,
+            0.0281939391326836,
+            0.025685787482623124,
+            0.0029932688462254147,
+            -0.018668945726139946,
+            -0.0009326901784812825,
+            -0.007880352863625262,
+            0.010492063055849727,
+            0.022849902214818293,
+            -0.022619228461640748,
+            -0.02772644874135741,
+            0.018566981184526675,
+            -0.0022608185345048414,
+            0.020031103834122595,
+            0.007544464625538984,
+            -0.020935181884939796,
+            0.003407355515103286,
+            -0.023178230166337178,
+            -0.0005418155871620593,
+            -0.031039187720144967,
+            -0.028591174077243604,
+            -0.0007659241904525392,
+            0.011945344534674635,
+            -0.01688153017778311,
+            0.03357788163306365,
+            0.017667778671101954,
+            -0.014289109343082812,
+            0.01604684528817524,
+            -0.011054587325539086,
+            -0.0036959091138078683,
+            -0.005819853870004682,
+            -0.00203755010992417,
+            -0.02157944474979592,
+            0.005138614763345542,
+            -0.0033509138150859593,
+            -0.0024871060745482883,
+            -0.016936575573598154,
+            -0.021901830366872595,
+            0.016375173843894617,
+            0.016742770709641697,
+            -0.006606252542838758,
+            0.039771905404651496,
+            -0.004238071340064403,
+            0.00018364315349545657,
+            0.011690521707403059,
+            0.0010018392078563795,
+            -0.013651921068215563,
+            -0.010829994512513501,
+            0.012362647379788833,
+            0.011321403447882543,
+            -0.018942043512252663,
+            0.005763773608775982,
+            -0.009131565038297889,
+            0.002089216671850269,
+            0.005438887787329238,
+            -0.008183835275629478,
+            0.006690280329947951,
+            -0.01673229763262922,
+            0.015833397876364723,
+            0.00295474085300386,
+            0.003186554859191664,
+            -0.011797859525926772,
+            -0.020991927615680524,
+            0.01576413532776261,
+            0.0032403514518527287,
+            0.018819824905201635,
+            -0.026755218746060662,
+            -0.00012946467986739473,
+            -0.009552681500884924,
+            -0.0028941357661144826,
+            -0.012533800273548772,
+            0.013975640628890588,
+            0.016864739319953755,
+            0.0063941926578529,
+            0.01781016436903586,
+            -0.0026268224264609724,
+            0.015368284773773678,
+            -0.018122416427266218,
+            0.008686009180697955,
+            -0.006801074032721299,
+            -0.018537521789181048,
+            0.0013237338264105415,
+            0.027863741654435434,
+            -0.004670012662973029,
+            0.011066853115455966,
+            0.011704884649534624,
+            0.018954056680143452,
+            -0.0012626215734593707,
+            -0.0030850559460101866,
+            0.015610404665171367,
+            -0.005838226405658572,
+            -0.006861883833171456,
+            0.020872966233916948,
+            0.032951653763679145,
+            -0.006716215648910268,
+            -0.0021292599951407863,
+            0.010886596327481677,
+            -0.004901684487088035,
+            -0.008644584974417106,
+            -0.009587037829603014,
+            -0.03158137900717491,
+            0.024369089446706894,
+            0.012092412439477474,
+            -0.009601042951586109,
+            0.022690862146548786,
+            0.020752792552278052,
+            0.011412743968296708,
+            -0.010471860708195726,
+            0.02291536038589634,
+            -0.01268035142112569,
+            0.024168869865226136,
+            0.032482831775271157,
+            0.004400973556358811,
+            0.0022312906286559167,
+            0.0021175489661843985,
+            0.01691348460546639,
+            0.006050357346231155,
+            -0.007068133214675996,
+            0.011656003476409986,
+            -0.018487923180009552,
+            0.01598530891381001,
+            -0.004432366596808033,
+            0.02808371007118948,
+            -0.01337145216499331,
+            -0.010473700033368562,
+            -0.00537325376270318,
+            0.00010726431372328729,
+            0.006445701087575359,
+            -0.003324002570256215,
+            0.005255443653813165,
+            0.0007072577303252132,
+            -0.017825649146944237,
+            0.003050128130393781,
+            0.004825967170196209,
+            0.009018799597337101,
+            0.004103653497834134,
+            0.007249957401824204,
+            -0.012360355929483,
+            0.01441885664789707,
+            0.004401820363035808,
+            -0.0026190938717927086,
+            -0.0007163225760398784,
+            0.011124538295057815,
+            -0.013232822471919533,
+            -0.004895624967029031,
+            0.005281228035321008,
+            0.0059275753353475414,
+            -0.007507924998741734,
+            -0.004484140989405051,
+            0.011908076549392969,
+            -0.0037917291321648483,
+            0.006351543504726163,
+            -0.004409464890606014,
+            -0.0037760571637986827,
+            0.005303661313434895,
+            -0.00974511544836468,
+            -0.004204167898342997,
+            -0.0007093125445559502,
+            -0.006869250680233214,
+            0.00786202375969793,
+            0.005971033227117194,
+            0.010606085272785863,
+            0.006969437776542303,
+            0.004293129058563288,
+            0.004114452739123398,
+            0.004071017374828283,
+            -0.002445681623750424,
+            -0.00592480769363354,
+            -0.004617501328052899,
+            -0.012067458701076778,
+            -0.008335239153829813,
+            0.009025028998949398,
+            0.008653779421856954,
+            -0.019363018792788574,
+            0.02048011578221327,
+            -0.001188320960771659,
+            0.00772165044232104,
+            -0.01602086976290928,
+            -0.000907553574811279,
+            0.005464880629734941,
+            0.008267702703609528,
+            0.016136762227127184,
+            -0.03654209526492064,
+            0.005179791231187316,
+            -0.0037321883493524766,
+            0.016026013709955444,
+            -0.008215703411373182,
+            -0.006979816286354862,
+            0.02564060526656963,
+            -0.01450763388985563,
+            -0.0009847758228920171,
+            -0.016142304341196283,
+            -0.01407778890808498,
+            0.010815725252959506,
+            0.02050940280233426,
+            0.020684808835848383,
+            -0.0004965802897137407,
+            -0.0005781445948759235,
+            0.00036179841583004193,
+            0.022775804864342608,
+            0.015545681427566235,
+            0.018697154644607316,
+            -0.004543500782944068,
+            0.007028436664826706,
+            -0.023057215501463973,
+            0.009912581556496658,
+            0.0011795417129232596,
+            -0.008215027850908477,
+            -0.008554137076267468,
+            -0.008703378564520125,
+            -0.010448538943182258,
+            -0.011773811327190638,
+            0.004106616839960258,
+            0.013895013678116032,
+            0.004190549543250594,
+            0.0031571190600981074,
+            -0.0225385788570314,
+            -0.006243193914542516,
+            -0.011229641377928767,
+            -0.01674214043453308,
+            -0.007367389908398082,
+            -0.02386334969150212,
+            -0.028021339684039734,
+            0.0002794207370355175,
+            -0.02411861258809571,
+            -0.015958336406924612,
+            0.005268815864773021,
+            -0.01546610225139129,
+            0.00898600137695074,
+            -0.008396481690803911,
+            -0.011873486990732417,
+            0.0074743033679134105,
+            0.015336444788692,
+            -0.017676977079590463,
+            -0.010165317702640217,
+            -0.01690524990506033,
+            -0.013460451917691695,
+            0.010918550283871637,
+            0.00595960977280955,
+            0.009250157926438213,
+            -0.012447979337850305,
+            0.0062932435120863365,
+            -0.012673905639312696,
+            0.015073013552721701,
+            -0.03769166067890086,
+            -0.029992249463405965,
+            -0.00014990754540999537,
+            0.007288036630677436,
+            -0.0020735742042505426,
+            -0.018695120921334806,
+            0.0003860116505754868,
+            -0.003608995261360232,
+            -0.014900191335144973,
+            -0.013866216738266176,
+            0.025251475305524053,
+            -0.006788483107616133,
+            0.01223227424986397,
+            0.0034937277040015626,
+            0.03438333560343413,
+            0.008803852645562129,
+            0.014666206816302367,
+            -0.005709421955160623,
+            0.003742433643393657,
+            -0.0033706258426092744,
+            0.006906464031352315,
+            0.0076578888497945185,
+            0.009276592191710524,
+            0.008025321004305927,
+            0.004576470294889512,
+            0.002887346543823031,
+            -0.001925468214331082,
+            0.02125241545487307,
+            0.025671028960818562,
+            -0.00037467906648160044,
+            0.008981063192139038,
+            0.0009387900084090577,
+            0.0002735789978680719,
+            -0.008374290340790827,
+            0.006547013284767913,
+            -0.008274637814401893,
+            0.004041066530254948,
+            0.012897549078838394,
+            -0.009150863940242148,
+            -0.010211568689323908,
+            0.016575088843292566,
+            -0.012380099999574783,
+            -0.03420455578618604,
+            0.026550654680719987,
+            0.005324512466403649,
+            -0.0035670106106754245,
+            -0.010657928468872608,
+            0.0037176027853158167,
+            -0.01103891334887315,
+            0.008096915671982816,
+            0.022587659771132487,
+            -0.0013527917944322358,
+            -0.0018085171207746743,
+            0.0038503794877980586,
+            -0.011870484733648887,
+            -0.007881489342351292,
+            0.006462772034381968,
+            0.0315696514747912,
+            0.01326417653779577,
+            -0.020400369539140532,
+            -0.015223808971473513,
+            0.020406862349006738,
+            -0.023142176712111575,
+            -0.006254959806032159,
+            0.009039506093607052,
+            0.0009087520492922078,
+            -0.022766046838415165,
+            0.0009662037093312435,
+            0.004701966191595815,
+            0.005946458676746661,
+            -0.006358986495019584,
+            0.03237532341275415,
+            0.001075992257265679,
+            0.015892020271402903,
+            -0.017189453434945788,
+            0.0017514632185789299,
+            -0.014189278671121677,
+            -0.014265447388465805,
+            -0.029809460132405256,
+            0.0005557491671071693,
+            -0.0030324198417091704,
+            0.011038984948790744,
+            0.024808067315720335,
+            0.01145103533194245,
+            0.014432246996007246,
+            0.010552852947578788,
+            0.01513689173494211,
+            -0.013653722180565311,
+            -0.029009757850432732,
+            -0.005171522291984226,
+            -0.018001048743500243,
+            -0.02297482628212251,
+            0.003962613844063587,
+            0.013645125285548414,
+            0.006278662690346082,
+            0.023786381386575035,
+            0.0021738516691483163,
+            0.0009580591750940011,
+            -2.690805459518147e-05,
+            0.008894091070322921,
+            -0.004952203441302471,
+            -0.006553986197076102,
+            0.003570189040872711,
+            -0.000652714137223501,
+            0.021228360881461853,
+            -0.005758946428212041,
+            7.897153237591187e-05,
+            0.014242175492137011,
+            -0.005622256348934008,
+            -0.0035009192529791146,
+            -0.017339926143652377,
+            0.006248842389703505,
+            -0.01622011586976866,
+            0.025783086900871728,
+            0.018450783573486916,
+            -0.009109295293546267,
+            -0.013469521573468915,
+            -0.011259555201313116,
+            0.02773097452736661,
+            0.004631282554715908,
+            0.010787055220617197,
+            0.018228322680582464,
+            0.01461254259249465,
+            0.01784830193964943,
+            0.004664725683522323,
+            -0.0034339781146649384,
+            -0.0007692550982545399,
+            0.021176646860433974,
+            -0.014700178590249292,
+            -0.002701208084390198,
+            -0.008842788249773623,
+            0.0021799005046816444,
+            -0.008431299000066363,
+            0.006086447848362669,
+            -0.010793016137040172,
+            -0.015251381840165954,
+            0.0189503457652375,
+            -0.02660767353735015,
+            0.016423753600066336,
+            -0.00593016170680532,
+            -0.00482891500342298,
+            -0.027459214027897378,
+            0.010728039528784636,
+            0.0027866316191608297,
+            0.0022519974195000687,
+            0.010770472550461752,
+            -0.021427634123401408,
+            -0.01867717541189363,
+            0.008532444554010673,
+            -0.011457117350255678,
+            -0.007152301670244725,
+            -0.00348619742388509,
+            0.0154715607296319,
+            -0.00017898955187849443,
+            -0.009925365923862246,
+            0.02085093731060963,
+            -0.007175309371250024,
+            0.01223979370120031,
+            0.005256127464980655,
+            -0.008756077893734508,
+            -0.0028438320920811362,
+            0.021022708898898203,
+            0.027557382862284808,
+            -0.008342206702778956,
+            0.014526066580760486,
+            -0.011883797998224506,
+            -0.01088335352319397,
+            -0.03479444009382274,
+            0.030104838505937447,
+            -0.013305732257226255,
+            0.026773747582411633,
+            -0.0006968265979630097,
+            -0.017037729111584094,
+            -0.016206207453228583,
+            0.01625225876515654,
+            -0.0024290584023212254,
+            -0.009366287391027166,
+            0.01465885301870365,
+            -0.02428628495473643,
+            0.0058336571592899735,
+            -0.0006627935593670669,
+            -0.011531762838191602,
+            0.013275508677482622,
+            0.0019223252930710324,
+            0.010738622785461513,
+            -0.02030405805119001,
+            0.0009661595529028194,
+            0.009263716373456188,
+            -0.016791855691574455,
+            0.006057830565306032,
+            -0.03730952786847912,
+            0.0029341721588188943,
+            -0.014833895684348656,
+            0.005725615593914644,
+            0.001498755253963699,
+            -0.01820708352937343,
+            -0.03652692232407375,
+            0.00043665701245599497,
+            0.01913883585802903,
+            -0.007975988799790905,
+            0.020538595326175857,
+            0.008243569241768276,
+            0.008421404577931517,
+            0.002314738067443065,
+            0.0016702403734608475,
+            0.008279538988821465,
+            0.03679456861213399,
+            -0.01775927634148114,
+            -0.004836606118958108,
+            -0.0012151329859941935,
+            0.010644195708319258,
+            -0.02028470571116305,
+            -0.009287515726906219,
+            -0.0036506912311261536,
+            -0.013134467570201307,
+            0.030615862647585222,
+            -0.020265555441523482,
+            0.025634449009756734,
+            -0.009300770899023007,
+            0.009274072905190077,
+            -0.02097685009125915,
+            -0.009797008690754956,
+            -0.010661540100131183,
+            0.0019532291898233895,
+            0.013465005282849381,
+            -0.009151430285942593,
+            0.0071467934925125145,
+            -0.006655608635025676,
+            0.01020327561163374,
+            -0.001472597695707201,
+            -0.02443508800166349,
+            -0.0024998811551575563,
+            -0.01107321258077183,
+            0.02456164421154801,
+            -0.0019619696941303388,
+            0.0005043179699158694,
+            0.010951580163871948,
+            -0.00853593956432799,
+            0.023927336608555987,
+            -0.008855702280766362,
+            0.0010093679746665581,
+            -0.004863269000020324,
+            -0.004005691415117686,
+            0.001698212044518236,
+            0.01822024022650613,
+            -0.005707404715849356,
+            0.03722587302228139,
+            -0.01708925706528118,
+            -0.0065500772743641925,
+            -0.00015538686795361824,
+            0.010852925330001283,
+            0.010035359503147549,
+            0.008754719437760327,
+            0.004067820167375348,
+            -0.014861417626433965,
+            0.009346669946626484,
+            -0.020136029447262147,
+            0.007501669351583119,
+            0.023178490169347605,
+            -0.006840359541885465,
+            0.0006676450965357345,
+            -0.01283320739092204,
+            -0.015610538827742276,
+            0.006308543164022682,
+            0.002329420005586303,
+            -0.002295739637504597,
+            -0.02181189272979859,
+            0.006298804075737111,
+            -0.014398470126723037,
+            0.004007419942728446,
+            -0.0007497656993578918,
+            0.013863867106289847,
+            0.016451166273826513,
+            -0.006740209707090658,
+            0.039292260347522444,
+            0.01871723939316731,
+            0.009109005310505462,
+            0.005674782595856756,
+            -0.040588444759736254,
+            -0.008993850998154852,
+            -0.0010197185016393919,
+            0.004866150984083949,
+            -0.01141354216871663,
+            -0.018944881712855702,
+            -0.002646110151132781,
+            0.000308585036997592,
+            -0.010681954707313127,
+            -0.009091585604191968,
+            0.01751599002710486,
+            -0.006034424764302979,
+            0.00872057856926618,
+            0.004146961781109887,
+            0.023758065635807138,
+            -0.009727802672285544,
+            0.01176257119851645,
+            -0.0071822083765114165,
+            -0.005456197751568526,
+            0.0006726765703478756,
+            0.011011628907288813,
+            -0.008031792262886802,
+            0.009763340867550022,
+            -0.004279891138185881,
+            0.008355720233269893,
+            -0.008552596384067825,
+            -0.023952129458400563,
+            -0.014972827087541732,
+            0.014347535404251018,
+            0.011651260496709961,
+            0.008137738462704715,
+            -0.013624435569539285,
+            0.0031728721674035352,
+            0.00675830277744976,
+            -0.009184784787581942,
+            0.009161771311910335,
+            0.017733082848622708,
+            -0.001411542568472906,
+            -0.0017830167499636113,
+            -0.0016130003349297027,
+            -0.012173528776899308,
+            0.0011196437753239162,
+            0.02284887358189995,
+            -0.003054479643937998,
+            0.014614283913976716,
+            0.010689486748036806,
+            0.009643292262102645,
+            -0.00810930837603355,
+            0.001915955511174678,
+            0.02341863048264366,
+            0.015022291783105098,
+            -0.004625715884342598,
+            -0.011422973894623295,
+            0.004259757717597968,
+            -0.010332744704689279,
+            0.007373328473086509,
+            -0.02117940113144938,
+            -0.004672698999328004,
+            0.008896311380763048,
+            -0.007028862917603513,
+            -0.01732432285099198,
+            -0.028026173555055017,
+            -0.016765805892242715,
+            0.019819329380181,
+            0.0027213760678977965,
+            -0.013291251876048186,
+            -0.0032919084721662363,
+            0.0031214883337300926,
+            0.002553005168048115,
+            -0.0071104806975682885,
+            0.005310583052470149,
+            0.007359183453835734,
+            -0.008674133980486269,
+            0.017665958608281126,
+            -0.000521269431423141,
+            -0.0008020251454616109,
+            -0.0005442374261144659,
+            0.0006127687598711624,
+            -0.016387752806126442,
+            0.01666161912459431,
+            -0.010949228376419892,
+            0.015198110119816624,
+            -0.008966726991404666,
+            0.008600499511690635,
+            -0.007617043351740469,
+            0.009458641139070436,
+            0.011597957400445055,
+            0.012755037944648426,
+            -0.004403680820063604,
+            0.008519241473418196,
+            0.004807816191718704,
+            -0.008147125240985166,
+            0.007017601015398136,
+            0.01296082929962557,
+            0.00582353932160707,
+            0.0013634237970372698,
+            -0.0008246894322793264,
+            0.0022035670958705576,
+            -0.002480976219285683,
+            0.0056293125704658135,
+            0.012714850523393719,
+            -0.002497639449055006,
+            -0.007135683201188999,
+            0.014988249771676699,
+            0.014955895630183352,
+            -0.022994948923833993,
+            0.007121634434082677,
+            0.005835971825639792,
+            -0.0030507120821557008,
+            0.002145266953387315,
+            -0.027334525318320587,
+            -0.020359474070022213,
+            0.01901995970350808,
+            -0.015260454474605018,
+            0.02850460517419317,
+            -0.011624138792780823,
+            -0.02284242513728605,
+            -0.01277489645734181,
+            -0.004209529633534193,
+            -0.022684500849916897,
+            -0.004894414098943742,
+            0.01919165657366256,
+            0.013401560972810082,
+            -0.004313549020265677,
+            -0.019089369171102138,
+            0.03863248408555474,
+            0.010497951840980463,
+            0.03508157919687289,
+            -0.008597970944278045,
+            0.0023354831464448313,
+            -0.01563401808449416,
+            0.015551176593436512,
+            0.001075882802133782,
+            -0.011575692834796734,
+            0.01505716108583101,
+            0.031836226915668404,
+            0.0034740889237207494,
+            -0.00617206855371193,
+            0.0018420207148235309,
+            0.00418433325800744,
+            -0.0008167766650900381,
+            0.011160989178055499,
+            -0.018497472318034093,
+            -0.011951262348399922,
+            0.0013093376768780122,
+            0.00216347903176653,
+            0.0059233460419514045,
+            -0.001778091863744448,
+            -0.013317639704511549,
+            0.011433308892643204,
+            0.011596810919469926,
+            -0.01518143105525228,
+            -0.007497373860338335,
+            0.0050957665991362405,
+            -0.018974405317819656,
+            0.01872472902494418,
+            0.011859522398789263,
+            0.000792413325332956,
+            -0.0006503947129906383,
+            0.00808635755624928,
+            0.003749813548995223,
+            0.0033657799335872368,
+            -0.007862215303617144,
+            -0.0028014669075116876,
+            0.0009892310856576799,
+            3.834379772674841e-06,
+            0.015402211364447726,
+            -0.015264893397161102,
+            0.02488855366696587,
+            -0.0025643639525691083,
+            0.00953993868921819,
+            -0.017460590918773188,
+            -0.0017911712931080155,
+            -0.00402374787529868,
+            0.01838051670627292,
+            0.006753072393532114,
+            -0.010130676216186492,
+            0.004921510943896829,
+            0.007975201081296845,
+            -0.012589786869771252,
+            0.00018319054705558588,
+            -0.0022606180909706688,
+            -0.014791528072476771,
+            -0.0031171076370692706,
+            0.017873825561816544,
+            0.007593688317551089,
+            0.0037375070591255375,
+            -0.015863143964928172,
+            0.007192324815779268,
+            0.007522990667369163,
+            0.024504875417877527,
+            -0.026309816223960274,
+            0.024779042752840836,
+            -0.022104087334088543,
+            0.02165781034613893,
+            0.0319051741666687,
+            0.007538146369110045,
+            -0.01690181186708103,
+            -0.006595947249169521,
+            0.006373991305923002,
+            -0.023410814842488123,
+            -0.005494017994983188,
+            0.016903736810723716,
+            -0.01792400152372258,
+            0.000330865061689199,
+            -0.014660413003063089,
+            0.0040825880154611895,
+            0.008248037865838212,
+            0.011540333451799686,
+            -0.0033578830043738085,
+            0.018327525495676525,
+            0.020171745352159914,
+            -0.01138759854371235,
+            -0.0004221149053107702,
+            -0.022321409200941777,
+            0.005046865873981027,
+            0.022509643273449936,
+            0.001111272240506322,
+            -0.009540467231190513,
+            0.018376261423745193,
+            0.008480532791922674,
+            -0.020801891594934313,
+            0.008436374227753589,
+            -0.036175982655998785,
+            -0.010454118523313122,
+            -0.009328507118928412,
+            -0.001218633166683384,
+            -0.004142773602181115,
+            -0.0002494371580772723,
+            -0.011365548842513265,
+            -7.699548642315061e-05,
+            -0.006889339420877534,
+            -0.0037173070609413474,
+            0.014953978004522174,
+            0.011678957032329096,
+            -0.003501047405385249,
+            -0.0073984932581048915,
+            0.008493128567102267,
+            0.006163937693564917,
+            -0.017034696006318337,
+            -0.013268418502042576,
+            0.0037122128689059085,
+            -0.020067656401141467,
+            0.005148329575092478,
+            0.00459729178556505,
+            0.014031669598917171,
+            -0.01849235806598133,
+            -0.009298127665365717,
+            0.019716512208448042,
+            0.0094466405158553,
+            -0.0008988916054855498,
+            0.004264260983102956,
+            -0.00032312548009733906,
+            0.002279987803645912,
+            -0.0008902345374346296,
+            -6.145528806973222e-06,
+            0.003602229465148649,
+            0.012503366799381277,
+            0.007759393855482671,
+            0.03694402922447051,
+            -0.016692148708641812,
+            0.005873765863350649,
+            -0.014187609702966002,
+            -0.0037592517335595654,
+            -0.00749939010396784,
+            -0.0004308157309481927,
+            -0.0022282234382577843,
+            0.0021999996319642854,
+            0.0052176773733353925,
+            0.009353171029508977,
+            0.002155405440765206,
+            -0.00823565584692709,
+            0.009691447250255694,
+            -0.015228173095533029,
+            0.003853858061067531,
+            -0.008207910291425473,
+            0.01017972550354623,
+            -0.009695280827515048,
+            0.0028705587799331246,
+            0.03849358164875786,
+            0.01130021300937487,
+            -0.004934175853966098,
+            -0.014300147534361459,
+            -0.0007796051618993811,
+            -0.01737877501558196,
+            -0.008579119859817895,
+            -0.005434792488602325,
+            -0.02617338653902575,
+            -0.00838916510137031,
+            0.015344260464413648,
+            0.011202567142310492,
+            0.014124853863170132,
+            -0.023894182185855824,
+            0.008897943489322673,
+            -0.03001670823127986,
+            -0.011951507895140876,
+            -0.001530967140156163,
+            -0.015467812535428609,
+            0.0010110999991167703,
+            0.023623196504289838,
+            -0.007762735863648054,
+            0.012654652822750325,
+            0.012867958170258448,
+            0.018777522196571934,
+            0.006390986057157016,
+            0.00038045736974183937,
+            -0.009444911783760569,
+            0.009534310165417939,
+            0.008806031651319704,
+            0.0080954543713255,
+            0.013161833638467553,
+            -0.0022840529693446553,
+            -0.018060162439023572,
+            0.005469288590000856,
+            0.014354475778235131,
+            -0.0033504276914600585,
+            -0.001615191819057537,
+            -0.011088493180661976,
+            0.016039040565630432,
+            -0.023559046333872156,
+            -0.009482398794557168,
+            0.009918576960319548,
+            -0.0027308309155029842,
+            0.009212063547988036,
+            0.0160181200081126,
+            -0.01881373001297218,
+            -0.0024168618343039703,
+            0.016244889977796682,
+            0.0024104546308670025,
+            -0.00022606425578303413,
+            0.016792244238218245,
+            -0.0007616857360728623,
+            -0.004279120886828465,
+            0.00997764266310123,
+            -0.0075631917138318655,
+            7.3878137053128405e-06,
+            -0.002416364915008295,
+            -0.0028253498168087964,
+            0.02054128727706496,
+            -0.021281801678871736,
+            -0.005379259148621041,
+            0.02257306103533458,
+            0.015687105622685975,
+            -0.007843745111084413,
+            6.258498042855072e-05,
+            -0.0219506172079994,
+            0.0019925886885244454,
+            -0.007768133999661569,
+            0.016524850592507592,
+            -0.007957949830060354,
+            0.0030085044673901665,
+            -0.01361024620290394,
+            0.011563781350927133,
+            -0.01277407169098418,
+            0.00873716215529134,
+            -0.002986943948333218,
+            -0.017567646684567492,
+            -0.0053109390854215125,
+            0.004561236136619266,
+            0.0014045175130801308,
+            0.044958041347040185,
+            -0.005539136232703934,
+            0.003729517522976541,
+            0.018086335927383804,
+            -0.014199453349727933,
+            0.02079719360289355,
+            0.016663293302592568,
+            0.00991445855219406,
+            -0.00033967370741635794,
+            -0.0018726748849858836,
+            0.023373354806639515,
+            0.013258226441642464,
+            0.00861986998150065,
+            -0.0014340218811281975,
+            -0.008516541227326329,
+            0.018239230461345617,
+            0.005458629943667832,
+            -0.012421896028975013,
+            0.012732721349758832,
+            0.013559877132508506,
+            -0.013720939472355347,
+            -0.011545210834395917,
+            0.01616358223872008,
+            -0.019904594319155373,
+            -0.018260016571763644,
+            -0.0036124537591368864,
+            -0.01614510591132948,
+            -0.03864699384205461,
+            0.01619401261888991,
+            0.018351607973242735,
+            0.006666313000123845,
+            0.013647410298896132,
+            0.002597680061427196,
+            0.004545530267244157,
+            -0.018531082430861854,
+            -0.0077618625909995275,
+            -0.008085144524135461,
+            0.003744735923004248,
+            0.011059457364156674,
+            -0.011335432904114519,
+            -0.009514379703581616,
+            -0.00034500702695418285,
+            -0.007433132060989644,
+            0.01971420270994951,
+            0.0020753610290475642,
+            0.0034506706065161024,
+            -0.00628437298309269,
+            0.017966485854173167,
+            0.026789678964152908,
+            -0.007532446825077476,
+            0.021886182869092073,
+            -0.002864230253056434,
+            -0.0015870360120997492,
+            -0.014393519359298496,
+            -0.0014439412842964876,
+            -0.011527182213472351,
+            0.0005453493437517964,
+            -0.0006791481533636673,
+            0.014093767997869645,
+            -0.03288824642147967,
+            -0.003957301296796813,
+            0.014444592778851293,
+            0.007538052312876599,
+            -0.005057755472988024,
+            0.011357919882949903,
+            -0.002964451028327082,
+            0.016977317604413924,
+            0.0075201055190542146,
+            0.023391646079994784,
+            -0.016588823248397513,
+            -0.004504858318770565,
+            0.006660877064954993,
+            0.0034631255931793638,
+            0.0014484203909961972,
+            0.01912983041187014,
+            0.023439941258581887,
+            -0.014715886385460541,
+            -0.0013596123366001458,
+            -0.0029020372473879614,
+            -0.021414221343490594,
+            0.014976806123253727,
+            0.0074729481304182865,
+            0.010967351226114886,
+            0.009806663105020302,
+            0.015465076881663691,
+            -0.02109587527354609,
+            0.0011671121739822202,
+            0.020567458365051382,
+            -0.008618333549105003,
+            -0.018815706906171112,
+            -0.009804829158883123,
+            -0.00966828714719191,
+            -0.006078893678150302,
+            -0.00803783313451146,
+            0.005523023167028105,
+            0.0022647506733877573,
+            0.01652997604669998,
+            -0.008204200434441975,
+            -0.008404609147866811,
+            0.008489247125986078,
+            0.007546785673285454,
+            -0.0033093451283570316,
+            0.010614050963656496,
+            0.0030633179859393587,
+            -0.006940540318011839,
+            0.00894884343338777,
+            -0.002787816848430143,
+            0.014347598920245487,
+            -0.02441769157218451,
+            0.004717742147185283,
+            -0.012096876825337124,
+            0.006405151160675699,
+            0.0041568790527989905,
+            0.0016702066682613428,
+            -0.0041159211459004,
+            0.0035217362856754353,
+            0.0008893479210743541,
+            -0.07433312151926533,
+            -0.012629681414004524,
+            -0.0035797012285864486,
+            0.033657838713146884,
+            -0.019695806809053914,
+            0.004477927056336216,
+            0.011966065644382396,
+            0.0096717896523898,
+            0.003070713232538593,
+            0.006136726101157844,
+            0.006112385569942848,
+            -0.0245258242481677,
+            -0.00015284418977092615,
+            0.004821040694679944,
+            0.0020347004045943454,
+            -0.007532408630122127,
+            0.00750565079228405,
+            0.015327710245879296,
+            0.020989926895388523,
+            -0.010970305087289488,
+            0.017227663358885748,
+            -0.003352718496981928,
+            -0.006068892126250978,
+            0.0014285750322333686,
+            -0.010153799722483554,
+            -0.017307415843726394,
+            -0.006495008086641125,
+            0.0068983699213297075,
+            -0.001982568363667046,
+            0.0044032687939897855,
+            0.003333596495330213,
+            -0.0032171586993948696,
+            0.0072365127663633925,
+            0.013920326463272646,
+            0.006415916101287133,
+            -0.013675562796300214,
+            0.003003782758807665,
+            -0.02500375035312058,
+            0.011945988879567646,
+            -0.012708975181025296,
+            -0.0009488161508767279,
+            -0.0047093877027523014,
+            -0.0035750724825159503,
+            -0.002213435677949806,
+            0.02043567538125188,
+            -0.014304009033474186,
+            -0.01802899775650995,
+            0.008187227768196673,
+            -0.011217013195839762,
+            0.008300740149622635,
+            -0.03382831122769974,
+            0.012386335491333264,
+            -0.0276029934382802,
+            -0.0002243752790856292,
+            -0.004521387639042107,
+            -0.0010631721066129095,
+            -0.009491184157278669,
+            -0.03274559774619223,
+            0.015845705808609598,
+            -0.007204751554079551,
+            -0.020338337341085228,
+            0.0007143424225187802,
+            -0.021267094083674304,
+            0.014150458196361147,
+            -0.01851165553224022,
+            0.0033450353123933058,
+            -0.018775439467431685,
+            0.002782289821140401,
+            -0.0018711350216089556,
+            0.010502195116579441,
+            -0.00557814442809019,
+            -0.0007130962242755068,
+            -0.013072988887382269,
+            -0.002121841212381393,
+            -0.006653444103331213,
+            0.009394907690516121,
+            0.0028678074167728367,
+            -0.009601577734902932,
+            0.011153571864462314,
+            0.013046116892519414,
+            0.019812860521973938,
+            -0.00539047787915588,
+            -0.00555139958411929,
+            -0.011553689090179406,
+            0.009852063270798534,
+            -0.016326212706634428,
+            -0.00827402913960903,
+            0.008982303148984887,
+            0.015216947130173639,
+            0.01026889105233137,
+            0.003352028462252994,
+            0.003150801750574548,
+            0.002531620731961765,
+            0.026744691368233237,
+            -0.0015120532859309638,
+            0.0008324701960729416,
+            -0.014508988496009931,
+            0.022127525293249926,
+            0.02772430884375744,
+            -0.0012206102581726523,
+            -0.00018859639594030037,
+            0.017437360169132967,
+            -0.004697052522250805,
+            -0.0038111184371524645,
+            0.024015246959315593,
+            -0.038396108183338655,
+            -0.008416825232533771,
+            -0.006788816882617399,
+            -0.006043428198944214,
+            -0.009587366247924467,
+            0.007294190273465812,
+            0.00041966080769004275,
+            0.0024709692753547823,
+            -0.0005250904505954556,
+            0.03452858797518386,
+            0.003403515654095309,
+            -0.0003392060101496309,
+            -0.0032980864446307116,
+            0.0019125121221204029,
+            0.0032987517853214346,
+            -0.009116761755779728,
+            -0.038030085122772624,
+            0.006006334000396066,
+            0.007932785488877407,
+            0.00975009749364395,
+            -0.008120121513849434,
+            -0.008864461019882246,
+            0.004909348738798099,
+            -0.02273129283865203,
+            -0.01372636307167243,
+            0.02625229144362647,
+            -0.008523028986127237,
+            0.029005877324974557,
+            -0.022435059061182174,
+            -0.013135836801818153,
+            -0.004126433322528265,
+            0.010283975013795537,
+            -0.019732856462444506,
+            0.004343860338939202,
+            -0.004022387065239562,
+            0.023733721026005954,
+            0.0077578144626961245,
+            -0.008371631446146718,
+            0.012386735614153616,
+            0.0064789380436815714,
+            -0.0026638642827013837,
+            0.00136333052963848,
+            -0.007209517067748103,
+            0.016916505654836485,
+            -0.011819784026452944,
+            -0.005833909403097991,
+            0.008977527508585844,
+            -0.0024556434204852946,
+            0.02509293948918158,
+            -0.0237762245071987,
+            0.005288171058938098,
+            -0.00019047961064753207,
+            0.01822542471604037,
+            0.008127347078975925,
+            0.008361588667386882,
+            -0.011835407590060994,
+            0.010004482616697591,
+            -0.03378107067882622,
+            -0.015433906677849963,
+            -0.007702910250795191,
+            -0.0013877702264578865,
+            -0.002059152251387601,
+            -0.015745218548968338,
+            -0.026393221735795527,
+            0.006294583740809379,
+            0.016267525575472445,
+            0.023291528020980265,
+            -0.004683197930892445,
+            0.012912830736650997,
+            -0.002831583991769044,
+            -0.008904271394710672,
+            -0.011441882849296418,
+            -0.012202629605208118,
+            -0.012470773094383122,
+            -0.014560694782340029,
+            -0.017045701603711442,
+            0.012139665577738337,
+            -0.020079061996873537,
+            0.013911518328924777,
+            0.0347210775522539,
+            -0.017176114991595492,
+            -0.0018941772168839354,
+            0.019702722555622765,
+            -0.004221469612316137,
+            0.01212088353057276,
+            0.0063707489618353725,
+            0.015298014532351866,
+            0.0038643174573634725,
+            -0.012181479717826009,
+            -0.007374005740809251,
+            0.019411457664902376,
+            -0.004902591844710158,
+            0.014591446725717338,
+            0.015069599884557399,
+            -0.008675672164385891,
+            0.002544198867387345,
+            0.01951865904496276,
+            -0.01762101777798757,
+            0.01665654707715257,
+            -0.00458402229480694,
+            -0.009936206685538294,
+            -0.013345740581684638,
+            -0.005336416724426228,
+            -0.019469457091169776,
+            0.0061608454109892425,
+            0.018931676738452464,
+            -0.00533013594306834,
+            0.009310871143727683,
+            -0.0037670793158045,
+            -0.005045151384034014,
+            0.0002041909886640482,
+            -0.04152751596227068,
+            -0.008340911336217609,
+            -0.0006651584370005978,
+            -0.02505748815737022,
+            0.025954102004571416,
+            0.007205602377144382,
+            0.022602344303400163,
+            -0.017168610245491406,
+            0.004459735690564842,
+            -0.005882024209102714,
+            0.013217517324383834,
+            0.02255689669623969,
+            0.0017984057469785134,
+            0.005699128497215882,
+            -0.02353849721739205,
+            -0.0034711576718386185,
+            0.015924913374553738,
+            -0.003604478141616307,
+            -0.0034993903155063817,
+            0.012592308877056896,
+            0.012864938348003884,
+            0.01567418642605425,
+            0.008801753550506258,
+            0.015606443428689292,
+            0.010589933994371628,
+            0.008928721556432368,
+            -0.005805129999077126,
+            -0.001025103064751441,
+            -0.015907755521365925,
+            0.0045874330999415985,
+            0.031701049450012876,
+            0.012594483199773746,
+            0.016783858873835704,
+            -0.01778536268185936,
+            0.00789523134933153,
+            -0.0012706411671644011,
+            0.008325108613530198,
+            -0.001677173669891956,
+            0.029540136288412976,
+            0.011739999605699826,
+            0.015670966746894652,
+            0.008732334090105053,
+            -0.016237731209426752,
+            0.045334690225148,
+            -0.016911928942220152,
+            -0.0021139311364009113,
+            -0.01058970056572469,
+            -0.021705338531028618,
+            0.007324792557651635,
+            -0.006556582104686972,
+            0.0018243188317438144,
+            0.010780292738415671,
+            -0.006471850255151019,
+            0.0006997395548596146,
+            0.003907433192252573,
+            -0.023129313269763224,
+            -0.002550875642832235,
+            -0.02237917712764273,
+            -0.003107775639788617,
+            -0.02621608419628401,
+            -0.00012482032027919036,
+            -0.016207377503696517,
+            -0.0038712739995960753,
+            -0.023315508569288476,
+            -0.014963211090562272,
+            -0.01633110043829169,
+            -0.004241081909202262,
+            -0.008295402942465353,
+            0.004976396647108597,
+            -0.007402107755970892,
+            -0.020321297548927468,
+            0.0020986169430507757,
+            0.016803273183070004,
+            0.01434729320047051,
+            0.018612970083178984,
+            0.00603868809307069,
+            0.014278001198851581,
+            -0.0026350854954985117,
+            0.017171016209107656,
+            -0.006968968261909459,
+            0.006881847166565678,
+            0.011454001418121107,
+            0.02140935294963631,
+            0.011121833270235448,
+            -0.012801756637470525,
+            0.0010870041670931706,
+            -0.009655388034828094,
+            0.002094601818569396,
+            -0.00828486034340067,
+            -0.009190975158350585,
+            0.0017762581946855365,
+            -0.007261996868906865,
+            0.004674405202069334,
+            0.0030526253833799426,
+            -0.0014082232541965316,
+            0.010419302200840906,
+            -0.0024200053377751805,
+            -0.012208685875688886,
+            -0.0033037330470612695,
+            -0.017975940046938498,
+            -0.006958042903103732,
+            -0.0014496522238488239,
+            -0.011711502944602638,
+            -0.01016130418405875,
+            0.003321436189059608,
+            -0.00538282506552848,
+            -0.009353581612407435,
+            0.013884147453136417,
+            0.0001309738696853529,
+            0.0005878330075918333,
+            0.023495280602974504,
+            0.006035028203642483,
+            0.008087594627921345,
+            0.013144288945385904,
+            -0.01023995000994454,
+            -0.022574958230635715,
+            -0.0243435727055662,
+            0.007867022215096418,
+            -0.01573478573423379,
+            0.02324699908478915,
+            -0.007929809922043611,
+            -0.00561630876431978,
+            0.0020982723335057625,
+            -0.013699317926847062,
+            0.008578431496569186,
+            -0.011271381061801356,
+            0.0014450645788554738,
+            -0.019581451799757368,
+            0.018059311800267413,
+            -0.010917911869373693,
+            -0.004302687883018328,
+            0.009466103226841532,
+            0.034049086327052606,
+            -0.0005689605174080737,
+            -0.007410970644432269,
+            -0.006594150877740108,
+            -0.00022810271313786974,
+            0.018766765773409123,
+            -0.0029041815756733333,
+            0.005758781908059541,
+            -0.01563519107805783,
+            0.006571628287171105,
+            -0.006132372252825422,
+            0.0012423261344297225,
+            0.01513074831647933,
+            -0.026306320707079913,
+            -0.015491710256868008,
+            -0.007857352164317122,
+            -0.024893430640551818,
+            0.002253574511878983,
+            0.017579477884206534,
+            0.020505606775521504,
+            -0.020324505029926956,
+            -0.02513332780748072,
+            0.005150966733930881,
+            -0.026580994694344065,
+            0.0037390577082486594,
+            0.1241031878517593,
+            -0.002138404898309707,
+            0.002720723037908521,
+            -0.011425036886011179,
+            0.0022399461009316216,
+            0.005389407121650262,
+            -0.019552543955456567,
+            0.05950746473186985,
+            -0.00638023587296763,
+            -0.004023798021748334,
+            0.002170545188776296,
+            0.001785855390676113,
+            -0.005831534307818096,
+            0.008643501093778434,
+            -0.017602787281866816,
+            -0.02184792441891879,
+            -0.0052215705748838905,
+            -0.021252169339074325,
+            -0.008509621422071373,
+            -0.013000137524907565,
+            0.002893880203681282,
+            -0.015821609145923173,
+            0.010397692426614709,
+            -0.01638916243796692,
+            -0.010283706937701179,
+            -0.01103621454761223,
+            -0.00373552043529251,
+            0.01563553822905027,
+            -0.03188533919653949,
+            -0.004873550301425776,
+            -0.005183590054152726,
+            0.018218971950313273,
+            0.006004328765748325,
+            0.004357734370247252,
+            -0.014230318971458851,
+            0.0064792070801456235,
+            0.0215589173714926,
+            0.0055272493764401305,
+            0.014643335833970546,
+            0.010618070641357003,
+            -0.0022177303813575846,
+            -0.0024402761885758007,
+            0.017541799738501007,
+            -0.005445774414228888,
+            -0.00023502936119626386,
+            -0.00537569046685066,
+            -0.005974703247446444,
+            -0.010712244203425036,
+            0.01843613166635215,
+            -0.009938564643456672,
+            -0.017388592763277214,
+            -0.0015797412875366156,
+            0.02028434083164951,
+            -0.012761534382003973,
+            0.0031906812114950982,
+            0.000788436210275669,
+            0.020589196433948156,
+            0.0007726855314648301,
+            -0.005481260996289263,
+            0.007165014826294827,
+            -0.02956334397512344,
+            0.004095715232997753,
+            0.0003568512047626187,
+            0.027290768799658912,
+            -0.0017631129386615335,
+            0.01552542586826711,
+            -0.020871286597113327,
+            0.004388125824324094,
+            -0.016926008631435403,
+            0.005099899758322398,
+            0.012560923968841436,
+            0.002781049890810361,
+            0.010889021753593997,
+            -0.008500358897706648,
+            -0.021353460542554444,
+            0.012663934009161902,
+            -0.018657094130860155,
+            -0.029156017587920958,
+            0.0018889388761887053,
+            -0.028158076315864772,
+            -0.003048854609750186,
+            0.006812072971079798,
+            -0.035217412591558014,
+            -0.02251675180883659,
+            0.007396524512996277,
+            -0.00995571600201846,
+            -0.011912753666073882,
+            0.010675240017836214,
+            -0.02645155743041798,
+            -0.018137648527851668,
+            0.0010666944556449141,
+            0.012588800077165337,
+            -0.004561825939541096,
+            0.007555558090958502,
+            -0.028558603070876605,
+            0.014816816621445352,
+            -0.01313764261744744,
+            0.016036065063831813,
+            -0.0037797584923144134,
+            0.022217516257304868,
+            0.011131597210722573,
+            0.008795376259031849,
+            -0.004950080558710245,
+            0.021409590863550987,
+            0.005574351961953293,
+            0.051358937509801736,
+            0.031972009383746405,
+            0.0019821003238211276,
+            -0.003251257665378234,
+            0.003313045083311649,
+            0.008294776423730887,
+            -0.00940948257229233,
+            0.005114631360856872,
+            0.028053227377164806,
+            0.016738013668710083,
+            -0.018755924318109216,
+            0.006606711923035118,
+            -0.02702692774986498,
+            0.017950884766781056,
+            -0.022054653845163,
+            0.0003755944703013671,
+            0.007766253634286795,
+            0.0016206993755471613,
+            0.010682347575070117,
+            0.0013842930079185415,
+            0.01705234255261425,
+            0.023975191051210525,
+            -0.002976402980885077,
+            0.017466027107477605,
+            0.012734909275073793,
+            0.018143161773162015,
+            -0.0038399373197265827,
+            0.006929440921265761,
+            -0.012003095293274786,
+            -0.0008883470597690258,
+            0.0020490527756307267,
+            -0.004231091383919565,
+            0.012128848448049833,
+            0.008340302519160299,
+            0.012989849936937843,
+            0.002974488532907779,
+            0.003198854381461613,
+            -0.005309753760578859,
+            -0.0017024982049936384,
+            -0.01763395035509857,
+            -0.01819680226734474,
+            0.028417724274372468,
+            -0.020217879775485576,
+            -0.050118184195774004,
+            0.0025509548991664425,
+            0.014185761006969567,
+            -0.0012021669304388173,
+            -0.0051260576846881515,
+            -0.0005774778872927327,
+            0.005718864494530703,
+            -0.016918749210038116,
+            -0.0015472638835434902,
+            0.008125845907627804,
+            0.020347547062899797,
+            -0.002036218493206914,
+            -0.01079739518866248,
+            0.0038112958713600906,
+            0.02169339303105242,
+            0.01667594322285835,
+            0.012338193092567552,
+            -0.009798409233733167,
+            0.0003198215558702285,
+            -1.6308740825163337e-05,
+            0.0076870827874737325,
+            0.004356644356656038,
+            0.003500023005601613,
+            -0.019968549869439228,
+            -0.008634740687755112,
+            0.012781565963545847,
+            -0.0025637758121512927,
+            0.010243567793632335,
+            -0.00042296044205132426,
+            -0.003069251553118065,
+            -0.009344199449486244,
+            -0.004694953215957636,
+            -0.009583993744231791,
+            0.013783878036521561,
+            -0.007705982042252044,
+            -0.01549567538665377,
+            0.00020666298495823114,
+            0.0005515775451348472,
+            0.001667367278489879,
+            -0.004518431526236256,
+            0.03302324454129141,
+            0.009531990824506103,
+            -0.0004937348108340977,
+            -0.004047613286672229,
+            -0.0327383112522095,
+            0.012456150771224677,
+            -0.009677533214778067,
+            0.0006455811519981003,
+            0.020012976827426605,
+            -0.00020277620481825933,
+            0.002168432791818704,
+            -0.015657551629104767,
+            0.0032924357559238502,
+            0.006161295735841883,
+            0.013814165416788555,
+            0.0005164426482972658,
+            0.01076512534105537,
+            -0.006498977875968572,
+            -0.010325144017898257,
+            0.004804158067334203,
+            0.0104064855510473,
+            -0.0012188100892986933,
+            0.005171197790296134
+          ],
+          "title": "Mixture-Of-Experts With Expert Choice Routing",
+          "keyphrases": [
+            "Mixture-Of-Experts",
+            "Expert Choice Routing",
+            "sparsely-activated models",
+            "training efficiency",
+            "downstream performance"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "65742957-6bda-446d-a61a-0fe20313c760",
+        "properties": {
+          "page_content": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity\n\nWilliam Fedus∗ liamfedus@google.com Barret Zoph∗ barretzoph@google.com Noam Shazeer noam@google.com Google, Mountain View, CA 94043, USA Editor: Alexander Clark\n\nAbstract\n\nIn deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) models defy this and instead select different parameters for each incoming example. The result is a sparsely-activated model—with an outrageous number of parameters—but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs, and training instability. We address these with the introduction of the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques mitigate the instabilities, and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large (Raffel et al., 2019) to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the \"Colossal Clean Crawled Corpus\", and achieve a 4x speedup over the T5-XXL model.12 Keywords: mixture-of-experts, natural language processing, sparsity, large-scale machine learning, distributed computing arXiv:2101.03961v3 [cs.LG] 16 Jun 2022 Contents\n\n1 Introduction 3 2 Switch Transformer 4 2.1 Simplifying Sparse Routing 5 2.2 Efficient Sparse Routing 6 2.3 Putting It All Together: The Switch Transformer 8 2.4 Improved Training and Fine-Tuning Techniques 8 3 Scaling Properties 11 3.1 Scaling Results on a Step-Basis 12 3.2 Scaling Results on a Time-Basis 13 3.3 Scaling Versus a Larger Dense Model 13 4 Downstream Results 14 4.1 Fine-Tuning 14 4.2 Distillation 16 4.3 Multilingual Learning 17 5 Designing Models with Data, Model, and Expert-Parallelism 18 5.1 Data Parallelism 20 5.2 Model Parallelism 20 5.3 Model and Data Parallelism 21 5.4 Expert and Data Parallelism 22 5.5 Expert, Model and Data Parallelism 22 5.6 Towards Trillion Parameter Models 22 6 Related Work 24 7 Discussion 25 8 Future Work 26 9 Conclusion 27 A Switch for Attention 27 B Preventing Token Dropping with No-Token-Left-Behind 29\n\nC Encouraging Exploration Across Experts 29 D Switch Transformers in Lower Compute Regimes 29 E Relation of Upstream to Downstream Model Performance 32 F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/switch_transformers.md"
+          },
+          "summary": "The Switch Transformer is a sparsely-activated model that simplifies the Mixture of Experts (MoE) approach, allowing for efficient scaling to trillion-parameter models with reduced communication and computational costs. It achieves significant speedups in pre-training and fine-tuning by activating only a subset of neural network weights for each input, maintaining a constant computational cost. The model demonstrates superior scaling properties and improved performance across various natural language processing tasks, including multilingual settings. It also allows for effective distillation into smaller dense models, preserving a portion of the quality gains. The architecture is designed to maximize parameter count while keeping floating point operations per example constant, making it suitable for both large-scale and smaller-scale applications.",
+          "headlines": {
+            "1 Introduction": [],
+            "2 Switch Transformer": [
+              "2.1 Simplifying Sparse Routing",
+              "2.2 Efficient Sparse Routing",
+              "2.3 Putting It All Together: The Switch Transformer",
+              "2.4 Improved Training and Fine-Tuning Techniques"
+            ],
+            "3 Scaling Properties": [
+              "3.1 Scaling Results on a Step-Basis",
+              "3.2 Scaling Results on a Time-Basis",
+              "3.3 Scaling Versus a Larger Dense Model"
+            ],
+            "4 Downstream Results": [
+              "4.1 Fine-Tuning",
+              "4.2 Distillation",
+              "4.3 Multilingual Learning"
+            ],
+            "5 Designing Models with Data, Model, and Expert-Parallelism": [
+              "5.1 Data Parallelism",
+              "5.2 Model Parallelism",
+              "5.3 Model and Data Parallelism",
+              "5.4 Expert and Data Parallelism",
+              "5.5 Expert, Model and Data Parallelism",
+              "5.6 Towards Trillion Parameter Models"
+            ],
+            "6 Related Work": [],
+            "7 Discussion": [],
+            "8 Future Work": [],
+            "9 Conclusion": [],
+            "A Switch for Attention": [],
+            "B Preventing Token Dropping with No-Token-Left-Behind": [],
+            "C Encouraging Exploration Across Experts": [],
+            "D Switch Transformers in Lower Compute Regimes": [],
+            "E Relation of Upstream to Downstream Model Performance": [],
+            "F Pseudo Code for Switch Transformers": []
+          },
+          "summary_embedding": [
+            -0.0060447100549936295,
+            -0.00390331051312387,
+            -0.01784370467066765,
+            -0.0021820589900016785,
+            0.013227885589003563,
+            0.011338187381625175,
+            0.0165580902248621,
+            0.024240797385573387,
+            -0.005986624862998724,
+            0.03596621751785278,
+            -0.004852031823247671,
+            -0.0068811336532235146,
+            -0.003574162255972624,
+            0.014242436736822128,
+            0.011578272096812725,
+            -0.033302053809165955,
+            -0.024519605562090874,
+            0.01050176378339529,
+            0.016093410551548004,
+            -0.006795942317694426,
+            0.007473600562661886,
+            -0.030513975769281387,
+            -0.06124480068683624,
+            -0.006838538218289614,
+            -0.0026099516544491053,
+            0.01029265858232975,
+            0.0010329445358365774,
+            -0.010873507708311081,
+            -0.053469158709049225,
+            0.023311438038945198,
+            0.025294071063399315,
+            0.005092116538435221,
+            -0.029584616422653198,
+            0.0009361362317577004,
+            -0.05161044001579285,
+            -0.04448534920811653,
+            0.0379488542675972,
+            0.012561844661831856,
+            -0.04364892467856407,
+            0.039249956607818604,
+            -0.004875265993177891,
+            -0.0022440163884311914,
+            -0.021499186754226685,
+            -0.0034909071400761604,
+            -0.007039899472147226,
+            -0.009851212613284588,
+            -0.02874819189310074,
+            0.0014037203509360552,
+            0.021886419504880905,
+            -0.0056962003000080585,
+            0.023264970630407333,
+            -0.024271776899695396,
+            0.01936165988445282,
+            0.04240977764129639,
+            -0.06957805901765823,
+            0.024659009650349617,
+            0.017549408599734306,
+            0.05120771750807762,
+            0.06511713564395905,
+            0.018339363858103752,
+            0.021669568493962288,
+            -0.014141756109893322,
+            -0.017766257748007774,
+            -0.011872569099068642,
+            0.004673904739320278,
+            -0.014970434829592705,
+            -0.01891246810555458,
+            0.008449427783489227,
+            -0.020461400970816612,
+            0.008209343999624252,
+            -0.05576157942414284,
+            -0.014730350114405155,
+            -0.034200433641672134,
+            -0.01140014547854662,
+            0.017642343416810036,
+            -0.018215449526906013,
+            0.030126741155982018,
+            0.02997184917330742,
+            0.002638994250446558,
+            0.0007822110201232135,
+            0.021049996837973595,
+            0.006509389728307724,
+            -0.0053438181057572365,
+            0.012321759946644306,
+            0.028159597888588905,
+            -0.01450575515627861,
+            -0.010106786154210567,
+            -0.03454120084643364,
+            0.01861817203462124,
+            0.01589204929769039,
+            -0.04194509983062744,
+            -0.012569589540362358,
+            -0.013824224472045898,
+            0.032341714948415756,
+            -0.009549170732498169,
+            0.04098476096987724,
+            -0.010083552449941635,
+            0.001383390510454774,
+            -0.031149037182331085,
+            -0.009463978931307793,
+            0.017069239169359207,
+            0.006172496825456619,
+            -0.016589069738984108,
+            -0.0012846461031585932,
+            0.016170857474207878,
+            0.006617815233767033,
+            -0.030142230913043022,
+            0.04488807171583176,
+            0.023497309535741806,
+            0.055296897888183594,
+            0.05579255893826485,
+            -0.037391237914562225,
+            -0.002445377642288804,
+            0.0344172865152359,
+            -0.0228312686085701,
+            -0.015040136873722076,
+            -0.034014564007520676,
+            -0.01768881268799305,
+            0.035935238003730774,
+            -0.0318460576236248,
+            0.01556677371263504,
+            0.028066661208868027,
+            -0.025712283328175545,
+            0.003488970920443535,
+            -0.03943582624197006,
+            0.003622566582635045,
+            0.01220559049397707,
+            0.006095050368458033,
+            -0.0026293133851140738,
+            -0.013615119270980358,
+            0.008906363509595394,
+            0.003055269829928875,
+            0.00800798274576664,
+            -0.04615819454193115,
+            0.018478767946362495,
+            -0.00454224506393075,
+            0.04033420979976654,
+            0.042502716183662415,
+            -0.02730768360197544,
+            -0.016743963584303856,
+            0.008557853288948536,
+            0.012104909867048264,
+            0.04188314080238342,
+            -0.007287728600203991,
+            -0.00121784838847816,
+            -0.0007836631848476827,
+            0.007322579622268677,
+            0.047552235424518585,
+            0.004646798130124807,
+            0.0020697612781077623,
+            -0.027230238541960716,
+            0.014978179708123207,
+            -0.019733402878046036,
+            0.027648448944091797,
+            0.023373395204544067,
+            0.02942972257733345,
+            -0.04048910364508629,
+            0.017518429085612297,
+            -0.011818356812000275,
+            -0.005568413063883781,
+            -0.002726121572777629,
+            -0.02856232039630413,
+            -0.0074426219798624516,
+            0.033983584493398666,
+            0.015365412458777428,
+            0.012956822291016579,
+            0.03271345794200897,
+            0.0012052633101120591,
+            -0.042502716183662415,
+            0.03804178908467293,
+            0.024550583213567734,
+            0.04640602692961693,
+            0.023466331884264946,
+            0.010796061716973782,
+            0.006571347359567881,
+            0.013375034555792809,
+            -0.019377149641513824,
+            0.015907539054751396,
+            0.02560385875403881,
+            0.009750531986355782,
+            -0.03167567402124405,
+            -0.0009501734166406095,
+            0.01178737822920084,
+            -0.004546117503196001,
+            0.03701949119567871,
+            0.04104671627283096,
+            0.020569827407598495,
+            0.05030933395028114,
+            0.0020155487582087517,
+            0.017936641350388527,
+            0.005936284549534321,
+            0.02820606529712677,
+            0.030467506498098373,
+            -0.005285732913762331,
+            0.005076627247035503,
+            -0.01136916596442461,
+            -0.03385967016220093,
+            0.014110777527093887,
+            0.01561324205249548,
+            -0.008077683858573437,
+            -0.01697630248963833,
+            0.006033093202859163,
+            0.009169681929051876,
+            -0.01795213110744953,
+            -0.017456471920013428,
+            0.005971135571599007,
+            0.018866000697016716,
+            -0.013576395809650421,
+            0.024689987301826477,
+            -0.04442339017987251,
+            0.029119934886693954,
+            -0.033209118992090225,
+            -0.003384418087080121,
+            -0.010641167871654034,
+            -0.0030843124259263277,
+            0.0041666291654109955,
+            0.011314953677356243,
+            0.013406013138592243,
+            0.002042655134573579,
+            -0.013909416273236275,
+            0.024519605562090874,
+            0.018447788432240486,
+            0.012073930352926254,
+            -0.03148980438709259,
+            -0.014397329650819302,
+            -0.0015673263696953654,
+            -0.008743725717067719,
+            -0.034138478338718414,
+            0.026966920122504234,
+            0.014048820361495018,
+            -0.03844451159238815,
+            0.017797237262129784,
+            0.029538147151470184,
+            0.014420564286410809,
+            -0.004790074657648802,
+            -0.0463130883872509,
+            0.04045812413096428,
+            -0.030436528846621513,
+            -0.033333033323287964,
+            -0.010478530079126358,
+            -0.0003146269591525197,
+            0.02894955314695835,
+            -0.0064358157105743885,
+            -0.050835974514484406,
+            0.01184933539479971,
+            0.020631784573197365,
+            0.00010763872705865651,
+            0.005731051322072744,
+            -0.019593998789787292,
+            0.00028631053282879293,
+            -0.03395260497927666,
+            0.0090844901278615,
+            0.026750069111585617,
+            0.008867640048265457,
+            -0.016418687999248505,
+            0.03351890295743942,
+            -0.0004121613164898008,
+            -0.018695618957281113,
+            -0.003889757441356778,
+            0.006885006092488766,
+            0.034912943840026855,
+            0.004836542531847954,
+            -0.02946070022881031,
+            0.05665995925664902,
+            -0.02100352756679058,
+            0.021344292908906937,
+            0.05631919577717781,
+            -0.019439106807112694,
+            -0.008472662419080734,
+            -0.002065889071673155,
+            0.0026196325197815895,
+            -0.0260530486702919,
+            -0.008937342092394829,
+            0.032341714948415756,
+            0.041728246957063675,
+            -0.018742086365818977,
+            -0.001398879918269813,
+            0.006780453026294708,
+            0.009425255469977856,
+            0.004948840010911226,
+            0.02692045085132122,
+            -0.012128143571317196,
+            -0.014559967443346977,
+            0.05198218300938606,
+            0.028314489871263504,
+            -0.012081675231456757,
+            -0.0164031982421875,
+            0.019888296723365784,
+            -0.024844881147146225,
+            -0.020244551822543144,
+            -0.02431824430823326,
+            -0.020414933562278748,
+            -0.0042518205009400845,
+            0.009525936096906662,
+            0.037484172731637955,
+            -0.03968365862965584,
+            0.012747716158628464,
+            0.021700548008084297,
+            0.02981695532798767,
+            -0.019532041624188423,
+            -0.05910727381706238,
+            -0.0016128262504935265,
+            0.03296129032969475,
+            -0.015458349138498306,
+            -0.049379974603652954,
+            0.015164051204919815,
+            0.0260530486702919,
+            0.05957195535302162,
+            -0.001461805310100317,
+            -0.014954946003854275,
+            0.061926331371068954,
+            0.03026614524424076,
+            -0.004383479710668325,
+            -0.009494957514107227,
+            -0.0127864396199584,
+            0.003396035172045231,
+            0.00465454300865531,
+            -0.03364282101392746,
+            -0.041170634329319,
+            -0.011841590516269207,
+            -0.010284913703799248,
+            0.008108663372695446,
+            -0.040922801941633224,
+            0.03531566634774208,
+            -0.021917399019002914,
+            0.010950954630970955,
+            0.015473837964236736,
+            0.037391237914562225,
+            -0.021344292908906937,
+            -0.01284839678555727,
+            -0.01572941243648529,
+            -0.024968795478343964,
+            -0.014010096900165081,
+            0.021917399019002914,
+            0.007682706229388714,
+            3.4699722164077684e-05,
+            0.010075807571411133,
+            -0.024659009650349617,
+            -0.051052823662757874,
+            0.0291044469922781,
+            0.0015605497173964977,
+            -0.015938518568873405,
+            -0.004623564425855875,
+            -0.01977987214922905,
+            -0.01122201792895794,
+            -0.012608313001692295,
+            -0.039249956607818604,
+            0.021111954003572464,
+            -0.01135367713868618,
+            0.015876561403274536,
+            0.0009250032599084079,
+            0.003659353591501713,
+            0.028624277561903,
+            0.01116006076335907,
+            0.0051811798475682735,
+            -0.03255856782197952,
+            -0.018277406692504883,
+            0.010122275911271572,
+            -0.00561100896447897,
+            -0.018339363858103752,
+            -0.013351799920201302,
+            -0.05037129297852516,
+            -0.03305422514677048,
+            -0.008054450154304504,
+            -0.05415068939328194,
+            0.0020697612781077623,
+            -0.014296649023890495,
+            -0.02232012152671814,
+            -0.005990497302263975,
+            0.03107159025967121,
+            -0.003235333366319537,
+            0.025294071063399315,
+            0.0076594725251197815,
+            0.037360258400440216,
+            0.040117356926202774,
+            0.008426194079220295,
+            0.03065337985754013,
+            -0.0029991210903972387,
+            -0.016867877915501595,
+            0.05486319959163666,
+            -0.01881953328847885,
+            0.013529927469789982,
+            -0.008604321628808975,
+            0.008604321628808975,
+            0.03299226611852646,
+            -0.003794885240495205,
+            0.02007416822016239,
+            -0.0311955064535141,
+            -0.07998688519001007,
+            0.01637221872806549,
+            0.028159597888588905,
+            -0.06418777257204056,
+            0.014691627584397793,
+            -0.020275529474020004,
+            0.02052335813641548,
+            0.02411688305437565,
+            -0.0009114501299336553,
+            -0.014064309187233448,
+            0.005595519673079252,
+            0.0020833145827054977,
+            -0.01456771232187748,
+            -0.002691270550712943,
+            -0.009417511522769928,
+            0.008774704299867153,
+            -0.017007282003760338,
+            -0.013406013138592243,
+            -0.034231413155794144,
+            0.010873507708311081,
+            0.01233724970370531,
+            -0.003969140350818634,
+            0.0011500825639814138,
+            -0.03277541697025299,
+            -0.007876322604715824,
+            -0.02145271934568882,
+            0.03382869064807892,
+            0.012058441527187824,
+            -0.030699847266077995,
+            -0.007814365439116955,
+            0.015264731831848621,
+            -0.007876322604715824,
+            0.0008906363509595394,
+            0.024132372811436653,
+            0.02103450708091259,
+            0.0028364830650389194,
+            -0.0009700191440060735,
+            0.02438020147383213,
+            0.007427132688462734,
+            -0.01720864325761795,
+            0.028794659301638603,
+            0.014288905076682568,
+            0.060160547494888306,
+            -0.03271345794200897,
+            0.001297231181524694,
+            -0.04135650396347046,
+            -0.031118059530854225,
+            0.006408709101378918,
+            0.014350862242281437,
+            0.007597515359520912,
+            -0.028252532705664635,
+            0.005978880450129509,
+            0.02004319056868553,
+            0.01469937153160572,
+            0.009704063646495342,
+            0.0037580979987978935,
+            -0.036740683019161224,
+            -0.000691695255227387,
+            -0.011214273050427437,
+            -0.0038781403563916683,
+            0.035656433552503586,
+            -0.008186109364032745,
+            -0.05151750147342682,
+            -0.0006418389966711402,
+            -0.03255856782197952,
+            -0.0353776253759861,
+            0.021313315257430077,
+            -0.00839521549642086,
+            -0.0039207362569868565,
+            -0.013986863195896149,
+            -0.000537770101800561,
+            -0.010524998418986797,
+            0.027509044855833054,
+            0.028345469385385513,
+            0.022490503266453743,
+            -0.05384090170264244,
+            0.009479468688368797,
+            0.00023766436788719147,
+            -0.05817791447043419,
+            0.0009709871956147254,
+            -0.007117345929145813,
+            0.011191039346158504,
+            -0.020213572308421135,
+            -0.02414786070585251,
+            0.011500826105475426,
+            0.03392162546515465,
+            0.014304393902420998,
+            0.0321558453142643,
+            -0.013367289677262306,
+            -0.09250226616859436,
+            0.018292896449565887,
+            -0.008689512498676777,
+            0.04126356914639473,
+            0.00865078903734684,
+            -0.004278926644474268,
+            -0.015845581889152527,
+            -0.00865078903734684,
+            -0.006943091284483671,
+            0.01283290795981884,
+            0.004530628211796284,
+            -0.005626498255878687,
+            0.0016631665639579296,
+            0.013708055019378662,
+            -0.005967263597995043,
+            0.007481344975531101,
+            0.003651608945801854,
+            -0.013189162127673626,
+            -0.06059424951672554,
+            -0.04055105894804001,
+            -0.018540725111961365,
+            0.050835974514484406,
+            -0.0069159846752882,
+            -0.046684831380844116,
+            -0.010850274004042149,
+            0.05058814212679863,
+            0.012259802781045437,
+            -0.0334569476544857,
+            -0.0007841471815481782,
+            0.0050882440991699696,
+            0.015458349138498306,
+            0.004546117503196001,
+            -0.027292195707559586,
+            0.02820606529712677,
+            0.0008412640891037881,
+            0.02300165221095085,
+            -0.018339363858103752,
+            -0.005312839522957802,
+            0.035780347883701324,
+            -0.024844881147146225,
+            0.023791607469320297,
+            0.0055219451896846294,
+            0.031629208475351334,
+            0.002952652983367443,
+            0.010060317814350128,
+            0.004704883322119713,
+            0.0186491496860981,
+            0.004275054205209017,
+            0.0276639387011528,
+            0.011462102644145489,
+            0.006428070832043886,
+            0.021623101085424423,
+            -0.009378788061439991,
+            -0.014513500034809113,
+            0.01885051093995571,
+            -0.0014695499558001757,
+            0.025232113897800446,
+            -0.020538847893476486,
+            0.002726121572777629,
+            -0.0016883367206901312,
+            -0.017967619001865387,
+            -0.040922801941633224,
+            -0.0007405834621749818,
+            0.021747015416622162,
+            -0.011671207845211029,
+            -0.0228312686085701,
+            -0.013715799897909164,
+            -0.01585332676768303,
+            0.0038490979932248592,
+            0.034169457852840424,
+            -0.0398695282638073,
+            0.00963436160236597,
+            -0.000830615172162652,
+            -0.03125746175646782,
+            -0.0239619892090559,
+            -0.01093546487390995,
+            0.014614180661737919,
+            -0.01652711257338524,
+            -0.021173911169171333,
+            -0.028469383716583252,
+            -0.008410705253481865,
+            0.006048582494258881,
+            -0.05065010115504265,
+            -0.004054331686347723,
+            0.019965743646025658,
+            0.0276639387011528,
+            -0.003099801717326045,
+            -0.02177799493074417,
+            0.0012468908680602908,
+            0.022351099178195,
+            0.02936776541173458,
+            0.014226946979761124,
+            0.027586491778492928,
+            -0.002178186783567071,
+            -0.0011675080750137568,
+            -0.0073148347437381744,
+            0.01307299267500639,
+            0.0011287847300991416,
+            0.036616768687963486,
+            0.0122210793197155,
+            -0.020306508988142014,
+            0.0013959755888208747,
+            -0.006431943271309137,
+            0.029181893914937973,
+            -0.02270735427737236,
+            -9.832092473516241e-05,
+            0.03531566634774208,
+            0.027741385623812675,
+            0.004933350719511509,
+            -0.0143121387809515,
+            -0.05142456665635109,
+            0.021592123433947563,
+            -0.01765783317387104,
+            0.031025122851133347,
+            0.016666516661643982,
+            0.003397971158847213,
+            -0.0034212053287774324,
+            0.012368228286504745,
+            -0.022087780758738518,
+            -0.004406713880598545,
+            -0.021142931655049324,
+            0.04389675334095955,
+            -0.043184246867895126,
+            0.008472662419080734,
+            0.029445212334394455,
+            0.011926782317459583,
+            -0.019082851707935333,
+            -0.005425137002021074,
+            0.029212871566414833,
+            -0.00162637943867594,
+            0.01308848150074482,
+            0.0058665829710662365,
+            -0.026935940608382225,
+            0.01202746294438839,
+            -0.031660184264183044,
+            0.02177799493074417,
+            -0.013436991721391678,
+            0.018447788432240486,
+            0.03342596814036369,
+            -0.0030397805385291576,
+            -0.0038239278364926577,
+            -0.016263794153928757,
+            -0.00969631876796484,
+            0.0662323608994484,
+            0.01054823212325573,
+            0.03726732358336449,
+            0.0018974426202476025,
+            -0.009773765690624714,
+            0.007969259284436703,
+            0.0063041565008461475,
+            -0.012329504825174809,
+            0.010494019836187363,
+            0.024178840219974518,
+            0.0012730291346088052,
+            -0.002412462839856744,
+            -0.019408127292990685,
+            -0.03438630700111389,
+            0.003727119415998459,
+            -0.030932186171412468,
+            0.022691864520311356,
+            0.009208405390381813,
+            0.017456471920013428,
+            0.008240322582423687,
+            -0.015907539054751396,
+            0.03200095146894455,
+            0.05461536720395088,
+            0.0019768255297094584,
+            0.013375034555792809,
+            0.007938280701637268,
+            0.007036027032881975,
+            -0.011338187381625175,
+            0.04702559858560562,
+            0.000662652775645256,
+            0.005169562995433807,
+            0.022536970674991608,
+            0.02145271934568882,
+            0.022939693182706833,
+            0.002304037567228079,
+            -0.011694442480802536,
+            0.010478530079126358,
+            -0.0006302220281213522,
+            0.004445436876267195,
+            -0.010695381090044975,
+            -0.027865299955010414,
+            0.007492962293326855,
+            0.01302652433514595,
+            0.022598929703235626,
+            0.021592123433947563,
+            -0.03178409859538078,
+            -0.01855621486902237,
+            0.03364282101392746,
+            -0.02997184917330742,
+            0.03612111136317253,
+            -0.0014424435794353485,
+            -0.023807097226381302,
+            -0.0058356039226055145,
+            0.016929835081100464,
+            0.02656419761478901,
+            -0.029181893914937973,
+            0.01804506592452526,
+            0.029693040996789932,
+            -0.0074426219798624516,
+            0.0005939188995398581,
+            0.012112653814256191,
+            0.02447313815355301,
+            -0.009293596260249615,
+            -0.0024027819745242596,
+            -0.0013320822035893798,
+            -0.011926782317459583,
+            -0.018370341509580612,
+            -0.0044725434854626656,
+            -0.022629907354712486,
+            0.005564541090279818,
+            -0.007365175057202578,
+            -0.003481226274743676,
+            -0.008581086993217468,
+            0.01154729351401329,
+            0.001979729626327753,
+            -0.019857317209243774,
+            -0.022335609421133995,
+            0.01652711257338524,
+            0.03887821361422539,
+            -0.018215449526906013,
+            -0.02106548473238945,
+            0.024442158639431,
+            -0.01759587600827217,
+            -0.01601596549153328,
+            0.036678727716207504,
+            0.013739033602178097,
+            0.007736918982118368,
+            -0.03358086198568344,
+            -0.030049296095967293,
+            -0.03274443745613098,
+            0.006296411622315645,
+            0.03785591572523117,
+            -0.0028752065263688564,
+            0.02569679357111454,
+            0.03701949119567871,
+            0.0057891360484063625,
+            -0.028934063389897346,
+            -0.014823286794126034,
+            -0.045104920864105225,
+            0.060160547494888306,
+            0.0011481463443487883,
+            0.05161044001579285,
+            0.016294771805405617,
+            -0.01456771232187748,
+            0.00419760774821043,
+            -0.013251119293272495,
+            0.026192452758550644,
+            0.02280029095709324,
+            0.025929134339094162,
+            0.006582964211702347,
+            0.022505993023514748,
+            -0.0037522895727306604,
+            -0.012197845615446568,
+            0.007307090330868959,
+            0.014536733739078045,
+            0.0002497654058970511,
+            -0.011392400600016117,
+            0.015783624723553658,
+            -0.01307299267500639,
+            -0.011361422017216682,
+            0.023295948281884193,
+            0.01368482131510973,
+            -0.012344993650913239,
+            0.014064309187233448,
+            -0.008565598167479038,
+            0.01239920686930418,
+            -0.023915521800518036,
+            0.022691864520311356,
+            -0.03262052312493324,
+            0.009587894193828106,
+            0.0015053689712658525,
+            -0.0073148347437381744,
+            0.024550583213567734,
+            -0.0027454833034425974,
+            -0.05071205645799637,
+            0.03550153970718384,
+            -0.008015726692974567,
+            0.0037213109899312258,
+            0.006381602957844734,
+            0.011392400600016117,
+            -0.0012623801594600081,
+            0.03807276859879494,
+            0.01239920686930418,
+            0.015969496220350266,
+            0.02199484594166279,
+            0.011911292560398579,
+            -0.008433938957750797,
+            -0.004096927121281624,
+            -0.0016137943603098392,
+            -0.01784370467066765,
+            0.025944624096155167,
+            0.0018674320308491588,
+            0.03962169960141182,
+            -0.018168980255723,
+            -0.02300165221095085,
+            0.028066661208868027,
+            0.007597515359520912,
+            -0.02351279929280281,
+            -0.0017377089243382215,
+            0.01202746294438839,
+            -0.009185170754790306,
+            0.027385130524635315,
+            0.01196550577878952,
+            0.019113829359412193,
+            -0.018060555681586266,
+            -0.004340883810073137,
+            -0.005072754807770252,
+            -0.015938518568873405,
+            0.02238207869231701,
+            0.020771188661456108,
+            0.011152315884828568,
+            -0.011880313977599144,
+            -0.01307299267500639,
+            0.020600805059075356,
+            0.035036858171224594,
+            -0.0002744515077210963,
+            0.005672966130077839,
+            0.017487451434135437,
+            0.002373739378526807,
+            -0.04470219835639,
+            0.016232814639806747,
+            -0.014025585725903511,
+            0.011214273050427437,
+            0.019593998789787292,
+            -0.021917399019002914,
+            -0.03196997195482254,
+            -0.011539548635482788,
+            -0.0414804182946682,
+            0.00493722315877676,
+            -0.0069159846752882,
+            0.013576395809650421,
+            -0.010176488198339939,
+            -0.0014831031439825892,
+            0.016666516661643982,
+            0.0038103745318949223,
+            0.02177799493074417,
+            -0.011678952723741531,
+            -0.012073930352926254,
+            -0.020213572308421135,
+            -0.020786678418517113,
+            0.004278926644474268,
+            -0.030405549332499504,
+            -0.004488032776862383,
+            0.021375272423028946,
+            -0.04783104360103607,
+            -0.03689557686448097,
+            0.02898053266108036,
+            -0.008503641001880169,
+            0.020755698904395103,
+            0.022955182939767838,
+            -0.015094350092113018,
+            -0.0008717587334103882,
+            -0.018695618957281113,
+            0.02258343994617462,
+            -0.007837600074708462,
+            0.026300877332687378,
+            -0.0027551641687750816,
+            0.0006292539183050394,
+            -0.02659517526626587,
+            0.020306508988142014,
+            0.005227648187428713,
+            0.012275291606783867,
+            -0.02390003204345703,
+            0.006815304048359394,
+            -0.03133491054177284,
+            -0.01784370467066765,
+            0.0180140882730484,
+            -0.011152315884828568,
+            0.027989214286208153,
+            -0.006060199346393347,
+            0.028438406065106392,
+            0.022722844034433365,
+            0.024751944467425346,
+            0.02676555886864662,
+            -0.02363671362400055,
+            0.008495896123349667,
+            0.03726732358336449,
+            0.00865078903734684,
+            -0.015876561403274536,
+            0.008170620538294315,
+            -0.04863648861646652,
+            -0.008062195032835007,
+            0.05030933395028114,
+            0.002127846470102668,
+            0.023141054436564445,
+            0.016294771805405617,
+            -0.002873270306736231,
+            -0.005688455421477556,
+            0.010269423946738243,
+            -0.04296739399433136,
+            -0.023172033950686455,
+            0.022629907354712486,
+            -0.021561143919825554,
+            6.54212328754511e-07,
+            0.03122648410499096,
+            0.012120398692786694,
+            -0.0009109660750254989,
+            -0.05108380317687988,
+            0.02100352756679058,
+            0.01178737822920084,
+            0.014443797990679741,
+            -0.010300402529537678,
+            0.026068538427352905,
+            0.01733255758881569,
+            0.038413532078266144,
+            0.012755461037158966,
+            -0.002679653698578477,
+            -0.03782493621110916,
+            0.009177426807582378,
+            -0.021685058251023293,
+            0.0011471783509477973,
+            0.0074503663927316666,
+            -0.02273833192884922,
+            -0.024782923981547356,
+            0.0043912241235375404,
+            0.004588713403791189,
+            0.011880313977599144,
+            0.022087780758738518,
+            0.009417511522769928,
+            -0.021204888820648193,
+            0.015404135920107365,
+            -0.05446047708392143,
+            -0.015450604259967804,
+            -0.023884542286396027,
+            0.02566581591963768,
+            0.020012211054563522,
+            0.010184233076870441,
+            0.005545179359614849,
+            0.010904486291110516,
+            -0.020988039672374725,
+            -0.009239383973181248,
+            0.020631784573197365,
+            0.011074868962168694,
+            -0.012840652838349342,
+            0.04113965481519699,
+            -0.00011126903700642288,
+            0.031210994347929955,
+            -0.02280029095709324,
+            0.002941036131232977,
+            -0.001241082325577736,
+            -0.03649285435676575,
+            0.009130958467721939,
+            0.01981084980070591,
+            0.02151467651128769,
+            -0.01861817203462124,
+            0.0024143988266587257,
+            0.01048627495765686,
+            0.004766840487718582,
+            0.038351576775312424,
+            0.027121812105178833,
+            -0.044206541031599045,
+            0.011663463898003101,
+            0.012995545752346516,
+            -0.009363298304378986,
+            0.048419639468193054,
+            -0.017518429085612297,
+            0.017673322930932045,
+            0.008240322582423687,
+            -0.01798310875892639,
+            0.02318752370774746,
+            -0.01469937153160572,
+            -0.013282098807394505,
+            0.00650551775470376,
+            0.009192915633320808,
+            -0.0001770381786627695,
+            -0.037577107548713684,
+            -0.017859194427728653,
+            -0.005719434004276991,
+            0.0038297362625598907,
+            0.01562873087823391,
+            0.013498948886990547,
+            0.009967382065951824,
+            0.005161818582564592,
+            -0.00945623405277729,
+            -0.0016089539276435971,
+            0.008124152198433876,
+            -0.0005532593932002783,
+            0.0019216446671634912,
+            -0.01926872320473194,
+            0.02698240801692009,
+            0.02572777308523655,
+            -0.0018354853382334113,
+            -0.04652994126081467,
+            -0.0036709706764668226,
+            0.0037077576853334904,
+            -0.017735280096530914,
+            -0.0035315665882080793,
+            -0.04519785940647125,
+            -0.01810702309012413,
+            0.0143121387809515,
+            -0.014614180661737919,
+            -0.0460033044219017,
+            0.009076746180653572,
+            -0.013886182568967342,
+            0.0063970922492444515,
+            -0.00932457484304905,
+            0.005762029904872179,
+            0.003651608945801854,
+            0.005680711008608341,
+            -0.005909178406000137,
+            0.05108380317687988,
+            -0.03466511517763138,
+            0.028051171451807022,
+            0.008565598167479038,
+            -0.012437930330634117,
+            -0.02261441759765148,
+            0.00887538492679596,
+            -0.044981006532907486,
+            -0.033209118992090225,
+            0.016743963584303856,
+            -0.01781272701919079,
+            0.011074868962168694,
+            0.06161654368042946,
+            0.007330324035137892,
+            -0.023342415690422058,
+            -0.01233724970370531,
+            -0.021049996837973595,
+            -0.04191412031650543,
+            0.003442503046244383,
+            0.005463860463351011,
+            -0.004948840010911226,
+            -0.03807276859879494,
+            -0.021716037765145302,
+            0.04615819454193115,
+            -0.012004229240119457,
+            0.012422440573573112,
+            0.01080380566418171,
+            -0.0019410063978284597,
+            -0.04860550910234451,
+            -0.021019017323851585,
+            0.0395597442984581,
+            0.021886419504880905,
+            -0.008426194079220295,
+            0.0028364830650389194,
+            -0.018277406692504883,
+            -0.003289545886218548,
+            0.006296411622315645,
+            -0.009107724763453007,
+            -0.03562545403838158,
+            0.02067825198173523,
+            0.0054832217283546925,
+            -0.023373395204544067,
+            -0.025805220007896423,
+            -0.03296129032969475,
+            0.009944148361682892,
+            0.02453509531915188,
+            -0.01968693546950817,
+            0.005823987070471048,
+            0.00027203132049180567,
+            -0.000881923595443368,
+            0.00042740863864310086,
+            -0.009959637187421322,
+            -0.019764382392168045,
+            -0.020151615142822266,
+            0.03268248215317726,
+            -0.0060059865936636925,
+            0.0074387495405972,
+            -0.03630698472261429,
+            -0.010974188335239887,
+            0.011322698555886745,
+            -0.002395037328824401,
+            -0.013452480547130108,
+            -0.009092235006392002,
+            -0.01739451475441456,
+            -0.009882191196084023,
+            -0.007876322604715824,
+            0.022660886868834496,
+            0.006168624851852655,
+            0.005409647710621357,
+            0.0023795480374246836,
+            -0.008581086993217468,
+            0.02898053266108036,
+            0.017874684184789658,
+            -0.014188224449753761,
+            -0.00920066051185131,
+            0.03451022133231163,
+            0.02665713243186474,
+            -0.009115469641983509,
+            0.01798310875892639,
+            0.023264970630407333,
+            -0.0010484338272362947,
+            -0.01196550577878952,
+            0.001922612776979804,
+            -0.0009550138493068516,
+            -0.0032972905319184065,
+            0.00845717266201973,
+            -0.015365412458777428,
+            0.0315207801759243,
+            0.04287445917725563,
+            -0.002145271748304367,
+            0.009587894193828106,
+            -0.003727119415998459,
+            -0.008472662419080734,
+            0.017069239169359207,
+            0.009316830895841122,
+            0.04625113308429718,
+            0.01098193321377039,
+            0.005680711008608341,
+            -0.033983584493398666,
+            -0.001229465357027948,
+            -0.01184933539479971,
+            0.03308520466089249,
+            0.02061629481613636,
+            0.00889087375253439,
+            -0.008379725739359856,
+            0.01344473659992218,
+            0.008341003209352493,
+            0.00661007035523653,
+            0.020569827407598495,
+            -0.016542602330446243,
+            -0.017286090180277824,
+            0.030467506498098373,
+            -0.00672236829996109,
+            0.01289486512541771,
+            0.02029101923108101,
+            -0.002089123008772731,
+            0.010842529125511646,
+            0.07100307941436768,
+            -0.006184114143252373,
+            0.015713922679424286,
+            -0.019082851707935333,
+            0.0015789433382451534,
+            -0.007640110794454813,
+            -0.03370477631688118,
+            0.018246427178382874,
+            0.0039962464943528175,
+            0.00865078903734684,
+            0.01198099460452795,
+            0.009053511545062065,
+            -0.007946024648845196,
+            -0.00963436160236597,
+            0.015195030719041824,
+            0.0019080914789810777,
+            0.00033568276558071375,
+            0.001336922636255622,
+            -0.0065210070461034775,
+            -0.004011735785752535,
+            -0.021948376670479774,
+            0.02763296104967594,
+            0.007140580099076033,
+            0.030606910586357117,
+            -0.055296897888183594,
+            0.030095763504505157,
+            -0.00789568480104208,
+            0.02554190158843994,
+            -0.016201836988329887,
+            -0.011090358719229698,
+            0.001609921921044588,
+            0.018153492361307144,
+            0.004491904750466347,
+            0.006462921854108572,
+            0.011888058856129646,
+            0.023233991116285324,
+            -0.03444826602935791,
+            0.017053749412298203,
+            0.023110076785087585,
+            0.0048675211146473885,
+            0.0003535923024173826,
+            0.005165690556168556,
+            0.015458349138498306,
+            0.007198664825409651,
+            -0.023048119619488716,
+            -0.0025692922063171864,
+            0.017611365765333176,
+            -0.02306360937654972,
+            0.0036980770528316498,
+            -0.006923729553818703,
+            0.004623564425855875,
+            -0.0014705180656164885,
+            -0.03509881719946861,
+            -0.00100099784322083,
+            -0.025743262842297554,
+            0.01923774555325508,
+            0.009556914679706097,
+            -0.025867177173495293,
+            0.003932353109121323,
+            0.025650326162576675,
+            -0.0003695656778290868,
+            0.00821708794683218,
+            0.0004683101433329284,
+            0.00869725737720728,
+            -0.00974278710782528,
+            -0.01493945624679327,
+            0.004801691509783268,
+            0.01531120017170906,
+            -0.01493945624679327,
+            -0.007431004662066698,
+            -0.009510447271168232,
+            0.030560443177819252,
+            0.0006674932083114982,
+            0.004189862869679928,
+            -0.005045648664236069,
+            -0.01858719252049923,
+            -0.0032237162813544273,
+            -0.021142931655049324,
+            0.004367990419268608,
+            -0.0009046735358424485,
+            0.04197607934474945,
+            0.0038161829579621553,
+            -0.02029101923108101,
+            0.0032972905319184065,
+            -0.03136589005589485,
+            0.0005000148667022586,
+            -0.007276111748069525,
+            -0.02004319056868553,
+            0.0015005286550149322,
+            0.0016525175888091326,
+            0.012631546705961227,
+            -0.03255856782197952,
+            -0.009301341138780117,
+            0.018416810780763626,
+            -0.00020680672605521977,
+            -0.010687636211514473,
+            -0.017828216776251793,
+            0.00035189816844649613,
+            -0.011338187381625175,
+            -0.027973726391792297,
+            -0.02850036323070526,
+            -0.0026719090528786182,
+            0.013491204008460045,
+            -0.018277406692504883,
+            -0.014993669465184212,
+            0.0031249718740582466,
+            0.021592123433947563,
+            -0.009781510569155216,
+            0.008302279748022556,
+            0.02241305634379387,
+            -0.008255811408162117,
+            -0.005514200776815414,
+            -0.000941460661124438,
+            -0.009130958467721939,
+            0.004267309792339802,
+            0.006300284061580896,
+            0.043339136987924576,
+            -0.032403673976659775,
+            -0.0018238683696836233,
+            0.014815541915595531,
+            0.0034153966698795557,
+            0.0025905899237841368,
+            0.002503462601453066,
+            -0.0007841471815481782,
+            0.01344473659992218,
+            0.01178737822920084,
+            0.015466093085706234,
+            0.00100099784322083,
+            -0.0033011629711836576,
+            0.006408709101378918,
+            -0.021173911169171333,
+            0.015721667557954788,
+            0.027958236634731293,
+            -0.0033999073784798384,
+            0.005525817628949881,
+            -0.013731288723647594,
+            -0.008759214542806149,
+            0.013134949840605259,
+            0.0026641644071787596,
+            -0.005068882368505001,
+            -0.009394276887178421,
+            0.017967619001865387,
+            0.019082851707935333,
+            0.006668155547231436,
+            -0.037453193217515945,
+            -0.02898053266108036,
+            -0.009959637187421322,
+            0.020569827407598495,
+            0.014916222542524338,
+            0.0010339126456528902,
+            -0.010540487244725227,
+            -0.009595638141036034,
+            -0.011903548613190651,
+            0.010989678092300892,
+            0.02721474878489971,
+            -0.032372694462537766,
+            0.0008214184199459851,
+            -0.029383255168795586,
+            0.03068435750901699,
+            0.0020000594668090343,
+            0.011446612887084484,
+            -0.0008185141487047076,
+            -0.015140817500650883,
+            -0.029181893914937973,
+            0.013824224472045898,
+            -0.004395096562802792,
+            0.001024231780320406,
+            -0.0031753121875226498,
+            0.0201051477342844,
+            0.012887120246887207,
+            -0.0014472840121015906,
+            0.01968693546950817,
+            0.008178364485502243,
+            0.016480645164847374,
+            -0.010153254494071007,
+            -0.04135650396347046,
+            0.019965743646025658,
+            0.007562664337456226,
+            0.014219203032553196,
+            0.004739734344184399,
+            0.004588713403791189,
+            -0.020771188661456108,
+            0.002724185585975647,
+            0.004333139397203922,
+            0.014405074529349804,
+            0.011028401553630829,
+            0.0025615475606173277,
+            0.011678952723741531,
+            -0.02228914201259613,
+            0.01929970271885395,
+            -0.01631026156246662,
+            0.01006806269288063,
+            0.004445436876267195,
+            -0.026873983442783356,
+            0.007547175046056509,
+            0.007822110317647457,
+            -0.0019187404541298747,
+            -0.017549408599734306,
+            0.02160761132836342,
+            -0.0034831624943763018,
+            -0.03509881719946861,
+            0.0360281765460968,
+            -0.0043447562493383884,
+            0.019346170127391815,
+            0.03209388628602028,
+            -0.01771979033946991,
+            -0.01993476413190365,
+            0.00906900130212307,
+            0.010408828034996986,
+            -0.002873270306736231,
+            0.017859194427728653,
+            -0.002919738180935383,
+            -0.02232012152671814,
+            0.017766257748007774,
+            -0.028918573632836342,
+            -0.014234691858291626,
+            0.020337486639618874,
+            -0.03624502569437027,
+            -0.0021026763133704662,
+            0.03175312280654907,
+            -0.024070415645837784,
+            0.005734923295676708,
+            0.005196669604629278,
+            -0.0019351978553459048,
+            0.0012236569309607148,
+            -0.006451305001974106,
+            0.03801080957055092,
+            -0.010865762829780579,
+            0.0201051477342844,
+            0.01929970271885395,
+            -0.023388884961605072,
+            0.01977987214922905,
+            -0.001109423115849495,
+            0.01675945147871971,
+            -0.003940097521990538,
+            0.03466511517763138,
+            -0.006017603911459446,
+            -0.011330443434417248,
+            0.006431943271309137,
+            0.008488151244819164,
+            -0.009959637187421322,
+            0.008681767620146275,
+            -0.018122512847185135,
+            0.027152791619300842,
+            0.02091059274971485,
+            -0.0002691270783543587,
+            0.013800990767776966,
+            0.0004421719058882445,
+            -0.020833145827054977,
+            0.00789568480104208,
+            -0.027121812105178833,
+            -0.009967382065951824,
+            0.01646515540778637,
+            -0.005762029904872179,
+            0.007380664348602295,
+            -0.02064727433025837,
+            0.028004704043269157,
+            0.0077524082735180855,
+            -0.013591884635388851,
+            -0.0008824076503515244,
+            0.023760627955198288,
+            -0.005432881414890289,
+            0.009355553425848484,
+            0.02070923149585724,
+            0.015713922679424286,
+            0.013615119270980358,
+            0.016263794153928757,
+            0.03351890295743942,
+            0.005328328814357519,
+            0.005568413063883781,
+            0.003519949736073613,
+            0.016991792246699333,
+            -0.008557853288948536,
+            0.018029578030109406,
+            0.0008093173382803798,
+            -0.013576395809650421,
+            0.021499186754226685,
+            -0.0018684001406654716,
+            -0.013622863218188286,
+            -0.0040891822427511215,
+            -0.004337011836469173,
+            0.03271345794200897,
+            0.022041313350200653,
+            -0.004468671046197414,
+            -0.024442158639431,
+            -0.00394784240052104,
+            0.016682006418704987,
+            -0.003167567541822791,
+            0.011136826127767563,
+            0.0007952801533974707,
+            -0.01586107164621353,
+            -0.009595638141036034,
+            0.0063583687879145145,
+            0.010586955584585667,
+            -0.008921852335333824,
+            0.010346870869398117,
+            -0.0228312686085701,
+            -0.004011735785752535,
+            0.00046613195445388556,
+            -0.004844287410378456,
+            0.0053438181057572365,
+            -0.011314953677356243,
+            0.004948840010911226,
+            0.016062432900071144,
+            0.01026167906820774,
+            0.004832670092582703,
+            0.006989559158682823,
+            0.012066186405718327,
+            0.009161937050521374,
+            -0.01627928391098976,
+            0.00428667152300477,
+            -0.005072754807770252,
+            -0.02345084212720394,
+            0.005831731949001551,
+            0.00902253296226263,
+            -0.01122201792895794,
+            0.009386532939970493,
+            -0.010958699509501457,
+            0.03296129032969475,
+            0.01030814740806818,
+            -0.037391237914562225,
+            -0.0033379499800503254,
+            -0.0016244432190433145,
+            0.014885243959724903,
+            0.012298526242375374,
+            0.008550108410418034,
+            -0.017518429085612297,
+            -0.0047823297791182995,
+            -0.009835722856223583,
+            0.0012827098835259676,
+            -0.012306271120905876,
+            0.017456471920013428,
+            0.022567950189113617,
+            -0.036337960511446,
+            0.01968693546950817,
+            -0.010432061739265919,
+            -0.023652203381061554,
+            0.020476890727877617,
+            0.011500826105475426,
+            0.01942361705005169,
+            -0.009402021765708923,
+            0.018896980211138725,
+            -0.0001331114035565406,
+            0.025510922074317932,
+            -0.004228586331009865,
+            -0.022087780758738518,
+            -0.012499887496232986,
+            -0.03209388628602028,
+            -0.0051811798475682735,
+            0.006819176487624645,
+            -0.00045524101005867124,
+            0.013111716136336327,
+            0.014226946979761124,
+            -0.025712283328175545,
+            0.007295473478734493,
+            0.032589543610811234,
+            0.014637414366006851,
+            0.023311438038945198,
+            -0.011276230216026306,
+            -0.0010794125264510512,
+            -0.0026060794480144978,
+            -0.013770012184977531,
+            -0.03212486580014229,
+            -0.0017493258928880095,
+            -0.036585792899131775,
+            0.005680711008608341,
+            -0.016387708485126495,
+            -0.013096226379275322,
+            0.023264970630407333,
+            -0.034138478338718414,
+            0.0014608372002840042,
+            0.013708055019378662,
+            0.010540487244725227,
+            0.012391461990773678,
+            -0.00162637943867594,
+            0.015032391995191574,
+            -0.029243851080536842,
+            -0.010284913703799248,
+            -0.004491904750466347,
+            -0.01981084980070591,
+            -0.007338068913668394,
+            -0.018060555681586266,
+            -0.013127204962074757,
+            -0.003566417610272765,
+            -0.010060317814350128,
+            0.002247888594865799,
+            0.0032062907703220844,
+            -0.014714861288666725,
+            0.03206290677189827,
+            0.003157886676490307,
+            0.018958937376737595,
+            0.0033418224193155766,
+            0.004449309315532446,
+            0.018788553774356842,
+            -0.0026331855915486813,
+            -0.0058278595097362995,
+            0.030963165685534477,
+            -0.00032672801171429455,
+            0.008759214542806149,
+            0.019082851707935333,
+            0.004724245052784681,
+            0.0038529701996594667,
+            -0.0260530486702919,
+            0.012887120246887207,
+            -0.027447087690234184,
+            0.005080499220639467,
+            -0.0077175572514534,
+            -0.024302754551172256,
+            -0.028128618374466896,
+            -0.006571347359567881,
+            0.012058441527187824,
+            -0.02898053266108036,
+            -0.00613377382978797,
+            0.002315654419362545,
+            -0.00400399137288332,
+            -0.002673845272511244,
+            0.007241260726004839,
+            0.029150914400815964,
+            -0.033209118992090225,
+            0.008751469664275646,
+            0.005595519673079252,
+            -0.0005053392960689962,
+            0.013297587633132935,
+            0.008038961328566074,
+            0.003343758638948202,
+            0.020848635584115982,
+            0.006637176964432001,
+            0.002458930714055896,
+            -0.0025576751213520765,
+            -0.014242436736822128,
+            0.004000118933618069,
+            0.02154565416276455,
+            0.007562664337456226,
+            0.018385831266641617,
+            -0.008914108388125896,
+            0.002621568739414215,
+            -0.01455222349613905,
+            0.005552923772484064,
+            -0.002867461647838354,
+            0.004309905227273703,
+            -0.012236569076776505,
+            0.008093173615634441,
+            0.0028771425131708384,
+            -0.0043447562493383884,
+            0.018060555681586266,
+            -0.002319526858627796,
+            0.016480645164847374,
+            0.004015608225017786,
+            0.015620986931025982,
+            -0.0035412474535405636,
+            0.01178737822920084,
+            0.0005731051205657423,
+            -0.017905661836266518,
+            -0.017022771760821342,
+            -0.039249956607818604,
+            0.007702067960053682,
+            0.03531566634774208,
+            -0.0009564659558236599,
+            -0.03444826602935791,
+            0.010145509615540504,
+            -0.011942271143198013,
+            -0.009975126944482327,
+            -0.022862248122692108,
+            0.013862947933375835,
+            0.012430185452103615,
+            0.007415515370666981,
+            0.012252057902514935,
+            -0.006997303571552038,
+            0.01026167906820774,
+            -0.005216030869632959,
+            0.008356492035090923,
+            -0.0010823167394846678,
+            -0.037422213703393936,
+            0.012344993650913239,
+            0.02216522768139839,
+            -0.002635121811181307,
+            0.0016089539276435971,
+            -0.006145390681922436,
+            0.022103270515799522,
+            -0.013971373438835144,
+            -0.00650551775470376,
+            -0.00046443779137916863,
+            0.00797700323164463,
+            -0.01795213110744953,
+            0.004778457339853048,
+            0.0755879208445549,
+            -0.007775642443448305,
+            0.006122156511992216,
+            0.009270362555980682,
+            -0.015148562379181385,
+            -0.01637221872806549,
+            -0.004561606794595718,
+            -0.020260039716959,
+            0.008844406343996525,
+            -0.0018819533288478851,
+            -0.0110981035977602,
+            0.012468908913433552,
+            0.0053050946444272995,
+            0.002031038049608469,
+            -0.008379725739359856,
+            0.010811550542712212,
+            0.009525936096906662,
+            0.010432061739265919,
+            0.02360573597252369,
+            0.011477591469883919,
+            -0.00858883187174797,
+            -0.011795123107731342,
+            0.01129171997308731,
+            0.0028577810153365135,
+            0.016697494313120842,
+            0.007907302118837833,
+            0.0010648912284523249,
+            0.019501063972711563,
+            -0.002309845993295312,
+            0.02180897258222103,
+            -2.2689444449497387e-05,
+            -0.009301341138780117,
+            -0.004236331209540367,
+            -0.015349923633038998,
+            0.024271776899695396,
+            -0.0020155487582087517,
+            0.0015450604259967804,
+            -0.020213572308421135,
+            0.0013979118084535003,
+            0.013243375346064568,
+            0.011957760900259018,
+            -0.0020407189149409533,
+            -0.0051347119733691216,
+            0.0007154133054427803,
+            -0.016356728971004486,
+            0.032806396484375,
+            0.007578153628855944,
+            0.0007681738352403045,
+            0.0008838597568683326,
+            0.0009288755827583373,
+            -0.0032972905319184065,
+            0.0024608669336885214,
+            -0.004766840487718582,
+            0.009549170732498169,
+            -0.010943209752440453,
+            -0.010796061716973782,
+            0.010075807571411133,
+            -0.02402394637465477,
+            0.003566417610272765,
+            -0.00558777479454875,
+            -0.008062195032835007,
+            0.0076207490637898445,
+            0.00015537731815129519,
+            -0.007392281666398048,
+            -0.0030939930584281683,
+            0.0023408245760947466,
+            0.013429246842861176,
+            0.004406713880598545,
+            0.010850274004042149,
+            0.012956822291016579,
+            0.005359307397156954,
+            -0.006629432085901499,
+            0.015512561425566673,
+            -0.006946963258087635,
+            -0.004228586331009865,
+            -0.0012468908680602908,
+            -0.019996721297502518,
+            -0.0009419447160325944,
+            -0.004708755761384964,
+            0.0015092412941157818,
+            0.0001705036120256409,
+            -8.749050175538287e-05,
+            -0.0018490385264158249,
+            -0.011988739483058453,
+            -0.001831613015383482,
+            -0.011663463898003101,
+            -0.014498010277748108,
+            0.010253935120999813,
+            -0.0041201612912118435,
+            -0.022939693182706833,
+            -0.015349923633038998,
+            -0.020662762224674225,
+            0.02145271934568882,
+            -0.004387352149933577,
+            -0.005800753366202116,
+            0.006733985152095556,
+            -0.010215211659669876,
+            -0.015706177800893784,
+            -0.008069939911365509,
+            -0.0205853171646595,
+            0.010703125037252903,
+            0.003214035416021943,
+            0.0006791101768612862,
+            0.00882891658693552,
+            0.014397329650819302,
+            -0.01589204929769039,
+            0.0111755495890975,
+            0.012182355858385563,
+            0.011361422017216682,
+            -0.006854027509689331,
+            0.0008911203476600349,
+            -0.014103032648563385,
+            0.02596011385321617,
+            0.021623101085424423,
+            0.005169562995433807,
+            -0.04132552444934845,
+            -0.005862710531800985,
+            -0.013344055972993374,
+            -0.010524998418986797,
+            -0.012120398692786694,
+            0.011771888472139835,
+            -0.0038820127956569195,
+            0.0013117524795234203,
+            0.006288666743785143,
+            0.0015063370810821652,
+            0.004367990419268608,
+            -0.0022382077295333147,
+            -0.007709812838584185,
+            -0.03212486580014229,
+            -0.02817508764564991,
+            -0.00863530021160841,
+            0.020461400970816612,
+            0.002838419284671545,
+            -0.0009453330421820283,
+            -0.0036380558740347624,
+            0.0027048238553106785,
+            -0.005882072262465954,
+            -0.005018542055040598,
+            -0.014753584749996662,
+            -0.012739972211420536,
+            0.0101300198584795,
+            0.00802347157150507,
+            0.014195968396961689,
+            -0.011950016021728516,
+            0.0028190575540065765,
+            0.018788553774356842,
+            0.007690451107919216,
+            -0.023357905447483063,
+            0.01135367713868618,
+            -0.010912231169641018,
+            -0.009270362555980682,
+            -0.009433000348508358,
+            -0.009471723809838295,
+            0.0016980175860226154,
+            -0.008317768573760986,
+            0.003945906180888414,
+            -0.03615209087729454,
+            -0.011330443434417248,
+            -0.0013901671627536416,
+            -0.010346870869398117,
+            0.0009864765452221036,
+            0.02524760365486145,
+            0.006815304048359394,
+            0.01432762760668993,
+            -0.0006733016925863922,
+            0.017502939328551292,
+            -0.0018984107300639153,
+            0.018726596608757973,
+            0.012763205915689468,
+            -0.00797700323164463,
+            0.011717676185071468,
+            0.014738094992935658,
+            -0.0033863543067127466,
+            0.015620986931025982,
+            -0.010168743319809437,
+            0.005285732913762331,
+            -0.009556914679706097,
+            -0.003214035416021943,
+            0.002495717955753207,
+            0.030901208519935608,
+            -0.003618694143369794,
+            0.00858883187174797,
+            0.01098193321377039,
+            -0.015241498127579689,
+            0.0026951429899781942,
+            0.014962689951062202,
+            0.01141563430428505,
+            0.007760153152048588,
+            0.0024434414226561785,
+            -0.024225307628512383,
+            0.004193735308945179,
+            0.005037903785705566,
+            0.0031520782504230738,
+            -0.014366351068019867,
+            0.0021859314292669296,
+            0.015675200149416924,
+            0.008550108410418034,
+            -0.025325050577521324,
+            0.016418687999248505,
+            0.0009138703462667763,
+            -0.003909118939191103,
+            0.029119934886693954,
+            -0.011268485337495804,
+            -0.005723306443542242,
+            0.009463978931307793,
+            0.0009980935137718916,
+            0.0030029932968318462,
+            0.008937342092394829,
+            0.0228312686085701,
+            -0.003988501615822315,
+            -0.032341714948415756,
+            -0.022459525614976883,
+            0.015706177800893784,
+            -0.0047823297791182995,
+            -0.022939693182706833,
+            -0.026703599840402603,
+            -0.003113354789093137,
+            -0.02228914201259613,
+            -0.0022323993034660816,
+            -0.020058678463101387,
+            -0.014056564308702946,
+            0.001268188701942563,
+            0.029987338930368423,
+            0.016201836988329887,
+            0.008224832825362682,
+            -0.011051635257899761,
+            -0.029956359416246414,
+            -0.024271776899695396,
+            0.01810702309012413,
+            -0.007055388763546944,
+            -0.005541306920349598,
+            0.00591692328453064,
+            -0.009951893240213394,
+            -0.008480406366288662,
+            0.011802867986261845,
+            0.027137301862239838,
+            -0.002747419523075223,
+            -0.029987338930368423,
+            -0.02688947319984436,
+            -0.014335372485220432,
+            -0.01840132102370262,
+            -0.017704300582408905,
+            -0.008627555333077908,
+            0.0045035220682621,
+            0.00030300996149890125,
+            0.016186347231268883,
+            0.014962689951062202,
+            -0.000480653194244951,
+            0.0028597170021384954,
+            -2.7711375878425315e-05,
+            -0.001763847190886736,
+            0.006888878531754017,
+            -0.004933350719511509,
+            0.026997897773981094,
+            0.014614180661737919,
+            -0.006594581063836813,
+            0.018711106851696968,
+            0.028314489871263504,
+            0.020430423319339752,
+            0.003157886676490307,
+            -0.010695381090044975,
+            -0.014606435783207417,
+            0.009727297350764275,
+            -0.013421501964330673,
+            0.023264970630407333,
+            0.022552460432052612,
+            -0.01480779703706503,
+            -0.014885243959724903,
+            -0.017905661836266518,
+            0.015210519544780254,
+            -0.007562664337456226,
+            -0.008224832825362682,
+            0.012368228286504745,
+            0.022335609421133995,
+            -0.007024410180747509,
+            0.002371803391724825,
+            -0.0010232636705040932,
+            -0.021731525659561157,
+            -0.006122156511992216,
+            -0.004596457816660404,
+            -0.008666278794407845,
+            0.008681767620146275,
+            0.008131897076964378,
+            -0.001928421319462359,
+            -0.006319645792245865,
+            0.005471604876220226,
+            -0.013367289677262306,
+            0.01301103550940752,
+            -0.00789568480104208,
+            0.03379771113395691,
+            -0.009138703346252441,
+            -0.0006733016925863922,
+            -0.022598929703235626,
+            -0.011446612887084484,
+            0.00845717266201973,
+            0.0017967619933187962,
+            0.01682141050696373,
+            -0.0011626676423475146,
+            -0.005688455421477556,
+            0.009913169778883457,
+            -0.0010687635513022542,
+            -0.001450188341550529,
+            -0.006892750971019268,
+            -0.010401083156466484,
+            0.0002884887217078358,
+            0.013258864171802998,
+            -0.006025348324328661,
+            0.0018054747488349676,
+            0.006095050368458033,
+            -0.0175339188426733,
+            0.029027000069618225,
+            0.023884542286396027,
+            0.017255110666155815,
+            0.014660648070275784,
+            0.0006394187803380191,
+            0.011647974140942097,
+            -0.003572226269170642,
+            0.0038820127956569195,
+            -0.020337486639618874,
+            0.021080974489450455,
+            -0.010811550542712212,
+            0.01846327818930149,
+            0.011655719019472599,
+            -0.015233753249049187,
+            -0.005022414494305849,
+            0.021406250074505806,
+            0.009967382065951824,
+            0.00889087375253439,
+            0.0017541663255542517,
+            -0.02470547705888748,
+            0.017766257748007774,
+            0.010099041275680065,
+            -0.020399443805217743,
+            0.019547531381249428,
+            -0.009471723809838295,
+            0.01233724970370531,
+            -0.0005300253978930414,
+            0.004418330732733011,
+            0.0058743273839354515,
+            0.021499186754226685,
+            0.011570527218282223,
+            -0.03884723410010338,
+            -0.013615119270980358,
+            -0.0017357728211209178,
+            0.006857899948954582,
+            -0.0036303112283349037,
+            -0.015233753249049187,
+            -0.041635312139987946,
+            0.01604694314301014,
+            0.009998360648751259,
+            -0.01643417589366436,
+            0.004987563472241163,
+            -0.002427952131256461,
+            -0.0042053526267409325,
+            -0.008170620538294315,
+            0.003186929039657116,
+            -0.004170501604676247,
+            0.02324948087334633,
+            0.0018984107300639153,
+            -0.006946963258087635,
+            0.013258864171802998,
+            0.01344473659992218,
+            -0.021948376670479774,
+            0.0009874446550384164,
+            0.0021375271026045084,
+            0.000636998622212559,
+            0.023683181032538414,
+            -0.014707116410136223,
+            -0.0017657832941040397,
+            -0.01685238815844059,
+            -0.0013069120468571782,
+            -0.021592123433947563,
+            -0.001188805908896029,
+            -0.015187285840511322,
+            -0.008511385880410671,
+            0.025030752643942833,
+            -0.011005166918039322,
+            0.010997422970831394,
+            -0.006687517277896404,
+            0.006943091284483671,
+            -0.006904367823153734,
+            -0.01936165988445282,
+            -0.0008267428493127227,
+            0.009975126944482327,
+            -0.000512115890160203,
+            0.002941036131232977,
+            8.482827252009884e-05,
+            0.002464739140123129,
+            0.008867640048265457,
+            0.006234454456716776,
+            -0.0075278133153915405,
+            0.02489134855568409,
+            -0.012073930352926254,
+            0.0016399325104430318,
+            0.011911292560398579,
+            0.0004743606550619006,
+            -0.010424317792057991,
+            0.025588368996977806,
+            -0.02139076218008995,
+            -0.00800798274576664,
+            -0.00014799568452872336,
+            -0.0016544538084417582,
+            -0.006513262167572975,
+            0.006718495860695839,
+            0.02946070022881031,
+            0.0062654330395162106,
+            0.010447551496326923,
+            -0.0281441081315279,
+            -0.011493081226944923,
+            0.012538610957562923,
+            -0.022273652255535126,
+            -0.0007739823195151985,
+            -0.0004322490422055125,
+            -0.00619960343465209,
+            0.026378324255347252,
+            -0.02022906206548214,
+            -0.01093546487390995,
+            -0.018664639443159103,
+            0.005405775271356106,
+            -0.011121337302029133,
+            0.008294534869492054,
+            -0.0009777637897059321,
+            0.006540368776768446,
+            0.016805920749902725,
+            -0.019563021138310432,
+            0.016031453385949135,
+            0.005119222681969404,
+            -0.016217326745390892,
+            0.010222956538200378,
+            -0.017580386251211166,
+            -0.0008199662552215159,
+            0.020275529474020004,
+            -0.00400399137288332,
+            0.008751469664275646,
+            -0.003363120136782527,
+            0.005208286456763744,
+            -0.006513262167572975,
+            0.0001492057926952839,
+            -0.007249005138874054,
+            0.014428308233618736,
+            -0.016805920749902725,
+            -0.017828216776251793,
+            -0.008821171708405018,
+            0.0016709112096577883,
+            0.021297825500369072,
+            0.011136826127767563,
+            -0.010044828988611698,
+            -0.011059380136430264,
+            -0.011694442480802536,
+            0.041604332625865936,
+            0.005762029904872179,
+            0.027230238541960716,
+            -0.02261441759765148,
+            0.00545998802408576,
+            0.025170156732201576,
+            -0.014242436736822128,
+            0.0012788375606760383,
+            -0.0022788674104958773,
+            0.0053050946444272995,
+            0.011562783271074295,
+            -0.01819995976984501,
+            0.0049139889888465405,
+            -0.00019216447253711522,
+            -0.00782985519617796,
+            -0.000656844291370362,
+            0.003442503046244383,
+            0.00020196630794089288,
+            0.016325751319527626,
+            -0.004027225077152252,
+            -0.016511622816324234,
+            -0.004429947584867477,
+            -0.0003216455806978047,
+            -0.027896279469132423,
+            0.013514438644051552,
+            0.016589069738984108,
+            0.01566745527088642,
+            0.027989214286208153,
+            -0.014513500034809113,
+            0.016031453385949135,
+            0.009533680975437164,
+            0.013901671394705772,
+            -0.010610189288854599,
+            0.008952830918133259,
+            -0.01595400646328926,
+            0.007446494419127703,
+            -0.03454120084643364,
+            -0.007887939922511578,
+            -0.00802347157150507,
+            0.01984182931482792,
+            0.028763681650161743,
+            0.002923610620200634,
+            0.006493900436908007,
+            0.000196883876924403,
+            0.0024202074855566025,
+            -0.01714668609201908,
+            -0.020182594656944275,
+            -0.0003337466041557491,
+            0.0029332914855331182,
+            0.0006950835813768208,
+            -0.004495777189731598,
+            0.00472037261351943,
+            -0.012391461990773678,
+            0.0026835259050130844,
+            -0.013134949840605259,
+            0.005971135571599007,
+            -0.0059982421807944775,
+            0.009603383019566536,
+            0.0019429425010457635,
+            0.020864123478531837,
+            0.02300165221095085,
+            0.004100799560546875,
+            0.0046971384435892105,
+            -0.024333734065294266,
+            0.00920066051185131,
+            -0.013847459107637405,
+            0.012329504825174809,
+            0.019408127292990685,
+            0.008248066529631615,
+            0.006571347359567881,
+            0.01233724970370531,
+            0.012918098829686642,
+            -0.0031772484071552753,
+            0.024287264794111252,
+            0.005014669615775347,
+            -0.0009070937521755695,
+            0.002052335999906063,
+            -0.014954946003854275,
+            -0.0070979841984808445,
+            -0.0015286030247807503,
+            0.0018848575418815017,
+            -0.004731989465653896,
+            0.01906736195087433,
+            0.015256987884640694,
+            -0.01050176378339529,
+            0.017874684184789658,
+            -0.03937387093901634,
+            0.017379024997353554,
+            -0.0027377386577427387,
+            -0.0060911779291927814,
+            -0.00545998802408576,
+            0.00665653869509697,
+            0.004515138920396566,
+            0.003669034456834197,
+            0.03773200139403343,
+            -0.010408828034996986,
+            0.003963331691920757,
+            -0.005847221240401268,
+            0.007121218368411064,
+            -0.020864123478531837,
+            0.008302279748022556,
+            0.005336073227226734,
+            0.02160761132836342,
+            -0.0056497324258089066,
+            -0.0014482521219179034,
+            0.02328045852482319,
+            0.008859895169734955,
+            0.03469609469175339,
+            -0.01566745527088642,
+            -0.004418330732733011,
+            0.01048627495765686,
+            0.0207402091473341,
+            -0.00580462533980608,
+            -0.008976065553724766,
+            0.007903429679572582,
+            0.0035586729645729065,
+            0.021824462339282036,
+            -0.00515407370403409,
+            -0.0006650729919783771,
+            0.0054367538541555405,
+            -0.0060911779291927814,
+            0.009735042229294777,
+            0.011252996511757374,
+            -0.009076746180653572,
+            0.007539430167526007,
+            0.021530164405703545,
+            -0.009975126944482327,
+            -0.008658533915877342,
+            -0.009649851359426975,
+            0.0010658593382686377,
+            -0.03068435750901699,
+            0.001649613375775516,
+            0.005359307397156954,
+            0.015187285840511322,
+            -0.0004603234410751611,
+            -0.009014789015054703,
+            0.021019017323851585,
+            0.017255110666155815,
+            -0.005936284549534321,
+            -0.0028132491279393435,
+            0.0023795480374246836,
+            -0.0017154429806396365,
+            -0.006160879973322153,
+            -0.0006248975405469537,
+            -0.002656419761478901,
+            0.014730350114405155,
+            -0.0015469965292140841,
+            -0.006617815233767033,
+            -0.00756653631106019,
+            0.0033302053343504667,
+            0.010834784246981144,
+            -0.005661349277943373,
+            0.013057502917945385,
+            0.016805920749902725,
+            -0.00031704717548564076,
+            0.019671445712447166,
+            0.01037010457366705,
+            -0.010447551496326923,
+            0.015938518568873405,
+            -0.007984748110175133,
+            0.006563602481037378,
+            0.0164031982421875,
+            -0.01277095079421997,
+            -0.01246116403490305,
+            0.007210282143205404,
+            0.00665653869509697,
+            0.028918573632836342,
+            0.014792308211326599,
+            -0.02109646424651146,
+            0.014908477663993835,
+            0.015768134966492653,
+            -0.013018779456615448,
+            0.003537375247105956,
+            -0.01784370467066765,
+            0.020260039716959,
+            0.019051872193813324,
+            -0.015535795129835606,
+            -0.017224133014678955,
+            -0.014947201125323772,
+            -0.005092116538435221,
+            0.006331262644380331,
+            0.0033263331279158592,
+            -0.00617636926472187,
+            -0.027989214286208153,
+            -0.0049139889888465405,
+            -0.018633661791682243,
+            0.013870692811906338,
+            -0.008674023672938347,
+            0.0007715621031820774,
+            -0.004511266481131315,
+            0.007984748110175133,
+            -0.0033882902935147285,
+            0.032465629279613495,
+            -0.0051811798475682735,
+            -0.02832997962832451,
+            0.004398969002068043,
+            0.016356728971004486,
+            0.012623801827430725,
+            -0.029243851080536842,
+            0.010463040322065353,
+            0.004805563949048519,
+            -0.012530866079032421,
+            0.005990497302263975,
+            -0.02258343994617462,
+            -0.007632365915924311,
+            -0.0009395244996994734,
+            -0.016743963584303856,
+            -0.027121812105178833,
+            -0.008433938957750797,
+            -0.006993431597948074,
+            -0.005401902832090855,
+            -0.009525936096906662,
+            -0.008720491081476212,
+            -0.011477591469883919,
+            0.02303262986242771,
+            -0.007899557240307331,
+            -0.001866463921032846,
+            -0.0014637414133176208,
+            0.00930908601731062,
+            -0.023776117712259293,
+            -0.015605497173964977,
+            0.006389347370713949,
+            -0.002594462363049388,
+            0.022815778851509094,
+            -0.012747716158628464,
+            0.005940156988799572,
+            -0.009316830895841122,
+            0.009851212613284588,
+            0.015241498127579689,
+            0.004956584889441729,
+            0.017440982162952423,
+            -0.009889936074614525,
+            0.0021026763133704662,
+            -0.008627555333077908,
+            -0.00018986526993103325,
+            -1.027075541060185e-05,
+            0.014428308233618736,
+            0.007624621503055096,
+            0.014141756109893322,
+            0.017487451434135437,
+            -0.013553161174058914,
+            0.005003052763640881,
+            -0.0003274540649726987,
+            -0.017255110666155815,
+            0.0027725896798074245,
+            -0.007984748110175133,
+            -0.0008896682411432266,
+            0.001758038648404181,
+            6.395397940650582e-05,
+            -0.02112744189798832,
+            -0.0009124181815423071,
+            0.0018809852190315723,
+            0.0010222955606877804,
+            -0.025201136246323586,
+            0.00889087375253439,
+            -0.00889087375253439,
+            -0.0014918158994987607,
+            -0.019113829359412193,
+            0.019795360043644905,
+            0.026300877332687378,
+            -0.004619691986590624,
+            0.00036206303047947586,
+            -0.013537672348320484,
+            -0.02049238048493862,
+            0.0001489637652412057,
+            -0.011314953677356243,
+            -0.011345932260155678,
+            -0.0022169100120663643,
+            -0.003214035416021943,
+            0.008805682882666588,
+            0.0004189379105810076,
+            -0.009433000348508358,
+            -0.004956584889441729,
+            -0.011764144524931908,
+            -0.017549408599734306,
+            -0.010401083156466484,
+            0.01933068037033081,
+            -0.0018016024259850383,
+            0.0020794421434402466,
+            -0.0032508226577192545,
+            0.0005271211848594248,
+            0.011748654767870903,
+            0.013475715182721615,
+            0.013344055972993374,
+            -0.023915521800518036,
+            -0.017890173941850662,
+            -0.013320821337401867,
+            0.0029487807769328356,
+            -0.007187047973275185,
+            0.004844287410378456,
+            0.026362836360931396,
+            -0.0009695350890979171,
+            -0.01091997604817152,
+            0.002327271504327655,
+            0.019051872193813324,
+            -0.031272951513528824,
+            -0.013390523381531239,
+            -0.025077221915125847,
+            0.02225816436111927,
+            -0.013351799920201302,
+            0.011516314931213856,
+            0.010191977955400944,
+            -0.025944624096155167,
+            0.006768836174160242,
+            -0.007686578668653965,
+            0.026037558913230896,
+            0.01634124107658863,
+            0.0005934348446317017,
+            -0.00944074522703886,
+            0.013622863218188286,
+            -0.0008722427301108837,
+            -0.013576395809650421,
+            0.002201420720666647,
+            0.014645159244537354,
+            -0.014691627584397793,
+            -0.009510447271168232,
+            -0.004507394507527351,
+            0.004313777666538954,
+            0.012763205915689468,
+            -0.0030339721124619246,
+            -0.004031097516417503,
+            -0.011624740436673164,
+            0.013344055972993374,
+            -0.003930416889488697,
+            0.012383717112243176,
+            0.007380664348602295,
+            -0.014056564308702946,
+            -0.019532041624188423,
+            0.033333033323287964,
+            0.00400399137288332,
+            0.00338054564781487,
+            -0.010997422970831394,
+            0.02740062028169632,
+            -0.009897680021822453,
+            0.010571465827524662,
+            -0.006408709101378918,
+            0.0031985461246222258,
+            0.012321759946644306,
+            -0.012104909867048264,
+            -0.01981084980070591,
+            0.021468207240104675,
+            -0.00364192808046937,
+            -0.0010193913476541638,
+            0.008310024626553059,
+            -0.020182594656944275,
+            -0.00663330452516675,
+            0.002180122770369053,
+            0.009541425853967667,
+            -0.011818356812000275,
+            0.006369986105710268,
+            0.017905661836266518,
+            0.003912991378456354,
+            -0.0007672057254239917,
+            0.019996721297502518,
+            -0.01672847382724285,
+            0.011717676185071468,
+            -0.010850274004042149,
+            0.0016825281782075763,
+            0.0006065039779059589,
+            -0.0048597767017781734,
+            0.0058743273839354515,
+            -0.014536733739078045,
+            0.011988739483058453,
+            -6.244135147426277e-05,
+            -0.013979118317365646,
+            0.007601387333124876,
+            -0.013359544798731804,
+            -0.016836898401379585,
+            0.041542377322912216,
+            -0.005301222205162048,
+            -0.0038665232714265585,
+            -0.0056962003000080585,
+            0.01350669376552105,
+            0.02823704481124878,
+            -0.005336073227226734,
+            -0.01153180468827486,
+            -0.010780571959912777,
+            -0.016573579981923103,
+            0.015907539054751396,
+            -0.01810702309012413,
+            0.002495717955753207,
+            -0.010532742366194725,
+            -0.007903429679572582,
+            0.004302160814404488,
+            -0.003516077296808362,
+            0.014722606167197227,
+            0.00950270239263773,
+            0.023357905447483063,
+            0.00797700323164463,
+            -0.00027348342700861394,
+            0.010393339209258556,
+            0.011663463898003101,
+            0.003243078012019396,
+            -0.010509508661925793,
+            -0.00011949774489039555,
+            -0.02894955314695835,
+            -0.008449427783489227,
+            -0.004863648675382137,
+            -0.001485039247199893,
+            -0.00428667152300477,
+            -0.01239920686930418,
+            0.006617815233767033,
+            -0.0022052929271012545,
+            0.013909416273236275,
+            -0.009254872798919678,
+            -0.017766257748007774,
+            0.03389064967632294,
+            0.004495777189731598,
+            0.006466794293373823,
+            -0.008286789990961552,
+            -0.03705047070980072,
+            -0.0032566310837864876,
+            -0.008116407319903374,
+            -0.013529927469789982,
+            0.004538373090326786,
+            0.01283290795981884,
+            -0.014831030741333961,
+            -0.028097640722990036,
+            0.0019390701781958342,
+            -0.006683644838631153,
+            0.02978597767651081,
+            -0.007798876613378525,
+            0.00419760774821043,
+            0.0037251831963658333,
+            0.0011558911064639688,
+            0.01239920686930418,
+            -0.00845717266201973,
+            -0.0022536972537636757,
+            0.016387708485126495,
+            -0.010858018882572651,
+            0.011051635257899761,
+            -0.010966443456709385,
+            -0.02554190158843994,
+            -0.01456771232187748,
+            0.021344292908906937,
+            -0.0004823473282158375,
+            -0.00024903935263864696,
+            0.014823286794126034,
+            0.02354377694427967,
+            0.011260741390287876,
+            0.00969631876796484,
+            0.006408709101378918,
+            -0.001035848748870194,
+            0.005975008010864258,
+            0.006095050368458033,
+            -0.0024163350462913513,
+            -0.013398268260061741,
+            0.017564896494150162,
+            0.009773765690624714,
+            -0.01810702309012413,
+            0.0008514289511367679,
+            0.0036612898111343384,
+            0.0168833676725626,
+            0.006935346405953169,
+            0.009316830895841122,
+            -0.0143121387809515,
+            -0.023760627955198288,
+            -0.00014932679187040776,
+            0.002873270306736231,
+            0.015721667557954788,
+            0.010579210706055164,
+            -0.014319883659482002,
+            0.0005963391158729792,
+            0.008712747134268284,
+            0.020213572308421135,
+            -0.024209819734096527,
+            0.019702425226569176,
+            -0.004673904739320278,
+            -0.0021084847394376993,
+            0.007946024648845196,
+            0.011632485315203667,
+            -0.011872569099068642,
+            -0.00650551775470376,
+            0.006249943748116493,
+            0.003157886676490307,
+            0.03172214329242706,
+            -0.0017260919557884336,
+            0.009665340185165405,
+            -0.010997422970831394,
+            -0.0002405686245765537,
+            0.0006728176376782358,
+            -0.0006820144481025636,
+            -0.002884887158870697,
+            -0.011020656675100327,
+            0.0018974426202476025,
+            -0.020306508988142014,
+            0.0024473138619214296,
+            0.00463130883872509,
+            -0.005684582982212305,
+            -0.0032972905319184065,
+            -0.019222255796194077,
+            -0.0022672503255307674,
+            0.003293418325483799,
+            0.014335372485220432,
+            -0.031691163778305054,
+            0.004972074180841446,
+            0.010672146454453468,
+            -0.002309845993295312,
+            -0.010416572913527489,
+            0.011423379182815552,
+            -0.0026060794480144978,
+            0.011988739483058453,
+            -0.0038142469711601734,
+            -0.0004905760288238525,
+            0.025681305676698685,
+            -0.005308967083692551,
+            0.014707116410136223,
+            -0.01026167906820774,
+            -0.002993312431499362,
+            -0.028252532705664635,
+            0.005235392600297928,
+            -0.007403898518532515,
+            -0.0012081675231456757,
+            -0.004402841441333294,
+            -0.01130720879882574,
+            -0.0026990154292434454,
+            0.002122037811204791,
+            0.008000237867236137,
+            0.012910354882478714,
+            -0.0007725302129983902,
+            -0.010075807571411133,
+            0.004832670092582703,
+            0.0075355577282607555,
+            -0.015520306304097176,
+            -0.0070437719114124775,
+            -0.020306508988142014,
+            0.029662061482667923,
+            -0.013731288723647594,
+            -0.0011123273288831115,
+            -0.020864123478531837,
+            0.010160998441278934,
+            -0.0063119009137153625,
+            -0.002782270545139909,
+            0.004708755761384964,
+            -0.015109838917851448,
+            0.01344473659992218,
+            0.008751469664275646,
+            0.01450575515627861,
+            -0.02151467651128769,
+            -0.01762685552239418,
+            -0.0044725434854626656,
+            -0.002735802438110113,
+            -0.010648912750184536,
+            -0.0005334137240424752,
+            0.002013612538576126,
+            0.021251358091831207,
+            -0.00032503384863957763,
+            0.0042053526267409325,
+            -0.003214035416021943,
+            0.017224133014678955,
+            -0.010137764737010002,
+            -0.013057502917945385,
+            0.0024260159116238356,
+            0.005994369741529226,
+            0.016186347231268883,
+            0.03158273920416832,
+            -0.020693741738796234,
+            -0.00272999401204288,
+            0.0032740565948188305,
+            0.010656657628715038,
+            -0.004085310269147158,
+            6.304640555754304e-05,
+            -0.027803342789411545,
+            -0.04464024305343628,
+            -0.002422143705189228,
+            0.013057502917945385,
+            -0.015636475756764412,
+            0.012670270167291164,
+            0.008743725717067719,
+            0.010021595284342766,
+            -0.005196669604629278,
+            0.02620794251561165,
+            -0.004793947096914053,
+            0.014304393902420998,
+            0.011733165942132473,
+            0.011911292560398579,
+            0.0009874446550384164,
+            0.0007459079497493804,
+            -0.018354853615164757,
+            -0.008255811408162117,
+            -0.0015954007394611835,
+            0.01765783317387104,
+            -0.027478067204356194,
+            0.008526874706149101,
+            -0.011516314931213856,
+            0.013839714229106903,
+            0.0003666614356916398,
+            0.005661349277943373,
+            -0.01771979033946991,
+            0.012445674277842045,
+            -0.0009337160154245794,
+            -0.025325050577521324,
+            0.004805563949048519,
+            0.011276230216026306,
+            -0.010362359695136547,
+            0.006877261213958263,
+            0.019950253888964653,
+            0.0023079097736626863,
+            0.008495896123349667,
+            -0.014126266352832317,
+            0.010703125037252903,
+            0.009649851359426975,
+            -0.021080974489450455,
+            -0.01233724970370531,
+            -0.007059261202812195,
+            -0.006335135083645582,
+            -0.0035470561124384403,
+            -0.00013105422840453684,
+            0.010176488198339939,
+            -0.016000475734472275,
+            0.010099041275680065,
+            -0.006559730041772127,
+            0.006400964688509703,
+            0.00944074522703886,
+            0.001115231541916728,
+            -0.013576395809650421,
+            0.012716737575829029,
+            -0.0218709297478199,
+            0.008426194079220295,
+            -0.04287445917725563,
+            -0.005847221240401268,
+            -0.015078860335052013,
+            0.005204414017498493,
+            -0.003798757679760456,
+            -0.004453181754797697,
+            -0.006203475873917341,
+            -0.005061137955635786,
+            0.00920066051185131,
+            0.021747015416622162,
+            0.012135888449847698,
+            -0.016620049253106117,
+            -0.007551047019660473,
+            0.005425137002021074,
+            0.0004854935978073627,
+            -0.013336311094462872,
+            0.009316830895841122,
+            -0.01576039008796215,
+            0.0009259713697247207,
+            0.006339007057249546,
+            -0.01072635967284441,
+            0.013452480547130108,
+            0.004731989465653896,
+            -0.003401843598112464,
+            0.013390523381531239,
+            -0.0050417762249708176,
+            -0.0057891360484063625,
+            -0.007272239308804274,
+            -0.013739033602178097,
+            -0.003818119177594781,
+            -0.003072695340961218,
+            0.012747716158628464,
+            0.001529571134597063,
+            -0.0003243077953811735,
+            -0.00782985519617796,
+            0.009154192171990871,
+            0.002220782218500972,
+            -0.0013146566925570369,
+            0.005773646757006645,
+            -0.02300165221095085,
+            0.0001287550403503701,
+            0.004755223635584116,
+            -0.0009487213101238012,
+            -0.006246071308851242,
+            -0.012151377275586128,
+            -0.011903548613190651,
+            -0.038165703415870667,
+            0.009363298304378986,
+            0.0019197085639461875,
+            -0.0076207490637898445,
+            0.0064822835847735405,
+            0.017255110666155815,
+            0.01456771232187748,
+            0.016743963584303856,
+            -0.020693741738796234,
+            -0.02303262986242771,
+            0.005556796211749315,
+            -0.009402021765708923,
+            0.02261441759765148,
+            0.004046586807817221,
+            -0.005847221240401268,
+            0.0013243375578895211,
+            -0.013258864171802998,
+            0.008929597213864326,
+            0.005723306443542242,
+            -0.001592496526427567,
+            0.005425137002021074,
+            -0.013692565262317657,
+            -0.010904486291110516,
+            0.01048627495765686,
+            0.032496608793735504,
+            -0.0004261985304765403,
+            0.0024105266202241182,
+            0.023295948281884193,
+            0.015466093085706234,
+            0.016356728971004486,
+            -0.021111954003572464,
+            0.011167804710566998,
+            0.010060317814350128,
+            -0.0018645278178155422,
+            -0.014892988838255405,
+            0.00513083953410387,
+            -0.012716737575829029,
+            -0.010664401575922966,
+            0.014993669465184212,
+            0.007853088900446892,
+            0.017735280096530914,
+            -0.02369867078959942,
+            0.00789568480104208,
+            0.00042765066609717906,
+            0.00013710475468542427,
+            0.011524059809744358,
+            0.012956822291016579,
+            0.03370477631688118,
+            0.027152791619300842,
+            0.020662762224674225,
+            -0.009285852313041687,
+            0.014366351068019867,
+            -0.009192915633320808,
+            -0.011090358719229698,
+            -0.019470084458589554,
+            -0.015249243006110191,
+            -0.011136826127767563,
+            0.017998598515987396,
+            0.008790193125605583,
+            -0.009355553425848484,
+            -0.005947901867330074,
+            0.01586107164621353,
+            -0.003909118939191103,
+            -0.004635181277990341,
+            0.01646515540778637,
+            -0.026269899681210518,
+            0.016836898401379585,
+            -0.000745423894841224,
+            -0.014668392948806286,
+            -0.014691627584397793,
+            -0.004182118456810713,
+            -0.012670270167291164,
+            0.0003964299685321748,
+            -0.020817656069993973,
+            0.013181418180465698,
+            0.024597052484750748,
+            -0.0030920570716261864,
+            -0.004100799560546875,
+            -0.011299464851617813,
+            -0.0005111477803438902,
+            0.018385831266641617,
+            0.0034773540683090687,
+            0.009192915633320808,
+            -0.0023872926831245422,
+            0.013924905098974705,
+            0.00524313747882843,
+            0.0027958236169070005,
+            -0.0007565568666905165,
+            -0.016263794153928757,
+            0.012197845615446568,
+            0.0026835259050130844,
+            0.012135888449847698,
+            -0.022242674604058266,
+            0.007938280701637268,
+            -0.011764144524931908,
+            -0.0023698671720921993,
+            -0.02228914201259613,
+            0.0189744271337986,
+            -0.01610890030860901,
+            0.001046497724018991,
+            0.003841353114694357,
+            0.0006224773242138326,
+            0.009580149315297604,
+            0.01610890030860901,
+            -0.023388884961605072,
+            -0.0003460896550677717,
+            -0.014908477663993835,
+            -0.021917399019002914,
+            -0.01607792265713215,
+            0.004728117026388645,
+            -0.019501063972711563,
+            0.005444498732686043,
+            -0.0025557391345500946,
+            -0.015125328674912453,
+            0.018478767946362495,
+            0.014691627584397793,
+            -0.004379607271403074,
+            0.002412462839856744,
+            0.005425137002021074,
+            -0.0014405074762180448,
+            0.01971791312098503,
+            0.00013274837692733854,
+            -0.0010048701660707593,
+            -0.009425255469977856,
+            -0.0168833676725626,
+            -0.0065752193331718445,
+            -0.035005878657102585,
+            0.0260530486702919,
+            -0.004670032300055027,
+            -0.016511622816324234,
+            0.014319883659482002,
+            0.009115469641983509,
+            -0.014629669487476349,
+            -0.007059261202812195,
+            0.016294771805405617,
+            -0.024287264794111252,
+            0.02244403585791588,
+            -0.009611127898097038,
+            -0.008774704299867153,
+            0.00784921646118164,
+            0.05154848098754883,
+            0.009146448224782944,
+            0.003492843359708786,
+            0.011036145500838757,
+            0.005839476361870766,
+            -0.0013165927957743406,
+            -0.023683181032538414,
+            0.005568413063883781,
+            -0.007190920412540436,
+            -0.003601268632337451,
+            0.002536377403885126,
+            -0.004801691509783268,
+            0.031149037182331085,
+            -0.004379607271403074,
+            -0.005289605353027582,
+            0.004484160337597132,
+            -0.02196386642754078,
+            -0.0001332324172835797,
+            0.023203013464808464,
+            -2.3128106477088295e-05,
+            -0.023590246215462685,
+            -0.006571347359567881,
+            -0.015071115456521511,
+            -0.013770012184977531,
+            -0.018989915028214455,
+            0.005231520161032677,
+            -0.012755461037158966,
+            -0.0033940989524126053,
+            0.0016660707769915462,
+            -0.005940156988799572,
+            -0.002582845278084278,
+            0.009603383019566536,
+            0.05362405255436897,
+            0.014103032648563385,
+            -0.006335135083645582,
+            -0.005854965653270483,
+            0.009192915633320808,
+            -0.010524998418986797,
+            -0.019795360043644905,
+            -0.009820234030485153,
+            -0.029507169499993324,
+            -0.012747716158628464,
+            -0.0143121387809515,
+            -0.010191977955400944,
+            -0.010277168825268745,
+            0.00963436160236597,
+            -0.018122512847185135,
+            -0.007628493942320347,
+            -0.020337486639618874,
+            0.002941036131232977,
+            -0.011268485337495804,
+            -0.014714861288666725,
+            -0.008054450154304504,
+            -0.04807887226343155,
+            0.005796880926936865,
+            0.019593998789787292,
+            0.01263929158449173,
+            0.002515079453587532,
+            0.004383479710668325,
+            -0.01993476413190365,
+            -0.030792783945798874,
+            0.013839714229106903,
+            0.011283975094556808,
+            0.011733165942132473,
+            0.03283737599849701,
+            -0.011709931306540966,
+            -0.00813964195549488,
+            -0.004693266469985247,
+            -0.0017251238459721208,
+            0.012066186405718327,
+            0.00969631876796484,
+            -0.004917861428111792,
+            -0.012375972233712673,
+            0.016217326745390892,
+            -0.01981084980070591,
+            -0.023683181032538414,
+            -0.01672847382724285,
+            0.006157007534056902,
+            0.00813964195549488,
+            -0.008054450154304504,
+            0.002973950933665037,
+            0.0056962003000080585,
+            -0.007159941829741001,
+            -0.00022423222253564745,
+            0.004890755284577608,
+            -0.023048119619488716,
+            0.0008490087930113077,
+            -0.002257569460198283,
+            0.0061376458033919334,
+            0.021266845986247063,
+            -0.010168743319809437,
+            -0.0014075926737859845,
+            -0.0053902859799563885,
+            -0.0028984404634684324,
+            -0.006764963734894991,
+            0.02620794251561165,
+            0.009154192171990871,
+            -0.009882191196084023,
+            -0.01011453103274107,
+            -0.01129171997308731,
+            -0.0011278166202828288,
+            -0.017255110666155815,
+            -0.016186347231268883,
+            0.0015566773945465684,
+            -0.016712984070181847,
+            0.013971373438835144,
+            -0.00876695942133665,
+            0.00400399137288332,
+            -0.014110777527093887,
+            0.02019808255136013,
+            0.01929970271885395,
+            -0.009711808525025845,
+            0.023776117712259293,
+            -0.007709812838584185,
+            -0.038382552564144135,
+            -0.00800798274576664,
+            0.01843230053782463,
+            -0.006118284538388252,
+            0.010037084110081196,
+            -0.011314953677356243,
+            0.008209343999624252,
+            0.00015319912927225232,
+            0.0018122512847185135,
+            -0.0003112386621069163,
+            0.027555514127016068,
+            0.015535795129835606,
+            -0.005560668651014566,
+            -0.005494839046150446,
+            0.004255692940205336,
+            0.009541425853967667,
+            0.017580386251211166,
+            0.012151377275586128,
+            -0.021019017323851585,
+            -0.014257925562560558,
+            -0.016263794153928757,
+            0.011012911796569824,
+            0.015125328674912453,
+            -0.0040891822427511215,
+            0.01691434532403946,
+            0.014126266352832317,
+            -0.019098341464996338,
+            0.0019884423818439245,
+            -0.008193854242563248,
+            -0.00158184755127877,
+            -0.03029712475836277,
+            0.011043890379369259,
+            0.0074387495405972,
+            -0.0072218989953398705,
+            0.036616768687963486,
+            0.009858956560492516,
+            0.037577107548713684,
+            0.018680129200220108,
+            0.0008974128868430853,
+            0.009045767597854137,
+            -0.018370341509580612,
+            0.013065247796475887,
+            0.01500141341239214,
+            -0.012724482454359531,
+            -0.005734923295676708,
+            0.0013446672819554806,
+            0.0007551047019660473,
+            0.012631546705961227,
+            0.02004319056868553,
+            -0.01795213110744953,
+            0.009773765690624714,
+            -0.015032391995191574,
+            -0.005250881891697645,
+            0.006555858068168163,
+            -0.013932649977505207,
+            -0.008495896123349667,
+            -0.00963436160236597,
+            0.019408127292990685,
+            -0.012600568123161793,
+            0.016573579981923103,
+            -0.009595638141036034,
+            0.015520306304097176,
+            -0.029724018648266792,
+            -0.003669034456834197,
+            -0.006679772399365902,
+            -0.019593998789787292,
+            -0.012887120246887207,
+            0.009820234030485153,
+            0.010633422993123531,
+            -0.009301341138780117,
+            -0.011562783271074295,
+            0.020368466153740883,
+            -0.004185990896075964,
+            0.013529927469789982,
+            0.021189400926232338,
+            0.00737291993573308,
+            0.024132372811436653,
+            0.0005605200421996415,
+            0.014164989814162254,
+            0.005463860463351011,
+            0.010362359695136547,
+            -0.004677776712924242,
+            0.005161818582564592,
+            0.00932457484304905,
+            0.014203713275492191,
+            0.017766257748007774,
+            0.013196907006204128,
+            -0.00598275288939476,
+            0.007419387809932232,
+            0.02016710489988327,
+            0.0016254113288596272,
+            -0.030312614515423775,
+            -0.009719553403556347,
+            0.0001736498816171661,
+            0.00924712885171175,
+            0.009185170754790306,
+            3.817877222900279e-05,
+            -0.01093546487390995,
+            0.011144571006298065,
+            0.00969631876796484,
+            0.002286612056195736,
+            0.006280922330915928,
+            -0.01456771232187748,
+            -0.020957060158252716,
+            0.006106667220592499,
+            0.01798310875892639,
+            0.013917161151766777,
+            -0.014722606167197227,
+            0.022784801200032234,
+            0.006145390681922436,
+            -0.021406250074505806,
+            -0.01840132102370262,
+            -0.0009559819591231644,
+            0.0004542729293461889,
+            -0.002265314105898142,
+            0.03720536455512047,
+            -0.0011539548868313432,
+            -0.005955646280199289,
+            -0.0054754773154854774,
+            0.02775687538087368,
+            0.02049238048493862,
+            0.009177426807582378
+          ],
+          "embedding": [
+            0.0008215984936325879,
+            -0.019349418284688353,
+            -0.02322660633029795,
+            0.009747820845194852,
+            0.007353771220903641,
+            0.015252908783199839,
+            -0.012176175829980202,
+            0.051898051094966605,
+            -0.0068130353886146165,
+            0.008189212816890595,
+            0.008102287831931905,
+            0.01682801136011614,
+            0.02005557996049711,
+            -0.020779879580497183,
+            0.002332165268329781,
+            0.02337485065616377,
+            0.006074992895540845,
+            0.02576575146535689,
+            0.0007517286429767504,
+            -0.03763909942314134,
+            0.0011204729720314833,
+            0.00041212613589632806,
+            -0.06089176800314795,
+            0.012852035873395388,
+            -0.008901127092058543,
+            0.01702347055484216,
+            0.005803446395466281,
+            0.017889570275475296,
+            -0.05375047382413552,
+            0.021623778774252724,
+            0.014784317465883424,
+            -0.0018881880857885708,
+            0.0004735328554290247,
+            0.012393802850407656,
+            -0.019234442968055952,
+            -0.019566283611638122,
+            0.04407789373296535,
+            -0.0053565873209102995,
+            -0.026240866282480922,
+            0.048107902270426675,
+            -0.0054681314939387235,
+            -0.0009441397948041965,
+            -0.015205079626883548,
+            0.006885981841893229,
+            0.01403724256866746,
+            -0.026086548246792754,
+            0.0034778946051893067,
+            0.00044587533948775473,
+            0.032851328435891634,
+            -0.03882097333255334,
+            -0.011126344604254894,
+            -0.01648943336384904,
+            0.035810456555802166,
+            -0.019384589526165754,
+            -0.019430702768170285,
+            0.014297440272239852,
+            -0.007883453204663151,
+            0.050996900793086844,
+            0.06007149434153715,
+            0.03406832053483379,
+            0.025193439934671852,
+            0.00027785129663604897,
+            -0.008014391056168083,
+            -0.018806730908490114,
+            0.027193788903098558,
+            -0.006195006670328595,
+            -0.020466000344352826,
+            0.0068939884219589185,
+            0.017252276167086602,
+            -0.045073444648704136,
+            -0.049242269917908754,
+            0.023021140866008313,
+            -0.015676872472071106,
+            -0.024497517908898602,
+            0.005873633718973869,
+            -0.022828585938203683,
+            0.011748341004940436,
+            0.04624205727927451,
+            -0.015442165499674641,
+            0.018565551071491767,
+            0.015246505395507096,
+            0.004119269796224144,
+            0.004862735250819252,
+            -0.000145323321960046,
+            0.05155838952887077,
+            0.0027052423345323936,
+            -0.016218063670586664,
+            -0.07462470920944148,
+            0.043695974746295135,
+            -0.027608273720377708,
+            -0.03860912774695494,
+            0.002823110565150435,
+            0.026471469849508766,
+            -0.0026063577189074103,
+            -0.029230652014151735,
+            0.05684164470477843,
+            0.00022182868300572677,
+            0.015590845552956364,
+            -0.024939617549529242,
+            0.01035803286203917,
+            -0.010225298012384327,
+            0.019571847648759157,
+            0.01841613041441054,
+            0.0156981470538354,
+            0.021764297694793343,
+            -0.018831230007033416,
+            -0.014419529638789993,
+            -0.0029965475899652273,
+            0.028881185012095884,
+            0.09163526460163937,
+            0.022644302207811818,
+            -0.017933909936330133,
+            -0.04663928709016869,
+            0.004962951444105128,
+            -0.041333041174526026,
+            -0.027545628007617527,
+            -0.037142394579894954,
+            -0.013358143162788402,
+            0.02719686432396677,
+            -0.0034971800937504205,
+            0.04413635817371732,
+            0.04903857110328917,
+            0.03186067155630788,
+            0.007831051050269414,
+            -0.025257538623313574,
+            0.023517513773109414,
+            -0.0016527403209505964,
+            -0.0016101860707429347,
+            0.014448515185903624,
+            0.009905018312477684,
+            0.0024114582235163955,
+            0.00172043851384822,
+            -0.03818316221102173,
+            -0.02304537008110578,
+            0.01024969233570089,
+            -0.01901116356327339,
+            0.019791121460848986,
+            0.07985067340821365,
+            -0.051903191272236665,
+            -0.025924646976556942,
+            0.014766511532849252,
+            -0.002073086076894999,
+            0.030959269926800852,
+            0.006394417986617089,
+            -0.028177586693713442,
+            0.010725285336674526,
+            0.0009614052276924192,
+            0.07500253788978564,
+            -0.00036794182149960157,
+            0.004058458332826519,
+            -0.013218269166951234,
+            0.027613539268683245,
+            -0.012995170442480582,
+            0.023388707018488407,
+            0.026389423313105165,
+            -0.003068639956061266,
+            -0.02661967649632026,
+            0.026145274570635913,
+            -0.03300591322174331,
+            -0.004556072066733024,
+            0.020045722084112896,
+            0.0033437343914784865,
+            -0.010224220905483085,
+            -0.0025025498684079703,
+            0.03460103245395426,
+            0.04514362357278504,
+            0.03797031988058962,
+            -0.004586662475576956,
+            -0.06351723722596923,
+            0.06037983484369874,
+            0.019415753124363902,
+            0.028850618647994476,
+            -0.0034810237045849006,
+            0.02957938154808314,
+            0.02642652523231673,
+            0.013187653724133275,
+            -0.007856320804109569,
+            -0.007065882659482447,
+            -0.016672784625197085,
+            0.029020161154209707,
+            -0.013243058921970134,
+            -0.014900932834241732,
+            0.006001769901896033,
+            0.008648456186944271,
+            0.03484763413464497,
+            0.012279368984306564,
+            0.02379643508687133,
+            0.04933044959399941,
+            -0.021047723935647383,
+            -0.007264823701706426,
+            0.0007058774795716792,
+            -0.010512071832195085,
+            0.027489671668749157,
+            0.009731157570361245,
+            -0.03195919640758579,
+            -0.0033626031972691915,
+            -0.04282419484651207,
+            0.01873977910485268,
+            -0.014398285316695653,
+            0.012854631447494207,
+            -0.0017372959978092432,
+            -0.037331867825735776,
+            0.001088209292238142,
+            -0.012345684323522151,
+            0.037096063326414734,
+            0.005050219690024652,
+            0.0008906051515353004,
+            -0.003697709449630797,
+            0.006548479142737107,
+            -0.010727663712299397,
+            -0.007486226350610755,
+            -0.02704618481069485,
+            -0.019752552016312926,
+            -0.014632581296303486,
+            -0.0037501138119724717,
+            0.014588411795703245,
+            -0.007119137071719376,
+            0.016440258608998535,
+            -0.03686645352111013,
+            -0.023546402448713415,
+            0.005628703432152666,
+            0.024988145751166143,
+            0.011971199595808573,
+            -0.010868778329190568,
+            -0.03270046433624425,
+            -0.021950559285357267,
+            -0.011889914022657738,
+            -0.005123971523701009,
+            0.02619713302280345,
+            -0.021990810305904974,
+            0.01165257474485436,
+            -0.009117901804315863,
+            -0.007492082755249399,
+            0.010725902070291232,
+            0.01862811996181951,
+            -0.026694281868322685,
+            -0.01623776415010199,
+            -0.009403997538684658,
+            -0.02207329027097839,
+            0.025939540497196416,
+            0.014178803271471762,
+            -0.001141821318145744,
+            -0.006281187017652108,
+            -0.035993365198554964,
+            0.0026192776111449633,
+            -0.015254976771874396,
+            -0.0116872453009606,
+            -0.015780016491993135,
+            0.021421328283124968,
+            0.02889432250124136,
+            -0.01871009416296692,
+            -0.026540472640514536,
+            0.015703370682110186,
+            -0.0002849235245164438,
+            -0.02594930889995505,
+            0.04491828744809679,
+            0.044063522197410734,
+            -0.037441864491547405,
+            -0.007005339187980476,
+            -0.030381794393931814,
+            0.06901083845539985,
+            -0.017604082400603037,
+            -0.04060591381445593,
+            0.06044850935418443,
+            -0.01756490171206357,
+            0.025539717592705295,
+            0.03452018789549939,
+            0.0016217577522523927,
+            0.00018825054264059277,
+            -0.002563606092068712,
+            0.04770277039047993,
+            -0.0032010143255574255,
+            -0.03425328601475849,
+            -0.02105909767072799,
+            -0.00840881022563103,
+            -0.028699447677549002,
+            -0.015925160861589006,
+            -0.0055750742281166,
+            0.017513987082479483,
+            0.010399718005952185,
+            0.02267201252273823,
+            0.01258511878601025,
+            0.0015081978484714903,
+            0.01874981572091931,
+            0.011158889421057756,
+            -0.002607810686437211,
+            -0.003732023411827874,
+            -0.007787189972165309,
+            -0.05524012485122728,
+            -0.025110689689279583,
+            -0.03946383236703947,
+            0.004443572529325475,
+            -0.018745077598477586,
+            0.005926160888274727,
+            0.039430331582190524,
+            -0.0049500324516682,
+            0.004352795787899792,
+            0.015566139991780173,
+            0.013148156356408315,
+            -0.015279560337042242,
+            -0.03316846614871165,
+            0.0001592209771257332,
+            0.018362411161743347,
+            -0.012706582418607377,
+            -0.03424477956681259,
+            -0.001991022841226589,
+            0.0033467286047492175,
+            0.03357019284647071,
+            -0.024359110796680374,
+            0.00359117029951049,
+            0.005061818038334802,
+            0.011359582931760315,
+            -0.026793992746244076,
+            -0.018170136079817746,
+            -0.017421603453826397,
+            0.03996608926471779,
+            0.02587013058343799,
+            0.015763410145648683,
+            -0.04512536449983358,
+            -0.02293473890636517,
+            0.048575083783480195,
+            -0.009589860756342604,
+            -0.06486262416944208,
+            0.03447666009581266,
+            0.016105654870253654,
+            0.003526582643722421,
+            -0.01354754623151814,
+            0.05401575248587799,
+            -0.03563190295559527,
+            -0.017365166993914202,
+            -0.0046288709698227735,
+            -0.033657212157212645,
+            -0.007686252059880836,
+            -0.001385273642139782,
+            0.048229062789530236,
+            -0.029083855744728694,
+            -0.030465559177859865,
+            0.0024368661249980416,
+            -0.024144031154493057,
+            0.021693278579520123,
+            -0.016405088055634124,
+            -0.018439935250344958,
+            0.006948603359995389,
+            -0.033943037395199695,
+            -0.003007796423804453,
+            -0.016783195426536633,
+            -0.02270589491206155,
+            0.020874438753828814,
+            0.016724666903417947,
+            0.027033184180344852,
+            0.0019847647852695785,
+            0.017749524526371133,
+            0.0204661893602285,
+            -0.005604929968008821,
+            0.028462234633072944,
+            -0.020790459496727066,
+            -0.04183816355941877,
+            0.03946487892663695,
+            0.01192861758684541,
+            -0.0018024841275380242,
+            0.01077082591611702,
+            -0.01475239020295694,
+            -0.03952356245715099,
+            0.03683950018557526,
+            -0.056767672022551764,
+            -0.007507600090017693,
+            0.009530544352246083,
+            -0.00788856051420074,
+            -0.011204694027562521,
+            -0.010587701466318598,
+            -0.028604940377315504,
+            0.023749537635431462,
+            -0.013672419168235691,
+            0.01953449229707597,
+            -0.011449939491554354,
+            -0.012238180529977605,
+            0.05795536247491174,
+            -0.021518581774137322,
+            -0.040672955795925836,
+            0.026066659238628543,
+            -0.014061106927896723,
+            0.03328357230532424,
+            0.03957107406164427,
+            0.030670947051333917,
+            -0.004278577327978564,
+            0.025885946540364826,
+            0.0319106101668716,
+            -0.003462160036198642,
+            -0.04686868330851868,
+            0.00555303933159587,
+            0.02156143971867743,
+            -0.03739529489075782,
+            -0.01025626126223168,
+            -0.02141970191890338,
+            0.022134910980548407,
+            0.017807484142010496,
+            -0.007232643256393455,
+            0.011318041192005274,
+            -0.010489012862455913,
+            -0.013003217163997554,
+            -0.00847279880392288,
+            0.02357274819910594,
+            -0.0016485066139744577,
+            -0.005873113634992993,
+            0.006424249429163587,
+            0.004008703814301658,
+            0.0054639858216457724,
+            0.010198079500408441,
+            0.017694720520798957,
+            -0.017087021465208255,
+            -0.0025558869389290554,
+            -0.016503579212015177,
+            -0.0420082404922411,
+            -0.003563173424359332,
+            0.022924161700566468,
+            0.012345933179990547,
+            -0.00729908668365632,
+            -0.004234683369642856,
+            -0.0022962545569187934,
+            -0.020666205489143392,
+            -0.014796149043465308,
+            0.024562653223873056,
+            0.030217494971086438,
+            0.007810961433007407,
+            0.007781263640449456,
+            -0.0030348559388780655,
+            0.007310876971216526,
+            0.016099591640649783,
+            0.02311929597780174,
+            -0.04160162038353416,
+            0.027152100845897007,
+            -0.00025396315502635146,
+            0.023717123370023598,
+            -0.04186918759097667,
+            -0.017988531857836703,
+            -0.03106663560912813,
+            -0.023595079699181436,
+            0.02437134274097885,
+            -0.02317559303912261,
+            -0.012327263775257306,
+            0.000989566600374836,
+            0.00954921216572729,
+            -0.005155079395685017,
+            -0.017508404061379164,
+            -0.04582321461886958,
+            -0.0014367264231457263,
+            -0.030417229142190933,
+            -0.017829710557332023,
+            -0.001056472801554354,
+            0.005016912084754242,
+            -0.06276644565618253,
+            0.0109965681725148,
+            -0.07270669271977365,
+            -0.004542796125637244,
+            0.040139998232966566,
+            -0.017890441783630643,
+            0.055909102081208445,
+            -0.05029104308417623,
+            0.01673962668706085,
+            -0.04507293803055318,
+            0.015385506084493484,
+            0.025677372277539627,
+            0.030235171586316813,
+            -0.022189978375852563,
+            0.008461188938299745,
+            0.008033707483259569,
+            -0.045089505755429415,
+            -0.04144698743458133,
+            -0.013640256640978024,
+            0.01036782197518379,
+            0.0012696958402094232,
+            -0.016090728932978384,
+            0.023838437877040163,
+            0.011946482002767135,
+            0.03342604921859589,
+            0.04211801335157251,
+            0.002708015395637621,
+            -0.047156646610976634,
+            -0.013104464812562618,
+            -0.003933116891120681,
+            0.03155321708116329,
+            -0.018866109579446184,
+            0.015376812010630058,
+            -0.01955557376085726,
+            0.016255217383093355,
+            0.017868332102274922,
+            -0.020325179506828335,
+            0.013434364256590097,
+            0.004176904701252156,
+            -0.03877060944639854,
+            0.0006475542749369515,
+            0.00042468300294954125,
+            -0.02116992978038693,
+            0.0032654387271108293,
+            -0.00575003786159731,
+            -0.05149954071159492,
+            -0.031382926943128234,
+            -0.015730563640253065,
+            0.0382312502769352,
+            0.004297939848910064,
+            0.019724983545557365,
+            -0.017907869879607064,
+            -0.005165628493894463,
+            0.03698976707579373,
+            0.008501001373266799,
+            -0.01725527821298066,
+            0.006879620077191753,
+            0.02455792200929138,
+            -0.03224193972301503,
+            -0.039181328884413266,
+            -0.0008206986106407542,
+            -0.023306758893533952,
+            0.01672946960876128,
+            -0.02658587698869307,
+            0.005873620623736886,
+            0.0025331170979801232,
+            -0.03490809296591325,
+            0.02456764314036355,
+            0.003291113462353163,
+            -0.004303423672882112,
+            -0.020359544091059775,
+            -0.016892415165120203,
+            0.017174797487220006,
+            0.04391489511478065,
+            0.007327321684948843,
+            -0.003943713832544148,
+            0.00862684231313155,
+            -0.006361172785753631,
+            0.05826688726534012,
+            -0.02284956969636637,
+            -0.026680023999604643,
+            0.027527436347931256,
+            -0.022597669279069723,
+            -0.006277863878949772,
+            -0.017343446085471857,
+            -0.012065192152203348,
+            -0.0004183712490442143,
+            0.019339311388446943,
+            -0.019423562920897294,
+            -0.010212527447243049,
+            0.004783006556748757,
+            0.012824133870627983,
+            -0.004855945108275706,
+            0.006239949458562236,
+            0.006847581690471813,
+            0.03094005096378827,
+            0.026993131409761086,
+            -0.011331881651539968,
+            0.03297014291876675,
+            0.010265600643742144,
+            4.666113318251278e-05,
+            0.001970879281172325,
+            0.002523664335535017,
+            0.0034107811520368574,
+            -0.03143890864030425,
+            -0.01624584278940617,
+            0.013468729048885906,
+            0.005233309338223403,
+            -0.004550544967856448,
+            -0.023783653541634683,
+            -0.009402494239255264,
+            0.03334923712760375,
+            -0.003260424721959948,
+            0.02748395114281811,
+            -0.0058374897745283065,
+            -0.01985840930273332,
+            0.017595418271610163,
+            0.02881743246631328,
+            -0.05928552440605384,
+            0.017062540894682354,
+            -0.0011508547522925793,
+            0.008953761538519997,
+            0.01610561914042553,
+            -0.008894160513648953,
+            0.013301991014533684,
+            0.04080863883897929,
+            0.010738051773755976,
+            -0.023345002072886145,
+            0.004215726310838674,
+            0.01541749448570608,
+            0.01291813299324752,
+            -0.022747942296528025,
+            0.0008640791861357522,
+            0.011008522066277018,
+            0.005893793007012561,
+            0.0026106450193140156,
+            -0.006278394080583205,
+            -0.025659188287146268,
+            0.024180550333596625,
+            -0.0044491292740074925,
+            0.026967973346698617,
+            0.04390909273429254,
+            0.009596855239692042,
+            -0.03686020608700941,
+            -0.013109951664921729,
+            0.0002799680359670127,
+            -0.0038537577602593366,
+            -0.0037552094562213916,
+            0.011105295862811114,
+            -0.04185615640149602,
+            0.019950890919369135,
+            0.006453241355231516,
+            0.015314645427201533,
+            -0.020031193483734364,
+            -0.024530173488182337,
+            0.0031949828520724337,
+            -0.01492645564652531,
+            0.012875827757040941,
+            0.005057170413681063,
+            -0.004319834710426384,
+            0.005386862713494232,
+            0.00030848054013413157,
+            0.008384941028326983,
+            -0.010811526993301901,
+            0.03175068297908541,
+            0.004680987487229685,
+            0.009264547472281849,
+            -0.02856314241700808,
+            -0.021083721348429348,
+            0.020113396771629096,
+            0.03572366771051171,
+            -0.01708529918264109,
+            0.03346921724975398,
+            -0.009042046660366718,
+            0.0033219155387035227,
+            0.0151720168318272,
+            -0.015805318210244414,
+            -0.008057236707694303,
+            -0.0005483009439015931,
+            -0.002917527004423299,
+            -0.014538511124345527,
+            0.007302979537905532,
+            -0.0208529958115422,
+            -0.02965227242147972,
+            0.020064178921373875,
+            -0.00011892622932349096,
+            0.016455188521316146,
+            0.010335044657747514,
+            -0.007816155992852236,
+            -0.0052262330850905165,
+            0.006959344748883874,
+            0.04432147882994085,
+            0.026734484553994495,
+            0.05547661378166294,
+            -0.005787997952888261,
+            -0.004559444854138438,
+            -0.0082295581123363,
+            -0.006588172755707144,
+            0.04524009609473585,
+            0.015915003261638767,
+            -0.004910285095602864,
+            0.017495607570191165,
+            0.01557886571266685,
+            0.03858259260136037,
+            0.0027648525902558316,
+            -0.016912796336835408,
+            0.04014755669075221,
+            -0.004077405782937414,
+            0.008616827127381418,
+            0.011102692199778469,
+            0.0024185742218798562,
+            0.00807141794538637,
+            0.02996274720815659,
+            -0.012429186019907761,
+            0.01680942269506771,
+            -0.04571916592313758,
+            -0.02775530246832996,
+            0.014370280587419414,
+            -0.004931905540824582,
+            0.03419959665162407,
+            0.01668521021731861,
+            -0.023105305446763272,
+            -0.024761385385514877,
+            0.021529428247380984,
+            -0.002603042335561961,
+            -0.024135326707477325,
+            -0.015906833215844473,
+            0.003286605488908195,
+            -0.0023437997044735617,
+            0.0060251848371355166,
+            -0.013481652076194643,
+            0.0002908426939460218,
+            0.01953362396253743,
+            0.0038544889405965057,
+            0.020760460182873287,
+            -0.016224396121239458,
+            -0.00259233446682179,
+            -0.009282861237653619,
+            -0.02952130765234087,
+            0.017988370145301916,
+            0.006138236653154911,
+            -0.007607182955958238,
+            -0.009516767050028553,
+            0.0048347778733230365,
+            -0.005819209642901871,
+            -0.0025851701689263706,
+            -0.01073014664868326,
+            0.012433465684360061,
+            0.025784014942833097,
+            -0.016875624785552303,
+            0.007944349714123104,
+            8.651589872318479e-05,
+            -0.022306044741692753,
+            -0.022127016745066307,
+            0.020143472307319017,
+            0.002792646249762062,
+            0.008768813346304779,
+            -0.004074214276300631,
+            -0.00015024341785520542,
+            0.002990550758627699,
+            0.01951746553838439,
+            0.03896451381932467,
+            0.004232673242619471,
+            0.027876586543280867,
+            0.0145860303663199,
+            0.03994234200854897,
+            0.009657017763624678,
+            0.00696607158576712,
+            -0.026055387694656488,
+            0.02819890357011778,
+            0.00018297502510753014,
+            0.043190622821775236,
+            0.022488402221902314,
+            -0.0407730101161195,
+            -0.0021946518523759066,
+            0.01669520746276111,
+            0.013212623257794862,
+            0.017009844136059122,
+            -0.004537512994041156,
+            -0.014041213424740049,
+            0.028135404528975894,
+            -0.022859586430516003,
+            0.0008637006862481223,
+            -0.004486332934188017,
+            -0.025031795510001887,
+            0.007351370254175368,
+            -0.004567816916440959,
+            0.030025413528233364,
+            0.0014976740344892735,
+            -0.022285096307825804,
+            0.022837170656498543,
+            -0.009551269792616045,
+            0.011045530488684207,
+            0.008809047185288755,
+            0.020065001354321994,
+            0.031993362733068575,
+            0.009143210434415064,
+            0.02133640814803879,
+            0.012241777474889248,
+            0.014771538127546474,
+            -0.008804192497858033,
+            -0.02341427260922065,
+            0.023167424359504862,
+            -0.0011598649565273595,
+            -0.029627604902846688,
+            0.0021991474725787303,
+            0.003261878079344105,
+            -0.0005935713331682358,
+            -0.009144689470851402,
+            -0.005580557862988033,
+            -0.007240836451875213,
+            0.019631377011298905,
+            -0.007085213886352692,
+            0.007017573363709754,
+            -0.004074171791476129,
+            -0.00822871835665378,
+            -0.0421101573155962,
+            0.004210914075286222,
+            0.003408427700622687,
+            -0.01752179330665445,
+            0.006384440404174512,
+            0.011121481025573962,
+            0.043020575455262985,
+            -0.022809979913698254,
+            -0.0005737138409049003,
+            0.019206623212867476,
+            0.0009562551889820241,
+            -0.023427467810802533,
+            -0.038931688376706186,
+            0.007172345672967499,
+            -0.03467050750542982,
+            0.023185301620612612,
+            -0.0007075596339842658,
+            0.02387977204565927,
+            -0.03043468414449851,
+            0.013909304572370437,
+            0.004784668158421625,
+            0.012524879174804915,
+            0.021654512058639427,
+            0.03142146922870715,
+            -0.02053473475540882,
+            -0.050409862638621725,
+            -0.005406667316025549,
+            0.00035138762584085976,
+            0.022018487016014856,
+            -0.026014580970974616,
+            0.00025633004699302783,
+            -0.003349094027291921,
+            -0.02401366769212778,
+            -0.01735996745614562,
+            0.01962340791446025,
+            -0.050011480783403936,
+            0.015042913576492952,
+            0.0262737424555697,
+            -0.02507473901367252,
+            -0.05408889321034101,
+            -0.018988587082325706,
+            -0.028217893770659686,
+            -0.009285770094319318,
+            -0.00661159230987788,
+            0.028597405154740078,
+            0.003525576256054959,
+            -0.011462219358952466,
+            0.025401668390029348,
+            0.006155837251830549,
+            0.0026050693705924593,
+            0.015985722392086222,
+            0.007289910674212967,
+            -0.011452511453623235,
+            -0.014599577191293152,
+            -0.04558584556100621,
+            0.010197536827229795,
+            -0.0045431962637690155,
+            -0.019372598050508005,
+            -0.030147301333718016,
+            -0.0137828704002156,
+            -0.027060515748528666,
+            -0.0017291966950107505,
+            0.053783684195381654,
+            0.03292494290301388,
+            0.005560172296221419,
+            -0.0026085670565688026,
+            -0.010466890489356092,
+            0.036247483457995895,
+            -0.006789746401727701,
+            0.03662690472838588,
+            0.001784522073210422,
+            -0.0242673253380092,
+            0.01169382884548301,
+            0.02892627181542449,
+            0.009858451662166387,
+            -0.0005592283645355213,
+            -0.006160132931009508,
+            -0.008782830470079547,
+            0.0031207630972208394,
+            0.025613233596604418,
+            0.003268170392795548,
+            -0.010062991425127038,
+            0.018106967407336815,
+            0.004480291699976396,
+            0.02010374270557836,
+            0.03555357587371324,
+            0.023489837757983054,
+            -0.021782173652026497,
+            -0.026581762796336082,
+            -0.0013561486781077014,
+            0.0072266106510934505,
+            0.03551967295942857,
+            -0.01651315514034218,
+            0.0001274396319698452,
+            -0.008272735507120127,
+            -0.00894931703949637,
+            0.011149952898504355,
+            -0.0029671593121986116,
+            -0.0030752010285951576,
+            0.01662711184497578,
+            -0.0011672263615879709,
+            -0.033448222499147764,
+            0.0078288560210653,
+            -0.023496331346917084,
+            0.0051040606404415545,
+            -0.004113104164416575,
+            -0.007076525381595441,
+            -0.002157548492138289,
+            0.006302962169276006,
+            -0.008065841090139872,
+            -0.00035242603988955366,
+            -0.01586406637763306,
+            0.008271894439780969,
+            0.007438312999409211,
+            -0.0069617481460406095,
+            0.011402932583388205,
+            0.002666895909532698,
+            0.0005219881149353855,
+            -0.012159425670629751,
+            -0.012042145306711322,
+            -0.00957276588761916,
+            -0.03401863879157551,
+            -0.02604353712472079,
+            -0.024949533021906844,
+            -0.0014209149651479556,
+            -0.006566685925666972,
+            -0.03187796739029736,
+            -0.01209152137926458,
+            -0.011110763124407752,
+            0.007756961097537267,
+            -0.013260548545917543,
+            0.005433285079114047,
+            -6.399098520041351e-05,
+            -0.0044960488826674555,
+            0.016615636438440516,
+            -0.003643291599919672,
+            -0.019488010211707343,
+            -0.03927818878149417,
+            0.008209336177358258,
+            -0.02805593125673831,
+            -0.0038816771550190788,
+            0.007607119219721711,
+            -0.00528681677614909,
+            -0.029502897838743516,
+            -0.009149794737430706,
+            0.008767001199311985,
+            0.003507217008280066,
+            -0.01773349756494436,
+            0.03579670773605949,
+            0.026971617790626075,
+            0.011875179879209696,
+            -0.019891342676443626,
+            0.0508929813275565,
+            -0.0047010735337056275,
+            -0.012544935684845649,
+            0.018297503869683027,
+            0.02355022268131519,
+            0.03273167043015095,
+            -0.024830243320788693,
+            0.0009370814985214882,
+            -0.021937793653233454,
+            0.00535662615596573,
+            -0.035009915581788176,
+            0.0028595659314725012,
+            -0.014395027018699446,
+            0.015596459769560532,
+            0.03397372881171701,
+            -0.013254226072981457,
+            0.011195747181764677,
+            -0.016299384375589934,
+            0.00730244731485402,
+            0.00569944090461748,
+            -0.012916284280345044,
+            -0.005812311882399539,
+            -0.022902815152116493,
+            0.020140276669979015,
+            0.005809799950176385,
+            -0.0008130655714656685,
+            0.0104085826652653,
+            -0.044723432238900435,
+            0.03286429597337671,
+            0.008730225944501239,
+            0.017599189944921396,
+            0.0029762446259779998,
+            -0.0003068671951232415,
+            0.011611551606224214,
+            0.011228401175698422,
+            -0.016029727902740706,
+            0.004345913009102136,
+            -0.004613061137378433,
+            -0.03850533909474302,
+            0.009363351681220008,
+            -0.03631184057978666,
+            0.014427784677340159,
+            0.016026872002233575,
+            -0.004170582842643461,
+            -0.025166189571409366,
+            -0.005428471012333673,
+            0.010827910811085384,
+            -0.009783069664327864,
+            -0.009987805234276443,
+            -0.04163358183060102,
+            -0.025236440364963034,
+            0.034097508113604576,
+            -0.002119474613596975,
+            -0.025826403511707147,
+            -0.010830738112107807,
+            -0.011351132636491575,
+            0.020207687645133,
+            -0.006542026283946858,
+            -0.0006263975804936843,
+            -0.04150011545311013,
+            -0.0073396962917864515,
+            0.011192225907726928,
+            0.027746213794734322,
+            -0.035970017302005566,
+            0.02708126943501373,
+            0.002105633542220187,
+            0.014262674845755191,
+            -0.01626664983003942,
+            -0.005631805361903626,
+            -0.013990539604372344,
+            -0.035877093361375685,
+            0.016487723013906077,
+            -0.008607758248055054,
+            0.004714436224135091,
+            0.04458009880881818,
+            -0.0036872068506712727,
+            -0.009466125348499464,
+            0.007600705817195143,
+            -0.0034590883553048063,
+            -0.010351576085287273,
+            -0.0028564505793999295,
+            0.001959661298507,
+            -0.007080697863126256,
+            -0.04183476693971616,
+            -0.00499888976530217,
+            0.04684391782633148,
+            0.006361744381879857,
+            0.013276793765092465,
+            0.013516220521120684,
+            -0.018335746557932003,
+            -0.045577810515431215,
+            -0.0071616111758423025,
+            -0.0032763880176386808,
+            -0.022708028392758567,
+            -0.0240284945459861,
+            -0.008735040856657758,
+            0.005844145402902923,
+            0.006359368165686724,
+            -0.01120763599008222,
+            -0.022676131361171057,
+            -0.027085968700452136,
+            -0.010526103176253857,
+            -0.02123944716136792,
+            -0.010792145796733375,
+            -0.027142270190771887,
+            -0.028742234538796804,
+            0.02880426953386151,
+            0.007575182232859564,
+            -0.004084088301176117,
+            -0.010005765476121966,
+            -0.01255640546813973,
+            -0.0004774761617900028,
+            -0.0025307994917461232,
+            -0.005822065709999144,
+            0.0038614286313282164,
+            -0.011390482620811868,
+            0.023152086963976554,
+            0.0021220701260546895,
+            0.018329162172128138,
+            -0.05772324571231113,
+            -0.002709083851090071,
+            0.021497896395795905,
+            0.0033025410488410913,
+            -0.005412445141910314,
+            0.029352650459792132,
+            0.011223947499453582,
+            -0.006539460282439711,
+            -0.031355428613718234,
+            0.020942976176820262,
+            0.024122993515785947,
+            0.00817615634503138,
+            -0.012994160510030828,
+            0.01221095914384204,
+            0.01629778384790159,
+            -0.004494985018246821,
+            -0.011161098655072933,
+            0.005586857967039181,
+            0.020941039642372112,
+            0.013580622629911925,
+            0.0013398071867077221,
+            0.030905022925366163,
+            0.0006862084391508121,
+            -0.03521231612862609,
+            -0.01908191995763771,
+            -0.01275995164030872,
+            -0.003584331305986364,
+            -0.015264425022763126,
+            0.005294329654809353,
+            -0.002355465010876059,
+            0.006335052149475905,
+            0.022655379849785256,
+            0.03593231939948758,
+            0.0146937743203089,
+            0.00937330328448886,
+            0.003167462724350819,
+            -0.0013235176101076284,
+            0.012102872918129403,
+            0.008638011773075119,
+            0.0350701240268404,
+            -0.014653221697693864,
+            -0.011003994230902835,
+            -0.0037767288519369537,
+            0.010100131061186566,
+            -0.004113527518662366,
+            0.020414393256136372,
+            0.03274709615649469,
+            -0.0037222949335916876,
+            -0.005604552180246868,
+            -0.006160118231975786,
+            0.0041467454414452565,
+            0.01807879127455274,
+            -0.01252134965321004,
+            0.004364858987501461,
+            0.005549300665159658,
+            -0.0071897162833861,
+            0.018567059255813558,
+            0.0002622074720860951,
+            -0.012983550838417323,
+            0.03266376564402191,
+            0.11449509387240629,
+            0.0017251916835047628,
+            0.007125215166916139,
+            -0.040229997815548584,
+            -0.020537987527148888,
+            -0.00245728735683473,
+            0.005779972828836593,
+            0.007004732428236457,
+            0.02225914982152608,
+            0.006505827715591914,
+            0.0006708045351142873,
+            0.031708522931495536,
+            -0.017334083511709092,
+            -0.002522861743921599,
+            -0.01745656559688129,
+            0.013890451996979048,
+            -0.014312605700272691,
+            -0.013784963572164409,
+            -0.01572417853955075,
+            0.004507425871748977,
+            -0.000601158156253668,
+            0.0016158265715634729,
+            0.01791303234734911,
+            0.0011926917387973374,
+            -0.026393351252156246,
+            0.0006694008039871878,
+            0.001026031515214626,
+            -0.004514419527575117,
+            -0.011320929719992686,
+            -0.02223001723942815,
+            0.016235841442653918,
+            0.014188311619681008,
+            -0.004895218832118094,
+            0.009244697376301004,
+            0.010558796153959677,
+            0.02707485418319703,
+            -0.028380342401606758,
+            0.0040204430418629405,
+            0.007023041225985294,
+            0.01606556581447933,
+            0.017617862031190843,
+            0.0034870847111552862,
+            -0.011253315705053224,
+            0.03238289399766116,
+            -0.010563282687352038,
+            0.0065093536270233385,
+            0.014966802208644462,
+            -0.03541611540945151,
+            -5.821682910348165e-05,
+            0.0061985444938402745,
+            -0.011716201661363997,
+            0.009580035961315467,
+            -0.03120692534097433,
+            -0.0039291243987978845,
+            0.009317468331974374,
+            0.017611312558635673,
+            0.008804996982574765,
+            -0.0034129169537151925,
+            0.010822226698822738,
+            0.012510661458054741,
+            -0.0027147975086409966,
+            -0.008855213644027594,
+            0.01238502686635956,
+            -0.0007687053446954701,
+            0.00450108863711973,
+            -0.00799170686059026,
+            0.0029755595812552293,
+            0.002290650917244691,
+            -0.01554856300008412,
+            -0.011779430686394945,
+            0.00438393126679289,
+            0.0036931891075561957,
+            0.001535662168147833,
+            0.012014408636737167,
+            -0.006087255045485002,
+            -0.0077816220325470226,
+            0.0040573724156794284,
+            0.003136943055301706,
+            -5.505014316363865e-05,
+            -0.030625245024561508,
+            0.01585923084309149,
+            0.0067595435845396405,
+            0.010013308598626031,
+            -0.005315626260228517,
+            -0.00903142414091241,
+            -0.007240920015018893,
+            -0.01228521172101564,
+            -0.010657894223173453,
+            0.007658605592936815,
+            -0.00724924993432376,
+            0.011124174211132038,
+            -0.03128027787357322,
+            -0.013988715435985316,
+            0.01231792993245938,
+            -0.01528161242159377,
+            0.0007675821323979293,
+            -0.008389914312201402,
+            0.0009417242487395995,
+            -0.005119304379761705,
+            0.007887436211143328,
+            -0.03731503169547854,
+            0.007878001990271541,
+            0.02456170736765724,
+            -0.00330924625934981,
+            -0.020259771366167557,
+            -0.009613930457931786,
+            0.0241314385136084,
+            -0.008267831641886344,
+            -0.01645888287387047,
+            0.008421304862524506,
+            0.010140813149093993,
+            -0.0006309118552875393,
+            0.0008237784055549769,
+            -0.003726984191590917,
+            0.01669125388798323,
+            0.02054777616420997,
+            0.041930770964107515,
+            -0.014789610432004559,
+            0.01092349703703152,
+            0.02866938974792608,
+            -0.03378576980722847,
+            0.014728113846952417,
+            0.02518409645155824,
+            0.014590531327650454,
+            -0.0004446956315190408,
+            0.001609777186372437,
+            0.0068939321280085475,
+            -0.014273492115311677,
+            0.009247676157551494,
+            0.013380104440744367,
+            -0.018407746260686045,
+            0.020444430509441124,
+            0.03140071554008003,
+            0.0025839998101985736,
+            0.01355205268903058,
+            -0.006268939830271642,
+            -0.013860692186849113,
+            0.014717364068116991,
+            0.031429975488060896,
+            0.0006919565473572909,
+            -0.018561500446336806,
+            0.019128244642842268,
+            0.006988095806264863,
+            -0.003678529269242034,
+            -0.0021993078588142664,
+            -0.021334919445200388,
+            -0.021003586791369228,
+            0.017304350704792804,
+            0.013508779408333964,
+            0.027794485858413365,
+            -0.010594702782952029,
+            -0.007818089330662161,
+            0.014293688127901867,
+            -0.022368568887004094,
+            0.015426482139881204,
+            -0.02160820059598751,
+            -0.00571499813703566,
+            -0.0077402210272634235,
+            0.03274360648629609,
+            0.001441304806200403,
+            0.02734331677589297,
+            0.016457027764064002,
+            -0.033609420134045206,
+            -0.029790535553657092,
+            -0.00712106584238484,
+            -0.02307933297713104,
+            0.010356561171963577,
+            -0.03901183838032092,
+            -0.00071735150189382,
+            0.008251646656432826,
+            -0.008079917288471352,
+            -0.0007087526632966007,
+            -0.006419191399519291,
+            0.013989344185652506,
+            0.013661633160336152,
+            -0.0228103143826663,
+            0.023644639659577426,
+            0.008343481940133684,
+            0.025998483906975782,
+            -0.02076055376006461,
+            0.019148568238437927,
+            -0.017705363646416224,
+            -0.02777313933338929,
+            0.014358130220942483,
+            0.023585731219788467,
+            0.01635364038245971,
+            -0.007122853965024528,
+            0.007110003583832299,
+            -0.03556761320536796,
+            0.027585457849964405,
+            -0.013064062623866305,
+            -0.012504161289952458,
+            -0.0032379276562094555,
+            -0.011931827532360686,
+            -0.002574161005800279,
+            0.0008533907465694845,
+            0.0034345232246102806,
+            -0.03131462673429805,
+            -0.0002450959360520896,
+            -0.03727416416766574,
+            -0.028944814604815072,
+            0.01360126708805471,
+            -0.00036361284498363273,
+            -0.0049097813612814616,
+            0.02933242705521636,
+            -0.026339136251046315,
+            -0.02693295650849956,
+            -0.009590247544093352,
+            0.02853096456887074,
+            -0.0220673798697511,
+            -0.004519050393279848,
+            0.0015034014871316069,
+            -0.01828226786700525,
+            0.004436383319946364,
+            -0.0012748481042235446,
+            -0.010416065158876917,
+            -0.01105146139270597,
+            -0.03229479099927737,
+            -0.005822133182096844,
+            0.02106215163620738,
+            -0.006957125617688484,
+            0.002746241228040847,
+            0.013312248331456934,
+            -0.010968629118224582,
+            -0.0029946900704643812,
+            0.0017875284441794235,
+            0.023304175718480834,
+            -0.021033941064173063,
+            0.007714334541806199,
+            0.005955442031111047,
+            0.0016405586860131207,
+            0.018406562522853148,
+            -0.0054922197063461825,
+            0.0037690571959124617,
+            0.0162126534419016,
+            -0.009969304596173302,
+            0.017959307638726878,
+            0.007691549802475525,
+            0.006681837561745192,
+            0.002775700618222622,
+            0.007712770283579111,
+            0.009752652973700739,
+            -0.013506020126078112,
+            0.016911141425375933,
+            0.022468715726349706,
+            0.00023233555209372766,
+            0.021066580799163234,
+            -0.02302516292580905,
+            -0.028618544628453015,
+            0.018297878810036087,
+            -0.0225078941299981,
+            -0.012735208680311652,
+            0.016459925496937065,
+            -0.0043032724578332855,
+            -0.00797609922711709,
+            -0.014986057073351055,
+            0.02260618561608301,
+            0.02132930921464508,
+            -0.020249788476562527,
+            -0.012385936327068884,
+            0.0072304431443959505,
+            -0.004373475447051833,
+            0.007086656081865733,
+            0.03195810137269375,
+            -0.015982210897392764,
+            0.006818890093901413,
+            -0.0038306138017521626,
+            0.0030753642449792215,
+            0.017830546304667783,
+            0.009470018737705222,
+            -0.0019815241970236146,
+            0.030100238416495066,
+            -0.001330614013444443,
+            0.016392226282459177,
+            0.0004888028477871524,
+            -0.010948833692356176,
+            0.006343890495626752,
+            -0.026652806306173643,
+            -0.008627570055007363,
+            -0.001203530223345183,
+            -0.009581494451831351,
+            0.07598616930784373,
+            0.003930153552069478,
+            -0.01295957554290434,
+            -0.001673007115291578,
+            -0.001368287102181597,
+            -0.006897179139643672,
+            -0.00960679451603541,
+            0.0287166325268306,
+            -0.016613613331623346,
+            -0.01894527592131826,
+            0.0008370014132002703,
+            -0.006785618888421608,
+            0.00919924251249592,
+            0.01816363936999796,
+            -0.010225961928154075,
+            -0.01604759761423727,
+            0.001577138242530474,
+            -0.003520564936698765,
+            -0.004395434431595494,
+            0.0052399043934902014,
+            -0.010127122101423486,
+            0.006456662777583096,
+            0.026560491731327886,
+            -0.013684536319659296,
+            -0.01425196172663986,
+            0.004084982328707292,
+            -0.0010682084059972982,
+            0.009758641913073978,
+            -0.00864879062601467,
+            -0.00481222050200678,
+            -0.014375330954464725,
+            -0.011378890549403524,
+            0.02505641780042516,
+            -0.02861164504722546,
+            0.0034060847953066354,
+            0.014937227531980433,
+            -0.00016904224334342915,
+            0.01541381851648916,
+            0.0033028855218525644,
+            -0.03615935765718852,
+            -0.013677774380513626,
+            -0.02073839691250734,
+            0.019821074228011413,
+            0.01688422012576042,
+            -0.0033731840788147442,
+            -0.013840993586441808,
+            -0.026149880178874887,
+            -0.007430563876740068,
+            -0.02466234818567362,
+            -0.020708282040996014,
+            -0.019276991448222817,
+            0.006137002799644838,
+            0.005162892426268572,
+            0.022881759407670583,
+            0.0025854880108157857,
+            -0.015058889448404231,
+            0.0029948608544278386,
+            -0.006722836436935071,
+            0.011845298212977357,
+            -0.02747944486818378,
+            0.015311272959213262,
+            -0.01610493464983465,
+            0.025239768880017402,
+            0.007425208902253236,
+            0.0010569008485112994,
+            -0.008822681490988575,
+            0.00010704167529046774,
+            0.014883787023445807,
+            -0.004186968199973372,
+            -0.0007781259002896265,
+            0.02787555889813697,
+            0.02087373813833492,
+            -0.0017783425135865238,
+            0.008743718448136432,
+            0.028026471206552342,
+            0.019081083283793984,
+            0.0325261942213039,
+            -0.013815526479760766,
+            0.005113676604826083,
+            -0.028873600114081546,
+            0.010510475843707614,
+            -0.010871039729515013,
+            -0.010201824683340651,
+            -0.0067946599549165735,
+            0.027756887504389347,
+            -0.007962601205446472,
+            -0.0006687976124826922,
+            0.03469894503563926,
+            -0.014185294847328063,
+            -0.00010543064621374546,
+            0.0007386418834840544,
+            -0.009759481309152999,
+            0.02742722730386853,
+            -0.0073590367291510046,
+            0.018456032721630843,
+            -0.019560882612141943,
+            -0.013705917411254969,
+            -0.011894560208199602,
+            -0.001142768182039817,
+            -0.02576754877141608,
+            -0.008755240066826174,
+            -0.02249287216087102,
+            0.00859172262856532,
+            -0.0046012388553094375,
+            0.018845920253446127,
+            0.0011617844393195566,
+            -0.005275799428018491,
+            0.029069367036565145,
+            0.0088080893110627,
+            0.018100411389202663,
+            -0.0004971948011851863,
+            0.01216566778850985,
+            0.02369448391321832,
+            0.002637001729688794,
+            0.005012948614448268,
+            -0.024213704079796478,
+            0.003420184574622472,
+            -0.0004359900462023086,
+            8.265488264269535e-05,
+            0.016267948861008616,
+            -0.019572032453100303,
+            -0.02216264534756073,
+            0.021871880626992544,
+            -0.012742926070754964,
+            0.011676769226089007,
+            0.0035362870996336194,
+            -0.010732229460038356,
+            0.0006838777488770268,
+            -0.03083438636386314,
+            0.008820746379709355,
+            -0.03216685858285627,
+            -0.032230808828686675,
+            -0.013547681710236494,
+            -0.009994106571421729,
+            0.0008758088338659712,
+            0.028447187234357855,
+            -0.00795317029726568,
+            -0.015797892954136448,
+            0.005363565557248085,
+            -0.011199252697624111,
+            -0.00817162221789202,
+            -0.0007867892785855576,
+            0.005856939999700809,
+            0.009461959355310273,
+            0.005997804620690314,
+            0.011648335019189696,
+            0.007491946658046298,
+            -0.008028800679169293,
+            -0.008837725220736191,
+            0.015889923814605387,
+            -0.0014409716087425222,
+            -0.01176424391171685,
+            0.030430130082762363,
+            -0.013300620185884817,
+            -0.0005592926632214984,
+            -0.0021975669766032472,
+            -0.0015687208316663597,
+            -0.0021652076181411244,
+            -0.014225289143358926,
+            0.01870568404189425,
+            8.334501002782455e-05,
+            -0.014288806218067678,
+            0.0032841743905975634,
+            0.003841780428153241,
+            -0.0036498339483411074,
+            0.008388922960269383,
+            -0.008147275785908849,
+            0.017348147498892,
+            -0.015113395971380741,
+            0.008740452410210708,
+            0.0038335161236289234,
+            0.0043482078469644385,
+            -0.0065755348640655865,
+            -0.01592809184384243,
+            0.009642593996701513,
+            0.012051872087663658,
+            0.02173264012652684,
+            -0.02525660485859878,
+            0.00876161653901566,
+            -0.005374533920781607,
+            -0.014534179652013885,
+            -0.016127866637369738,
+            0.00982303248417706,
+            0.022322910435388785,
+            0.0063086346898091296,
+            0.012139263756190026,
+            0.00013383660542132047,
+            0.005848259030010402,
+            -0.014767198851314148,
+            0.010570158368602513,
+            -0.011355774405066537,
+            -0.018439261453084426,
+            -0.005243587266408389,
+            0.028894120979627323,
+            -0.005492024421080143,
+            0.017938747701972086,
+            0.0021350711250093858,
+            0.029007356490801176,
+            -0.008686781541806358,
+            0.0014883465513564527,
+            0.0039866022469891155,
+            -0.01295271525889559,
+            -0.004272008629260297,
+            0.01976273843646234,
+            0.04882868638601926,
+            0.010204475902027572,
+            -0.0009155109545661691,
+            0.010069071220988797,
+            0.005626418787776309,
+            0.002148101019776153,
+            -0.00805781810841286,
+            -0.030431071067211577,
+            0.008883707378543252,
+            0.005628218215212441,
+            -0.01997825125961566,
+            0.019838234890736068,
+            0.026483365072033395,
+            0.023040623182975964,
+            -0.011585643598171132,
+            0.016539930468533123,
+            -0.010092529089936188,
+            0.0177008735686489,
+            0.028665424114962996,
+            0.013810061902648515,
+            -0.010487150104822674,
+            -0.0020847870330548534,
+            0.006329342599564054,
+            0.004186981876004016,
+            0.00026239408223817384,
+            0.015190614519766204,
+            -0.019155180136775043,
+            0.01005728016778505,
+            0.014725826697964637,
+            0.03322436221319799,
+            8.049930133627564e-05,
+            -0.01645191835557271,
+            0.0016249281996128196,
+            -0.009976155662939693,
+            0.00856399492362911,
+            0.000632427127231252,
+            0.0032242260692672124,
+            -0.009688119824276678,
+            -0.012682779503915694,
+            -0.002851918183031152,
+            0.009403205508474241,
+            0.0021188716361101698,
+            0.002759004011231831,
+            0.007641925150177999,
+            0.0022830062031146204,
+            0.01713024474384216,
+            -0.0011181980841546123,
+            -0.00029141889484139217,
+            0.004601090256674874,
+            0.010509199590505558,
+            -0.01649852835819883,
+            -0.017548369964263514,
+            0.004337237475053179,
+            0.012217494210546133,
+            -0.013159179948224596,
+            -0.013241981632059755,
+            0.015704543608959117,
+            -0.01913575130892795,
+            0.024934102868065366,
+            -0.00942381892138331,
+            -0.004543611336526587,
+            0.000343200128934031,
+            -0.00727538880611703,
+            -0.0038751118497522673,
+            -0.0030064279504724797,
+            0.003984715814505561,
+            0.0019261818110367147,
+            0.007472350699291607,
+            0.008472369788801792,
+            0.014568547873264247,
+            0.006043895352236181,
+            0.0032580830408778395,
+            0.009109104498825864,
+            -0.014523846294121576,
+            -0.014800649887278973,
+            -0.002704791598011488,
+            -0.008895394038700809,
+            0.00902016867266151,
+            -0.006607182154874112,
+            0.00845408382182461,
+            -0.015501867835442603,
+            0.015573642225745032,
+            -0.012200552219016517,
+            -0.0008900215712478553,
+            -0.009666233545144905,
+            -2.7470277092642562e-05,
+            0.008191697904514857,
+            0.014605287653054787,
+            0.0070552942843382215,
+            -0.031921405225964206,
+            0.0017403427534633655,
+            -0.008846568352320488,
+            0.03446319360336241,
+            0.002612606662691094,
+            -0.01060863123202008,
+            0.020352181356568008,
+            -0.01592124132155158,
+            -0.00027704668715660693,
+            -0.01602075523652062,
+            -0.004797557413047212,
+            0.00707939446796679,
+            0.016399017978990946,
+            0.02759049512664933,
+            -0.005697020356865688,
+            -0.007995305872076221,
+            0.006384047872862793,
+            0.029590696670334232,
+            0.012246955508890425,
+            0.024748351124415785,
+            -0.007633712004672087,
+            0.014348228716100414,
+            0.007773260552660029,
+            0.0076045488653348975,
+            0.004321689250440021,
+            -0.002591801797145718,
+            -0.011698574606499644,
+            -0.014974971814360001,
+            -0.0037563183330623384,
+            -0.016987915346053498,
+            0.00012247833530362867,
+            0.021344400276501117,
+            -0.00475351423959752,
+            -0.0033847161875730026,
+            -0.01874186437694691,
+            0.0033461670218611346,
+            -0.010029445302549983,
+            -0.00367421246775698,
+            -0.013133469163793961,
+            -0.0254122035745965,
+            -0.018556369080030412,
+            -0.002155244024999317,
+            -0.00929309186646207,
+            -0.015194269781263309,
+            0.0070354690219415535,
+            -0.016305117309271634,
+            0.0080837166473439,
+            -0.00448473417024141,
+            -0.016508775226236883,
+            -0.00958925666995396,
+            0.01100335153658195,
+            -0.0041819993634961685,
+            -0.01929493648733792,
+            -0.01282300765645703,
+            -0.011300260564838231,
+            0.0051733704121938884,
+            0.005996780699975213,
+            -0.0038111648304178544,
+            -0.0028279639348479987,
+            0.016640121845366295,
+            -0.010197365811240557,
+            0.009878730312873492,
+            -0.03335474019383245,
+            -0.03551826040406959,
+            0.004497847402931096,
+            0.01507007013703286,
+            0.0013085881889541252,
+            -0.02078680101065954,
+            0.010806315988966003,
+            -0.016499186313125556,
+            -0.00858013792516617,
+            -0.008522534127498645,
+            0.009649521088411411,
+            -0.007283003380127847,
+            0.01299542385123007,
+            -5.3226845309105124e-05,
+            0.030477683597373687,
+            0.0074153455615142755,
+            0.004161881195438522,
+            -0.00679081048132963,
+            0.008354912600159931,
+            -0.001621776416108105,
+            0.004080191054849395,
+            0.02120524874226696,
+            0.012440848502230418,
+            -0.00218041171518331,
+            0.006458415098595872,
+            0.005678055538564355,
+            -0.017186537377121235,
+            0.02507960402044456,
+            0.024678178106806115,
+            -0.008205288425409454,
+            0.004108402466376321,
+            0.005783335302169668,
+            -0.008846663924040241,
+            -0.006897707231869481,
+            0.006527795262026734,
+            -0.018474181903007085,
+            -0.0008721381678008483,
+            0.010598084358464908,
+            -0.02255441909004843,
+            -0.004840511019512733,
+            0.021745651362109403,
+            0.004631567631335321,
+            -0.018558972627958834,
+            0.02764788406651861,
+            -0.004942361238163609,
+            -0.0027290999934532494,
+            -0.010876135202650048,
+            0.01671497143541799,
+            -0.012173527207375091,
+            0.008443317346655462,
+            0.011021541861818043,
+            -0.006295639322287843,
+            0.003083685676540999,
+            0.008277206401503182,
+            -0.004504302913715969,
+            -0.0068075424927125676,
+            -0.0010508416962215848,
+            0.010167160518175114,
+            0.0036980815941666336,
+            -0.022704963865525812,
+            -0.006020073261002279,
+            0.02934227992246887,
+            -0.015082922782242507,
+            -0.0050916398557337705,
+            0.005376294716056329,
+            0.007709342013963906,
+            -0.003476236561444464,
+            -0.0023399991233097337,
+            -0.00787486645740744,
+            0.010356414979428607,
+            -0.015950083845400274,
+            0.02076360898078935,
+            0.025608263494550438,
+            0.024504236990840842,
+            -0.014814235990238182,
+            0.010828584133720797,
+            -0.011703992394890898,
+            -0.011047470905668532,
+            -0.015542835222173864,
+            -0.001658296303013357,
+            0.001970208852236451,
+            0.013107754130616248,
+            0.009666887546609378,
+            0.0235853683657408,
+            0.006473544089109194,
+            0.008973300312205858,
+            0.014118601565410305,
+            -0.003902202854013708,
+            -0.018303438244719996,
+            -0.014584226585460755,
+            -0.011977613285028946,
+            -0.01756345491020528,
+            0.0004563602312336091,
+            0.020461659996271687,
+            0.012417744207683839,
+            0.024136626592375227,
+            0.002119213701333967,
+            0.013759902247467922,
+            -2.9525392347047554e-05,
+            -0.000892319792935443,
+            0.00045783825646242434,
+            -0.005238483120312843,
+            0.019255756091120846,
+            -0.004599752813817219,
+            0.01656589522110913,
+            0.0006443773300684925,
+            0.007133940816771615,
+            0.013554761340256566,
+            -0.01704388669741371,
+            -0.003443100892108368,
+            -0.010029620309829285,
+            0.000941391966877521,
+            -0.019633111750884034,
+            0.01934571074127927,
+            0.010273330977233852,
+            0.0017121345993533963,
+            -0.018327189396014198,
+            -0.0029706385099949693,
+            0.03448635131998772,
+            0.019458999453011265,
+            0.0050734354363614525,
+            0.02632443373898792,
+            0.013018762352749236,
+            0.014443611411502168,
+            0.010153370950346075,
+            0.02356766253713842,
+            -0.023479811300626968,
+            0.02095255383200973,
+            -0.011590347850340786,
+            -0.007414136601134069,
+            -0.0035774888741646705,
+            0.0025141508183687115,
+            -0.013588691733097813,
+            0.00982877622088977,
+            -0.0017916539399655826,
+            -0.025386211797192578,
+            0.009164316443206016,
+            -0.008811052979979271,
+            0.010464442616928296,
+            -0.007774569252594923,
+            -0.0002806136360286729,
+            -0.0127084026190163,
+            -0.015184317235990752,
+            0.001259201880504347,
+            -0.009158429132281081,
+            0.023908150915412617,
+            -0.021210848953875233,
+            -0.01321384014436847,
+            0.013997502713857512,
+            -0.006869443694669701,
+            -0.011364239609894453,
+            -0.00374449980963666,
+            0.023673050288879224,
+            -0.009243341446512421,
+            -0.01359367519009039,
+            0.014464166953530989,
+            0.003834735113400469,
+            0.0059713317014342395,
+            -0.007508862694272452,
+            -0.008428705474135382,
+            -0.003280788036976103,
+            0.005998840043285335,
+            0.026615077617284676,
+            -0.014059930972065399,
+            -0.0032833264512892022,
+            -0.009189999799498163,
+            0.004118204537632102,
+            -0.02359889635281631,
+            0.02771075351384059,
+            -0.008871395303200375,
+            0.021705177758560575,
+            -0.0015924449308442096,
+            -0.006239188158271628,
+            -0.024231036074349822,
+            0.027522945599928003,
+            0.003683870966132199,
+            -0.010221180899173243,
+            0.0012768725460662862,
+            -0.052005292736885915,
+            0.016636109796947025,
+            0.008429895913163985,
+            -0.019507994429834353,
+            0.003697519981610208,
+            -0.004773584856005853,
+            0.018983847580558853,
+            -0.022966993228850697,
+            -0.014078986007109447,
+            0.008283766047449966,
+            -0.01153285124362746,
+            0.0031439180465649996,
+            -0.02527749501821793,
+            -0.005754537200570427,
+            0.0011760345167229269,
+            -0.005276936074280004,
+            -0.001705933750221904,
+            -0.020022761608132548,
+            -0.025972227850795115,
+            0.017666017379157135,
+            0.015820875579600418,
+            -0.002278262448951789,
+            0.01320818011561616,
+            0.008709899932124187,
+            0.00836573261091119,
+            0.016034263394940008,
+            0.007124509234103267,
+            0.0081473711608784,
+            0.045498533239850446,
+            -0.019846323044145485,
+            0.007371629414298193,
+            -0.0014854149174929534,
+            0.013951741898629716,
+            -0.030467924817555014,
+            -0.0055144480998344705,
+            0.0062802744565489945,
+            -0.0049373689340857915,
+            0.03510454786147091,
+            -0.017326094216003073,
+            0.01694160526735264,
+            -0.0026075809870445238,
+            0.0007420394265014909,
+            -0.02042003877666893,
+            -0.026156937726134134,
+            -0.014294938857971834,
+            0.0011857406889443987,
+            -0.0010047941261608975,
+            -0.011946583993069242,
+            0.006797001710574735,
+            0.004917161312925866,
+            -0.0020713379451245515,
+            0.015652523103715316,
+            -0.02952061566240878,
+            0.007356800041369564,
+            -0.010733578620243498,
+            0.020008504830359524,
+            -0.00990331048284697,
+            -0.0013319400098308093,
+            0.007236960885181047,
+            -0.004122868383988748,
+            0.013216043472071927,
+            -0.0038337251451538677,
+            0.004227115653354611,
+            -0.005221321656766545,
+            -0.01794343569722877,
+            -0.0016435261495179767,
+            0.021429008432348098,
+            -0.008349684096540855,
+            0.031864497493526205,
+            -0.01264743878037828,
+            0.009174982784686633,
+            0.010692990337462762,
+            0.0006628831036334152,
+            -0.005286285851702483,
+            0.009052971814211658,
+            0.0021431970274545893,
+            -0.008690520448278274,
+            0.006934310986093342,
+            -0.018349449937310143,
+            0.008494451785507706,
+            0.03394632739506833,
+            0.0018511328780472266,
+            -0.0007519661670070526,
+            -0.007266663179613392,
+            -0.01335844201823745,
+            0.018540713029383066,
+            0.005488273906733518,
+            -0.002546531152114063,
+            -0.027536660869590543,
+            0.02007746621157705,
+            0.0012265798861291223,
+            -0.004755325069578531,
+            0.012466557901411086,
+            0.0020495867632654885,
+            0.019989531151046736,
+            -0.010171678639113316,
+            0.030757957899447096,
+            0.010050036220124393,
+            0.002959833051227505,
+            0.0038556928367700113,
+            -0.036655749074851536,
+            -0.01652152778069314,
+            -0.00019357502472740266,
+            0.014271581054865031,
+            -0.012430647252826173,
+            -0.022432580814844155,
+            0.0010346169306977334,
+            0.01148004051449691,
+            -0.0033898642762881046,
+            -0.009650665305271706,
+            0.01560757541276278,
+            -0.020342213902670575,
+            0.006701302394408393,
+            0.007693672045981921,
+            0.0287100214318104,
+            -0.0020735384554022153,
+            -0.0012324838849990056,
+            0.000626416803904792,
+            -0.0055669860801192935,
+            -0.0008037366126870824,
+            0.017251369734470945,
+            -0.013661316947002466,
+            0.0175926211217534,
+            -0.012735241490550407,
+            0.0064713656749838585,
+            -0.0021092378432037563,
+            -0.010629579860050812,
+            -0.020114807971202787,
+            0.0029609495764307357,
+            0.009378003163375899,
+            0.006920144698428806,
+            -0.010438984881449953,
+            0.010715409812532968,
+            0.012197673689706352,
+            -0.00753962861484439,
+            0.014519905132928041,
+            0.022173989895384192,
+            -0.007687503607783572,
+            0.013633321414009471,
+            -0.0041446334728745095,
+            -0.014056264816568229,
+            -0.009985463824440833,
+            0.023929750617160402,
+            -0.0005563806942667597,
+            0.008756997373618809,
+            0.00388125051641897,
+            0.0007856447663040696,
+            -0.011226530517316183,
+            -0.001653447552601681,
+            0.019294448360333633,
+            0.011831934675073843,
+            -0.0013888534822008192,
+            -0.0010525764356699885,
+            -0.0014566936525653407,
+            -0.009427557739483988,
+            0.02049188741459728,
+            -0.022030783927622997,
+            -0.0019604940727076573,
+            0.00461018148439991,
+            -0.022589816803875618,
+            0.0001863117949313752,
+            -0.024570217962989905,
+            -0.01569952241187498,
+            0.011763033781122343,
+            -0.0022530631823737554,
+            -0.016028810867854907,
+            -0.0003242701040942633,
+            7.424127301806556e-05,
+            -0.007815789940508527,
+            -0.004135500768181258,
+            0.006115909992703774,
+            -0.00034193217510010117,
+            -0.0032262189374874842,
+            0.013430154310506884,
+            -0.009996622196870008,
+            0.015431528911575157,
+            0.0067029980830614145,
+            -0.015702167721800483,
+            0.0021406610839557003,
+            0.018500219544264753,
+            -0.0058910087864916085,
+            0.00787021425357441,
+            -0.014681907249972176,
+            -0.0037495551894426303,
+            -0.005787661351944086,
+            0.006051351977821761,
+            0.024849195780885833,
+            0.012969434450643372,
+            0.01139340445077682,
+            0.017405361843075933,
+            0.004325000715054734,
+            -0.01572297405297993,
+            0.0007416798945675923,
+            0.01682727189828363,
+            0.010464036837245312,
+            0.009804326468113703,
+            0.0008975556667684365,
+            0.0043704860361487635,
+            -0.005142176468800175,
+            0.009317064072855551,
+            0.02915482449687158,
+            0.0013434689612997423,
+            0.008696141010760192,
+            0.017276730345147134,
+            0.027038741396131265,
+            -0.0035088997194497725,
+            -0.007469614319788499,
+            0.008739942310300076,
+            -0.009657991604173173,
+            -0.001081629174976152,
+            -0.017000351183542026,
+            -0.012310275736750868,
+            0.016510505259358398,
+            -0.006084627062230492,
+            0.02848913486719956,
+            -0.0010655806791907225,
+            -0.02026498848361482,
+            -0.01565505608194376,
+            0.001404549479440637,
+            -0.01362132895080652,
+            0.005287566256051299,
+            0.014187265685427415,
+            0.01917046513020831,
+            0.004687191682875024,
+            -0.022258337721085656,
+            0.037364962197399806,
+            0.01757108963349427,
+            0.019377870795760123,
+            -0.019422840506328382,
+            -0.006159311237412833,
+            -0.009077095160676276,
+            0.02116697012944498,
+            -0.00880692095488465,
+            -0.009875674856043954,
+            0.024014332004451508,
+            0.017812606845656484,
+            0.005679806286464216,
+            0.006736609904079586,
+            0.0020048663439319585,
+            0.0025414630253661444,
+            -0.015831593851012047,
+            0.017957902498542212,
+            -0.005858125698347523,
+            -0.0058068554681761945,
+            0.017529514573571298,
+            -0.005989581792993925,
+            -0.0030306021203732347,
+            0.00015839997806822045,
+            -0.019697052145531482,
+            0.011618112886207201,
+            0.008256007601576028,
+            -0.017229774429089857,
+            -0.0014757560402567304,
+            -0.0002404480326945295,
+            -0.014118694740134205,
+            0.02004885146995733,
+            0.01694893356262835,
+            -0.0013248884139405398,
+            -0.0051752815372013315,
+            0.008223868147393114,
+            0.007000679132597393,
+            0.005256194890343006,
+            -0.019496098876977206,
+            0.005230597149342019,
+            0.005103909294824105,
+            -0.0014716649591412774,
+            0.010274346323051634,
+            -0.009030594825985526,
+            0.033740380586893826,
+            -0.006505788197604522,
+            0.013065658941136721,
+            -0.0043634731684193745,
+            -0.0055529072007985125,
+            0.001816020676232658,
+            0.013054468478598695,
+            0.009753501030700457,
+            -0.00970446155762939,
+            -0.004172354962129865,
+            0.018697982092775737,
+            -0.011983156146516074,
+            0.0013300791598046338,
+            0.01564072865804035,
+            -0.004858814041438713,
+            0.004708235083775013,
+            0.008635204790345423,
+            -0.004489505237783172,
+            0.01801512858970922,
+            -0.016482572999490366,
+            0.0023953436660054026,
+            0.010694869684962088,
+            0.02390864159830096,
+            -0.019152187046410703,
+            0.021786596486708207,
+            -0.017951050280323393,
+            0.026538582634436314,
+            0.014484894489694262,
+            -0.0035222259457311444,
+            -0.01946200885147006,
+            -0.00850690913394181,
+            0.011654565129063417,
+            -0.021423803876241532,
+            -0.0007985948078290641,
+            -0.0007290222944686444,
+            -0.021367628543092133,
+            -0.011189364017231132,
+            -0.009961779129778331,
+            0.010701637745810138,
+            0.001918633898887013,
+            0.010380983868185359,
+            0.008002182827196827,
+            0.010784760574518087,
+            0.014405640442760822,
+            0.0008958107769384412,
+            -0.010743001445440941,
+            -0.023795589003655273,
+            0.016987708698679597,
+            0.00438463274382093,
+            0.0007862212695139549,
+            -0.01627468994515667,
+            0.011936651269737248,
+            0.013250643815768723,
+            -0.009385605548926286,
+            0.009121882469425302,
+            -0.03716901103204463,
+            -0.004552808410398574,
+            -0.0035704811925430366,
+            -0.0020309197365173475,
+            -0.00908678158053104,
+            0.01559445415882485,
+            -0.013682507836005851,
+            -0.0009590922904844632,
+            0.00011696985057536548,
+            -0.0074756526903548915,
+            -1.5467890432254378e-05,
+            0.020649928837031005,
+            0.008438785939908428,
+            -0.0037382105257999757,
+            0.010376213224654074,
+            0.008826031303751312,
+            -0.013900359449287676,
+            -0.01527474712786084,
+            0.019035366035884273,
+            -0.018776918681908793,
+            0.0063098418544296165,
+            0.01677916431403883,
+            0.020778813096283988,
+            -0.02891862501830075,
+            -0.0007231469961304435,
+            0.022093939332021185,
+            -0.0012807635799950273,
+            0.006370841109790851,
+            0.008781853138377092,
+            0.006754082980602544,
+            0.0029102159062355374,
+            -0.003711173531879644,
+            -4.843770941622283e-06,
+            -0.003940474588075707,
+            0.007155539022862352,
+            0.004366433868349658,
+            0.036453728071874844,
+            -0.013661129348165794,
+            0.002541308805735001,
+            -0.0058838191638835155,
+            -0.00347401304572066,
+            -0.00593469688967835,
+            0.004407665856950772,
+            -0.010055727994798024,
+            0.004330827357536078,
+            -0.001570861535288093,
+            -0.0003731129422346693,
+            0.004883969598300302,
+            -0.001701673379646408,
+            0.007036892564844683,
+            -0.011863650867472625,
+            -0.0003979727750011258,
+            -0.020174851412609312,
+            0.021117284557875267,
+            0.00011666551282792189,
+            0.003662462447326642,
+            0.03723741246290771,
+            0.00476064510662738,
+            -0.00238355131307445,
+            -0.012880057949435798,
+            -0.013551338766321525,
+            -0.012986444917747435,
+            -0.020703821629987552,
+            -0.0036665673548525388,
+            -0.021006311026247164,
+            -0.0052061406234528915,
+            0.010362862594785442,
+            -0.011540184420787572,
+            -0.004023659825622843,
+            -0.015881589631889473,
+            -0.006531981517815174,
+            -0.03114211633687585,
+            -0.01124607488650473,
+            0.007493228366452009,
+            -0.019149285949049886,
+            -0.0039932870926865805,
+            0.026850121590606017,
+            0.001095360635760988,
+            0.01387696424607248,
+            0.0038058248676879625,
+            0.008117686785954085,
+            -0.0045937472017383955,
+            -0.0005669025036141417,
+            -0.01928244877716143,
+            0.003391980330529026,
+            0.008525354854576925,
+            -0.00361507909530282,
+            0.0070008404454106905,
+            -0.008835727374865623,
+            -0.016950017602980382,
+            0.0021654956715219517,
+            0.009334379529385142,
+            -0.006639140003365139,
+            -0.017587472740268683,
+            -0.011049929670847685,
+            0.010401189523318744,
+            -0.02283429470064187,
+            0.006257145380266984,
+            0.016439581169238607,
+            -0.009363030375624585,
+            0.004651575425738153,
+            0.019809183749755425,
+            -0.011810674576781702,
+            -0.005811328501528397,
+            0.008471992070998505,
+            0.013891259999721588,
+            0.0095606134736748,
+            0.02954498207274139,
+            0.0026596809804008996,
+            0.000561870602466584,
+            0.004889263753490895,
+            -0.004277061142504552,
+            -0.017423200644235665,
+            -0.006243787897080996,
+            0.0038507494787521856,
+            0.012873996671952184,
+            -0.009879417960879032,
+            -0.00048575277472139055,
+            0.01682604801673206,
+            0.004530939990187663,
+            -0.0033901248079550284,
+            -0.011868399508738274,
+            -0.020080331108085103,
+            0.0015580259874809898,
+            -0.01362200460317948,
+            0.015354347040143093,
+            0.003390601069331352,
+            0.0015284610287850575,
+            -0.004239880336806429,
+            0.01828645373058079,
+            -0.005013418527935815,
+            0.012749833447770729,
+            -0.006148890994661865,
+            -0.004587831231448011,
+            0.007986533783001116,
+            0.009166578151006778,
+            0.0004939857995863664,
+            0.05421019830078428,
+            0.002838056604877719,
+            0.01333205648597435,
+            0.013419127695838135,
+            -0.014523733369484934,
+            0.014086525147734823,
+            0.012688569853552613,
+            0.018497344547821693,
+            0.013037819487160016,
+            -0.012353776077273462,
+            0.017354801119939964,
+            0.0024084723134891487,
+            0.003909681065813095,
+            0.0022202438064449653,
+            -0.004528576079628071,
+            0.019594683950225115,
+            0.004070222328547378,
+            -0.0074362443729372105,
+            0.004453618452623641,
+            0.005007543168224267,
+            -0.013105524222311019,
+            -0.0112958915514782,
+            0.027105208848863218,
+            -0.01219417557128091,
+            -0.018749971743148304,
+            0.006706594269933445,
+            -0.02008458683853808,
+            -0.03763686535076864,
+            -0.0021332074289244973,
+            0.009992361098878196,
+            0.002795183524495193,
+            -0.00253695604043486,
+            0.01614298064601032,
+            0.016730443047719683,
+            -0.016985296937992726,
+            -0.018245902728910347,
+            0.0025257780765481825,
+            0.0016333325293570182,
+            0.01861807236979066,
+            -0.023168663546030196,
+            -0.006378864850313035,
+            -1.3795465857380962e-05,
+            -0.006829008132963306,
+            0.004850306884033312,
+            0.004703280112191218,
+            -0.0071714759384722965,
+            -0.0027773519096287217,
+            0.002256846645341137,
+            0.01975383140755998,
+            0.013198754584066554,
+            0.01572971337890945,
+            0.002600741959577911,
+            -0.005033508062449482,
+            -0.016817359058724685,
+            -0.005450011991586895,
+            -0.013478330855299777,
+            -0.009091854345715177,
+            -0.003042052092498012,
+            0.008217376329142566,
+            -0.025053469059115735,
+            -0.011965083373220982,
+            0.0119294363235832,
+            0.00391377331459168,
+            0.0003099203772539202,
+            0.007607672728136279,
+            -0.003830132495364688,
+            0.010331053246451333,
+            0.013823890473448383,
+            0.0076112887952959766,
+            -0.004723377809559611,
+            0.009600411526173948,
+            0.011460781436250457,
+            0.006015477068316837,
+            -0.005072711638895421,
+            0.023859899048048694,
+            0.01821920335673539,
+            -0.01680365156728093,
+            -0.002149107383573353,
+            -0.0027829676356423997,
+            -0.01884308738652242,
+            0.009665830801645276,
+            0.00880322398336952,
+            0.008308268605449346,
+            0.003906369351868086,
+            0.00974531700889854,
+            -0.013875904819275925,
+            -0.0050570811397453235,
+            0.002870120865445989,
+            -0.005729556771404781,
+            -0.01901337075686543,
+            0.000297242372701423,
+            -0.011987192261104132,
+            -0.011327157550660895,
+            -0.01409710922165158,
+            -0.003221240464411861,
+            0.005283472511900512,
+            0.021472187640986972,
+            -0.004410881709548286,
+            -0.001855852749039012,
+            0.01351697599517034,
+            0.010060125186506398,
+            0.0021658738823899905,
+            0.0002206474541239167,
+            0.005418828324605768,
+            -0.0027919130944783397,
+            0.0008094042430809068,
+            -0.0015826797905407809,
+            0.021089276765434876,
+            -0.02016611086095107,
+            -0.009413977569213423,
+            -0.009288078098692112,
+            0.005842197094339927,
+            0.028793032462603636,
+            -0.007165634502373454,
+            -0.010104002083017078,
+            -0.01994315293556678,
+            0.0003786527922888834,
+            -0.07810609175343784,
+            -0.005210636756258548,
+            0.001870513618910238,
+            0.03126380113105983,
+            -0.010817888476436834,
+            0.0033673768058629823,
+            -0.002204487854177595,
+            0.014542299655712928,
+            -0.001760107965206874,
+            -0.0013283746982194259,
+            0.0071954806737858056,
+            -0.005640400818010697,
+            -0.006210989097047217,
+            0.005904163661619133,
+            -9.945941028848348e-05,
+            -0.010266938705916303,
+            0.00057263080955709,
+            0.02209498395848068,
+            0.0159398346608872,
+            -0.006575439759804175,
+            0.01816748552021211,
+            -0.00719294553484007,
+            -0.009838738592324234,
+            0.005639288780502127,
+            -0.0019007683459126327,
+            -0.015369619176147573,
+            -0.011475977270803784,
+            0.009862808084641291,
+            -0.030069890644150284,
+            -0.00010725753918890802,
+            0.00943698424646706,
+            -0.013540245857270492,
+            -0.008293042205008774,
+            0.007212974960781723,
+            0.006708343474532583,
+            -0.0012017968151025369,
+            -0.0078026000239507145,
+            -0.02527526731750743,
+            0.0027578400032925625,
+            -0.002446811971857853,
+            -0.004899702349784173,
+            0.0019960465173086183,
+            0.001479503476855694,
+            -0.012645905858466798,
+            0.01216748847452763,
+            -0.013378877368040743,
+            -0.012053251571418766,
+            0.012508842412556306,
+            -0.00027167241745418584,
+            0.01682680930386839,
+            -0.020607299598265426,
+            0.0010702151063620052,
+            -0.028075439824932304,
+            -0.007671848153932823,
+            -0.015486630894165054,
+            -0.0009776136330459065,
+            -0.0012939921569540803,
+            -0.03481880531471923,
+            0.012889447665794033,
+            -0.005200149829198305,
+            -0.006094702500154477,
+            -0.00459778303079732,
+            -0.024583504817899775,
+            0.006072281238752663,
+            0.00607350167867964,
+            -0.006506508970816635,
+            -0.016816439635445445,
+            0.008428048142588516,
+            -0.0011040593013238872,
+            0.004873670535200424,
+            -0.007475800310154908,
+            -0.0005638920132076605,
+            -0.013047135093134946,
+            -0.005463454269764494,
+            -0.0008522562324122834,
+            0.01827123304871433,
+            0.0044055851550317835,
+            -0.014526381222031981,
+            0.016862445236481526,
+            0.013250692156820124,
+            0.01602714293656341,
+            -0.0071089293427753115,
+            -0.0004265739315250755,
+            -0.02750630108787212,
+            -0.00327225092954903,
+            -0.023687644154808972,
+            0.00041213204365507744,
+            0.009447660230507965,
+            0.0203905264889617,
+            0.012694421477479161,
+            0.004620617778996077,
+            -0.001557511764988156,
+            0.0066751090724502,
+            0.02696959560423252,
+            -0.01099183126742224,
+            -0.006969111598423376,
+            -0.01884504192789845,
+            0.01113023717919988,
+            0.025135748541199255,
+            0.003381303139273845,
+            -0.011317957771935042,
+            0.008112598754655146,
+            -0.0009247845526290428,
+            -0.010302980598863016,
+            0.02889587382004902,
+            -0.0391134481322538,
+            -0.015599591108049343,
+            -0.004605824192962023,
+            0.007406520608110251,
+            -0.015596917115435033,
+            0.00770468362805995,
+            -0.012614339656105264,
+            0.001699049329817941,
+            -0.005851477164035241,
+            0.03535087340659861,
+            -0.0031950960727572925,
+            0.0006909856233673865,
+            -0.003395233050301427,
+            0.003531135044131077,
+            0.012355948349937309,
+            -0.004444221317023453,
+            -0.030726300006909638,
+            -0.0022983766869702346,
+            -0.011198252640110788,
+            0.009274666307515955,
+            -0.016207714291658314,
+            -0.017342435976846468,
+            -0.009424737744066259,
+            -0.021827243671837953,
+            -0.012403084982789827,
+            0.029747453318486338,
+            -0.005824001042844527,
+            0.01747569398142269,
+            -0.005219060548639168,
+            -0.008365003057800934,
+            0.00021272411948388247,
+            0.01344335658390732,
+            -0.028498240040284714,
+            0.00041963499785238845,
+            -0.002322931702641349,
+            0.0126393550661767,
+            -0.001595134597082556,
+            -0.00983073906977773,
+            0.006903313245694627,
+            0.0010522208685011098,
+            -0.015648940549713764,
+            -0.0007573245570947632,
+            -0.008533504878052644,
+            0.020475329445149498,
+            0.00027791387213409814,
+            0.001074213629646005,
+            0.013864331671732542,
+            -0.02102352993295736,
+            0.01576938071930521,
+            -0.021722775924283996,
+            0.01893267442161547,
+            -0.006577103101038035,
+            0.005772016664149475,
+            0.008283223343661555,
+            0.014068789332250016,
+            -0.014030042849491258,
+            -0.00663752723080222,
+            -0.034436220972634726,
+            -0.0122621879114314,
+            -0.001946072976826902,
+            -0.005308941654794638,
+            -0.014773123927909993,
+            -0.0011054822266082056,
+            -0.026205461509573584,
+            -0.006460373065662098,
+            0.012822188046808264,
+            0.019900312098359022,
+            -0.008409070094265829,
+            0.014163132434889265,
+            -0.006743711872055091,
+            -0.0028230771810882492,
+            -0.012652021749331811,
+            -0.018061180380009395,
+            -0.01083672324723357,
+            -0.01890966636827258,
+            -0.0018635408070944168,
+            0.004709222372611061,
+            -0.015760618199235518,
+            0.011435131737642627,
+            0.03198874888333405,
+            -0.01567271472650171,
+            0.008243406188694909,
+            0.016411496328333205,
+            0.0034236971031973503,
+            0.011728079139322837,
+            0.0038311415233052795,
+            0.004621789116267879,
+            -0.011109450058055078,
+            -0.011473158133621661,
+            0.006524490579223986,
+            0.00954511918454185,
+            -0.004913142601134428,
+            0.024347072255007807,
+            0.0008745419206472714,
+            0.005045794664611216,
+            0.005304506902446091,
+            0.011488187043525824,
+            -0.013135271908307747,
+            0.019746484188715552,
+            -0.002055210086078267,
+            -0.015571665869889405,
+            -0.01535499169234956,
+            -0.01105419264826655,
+            -0.018626399757822328,
+            0.0017108083455869296,
+            0.0181439079855921,
+            -0.0012493542718277685,
+            0.012137753522284387,
+            -0.00924460959150499,
+            -0.005019437549649743,
+            -0.0027156377206998474,
+            -0.038211954658821295,
+            -0.0021878970332906457,
+            -0.018809487178512628,
+            -0.02498945889484935,
+            0.03505216568400238,
+            -4.26477081213836e-05,
+            0.00997643255528116,
+            -0.010310197159232386,
+            -0.0002454902098986129,
+            -0.011065403358802515,
+            0.014954998647970473,
+            0.004624123403100598,
+            0.008872923678983478,
+            0.008560280683876702,
+            -0.016419500515801093,
+            -0.002997557798375716,
+            0.007472390947008284,
+            -0.00374822278027057,
+            0.012680407710694952,
+            0.015353684671120731,
+            0.020886474047003733,
+            0.011161262397502876,
+            0.002045862158732102,
+            0.00887271411363814,
+            0.009563361150297699,
+            -0.0003513237086717377,
+            -0.010459621030424415,
+            0.015419039131459582,
+            -0.02204848739950703,
+            0.012172257374410352,
+            0.04346634265794738,
+            0.019197433209794286,
+            0.029488039318675732,
+            -0.02272810439246353,
+            0.01448092574459596,
+            -0.003801043409441206,
+            0.009476229966122152,
+            0.0049309968883087685,
+            0.02346191840355209,
+            0.003973840862678076,
+            0.0073546443568171405,
+            0.010647581332452525,
+            -0.00972051412996416,
+            0.025644022498404723,
+            -0.00846669916801108,
+            5.1518540791212095e-05,
+            -0.004407265804145216,
+            -0.026400029105519137,
+            0.01615193008014211,
+            0.0081722536401398,
+            0.0010702314763197463,
+            0.0010263623479474385,
+            -0.0019566604229607643,
+            0.007982949510899019,
+            0.007079322685771456,
+            -0.015274148711633536,
+            -0.003886053737590683,
+            -0.018527455865296014,
+            0.0012864569064749438,
+            -0.024337133883771334,
+            0.010466651250597847,
+            -0.017568706814132484,
+            -0.014822354159709854,
+            -0.021666681078831483,
+            -0.013232016867264097,
+            -0.02011653224168991,
+            0.0035858355517491515,
+            -0.008632755352062642,
+            0.013025728891203198,
+            -0.016962939287139683,
+            -0.021494971929779273,
+            0.008430887589398292,
+            0.00870802359711456,
+            0.007481282677955534,
+            0.009728368603458082,
+            0.014996157358749115,
+            0.01736292711410622,
+            -0.022712386143092265,
+            -0.023163001611599246,
+            -0.006682209717364464,
+            -0.003531495447167166,
+            0.006762379833125297,
+            0.00933681196321128,
+            0.012884185500343249,
+            -0.007077895849501418,
+            -0.007554762054641417,
+            -0.013283067570021503,
+            0.006197966297581871,
+            -0.003993067576677973,
+            -0.009464789349849612,
+            -0.0015662739825834614,
+            -0.0061389795593180686,
+            0.002753664732282926,
+            -0.0001791088888074743,
+            0.0007595261237141144,
+            0.017122834676182343,
+            -0.008176319370751018,
+            -0.009787747261447871,
+            -0.02027228453943018,
+            -0.010683273300243025,
+            -0.015604745521156111,
+            -0.0016279406232527138,
+            -0.006506713191861036,
+            -0.014420323628589382,
+            0.010888206569219648,
+            -0.011930708571093279,
+            -0.00887210117616784,
+            0.010807147705360856,
+            0.0073900910661282935,
+            -0.003431679006289002,
+            0.037804020313161775,
+            0.0011945597874820288,
+            0.00778778394048267,
+            -0.003366543306746973,
+            0.005296727102519836,
+            -0.017761251541598876,
+            -0.026382400396890007,
+            0.0010357716464853195,
+            -0.01775777668381489,
+            0.010807713750049491,
+            -0.008515368407459745,
+            0.0016051389544388124,
+            -0.00034958600424912995,
+            -0.009585854918521076,
+            0.0036549720706351473,
+            -0.008110927907950148,
+            0.004212219431447508,
+            -0.03507890639477636,
+            0.011409377585617926,
+            -0.01819120811524444,
+            -0.008962242411035718,
+            0.01710177594046791,
+            0.037560936862121835,
+            0.0028513388998398104,
+            0.00030058353811717895,
+            0.0014792184333546447,
+            0.004427921639326168,
+            0.021438127785303025,
+            0.006339483221420334,
+            -0.001187747087595264,
+            -0.02237982599829809,
+            0.0062435650294483695,
+            0.002200578793697352,
+            0.010092828474934744,
+            0.023105023225115266,
+            -0.016797326021070008,
+            -0.015378520287395565,
+            -0.012732527281000097,
+            -0.010797463778182385,
+            0.009881499418260473,
+            0.02801767062785128,
+            0.01811139137849365,
+            -0.017358098300028903,
+            -0.020207173948076473,
+            0.0016242901745373126,
+            -0.026331540602868762,
+            0.007269751661479859,
+            0.11940631114242901,
+            -0.0045382135734049305,
+            -0.005994407567723939,
+            -0.008679712231646732,
+            -0.004482553208064535,
+            0.016709571294385293,
+            -0.013387043582971358,
+            0.069179848974036,
+            -0.007221545718220735,
+            0.0037392229364642915,
+            0.0028916224964057385,
+            0.00462729158956839,
+            -0.009727339449635306,
+            0.011570145487622916,
+            -0.011287365791681587,
+            -0.012991994493845187,
+            -0.013076194601468383,
+            -0.0180830261130804,
+            -0.007272768410218907,
+            -0.004155564814758654,
+            0.004204993170487392,
+            -0.005525575983182816,
+            0.008051108079520902,
+            -0.017551645696802648,
+            -0.010800070994485792,
+            0.013007176804761548,
+            -0.0023627485852619817,
+            0.005390986239687555,
+            -0.029668390278675554,
+            -0.0007650391259819867,
+            -0.0002446394046947245,
+            0.021280292985495608,
+            0.003137771886236958,
+            -0.0019077298815862946,
+            -0.010810159675564049,
+            0.007040107942856957,
+            0.023835993694726953,
+            0.021875366566457236,
+            0.0180511566130312,
+            0.012135351541044354,
+            0.020483975362807574,
+            -0.008667079096167644,
+            0.026324312888512866,
+            -0.001552207146726937,
+            -0.00397810465151493,
+            0.0077577136352257004,
+            -0.009849600842081875,
+            -0.00816392268697345,
+            0.00677353736462267,
+            -0.017288900282504148,
+            -0.020418071729964713,
+            -0.020383762873109595,
+            0.025741408097358233,
+            -0.011755094311186617,
+            0.002737129201159793,
+            -0.008266210469668866,
+            0.027287436144516414,
+            -0.006283892307996516,
+            -0.0029440642954701632,
+            0.009428876742171362,
+            -0.03788574758692254,
+            0.008834605607057885,
+            0.006463342392514337,
+            0.026711196829464153,
+            -3.9827332346130654e-05,
+            0.005565561841469247,
+            -0.0006738085601962865,
+            -0.006104320995853506,
+            -0.010792830169647851,
+            2.4565453208679464e-05,
+            0.017983908748172493,
+            0.0035181716362611077,
+            0.016970206528324653,
+            -0.005449808667944701,
+            -0.023171388778833197,
+            0.007092509502738752,
+            -0.01555320330471131,
+            -0.023306285898839978,
+            -0.002028783456704567,
+            -0.03196436161255752,
+            0.0033412618908711163,
+            0.007527633574188688,
+            -0.01890570959143866,
+            -0.017859735533232564,
+            0.010329340599904344,
+            0.004382396539605985,
+            -0.0016746397858953155,
+            0.006643785064338525,
+            -0.03307812909454471,
+            -0.01927475592253288,
+            0.004686970010943116,
+            0.012733525168279355,
+            -0.012993441126970916,
+            0.015291832952199082,
+            -0.020491167895268936,
+            0.016451976170508816,
+            -0.012259387189535237,
+            0.018393274368854132,
+            -0.006997668664226754,
+            0.011843450716200625,
+            0.011457415642860215,
+            0.009560877634897626,
+            -0.01790001115302499,
+            0.0214879415944361,
+            0.0066829070506954115,
+            0.03900547021123392,
+            0.02512484499576633,
+            -0.012651294056575184,
+            -0.0018425680226742782,
+            -0.00011361458320146007,
+            0.014982647266052349,
+            0.0061874424115559,
+            -0.003689464024837091,
+            0.02705902205734107,
+            0.018239469373522537,
+            -0.021070704644859155,
+            0.020037508334843183,
+            -0.00973804771680803,
+            0.018876283979591423,
+            -0.013303080480396818,
+            0.010592424494628194,
+            0.004717870260467274,
+            0.01070131238869639,
+            0.009087656015197565,
+            0.006708673669681232,
+            0.025928065153829043,
+            0.03663367449576964,
+            -0.00511073108390551,
+            0.002161381031392467,
+            0.004864092591944181,
+            0.012814518637357098,
+            0.001556833001747549,
+            0.007915766323215207,
+            -0.007012785514562176,
+            0.005834786334750152,
+            -0.002311533032619821,
+            -0.00122162928193159,
+            0.019525999280692766,
+            0.000824163362963678,
+            0.012875575199417254,
+            0.013259376059185821,
+            -0.002390133687709065,
+            -0.005682644446302278,
+            0.0019262218417782657,
+            -0.018193478045923098,
+            -0.015131696847666434,
+            0.029463723011600933,
+            -0.011447039651934148,
+            -0.008083114389964213,
+            0.0020642857121204753,
+            0.012160202510990566,
+            -0.00866944494225225,
+            -0.005378300635997713,
+            0.0038280364584418335,
+            0.014427318067525318,
+            -0.01799358399064332,
+            0.003381165206832161,
+            0.005252552821854107,
+            0.019433554824232185,
+            -0.00028791985674147885,
+            -0.013487678738391956,
+            0.0005317922815671783,
+            0.014982152801567637,
+            0.011902348165369078,
+            0.0016056390299304837,
+            -0.006935777197137747,
+            -0.00446584737370504,
+            0.004586814866182051,
+            0.006269466956110759,
+            0.0027516492952993445,
+            0.008157662045793032,
+            -0.0025001885993292637,
+            -0.016648747436203294,
+            0.017735757469839637,
+            -0.0030669873770852725,
+            0.018058248627079793,
+            -0.0038238312452176745,
+            -0.0147172470474118,
+            0.0043328417868095355,
+            -0.001888507803054499,
+            -0.018556943283494755,
+            0.009407930200720534,
+            0.002560949424273885,
+            0.010411159836451958,
+            0.0052571230238294825,
+            -0.007499470181994448,
+            -6.145889479758644e-05,
+            -0.015163819825329975,
+            0.022836682800304944,
+            0.016390944816242627,
+            0.001713572907204073,
+            -0.0035687505973373878,
+            -0.031210299824387475,
+            0.0029076511444359886,
+            -0.007822231980296341,
+            0.017821806420465356,
+            0.008250605126658402,
+            0.007908773724009905,
+            0.008351355849444162,
+            -0.00743970642051451,
+            -0.002411981861500092,
+            0.012543476434788293,
+            0.0202407777715137,
+            -0.0024635818448627447,
+            0.02535771413579989,
+            -0.012533449594824386,
+            -0.0022489292647974992,
+            0.0012362917358624392,
+            0.01578939075186126,
+            -0.0008001783766371555,
+            0.006311837812042848
+          ],
+          "keyphrases": [
+            "Switch Transformers",
+            "Mixture of Experts",
+            "sparsely-activated model",
+            "training instability",
+            "multilingual settings"
+          ],
+          "title": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity"
+        },
+        "type": "document"
+      },
+      "bidirectional": true,
+      "properties": {
+        "cosine_similarity": 0.8535917850971473
+      }
+    },
+    {
+      "id": "e8db17b8-6913-4e72-b163-d719d401cbb8",
+      "type": "cosine_similarity",
+      "source": {
+        "id": "b570d71e-4f83-486e-b70f-b0c6217db455",
+        "properties": {
+          "page_content": "Mixture-Of-Experts With Expert Choice Routing\n\nYanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon Google, Mountain View, CA, USA {yanqiz, taole, hanxiaol, dunan, huangyp, vzhao, adai, zhifengc, qvl, jlaudon}@google.com\n\nAbstract\n\nSparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choice method. Instead of letting tokens select the top-k experts, we have experts selecting the top-k tokens. As a result, each token can be routed to a variable number of experts and each expert can have a fixed bucket size. We systematically study pre-training speedups using the same computational resources of the Switch Transformer top-1 and GShard top-2 gating of prior work and find that our method improves training convergence time by more than 2×. For the same computational cost, our method demonstrates higher performance in fine-tuning 11 selected tasks in the GLUE and SuperGLUE benchmarks. For a smaller activation cost, our method outperforms the T5 dense model in 7 out of the 11 tasks.\n\n1 Introduction\n\nScaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27]. The final model quality has been found to have a power-law relationship with the amount of data, model size, and compute time [16, 20]. However, training efficiency, which is defined as the total amount of computation used to achieve superior model quality than the state of the art system [21], should receive greater attention as we increase our efforts towards green AI [29].\n\nSparsely gated mixture-of-experts [31] (MoE) provides an effective way to scale model capacity given a fixed computational cost, and has recently played an important role in increasing the training efficiency of large-scale language models [10, 21]. MoE operate by adopting a number of experts, each as a sub-network, and by activating only one or a few experts for each input token. A gating network must be chosen and optimized in order to route each token to the most suited expert(s). For example, recent work has implemented sparse routing via k-means clustering [12], linear assignment to maximize token-expert affinities [22], or hashing [8, 28]. Many of the prior work use a routing strategy concerning the token choice, where each token selects the best one or two experts.\n\nWe argue that the independent token choice of prior work often leads to an imbalanced load of experts, which causes training inefficiency and sub-optimal training of the model. In order to mitigate this\n\nissue, previous sparsely gated networks introduce additional auxiliary losses as regularization to prevent too many tokens being routed to a single expert, but the effectiveness is still limited. Recent approaches [8, 22, 28] explore alternative strategies for routing, but they focus on pre-training only and do not demonstrate performance gain on downstream tasks. Moreover, none of the previous methods consider allocating a variable number of experts to each token based on importance, which can be beneficial.\n\nWe propose a very simple yet effective routing method we are calling expert choice. Unlike conventional MoE where tokens select one or two top-scoring experts, our method lets each expert pick the top-k tokens. Our method guarantees perfect load balancing, allows a variable number of experts for each token, and achieves substantial gains in training efficiency and downstream performance as demonstrated in our experiments. Our major contributions include:\n\nWe identify common pitfalls in conventional MoE such as load imbalance as described in Section 3.1. We then propose a heterogeneous, expert choice method to provide a fluid allocation of model parameters based on a learnt token-to-expert importance. This method intrinsically guarantees load balance without imposing an auxiliary loss.\n\nWe show our method provides over 2× faster training convergence in a 8B/64E (8 billion activated parameters, 64 experts) model, compared to the top-1 and top-2 gating counterparts in Switch Transformer [10] and GShard [21].\n\nWe show our method demonstrates strong scaling when increasing the number of experts from 16 to 128, evaluated in training perplexity.\n\nWe show our method demonstrates strong performance on downstream tasks selected from GLUE and SuperGLUE at all the evaluated scales. More specifically, our 8B/64E model outperforms a T5 11B dense model in 7 out of 11 tasks evaluated.\n\n2 Related Work\n\nScaling: Various approaches have been proposed to scale up neural network capacity to improve performance. Recent works have successfully scaled models to billions of parameters via various forms of model parallelism [2, 21, 26, 27, 33]. Model parallelism [30] splits weights and tensors across multiple cores while pipeline parallelism [18, 24] splits different layers across devices with micro-batches pipelined to the different layers. To enable continued scaling of neural networks, improving model training and serving efficiency has become a critical research area.\n\nConditional Computation: Computation decisions can be made dynamically based on the input [23, 25]. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation, by activating certain parameters and computation on demand, on a per-example or per-token basis [3]. Conditional convolution layers [1] with task-specific gating has been used to combat catastrophic forgetting when a sequence of learning problems are optimized. The gating decisions may be binary or sparse and continuous, stochastic or deterministic.\n\nMixture of Experts: Sparsely-gated MoE [31] is the first model to demonstrate massive improvements in model capacity, training time, or model quality with gating. Switch Transformer [10] simplifies the gating by selecting only the top expert per token using a softmax over the hidden state and demonstrates better scaling than previous work. All the prior work requires an auxiliary loss to explicitly encourage balancing. This loss term has to be carefully weighted to not overwhelm the primary loss. However, auxiliary loss does not guarantee balancing and a hard capacity factor has to be imposed. As a result, many tokens can still be unprocessed by the MoE layer. Hard MoE [12] with a single decoding layer can be efficiently trained to good effect on large scale hashtag prediction tasks.\n\nBase Layers [22] formulate a linear assignment that maximizes token-expert affinities while ensuring each expert receives an equal number of tokens. Hash layers [8, 28] devise hashing techniques on input tokens. However, the evaluations are limited to pre-training perplexity. THOR [? ] randomly activates experts during training and inference and is trained with a consistency regularization loss.\n\nTHOR has demonstrated strong performance on translation tasks. Different from these prior works, our method is a learnt method that enables heterogeneous MoE and effectively improves downstream fine-tuning performance.\n\n3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n5 Conclusion\n\nWe propose a new routing method for sparsely activated mixture-of-experts (MoE) models. This method addresses load imbalance and under-utilization of experts in conventional MoE methods, and enables selecting different numbers of experts for each token. Our model demonstrates more than 2x training efficiency improvements when compared to the state-of-the-art GShard and Switch Transformer models, and also achieves strong gains when finetuning on 11 datasets in the GLUE and SuperGLUE benchmark.\n\n6 Limitations\n\nThe expert choice method might not immediately apply to auto-regressive text generation as our current implementation takes in the past and future tokens to perform the top-k selection. One possible solution is to collect a large batch of input sequences, dispatch tokens of the same sequence into separate groups, and perform expert choice routing for each group. Another scenario where the expert choice method does not immediately apply is when the batch size becomes very small during serving or inference. A global top-k can be selected instead and we can cap the number of times each expert or token gets selected. We leave these possible improvements for future work.\n\nAnother long-standing issue with MoE has been the large memory footprint. Even though computational cost can be reduced using sparsely gated networks, the total number of parameters increases linearly or sub-linearly with the number of experts. Increasing the number of experts requires reservation of a large number of hardware devices. Therefore, dynamic (used) power is saved while static (reserved) power is not. Power saving techniques such as the ability to put hardware devices into low power states while not in use [17] can help with reducing the reserved power requirements.\n\nReferences\n\n[1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pages 3930–3939. Computer Vision Foundation / IEEE, 2020.\n\n[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. In Advances in Neural Information Processing Systems.\n\n[3] Kyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning, 2014.\n\n[4] Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems, 2021.\n\n[5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.\n\nTransformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.\n\n[6] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 933–941. JMLR.org, 2017.\n\n[7] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixtureof-experts, 2021.\n\n[8] Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan.\n\nTricks for training sparse translation models, 2021.\n\n[9] Richard L Dykstra. An iterative procedure for obtaining i-projections onto the intersection of convex sets. The annals of Probability, pages 975–984, 1985.\n\n[10] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.\n\n[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. NAS-FPN: learning scalable feature pyramid architecture for object detection. In CVPR, pages 7036–7045. Computer Vision Foundation / IEEE, 2019.\n\n[12] Sam Gross, Marc'Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision, 2017.\n\n[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.\n\n[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 630–645, Cham, 2016. Springer International Publishing.\n\n[15] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2016.\n\n[16] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.\n\n[17] Ping Huang, Zuocheng Xing, Tianran Wang, Qiang Wei, Hongyan Wang, and Guitao Fu. A brief survey on power gating design. In 2010 10th IEEE International Conference on Solid-State and Integrated Circuit Technology, pages 788–790, 2010.\n\n[18] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 103–112, 2019.\n\n[19] Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David A. Patterson. A domain-specific supercomputer for training deep neural networks. Commun. ACM, 63(7):67–78, 2020.\n\n[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.\n\n[21] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.\n\n[22] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6265–6274. PMLR, 18–24 Jul 2021.\n\n[24] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. New York, NY, USA, 2019. Association for Computing Machinery.\n\n[25] Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, Cédric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. In ICLR. OpenReview.net, 2021.\n\n[26] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.\n\n[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.\n\n[28] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models, 2021.\n\n[30] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 10435–10444, Red Hook, NY, USA, 2018. Curran Associates Inc.\n\n[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E.\n\nHinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In ICLR (Poster). OpenReview.net, 2017.\n\n[32] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 10–15 Jul 2018. [23] Min Lin, Jie Fu, and Yoshua Bengio. Conditional computation for continual learning, 2019.\n\n[29] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.\n\n[33] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.\n\n[34] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc.\n\n[35] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.\n\nGLUE: A multi-task benchmark and analysis platform for natural language understanding.\n\nIn Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, November 2018. Association for Computational Linguistics.\n\n[36] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake A. Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen. GSPMD: general and scalable parallelization for ML computation graphs. CoRR, abs/2105.04663, 2021.\n\n7 Checklist\n\n(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Yes (b) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes (c) Did you discuss any potential negative societal impacts of your work? N/A. Not any. (d) Did you describe the limitations of your work? Yes (a) Did you include the code, data, and instructions needed to reproduce the main experimental results? Yes. We include details in the experiment setup to help reproduce the main results. (b) Did you specify all the training details? Yes (c) Did you report error bars? Yes (d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? Yes (a) If your work uses existing assets, did you cite the creators? Yes (b) Did you mention the license of the assets? No. The used dataset is not released yet. (c) Did you include any new assets either in the supplemental material or as a URL? No. The dataset is not released yet.\n\n(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? No. Not using persons' data. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes. The dataset does not contain any personally identifiable information or offensive content.\n\nA Comparison On Fine-Tuning With A Dense Model\n\nOur 8B MoE model achieves stronger pre-training perplexity than its dense counterpart. However, a better perplexity does not always directly translate to downstream performance as demonstrated in Section 4.4. To this end, we compare fine-tuning performance of the 8B dense model and MoE model in Table 1. As shown in the table, our MoE model using expert choice routing consistently outperforms the dense model across the 11 tasks in GLUE and SuperGLUE.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 1: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nB Capacity Factor\n\nWe evaluate the downstream task fine-tuning performance by varying the capacity factors. Note that a capacity factor of n indicates on average how many experts each token can be received. EC-CF2 is our baseline expert choice, which matches GShard top-2 gating computational footprint. EC-CF1, however, matches Switch Transformer top-1 gating computational footprint. EC-CF0.5 further verifies that an aggressively lowered capacity factor can provide strong enough performance, that almost matches the top-2 gating baseline.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Top-2 78.1 87.0 88.3 85.0 82.6 90.1 90.7 81.6 94.7 68.2 67.2 83.0±0.3 EC-CAP2 78.2 88.0 88.5 85.7 83.0 90.8 91.1 80.0 95.4 70.4 64.1 83.2±0.4 EC-CAP3 78.5 91.7 89.3 86.3 83.5 90.9 91.1 81.8 94.9 70.0 65.6 84.0±0.4 EC-CF2 79.1 89.6 89.3 86.8 84.3 91.3 91.2 81.1 95.2 68.1 68.0 84.0±0.2 EC-CF1 77.4 90.6 88.0 85.5 83.6 90.3 91.2 79.8 95.3 66.5 64.9 83.0±0.2 EC-CF0.5 77.4 89.6 86.3 85.2 82.7 91.7 91.0 79.6 94.9 67.3 63.5 83.0 ±0.05 Hash Layers 76.1 85.2 86.7 83.4 82.5 90.0 90.3 75.7 94.0 67.4 63.3 81.3±1.0\n\nTable 2: Comparison between different routing methods in fine-tuning of 100M/64E models. We perform 3 independent fine-tuning runs for each method and report the average results. This gives more accurate difference between the variants of expert choice method, since they achieve close fine-tuning results. We do not report averaged results in other experiments.\n\nC Capped Expert Choice\n\nAs described in Section 4.5, the maximum number of experts each token is assigned can be capped by an entropy-regularized linear programming. Figure 1 compares the validation perplexity when training the 100M/64E models using the base expert choice method (EC-BASE), expert choice capped by two experts per token (EC-CAP2), expert choice capped by three experts per token (EC-CAP3), and GShard top-2 gating.\n\nAs shown in the figure, restricting the number of experts to 2 degrades the perplexity compared to the base expert choice method. This suggests that a more flexible allocation of experts (e.g. more than 2 experts for a token) can enhance model expressiveness. On the other hand, our EC-CAP2 and EC-CAP3 methods still outperform the top-2 gating method by a clear margin. We believe this confirms the effectiveness of a load balanced training, provided by our method. Finally, EC-CAP3 obtains comparable perplexity to EC-BASE. As indicated by Figure 3, only a little fraction of tokens use more than 3 experts therefore we see little or no difference between EC-BASE and EC-CAP3 variants. We present the fine-tuning results of these methods in Table 2.\n\nD Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [? ]. We use mod x to map a token ID to an expert ID. This in some way ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 2. Hashing based routing performs much worse than expert choice in terms of average scores and variance.\n\nE Fine-Tuning Details\n\nWe did a hyperparameter search for both baseline models and expert choice method. For fine-tuning of the 8B dense model, we use a constant learning rate of 0.0001 and a dropout rate of 0.1. We freeze the attention layer and feed-forward layer while leaving the embedding and layer normalization trainable. This setting has been found optimal for the 8B dense model. For MoE 8B/64E models including GShard top-2 gating and expert choice, we found continuing the learning rate from the pre-trained model while using a square root learning rate decay works better. In addition, we do not apply parameter freezing for fine-tuning MoE models. For models with 100M expert size, we use a constant learning rate of 0.0001 and no dropout is used.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/2202.09368v2.md"
+          },
+          "headlines": {
+            "1 Introduction": [],
+            "2 Related Work": [],
+            "3 Method": [
+              "3.1 Pitfalls Of Token-Choice Routing",
+              "3.2 Heterogeneous Moe Via Expert Choice",
+              "3.3 Expert Choice With Additional Constraint",
+              "3.4 Model Architecture"
+            ],
+            "4 Experiments": [
+              "4.1 Setup",
+              "4.2 Training Efficiency",
+              "4.3 Scaling The Number Of Experts",
+              "4.4 Fine-Tuning On Glue And Superglue",
+              "4.5 Heterogeneity Matters",
+              "4.6 Comparison With Hash Layer",
+              "4.7 Ablation"
+            ],
+            "5 Conclusion": [],
+            "6 Limitations": [],
+            "A Comparison On Fine-Tuning With A Dense Model": [],
+            "B Capacity Factor": [],
+            "C Capped Expert Choice": [],
+            "D Comparison With Hash Layer": [],
+            "E Fine-Tuning Details": []
+          },
+          "summary": "The paper introduces a new routing method for sparsely-activated Mixture-of-Experts (MoE) models called 'expert choice,' which addresses load imbalance and under-utilization issues in conventional MoE methods. Unlike traditional token-choice routing, expert choice allows experts to select the top-k tokens, ensuring perfect load balancing and enabling variable expert allocation per token. This method improves training efficiency, achieving over 2x faster convergence compared to existing models like Switch Transformer and GShard. It also demonstrates superior performance on downstream tasks in the GLUE and SuperGLUE benchmarks. The study highlights the benefits of flexible expert allocation and shows that expert choice outperforms hashing-based routing methods. The paper also discusses potential limitations and future improvements for the expert choice method.",
+          "summary_embedding": [
+            -0.009726986289024353,
+            -0.0337262786924839,
+            -0.023302342742681503,
+            -0.019878201186656952,
+            -0.025953780859708786,
+            0.04560471698641777,
+            -0.006874797400087118,
+            0.039665497839450836,
+            -0.006973279174417257,
+            0.034544438123703,
+            -0.025787118822336197,
+            -0.00363057479262352,
+            -0.01052999310195446,
+            0.0054657477885484695,
+            -0.024499278515577316,
+            0.036786794662475586,
+            0.010007280856370926,
+            0.003272630739957094,
+            0.006102093029767275,
+            -0.04048365354537964,
+            -0.018666116520762444,
+            -0.013522328808903694,
+            -0.03230207785964012,
+            0.008969432674348354,
+            0.02484775334596634,
+            0.0016713523073121905,
+            0.0026230288203805685,
+            0.003624893259257078,
+            -0.05311964824795723,
+            0.04339266195893288,
+            0.018681267276406288,
+            0.023711422458291054,
+            -0.016332851722836494,
+            -0.012832955457270145,
+            -0.039120059460401535,
+            -0.0197418425232172,
+            0.030786970630288124,
+            0.039968520402908325,
+            -0.019287310540676117,
+            0.03357476741075516,
+            0.018060073256492615,
+            0.02454473078250885,
+            -0.021272100508213043,
+            -0.029817303642630577,
+            -0.0061892117373645306,
+            -0.007840678095817566,
+            -0.008355814032256603,
+            0.01242387667298317,
+            0.016984347254037857,
+            -0.013007191941142082,
+            0.0212569497525692,
+            -0.01371171697974205,
+            0.03384748846292496,
+            0.008916404098272324,
+            -0.042332086712121964,
+            0.03984731063246727,
+            -0.008007340133190155,
+            0.012105703353881836,
+            0.03281721472740173,
+            0.010333028621971607,
+            -0.005287723150104284,
+            -0.021544819697737694,
+            -0.03378688171505928,
+            0.0010529992869123816,
+            0.00014014738553669304,
+            0.005950582213699818,
+            -0.003069985192269087,
+            -0.009742137044668198,
+            0.03227177634835243,
+            -0.003939277958124876,
+            -0.0740584209561348,
+            0.050089433789253235,
+            -0.026590125635266304,
+            -0.007628562860190868,
+            -0.028938541188836098,
+            -0.01736312545835972,
+            0.037362534552812576,
+            0.005662711802870035,
+            0.0098709212616086,
+            0.0106284748762846,
+            -0.0007892759749665856,
+            0.028923390433192253,
+            0.00910579226911068,
+            -0.024923507124185562,
+            0.04478655755519867,
+            -0.04790768027305603,
+            0.01801462098956108,
+            -0.04336235672235489,
+            0.022756904363632202,
+            1.6393929399782792e-05,
+            -0.020726662129163742,
+            -0.0013304532039910555,
+            -0.016666175797581673,
+            0.0218023881316185,
+            -0.01730252057313919,
+            0.026802239939570427,
+            0.013825350441038609,
+            -0.015802564099431038,
+            -0.029817303642630577,
+            0.025378039106726646,
+            0.003367325058206916,
+            0.010499690659344196,
+            -0.02075696364045143,
+            0.014332910999655724,
+            0.04393810033798218,
+            -0.02593863010406494,
+            -0.02271145209670067,
+            0.027681002393364906,
+            0.014408666640520096,
+            0.0633314698934555,
+            0.036544378846883774,
+            0.006624804809689522,
+            0.022362977266311646,
+            0.020575150847434998,
+            -0.028968842700123787,
+            -0.01963578537106514,
+            -0.046362269669771194,
+            -0.04757435619831085,
+            0.019878201186656952,
+            -0.04566532000899315,
+            0.0050869714468717575,
+            0.038544319570064545,
+            -0.013704141601920128,
+            -0.014446543529629707,
+            -0.04021093621850014,
+            0.017847958952188492,
+            0.0003908502112608403,
+            0.011976920068264008,
+            -0.0022082682698965073,
+            0.00033285003155469894,
+            -0.011825408786535263,
+            0.008878526277840137,
+            -0.03778676688671112,
+            -0.025196226313710213,
+            0.025181075558066368,
+            0.022862963378429413,
+            0.024120500311255455,
+            0.028802180662751198,
+            -0.03077181987464428,
+            -0.009984553791582584,
+            0.021378157660365105,
+            -0.04242299124598503,
+            -0.006556625012308359,
+            0.011332998983561993,
+            0.0026703758630901575,
+            0.003872992005199194,
+            0.032786913216114044,
+            0.052998438477516174,
+            -0.03172633796930313,
+            0.02065090648829937,
+            -0.010431510396301746,
+            0.04957429692149162,
+            0.00024691506405360997,
+            0.028968842700123787,
+            -0.019514575600624084,
+            -0.0134389977902174,
+            -0.029529431834816933,
+            -0.009795165620744228,
+            -0.01356020662933588,
+            -0.008484598249197006,
+            -0.002998017705976963,
+            -0.009575475007295609,
+            0.0040453351102769375,
+            0.0331808403134346,
+            0.00273855566047132,
+            0.030680913478136063,
+            0.014325335621833801,
+            0.0051475754007697105,
+            -0.08745196461677551,
+            0.026059838011860847,
+            -0.0033616432920098305,
+            0.005859675817191601,
+            0.0062460280023515224,
+            0.03013547509908676,
+            -0.0312718041241169,
+            0.020938776433467865,
+            0.026620427146553993,
+            0.0011145505122840405,
+            0.016969196498394012,
+            0.036847397685050964,
+            -0.04181694984436035,
+            0.009643655270338058,
+            0.012590537779033184,
+            -0.0033881578128784895,
+            0.04584713280200958,
+            0.011628445237874985,
+            0.019514575600624084,
+            0.018529755994677544,
+            0.003569970605894923,
+            0.00297907879576087,
+            0.006533898413181305,
+            0.03724132478237152,
+            0.003462019143626094,
+            -0.009833043441176414,
+            0.014976831153035164,
+            -0.010757259093225002,
+            -0.011219366453588009,
+            -0.037998881191015244,
+            0.014454119838774204,
+            0.008742166683077812,
+            -0.017454031854867935,
+            -0.02959003672003746,
+            0.0026949963066726923,
+            -0.031908150762319565,
+            0.026559824123978615,
+            -0.002445003716275096,
+            0.005738467443734407,
+            -0.010454237461090088,
+            -0.006757376715540886,
+            -0.035423196852207184,
+            0.028453707695007324,
+            -0.02095392718911171,
+            0.0053142374381423,
+            0.0007258309051394463,
+            -0.029438525438308716,
+            0.0060073984786868095,
+            0.017863109707832336,
+            -0.0025965142995119095,
+            0.0025302283465862274,
+            -0.02349930815398693,
+            0.024468975141644478,
+            0.025635607540607452,
+            0.004518806003034115,
+            -0.014037464745342731,
+            -0.013113250024616718,
+            -0.003605954349040985,
+            -0.009742137044668198,
+            -0.016196493059396744,
+            0.024468975141644478,
+            -0.014810169115662575,
+            0.00308892410248518,
+            0.021847840398550034,
+            0.014264730736613274,
+            -0.007045246660709381,
+            -0.004007457755506039,
+            -0.04066546633839607,
+            -0.01218903437256813,
+            -0.04760465770959854,
+            -0.011628445237874985,
+            -0.0034601252991706133,
+            -0.028696123510599136,
+            0.0061513339169323444,
+            -0.04481685906648636,
+            -0.01235569640994072,
+            0.02309022843837738,
+            0.018135828897356987,
+            0.0019488062243908644,
+            -0.009173971600830555,
+            -0.022605394944548607,
+            8.486965816700831e-05,
+            -0.01763584464788437,
+            -0.017044952139258385,
+            0.015393485315144062,
+            0.031059689819812775,
+            -0.03101423755288124,
+            0.04024123772978783,
+            0.003893824527040124,
+            -0.08066429197788239,
+            0.010393633507192135,
+            -0.026362858712673187,
+            0.07011914253234863,
+            -0.0025453795678913593,
+            0.014075342565774918,
+            0.06781618297100067,
+            -0.012431452050805092,
+            -0.02007516659796238,
+            0.03142331540584564,
+            -0.008249756880104542,
+            -1.1311515663692262e-05,
+            -0.02884763479232788,
+            -0.02739313244819641,
+            -0.053271155804395676,
+            -0.009704259224236012,
+            -0.010787560604512691,
+            0.03360506892204285,
+            -0.024317465722560883,
+            -0.0007656024536117911,
+            0.012620840221643448,
+            0.02983245439827442,
+            0.016635872423648834,
+            -0.0015681355725973845,
+            0.008272483013570309,
+            0.02881733328104019,
+            0.0274234339594841,
+            -0.0036154238041490316,
+            0.00814369972795248,
+            -0.034211114048957825,
+            -0.0068066176027059555,
+            -0.0493621826171875,
+            0.011302697472274303,
+            -0.02596893161535263,
+            -0.02596893161535263,
+            0.007643714081496,
+            -0.011476934887468815,
+            0.03748374432325363,
+            -0.024696242064237595,
+            -0.007503566797822714,
+            0.005037730094045401,
+            0.012855681590735912,
+            -0.017757052555680275,
+            -0.03612014651298523,
+            -0.0028502948116511106,
+            0.0026381798088550568,
+            0.007560383062809706,
+            -0.053240854293107986,
+            0.004499867558479309,
+            -0.018469152972102165,
+            0.0279688723385334,
+            -0.02139330841600895,
+            -0.013688990846276283,
+            0.042029064148664474,
+            0.011363301426172256,
+            0.027499189600348473,
+            -0.026468917727470398,
+            0.0017849853029474616,
+            0.012825379148125648,
+            -0.0020056227222085,
+            0.032514192163944244,
+            -0.030574856325984,
+            -0.03257479891180992,
+            0.001548249856568873,
+            -0.007840678095817566,
+            -0.0625436082482338,
+            -0.011386028490960598,
+            -0.00981789268553257,
+            0.017317671328783035,
+            0.0446956530213356,
+            0.04557441547513008,
+            -0.01068150345236063,
+            0.002174178371205926,
+            -0.02213571034371853,
+            -0.04542290419340134,
+            -0.02643861435353756,
+            0.0016486257081851363,
+            -0.019999410957098007,
+            -0.02789311669766903,
+            -0.0018313854234293103,
+            0.011211791075766087,
+            -0.013651113025844097,
+            0.01134057529270649,
+            -0.018560059368610382,
+            -0.01862066239118576,
+            -0.012332970276474953,
+            0.018575210124254227,
+            0.009901223704218864,
+            -0.0018796793883666396,
+            -0.015037435106933117,
+            -0.00980274099856615,
+            0.018969137221574783,
+            -0.011279970407485962,
+            0.0009388927719555795,
+            0.01188601367175579,
+            0.024347767233848572,
+            -0.007083124481141567,
+            0.010961798019707203,
+            -0.034756552428007126,
+            -0.01966608688235283,
+            0.007984613068401814,
+            0.002719616750255227,
+            0.020711511373519897,
+            0.008742166683077812,
+            -0.04275631532073021,
+            -0.044301725924015045,
+            0.016044981777668,
+            -0.0562104657292366,
+            0.028120383620262146,
+            -0.03327174484729767,
+            -0.012514783069491386,
+            -0.02671133354306221,
+            0.0013418165035545826,
+            -0.01828734017908573,
+            0.022438732907176018,
+            0.03778676688671112,
+            -0.0011486404109746218,
+            0.005503625608980656,
+            0.014658658765256405,
+            0.04384719207882881,
+            -0.05021063983440399,
+            -0.041180603206157684,
+            0.03754434734582901,
+            -0.01601468026638031,
+            0.0185146052390337,
+            0.00021744151308666915,
+            0.010348179377615452,
+            -0.014431392773985863,
+            0.022772056981921196,
+            0.013757170177996159,
+            -0.008764892816543579,
+            -0.030574856325984,
+            0.04945308715105057,
+            -0.023650817573070526,
+            -0.049513693898916245,
+            0.0018323323456570506,
+            0.018878230825066566,
+            0.018635815009474754,
+            -0.014870774000883102,
+            -0.02907489985227585,
+            -0.024317465722560883,
+            -0.006060427520424128,
+            -0.00947699323296547,
+            -0.03402930125594139,
+            0.011363301426172256,
+            0.0048445542342960835,
+            0.016666175797581673,
+            -0.00645056739449501,
+            -0.0073141781613230705,
+            -0.009370936080813408,
+            0.022559940814971924,
+            0.012635990977287292,
+            -0.03175663948059082,
+            -0.017484333366155624,
+            -0.02528713271021843,
+            0.004359719809144735,
+            0.010136065073311329,
+            0.022166013717651367,
+            0.024287162348628044,
+            -0.0323626808822155,
+            0.04218057543039322,
+            0.01550711877644062,
+            -0.008727015927433968,
+            -0.0019317612750455737,
+            0.02478714846074581,
+            0.04324115067720413,
+            0.0218023881316185,
+            0.026514369994401932,
+            0.002107892418280244,
+            8.705945947440341e-05,
+            -0.015620751306414604,
+            -0.0011391709558665752,
+            0.01862066239118576,
+            0.04911976307630539,
+            -0.012151156552135944,
+            0.01716616004705429,
+            -0.03578682243824005,
+            0.00396200455725193,
+            -0.02580226957798004,
+            -0.030786970630288124,
+            0.03745344281196594,
+            -0.015060162171721458,
+            0.0191964041441679,
+            0.025044716894626617,
+            -0.022469034418463707,
+            -0.011075431481003761,
+            -0.019423669204115868,
+            -0.026181045919656754,
+            -0.01702980138361454,
+            -0.03281721472740173,
+            -0.018135828897356987,
+            -0.002653330797329545,
+            -0.035635314881801605,
+            -0.04987731948494911,
+            -0.004329417832195759,
+            -0.046392571181058884,
+            -0.010507266037166119,
+            0.017378276214003563,
+            -0.011166337877511978,
+            -0.025711363181471825,
+            -0.005336964037269354,
+            0.003973367623984814,
+            -0.022196315228939056,
+            -0.0066588944755494595,
+            0.008439145050942898,
+            0.035938333719968796,
+            -0.02298417128622532,
+            -0.013060221448540688,
+            -0.0024317465722560883,
+            -0.02887793630361557,
+            -0.007878555916249752,
+            0.008999735116958618,
+            0.007723257411271334,
+            -0.0012357590021565557,
+            -0.05196816474199295,
+            -0.006916462909430265,
+            -0.012992041185498238,
+            0.029196109622716904,
+            0.026468917727470398,
+            0.0012168202083557844,
+            -0.02783251367509365,
+            0.008749742060899734,
+            -0.0072535742074251175,
+            0.02339324913918972,
+            -0.007992188446223736,
+            -0.022908415645360947,
+            -0.02034788578748703,
+            0.030786970630288124,
+            -0.0159086212515831,
+            0.009113367646932602,
+            0.009976978413760662,
+            0.0121360057964921,
+            -0.03899884968996048,
+            -0.001445980160497129,
+            0.0005648507503792644,
+            -0.010128489695489407,
+            0.00423093605786562,
+            -0.006738437805324793,
+            -0.007079336792230606,
+            -0.022090258076786995,
+            -0.0218023881316185,
+            0.03818069398403168,
+            -0.009113367646932602,
+            -0.0035661826841533184,
+            -0.01088604237884283,
+            0.03436262533068657,
+            -0.00595815759152174,
+            -0.01993880607187748,
+            -0.019347915425896645,
+            0.014431392773985863,
+            0.05105910077691078,
+            -0.028529461473226547,
+            -0.03984731063246727,
+            0.0011136035900563002,
+            0.030150625854730606,
+            0.004140029661357403,
+            -0.016908591613173485,
+            0.0035851215943694115,
+            0.056392278522253036,
+            -0.011742077767848969,
+            0.028362801298499107,
+            0.027771908789873123,
+            0.02295386977493763,
+            -0.02021152526140213,
+            -0.01678738370537758,
+            -0.020999381318688393,
+            0.011098157614469528,
+            -0.03027183562517166,
+            0.027059808373451233,
+            0.018969137221574783,
+            0.01538590993732214,
+            0.05054396390914917,
+            -0.03178694099187851,
+            -0.02515077404677868,
+            0.008499749936163425,
+            0.0021722845267504454,
+            0.011007251217961311,
+            -0.007727045100182295,
+            -0.01801462098956108,
+            -0.001950700068846345,
+            -0.014022313989698887,
+            -0.019817598164081573,
+            -0.020938776433467865,
+            0.00954517349600792,
+            0.015696506947278976,
+            0.002649543108418584,
+            0.010863316245377064,
+            0.0016590419691056013,
+            0.01686313934624195,
+            0.01321930717676878,
+            -0.0444229319691658,
+            -7.936555630294606e-05,
+            0.03378688171505928,
+            -0.0293476190418005,
+            -0.012264790013432503,
+            -0.016060132533311844,
+            -0.004439263138920069,
+            -0.01706010289490223,
+            -0.02762039750814438,
+            -0.006874797400087118,
+            -0.01889338344335556,
+            -0.00814369972795248,
+            -0.028196139261126518,
+            -0.009992130100727081,
+            0.01949942484498024,
+            0.022332675755023956,
+            0.001196934375911951,
+            -0.006783891003578901,
+            -0.005901341326534748,
+            0.02092362567782402,
+            0.032514192163944244,
+            0.01952972821891308,
+            0.0058331615291535854,
+            -0.015158643946051598,
+            -0.004295328166335821,
+            -0.012370847165584564,
+            0.016060132533311844,
+            0.012537509202957153,
+            6.344509893096983e-05,
+            0.03493836522102356,
+            0.004170331638306379,
+            -0.022726602852344513,
+            -0.001230077352374792,
+            0.024923507124185562,
+            -0.021590271964669228,
+            -0.0108330138027668,
+            0.040029123425483704,
+            0.0085679292678833,
+            -0.0024506852496415377,
+            -0.019287310540676117,
+            -0.019075196236371994,
+            0.02630225569009781,
+            -0.02133270539343357,
+            0.002083271974697709,
+            0.0482713058590889,
+            0.00035344602656550705,
+            -0.029802151024341583,
+            0.017954016104340553,
+            -0.013242034241557121,
+            -0.018560059368610382,
+            0.01848430372774601,
+            0.0012859469279646873,
+            -0.039968520402908325,
+            0.019014591351151466,
+            0.018408548086881638,
+            0.022635696455836296,
+            -0.02918095886707306,
+            0.0002127068000845611,
+            0.03233237937092781,
+            0.004037759732455015,
+            0.001595596899278462,
+            -0.008530051447451115,
+            -0.0006065162015147507,
+            0.010431510396301746,
+            0.001746160676702857,
+            0.02854461409151554,
+            -0.0030434709042310715,
+            0.027771908789873123,
+            -0.006727074272930622,
+            -0.008863375522196293,
+            0.019181253388524055,
+            -0.02109028771519661,
+            0.013166278600692749,
+            0.0685434341430664,
+            0.0021950111258774996,
+            0.025847723707556725,
+            0.007181606721132994,
+            0.024347767233848572,
+            0.02018122375011444,
+            -0.027271922677755356,
+            -0.02021152526140213,
+            -0.005397567991167307,
+            -0.0014033677289262414,
+            -0.03787767142057419,
+            0.011128460057079792,
+            -0.021302402019500732,
+            -0.013249609619379044,
+            -0.0022347825579345226,
+            -0.002316219499334693,
+            -0.006405114196240902,
+            0.034211114048957825,
+            -0.0014630251098424196,
+            -0.018560059368610382,
+            -0.035423196852207184,
+            0.037726160138845444,
+            0.04048365354537964,
+            0.0389382466673851,
+            -0.005984671879559755,
+            0.03548380360007286,
+            0.0030756669584661722,
+            0.0025302283465862274,
+            0.041453324258327484,
+            0.004791525658220053,
+            -0.011567841283977032,
+            0.02295386977493763,
+            0.012416300363838673,
+            0.03351416438817978,
+            -0.02045394293963909,
+            -0.02630225569009781,
+            -0.003299145260825753,
+            -0.026423463597893715,
+            0.012492056004703045,
+            -0.019726691767573357,
+            0.0018768386216834188,
+            0.007117214612662792,
+            0.015196521766483784,
+            -0.016044981777668,
+            0.014719262719154358,
+            -0.006102093029767275,
+            -0.021014532074332237,
+            0.011719351634383202,
+            -0.007780073676258326,
+            0.011961768381297588,
+            -0.02262054570019245,
+            0.001138224033638835,
+            -0.04063516482710838,
+            0.008280059322714806,
+            0.014022313989698887,
+            -0.01598437689244747,
+            0.03205966204404831,
+            0.020969079807400703,
+            -0.01801462098956108,
+            -0.0009682480012997985,
+            0.0034714885987341404,
+            -0.010393633507192135,
+            -0.011067855171859264,
+            -0.019923655316233635,
+            0.02786281518638134,
+            -0.015802564099431038,
+            -0.025908326730132103,
+            0.02454473078250885,
+            -0.01743887923657894,
+            0.015151068568229675,
+            -0.004299115855246782,
+            0.00888610165566206,
+            -0.019120648503303528,
+            0.02284781076014042,
+            -0.023741723969578743,
+            -0.0009128518868237734,
+            -0.020287280902266502,
+            0.0028389315120875835,
+            0.014135947450995445,
+            -0.018272189423441887,
+            -0.02139330841600895,
+            0.019150950014591217,
+            -0.005420294590294361,
+            -0.003931702114641666,
+            0.010014856234192848,
+            0.0030983935575932264,
+            -0.0029715034179389477,
+            -0.004549108445644379,
+            -0.03281721472740173,
+            -0.02915065549314022,
+            0.0033086147159337997,
+            0.011386028490960598,
+            0.015257125720381737,
+            0.027574945241212845,
+            0.03715042024850845,
+            0.006052851676940918,
+            0.0036949669010937214,
+            0.0009327376610599458,
+            -0.020726662129163742,
+            0.042635105550289154,
+            -0.020469093695282936,
+            0.040847279131412506,
+            0.032544493675231934,
+            -0.0197418425232172,
+            -0.023908386006951332,
+            0.0029052174650132656,
+            0.03115059621632099,
+            0.03151422366499901,
+            0.025211378931999207,
+            0.012310243211686611,
+            0.03690800443291664,
+            -0.004359719809144735,
+            -0.032211173325777054,
+            0.015802564099431038,
+            0.022256920114159584,
+            -0.008105821907520294,
+            0.009673957712948322,
+            0.027438584715127945,
+            0.013279912061989307,
+            -0.007757347077131271,
+            0.030150625854730606,
+            0.0012234487803652883,
+            0.02396899089217186,
+            0.02498411200940609,
+            -0.02190844528377056,
+            0.01743887923657894,
+            -0.0001154677156591788,
+            0.006727074272930622,
+            -0.009840618818998337,
+            0.004348356742411852,
+            0.011810258030891418,
+            0.018272189423441887,
+            0.018908534198999405,
+            0.003246116451919079,
+            -0.07060398161411285,
+            0.019726691767573357,
+            0.004507442936301231,
+            -0.02160542458295822,
+            0.00390140013769269,
+            -0.0024866692256182432,
+            0.012704171240329742,
+            0.02105998620390892,
+            0.012461753562092781,
+            -0.01790856383740902,
+            0.0013626492582261562,
+            0.011961768381297588,
+            -0.00595815759152174,
+            0.04730163514614105,
+            0.0019791084341704845,
+            -0.01437078882008791,
+            -0.016105586662888527,
+            0.013726867735385895,
+            0.02525683119893074,
+            -0.02355991117656231,
+            0.006325571332126856,
+            0.005621046293526888,
+            -0.004147605039179325,
+            -0.010742107406258583,
+            -0.030377892777323723,
+            -0.005280147306621075,
+            -0.03436262533068657,
+            0.023150833323597908,
+            0.015491968020796776,
+            0.014764715917408466,
+            -0.025044716894626617,
+            -0.004280176945030689,
+            0.0046400148421525955,
+            -0.021287251263856888,
+            0.003261267440393567,
+            0.01161329448223114,
+            0.0005104016163386405,
+            -0.020984230563044548,
+            0.00959820207208395,
+            0.013469300232827663,
+            0.06848283112049103,
+            0.0007442962378263474,
+            0.029771849513053894,
+            0.0422411784529686,
+            -0.0012243957025930285,
+            -0.006371024530380964,
+            0.0036286809481680393,
+            -0.019953956827521324,
+            0.003473382443189621,
+            0.025878025218844414,
+            -0.03224147483706474,
+            -0.055058982223272324,
+            -0.007545232307165861,
+            -0.04624105989933014,
+            0.001489539397880435,
+            0.014711687341332436,
+            0.023044776171445847,
+            -0.0037782976869493723,
+            -0.034817155450582504,
+            0.016908591613173485,
+            -0.003071879269555211,
+            0.007014944683760405,
+            -0.02593863010406494,
+            -0.0294839795678854,
+            -0.026014383882284164,
+            -0.02339324913918972,
+            -0.012908710166811943,
+            -0.022726602852344513,
+            0.012590537779033184,
+            -0.0015103721525520086,
+            -0.06587684899568558,
+            -0.009780014865100384,
+            0.016166189685463905,
+            0.001200722181238234,
+            0.011939042247831821,
+            0.011083006858825684,
+            -0.007083124481141567,
+            -0.0018275976181030273,
+            -0.022559940814971924,
+            0.03124150261282921,
+            0.005177877843379974,
+            0.03584742918610573,
+            -0.006503595970571041,
+            -0.009128518402576447,
+            0.01073453202843666,
+            0.04527139291167259,
+            0.0017916138749569654,
+            0.010226971469819546,
+            -0.020560000091791153,
+            0.0015065843472257257,
+            -0.02284781076014042,
+            -0.0031135445460677147,
+            -0.008863375522196293,
+            -0.0021211495622992516,
+            0.015893470495939255,
+            0.0019753205124288797,
+            -0.010204244405031204,
+            0.025438643991947174,
+            0.02251448854804039,
+            -0.009605777449905872,
+            -0.015969226136803627,
+            -0.02437806874513626,
+            -0.014545026235282421,
+            -0.0002627763315103948,
+            -0.010007280856370926,
+            -0.022453883662819862,
+            -0.020878173410892487,
+            -0.002566212322562933,
+            -0.02559015527367592,
+            0.022529639303684235,
+            0.01220418605953455,
+            -4.882668508798815e-05,
+            -0.006723286584019661,
+            -0.038544319570064545,
+            -0.003727162955328822,
+            -0.03118089959025383,
+            -0.005374841392040253,
+            0.03287781774997711,
+            -0.03745344281196594,
+            -0.0002231231628684327,
+            0.014491996727883816,
+            -0.01993880607187748,
+            0.0028938541654497385,
+            0.008961857296526432,
+            0.02769615314900875,
+            -0.027408283203840256,
+            0.0001795638381736353,
+            0.006954340264201164,
+            0.016908591613173485,
+            -0.009878496639430523,
+            0.027196168899536133,
+            -0.04275631532073021,
+            0.011037553660571575,
+            -0.037120118737220764,
+            -0.020423639565706253,
+            -0.01523439958691597,
+            0.008363389410078526,
+            -0.005018791649490595,
+            -0.02637801133096218,
+            -0.010454237461090088,
+            0.006026337388902903,
+            -0.00693918950855732,
+            -0.01902974210679531,
+            -0.010620899498462677,
+            0.027438584715127945,
+            0.00931033119559288,
+            0.007977037690579891,
+            -0.02955973520874977,
+            -0.01631770096719265,
+            0.006280118133872747,
+            0.052089374512434006,
+            -0.01584801822900772,
+            0.014938953332602978,
+            -0.006639956030994654,
+            0.02342355251312256,
+            -0.028484009206295013,
+            -0.01872672140598297,
+            0.0315445251762867,
+            0.013946558348834515,
+            0.007064185570925474,
+            0.028953691944479942,
+            0.027196168899536133,
+            0.027105262503027916,
+            -0.04654408246278763,
+            0.009886072017252445,
+            -0.002613559365272522,
+            -0.008757317438721657,
+            0.03402930125594139,
+            0.01916610263288021,
+            0.022150862962007523,
+            0.0009744031121954322,
+            -0.01730252057313919,
+            0.0139844361692667,
+            0.014885924756526947,
+            0.007681591901928186,
+            -0.014242004603147507,
+            -0.02660527639091015,
+            0.003367325058206916,
+            -0.007992188446223736,
+            0.00047276070108637214,
+            0.020726662129163742,
+            -0.009052763693034649,
+            0.030999086797237396,
+            -0.014605630189180374,
+            -0.02593863010406494,
+            0.008916404098272324,
+            -0.02268115058541298,
+            -0.0041741193272173405,
+            0.0004476667381823063,
+            0.010083036497235298,
+            -0.023847781121730804,
+            -0.04669559374451637,
+            0.019408518448472023,
+            0.028393102809786797,
+            0.005579380784183741,
+            0.01328748743981123,
+            -0.035332292318344116,
+            0.008992158807814121,
+            0.012310243211686611,
+            0.011348150670528412,
+            0.0019847899675369263,
+            0.035362593829631805,
+            -0.00028834378463216126,
+            0.012885984033346176,
+            -0.008620957843959332,
+            0.028362801298499107,
+            0.00667025800794363,
+            -0.0323626808822155,
+            -0.0378473699092865,
+            -0.01513591781258583,
+            0.015257125720381737,
+            0.006692984607070684,
+            -0.011840560473501682,
+            -0.02278720773756504,
+            -0.038817036896944046,
+            0.015787413343787193,
+            -0.012916285544633865,
+            -0.038271598517894745,
+            -0.010158791206777096,
+            -0.009742137044668198,
+            0.016635872423648834,
+            -0.006916462909430265,
+            -0.016802534461021423,
+            -0.016090434044599533,
+            0.005821797996759415,
+            7.161247049225494e-05,
+            0.0244083721190691,
+            -0.001251857029274106,
+            -0.006855858489871025,
+            -0.016393456608057022,
+            -0.01188601367175579,
+            -0.0035074723418802023,
+            0.03602924197912216,
+            -0.014219277538359165,
+            -0.02383263036608696,
+            0.014984406530857086,
+            0.02657497487962246,
+            0.011636020615696907,
+            0.03227177634835243,
+            0.007181606721132994,
+            -0.014022313989698887,
+            0.003016956616193056,
+            -0.033029329031705856,
+            -0.0046400148421525955,
+            0.005196816753596067,
+            -0.006427840795367956,
+            -0.015651054680347443,
+            -0.01815097965300083,
+            -0.015832867473363876,
+            0.030756669119000435,
+            -0.02487805485725403,
+            0.0218023881316185,
+            0.030347591266036034,
+            0.003174148965626955,
+            -0.02915065549314022,
+            0.013598084449768066,
+            -0.0072005451656877995,
+            0.01862066239118576,
+            0.0019014590652659535,
+            0.028014326468110085,
+            -0.007461901288479567,
+            -0.004818039946258068,
+            -0.02501441352069378,
+            -0.03696860745549202,
+            -0.023484155535697937,
+            -0.008817922323942184,
+            0.0027537066489458084,
+            0.020984230563044548,
+            -0.0375746488571167,
+            -0.017757052555680275,
+            0.009855770505964756,
+            0.015757111832499504,
+            -0.03027183562517166,
+            -0.0025908327661454678,
+            -0.010158791206777096,
+            -0.0004055278259329498,
+            0.005140000022947788,
+            -0.02227207086980343,
+            -0.008977008052170277,
+            -0.015726808458566666,
+            -0.0010435298318043351,
+            0.006166485138237476,
+            -0.0007409819518215954,
+            -0.05805889517068863,
+            -0.013969285413622856,
+            0.018635815009474754,
+            0.0026400736533105373,
+            0.008908828720450401,
+            0.015938924625515938,
+            -0.0111739132553339,
+            0.0031533162109553814,
+            -0.026756787672638893,
+            0.023893235251307487,
+            -0.009492144919931889,
+            0.009355784393846989,
+            -0.013742019422352314,
+            0.012514783069491386,
+            0.014014738611876965,
+            0.004795313347131014,
+            0.0010681503918021917,
+            -0.0054051438346505165,
+            0.01601468026638031,
+            0.028377952054142952,
+            -0.023756874725222588,
+            0.011454207822680473,
+            -0.0002514130319468677,
+            -0.0032158142421394587,
+            -0.029196109622716904,
+            0.00509454682469368,
+            0.003931702114641666,
+            -0.005109698045998812,
+            0.016651025041937828,
+            -0.036544378846883774,
+            0.01662072166800499,
+            0.02177208475768566,
+            0.032544493675231934,
+            0.014832896180450916,
+            -0.017287369817495346,
+            -0.011658747680485249,
+            -0.012385998852550983,
+            -0.002624922664836049,
+            0.022181164473295212,
+            0.012120855040848255,
+            -0.01815097965300083,
+            -0.024287162348628044,
+            0.011249667964875698,
+            0.0005283934879116714,
+            -0.01203752402216196,
+            0.010795135982334614,
+            -0.013166278600692749,
+            0.01349202636629343,
+            0.012711746618151665,
+            0.00866641104221344,
+            0.00888610165566206,
+            0.022862963378429413,
+            -0.011878437362611294,
+            -0.017711598426103592,
+            0.00656041270121932,
+            -0.008916404098272324,
+            -0.0034089903347194195,
+            -0.012598113156855106,
+            0.010795135982334614,
+            0.014878349378705025,
+            0.06490717828273773,
+            -0.013143551535904408,
+            0.0009526234352961183,
+            -0.020438792183995247,
+            -0.031938452273607254,
+            -0.0017944546416401863,
+            -0.020908474922180176,
+            0.0111739132553339,
+            0.008370965719223022,
+            0.01598437689244747,
+            0.012022373266518116,
+            -0.0021912232041358948,
+            0.00795431062579155,
+            0.019317612051963806,
+            -0.010423935018479824,
+            7.255941454786807e-05,
+            -0.006503595970571041,
+            0.0013683309080079198,
+            0.0035851215943694115,
+            -0.011719351634383202,
+            -0.0026949963066726923,
+            0.035695917904376984,
+            0.01068150345236063,
+            0.011651171371340752,
+            -0.03445352986454964,
+            0.0020435003098100424,
+            0.020150920376181602,
+            0.007602048572152853,
+            -0.008302785456180573,
+            -0.01563590206205845,
+            0.01706010289490223,
+            -0.015317730605602264,
+            -0.0022499337792396545,
+            0.002888172399252653,
+            -0.010151215828955173,
+            0.04581683129072189,
+            -0.019787294790148735,
+            -0.0011742077767848969,
+            0.031847547739744186,
+            0.0030207443051040173,
+            -0.0040983641520142555,
+            0.005408931523561478,
+            -0.003473382443189621,
+            -0.0069013116881251335,
+            -0.02139330841600895,
+            0.0031381649896502495,
+            0.00667025800794363,
+            -0.019287310540676117,
+            -0.0072005451656877995,
+            -0.007719469256699085,
+            0.006897523999214172,
+            0.023105379194021225,
+            -0.009492144919931889,
+            -0.00016145357221830636,
+            -0.008575504645705223,
+            0.009848194196820259,
+            -0.0011912527261301875,
+            -0.044301725924015045,
+            -0.006147546228021383,
+            0.021181194111704826,
+            0.000973456131760031,
+            0.0005757405888289213,
+            -0.007450537756085396,
+            0.005007428117096424,
+            -0.017423728480935097,
+            -0.008454295806586742,
+            0.02369627170264721,
+            0.01736312545835972,
+            -0.009113367646932602,
+            0.010863316245377064,
+            -0.0035680767614394426,
+            0.00433320552110672,
+            -0.020014561712741852,
+            -0.01095422264188528,
+            0.003462019143626094,
+            0.015272277407348156,
+            -0.0009819786064326763,
+            -0.029029447585344315,
+            -0.0012272365856915712,
+            -0.00024194362049456686,
+            0.02918095886707306,
+            0.01486319862306118,
+            -0.008961857296526432,
+            -0.021211495622992516,
+            -0.013242034241557121,
+            -0.001237652963027358,
+            -0.015726808458566666,
+            -0.009620928205549717,
+            0.011855711229145527,
+            0.0009137988090515137,
+            0.008992158807814121,
+            -0.018135828897356987,
+            -0.027347678318619728,
+            -0.016166189685463905,
+            0.0018465365283191204,
+            -0.012870832346379757,
+            0.003340810537338257,
+            0.015514694154262543,
+            -0.03430201858282089,
+            -0.015060162171721458,
+            -0.03805948421359062,
+            0.008280059322714806,
+            -0.0029658216517418623,
+            -0.02024182677268982,
+            -0.023605365306138992,
+            0.009484568610787392,
+            0.019181253388524055,
+            -0.0015548784285783768,
+            0.019847899675369263,
+            -0.000889651826582849,
+            0.020060013979673386,
+            -0.0073141781613230705,
+            0.0031703610438853502,
+            -0.013007191941142082,
+            0.0030245319940149784,
+            0.008333087898790836,
+            0.02890823967754841,
+            -0.023075077682733536,
+            -0.006503595970571041,
+            -0.004685468040406704,
+            0.004746072459965944,
+            0.012658718042075634,
+            0.014613205567002296,
+            0.020741812884807587,
+            -0.008272483013570309,
+            -0.0027612822595983744,
+            0.012484480626881123,
+            -0.003799130441620946,
+            0.007295239716768265,
+            0.002390081062912941,
+            0.0019677451346069574,
+            0.004162756260484457,
+            0.008477022871375084,
+            0.0053710537031292915,
+            0.027650700882077217,
+            -0.0028275682125240564,
+            0.0031703610438853502,
+            0.024620486423373222,
+            0.02322658896446228,
+            -0.018878230825066566,
+            -0.028074929490685463,
+            0.002024561632424593,
+            0.050695475190877914,
+            0.007245998363941908,
+            -0.006810405291616917,
+            -0.03927157074213028,
+            -0.0031760428100824356,
+            0.028635520488023758,
+            0.023650817573070526,
+            0.01584801822900772,
+            -0.010567869991064072,
+            -0.012098127976059914,
+            0.02772645466029644,
+            0.013295062817633152,
+            0.02616589516401291,
+            -0.025302285328507423,
+            0.010704229585826397,
+            -0.03869583085179329,
+            0.0449683703482151,
+            0.004439263138920069,
+            -0.014476845972239971,
+            0.0011884119594469666,
+            0.010386057198047638,
+            -0.004537744913250208,
+            0.006431628484278917,
+            -5.8059369621332735e-05,
+            -0.0008086882880888879,
+            0.011923890560865402,
+            0.012007221579551697,
+            0.004265025723725557,
+            -0.02399929240345955,
+            0.007359631359577179,
+            0.004272601567208767,
+            0.03005971945822239,
+            0.005431658122688532,
+            -0.0013863227795809507,
+            0.012385998852550983,
+            -0.005810434930026531,
+            0.008855799213051796,
+            -0.010719381272792816,
+            0.020469093695282936,
+            -0.02918095886707306,
+            1.0986003871948924e-05,
+            0.02051454596221447,
+            -0.0015785519499331713,
+            0.0038199631962925196,
+            -0.023741723969578743,
+            0.01186328660696745,
+            -0.052968136966228485,
+            0.021166043356060982,
+            -0.018741872161626816,
+            0.021741783246397972,
+            0.019590331241488457,
+            -0.009734561666846275,
+            -0.012052674777805805,
+            0.03545350208878517,
+            0.0026760573964565992,
+            -0.014264730736613274,
+            -0.010643625631928444,
+            -0.017999470233917236,
+            -0.02830219641327858,
+            0.041968461126089096,
+            -0.0006117243901826441,
+            0.0028275682125240564,
+            0.0024222771171480417,
+            -0.010666352696716785,
+            -0.008923979476094246,
+            0.001954487757757306,
+            0.020499395206570625,
+            -0.03369597718119621,
+            -0.011128460057079792,
+            -0.008689138107001781,
+            -0.02566591091454029,
+            -0.00020300064352340996,
+            -0.011961768381297588,
+            0.026211349293589592,
+            0.01240872498601675,
+            -0.03696860745549202,
+            0.003780191531404853,
+            0.023650817573070526,
+            0.001603172393515706,
+            0.018120678141713142,
+            0.002130619017407298,
+            0.0010795136913657188,
+            0.009742137044668198,
+            -0.010476963594555855,
+            0.026559824123978615,
+            -0.009204274043440819,
+            0.008658835664391518,
+            -0.001950700068846345,
+            -0.017287369817495346,
+            0.017969166859984398,
+            0.004083212930709124,
+            0.024120500311255455,
+            0.011954193003475666,
+            0.011605718173086643,
+            0.01112088467925787,
+            -0.019878201186656952,
+            0.020090317353606224,
+            -0.0018891488434746861,
+            0.009075489826500416,
+            0.024726543575525284,
+            -0.005863463506102562,
+            0.020832719281315804,
+            0.015257125720381737,
+            -0.00020229043730068952,
+            0.011295121163129807,
+            -0.020120618864893913,
+            -0.011779955588281155,
+            0.0085679292678833,
+            -0.009605777449905872,
+            -0.010325453244149685,
+            0.009901223704218864,
+            -0.010105762630701065,
+            -0.015052586793899536,
+            -0.0071247899904847145,
+            0.016635872423648834,
+            0.0028976418543606997,
+            -0.02610529027879238,
+            -1.7326075976598077e-05,
+            0.008901252411305904,
+            0.006166485138237476,
+            0.018741872161626816,
+            0.011317848227918148,
+            -0.006692984607070684,
+            0.010363331064581871,
+            0.011476934887468815,
+            0.0053104497492313385,
+            -0.005079395603388548,
+            -0.013431422412395477,
+            -0.0024241709616035223,
+            0.018090376630425453,
+            0.009810317307710648,
+            -0.01922670565545559,
+            0.004564259201288223,
+            0.0008877579239197075,
+            -0.01420412678271532,
+            -0.019347915425896645,
+            -0.0038483713287860155,
+            -0.004170331638306379,
+            0.0027915844693779945,
+            0.0509984977543354,
+            -0.0006216673064045608,
+            0.007893706671893597,
+            -0.025741664692759514,
+            -0.006299056578427553,
+            -0.00027556007262319326,
+            -0.0036097420379519463,
+            0.01272689737379551,
+            -0.01824188604950905,
+            -0.03724132478237152,
+            0.012711746618151665,
+            -0.0012584856012836099,
+            0.014923802576959133,
+            -0.00297907879576087,
+            0.014226853847503662,
+            -0.017347972840070724,
+            0.01916610263288021,
+            -0.0011117096291854978,
+            -0.016029831022024155,
+            0.0029374135192483664,
+            -0.011143610812723637,
+            -0.021317552775144577,
+            0.005916492082178593,
+            -0.006848283112049103,
+            -0.03896854817867279,
+            0.0029885482508689165,
+            -0.010423935018479824,
+            0.0011798894265666604,
+            -0.003912763670086861,
+            0.020166072994470596,
+            -0.0061513339169323444,
+            0.0008697660523466766,
+            0.02177208475768566,
+            0.0017026013229042292,
+            0.01601468026638031,
+            -0.014499573037028313,
+            0.010999675840139389,
+            0.012598113156855106,
+            0.02210540883243084,
+            -0.032180868089199066,
+            -0.01393140759319067,
+            0.014211702160537243,
+            0.0026116655208170414,
+            -0.005382417235523462,
+            0.020635755732655525,
+            -0.01294658798724413,
+            0.019650936126708984,
+            -0.0005577487172558904,
+            -0.011704200878739357,
+            -0.03524138405919075,
+            -0.016529815271496773,
+            -3.343888238305226e-05,
+            -0.008439145050942898,
+            0.025317436084151268,
+            0.01760554127395153,
+            -0.0108330138027668,
+            0.026468917727470398,
+            0.0037574649322777987,
+            0.014598054811358452,
+            -0.014257155358791351,
+            -0.005999823100864887,
+            -0.0003773563075810671,
+            -0.0119314668700099,
+            0.015264702029526234,
+            -0.03512017801403999,
+            0.0008342557121068239,
+            -0.00839369185268879,
+            0.005242269951850176,
+            -0.0254992488771677,
+            0.018257036805152893,
+            0.025817420333623886,
+            -0.0031419529113918543,
+            -0.005037730094045401,
+            -0.022802358493208885,
+            0.028968842700123787,
+            0.03049910068511963,
+            0.0024828813038766384,
+            -0.0013825349742546678,
+            -0.00352072948589921,
+            0.0033218718599528074,
+            0.0211205892264843,
+            -0.037998881191015244,
+            -0.009098216891288757,
+            -0.028696123510599136,
+            0.042332086712121964,
+            -0.008098246529698372,
+            -0.013431422412395477,
+            0.01862066239118576,
+            -0.0024715180043131113,
+            0.02355991117656231,
+            -0.004689255729317665,
+            -0.004855917301028967,
+            0.03148392215371132,
+            0.011514811776578426,
+            0.02772645466029644,
+            -0.04015033319592476,
+            0.014704111963510513,
+            -0.006867222022265196,
+            0.004867280833423138,
+            -0.013484450988471508,
+            0.005302873905748129,
+            0.02580226957798004,
+            -0.018060073256492615,
+            0.0068066176027059555,
+            0.030665762722492218,
+            0.00748084019869566,
+            0.002448791405186057,
+            0.0165904201567173,
+            0.003890036838129163,
+            0.010855740867555141,
+            0.000841831264551729,
+            0.0052574207074940205,
+            0.022423582151532173,
+            0.014454119838774204,
+            -0.016029831022024155,
+            -0.015120766125619411,
+            -0.00044790346873924136,
+            -0.004450626205652952,
+            0.005261208396404982,
+            0.013242034241557121,
+            0.0061172437854111195,
+            -0.015681356191635132,
+            0.01052999310195446,
+            0.014158673584461212,
+            0.013128400780260563,
+            -0.0039847311563789845,
+            -0.03412020578980446,
+            -0.02443867363035679,
+            -0.021438762545585632,
+            0.003104075090959668,
+            -0.0008281006012111902,
+            -0.0021211495622992516,
+            0.0017954016802832484,
+            0.017347972840070724,
+            -0.0008352026343345642,
+            0.00379155483096838,
+            0.01686313934624195,
+            -0.01415109820663929,
+            -0.022090258076786995,
+            -0.0035964848939329386,
+            -0.027529491111636162,
+            0.0053710537031292915,
+            0.001844642567448318,
+            -0.00965123064815998,
+            0.008530051447451115,
+            -0.0046059247106313705,
+            -0.007223271764814854,
+            -0.012196609750390053,
+            -0.02007516659796238,
+            0.005780132487416267,
+            0.016878290101885796,
+            0.005730891600251198,
+            -0.009272454306483269,
+            -0.003939277958124876,
+            0.011575416661798954,
+            0.017484333366155624,
+            0.017317671328783035,
+            -0.009295180439949036,
+            0.009628504514694214,
+            0.0004022135108243674,
+            0.004670316819101572,
+            -0.01834794320166111,
+            0.0045869858004152775,
+            -0.0027953721582889557,
+            -0.008658835664391518,
+            0.01105270441621542,
+            -0.00031201730598695576,
+            0.01628739945590496,
+            -0.013090522959828377,
+            0.005624833982437849,
+            0.010333028621971607,
+            -0.020590301603078842,
+            -0.0067308624275028706,
+            -0.01692374423146248,
+            0.026484068483114243,
+            -0.00504530593752861,
+            0.00045405857963487506,
+            -0.02978700026869774,
+            0.0033616432920098305,
+            0.000848933297675103,
+            -0.000845619011670351,
+            -0.012764775194227695,
+            0.010749682784080505,
+            0.03242328763008118,
+            0.0005179771687835455,
+            0.0216660276055336,
+            -0.0036154238041490316,
+            0.0007480839849449694,
+            -0.00785582885146141,
+            0.004977126140147448,
+            -0.004916521720588207,
+            -0.009901223704218864,
+            -0.010863316245377064,
+            0.021726632490754128,
+            -0.0033029329497367144,
+            0.0003356908564455807,
+            0.011332998983561993,
+            -0.012067826464772224,
+            0.0014620780711993575,
+            -0.0047687990590929985,
+            0.00971941091120243,
+            0.0061702728271484375,
+            0.009113367646932602,
+            -0.01176480483263731,
+            0.05451354384422302,
+            -0.0106284748762846,
+            0.005537715740501881,
+            -0.00439380994066596,
+            -0.022226616740226746,
+            -0.017317671328783035,
+            0.012711746618151665,
+            -0.017544938251376152,
+            0.03375658020377159,
+            0.024150803685188293,
+            0.0006467612111009657,
+            0.010393633507192135,
+            -0.006958128418773413,
+            0.023923536762595177,
+            -0.002602196065708995,
+            0.0010795136913657188,
+            -0.004855917301028967,
+            0.015802564099431038,
+            0.04511988162994385,
+            0.017393426969647408,
+            -0.001152428099885583,
+            0.00829521007835865,
+            0.020575150847434998,
+            0.00814369972795248,
+            -0.0020075165666639805,
+            0.014219277538359165,
+            -0.01024212222546339,
+            0.00900731049478054,
+            -0.004458202049136162,
+            0.027438584715127945,
+            0.00814369972795248,
+            -0.006310420110821724,
+            -0.0036836036015301943,
+            -0.0008209985098801553,
+            0.014173824340105057,
+            0.006950552575290203,
+            0.01963578537106514,
+            -0.00461350055411458,
+            -0.0012575386790558696,
+            0.013870803639292717,
+            0.014491996727883816,
+            -0.00552635220810771,
+            0.001724380999803543,
+            -0.01435563713312149,
+            -0.010204244405031204,
+            0.013575357384979725,
+            0.006086941808462143,
+            0.006310420110821724,
+            -0.014514723792672157,
+            -0.00980274099856615,
+            -0.004670316819101572,
+            0.0015435151290148497,
+            0.015052586793899536,
+            0.011060279794037342,
+            -0.004814252257347107,
+            0.006052851676940918,
+            0.013825350441038609,
+            -0.014014738611876965,
+            0.006147546228021383,
+            0.007208121009171009,
+            0.008855799213051796,
+            0.004787737503647804,
+            -0.010817863047122955,
+            -0.00428396463394165,
+            -0.0063937511295080185,
+            0.0012670081341639161,
+            0.016832837834954262,
+            0.013181429356336594,
+            0.0049581872299313545,
+            0.027529491111636162,
+            -0.00839369185268879,
+            -0.012302667833864689,
+            0.02339324913918972,
+            0.011257244274020195,
+            0.0031703610438853502,
+            0.014651083387434483,
+            -0.012635990977287292,
+            -0.012446602806448936,
+            0.006655106786638498,
+            0.004848341923207045,
+            0.006049063988029957,
+            0.009151245467364788,
+            0.00450365524739027,
+            0.007333117071539164,
+            -0.007098275702446699,
+            -0.007427811156958342,
+            0.0014810169814154506,
+            0.01642375811934471,
+            0.020393338054418564,
+            -0.024968961253762245,
+            -0.0018806264270097017,
+            0.0009052763343788683,
+            0.010878467001020908,
+            -0.003316190093755722,
+            0.012310243211686611,
+            5.65205882594455e-05,
+            -0.0074240234680473804,
+            0.010158791206777096,
+            -0.02119634486734867,
+            -0.012681444175541401,
+            0.0012717428617179394,
+            0.013037494383752346,
+            -0.019590331241488457,
+            0.018226735293865204,
+            -0.008893677033483982,
+            -0.013348091393709183,
+            0.013893529772758484,
+            0.020014561712741852,
+            0.004090788308531046,
+            -0.007738408166915178,
+            -0.021862991154193878,
+            -0.01127239502966404,
+            0.016060132533311844,
+            0.020272130146622658,
+            0.000947888707742095,
+            -0.0006472347304224968,
+            0.010173942893743515,
+            -0.013113250024616718,
+            -0.023605365306138992,
+            -0.005230906419456005,
+            0.010386057198047638,
+            0.01746918261051178,
+            0.018832778558135033,
+            -0.006685409229248762,
+            0.02443867363035679,
+            -0.0059127043932676315,
+            -0.015499543398618698,
+            2.86597842205083e-05,
+            -0.02802947722375393,
+            -0.017423728480935097,
+            0.00013778003631159663,
+            0.00246773031540215,
+            0.008598231710493565,
+            0.0008555619278922677,
+            0.01943882182240486,
+            0.000904329412151128,
+            0.017590390518307686,
+            -0.000659071491099894,
+            -0.002009410411119461,
+            -0.0013995799235999584,
+            -0.0015340456739068031,
+            0.02254479005932808,
+            -0.012166308239102364,
+            -0.01176480483263731,
+            0.022302372381091118,
+            0.006852070800960064,
+            0.014719262719154358,
+            -0.027635548263788223,
+            0.013757170177996159,
+            -0.015499543398618698,
+            0.017423728480935097,
+            -0.03408990427851677,
+            0.006420265417546034,
+            0.01489350013434887,
+            -0.010348179377615452,
+            -0.020499395206570625,
+            -0.029650641605257988,
+            -0.024150803685188293,
+            -0.0038710979279130697,
+            -0.010855740867555141,
+            -0.016847988590598106,
+            0.006121031939983368,
+            -0.005802859086543322,
+            0.018499454483389854,
+            -0.007079336792230606,
+            0.033877789974212646,
+            0.006969491485506296,
+            -0.004056698642671108,
+            0.003835114184767008,
+            0.022469034418463707,
+            -0.006102093029767275,
+            0.02109028771519661,
+            0.004408961161971092,
+            -0.001467759720981121,
+            0.013681414537131786,
+            -0.006552837323397398,
+            -0.002361672930419445,
+            0.005821797996759415,
+            -0.010484539903700352,
+            0.022014502435922623,
+            0.005924067925661802,
+            -0.007708106189966202,
+            -0.0006775368237867951,
+            -0.011454207822680473,
+            0.014726839028298855,
+            0.006530110724270344,
+            0.0022480397019535303,
+            -0.011295121163129807,
+            0.011249667964875698,
+            -0.004859705455601215,
+            0.0040642740204930305,
+            0.009484568610787392,
+            -0.01774190180003643,
+            -0.028862785547971725,
+            0.011416330002248287,
+            0.019120648503303528,
+            0.006382387597113848,
+            -0.0055755930952727795,
+            -0.0025681061670184135,
+            0.00839369185268879,
+            0.010651201009750366,
+            0.022150862962007523,
+            -0.012984465807676315,
+            -8.889415767043829e-05,
+            0.013204156421124935,
+            -0.007170243188738823,
+            -0.01056029461324215,
+            0.00487485621124506,
+            0.023438703268766403,
+            0.0068823727779090405,
+            -0.04327145218849182,
+            -0.018469152972102165,
+            0.01909034699201584,
+            0.007136153522878885,
+            -0.04154422879219055,
+            0.0026381798088550568,
+            -0.003689285134896636,
+            -0.023787178099155426,
+            0.016105586662888527,
+            -0.0011817833874374628,
+            -0.01828734017908573,
+            0.028756728395819664,
+            0.02830219641327858,
+            0.01834794320166111,
+            2.6987840101355687e-05,
+            -0.013332940638065338,
+            -0.0318172425031662,
+            -0.023590214550495148,
+            0.008280059322714806,
+            -0.02660527639091015,
+            -0.0072725131176412106,
+            0.011242092587053776,
+            -0.017454031854867935,
+            0.02972639724612236,
+            -0.008507325313985348,
+            0.026938600465655327,
+            0.011446632444858551,
+            0.001691238023340702,
+            -0.013446573168039322,
+            -0.018802475184202194,
+            -0.01350717805325985,
+            -0.008196728304028511,
+            0.014090493321418762,
+            0.017923714593052864,
+            0.011007251217961311,
+            0.011332998983561993,
+            0.02251448854804039,
+            -0.012158732861280441,
+            0.007969462312757969,
+            -2.888172457460314e-05,
+            -0.008961857296526432,
+            0.0026343921199440956,
+            -0.018393397331237793,
+            0.00868156272917986,
+            0.0062308767810463905,
+            0.005458172410726547,
+            -0.007208121009171009,
+            0.010742107406258583,
+            0.02712041325867176,
+            -0.018847929313778877,
+            0.011401179246604443,
+            -0.031362712383270264,
+            -0.014787442982196808,
+            -0.01056029461324215,
+            0.026862844824790955,
+            0.007757347077131271,
+            -0.015726808458566666,
+            -0.007514929864555597,
+            -0.01545409020036459,
+            0.012885984033346176,
+            0.0032442226074635983,
+            0.005764981731772423,
+            0.015817714855074883,
+            0.021893294528126717,
+            0.01746918261051178,
+            -0.00795431062579155,
+            0.006852070800960064,
+            0.004674104508012533,
+            0.028090080246329308,
+            -0.01777220331132412,
+            0.020802417770028114,
+            0.009954252280294895,
+            -0.003935490269213915,
+            0.005685438401997089,
+            -0.014257155358791351,
+            0.008530051447451115,
+            -0.007325541693717241,
+            0.030514251440763474,
+            0.0033067206386476755,
+            0.014075342565774918,
+            -0.02062060497701168,
+            0.006575563922524452,
+            -0.03172633796930313,
+            0.009022461250424385,
+            0.013704141601920128,
+            0.012976890429854393,
+            -0.007677804213017225,
+            -0.024181105196475983,
+            -0.0119314668700099,
+            0.0057990713976323605,
+            -0.01279507763683796,
+            -0.01046181283891201,
+            -0.0034828518982976675,
+            0.012446602806448936,
+            0.018847929313778877,
+            -0.01848430372774601,
+            -0.019923655316233635,
+            -0.009583051316440105,
+            0.02898399345576763,
+            -0.015787413343787193,
+            -0.001471547526307404,
+            0.02413565292954445,
+            0.021241798996925354,
+            0.015022284351289272,
+            0.002155239460989833,
+            0.020090317353606224,
+            -0.0007111532613635063,
+            -0.0237265732139349,
+            -0.02813553437590599,
+            0.007302815094590187,
+            -0.006912675220519304,
+            0.01943882182240486,
+            -0.0035548193845897913,
+            -0.005658924113959074,
+            -0.0034033088013529778,
+            -0.007503566797822714,
+            -0.0165904201567173,
+            -0.01787826046347618,
+            0.0021647089160978794,
+            -0.008454295806586742,
+            0.005863463506102562,
+            -0.0002795845502987504,
+            -0.013105674646794796,
+            0.013401119969785213,
+            -0.014075342565774918,
+            0.005772557109594345,
+            -0.002653330797329545,
+            0.001962063368409872,
+            0.0035926972050219774,
+            0.009113367646932602,
+            -0.007336904760450125,
+            -0.06381630152463913,
+            -0.013105674646794796,
+            -0.003704436356201768,
+            0.006071790587157011,
+            -0.00019578021601773798,
+            -0.016635872423648834,
+            -0.013196581043303013,
+            0.01706010289490223,
+            0.008863375522196293,
+            -0.013870803639292717,
+            0.016954045742750168,
+            0.009961827658116817,
+            0.003622999181970954,
+            -0.024044746533036232,
+            -0.0008148433989845216,
+            0.008757317438721657,
+            0.032756611704826355,
+            -0.008643684908747673,
+            -0.0003839849086944014,
+            0.0036684523802250624,
+            -0.0033332351595163345,
+            -0.010052734054625034,
+            -0.014582903124392033,
+            -0.011848135851323605,
+            0.011908739805221558,
+            0.01790856383740902,
+            -0.014264730736613274,
+            -0.02007516659796238,
+            -0.0007106798002496362,
+            0.012931437231600285,
+            -0.026590125635266304,
+            0.0007244104635901749,
+            -0.007787649054080248,
+            -0.01052999310195446,
+            0.018302490934729576,
+            -0.019014591351151466,
+            0.005249845329672098,
+            0.00048294031876139343,
+            0.008507325313985348,
+            -0.023544760420918465,
+            -0.024893205612897873,
+            0.0035131541080772877,
+            -0.023847781121730804,
+            0.03612014651298523,
+            -0.021590271964669228,
+            -0.012082977220416069,
+            0.034483831375837326,
+            -0.0037423139438033104,
+            0.01702980138361454,
+            0.0023995505180209875,
+            0.02739313244819641,
+            -0.011439057067036629,
+            -0.00617784820497036,
+            0.012499631382524967,
+            0.009575475007295609,
+            0.0032593735959380865,
+            0.03775646165013313,
+            -0.01257538702338934,
+            -0.013901105150580406,
+            -0.005556654185056686,
+            -0.012954163365066051,
+            0.0006775368237867951,
+            0.014431392773985863,
+            0.028393102809786797,
+            0.004257450345903635,
+            -0.00628769351169467,
+            -0.006143758539110422,
+            -0.0014649189542979002,
+            -0.004242299124598503,
+            -0.00947699323296547,
+            -0.00715130427852273,
+            0.017287369817495346,
+            -0.025241680443286896,
+            0.016711628064513206,
+            -0.006420265417546034,
+            0.021135739982128143,
+            -0.011757229454815388,
+            0.007007369305938482,
+            -0.008961857296526432,
+            0.003492321353405714,
+            0.0023635667748749256,
+            -0.005109698045998812,
+            0.009287605062127113,
+            -0.014772292226552963,
+            0.008454295806586742,
+            0.013742019422352314,
+            0.0011183382011950016,
+            -0.010590597055852413,
+            -0.019272159785032272,
+            0.0022972808219492435,
+            -0.000710206339135766,
+            0.01181783340871334,
+            -0.0047498601488769054,
+            -0.000981031684204936,
+            0.0002921315317507833,
+            -0.014476845972239971,
+            -0.009575475007295609,
+            0.0006050958181731403,
+            0.005302873905748129,
+            -0.008961857296526432,
+            0.010378481820225716,
+            -0.005693014245480299,
+            -0.00034160923678427935,
+            0.02627195231616497,
+            0.006526322569698095,
+            -0.007632351014763117,
+            -0.014098069630563259,
+            0.01563590206205845,
+            0.015370759181678295,
+            -0.007454325910657644,
+            0.01933276280760765,
+            -0.016196493059396744,
+            -0.0008366230758838356,
+            0.013916256837546825,
+            -0.009219424799084663,
+            0.0017698341980576515,
+            -0.008196728304028511,
+            0.03554440662264824,
+            -0.0008787619881331921,
+            -0.0035415622405707836,
+            -0.0006401326390914619,
+            0.012976890429854393,
+            -0.0026779514737427235,
+            0.001182730309665203,
+            0.01132542360574007,
+            0.005321812815964222,
+            0.0185146052390337,
+            0.00328020635060966,
+            -0.03378688171505928,
+            -0.007780073676258326,
+            -0.0065490491688251495,
+            -0.008090670220553875,
+            0.009022461250424385,
+            0.025196226313710213,
+            0.01990850456058979,
+            0.006408901885151863,
+            0.001881573349237442,
+            0.018423698842525482,
+            0.015757111832499504,
+            -0.01489350013434887,
+            0.016363153234124184,
+            0.02802947722375393,
+            -0.010030006989836693,
+            0.022741753607988358,
+            -0.005253633018583059,
+            -0.008893677033483982,
+            0.008946705609560013,
+            -0.0015378334792330861,
+            -0.017393426969647408,
+            -0.015681356191635132,
+            0.005022579338401556,
+            -0.008999735116958618,
+            -0.0022897052112966776,
+            -0.018605511635541916,
+            0.004670316819101572,
+            0.017105557024478912,
+            -0.004965762607753277,
+            -3.5243989259470254e-05,
+            0.019484274089336395,
+            -0.006965703796595335,
+            -0.016908591613173485,
+            0.006325571332126856,
+            -0.015007133595645428,
+            0.007946735247969627,
+            -0.013666263781487942,
+            0.010492115281522274,
+            -0.006098304875195026,
+            0.018817627802491188,
+            0.005446809343993664,
+            -0.008090670220553875,
+            -0.0008958069374784827,
+            -0.018984289839863777,
+            0.02065090648829937,
+            -0.033089932054281235,
+            0.00839369185268879,
+            -0.026226500049233437,
+            -0.0009781909175217152,
+            0.0008849170990288258,
+            0.014605630189180374,
+            -0.001218714052811265,
+            0.019696388393640518,
+            -0.002751812804490328,
+            -0.020090317353606224,
+            -0.007086912170052528,
+            -0.0010634156642481685,
+            -0.01618134044110775,
+            -0.030923331156373024,
+            -0.004272601567208767,
+            0.0036267871037125587,
+            -0.004711982328444719,
+            0.0021135739516466856,
+            0.017984317615628242,
+            -0.005613470915704966,
+            0.009173971600830555,
+            -0.02690829709172249,
+            -0.007416448090225458,
+            -0.00355103169567883,
+            0.0010776197304949164,
+            -0.011242092587053776,
+            0.0010596278589218855,
+            0.011226941831409931,
+            -0.019044892862439156,
+            0.013461724855005741,
+            -0.021378157660365105,
+            -0.004371083341538906,
+            -0.008507325313985348,
+            -0.00135412672534585,
+            -0.013446573168039322,
+            -0.028120383620262146,
+            0.025029566138982773,
+            -0.00021495578403118998,
+            -0.003528305096551776,
+            -0.011999646201729774,
+            0.023181134834885597,
+            0.002914686920121312,
+            0.027817361056804657,
+            -0.03157482668757439,
+            0.0013427634257823229,
+            0.011552689597010612,
+            0.0025984083767980337,
+            -0.017287369817495346,
+            0.004746072459965944,
+            -0.005234694108366966,
+            0.02210540883243084,
+            -0.0011694730492308736,
+            -0.0005340751959010959,
+            -0.008234606124460697,
+            0.030999086797237396,
+            0.009136094711720943,
+            0.0012944694608449936,
+            -0.013226882554590702,
+            -0.008060368709266186,
+            -0.002759388415142894,
+            -0.0011069749016314745,
+            0.012855681590735912,
+            0.01290113478899002,
+            -0.012105703353881836,
+            0.0013607552973553538,
+            -0.006878585088998079,
+            0.0011694730492308736,
+            -0.002935519441962242,
+            -0.0004966709529981017,
+            -0.01149208564311266,
+            0.003257479751482606,
+            0.016272246837615967,
+            -0.00780280027538538,
+            -0.006480869371443987,
+            0.0007750718505121768,
+            0.009181547909975052,
+            0.012461753562092781,
+            -0.012742049060761929,
+            -0.018226735293865204,
+            -0.011908739805221558,
+            0.016029831022024155,
+            0.010522416792809963,
+            -0.008204303681850433,
+            -0.013234458863735199,
+            -0.02583257108926773,
+            0.01169662456959486,
+            -0.03293842449784279,
+            -0.012242063879966736,
+            0.005821797996759415,
+            0.028559764847159386,
+            0.019999410957098007,
+            0.012226912193000317,
+            0.001332347048446536,
+            0.002950670663267374,
+            0.00023342114582192153,
+            -0.01916610263288021,
+            0.014408666640520096,
+            -0.011143610812723637,
+            -0.018166130408644676,
+            0.0075906855054199696,
+            0.011939042247831821,
+            0.03148392215371132,
+            0.006942977197468281,
+            -0.0010160685051232576,
+            0.01686313934624195,
+            -0.001218714052811265,
+            -0.013582932762801647,
+            0.01420412678271532,
+            -0.024256860837340355,
+            0.016438908874988556,
+            0.02860521711409092,
+            -0.004780162125825882,
+            0.0008531945641152561,
+            -0.020938776433467865,
+            0.00482561532407999,
+            -0.011529963463544846,
+            -0.002177966060116887,
+            -0.003287781961262226,
+            -0.03460504114627838,
+            0.007946735247969627,
+            -0.006825556512922049,
+            0.01046181283891201,
+            -0.00814369972795248,
+            0.006655106786638498,
+            -0.012431452050805092,
+            0.004253662656992674,
+            0.01513591781258583,
+            0.016105586662888527,
+            0.007060397882014513,
+            -0.025211378931999207,
+            -0.009196698665618896,
+            0.008105821907520294,
+            -0.01349202636629343,
+            -0.02295386977493763,
+            -0.006852070800960064,
+            0.009840618818998337,
+            -0.01061332318931818,
+            -0.00488243205472827,
+            -0.028180986642837524,
+            -0.013681414537131786,
+            -0.0014327229000627995,
+            -0.017817657440900803,
+            -0.005556654185056686,
+            -0.003100287402048707,
+            0.01571165770292282,
+            -0.0049733384512364864,
+            -0.006185423582792282,
+            0.004742284305393696,
+            -0.023878084495663643,
+            0.0037423139438033104,
+            -0.008280059322714806,
+            0.0008266801596619189,
+            -0.0032517979852855206,
+            -0.0014762822538614273,
+            -0.007969462312757969,
+            -0.014332910999655724,
+            0.006893736310303211,
+            -0.016469212248921394,
+            0.002954458352178335,
+            -0.015173795633018017,
+            -0.016499513760209084,
+            -0.00682934420183301,
+            0.00775355938822031,
+            0.021575121209025383,
+            0.006541473791003227,
+            0.021484214812517166,
+            0.005469535943120718,
+            0.028680972754955292,
+            0.0003953482082579285,
+            0.003041577059775591,
+            -8.958956641436089e-06,
+            0.003232859307900071,
+            0.019347915425896645,
+            0.006571775767952204,
+            0.018984289839863777,
+            -0.008742166683077812,
+            0.010514841414988041,
+            -0.010355755686759949,
+            0.010014856234192848,
+            0.006333146709948778,
+            -0.006473293993622065,
+            -0.003799130441620946,
+            -0.010348179377615452,
+            -0.0031419529113918543,
+            -0.016544966027140617,
+            0.0027366618160158396,
+            0.00693161366507411,
+            0.0071588801220059395,
+            -0.011582992039620876,
+            -0.005230906419456005,
+            -0.014704111963510513,
+            -0.0064354161731898785,
+            -0.01831764169037342,
+            0.014385939575731754,
+            0.029226411134004593,
+            -0.016332851722836494,
+            -0.01977214403450489,
+            -0.007772498298436403,
+            0.0007296186522580683,
+            -0.006636167876422405,
+            -0.01240872498601675,
+            -0.009151245467364788,
+            -0.035059571266174316,
+            -0.017666146159172058,
+            0.007586897816509008,
+            0.02484775334596634,
+            0.01387837901711464,
+            -0.013628385961055756,
+            -0.0006983695784583688,
+            -0.016984347254037857,
+            -0.010787560604512691,
+            -0.01535560842603445,
+            -0.02054484933614731,
+            -0.02498411200940609,
+            0.0077952248975634575,
+            -0.004033972043544054,
+            0.014931377954781055,
+            0.008113397285342216,
+            0.0032631612848490477,
+            -0.004371083341538906,
+            -0.0024109138175845146,
+            -0.013196581043303013,
+            0.0012925754999741912,
+            -0.004787737503647804,
+            -0.0017291157273575664,
+            0.03554440662264824,
+            -0.008098246529698372,
+            -0.005871038883924484,
+            0.011060279794037342,
+            0.011128460057079792,
+            -0.013764745555818081,
+            0.0006642796797677875,
+            -0.011249667964875698,
+            0.021847840398550034,
+            -0.021620575338602066,
+            0.019257009029388428,
+            0.0014507147716358304,
+            -0.010840589180588722,
+            0.028999146074056625,
+            0.01892368495464325,
+            0.00743159931153059,
+            0.006579351611435413,
+            0.01672677882015705,
+            0.0044316877610981464,
+            0.016196493059396744,
+            -0.0026552248746156693,
+            -0.004632438998669386,
+            0.009295180439949036,
+            0.015681356191635132,
+            -0.000702630786690861,
+            -0.006942977197468281,
+            0.016438908874988556,
+            0.004056698642671108,
+            0.0075414446182549,
+            -0.01356020662933588,
+            -0.008961857296526432,
+            0.001507531269453466,
+            0.006257391534745693,
+            -0.005939218681305647,
+            -0.0026003022212535143,
+            0.005530139897018671,
+            0.003943065647035837,
+            -0.013302638195455074,
+            0.01689344085752964,
+            -0.016166189685463905,
+            0.008742166683077812,
+            -0.0037972365971654654,
+            0.0121360057964921,
+            -0.006799042224884033,
+            0.0013153020991012454,
+            -0.01818128302693367,
+            -0.003869204083457589,
+            0.0038597346283495426,
+            0.012173883616924286,
+            -0.01262841559946537,
+            0.025484098121523857,
+            0.008166425861418247,
+            0.006602078210562468,
+            0.0059467945247888565,
+            -0.012855681590735912,
+            0.027211319655179977,
+            0.005969521123915911,
+            0.009886072017252445,
+            -0.008893677033483982,
+            0.011136035434901714,
+            0.035362593829631805,
+            0.03060515783727169,
+            0.014135947450995445,
+            0.004469565115869045,
+            -0.018999440595507622,
+            0.015120766125619411,
+            0.0018427487229928374,
+            0.0023806116078048944,
+            0.00863610953092575,
+            -0.011348150670528412,
+            0.01305264513939619,
+            -0.009484568610787392,
+            0.015241974964737892,
+            -0.021272100508213043,
+            -0.011749654076993465,
+            -0.014408666640520096,
+            -0.021075136959552765,
+            -0.01430260855704546,
+            0.034180812537670135,
+            -0.002346521709114313,
+            -0.010393633507192135,
+            0.0045112306252121925,
+            0.008477022871375084,
+            -0.006079366430640221,
+            0.0053142374381423,
+            -0.008204303681850433,
+            -0.013749594800174236,
+            -0.01889338344335556,
+            0.006628592498600483,
+            -0.01550711877644062,
+            0.007651289459317923,
+            -0.003304826794192195,
+            0.0037688282318413258,
+            0.007014944683760405,
+            0.018105527386069298,
+            0.00530666159465909,
+            -0.00135412672534585,
+            0.0014033677289262414,
+            0.0016865032957866788,
+            -0.00931033119559288,
+            0.0108330138027668,
+            0.015408637002110481,
+            -0.0011921997647732496,
+            -0.011279970407485962,
+            -0.009136094711720943,
+            -0.042877525091171265,
+            0.0006264019757509232,
+            -0.013128400780260563,
+            0.0064354161731898785,
+            -0.011196639388799667,
+            0.0022234192583709955,
+            0.015863168984651566,
+            -0.0054051438346505165,
+            -0.0010918239131569862,
+            0.02193874679505825,
+            0.013014768250286579,
+            0.021075136959552765,
+            0.02065090648829937,
+            0.014188976027071476,
+            -0.015302578918635845,
+            -0.03469594568014145,
+            0.002221525413915515,
+            -0.010030006989836693,
+            0.013772320933640003,
+            0.004238511435687542,
+            0.016060132533311844,
+            -0.023893235251307487,
+            -0.011878437362611294,
+            -0.006662682630121708,
+            0.014431392773985863,
+            0.01491622719913721,
+            -0.0032271775417029858,
+            0.02690829709172249,
+            0.009280029684305191,
+            0.0020510759204626083,
+            -0.02281750924885273,
+            0.003312402404844761,
+            0.005321812815964222,
+            -0.008370965719223022,
+            -0.016681326553225517,
+            -0.0004552422615233809,
+            -0.0012168202083557844,
+            -0.0007817004225216806,
+            -0.021650876849889755,
+            0.018832778558135033,
+            -0.004018820822238922,
+            0.0037574649322777987,
+            0.013840501196682453,
+            0.01516621932387352,
+            0.017984317615628242,
+            0.008370965719223022,
+            0.001497115008533001,
+            -0.006605865899473429,
+            -0.0007973249885253608,
+            -0.0005828426801599562,
+            -0.007389933802187443,
+            -0.005598319694399834,
+            0.006382387597113848,
+            -0.01134057529270649,
+            0.008810346014797688,
+            0.000896753859706223,
+            0.006113456096500158,
+            0.010605747811496258,
+            -0.004836978856474161,
+            -0.011628445237874985,
+            0.014628356322646141,
+            -0.0010548931313678622,
+            -0.0055755930952727795,
+            -0.011060279794037342,
+            0.003624893259257078,
+            0.013575357384979725,
+            -0.003096499713137746,
+            -0.013499601744115353,
+            0.00910579226911068,
+            0.002009410411119461,
+            -0.027438584715127945,
+            0.015499543398618698,
+            0.010295150801539421,
+            -0.013370818458497524,
+            0.004980913829058409,
+            -0.003371112747117877,
+            -0.03027183562517166,
+            0.004791525658220053,
+            0.002660906407982111,
+            -0.010643625631928444,
+            0.028938541188836098,
+            -0.003821857040748,
+            0.029256712645292282,
+            -0.004980913829058409,
+            -0.006905099377036095,
+            -0.011355726048350334,
+            0.009416389279067516,
+            -0.018681267276406288,
+            -0.0037953427527099848,
+            -0.008727015927433968,
+            -0.00915882084518671,
+            -0.0009838725673034787,
+            -0.005185453221201897,
+            0.006984642706811428,
+            0.005121061112731695,
+            -0.012704171240329742,
+            0.006757376715540886,
+            -0.010257272981107235,
+            -0.00020856392802670598,
+            0.0006718551740050316,
+            -0.008408842608332634,
+            0.013885954394936562,
+            0.005996035411953926,
+            -0.008340663276612759,
+            -0.0005032995832152665,
+            0.027817361056804657,
+            0.05051366239786148,
+            -0.018863080069422722,
+            0.0009270560112781823,
+            0.01452229917049408,
+            -0.0028824908658862114,
+            -0.0021590273827314377,
+            -0.011976920068264008,
+            0.005821797996759415,
+            -0.019423669204115868,
+            0.013317788951098919,
+            0.006473293993622065,
+            -0.008833073079586029,
+            0.003202557098120451,
+            -0.016711628064513206,
+            0.002621134975925088,
+            -0.0018058179412037134,
+            -0.018681267276406288,
+            -0.015529844909906387,
+            -0.012060250155627728,
+            -0.024181105196475983,
+            0.0004235197266098112,
+            -0.0008546149474568665,
+            -0.013870803639292717,
+            0.006579351611435413,
+            0.008431569673120975,
+            0.032483890652656555,
+            0.0011912527261301875,
+            0.010128489695489407,
+            -2.1513333194889128e-05,
+            -0.0010416359873488545,
+            -0.005988460034132004,
+            -0.01523439958691597,
+            0.00015387803432531655,
+            -0.002342734020203352,
+            0.022590242326259613,
+            0.005549078807234764,
+            0.017560089007019997,
+            0.0017887729918584228,
+            -0.016514664515852928,
+            -0.00596573343500495,
+            -0.007408872712403536,
+            -0.018060073256492615,
+            0.0062649669125676155,
+            -0.005473323632031679,
+            0.009605777449905872,
+            -0.015863168984651566,
+            -0.02731737680733204,
+            0.00792400911450386,
+            0.012234487570822239,
+            0.028514310717582703,
+            -0.0035623949952423573,
+            0.001716805505566299,
+            -0.005533927585929632,
+            0.016408607363700867,
+            0.032180868089199066,
+            -0.0008550884085707366,
+            0.006950552575290203,
+            -0.019999410957098007,
+            0.0017404790269210935,
+            0.013704141601920128,
+            -0.008401267230510712,
+            -0.01821158453822136,
+            0.003939277958124876,
+            -0.020635755732655525,
+            0.004583198111504316,
+            -0.020393338054418564,
+            0.030408194288611412,
+            -0.0023522034753113985,
+            0.008105821907520294,
+            0.004310478921979666,
+            0.01760554127395153,
+            -0.0031646795105189085,
+            0.010545143857598305,
+            0.013916256837546825,
+            0.014908651821315289,
+            0.0009980766335502267,
+            -0.011189064010977745,
+            -0.03790797293186188,
+            0.001859793672338128,
+            0.00817400123924017,
+            0.018635815009474754,
+            -0.01980244740843773,
+            0.011681473813951015,
+            -0.01296931505203247,
+            -0.020363036543130875,
+            -0.01003758329898119,
+            0.011575416661798954,
+            -0.006139970384538174,
+            0.03378688171505928,
+            -0.005874827038496733,
+            -0.008204303681850433,
+            0.005136212334036827,
+            0.010454237461090088,
+            0.0066778333857655525,
+            0.014704111963510513,
+            0.0326354019343853,
+            0.010196669027209282,
+            0.014173824340105057,
+            -0.004859705455601215,
+            0.02399929240345955,
+            -0.003931702114641666,
+            0.003954428713768721,
+            0.00509454682469368,
+            -0.022832660004496574,
+            -0.0040983641520142555,
+            -0.003931702114641666,
+            -0.016711628064513206,
+            -0.007458113599568605,
+            0.0032366469968110323,
+            0.008802770636975765,
+            -0.014045040123164654,
+            -0.00649980828166008,
+            -0.01350717805325985,
+            0.01218903437256813,
+            0.002730980049818754,
+            0.018302490934729576,
+            -0.029802151024341583,
+            0.009825468063354492,
+            -0.017484333366155624,
+            -0.017044952139258385,
+            0.006511171814054251,
+            -0.0038332203403115273,
+            0.006855858489871025,
+            -0.018272189423441887,
+            -0.01318900566548109,
+            0.012817803770303726,
+            0.0007897494360804558,
+            0.013060221448540688,
+            -0.009825468063354492,
+            0.0009706153068691492,
+            0.006064215209335089,
+            -0.009833043441176414,
+            -0.010961798019707203,
+            -0.007977037690579891,
+            -0.009681533090770245,
+            -0.008848223835229874,
+            0.005621046293526888,
+            0.01656011864542961,
+            -0.013742019422352314,
+            -0.003685497445985675,
+            0.016135888174176216,
+            0.006416477728635073,
+            -0.008166425861418247,
+            0.004075637552887201,
+            -0.003793448908254504,
+            -9.948806837201118e-05,
+            0.0013740125577896833,
+            0.012439027428627014,
+            -0.0017632056260481477,
+            0.024665940552949905,
+            -0.02654467150568962,
+            0.007598260883241892,
+            -0.0054051438346505165,
+            0.005446809343993664,
+            0.018226735293865204,
+            -0.02366596832871437,
+            -0.013082947582006454,
+            0.006276329979300499,
+            -0.01966608688235283,
+            -0.008060368709266186,
+            0.0054051438346505165,
+            -0.00016405766655225307,
+            -0.019923655316233635,
+            -0.012923861853778362,
+            -0.034786853939294815,
+            -0.02309022843837738,
+            0.024665940552949905,
+            0.008401267230510712,
+            -0.0073672072030603886,
+            0.018060073256492615,
+            -0.018438849598169327,
+            0.0172722190618515,
+            -0.002225313102826476,
+            -0.006613441277295351,
+            0.003935490269213915,
+            0.007674016058444977,
+            0.025362888351082802,
+            0.002653330797329545,
+            0.008264907635748386,
+            -0.0042082094587385654,
+            -0.014931377954781055,
+            -0.006321783177554607,
+            0.01980244740843773,
+            0.02471139281988144,
+            0.018605511635541916,
+            -0.018575210124254227,
+            0.010742107406258583,
+            -0.011749654076993465,
+            0.01818128302693367,
+            -0.006537686102092266,
+            0.00039037675014697015,
+            0.01859036087989807,
+            0.007969462312757969,
+            0.025241680443286896,
+            0.019923655316233635,
+            0.006852070800960064,
+            0.014128371141850948,
+            0.011908739805221558,
+            0.001204509986564517,
+            -0.012832955457270145,
+            -0.01154511421918869,
+            -0.02840825356543064,
+            0.042847223579883575,
+            0.0025434857234358788,
+            0.008317936211824417,
+            -0.027635548263788223,
+            -0.00021389048197306693,
+            -0.0034942151978611946,
+            -0.018832778558135033,
+            0.005939218681305647,
+            0.03260510042309761,
+            0.03284751623868942,
+            0.021438762545585632,
+            -2.888172457460314e-05,
+            0.0012234487803652883,
+            0.0329081192612648,
+            -0.013673839159309864,
+            -0.00937851145863533,
+            -0.01034060399979353,
+            -0.018393397331237793,
+            -0.006852070800960064,
+            -0.0032423287630081177,
+            -0.01031030248850584,
+            -0.0061172437854111195,
+            -0.009272454306483269,
+            0.0238629337400198,
+            0.0001374249259242788,
+            -0.0022594030015170574,
+            0.006605865899473429,
+            -0.020878173410892487,
+            0.011537538841366768,
+            -0.02813553437590599,
+            -0.013431422412395477,
+            -0.008651260286569595,
+            -0.006950552575290203,
+            -0.018802475184202194,
+            0.008643684908747673,
+            -0.006942977197468281,
+            0.021438762545585632,
+            0.002914686920121312,
+            -0.023590214550495148,
+            0.007530081085860729,
+            0.015317730605602264,
+            -0.0003825644962489605,
+            0.014037464745342731,
+            -0.003890036838129163,
+            0.009060339070856571,
+            0.0045112306252121925,
+            0.011681473813951015,
+            0.016954045742750168,
+            0.010651201009750366,
+            0.009030036628246307,
+            0.003437398700043559,
+            -0.001167579204775393,
+            0.01878732442855835,
+            -0.0007163614500313997,
+            -0.03405960276722908,
+            2.9236827685963362e-05,
+            -0.014628356322646141,
+            0.029499130323529243,
+            -0.009795165620744228,
+            0.009514871053397655,
+            0.008105821907520294,
+            0.007098275702446699,
+            -0.0067119235172867775,
+            0.007916433736681938,
+            -0.008742166683077812,
+            0.0052763596177101135,
+            0.015832867473363876,
+            -0.0042877523228526115,
+            -0.0014980619307607412,
+            0.007840678095817566,
+            0.00407184986397624,
+            -0.005454384721815586,
+            -0.019711541011929512,
+            -0.012287517078220844,
+            0.00726114958524704,
+            -0.02065090648829937,
+            -0.004367295652627945,
+            0.01161329448223114,
+            -0.005454384721815586,
+            -0.009620928205549717,
+            0.01393140759319067,
+            0.02136300690472126,
+            -0.002382505452260375,
+            0.022893264889717102,
+            -0.01349202636629343,
+            -0.009833043441176414,
+            -0.007984613068401814,
+            -0.004374871030449867,
+            -0.041120000183582306,
+            0.025984082370996475,
+            0.01068150345236063,
+            -0.006871009711176157,
+            0.01895398646593094,
+            0.0022499337792396545,
+            -0.002954458352178335,
+            -0.0011041341349482536,
+            0.00010499217023607343,
+            -0.03281721472740173,
+            0.03660498186945915,
+            -0.007007369305938482,
+            0.0005449649761430919,
+            0.010272424668073654,
+            0.042029064148664474,
+            -0.004590773954987526,
+            -0.005859675817191601,
+            0.01225721463561058,
+            0.005336964037269354,
+            0.031938452273607254,
+            -0.010416359640657902,
+            -0.014340486377477646,
+            -0.01815097965300083,
+            -0.004833190701901913,
+            -0.01939336769282818,
+            -0.0038313264958560467,
+            0.014037464745342731,
+            -0.019650936126708984,
+            -0.011242092587053776,
+            -0.001287840772420168,
+            -0.012537509202957153,
+            -0.0010747789638116956,
+            0.008598231710493565,
+            0.017832808196544647,
+            -0.011348150670528412,
+            -0.004450626205652952,
+            -2.2904154320713133e-05,
+            -0.005003640428185463,
+            0.0185146052390337,
+            0.0020056227222085,
+            -0.01702980138361454,
+            0.01321930717676878,
+            -0.00526499655097723,
+            0.006749800872057676,
+            0.0074429623782634735,
+            0.007939159870147705,
+            0.05145302787423134,
+            0.010317877866327763,
+            -0.001339922659099102,
+            -0.01607528328895569,
+            0.023196285590529442,
+            -0.01749948412179947,
+            -0.014181400649249554,
+            -0.02119634486734867,
+            -0.0029923359397798777,
+            -0.007060397882014513,
+            -0.028014326468110085,
+            -9.137987944995984e-05,
+            0.0016221113037317991,
+            -0.0024980325251817703,
+            -0.010045158676803112,
+            0.0005208179936744273,
+            -0.011529963463544846,
+            -0.006954340264201164,
+            -0.008545203134417534,
+            -0.016044981777668,
+            0.02240842953324318,
+            -0.021817538887262344,
+            -0.008855799213051796,
+            0.011514811776578426,
+            0.03118089959025383,
+            -0.005011215806007385,
+            -0.014173824340105057,
+            -0.019817598164081573,
+            0.0025643182452768087,
+            -0.01763584464788437,
+            0.012181458994746208,
+            -0.01225721463561058,
+            0.015045011416077614,
+            -0.03142331540584564,
+            -0.006079366430640221,
+            -0.008992158807814121,
+            0.012840530835092068,
+            0.019817598164081573,
+            -0.0025093958247452974,
+            0.00606800289824605,
+            -0.010295150801539421,
+            0.030211230739951134,
+            -0.00012286569108255208,
+            -0.018605511635541916,
+            0.00873459130525589,
+            0.007977037690579891,
+            0.012537509202957153,
+            -0.004435475450009108,
+            0.00257757562212646,
+            0.011567841283977032,
+            0.012742049060761929,
+            -0.01634800247848034,
+            -0.003174148965626955,
+            -0.017969166859984398,
+            0.005996035411953926,
+            0.006999793462455273,
+            0.021711481735110283,
+            -0.0013550736475735903,
+            0.018272189423441887,
+            -0.01563590206205845,
+            0.00019412307301536202,
+            -0.011749654076993465,
+            -0.0082270298153162,
+            -0.0028237802907824516,
+            -0.003958216868340969,
+            0.006685409229248762,
+            -0.028211290016770363,
+            -0.01274962443858385,
+            0.005287723150104284,
+            -0.004992276895791292,
+            -0.03405960276722908,
+            0.0053710537031292915,
+            -0.02725677192211151,
+            0.013128400780260563,
+            -0.002397656673565507,
+            -0.017847958952188492,
+            -0.00030846628942526877,
+            0.026332557201385498,
+            -0.003677921835333109,
+            -0.012537509202957153,
+            0.015544996596872807,
+            -0.006757376715540886,
+            -0.02119634486734867,
+            -0.005420294590294361,
+            0.014658658765256405,
+            -0.0010719380807131529,
+            -0.012393574230372906,
+            -0.029635490849614143,
+            0.012992041185498238,
+            0.0026874206960201263,
+            -0.007386146113276482,
+            0.010893618687987328,
+            0.025953780859708786,
+            0.008151275105774403,
+            -0.005924067925661802,
+            0.0015757111832499504,
+            0.0049581872299313545,
+            0.00300938100554049,
+            0.020287280902266502,
+            0.02728707529604435,
+            -0.008014915511012077,
+            -0.0011145505122840405,
+            -0.014954105019569397,
+            0.015802564099431038,
+            0.0013020449550822377,
+            0.010507266037166119,
+            0.021893294528126717,
+            0.007045246660709381,
+            -0.005738467443734407,
+            0.003905187826603651,
+            -0.023135682567954063,
+            0.01420412678271532,
+            -0.012272365391254425,
+            0.000622140767518431,
+            0.008749742060899734,
+            -0.007283876184374094,
+            0.01437078882008791,
+            -0.0015245762187987566,
+            0.0339989997446537,
+            0.025211378931999207,
+            0.0043218424543738365,
+            0.0007362472242675722,
+            -0.028559764847159386,
+            0.003547244006767869,
+            0.008651260286569595,
+            0.014499573037028313,
+            -0.0076664406806230545,
+            0.006261179223656654,
+            -0.004355932120233774,
+            0.00959820207208395,
+            -0.0065490491688251495,
+            -0.015893470495939255,
+            0.006113456096500158,
+            0.0012689019786193967,
+            0.006795254070311785,
+            0.0023408401757478714,
+            -0.008863375522196293,
+            -0.024362917989492416,
+            -0.025984082370996475,
+            0.021984200924634933,
+            -0.03918066248297691,
+            0.0073293293826282024,
+            0.011234517209231853,
+            0.03324144333600998,
+            -0.008158850483596325,
+            -0.004980913829058409,
+            0.006276329979300499,
+            -0.016711628064513206,
+            -0.004147605039179325,
+            0.008204303681850433,
+            0.012385998852550983,
+            -0.002420383272692561,
+            0.00974971242249012,
+            0.01318900566548109,
+            -0.012378423474729061,
+            0.03660498186945915,
+            0.016514664515852928,
+            0.01467380952090025,
+            -0.004007457755506039,
+            0.01312082540243864,
+            0.0066172294318675995,
+            0.019044892862439156,
+            0.025165924802422523,
+            -0.017454031854867935,
+            -0.002365460619330406,
+            -0.02292356640100479,
+            0.024165954440832138,
+            0.00726114958524704,
+            0.004556683823466301,
+            -0.008310360834002495,
+            0.014870774000883102,
+            0.0017954016802832484,
+            0.0028199926018714905,
+            -0.006696772295981646,
+            -0.010173942893743515,
+            0.019241856411099434,
+            0.0034317169338464737,
+            0.014870774000883102,
+            4.8501173296244815e-05,
+            -0.005503625608980656,
+            0.006052851676940918,
+            0.014832896180450916,
+            -0.023590214550495148,
+            -0.00203592493198812,
+            -0.019484274089336395,
+            -0.03202936053276062,
+            0.009022461250424385,
+            0.0016135887708514929,
+            -0.007052822504192591,
+            0.03230207785964012,
+            -0.010083036497235298,
+            -0.019787294790148735,
+            -0.008067944087088108,
+            -0.013113250024616718,
+            -0.010469388216733932,
+            0.0008271536789834499,
+            0.003908975515514612,
+            -0.00012428610352799296,
+            -0.009317907504737377,
+            -0.005749830510467291,
+            0.01990850456058979,
+            -0.0030358952935785055,
+            -0.00753386877477169,
+            0.00868156272917986
+          ],
+          "embedding": [
+            0.0040707582550429606,
+            -0.010230625151989194,
+            -0.021520242101903186,
+            -0.010115756270640508,
+            0.026738734586159035,
+            0.01796370233223267,
+            -0.012245199029994153,
+            0.06506924494147726,
+            -0.0014087587222033442,
+            0.009300068719271429,
+            0.003787396973047584,
+            0.002818772639754174,
+            0.022129077062176306,
+            -0.021526709808445683,
+            -0.0034388135367225824,
+            0.02763761747879405,
+            0.013210013805395276,
+            0.025255872672296602,
+            0.00029011480909986703,
+            -0.05438974762290585,
+            0.016671640187490865,
+            -0.016426252259893328,
+            -0.056591180651012554,
+            0.009335648425523075,
+            0.006141624459418827,
+            0.02568943956694641,
+            0.019377132054042116,
+            0.017946464337078746,
+            -0.057184776985927686,
+            0.03208527139096687,
+            0.030132392063755547,
+            -0.003846032371791839,
+            -0.008376524427531415,
+            -0.0010513963058713435,
+            -0.02734868388724371,
+            -0.014879400370545479,
+            0.04383920959748825,
+            0.02563613659516289,
+            -0.020117467074438997,
+            0.0398016699155716,
+            0.029896497884197775,
+            -0.0004195844530067313,
+            -0.023302720788253625,
+            -0.001168337916577517,
+            0.009149021369661628,
+            -0.02508145721900992,
+            -0.004307594138480351,
+            0.00550940415564303,
+            0.014016998918718476,
+            -0.028607379616884317,
+            -0.0018317656587526954,
+            -0.01973464940415756,
+            0.03762738392056255,
+            0.015358106058583114,
+            -0.03931907682104936,
+            0.011519637298832444,
+            0.004482080558226775,
+            0.0747867063383242,
+            0.05550679018263077,
+            0.023243129601931176,
+            0.02671631686665238,
+            -0.009686146566573536,
+            -0.04088582189725495,
+            -0.0065344623594924085,
+            0.019617295999325712,
+            -0.004722087402575777,
+            -0.002871844114437583,
+            0.003935087265300519,
+            0.03229890413393201,
+            -0.02813577047904649,
+            -0.05124172132597234,
+            0.03325235407841968,
+            -0.008687655557217192,
+            -0.011027174838246948,
+            -0.003653553063547375,
+            -0.02350737945182988,
+            0.016223087939183468,
+            0.03789252356462714,
+            0.0034350623378603083,
+            0.00195158657888739,
+            0.011789712862232108,
+            -0.0004435439660769551,
+            -0.0037330456918125894,
+            0.0031552713243866858,
+            0.04465432992298031,
+            -0.0032465421571915712,
+            -0.0075366743861922745,
+            -0.059958752731410256,
+            0.016784616861008125,
+            -0.028734506065772967,
+            -0.03035575736273943,
+            0.02680390892611741,
+            0.02338368719038086,
+            -0.0030658375437784277,
+            -0.03754896050794851,
+            0.07005659523248504,
+            0.0020711575553570695,
+            0.013034425083076923,
+            -0.02316832262132332,
+            0.021332142030732894,
+            -0.0318756233020585,
+            0.02270782509507388,
+            0.01570810309454529,
+            0.004788124787612736,
+            0.026727250200142456,
+            -0.022380142778058276,
+            -0.008593910379436904,
+            0.014439754382387034,
+            0.015007697617383174,
+            0.07662531963612977,
+            0.011195998137723795,
+            -0.010809283538470098,
+            -0.042227822120219305,
+            0.003824320933275175,
+            -0.03797629427607094,
+            -0.02347390684453348,
+            -0.0526876440280276,
+            -0.017799861680287913,
+            0.01701482796266917,
+            0.002730395144917155,
+            0.03789451759899825,
+            0.038338545350313946,
+            0.028463789278406586,
+            0.015494583705011921,
+            -0.05700237266350479,
+            0.047895170359075166,
+            -0.0064406894534929406,
+            -0.0007954721295065988,
+            0.012543029839729358,
+            -0.004703334868741216,
+            -0.0060569679284068,
+            0.012324901254711192,
+            -0.019883935809784068,
+            -0.04253565942154215,
+            0.02164409276211115,
+            -0.0075367555415808635,
+            0.03617159147722157,
+            0.07567745988686005,
+            -0.04225977020036783,
+            -0.022548837589622003,
+            0.029811602145850144,
+            -0.02175792196819533,
+            0.02069861458115363,
+            -0.00941331829664397,
+            -0.035479639885943264,
+            0.020352552580119105,
+            0.003678588588550461,
+            0.06776105274336729,
+            -0.02346047721661687,
+            0.02721075076349443,
+            -0.029386781707644253,
+            0.023530831921117993,
+            -0.02077660533100372,
+            0.020022475323372906,
+            0.024195118676513715,
+            -0.028378518720494666,
+            -0.012900969023913307,
+            0.019264089970719624,
+            -0.0374983624629265,
+            -0.019726171557384355,
+            0.00979636085642416,
+            -0.014479528435832063,
+            -0.014533384583777662,
+            0.005904623758767224,
+            0.010344475500263744,
+            0.05659646358026115,
+            0.04163866113482093,
+            0.007152494819169405,
+            -0.07584500130974763,
+            0.05471858597878099,
+            0.03580211013427615,
+            0.044161429444475306,
+            0.02258180566046248,
+            0.00980779221334715,
+            0.0011193505603397395,
+            0.018086566768892507,
+            0.015605066702191871,
+            -0.00954727399396313,
+            -0.027950079146919375,
+            0.03028514025485407,
+            -0.006351669761257431,
+            0.005095804801686436,
+            0.009620040085743162,
+            0.019748001227987134,
+            0.033709535059510916,
+            0.008203559956940196,
+            0.007389752602218563,
+            0.04197771878351742,
+            -0.02067363758471765,
+            0.007543739188471913,
+            0.011207941261799052,
+            0.013964827251172813,
+            0.03839562075248879,
+            0.0009775244543292637,
+            0.006469507611492216,
+            -0.022808235395746654,
+            -0.024795004398633292,
+            0.006249024864814684,
+            0.011292877109066925,
+            0.01596839619826819,
+            -0.0014105396434174883,
+            -0.04152075438277094,
+            0.008267788411079337,
+            -0.028834191212891387,
+            0.030940526405027906,
+            -0.0018271056434014425,
+            0.006185475047820603,
+            -0.007920574674727123,
+            0.012907560753043616,
+            -0.015165704767187252,
+            -0.021962244496450727,
+            0.0037071053072833147,
+            -0.013508376064336042,
+            -0.015178823540513864,
+            -0.02518557053061311,
+            0.011693256916760944,
+            -0.01716015674592331,
+            0.02048897513447166,
+            -0.003412078966361745,
+            -0.001807298511378529,
+            -0.0012575655363779904,
+            0.03116103961636643,
+            0.011668533361641842,
+            -0.011637010548030312,
+            -0.040260195687925236,
+            -0.011129376691522074,
+            -0.0034493180842871908,
+            -0.02098244789711689,
+            0.026658743116016938,
+            -0.01923736325953796,
+            -0.009723937181300192,
+            -0.013134679989565051,
+            -0.011974548914818263,
+            -0.004028712301052159,
+            0.019628329175867303,
+            -0.028192034966932684,
+            -0.012638520793012082,
+            -0.011568652932013111,
+            -0.0025919910519445737,
+            0.012497908463405111,
+            0.015113186712897145,
+            0.014880607940212641,
+            -0.03059227859452334,
+            -0.021899716217105084,
+            -0.0008513506358489523,
+            0.011985931118464679,
+            -0.01321584595585435,
+            -0.011735288366213981,
+            0.010710683541435907,
+            0.01961722172525993,
+            -0.023571946726192585,
+            -0.03364281880125354,
+            0.0014460076743756465,
+            0.016563167828467187,
+            -0.02891589059260944,
+            0.04595166331401113,
+            0.034271553243501465,
+            -0.027557971577291153,
+            -0.007138821520412525,
+            -0.039508407807651844,
+            0.06814329636830853,
+            0.00022136123564675344,
+            -0.039718016705753735,
+            0.0503457594963753,
+            -0.003580581891576583,
+            0.013820796398521455,
+            0.02054050419088757,
+            0.035682661890720495,
+            -0.0013220739026852943,
+            -0.0052759394697328675,
+            0.04733018719162478,
+            -0.0187845855131196,
+            -0.03982699468608443,
+            -0.023905460465070062,
+            0.002631233133039351,
+            -0.03090995554989513,
+            -0.012617612051488022,
+            -0.013605787969473513,
+            0.039672783793764745,
+            0.008669582242097126,
+            0.009974277556736362,
+            0.005535636914843583,
+            0.014180668493093584,
+            0.012042659947850441,
+            0.011870115226306776,
+            0.00451259454348581,
+            -0.014023048592193627,
+            -0.007769033996392623,
+            -0.05715987527668358,
+            0.003374564518384728,
+            -0.028801158523506985,
+            -0.00756331369170698,
+            -0.02163054894055572,
+            0.005367181573233089,
+            0.02308705505078254,
+            -0.005438909398653553,
+            -0.00873620909426313,
+            -0.0036530652237719077,
+            0.013920407174296543,
+            -0.01539613379172909,
+            -0.03908211213269472,
+            -0.016810726548844992,
+            0.007460173854386443,
+            -0.007069189426742819,
+            -0.041197222498539215,
+            -0.003843233633335066,
+            0.004373367480223065,
+            0.016194121660459908,
+            -0.029165150066068787,
+            -0.00735139702658342,
+            0.017257895082337448,
+            0.014153313280536095,
+            -0.002376178042741926,
+            -0.023223700836427403,
+            -0.0319877537205969,
+            0.02088060774052704,
+            0.0012829290529515294,
+            0.03251550040316258,
+            -0.04128453069131847,
+            -0.023376114412063578,
+            0.04083022735076889,
+            0.005690491480259003,
+            -0.05810403800689653,
+            0.016582434816685452,
+            0.010402038964417569,
+            0.016795644054313566,
+            -0.00908590043098552,
+            0.05003656634510042,
+            -0.037216197954986735,
+            -0.013027960261454162,
+            -0.007090943677204207,
+            -0.05175371787508879,
+            -0.0020065660448063322,
+            -0.014908489552053589,
+            0.010627472758306387,
+            -0.03275473161158777,
+            -0.017334608434133866,
+            0.025321047439048318,
+            -0.0033515195870701753,
+            0.011526068468647207,
+            -0.026783879816504975,
+            -0.0012830935419413407,
+            -0.017506908439998053,
+            -0.02114853058440477,
+            0.004878884893669383,
+            -0.020936900837115968,
+            -0.0025587027781733642,
+            0.01557850246191957,
+            0.023652335794299834,
+            0.011489428313303988,
+            0.0019089137782552768,
+            0.02054907941022243,
+            0.014361232983780595,
+            -0.0056413109308506085,
+            -0.0010300050480036684,
+            -0.03413262505887213,
+            -0.018821160485010483,
+            0.03361989907065425,
+            0.013125889701529858,
+            -0.003742167496379789,
+            0.008813100942466033,
+            -0.0383265293638935,
+            -0.035316957263713485,
+            0.03020499323039515,
+            -0.05001623971243151,
+            -0.00855479667673597,
+            -0.005939674940109814,
+            -0.026451831943555135,
+            -0.008032873049527808,
+            0.0009431571685654594,
+            -0.014387605128858954,
+            0.016382559541755014,
+            -0.024909510232038502,
+            0.003715736505444385,
+            -0.006016712902409688,
+            -0.009110197692342108,
+            0.05507263160354303,
+            -0.022103031032598353,
+            -0.0475415595313825,
+            0.021958434235402655,
+            -0.016670809659931243,
+            0.0323519363801999,
+            0.03867494951405658,
+            0.011124508964363016,
+            -0.008002445714431074,
+            0.034041838454818577,
+            0.018400123235267234,
+            -0.01161754789471199,
+            -0.025901844576206774,
+            0.021800298132870245,
+            0.00943488425697727,
+            -0.05906691123008531,
+            -0.00782549274387971,
+            -0.019357446956797447,
+            0.008682376344965976,
+            0.02863077694455766,
+            0.0024925977373894447,
+            -0.002682170968419412,
+            -0.013590235283325244,
+            -0.021511149507762284,
+            -0.03255453916518719,
+            0.009039307015853905,
+            0.008305286476500875,
+            0.009312625027668458,
+            -0.0014144165766983945,
+            -0.007853869334376857,
+            0.009067065760942163,
+            0.022485201611182105,
+            0.03966189080765722,
+            -0.041627289519629344,
+            0.002707086456239785,
+            -0.022678178050044222,
+            -0.03653753142622053,
+            0.0036574646778835588,
+            0.029996160786576367,
+            0.02901163765559891,
+            -0.01971777193183787,
+            -0.0036589934841109884,
+            0.009181234938554578,
+            -0.025979483002227535,
+            -0.00773645103995767,
+            0.004280770859737866,
+            0.04116668301017644,
+            0.006201085381597264,
+            0.01402728258112097,
+            0.001060407317110527,
+            -0.00920801784000087,
+            0.02109880889040528,
+            0.025211022083982727,
+            -0.029866678264942617,
+            0.014439702646820226,
+            0.009379206296908577,
+            0.04156435812161122,
+            -0.037309655469525846,
+            -0.011819731348299495,
+            -0.02766498020409985,
+            -0.010454143230794476,
+            0.027911532296566614,
+            -0.04127890742215716,
+            -0.006948536183086887,
+            0.010207784113909938,
+            -0.0016584043356136647,
+            0.016437495428919636,
+            -0.014607861750686952,
+            -0.03222311919805017,
+            -0.000610723203118213,
+            -0.02867445198803213,
+            -0.024772816792038133,
+            -0.016510309753766612,
+            -0.018581201775667577,
+            -0.07450654873867177,
+            0.00026832248829154255,
+            -0.06724696990816072,
+            -0.02447002541264633,
+            0.04885569249258513,
+            -0.010195604207134053,
+            0.03332905935704749,
+            -0.02815272112271314,
+            -0.005296340304911926,
+            -0.05908835042526019,
+            0.0038476172058970774,
+            0.025753576863447866,
+            0.02386003857017594,
+            0.005206447371391696,
+            0.024799057871126032,
+            0.012060738268265606,
+            -0.04146503002980051,
+            -0.024949907556862643,
+            -0.014503475223770958,
+            -0.0016340466065744286,
+            -0.0034009167013795906,
+            -0.031298432995411005,
+            0.036241272149880994,
+            0.008902644391348814,
+            0.03550223922667804,
+            0.04463301409716232,
+            -0.010385984413759852,
+            -0.03305004479734889,
+            -0.006952739338820577,
+            -0.0175981588092108,
+            0.042668533202572856,
+            -0.017112184620086308,
+            0.005149173014526686,
+            -0.03219589107997635,
+            0.024181808150698054,
+            0.0014339790422945406,
+            0.01254721190611376,
+            0.011099458649404119,
+            0.0027626836767364253,
+            -0.026883640446090158,
+            0.0024160770766463607,
+            0.023874514847956222,
+            -0.022558433617669373,
+            0.00917667715473093,
+            -0.006754091166465801,
+            -0.033686935718776144,
+            -0.01078389638118685,
+            -0.013595327486167114,
+            0.024712781935748277,
+            0.013168464807451798,
+            0.005875864604862434,
+            -0.02417814420558463,
+            -0.0023978800459714144,
+            0.028650463366555304,
+            0.020176980768301288,
+            -0.02118184201861616,
+            0.019082340279671355,
+            0.019734925740100277,
+            -0.04209645071102001,
+            -0.04217310001316092,
+            0.006553913719098956,
+            -0.003735719227468833,
+            0.0061793909220459,
+            -0.028285889794908054,
+            0.008181788076484304,
+            0.004432488437404309,
+            -0.01743905248466261,
+            0.041221269621866216,
+            -0.014892354419172107,
+            0.005087102208335201,
+            -0.022198383716186407,
+            -0.011936388003605221,
+            -0.00027487042024290816,
+            0.044599215801158845,
+            0.008068975171827777,
+            0.003496950921841073,
+            0.02238779276447619,
+            0.0054839808262270315,
+            0.04816723616939378,
+            -0.02685221260466647,
+            -0.0190356946337995,
+            0.016464136459560113,
+            -0.0176092780782677,
+            -0.012513274095432353,
+            -0.010695421445334002,
+            -0.004784348908457072,
+            -0.022203041334932165,
+            0.010788155380415583,
+            -0.018353897272137543,
+            -0.02277431189907027,
+            -0.003945465850796918,
+            0.016024639792926797,
+            -0.006248368166794934,
+            -0.0038769803892794875,
+            0.0016292556540434277,
+            0.02412864830338344,
+            0.02892955453819876,
+            -0.02408022470555968,
+            0.019190282372105023,
+            0.018941584034503536,
+            -0.006709516380602965,
+            -0.020956377933747532,
+            -0.01964885206488053,
+            0.01434623424964373,
+            -0.01352968586212152,
+            -0.013872556892707697,
+            0.004052798542613835,
+            0.009201147374299997,
+            -0.004086067985859617,
+            -0.04185829685685201,
+            -0.014668698123513235,
+            0.003179403563664022,
+            -0.0006415014901450319,
+            0.019360184344240947,
+            -0.0150817538567364,
+            -0.029771999278536083,
+            0.014221776097046751,
+            0.022944884879589678,
+            -0.04493425220030206,
+            0.02117577719533236,
+            -0.0030648751497261983,
+            -0.00016251533643103604,
+            -3.817296018601414e-05,
+            0.008727767382077138,
+            0.026914779443266063,
+            0.03456565094655899,
+            0.01974806772696219,
+            0.00450800007152441,
+            -0.008952599700759673,
+            0.025184190461155245,
+            0.009846044083064704,
+            -0.025168735183134738,
+            0.010983353565382029,
+            0.016740721743326522,
+            0.0070838930012654296,
+            0.009173879407277282,
+            -0.006078288608766152,
+            -0.018269347893097643,
+            0.013181494188066667,
+            -0.01435693747022231,
+            0.01678907949532146,
+            0.05154463552784974,
+            -0.00039632596424461964,
+            -0.02461439550772063,
+            0.008418147037164186,
+            -0.006544319842335593,
+            -0.010065129286193278,
+            0.02214628550460128,
+            0.0029515574751370417,
+            -0.030862152033847717,
+            0.007509556025223583,
+            0.00675485304374519,
+            0.01894936231559203,
+            -0.02355525295639599,
+            -0.01387885858544803,
+            -0.00431398020087544,
+            0.0021208056468763523,
+            0.01580766504149989,
+            -0.004739477121073512,
+            -0.013204823447960706,
+            0.02994098229517376,
+            0.025496728234662373,
+            -0.00026124946738213036,
+            0.015060064980768175,
+            0.022262345898533903,
+            0.009968408664790368,
+            0.007588431141088797,
+            -0.0083063413609779,
+            -0.031336272171937095,
+            0.0018957858383561667,
+            0.05800811417728381,
+            -0.014937085296865802,
+            0.04313875432235027,
+            0.02018476425923236,
+            0.015657644644987107,
+            0.017880263053253607,
+            -0.01662268082870562,
+            -0.008740053682875323,
+            -0.0017630394204179638,
+            -0.008294417990024647,
+            -0.027613464270562948,
+            -0.005953209556351086,
+            -0.02379961210881074,
+            -0.013878676554701543,
+            0.015418315245979704,
+            -0.014764728065382074,
+            0.0027420640870858457,
+            0.010395538988699663,
+            -0.006590633833835581,
+            -0.00841537209927409,
+            -0.003943772700797381,
+            0.03376338017637296,
+            0.011980354721458463,
+            0.027761929822019422,
+            -0.00011128539412582973,
+            0.009342430952107655,
+            -0.018419526564396063,
+            -0.006839752805318664,
+            0.03271190887864607,
+            0.006011520427944665,
+            0.004652835471939905,
+            0.01485920920301398,
+            0.014305626381398871,
+            0.038015677219487023,
+            -0.005102005401565164,
+            -0.018204460895989052,
+            0.02065495133597456,
+            -0.0049699013356374775,
+            0.019149036693562712,
+            0.008825981577388056,
+            -0.002660055848523568,
+            -0.004292379561953203,
+            0.03090270710860331,
+            -0.020243440437217503,
+            0.018735727616321387,
+            -0.02238236089688497,
+            -0.0276584413929402,
+            0.023028408906176946,
+            0.008320436733620496,
+            0.04047641118026783,
+            0.011463851066078041,
+            -0.02247143742166039,
+            -0.034588725413285185,
+            0.009827713930789184,
+            0.00337071450636392,
+            -0.024446280199879062,
+            0.017542777160283324,
+            0.00780836483490212,
+            0.008453886667283516,
+            0.016624116261219637,
+            -0.004175682824655864,
+            -0.0042784362933758835,
+            -0.001546656717896957,
+            -0.008819739495525477,
+            0.01406790007187259,
+            -0.015487658289541583,
+            -0.021457182907140035,
+            0.024687169583245905,
+            -0.03915911962859764,
+            0.01771290139419629,
+            -0.006510046907173026,
+            -0.00997450880725535,
+            -0.02059560682587115,
+            1.5939836949198004e-05,
+            -0.007159493752862143,
+            -0.0004782363814453761,
+            -0.017503181957042913,
+            0.016643971600145645,
+            0.024056442566869802,
+            -0.013209211443909658,
+            -0.00445521990927108,
+            0.0020313626821877526,
+            -0.029235692455820056,
+            0.006662209097255912,
+            0.016993027053110635,
+            0.00529940612845993,
+            0.0026104743851393716,
+            0.0001707934714983891,
+            -0.031289446611163674,
+            -0.005782208620284939,
+            0.03053516889069683,
+            0.05590704747438521,
+            0.021628254147289437,
+            0.02403314981042378,
+            0.02908606663381421,
+            0.02717206823892633,
+            -0.003198844932040613,
+            0.0016217508103442348,
+            -0.01620660606431983,
+            0.027098323143010743,
+            -0.011664168290676943,
+            0.04322175021589083,
+            0.008652323184522541,
+            -0.049493020736445056,
+            0.004537958157696794,
+            0.00943239454147692,
+            0.034918603888916544,
+            0.016678716235690318,
+            0.0005887414054803805,
+            -0.005894338664998101,
+            0.03279165663336323,
+            -0.02735726709032382,
+            -0.011962070182991011,
+            -0.00036014407560325284,
+            -0.019079003730607717,
+            -0.015938295169860884,
+            0.010357611943935459,
+            0.04278924678845588,
+            0.0009957483598272451,
+            -0.0071422790468850215,
+            0.01571201177743999,
+            -0.009445307674383,
+            0.02286338492075306,
+            0.007336440384050498,
+            0.0042809497342459904,
+            0.025581467690095977,
+            -0.0025007871391868105,
+            0.02300539823997411,
+            -0.000628898771317254,
+            0.004582863909134464,
+            -0.015960578501883316,
+            -0.00930649396114688,
+            0.027838455754803444,
+            -0.011112268986721,
+            -0.029253746504396393,
+            -0.008734785561831333,
+            0.00472579547270089,
+            -0.0003805825712986138,
+            0.0021854358149324227,
+            -0.01183885533661493,
+            -0.010840951201591629,
+            0.019965691788904057,
+            -0.004922943301424112,
+            0.003800048754664761,
+            0.012664049429916252,
+            -0.0030124219674414897,
+            -0.01910844996750822,
+            0.01889983984736845,
+            -0.003858283778519141,
+            -0.024168821338992813,
+            0.020083955711766945,
+            0.01850497889545421,
+            0.045156567155119454,
+            -0.01586824055785609,
+            0.004612757419198795,
+            0.017497774574602505,
+            -0.002172088240279357,
+            -0.019617327081566827,
+            -0.014944795772093997,
+            -0.018212462639041492,
+            -0.015975283672064702,
+            0.012837956594691596,
+            -0.01243789877725244,
+            0.019500571353639314,
+            -0.0380735226388175,
+            0.012590900534826689,
+            0.004832168601136412,
+            0.007505385730121269,
+            0.016011983211212992,
+            0.025994560026860358,
+            0.002157436199771672,
+            -0.033465476168065784,
+            -0.01014599891204166,
+            0.007845373376916211,
+            0.010093432683262838,
+            -0.002849185178237036,
+            -0.018147004183434554,
+            0.003633790611809425,
+            -0.0266147420637404,
+            -0.018644422659309035,
+            0.022926737299761104,
+            -0.04798772015599113,
+            0.017908415301694906,
+            0.024859125158586466,
+            -0.04343699525707025,
+            -0.04886699607556352,
+            -0.011459188432595229,
+            -0.036774381316312464,
+            0.0014128005215813578,
+            -0.0005256067584181517,
+            0.011194992852264102,
+            -0.009342140079829378,
+            -0.018598671981710183,
+            0.01584789414301952,
+            -0.0036412641242200418,
+            -0.004225840239935549,
+            -0.00616696736018045,
+            -0.0025598631463178384,
+            -0.010188803157718543,
+            -0.03244694662411977,
+            -0.03290891136780909,
+            -0.00033742479296272127,
+            -0.009743927051414455,
+            -0.02334288359945272,
+            -0.03447062782872289,
+            -0.008728968107967028,
+            -0.037753659749578956,
+            -0.00515561961442376,
+            0.04070890150714597,
+            0.027517147608335957,
+            0.009449414978133334,
+            0.008086900089929912,
+            -0.013989034604490105,
+            0.05691047488419298,
+            -0.020425883381389086,
+            0.039466620642512155,
+            0.005301493569940605,
+            -0.0037222353899918703,
+            0.013075425214878277,
+            0.027710826422351564,
+            -0.006253665106995653,
+            -0.0017412928073238833,
+            -0.01928096630475438,
+            -0.012920476859996416,
+            0.009444408011467955,
+            0.015799884767184885,
+            -0.0025083061619197524,
+            0.001274457251739847,
+            0.00930533161700852,
+            -0.0006652478540264343,
+            0.017258038491335247,
+            0.03996268817486439,
+            0.003833844612473671,
+            -0.00826404141706295,
+            -0.018822002570963614,
+            -0.003188844900317347,
+            -0.011807006674248829,
+            0.010482058734260632,
+            -0.015410635475145028,
+            -0.011540803927522013,
+            -0.01249428777269578,
+            -0.017081856786754605,
+            -0.002183034976477535,
+            0.003114834030705039,
+            0.0027475411722881747,
+            0.02261627635192778,
+            -0.015872027027080824,
+            -0.022487581027490516,
+            -0.0035823117885461506,
+            -0.02398778449664045,
+            8.13716819067491e-05,
+            -0.01050060782407259,
+            -0.015616181283381218,
+            0.013676480044963827,
+            -0.021513173575512936,
+            -0.01091399089076612,
+            -0.008699176177861256,
+            -0.0006986953850759474,
+            0.005853723710671249,
+            -0.00993355748621621,
+            -0.00010241843582159751,
+            0.01150872694850241,
+            0.005559552266827916,
+            -0.012336449703762907,
+            0.0039431718621504,
+            -0.012740550272925562,
+            -0.004432059174951158,
+            -0.03592309122711106,
+            -0.019665686718735042,
+            -0.01846795637714379,
+            0.005642094448288269,
+            -0.005108964028543505,
+            -0.020586678054151854,
+            0.00668364061048941,
+            0.0034775127050453774,
+            0.004713169526661792,
+            -0.010723667875209928,
+            0.013944735241748791,
+            0.010882420168586074,
+            -0.003518245916227596,
+            0.013827334494230662,
+            -0.001330399936559009,
+            -0.020286697504699764,
+            -0.011139922402071095,
+            0.03538258948179278,
+            -0.024961826464903997,
+            -0.0018019328351763667,
+            -0.0026423695690168066,
+            0.00597003516870726,
+            -0.03071835946501977,
+            -0.013634212232744325,
+            0.02164400349828422,
+            -0.005846552894831441,
+            -0.026367014426434226,
+            0.03399116480234033,
+            0.03473258231629753,
+            0.010041605592929843,
+            -0.013423384340589794,
+            0.04730054254696893,
+            -0.012968978954938308,
+            -0.018274800841400665,
+            0.026826644936918647,
+            0.0278389611094493,
+            0.02525164095926168,
+            -0.029782550447890466,
+            -0.003010713086487275,
+            -0.02393629084357885,
+            0.017993366287762866,
+            -0.0263476663180971,
+            0.01864011499179424,
+            -0.02332263152537849,
+            0.016014908881756034,
+            0.01868464825773604,
+            -0.003104041044182249,
+            0.026313491086684425,
+            -0.0052945248919367815,
+            0.015253731232325074,
+            0.0012371308759955573,
+            -0.02119765517013431,
+            0.007795466385020642,
+            -0.006469500272298062,
+            0.025762169721880456,
+            0.00833992768202466,
+            0.01801901639978976,
+            0.022019940825351116,
+            -0.0409522298213741,
+            0.02003698451081499,
+            -0.001367916864679543,
+            0.01782984895971748,
+            0.0038516134262842373,
+            -0.007885432294782304,
+            0.014221728513400065,
+            0.010991352188016265,
+            -0.012851099261924757,
+            0.003840539873231741,
+            0.0008923482291826014,
+            -0.028928104542716937,
+            0.007066004768771198,
+            -0.026914574065060926,
+            0.025866907400807087,
+            -0.005526774784152086,
+            -0.00998122210956923,
+            -0.020744532673097317,
+            -0.0071425207262770865,
+            0.003754468896273156,
+            -0.012747874997429746,
+            -0.011385805601999295,
+            -0.024700457669046855,
+            -0.02401291382882952,
+            0.029591582802488962,
+            -0.014054332096344003,
+            -0.04015978298687865,
+            0.001234440863385445,
+            -0.023301255402129767,
+            0.01193125007773239,
+            -0.005793124129267758,
+            -0.024440100649464773,
+            -0.042563399082257154,
+            -0.017548888269389457,
+            -0.0009394887961485922,
+            0.022739763644266633,
+            -0.00114156949394707,
+            0.041791395064755515,
+            0.0008816696205437911,
+            0.022207132857533028,
+            -0.015547580543606755,
+            0.005319731264840819,
+            -0.02071229859314312,
+            -0.041495579742912285,
+            0.03660432378985971,
+            -0.014637031447181937,
+            0.025854844497435178,
+            0.03324325296704186,
+            -0.013009347371118843,
+            0.001354504885302625,
+            0.02652570709474851,
+            -0.014618100923440427,
+            -0.010782232747863238,
+            -0.0034936699515378212,
+            0.003461670978569965,
+            -0.006468116285021298,
+            -0.027793221545159466,
+            -0.007752458229193391,
+            0.04193706715422074,
+            0.017894069367616637,
+            0.021928994684423043,
+            0.022606401934291136,
+            -0.017972404868085358,
+            -0.026402048654300563,
+            0.003656646576244765,
+            -0.013129722990153284,
+            -0.018090360146368677,
+            -0.01624598856366761,
+            0.012397150789705746,
+            0.011741002980073786,
+            -0.004052422390675261,
+            -0.01836750423615859,
+            -0.020929335029248472,
+            -0.016327412417122416,
+            -0.0042615345226698015,
+            -0.008458117095535,
+            -0.005002582649405199,
+            -0.012158173174049797,
+            -0.020409616620022143,
+            0.03706368540614453,
+            0.0285565642978937,
+            -0.0041732933884846566,
+            -0.00607154864704767,
+            -0.011142150176334957,
+            0.008834431806990914,
+            0.006021232046087652,
+            -0.005028870471297621,
+            -2.3396043292139426e-05,
+            0.005112033372713154,
+            0.022822312435625314,
+            0.007271235299348001,
+            0.0035791789459533856,
+            -0.06340305811365254,
+            -0.005392000517722054,
+            0.022560201096078862,
+            0.012581401789917604,
+            -0.0032248219865223496,
+            0.025180718735801837,
+            0.0006451724964970927,
+            -0.009714514747463593,
+            -0.018389422219956937,
+            0.024728862150150004,
+            0.008760329201960964,
+            0.020988700992999768,
+            -0.014974763313686516,
+            0.003845297486923296,
+            0.0006445983847138896,
+            0.00750394530065325,
+            -0.0019792992847026793,
+            -0.008578254004439607,
+            0.025776677078006516,
+            0.016593438298233645,
+            -0.005695578040595484,
+            0.01604018433477423,
+            -0.0017497260488899657,
+            -0.021460543443428924,
+            -0.010265213095609554,
+            0.00042017209251926397,
+            -0.010570864288876387,
+            -0.018015790888295944,
+            0.017569414447505878,
+            -0.002601036282535863,
+            0.0072399377790164435,
+            0.030281707151249355,
+            0.0423183067569767,
+            0.009566493398534713,
+            -0.005928603293907002,
+            -0.0037936911251953654,
+            0.015477873304924939,
+            0.01513127995244932,
+            0.010741256034652754,
+            0.03015858543221727,
+            -0.029247492875021843,
+            -0.006690306960497428,
+            0.0046653727410352576,
+            0.004610405633429778,
+            -0.011692179099382171,
+            0.030055897497741527,
+            0.04569730592994177,
+            0.006736813244286893,
+            -0.004799978513117371,
+            0.002121067028052434,
+            0.011299889107780231,
+            0.007768137218140627,
+            -0.018927080714961783,
+            -0.0012222962555201746,
+            -0.006009911026251445,
+            -0.014001766533604134,
+            0.026941585318261085,
+            -0.003960837039351648,
+            -0.011848232223811548,
+            0.02903357274500332,
+            0.1105267435019781,
+            -0.006011029338781496,
+            -0.0053646998281195206,
+            -0.03242321704621517,
+            -0.015194102811332289,
+            -0.008860590919147765,
+            0.0056988205877963076,
+            0.01635957585050017,
+            0.020327661913559888,
+            -0.0008943408274661007,
+            0.016820860249955754,
+            0.02822175614376368,
+            -0.013854827839129885,
+            0.007906244895327178,
+            -0.030891381690403466,
+            0.015702117924362928,
+            -0.005383974926093977,
+            -0.012032955400313954,
+            -0.0026169511828034054,
+            -0.008618307035330928,
+            -0.0037356680273096105,
+            0.0009666430093455632,
+            0.011296307527684261,
+            0.004427516814276546,
+            -0.014484198313369681,
+            0.002829529156309784,
+            -0.008595300985863424,
+            -0.005122996529848575,
+            -0.02393571073523279,
+            -0.024119245543788293,
+            0.005223147047205755,
+            0.0019389305548629142,
+            6.344308292376579e-05,
+            0.006848965054539454,
+            -0.003810712095343454,
+            0.030149081317333418,
+            -0.01700322008272963,
+            0.0012288912810468446,
+            0.01193226302726573,
+            0.012332749933413344,
+            0.01688602002694633,
+            0.0028600984823761366,
+            -1.9882692552530084e-06,
+            0.021172977047420227,
+            -0.010514036509402561,
+            0.01460334896316005,
+            0.013091417303145243,
+            -0.04103572381458182,
+            0.0089746401463863,
+            -0.0027841289484449873,
+            -0.024252447218794344,
+            0.004446472209246425,
+            -0.018987867343500102,
+            -0.00881974688349601,
+            0.0005365625696166774,
+            0.012054719659248512,
+            0.01960300421041832,
+            -0.008553943699846696,
+            0.02111201386480852,
+            -0.0003501657740808554,
+            0.004177088753111459,
+            -0.012879944670340026,
+            0.0065222126975051685,
+            -0.007563354540402343,
+            0.0008168128607271199,
+            -0.018213241937597355,
+            0.012526561612175514,
+            -0.0008285725179894544,
+            -0.015737003159436623,
+            -0.011337688275309921,
+            0.004737346137976519,
+            -0.0027993545140114705,
+            0.0030843163114700313,
+            0.010153740340626154,
+            -0.008815302484351509,
+            0.004919275910399053,
+            -0.0017541042538165297,
+            0.002562816687714991,
+            0.014285249186421434,
+            -0.029816964593980453,
+            0.005357227738099325,
+            0.009154842495811638,
+            -0.00048782303639333943,
+            -0.013074976152495576,
+            -0.004054505559072866,
+            0.0028322102449485015,
+            -0.016877328079971877,
+            -0.012128199639779175,
+            0.023421330164419888,
+            -0.012565145848820333,
+            0.008609070188894315,
+            -0.010936598722317274,
+            -0.018401801734378692,
+            -0.01066119413133089,
+            -0.005254229780560808,
+            0.01234969508270852,
+            -0.0038427767204741877,
+            -0.006947594635952202,
+            -0.01756894543880097,
+            -0.007890897373987378,
+            -0.047246698568222466,
+            0.009617876521097624,
+            0.027157555263862723,
+            -0.011355758309667893,
+            -0.04050311381480088,
+            -0.01017774338301804,
+            0.02437257208967103,
+            -0.004056329295095018,
+            -0.008345450168215274,
+            -0.001599171409135998,
+            0.018358386463243014,
+            0.0027207445181251406,
+            -0.007292967749349572,
+            -0.011267861428599221,
+            0.01657507035043879,
+            0.015105061280816214,
+            0.021419854579631276,
+            -0.02088952706825817,
+            0.006295984081511286,
+            0.020973020776936996,
+            -0.03073246543555915,
+            0.0023720435876790876,
+            0.03363966216709148,
+            0.010412588613998134,
+            0.013023311559810034,
+            -0.0010217662933572685,
+            -0.0075567381078977215,
+            -0.008248818794295818,
+            0.011848200323549865,
+            0.005066463524567732,
+            -0.023174489297821385,
+            0.021823825355858556,
+            0.020368868756448753,
+            0.0026042856858509524,
+            0.003457067317891926,
+            0.0005539745115410357,
+            0.01062670844108589,
+            0.009948889803890108,
+            0.05100741556350918,
+            -0.006493714124814164,
+            -0.014021751055930108,
+            0.015063988102064609,
+            0.02488059237697814,
+            -0.0023499941119596228,
+            0.01087282522970853,
+            -0.02612218935931224,
+            -0.021423880914461062,
+            0.023389294741183817,
+            0.008967763132667892,
+            0.009637806238448432,
+            -0.010815706635624337,
+            -0.011390999462441815,
+            0.031750675158698435,
+            -0.01307829462211914,
+            0.00759749911213318,
+            -0.02169895957209634,
+            0.01387879409130081,
+            -0.008606362268863441,
+            0.032229658009209815,
+            -0.0009066257436946125,
+            0.011062436424260079,
+            0.028010486866468023,
+            -0.007835805879241447,
+            -0.019657486763625547,
+            0.003967480939987674,
+            -0.015577952617624695,
+            0.02188429826317828,
+            -0.020502758155233298,
+            -0.0002362500702658591,
+            0.0040935406015014265,
+            -0.010056462958039452,
+            -0.0057038083589231615,
+            0.011475170331848666,
+            0.0015345076181482267,
+            0.006359262977587827,
+            -0.020293014401106,
+            0.01369706440475074,
+            -0.003989482215141903,
+            0.007887824197823247,
+            -0.015925992045349828,
+            0.014370223626942766,
+            -0.02309139937510304,
+            -0.021808736033141787,
+            0.008142621179624918,
+            0.0108996933426044,
+            -0.007589050511458233,
+            -0.025971539510030145,
+            0.02250992080022228,
+            -0.024980833805349545,
+            0.024979008145354243,
+            -0.006622339940663538,
+            0.002566378488088469,
+            0.004913874740363343,
+            -0.008102103508732738,
+            -0.0028098905427027783,
+            0.0057865587191235565,
+            -0.0073509382400023995,
+            -0.014123756810537343,
+            -0.019453380788786282,
+            -0.04083629135586275,
+            -0.03177927047231389,
+            0.022927581786088044,
+            0.011557232599801421,
+            0.0017073832658476193,
+            0.02339552660545296,
+            -0.02247471449131386,
+            -0.017890866287500053,
+            -0.009775068417716798,
+            0.025184899522229628,
+            -0.04408541669037361,
+            0.012104255392615549,
+            -0.014669458760022245,
+            -0.0245258242481677,
+            0.006274252098230249,
+            -0.004161960147481513,
+            0.009299171724026455,
+            -0.010779421730862708,
+            -0.04713627671791054,
+            -0.005192684784377356,
+            0.02330975631183273,
+            -0.0036802876364168935,
+            -0.006197548344287773,
+            0.03048284650641829,
+            -0.0019503797069087845,
+            -0.005038254331399784,
+            -0.005524875459245342,
+            0.027783596017224226,
+            -0.024950083604372984,
+            0.010988398531346805,
+            0.008304937912856357,
+            -0.008485065895193593,
+            0.006572665697403556,
+            0.0011518846966102158,
+            0.011927348472282199,
+            0.015632588644277183,
+            -0.015559413716826264,
+            0.02262990861928957,
+            0.011446850286896198,
+            0.006506075777765303,
+            0.008015225053785824,
+            0.00010054430757843335,
+            0.0022975742216194894,
+            -0.024948925449668525,
+            0.026047924967851216,
+            0.01888211557921096,
+            0.0001632181017876392,
+            0.012553094476616919,
+            -0.015042621225242777,
+            -0.02782009986290425,
+            0.02606646297992026,
+            -0.016847653820092335,
+            -0.00045272379465939274,
+            0.020053091486344633,
+            -0.016210036358455773,
+            -0.007930550661303182,
+            -0.01123514849773852,
+            -0.0162675150307978,
+            0.01803076559921804,
+            -0.012160217543572069,
+            -0.02216594536057653,
+            0.002594675636753233,
+            0.016443123828746582,
+            0.02134480656850119,
+            0.026694575953313482,
+            -0.011470930697351533,
+            0.017768927048191052,
+            0.0008653408391607816,
+            0.001865765599795092,
+            0.025745954171649543,
+            0.01702677011968492,
+            -0.008474889109168202,
+            0.025347176686722313,
+            -0.0016686223698600755,
+            -0.002421921060670558,
+            -0.012265504084322075,
+            -0.0165214436458303,
+            -0.007545471138114208,
+            -0.015333235556146196,
+            -0.0051830106590765685,
+            0.000762746782664525,
+            -0.0018118022696726346,
+            0.10876576838038368,
+            0.0034948044540582626,
+            -0.011608460689239234,
+            -0.005870263500417206,
+            0.007510774308250375,
+            0.0023525201199670765,
+            -0.006846154764389575,
+            0.016082721103476375,
+            -0.018616626204086575,
+            -0.023593744079394,
+            0.004743481974389998,
+            -0.0009262286319469741,
+            0.00870122878944883,
+            0.02029523251993269,
+            -0.008608837818169835,
+            -0.0200567711498164,
+            0.007089477722191928,
+            0.005535149644041221,
+            0.0023640746516071774,
+            0.005256431824732927,
+            -0.0006118671321824933,
+            0.008209920240361308,
+            0.02448422122961942,
+            -0.015622892878860465,
+            -0.029770836607612903,
+            0.010535371601298532,
+            -0.00015994156470228762,
+            -0.007193425789067776,
+            -0.011046316463797128,
+            0.014728841492009627,
+            -0.0011488451990980268,
+            -0.02751404133172324,
+            0.017663132712536528,
+            -0.019055335178900415,
+            0.014643246200933506,
+            0.0025348345163347464,
+            -0.00016387173410118788,
+            0.022565953069005714,
+            0.013101158251834834,
+            -0.04272085076882881,
+            -0.0034442020309914823,
+            0.01041205537746844,
+            0.011059422927946552,
+            0.0116446016862942,
+            0.0012992535381387887,
+            -0.018670801241345514,
+            -0.011788984490283379,
+            -0.009109849103208542,
+            -0.021268488414296117,
+            -0.012523033785938503,
+            -0.011393853408628214,
+            0.010743646386439633,
+            0.0012964030814325153,
+            0.023809566099779113,
+            -0.003030535542277875,
+            -0.006526351867141901,
+            -0.000430699447948508,
+            0.00880561288237257,
+            0.009694646009259077,
+            -0.02079011138439714,
+            0.008294338719512953,
+            -0.01823922789220214,
+            0.011819769685519291,
+            0.006416044253693268,
+            -0.022799568832306067,
+            -0.009911619276012527,
+            0.007152786695893951,
+            0.008393677195358034,
+            -0.013569953300860946,
+            -0.006503988731872938,
+            0.02299498754558268,
+            0.018870619152710703,
+            0.012233964644738905,
+            0.024994616203196145,
+            0.03838969722235787,
+            0.005744255912479976,
+            0.027682123067526515,
+            -0.016791471092091604,
+            0.008459745383680297,
+            -0.022493868174622695,
+            0.012902262638928338,
+            -0.00460399519266689,
+            -0.0050431343566427886,
+            -0.018188192498411616,
+            0.033765105245412236,
+            -0.009159751849801773,
+            -0.0027150644587327763,
+            0.034778224828120714,
+            -0.0066502021793471075,
+            0.004525230911930458,
+            0.005096748531775903,
+            -0.016067049494073893,
+            0.033274672789154185,
+            0.01686831040872425,
+            0.016353966592947262,
+            -0.021631941883929817,
+            0.011748317746828848,
+            -0.005675701689448899,
+            0.0001273442591116645,
+            -0.020738819037390263,
+            -0.0037528323797385733,
+            0.0027725541343763104,
+            0.0059860509282538546,
+            -0.016556134518370437,
+            0.02033623476757896,
+            0.00019647023567641008,
+            0.0052895744489259875,
+            0.03253462247073863,
+            0.01043311189804924,
+            0.019182938878855655,
+            0.010051051068017807,
+            0.019523739618278902,
+            0.0281939391326836,
+            0.025685787482623124,
+            0.0029932688462254147,
+            -0.018668945726139946,
+            -0.0009326901784812825,
+            -0.007880352863625262,
+            0.010492063055849727,
+            0.022849902214818293,
+            -0.022619228461640748,
+            -0.02772644874135741,
+            0.018566981184526675,
+            -0.0022608185345048414,
+            0.020031103834122595,
+            0.007544464625538984,
+            -0.020935181884939796,
+            0.003407355515103286,
+            -0.023178230166337178,
+            -0.0005418155871620593,
+            -0.031039187720144967,
+            -0.028591174077243604,
+            -0.0007659241904525392,
+            0.011945344534674635,
+            -0.01688153017778311,
+            0.03357788163306365,
+            0.017667778671101954,
+            -0.014289109343082812,
+            0.01604684528817524,
+            -0.011054587325539086,
+            -0.0036959091138078683,
+            -0.005819853870004682,
+            -0.00203755010992417,
+            -0.02157944474979592,
+            0.005138614763345542,
+            -0.0033509138150859593,
+            -0.0024871060745482883,
+            -0.016936575573598154,
+            -0.021901830366872595,
+            0.016375173843894617,
+            0.016742770709641697,
+            -0.006606252542838758,
+            0.039771905404651496,
+            -0.004238071340064403,
+            0.00018364315349545657,
+            0.011690521707403059,
+            0.0010018392078563795,
+            -0.013651921068215563,
+            -0.010829994512513501,
+            0.012362647379788833,
+            0.011321403447882543,
+            -0.018942043512252663,
+            0.005763773608775982,
+            -0.009131565038297889,
+            0.002089216671850269,
+            0.005438887787329238,
+            -0.008183835275629478,
+            0.006690280329947951,
+            -0.01673229763262922,
+            0.015833397876364723,
+            0.00295474085300386,
+            0.003186554859191664,
+            -0.011797859525926772,
+            -0.020991927615680524,
+            0.01576413532776261,
+            0.0032403514518527287,
+            0.018819824905201635,
+            -0.026755218746060662,
+            -0.00012946467986739473,
+            -0.009552681500884924,
+            -0.0028941357661144826,
+            -0.012533800273548772,
+            0.013975640628890588,
+            0.016864739319953755,
+            0.0063941926578529,
+            0.01781016436903586,
+            -0.0026268224264609724,
+            0.015368284773773678,
+            -0.018122416427266218,
+            0.008686009180697955,
+            -0.006801074032721299,
+            -0.018537521789181048,
+            0.0013237338264105415,
+            0.027863741654435434,
+            -0.004670012662973029,
+            0.011066853115455966,
+            0.011704884649534624,
+            0.018954056680143452,
+            -0.0012626215734593707,
+            -0.0030850559460101866,
+            0.015610404665171367,
+            -0.005838226405658572,
+            -0.006861883833171456,
+            0.020872966233916948,
+            0.032951653763679145,
+            -0.006716215648910268,
+            -0.0021292599951407863,
+            0.010886596327481677,
+            -0.004901684487088035,
+            -0.008644584974417106,
+            -0.009587037829603014,
+            -0.03158137900717491,
+            0.024369089446706894,
+            0.012092412439477474,
+            -0.009601042951586109,
+            0.022690862146548786,
+            0.020752792552278052,
+            0.011412743968296708,
+            -0.010471860708195726,
+            0.02291536038589634,
+            -0.01268035142112569,
+            0.024168869865226136,
+            0.032482831775271157,
+            0.004400973556358811,
+            0.0022312906286559167,
+            0.0021175489661843985,
+            0.01691348460546639,
+            0.006050357346231155,
+            -0.007068133214675996,
+            0.011656003476409986,
+            -0.018487923180009552,
+            0.01598530891381001,
+            -0.004432366596808033,
+            0.02808371007118948,
+            -0.01337145216499331,
+            -0.010473700033368562,
+            -0.00537325376270318,
+            0.00010726431372328729,
+            0.006445701087575359,
+            -0.003324002570256215,
+            0.005255443653813165,
+            0.0007072577303252132,
+            -0.017825649146944237,
+            0.003050128130393781,
+            0.004825967170196209,
+            0.009018799597337101,
+            0.004103653497834134,
+            0.007249957401824204,
+            -0.012360355929483,
+            0.01441885664789707,
+            0.004401820363035808,
+            -0.0026190938717927086,
+            -0.0007163225760398784,
+            0.011124538295057815,
+            -0.013232822471919533,
+            -0.004895624967029031,
+            0.005281228035321008,
+            0.0059275753353475414,
+            -0.007507924998741734,
+            -0.004484140989405051,
+            0.011908076549392969,
+            -0.0037917291321648483,
+            0.006351543504726163,
+            -0.004409464890606014,
+            -0.0037760571637986827,
+            0.005303661313434895,
+            -0.00974511544836468,
+            -0.004204167898342997,
+            -0.0007093125445559502,
+            -0.006869250680233214,
+            0.00786202375969793,
+            0.005971033227117194,
+            0.010606085272785863,
+            0.006969437776542303,
+            0.004293129058563288,
+            0.004114452739123398,
+            0.004071017374828283,
+            -0.002445681623750424,
+            -0.00592480769363354,
+            -0.004617501328052899,
+            -0.012067458701076778,
+            -0.008335239153829813,
+            0.009025028998949398,
+            0.008653779421856954,
+            -0.019363018792788574,
+            0.02048011578221327,
+            -0.001188320960771659,
+            0.00772165044232104,
+            -0.01602086976290928,
+            -0.000907553574811279,
+            0.005464880629734941,
+            0.008267702703609528,
+            0.016136762227127184,
+            -0.03654209526492064,
+            0.005179791231187316,
+            -0.0037321883493524766,
+            0.016026013709955444,
+            -0.008215703411373182,
+            -0.006979816286354862,
+            0.02564060526656963,
+            -0.01450763388985563,
+            -0.0009847758228920171,
+            -0.016142304341196283,
+            -0.01407778890808498,
+            0.010815725252959506,
+            0.02050940280233426,
+            0.020684808835848383,
+            -0.0004965802897137407,
+            -0.0005781445948759235,
+            0.00036179841583004193,
+            0.022775804864342608,
+            0.015545681427566235,
+            0.018697154644607316,
+            -0.004543500782944068,
+            0.007028436664826706,
+            -0.023057215501463973,
+            0.009912581556496658,
+            0.0011795417129232596,
+            -0.008215027850908477,
+            -0.008554137076267468,
+            -0.008703378564520125,
+            -0.010448538943182258,
+            -0.011773811327190638,
+            0.004106616839960258,
+            0.013895013678116032,
+            0.004190549543250594,
+            0.0031571190600981074,
+            -0.0225385788570314,
+            -0.006243193914542516,
+            -0.011229641377928767,
+            -0.01674214043453308,
+            -0.007367389908398082,
+            -0.02386334969150212,
+            -0.028021339684039734,
+            0.0002794207370355175,
+            -0.02411861258809571,
+            -0.015958336406924612,
+            0.005268815864773021,
+            -0.01546610225139129,
+            0.00898600137695074,
+            -0.008396481690803911,
+            -0.011873486990732417,
+            0.0074743033679134105,
+            0.015336444788692,
+            -0.017676977079590463,
+            -0.010165317702640217,
+            -0.01690524990506033,
+            -0.013460451917691695,
+            0.010918550283871637,
+            0.00595960977280955,
+            0.009250157926438213,
+            -0.012447979337850305,
+            0.0062932435120863365,
+            -0.012673905639312696,
+            0.015073013552721701,
+            -0.03769166067890086,
+            -0.029992249463405965,
+            -0.00014990754540999537,
+            0.007288036630677436,
+            -0.0020735742042505426,
+            -0.018695120921334806,
+            0.0003860116505754868,
+            -0.003608995261360232,
+            -0.014900191335144973,
+            -0.013866216738266176,
+            0.025251475305524053,
+            -0.006788483107616133,
+            0.01223227424986397,
+            0.0034937277040015626,
+            0.03438333560343413,
+            0.008803852645562129,
+            0.014666206816302367,
+            -0.005709421955160623,
+            0.003742433643393657,
+            -0.0033706258426092744,
+            0.006906464031352315,
+            0.0076578888497945185,
+            0.009276592191710524,
+            0.008025321004305927,
+            0.004576470294889512,
+            0.002887346543823031,
+            -0.001925468214331082,
+            0.02125241545487307,
+            0.025671028960818562,
+            -0.00037467906648160044,
+            0.008981063192139038,
+            0.0009387900084090577,
+            0.0002735789978680719,
+            -0.008374290340790827,
+            0.006547013284767913,
+            -0.008274637814401893,
+            0.004041066530254948,
+            0.012897549078838394,
+            -0.009150863940242148,
+            -0.010211568689323908,
+            0.016575088843292566,
+            -0.012380099999574783,
+            -0.03420455578618604,
+            0.026550654680719987,
+            0.005324512466403649,
+            -0.0035670106106754245,
+            -0.010657928468872608,
+            0.0037176027853158167,
+            -0.01103891334887315,
+            0.008096915671982816,
+            0.022587659771132487,
+            -0.0013527917944322358,
+            -0.0018085171207746743,
+            0.0038503794877980586,
+            -0.011870484733648887,
+            -0.007881489342351292,
+            0.006462772034381968,
+            0.0315696514747912,
+            0.01326417653779577,
+            -0.020400369539140532,
+            -0.015223808971473513,
+            0.020406862349006738,
+            -0.023142176712111575,
+            -0.006254959806032159,
+            0.009039506093607052,
+            0.0009087520492922078,
+            -0.022766046838415165,
+            0.0009662037093312435,
+            0.004701966191595815,
+            0.005946458676746661,
+            -0.006358986495019584,
+            0.03237532341275415,
+            0.001075992257265679,
+            0.015892020271402903,
+            -0.017189453434945788,
+            0.0017514632185789299,
+            -0.014189278671121677,
+            -0.014265447388465805,
+            -0.029809460132405256,
+            0.0005557491671071693,
+            -0.0030324198417091704,
+            0.011038984948790744,
+            0.024808067315720335,
+            0.01145103533194245,
+            0.014432246996007246,
+            0.010552852947578788,
+            0.01513689173494211,
+            -0.013653722180565311,
+            -0.029009757850432732,
+            -0.005171522291984226,
+            -0.018001048743500243,
+            -0.02297482628212251,
+            0.003962613844063587,
+            0.013645125285548414,
+            0.006278662690346082,
+            0.023786381386575035,
+            0.0021738516691483163,
+            0.0009580591750940011,
+            -2.690805459518147e-05,
+            0.008894091070322921,
+            -0.004952203441302471,
+            -0.006553986197076102,
+            0.003570189040872711,
+            -0.000652714137223501,
+            0.021228360881461853,
+            -0.005758946428212041,
+            7.897153237591187e-05,
+            0.014242175492137011,
+            -0.005622256348934008,
+            -0.0035009192529791146,
+            -0.017339926143652377,
+            0.006248842389703505,
+            -0.01622011586976866,
+            0.025783086900871728,
+            0.018450783573486916,
+            -0.009109295293546267,
+            -0.013469521573468915,
+            -0.011259555201313116,
+            0.02773097452736661,
+            0.004631282554715908,
+            0.010787055220617197,
+            0.018228322680582464,
+            0.01461254259249465,
+            0.01784830193964943,
+            0.004664725683522323,
+            -0.0034339781146649384,
+            -0.0007692550982545399,
+            0.021176646860433974,
+            -0.014700178590249292,
+            -0.002701208084390198,
+            -0.008842788249773623,
+            0.0021799005046816444,
+            -0.008431299000066363,
+            0.006086447848362669,
+            -0.010793016137040172,
+            -0.015251381840165954,
+            0.0189503457652375,
+            -0.02660767353735015,
+            0.016423753600066336,
+            -0.00593016170680532,
+            -0.00482891500342298,
+            -0.027459214027897378,
+            0.010728039528784636,
+            0.0027866316191608297,
+            0.0022519974195000687,
+            0.010770472550461752,
+            -0.021427634123401408,
+            -0.01867717541189363,
+            0.008532444554010673,
+            -0.011457117350255678,
+            -0.007152301670244725,
+            -0.00348619742388509,
+            0.0154715607296319,
+            -0.00017898955187849443,
+            -0.009925365923862246,
+            0.02085093731060963,
+            -0.007175309371250024,
+            0.01223979370120031,
+            0.005256127464980655,
+            -0.008756077893734508,
+            -0.0028438320920811362,
+            0.021022708898898203,
+            0.027557382862284808,
+            -0.008342206702778956,
+            0.014526066580760486,
+            -0.011883797998224506,
+            -0.01088335352319397,
+            -0.03479444009382274,
+            0.030104838505937447,
+            -0.013305732257226255,
+            0.026773747582411633,
+            -0.0006968265979630097,
+            -0.017037729111584094,
+            -0.016206207453228583,
+            0.01625225876515654,
+            -0.0024290584023212254,
+            -0.009366287391027166,
+            0.01465885301870365,
+            -0.02428628495473643,
+            0.0058336571592899735,
+            -0.0006627935593670669,
+            -0.011531762838191602,
+            0.013275508677482622,
+            0.0019223252930710324,
+            0.010738622785461513,
+            -0.02030405805119001,
+            0.0009661595529028194,
+            0.009263716373456188,
+            -0.016791855691574455,
+            0.006057830565306032,
+            -0.03730952786847912,
+            0.0029341721588188943,
+            -0.014833895684348656,
+            0.005725615593914644,
+            0.001498755253963699,
+            -0.01820708352937343,
+            -0.03652692232407375,
+            0.00043665701245599497,
+            0.01913883585802903,
+            -0.007975988799790905,
+            0.020538595326175857,
+            0.008243569241768276,
+            0.008421404577931517,
+            0.002314738067443065,
+            0.0016702403734608475,
+            0.008279538988821465,
+            0.03679456861213399,
+            -0.01775927634148114,
+            -0.004836606118958108,
+            -0.0012151329859941935,
+            0.010644195708319258,
+            -0.02028470571116305,
+            -0.009287515726906219,
+            -0.0036506912311261536,
+            -0.013134467570201307,
+            0.030615862647585222,
+            -0.020265555441523482,
+            0.025634449009756734,
+            -0.009300770899023007,
+            0.009274072905190077,
+            -0.02097685009125915,
+            -0.009797008690754956,
+            -0.010661540100131183,
+            0.0019532291898233895,
+            0.013465005282849381,
+            -0.009151430285942593,
+            0.0071467934925125145,
+            -0.006655608635025676,
+            0.01020327561163374,
+            -0.001472597695707201,
+            -0.02443508800166349,
+            -0.0024998811551575563,
+            -0.01107321258077183,
+            0.02456164421154801,
+            -0.0019619696941303388,
+            0.0005043179699158694,
+            0.010951580163871948,
+            -0.00853593956432799,
+            0.023927336608555987,
+            -0.008855702280766362,
+            0.0010093679746665581,
+            -0.004863269000020324,
+            -0.004005691415117686,
+            0.001698212044518236,
+            0.01822024022650613,
+            -0.005707404715849356,
+            0.03722587302228139,
+            -0.01708925706528118,
+            -0.0065500772743641925,
+            -0.00015538686795361824,
+            0.010852925330001283,
+            0.010035359503147549,
+            0.008754719437760327,
+            0.004067820167375348,
+            -0.014861417626433965,
+            0.009346669946626484,
+            -0.020136029447262147,
+            0.007501669351583119,
+            0.023178490169347605,
+            -0.006840359541885465,
+            0.0006676450965357345,
+            -0.01283320739092204,
+            -0.015610538827742276,
+            0.006308543164022682,
+            0.002329420005586303,
+            -0.002295739637504597,
+            -0.02181189272979859,
+            0.006298804075737111,
+            -0.014398470126723037,
+            0.004007419942728446,
+            -0.0007497656993578918,
+            0.013863867106289847,
+            0.016451166273826513,
+            -0.006740209707090658,
+            0.039292260347522444,
+            0.01871723939316731,
+            0.009109005310505462,
+            0.005674782595856756,
+            -0.040588444759736254,
+            -0.008993850998154852,
+            -0.0010197185016393919,
+            0.004866150984083949,
+            -0.01141354216871663,
+            -0.018944881712855702,
+            -0.002646110151132781,
+            0.000308585036997592,
+            -0.010681954707313127,
+            -0.009091585604191968,
+            0.01751599002710486,
+            -0.006034424764302979,
+            0.00872057856926618,
+            0.004146961781109887,
+            0.023758065635807138,
+            -0.009727802672285544,
+            0.01176257119851645,
+            -0.0071822083765114165,
+            -0.005456197751568526,
+            0.0006726765703478756,
+            0.011011628907288813,
+            -0.008031792262886802,
+            0.009763340867550022,
+            -0.004279891138185881,
+            0.008355720233269893,
+            -0.008552596384067825,
+            -0.023952129458400563,
+            -0.014972827087541732,
+            0.014347535404251018,
+            0.011651260496709961,
+            0.008137738462704715,
+            -0.013624435569539285,
+            0.0031728721674035352,
+            0.00675830277744976,
+            -0.009184784787581942,
+            0.009161771311910335,
+            0.017733082848622708,
+            -0.001411542568472906,
+            -0.0017830167499636113,
+            -0.0016130003349297027,
+            -0.012173528776899308,
+            0.0011196437753239162,
+            0.02284887358189995,
+            -0.003054479643937998,
+            0.014614283913976716,
+            0.010689486748036806,
+            0.009643292262102645,
+            -0.00810930837603355,
+            0.001915955511174678,
+            0.02341863048264366,
+            0.015022291783105098,
+            -0.004625715884342598,
+            -0.011422973894623295,
+            0.004259757717597968,
+            -0.010332744704689279,
+            0.007373328473086509,
+            -0.02117940113144938,
+            -0.004672698999328004,
+            0.008896311380763048,
+            -0.007028862917603513,
+            -0.01732432285099198,
+            -0.028026173555055017,
+            -0.016765805892242715,
+            0.019819329380181,
+            0.0027213760678977965,
+            -0.013291251876048186,
+            -0.0032919084721662363,
+            0.0031214883337300926,
+            0.002553005168048115,
+            -0.0071104806975682885,
+            0.005310583052470149,
+            0.007359183453835734,
+            -0.008674133980486269,
+            0.017665958608281126,
+            -0.000521269431423141,
+            -0.0008020251454616109,
+            -0.0005442374261144659,
+            0.0006127687598711624,
+            -0.016387752806126442,
+            0.01666161912459431,
+            -0.010949228376419892,
+            0.015198110119816624,
+            -0.008966726991404666,
+            0.008600499511690635,
+            -0.007617043351740469,
+            0.009458641139070436,
+            0.011597957400445055,
+            0.012755037944648426,
+            -0.004403680820063604,
+            0.008519241473418196,
+            0.004807816191718704,
+            -0.008147125240985166,
+            0.007017601015398136,
+            0.01296082929962557,
+            0.00582353932160707,
+            0.0013634237970372698,
+            -0.0008246894322793264,
+            0.0022035670958705576,
+            -0.002480976219285683,
+            0.0056293125704658135,
+            0.012714850523393719,
+            -0.002497639449055006,
+            -0.007135683201188999,
+            0.014988249771676699,
+            0.014955895630183352,
+            -0.022994948923833993,
+            0.007121634434082677,
+            0.005835971825639792,
+            -0.0030507120821557008,
+            0.002145266953387315,
+            -0.027334525318320587,
+            -0.020359474070022213,
+            0.01901995970350808,
+            -0.015260454474605018,
+            0.02850460517419317,
+            -0.011624138792780823,
+            -0.02284242513728605,
+            -0.01277489645734181,
+            -0.004209529633534193,
+            -0.022684500849916897,
+            -0.004894414098943742,
+            0.01919165657366256,
+            0.013401560972810082,
+            -0.004313549020265677,
+            -0.019089369171102138,
+            0.03863248408555474,
+            0.010497951840980463,
+            0.03508157919687289,
+            -0.008597970944278045,
+            0.0023354831464448313,
+            -0.01563401808449416,
+            0.015551176593436512,
+            0.001075882802133782,
+            -0.011575692834796734,
+            0.01505716108583101,
+            0.031836226915668404,
+            0.0034740889237207494,
+            -0.00617206855371193,
+            0.0018420207148235309,
+            0.00418433325800744,
+            -0.0008167766650900381,
+            0.011160989178055499,
+            -0.018497472318034093,
+            -0.011951262348399922,
+            0.0013093376768780122,
+            0.00216347903176653,
+            0.0059233460419514045,
+            -0.001778091863744448,
+            -0.013317639704511549,
+            0.011433308892643204,
+            0.011596810919469926,
+            -0.01518143105525228,
+            -0.007497373860338335,
+            0.0050957665991362405,
+            -0.018974405317819656,
+            0.01872472902494418,
+            0.011859522398789263,
+            0.000792413325332956,
+            -0.0006503947129906383,
+            0.00808635755624928,
+            0.003749813548995223,
+            0.0033657799335872368,
+            -0.007862215303617144,
+            -0.0028014669075116876,
+            0.0009892310856576799,
+            3.834379772674841e-06,
+            0.015402211364447726,
+            -0.015264893397161102,
+            0.02488855366696587,
+            -0.0025643639525691083,
+            0.00953993868921819,
+            -0.017460590918773188,
+            -0.0017911712931080155,
+            -0.00402374787529868,
+            0.01838051670627292,
+            0.006753072393532114,
+            -0.010130676216186492,
+            0.004921510943896829,
+            0.007975201081296845,
+            -0.012589786869771252,
+            0.00018319054705558588,
+            -0.0022606180909706688,
+            -0.014791528072476771,
+            -0.0031171076370692706,
+            0.017873825561816544,
+            0.007593688317551089,
+            0.0037375070591255375,
+            -0.015863143964928172,
+            0.007192324815779268,
+            0.007522990667369163,
+            0.024504875417877527,
+            -0.026309816223960274,
+            0.024779042752840836,
+            -0.022104087334088543,
+            0.02165781034613893,
+            0.0319051741666687,
+            0.007538146369110045,
+            -0.01690181186708103,
+            -0.006595947249169521,
+            0.006373991305923002,
+            -0.023410814842488123,
+            -0.005494017994983188,
+            0.016903736810723716,
+            -0.01792400152372258,
+            0.000330865061689199,
+            -0.014660413003063089,
+            0.0040825880154611895,
+            0.008248037865838212,
+            0.011540333451799686,
+            -0.0033578830043738085,
+            0.018327525495676525,
+            0.020171745352159914,
+            -0.01138759854371235,
+            -0.0004221149053107702,
+            -0.022321409200941777,
+            0.005046865873981027,
+            0.022509643273449936,
+            0.001111272240506322,
+            -0.009540467231190513,
+            0.018376261423745193,
+            0.008480532791922674,
+            -0.020801891594934313,
+            0.008436374227753589,
+            -0.036175982655998785,
+            -0.010454118523313122,
+            -0.009328507118928412,
+            -0.001218633166683384,
+            -0.004142773602181115,
+            -0.0002494371580772723,
+            -0.011365548842513265,
+            -7.699548642315061e-05,
+            -0.006889339420877534,
+            -0.0037173070609413474,
+            0.014953978004522174,
+            0.011678957032329096,
+            -0.003501047405385249,
+            -0.0073984932581048915,
+            0.008493128567102267,
+            0.006163937693564917,
+            -0.017034696006318337,
+            -0.013268418502042576,
+            0.0037122128689059085,
+            -0.020067656401141467,
+            0.005148329575092478,
+            0.00459729178556505,
+            0.014031669598917171,
+            -0.01849235806598133,
+            -0.009298127665365717,
+            0.019716512208448042,
+            0.0094466405158553,
+            -0.0008988916054855498,
+            0.004264260983102956,
+            -0.00032312548009733906,
+            0.002279987803645912,
+            -0.0008902345374346296,
+            -6.145528806973222e-06,
+            0.003602229465148649,
+            0.012503366799381277,
+            0.007759393855482671,
+            0.03694402922447051,
+            -0.016692148708641812,
+            0.005873765863350649,
+            -0.014187609702966002,
+            -0.0037592517335595654,
+            -0.00749939010396784,
+            -0.0004308157309481927,
+            -0.0022282234382577843,
+            0.0021999996319642854,
+            0.0052176773733353925,
+            0.009353171029508977,
+            0.002155405440765206,
+            -0.00823565584692709,
+            0.009691447250255694,
+            -0.015228173095533029,
+            0.003853858061067531,
+            -0.008207910291425473,
+            0.01017972550354623,
+            -0.009695280827515048,
+            0.0028705587799331246,
+            0.03849358164875786,
+            0.01130021300937487,
+            -0.004934175853966098,
+            -0.014300147534361459,
+            -0.0007796051618993811,
+            -0.01737877501558196,
+            -0.008579119859817895,
+            -0.005434792488602325,
+            -0.02617338653902575,
+            -0.00838916510137031,
+            0.015344260464413648,
+            0.011202567142310492,
+            0.014124853863170132,
+            -0.023894182185855824,
+            0.008897943489322673,
+            -0.03001670823127986,
+            -0.011951507895140876,
+            -0.001530967140156163,
+            -0.015467812535428609,
+            0.0010110999991167703,
+            0.023623196504289838,
+            -0.007762735863648054,
+            0.012654652822750325,
+            0.012867958170258448,
+            0.018777522196571934,
+            0.006390986057157016,
+            0.00038045736974183937,
+            -0.009444911783760569,
+            0.009534310165417939,
+            0.008806031651319704,
+            0.0080954543713255,
+            0.013161833638467553,
+            -0.0022840529693446553,
+            -0.018060162439023572,
+            0.005469288590000856,
+            0.014354475778235131,
+            -0.0033504276914600585,
+            -0.001615191819057537,
+            -0.011088493180661976,
+            0.016039040565630432,
+            -0.023559046333872156,
+            -0.009482398794557168,
+            0.009918576960319548,
+            -0.0027308309155029842,
+            0.009212063547988036,
+            0.0160181200081126,
+            -0.01881373001297218,
+            -0.0024168618343039703,
+            0.016244889977796682,
+            0.0024104546308670025,
+            -0.00022606425578303413,
+            0.016792244238218245,
+            -0.0007616857360728623,
+            -0.004279120886828465,
+            0.00997764266310123,
+            -0.0075631917138318655,
+            7.3878137053128405e-06,
+            -0.002416364915008295,
+            -0.0028253498168087964,
+            0.02054128727706496,
+            -0.021281801678871736,
+            -0.005379259148621041,
+            0.02257306103533458,
+            0.015687105622685975,
+            -0.007843745111084413,
+            6.258498042855072e-05,
+            -0.0219506172079994,
+            0.0019925886885244454,
+            -0.007768133999661569,
+            0.016524850592507592,
+            -0.007957949830060354,
+            0.0030085044673901665,
+            -0.01361024620290394,
+            0.011563781350927133,
+            -0.01277407169098418,
+            0.00873716215529134,
+            -0.002986943948333218,
+            -0.017567646684567492,
+            -0.0053109390854215125,
+            0.004561236136619266,
+            0.0014045175130801308,
+            0.044958041347040185,
+            -0.005539136232703934,
+            0.003729517522976541,
+            0.018086335927383804,
+            -0.014199453349727933,
+            0.02079719360289355,
+            0.016663293302592568,
+            0.00991445855219406,
+            -0.00033967370741635794,
+            -0.0018726748849858836,
+            0.023373354806639515,
+            0.013258226441642464,
+            0.00861986998150065,
+            -0.0014340218811281975,
+            -0.008516541227326329,
+            0.018239230461345617,
+            0.005458629943667832,
+            -0.012421896028975013,
+            0.012732721349758832,
+            0.013559877132508506,
+            -0.013720939472355347,
+            -0.011545210834395917,
+            0.01616358223872008,
+            -0.019904594319155373,
+            -0.018260016571763644,
+            -0.0036124537591368864,
+            -0.01614510591132948,
+            -0.03864699384205461,
+            0.01619401261888991,
+            0.018351607973242735,
+            0.006666313000123845,
+            0.013647410298896132,
+            0.002597680061427196,
+            0.004545530267244157,
+            -0.018531082430861854,
+            -0.0077618625909995275,
+            -0.008085144524135461,
+            0.003744735923004248,
+            0.011059457364156674,
+            -0.011335432904114519,
+            -0.009514379703581616,
+            -0.00034500702695418285,
+            -0.007433132060989644,
+            0.01971420270994951,
+            0.0020753610290475642,
+            0.0034506706065161024,
+            -0.00628437298309269,
+            0.017966485854173167,
+            0.026789678964152908,
+            -0.007532446825077476,
+            0.021886182869092073,
+            -0.002864230253056434,
+            -0.0015870360120997492,
+            -0.014393519359298496,
+            -0.0014439412842964876,
+            -0.011527182213472351,
+            0.0005453493437517964,
+            -0.0006791481533636673,
+            0.014093767997869645,
+            -0.03288824642147967,
+            -0.003957301296796813,
+            0.014444592778851293,
+            0.007538052312876599,
+            -0.005057755472988024,
+            0.011357919882949903,
+            -0.002964451028327082,
+            0.016977317604413924,
+            0.0075201055190542146,
+            0.023391646079994784,
+            -0.016588823248397513,
+            -0.004504858318770565,
+            0.006660877064954993,
+            0.0034631255931793638,
+            0.0014484203909961972,
+            0.01912983041187014,
+            0.023439941258581887,
+            -0.014715886385460541,
+            -0.0013596123366001458,
+            -0.0029020372473879614,
+            -0.021414221343490594,
+            0.014976806123253727,
+            0.0074729481304182865,
+            0.010967351226114886,
+            0.009806663105020302,
+            0.015465076881663691,
+            -0.02109587527354609,
+            0.0011671121739822202,
+            0.020567458365051382,
+            -0.008618333549105003,
+            -0.018815706906171112,
+            -0.009804829158883123,
+            -0.00966828714719191,
+            -0.006078893678150302,
+            -0.00803783313451146,
+            0.005523023167028105,
+            0.0022647506733877573,
+            0.01652997604669998,
+            -0.008204200434441975,
+            -0.008404609147866811,
+            0.008489247125986078,
+            0.007546785673285454,
+            -0.0033093451283570316,
+            0.010614050963656496,
+            0.0030633179859393587,
+            -0.006940540318011839,
+            0.00894884343338777,
+            -0.002787816848430143,
+            0.014347598920245487,
+            -0.02441769157218451,
+            0.004717742147185283,
+            -0.012096876825337124,
+            0.006405151160675699,
+            0.0041568790527989905,
+            0.0016702066682613428,
+            -0.0041159211459004,
+            0.0035217362856754353,
+            0.0008893479210743541,
+            -0.07433312151926533,
+            -0.012629681414004524,
+            -0.0035797012285864486,
+            0.033657838713146884,
+            -0.019695806809053914,
+            0.004477927056336216,
+            0.011966065644382396,
+            0.0096717896523898,
+            0.003070713232538593,
+            0.006136726101157844,
+            0.006112385569942848,
+            -0.0245258242481677,
+            -0.00015284418977092615,
+            0.004821040694679944,
+            0.0020347004045943454,
+            -0.007532408630122127,
+            0.00750565079228405,
+            0.015327710245879296,
+            0.020989926895388523,
+            -0.010970305087289488,
+            0.017227663358885748,
+            -0.003352718496981928,
+            -0.006068892126250978,
+            0.0014285750322333686,
+            -0.010153799722483554,
+            -0.017307415843726394,
+            -0.006495008086641125,
+            0.0068983699213297075,
+            -0.001982568363667046,
+            0.0044032687939897855,
+            0.003333596495330213,
+            -0.0032171586993948696,
+            0.0072365127663633925,
+            0.013920326463272646,
+            0.006415916101287133,
+            -0.013675562796300214,
+            0.003003782758807665,
+            -0.02500375035312058,
+            0.011945988879567646,
+            -0.012708975181025296,
+            -0.0009488161508767279,
+            -0.0047093877027523014,
+            -0.0035750724825159503,
+            -0.002213435677949806,
+            0.02043567538125188,
+            -0.014304009033474186,
+            -0.01802899775650995,
+            0.008187227768196673,
+            -0.011217013195839762,
+            0.008300740149622635,
+            -0.03382831122769974,
+            0.012386335491333264,
+            -0.0276029934382802,
+            -0.0002243752790856292,
+            -0.004521387639042107,
+            -0.0010631721066129095,
+            -0.009491184157278669,
+            -0.03274559774619223,
+            0.015845705808609598,
+            -0.007204751554079551,
+            -0.020338337341085228,
+            0.0007143424225187802,
+            -0.021267094083674304,
+            0.014150458196361147,
+            -0.01851165553224022,
+            0.0033450353123933058,
+            -0.018775439467431685,
+            0.002782289821140401,
+            -0.0018711350216089556,
+            0.010502195116579441,
+            -0.00557814442809019,
+            -0.0007130962242755068,
+            -0.013072988887382269,
+            -0.002121841212381393,
+            -0.006653444103331213,
+            0.009394907690516121,
+            0.0028678074167728367,
+            -0.009601577734902932,
+            0.011153571864462314,
+            0.013046116892519414,
+            0.019812860521973938,
+            -0.00539047787915588,
+            -0.00555139958411929,
+            -0.011553689090179406,
+            0.009852063270798534,
+            -0.016326212706634428,
+            -0.00827402913960903,
+            0.008982303148984887,
+            0.015216947130173639,
+            0.01026889105233137,
+            0.003352028462252994,
+            0.003150801750574548,
+            0.002531620731961765,
+            0.026744691368233237,
+            -0.0015120532859309638,
+            0.0008324701960729416,
+            -0.014508988496009931,
+            0.022127525293249926,
+            0.02772430884375744,
+            -0.0012206102581726523,
+            -0.00018859639594030037,
+            0.017437360169132967,
+            -0.004697052522250805,
+            -0.0038111184371524645,
+            0.024015246959315593,
+            -0.038396108183338655,
+            -0.008416825232533771,
+            -0.006788816882617399,
+            -0.006043428198944214,
+            -0.009587366247924467,
+            0.007294190273465812,
+            0.00041966080769004275,
+            0.0024709692753547823,
+            -0.0005250904505954556,
+            0.03452858797518386,
+            0.003403515654095309,
+            -0.0003392060101496309,
+            -0.0032980864446307116,
+            0.0019125121221204029,
+            0.0032987517853214346,
+            -0.009116761755779728,
+            -0.038030085122772624,
+            0.006006334000396066,
+            0.007932785488877407,
+            0.00975009749364395,
+            -0.008120121513849434,
+            -0.008864461019882246,
+            0.004909348738798099,
+            -0.02273129283865203,
+            -0.01372636307167243,
+            0.02625229144362647,
+            -0.008523028986127237,
+            0.029005877324974557,
+            -0.022435059061182174,
+            -0.013135836801818153,
+            -0.004126433322528265,
+            0.010283975013795537,
+            -0.019732856462444506,
+            0.004343860338939202,
+            -0.004022387065239562,
+            0.023733721026005954,
+            0.0077578144626961245,
+            -0.008371631446146718,
+            0.012386735614153616,
+            0.0064789380436815714,
+            -0.0026638642827013837,
+            0.00136333052963848,
+            -0.007209517067748103,
+            0.016916505654836485,
+            -0.011819784026452944,
+            -0.005833909403097991,
+            0.008977527508585844,
+            -0.0024556434204852946,
+            0.02509293948918158,
+            -0.0237762245071987,
+            0.005288171058938098,
+            -0.00019047961064753207,
+            0.01822542471604037,
+            0.008127347078975925,
+            0.008361588667386882,
+            -0.011835407590060994,
+            0.010004482616697591,
+            -0.03378107067882622,
+            -0.015433906677849963,
+            -0.007702910250795191,
+            -0.0013877702264578865,
+            -0.002059152251387601,
+            -0.015745218548968338,
+            -0.026393221735795527,
+            0.006294583740809379,
+            0.016267525575472445,
+            0.023291528020980265,
+            -0.004683197930892445,
+            0.012912830736650997,
+            -0.002831583991769044,
+            -0.008904271394710672,
+            -0.011441882849296418,
+            -0.012202629605208118,
+            -0.012470773094383122,
+            -0.014560694782340029,
+            -0.017045701603711442,
+            0.012139665577738337,
+            -0.020079061996873537,
+            0.013911518328924777,
+            0.0347210775522539,
+            -0.017176114991595492,
+            -0.0018941772168839354,
+            0.019702722555622765,
+            -0.004221469612316137,
+            0.01212088353057276,
+            0.0063707489618353725,
+            0.015298014532351866,
+            0.0038643174573634725,
+            -0.012181479717826009,
+            -0.007374005740809251,
+            0.019411457664902376,
+            -0.004902591844710158,
+            0.014591446725717338,
+            0.015069599884557399,
+            -0.008675672164385891,
+            0.002544198867387345,
+            0.01951865904496276,
+            -0.01762101777798757,
+            0.01665654707715257,
+            -0.00458402229480694,
+            -0.009936206685538294,
+            -0.013345740581684638,
+            -0.005336416724426228,
+            -0.019469457091169776,
+            0.0061608454109892425,
+            0.018931676738452464,
+            -0.00533013594306834,
+            0.009310871143727683,
+            -0.0037670793158045,
+            -0.005045151384034014,
+            0.0002041909886640482,
+            -0.04152751596227068,
+            -0.008340911336217609,
+            -0.0006651584370005978,
+            -0.02505748815737022,
+            0.025954102004571416,
+            0.007205602377144382,
+            0.022602344303400163,
+            -0.017168610245491406,
+            0.004459735690564842,
+            -0.005882024209102714,
+            0.013217517324383834,
+            0.02255689669623969,
+            0.0017984057469785134,
+            0.005699128497215882,
+            -0.02353849721739205,
+            -0.0034711576718386185,
+            0.015924913374553738,
+            -0.003604478141616307,
+            -0.0034993903155063817,
+            0.012592308877056896,
+            0.012864938348003884,
+            0.01567418642605425,
+            0.008801753550506258,
+            0.015606443428689292,
+            0.010589933994371628,
+            0.008928721556432368,
+            -0.005805129999077126,
+            -0.001025103064751441,
+            -0.015907755521365925,
+            0.0045874330999415985,
+            0.031701049450012876,
+            0.012594483199773746,
+            0.016783858873835704,
+            -0.01778536268185936,
+            0.00789523134933153,
+            -0.0012706411671644011,
+            0.008325108613530198,
+            -0.001677173669891956,
+            0.029540136288412976,
+            0.011739999605699826,
+            0.015670966746894652,
+            0.008732334090105053,
+            -0.016237731209426752,
+            0.045334690225148,
+            -0.016911928942220152,
+            -0.0021139311364009113,
+            -0.01058970056572469,
+            -0.021705338531028618,
+            0.007324792557651635,
+            -0.006556582104686972,
+            0.0018243188317438144,
+            0.010780292738415671,
+            -0.006471850255151019,
+            0.0006997395548596146,
+            0.003907433192252573,
+            -0.023129313269763224,
+            -0.002550875642832235,
+            -0.02237917712764273,
+            -0.003107775639788617,
+            -0.02621608419628401,
+            -0.00012482032027919036,
+            -0.016207377503696517,
+            -0.0038712739995960753,
+            -0.023315508569288476,
+            -0.014963211090562272,
+            -0.01633110043829169,
+            -0.004241081909202262,
+            -0.008295402942465353,
+            0.004976396647108597,
+            -0.007402107755970892,
+            -0.020321297548927468,
+            0.0020986169430507757,
+            0.016803273183070004,
+            0.01434729320047051,
+            0.018612970083178984,
+            0.00603868809307069,
+            0.014278001198851581,
+            -0.0026350854954985117,
+            0.017171016209107656,
+            -0.006968968261909459,
+            0.006881847166565678,
+            0.011454001418121107,
+            0.02140935294963631,
+            0.011121833270235448,
+            -0.012801756637470525,
+            0.0010870041670931706,
+            -0.009655388034828094,
+            0.002094601818569396,
+            -0.00828486034340067,
+            -0.009190975158350585,
+            0.0017762581946855365,
+            -0.007261996868906865,
+            0.004674405202069334,
+            0.0030526253833799426,
+            -0.0014082232541965316,
+            0.010419302200840906,
+            -0.0024200053377751805,
+            -0.012208685875688886,
+            -0.0033037330470612695,
+            -0.017975940046938498,
+            -0.006958042903103732,
+            -0.0014496522238488239,
+            -0.011711502944602638,
+            -0.01016130418405875,
+            0.003321436189059608,
+            -0.00538282506552848,
+            -0.009353581612407435,
+            0.013884147453136417,
+            0.0001309738696853529,
+            0.0005878330075918333,
+            0.023495280602974504,
+            0.006035028203642483,
+            0.008087594627921345,
+            0.013144288945385904,
+            -0.01023995000994454,
+            -0.022574958230635715,
+            -0.0243435727055662,
+            0.007867022215096418,
+            -0.01573478573423379,
+            0.02324699908478915,
+            -0.007929809922043611,
+            -0.00561630876431978,
+            0.0020982723335057625,
+            -0.013699317926847062,
+            0.008578431496569186,
+            -0.011271381061801356,
+            0.0014450645788554738,
+            -0.019581451799757368,
+            0.018059311800267413,
+            -0.010917911869373693,
+            -0.004302687883018328,
+            0.009466103226841532,
+            0.034049086327052606,
+            -0.0005689605174080737,
+            -0.007410970644432269,
+            -0.006594150877740108,
+            -0.00022810271313786974,
+            0.018766765773409123,
+            -0.0029041815756733333,
+            0.005758781908059541,
+            -0.01563519107805783,
+            0.006571628287171105,
+            -0.006132372252825422,
+            0.0012423261344297225,
+            0.01513074831647933,
+            -0.026306320707079913,
+            -0.015491710256868008,
+            -0.007857352164317122,
+            -0.024893430640551818,
+            0.002253574511878983,
+            0.017579477884206534,
+            0.020505606775521504,
+            -0.020324505029926956,
+            -0.02513332780748072,
+            0.005150966733930881,
+            -0.026580994694344065,
+            0.0037390577082486594,
+            0.1241031878517593,
+            -0.002138404898309707,
+            0.002720723037908521,
+            -0.011425036886011179,
+            0.0022399461009316216,
+            0.005389407121650262,
+            -0.019552543955456567,
+            0.05950746473186985,
+            -0.00638023587296763,
+            -0.004023798021748334,
+            0.002170545188776296,
+            0.001785855390676113,
+            -0.005831534307818096,
+            0.008643501093778434,
+            -0.017602787281866816,
+            -0.02184792441891879,
+            -0.0052215705748838905,
+            -0.021252169339074325,
+            -0.008509621422071373,
+            -0.013000137524907565,
+            0.002893880203681282,
+            -0.015821609145923173,
+            0.010397692426614709,
+            -0.01638916243796692,
+            -0.010283706937701179,
+            -0.01103621454761223,
+            -0.00373552043529251,
+            0.01563553822905027,
+            -0.03188533919653949,
+            -0.004873550301425776,
+            -0.005183590054152726,
+            0.018218971950313273,
+            0.006004328765748325,
+            0.004357734370247252,
+            -0.014230318971458851,
+            0.0064792070801456235,
+            0.0215589173714926,
+            0.0055272493764401305,
+            0.014643335833970546,
+            0.010618070641357003,
+            -0.0022177303813575846,
+            -0.0024402761885758007,
+            0.017541799738501007,
+            -0.005445774414228888,
+            -0.00023502936119626386,
+            -0.00537569046685066,
+            -0.005974703247446444,
+            -0.010712244203425036,
+            0.01843613166635215,
+            -0.009938564643456672,
+            -0.017388592763277214,
+            -0.0015797412875366156,
+            0.02028434083164951,
+            -0.012761534382003973,
+            0.0031906812114950982,
+            0.000788436210275669,
+            0.020589196433948156,
+            0.0007726855314648301,
+            -0.005481260996289263,
+            0.007165014826294827,
+            -0.02956334397512344,
+            0.004095715232997753,
+            0.0003568512047626187,
+            0.027290768799658912,
+            -0.0017631129386615335,
+            0.01552542586826711,
+            -0.020871286597113327,
+            0.004388125824324094,
+            -0.016926008631435403,
+            0.005099899758322398,
+            0.012560923968841436,
+            0.002781049890810361,
+            0.010889021753593997,
+            -0.008500358897706648,
+            -0.021353460542554444,
+            0.012663934009161902,
+            -0.018657094130860155,
+            -0.029156017587920958,
+            0.0018889388761887053,
+            -0.028158076315864772,
+            -0.003048854609750186,
+            0.006812072971079798,
+            -0.035217412591558014,
+            -0.02251675180883659,
+            0.007396524512996277,
+            -0.00995571600201846,
+            -0.011912753666073882,
+            0.010675240017836214,
+            -0.02645155743041798,
+            -0.018137648527851668,
+            0.0010666944556449141,
+            0.012588800077165337,
+            -0.004561825939541096,
+            0.007555558090958502,
+            -0.028558603070876605,
+            0.014816816621445352,
+            -0.01313764261744744,
+            0.016036065063831813,
+            -0.0037797584923144134,
+            0.022217516257304868,
+            0.011131597210722573,
+            0.008795376259031849,
+            -0.004950080558710245,
+            0.021409590863550987,
+            0.005574351961953293,
+            0.051358937509801736,
+            0.031972009383746405,
+            0.0019821003238211276,
+            -0.003251257665378234,
+            0.003313045083311649,
+            0.008294776423730887,
+            -0.00940948257229233,
+            0.005114631360856872,
+            0.028053227377164806,
+            0.016738013668710083,
+            -0.018755924318109216,
+            0.006606711923035118,
+            -0.02702692774986498,
+            0.017950884766781056,
+            -0.022054653845163,
+            0.0003755944703013671,
+            0.007766253634286795,
+            0.0016206993755471613,
+            0.010682347575070117,
+            0.0013842930079185415,
+            0.01705234255261425,
+            0.023975191051210525,
+            -0.002976402980885077,
+            0.017466027107477605,
+            0.012734909275073793,
+            0.018143161773162015,
+            -0.0038399373197265827,
+            0.006929440921265761,
+            -0.012003095293274786,
+            -0.0008883470597690258,
+            0.0020490527756307267,
+            -0.004231091383919565,
+            0.012128848448049833,
+            0.008340302519160299,
+            0.012989849936937843,
+            0.002974488532907779,
+            0.003198854381461613,
+            -0.005309753760578859,
+            -0.0017024982049936384,
+            -0.01763395035509857,
+            -0.01819680226734474,
+            0.028417724274372468,
+            -0.020217879775485576,
+            -0.050118184195774004,
+            0.0025509548991664425,
+            0.014185761006969567,
+            -0.0012021669304388173,
+            -0.0051260576846881515,
+            -0.0005774778872927327,
+            0.005718864494530703,
+            -0.016918749210038116,
+            -0.0015472638835434902,
+            0.008125845907627804,
+            0.020347547062899797,
+            -0.002036218493206914,
+            -0.01079739518866248,
+            0.0038112958713600906,
+            0.02169339303105242,
+            0.01667594322285835,
+            0.012338193092567552,
+            -0.009798409233733167,
+            0.0003198215558702285,
+            -1.6308740825163337e-05,
+            0.0076870827874737325,
+            0.004356644356656038,
+            0.003500023005601613,
+            -0.019968549869439228,
+            -0.008634740687755112,
+            0.012781565963545847,
+            -0.0025637758121512927,
+            0.010243567793632335,
+            -0.00042296044205132426,
+            -0.003069251553118065,
+            -0.009344199449486244,
+            -0.004694953215957636,
+            -0.009583993744231791,
+            0.013783878036521561,
+            -0.007705982042252044,
+            -0.01549567538665377,
+            0.00020666298495823114,
+            0.0005515775451348472,
+            0.001667367278489879,
+            -0.004518431526236256,
+            0.03302324454129141,
+            0.009531990824506103,
+            -0.0004937348108340977,
+            -0.004047613286672229,
+            -0.0327383112522095,
+            0.012456150771224677,
+            -0.009677533214778067,
+            0.0006455811519981003,
+            0.020012976827426605,
+            -0.00020277620481825933,
+            0.002168432791818704,
+            -0.015657551629104767,
+            0.0032924357559238502,
+            0.006161295735841883,
+            0.013814165416788555,
+            0.0005164426482972658,
+            0.01076512534105537,
+            -0.006498977875968572,
+            -0.010325144017898257,
+            0.004804158067334203,
+            0.0104064855510473,
+            -0.0012188100892986933,
+            0.005171197790296134
+          ],
+          "title": "Mixture-Of-Experts With Expert Choice Routing",
+          "keyphrases": [
+            "Mixture-Of-Experts",
+            "Expert Choice Routing",
+            "sparsely-activated models",
+            "training efficiency",
+            "downstream performance"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "86d0cb2e-1f6d-4e38-a077-7b7de2fe9e4a",
+        "properties": {
+          "page_content": "3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n",
+          "embedding": [
+            0.010104401037096977,
+            -0.020574966445565224,
+            -0.02167346142232418,
+            -0.031542472541332245,
+            0.03588413819670677,
+            0.03464615345001221,
+            -0.009755672886967659,
+            0.046869076788425446,
+            0.004803731571882963,
+            0.0364421010017395,
+            0.016547156497836113,
+            -0.007223033811897039,
+            0.009546436369419098,
+            -0.0017371026333421469,
+            0.005762734450399876,
+            0.04048734903335571,
+            0.011603932827711105,
+            0.01784616895020008,
+            0.011150586418807507,
+            -0.02915368229150772,
+            0.005383492447435856,
+            -0.01708768494427204,
+            -0.04317255690693855,
+            -0.006194285582751036,
+            0.009546436369419098,
+            0.005871712230145931,
+            0.027340294793248177,
+            0.0036812627222388983,
+            -0.055901139974594116,
+            0.023800702765583992,
+            0.021708333864808083,
+            0.010984940454363823,
+            -0.01007824670523405,
+            -0.010601338930428028,
+            -0.026573093608021736,
+            -0.021969880908727646,
+            0.04634598642587662,
+            0.007685098797082901,
+            -0.006399163510650396,
+            0.042475100606679916,
+            0.03703493997454643,
+            0.012728581205010414,
+            -0.012772172689437866,
+            -0.014175803400576115,
+            -0.003430614247918129,
+            -0.0012979229213669896,
+            -0.004712190479040146,
+            0.0065735275857150555,
+            0.009311044588685036,
+            -0.026172054931521416,
+            -0.001842810888774693,
+            -0.021900134161114693,
+            0.02798544242978096,
+            0.020609840750694275,
+            -0.049379922449588776,
+            0.028072625398635864,
+            -0.004890913609415293,
+            0.06461934745311737,
+            0.047461915761232376,
+            0.014602995477616787,
+            0.03794163465499878,
+            -0.034035876393318176,
+            -0.029275737702846527,
+            -0.003177786245942116,
+            0.03234454616904259,
+            0.007654585409909487,
+            -0.01033979281783104,
+            0.0010679803090170026,
+            0.011717269197106361,
+            -0.005706066265702248,
+            -0.044532597064971924,
+            0.010496720671653748,
+            -0.014925569295883179,
+            0.002881367225199938,
+            0.004034350160509348,
+            -0.020017001777887344,
+            0.03030448593199253,
+            0.0268520750105381,
+            0.014899414964020252,
+            0.010740830563008785,
+            0.010653648525476456,
+            0.03204812481999397,
+            -0.0026961052790284157,
+            0.009555154480040073,
+            0.0364421010017395,
+            -0.029415227472782135,
+            -0.01360040158033371,
+            -0.062178246676921844,
+            0.010540311224758625,
+            0.0028072623535990715,
+            -0.02815980650484562,
+            0.0018340926617383957,
+            -0.006569168530404568,
+            0.0036986989434808493,
+            -0.033094312995672226,
+            0.0727447122335434,
+            0.010653648525476456,
+            0.00023280335881281644,
+            -0.024044813588261604,
+            0.006394804455339909,
+            -0.015762517228722572,
+            0.021969880908727646,
+            0.0006560450419783592,
+            0.004008195362985134,
+            0.0229811929166317,
+            -0.02859571762382984,
+            -0.004690395202487707,
+            0.03815086930990219,
+            0.02148166112601757,
+            0.08676359057426453,
+            0.01007824670523405,
+            -0.017593340948224068,
+            -0.011211613193154335,
+            0.018482597544789314,
+            -0.04278895631432533,
+            -0.017689241096377373,
+            -0.03965040296316147,
+            -0.03215274587273598,
+            0.0114818774163723,
+            -0.025038689374923706,
+            0.01996469311416149,
+            0.02566639892756939,
+            -0.0011344566009938717,
+            -0.025422289967536926,
+            -0.07155904173851013,
+            0.04292844980955124,
+            -0.023696085438132286,
+            0.01780257746577263,
+            0.001439593848772347,
+            -0.0022144243121147156,
+            -0.00422179140150547,
+            0.0037183149252086878,
+            -0.020261110737919807,
+            -0.041882261633872986,
+            0.01856978051364422,
+            0.010479284450411797,
+            0.046834204345941544,
+            0.0633290484547615,
+            -0.021132932975888252,
+            -0.0072448295541107655,
+            0.017375385388731956,
+            -0.022894009947776794,
+            0.003243172774091363,
+            -0.0038142153061926365,
+            -0.029450101777911186,
+            0.013975284993648529,
+            0.028508534654974937,
+            0.08578714728355408,
+            -0.01956365443766117,
+            0.012022406794130802,
+            -0.033286113291978836,
+            0.037662651389837265,
+            -0.009799264371395111,
+            0.023417102172970772,
+            -0.025404853746294975,
+            -0.021708333864808083,
+            -0.025352545082569122,
+            0.010705957189202309,
+            -0.014995315112173557,
+            0.0059632533229887486,
+            -0.005889148451387882,
+            -0.0062945447862148285,
+            -0.018691834062337875,
+            0.010470565408468246,
+            0.007685098797082901,
+            0.047985006123781204,
+            0.029816266149282455,
+            0.005858634598553181,
+            -0.07769665867090225,
+            0.038325235247612,
+            0.017514877021312714,
+            0.036546722054481506,
+            0.01543994341045618,
+            0.033443041145801544,
+            -0.026538221165537834,
+            0.017680522054433823,
+            0.0032475318294018507,
+            0.01094134896993637,
+            -0.03197837993502617,
+            0.044916197657585144,
+            -0.00843050517141819,
+            0.002720080316066742,
+            0.01972058229148388,
+            0.013312701135873795,
+            0.04062684252858162,
+            0.027915697544813156,
+            0.034210242331027985,
+            0.03815086930990219,
+            -0.01332141924649477,
+            0.010453129187226295,
+            0.029380355030298233,
+            0.03515180945396423,
+            0.02796800620853901,
+            0.0027636713348329067,
+            0.0041890982538461685,
+            -0.026590529829263687,
+            -0.006721737328916788,
+            0.009964910335838795,
+            0.007039951626211405,
+            0.016390228644013405,
+            -0.012780890800058842,
+            -0.028124934062361717,
+            0.0220221895724535,
+            -0.028648026287555695,
+            0.030740395188331604,
+            0.008626664988696575,
+            -0.01083673071116209,
+            0.013208082877099514,
+            0.009895164519548416,
+            -0.029833702370524406,
+            0.004243587143719196,
+            -0.006250953767448664,
+            -0.004468080587685108,
+            -0.025021253153681755,
+            -0.01933698169887066,
+            0.00797280017286539,
+            0.0029532923363149166,
+            0.005021686665713787,
+            0.009380790404975414,
+            -0.022545281797647476,
+            0.004284998401999474,
+            0.02127242274582386,
+            0.022876573726534843,
+            -0.018325669690966606,
+            -0.018901070579886436,
+            -0.011438286863267422,
+            -0.01914518140256405,
+            -0.011865478940308094,
+            0.044393107295036316,
+            -0.012711144983768463,
+            -0.009694645181298256,
+            0.020574966445565224,
+            0.006774046458303928,
+            -0.005043482407927513,
+            0.017872322350740433,
+            -0.029484974220395088,
+            -0.010932630859315395,
+            -0.0035592077765613794,
+            0.011072122491896152,
+            0.010243892669677734,
+            -0.00997362844645977,
+            0.01550968922674656,
+            -0.050879452377557755,
+            -0.010932630859315395,
+            0.007881258614361286,
+            0.015082497149705887,
+            -0.0110285310074687,
+            0.02339966595172882,
+            0.012161897495388985,
+            0.016224581748247147,
+            -0.003081885864958167,
+            -0.014829669147729874,
+            0.008221268653869629,
+            0.017384104430675507,
+            0.007881258614361286,
+            0.05712168663740158,
+            0.02338222973048687,
+            -0.03947603702545166,
+            0.006538654677569866,
+            -0.029973194003105164,
+            0.05579652264714241,
+            0.007994594983756542,
+            -0.029833702370524406,
+            0.053215932101011276,
+            -0.023434538394212723,
+            -0.0024193022400140762,
+            0.060853078961372375,
+            -0.0007007258245721459,
+            -0.013548092916607857,
+            0.0013011923292651772,
+            0.02435866929590702,
+            -0.056075502187013626,
+            -0.04899631813168526,
+            -0.02148166112601757,
+            0.02857828140258789,
+            -0.02453303337097168,
+            0.0077156126499176025,
+            -0.015405070967972279,
+            0.04735729843378067,
+            0.036721084266901016,
+            0.018116433173418045,
+            -0.008731283247470856,
+            0.02008674666285515,
+            0.04749678820371628,
+            -0.007977158762514591,
+            -0.004742704331874847,
+            -0.02833417057991028,
+            0.005152460187673569,
+            -0.033216364681720734,
+            -0.009485408663749695,
+            -0.027462350204586983,
+            -0.011133149266242981,
+            -0.02125498652458191,
+            -0.0030317562632262707,
+            0.022336045280098915,
+            0.0041389684192836285,
+            5.1253518904559314e-05,
+            -0.0020389705896377563,
+            0.040522221475839615,
+            -0.024951506406068802,
+            -0.04470696300268173,
+            -0.00825178250670433,
+            0.01704409345984459,
+            0.0038556267973035574,
+            -0.05475033447146416,
+            -0.006250953767448664,
+            -0.0029707287903875113,
+            0.01304243691265583,
+            -0.023539157584309578,
+            -0.00863538309931755,
+            0.016573309898376465,
+            0.002870469354093075,
+            0.009807982482016087,
+            -0.02800287865102291,
+            -0.014629150740802288,
+            0.007406116463243961,
+            0.01841285265982151,
+            0.02817724272608757,
+            -0.0555872842669487,
+            -0.03225736320018768,
+            0.02067958563566208,
+            -0.009040780365467072,
+            -0.06409624963998795,
+            0.007833308540284634,
+            -0.001987751107662916,
+            0.01799437776207924,
+            0.016948193311691284,
+            0.04160328209400177,
+            -0.019581090658903122,
+            -0.022283734753727913,
+            -0.016599465161561966,
+            -0.0364072285592556,
+            -0.028700334951281548,
+            -0.009712082333862782,
+            -0.008299732580780983,
+            -0.028857262805104256,
+            -0.0017839629435911775,
+            0.01342603750526905,
+            -0.03759290650486946,
+            0.006176849361509085,
+            -0.021603714674711227,
+            0.0028486738447099924,
+            -0.03703493997454643,
+            0.007842026650905609,
+            -0.0030339357908815145,
+            -0.01613740064203739,
+            -0.02125498652458191,
+            -0.01561430748552084,
+            0.014725050888955593,
+            -0.0002036518562817946,
+            0.0017142173601314425,
+            0.0229637548327446,
+            0.007484580390155315,
+            0.009328480809926987,
+            -0.008535124361515045,
+            -0.029798829928040504,
+            -0.013382446952164173,
+            0.006586604751646519,
+            0.018534906208515167,
+            0.019999565556645393,
+            0.005501188337802887,
+            -0.04958915710449219,
+            -0.03584926575422287,
+            0.021621152758598328,
+            -0.05890020355582237,
+            0.006394804455339909,
+            -0.0017425514524802566,
+            -0.008622306399047375,
+            0.0033150978852063417,
+            -0.005832480266690254,
+            -0.026294110342860222,
+            0.015570716932415962,
+            0.0086135882884264,
+            0.012275234796106815,
+            0.007488939445465803,
+            0.015980472788214684,
+            0.02852597087621689,
+            -0.02451559714972973,
+            -0.05788889154791832,
+            0.03427998721599579,
+            -0.019075434654951096,
+            0.014210676774382591,
+            0.0440792515873909,
+            0.005475033540278673,
+            -0.006551732309162617,
+            0.025003815069794655,
+            0.013399883173406124,
+            -0.010034655220806599,
+            -0.03407074883580208,
+            0.04376539587974548,
+            -0.003430614247918129,
+            -0.07121030986309052,
+            0.005479392595589161,
+            0.0014810053398832679,
+            0.013103464618325233,
+            0.015283015556633472,
+            -0.003430614247918129,
+            -0.023643774911761284,
+            -0.01956365443766117,
+            -0.014088621363043785,
+            -0.018168741837143898,
+            0.029484974220395088,
+            0.003539591794833541,
+            0.005004250444471836,
+            -0.0006091846735216677,
+            -0.0014341449132189155,
+            0.004677317570894957,
+            0.014027593657374382,
+            0.04080120474100113,
+            -0.0364769771695137,
+            0.0025958458427339792,
+            -0.04275408387184143,
+            -0.03874370828270912,
+            -0.0046380856074392796,
+            0.018360542133450508,
+            0.026485910639166832,
+            -0.025910509750247,
+            0.02320786565542221,
+            0.0100608104839921,
+            -0.015675334259867668,
+            -0.005008609499782324,
+            0.028090061619877815,
+            0.025369981303811073,
+            0.01285063661634922,
+            -0.004533467348664999,
+            0.014690177515149117,
+            -0.012623962946236134,
+            0.0026045641861855984,
+            0.01574508100748062,
+            -0.00010305192699888721,
+            0.03490769863128662,
+            0.001937621389515698,
+            0.018918508663773537,
+            -0.04693882167339325,
+            -0.02699156664311886,
+            -0.027689022943377495,
+            -0.0172620490193367,
+            0.03874370828270912,
+            -0.006503781769424677,
+            -0.012728581205010414,
+            0.020261110737919807,
+            0.004282819107174873,
+            0.0017044093692675233,
+            -0.04066171497106552,
+            -0.028630590066313744,
+            -0.022283734753727913,
+            -0.044916197657585144,
+            -0.005618884228169918,
+            -0.02183038927614689,
+            -0.019057998433709145,
+            -0.05342517048120499,
+            0.0010527233825996518,
+            -0.07811512798070908,
+            -0.005370415281504393,
+            0.04048734903335571,
+            0.004686036147177219,
+            0.009424380958080292,
+            -0.028682898730039597,
+            0.01025261078029871,
+            -0.036337483674287796,
+            -0.02568383514881134,
+            0.017785141244530678,
+            0.04387001320719719,
+            -0.011813169345259666,
+            0.0019997386261820793,
+            0.019057998433709145,
+            -0.05056559666991234,
+            -0.005370415281504393,
+            -0.0031799657735973597,
+            0.007641507778316736,
+            0.012920381501317024,
+            -0.016974348574876785,
+            0.05457597225904465,
+            0.011054685339331627,
+            0.02223142609000206,
+            0.02606743760406971,
+            -0.00020024630066473037,
+            -0.038046251982450485,
+            0.017584621906280518,
+            0.008535124361515045,
+            0.02796800620853901,
+            0.0015681873774155974,
+            0.0073014977388083935,
+            -0.019807765260338783,
+            0.008979752659797668,
+            -0.009485408663749695,
+            0.01646869257092476,
+            0.018866198137402534,
+            0.02645103819668293,
+            -0.008644101209938526,
+            -0.002175192581489682,
+            0.0033434322103857994,
+            -0.0034785643219947815,
+            0.028700334951281548,
+            0.0076371487230062485,
+            -0.038848329335451126,
+            0.006813278421759605,
+            -0.002189359627664089,
+            0.023120682686567307,
+            0.030374230816960335,
+            -0.04108018800616264,
+            -0.02338222973048687,
+            -0.002796364715322852,
+            0.016974348574876785,
+            0.011019812896847725,
+            -0.020662149414420128,
+            -0.0031668883748352528,
+            0.044602345675230026,
+            -0.010479284450411797,
+            -0.032484035938978195,
+            0.01293781865388155,
+            -0.0018504393519833684,
+            0.0059806895442306995,
+            0.0024476363323628902,
+            0.012772172689437866,
+            0.013469628989696503,
+            -0.019598528742790222,
+            0.04087095335125923,
+            -0.004965018481016159,
+            0.013208082877099514,
+            -0.012031124904751778,
+            -0.010052092373371124,
+            -0.015195833519101143,
+            0.03926680237054825,
+            -0.0047906544059515,
+            0.02013905718922615,
+            0.024864325299859047,
+            0.006338136270642281,
+            0.05216974765062332,
+            -0.04125455394387245,
+            -0.03724417835474014,
+            0.005971971433609724,
+            0.0016215863870456815,
+            0.005871712230145931,
+            -0.011272640898823738,
+            -0.030932195484638214,
+            -0.012074715457856655,
+            0.004768858663737774,
+            -0.015535843558609486,
+            -0.015684053301811218,
+            0.00019043832435272634,
+            0.022144244983792305,
+            0.0011257383739575744,
+            0.005475033540278673,
+            0.00287700816988945,
+            0.014585559256374836,
+            0.03581439331173897,
+            -0.02589307352900505,
+            0.010758266784250736,
+            0.03147272393107414,
+            -0.009136680513620377,
+            -0.026764893904328346,
+            -0.027305422350764275,
+            0.015073779039084911,
+            -0.018918508663773537,
+            -0.028874699026346207,
+            3.807404209510423e-05,
+            -0.0034872826654464006,
+            0.008391273207962513,
+            -0.03584926575422287,
+            -0.0076763806864619255,
+            0.0021283321548253298,
+            -0.00196377607062459,
+            0.0007078094058670104,
+            -0.022318609058856964,
+            -0.013591683469712734,
+            0.01741897687315941,
+            0.024219177663326263,
+            0.005727861542254686,
+            0.03028704971075058,
+            -0.010243892669677734,
+            -0.00016768925706855953,
+            0.006525577511638403,
+            0.01589328981935978,
+            0.023050937801599503,
+            0.023312482982873917,
+            0.026573093608021736,
+            0.0009344827267341316,
+            -0.012327543459832668,
+            0.004252305254340172,
+            0.032484035938978195,
+            -0.016058936715126038,
+            -0.0030993223190307617,
+            0.014812232926487923,
+            0.03274558484554291,
+            0.010627493262290955,
+            -0.012188052758574486,
+            -0.02528279833495617,
+            0.015431225299835205,
+            -0.022283734753727913,
+            0.012510626576840878,
+            0.04777577146887779,
+            0.004393976181745529,
+            -0.029240863397717476,
+            0.02430635876953602,
+            -0.009546436369419098,
+            -0.024044813588261604,
+            0.027165930718183517,
+            0.003757546888664365,
+            -0.03271070867776871,
+            -6.98818766977638e-05,
+            -0.00018321855168323964,
+            0.014794795773923397,
+            -0.026015127077698708,
+            -0.015579435043036938,
+            0.02322530187666416,
+            0.009293608367443085,
+            0.0032627887558192015,
+            -0.021603714674711227,
+            0.001829733606427908,
+            0.015344043262302876,
+            0.0018177460879087448,
+            0.009424380958080292,
+            0.009145398624241352,
+            0.01505634281784296,
+            0.006939692422747612,
+            -0.013190646655857563,
+            -0.0032344546634703875,
+            -0.02720080316066742,
+            -0.006098385434597731,
+            0.0652819275856018,
+            -0.007292779628187418,
+            0.03686057776212692,
+            0.0015213270671665668,
+            0.012868072837591171,
+            0.025788454338908195,
+            -0.025160742923617363,
+            -0.02437610551714897,
+            -0.005047841463238001,
+            0.006961487699300051,
+            -0.04292844980955124,
+            -0.013269110582768917,
+            -0.023818138986825943,
+            -0.030217302963137627,
+            0.01933698169887066,
+            0.0005187332862988114,
+            0.0019267236348241568,
+            0.024829450994729996,
+            0.0051742554642260075,
+            -0.015038905665278435,
+            -0.017009221017360687,
+            0.0383601076900959,
+            0.016625620424747467,
+            0.028229551389813423,
+            -0.015989189967513084,
+            0.01625945419073105,
+            -0.008099214173853397,
+            0.003171247662976384,
+            0.03236198052763939,
+            -0.0026982848066836596,
+            -0.0030186788644641638,
+            0.009197707287967205,
+            0.01881388947367668,
+            0.040731459856033325,
+            -0.01745384931564331,
+            -0.030827578157186508,
+            -0.008914366364479065,
+            -0.014236831106245518,
+            0.021969880908727646,
+            -0.0009491947130300105,
+            -0.00041084547410719097,
+            -0.003934090491384268,
+            0.04352128505706787,
+            -0.029572155326604843,
+            0.014428631402552128,
+            -0.024445850402116776,
+            -0.039022691547870636,
+            0.017706677317619324,
+            -0.006102744489908218,
+            0.032693274319171906,
+            0.006883023772388697,
+            -0.007833308540284634,
+            -0.032867636531591415,
+            0.0016749853966757655,
+            0.0086179468780756,
+            -0.03194350749254227,
+            0.020418038591742516,
+            0.02472483366727829,
+            -0.00651250034570694,
+            0.0033063797745853662,
+            -0.016782546415925026,
+            0.008142804726958275,
+            -0.013844511471688747,
+            -0.022946318611502647,
+            0.022737082093954086,
+            -0.004298075567930937,
+            -0.028055187314748764,
+            0.006686864420771599,
+            -0.022458098828792572,
+            0.004707831423729658,
+            -0.0016771649243310094,
+            -0.00017014124023262411,
+            -0.00797280017286539,
+            0.010461847297847271,
+            -0.007571762427687645,
+            -0.01073211245238781,
+            -0.022353481501340866,
+            0.009991064667701721,
+            0.025631526485085487,
+            0.0017588981427252293,
+            -0.004672958515584469,
+            0.006848151329904795,
+            -0.03658159449696541,
+            0.00574093870818615,
+            0.02413199469447136,
+            -0.012205488979816437,
+            -0.008713847026228905,
+            -0.018255924805998802,
+            -0.029380355030298233,
+            -0.002366993110626936,
+            0.01130751334130764,
+            0.03584926575422287,
+            0.025788454338908195,
+            0.030409103259444237,
+            0.03391382470726967,
+            0.010810576379299164,
+            -0.00398204056546092,
+            0.009127962402999401,
+            -0.029031626880168915,
+            0.04006887599825859,
+            0.0040823002345860004,
+            0.02608487382531166,
+            0.016172273084521294,
+            -0.0459623858332634,
+            -0.00968592707067728,
+            0.014681459404528141,
+            0.027898261323571205,
+            0.044148996472358704,
+            0.025038689374923706,
+            -0.0010232995264232159,
+            0.027654150500893593,
+            -0.022196553647518158,
+            -0.02878751792013645,
+            0.01956365443766117,
+            -0.005235282704234123,
+            -0.00918898917734623,
+            -0.0013131798477843404,
+            0.021533969789743423,
+            -0.011385977268218994,
+            0.016058936715126038,
+            0.03415793180465698,
+            -0.008461019024252892,
+            0.012135743163526058,
+            0.0005590550135821104,
+            -0.008286654949188232,
+            0.026764893904328346,
+            -0.007214315701276064,
+            0.03485538810491562,
+            -0.01650356501340866,
+            -0.0018656961619853973,
+            0.000780279457103461,
+            -0.010766984894871712,
+            0.028020314872264862,
+            0.007270984351634979,
+            -0.06172490119934082,
+            0.00873564276844263,
+            0.011063403449952602,
+            0.013835793361067772,
+            -0.001278307056054473,
+            -0.0033957413397729397,
+            -0.025788454338908195,
+            0.019040562212467194,
+            0.008129727095365524,
+            0.02472483366727829,
+            0.010679802857339382,
+            0.019493909552693367,
+            -0.016643056645989418,
+            0.016599465161561966,
+            -0.0026590528432279825,
+            -0.02854340709745884,
+            0.007266624830663204,
+            0.013181927613914013,
+            0.0402083694934845,
+            -0.026538221165537834,
+            -0.002582768676802516,
+            0.010889039374887943,
+            -0.006970205809921026,
+            -0.011194176971912384,
+            -0.030443977564573288,
+            -0.0010756086558103561,
+            -0.028421353548765182,
+            0.02416686899960041,
+            -0.005139382556080818,
+            0.019406726583838463,
+            -0.027357731014490128,
+            0.0063337767496705055,
+            0.00043264098349027336,
+            -0.027654150500893593,
+            0.006390445400029421,
+            0.01782001368701458,
+            0.0031189383007586002,
+            -0.031176306307315826,
+            -0.006006844341754913,
+            0.023242738097906113,
+            0.03619799390435219,
+            0.006713018752634525,
+            -0.006848151329904795,
+            0.021533969789743423,
+            -0.015579435043036938,
+            -0.006150694563984871,
+            0.036546722054481506,
+            -0.025910509750247,
+            0.012231643311679363,
+            0.03759290650486946,
+            -0.03543078899383545,
+            -0.03196094557642937,
+            -0.0004021272761747241,
+            -0.04142891615629196,
+            0.001680434332229197,
+            -0.002975087845697999,
+            0.00835204217582941,
+            -0.0009715351043269038,
+            -0.027113622054457664,
+            0.024812014773488045,
+            -0.0003577189054340124,
+            0.013373728841543198,
+            -0.006848151329904795,
+            0.0021065366454422474,
+            -0.025143306702375412,
+            -0.01862208917737007,
+            -0.010793139226734638,
+            0.00015651904686819762,
+            0.0023059656377881765,
+            -0.008112290874123573,
+            -0.03944116458296776,
+            -0.020976005122065544,
+            0.007257906720042229,
+            0.00045770584256388247,
+            0.03584926575422287,
+            0.018482597544789314,
+            0.012188052758574486,
+            -0.02165602520108223,
+            -0.01898825354874134,
+            0.04756653308868408,
+            -0.026416165754199028,
+            0.0402432419359684,
+            -0.012135743163526058,
+            -0.017488721758127213,
+            0.002946753753349185,
+            0.02625923790037632,
+            -0.020923694595694542,
+            0.016817420721054077,
+            -0.01561430748552084,
+            0.005719143431633711,
+            -0.01409733947366476,
+            -0.004755781497806311,
+            0.009642336517572403,
+            -0.0017796038882806897,
+            0.018255924805998802,
+            0.0009072383400052786,
+            -0.0031123997177928686,
+            0.03183889016509056,
+            0.013652711175382137,
+            0.007763562723994255,
+            -0.016773829236626625,
+            -0.012222925201058388,
+            -0.010766984894871712,
+            0.007087901700288057,
+            -0.015073779039084911,
+            -0.007066106423735619,
+            -0.029659338295459747,
+            0.001694601378403604,
+            0.0007154378108680248,
+            -0.015256861224770546,
+            0.01235369872301817,
+            0.02706131339073181,
+            -0.013295264914631844,
+            -0.03157734498381615,
+            -0.006229158490896225,
+            -0.03944116458296776,
+            -0.021045750007033348,
+            0.019511345773935318,
+            -0.0005190056981518865,
+            0.009127962402999401,
+            0.004250125493854284,
+            0.00043509298120625317,
+            -0.022946318611502647,
+            -0.017976941540837288,
+            0.00714892940595746,
+            -0.009572590701282024,
+            0.00107724335975945,
+            0.00131862866692245,
+            0.01822105050086975,
+            -0.010156710632145405,
+            0.02800287865102291,
+            -0.017497440800070763,
+            0.015265579335391521,
+            -0.029851138591766357,
+            -0.02760184183716774,
+            -0.03199581801891327,
+            0.01304243691265583,
+            -0.014611713588237762,
+            -0.008195114322006702,
+            -0.010113119147717953,
+            0.0004999346565455198,
+            0.007567403372377157,
+            -0.009441817179322243,
+            0.00842614658176899,
+            0.011603932827711105,
+            0.013303983025252819,
+            0.007406116463243961,
+            -0.03244916349649429,
+            -0.02029598504304886,
+            -0.02397506684064865,
+            0.043974634259939194,
+            -0.030513722449541092,
+            0.01703537441790104,
+            -0.017392821609973907,
+            0.005823761690407991,
+            -0.0363723561167717,
+            -0.019075434654951096,
+            0.008953598327934742,
+            0.020749330520629883,
+            -0.025910509750247,
+            0.02741004154086113,
+            0.02934548258781433,
+            0.03013012185692787,
+            -0.036546722054481506,
+            0.025352545082569122,
+            -0.010505438782274723,
+            -0.02512587048113346,
+            0.027357731014490128,
+            0.03731392323970795,
+            0.03679082915186882,
+            -0.013460910879075527,
+            -0.012589089572429657,
+            -0.014690177515149117,
+            0.012318825349211693,
+            -0.0021555765997618437,
+            0.005801966413855553,
+            -0.037069812417030334,
+            0.01862208917737007,
+            0.009895164519548416,
+            0.005121946334838867,
+            0.03773239627480507,
+            -0.01898825354874134,
+            0.02054009400308132,
+            0.007471502758562565,
+            -0.01839541457593441,
+            0.006974565330892801,
+            0.000942111189942807,
+            -0.0004040343628730625,
+            0.003602798795327544,
+            0.017584621906280518,
+            0.005553497467190027,
+            -0.054541099816560745,
+            0.006582245696336031,
+            0.027898261323571205,
+            0.020418038591742516,
+            0.005235282704234123,
+            -0.026398729532957077,
+            0.014629150740802288,
+            0.015823544934391975,
+            0.002135960618034005,
+            0.007035592570900917,
+            0.016538437455892563,
+            -0.011080840602517128,
+            0.00042283302173018456,
+            0.0030339357908815145,
+            0.03720930591225624,
+            0.01475120522081852,
+            -0.013190646655857563,
+            -0.025056125596165657,
+            -0.016364073380827904,
+            0.0066519915126264095,
+            -0.005915303248912096,
+            -0.0004182014672551304,
+            -0.037488289177417755,
+            -0.03598875552415848,
+            0.013521937653422356,
+            -0.01818617805838585,
+            -0.03815086930990219,
+            -0.0042741005308926105,
+            -0.03222249075770378,
+            0.005200410261750221,
+            -0.014890696853399277,
+            -0.019319545477628708,
+            -0.03926680237054825,
+            -0.0038970382884144783,
+            0.00012893410166725516,
+            0.04045247659087181,
+            0.00037842465098947287,
+            0.03476820886135101,
+            -0.007253547664731741,
+            -0.0013676686212420464,
+            -0.004143327474594116,
+            0.022876573726534843,
+            -0.03147272393107414,
+            -0.04529980197548866,
+            0.026573093608021736,
+            0.005832480266690254,
+            0.016024062409996986,
+            0.035744644701480865,
+            -0.013870666734874249,
+            -0.010200301185250282,
+            0.024864325299859047,
+            -0.016747673973441124,
+            -0.010984940454363823,
+            -0.0022231426555663347,
+            0.0025980256032198668,
+            -0.012231643311679363,
+            -0.013521937653422356,
+            -0.011656241491436958,
+            0.05157690867781639,
+            0.005575293209403753,
+            0.027845950797200203,
+            0.04038273170590401,
+            0.0010271136416122317,
+            -0.04059197008609772,
+            0.0032889433205127716,
+            -0.0091018071398139,
+            -0.009642336517572403,
+            -0.004625008441507816,
+            0.033024564385414124,
+            0.001587803359143436,
+            0.017872322350740433,
+            -0.0042174323461949825,
+            -0.016416382044553757,
+            -0.020418038591742516,
+            -0.012318825349211693,
+            -0.00761535344645381,
+            -0.004145507235080004,
+            -0.03373945876955986,
+            -0.01256293524056673,
+            0.031716834753751755,
+            0.025335106998682022,
+            -0.022388353943824768,
+            -0.004925786517560482,
+            -0.01782001368701458,
+            -0.007672021631151438,
+            0.010479284450411797,
+            -0.002959831152111292,
+            -0.00416948227211833,
+            0.009250016883015633,
+            0.018133869394659996,
+            0.01007824670523405,
+            0.00766766257584095,
+            -0.05715656280517578,
+            -0.0019398009171709418,
+            0.011351104825735092,
+            0.010400820523500443,
+            -0.003968963399529457,
+            0.014541968703269958,
+            -0.012580371461808681,
+            -0.01265011727809906,
+            -0.01975545659661293,
+            0.0326235294342041,
+            0.006355572491884232,
+            0.030426539480686188,
+            -0.0005541510181501508,
+            0.009075652807950974,
+            0.02149909734725952,
+            0.004101916216313839,
+            0.012807045131921768,
+            0.00199102028273046,
+            0.019807765260338783,
+            0.00986029114574194,
+            -0.03142041712999344,
+            0.012414725497364998,
+            0.0010429153917357326,
+            -0.0024519956205040216,
+            -0.017872322350740433,
+            0.006922255735844374,
+            0.0023757112212479115,
+            -0.013949130661785603,
+            0.0041215321980416775,
+            -0.022370917722582817,
+            0.005954534746706486,
+            0.031089123338460922,
+            0.033233802765607834,
+            0.009206426329910755,
+            -0.015169679187238216,
+            -0.010810576379299164,
+            0.002057496691122651,
+            0.02008674666285515,
+            0.025404853746294975,
+            0.025003815069794655,
+            -0.0268172025680542,
+            -0.014393758960068226,
+            -0.003729212796315551,
+            -0.001253242138773203,
+            -0.010932630859315395,
+            0.03046141378581524,
+            0.02833417057991028,
+            -0.0008914365898817778,
+            0.0028835467528551817,
+            -0.0051916916854679585,
+            0.01523070689290762,
+            0.01898825354874134,
+            -0.017340512946248055,
+            0.0017665266059339046,
+            0.022283734753727913,
+            -0.020191365852952003,
+            0.022946318611502647,
+            -0.014672741293907166,
+            -0.011429568752646446,
+            0.02013905718922615,
+            0.09366840869188309,
+            -0.005248360335826874,
+            0.01120289508253336,
+            -0.028700334951281548,
+            -0.019214926287531853,
+            -0.008474096655845642,
+            -0.0035330532118678093,
+            0.016765110194683075,
+            0.026503346860408783,
+            0.009511562995612621,
+            0.021359605714678764,
+            0.015466097742319107,
+            -0.01666049286723137,
+            0.014411195181310177,
+            -0.009694645181298256,
+            0.016198428347706795,
+            -0.01362655684351921,
+            -0.004267562180757523,
+            0.010069528594613075,
+            -0.001332795829512179,
+            0.009816700592637062,
+            0.018674397841095924,
+            0.016799984499812126,
+            0.013190646655857563,
+            -0.024864325299859047,
+            0.00949412677437067,
+            0.002284170128405094,
+            -0.007763562723994255,
+            -0.02013905718922615,
+            -0.02322530187666416,
+            0.004999891389161348,
+            0.004428848624229431,
+            0.005296310409903526,
+            0.010331074707210064,
+            -0.0032889433205127716,
+            0.03811599686741829,
+            -0.03319893032312393,
+            0.01496044173836708,
+            0.006917896680533886,
+            -0.013103464618325233,
+            0.011935224756598473,
+            0.016198428347706795,
+            0.004049607086926699,
+            0.012249079532921314,
+            -0.011569059453904629,
+            0.014245549216866493,
+            0.02052265778183937,
+            -0.029781393706798553,
+            -0.00266123260371387,
+            -0.016372792422771454,
+            -0.008474096655845642,
+            0.0026677711866796017,
+            -0.016172273084521294,
+            -0.006006844341754913,
+            -0.00546195637434721,
+            0.01169983297586441,
+            0.0024825092405080795,
+            -0.0325886569917202,
+            0.005060918629169464,
+            0.00261110276915133,
+            0.005928380414843559,
+            0.018168741837143898,
+            0.01632048189640045,
+            0.01765436865389347,
+            -0.011525468900799751,
+            -0.01975545659661293,
+            0.006285826675593853,
+            0.0045029534958302975,
+            -0.01073211245238781,
+            0.0023473771288990974,
+            0.0024912275839596987,
+            0.01588457264006138,
+            -0.011185458861291409,
+            -0.005823761690407991,
+            -0.004141148179769516,
+            0.011664960533380508,
+            0.0031516316812485456,
+            -0.014541968703269958,
+            0.003940629307180643,
+            -0.021150369197130203,
+            0.020644713193178177,
+            0.011577778495848179,
+            -0.0019932000432163477,
+            -0.02088882215321064,
+            -0.02873520739376545,
+            0.01332141924649477,
+            -0.020836513489484787,
+            0.0031952227000147104,
+            0.015771236270666122,
+            -0.011673678644001484,
+            0.013260391540825367,
+            -0.018255924805998802,
+            -0.018866198137402534,
+            -0.0114818774163723,
+            -0.00473834527656436,
+            -0.005226564593613148,
+            -0.007576121482998133,
+            0.007502016611397266,
+            -0.01823848858475685,
+            -0.0306706503033638,
+            -0.033634841442108154,
+            0.019023125991225243,
+            0.01670408435165882,
+            -0.009467972442507744,
+            -0.015823544934391975,
+            -0.004642445128411055,
+            0.02319042943418026,
+            -0.0034415118861943483,
+            0.0007110786973498762,
+            -0.0026830281130969524,
+            0.013391165062785149,
+            -0.007610994391143322,
+            0.00016441992192994803,
+            -0.02284170128405094,
+            0.00044217653339728713,
+            0.0011725988006219268,
+            0.013391165062785149,
+            -0.027357731014490128,
+            -0.003380484413355589,
+            0.0003002332232426852,
+            -0.018098996952176094,
+            -0.004476799163967371,
+            0.04254484549164772,
+            -0.013417319394648075,
+            0.008979752659797668,
+            0.021551406010985374,
+            0.010104401037096977,
+            -0.00977310910820961,
+            0.010095682926476002,
+            0.007850744761526585,
+            -0.0028857262805104256,
+            0.026538221165537834,
+            0.022911446169018745,
+            0.01914518140256405,
+            0.01670408435165882,
+            -0.022161681205034256,
+            0.012528062798082829,
+            0.021969880908727646,
+            0.03752316161990166,
+            -0.017331793904304504,
+            -0.021952442824840546,
+            0.019860073924064636,
+            0.04676445946097374,
+            0.0021032672375440598,
+            -0.0006200824282132089,
+            -0.028648026287555695,
+            -0.017201021313667297,
+            0.024689961224794388,
+            0.020557530224323273,
+            0.016224581748247147,
+            -0.010121837258338928,
+            -0.031507596373558044,
+            0.014280421659350395,
+            -0.00309278373606503,
+            0.006865587551146746,
+            -0.01996469311416149,
+            0.01688716560602188,
+            -0.015004033222794533,
+            0.04257972165942192,
+            -0.012658835388720036,
+            0.004829886369407177,
+            0.017148712649941444,
+            0.011150586418807507,
+            -0.022318609058856964,
+            0.0009263094398193061,
+            -0.01666049286723137,
+            0.027933133766055107,
+            -0.018883634358644485,
+            0.003618055721744895,
+            0.01914518140256405,
+            -0.016608182340860367,
+            -0.014533249661326408,
+            0.004191277548670769,
+            0.019459037110209465,
+            0.006107103545218706,
+            -0.025247925892472267,
+            0.018098996952176094,
+            0.0019190951716154814,
+            0.009171552956104279,
+            -0.009930036962032318,
+            0.023870449513196945,
+            -0.027636714279651642,
+            -0.015387633815407753,
+            0.025003815069794655,
+            0.0041607641614973545,
+            0.013565529137849808,
+            -0.026311546564102173,
+            0.009380790404975414,
+            -0.029049063101410866,
+            0.02336479350924492,
+            0.0032584297005087137,
+            0.014280421659350395,
+            0.0015147883677855134,
+            0.002866110298782587,
+            -0.0015115190763026476,
+            0.014446067623794079,
+            -0.000550881668459624,
+            -0.022527845576405525,
+            -0.01569277234375477,
+            -0.03696519508957863,
+            -0.022737082093954086,
+            0.023818138986825943,
+            -0.002715721260756254,
+            0.003528693923726678,
+            0.02430635876953602,
+            -0.027462350204586983,
+            -0.01435888558626175,
+            0.009572590701282024,
+            0.02549203485250473,
+            -0.043625906109809875,
+            4.093470488442108e-05,
+            0.0013829254312440753,
+            -0.019232362508773804,
+            0.010627493262290955,
+            -0.005440160632133484,
+            0.015500971116125584,
+            0.0018776836805045605,
+            -0.037662651389837265,
+            0.011586496606469154,
+            0.03762777894735336,
+            -0.017924632877111435,
+            0.0028225192800164223,
+            0.017706677317619324,
+            0.012493189424276352,
+            -0.0012390750925987959,
+            -0.0017033196054399014,
+            0.01877901703119278,
+            -0.025945382192730904,
+            0.026102310046553612,
+            0.017026657238602638,
+            -0.014881978742778301,
+            0.007842026650905609,
+            -0.008753078989684582,
+            0.020731894299387932,
+            -0.0013208083109930158,
+            -0.013155773282051086,
+            0.017375385388731956,
+            0.002195898210629821,
+            0.01349578332155943,
+            -0.001912556472234428,
+            0.004304614383727312,
+            0.03107168711721897,
+            -0.01609380915760994,
+            0.02280682697892189,
+            0.01994725689291954,
+            3.6405323044164106e-05,
+            0.015405070967972279,
+            -0.019877510145306587,
+            -0.03312918543815613,
+            0.02221398986876011,
+            -0.013591683469712734,
+            -0.007310215849429369,
+            0.023347357288002968,
+            -0.010453129187226295,
+            -0.006250953767448664,
+            -0.004282819107174873,
+            0.004077941179275513,
+            0.012406007386744022,
+            -0.0230160653591156,
+            -0.015631744638085365,
+            0.022318609058856964,
+            0.017131276428699493,
+            0.022179117426276207,
+            0.023800702765583992,
+            -0.015378915704786777,
+            0.03182145208120346,
+            0.009223862551152706,
+            0.0065735275857150555,
+            0.0076371487230062485,
+            0.008395632728934288,
+            -0.005871712230145931,
+            0.01933698169887066,
+            0.014114775694906712,
+            0.007336370646953583,
+            -0.003565746359527111,
+            -0.02029598504304886,
+            -0.011447004973888397,
+            -0.027532095089554787,
+            -0.00045825072447769344,
+            -0.013914257287979126,
+            0.00704866973683238,
+            0.07797563821077347,
+            -0.006682505365461111,
+            0.0018144766800105572,
+            -0.007868180982768536,
+            0.000739412906114012,
+            0.0016771649243310094,
+            -0.005209128372371197,
+            0.016538437455892563,
+            -0.020662149414420128,
+            -0.020749330520629883,
+            -0.006163771729916334,
+            -0.010967503301799297,
+            0.013460910879075527,
+            0.01741897687315941,
+            0.0030034221708774567,
+            -0.02277195453643799,
+            0.011141867376863956,
+            -0.016355354338884354,
+            -0.011106994934380054,
+            0.010348510928452015,
+            -0.007231752388179302,
+            -0.011656241491436958,
+            0.03295481950044632,
+            -0.009206426329910755,
+            -0.03147272393107414,
+            0.015640461817383766,
+            -0.0024541751481592655,
+            -0.005976330488920212,
+            0.00470347236841917,
+            0.00853076484054327,
+            -0.004986814223229885,
+            0.0005544234300032258,
+            0.013504501432180405,
+            -0.002068394562229514,
+            0.005414006300270557,
+            0.005318105686455965,
+            -0.013565529137849808,
+            0.017296921461820602,
+            0.022527845576405525,
+            -0.033477913588285446,
+            -0.004755781497806311,
+            -0.013940411619842052,
+            0.010479284450411797,
+            0.018709270283579826,
+            0.015535843558609486,
+            -0.022580154240131378,
+            0.0025871277321130037,
+            -0.009154116734862328,
+            -0.012702426873147488,
+            -0.02741004154086113,
+            -0.018255924805998802,
+            0.00723611144348979,
+            -0.00851768720895052,
+            0.018656961619853973,
+            0.007907412946224213,
+            -0.005745298229157925,
+            0.005012968555092812,
+            -0.005178614519536495,
+            0.015518407337367535,
+            -0.0326235294342041,
+            0.010461847297847271,
+            -0.009337198920547962,
+            0.009581308811903,
+            0.01055774837732315,
+            -0.022388353943824768,
+            -0.014759923331439495,
+            -0.023417102172970772,
+            0.016555873677134514,
+            -0.010086964815855026,
+            -0.0020934592466801405,
+            0.02008674666285515,
+            0.0009088730439543724,
+            0.015413789078593254,
+            -0.001491903094574809,
+            0.023242738097906113,
+            0.014140930958092213,
+            0.016590746119618416,
+            -0.01215317938476801,
+            -0.001640112604945898,
+            -0.011124431155622005,
+            0.012719863094389439,
+            -0.02303350158035755,
+            -0.006813278421759605,
+            -0.026015127077698708,
+            0.03598875552415848,
+            -0.007602275814861059,
+            -0.01588457264006138,
+            0.02857828140258789,
+            -0.012414725497364998,
+            -0.0037052377592772245,
+            0.0058847893960773945,
+            -0.015431225299835205,
+            0.023451974615454674,
+            0.0004955755430273712,
+            0.011499314568936825,
+            -0.02549203485250473,
+            0.00863102450966835,
+            -0.007087901700288057,
+            0.006076589692384005,
+            -0.02128986082971096,
+            -0.002423661295324564,
+            0.012135743163526058,
+            -0.010426974855363369,
+            0.00411717314273119,
+            0.021185241639614105,
+            0.005331183318048716,
+            -0.01744513027369976,
+            0.03951091319322586,
+            -0.004025631584227085,
+            0.010653648525476456,
+            0.005475033540278673,
+            -0.0009028792264871299,
+            0.016799984499812126,
+            0.01742769405245781,
+            -0.004925786517560482,
+            -0.016547156497836113,
+            0.0032235567923635244,
+            -0.015021469444036484,
+            0.01561430748552084,
+            0.016233300790190697,
+            -0.02125498652458191,
+            -0.032658401876688004,
+            0.01779385842382908,
+            0.002061855746433139,
+            0.028072625398635864,
+            0.006983283441513777,
+            -0.025997690856456757,
+            -0.005296310409903526,
+            -0.03815086930990219,
+            0.012885509058833122,
+            -0.01801181398332119,
+            -0.02437610551714897,
+            0.002785466844215989,
+            -0.00986029114574194,
+            -0.01856978051364422,
+            0.02512587048113346,
+            0.013652711175382137,
+            -0.02552690915763378,
+            0.010889039374887943,
+            0.0013011923292651772,
+            -0.017549749463796616,
+            0.010052092373371124,
+            0.004350385162979364,
+            -0.012493189424276352,
+            0.007410475518554449,
+            0.005130664445459843,
+            -0.006978924386203289,
+            -0.02890957146883011,
+            -0.01670408435165882,
+            0.01972058229148388,
+            0.010993658564984798,
+            -0.0058455574326217175,
+            0.011290077120065689,
+            -0.00997362844645977,
+            0.0033456117380410433,
+            0.0035221553407609463,
+            0.00901462510228157,
+            -0.013051155023276806,
+            -0.007371243555098772,
+            -0.007162006571888924,
+            0.01973801851272583,
+            -0.016024062409996986,
+            0.007990236394107342,
+            -0.013443474657833576,
+            -0.007981518283486366,
+            0.011429568752646446,
+            -0.0013774766121059656,
+            0.014986597001552582,
+            -0.018744144588708878,
+            0.0073407297022640705,
+            0.01189163327217102,
+            -0.008883852511644363,
+            -0.010671084746718407,
+            -0.02437610551714897,
+            0.019110308960080147,
+            0.01438504084944725,
+            0.004694754257798195,
+            -0.03679082915186882,
+            0.003951527178287506,
+            -0.010453129187226295,
+            -0.0009775288635864854,
+            -0.008796670474112034,
+            0.019249800592660904,
+            0.018831325694918633,
+            0.007985876873135567,
+            0.013277828693389893,
+            -0.0023081451654434204,
+            0.013548092916607857,
+            -0.012414725497364998,
+            0.017593340948224068,
+            -0.004036529455333948,
+            -0.017575904726982117,
+            -0.005714784376323223,
+            0.019406726583838463,
+            -0.001491903094574809,
+            0.007903054356575012,
+            0.004372180439531803,
+            0.010226456448435783,
+            -0.012275234796106815,
+            -0.007493298500776291,
+            0.014045030809938908,
+            0.008378196507692337,
+            -0.0004288267809897661,
+            0.009250016883015633,
+            0.06758353859186172,
+            -0.00870512891560793,
+            0.023853013291954994,
+            0.006311981473118067,
+            -0.009345917031168938,
+            -0.008112290874123573,
+            0.0019932000432163477,
+            -0.02951984666287899,
+            0.023329921066761017,
+            0.011970097199082375,
+            -0.009180271066725254,
+            0.00036126066697761416,
+            -0.006726096384227276,
+            0.018325669690966606,
+            -0.009694645181298256,
+            0.016948193311691284,
+            -0.010052092373371124,
+            0.02435866929590702,
+            0.019162617623806,
+            0.005640679504722357,
+            0.005536061245948076,
+            0.0028203397523611784,
+            0.019459037110209465,
+            0.012868072837591171,
+            -0.002165384590625763,
+            -0.003123297356069088,
+            -0.009877728298306465,
+            0.019389290362596512,
+            -0.001910376944579184,
+            0.030792705714702606,
+            -0.0022144243121147156,
+            -0.005182973574846983,
+            -0.009842854924499989,
+            0.0041607641614973545,
+            0.011569059453904629,
+            0.009520281106233597,
+            0.012876790948212147,
+            -0.0013567708665505052,
+            -0.010679802857339382,
+            0.011909069493412971,
+            0.00863102450966835,
+            0.005326824262738228,
+            -0.0032518908847123384,
+            0.0045247492380440235,
+            -0.008404350839555264,
+            0.014001439325511456,
+            -0.0024781501851975918,
+            0.0015354941133409739,
+            -0.008774874731898308,
+            0.004426669329404831,
+            -0.01312961895018816,
+            0.0005348074482753873,
+            0.009607463143765926,
+            0.012868072837591171,
+            0.0015213270671665668,
+            -0.002273272257298231,
+            0.015405070967972279,
+            0.0025980256032198668,
+            0.0073014977388083935,
+            0.0009486498311161995,
+            -0.009537717327475548,
+            0.019057998433709145,
+            -0.01033979281783104,
+            -0.0019212746992707253,
+            0.010705957189202309,
+            0.006067871581763029,
+            0.008260500617325306,
+            0.01496916078031063,
+            0.009354635141789913,
+            0.003968963399529457,
+            0.0013545913388952613,
+            -0.00565375667065382,
+            -0.003535232739523053,
+            0.0036267738323658705,
+            0.0016771649243310094,
+            0.01646869257092476,
+            -0.019459037110209465,
+            -0.007039951626211405,
+            0.010679802857339382,
+            0.011438286863267422,
+            0.0017839629435911775,
+            0.015684053301811218,
+            0.014428631402552128,
+            0.01994725689291954,
+            -0.022562718018889427,
+            0.003831651760265231,
+            -0.002920599188655615,
+            0.019057998433709145,
+            0.008770515210926533,
+            -0.024114558473229408,
+            0.0003645300166681409,
+            -0.01063621137291193,
+            0.012676272541284561,
+            -0.0067086596973240376,
+            0.0023473771288990974,
+            0.01665177382528782,
+            -0.010121837258338928,
+            -0.009930036962032318,
+            -0.019424164667725563,
+            -0.016983065754175186,
+            0.010976221412420273,
+            0.014437349513173103,
+            0.0014188881032168865,
+            0.019232362508773804,
+            -0.0012412546202540398,
+            -0.008474096655845642,
+            0.030827578157186508,
+            0.016765110194683075,
+            -0.0004157494695391506,
+            -0.008962316438555717,
+            -0.0006544103380292654,
+            -0.015082497149705887,
+            0.01074954867362976,
+            0.004426669329404831,
+            0.005108869168907404,
+            0.003387023229151964,
+            9.174550359603018e-05,
+            -0.016712801530957222,
+            -0.014620431698858738,
+            -0.004668599460273981,
+            0.011045967228710651,
+            -0.003751008305698633,
+            0.009197707287967205,
+            -0.01609380915760994,
+            0.003121117828413844,
+            -0.008173318579792976,
+            -0.010714675299823284,
+            -0.008356400765478611,
+            -0.039406292140483856,
+            -0.02067958563566208,
+            0.0003173972072545439,
+            -0.00473834527656436,
+            -0.002358274767175317,
+            0.009223862551152706,
+            -0.003672544378787279,
+            0.013670147396624088,
+            -0.0015551100950688124,
+            0.0047906544059515,
+            0.008657178841531277,
+            0.01561430748552084,
+            -0.013504501432180405,
+            -0.0006631285650655627,
+            -0.014071185141801834,
+            -0.026782330125570297,
+            0.01707896590232849,
+            0.003199581755325198,
+            0.014446067623794079,
+            -0.015187115408480167,
+            0.009528999216854572,
+            -0.012589089572429657,
+            0.014463504776358604,
+            -0.0421612448990345,
+            -0.009520281106233597,
+            0.007031233515590429,
+            -0.0075979167595505714,
+            -0.005496829282492399,
+            -0.025369981303811073,
+            -0.0012205488746985793,
+            -0.011054685339331627,
+            -0.01666049286723137,
+            -0.019459037110209465,
+            0.00574093870818615,
+            -0.009311044588685036,
+            0.008591792546212673,
+            -0.006120180711150169,
+            0.02761927805840969,
+            0.00532246520742774,
+            -0.001346962875686586,
+            -0.006128899287432432,
+            0.009206426329910755,
+            -0.0007503106608055532,
+            0.012885509058833122,
+            0.008648460730910301,
+            0.005823761690407991,
+            0.023434538394212723,
+            0.0009339378448203206,
+            0.004690395202487707,
+            -0.006141976453363895,
+            0.014541968703269958,
+            0.024689961224794388,
+            0.011342386715114117,
+            0.011455723084509373,
+            0.005967612378299236,
+            0.0005950175691395998,
+            0.011429568752646446,
+            0.017392821609973907,
+            -0.003990759141743183,
+            -0.009171552956104279,
+            0.004869118332862854,
+            -0.01761949434876442,
+            -0.0033347138669341803,
+            0.012885509058833122,
+            -0.006128899287432432,
+            -0.02706131339073181,
+            0.01285063661634922,
+            0.004317691549658775,
+            -0.006673786789178848,
+            -0.016948193311691284,
+            -0.00661711860448122,
+            0.004646804183721542,
+            0.004729627165943384,
+            0.024236613884568214,
+            -0.022144244983792305,
+            -0.008644101209938526,
+            0.011917788535356522,
+            -0.014446067623794079,
+            -0.01063621137291193,
+            0.009790545329451561,
+            0.029484974220395088,
+            0.003496000776067376,
+            -0.05307644233107567,
+            -0.012467035092413425,
+            0.014236831106245518,
+            -0.01503018755465746,
+            -0.025980254635214806,
+            -0.0014428631402552128,
+            0.0038578063249588013,
+            -0.014010157436132431,
+            0.006625836715102196,
+            0.0011606112821027637,
+            -0.011708551086485386,
+            0.017139993607997894,
+            0.028874699026346207,
+            0.01312961895018816,
+            0.0114818774163723,
+            -0.025213053449988365,
+            -0.0008952508214861155,
+            -0.01914518140256405,
+            0.011752142570912838,
+            -0.030566031113266945,
+            -0.013984003104269505,
+            -0.0023342997301369905,
+            0.004141148179769516,
+            0.027444913983345032,
+            0.014376322738826275,
+            0.025962818413972855,
+            0.006778405513614416,
+            0.0037989583797752857,
+            -0.02125498652458191,
+            -0.018447725102305412,
+            0.0033456117380410433,
+            -0.0008304091752506793,
+            -0.002329940674826503,
+            -0.008216909132897854,
+            0.012275234796106815,
+            0.012039843015372753,
+            0.026172054931521416,
+            0.005087073426693678,
+            0.005553497467190027,
+            -2.538836997700855e-05,
+            -0.00494322320446372,
+            0.007593557704240084,
+            -0.019232362508773804,
+            -0.0029925242997705936,
+            -0.004559622146189213,
+            0.015405070967972279,
+            0.0045029534958302975,
+            0.0022275017108768225,
+            0.017357949167490005,
+            -0.0030099607538431883,
+            0.003729212796315551,
+            -0.019982129335403442,
+            -0.014672741293907166,
+            -0.0032518908847123384,
+            0.03525642678141594,
+            0.003971143160015345,
+            -0.02761927805840969,
+            -0.016224581748247147,
+            -0.0153353251516819,
+            0.030792705714702606,
+            -0.009581308811903,
+            0.004315512254834175,
+            0.02683463878929615,
+            0.01593688130378723,
+            0.012388571165502071,
+            0.006399163510650396,
+            -0.01822105050086975,
+            0.005021686665713787,
+            0.02931061014533043,
+            -0.0030317562632262707,
+            -0.010086964815855026,
+            -0.0066912234760820866,
+            0.007144570350646973,
+            -0.007798435632139444,
+            0.007632789667695761,
+            -0.0035068984143435955,
+            -0.013417319394648075,
+            0.029554719105362892,
+            -0.00968592707067728,
+            0.022370917722582817,
+            -0.010130555368959904,
+            0.00268738716840744,
+            -0.038848329335451126,
+            -0.002004097681492567,
+            0.0011453543556854129,
+            0.0019692247733473778,
+            0.004555262625217438,
+            -0.01881388947367668,
+            -0.008753078989684582,
+            0.011839324608445168,
+            -0.01276345457881689,
+            -0.007885618135333061,
+            -0.006137617398053408,
+            0.002983806189149618,
+            0.01977289281785488,
+            -0.00947669055312872,
+            0.0009857022669166327,
+            -0.0210980586707592,
+            0.01256293524056673,
+            -0.00282687833532691,
+            0.0017567186150699854,
+            0.010453129187226295,
+            0.007009437773376703,
+            0.03502975404262543,
+            0.01094134896993637,
+            0.005531701724976301,
+            -0.0032780456822365522,
+            -0.014053748920559883,
+            -0.03218761831521988,
+            0.013295264914631844,
+            -0.010453129187226295,
+            0.023800702765583992,
+            0.0036703648511320353,
+            -0.008103572763502598,
+            -0.0051960512064397335,
+            0.0037139558698982,
+            0.009590026922523975,
+            -0.0041106343269348145,
+            0.013478347100317478,
+            -0.010243892669677734,
+            0.021167805418372154,
+            0.0031298361718654633,
+            -0.005121946334838867,
+            0.012571653351187706,
+            -0.017689241096377373,
+            0.01055774837732315,
+            -0.01744513027369976,
+            0.0028225192800164223,
+            0.01360040158033371,
+            -0.010296201333403587,
+            0.0056276023387908936,
+            -0.052588220685720444,
+            -0.001197663601487875,
+            -0.016521001234650612,
+            -0.00675660977140069,
+            -0.008456660434603691,
+            -0.021394478157162666,
+            -0.03319893032312393,
+            -0.0036660057958215475,
+            0.015370197594165802,
+            0.006551732309162617,
+            0.023260174319148064,
+            0.0009895163821056485,
+            0.012257798574864864,
+            0.012807045131921768,
+            -0.0047906544059515,
+            0.012519344687461853,
+            0.033861514180898666,
+            -0.027741333469748497,
+            -0.019685709848999977,
+            0.005257078446447849,
+            -0.00028552126605063677,
+            -0.015788672491908073,
+            -0.013103464618325233,
+            -0.0017665266059339046,
+            0.0024955866392701864,
+            0.030443977564573288,
+            -0.020191365852952003,
+            -0.005579652264714241,
+            -0.007053028792142868,
+            0.004433208145201206,
+            -0.01918005384504795,
+            -0.0031450928654521704,
+            -0.011708551086485386,
+            -0.012780890800058842,
+            0.016041500493884087,
+            -0.011089558713138103,
+            0.011159304529428482,
+            -0.019511345773935318,
+            0.008626664988696575,
+            -0.015378915704786777,
+            -0.010627493262290955,
+            -0.0031690681353211403,
+            -0.010461847297847271,
+            0.030164994299411774,
+            -0.0041890982538461685,
+            0.0048386044800281525,
+            0.01670408435165882,
+            -0.00853076484054327,
+            0.011603932827711105,
+            -0.0006124540232121944,
+            0.007898694835603237,
+            -0.002704823622480035,
+            -0.0086135882884264,
+            0.0015278656501322985,
+            0.009537717327475548,
+            -0.010950067080557346,
+            0.03846472501754761,
+            -0.013731175102293491,
+            -0.008879492990672588,
+            -0.003859985852614045,
+            -0.004511671606451273,
+            0.002425840822979808,
+            0.0021762822289019823,
+            0.010984940454363823,
+            -0.0012870251666754484,
+            0.006835073698312044,
+            -0.024219177663326263,
+            -0.011717269197106361,
+            0.0013164491392672062,
+            0.006468909326940775,
+            0.003321636701002717,
+            0.008016390725970268,
+            -0.025579217821359634,
+            -0.0020782025530934334,
+            -0.012859354726970196,
+            0.005475033540278673,
+            -0.011499314568936825,
+            0.007332011591643095,
+            -0.01215317938476801,
+            -0.0013295264216139913,
+            0.009337198920547962,
+            0.012946536764502525,
+            0.013861947692930698,
+            -0.006608400493860245,
+            0.026573093608021736,
+            0.014899414964020252,
+            0.008709488436579704,
+            0.005405288189649582,
+            -0.033216364681720734,
+            -0.007802794687449932,
+            0.010365947149693966,
+            0.007519452832639217,
+            0.009232580661773682,
+            -0.017689241096377373,
+            -0.004176021087914705,
+            0.0029402151703834534,
+            -0.007039951626211405,
+            4.651980270864442e-05,
+            0.011080840602517128,
+            -0.010627493262290955,
+            0.005505547393113375,
+            -0.00031521765049546957,
+            0.0027549532242119312,
+            -0.006460190750658512,
+            0.012188052758574486,
+            -0.013286546804010868,
+            -0.008168959058821201,
+            0.0211154967546463,
+            0.02760184183716774,
+            -0.005531701724976301,
+            0.023696085438132286,
+            -0.021917570382356644,
+            0.008487174287438393,
+            -0.005043482407927513,
+            -0.014149649068713188,
+            -0.015300451777875423,
+            0.006883023772388697,
+            0.014777359552681446,
+            0.004760140553116798,
+            -0.013155773282051086,
+            -0.003027397207915783,
+            0.016983065754175186,
+            -0.0010952246375381947,
+            0.006216081324964762,
+            0.01593688130378723,
+            -0.00026971951592713594,
+            0.00017667989595793188,
+            -0.006002485286444426,
+            -0.01876158080995083,
+            -0.006290185730904341,
+            0.017924632877111435,
+            -0.0073799616657197475,
+            0.014262985438108444,
+            0.02320786565542221,
+            0.0005429807933978736,
+            0.011351104825735092,
+            -0.0038469084538519382,
+            0.01351321954280138,
+            0.01505634281784296,
+            -0.005235282704234123,
+            -0.0030600903555750847,
+            -0.0036965194158256054,
+            0.00011858122888952494,
+            0.012022406794130802,
+            -0.01741897687315941,
+            -0.011333668604493141,
+            0.0114818774163723,
+            -0.002121793571859598,
+            -0.008404350839555264,
+            -0.023486848920583725,
+            0.0016313943779096007,
+            0.0077156126499176025,
+            0.011447004973888397,
+            -0.016425101086497307,
+            -0.0005239096935838461,
+            0.000875089957844466,
+            0.010801857337355614,
+            -0.007497657556086779,
+            0.01159521471709013,
+            -0.007907412946224213,
+            -0.00863102450966835,
+            0.006381727289408445,
+            0.0015507509233430028,
+            0.007414834573864937,
+            -0.009999782778322697,
+            0.0015987011138349771,
+            -0.008025108836591244,
+            0.026590529829263687,
+            0.008317168802022934,
+            -0.0013011923292651772,
+            -0.005640679504722357,
+            0.004995532333850861,
+            0.008421787060797215,
+            -0.0100608104839921,
+            0.012815763242542744,
+            0.006107103545218706,
+            -0.002933676354587078,
+            0.010592620819807053,
+            0.015283015556633472,
+            0.0025631526950746775,
+            0.009328480809926987,
+            0.003443691413849592,
+            0.0059022256173193455,
+            -0.007022515404969454,
+            0.0017861425876617432,
+            -0.01550968922674656,
+            -0.020801641047000885,
+            -0.0035831828135997057,
+            0.016991784796118736,
+            -0.010583902709186077,
+            -0.001182406791485846,
+            0.017924632877111435,
+            0.01371373888105154,
+            -0.005034764297306538,
+            0.00551426550373435,
+            0.002217693720012903,
+            -0.0015529305674135685,
+            0.003323816228657961,
+            -0.014280421659350395,
+            -0.012493189424276352,
+            0.008726924657821655,
+            -0.02087138593196869,
+            0.01952878199517727,
+            -0.02165602520108223,
+            -0.035361044108867645,
+            -0.008831542916595936,
+            -0.009197707287967205,
+            -0.01707896590232849,
+            -0.013574247248470783,
+            0.023661212995648384,
+            0.016207145527005196,
+            -0.009354635141789913,
+            -0.034384604543447495,
+            0.03155990689992905,
+            0.00918898917734623,
+            0.027933133766055107,
+            -0.015544561669230461,
+            0.004343846347182989,
+            -0.0015213270671665668,
+            0.014707613736391068,
+            -0.006163771729916334,
+            -0.006686864420771599,
+            0.0091018071398139,
+            0.014620431698858738,
+            0.010897758416831493,
+            0.003245352301746607,
+            -0.003160349791869521,
+            0.007279702462255955,
+            0.002530459314584732,
+            0.0022285915911197662,
+            -0.008583074435591698,
+            -0.02282426506280899,
+            -0.009781827218830585,
+            0.017942069098353386,
+            0.022266298532485962,
+            0.012746017426252365,
+            -0.017131276428699493,
+            0.0020520477555692196,
+            -0.00484296353533864,
+            -0.014053748920559883,
+            -0.009991064667701721,
+            0.009642336517572403,
+            -0.033617403358221054,
+            0.02317299321293831,
+            0.004760140553116798,
+            -0.00627274950966239,
+            -0.0015006213216111064,
+            0.003836010815575719,
+            0.005679911468178034,
+            -0.004313332494348288,
+            0.0002961465797852725,
+            -0.002532638842239976,
+            0.010897758416831493,
+            0.016599465161561966,
+            0.005100150592625141,
+            -0.010400820523500443,
+            0.012388571165502071,
+            -0.0031407338101416826,
+            -0.002761491807177663,
+            -0.013914257287979126,
+            -0.006011203397065401,
+            -0.009502844884991646,
+            0.010261328890919685,
+            0.01483838725835085,
+            0.0054924702271819115,
+            0.006791482679545879,
+            0.004335128236562014,
+            -0.00013084120291750878,
+            -0.0002569146454334259,
+            0.008469737134873867,
+            -0.0005070181796327233,
+            -0.0045639812014997005,
+            0.0010821473551914096,
+            -0.0014679280575364828,
+            0.02320786565542221,
+            -0.00532246520742774,
+            -0.010950067080557346,
+            0.007196879480034113,
+            0.02033085748553276,
+            -0.022370917722582817,
+            0.022091934457421303,
+            -0.022179117426276207,
+            0.023853013291954994,
+            0.03482051566243172,
+            0.013975284993648529,
+            -0.011717269197106361,
+            -0.007096619810909033,
+            0.012815763242542744,
+            -0.012510626576840878,
+            -0.0009600924677215517,
+            0.0018079380970448256,
+            -0.022580154240131378,
+            0.015178397297859192,
+            -0.023347357288002968,
+            0.014280421659350395,
+            0.005121946334838867,
+            -0.00852640625089407,
+            -0.0073232934810221195,
+            -0.0009764390997588634,
+            0.018133869394659996,
+            -0.003990759141743183,
+            -0.0006255313055589795,
+            -0.03028704971075058,
+            -0.006930973846465349,
+            0.02163858897984028,
+            -0.0045247492380440235,
+            -0.013949130661785603,
+            0.007253547664731741,
+            -8.973622243502177e-06,
+            -0.016154836863279343,
+            -0.005876071285456419,
+            -0.03166452422738075,
+            -0.008142804726958275,
+            -0.02264990098774433,
+            -0.0075979167595505714,
+            -0.011577778495848179,
+            -0.012292671017348766,
+            -0.0038512677419930696,
+            -0.016276890411973,
+            0.001949608908034861,
+            -0.001063076313585043,
+            0.0033020207192748785,
+            0.0027091826777905226,
+            -0.009563872590661049,
+            -0.003968963399529457,
+            0.021324733272194862,
+            0.010705957189202309,
+            -0.016041500493884087,
+            -0.026206929236650467,
+            0.011638805270195007,
+            -0.012205488979816437,
+            -0.007593557704240084,
+            0.001547481631860137,
+            -0.005915303248912096,
+            -0.019267236813902855,
+            -0.01860465295612812,
+            0.005967612378299236,
+            0.0003718859807122499,
+            0.0051175872795283794,
+            0.013582965359091759,
+            0.014341449365019798,
+            -0.0019180054077878594,
+            -0.006734814494848251,
+            -1.330718441749923e-05,
+            -0.002195898210629821,
+            0.014027593657374382,
+            0.004348205402493477,
+            0.024445850402116776,
+            -0.008744360879063606,
+            0.0028944446239620447,
+            -0.017924632877111435,
+            0.00449859444051981,
+            -0.007393038831651211,
+            -0.005679911468178034,
+            -0.004064863547682762,
+            -0.011089558713138103,
+            0.005993766710162163,
+            -0.0011126610916107893,
+            -0.00247161160223186,
+            -0.007911772467195988,
+            -0.0020912797190248966,
+            -0.01707024872303009,
+            -0.0029794471338391304,
+            -0.020557530224323273,
+            -0.0013905538944527507,
+            -0.014916851185262203,
+            0.017000501975417137,
+            0.026172054931521416,
+            -0.00513502350077033,
+            -0.0006091846735216677,
+            -0.011229049414396286,
+            -4.08665910072159e-05,
+            -0.011909069493412971,
+            -0.00637736776843667,
+            -0.016416382044553757,
+            -0.0325189083814621,
+            -0.016904601827263832,
+            0.009049498476088047,
+            0.024288922548294067,
+            0.0019398009171709418,
+            -0.024986378848552704,
+            0.0029511128086596727,
+            -0.026607966050505638,
+            -0.014228112995624542,
+            -0.01496916078031063,
+            -0.006529936566948891,
+            -0.009136680513620377,
+            0.017392821609973907,
+            -0.0026350778061896563,
+            0.01571020856499672,
+            0.01747128553688526,
+            0.010906476527452469,
+            0.007497657556086779,
+            0.010880321264266968,
+            -0.009528999216854572,
+            -0.00016932391736190766,
+            0.009869009256362915,
+            -0.006311981473118067,
+            0.011839324608445168,
+            -0.00815152283757925,
+            -0.0011933045461773872,
+            0.01879645325243473,
+            0.008016390725970268,
+            -0.006255313288420439,
+            -0.0026699507143348455,
+            -0.02742747776210308,
+            0.017506157979369164,
+            -0.018552342429757118,
+            -0.00258494820445776,
+            -0.002944574225693941,
+            -0.016738956794142723,
+            0.02319042943418026,
+            0.017323076725006104,
+            -0.005021686665713787,
+            -0.0027549532242119312,
+            0.017889760434627533,
+            -0.006451472640037537,
+            0.00398204056546092,
+            0.0025762300938367844,
+            -0.0033281752839684486,
+            0.009790545329451561,
+            0.0027636713348329067,
+            -0.004577058367431164,
+            -0.0025587936397641897,
+            0.012196770869195461,
+            0.003901397343724966,
+            0.013530656695365906,
+            -0.027131058275699615,
+            -0.010566466487944126,
+            -0.0010303830495104194,
+            0.01935441792011261,
+            -0.005405288189649582,
+            -0.0007906323298811913,
+            -0.013312701135873795,
+            -0.0008854428306221962,
+            -0.005719143431633711,
+            0.023818138986825943,
+            -0.01992982067167759,
+            0.006120180711150169,
+            -0.008395632728934288,
+            0.013861947692930698,
+            -0.0032191977370530367,
+            0.010801857337355614,
+            -0.0007955363253131509,
+            -1.3375295566220302e-05,
+            0.009964910335838795,
+            0.012981409206986427,
+            -0.0022024367935955524,
+            0.024027377367019653,
+            -0.0004737800336442888,
+            -0.006403522565960884,
+            0.0027745692059397697,
+            -0.01895338110625744,
+            0.0229637548327446,
+            0.003402280155569315,
+            0.004315512254834175,
+            -0.0058629936538636684,
+            -0.008199472911655903,
+            0.039789892733097076,
+            0.021516533568501472,
+            0.010392102412879467,
+            -0.000592293159570545,
+            -0.01494300551712513,
+            0.028229551389813423,
+            0.007519452832639217,
+            0.00930232647806406,
+            -0.004012554418295622,
+            0.0027113622054457664,
+            -0.015178397297859192,
+            -0.010976221412420273,
+            0.019807765260338783,
+            -0.019999565556645393,
+            -0.008731283247470856,
+            -0.0026328982785344124,
+            -0.019703146070241928,
+            -0.029293173924088478,
+            0.015474816784262657,
+            0.009764390997588634,
+            0.0028443147893995047,
+            0.008495892398059368,
+            0.0037335718516260386,
+            -0.0015028008492663503,
+            -0.011647523380815983,
+            -0.003986400086432695,
+            -0.008574356324970722,
+            -0.01044441107660532,
+            0.006848151329904795,
+            -0.012004970572888851,
+            0.006717377807945013,
+            -0.008256141096353531,
+            -0.00024533577379770577,
+            0.01285063661634922,
+            0.005091432482004166,
+            0.0062771085649728775,
+            0.003443691413849592,
+            0.005078355316072702,
+            0.01591944508254528,
+            -0.013757329434156418,
+            0.016346637159585953,
+            0.017113838344812393,
+            0.0062945447862148285,
+            -0.0034131777938455343,
+            0.0023103246930986643,
+            -0.029188554733991623,
+            -0.00022381270537152886,
+            -0.0016869729151949286,
+            0.012632681056857109,
+            -0.02163858897984028,
+            0.004733986221253872,
+            0.021568842232227325,
+            -0.0016172273317351937,
+            0.009816700592637062,
+            0.014864541590213776,
+            -0.0001531135058030486,
+            0.028037751093506813,
+            0.011342386715114117,
+            0.02416686899960041,
+            -0.009459254331886768,
+            -0.02852597087621689,
+            0.006368649657815695,
+            -0.0038904997054487467,
+            0.014376322738826275,
+            0.0091018071398139,
+            0.0201216209679842,
+            -0.01369630265980959,
+            -0.006695582531392574,
+            0.007789717521518469,
+            0.0018918508430942893,
+            0.014306576922535896,
+            0.003217018209397793,
+            0.019302109256386757,
+            0.009119244292378426,
+            -0.0008625575574114919,
+            -0.007737408392131329,
+            8.588795753894374e-05,
+            0.006250953767448664,
+            -0.003500359831377864,
+            -0.03047885000705719,
+            0.0016390228411182761,
+            -0.004834245424717665,
+            0.004978095646947622,
+            -0.014245549216866493,
+            0.010409538634121418,
+            -0.0010015039006248116,
+            0.022196553647518158,
+            -0.0041498662903904915,
+            0.004064863547682762,
+            0.01708768494427204,
+            0.00938950851559639,
+            0.004991173278540373,
+            0.015300451777875423,
+            0.005658115725964308,
+            0.004511671606451273,
+            -0.008038186468183994,
+            -0.019493909552693367,
+            0.019162617623806,
+            -0.027462350204586983,
+            0.010653648525476456,
+            -0.00873564276844263,
+            0.004317691549658775,
+            0.003925372380763292,
+            -0.006887382827699184,
+            -0.002567511750385165,
+            0.003607157850638032,
+            0.001491903094574809,
+            -0.059144310653209686,
+            0.0037706242874264717,
+            0.0007388679659925401,
+            0.020452912896871567,
+            -0.00881846621632576,
+            0.0007590288296341896,
+            0.0026394370943307877,
+            0.00390357687138021,
+            -0.013347573578357697,
+            -0.0023604542948305607,
+            0.01571892574429512,
+            -0.01858721673488617,
+            -0.0059632533229887486,
+            0.004038709215819836,
+            -0.005557856522500515,
+            0.0035940806847065687,
+            -0.008779234252870083,
+            -0.002622000640258193,
+            0.02838647924363613,
+            -0.001655369414947927,
+            0.03140297904610634,
+            0.002506484277546406,
+            -0.0004756871494464576,
+            -0.01666049286723137,
+            0.007410475518554449,
+            -0.016738956794142723,
+            -0.00425884360447526,
+            0.008927443064749241,
+            -0.018726706504821777,
+            -0.00123580580111593,
+            0.001693511614575982,
+            -0.0013382446486502886,
+            -0.0077722808346152306,
+            -0.001763257198035717,
+            0.0010429153917357326,
+            -0.02048778533935547,
+            0.015997909009456635,
+            -0.023853013291954994,
+            0.01625945419073105,
+            -0.0034371528308838606,
+            -0.002946753753349185,
+            -0.0013033718569204211,
+            -0.0038098562508821487,
+            0.005666834302246571,
+            0.02531767077744007,
+            -0.015448661521077156,
+            -0.00522220553830266,
+            0.008365118876099586,
+            -0.003567925887182355,
+            0.002730978187173605,
+            -0.008761797100305557,
+            0.006656350567936897,
+            -0.0325886569917202,
+            -0.008112290874123573,
+            0.004413592163473368,
+            0.0031908636447042227,
+            -0.001236895564943552,
+            -0.02758440561592579,
+            0.003960245288908482,
+            -0.00854384247213602,
+            -0.024201741442084312,
+            -0.00027380615938454866,
+            -0.020592402666807175,
+            0.006717377807945013,
+            -0.00881846621632576,
+            0.01761949434876442,
+            -0.014568123035132885,
+            0.011098276823759079,
+            -0.014367603696882725,
+            0.016006626188755035,
+            -0.007406116463243961,
+            -0.009398226626217365,
+            -0.0069266147911548615,
+            -0.006338136270642281,
+            0.01044441107660532,
+            -0.014916851185262203,
+            -0.009511562995612621,
+            -0.007641507778316736,
+            0.005161178298294544,
+            0.011045967228710651,
+            0.013574247248470783,
+            0.0011333668371662498,
+            -0.00880538858473301,
+            -0.016547156497836113,
+            -0.005392210558056831,
+            -0.014881978742778301,
+            -0.0009301236714236438,
+            0.0042174323461949825,
+            0.020853949710726738,
+            0.008160240948200226,
+            -0.003753187833353877,
+            0.011586496606469154,
+            0.020801641047000885,
+            0.014132212847471237,
+            -0.014777359552681446,
+            0.0021697436459362507,
+            -0.010095682926476002,
+            0.013417319394648075,
+            0.022144244983792305,
+            0.002098908182233572,
+            0.002796364715322852,
+            0.006085308268666267,
+            0.007951004430651665,
+            -0.004396155476570129,
+            0.008291014470160007,
+            -0.025457162410020828,
+            -0.012667553499341011,
+            -0.006569168530404568,
+            0.005387851502746344,
+            -0.015378915704786777,
+            0.029973194003105164,
+            -0.003609337378293276,
+            0.004472440108656883,
+            -0.002182820811867714,
+            0.0326409637928009,
+            0.007196879480034113,
+            0.00473834527656436,
+            0.002874828642234206,
+            9.467427298659459e-05,
+            0.009136680513620377,
+            -0.01761077716946602,
+            -0.03888320177793503,
+            0.003254070645198226,
+            -0.000592293159570545,
+            0.01766308583319187,
+            -0.01514352485537529,
+            0.006878664717078209,
+            -0.0036899808328598738,
+            -0.01282448135316372,
+            -0.009633618406951427,
+            0.03138554468750954,
+            -0.005466315429657698,
+            0.03387894853949547,
+            -0.007558684796094894,
+            -0.010104401037096977,
+            0.021725770086050034,
+            0.021900134161114693,
+            -0.014210676774382591,
+            0.005531701724976301,
+            0.015762517228722572,
+            0.020278548821806908,
+            0.005209128372371197,
+            -0.00806434080004692,
+            0.02627667412161827,
+            0.012946536764502525,
+            -0.012807045131921768,
+            -0.0026350778061896563,
+            -0.009572590701282024,
+            0.006726096384227276,
+            -0.012301389127969742,
+            0.005387851502746344,
+            0.006434036418795586,
+            0.004986814223229885,
+            0.012963972985744476,
+            -0.01879645325243473,
+            -0.003831651760265231,
+            0.0073014977388083935,
+            0.013966566883027554,
+            0.00651250034570694,
+            0.017558468505740166,
+            -0.023643774911761284,
+            0.0028966241516172886,
+            -0.02493407018482685,
+            -0.008103572763502598,
+            0.0029968833550810814,
+            0.001748000388033688,
+            0.008875134401023388,
+            -0.01592816226184368,
+            -0.029484974220395088,
+            0.011516750790178776,
+            0.01708768494427204,
+            0.02568383514881134,
+            0.0011126610916107893,
+            0.008914366364479065,
+            0.00560144754126668,
+            -0.013739893212914467,
+            -0.00805998221039772,
+            -0.00997362844645977,
+            -0.01073211245238781,
+            -0.011159304529428482,
+            -0.002761491807177663,
+            0.009119244292378426,
+            -0.010592620819807053,
+            0.0006821996648795903,
+            0.016346637159585953,
+            -0.011717269197106361,
+            -0.0051960512064397335,
+            0.015535843558609486,
+            -0.005008609499782324,
+            0.0007977159111760557,
+            0.0016891524428501725,
+            0.013086027465760708,
+            0.0034480507019907236,
+            -0.0063730087131261826,
+            -0.01591944508254528,
+            0.01215317938476801,
+            -0.00417166156694293,
+            0.0016074193408712745,
+            0.00708354264497757,
+            -0.007933568209409714,
+            -0.0047819362953305244,
+            0.013373728841543198,
+            -0.00854384247213602,
+            0.0016259454423561692,
+            -0.007606634870171547,
+            -0.017549749463796616,
+            -0.0076938169077038765,
+            0.006394804455339909,
+            -0.026730021461844444,
+            -0.013478347100317478,
+            0.016233300790190697,
+            -0.003570105414837599,
+            0.001546391868032515,
+            -0.0005138292908668518,
+            -0.011124431155622005,
+            0.003160349791869521,
+            -0.019598528742790222,
+            -0.01064493041485548,
+            -0.004864759277552366,
+            -0.02662540227174759,
+            0.017549749463796616,
+            0.015091215260326862,
+            0.0033172776456922293,
+            -0.00699636060744524,
+            -0.005592729430645704,
+            0.002567511750385165,
+            0.010348510928452015,
+            0.01763693243265152,
+            0.009311044588685036,
+            -0.005592729430645704,
+            -0.019685709848999977,
+            -0.003443691413849592,
+            0.021324733272194862,
+            -6.375188240781426e-05,
+            -0.016590746119618416,
+            0.020609840750694275,
+            0.007109697442501783,
+            0.014306576922535896,
+            0.0055883703753352165,
+            0.0023451976012438536,
+            0.005383492447435856,
+            0.014114775694906712,
+            -0.0001486181718064472,
+            -0.01783744990825653,
+            -0.013984003104269505,
+            0.0004890369018539786,
+            0.03140297904610634,
+            0.00708354264497757,
+            0.017139993607997894,
+            -0.03623286634683609,
+            0.0006642183288931847,
+            -0.005335542373359203,
+            -0.0038490882143378258,
+            -0.0045988536439836025,
+            0.027671586722135544,
+            0.010766984894871712,
+            -0.0001184450084110722,
+            0.015605589374899864,
+            -0.010043373331427574,
+            0.0422309935092926,
+            -0.015378915704786777,
+            -0.00737560261040926,
+            -0.010052092373371124,
+            -0.01685229316353798,
+            -0.003940629307180643,
+            -0.0011769578559324145,
+            -0.01689588464796543,
+            -0.006647632457315922,
+            -0.006320699583739042,
+            0.021411914378404617,
+            0.0046380856074392796,
+            -0.01665177382528782,
+            0.0017065888969227672,
+            -0.023835577070713043,
+            -0.0010946798138320446,
+            -0.022475536912679672,
+            -0.006403522565960884,
+            -0.012327543459832668,
+            -0.01349578332155943,
+            -0.020993441343307495,
+            -0.02031342126429081,
+            -0.004908350296318531,
+            0.016547156497836113,
+            0.00532246520742774,
+            -0.0069091785699129105,
+            -0.0021043571177870035,
+            -0.014402477070689201,
+            -0.010278765112161636,
+            0.02144678682088852,
+            0.015605589374899864,
+            0.012432162649929523,
+            0.01332141924649477,
+            0.01685229316353798,
+            0.0037597264163196087,
+            -0.034593842923641205,
+            0.006128899287432432,
+            0.0041607641614973545,
+            -0.005435801576822996,
+            0.01782873272895813,
+            0.000511104823090136,
+            -0.010793139226734638,
+            0.013399883173406124,
+            -0.007593557704240084,
+            0.018098996952176094,
+            -0.003212659154087305,
+            -0.004184739198535681,
+            -0.005536061245948076,
+            -0.008740002289414406,
+            -0.007371243555098772,
+            0.00455090356990695,
+            0.0021043571177870035,
+            0.013643993064761162,
+            0.0005775811732746661,
+            -0.004625008441507816,
+            -0.011333668604493141,
+            0.0014602995943278074,
+            -0.0032257363200187683,
+            -0.001628125086426735,
+            -0.023835577070713043,
+            -0.012170616537332535,
+            0.0017120378324761987,
+            -0.0086179468780756,
+            -0.01235369872301817,
+            0.02856084331870079,
+            -0.0017011400777846575,
+            -0.0008009852026589215,
+            0.010688520967960358,
+            0.015675334259867668,
+            0.001266319421119988,
+            0.01837797835469246,
+            -0.012170616537332535,
+            -0.011185458861291409,
+            -0.01649484597146511,
+            -0.01282448135316372,
+            -0.01122033130377531,
+            0.02090625837445259,
+            0.0033979208674281836,
+            -0.00708354264497757,
+            0.010418256744742393,
+            -0.011106994934380054,
+            -0.005056559573858976,
+            -0.0091018071398139,
+            0.011708551086485386,
+            -0.02969421073794365,
+            0.02069702185690403,
+            -0.012711144983768463,
+            0.006024280562996864,
+            0.018639525398612022,
+            0.039231929928064346,
+            -0.006608400493860245,
+            -0.006869946606457233,
+            0.010034655220806599,
+            -0.003068808699026704,
+            0.029205990955233574,
+            -0.0059414575807750225,
+            0.004237048327922821,
+            -0.021551406010985374,
+            0.0018449904164299369,
+            -0.01074954867362976,
+            0.0012815763475373387,
+            0.01991238258779049,
+            -0.0191974900662899,
+            -0.013722456991672516,
+            -0.00675660977140069,
+            -0.016058936715126038,
+            -0.00823870487511158,
+            0.022283734753727913,
+            0.011377259157598019,
+            -0.006111462600529194,
+            -0.008273578248918056,
+            -0.005736579652875662,
+            -0.024201741442084312,
+            0.014716332778334618,
+            0.08174190670251846,
+            -0.009982346557080746,
+            -0.0001858067698776722,
+            0.0003367407189216465,
+            0.0017806936521083117,
+            0.007907412946224213,
+            -0.00031821453012526035,
+            0.05956278741359711,
+            -0.003509078174829483,
+            0.003899217816069722,
+            -0.012998845428228378,
+            0.01083673071116209,
+            -0.01438504084944725,
+            -0.005684270523488522,
+            -0.009712082333862782,
+            -0.01197881530970335,
+            -0.0008167869527824223,
+            -0.02203962579369545,
+            -0.001063076313585043,
+            -0.011996251530945301,
+            -0.005021686665713787,
+            -0.021324733272194862,
+            0.017514877021312714,
+            -0.02087138593196869,
+            -0.01648612879216671,
+            -0.004594494588673115,
+            -0.010705957189202309,
+            0.012118306942284107,
+            -0.016555873677134514,
+            0.002380070276558399,
+            0.007270984351634979,
+            0.014228112995624542,
+            0.007475861813873053,
+            -0.008199472911655903,
+            -0.008408710360527039,
+            -0.0076197125017642975,
+            0.004986814223229885,
+            0.0069876424968242645,
+            -0.0011088468600064516,
+            0.014637868851423264,
+            -0.02088882215321064,
+            -0.016058936715126038,
+            0.007589198648929596,
+            0.0017817834159359336,
+            0.009502844884991646,
+            -0.007820230908691883,
+            0.007449707482010126,
+            0.00020882829267065972,
+            0.022754518315196037,
+            -0.010156710632145405,
+            -0.0268520750105381,
+            -0.007933568209409714,
+            0.008644101209938526,
+            -0.013138337060809135,
+            -0.006102744489908218,
+            0.005331183318048716,
+            0.009807982482016087,
+            0.008081777021288872,
+            -0.009877728298306465,
+            0.0007110786973498762,
+            -0.018918508663773537,
+            0.013373728841543198,
+            0.003238813718780875,
+            0.022196553647518158,
+            -0.012667553499341011,
+            0.010880321264266968,
+            -0.02683463878929615,
+            -0.002574050333350897,
+            -0.014419913291931152,
+            0.0008592882077209651,
+            0.008434864692389965,
+            0.004742704331874847,
+            0.01684357412159443,
+            -0.017192302271723747,
+            -0.017096402123570442,
+            0.009311044588685036,
+            -0.01782873272895813,
+            -0.02376583032310009,
+            -0.0018471699440851808,
+            -0.02624180167913437,
+            -0.0038163948338478804,
+            0.007532530464231968,
+            -0.019389290362596512,
+            -0.01369630265980959,
+            0.0065953233279287815,
+            -0.0033412526827305555,
+            -0.014611713588237762,
+            0.017183585092425346,
+            -0.011970097199082375,
+            -0.03544822707772255,
+            0.003727033268660307,
+            0.008382555097341537,
+            -0.004729627165943384,
+            0.006747891660779715,
+            -0.024410977959632874,
+            0.016015345230698586,
+            -0.002397506730630994,
+            0.004293716512620449,
+            0.008029468357563019,
+            0.02606743760406971,
+            0.0010309278732165694,
+            0.0034545892849564552,
+            -0.005544779356569052,
+            0.014411195181310177,
+            0.005801966413855553,
+            0.023504285141825676,
+            0.026416165754199028,
+            0.0015529305674135685,
+            -0.010958785191178322,
+            -0.01081929448992014,
+            0.01592816226184368,
+            -0.011586496606469154,
+            -0.0048386044800281525,
+            0.022737082093954086,
+            0.013827075250446796,
+            -0.02203962579369545,
+            0.00017259323794860393,
+            -0.02149909734725952,
+            0.0069876424968242645,
+            -0.017148712649941444,
+            0.009467972442507744,
+            0.008731283247470856,
+            -0.005073996260762215,
+            0.02069702185690403,
+            0.010095682926476002,
+            0.030984506011009216,
+            0.01991238258779049,
+            0.002920599188655615,
+            0.011412132531404495,
+            -0.006451472640037537,
+            0.010470565408468246,
+            0.004537826403975487,
+            0.0022275017108768225,
+            -0.006608400493860245,
+            0.006930973846465349,
+            0.0023451976012438536,
+            -0.001170419156551361,
+            0.008583074435591698,
+            -0.007471502758562565,
+            0.009921318851411343,
+            0.0005939278053119779,
+            0.0055273426696658134,
+            -0.004287178162485361,
+            -0.010575184598565102,
+            -0.030339358374476433,
+            -0.003132015699520707,
+            0.020958568900823593,
+            -0.023539157584309578,
+            -0.008003313094377518,
+            0.005932739470154047,
+            0.016983065754175186,
+            -0.008282296359539032,
+            -0.012798327021300793,
+            -0.01708768494427204,
+            0.003105861134827137,
+            -0.0004893093137070537,
+            -0.010313638485968113,
+            0.01747128553688526,
+            0.0009546435903757811,
+            -0.004951941315084696,
+            0.008953598327934742,
+            -0.00680456031113863,
+            0.028212115168571472,
+            0.01505634281784296,
+            0.011150586418807507,
+            -0.012100870721042156,
+            0.014201958663761616,
+            -0.012711144983768463,
+            0.007680739741772413,
+            0.009799264371395111,
+            -0.006416599731892347,
+            -0.021220114082098007,
+            -0.017558468505740166,
+            0.023451974615454674,
+            0.006848151329904795,
+            0.012222925201058388,
+            -0.011987533420324326,
+            -0.0012815763475373387,
+            0.007475861813873053,
+            0.004834245424717665,
+            -0.01611996442079544,
+            -0.00048631246318109334,
+            0.0061986446380615234,
+            0.038604218512773514,
+            0.005087073426693678,
+            0.01072339341044426,
+            -0.0035984397400170565,
+            -0.0002066487359115854,
+            0.019162617623806,
+            -0.01952878199517727,
+            -0.006503781769424677,
+            -0.005832480266690254,
+            -0.02739260531961918,
+            0.016573309898376465,
+            -0.01685229316353798,
+            -0.016451256349682808,
+            0.010976221412420273,
+            -0.00041356991278007627,
+            -0.004422310274094343,
+            -0.01284191757440567,
+            -0.008766156621277332,
+            0.00794228632003069,
+            0.010810576379299164,
+            -0.009136680513620377,
+            0.00532246520742774,
+            0.00357446470297873,
+            -0.0035984397400170565,
+            0.02127242274582386,
+            0.0006467819330282509,
+            -0.002837776206433773,
+            0.007358166389167309
+          ],
+          "title": "Method",
+          "keyphrases": [
+            "mixture-of-experts (MoE) models",
+            "token-choice routing",
+            "load imbalance",
+            "expert choice method",
+            "Transformer architecture"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": true,
+      "properties": {
+        "cosine_similarity": 0.8137348809038109
+      }
+    },
+    {
+      "id": "efbbda7f-a6fe-416a-9db1-5749e7344cd4",
+      "type": "cosine_similarity",
+      "source": {
+        "id": "b570d71e-4f83-486e-b70f-b0c6217db455",
+        "properties": {
+          "page_content": "Mixture-Of-Experts With Expert Choice Routing\n\nYanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon Google, Mountain View, CA, USA {yanqiz, taole, hanxiaol, dunan, huangyp, vzhao, adai, zhifengc, qvl, jlaudon}@google.com\n\nAbstract\n\nSparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choice method. Instead of letting tokens select the top-k experts, we have experts selecting the top-k tokens. As a result, each token can be routed to a variable number of experts and each expert can have a fixed bucket size. We systematically study pre-training speedups using the same computational resources of the Switch Transformer top-1 and GShard top-2 gating of prior work and find that our method improves training convergence time by more than 2×. For the same computational cost, our method demonstrates higher performance in fine-tuning 11 selected tasks in the GLUE and SuperGLUE benchmarks. For a smaller activation cost, our method outperforms the T5 dense model in 7 out of the 11 tasks.\n\n1 Introduction\n\nScaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27]. The final model quality has been found to have a power-law relationship with the amount of data, model size, and compute time [16, 20]. However, training efficiency, which is defined as the total amount of computation used to achieve superior model quality than the state of the art system [21], should receive greater attention as we increase our efforts towards green AI [29].\n\nSparsely gated mixture-of-experts [31] (MoE) provides an effective way to scale model capacity given a fixed computational cost, and has recently played an important role in increasing the training efficiency of large-scale language models [10, 21]. MoE operate by adopting a number of experts, each as a sub-network, and by activating only one or a few experts for each input token. A gating network must be chosen and optimized in order to route each token to the most suited expert(s). For example, recent work has implemented sparse routing via k-means clustering [12], linear assignment to maximize token-expert affinities [22], or hashing [8, 28]. Many of the prior work use a routing strategy concerning the token choice, where each token selects the best one or two experts.\n\nWe argue that the independent token choice of prior work often leads to an imbalanced load of experts, which causes training inefficiency and sub-optimal training of the model. In order to mitigate this\n\nissue, previous sparsely gated networks introduce additional auxiliary losses as regularization to prevent too many tokens being routed to a single expert, but the effectiveness is still limited. Recent approaches [8, 22, 28] explore alternative strategies for routing, but they focus on pre-training only and do not demonstrate performance gain on downstream tasks. Moreover, none of the previous methods consider allocating a variable number of experts to each token based on importance, which can be beneficial.\n\nWe propose a very simple yet effective routing method we are calling expert choice. Unlike conventional MoE where tokens select one or two top-scoring experts, our method lets each expert pick the top-k tokens. Our method guarantees perfect load balancing, allows a variable number of experts for each token, and achieves substantial gains in training efficiency and downstream performance as demonstrated in our experiments. Our major contributions include:\n\nWe identify common pitfalls in conventional MoE such as load imbalance as described in Section 3.1. We then propose a heterogeneous, expert choice method to provide a fluid allocation of model parameters based on a learnt token-to-expert importance. This method intrinsically guarantees load balance without imposing an auxiliary loss.\n\nWe show our method provides over 2× faster training convergence in a 8B/64E (8 billion activated parameters, 64 experts) model, compared to the top-1 and top-2 gating counterparts in Switch Transformer [10] and GShard [21].\n\nWe show our method demonstrates strong scaling when increasing the number of experts from 16 to 128, evaluated in training perplexity.\n\nWe show our method demonstrates strong performance on downstream tasks selected from GLUE and SuperGLUE at all the evaluated scales. More specifically, our 8B/64E model outperforms a T5 11B dense model in 7 out of 11 tasks evaluated.\n\n2 Related Work\n\nScaling: Various approaches have been proposed to scale up neural network capacity to improve performance. Recent works have successfully scaled models to billions of parameters via various forms of model parallelism [2, 21, 26, 27, 33]. Model parallelism [30] splits weights and tensors across multiple cores while pipeline parallelism [18, 24] splits different layers across devices with micro-batches pipelined to the different layers. To enable continued scaling of neural networks, improving model training and serving efficiency has become a critical research area.\n\nConditional Computation: Computation decisions can be made dynamically based on the input [23, 25]. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation, by activating certain parameters and computation on demand, on a per-example or per-token basis [3]. Conditional convolution layers [1] with task-specific gating has been used to combat catastrophic forgetting when a sequence of learning problems are optimized. The gating decisions may be binary or sparse and continuous, stochastic or deterministic.\n\nMixture of Experts: Sparsely-gated MoE [31] is the first model to demonstrate massive improvements in model capacity, training time, or model quality with gating. Switch Transformer [10] simplifies the gating by selecting only the top expert per token using a softmax over the hidden state and demonstrates better scaling than previous work. All the prior work requires an auxiliary loss to explicitly encourage balancing. This loss term has to be carefully weighted to not overwhelm the primary loss. However, auxiliary loss does not guarantee balancing and a hard capacity factor has to be imposed. As a result, many tokens can still be unprocessed by the MoE layer. Hard MoE [12] with a single decoding layer can be efficiently trained to good effect on large scale hashtag prediction tasks.\n\nBase Layers [22] formulate a linear assignment that maximizes token-expert affinities while ensuring each expert receives an equal number of tokens. Hash layers [8, 28] devise hashing techniques on input tokens. However, the evaluations are limited to pre-training perplexity. THOR [? ] randomly activates experts during training and inference and is trained with a consistency regularization loss.\n\nTHOR has demonstrated strong performance on translation tasks. Different from these prior works, our method is a learnt method that enables heterogeneous MoE and effectively improves downstream fine-tuning performance.\n\n3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n5 Conclusion\n\nWe propose a new routing method for sparsely activated mixture-of-experts (MoE) models. This method addresses load imbalance and under-utilization of experts in conventional MoE methods, and enables selecting different numbers of experts for each token. Our model demonstrates more than 2x training efficiency improvements when compared to the state-of-the-art GShard and Switch Transformer models, and also achieves strong gains when finetuning on 11 datasets in the GLUE and SuperGLUE benchmark.\n\n6 Limitations\n\nThe expert choice method might not immediately apply to auto-regressive text generation as our current implementation takes in the past and future tokens to perform the top-k selection. One possible solution is to collect a large batch of input sequences, dispatch tokens of the same sequence into separate groups, and perform expert choice routing for each group. Another scenario where the expert choice method does not immediately apply is when the batch size becomes very small during serving or inference. A global top-k can be selected instead and we can cap the number of times each expert or token gets selected. We leave these possible improvements for future work.\n\nAnother long-standing issue with MoE has been the large memory footprint. Even though computational cost can be reduced using sparsely gated networks, the total number of parameters increases linearly or sub-linearly with the number of experts. Increasing the number of experts requires reservation of a large number of hardware devices. Therefore, dynamic (used) power is saved while static (reserved) power is not. Power saving techniques such as the ability to put hardware devices into low power states while not in use [17] can help with reducing the reserved power requirements.\n\nReferences\n\n[1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pages 3930–3939. Computer Vision Foundation / IEEE, 2020.\n\n[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. In Advances in Neural Information Processing Systems.\n\n[3] Kyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning, 2014.\n\n[4] Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems, 2021.\n\n[5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.\n\nTransformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.\n\n[6] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 933–941. JMLR.org, 2017.\n\n[7] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixtureof-experts, 2021.\n\n[8] Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan.\n\nTricks for training sparse translation models, 2021.\n\n[9] Richard L Dykstra. An iterative procedure for obtaining i-projections onto the intersection of convex sets. The annals of Probability, pages 975–984, 1985.\n\n[10] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.\n\n[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. NAS-FPN: learning scalable feature pyramid architecture for object detection. In CVPR, pages 7036–7045. Computer Vision Foundation / IEEE, 2019.\n\n[12] Sam Gross, Marc'Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision, 2017.\n\n[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.\n\n[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 630–645, Cham, 2016. Springer International Publishing.\n\n[15] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2016.\n\n[16] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.\n\n[17] Ping Huang, Zuocheng Xing, Tianran Wang, Qiang Wei, Hongyan Wang, and Guitao Fu. A brief survey on power gating design. In 2010 10th IEEE International Conference on Solid-State and Integrated Circuit Technology, pages 788–790, 2010.\n\n[18] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 103–112, 2019.\n\n[19] Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David A. Patterson. A domain-specific supercomputer for training deep neural networks. Commun. ACM, 63(7):67–78, 2020.\n\n[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.\n\n[21] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.\n\n[22] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6265–6274. PMLR, 18–24 Jul 2021.\n\n[24] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. New York, NY, USA, 2019. Association for Computing Machinery.\n\n[25] Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, Cédric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. In ICLR. OpenReview.net, 2021.\n\n[26] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.\n\n[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.\n\n[28] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models, 2021.\n\n[30] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 10435–10444, Red Hook, NY, USA, 2018. Curran Associates Inc.\n\n[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E.\n\nHinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In ICLR (Poster). OpenReview.net, 2017.\n\n[32] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 10–15 Jul 2018. [23] Min Lin, Jie Fu, and Yoshua Bengio. Conditional computation for continual learning, 2019.\n\n[29] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.\n\n[33] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.\n\n[34] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc.\n\n[35] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.\n\nGLUE: A multi-task benchmark and analysis platform for natural language understanding.\n\nIn Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, November 2018. Association for Computational Linguistics.\n\n[36] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake A. Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen. GSPMD: general and scalable parallelization for ML computation graphs. CoRR, abs/2105.04663, 2021.\n\n7 Checklist\n\n(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Yes (b) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes (c) Did you discuss any potential negative societal impacts of your work? N/A. Not any. (d) Did you describe the limitations of your work? Yes (a) Did you include the code, data, and instructions needed to reproduce the main experimental results? Yes. We include details in the experiment setup to help reproduce the main results. (b) Did you specify all the training details? Yes (c) Did you report error bars? Yes (d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? Yes (a) If your work uses existing assets, did you cite the creators? Yes (b) Did you mention the license of the assets? No. The used dataset is not released yet. (c) Did you include any new assets either in the supplemental material or as a URL? No. The dataset is not released yet.\n\n(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? No. Not using persons' data. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes. The dataset does not contain any personally identifiable information or offensive content.\n\nA Comparison On Fine-Tuning With A Dense Model\n\nOur 8B MoE model achieves stronger pre-training perplexity than its dense counterpart. However, a better perplexity does not always directly translate to downstream performance as demonstrated in Section 4.4. To this end, we compare fine-tuning performance of the 8B dense model and MoE model in Table 1. As shown in the table, our MoE model using expert choice routing consistently outperforms the dense model across the 11 tasks in GLUE and SuperGLUE.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 1: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nB Capacity Factor\n\nWe evaluate the downstream task fine-tuning performance by varying the capacity factors. Note that a capacity factor of n indicates on average how many experts each token can be received. EC-CF2 is our baseline expert choice, which matches GShard top-2 gating computational footprint. EC-CF1, however, matches Switch Transformer top-1 gating computational footprint. EC-CF0.5 further verifies that an aggressively lowered capacity factor can provide strong enough performance, that almost matches the top-2 gating baseline.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Top-2 78.1 87.0 88.3 85.0 82.6 90.1 90.7 81.6 94.7 68.2 67.2 83.0±0.3 EC-CAP2 78.2 88.0 88.5 85.7 83.0 90.8 91.1 80.0 95.4 70.4 64.1 83.2±0.4 EC-CAP3 78.5 91.7 89.3 86.3 83.5 90.9 91.1 81.8 94.9 70.0 65.6 84.0±0.4 EC-CF2 79.1 89.6 89.3 86.8 84.3 91.3 91.2 81.1 95.2 68.1 68.0 84.0±0.2 EC-CF1 77.4 90.6 88.0 85.5 83.6 90.3 91.2 79.8 95.3 66.5 64.9 83.0±0.2 EC-CF0.5 77.4 89.6 86.3 85.2 82.7 91.7 91.0 79.6 94.9 67.3 63.5 83.0 ±0.05 Hash Layers 76.1 85.2 86.7 83.4 82.5 90.0 90.3 75.7 94.0 67.4 63.3 81.3±1.0\n\nTable 2: Comparison between different routing methods in fine-tuning of 100M/64E models. We perform 3 independent fine-tuning runs for each method and report the average results. This gives more accurate difference between the variants of expert choice method, since they achieve close fine-tuning results. We do not report averaged results in other experiments.\n\nC Capped Expert Choice\n\nAs described in Section 4.5, the maximum number of experts each token is assigned can be capped by an entropy-regularized linear programming. Figure 1 compares the validation perplexity when training the 100M/64E models using the base expert choice method (EC-BASE), expert choice capped by two experts per token (EC-CAP2), expert choice capped by three experts per token (EC-CAP3), and GShard top-2 gating.\n\nAs shown in the figure, restricting the number of experts to 2 degrades the perplexity compared to the base expert choice method. This suggests that a more flexible allocation of experts (e.g. more than 2 experts for a token) can enhance model expressiveness. On the other hand, our EC-CAP2 and EC-CAP3 methods still outperform the top-2 gating method by a clear margin. We believe this confirms the effectiveness of a load balanced training, provided by our method. Finally, EC-CAP3 obtains comparable perplexity to EC-BASE. As indicated by Figure 3, only a little fraction of tokens use more than 3 experts therefore we see little or no difference between EC-BASE and EC-CAP3 variants. We present the fine-tuning results of these methods in Table 2.\n\nD Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [? ]. We use mod x to map a token ID to an expert ID. This in some way ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 2. Hashing based routing performs much worse than expert choice in terms of average scores and variance.\n\nE Fine-Tuning Details\n\nWe did a hyperparameter search for both baseline models and expert choice method. For fine-tuning of the 8B dense model, we use a constant learning rate of 0.0001 and a dropout rate of 0.1. We freeze the attention layer and feed-forward layer while leaving the embedding and layer normalization trainable. This setting has been found optimal for the 8B dense model. For MoE 8B/64E models including GShard top-2 gating and expert choice, we found continuing the learning rate from the pre-trained model while using a square root learning rate decay works better. In addition, we do not apply parameter freezing for fine-tuning MoE models. For models with 100M expert size, we use a constant learning rate of 0.0001 and no dropout is used.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/2202.09368v2.md"
+          },
+          "headlines": {
+            "1 Introduction": [],
+            "2 Related Work": [],
+            "3 Method": [
+              "3.1 Pitfalls Of Token-Choice Routing",
+              "3.2 Heterogeneous Moe Via Expert Choice",
+              "3.3 Expert Choice With Additional Constraint",
+              "3.4 Model Architecture"
+            ],
+            "4 Experiments": [
+              "4.1 Setup",
+              "4.2 Training Efficiency",
+              "4.3 Scaling The Number Of Experts",
+              "4.4 Fine-Tuning On Glue And Superglue",
+              "4.5 Heterogeneity Matters",
+              "4.6 Comparison With Hash Layer",
+              "4.7 Ablation"
+            ],
+            "5 Conclusion": [],
+            "6 Limitations": [],
+            "A Comparison On Fine-Tuning With A Dense Model": [],
+            "B Capacity Factor": [],
+            "C Capped Expert Choice": [],
+            "D Comparison With Hash Layer": [],
+            "E Fine-Tuning Details": []
+          },
+          "summary": "The paper introduces a new routing method for sparsely-activated Mixture-of-Experts (MoE) models called 'expert choice,' which addresses load imbalance and under-utilization issues in conventional MoE methods. Unlike traditional token-choice routing, expert choice allows experts to select the top-k tokens, ensuring perfect load balancing and enabling variable expert allocation per token. This method improves training efficiency, achieving over 2x faster convergence compared to existing models like Switch Transformer and GShard. It also demonstrates superior performance on downstream tasks in the GLUE and SuperGLUE benchmarks. The study highlights the benefits of flexible expert allocation and shows that expert choice outperforms hashing-based routing methods. The paper also discusses potential limitations and future improvements for the expert choice method.",
+          "summary_embedding": [
+            -0.009726986289024353,
+            -0.0337262786924839,
+            -0.023302342742681503,
+            -0.019878201186656952,
+            -0.025953780859708786,
+            0.04560471698641777,
+            -0.006874797400087118,
+            0.039665497839450836,
+            -0.006973279174417257,
+            0.034544438123703,
+            -0.025787118822336197,
+            -0.00363057479262352,
+            -0.01052999310195446,
+            0.0054657477885484695,
+            -0.024499278515577316,
+            0.036786794662475586,
+            0.010007280856370926,
+            0.003272630739957094,
+            0.006102093029767275,
+            -0.04048365354537964,
+            -0.018666116520762444,
+            -0.013522328808903694,
+            -0.03230207785964012,
+            0.008969432674348354,
+            0.02484775334596634,
+            0.0016713523073121905,
+            0.0026230288203805685,
+            0.003624893259257078,
+            -0.05311964824795723,
+            0.04339266195893288,
+            0.018681267276406288,
+            0.023711422458291054,
+            -0.016332851722836494,
+            -0.012832955457270145,
+            -0.039120059460401535,
+            -0.0197418425232172,
+            0.030786970630288124,
+            0.039968520402908325,
+            -0.019287310540676117,
+            0.03357476741075516,
+            0.018060073256492615,
+            0.02454473078250885,
+            -0.021272100508213043,
+            -0.029817303642630577,
+            -0.0061892117373645306,
+            -0.007840678095817566,
+            -0.008355814032256603,
+            0.01242387667298317,
+            0.016984347254037857,
+            -0.013007191941142082,
+            0.0212569497525692,
+            -0.01371171697974205,
+            0.03384748846292496,
+            0.008916404098272324,
+            -0.042332086712121964,
+            0.03984731063246727,
+            -0.008007340133190155,
+            0.012105703353881836,
+            0.03281721472740173,
+            0.010333028621971607,
+            -0.005287723150104284,
+            -0.021544819697737694,
+            -0.03378688171505928,
+            0.0010529992869123816,
+            0.00014014738553669304,
+            0.005950582213699818,
+            -0.003069985192269087,
+            -0.009742137044668198,
+            0.03227177634835243,
+            -0.003939277958124876,
+            -0.0740584209561348,
+            0.050089433789253235,
+            -0.026590125635266304,
+            -0.007628562860190868,
+            -0.028938541188836098,
+            -0.01736312545835972,
+            0.037362534552812576,
+            0.005662711802870035,
+            0.0098709212616086,
+            0.0106284748762846,
+            -0.0007892759749665856,
+            0.028923390433192253,
+            0.00910579226911068,
+            -0.024923507124185562,
+            0.04478655755519867,
+            -0.04790768027305603,
+            0.01801462098956108,
+            -0.04336235672235489,
+            0.022756904363632202,
+            1.6393929399782792e-05,
+            -0.020726662129163742,
+            -0.0013304532039910555,
+            -0.016666175797581673,
+            0.0218023881316185,
+            -0.01730252057313919,
+            0.026802239939570427,
+            0.013825350441038609,
+            -0.015802564099431038,
+            -0.029817303642630577,
+            0.025378039106726646,
+            0.003367325058206916,
+            0.010499690659344196,
+            -0.02075696364045143,
+            0.014332910999655724,
+            0.04393810033798218,
+            -0.02593863010406494,
+            -0.02271145209670067,
+            0.027681002393364906,
+            0.014408666640520096,
+            0.0633314698934555,
+            0.036544378846883774,
+            0.006624804809689522,
+            0.022362977266311646,
+            0.020575150847434998,
+            -0.028968842700123787,
+            -0.01963578537106514,
+            -0.046362269669771194,
+            -0.04757435619831085,
+            0.019878201186656952,
+            -0.04566532000899315,
+            0.0050869714468717575,
+            0.038544319570064545,
+            -0.013704141601920128,
+            -0.014446543529629707,
+            -0.04021093621850014,
+            0.017847958952188492,
+            0.0003908502112608403,
+            0.011976920068264008,
+            -0.0022082682698965073,
+            0.00033285003155469894,
+            -0.011825408786535263,
+            0.008878526277840137,
+            -0.03778676688671112,
+            -0.025196226313710213,
+            0.025181075558066368,
+            0.022862963378429413,
+            0.024120500311255455,
+            0.028802180662751198,
+            -0.03077181987464428,
+            -0.009984553791582584,
+            0.021378157660365105,
+            -0.04242299124598503,
+            -0.006556625012308359,
+            0.011332998983561993,
+            0.0026703758630901575,
+            0.003872992005199194,
+            0.032786913216114044,
+            0.052998438477516174,
+            -0.03172633796930313,
+            0.02065090648829937,
+            -0.010431510396301746,
+            0.04957429692149162,
+            0.00024691506405360997,
+            0.028968842700123787,
+            -0.019514575600624084,
+            -0.0134389977902174,
+            -0.029529431834816933,
+            -0.009795165620744228,
+            -0.01356020662933588,
+            -0.008484598249197006,
+            -0.002998017705976963,
+            -0.009575475007295609,
+            0.0040453351102769375,
+            0.0331808403134346,
+            0.00273855566047132,
+            0.030680913478136063,
+            0.014325335621833801,
+            0.0051475754007697105,
+            -0.08745196461677551,
+            0.026059838011860847,
+            -0.0033616432920098305,
+            0.005859675817191601,
+            0.0062460280023515224,
+            0.03013547509908676,
+            -0.0312718041241169,
+            0.020938776433467865,
+            0.026620427146553993,
+            0.0011145505122840405,
+            0.016969196498394012,
+            0.036847397685050964,
+            -0.04181694984436035,
+            0.009643655270338058,
+            0.012590537779033184,
+            -0.0033881578128784895,
+            0.04584713280200958,
+            0.011628445237874985,
+            0.019514575600624084,
+            0.018529755994677544,
+            0.003569970605894923,
+            0.00297907879576087,
+            0.006533898413181305,
+            0.03724132478237152,
+            0.003462019143626094,
+            -0.009833043441176414,
+            0.014976831153035164,
+            -0.010757259093225002,
+            -0.011219366453588009,
+            -0.037998881191015244,
+            0.014454119838774204,
+            0.008742166683077812,
+            -0.017454031854867935,
+            -0.02959003672003746,
+            0.0026949963066726923,
+            -0.031908150762319565,
+            0.026559824123978615,
+            -0.002445003716275096,
+            0.005738467443734407,
+            -0.010454237461090088,
+            -0.006757376715540886,
+            -0.035423196852207184,
+            0.028453707695007324,
+            -0.02095392718911171,
+            0.0053142374381423,
+            0.0007258309051394463,
+            -0.029438525438308716,
+            0.0060073984786868095,
+            0.017863109707832336,
+            -0.0025965142995119095,
+            0.0025302283465862274,
+            -0.02349930815398693,
+            0.024468975141644478,
+            0.025635607540607452,
+            0.004518806003034115,
+            -0.014037464745342731,
+            -0.013113250024616718,
+            -0.003605954349040985,
+            -0.009742137044668198,
+            -0.016196493059396744,
+            0.024468975141644478,
+            -0.014810169115662575,
+            0.00308892410248518,
+            0.021847840398550034,
+            0.014264730736613274,
+            -0.007045246660709381,
+            -0.004007457755506039,
+            -0.04066546633839607,
+            -0.01218903437256813,
+            -0.04760465770959854,
+            -0.011628445237874985,
+            -0.0034601252991706133,
+            -0.028696123510599136,
+            0.0061513339169323444,
+            -0.04481685906648636,
+            -0.01235569640994072,
+            0.02309022843837738,
+            0.018135828897356987,
+            0.0019488062243908644,
+            -0.009173971600830555,
+            -0.022605394944548607,
+            8.486965816700831e-05,
+            -0.01763584464788437,
+            -0.017044952139258385,
+            0.015393485315144062,
+            0.031059689819812775,
+            -0.03101423755288124,
+            0.04024123772978783,
+            0.003893824527040124,
+            -0.08066429197788239,
+            0.010393633507192135,
+            -0.026362858712673187,
+            0.07011914253234863,
+            -0.0025453795678913593,
+            0.014075342565774918,
+            0.06781618297100067,
+            -0.012431452050805092,
+            -0.02007516659796238,
+            0.03142331540584564,
+            -0.008249756880104542,
+            -1.1311515663692262e-05,
+            -0.02884763479232788,
+            -0.02739313244819641,
+            -0.053271155804395676,
+            -0.009704259224236012,
+            -0.010787560604512691,
+            0.03360506892204285,
+            -0.024317465722560883,
+            -0.0007656024536117911,
+            0.012620840221643448,
+            0.02983245439827442,
+            0.016635872423648834,
+            -0.0015681355725973845,
+            0.008272483013570309,
+            0.02881733328104019,
+            0.0274234339594841,
+            -0.0036154238041490316,
+            0.00814369972795248,
+            -0.034211114048957825,
+            -0.0068066176027059555,
+            -0.0493621826171875,
+            0.011302697472274303,
+            -0.02596893161535263,
+            -0.02596893161535263,
+            0.007643714081496,
+            -0.011476934887468815,
+            0.03748374432325363,
+            -0.024696242064237595,
+            -0.007503566797822714,
+            0.005037730094045401,
+            0.012855681590735912,
+            -0.017757052555680275,
+            -0.03612014651298523,
+            -0.0028502948116511106,
+            0.0026381798088550568,
+            0.007560383062809706,
+            -0.053240854293107986,
+            0.004499867558479309,
+            -0.018469152972102165,
+            0.0279688723385334,
+            -0.02139330841600895,
+            -0.013688990846276283,
+            0.042029064148664474,
+            0.011363301426172256,
+            0.027499189600348473,
+            -0.026468917727470398,
+            0.0017849853029474616,
+            0.012825379148125648,
+            -0.0020056227222085,
+            0.032514192163944244,
+            -0.030574856325984,
+            -0.03257479891180992,
+            0.001548249856568873,
+            -0.007840678095817566,
+            -0.0625436082482338,
+            -0.011386028490960598,
+            -0.00981789268553257,
+            0.017317671328783035,
+            0.0446956530213356,
+            0.04557441547513008,
+            -0.01068150345236063,
+            0.002174178371205926,
+            -0.02213571034371853,
+            -0.04542290419340134,
+            -0.02643861435353756,
+            0.0016486257081851363,
+            -0.019999410957098007,
+            -0.02789311669766903,
+            -0.0018313854234293103,
+            0.011211791075766087,
+            -0.013651113025844097,
+            0.01134057529270649,
+            -0.018560059368610382,
+            -0.01862066239118576,
+            -0.012332970276474953,
+            0.018575210124254227,
+            0.009901223704218864,
+            -0.0018796793883666396,
+            -0.015037435106933117,
+            -0.00980274099856615,
+            0.018969137221574783,
+            -0.011279970407485962,
+            0.0009388927719555795,
+            0.01188601367175579,
+            0.024347767233848572,
+            -0.007083124481141567,
+            0.010961798019707203,
+            -0.034756552428007126,
+            -0.01966608688235283,
+            0.007984613068401814,
+            0.002719616750255227,
+            0.020711511373519897,
+            0.008742166683077812,
+            -0.04275631532073021,
+            -0.044301725924015045,
+            0.016044981777668,
+            -0.0562104657292366,
+            0.028120383620262146,
+            -0.03327174484729767,
+            -0.012514783069491386,
+            -0.02671133354306221,
+            0.0013418165035545826,
+            -0.01828734017908573,
+            0.022438732907176018,
+            0.03778676688671112,
+            -0.0011486404109746218,
+            0.005503625608980656,
+            0.014658658765256405,
+            0.04384719207882881,
+            -0.05021063983440399,
+            -0.041180603206157684,
+            0.03754434734582901,
+            -0.01601468026638031,
+            0.0185146052390337,
+            0.00021744151308666915,
+            0.010348179377615452,
+            -0.014431392773985863,
+            0.022772056981921196,
+            0.013757170177996159,
+            -0.008764892816543579,
+            -0.030574856325984,
+            0.04945308715105057,
+            -0.023650817573070526,
+            -0.049513693898916245,
+            0.0018323323456570506,
+            0.018878230825066566,
+            0.018635815009474754,
+            -0.014870774000883102,
+            -0.02907489985227585,
+            -0.024317465722560883,
+            -0.006060427520424128,
+            -0.00947699323296547,
+            -0.03402930125594139,
+            0.011363301426172256,
+            0.0048445542342960835,
+            0.016666175797581673,
+            -0.00645056739449501,
+            -0.0073141781613230705,
+            -0.009370936080813408,
+            0.022559940814971924,
+            0.012635990977287292,
+            -0.03175663948059082,
+            -0.017484333366155624,
+            -0.02528713271021843,
+            0.004359719809144735,
+            0.010136065073311329,
+            0.022166013717651367,
+            0.024287162348628044,
+            -0.0323626808822155,
+            0.04218057543039322,
+            0.01550711877644062,
+            -0.008727015927433968,
+            -0.0019317612750455737,
+            0.02478714846074581,
+            0.04324115067720413,
+            0.0218023881316185,
+            0.026514369994401932,
+            0.002107892418280244,
+            8.705945947440341e-05,
+            -0.015620751306414604,
+            -0.0011391709558665752,
+            0.01862066239118576,
+            0.04911976307630539,
+            -0.012151156552135944,
+            0.01716616004705429,
+            -0.03578682243824005,
+            0.00396200455725193,
+            -0.02580226957798004,
+            -0.030786970630288124,
+            0.03745344281196594,
+            -0.015060162171721458,
+            0.0191964041441679,
+            0.025044716894626617,
+            -0.022469034418463707,
+            -0.011075431481003761,
+            -0.019423669204115868,
+            -0.026181045919656754,
+            -0.01702980138361454,
+            -0.03281721472740173,
+            -0.018135828897356987,
+            -0.002653330797329545,
+            -0.035635314881801605,
+            -0.04987731948494911,
+            -0.004329417832195759,
+            -0.046392571181058884,
+            -0.010507266037166119,
+            0.017378276214003563,
+            -0.011166337877511978,
+            -0.025711363181471825,
+            -0.005336964037269354,
+            0.003973367623984814,
+            -0.022196315228939056,
+            -0.0066588944755494595,
+            0.008439145050942898,
+            0.035938333719968796,
+            -0.02298417128622532,
+            -0.013060221448540688,
+            -0.0024317465722560883,
+            -0.02887793630361557,
+            -0.007878555916249752,
+            0.008999735116958618,
+            0.007723257411271334,
+            -0.0012357590021565557,
+            -0.05196816474199295,
+            -0.006916462909430265,
+            -0.012992041185498238,
+            0.029196109622716904,
+            0.026468917727470398,
+            0.0012168202083557844,
+            -0.02783251367509365,
+            0.008749742060899734,
+            -0.0072535742074251175,
+            0.02339324913918972,
+            -0.007992188446223736,
+            -0.022908415645360947,
+            -0.02034788578748703,
+            0.030786970630288124,
+            -0.0159086212515831,
+            0.009113367646932602,
+            0.009976978413760662,
+            0.0121360057964921,
+            -0.03899884968996048,
+            -0.001445980160497129,
+            0.0005648507503792644,
+            -0.010128489695489407,
+            0.00423093605786562,
+            -0.006738437805324793,
+            -0.007079336792230606,
+            -0.022090258076786995,
+            -0.0218023881316185,
+            0.03818069398403168,
+            -0.009113367646932602,
+            -0.0035661826841533184,
+            -0.01088604237884283,
+            0.03436262533068657,
+            -0.00595815759152174,
+            -0.01993880607187748,
+            -0.019347915425896645,
+            0.014431392773985863,
+            0.05105910077691078,
+            -0.028529461473226547,
+            -0.03984731063246727,
+            0.0011136035900563002,
+            0.030150625854730606,
+            0.004140029661357403,
+            -0.016908591613173485,
+            0.0035851215943694115,
+            0.056392278522253036,
+            -0.011742077767848969,
+            0.028362801298499107,
+            0.027771908789873123,
+            0.02295386977493763,
+            -0.02021152526140213,
+            -0.01678738370537758,
+            -0.020999381318688393,
+            0.011098157614469528,
+            -0.03027183562517166,
+            0.027059808373451233,
+            0.018969137221574783,
+            0.01538590993732214,
+            0.05054396390914917,
+            -0.03178694099187851,
+            -0.02515077404677868,
+            0.008499749936163425,
+            0.0021722845267504454,
+            0.011007251217961311,
+            -0.007727045100182295,
+            -0.01801462098956108,
+            -0.001950700068846345,
+            -0.014022313989698887,
+            -0.019817598164081573,
+            -0.020938776433467865,
+            0.00954517349600792,
+            0.015696506947278976,
+            0.002649543108418584,
+            0.010863316245377064,
+            0.0016590419691056013,
+            0.01686313934624195,
+            0.01321930717676878,
+            -0.0444229319691658,
+            -7.936555630294606e-05,
+            0.03378688171505928,
+            -0.0293476190418005,
+            -0.012264790013432503,
+            -0.016060132533311844,
+            -0.004439263138920069,
+            -0.01706010289490223,
+            -0.02762039750814438,
+            -0.006874797400087118,
+            -0.01889338344335556,
+            -0.00814369972795248,
+            -0.028196139261126518,
+            -0.009992130100727081,
+            0.01949942484498024,
+            0.022332675755023956,
+            0.001196934375911951,
+            -0.006783891003578901,
+            -0.005901341326534748,
+            0.02092362567782402,
+            0.032514192163944244,
+            0.01952972821891308,
+            0.0058331615291535854,
+            -0.015158643946051598,
+            -0.004295328166335821,
+            -0.012370847165584564,
+            0.016060132533311844,
+            0.012537509202957153,
+            6.344509893096983e-05,
+            0.03493836522102356,
+            0.004170331638306379,
+            -0.022726602852344513,
+            -0.001230077352374792,
+            0.024923507124185562,
+            -0.021590271964669228,
+            -0.0108330138027668,
+            0.040029123425483704,
+            0.0085679292678833,
+            -0.0024506852496415377,
+            -0.019287310540676117,
+            -0.019075196236371994,
+            0.02630225569009781,
+            -0.02133270539343357,
+            0.002083271974697709,
+            0.0482713058590889,
+            0.00035344602656550705,
+            -0.029802151024341583,
+            0.017954016104340553,
+            -0.013242034241557121,
+            -0.018560059368610382,
+            0.01848430372774601,
+            0.0012859469279646873,
+            -0.039968520402908325,
+            0.019014591351151466,
+            0.018408548086881638,
+            0.022635696455836296,
+            -0.02918095886707306,
+            0.0002127068000845611,
+            0.03233237937092781,
+            0.004037759732455015,
+            0.001595596899278462,
+            -0.008530051447451115,
+            -0.0006065162015147507,
+            0.010431510396301746,
+            0.001746160676702857,
+            0.02854461409151554,
+            -0.0030434709042310715,
+            0.027771908789873123,
+            -0.006727074272930622,
+            -0.008863375522196293,
+            0.019181253388524055,
+            -0.02109028771519661,
+            0.013166278600692749,
+            0.0685434341430664,
+            0.0021950111258774996,
+            0.025847723707556725,
+            0.007181606721132994,
+            0.024347767233848572,
+            0.02018122375011444,
+            -0.027271922677755356,
+            -0.02021152526140213,
+            -0.005397567991167307,
+            -0.0014033677289262414,
+            -0.03787767142057419,
+            0.011128460057079792,
+            -0.021302402019500732,
+            -0.013249609619379044,
+            -0.0022347825579345226,
+            -0.002316219499334693,
+            -0.006405114196240902,
+            0.034211114048957825,
+            -0.0014630251098424196,
+            -0.018560059368610382,
+            -0.035423196852207184,
+            0.037726160138845444,
+            0.04048365354537964,
+            0.0389382466673851,
+            -0.005984671879559755,
+            0.03548380360007286,
+            0.0030756669584661722,
+            0.0025302283465862274,
+            0.041453324258327484,
+            0.004791525658220053,
+            -0.011567841283977032,
+            0.02295386977493763,
+            0.012416300363838673,
+            0.03351416438817978,
+            -0.02045394293963909,
+            -0.02630225569009781,
+            -0.003299145260825753,
+            -0.026423463597893715,
+            0.012492056004703045,
+            -0.019726691767573357,
+            0.0018768386216834188,
+            0.007117214612662792,
+            0.015196521766483784,
+            -0.016044981777668,
+            0.014719262719154358,
+            -0.006102093029767275,
+            -0.021014532074332237,
+            0.011719351634383202,
+            -0.007780073676258326,
+            0.011961768381297588,
+            -0.02262054570019245,
+            0.001138224033638835,
+            -0.04063516482710838,
+            0.008280059322714806,
+            0.014022313989698887,
+            -0.01598437689244747,
+            0.03205966204404831,
+            0.020969079807400703,
+            -0.01801462098956108,
+            -0.0009682480012997985,
+            0.0034714885987341404,
+            -0.010393633507192135,
+            -0.011067855171859264,
+            -0.019923655316233635,
+            0.02786281518638134,
+            -0.015802564099431038,
+            -0.025908326730132103,
+            0.02454473078250885,
+            -0.01743887923657894,
+            0.015151068568229675,
+            -0.004299115855246782,
+            0.00888610165566206,
+            -0.019120648503303528,
+            0.02284781076014042,
+            -0.023741723969578743,
+            -0.0009128518868237734,
+            -0.020287280902266502,
+            0.0028389315120875835,
+            0.014135947450995445,
+            -0.018272189423441887,
+            -0.02139330841600895,
+            0.019150950014591217,
+            -0.005420294590294361,
+            -0.003931702114641666,
+            0.010014856234192848,
+            0.0030983935575932264,
+            -0.0029715034179389477,
+            -0.004549108445644379,
+            -0.03281721472740173,
+            -0.02915065549314022,
+            0.0033086147159337997,
+            0.011386028490960598,
+            0.015257125720381737,
+            0.027574945241212845,
+            0.03715042024850845,
+            0.006052851676940918,
+            0.0036949669010937214,
+            0.0009327376610599458,
+            -0.020726662129163742,
+            0.042635105550289154,
+            -0.020469093695282936,
+            0.040847279131412506,
+            0.032544493675231934,
+            -0.0197418425232172,
+            -0.023908386006951332,
+            0.0029052174650132656,
+            0.03115059621632099,
+            0.03151422366499901,
+            0.025211378931999207,
+            0.012310243211686611,
+            0.03690800443291664,
+            -0.004359719809144735,
+            -0.032211173325777054,
+            0.015802564099431038,
+            0.022256920114159584,
+            -0.008105821907520294,
+            0.009673957712948322,
+            0.027438584715127945,
+            0.013279912061989307,
+            -0.007757347077131271,
+            0.030150625854730606,
+            0.0012234487803652883,
+            0.02396899089217186,
+            0.02498411200940609,
+            -0.02190844528377056,
+            0.01743887923657894,
+            -0.0001154677156591788,
+            0.006727074272930622,
+            -0.009840618818998337,
+            0.004348356742411852,
+            0.011810258030891418,
+            0.018272189423441887,
+            0.018908534198999405,
+            0.003246116451919079,
+            -0.07060398161411285,
+            0.019726691767573357,
+            0.004507442936301231,
+            -0.02160542458295822,
+            0.00390140013769269,
+            -0.0024866692256182432,
+            0.012704171240329742,
+            0.02105998620390892,
+            0.012461753562092781,
+            -0.01790856383740902,
+            0.0013626492582261562,
+            0.011961768381297588,
+            -0.00595815759152174,
+            0.04730163514614105,
+            0.0019791084341704845,
+            -0.01437078882008791,
+            -0.016105586662888527,
+            0.013726867735385895,
+            0.02525683119893074,
+            -0.02355991117656231,
+            0.006325571332126856,
+            0.005621046293526888,
+            -0.004147605039179325,
+            -0.010742107406258583,
+            -0.030377892777323723,
+            -0.005280147306621075,
+            -0.03436262533068657,
+            0.023150833323597908,
+            0.015491968020796776,
+            0.014764715917408466,
+            -0.025044716894626617,
+            -0.004280176945030689,
+            0.0046400148421525955,
+            -0.021287251263856888,
+            0.003261267440393567,
+            0.01161329448223114,
+            0.0005104016163386405,
+            -0.020984230563044548,
+            0.00959820207208395,
+            0.013469300232827663,
+            0.06848283112049103,
+            0.0007442962378263474,
+            0.029771849513053894,
+            0.0422411784529686,
+            -0.0012243957025930285,
+            -0.006371024530380964,
+            0.0036286809481680393,
+            -0.019953956827521324,
+            0.003473382443189621,
+            0.025878025218844414,
+            -0.03224147483706474,
+            -0.055058982223272324,
+            -0.007545232307165861,
+            -0.04624105989933014,
+            0.001489539397880435,
+            0.014711687341332436,
+            0.023044776171445847,
+            -0.0037782976869493723,
+            -0.034817155450582504,
+            0.016908591613173485,
+            -0.003071879269555211,
+            0.007014944683760405,
+            -0.02593863010406494,
+            -0.0294839795678854,
+            -0.026014383882284164,
+            -0.02339324913918972,
+            -0.012908710166811943,
+            -0.022726602852344513,
+            0.012590537779033184,
+            -0.0015103721525520086,
+            -0.06587684899568558,
+            -0.009780014865100384,
+            0.016166189685463905,
+            0.001200722181238234,
+            0.011939042247831821,
+            0.011083006858825684,
+            -0.007083124481141567,
+            -0.0018275976181030273,
+            -0.022559940814971924,
+            0.03124150261282921,
+            0.005177877843379974,
+            0.03584742918610573,
+            -0.006503595970571041,
+            -0.009128518402576447,
+            0.01073453202843666,
+            0.04527139291167259,
+            0.0017916138749569654,
+            0.010226971469819546,
+            -0.020560000091791153,
+            0.0015065843472257257,
+            -0.02284781076014042,
+            -0.0031135445460677147,
+            -0.008863375522196293,
+            -0.0021211495622992516,
+            0.015893470495939255,
+            0.0019753205124288797,
+            -0.010204244405031204,
+            0.025438643991947174,
+            0.02251448854804039,
+            -0.009605777449905872,
+            -0.015969226136803627,
+            -0.02437806874513626,
+            -0.014545026235282421,
+            -0.0002627763315103948,
+            -0.010007280856370926,
+            -0.022453883662819862,
+            -0.020878173410892487,
+            -0.002566212322562933,
+            -0.02559015527367592,
+            0.022529639303684235,
+            0.01220418605953455,
+            -4.882668508798815e-05,
+            -0.006723286584019661,
+            -0.038544319570064545,
+            -0.003727162955328822,
+            -0.03118089959025383,
+            -0.005374841392040253,
+            0.03287781774997711,
+            -0.03745344281196594,
+            -0.0002231231628684327,
+            0.014491996727883816,
+            -0.01993880607187748,
+            0.0028938541654497385,
+            0.008961857296526432,
+            0.02769615314900875,
+            -0.027408283203840256,
+            0.0001795638381736353,
+            0.006954340264201164,
+            0.016908591613173485,
+            -0.009878496639430523,
+            0.027196168899536133,
+            -0.04275631532073021,
+            0.011037553660571575,
+            -0.037120118737220764,
+            -0.020423639565706253,
+            -0.01523439958691597,
+            0.008363389410078526,
+            -0.005018791649490595,
+            -0.02637801133096218,
+            -0.010454237461090088,
+            0.006026337388902903,
+            -0.00693918950855732,
+            -0.01902974210679531,
+            -0.010620899498462677,
+            0.027438584715127945,
+            0.00931033119559288,
+            0.007977037690579891,
+            -0.02955973520874977,
+            -0.01631770096719265,
+            0.006280118133872747,
+            0.052089374512434006,
+            -0.01584801822900772,
+            0.014938953332602978,
+            -0.006639956030994654,
+            0.02342355251312256,
+            -0.028484009206295013,
+            -0.01872672140598297,
+            0.0315445251762867,
+            0.013946558348834515,
+            0.007064185570925474,
+            0.028953691944479942,
+            0.027196168899536133,
+            0.027105262503027916,
+            -0.04654408246278763,
+            0.009886072017252445,
+            -0.002613559365272522,
+            -0.008757317438721657,
+            0.03402930125594139,
+            0.01916610263288021,
+            0.022150862962007523,
+            0.0009744031121954322,
+            -0.01730252057313919,
+            0.0139844361692667,
+            0.014885924756526947,
+            0.007681591901928186,
+            -0.014242004603147507,
+            -0.02660527639091015,
+            0.003367325058206916,
+            -0.007992188446223736,
+            0.00047276070108637214,
+            0.020726662129163742,
+            -0.009052763693034649,
+            0.030999086797237396,
+            -0.014605630189180374,
+            -0.02593863010406494,
+            0.008916404098272324,
+            -0.02268115058541298,
+            -0.0041741193272173405,
+            0.0004476667381823063,
+            0.010083036497235298,
+            -0.023847781121730804,
+            -0.04669559374451637,
+            0.019408518448472023,
+            0.028393102809786797,
+            0.005579380784183741,
+            0.01328748743981123,
+            -0.035332292318344116,
+            0.008992158807814121,
+            0.012310243211686611,
+            0.011348150670528412,
+            0.0019847899675369263,
+            0.035362593829631805,
+            -0.00028834378463216126,
+            0.012885984033346176,
+            -0.008620957843959332,
+            0.028362801298499107,
+            0.00667025800794363,
+            -0.0323626808822155,
+            -0.0378473699092865,
+            -0.01513591781258583,
+            0.015257125720381737,
+            0.006692984607070684,
+            -0.011840560473501682,
+            -0.02278720773756504,
+            -0.038817036896944046,
+            0.015787413343787193,
+            -0.012916285544633865,
+            -0.038271598517894745,
+            -0.010158791206777096,
+            -0.009742137044668198,
+            0.016635872423648834,
+            -0.006916462909430265,
+            -0.016802534461021423,
+            -0.016090434044599533,
+            0.005821797996759415,
+            7.161247049225494e-05,
+            0.0244083721190691,
+            -0.001251857029274106,
+            -0.006855858489871025,
+            -0.016393456608057022,
+            -0.01188601367175579,
+            -0.0035074723418802023,
+            0.03602924197912216,
+            -0.014219277538359165,
+            -0.02383263036608696,
+            0.014984406530857086,
+            0.02657497487962246,
+            0.011636020615696907,
+            0.03227177634835243,
+            0.007181606721132994,
+            -0.014022313989698887,
+            0.003016956616193056,
+            -0.033029329031705856,
+            -0.0046400148421525955,
+            0.005196816753596067,
+            -0.006427840795367956,
+            -0.015651054680347443,
+            -0.01815097965300083,
+            -0.015832867473363876,
+            0.030756669119000435,
+            -0.02487805485725403,
+            0.0218023881316185,
+            0.030347591266036034,
+            0.003174148965626955,
+            -0.02915065549314022,
+            0.013598084449768066,
+            -0.0072005451656877995,
+            0.01862066239118576,
+            0.0019014590652659535,
+            0.028014326468110085,
+            -0.007461901288479567,
+            -0.004818039946258068,
+            -0.02501441352069378,
+            -0.03696860745549202,
+            -0.023484155535697937,
+            -0.008817922323942184,
+            0.0027537066489458084,
+            0.020984230563044548,
+            -0.0375746488571167,
+            -0.017757052555680275,
+            0.009855770505964756,
+            0.015757111832499504,
+            -0.03027183562517166,
+            -0.0025908327661454678,
+            -0.010158791206777096,
+            -0.0004055278259329498,
+            0.005140000022947788,
+            -0.02227207086980343,
+            -0.008977008052170277,
+            -0.015726808458566666,
+            -0.0010435298318043351,
+            0.006166485138237476,
+            -0.0007409819518215954,
+            -0.05805889517068863,
+            -0.013969285413622856,
+            0.018635815009474754,
+            0.0026400736533105373,
+            0.008908828720450401,
+            0.015938924625515938,
+            -0.0111739132553339,
+            0.0031533162109553814,
+            -0.026756787672638893,
+            0.023893235251307487,
+            -0.009492144919931889,
+            0.009355784393846989,
+            -0.013742019422352314,
+            0.012514783069491386,
+            0.014014738611876965,
+            0.004795313347131014,
+            0.0010681503918021917,
+            -0.0054051438346505165,
+            0.01601468026638031,
+            0.028377952054142952,
+            -0.023756874725222588,
+            0.011454207822680473,
+            -0.0002514130319468677,
+            -0.0032158142421394587,
+            -0.029196109622716904,
+            0.00509454682469368,
+            0.003931702114641666,
+            -0.005109698045998812,
+            0.016651025041937828,
+            -0.036544378846883774,
+            0.01662072166800499,
+            0.02177208475768566,
+            0.032544493675231934,
+            0.014832896180450916,
+            -0.017287369817495346,
+            -0.011658747680485249,
+            -0.012385998852550983,
+            -0.002624922664836049,
+            0.022181164473295212,
+            0.012120855040848255,
+            -0.01815097965300083,
+            -0.024287162348628044,
+            0.011249667964875698,
+            0.0005283934879116714,
+            -0.01203752402216196,
+            0.010795135982334614,
+            -0.013166278600692749,
+            0.01349202636629343,
+            0.012711746618151665,
+            0.00866641104221344,
+            0.00888610165566206,
+            0.022862963378429413,
+            -0.011878437362611294,
+            -0.017711598426103592,
+            0.00656041270121932,
+            -0.008916404098272324,
+            -0.0034089903347194195,
+            -0.012598113156855106,
+            0.010795135982334614,
+            0.014878349378705025,
+            0.06490717828273773,
+            -0.013143551535904408,
+            0.0009526234352961183,
+            -0.020438792183995247,
+            -0.031938452273607254,
+            -0.0017944546416401863,
+            -0.020908474922180176,
+            0.0111739132553339,
+            0.008370965719223022,
+            0.01598437689244747,
+            0.012022373266518116,
+            -0.0021912232041358948,
+            0.00795431062579155,
+            0.019317612051963806,
+            -0.010423935018479824,
+            7.255941454786807e-05,
+            -0.006503595970571041,
+            0.0013683309080079198,
+            0.0035851215943694115,
+            -0.011719351634383202,
+            -0.0026949963066726923,
+            0.035695917904376984,
+            0.01068150345236063,
+            0.011651171371340752,
+            -0.03445352986454964,
+            0.0020435003098100424,
+            0.020150920376181602,
+            0.007602048572152853,
+            -0.008302785456180573,
+            -0.01563590206205845,
+            0.01706010289490223,
+            -0.015317730605602264,
+            -0.0022499337792396545,
+            0.002888172399252653,
+            -0.010151215828955173,
+            0.04581683129072189,
+            -0.019787294790148735,
+            -0.0011742077767848969,
+            0.031847547739744186,
+            0.0030207443051040173,
+            -0.0040983641520142555,
+            0.005408931523561478,
+            -0.003473382443189621,
+            -0.0069013116881251335,
+            -0.02139330841600895,
+            0.0031381649896502495,
+            0.00667025800794363,
+            -0.019287310540676117,
+            -0.0072005451656877995,
+            -0.007719469256699085,
+            0.006897523999214172,
+            0.023105379194021225,
+            -0.009492144919931889,
+            -0.00016145357221830636,
+            -0.008575504645705223,
+            0.009848194196820259,
+            -0.0011912527261301875,
+            -0.044301725924015045,
+            -0.006147546228021383,
+            0.021181194111704826,
+            0.000973456131760031,
+            0.0005757405888289213,
+            -0.007450537756085396,
+            0.005007428117096424,
+            -0.017423728480935097,
+            -0.008454295806586742,
+            0.02369627170264721,
+            0.01736312545835972,
+            -0.009113367646932602,
+            0.010863316245377064,
+            -0.0035680767614394426,
+            0.00433320552110672,
+            -0.020014561712741852,
+            -0.01095422264188528,
+            0.003462019143626094,
+            0.015272277407348156,
+            -0.0009819786064326763,
+            -0.029029447585344315,
+            -0.0012272365856915712,
+            -0.00024194362049456686,
+            0.02918095886707306,
+            0.01486319862306118,
+            -0.008961857296526432,
+            -0.021211495622992516,
+            -0.013242034241557121,
+            -0.001237652963027358,
+            -0.015726808458566666,
+            -0.009620928205549717,
+            0.011855711229145527,
+            0.0009137988090515137,
+            0.008992158807814121,
+            -0.018135828897356987,
+            -0.027347678318619728,
+            -0.016166189685463905,
+            0.0018465365283191204,
+            -0.012870832346379757,
+            0.003340810537338257,
+            0.015514694154262543,
+            -0.03430201858282089,
+            -0.015060162171721458,
+            -0.03805948421359062,
+            0.008280059322714806,
+            -0.0029658216517418623,
+            -0.02024182677268982,
+            -0.023605365306138992,
+            0.009484568610787392,
+            0.019181253388524055,
+            -0.0015548784285783768,
+            0.019847899675369263,
+            -0.000889651826582849,
+            0.020060013979673386,
+            -0.0073141781613230705,
+            0.0031703610438853502,
+            -0.013007191941142082,
+            0.0030245319940149784,
+            0.008333087898790836,
+            0.02890823967754841,
+            -0.023075077682733536,
+            -0.006503595970571041,
+            -0.004685468040406704,
+            0.004746072459965944,
+            0.012658718042075634,
+            0.014613205567002296,
+            0.020741812884807587,
+            -0.008272483013570309,
+            -0.0027612822595983744,
+            0.012484480626881123,
+            -0.003799130441620946,
+            0.007295239716768265,
+            0.002390081062912941,
+            0.0019677451346069574,
+            0.004162756260484457,
+            0.008477022871375084,
+            0.0053710537031292915,
+            0.027650700882077217,
+            -0.0028275682125240564,
+            0.0031703610438853502,
+            0.024620486423373222,
+            0.02322658896446228,
+            -0.018878230825066566,
+            -0.028074929490685463,
+            0.002024561632424593,
+            0.050695475190877914,
+            0.007245998363941908,
+            -0.006810405291616917,
+            -0.03927157074213028,
+            -0.0031760428100824356,
+            0.028635520488023758,
+            0.023650817573070526,
+            0.01584801822900772,
+            -0.010567869991064072,
+            -0.012098127976059914,
+            0.02772645466029644,
+            0.013295062817633152,
+            0.02616589516401291,
+            -0.025302285328507423,
+            0.010704229585826397,
+            -0.03869583085179329,
+            0.0449683703482151,
+            0.004439263138920069,
+            -0.014476845972239971,
+            0.0011884119594469666,
+            0.010386057198047638,
+            -0.004537744913250208,
+            0.006431628484278917,
+            -5.8059369621332735e-05,
+            -0.0008086882880888879,
+            0.011923890560865402,
+            0.012007221579551697,
+            0.004265025723725557,
+            -0.02399929240345955,
+            0.007359631359577179,
+            0.004272601567208767,
+            0.03005971945822239,
+            0.005431658122688532,
+            -0.0013863227795809507,
+            0.012385998852550983,
+            -0.005810434930026531,
+            0.008855799213051796,
+            -0.010719381272792816,
+            0.020469093695282936,
+            -0.02918095886707306,
+            1.0986003871948924e-05,
+            0.02051454596221447,
+            -0.0015785519499331713,
+            0.0038199631962925196,
+            -0.023741723969578743,
+            0.01186328660696745,
+            -0.052968136966228485,
+            0.021166043356060982,
+            -0.018741872161626816,
+            0.021741783246397972,
+            0.019590331241488457,
+            -0.009734561666846275,
+            -0.012052674777805805,
+            0.03545350208878517,
+            0.0026760573964565992,
+            -0.014264730736613274,
+            -0.010643625631928444,
+            -0.017999470233917236,
+            -0.02830219641327858,
+            0.041968461126089096,
+            -0.0006117243901826441,
+            0.0028275682125240564,
+            0.0024222771171480417,
+            -0.010666352696716785,
+            -0.008923979476094246,
+            0.001954487757757306,
+            0.020499395206570625,
+            -0.03369597718119621,
+            -0.011128460057079792,
+            -0.008689138107001781,
+            -0.02566591091454029,
+            -0.00020300064352340996,
+            -0.011961768381297588,
+            0.026211349293589592,
+            0.01240872498601675,
+            -0.03696860745549202,
+            0.003780191531404853,
+            0.023650817573070526,
+            0.001603172393515706,
+            0.018120678141713142,
+            0.002130619017407298,
+            0.0010795136913657188,
+            0.009742137044668198,
+            -0.010476963594555855,
+            0.026559824123978615,
+            -0.009204274043440819,
+            0.008658835664391518,
+            -0.001950700068846345,
+            -0.017287369817495346,
+            0.017969166859984398,
+            0.004083212930709124,
+            0.024120500311255455,
+            0.011954193003475666,
+            0.011605718173086643,
+            0.01112088467925787,
+            -0.019878201186656952,
+            0.020090317353606224,
+            -0.0018891488434746861,
+            0.009075489826500416,
+            0.024726543575525284,
+            -0.005863463506102562,
+            0.020832719281315804,
+            0.015257125720381737,
+            -0.00020229043730068952,
+            0.011295121163129807,
+            -0.020120618864893913,
+            -0.011779955588281155,
+            0.0085679292678833,
+            -0.009605777449905872,
+            -0.010325453244149685,
+            0.009901223704218864,
+            -0.010105762630701065,
+            -0.015052586793899536,
+            -0.0071247899904847145,
+            0.016635872423648834,
+            0.0028976418543606997,
+            -0.02610529027879238,
+            -1.7326075976598077e-05,
+            0.008901252411305904,
+            0.006166485138237476,
+            0.018741872161626816,
+            0.011317848227918148,
+            -0.006692984607070684,
+            0.010363331064581871,
+            0.011476934887468815,
+            0.0053104497492313385,
+            -0.005079395603388548,
+            -0.013431422412395477,
+            -0.0024241709616035223,
+            0.018090376630425453,
+            0.009810317307710648,
+            -0.01922670565545559,
+            0.004564259201288223,
+            0.0008877579239197075,
+            -0.01420412678271532,
+            -0.019347915425896645,
+            -0.0038483713287860155,
+            -0.004170331638306379,
+            0.0027915844693779945,
+            0.0509984977543354,
+            -0.0006216673064045608,
+            0.007893706671893597,
+            -0.025741664692759514,
+            -0.006299056578427553,
+            -0.00027556007262319326,
+            -0.0036097420379519463,
+            0.01272689737379551,
+            -0.01824188604950905,
+            -0.03724132478237152,
+            0.012711746618151665,
+            -0.0012584856012836099,
+            0.014923802576959133,
+            -0.00297907879576087,
+            0.014226853847503662,
+            -0.017347972840070724,
+            0.01916610263288021,
+            -0.0011117096291854978,
+            -0.016029831022024155,
+            0.0029374135192483664,
+            -0.011143610812723637,
+            -0.021317552775144577,
+            0.005916492082178593,
+            -0.006848283112049103,
+            -0.03896854817867279,
+            0.0029885482508689165,
+            -0.010423935018479824,
+            0.0011798894265666604,
+            -0.003912763670086861,
+            0.020166072994470596,
+            -0.0061513339169323444,
+            0.0008697660523466766,
+            0.02177208475768566,
+            0.0017026013229042292,
+            0.01601468026638031,
+            -0.014499573037028313,
+            0.010999675840139389,
+            0.012598113156855106,
+            0.02210540883243084,
+            -0.032180868089199066,
+            -0.01393140759319067,
+            0.014211702160537243,
+            0.0026116655208170414,
+            -0.005382417235523462,
+            0.020635755732655525,
+            -0.01294658798724413,
+            0.019650936126708984,
+            -0.0005577487172558904,
+            -0.011704200878739357,
+            -0.03524138405919075,
+            -0.016529815271496773,
+            -3.343888238305226e-05,
+            -0.008439145050942898,
+            0.025317436084151268,
+            0.01760554127395153,
+            -0.0108330138027668,
+            0.026468917727470398,
+            0.0037574649322777987,
+            0.014598054811358452,
+            -0.014257155358791351,
+            -0.005999823100864887,
+            -0.0003773563075810671,
+            -0.0119314668700099,
+            0.015264702029526234,
+            -0.03512017801403999,
+            0.0008342557121068239,
+            -0.00839369185268879,
+            0.005242269951850176,
+            -0.0254992488771677,
+            0.018257036805152893,
+            0.025817420333623886,
+            -0.0031419529113918543,
+            -0.005037730094045401,
+            -0.022802358493208885,
+            0.028968842700123787,
+            0.03049910068511963,
+            0.0024828813038766384,
+            -0.0013825349742546678,
+            -0.00352072948589921,
+            0.0033218718599528074,
+            0.0211205892264843,
+            -0.037998881191015244,
+            -0.009098216891288757,
+            -0.028696123510599136,
+            0.042332086712121964,
+            -0.008098246529698372,
+            -0.013431422412395477,
+            0.01862066239118576,
+            -0.0024715180043131113,
+            0.02355991117656231,
+            -0.004689255729317665,
+            -0.004855917301028967,
+            0.03148392215371132,
+            0.011514811776578426,
+            0.02772645466029644,
+            -0.04015033319592476,
+            0.014704111963510513,
+            -0.006867222022265196,
+            0.004867280833423138,
+            -0.013484450988471508,
+            0.005302873905748129,
+            0.02580226957798004,
+            -0.018060073256492615,
+            0.0068066176027059555,
+            0.030665762722492218,
+            0.00748084019869566,
+            0.002448791405186057,
+            0.0165904201567173,
+            0.003890036838129163,
+            0.010855740867555141,
+            0.000841831264551729,
+            0.0052574207074940205,
+            0.022423582151532173,
+            0.014454119838774204,
+            -0.016029831022024155,
+            -0.015120766125619411,
+            -0.00044790346873924136,
+            -0.004450626205652952,
+            0.005261208396404982,
+            0.013242034241557121,
+            0.0061172437854111195,
+            -0.015681356191635132,
+            0.01052999310195446,
+            0.014158673584461212,
+            0.013128400780260563,
+            -0.0039847311563789845,
+            -0.03412020578980446,
+            -0.02443867363035679,
+            -0.021438762545585632,
+            0.003104075090959668,
+            -0.0008281006012111902,
+            -0.0021211495622992516,
+            0.0017954016802832484,
+            0.017347972840070724,
+            -0.0008352026343345642,
+            0.00379155483096838,
+            0.01686313934624195,
+            -0.01415109820663929,
+            -0.022090258076786995,
+            -0.0035964848939329386,
+            -0.027529491111636162,
+            0.0053710537031292915,
+            0.001844642567448318,
+            -0.00965123064815998,
+            0.008530051447451115,
+            -0.0046059247106313705,
+            -0.007223271764814854,
+            -0.012196609750390053,
+            -0.02007516659796238,
+            0.005780132487416267,
+            0.016878290101885796,
+            0.005730891600251198,
+            -0.009272454306483269,
+            -0.003939277958124876,
+            0.011575416661798954,
+            0.017484333366155624,
+            0.017317671328783035,
+            -0.009295180439949036,
+            0.009628504514694214,
+            0.0004022135108243674,
+            0.004670316819101572,
+            -0.01834794320166111,
+            0.0045869858004152775,
+            -0.0027953721582889557,
+            -0.008658835664391518,
+            0.01105270441621542,
+            -0.00031201730598695576,
+            0.01628739945590496,
+            -0.013090522959828377,
+            0.005624833982437849,
+            0.010333028621971607,
+            -0.020590301603078842,
+            -0.0067308624275028706,
+            -0.01692374423146248,
+            0.026484068483114243,
+            -0.00504530593752861,
+            0.00045405857963487506,
+            -0.02978700026869774,
+            0.0033616432920098305,
+            0.000848933297675103,
+            -0.000845619011670351,
+            -0.012764775194227695,
+            0.010749682784080505,
+            0.03242328763008118,
+            0.0005179771687835455,
+            0.0216660276055336,
+            -0.0036154238041490316,
+            0.0007480839849449694,
+            -0.00785582885146141,
+            0.004977126140147448,
+            -0.004916521720588207,
+            -0.009901223704218864,
+            -0.010863316245377064,
+            0.021726632490754128,
+            -0.0033029329497367144,
+            0.0003356908564455807,
+            0.011332998983561993,
+            -0.012067826464772224,
+            0.0014620780711993575,
+            -0.0047687990590929985,
+            0.00971941091120243,
+            0.0061702728271484375,
+            0.009113367646932602,
+            -0.01176480483263731,
+            0.05451354384422302,
+            -0.0106284748762846,
+            0.005537715740501881,
+            -0.00439380994066596,
+            -0.022226616740226746,
+            -0.017317671328783035,
+            0.012711746618151665,
+            -0.017544938251376152,
+            0.03375658020377159,
+            0.024150803685188293,
+            0.0006467612111009657,
+            0.010393633507192135,
+            -0.006958128418773413,
+            0.023923536762595177,
+            -0.002602196065708995,
+            0.0010795136913657188,
+            -0.004855917301028967,
+            0.015802564099431038,
+            0.04511988162994385,
+            0.017393426969647408,
+            -0.001152428099885583,
+            0.00829521007835865,
+            0.020575150847434998,
+            0.00814369972795248,
+            -0.0020075165666639805,
+            0.014219277538359165,
+            -0.01024212222546339,
+            0.00900731049478054,
+            -0.004458202049136162,
+            0.027438584715127945,
+            0.00814369972795248,
+            -0.006310420110821724,
+            -0.0036836036015301943,
+            -0.0008209985098801553,
+            0.014173824340105057,
+            0.006950552575290203,
+            0.01963578537106514,
+            -0.00461350055411458,
+            -0.0012575386790558696,
+            0.013870803639292717,
+            0.014491996727883816,
+            -0.00552635220810771,
+            0.001724380999803543,
+            -0.01435563713312149,
+            -0.010204244405031204,
+            0.013575357384979725,
+            0.006086941808462143,
+            0.006310420110821724,
+            -0.014514723792672157,
+            -0.00980274099856615,
+            -0.004670316819101572,
+            0.0015435151290148497,
+            0.015052586793899536,
+            0.011060279794037342,
+            -0.004814252257347107,
+            0.006052851676940918,
+            0.013825350441038609,
+            -0.014014738611876965,
+            0.006147546228021383,
+            0.007208121009171009,
+            0.008855799213051796,
+            0.004787737503647804,
+            -0.010817863047122955,
+            -0.00428396463394165,
+            -0.0063937511295080185,
+            0.0012670081341639161,
+            0.016832837834954262,
+            0.013181429356336594,
+            0.0049581872299313545,
+            0.027529491111636162,
+            -0.00839369185268879,
+            -0.012302667833864689,
+            0.02339324913918972,
+            0.011257244274020195,
+            0.0031703610438853502,
+            0.014651083387434483,
+            -0.012635990977287292,
+            -0.012446602806448936,
+            0.006655106786638498,
+            0.004848341923207045,
+            0.006049063988029957,
+            0.009151245467364788,
+            0.00450365524739027,
+            0.007333117071539164,
+            -0.007098275702446699,
+            -0.007427811156958342,
+            0.0014810169814154506,
+            0.01642375811934471,
+            0.020393338054418564,
+            -0.024968961253762245,
+            -0.0018806264270097017,
+            0.0009052763343788683,
+            0.010878467001020908,
+            -0.003316190093755722,
+            0.012310243211686611,
+            5.65205882594455e-05,
+            -0.0074240234680473804,
+            0.010158791206777096,
+            -0.02119634486734867,
+            -0.012681444175541401,
+            0.0012717428617179394,
+            0.013037494383752346,
+            -0.019590331241488457,
+            0.018226735293865204,
+            -0.008893677033483982,
+            -0.013348091393709183,
+            0.013893529772758484,
+            0.020014561712741852,
+            0.004090788308531046,
+            -0.007738408166915178,
+            -0.021862991154193878,
+            -0.01127239502966404,
+            0.016060132533311844,
+            0.020272130146622658,
+            0.000947888707742095,
+            -0.0006472347304224968,
+            0.010173942893743515,
+            -0.013113250024616718,
+            -0.023605365306138992,
+            -0.005230906419456005,
+            0.010386057198047638,
+            0.01746918261051178,
+            0.018832778558135033,
+            -0.006685409229248762,
+            0.02443867363035679,
+            -0.0059127043932676315,
+            -0.015499543398618698,
+            2.86597842205083e-05,
+            -0.02802947722375393,
+            -0.017423728480935097,
+            0.00013778003631159663,
+            0.00246773031540215,
+            0.008598231710493565,
+            0.0008555619278922677,
+            0.01943882182240486,
+            0.000904329412151128,
+            0.017590390518307686,
+            -0.000659071491099894,
+            -0.002009410411119461,
+            -0.0013995799235999584,
+            -0.0015340456739068031,
+            0.02254479005932808,
+            -0.012166308239102364,
+            -0.01176480483263731,
+            0.022302372381091118,
+            0.006852070800960064,
+            0.014719262719154358,
+            -0.027635548263788223,
+            0.013757170177996159,
+            -0.015499543398618698,
+            0.017423728480935097,
+            -0.03408990427851677,
+            0.006420265417546034,
+            0.01489350013434887,
+            -0.010348179377615452,
+            -0.020499395206570625,
+            -0.029650641605257988,
+            -0.024150803685188293,
+            -0.0038710979279130697,
+            -0.010855740867555141,
+            -0.016847988590598106,
+            0.006121031939983368,
+            -0.005802859086543322,
+            0.018499454483389854,
+            -0.007079336792230606,
+            0.033877789974212646,
+            0.006969491485506296,
+            -0.004056698642671108,
+            0.003835114184767008,
+            0.022469034418463707,
+            -0.006102093029767275,
+            0.02109028771519661,
+            0.004408961161971092,
+            -0.001467759720981121,
+            0.013681414537131786,
+            -0.006552837323397398,
+            -0.002361672930419445,
+            0.005821797996759415,
+            -0.010484539903700352,
+            0.022014502435922623,
+            0.005924067925661802,
+            -0.007708106189966202,
+            -0.0006775368237867951,
+            -0.011454207822680473,
+            0.014726839028298855,
+            0.006530110724270344,
+            0.0022480397019535303,
+            -0.011295121163129807,
+            0.011249667964875698,
+            -0.004859705455601215,
+            0.0040642740204930305,
+            0.009484568610787392,
+            -0.01774190180003643,
+            -0.028862785547971725,
+            0.011416330002248287,
+            0.019120648503303528,
+            0.006382387597113848,
+            -0.0055755930952727795,
+            -0.0025681061670184135,
+            0.00839369185268879,
+            0.010651201009750366,
+            0.022150862962007523,
+            -0.012984465807676315,
+            -8.889415767043829e-05,
+            0.013204156421124935,
+            -0.007170243188738823,
+            -0.01056029461324215,
+            0.00487485621124506,
+            0.023438703268766403,
+            0.0068823727779090405,
+            -0.04327145218849182,
+            -0.018469152972102165,
+            0.01909034699201584,
+            0.007136153522878885,
+            -0.04154422879219055,
+            0.0026381798088550568,
+            -0.003689285134896636,
+            -0.023787178099155426,
+            0.016105586662888527,
+            -0.0011817833874374628,
+            -0.01828734017908573,
+            0.028756728395819664,
+            0.02830219641327858,
+            0.01834794320166111,
+            2.6987840101355687e-05,
+            -0.013332940638065338,
+            -0.0318172425031662,
+            -0.023590214550495148,
+            0.008280059322714806,
+            -0.02660527639091015,
+            -0.0072725131176412106,
+            0.011242092587053776,
+            -0.017454031854867935,
+            0.02972639724612236,
+            -0.008507325313985348,
+            0.026938600465655327,
+            0.011446632444858551,
+            0.001691238023340702,
+            -0.013446573168039322,
+            -0.018802475184202194,
+            -0.01350717805325985,
+            -0.008196728304028511,
+            0.014090493321418762,
+            0.017923714593052864,
+            0.011007251217961311,
+            0.011332998983561993,
+            0.02251448854804039,
+            -0.012158732861280441,
+            0.007969462312757969,
+            -2.888172457460314e-05,
+            -0.008961857296526432,
+            0.0026343921199440956,
+            -0.018393397331237793,
+            0.00868156272917986,
+            0.0062308767810463905,
+            0.005458172410726547,
+            -0.007208121009171009,
+            0.010742107406258583,
+            0.02712041325867176,
+            -0.018847929313778877,
+            0.011401179246604443,
+            -0.031362712383270264,
+            -0.014787442982196808,
+            -0.01056029461324215,
+            0.026862844824790955,
+            0.007757347077131271,
+            -0.015726808458566666,
+            -0.007514929864555597,
+            -0.01545409020036459,
+            0.012885984033346176,
+            0.0032442226074635983,
+            0.005764981731772423,
+            0.015817714855074883,
+            0.021893294528126717,
+            0.01746918261051178,
+            -0.00795431062579155,
+            0.006852070800960064,
+            0.004674104508012533,
+            0.028090080246329308,
+            -0.01777220331132412,
+            0.020802417770028114,
+            0.009954252280294895,
+            -0.003935490269213915,
+            0.005685438401997089,
+            -0.014257155358791351,
+            0.008530051447451115,
+            -0.007325541693717241,
+            0.030514251440763474,
+            0.0033067206386476755,
+            0.014075342565774918,
+            -0.02062060497701168,
+            0.006575563922524452,
+            -0.03172633796930313,
+            0.009022461250424385,
+            0.013704141601920128,
+            0.012976890429854393,
+            -0.007677804213017225,
+            -0.024181105196475983,
+            -0.0119314668700099,
+            0.0057990713976323605,
+            -0.01279507763683796,
+            -0.01046181283891201,
+            -0.0034828518982976675,
+            0.012446602806448936,
+            0.018847929313778877,
+            -0.01848430372774601,
+            -0.019923655316233635,
+            -0.009583051316440105,
+            0.02898399345576763,
+            -0.015787413343787193,
+            -0.001471547526307404,
+            0.02413565292954445,
+            0.021241798996925354,
+            0.015022284351289272,
+            0.002155239460989833,
+            0.020090317353606224,
+            -0.0007111532613635063,
+            -0.0237265732139349,
+            -0.02813553437590599,
+            0.007302815094590187,
+            -0.006912675220519304,
+            0.01943882182240486,
+            -0.0035548193845897913,
+            -0.005658924113959074,
+            -0.0034033088013529778,
+            -0.007503566797822714,
+            -0.0165904201567173,
+            -0.01787826046347618,
+            0.0021647089160978794,
+            -0.008454295806586742,
+            0.005863463506102562,
+            -0.0002795845502987504,
+            -0.013105674646794796,
+            0.013401119969785213,
+            -0.014075342565774918,
+            0.005772557109594345,
+            -0.002653330797329545,
+            0.001962063368409872,
+            0.0035926972050219774,
+            0.009113367646932602,
+            -0.007336904760450125,
+            -0.06381630152463913,
+            -0.013105674646794796,
+            -0.003704436356201768,
+            0.006071790587157011,
+            -0.00019578021601773798,
+            -0.016635872423648834,
+            -0.013196581043303013,
+            0.01706010289490223,
+            0.008863375522196293,
+            -0.013870803639292717,
+            0.016954045742750168,
+            0.009961827658116817,
+            0.003622999181970954,
+            -0.024044746533036232,
+            -0.0008148433989845216,
+            0.008757317438721657,
+            0.032756611704826355,
+            -0.008643684908747673,
+            -0.0003839849086944014,
+            0.0036684523802250624,
+            -0.0033332351595163345,
+            -0.010052734054625034,
+            -0.014582903124392033,
+            -0.011848135851323605,
+            0.011908739805221558,
+            0.01790856383740902,
+            -0.014264730736613274,
+            -0.02007516659796238,
+            -0.0007106798002496362,
+            0.012931437231600285,
+            -0.026590125635266304,
+            0.0007244104635901749,
+            -0.007787649054080248,
+            -0.01052999310195446,
+            0.018302490934729576,
+            -0.019014591351151466,
+            0.005249845329672098,
+            0.00048294031876139343,
+            0.008507325313985348,
+            -0.023544760420918465,
+            -0.024893205612897873,
+            0.0035131541080772877,
+            -0.023847781121730804,
+            0.03612014651298523,
+            -0.021590271964669228,
+            -0.012082977220416069,
+            0.034483831375837326,
+            -0.0037423139438033104,
+            0.01702980138361454,
+            0.0023995505180209875,
+            0.02739313244819641,
+            -0.011439057067036629,
+            -0.00617784820497036,
+            0.012499631382524967,
+            0.009575475007295609,
+            0.0032593735959380865,
+            0.03775646165013313,
+            -0.01257538702338934,
+            -0.013901105150580406,
+            -0.005556654185056686,
+            -0.012954163365066051,
+            0.0006775368237867951,
+            0.014431392773985863,
+            0.028393102809786797,
+            0.004257450345903635,
+            -0.00628769351169467,
+            -0.006143758539110422,
+            -0.0014649189542979002,
+            -0.004242299124598503,
+            -0.00947699323296547,
+            -0.00715130427852273,
+            0.017287369817495346,
+            -0.025241680443286896,
+            0.016711628064513206,
+            -0.006420265417546034,
+            0.021135739982128143,
+            -0.011757229454815388,
+            0.007007369305938482,
+            -0.008961857296526432,
+            0.003492321353405714,
+            0.0023635667748749256,
+            -0.005109698045998812,
+            0.009287605062127113,
+            -0.014772292226552963,
+            0.008454295806586742,
+            0.013742019422352314,
+            0.0011183382011950016,
+            -0.010590597055852413,
+            -0.019272159785032272,
+            0.0022972808219492435,
+            -0.000710206339135766,
+            0.01181783340871334,
+            -0.0047498601488769054,
+            -0.000981031684204936,
+            0.0002921315317507833,
+            -0.014476845972239971,
+            -0.009575475007295609,
+            0.0006050958181731403,
+            0.005302873905748129,
+            -0.008961857296526432,
+            0.010378481820225716,
+            -0.005693014245480299,
+            -0.00034160923678427935,
+            0.02627195231616497,
+            0.006526322569698095,
+            -0.007632351014763117,
+            -0.014098069630563259,
+            0.01563590206205845,
+            0.015370759181678295,
+            -0.007454325910657644,
+            0.01933276280760765,
+            -0.016196493059396744,
+            -0.0008366230758838356,
+            0.013916256837546825,
+            -0.009219424799084663,
+            0.0017698341980576515,
+            -0.008196728304028511,
+            0.03554440662264824,
+            -0.0008787619881331921,
+            -0.0035415622405707836,
+            -0.0006401326390914619,
+            0.012976890429854393,
+            -0.0026779514737427235,
+            0.001182730309665203,
+            0.01132542360574007,
+            0.005321812815964222,
+            0.0185146052390337,
+            0.00328020635060966,
+            -0.03378688171505928,
+            -0.007780073676258326,
+            -0.0065490491688251495,
+            -0.008090670220553875,
+            0.009022461250424385,
+            0.025196226313710213,
+            0.01990850456058979,
+            0.006408901885151863,
+            0.001881573349237442,
+            0.018423698842525482,
+            0.015757111832499504,
+            -0.01489350013434887,
+            0.016363153234124184,
+            0.02802947722375393,
+            -0.010030006989836693,
+            0.022741753607988358,
+            -0.005253633018583059,
+            -0.008893677033483982,
+            0.008946705609560013,
+            -0.0015378334792330861,
+            -0.017393426969647408,
+            -0.015681356191635132,
+            0.005022579338401556,
+            -0.008999735116958618,
+            -0.0022897052112966776,
+            -0.018605511635541916,
+            0.004670316819101572,
+            0.017105557024478912,
+            -0.004965762607753277,
+            -3.5243989259470254e-05,
+            0.019484274089336395,
+            -0.006965703796595335,
+            -0.016908591613173485,
+            0.006325571332126856,
+            -0.015007133595645428,
+            0.007946735247969627,
+            -0.013666263781487942,
+            0.010492115281522274,
+            -0.006098304875195026,
+            0.018817627802491188,
+            0.005446809343993664,
+            -0.008090670220553875,
+            -0.0008958069374784827,
+            -0.018984289839863777,
+            0.02065090648829937,
+            -0.033089932054281235,
+            0.00839369185268879,
+            -0.026226500049233437,
+            -0.0009781909175217152,
+            0.0008849170990288258,
+            0.014605630189180374,
+            -0.001218714052811265,
+            0.019696388393640518,
+            -0.002751812804490328,
+            -0.020090317353606224,
+            -0.007086912170052528,
+            -0.0010634156642481685,
+            -0.01618134044110775,
+            -0.030923331156373024,
+            -0.004272601567208767,
+            0.0036267871037125587,
+            -0.004711982328444719,
+            0.0021135739516466856,
+            0.017984317615628242,
+            -0.005613470915704966,
+            0.009173971600830555,
+            -0.02690829709172249,
+            -0.007416448090225458,
+            -0.00355103169567883,
+            0.0010776197304949164,
+            -0.011242092587053776,
+            0.0010596278589218855,
+            0.011226941831409931,
+            -0.019044892862439156,
+            0.013461724855005741,
+            -0.021378157660365105,
+            -0.004371083341538906,
+            -0.008507325313985348,
+            -0.00135412672534585,
+            -0.013446573168039322,
+            -0.028120383620262146,
+            0.025029566138982773,
+            -0.00021495578403118998,
+            -0.003528305096551776,
+            -0.011999646201729774,
+            0.023181134834885597,
+            0.002914686920121312,
+            0.027817361056804657,
+            -0.03157482668757439,
+            0.0013427634257823229,
+            0.011552689597010612,
+            0.0025984083767980337,
+            -0.017287369817495346,
+            0.004746072459965944,
+            -0.005234694108366966,
+            0.02210540883243084,
+            -0.0011694730492308736,
+            -0.0005340751959010959,
+            -0.008234606124460697,
+            0.030999086797237396,
+            0.009136094711720943,
+            0.0012944694608449936,
+            -0.013226882554590702,
+            -0.008060368709266186,
+            -0.002759388415142894,
+            -0.0011069749016314745,
+            0.012855681590735912,
+            0.01290113478899002,
+            -0.012105703353881836,
+            0.0013607552973553538,
+            -0.006878585088998079,
+            0.0011694730492308736,
+            -0.002935519441962242,
+            -0.0004966709529981017,
+            -0.01149208564311266,
+            0.003257479751482606,
+            0.016272246837615967,
+            -0.00780280027538538,
+            -0.006480869371443987,
+            0.0007750718505121768,
+            0.009181547909975052,
+            0.012461753562092781,
+            -0.012742049060761929,
+            -0.018226735293865204,
+            -0.011908739805221558,
+            0.016029831022024155,
+            0.010522416792809963,
+            -0.008204303681850433,
+            -0.013234458863735199,
+            -0.02583257108926773,
+            0.01169662456959486,
+            -0.03293842449784279,
+            -0.012242063879966736,
+            0.005821797996759415,
+            0.028559764847159386,
+            0.019999410957098007,
+            0.012226912193000317,
+            0.001332347048446536,
+            0.002950670663267374,
+            0.00023342114582192153,
+            -0.01916610263288021,
+            0.014408666640520096,
+            -0.011143610812723637,
+            -0.018166130408644676,
+            0.0075906855054199696,
+            0.011939042247831821,
+            0.03148392215371132,
+            0.006942977197468281,
+            -0.0010160685051232576,
+            0.01686313934624195,
+            -0.001218714052811265,
+            -0.013582932762801647,
+            0.01420412678271532,
+            -0.024256860837340355,
+            0.016438908874988556,
+            0.02860521711409092,
+            -0.004780162125825882,
+            0.0008531945641152561,
+            -0.020938776433467865,
+            0.00482561532407999,
+            -0.011529963463544846,
+            -0.002177966060116887,
+            -0.003287781961262226,
+            -0.03460504114627838,
+            0.007946735247969627,
+            -0.006825556512922049,
+            0.01046181283891201,
+            -0.00814369972795248,
+            0.006655106786638498,
+            -0.012431452050805092,
+            0.004253662656992674,
+            0.01513591781258583,
+            0.016105586662888527,
+            0.007060397882014513,
+            -0.025211378931999207,
+            -0.009196698665618896,
+            0.008105821907520294,
+            -0.01349202636629343,
+            -0.02295386977493763,
+            -0.006852070800960064,
+            0.009840618818998337,
+            -0.01061332318931818,
+            -0.00488243205472827,
+            -0.028180986642837524,
+            -0.013681414537131786,
+            -0.0014327229000627995,
+            -0.017817657440900803,
+            -0.005556654185056686,
+            -0.003100287402048707,
+            0.01571165770292282,
+            -0.0049733384512364864,
+            -0.006185423582792282,
+            0.004742284305393696,
+            -0.023878084495663643,
+            0.0037423139438033104,
+            -0.008280059322714806,
+            0.0008266801596619189,
+            -0.0032517979852855206,
+            -0.0014762822538614273,
+            -0.007969462312757969,
+            -0.014332910999655724,
+            0.006893736310303211,
+            -0.016469212248921394,
+            0.002954458352178335,
+            -0.015173795633018017,
+            -0.016499513760209084,
+            -0.00682934420183301,
+            0.00775355938822031,
+            0.021575121209025383,
+            0.006541473791003227,
+            0.021484214812517166,
+            0.005469535943120718,
+            0.028680972754955292,
+            0.0003953482082579285,
+            0.003041577059775591,
+            -8.958956641436089e-06,
+            0.003232859307900071,
+            0.019347915425896645,
+            0.006571775767952204,
+            0.018984289839863777,
+            -0.008742166683077812,
+            0.010514841414988041,
+            -0.010355755686759949,
+            0.010014856234192848,
+            0.006333146709948778,
+            -0.006473293993622065,
+            -0.003799130441620946,
+            -0.010348179377615452,
+            -0.0031419529113918543,
+            -0.016544966027140617,
+            0.0027366618160158396,
+            0.00693161366507411,
+            0.0071588801220059395,
+            -0.011582992039620876,
+            -0.005230906419456005,
+            -0.014704111963510513,
+            -0.0064354161731898785,
+            -0.01831764169037342,
+            0.014385939575731754,
+            0.029226411134004593,
+            -0.016332851722836494,
+            -0.01977214403450489,
+            -0.007772498298436403,
+            0.0007296186522580683,
+            -0.006636167876422405,
+            -0.01240872498601675,
+            -0.009151245467364788,
+            -0.035059571266174316,
+            -0.017666146159172058,
+            0.007586897816509008,
+            0.02484775334596634,
+            0.01387837901711464,
+            -0.013628385961055756,
+            -0.0006983695784583688,
+            -0.016984347254037857,
+            -0.010787560604512691,
+            -0.01535560842603445,
+            -0.02054484933614731,
+            -0.02498411200940609,
+            0.0077952248975634575,
+            -0.004033972043544054,
+            0.014931377954781055,
+            0.008113397285342216,
+            0.0032631612848490477,
+            -0.004371083341538906,
+            -0.0024109138175845146,
+            -0.013196581043303013,
+            0.0012925754999741912,
+            -0.004787737503647804,
+            -0.0017291157273575664,
+            0.03554440662264824,
+            -0.008098246529698372,
+            -0.005871038883924484,
+            0.011060279794037342,
+            0.011128460057079792,
+            -0.013764745555818081,
+            0.0006642796797677875,
+            -0.011249667964875698,
+            0.021847840398550034,
+            -0.021620575338602066,
+            0.019257009029388428,
+            0.0014507147716358304,
+            -0.010840589180588722,
+            0.028999146074056625,
+            0.01892368495464325,
+            0.00743159931153059,
+            0.006579351611435413,
+            0.01672677882015705,
+            0.0044316877610981464,
+            0.016196493059396744,
+            -0.0026552248746156693,
+            -0.004632438998669386,
+            0.009295180439949036,
+            0.015681356191635132,
+            -0.000702630786690861,
+            -0.006942977197468281,
+            0.016438908874988556,
+            0.004056698642671108,
+            0.0075414446182549,
+            -0.01356020662933588,
+            -0.008961857296526432,
+            0.001507531269453466,
+            0.006257391534745693,
+            -0.005939218681305647,
+            -0.0026003022212535143,
+            0.005530139897018671,
+            0.003943065647035837,
+            -0.013302638195455074,
+            0.01689344085752964,
+            -0.016166189685463905,
+            0.008742166683077812,
+            -0.0037972365971654654,
+            0.0121360057964921,
+            -0.006799042224884033,
+            0.0013153020991012454,
+            -0.01818128302693367,
+            -0.003869204083457589,
+            0.0038597346283495426,
+            0.012173883616924286,
+            -0.01262841559946537,
+            0.025484098121523857,
+            0.008166425861418247,
+            0.006602078210562468,
+            0.0059467945247888565,
+            -0.012855681590735912,
+            0.027211319655179977,
+            0.005969521123915911,
+            0.009886072017252445,
+            -0.008893677033483982,
+            0.011136035434901714,
+            0.035362593829631805,
+            0.03060515783727169,
+            0.014135947450995445,
+            0.004469565115869045,
+            -0.018999440595507622,
+            0.015120766125619411,
+            0.0018427487229928374,
+            0.0023806116078048944,
+            0.00863610953092575,
+            -0.011348150670528412,
+            0.01305264513939619,
+            -0.009484568610787392,
+            0.015241974964737892,
+            -0.021272100508213043,
+            -0.011749654076993465,
+            -0.014408666640520096,
+            -0.021075136959552765,
+            -0.01430260855704546,
+            0.034180812537670135,
+            -0.002346521709114313,
+            -0.010393633507192135,
+            0.0045112306252121925,
+            0.008477022871375084,
+            -0.006079366430640221,
+            0.0053142374381423,
+            -0.008204303681850433,
+            -0.013749594800174236,
+            -0.01889338344335556,
+            0.006628592498600483,
+            -0.01550711877644062,
+            0.007651289459317923,
+            -0.003304826794192195,
+            0.0037688282318413258,
+            0.007014944683760405,
+            0.018105527386069298,
+            0.00530666159465909,
+            -0.00135412672534585,
+            0.0014033677289262414,
+            0.0016865032957866788,
+            -0.00931033119559288,
+            0.0108330138027668,
+            0.015408637002110481,
+            -0.0011921997647732496,
+            -0.011279970407485962,
+            -0.009136094711720943,
+            -0.042877525091171265,
+            0.0006264019757509232,
+            -0.013128400780260563,
+            0.0064354161731898785,
+            -0.011196639388799667,
+            0.0022234192583709955,
+            0.015863168984651566,
+            -0.0054051438346505165,
+            -0.0010918239131569862,
+            0.02193874679505825,
+            0.013014768250286579,
+            0.021075136959552765,
+            0.02065090648829937,
+            0.014188976027071476,
+            -0.015302578918635845,
+            -0.03469594568014145,
+            0.002221525413915515,
+            -0.010030006989836693,
+            0.013772320933640003,
+            0.004238511435687542,
+            0.016060132533311844,
+            -0.023893235251307487,
+            -0.011878437362611294,
+            -0.006662682630121708,
+            0.014431392773985863,
+            0.01491622719913721,
+            -0.0032271775417029858,
+            0.02690829709172249,
+            0.009280029684305191,
+            0.0020510759204626083,
+            -0.02281750924885273,
+            0.003312402404844761,
+            0.005321812815964222,
+            -0.008370965719223022,
+            -0.016681326553225517,
+            -0.0004552422615233809,
+            -0.0012168202083557844,
+            -0.0007817004225216806,
+            -0.021650876849889755,
+            0.018832778558135033,
+            -0.004018820822238922,
+            0.0037574649322777987,
+            0.013840501196682453,
+            0.01516621932387352,
+            0.017984317615628242,
+            0.008370965719223022,
+            0.001497115008533001,
+            -0.006605865899473429,
+            -0.0007973249885253608,
+            -0.0005828426801599562,
+            -0.007389933802187443,
+            -0.005598319694399834,
+            0.006382387597113848,
+            -0.01134057529270649,
+            0.008810346014797688,
+            0.000896753859706223,
+            0.006113456096500158,
+            0.010605747811496258,
+            -0.004836978856474161,
+            -0.011628445237874985,
+            0.014628356322646141,
+            -0.0010548931313678622,
+            -0.0055755930952727795,
+            -0.011060279794037342,
+            0.003624893259257078,
+            0.013575357384979725,
+            -0.003096499713137746,
+            -0.013499601744115353,
+            0.00910579226911068,
+            0.002009410411119461,
+            -0.027438584715127945,
+            0.015499543398618698,
+            0.010295150801539421,
+            -0.013370818458497524,
+            0.004980913829058409,
+            -0.003371112747117877,
+            -0.03027183562517166,
+            0.004791525658220053,
+            0.002660906407982111,
+            -0.010643625631928444,
+            0.028938541188836098,
+            -0.003821857040748,
+            0.029256712645292282,
+            -0.004980913829058409,
+            -0.006905099377036095,
+            -0.011355726048350334,
+            0.009416389279067516,
+            -0.018681267276406288,
+            -0.0037953427527099848,
+            -0.008727015927433968,
+            -0.00915882084518671,
+            -0.0009838725673034787,
+            -0.005185453221201897,
+            0.006984642706811428,
+            0.005121061112731695,
+            -0.012704171240329742,
+            0.006757376715540886,
+            -0.010257272981107235,
+            -0.00020856392802670598,
+            0.0006718551740050316,
+            -0.008408842608332634,
+            0.013885954394936562,
+            0.005996035411953926,
+            -0.008340663276612759,
+            -0.0005032995832152665,
+            0.027817361056804657,
+            0.05051366239786148,
+            -0.018863080069422722,
+            0.0009270560112781823,
+            0.01452229917049408,
+            -0.0028824908658862114,
+            -0.0021590273827314377,
+            -0.011976920068264008,
+            0.005821797996759415,
+            -0.019423669204115868,
+            0.013317788951098919,
+            0.006473293993622065,
+            -0.008833073079586029,
+            0.003202557098120451,
+            -0.016711628064513206,
+            0.002621134975925088,
+            -0.0018058179412037134,
+            -0.018681267276406288,
+            -0.015529844909906387,
+            -0.012060250155627728,
+            -0.024181105196475983,
+            0.0004235197266098112,
+            -0.0008546149474568665,
+            -0.013870803639292717,
+            0.006579351611435413,
+            0.008431569673120975,
+            0.032483890652656555,
+            0.0011912527261301875,
+            0.010128489695489407,
+            -2.1513333194889128e-05,
+            -0.0010416359873488545,
+            -0.005988460034132004,
+            -0.01523439958691597,
+            0.00015387803432531655,
+            -0.002342734020203352,
+            0.022590242326259613,
+            0.005549078807234764,
+            0.017560089007019997,
+            0.0017887729918584228,
+            -0.016514664515852928,
+            -0.00596573343500495,
+            -0.007408872712403536,
+            -0.018060073256492615,
+            0.0062649669125676155,
+            -0.005473323632031679,
+            0.009605777449905872,
+            -0.015863168984651566,
+            -0.02731737680733204,
+            0.00792400911450386,
+            0.012234487570822239,
+            0.028514310717582703,
+            -0.0035623949952423573,
+            0.001716805505566299,
+            -0.005533927585929632,
+            0.016408607363700867,
+            0.032180868089199066,
+            -0.0008550884085707366,
+            0.006950552575290203,
+            -0.019999410957098007,
+            0.0017404790269210935,
+            0.013704141601920128,
+            -0.008401267230510712,
+            -0.01821158453822136,
+            0.003939277958124876,
+            -0.020635755732655525,
+            0.004583198111504316,
+            -0.020393338054418564,
+            0.030408194288611412,
+            -0.0023522034753113985,
+            0.008105821907520294,
+            0.004310478921979666,
+            0.01760554127395153,
+            -0.0031646795105189085,
+            0.010545143857598305,
+            0.013916256837546825,
+            0.014908651821315289,
+            0.0009980766335502267,
+            -0.011189064010977745,
+            -0.03790797293186188,
+            0.001859793672338128,
+            0.00817400123924017,
+            0.018635815009474754,
+            -0.01980244740843773,
+            0.011681473813951015,
+            -0.01296931505203247,
+            -0.020363036543130875,
+            -0.01003758329898119,
+            0.011575416661798954,
+            -0.006139970384538174,
+            0.03378688171505928,
+            -0.005874827038496733,
+            -0.008204303681850433,
+            0.005136212334036827,
+            0.010454237461090088,
+            0.0066778333857655525,
+            0.014704111963510513,
+            0.0326354019343853,
+            0.010196669027209282,
+            0.014173824340105057,
+            -0.004859705455601215,
+            0.02399929240345955,
+            -0.003931702114641666,
+            0.003954428713768721,
+            0.00509454682469368,
+            -0.022832660004496574,
+            -0.0040983641520142555,
+            -0.003931702114641666,
+            -0.016711628064513206,
+            -0.007458113599568605,
+            0.0032366469968110323,
+            0.008802770636975765,
+            -0.014045040123164654,
+            -0.00649980828166008,
+            -0.01350717805325985,
+            0.01218903437256813,
+            0.002730980049818754,
+            0.018302490934729576,
+            -0.029802151024341583,
+            0.009825468063354492,
+            -0.017484333366155624,
+            -0.017044952139258385,
+            0.006511171814054251,
+            -0.0038332203403115273,
+            0.006855858489871025,
+            -0.018272189423441887,
+            -0.01318900566548109,
+            0.012817803770303726,
+            0.0007897494360804558,
+            0.013060221448540688,
+            -0.009825468063354492,
+            0.0009706153068691492,
+            0.006064215209335089,
+            -0.009833043441176414,
+            -0.010961798019707203,
+            -0.007977037690579891,
+            -0.009681533090770245,
+            -0.008848223835229874,
+            0.005621046293526888,
+            0.01656011864542961,
+            -0.013742019422352314,
+            -0.003685497445985675,
+            0.016135888174176216,
+            0.006416477728635073,
+            -0.008166425861418247,
+            0.004075637552887201,
+            -0.003793448908254504,
+            -9.948806837201118e-05,
+            0.0013740125577896833,
+            0.012439027428627014,
+            -0.0017632056260481477,
+            0.024665940552949905,
+            -0.02654467150568962,
+            0.007598260883241892,
+            -0.0054051438346505165,
+            0.005446809343993664,
+            0.018226735293865204,
+            -0.02366596832871437,
+            -0.013082947582006454,
+            0.006276329979300499,
+            -0.01966608688235283,
+            -0.008060368709266186,
+            0.0054051438346505165,
+            -0.00016405766655225307,
+            -0.019923655316233635,
+            -0.012923861853778362,
+            -0.034786853939294815,
+            -0.02309022843837738,
+            0.024665940552949905,
+            0.008401267230510712,
+            -0.0073672072030603886,
+            0.018060073256492615,
+            -0.018438849598169327,
+            0.0172722190618515,
+            -0.002225313102826476,
+            -0.006613441277295351,
+            0.003935490269213915,
+            0.007674016058444977,
+            0.025362888351082802,
+            0.002653330797329545,
+            0.008264907635748386,
+            -0.0042082094587385654,
+            -0.014931377954781055,
+            -0.006321783177554607,
+            0.01980244740843773,
+            0.02471139281988144,
+            0.018605511635541916,
+            -0.018575210124254227,
+            0.010742107406258583,
+            -0.011749654076993465,
+            0.01818128302693367,
+            -0.006537686102092266,
+            0.00039037675014697015,
+            0.01859036087989807,
+            0.007969462312757969,
+            0.025241680443286896,
+            0.019923655316233635,
+            0.006852070800960064,
+            0.014128371141850948,
+            0.011908739805221558,
+            0.001204509986564517,
+            -0.012832955457270145,
+            -0.01154511421918869,
+            -0.02840825356543064,
+            0.042847223579883575,
+            0.0025434857234358788,
+            0.008317936211824417,
+            -0.027635548263788223,
+            -0.00021389048197306693,
+            -0.0034942151978611946,
+            -0.018832778558135033,
+            0.005939218681305647,
+            0.03260510042309761,
+            0.03284751623868942,
+            0.021438762545585632,
+            -2.888172457460314e-05,
+            0.0012234487803652883,
+            0.0329081192612648,
+            -0.013673839159309864,
+            -0.00937851145863533,
+            -0.01034060399979353,
+            -0.018393397331237793,
+            -0.006852070800960064,
+            -0.0032423287630081177,
+            -0.01031030248850584,
+            -0.0061172437854111195,
+            -0.009272454306483269,
+            0.0238629337400198,
+            0.0001374249259242788,
+            -0.0022594030015170574,
+            0.006605865899473429,
+            -0.020878173410892487,
+            0.011537538841366768,
+            -0.02813553437590599,
+            -0.013431422412395477,
+            -0.008651260286569595,
+            -0.006950552575290203,
+            -0.018802475184202194,
+            0.008643684908747673,
+            -0.006942977197468281,
+            0.021438762545585632,
+            0.002914686920121312,
+            -0.023590214550495148,
+            0.007530081085860729,
+            0.015317730605602264,
+            -0.0003825644962489605,
+            0.014037464745342731,
+            -0.003890036838129163,
+            0.009060339070856571,
+            0.0045112306252121925,
+            0.011681473813951015,
+            0.016954045742750168,
+            0.010651201009750366,
+            0.009030036628246307,
+            0.003437398700043559,
+            -0.001167579204775393,
+            0.01878732442855835,
+            -0.0007163614500313997,
+            -0.03405960276722908,
+            2.9236827685963362e-05,
+            -0.014628356322646141,
+            0.029499130323529243,
+            -0.009795165620744228,
+            0.009514871053397655,
+            0.008105821907520294,
+            0.007098275702446699,
+            -0.0067119235172867775,
+            0.007916433736681938,
+            -0.008742166683077812,
+            0.0052763596177101135,
+            0.015832867473363876,
+            -0.0042877523228526115,
+            -0.0014980619307607412,
+            0.007840678095817566,
+            0.00407184986397624,
+            -0.005454384721815586,
+            -0.019711541011929512,
+            -0.012287517078220844,
+            0.00726114958524704,
+            -0.02065090648829937,
+            -0.004367295652627945,
+            0.01161329448223114,
+            -0.005454384721815586,
+            -0.009620928205549717,
+            0.01393140759319067,
+            0.02136300690472126,
+            -0.002382505452260375,
+            0.022893264889717102,
+            -0.01349202636629343,
+            -0.009833043441176414,
+            -0.007984613068401814,
+            -0.004374871030449867,
+            -0.041120000183582306,
+            0.025984082370996475,
+            0.01068150345236063,
+            -0.006871009711176157,
+            0.01895398646593094,
+            0.0022499337792396545,
+            -0.002954458352178335,
+            -0.0011041341349482536,
+            0.00010499217023607343,
+            -0.03281721472740173,
+            0.03660498186945915,
+            -0.007007369305938482,
+            0.0005449649761430919,
+            0.010272424668073654,
+            0.042029064148664474,
+            -0.004590773954987526,
+            -0.005859675817191601,
+            0.01225721463561058,
+            0.005336964037269354,
+            0.031938452273607254,
+            -0.010416359640657902,
+            -0.014340486377477646,
+            -0.01815097965300083,
+            -0.004833190701901913,
+            -0.01939336769282818,
+            -0.0038313264958560467,
+            0.014037464745342731,
+            -0.019650936126708984,
+            -0.011242092587053776,
+            -0.001287840772420168,
+            -0.012537509202957153,
+            -0.0010747789638116956,
+            0.008598231710493565,
+            0.017832808196544647,
+            -0.011348150670528412,
+            -0.004450626205652952,
+            -2.2904154320713133e-05,
+            -0.005003640428185463,
+            0.0185146052390337,
+            0.0020056227222085,
+            -0.01702980138361454,
+            0.01321930717676878,
+            -0.00526499655097723,
+            0.006749800872057676,
+            0.0074429623782634735,
+            0.007939159870147705,
+            0.05145302787423134,
+            0.010317877866327763,
+            -0.001339922659099102,
+            -0.01607528328895569,
+            0.023196285590529442,
+            -0.01749948412179947,
+            -0.014181400649249554,
+            -0.02119634486734867,
+            -0.0029923359397798777,
+            -0.007060397882014513,
+            -0.028014326468110085,
+            -9.137987944995984e-05,
+            0.0016221113037317991,
+            -0.0024980325251817703,
+            -0.010045158676803112,
+            0.0005208179936744273,
+            -0.011529963463544846,
+            -0.006954340264201164,
+            -0.008545203134417534,
+            -0.016044981777668,
+            0.02240842953324318,
+            -0.021817538887262344,
+            -0.008855799213051796,
+            0.011514811776578426,
+            0.03118089959025383,
+            -0.005011215806007385,
+            -0.014173824340105057,
+            -0.019817598164081573,
+            0.0025643182452768087,
+            -0.01763584464788437,
+            0.012181458994746208,
+            -0.01225721463561058,
+            0.015045011416077614,
+            -0.03142331540584564,
+            -0.006079366430640221,
+            -0.008992158807814121,
+            0.012840530835092068,
+            0.019817598164081573,
+            -0.0025093958247452974,
+            0.00606800289824605,
+            -0.010295150801539421,
+            0.030211230739951134,
+            -0.00012286569108255208,
+            -0.018605511635541916,
+            0.00873459130525589,
+            0.007977037690579891,
+            0.012537509202957153,
+            -0.004435475450009108,
+            0.00257757562212646,
+            0.011567841283977032,
+            0.012742049060761929,
+            -0.01634800247848034,
+            -0.003174148965626955,
+            -0.017969166859984398,
+            0.005996035411953926,
+            0.006999793462455273,
+            0.021711481735110283,
+            -0.0013550736475735903,
+            0.018272189423441887,
+            -0.01563590206205845,
+            0.00019412307301536202,
+            -0.011749654076993465,
+            -0.0082270298153162,
+            -0.0028237802907824516,
+            -0.003958216868340969,
+            0.006685409229248762,
+            -0.028211290016770363,
+            -0.01274962443858385,
+            0.005287723150104284,
+            -0.004992276895791292,
+            -0.03405960276722908,
+            0.0053710537031292915,
+            -0.02725677192211151,
+            0.013128400780260563,
+            -0.002397656673565507,
+            -0.017847958952188492,
+            -0.00030846628942526877,
+            0.026332557201385498,
+            -0.003677921835333109,
+            -0.012537509202957153,
+            0.015544996596872807,
+            -0.006757376715540886,
+            -0.02119634486734867,
+            -0.005420294590294361,
+            0.014658658765256405,
+            -0.0010719380807131529,
+            -0.012393574230372906,
+            -0.029635490849614143,
+            0.012992041185498238,
+            0.0026874206960201263,
+            -0.007386146113276482,
+            0.010893618687987328,
+            0.025953780859708786,
+            0.008151275105774403,
+            -0.005924067925661802,
+            0.0015757111832499504,
+            0.0049581872299313545,
+            0.00300938100554049,
+            0.020287280902266502,
+            0.02728707529604435,
+            -0.008014915511012077,
+            -0.0011145505122840405,
+            -0.014954105019569397,
+            0.015802564099431038,
+            0.0013020449550822377,
+            0.010507266037166119,
+            0.021893294528126717,
+            0.007045246660709381,
+            -0.005738467443734407,
+            0.003905187826603651,
+            -0.023135682567954063,
+            0.01420412678271532,
+            -0.012272365391254425,
+            0.000622140767518431,
+            0.008749742060899734,
+            -0.007283876184374094,
+            0.01437078882008791,
+            -0.0015245762187987566,
+            0.0339989997446537,
+            0.025211378931999207,
+            0.0043218424543738365,
+            0.0007362472242675722,
+            -0.028559764847159386,
+            0.003547244006767869,
+            0.008651260286569595,
+            0.014499573037028313,
+            -0.0076664406806230545,
+            0.006261179223656654,
+            -0.004355932120233774,
+            0.00959820207208395,
+            -0.0065490491688251495,
+            -0.015893470495939255,
+            0.006113456096500158,
+            0.0012689019786193967,
+            0.006795254070311785,
+            0.0023408401757478714,
+            -0.008863375522196293,
+            -0.024362917989492416,
+            -0.025984082370996475,
+            0.021984200924634933,
+            -0.03918066248297691,
+            0.0073293293826282024,
+            0.011234517209231853,
+            0.03324144333600998,
+            -0.008158850483596325,
+            -0.004980913829058409,
+            0.006276329979300499,
+            -0.016711628064513206,
+            -0.004147605039179325,
+            0.008204303681850433,
+            0.012385998852550983,
+            -0.002420383272692561,
+            0.00974971242249012,
+            0.01318900566548109,
+            -0.012378423474729061,
+            0.03660498186945915,
+            0.016514664515852928,
+            0.01467380952090025,
+            -0.004007457755506039,
+            0.01312082540243864,
+            0.0066172294318675995,
+            0.019044892862439156,
+            0.025165924802422523,
+            -0.017454031854867935,
+            -0.002365460619330406,
+            -0.02292356640100479,
+            0.024165954440832138,
+            0.00726114958524704,
+            0.004556683823466301,
+            -0.008310360834002495,
+            0.014870774000883102,
+            0.0017954016802832484,
+            0.0028199926018714905,
+            -0.006696772295981646,
+            -0.010173942893743515,
+            0.019241856411099434,
+            0.0034317169338464737,
+            0.014870774000883102,
+            4.8501173296244815e-05,
+            -0.005503625608980656,
+            0.006052851676940918,
+            0.014832896180450916,
+            -0.023590214550495148,
+            -0.00203592493198812,
+            -0.019484274089336395,
+            -0.03202936053276062,
+            0.009022461250424385,
+            0.0016135887708514929,
+            -0.007052822504192591,
+            0.03230207785964012,
+            -0.010083036497235298,
+            -0.019787294790148735,
+            -0.008067944087088108,
+            -0.013113250024616718,
+            -0.010469388216733932,
+            0.0008271536789834499,
+            0.003908975515514612,
+            -0.00012428610352799296,
+            -0.009317907504737377,
+            -0.005749830510467291,
+            0.01990850456058979,
+            -0.0030358952935785055,
+            -0.00753386877477169,
+            0.00868156272917986
+          ],
+          "embedding": [
+            0.0040707582550429606,
+            -0.010230625151989194,
+            -0.021520242101903186,
+            -0.010115756270640508,
+            0.026738734586159035,
+            0.01796370233223267,
+            -0.012245199029994153,
+            0.06506924494147726,
+            -0.0014087587222033442,
+            0.009300068719271429,
+            0.003787396973047584,
+            0.002818772639754174,
+            0.022129077062176306,
+            -0.021526709808445683,
+            -0.0034388135367225824,
+            0.02763761747879405,
+            0.013210013805395276,
+            0.025255872672296602,
+            0.00029011480909986703,
+            -0.05438974762290585,
+            0.016671640187490865,
+            -0.016426252259893328,
+            -0.056591180651012554,
+            0.009335648425523075,
+            0.006141624459418827,
+            0.02568943956694641,
+            0.019377132054042116,
+            0.017946464337078746,
+            -0.057184776985927686,
+            0.03208527139096687,
+            0.030132392063755547,
+            -0.003846032371791839,
+            -0.008376524427531415,
+            -0.0010513963058713435,
+            -0.02734868388724371,
+            -0.014879400370545479,
+            0.04383920959748825,
+            0.02563613659516289,
+            -0.020117467074438997,
+            0.0398016699155716,
+            0.029896497884197775,
+            -0.0004195844530067313,
+            -0.023302720788253625,
+            -0.001168337916577517,
+            0.009149021369661628,
+            -0.02508145721900992,
+            -0.004307594138480351,
+            0.00550940415564303,
+            0.014016998918718476,
+            -0.028607379616884317,
+            -0.0018317656587526954,
+            -0.01973464940415756,
+            0.03762738392056255,
+            0.015358106058583114,
+            -0.03931907682104936,
+            0.011519637298832444,
+            0.004482080558226775,
+            0.0747867063383242,
+            0.05550679018263077,
+            0.023243129601931176,
+            0.02671631686665238,
+            -0.009686146566573536,
+            -0.04088582189725495,
+            -0.0065344623594924085,
+            0.019617295999325712,
+            -0.004722087402575777,
+            -0.002871844114437583,
+            0.003935087265300519,
+            0.03229890413393201,
+            -0.02813577047904649,
+            -0.05124172132597234,
+            0.03325235407841968,
+            -0.008687655557217192,
+            -0.011027174838246948,
+            -0.003653553063547375,
+            -0.02350737945182988,
+            0.016223087939183468,
+            0.03789252356462714,
+            0.0034350623378603083,
+            0.00195158657888739,
+            0.011789712862232108,
+            -0.0004435439660769551,
+            -0.0037330456918125894,
+            0.0031552713243866858,
+            0.04465432992298031,
+            -0.0032465421571915712,
+            -0.0075366743861922745,
+            -0.059958752731410256,
+            0.016784616861008125,
+            -0.028734506065772967,
+            -0.03035575736273943,
+            0.02680390892611741,
+            0.02338368719038086,
+            -0.0030658375437784277,
+            -0.03754896050794851,
+            0.07005659523248504,
+            0.0020711575553570695,
+            0.013034425083076923,
+            -0.02316832262132332,
+            0.021332142030732894,
+            -0.0318756233020585,
+            0.02270782509507388,
+            0.01570810309454529,
+            0.004788124787612736,
+            0.026727250200142456,
+            -0.022380142778058276,
+            -0.008593910379436904,
+            0.014439754382387034,
+            0.015007697617383174,
+            0.07662531963612977,
+            0.011195998137723795,
+            -0.010809283538470098,
+            -0.042227822120219305,
+            0.003824320933275175,
+            -0.03797629427607094,
+            -0.02347390684453348,
+            -0.0526876440280276,
+            -0.017799861680287913,
+            0.01701482796266917,
+            0.002730395144917155,
+            0.03789451759899825,
+            0.038338545350313946,
+            0.028463789278406586,
+            0.015494583705011921,
+            -0.05700237266350479,
+            0.047895170359075166,
+            -0.0064406894534929406,
+            -0.0007954721295065988,
+            0.012543029839729358,
+            -0.004703334868741216,
+            -0.0060569679284068,
+            0.012324901254711192,
+            -0.019883935809784068,
+            -0.04253565942154215,
+            0.02164409276211115,
+            -0.0075367555415808635,
+            0.03617159147722157,
+            0.07567745988686005,
+            -0.04225977020036783,
+            -0.022548837589622003,
+            0.029811602145850144,
+            -0.02175792196819533,
+            0.02069861458115363,
+            -0.00941331829664397,
+            -0.035479639885943264,
+            0.020352552580119105,
+            0.003678588588550461,
+            0.06776105274336729,
+            -0.02346047721661687,
+            0.02721075076349443,
+            -0.029386781707644253,
+            0.023530831921117993,
+            -0.02077660533100372,
+            0.020022475323372906,
+            0.024195118676513715,
+            -0.028378518720494666,
+            -0.012900969023913307,
+            0.019264089970719624,
+            -0.0374983624629265,
+            -0.019726171557384355,
+            0.00979636085642416,
+            -0.014479528435832063,
+            -0.014533384583777662,
+            0.005904623758767224,
+            0.010344475500263744,
+            0.05659646358026115,
+            0.04163866113482093,
+            0.007152494819169405,
+            -0.07584500130974763,
+            0.05471858597878099,
+            0.03580211013427615,
+            0.044161429444475306,
+            0.02258180566046248,
+            0.00980779221334715,
+            0.0011193505603397395,
+            0.018086566768892507,
+            0.015605066702191871,
+            -0.00954727399396313,
+            -0.027950079146919375,
+            0.03028514025485407,
+            -0.006351669761257431,
+            0.005095804801686436,
+            0.009620040085743162,
+            0.019748001227987134,
+            0.033709535059510916,
+            0.008203559956940196,
+            0.007389752602218563,
+            0.04197771878351742,
+            -0.02067363758471765,
+            0.007543739188471913,
+            0.011207941261799052,
+            0.013964827251172813,
+            0.03839562075248879,
+            0.0009775244543292637,
+            0.006469507611492216,
+            -0.022808235395746654,
+            -0.024795004398633292,
+            0.006249024864814684,
+            0.011292877109066925,
+            0.01596839619826819,
+            -0.0014105396434174883,
+            -0.04152075438277094,
+            0.008267788411079337,
+            -0.028834191212891387,
+            0.030940526405027906,
+            -0.0018271056434014425,
+            0.006185475047820603,
+            -0.007920574674727123,
+            0.012907560753043616,
+            -0.015165704767187252,
+            -0.021962244496450727,
+            0.0037071053072833147,
+            -0.013508376064336042,
+            -0.015178823540513864,
+            -0.02518557053061311,
+            0.011693256916760944,
+            -0.01716015674592331,
+            0.02048897513447166,
+            -0.003412078966361745,
+            -0.001807298511378529,
+            -0.0012575655363779904,
+            0.03116103961636643,
+            0.011668533361641842,
+            -0.011637010548030312,
+            -0.040260195687925236,
+            -0.011129376691522074,
+            -0.0034493180842871908,
+            -0.02098244789711689,
+            0.026658743116016938,
+            -0.01923736325953796,
+            -0.009723937181300192,
+            -0.013134679989565051,
+            -0.011974548914818263,
+            -0.004028712301052159,
+            0.019628329175867303,
+            -0.028192034966932684,
+            -0.012638520793012082,
+            -0.011568652932013111,
+            -0.0025919910519445737,
+            0.012497908463405111,
+            0.015113186712897145,
+            0.014880607940212641,
+            -0.03059227859452334,
+            -0.021899716217105084,
+            -0.0008513506358489523,
+            0.011985931118464679,
+            -0.01321584595585435,
+            -0.011735288366213981,
+            0.010710683541435907,
+            0.01961722172525993,
+            -0.023571946726192585,
+            -0.03364281880125354,
+            0.0014460076743756465,
+            0.016563167828467187,
+            -0.02891589059260944,
+            0.04595166331401113,
+            0.034271553243501465,
+            -0.027557971577291153,
+            -0.007138821520412525,
+            -0.039508407807651844,
+            0.06814329636830853,
+            0.00022136123564675344,
+            -0.039718016705753735,
+            0.0503457594963753,
+            -0.003580581891576583,
+            0.013820796398521455,
+            0.02054050419088757,
+            0.035682661890720495,
+            -0.0013220739026852943,
+            -0.0052759394697328675,
+            0.04733018719162478,
+            -0.0187845855131196,
+            -0.03982699468608443,
+            -0.023905460465070062,
+            0.002631233133039351,
+            -0.03090995554989513,
+            -0.012617612051488022,
+            -0.013605787969473513,
+            0.039672783793764745,
+            0.008669582242097126,
+            0.009974277556736362,
+            0.005535636914843583,
+            0.014180668493093584,
+            0.012042659947850441,
+            0.011870115226306776,
+            0.00451259454348581,
+            -0.014023048592193627,
+            -0.007769033996392623,
+            -0.05715987527668358,
+            0.003374564518384728,
+            -0.028801158523506985,
+            -0.00756331369170698,
+            -0.02163054894055572,
+            0.005367181573233089,
+            0.02308705505078254,
+            -0.005438909398653553,
+            -0.00873620909426313,
+            -0.0036530652237719077,
+            0.013920407174296543,
+            -0.01539613379172909,
+            -0.03908211213269472,
+            -0.016810726548844992,
+            0.007460173854386443,
+            -0.007069189426742819,
+            -0.041197222498539215,
+            -0.003843233633335066,
+            0.004373367480223065,
+            0.016194121660459908,
+            -0.029165150066068787,
+            -0.00735139702658342,
+            0.017257895082337448,
+            0.014153313280536095,
+            -0.002376178042741926,
+            -0.023223700836427403,
+            -0.0319877537205969,
+            0.02088060774052704,
+            0.0012829290529515294,
+            0.03251550040316258,
+            -0.04128453069131847,
+            -0.023376114412063578,
+            0.04083022735076889,
+            0.005690491480259003,
+            -0.05810403800689653,
+            0.016582434816685452,
+            0.010402038964417569,
+            0.016795644054313566,
+            -0.00908590043098552,
+            0.05003656634510042,
+            -0.037216197954986735,
+            -0.013027960261454162,
+            -0.007090943677204207,
+            -0.05175371787508879,
+            -0.0020065660448063322,
+            -0.014908489552053589,
+            0.010627472758306387,
+            -0.03275473161158777,
+            -0.017334608434133866,
+            0.025321047439048318,
+            -0.0033515195870701753,
+            0.011526068468647207,
+            -0.026783879816504975,
+            -0.0012830935419413407,
+            -0.017506908439998053,
+            -0.02114853058440477,
+            0.004878884893669383,
+            -0.020936900837115968,
+            -0.0025587027781733642,
+            0.01557850246191957,
+            0.023652335794299834,
+            0.011489428313303988,
+            0.0019089137782552768,
+            0.02054907941022243,
+            0.014361232983780595,
+            -0.0056413109308506085,
+            -0.0010300050480036684,
+            -0.03413262505887213,
+            -0.018821160485010483,
+            0.03361989907065425,
+            0.013125889701529858,
+            -0.003742167496379789,
+            0.008813100942466033,
+            -0.0383265293638935,
+            -0.035316957263713485,
+            0.03020499323039515,
+            -0.05001623971243151,
+            -0.00855479667673597,
+            -0.005939674940109814,
+            -0.026451831943555135,
+            -0.008032873049527808,
+            0.0009431571685654594,
+            -0.014387605128858954,
+            0.016382559541755014,
+            -0.024909510232038502,
+            0.003715736505444385,
+            -0.006016712902409688,
+            -0.009110197692342108,
+            0.05507263160354303,
+            -0.022103031032598353,
+            -0.0475415595313825,
+            0.021958434235402655,
+            -0.016670809659931243,
+            0.0323519363801999,
+            0.03867494951405658,
+            0.011124508964363016,
+            -0.008002445714431074,
+            0.034041838454818577,
+            0.018400123235267234,
+            -0.01161754789471199,
+            -0.025901844576206774,
+            0.021800298132870245,
+            0.00943488425697727,
+            -0.05906691123008531,
+            -0.00782549274387971,
+            -0.019357446956797447,
+            0.008682376344965976,
+            0.02863077694455766,
+            0.0024925977373894447,
+            -0.002682170968419412,
+            -0.013590235283325244,
+            -0.021511149507762284,
+            -0.03255453916518719,
+            0.009039307015853905,
+            0.008305286476500875,
+            0.009312625027668458,
+            -0.0014144165766983945,
+            -0.007853869334376857,
+            0.009067065760942163,
+            0.022485201611182105,
+            0.03966189080765722,
+            -0.041627289519629344,
+            0.002707086456239785,
+            -0.022678178050044222,
+            -0.03653753142622053,
+            0.0036574646778835588,
+            0.029996160786576367,
+            0.02901163765559891,
+            -0.01971777193183787,
+            -0.0036589934841109884,
+            0.009181234938554578,
+            -0.025979483002227535,
+            -0.00773645103995767,
+            0.004280770859737866,
+            0.04116668301017644,
+            0.006201085381597264,
+            0.01402728258112097,
+            0.001060407317110527,
+            -0.00920801784000087,
+            0.02109880889040528,
+            0.025211022083982727,
+            -0.029866678264942617,
+            0.014439702646820226,
+            0.009379206296908577,
+            0.04156435812161122,
+            -0.037309655469525846,
+            -0.011819731348299495,
+            -0.02766498020409985,
+            -0.010454143230794476,
+            0.027911532296566614,
+            -0.04127890742215716,
+            -0.006948536183086887,
+            0.010207784113909938,
+            -0.0016584043356136647,
+            0.016437495428919636,
+            -0.014607861750686952,
+            -0.03222311919805017,
+            -0.000610723203118213,
+            -0.02867445198803213,
+            -0.024772816792038133,
+            -0.016510309753766612,
+            -0.018581201775667577,
+            -0.07450654873867177,
+            0.00026832248829154255,
+            -0.06724696990816072,
+            -0.02447002541264633,
+            0.04885569249258513,
+            -0.010195604207134053,
+            0.03332905935704749,
+            -0.02815272112271314,
+            -0.005296340304911926,
+            -0.05908835042526019,
+            0.0038476172058970774,
+            0.025753576863447866,
+            0.02386003857017594,
+            0.005206447371391696,
+            0.024799057871126032,
+            0.012060738268265606,
+            -0.04146503002980051,
+            -0.024949907556862643,
+            -0.014503475223770958,
+            -0.0016340466065744286,
+            -0.0034009167013795906,
+            -0.031298432995411005,
+            0.036241272149880994,
+            0.008902644391348814,
+            0.03550223922667804,
+            0.04463301409716232,
+            -0.010385984413759852,
+            -0.03305004479734889,
+            -0.006952739338820577,
+            -0.0175981588092108,
+            0.042668533202572856,
+            -0.017112184620086308,
+            0.005149173014526686,
+            -0.03219589107997635,
+            0.024181808150698054,
+            0.0014339790422945406,
+            0.01254721190611376,
+            0.011099458649404119,
+            0.0027626836767364253,
+            -0.026883640446090158,
+            0.0024160770766463607,
+            0.023874514847956222,
+            -0.022558433617669373,
+            0.00917667715473093,
+            -0.006754091166465801,
+            -0.033686935718776144,
+            -0.01078389638118685,
+            -0.013595327486167114,
+            0.024712781935748277,
+            0.013168464807451798,
+            0.005875864604862434,
+            -0.02417814420558463,
+            -0.0023978800459714144,
+            0.028650463366555304,
+            0.020176980768301288,
+            -0.02118184201861616,
+            0.019082340279671355,
+            0.019734925740100277,
+            -0.04209645071102001,
+            -0.04217310001316092,
+            0.006553913719098956,
+            -0.003735719227468833,
+            0.0061793909220459,
+            -0.028285889794908054,
+            0.008181788076484304,
+            0.004432488437404309,
+            -0.01743905248466261,
+            0.041221269621866216,
+            -0.014892354419172107,
+            0.005087102208335201,
+            -0.022198383716186407,
+            -0.011936388003605221,
+            -0.00027487042024290816,
+            0.044599215801158845,
+            0.008068975171827777,
+            0.003496950921841073,
+            0.02238779276447619,
+            0.0054839808262270315,
+            0.04816723616939378,
+            -0.02685221260466647,
+            -0.0190356946337995,
+            0.016464136459560113,
+            -0.0176092780782677,
+            -0.012513274095432353,
+            -0.010695421445334002,
+            -0.004784348908457072,
+            -0.022203041334932165,
+            0.010788155380415583,
+            -0.018353897272137543,
+            -0.02277431189907027,
+            -0.003945465850796918,
+            0.016024639792926797,
+            -0.006248368166794934,
+            -0.0038769803892794875,
+            0.0016292556540434277,
+            0.02412864830338344,
+            0.02892955453819876,
+            -0.02408022470555968,
+            0.019190282372105023,
+            0.018941584034503536,
+            -0.006709516380602965,
+            -0.020956377933747532,
+            -0.01964885206488053,
+            0.01434623424964373,
+            -0.01352968586212152,
+            -0.013872556892707697,
+            0.004052798542613835,
+            0.009201147374299997,
+            -0.004086067985859617,
+            -0.04185829685685201,
+            -0.014668698123513235,
+            0.003179403563664022,
+            -0.0006415014901450319,
+            0.019360184344240947,
+            -0.0150817538567364,
+            -0.029771999278536083,
+            0.014221776097046751,
+            0.022944884879589678,
+            -0.04493425220030206,
+            0.02117577719533236,
+            -0.0030648751497261983,
+            -0.00016251533643103604,
+            -3.817296018601414e-05,
+            0.008727767382077138,
+            0.026914779443266063,
+            0.03456565094655899,
+            0.01974806772696219,
+            0.00450800007152441,
+            -0.008952599700759673,
+            0.025184190461155245,
+            0.009846044083064704,
+            -0.025168735183134738,
+            0.010983353565382029,
+            0.016740721743326522,
+            0.0070838930012654296,
+            0.009173879407277282,
+            -0.006078288608766152,
+            -0.018269347893097643,
+            0.013181494188066667,
+            -0.01435693747022231,
+            0.01678907949532146,
+            0.05154463552784974,
+            -0.00039632596424461964,
+            -0.02461439550772063,
+            0.008418147037164186,
+            -0.006544319842335593,
+            -0.010065129286193278,
+            0.02214628550460128,
+            0.0029515574751370417,
+            -0.030862152033847717,
+            0.007509556025223583,
+            0.00675485304374519,
+            0.01894936231559203,
+            -0.02355525295639599,
+            -0.01387885858544803,
+            -0.00431398020087544,
+            0.0021208056468763523,
+            0.01580766504149989,
+            -0.004739477121073512,
+            -0.013204823447960706,
+            0.02994098229517376,
+            0.025496728234662373,
+            -0.00026124946738213036,
+            0.015060064980768175,
+            0.022262345898533903,
+            0.009968408664790368,
+            0.007588431141088797,
+            -0.0083063413609779,
+            -0.031336272171937095,
+            0.0018957858383561667,
+            0.05800811417728381,
+            -0.014937085296865802,
+            0.04313875432235027,
+            0.02018476425923236,
+            0.015657644644987107,
+            0.017880263053253607,
+            -0.01662268082870562,
+            -0.008740053682875323,
+            -0.0017630394204179638,
+            -0.008294417990024647,
+            -0.027613464270562948,
+            -0.005953209556351086,
+            -0.02379961210881074,
+            -0.013878676554701543,
+            0.015418315245979704,
+            -0.014764728065382074,
+            0.0027420640870858457,
+            0.010395538988699663,
+            -0.006590633833835581,
+            -0.00841537209927409,
+            -0.003943772700797381,
+            0.03376338017637296,
+            0.011980354721458463,
+            0.027761929822019422,
+            -0.00011128539412582973,
+            0.009342430952107655,
+            -0.018419526564396063,
+            -0.006839752805318664,
+            0.03271190887864607,
+            0.006011520427944665,
+            0.004652835471939905,
+            0.01485920920301398,
+            0.014305626381398871,
+            0.038015677219487023,
+            -0.005102005401565164,
+            -0.018204460895989052,
+            0.02065495133597456,
+            -0.0049699013356374775,
+            0.019149036693562712,
+            0.008825981577388056,
+            -0.002660055848523568,
+            -0.004292379561953203,
+            0.03090270710860331,
+            -0.020243440437217503,
+            0.018735727616321387,
+            -0.02238236089688497,
+            -0.0276584413929402,
+            0.023028408906176946,
+            0.008320436733620496,
+            0.04047641118026783,
+            0.011463851066078041,
+            -0.02247143742166039,
+            -0.034588725413285185,
+            0.009827713930789184,
+            0.00337071450636392,
+            -0.024446280199879062,
+            0.017542777160283324,
+            0.00780836483490212,
+            0.008453886667283516,
+            0.016624116261219637,
+            -0.004175682824655864,
+            -0.0042784362933758835,
+            -0.001546656717896957,
+            -0.008819739495525477,
+            0.01406790007187259,
+            -0.015487658289541583,
+            -0.021457182907140035,
+            0.024687169583245905,
+            -0.03915911962859764,
+            0.01771290139419629,
+            -0.006510046907173026,
+            -0.00997450880725535,
+            -0.02059560682587115,
+            1.5939836949198004e-05,
+            -0.007159493752862143,
+            -0.0004782363814453761,
+            -0.017503181957042913,
+            0.016643971600145645,
+            0.024056442566869802,
+            -0.013209211443909658,
+            -0.00445521990927108,
+            0.0020313626821877526,
+            -0.029235692455820056,
+            0.006662209097255912,
+            0.016993027053110635,
+            0.00529940612845993,
+            0.0026104743851393716,
+            0.0001707934714983891,
+            -0.031289446611163674,
+            -0.005782208620284939,
+            0.03053516889069683,
+            0.05590704747438521,
+            0.021628254147289437,
+            0.02403314981042378,
+            0.02908606663381421,
+            0.02717206823892633,
+            -0.003198844932040613,
+            0.0016217508103442348,
+            -0.01620660606431983,
+            0.027098323143010743,
+            -0.011664168290676943,
+            0.04322175021589083,
+            0.008652323184522541,
+            -0.049493020736445056,
+            0.004537958157696794,
+            0.00943239454147692,
+            0.034918603888916544,
+            0.016678716235690318,
+            0.0005887414054803805,
+            -0.005894338664998101,
+            0.03279165663336323,
+            -0.02735726709032382,
+            -0.011962070182991011,
+            -0.00036014407560325284,
+            -0.019079003730607717,
+            -0.015938295169860884,
+            0.010357611943935459,
+            0.04278924678845588,
+            0.0009957483598272451,
+            -0.0071422790468850215,
+            0.01571201177743999,
+            -0.009445307674383,
+            0.02286338492075306,
+            0.007336440384050498,
+            0.0042809497342459904,
+            0.025581467690095977,
+            -0.0025007871391868105,
+            0.02300539823997411,
+            -0.000628898771317254,
+            0.004582863909134464,
+            -0.015960578501883316,
+            -0.00930649396114688,
+            0.027838455754803444,
+            -0.011112268986721,
+            -0.029253746504396393,
+            -0.008734785561831333,
+            0.00472579547270089,
+            -0.0003805825712986138,
+            0.0021854358149324227,
+            -0.01183885533661493,
+            -0.010840951201591629,
+            0.019965691788904057,
+            -0.004922943301424112,
+            0.003800048754664761,
+            0.012664049429916252,
+            -0.0030124219674414897,
+            -0.01910844996750822,
+            0.01889983984736845,
+            -0.003858283778519141,
+            -0.024168821338992813,
+            0.020083955711766945,
+            0.01850497889545421,
+            0.045156567155119454,
+            -0.01586824055785609,
+            0.004612757419198795,
+            0.017497774574602505,
+            -0.002172088240279357,
+            -0.019617327081566827,
+            -0.014944795772093997,
+            -0.018212462639041492,
+            -0.015975283672064702,
+            0.012837956594691596,
+            -0.01243789877725244,
+            0.019500571353639314,
+            -0.0380735226388175,
+            0.012590900534826689,
+            0.004832168601136412,
+            0.007505385730121269,
+            0.016011983211212992,
+            0.025994560026860358,
+            0.002157436199771672,
+            -0.033465476168065784,
+            -0.01014599891204166,
+            0.007845373376916211,
+            0.010093432683262838,
+            -0.002849185178237036,
+            -0.018147004183434554,
+            0.003633790611809425,
+            -0.0266147420637404,
+            -0.018644422659309035,
+            0.022926737299761104,
+            -0.04798772015599113,
+            0.017908415301694906,
+            0.024859125158586466,
+            -0.04343699525707025,
+            -0.04886699607556352,
+            -0.011459188432595229,
+            -0.036774381316312464,
+            0.0014128005215813578,
+            -0.0005256067584181517,
+            0.011194992852264102,
+            -0.009342140079829378,
+            -0.018598671981710183,
+            0.01584789414301952,
+            -0.0036412641242200418,
+            -0.004225840239935549,
+            -0.00616696736018045,
+            -0.0025598631463178384,
+            -0.010188803157718543,
+            -0.03244694662411977,
+            -0.03290891136780909,
+            -0.00033742479296272127,
+            -0.009743927051414455,
+            -0.02334288359945272,
+            -0.03447062782872289,
+            -0.008728968107967028,
+            -0.037753659749578956,
+            -0.00515561961442376,
+            0.04070890150714597,
+            0.027517147608335957,
+            0.009449414978133334,
+            0.008086900089929912,
+            -0.013989034604490105,
+            0.05691047488419298,
+            -0.020425883381389086,
+            0.039466620642512155,
+            0.005301493569940605,
+            -0.0037222353899918703,
+            0.013075425214878277,
+            0.027710826422351564,
+            -0.006253665106995653,
+            -0.0017412928073238833,
+            -0.01928096630475438,
+            -0.012920476859996416,
+            0.009444408011467955,
+            0.015799884767184885,
+            -0.0025083061619197524,
+            0.001274457251739847,
+            0.00930533161700852,
+            -0.0006652478540264343,
+            0.017258038491335247,
+            0.03996268817486439,
+            0.003833844612473671,
+            -0.00826404141706295,
+            -0.018822002570963614,
+            -0.003188844900317347,
+            -0.011807006674248829,
+            0.010482058734260632,
+            -0.015410635475145028,
+            -0.011540803927522013,
+            -0.01249428777269578,
+            -0.017081856786754605,
+            -0.002183034976477535,
+            0.003114834030705039,
+            0.0027475411722881747,
+            0.02261627635192778,
+            -0.015872027027080824,
+            -0.022487581027490516,
+            -0.0035823117885461506,
+            -0.02398778449664045,
+            8.13716819067491e-05,
+            -0.01050060782407259,
+            -0.015616181283381218,
+            0.013676480044963827,
+            -0.021513173575512936,
+            -0.01091399089076612,
+            -0.008699176177861256,
+            -0.0006986953850759474,
+            0.005853723710671249,
+            -0.00993355748621621,
+            -0.00010241843582159751,
+            0.01150872694850241,
+            0.005559552266827916,
+            -0.012336449703762907,
+            0.0039431718621504,
+            -0.012740550272925562,
+            -0.004432059174951158,
+            -0.03592309122711106,
+            -0.019665686718735042,
+            -0.01846795637714379,
+            0.005642094448288269,
+            -0.005108964028543505,
+            -0.020586678054151854,
+            0.00668364061048941,
+            0.0034775127050453774,
+            0.004713169526661792,
+            -0.010723667875209928,
+            0.013944735241748791,
+            0.010882420168586074,
+            -0.003518245916227596,
+            0.013827334494230662,
+            -0.001330399936559009,
+            -0.020286697504699764,
+            -0.011139922402071095,
+            0.03538258948179278,
+            -0.024961826464903997,
+            -0.0018019328351763667,
+            -0.0026423695690168066,
+            0.00597003516870726,
+            -0.03071835946501977,
+            -0.013634212232744325,
+            0.02164400349828422,
+            -0.005846552894831441,
+            -0.026367014426434226,
+            0.03399116480234033,
+            0.03473258231629753,
+            0.010041605592929843,
+            -0.013423384340589794,
+            0.04730054254696893,
+            -0.012968978954938308,
+            -0.018274800841400665,
+            0.026826644936918647,
+            0.0278389611094493,
+            0.02525164095926168,
+            -0.029782550447890466,
+            -0.003010713086487275,
+            -0.02393629084357885,
+            0.017993366287762866,
+            -0.0263476663180971,
+            0.01864011499179424,
+            -0.02332263152537849,
+            0.016014908881756034,
+            0.01868464825773604,
+            -0.003104041044182249,
+            0.026313491086684425,
+            -0.0052945248919367815,
+            0.015253731232325074,
+            0.0012371308759955573,
+            -0.02119765517013431,
+            0.007795466385020642,
+            -0.006469500272298062,
+            0.025762169721880456,
+            0.00833992768202466,
+            0.01801901639978976,
+            0.022019940825351116,
+            -0.0409522298213741,
+            0.02003698451081499,
+            -0.001367916864679543,
+            0.01782984895971748,
+            0.0038516134262842373,
+            -0.007885432294782304,
+            0.014221728513400065,
+            0.010991352188016265,
+            -0.012851099261924757,
+            0.003840539873231741,
+            0.0008923482291826014,
+            -0.028928104542716937,
+            0.007066004768771198,
+            -0.026914574065060926,
+            0.025866907400807087,
+            -0.005526774784152086,
+            -0.00998122210956923,
+            -0.020744532673097317,
+            -0.0071425207262770865,
+            0.003754468896273156,
+            -0.012747874997429746,
+            -0.011385805601999295,
+            -0.024700457669046855,
+            -0.02401291382882952,
+            0.029591582802488962,
+            -0.014054332096344003,
+            -0.04015978298687865,
+            0.001234440863385445,
+            -0.023301255402129767,
+            0.01193125007773239,
+            -0.005793124129267758,
+            -0.024440100649464773,
+            -0.042563399082257154,
+            -0.017548888269389457,
+            -0.0009394887961485922,
+            0.022739763644266633,
+            -0.00114156949394707,
+            0.041791395064755515,
+            0.0008816696205437911,
+            0.022207132857533028,
+            -0.015547580543606755,
+            0.005319731264840819,
+            -0.02071229859314312,
+            -0.041495579742912285,
+            0.03660432378985971,
+            -0.014637031447181937,
+            0.025854844497435178,
+            0.03324325296704186,
+            -0.013009347371118843,
+            0.001354504885302625,
+            0.02652570709474851,
+            -0.014618100923440427,
+            -0.010782232747863238,
+            -0.0034936699515378212,
+            0.003461670978569965,
+            -0.006468116285021298,
+            -0.027793221545159466,
+            -0.007752458229193391,
+            0.04193706715422074,
+            0.017894069367616637,
+            0.021928994684423043,
+            0.022606401934291136,
+            -0.017972404868085358,
+            -0.026402048654300563,
+            0.003656646576244765,
+            -0.013129722990153284,
+            -0.018090360146368677,
+            -0.01624598856366761,
+            0.012397150789705746,
+            0.011741002980073786,
+            -0.004052422390675261,
+            -0.01836750423615859,
+            -0.020929335029248472,
+            -0.016327412417122416,
+            -0.0042615345226698015,
+            -0.008458117095535,
+            -0.005002582649405199,
+            -0.012158173174049797,
+            -0.020409616620022143,
+            0.03706368540614453,
+            0.0285565642978937,
+            -0.0041732933884846566,
+            -0.00607154864704767,
+            -0.011142150176334957,
+            0.008834431806990914,
+            0.006021232046087652,
+            -0.005028870471297621,
+            -2.3396043292139426e-05,
+            0.005112033372713154,
+            0.022822312435625314,
+            0.007271235299348001,
+            0.0035791789459533856,
+            -0.06340305811365254,
+            -0.005392000517722054,
+            0.022560201096078862,
+            0.012581401789917604,
+            -0.0032248219865223496,
+            0.025180718735801837,
+            0.0006451724964970927,
+            -0.009714514747463593,
+            -0.018389422219956937,
+            0.024728862150150004,
+            0.008760329201960964,
+            0.020988700992999768,
+            -0.014974763313686516,
+            0.003845297486923296,
+            0.0006445983847138896,
+            0.00750394530065325,
+            -0.0019792992847026793,
+            -0.008578254004439607,
+            0.025776677078006516,
+            0.016593438298233645,
+            -0.005695578040595484,
+            0.01604018433477423,
+            -0.0017497260488899657,
+            -0.021460543443428924,
+            -0.010265213095609554,
+            0.00042017209251926397,
+            -0.010570864288876387,
+            -0.018015790888295944,
+            0.017569414447505878,
+            -0.002601036282535863,
+            0.0072399377790164435,
+            0.030281707151249355,
+            0.0423183067569767,
+            0.009566493398534713,
+            -0.005928603293907002,
+            -0.0037936911251953654,
+            0.015477873304924939,
+            0.01513127995244932,
+            0.010741256034652754,
+            0.03015858543221727,
+            -0.029247492875021843,
+            -0.006690306960497428,
+            0.0046653727410352576,
+            0.004610405633429778,
+            -0.011692179099382171,
+            0.030055897497741527,
+            0.04569730592994177,
+            0.006736813244286893,
+            -0.004799978513117371,
+            0.002121067028052434,
+            0.011299889107780231,
+            0.007768137218140627,
+            -0.018927080714961783,
+            -0.0012222962555201746,
+            -0.006009911026251445,
+            -0.014001766533604134,
+            0.026941585318261085,
+            -0.003960837039351648,
+            -0.011848232223811548,
+            0.02903357274500332,
+            0.1105267435019781,
+            -0.006011029338781496,
+            -0.0053646998281195206,
+            -0.03242321704621517,
+            -0.015194102811332289,
+            -0.008860590919147765,
+            0.0056988205877963076,
+            0.01635957585050017,
+            0.020327661913559888,
+            -0.0008943408274661007,
+            0.016820860249955754,
+            0.02822175614376368,
+            -0.013854827839129885,
+            0.007906244895327178,
+            -0.030891381690403466,
+            0.015702117924362928,
+            -0.005383974926093977,
+            -0.012032955400313954,
+            -0.0026169511828034054,
+            -0.008618307035330928,
+            -0.0037356680273096105,
+            0.0009666430093455632,
+            0.011296307527684261,
+            0.004427516814276546,
+            -0.014484198313369681,
+            0.002829529156309784,
+            -0.008595300985863424,
+            -0.005122996529848575,
+            -0.02393571073523279,
+            -0.024119245543788293,
+            0.005223147047205755,
+            0.0019389305548629142,
+            6.344308292376579e-05,
+            0.006848965054539454,
+            -0.003810712095343454,
+            0.030149081317333418,
+            -0.01700322008272963,
+            0.0012288912810468446,
+            0.01193226302726573,
+            0.012332749933413344,
+            0.01688602002694633,
+            0.0028600984823761366,
+            -1.9882692552530084e-06,
+            0.021172977047420227,
+            -0.010514036509402561,
+            0.01460334896316005,
+            0.013091417303145243,
+            -0.04103572381458182,
+            0.0089746401463863,
+            -0.0027841289484449873,
+            -0.024252447218794344,
+            0.004446472209246425,
+            -0.018987867343500102,
+            -0.00881974688349601,
+            0.0005365625696166774,
+            0.012054719659248512,
+            0.01960300421041832,
+            -0.008553943699846696,
+            0.02111201386480852,
+            -0.0003501657740808554,
+            0.004177088753111459,
+            -0.012879944670340026,
+            0.0065222126975051685,
+            -0.007563354540402343,
+            0.0008168128607271199,
+            -0.018213241937597355,
+            0.012526561612175514,
+            -0.0008285725179894544,
+            -0.015737003159436623,
+            -0.011337688275309921,
+            0.004737346137976519,
+            -0.0027993545140114705,
+            0.0030843163114700313,
+            0.010153740340626154,
+            -0.008815302484351509,
+            0.004919275910399053,
+            -0.0017541042538165297,
+            0.002562816687714991,
+            0.014285249186421434,
+            -0.029816964593980453,
+            0.005357227738099325,
+            0.009154842495811638,
+            -0.00048782303639333943,
+            -0.013074976152495576,
+            -0.004054505559072866,
+            0.0028322102449485015,
+            -0.016877328079971877,
+            -0.012128199639779175,
+            0.023421330164419888,
+            -0.012565145848820333,
+            0.008609070188894315,
+            -0.010936598722317274,
+            -0.018401801734378692,
+            -0.01066119413133089,
+            -0.005254229780560808,
+            0.01234969508270852,
+            -0.0038427767204741877,
+            -0.006947594635952202,
+            -0.01756894543880097,
+            -0.007890897373987378,
+            -0.047246698568222466,
+            0.009617876521097624,
+            0.027157555263862723,
+            -0.011355758309667893,
+            -0.04050311381480088,
+            -0.01017774338301804,
+            0.02437257208967103,
+            -0.004056329295095018,
+            -0.008345450168215274,
+            -0.001599171409135998,
+            0.018358386463243014,
+            0.0027207445181251406,
+            -0.007292967749349572,
+            -0.011267861428599221,
+            0.01657507035043879,
+            0.015105061280816214,
+            0.021419854579631276,
+            -0.02088952706825817,
+            0.006295984081511286,
+            0.020973020776936996,
+            -0.03073246543555915,
+            0.0023720435876790876,
+            0.03363966216709148,
+            0.010412588613998134,
+            0.013023311559810034,
+            -0.0010217662933572685,
+            -0.0075567381078977215,
+            -0.008248818794295818,
+            0.011848200323549865,
+            0.005066463524567732,
+            -0.023174489297821385,
+            0.021823825355858556,
+            0.020368868756448753,
+            0.0026042856858509524,
+            0.003457067317891926,
+            0.0005539745115410357,
+            0.01062670844108589,
+            0.009948889803890108,
+            0.05100741556350918,
+            -0.006493714124814164,
+            -0.014021751055930108,
+            0.015063988102064609,
+            0.02488059237697814,
+            -0.0023499941119596228,
+            0.01087282522970853,
+            -0.02612218935931224,
+            -0.021423880914461062,
+            0.023389294741183817,
+            0.008967763132667892,
+            0.009637806238448432,
+            -0.010815706635624337,
+            -0.011390999462441815,
+            0.031750675158698435,
+            -0.01307829462211914,
+            0.00759749911213318,
+            -0.02169895957209634,
+            0.01387879409130081,
+            -0.008606362268863441,
+            0.032229658009209815,
+            -0.0009066257436946125,
+            0.011062436424260079,
+            0.028010486866468023,
+            -0.007835805879241447,
+            -0.019657486763625547,
+            0.003967480939987674,
+            -0.015577952617624695,
+            0.02188429826317828,
+            -0.020502758155233298,
+            -0.0002362500702658591,
+            0.0040935406015014265,
+            -0.010056462958039452,
+            -0.0057038083589231615,
+            0.011475170331848666,
+            0.0015345076181482267,
+            0.006359262977587827,
+            -0.020293014401106,
+            0.01369706440475074,
+            -0.003989482215141903,
+            0.007887824197823247,
+            -0.015925992045349828,
+            0.014370223626942766,
+            -0.02309139937510304,
+            -0.021808736033141787,
+            0.008142621179624918,
+            0.0108996933426044,
+            -0.007589050511458233,
+            -0.025971539510030145,
+            0.02250992080022228,
+            -0.024980833805349545,
+            0.024979008145354243,
+            -0.006622339940663538,
+            0.002566378488088469,
+            0.004913874740363343,
+            -0.008102103508732738,
+            -0.0028098905427027783,
+            0.0057865587191235565,
+            -0.0073509382400023995,
+            -0.014123756810537343,
+            -0.019453380788786282,
+            -0.04083629135586275,
+            -0.03177927047231389,
+            0.022927581786088044,
+            0.011557232599801421,
+            0.0017073832658476193,
+            0.02339552660545296,
+            -0.02247471449131386,
+            -0.017890866287500053,
+            -0.009775068417716798,
+            0.025184899522229628,
+            -0.04408541669037361,
+            0.012104255392615549,
+            -0.014669458760022245,
+            -0.0245258242481677,
+            0.006274252098230249,
+            -0.004161960147481513,
+            0.009299171724026455,
+            -0.010779421730862708,
+            -0.04713627671791054,
+            -0.005192684784377356,
+            0.02330975631183273,
+            -0.0036802876364168935,
+            -0.006197548344287773,
+            0.03048284650641829,
+            -0.0019503797069087845,
+            -0.005038254331399784,
+            -0.005524875459245342,
+            0.027783596017224226,
+            -0.024950083604372984,
+            0.010988398531346805,
+            0.008304937912856357,
+            -0.008485065895193593,
+            0.006572665697403556,
+            0.0011518846966102158,
+            0.011927348472282199,
+            0.015632588644277183,
+            -0.015559413716826264,
+            0.02262990861928957,
+            0.011446850286896198,
+            0.006506075777765303,
+            0.008015225053785824,
+            0.00010054430757843335,
+            0.0022975742216194894,
+            -0.024948925449668525,
+            0.026047924967851216,
+            0.01888211557921096,
+            0.0001632181017876392,
+            0.012553094476616919,
+            -0.015042621225242777,
+            -0.02782009986290425,
+            0.02606646297992026,
+            -0.016847653820092335,
+            -0.00045272379465939274,
+            0.020053091486344633,
+            -0.016210036358455773,
+            -0.007930550661303182,
+            -0.01123514849773852,
+            -0.0162675150307978,
+            0.01803076559921804,
+            -0.012160217543572069,
+            -0.02216594536057653,
+            0.002594675636753233,
+            0.016443123828746582,
+            0.02134480656850119,
+            0.026694575953313482,
+            -0.011470930697351533,
+            0.017768927048191052,
+            0.0008653408391607816,
+            0.001865765599795092,
+            0.025745954171649543,
+            0.01702677011968492,
+            -0.008474889109168202,
+            0.025347176686722313,
+            -0.0016686223698600755,
+            -0.002421921060670558,
+            -0.012265504084322075,
+            -0.0165214436458303,
+            -0.007545471138114208,
+            -0.015333235556146196,
+            -0.0051830106590765685,
+            0.000762746782664525,
+            -0.0018118022696726346,
+            0.10876576838038368,
+            0.0034948044540582626,
+            -0.011608460689239234,
+            -0.005870263500417206,
+            0.007510774308250375,
+            0.0023525201199670765,
+            -0.006846154764389575,
+            0.016082721103476375,
+            -0.018616626204086575,
+            -0.023593744079394,
+            0.004743481974389998,
+            -0.0009262286319469741,
+            0.00870122878944883,
+            0.02029523251993269,
+            -0.008608837818169835,
+            -0.0200567711498164,
+            0.007089477722191928,
+            0.005535149644041221,
+            0.0023640746516071774,
+            0.005256431824732927,
+            -0.0006118671321824933,
+            0.008209920240361308,
+            0.02448422122961942,
+            -0.015622892878860465,
+            -0.029770836607612903,
+            0.010535371601298532,
+            -0.00015994156470228762,
+            -0.007193425789067776,
+            -0.011046316463797128,
+            0.014728841492009627,
+            -0.0011488451990980268,
+            -0.02751404133172324,
+            0.017663132712536528,
+            -0.019055335178900415,
+            0.014643246200933506,
+            0.0025348345163347464,
+            -0.00016387173410118788,
+            0.022565953069005714,
+            0.013101158251834834,
+            -0.04272085076882881,
+            -0.0034442020309914823,
+            0.01041205537746844,
+            0.011059422927946552,
+            0.0116446016862942,
+            0.0012992535381387887,
+            -0.018670801241345514,
+            -0.011788984490283379,
+            -0.009109849103208542,
+            -0.021268488414296117,
+            -0.012523033785938503,
+            -0.011393853408628214,
+            0.010743646386439633,
+            0.0012964030814325153,
+            0.023809566099779113,
+            -0.003030535542277875,
+            -0.006526351867141901,
+            -0.000430699447948508,
+            0.00880561288237257,
+            0.009694646009259077,
+            -0.02079011138439714,
+            0.008294338719512953,
+            -0.01823922789220214,
+            0.011819769685519291,
+            0.006416044253693268,
+            -0.022799568832306067,
+            -0.009911619276012527,
+            0.007152786695893951,
+            0.008393677195358034,
+            -0.013569953300860946,
+            -0.006503988731872938,
+            0.02299498754558268,
+            0.018870619152710703,
+            0.012233964644738905,
+            0.024994616203196145,
+            0.03838969722235787,
+            0.005744255912479976,
+            0.027682123067526515,
+            -0.016791471092091604,
+            0.008459745383680297,
+            -0.022493868174622695,
+            0.012902262638928338,
+            -0.00460399519266689,
+            -0.0050431343566427886,
+            -0.018188192498411616,
+            0.033765105245412236,
+            -0.009159751849801773,
+            -0.0027150644587327763,
+            0.034778224828120714,
+            -0.0066502021793471075,
+            0.004525230911930458,
+            0.005096748531775903,
+            -0.016067049494073893,
+            0.033274672789154185,
+            0.01686831040872425,
+            0.016353966592947262,
+            -0.021631941883929817,
+            0.011748317746828848,
+            -0.005675701689448899,
+            0.0001273442591116645,
+            -0.020738819037390263,
+            -0.0037528323797385733,
+            0.0027725541343763104,
+            0.0059860509282538546,
+            -0.016556134518370437,
+            0.02033623476757896,
+            0.00019647023567641008,
+            0.0052895744489259875,
+            0.03253462247073863,
+            0.01043311189804924,
+            0.019182938878855655,
+            0.010051051068017807,
+            0.019523739618278902,
+            0.0281939391326836,
+            0.025685787482623124,
+            0.0029932688462254147,
+            -0.018668945726139946,
+            -0.0009326901784812825,
+            -0.007880352863625262,
+            0.010492063055849727,
+            0.022849902214818293,
+            -0.022619228461640748,
+            -0.02772644874135741,
+            0.018566981184526675,
+            -0.0022608185345048414,
+            0.020031103834122595,
+            0.007544464625538984,
+            -0.020935181884939796,
+            0.003407355515103286,
+            -0.023178230166337178,
+            -0.0005418155871620593,
+            -0.031039187720144967,
+            -0.028591174077243604,
+            -0.0007659241904525392,
+            0.011945344534674635,
+            -0.01688153017778311,
+            0.03357788163306365,
+            0.017667778671101954,
+            -0.014289109343082812,
+            0.01604684528817524,
+            -0.011054587325539086,
+            -0.0036959091138078683,
+            -0.005819853870004682,
+            -0.00203755010992417,
+            -0.02157944474979592,
+            0.005138614763345542,
+            -0.0033509138150859593,
+            -0.0024871060745482883,
+            -0.016936575573598154,
+            -0.021901830366872595,
+            0.016375173843894617,
+            0.016742770709641697,
+            -0.006606252542838758,
+            0.039771905404651496,
+            -0.004238071340064403,
+            0.00018364315349545657,
+            0.011690521707403059,
+            0.0010018392078563795,
+            -0.013651921068215563,
+            -0.010829994512513501,
+            0.012362647379788833,
+            0.011321403447882543,
+            -0.018942043512252663,
+            0.005763773608775982,
+            -0.009131565038297889,
+            0.002089216671850269,
+            0.005438887787329238,
+            -0.008183835275629478,
+            0.006690280329947951,
+            -0.01673229763262922,
+            0.015833397876364723,
+            0.00295474085300386,
+            0.003186554859191664,
+            -0.011797859525926772,
+            -0.020991927615680524,
+            0.01576413532776261,
+            0.0032403514518527287,
+            0.018819824905201635,
+            -0.026755218746060662,
+            -0.00012946467986739473,
+            -0.009552681500884924,
+            -0.0028941357661144826,
+            -0.012533800273548772,
+            0.013975640628890588,
+            0.016864739319953755,
+            0.0063941926578529,
+            0.01781016436903586,
+            -0.0026268224264609724,
+            0.015368284773773678,
+            -0.018122416427266218,
+            0.008686009180697955,
+            -0.006801074032721299,
+            -0.018537521789181048,
+            0.0013237338264105415,
+            0.027863741654435434,
+            -0.004670012662973029,
+            0.011066853115455966,
+            0.011704884649534624,
+            0.018954056680143452,
+            -0.0012626215734593707,
+            -0.0030850559460101866,
+            0.015610404665171367,
+            -0.005838226405658572,
+            -0.006861883833171456,
+            0.020872966233916948,
+            0.032951653763679145,
+            -0.006716215648910268,
+            -0.0021292599951407863,
+            0.010886596327481677,
+            -0.004901684487088035,
+            -0.008644584974417106,
+            -0.009587037829603014,
+            -0.03158137900717491,
+            0.024369089446706894,
+            0.012092412439477474,
+            -0.009601042951586109,
+            0.022690862146548786,
+            0.020752792552278052,
+            0.011412743968296708,
+            -0.010471860708195726,
+            0.02291536038589634,
+            -0.01268035142112569,
+            0.024168869865226136,
+            0.032482831775271157,
+            0.004400973556358811,
+            0.0022312906286559167,
+            0.0021175489661843985,
+            0.01691348460546639,
+            0.006050357346231155,
+            -0.007068133214675996,
+            0.011656003476409986,
+            -0.018487923180009552,
+            0.01598530891381001,
+            -0.004432366596808033,
+            0.02808371007118948,
+            -0.01337145216499331,
+            -0.010473700033368562,
+            -0.00537325376270318,
+            0.00010726431372328729,
+            0.006445701087575359,
+            -0.003324002570256215,
+            0.005255443653813165,
+            0.0007072577303252132,
+            -0.017825649146944237,
+            0.003050128130393781,
+            0.004825967170196209,
+            0.009018799597337101,
+            0.004103653497834134,
+            0.007249957401824204,
+            -0.012360355929483,
+            0.01441885664789707,
+            0.004401820363035808,
+            -0.0026190938717927086,
+            -0.0007163225760398784,
+            0.011124538295057815,
+            -0.013232822471919533,
+            -0.004895624967029031,
+            0.005281228035321008,
+            0.0059275753353475414,
+            -0.007507924998741734,
+            -0.004484140989405051,
+            0.011908076549392969,
+            -0.0037917291321648483,
+            0.006351543504726163,
+            -0.004409464890606014,
+            -0.0037760571637986827,
+            0.005303661313434895,
+            -0.00974511544836468,
+            -0.004204167898342997,
+            -0.0007093125445559502,
+            -0.006869250680233214,
+            0.00786202375969793,
+            0.005971033227117194,
+            0.010606085272785863,
+            0.006969437776542303,
+            0.004293129058563288,
+            0.004114452739123398,
+            0.004071017374828283,
+            -0.002445681623750424,
+            -0.00592480769363354,
+            -0.004617501328052899,
+            -0.012067458701076778,
+            -0.008335239153829813,
+            0.009025028998949398,
+            0.008653779421856954,
+            -0.019363018792788574,
+            0.02048011578221327,
+            -0.001188320960771659,
+            0.00772165044232104,
+            -0.01602086976290928,
+            -0.000907553574811279,
+            0.005464880629734941,
+            0.008267702703609528,
+            0.016136762227127184,
+            -0.03654209526492064,
+            0.005179791231187316,
+            -0.0037321883493524766,
+            0.016026013709955444,
+            -0.008215703411373182,
+            -0.006979816286354862,
+            0.02564060526656963,
+            -0.01450763388985563,
+            -0.0009847758228920171,
+            -0.016142304341196283,
+            -0.01407778890808498,
+            0.010815725252959506,
+            0.02050940280233426,
+            0.020684808835848383,
+            -0.0004965802897137407,
+            -0.0005781445948759235,
+            0.00036179841583004193,
+            0.022775804864342608,
+            0.015545681427566235,
+            0.018697154644607316,
+            -0.004543500782944068,
+            0.007028436664826706,
+            -0.023057215501463973,
+            0.009912581556496658,
+            0.0011795417129232596,
+            -0.008215027850908477,
+            -0.008554137076267468,
+            -0.008703378564520125,
+            -0.010448538943182258,
+            -0.011773811327190638,
+            0.004106616839960258,
+            0.013895013678116032,
+            0.004190549543250594,
+            0.0031571190600981074,
+            -0.0225385788570314,
+            -0.006243193914542516,
+            -0.011229641377928767,
+            -0.01674214043453308,
+            -0.007367389908398082,
+            -0.02386334969150212,
+            -0.028021339684039734,
+            0.0002794207370355175,
+            -0.02411861258809571,
+            -0.015958336406924612,
+            0.005268815864773021,
+            -0.01546610225139129,
+            0.00898600137695074,
+            -0.008396481690803911,
+            -0.011873486990732417,
+            0.0074743033679134105,
+            0.015336444788692,
+            -0.017676977079590463,
+            -0.010165317702640217,
+            -0.01690524990506033,
+            -0.013460451917691695,
+            0.010918550283871637,
+            0.00595960977280955,
+            0.009250157926438213,
+            -0.012447979337850305,
+            0.0062932435120863365,
+            -0.012673905639312696,
+            0.015073013552721701,
+            -0.03769166067890086,
+            -0.029992249463405965,
+            -0.00014990754540999537,
+            0.007288036630677436,
+            -0.0020735742042505426,
+            -0.018695120921334806,
+            0.0003860116505754868,
+            -0.003608995261360232,
+            -0.014900191335144973,
+            -0.013866216738266176,
+            0.025251475305524053,
+            -0.006788483107616133,
+            0.01223227424986397,
+            0.0034937277040015626,
+            0.03438333560343413,
+            0.008803852645562129,
+            0.014666206816302367,
+            -0.005709421955160623,
+            0.003742433643393657,
+            -0.0033706258426092744,
+            0.006906464031352315,
+            0.0076578888497945185,
+            0.009276592191710524,
+            0.008025321004305927,
+            0.004576470294889512,
+            0.002887346543823031,
+            -0.001925468214331082,
+            0.02125241545487307,
+            0.025671028960818562,
+            -0.00037467906648160044,
+            0.008981063192139038,
+            0.0009387900084090577,
+            0.0002735789978680719,
+            -0.008374290340790827,
+            0.006547013284767913,
+            -0.008274637814401893,
+            0.004041066530254948,
+            0.012897549078838394,
+            -0.009150863940242148,
+            -0.010211568689323908,
+            0.016575088843292566,
+            -0.012380099999574783,
+            -0.03420455578618604,
+            0.026550654680719987,
+            0.005324512466403649,
+            -0.0035670106106754245,
+            -0.010657928468872608,
+            0.0037176027853158167,
+            -0.01103891334887315,
+            0.008096915671982816,
+            0.022587659771132487,
+            -0.0013527917944322358,
+            -0.0018085171207746743,
+            0.0038503794877980586,
+            -0.011870484733648887,
+            -0.007881489342351292,
+            0.006462772034381968,
+            0.0315696514747912,
+            0.01326417653779577,
+            -0.020400369539140532,
+            -0.015223808971473513,
+            0.020406862349006738,
+            -0.023142176712111575,
+            -0.006254959806032159,
+            0.009039506093607052,
+            0.0009087520492922078,
+            -0.022766046838415165,
+            0.0009662037093312435,
+            0.004701966191595815,
+            0.005946458676746661,
+            -0.006358986495019584,
+            0.03237532341275415,
+            0.001075992257265679,
+            0.015892020271402903,
+            -0.017189453434945788,
+            0.0017514632185789299,
+            -0.014189278671121677,
+            -0.014265447388465805,
+            -0.029809460132405256,
+            0.0005557491671071693,
+            -0.0030324198417091704,
+            0.011038984948790744,
+            0.024808067315720335,
+            0.01145103533194245,
+            0.014432246996007246,
+            0.010552852947578788,
+            0.01513689173494211,
+            -0.013653722180565311,
+            -0.029009757850432732,
+            -0.005171522291984226,
+            -0.018001048743500243,
+            -0.02297482628212251,
+            0.003962613844063587,
+            0.013645125285548414,
+            0.006278662690346082,
+            0.023786381386575035,
+            0.0021738516691483163,
+            0.0009580591750940011,
+            -2.690805459518147e-05,
+            0.008894091070322921,
+            -0.004952203441302471,
+            -0.006553986197076102,
+            0.003570189040872711,
+            -0.000652714137223501,
+            0.021228360881461853,
+            -0.005758946428212041,
+            7.897153237591187e-05,
+            0.014242175492137011,
+            -0.005622256348934008,
+            -0.0035009192529791146,
+            -0.017339926143652377,
+            0.006248842389703505,
+            -0.01622011586976866,
+            0.025783086900871728,
+            0.018450783573486916,
+            -0.009109295293546267,
+            -0.013469521573468915,
+            -0.011259555201313116,
+            0.02773097452736661,
+            0.004631282554715908,
+            0.010787055220617197,
+            0.018228322680582464,
+            0.01461254259249465,
+            0.01784830193964943,
+            0.004664725683522323,
+            -0.0034339781146649384,
+            -0.0007692550982545399,
+            0.021176646860433974,
+            -0.014700178590249292,
+            -0.002701208084390198,
+            -0.008842788249773623,
+            0.0021799005046816444,
+            -0.008431299000066363,
+            0.006086447848362669,
+            -0.010793016137040172,
+            -0.015251381840165954,
+            0.0189503457652375,
+            -0.02660767353735015,
+            0.016423753600066336,
+            -0.00593016170680532,
+            -0.00482891500342298,
+            -0.027459214027897378,
+            0.010728039528784636,
+            0.0027866316191608297,
+            0.0022519974195000687,
+            0.010770472550461752,
+            -0.021427634123401408,
+            -0.01867717541189363,
+            0.008532444554010673,
+            -0.011457117350255678,
+            -0.007152301670244725,
+            -0.00348619742388509,
+            0.0154715607296319,
+            -0.00017898955187849443,
+            -0.009925365923862246,
+            0.02085093731060963,
+            -0.007175309371250024,
+            0.01223979370120031,
+            0.005256127464980655,
+            -0.008756077893734508,
+            -0.0028438320920811362,
+            0.021022708898898203,
+            0.027557382862284808,
+            -0.008342206702778956,
+            0.014526066580760486,
+            -0.011883797998224506,
+            -0.01088335352319397,
+            -0.03479444009382274,
+            0.030104838505937447,
+            -0.013305732257226255,
+            0.026773747582411633,
+            -0.0006968265979630097,
+            -0.017037729111584094,
+            -0.016206207453228583,
+            0.01625225876515654,
+            -0.0024290584023212254,
+            -0.009366287391027166,
+            0.01465885301870365,
+            -0.02428628495473643,
+            0.0058336571592899735,
+            -0.0006627935593670669,
+            -0.011531762838191602,
+            0.013275508677482622,
+            0.0019223252930710324,
+            0.010738622785461513,
+            -0.02030405805119001,
+            0.0009661595529028194,
+            0.009263716373456188,
+            -0.016791855691574455,
+            0.006057830565306032,
+            -0.03730952786847912,
+            0.0029341721588188943,
+            -0.014833895684348656,
+            0.005725615593914644,
+            0.001498755253963699,
+            -0.01820708352937343,
+            -0.03652692232407375,
+            0.00043665701245599497,
+            0.01913883585802903,
+            -0.007975988799790905,
+            0.020538595326175857,
+            0.008243569241768276,
+            0.008421404577931517,
+            0.002314738067443065,
+            0.0016702403734608475,
+            0.008279538988821465,
+            0.03679456861213399,
+            -0.01775927634148114,
+            -0.004836606118958108,
+            -0.0012151329859941935,
+            0.010644195708319258,
+            -0.02028470571116305,
+            -0.009287515726906219,
+            -0.0036506912311261536,
+            -0.013134467570201307,
+            0.030615862647585222,
+            -0.020265555441523482,
+            0.025634449009756734,
+            -0.009300770899023007,
+            0.009274072905190077,
+            -0.02097685009125915,
+            -0.009797008690754956,
+            -0.010661540100131183,
+            0.0019532291898233895,
+            0.013465005282849381,
+            -0.009151430285942593,
+            0.0071467934925125145,
+            -0.006655608635025676,
+            0.01020327561163374,
+            -0.001472597695707201,
+            -0.02443508800166349,
+            -0.0024998811551575563,
+            -0.01107321258077183,
+            0.02456164421154801,
+            -0.0019619696941303388,
+            0.0005043179699158694,
+            0.010951580163871948,
+            -0.00853593956432799,
+            0.023927336608555987,
+            -0.008855702280766362,
+            0.0010093679746665581,
+            -0.004863269000020324,
+            -0.004005691415117686,
+            0.001698212044518236,
+            0.01822024022650613,
+            -0.005707404715849356,
+            0.03722587302228139,
+            -0.01708925706528118,
+            -0.0065500772743641925,
+            -0.00015538686795361824,
+            0.010852925330001283,
+            0.010035359503147549,
+            0.008754719437760327,
+            0.004067820167375348,
+            -0.014861417626433965,
+            0.009346669946626484,
+            -0.020136029447262147,
+            0.007501669351583119,
+            0.023178490169347605,
+            -0.006840359541885465,
+            0.0006676450965357345,
+            -0.01283320739092204,
+            -0.015610538827742276,
+            0.006308543164022682,
+            0.002329420005586303,
+            -0.002295739637504597,
+            -0.02181189272979859,
+            0.006298804075737111,
+            -0.014398470126723037,
+            0.004007419942728446,
+            -0.0007497656993578918,
+            0.013863867106289847,
+            0.016451166273826513,
+            -0.006740209707090658,
+            0.039292260347522444,
+            0.01871723939316731,
+            0.009109005310505462,
+            0.005674782595856756,
+            -0.040588444759736254,
+            -0.008993850998154852,
+            -0.0010197185016393919,
+            0.004866150984083949,
+            -0.01141354216871663,
+            -0.018944881712855702,
+            -0.002646110151132781,
+            0.000308585036997592,
+            -0.010681954707313127,
+            -0.009091585604191968,
+            0.01751599002710486,
+            -0.006034424764302979,
+            0.00872057856926618,
+            0.004146961781109887,
+            0.023758065635807138,
+            -0.009727802672285544,
+            0.01176257119851645,
+            -0.0071822083765114165,
+            -0.005456197751568526,
+            0.0006726765703478756,
+            0.011011628907288813,
+            -0.008031792262886802,
+            0.009763340867550022,
+            -0.004279891138185881,
+            0.008355720233269893,
+            -0.008552596384067825,
+            -0.023952129458400563,
+            -0.014972827087541732,
+            0.014347535404251018,
+            0.011651260496709961,
+            0.008137738462704715,
+            -0.013624435569539285,
+            0.0031728721674035352,
+            0.00675830277744976,
+            -0.009184784787581942,
+            0.009161771311910335,
+            0.017733082848622708,
+            -0.001411542568472906,
+            -0.0017830167499636113,
+            -0.0016130003349297027,
+            -0.012173528776899308,
+            0.0011196437753239162,
+            0.02284887358189995,
+            -0.003054479643937998,
+            0.014614283913976716,
+            0.010689486748036806,
+            0.009643292262102645,
+            -0.00810930837603355,
+            0.001915955511174678,
+            0.02341863048264366,
+            0.015022291783105098,
+            -0.004625715884342598,
+            -0.011422973894623295,
+            0.004259757717597968,
+            -0.010332744704689279,
+            0.007373328473086509,
+            -0.02117940113144938,
+            -0.004672698999328004,
+            0.008896311380763048,
+            -0.007028862917603513,
+            -0.01732432285099198,
+            -0.028026173555055017,
+            -0.016765805892242715,
+            0.019819329380181,
+            0.0027213760678977965,
+            -0.013291251876048186,
+            -0.0032919084721662363,
+            0.0031214883337300926,
+            0.002553005168048115,
+            -0.0071104806975682885,
+            0.005310583052470149,
+            0.007359183453835734,
+            -0.008674133980486269,
+            0.017665958608281126,
+            -0.000521269431423141,
+            -0.0008020251454616109,
+            -0.0005442374261144659,
+            0.0006127687598711624,
+            -0.016387752806126442,
+            0.01666161912459431,
+            -0.010949228376419892,
+            0.015198110119816624,
+            -0.008966726991404666,
+            0.008600499511690635,
+            -0.007617043351740469,
+            0.009458641139070436,
+            0.011597957400445055,
+            0.012755037944648426,
+            -0.004403680820063604,
+            0.008519241473418196,
+            0.004807816191718704,
+            -0.008147125240985166,
+            0.007017601015398136,
+            0.01296082929962557,
+            0.00582353932160707,
+            0.0013634237970372698,
+            -0.0008246894322793264,
+            0.0022035670958705576,
+            -0.002480976219285683,
+            0.0056293125704658135,
+            0.012714850523393719,
+            -0.002497639449055006,
+            -0.007135683201188999,
+            0.014988249771676699,
+            0.014955895630183352,
+            -0.022994948923833993,
+            0.007121634434082677,
+            0.005835971825639792,
+            -0.0030507120821557008,
+            0.002145266953387315,
+            -0.027334525318320587,
+            -0.020359474070022213,
+            0.01901995970350808,
+            -0.015260454474605018,
+            0.02850460517419317,
+            -0.011624138792780823,
+            -0.02284242513728605,
+            -0.01277489645734181,
+            -0.004209529633534193,
+            -0.022684500849916897,
+            -0.004894414098943742,
+            0.01919165657366256,
+            0.013401560972810082,
+            -0.004313549020265677,
+            -0.019089369171102138,
+            0.03863248408555474,
+            0.010497951840980463,
+            0.03508157919687289,
+            -0.008597970944278045,
+            0.0023354831464448313,
+            -0.01563401808449416,
+            0.015551176593436512,
+            0.001075882802133782,
+            -0.011575692834796734,
+            0.01505716108583101,
+            0.031836226915668404,
+            0.0034740889237207494,
+            -0.00617206855371193,
+            0.0018420207148235309,
+            0.00418433325800744,
+            -0.0008167766650900381,
+            0.011160989178055499,
+            -0.018497472318034093,
+            -0.011951262348399922,
+            0.0013093376768780122,
+            0.00216347903176653,
+            0.0059233460419514045,
+            -0.001778091863744448,
+            -0.013317639704511549,
+            0.011433308892643204,
+            0.011596810919469926,
+            -0.01518143105525228,
+            -0.007497373860338335,
+            0.0050957665991362405,
+            -0.018974405317819656,
+            0.01872472902494418,
+            0.011859522398789263,
+            0.000792413325332956,
+            -0.0006503947129906383,
+            0.00808635755624928,
+            0.003749813548995223,
+            0.0033657799335872368,
+            -0.007862215303617144,
+            -0.0028014669075116876,
+            0.0009892310856576799,
+            3.834379772674841e-06,
+            0.015402211364447726,
+            -0.015264893397161102,
+            0.02488855366696587,
+            -0.0025643639525691083,
+            0.00953993868921819,
+            -0.017460590918773188,
+            -0.0017911712931080155,
+            -0.00402374787529868,
+            0.01838051670627292,
+            0.006753072393532114,
+            -0.010130676216186492,
+            0.004921510943896829,
+            0.007975201081296845,
+            -0.012589786869771252,
+            0.00018319054705558588,
+            -0.0022606180909706688,
+            -0.014791528072476771,
+            -0.0031171076370692706,
+            0.017873825561816544,
+            0.007593688317551089,
+            0.0037375070591255375,
+            -0.015863143964928172,
+            0.007192324815779268,
+            0.007522990667369163,
+            0.024504875417877527,
+            -0.026309816223960274,
+            0.024779042752840836,
+            -0.022104087334088543,
+            0.02165781034613893,
+            0.0319051741666687,
+            0.007538146369110045,
+            -0.01690181186708103,
+            -0.006595947249169521,
+            0.006373991305923002,
+            -0.023410814842488123,
+            -0.005494017994983188,
+            0.016903736810723716,
+            -0.01792400152372258,
+            0.000330865061689199,
+            -0.014660413003063089,
+            0.0040825880154611895,
+            0.008248037865838212,
+            0.011540333451799686,
+            -0.0033578830043738085,
+            0.018327525495676525,
+            0.020171745352159914,
+            -0.01138759854371235,
+            -0.0004221149053107702,
+            -0.022321409200941777,
+            0.005046865873981027,
+            0.022509643273449936,
+            0.001111272240506322,
+            -0.009540467231190513,
+            0.018376261423745193,
+            0.008480532791922674,
+            -0.020801891594934313,
+            0.008436374227753589,
+            -0.036175982655998785,
+            -0.010454118523313122,
+            -0.009328507118928412,
+            -0.001218633166683384,
+            -0.004142773602181115,
+            -0.0002494371580772723,
+            -0.011365548842513265,
+            -7.699548642315061e-05,
+            -0.006889339420877534,
+            -0.0037173070609413474,
+            0.014953978004522174,
+            0.011678957032329096,
+            -0.003501047405385249,
+            -0.0073984932581048915,
+            0.008493128567102267,
+            0.006163937693564917,
+            -0.017034696006318337,
+            -0.013268418502042576,
+            0.0037122128689059085,
+            -0.020067656401141467,
+            0.005148329575092478,
+            0.00459729178556505,
+            0.014031669598917171,
+            -0.01849235806598133,
+            -0.009298127665365717,
+            0.019716512208448042,
+            0.0094466405158553,
+            -0.0008988916054855498,
+            0.004264260983102956,
+            -0.00032312548009733906,
+            0.002279987803645912,
+            -0.0008902345374346296,
+            -6.145528806973222e-06,
+            0.003602229465148649,
+            0.012503366799381277,
+            0.007759393855482671,
+            0.03694402922447051,
+            -0.016692148708641812,
+            0.005873765863350649,
+            -0.014187609702966002,
+            -0.0037592517335595654,
+            -0.00749939010396784,
+            -0.0004308157309481927,
+            -0.0022282234382577843,
+            0.0021999996319642854,
+            0.0052176773733353925,
+            0.009353171029508977,
+            0.002155405440765206,
+            -0.00823565584692709,
+            0.009691447250255694,
+            -0.015228173095533029,
+            0.003853858061067531,
+            -0.008207910291425473,
+            0.01017972550354623,
+            -0.009695280827515048,
+            0.0028705587799331246,
+            0.03849358164875786,
+            0.01130021300937487,
+            -0.004934175853966098,
+            -0.014300147534361459,
+            -0.0007796051618993811,
+            -0.01737877501558196,
+            -0.008579119859817895,
+            -0.005434792488602325,
+            -0.02617338653902575,
+            -0.00838916510137031,
+            0.015344260464413648,
+            0.011202567142310492,
+            0.014124853863170132,
+            -0.023894182185855824,
+            0.008897943489322673,
+            -0.03001670823127986,
+            -0.011951507895140876,
+            -0.001530967140156163,
+            -0.015467812535428609,
+            0.0010110999991167703,
+            0.023623196504289838,
+            -0.007762735863648054,
+            0.012654652822750325,
+            0.012867958170258448,
+            0.018777522196571934,
+            0.006390986057157016,
+            0.00038045736974183937,
+            -0.009444911783760569,
+            0.009534310165417939,
+            0.008806031651319704,
+            0.0080954543713255,
+            0.013161833638467553,
+            -0.0022840529693446553,
+            -0.018060162439023572,
+            0.005469288590000856,
+            0.014354475778235131,
+            -0.0033504276914600585,
+            -0.001615191819057537,
+            -0.011088493180661976,
+            0.016039040565630432,
+            -0.023559046333872156,
+            -0.009482398794557168,
+            0.009918576960319548,
+            -0.0027308309155029842,
+            0.009212063547988036,
+            0.0160181200081126,
+            -0.01881373001297218,
+            -0.0024168618343039703,
+            0.016244889977796682,
+            0.0024104546308670025,
+            -0.00022606425578303413,
+            0.016792244238218245,
+            -0.0007616857360728623,
+            -0.004279120886828465,
+            0.00997764266310123,
+            -0.0075631917138318655,
+            7.3878137053128405e-06,
+            -0.002416364915008295,
+            -0.0028253498168087964,
+            0.02054128727706496,
+            -0.021281801678871736,
+            -0.005379259148621041,
+            0.02257306103533458,
+            0.015687105622685975,
+            -0.007843745111084413,
+            6.258498042855072e-05,
+            -0.0219506172079994,
+            0.0019925886885244454,
+            -0.007768133999661569,
+            0.016524850592507592,
+            -0.007957949830060354,
+            0.0030085044673901665,
+            -0.01361024620290394,
+            0.011563781350927133,
+            -0.01277407169098418,
+            0.00873716215529134,
+            -0.002986943948333218,
+            -0.017567646684567492,
+            -0.0053109390854215125,
+            0.004561236136619266,
+            0.0014045175130801308,
+            0.044958041347040185,
+            -0.005539136232703934,
+            0.003729517522976541,
+            0.018086335927383804,
+            -0.014199453349727933,
+            0.02079719360289355,
+            0.016663293302592568,
+            0.00991445855219406,
+            -0.00033967370741635794,
+            -0.0018726748849858836,
+            0.023373354806639515,
+            0.013258226441642464,
+            0.00861986998150065,
+            -0.0014340218811281975,
+            -0.008516541227326329,
+            0.018239230461345617,
+            0.005458629943667832,
+            -0.012421896028975013,
+            0.012732721349758832,
+            0.013559877132508506,
+            -0.013720939472355347,
+            -0.011545210834395917,
+            0.01616358223872008,
+            -0.019904594319155373,
+            -0.018260016571763644,
+            -0.0036124537591368864,
+            -0.01614510591132948,
+            -0.03864699384205461,
+            0.01619401261888991,
+            0.018351607973242735,
+            0.006666313000123845,
+            0.013647410298896132,
+            0.002597680061427196,
+            0.004545530267244157,
+            -0.018531082430861854,
+            -0.0077618625909995275,
+            -0.008085144524135461,
+            0.003744735923004248,
+            0.011059457364156674,
+            -0.011335432904114519,
+            -0.009514379703581616,
+            -0.00034500702695418285,
+            -0.007433132060989644,
+            0.01971420270994951,
+            0.0020753610290475642,
+            0.0034506706065161024,
+            -0.00628437298309269,
+            0.017966485854173167,
+            0.026789678964152908,
+            -0.007532446825077476,
+            0.021886182869092073,
+            -0.002864230253056434,
+            -0.0015870360120997492,
+            -0.014393519359298496,
+            -0.0014439412842964876,
+            -0.011527182213472351,
+            0.0005453493437517964,
+            -0.0006791481533636673,
+            0.014093767997869645,
+            -0.03288824642147967,
+            -0.003957301296796813,
+            0.014444592778851293,
+            0.007538052312876599,
+            -0.005057755472988024,
+            0.011357919882949903,
+            -0.002964451028327082,
+            0.016977317604413924,
+            0.0075201055190542146,
+            0.023391646079994784,
+            -0.016588823248397513,
+            -0.004504858318770565,
+            0.006660877064954993,
+            0.0034631255931793638,
+            0.0014484203909961972,
+            0.01912983041187014,
+            0.023439941258581887,
+            -0.014715886385460541,
+            -0.0013596123366001458,
+            -0.0029020372473879614,
+            -0.021414221343490594,
+            0.014976806123253727,
+            0.0074729481304182865,
+            0.010967351226114886,
+            0.009806663105020302,
+            0.015465076881663691,
+            -0.02109587527354609,
+            0.0011671121739822202,
+            0.020567458365051382,
+            -0.008618333549105003,
+            -0.018815706906171112,
+            -0.009804829158883123,
+            -0.00966828714719191,
+            -0.006078893678150302,
+            -0.00803783313451146,
+            0.005523023167028105,
+            0.0022647506733877573,
+            0.01652997604669998,
+            -0.008204200434441975,
+            -0.008404609147866811,
+            0.008489247125986078,
+            0.007546785673285454,
+            -0.0033093451283570316,
+            0.010614050963656496,
+            0.0030633179859393587,
+            -0.006940540318011839,
+            0.00894884343338777,
+            -0.002787816848430143,
+            0.014347598920245487,
+            -0.02441769157218451,
+            0.004717742147185283,
+            -0.012096876825337124,
+            0.006405151160675699,
+            0.0041568790527989905,
+            0.0016702066682613428,
+            -0.0041159211459004,
+            0.0035217362856754353,
+            0.0008893479210743541,
+            -0.07433312151926533,
+            -0.012629681414004524,
+            -0.0035797012285864486,
+            0.033657838713146884,
+            -0.019695806809053914,
+            0.004477927056336216,
+            0.011966065644382396,
+            0.0096717896523898,
+            0.003070713232538593,
+            0.006136726101157844,
+            0.006112385569942848,
+            -0.0245258242481677,
+            -0.00015284418977092615,
+            0.004821040694679944,
+            0.0020347004045943454,
+            -0.007532408630122127,
+            0.00750565079228405,
+            0.015327710245879296,
+            0.020989926895388523,
+            -0.010970305087289488,
+            0.017227663358885748,
+            -0.003352718496981928,
+            -0.006068892126250978,
+            0.0014285750322333686,
+            -0.010153799722483554,
+            -0.017307415843726394,
+            -0.006495008086641125,
+            0.0068983699213297075,
+            -0.001982568363667046,
+            0.0044032687939897855,
+            0.003333596495330213,
+            -0.0032171586993948696,
+            0.0072365127663633925,
+            0.013920326463272646,
+            0.006415916101287133,
+            -0.013675562796300214,
+            0.003003782758807665,
+            -0.02500375035312058,
+            0.011945988879567646,
+            -0.012708975181025296,
+            -0.0009488161508767279,
+            -0.0047093877027523014,
+            -0.0035750724825159503,
+            -0.002213435677949806,
+            0.02043567538125188,
+            -0.014304009033474186,
+            -0.01802899775650995,
+            0.008187227768196673,
+            -0.011217013195839762,
+            0.008300740149622635,
+            -0.03382831122769974,
+            0.012386335491333264,
+            -0.0276029934382802,
+            -0.0002243752790856292,
+            -0.004521387639042107,
+            -0.0010631721066129095,
+            -0.009491184157278669,
+            -0.03274559774619223,
+            0.015845705808609598,
+            -0.007204751554079551,
+            -0.020338337341085228,
+            0.0007143424225187802,
+            -0.021267094083674304,
+            0.014150458196361147,
+            -0.01851165553224022,
+            0.0033450353123933058,
+            -0.018775439467431685,
+            0.002782289821140401,
+            -0.0018711350216089556,
+            0.010502195116579441,
+            -0.00557814442809019,
+            -0.0007130962242755068,
+            -0.013072988887382269,
+            -0.002121841212381393,
+            -0.006653444103331213,
+            0.009394907690516121,
+            0.0028678074167728367,
+            -0.009601577734902932,
+            0.011153571864462314,
+            0.013046116892519414,
+            0.019812860521973938,
+            -0.00539047787915588,
+            -0.00555139958411929,
+            -0.011553689090179406,
+            0.009852063270798534,
+            -0.016326212706634428,
+            -0.00827402913960903,
+            0.008982303148984887,
+            0.015216947130173639,
+            0.01026889105233137,
+            0.003352028462252994,
+            0.003150801750574548,
+            0.002531620731961765,
+            0.026744691368233237,
+            -0.0015120532859309638,
+            0.0008324701960729416,
+            -0.014508988496009931,
+            0.022127525293249926,
+            0.02772430884375744,
+            -0.0012206102581726523,
+            -0.00018859639594030037,
+            0.017437360169132967,
+            -0.004697052522250805,
+            -0.0038111184371524645,
+            0.024015246959315593,
+            -0.038396108183338655,
+            -0.008416825232533771,
+            -0.006788816882617399,
+            -0.006043428198944214,
+            -0.009587366247924467,
+            0.007294190273465812,
+            0.00041966080769004275,
+            0.0024709692753547823,
+            -0.0005250904505954556,
+            0.03452858797518386,
+            0.003403515654095309,
+            -0.0003392060101496309,
+            -0.0032980864446307116,
+            0.0019125121221204029,
+            0.0032987517853214346,
+            -0.009116761755779728,
+            -0.038030085122772624,
+            0.006006334000396066,
+            0.007932785488877407,
+            0.00975009749364395,
+            -0.008120121513849434,
+            -0.008864461019882246,
+            0.004909348738798099,
+            -0.02273129283865203,
+            -0.01372636307167243,
+            0.02625229144362647,
+            -0.008523028986127237,
+            0.029005877324974557,
+            -0.022435059061182174,
+            -0.013135836801818153,
+            -0.004126433322528265,
+            0.010283975013795537,
+            -0.019732856462444506,
+            0.004343860338939202,
+            -0.004022387065239562,
+            0.023733721026005954,
+            0.0077578144626961245,
+            -0.008371631446146718,
+            0.012386735614153616,
+            0.0064789380436815714,
+            -0.0026638642827013837,
+            0.00136333052963848,
+            -0.007209517067748103,
+            0.016916505654836485,
+            -0.011819784026452944,
+            -0.005833909403097991,
+            0.008977527508585844,
+            -0.0024556434204852946,
+            0.02509293948918158,
+            -0.0237762245071987,
+            0.005288171058938098,
+            -0.00019047961064753207,
+            0.01822542471604037,
+            0.008127347078975925,
+            0.008361588667386882,
+            -0.011835407590060994,
+            0.010004482616697591,
+            -0.03378107067882622,
+            -0.015433906677849963,
+            -0.007702910250795191,
+            -0.0013877702264578865,
+            -0.002059152251387601,
+            -0.015745218548968338,
+            -0.026393221735795527,
+            0.006294583740809379,
+            0.016267525575472445,
+            0.023291528020980265,
+            -0.004683197930892445,
+            0.012912830736650997,
+            -0.002831583991769044,
+            -0.008904271394710672,
+            -0.011441882849296418,
+            -0.012202629605208118,
+            -0.012470773094383122,
+            -0.014560694782340029,
+            -0.017045701603711442,
+            0.012139665577738337,
+            -0.020079061996873537,
+            0.013911518328924777,
+            0.0347210775522539,
+            -0.017176114991595492,
+            -0.0018941772168839354,
+            0.019702722555622765,
+            -0.004221469612316137,
+            0.01212088353057276,
+            0.0063707489618353725,
+            0.015298014532351866,
+            0.0038643174573634725,
+            -0.012181479717826009,
+            -0.007374005740809251,
+            0.019411457664902376,
+            -0.004902591844710158,
+            0.014591446725717338,
+            0.015069599884557399,
+            -0.008675672164385891,
+            0.002544198867387345,
+            0.01951865904496276,
+            -0.01762101777798757,
+            0.01665654707715257,
+            -0.00458402229480694,
+            -0.009936206685538294,
+            -0.013345740581684638,
+            -0.005336416724426228,
+            -0.019469457091169776,
+            0.0061608454109892425,
+            0.018931676738452464,
+            -0.00533013594306834,
+            0.009310871143727683,
+            -0.0037670793158045,
+            -0.005045151384034014,
+            0.0002041909886640482,
+            -0.04152751596227068,
+            -0.008340911336217609,
+            -0.0006651584370005978,
+            -0.02505748815737022,
+            0.025954102004571416,
+            0.007205602377144382,
+            0.022602344303400163,
+            -0.017168610245491406,
+            0.004459735690564842,
+            -0.005882024209102714,
+            0.013217517324383834,
+            0.02255689669623969,
+            0.0017984057469785134,
+            0.005699128497215882,
+            -0.02353849721739205,
+            -0.0034711576718386185,
+            0.015924913374553738,
+            -0.003604478141616307,
+            -0.0034993903155063817,
+            0.012592308877056896,
+            0.012864938348003884,
+            0.01567418642605425,
+            0.008801753550506258,
+            0.015606443428689292,
+            0.010589933994371628,
+            0.008928721556432368,
+            -0.005805129999077126,
+            -0.001025103064751441,
+            -0.015907755521365925,
+            0.0045874330999415985,
+            0.031701049450012876,
+            0.012594483199773746,
+            0.016783858873835704,
+            -0.01778536268185936,
+            0.00789523134933153,
+            -0.0012706411671644011,
+            0.008325108613530198,
+            -0.001677173669891956,
+            0.029540136288412976,
+            0.011739999605699826,
+            0.015670966746894652,
+            0.008732334090105053,
+            -0.016237731209426752,
+            0.045334690225148,
+            -0.016911928942220152,
+            -0.0021139311364009113,
+            -0.01058970056572469,
+            -0.021705338531028618,
+            0.007324792557651635,
+            -0.006556582104686972,
+            0.0018243188317438144,
+            0.010780292738415671,
+            -0.006471850255151019,
+            0.0006997395548596146,
+            0.003907433192252573,
+            -0.023129313269763224,
+            -0.002550875642832235,
+            -0.02237917712764273,
+            -0.003107775639788617,
+            -0.02621608419628401,
+            -0.00012482032027919036,
+            -0.016207377503696517,
+            -0.0038712739995960753,
+            -0.023315508569288476,
+            -0.014963211090562272,
+            -0.01633110043829169,
+            -0.004241081909202262,
+            -0.008295402942465353,
+            0.004976396647108597,
+            -0.007402107755970892,
+            -0.020321297548927468,
+            0.0020986169430507757,
+            0.016803273183070004,
+            0.01434729320047051,
+            0.018612970083178984,
+            0.00603868809307069,
+            0.014278001198851581,
+            -0.0026350854954985117,
+            0.017171016209107656,
+            -0.006968968261909459,
+            0.006881847166565678,
+            0.011454001418121107,
+            0.02140935294963631,
+            0.011121833270235448,
+            -0.012801756637470525,
+            0.0010870041670931706,
+            -0.009655388034828094,
+            0.002094601818569396,
+            -0.00828486034340067,
+            -0.009190975158350585,
+            0.0017762581946855365,
+            -0.007261996868906865,
+            0.004674405202069334,
+            0.0030526253833799426,
+            -0.0014082232541965316,
+            0.010419302200840906,
+            -0.0024200053377751805,
+            -0.012208685875688886,
+            -0.0033037330470612695,
+            -0.017975940046938498,
+            -0.006958042903103732,
+            -0.0014496522238488239,
+            -0.011711502944602638,
+            -0.01016130418405875,
+            0.003321436189059608,
+            -0.00538282506552848,
+            -0.009353581612407435,
+            0.013884147453136417,
+            0.0001309738696853529,
+            0.0005878330075918333,
+            0.023495280602974504,
+            0.006035028203642483,
+            0.008087594627921345,
+            0.013144288945385904,
+            -0.01023995000994454,
+            -0.022574958230635715,
+            -0.0243435727055662,
+            0.007867022215096418,
+            -0.01573478573423379,
+            0.02324699908478915,
+            -0.007929809922043611,
+            -0.00561630876431978,
+            0.0020982723335057625,
+            -0.013699317926847062,
+            0.008578431496569186,
+            -0.011271381061801356,
+            0.0014450645788554738,
+            -0.019581451799757368,
+            0.018059311800267413,
+            -0.010917911869373693,
+            -0.004302687883018328,
+            0.009466103226841532,
+            0.034049086327052606,
+            -0.0005689605174080737,
+            -0.007410970644432269,
+            -0.006594150877740108,
+            -0.00022810271313786974,
+            0.018766765773409123,
+            -0.0029041815756733333,
+            0.005758781908059541,
+            -0.01563519107805783,
+            0.006571628287171105,
+            -0.006132372252825422,
+            0.0012423261344297225,
+            0.01513074831647933,
+            -0.026306320707079913,
+            -0.015491710256868008,
+            -0.007857352164317122,
+            -0.024893430640551818,
+            0.002253574511878983,
+            0.017579477884206534,
+            0.020505606775521504,
+            -0.020324505029926956,
+            -0.02513332780748072,
+            0.005150966733930881,
+            -0.026580994694344065,
+            0.0037390577082486594,
+            0.1241031878517593,
+            -0.002138404898309707,
+            0.002720723037908521,
+            -0.011425036886011179,
+            0.0022399461009316216,
+            0.005389407121650262,
+            -0.019552543955456567,
+            0.05950746473186985,
+            -0.00638023587296763,
+            -0.004023798021748334,
+            0.002170545188776296,
+            0.001785855390676113,
+            -0.005831534307818096,
+            0.008643501093778434,
+            -0.017602787281866816,
+            -0.02184792441891879,
+            -0.0052215705748838905,
+            -0.021252169339074325,
+            -0.008509621422071373,
+            -0.013000137524907565,
+            0.002893880203681282,
+            -0.015821609145923173,
+            0.010397692426614709,
+            -0.01638916243796692,
+            -0.010283706937701179,
+            -0.01103621454761223,
+            -0.00373552043529251,
+            0.01563553822905027,
+            -0.03188533919653949,
+            -0.004873550301425776,
+            -0.005183590054152726,
+            0.018218971950313273,
+            0.006004328765748325,
+            0.004357734370247252,
+            -0.014230318971458851,
+            0.0064792070801456235,
+            0.0215589173714926,
+            0.0055272493764401305,
+            0.014643335833970546,
+            0.010618070641357003,
+            -0.0022177303813575846,
+            -0.0024402761885758007,
+            0.017541799738501007,
+            -0.005445774414228888,
+            -0.00023502936119626386,
+            -0.00537569046685066,
+            -0.005974703247446444,
+            -0.010712244203425036,
+            0.01843613166635215,
+            -0.009938564643456672,
+            -0.017388592763277214,
+            -0.0015797412875366156,
+            0.02028434083164951,
+            -0.012761534382003973,
+            0.0031906812114950982,
+            0.000788436210275669,
+            0.020589196433948156,
+            0.0007726855314648301,
+            -0.005481260996289263,
+            0.007165014826294827,
+            -0.02956334397512344,
+            0.004095715232997753,
+            0.0003568512047626187,
+            0.027290768799658912,
+            -0.0017631129386615335,
+            0.01552542586826711,
+            -0.020871286597113327,
+            0.004388125824324094,
+            -0.016926008631435403,
+            0.005099899758322398,
+            0.012560923968841436,
+            0.002781049890810361,
+            0.010889021753593997,
+            -0.008500358897706648,
+            -0.021353460542554444,
+            0.012663934009161902,
+            -0.018657094130860155,
+            -0.029156017587920958,
+            0.0018889388761887053,
+            -0.028158076315864772,
+            -0.003048854609750186,
+            0.006812072971079798,
+            -0.035217412591558014,
+            -0.02251675180883659,
+            0.007396524512996277,
+            -0.00995571600201846,
+            -0.011912753666073882,
+            0.010675240017836214,
+            -0.02645155743041798,
+            -0.018137648527851668,
+            0.0010666944556449141,
+            0.012588800077165337,
+            -0.004561825939541096,
+            0.007555558090958502,
+            -0.028558603070876605,
+            0.014816816621445352,
+            -0.01313764261744744,
+            0.016036065063831813,
+            -0.0037797584923144134,
+            0.022217516257304868,
+            0.011131597210722573,
+            0.008795376259031849,
+            -0.004950080558710245,
+            0.021409590863550987,
+            0.005574351961953293,
+            0.051358937509801736,
+            0.031972009383746405,
+            0.0019821003238211276,
+            -0.003251257665378234,
+            0.003313045083311649,
+            0.008294776423730887,
+            -0.00940948257229233,
+            0.005114631360856872,
+            0.028053227377164806,
+            0.016738013668710083,
+            -0.018755924318109216,
+            0.006606711923035118,
+            -0.02702692774986498,
+            0.017950884766781056,
+            -0.022054653845163,
+            0.0003755944703013671,
+            0.007766253634286795,
+            0.0016206993755471613,
+            0.010682347575070117,
+            0.0013842930079185415,
+            0.01705234255261425,
+            0.023975191051210525,
+            -0.002976402980885077,
+            0.017466027107477605,
+            0.012734909275073793,
+            0.018143161773162015,
+            -0.0038399373197265827,
+            0.006929440921265761,
+            -0.012003095293274786,
+            -0.0008883470597690258,
+            0.0020490527756307267,
+            -0.004231091383919565,
+            0.012128848448049833,
+            0.008340302519160299,
+            0.012989849936937843,
+            0.002974488532907779,
+            0.003198854381461613,
+            -0.005309753760578859,
+            -0.0017024982049936384,
+            -0.01763395035509857,
+            -0.01819680226734474,
+            0.028417724274372468,
+            -0.020217879775485576,
+            -0.050118184195774004,
+            0.0025509548991664425,
+            0.014185761006969567,
+            -0.0012021669304388173,
+            -0.0051260576846881515,
+            -0.0005774778872927327,
+            0.005718864494530703,
+            -0.016918749210038116,
+            -0.0015472638835434902,
+            0.008125845907627804,
+            0.020347547062899797,
+            -0.002036218493206914,
+            -0.01079739518866248,
+            0.0038112958713600906,
+            0.02169339303105242,
+            0.01667594322285835,
+            0.012338193092567552,
+            -0.009798409233733167,
+            0.0003198215558702285,
+            -1.6308740825163337e-05,
+            0.0076870827874737325,
+            0.004356644356656038,
+            0.003500023005601613,
+            -0.019968549869439228,
+            -0.008634740687755112,
+            0.012781565963545847,
+            -0.0025637758121512927,
+            0.010243567793632335,
+            -0.00042296044205132426,
+            -0.003069251553118065,
+            -0.009344199449486244,
+            -0.004694953215957636,
+            -0.009583993744231791,
+            0.013783878036521561,
+            -0.007705982042252044,
+            -0.01549567538665377,
+            0.00020666298495823114,
+            0.0005515775451348472,
+            0.001667367278489879,
+            -0.004518431526236256,
+            0.03302324454129141,
+            0.009531990824506103,
+            -0.0004937348108340977,
+            -0.004047613286672229,
+            -0.0327383112522095,
+            0.012456150771224677,
+            -0.009677533214778067,
+            0.0006455811519981003,
+            0.020012976827426605,
+            -0.00020277620481825933,
+            0.002168432791818704,
+            -0.015657551629104767,
+            0.0032924357559238502,
+            0.006161295735841883,
+            0.013814165416788555,
+            0.0005164426482972658,
+            0.01076512534105537,
+            -0.006498977875968572,
+            -0.010325144017898257,
+            0.004804158067334203,
+            0.0104064855510473,
+            -0.0012188100892986933,
+            0.005171197790296134
+          ],
+          "title": "Mixture-Of-Experts With Expert Choice Routing",
+          "keyphrases": [
+            "Mixture-Of-Experts",
+            "Expert Choice Routing",
+            "sparsely-activated models",
+            "training efficiency",
+            "downstream performance"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "33412f87-7418-4ce2-9116-377a27fd2732",
+        "properties": {
+          "page_content": "4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n",
+          "title": "4 Experiments",
+          "embedding": [
+            0.009938324801623821,
+            -0.009144282899796963,
+            -0.026211904361844063,
+            -0.022352691739797592,
+            0.016043050214648247,
+            0.016247965395450592,
+            -0.02783413790166378,
+            0.0537557490170002,
+            -0.02827811799943447,
+            -0.01755429059267044,
+            0.010237157344818115,
+            0.001297788112424314,
+            0.014762339182198048,
+            -0.01001516729593277,
+            0.007487895432859659,
+            0.015283161774277687,
+            0.019347287714481354,
+            0.019910801202058792,
+            0.02535809576511383,
+            -0.04251963645219803,
+            0.0020213902462273836,
+            -0.032034873962402344,
+            -0.058332156389951706,
+            0.0034600566141307354,
+            0.001006959704682231,
+            0.01755429059267044,
+            0.011645940132439137,
+            -0.0015795447397977114,
+            -0.053584985435009,
+            0.04794985428452492,
+            0.016555335372686386,
+            0.0044910297729074955,
+            -0.007833687588572502,
+            -0.004798400681465864,
+            -0.04204150289297104,
+            -0.028961164876818657,
+            0.05720513314008713,
+            0.005165537819266319,
+            -0.005519868340343237,
+            0.05481446906924248,
+            0.020252324640750885,
+            0.010245694778859615,
+            -0.03961668908596039,
+            -0.009904172271490097,
+            0.010134699754416943,
+            0.001490962109528482,
+            -0.0010427129454910755,
+            -0.007479357533156872,
+            0.02295035682618618,
+            -0.04146091267466545,
+            -0.005549751687794924,
+            -0.03811398521065712,
+            0.01863008923828602,
+            0.013780459761619568,
+            -0.04426140338182449,
+            0.023223575204610825,
+            -0.0052978782914578915,
+            0.03527934476733208,
+            0.057854026556015015,
+            0.02783413790166378,
+            0.024162763729691505,
+            -0.02590453252196312,
+            -0.04340759664773941,
+            -0.0036180110182613134,
+            0.010143238119781017,
+            0.008119713515043259,
+            -0.01344747468829155,
+            -0.010527451522648335,
+            0.046173933893442154,
+            -0.023701708763837814,
+            -0.05327761545777321,
+            0.027680452913045883,
+            -0.013584083877503872,
+            -0.008081291802227497,
+            0.0036372218746691942,
+            -0.03517688810825348,
+            0.012807118706405163,
+            0.03548425808548927,
+            0.01004078146070242,
+            0.015018481761217117,
+            0.001224147155880928,
+            0.008418546058237553,
+            -0.0014920294051989913,
+            0.017912890762090683,
+            0.028517184779047966,
+            -0.01417321152985096,
+            -0.006309640593826771,
+            -0.0686461552977562,
+            0.025767924264073372,
+            -0.007368362508714199,
+            -0.020115714520215988,
+            0.00026281276950612664,
+            -0.007099412847310305,
+            0.0033597343135625124,
+            -0.020781684666872025,
+            0.07472527027130127,
+            -0.013105952180922031,
+            0.014386664144694805,
+            -0.02096952311694622,
+            0.007300057914108038,
+            -0.031932417303323746,
+            0.02236976847052574,
+            0.01146664097905159,
+            0.005916888825595379,
+            0.038318898528814316,
+            -0.02834642305970192,
+            -0.005869929678738117,
+            0.0122009152546525,
+            0.0292856115847826,
+            0.08387809246778488,
+            0.01419028826057911,
+            -0.014352511614561081,
+            -0.025563009083271027,
+            -0.016538260504603386,
+            -0.046105626970529556,
+            -0.0292856115847826,
+            -0.025170257315039635,
+            -0.027099864557385445,
+            0.036338064819574356,
+            -0.0010613900376483798,
+            0.012388752773404121,
+            0.05338007211685181,
+            0.015863751992583275,
+            -0.00874299369752407,
+            -0.039241012185811996,
+            0.0438857264816761,
+            -0.006198645569384098,
+            0.010809208266437054,
+            0.002330895746126771,
+            -0.0146428057923913,
+            -0.006185838486999273,
+            0.011833777651190758,
+            -0.013763383962213993,
+            -0.029456373304128647,
+            0.031949494034051895,
+            -0.01727253384888172,
+            0.03944592550396919,
+            0.06239628419280052,
+            -0.03562086820602417,
+            -0.02001325786113739,
+            0.03189826384186745,
+            -0.035825781524181366,
+            0.018288565799593925,
+            -0.0005250918911769986,
+            -0.030993228778243065,
+            0.026246055960655212,
+            0.01562468521296978,
+            0.06092773377895355,
+            -0.024196917191147804,
+            0.028705023229122162,
+            -0.019176525995135307,
+            0.01628211699426174,
+            -0.020627999678254128,
+            0.03596239164471626,
+            0.0023351646959781647,
+            -0.024265222251415253,
+            -0.015752756968140602,
+            0.014694035053253174,
+            -0.014275669120252132,
+            -0.022830823436379433,
+            0.019740039482712746,
+            0.02100367471575737,
+            -0.010450609028339386,
+            -0.0049051265232264996,
+            0.012013077735900879,
+            0.04989653453230858,
+            0.016205275431275368,
+            -0.014130521565675735,
+            -0.07479357719421387,
+            0.04801815748214722,
+            0.021703798323869705,
+            0.01958635449409485,
+            0.023104043677449226,
+            0.035825781524181366,
+            -0.001056053675711155,
+            0.030566323548555374,
+            0.0019007899099960923,
+            -0.013601160608232021,
+            -0.02044016122817993,
+            0.043646663427352905,
+            -0.0039019023533910513,
+            -0.0019274713704362512,
+            0.0076373121701180935,
+            0.018288565799593925,
+            0.03123229369521141,
+            0.005430218297988176,
+            0.010399380698800087,
+            0.039719145745038986,
+            -0.011586173437535763,
+            0.01414759736508131,
+            0.009460192173719406,
+            0.02091829478740692,
+            0.04299776628613472,
+            0.02245514839887619,
+            -0.008670419454574585,
+            -0.02194286324083805,
+            0.00032551426556892693,
+            0.0219599399715662,
+            0.01811780408024788,
+            0.02199409157037735,
+            -0.012013077735900879,
+            -0.0220111683011055,
+            0.009383349679410458,
+            -0.02199409157037735,
+            0.01682855375111103,
+            -0.00167666538618505,
+            0.01859593763947487,
+            0.013865840621292591,
+            -0.006258412264287472,
+            -0.01677732542157173,
+            0.0002502724528312683,
+            -0.018186109140515327,
+            -0.008380125276744366,
+            -0.010945817455649376,
+            -0.016495568677783012,
+            0.03364003449678421,
+            -0.020747533068060875,
+            0.006599935237318277,
+            -0.01810072734951973,
+            -0.011885006912052631,
+            -0.005835777148604393,
+            0.03596239164471626,
+            0.012209453620016575,
+            -0.01851055584847927,
+            -0.026058217510581017,
+            -0.011108040809631348,
+            -0.018817927688360214,
+            -0.006463326048105955,
+            0.0342206209897995,
+            -0.01750306226313114,
+            0.006659701932221651,
+            -0.004211407620459795,
+            -0.002409872831776738,
+            0.004258367232978344,
+            0.033435117453336716,
+            -0.03082246705889702,
+            -0.01676025055348873,
+            0.023582175374031067,
+            0.008059946820139885,
+            0.029046546667814255,
+            -0.016888320446014404,
+            -0.006275488529354334,
+            -0.024231068789958954,
+            -0.016973702237010002,
+            -0.01643580198287964,
+            -0.005118578672409058,
+            -0.02199409157037735,
+            -0.016025975346565247,
+            0.02790244296193123,
+            0.013686541467905045,
+            -0.00790199264883995,
+            -0.019313136115670204,
+            -0.002333030104637146,
+            0.02542640082538128,
+            -0.013592622242867947,
+            0.04064125940203667,
+            0.050545431673526764,
+            -0.05713682621717453,
+            -0.0122521435841918,
+            -0.03661128506064415,
+            0.07684271782636642,
+            -0.015377080999314785,
+            -0.03032725863158703,
+            0.033964481204748154,
+            -0.029388070106506348,
+            -0.008990597911179066,
+            0.026997406035661697,
+            0.025272715836763382,
+            -0.014719649218022823,
+            0.01123611256480217,
+            0.03756754845380783,
+            -0.030139420181512833,
+            -0.0439881831407547,
+            -0.03777246177196503,
+            0.019859572872519493,
+            -0.025289790704846382,
+            -0.01419028826057911,
+            -0.0110311983153224,
+            0.023530947044491768,
+            0.01801534742116928,
+            0.010664060711860657,
+            0.007547662127763033,
+            0.006843270733952522,
+            0.021157361567020416,
+            0.02144765481352806,
+            0.007073798682540655,
+            -0.03237639740109444,
+            0.0003433908859733492,
+            -0.03944592550396919,
+            -0.009093054570257664,
+            -0.025289790704846382,
+            -0.02095244638621807,
+            -0.023821242153644562,
+            0.00949434470385313,
+            0.03275207430124283,
+            -0.0008143192972056568,
+            0.0067365444265306,
+            0.015044095925986767,
+            0.015488076023757458,
+            -0.03807983547449112,
+            -0.03555256128311157,
+            0.006348061840981245,
+            0.01682855375111103,
+            -0.0055625587701797485,
+            -0.059220120310783386,
+            -0.00042770442087203264,
+            -0.003165493020787835,
+            -0.005016121547669172,
+            -0.028909936547279358,
+            0.010476223193109035,
+            0.013345018029212952,
+            0.010237157344818115,
+            0.0110311983153224,
+            -0.027526767924427986,
+            -0.014651344157755375,
+            -0.0036073385272175074,
+            0.006019345950335264,
+            0.02100367471575737,
+            -0.040709562599658966,
+            -0.022830823436379433,
+            0.03818229213356972,
+            0.0003810117777902633,
+            -0.06598227471113205,
+            -0.005387527868151665,
+            -0.010757979936897755,
+            0.015436847694218159,
+            -0.0005656477296724916,
+            0.034476764500141144,
+            -0.03121521882712841,
+            -0.019296059384942055,
+            -0.020815838128328323,
+            -0.051979828625917435,
+            -0.03531349450349808,
+            -0.015436847694218159,
+            0.007389707490801811,
+            -0.038353051990270615,
+            -0.024265222251415253,
+            0.004111085087060928,
+            -0.0268949493765831,
+            0.007398245856165886,
+            -0.013567008078098297,
+            0.015633223578333855,
+            -0.03852381557226181,
+            -0.010092009790241718,
+            0.004262635949999094,
+            -0.01861301250755787,
+            -0.005507061257958412,
+            0.014369587413966656,
+            0.013925607316195965,
+            0.014719649218022823,
+            0.0017268265364691615,
+            0.015266085974872112,
+            0.0062071834690868855,
+            -0.00769280968233943,
+            0.007833687588572502,
+            -0.04091447591781616,
+            0.0014461372047662735,
+            0.018852079287171364,
+            0.011432488448917866,
+            -0.011082426644861698,
+            0.004151640925556421,
+            -0.010373766534030437,
+            -0.03034433349967003,
+            0.029917430132627487,
+            -0.05419972911477089,
+            0.0024141420144587755,
+            0.005554020404815674,
+            -0.016410188749432564,
+            -0.008384393528103828,
+            -0.009690719656646252,
+            -0.03708941861987114,
+            -0.01248267199844122,
+            0.0039019023533910513,
+            0.00019050588889513165,
+            0.009349197149276733,
+            0.002870929194614291,
+            0.05471201241016388,
+            -0.003165493020787835,
+            -0.04484199360013008,
+            0.023308956995606422,
+            -0.03671374171972275,
+            0.018749622628092766,
+            0.04245132952928543,
+            0.020713381469249725,
+            -0.017212767153978348,
+            0.04791570082306862,
+            0.0195692777633667,
+            0.000224524803343229,
+            -0.016999315470457077,
+            0.025153182446956635,
+            -0.007099412847310305,
+            -0.061303410679101944,
+            -0.004448339343070984,
+            -0.018749622628092766,
+            0.018664240837097168,
+            0.016674868762493134,
+            -0.002407738473266363,
+            -0.010032243095338345,
+            0.009426039643585682,
+            -0.01510386262089014,
+            -0.017443295568227768,
+            0.01705908216536045,
+            -0.011458102613687515,
+            -0.004525181837379932,
+            0.007526317145675421,
+            0.013148642145097256,
+            0.01271320041269064,
+            0.008853988721966743,
+            0.023992003872990608,
+            -0.045627497136592865,
+            0.009127207100391388,
+            -0.01997910626232624,
+            -0.036918655037879944,
+            -0.002990462351590395,
+            0.01318279467523098,
+            0.04183658957481384,
+            -0.02779998630285263,
+            0.0019253368955105543,
+            0.005007583647966385,
+            -0.013515779748558998,
+            -0.021106131374835968,
+            0.00341309723444283,
+            0.04600317031145096,
+            -0.0015763428527861834,
+            0.021618416532874107,
+            -0.011304416693747044,
+            -0.03562086820602417,
+            0.008811297826468945,
+            0.04142676293849945,
+            -0.005028928630053997,
+            0.01663217879831791,
+            0.009434578008949757,
+            0.03548425808548927,
+            -0.04590071365237236,
+            -0.007833687588572502,
+            -0.02344556525349617,
+            -0.01813488081097603,
+            0.04105108603835106,
+            -0.03075416199862957,
+            -0.01365238893777132,
+            0.03524519130587578,
+            -0.005306416191160679,
+            0.02441890724003315,
+            -0.018271489068865776,
+            -0.02588745579123497,
+            0.0014600115828216076,
+            -0.03712356835603714,
+            -0.00708660576492548,
+            -0.01806657575070858,
+            -0.006160224322229624,
+            -0.06557244807481766,
+            0.00010459146869834512,
+            -0.07431544363498688,
+            -0.002902946900576353,
+            0.025136105716228485,
+            -0.006552976090461016,
+            0.04429555684328079,
+            -0.026228979229927063,
+            0.02098659984767437,
+            -0.05553166940808296,
+            -0.01367800310254097,
+            0.00978463888168335,
+            0.03375956416130066,
+            0.007611697539687157,
+            -0.001995776081457734,
+            0.0015507286880165339,
+            -0.026997406035661697,
+            -0.0018303507240489125,
+            -0.007282981649041176,
+            -0.001387437921948731,
+            4.89605481561739e-05,
+            -0.018852079287171364,
+            0.00950288213789463,
+            0.000838332693092525,
+            0.01610281690955162,
+            0.031454283744096756,
+            0.022711291909217834,
+            -0.03070293366909027,
+            -0.008862526156008244,
+            0.005195421166718006,
+            0.02587038092315197,
+            -0.02978082001209259,
+            0.003897633170709014,
+            -0.03464752435684204,
+            0.014864795841276646,
+            5.636466448777355e-05,
+            0.016657792031764984,
+            0.008008718490600586,
+            0.017016392201185226,
+            -0.040197279304265976,
+            0.0030160765163600445,
+            -0.0016254368238151073,
+            -0.027202321216464043,
+            0.020867066457867622,
+            0.002305281348526478,
+            -0.022147778421640396,
+            0.0054472945630550385,
+            -0.004207138437777758,
+            0.024213993921875954,
+            0.04473953694105148,
+            -0.0010592554463073611,
+            -0.02930268831551075,
+            -0.012704662047326565,
+            0.042212262749671936,
+            0.02633143588900566,
+            -0.028483033180236816,
+            0.017118848860263824,
+            0.02103782817721367,
+            -0.02341141365468502,
+            -0.021089056506752968,
+            0.013925607316195965,
+            -0.016179660335183144,
+            -0.0007694943924434483,
+            -0.005468640010803938,
+            0.010399380698800087,
+            0.01240582950413227,
+            -0.029883278533816338,
+            0.05676115304231644,
+            -0.016820017248392105,
+            -0.006834732368588448,
+            -0.03034433349967003,
+            -0.006168762221932411,
+            -0.014531810767948627,
+            0.05553166940808296,
+            0.0028175662737339735,
+            0.02728770114481449,
+            0.01658095046877861,
+            0.002264725510030985,
+            0.05628301948308945,
+            -0.03678204491734505,
+            -0.03451091796159744,
+            0.018681317567825317,
+            -0.0055625587701797485,
+            -0.00671093026176095,
+            -0.022728366777300835,
+            -0.014207364059984684,
+            -0.013353556394577026,
+            0.016965163871645927,
+            -0.0268778745085001,
+            -0.012260681949555874,
+            0.019740039482712746,
+            0.017690900713205338,
+            -2.0469710761972237e-06,
+            0.02293328195810318,
+            0.012644895352423191,
+            0.014284206554293633,
+            0.017878737300634384,
+            -0.020867066457867622,
+            0.019142374396324158,
+            0.024111535400152206,
+            0.009946862235665321,
+            -0.01763967238366604,
+            -0.017878737300634384,
+            0.007347017526626587,
+            -0.0038613462820649147,
+            -0.007125027012079954,
+            0.01437812577933073,
+            0.007043915335088968,
+            -0.004047049675136805,
+            -0.027151092886924744,
+            -0.018356870859861374,
+            0.02093537151813507,
+            -0.0023757205344736576,
+            0.01539415679872036,
+            -0.00928943045437336,
+            -0.010612832382321358,
+            0.03763585537672043,
+            0.01861301250755787,
+            -0.046139780431985855,
+            0.026280207559466362,
+            0.01051037572324276,
+            -0.006228528916835785,
+            -0.013729231432080269,
+            0.005626594182103872,
+            0.009101592935621738,
+            0.010220080614089966,
+            0.016572412103414536,
+            -0.005097233224660158,
+            -0.03328143432736397,
+            0.01564176194369793,
+            0.026211904361844063,
+            -0.020798761397600174,
+            0.012021616101264954,
+            0.00017663151083979756,
+            0.022762520238757133,
+            -0.005028928630053997,
+            -0.012311910279095173,
+            -0.01562468521296978,
+            0.009741948917508125,
+            -0.012235067784786224,
+            0.01123611256480217,
+            0.037396788597106934,
+            0.015914980322122574,
+            -0.03565501794219017,
+            0.015735680237412453,
+            -0.01686270721256733,
+            -0.015684451907873154,
+            0.03261546418070793,
+            0.008264861069619656,
+            -0.02636558935046196,
+            -0.007142103277146816,
+            -0.006527361460030079,
+            0.02001325786113739,
+            -0.02102075144648552,
+            -0.008508196100592613,
+            0.004418455995619297,
+            0.0035646480973809958,
+            0.009938324801623821,
+            0.006745082791894674,
+            0.005016121547669172,
+            0.019159449264407158,
+            0.013806073926389217,
+            -0.0005186883499845862,
+            0.012730276212096214,
+            0.011577636003494263,
+            0.002902946900576353,
+            0.0075391242280602455,
+            -0.010997045785188675,
+            -0.02295035682618618,
+            0.004734364803880453,
+            0.05945918336510658,
+            -0.005852853413671255,
+            0.03876287862658501,
+            0.010407919064164162,
+            0.028961164876818657,
+            0.021686721593141556,
+            -0.024299373850226402,
+            -0.01681147888302803,
+            -0.018151957541704178,
+            -0.021737949922680855,
+            -0.04453462362289429,
+            -0.01510386262089014,
+            -0.01629919372498989,
+            -0.013404784724116325,
+            0.005694898776710033,
+            -0.00013474156730808318,
+            -0.0013116624904796481,
+            0.011355645954608917,
+            -0.002003246918320656,
+            -0.012491210363805294,
+            -0.003355465130880475,
+            0.05324346199631691,
+            -0.0011056812945753336,
+            0.017912890762090683,
+            -0.004576410632580519,
+            0.015539304353296757,
+            -0.007300057914108038,
+            -0.016111355274915695,
+            0.029131926596164703,
+            0.0035561099648475647,
+            -0.0007540191290900111,
+            0.012106996960937977,
+            0.004469684325158596,
+            0.04149506613612175,
+            -0.005887005943804979,
+            -0.026058217510581017,
+            0.011910621076822281,
+            -0.017187153920531273,
+            0.020371858030557632,
+            0.00769707839936018,
+            -0.007607428822666407,
+            0.001932807732373476,
+            0.012909576296806335,
+            -0.027116939425468445,
+            0.019705887883901596,
+            -0.020645076408982277,
+            -0.045149363577365875,
+            0.009280892089009285,
+            -0.01175693515688181,
+            0.023035738617181778,
+            0.001117421081289649,
+            -0.010800670832395554,
+            -0.03507443144917488,
+            0.0021387888118624687,
+            0.0039040367119014263,
+            -0.03372541442513466,
+            0.006548706907778978,
+            0.01863008923828602,
+            0.015539304353296757,
+            0.009485806338489056,
+            -0.012465596199035645,
+            0.003758889390155673,
+            0.0031847036443650723,
+            -0.012371676973998547,
+            0.02776583470404148,
+            -0.004742903169244528,
+            -0.031983643770217896,
+            0.035859934985637665,
+            -0.02733892947435379,
+            0.009468729607760906,
+            -0.020730456337332726,
+            -0.012346062809228897,
+            -0.01608574204146862,
+            0.00441418681293726,
+            -0.02148180827498436,
+            -0.002561423694714904,
+            -0.02983205020427704,
+            0.027577996253967285,
+            0.009383349679410458,
+            -0.015249010175466537,
+            -0.004862436093389988,
+            0.009101592935621738,
+            -0.025255639106035233,
+            0.007846495136618614,
+            0.012585128657519817,
+            -0.010646984912455082,
+            -0.01367800310254097,
+            -0.00706099160015583,
+            -0.0014685495989397168,
+            0.005972386337816715,
+            0.0158210601657629,
+            0.04672037065029144,
+            0.004431263078004122,
+            0.03712356835603714,
+            0.02337726205587387,
+            0.013532855547964573,
+            0.0037012575194239616,
+            0.0047898623161017895,
+            -0.0067322757095098495,
+            0.04012897238135338,
+            0.006249873898923397,
+            0.0219599399715662,
+            0.008247784338891506,
+            -0.044671230018138885,
+            0.008503926917910576,
+            0.03264961391687393,
+            0.026621730998158455,
+            0.018715469166636467,
+            0.009588262997567654,
+            -0.0024013349320739508,
+            0.021191513165831566,
+            -0.020252324640750885,
+            -0.01705908216536045,
+            0.0035902622621506453,
+            -0.007530585862696171,
+            -0.0056650154292583466,
+            -0.009716334752738476,
+            0.03387909755110741,
+            -0.003229528432711959,
+            -0.008862526156008244,
+            0.018186109140515327,
+            -0.007432397920638323,
+            0.00475997943431139,
+            0.006245605181902647,
+            0.0011163539020344615,
+            0.02739015780389309,
+            0.008683227002620697,
+            0.027680452913045883,
+            0.005391797050833702,
+            0.007573276292532682,
+            0.0058571225963532925,
+            -0.009451653808355331,
+            0.018920384347438812,
+            -0.0097590247169137,
+            -0.033366814255714417,
+            -0.009443115442991257,
+            0.0016393112018704414,
+            0.014070754870772362,
+            0.0004023569927085191,
+            -0.014062216505408287,
+            -0.011893544346094131,
+            0.007859301753342152,
+            0.012533900327980518,
+            0.02107197977602482,
+            0.018169032409787178,
+            -0.005016121547669172,
+            -0.024726277217268944,
+            0.012593667022883892,
+            0.011056812480092049,
+            -0.01861301250755787,
+            0.008465506136417389,
+            0.012542438693344593,
+            0.057410046458244324,
+            -0.044705383479595184,
+            0.016538260504603386,
+            0.02151595987379551,
+            -0.004896588623523712,
+            -0.015249010175466537,
+            -0.010851899161934853,
+            0.0031633584294468164,
+            -0.017477449029684067,
+            0.014975790865719318,
+            -0.01798119582235813,
+            0.011193421669304371,
+            -0.014745263382792473,
+            0.00743666710332036,
+            0.012866885401308537,
+            -0.009682182222604752,
+            0.003942457959055901,
+            0.020371858030557632,
+            0.0028068935498595238,
+            -0.0077867284417152405,
+            0.005387527868151665,
+            0.01390853151679039,
+            0.020354781299829483,
+            0.006066305562853813,
+            -0.0005933965439908206,
+            0.019876649603247643,
+            -0.028175661340355873,
+            -0.025170257315039635,
+            0.01909114606678486,
+            -0.03551841154694557,
+            0.005472908727824688,
+            0.02790244296193123,
+            -0.04296361654996872,
+            -0.05020390823483467,
+            0.0027407235465943813,
+            -0.039719145745038986,
+            0.0004191663465462625,
+            -0.0023287611547857523,
+            0.01536854263395071,
+            -0.004057722166180611,
+            -0.017212767153978348,
+            0.0146428057923913,
+            -0.012397291138768196,
+            -0.0067322757095098495,
+            0.009835867211222649,
+            0.009016212075948715,
+            -0.024572592228651047,
+            -0.0016222350532189012,
+            -0.030890770256519318,
+            0.009169897064566612,
+            -0.017844585701823235,
+            -0.019142374396324158,
+            -0.025528857484459877,
+            -0.0005656477296724916,
+            -0.010638446547091007,
+            -0.001638244022615254,
+            0.051501695066690445,
+            0.021652569994330406,
+            0.020166942849755287,
+            -0.007726961746811867,
+            -0.023650480434298515,
+            0.05457540228962898,
+            0.0018132745753973722,
+            0.03814813867211342,
+            -0.012380215339362621,
+            -0.0002006448630709201,
+            -0.0008852921309880912,
+            0.022113624960184097,
+            -0.017101772129535675,
+            -0.0011547752656042576,
+            -0.011022660881280899,
+            -0.00586566049605608,
+            -0.0070780678652226925,
+            0.014045140706002712,
+            0.015974747017025948,
+            -0.002610517665743828,
+            0.014156135730445385,
+            0.01221799198538065,
+            0.022694215178489685,
+            0.03130060061812401,
+            0.016230888664722443,
+            -0.0034664601553231478,
+            -0.022199006751179695,
+            -0.012918113730847836,
+            -0.008200825192034245,
+            0.008999135345220566,
+            -0.008525271899998188,
+            -0.0031847036443650723,
+            -0.02631436102092266,
+            -0.020730456337332726,
+            -0.011312955059111118,
+            -0.01292665209621191,
+            -0.0031462821643799543,
+            0.02641681768000126,
+            -0.002469639526680112,
+            -0.03763585537672043,
+            -0.01511240005493164,
+            -0.03524519130587578,
+            -0.015607609413564205,
+            -0.0017876604106277227,
+            -0.01748598739504814,
+            0.016538260504603386,
+            -0.003780234605073929,
+            -0.007833687588572502,
+            -0.0008964983280748129,
+            -0.018169032409787178,
+            0.009690719656646252,
+            -0.011048275046050549,
+            -0.000543768925126642,
+            -0.0017503062263131142,
+            0.013345018029212952,
+            -0.008205094374716282,
+            0.008973521180450916,
+            -0.01412198320031166,
+            0.006868884898722172,
+            -0.048120614141225815,
+            -0.027731681242585182,
+            -0.027099864557385445,
+            0.022557605057954788,
+            -0.015214857645332813,
+            -0.023189423605799675,
+            0.008734455332159996,
+            -0.013686541467905045,
+            0.008939368650317192,
+            -0.012021616101264954,
+            0.008388662710785866,
+            0.019210679456591606,
+            0.009024749509990215,
+            -0.002614786848425865,
+            -0.013669464737176895,
+            -0.02390662208199501,
+            -0.02585330419242382,
+            0.023650480434298515,
+            -0.026485122740268707,
+            0.01071528997272253,
+            -0.012021616101264954,
+            0.008585038594901562,
+            -0.029917430132627487,
+            -0.020730456337332726,
+            0.0044910297729074955,
+            0.011347107589244843,
+            -0.021140284836292267,
+            0.044671230018138885,
+            0.04043634235858917,
+            0.029985735192894936,
+            -0.020252324640750885,
+            0.05139923840761185,
+            -0.022796671837568283,
+            0.007312864996492863,
+            0.036440521478652954,
+            0.030446790158748627,
+            0.026092370972037315,
+            -0.015581995248794556,
+            -0.012294834479689598,
+            -0.015496614389121532,
+            0.013404784724116325,
+            -0.024333525449037552,
+            0.002821835223585367,
+            -0.02042308636009693,
+            0.013643850572407246,
+            0.03451091796159744,
+            -0.004917933605611324,
+            0.029934506863355637,
+            -0.013720693998038769,
+            0.030361410230398178,
+            0.0017652479000389576,
+            -0.026587579399347305,
+            0.015667375177145004,
+            -0.014232978224754333,
+            0.004418455995619297,
+            0.017964119091629982,
+            -0.006087650544941425,
+            0.013575546443462372,
+            -0.061337560415267944,
+            0.028141509741544724,
+            0.01758844405412674,
+            0.01123611256480217,
+            -0.002180411946028471,
+            -0.024726277217268944,
+            -0.0021142419427633286,
+            0.009340658783912659,
+            -0.0016745307948440313,
+            0.011859391815960407,
+            0.01193623524159193,
+            -0.033418044447898865,
+            0.010578680783510208,
+            -0.013635313138365746,
+            0.024299373850226402,
+            0.006258412264287472,
+            0.0007476155878975987,
+            -0.03961668908596039,
+            -0.00560524920001626,
+            0.014830644242465496,
+            0.01095435582101345,
+            -0.005989462602883577,
+            -0.04624223709106445,
+            -0.030105268582701683,
+            0.025563009083271027,
+            0.005110040307044983,
+            -0.05167245492339134,
+            0.014557424932718277,
+            -0.015334390103816986,
+            0.018766699358820915,
+            -0.008708841167390347,
+            -0.025972837582230568,
+            -0.029985735192894936,
+            -0.017187153920531273,
+            0.016726097092032433,
+            0.048564594238996506,
+            0.004207138437777758,
+            0.013345018029212952,
+            -0.0048538981936872005,
+            0.006651164032518864,
+            -0.024760430678725243,
+            0.012030153535306454,
+            -0.027151092886924744,
+            -0.02971251681447029,
+            0.029012393206357956,
+            0.009981014765799046,
+            -0.011330030858516693,
+            0.019330210983753204,
+            -0.017084697261452675,
+            -0.0014141193823888898,
+            0.02240392006933689,
+            -0.0006163425859995186,
+            -0.007295788731426001,
+            -0.010382304899394512,
+            0.013618236407637596,
+            -0.0014354645973071456,
+            -0.01762259565293789,
+            -0.009571187198162079,
+            0.04241717979311943,
+            0.010092009790241718,
+            0.018835002556443214,
+            0.03261546418070793,
+            -0.0018933190731331706,
+            -0.02400907874107361,
+            -0.0027215126901865005,
+            -0.026638807728886604,
+            -0.017033468931913376,
+            -0.014548887498676777,
+            0.016674868762493134,
+            0.023530947044491768,
+            0.015249010175466537,
+            -0.027526767924427986,
+            -0.030019886791706085,
+            -0.01950097270309925,
+            -0.022199006751179695,
+            -0.0034493841230869293,
+            -0.0004213008505757898,
+            -0.026502197608351707,
+            0.007005494087934494,
+            0.0341181643307209,
+            0.03357172757387161,
+            -0.0010870042024180293,
+            -0.009016212075948715,
+            -0.010638446547091007,
+            0.0007721625734120607,
+            0.015231933444738388,
+            -0.015018481761217117,
+            0.0012326851719990373,
+            0.005028928630053997,
+            0.005383259151130915,
+            0.0029285612981766462,
+            -0.007095144130289555,
+            -0.05747834965586662,
+            0.003961668815463781,
+            0.02544347569346428,
+            0.005084426142275333,
+            -0.0008377990452572703,
+            0.02629728429019451,
+            0.0008276600856333971,
+            0.008123982697725296,
+            -0.03142013028264046,
+            0.011108040809631348,
+            0.01803242415189743,
+            0.015163629315793514,
+            -0.018151957541704178,
+            0.0122521435841918,
+            0.0064206356182694435,
+            0.004811207763850689,
+            -0.00030950538348406553,
+            0.010143238119781017,
+            0.028636718168854713,
+            0.019620506092905998,
+            -0.014736725017428398,
+            0.023821242153644562,
+            0.0006435577524825931,
+            -0.012072844430804253,
+            -0.024333525449037552,
+            -0.001040044822730124,
+            0.0029883277602493763,
+            -0.011176345869898796,
+            0.014702572487294674,
+            -0.0023949311580508947,
+            0.0007956422632560134,
+            0.04139260947704315,
+            0.0437491200864315,
+            0.013481627218425274,
+            -0.011295879259705544,
+            0.003304236801341176,
+            0.026502197608351707,
+            0.020132791250944138,
+            0.004862436093389988,
+            0.011458102613687515,
+            -0.03331558406352997,
+            -0.019620506092905998,
+            -0.002981924219056964,
+            0.0014290610561147332,
+            -0.0026788224931806326,
+            0.03876287862658501,
+            0.029524678364396095,
+            0.009682182222604752,
+            -0.005741858389228582,
+            -0.011364183388650417,
+            0.013643850572407246,
+            0.020201096311211586,
+            -0.023599252104759216,
+            -0.0038015798199921846,
+            0.0016745307948440313,
+            -0.017170077189803123,
+            0.02288205362856388,
+            -0.0070311082527041435,
+            -0.00928943045437336,
+            0.02645096927881241,
+            0.09186973422765732,
+            -0.00464471522718668,
+            0.012141148559749126,
+            -0.03948007896542549,
+            -0.021823331713676453,
+            -0.006975610740482807,
+            -0.0036671049892902374,
+            0.015402695164084435,
+            0.02739015780389309,
+            0.009135744534432888,
+            0.007133565377444029,
+            0.017878737300634384,
+            -0.004606293980032206,
+            0.012508286163210869,
+            -0.02826104313135147,
+            0.020166942849755287,
+            -0.005007583647966385,
+            -0.0038335975259542465,
+            -0.011389797553420067,
+            -0.005989462602883577,
+            0.011765473522245884,
+            0.005468640010803938,
+            0.017016392201185226,
+            0.0019893725402653217,
+            -0.0024760430678725243,
+            0.0014333301223814487,
+            0.003031018190085888,
+            -0.010877513326704502,
+            -0.009383349679410458,
+            -0.021396426483988762,
+            0.003827193984761834,
+            0.0009914844995364547,
+            -0.007884915918111801,
+            0.012755890376865864,
+            0.005502792075276375,
+            0.024043232202529907,
+            -0.028175661340355873,
+            -0.0030609015375375748,
+            0.00633525475859642,
+            -0.008956445381045341,
+            0.010390842333436012,
+            0.012380215339362621,
+            -0.008640536107122898,
+            0.022830823436379433,
+            -0.01466841995716095,
+            0.014762339182198048,
+            0.01170570682734251,
+            -0.021601341664791107,
+            -0.004155910108238459,
+            -0.009938324801623821,
+            -0.02192578837275505,
+            0.020252324640750885,
+            -0.018715469166636467,
+            0.005033197812736034,
+            0.00073320756200701,
+            0.006774966139346361,
+            -0.0010496501345187426,
+            -0.01740914396941662,
+            0.010348152369260788,
+            -0.008785683661699295,
+            0.006839001551270485,
+            -0.005058811977505684,
+            0.018254414200782776,
+            0.012166762724518776,
+            0.0028090281412005424,
+            -0.0011622461024671793,
+            0.014600115828216076,
+            -0.00372900627553463,
+            -0.018886230885982513,
+            -0.0036713741719722748,
+            0.01633334532380104,
+            0.01051037572324276,
+            0.0002606782363727689,
+            -0.010305461473762989,
+            -0.0146342683583498,
+            0.014139058999717236,
+            0.000977076473645866,
+            -0.0018122073961421847,
+            -0.0005285604856908321,
+            -0.03760170191526413,
+            0.002606248715892434,
+            0.013532855547964573,
+            0.012704662047326565,
+            -0.005818700883537531,
+            -0.017477449029684067,
+            0.007983104325830936,
+            -0.004190062638372183,
+            -0.0003204447857569903,
+            0.01661510206758976,
+            -0.019774191081523895,
+            0.01002370472997427,
+            -0.014779414981603622,
+            -0.03278622403740883,
+            -0.0029776552692055702,
+            -0.008380125276744366,
+            0.005822970066219568,
+            -0.005592442117631435,
+            -0.0071805245243012905,
+            -0.01414759736508131,
+            -0.005741858389228582,
+            -0.02979789674282074,
+            0.008947907015681267,
+            0.0317445807158947,
+            0.004473953507840633,
+            -0.028978241607546806,
+            -0.012158225290477276,
+            0.025580085813999176,
+            -0.0024931191001087427,
+            -0.006915844511240721,
+            0.003009672975167632,
+            0.0012572321575134993,
+            -0.005784548819065094,
+            -0.009656568057835102,
+            -0.012935190461575985,
+            0.013063261285424232,
+            0.01320840884000063,
+            0.011244650930166245,
+            -0.03357172757387161,
+            0.010664060711860657,
+            0.01813488081097603,
+            -0.02042308636009693,
+            0.011748396791517735,
+            0.028585489839315414,
+            0.0029435029719024897,
+            0.010399380698800087,
+            0.007334209978580475,
+            0.008286206051707268,
+            -0.017153002321720123,
+            0.016683407127857208,
+            0.009186973795294762,
+            -0.012559514492750168,
+            0.004563603550195694,
+            0.007381169591099024,
+            0.008542348630726337,
+            0.01960342936217785,
+            -0.011603250168263912,
+            0.013609698973596096,
+            -0.0068816919811069965,
+            0.03321312740445137,
+            -0.011919158510863781,
+            -0.014881872572004795,
+            0.03229101747274399,
+            0.027560919523239136,
+            -0.004563603550195694,
+            0.017246920615434647,
+            -0.01943266950547695,
+            -0.01267050951719284,
+            0.019296059384942055,
+            0.0005907283630222082,
+            0.019313136115670204,
+            -0.011304416693747044,
+            -0.02595576085150242,
+            0.029012393206357956,
+            -0.008106905966997147,
+            -0.001259366748854518,
+            -0.015829598531126976,
+            0.0043458822183310986,
+            -0.0005789885181002319,
+            0.037328481674194336,
+            0.012653433717787266,
+            0.0011985328746959567,
+            0.026570502668619156,
+            -0.00769280968233943,
+            -0.025067800655961037,
+            0.001990439835935831,
+            -0.015214857645332813,
+            0.013336479663848877,
+            -0.027970748022198677,
+            -0.00560524920001626,
+            0.014506196603178978,
+            0.003026749240234494,
+            -0.002644670195877552,
+            0.01666633039712906,
+            -0.007513510063290596,
+            0.011791087687015533,
+            -0.019244831055402756,
+            0.016717558726668358,
+            -0.011517869308590889,
+            0.004473953507840633,
+            -0.011910621076822281,
+            0.026092370972037315,
+            -0.025563009083271027,
+            -0.00999809056520462,
+            0.013660927303135395,
+            -0.008354511111974716,
+            0.0006376878009177744,
+            -0.02252345345914364,
+            -0.008089830167591572,
+            -0.020371858030557632,
+            0.01679440215229988,
+            -0.005682091694325209,
+            0.011902082711458206,
+            -0.00306303589604795,
+            0.0025528857950121164,
+            -0.008585038594901562,
+            0.010322538204491138,
+            -0.0009199780761264265,
+            -0.017417682334780693,
+            -0.005968117620795965,
+            -0.03025895357131958,
+            -0.030361410230398178,
+            0.03072001039981842,
+            -0.00880275946110487,
+            0.003923247568309307,
+            0.02728770114481449,
+            -0.02540932409465313,
+            -0.005485715810209513,
+            0.001918933354318142,
+            0.022728366777300835,
+            -0.030651705339550972,
+            0.007914799265563488,
+            -0.009391887113451958,
+            -0.016452878713607788,
+            0.007184793706983328,
+            -0.0008874266059137881,
+            0.015880826860666275,
+            -0.0025635582860559225,
+            -0.04924764111638069,
+            -0.0007081269286572933,
+            0.017135925590991974,
+            -0.013806073926389217,
+            0.008081291802227497,
+            0.02151595987379551,
+            -0.009630953893065453,
+            -0.008439891040325165,
+            -0.0011334300506860018,
+            0.025989914312958717,
+            -0.022625910118222237,
+            0.030019886791706085,
+            0.005767472553998232,
+            0.004713019821792841,
+            0.005340568721294403,
+            0.002962713595479727,
+            0.01730668731033802,
+            0.002080089645460248,
+            0.004258367232978344,
+            0.025238562375307083,
+            0.013148642145097256,
+            0.0037418133579194546,
+            -0.0032978332601487637,
+            -0.0030352871399372816,
+            0.016538260504603386,
+            -0.018442250788211823,
+            0.02100367471575737,
+            -0.0004714620590675622,
+            0.00013560871593654156,
+            0.016256503760814667,
+            -0.01484772004187107,
+            -0.027219397947192192,
+            0.03075416199862957,
+            -0.022335615009069443,
+            0.01757136732339859,
+            0.023770013824105263,
+            -0.01339624635875225,
+            -0.004922202788293362,
+            -0.007073798682540655,
+            0.001534719718620181,
+            0.018339794129133224,
+            -0.0134218605235219,
+            -0.023308956995606422,
+            -0.00030790449818596244,
+            0.0220111683011055,
+            0.02293328195810318,
+            0.028175661340355873,
+            -0.021140284836292267,
+            0.01412198320031166,
+            0.007466550450772047,
+            -0.008311820216476917,
+            0.016469955444335938,
+            0.008452698588371277,
+            -0.0025272713974118233,
+            0.021293969824910164,
+            0.0075903525575995445,
+            0.004576410632580519,
+            -0.008896678686141968,
+            -0.013242561370134354,
+            -0.01267050951719284,
+            -0.006749351508915424,
+            -0.006164493504911661,
+            -0.008264861069619656,
+            -0.01709323562681675,
+            0.09548988193273544,
+            -0.011867930181324482,
+            -0.0028410458471626043,
+            -0.009195511229336262,
+            0.0026041141245514154,
+            -0.006949996575713158,
+            0.0013479492627084255,
+            0.023086966946721077,
+            -0.022728366777300835,
+            -0.03671374171972275,
+            0.011167807504534721,
+            -0.004070529248565435,
+            0.006288295611739159,
+            0.03712356835603714,
+            0.00038314631092362106,
+            -0.016589488834142685,
+            -6.773765198886395e-05,
+            -0.003372541395947337,
+            0.01295226626098156,
+            0.0028197006322443485,
+            -0.0036158766597509384,
+            0.0109799699857831,
+            0.029200231656432152,
+            -0.017810434103012085,
+            -0.02144765481352806,
+            -0.012525362893939018,
+            0.0032615463715046644,
+            -0.002578499959781766,
+            0.003707661060616374,
+            0.0051996903494000435,
+            -0.014053679071366787,
+            -0.0134218605235219,
+            0.018817927688360214,
+            -0.016094280406832695,
+            0.0008025794522836804,
+            -0.012627819553017616,
+            -0.008222170174121857,
+            0.01753721572458744,
+            0.01757136732339859,
+            -0.03406693786382675,
+            -0.009921248070895672,
+            -0.03855796530842781,
+            0.018749622628092766,
+            0.01643580198287964,
+            0.014540349133312702,
+            -0.020884141325950623,
+            -0.012644895352423191,
+            -0.004520913120359182,
+            -0.02981497347354889,
+            -0.02243807166814804,
+            -0.026536351069808006,
+            0.009605339728295803,
+            0.00365856708958745,
+            0.027595072984695435,
+            0.005259457044303417,
+            -0.006262681446969509,
+            0.00801725685596466,
+            -0.01025423314422369,
+            0.00755620002746582,
+            -0.02146473154425621,
+            0.007107951212674379,
+            -0.0011409008875489235,
+            0.0012529630912467837,
+            0.005592442117631435,
+            -0.008853988721966743,
+            -0.01735791563987732,
+            -0.0079276068136096,
+            0.01466841995716095,
+            -0.015752756968140602,
+            -0.00821790099143982,
+            0.020798761397600174,
+            0.009383349679410458,
+            0.00931504461914301,
+            0.015616146847605705,
+            0.045661646872758865,
+            0.008273398503661156,
+            0.00783795677125454,
+            -0.011056812480092049,
+            0.009648029692471027,
+            -0.01813488081097603,
+            0.00745801255106926,
+            -0.019330210983753204,
+            -0.006292564328759909,
+            -0.019722962751984596,
+            0.04012897238135338,
+            -0.009366272948682308,
+            -0.01635042205452919,
+            0.033435117453336716,
+            -0.011808163486421108,
+            0.01910822093486786,
+            0.0035475720651447773,
+            -0.022830823436379433,
+            0.03352050110697746,
+            0.003876287955790758,
+            0.012918113730847836,
+            -0.025682542473077774,
+            0.0035859933122992516,
+            -0.00877714529633522,
+            0.006032153032720089,
+            -0.011586173437535763,
+            0.003494208911433816,
+            0.0023885276168584824,
+            -0.004841091111302376,
+            -0.008760069496929646,
+            0.018698394298553467,
+            0.006595666520297527,
+            0.0002183880569646135,
+            0.028056127950549126,
+            -0.000954130373429507,
+            0.018220260739326477,
+            6.110063259257004e-05,
+            0.006010808050632477,
+            0.027680452913045883,
+            0.002821835223585367,
+            0.006809118203818798,
+            -0.01714446395635605,
+            -0.0009199780761264265,
+            -0.009485806338489056,
+            0.009596801362931728,
+            0.02639974094927311,
+            -0.010612832382321358,
+            -0.031078608706593513,
+            0.022574681788682938,
+            -0.007048184517771006,
+            0.015966208651661873,
+            0.002999000484123826,
+            -0.014062216505408287,
+            -0.0023437028285115957,
+            -0.027680452913045883,
+            0.014454968273639679,
+            -0.017349377274513245,
+            -0.024265222251415253,
+            -0.007231753319501877,
+            -0.01045914739370346,
+            -0.004845359828323126,
+            0.038353051990270615,
+            0.012474133633077145,
+            -0.024572592228651047,
+            0.006817656569182873,
+            -0.003974475897848606,
+            -0.005989462602883577,
+            0.0057119750417768955,
+            0.005976655520498753,
+            0.004098278004676104,
+            0.010945817455649376,
+            -0.0065700518898665905,
+            -0.01320840884000063,
+            -0.03324728086590767,
+            -0.02240392006933689,
+            0.014941639266908169,
+            0.00028068936080671847,
+            -0.01315718051046133,
+            0.028483033180236816,
+            -0.01002370472997427,
+            -0.005575365852564573,
+            0.0071292961947619915,
+            0.00013887719251215458,
+            -0.0022796671837568283,
+            -0.017844585701823235,
+            0.002273263642564416,
+            0.006245605181902647,
+            -0.014950176700949669,
+            0.009844405576586723,
+            0.0032124524004757404,
+            -0.014540349133312702,
+            0.005635132547467947,
+            -0.0015027020126581192,
+            -0.003562513506039977,
+            -0.02481165900826454,
+            0.0059595792554318905,
+            0.00708233704790473,
+            -0.0018036692636087537,
+            0.002390662208199501,
+            -0.018169032409787178,
+            0.01801534742116928,
+            0.001955220242962241,
+            0.0071762558072805405,
+            -0.029166078194975853,
+            0.007282981649041176,
+            -0.006740813609212637,
+            0.0015080382581800222,
+            -0.0195692777633667,
+            0.009161358699202538,
+            0.020798761397600174,
+            0.012491210363805294,
+            -0.0005250918911769986,
+            0.010442070662975311,
+            0.008175211027264595,
+            -0.012704662047326565,
+            0.003950996324419975,
+            -0.02050846628844738,
+            -0.007308595813810825,
+            -0.010057857260107994,
+            0.011372721754014492,
+            -0.004219945520162582,
+            0.015983285382390022,
+            0.009511420503258705,
+            0.005600980017334223,
+            -0.009699258022010326,
+            0.00743666710332036,
+            0.01292665209621191,
+            0.005874198395758867,
+            -0.0011099503608420491,
+            0.017827508971095085,
+            0.07363239675760269,
+            0.005336299538612366,
+            0.013618236407637596,
+            6.870485958643258e-05,
+            -0.013071799650788307,
+            0.0015624684747308493,
+            -0.004939279053360224,
+            -0.01748598739504814,
+            0.026126522570848465,
+            0.026587579399347305,
+            -0.006382214371114969,
+            0.0004917400074191391,
+            0.009016212075948715,
+            0.005558289587497711,
+            0.005485715810209513,
+            0.022216081619262695,
+            -0.0077867284417152405,
+            0.01753721572458744,
+            0.024179840460419655,
+            0.005541213322430849,
+            -0.0019093280425295234,
+            -0.0022305732127279043,
+            0.009152821265161037,
+            0.004209273029118776,
+            -0.009050363674759865,
+            0.01150079257786274,
+            -0.01437812577933073,
+            0.01909114606678486,
+            0.011389797553420067,
+            0.02146473154425621,
+            0.007073798682540655,
+            -0.01704200729727745,
+            -0.018356870859861374,
+            0.0041281613521277905,
+            -0.0013479492627084255,
+            0.0059083509258925915,
+            0.0063224476762115955,
+            0.005477177910506725,
+            -0.011611787602305412,
+            -0.0041793896816670895,
+            0.010356690734624863,
+            -0.002471773885190487,
+            -0.0008500725380145013,
+            0.01757136732339859,
+            -0.01268758624792099,
+            0.01168009266257286,
+            -0.001839956152252853,
+            -0.0009984216885641217,
+            0.009682182222604752,
+            0.0014354645973071456,
+            -0.011825240217149258,
+            -0.004166582599282265,
+            0.0021003675647079945,
+            0.01217530108988285,
+            -0.013259637169539928,
+            -0.01048476155847311,
+            0.011825240217149258,
+            -0.006168762221932411,
+            0.009169897064566612,
+            -0.009579724632203579,
+            -0.0025315405800938606,
+            -0.00039648707024753094,
+            -0.008085560984909534,
+            -0.0014578771078959107,
+            0.007820880971848965,
+            0.0008324627415277064,
+            0.010518914088606834,
+            0.011560559272766113,
+            0.01150079257786274,
+            0.003560379147529602,
+            0.004002224653959274,
+            -0.002964847953990102,
+            -0.0064206356182694435,
+            -0.011432488448917866,
+            8.15119783510454e-05,
+            0.005776010919362307,
+            -0.02242099680006504,
+            -0.00024800453684292734,
+            0.008465506136417389,
+            0.005046004895120859,
+            -0.0007710952777415514,
+            0.020849989727139473,
+            0.001264702994376421,
+            0.007820880971848965,
+            -0.025221485644578934,
+            0.0011238247388973832,
+            0.006890229880809784,
+            0.007718423847109079,
+            0.012798581272363663,
+            -0.028636718168854713,
+            -0.0038528083823621273,
+            -0.0038634808734059334,
+            0.0171273872256279,
+            -0.00011606451880652457,
+            -0.0033106403425335884,
+            0.021686721593141556,
+            -0.0007833687704987824,
+            -0.003430173499509692,
+            -0.00755620002746582,
+            -0.005579635035246611,
+            0.008004449307918549,
+            0.012994957156479359,
+            0.024623820558190346,
+            -0.0019968433771282434,
+            -0.0055967108346521854,
+            -0.00809409935027361,
+            0.0244530588388443,
+            0.015249010175466537,
+            0.01863008923828602,
+            -0.0219087116420269,
+            0.010638446547091007,
+            -0.013865840621292591,
+            0.009067440405488014,
+            0.011714245192706585,
+            -0.010911665856838226,
+            0.0004290384822525084,
+            -0.010757979936897755,
+            -0.017135925590991974,
+            -0.022711291909217834,
+            0.00502039073035121,
+            0.02839765138924122,
+            -0.0002676154254004359,
+            0.007193332072347403,
+            -0.016358960419893265,
+            -0.0018442251021042466,
+            -0.011312955059111118,
+            -0.025750847533345222,
+            -0.016222350299358368,
+            -0.021772101521492004,
+            -0.02141350321471691,
+            -0.0038592119235545397,
+            -0.002131317974999547,
+            -0.01907406933605671,
+            0.00013153978215996176,
+            -0.0012817791430279613,
+            0.01902284100651741,
+            -0.0014888276346027851,
+            0.0015624684747308493,
+            0.012337524443864822,
+            0.012943727895617485,
+            -0.012824195437133312,
+            -0.004228483885526657,
+            -0.019261907786130905,
+            -0.024179840460419655,
+            0.0032487392891198397,
+            -0.007726961746811867,
+            0.013874378986656666,
+            -0.008234977722167969,
+            0.004260501824319363,
+            -0.017187153920531273,
+            0.014702572487294674,
+            -0.03579162806272507,
+            -0.024128612130880356,
+            0.0006531630642712116,
+            -0.0040299734100699425,
+            -0.009605339728295803,
+            -0.02585330419242382,
+            0.01198746357113123,
+            -0.019620506092905998,
+            0.002104636514559388,
+            -0.018442250788211823,
+            0.011398335918784142,
+            -0.015283161774277687,
+            0.018681317567825317,
+            -0.00025520852068439126,
+            0.03179580718278885,
+            0.004405648913234472,
+            0.007978835143148899,
+            -0.004525181837379932,
+            0.003327716374769807,
+            -0.005468640010803938,
+            0.006365138106048107,
+            0.006988417822867632,
+            0.008055677637457848,
+            0.007825150154531002,
+            -0.004768517334014177,
+            0.004061991348862648,
+            0.005110040307044983,
+            0.030515095219016075,
+            0.028021976351737976,
+            0.0027706066612154245,
+            0.0014674824196845293,
+            0.0034685947466641665,
+            0.004576410632580519,
+            -0.00038021133514121175,
+            0.008862526156008244,
+            -0.011287340894341469,
+            -0.005391797050833702,
+            0.01417321152985096,
+            -0.012448519468307495,
+            -0.0037695621140301228,
+            0.015377080999314785,
+            -0.015086785890161991,
+            -0.026092370972037315,
+            0.021823331713676453,
+            -0.006552976090461016,
+            -0.002450428670272231,
+            -0.006689585279673338,
+            0.008115444332361221,
+            -0.005946772173047066,
+            0.004194331355392933,
+            0.028107358142733574,
+            -0.005737589206546545,
+            -0.010945817455649376,
+            0.012636357918381691,
+            -0.002330895746126771,
+            -0.007948951795697212,
+            0.02291620522737503,
+            0.035894084721803665,
+            0.00999809056520462,
+            -0.026041142642498016,
+            -0.007381169591099024,
+            0.02144765481352806,
+            -0.00973341055214405,
+            -0.021755026653409004,
+            0.008273398503661156,
+            0.007522047962993383,
+            -0.005421680398285389,
+            0.018937459215521812,
+            0.004239156376570463,
+            -0.010792132467031479,
+            0.009041826240718365,
+            0.022711291909217834,
+            0.021686721593141556,
+            0.008947907015681267,
+            -0.023548023775219917,
+            0.01025423314422369,
+            -0.026228979229927063,
+            -0.013831689022481441,
+            -0.023821242153644562,
+            0.003581724362447858,
+            0.003857077332213521,
+            -0.003257277188822627,
+            0.026536351069808006,
+            0.011876468546688557,
+            0.011645940132439137,
+            0.0030950538348406553,
+            0.023052813485264778,
+            -0.015223395079374313,
+            -0.028107358142733574,
+            -0.0009733410552144051,
+            -0.015479537658393383,
+            -0.0018282162491232157,
+            0.002166537567973137,
+            0.007150641642510891,
+            0.005852853413671255,
+            0.01953512616455555,
+            0.000376209121895954,
+            0.0019584218971431255,
+            -2.4263486920972355e-05,
+            0.010100548155605793,
+            -0.015727141872048378,
+            -0.019705887883901596,
+            -0.008922292850911617,
+            6.470263178925961e-05,
+            0.011637401767075062,
+            -0.005208228249102831,
+            -3.448583447607234e-05,
+            0.007846495136618614,
+            -0.015897903591394424,
+            -0.0097590247169137,
+            -0.014412278309464455,
+            -0.0029306956566870213,
+            -0.005071619059890509,
+            0.03136890381574631,
+            0.014437892474234104,
+            -0.004747172351926565,
+            -0.011543483473360538,
+            -0.005741858389228582,
+            0.03370833769440651,
+            0.0017204229952767491,
+            0.012422905303537846,
+            0.0146342683583498,
+            0.015684451907873154,
+            0.027014482766389847,
+            0.012807118706405163,
+            0.003018211107701063,
+            0.002516598906368017,
+            0.02346264198422432,
+            -0.013951221480965614,
+            0.008730186149477959,
+            -0.011645940132439137,
+            -0.003521957667544484,
+            -0.010937280021607876,
+            0.015078248456120491,
+            -0.013003494590520859,
+            -0.011278802528977394,
+            0.023069890215992928,
+            -0.0004381102044135332,
+            0.013276712968945503,
+            -0.0004242358263581991,
+            -0.003953130915760994,
+            -0.02534101903438568,
+            0.0015464596217498183,
+            0.0006531630642712116,
+            -0.004401379730552435,
+            0.010578680783510208,
+            -0.027475539594888687,
+            0.0013660927070304751,
+            0.015240471810102463,
+            -0.007116489112377167,
+            0.00038981667603366077,
+            -0.010826284997165203,
+            0.005904081743210554,
+            0.00011606451880652457,
+            -0.02385539375245571,
+            0.015488076023757458,
+            -0.006698123179376125,
+            0.01243144366890192,
+            -0.0018548977095633745,
+            -0.013788998126983643,
+            0.008687496185302734,
+            0.019296059384942055,
+            0.027560919523239136,
+            -0.007825150154531002,
+            0.0005528406472876668,
+            -0.0001855698210420087,
+            -0.006027883850038052,
+            -0.02737308293581009,
+            0.02054261974990368,
+            -0.007863570936024189,
+            0.01953512616455555,
+            -0.009682182222604752,
+            -0.018186109140515327,
+            -0.014856258407235146,
+            0.011927696876227856,
+            0.007603159639984369,
+            0.0010304395109415054,
+            0.01489894837141037,
+            -0.024077383801341057,
+            0.020867066457867622,
+            0.0033383890986442566,
+            -0.0064718639478087425,
+            0.008303281851112843,
+            0.004230618476867676,
+            0.004695943556725979,
+            -0.008384393528103828,
+            0.007658657152205706,
+            0.007569007575511932,
+            -0.007253098301589489,
+            0.004326671827584505,
+            -0.04057295247912407,
+            0.010732365772128105,
+            -0.011090965010225773,
+            0.005878467578440905,
+            -0.004742903169244528,
+            -0.022745443508028984,
+            -0.035825781524181366,
+            0.004567872267216444,
+            0.019313136115670204,
+            -0.020235247910022736,
+            0.018408099189400673,
+            0.012124072760343552,
+            0.014574501663446426,
+            0.01859593763947487,
+            0.008853988721966743,
+            0.01461719162762165,
+            0.0438857264816761,
+            -0.024231068789958954,
+            6.483604374807328e-05,
+            0.008734455332159996,
+            0.011381260119378567,
+            -0.013404784724116325,
+            -0.010126162320375443,
+            0.005784548819065094,
+            -0.0015197781613096595,
+            0.04067540913820267,
+            -0.019210679456591606,
+            0.010886050760746002,
+            -0.010962894186377525,
+            0.008371586911380291,
+            -0.009562648832798004,
+            -0.0036158766597509384,
+            -0.012559514492750168,
+            -0.011893544346094131,
+            0.014745263382792473,
+            -0.019671734422445297,
+            -0.005460101645439863,
+            -0.007282981649041176,
+            0.01048476155847311,
+            -0.006096188444644213,
+            -0.020730456337332726,
+            0.018647165969014168,
+            -0.0018079383298754692,
+            0.029968658462166786,
+            -0.012755890376865864,
+            0.00574612757191062,
+            0.022591758519411087,
+            -0.010604294948279858,
+            0.018732545897364616,
+            -0.004362958483397961,
+            0.008853988721966743,
+            0.0055625587701797485,
+            -0.0065188235603272915,
+            -0.0014685495989397168,
+            0.01389145478606224,
+            -0.0027385889552533627,
+            0.029934506863355637,
+            -0.017400605604052544,
+            -0.009818791411817074,
+            0.009707796387374401,
+            0.0073769004084169865,
+            0.0042370217852294445,
+            0.003878422547131777,
+            0.009579724632203579,
+            -0.01071528997272253,
+            0.0009728074073791504,
+            -0.021686721593141556,
+            -0.015453923493623734,
+            -0.0011483717244118452,
+            -0.004683136474341154,
+            -0.0018730411538854241,
+            -0.002467504935339093,
+            -0.02044016122817993,
+            0.004627638962119818,
+            -0.0021579996682703495,
+            -8.451364556094632e-05,
+            -0.024487212300300598,
+            -0.00769280968233943,
+            -0.010057857260107994,
+            -0.005161269102245569,
+            -0.003139878623187542,
+            0.018271489068865776,
+            0.0034856710117310286,
+            -0.004862436093389988,
+            0.0293197650462389,
+            0.005404604133218527,
+            0.01536854263395071,
+            0.00280262460000813,
+            -0.05334591865539551,
+            -0.011014122515916824,
+            -0.004563603550195694,
+            -0.009912709705531597,
+            -0.007261636666953564,
+            -0.008456967771053314,
+            -0.010681137442588806,
+            -0.0019061262719333172,
+            0.0032914294861257076,
+            -0.004170851781964302,
+            0.0122521435841918,
+            -0.01700785383582115,
+            0.025494705885648727,
+            -0.004017166327685118,
+            0.015744218602776527,
+            -0.011722782626748085,
+            0.011654478497803211,
+            -0.006194376386702061,
+            -0.010851899161934853,
+            0.010544528253376484,
+            0.0146342683583498,
+            -0.0030544979963451624,
+            0.020371858030557632,
+            -0.014736725017428398,
+            0.003197510726749897,
+            -0.008303281851112843,
+            -0.02143057994544506,
+            -0.015991821885108948,
+            0.011295879259705544,
+            0.03130060061812401,
+            0.010066395625472069,
+            -0.0037524858489632607,
+            0.004072663839906454,
+            0.021669644862413406,
+            -0.004352285992354155,
+            0.0063181789591908455,
+            0.03025895357131958,
+            -0.0010982104577124119,
+            0.017742129042744637,
+            0.0001935742620844394,
+            -0.005028928630053997,
+            -0.002362913452088833,
+            0.01808365248143673,
+            -0.002766337711364031,
+            0.0057119750417768955,
+            0.02441890724003315,
+            0.010997045785188675,
+            -0.0064718639478087425,
+            0.008230708539485931,
+            0.014309820719063282,
+            0.013874378986656666,
+            -0.015351466834545135,
+            -0.0122009152546525,
+            -0.011534945107996464,
+            -0.015778370201587677,
+            0.007291519548743963,
+            -0.019261907786130905,
+            -0.004049184266477823,
+            0.007829418405890465,
+            -0.015138015151023865,
+            -0.012807118706405163,
+            -0.023548023775219917,
+            -0.015914980322122574,
+            0.01863008923828602,
+            0.005357644986361265,
+            -0.01465988252311945,
+            -0.004369362257421017,
+            0.0013714289525523782,
+            0.006181569304317236,
+            -0.0054985228925943375,
+            0.021840406581759453,
+            -0.004098278004676104,
+            -0.011125117540359497,
+            0.019330210983753204,
+            0.0017897948855534196,
+            0.005097233224660158,
+            -0.0064206356182694435,
+            0.004550796467810869,
+            -0.011364183388650417,
+            0.022096550092101097,
+            -0.003470729338005185,
+            0.012636357918381691,
+            -0.0037033918779343367,
+            0.004631908144801855,
+            -0.001978699816390872,
+            0.00821790099143982,
+            0.004017166327685118,
+            0.010629909113049507,
+            -0.005656477529555559,
+            0.004525181837379932,
+            0.012491210363805294,
+            -0.01641872711479664,
+            0.007265905383974314,
+            0.004559334367513657,
+            -4.365760105429217e-05,
+            0.005780279636383057,
+            -0.007645850069820881,
+            -0.005417411215603352,
+            -0.015488076023757458,
+            0.010100548155605793,
+            0.030873695388436317,
+            -0.0016649254830554128,
+            -0.007240291219204664,
+            0.01704200729727745,
+            0.015735680237412453,
+            -0.003816521493718028,
+            0.0061218030750751495,
+            0.009280892089009285,
+            -0.003148416755720973,
+            0.01370361726731062,
+            -0.024487212300300598,
+            -0.029524678364396095,
+            0.01763967238366604,
+            -0.020867066457867622,
+            0.022779595106840134,
+            -0.019210679456591606,
+            -0.027731681242585182,
+            -0.01847640424966812,
+            -0.0032530082389712334,
+            -0.019842496141791344,
+            -0.01611989364027977,
+            0.02532394416630268,
+            0.00685180863365531,
+            -0.006949996575713158,
+            -0.031112760305404663,
+            0.04050464928150177,
+            0.014489120803773403,
+            0.038318898528814316,
+            -0.011603250168263912,
+            -2.554753518779762e-05,
+            -0.007504971697926521,
+            0.012064306065440178,
+            -0.01146664097905159,
+            -0.014258592389523983,
+            0.007509240880608559,
+            0.028175661340355873,
+            0.015231933444738388,
+            -0.015223395079374313,
+            -0.007944682613015175,
+            0.015991821885108948,
+            -0.007043915335088968,
+            -0.0005480379913933575,
+            -0.008836911991238594,
+            -0.014830644242465496,
+            -0.010066395625472069,
+            -0.0005312286084517837,
+            0.012243606150150299,
+            0.016179660335183144,
+            -0.0007257367251440883,
+            0.01390853151679039,
+            -0.009955400601029396,
+            -0.014420815743505955,
+            -0.004602024797350168,
+            0.005327761638909578,
+            -0.02252345345914364,
+            0.02105490304529667,
+            0.002292474266141653,
+            -0.009537034668028355,
+            0.0030865157023072243,
+            -0.004572141449898481,
+            0.012602205388247967,
+            0.009391887113451958,
+            -0.0038421356584876776,
+            -0.0036137420684099197,
+            0.00572051340714097,
+            -0.001715086749754846,
+            0.015001405030488968,
+            -0.011714245192706585,
+            0.011842316016554832,
+            -0.007432397920638323,
+            -0.00019931078713852912,
+            -0.009648029692471027,
+            -0.005238111596554518,
+            -0.010493299923837185,
+            0.0028794673271477222,
+            0.012943727895617485,
+            -0.014548887498676777,
+            0.012807118706405163,
+            -0.0008132520597428083,
+            -0.00949434470385313,
+            -0.015138015151023865,
+            -0.0055155991576612,
+            -0.011364183388650417,
+            -0.016683407127857208,
+            0.007069529499858618,
+            0.0007855033036321402,
+            0.019347287714481354,
+            0.0030054040253162384,
+            -0.004572141449898481,
+            -0.00017262929759453982,
+            0.018903307616710663,
+            -0.02580207586288452,
+            0.03403278440237045,
+            -0.00010045584349427372,
+            0.022711291909217834,
+            0.04251963645219803,
+            0.00954557303339243,
+            -0.006663971114903688,
+            -0.013754845596849918,
+            0.005801625084131956,
+            -0.013609698973596096,
+            0.009981014765799046,
+            0.014335434883832932,
+            -0.03235932067036629,
+            -0.003944592550396919,
+            -0.013976835645735264,
+            0.011586173437535763,
+            -0.007957490161061287,
+            0.0032850259449332952,
+            -0.010587218217551708,
+            0.011765473522245884,
+            0.01679440215229988,
+            -0.009075978770852089,
+            -0.004311730153858662,
+            -0.027492616325616837,
+            0.00032524747075513005,
+            0.016674868762493134,
+            -0.010911665856838226,
+            -0.01684563048183918,
+            0.010945817455649376,
+            -0.007218946237117052,
+            -0.015453923493623734,
+            -0.006898768246173859,
+            -0.03278622403740883,
+            -0.00020504731219261885,
+            -0.01076651830226183,
+            -0.003927516285330057,
+            -0.0028794673271477222,
+            0.0011024795239791274,
+            -0.0033917520195245743,
+            -0.0067066615447402,
+            -0.0025891726836562157,
+            0.0067664277739822865,
+            0.005011852364987135,
+            0.002080089645460248,
+            0.0005987327895127237,
+            -0.00023506399884354323,
+            0.009255277924239635,
+            0.010168852284550667,
+            -0.007419590838253498,
+            -0.017212767153978348,
+            0.010834822431206703,
+            -0.020730456337332726,
+            0.005938234273344278,
+            0.00783795677125454,
+            0.011039736680686474,
+            -0.0220111683011055,
+            -0.010868974961340427,
+            0.009818791411817074,
+            0.003835732117295265,
+            0.01902284100651741,
+            0.010886050760746002,
+            0.0022455148864537477,
+            -0.0035646480973809958,
+            0.0028495839796960354,
+            -1.0947752343781758e-05,
+            -0.0012508286163210869,
+            0.015308775939047337,
+            0.0006504949415102601,
+            0.02974666841328144,
+            -0.012055767700076103,
+            0.011201960034668446,
+            -0.009340658783912659,
+            0.0013842361513525248,
+            -0.009741948917508125,
+            -0.001973363570868969,
+            -0.004194331355392933,
+            -0.006484671495854855,
+            0.013148642145097256,
+            0.007714154664427042,
+            -0.004239156376570463,
+            -0.01416467409580946,
+            0.01194477267563343,
+            -0.023035738617181778,
+            -0.012269220314919949,
+            -0.00881983619183302,
+            0.012935190461575985,
+            -0.0053960662335157394,
+            0.01170570682734251,
+            0.03457922115921974,
+            -0.0018559650052338839,
+            0.0007860369514673948,
+            -0.013746308162808418,
+            -0.008311820216476917,
+            -0.02728770114481449,
+            -0.01510386262089014,
+            -0.008922292850911617,
+            -0.03454506769776344,
+            -0.011919158510863781,
+            0.008239246904850006,
+            0.008111175149679184,
+            0.0014237246941775084,
+            -0.019176525995135307,
+            -0.008888140320777893,
+            -0.01492456253618002,
+            -0.017460372298955917,
+            -0.00924673955887556,
+            -0.01175693515688181,
+            -0.0037119300104677677,
+            0.020884141325950623,
+            -0.004162313882261515,
+            0.014267130754888058,
+            0.016939548775553703,
+            1.3132299500284716e-05,
+            0.015334390103816986,
+            0.009665105491876602,
+            -0.009127207100391388,
+            -0.004725826904177666,
+            0.006885961163789034,
+            0.0027706066612154245,
+            0.018749622628092766,
+            -0.006702392362058163,
+            -0.014796491712331772,
+            0.004298923071473837,
+            0.005571096669882536,
+            -0.012346062809228897,
+            0.010655523277819157,
+            -0.00476424815133214,
+            0.02054261974990368,
+            -0.018254414200782776,
+            -0.011885006912052631,
+            0.014472044073045254,
+            -0.008862526156008244,
+            0.02394077368080616,
+            0.013951221480965614,
+            -0.015291700139641762,
+            0.007167717441916466,
+            0.013029108755290508,
+            0.007615966722369194,
+            0.0018495614640414715,
+            0.022147778421640396,
+            -0.0011846584966406226,
+            0.0031590894795954227,
+            0.010237157344818115,
+            -0.018835002556443214,
+            -0.02042308636009693,
+            -0.003118533408269286,
+            0.02247222512960434,
+            0.007961759343743324,
+            -0.016222350299358368,
+            0.0037930416874587536,
+            0.012849809601902962,
+            0.012072844430804253,
+            0.0039787450805306435,
+            -0.001204936415888369,
+            -0.02248930186033249,
+            0.0035155541263520718,
+            0.0001910395221784711,
+            0.018664240837097168,
+            -0.011116579174995422,
+            0.001461078878492117,
+            -0.011389797553420067,
+            0.012747352942824364,
+            -0.005221035331487656,
+            0.01193623524159193,
+            0.0011761204805225134,
+            -0.005353375803679228,
+            0.0001780989987310022,
+            0.014941639266908169,
+            0.014070754870772362,
+            0.037738312035799026,
+            -0.005916888825595379,
+            -0.006480402313172817,
+            0.017170077189803123,
+            -0.012388752773404121,
+            0.018715469166636467,
+            0.006497478578239679,
+            0.0170249305665493,
+            -0.011090965010225773,
+            0.0011344973463565111,
+            0.04115354269742966,
+            0.015949131920933723,
+            -6.583659705938771e-05,
+            -0.008760069496929646,
+            -0.00805140845477581,
+            0.0244530588388443,
+            0.009067440405488014,
+            -2.7882166250492446e-05,
+            0.015744218602776527,
+            0.006096188444644213,
+            -0.016324806958436966,
+            -0.012918113730847836,
+            0.03555256128311157,
+            -0.007718423847109079,
+            -0.022045321762561798,
+            -0.0009311842732131481,
+            -0.009383349679410458,
+            -0.02884163148701191,
+            0.013959759846329689,
+            0.013788998126983643,
+            -0.008162403479218483,
+            0.006992687005549669,
+            0.008439891040325165,
+            -0.004117488861083984,
+            -0.004435532260686159,
+            0.0005346972029656172,
+            -0.0007049251580610871,
+            0.006177300587296486,
+            0.005092964507639408,
+            -0.014463506639003754,
+            -0.006787773221731186,
+            -0.005669284611940384,
+            -0.004064125940203667,
+            0.009016212075948715,
+            0.005058811977505684,
+            0.004870974458754063,
+            -0.008482581935822964,
+            0.018339794129133224,
+            0.02298451028764248,
+            -0.0122009152546525,
+            0.012115534394979477,
+            0.005233842879533768,
+            -0.0146428057923913,
+            -0.004614831879734993,
+            -0.01121903583407402,
+            -0.026570502668619156,
+            -0.006113264709711075,
+            0.0033490615896880627,
+            0.015428309328854084,
+            -0.022796671837568283,
+            0.0008500725380145013,
+            0.007790997624397278,
+            2.7365211280994117e-05,
+            -0.008879602886736393,
+            0.007573276292532682,
+            -0.00044157879892736673,
+            0.022301463410258293,
+            0.019808344542980194,
+            0.025631314143538475,
+            -0.008414276875555515,
+            -0.00499904528260231,
+            0.014301283285021782,
+            0.00433947890996933,
+            0.009374811314046383,
+            0.027543844655156136,
+            0.022164853289723396,
+            -0.013857303187251091,
+            0.006429173983633518,
+            0.000640355923678726,
+            -0.014250054024159908,
+            0.0066682398319244385,
+            0.013054722920060158,
+            0.012414366938173771,
+            0.0028410458471626043,
+            -0.0013415457215160131,
+            -0.015957670286297798,
+            0.004824014846235514,
+            0.018408099189400673,
+            -0.017118848860263824,
+            -0.03471583127975464,
+            -0.011842316016554832,
+            -0.012158225290477276,
+            -0.00624133599922061,
+            -0.012875423766672611,
+            0.01440373994410038,
+            0.004418455995619297,
+            0.019705887883901596,
+            0.0002309283590875566,
+            -0.01022861897945404,
+            0.014250054024159908,
+            0.005110040307044983,
+            -0.0016638581873849034,
+            0.0005352308508008718,
+            0.007231753319501877,
+            -0.0030331527814269066,
+            -0.0022476494777947664,
+            -0.00027055040118284523,
+            0.016239427030086517,
+            -0.02532394416630268,
+            0.008947907015681267,
+            -0.017067620530724525,
+            0.0017951311310753226,
+            0.015249010175466537,
+            -0.003775965655222535,
+            -0.016025975346565247,
+            0.0011216901475563645,
+            -0.008619191125035286,
+            -0.07288104295730591,
+            -0.005882736761122942,
+            0.004512374754995108,
+            0.023992003872990608,
+            -0.013336479663848877,
+            -0.0024824466090649366,
+            0.0018474268727004528,
+            0.008657612837851048,
+            -0.010134699754416943,
+            0.003995821345597506,
+            0.010390842333436012,
+            -0.003319178242236376,
+            -0.009221125394105911,
+            0.002508060773834586,
+            -0.008106905966997147,
+            -0.0038805571384727955,
+            -0.00783795677125454,
+            0.003402424743399024,
+            0.025221485644578934,
+            -0.010868974961340427,
+            0.024657974019646645,
+            0.009699258022010326,
+            -0.0075433929450809956,
+            -0.0027428579051047564,
+            -0.007761114276945591,
+            -0.027543844655156136,
+            -0.012397291138768196,
+            -0.003396020969375968,
+            -0.01292665209621191,
+            0.0029264267068356276,
+            0.001692674239166081,
+            -0.007167717441916466,
+            -0.004348016809672117,
+            0.0021547977812588215,
+            0.00030336863710545003,
+            -0.022147778421640396,
+            -0.0009407896432094276,
+            -0.014267130754888058,
+            0.02105490304529667,
+            -0.012371676973998547,
+            -0.004461146425455809,
+            0.0030544979963451624,
+            -0.010109085589647293,
+            0.004580679349601269,
+            0.019313136115670204,
+            -0.007308595813810825,
+            -0.009682182222604752,
+            0.014045140706002712,
+            -0.003690584795549512,
+            0.00185916677583009,
+            -0.01798119582235813,
+            0.010032243095338345,
+            -0.03232516720890999,
+            -0.003583858720958233,
+            -0.009981014765799046,
+            -0.0031847036443650723,
+            -0.0067151994444429874,
+            -0.03162504732608795,
+            0.015138015151023865,
+            -0.007039646618068218,
+            -0.02346264198422432,
+            -0.0011793222511187196,
+            -0.01859593763947487,
+            0.01419028826057911,
+            -0.01487333420664072,
+            0.01393414568156004,
+            -0.020645076408982277,
+            0.012533900327980518,
+            0.012798581272363663,
+            0.021123208105564117,
+            0.006493209395557642,
+            -0.005302147474139929,
+            0.002676687901839614,
+            0.0018730411538854241,
+            0.014352511614561081,
+            -0.008905217051506042,
+            0.00462336977943778,
+            -0.002390662208199501,
+            0.005737589206546545,
+            0.009426039643585682,
+            0.02107197977602482,
+            -0.0005896611255593598,
+            -0.0037823691964149475,
+            -0.017468910664319992,
+            0.00757754547521472,
+            -0.01559907104820013,
+            0.005485715810209513,
+            0.0025251370389014482,
+            0.023787088692188263,
+            0.010886050760746002,
+            0.0035859933122992516,
+            0.004900857340544462,
+            0.004520913120359182,
+            0.01861301250755787,
+            -0.006890229880809784,
+            -0.004747172351926565,
+            -0.018186109140515327,
+            0.019688811153173447,
+            0.007705616764724255,
+            -0.005306416191160679,
+            -0.011902082711458206,
+            0.006010808050632477,
+            0.002134519862011075,
+            0.0017577770631760359,
+            0.02288205362856388,
+            -0.02974666841328144,
+            -0.008964983746409416,
+            -0.007808073423802853,
+            -0.009682182222604752,
+            -0.017844585701823235,
+            0.017084697261452675,
+            -0.008012987673282623,
+            0.00461910106241703,
+            -0.012790042906999588,
+            0.033418044447898865,
+            0.002969117136672139,
+            0.00513992365449667,
+            0.013729231432080269,
+            -0.00024373548512812704,
+            0.017844585701823235,
+            -0.008367317728698254,
+            -0.04036803916096687,
+            -0.0031911071855574846,
+            0.0034515184815973043,
+            0.007645850069820881,
+            -0.010903127491474152,
+            0.004862436093389988,
+            -0.006339523941278458,
+            -0.01854470930993557,
+            -0.014232978224754333,
+            0.037840768694877625,
+            -0.005225304514169693,
+            0.017947042360901833,
+            -0.016145508736371994,
+            -0.0033682722132653,
+            0.007107951212674379,
+            0.022762520238757133,
+            -0.012602205388247967,
+            0.00791053008288145,
+            0.009571187198162079,
+            0.02199409157037735,
+            0.00978463888168335,
+            -0.01565883681178093,
+            0.023735860362648964,
+            0.005741858389228582,
+            -0.0023565099108964205,
+            0.005276532843708992,
+            -0.0030352871399372816,
+            0.016000360250473022,
+            -0.014813567511737347,
+            -0.011876468546688557,
+            0.002303146990016103,
+            -0.010168852284550667,
+            0.0037055264692753553,
+            -0.024589668959379196,
+            0.00037540867924690247,
+            -0.009844405576586723,
+            0.021089056506752968,
+            0.01027130987495184,
+            0.00877714529633522,
+            -0.019637582823634148,
+            -0.0048026698641479015,
+            -0.02645096927881241,
+            -0.014574501663446426,
+            -0.01022861897945404,
+            -0.0011451698374003172,
+            -0.004431263078004122,
+            -0.007765382993966341,
+            -0.03133475035429001,
+            -0.004202869720757008,
+            0.013541393913328648,
+            0.014975790865719318,
+            -0.0013714289525523782,
+            0.012089920230209827,
+            -0.00019544197130016983,
+            -0.008106905966997147,
+            -0.02236976847052574,
+            -0.010407919064164162,
+            -0.02091829478740692,
+            -0.03123229369521141,
+            -0.022199006751179695,
+            0.008670419454574585,
+            -0.012021616101264954,
+            -0.002941368380561471,
+            0.028961164876818657,
+            -0.022130701690912247,
+            0.007513510063290596,
+            0.005400335416197777,
+            0.0028666602447628975,
+            0.014856258407235146,
+            0.0054472945630550385,
+            0.0003634020104072988,
+            0.0037652929313480854,
+            -0.008341703563928604,
+            -0.008081291802227497,
+            0.01799827069044113,
+            -0.003331985557451844,
+            0.005097233224660158,
+            0.010809208266437054,
+            -0.006749351508915424,
+            -0.005438756663352251,
+            0.02392369881272316,
+            -0.015718603506684303,
+            0.002044870052486658,
+            -0.008610652759671211,
+            -0.014079293236136436,
+            -0.00949434470385313,
+            -0.005793086718767881,
+            -0.01813488081097603,
+            -0.015240471810102463,
+            0.01689685881137848,
+            -0.0004028906114399433,
+            0.016025975346565247,
+            -0.005289340391755104,
+            -0.005007583647966385,
+            0.014437892474234104,
+            -0.03558671474456787,
+            -0.003402424743399024,
+            0.006377945188432932,
+            -0.026211904361844063,
+            0.02484581060707569,
+            0.005276532843708992,
+            0.01605158858001232,
+            -0.013780459761619568,
+            0.0026318628806620836,
+            -0.006365138106048107,
+            0.004358689300715923,
+            0.018903307616710663,
+            0.007308595813810825,
+            0.004858167376369238,
+            -0.006689585279673338,
+            -0.008047140203416348,
+            0.016077203676104546,
+            -0.007415322121232748,
+            0.0018719738582149148,
+            0.016785863786935806,
+            0.018425175920128822,
+            0.012004539370536804,
+            0.0059553105384111404,
+            0.013071799650788307,
+            0.01750306226313114,
+            -0.007043915335088968,
+            -0.005835777148604393,
+            -0.013481627218425274,
+            -0.00802579429000616,
+            -0.0013105951948091388,
+            0.03859211876988411,
+            0.018920384347438812,
+            0.007385438773781061,
+            -0.02054261974990368,
+            0.013456013053655624,
+            0.0043928418308496475,
+            6.841302820248529e-06,
+            -0.0027385889552533627,
+            0.02293328195810318,
+            0.0024952536914497614,
+            0.007125027012079954,
+            0.006258412264287472,
+            -0.017212767153978348,
+            0.03309359401464462,
+            -0.020713381469249725,
+            0.0008532743086107075,
+            -0.013319403864443302,
+            -0.012166762724518776,
+            0.015530765987932682,
+            0.002104636514559388,
+            -0.004164448007941246,
+            -0.006296833511441946,
+            -0.0033575997222214937,
+            0.013515779748558998,
+            0.008499657735228539,
+            -0.012875423766672611,
+            0.004516643937677145,
+            -0.02542640082538128,
+            0.004315999336540699,
+            -0.015334390103816986,
+            0.021755026653409004,
+            -0.004559334367513657,
+            -0.004674598574638367,
+            -0.019261907786130905,
+            -0.017067620530724525,
+            -0.00999809056520462,
+            0.002403469290584326,
+            -0.004755710251629353,
+            0.004422725178301334,
+            -0.009468729607760906,
+            -0.008499657735228539,
+            0.0027599341701716185,
+            0.0036628360394388437,
+            0.012465596199035645,
+            0.013669464737176895,
+            0.019893724471330643,
+            0.017759205773472786,
+            -0.01536000519990921,
+            -0.0341864712536335,
+            0.0006056699785403907,
+            0.005084426142275333,
+            -0.0036436254158616066,
+            0.018664240837097168,
+            -0.001938143977895379,
+            -0.0009968207450583577,
+            -0.0007150641758926213,
+            0.0005419012159109116,
+            0.02004740945994854,
+            -0.011014122515916824,
+            -0.016657792031764984,
+            0.00021545309573411942,
+            0.008853988721966743,
+            0.00502039073035121,
+            -0.0025934416335076094,
+            -0.010296924039721489,
+            -0.0009455922991037369,
+            0.0020341973286122084,
+            0.006032153032720089,
+            -0.009101592935621738,
+            -0.0010496501345187426,
+            -0.0013671600027009845,
+            -0.0030544979963451624,
+            -0.006254143081605434,
+            -0.009844405576586723,
+            0.008222170174121857,
+            -0.012602205388247967,
+            -0.01123611256480217,
+            0.016222350299358368,
+            -0.015086785890161991,
+            0.001136631821282208,
+            0.023821242153644562,
+            0.022557605057954788,
+            -0.0008842248353175819,
+            0.006774966139346361,
+            -0.007936145178973675,
+            -0.011517869308590889,
+            -0.02785121463239193,
+            -0.019637582823634148,
+            -0.00806848518550396,
+            0.017947042360901833,
+            -0.002053408185020089,
+            -0.004049184266477823,
+            -0.0018869155319407582,
+            -0.008580769412219524,
+            0.008444160223007202,
+            -0.013814612291753292,
+            0.011620325967669487,
+            -0.034425534307956696,
+            0.013088875450193882,
+            -0.006702392362058163,
+            0.0010485828388482332,
+            0.015009943395853043,
+            0.04351005330681801,
+            -0.007351286243647337,
+            0.0029541754629462957,
+            0.001540056080557406,
+            0.01389145478606224,
+            0.028209814801812172,
+            0.004369362257421017,
+            0.0006136744632385671,
+            -0.01765674725174904,
+            0.004091874696314335,
+            0.002264725510030985,
+            -0.013456013053655624,
+            0.009579724632203579,
+            -0.023035738617181778,
+            -0.023155272006988525,
+            -0.009067440405488014,
+            -0.021123208105564117,
+            -0.002608383307233453,
+            0.002011785050854087,
+            0.012883962132036686,
+            -0.009878558106720448,
+            -0.012098458595573902,
+            -0.0005656477296724916,
+            -0.032991137355566025,
+            0.016709022223949432,
+            0.09665105491876602,
+            0.00875153113156557,
+            -0.003908305894583464,
+            -0.0021515958942472935,
+            0.002499522641301155,
+            0.009562648832798004,
+            -0.004768517334014177,
+            0.055360905826091766,
+            -0.018852079287171364,
+            0.008311820216476917,
+            -0.0008105839369818568,
+            0.008064216002821922,
+            -0.007223214954137802,
+            0.015530765987932682,
+            -0.012209453620016575,
+            -0.005951041355729103,
+            -0.009460192173719406,
+            -0.01123611256480217,
+            -0.0017246920615434647,
+            -0.016640717163681984,
+            -0.004986238200217485,
+            -0.020645076408982277,
+            0.013336479663848877,
+            -0.03234224393963814,
+            -0.011509330943226814,
+            -0.012474133633077145,
+            -0.0063224476762115955,
+            0.006066305562853813,
+            -0.015684451907873154,
+            -0.002595576224848628,
+            -0.002099300269037485,
+            0.015402695164084435,
+            0.0013863706262782216,
+            -0.00232235761359334,
+            -0.008328896015882492,
+            0.0005405671545304358,
+            0.005024659913033247,
+            0.003940323833376169,
+            0.006557244807481766,
+            0.006792041938751936,
+            -0.005848584230989218,
+            -0.008811297826468945,
+            0.010655523277819157,
+            -0.002510195365175605,
+            -0.0027919518761336803,
+            -0.008871064521372318,
+            0.010382304899394512,
+            -0.013481627218425274,
+            0.018356870859861374,
+            -0.013840226456522942,
+            -0.029507601633667946,
+            -0.015547842718660831,
+            0.02100367471575737,
+            -0.01996202953159809,
+            -0.0039381892420351505,
+            0.010553065687417984,
+            0.02829519473016262,
+            0.004113219678401947,
+            -0.001529383473098278,
+            -0.004909395705908537,
+            -0.017204230651259422,
+            0.009511420503258705,
+            0.008196556009352207,
+            0.018442250788211823,
+            -0.012371676973998547,
+            0.016487032175064087,
+            -0.018288565799593925,
+            0.006177300587296486,
+            -0.01663217879831791,
+            0.0006083381595090032,
+            0.00011866596469189972,
+            -0.0041793896816670895,
+            0.01608574204146862,
+            -0.01534292846918106,
+            -0.026263132691383362,
+            0.01900576427578926,
+            -0.013635313138365746,
+            -0.03307652100920677,
+            0.003374675754457712,
+            -0.034937821328639984,
+            -0.0025827689096331596,
+            0.009716334752738476,
+            -0.02149888314306736,
+            -0.018305642530322075,
+            0.007487895432859659,
+            -0.010066395625472069,
+            -0.008871064521372318,
+            0.00502039073035121,
+            -0.01689685881137848,
+            -0.012149686925113201,
+            -0.001152640674263239,
+            0.006753620691597462,
+            -0.00499477656558156,
+            0.0057632033713161945,
+            -0.026075294241309166,
+            0.014010988175868988,
+            -0.0170249305665493,
+            0.01144102681428194,
+            -0.010587218217551708,
+            0.022762520238757133,
+            0.011748396791517735,
+            -0.010356690734624863,
+            -0.0008303282083943486,
+            0.006036422215402126,
+            0.0005264259525574744,
+            0.026126522570848465,
+            0.03237639740109444,
+            -0.0011889275629073381,
+            0.008469774387776852,
+            0.007586083374917507,
+            0.003921112976968288,
+            -0.011014122515916824,
+            0.0015923518221825361,
+            0.014830644242465496,
+            0.01992787793278694,
+            -0.013729231432080269,
+            0.008879602886736393,
+            -0.026672959327697754,
+            0.02103782817721367,
+            -0.022096550092101097,
+            -0.000982412719167769,
+            0.010066395625472069,
+            -0.004777055233716965,
+            0.007090874947607517,
+            0.0066682398319244385,
+            0.021362274885177612,
+            0.03375956416130066,
+            0.00807702261954546,
+            0.008879602886736393,
+            0.007257367484271526,
+            -0.004798400681465864,
+            0.002044870052486658,
+            0.009135744534432888,
+            -0.005869929678738117,
+            -0.0040940092876553535,
+            -0.010083471424877644,
+            -0.006685316096991301,
+            0.016205275431275368,
+            0.0008079157560132444,
+            0.012602205388247967,
+            0.007018301170319319,
+            -0.007428129203617573,
+            -0.005669284611940384,
+            0.007014031987637281,
+            -0.023787088692188263,
+            -0.00950288213789463,
+            0.025221485644578934,
+            -0.013814612291753292,
+            -0.043202683329582214,
+            -0.0024333526380360126,
+            0.008853988721966743,
+            -0.017964119091629982,
+            -0.01245705783367157,
+            -0.004772786516696215,
+            0.010117623955011368,
+            0.0007241358398459852,
+            -0.007569007575511932,
+            0.028175661340355873,
+            0.00502039073035121,
+            -0.005169807001948357,
+            0.003571051638573408,
+            -2.8232359909452498e-05,
+            0.019244831055402756,
+            0.01536000519990921,
+            0.008294744417071342,
+            -0.03266669064760208,
+            -0.007304327096790075,
+            -0.005780279636383057,
+            0.013114489614963531,
+            0.002813297091051936,
+            0.003423769725486636,
+            -0.019740039482712746,
+            -0.020798761397600174,
+            0.019210679456591606,
+            0.004935009870678186,
+            0.019825419411063194,
+            -0.00867895781993866,
+            -0.006548706907778978,
+            -0.004841091111302376,
+            -0.004742903169244528,
+            0.00499477656558156,
+            0.0076415808871388435,
+            0.0038933642208576202,
+            0.03377664089202881,
+            -0.001280711847357452,
+            0.004042780492454767,
+            -0.0036350872833281755,
+            -0.0002441357064526528,
+            0.014787953346967697,
+            -0.00790199264883995,
+            0.008905217051506042,
+            0.018800850957632065,
+            -0.02887578494846821,
+            0.011398335918784142,
+            -0.01679440215229988,
+            -0.00828193686902523,
+            0.016000360250473022,
+            -0.0005320290802046657,
+            -0.004922202788293362,
+            -0.014583039097487926,
+            0.0008127184119075537,
+            0.007752575911581516,
+            0.01996202953159809,
+            -0.004091874696314335,
+            0.011287340894341469,
+            -0.009648029692471027,
+            -0.004305326379835606,
+            0.010083471424877644,
+            0.006087650544941425,
+            -0.0033213128335773945,
+            0.006232798099517822
+          ],
+          "keyphrases": [
+            "MoE models",
+            "expert choice",
+            "training efficiency",
+            "fine-tuning on GLUE and SuperGLUE",
+            "capacity factor"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": true,
+      "properties": {
+        "cosine_similarity": 0.854622020797967
+      }
+    },
+    {
+      "id": "081cd282-43e4-4153-82d9-e8a3e991c311",
+      "type": "cosine_similarity",
+      "source": {
+        "id": "b570d71e-4f83-486e-b70f-b0c6217db455",
+        "properties": {
+          "page_content": "Mixture-Of-Experts With Expert Choice Routing\n\nYanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon Google, Mountain View, CA, USA {yanqiz, taole, hanxiaol, dunan, huangyp, vzhao, adai, zhifengc, qvl, jlaudon}@google.com\n\nAbstract\n\nSparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choice method. Instead of letting tokens select the top-k experts, we have experts selecting the top-k tokens. As a result, each token can be routed to a variable number of experts and each expert can have a fixed bucket size. We systematically study pre-training speedups using the same computational resources of the Switch Transformer top-1 and GShard top-2 gating of prior work and find that our method improves training convergence time by more than 2×. For the same computational cost, our method demonstrates higher performance in fine-tuning 11 selected tasks in the GLUE and SuperGLUE benchmarks. For a smaller activation cost, our method outperforms the T5 dense model in 7 out of the 11 tasks.\n\n1 Introduction\n\nScaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27]. The final model quality has been found to have a power-law relationship with the amount of data, model size, and compute time [16, 20]. However, training efficiency, which is defined as the total amount of computation used to achieve superior model quality than the state of the art system [21], should receive greater attention as we increase our efforts towards green AI [29].\n\nSparsely gated mixture-of-experts [31] (MoE) provides an effective way to scale model capacity given a fixed computational cost, and has recently played an important role in increasing the training efficiency of large-scale language models [10, 21]. MoE operate by adopting a number of experts, each as a sub-network, and by activating only one or a few experts for each input token. A gating network must be chosen and optimized in order to route each token to the most suited expert(s). For example, recent work has implemented sparse routing via k-means clustering [12], linear assignment to maximize token-expert affinities [22], or hashing [8, 28]. Many of the prior work use a routing strategy concerning the token choice, where each token selects the best one or two experts.\n\nWe argue that the independent token choice of prior work often leads to an imbalanced load of experts, which causes training inefficiency and sub-optimal training of the model. In order to mitigate this\n\nissue, previous sparsely gated networks introduce additional auxiliary losses as regularization to prevent too many tokens being routed to a single expert, but the effectiveness is still limited. Recent approaches [8, 22, 28] explore alternative strategies for routing, but they focus on pre-training only and do not demonstrate performance gain on downstream tasks. Moreover, none of the previous methods consider allocating a variable number of experts to each token based on importance, which can be beneficial.\n\nWe propose a very simple yet effective routing method we are calling expert choice. Unlike conventional MoE where tokens select one or two top-scoring experts, our method lets each expert pick the top-k tokens. Our method guarantees perfect load balancing, allows a variable number of experts for each token, and achieves substantial gains in training efficiency and downstream performance as demonstrated in our experiments. Our major contributions include:\n\nWe identify common pitfalls in conventional MoE such as load imbalance as described in Section 3.1. We then propose a heterogeneous, expert choice method to provide a fluid allocation of model parameters based on a learnt token-to-expert importance. This method intrinsically guarantees load balance without imposing an auxiliary loss.\n\nWe show our method provides over 2× faster training convergence in a 8B/64E (8 billion activated parameters, 64 experts) model, compared to the top-1 and top-2 gating counterparts in Switch Transformer [10] and GShard [21].\n\nWe show our method demonstrates strong scaling when increasing the number of experts from 16 to 128, evaluated in training perplexity.\n\nWe show our method demonstrates strong performance on downstream tasks selected from GLUE and SuperGLUE at all the evaluated scales. More specifically, our 8B/64E model outperforms a T5 11B dense model in 7 out of 11 tasks evaluated.\n\n2 Related Work\n\nScaling: Various approaches have been proposed to scale up neural network capacity to improve performance. Recent works have successfully scaled models to billions of parameters via various forms of model parallelism [2, 21, 26, 27, 33]. Model parallelism [30] splits weights and tensors across multiple cores while pipeline parallelism [18, 24] splits different layers across devices with micro-batches pipelined to the different layers. To enable continued scaling of neural networks, improving model training and serving efficiency has become a critical research area.\n\nConditional Computation: Computation decisions can be made dynamically based on the input [23, 25]. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation, by activating certain parameters and computation on demand, on a per-example or per-token basis [3]. Conditional convolution layers [1] with task-specific gating has been used to combat catastrophic forgetting when a sequence of learning problems are optimized. The gating decisions may be binary or sparse and continuous, stochastic or deterministic.\n\nMixture of Experts: Sparsely-gated MoE [31] is the first model to demonstrate massive improvements in model capacity, training time, or model quality with gating. Switch Transformer [10] simplifies the gating by selecting only the top expert per token using a softmax over the hidden state and demonstrates better scaling than previous work. All the prior work requires an auxiliary loss to explicitly encourage balancing. This loss term has to be carefully weighted to not overwhelm the primary loss. However, auxiliary loss does not guarantee balancing and a hard capacity factor has to be imposed. As a result, many tokens can still be unprocessed by the MoE layer. Hard MoE [12] with a single decoding layer can be efficiently trained to good effect on large scale hashtag prediction tasks.\n\nBase Layers [22] formulate a linear assignment that maximizes token-expert affinities while ensuring each expert receives an equal number of tokens. Hash layers [8, 28] devise hashing techniques on input tokens. However, the evaluations are limited to pre-training perplexity. THOR [? ] randomly activates experts during training and inference and is trained with a consistency regularization loss.\n\nTHOR has demonstrated strong performance on translation tasks. Different from these prior works, our method is a learnt method that enables heterogeneous MoE and effectively improves downstream fine-tuning performance.\n\n3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n5 Conclusion\n\nWe propose a new routing method for sparsely activated mixture-of-experts (MoE) models. This method addresses load imbalance and under-utilization of experts in conventional MoE methods, and enables selecting different numbers of experts for each token. Our model demonstrates more than 2x training efficiency improvements when compared to the state-of-the-art GShard and Switch Transformer models, and also achieves strong gains when finetuning on 11 datasets in the GLUE and SuperGLUE benchmark.\n\n6 Limitations\n\nThe expert choice method might not immediately apply to auto-regressive text generation as our current implementation takes in the past and future tokens to perform the top-k selection. One possible solution is to collect a large batch of input sequences, dispatch tokens of the same sequence into separate groups, and perform expert choice routing for each group. Another scenario where the expert choice method does not immediately apply is when the batch size becomes very small during serving or inference. A global top-k can be selected instead and we can cap the number of times each expert or token gets selected. We leave these possible improvements for future work.\n\nAnother long-standing issue with MoE has been the large memory footprint. Even though computational cost can be reduced using sparsely gated networks, the total number of parameters increases linearly or sub-linearly with the number of experts. Increasing the number of experts requires reservation of a large number of hardware devices. Therefore, dynamic (used) power is saved while static (reserved) power is not. Power saving techniques such as the ability to put hardware devices into low power states while not in use [17] can help with reducing the reserved power requirements.\n\nReferences\n\n[1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pages 3930–3939. Computer Vision Foundation / IEEE, 2020.\n\n[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. In Advances in Neural Information Processing Systems.\n\n[3] Kyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning, 2014.\n\n[4] Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems, 2021.\n\n[5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.\n\nTransformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.\n\n[6] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 933–941. JMLR.org, 2017.\n\n[7] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixtureof-experts, 2021.\n\n[8] Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan.\n\nTricks for training sparse translation models, 2021.\n\n[9] Richard L Dykstra. An iterative procedure for obtaining i-projections onto the intersection of convex sets. The annals of Probability, pages 975–984, 1985.\n\n[10] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.\n\n[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. NAS-FPN: learning scalable feature pyramid architecture for object detection. In CVPR, pages 7036–7045. Computer Vision Foundation / IEEE, 2019.\n\n[12] Sam Gross, Marc'Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision, 2017.\n\n[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.\n\n[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 630–645, Cham, 2016. Springer International Publishing.\n\n[15] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2016.\n\n[16] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.\n\n[17] Ping Huang, Zuocheng Xing, Tianran Wang, Qiang Wei, Hongyan Wang, and Guitao Fu. A brief survey on power gating design. In 2010 10th IEEE International Conference on Solid-State and Integrated Circuit Technology, pages 788–790, 2010.\n\n[18] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 103–112, 2019.\n\n[19] Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David A. Patterson. A domain-specific supercomputer for training deep neural networks. Commun. ACM, 63(7):67–78, 2020.\n\n[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.\n\n[21] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.\n\n[22] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6265–6274. PMLR, 18–24 Jul 2021.\n\n[24] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. New York, NY, USA, 2019. Association for Computing Machinery.\n\n[25] Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, Cédric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. In ICLR. OpenReview.net, 2021.\n\n[26] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.\n\n[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.\n\n[28] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models, 2021.\n\n[30] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 10435–10444, Red Hook, NY, USA, 2018. Curran Associates Inc.\n\n[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E.\n\nHinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In ICLR (Poster). OpenReview.net, 2017.\n\n[32] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 10–15 Jul 2018. [23] Min Lin, Jie Fu, and Yoshua Bengio. Conditional computation for continual learning, 2019.\n\n[29] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.\n\n[33] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.\n\n[34] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc.\n\n[35] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.\n\nGLUE: A multi-task benchmark and analysis platform for natural language understanding.\n\nIn Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, November 2018. Association for Computational Linguistics.\n\n[36] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake A. Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen. GSPMD: general and scalable parallelization for ML computation graphs. CoRR, abs/2105.04663, 2021.\n\n7 Checklist\n\n(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Yes (b) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes (c) Did you discuss any potential negative societal impacts of your work? N/A. Not any. (d) Did you describe the limitations of your work? Yes (a) Did you include the code, data, and instructions needed to reproduce the main experimental results? Yes. We include details in the experiment setup to help reproduce the main results. (b) Did you specify all the training details? Yes (c) Did you report error bars? Yes (d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? Yes (a) If your work uses existing assets, did you cite the creators? Yes (b) Did you mention the license of the assets? No. The used dataset is not released yet. (c) Did you include any new assets either in the supplemental material or as a URL? No. The dataset is not released yet.\n\n(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? No. Not using persons' data. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes. The dataset does not contain any personally identifiable information or offensive content.\n\nA Comparison On Fine-Tuning With A Dense Model\n\nOur 8B MoE model achieves stronger pre-training perplexity than its dense counterpart. However, a better perplexity does not always directly translate to downstream performance as demonstrated in Section 4.4. To this end, we compare fine-tuning performance of the 8B dense model and MoE model in Table 1. As shown in the table, our MoE model using expert choice routing consistently outperforms the dense model across the 11 tasks in GLUE and SuperGLUE.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 1: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nB Capacity Factor\n\nWe evaluate the downstream task fine-tuning performance by varying the capacity factors. Note that a capacity factor of n indicates on average how many experts each token can be received. EC-CF2 is our baseline expert choice, which matches GShard top-2 gating computational footprint. EC-CF1, however, matches Switch Transformer top-1 gating computational footprint. EC-CF0.5 further verifies that an aggressively lowered capacity factor can provide strong enough performance, that almost matches the top-2 gating baseline.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Top-2 78.1 87.0 88.3 85.0 82.6 90.1 90.7 81.6 94.7 68.2 67.2 83.0±0.3 EC-CAP2 78.2 88.0 88.5 85.7 83.0 90.8 91.1 80.0 95.4 70.4 64.1 83.2±0.4 EC-CAP3 78.5 91.7 89.3 86.3 83.5 90.9 91.1 81.8 94.9 70.0 65.6 84.0±0.4 EC-CF2 79.1 89.6 89.3 86.8 84.3 91.3 91.2 81.1 95.2 68.1 68.0 84.0±0.2 EC-CF1 77.4 90.6 88.0 85.5 83.6 90.3 91.2 79.8 95.3 66.5 64.9 83.0±0.2 EC-CF0.5 77.4 89.6 86.3 85.2 82.7 91.7 91.0 79.6 94.9 67.3 63.5 83.0 ±0.05 Hash Layers 76.1 85.2 86.7 83.4 82.5 90.0 90.3 75.7 94.0 67.4 63.3 81.3±1.0\n\nTable 2: Comparison between different routing methods in fine-tuning of 100M/64E models. We perform 3 independent fine-tuning runs for each method and report the average results. This gives more accurate difference between the variants of expert choice method, since they achieve close fine-tuning results. We do not report averaged results in other experiments.\n\nC Capped Expert Choice\n\nAs described in Section 4.5, the maximum number of experts each token is assigned can be capped by an entropy-regularized linear programming. Figure 1 compares the validation perplexity when training the 100M/64E models using the base expert choice method (EC-BASE), expert choice capped by two experts per token (EC-CAP2), expert choice capped by three experts per token (EC-CAP3), and GShard top-2 gating.\n\nAs shown in the figure, restricting the number of experts to 2 degrades the perplexity compared to the base expert choice method. This suggests that a more flexible allocation of experts (e.g. more than 2 experts for a token) can enhance model expressiveness. On the other hand, our EC-CAP2 and EC-CAP3 methods still outperform the top-2 gating method by a clear margin. We believe this confirms the effectiveness of a load balanced training, provided by our method. Finally, EC-CAP3 obtains comparable perplexity to EC-BASE. As indicated by Figure 3, only a little fraction of tokens use more than 3 experts therefore we see little or no difference between EC-BASE and EC-CAP3 variants. We present the fine-tuning results of these methods in Table 2.\n\nD Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [? ]. We use mod x to map a token ID to an expert ID. This in some way ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 2. Hashing based routing performs much worse than expert choice in terms of average scores and variance.\n\nE Fine-Tuning Details\n\nWe did a hyperparameter search for both baseline models and expert choice method. For fine-tuning of the 8B dense model, we use a constant learning rate of 0.0001 and a dropout rate of 0.1. We freeze the attention layer and feed-forward layer while leaving the embedding and layer normalization trainable. This setting has been found optimal for the 8B dense model. For MoE 8B/64E models including GShard top-2 gating and expert choice, we found continuing the learning rate from the pre-trained model while using a square root learning rate decay works better. In addition, we do not apply parameter freezing for fine-tuning MoE models. For models with 100M expert size, we use a constant learning rate of 0.0001 and no dropout is used.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/2202.09368v2.md"
+          },
+          "headlines": {
+            "1 Introduction": [],
+            "2 Related Work": [],
+            "3 Method": [
+              "3.1 Pitfalls Of Token-Choice Routing",
+              "3.2 Heterogeneous Moe Via Expert Choice",
+              "3.3 Expert Choice With Additional Constraint",
+              "3.4 Model Architecture"
+            ],
+            "4 Experiments": [
+              "4.1 Setup",
+              "4.2 Training Efficiency",
+              "4.3 Scaling The Number Of Experts",
+              "4.4 Fine-Tuning On Glue And Superglue",
+              "4.5 Heterogeneity Matters",
+              "4.6 Comparison With Hash Layer",
+              "4.7 Ablation"
+            ],
+            "5 Conclusion": [],
+            "6 Limitations": [],
+            "A Comparison On Fine-Tuning With A Dense Model": [],
+            "B Capacity Factor": [],
+            "C Capped Expert Choice": [],
+            "D Comparison With Hash Layer": [],
+            "E Fine-Tuning Details": []
+          },
+          "summary": "The paper introduces a new routing method for sparsely-activated Mixture-of-Experts (MoE) models called 'expert choice,' which addresses load imbalance and under-utilization issues in conventional MoE methods. Unlike traditional token-choice routing, expert choice allows experts to select the top-k tokens, ensuring perfect load balancing and enabling variable expert allocation per token. This method improves training efficiency, achieving over 2x faster convergence compared to existing models like Switch Transformer and GShard. It also demonstrates superior performance on downstream tasks in the GLUE and SuperGLUE benchmarks. The study highlights the benefits of flexible expert allocation and shows that expert choice outperforms hashing-based routing methods. The paper also discusses potential limitations and future improvements for the expert choice method.",
+          "summary_embedding": [
+            -0.009726986289024353,
+            -0.0337262786924839,
+            -0.023302342742681503,
+            -0.019878201186656952,
+            -0.025953780859708786,
+            0.04560471698641777,
+            -0.006874797400087118,
+            0.039665497839450836,
+            -0.006973279174417257,
+            0.034544438123703,
+            -0.025787118822336197,
+            -0.00363057479262352,
+            -0.01052999310195446,
+            0.0054657477885484695,
+            -0.024499278515577316,
+            0.036786794662475586,
+            0.010007280856370926,
+            0.003272630739957094,
+            0.006102093029767275,
+            -0.04048365354537964,
+            -0.018666116520762444,
+            -0.013522328808903694,
+            -0.03230207785964012,
+            0.008969432674348354,
+            0.02484775334596634,
+            0.0016713523073121905,
+            0.0026230288203805685,
+            0.003624893259257078,
+            -0.05311964824795723,
+            0.04339266195893288,
+            0.018681267276406288,
+            0.023711422458291054,
+            -0.016332851722836494,
+            -0.012832955457270145,
+            -0.039120059460401535,
+            -0.0197418425232172,
+            0.030786970630288124,
+            0.039968520402908325,
+            -0.019287310540676117,
+            0.03357476741075516,
+            0.018060073256492615,
+            0.02454473078250885,
+            -0.021272100508213043,
+            -0.029817303642630577,
+            -0.0061892117373645306,
+            -0.007840678095817566,
+            -0.008355814032256603,
+            0.01242387667298317,
+            0.016984347254037857,
+            -0.013007191941142082,
+            0.0212569497525692,
+            -0.01371171697974205,
+            0.03384748846292496,
+            0.008916404098272324,
+            -0.042332086712121964,
+            0.03984731063246727,
+            -0.008007340133190155,
+            0.012105703353881836,
+            0.03281721472740173,
+            0.010333028621971607,
+            -0.005287723150104284,
+            -0.021544819697737694,
+            -0.03378688171505928,
+            0.0010529992869123816,
+            0.00014014738553669304,
+            0.005950582213699818,
+            -0.003069985192269087,
+            -0.009742137044668198,
+            0.03227177634835243,
+            -0.003939277958124876,
+            -0.0740584209561348,
+            0.050089433789253235,
+            -0.026590125635266304,
+            -0.007628562860190868,
+            -0.028938541188836098,
+            -0.01736312545835972,
+            0.037362534552812576,
+            0.005662711802870035,
+            0.0098709212616086,
+            0.0106284748762846,
+            -0.0007892759749665856,
+            0.028923390433192253,
+            0.00910579226911068,
+            -0.024923507124185562,
+            0.04478655755519867,
+            -0.04790768027305603,
+            0.01801462098956108,
+            -0.04336235672235489,
+            0.022756904363632202,
+            1.6393929399782792e-05,
+            -0.020726662129163742,
+            -0.0013304532039910555,
+            -0.016666175797581673,
+            0.0218023881316185,
+            -0.01730252057313919,
+            0.026802239939570427,
+            0.013825350441038609,
+            -0.015802564099431038,
+            -0.029817303642630577,
+            0.025378039106726646,
+            0.003367325058206916,
+            0.010499690659344196,
+            -0.02075696364045143,
+            0.014332910999655724,
+            0.04393810033798218,
+            -0.02593863010406494,
+            -0.02271145209670067,
+            0.027681002393364906,
+            0.014408666640520096,
+            0.0633314698934555,
+            0.036544378846883774,
+            0.006624804809689522,
+            0.022362977266311646,
+            0.020575150847434998,
+            -0.028968842700123787,
+            -0.01963578537106514,
+            -0.046362269669771194,
+            -0.04757435619831085,
+            0.019878201186656952,
+            -0.04566532000899315,
+            0.0050869714468717575,
+            0.038544319570064545,
+            -0.013704141601920128,
+            -0.014446543529629707,
+            -0.04021093621850014,
+            0.017847958952188492,
+            0.0003908502112608403,
+            0.011976920068264008,
+            -0.0022082682698965073,
+            0.00033285003155469894,
+            -0.011825408786535263,
+            0.008878526277840137,
+            -0.03778676688671112,
+            -0.025196226313710213,
+            0.025181075558066368,
+            0.022862963378429413,
+            0.024120500311255455,
+            0.028802180662751198,
+            -0.03077181987464428,
+            -0.009984553791582584,
+            0.021378157660365105,
+            -0.04242299124598503,
+            -0.006556625012308359,
+            0.011332998983561993,
+            0.0026703758630901575,
+            0.003872992005199194,
+            0.032786913216114044,
+            0.052998438477516174,
+            -0.03172633796930313,
+            0.02065090648829937,
+            -0.010431510396301746,
+            0.04957429692149162,
+            0.00024691506405360997,
+            0.028968842700123787,
+            -0.019514575600624084,
+            -0.0134389977902174,
+            -0.029529431834816933,
+            -0.009795165620744228,
+            -0.01356020662933588,
+            -0.008484598249197006,
+            -0.002998017705976963,
+            -0.009575475007295609,
+            0.0040453351102769375,
+            0.0331808403134346,
+            0.00273855566047132,
+            0.030680913478136063,
+            0.014325335621833801,
+            0.0051475754007697105,
+            -0.08745196461677551,
+            0.026059838011860847,
+            -0.0033616432920098305,
+            0.005859675817191601,
+            0.0062460280023515224,
+            0.03013547509908676,
+            -0.0312718041241169,
+            0.020938776433467865,
+            0.026620427146553993,
+            0.0011145505122840405,
+            0.016969196498394012,
+            0.036847397685050964,
+            -0.04181694984436035,
+            0.009643655270338058,
+            0.012590537779033184,
+            -0.0033881578128784895,
+            0.04584713280200958,
+            0.011628445237874985,
+            0.019514575600624084,
+            0.018529755994677544,
+            0.003569970605894923,
+            0.00297907879576087,
+            0.006533898413181305,
+            0.03724132478237152,
+            0.003462019143626094,
+            -0.009833043441176414,
+            0.014976831153035164,
+            -0.010757259093225002,
+            -0.011219366453588009,
+            -0.037998881191015244,
+            0.014454119838774204,
+            0.008742166683077812,
+            -0.017454031854867935,
+            -0.02959003672003746,
+            0.0026949963066726923,
+            -0.031908150762319565,
+            0.026559824123978615,
+            -0.002445003716275096,
+            0.005738467443734407,
+            -0.010454237461090088,
+            -0.006757376715540886,
+            -0.035423196852207184,
+            0.028453707695007324,
+            -0.02095392718911171,
+            0.0053142374381423,
+            0.0007258309051394463,
+            -0.029438525438308716,
+            0.0060073984786868095,
+            0.017863109707832336,
+            -0.0025965142995119095,
+            0.0025302283465862274,
+            -0.02349930815398693,
+            0.024468975141644478,
+            0.025635607540607452,
+            0.004518806003034115,
+            -0.014037464745342731,
+            -0.013113250024616718,
+            -0.003605954349040985,
+            -0.009742137044668198,
+            -0.016196493059396744,
+            0.024468975141644478,
+            -0.014810169115662575,
+            0.00308892410248518,
+            0.021847840398550034,
+            0.014264730736613274,
+            -0.007045246660709381,
+            -0.004007457755506039,
+            -0.04066546633839607,
+            -0.01218903437256813,
+            -0.04760465770959854,
+            -0.011628445237874985,
+            -0.0034601252991706133,
+            -0.028696123510599136,
+            0.0061513339169323444,
+            -0.04481685906648636,
+            -0.01235569640994072,
+            0.02309022843837738,
+            0.018135828897356987,
+            0.0019488062243908644,
+            -0.009173971600830555,
+            -0.022605394944548607,
+            8.486965816700831e-05,
+            -0.01763584464788437,
+            -0.017044952139258385,
+            0.015393485315144062,
+            0.031059689819812775,
+            -0.03101423755288124,
+            0.04024123772978783,
+            0.003893824527040124,
+            -0.08066429197788239,
+            0.010393633507192135,
+            -0.026362858712673187,
+            0.07011914253234863,
+            -0.0025453795678913593,
+            0.014075342565774918,
+            0.06781618297100067,
+            -0.012431452050805092,
+            -0.02007516659796238,
+            0.03142331540584564,
+            -0.008249756880104542,
+            -1.1311515663692262e-05,
+            -0.02884763479232788,
+            -0.02739313244819641,
+            -0.053271155804395676,
+            -0.009704259224236012,
+            -0.010787560604512691,
+            0.03360506892204285,
+            -0.024317465722560883,
+            -0.0007656024536117911,
+            0.012620840221643448,
+            0.02983245439827442,
+            0.016635872423648834,
+            -0.0015681355725973845,
+            0.008272483013570309,
+            0.02881733328104019,
+            0.0274234339594841,
+            -0.0036154238041490316,
+            0.00814369972795248,
+            -0.034211114048957825,
+            -0.0068066176027059555,
+            -0.0493621826171875,
+            0.011302697472274303,
+            -0.02596893161535263,
+            -0.02596893161535263,
+            0.007643714081496,
+            -0.011476934887468815,
+            0.03748374432325363,
+            -0.024696242064237595,
+            -0.007503566797822714,
+            0.005037730094045401,
+            0.012855681590735912,
+            -0.017757052555680275,
+            -0.03612014651298523,
+            -0.0028502948116511106,
+            0.0026381798088550568,
+            0.007560383062809706,
+            -0.053240854293107986,
+            0.004499867558479309,
+            -0.018469152972102165,
+            0.0279688723385334,
+            -0.02139330841600895,
+            -0.013688990846276283,
+            0.042029064148664474,
+            0.011363301426172256,
+            0.027499189600348473,
+            -0.026468917727470398,
+            0.0017849853029474616,
+            0.012825379148125648,
+            -0.0020056227222085,
+            0.032514192163944244,
+            -0.030574856325984,
+            -0.03257479891180992,
+            0.001548249856568873,
+            -0.007840678095817566,
+            -0.0625436082482338,
+            -0.011386028490960598,
+            -0.00981789268553257,
+            0.017317671328783035,
+            0.0446956530213356,
+            0.04557441547513008,
+            -0.01068150345236063,
+            0.002174178371205926,
+            -0.02213571034371853,
+            -0.04542290419340134,
+            -0.02643861435353756,
+            0.0016486257081851363,
+            -0.019999410957098007,
+            -0.02789311669766903,
+            -0.0018313854234293103,
+            0.011211791075766087,
+            -0.013651113025844097,
+            0.01134057529270649,
+            -0.018560059368610382,
+            -0.01862066239118576,
+            -0.012332970276474953,
+            0.018575210124254227,
+            0.009901223704218864,
+            -0.0018796793883666396,
+            -0.015037435106933117,
+            -0.00980274099856615,
+            0.018969137221574783,
+            -0.011279970407485962,
+            0.0009388927719555795,
+            0.01188601367175579,
+            0.024347767233848572,
+            -0.007083124481141567,
+            0.010961798019707203,
+            -0.034756552428007126,
+            -0.01966608688235283,
+            0.007984613068401814,
+            0.002719616750255227,
+            0.020711511373519897,
+            0.008742166683077812,
+            -0.04275631532073021,
+            -0.044301725924015045,
+            0.016044981777668,
+            -0.0562104657292366,
+            0.028120383620262146,
+            -0.03327174484729767,
+            -0.012514783069491386,
+            -0.02671133354306221,
+            0.0013418165035545826,
+            -0.01828734017908573,
+            0.022438732907176018,
+            0.03778676688671112,
+            -0.0011486404109746218,
+            0.005503625608980656,
+            0.014658658765256405,
+            0.04384719207882881,
+            -0.05021063983440399,
+            -0.041180603206157684,
+            0.03754434734582901,
+            -0.01601468026638031,
+            0.0185146052390337,
+            0.00021744151308666915,
+            0.010348179377615452,
+            -0.014431392773985863,
+            0.022772056981921196,
+            0.013757170177996159,
+            -0.008764892816543579,
+            -0.030574856325984,
+            0.04945308715105057,
+            -0.023650817573070526,
+            -0.049513693898916245,
+            0.0018323323456570506,
+            0.018878230825066566,
+            0.018635815009474754,
+            -0.014870774000883102,
+            -0.02907489985227585,
+            -0.024317465722560883,
+            -0.006060427520424128,
+            -0.00947699323296547,
+            -0.03402930125594139,
+            0.011363301426172256,
+            0.0048445542342960835,
+            0.016666175797581673,
+            -0.00645056739449501,
+            -0.0073141781613230705,
+            -0.009370936080813408,
+            0.022559940814971924,
+            0.012635990977287292,
+            -0.03175663948059082,
+            -0.017484333366155624,
+            -0.02528713271021843,
+            0.004359719809144735,
+            0.010136065073311329,
+            0.022166013717651367,
+            0.024287162348628044,
+            -0.0323626808822155,
+            0.04218057543039322,
+            0.01550711877644062,
+            -0.008727015927433968,
+            -0.0019317612750455737,
+            0.02478714846074581,
+            0.04324115067720413,
+            0.0218023881316185,
+            0.026514369994401932,
+            0.002107892418280244,
+            8.705945947440341e-05,
+            -0.015620751306414604,
+            -0.0011391709558665752,
+            0.01862066239118576,
+            0.04911976307630539,
+            -0.012151156552135944,
+            0.01716616004705429,
+            -0.03578682243824005,
+            0.00396200455725193,
+            -0.02580226957798004,
+            -0.030786970630288124,
+            0.03745344281196594,
+            -0.015060162171721458,
+            0.0191964041441679,
+            0.025044716894626617,
+            -0.022469034418463707,
+            -0.011075431481003761,
+            -0.019423669204115868,
+            -0.026181045919656754,
+            -0.01702980138361454,
+            -0.03281721472740173,
+            -0.018135828897356987,
+            -0.002653330797329545,
+            -0.035635314881801605,
+            -0.04987731948494911,
+            -0.004329417832195759,
+            -0.046392571181058884,
+            -0.010507266037166119,
+            0.017378276214003563,
+            -0.011166337877511978,
+            -0.025711363181471825,
+            -0.005336964037269354,
+            0.003973367623984814,
+            -0.022196315228939056,
+            -0.0066588944755494595,
+            0.008439145050942898,
+            0.035938333719968796,
+            -0.02298417128622532,
+            -0.013060221448540688,
+            -0.0024317465722560883,
+            -0.02887793630361557,
+            -0.007878555916249752,
+            0.008999735116958618,
+            0.007723257411271334,
+            -0.0012357590021565557,
+            -0.05196816474199295,
+            -0.006916462909430265,
+            -0.012992041185498238,
+            0.029196109622716904,
+            0.026468917727470398,
+            0.0012168202083557844,
+            -0.02783251367509365,
+            0.008749742060899734,
+            -0.0072535742074251175,
+            0.02339324913918972,
+            -0.007992188446223736,
+            -0.022908415645360947,
+            -0.02034788578748703,
+            0.030786970630288124,
+            -0.0159086212515831,
+            0.009113367646932602,
+            0.009976978413760662,
+            0.0121360057964921,
+            -0.03899884968996048,
+            -0.001445980160497129,
+            0.0005648507503792644,
+            -0.010128489695489407,
+            0.00423093605786562,
+            -0.006738437805324793,
+            -0.007079336792230606,
+            -0.022090258076786995,
+            -0.0218023881316185,
+            0.03818069398403168,
+            -0.009113367646932602,
+            -0.0035661826841533184,
+            -0.01088604237884283,
+            0.03436262533068657,
+            -0.00595815759152174,
+            -0.01993880607187748,
+            -0.019347915425896645,
+            0.014431392773985863,
+            0.05105910077691078,
+            -0.028529461473226547,
+            -0.03984731063246727,
+            0.0011136035900563002,
+            0.030150625854730606,
+            0.004140029661357403,
+            -0.016908591613173485,
+            0.0035851215943694115,
+            0.056392278522253036,
+            -0.011742077767848969,
+            0.028362801298499107,
+            0.027771908789873123,
+            0.02295386977493763,
+            -0.02021152526140213,
+            -0.01678738370537758,
+            -0.020999381318688393,
+            0.011098157614469528,
+            -0.03027183562517166,
+            0.027059808373451233,
+            0.018969137221574783,
+            0.01538590993732214,
+            0.05054396390914917,
+            -0.03178694099187851,
+            -0.02515077404677868,
+            0.008499749936163425,
+            0.0021722845267504454,
+            0.011007251217961311,
+            -0.007727045100182295,
+            -0.01801462098956108,
+            -0.001950700068846345,
+            -0.014022313989698887,
+            -0.019817598164081573,
+            -0.020938776433467865,
+            0.00954517349600792,
+            0.015696506947278976,
+            0.002649543108418584,
+            0.010863316245377064,
+            0.0016590419691056013,
+            0.01686313934624195,
+            0.01321930717676878,
+            -0.0444229319691658,
+            -7.936555630294606e-05,
+            0.03378688171505928,
+            -0.0293476190418005,
+            -0.012264790013432503,
+            -0.016060132533311844,
+            -0.004439263138920069,
+            -0.01706010289490223,
+            -0.02762039750814438,
+            -0.006874797400087118,
+            -0.01889338344335556,
+            -0.00814369972795248,
+            -0.028196139261126518,
+            -0.009992130100727081,
+            0.01949942484498024,
+            0.022332675755023956,
+            0.001196934375911951,
+            -0.006783891003578901,
+            -0.005901341326534748,
+            0.02092362567782402,
+            0.032514192163944244,
+            0.01952972821891308,
+            0.0058331615291535854,
+            -0.015158643946051598,
+            -0.004295328166335821,
+            -0.012370847165584564,
+            0.016060132533311844,
+            0.012537509202957153,
+            6.344509893096983e-05,
+            0.03493836522102356,
+            0.004170331638306379,
+            -0.022726602852344513,
+            -0.001230077352374792,
+            0.024923507124185562,
+            -0.021590271964669228,
+            -0.0108330138027668,
+            0.040029123425483704,
+            0.0085679292678833,
+            -0.0024506852496415377,
+            -0.019287310540676117,
+            -0.019075196236371994,
+            0.02630225569009781,
+            -0.02133270539343357,
+            0.002083271974697709,
+            0.0482713058590889,
+            0.00035344602656550705,
+            -0.029802151024341583,
+            0.017954016104340553,
+            -0.013242034241557121,
+            -0.018560059368610382,
+            0.01848430372774601,
+            0.0012859469279646873,
+            -0.039968520402908325,
+            0.019014591351151466,
+            0.018408548086881638,
+            0.022635696455836296,
+            -0.02918095886707306,
+            0.0002127068000845611,
+            0.03233237937092781,
+            0.004037759732455015,
+            0.001595596899278462,
+            -0.008530051447451115,
+            -0.0006065162015147507,
+            0.010431510396301746,
+            0.001746160676702857,
+            0.02854461409151554,
+            -0.0030434709042310715,
+            0.027771908789873123,
+            -0.006727074272930622,
+            -0.008863375522196293,
+            0.019181253388524055,
+            -0.02109028771519661,
+            0.013166278600692749,
+            0.0685434341430664,
+            0.0021950111258774996,
+            0.025847723707556725,
+            0.007181606721132994,
+            0.024347767233848572,
+            0.02018122375011444,
+            -0.027271922677755356,
+            -0.02021152526140213,
+            -0.005397567991167307,
+            -0.0014033677289262414,
+            -0.03787767142057419,
+            0.011128460057079792,
+            -0.021302402019500732,
+            -0.013249609619379044,
+            -0.0022347825579345226,
+            -0.002316219499334693,
+            -0.006405114196240902,
+            0.034211114048957825,
+            -0.0014630251098424196,
+            -0.018560059368610382,
+            -0.035423196852207184,
+            0.037726160138845444,
+            0.04048365354537964,
+            0.0389382466673851,
+            -0.005984671879559755,
+            0.03548380360007286,
+            0.0030756669584661722,
+            0.0025302283465862274,
+            0.041453324258327484,
+            0.004791525658220053,
+            -0.011567841283977032,
+            0.02295386977493763,
+            0.012416300363838673,
+            0.03351416438817978,
+            -0.02045394293963909,
+            -0.02630225569009781,
+            -0.003299145260825753,
+            -0.026423463597893715,
+            0.012492056004703045,
+            -0.019726691767573357,
+            0.0018768386216834188,
+            0.007117214612662792,
+            0.015196521766483784,
+            -0.016044981777668,
+            0.014719262719154358,
+            -0.006102093029767275,
+            -0.021014532074332237,
+            0.011719351634383202,
+            -0.007780073676258326,
+            0.011961768381297588,
+            -0.02262054570019245,
+            0.001138224033638835,
+            -0.04063516482710838,
+            0.008280059322714806,
+            0.014022313989698887,
+            -0.01598437689244747,
+            0.03205966204404831,
+            0.020969079807400703,
+            -0.01801462098956108,
+            -0.0009682480012997985,
+            0.0034714885987341404,
+            -0.010393633507192135,
+            -0.011067855171859264,
+            -0.019923655316233635,
+            0.02786281518638134,
+            -0.015802564099431038,
+            -0.025908326730132103,
+            0.02454473078250885,
+            -0.01743887923657894,
+            0.015151068568229675,
+            -0.004299115855246782,
+            0.00888610165566206,
+            -0.019120648503303528,
+            0.02284781076014042,
+            -0.023741723969578743,
+            -0.0009128518868237734,
+            -0.020287280902266502,
+            0.0028389315120875835,
+            0.014135947450995445,
+            -0.018272189423441887,
+            -0.02139330841600895,
+            0.019150950014591217,
+            -0.005420294590294361,
+            -0.003931702114641666,
+            0.010014856234192848,
+            0.0030983935575932264,
+            -0.0029715034179389477,
+            -0.004549108445644379,
+            -0.03281721472740173,
+            -0.02915065549314022,
+            0.0033086147159337997,
+            0.011386028490960598,
+            0.015257125720381737,
+            0.027574945241212845,
+            0.03715042024850845,
+            0.006052851676940918,
+            0.0036949669010937214,
+            0.0009327376610599458,
+            -0.020726662129163742,
+            0.042635105550289154,
+            -0.020469093695282936,
+            0.040847279131412506,
+            0.032544493675231934,
+            -0.0197418425232172,
+            -0.023908386006951332,
+            0.0029052174650132656,
+            0.03115059621632099,
+            0.03151422366499901,
+            0.025211378931999207,
+            0.012310243211686611,
+            0.03690800443291664,
+            -0.004359719809144735,
+            -0.032211173325777054,
+            0.015802564099431038,
+            0.022256920114159584,
+            -0.008105821907520294,
+            0.009673957712948322,
+            0.027438584715127945,
+            0.013279912061989307,
+            -0.007757347077131271,
+            0.030150625854730606,
+            0.0012234487803652883,
+            0.02396899089217186,
+            0.02498411200940609,
+            -0.02190844528377056,
+            0.01743887923657894,
+            -0.0001154677156591788,
+            0.006727074272930622,
+            -0.009840618818998337,
+            0.004348356742411852,
+            0.011810258030891418,
+            0.018272189423441887,
+            0.018908534198999405,
+            0.003246116451919079,
+            -0.07060398161411285,
+            0.019726691767573357,
+            0.004507442936301231,
+            -0.02160542458295822,
+            0.00390140013769269,
+            -0.0024866692256182432,
+            0.012704171240329742,
+            0.02105998620390892,
+            0.012461753562092781,
+            -0.01790856383740902,
+            0.0013626492582261562,
+            0.011961768381297588,
+            -0.00595815759152174,
+            0.04730163514614105,
+            0.0019791084341704845,
+            -0.01437078882008791,
+            -0.016105586662888527,
+            0.013726867735385895,
+            0.02525683119893074,
+            -0.02355991117656231,
+            0.006325571332126856,
+            0.005621046293526888,
+            -0.004147605039179325,
+            -0.010742107406258583,
+            -0.030377892777323723,
+            -0.005280147306621075,
+            -0.03436262533068657,
+            0.023150833323597908,
+            0.015491968020796776,
+            0.014764715917408466,
+            -0.025044716894626617,
+            -0.004280176945030689,
+            0.0046400148421525955,
+            -0.021287251263856888,
+            0.003261267440393567,
+            0.01161329448223114,
+            0.0005104016163386405,
+            -0.020984230563044548,
+            0.00959820207208395,
+            0.013469300232827663,
+            0.06848283112049103,
+            0.0007442962378263474,
+            0.029771849513053894,
+            0.0422411784529686,
+            -0.0012243957025930285,
+            -0.006371024530380964,
+            0.0036286809481680393,
+            -0.019953956827521324,
+            0.003473382443189621,
+            0.025878025218844414,
+            -0.03224147483706474,
+            -0.055058982223272324,
+            -0.007545232307165861,
+            -0.04624105989933014,
+            0.001489539397880435,
+            0.014711687341332436,
+            0.023044776171445847,
+            -0.0037782976869493723,
+            -0.034817155450582504,
+            0.016908591613173485,
+            -0.003071879269555211,
+            0.007014944683760405,
+            -0.02593863010406494,
+            -0.0294839795678854,
+            -0.026014383882284164,
+            -0.02339324913918972,
+            -0.012908710166811943,
+            -0.022726602852344513,
+            0.012590537779033184,
+            -0.0015103721525520086,
+            -0.06587684899568558,
+            -0.009780014865100384,
+            0.016166189685463905,
+            0.001200722181238234,
+            0.011939042247831821,
+            0.011083006858825684,
+            -0.007083124481141567,
+            -0.0018275976181030273,
+            -0.022559940814971924,
+            0.03124150261282921,
+            0.005177877843379974,
+            0.03584742918610573,
+            -0.006503595970571041,
+            -0.009128518402576447,
+            0.01073453202843666,
+            0.04527139291167259,
+            0.0017916138749569654,
+            0.010226971469819546,
+            -0.020560000091791153,
+            0.0015065843472257257,
+            -0.02284781076014042,
+            -0.0031135445460677147,
+            -0.008863375522196293,
+            -0.0021211495622992516,
+            0.015893470495939255,
+            0.0019753205124288797,
+            -0.010204244405031204,
+            0.025438643991947174,
+            0.02251448854804039,
+            -0.009605777449905872,
+            -0.015969226136803627,
+            -0.02437806874513626,
+            -0.014545026235282421,
+            -0.0002627763315103948,
+            -0.010007280856370926,
+            -0.022453883662819862,
+            -0.020878173410892487,
+            -0.002566212322562933,
+            -0.02559015527367592,
+            0.022529639303684235,
+            0.01220418605953455,
+            -4.882668508798815e-05,
+            -0.006723286584019661,
+            -0.038544319570064545,
+            -0.003727162955328822,
+            -0.03118089959025383,
+            -0.005374841392040253,
+            0.03287781774997711,
+            -0.03745344281196594,
+            -0.0002231231628684327,
+            0.014491996727883816,
+            -0.01993880607187748,
+            0.0028938541654497385,
+            0.008961857296526432,
+            0.02769615314900875,
+            -0.027408283203840256,
+            0.0001795638381736353,
+            0.006954340264201164,
+            0.016908591613173485,
+            -0.009878496639430523,
+            0.027196168899536133,
+            -0.04275631532073021,
+            0.011037553660571575,
+            -0.037120118737220764,
+            -0.020423639565706253,
+            -0.01523439958691597,
+            0.008363389410078526,
+            -0.005018791649490595,
+            -0.02637801133096218,
+            -0.010454237461090088,
+            0.006026337388902903,
+            -0.00693918950855732,
+            -0.01902974210679531,
+            -0.010620899498462677,
+            0.027438584715127945,
+            0.00931033119559288,
+            0.007977037690579891,
+            -0.02955973520874977,
+            -0.01631770096719265,
+            0.006280118133872747,
+            0.052089374512434006,
+            -0.01584801822900772,
+            0.014938953332602978,
+            -0.006639956030994654,
+            0.02342355251312256,
+            -0.028484009206295013,
+            -0.01872672140598297,
+            0.0315445251762867,
+            0.013946558348834515,
+            0.007064185570925474,
+            0.028953691944479942,
+            0.027196168899536133,
+            0.027105262503027916,
+            -0.04654408246278763,
+            0.009886072017252445,
+            -0.002613559365272522,
+            -0.008757317438721657,
+            0.03402930125594139,
+            0.01916610263288021,
+            0.022150862962007523,
+            0.0009744031121954322,
+            -0.01730252057313919,
+            0.0139844361692667,
+            0.014885924756526947,
+            0.007681591901928186,
+            -0.014242004603147507,
+            -0.02660527639091015,
+            0.003367325058206916,
+            -0.007992188446223736,
+            0.00047276070108637214,
+            0.020726662129163742,
+            -0.009052763693034649,
+            0.030999086797237396,
+            -0.014605630189180374,
+            -0.02593863010406494,
+            0.008916404098272324,
+            -0.02268115058541298,
+            -0.0041741193272173405,
+            0.0004476667381823063,
+            0.010083036497235298,
+            -0.023847781121730804,
+            -0.04669559374451637,
+            0.019408518448472023,
+            0.028393102809786797,
+            0.005579380784183741,
+            0.01328748743981123,
+            -0.035332292318344116,
+            0.008992158807814121,
+            0.012310243211686611,
+            0.011348150670528412,
+            0.0019847899675369263,
+            0.035362593829631805,
+            -0.00028834378463216126,
+            0.012885984033346176,
+            -0.008620957843959332,
+            0.028362801298499107,
+            0.00667025800794363,
+            -0.0323626808822155,
+            -0.0378473699092865,
+            -0.01513591781258583,
+            0.015257125720381737,
+            0.006692984607070684,
+            -0.011840560473501682,
+            -0.02278720773756504,
+            -0.038817036896944046,
+            0.015787413343787193,
+            -0.012916285544633865,
+            -0.038271598517894745,
+            -0.010158791206777096,
+            -0.009742137044668198,
+            0.016635872423648834,
+            -0.006916462909430265,
+            -0.016802534461021423,
+            -0.016090434044599533,
+            0.005821797996759415,
+            7.161247049225494e-05,
+            0.0244083721190691,
+            -0.001251857029274106,
+            -0.006855858489871025,
+            -0.016393456608057022,
+            -0.01188601367175579,
+            -0.0035074723418802023,
+            0.03602924197912216,
+            -0.014219277538359165,
+            -0.02383263036608696,
+            0.014984406530857086,
+            0.02657497487962246,
+            0.011636020615696907,
+            0.03227177634835243,
+            0.007181606721132994,
+            -0.014022313989698887,
+            0.003016956616193056,
+            -0.033029329031705856,
+            -0.0046400148421525955,
+            0.005196816753596067,
+            -0.006427840795367956,
+            -0.015651054680347443,
+            -0.01815097965300083,
+            -0.015832867473363876,
+            0.030756669119000435,
+            -0.02487805485725403,
+            0.0218023881316185,
+            0.030347591266036034,
+            0.003174148965626955,
+            -0.02915065549314022,
+            0.013598084449768066,
+            -0.0072005451656877995,
+            0.01862066239118576,
+            0.0019014590652659535,
+            0.028014326468110085,
+            -0.007461901288479567,
+            -0.004818039946258068,
+            -0.02501441352069378,
+            -0.03696860745549202,
+            -0.023484155535697937,
+            -0.008817922323942184,
+            0.0027537066489458084,
+            0.020984230563044548,
+            -0.0375746488571167,
+            -0.017757052555680275,
+            0.009855770505964756,
+            0.015757111832499504,
+            -0.03027183562517166,
+            -0.0025908327661454678,
+            -0.010158791206777096,
+            -0.0004055278259329498,
+            0.005140000022947788,
+            -0.02227207086980343,
+            -0.008977008052170277,
+            -0.015726808458566666,
+            -0.0010435298318043351,
+            0.006166485138237476,
+            -0.0007409819518215954,
+            -0.05805889517068863,
+            -0.013969285413622856,
+            0.018635815009474754,
+            0.0026400736533105373,
+            0.008908828720450401,
+            0.015938924625515938,
+            -0.0111739132553339,
+            0.0031533162109553814,
+            -0.026756787672638893,
+            0.023893235251307487,
+            -0.009492144919931889,
+            0.009355784393846989,
+            -0.013742019422352314,
+            0.012514783069491386,
+            0.014014738611876965,
+            0.004795313347131014,
+            0.0010681503918021917,
+            -0.0054051438346505165,
+            0.01601468026638031,
+            0.028377952054142952,
+            -0.023756874725222588,
+            0.011454207822680473,
+            -0.0002514130319468677,
+            -0.0032158142421394587,
+            -0.029196109622716904,
+            0.00509454682469368,
+            0.003931702114641666,
+            -0.005109698045998812,
+            0.016651025041937828,
+            -0.036544378846883774,
+            0.01662072166800499,
+            0.02177208475768566,
+            0.032544493675231934,
+            0.014832896180450916,
+            -0.017287369817495346,
+            -0.011658747680485249,
+            -0.012385998852550983,
+            -0.002624922664836049,
+            0.022181164473295212,
+            0.012120855040848255,
+            -0.01815097965300083,
+            -0.024287162348628044,
+            0.011249667964875698,
+            0.0005283934879116714,
+            -0.01203752402216196,
+            0.010795135982334614,
+            -0.013166278600692749,
+            0.01349202636629343,
+            0.012711746618151665,
+            0.00866641104221344,
+            0.00888610165566206,
+            0.022862963378429413,
+            -0.011878437362611294,
+            -0.017711598426103592,
+            0.00656041270121932,
+            -0.008916404098272324,
+            -0.0034089903347194195,
+            -0.012598113156855106,
+            0.010795135982334614,
+            0.014878349378705025,
+            0.06490717828273773,
+            -0.013143551535904408,
+            0.0009526234352961183,
+            -0.020438792183995247,
+            -0.031938452273607254,
+            -0.0017944546416401863,
+            -0.020908474922180176,
+            0.0111739132553339,
+            0.008370965719223022,
+            0.01598437689244747,
+            0.012022373266518116,
+            -0.0021912232041358948,
+            0.00795431062579155,
+            0.019317612051963806,
+            -0.010423935018479824,
+            7.255941454786807e-05,
+            -0.006503595970571041,
+            0.0013683309080079198,
+            0.0035851215943694115,
+            -0.011719351634383202,
+            -0.0026949963066726923,
+            0.035695917904376984,
+            0.01068150345236063,
+            0.011651171371340752,
+            -0.03445352986454964,
+            0.0020435003098100424,
+            0.020150920376181602,
+            0.007602048572152853,
+            -0.008302785456180573,
+            -0.01563590206205845,
+            0.01706010289490223,
+            -0.015317730605602264,
+            -0.0022499337792396545,
+            0.002888172399252653,
+            -0.010151215828955173,
+            0.04581683129072189,
+            -0.019787294790148735,
+            -0.0011742077767848969,
+            0.031847547739744186,
+            0.0030207443051040173,
+            -0.0040983641520142555,
+            0.005408931523561478,
+            -0.003473382443189621,
+            -0.0069013116881251335,
+            -0.02139330841600895,
+            0.0031381649896502495,
+            0.00667025800794363,
+            -0.019287310540676117,
+            -0.0072005451656877995,
+            -0.007719469256699085,
+            0.006897523999214172,
+            0.023105379194021225,
+            -0.009492144919931889,
+            -0.00016145357221830636,
+            -0.008575504645705223,
+            0.009848194196820259,
+            -0.0011912527261301875,
+            -0.044301725924015045,
+            -0.006147546228021383,
+            0.021181194111704826,
+            0.000973456131760031,
+            0.0005757405888289213,
+            -0.007450537756085396,
+            0.005007428117096424,
+            -0.017423728480935097,
+            -0.008454295806586742,
+            0.02369627170264721,
+            0.01736312545835972,
+            -0.009113367646932602,
+            0.010863316245377064,
+            -0.0035680767614394426,
+            0.00433320552110672,
+            -0.020014561712741852,
+            -0.01095422264188528,
+            0.003462019143626094,
+            0.015272277407348156,
+            -0.0009819786064326763,
+            -0.029029447585344315,
+            -0.0012272365856915712,
+            -0.00024194362049456686,
+            0.02918095886707306,
+            0.01486319862306118,
+            -0.008961857296526432,
+            -0.021211495622992516,
+            -0.013242034241557121,
+            -0.001237652963027358,
+            -0.015726808458566666,
+            -0.009620928205549717,
+            0.011855711229145527,
+            0.0009137988090515137,
+            0.008992158807814121,
+            -0.018135828897356987,
+            -0.027347678318619728,
+            -0.016166189685463905,
+            0.0018465365283191204,
+            -0.012870832346379757,
+            0.003340810537338257,
+            0.015514694154262543,
+            -0.03430201858282089,
+            -0.015060162171721458,
+            -0.03805948421359062,
+            0.008280059322714806,
+            -0.0029658216517418623,
+            -0.02024182677268982,
+            -0.023605365306138992,
+            0.009484568610787392,
+            0.019181253388524055,
+            -0.0015548784285783768,
+            0.019847899675369263,
+            -0.000889651826582849,
+            0.020060013979673386,
+            -0.0073141781613230705,
+            0.0031703610438853502,
+            -0.013007191941142082,
+            0.0030245319940149784,
+            0.008333087898790836,
+            0.02890823967754841,
+            -0.023075077682733536,
+            -0.006503595970571041,
+            -0.004685468040406704,
+            0.004746072459965944,
+            0.012658718042075634,
+            0.014613205567002296,
+            0.020741812884807587,
+            -0.008272483013570309,
+            -0.0027612822595983744,
+            0.012484480626881123,
+            -0.003799130441620946,
+            0.007295239716768265,
+            0.002390081062912941,
+            0.0019677451346069574,
+            0.004162756260484457,
+            0.008477022871375084,
+            0.0053710537031292915,
+            0.027650700882077217,
+            -0.0028275682125240564,
+            0.0031703610438853502,
+            0.024620486423373222,
+            0.02322658896446228,
+            -0.018878230825066566,
+            -0.028074929490685463,
+            0.002024561632424593,
+            0.050695475190877914,
+            0.007245998363941908,
+            -0.006810405291616917,
+            -0.03927157074213028,
+            -0.0031760428100824356,
+            0.028635520488023758,
+            0.023650817573070526,
+            0.01584801822900772,
+            -0.010567869991064072,
+            -0.012098127976059914,
+            0.02772645466029644,
+            0.013295062817633152,
+            0.02616589516401291,
+            -0.025302285328507423,
+            0.010704229585826397,
+            -0.03869583085179329,
+            0.0449683703482151,
+            0.004439263138920069,
+            -0.014476845972239971,
+            0.0011884119594469666,
+            0.010386057198047638,
+            -0.004537744913250208,
+            0.006431628484278917,
+            -5.8059369621332735e-05,
+            -0.0008086882880888879,
+            0.011923890560865402,
+            0.012007221579551697,
+            0.004265025723725557,
+            -0.02399929240345955,
+            0.007359631359577179,
+            0.004272601567208767,
+            0.03005971945822239,
+            0.005431658122688532,
+            -0.0013863227795809507,
+            0.012385998852550983,
+            -0.005810434930026531,
+            0.008855799213051796,
+            -0.010719381272792816,
+            0.020469093695282936,
+            -0.02918095886707306,
+            1.0986003871948924e-05,
+            0.02051454596221447,
+            -0.0015785519499331713,
+            0.0038199631962925196,
+            -0.023741723969578743,
+            0.01186328660696745,
+            -0.052968136966228485,
+            0.021166043356060982,
+            -0.018741872161626816,
+            0.021741783246397972,
+            0.019590331241488457,
+            -0.009734561666846275,
+            -0.012052674777805805,
+            0.03545350208878517,
+            0.0026760573964565992,
+            -0.014264730736613274,
+            -0.010643625631928444,
+            -0.017999470233917236,
+            -0.02830219641327858,
+            0.041968461126089096,
+            -0.0006117243901826441,
+            0.0028275682125240564,
+            0.0024222771171480417,
+            -0.010666352696716785,
+            -0.008923979476094246,
+            0.001954487757757306,
+            0.020499395206570625,
+            -0.03369597718119621,
+            -0.011128460057079792,
+            -0.008689138107001781,
+            -0.02566591091454029,
+            -0.00020300064352340996,
+            -0.011961768381297588,
+            0.026211349293589592,
+            0.01240872498601675,
+            -0.03696860745549202,
+            0.003780191531404853,
+            0.023650817573070526,
+            0.001603172393515706,
+            0.018120678141713142,
+            0.002130619017407298,
+            0.0010795136913657188,
+            0.009742137044668198,
+            -0.010476963594555855,
+            0.026559824123978615,
+            -0.009204274043440819,
+            0.008658835664391518,
+            -0.001950700068846345,
+            -0.017287369817495346,
+            0.017969166859984398,
+            0.004083212930709124,
+            0.024120500311255455,
+            0.011954193003475666,
+            0.011605718173086643,
+            0.01112088467925787,
+            -0.019878201186656952,
+            0.020090317353606224,
+            -0.0018891488434746861,
+            0.009075489826500416,
+            0.024726543575525284,
+            -0.005863463506102562,
+            0.020832719281315804,
+            0.015257125720381737,
+            -0.00020229043730068952,
+            0.011295121163129807,
+            -0.020120618864893913,
+            -0.011779955588281155,
+            0.0085679292678833,
+            -0.009605777449905872,
+            -0.010325453244149685,
+            0.009901223704218864,
+            -0.010105762630701065,
+            -0.015052586793899536,
+            -0.0071247899904847145,
+            0.016635872423648834,
+            0.0028976418543606997,
+            -0.02610529027879238,
+            -1.7326075976598077e-05,
+            0.008901252411305904,
+            0.006166485138237476,
+            0.018741872161626816,
+            0.011317848227918148,
+            -0.006692984607070684,
+            0.010363331064581871,
+            0.011476934887468815,
+            0.0053104497492313385,
+            -0.005079395603388548,
+            -0.013431422412395477,
+            -0.0024241709616035223,
+            0.018090376630425453,
+            0.009810317307710648,
+            -0.01922670565545559,
+            0.004564259201288223,
+            0.0008877579239197075,
+            -0.01420412678271532,
+            -0.019347915425896645,
+            -0.0038483713287860155,
+            -0.004170331638306379,
+            0.0027915844693779945,
+            0.0509984977543354,
+            -0.0006216673064045608,
+            0.007893706671893597,
+            -0.025741664692759514,
+            -0.006299056578427553,
+            -0.00027556007262319326,
+            -0.0036097420379519463,
+            0.01272689737379551,
+            -0.01824188604950905,
+            -0.03724132478237152,
+            0.012711746618151665,
+            -0.0012584856012836099,
+            0.014923802576959133,
+            -0.00297907879576087,
+            0.014226853847503662,
+            -0.017347972840070724,
+            0.01916610263288021,
+            -0.0011117096291854978,
+            -0.016029831022024155,
+            0.0029374135192483664,
+            -0.011143610812723637,
+            -0.021317552775144577,
+            0.005916492082178593,
+            -0.006848283112049103,
+            -0.03896854817867279,
+            0.0029885482508689165,
+            -0.010423935018479824,
+            0.0011798894265666604,
+            -0.003912763670086861,
+            0.020166072994470596,
+            -0.0061513339169323444,
+            0.0008697660523466766,
+            0.02177208475768566,
+            0.0017026013229042292,
+            0.01601468026638031,
+            -0.014499573037028313,
+            0.010999675840139389,
+            0.012598113156855106,
+            0.02210540883243084,
+            -0.032180868089199066,
+            -0.01393140759319067,
+            0.014211702160537243,
+            0.0026116655208170414,
+            -0.005382417235523462,
+            0.020635755732655525,
+            -0.01294658798724413,
+            0.019650936126708984,
+            -0.0005577487172558904,
+            -0.011704200878739357,
+            -0.03524138405919075,
+            -0.016529815271496773,
+            -3.343888238305226e-05,
+            -0.008439145050942898,
+            0.025317436084151268,
+            0.01760554127395153,
+            -0.0108330138027668,
+            0.026468917727470398,
+            0.0037574649322777987,
+            0.014598054811358452,
+            -0.014257155358791351,
+            -0.005999823100864887,
+            -0.0003773563075810671,
+            -0.0119314668700099,
+            0.015264702029526234,
+            -0.03512017801403999,
+            0.0008342557121068239,
+            -0.00839369185268879,
+            0.005242269951850176,
+            -0.0254992488771677,
+            0.018257036805152893,
+            0.025817420333623886,
+            -0.0031419529113918543,
+            -0.005037730094045401,
+            -0.022802358493208885,
+            0.028968842700123787,
+            0.03049910068511963,
+            0.0024828813038766384,
+            -0.0013825349742546678,
+            -0.00352072948589921,
+            0.0033218718599528074,
+            0.0211205892264843,
+            -0.037998881191015244,
+            -0.009098216891288757,
+            -0.028696123510599136,
+            0.042332086712121964,
+            -0.008098246529698372,
+            -0.013431422412395477,
+            0.01862066239118576,
+            -0.0024715180043131113,
+            0.02355991117656231,
+            -0.004689255729317665,
+            -0.004855917301028967,
+            0.03148392215371132,
+            0.011514811776578426,
+            0.02772645466029644,
+            -0.04015033319592476,
+            0.014704111963510513,
+            -0.006867222022265196,
+            0.004867280833423138,
+            -0.013484450988471508,
+            0.005302873905748129,
+            0.02580226957798004,
+            -0.018060073256492615,
+            0.0068066176027059555,
+            0.030665762722492218,
+            0.00748084019869566,
+            0.002448791405186057,
+            0.0165904201567173,
+            0.003890036838129163,
+            0.010855740867555141,
+            0.000841831264551729,
+            0.0052574207074940205,
+            0.022423582151532173,
+            0.014454119838774204,
+            -0.016029831022024155,
+            -0.015120766125619411,
+            -0.00044790346873924136,
+            -0.004450626205652952,
+            0.005261208396404982,
+            0.013242034241557121,
+            0.0061172437854111195,
+            -0.015681356191635132,
+            0.01052999310195446,
+            0.014158673584461212,
+            0.013128400780260563,
+            -0.0039847311563789845,
+            -0.03412020578980446,
+            -0.02443867363035679,
+            -0.021438762545585632,
+            0.003104075090959668,
+            -0.0008281006012111902,
+            -0.0021211495622992516,
+            0.0017954016802832484,
+            0.017347972840070724,
+            -0.0008352026343345642,
+            0.00379155483096838,
+            0.01686313934624195,
+            -0.01415109820663929,
+            -0.022090258076786995,
+            -0.0035964848939329386,
+            -0.027529491111636162,
+            0.0053710537031292915,
+            0.001844642567448318,
+            -0.00965123064815998,
+            0.008530051447451115,
+            -0.0046059247106313705,
+            -0.007223271764814854,
+            -0.012196609750390053,
+            -0.02007516659796238,
+            0.005780132487416267,
+            0.016878290101885796,
+            0.005730891600251198,
+            -0.009272454306483269,
+            -0.003939277958124876,
+            0.011575416661798954,
+            0.017484333366155624,
+            0.017317671328783035,
+            -0.009295180439949036,
+            0.009628504514694214,
+            0.0004022135108243674,
+            0.004670316819101572,
+            -0.01834794320166111,
+            0.0045869858004152775,
+            -0.0027953721582889557,
+            -0.008658835664391518,
+            0.01105270441621542,
+            -0.00031201730598695576,
+            0.01628739945590496,
+            -0.013090522959828377,
+            0.005624833982437849,
+            0.010333028621971607,
+            -0.020590301603078842,
+            -0.0067308624275028706,
+            -0.01692374423146248,
+            0.026484068483114243,
+            -0.00504530593752861,
+            0.00045405857963487506,
+            -0.02978700026869774,
+            0.0033616432920098305,
+            0.000848933297675103,
+            -0.000845619011670351,
+            -0.012764775194227695,
+            0.010749682784080505,
+            0.03242328763008118,
+            0.0005179771687835455,
+            0.0216660276055336,
+            -0.0036154238041490316,
+            0.0007480839849449694,
+            -0.00785582885146141,
+            0.004977126140147448,
+            -0.004916521720588207,
+            -0.009901223704218864,
+            -0.010863316245377064,
+            0.021726632490754128,
+            -0.0033029329497367144,
+            0.0003356908564455807,
+            0.011332998983561993,
+            -0.012067826464772224,
+            0.0014620780711993575,
+            -0.0047687990590929985,
+            0.00971941091120243,
+            0.0061702728271484375,
+            0.009113367646932602,
+            -0.01176480483263731,
+            0.05451354384422302,
+            -0.0106284748762846,
+            0.005537715740501881,
+            -0.00439380994066596,
+            -0.022226616740226746,
+            -0.017317671328783035,
+            0.012711746618151665,
+            -0.017544938251376152,
+            0.03375658020377159,
+            0.024150803685188293,
+            0.0006467612111009657,
+            0.010393633507192135,
+            -0.006958128418773413,
+            0.023923536762595177,
+            -0.002602196065708995,
+            0.0010795136913657188,
+            -0.004855917301028967,
+            0.015802564099431038,
+            0.04511988162994385,
+            0.017393426969647408,
+            -0.001152428099885583,
+            0.00829521007835865,
+            0.020575150847434998,
+            0.00814369972795248,
+            -0.0020075165666639805,
+            0.014219277538359165,
+            -0.01024212222546339,
+            0.00900731049478054,
+            -0.004458202049136162,
+            0.027438584715127945,
+            0.00814369972795248,
+            -0.006310420110821724,
+            -0.0036836036015301943,
+            -0.0008209985098801553,
+            0.014173824340105057,
+            0.006950552575290203,
+            0.01963578537106514,
+            -0.00461350055411458,
+            -0.0012575386790558696,
+            0.013870803639292717,
+            0.014491996727883816,
+            -0.00552635220810771,
+            0.001724380999803543,
+            -0.01435563713312149,
+            -0.010204244405031204,
+            0.013575357384979725,
+            0.006086941808462143,
+            0.006310420110821724,
+            -0.014514723792672157,
+            -0.00980274099856615,
+            -0.004670316819101572,
+            0.0015435151290148497,
+            0.015052586793899536,
+            0.011060279794037342,
+            -0.004814252257347107,
+            0.006052851676940918,
+            0.013825350441038609,
+            -0.014014738611876965,
+            0.006147546228021383,
+            0.007208121009171009,
+            0.008855799213051796,
+            0.004787737503647804,
+            -0.010817863047122955,
+            -0.00428396463394165,
+            -0.0063937511295080185,
+            0.0012670081341639161,
+            0.016832837834954262,
+            0.013181429356336594,
+            0.0049581872299313545,
+            0.027529491111636162,
+            -0.00839369185268879,
+            -0.012302667833864689,
+            0.02339324913918972,
+            0.011257244274020195,
+            0.0031703610438853502,
+            0.014651083387434483,
+            -0.012635990977287292,
+            -0.012446602806448936,
+            0.006655106786638498,
+            0.004848341923207045,
+            0.006049063988029957,
+            0.009151245467364788,
+            0.00450365524739027,
+            0.007333117071539164,
+            -0.007098275702446699,
+            -0.007427811156958342,
+            0.0014810169814154506,
+            0.01642375811934471,
+            0.020393338054418564,
+            -0.024968961253762245,
+            -0.0018806264270097017,
+            0.0009052763343788683,
+            0.010878467001020908,
+            -0.003316190093755722,
+            0.012310243211686611,
+            5.65205882594455e-05,
+            -0.0074240234680473804,
+            0.010158791206777096,
+            -0.02119634486734867,
+            -0.012681444175541401,
+            0.0012717428617179394,
+            0.013037494383752346,
+            -0.019590331241488457,
+            0.018226735293865204,
+            -0.008893677033483982,
+            -0.013348091393709183,
+            0.013893529772758484,
+            0.020014561712741852,
+            0.004090788308531046,
+            -0.007738408166915178,
+            -0.021862991154193878,
+            -0.01127239502966404,
+            0.016060132533311844,
+            0.020272130146622658,
+            0.000947888707742095,
+            -0.0006472347304224968,
+            0.010173942893743515,
+            -0.013113250024616718,
+            -0.023605365306138992,
+            -0.005230906419456005,
+            0.010386057198047638,
+            0.01746918261051178,
+            0.018832778558135033,
+            -0.006685409229248762,
+            0.02443867363035679,
+            -0.0059127043932676315,
+            -0.015499543398618698,
+            2.86597842205083e-05,
+            -0.02802947722375393,
+            -0.017423728480935097,
+            0.00013778003631159663,
+            0.00246773031540215,
+            0.008598231710493565,
+            0.0008555619278922677,
+            0.01943882182240486,
+            0.000904329412151128,
+            0.017590390518307686,
+            -0.000659071491099894,
+            -0.002009410411119461,
+            -0.0013995799235999584,
+            -0.0015340456739068031,
+            0.02254479005932808,
+            -0.012166308239102364,
+            -0.01176480483263731,
+            0.022302372381091118,
+            0.006852070800960064,
+            0.014719262719154358,
+            -0.027635548263788223,
+            0.013757170177996159,
+            -0.015499543398618698,
+            0.017423728480935097,
+            -0.03408990427851677,
+            0.006420265417546034,
+            0.01489350013434887,
+            -0.010348179377615452,
+            -0.020499395206570625,
+            -0.029650641605257988,
+            -0.024150803685188293,
+            -0.0038710979279130697,
+            -0.010855740867555141,
+            -0.016847988590598106,
+            0.006121031939983368,
+            -0.005802859086543322,
+            0.018499454483389854,
+            -0.007079336792230606,
+            0.033877789974212646,
+            0.006969491485506296,
+            -0.004056698642671108,
+            0.003835114184767008,
+            0.022469034418463707,
+            -0.006102093029767275,
+            0.02109028771519661,
+            0.004408961161971092,
+            -0.001467759720981121,
+            0.013681414537131786,
+            -0.006552837323397398,
+            -0.002361672930419445,
+            0.005821797996759415,
+            -0.010484539903700352,
+            0.022014502435922623,
+            0.005924067925661802,
+            -0.007708106189966202,
+            -0.0006775368237867951,
+            -0.011454207822680473,
+            0.014726839028298855,
+            0.006530110724270344,
+            0.0022480397019535303,
+            -0.011295121163129807,
+            0.011249667964875698,
+            -0.004859705455601215,
+            0.0040642740204930305,
+            0.009484568610787392,
+            -0.01774190180003643,
+            -0.028862785547971725,
+            0.011416330002248287,
+            0.019120648503303528,
+            0.006382387597113848,
+            -0.0055755930952727795,
+            -0.0025681061670184135,
+            0.00839369185268879,
+            0.010651201009750366,
+            0.022150862962007523,
+            -0.012984465807676315,
+            -8.889415767043829e-05,
+            0.013204156421124935,
+            -0.007170243188738823,
+            -0.01056029461324215,
+            0.00487485621124506,
+            0.023438703268766403,
+            0.0068823727779090405,
+            -0.04327145218849182,
+            -0.018469152972102165,
+            0.01909034699201584,
+            0.007136153522878885,
+            -0.04154422879219055,
+            0.0026381798088550568,
+            -0.003689285134896636,
+            -0.023787178099155426,
+            0.016105586662888527,
+            -0.0011817833874374628,
+            -0.01828734017908573,
+            0.028756728395819664,
+            0.02830219641327858,
+            0.01834794320166111,
+            2.6987840101355687e-05,
+            -0.013332940638065338,
+            -0.0318172425031662,
+            -0.023590214550495148,
+            0.008280059322714806,
+            -0.02660527639091015,
+            -0.0072725131176412106,
+            0.011242092587053776,
+            -0.017454031854867935,
+            0.02972639724612236,
+            -0.008507325313985348,
+            0.026938600465655327,
+            0.011446632444858551,
+            0.001691238023340702,
+            -0.013446573168039322,
+            -0.018802475184202194,
+            -0.01350717805325985,
+            -0.008196728304028511,
+            0.014090493321418762,
+            0.017923714593052864,
+            0.011007251217961311,
+            0.011332998983561993,
+            0.02251448854804039,
+            -0.012158732861280441,
+            0.007969462312757969,
+            -2.888172457460314e-05,
+            -0.008961857296526432,
+            0.0026343921199440956,
+            -0.018393397331237793,
+            0.00868156272917986,
+            0.0062308767810463905,
+            0.005458172410726547,
+            -0.007208121009171009,
+            0.010742107406258583,
+            0.02712041325867176,
+            -0.018847929313778877,
+            0.011401179246604443,
+            -0.031362712383270264,
+            -0.014787442982196808,
+            -0.01056029461324215,
+            0.026862844824790955,
+            0.007757347077131271,
+            -0.015726808458566666,
+            -0.007514929864555597,
+            -0.01545409020036459,
+            0.012885984033346176,
+            0.0032442226074635983,
+            0.005764981731772423,
+            0.015817714855074883,
+            0.021893294528126717,
+            0.01746918261051178,
+            -0.00795431062579155,
+            0.006852070800960064,
+            0.004674104508012533,
+            0.028090080246329308,
+            -0.01777220331132412,
+            0.020802417770028114,
+            0.009954252280294895,
+            -0.003935490269213915,
+            0.005685438401997089,
+            -0.014257155358791351,
+            0.008530051447451115,
+            -0.007325541693717241,
+            0.030514251440763474,
+            0.0033067206386476755,
+            0.014075342565774918,
+            -0.02062060497701168,
+            0.006575563922524452,
+            -0.03172633796930313,
+            0.009022461250424385,
+            0.013704141601920128,
+            0.012976890429854393,
+            -0.007677804213017225,
+            -0.024181105196475983,
+            -0.0119314668700099,
+            0.0057990713976323605,
+            -0.01279507763683796,
+            -0.01046181283891201,
+            -0.0034828518982976675,
+            0.012446602806448936,
+            0.018847929313778877,
+            -0.01848430372774601,
+            -0.019923655316233635,
+            -0.009583051316440105,
+            0.02898399345576763,
+            -0.015787413343787193,
+            -0.001471547526307404,
+            0.02413565292954445,
+            0.021241798996925354,
+            0.015022284351289272,
+            0.002155239460989833,
+            0.020090317353606224,
+            -0.0007111532613635063,
+            -0.0237265732139349,
+            -0.02813553437590599,
+            0.007302815094590187,
+            -0.006912675220519304,
+            0.01943882182240486,
+            -0.0035548193845897913,
+            -0.005658924113959074,
+            -0.0034033088013529778,
+            -0.007503566797822714,
+            -0.0165904201567173,
+            -0.01787826046347618,
+            0.0021647089160978794,
+            -0.008454295806586742,
+            0.005863463506102562,
+            -0.0002795845502987504,
+            -0.013105674646794796,
+            0.013401119969785213,
+            -0.014075342565774918,
+            0.005772557109594345,
+            -0.002653330797329545,
+            0.001962063368409872,
+            0.0035926972050219774,
+            0.009113367646932602,
+            -0.007336904760450125,
+            -0.06381630152463913,
+            -0.013105674646794796,
+            -0.003704436356201768,
+            0.006071790587157011,
+            -0.00019578021601773798,
+            -0.016635872423648834,
+            -0.013196581043303013,
+            0.01706010289490223,
+            0.008863375522196293,
+            -0.013870803639292717,
+            0.016954045742750168,
+            0.009961827658116817,
+            0.003622999181970954,
+            -0.024044746533036232,
+            -0.0008148433989845216,
+            0.008757317438721657,
+            0.032756611704826355,
+            -0.008643684908747673,
+            -0.0003839849086944014,
+            0.0036684523802250624,
+            -0.0033332351595163345,
+            -0.010052734054625034,
+            -0.014582903124392033,
+            -0.011848135851323605,
+            0.011908739805221558,
+            0.01790856383740902,
+            -0.014264730736613274,
+            -0.02007516659796238,
+            -0.0007106798002496362,
+            0.012931437231600285,
+            -0.026590125635266304,
+            0.0007244104635901749,
+            -0.007787649054080248,
+            -0.01052999310195446,
+            0.018302490934729576,
+            -0.019014591351151466,
+            0.005249845329672098,
+            0.00048294031876139343,
+            0.008507325313985348,
+            -0.023544760420918465,
+            -0.024893205612897873,
+            0.0035131541080772877,
+            -0.023847781121730804,
+            0.03612014651298523,
+            -0.021590271964669228,
+            -0.012082977220416069,
+            0.034483831375837326,
+            -0.0037423139438033104,
+            0.01702980138361454,
+            0.0023995505180209875,
+            0.02739313244819641,
+            -0.011439057067036629,
+            -0.00617784820497036,
+            0.012499631382524967,
+            0.009575475007295609,
+            0.0032593735959380865,
+            0.03775646165013313,
+            -0.01257538702338934,
+            -0.013901105150580406,
+            -0.005556654185056686,
+            -0.012954163365066051,
+            0.0006775368237867951,
+            0.014431392773985863,
+            0.028393102809786797,
+            0.004257450345903635,
+            -0.00628769351169467,
+            -0.006143758539110422,
+            -0.0014649189542979002,
+            -0.004242299124598503,
+            -0.00947699323296547,
+            -0.00715130427852273,
+            0.017287369817495346,
+            -0.025241680443286896,
+            0.016711628064513206,
+            -0.006420265417546034,
+            0.021135739982128143,
+            -0.011757229454815388,
+            0.007007369305938482,
+            -0.008961857296526432,
+            0.003492321353405714,
+            0.0023635667748749256,
+            -0.005109698045998812,
+            0.009287605062127113,
+            -0.014772292226552963,
+            0.008454295806586742,
+            0.013742019422352314,
+            0.0011183382011950016,
+            -0.010590597055852413,
+            -0.019272159785032272,
+            0.0022972808219492435,
+            -0.000710206339135766,
+            0.01181783340871334,
+            -0.0047498601488769054,
+            -0.000981031684204936,
+            0.0002921315317507833,
+            -0.014476845972239971,
+            -0.009575475007295609,
+            0.0006050958181731403,
+            0.005302873905748129,
+            -0.008961857296526432,
+            0.010378481820225716,
+            -0.005693014245480299,
+            -0.00034160923678427935,
+            0.02627195231616497,
+            0.006526322569698095,
+            -0.007632351014763117,
+            -0.014098069630563259,
+            0.01563590206205845,
+            0.015370759181678295,
+            -0.007454325910657644,
+            0.01933276280760765,
+            -0.016196493059396744,
+            -0.0008366230758838356,
+            0.013916256837546825,
+            -0.009219424799084663,
+            0.0017698341980576515,
+            -0.008196728304028511,
+            0.03554440662264824,
+            -0.0008787619881331921,
+            -0.0035415622405707836,
+            -0.0006401326390914619,
+            0.012976890429854393,
+            -0.0026779514737427235,
+            0.001182730309665203,
+            0.01132542360574007,
+            0.005321812815964222,
+            0.0185146052390337,
+            0.00328020635060966,
+            -0.03378688171505928,
+            -0.007780073676258326,
+            -0.0065490491688251495,
+            -0.008090670220553875,
+            0.009022461250424385,
+            0.025196226313710213,
+            0.01990850456058979,
+            0.006408901885151863,
+            0.001881573349237442,
+            0.018423698842525482,
+            0.015757111832499504,
+            -0.01489350013434887,
+            0.016363153234124184,
+            0.02802947722375393,
+            -0.010030006989836693,
+            0.022741753607988358,
+            -0.005253633018583059,
+            -0.008893677033483982,
+            0.008946705609560013,
+            -0.0015378334792330861,
+            -0.017393426969647408,
+            -0.015681356191635132,
+            0.005022579338401556,
+            -0.008999735116958618,
+            -0.0022897052112966776,
+            -0.018605511635541916,
+            0.004670316819101572,
+            0.017105557024478912,
+            -0.004965762607753277,
+            -3.5243989259470254e-05,
+            0.019484274089336395,
+            -0.006965703796595335,
+            -0.016908591613173485,
+            0.006325571332126856,
+            -0.015007133595645428,
+            0.007946735247969627,
+            -0.013666263781487942,
+            0.010492115281522274,
+            -0.006098304875195026,
+            0.018817627802491188,
+            0.005446809343993664,
+            -0.008090670220553875,
+            -0.0008958069374784827,
+            -0.018984289839863777,
+            0.02065090648829937,
+            -0.033089932054281235,
+            0.00839369185268879,
+            -0.026226500049233437,
+            -0.0009781909175217152,
+            0.0008849170990288258,
+            0.014605630189180374,
+            -0.001218714052811265,
+            0.019696388393640518,
+            -0.002751812804490328,
+            -0.020090317353606224,
+            -0.007086912170052528,
+            -0.0010634156642481685,
+            -0.01618134044110775,
+            -0.030923331156373024,
+            -0.004272601567208767,
+            0.0036267871037125587,
+            -0.004711982328444719,
+            0.0021135739516466856,
+            0.017984317615628242,
+            -0.005613470915704966,
+            0.009173971600830555,
+            -0.02690829709172249,
+            -0.007416448090225458,
+            -0.00355103169567883,
+            0.0010776197304949164,
+            -0.011242092587053776,
+            0.0010596278589218855,
+            0.011226941831409931,
+            -0.019044892862439156,
+            0.013461724855005741,
+            -0.021378157660365105,
+            -0.004371083341538906,
+            -0.008507325313985348,
+            -0.00135412672534585,
+            -0.013446573168039322,
+            -0.028120383620262146,
+            0.025029566138982773,
+            -0.00021495578403118998,
+            -0.003528305096551776,
+            -0.011999646201729774,
+            0.023181134834885597,
+            0.002914686920121312,
+            0.027817361056804657,
+            -0.03157482668757439,
+            0.0013427634257823229,
+            0.011552689597010612,
+            0.0025984083767980337,
+            -0.017287369817495346,
+            0.004746072459965944,
+            -0.005234694108366966,
+            0.02210540883243084,
+            -0.0011694730492308736,
+            -0.0005340751959010959,
+            -0.008234606124460697,
+            0.030999086797237396,
+            0.009136094711720943,
+            0.0012944694608449936,
+            -0.013226882554590702,
+            -0.008060368709266186,
+            -0.002759388415142894,
+            -0.0011069749016314745,
+            0.012855681590735912,
+            0.01290113478899002,
+            -0.012105703353881836,
+            0.0013607552973553538,
+            -0.006878585088998079,
+            0.0011694730492308736,
+            -0.002935519441962242,
+            -0.0004966709529981017,
+            -0.01149208564311266,
+            0.003257479751482606,
+            0.016272246837615967,
+            -0.00780280027538538,
+            -0.006480869371443987,
+            0.0007750718505121768,
+            0.009181547909975052,
+            0.012461753562092781,
+            -0.012742049060761929,
+            -0.018226735293865204,
+            -0.011908739805221558,
+            0.016029831022024155,
+            0.010522416792809963,
+            -0.008204303681850433,
+            -0.013234458863735199,
+            -0.02583257108926773,
+            0.01169662456959486,
+            -0.03293842449784279,
+            -0.012242063879966736,
+            0.005821797996759415,
+            0.028559764847159386,
+            0.019999410957098007,
+            0.012226912193000317,
+            0.001332347048446536,
+            0.002950670663267374,
+            0.00023342114582192153,
+            -0.01916610263288021,
+            0.014408666640520096,
+            -0.011143610812723637,
+            -0.018166130408644676,
+            0.0075906855054199696,
+            0.011939042247831821,
+            0.03148392215371132,
+            0.006942977197468281,
+            -0.0010160685051232576,
+            0.01686313934624195,
+            -0.001218714052811265,
+            -0.013582932762801647,
+            0.01420412678271532,
+            -0.024256860837340355,
+            0.016438908874988556,
+            0.02860521711409092,
+            -0.004780162125825882,
+            0.0008531945641152561,
+            -0.020938776433467865,
+            0.00482561532407999,
+            -0.011529963463544846,
+            -0.002177966060116887,
+            -0.003287781961262226,
+            -0.03460504114627838,
+            0.007946735247969627,
+            -0.006825556512922049,
+            0.01046181283891201,
+            -0.00814369972795248,
+            0.006655106786638498,
+            -0.012431452050805092,
+            0.004253662656992674,
+            0.01513591781258583,
+            0.016105586662888527,
+            0.007060397882014513,
+            -0.025211378931999207,
+            -0.009196698665618896,
+            0.008105821907520294,
+            -0.01349202636629343,
+            -0.02295386977493763,
+            -0.006852070800960064,
+            0.009840618818998337,
+            -0.01061332318931818,
+            -0.00488243205472827,
+            -0.028180986642837524,
+            -0.013681414537131786,
+            -0.0014327229000627995,
+            -0.017817657440900803,
+            -0.005556654185056686,
+            -0.003100287402048707,
+            0.01571165770292282,
+            -0.0049733384512364864,
+            -0.006185423582792282,
+            0.004742284305393696,
+            -0.023878084495663643,
+            0.0037423139438033104,
+            -0.008280059322714806,
+            0.0008266801596619189,
+            -0.0032517979852855206,
+            -0.0014762822538614273,
+            -0.007969462312757969,
+            -0.014332910999655724,
+            0.006893736310303211,
+            -0.016469212248921394,
+            0.002954458352178335,
+            -0.015173795633018017,
+            -0.016499513760209084,
+            -0.00682934420183301,
+            0.00775355938822031,
+            0.021575121209025383,
+            0.006541473791003227,
+            0.021484214812517166,
+            0.005469535943120718,
+            0.028680972754955292,
+            0.0003953482082579285,
+            0.003041577059775591,
+            -8.958956641436089e-06,
+            0.003232859307900071,
+            0.019347915425896645,
+            0.006571775767952204,
+            0.018984289839863777,
+            -0.008742166683077812,
+            0.010514841414988041,
+            -0.010355755686759949,
+            0.010014856234192848,
+            0.006333146709948778,
+            -0.006473293993622065,
+            -0.003799130441620946,
+            -0.010348179377615452,
+            -0.0031419529113918543,
+            -0.016544966027140617,
+            0.0027366618160158396,
+            0.00693161366507411,
+            0.0071588801220059395,
+            -0.011582992039620876,
+            -0.005230906419456005,
+            -0.014704111963510513,
+            -0.0064354161731898785,
+            -0.01831764169037342,
+            0.014385939575731754,
+            0.029226411134004593,
+            -0.016332851722836494,
+            -0.01977214403450489,
+            -0.007772498298436403,
+            0.0007296186522580683,
+            -0.006636167876422405,
+            -0.01240872498601675,
+            -0.009151245467364788,
+            -0.035059571266174316,
+            -0.017666146159172058,
+            0.007586897816509008,
+            0.02484775334596634,
+            0.01387837901711464,
+            -0.013628385961055756,
+            -0.0006983695784583688,
+            -0.016984347254037857,
+            -0.010787560604512691,
+            -0.01535560842603445,
+            -0.02054484933614731,
+            -0.02498411200940609,
+            0.0077952248975634575,
+            -0.004033972043544054,
+            0.014931377954781055,
+            0.008113397285342216,
+            0.0032631612848490477,
+            -0.004371083341538906,
+            -0.0024109138175845146,
+            -0.013196581043303013,
+            0.0012925754999741912,
+            -0.004787737503647804,
+            -0.0017291157273575664,
+            0.03554440662264824,
+            -0.008098246529698372,
+            -0.005871038883924484,
+            0.011060279794037342,
+            0.011128460057079792,
+            -0.013764745555818081,
+            0.0006642796797677875,
+            -0.011249667964875698,
+            0.021847840398550034,
+            -0.021620575338602066,
+            0.019257009029388428,
+            0.0014507147716358304,
+            -0.010840589180588722,
+            0.028999146074056625,
+            0.01892368495464325,
+            0.00743159931153059,
+            0.006579351611435413,
+            0.01672677882015705,
+            0.0044316877610981464,
+            0.016196493059396744,
+            -0.0026552248746156693,
+            -0.004632438998669386,
+            0.009295180439949036,
+            0.015681356191635132,
+            -0.000702630786690861,
+            -0.006942977197468281,
+            0.016438908874988556,
+            0.004056698642671108,
+            0.0075414446182549,
+            -0.01356020662933588,
+            -0.008961857296526432,
+            0.001507531269453466,
+            0.006257391534745693,
+            -0.005939218681305647,
+            -0.0026003022212535143,
+            0.005530139897018671,
+            0.003943065647035837,
+            -0.013302638195455074,
+            0.01689344085752964,
+            -0.016166189685463905,
+            0.008742166683077812,
+            -0.0037972365971654654,
+            0.0121360057964921,
+            -0.006799042224884033,
+            0.0013153020991012454,
+            -0.01818128302693367,
+            -0.003869204083457589,
+            0.0038597346283495426,
+            0.012173883616924286,
+            -0.01262841559946537,
+            0.025484098121523857,
+            0.008166425861418247,
+            0.006602078210562468,
+            0.0059467945247888565,
+            -0.012855681590735912,
+            0.027211319655179977,
+            0.005969521123915911,
+            0.009886072017252445,
+            -0.008893677033483982,
+            0.011136035434901714,
+            0.035362593829631805,
+            0.03060515783727169,
+            0.014135947450995445,
+            0.004469565115869045,
+            -0.018999440595507622,
+            0.015120766125619411,
+            0.0018427487229928374,
+            0.0023806116078048944,
+            0.00863610953092575,
+            -0.011348150670528412,
+            0.01305264513939619,
+            -0.009484568610787392,
+            0.015241974964737892,
+            -0.021272100508213043,
+            -0.011749654076993465,
+            -0.014408666640520096,
+            -0.021075136959552765,
+            -0.01430260855704546,
+            0.034180812537670135,
+            -0.002346521709114313,
+            -0.010393633507192135,
+            0.0045112306252121925,
+            0.008477022871375084,
+            -0.006079366430640221,
+            0.0053142374381423,
+            -0.008204303681850433,
+            -0.013749594800174236,
+            -0.01889338344335556,
+            0.006628592498600483,
+            -0.01550711877644062,
+            0.007651289459317923,
+            -0.003304826794192195,
+            0.0037688282318413258,
+            0.007014944683760405,
+            0.018105527386069298,
+            0.00530666159465909,
+            -0.00135412672534585,
+            0.0014033677289262414,
+            0.0016865032957866788,
+            -0.00931033119559288,
+            0.0108330138027668,
+            0.015408637002110481,
+            -0.0011921997647732496,
+            -0.011279970407485962,
+            -0.009136094711720943,
+            -0.042877525091171265,
+            0.0006264019757509232,
+            -0.013128400780260563,
+            0.0064354161731898785,
+            -0.011196639388799667,
+            0.0022234192583709955,
+            0.015863168984651566,
+            -0.0054051438346505165,
+            -0.0010918239131569862,
+            0.02193874679505825,
+            0.013014768250286579,
+            0.021075136959552765,
+            0.02065090648829937,
+            0.014188976027071476,
+            -0.015302578918635845,
+            -0.03469594568014145,
+            0.002221525413915515,
+            -0.010030006989836693,
+            0.013772320933640003,
+            0.004238511435687542,
+            0.016060132533311844,
+            -0.023893235251307487,
+            -0.011878437362611294,
+            -0.006662682630121708,
+            0.014431392773985863,
+            0.01491622719913721,
+            -0.0032271775417029858,
+            0.02690829709172249,
+            0.009280029684305191,
+            0.0020510759204626083,
+            -0.02281750924885273,
+            0.003312402404844761,
+            0.005321812815964222,
+            -0.008370965719223022,
+            -0.016681326553225517,
+            -0.0004552422615233809,
+            -0.0012168202083557844,
+            -0.0007817004225216806,
+            -0.021650876849889755,
+            0.018832778558135033,
+            -0.004018820822238922,
+            0.0037574649322777987,
+            0.013840501196682453,
+            0.01516621932387352,
+            0.017984317615628242,
+            0.008370965719223022,
+            0.001497115008533001,
+            -0.006605865899473429,
+            -0.0007973249885253608,
+            -0.0005828426801599562,
+            -0.007389933802187443,
+            -0.005598319694399834,
+            0.006382387597113848,
+            -0.01134057529270649,
+            0.008810346014797688,
+            0.000896753859706223,
+            0.006113456096500158,
+            0.010605747811496258,
+            -0.004836978856474161,
+            -0.011628445237874985,
+            0.014628356322646141,
+            -0.0010548931313678622,
+            -0.0055755930952727795,
+            -0.011060279794037342,
+            0.003624893259257078,
+            0.013575357384979725,
+            -0.003096499713137746,
+            -0.013499601744115353,
+            0.00910579226911068,
+            0.002009410411119461,
+            -0.027438584715127945,
+            0.015499543398618698,
+            0.010295150801539421,
+            -0.013370818458497524,
+            0.004980913829058409,
+            -0.003371112747117877,
+            -0.03027183562517166,
+            0.004791525658220053,
+            0.002660906407982111,
+            -0.010643625631928444,
+            0.028938541188836098,
+            -0.003821857040748,
+            0.029256712645292282,
+            -0.004980913829058409,
+            -0.006905099377036095,
+            -0.011355726048350334,
+            0.009416389279067516,
+            -0.018681267276406288,
+            -0.0037953427527099848,
+            -0.008727015927433968,
+            -0.00915882084518671,
+            -0.0009838725673034787,
+            -0.005185453221201897,
+            0.006984642706811428,
+            0.005121061112731695,
+            -0.012704171240329742,
+            0.006757376715540886,
+            -0.010257272981107235,
+            -0.00020856392802670598,
+            0.0006718551740050316,
+            -0.008408842608332634,
+            0.013885954394936562,
+            0.005996035411953926,
+            -0.008340663276612759,
+            -0.0005032995832152665,
+            0.027817361056804657,
+            0.05051366239786148,
+            -0.018863080069422722,
+            0.0009270560112781823,
+            0.01452229917049408,
+            -0.0028824908658862114,
+            -0.0021590273827314377,
+            -0.011976920068264008,
+            0.005821797996759415,
+            -0.019423669204115868,
+            0.013317788951098919,
+            0.006473293993622065,
+            -0.008833073079586029,
+            0.003202557098120451,
+            -0.016711628064513206,
+            0.002621134975925088,
+            -0.0018058179412037134,
+            -0.018681267276406288,
+            -0.015529844909906387,
+            -0.012060250155627728,
+            -0.024181105196475983,
+            0.0004235197266098112,
+            -0.0008546149474568665,
+            -0.013870803639292717,
+            0.006579351611435413,
+            0.008431569673120975,
+            0.032483890652656555,
+            0.0011912527261301875,
+            0.010128489695489407,
+            -2.1513333194889128e-05,
+            -0.0010416359873488545,
+            -0.005988460034132004,
+            -0.01523439958691597,
+            0.00015387803432531655,
+            -0.002342734020203352,
+            0.022590242326259613,
+            0.005549078807234764,
+            0.017560089007019997,
+            0.0017887729918584228,
+            -0.016514664515852928,
+            -0.00596573343500495,
+            -0.007408872712403536,
+            -0.018060073256492615,
+            0.0062649669125676155,
+            -0.005473323632031679,
+            0.009605777449905872,
+            -0.015863168984651566,
+            -0.02731737680733204,
+            0.00792400911450386,
+            0.012234487570822239,
+            0.028514310717582703,
+            -0.0035623949952423573,
+            0.001716805505566299,
+            -0.005533927585929632,
+            0.016408607363700867,
+            0.032180868089199066,
+            -0.0008550884085707366,
+            0.006950552575290203,
+            -0.019999410957098007,
+            0.0017404790269210935,
+            0.013704141601920128,
+            -0.008401267230510712,
+            -0.01821158453822136,
+            0.003939277958124876,
+            -0.020635755732655525,
+            0.004583198111504316,
+            -0.020393338054418564,
+            0.030408194288611412,
+            -0.0023522034753113985,
+            0.008105821907520294,
+            0.004310478921979666,
+            0.01760554127395153,
+            -0.0031646795105189085,
+            0.010545143857598305,
+            0.013916256837546825,
+            0.014908651821315289,
+            0.0009980766335502267,
+            -0.011189064010977745,
+            -0.03790797293186188,
+            0.001859793672338128,
+            0.00817400123924017,
+            0.018635815009474754,
+            -0.01980244740843773,
+            0.011681473813951015,
+            -0.01296931505203247,
+            -0.020363036543130875,
+            -0.01003758329898119,
+            0.011575416661798954,
+            -0.006139970384538174,
+            0.03378688171505928,
+            -0.005874827038496733,
+            -0.008204303681850433,
+            0.005136212334036827,
+            0.010454237461090088,
+            0.0066778333857655525,
+            0.014704111963510513,
+            0.0326354019343853,
+            0.010196669027209282,
+            0.014173824340105057,
+            -0.004859705455601215,
+            0.02399929240345955,
+            -0.003931702114641666,
+            0.003954428713768721,
+            0.00509454682469368,
+            -0.022832660004496574,
+            -0.0040983641520142555,
+            -0.003931702114641666,
+            -0.016711628064513206,
+            -0.007458113599568605,
+            0.0032366469968110323,
+            0.008802770636975765,
+            -0.014045040123164654,
+            -0.00649980828166008,
+            -0.01350717805325985,
+            0.01218903437256813,
+            0.002730980049818754,
+            0.018302490934729576,
+            -0.029802151024341583,
+            0.009825468063354492,
+            -0.017484333366155624,
+            -0.017044952139258385,
+            0.006511171814054251,
+            -0.0038332203403115273,
+            0.006855858489871025,
+            -0.018272189423441887,
+            -0.01318900566548109,
+            0.012817803770303726,
+            0.0007897494360804558,
+            0.013060221448540688,
+            -0.009825468063354492,
+            0.0009706153068691492,
+            0.006064215209335089,
+            -0.009833043441176414,
+            -0.010961798019707203,
+            -0.007977037690579891,
+            -0.009681533090770245,
+            -0.008848223835229874,
+            0.005621046293526888,
+            0.01656011864542961,
+            -0.013742019422352314,
+            -0.003685497445985675,
+            0.016135888174176216,
+            0.006416477728635073,
+            -0.008166425861418247,
+            0.004075637552887201,
+            -0.003793448908254504,
+            -9.948806837201118e-05,
+            0.0013740125577896833,
+            0.012439027428627014,
+            -0.0017632056260481477,
+            0.024665940552949905,
+            -0.02654467150568962,
+            0.007598260883241892,
+            -0.0054051438346505165,
+            0.005446809343993664,
+            0.018226735293865204,
+            -0.02366596832871437,
+            -0.013082947582006454,
+            0.006276329979300499,
+            -0.01966608688235283,
+            -0.008060368709266186,
+            0.0054051438346505165,
+            -0.00016405766655225307,
+            -0.019923655316233635,
+            -0.012923861853778362,
+            -0.034786853939294815,
+            -0.02309022843837738,
+            0.024665940552949905,
+            0.008401267230510712,
+            -0.0073672072030603886,
+            0.018060073256492615,
+            -0.018438849598169327,
+            0.0172722190618515,
+            -0.002225313102826476,
+            -0.006613441277295351,
+            0.003935490269213915,
+            0.007674016058444977,
+            0.025362888351082802,
+            0.002653330797329545,
+            0.008264907635748386,
+            -0.0042082094587385654,
+            -0.014931377954781055,
+            -0.006321783177554607,
+            0.01980244740843773,
+            0.02471139281988144,
+            0.018605511635541916,
+            -0.018575210124254227,
+            0.010742107406258583,
+            -0.011749654076993465,
+            0.01818128302693367,
+            -0.006537686102092266,
+            0.00039037675014697015,
+            0.01859036087989807,
+            0.007969462312757969,
+            0.025241680443286896,
+            0.019923655316233635,
+            0.006852070800960064,
+            0.014128371141850948,
+            0.011908739805221558,
+            0.001204509986564517,
+            -0.012832955457270145,
+            -0.01154511421918869,
+            -0.02840825356543064,
+            0.042847223579883575,
+            0.0025434857234358788,
+            0.008317936211824417,
+            -0.027635548263788223,
+            -0.00021389048197306693,
+            -0.0034942151978611946,
+            -0.018832778558135033,
+            0.005939218681305647,
+            0.03260510042309761,
+            0.03284751623868942,
+            0.021438762545585632,
+            -2.888172457460314e-05,
+            0.0012234487803652883,
+            0.0329081192612648,
+            -0.013673839159309864,
+            -0.00937851145863533,
+            -0.01034060399979353,
+            -0.018393397331237793,
+            -0.006852070800960064,
+            -0.0032423287630081177,
+            -0.01031030248850584,
+            -0.0061172437854111195,
+            -0.009272454306483269,
+            0.0238629337400198,
+            0.0001374249259242788,
+            -0.0022594030015170574,
+            0.006605865899473429,
+            -0.020878173410892487,
+            0.011537538841366768,
+            -0.02813553437590599,
+            -0.013431422412395477,
+            -0.008651260286569595,
+            -0.006950552575290203,
+            -0.018802475184202194,
+            0.008643684908747673,
+            -0.006942977197468281,
+            0.021438762545585632,
+            0.002914686920121312,
+            -0.023590214550495148,
+            0.007530081085860729,
+            0.015317730605602264,
+            -0.0003825644962489605,
+            0.014037464745342731,
+            -0.003890036838129163,
+            0.009060339070856571,
+            0.0045112306252121925,
+            0.011681473813951015,
+            0.016954045742750168,
+            0.010651201009750366,
+            0.009030036628246307,
+            0.003437398700043559,
+            -0.001167579204775393,
+            0.01878732442855835,
+            -0.0007163614500313997,
+            -0.03405960276722908,
+            2.9236827685963362e-05,
+            -0.014628356322646141,
+            0.029499130323529243,
+            -0.009795165620744228,
+            0.009514871053397655,
+            0.008105821907520294,
+            0.007098275702446699,
+            -0.0067119235172867775,
+            0.007916433736681938,
+            -0.008742166683077812,
+            0.0052763596177101135,
+            0.015832867473363876,
+            -0.0042877523228526115,
+            -0.0014980619307607412,
+            0.007840678095817566,
+            0.00407184986397624,
+            -0.005454384721815586,
+            -0.019711541011929512,
+            -0.012287517078220844,
+            0.00726114958524704,
+            -0.02065090648829937,
+            -0.004367295652627945,
+            0.01161329448223114,
+            -0.005454384721815586,
+            -0.009620928205549717,
+            0.01393140759319067,
+            0.02136300690472126,
+            -0.002382505452260375,
+            0.022893264889717102,
+            -0.01349202636629343,
+            -0.009833043441176414,
+            -0.007984613068401814,
+            -0.004374871030449867,
+            -0.041120000183582306,
+            0.025984082370996475,
+            0.01068150345236063,
+            -0.006871009711176157,
+            0.01895398646593094,
+            0.0022499337792396545,
+            -0.002954458352178335,
+            -0.0011041341349482536,
+            0.00010499217023607343,
+            -0.03281721472740173,
+            0.03660498186945915,
+            -0.007007369305938482,
+            0.0005449649761430919,
+            0.010272424668073654,
+            0.042029064148664474,
+            -0.004590773954987526,
+            -0.005859675817191601,
+            0.01225721463561058,
+            0.005336964037269354,
+            0.031938452273607254,
+            -0.010416359640657902,
+            -0.014340486377477646,
+            -0.01815097965300083,
+            -0.004833190701901913,
+            -0.01939336769282818,
+            -0.0038313264958560467,
+            0.014037464745342731,
+            -0.019650936126708984,
+            -0.011242092587053776,
+            -0.001287840772420168,
+            -0.012537509202957153,
+            -0.0010747789638116956,
+            0.008598231710493565,
+            0.017832808196544647,
+            -0.011348150670528412,
+            -0.004450626205652952,
+            -2.2904154320713133e-05,
+            -0.005003640428185463,
+            0.0185146052390337,
+            0.0020056227222085,
+            -0.01702980138361454,
+            0.01321930717676878,
+            -0.00526499655097723,
+            0.006749800872057676,
+            0.0074429623782634735,
+            0.007939159870147705,
+            0.05145302787423134,
+            0.010317877866327763,
+            -0.001339922659099102,
+            -0.01607528328895569,
+            0.023196285590529442,
+            -0.01749948412179947,
+            -0.014181400649249554,
+            -0.02119634486734867,
+            -0.0029923359397798777,
+            -0.007060397882014513,
+            -0.028014326468110085,
+            -9.137987944995984e-05,
+            0.0016221113037317991,
+            -0.0024980325251817703,
+            -0.010045158676803112,
+            0.0005208179936744273,
+            -0.011529963463544846,
+            -0.006954340264201164,
+            -0.008545203134417534,
+            -0.016044981777668,
+            0.02240842953324318,
+            -0.021817538887262344,
+            -0.008855799213051796,
+            0.011514811776578426,
+            0.03118089959025383,
+            -0.005011215806007385,
+            -0.014173824340105057,
+            -0.019817598164081573,
+            0.0025643182452768087,
+            -0.01763584464788437,
+            0.012181458994746208,
+            -0.01225721463561058,
+            0.015045011416077614,
+            -0.03142331540584564,
+            -0.006079366430640221,
+            -0.008992158807814121,
+            0.012840530835092068,
+            0.019817598164081573,
+            -0.0025093958247452974,
+            0.00606800289824605,
+            -0.010295150801539421,
+            0.030211230739951134,
+            -0.00012286569108255208,
+            -0.018605511635541916,
+            0.00873459130525589,
+            0.007977037690579891,
+            0.012537509202957153,
+            -0.004435475450009108,
+            0.00257757562212646,
+            0.011567841283977032,
+            0.012742049060761929,
+            -0.01634800247848034,
+            -0.003174148965626955,
+            -0.017969166859984398,
+            0.005996035411953926,
+            0.006999793462455273,
+            0.021711481735110283,
+            -0.0013550736475735903,
+            0.018272189423441887,
+            -0.01563590206205845,
+            0.00019412307301536202,
+            -0.011749654076993465,
+            -0.0082270298153162,
+            -0.0028237802907824516,
+            -0.003958216868340969,
+            0.006685409229248762,
+            -0.028211290016770363,
+            -0.01274962443858385,
+            0.005287723150104284,
+            -0.004992276895791292,
+            -0.03405960276722908,
+            0.0053710537031292915,
+            -0.02725677192211151,
+            0.013128400780260563,
+            -0.002397656673565507,
+            -0.017847958952188492,
+            -0.00030846628942526877,
+            0.026332557201385498,
+            -0.003677921835333109,
+            -0.012537509202957153,
+            0.015544996596872807,
+            -0.006757376715540886,
+            -0.02119634486734867,
+            -0.005420294590294361,
+            0.014658658765256405,
+            -0.0010719380807131529,
+            -0.012393574230372906,
+            -0.029635490849614143,
+            0.012992041185498238,
+            0.0026874206960201263,
+            -0.007386146113276482,
+            0.010893618687987328,
+            0.025953780859708786,
+            0.008151275105774403,
+            -0.005924067925661802,
+            0.0015757111832499504,
+            0.0049581872299313545,
+            0.00300938100554049,
+            0.020287280902266502,
+            0.02728707529604435,
+            -0.008014915511012077,
+            -0.0011145505122840405,
+            -0.014954105019569397,
+            0.015802564099431038,
+            0.0013020449550822377,
+            0.010507266037166119,
+            0.021893294528126717,
+            0.007045246660709381,
+            -0.005738467443734407,
+            0.003905187826603651,
+            -0.023135682567954063,
+            0.01420412678271532,
+            -0.012272365391254425,
+            0.000622140767518431,
+            0.008749742060899734,
+            -0.007283876184374094,
+            0.01437078882008791,
+            -0.0015245762187987566,
+            0.0339989997446537,
+            0.025211378931999207,
+            0.0043218424543738365,
+            0.0007362472242675722,
+            -0.028559764847159386,
+            0.003547244006767869,
+            0.008651260286569595,
+            0.014499573037028313,
+            -0.0076664406806230545,
+            0.006261179223656654,
+            -0.004355932120233774,
+            0.00959820207208395,
+            -0.0065490491688251495,
+            -0.015893470495939255,
+            0.006113456096500158,
+            0.0012689019786193967,
+            0.006795254070311785,
+            0.0023408401757478714,
+            -0.008863375522196293,
+            -0.024362917989492416,
+            -0.025984082370996475,
+            0.021984200924634933,
+            -0.03918066248297691,
+            0.0073293293826282024,
+            0.011234517209231853,
+            0.03324144333600998,
+            -0.008158850483596325,
+            -0.004980913829058409,
+            0.006276329979300499,
+            -0.016711628064513206,
+            -0.004147605039179325,
+            0.008204303681850433,
+            0.012385998852550983,
+            -0.002420383272692561,
+            0.00974971242249012,
+            0.01318900566548109,
+            -0.012378423474729061,
+            0.03660498186945915,
+            0.016514664515852928,
+            0.01467380952090025,
+            -0.004007457755506039,
+            0.01312082540243864,
+            0.0066172294318675995,
+            0.019044892862439156,
+            0.025165924802422523,
+            -0.017454031854867935,
+            -0.002365460619330406,
+            -0.02292356640100479,
+            0.024165954440832138,
+            0.00726114958524704,
+            0.004556683823466301,
+            -0.008310360834002495,
+            0.014870774000883102,
+            0.0017954016802832484,
+            0.0028199926018714905,
+            -0.006696772295981646,
+            -0.010173942893743515,
+            0.019241856411099434,
+            0.0034317169338464737,
+            0.014870774000883102,
+            4.8501173296244815e-05,
+            -0.005503625608980656,
+            0.006052851676940918,
+            0.014832896180450916,
+            -0.023590214550495148,
+            -0.00203592493198812,
+            -0.019484274089336395,
+            -0.03202936053276062,
+            0.009022461250424385,
+            0.0016135887708514929,
+            -0.007052822504192591,
+            0.03230207785964012,
+            -0.010083036497235298,
+            -0.019787294790148735,
+            -0.008067944087088108,
+            -0.013113250024616718,
+            -0.010469388216733932,
+            0.0008271536789834499,
+            0.003908975515514612,
+            -0.00012428610352799296,
+            -0.009317907504737377,
+            -0.005749830510467291,
+            0.01990850456058979,
+            -0.0030358952935785055,
+            -0.00753386877477169,
+            0.00868156272917986
+          ],
+          "embedding": [
+            0.0040707582550429606,
+            -0.010230625151989194,
+            -0.021520242101903186,
+            -0.010115756270640508,
+            0.026738734586159035,
+            0.01796370233223267,
+            -0.012245199029994153,
+            0.06506924494147726,
+            -0.0014087587222033442,
+            0.009300068719271429,
+            0.003787396973047584,
+            0.002818772639754174,
+            0.022129077062176306,
+            -0.021526709808445683,
+            -0.0034388135367225824,
+            0.02763761747879405,
+            0.013210013805395276,
+            0.025255872672296602,
+            0.00029011480909986703,
+            -0.05438974762290585,
+            0.016671640187490865,
+            -0.016426252259893328,
+            -0.056591180651012554,
+            0.009335648425523075,
+            0.006141624459418827,
+            0.02568943956694641,
+            0.019377132054042116,
+            0.017946464337078746,
+            -0.057184776985927686,
+            0.03208527139096687,
+            0.030132392063755547,
+            -0.003846032371791839,
+            -0.008376524427531415,
+            -0.0010513963058713435,
+            -0.02734868388724371,
+            -0.014879400370545479,
+            0.04383920959748825,
+            0.02563613659516289,
+            -0.020117467074438997,
+            0.0398016699155716,
+            0.029896497884197775,
+            -0.0004195844530067313,
+            -0.023302720788253625,
+            -0.001168337916577517,
+            0.009149021369661628,
+            -0.02508145721900992,
+            -0.004307594138480351,
+            0.00550940415564303,
+            0.014016998918718476,
+            -0.028607379616884317,
+            -0.0018317656587526954,
+            -0.01973464940415756,
+            0.03762738392056255,
+            0.015358106058583114,
+            -0.03931907682104936,
+            0.011519637298832444,
+            0.004482080558226775,
+            0.0747867063383242,
+            0.05550679018263077,
+            0.023243129601931176,
+            0.02671631686665238,
+            -0.009686146566573536,
+            -0.04088582189725495,
+            -0.0065344623594924085,
+            0.019617295999325712,
+            -0.004722087402575777,
+            -0.002871844114437583,
+            0.003935087265300519,
+            0.03229890413393201,
+            -0.02813577047904649,
+            -0.05124172132597234,
+            0.03325235407841968,
+            -0.008687655557217192,
+            -0.011027174838246948,
+            -0.003653553063547375,
+            -0.02350737945182988,
+            0.016223087939183468,
+            0.03789252356462714,
+            0.0034350623378603083,
+            0.00195158657888739,
+            0.011789712862232108,
+            -0.0004435439660769551,
+            -0.0037330456918125894,
+            0.0031552713243866858,
+            0.04465432992298031,
+            -0.0032465421571915712,
+            -0.0075366743861922745,
+            -0.059958752731410256,
+            0.016784616861008125,
+            -0.028734506065772967,
+            -0.03035575736273943,
+            0.02680390892611741,
+            0.02338368719038086,
+            -0.0030658375437784277,
+            -0.03754896050794851,
+            0.07005659523248504,
+            0.0020711575553570695,
+            0.013034425083076923,
+            -0.02316832262132332,
+            0.021332142030732894,
+            -0.0318756233020585,
+            0.02270782509507388,
+            0.01570810309454529,
+            0.004788124787612736,
+            0.026727250200142456,
+            -0.022380142778058276,
+            -0.008593910379436904,
+            0.014439754382387034,
+            0.015007697617383174,
+            0.07662531963612977,
+            0.011195998137723795,
+            -0.010809283538470098,
+            -0.042227822120219305,
+            0.003824320933275175,
+            -0.03797629427607094,
+            -0.02347390684453348,
+            -0.0526876440280276,
+            -0.017799861680287913,
+            0.01701482796266917,
+            0.002730395144917155,
+            0.03789451759899825,
+            0.038338545350313946,
+            0.028463789278406586,
+            0.015494583705011921,
+            -0.05700237266350479,
+            0.047895170359075166,
+            -0.0064406894534929406,
+            -0.0007954721295065988,
+            0.012543029839729358,
+            -0.004703334868741216,
+            -0.0060569679284068,
+            0.012324901254711192,
+            -0.019883935809784068,
+            -0.04253565942154215,
+            0.02164409276211115,
+            -0.0075367555415808635,
+            0.03617159147722157,
+            0.07567745988686005,
+            -0.04225977020036783,
+            -0.022548837589622003,
+            0.029811602145850144,
+            -0.02175792196819533,
+            0.02069861458115363,
+            -0.00941331829664397,
+            -0.035479639885943264,
+            0.020352552580119105,
+            0.003678588588550461,
+            0.06776105274336729,
+            -0.02346047721661687,
+            0.02721075076349443,
+            -0.029386781707644253,
+            0.023530831921117993,
+            -0.02077660533100372,
+            0.020022475323372906,
+            0.024195118676513715,
+            -0.028378518720494666,
+            -0.012900969023913307,
+            0.019264089970719624,
+            -0.0374983624629265,
+            -0.019726171557384355,
+            0.00979636085642416,
+            -0.014479528435832063,
+            -0.014533384583777662,
+            0.005904623758767224,
+            0.010344475500263744,
+            0.05659646358026115,
+            0.04163866113482093,
+            0.007152494819169405,
+            -0.07584500130974763,
+            0.05471858597878099,
+            0.03580211013427615,
+            0.044161429444475306,
+            0.02258180566046248,
+            0.00980779221334715,
+            0.0011193505603397395,
+            0.018086566768892507,
+            0.015605066702191871,
+            -0.00954727399396313,
+            -0.027950079146919375,
+            0.03028514025485407,
+            -0.006351669761257431,
+            0.005095804801686436,
+            0.009620040085743162,
+            0.019748001227987134,
+            0.033709535059510916,
+            0.008203559956940196,
+            0.007389752602218563,
+            0.04197771878351742,
+            -0.02067363758471765,
+            0.007543739188471913,
+            0.011207941261799052,
+            0.013964827251172813,
+            0.03839562075248879,
+            0.0009775244543292637,
+            0.006469507611492216,
+            -0.022808235395746654,
+            -0.024795004398633292,
+            0.006249024864814684,
+            0.011292877109066925,
+            0.01596839619826819,
+            -0.0014105396434174883,
+            -0.04152075438277094,
+            0.008267788411079337,
+            -0.028834191212891387,
+            0.030940526405027906,
+            -0.0018271056434014425,
+            0.006185475047820603,
+            -0.007920574674727123,
+            0.012907560753043616,
+            -0.015165704767187252,
+            -0.021962244496450727,
+            0.0037071053072833147,
+            -0.013508376064336042,
+            -0.015178823540513864,
+            -0.02518557053061311,
+            0.011693256916760944,
+            -0.01716015674592331,
+            0.02048897513447166,
+            -0.003412078966361745,
+            -0.001807298511378529,
+            -0.0012575655363779904,
+            0.03116103961636643,
+            0.011668533361641842,
+            -0.011637010548030312,
+            -0.040260195687925236,
+            -0.011129376691522074,
+            -0.0034493180842871908,
+            -0.02098244789711689,
+            0.026658743116016938,
+            -0.01923736325953796,
+            -0.009723937181300192,
+            -0.013134679989565051,
+            -0.011974548914818263,
+            -0.004028712301052159,
+            0.019628329175867303,
+            -0.028192034966932684,
+            -0.012638520793012082,
+            -0.011568652932013111,
+            -0.0025919910519445737,
+            0.012497908463405111,
+            0.015113186712897145,
+            0.014880607940212641,
+            -0.03059227859452334,
+            -0.021899716217105084,
+            -0.0008513506358489523,
+            0.011985931118464679,
+            -0.01321584595585435,
+            -0.011735288366213981,
+            0.010710683541435907,
+            0.01961722172525993,
+            -0.023571946726192585,
+            -0.03364281880125354,
+            0.0014460076743756465,
+            0.016563167828467187,
+            -0.02891589059260944,
+            0.04595166331401113,
+            0.034271553243501465,
+            -0.027557971577291153,
+            -0.007138821520412525,
+            -0.039508407807651844,
+            0.06814329636830853,
+            0.00022136123564675344,
+            -0.039718016705753735,
+            0.0503457594963753,
+            -0.003580581891576583,
+            0.013820796398521455,
+            0.02054050419088757,
+            0.035682661890720495,
+            -0.0013220739026852943,
+            -0.0052759394697328675,
+            0.04733018719162478,
+            -0.0187845855131196,
+            -0.03982699468608443,
+            -0.023905460465070062,
+            0.002631233133039351,
+            -0.03090995554989513,
+            -0.012617612051488022,
+            -0.013605787969473513,
+            0.039672783793764745,
+            0.008669582242097126,
+            0.009974277556736362,
+            0.005535636914843583,
+            0.014180668493093584,
+            0.012042659947850441,
+            0.011870115226306776,
+            0.00451259454348581,
+            -0.014023048592193627,
+            -0.007769033996392623,
+            -0.05715987527668358,
+            0.003374564518384728,
+            -0.028801158523506985,
+            -0.00756331369170698,
+            -0.02163054894055572,
+            0.005367181573233089,
+            0.02308705505078254,
+            -0.005438909398653553,
+            -0.00873620909426313,
+            -0.0036530652237719077,
+            0.013920407174296543,
+            -0.01539613379172909,
+            -0.03908211213269472,
+            -0.016810726548844992,
+            0.007460173854386443,
+            -0.007069189426742819,
+            -0.041197222498539215,
+            -0.003843233633335066,
+            0.004373367480223065,
+            0.016194121660459908,
+            -0.029165150066068787,
+            -0.00735139702658342,
+            0.017257895082337448,
+            0.014153313280536095,
+            -0.002376178042741926,
+            -0.023223700836427403,
+            -0.0319877537205969,
+            0.02088060774052704,
+            0.0012829290529515294,
+            0.03251550040316258,
+            -0.04128453069131847,
+            -0.023376114412063578,
+            0.04083022735076889,
+            0.005690491480259003,
+            -0.05810403800689653,
+            0.016582434816685452,
+            0.010402038964417569,
+            0.016795644054313566,
+            -0.00908590043098552,
+            0.05003656634510042,
+            -0.037216197954986735,
+            -0.013027960261454162,
+            -0.007090943677204207,
+            -0.05175371787508879,
+            -0.0020065660448063322,
+            -0.014908489552053589,
+            0.010627472758306387,
+            -0.03275473161158777,
+            -0.017334608434133866,
+            0.025321047439048318,
+            -0.0033515195870701753,
+            0.011526068468647207,
+            -0.026783879816504975,
+            -0.0012830935419413407,
+            -0.017506908439998053,
+            -0.02114853058440477,
+            0.004878884893669383,
+            -0.020936900837115968,
+            -0.0025587027781733642,
+            0.01557850246191957,
+            0.023652335794299834,
+            0.011489428313303988,
+            0.0019089137782552768,
+            0.02054907941022243,
+            0.014361232983780595,
+            -0.0056413109308506085,
+            -0.0010300050480036684,
+            -0.03413262505887213,
+            -0.018821160485010483,
+            0.03361989907065425,
+            0.013125889701529858,
+            -0.003742167496379789,
+            0.008813100942466033,
+            -0.0383265293638935,
+            -0.035316957263713485,
+            0.03020499323039515,
+            -0.05001623971243151,
+            -0.00855479667673597,
+            -0.005939674940109814,
+            -0.026451831943555135,
+            -0.008032873049527808,
+            0.0009431571685654594,
+            -0.014387605128858954,
+            0.016382559541755014,
+            -0.024909510232038502,
+            0.003715736505444385,
+            -0.006016712902409688,
+            -0.009110197692342108,
+            0.05507263160354303,
+            -0.022103031032598353,
+            -0.0475415595313825,
+            0.021958434235402655,
+            -0.016670809659931243,
+            0.0323519363801999,
+            0.03867494951405658,
+            0.011124508964363016,
+            -0.008002445714431074,
+            0.034041838454818577,
+            0.018400123235267234,
+            -0.01161754789471199,
+            -0.025901844576206774,
+            0.021800298132870245,
+            0.00943488425697727,
+            -0.05906691123008531,
+            -0.00782549274387971,
+            -0.019357446956797447,
+            0.008682376344965976,
+            0.02863077694455766,
+            0.0024925977373894447,
+            -0.002682170968419412,
+            -0.013590235283325244,
+            -0.021511149507762284,
+            -0.03255453916518719,
+            0.009039307015853905,
+            0.008305286476500875,
+            0.009312625027668458,
+            -0.0014144165766983945,
+            -0.007853869334376857,
+            0.009067065760942163,
+            0.022485201611182105,
+            0.03966189080765722,
+            -0.041627289519629344,
+            0.002707086456239785,
+            -0.022678178050044222,
+            -0.03653753142622053,
+            0.0036574646778835588,
+            0.029996160786576367,
+            0.02901163765559891,
+            -0.01971777193183787,
+            -0.0036589934841109884,
+            0.009181234938554578,
+            -0.025979483002227535,
+            -0.00773645103995767,
+            0.004280770859737866,
+            0.04116668301017644,
+            0.006201085381597264,
+            0.01402728258112097,
+            0.001060407317110527,
+            -0.00920801784000087,
+            0.02109880889040528,
+            0.025211022083982727,
+            -0.029866678264942617,
+            0.014439702646820226,
+            0.009379206296908577,
+            0.04156435812161122,
+            -0.037309655469525846,
+            -0.011819731348299495,
+            -0.02766498020409985,
+            -0.010454143230794476,
+            0.027911532296566614,
+            -0.04127890742215716,
+            -0.006948536183086887,
+            0.010207784113909938,
+            -0.0016584043356136647,
+            0.016437495428919636,
+            -0.014607861750686952,
+            -0.03222311919805017,
+            -0.000610723203118213,
+            -0.02867445198803213,
+            -0.024772816792038133,
+            -0.016510309753766612,
+            -0.018581201775667577,
+            -0.07450654873867177,
+            0.00026832248829154255,
+            -0.06724696990816072,
+            -0.02447002541264633,
+            0.04885569249258513,
+            -0.010195604207134053,
+            0.03332905935704749,
+            -0.02815272112271314,
+            -0.005296340304911926,
+            -0.05908835042526019,
+            0.0038476172058970774,
+            0.025753576863447866,
+            0.02386003857017594,
+            0.005206447371391696,
+            0.024799057871126032,
+            0.012060738268265606,
+            -0.04146503002980051,
+            -0.024949907556862643,
+            -0.014503475223770958,
+            -0.0016340466065744286,
+            -0.0034009167013795906,
+            -0.031298432995411005,
+            0.036241272149880994,
+            0.008902644391348814,
+            0.03550223922667804,
+            0.04463301409716232,
+            -0.010385984413759852,
+            -0.03305004479734889,
+            -0.006952739338820577,
+            -0.0175981588092108,
+            0.042668533202572856,
+            -0.017112184620086308,
+            0.005149173014526686,
+            -0.03219589107997635,
+            0.024181808150698054,
+            0.0014339790422945406,
+            0.01254721190611376,
+            0.011099458649404119,
+            0.0027626836767364253,
+            -0.026883640446090158,
+            0.0024160770766463607,
+            0.023874514847956222,
+            -0.022558433617669373,
+            0.00917667715473093,
+            -0.006754091166465801,
+            -0.033686935718776144,
+            -0.01078389638118685,
+            -0.013595327486167114,
+            0.024712781935748277,
+            0.013168464807451798,
+            0.005875864604862434,
+            -0.02417814420558463,
+            -0.0023978800459714144,
+            0.028650463366555304,
+            0.020176980768301288,
+            -0.02118184201861616,
+            0.019082340279671355,
+            0.019734925740100277,
+            -0.04209645071102001,
+            -0.04217310001316092,
+            0.006553913719098956,
+            -0.003735719227468833,
+            0.0061793909220459,
+            -0.028285889794908054,
+            0.008181788076484304,
+            0.004432488437404309,
+            -0.01743905248466261,
+            0.041221269621866216,
+            -0.014892354419172107,
+            0.005087102208335201,
+            -0.022198383716186407,
+            -0.011936388003605221,
+            -0.00027487042024290816,
+            0.044599215801158845,
+            0.008068975171827777,
+            0.003496950921841073,
+            0.02238779276447619,
+            0.0054839808262270315,
+            0.04816723616939378,
+            -0.02685221260466647,
+            -0.0190356946337995,
+            0.016464136459560113,
+            -0.0176092780782677,
+            -0.012513274095432353,
+            -0.010695421445334002,
+            -0.004784348908457072,
+            -0.022203041334932165,
+            0.010788155380415583,
+            -0.018353897272137543,
+            -0.02277431189907027,
+            -0.003945465850796918,
+            0.016024639792926797,
+            -0.006248368166794934,
+            -0.0038769803892794875,
+            0.0016292556540434277,
+            0.02412864830338344,
+            0.02892955453819876,
+            -0.02408022470555968,
+            0.019190282372105023,
+            0.018941584034503536,
+            -0.006709516380602965,
+            -0.020956377933747532,
+            -0.01964885206488053,
+            0.01434623424964373,
+            -0.01352968586212152,
+            -0.013872556892707697,
+            0.004052798542613835,
+            0.009201147374299997,
+            -0.004086067985859617,
+            -0.04185829685685201,
+            -0.014668698123513235,
+            0.003179403563664022,
+            -0.0006415014901450319,
+            0.019360184344240947,
+            -0.0150817538567364,
+            -0.029771999278536083,
+            0.014221776097046751,
+            0.022944884879589678,
+            -0.04493425220030206,
+            0.02117577719533236,
+            -0.0030648751497261983,
+            -0.00016251533643103604,
+            -3.817296018601414e-05,
+            0.008727767382077138,
+            0.026914779443266063,
+            0.03456565094655899,
+            0.01974806772696219,
+            0.00450800007152441,
+            -0.008952599700759673,
+            0.025184190461155245,
+            0.009846044083064704,
+            -0.025168735183134738,
+            0.010983353565382029,
+            0.016740721743326522,
+            0.0070838930012654296,
+            0.009173879407277282,
+            -0.006078288608766152,
+            -0.018269347893097643,
+            0.013181494188066667,
+            -0.01435693747022231,
+            0.01678907949532146,
+            0.05154463552784974,
+            -0.00039632596424461964,
+            -0.02461439550772063,
+            0.008418147037164186,
+            -0.006544319842335593,
+            -0.010065129286193278,
+            0.02214628550460128,
+            0.0029515574751370417,
+            -0.030862152033847717,
+            0.007509556025223583,
+            0.00675485304374519,
+            0.01894936231559203,
+            -0.02355525295639599,
+            -0.01387885858544803,
+            -0.00431398020087544,
+            0.0021208056468763523,
+            0.01580766504149989,
+            -0.004739477121073512,
+            -0.013204823447960706,
+            0.02994098229517376,
+            0.025496728234662373,
+            -0.00026124946738213036,
+            0.015060064980768175,
+            0.022262345898533903,
+            0.009968408664790368,
+            0.007588431141088797,
+            -0.0083063413609779,
+            -0.031336272171937095,
+            0.0018957858383561667,
+            0.05800811417728381,
+            -0.014937085296865802,
+            0.04313875432235027,
+            0.02018476425923236,
+            0.015657644644987107,
+            0.017880263053253607,
+            -0.01662268082870562,
+            -0.008740053682875323,
+            -0.0017630394204179638,
+            -0.008294417990024647,
+            -0.027613464270562948,
+            -0.005953209556351086,
+            -0.02379961210881074,
+            -0.013878676554701543,
+            0.015418315245979704,
+            -0.014764728065382074,
+            0.0027420640870858457,
+            0.010395538988699663,
+            -0.006590633833835581,
+            -0.00841537209927409,
+            -0.003943772700797381,
+            0.03376338017637296,
+            0.011980354721458463,
+            0.027761929822019422,
+            -0.00011128539412582973,
+            0.009342430952107655,
+            -0.018419526564396063,
+            -0.006839752805318664,
+            0.03271190887864607,
+            0.006011520427944665,
+            0.004652835471939905,
+            0.01485920920301398,
+            0.014305626381398871,
+            0.038015677219487023,
+            -0.005102005401565164,
+            -0.018204460895989052,
+            0.02065495133597456,
+            -0.0049699013356374775,
+            0.019149036693562712,
+            0.008825981577388056,
+            -0.002660055848523568,
+            -0.004292379561953203,
+            0.03090270710860331,
+            -0.020243440437217503,
+            0.018735727616321387,
+            -0.02238236089688497,
+            -0.0276584413929402,
+            0.023028408906176946,
+            0.008320436733620496,
+            0.04047641118026783,
+            0.011463851066078041,
+            -0.02247143742166039,
+            -0.034588725413285185,
+            0.009827713930789184,
+            0.00337071450636392,
+            -0.024446280199879062,
+            0.017542777160283324,
+            0.00780836483490212,
+            0.008453886667283516,
+            0.016624116261219637,
+            -0.004175682824655864,
+            -0.0042784362933758835,
+            -0.001546656717896957,
+            -0.008819739495525477,
+            0.01406790007187259,
+            -0.015487658289541583,
+            -0.021457182907140035,
+            0.024687169583245905,
+            -0.03915911962859764,
+            0.01771290139419629,
+            -0.006510046907173026,
+            -0.00997450880725535,
+            -0.02059560682587115,
+            1.5939836949198004e-05,
+            -0.007159493752862143,
+            -0.0004782363814453761,
+            -0.017503181957042913,
+            0.016643971600145645,
+            0.024056442566869802,
+            -0.013209211443909658,
+            -0.00445521990927108,
+            0.0020313626821877526,
+            -0.029235692455820056,
+            0.006662209097255912,
+            0.016993027053110635,
+            0.00529940612845993,
+            0.0026104743851393716,
+            0.0001707934714983891,
+            -0.031289446611163674,
+            -0.005782208620284939,
+            0.03053516889069683,
+            0.05590704747438521,
+            0.021628254147289437,
+            0.02403314981042378,
+            0.02908606663381421,
+            0.02717206823892633,
+            -0.003198844932040613,
+            0.0016217508103442348,
+            -0.01620660606431983,
+            0.027098323143010743,
+            -0.011664168290676943,
+            0.04322175021589083,
+            0.008652323184522541,
+            -0.049493020736445056,
+            0.004537958157696794,
+            0.00943239454147692,
+            0.034918603888916544,
+            0.016678716235690318,
+            0.0005887414054803805,
+            -0.005894338664998101,
+            0.03279165663336323,
+            -0.02735726709032382,
+            -0.011962070182991011,
+            -0.00036014407560325284,
+            -0.019079003730607717,
+            -0.015938295169860884,
+            0.010357611943935459,
+            0.04278924678845588,
+            0.0009957483598272451,
+            -0.0071422790468850215,
+            0.01571201177743999,
+            -0.009445307674383,
+            0.02286338492075306,
+            0.007336440384050498,
+            0.0042809497342459904,
+            0.025581467690095977,
+            -0.0025007871391868105,
+            0.02300539823997411,
+            -0.000628898771317254,
+            0.004582863909134464,
+            -0.015960578501883316,
+            -0.00930649396114688,
+            0.027838455754803444,
+            -0.011112268986721,
+            -0.029253746504396393,
+            -0.008734785561831333,
+            0.00472579547270089,
+            -0.0003805825712986138,
+            0.0021854358149324227,
+            -0.01183885533661493,
+            -0.010840951201591629,
+            0.019965691788904057,
+            -0.004922943301424112,
+            0.003800048754664761,
+            0.012664049429916252,
+            -0.0030124219674414897,
+            -0.01910844996750822,
+            0.01889983984736845,
+            -0.003858283778519141,
+            -0.024168821338992813,
+            0.020083955711766945,
+            0.01850497889545421,
+            0.045156567155119454,
+            -0.01586824055785609,
+            0.004612757419198795,
+            0.017497774574602505,
+            -0.002172088240279357,
+            -0.019617327081566827,
+            -0.014944795772093997,
+            -0.018212462639041492,
+            -0.015975283672064702,
+            0.012837956594691596,
+            -0.01243789877725244,
+            0.019500571353639314,
+            -0.0380735226388175,
+            0.012590900534826689,
+            0.004832168601136412,
+            0.007505385730121269,
+            0.016011983211212992,
+            0.025994560026860358,
+            0.002157436199771672,
+            -0.033465476168065784,
+            -0.01014599891204166,
+            0.007845373376916211,
+            0.010093432683262838,
+            -0.002849185178237036,
+            -0.018147004183434554,
+            0.003633790611809425,
+            -0.0266147420637404,
+            -0.018644422659309035,
+            0.022926737299761104,
+            -0.04798772015599113,
+            0.017908415301694906,
+            0.024859125158586466,
+            -0.04343699525707025,
+            -0.04886699607556352,
+            -0.011459188432595229,
+            -0.036774381316312464,
+            0.0014128005215813578,
+            -0.0005256067584181517,
+            0.011194992852264102,
+            -0.009342140079829378,
+            -0.018598671981710183,
+            0.01584789414301952,
+            -0.0036412641242200418,
+            -0.004225840239935549,
+            -0.00616696736018045,
+            -0.0025598631463178384,
+            -0.010188803157718543,
+            -0.03244694662411977,
+            -0.03290891136780909,
+            -0.00033742479296272127,
+            -0.009743927051414455,
+            -0.02334288359945272,
+            -0.03447062782872289,
+            -0.008728968107967028,
+            -0.037753659749578956,
+            -0.00515561961442376,
+            0.04070890150714597,
+            0.027517147608335957,
+            0.009449414978133334,
+            0.008086900089929912,
+            -0.013989034604490105,
+            0.05691047488419298,
+            -0.020425883381389086,
+            0.039466620642512155,
+            0.005301493569940605,
+            -0.0037222353899918703,
+            0.013075425214878277,
+            0.027710826422351564,
+            -0.006253665106995653,
+            -0.0017412928073238833,
+            -0.01928096630475438,
+            -0.012920476859996416,
+            0.009444408011467955,
+            0.015799884767184885,
+            -0.0025083061619197524,
+            0.001274457251739847,
+            0.00930533161700852,
+            -0.0006652478540264343,
+            0.017258038491335247,
+            0.03996268817486439,
+            0.003833844612473671,
+            -0.00826404141706295,
+            -0.018822002570963614,
+            -0.003188844900317347,
+            -0.011807006674248829,
+            0.010482058734260632,
+            -0.015410635475145028,
+            -0.011540803927522013,
+            -0.01249428777269578,
+            -0.017081856786754605,
+            -0.002183034976477535,
+            0.003114834030705039,
+            0.0027475411722881747,
+            0.02261627635192778,
+            -0.015872027027080824,
+            -0.022487581027490516,
+            -0.0035823117885461506,
+            -0.02398778449664045,
+            8.13716819067491e-05,
+            -0.01050060782407259,
+            -0.015616181283381218,
+            0.013676480044963827,
+            -0.021513173575512936,
+            -0.01091399089076612,
+            -0.008699176177861256,
+            -0.0006986953850759474,
+            0.005853723710671249,
+            -0.00993355748621621,
+            -0.00010241843582159751,
+            0.01150872694850241,
+            0.005559552266827916,
+            -0.012336449703762907,
+            0.0039431718621504,
+            -0.012740550272925562,
+            -0.004432059174951158,
+            -0.03592309122711106,
+            -0.019665686718735042,
+            -0.01846795637714379,
+            0.005642094448288269,
+            -0.005108964028543505,
+            -0.020586678054151854,
+            0.00668364061048941,
+            0.0034775127050453774,
+            0.004713169526661792,
+            -0.010723667875209928,
+            0.013944735241748791,
+            0.010882420168586074,
+            -0.003518245916227596,
+            0.013827334494230662,
+            -0.001330399936559009,
+            -0.020286697504699764,
+            -0.011139922402071095,
+            0.03538258948179278,
+            -0.024961826464903997,
+            -0.0018019328351763667,
+            -0.0026423695690168066,
+            0.00597003516870726,
+            -0.03071835946501977,
+            -0.013634212232744325,
+            0.02164400349828422,
+            -0.005846552894831441,
+            -0.026367014426434226,
+            0.03399116480234033,
+            0.03473258231629753,
+            0.010041605592929843,
+            -0.013423384340589794,
+            0.04730054254696893,
+            -0.012968978954938308,
+            -0.018274800841400665,
+            0.026826644936918647,
+            0.0278389611094493,
+            0.02525164095926168,
+            -0.029782550447890466,
+            -0.003010713086487275,
+            -0.02393629084357885,
+            0.017993366287762866,
+            -0.0263476663180971,
+            0.01864011499179424,
+            -0.02332263152537849,
+            0.016014908881756034,
+            0.01868464825773604,
+            -0.003104041044182249,
+            0.026313491086684425,
+            -0.0052945248919367815,
+            0.015253731232325074,
+            0.0012371308759955573,
+            -0.02119765517013431,
+            0.007795466385020642,
+            -0.006469500272298062,
+            0.025762169721880456,
+            0.00833992768202466,
+            0.01801901639978976,
+            0.022019940825351116,
+            -0.0409522298213741,
+            0.02003698451081499,
+            -0.001367916864679543,
+            0.01782984895971748,
+            0.0038516134262842373,
+            -0.007885432294782304,
+            0.014221728513400065,
+            0.010991352188016265,
+            -0.012851099261924757,
+            0.003840539873231741,
+            0.0008923482291826014,
+            -0.028928104542716937,
+            0.007066004768771198,
+            -0.026914574065060926,
+            0.025866907400807087,
+            -0.005526774784152086,
+            -0.00998122210956923,
+            -0.020744532673097317,
+            -0.0071425207262770865,
+            0.003754468896273156,
+            -0.012747874997429746,
+            -0.011385805601999295,
+            -0.024700457669046855,
+            -0.02401291382882952,
+            0.029591582802488962,
+            -0.014054332096344003,
+            -0.04015978298687865,
+            0.001234440863385445,
+            -0.023301255402129767,
+            0.01193125007773239,
+            -0.005793124129267758,
+            -0.024440100649464773,
+            -0.042563399082257154,
+            -0.017548888269389457,
+            -0.0009394887961485922,
+            0.022739763644266633,
+            -0.00114156949394707,
+            0.041791395064755515,
+            0.0008816696205437911,
+            0.022207132857533028,
+            -0.015547580543606755,
+            0.005319731264840819,
+            -0.02071229859314312,
+            -0.041495579742912285,
+            0.03660432378985971,
+            -0.014637031447181937,
+            0.025854844497435178,
+            0.03324325296704186,
+            -0.013009347371118843,
+            0.001354504885302625,
+            0.02652570709474851,
+            -0.014618100923440427,
+            -0.010782232747863238,
+            -0.0034936699515378212,
+            0.003461670978569965,
+            -0.006468116285021298,
+            -0.027793221545159466,
+            -0.007752458229193391,
+            0.04193706715422074,
+            0.017894069367616637,
+            0.021928994684423043,
+            0.022606401934291136,
+            -0.017972404868085358,
+            -0.026402048654300563,
+            0.003656646576244765,
+            -0.013129722990153284,
+            -0.018090360146368677,
+            -0.01624598856366761,
+            0.012397150789705746,
+            0.011741002980073786,
+            -0.004052422390675261,
+            -0.01836750423615859,
+            -0.020929335029248472,
+            -0.016327412417122416,
+            -0.0042615345226698015,
+            -0.008458117095535,
+            -0.005002582649405199,
+            -0.012158173174049797,
+            -0.020409616620022143,
+            0.03706368540614453,
+            0.0285565642978937,
+            -0.0041732933884846566,
+            -0.00607154864704767,
+            -0.011142150176334957,
+            0.008834431806990914,
+            0.006021232046087652,
+            -0.005028870471297621,
+            -2.3396043292139426e-05,
+            0.005112033372713154,
+            0.022822312435625314,
+            0.007271235299348001,
+            0.0035791789459533856,
+            -0.06340305811365254,
+            -0.005392000517722054,
+            0.022560201096078862,
+            0.012581401789917604,
+            -0.0032248219865223496,
+            0.025180718735801837,
+            0.0006451724964970927,
+            -0.009714514747463593,
+            -0.018389422219956937,
+            0.024728862150150004,
+            0.008760329201960964,
+            0.020988700992999768,
+            -0.014974763313686516,
+            0.003845297486923296,
+            0.0006445983847138896,
+            0.00750394530065325,
+            -0.0019792992847026793,
+            -0.008578254004439607,
+            0.025776677078006516,
+            0.016593438298233645,
+            -0.005695578040595484,
+            0.01604018433477423,
+            -0.0017497260488899657,
+            -0.021460543443428924,
+            -0.010265213095609554,
+            0.00042017209251926397,
+            -0.010570864288876387,
+            -0.018015790888295944,
+            0.017569414447505878,
+            -0.002601036282535863,
+            0.0072399377790164435,
+            0.030281707151249355,
+            0.0423183067569767,
+            0.009566493398534713,
+            -0.005928603293907002,
+            -0.0037936911251953654,
+            0.015477873304924939,
+            0.01513127995244932,
+            0.010741256034652754,
+            0.03015858543221727,
+            -0.029247492875021843,
+            -0.006690306960497428,
+            0.0046653727410352576,
+            0.004610405633429778,
+            -0.011692179099382171,
+            0.030055897497741527,
+            0.04569730592994177,
+            0.006736813244286893,
+            -0.004799978513117371,
+            0.002121067028052434,
+            0.011299889107780231,
+            0.007768137218140627,
+            -0.018927080714961783,
+            -0.0012222962555201746,
+            -0.006009911026251445,
+            -0.014001766533604134,
+            0.026941585318261085,
+            -0.003960837039351648,
+            -0.011848232223811548,
+            0.02903357274500332,
+            0.1105267435019781,
+            -0.006011029338781496,
+            -0.0053646998281195206,
+            -0.03242321704621517,
+            -0.015194102811332289,
+            -0.008860590919147765,
+            0.0056988205877963076,
+            0.01635957585050017,
+            0.020327661913559888,
+            -0.0008943408274661007,
+            0.016820860249955754,
+            0.02822175614376368,
+            -0.013854827839129885,
+            0.007906244895327178,
+            -0.030891381690403466,
+            0.015702117924362928,
+            -0.005383974926093977,
+            -0.012032955400313954,
+            -0.0026169511828034054,
+            -0.008618307035330928,
+            -0.0037356680273096105,
+            0.0009666430093455632,
+            0.011296307527684261,
+            0.004427516814276546,
+            -0.014484198313369681,
+            0.002829529156309784,
+            -0.008595300985863424,
+            -0.005122996529848575,
+            -0.02393571073523279,
+            -0.024119245543788293,
+            0.005223147047205755,
+            0.0019389305548629142,
+            6.344308292376579e-05,
+            0.006848965054539454,
+            -0.003810712095343454,
+            0.030149081317333418,
+            -0.01700322008272963,
+            0.0012288912810468446,
+            0.01193226302726573,
+            0.012332749933413344,
+            0.01688602002694633,
+            0.0028600984823761366,
+            -1.9882692552530084e-06,
+            0.021172977047420227,
+            -0.010514036509402561,
+            0.01460334896316005,
+            0.013091417303145243,
+            -0.04103572381458182,
+            0.0089746401463863,
+            -0.0027841289484449873,
+            -0.024252447218794344,
+            0.004446472209246425,
+            -0.018987867343500102,
+            -0.00881974688349601,
+            0.0005365625696166774,
+            0.012054719659248512,
+            0.01960300421041832,
+            -0.008553943699846696,
+            0.02111201386480852,
+            -0.0003501657740808554,
+            0.004177088753111459,
+            -0.012879944670340026,
+            0.0065222126975051685,
+            -0.007563354540402343,
+            0.0008168128607271199,
+            -0.018213241937597355,
+            0.012526561612175514,
+            -0.0008285725179894544,
+            -0.015737003159436623,
+            -0.011337688275309921,
+            0.004737346137976519,
+            -0.0027993545140114705,
+            0.0030843163114700313,
+            0.010153740340626154,
+            -0.008815302484351509,
+            0.004919275910399053,
+            -0.0017541042538165297,
+            0.002562816687714991,
+            0.014285249186421434,
+            -0.029816964593980453,
+            0.005357227738099325,
+            0.009154842495811638,
+            -0.00048782303639333943,
+            -0.013074976152495576,
+            -0.004054505559072866,
+            0.0028322102449485015,
+            -0.016877328079971877,
+            -0.012128199639779175,
+            0.023421330164419888,
+            -0.012565145848820333,
+            0.008609070188894315,
+            -0.010936598722317274,
+            -0.018401801734378692,
+            -0.01066119413133089,
+            -0.005254229780560808,
+            0.01234969508270852,
+            -0.0038427767204741877,
+            -0.006947594635952202,
+            -0.01756894543880097,
+            -0.007890897373987378,
+            -0.047246698568222466,
+            0.009617876521097624,
+            0.027157555263862723,
+            -0.011355758309667893,
+            -0.04050311381480088,
+            -0.01017774338301804,
+            0.02437257208967103,
+            -0.004056329295095018,
+            -0.008345450168215274,
+            -0.001599171409135998,
+            0.018358386463243014,
+            0.0027207445181251406,
+            -0.007292967749349572,
+            -0.011267861428599221,
+            0.01657507035043879,
+            0.015105061280816214,
+            0.021419854579631276,
+            -0.02088952706825817,
+            0.006295984081511286,
+            0.020973020776936996,
+            -0.03073246543555915,
+            0.0023720435876790876,
+            0.03363966216709148,
+            0.010412588613998134,
+            0.013023311559810034,
+            -0.0010217662933572685,
+            -0.0075567381078977215,
+            -0.008248818794295818,
+            0.011848200323549865,
+            0.005066463524567732,
+            -0.023174489297821385,
+            0.021823825355858556,
+            0.020368868756448753,
+            0.0026042856858509524,
+            0.003457067317891926,
+            0.0005539745115410357,
+            0.01062670844108589,
+            0.009948889803890108,
+            0.05100741556350918,
+            -0.006493714124814164,
+            -0.014021751055930108,
+            0.015063988102064609,
+            0.02488059237697814,
+            -0.0023499941119596228,
+            0.01087282522970853,
+            -0.02612218935931224,
+            -0.021423880914461062,
+            0.023389294741183817,
+            0.008967763132667892,
+            0.009637806238448432,
+            -0.010815706635624337,
+            -0.011390999462441815,
+            0.031750675158698435,
+            -0.01307829462211914,
+            0.00759749911213318,
+            -0.02169895957209634,
+            0.01387879409130081,
+            -0.008606362268863441,
+            0.032229658009209815,
+            -0.0009066257436946125,
+            0.011062436424260079,
+            0.028010486866468023,
+            -0.007835805879241447,
+            -0.019657486763625547,
+            0.003967480939987674,
+            -0.015577952617624695,
+            0.02188429826317828,
+            -0.020502758155233298,
+            -0.0002362500702658591,
+            0.0040935406015014265,
+            -0.010056462958039452,
+            -0.0057038083589231615,
+            0.011475170331848666,
+            0.0015345076181482267,
+            0.006359262977587827,
+            -0.020293014401106,
+            0.01369706440475074,
+            -0.003989482215141903,
+            0.007887824197823247,
+            -0.015925992045349828,
+            0.014370223626942766,
+            -0.02309139937510304,
+            -0.021808736033141787,
+            0.008142621179624918,
+            0.0108996933426044,
+            -0.007589050511458233,
+            -0.025971539510030145,
+            0.02250992080022228,
+            -0.024980833805349545,
+            0.024979008145354243,
+            -0.006622339940663538,
+            0.002566378488088469,
+            0.004913874740363343,
+            -0.008102103508732738,
+            -0.0028098905427027783,
+            0.0057865587191235565,
+            -0.0073509382400023995,
+            -0.014123756810537343,
+            -0.019453380788786282,
+            -0.04083629135586275,
+            -0.03177927047231389,
+            0.022927581786088044,
+            0.011557232599801421,
+            0.0017073832658476193,
+            0.02339552660545296,
+            -0.02247471449131386,
+            -0.017890866287500053,
+            -0.009775068417716798,
+            0.025184899522229628,
+            -0.04408541669037361,
+            0.012104255392615549,
+            -0.014669458760022245,
+            -0.0245258242481677,
+            0.006274252098230249,
+            -0.004161960147481513,
+            0.009299171724026455,
+            -0.010779421730862708,
+            -0.04713627671791054,
+            -0.005192684784377356,
+            0.02330975631183273,
+            -0.0036802876364168935,
+            -0.006197548344287773,
+            0.03048284650641829,
+            -0.0019503797069087845,
+            -0.005038254331399784,
+            -0.005524875459245342,
+            0.027783596017224226,
+            -0.024950083604372984,
+            0.010988398531346805,
+            0.008304937912856357,
+            -0.008485065895193593,
+            0.006572665697403556,
+            0.0011518846966102158,
+            0.011927348472282199,
+            0.015632588644277183,
+            -0.015559413716826264,
+            0.02262990861928957,
+            0.011446850286896198,
+            0.006506075777765303,
+            0.008015225053785824,
+            0.00010054430757843335,
+            0.0022975742216194894,
+            -0.024948925449668525,
+            0.026047924967851216,
+            0.01888211557921096,
+            0.0001632181017876392,
+            0.012553094476616919,
+            -0.015042621225242777,
+            -0.02782009986290425,
+            0.02606646297992026,
+            -0.016847653820092335,
+            -0.00045272379465939274,
+            0.020053091486344633,
+            -0.016210036358455773,
+            -0.007930550661303182,
+            -0.01123514849773852,
+            -0.0162675150307978,
+            0.01803076559921804,
+            -0.012160217543572069,
+            -0.02216594536057653,
+            0.002594675636753233,
+            0.016443123828746582,
+            0.02134480656850119,
+            0.026694575953313482,
+            -0.011470930697351533,
+            0.017768927048191052,
+            0.0008653408391607816,
+            0.001865765599795092,
+            0.025745954171649543,
+            0.01702677011968492,
+            -0.008474889109168202,
+            0.025347176686722313,
+            -0.0016686223698600755,
+            -0.002421921060670558,
+            -0.012265504084322075,
+            -0.0165214436458303,
+            -0.007545471138114208,
+            -0.015333235556146196,
+            -0.0051830106590765685,
+            0.000762746782664525,
+            -0.0018118022696726346,
+            0.10876576838038368,
+            0.0034948044540582626,
+            -0.011608460689239234,
+            -0.005870263500417206,
+            0.007510774308250375,
+            0.0023525201199670765,
+            -0.006846154764389575,
+            0.016082721103476375,
+            -0.018616626204086575,
+            -0.023593744079394,
+            0.004743481974389998,
+            -0.0009262286319469741,
+            0.00870122878944883,
+            0.02029523251993269,
+            -0.008608837818169835,
+            -0.0200567711498164,
+            0.007089477722191928,
+            0.005535149644041221,
+            0.0023640746516071774,
+            0.005256431824732927,
+            -0.0006118671321824933,
+            0.008209920240361308,
+            0.02448422122961942,
+            -0.015622892878860465,
+            -0.029770836607612903,
+            0.010535371601298532,
+            -0.00015994156470228762,
+            -0.007193425789067776,
+            -0.011046316463797128,
+            0.014728841492009627,
+            -0.0011488451990980268,
+            -0.02751404133172324,
+            0.017663132712536528,
+            -0.019055335178900415,
+            0.014643246200933506,
+            0.0025348345163347464,
+            -0.00016387173410118788,
+            0.022565953069005714,
+            0.013101158251834834,
+            -0.04272085076882881,
+            -0.0034442020309914823,
+            0.01041205537746844,
+            0.011059422927946552,
+            0.0116446016862942,
+            0.0012992535381387887,
+            -0.018670801241345514,
+            -0.011788984490283379,
+            -0.009109849103208542,
+            -0.021268488414296117,
+            -0.012523033785938503,
+            -0.011393853408628214,
+            0.010743646386439633,
+            0.0012964030814325153,
+            0.023809566099779113,
+            -0.003030535542277875,
+            -0.006526351867141901,
+            -0.000430699447948508,
+            0.00880561288237257,
+            0.009694646009259077,
+            -0.02079011138439714,
+            0.008294338719512953,
+            -0.01823922789220214,
+            0.011819769685519291,
+            0.006416044253693268,
+            -0.022799568832306067,
+            -0.009911619276012527,
+            0.007152786695893951,
+            0.008393677195358034,
+            -0.013569953300860946,
+            -0.006503988731872938,
+            0.02299498754558268,
+            0.018870619152710703,
+            0.012233964644738905,
+            0.024994616203196145,
+            0.03838969722235787,
+            0.005744255912479976,
+            0.027682123067526515,
+            -0.016791471092091604,
+            0.008459745383680297,
+            -0.022493868174622695,
+            0.012902262638928338,
+            -0.00460399519266689,
+            -0.0050431343566427886,
+            -0.018188192498411616,
+            0.033765105245412236,
+            -0.009159751849801773,
+            -0.0027150644587327763,
+            0.034778224828120714,
+            -0.0066502021793471075,
+            0.004525230911930458,
+            0.005096748531775903,
+            -0.016067049494073893,
+            0.033274672789154185,
+            0.01686831040872425,
+            0.016353966592947262,
+            -0.021631941883929817,
+            0.011748317746828848,
+            -0.005675701689448899,
+            0.0001273442591116645,
+            -0.020738819037390263,
+            -0.0037528323797385733,
+            0.0027725541343763104,
+            0.0059860509282538546,
+            -0.016556134518370437,
+            0.02033623476757896,
+            0.00019647023567641008,
+            0.0052895744489259875,
+            0.03253462247073863,
+            0.01043311189804924,
+            0.019182938878855655,
+            0.010051051068017807,
+            0.019523739618278902,
+            0.0281939391326836,
+            0.025685787482623124,
+            0.0029932688462254147,
+            -0.018668945726139946,
+            -0.0009326901784812825,
+            -0.007880352863625262,
+            0.010492063055849727,
+            0.022849902214818293,
+            -0.022619228461640748,
+            -0.02772644874135741,
+            0.018566981184526675,
+            -0.0022608185345048414,
+            0.020031103834122595,
+            0.007544464625538984,
+            -0.020935181884939796,
+            0.003407355515103286,
+            -0.023178230166337178,
+            -0.0005418155871620593,
+            -0.031039187720144967,
+            -0.028591174077243604,
+            -0.0007659241904525392,
+            0.011945344534674635,
+            -0.01688153017778311,
+            0.03357788163306365,
+            0.017667778671101954,
+            -0.014289109343082812,
+            0.01604684528817524,
+            -0.011054587325539086,
+            -0.0036959091138078683,
+            -0.005819853870004682,
+            -0.00203755010992417,
+            -0.02157944474979592,
+            0.005138614763345542,
+            -0.0033509138150859593,
+            -0.0024871060745482883,
+            -0.016936575573598154,
+            -0.021901830366872595,
+            0.016375173843894617,
+            0.016742770709641697,
+            -0.006606252542838758,
+            0.039771905404651496,
+            -0.004238071340064403,
+            0.00018364315349545657,
+            0.011690521707403059,
+            0.0010018392078563795,
+            -0.013651921068215563,
+            -0.010829994512513501,
+            0.012362647379788833,
+            0.011321403447882543,
+            -0.018942043512252663,
+            0.005763773608775982,
+            -0.009131565038297889,
+            0.002089216671850269,
+            0.005438887787329238,
+            -0.008183835275629478,
+            0.006690280329947951,
+            -0.01673229763262922,
+            0.015833397876364723,
+            0.00295474085300386,
+            0.003186554859191664,
+            -0.011797859525926772,
+            -0.020991927615680524,
+            0.01576413532776261,
+            0.0032403514518527287,
+            0.018819824905201635,
+            -0.026755218746060662,
+            -0.00012946467986739473,
+            -0.009552681500884924,
+            -0.0028941357661144826,
+            -0.012533800273548772,
+            0.013975640628890588,
+            0.016864739319953755,
+            0.0063941926578529,
+            0.01781016436903586,
+            -0.0026268224264609724,
+            0.015368284773773678,
+            -0.018122416427266218,
+            0.008686009180697955,
+            -0.006801074032721299,
+            -0.018537521789181048,
+            0.0013237338264105415,
+            0.027863741654435434,
+            -0.004670012662973029,
+            0.011066853115455966,
+            0.011704884649534624,
+            0.018954056680143452,
+            -0.0012626215734593707,
+            -0.0030850559460101866,
+            0.015610404665171367,
+            -0.005838226405658572,
+            -0.006861883833171456,
+            0.020872966233916948,
+            0.032951653763679145,
+            -0.006716215648910268,
+            -0.0021292599951407863,
+            0.010886596327481677,
+            -0.004901684487088035,
+            -0.008644584974417106,
+            -0.009587037829603014,
+            -0.03158137900717491,
+            0.024369089446706894,
+            0.012092412439477474,
+            -0.009601042951586109,
+            0.022690862146548786,
+            0.020752792552278052,
+            0.011412743968296708,
+            -0.010471860708195726,
+            0.02291536038589634,
+            -0.01268035142112569,
+            0.024168869865226136,
+            0.032482831775271157,
+            0.004400973556358811,
+            0.0022312906286559167,
+            0.0021175489661843985,
+            0.01691348460546639,
+            0.006050357346231155,
+            -0.007068133214675996,
+            0.011656003476409986,
+            -0.018487923180009552,
+            0.01598530891381001,
+            -0.004432366596808033,
+            0.02808371007118948,
+            -0.01337145216499331,
+            -0.010473700033368562,
+            -0.00537325376270318,
+            0.00010726431372328729,
+            0.006445701087575359,
+            -0.003324002570256215,
+            0.005255443653813165,
+            0.0007072577303252132,
+            -0.017825649146944237,
+            0.003050128130393781,
+            0.004825967170196209,
+            0.009018799597337101,
+            0.004103653497834134,
+            0.007249957401824204,
+            -0.012360355929483,
+            0.01441885664789707,
+            0.004401820363035808,
+            -0.0026190938717927086,
+            -0.0007163225760398784,
+            0.011124538295057815,
+            -0.013232822471919533,
+            -0.004895624967029031,
+            0.005281228035321008,
+            0.0059275753353475414,
+            -0.007507924998741734,
+            -0.004484140989405051,
+            0.011908076549392969,
+            -0.0037917291321648483,
+            0.006351543504726163,
+            -0.004409464890606014,
+            -0.0037760571637986827,
+            0.005303661313434895,
+            -0.00974511544836468,
+            -0.004204167898342997,
+            -0.0007093125445559502,
+            -0.006869250680233214,
+            0.00786202375969793,
+            0.005971033227117194,
+            0.010606085272785863,
+            0.006969437776542303,
+            0.004293129058563288,
+            0.004114452739123398,
+            0.004071017374828283,
+            -0.002445681623750424,
+            -0.00592480769363354,
+            -0.004617501328052899,
+            -0.012067458701076778,
+            -0.008335239153829813,
+            0.009025028998949398,
+            0.008653779421856954,
+            -0.019363018792788574,
+            0.02048011578221327,
+            -0.001188320960771659,
+            0.00772165044232104,
+            -0.01602086976290928,
+            -0.000907553574811279,
+            0.005464880629734941,
+            0.008267702703609528,
+            0.016136762227127184,
+            -0.03654209526492064,
+            0.005179791231187316,
+            -0.0037321883493524766,
+            0.016026013709955444,
+            -0.008215703411373182,
+            -0.006979816286354862,
+            0.02564060526656963,
+            -0.01450763388985563,
+            -0.0009847758228920171,
+            -0.016142304341196283,
+            -0.01407778890808498,
+            0.010815725252959506,
+            0.02050940280233426,
+            0.020684808835848383,
+            -0.0004965802897137407,
+            -0.0005781445948759235,
+            0.00036179841583004193,
+            0.022775804864342608,
+            0.015545681427566235,
+            0.018697154644607316,
+            -0.004543500782944068,
+            0.007028436664826706,
+            -0.023057215501463973,
+            0.009912581556496658,
+            0.0011795417129232596,
+            -0.008215027850908477,
+            -0.008554137076267468,
+            -0.008703378564520125,
+            -0.010448538943182258,
+            -0.011773811327190638,
+            0.004106616839960258,
+            0.013895013678116032,
+            0.004190549543250594,
+            0.0031571190600981074,
+            -0.0225385788570314,
+            -0.006243193914542516,
+            -0.011229641377928767,
+            -0.01674214043453308,
+            -0.007367389908398082,
+            -0.02386334969150212,
+            -0.028021339684039734,
+            0.0002794207370355175,
+            -0.02411861258809571,
+            -0.015958336406924612,
+            0.005268815864773021,
+            -0.01546610225139129,
+            0.00898600137695074,
+            -0.008396481690803911,
+            -0.011873486990732417,
+            0.0074743033679134105,
+            0.015336444788692,
+            -0.017676977079590463,
+            -0.010165317702640217,
+            -0.01690524990506033,
+            -0.013460451917691695,
+            0.010918550283871637,
+            0.00595960977280955,
+            0.009250157926438213,
+            -0.012447979337850305,
+            0.0062932435120863365,
+            -0.012673905639312696,
+            0.015073013552721701,
+            -0.03769166067890086,
+            -0.029992249463405965,
+            -0.00014990754540999537,
+            0.007288036630677436,
+            -0.0020735742042505426,
+            -0.018695120921334806,
+            0.0003860116505754868,
+            -0.003608995261360232,
+            -0.014900191335144973,
+            -0.013866216738266176,
+            0.025251475305524053,
+            -0.006788483107616133,
+            0.01223227424986397,
+            0.0034937277040015626,
+            0.03438333560343413,
+            0.008803852645562129,
+            0.014666206816302367,
+            -0.005709421955160623,
+            0.003742433643393657,
+            -0.0033706258426092744,
+            0.006906464031352315,
+            0.0076578888497945185,
+            0.009276592191710524,
+            0.008025321004305927,
+            0.004576470294889512,
+            0.002887346543823031,
+            -0.001925468214331082,
+            0.02125241545487307,
+            0.025671028960818562,
+            -0.00037467906648160044,
+            0.008981063192139038,
+            0.0009387900084090577,
+            0.0002735789978680719,
+            -0.008374290340790827,
+            0.006547013284767913,
+            -0.008274637814401893,
+            0.004041066530254948,
+            0.012897549078838394,
+            -0.009150863940242148,
+            -0.010211568689323908,
+            0.016575088843292566,
+            -0.012380099999574783,
+            -0.03420455578618604,
+            0.026550654680719987,
+            0.005324512466403649,
+            -0.0035670106106754245,
+            -0.010657928468872608,
+            0.0037176027853158167,
+            -0.01103891334887315,
+            0.008096915671982816,
+            0.022587659771132487,
+            -0.0013527917944322358,
+            -0.0018085171207746743,
+            0.0038503794877980586,
+            -0.011870484733648887,
+            -0.007881489342351292,
+            0.006462772034381968,
+            0.0315696514747912,
+            0.01326417653779577,
+            -0.020400369539140532,
+            -0.015223808971473513,
+            0.020406862349006738,
+            -0.023142176712111575,
+            -0.006254959806032159,
+            0.009039506093607052,
+            0.0009087520492922078,
+            -0.022766046838415165,
+            0.0009662037093312435,
+            0.004701966191595815,
+            0.005946458676746661,
+            -0.006358986495019584,
+            0.03237532341275415,
+            0.001075992257265679,
+            0.015892020271402903,
+            -0.017189453434945788,
+            0.0017514632185789299,
+            -0.014189278671121677,
+            -0.014265447388465805,
+            -0.029809460132405256,
+            0.0005557491671071693,
+            -0.0030324198417091704,
+            0.011038984948790744,
+            0.024808067315720335,
+            0.01145103533194245,
+            0.014432246996007246,
+            0.010552852947578788,
+            0.01513689173494211,
+            -0.013653722180565311,
+            -0.029009757850432732,
+            -0.005171522291984226,
+            -0.018001048743500243,
+            -0.02297482628212251,
+            0.003962613844063587,
+            0.013645125285548414,
+            0.006278662690346082,
+            0.023786381386575035,
+            0.0021738516691483163,
+            0.0009580591750940011,
+            -2.690805459518147e-05,
+            0.008894091070322921,
+            -0.004952203441302471,
+            -0.006553986197076102,
+            0.003570189040872711,
+            -0.000652714137223501,
+            0.021228360881461853,
+            -0.005758946428212041,
+            7.897153237591187e-05,
+            0.014242175492137011,
+            -0.005622256348934008,
+            -0.0035009192529791146,
+            -0.017339926143652377,
+            0.006248842389703505,
+            -0.01622011586976866,
+            0.025783086900871728,
+            0.018450783573486916,
+            -0.009109295293546267,
+            -0.013469521573468915,
+            -0.011259555201313116,
+            0.02773097452736661,
+            0.004631282554715908,
+            0.010787055220617197,
+            0.018228322680582464,
+            0.01461254259249465,
+            0.01784830193964943,
+            0.004664725683522323,
+            -0.0034339781146649384,
+            -0.0007692550982545399,
+            0.021176646860433974,
+            -0.014700178590249292,
+            -0.002701208084390198,
+            -0.008842788249773623,
+            0.0021799005046816444,
+            -0.008431299000066363,
+            0.006086447848362669,
+            -0.010793016137040172,
+            -0.015251381840165954,
+            0.0189503457652375,
+            -0.02660767353735015,
+            0.016423753600066336,
+            -0.00593016170680532,
+            -0.00482891500342298,
+            -0.027459214027897378,
+            0.010728039528784636,
+            0.0027866316191608297,
+            0.0022519974195000687,
+            0.010770472550461752,
+            -0.021427634123401408,
+            -0.01867717541189363,
+            0.008532444554010673,
+            -0.011457117350255678,
+            -0.007152301670244725,
+            -0.00348619742388509,
+            0.0154715607296319,
+            -0.00017898955187849443,
+            -0.009925365923862246,
+            0.02085093731060963,
+            -0.007175309371250024,
+            0.01223979370120031,
+            0.005256127464980655,
+            -0.008756077893734508,
+            -0.0028438320920811362,
+            0.021022708898898203,
+            0.027557382862284808,
+            -0.008342206702778956,
+            0.014526066580760486,
+            -0.011883797998224506,
+            -0.01088335352319397,
+            -0.03479444009382274,
+            0.030104838505937447,
+            -0.013305732257226255,
+            0.026773747582411633,
+            -0.0006968265979630097,
+            -0.017037729111584094,
+            -0.016206207453228583,
+            0.01625225876515654,
+            -0.0024290584023212254,
+            -0.009366287391027166,
+            0.01465885301870365,
+            -0.02428628495473643,
+            0.0058336571592899735,
+            -0.0006627935593670669,
+            -0.011531762838191602,
+            0.013275508677482622,
+            0.0019223252930710324,
+            0.010738622785461513,
+            -0.02030405805119001,
+            0.0009661595529028194,
+            0.009263716373456188,
+            -0.016791855691574455,
+            0.006057830565306032,
+            -0.03730952786847912,
+            0.0029341721588188943,
+            -0.014833895684348656,
+            0.005725615593914644,
+            0.001498755253963699,
+            -0.01820708352937343,
+            -0.03652692232407375,
+            0.00043665701245599497,
+            0.01913883585802903,
+            -0.007975988799790905,
+            0.020538595326175857,
+            0.008243569241768276,
+            0.008421404577931517,
+            0.002314738067443065,
+            0.0016702403734608475,
+            0.008279538988821465,
+            0.03679456861213399,
+            -0.01775927634148114,
+            -0.004836606118958108,
+            -0.0012151329859941935,
+            0.010644195708319258,
+            -0.02028470571116305,
+            -0.009287515726906219,
+            -0.0036506912311261536,
+            -0.013134467570201307,
+            0.030615862647585222,
+            -0.020265555441523482,
+            0.025634449009756734,
+            -0.009300770899023007,
+            0.009274072905190077,
+            -0.02097685009125915,
+            -0.009797008690754956,
+            -0.010661540100131183,
+            0.0019532291898233895,
+            0.013465005282849381,
+            -0.009151430285942593,
+            0.0071467934925125145,
+            -0.006655608635025676,
+            0.01020327561163374,
+            -0.001472597695707201,
+            -0.02443508800166349,
+            -0.0024998811551575563,
+            -0.01107321258077183,
+            0.02456164421154801,
+            -0.0019619696941303388,
+            0.0005043179699158694,
+            0.010951580163871948,
+            -0.00853593956432799,
+            0.023927336608555987,
+            -0.008855702280766362,
+            0.0010093679746665581,
+            -0.004863269000020324,
+            -0.004005691415117686,
+            0.001698212044518236,
+            0.01822024022650613,
+            -0.005707404715849356,
+            0.03722587302228139,
+            -0.01708925706528118,
+            -0.0065500772743641925,
+            -0.00015538686795361824,
+            0.010852925330001283,
+            0.010035359503147549,
+            0.008754719437760327,
+            0.004067820167375348,
+            -0.014861417626433965,
+            0.009346669946626484,
+            -0.020136029447262147,
+            0.007501669351583119,
+            0.023178490169347605,
+            -0.006840359541885465,
+            0.0006676450965357345,
+            -0.01283320739092204,
+            -0.015610538827742276,
+            0.006308543164022682,
+            0.002329420005586303,
+            -0.002295739637504597,
+            -0.02181189272979859,
+            0.006298804075737111,
+            -0.014398470126723037,
+            0.004007419942728446,
+            -0.0007497656993578918,
+            0.013863867106289847,
+            0.016451166273826513,
+            -0.006740209707090658,
+            0.039292260347522444,
+            0.01871723939316731,
+            0.009109005310505462,
+            0.005674782595856756,
+            -0.040588444759736254,
+            -0.008993850998154852,
+            -0.0010197185016393919,
+            0.004866150984083949,
+            -0.01141354216871663,
+            -0.018944881712855702,
+            -0.002646110151132781,
+            0.000308585036997592,
+            -0.010681954707313127,
+            -0.009091585604191968,
+            0.01751599002710486,
+            -0.006034424764302979,
+            0.00872057856926618,
+            0.004146961781109887,
+            0.023758065635807138,
+            -0.009727802672285544,
+            0.01176257119851645,
+            -0.0071822083765114165,
+            -0.005456197751568526,
+            0.0006726765703478756,
+            0.011011628907288813,
+            -0.008031792262886802,
+            0.009763340867550022,
+            -0.004279891138185881,
+            0.008355720233269893,
+            -0.008552596384067825,
+            -0.023952129458400563,
+            -0.014972827087541732,
+            0.014347535404251018,
+            0.011651260496709961,
+            0.008137738462704715,
+            -0.013624435569539285,
+            0.0031728721674035352,
+            0.00675830277744976,
+            -0.009184784787581942,
+            0.009161771311910335,
+            0.017733082848622708,
+            -0.001411542568472906,
+            -0.0017830167499636113,
+            -0.0016130003349297027,
+            -0.012173528776899308,
+            0.0011196437753239162,
+            0.02284887358189995,
+            -0.003054479643937998,
+            0.014614283913976716,
+            0.010689486748036806,
+            0.009643292262102645,
+            -0.00810930837603355,
+            0.001915955511174678,
+            0.02341863048264366,
+            0.015022291783105098,
+            -0.004625715884342598,
+            -0.011422973894623295,
+            0.004259757717597968,
+            -0.010332744704689279,
+            0.007373328473086509,
+            -0.02117940113144938,
+            -0.004672698999328004,
+            0.008896311380763048,
+            -0.007028862917603513,
+            -0.01732432285099198,
+            -0.028026173555055017,
+            -0.016765805892242715,
+            0.019819329380181,
+            0.0027213760678977965,
+            -0.013291251876048186,
+            -0.0032919084721662363,
+            0.0031214883337300926,
+            0.002553005168048115,
+            -0.0071104806975682885,
+            0.005310583052470149,
+            0.007359183453835734,
+            -0.008674133980486269,
+            0.017665958608281126,
+            -0.000521269431423141,
+            -0.0008020251454616109,
+            -0.0005442374261144659,
+            0.0006127687598711624,
+            -0.016387752806126442,
+            0.01666161912459431,
+            -0.010949228376419892,
+            0.015198110119816624,
+            -0.008966726991404666,
+            0.008600499511690635,
+            -0.007617043351740469,
+            0.009458641139070436,
+            0.011597957400445055,
+            0.012755037944648426,
+            -0.004403680820063604,
+            0.008519241473418196,
+            0.004807816191718704,
+            -0.008147125240985166,
+            0.007017601015398136,
+            0.01296082929962557,
+            0.00582353932160707,
+            0.0013634237970372698,
+            -0.0008246894322793264,
+            0.0022035670958705576,
+            -0.002480976219285683,
+            0.0056293125704658135,
+            0.012714850523393719,
+            -0.002497639449055006,
+            -0.007135683201188999,
+            0.014988249771676699,
+            0.014955895630183352,
+            -0.022994948923833993,
+            0.007121634434082677,
+            0.005835971825639792,
+            -0.0030507120821557008,
+            0.002145266953387315,
+            -0.027334525318320587,
+            -0.020359474070022213,
+            0.01901995970350808,
+            -0.015260454474605018,
+            0.02850460517419317,
+            -0.011624138792780823,
+            -0.02284242513728605,
+            -0.01277489645734181,
+            -0.004209529633534193,
+            -0.022684500849916897,
+            -0.004894414098943742,
+            0.01919165657366256,
+            0.013401560972810082,
+            -0.004313549020265677,
+            -0.019089369171102138,
+            0.03863248408555474,
+            0.010497951840980463,
+            0.03508157919687289,
+            -0.008597970944278045,
+            0.0023354831464448313,
+            -0.01563401808449416,
+            0.015551176593436512,
+            0.001075882802133782,
+            -0.011575692834796734,
+            0.01505716108583101,
+            0.031836226915668404,
+            0.0034740889237207494,
+            -0.00617206855371193,
+            0.0018420207148235309,
+            0.00418433325800744,
+            -0.0008167766650900381,
+            0.011160989178055499,
+            -0.018497472318034093,
+            -0.011951262348399922,
+            0.0013093376768780122,
+            0.00216347903176653,
+            0.0059233460419514045,
+            -0.001778091863744448,
+            -0.013317639704511549,
+            0.011433308892643204,
+            0.011596810919469926,
+            -0.01518143105525228,
+            -0.007497373860338335,
+            0.0050957665991362405,
+            -0.018974405317819656,
+            0.01872472902494418,
+            0.011859522398789263,
+            0.000792413325332956,
+            -0.0006503947129906383,
+            0.00808635755624928,
+            0.003749813548995223,
+            0.0033657799335872368,
+            -0.007862215303617144,
+            -0.0028014669075116876,
+            0.0009892310856576799,
+            3.834379772674841e-06,
+            0.015402211364447726,
+            -0.015264893397161102,
+            0.02488855366696587,
+            -0.0025643639525691083,
+            0.00953993868921819,
+            -0.017460590918773188,
+            -0.0017911712931080155,
+            -0.00402374787529868,
+            0.01838051670627292,
+            0.006753072393532114,
+            -0.010130676216186492,
+            0.004921510943896829,
+            0.007975201081296845,
+            -0.012589786869771252,
+            0.00018319054705558588,
+            -0.0022606180909706688,
+            -0.014791528072476771,
+            -0.0031171076370692706,
+            0.017873825561816544,
+            0.007593688317551089,
+            0.0037375070591255375,
+            -0.015863143964928172,
+            0.007192324815779268,
+            0.007522990667369163,
+            0.024504875417877527,
+            -0.026309816223960274,
+            0.024779042752840836,
+            -0.022104087334088543,
+            0.02165781034613893,
+            0.0319051741666687,
+            0.007538146369110045,
+            -0.01690181186708103,
+            -0.006595947249169521,
+            0.006373991305923002,
+            -0.023410814842488123,
+            -0.005494017994983188,
+            0.016903736810723716,
+            -0.01792400152372258,
+            0.000330865061689199,
+            -0.014660413003063089,
+            0.0040825880154611895,
+            0.008248037865838212,
+            0.011540333451799686,
+            -0.0033578830043738085,
+            0.018327525495676525,
+            0.020171745352159914,
+            -0.01138759854371235,
+            -0.0004221149053107702,
+            -0.022321409200941777,
+            0.005046865873981027,
+            0.022509643273449936,
+            0.001111272240506322,
+            -0.009540467231190513,
+            0.018376261423745193,
+            0.008480532791922674,
+            -0.020801891594934313,
+            0.008436374227753589,
+            -0.036175982655998785,
+            -0.010454118523313122,
+            -0.009328507118928412,
+            -0.001218633166683384,
+            -0.004142773602181115,
+            -0.0002494371580772723,
+            -0.011365548842513265,
+            -7.699548642315061e-05,
+            -0.006889339420877534,
+            -0.0037173070609413474,
+            0.014953978004522174,
+            0.011678957032329096,
+            -0.003501047405385249,
+            -0.0073984932581048915,
+            0.008493128567102267,
+            0.006163937693564917,
+            -0.017034696006318337,
+            -0.013268418502042576,
+            0.0037122128689059085,
+            -0.020067656401141467,
+            0.005148329575092478,
+            0.00459729178556505,
+            0.014031669598917171,
+            -0.01849235806598133,
+            -0.009298127665365717,
+            0.019716512208448042,
+            0.0094466405158553,
+            -0.0008988916054855498,
+            0.004264260983102956,
+            -0.00032312548009733906,
+            0.002279987803645912,
+            -0.0008902345374346296,
+            -6.145528806973222e-06,
+            0.003602229465148649,
+            0.012503366799381277,
+            0.007759393855482671,
+            0.03694402922447051,
+            -0.016692148708641812,
+            0.005873765863350649,
+            -0.014187609702966002,
+            -0.0037592517335595654,
+            -0.00749939010396784,
+            -0.0004308157309481927,
+            -0.0022282234382577843,
+            0.0021999996319642854,
+            0.0052176773733353925,
+            0.009353171029508977,
+            0.002155405440765206,
+            -0.00823565584692709,
+            0.009691447250255694,
+            -0.015228173095533029,
+            0.003853858061067531,
+            -0.008207910291425473,
+            0.01017972550354623,
+            -0.009695280827515048,
+            0.0028705587799331246,
+            0.03849358164875786,
+            0.01130021300937487,
+            -0.004934175853966098,
+            -0.014300147534361459,
+            -0.0007796051618993811,
+            -0.01737877501558196,
+            -0.008579119859817895,
+            -0.005434792488602325,
+            -0.02617338653902575,
+            -0.00838916510137031,
+            0.015344260464413648,
+            0.011202567142310492,
+            0.014124853863170132,
+            -0.023894182185855824,
+            0.008897943489322673,
+            -0.03001670823127986,
+            -0.011951507895140876,
+            -0.001530967140156163,
+            -0.015467812535428609,
+            0.0010110999991167703,
+            0.023623196504289838,
+            -0.007762735863648054,
+            0.012654652822750325,
+            0.012867958170258448,
+            0.018777522196571934,
+            0.006390986057157016,
+            0.00038045736974183937,
+            -0.009444911783760569,
+            0.009534310165417939,
+            0.008806031651319704,
+            0.0080954543713255,
+            0.013161833638467553,
+            -0.0022840529693446553,
+            -0.018060162439023572,
+            0.005469288590000856,
+            0.014354475778235131,
+            -0.0033504276914600585,
+            -0.001615191819057537,
+            -0.011088493180661976,
+            0.016039040565630432,
+            -0.023559046333872156,
+            -0.009482398794557168,
+            0.009918576960319548,
+            -0.0027308309155029842,
+            0.009212063547988036,
+            0.0160181200081126,
+            -0.01881373001297218,
+            -0.0024168618343039703,
+            0.016244889977796682,
+            0.0024104546308670025,
+            -0.00022606425578303413,
+            0.016792244238218245,
+            -0.0007616857360728623,
+            -0.004279120886828465,
+            0.00997764266310123,
+            -0.0075631917138318655,
+            7.3878137053128405e-06,
+            -0.002416364915008295,
+            -0.0028253498168087964,
+            0.02054128727706496,
+            -0.021281801678871736,
+            -0.005379259148621041,
+            0.02257306103533458,
+            0.015687105622685975,
+            -0.007843745111084413,
+            6.258498042855072e-05,
+            -0.0219506172079994,
+            0.0019925886885244454,
+            -0.007768133999661569,
+            0.016524850592507592,
+            -0.007957949830060354,
+            0.0030085044673901665,
+            -0.01361024620290394,
+            0.011563781350927133,
+            -0.01277407169098418,
+            0.00873716215529134,
+            -0.002986943948333218,
+            -0.017567646684567492,
+            -0.0053109390854215125,
+            0.004561236136619266,
+            0.0014045175130801308,
+            0.044958041347040185,
+            -0.005539136232703934,
+            0.003729517522976541,
+            0.018086335927383804,
+            -0.014199453349727933,
+            0.02079719360289355,
+            0.016663293302592568,
+            0.00991445855219406,
+            -0.00033967370741635794,
+            -0.0018726748849858836,
+            0.023373354806639515,
+            0.013258226441642464,
+            0.00861986998150065,
+            -0.0014340218811281975,
+            -0.008516541227326329,
+            0.018239230461345617,
+            0.005458629943667832,
+            -0.012421896028975013,
+            0.012732721349758832,
+            0.013559877132508506,
+            -0.013720939472355347,
+            -0.011545210834395917,
+            0.01616358223872008,
+            -0.019904594319155373,
+            -0.018260016571763644,
+            -0.0036124537591368864,
+            -0.01614510591132948,
+            -0.03864699384205461,
+            0.01619401261888991,
+            0.018351607973242735,
+            0.006666313000123845,
+            0.013647410298896132,
+            0.002597680061427196,
+            0.004545530267244157,
+            -0.018531082430861854,
+            -0.0077618625909995275,
+            -0.008085144524135461,
+            0.003744735923004248,
+            0.011059457364156674,
+            -0.011335432904114519,
+            -0.009514379703581616,
+            -0.00034500702695418285,
+            -0.007433132060989644,
+            0.01971420270994951,
+            0.0020753610290475642,
+            0.0034506706065161024,
+            -0.00628437298309269,
+            0.017966485854173167,
+            0.026789678964152908,
+            -0.007532446825077476,
+            0.021886182869092073,
+            -0.002864230253056434,
+            -0.0015870360120997492,
+            -0.014393519359298496,
+            -0.0014439412842964876,
+            -0.011527182213472351,
+            0.0005453493437517964,
+            -0.0006791481533636673,
+            0.014093767997869645,
+            -0.03288824642147967,
+            -0.003957301296796813,
+            0.014444592778851293,
+            0.007538052312876599,
+            -0.005057755472988024,
+            0.011357919882949903,
+            -0.002964451028327082,
+            0.016977317604413924,
+            0.0075201055190542146,
+            0.023391646079994784,
+            -0.016588823248397513,
+            -0.004504858318770565,
+            0.006660877064954993,
+            0.0034631255931793638,
+            0.0014484203909961972,
+            0.01912983041187014,
+            0.023439941258581887,
+            -0.014715886385460541,
+            -0.0013596123366001458,
+            -0.0029020372473879614,
+            -0.021414221343490594,
+            0.014976806123253727,
+            0.0074729481304182865,
+            0.010967351226114886,
+            0.009806663105020302,
+            0.015465076881663691,
+            -0.02109587527354609,
+            0.0011671121739822202,
+            0.020567458365051382,
+            -0.008618333549105003,
+            -0.018815706906171112,
+            -0.009804829158883123,
+            -0.00966828714719191,
+            -0.006078893678150302,
+            -0.00803783313451146,
+            0.005523023167028105,
+            0.0022647506733877573,
+            0.01652997604669998,
+            -0.008204200434441975,
+            -0.008404609147866811,
+            0.008489247125986078,
+            0.007546785673285454,
+            -0.0033093451283570316,
+            0.010614050963656496,
+            0.0030633179859393587,
+            -0.006940540318011839,
+            0.00894884343338777,
+            -0.002787816848430143,
+            0.014347598920245487,
+            -0.02441769157218451,
+            0.004717742147185283,
+            -0.012096876825337124,
+            0.006405151160675699,
+            0.0041568790527989905,
+            0.0016702066682613428,
+            -0.0041159211459004,
+            0.0035217362856754353,
+            0.0008893479210743541,
+            -0.07433312151926533,
+            -0.012629681414004524,
+            -0.0035797012285864486,
+            0.033657838713146884,
+            -0.019695806809053914,
+            0.004477927056336216,
+            0.011966065644382396,
+            0.0096717896523898,
+            0.003070713232538593,
+            0.006136726101157844,
+            0.006112385569942848,
+            -0.0245258242481677,
+            -0.00015284418977092615,
+            0.004821040694679944,
+            0.0020347004045943454,
+            -0.007532408630122127,
+            0.00750565079228405,
+            0.015327710245879296,
+            0.020989926895388523,
+            -0.010970305087289488,
+            0.017227663358885748,
+            -0.003352718496981928,
+            -0.006068892126250978,
+            0.0014285750322333686,
+            -0.010153799722483554,
+            -0.017307415843726394,
+            -0.006495008086641125,
+            0.0068983699213297075,
+            -0.001982568363667046,
+            0.0044032687939897855,
+            0.003333596495330213,
+            -0.0032171586993948696,
+            0.0072365127663633925,
+            0.013920326463272646,
+            0.006415916101287133,
+            -0.013675562796300214,
+            0.003003782758807665,
+            -0.02500375035312058,
+            0.011945988879567646,
+            -0.012708975181025296,
+            -0.0009488161508767279,
+            -0.0047093877027523014,
+            -0.0035750724825159503,
+            -0.002213435677949806,
+            0.02043567538125188,
+            -0.014304009033474186,
+            -0.01802899775650995,
+            0.008187227768196673,
+            -0.011217013195839762,
+            0.008300740149622635,
+            -0.03382831122769974,
+            0.012386335491333264,
+            -0.0276029934382802,
+            -0.0002243752790856292,
+            -0.004521387639042107,
+            -0.0010631721066129095,
+            -0.009491184157278669,
+            -0.03274559774619223,
+            0.015845705808609598,
+            -0.007204751554079551,
+            -0.020338337341085228,
+            0.0007143424225187802,
+            -0.021267094083674304,
+            0.014150458196361147,
+            -0.01851165553224022,
+            0.0033450353123933058,
+            -0.018775439467431685,
+            0.002782289821140401,
+            -0.0018711350216089556,
+            0.010502195116579441,
+            -0.00557814442809019,
+            -0.0007130962242755068,
+            -0.013072988887382269,
+            -0.002121841212381393,
+            -0.006653444103331213,
+            0.009394907690516121,
+            0.0028678074167728367,
+            -0.009601577734902932,
+            0.011153571864462314,
+            0.013046116892519414,
+            0.019812860521973938,
+            -0.00539047787915588,
+            -0.00555139958411929,
+            -0.011553689090179406,
+            0.009852063270798534,
+            -0.016326212706634428,
+            -0.00827402913960903,
+            0.008982303148984887,
+            0.015216947130173639,
+            0.01026889105233137,
+            0.003352028462252994,
+            0.003150801750574548,
+            0.002531620731961765,
+            0.026744691368233237,
+            -0.0015120532859309638,
+            0.0008324701960729416,
+            -0.014508988496009931,
+            0.022127525293249926,
+            0.02772430884375744,
+            -0.0012206102581726523,
+            -0.00018859639594030037,
+            0.017437360169132967,
+            -0.004697052522250805,
+            -0.0038111184371524645,
+            0.024015246959315593,
+            -0.038396108183338655,
+            -0.008416825232533771,
+            -0.006788816882617399,
+            -0.006043428198944214,
+            -0.009587366247924467,
+            0.007294190273465812,
+            0.00041966080769004275,
+            0.0024709692753547823,
+            -0.0005250904505954556,
+            0.03452858797518386,
+            0.003403515654095309,
+            -0.0003392060101496309,
+            -0.0032980864446307116,
+            0.0019125121221204029,
+            0.0032987517853214346,
+            -0.009116761755779728,
+            -0.038030085122772624,
+            0.006006334000396066,
+            0.007932785488877407,
+            0.00975009749364395,
+            -0.008120121513849434,
+            -0.008864461019882246,
+            0.004909348738798099,
+            -0.02273129283865203,
+            -0.01372636307167243,
+            0.02625229144362647,
+            -0.008523028986127237,
+            0.029005877324974557,
+            -0.022435059061182174,
+            -0.013135836801818153,
+            -0.004126433322528265,
+            0.010283975013795537,
+            -0.019732856462444506,
+            0.004343860338939202,
+            -0.004022387065239562,
+            0.023733721026005954,
+            0.0077578144626961245,
+            -0.008371631446146718,
+            0.012386735614153616,
+            0.0064789380436815714,
+            -0.0026638642827013837,
+            0.00136333052963848,
+            -0.007209517067748103,
+            0.016916505654836485,
+            -0.011819784026452944,
+            -0.005833909403097991,
+            0.008977527508585844,
+            -0.0024556434204852946,
+            0.02509293948918158,
+            -0.0237762245071987,
+            0.005288171058938098,
+            -0.00019047961064753207,
+            0.01822542471604037,
+            0.008127347078975925,
+            0.008361588667386882,
+            -0.011835407590060994,
+            0.010004482616697591,
+            -0.03378107067882622,
+            -0.015433906677849963,
+            -0.007702910250795191,
+            -0.0013877702264578865,
+            -0.002059152251387601,
+            -0.015745218548968338,
+            -0.026393221735795527,
+            0.006294583740809379,
+            0.016267525575472445,
+            0.023291528020980265,
+            -0.004683197930892445,
+            0.012912830736650997,
+            -0.002831583991769044,
+            -0.008904271394710672,
+            -0.011441882849296418,
+            -0.012202629605208118,
+            -0.012470773094383122,
+            -0.014560694782340029,
+            -0.017045701603711442,
+            0.012139665577738337,
+            -0.020079061996873537,
+            0.013911518328924777,
+            0.0347210775522539,
+            -0.017176114991595492,
+            -0.0018941772168839354,
+            0.019702722555622765,
+            -0.004221469612316137,
+            0.01212088353057276,
+            0.0063707489618353725,
+            0.015298014532351866,
+            0.0038643174573634725,
+            -0.012181479717826009,
+            -0.007374005740809251,
+            0.019411457664902376,
+            -0.004902591844710158,
+            0.014591446725717338,
+            0.015069599884557399,
+            -0.008675672164385891,
+            0.002544198867387345,
+            0.01951865904496276,
+            -0.01762101777798757,
+            0.01665654707715257,
+            -0.00458402229480694,
+            -0.009936206685538294,
+            -0.013345740581684638,
+            -0.005336416724426228,
+            -0.019469457091169776,
+            0.0061608454109892425,
+            0.018931676738452464,
+            -0.00533013594306834,
+            0.009310871143727683,
+            -0.0037670793158045,
+            -0.005045151384034014,
+            0.0002041909886640482,
+            -0.04152751596227068,
+            -0.008340911336217609,
+            -0.0006651584370005978,
+            -0.02505748815737022,
+            0.025954102004571416,
+            0.007205602377144382,
+            0.022602344303400163,
+            -0.017168610245491406,
+            0.004459735690564842,
+            -0.005882024209102714,
+            0.013217517324383834,
+            0.02255689669623969,
+            0.0017984057469785134,
+            0.005699128497215882,
+            -0.02353849721739205,
+            -0.0034711576718386185,
+            0.015924913374553738,
+            -0.003604478141616307,
+            -0.0034993903155063817,
+            0.012592308877056896,
+            0.012864938348003884,
+            0.01567418642605425,
+            0.008801753550506258,
+            0.015606443428689292,
+            0.010589933994371628,
+            0.008928721556432368,
+            -0.005805129999077126,
+            -0.001025103064751441,
+            -0.015907755521365925,
+            0.0045874330999415985,
+            0.031701049450012876,
+            0.012594483199773746,
+            0.016783858873835704,
+            -0.01778536268185936,
+            0.00789523134933153,
+            -0.0012706411671644011,
+            0.008325108613530198,
+            -0.001677173669891956,
+            0.029540136288412976,
+            0.011739999605699826,
+            0.015670966746894652,
+            0.008732334090105053,
+            -0.016237731209426752,
+            0.045334690225148,
+            -0.016911928942220152,
+            -0.0021139311364009113,
+            -0.01058970056572469,
+            -0.021705338531028618,
+            0.007324792557651635,
+            -0.006556582104686972,
+            0.0018243188317438144,
+            0.010780292738415671,
+            -0.006471850255151019,
+            0.0006997395548596146,
+            0.003907433192252573,
+            -0.023129313269763224,
+            -0.002550875642832235,
+            -0.02237917712764273,
+            -0.003107775639788617,
+            -0.02621608419628401,
+            -0.00012482032027919036,
+            -0.016207377503696517,
+            -0.0038712739995960753,
+            -0.023315508569288476,
+            -0.014963211090562272,
+            -0.01633110043829169,
+            -0.004241081909202262,
+            -0.008295402942465353,
+            0.004976396647108597,
+            -0.007402107755970892,
+            -0.020321297548927468,
+            0.0020986169430507757,
+            0.016803273183070004,
+            0.01434729320047051,
+            0.018612970083178984,
+            0.00603868809307069,
+            0.014278001198851581,
+            -0.0026350854954985117,
+            0.017171016209107656,
+            -0.006968968261909459,
+            0.006881847166565678,
+            0.011454001418121107,
+            0.02140935294963631,
+            0.011121833270235448,
+            -0.012801756637470525,
+            0.0010870041670931706,
+            -0.009655388034828094,
+            0.002094601818569396,
+            -0.00828486034340067,
+            -0.009190975158350585,
+            0.0017762581946855365,
+            -0.007261996868906865,
+            0.004674405202069334,
+            0.0030526253833799426,
+            -0.0014082232541965316,
+            0.010419302200840906,
+            -0.0024200053377751805,
+            -0.012208685875688886,
+            -0.0033037330470612695,
+            -0.017975940046938498,
+            -0.006958042903103732,
+            -0.0014496522238488239,
+            -0.011711502944602638,
+            -0.01016130418405875,
+            0.003321436189059608,
+            -0.00538282506552848,
+            -0.009353581612407435,
+            0.013884147453136417,
+            0.0001309738696853529,
+            0.0005878330075918333,
+            0.023495280602974504,
+            0.006035028203642483,
+            0.008087594627921345,
+            0.013144288945385904,
+            -0.01023995000994454,
+            -0.022574958230635715,
+            -0.0243435727055662,
+            0.007867022215096418,
+            -0.01573478573423379,
+            0.02324699908478915,
+            -0.007929809922043611,
+            -0.00561630876431978,
+            0.0020982723335057625,
+            -0.013699317926847062,
+            0.008578431496569186,
+            -0.011271381061801356,
+            0.0014450645788554738,
+            -0.019581451799757368,
+            0.018059311800267413,
+            -0.010917911869373693,
+            -0.004302687883018328,
+            0.009466103226841532,
+            0.034049086327052606,
+            -0.0005689605174080737,
+            -0.007410970644432269,
+            -0.006594150877740108,
+            -0.00022810271313786974,
+            0.018766765773409123,
+            -0.0029041815756733333,
+            0.005758781908059541,
+            -0.01563519107805783,
+            0.006571628287171105,
+            -0.006132372252825422,
+            0.0012423261344297225,
+            0.01513074831647933,
+            -0.026306320707079913,
+            -0.015491710256868008,
+            -0.007857352164317122,
+            -0.024893430640551818,
+            0.002253574511878983,
+            0.017579477884206534,
+            0.020505606775521504,
+            -0.020324505029926956,
+            -0.02513332780748072,
+            0.005150966733930881,
+            -0.026580994694344065,
+            0.0037390577082486594,
+            0.1241031878517593,
+            -0.002138404898309707,
+            0.002720723037908521,
+            -0.011425036886011179,
+            0.0022399461009316216,
+            0.005389407121650262,
+            -0.019552543955456567,
+            0.05950746473186985,
+            -0.00638023587296763,
+            -0.004023798021748334,
+            0.002170545188776296,
+            0.001785855390676113,
+            -0.005831534307818096,
+            0.008643501093778434,
+            -0.017602787281866816,
+            -0.02184792441891879,
+            -0.0052215705748838905,
+            -0.021252169339074325,
+            -0.008509621422071373,
+            -0.013000137524907565,
+            0.002893880203681282,
+            -0.015821609145923173,
+            0.010397692426614709,
+            -0.01638916243796692,
+            -0.010283706937701179,
+            -0.01103621454761223,
+            -0.00373552043529251,
+            0.01563553822905027,
+            -0.03188533919653949,
+            -0.004873550301425776,
+            -0.005183590054152726,
+            0.018218971950313273,
+            0.006004328765748325,
+            0.004357734370247252,
+            -0.014230318971458851,
+            0.0064792070801456235,
+            0.0215589173714926,
+            0.0055272493764401305,
+            0.014643335833970546,
+            0.010618070641357003,
+            -0.0022177303813575846,
+            -0.0024402761885758007,
+            0.017541799738501007,
+            -0.005445774414228888,
+            -0.00023502936119626386,
+            -0.00537569046685066,
+            -0.005974703247446444,
+            -0.010712244203425036,
+            0.01843613166635215,
+            -0.009938564643456672,
+            -0.017388592763277214,
+            -0.0015797412875366156,
+            0.02028434083164951,
+            -0.012761534382003973,
+            0.0031906812114950982,
+            0.000788436210275669,
+            0.020589196433948156,
+            0.0007726855314648301,
+            -0.005481260996289263,
+            0.007165014826294827,
+            -0.02956334397512344,
+            0.004095715232997753,
+            0.0003568512047626187,
+            0.027290768799658912,
+            -0.0017631129386615335,
+            0.01552542586826711,
+            -0.020871286597113327,
+            0.004388125824324094,
+            -0.016926008631435403,
+            0.005099899758322398,
+            0.012560923968841436,
+            0.002781049890810361,
+            0.010889021753593997,
+            -0.008500358897706648,
+            -0.021353460542554444,
+            0.012663934009161902,
+            -0.018657094130860155,
+            -0.029156017587920958,
+            0.0018889388761887053,
+            -0.028158076315864772,
+            -0.003048854609750186,
+            0.006812072971079798,
+            -0.035217412591558014,
+            -0.02251675180883659,
+            0.007396524512996277,
+            -0.00995571600201846,
+            -0.011912753666073882,
+            0.010675240017836214,
+            -0.02645155743041798,
+            -0.018137648527851668,
+            0.0010666944556449141,
+            0.012588800077165337,
+            -0.004561825939541096,
+            0.007555558090958502,
+            -0.028558603070876605,
+            0.014816816621445352,
+            -0.01313764261744744,
+            0.016036065063831813,
+            -0.0037797584923144134,
+            0.022217516257304868,
+            0.011131597210722573,
+            0.008795376259031849,
+            -0.004950080558710245,
+            0.021409590863550987,
+            0.005574351961953293,
+            0.051358937509801736,
+            0.031972009383746405,
+            0.0019821003238211276,
+            -0.003251257665378234,
+            0.003313045083311649,
+            0.008294776423730887,
+            -0.00940948257229233,
+            0.005114631360856872,
+            0.028053227377164806,
+            0.016738013668710083,
+            -0.018755924318109216,
+            0.006606711923035118,
+            -0.02702692774986498,
+            0.017950884766781056,
+            -0.022054653845163,
+            0.0003755944703013671,
+            0.007766253634286795,
+            0.0016206993755471613,
+            0.010682347575070117,
+            0.0013842930079185415,
+            0.01705234255261425,
+            0.023975191051210525,
+            -0.002976402980885077,
+            0.017466027107477605,
+            0.012734909275073793,
+            0.018143161773162015,
+            -0.0038399373197265827,
+            0.006929440921265761,
+            -0.012003095293274786,
+            -0.0008883470597690258,
+            0.0020490527756307267,
+            -0.004231091383919565,
+            0.012128848448049833,
+            0.008340302519160299,
+            0.012989849936937843,
+            0.002974488532907779,
+            0.003198854381461613,
+            -0.005309753760578859,
+            -0.0017024982049936384,
+            -0.01763395035509857,
+            -0.01819680226734474,
+            0.028417724274372468,
+            -0.020217879775485576,
+            -0.050118184195774004,
+            0.0025509548991664425,
+            0.014185761006969567,
+            -0.0012021669304388173,
+            -0.0051260576846881515,
+            -0.0005774778872927327,
+            0.005718864494530703,
+            -0.016918749210038116,
+            -0.0015472638835434902,
+            0.008125845907627804,
+            0.020347547062899797,
+            -0.002036218493206914,
+            -0.01079739518866248,
+            0.0038112958713600906,
+            0.02169339303105242,
+            0.01667594322285835,
+            0.012338193092567552,
+            -0.009798409233733167,
+            0.0003198215558702285,
+            -1.6308740825163337e-05,
+            0.0076870827874737325,
+            0.004356644356656038,
+            0.003500023005601613,
+            -0.019968549869439228,
+            -0.008634740687755112,
+            0.012781565963545847,
+            -0.0025637758121512927,
+            0.010243567793632335,
+            -0.00042296044205132426,
+            -0.003069251553118065,
+            -0.009344199449486244,
+            -0.004694953215957636,
+            -0.009583993744231791,
+            0.013783878036521561,
+            -0.007705982042252044,
+            -0.01549567538665377,
+            0.00020666298495823114,
+            0.0005515775451348472,
+            0.001667367278489879,
+            -0.004518431526236256,
+            0.03302324454129141,
+            0.009531990824506103,
+            -0.0004937348108340977,
+            -0.004047613286672229,
+            -0.0327383112522095,
+            0.012456150771224677,
+            -0.009677533214778067,
+            0.0006455811519981003,
+            0.020012976827426605,
+            -0.00020277620481825933,
+            0.002168432791818704,
+            -0.015657551629104767,
+            0.0032924357559238502,
+            0.006161295735841883,
+            0.013814165416788555,
+            0.0005164426482972658,
+            0.01076512534105537,
+            -0.006498977875968572,
+            -0.010325144017898257,
+            0.004804158067334203,
+            0.0104064855510473,
+            -0.0012188100892986933,
+            0.005171197790296134
+          ],
+          "title": "Mixture-Of-Experts With Expert Choice Routing",
+          "keyphrases": [
+            "Mixture-Of-Experts",
+            "Expert Choice Routing",
+            "sparsely-activated models",
+            "training efficiency",
+            "downstream performance"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "addd7c61-bdd3-470e-8c4f-439af824f89e",
+        "properties": {
+          "page_content": "F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+          "embedding": [
+            0.0002445570513886185,
+            -0.025035678422212698,
+            -0.026056660824720954,
+            0.0021234386221176942,
+            -0.007526403638487191,
+            0.018963965001103752,
+            -0.007728070932730228,
+            0.062213823247851435,
+            -0.00501600735659763,
+            0.031461940272958254,
+            0.014585783857839997,
+            0.010118033107401152,
+            0.009983318452435566,
+            -0.03282616973330728,
+            0.015192449242879527,
+            0.01480596701302375,
+            0.019405922018728892,
+            0.022192910412596127,
+            0.008048967251594976,
+            -0.0367043822293015,
+            0.017737679976556845,
+            -0.022757028540067756,
+            -0.06290354368907101,
+            0.016842087557557506,
+            -0.0050319451760917995,
+            0.019311667659179976,
+            0.008548884527537198,
+            0.02344214718766695,
+            -0.042417710048062586,
+            0.01950972309054391,
+            0.021637767293446648,
+            0.010581067128107188,
+            -0.014214247204390872,
+            0.01016365063627519,
+            -0.025782209417014867,
+            -0.02611397356963557,
+            0.034251067799009986,
+            0.0071402676991737055,
+            -0.020147891665364803,
+            0.056472902950962155,
+            -0.006027534100459245,
+            0.005473493744726429,
+            -0.03139313966858068,
+            0.00043634965151753214,
+            0.013171990459657583,
+            -0.016474119463665594,
+            0.0075792906007691105,
+            -0.0032442837467945356,
+            0.02492845005795393,
+            -0.04366675539501486,
+            -0.006033838756045725,
+            -0.018660330752604155,
+            0.02088611082654532,
+            -0.019387265112916365,
+            -0.01222746328548615,
+            0.005977009213829137,
+            0.0012690574665308455,
+            0.05430319716489232,
+            0.058075720286678886,
+            0.02554562736016709,
+            0.02997758641056458,
+            0.00014178556329408164,
+            0.004139467932725944,
+            -0.004454862143349584,
+            0.031094774362479788,
+            0.0017837745263941452,
+            -0.013542834385584962,
+            0.01983846902748998,
+            0.010136662804503968,
+            -0.0387179389182337,
+            -0.06156798393873734,
+            0.015844092498186847,
+            -0.020820695961906926,
+            -0.02678200120240037,
+            -4.570339833017677e-05,
+            -0.015544616932001905,
+            0.005434882475939711,
+            0.050835936350804696,
+            -0.004985250023269435,
+            0.0016509410141492726,
+            0.022801366331672433,
+            -0.004080282248725238,
+            0.0078367492683837,
+            -0.00034876063727630653,
+            0.039772801289415724,
+            0.0288510926430687,
+            -0.022609202992981695,
+            -0.07091181813750308,
+            0.045194222785472735,
+            -0.03225547853169315,
+            -0.032585463285183205,
+            0.01670145480899693,
+            0.03286138231600811,
+            -0.009733112712376757,
+            -0.030197857075781466,
+            0.0731500616714208,
+            -0.005545147701735842,
+            0.017072915518860705,
+            -0.021002529816498382,
+            0.010525361666209598,
+            -0.012745643075817463,
+            0.025352505498532282,
+            -0.003102906957075941,
+            0.016391641977220053,
+            0.02534354270624064,
+            -0.008108690979345334,
+            -0.006451703723989414,
+            0.0065097623653779324,
+            0.04453924610212246,
+            0.0889545538715994,
+            0.019774398083206815,
+            -0.003981259900744729,
+            -0.046615274629371053,
+            0.00732791197790897,
+            -0.02916202133853672,
+            -0.04303144138453202,
+            -0.04367640508681853,
+            -0.010910501188273782,
+            0.023755744360759886,
+            -0.003399898430475389,
+            0.03910227884474275,
+            0.04137187598162877,
+            0.03363334616934253,
+            0.00916121492790106,
+            -0.031686589382074935,
+            0.026907320660055866,
+            -0.01693827759642556,
+            0.002542422168436765,
+            0.00734783450287368,
+            0.008189080458799796,
+            -0.006352648760821389,
+            -0.002333605867785783,
+            -0.03831808898525347,
+            -0.036762698962852246,
+            0.012773386000855496,
+            -0.008349610778914014,
+            0.027936454763857345,
+            0.07015560257394027,
+            -0.03713244149832974,
+            -0.0066245405611366675,
+            0.02802618921440782,
+            0.0017169880395367697,
+            0.023346396749133396,
+            0.007605522879978981,
+            -0.023484167590494896,
+            0.008594274917753499,
+            -0.0071765874950466295,
+            0.0677537727828362,
+            -0.008126000179550069,
+            0.011046754551416857,
+            -0.019516727128214784,
+            0.031654663874166704,
+            0.0007057013357055938,
+            0.021535694192642306,
+            0.03313372731162823,
+            -0.005794425715079591,
+            -0.03865395674172617,
+            0.032271538528276474,
+            -0.029790294001121286,
+            0.0013647166052139265,
+            0.014194415049797637,
+            -0.004899479294176156,
+            -0.015920769574443834,
+            0.0013708196727143433,
+            0.029757298661028133,
+            0.045717552753231847,
+            0.042911032378569906,
+            0.0016019404554248433,
+            -0.06844613496660937,
+            0.06365820914938357,
+            0.012895947255192629,
+            0.03208498337942363,
+            0.009724409704032419,
+            0.042361427690196844,
+            0.008120380964731038,
+            -0.0004384537671641681,
+            -0.005540416382489673,
+            -0.005125920468958917,
+            -0.021186731704155592,
+            0.03258917022091572,
+            -0.01443327736487225,
+            -0.007107908179884635,
+            -0.00206346150761894,
+            0.014042865018508809,
+            0.028517557440008168,
+            0.017283745982594707,
+            0.0029564774390786045,
+            0.04378212980825259,
+            -0.019454732028746872,
+            -0.009437148551161395,
+            0.018820938021162286,
+            0.0008903198503990673,
+            0.025297040557527045,
+            -0.0011429799732923295,
+            -0.01670583390791835,
+            -0.009421860087826363,
+            -0.0328764813310764,
+            0.009563088287577815,
+            -0.0015347427243049344,
+            0.019228666906882602,
+            -0.007174098839673089,
+            -0.04068735475327182,
+            0.008903031576514094,
+            -0.011472954071988202,
+            0.015735403376626513,
+            -0.007084760348074423,
+            0.008729329609340102,
+            -0.00638454270428521,
+            0.016210007209816298,
+            -0.02460475066044075,
+            -0.009047036293783832,
+            -0.022598561172568143,
+            -0.020144302113939427,
+            -0.010886272544512816,
+            -0.010886770387412563,
+            0.008591903125597681,
+            -0.012216526799178626,
+            0.023956058440750026,
+            -0.017402060516065108,
+            -0.018328740623422317,
+            -0.00046940812181371226,
+            0.022921229116282273,
+            0.009026508358954174,
+            -0.00446208425714566,
+            -0.04421346673289319,
+            -0.01214251475197077,
+            -0.016860336137191526,
+            0.005985059770480569,
+            0.02804423359971541,
+            -0.03510963462663318,
+            0.012588036039507566,
+            -0.0018909139908325232,
+            0.0019847602212350957,
+            0.011374423039138836,
+            0.015486762163219968,
+            -0.027040748338117617,
+            -0.016329845569406535,
+            -0.012600824227928017,
+            -0.01765236203915568,
+            0.01673827406092187,
+            0.0016641690585652725,
+            0.003997011410859195,
+            -0.00499922800986537,
+            -0.02782499674212141,
+            -0.008774389172636544,
+            -0.011506812701874534,
+            -0.009783555199790996,
+            -0.015740884114025612,
+            0.013493654174963239,
+            0.014254624157312108,
+            -0.033270488877124596,
+            -0.018446763439664365,
+            0.02247061755496804,
+            -0.0035396859846893343,
+            -0.03759408156279411,
+            0.04544560120093284,
+            0.052893430652542234,
+            -0.02220682841881151,
+            0.002828355769475809,
+            -0.03469141962159207,
+            0.05950248906327849,
+            -0.03444371252359212,
+            -0.03378776392860449,
+            0.06521072243320349,
+            -0.009713026070923203,
+            0.015249223153809483,
+            0.03131206469454657,
+            0.0034865285730464384,
+            -0.0025997752599464445,
+            0.00912836988165194,
+            0.033031075123542926,
+            -0.014878258361522118,
+            -0.013101723481793925,
+            -0.01797681408884685,
+            -0.002397237505725697,
+            -0.03438738791694146,
+            -0.021713511748922477,
+            0.00836495266415799,
+            0.013064910474172053,
+            0.017110113503472666,
+            0.017514586931181946,
+            0.016195079102773242,
+            -0.0024109742901844617,
+            0.01545592172528996,
+            0.001046760697009891,
+            -8.863738247355372e-05,
+            -0.010979717992873124,
+            -0.017762086340346584,
+            -0.052667758929128256,
+            -0.018996338856872796,
+            -0.04421628171000429,
+            -0.005982494173649518,
+            0.0012420734363470585,
+            0.009886466834005627,
+            0.04949340056515024,
+            -0.01295176306623405,
+            0.003508142432105508,
+            0.010930632194984236,
+            0.01019261004578912,
+            -0.006556730312565122,
+            -0.032765993356945154,
+            -0.006485728779853869,
+            0.025709969056450885,
+            -0.022932446214570953,
+            -0.021792413124305354,
+            -0.007047777916806282,
+            0.0038659532636633486,
+            0.02321123092409853,
+            -0.026003302747277822,
+            0.017643847938685852,
+            0.010129447679208158,
+            0.009551458127273026,
+            -0.017747795509094742,
+            -0.02891499441428287,
+            -0.030321565228144824,
+            0.039350178167852384,
+            0.02899873253152933,
+            0.011338359735211436,
+            -0.033634061116901,
+            -0.013629902520109783,
+            0.0376352218933715,
+            -0.0019251015746424632,
+            -0.07267257827375968,
+            0.036685408572352814,
+            0.0050526642351482985,
+            -0.00019289056191322955,
+            -0.013581714265389013,
+            0.05300479823846402,
+            -0.04139717386779843,
+            -0.03335524761191713,
+            -0.00920987783097114,
+            -0.022616612319255443,
+            -0.005274093640024995,
+            -0.0012667939280239298,
+            0.032502321377997974,
+            -0.020089629343721754,
+            -0.024354491736044894,
+            -0.010206744478678104,
+            -0.02465036256948816,
+            0.017966455981144307,
+            -0.012995995315432267,
+            -0.02135780149071486,
+            0.010967719068923269,
+            -0.03402154331207373,
+            -2.0626689329697705e-05,
+            -0.01092309016851862,
+            -0.018683564965672157,
+            0.01571041043301007,
+            0.016259448102797114,
+            0.015403085986247105,
+            0.0020047641027155683,
+            0.015172380396260792,
+            0.008099191720720747,
+            0.004074377809743208,
+            0.01893667379047588,
+            -0.03384452762579117,
+            -0.038822883991989264,
+            0.050315952008617104,
+            0.0008458969404873289,
+            0.0012049589464131246,
+            0.006406225290437734,
+            -0.0042817559898658255,
+            -0.0411169553122232,
+            0.012933323148113837,
+            -0.05784927886369837,
+            0.006262727192870561,
+            0.012928199517490538,
+            -0.01786898028943474,
+            -0.02291183998443847,
+            0.00046953379111417065,
+            -0.024864892714864046,
+            0.02151263275001755,
+            -0.017247234755598105,
+            0.020405349906524557,
+            -0.023068655052252782,
+            -0.006872857342813033,
+            0.05842934123173711,
+            -0.02308642746390288,
+            -0.04925938088613823,
+            0.032829595473385986,
+            -0.007184802367095134,
+            0.03878513331161335,
+            0.055971017386540985,
+            0.016337316575743225,
+            0.006439501224944926,
+            0.026084366083114555,
+            0.013204436213767278,
+            -0.012028383848550473,
+            -0.0383295105300241,
+            0.015805232298071025,
+            0.030751727443425712,
+            -0.04685643586863969,
+            -0.010258856888449854,
+            -0.02541648783174773,
+            0.01599307149086869,
+            0.020046184897703115,
+            -0.0044598427609889935,
+            -0.0012270315730225135,
+            -0.009664326045898607,
+            -0.01520444928791987,
+            -0.02631191309701645,
+            0.02193635473056903,
+            -0.013740214173212537,
+            0.0007504924364311022,
+            -0.001111971939163715,
+            0.0012797636039507028,
+            0.008478066024469051,
+            0.01501219347281274,
+            0.027440349509054086,
+            -0.022728634568050663,
+            0.006916913421268421,
+            -0.008499438828073956,
+            -0.03788213665656746,
+            -0.006718262893819488,
+            0.029198375723488434,
+            0.006169843558767785,
+            -0.016427547826608675,
+            -0.0049009468913267205,
+            0.0022509753971959087,
+            -0.016307639896946134,
+            0.00372456949880952,
+            0.010347360137213142,
+            0.035374109073612686,
+            0.01856811791368892,
+            0.019247903053478954,
+            0.0035591273657627456,
+            0.010292179556783772,
+            0.004040044337368795,
+            0.025869815038962875,
+            -0.04526297785801751,
+            0.030173313035569235,
+            0.01521363974209499,
+            0.009059160353158267,
+            -0.03914219183565228,
+            -0.02367109515190293,
+            -0.025737979702884085,
+            -0.021654077593318197,
+            0.02346511753268162,
+            -0.02730833269330862,
+            -0.006289840626554075,
+            0.006531934540756005,
+            -0.0006804032161296996,
+            0.003928945341546481,
+            -0.0033962179480786624,
+            -0.037975452991752895,
+            -0.009840510213526213,
+            -0.04023861493405766,
+            -0.0241245373830213,
+            0.003822063151525241,
+            0.003249755356115233,
+            -0.06594891648887229,
+            -0.001768385222533634,
+            -0.06222620899240395,
+            -0.013594921028316139,
+            0.03441942883093182,
+            -0.024438842229617237,
+            0.04820486752403123,
+            -0.05486229161412509,
+            0.018219866569279347,
+            -0.046562594596398506,
+            0.011354841468658838,
+            0.021738373879790916,
+            0.04347115691988908,
+            -0.007572536447300832,
+            0.014626550834376365,
+            0.009906079088662038,
+            -0.04530647508884267,
+            -0.04520033458696902,
+            -0.011400230794947595,
+            0.011239630851440557,
+            0.003287355301854498,
+            -0.012304101523783504,
+            0.02623659451177029,
+            0.015903981896266153,
+            0.04300392046214319,
+            0.03414364576071245,
+            -0.0005691335103219632,
+            -0.049506948352089984,
+            -0.027336474868358992,
+            -0.014910043127825028,
+            0.02552137904202476,
+            -0.03055258097953423,
+            0.003401887357920115,
+            -0.03228200232009397,
+            0.013338682663233296,
+            0.018207379090305786,
+            -0.02224210534533673,
+            0.032569850137027664,
+            0.014446434357859929,
+            -0.040301668913492676,
+            0.0012399866793165108,
+            0.0002450513991178417,
+            -0.038300625347349834,
+            0.008119751344998158,
+            -0.013712452991626932,
+            -0.05381257202269606,
+            -0.022199278450525977,
+            -0.014717114803771204,
+            0.02206967996899982,
+            -0.0023470596291977116,
+            0.017222681546831476,
+            -0.029965631581610256,
+            0.0053103260868707786,
+            0.039279027237044066,
+            -0.001286692479487674,
+            -0.01148541970963179,
+            -0.0008706049086180766,
+            0.022795196067681996,
+            -0.01968201411368477,
+            -0.045534614872904305,
+            0.009877856016973188,
+            -0.020566606587457816,
+            0.016002665913994348,
+            -0.015445999456355206,
+            -0.0052940252527317235,
+            0.010360748741468617,
+            -0.03170100753615497,
+            0.02004511444257144,
+            0.009722541379665509,
+            0.006411072442536516,
+            -0.017460683536795665,
+            -0.0033423349847155888,
+            0.015743901543629382,
+            0.047005401368949463,
+            0.0007329338628268773,
+            0.006153613674047577,
+            0.02501606022399451,
+            -0.013054974002495664,
+            0.05036387123988784,
+            -0.02807526639433245,
+            -0.03099731580589117,
+            0.04344605263104773,
+            -0.019489740746149623,
+            -0.017636713524640496,
+            -0.017960525325193457,
+            -0.009792405274000215,
+            -0.0062801692104839674,
+            0.02801421653619339,
+            -0.00712206060099242,
+            -0.005749470909187103,
+            0.020504942834135256,
+            0.013170883165927672,
+            -0.011218763497707175,
+            0.008523945766715354,
+            0.01511514654097483,
+            0.027931571407773977,
+            0.03214582418497515,
+            -0.02345593941074465,
+            0.0333746849861944,
+            0.009324947617983663,
+            -0.004234113006464689,
+            0.005131158074842622,
+            0.0020437263732432953,
+            0.00176803337042029,
+            -0.029372739581870692,
+            -0.01539406767336474,
+            0.015958225909710817,
+            0.007396588215584237,
+            -0.003291618431912445,
+            -0.0316834931316845,
+            -0.0028368107695447525,
+            0.02614797456533501,
+            0.003832488818435695,
+            0.026426519154286857,
+            -0.015605455255333021,
+            -0.02018445776455915,
+            0.008876565653043164,
+            0.023669947107607083,
+            -0.055401884612295824,
+            0.01882053182190406,
+            0.011046506385076857,
+            -2.6003594081234644e-05,
+            0.01591257026228522,
+            -0.0046432737555961385,
+            0.005094314300671912,
+            0.0482159962414822,
+            0.015126643838581288,
+            -0.013318945197936635,
+            0.0020418779929576307,
+            0.015136735547940301,
+            0.001727083579122356,
+            -0.020116332736417462,
+            0.002896210161586164,
+            0.012858057228188658,
+            0.0092589465483404,
+            -0.012984552574495295,
+            -0.005651788746934606,
+            -0.016856929103895944,
+            0.03249171052801259,
+            -0.005843029913406538,
+            0.02336833175997056,
+            0.04800276014595125,
+            0.02261889070736121,
+            -0.03578655846417886,
+            -0.011451513080556048,
+            0.0077334037670644135,
+            0.009415939321950421,
+            -0.012399032514032437,
+            0.006084821107593102,
+            -0.034655634244304866,
+            0.013104268092544735,
+            -0.0016781301605750378,
+            0.012249966178417542,
+            -0.025567673456449492,
+            -0.023488408901029257,
+            0.007732021927617865,
+            -0.006986715221001125,
+            0.01461433573263053,
+            0.00011845838735106802,
+            -0.005506714145045683,
+            -0.006827219009979501,
+            0.0005242424465709722,
+            0.002885486381705293,
+            -0.0042669493792680474,
+            0.021618593454103052,
+            0.00010034640948597114,
+            0.011602625527297699,
+            -0.023477647311478233,
+            -0.0162254255990209,
+            0.031352657169434826,
+            0.03529846802317018,
+            -0.0007059281329032357,
+            0.02568807107901636,
+            -0.00277252926066966,
+            0.010828396988798257,
+            0.00810821945842862,
+            -0.010307381937016405,
+            0.0003407672690264756,
+            -0.003768030765932326,
+            -0.0027367413382233317,
+            -0.012887542072043335,
+            0.0031930180018598015,
+            -0.014086667286395425,
+            -0.01504545079945656,
+            0.02010182276528891,
+            -0.002519095978589373,
+            0.011477055159845283,
+            0.006165264376334778,
+            -0.01139265727997407,
+            -0.016539552549751473,
+            0.01574258576985793,
+            0.05233886430069725,
+            0.022428487725773775,
+            0.044334901638531425,
+            -0.00706941862193002,
+            0.011585028247041065,
+            -0.016760013805097753,
+            0.001211362507398395,
+            0.048283299647824234,
+            0.015403290676022106,
+            -0.013446978976128664,
+            0.024015090204073844,
+            0.013753743307879434,
+            0.0398341544586196,
+            0.0034302789804899806,
+            -0.020152697770903245,
+            0.041614050886963766,
+            0.00392650113407049,
+            0.013465009394885544,
+            0.020487015678247938,
+            0.002430045518615804,
+            0.0035516244267459565,
+            0.02938659684630997,
+            -0.010142701862398358,
+            0.010335745659719125,
+            -0.033187389861872627,
+            -0.021792906328508686,
+            0.02532137764121119,
+            0.0005114146307176893,
+            0.033795722059780535,
+            0.007069260093915293,
+            -0.017628598898476375,
+            -0.017395163521872672,
+            0.021985468266342154,
+            -0.007987351575783343,
+            -0.025037973555114328,
+            0.002627649768366858,
+            0.008700784263932844,
+            -0.004480915144455101,
+            0.006798551384053197,
+            -0.014580806303786968,
+            -0.009741248878656092,
+            0.025066050175629597,
+            0.007330922027951371,
+            0.010472464570407973,
+            -0.024286928122882124,
+            -0.011159235410800293,
+            0.006107141408585794,
+            -0.04181412282877586,
+            0.008647396691893692,
+            0.00528510107024571,
+            -0.006647986735406744,
+            -0.002548771993161688,
+            -0.0011322362644692566,
+            0.00513401828632921,
+            -0.0035963203271502263,
+            -0.014021183815351008,
+            0.022993860073007393,
+            0.02875036422479212,
+            -0.012996574913859303,
+            0.014373647963439562,
+            -0.0093541571449835,
+            -0.019474535598737786,
+            -0.0233122546137944,
+            0.014269723193443946,
+            0.008784376430480408,
+            0.004317715767550436,
+            -0.008273153355003107,
+            0.0010203136295909875,
+            0.001909823045834306,
+            0.02406426043795662,
+            0.049377235716785324,
+            0.005265955549193235,
+            0.029166436381966566,
+            0.017136545810044906,
+            0.03882940870384413,
+            0.0007533463090350569,
+            0.016233986042907378,
+            -0.03311355052072762,
+            0.03082356484543373,
+            -0.0009431717656252606,
+            0.04077734731808,
+            0.0369941736537998,
+            -0.05477277258937169,
+            -0.014658721668336077,
+            0.028327060073149418,
+            0.010896122046930724,
+            0.01436699901738847,
+            -0.003307011398652852,
+            -0.01163199508062231,
+            0.029659278016495517,
+            -0.023936970669211955,
+            -0.010748756889752312,
+            -0.00767811402724157,
+            -0.010785978668072403,
+            0.009408061520238972,
+            -0.0038320809092061126,
+            0.038383104318077226,
+            0.0070413219383093,
+            -0.009684310528333753,
+            0.019154521368080882,
+            -0.006292471963504016,
+            0.01596901434992771,
+            0.0037167338451907566,
+            0.0019879266696342887,
+            0.03104390318244226,
+            -0.003561483519923973,
+            0.028949607591091022,
+            0.015156617705082505,
+            0.017473301120296506,
+            -0.0029321911929339165,
+            -0.017573294769065502,
+            0.027284173131869906,
+            0.009113069584708108,
+            -0.01971507323375489,
+            0.003667023499937692,
+            -0.0025339283534597176,
+            -0.0077697455666997945,
+            -0.010491575421861018,
+            -9.407001004405342e-05,
+            -0.00211520268689493,
+            0.02087980536494742,
+            -0.005582301812064227,
+            -0.0016376727058019977,
+            -0.0064225805186745605,
+            0.0052835419390265485,
+            -0.019662445004865352,
+            0.015117937189193789,
+            0.007575238218238001,
+            -0.018537762525347407,
+            0.014673223139419801,
+            0.019165766004277593,
+            0.0343151018851732,
+            -0.023949650834192267,
+            -0.0009279394158895472,
+            0.01730410035639405,
+            0.005928527001253011,
+            -0.018975612465080345,
+            -0.033873819454780706,
+            0.00825263613547798,
+            -0.01745801647450136,
+            0.027401724356781083,
+            0.008830481774058282,
+            0.0347823637819511,
+            -0.02309575756624861,
+            0.020559439488958053,
+            -1.2525886083543457e-05,
+            0.01593239963830551,
+            0.01455178268663934,
+            0.027993448238508672,
+            -0.017573784980205873,
+            -0.0379374065982424,
+            -0.0030450459335320053,
+            -0.0029968228865394214,
+            0.016851016821808874,
+            -0.013002468483615099,
+            -0.003337198735398279,
+            -0.0019044899095842511,
+            -0.029686441224288696,
+            -0.021487593939979652,
+            0.027796807698461906,
+            -0.04637191752173605,
+            0.0030176823995715597,
+            0.03281049040516897,
+            -0.030047693785808505,
+            -0.04246000386774685,
+            -0.0014603539776430917,
+            -0.03560639883119047,
+            0.0010602583020630973,
+            -0.01222013383414755,
+            0.029128085483749913,
+            0.007388529661657186,
+            -0.020518571527741426,
+            0.018184513437823,
+            0.0076594003031722,
+            -0.004161472492195519,
+            0.006100954284405263,
+            -0.005239699097592758,
+            -0.028136388241670696,
+            -0.015254134018585082,
+            -0.03037392857155718,
+            -1.6317930920712043e-06,
+            -0.002478230556191563,
+            -0.012845180214134591,
+            -0.024383682688014996,
+            -0.000484419407302015,
+            -0.014810859291916117,
+            3.958274758340876e-06,
+            0.05136363130801806,
+            0.02120810926469503,
+            0.005060247818396135,
+            0.012869784532979883,
+            -0.019665446839172217,
+            0.04743686110578455,
+            0.003514534022065111,
+            0.03583012040901262,
+            0.002062991366066978,
+            -0.02173965243546218,
+            -0.0005002960506349255,
+            0.0319890423992129,
+            -0.0016894669649440992,
+            -0.017249094049398075,
+            0.006136324758346181,
+            -0.009220681391996718,
+            0.00837047468537671,
+            0.012871598576797918,
+            0.01579543926540845,
+            -0.0061999233512262265,
+            0.014725777187858035,
+            -0.003927118678273384,
+            0.009336164668870259,
+            0.03122006739418818,
+            0.019771411252821138,
+            -0.024716950288097537,
+            -0.037736399088695305,
+            -0.012523788001500705,
+            0.001234987976751515,
+            0.034477269953960235,
+            -0.013836458205716058,
+            0.003837278407556882,
+            -0.011027628278972294,
+            -0.0054689562005572675,
+            0.005739209890610896,
+            -0.006985434270987544,
+            0.0015806421600885957,
+            0.012196603223058041,
+            -0.006859901421089723,
+            -0.025670287745499374,
+            -0.004011122451643543,
+            -0.023258896941014143,
+            0.012997299613183496,
+            -0.010144866692641839,
+            -0.006816231069049567,
+            -0.005213359047878286,
+            -0.0014167278420750282,
+            -0.0062535968890290545,
+            0.0006662897611595949,
+            -0.014236807237438571,
+            0.01012846684096945,
+            -0.00899852930003816,
+            0.0016620397335915663,
+            0.007838787681725669,
+            0.007725632848979133,
+            0.0033110678307931205,
+            -0.008206916937522372,
+            -0.008777095103563134,
+            -0.01631820698659747,
+            -0.032412484976539334,
+            -0.03000207979123948,
+            -0.02276516767125653,
+            -0.003870969029785218,
+            0.0034978865240734053,
+            -0.032594545526009384,
+            -0.020635972136797354,
+            -0.002367523739258502,
+            0.0030103817989158062,
+            -0.014587341442812854,
+            0.0014779462725289245,
+            0.007969995140846214,
+            -0.0029319878689507335,
+            0.019355924479361485,
+            -0.005448177691636249,
+            -0.021203447333791942,
+            -0.028956667864566195,
+            0.017851274292380093,
+            -0.015891501231440428,
+            -0.010070179095955473,
+            0.0064388768290910715,
+            -0.005633920123057214,
+            -0.03957042012546069,
+            -0.008945181709739484,
+            0.010992989857329521,
+            0.0035378401849368002,
+            -0.025936727040303652,
+            0.031903561853958325,
+            0.030132441728296204,
+            0.01845519141895959,
+            -0.017902241158092712,
+            0.04710441323879975,
+            -0.006739916600200837,
+            -0.012103058303430145,
+            0.016276540022040643,
+            0.023963592487909288,
+            0.028774476787161263,
+            -0.02643829153200483,
+            -0.005349178006882811,
+            -0.010545703050010008,
+            0.005354643441490569,
+            -0.02271588009302058,
+            0.008468528451368553,
+            -0.013601178278798316,
+            0.011576067943282815,
+            0.019227032134009135,
+            -0.013197207810056759,
+            0.02629307658052319,
+            0.0045263515787155985,
+            -6.067227645639678e-05,
+            -0.0037770406201685248,
+            -0.016287414859552752,
+            -0.012319957965328086,
+            -0.034674880061192896,
+            0.020297690806102293,
+            0.01889209267823917,
+            0.003953684662429981,
+            0.021845046751532583,
+            -0.049434011185276405,
+            0.02806744160678342,
+            0.0017849117483741536,
+            0.021763434864703356,
+            0.013889518910180847,
+            -0.001843068102127134,
+            0.013886691089114936,
+            -0.0032117229736863266,
+            -0.019971320584891997,
+            0.006079730961157539,
+            -0.006352535944493006,
+            -0.03316544192496317,
+            0.014736816574462923,
+            -0.03268321622529288,
+            0.013639134163846678,
+            0.015454170145821088,
+            -0.010170396823729307,
+            -0.021708948733883182,
+            -0.008303375382721087,
+            0.0010570623547459114,
+            -0.0014149688171089135,
+            -0.006849845444850982,
+            -0.04346483266888068,
+            -0.03020319096411844,
+            0.03124837430439466,
+            -0.0016720020681567616,
+            -0.03404132565569305,
+            -0.007209824644083462,
+            -0.02711263625315296,
+            0.004780610743996515,
+            -0.004212647822173262,
+            0.0025676935176963222,
+            -0.048750272936498695,
+            -0.01018434342955251,
+            0.011179560412168381,
+            0.031648611061279724,
+            -0.029298960703078986,
+            0.021370266671953315,
+            3.770954396468699e-05,
+            0.006629905234129844,
+            -0.00943632381701963,
+            -0.00484754585233744,
+            -0.012115337995959813,
+            -0.02607251387584821,
+            0.021868809148577698,
+            -0.0020587964818437345,
+            -0.0013502026980372474,
+            0.0341522429571705,
+            -0.00669289309893371,
+            -0.010113192232031375,
+            0.0005934017402521538,
+            -0.011327797122243258,
+            -0.017269539623150844,
+            -0.010467173785198161,
+            0.0005779464354975677,
+            0.0030406502310922306,
+            -0.024755136158113783,
+            0.004943446717510779,
+            0.04566273647510376,
+            0.013607936503018211,
+            0.02534200532143803,
+            0.017104750405207106,
+            -0.03792550084708064,
+            -0.04394052120951292,
+            0.0012600504352102988,
+            -0.005807815271334456,
+            -0.01944685298932453,
+            -0.01732108197375621,
+            0.0011677414026425442,
+            0.00790730598557849,
+            0.0037240231183698074,
+            -0.011427532982681041,
+            -0.02086964367350831,
+            -0.032539146090370744,
+            -0.0136797717535217,
+            -0.009464326388205681,
+            -0.007302204556505735,
+            -0.032703238152349344,
+            -0.015929357859286307,
+            0.026848214129378756,
+            0.01012098852614113,
+            -0.005646819862636667,
+            -0.0107564788588665,
+            -0.024118155140225247,
+            0.007122339033521904,
+            -0.0027183981945039814,
+            -0.009219030314883272,
+            -0.004672057262936543,
+            -0.016450690196817726,
+            0.02348093022809639,
+            0.009139210149358477,
+            0.024997129282118148,
+            -0.058533911093319214,
+            -0.014411932474734742,
+            0.014256038602455438,
+            0.0020188651815054508,
+            0.001417136751974749,
+            0.029836680491427268,
+            0.0028786535004858044,
+            -0.006526872714072965,
+            -0.02967220833325222,
+            0.028817295504169067,
+            0.027449694157944637,
+            0.01800200378294509,
+            -0.010560094796033294,
+            0.0033996190760546383,
+            0.006004691351249025,
+            -0.005679195734863024,
+            -0.011927786207717698,
+            0.0021260244673648285,
+            0.018998574211603905,
+            0.017174632765532803,
+            0.014659925822127794,
+            0.02455156987384756,
+            -0.005882590537857664,
+            -0.03911277764764406,
+            -0.009781110378157883,
+            -0.0119630929326396,
+            -0.004815125964940769,
+            -0.0026631770014339242,
+            0.008903827530107938,
+            -0.011824767329332145,
+            0.013088406716464123,
+            0.030264448923574067,
+            0.042592268021664714,
+            0.021292401951863188,
+            0.006150997237661769,
+            -0.005463801502425317,
+            0.0007399518998122848,
+            0.0038649546282459707,
+            0.011355675550395135,
+            0.035762872771750435,
+            -0.018147218572765546,
+            -0.019704502923991365,
+            0.0015558311499702044,
+            0.023543487977413267,
+            -0.007150197116420024,
+            0.02608209797565159,
+            0.023985156585794257,
+            -0.0003434024317322991,
+            -0.01271338268931938,
+            0.000985282363050309,
+            0.006710364567868783,
+            0.014540943712513688,
+            -0.02154205198718209,
+            0.004039619025717627,
+            0.001865104196024055,
+            -0.009756600776161738,
+            0.029738871754072076,
+            0.00039154552800346406,
+            -0.012981811693538832,
+            0.03825344802886,
+            0.11025963334575106,
+            0.001476339401950677,
+            0.0031187927813794386,
+            -0.031522639386920355,
+            -0.015113380587160163,
+            0.0021786482865071505,
+            -0.005948756393737655,
+            0.020082591515778573,
+            0.012751062800862873,
+            0.0055847042526916265,
+            0.010100961720464159,
+            0.038084609873482715,
+            -0.009203643420546616,
+            0.009804311800033327,
+            -0.01878272400403416,
+            0.017185477997947226,
+            -0.00972716681599207,
+            -0.00857276597428136,
+            -0.007641511960016827,
+            0.004167416313632853,
+            0.00984005584819001,
+            -0.004596999098606572,
+            0.018810908413859465,
+            0.002496310438422421,
+            -0.01613133688770545,
+            0.011505030137914962,
+            0.007437924384075958,
+            -0.005386885126779478,
+            -0.031755982499042136,
+            -0.013811586180238287,
+            0.01190667400285204,
+            -0.0013786018401773159,
+            0.0007177632889203917,
+            0.012342640512536284,
+            0.01205074763420124,
+            0.031217897495485192,
+            -0.0244324065837541,
+            0.0025071378510626696,
+            0.0070486374045588465,
+            0.0159322020159265,
+            0.01098799814774876,
+            0.00029844505852147526,
+            -0.015456787478044924,
+            0.027139764983358832,
+            -0.011978684728746428,
+            0.010378658252224036,
+            0.01146607671795495,
+            -0.04131658683485702,
+            -0.0009967932006190922,
+            -0.005916133731248287,
+            -0.015698754902727707,
+            0.013905118384587169,
+            -0.040163447416240027,
+            -0.005682084326086952,
+            0.004297114532998809,
+            0.010724201884417054,
+            0.0058749102353125955,
+            -0.017715270733111323,
+            0.009962853888582856,
+            0.00955216765250444,
+            0.0025392532243413593,
+            -0.006356691980285544,
+            -0.006218169970979339,
+            -0.008606942887730427,
+            -0.004280171880086814,
+            -0.015669721147884084,
+            0.0057481312044666884,
+            0.0077227094511494055,
+            -0.018751123856436715,
+            -0.01585864749751703,
+            0.009183099781375234,
+            0.006837438250956817,
+            -0.001445622524955126,
+            0.004828557942941401,
+            -0.010034012200008746,
+            -0.004543834537450213,
+            0.01097540631691429,
+            -0.009334158551770033,
+            -0.004804430123454511,
+            -0.02609985520241612,
+            0.013342299390862539,
+            0.011103136832190223,
+            -0.008261332966534102,
+            -0.006944131083745896,
+            -0.016931502665633764,
+            -0.0031318730206304824,
+            -0.0026508016484082988,
+            0.0016110884325639078,
+            0.017071266355203386,
+            -0.009906320666640836,
+            0.010877416311741906,
+            -0.03496707406301918,
+            -0.015357787514985604,
+            0.01622830705930841,
+            -0.01290003882907745,
+            -0.005625420341281716,
+            -0.009838333350984609,
+            0.0006911383780218238,
+            -0.007545081680638308,
+            0.006074752194220675,
+            -0.041320607445282646,
+            0.0035102345636816475,
+            0.018825308640738208,
+            -0.009723969339534365,
+            -0.020829421583878845,
+            -0.009055117268112308,
+            0.010487235049013567,
+            -0.015831829161845813,
+            -0.01780922120240258,
+            0.0016219998217122911,
+            0.010559783293374071,
+            -0.0055135690971065965,
+            -0.005160232484458576,
+            0.006163424486940105,
+            0.020247627989655837,
+            0.015220964429515309,
+            0.04337432385613251,
+            -0.011915355138717245,
+            0.009617347907919507,
+            0.027812934121040484,
+            -0.030652481946156104,
+            0.01037470561447702,
+            0.02599512377987521,
+            0.0037340561802882417,
+            -0.0013375196064018082,
+            -0.010197691376747194,
+            0.004161692215514214,
+            -0.012254402844820636,
+            0.008715552098450461,
+            0.004697361473725416,
+            -0.027978648968732336,
+            0.017196771853396124,
+            0.019448218659009996,
+            0.0038720194629308794,
+            0.0006995062900842749,
+            -0.01527901684789497,
+            -0.011825577013200781,
+            0.019194236643713274,
+            0.027690065919038755,
+            0.005460028121722145,
+            -0.015512497062157167,
+            0.02370643287056251,
+            0.012916257803027072,
+            -0.004973882005506947,
+            0.0022332716531575122,
+            -0.016984605526823043,
+            -0.01443316577897661,
+            0.030942241604983316,
+            0.011563751997976407,
+            0.01908030186367601,
+            -0.021614212853976106,
+            -0.003982310717524619,
+            0.019931579069642925,
+            -0.021029366993395197,
+            0.013006879711144009,
+            -0.019497570222553013,
+            -0.014067925104548129,
+            -0.010687576618198498,
+            0.02884494104893037,
+            0.009229770955288097,
+            0.026701705044769222,
+            0.012715371448003833,
+            -0.021444875656527786,
+            -0.021050591467198086,
+            -0.0027703267406452336,
+            -0.02721606030674425,
+            0.01688256721881862,
+            -0.055990128879151335,
+            0.0016563858333845547,
+            0.010911810464298247,
+            -0.008604898990983188,
+            0.004039521277648332,
+            -0.007858378479197747,
+            0.010021929060776957,
+            0.020926274809879138,
+            -0.02956229623532218,
+            0.01256060127924585,
+            0.01156331569664224,
+            0.0218688589850221,
+            -0.018861065515394883,
+            0.015339087155577661,
+            -0.00841001512858443,
+            -0.027396412391138852,
+            0.008323898095545889,
+            0.01763697170382491,
+            0.014160184414353919,
+            -0.011992345081843988,
+            0.004485633614765869,
+            -0.04000932898876895,
+            0.027999519977069676,
+            -0.016823500188831896,
+            0.002010511519234648,
+            -0.004537991252901591,
+            -0.0067219583064939085,
+            -0.008523354091034399,
+            2.682477256914193e-05,
+            -0.0005371719103087346,
+            -0.030920653652724776,
+            0.0012939076000836422,
+            -0.04144943703666512,
+            -0.02260198516292792,
+            0.013327519374598822,
+            -0.0012897465780583367,
+            -0.005059170593504863,
+            0.02999548027478466,
+            -0.02863650640718448,
+            -0.027272295036911975,
+            -0.014376307736824656,
+            0.03534844385341626,
+            -0.015358480569131909,
+            -0.000540068394391221,
+            -0.009287193719601908,
+            -0.02236594406404711,
+            0.01091078016405303,
+            -0.006469935602381157,
+            -0.0019272207412132534,
+            -0.01387843655423569,
+            -0.03972775413987396,
+            -0.005948332199741248,
+            0.019500395452013777,
+            0.0006122786247500248,
+            0.003656719755271774,
+            0.014473976840529128,
+            -0.0072278373928449675,
+            -0.0036623750417377524,
+            -0.0038809301345520044,
+            0.026929584793529297,
+            -0.022225170775945557,
+            0.009744440210592443,
+            0.004704883568603746,
+            -0.0005991305039095727,
+            0.017317686588588826,
+            0.000351879037187266,
+            0.0024030356296394478,
+            0.0029060795825935717,
+            -0.0035301907818594274,
+            0.02922907976800865,
+            0.0064070048516525,
+            0.0016291092834589405,
+            0.0001988403049139903,
+            0.004746570638478093,
+            0.01395197992587789,
+            -0.01069245020264236,
+            0.0157940052093005,
+            0.02865126209030693,
+            0.00021287181382353683,
+            0.018345429962063935,
+            -0.015921361819192573,
+            -0.036059435316760584,
+            0.019807131219096722,
+            -0.029642240399677812,
+            -0.003504127677138975,
+            0.022473005924273325,
+            -0.012903852741467797,
+            -0.0031750649075827082,
+            -0.01107957248024579,
+            0.027243132530764567,
+            0.023512079029549708,
+            -0.014856101416774535,
+            -0.015142539116300868,
+            0.004337433267586975,
+            -0.010210607143984678,
+            0.010939153927790107,
+            0.029382995339774957,
+            -0.011230337698862454,
+            0.011717621080350227,
+            0.001161341258722717,
+            -0.003778338045364591,
+            0.019300005069271462,
+            0.0027681684372537606,
+            -0.007401677666787755,
+            0.025628069381915978,
+            -0.008115857591022265,
+            0.013043669211084909,
+            -0.004672358463640539,
+            -0.012902494254220482,
+            -0.003106794914564588,
+            -0.017558145644151636,
+            -0.01225024815282763,
+            0.0011522376202485873,
+            -0.005528597019035285,
+            0.06711658458916447,
+            0.010930891527455397,
+            -0.017888423424597618,
+            -0.0059190199704595775,
+            0.00955567741445378,
+            -0.000966061435587513,
+            -0.009466328431452424,
+            0.027063767833815658,
+            -0.016888493246695876,
+            -0.009681104485277698,
+            -3.814655691584766e-05,
+            -0.0233149747136092,
+            0.0030596902889784496,
+            0.010827987186229144,
+            0.0012052973350181351,
+            -0.013994504803884577,
+            0.005318095523120499,
+            -0.004843763992616558,
+            0.0034188401938723923,
+            0.003678626789714963,
+            -0.0025749484272345573,
+            -0.005333294277998365,
+            0.02307826358450567,
+            -0.012791914224263883,
+            -0.008519982879775168,
+            0.01173424692936401,
+            0.001777123104204978,
+            0.003219553788359312,
+            -0.004564745566711077,
+            -0.002015107048152335,
+            -0.017123545909339455,
+            -0.011278616226036531,
+            0.023713473402287875,
+            -0.02733523566514418,
+            -0.00150943777243135,
+            0.017328382962199716,
+            -0.005705544203971564,
+            0.016650187392502975,
+            9.750950361601131e-05,
+            -0.02100066506382883,
+            0.00010090567939930427,
+            -0.024778741347883073,
+            0.010484659495766551,
+            0.013963183459158656,
+            -0.009531952698151101,
+            -0.020878167882780043,
+            -0.027881362421494915,
+            -0.010190031350698889,
+            -0.027246821338460653,
+            -0.0280692647056191,
+            -0.02020582179742232,
+            0.009778340348478517,
+            0.005715364307462615,
+            0.021929057840860695,
+            -0.006380523858228711,
+            -0.027552858236741976,
+            0.015095163660340326,
+            -0.013269344817111162,
+            0.008282918921343354,
+            -0.019991891695114384,
+            0.01754833974582726,
+            -0.014720830704852575,
+            0.02112161088830394,
+            -0.00041409266970033855,
+            -0.004490472334793758,
+            -0.012076561942010787,
+            0.003976913901514849,
+            0.008101357042612863,
+            0.00176590819486174,
+            -0.008977295353115363,
+            0.026242190932756135,
+            0.01707634775369835,
+            -0.007385093173634561,
+            0.01604386075569055,
+            0.031855080733360075,
+            0.014762206220119051,
+            0.038860660002328094,
+            -0.007238177370595944,
+            -0.009237098188374397,
+            -0.02764495845127417,
+            -0.004407870047633689,
+            -0.006820430763682645,
+            -0.004516837004332235,
+            -0.015251921783911087,
+            0.03164122113689215,
+            -0.010371400055822811,
+            -0.01686972159902524,
+            0.03979035122593764,
+            -0.021846728842184655,
+            -0.007084905708666271,
+            0.008642771511425465,
+            -0.013996636040226265,
+            0.03176661229383477,
+            0.007275612529363987,
+            0.018452818352301472,
+            -0.015039890816023917,
+            -0.015988701055097997,
+            -0.0019911190487748244,
+            -0.002806138139429132,
+            -0.020828646560807865,
+            -0.0063794261154720665,
+            -0.01614570050728432,
+            -0.0026319562705484634,
+            -0.00570753504957681,
+            0.015917008930969468,
+            0.0013916917451795444,
+            -0.015420032290910069,
+            0.04488171544231248,
+            0.005033499525006087,
+            0.022768662319611596,
+            0.010777933460735837,
+            0.012372342667053587,
+            0.015819823672091563,
+            0.006759824420801962,
+            -0.0001668726099910387,
+            -0.02211654164646909,
+            0.004261169256891085,
+            -0.005641717035723983,
+            0.0041005490092982915,
+            0.02012094437568505,
+            -0.024436624754739972,
+            -0.029708548399946657,
+            0.024493078535528394,
+            -0.01833101179919427,
+            0.0030908990956393764,
+            -0.0031025537219982234,
+            -0.010059653447038065,
+            -0.011612233298907742,
+            -0.03293899346571458,
+            -0.0005929867206193162,
+            -0.027438115397399914,
+            -0.03635734152790657,
+            -0.006287687204976032,
+            -0.007850474352061472,
+            -0.003574235211373211,
+            0.02449669755605501,
+            0.004269460596739739,
+            -0.029959856511873897,
+            0.01782698044631415,
+            -0.013252733005984911,
+            -0.013342147906120882,
+            0.0029581361690999564,
+            0.004786570391076097,
+            -0.0027618259744494854,
+            0.010720003765859173,
+            0.007898965393358715,
+            0.007358694001293221,
+            -0.010710502803266644,
+            -0.004049671482838962,
+            0.0013263710826446039,
+            0.0030943253037901135,
+            -0.013325799375903166,
+            0.030553973934607993,
+            -0.018812559793353154,
+            -0.004819166852190956,
+            -0.0006499911473215254,
+            0.0034556893023524674,
+            -0.0110929170801761,
+            -0.014862886414702483,
+            0.021004665462647988,
+            0.0031088550726355926,
+            -0.012570481398485388,
+            0.0096952264731202,
+            -0.0018868240406799254,
+            -0.0024767709620225923,
+            0.01176108203579327,
+            -0.002080730326597438,
+            0.014924914157541781,
+            -0.011040818699356567,
+            0.01490073610670544,
+            0.0034649445207609085,
+            0.008645740731109337,
+            0.003603641937848282,
+            -0.01905231764119889,
+            0.009780124784430865,
+            0.016356631955067272,
+            0.013415470760506568,
+            -0.029496530287207723,
+            0.008881639987471632,
+            -0.005975399013716009,
+            -0.009870580036511721,
+            -0.024372445588075428,
+            0.002590423942920311,
+            0.028960961301291905,
+            0.012044179251559706,
+            0.007934516374398097,
+            -0.0077949340827692054,
+            -0.005480592942515702,
+            -0.019923500717513334,
+            0.009041031662948522,
+            -0.01095651774810384,
+            -0.025157908389819465,
+            -0.007578712371715301,
+            0.03430545904801765,
+            -0.010740874257824011,
+            0.01343875386664532,
+            -0.005526757815283282,
+            0.02436552949831661,
+            -0.0027538610388853284,
+            0.0013690344627799137,
+            0.008416180197734094,
+            -0.006091433734127276,
+            -0.001708607133233843,
+            0.013349212286052506,
+            0.041066250931277824,
+            0.0038375614886832564,
+            -0.0050950709988837216,
+            0.01149207527692021,
+            -0.00326389523728132,
+            -0.005712157453687712,
+            -0.00309579814793649,
+            -0.025491974096530076,
+            0.015133356727314608,
+            0.0010745378801251735,
+            -0.017162228172591873,
+            0.021785292616181687,
+            0.027891598422935242,
+            0.013262622112280296,
+            -0.0178471114368855,
+            0.014456217536546996,
+            -0.009278392107874806,
+            0.01629461595238019,
+            0.02423194076020779,
+            0.010067296472207578,
+            -0.011865648674106681,
+            -0.0070073298858753805,
+            0.008985354045706703,
+            0.008096016282907443,
+            0.0004236756235618935,
+            0.020370823466310733,
+            -0.028334250141960886,
+            0.01483270667238479,
+            0.007137078270349882,
+            0.03314918453195744,
+            -0.0006231629401653488,
+            -0.014397909717816884,
+            -0.008918359605259833,
+            -0.002915473990900902,
+            0.011759189078587877,
+            0.003523992368256188,
+            -0.0006853609664806012,
+            -0.004749148984168426,
+            -0.014181883848149463,
+            -0.006778789302277201,
+            0.012376777107762334,
+            -0.0031646329407784245,
+            0.0030514749941358095,
+            -0.0007326603315688041,
+            -0.007811941797535734,
+            0.015661953163847418,
+            0.001669500869963878,
+            -0.0007143628544145883,
+            0.0016729300148826723,
+            6.604560293656475e-06,
+            -0.00467413979302959,
+            -0.019114577369302277,
+            0.008922356418978664,
+            0.00867683974577237,
+            -0.006578536023513047,
+            -0.006467824108991175,
+            0.009324373505311057,
+            -0.01432602608910974,
+            0.01852478469282692,
+            -0.0061254590933158826,
+            -0.006603842196861297,
+            0.00014277641689784565,
+            -0.012328738339684425,
+            -0.003642202698645936,
+            0.0034895681343535914,
+            0.0051453312116433265,
+            -0.000908593523739809,
+            0.0036062764245630334,
+            0.003940522150829955,
+            0.01083964694261132,
+            0.002824079422783228,
+            0.0015266576108775834,
+            0.007280263953807289,
+            -0.01039293741283943,
+            -0.007403606843286745,
+            -0.004993622657038095,
+            -0.006341662331986892,
+            0.007288958567606623,
+            -0.004364975055335228,
+            0.010457360052582658,
+            -0.0062292104145735515,
+            0.020471836369529147,
+            -0.006133009423930811,
+            -0.0024055488697334888,
+            -0.017695183455939345,
+            -0.007234478249839231,
+            0.007156119488831356,
+            0.007878044165936736,
+            0.009359297231370758,
+            -0.04487830669749157,
+            0.005038301058510894,
+            -0.0054192943462008595,
+            0.02941616633783806,
+            -0.009828941844770915,
+            -0.004335130343692854,
+            0.016067824555917796,
+            -0.019100842323053376,
+            0.0037066021889614513,
+            -0.013965982913833292,
+            0.003695036285467973,
+            0.0034558765116311525,
+            0.014205693020177169,
+            0.031307058208919394,
+            -0.007482180869231769,
+            -0.009383819319265733,
+            -0.006384083847148052,
+            0.026967380724567013,
+            0.018643952476557566,
+            0.023548246269020395,
+            -0.011885146764471029,
+            0.00960518654541979,
+            0.001357650512536872,
+            0.018271473493526338,
+            -0.009663920325675169,
+            -0.011163336807225407,
+            -0.000516828412255004,
+            -0.010288703356943627,
+            -0.0032111641654394615,
+            -0.021092460896394025,
+            -0.0033456309643941965,
+            0.025876107752976232,
+            0.0038584058642743987,
+            -0.0036689325941524813,
+            -0.015624044740149702,
+            2.2644755460272697e-05,
+            -0.011931276845804481,
+            -0.0055870010742218364,
+            -0.015697330226950804,
+            -0.028927689083316704,
+            -0.016192017705678192,
+            -0.002914436546908901,
+            -0.017224654082473787,
+            -0.009460990414684924,
+            0.002071634636546831,
+            -0.015533259213802986,
+            0.004918377471991415,
+            -0.015669642231243106,
+            -0.010915412245035698,
+            -0.00046472075835247,
+            0.004672390433470177,
+            -0.004682173537415046,
+            -0.024109027987987815,
+            -0.013628794248618615,
+            -0.0022054473551170926,
+            0.006500170428351522,
+            0.010016298501187265,
+            0.008324628078950177,
+            -0.0069374343736907625,
+            0.011201167788973098,
+            -0.009746811993090597,
+            0.010155637867574787,
+            -0.02773759960959338,
+            -0.04317847702979315,
+            -0.000791583637601122,
+            0.009241551203900427,
+            0.001979688190287368,
+            -0.01884488538245213,
+            0.0038489955050695704,
+            -0.011280060242688173,
+            -0.00847814532468811,
+            -0.007522173019881079,
+            0.01573091287228822,
+            0.004138348969142638,
+            0.013463997539337106,
+            0.0042065270170965705,
+            0.0285467221116185,
+            0.012635383326761538,
+            0.01030078781909351,
+            -0.003092067773698843,
+            0.004763648809248413,
+            -0.0022835042548997662,
+            0.009077632103272405,
+            0.02331475506951857,
+            0.0076225652102231605,
+            -0.002326207128504976,
+            0.00409858249509217,
+            -0.0005698076375460636,
+            -0.005442088641415082,
+            0.018565262151290427,
+            0.02915691852605287,
+            -0.0069811288298562155,
+            0.008120076286611811,
+            -0.0009110258445641983,
+            -0.009900329477406562,
+            0.0038288023984988134,
+            0.005765264445222657,
+            -0.014230725839572923,
+            -0.0024175374911696653,
+            0.009367987848760041,
+            -0.024664375361024386,
+            -0.004113757123212052,
+            0.020371264792326226,
+            0.0024813232005061487,
+            -0.016599537494340785,
+            0.03488073638324099,
+            -0.003709931703179228,
+            0.003010315958426392,
+            -0.00944047118505417,
+            0.016362178907316286,
+            -0.017989464717541103,
+            0.0005390907615384354,
+            0.013131172782591922,
+            -0.005251614069791733,
+            0.005783823857695169,
+            0.003744510952053123,
+            -0.010417674399178015,
+            -0.010824994224858631,
+            -0.0033951133363820767,
+            0.01085129781083312,
+            0.002642227227069058,
+            -0.024446190958178605,
+            -0.017731767444262785,
+            0.019381370848622046,
+            -0.01001206050737818,
+            -0.008290790009659911,
+            0.0009331678418736739,
+            0.0029270629970877303,
+            -0.004637651319894003,
+            -0.004290917518172777,
+            -0.003574121539057569,
+            0.009121344354139133,
+            -0.009485343005168732,
+            0.01637203851901565,
+            0.004043708359084711,
+            0.01843096334879628,
+            -0.011588935298420727,
+            0.005808551475129831,
+            -0.001368952044937682,
+            -0.007242291047654279,
+            -0.019004856431577073,
+            0.0059537808323522675,
+            0.0073777483129045685,
+            0.016051495408653346,
+            0.011060627996179347,
+            0.018569088420830965,
+            0.011345955812555888,
+            0.0039209921807600205,
+            0.001180658155391396,
+            -0.011006781869601802,
+            -0.016371246428912994,
+            -0.014232536816028922,
+            -0.007416908142074357,
+            -0.019561375671062513,
+            -0.0013253236527051844,
+            0.017798265625542643,
+            0.014528038148955117,
+            0.017704367192118375,
+            -0.0048389172759564884,
+            0.016018290057587425,
+            -2.7789810195444745e-05,
+            0.0009215076153434622,
+            -0.005897137279173464,
+            0.001794778099552753,
+            0.016314258230521846,
+            -0.003313666704140227,
+            0.0063152813149371136,
+            0.0002935873451081568,
+            0.01645203621687452,
+            0.022650990579075773,
+            -0.018899997585540014,
+            -0.0014442758993479705,
+            -0.006467599303099793,
+            -0.0021225224281092863,
+            -0.019894752041664215,
+            0.020266472457121338,
+            0.014019662562577204,
+            -0.005002061958298289,
+            -0.00943116174529331,
+            -0.004354703314807638,
+            0.02678118874395904,
+            0.020206530983694557,
+            0.006044260997975214,
+            0.026222728100119382,
+            0.01976601108150248,
+            0.01984865298844998,
+            0.011918890024498122,
+            0.009052182076567042,
+            -0.013092243192366595,
+            0.018358346459726883,
+            -0.01681770553983246,
+            0.0008504895530001561,
+            -0.003223133403363917,
+            0.002743153478851782,
+            -0.007937179998086906,
+            0.013021430914653432,
+            -0.00030955362267617273,
+            -0.025935204965869492,
+            0.007680669831594167,
+            -0.013183173230049736,
+            0.001375652741144253,
+            6.07690288088255e-06,
+            -0.001857629339163609,
+            -0.017652365850892467,
+            -0.0082709435899296,
+            0.0008448530375262078,
+            -0.01435595518048752,
+            0.023866770609514233,
+            -0.026826645910954506,
+            -0.02140002089501948,
+            0.00988141127158139,
+            -0.004436844338114915,
+            -0.012632978351515892,
+            -0.006972217855086974,
+            0.021864041851582767,
+            -0.005139891275508354,
+            -0.016596866242234234,
+            0.014168148679282804,
+            0.0005869000756971533,
+            0.00412958763467087,
+            0.0012364712526909765,
+            -0.008001798306987228,
+            -0.0030416500521329983,
+            0.006860667487819997,
+            0.028693213258586023,
+            -0.011895509688811421,
+            0.0016632663352826491,
+            -0.013799461077520309,
+            -0.0037922015693749754,
+            -0.02974150200357857,
+            0.027752493864472517,
+            -0.011830981692069491,
+            0.01989293240244598,
+            -0.004540523332199857,
+            -0.007667738626842118,
+            -0.021173122202316107,
+            0.035703796333655495,
+            0.004594443089494466,
+            -0.0006664409717898689,
+            0.007256660335524136,
+            -0.047588498838074554,
+            0.01105469709139337,
+            0.006498789757276137,
+            -0.023261890747840144,
+            -0.000964111432535798,
+            -0.003995007813335293,
+            0.02143067593410374,
+            -0.019461700447829744,
+            -0.012777737727876326,
+            0.0021024453304811634,
+            -0.021925886491102078,
+            -0.007056889535073896,
+            -0.0340604475320119,
+            -0.006137802768771592,
+            0.006503135515471076,
+            0.0038385769709340417,
+            0.0020799029890387023,
+            -0.021308499975514292,
+            -0.0309544649831362,
+            0.013308972238482328,
+            0.01651668347077017,
+            0.010007831710070633,
+            0.01200701454477496,
+            0.0067050757442456005,
+            0.011425518484475204,
+            0.00865023114755134,
+            0.00974582806570851,
+            0.008884712478547614,
+            0.039109252942848706,
+            -0.016122498485808056,
+            0.008671479031657285,
+            0.0005699279397862279,
+            0.02156543966774511,
+            -0.02391743870774471,
+            -0.009877938786506814,
+            0.015475531666380014,
+            0.003045729682805746,
+            0.030465388674890067,
+            -0.01511217342930631,
+            0.005735960079359902,
+            0.0030955813611916066,
+            0.003343849304714193,
+            -0.017685941897160877,
+            -0.019795694199965495,
+            -0.015889472669342385,
+            0.0005064000414834533,
+            -0.0007719666713392906,
+            -0.012817517834744753,
+            0.01370266779873307,
+            0.0029457043321269045,
+            0.004380574233134056,
+            0.003997906783171028,
+            -0.02610528890228495,
+            0.0033844277218836307,
+            -0.008397899012048187,
+            0.013703609594527308,
+            -0.0012702392316682096,
+            2.036849934035612e-05,
+            0.007261982817787643,
+            -0.00406175072535136,
+            0.018353316401139956,
+            -0.007206833180919801,
+            -0.004174154024217347,
+            -0.004097578846173991,
+            -0.00573702992914836,
+            -0.0017184308802689716,
+            0.022375713882789285,
+            -0.008746034802967245,
+            0.032729322137817145,
+            -0.014835515993879445,
+            0.01049714579786462,
+            0.013344412726432523,
+            0.006468296817982878,
+            -0.0036641997047404426,
+            0.0003384623491490165,
+            -0.0035345323089769998,
+            -0.006579417470328358,
+            0.015682816949756224,
+            -0.018934655843627496,
+            0.006860259740057359,
+            0.03727162736659208,
+            -0.002536806673033066,
+            -0.00615871353663963,
+            -0.008853057427430264,
+            -0.012968501800661,
+            0.028268388127570685,
+            0.005935705363468247,
+            -0.0013092005585187737,
+            -0.03024974000885413,
+            0.015180604705329429,
+            -0.0037501687534873948,
+            0.0016387074669205704,
+            0.00860407823411825,
+            0.00044285387135976006,
+            0.020127437993717013,
+            -0.0011272951769363546,
+            0.021290480413673304,
+            0.010032359217391118,
+            -0.0010666180914844765,
+            0.011529468490024128,
+            -0.036558434912810016,
+            -0.008579049583792913,
+            0.0038048395503202733,
+            0.019444233855445248,
+            -0.0006623807457136332,
+            -0.015853103419331053,
+            -0.004242407392496502,
+            0.009029483146616182,
+            -0.01059695082306233,
+            -0.013492380798775181,
+            0.016259633969459952,
+            -0.019096544374819852,
+            0.011560512412214506,
+            0.008617334373339341,
+            0.02512750386847096,
+            0.0041202753709624245,
+            0.002613588241299414,
+            0.0035069606381226207,
+            0.006602772841623022,
+            -0.002625433099855357,
+            0.011789793979724633,
+            -0.011592113209370399,
+            0.02102479518464762,
+            -0.008294039267346376,
+            0.007068661518691129,
+            0.0032541671568032306,
+            -0.022166914870027622,
+            -0.018234159989683987,
+            -0.0023642465680105544,
+            0.021312775708470108,
+            0.006016177710817625,
+            -0.012618763344460982,
+            0.0055411096806088885,
+            0.004690341026678943,
+            -0.00547301082703577,
+            0.009118596747082881,
+            0.024398283179066196,
+            -0.0028098889343668772,
+            0.01068095200128235,
+            -0.007137857371825465,
+            -0.027832295014375272,
+            -0.007421428223393234,
+            0.021707007996690945,
+            0.00020946296332069377,
+            0.016793613517175475,
+            0.014396690762872469,
+            0.012889706504719996,
+            -0.00488698200749025,
+            -0.005354238088985329,
+            0.018317372626432005,
+            0.014340608961907827,
+            -0.0048105230366691145,
+            0.001495716612807014,
+            -0.0018529641162211795,
+            -0.017240719704808234,
+            0.026933447120216628,
+            -0.024232238634425658,
+            -0.003438477892192365,
+            0.00892792234782449,
+            -0.012586276728773793,
+            -0.004923824123059311,
+            -0.023468301644204742,
+            -0.02117100775850015,
+            0.014296432515879623,
+            -0.003314359969711106,
+            -0.01750062790930347,
+            -0.006856892040084997,
+            6.621834660421394e-05,
+            -0.003983545180909168,
+            -0.010826144842427663,
+            -0.0028995230412062357,
+            0.0018029406642546938,
+            -0.01286603500476852,
+            0.00937188453692113,
+            -0.00635685147895941,
+            0.010242542894487791,
+            0.0062482230941642404,
+            -0.022974002980633818,
+            0.005235079251883088,
+            0.020160136045340955,
+            -0.008953506137454955,
+            0.0007027144749750591,
+            -0.015318432934167311,
+            -0.006155069535076527,
+            -8.304773776972749e-05,
+            0.010665400866776814,
+            0.013561082526851403,
+            0.019453190615040503,
+            0.008724224096424808,
+            0.017716608184445207,
+            0.0035170519664773133,
+            -0.013169473712817174,
+            0.009862672596286218,
+            0.011645070824075182,
+            0.011745045118639314,
+            0.011761235635339118,
+            -0.0051506969158396755,
+            0.00628898514628358,
+            -0.0049857719357619505,
+            0.008525048536699957,
+            0.027507594807546505,
+            0.0013209667681023782,
+            0.017047984110417937,
+            0.016368796085341072,
+            0.020105811276523177,
+            -0.003759327296302425,
+            3.4626946806238876e-05,
+            0.00035688794161846735,
+            -0.001063984476108484,
+            0.003291497239446139,
+            -0.018279629509070614,
+            -0.01135428882896519,
+            0.017033095140582537,
+            -5.492833742539044e-05,
+            0.024685530931127835,
+            -0.0010090516352729089,
+            -0.0184210790691242,
+            -0.006072072044463014,
+            0.0019766255786695347,
+            -0.0143465314042477,
+            -0.00209268265697904,
+            0.008867797876246725,
+            0.02090777280180668,
+            0.007785784409274212,
+            -0.014694559759226521,
+            0.03266241399157856,
+            0.017596572028431588,
+            0.02301544904486291,
+            -0.02019106568899837,
+            -0.008116927224141509,
+            -0.007678137378694171,
+            0.020318874678520133,
+            -0.0068566512882342135,
+            -0.002510061555870801,
+            0.021473513752723834,
+            0.006645783892214728,
+            0.011211922100954015,
+            0.012286872605920514,
+            0.0003084092164612384,
+            0.0028492628853667647,
+            -0.015161921733689642,
+            0.011855756525919763,
+            -0.005289433905822296,
+            -0.006904878573181685,
+            0.021517401941638887,
+            -0.010444313612915583,
+            -0.009104739247715266,
+            0.005434542886627656,
+            -0.011460046711120444,
+            0.013324525803587403,
+            -0.0016717915349416276,
+            -0.013225789210352699,
+            -0.014868291792978007,
+            -0.004059908595020105,
+            -0.017086039690741954,
+            0.018935544362274584,
+            0.006810606440007176,
+            -0.0016747502037076753,
+            -0.005949836930379681,
+            0.004577809518378731,
+            -0.002387527459083855,
+            -0.0006415824062693734,
+            -0.015923707971787795,
+            0.0035047065077239213,
+            0.009700583796235219,
+            -0.0031324165789907116,
+            0.010643108490671052,
+            -0.013353561795075988,
+            0.033779753889958034,
+            -0.003236875179603599,
+            0.0033068852921624326,
+            0.005106828066670305,
+            -0.003034255184307467,
+            0.001481657575335246,
+            0.013519816427440722,
+            0.009066434673988428,
+            -0.008339461460990311,
+            -0.0009211583209818409,
+            0.013497039042975171,
+            -0.018668722901146233,
+            0.0013796912320793106,
+            0.013796986913155082,
+            -0.015849997808276967,
+            0.0049427770053857125,
+            0.011400118227502903,
+            -0.00639345486098034,
+            0.015340608222895147,
+            -0.01371015934577743,
+            0.007008681477695071,
+            0.009775206231977338,
+            0.028490233702896408,
+            -0.01966685875736091,
+            0.01928758485650401,
+            -0.025635008258084145,
+            0.031283028009357144,
+            0.012122242939206308,
+            -0.005796910058436286,
+            -0.025521126197222552,
+            -0.011593941553769327,
+            0.01283377684428954,
+            -0.021686338476615465,
+            0.0014883363235708173,
+            0.004930128822274494,
+            -0.013589466747020829,
+            -0.014766714233107183,
+            -0.01202283835955618,
+            0.011738578705966321,
+            0.0006399892787922201,
+            0.01589633486439762,
+            -0.001565909739326631,
+            0.006195453613357957,
+            0.00799709051068248,
+            -0.007262779704915372,
+            -0.006717513726969218,
+            -0.01982600498555257,
+            0.01452043414139846,
+            0.004256754663990435,
+            -0.005262312998149605,
+            -0.0224549938750886,
+            0.009397743096285409,
+            0.017707219262338467,
+            -0.005753255460862127,
+            0.013047488144162397,
+            -0.03383526087792666,
+            0.0009452367103428395,
+            -0.005822570832057675,
+            -0.0027621026437538738,
+            -0.011741384193082396,
+            0.005215022845011562,
+            -0.015301048137828595,
+            0.002594097201808009,
+            -0.006446736651739076,
+            -0.016963075831388622,
+            0.002297125878220545,
+            0.02099207084363147,
+            -0.0032941933998622765,
+            -0.0028818889706579017,
+            0.01784874229400347,
+            0.008499720521713671,
+            -0.01436426626087195,
+            -0.004000179664908013,
+            0.006397941984552902,
+            -0.018873859156915206,
+            0.0011599963694687036,
+            0.02198209878751397,
+            0.012652203334020664,
+            -0.026378312815268385,
+            -0.0016397712646032201,
+            0.013833745513458322,
+            0.008230921549345205,
+            0.008667892596847433,
+            0.0040942632709300665,
+            0.014593014365768302,
+            -0.006350891941908016,
+            -0.001374093276890415,
+            -6.3005496707736715e-06,
+            -0.006943874751919358,
+            0.011912879044321251,
+            0.003688355066059739,
+            0.040741622477833674,
+            -0.010639328654064204,
+            0.011006744363538067,
+            -0.01080331270092793,
+            -0.004477468739500037,
+            -0.004503151926350927,
+            0.0027311865257027917,
+            -0.013139063275489816,
+            -0.0001322676781706513,
+            0.0021051078165230533,
+            -0.0019788024041678294,
+            0.00649268452600941,
+            0.003305903531478554,
+            0.013176851471483939,
+            -0.017229299828998786,
+            -0.0024884474570636707,
+            -0.009910839696360717,
+            0.019347666346683084,
+            -0.010724923526200773,
+            0.01051871403061451,
+            0.03256893707453101,
+            0.013242686936629418,
+            0.00022445440868234016,
+            -0.014947304198470308,
+            -0.008836459838385262,
+            -0.008992977994567497,
+            -0.01739161854615299,
+            -0.012908166671408042,
+            -0.022654409684101404,
+            -0.00294050967213434,
+            0.006390611639719366,
+            -0.0010555036392130776,
+            -0.002785796833161689,
+            -0.027340941664129582,
+            -0.00792516100590229,
+            -0.03210910565574123,
+            -0.013679170743667392,
+            0.0056845041078574104,
+            -0.021558178294402857,
+            0.005195998493021035,
+            0.029445968279023734,
+            -0.008256723194649932,
+            0.018403154314213917,
+            0.006923789005793937,
+            0.011433558571124386,
+            -0.0022324062216246424,
+            -0.011870704203994723,
+            -0.011035736680889242,
+            0.00032924465177187114,
+            0.003455564411617082,
+            0.004938023824535148,
+            0.00916834019939669,
+            -0.01732811783799426,
+            -0.010720420451770168,
+            0.00025951769018448803,
+            0.01732173296763328,
+            -0.0036441011455210317,
+            -0.014654182169779525,
+            -0.009890543675136605,
+            0.00934755871928037,
+            -0.017560953324850772,
+            0.0028803276180087,
+            0.008144645056336303,
+            -0.006844751836577803,
+            0.009686357271083834,
+            0.00910269542748879,
+            -0.003634975911996508,
+            -0.01125116892179341,
+            0.015123292946443665,
+            0.011417498955214564,
+            -0.0005683452620824292,
+            0.009547451569266946,
+            0.0031907584166949686,
+            -0.004914343407341038,
+            0.0004946517266559977,
+            -0.013981094295278457,
+            -0.008438177812073363,
+            -0.009698303451884585,
+            -0.0004873949117926506,
+            0.008310275049300703,
+            -0.011244097579407173,
+            0.002589333020166005,
+            0.021677389796728952,
+            0.0016179416889548232,
+            -0.011231602157838225,
+            -0.0045296258349024725,
+            -0.03198651985714177,
+            -0.005752566699226788,
+            -0.017524199510403467,
+            0.014980719037744585,
+            -0.002529640898744345,
+            0.01028755110586937,
+            -0.0021848874276835985,
+            0.015817733088597956,
+            -0.005450667684071798,
+            0.012212046004369696,
+            -0.0008714643856749736,
+            -0.004560548606687623,
+            0.006746171236132563,
+            0.0034423786329218276,
+            -0.0009995673054631,
+            0.05643134636651925,
+            0.010296478797092773,
+            0.014377556149215558,
+            0.013825497434998718,
+            -0.017485595187900386,
+            0.013330532940316836,
+            0.013333976202885402,
+            0.012417102160127097,
+            0.004536479576179834,
+            -0.017256228281240734,
+            0.024337236913894427,
+            0.012755805964855683,
+            0.009622694631579495,
+            -0.0018516129725815202,
+            -0.0015877067248695194,
+            0.019827793457522933,
+            0.018266833138588977,
+            -0.009808363592736615,
+            0.011083663940891296,
+            0.004458465015627857,
+            -0.01068421391840819,
+            -0.0050807436529387556,
+            0.024563333063926475,
+            -0.01325209293057542,
+            -0.016539133573613984,
+            0.0059168302415277515,
+            -0.02488439856049784,
+            -0.029755917745609557,
+            0.001920092403971772,
+            0.006364691645397147,
+            0.002128416061225683,
+            -0.0021827593501562285,
+            0.00636889353893684,
+            0.01505350329221805,
+            -0.02223481012881443,
+            -0.01913380824474348,
+            0.004876437698740333,
+            -0.0031323018872010133,
+            0.015711985535422395,
+            -0.02142499388661488,
+            -0.011646214273628826,
+            0.003244037052013272,
+            -0.006889665577990765,
+            0.004421548112756594,
+            0.004704830748153097,
+            -0.004955793799612646,
+            -0.005239972361031197,
+            0.0007828188358756139,
+            0.019667585310240637,
+            0.017115475760556154,
+            0.02008255444898991,
+            0.012206187649127842,
+            -0.004915394518125827,
+            -0.00997481157928988,
+            0.002664275191354035,
+            -0.017453428141032505,
+            -0.005114569347643741,
+            0.006659824028894164,
+            0.013233844584497872,
+            -0.023794827807567007,
+            -0.007354308951883939,
+            0.012842257789962674,
+            0.011294252132532885,
+            -0.00824836528452984,
+            0.004256336058369505,
+            -0.011193345143979632,
+            0.009248664100690892,
+            0.010363170772580968,
+            0.010981447978991254,
+            -0.01250390246078834,
+            0.008163162620777558,
+            0.002528845973804195,
+            0.007003579605495149,
+            -0.005812703866552746,
+            0.008764871082692903,
+            0.013124123077340675,
+            -0.012212447674686537,
+            -0.0013940507123620205,
+            -0.001544873017903471,
+            -0.020493969663982807,
+            0.005782189204516662,
+            0.007033733269347455,
+            0.012878926833341835,
+            -0.008338295354334415,
+            0.015844568159504493,
+            -0.013423240193733334,
+            -0.00468920516522183,
+            0.0063319500880227525,
+            -0.004924778574104498,
+            -0.012053197667708071,
+            0.0017742020845283848,
+            -0.0035640186446405536,
+            -0.008666749197879684,
+            -0.0013894177719374007,
+            0.0016400693271668141,
+            0.004501038858023909,
+            0.02043221461176279,
+            0.001901506572332553,
+            -0.0056076433740056926,
+            0.019660750419391273,
+            0.003538773720733224,
+            -0.0018659146421562808,
+            0.005354994676719758,
+            0.0006789623595561136,
+            -0.0021516503031452773,
+            -0.0027566542393029298,
+            -0.010552067426241374,
+            0.01699044655141112,
+            -0.01883386136434379,
+            -0.006428123254736374,
+            -0.012514829515253224,
+            0.006923832110523631,
+            0.023065674483579466,
+            -0.0031719793455512423,
+            -0.006131653355136594,
+            -0.015640226612880566,
+            0.003589774262421098,
+            -0.07234225978543331,
+            -0.010636515019878534,
+            0.005436135654579842,
+            0.030362090167699224,
+            -0.008735845879763853,
+            0.0012226978854869956,
+            -0.00802187367908004,
+            0.010460901462092434,
+            -0.0024797678489701584,
+            0.005049311815982583,
+            0.0037643360764659196,
+            -0.003379308997651945,
+            -0.0029803517660911536,
+            0.0047612984744651015,
+            -0.010726918354479238,
+            -0.018870259480662628,
+            0.0028549213021684043,
+            0.025034678854420084,
+            0.009004274035878184,
+            0.004138885468102832,
+            0.009765532829693857,
+            -0.008537379113922283,
+            -0.0014902115840393893,
+            0.008462743183234337,
+            0.007328272099011593,
+            -0.009076904245691212,
+            -0.0008002047467975742,
+            0.008347212876126893,
+            -0.021799374855548816,
+            0.0020253943807330738,
+            0.012246389250647798,
+            -0.008912465890948629,
+            -0.006244789880293639,
+            0.002413038795382646,
+            0.004202974630810528,
+            -0.004339960146144246,
+            -0.009919301413657872,
+            -0.023862183602380987,
+            0.006400371798756263,
+            -0.007410957922309058,
+            -0.0001606112434129549,
+            0.0031662502228666104,
+            0.00012871401736865864,
+            -0.01687187870925491,
+            0.01374600947628664,
+            -0.008834671549515985,
+            -0.0033344289547729495,
+            0.012937173759658114,
+            -0.003267867578821276,
+            0.01864135643890992,
+            -0.014050228437162755,
+            0.007864998326137568,
+            -0.018988002335416778,
+            -0.003944450943320823,
+            -0.0032084001089808037,
+            -0.0023149926514803745,
+            -0.007750311931438483,
+            -0.045019596490520515,
+            0.013573650825957857,
+            -0.004374086208559833,
+            -0.005869005771503507,
+            -0.01250037134701792,
+            -0.016625210662158708,
+            0.011639132814162461,
+            0.004045997475412364,
+            -0.0014885281342871186,
+            -0.020321856013396592,
+            0.006222318025479057,
+            0.011121009394355636,
+            0.00904225037228108,
+            -0.011585148988670874,
+            -0.0018474735072673144,
+            -0.011325387916132704,
+            -0.0025890917877355725,
+            -0.009148717095934154,
+            0.018495348432506,
+            0.002404017871470538,
+            -0.010232069372010613,
+            0.022941087706292104,
+            -0.0024570420644715457,
+            0.020988341201251853,
+            -0.00442363926883869,
+            -0.004485898239018673,
+            -0.020447941631096866,
+            -0.004158168168969816,
+            -0.019358617104349624,
+            0.00036464612100595666,
+            0.0017749921060331022,
+            0.02078748568527368,
+            0.007256675717496499,
+            0.012612722550842472,
+            0.006032799184116686,
+            0.00631983142330159,
+            0.030151438614535002,
+            -0.010852408231267204,
+            0.0022451749744724233,
+            -0.017863828302314567,
+            0.010653806941450981,
+            0.028289026813870025,
+            0.009882042879445822,
+            -0.006920783006708277,
+            0.014053717954960316,
+            0.004932716665921658,
+            -0.00616100713292227,
+            0.025975267677767474,
+            -0.038836632659249616,
+            -0.021059874446788137,
+            -0.0061122622002863615,
+            0.001439949038430725,
+            -3.496872289060673e-05,
+            0.005030571386336924,
+            -0.011007922189367617,
+            0.009222155733548683,
+            0.0026655561130744807,
+            0.03101907459085608,
+            -0.002317540524509921,
+            -0.007348120500263025,
+            0.003987101796623003,
+            0.001511347369284844,
+            0.021229069065914295,
+            -0.01197457000398312,
+            -0.038429200853184124,
+            -0.0031917847963042626,
+            -0.0039488592790253305,
+            0.008371872140716302,
+            -0.012355524487988753,
+            -0.010839396359961,
+            -0.004574476927098761,
+            -0.01882180934875166,
+            -0.010292867074240718,
+            0.025595330740853118,
+            -0.007162452868646569,
+            0.01788200594893745,
+            -0.0076384034977797525,
+            -0.008112046063674713,
+            0.00683079906571602,
+            0.011572398977364099,
+            -0.01932794114157753,
+            0.0034055065356524052,
+            -0.0027824032743918262,
+            0.01627342501880651,
+            0.011718826886349412,
+            -0.013342112644751174,
+            0.006234666245077415,
+            0.01365078200475753,
+            -0.015476695744659024,
+            -0.0015378873781631448,
+            -0.0069121070205125585,
+            0.014866073481143516,
+            -0.003093423905641203,
+            -0.0033762696975023986,
+            0.011958295888528918,
+            -0.021982979614167224,
+            0.01697002117600754,
+            -0.02183323542526832,
+            0.022295922951583082,
+            -0.0006195453455557405,
+            -0.0006320640234089136,
+            0.008423668060539851,
+            0.008314680408345411,
+            -0.013378756052181496,
+            0.0016194432891316378,
+            -0.03416272433582183,
+            -0.009962172336151867,
+            -0.015051382650973002,
+            0.0038009542554177348,
+            -0.010106196779819915,
+            -0.01203081704994182,
+            -0.021869748479536547,
+            -0.013231917312580397,
+            0.009488867268928662,
+            0.02421155136770049,
+            -0.004349181861466944,
+            0.015665735941744613,
+            -0.008077879618601087,
+            -0.00346140349979159,
+            -0.019022501109520248,
+            -0.020158795455478176,
+            -0.011854460388465065,
+            -0.01570428154296937,
+            -0.007073290993912972,
+            0.004794071576762192,
+            -0.014492538032767992,
+            0.013391092874532436,
+            0.03615243857749714,
+            -0.017029836110560772,
+            0.008609740828901341,
+            0.019179133878629843,
+            0.0009923228421593012,
+            0.015393048602064972,
+            0.003790868078763006,
+            0.011710499244312703,
+            -0.014885526287290077,
+            -0.011829126494406653,
+            -0.0005627435441977781,
+            0.0020582596099707785,
+            0.003728052108395647,
+            0.027914733124690644,
+            0.0038604865752177283,
+            0.005302574635039923,
+            0.012293011832771443,
+            0.013320598642264739,
+            -0.010706185765391238,
+            0.015853648683536427,
+            0.004090821159159474,
+            -0.016980287717038522,
+            -0.0031197192883179663,
+            -0.006894138060845641,
+            -0.012739354473335233,
+            -0.002311031608856898,
+            0.0159372988694303,
+            -0.008091532959711314,
+            0.015169701976466646,
+            -0.004316674756425857,
+            -0.015090694840235596,
+            0.0007287957651103694,
+            -0.03810971752015664,
+            -0.0018729979445690102,
+            -0.01301867399306695,
+            -0.023136174288569147,
+            0.04241471786668668,
+            0.00415464531754789,
+            0.020836251111427635,
+            -0.006703690210920993,
+            0.006147438331654669,
+            -0.020673361951422737,
+            0.011783676068789637,
+            -0.0037516561524537765,
+            0.009702463136945385,
+            0.009366943837185086,
+            -0.018879598518030567,
+            -0.01132474738893142,
+            0.018484010977000243,
+            0.0008964474295952695,
+            0.001899853004269779,
+            0.014452586238316783,
+            0.019746940887134506,
+            0.01136527620507647,
+            -0.0020960934961233306,
+            0.02093114800466563,
+            0.018023169361765227,
+            -0.0002475129435970239,
+            -0.007118840223520255,
+            0.013527604253725194,
+            -0.02531530780276405,
+            0.014601163833111726,
+            0.03268673940622878,
+            0.013556815596615061,
+            0.020396267313433348,
+            -0.016161264086739215,
+            0.016259608464481445,
+            -0.006718110409512467,
+            0.017378727943854342,
+            0.001211924791251576,
+            0.01911265175070918,
+            0.014022657097776944,
+            0.01308266544853342,
+            0.006784756253744209,
+            -0.013367188766230505,
+            0.024435211011756285,
+            -0.008342072323415817,
+            -0.0004209982405190692,
+            -0.002823543536521797,
+            -0.023279946325307846,
+            0.01506543259962226,
+            0.0017095020186682082,
+            0.003411744637061027,
+            -0.0035317155649076414,
+            -0.001957702240702538,
+            0.008037874596341196,
+            0.008176809711854463,
+            -0.015780325305129808,
+            0.0031090160698441006,
+            -0.01924979849975192,
+            -0.008566500568783712,
+            -0.024111269290772643,
+            0.003704817310374187,
+            -0.015182531891766549,
+            -0.009971369164255839,
+            -0.019503345629597463,
+            -0.011275291123690537,
+            -0.010129027262987365,
+            0.00026549411615768147,
+            -0.0077553590287942975,
+            0.014297350661562049,
+            -0.017119422159171904,
+            -0.021341131823214652,
+            0.007945720599865192,
+            0.01439604029848557,
+            0.0037256533134793816,
+            0.005472351446140726,
+            0.008325407758477418,
+            0.009082034215555286,
+            -0.010830069811539129,
+            -0.010352768708989164,
+            0.001902868536460922,
+            0.0010307757364948558,
+            0.011572880679973516,
+            0.01480960939553474,
+            0.011328449881374384,
+            -0.013429607187664061,
+            0.0019326261581148884,
+            -0.014862631307250641,
+            0.004686987038977201,
+            -0.0011432874921856958,
+            -0.007026108142521568,
+            -0.0023122147402109927,
+            -0.01067305755615663,
+            0.008329587642130216,
+            0.0051774766682198365,
+            0.0014058609424645312,
+            0.0175095191841222,
+            -0.0016739751490535205,
+            -0.01295313808457633,
+            -0.012767766732175707,
+            -0.011653259281510757,
+            -0.015653450921388538,
+            0.006739771869002279,
+            -0.006908534840842188,
+            -0.006309735406143942,
+            0.00969748644524724,
+            -0.004300978285892996,
+            -0.0071082865141610775,
+            0.009870599453242536,
+            0.0003007194077865931,
+            -0.0011846789159672513,
+            0.03344372215649544,
+            -0.004482311849310301,
+            0.004153681423879411,
+            0.00022516282287549753,
+            -0.00096173441700326,
+            -0.01608715959652527,
+            -0.026963713856310283,
+            -0.0014323662880336336,
+            -0.02273696636049197,
+            0.017938234567205,
+            -0.0058065575851078514,
+            -0.001006759040376689,
+            0.002493842516193206,
+            -0.0042740143630253385,
+            -6.58295494211919e-05,
+            -0.011768915967276026,
+            0.0022431022211178383,
+            -0.029090994997079067,
+            0.009898341521414653,
+            -0.009544941198003441,
+            -0.02154260971685405,
+            0.01133866603878075,
+            0.040150135462722274,
+            -0.0034082812055093122,
+            0.0006477463440760081,
+            0.0010662318198694314,
+            0.010346430535548179,
+            0.014953186668393513,
+            -0.0019239187336010772,
+            -0.01603518108536549,
+            -0.010550241019652155,
+            0.007810013487665037,
+            0.0003366114893320438,
+            0.0030211673366186986,
+            0.025036691996012227,
+            -0.019097101590128545,
+            -0.016027314610013832,
+            -0.011102216330632247,
+            -0.011450582072890358,
+            0.009254748792585386,
+            0.019696746434919966,
+            0.013185751991632784,
+            -0.020709092762886617,
+            -0.017120964154502993,
+            0.00025737692261907426,
+            -0.02662183259737709,
+            -0.0015175233130328546,
+            0.12569047385271798,
+            -0.004250752280902458,
+            -0.009469510190778525,
+            -0.016304049953896647,
+            -0.00863767582150345,
+            0.02183050591695681,
+            -0.008697910752438377,
+            0.06936580824413162,
+            0.0006861269173917291,
+            -0.0029257096931296273,
+            -0.0020896063525880225,
+            -0.002993306834763656,
+            -0.013871927123029641,
+            0.001864949832879178,
+            -0.016847962023455045,
+            -0.01637518023758693,
+            -0.006332388047537449,
+            -0.013325712872379694,
+            -0.0017155737214029734,
+            -0.0061798136617018905,
+            0.004759205228314821,
+            -0.007830001529147273,
+            0.011495001353183024,
+            -0.026107152853762815,
+            -0.010130127689151806,
+            0.005962957535201191,
+            -0.011637257947784772,
+            0.0011307756612654312,
+            -0.030816554116329904,
+            -0.00441741097001083,
+            7.409036337997905e-05,
+            0.016501246370776126,
+            0.0015748302561177006,
+            -0.00940138978060357,
+            -0.011216292373500447,
+            0.006508185064247167,
+            0.021984022174755303,
+            0.02029850289834145,
+            0.020629050747183544,
+            0.009741513502990842,
+            0.017418812192334685,
+            -0.007031919478404256,
+            0.02625468226435525,
+            0.006196721544177288,
+            0.002206946290342702,
+            0.005596780938975919,
+            -0.01211153541496144,
+            -0.008534755574472074,
+            0.0074101096913398124,
+            -0.0219026467376419,
+            -0.0238583746438936,
+            -0.012260621283748488,
+            0.025739429552984337,
+            -0.009054977254128021,
+            -0.0019759724570709175,
+            -0.0030737356218219617,
+            0.024727831360321823,
+            -0.005149317224364814,
+            0.002332136352952536,
+            0.015587381251136284,
+            -0.028035348333895063,
+            0.008643405636450286,
+            0.007622365990511162,
+            0.02564089666732,
+            -0.0032690983288409517,
+            0.0030897564314706603,
+            -0.009711753474608342,
+            -0.009285462125807194,
+            -0.009613024258506932,
+            0.004240295765592783,
+            0.018961596475867332,
+            0.005797211858953558,
+            0.009922313409983964,
+            -0.006335322307636977,
+            -0.015248479856831972,
+            0.018517674622994848,
+            -0.018526204881631703,
+            -0.017805480495530986,
+            -0.0031014632675223346,
+            -0.023499333544416315,
+            0.007508363381003107,
+            0.006594465447638183,
+            -0.013304818586245387,
+            -0.0232989529893049,
+            0.020407788403667446,
+            0.003426820233878876,
+            -0.006520960224510217,
+            0.0028020004149410366,
+            -0.03282622678411901,
+            -0.018099884128291804,
+            0.004341732255376245,
+            0.012442902960644058,
+            -0.006914419410521592,
+            0.01684289728480369,
+            -0.0213314411022478,
+            0.021170682961259797,
+            -0.018444729425471068,
+            0.015400297173773751,
+            -0.005856807329212788,
+            0.01661347113847185,
+            0.011889835104373263,
+            0.0013585650928227913,
+            -0.019926344687386053,
+            0.02233564528528766,
+            0.0006518460244140545,
+            0.029977976063273147,
+            0.026669882018303386,
+            -0.005623183408671759,
+            -0.004497323576380724,
+            -0.017964594171161242,
+            0.011149712487916564,
+            0.0034380718823694635,
+            0.00295265329839474,
+            0.023835186931253683,
+            0.02803963164846445,
+            -0.028097235508129332,
+            0.017703040589712025,
+            -0.012893023698870347,
+            0.0089348477172039,
+            -0.011630331709306501,
+            0.0096768368423985,
+            0.00669273666692988,
+            0.003825890215203911,
+            0.008289398654583944,
+            0.011807275797649763,
+            0.024893949220958504,
+            0.02893733381281628,
+            0.004069052493622539,
+            0.012465630768347759,
+            -0.003600044227856378,
+            0.004954707230784128,
+            0.005985890100113585,
+            0.0014569711209781388,
+            -0.00648453455551435,
+            0.0031945554565833674,
+            -0.006090168119075504,
+            -0.003744222112779555,
+            0.015266602618825367,
+            -0.00028966361595811053,
+            0.015634624351841753,
+            0.012395295870300452,
+            -0.006577117978669073,
+            -0.003069372773552921,
+            -0.0029369830414480856,
+            -0.007598245840968154,
+            -0.017753359726409172,
+            0.025057809370078198,
+            -0.01606095813358513,
+            -0.019631204097884002,
+            0.0008782653868466668,
+            0.0031106969086903075,
+            -0.013675045629189149,
+            -0.008463196400259825,
+            0.008777824363912681,
+            0.014471929439779247,
+            -0.018478175346087953,
+            0.0017862044141924377,
+            -0.0004910793014860654,
+            0.013862986726603731,
+            -0.003641460118879092,
+            -0.00902817956483427,
+            -0.002076665354355939,
+            0.010393957588184604,
+            0.016413601414994997,
+            0.0038017609859922327,
+            -0.008644753777502864,
+            -0.0005484639867925445,
+            0.006452371249117759,
+            0.007391332924183688,
+            0.0036307379469783137,
+            -4.895894824382588e-05,
+            -0.008129811261078888,
+            -0.013655410823646597,
+            0.018829640339982075,
+            -0.007579460691893014,
+            0.016191139169828096,
+            -0.004984562798832502,
+            -0.011846989328030879,
+            0.004110855640416496,
+            0.0004775146965349116,
+            -0.014308885325290177,
+            0.002778951512640393,
+            -0.003583348080466357,
+            0.0017351729844533707,
+            0.006444402813802743,
+            -0.004945263366121632,
+            0.003803172980003619,
+            -0.008178062077842394,
+            0.02700102856098108,
+            0.015477045085093356,
+            0.005818362056942523,
+            -0.00876269298951217,
+            -0.024697543974427114,
+            0.008183329610246195,
+            0.00518473714350693,
+            0.018661120751406378,
+            0.014464207183253898,
+            0.0019868192486975638,
+            0.009827229842256728,
+            -0.013395371817208287,
+            -0.0022376892192351944,
+            0.009711801873474911,
+            0.017963581533215926,
+            -0.0059700736266576305,
+            0.019169874693002874,
+            -0.010316234316657931,
+            -0.0008125759237295765,
+            0.004003428870809722,
+            0.017109043212624506,
+            -0.004590883969173384,
+            0.013958072401225709
+          ],
+          "title": "F Pseudo Code for Switch Transformers 33",
+          "keyphrases": [
+            "Switch Transformers",
+            "Mixture-of-Experts",
+            "sparsely-activated expert model",
+            "natural language tasks",
+            "pre-training and fine-tuning"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": true,
+      "properties": {
+        "cosine_similarity": 0.8461173399164739
+      }
+    },
+    {
+      "id": "1f346181-4bd6-4596-8a44-75df4e078cd7",
+      "type": "cosine_similarity",
+      "source": {
+        "id": "14339b1f-53f8-479b-9da7-9430b174b9ee",
+        "properties": {
+          "page_content": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer\n\nNoam Shazeer1, Azalia Mirhoseini∗†1, Krzysztof Maziarz∗2, Andy Davis1, Quoc Le1, Geoffrey Hinton1and Jeff Dean1 1Google Brain, {noam,azalia,andydavis,qvl,geoffhinton,jeff}@google.com 2Jagiellonian University, Cracow, krzysztof.maziarz@student.uj.edu.pl\n\nAbstract\n\nThe capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost.\n\n1 Introduction And Related Work 1.1 Conditional Computation\n\nExploiting scale in both training data and model size has been central to the success of deep learning. When datasets are sufficiently large, increasing the capacity (number of parameters) of neural networks can give much better prediction accuracy. This has been shown in domains such as text (Sutskever et al., 2014; Bahdanau et al., 2014; Jozefowicz et al., 2016; Wu et al., 2016), images (Krizhevsky et al., 2012; Le et al., 2012), and audio (Hinton et al., 2012; Amodei et al., 2015). For typical deep learning models, where the entire model is activated for every example, this leads to a roughly quadratic blow-up in training costs, as both the model size and the number of training examples increase. Unfortunately, the advances in computing power and distributed computation fall short of meeting such demand.\n\nVarious forms of conditional computation have been proposed as a way to increase model capacity without a proportional increase in computational costs (Davis & Arel, 2013; Bengio et al., 2013; Eigen et al., 2013; Ludovic Denoyer, 2014; Cho & Bengio, 2014; Bengio et al., 2015; Almahairi et al., 2015). In these schemes, large parts of a network are active or inactive on a per-example basis. The gating decisions may be binary or sparse and continuous, stochastic or deterministic. Various forms of reinforcement learning and back-propagation are proposed for trarining the gating decisions.\n\nWhile these ideas are promising in theory, no work to date has yet demonstrated massive improvements in model capacity, training time, or model quality. We blame this on a combination of the following challenges:\n\nModern computing devices, especially GPUs, are much faster at arithmetic than at branching. Most of the works above recognize this and propose turning on/off large chunks of the network with each gating decision.\n\nLarge batch sizes are critical for performance, as they amortize the costs of parameter transfers and updates. Conditional computation reduces the batch sizes for the conditionally active chunks of the network.\n\nNetwork bandwidth can be a bottleneck. A cluster of GPUs may have computational power thousands of times greater than the aggregate inter-device network bandwidth. To be computationally efficient, the relative computational versus network demands of an algorithm must exceed this ratio. Embedding layers, which can be seen as a form of conditional computation, are handicapped by this very problem. Since the embeddings generally need to be sent across the network, the number of (example, parameter) interactions is limited by network bandwidth instead of computational capacity.\n\nDepending on the scheme, loss terms may be necessary to achieve the desired level of sparsity per-chunk and/or per example. Bengio et al. (2015) use three such terms. These issues can affect both model quality and load-balancing.\n\nModel capacity is most critical for very large data sets. The existing literature on conditional computation deals with relatively small image recognition data sets consisting of up to 600,000 images. It is hard to imagine that the labels of these images provide a sufficient signal to adequately train a model with millions, let alone billions of parameters. In this work, we for the first time address all of the above challenges and finally realize the promise of conditional computation. We obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets.\n\n1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1). All parts of the network are trained jointly by back-propagation.\n\nWhile the introduced technique is generic, in this paper we focus on language modeling and machine translation tasks, which are known to benefit from very large models. In particular, we apply a MoE convolutionally between stacked LSTM layers (Hochreiter & Schmidhuber, 1997), as in Figure 1.\n\nThe MoE is called once for each position in the text, selecting a potentially different combination of experts at each position. The different experts tend to become highly specialized based on syntax and semantics (see Appendix E Table 9). On both language modeling and machine translation benchmarks, we improve on best published results at a fraction of the computational cost.\n\n1.3 Related Work On Mixtures Of Experts\n\nSince its introduction more than two decades ago (Jacobs et al., 1991; Jordan & Jacobs, 1994), the mixture-of-experts approach has been the subject of much research. Different types of expert architectures hae been proposed such as SVMs (Collobert et al., 2002), Gaussian Processes (Tresp, 2001; Theis & Bethge, 2015; Deisenroth & Ng, 2015), Dirichlet Processes (Shahbaba & Neal, 2009), and deep networks. Other work has focused on different expert configurations such as a hierarchical structure (Yao et al., 2009), infinite numbers of experts (Rasmussen & Ghahramani, 2002), and adding experts sequentially (Aljundi et al., 2016). Garmash & Monz (2016) suggest an ensemble model in the format of mixture of experts for machine translation. The gating network is trained on a pre-trained ensemble NMT model. The works above concern top-level mixtures of experts. The mixture of experts is the whole model. Eigen et al. (2013) introduce the idea of using multiple MoEs with their own gating networks as parts of a deep model. It is intuitive that the latter approach is more powerful, since complex problems may contain many sub-problems each requiring different experts. They also allude in their conclusion to the potential to introduce sparsity, turning MoEs into a vehicle for computational computation.\n\nOur work builds on this use of MoEs as a general purpose neural network component. While Eigen et al. (2013) uses two stacked MoEs allowing for two sets of gating decisions, our convolutional application of the MoE allows for different gating decisions at each position in the text. We also realize sparse gating and demonstrate its use as a practical way to massively increase model capacity.\n\n2 The Structure Of The Mixture-Of-Experts Layer\n\nThe Mixture-of-Experts (MoE) layer consists of a set of n \"expert networks\" E1, · · · , En, and a \"gating network\" G whose output is a sparse n-dimensional vector. Figure 1 shows an overview of the MoE module. The experts are themselves neural networks, each with their own parameters.\n\nAlthough in principle we only require that the experts accept the same sized inputs and produce the same-sized outputs, in our initial investigations in this paper, we restrict ourselves to the case where the models are feed-forward networks with identical architectures, but with separate parameters.\n\nLet us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows:\n\n$$y=\\sum_{i=1}^{n}G(x){i}E{i}(x)$$ $$(\\mathbf{l})$$ G(x)iEi(x) (1) We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example. If the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network. In the following we focus on ordinary MoEs. We provide more details on hierarchical MoEs in Appendix B. Our implementation is related to other models of conditional computation. A MoE whose experts are simple weight matrices is similar to the parameterized weight matrix proposed in (Cho & Bengio, 2014). A MoE whose experts have one hidden layer is similar to the block-wise dropout described in (Bengio et al., 2015), where the dropped-out layer is sandwiched between fully-activated layers.\n\n2.1 Gating Network\n\nSoftmax Gating: A simple choice of non-sparse gating function (Jordan & Jacobs, 1994) is to multiply the input by a trainable weight matrix Wg and then apply the Sof tmax function.\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ $\\eqref{eq:walpha}$. Gσ(x) = Sof tmax(x · Wg) (2) Noisy Top-K Gating: We add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to −∞ (which causes the corresponding gate values to equal 0). The sparsity serves to save computation, as described above. While this form of sparsity creates some theoretically scary discontinuities in the output of gating function, we have not yet observed this to be a problem in practice. The noise term helps with load balancing, as will be discussed in Appendix A. The amount of noise per component is controlled by a second trainable weight matrix Wnoise.\n\n$$G(x)=S o f t m a x(K e e p T o p K(H(x),k))$$ $$(4)$$ $$(S)$$ G(x) = Sof tmax(KeepT opK(H(x), k)) (3) $$H(x){i}=(x\\cdot W{g}){i}+S t a n d a r d N o r m a l()\\cdot S o f t p l u s((x\\cdot W{n o i s e}){i})$$ $KeepTopK(v,k){i}=\\begin{cases}v_{i}&\\text{if}v_{i}\\text{is in the top}k\\text{elements of}v.\\ -\\infty&\\text{otherwise.}\\end{cases}$ Training the Gating Network We train the gating network by simple back-propagation, along with the rest of the model. If we choose k > 1, the gate values for the top k experts have nonzero derivatives with respect to the weights of the gating network. This type of occasionally-sensitive behavior is described in (Bengio et al., 2013) with respect to noisy rectifiers. Gradients also backpropagate through the gating network to its inputs. Our method differs here from (Bengio et al., 2015) who use boolean gates and a REINFORCE-style approach to train the gating network.\n\n3 Addressing Performance Challenges 3.1 The Shrinking Batch Problem\n\nOn modern CPUs and GPUs, large batch sizes are necessary for computational efficiency, so as to amortize the overhead of parameter loads and updates. If the gating network chooses k out of n experts for each example, then for a batch of b examples, each expert receives a much smaller batch of approximately kb n b examples. This causes a naive MoE implementation to become very inefficient as the number of experts increases. The solution to this shrinking batch problem is to make the original batch size as large as possible. However, batch size tends to be limited by the memory necessary to store activations between the forwards and backwards passes. We propose the following techniques for increasing the batch size: Mixing Data Parallelism and Model Parallelism: In a conventional distributed training setting, multiple copies of the model on different devices asynchronously process distinct batches of data, and parameters are synchronized through a set of parameter servers. In our technique, these different batches run synchronously so that they can be combined for the MoE layer. We distribute the standard layers of the model and the gating network according to conventional data-parallel schemes, but keep only one shared copy of each expert. Each expert in the MoE layer receives a combined batch consisting of the relevant examples from all of the data-parallel input batches. The same set of devices function as data-parallel replicas (for the standard layers and the gating networks) and as model-parallel shards (each hosting a subset of the experts). If the model is distributed over d devices, and each device processes a batch of size b, each expert receives a batch of approximately kbd nexamples. Thus, we achieve a factor of d improvement in expert batch size.\n\nIn the case of a hierarchical MoE (Section B), the primary gating network employs data parallelism, and the secondary MoEs employ model parallelism. Each secondary MoE resides on one device.\n\nThis technique allows us to increase the number of experts (and hence the number of parameters) by proportionally increasing the number of devices in the training cluster. The total batch size increases, keeping the batch size per expert constant. The memory and bandwidth requirements per device also remain constant, as do the step times, as does the amount of time necessary to process a number of training examples equal to the number of parameters in the model. It is our goal to train a trillionparameter model on a trillion-word corpus. We have not scaled our systems this far as of the writing of this paper, but it should be possible by adding more hardware.\n\nTaking Advantage of Convolutionality: In our language models, we apply the same MoE to each time step of the previous layer. If we wait for the previous layer to finish, we can apply the MoE to all the time steps together as one big batch. Doing so increases the size of the input batch to the MoE layer by a factor of the number of unrolled time steps.\n\nIncreasing Batch Size for a Recurrent MoE: We suspect that even more powerful models may involve applying a MoE recurrently. For example, the weight matrices of a LSTM or other RNN could be replaced by a MoE. Sadly, such models break the convolutional trick from the last paragraph, since the input to the MoE at one timestep depends on the output of the MoE at the previous timestep. Gruslys et al. (2016) describe a technique for drastically reducing the number of stored activations in an unrolled RNN, at the cost of recomputing forward activations. This would allow for a large increase in batch size.\n\n3.2 Network Bandwidth\n\nAnother major performance concern in distributed computing is network bandwidth. Since the experts are stationary (see above) and the number of gating parameters is small, most of the communication involves sending the inputs and outputs of the experts across the network. To maintain computational efficiency, the ratio of an expert's computation to the size of its input and output must exceed the ratio of computational to network capacity of the computing device. For GPUs, this may be thousands to one. In our experiments, we use experts with one hidden layer containing thousands of RELU-activated units. Since the weight matrices in the expert have sizes input_size×hidden_size and hidden_size × output_size, the ratio of computation to input and output is equal to the size of the hidden layer. Conveniently, we can increase computational efficiency simply by using a larger hidden layer, or more hidden layers.\n\n4 Balancing Expert Utilization\n\nWe have observed that the gating network tends to converge to a state where it always produces large weights for the same few experts. This imbalance is self-reinforcing, as the favored experts are trained more rapidly and thus are selected even more by the gating network. Eigen et al. (2013) describe the same phenomenon, and use a hard constraint at the beginning of training to avoid this local minimum. Bengio et al. (2015) include a soft constraint on the batch-wise average of each gate.1 We take a soft constraint approach. We define the importance of an expert relative to a batch of training examples to be the batchwise sum of the gate values for that expert. We define an additional loss Limportance, which is added to the overall loss function for the model. This loss is equal to the square of the coefficient of variation of the set of importance values, multiplied by a hand-tuned scaling factor wimportance. This additional loss encourages all experts to have equal importance.\n\n$$I m p o r t a n c e(X)=\\sum_{x\\in X}G(x)$$ $$L_{importance}(X)=w_{importance}\\cdot CV(Importance(X))^{2}\\tag{7}$$ $$(6)$$\n\nWhile this loss function can ensure equal importance, experts may still receive very different numbers of examples. For example, one expert may receive a few examples with large weights, and another may receive many examples with small weights. This can cause memory and performance problems on distributed hardware. To solve this problem, we introduce a second loss function, Lload , which ensures balanced loads. Appendix A contains the definition of this function, along with experimental results.\n\n5 Experiments 5.1 1 Billion Word Language Modeling Benchmark\n\nDataset: This dataset, introduced by (Chelba et al., 2013) consists of shuffled unique sentences from news articles, totaling approximately 829 million words, with a vocabulary of 793,471 words.\n\nPrevious State-of-the-Art: The best previously published results (Jozefowicz et al., 2016) use models consisting of one or more stacked Long Short-Term Memory (LSTM) layers (Hochreiter & Schmidhuber, 1997; Gers et al., 2000). The number of parameters in the LSTM layers of these models vary from 2 million to 151 million. Quality increases greatly with parameter count, as do computational costs. Results for these models form the top line of Figure 2-right. MoE Models: Our models consist of two stacked LSTM layers with a MoE layer between them (see Figure 1). We vary the sizes of the layers and the number of experts. For full details on model architecture, training regimen, additional baselines and results, see Appendix C.\n\nLow Computation, Varied Capacity: To investigate the effects of adding capacity, we trained\n\na series of MoE models all with roughly equal computational costs: about 8 million multiply-andadds per training example per timestep in the forwards pass, excluding the softmax layer. We call this metric (ops/timestep). We trained models with flat MoEs containing 4, 32, and 256 experts, and models with hierarchical MoEs containing 256, 1024, and 4096 experts. Each expert had about 1 million parameters. For all the MoE layers, 4 experts were active per input.\n\nThe results of these models are shown in Figure 2-left. The model with 4 always-active experts performed (unsurprisingly) similarly to the computationally-matched baseline models, while the largest of the models (4096 experts) achieved an impressive 24% lower perplexity on the test set. Figure 2: Model comparison on 1-Billion-Word Language-Modeling Benchmark. On the left, we plot test perplexity as a function of model capacity for models with similar computational budgets of approximately 8-million-ops-per-timestep. On the right, we plot test perplexity as a function of computational budget. The top line represents the LSTM models from (Jozefowicz et al., 2016).\n\nThe bottom line represents 4-billion parameter MoE models with different computational budgets.\n\nVaried Computation, High Capacity: In addition to the largest model from the previous section, we trained two more MoE models with similarly high capacity (4 billion parameters), but higher computation budgets. These models had larger LSTMs, and fewer but larger and experts. Details\n\nTest Test #Parameters ops/timestep Training TFLOPS Perplexity Perplexity excluding embedding Time /GPU 10 epochs 100 epochs and softmax layers 10 epochs Best Published Results 34.7 30.6 151 million 151 million 59 hours, 32 k40s 1.09 Low-Budget MoE Model 34.1 4303 million 8.9 million 15 hours, 16 k40s 0.74 Medium-Budget MoE Model 31.3 4313 million 33.8 million 17 hours, 32 k40s 1.22 High-Budget MoE Model 28.0 4371 million 142.7 million 47 hours, 32 k40s 1.56\n\nTable 1: Summary of high-capacity MoE-augmented models with varying computational budgets, vs. best previously published results (Jozefowicz et al., 2016). Details in Appendix C. can be found in Appendix C.2. Results of these three models form the bottom line of Figure 2-right.\n\nTable 1 compares the results of these models to the best previously-published result on this dataset .\n\nEven the fastest of these models beats the best published result (when controlling for the number of training epochs), despite requiring only 6% of the computation.\n\nComputational Efficiency: We trained our models using TensorFlow (Abadi et al., 2016) on clusters containing 16-32 Tesla K40 GPUs. For each of our models, we determine computational efficiency in TFLOPS/GPU by dividing the number of floating point operations required to process one training batch by the observed step time and the number of GPUs in the cluster. The operation counts used here are higher than the ones we report in our ops/timestep numbers in that we include the backwards pass, we include the importance-sampling-based training of the softmax layer, and we count a multiply-and-add as two separate operations. For all of our MoE models, the floating point operations involved in the experts represent between 37% and 46% of the total.\n\nFor our baseline models wtih no MoE, observed computational efficiency ranged from 1.07-1.29 TFLOPS/GPU. For our low-computation MoE models, computation efficiency ranged from 0.740.90 TFLOPS/GPU, except for the 4-expert model which did not make full use of the available parallelism. Our highest-computation MoE model was more efficient at 1.56 TFLOPS/GPU, likely due to the larger matrices. These numbers represent a significant fraction of the theoretical maximum of 4.29 TFLOPS/GPU claimed by NVIDIA. Detailed results are in Appendix C, Table 7.\n\n5.2 100 BILLION WORD GOOGLE NEWS CORPUS\n\nOn the 1-billion-word corpus, adding additional capacity seems to produce diminishing returns as the number of parameters in the MoE layer exceeds 1 billion, as can be seen in Figure 2-left. We hypothesized that for a larger training set, even higher capacities would produce significant quality improvements. We constructed a similar training set consisting of shuffled unique sentences from Google's internal news corpus, totalling roughly 100 billion words. Similarly to the previous section, we tested a series of models with similar computational costs of about 8 million ops/timestep. In addition to a baseline LSTM model, we trained models augmented with MoE layers containing 32, 256, 1024, 4096, 16384, 65536, and 131072 experts. This corresponds to up to 137 billion parameters in the MoE layer. Details on architecture, training, and results are given in Appendix D.\n\nResults: Figure 3 shows test perplexity as a function of capacity after training on 10 billion words (top line) and 100 billion words (bottom line). When training over the full 100 billion words, test perplexity improves significantly up to 65536 experts (68 billion parameters), dropping 39% lower than the computationally matched baseline, but degrades at 131072 experts, possibly a result of too much sparsity. The widening gap between the two lines demonstrates (unsurprisingly) that increased model capacity helps more on larger training sets. Even at 65536 experts (99.994% layer sparsity), computational efficiency for the model stays at a respectable 0.72 TFLOPS/GPU.\n\n5.3 Machine Translation (Single Language Pair)\n\nModel Architecture: Our model was a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decreased the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We inserted MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). Each MoE layer contained up to 2048 experts each with about two million parameters, adding a total of about 8 billion parameters to the models. Further details on model architecture, testing procedure and results can be found in Appendix E.\n\nDatasets: We benchmarked our method on the WMT'14 En→Fr and En→De corpora, whose training sets have 36M sentence pairs and 5M sentence pairs, respectively. The experimental protocols were also similar to those in (Wu et al., 2016): newstest2014 was used as the test set to compare against previous work (Luong et al., 2015a; Zhou et al., 2016; Wu et al., 2016), while the combination of newstest2012 and newstest2013 was used as the development set. We also tested the same model on a Google's Production English to French data.\n\nTable 2: Results on WMT'14 En→ Fr newstest2014 (bold values represent best results). Model Test Test ops/timenstep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.69 40.35 85M 8.7B 3 days/64 k40s MoE with 2048 Experts (longer training) 2.63 40.56 85M 8.7B 6 days/64 k40s GNMT (Wu et al., 2016) 2.79 39.22 214M 278M 6 days/96 k80s GNMT+RL (Wu et al., 2016) 2.96 39.92 214M 278M 6 days/96 k80s PBMT (Durrani et al., 2014) 37.0 LSTM (6-layer) (Luong et al., 2015b) 31.5 LSTM (6-layer+PosUnk) (Luong et al., 2015b) 33.1 DeepAtt (Zhou et al., 2016) 37.7 DeepAtt+PosUnk (Zhou et al., 2016) 39.2\n\nTable 3: Results on WMT'14 En → De newstest2014 (bold values represent best results). Model Test Test ops/timestep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 4.64 26.03 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 5.25 24.91 214M 278M 1 day/96 k80s GNMT +RL (Wu et al., 2016) 8.08 24.66 214M 278M 1 day/96 k80s PBMT (Durrani et al., 2014) 20.7 DeepAtt (Zhou et al., 2016) 20.6\n\nTable 4: Results on the Google Production En→ Fr dataset (bold values represent best results). Model Eval Eval Test Test ops/timestep Total Training Perplexity BLEU Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.60 37.27 2.69 36.57 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 2.78 35.80 2.87 35.56 214M 278M 6 days/96 k80s\n\nResults: Tables 2, 3, and 4 show the results of our largest models, compared with published results. Our approach achieved BLEU scores of 40.56 and 26.03 on the WMT'14 En→Fr and En→De benchmarks. As our models did not use RL refinement, these results constitute significant gains of 1.34 and 1.12 BLEU score on top of the strong baselines in (Wu et al., 2016). The perplexity scores are also better.2 On the Google Production dataset, our model achieved 1.01 higher test BLEU score even after training for only one sixth of the time.\n\n5.4 Multilingual Machine Translation\n\nDataset: (Johnson et al., 2016) train a single GNMT (Wu et al., 2016) model on a very large combined dataset of twelve language pairs. Results are somewhat worse than those for 12 separately trained single-pair GNMT models. This is not surprising, given that the twelve models have 12 times the capacity and twelve times the aggregate training of the one model. We repeat this experiment with a single MoE-augmented model. See Appendix E for details on model architecture.\n\nWe train our model on the same dataset as (Johnson et al., 2016) and process the same number of training examples (about 3 billion sentence pairs). Our training time was shorter due to the lower computational budget of our model.\n\nResults: Results for the single-pair GNMT models, the multilingual GNMT model and the multilingual MoE model are given in Table 5. The MoE model achieves 19% lower perplexity on the dev set than the multilingual GNMT model. On BLEU score, the MoE model significantly beats the multilingual GNMT model on 11 of the 12 language pairs (by as much as 5.84 points), and even beats the monolingual GNMT models on 8 of 12 language pairs. The poor performance on English → Korean seems to be a result of severe overtraining, as for the rarer language pairs a small number of real examples were highly oversampled in the training corpus.\n\nTable 5: Multilingual Machine Translation (bold values represent best results). GNMT-Mono GNMT-Multi MoE-Multi MoE-Multi vs. GNMT-Multi Parameters 278M / model 278M 8.7B ops/timestep 212M 212M 102M training time, hardware various 21 days, 96 k20s 12 days, 64 k40s Perplexity (dev) 4.14 3.35 -19% French → English Test BLEU 36.47 34.40 37.46 +3.06 German → English Test BLEU 31.77 31.17 34.80 +3.63 Japanese → English Test BLEU 23.41 21.62 25.91 +4.29 Korean → English Test BLEU 25.42 22.87 28.71 +5.84 Portuguese → English Test BLEU 44.40 42.53 46.13 +3.60 Spanish → English Test BLEU 38.00 36.04 39.39 +3.35 English → French Test BLEU 35.37 34.00 36.59 +2.59 English → German Test BLEU 26.43 23.15 24.53 +1.38 English → Japanese Test BLEU 23.66 21.10 22.78 +1.68 English → Korean Test BLEU 19.75 18.41 16.62 -1.79 English → Portuguese Test BLEU 38.40 37.35 37.90 +0.55 English → Spanish Test BLEU 34.50 34.25 36.21 +1.96\n\n6 Conclusion\n\nThis work is the first to demonstrate major wins from conditional computation in deep networks.\n\nWe carefully identified the design considerations and challenges of conditional computing and addressed them with a combination of algorithmic and engineering solutions. While we focused on text, conditional computation may help in other domains as well, provided sufficiently large training sets. We look forward to seeing many novel implementations and applications of conditional computation in the years to come.\n\nAcknowledgments\n\nWe would like to thank all of the members of the Google Brain and Google Translate teams who helped us with this project, in particular Zhifeng Chen, Yonghui Wu, and Melvin Johnson. Thanks also to our anonymous ICLR reviewers for the helpful suggestions on making this paper better.\n\n2Reported perplexities relative to the tokenization used by both our models and GNMT.\n\nReferences\n\nMartín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Gordon Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda B. Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. CoRR, abs/1603.04467, 2016. URL http://arxiv.org/abs/1603.04467.\n\nRahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. CoRR, abs/1611.06194, 2016. URL http://arxiv.org/abs/1611.\n\n06194.\n\nA. Almahairi, N. Ballas, T. Cooijmans, Y. Zheng, H. Larochelle, and A. Courville. Dynamic Capacity Networks. ArXiv e-prints, November 2015.\n\nDario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan, Christopher Fougner, Tony Han, Awni Y. Hannun, Billy Jun, Patrick LeGresley, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595, 2015. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.\n\nEmmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297, 2015.\n\nYoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.\n\nCiprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling.\n\narXiv preprint arXiv:1312.3005, 2013.\n\nK. Cho and Y. Bengio. Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning. ArXiv e-prints, June 2014. Ronan Collobert, Samy Bengio, and Yoshua Bengio. A parallel mixture of SVMs for very large scale problems. Neural Computing, 2002.\n\nAndrew Davis and Itamar Arel. Low-rank approximations for conditional feedforward computation in deep neural networks. arXiv preprint arXiv:1312.4461, 2013. Marc Peter Deisenroth and Jun Wei Ng. Distributed Gaussian processes. In ICML, 2015.\n\nJohn Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization, 2010.\n\nNadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. Edinburgh's phrase-based machine translation systems for wmt-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation, 2014.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013. Ekaterina Garmash and Christof Monz. Ensemble learning for multi-source neural machine translation. In staff.science.uva.nl/c.monz, 2016.\n\nFelix A. Gers, Jürgen A. Schmidhuber, and Fred A. Cummins. Learning to forget: Continual prediction with lstm. Neural Computation, 2000.\n\nAudrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. Memory-efficient backpropagation through time. CoRR, abs/1606.03401, 2016. URL http://arxiv.org/ abs/1606.03401. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2015.\n\nGeoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 2012. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.\n\nSergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.\n\nRobert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computing, 1991.\n\nMelvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation.\n\nCoRR, abs/1611.04558, 2016. URL http://arxiv.org/abs/1611.04558.\n\nMichael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm.\n\nNeural Computing, 1994. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.\n\nDiederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.\n\nReinhard Kneser and Hermann. Ney. Improved backingoff for m-gram language modeling., 1995.\n\nAlex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.\n\nQuoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeffrey Dean, and Andrew Y. Ng. Building high-level features using large scale unsupervised learning. In ICML, 2012. Patrick Gallinari Ludovic Denoyer. Deep sequential neural network. arXiv preprint arXiv:1410.0510, 2014.\n\nMinh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attentionbased neural machine translation. EMNLP, 2015a.\n\nMinh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the rare word problem in neural machine translation. ACL, 2015b.\n\nCarl Edward Rasmussen and Zoubin Ghahramani. Infinite mixtures of Gaussian process experts.\n\nNIPS, 2002.\n\nHasim Sak, Andrew W Senior, and Françoise Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH, pp. 338–342, 2014.\n\nMike Schuster and Kaisuke Nakajima. Japanese and Korean voice search. ICASSP, 2012.\n\nBabak Shahbaba and Radford Neal. Nonlinear models using dirichlet process mixtures. JMLR, 2009.\n\nIlya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks.\n\nIn NIPS, 2014.\n\nLucas Theis and Matthias Bethge. Generative image modeling using spatial LSTMs. In NIPS, 2015. Volker Tresp. Mixtures of Gaussian Processes. In NIPS, 2001.\n\nYonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.\n\nBangpeng Yao, Dirk Walther, Diane Beck, and Li Fei-fei. Hierarchical mixture of classification experts uncovers interactions between brain regions. In NIPS. 2009.\n\nWojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization.\n\narXiv preprint arXiv:1409.2329, 2014.\n\nJie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with fast-forward connections for neural machine translation. arXiv preprint arXiv:1606.04199, 2016.\n\nAppendices A Load-Balancing Loss\n\nAs discussed in section 4, for load-balancing purposes, we want to define an additional loss function to encourage experts to receive roughly equal numbers of training examples. Unfortunately, the number of examples received by an expert is a discrete quantity, so it can not be used in backpropagation. Instead, we define a smooth estimator Load(X) of the number of examples assigned to each expert for a batch X of inputs. The smoothness allows us to back-propagate gradients through the estimator. This is the purpose of the noise term in the gating function. We define P(x, i) as the probability that G(x)iis nonzero, given a new random choice of noise on element i, but keeping the already-sampled choices of noise on the other elements. To compute P(x, i), we note that the G(x)iis nonzero if and only if H(x)iis greater than the k th-greatest element of H(x) excluding itself. The probability works out to be:\n\n$$P(x,i)=P r\\Big((x\\cdot W_{g}){i}+S t a n d x N o r a l()\\cdot S o f t p u s((x\\cdot W{n o i s e})_{i})$$ $$>k t h_e x c l u d i n(H(x),k,i)\\Big)$$ $$({\\boldsymbol{8}})$$\n\n$$(9)$$ $$(10)$$ (8) Where kth_excluding(v, k, i) means the kth highest component of v, excluding component i. Simplifying, we get:\n\n$$P(x,i)=\\Phi{\\Big(}{\\frac{(x\\cdot W_{g}){i}-k t h_c x c l u d i n g(H(x),k,i)}{S o f t p l u s((x\\cdot W{n o i s e})_{i})}}{\\Big)}$$\n\nWhere Φ is the CDF of the standard normal distribution.\n\n$$L o a d(X){i}=\\sum{x\\in X}P(x,i)$$\n\n$$(11)$$ P(x, i) (10) We can now define the load loss to be the square of the coefficient of variation of the load vector, multiplied by a hand-tuned scaling factor wload.\n\n$$L_{l o a d}(X)=w_{l o a d}\\cdot C V(L o a d(X))^{2}$$ Lload(X) = wload · CV (Load(X))2(11) Initial Load Imbalance: To avoid out-of-memory errors, we need to initialize the network in a state of approximately equal expert load (since the soft constraints need some time to work). To accomplish this, we initialize the matrices Wg and Wnoise to all zeros, which yields no signal and some noise.\n\nExperiments: We trained a set of models with identical architecture (the MoE-256 model described in Appendix C), using different values of wimportance and wload. We trained each model for 10 epochs, then measured perplexity on the test set. We also measured the coefficients of variation in Importance and Load, as well as ratio of the load on the most overloaded expert to the average load. This last value is significant for load balancing purposes on distributed hardware. All of these metrics were averaged over several training batches.\n\nwimportance wload Test Perplexity CV (Importance(X)) CV (Load(X)) max(Load(X)) mean(Load(X)) 0.0 0.0 39.8 3.04 3.01 17.80 0.2 0.0 35.6 0.06 0.17 1.47 0.0 0.2 35.7 0.22 0.04 1.15 0.1 0.1 35.6 0.06 0.05 1.14 0.01 0.01 35.7 0.48 0.11 1.37 1.0 1.0 35.7 0.03 0.02 1.07\n\nTable 6: Experiments with different combinations of losses.\n\n13 Results: Results are reported in Table 6. All the combinations containing at least one the two losses led to very similar model quality, where having no loss was much worse. Models with higher values of wload had lower loads on the most overloaded expert.\n\nB Hierachical Mixture Of Experts\n\nIf the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network.3If the hierarchical MoE consists of a groups of b experts each, we denote the primary gating network by Gprimary, the secondary gating networks by (G1, G2..Ga), and the expert networks by (E0,0, E0,1..Ea,b). The output of the MoE is given by:\n\n$$y_{H}=\\sum_{i=1}^{a}\\sum_{j=1}^{b}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}\\cdot E{i,j}(x)$$ $$(12)$$ $$(13)$$ $$(14)$$\n\nOur metrics of expert utilization change to the following:\n\n$$I m p o r t a n c e_{H}(X){i,j}=\\sum{x\\in X}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}$$ $$L o a d{H}(X){i,j}={\\frac{L o a d{p r i m a r y}(X){i}\\cdot L o a d{i}(X^{(i)})_{j}}{|X^{(i)}|}}$$\n\nLoadprimary and Loadi deonte the Load functions for the primary gating network and i th secondary gating network respectively. X(i) denotes the subset of X for which Gprimary(x)i > 0.\n\nIt would seem simpler to let LoadH(X)i,j = Loadi(Xi)j , but this would not have a gradient with respect to the primary gating network, so we use the formulation above.\n\nC 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS C.1 8-MILLION-OPERATIONS-PER-TIMESTEP MODELS Model Architecture: Our model consists of five layers: a word embedding layer, a recurrent Long Short-Term Memory (LSTM) layer (Hochreiter & Schmidhuber, 1997; Gers et al., 2000), a MoE layer, a second LSTM layer, and a softmax layer. The dimensionality of the embedding layer, the number of units in each LSTM layer, and the input and output dimensionality of the MoE layer are all equal to 512. For every layer other than the softmax, we apply drouput (Zaremba et al., 2014) to the layer output, dropping each activation with probability DropP rob, otherwise dividing by (1 − DropP rob). After dropout, the output of the previous layer is added to the layer output.\n\nThis residual connection encourages gradient flow (He et al., 2015).\n\nMoE Layer Architecture: Each expert in the MoE layer is a feed forward network with one ReLU-activated hidden layer of size 1024 and an output layer of size 512. Thus, each expert contains [512 ∗ 1024] + [1024 ∗ 512] = 1M parameters. The output of the MoE layer is passed through a sigmoid function before dropout. We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts.\n\nWe call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers. Thus, each example is processed by exactly 4 experts for a total of 4M ops/timestep. The two LSTM layers contribute 2M ops/timestep each for the desired total of 8M.\n\n3 We have not found the need for deeper hierarchies.\n\nComputationally-Matched Baselines: The MoE-4 model does not employ sparsity, since all 4 experts are always used. In addition, we trained four more computationally-matched baseline models with no sparsity:\n\nMoE-1-Wide: The MoE layer consists of a single \"expert\" containing one ReLU-activated hidden layer of size 4096.\n\nMoE-1-Deep: The MoE layer consists of a single \"expert\" containing four ReLU-activated hidden layers, each with size 1024.\n\n4xLSTM-512: We replace the MoE layer with two additional 512-unit LSTM layers.\n\nLSTM-2048-512: The model contains one 2048-unit LSTM layer (and no MoE). The output of the LSTM is projected down to 512 dimensions (Sak et al., 2014). The next timestep of the LSTM receives the projected output. This is identical to one of the models published in (Jozefowicz et al., 2016). We re-ran it to account for differences in training regimen, and obtained results very similar to the published ones. Training: The models were trained on a cluster of 16 K40 GPUs using the synchronous method described in Section 3. Each batch consisted of a set of sentences totaling roughly 300,000 words. In the interest of time, we limited training to 10 epochs, (27,000 steps). Training took 12-16 hours for all models, except for MoE-4, which took 18 hours (since all the expert computation was performed on only 4 of 16 GPUs). We used the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 1000 training steps, and decreased after that so as to be proportional to the inverse square root of the step number. The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016). For each model, we performed a hyper-parmeter search to find the best dropout probability, in increments of 0.1.\n\nTo ensure balanced expert utilization we set wimportance = 0.1 and wload = 0.1, as described in Section 4 and Appendix A.\n\nResults: We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol. Results are reported in Table 7. For each model, we report the test perplexity, the computational budget, the parameter counts, the value of DropP rob, and the computational efficiency.\n\nModel Test Test ops/timestep #Params excluding Total Drop- TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params P rob per GPU 10 epochs (final) (millions) (billions) (observed) Kneser-Ney 5-gram* 67.6 0.00001 1.8 LSTM-512-512* 54.1 2.4 2.4 0.8 0.1 LSTM-1024-512* 48.2 4.7 4.7 0.8 0.1 LSTM-2048-512* 45.0 43.7 9.4 9.4 0.8 0.1 0.61 LSTM-2048-512 44.7 9.4 9.4 0.8 0.1 1.21 4xLSTM-512 46.0 8.4 8.4 0.8 0.1 1.07 MoE-1-Wide 46.1 8.4 8.4 0.8 0.1 1.29 MoE-1-Deep 45.7 8.4 8.4 0.8 0.1 1.29 MoE-4 45.0 8.4 8.4 0.8 0.1 0.52 MoE-32 39.7 8.4 37.8 0.9 0.1 0.87 MoE-256 35.7 8.6 272.9 1.1 0.1 0.81 MoE-256-h 36.0 8.4 272.9 1.1 0.1 0.89 MoE-1024-h 34.6 8.5 1079.0 1.9 0.2 0.90 MoE-4096-h 34.1 8.9 4303.4 5.1 0.2 0.74 2xLSTM-8192-1024* 34.7 30.6 151.0 151.0 1.8 0.25 1.09 MoE-34M 31.3 33.8 4313.9 6.0 0.3 1.22 MoE-143M 28.0 142.7 4371.1 6.0 0.4 1.56\n\nC.2 More Expensive Models\n\nWe ran two additional models (MoE-34M and MoE-143M) to investigate the effects of adding more computation in the presence of a large MoE layer. These models have computation budgets of 34M and 143M ops/timestep. Similar to the models above, these models use a MoE layer between two LSTM layers. The dimensionality of the embedding layer, and the input and output dimensionality of the MoE layer are set to 1024 instead of 512. For MoE-34M, the LSTM layers have 1024 units.\n\nFor MoE-143M, the LSTM layers have 4096 units and an output projection of size 1024 (Sak et al., 2014). MoE-34M uses a hierarchical MoE layer with 1024 experts, each with a hidden layer of size 2048. MoE-143M uses a hierarchical MoE layer with 256 experts, each with a hidden layer of size 8192. Both models have 4B parameters in the MoE layers. We searched for the best DropP rob for each model, and trained each model for 10 epochs.\n\nThe two models achieved test perplexity of 31.3 and 28.0 respectively, showing that even in the presence of a large MoE, more computation is still useful. Results are reported at the bottom of Table 7. The larger of the two models has a similar computational budget to the best published model from the literature, and training times are similar. Comparing after 10 epochs, our model has a lower test perplexity by 18%.\n\nD 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS Model Architecture: The models are similar in structure to the 8-million-operations-per-timestep models described in the previous section. We vary the number of experts between models, using an ordinary MoE layer with 32 experts and hierarchical MoE layers with 256, 1024, 4096, 16384, 65536 and 131072 experts. For the hierarchical MoE layers, the first level branching factors are 32, 32, 64, 128, 256 and 256, respectively.\n\nTraining: Models are trained on a cluster of 32 Tesla K40 GPUs, except for the last two models, which are trained on clusters of 64 and 128 GPUs so as to have enough memory for all the parameters. For all models, training batch sizes are approximately 2.5 million words. Models are trained once-through over about 100 billion words. We implement several memory optimizations in order to fit up to 1 billion parameters per GPU. First, we do not store the activations of the hidden layers of the experts, but instead recompute them on the backwards pass. Secondly, we modify the optimizer on the expert parameters to require less auxiliary storage: The Adam optimizer (Kingma & Ba, 2015) keeps first and second moment estimates of the perparameter gradients. This triples the required memory. To avoid keeping a first-moment estimator, we set β1 = 0. To reduce the size of the second moment estimator, we replace it with a factored approximation. For a matrix of parameters, instead of maintaining a full matrix of second-moment estimators, we maintain vectors of row-wise and column-wise averages of that matrix. At each step, the matrix of estimators is taken to be the outer product of those two vectors divided by the mean of either one. This technique could similarly be applied to Adagrad (Duchi et al., 2010).\n\nModel Test Test ops/timestep #Params excluding Total TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params per GPU .1 epochs 1 epoch (millions) (billions) (observed) Kneser-Ney 5-gram 67.1 45.3 0.00001 76.0 4xLSTM-512 54.5 47.0 8.4 8.4 0.1 1.23 MoE-32 48.5 40.4 8.4 37.8 0.1 0.83 MoE-256-h 42.8 35.3 8.4 272.9 0.4 1.11 MoE-1024-h 40.3 32.7 8.5 1079.0 1.2 1.14 MoE-4096-h 38.9 30.9 8.6 4303.4 4.4 1.07 MoE-16384-h 38.2 29.7 8.8 17201.0 17.3 0.96 MoE-65536-h 38.2 28.9 9.2 68791.0 68.9 0.72 MoE-131072-h 39.8 29.2 9.7 137577.6 137.7 0.30\n\nResults: We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8. Perplexity after 100 billion training words is 39% lower for the 68-billion-parameter MoE model than for the baseline model. It is notable that the measured computational efficiency of the largest model (0.30 TFLOPS/GPU) is very low compared to the other models. This is likely a result of the fact that, for purposes of comparison to the other models, we did not increase the training batch size proportionally to the number of GPUs. For comparison, we include results for a computationally matched baseline model consisting of 4 LSTMs, and for an unpruned 5-gram model with Kneser-Ney smoothing (Kneser & Ney, 1995).4\n\nE Machine Translation - Experimental Details\n\nModel Architecture for Single Language Pair MoE Models: Our model is a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decrease the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We insert MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). We use an attention mechanism between the encoder and decoder, with the first decoder LSTM receiving output from and providing input for the attention 5. All of the layers in our model have input and output dimensionality of 512. Our LSTM layers have 2048 hidden units, with a 512-dimensional output projection. We add residual connections around all LSTM and MoE layers to encourage gradient flow (He et al., 2015). Similar to GNMT, to effectively deal with rare words, we used subword units (also known as \"wordpieces\") (Schuster & Nakajima, 2012) for inputs and outputs in our system.\n\nWe use a shared source and target vocabulary of 32K wordpieces. We also used the same beam search technique as proposed in (Wu et al., 2016).\n\nWe train models with different numbers of experts in the MoE layers. In addition to a baseline model with no MoE layers, we train models with flat MoE layers containing 32 experts, and models with hierarchical MoE layers containing 512 and 2048 experts. The flat MoE layers use k = 4 and the hierarchical MoE models use k = 2 at each level of the gating network. Thus, each input is processed by exactly 4 experts in each MoE layer. Each expert in the MoE layer is a feed forward network with one hidden layer of size 2048 and ReLU activation. Thus, each expert contains [512 ∗ 2048] + [2048 ∗ 512] = 2M parameters. The output of the MoE layer is passed through a sigmoid function. We use the strictly-balanced gating function described in Appendix F.\n\nModel Architecture for Multilingual MoE Model: We used the same model architecture as for the single-language-pair models, with the following exceptions: We used noisy-top-k gating as described in Section 2.1, not the scheme from Appendix F. The MoE layers in the encoder and decoder are non-hierarchical MoEs with n = 512 experts, and k = 2. Each expert has a larger hidden layer of size 8192. This doubles the amount of computation in the MoE layers, raising the computational budget of the entire model from 85M to 102M ops/timestep. Training: We trained our networks using the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 2000 training steps, held constant for an additional 8000 steps, and decreased after that so as to be proportional to the inverse square root of the step number. For the single-language-pair models, similarly to (Wu et al., 2016), we applied dropout (Zaremba et al., 2014) to the output of all embedding, LSTM and MoE layers, using DropP rob = 0.4. Training was done synchronously on a cluster of up to 64 GPUs as described in section 3. Each training batch consisted of a set of sentence pairs containing roughly 16000 words per GPU.\n\nTo ensure balanced expert utilization we set wimportance = 0.01 and wload = 0.01, as described in Section 4 and Appendix A. Metrics: We evaluated our models using the perplexity and the standard BLEU score metric. We reported tokenized BLEU score as computed by the multi-bleu.pl script, downloaded from the public implementation of Moses (on Github), which was also used in (Luong et al., 2015a). Results: Tables 2, 3 and 4 in Section 5.3 show comparisons of our results to other published methods. Figure 4 shows test perplexity as a function of number of words in the (training data's) source sentences processed for models with different numbers of experts. As can be seen from the Figure, as we increased the number of experts to approach 2048, the test perplexity of our model continued to improve.\n\nWe found that the experts indeed become highly specialized by syntax and/or semantics, as can be seen in Table 9. For example, one expert is used when the indefinite article \"a\" introduces the direct object in a verb phrase indicating importance or leadership.\n\nExpert 381 Expert 752 Expert 2004 ... with researchers , ... ... plays a core ... ... with rapidly growing ... ... to innovation . ... plays a critical ... ... under static conditions ... ... tics researchers . ... provides a legislative ... ... to swift ly ... ... the generation of ... ... play a leading ... ... to dras tically ... ... technology innovations is ... ... assume a leadership ... ... the rapid and ... ... technological innovations , ... ... plays a central ... ... the fast est ... ... support innovation throughout ... ... taken a leading ... ... the Quick Method ... ... role innovation will ... ... established a reconciliation ... ... rec urrent ) ... ... research scienti st ... ... played a vital ... ... provides quick access ... ... promoting innovation where ... ... have a central ... ... of volatile organic ... ... ... ...\n\nDue to some peculiarities in our infrastructure which have since been fixed, at the time we ran some of the machine translation experiments, our models ran faster if every expert received exactly the same batch size. To accommodate this, we used a different gating function which we describe below. Recall that we define the softmax gating function to be:\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ Gσ(x) = Sof tmax(x · Wg) (15) Sparse Gating (alternate formulation): To obtain a sparse gating vector, we multiply Gσ(x) component-wise with a sparse mask M(Gσ(x)) and normalize the output. The mask itself is a function of Gσ(x) and specifies which experts are assigned to each input example:\n\n$$(15)$$ $$G(x){i}=\\frac{G{\\sigma}(x){i}M(G{\\sigma}(x)){i}}{\\sum{j=1}^{n}G_{\\sigma}(x){j}M(G{\\sigma}(x))_{j}}\\tag{1}$$ $$(17)$$\n\n$$(16)$$\n\nTop-K Mask: To implement top-k gating in this formulation, we would let M(v) = T opK(v, k), where:\n\n$$T o p K(v,k){i}=\\begin{cases}1&{\\mathrm{if~}}v{i}{\\mathrm{~is~in~the~top~}}k{\\mathrm{~elements~of~}}v.\\ 0&{\\mathrm{otherwise.}}\\end{cases}$$ 0 otherwise. (17) Batchwise Mask: To force each expert to receive the exact same number of examples, we introduce an alternative mask function, Mbatchwise(X, m), which operates over batches of input vectors.\n\nInstead of keeping the top k values per example, we keep the top m values per expert across the training batch, where m = k|X| n, so that each example is sent to an average of k experts.\n\n$$M_{b a t c h w i s e}(X,m){j,i}=\\begin{cases}1&\\text{if}X{j,i}\\text{is in the top}m\\text{values for to expert}i\\ 0&\\text{otherwise}\\end{cases}$$\n\nAs our experiments suggest and also observed in (Ioffe & Szegedy, 2015), using a batchwise function during training (such as Mbatchwise) requires modifications to the inference when we may not have a large batch of examples. Our solution to this is to train a vector T of per-expert threshold values to approximate the effects of the batchwise mask. We use the following mask at inference time:\n\n$$(18)$$ $$M_{threshold}(x,T){i}=\\begin{cases}1&\\text{if}x{i}>T_{i}\\ 0&\\text{otherwise}\\end{cases}\\tag{1}$$ $$(19)$$\n\nTo learn the threshold values, we apply an additional loss at training time which is minimized when the batchwise mask and the threshold mask are identical.\n\n$$L_{batchwise}(X,T,m)=\\sum_{j=1}^{|X|}\\sum_{i=1}^{n}(M_{threshold}(x,T){i}-M{batchwise}(X,m){j,i})(X{j,i}-T_{i})\\tag{20}$$\n\nG ATTENTION FUNCTION The attention mechanism described in GNMT (Wu et al., 2016) involves a learned \"Attention Function\" A(xi, yj ) which takes a \"source vector\" xi and a \"target vector\" yj , and must be computed for every source time step i and target time step j. In GNMT, the attention function is implemented as a feed forward neural network with a hidden layer of size n. It can be expressed as:\n\n$$A_{G N M T}(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}t a n h((x_{i}U){d}+(y{j}W)_{d})$$\n\nWhere U and W are trainable weight matrices and V is a trainable weight vector.\n\nFor performance reasons, in our models, we used a slightly different attention function:\n\n$$A(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}tanh((x_{i}U){d})tanh((y{j}W)_{d})\\tag{22}$$\n\nWith our attention function, we can simultaneously compute the attention function on multiple source time steps and multiple target time steps using optimized matrix multiplications. We found little difference in quality between the two functions.\n\n$$(21)$$",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/1701.06538v1.md"
+          },
+          "headlines": {
+            "1 Introduction And Related Work": [
+              "1.1 Conditional Computation",
+              "1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER",
+              "1.3 Related Work On Mixtures Of Experts"
+            ],
+            "2 The Structure Of The Mixture-Of-Experts Layer": [
+              "2.1 Gating Network"
+            ],
+            "3 Addressing Performance Challenges": [
+              "3.1 The Shrinking Batch Problem",
+              "3.2 Network Bandwidth"
+            ],
+            "4 Balancing Expert Utilization": [],
+            "5 Experiments": [
+              "5.1 1 Billion Word Language Modeling Benchmark",
+              "5.2 100 BILLION WORD GOOGLE NEWS CORPUS",
+              "5.3 Machine Translation (Single Language Pair)",
+              "5.4 Multilingual Machine Translation"
+            ],
+            "6 Conclusion": [],
+            "Appendices": [
+              "A Load-Balancing Loss",
+              "B Hierachical Mixture Of Experts",
+              "C 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS",
+              "D 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS",
+              "E Machine Translation - Experimental Details"
+            ]
+          },
+          "summary": "The paper introduces a Sparsely-Gated Mixture-of-Experts (MoE) layer to enhance neural network capacity without a proportional increase in computation. This approach uses a trainable gating network to select a sparse combination of experts for each input, significantly improving model capacity and performance in language modeling and machine translation tasks. The MoE layer, with up to 137 billion parameters, achieves better results than state-of-the-art models at lower computational costs. The paper addresses challenges like batch size and network bandwidth, proposing solutions like mixing data and model parallelism. Experiments show that MoE models outperform traditional models in various benchmarks, demonstrating the potential of conditional computation in deep learning.",
+          "summary_embedding": [
+            -0.01644345559179783,
+            -0.016995344310998917,
+            -0.016372699290513992,
+            -0.014349107630550861,
+            0.02282555401325226,
+            0.045934125781059265,
+            -0.014646278694272041,
+            0.03563220426440239,
+            -0.004829026758670807,
+            0.005617944523692131,
+            -0.00932550523430109,
+            -0.008143897168338299,
+            0.005575491581112146,
+            0.01538213063031435,
+            -0.005653322208672762,
+            -0.020306676626205444,
+            -0.013832597061991692,
+            -0.023235931992530823,
+            0.028344441205263138,
+            -0.020080260932445526,
+            0.011823155917227268,
+            -0.020943470299243927,
+            -0.025868017226457596,
+            -0.016500059515237808,
+            0.025061409920454025,
+            0.024806693196296692,
+            0.02583971433341503,
+            0.003827843815088272,
+            -0.03492465242743492,
+            0.033651065081357956,
+            0.014674580655992031,
+            0.012488252483308315,
+            -0.031500112265348434,
+            -0.009162768721580505,
+            -0.05351905897259712,
+            -0.05227377265691757,
+            0.04143410921096802,
+            0.022245362401008606,
+            -0.016231190413236618,
+            0.023717066273093224,
+            0.017264213413000107,
+            0.005816058721393347,
+            -0.020632149651646614,
+            -0.020844414830207825,
+            0.003700484987348318,
+            -0.013103820383548737,
+            -0.01983969286084175,
+            -0.013528349809348583,
+            -0.01828308403491974,
+            -0.01577835902571678,
+            0.009551920928061008,
+            0.0002856731880456209,
+            0.0029186424799263477,
+            -0.003424540627747774,
+            -0.047830358147621155,
+            0.0248208437114954,
+            0.019853845238685608,
+            0.03492465242743492,
+            0.04633035510778427,
+            0.013011839240789413,
+            0.01516986545175314,
+            -0.014271277002990246,
+            -0.04021712392568588,
+            0.005497660953551531,
+            -0.00041656987741589546,
+            -0.005380915477871895,
+            -0.005819596350193024,
+            0.008575502783060074,
+            0.008724087849259377,
+            -0.01143400277942419,
+            -0.04202845320105553,
+            0.013952880166471004,
+            -0.020561393350362778,
+            0.00010143388499272987,
+            0.01354957651346922,
+            0.008292482234537601,
+            0.02677368000149727,
+            0.03195294365286827,
+            0.023971782997250557,
+            -0.008724087849259377,
+            -0.01340806670486927,
+            0.019924599677324295,
+            -0.0022676968947052956,
+            0.007018893025815487,
+            0.04570771008729935,
+            -0.04033033177256584,
+            0.02163686975836754,
+            -0.025726506486535072,
+            -0.015339677222073078,
+            -0.005349075887352228,
+            -0.022471778094768524,
+            0.02806142158806324,
+            -0.007344366051256657,
+            0.017801949754357338,
+            0.017094401642680168,
+            0.01594817079603672,
+            0.013330236077308655,
+            -0.001894464367069304,
+            -0.022938761860132217,
+            0.0018360915128141642,
+            -0.01541043259203434,
+            0.02114158496260643,
+            0.0015230007702484727,
+            0.005469358991831541,
+            0.027594437822699547,
+            0.031160488724708557,
+            -0.023490650579333305,
+            0.009502392262220383,
+            0.02422650158405304,
+            0.06967949122190475,
+            0.015424583107233047,
+            -0.006435164716094732,
+            -0.0024640418123453856,
+            0.04143410921096802,
+            0.0015513027319684625,
+            0.0038349193055182695,
+            -0.06826439499855042,
+            -0.03022652305662632,
+            -0.0011365016689524055,
+            -0.02254253253340721,
+            0.02227366529405117,
+            0.01002597901970148,
+            -0.009750034660100937,
+            -0.022457627579569817,
+            -0.04516997188329697,
+            0.05190584436058998,
+            -0.00545874610543251,
+            -0.016259491443634033,
+            -0.005554265342652798,
+            -0.05340585112571716,
+            0.00039136342820711434,
+            0.015042506158351898,
+            -0.018481198698282242,
+            -0.03670767694711685,
+            0.036141637712717056,
+            0.0007398316520266235,
+            0.03741522878408432,
+            0.012459950521588326,
+            -0.01708024926483631,
+            -0.020094411447644234,
+            0.022259512916207314,
+            -0.011639192700386047,
+            -0.0038313816767185926,
+            -0.027339721098542213,
+            0.0065236082300543785,
+            -0.028103873133659363,
+            -0.0014566679019480944,
+            0.06730213016271591,
+            -0.04816998168826103,
+            0.02946236915886402,
+            -0.029660483822226524,
+            0.023660462349653244,
+            -0.008575502783060074,
+            0.0278916098177433,
+            -0.010238244198262691,
+            0.005840822588652372,
+            -0.013457595370709896,
+            -0.0010586712742224336,
+            -0.020745357498526573,
+            0.0034210027661174536,
+            -0.016259491443634033,
+            -0.027750099077820778,
+            -0.007054270710796118,
+            0.031273696571588516,
+            0.01698119193315506,
+            0.025655752047896385,
+            0.025429336354136467,
+            0.01246702577918768,
+            -0.04935866594314575,
+            0.02595292218029499,
+            0.0018926954362541437,
+            0.013514199294149876,
+            0.013181651011109352,
+            -0.0010418669553473592,
+            0.025599148124456406,
+            0.012049571610987186,
+            0.025061409920454025,
+            0.037443529814481735,
+            0.0027682881336659193,
+            0.04737752676010132,
+            -0.015693452209234238,
+            -0.005292471963912249,
+            0.028528403490781784,
+            0.000505455827806145,
+            0.02465103194117546,
+            -0.001686621573753655,
+            0.013330236077308655,
+            0.000776978034991771,
+            0.03834919258952141,
+            -0.006339645478874445,
+            0.029518973082304,
+            0.021608568727970123,
+            0.028316138312220573,
+            -0.014759486541152,
+            0.023122724145650864,
+            -0.019754787907004356,
+            -0.02292460948228836,
+            0.035547297447919846,
+            0.004082561936229467,
+            0.01886327564716339,
+            -0.028457649052143097,
+            -0.030679354444146156,
+            0.015452885068953037,
+            -0.01855195313692093,
+            0.016556663438677788,
+            0.03639635816216469,
+            0.0008273909334093332,
+            -0.0025719432160258293,
+            0.026292547583580017,
+            -0.04245298355817795,
+            0.012856177985668182,
+            -0.01532552670687437,
+            -0.02855670638382435,
+            -0.019528372213244438,
+            -0.00875946506857872,
+            -0.0018095583654940128,
+            -0.013691086322069168,
+            -0.0037570889107882977,
+            -0.007563706487417221,
+            -0.02459442801773548,
+            0.00935380719602108,
+            0.013627407141029835,
+            0.009849091991782188,
+            -0.018665160983800888,
+            0.0019952901639044285,
+            -0.008257105015218258,
+            -0.0160047747194767,
+            -0.03752843663096428,
+            0.015268922783434391,
+            -0.0024888061452656984,
+            0.009191070683300495,
+            0.04078316316008568,
+            0.028485950082540512,
+            -0.005055442452430725,
+            0.0329718142747879,
+            -0.033707667142152786,
+            0.017915157601237297,
+            -0.008320784196257591,
+            0.009191070683300495,
+            0.002989397384226322,
+            -0.025542544201016426,
+            0.027424626052379608,
+            -0.03792466223239899,
+            -0.038151077926158905,
+            0.022995365783572197,
+            0.011858533136546612,
+            0.0017096169758588076,
+            -0.019386861473321915,
+            -0.017717044800519943,
+            0.0008499440737068653,
+            0.002752368338406086,
+            -0.018834972754120827,
+            -0.01019579079002142,
+            0.02227366529405117,
+            -0.025500090792775154,
+            0.036085035651922226,
+            0.023391593247652054,
+            -0.03520767390727997,
+            -0.00608492735773325,
+            -0.01903308741748333,
+            0.059037946164608,
+            -0.0011506526498124003,
+            -0.010301923379302025,
+            0.022938761860132217,
+            -0.027268964797258377,
+            -0.004524780437350273,
+            0.022301966324448586,
+            0.01828308403491974,
+            -0.008278331719338894,
+            0.003983504604548216,
+            -0.022287815809249878,
+            -0.05281151086091995,
+            -0.010577867738902569,
+            -0.0003305583668407053,
+            0.054368119686841965,
+            -0.012367968447506428,
+            0.0020695829298347235,
+            -0.006413938011974096,
+            0.004100250545889139,
+            0.03704730048775673,
+            0.002140337834134698,
+            0.0025082635693252087,
+            -0.013698162510991096,
+            0.0371888130903244,
+            0.022316116839647293,
+            0.005883275531232357,
+            -0.012162779457867146,
+            0.005377377849072218,
+            -0.04899074137210846,
+            -0.0009118546731770039,
+            0.005741765722632408,
+            0.007386818993836641,
+            -0.025910470634698868,
+            0.017717044800519943,
+            0.009205221198499203,
+            -0.0024693484883755445,
+            -0.009792488068342209,
+            0.01173117384314537,
+            0.025429336354136467,
+            -0.019938750192523003,
+            -0.05700020492076874,
+            0.0034705311991274357,
+            0.012014193460345268,
+            -0.004054259974509478,
+            -0.019401011988520622,
+            0.006792477332055569,
+            0.0021934041287750006,
+            0.022698193788528442,
+            -0.011271266266703606,
+            -0.0029221803415566683,
+            0.040585052222013474,
+            0.009516543708741665,
+            -0.0016813150141388178,
+            -0.017717044800519943,
+            -0.011794853024184704,
+            0.018070818856358528,
+            0.004740582779049873,
+            0.0035289041697978973,
+            -0.01751893013715744,
+            -0.029264256358146667,
+            0.02800481766462326,
+            0.019967053085565567,
+            -0.048905834555625916,
+            -0.008200501091778278,
+            -0.019273653626441956,
+            0.0019599127117544413,
+            0.008695785887539387,
+            0.04613224044442177,
+            -0.00590803986415267,
+            0.008313708938658237,
+            -0.020179318264126778,
+            -0.05003791302442551,
+            -0.02708500251173973,
+            0.03707560524344444,
+            -0.028684064745903015,
+            -0.04276430234313011,
+            -0.010252394713461399,
+            0.016740625724196434,
+            -0.009403335861861706,
+            -0.018679311498999596,
+            -0.011738249100744724,
+            -0.04969828948378563,
+            -0.003497064346447587,
+            0.0337359681725502,
+            0.017264213413000107,
+            -0.008441068232059479,
+            -0.033877480775117874,
+            0.019641580060124397,
+            -0.007160402834415436,
+            -0.006474079564213753,
+            0.0007862646016292274,
+            0.008794843219220638,
+            0.02400008589029312,
+            0.011879759840667248,
+            -0.02763689123094082,
+            -0.03427370637655258,
+            -0.0024888061452656984,
+            0.014929298311471939,
+            -0.00878776703029871,
+            0.0245378240942955,
+            0.024679332971572876,
+            -0.022995365783572197,
+            -0.01654251106083393,
+            0.015750056132674217,
+            -0.05779265984892845,
+            -0.010287772864103317,
+            -0.029235953465104103,
+            -0.006183984223753214,
+            0.0009613831643946469,
+            0.0061096916906535625,
+            0.00012304731353651732,
+            0.03846240043640137,
+            0.008801918476819992,
+            0.006014172453433275,
+            0.009191070683300495,
+            0.01913214474916458,
+            0.028471799567341805,
+            0.009155693463981152,
+            -0.03517937287688255,
+            0.02850010246038437,
+            -0.017051948234438896,
+            0.007811348885297775,
+            -0.025754809379577637,
+            -0.013033065013587475,
+            0.00443633645772934,
+            0.02725481428205967,
+            -0.000503244751598686,
+            0.018849123269319534,
+            -0.021919889375567436,
+            0.037500135600566864,
+            -0.029915200546383858,
+            -0.05278320610523224,
+            -0.02005195803940296,
+            -0.008285406976938248,
+            0.015764206647872925,
+            0.017278363928198814,
+            -0.019924599677324295,
+            -0.0286557637155056,
+            -0.021750077605247498,
+            0.006470541935414076,
+            0.010542490519583225,
+            -0.021099131554365158,
+            -0.010627396404743195,
+            0.01502835564315319,
+            -0.00404364662244916,
+            -0.01886327564716339,
+            0.013684011064469814,
+            0.015693452209234238,
+            0.03169822692871094,
+            -0.03308502212166786,
+            0.004287751391530037,
+            -0.025967074558138847,
+            -0.04327373951673508,
+            0.004096712917089462,
+            0.03166992589831352,
+            0.027509532868862152,
+            -0.018636859953403473,
+            0.010754755698144436,
+            0.003990580327808857,
+            0.004234685096889734,
+            -0.0033803186379373074,
+            0.023603858426213264,
+            0.03291521221399307,
+            0.02763689123094082,
+            -0.011653343215584755,
+            0.004921008367091417,
+            -0.01559439580887556,
+            -0.0011108530452474952,
+            0.01865101046860218,
+            -0.017816102132201195,
+            0.033877480775117874,
+            -0.029405765235424042,
+            0.02104252763092518,
+            -0.035122767090797424,
+            0.006141531281173229,
+            -0.012099100276827812,
+            -0.00404364662244916,
+            0.04833979532122612,
+            -0.017646290361881256,
+            -0.0019546060357242823,
+            -0.008101443760097027,
+            -0.014575524255633354,
+            -0.02158026583492756,
+            0.008766540326178074,
+            -0.04033033177256584,
+            -0.012367968447506428,
+            -0.028429346159100533,
+            0.003424540627747774,
+            0.019910449162125587,
+            -0.06277380883693695,
+            -0.06237758323550224,
+            0.012275987304747105,
+            -0.04918885603547096,
+            -0.026419905945658684,
+            0.02963218092918396,
+            -0.002984090941026807,
+            0.0012780115939676762,
+            -0.0024003623984754086,
+            0.0068101659417152405,
+            -0.029207652434706688,
+            0.00425944896414876,
+            -0.000887974863871932,
+            0.03271709755063057,
+            -0.021056678146123886,
+            -0.008122670464217663,
+            0.004878554958850145,
+            -0.06860402226448059,
+            -0.028344441205263138,
+            0.011271266266703606,
+            -0.009459939785301685,
+            -0.0066615804098546505,
+            -0.017915157601237297,
+            -0.004383270163089037,
+            0.0008950503543019295,
+            0.0032830305863171816,
+            0.05801907554268837,
+            -0.01983969286084175,
+            -0.053773775696754456,
+            0.03633975237607956,
+            0.00887267291545868,
+            0.029603879898786545,
+            -0.025556694716215134,
+            -0.017533080652356148,
+            -0.01855195313692093,
+            -0.020504789426922798,
+            -0.04123599827289581,
+            0.00404364662244916,
+            -0.005918653216212988,
+            0.009212297387421131,
+            -0.0009693430620245636,
+            0.02763689123094082,
+            0.029235953465104103,
+            0.023108573630452156,
+            0.012750045396387577,
+            0.01079013291746378,
+            -0.022372720763087273,
+            -0.021594416350126266,
+            0.0012222920777276158,
+            0.04683978855609894,
+            0.024636881425976753,
+            -0.02675952948629856,
+            -0.011504758149385452,
+            0.03953787684440613,
+            0.0419718474149704,
+            -0.01974063739180565,
+            -0.02276895008981228,
+            0.00764861237257719,
+            0.03266049548983574,
+            -0.04477374628186226,
+            0.005932804197072983,
+            0.03410389646887779,
+            0.014405711553990841,
+            -0.005356151144951582,
+            -0.010294848121702671,
+            0.022570835426449776,
+            0.03152841329574585,
+            -0.008724087849259377,
+            0.01876421831548214,
+            0.016910437494516373,
+            0.029349161311984062,
+            -0.0067889392375946045,
+            0.0005766529939137399,
+            -0.024735936895012856,
+            0.055188875645399094,
+            -0.03059444949030876,
+            0.02909444272518158,
+            0.010316074825823307,
+            -0.00593634182587266,
+            0.039056744426488876,
+            -0.016641568392515182,
+            -0.0431322306394577,
+            0.01913214474916458,
+            -0.006979977712035179,
+            -0.0055613406002521515,
+            -0.03444352000951767,
+            -0.007592008449137211,
+            -0.007507102098315954,
+            -0.006714646704494953,
+            -0.022556684911251068,
+            -0.02606613002717495,
+            0.004415110219269991,
+            0.00973588414490223,
+            -0.006484692916274071,
+            0.00698705343529582,
+            0.02092931978404522,
+            -0.013507124036550522,
+            0.02525952458381653,
+            -0.046047333627939224,
+            -0.0008366775000467896,
+            -0.0038596836384385824,
+            -0.025712355971336365,
+            -0.0345001220703125,
+            -0.0005770951975136995,
+            -0.0024251264985650778,
+            -0.024622729048132896,
+            -0.00914154201745987,
+            -0.008752389810979366,
+            -0.0023119186516851187,
+            -0.004938696976751089,
+            -0.055188875645399094,
+            0.008695785887539387,
+            0.010846736840903759,
+            -0.002713453024625778,
+            0.0055507272481918335,
+            -0.010613245889544487,
+            -0.03039633482694626,
+            0.03039633482694626,
+            0.04129260033369064,
+            0.033707667142152786,
+            0.014334957115352154,
+            -0.0074646491557359695,
+            -0.017165156081318855,
+            0.006060163024812937,
+            0.021721776574850082,
+            0.00902833417057991,
+            0.007846726104617119,
+            0.03877372294664383,
+            0.00011331850691931322,
+            -0.008349086157977581,
+            -0.009764186106622219,
+            0.028740668669342995,
+            -0.001085204305127263,
+            0.023603858426213264,
+            0.050802066922187805,
+            0.030424637719988823,
+            0.020589696243405342,
+            -0.01681138016283512,
+            -0.02010856196284294,
+            -0.00851889792829752,
+            -0.02855670638382435,
+            0.0364246591925621,
+            0.012127402238547802,
+            -0.005232329946011305,
+            -0.011540135368704796,
+            0.019061388447880745,
+            -0.033339742571115494,
+            -0.0037358624394983053,
+            0.00048422932741232216,
+            0.013924578204751015,
+            -0.03379257395863533,
+            0.0002480846014805138,
+            0.031613320112228394,
+            0.014957600273191929,
+            -0.016995344310998917,
+            -0.0188774261623621,
+            0.010549565777182579,
+            0.020462337881326675,
+            0.018948180601000786,
+            0.00037743354914709926,
+            0.00897173024713993,
+            -0.00022121983056422323,
+            -0.0005319889169186354,
+            0.01590571738779545,
+            -0.007952858693897724,
+            0.036254845559597015,
+            -0.003525366308167577,
+            -0.01325240544974804,
+            -0.0037500134203583,
+            -0.021438755095005035,
+            -0.024467067793011665,
+            0.043556760996580124,
+            0.023971782997250557,
+            0.04661337286233902,
+            -0.005660397466272116,
+            -0.004245297983288765,
+            0.025825563818216324,
+            0.008221727795898914,
+            -0.03288691118359566,
+            0.000265110022155568,
+            0.005281858611851931,
+            -0.02892463095486164,
+            0.016641568392515182,
+            -0.02476423978805542,
+            -0.016896286979317665,
+            -0.006771250627934933,
+            -0.0015557248843833804,
+            -0.028188779950141907,
+            0.01703779585659504,
+            0.0031309074256569147,
+            -0.0027099153958261013,
+            -0.039764292538166046,
+            0.006880920846015215,
+            0.020419884473085403,
+            0.015311375260353088,
+            0.0037358624394983053,
+            0.011525984853506088,
+            -0.016613267362117767,
+            0.002126186853274703,
+            0.03379257395863533,
+            0.004878554958850145,
+            0.02005195803940296,
+            0.0027647505048662424,
+            0.023773670196533203,
+            0.03633975237607956,
+            -0.01681138016283512,
+            -0.02189158834517002,
+            0.01519816741347313,
+            -0.02233026921749115,
+            0.029943503439426422,
+            0.004889168310910463,
+            -0.012283062562346458,
+            0.022033097222447395,
+            -0.009827865287661552,
+            -0.0027346794959157705,
+            0.01659911684691906,
+            -0.022740647196769714,
+            -0.017717044800519943,
+            0.03183973580598831,
+            -0.039452970027923584,
+            0.012856177985668182,
+            -0.008547200821340084,
+            0.015863263979554176,
+            -0.023066120222210884,
+            0.011172209866344929,
+            0.007100261282175779,
+            -0.01573590561747551,
+            0.010797208175063133,
+            0.009693430736660957,
+            0.00545874610543251,
+            0.0018926954362541437,
+            0.02179253101348877,
+            -0.0006049550138413906,
+            -0.011257115751504898,
+            -0.018155725672841072,
+            -0.004694592207670212,
+            -0.002614396158605814,
+            -0.016556663438677788,
+            0.01865101046860218,
+            -0.041066184639930725,
+            -0.013280707411468029,
+            -0.008879749104380608,
+            -0.013988257385790348,
+            -0.010011828504502773,
+            0.024184048175811768,
+            0.025669902563095093,
+            -0.002361447084695101,
+            -0.031217092648148537,
+            0.01790100708603859,
+            0.024198200553655624,
+            0.003654494183138013,
+            -0.023646311834454536,
+            0.0302831269800663,
+            -0.008936353027820587,
+            0.014971751719713211,
+            0.026136886328458786,
+            0.014094389975070953,
+            0.004595535341650248,
+            -0.010500037111341953,
+            -0.02114158496260643,
+            0.01481609046459198,
+            0.010507113300263882,
+            0.033537857234478,
+            0.014745336025953293,
+            0.02071705460548401,
+            0.01594817079603672,
+            -0.023320838809013367,
+            -0.011214662343263626,
+            -0.003482913365587592,
+            -0.022514231503009796,
+            0.03093407303094864,
+            0.0006129149114713073,
+            0.02114158496260643,
+            0.026618018746376038,
+            -0.03319823369383812,
+            -0.01187268365174532,
+            0.032688796520233154,
+            0.02779255248606205,
+            0.05108508840203285,
+            0.025316128507256508,
+            0.015396281145513058,
+            0.027226513251662254,
+            -0.03585861995816231,
+            -0.036368053406476974,
+            0.001548649393953383,
+            0.04256619140505791,
+            -0.008426916785538197,
+            0.00967928022146225,
+            0.03900013864040375,
+            -0.006781863979995251,
+            -0.01611798256635666,
+            0.019457615911960602,
+            0.0026710000820457935,
+            0.002448122017085552,
+            -0.0033113325480371714,
+            -0.01335146278142929,
+            0.03175482898950577,
+            -0.0051474240608513355,
+            0.013570803217589855,
+            -0.019273653626441956,
+            0.027113303542137146,
+            0.0001799829478841275,
+            -0.003229964291676879,
+            0.028344441205263138,
+            -0.002660386962816119,
+            -0.051990751177072525,
+            0.024212351068854332,
+            0.019811391830444336,
+            -0.011561362072825432,
+            -0.0036332677118480206,
+            -0.003792466362938285,
+            0.04567940905690193,
+            0.034698236733675,
+            0.029009537771344185,
+            -0.0020501252729445696,
+            0.0168679840862751,
+            0.018792519345879555,
+            0.0053667644970119,
+            0.014278353191912174,
+            -0.0017025414854288101,
+            -0.007853801362216473,
+            0.021056678146123886,
+            -0.020787810906767845,
+            0.03226426616311073,
+            -0.016570813953876495,
+            -0.020122714340686798,
+            0.008738238364458084,
+            -0.011172209866344929,
+            -0.0009083169279620051,
+            -0.006311343517154455,
+            -0.0008813415770418942,
+            -0.0029664020985364914,
+            0.016938740387558937,
+            -0.004446949809789658,
+            0.04661337286233902,
+            -0.013931653462350368,
+            -0.008808993734419346,
+            0.006972902454435825,
+            -0.025981225073337555,
+            0.0013346156338229775,
+            0.0028726516757160425,
+            0.027000095695257187,
+            -0.01989629678428173,
+            -0.00970758218318224,
+            0.01606137864291668,
+            0.06039644405245781,
+            -0.008639181964099407,
+            -0.0026267783250659704,
+            0.027226513251662254,
+            -0.002787745790556073,
+            -0.03551899641752243,
+            0.012919857166707516,
+            -0.04494355618953705,
+            0.017589686438441277,
+            0.026349151507019997,
+            -0.04986810311675072,
+            -0.022089701145887375,
+            -0.02347649820148945,
+            -0.02043403498828411,
+            0.023434046655893326,
+            -0.009976450353860855,
+            0.013867974281311035,
+            -0.017801949754357338,
+            -0.009014182724058628,
+            0.020518941804766655,
+            0.009176919236779213,
+            0.03735862299799919,
+            -0.027551984414458275,
+            -0.016783079132437706,
+            -0.027820853516459465,
+            -0.04304732382297516,
+            0.009558996185660362,
+            -0.0278916098177433,
+            0.0027099153958261013,
+            -0.0016291331266984344,
+            -0.040302030742168427,
+            -0.037981268018484116,
+            0.005278320983052254,
+            -0.017985913902521133,
+            0.01516986545175314,
+            0.020915169268846512,
+            0.02731141820549965,
+            -0.0020253609400242567,
+            -0.030198222026228905,
+            0.029235953465104103,
+            -0.007344366051256657,
+            0.03223596513271332,
+            -0.00689507182687521,
+            0.005066055804491043,
+            -0.0070436573587358,
+            0.031160488724708557,
+            -0.023009516298770905,
+            0.018792519345879555,
+            -0.026801982894539833,
+            -0.004086099565029144,
+            -0.018990634009242058,
+            -0.01774534583091736,
+            0.004224071744829416,
+            0.004521242808550596,
+            7.888073741924018e-05,
+            0.0048750173300504684,
+            0.005777143407613039,
+            0.024806693196296692,
+            0.0021421066485345364,
+            0.02589631825685501,
+            -0.027877457439899445,
+            -0.025316128507256508,
+            0.01719345711171627,
+            -0.02077365852892399,
+            0.005678086541593075,
+            -0.006261814851313829,
+            -0.0302831269800663,
+            -0.01202834490686655,
+            0.023391593247652054,
+            0.02162271924316883,
+            -0.005490585695952177,
+            0.002644466934725642,
+            0.02206140011548996,
+            -0.026023678481578827,
+            0.00437973253428936,
+            -0.03781145438551903,
+            -0.018396291881799698,
+            0.029773691669106483,
+            -0.03195294365286827,
+            0.020745357498526573,
+            0.003185742534697056,
+            0.03483974561095238,
+            -0.017292514443397522,
+            -0.01947176828980446,
+            0.01654251106083393,
+            -0.01008258294314146,
+            0.004142703488469124,
+            0.01606137864291668,
+            0.01616043597459793,
+            -0.0015070808585733175,
+            0.05380208045244217,
+            0.0008167776977643371,
+            0.006909222807735205,
+            -0.049075648188591,
+            0.004496478475630283,
+            -0.044858649373054504,
+            0.0002768288250081241,
+            -0.013337312266230583,
+            -0.02514631673693657,
+            0.0014027173165231943,
+            0.001001182827167213,
+            0.001963450340554118,
+            -0.01913214474916458,
+            -0.01989629678428173,
+            0.011596739292144775,
+            -0.015863263979554176,
+            0.004928083624690771,
+            -0.06418890506029129,
+            -0.01855195313692093,
+            -0.004974074196070433,
+            0.0043337419629096985,
+            0.0007699025445617735,
+            0.03342464938759804,
+            -0.01478778850287199,
+            0.010117961093783379,
+            -0.01549533847719431,
+            -0.026363302022218704,
+            0.04010391607880592,
+            0.018750067800283432,
+            -0.023773670196533203,
+            0.024693485349416733,
+            0.012106175534427166,
+            0.03099067695438862,
+            -0.0318114347755909,
+            0.020349130034446716,
+            0.018523650243878365,
+            -0.0029823218937963247,
+            0.006551910191774368,
+            0.008575502783060074,
+            0.01832553744316101,
+            -0.015764206647872925,
+            -0.0003632825391832739,
+            0.011419852264225483,
+            0.010033054277300835,
+            0.031924642622470856,
+            0.01403071079403162,
+            -0.03183973580598831,
+            0.019047237932682037,
+            0.01616043597459793,
+            0.02298121340572834,
+            0.033820874989032745,
+            -0.027481229975819588,
+            0.038575608283281326,
+            -0.010386829264461994,
+            -0.024042539298534393,
+            0.020801961421966553,
+            -0.032688796520233154,
+            -0.006679269019514322,
+            0.030735958367586136,
+            0.04398128762841225,
+            -0.0015566092915832996,
+            -0.044632233679294586,
+            0.01535382866859436,
+            0.01316749956458807,
+            0.0027806703001260757,
+            0.028811423107981682,
+            0.011264191009104252,
+            0.021820832043886185,
+            0.025273675099015236,
+            0.011419852264225483,
+            -0.0037252490874379873,
+            0.024962352588772774,
+            -0.008101443760097027,
+            -0.004446949809789658,
+            -0.0007035697344690561,
+            0.03206615149974823,
+            0.014702882617712021,
+            -0.017603836953639984,
+            -0.036962397396564484,
+            -0.007620310410857201,
+            0.009452863596379757,
+            -0.007903330028057098,
+            -0.027127455919981003,
+            -0.03950957581400871,
+            -0.026787832379341125,
+            -0.005667473189532757,
+            0.01505665760487318,
+            -0.06724552065134048,
+            -0.007620310410857201,
+            -0.012233533896505833,
+            0.017717044800519943,
+            0.0022906921803951263,
+            0.0181132722645998,
+            0.001277127186767757,
+            -0.009403335861861706,
+            0.004209920763969421,
+            0.050802066922187805,
+            -0.03059444949030876,
+            0.014242975041270256,
+            0.011292492970824242,
+            0.0032105068676173687,
+            0.002787745790556073,
+            0.038688816130161285,
+            -0.010988246649503708,
+            -0.030424637719988823,
+            0.011575513519346714,
+            -0.017292514443397522,
+            0.003383856499567628,
+            0.05006621778011322,
+            -0.009481165558099747,
+            -0.027495380491018295,
+            0.020915169268846512,
+            -0.009148617275059223,
+            -0.015523640438914299,
+            -0.0027258351910859346,
+            -0.006035399157553911,
+            0.0006283925613388419,
+            -0.03832089155912399,
+            -0.015750056132674217,
+            0.03212275728583336,
+            0.008617955259978771,
+            0.014603826217353344,
+            0.025344429537653923,
+            0.010209942236542702,
+            -0.030481241643428802,
+            0.027481229975819588,
+            -0.0005678086308762431,
+            0.005221716593950987,
+            0.013323160819709301,
+            0.03424540534615517,
+            -0.014264201745390892,
+            -0.020264223217964172,
+            -0.040245428681373596,
+            -0.013853822834789753,
+            -0.0463869571685791,
+            0.001943992800079286,
+            0.006919836159795523,
+            -0.012919857166707516,
+            -0.04366996884346008,
+            -0.0329718142747879,
+            0.021127434447407722,
+            0.03178313374519348,
+            -0.017023645341396332,
+            0.0025294902734458447,
+            -0.008483520708978176,
+            -0.007882103323936462,
+            -0.0017520699184387922,
+            -0.013217028230428696,
+            -0.006378560326993465,
+            -0.006229975260794163,
+            0.015452885068953037,
+            -0.004652139265090227,
+            -0.0036934094969183207,
+            -0.0498398020863533,
+            -0.018778368830680847,
+            0.004082561936229467,
+            -0.008221727795898914,
+            0.007323139347136021,
+            -0.006955213379114866,
+            -0.002727604005485773,
+            -0.012474101036787033,
+            -0.03829259052872658,
+            0.024184048175811768,
+            0.017278363928198814,
+            0.0035501306410878897,
+            -0.008922201581299305,
+            -0.018367990851402283,
+            0.03294351324439049,
+            0.012523629702627659,
+            -0.017547233030200005,
+            -0.018580254167318344,
+            0.025981225073337555,
+            0.026957644149661064,
+            0.008122670464217663,
+            -0.006137993652373552,
+            0.029886899515986443,
+            0.005695775151252747,
+            -0.022938761860132217,
+            -0.003983504604548216,
+            -0.005055442452430725,
+            -0.0038844477385282516,
+            0.020561393350362778,
+            -0.010556641966104507,
+            0.0300001073628664,
+            0.038717120885849,
+            0.012870329432189465,
+            0.0039198254235088825,
+            -0.0033714743331074715,
+            -0.008922201581299305,
+            0.015976471826434135,
+            0.006481155287474394,
+            0.008525974117219448,
+            0.014759486541152,
+            -0.016358548775315285,
+            -0.020405733957886696,
+            0.006215824279934168,
+            -0.007107336539775133,
+            0.020122714340686798,
+            0.01708024926483631,
+            0.016429303213953972,
+            0.0007610581233166158,
+            -0.028287837281823158,
+            -0.0018024828750640154,
+            0.02411329373717308,
+            0.02010856196284294,
+            -0.004655676893889904,
+            -0.023264234885573387,
+            0.014398636296391487,
+            -0.015806660056114197,
+            -0.009969375096261501,
+            0.0033431723713874817,
+            0.006229975260794163,
+            -0.003083147807046771,
+            0.0571134127676487,
+            -0.020193468779325485,
+            -0.011419852264225483,
+            -0.010287772864103317,
+            -0.010867963545024395,
+            0.0012072566896677017,
+            -0.01828308403491974,
+            0.008964654989540577,
+            0.025613298639655113,
+            -0.009764186106622219,
+            0.03568880632519722,
+            0.0228114016354084,
+            0.006912760436534882,
+            0.005260631907731295,
+            -0.027070851996541023,
+            0.012304289266467094,
+            -0.016047226265072823,
+            0.005660397466272116,
+            0.018467046320438385,
+            0.017320817336440086,
+            0.0076132346875965595,
+            0.026306698098778725,
+            0.019386861473321915,
+            0.016684021800756454,
+            -0.022783100605010986,
+            0.025457637384533882,
+            -0.007832575589418411,
+            0.027877457439899445,
+            -0.01611798256635666,
+            -0.002253545681014657,
+            -0.012990612536668777,
+            0.008681634441018105,
+            0.002416282193735242,
+            -0.0019351483788341284,
+            0.00036129256477579474,
+            0.016783079132437706,
+            -0.014589674770832062,
+            0.01062032114714384,
+            0.013316085562109947,
+            -0.027919910848140717,
+            -0.010372678749263287,
+            0.016344398260116577,
+            -0.00668988237157464,
+            0.006877383217215538,
+            -0.011865608394145966,
+            -0.0017202302115038037,
+            0.011009473353624344,
+            -0.0436982698738575,
+            0.0104929618537426,
+            0.006279503460973501,
+            -0.015155714936554432,
+            0.014801939949393272,
+            -0.009799563325941563,
+            -0.008235878311097622,
+            -0.013592029921710491,
+            0.003792466362938285,
+            -0.00030225637601688504,
+            -0.0018290160223841667,
+            -0.00022995365725364536,
+            0.00389859895221889,
+            0.010047205723822117,
+            -0.0045990729704499245,
+            -0.010450509376823902,
+            -0.0028638073708862066,
+            -0.02265574224293232,
+            -0.0010047205723822117,
+            0.018735915422439575,
+            -0.004591997712850571,
+            0.0005222601466812193,
+            0.0145189194008708,
+            -0.005830209702253342,
+            0.010556641966104507,
+            -0.005221716593950987,
+            -0.008320784196257591,
+            -0.010768906213343143,
+            -0.033764272928237915,
+            0.0026710000820457935,
+            -0.015254771336913109,
+            0.008865597657859325,
+            -0.012544856406748295,
+            0.04027372971177101,
+            0.01022409275174141,
+            -0.00837738811969757,
+            -0.002796590095385909,
+            -0.0228114016354084,
+            -0.004475251771509647,
+            -0.016938740387558937,
+            -0.01184438169002533,
+            0.008342010900378227,
+            0.012629762291908264,
+            0.005614406894892454,
+            -0.018410442396998405,
+            0.003665107535198331,
+            0.0030955299735069275,
+            -0.005575491581112146,
+            -0.008370312862098217,
+            0.010853812098503113,
+            -0.0006248548161238432,
+            -0.01800006441771984,
+            -0.01957082375884056,
+            -0.027424626052379608,
+            0.0150000536814332,
+            0.014589674770832062,
+            -0.01011088490486145,
+            -0.022754797711968422,
+            -0.007471724878996611,
+            0.007107336539775133,
+            -0.02064630016684532,
+            0.0006779210525564849,
+            0.006891534198075533,
+            0.008717012591660023,
+            0.004117939155548811,
+            0.009636826813220978,
+            0.0007699025445617735,
+            0.0121698547154665,
+            0.018566103652119637,
+            0.03271709755063057,
+            -0.0396793857216835,
+            -0.009537769481539726,
+            0.0057028504088521,
+            -0.001465512323193252,
+            -0.013853822834789753,
+            0.009672204032540321,
+            0.0024375086650252342,
+            -0.0026426981203258038,
+            0.024424616247415543,
+            -0.008044839836657047,
+            -0.01505665760487318,
+            0.01184438169002533,
+            0.00887267291545868,
+            -0.019075538963079453,
+            0.02746707946062088,
+            0.006169833242893219,
+            -0.01011088490486145,
+            0.02259913645684719,
+            -0.012360893189907074,
+            0.012658064253628254,
+            -0.005522425286471844,
+            0.023660462349653244,
+            -0.019004784524440765,
+            0.0006306036957539618,
+            0.015665151178836823,
+            0.03883032873272896,
+            -0.00437973253428936,
+            -0.004567233379930258,
+            -0.03421710431575775,
+            0.005115584470331669,
+            0.028853876516222954,
+            0.02666047215461731,
+            -0.0012346742441877723,
+            -0.005596718285232782,
+            -0.0019864458590745926,
+            0.010768906213343143,
+            0.011589664034545422,
+            0.017561383545398712,
+            -0.029547275975346565,
+            0.00022011429246049374,
+            -0.016938740387558937,
+            0.025500090792775154,
+            0.014632128179073334,
+            0.006951675750315189,
+            0.04072656109929085,
+            0.0027789014857262373,
+            0.003841994795948267,
+            0.027452928945422173,
+            -0.00779719790443778,
+            0.003546592779457569,
+            -0.015566092915832996,
+            0.0003175129240844399,
+            0.022853855043649673,
+            0.0016370931407436728,
+            0.0020784272346645594,
+            0.013337312266230583,
+            0.029320860281586647,
+            -0.022471778094768524,
+            -0.023929331451654434,
+            0.003482913365587592,
+            0.007185167167335749,
+            -0.010570792481303215,
+            0.0038738346192985773,
+            0.006275965832173824,
+            -0.006007096730172634,
+            0.008900974877178669,
+            0.006332569755613804,
+            -0.0003166284877806902,
+            0.0039198254235088825,
+            -0.00308845448307693,
+            0.012962310574948788,
+            -0.02702839858829975,
+            0.036849189549684525,
+            0.004439874552190304,
+            0.010075507685542107,
+            0.015268922783434391,
+            0.001627364312298596,
+            0.004917470272630453,
+            0.005869124550372362,
+            0.010188715532422066,
+            0.018636859953403473,
+            -0.00956607237458229,
+            -0.027481229975819588,
+            -0.03166992589831352,
+            0.020363280549645424,
+            -0.01173117384314537,
+            0.0035872769076377153,
+            0.0074646491557359695,
+            -0.01828308403491974,
+            -0.007507102098315954,
+            -0.009976450353860855,
+            0.015764206647872925,
+            -0.03189634159207344,
+            0.012990612536668777,
+            -0.0043231286108493805,
+            -0.028669914230704308,
+            0.00851889792829752,
+            -0.010181640274822712,
+            -0.0045884596183896065,
+            -0.0035094465129077435,
+            -0.029264256358146667,
+            -0.007931631989777088,
+            0.03908504545688629,
+            -0.02141045406460762,
+            0.024184048175811768,
+            0.008257105015218258,
+            -0.00014449491573031992,
+            -0.0040365708991885185,
+            -0.013563727959990501,
+            0.005331386812031269,
+            -0.01778779923915863,
+            0.025457637384533882,
+            0.010294848121702671,
+            -0.013924578204751015,
+            0.0322076641023159,
+            -0.011179285123944283,
+            0.027438776567578316,
+            -0.018580254167318344,
+            0.036198243498802185,
+            0.007422196213155985,
+            0.00674648629501462,
+            0.01974063739180565,
+            -0.004025958012789488,
+            0.008228803053498268,
+            0.04559450224041939,
+            -0.018523650243878365,
+            0.02460857853293419,
+            0.020349130034446716,
+            -0.00022685811563860625,
+            0.017915157601237297,
+            0.0028390432707965374,
+            -0.023165177553892136,
+            0.009495317004621029,
+            -0.01135617308318615,
+            -0.009955224581062794,
+            0.01583496294915676,
+            -0.019316107034683228,
+            -0.018721764907240868,
+            -0.025613298639655113,
+            0.0029416377656161785,
+            -0.003350247861817479,
+            -0.0077618202194571495,
+            -0.0019616815261542797,
+            0.01454722136259079,
+            0.008724087849259377,
+            0.025669902563095093,
+            0.0008326975512318313,
+            0.011971740983426571,
+            0.01511326152831316,
+            -0.00035576484515331686,
+            0.02313687466084957,
+            -0.03229256719350815,
+            0.01246702577918768,
+            -0.004517704714089632,
+            -0.00509789539501071,
+            0.023858575150370598,
+            0.0003747802402358502,
+            -0.0004877670726273209,
+            -0.0024888061452656984,
+            0.0005496777012012899,
+            -0.014589674770832062,
+            -0.0057099261321127415,
+            0.000907432462554425,
+            -0.005175726022571325,
+            0.04299071803689003,
+            -0.00701535539701581,
+            -0.004772422835230827,
+            -0.039169952273368835,
+            -0.01918874867260456,
+            0.00623351288959384,
+            -0.01583496294915676,
+            -0.018778368830680847,
+            -0.006307805422693491,
+            0.005801907740533352,
+            -0.04893413558602333,
+            0.008342010900378227,
+            0.03336804360151291,
+            -0.012219383381307125,
+            0.00259493850171566,
+            -0.010981171391904354,
+            -0.006969364359974861,
+            -0.0029115669894963503,
+            -7.766463386360556e-05,
+            -0.002412744564935565,
+            -0.020830262452363968,
+            -0.005617944523692131,
+            0.003364398842677474,
+            -0.0008795727044343948,
+            -0.025047259405255318,
+            0.0006177793256938457,
+            0.018849123269319534,
+            -0.003979966975748539,
+            -0.0011152751976624131,
+            -0.001825478277169168,
+            0.006399787031114101,
+            -0.01508495956659317,
+            0.008009462617337704,
+            -0.0076344613917171955,
+            0.003841994795948267,
+            -0.0030495391692966223,
+            -0.002214630600064993,
+            0.02844349853694439,
+            0.021594416350126266,
+            -0.04808507859706879,
+            -0.01405901275575161,
+            0.023009516298770905,
+            0.007280686404556036,
+            -0.004740582779049873,
+            0.01178777776658535,
+            -0.031613320112228394,
+            -0.00028943203506059945,
+            -0.026787832379341125,
+            0.02200479619204998,
+            -0.009205221198499203,
+            -0.007386818993836641,
+            0.016967041417956352,
+            -0.026632171124219894,
+            0.016344398260116577,
+            0.00794578343629837,
+            -0.00030181417241692543,
+            -0.000493515923153609,
+            0.0063856360502541065,
+            0.0008972614887170494,
+            -0.01659911684691906,
+            0.006442239973694086,
+            -0.0048502529971301556,
+            0.007959933951497078,
+            -0.005087282508611679,
+            -0.05527378246188164,
+            -0.009933997876942158,
+            -0.005497660953551531,
+            -0.008936353027820587,
+            -0.012820800766348839,
+            0.008136821910738945,
+            0.02827368676662445,
+            0.01606137864291668,
+            -0.009863242506980896,
+            -4.8312376748071983e-05,
+            0.033000119030475616,
+            -0.0008300442132167518,
+            0.009955224581062794,
+            -0.011349096894264221,
+            0.00015709814033471048,
+            -0.0006911876262165606,
+            -0.014660430140793324,
+            -0.019061388447880745,
+            0.024042539298534393,
+            -0.028344441205263138,
+            0.02378782071173191,
+            -0.01843874529004097,
+            -0.0130825936794281,
+            0.01757553406059742,
+            -0.032575588673353195,
+            -0.005108508747071028,
+            0.007450498174875975,
+            0.0090566361322999,
+            0.021608568727970123,
+            0.0032794929575175047,
+            0.020589696243405342,
+            -0.03716050833463669,
+            -0.009134466759860516,
+            -0.010160413570702076,
+            -0.00794578343629837,
+            -0.010231168940663338,
+            -0.011596739292144775,
+            0.021863285452127457,
+            -0.015240620821714401,
+            -0.016952890902757645,
+            0.007054270710796118,
+            0.0030212372075766325,
+            -0.011313719674944878,
+            0.015750056132674217,
+            0.00017810352437663823,
+            0.004712280817329884,
+            0.013011839240789413,
+            -0.0008923970744945109,
+            0.04086806997656822,
+            0.0253019779920578,
+            -0.0030212372075766325,
+            -0.008568426594138145,
+            0.021438755095005035,
+            0.0004285097820684314,
+            0.0150000536814332,
+            0.009375033900141716,
+            0.023264234885573387,
+            -0.02092931978404522,
+            0.009997677057981491,
+            -0.005501199048012495,
+            0.014603826217353344,
+            -0.009495317004621029,
+            -0.015339677222073078,
+            0.00479718716815114,
+            -0.01577835902571678,
+            0.010153338313102722,
+            -0.010860888287425041,
+            0.008221727795898914,
+            0.01913214474916458,
+            0.002467579673975706,
+            -0.00941041111946106,
+            0.01146230474114418,
+            0.02075950801372528,
+            -0.025641601532697678,
+            0.008851447142660618,
+            -0.0031998935155570507,
+            -0.0021916350815445185,
+            0.021127434447407722,
+            0.010301923379302025,
+            -0.01594817079603672,
+            0.01573590561747551,
+            -0.008879749104380608,
+            -0.0015053119277581573,
+            -0.01882082223892212,
+            -0.03291521221399307,
+            -0.0015283073298633099,
+            0.012396270409226418,
+            0.004800724796950817,
+            0.0150000536814332,
+            0.011264191009104252,
+            -0.0008950503543019295,
+            -0.006601438857614994,
+            0.0008433107868768275,
+            -0.013485897332429886,
+            -0.004963461309671402,
+            0.010231168940663338,
+            0.017051948234438896,
+            -0.0037217114586383104,
+            0.015650998800992966,
+            0.0088160689920187,
+            0.006137993652373552,
+            -0.0010480580385774374,
+            0.0036138100549578667,
+            -0.002060738392174244,
+            -0.007783046457916498,
+            0.019938750192523003,
+            0.019075538963079453,
+            -0.018509499728679657,
+            -0.0024870370980352163,
+            -0.02141045406460762,
+            0.025882167741656303,
+            0.022188758477568626,
+            0.011165134608745575,
+            -0.04222656786441803,
+            0.00012050455552525818,
+            0.00752832880243659,
+            -0.0013744152383878827,
+            -0.031103884801268578,
+            0.008172199130058289,
+            0.005221716593950987,
+            0.004330204334110022,
+            0.020844414830207825,
+            -0.018424594774842262,
+            0.010853812098503113,
+            0.0070861103013157845,
+            -0.0005766529939137399,
+            0.01019579079002142,
+            -0.0073372903279960155,
+            0.008688710629940033,
+            0.038717120885849,
+            -0.018410442396998405,
+            0.005303084850311279,
+            0.0062370505183935165,
+            0.023320838809013367,
+            -0.011228813789784908,
+            -0.00031485961517319083,
+            0.00800238735973835,
+            0.021608568727970123,
+            -0.012934008613228798,
+            0.005883275531232357,
+            0.067075714468956,
+            -0.012912781909108162,
+            0.031613320112228394,
+            0.020533092319965363,
+            -0.004970536567270756,
+            -0.0104929618537426,
+            -0.012445799075067043,
+            -0.0181132722645998,
+            0.021283095702528954,
+            0.01175947580486536,
+            0.003419233951717615,
+            0.01941516436636448,
+            -0.01610383205115795,
+            0.006636816076934338,
+            0.017547233030200005,
+            0.012601460330188274,
+            -0.007379743270576,
+            -0.0036173479165881872,
+            0.019004784524440765,
+            0.018509499728679657,
+            -0.0030849166214466095,
+            0.00019424450874794275,
+            0.01508495956659317,
+            -0.022075550630688667,
+            -0.012530704960227013,
+            0.01644345559179783,
+            -0.0037358624394983053,
+            0.017773648723959923,
+            -0.02227366529405117,
+            0.024198200553655624,
+            -0.004531855694949627,
+            -0.017179306596517563,
+            0.004698129836469889,
+            -0.007514177821576595,
+            0.022033097222447395,
+            -0.001578720286488533,
+            0.004857328720390797,
+            0.000821199850179255,
+            -0.0018015984678640962,
+            0.01349297259002924,
+            0.002826661104336381,
+            -0.0005120891146361828,
+            0.010507113300263882,
+            -0.0009702275274321437,
+            -0.0013496510218828917,
+            0.024509521201252937,
+            1.9374700059415773e-05,
+            0.005989408120512962,
+            0.0032936439383774996,
+            -0.009261825121939182,
+            -0.004429261200129986,
+            -0.004443412180989981,
+            0.0011842612875625491,
+            0.003877372248098254,
+            -0.007029506377875805,
+            -0.02384442463517189,
+            0.022471778094768524,
+            -0.008540124632418156,
+            0.01175947580486536,
+            0.011349096894264221,
+            0.003665107535198331,
+            0.006824316922575235,
+            -0.009771261364221573,
+            -0.017320817336440086,
+            0.0006633278680965304,
+            -0.01617458648979664,
+            0.006912760436534882,
+            -0.001656550681218505,
+            0.0020235921256244183,
+            0.009445788338780403,
+            -0.002570174168795347,
+            0.0015123875346034765,
+            0.011306644417345524,
+            0.011497682891786098,
+            0.003254728624597192,
+            0.004577846731990576,
+            -0.018721764907240868,
+            -0.020915169268846512,
+            0.005288933869451284,
+            0.013924578204751015,
+            0.004273600410670042,
+            0.0015309606678783894,
+            0.017603836953639984,
+            -0.006166295614093542,
+            -0.022967062890529633,
+            -0.010117961093783379,
+            -0.01584911346435547,
+            0.039396367967128754,
+            0.01800006441771984,
+            -0.02525952458381653,
+            -0.000530220044311136,
+            -0.009728807955980301,
+            0.010450509376823902,
+            -0.00404364662244916,
+            0.014575524255633354,
+            0.005476434715092182,
+            -0.02426895499229431,
+            -0.016797229647636414,
+            -0.010662773624062538,
+            -0.005257094278931618,
+            0.020080260932445526,
+            0.027014248073101044,
+            -0.004956385586410761,
+            0.0061026159673929214,
+            0.019584976136684418,
+            -0.010938717983663082,
+            0.009849091991782188,
+            0.020957622677087784,
+            0.0031698227394372225,
+            -0.007570781745016575,
+            -0.004553082399070263,
+            -0.016514210030436516,
+            0.024467067793011665,
+            0.014250051230192184,
+            -0.005104971118271351,
+            -0.02465103194117546,
+            0.004110863897949457,
+            -0.02428310550749302,
+            -0.01616043597459793,
+            -0.007301913108676672,
+            -0.014143918640911579,
+            -0.004931621253490448,
+            0.0027594438288360834,
+            -0.004836102016270161,
+            0.011278342455625534,
+            -0.0031592093873769045,
+            -0.004531855694949627,
+            -0.0041851564310491085,
+            -0.025584997609257698,
+            -0.02524537406861782,
+            0.007691065315157175,
+            -0.003005317412316799,
+            0.005596718285232782,
+            0.000650945701636374,
+            -0.013400991447269917,
+            -0.011589664034545422,
+            0.00970758218318224,
+            -0.007988235913217068,
+            0.012813725508749485,
+            0.0031149876303970814,
+            -0.009183995425701141,
+            0.0017193458043038845,
+            -0.0012311364989727736,
+            -0.0059752571396529675,
+            0.019924599677324295,
+            0.00916984397917986,
+            0.011724098585546017,
+            -0.03118879161775112,
+            -0.0021633331198245287,
+            0.008349086157977581,
+            0.01828308403491974,
+            -0.02244347706437111,
+            -0.008313708938658237,
+            0.005179263651371002,
+            -0.011228813789784908,
+            -0.013507124036550522,
+            -0.028952933847904205,
+            -0.0030460015404969454,
+            8.175515540642664e-05,
+            -0.018636859953403473,
+            -0.029886899515986443,
+            0.0009038947173394263,
+            -0.007860877551138401,
+            0.016882136464118958,
+            -0.006643891800194979,
+            0.01790100708603859,
+            0.00870286114513874,
+            0.0036173479165881872,
+            0.011554286815226078,
+            0.0025082635693252087,
+            -0.013266556896269321,
+            0.008426916785538197,
+            -0.02671707607805729,
+            -0.01516986545175314,
+            0.009792488068342209,
+            0.014214673079550266,
+            -0.0026126273442059755,
+            5.074458022136241e-05,
+            0.0058372849598526955,
+            0.024254804477095604,
+            0.013584954664111137,
+            0.012474101036787033,
+            0.003042463678866625,
+            -0.005303084850311279,
+            0.005568416323512793,
+            0.0227123461663723,
+            0.008603804744780064,
+            0.004011806566268206,
+            0.007592008449137211,
+            0.00389859895221889,
+            0.0009932229295372963,
+            -0.005943417549133301,
+            -0.00843399204313755,
+            -0.021537812426686287,
+            -0.008908051066100597,
+            0.019938750192523003,
+            -0.010422207415103912,
+            -0.03005671128630638,
+            0.0006023016758263111,
+            -0.0032777239102870226,
+            0.007988235913217068,
+            0.02595292218029499,
+            -0.014172220602631569,
+            0.004970536567270756,
+            0.005536576267331839,
+            -0.007910405285656452,
+            -0.00614860700443387,
+            0.022967062890529633,
+            0.02639160305261612,
+            -0.0034404604230076075,
+            -0.04800017178058624,
+            -0.01671232469379902,
+            0.004305440001189709,
+            -0.007768895477056503,
+            -0.034047290682792664,
+            -0.015424583107233047,
+            0.002757675014436245,
+            -0.03206615149974823,
+            0.010068432427942753,
+            -0.012948159128427505,
+            -0.01214155275374651,
+            0.020264223217964172,
+            0.03223596513271332,
+            0.005154499784111977,
+            0.00014847487909719348,
+            -0.0214670579880476,
+            -0.0021686397958546877,
+            -0.021382151171565056,
+            0.010174565017223358,
+            -0.008801918476819992,
+            -0.010584943927824497,
+            0.0059611061587929726,
+            -0.012191081419587135,
+            0.013266556896269321,
+            0.010797208175063133,
+            0.024778390303254128,
+            0.01584911346435547,
+            -0.01876421831548214,
+            -0.0024746551644057035,
+            -0.01108730398118496,
+            -0.0016450530383735895,
+            -0.009820790030062199,
+            0.006360871717333794,
+            -0.0031592093873769045,
+            0.011264191009104252,
+            0.015184016898274422,
+            0.004432798828929663,
+            0.00046786724124103785,
+            0.038151077926158905,
+            -2.4487851987942122e-05,
+            -0.0003418349369894713,
+            0.006803090218454599,
+            -0.015750056132674217,
+            0.008837295696139336,
+            0.012191081419587135,
+            0.005051904823631048,
+            -0.007698140572756529,
+            0.006739411037415266,
+            0.03099067695438862,
+            -0.01516986545175314,
+            -0.009014182724058628,
+            -0.023504801094532013,
+            0.007549555506557226,
+            -0.0004656561359297484,
+            0.029971804469823837,
+            0.020533092319965363,
+            -0.027509532868862152,
+            -0.016018925234675407,
+            -0.02179253101348877,
+            0.01267929095774889,
+            -0.011497682891786098,
+            -0.003704022616147995,
+            0.012240610085427761,
+            0.02367461286485195,
+            0.0018838511314243078,
+            -0.014455240219831467,
+            -0.008950503543019295,
+            0.008943428285419941,
+            0.014617976732552052,
+            0.003497064346447587,
+            0.007570781745016575,
+            0.00743634719401598,
+            0.008886824361979961,
+            -0.014801939949393272,
+            -0.012254760600626469,
+            0.00767691433429718,
+            -0.010202866978943348,
+            0.01794346049427986,
+            -0.009403335861861706,
+            0.016768928617239,
+            -0.004645063541829586,
+            -0.011964665725827217,
+            -0.0320095494389534,
+            -0.002292460994794965,
+            0.005037753842771053,
+            0.004627374932169914,
+            0.0034705311991274357,
+            -0.0022217060904949903,
+            -0.013160424306988716,
+            0.004673365969210863,
+            0.005073131527751684,
+            1.875283123808913e-05,
+            -0.008313708938658237,
+            -0.00479718716815114,
+            0.004252373706549406,
+            -0.001031253719702363,
+            -0.0073372903279960155,
+            -0.0059858704917132854,
+            0.018481198698282242,
+            0.0066509670577943325,
+            0.00800238735973835,
+            0.03234917297959328,
+            0.02974538877606392,
+            -0.0016353242099285126,
+            -0.004606148693710566,
+            -0.00023304918431676924,
+            -0.01589156687259674,
+            -0.0057205394841730595,
+            -0.023320838809013367,
+            0.022259512916207314,
+            -0.009665128774940968,
+            0.021820832043886185,
+            0.019457615911960602,
+            0.0039304387755692005,
+            -0.008391539566218853,
+            -0.0011586126638576388,
+            -0.007308988366276026,
+            -0.003560743760317564,
+            0.016641568392515182,
+            0.006742948666214943,
+            0.01226891204714775,
+            0.00030933189555071294,
+            -0.028401045128703117,
+            0.026405755430459976,
+            0.000579748535528779,
+            0.0034015451092272997,
+            0.008908051066100597,
+            0.014129767194390297,
+            0.013209952972829342,
+            -0.01235381793230772,
+            0.01768874190747738,
+            -0.03280200436711311,
+            -0.0033484790474176407,
+            -0.016500059515237808,
+            0.002076658420264721,
+            -0.0018520113080739975,
+            -0.0008145665633492172,
+            -0.05354736000299454,
+            0.0003842879377771169,
+            0.005313698202371597,
+            -0.015339677222073078,
+            0.029349161311984062,
+            -0.020589696243405342,
+            -0.017165156081318855,
+            -0.032858606427907944,
+            -0.010238244198262691,
+            0.018099121749401093,
+            0.005617944523692131,
+            0.00035178486723452806,
+            -0.02660386823117733,
+            0.015750056132674217,
+            -0.0047476585023105145,
+            -0.00743634719401598,
+            -0.029858596622943878,
+            -0.010938717983663082,
+            0.009403335861861706,
+            0.004485865123569965,
+            -0.0130825936794281,
+            0.009898620657622814,
+            -0.002890340518206358,
+            0.010867963545024395,
+            -0.0138467475771904,
+            -0.00420284504070878,
+            0.0010020672343671322,
+            -0.005363226868212223,
+            0.012615610845386982,
+            -0.030537845566868782,
+            0.011391550302505493,
+            -0.01352127455174923,
+            0.0008548084879294038,
+            -0.017858553677797318,
+            -0.027707645669579506,
+            0.0020288988016545773,
+            -0.016132133081555367,
+            0.039990708231925964,
+            0.010188715532422066,
+            -0.01481609046459198,
+            0.016995344310998917,
+            -0.002080196049064398,
+            0.017377421259880066,
+            0.0014955832157284021,
+            0.017023645341396332,
+            -0.011971740983426571,
+            0.021056678146123886,
+            -0.0041745430789887905,
+            0.02156611531972885,
+            0.00600002147257328,
+            0.034047290682792664,
+            -0.02033497765660286,
+            0.0070825726725161076,
+            0.003792466362938285,
+            0.005798369646072388,
+            -0.0026816134341061115,
+            0.021424604579806328,
+            0.017985913902521133,
+            0.019698183983564377,
+            0.0037110981065779924,
+            -0.024084990844130516,
+            -0.006137993652373552,
+            0.004867942072451115,
+            0.023858575150370598,
+            0.011150983162224293,
+            -0.016415152698755264,
+            -0.021056678146123886,
+            0.015226469375193119,
+            -0.007860877551138401,
+            0.006771250627934933,
+            -0.00851889792829752,
+            0.00037544354563578963,
+            -0.013259481638669968,
+            0.012233533896505833,
+            -0.012523629702627659,
+            0.0032335021533071995,
+            0.01043635793030262,
+            -0.027806703001260757,
+            0.01511326152831316,
+            -0.00371817359700799,
+            -0.014313730411231518,
+            0.023320838809013367,
+            -0.015877414494752884,
+            -0.013988257385790348,
+            0.005501199048012495,
+            0.008610880002379417,
+            0.01746232621371746,
+            -0.002761212643235922,
+            0.002669231267645955,
+            -0.020504789426922798,
+            -0.012445799075067043,
+            0.004591997712850571,
+            0.02210385352373123,
+            -0.015820810571312904,
+            0.012431648559868336,
+            -0.018099121749401093,
+            0.01364863384515047,
+            0.016202887520194054,
+            0.007149789482355118,
+            0.00573115237057209,
+            -0.007726442534476519,
+            -0.0011471149045974016,
+            0.032151058316230774,
+            -0.0029876285698264837,
+            0.02704254910349846,
+            -0.014915147796273232,
+            0.009615600109100342,
+            0.00518987700343132,
+            -0.013089669868350029,
+            -0.014589674770832062,
+            0.011108530685305595,
+            0.02428310550749302,
+            -0.009530694223940372,
+            -0.023660462349653244,
+            0.0018484735628589988,
+            0.014264201745390892,
+            0.010337301529943943,
+            0.009403335861861706,
+            -0.001997058978304267,
+            -0.0037358624394983053,
+            0.01484439242631197,
+            -0.0032918748911470175,
+            -0.0286557637155056,
+            -0.01633024774491787,
+            -0.004400959238409996,
+            -0.017490629106760025,
+            0.020745357498526573,
+            0.02367461286485195,
+            0.010507113300263882,
+            0.0037570889107882977,
+            0.005540114361792803,
+            0.018297234550118446,
+            0.01843874529004097,
+            0.013330236077308655,
+            0.009304278530180454,
+            0.012700516730546951,
+            -0.004319590982049704,
+            0.02800481766462326,
+            -0.010860888287425041,
+            -0.01046465989202261,
+            -0.015707602724432945,
+            0.0016636261716485023,
+            -0.005193414632230997,
+            -0.012431648559868336,
+            -0.0069481381215155125,
+            0.004369119182229042,
+            0.013330236077308655,
+            -0.012240610085427761,
+            -0.007149789482355118,
+            0.01016748882830143,
+            0.011971740983426571,
+            0.01403071079403162,
+            0.01719345711171627,
+            -0.01633024774491787,
+            -0.011278342455625534,
+            0.0009834941010922194,
+            -0.02844349853694439,
+            -0.0160047747194767,
+            -0.01767459139227867,
+            0.009580222889780998,
+            -0.014469391666352749,
+            0.03724541515111923,
+            0.028471799567341805,
+            0.0073514413088560104,
+            0.031613320112228394,
+            -0.028401045128703117,
+            0.00970758218318224,
+            -0.014398636296391487,
+            0.007132100872695446,
+            -0.004188694059848785,
+            0.003902136581018567,
+            0.002448122017085552,
+            0.020419884473085403,
+            0.00791748147457838,
+            0.014264201745390892,
+            0.008150972425937653,
+            -0.021934039890766144,
+            -0.00662620272487402,
+            -0.012559006921947002,
+            -0.016401002183556557,
+            -0.02595292218029499,
+            0.013719388283789158,
+            0.005087282508611679,
+            -0.003074303502216935,
+            -0.0002704166399780661,
+            0.006566061172634363,
+            -0.0013195802457630634,
+            0.010443433187901974,
+            -0.027608588337898254,
+            -0.01311089564114809,
+            0.0069269114173948765,
+            0.01449061743915081,
+            -0.02114158496260643,
+            0.0014637433923780918,
+            0.008900974877178669,
+            -0.004602610599249601,
+            0.035660505294799805,
+            -0.014674580655992031,
+            -0.008610880002379417,
+            0.0027806703001260757,
+            0.005062518175691366,
+            -0.017660440877079964,
+            -0.02319347858428955,
+            0.0070648835971951485,
+            0.011575513519346714,
+            0.010867963545024395,
+            -0.028570856899023056,
+            0.02087271586060524,
+            -0.0054339817725121975,
+            0.03832089155912399,
+            0.00401534466072917,
+            -0.013337312266230583,
+            0.003962278366088867,
+            0.0011860302183777094,
+            -0.012360893189907074,
+            0.002060738392174244,
+            -0.000340508297085762,
+            0.003700484987348318,
+            0.011228813789784908,
+            -0.014801939949393272,
+            -0.016967041417956352,
+            0.016910437494516373,
+            -0.011476456187665462,
+            -0.0035395172890275717,
+            -0.00943871308118105,
+            -0.016669871285557747,
+            0.007308988366276026,
+            0.025047259405255318,
+            0.0041745430789887905,
+            0.001242634141817689,
+            -0.011632117442786694,
+            -0.0038596836384385824,
+            -0.008575502783060074,
+            0.0011718792375177145,
+            -0.0207029040902853,
+            0.027396325021982193,
+            -0.01801421493291855,
+            0.002410975517705083,
+            0.021099131554365158,
+            0.008709936402738094,
+            -0.003944589756429195,
+            0.016316095367074013,
+            0.026363302022218704,
+            0.01205664686858654,
+            0.0031539027113467455,
+            -0.00902833417057991,
+            1.648646139074117e-05,
+            0.014957600273191929,
+            0.0015123875346034765,
+            -0.0015663381200283766,
+            0.0032140444964170456,
+            -0.006583749782294035,
+            -0.001859086798503995,
+            -0.017561383545398712,
+            0.005564878229051828,
+            0.02125479280948639,
+            -0.004132090136408806,
+            0.020476488396525383,
+            0.015509488992393017,
+            0.011625041253864765,
+            0.010726453736424446,
+            -0.0012390963966026902,
+            -0.002150951186195016,
+            -0.0021668709814548492,
+            -0.007393894251435995,
+            -0.018297234550118446,
+            0.006470541935414076,
+            0.0034493047278374434,
+            0.01822648011147976,
+            0.020504789426922798,
+            -0.02308027073740959,
+            0.010146263055503368,
+            0.014985902234911919,
+            -0.013011839240789413,
+            0.0235755555331707,
+            -0.02006610855460167,
+            0.0024605041835457087,
+            0.02156611531972885,
+            0.013337312266230583,
+            -0.014271277002990246,
+            -0.009219372645020485,
+            0.004662752617150545,
+            -0.01543873455375433,
+            0.000872939417604357,
+            -0.01478778850287199,
+            -0.010783057659864426,
+            0.014533070847392082,
+            -0.01490099634975195,
+            0.01229721400886774,
+            -0.006969364359974861,
+            0.0070967236533761024,
+            -0.017165156081318855,
+            0.015014204196631908,
+            0.008632106706500053,
+            0.01757553406059742,
+            0.0021279556676745415,
+            -0.02518877014517784,
+            -0.006290116813033819,
+            0.021169887855648994,
+            0.009509467519819736,
+            -0.02497650496661663,
+            0.019613277167081833,
+            0.0020094411447644234,
+            -0.019528372213244438,
+            -0.022401023656129837,
+            -0.012686366215348244,
+            -0.005055442452430725,
+            -0.017264213413000107,
+            -0.0011276573641225696,
+            -0.024792540818452835,
+            0.003972891718149185,
+            0.009106164798140526,
+            -0.011978816241025925,
+            -0.019273653626441956,
+            -0.0002315014135092497,
+            -0.01040805596858263,
+            0.006219361908733845,
+            -0.01132787112146616,
+            -0.019853845238685608,
+            0.0070436573587358,
+            0.012127402238547802,
+            -0.0027647505048662424,
+            -0.005646246485412121,
+            -0.02562745101749897,
+            -0.006919836159795523,
+            0.020179318264126778,
+            -0.003994117956608534,
+            -0.004974074196070433,
+            -0.01703779585659504,
+            0.01108730398118496,
+            0.011165134608745575,
+            0.020901018753647804,
+            0.010931642726063728,
+            0.005674548447132111,
+            0.013634482398629189,
+            -0.0029416377656161785,
+            -0.007156865205615759,
+            -9.645892532716971e-06,
+            0.004284213297069073,
+            0.0069233737885952,
+            0.0016636261716485023,
+            0.020235922187566757,
+            -0.01692458800971508,
+            0.018934030085802078,
+            -0.012290137819945812,
+            0.008794843219220638,
+            -0.01855195313692093,
+            -0.0003155229496769607,
+            -0.01067692507058382,
+            0.00015002264990471303,
+            -0.0038455326575785875,
+            -0.009544845670461655,
+            1.9595809135353193e-05,
+            0.00143986358307302,
+            0.008511822670698166,
+            -0.016783079132437706,
+            0.009615600109100342,
+            -0.010839661583304405,
+            -0.004308977629989386,
+            -0.019542522728443146,
+            0.017448175698518753,
+            0.013344387523829937,
+            0.007308988366276026,
+            -0.008278331719338894,
+            -0.006300730165094137,
+            -0.02282555401325226,
+            0.008582578040659428,
+            -0.0014637433923780918,
+            0.002214630600064993,
+            -0.028301987797021866,
+            -0.004358506295830011,
+            0.0024587351363152266,
+            0.020306676626205444,
+            0.0033661676570773125,
+            -0.010712302289903164,
+            0.008978805504739285,
+            -0.023377442732453346,
+            -0.0030159305315464735,
+            -0.023349139839410782,
+            -0.004075486212968826,
+            -0.005112046375870705,
+            0.010103809647262096,
+            0.008101443760097027,
+            0.01549533847719431,
+            -0.009898620657622814,
+            0.01478778850287199,
+            -0.004793649073690176,
+            0.012049571610987186,
+            -0.0012152165872976184,
+            0.01337976474314928,
+            -0.0003111007681582123,
+            -0.01505665760487318,
+            0.02158026583492756,
+            -0.008738238364458084,
+            0.001741456682793796,
+            -0.0052853962406516075,
+            0.0027346794959157705,
+            -0.017702894285321236,
+            -0.004553082399070263,
+            -0.014455240219831467,
+            0.01957082375884056,
+            -0.022584985941648483,
+            0.004153316840529442,
+            -0.005055442452430725,
+            -0.016075529158115387,
+            0.029434068128466606,
+            0.022896308451890945,
+            -0.0035483618266880512,
+            0.015070809051394463,
+            0.0017989451298490167,
+            0.0015061964513733983,
+            0.016202887520194054,
+            -0.015806660056114197,
+            -0.013025989755988121,
+            0.035066161304712296,
+            0.0176321379840374,
+            0.0004313841927796602,
+            -0.0023260696325451136,
+            -0.0037323245778679848,
+            0.0032193511724472046,
+            0.008023613132536411,
+            -0.010450509376823902,
+            -0.01141277700662613,
+            -0.017051948234438896,
+            0.009997677057981491,
+            -0.006155682262033224,
+            0.004446949809789658,
+            -0.00025891896802932024,
+            -7.617214578203857e-05,
+            -0.01516986545175314,
+            0.022584985941648483,
+            -0.017717044800519943,
+            -0.00602832343429327,
+            0.004687516950070858,
+            0.0207029040902853,
+            -0.02541518583893776,
+            0.01616043597459793,
+            -0.007107336539775133,
+            -0.013429293408989906,
+            0.012573158368468285,
+            -0.015127412974834442,
+            -0.0107335289940238,
+            -0.003175129182636738,
+            -0.008009462617337704,
+            -0.022754797711968422,
+            0.017929309979081154,
+            -0.013627407141029835,
+            0.015650998800992966,
+            -0.00287972716614604,
+            0.000562502013053745,
+            0.002057200763374567,
+            0.0032582662533968687,
+            0.02250008098781109,
+            0.030735958367586136,
+            0.010415131226181984,
+            -0.005465821363031864,
+            -0.021000074222683907,
+            0.012983537279069424,
+            0.0020589695777744055,
+            0.013202877715229988,
+            -0.002253545681014657,
+            -0.002135031158104539,
+            -0.016613267362117767,
+            -0.023122724145650864,
+            -0.0013010069960728288,
+            -0.01761798746883869,
+            -0.02552839368581772,
+            -0.01774534583091736,
+            -0.007047194987535477,
+            0.011349096894264221,
+            0.020405733957886696,
+            0.0009755341452546418,
+            -0.005384453106671572,
+            0.01559439580887556,
+            -0.006941062398254871,
+            0.002080196049064398,
+            -0.006594363134354353,
+            -0.002032436430454254,
+            -0.008200501091778278,
+            -0.021863285452127457,
+            0.029660483822226524,
+            -0.010103809647262096,
+            0.01141277700662613,
+            -0.005472897086292505,
+            -0.00965097825974226,
+            0.005575491581112146,
+            0.00900003220885992,
+            0.024849144741892815,
+            0.006452853325754404,
+            0.03141520544886589,
+            0.020151015371084213,
+            -0.018891576677560806,
+            -0.0006071660900488496,
+            0.022684043273329735,
+            -0.0037570889107882977,
+            -0.004889168310910463,
+            -0.0031786670442670584,
+            -0.017985913902521133,
+            -0.012007118202745914,
+            0.007839650847017765,
+            -0.013471745885908604,
+            -0.006636816076934338,
+            -0.00837738811969757,
+            0.018070818856358528,
+            -0.017773648723959923,
+            0.025698205456137657,
+            -0.006488230545073748,
+            0.005154499784111977,
+            0.031924642622470856,
+            0.014363259077072144,
+            0.019698183983564377,
+            -0.011794853024184704,
+            -0.031075581908226013,
+            -0.00683846790343523,
+            -0.007882103323936462,
+            0.009558996185660362,
+            -0.01062032114714384,
+            0.005204027984291315,
+            -0.02006610855460167,
+            -0.018948180601000786,
+            0.012863253243267536,
+            -0.017476476728916168,
+            0.013924578204751015,
+            -0.009955224581062794,
+            0.027226513251662254,
+            -0.001228483160957694,
+            0.0121698547154665,
+            -0.002614396158605814,
+            -0.017801949754357338,
+            0.01205664686858654,
+            0.004089637193828821,
+            -0.023943481966853142,
+            0.010337301529943943,
+            -0.001864393474534154,
+            -0.014037786051630974,
+            -0.018410442396998405,
+            -0.001228483160957694,
+            -0.005200490355491638,
+            -0.009658053517341614,
+            0.025457637384533882,
+            0.014101465232670307,
+            0.016231190413236618,
+            0.0035359796602278948,
+            0.0016335553955286741,
+            -0.011073152534663677,
+            0.009573147632181644,
+            -0.006803090218454599,
+            0.000612030504271388,
+            -0.0034404604230076075,
+            0.011893910355865955,
+            -0.004252373706549406,
+            -0.01543873455375433,
+            0.002324300818145275,
+            0.008561351336538792,
+            0.0143278818577528,
+            5.947839599684812e-05,
+            0.009587298147380352,
+            0.0009790719486773014,
+            -0.009764186106622219,
+            -0.00046388726332224905,
+            -0.0060884649865329266,
+            0.0069375247694551945,
+            0.02400008589029312,
+            -0.012516554445028305,
+            -0.01681138016283512,
+            0.014504768885672092,
+            0.004033033270388842,
+            -0.017589686438441277,
+            0.018367990851402283,
+            0.004103788174688816,
+            -0.022358570247888565,
+            0.0035412863362580538,
+            0.010662773624062538,
+            -0.009523618966341019,
+            -0.014469391666352749,
+            -0.0010339069413021207,
+            -0.015509488992393017,
+            0.03130199760198593,
+            0.00025073791039176285,
+            0.016910437494516373,
+            -0.01062032114714384,
+            -0.0025967073161154985,
+            -0.013309009373188019,
+            0.013103820383548737,
+            -0.015750056132674217,
+            -0.007273610681295395,
+            0.00035620704875327647,
+            -0.02298121340572834,
+            0.01741987280547619,
+            0.0013425755314528942,
+            0.00794578343629837,
+            0.003436922561377287,
+            -0.010556641966104507,
+            0.009820790030062199,
+            0.0018237093463540077,
+            0.006668656133115292,
+            -0.014915147796273232,
+            0.023618008941411972,
+            0.006413938011974096,
+            0.011235889047384262,
+            -0.01352127455174923,
+            -0.01546703651547432,
+            0.002253545681014657,
+            0.04134920611977577,
+            -0.01306136790663004,
+            0.0002593612007331103,
+            0.01562269777059555,
+            7.081024523358792e-05,
+            0.004156854469329119,
+            -0.0023755980655550957,
+            -0.0034493047278374434,
+            -0.034585028886795044,
+            0.005370302125811577,
+            -0.0006328147719614208,
+            0.008073141798377037,
+            -0.008992956951260567,
+            -0.005430444143712521,
+            -0.0040684109553694725,
+            -0.008625030517578125,
+            -0.00314859626814723,
+            -0.01478778850287199,
+            0.008363237604498863,
+            -0.014533070847392082,
+            -0.028910480439662933,
+            0.03166992589831352,
+            -0.006887996103614569,
+            -0.00533846253529191,
+            -0.021396303549408913,
+            0.007570781745016575,
+            -0.01370523776859045,
+            0.002663924591615796,
+            0.010967019945383072,
+            0.008865597657859325,
+            0.008738238364458084,
+            0.002762981690466404,
+            0.0009516543359495699,
+            -0.004255911335349083,
+            0.010273621417582035,
+            -0.00835616234689951,
+            -0.0005943417199887335,
+            -0.0014186371117830276,
+            -0.02725481428205967,
+            -0.01405901275575161,
+            -0.010592019185423851,
+            -0.007811348885297775,
+            -0.00435143057256937,
+            0.012608535587787628,
+            0.007174553815275431,
+            -0.0017485321732237935,
+            -0.022938761860132217,
+            -0.014745336025953293,
+            -0.0034103896468877792,
+            0.008349086157977581,
+            -0.01838214136660099,
+            0.01064862310886383,
+            0.0016830838285386562,
+            0.009976450353860855,
+            0.014292503707110882,
+            0.00302831269800663,
+            0.0016742395237088203,
+            0.003458149265497923,
+            0.014886845834553242,
+            0.012601460330188274,
+            -0.005801907740533352,
+            -0.021382151171565056,
+            -0.0024605041835457087,
+            -0.007068421691656113,
+            -0.0034687623847275972,
+            -0.012332591228187084,
+            0.03039633482694626,
+            0.0029805530793964863,
+            0.0176321379840374,
+            0.007832575589418411,
+            0.009573147632181644,
+            0.010507113300263882,
+            0.029377464205026627,
+            0.006049550138413906,
+            -0.0029929352458566427,
+            -0.008603804744780064,
+            -0.02141045406460762,
+            -0.013146273791790009,
+            0.009898620657622814,
+            0.013294858857989311,
+            0.03597182780504227,
+            -0.034896351397037506,
+            0.012162779457867146,
+            -0.011101454496383667,
+            -0.008264180272817612,
+            -0.008533049374818802,
+            0.0044717141427099705,
+            0.010068432427942753,
+            0.014370334334671497,
+            -0.0023402206134051085,
+            -0.01175947580486536,
+            0.009070787578821182,
+            0.023547254502773285,
+            -0.012983537279069424,
+            0.03016991913318634,
+            0.027325568720698357,
+            0.023066120222210884,
+            0.01761798746883869,
+            -0.012537780217826366,
+            0.026462359353899956,
+            -6.484030018327758e-05,
+            0.0006261814851313829,
+            -0.018084971234202385,
+            -0.00587973790243268,
+            -0.004294826649129391,
+            -0.004362043924629688,
+            0.013316085562109947,
+            -0.0025471788831055164,
+            -0.0008468485320918262,
+            0.015099111013114452,
+            -0.0037323245778679848,
+            -3.927453508367762e-05,
+            -0.0059752571396529675,
+            0.015127412974834442,
+            0.0002699744363781065,
+            0.019457615911960602,
+            -0.04270770028233528,
+            0.019726485013961792,
+            -0.021396303549408913,
+            -0.003102605463936925,
+            -0.0020890405867248774,
+            0.009523618966341019,
+            0.00764861237257719,
+            -0.007142714224755764,
+            -0.014334957115352154,
+            0.004524780437350273,
+            0.014066088013350964,
+            -0.0006726144347339869,
+            0.005080206785351038,
+            -0.0027541371528059244,
+            0.0017228835495188832,
+            -0.010089659132063389,
+            -0.005862049292773008,
+            5.842812606715597e-05,
+            -0.010570792481303215,
+            -0.008646257221698761,
+            -0.03232087194919586,
+            0.01062032114714384,
+            0.010209942236542702,
+            0.007294837385416031,
+            0.004252373706549406,
+            -0.022018946707248688,
+            -0.012318440712988377,
+            0.012948159128427505,
+            -0.01132787112146616,
+            -0.002570174168795347,
+            -0.0020005968399345875,
+            0.005387990735471249,
+            0.0004705205501522869,
+            0.010160413570702076,
+            -0.003472300246357918,
+            0.010938717983663082,
+            -0.014603826217353344,
+            0.018084971234202385,
+            0.03229256719350815,
+            -0.00533846253529191,
+            -0.019967053085565567,
+            0.000907432462554425,
+            -0.009077862836420536,
+            0.003813692834228277,
+            0.007641536649316549,
+            -0.021962342783808708,
+            -0.02173592709004879,
+            0.005087282508611679,
+            -0.013153349049389362,
+            -0.007783046457916498,
+            0.014830241911113262,
+            0.005221716593950987,
+            0.010839661583304405,
+            0.01751893013715744,
+            0.010889190249145031,
+            0.00840569008141756,
+            -0.026533113792538643,
+            0.0006734988419339061,
+            -0.012686366215348244,
+            -0.00590803986415267,
+            0.002239394700154662,
+            -0.0017591454088687897,
+            0.004591997712850571,
+            -0.00010115749319083989,
+            -0.011200511828064919,
+            -0.0007057808106765151,
+            0.010903340764343739,
+            0.027919910848140717,
+            -0.004146241117268801,
+            -0.0022570835426449776,
+            -0.014257126487791538,
+            0.0010100272484123707,
+            0.036481261253356934,
+            -0.012275987304747105,
+            -0.017434025183320045,
+            0.01757553406059742,
+            0.027452928945422173,
+            0.005954030901193619,
+            -0.0032529598101973534,
+            -0.0027364485431462526,
+            0.00039600671152584255,
+            0.011978816241025925,
+            0.007719367276877165,
+            -0.002692226553335786,
+            -0.002131493529304862,
+            0.004761809483170509,
+            0.035773713141679764,
+            -0.0015203474322333932,
+            -0.010330225341022015,
+            -0.024240652099251747,
+            0.010974096134305,
+            0.00277182599529624,
+            0.004754733759909868,
+            0.002313687466084957,
+            0.023504801094532013,
+            0.019655730575323105,
+            0.014858543872833252,
+            0.00470520555973053,
+            -0.009771261364221573,
+            0.03005671128630638,
+            -0.008172199130058289,
+            -0.004542469047009945,
+            -0.02838689461350441,
+            -0.005087282508611679,
+            -0.015650998800992966,
+            0.008929276838898659,
+            -0.0002221042668679729,
+            -0.005016527138650417,
+            -0.013422218151390553,
+            0.019726485013961792,
+            0.01064862310886383,
+            -0.011073152534663677,
+            0.012608535587787628,
+            -0.027070851996541023,
+            0.02346234768629074,
+            -0.03826428949832916,
+            -0.0071993181481957436,
+            -0.006421013735234737,
+            -0.010514188557863235,
+            -0.02627839520573616,
+            -0.0030353881884366274,
+            0.007372668012976646,
+            0.014250051230192184,
+            0.0022570835426449776,
+            0.0022818478755652905,
+            0.02282555401325226,
+            0.0016415152931585908,
+            -0.01508495956659317,
+            0.017985913902521133,
+            0.03492465242743492,
+            0.012926933355629444,
+            0.00155837822239846,
+            0.015990622341632843,
+            0.0027789014857262373,
+            0.0021633331198245287,
+            0.00788917951285839,
+            -0.0034493047278374434,
+            -0.007938707247376442,
+            0.008582578040659428,
+            0.006990591064095497,
+            -0.02763689123094082,
+            0.003824306186288595,
+            -0.00010408719390397891,
+            -0.009898620657622814,
+            -0.022429324686527252,
+            0.011568437330424786,
+            0.01913214474916458,
+            -0.004224071744829416,
+            -0.006771250627934933,
+            -0.0043337419629096985,
+            0.0006836698739789426,
+            0.006785401608794928,
+            -0.002412744564935565,
+            -0.000515184598043561,
+            -0.0015176940942183137,
+            -0.02368876338005066,
+            -0.010316074825823307,
+            0.013570803217589855,
+            -0.005175726022571325,
+            0.010358527302742004,
+            0.005242943298071623,
+            -0.018792519345879555,
+            -0.016938740387558937,
+            0.006852618884295225,
+            -0.016188737004995346,
+            0.01364863384515047,
+            0.00027948213391937315,
+            -0.0023172253277152777,
+            0.024962352588772774,
+            0.03594352304935455,
+            -0.013188726268708706,
+            0.005119122099131346,
+            -0.019386861473321915,
+            -0.0245378240942955,
+            -0.019004784524440765,
+            0.022613288834691048,
+            0.008306633681058884,
+            -0.018792519345879555,
+            0.012304289266467094,
+            -0.002752368338406086,
+            -0.008865597657859325,
+            -0.0066721937619149685,
+            -0.006845543161034584,
+            -0.008547200821340084,
+            0.020533092319965363,
+            -0.004935159347951412,
+            0.0038950610905885696,
+            0.019047237932682037,
+            0.04075486212968826,
+            -0.008292482234537601,
+            -0.01349297259002924,
+            0.0054339817725121975,
+            0.004878554958850145,
+            0.0018626245437189937,
+            -0.004814875777810812,
+            -0.008016537874937057,
+            -0.0185944065451622,
+            -0.01740572229027748,
+            -0.009615600109100342,
+            -0.0020041344687342644,
+            0.020122714340686798,
+            -0.007768895477056503,
+            -0.01354957651346922,
+            -0.0245378240942955,
+            -0.028188779950141907,
+            -0.006580212153494358,
+            0.02319347858428955,
+            0.017179306596517563,
+            -0.00013200224202591926,
+            -4.3337418901501223e-05,
+            -0.012042495422065258,
+            -0.01968403346836567,
+            0.015509488992393017,
+            0.00037876018905080855,
+            -0.01773119531571865,
+            -0.0065448349341750145,
+            -0.015919867902994156,
+            0.005787756759673357,
+            -0.0051368107087910175,
+            0.004574308637529612,
+            0.05057565122842789,
+            0.0005271245026960969,
+            0.008441068232059479,
+            -0.026575567200779915,
+            0.009106164798140526,
+            -0.01583496294915676,
+            -0.017377421259880066,
+            -0.00012702727690339088,
+            -0.022301966324448586,
+            -0.0030955299735069275,
+            -0.018792519345879555,
+            -0.007386818993836641,
+            0.008568426594138145,
+            0.009799563325941563,
+            -0.013896276243031025,
+            -0.013669860549271107,
+            -0.024693485349416733,
+            -0.0035872769076377153,
+            -0.001539805089123547,
+            0.009311353787779808,
+            0.007825499400496483,
+            -0.03693409264087677,
+            0.002536565763875842,
+            -0.0009507699287496507,
+            0.0005779796629212797,
+            0.01208494883030653,
+            -0.008667483925819397,
+            -0.003063690150156617,
+            -0.0214670579880476,
+            0.006452853325754404,
+            0.007103798910975456,
+            -0.013500047847628593,
+            0.012375044636428356,
+            -0.012863253243267536,
+            -0.007669838611036539,
+            -0.0005426021525636315,
+            0.03379257395863533,
+            0.011794853024184704,
+            -0.0025153392925858498,
+            -0.00654129683971405,
+            -0.011150983162224293,
+            0.02654726430773735,
+            -0.011349096894264221,
+            -0.026405755430459976,
+            0.028910480439662933,
+            0.013245330192148685,
+            0.0005974372616037726,
+            0.020151015371084213,
+            -0.0017936384538188577,
+            -0.01397410687059164,
+            0.009601449593901634,
+            0.004266524687409401,
+            0.012424572370946407,
+            -0.00014460546663030982,
+            0.0214670579880476,
+            0.0018555490532889962,
+            0.0027399861719459295,
+            -0.0036226543597877026,
+            0.004687516950070858,
+            -0.03093407303094864,
+            -0.0006549257086589932,
+            -0.003102605463936925,
+            -0.008398614823818207,
+            0.011207587085664272,
+            -0.002421588869765401,
+            0.006558985915035009,
+            -0.022188758477568626,
+            -0.028485950082540512,
+            0.011342021636664867,
+            -0.016231190413236618,
+            -0.021396303549408913,
+            0.015339677222073078,
+            -0.0007641536649316549,
+            0.00614860700443387,
+            -0.007443422917276621,
+            -0.023278385400772095,
+            -0.019386861473321915,
+            0.02064630016684532,
+            0.012212308123707771,
+            -0.01190098561346531,
+            0.012750045396387577,
+            -0.021113283932209015,
+            -0.012948159128427505,
+            -0.001894464367069304,
+            0.0090566361322999,
+            -0.0019864458590745926,
+            0.01325240544974804,
+            -0.012198156677186489,
+            -0.00334494118578732,
+            0.0031592093873769045,
+            -0.011589664034545422,
+            0.008158047683537006,
+            0.01978308893740177,
+            0.008193425834178925,
+            -0.009339655749499798,
+            0.003422771580517292,
+            -0.011922212317585945,
+            0.013500047847628593,
+            0.015537790954113007,
+            0.018311386927962303,
+            -0.006498843897134066,
+            0.0020412809681147337,
+            -0.01370523776859045,
+            0.0063962494023144245,
+            -0.009827865287661552,
+            -0.004878554958850145,
+            -0.0005333155859261751,
+            0.004921008367091417,
+            -0.011179285123944283,
+            -0.0025772498920559883,
+            -0.026830283924937248,
+            -0.0013823752524331212,
+            -0.020533092319965363,
+            0.005303084850311279,
+            0.023207630962133408,
+            -0.007514177821576595,
+            0.0006045127520337701,
+            -0.0039092120714485645,
+            0.014476466923952103,
+            0.020688753575086594,
+            0.00870286114513874,
+            0.00482548912987113,
+            -0.006081389728933573,
+            0.02179253101348877,
+            0.011603815481066704,
+            -0.017816102132201195,
+            0.00164239970035851,
+            -0.001509734196588397,
+            -0.0014336726162582636,
+            0.011625041253864765,
+            0.004443412180989981,
+            -0.020844414830207825,
+            0.0028602697420865297,
+            -0.010089659132063389,
+            0.007818424142897129,
+            0.014702882617712021,
+            -0.00031906069489195943,
+            -0.02486329711973667,
+            -0.009827865287661552,
+            0.021919889375567436,
+            -0.015070809051394463,
+            -0.0008291598060168326,
+            0.00764861237257719,
+            0.023264234885573387,
+            -0.024792540818452835,
+            -0.006831392180174589,
+            -0.0023455272894352674,
+            -0.003933976404368877,
+            -0.005193414632230997,
+            0.0010489424457773566,
+            0.0214670579880476,
+            0.004415110219269991,
+            -0.004234685096889734,
+            0.005724077112972736,
+            0.006470541935414076,
+            0.02325008250772953,
+            0.0040577976033091545,
+            0.028047269210219383,
+            -0.00943871308118105,
+            -0.0023809047415852547,
+            -0.0019988277927041054,
+            -4.775960405822843e-05,
+            0.0160047747194767,
+            -0.005430444143712521,
+            0.020320827141404152,
+            0.009445788338780403,
+            0.0036934094969183207,
+            0.016514210030436516,
+            0.02319347858428955,
+            -0.006767712999135256,
+            -0.004793649073690176,
+            0.019160445779561996,
+            0.0053667644970119,
+            -0.008858522400259972,
+            -0.02194819226861,
+            0.01141277700662613,
+            0.0014124460285529494,
+            -0.0006956097786314785,
+            0.012106175534427166,
+            0.0024375086650252342,
+            0.006300730165094137,
+            0.008172199130058289,
+            -0.016684021800756454,
+            0.004241760354489088,
+            -0.004698129836469889,
+            -0.01860855706036091,
+            0.010910416021943092,
+            -0.012092024087905884,
+            -5.201595922699198e-05,
+            -0.0010002984199672937,
+            0.014165145345032215,
+            -0.01703779585659504,
+            -0.0019103841623291373,
+            -0.0070719593204557896,
+            -1.3024719009990804e-05,
+            -0.0009162768837995827,
+            0.008320784196257591,
+            -0.006254739128053188,
+            -0.010938717983663082,
+            -0.010938717983663082,
+            0.027509532868862152,
+            -0.004991763271391392,
+            0.003916287794709206,
+            0.019330257549881935
+          ],
+          "embedding": [
+            0.007204997251390758,
+            -0.03173324616100502,
+            -0.020505790884701024,
+            0.0037451608231038273,
+            0.009817085391002229,
+            0.026883177166072393,
+            -0.016477826654817607,
+            0.06949311853773875,
+            -0.01224876667472692,
+            0.031952566873802865,
+            0.013633671951171586,
+            -0.005041437355104989,
+            0.02663392750069342,
+            -0.022099132865267587,
+            0.006665702000169926,
+            0.0030467129715919033,
+            0.007931802508415542,
+            0.009984477355413895,
+            -0.010362836464183985,
+            -0.03180848181251013,
+            -0.013542377333474449,
+            -0.013572345022577053,
+            -0.03979985300694044,
+            -0.007698412130292187,
+            -0.005953452234356532,
+            0.018890228917932694,
+            0.02217446994660065,
+            0.005158357501784182,
+            -0.05468694715835175,
+            0.00789994048345594,
+            0.017130184174786147,
+            -0.0026591475669416114,
+            -0.02081141846859738,
+            -0.008403928318906045,
+            -0.030925346724945978,
+            -0.03855023691059801,
+            0.038853354785097736,
+            0.01612011653434478,
+            -0.035635702395196575,
+            0.026266623757525934,
+            0.043450373922427316,
+            -0.007254334835408145,
+            -0.015376804869420366,
+            -0.0017276202867766907,
+            0.004650414207094434,
+            -0.002365514361535186,
+            0.011906139905776183,
+            -0.0005664194302009289,
+            -0.0191723447249157,
+            -0.025546124518037998,
+            0.01025005257280377,
+            -0.014436532079436437,
+            0.029277529346694118,
+            -0.0045414983847733344,
+            -0.02260315456913697,
+            0.01717918656248626,
+            0.013666132349359846,
+            0.057804090189845364,
+            0.056600576093100735,
+            0.051208022172527734,
+            0.03553760896972431,
+            0.02094599510567085,
+            -0.02438674472832137,
+            -0.0027797634437333432,
+            0.03614248430646102,
+            -0.026729308357113124,
+            -0.002251394534452815,
+            0.018929251842788906,
+            0.00689697483318378,
+            -0.026603606434115974,
+            -0.0541443728344337,
+            0.027164630314707807,
+            -0.004335272389193875,
+            -0.013967321193992958,
+            0.003323121377857836,
+            -0.0245295956412367,
+            0.024489335129134787,
+            0.06212059558640156,
+            0.009214058207162371,
+            -0.002082293435905329,
+            0.013659934201796686,
+            -0.003344037927706573,
+            -0.00689041317919184,
+            0.020235785390418023,
+            0.0212739457082623,
+            0.0016782346585238381,
+            -0.013484221664059555,
+            -0.06465582506604173,
+            0.011882797526607326,
+            -0.032970655964890155,
+            -0.04027700439305653,
+            0.025396847912528105,
+            0.013861157517716707,
+            0.00604596157583048,
+            -0.00744611407420936,
+            0.04038084298991919,
+            0.021507532290328625,
+            0.005983822806623904,
+            -0.012711110369915742,
+            0.02410138142963901,
+            -0.018372791517880082,
+            0.021016674128298246,
+            0.013387285956415756,
+            0.02039383525127615,
+            0.010700292312227496,
+            0.0053934660497349735,
+            -0.014807678014449691,
+            0.009422805142740376,
+            0.0414357625490679,
+            0.12206398267002624,
+            -0.010877509834324155,
+            -0.011079302503786522,
+            -0.04107714853756596,
+            0.006631397986165343,
+            -0.031914267104222384,
+            -0.0179965295457466,
+            -0.057530036803978266,
+            -0.0320488164246567,
+            0.008208202363320509,
+            -0.0020203090915064162,
+            0.03502055780919601,
+            0.04464599164916666,
+            0.019332152079287082,
+            0.017133943087431935,
+            -0.039885565008919065,
+            0.04703059988373572,
+            -0.029370696964304416,
+            0.0008494086835971101,
+            0.0014082189719987074,
+            -0.030973762582071615,
+            -0.006561157852024518,
+            0.011634653995510726,
+            -0.019896826887411713,
+            -0.034275921680252924,
+            0.026274465530039866,
+            -0.007728234125236442,
+            0.04814514040690998,
+            0.06550407296280117,
+            -0.02749051703615343,
+            -0.010041099502187074,
+            0.02019601967763537,
+            -0.01871701432558228,
+            0.008532829759864928,
+            -0.020164138072967613,
+            -0.045196925901010855,
+            -0.004523517988516156,
+            -0.002113469149506399,
+            0.07929650663710812,
+            -0.01111269708042164,
+            0.03358254444720164,
+            -0.019973382759721527,
+            -0.0007145592863325417,
+            -0.0011740862928390678,
+            0.02166182480535788,
+            0.004543228023398081,
+            -0.013377688428761516,
+            -0.01758250073647576,
+            0.009548292014596834,
+            -0.04753617258004735,
+            -0.007125744019792479,
+            -0.009930358894299303,
+            -0.007931672044235996,
+            -0.024561936537429774,
+            0.012979195012881556,
+            0.009209145250061479,
+            0.04586756662401364,
+            0.04311807301838248,
+            -0.0036241105681042675,
+            -0.06579256720896123,
+            0.04967191755416882,
+            0.015475886231466817,
+            0.03157651656768371,
+            0.02515941109300714,
+            0.01773554572598468,
+            0.005824418191900522,
+            0.016317635547562007,
+            0.007650872352957098,
+            0.010015166465492963,
+            -0.033203307142867844,
+            0.040272188258443496,
+            -0.013408423788025047,
+            0.01036004294342104,
+            0.012391622448703885,
+            0.04214857862444935,
+            0.03418032388760535,
+            0.015263018079889338,
+            0.018786704580603317,
+            0.02704685343985817,
+            0.015348111464412869,
+            0.0013031600434754704,
+            0.03241427528588285,
+            0.011689924942195664,
+            0.04544406114742012,
+            0.016186615914968734,
+            0.011624500626103958,
+            -0.008272953078864385,
+            -0.02363080189189375,
+            0.02784298119317616,
+            -0.010585980875564109,
+            0.01455013604305903,
+            0.01749228557615982,
+            -0.054742056843690476,
+            -0.005876213780171779,
+            -0.010556605098730201,
+            0.003967456397644932,
+            0.014663888851767762,
+            0.0015548476559947807,
+            -0.003858367677774835,
+            0.012504286824484335,
+            -0.025806882498628832,
+            -0.014216299453397403,
+            0.0031620924516417956,
+            -0.006448390918604307,
+            -0.02785577502346934,
+            -0.00766402687921159,
+            0.005275000635711884,
+            -0.0090448751942,
+            0.019702662630533,
+            0.007009091837467867,
+            -0.01847307100043082,
+            -0.0006774105206839772,
+            0.032952656101827446,
+            0.015900033044760432,
+            -0.02170121261931376,
+            -0.017964879081469612,
+            -0.005062849745084668,
+            -0.03171847232054206,
+            -0.038177398521606544,
+            0.029426291950046455,
+            -0.025687201683643784,
+            0.030158867757101797,
+            0.015780411479509624,
+            -0.007011484049169482,
+            -0.012114023533118393,
+            0.020661407942701628,
+            -0.025067530380881376,
+            0.009777190971413269,
+            -0.004160771288889298,
+            -0.028825369485727344,
+            0.008307290529469729,
+            -0.014415903546305744,
+            -0.007561235629544945,
+            -0.03201704135271719,
+            -0.015182869388110948,
+            -0.00501561450194676,
+            -0.0003863746169672397,
+            -0.01073403445816663,
+            -0.03321103354401307,
+            -0.002304357889348034,
+            0.017001213960120047,
+            0.00028222146729658097,
+            -0.04002882852650754,
+            0.014509588052058002,
+            -0.004150541038999649,
+            -0.034333837380045466,
+            0.0564137596224324,
+            0.03584898691608747,
+            -0.006830698606679202,
+            0.006121456445207273,
+            -0.04505674213649008,
+            0.04358467937307492,
+            0.010847859073780811,
+            -0.04918983713702805,
+            0.049190442935189385,
+            -0.00528960443684769,
+            0.01419195371129309,
+            0.040316024601793406,
+            0.004600843996974165,
+            -0.006780970176765864,
+            0.005206651850992678,
+            0.03397517915308201,
+            -0.010037223444627569,
+            -0.027377913828275806,
+            -0.03328136706915221,
+            0.02562568440972934,
+            -0.03485770844248482,
+            -0.020069979230687374,
+            -0.0052041408068332524,
+            0.029818953377223505,
+            0.028789460147370508,
+            0.004071274130619537,
+            0.01928375754300958,
+            0.0014970947734202088,
+            0.010502599795027616,
+            0.016914145702515263,
+            0.011942576738188502,
+            -0.014909953528043828,
+            -0.003432489584767786,
+            -0.05733768990013957,
+            0.0030710780388681002,
+            -0.02627280695622545,
+            0.007930906480371135,
+            -0.031227917075485834,
+            0.013132797714872345,
+            0.002514351081823354,
+            -0.007152021475569949,
+            -0.0066245205132327055,
+            -0.005279224195782313,
+            0.028512607222275155,
+            -0.00921960825771865,
+            -0.0359704105589433,
+            -0.021347597388353182,
+            0.011200044645812517,
+            -0.0004171294784927543,
+            -0.016784468873218786,
+            -0.0018746544944553878,
+            0.007749103417240419,
+            0.02390272097796833,
+            -0.005128540108845805,
+            0.006397849195755672,
+            0.01214922062611769,
+            0.01163371266250103,
+            0.012307186722130844,
+            -0.004982589662882946,
+            -0.016343724553037307,
+            0.019806887143115643,
+            0.013139657463054087,
+            0.0205801609265216,
+            -0.03767676007629612,
+            -0.0249879011550972,
+            0.05547392586363707,
+            0.005903313549319051,
+            -0.05568806298774435,
+            0.018827693208840233,
+            -0.007931178941307645,
+            0.008183992600768387,
+            -0.010319022946041053,
+            0.05111352384595183,
+            -0.024897203290807753,
+            -0.02096678205188722,
+            0.006436523335212348,
+            -0.04917084523640266,
+            -0.010459284852442706,
+            -0.011138558192296276,
+            0.014789370043397917,
+            -0.03105725978752438,
+            -0.004934745004818259,
+            0.015595801375817122,
+            -4.687585727052446e-05,
+            0.0028886067775750254,
+            -0.00850352807499054,
+            -0.0034885579171087815,
+            0.016339681487281293,
+            -0.0034050317046826813,
+            -0.008309570448487812,
+            -0.003615165868022203,
+            -0.027638409919916757,
+            0.023352910161586186,
+            0.026364511529991978,
+            0.001249936082895606,
+            0.001987948967948376,
+            0.01756629246122769,
+            0.004656075485833687,
+            -0.006752579977941239,
+            -0.013681608638716132,
+            -0.03405936202918687,
+            -0.0401867651078737,
+            0.022991180183813725,
+            -0.02318908140203705,
+            0.018299512436825923,
+            0.00904081847012441,
+            0.019697091729447287,
+            -0.019236130984371835,
+            0.015159917756323988,
+            -0.0387297415849285,
+            -0.018793498297161176,
+            -0.005256288213225733,
+            -0.010414697711744128,
+            0.0007039894599589104,
+            -0.01739645696111503,
+            -0.013728845328097506,
+            0.022709522955937343,
+            -0.022789243441781803,
+            -0.0036763802589970026,
+            -0.01429671717544519,
+            0.0023674126354206932,
+            0.049448298344058644,
+            -0.009926033487593551,
+            -0.023371285193090733,
+            0.020846792916650956,
+            -0.026996519120163966,
+            0.011246908457705283,
+            0.04091365159640201,
+            0.005469162850197613,
+            0.005640919201612091,
+            0.029575753750229278,
+            -0.0007860452837141593,
+            -0.0052805237594885516,
+            -0.009782745443933828,
+            0.021178503581190618,
+            -0.0009176512672832309,
+            -0.05245650420332918,
+            -0.007698364493410058,
+            -0.039027418406703554,
+            0.005448840777995117,
+            0.017581625031764185,
+            0.025522517097715556,
+            -0.020775492472580457,
+            -0.015906320383873178,
+            -0.000643669839811819,
+            -0.0086896084429604,
+            0.0029399233482444638,
+            0.014381240985647907,
+            0.021086693776922936,
+            0.002436575081260609,
+            -0.004913514868101904,
+            -0.011582101006213432,
+            0.014400297663309155,
+            0.0467181039341492,
+            -0.01722692180137712,
+            0.0011622224560709315,
+            -0.013036540368441797,
+            -0.06208454638917413,
+            -0.019160212331432766,
+            0.0351496273293468,
+            0.02429814055116254,
+            -0.009896662169762772,
+            -0.01647101145477158,
+            -0.002915821099802562,
+            -0.03557863734177486,
+            -0.01786433999615293,
+            -0.010699364747311715,
+            0.02390901423299708,
+            0.007213809581960646,
+            0.0005601364043388203,
+            0.00916614601011734,
+            0.014737491634516238,
+            0.019303599465763203,
+            0.008864535607492504,
+            -0.0575839606069681,
+            0.018051837786406608,
+            0.006315830080225406,
+            0.02525619013082567,
+            -0.015671924543113065,
+            0.003988943108210209,
+            -0.03179878281917948,
+            -0.029440641244429775,
+            0.040586322748158446,
+            -0.03418722657593859,
+            -0.030650542761032924,
+            0.006566745611226466,
+            0.009571089937153852,
+            0.004491158558315032,
+            -0.011352813042040581,
+            -0.02535505987721779,
+            -0.013381948772812133,
+            -0.030918676056735445,
+            -0.02192708443081135,
+            -0.005232667170255381,
+            -0.03342408726436631,
+            -0.07498634115550039,
+            -0.006245638452568239,
+            -0.049946896465756174,
+            -0.03900385398225758,
+            0.04458524125677807,
+            -0.006106489377959481,
+            0.02398410968928338,
+            -0.04615183642824994,
+            -0.01165382599292015,
+            -0.04890013373371541,
+            0.013624324077017377,
+            0.01476398305411351,
+            0.020202554298470666,
+            0.00014553149242532853,
+            0.036224316512987846,
+            0.00021498496267774312,
+            -0.05055208658013767,
+            -0.057482169168904106,
+            -0.003960451575823462,
+            -0.010217078509789989,
+            -0.005593738972150311,
+            -0.034090261748104665,
+            0.05835341009022349,
+            0.0116359661127526,
+            0.04203678184748103,
+            0.03447818335186531,
+            -0.006747564131379555,
+            -0.025756962754841885,
+            -0.028123416424745064,
+            -0.008073625379614702,
+            0.047544241458556825,
+            -0.029605671083240564,
+            0.008171081332993893,
+            -0.026007452205474395,
+            0.0067084589781962805,
+            -0.0070645448617940995,
+            -0.0012298101311792947,
+            0.012372286141738867,
+            0.006377319686763927,
+            -0.015024537877377194,
+            0.007934745664790507,
+            0.026041755081605937,
+            -0.007303731324541215,
+            0.0407000170257006,
+            -0.012990703097937777,
+            -0.033189084013863494,
+            -0.022525679964848666,
+            0.01657717698191603,
+            0.03440284885162039,
+            0.010230633738702899,
+            -0.008707808754455952,
+            -0.0363777020540216,
+            0.011184076269109602,
+            0.031774012431843524,
+            0.01504703222643543,
+            -0.022509833303978505,
+            0.025230974050270107,
+            0.017502662694813216,
+            -0.018700579139537456,
+            -0.03511734747709348,
+            0.015162183119327901,
+            -0.017711638132862063,
+            0.018190335050681287,
+            -0.01590174153215615,
+            0.011103110869700727,
+            0.0025189370696524936,
+            -0.00920305384736575,
+            0.01929652151514404,
+            -0.006196715938358087,
+            0.011426611271994685,
+            0.007058186102245829,
+            -0.018191352599117545,
+            0.0027202185072220937,
+            0.057310566886380025,
+            -0.000531639834386418,
+            0.01494975681909879,
+            0.03845692752400366,
+            -0.001971953300066856,
+            0.045554909803324765,
+            -0.02532332999916258,
+            -0.042616337063845953,
+            0.02633963974172818,
+            -0.02970705577683783,
+            -0.01624154998005974,
+            -0.02620993449678864,
+            -0.021490817420853802,
+            -0.004038424045768319,
+            0.009940887982818171,
+            -0.02692933845395842,
+            -0.023801570856681796,
+            -0.001948460540662095,
+            0.02290911497874141,
+            -0.03300532378029689,
+            0.006525488380791647,
+            0.007039306647666766,
+            -0.003685585408710752,
+            0.03183735852538168,
+            -0.027197064244321705,
+            0.006861928286688311,
+            0.011209869849682412,
+            -0.001094897323854524,
+            -0.005328676992017907,
+            -0.004007474089818478,
+            0.010796946231051235,
+            -0.030543173497906445,
+            -0.012879972141706173,
+            0.00935894527375688,
+            0.0017573071057611829,
+            -0.010050882212831222,
+            -0.03102132152986303,
+            -0.002915141291985554,
+            0.007634137769426993,
+            -0.011094093612593696,
+            0.01199609810193786,
+            -0.010458016758814742,
+            -0.04929347607956551,
+            0.010973132159729983,
+            0.028148117094907874,
+            -0.025504019123994758,
+            0.012470127357818321,
+            0.009182729379865986,
+            -0.016020133756226275,
+            0.016320111225714855,
+            -0.012051271446461976,
+            0.024098195821827578,
+            0.04317620443481784,
+            0.024592886889234396,
+            0.007727787037954994,
+            0.00771746776156926,
+            -0.0003147538723468374,
+            0.022572868970938624,
+            -0.016683340264320543,
+            0.013672085891597506,
+            0.020827021901755415,
+            0.02092309219721837,
+            0.0057130785714458055,
+            -0.003901728981013356,
+            -0.0036968415923910314,
+            0.002227024825579113,
+            -0.02106678364798051,
+            0.014578546755457675,
+            0.03128101864424301,
+            -0.005234627629617124,
+            -0.022394744692442837,
+            0.002430617645746389,
+            -0.015780389432069438,
+            4.7277885648917e-05,
+            0.013357803020920043,
+            0.013777651543690907,
+            -0.027713499926132385,
+            -0.0018999112228756628,
+            0.028364630573666852,
+            0.009146793705085586,
+            -0.009891297866910313,
+            -0.01644269527910728,
+            -0.0003903768506971414,
+            0.006512217314869576,
+            0.02408785204739448,
+            0.0018406499545367782,
+            -0.007360473750031044,
+            0.01484497577088404,
+            0.016723277095327158,
+            0.003629911330658209,
+            -0.0004820567356447713,
+            0.02394406451074397,
+            0.023794019615473026,
+            0.001632266615911801,
+            -0.02064379152040184,
+            -0.02755693521684878,
+            0.027335938400284726,
+            0.02769248863677192,
+            -0.006832353240623308,
+            0.027324973360715134,
+            0.003937207909967385,
+            0.013580154030133622,
+            0.02076818768224654,
+            -0.01784151382870951,
+            -0.017617903734311997,
+            0.0017316476451235418,
+            -0.009605294810027804,
+            -0.009829196696575069,
+            0.004891780008779742,
+            -0.011655402383771472,
+            -0.01124379942906224,
+            0.043205454954628145,
+            0.004713701851783307,
+            0.003560551284131962,
+            0.007894840447501912,
+            -0.018881214526749082,
+            -0.00945538175144614,
+            0.0019040259741796817,
+            0.029505863720301916,
+            -0.001400002868254808,
+            0.03315866433970851,
+            -0.00830542045504805,
+            0.008602408964961391,
+            -0.03202191233287069,
+            -0.006978289755114341,
+            0.0326499698728755,
+            0.015314305362524091,
+            0.015104436300213667,
+            0.0021257042489266787,
+            -0.0010840650756005028,
+            0.02434348355847526,
+            -0.007174433778817674,
+            -0.04839373063710409,
+            0.022696994450018916,
+            -0.004779511451775743,
+            0.023588216956749154,
+            0.0382733980620746,
+            -0.004232286553944853,
+            0.003815507826545929,
+            0.023531223314846965,
+            -0.01887650634213847,
+            0.015055403379782985,
+            -0.01918944132635701,
+            -0.017597900721414636,
+            0.02176630010543207,
+            -0.008896608357798277,
+            0.01732387427588561,
+            0.01764363756727683,
+            -0.012180043444768312,
+            -0.019003360499321047,
+            0.01229139850966224,
+            0.0025710091163655626,
+            -0.017075658234265562,
+            0.01797412486853335,
+            0.018372010308966135,
+            0.006723179570342314,
+            0.015417677746651683,
+            -0.00047634684109332134,
+            0.017608296628666153,
+            -0.001049984114818522,
+            -0.008416990841821452,
+            0.01827827907328822,
+            -0.009568717277044265,
+            -0.01658089563620308,
+            0.009808478422454319,
+            -0.05193977747210182,
+            0.010644335931025957,
+            0.0016781339270411392,
+            -0.01562285100294917,
+            0.00405036330963539,
+            0.007102724536527262,
+            0.005195069570261247,
+            0.0028773630901681767,
+            0.004610474981005765,
+            0.042921486858787584,
+            0.013213672397112759,
+            -0.0064030584930089415,
+            -0.010711457508461704,
+            -0.017480309425763254,
+            -0.027151012983864797,
+            -0.005204425551873271,
+            0.009927884476991038,
+            0.022601230152836282,
+            0.008046297278661014,
+            -0.0031858480011267704,
+            -0.02589628103025927,
+            0.019723430185137954,
+            0.03604835658149313,
+            0.062423099201768716,
+            0.010930268386637612,
+            0.02925564269324381,
+            0.02439517070384333,
+            0.023930432096006114,
+            0.013036828091946605,
+            0.006910311087503408,
+            -0.032995841694293995,
+            0.012813650252889403,
+            0.01333714655191133,
+            0.03678966264802025,
+            0.012571138353467183,
+            -0.05610151838497368,
+            3.0777966587107344e-05,
+            0.009791774904454618,
+            0.041959488802722524,
+            0.01956296954085397,
+            0.000155217569370113,
+            -0.007285602454264675,
+            0.031547870838711856,
+            -0.026518385135879206,
+            -0.012299593280363843,
+            -0.0026551378470968594,
+            -0.006190117573235779,
+            -0.003807907001161504,
+            -0.004991957475076049,
+            0.036983035940964804,
+            -0.015648565314743963,
+            -0.019256463332683493,
+            0.0004300362961407231,
+            -0.0015686065121077472,
+            0.0011311524136286674,
+            0.0032236641835276464,
+            0.013795345278976164,
+            0.015385029265915279,
+            -0.004800448639353721,
+            0.026514926384796363,
+            -0.008699453553651681,
+            0.022186817580012775,
+            -0.013427553742493343,
+            -0.011927259975371873,
+            0.017776228476138103,
+            -0.021243797085501972,
+            -0.01667280589187595,
+            -0.003688045116580565,
+            0.006189970874536939,
+            0.0020911968266404467,
+            0.0006097761986763103,
+            -0.004685468039463817,
+            -0.01011357666327168,
+            0.022631082517271542,
+            0.012398000826637277,
+            -0.006721588925828951,
+            0.018142966317821054,
+            -0.0021330638043572654,
+            -0.011869184647489572,
+            0.025073455250141923,
+            -0.0015392945266926425,
+            -0.0337085890598252,
+            0.01267302113143573,
+            -0.013438601165482187,
+            0.02673290895281225,
+            -0.009586762706902736,
+            -0.011208527168019762,
+            0.019144928964011168,
+            -0.0017591316941248897,
+            -0.005459150067129732,
+            -0.029230079356228025,
+            0.004679497471366242,
+            -0.011723195942933286,
+            0.015899377439148873,
+            0.0022054234777254627,
+            0.029187759252864872,
+            -0.018290694895057157,
+            0.00792533824593637,
+            0.011719198071061566,
+            0.014612375997487872,
+            0.005419943379355563,
+            0.010515265723144554,
+            -0.006893314445458116,
+            -0.04424495426064596,
+            5.8241472764594054e-05,
+            -0.005719143565673274,
+            0.0038220021965914754,
+            0.013106940982933836,
+            -0.01203955335201393,
+            0.011260626096963215,
+            -0.0279896016489392,
+            -0.0295228261396555,
+            0.036474811902902415,
+            -0.04900747926861378,
+            0.019261089742811332,
+            0.00014104646541594166,
+            -0.03057197731717424,
+            -0.04331886560073797,
+            -0.03232957159604667,
+            -0.019401402365464362,
+            0.004102283356333861,
+            0.0002610070704924167,
+            0.0012280372705500212,
+            -0.014025012323929194,
+            -0.014135990809256075,
+            0.013269444858490211,
+            0.00992155580941576,
+            0.011225162609266308,
+            -0.006545044517282357,
+            0.002368862850663672,
+            -0.02063183105457681,
+            -0.026981383275694142,
+            -0.02774194980145225,
+            0.013337303400678109,
+            -0.0025902548982556788,
+            -0.02794339093668326,
+            -0.025606606546251152,
+            -0.0029852498102770587,
+            -0.01906873442473352,
+            0.006508845592362198,
+            0.03268861529625085,
+            0.0355975143313428,
+            0.010066906387952277,
+            0.01690250467284985,
+            -0.017201544758038183,
+            0.048345309114140714,
+            -0.019031171948182872,
+            0.03188329748476941,
+            -0.0005845169448621133,
+            -0.024140202991373587,
+            0.00011597263700677706,
+            0.03203623528903764,
+            -0.019285565857567896,
+            -0.01332755507465857,
+            -0.010914187315481855,
+            -0.003374912271519852,
+            0.02419326154642231,
+            0.02443825785057301,
+            -0.0037923303809811746,
+            0.00022800399604730175,
+            0.015893265651968885,
+            0.003154527829977343,
+            0.007688529734165207,
+            0.030623379985627763,
+            0.014052282574575454,
+            0.01127235345371133,
+            -0.04164707563955819,
+            -0.011811587719246128,
+            -0.003763086694358232,
+            0.005909410979841885,
+            0.002913220159138154,
+            -0.002671613084376487,
+            0.0045765194162907885,
+            -0.007241703279856166,
+            -0.005855949422953439,
+            0.0016973541436777935,
+            -0.001576326424468957,
+            0.008393873711385013,
+            -0.0037217094191171083,
+            -0.03498879678492834,
+            -0.026096985039773694,
+            -0.021280075788049788,
+            0.006437497348154639,
+            0.0070214556591898375,
+            -0.00972721128392027,
+            0.01949553762684561,
+            -0.0029404319702066146,
+            0.000271037702827666,
+            -0.010734867909488861,
+            -0.0029784341515646075,
+            -0.01206464664516104,
+            -0.019544778160755956,
+            0.00035139045159739705,
+            0.013457601448003152,
+            0.00025947376767512006,
+            -0.03165522602526639,
+            0.011979811705146604,
+            0.006642849985850598,
+            0.013181162886675883,
+            -0.024499404540711422,
+            -0.02310714308893576,
+            -0.028028912907092744,
+            0.010570122662701183,
+            -0.0066866321204157725,
+            -0.024617571284817428,
+            -0.004097807033448707,
+            -0.0074576460628928256,
+            -0.0020417256533259887,
+            -0.02470953739335365,
+            -0.011408461668188081,
+            0.023537349682260345,
+            -0.006373652338136973,
+            0.01936338676879376,
+            -0.017636435070085086,
+            -0.017036091876814843,
+            -0.002198685044404603,
+            0.015584758061011048,
+            -0.038943923931820196,
+            -0.002076529978164001,
+            -0.008452513321945455,
+            0.006551055965075869,
+            -0.028840587857625097,
+            -0.007128279504360441,
+            0.0199397499488538,
+            0.012424319366271584,
+            -0.03518875279748495,
+            0.013596027106451115,
+            0.022191403935861113,
+            0.01218507466930854,
+            -0.0038551718784437783,
+            0.03910075115429626,
+            -0.0025215641203030336,
+            -0.005422688956956598,
+            0.016759888275451905,
+            0.04462538468871483,
+            0.023708097134660096,
+            -0.03152743757744915,
+            -0.008668774531128668,
+            -0.015224825785549947,
+            0.004102648962663789,
+            -0.025867795193966987,
+            0.004676262922216584,
+            -0.011739753544166856,
+            0.01645951757031186,
+            0.011253400919230348,
+            0.013958707917471253,
+            0.0376596076542735,
+            -0.005077354977296842,
+            0.02668910096392324,
+            0.0010332284250006117,
+            -0.00856723931523911,
+            0.0014431148305642308,
+            -0.01412579094241589,
+            0.029892393553325205,
+            0.02269599036836208,
+            0.014421409701746036,
+            0.029248195108770955,
+            -0.03588867295960255,
+            0.02458655544632528,
+            -0.0010013539341895672,
+            0.008368648583431747,
+            0.018150734970019018,
+            0.02628022565029391,
+            0.015423867373844644,
+            0.012477080028047823,
+            -0.02513018750675568,
+            -0.022180001014682045,
+            -0.0027550350630288997,
+            -0.031944843677988374,
+            -0.0014668093006914267,
+            -0.009789309083948147,
+            0.026386027599666254,
+            0.0046988411391149,
+            -0.019803160322638583,
+            -0.003931806845108628,
+            0.004482097428470392,
+            -0.00244660100774939,
+            -0.010441732346892935,
+            -0.004649318716095894,
+            -0.01869464992812615,
+            -0.017838643679294407,
+            0.023677878122153204,
+            -0.023958455640817666,
+            -0.025238725541146534,
+            0.014582798102040157,
+            0.0013223590277629197,
+            -0.002295641913710489,
+            0.0008238518461567685,
+            -0.02115533275282518,
+            -0.04214275933987015,
+            -0.027817083427786927,
+            0.0037588803464689412,
+            0.03565056002940463,
+            -0.026788865866928923,
+            0.014162550819567465,
+            -0.011336583458519,
+            0.019175861037658344,
+            -0.005127229083828615,
+            0.015531116410874209,
+            -0.018251290727445846,
+            -0.044630374763679266,
+            0.021833307809708984,
+            -0.012087009688208446,
+            0.02720742725750304,
+            0.021497809277370574,
+            -0.026937653159100156,
+            -0.013572733244188407,
+            0.01358214612023832,
+            0.0020408253706382332,
+            -0.009652962552750464,
+            -0.0063411929391088626,
+            -0.010533309167065193,
+            0.003901271588567942,
+            -0.04496345296706673,
+            -0.008761188928422701,
+            0.06948704056954039,
+            0.005916180375965137,
+            0.03157905519638486,
+            0.014668818318138927,
+            -0.028448496686381258,
+            -0.04008949536631331,
+            -0.00462253589823598,
+            -0.0003701522833020314,
+            -0.00024410762276269235,
+            -0.01850585249309217,
+            0.017699615637510956,
+            -0.002712069912576265,
+            -0.009767338815933488,
+            -0.013119464222297351,
+            -0.013743610418034631,
+            -0.030375365173543157,
+            -0.0030316327877227783,
+            -0.008632689470358149,
+            -0.006565457264917239,
+            -0.0328089275944709,
+            -0.011713996290006943,
+            0.029773910045828737,
+            0.02740100904353622,
+            -0.0061926915497348915,
+            0.00638737826122608,
+            -0.010244154924811975,
+            0.0010396203272677069,
+            -0.001264454445070139,
+            -0.004059969507786729,
+            0.008715730726831364,
+            0.0014767015415289076,
+            0.031074767756160294,
+            0.004761858344125113,
+            0.00987870172044607,
+            -0.059926117793699066,
+            -0.009294021263133636,
+            0.021188432871397475,
+            0.016602826531924377,
+            -0.009125784136292983,
+            0.013946188378030194,
+            -0.0017670794602549428,
+            -0.006837334179368295,
+            -0.028290055353601393,
+            0.024556541126233806,
+            0.04543639597357669,
+            0.0031311343860416473,
+            -0.011199639773247286,
+            0.014965677855595444,
+            0.0018248533506196013,
+            0.015507721118783542,
+            -0.011409402378400194,
+            -0.01338737990904486,
+            0.03063410115541065,
+            0.009305362935335817,
+            0.0025578631174065385,
+            0.012727648343848194,
+            -0.0038680554900795763,
+            -0.01173215289303824,
+            -0.01568862495293563,
+            -0.004101703129244437,
+            -0.03649200408657064,
+            -0.0022107849490954137,
+            0.009659780679763728,
+            0.0034091693454195806,
+            0.008848380026939594,
+            0.01842435611499587,
+            0.03546926214224266,
+            0.013478278853234764,
+            0.005970293629431872,
+            -0.0069197533761813455,
+            0.023196445250565058,
+            0.010403914879205351,
+            0.010017272564548626,
+            0.015076807790462658,
+            -0.029649260409273275,
+            -0.012737152920662164,
+            0.00373276777994811,
+            0.013239909450830258,
+            0.0001682544254742693,
+            0.02030652051051447,
+            0.014143961923579586,
+            0.0010147733405793957,
+            -0.012509446388074923,
+            0.0020437769941487626,
+            0.007405578085797006,
+            0.013458276946120546,
+            0.0006779114540819997,
+            -0.004250745498730309,
+            0.0009149988500358458,
+            -0.008355906072311008,
+            0.040391939952386505,
+            -0.01831218885700067,
+            -0.016793844108223176,
+            0.035064882870542116,
+            0.11416006426189942,
+            0.0010630856112440689,
+            -0.015660014600661857,
+            -0.017142534874931403,
+            -0.0086899013413391,
+            0.016611825572132997,
+            0.01609650977226868,
+            0.009780106186629793,
+            0.009622367309076686,
+            -0.000672926017804395,
+            0.024310949465350565,
+            0.02944698490225121,
+            -0.0037169531700178868,
+            -0.010990838766452194,
+            -0.036819646475041955,
+            0.022923962979125287,
+            -0.004596366586768123,
+            -0.007108041562635549,
+            -0.002604794313552557,
+            0.012641223813290853,
+            0.013668775645501733,
+            0.005125100040230003,
+            0.0077993841846437815,
+            -0.011500851674334447,
+            -0.017334458072700107,
+            0.0056819751688881774,
+            -0.015982234644378358,
+            -0.0076533013358874935,
+            -0.013127172654362694,
+            -0.011837346554842119,
+            -0.006562312362371076,
+            0.0029877420008976425,
+            -0.008572723854050838,
+            0.00854083969164562,
+            -0.010939639783598493,
+            0.016185642759084087,
+            -0.008052282332943615,
+            -0.0011992212936609377,
+            0.0010378646897070758,
+            -0.0025399515501741718,
+            0.010712865639618352,
+            0.01451357949543628,
+            -0.02521401163091652,
+            0.020836357346214075,
+            -0.008252813367155418,
+            0.0006409807249997337,
+            0.015338809580802425,
+            -0.04672526649678653,
+            0.022728876784937357,
+            0.005129225753457052,
+            -0.018372222092794564,
+            0.0066182736895667805,
+            -0.02048375780196863,
+            -0.00896382274306482,
+            0.0021738974344880794,
+            0.004491167150658828,
+            0.0034280799511763956,
+            0.015611133773752265,
+            0.018126362497548308,
+            0.00014383671280220997,
+            0.007927708693242186,
+            -0.006973076292062532,
+            0.008118313055163714,
+            -0.006791836664375275,
+            -0.01164567445668625,
+            -0.010894577039363014,
+            0.00014731542306100506,
+            -0.008865132658571428,
+            -0.011415696755798914,
+            -0.017922318360382588,
+            0.00358538001727405,
+            0.00289327637128695,
+            0.010406054711003452,
+            -0.007757607207626101,
+            -0.010994775479337082,
+            -0.011716096880261664,
+            -0.0013333474316266438,
+            -0.01231144419953554,
+            0.009792603083441917,
+            -0.02884953407490196,
+            0.0029097409399144306,
+            0.025277274938032477,
+            0.009263530315933738,
+            -0.028037993274175277,
+            -0.02532252134134532,
+            -0.0018596440554143638,
+            -0.013847737486669237,
+            0.009910107581119272,
+            0.013952962280235523,
+            -0.011807495289410774,
+            0.003896325037406752,
+            -0.02144921729633576,
+            -0.01628545945817568,
+            -0.0040148671532777005,
+            -0.016032493124255323,
+            0.004580041957001551,
+            -0.011375180364988195,
+            -0.007789996450583554,
+            -0.022041665680945066,
+            -0.005890065942573431,
+            -0.050455058269193866,
+            0.014401045842891128,
+            0.030219548461217365,
+            -0.004580747668649223,
+            -0.028684680969792875,
+            -0.015525870112797619,
+            0.011121401958547706,
+            -0.008879749268609245,
+            -0.019454243014778435,
+            -0.007839434838464604,
+            0.018908534797107714,
+            -0.0019283067432864216,
+            0.0024769961730655835,
+            -0.006732653593696035,
+            0.027604805021030646,
+            0.01366593691107021,
+            0.03740746059041736,
+            -0.014662268336336578,
+            0.02539666271785517,
+            0.02371809783638648,
+            -0.036126848124425154,
+            -0.007627445211958312,
+            0.03657481892016626,
+            0.01092515982671624,
+            0.0011046573579274704,
+            0.012350047525582187,
+            -0.006340802546498992,
+            -0.031330519599469066,
+            -0.0013691360532301728,
+            -0.0014676630169621005,
+            -0.023111265345539388,
+            0.027359543575867225,
+            0.018091458372134273,
+            0.014237156751820934,
+            0.0030993618578507172,
+            -0.0014037082734049385,
+            0.013257269860435359,
+            0.01021315143964527,
+            0.030885417052806473,
+            -0.003225065956381245,
+            -0.012825341172770057,
+            0.013476039084668826,
+            0.01852164843904835,
+            -0.009965495459579544,
+            0.012541079912270015,
+            -0.00824122613985912,
+            -0.018124134280975132,
+            0.03192263766222747,
+            0.0001484308749572035,
+            0.010888733498667343,
+            -0.010649515858317311,
+            -0.013082335214246402,
+            0.032614733822945205,
+            -0.00999075761269065,
+            0.004257643407198955,
+            -0.030723533031261387,
+            -0.003820776505926347,
+            -0.010471792928758869,
+            0.01737608180148319,
+            0.004597677757231315,
+            0.02095148515295301,
+            0.029253627700524396,
+            -0.011604450820008121,
+            -0.01996325509217509,
+            -0.001158626158150122,
+            -0.020583793738536985,
+            0.004168757459099133,
+            -0.03632338333261702,
+            -0.019373660230711246,
+            0.022259216603413172,
+            -0.010364620831688015,
+            -0.004751193403783363,
+            0.00444187509644351,
+            0.0008572202661850934,
+            0.014943779567410408,
+            -0.025408263453313398,
+            0.017755758370543148,
+            0.004627392860417132,
+            0.007402073071525438,
+            -0.024913329140362466,
+            0.013760771254234878,
+            -0.006087346663657887,
+            -0.03780469033982951,
+            0.005509388120163784,
+            0.03756141956381493,
+            0.02381019168406692,
+            -0.024580249929652097,
+            0.01807584361080568,
+            -0.03111039245576191,
+            0.023245542501125073,
+            -0.004868090951368273,
+            5.1544460403943004e-05,
+            0.00640235757127173,
+            -0.004698373089469779,
+            -0.00621383035264185,
+            -0.0025132415458431587,
+            -0.008746634331769936,
+            -0.01611801559493664,
+            -0.01193535623883546,
+            -0.03381789115349148,
+            -0.024119754887378317,
+            0.013881102852385332,
+            0.0019232668966222687,
+            -0.0095263729740736,
+            0.018652966857927805,
+            -0.03775912497511837,
+            -0.012022352025516115,
+            -0.023289377270709736,
+            0.023579672990954228,
+            -0.028605627815260902,
+            0.019235397553197357,
+            -0.025838655372560505,
+            -0.021458526449310157,
+            0.01508157885177254,
+            -0.010064396230734452,
+            -0.0016708624560165368,
+            -0.015042071642751917,
+            -0.03670785416972153,
+            -0.006267614265977843,
+            0.02298763967146357,
+            -0.012506444728541897,
+            0.00029871598584066644,
+            0.02671390618094565,
+            -0.003059865833779805,
+            -0.011103452976820622,
+            -0.0017382684168257506,
+            0.010923055147701782,
+            -0.02938686260440486,
+            0.014300097108092281,
+            0.0038571851420598074,
+            -0.0037614499297117365,
+            0.0010991440540809893,
+            -0.010108413486243776,
+            0.02127952161226619,
+            0.009142545227414532,
+            -0.0030260119510809205,
+            0.040589877142650374,
+            0.008677453889680373,
+            0.0017339508605300814,
+            0.008732950895530873,
+            0.01063352447960556,
+            0.006618635029823979,
+            -0.014986307792626314,
+            0.016905861015681957,
+            0.013101237900847744,
+            0.0002162791516784468,
+            0.019309343995079022,
+            0.0021263376627204313,
+            -0.024516267132902416,
+            0.013339486637573757,
+            -0.02311659751478777,
+            0.0020575982953961246,
+            0.02391695586495566,
+            -0.020440172706421535,
+            -0.001656132876701879,
+            -0.006304133667169861,
+            0.000543457661904495,
+            0.022135707414160034,
+            -0.0230903871986617,
+            -0.023444763940153112,
+            0.009604600033024832,
+            0.01721387169484476,
+            0.01827905276638783,
+            0.02824773106648473,
+            -0.007100625424382998,
+            0.01348510178676837,
+            0.0002604386361122316,
+            -0.011802178189607182,
+            0.007664157334475143,
+            0.0074175377458313594,
+            -0.0044008460682833875,
+            0.025491035291032547,
+            -0.0017015687016376205,
+            -0.006134010425966364,
+            -0.005379657499404915,
+            -0.01584573519366957,
+            0.0034386456145185213,
+            -0.013860742817569676,
+            -0.013355238174583658,
+            0.0040342831827141845,
+            -0.0208160950322342,
+            0.10904592104153767,
+            -0.0037038089210379766,
+            -0.019472012964045247,
+            0.0063554319446707044,
+            0.003716719884684827,
+            -0.003692786470980806,
+            -0.003353843210051379,
+            0.007788605566526459,
+            -0.011254480693190834,
+            -0.0045751956417398095,
+            -0.013976039724381875,
+            -0.001989262807706565,
+            0.010067374586433014,
+            0.009201983771978006,
+            -0.005902980676079493,
+            -0.02029714318363332,
+            0.013031676901335606,
+            -0.016970144750162643,
+            0.008874670945038486,
+            0.0008311255074953139,
+            0.00884167159995975,
+            0.001716603195908536,
+            0.01854951250316076,
+            -0.009310881405486074,
+            -0.01325242738760595,
+            0.029667589863813217,
+            -0.005616511284128333,
+            -0.0038179509972925954,
+            -0.0004690716809300012,
+            -0.005068974756744239,
+            0.0021204231125051274,
+            -0.027958274497127258,
+            0.022127088236172387,
+            -0.017271838492631593,
+            0.010267709087809265,
+            0.01316800560257366,
+            0.005599367128954229,
+            0.01817806970002991,
+            -0.007115594036834107,
+            -0.02627643123342057,
+            0.006542742528003751,
+            -0.003433874053082844,
+            0.006631427284537971,
+            0.02316295746074422,
+            -0.014573351469966297,
+            -0.033520864104177005,
+            -0.013707022143307006,
+            -0.01877574599842729,
+            -0.005992004412448576,
+            -0.0043698881826095435,
+            -0.01481352026460129,
+            0.006424121402963946,
+            0.01382938882387991,
+            0.018136021761515388,
+            0.0012697483790911433,
+            -0.011531136974861559,
+            0.00609187274277241,
+            0.01076133059240578,
+            0.008400887242930113,
+            -0.01533504932731406,
+            0.011439227832642668,
+            -0.028718487721012004,
+            0.01522697877548034,
+            -0.012452665599305595,
+            -0.024938748080573947,
+            -0.0022082450217489366,
+            0.017589399637694264,
+            0.0037824832255756763,
+            -0.004355445812154662,
+            -0.018778668474337727,
+            0.019825087149714933,
+            0.024692878851503508,
+            0.003883497401229232,
+            0.019470881143244197,
+            0.041834635328088364,
+            -0.003544509491805351,
+            0.027139164223766474,
+            -0.02129723143323512,
+            0.01461354860170749,
+            -0.03441729954693155,
+            0.001893713808284104,
+            -0.004222657393633789,
+            0.0013677766871517654,
+            -0.003811928806805752,
+            0.020142937759360904,
+            -0.009158202577174012,
+            0.0030638058587375543,
+            0.024854053046139163,
+            -0.018665378030958756,
+            0.00805903734779081,
+            0.010625429315145887,
+            -0.012856237251902233,
+            0.028535299711292306,
+            -0.006335094045840286,
+            0.00305422126851784,
+            -0.0297883488914759,
+            -0.0029192872993071216,
+            0.0026067044379754047,
+            -0.006097749614529602,
+            -0.00968023644800927,
+            -0.007433916539502931,
+            -0.006202274026852405,
+            0.0038354607448269555,
+            -0.021471361566683354,
+            0.003195646827154078,
+            -0.004390751282171024,
+            -0.003323447423321121,
+            0.02518759723554948,
+            0.025857746688368054,
+            0.016893707217894103,
+            0.012739170140289531,
+            0.008553625305650129,
+            0.019805351588748434,
+            0.01654964927067909,
+            0.011549329129728739,
+            -0.020026013396046632,
+            0.01150644479272541,
+            -0.004159611921476011,
+            -0.0022937297602724912,
+            0.014815466072709131,
+            -0.01572171840394285,
+            -0.02276187015864222,
+            0.024693009795978448,
+            -0.019725315346836404,
+            0.00998768979020729,
+            -1.1702210281215188e-06,
+            -0.011129745730507545,
+            -0.002591293237053776,
+            -0.018034357254990988,
+            0.00012070821244376275,
+            -0.023051045150566527,
+            -0.028465270110388716,
+            -0.00048603008831669926,
+            -0.000574546300981984,
+            -0.012992623897880395,
+            0.02064747165046032,
+            0.014820638761258362,
+            -0.024820174379739057,
+            0.02666518374464728,
+            -0.017430076811109155,
+            -0.01495051200726725,
+            -0.005861181182847845,
+            0.00100881727574559,
+            -0.0055223282819321295,
+            0.0034719859452138017,
+            -0.010180752654175722,
+            -0.009961678082468036,
+            -0.012134423047419183,
+            -0.028073672819317354,
+            0.011574683159318163,
+            0.016120674453124317,
+            -0.015043147648505721,
+            0.031834693135959095,
+            -0.008802805725050891,
+            -0.0018698230152007877,
+            0.016179742266521146,
+            0.005096232332051617,
+            -0.017187077376166815,
+            -0.018999569809271876,
+            0.012251308792871571,
+            0.013279090498325945,
+            -0.009588297039279112,
+            0.006630431200804809,
+            -0.014130176783207386,
+            -0.005092662672039885,
+            0.007020236623084732,
+            -0.005049363863784624,
+            0.0055051712211635915,
+            -0.00995314895304919,
+            0.0052743456988306485,
+            0.005841370337169303,
+            -0.0029434616355351975,
+            0.004677049987338193,
+            -0.018568941264514256,
+            0.020031253265629777,
+            0.0018725390936655962,
+            0.01332532735283085,
+            -0.03518855189467458,
+            -0.0056298723325832025,
+            -0.005369475318146738,
+            -0.004650364775945294,
+            -0.020833510091412353,
+            0.020445280915606132,
+            0.0159991245532085,
+            0.013202409908533162,
+            0.023094507005426774,
+            -0.014905666713082246,
+            0.01572052456975675,
+            -0.020945766915121455,
+            0.013114000985348841,
+            0.003349101658930482,
+            -0.019829605346106578,
+            -0.001423750749031516,
+            0.04557459007627153,
+            -0.005847231973329081,
+            0.011986110554869603,
+            0.0012709842140249691,
+            0.026758922819943853,
+            -0.0001635536762342011,
+            0.0065419519929442995,
+            0.005997692702992206,
+            -0.009935128108338126,
+            -0.0008057798606777003,
+            0.00865712443881613,
+            0.046254320605750854,
+            -0.01181132866321511,
+            0.008170688314239364,
+            0.013099051984057827,
+            0.0034971494510865624,
+            -0.0008386400616584293,
+            -0.015346506644390764,
+            -0.03067678541405639,
+            0.019305962963428017,
+            0.007736916644077731,
+            -0.012756525606221703,
+            0.008734673666457216,
+            0.015137771675476126,
+            0.012891824153867332,
+            0.003895886821685762,
+            0.02155154331885366,
+            -0.012761690789748027,
+            0.018528313792219413,
+            0.024663453412919027,
+            0.009077230086690327,
+            9.184526542175535e-05,
+            -0.000751529849753638,
+            0.006791927267880182,
+            -0.0004433479107716552,
+            -0.009832230053513483,
+            0.015858232096630655,
+            -0.014861341148074698,
+            0.017790924034108806,
+            2.7216806238946785e-05,
+            0.03744841066552559,
+            -0.015216207585985075,
+            -0.010138319613387346,
+            -0.007774325714088762,
+            -0.0016997034352120362,
+            0.003971564314593291,
+            0.0018686537276849653,
+            0.006051669054361943,
+            -0.0008319624938748146,
+            -0.017326561560776794,
+            0.01667737942301944,
+            0.005719808546053257,
+            -0.0007405637872110778,
+            0.013750752065593804,
+            0.0038732323799683453,
+            -0.00367645612814479,
+            0.023580906347714314,
+            0.006238541579764762,
+            -0.005235795826199542,
+            0.0028182323816636243,
+            -0.006443210286549713,
+            -0.016255209299456314,
+            -0.003061441395209219,
+            0.004597970789372988,
+            -0.0016590920570710628,
+            -0.011138261849069419,
+            -0.0019423173609729848,
+            0.020355534503272887,
+            -0.004612410336124759,
+            0.013346750536413418,
+            -0.005400060374036387,
+            0.006252154576727367,
+            0.010601459925435238,
+            -0.012346569079682625,
+            0.0014154620309235822,
+            0.010406076751218612,
+            0.0013983417564564663,
+            -0.007140180075225315,
+            0.0010544410919327675,
+            0.007757599672903709,
+            0.006505350216372801,
+            0.00483063937962671,
+            0.00836630436955631,
+            0.013284451514874498,
+            0.0033749897293287438,
+            -0.006243293993633622,
+            0.0013949831763016042,
+            -0.013269921385800992,
+            -0.006308639321247727,
+            0.0010906169976769569,
+            -0.0010152835709869353,
+            -0.02017142197515583,
+            0.010787630901414118,
+            0.005355784839945777,
+            0.007838485470991034,
+            -0.01706932891696025,
+            0.009576030158827565,
+            -0.0008144311254998525,
+            0.02829832828245172,
+            0.007407953684956539,
+            -0.02769403281764041,
+            0.014266001222243134,
+            -0.010492199740754951,
+            0.022575782293320284,
+            -0.0016349316593207844,
+            -0.005656505606143177,
+            0.008025514398322551,
+            -0.009634281579228592,
+            -0.005991026158213869,
+            -0.022366083313133788,
+            -0.025825209755160946,
+            0.016871087849192924,
+            0.013583173973358155,
+            0.021125252303971605,
+            0.00012555627355712715,
+            0.015347535950809997,
+            0.0020128011217859444,
+            0.015164099887550086,
+            0.0024544903332736167,
+            0.012507136277293888,
+            -0.011798050869794805,
+            0.0104828761281283,
+            -0.0137805761114781,
+            0.014620357150314097,
+            -0.005246639228547017,
+            -0.006986408834884282,
+            -0.008849580196616098,
+            -0.005442165099441538,
+            -0.020891551850452786,
+            -0.009155930286949747,
+            -0.0013590151948670868,
+            0.010665872513236615,
+            -0.0005386423407222223,
+            -0.0027671778465775897,
+            -0.03237311596935737,
+            -0.004875062533405715,
+            -0.006670982068843501,
+            -0.01942244110096111,
+            -0.0045975373348006885,
+            -0.026137783023267008,
+            -0.01657173843643047,
+            -0.0007784046498232684,
+            -0.015037306676062236,
+            -0.006889680260149007,
+            -0.02009705080543122,
+            -0.027192867101205106,
+            3.0161964469866572e-05,
+            -0.009647644858034833,
+            -0.005380165957043753,
+            0.013815092649296373,
+            0.011902366689169847,
+            -0.01572592170758115,
+            -0.022321702398924333,
+            -0.01088937403483412,
+            -0.0165754067023749,
+            0.01646435743868707,
+            0.009914938007210463,
+            0.018834863207293908,
+            -0.003724245912768107,
+            0.005643505694722323,
+            0.0009581443359000331,
+            0.02327704258596552,
+            -0.02532422120223976,
+            -0.028694220894404807,
+            0.0019298778426469338,
+            0.0025970175688368323,
+            0.006086164866310174,
+            -0.009414665394188936,
+            0.001200390869257535,
+            -0.007782899559317819,
+            -0.0228264209954853,
+            -0.00898519580978142,
+            0.008156835270668449,
+            -0.010036630164522355,
+            0.02230664068547531,
+            0.012332230358869292,
+            0.013699197379189934,
+            0.010593587938184292,
+            0.01576174456497367,
+            -0.0001654241831584687,
+            0.014471905257729428,
+            -0.008675915624727887,
+            -0.0031070346099204425,
+            0.0038358404385932777,
+            0.005878417037187274,
+            -0.00625022998235275,
+            0.028077114529659795,
+            -0.0011836453716817525,
+            0.0002962695480296182,
+            0.014405782572641941,
+            0.02697667559474498,
+            -0.004710313427099278,
+            0.0210512043261309,
+            -0.002393165025068831,
+            -0.006135816286212752,
+            -0.008429776018235104,
+            0.008607617299702318,
+            -0.009529291136733401,
+            0.011133497451942684,
+            0.019864404836361926,
+            -0.014041315457579166,
+            -0.007351413712676263,
+            0.02950254974910368,
+            -0.014564388091367897,
+            -0.01825317106331622,
+            0.02325964952838705,
+            -0.007414295676409368,
+            -0.0039042587057605484,
+            0.00377144546378926,
+            0.007169856503233239,
+            -0.004036744966949995,
+            -0.012928341573802416,
+            0.016090527447479415,
+            -0.00011159893712484579,
+            0.014146137810890391,
+            -0.0004655839804281017,
+            -0.006372231942983921,
+            -0.010047382825720416,
+            0.002373224247019062,
+            0.026818813126567614,
+            0.013483627627077692,
+            -0.017895710143841333,
+            -0.013785856764479817,
+            0.010826610341182479,
+            -0.01538893818197323,
+            -0.004425206001945227,
+            0.02155595194812746,
+            0.0028768153668128553,
+            -0.020543027467698483,
+            0.008028975884363618,
+            0.0003835165718138906,
+            0.00037046947047892953,
+            -0.00029363296913641716,
+            0.030135739208689217,
+            0.001343516577762509,
+            0.00018948658255495327,
+            -0.00875053830853609,
+            0.008606573235959595,
+            -0.01280305089741873,
+            0.012786478836532904,
+            -0.02437993139337839,
+            0.007648181775166282,
+            -0.00629541989541625,
+            0.004579605315599212,
+            0.016658386927051592,
+            0.01218098627022323,
+            0.015031618813778413,
+            0.008237200519537417,
+            -0.008942965899756683,
+            0.0009093265012901892,
+            -0.025302175262520477,
+            -0.001333639155622843,
+            -0.017971876771137367,
+            -0.02036433247634014,
+            0.0002463842837918767,
+            0.022144712689630985,
+            0.018559482073124734,
+            0.011579840028464503,
+            -0.008657014440886656,
+            0.0166696911919226,
+            -2.9834620942882283e-05,
+            0.011845827764518911,
+            -0.00037901176785495324,
+            -0.006138374154553732,
+            0.004711058953619512,
+            -0.003622694588720417,
+            0.013542405259906128,
+            -0.014730567462694827,
+            0.012076044672004906,
+            0.01690223095263866,
+            -0.009108632851807137,
+            -0.008946251274440141,
+            -0.013992423874475428,
+            0.00048381889145447367,
+            -0.015909599902931202,
+            0.035489252674454214,
+            0.02252772349379229,
+            -0.01210464970923921,
+            -0.010469120693943236,
+            -0.014164999313775756,
+            0.019496048806752753,
+            0.010427539351244087,
+            0.0028018908451425276,
+            0.017091768406027965,
+            0.021395638416767112,
+            0.008817247593404623,
+            -0.006081063711583319,
+            -0.0021216503501144983,
+            -0.003781810594530787,
+            0.020143098218482827,
+            -0.003078172339607037,
+            -0.011350171884507555,
+            -0.012516379645909518,
+            0.005766102930704529,
+            -0.01626433668691071,
+            0.0019294643928566657,
+            0.008641558521764407,
+            -0.015057688533773374,
+            0.019407986696657772,
+            -0.028126197703647868,
+            0.012364374199657305,
+            -0.0044278434183731225,
+            -0.007251147925369614,
+            -0.03496220945653279,
+            0.006389489026974697,
+            0.008941259259632024,
+            0.003941752277454314,
+            0.023003838655784314,
+            -0.024528514847470928,
+            -0.018361931455105486,
+            0.0037791061321962543,
+            0.00047162060848090104,
+            -0.00711894577052754,
+            0.0010138871688747137,
+            0.004128392769213672,
+            -0.00674546985015164,
+            0.0017307097443508704,
+            0.0033354800298095546,
+            -0.002829898077850263,
+            0.013252947102333348,
+            0.0081724269619999,
+            -0.021449979263734438,
+            0.001368791856860285,
+            0.01590656734732547,
+            0.01452008402429858,
+            -0.02391847163018108,
+            0.006331397860689212,
+            -0.02064812934794864,
+            -0.000631340242718475,
+            -0.031272866000888556,
+            0.016859793097564958,
+            -0.0069008624795683276,
+            0.01899718801899956,
+            -0.0015377436942771224,
+            -0.0065810685260109004,
+            -0.012822620655323715,
+            0.02137604937338179,
+            -0.0019121137078344135,
+            -0.0025598772127157477,
+            0.015678293057066646,
+            -0.025700865215693987,
+            0.017825186616033165,
+            -0.0026230848396158204,
+            -0.017342967698019966,
+            0.017199139867162275,
+            0.004536019178633225,
+            0.007224666811309087,
+            -0.023505589665315567,
+            0.0007208041198255396,
+            0.003900164485347562,
+            -0.017250324025622685,
+            0.004470862192961212,
+            -0.030807400122398566,
+            0.0006579147990327597,
+            -0.013222907940559794,
+            -0.0020514994691204847,
+            -0.0046763235724575445,
+            -0.014050035576183385,
+            -0.039181147615786484,
+            -0.005429639279537522,
+            0.016065742005148733,
+            -0.0009029636981539927,
+            0.01800715198737081,
+            -0.002730931639143803,
+            0.002947680833843522,
+            0.004912692811020822,
+            0.00818378920043427,
+            0.006915895602313347,
+            0.02274116906914758,
+            -0.0057854922886970365,
+            0.00471999900678522,
+            0.007058014277137233,
+            0.014638081356210031,
+            -0.01719565951201012,
+            -0.015170377598372723,
+            -0.016015921063369496,
+            -0.006351213642269936,
+            0.01739665344303718,
+            -0.017488885259028475,
+            0.013303332445465686,
+            -0.00429428898520701,
+            0.014825338166443021,
+            -0.008686427208690176,
+            -0.01575945086724703,
+            0.0028564767084934386,
+            -0.007570417028925067,
+            0.001007670647768344,
+            -0.02323908008484089,
+            0.02019729564706863,
+            -0.00847581038203716,
+            0.010508144473197669,
+            0.01729824052022313,
+            -0.021364529179605813,
+            -0.009742851959432839,
+            -0.014342281441152413,
+            0.029896210049029176,
+            0.013925163184699139,
+            -0.009392252140772913,
+            0.003927230832659267,
+            0.008873561269230905,
+            0.011529536139797032,
+            -0.012209070214071445,
+            -0.0027679974125029362,
+            0.002411914349172495,
+            -0.0062056207925691356,
+            -0.002025687569076897,
+            0.03090134105002816,
+            0.0038285457265343245,
+            0.024877474647546086,
+            -0.013214997316123852,
+            -0.0033763842539861907,
+            0.00023379106197363714,
+            0.005134764372494098,
+            0.01183867975321486,
+            -0.0017966997954747142,
+            -0.008029791503028367,
+            -0.01179500476055557,
+            0.02599911330072515,
+            -0.021852083993691557,
+            0.007024505421035273,
+            0.026784024996764876,
+            0.001390202993990949,
+            0.0029956956045771046,
+            -0.013131691460546865,
+            -0.006298736864509989,
+            0.02523640664202355,
+            0.010030824059858138,
+            0.0041523782603032225,
+            -0.026465819482580182,
+            0.0032128899272377593,
+            -0.0061901409332948115,
+            -0.008681758126710213,
+            0.009866461769956222,
+            -0.0028095454857224164,
+            0.01188012041754054,
+            -0.012191996383095982,
+            0.028459939543805705,
+            0.010711502132045176,
+            0.01047154118931583,
+            0.025888652452698974,
+            -0.02079614305315134,
+            0.0042203106489994615,
+            -0.010829796140534066,
+            0.0006773493086316061,
+            0.005596640511307504,
+            -0.027510562413905437,
+            0.001969898603871003,
+            0.010390866563404346,
+            -0.005642706366801436,
+            -0.01882994179544761,
+            0.01187970346127799,
+            -0.019814385265412387,
+            0.024479579275617887,
+            0.003231341626501863,
+            0.020056914356101588,
+            -0.0053276448188067154,
+            0.009834685132275868,
+            0.0002742942778577165,
+            0.0031361331890882127,
+            -0.0008074598092128465,
+            0.009009012303347016,
+            0.005786777410144894,
+            0.013719989127068968,
+            -0.025528370602188686,
+            0.006821572597883499,
+            -0.0029197127345352626,
+            -0.030539872624857384,
+            -0.013383097680839732,
+            0.010291485622655026,
+            0.012218533188117704,
+            -0.009144425807890594,
+            -0.010378679518611277,
+            0.0030042579827305603,
+            0.005748243116238633,
+            -0.006692403885450407,
+            0.006913672151000981,
+            0.01629516912493545,
+            -0.002533517074771728,
+            -0.0019083178803585312,
+            -0.0035007340321378893,
+            -0.01154609935927537,
+            -0.010016998439072824,
+            0.023193275321990937,
+            -0.00482990342915347,
+            0.017943025772239603,
+            0.025873224580389702,
+            0.0039876567755322,
+            -0.0057577932736143506,
+            -0.005460363755206188,
+            0.01478435999841998,
+            0.014195457012279476,
+            0.004669295894868759,
+            -0.009702220679239768,
+            0.007974305230005956,
+            -0.007760086293456636,
+            0.008375077126777163,
+            -0.020260917110786147,
+            0.002199401871514877,
+            0.0026191373298767766,
+            -0.0022089262577237776,
+            -0.015213807046175918,
+            -0.01956788452420682,
+            -0.021898368819136302,
+            0.012634659463647929,
+            0.00034643102239488474,
+            -0.009241772775582378,
+            -0.013645250997151429,
+            0.0016665106990117016,
+            0.0025630500346914592,
+            0.0002095521560180936,
+            -0.007783022628694597,
+            0.009546147813008391,
+            -0.01265287317178631,
+            0.007445223311736028,
+            -0.004007029181436962,
+            0.0051846833906574515,
+            -0.0016916014415925766,
+            -0.016413938859402438,
+            -0.0210868129634149,
+            0.01920794362224444,
+            -0.012897042227841011,
+            0.02269303737601583,
+            -0.0014935878147854765,
+            3.062680855559304e-05,
+            0.0033132626755401546,
+            0.007274487651476732,
+            0.012110042834251237,
+            0.011426424896797977,
+            0.004307086698938411,
+            0.008206932254416481,
+            0.004673427176334148,
+            -0.013124571552764586,
+            0.004324963759928715,
+            0.0039216033705804084,
+            -0.010449887922072297,
+            0.005298323150690876,
+            -0.0010196552573491,
+            0.006128327172621373,
+            0.012167492341907742,
+            0.01923525204293908,
+            0.01908107550907106,
+            -0.0012031208519043073,
+            -0.0010601912677170903,
+            0.014474374858125607,
+            0.004540954805526089,
+            -0.004883537463424679,
+            -0.002772988629590886,
+            -0.013363713078069854,
+            -0.002572403646338979,
+            0.011541515690646252,
+            -0.02376161747414617,
+            -0.018215231053002656,
+            0.021831219299143226,
+            -0.004991882316041097,
+            0.033796174834149546,
+            -0.013140970283091536,
+            -0.016197177724633614,
+            -0.00464190076219332,
+            0.0026389653040644607,
+            -0.013788049032378498,
+            -0.010300859744205445,
+            0.010545541278382187,
+            0.013848738866656793,
+            0.009903590151120531,
+            -0.0005984085581408575,
+            0.02922816576443645,
+            0.0019900932217491907,
+            0.02245395592422119,
+            0.0018978407151245295,
+            -0.007029294098689854,
+            -0.01440575120349887,
+            0.010840472832356266,
+            0.005529335652955532,
+            0.009050748012161991,
+            0.017791269808484802,
+            0.014502057454389247,
+            0.006265567405788131,
+            0.013928705812292101,
+            0.0012903002372833693,
+            0.0010858377120770652,
+            -0.002355229463060164,
+            -0.01692711767136341,
+            -0.01456786846097011,
+            -0.01627640642218236,
+            0.003873864312554098,
+            0.012620350492241247,
+            0.00525890570023481,
+            0.0007393789113866752,
+            -0.007153210751899716,
+            0.009254566395058412,
+            0.0065087881512835375,
+            -0.012279152412209152,
+            -0.016530068283509593,
+            0.004060671955726967,
+            -0.02210570272789274,
+            0.007849467899009437,
+            0.01554233481902349,
+            0.00219885032769439,
+            -0.002655449524219968,
+            0.0006843524304842362,
+            0.006272944169764281,
+            0.005531927368102191,
+            -0.0036710328198476905,
+            -0.00544113514775027,
+            0.011264661763762068,
+            -0.006379654234440906,
+            0.009537150372938296,
+            -0.00290031548037575,
+            0.03718190949929003,
+            -0.011976080887998194,
+            0.012120408392610773,
+            -0.001751472085435711,
+            0.008481703962951994,
+            0.008067669857949423,
+            0.010772687289903202,
+            -0.003805089040741351,
+            -0.0022164543131735167,
+            -0.008753625032781523,
+            0.006751953442322185,
+            -0.01582177621255226,
+            0.013631301238130705,
+            -0.00030685726238985963,
+            -0.010728958775974582,
+            0.01055472059121978,
+            0.006788316621149835,
+            0.004413382316882087,
+            -0.003262024961860459,
+            -0.01972283203505511,
+            0.014937729522976615,
+            0.0018799533869990667,
+            0.019247008681353865,
+            -0.023276106189004046,
+            0.029043727846427652,
+            -0.02019024213343764,
+            0.023083643395685142,
+            0.01865875504942157,
+            0.001179436167200093,
+            -0.02070329911663634,
+            -0.007618131527214554,
+            0.014871709122115308,
+            -0.02659818889735726,
+            -0.006234719259825804,
+            0.005986829551040766,
+            -0.014639912579831785,
+            0.00037813423546252645,
+            -0.01924149909676064,
+            0.00010832498610532006,
+            0.011013130149723431,
+            -0.011782407678373089,
+            0.00438725832178584,
+            0.01608349750921335,
+            0.019347883652009713,
+            -0.009760017612545174,
+            0.003869246694835168,
+            -0.024240324890794748,
+            0.010958789926998691,
+            0.020511684636649225,
+            0.008883830392748497,
+            -0.01761944522796127,
+            0.021817973960311446,
+            0.010652131228027521,
+            -0.012575679152857703,
+            -0.0012753095453828826,
+            -0.028191274775392804,
+            -0.012421170815054556,
+            -0.02188248722244861,
+            -0.0011670175621882146,
+            -0.014436683528976771,
+            -0.0029783880907228383,
+            0.0009506986717516226,
+            -0.004719165363484719,
+            -0.01179163390107033,
+            -0.019543732196676634,
+            0.016606805875498898,
+            0.0227313413004497,
+            -0.006738735201626204,
+            -0.014547240981002825,
+            0.013866974327185123,
+            0.031694077725890536,
+            -0.016189209297529324,
+            -0.00542110554672736,
+            0.0028061798782555557,
+            -0.012964876141500577,
+            0.018946046750284537,
+            0.0039419760434645495,
+            0.000322115192590907,
+            -0.01991851369210666,
+            0.002489725351392778,
+            0.010835458511052207,
+            0.01684964045708685,
+            0.0005541314524621196,
+            0.010855852688653808,
+            -0.007549882748861744,
+            0.010096800951420057,
+            -0.011832260836468588,
+            -3.6179995603517093e-06,
+            -0.007800366796516945,
+            0.007001049318404058,
+            -0.004424776169203575,
+            0.034356203288485945,
+            -0.02817071276766299,
+            0.004989631403681216,
+            -0.014539644354369607,
+            -0.010888763939716889,
+            -0.0027322086509797843,
+            0.0004865534340981866,
+            -0.010775238981581461,
+            -0.005162220412866082,
+            -0.0011263271110752205,
+            0.00913081242511139,
+            0.005844327320514489,
+            -0.013886529519306775,
+            0.019171827605726503,
+            -0.030984919318656618,
+            0.004469458202753611,
+            -0.008553175891755315,
+            0.010511707988936336,
+            -0.0151185694875566,
+            0.013830711686954931,
+            0.01301959345025159,
+            0.008205984085944315,
+            0.007740367849216328,
+            -0.01532123052721834,
+            -0.01678314222392063,
+            0.001376468747213867,
+            0.0036953018347776244,
+            -0.00931855481896886,
+            -0.033391028922085425,
+            -0.007072913446166969,
+            0.005331350534425606,
+            0.011614453225043,
+            0.008156128603003278,
+            -0.02345206873048703,
+            0.005003870024809975,
+            -0.024178720646704597,
+            -0.012115112684614331,
+            -0.007750557665424973,
+            -0.011580152272614108,
+            -0.006117298954192534,
+            0.026449765548206453,
+            0.007989237291215073,
+            0.016691517078382376,
+            0.004954133565782356,
+            0.02742809127725913,
+            0.011670893621991404,
+            -0.011736840457474895,
+            -0.003349151747880166,
+            0.00035419034843196817,
+            0.005507651806218584,
+            -0.008706590052024251,
+            0.011793241980257321,
+            -0.0013295035978611142,
+            -0.0065061682758755625,
+            -0.004398354347597199,
+            0.009372816666628725,
+            0.014188676762501037,
+            -0.003007980559117429,
+            -0.008724255229333351,
+            0.015749776067989826,
+            -0.030899793617096834,
+            -0.015215033379100091,
+            0.0016152308157299517,
+            -0.007813175908301852,
+            0.017806770680528907,
+            0.01169056345888971,
+            -0.026835412610223673,
+            -0.004538235170332766,
+            0.010532316940606419,
+            0.004755930024691789,
+            0.0095408889025506,
+            0.00954782703036282,
+            0.012079064562163901,
+            -0.007829549014214643,
+            0.01046722658021748,
+            -0.011519369840999348,
+            -0.007092961453184232,
+            -0.007241799314569802,
+            -0.008077934089061052,
+            0.021467547268987214,
+            -0.025816030568238717,
+            -0.006558853543772633,
+            0.01134670818701549,
+            0.013027491377727365,
+            -0.002301991242932548,
+            -0.006299073927206214,
+            -0.017357989759336386,
+            0.008311464809363397,
+            -0.013922568511594478,
+            0.025633907424998367,
+            -0.004854655498817219,
+            -0.005336807899100369,
+            -0.005228056880188717,
+            0.01958682721272445,
+            -0.016774409737540904,
+            0.02072573220949266,
+            -0.014449250679704904,
+            -0.007857351393427215,
+            0.001673929749689194,
+            0.006890008607998747,
+            -0.002948835017234416,
+            0.03734128725793149,
+            -0.00556474133167756,
+            -0.0014544201137417158,
+            0.02209084868209573,
+            -0.01883610080898967,
+            0.017901421953179442,
+            0.01563604483998435,
+            0.012327159100148019,
+            0.0018161034827768648,
+            -0.010253034391420751,
+            0.019979728662282904,
+            0.020823925491027635,
+            0.0139490143219451,
+            0.009577521724445478,
+            -0.013478761809038993,
+            0.014934933784421556,
+            -0.0003725422088504494,
+            -0.0024362115214204105,
+            0.012785534912074068,
+            0.016441944143888537,
+            -0.02164522371903646,
+            -0.013152102018736346,
+            0.0063548333449661805,
+            -0.016547629104776748,
+            -0.024875443975589356,
+            -0.008991095813132973,
+            -0.021885127280977718,
+            -0.023336714382596175,
+            0.00688008599534903,
+            0.016482610587564443,
+            0.0035797836729533367,
+            0.013111012464332918,
+            0.006820471349350837,
+            0.018274678506489214,
+            -0.00980248993077149,
+            -0.01582838461385605,
+            -0.0009454586621769173,
+            -0.0004906493597104733,
+            0.009787688665732045,
+            -0.01835583040075359,
+            -0.013977939077760832,
+            0.004340902272034388,
+            -0.020118484730757877,
+            0.01421736173162507,
+            -0.006428386730058143,
+            0.016756376129345597,
+            -0.0004630315831872392,
+            0.011701948921076007,
+            0.02621773278421472,
+            -0.001391567406453007,
+            0.01262026995055899,
+            0.004696906079976175,
+            -0.0030577364576645635,
+            -0.010624066224426196,
+            0.0009420498021845824,
+            -0.0044079288776429516,
+            -0.0004845606666642234,
+            0.006481113327122954,
+            0.020269778435314,
+            -0.03321232649491074,
+            -0.009425498978427855,
+            0.01246754422779043,
+            -0.001658574527323383,
+            -0.005816985985181387,
+            0.013050062223188965,
+            0.00011195028631856881,
+            0.016573827662919836,
+            -0.0017453203044868104,
+            0.01417371165305189,
+            -0.008165910519539772,
+            -0.0007461950113024975,
+            0.0016492309267666983,
+            0.005076015595439873,
+            0.005829299318055953,
+            0.012146800191035666,
+            0.01148884639506138,
+            -0.021051327863689618,
+            -0.0026357096198729156,
+            0.0036146148489310443,
+            -0.010673282438301611,
+            0.014967444393862347,
+            0.005737933867764028,
+            0.01391411712868596,
+            0.0029556189893790237,
+            0.01167806258173173,
+            -0.024440974650111174,
+            -0.01029944007931845,
+            0.01385509328958837,
+            0.00869671728965227,
+            -0.00998315356917989,
+            -0.016580053751364735,
+            -0.007061005072623325,
+            -0.00477175200214453,
+            -0.0021182980051220428,
+            -0.009871524928312118,
+            -0.0005933941177237234,
+            0.006284920058476413,
+            5.466820734762947e-05,
+            -0.012842295882002122,
+            0.015810530885294202,
+            0.004572385791133295,
+            0.003685966629168175,
+            0.005093714133656017,
+            0.011877541830396259,
+            -0.013130170365831912,
+            0.01258190257102377,
+            0.011433121961378686,
+            0.01142665981860032,
+            -0.013710025169815697,
+            -0.001774030746825879,
+            -0.01109881340948764,
+            0.0065684584345163735,
+            0.009983753894796107,
+            0.005346443477816739,
+            -0.002062387245572686,
+            0.004950044847352647,
+            0.006105199826576752,
+            -0.06391185159544624,
+            -0.02131511418723256,
+            0.0027165737784844413,
+            0.033389688791557785,
+            -0.007167593552251769,
+            -0.0037646833460996797,
+            0.002883510089742145,
+            0.01356887065340832,
+            0.009730171586767032,
+            0.00418566245984083,
+            0.020320934773473808,
+            -0.008814826860651366,
+            -0.003014193747732543,
+            0.001559573791453607,
+            0.003499420226345028,
+            -0.008703837291821111,
+            0.0044844701302276645,
+            0.0053963630238108035,
+            0.027764369548110287,
+            0.001469968540767627,
+            0.015793086297019008,
+            -0.008505290826588508,
+            -0.004075888558568453,
+            -0.009166710762668385,
+            -0.002860047211143503,
+            -0.011127820563172684,
+            0.003937110717404271,
+            0.02054186721680431,
+            -0.006126180963243227,
+            0.009346433819831456,
+            0.01057504086030076,
+            0.0001650851447316893,
+            -0.012033423322524951,
+            0.0049587335923558,
+            -0.0032240390628760535,
+            -0.005568976687946761,
+            -0.0024667040429312737,
+            -0.03474027725033016,
+            0.026855947684394643,
+            -0.004369626674503285,
+            0.009252512430957932,
+            -0.005081834388513357,
+            -0.0037896610731020337,
+            -0.014201609270756739,
+            0.017871088029615106,
+            -0.008249444723433136,
+            -0.020092257756971717,
+            0.009532994544485572,
+            -0.004991989149115383,
+            0.006464417738670472,
+            -0.02643078108339741,
+            0.0029138922065812495,
+            -0.017232410749655366,
+            -0.003999293090834831,
+            -0.006924488971625995,
+            0.01318082244178942,
+            -0.007232695119856389,
+            -0.02660711037908311,
+            0.01979251106803667,
+            -0.019485860712345603,
+            -0.011977904020170568,
+            -0.0177530973032872,
+            -0.014524003566945937,
+            0.004874952064727108,
+            -0.015326044325348929,
+            0.002047771741041311,
+            -0.009697340332342246,
+            -0.003652304998155628,
+            -0.0008878698629674222,
+            0.0017782435347417623,
+            -0.013017776298713044,
+            -0.018272697230002275,
+            -0.01353020425912225,
+            -0.00757239449706798,
+            -0.0008250000619957538,
+            0.01299142281772714,
+            -0.001679606555140765,
+            -0.007329941421303076,
+            0.0044605462460153116,
+            0.010231192277666706,
+            0.0006558644644452857,
+            0.0024231733856020414,
+            -0.010728844680699885,
+            -0.014137434553169443,
+            0.005230053181525096,
+            -0.012086468475542818,
+            -0.009459717916037003,
+            -0.0028185227600210556,
+            0.014984527039312847,
+            0.008189464334213754,
+            -0.009139240774931914,
+            -0.005639157579163322,
+            0.014325949891859377,
+            0.009020303519690971,
+            -0.013310205617445609,
+            0.01567574314514385,
+            -0.025369802629268523,
+            0.021964786021408878,
+            0.030536905527039065,
+            0.004945288795196981,
+            -0.01168303908203322,
+            0.02281128628230144,
+            -0.004072286832145419,
+            0.0028278384348551146,
+            0.01740522162288968,
+            -0.042558465579514926,
+            -0.013146562547084278,
+            0.002903743316339479,
+            -0.015297177099602701,
+            -0.010963312697387696,
+            0.008804938803398296,
+            -0.0026727734429658817,
+            0.005273238046953793,
+            0.004257613402172282,
+            0.03512246556126453,
+            0.002258612643297096,
+            -0.0002885708728599208,
+            -0.0019720190285930487,
+            0.012312613129996533,
+            0.010451003939611476,
+            -0.015556620910136185,
+            -0.02387856865043814,
+            -0.004583090523619507,
+            -0.0037828776396525835,
+            -0.0014588238692793338,
+            -0.004361834167147271,
+            -0.008159679676329573,
+            0.012243319920992457,
+            -0.017279964797769668,
+            -0.013748032758693542,
+            0.018394195628121487,
+            -0.006411113522842359,
+            0.019002611495040334,
+            -0.014970362556522149,
+            -0.0077437581301661946,
+            0.0017311115715410025,
+            0.024439691379769795,
+            -0.025013956798520212,
+            0.007359134924497539,
+            -0.0031406470488477937,
+            0.017574824561755627,
+            0.00419409250132525,
+            -0.0046803792187546656,
+            0.010556972371044426,
+            0.015229789583480183,
+            -0.004608293492221483,
+            -0.014449487985966966,
+            -0.01172850291078532,
+            0.016823024209386785,
+            -0.012304887597699588,
+            0.0011773351576246157,
+            0.002262670226244277,
+            -0.0023810578015456716,
+            0.03477119000959688,
+            -0.03442178291363675,
+            0.004062693040714608,
+            0.0029249583507755944,
+            0.003291349443702718,
+            0.006461525089528308,
+            0.006047777261425826,
+            -0.020642802018978404,
+            0.008916088381876748,
+            -0.03316762661930299,
+            -0.006367254689957291,
+            -0.005404276207844267,
+            0.0009189006423526284,
+            0.009401330264376041,
+            -0.011106443887384533,
+            -0.02259405372588916,
+            -0.006010439054323966,
+            0.0070744551115721095,
+            0.010262607009446879,
+            -0.002856616377476421,
+            0.019719451109535182,
+            0.008426509638139982,
+            0.0015877648339862691,
+            -0.010655792508751512,
+            -0.015527851374834499,
+            -0.015460974859638219,
+            -0.017847462632188386,
+            -0.02236964311096428,
+            0.011841749854626384,
+            -0.00576688153869993,
+            0.009054902130717224,
+            0.014317707346504314,
+            -0.015366657284925976,
+            0.002910211008094564,
+            0.017761685988205086,
+            0.007974806171830457,
+            0.002692237107055498,
+            0.0032405477366987947,
+            0.02685230091638845,
+            -0.0009116884421301454,
+            -0.0032228977180251673,
+            0.003158213042682977,
+            0.015872199488167846,
+            0.0004966905858799325,
+            0.019981401695749582,
+            0.015201538351780685,
+            0.005359075814444414,
+            -0.004668738193285921,
+            0.019853988551405383,
+            -0.009844394286458195,
+            0.013476144827747423,
+            -0.00411789677405284,
+            -0.00830656434595337,
+            -0.008496132520586954,
+            0.006442581751999304,
+            -0.008138786054348571,
+            0.006977337946948795,
+            0.02419128791801387,
+            -0.0005734827808485311,
+            -0.0028842142002271044,
+            0.005503033191779872,
+            -0.01696639779775455,
+            -0.00389835761601045,
+            -0.03366077608692946,
+            -0.009942613075976742,
+            -0.01708630689497001,
+            -0.02910756562213045,
+            0.026536268897199546,
+            0.00219827495662445,
+            0.02586503862171508,
+            -0.01905122645609761,
+            0.008935637454859214,
+            -0.01385806889332683,
+            0.029747093931541325,
+            0.009163959012862627,
+            0.0008611776492791918,
+            0.003598831885031635,
+            -0.02103045281939351,
+            -0.01150941825152158,
+            0.027341774725604333,
+            -0.0006700608710615124,
+            -0.012089044078073347,
+            0.022396400579038037,
+            0.017491565611764764,
+            0.0051760367056280536,
+            -0.00559536785318233,
+            0.01979262595019399,
+            0.002306380546194741,
+            0.011973863166872444,
+            -0.0028136750606972223,
+            0.012911235736067303,
+            -0.01510421118359169,
+            -0.003338602157155174,
+            0.039758808572572274,
+            0.007290642519242007,
+            -0.001802921446521437,
+            -0.013696774990705774,
+            0.003941198181491921,
+            -0.0008308293948698925,
+            0.0200351513103389,
+            -0.00014327587644797734,
+            0.028291423013030337,
+            -0.0001085859276231927,
+            -0.010723064585989723,
+            0.020443848393732245,
+            -0.017860296930396913,
+            0.04540901629084378,
+            -0.018671578423261854,
+            0.000852031295629291,
+            -0.010639853151638835,
+            -0.019017194262730474,
+            0.003861296738239369,
+            0.0010977143012920602,
+            0.0008387482016061222,
+            0.0060155816314618,
+            -0.010780239332444652,
+            0.002152214497259421,
+            0.010556338812873054,
+            -0.027636192450621728,
+            0.0010938992111792307,
+            -0.014385202899929138,
+            -0.01353192238925814,
+            -0.026667778226211785,
+            -0.00017403398236549974,
+            -0.015357674131635848,
+            -0.013956722505869091,
+            -0.01985038526848704,
+            -0.008927903879720431,
+            -0.009580214158077466,
+            -0.0020314441852913386,
+            -0.004755584878514854,
+            -0.00018388548107372635,
+            -0.002918855567842974,
+            -0.0015941852262316128,
+            -0.014230272061190995,
+            0.02133992212037431,
+            0.015283345362445234,
+            0.004642765060980717,
+            0.012829570028541471,
+            0.01708952142208596,
+            -0.004770647543823141,
+            0.006604762788847952,
+            -0.0025369836845081078,
+            0.008368000846861623,
+            0.014785031726639733,
+            0.012074738887486398,
+            0.02124518119032885,
+            -0.015459818988572564,
+            -0.004992865135357168,
+            -0.012722391036055767,
+            -0.012210202603282509,
+            -0.017850065946244383,
+            -0.004300429148015694,
+            -0.0014387298144714717,
+            -0.023128300275416423,
+            0.004468952290723408,
+            -0.0026602931688877443,
+            -0.005768852559887139,
+            0.007815212168719318,
+            0.0005862972228454734,
+            -0.002265719947629544,
+            -0.00019323737674204774,
+            -0.02735820445266249,
+            0.00024009122317215594,
+            -0.00018701993387558173,
+            -0.014663710240203917,
+            -0.00450797561624596,
+            0.010584731989847334,
+            -0.011966527748437568,
+            -0.00602367292291343,
+            0.0111155226518017,
+            0.0015565179432852256,
+            0.006062862961835896,
+            0.016952559697728433,
+            0.0005012157272316869,
+            0.011160668242334889,
+            0.014993782560797772,
+            -0.01591303023837098,
+            -0.015356412848444082,
+            -0.0236025426434855,
+            -0.004514538154596937,
+            -0.009753334017065119,
+            0.007553951535816575,
+            -0.004616595532996391,
+            -0.013023153809236325,
+            0.006040285197946957,
+            -0.0006343451398642784,
+            0.008338564073158257,
+            -0.017608246105292328,
+            -8.346168119689692e-05,
+            -0.020944870308705715,
+            0.009230123321316848,
+            -0.00531237224627929,
+            -0.00445985089300067,
+            0.01410675485522911,
+            0.02895945580213926,
+            -0.007799830351886844,
+            -0.009455128322575888,
+            -0.005663072244883114,
+            0.012817229945992764,
+            0.012474431030896143,
+            -0.004423141749110039,
+            -0.0018294512146253298,
+            -0.012817101093394574,
+            0.0004755435377918658,
+            -0.01129485262081729,
+            0.0062747111125005155,
+            0.017199374798326412,
+            -0.021965117839794063,
+            -0.01296097993683748,
+            -0.016765996091978336,
+            -0.016882855472266523,
+            0.004372970343291638,
+            0.017290565252244258,
+            0.022670788364378462,
+            -0.008456547449078897,
+            -0.019564049850942417,
+            0.006898353991539946,
+            -0.01620125883985687,
+            0.0037108638122592746,
+            0.12887455743463638,
+            -0.0028864852421660366,
+            -0.011487804449328495,
+            -0.022683334123299446,
+            -0.005467192611880829,
+            0.01210557696712196,
+            -0.01555159294496261,
+            0.06709672179195036,
+            -0.015071089552631105,
+            0.004364902817496082,
+            -0.004899872946473657,
+            0.0006310685609937571,
+            -0.017610935588191347,
+            -0.005962991715597585,
+            -0.007768819050529025,
+            -0.02688607975562596,
+            0.000883763319535686,
+            -0.009935313912803587,
+            -0.012883104064962595,
+            0.002126368813790577,
+            -0.0013697265269800346,
+            -0.016131186838419288,
+            0.01575514418037863,
+            -0.020677990768697592,
+            -0.007251917191831732,
+            -0.0029325623780992118,
+            -0.00600710212646933,
+            0.014573093764769918,
+            -0.03253336117394778,
+            0.002973725493330224,
+            -0.013601417220439602,
+            0.011905788192118131,
+            0.001542117126832946,
+            -0.010789486980776022,
+            -0.013862139885514524,
+            0.007086571810314664,
+            0.015658913897173155,
+            0.017039624764452546,
+            0.01901374401756777,
+            0.004797954569087795,
+            0.003928999581093576,
+            -0.008046722893614655,
+            0.021974371139905104,
+            -0.00697335955527197,
+            0.0008975473383295958,
+            -0.00970589295353857,
+            -0.004346869073499034,
+            0.0031279458769925416,
+            0.02303983493541673,
+            -0.012560380384977343,
+            -0.01536378830925542,
+            0.0012035987157572355,
+            0.033313215063595594,
+            -0.010845084209743817,
+            0.013587126023659577,
+            0.0029867620644300073,
+            0.020679673435988446,
+            -0.0006640787253300063,
+            0.0025071780490108306,
+            0.022554193713122565,
+            -0.018082906254599396,
+            0.018230000220500718,
+            -0.004451773302171552,
+            0.01915083025790679,
+            -0.004183521043835777,
+            0.015019320101074747,
+            -0.027014055979204272,
+            -0.005362044589023088,
+            -0.012355559881051253,
+            0.003995309003198048,
+            0.020680455787871383,
+            0.0054391755324872375,
+            0.009865676762244514,
+            -0.01645386866657314,
+            -0.01760657627715638,
+            0.022969828924328126,
+            -0.021566687194482292,
+            -0.015826424849757402,
+            0.00012804774676298237,
+            -0.015021970627042762,
+            0.006000610438452896,
+            0.010126179023323158,
+            -0.02912121211344025,
+            -0.02046150879033127,
+            0.004881850039543051,
+            -0.0023469840669773042,
+            -0.00915786433296249,
+            0.006547377770403066,
+            -0.022329063637464074,
+            -0.015702777677743352,
+            -0.005692904224111613,
+            0.01207999389934878,
+            -0.008429160038192635,
+            0.010075061963131637,
+            -0.020106712664936504,
+            0.0036787045650556907,
+            -0.002764965946892891,
+            0.010307702251974629,
+            0.004697169247387421,
+            0.011192285826515207,
+            0.0030113901179382946,
+            0.0014774681536380047,
+            -0.02778270000997313,
+            0.010043261430740039,
+            -5.178406450222847e-05,
+            0.04216636255860709,
+            0.030977096295679525,
+            0.004479857028809617,
+            -5.798020745008712e-05,
+            -0.009996471945562616,
+            0.012476009017187798,
+            -0.019306907391548302,
+            -0.005092322343599383,
+            0.01856463597985508,
+            0.02195467141594795,
+            -0.02236232264139305,
+            0.0052217593371508145,
+            -0.016766955100332014,
+            0.016133299654723637,
+            -0.011208763137897244,
+            0.006248024409576809,
+            0.011601988603146743,
+            0.0022330878390492527,
+            0.010317781892717903,
+            -0.005895142237225119,
+            0.019670196605183973,
+            0.01186677200787666,
+            -0.002339367025714548,
+            0.02367408853110795,
+            0.006858134385159451,
+            0.004479683673962342,
+            0.004798719889930319,
+            -0.009604173975576537,
+            -0.009045602998213713,
+            0.009230969488015385,
+            0.004083616606774546,
+            -0.007944628589259331,
+            0.011836323287097144,
+            0.015100350195476968,
+            0.01885317701149666,
+            -0.0011707847499298092,
+            -0.00684548433712413,
+            -0.006550059418141426,
+            0.001899758393077906,
+            -0.024577381680573596,
+            -0.02855511854789239,
+            0.021238935710272536,
+            -0.010427135883958709,
+            -0.019187769881105637,
+            -0.0015864640188683347,
+            0.016360037306061392,
+            -0.008853972319021563,
+            -0.01627167079143546,
+            0.001270414822216867,
+            0.02255755055961612,
+            -0.005959411125391464,
+            0.006009654905468478,
+            0.005909233773418581,
+            0.005874361519568704,
+            -0.0027859463941992753,
+            -0.024599789577567636,
+            0.0036612374968223453,
+            0.03221346865585554,
+            0.016461563205582293,
+            0.016139782455492073,
+            -0.007158168059808459,
+            0.001838455290310879,
+            0.007530954381521755,
+            0.005438791523412295,
+            0.015674000912615536,
+            0.008604106101542998,
+            -0.006990462920227135,
+            0.00244736628387359,
+            0.012213048255418866,
+            -0.007329299946651028,
+            0.017424711328323798,
+            0.001973882532418776,
+            -0.0133178791530314,
+            -0.008373452013951822,
+            -0.010462150707776556,
+            -0.014575483571750984,
+            0.0027958704799249897,
+            -0.011401916623878957,
+            0.0008016041012669959,
+            0.003953958717263787,
+            0.010260753041492368,
+            0.013961649189650642,
+            -0.0019275791443041316,
+            0.017019168629298462,
+            0.011534594243860177,
+            -0.0017859096307142257,
+            -0.002970739203219076,
+            -0.03142778833370656,
+            0.010207789755749909,
+            -0.014730553506704018,
+            -0.0037611416875830746,
+            0.01825790347981639,
+            -0.007766890847082858,
+            -0.0032778433418881666,
+            -0.021003405429807113,
+            0.0024512381432602736,
+            0.008768616692363321,
+            0.01558713989712387,
+            -0.012819542714293536,
+            0.009201155584074711,
+            -0.0012254337002053017,
+            -0.0057451574939563045,
+            0.00977251392462179,
+            0.0042356831792011935,
+            -0.0033433823868435786,
+            0.011499764125503874
+          ],
+          "title": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer",
+          "keyphrases": [
+            "Sparsely-Gated Mixture-of-Experts Layer",
+            "conditional computation",
+            "neural network",
+            "language modeling",
+            "machine translation"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "addd7c61-bdd3-470e-8c4f-439af824f89e",
+        "properties": {
+          "page_content": "F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+          "embedding": [
+            0.0002445570513886185,
+            -0.025035678422212698,
+            -0.026056660824720954,
+            0.0021234386221176942,
+            -0.007526403638487191,
+            0.018963965001103752,
+            -0.007728070932730228,
+            0.062213823247851435,
+            -0.00501600735659763,
+            0.031461940272958254,
+            0.014585783857839997,
+            0.010118033107401152,
+            0.009983318452435566,
+            -0.03282616973330728,
+            0.015192449242879527,
+            0.01480596701302375,
+            0.019405922018728892,
+            0.022192910412596127,
+            0.008048967251594976,
+            -0.0367043822293015,
+            0.017737679976556845,
+            -0.022757028540067756,
+            -0.06290354368907101,
+            0.016842087557557506,
+            -0.0050319451760917995,
+            0.019311667659179976,
+            0.008548884527537198,
+            0.02344214718766695,
+            -0.042417710048062586,
+            0.01950972309054391,
+            0.021637767293446648,
+            0.010581067128107188,
+            -0.014214247204390872,
+            0.01016365063627519,
+            -0.025782209417014867,
+            -0.02611397356963557,
+            0.034251067799009986,
+            0.0071402676991737055,
+            -0.020147891665364803,
+            0.056472902950962155,
+            -0.006027534100459245,
+            0.005473493744726429,
+            -0.03139313966858068,
+            0.00043634965151753214,
+            0.013171990459657583,
+            -0.016474119463665594,
+            0.0075792906007691105,
+            -0.0032442837467945356,
+            0.02492845005795393,
+            -0.04366675539501486,
+            -0.006033838756045725,
+            -0.018660330752604155,
+            0.02088611082654532,
+            -0.019387265112916365,
+            -0.01222746328548615,
+            0.005977009213829137,
+            0.0012690574665308455,
+            0.05430319716489232,
+            0.058075720286678886,
+            0.02554562736016709,
+            0.02997758641056458,
+            0.00014178556329408164,
+            0.004139467932725944,
+            -0.004454862143349584,
+            0.031094774362479788,
+            0.0017837745263941452,
+            -0.013542834385584962,
+            0.01983846902748998,
+            0.010136662804503968,
+            -0.0387179389182337,
+            -0.06156798393873734,
+            0.015844092498186847,
+            -0.020820695961906926,
+            -0.02678200120240037,
+            -4.570339833017677e-05,
+            -0.015544616932001905,
+            0.005434882475939711,
+            0.050835936350804696,
+            -0.004985250023269435,
+            0.0016509410141492726,
+            0.022801366331672433,
+            -0.004080282248725238,
+            0.0078367492683837,
+            -0.00034876063727630653,
+            0.039772801289415724,
+            0.0288510926430687,
+            -0.022609202992981695,
+            -0.07091181813750308,
+            0.045194222785472735,
+            -0.03225547853169315,
+            -0.032585463285183205,
+            0.01670145480899693,
+            0.03286138231600811,
+            -0.009733112712376757,
+            -0.030197857075781466,
+            0.0731500616714208,
+            -0.005545147701735842,
+            0.017072915518860705,
+            -0.021002529816498382,
+            0.010525361666209598,
+            -0.012745643075817463,
+            0.025352505498532282,
+            -0.003102906957075941,
+            0.016391641977220053,
+            0.02534354270624064,
+            -0.008108690979345334,
+            -0.006451703723989414,
+            0.0065097623653779324,
+            0.04453924610212246,
+            0.0889545538715994,
+            0.019774398083206815,
+            -0.003981259900744729,
+            -0.046615274629371053,
+            0.00732791197790897,
+            -0.02916202133853672,
+            -0.04303144138453202,
+            -0.04367640508681853,
+            -0.010910501188273782,
+            0.023755744360759886,
+            -0.003399898430475389,
+            0.03910227884474275,
+            0.04137187598162877,
+            0.03363334616934253,
+            0.00916121492790106,
+            -0.031686589382074935,
+            0.026907320660055866,
+            -0.01693827759642556,
+            0.002542422168436765,
+            0.00734783450287368,
+            0.008189080458799796,
+            -0.006352648760821389,
+            -0.002333605867785783,
+            -0.03831808898525347,
+            -0.036762698962852246,
+            0.012773386000855496,
+            -0.008349610778914014,
+            0.027936454763857345,
+            0.07015560257394027,
+            -0.03713244149832974,
+            -0.0066245405611366675,
+            0.02802618921440782,
+            0.0017169880395367697,
+            0.023346396749133396,
+            0.007605522879978981,
+            -0.023484167590494896,
+            0.008594274917753499,
+            -0.0071765874950466295,
+            0.0677537727828362,
+            -0.008126000179550069,
+            0.011046754551416857,
+            -0.019516727128214784,
+            0.031654663874166704,
+            0.0007057013357055938,
+            0.021535694192642306,
+            0.03313372731162823,
+            -0.005794425715079591,
+            -0.03865395674172617,
+            0.032271538528276474,
+            -0.029790294001121286,
+            0.0013647166052139265,
+            0.014194415049797637,
+            -0.004899479294176156,
+            -0.015920769574443834,
+            0.0013708196727143433,
+            0.029757298661028133,
+            0.045717552753231847,
+            0.042911032378569906,
+            0.0016019404554248433,
+            -0.06844613496660937,
+            0.06365820914938357,
+            0.012895947255192629,
+            0.03208498337942363,
+            0.009724409704032419,
+            0.042361427690196844,
+            0.008120380964731038,
+            -0.0004384537671641681,
+            -0.005540416382489673,
+            -0.005125920468958917,
+            -0.021186731704155592,
+            0.03258917022091572,
+            -0.01443327736487225,
+            -0.007107908179884635,
+            -0.00206346150761894,
+            0.014042865018508809,
+            0.028517557440008168,
+            0.017283745982594707,
+            0.0029564774390786045,
+            0.04378212980825259,
+            -0.019454732028746872,
+            -0.009437148551161395,
+            0.018820938021162286,
+            0.0008903198503990673,
+            0.025297040557527045,
+            -0.0011429799732923295,
+            -0.01670583390791835,
+            -0.009421860087826363,
+            -0.0328764813310764,
+            0.009563088287577815,
+            -0.0015347427243049344,
+            0.019228666906882602,
+            -0.007174098839673089,
+            -0.04068735475327182,
+            0.008903031576514094,
+            -0.011472954071988202,
+            0.015735403376626513,
+            -0.007084760348074423,
+            0.008729329609340102,
+            -0.00638454270428521,
+            0.016210007209816298,
+            -0.02460475066044075,
+            -0.009047036293783832,
+            -0.022598561172568143,
+            -0.020144302113939427,
+            -0.010886272544512816,
+            -0.010886770387412563,
+            0.008591903125597681,
+            -0.012216526799178626,
+            0.023956058440750026,
+            -0.017402060516065108,
+            -0.018328740623422317,
+            -0.00046940812181371226,
+            0.022921229116282273,
+            0.009026508358954174,
+            -0.00446208425714566,
+            -0.04421346673289319,
+            -0.01214251475197077,
+            -0.016860336137191526,
+            0.005985059770480569,
+            0.02804423359971541,
+            -0.03510963462663318,
+            0.012588036039507566,
+            -0.0018909139908325232,
+            0.0019847602212350957,
+            0.011374423039138836,
+            0.015486762163219968,
+            -0.027040748338117617,
+            -0.016329845569406535,
+            -0.012600824227928017,
+            -0.01765236203915568,
+            0.01673827406092187,
+            0.0016641690585652725,
+            0.003997011410859195,
+            -0.00499922800986537,
+            -0.02782499674212141,
+            -0.008774389172636544,
+            -0.011506812701874534,
+            -0.009783555199790996,
+            -0.015740884114025612,
+            0.013493654174963239,
+            0.014254624157312108,
+            -0.033270488877124596,
+            -0.018446763439664365,
+            0.02247061755496804,
+            -0.0035396859846893343,
+            -0.03759408156279411,
+            0.04544560120093284,
+            0.052893430652542234,
+            -0.02220682841881151,
+            0.002828355769475809,
+            -0.03469141962159207,
+            0.05950248906327849,
+            -0.03444371252359212,
+            -0.03378776392860449,
+            0.06521072243320349,
+            -0.009713026070923203,
+            0.015249223153809483,
+            0.03131206469454657,
+            0.0034865285730464384,
+            -0.0025997752599464445,
+            0.00912836988165194,
+            0.033031075123542926,
+            -0.014878258361522118,
+            -0.013101723481793925,
+            -0.01797681408884685,
+            -0.002397237505725697,
+            -0.03438738791694146,
+            -0.021713511748922477,
+            0.00836495266415799,
+            0.013064910474172053,
+            0.017110113503472666,
+            0.017514586931181946,
+            0.016195079102773242,
+            -0.0024109742901844617,
+            0.01545592172528996,
+            0.001046760697009891,
+            -8.863738247355372e-05,
+            -0.010979717992873124,
+            -0.017762086340346584,
+            -0.052667758929128256,
+            -0.018996338856872796,
+            -0.04421628171000429,
+            -0.005982494173649518,
+            0.0012420734363470585,
+            0.009886466834005627,
+            0.04949340056515024,
+            -0.01295176306623405,
+            0.003508142432105508,
+            0.010930632194984236,
+            0.01019261004578912,
+            -0.006556730312565122,
+            -0.032765993356945154,
+            -0.006485728779853869,
+            0.025709969056450885,
+            -0.022932446214570953,
+            -0.021792413124305354,
+            -0.007047777916806282,
+            0.0038659532636633486,
+            0.02321123092409853,
+            -0.026003302747277822,
+            0.017643847938685852,
+            0.010129447679208158,
+            0.009551458127273026,
+            -0.017747795509094742,
+            -0.02891499441428287,
+            -0.030321565228144824,
+            0.039350178167852384,
+            0.02899873253152933,
+            0.011338359735211436,
+            -0.033634061116901,
+            -0.013629902520109783,
+            0.0376352218933715,
+            -0.0019251015746424632,
+            -0.07267257827375968,
+            0.036685408572352814,
+            0.0050526642351482985,
+            -0.00019289056191322955,
+            -0.013581714265389013,
+            0.05300479823846402,
+            -0.04139717386779843,
+            -0.03335524761191713,
+            -0.00920987783097114,
+            -0.022616612319255443,
+            -0.005274093640024995,
+            -0.0012667939280239298,
+            0.032502321377997974,
+            -0.020089629343721754,
+            -0.024354491736044894,
+            -0.010206744478678104,
+            -0.02465036256948816,
+            0.017966455981144307,
+            -0.012995995315432267,
+            -0.02135780149071486,
+            0.010967719068923269,
+            -0.03402154331207373,
+            -2.0626689329697705e-05,
+            -0.01092309016851862,
+            -0.018683564965672157,
+            0.01571041043301007,
+            0.016259448102797114,
+            0.015403085986247105,
+            0.0020047641027155683,
+            0.015172380396260792,
+            0.008099191720720747,
+            0.004074377809743208,
+            0.01893667379047588,
+            -0.03384452762579117,
+            -0.038822883991989264,
+            0.050315952008617104,
+            0.0008458969404873289,
+            0.0012049589464131246,
+            0.006406225290437734,
+            -0.0042817559898658255,
+            -0.0411169553122232,
+            0.012933323148113837,
+            -0.05784927886369837,
+            0.006262727192870561,
+            0.012928199517490538,
+            -0.01786898028943474,
+            -0.02291183998443847,
+            0.00046953379111417065,
+            -0.024864892714864046,
+            0.02151263275001755,
+            -0.017247234755598105,
+            0.020405349906524557,
+            -0.023068655052252782,
+            -0.006872857342813033,
+            0.05842934123173711,
+            -0.02308642746390288,
+            -0.04925938088613823,
+            0.032829595473385986,
+            -0.007184802367095134,
+            0.03878513331161335,
+            0.055971017386540985,
+            0.016337316575743225,
+            0.006439501224944926,
+            0.026084366083114555,
+            0.013204436213767278,
+            -0.012028383848550473,
+            -0.0383295105300241,
+            0.015805232298071025,
+            0.030751727443425712,
+            -0.04685643586863969,
+            -0.010258856888449854,
+            -0.02541648783174773,
+            0.01599307149086869,
+            0.020046184897703115,
+            -0.0044598427609889935,
+            -0.0012270315730225135,
+            -0.009664326045898607,
+            -0.01520444928791987,
+            -0.02631191309701645,
+            0.02193635473056903,
+            -0.013740214173212537,
+            0.0007504924364311022,
+            -0.001111971939163715,
+            0.0012797636039507028,
+            0.008478066024469051,
+            0.01501219347281274,
+            0.027440349509054086,
+            -0.022728634568050663,
+            0.006916913421268421,
+            -0.008499438828073956,
+            -0.03788213665656746,
+            -0.006718262893819488,
+            0.029198375723488434,
+            0.006169843558767785,
+            -0.016427547826608675,
+            -0.0049009468913267205,
+            0.0022509753971959087,
+            -0.016307639896946134,
+            0.00372456949880952,
+            0.010347360137213142,
+            0.035374109073612686,
+            0.01856811791368892,
+            0.019247903053478954,
+            0.0035591273657627456,
+            0.010292179556783772,
+            0.004040044337368795,
+            0.025869815038962875,
+            -0.04526297785801751,
+            0.030173313035569235,
+            0.01521363974209499,
+            0.009059160353158267,
+            -0.03914219183565228,
+            -0.02367109515190293,
+            -0.025737979702884085,
+            -0.021654077593318197,
+            0.02346511753268162,
+            -0.02730833269330862,
+            -0.006289840626554075,
+            0.006531934540756005,
+            -0.0006804032161296996,
+            0.003928945341546481,
+            -0.0033962179480786624,
+            -0.037975452991752895,
+            -0.009840510213526213,
+            -0.04023861493405766,
+            -0.0241245373830213,
+            0.003822063151525241,
+            0.003249755356115233,
+            -0.06594891648887229,
+            -0.001768385222533634,
+            -0.06222620899240395,
+            -0.013594921028316139,
+            0.03441942883093182,
+            -0.024438842229617237,
+            0.04820486752403123,
+            -0.05486229161412509,
+            0.018219866569279347,
+            -0.046562594596398506,
+            0.011354841468658838,
+            0.021738373879790916,
+            0.04347115691988908,
+            -0.007572536447300832,
+            0.014626550834376365,
+            0.009906079088662038,
+            -0.04530647508884267,
+            -0.04520033458696902,
+            -0.011400230794947595,
+            0.011239630851440557,
+            0.003287355301854498,
+            -0.012304101523783504,
+            0.02623659451177029,
+            0.015903981896266153,
+            0.04300392046214319,
+            0.03414364576071245,
+            -0.0005691335103219632,
+            -0.049506948352089984,
+            -0.027336474868358992,
+            -0.014910043127825028,
+            0.02552137904202476,
+            -0.03055258097953423,
+            0.003401887357920115,
+            -0.03228200232009397,
+            0.013338682663233296,
+            0.018207379090305786,
+            -0.02224210534533673,
+            0.032569850137027664,
+            0.014446434357859929,
+            -0.040301668913492676,
+            0.0012399866793165108,
+            0.0002450513991178417,
+            -0.038300625347349834,
+            0.008119751344998158,
+            -0.013712452991626932,
+            -0.05381257202269606,
+            -0.022199278450525977,
+            -0.014717114803771204,
+            0.02206967996899982,
+            -0.0023470596291977116,
+            0.017222681546831476,
+            -0.029965631581610256,
+            0.0053103260868707786,
+            0.039279027237044066,
+            -0.001286692479487674,
+            -0.01148541970963179,
+            -0.0008706049086180766,
+            0.022795196067681996,
+            -0.01968201411368477,
+            -0.045534614872904305,
+            0.009877856016973188,
+            -0.020566606587457816,
+            0.016002665913994348,
+            -0.015445999456355206,
+            -0.0052940252527317235,
+            0.010360748741468617,
+            -0.03170100753615497,
+            0.02004511444257144,
+            0.009722541379665509,
+            0.006411072442536516,
+            -0.017460683536795665,
+            -0.0033423349847155888,
+            0.015743901543629382,
+            0.047005401368949463,
+            0.0007329338628268773,
+            0.006153613674047577,
+            0.02501606022399451,
+            -0.013054974002495664,
+            0.05036387123988784,
+            -0.02807526639433245,
+            -0.03099731580589117,
+            0.04344605263104773,
+            -0.019489740746149623,
+            -0.017636713524640496,
+            -0.017960525325193457,
+            -0.009792405274000215,
+            -0.0062801692104839674,
+            0.02801421653619339,
+            -0.00712206060099242,
+            -0.005749470909187103,
+            0.020504942834135256,
+            0.013170883165927672,
+            -0.011218763497707175,
+            0.008523945766715354,
+            0.01511514654097483,
+            0.027931571407773977,
+            0.03214582418497515,
+            -0.02345593941074465,
+            0.0333746849861944,
+            0.009324947617983663,
+            -0.004234113006464689,
+            0.005131158074842622,
+            0.0020437263732432953,
+            0.00176803337042029,
+            -0.029372739581870692,
+            -0.01539406767336474,
+            0.015958225909710817,
+            0.007396588215584237,
+            -0.003291618431912445,
+            -0.0316834931316845,
+            -0.0028368107695447525,
+            0.02614797456533501,
+            0.003832488818435695,
+            0.026426519154286857,
+            -0.015605455255333021,
+            -0.02018445776455915,
+            0.008876565653043164,
+            0.023669947107607083,
+            -0.055401884612295824,
+            0.01882053182190406,
+            0.011046506385076857,
+            -2.6003594081234644e-05,
+            0.01591257026228522,
+            -0.0046432737555961385,
+            0.005094314300671912,
+            0.0482159962414822,
+            0.015126643838581288,
+            -0.013318945197936635,
+            0.0020418779929576307,
+            0.015136735547940301,
+            0.001727083579122356,
+            -0.020116332736417462,
+            0.002896210161586164,
+            0.012858057228188658,
+            0.0092589465483404,
+            -0.012984552574495295,
+            -0.005651788746934606,
+            -0.016856929103895944,
+            0.03249171052801259,
+            -0.005843029913406538,
+            0.02336833175997056,
+            0.04800276014595125,
+            0.02261889070736121,
+            -0.03578655846417886,
+            -0.011451513080556048,
+            0.0077334037670644135,
+            0.009415939321950421,
+            -0.012399032514032437,
+            0.006084821107593102,
+            -0.034655634244304866,
+            0.013104268092544735,
+            -0.0016781301605750378,
+            0.012249966178417542,
+            -0.025567673456449492,
+            -0.023488408901029257,
+            0.007732021927617865,
+            -0.006986715221001125,
+            0.01461433573263053,
+            0.00011845838735106802,
+            -0.005506714145045683,
+            -0.006827219009979501,
+            0.0005242424465709722,
+            0.002885486381705293,
+            -0.0042669493792680474,
+            0.021618593454103052,
+            0.00010034640948597114,
+            0.011602625527297699,
+            -0.023477647311478233,
+            -0.0162254255990209,
+            0.031352657169434826,
+            0.03529846802317018,
+            -0.0007059281329032357,
+            0.02568807107901636,
+            -0.00277252926066966,
+            0.010828396988798257,
+            0.00810821945842862,
+            -0.010307381937016405,
+            0.0003407672690264756,
+            -0.003768030765932326,
+            -0.0027367413382233317,
+            -0.012887542072043335,
+            0.0031930180018598015,
+            -0.014086667286395425,
+            -0.01504545079945656,
+            0.02010182276528891,
+            -0.002519095978589373,
+            0.011477055159845283,
+            0.006165264376334778,
+            -0.01139265727997407,
+            -0.016539552549751473,
+            0.01574258576985793,
+            0.05233886430069725,
+            0.022428487725773775,
+            0.044334901638531425,
+            -0.00706941862193002,
+            0.011585028247041065,
+            -0.016760013805097753,
+            0.001211362507398395,
+            0.048283299647824234,
+            0.015403290676022106,
+            -0.013446978976128664,
+            0.024015090204073844,
+            0.013753743307879434,
+            0.0398341544586196,
+            0.0034302789804899806,
+            -0.020152697770903245,
+            0.041614050886963766,
+            0.00392650113407049,
+            0.013465009394885544,
+            0.020487015678247938,
+            0.002430045518615804,
+            0.0035516244267459565,
+            0.02938659684630997,
+            -0.010142701862398358,
+            0.010335745659719125,
+            -0.033187389861872627,
+            -0.021792906328508686,
+            0.02532137764121119,
+            0.0005114146307176893,
+            0.033795722059780535,
+            0.007069260093915293,
+            -0.017628598898476375,
+            -0.017395163521872672,
+            0.021985468266342154,
+            -0.007987351575783343,
+            -0.025037973555114328,
+            0.002627649768366858,
+            0.008700784263932844,
+            -0.004480915144455101,
+            0.006798551384053197,
+            -0.014580806303786968,
+            -0.009741248878656092,
+            0.025066050175629597,
+            0.007330922027951371,
+            0.010472464570407973,
+            -0.024286928122882124,
+            -0.011159235410800293,
+            0.006107141408585794,
+            -0.04181412282877586,
+            0.008647396691893692,
+            0.00528510107024571,
+            -0.006647986735406744,
+            -0.002548771993161688,
+            -0.0011322362644692566,
+            0.00513401828632921,
+            -0.0035963203271502263,
+            -0.014021183815351008,
+            0.022993860073007393,
+            0.02875036422479212,
+            -0.012996574913859303,
+            0.014373647963439562,
+            -0.0093541571449835,
+            -0.019474535598737786,
+            -0.0233122546137944,
+            0.014269723193443946,
+            0.008784376430480408,
+            0.004317715767550436,
+            -0.008273153355003107,
+            0.0010203136295909875,
+            0.001909823045834306,
+            0.02406426043795662,
+            0.049377235716785324,
+            0.005265955549193235,
+            0.029166436381966566,
+            0.017136545810044906,
+            0.03882940870384413,
+            0.0007533463090350569,
+            0.016233986042907378,
+            -0.03311355052072762,
+            0.03082356484543373,
+            -0.0009431717656252606,
+            0.04077734731808,
+            0.0369941736537998,
+            -0.05477277258937169,
+            -0.014658721668336077,
+            0.028327060073149418,
+            0.010896122046930724,
+            0.01436699901738847,
+            -0.003307011398652852,
+            -0.01163199508062231,
+            0.029659278016495517,
+            -0.023936970669211955,
+            -0.010748756889752312,
+            -0.00767811402724157,
+            -0.010785978668072403,
+            0.009408061520238972,
+            -0.0038320809092061126,
+            0.038383104318077226,
+            0.0070413219383093,
+            -0.009684310528333753,
+            0.019154521368080882,
+            -0.006292471963504016,
+            0.01596901434992771,
+            0.0037167338451907566,
+            0.0019879266696342887,
+            0.03104390318244226,
+            -0.003561483519923973,
+            0.028949607591091022,
+            0.015156617705082505,
+            0.017473301120296506,
+            -0.0029321911929339165,
+            -0.017573294769065502,
+            0.027284173131869906,
+            0.009113069584708108,
+            -0.01971507323375489,
+            0.003667023499937692,
+            -0.0025339283534597176,
+            -0.0077697455666997945,
+            -0.010491575421861018,
+            -9.407001004405342e-05,
+            -0.00211520268689493,
+            0.02087980536494742,
+            -0.005582301812064227,
+            -0.0016376727058019977,
+            -0.0064225805186745605,
+            0.0052835419390265485,
+            -0.019662445004865352,
+            0.015117937189193789,
+            0.007575238218238001,
+            -0.018537762525347407,
+            0.014673223139419801,
+            0.019165766004277593,
+            0.0343151018851732,
+            -0.023949650834192267,
+            -0.0009279394158895472,
+            0.01730410035639405,
+            0.005928527001253011,
+            -0.018975612465080345,
+            -0.033873819454780706,
+            0.00825263613547798,
+            -0.01745801647450136,
+            0.027401724356781083,
+            0.008830481774058282,
+            0.0347823637819511,
+            -0.02309575756624861,
+            0.020559439488958053,
+            -1.2525886083543457e-05,
+            0.01593239963830551,
+            0.01455178268663934,
+            0.027993448238508672,
+            -0.017573784980205873,
+            -0.0379374065982424,
+            -0.0030450459335320053,
+            -0.0029968228865394214,
+            0.016851016821808874,
+            -0.013002468483615099,
+            -0.003337198735398279,
+            -0.0019044899095842511,
+            -0.029686441224288696,
+            -0.021487593939979652,
+            0.027796807698461906,
+            -0.04637191752173605,
+            0.0030176823995715597,
+            0.03281049040516897,
+            -0.030047693785808505,
+            -0.04246000386774685,
+            -0.0014603539776430917,
+            -0.03560639883119047,
+            0.0010602583020630973,
+            -0.01222013383414755,
+            0.029128085483749913,
+            0.007388529661657186,
+            -0.020518571527741426,
+            0.018184513437823,
+            0.0076594003031722,
+            -0.004161472492195519,
+            0.006100954284405263,
+            -0.005239699097592758,
+            -0.028136388241670696,
+            -0.015254134018585082,
+            -0.03037392857155718,
+            -1.6317930920712043e-06,
+            -0.002478230556191563,
+            -0.012845180214134591,
+            -0.024383682688014996,
+            -0.000484419407302015,
+            -0.014810859291916117,
+            3.958274758340876e-06,
+            0.05136363130801806,
+            0.02120810926469503,
+            0.005060247818396135,
+            0.012869784532979883,
+            -0.019665446839172217,
+            0.04743686110578455,
+            0.003514534022065111,
+            0.03583012040901262,
+            0.002062991366066978,
+            -0.02173965243546218,
+            -0.0005002960506349255,
+            0.0319890423992129,
+            -0.0016894669649440992,
+            -0.017249094049398075,
+            0.006136324758346181,
+            -0.009220681391996718,
+            0.00837047468537671,
+            0.012871598576797918,
+            0.01579543926540845,
+            -0.0061999233512262265,
+            0.014725777187858035,
+            -0.003927118678273384,
+            0.009336164668870259,
+            0.03122006739418818,
+            0.019771411252821138,
+            -0.024716950288097537,
+            -0.037736399088695305,
+            -0.012523788001500705,
+            0.001234987976751515,
+            0.034477269953960235,
+            -0.013836458205716058,
+            0.003837278407556882,
+            -0.011027628278972294,
+            -0.0054689562005572675,
+            0.005739209890610896,
+            -0.006985434270987544,
+            0.0015806421600885957,
+            0.012196603223058041,
+            -0.006859901421089723,
+            -0.025670287745499374,
+            -0.004011122451643543,
+            -0.023258896941014143,
+            0.012997299613183496,
+            -0.010144866692641839,
+            -0.006816231069049567,
+            -0.005213359047878286,
+            -0.0014167278420750282,
+            -0.0062535968890290545,
+            0.0006662897611595949,
+            -0.014236807237438571,
+            0.01012846684096945,
+            -0.00899852930003816,
+            0.0016620397335915663,
+            0.007838787681725669,
+            0.007725632848979133,
+            0.0033110678307931205,
+            -0.008206916937522372,
+            -0.008777095103563134,
+            -0.01631820698659747,
+            -0.032412484976539334,
+            -0.03000207979123948,
+            -0.02276516767125653,
+            -0.003870969029785218,
+            0.0034978865240734053,
+            -0.032594545526009384,
+            -0.020635972136797354,
+            -0.002367523739258502,
+            0.0030103817989158062,
+            -0.014587341442812854,
+            0.0014779462725289245,
+            0.007969995140846214,
+            -0.0029319878689507335,
+            0.019355924479361485,
+            -0.005448177691636249,
+            -0.021203447333791942,
+            -0.028956667864566195,
+            0.017851274292380093,
+            -0.015891501231440428,
+            -0.010070179095955473,
+            0.0064388768290910715,
+            -0.005633920123057214,
+            -0.03957042012546069,
+            -0.008945181709739484,
+            0.010992989857329521,
+            0.0035378401849368002,
+            -0.025936727040303652,
+            0.031903561853958325,
+            0.030132441728296204,
+            0.01845519141895959,
+            -0.017902241158092712,
+            0.04710441323879975,
+            -0.006739916600200837,
+            -0.012103058303430145,
+            0.016276540022040643,
+            0.023963592487909288,
+            0.028774476787161263,
+            -0.02643829153200483,
+            -0.005349178006882811,
+            -0.010545703050010008,
+            0.005354643441490569,
+            -0.02271588009302058,
+            0.008468528451368553,
+            -0.013601178278798316,
+            0.011576067943282815,
+            0.019227032134009135,
+            -0.013197207810056759,
+            0.02629307658052319,
+            0.0045263515787155985,
+            -6.067227645639678e-05,
+            -0.0037770406201685248,
+            -0.016287414859552752,
+            -0.012319957965328086,
+            -0.034674880061192896,
+            0.020297690806102293,
+            0.01889209267823917,
+            0.003953684662429981,
+            0.021845046751532583,
+            -0.049434011185276405,
+            0.02806744160678342,
+            0.0017849117483741536,
+            0.021763434864703356,
+            0.013889518910180847,
+            -0.001843068102127134,
+            0.013886691089114936,
+            -0.0032117229736863266,
+            -0.019971320584891997,
+            0.006079730961157539,
+            -0.006352535944493006,
+            -0.03316544192496317,
+            0.014736816574462923,
+            -0.03268321622529288,
+            0.013639134163846678,
+            0.015454170145821088,
+            -0.010170396823729307,
+            -0.021708948733883182,
+            -0.008303375382721087,
+            0.0010570623547459114,
+            -0.0014149688171089135,
+            -0.006849845444850982,
+            -0.04346483266888068,
+            -0.03020319096411844,
+            0.03124837430439466,
+            -0.0016720020681567616,
+            -0.03404132565569305,
+            -0.007209824644083462,
+            -0.02711263625315296,
+            0.004780610743996515,
+            -0.004212647822173262,
+            0.0025676935176963222,
+            -0.048750272936498695,
+            -0.01018434342955251,
+            0.011179560412168381,
+            0.031648611061279724,
+            -0.029298960703078986,
+            0.021370266671953315,
+            3.770954396468699e-05,
+            0.006629905234129844,
+            -0.00943632381701963,
+            -0.00484754585233744,
+            -0.012115337995959813,
+            -0.02607251387584821,
+            0.021868809148577698,
+            -0.0020587964818437345,
+            -0.0013502026980372474,
+            0.0341522429571705,
+            -0.00669289309893371,
+            -0.010113192232031375,
+            0.0005934017402521538,
+            -0.011327797122243258,
+            -0.017269539623150844,
+            -0.010467173785198161,
+            0.0005779464354975677,
+            0.0030406502310922306,
+            -0.024755136158113783,
+            0.004943446717510779,
+            0.04566273647510376,
+            0.013607936503018211,
+            0.02534200532143803,
+            0.017104750405207106,
+            -0.03792550084708064,
+            -0.04394052120951292,
+            0.0012600504352102988,
+            -0.005807815271334456,
+            -0.01944685298932453,
+            -0.01732108197375621,
+            0.0011677414026425442,
+            0.00790730598557849,
+            0.0037240231183698074,
+            -0.011427532982681041,
+            -0.02086964367350831,
+            -0.032539146090370744,
+            -0.0136797717535217,
+            -0.009464326388205681,
+            -0.007302204556505735,
+            -0.032703238152349344,
+            -0.015929357859286307,
+            0.026848214129378756,
+            0.01012098852614113,
+            -0.005646819862636667,
+            -0.0107564788588665,
+            -0.024118155140225247,
+            0.007122339033521904,
+            -0.0027183981945039814,
+            -0.009219030314883272,
+            -0.004672057262936543,
+            -0.016450690196817726,
+            0.02348093022809639,
+            0.009139210149358477,
+            0.024997129282118148,
+            -0.058533911093319214,
+            -0.014411932474734742,
+            0.014256038602455438,
+            0.0020188651815054508,
+            0.001417136751974749,
+            0.029836680491427268,
+            0.0028786535004858044,
+            -0.006526872714072965,
+            -0.02967220833325222,
+            0.028817295504169067,
+            0.027449694157944637,
+            0.01800200378294509,
+            -0.010560094796033294,
+            0.0033996190760546383,
+            0.006004691351249025,
+            -0.005679195734863024,
+            -0.011927786207717698,
+            0.0021260244673648285,
+            0.018998574211603905,
+            0.017174632765532803,
+            0.014659925822127794,
+            0.02455156987384756,
+            -0.005882590537857664,
+            -0.03911277764764406,
+            -0.009781110378157883,
+            -0.0119630929326396,
+            -0.004815125964940769,
+            -0.0026631770014339242,
+            0.008903827530107938,
+            -0.011824767329332145,
+            0.013088406716464123,
+            0.030264448923574067,
+            0.042592268021664714,
+            0.021292401951863188,
+            0.006150997237661769,
+            -0.005463801502425317,
+            0.0007399518998122848,
+            0.0038649546282459707,
+            0.011355675550395135,
+            0.035762872771750435,
+            -0.018147218572765546,
+            -0.019704502923991365,
+            0.0015558311499702044,
+            0.023543487977413267,
+            -0.007150197116420024,
+            0.02608209797565159,
+            0.023985156585794257,
+            -0.0003434024317322991,
+            -0.01271338268931938,
+            0.000985282363050309,
+            0.006710364567868783,
+            0.014540943712513688,
+            -0.02154205198718209,
+            0.004039619025717627,
+            0.001865104196024055,
+            -0.009756600776161738,
+            0.029738871754072076,
+            0.00039154552800346406,
+            -0.012981811693538832,
+            0.03825344802886,
+            0.11025963334575106,
+            0.001476339401950677,
+            0.0031187927813794386,
+            -0.031522639386920355,
+            -0.015113380587160163,
+            0.0021786482865071505,
+            -0.005948756393737655,
+            0.020082591515778573,
+            0.012751062800862873,
+            0.0055847042526916265,
+            0.010100961720464159,
+            0.038084609873482715,
+            -0.009203643420546616,
+            0.009804311800033327,
+            -0.01878272400403416,
+            0.017185477997947226,
+            -0.00972716681599207,
+            -0.00857276597428136,
+            -0.007641511960016827,
+            0.004167416313632853,
+            0.00984005584819001,
+            -0.004596999098606572,
+            0.018810908413859465,
+            0.002496310438422421,
+            -0.01613133688770545,
+            0.011505030137914962,
+            0.007437924384075958,
+            -0.005386885126779478,
+            -0.031755982499042136,
+            -0.013811586180238287,
+            0.01190667400285204,
+            -0.0013786018401773159,
+            0.0007177632889203917,
+            0.012342640512536284,
+            0.01205074763420124,
+            0.031217897495485192,
+            -0.0244324065837541,
+            0.0025071378510626696,
+            0.0070486374045588465,
+            0.0159322020159265,
+            0.01098799814774876,
+            0.00029844505852147526,
+            -0.015456787478044924,
+            0.027139764983358832,
+            -0.011978684728746428,
+            0.010378658252224036,
+            0.01146607671795495,
+            -0.04131658683485702,
+            -0.0009967932006190922,
+            -0.005916133731248287,
+            -0.015698754902727707,
+            0.013905118384587169,
+            -0.040163447416240027,
+            -0.005682084326086952,
+            0.004297114532998809,
+            0.010724201884417054,
+            0.0058749102353125955,
+            -0.017715270733111323,
+            0.009962853888582856,
+            0.00955216765250444,
+            0.0025392532243413593,
+            -0.006356691980285544,
+            -0.006218169970979339,
+            -0.008606942887730427,
+            -0.004280171880086814,
+            -0.015669721147884084,
+            0.0057481312044666884,
+            0.0077227094511494055,
+            -0.018751123856436715,
+            -0.01585864749751703,
+            0.009183099781375234,
+            0.006837438250956817,
+            -0.001445622524955126,
+            0.004828557942941401,
+            -0.010034012200008746,
+            -0.004543834537450213,
+            0.01097540631691429,
+            -0.009334158551770033,
+            -0.004804430123454511,
+            -0.02609985520241612,
+            0.013342299390862539,
+            0.011103136832190223,
+            -0.008261332966534102,
+            -0.006944131083745896,
+            -0.016931502665633764,
+            -0.0031318730206304824,
+            -0.0026508016484082988,
+            0.0016110884325639078,
+            0.017071266355203386,
+            -0.009906320666640836,
+            0.010877416311741906,
+            -0.03496707406301918,
+            -0.015357787514985604,
+            0.01622830705930841,
+            -0.01290003882907745,
+            -0.005625420341281716,
+            -0.009838333350984609,
+            0.0006911383780218238,
+            -0.007545081680638308,
+            0.006074752194220675,
+            -0.041320607445282646,
+            0.0035102345636816475,
+            0.018825308640738208,
+            -0.009723969339534365,
+            -0.020829421583878845,
+            -0.009055117268112308,
+            0.010487235049013567,
+            -0.015831829161845813,
+            -0.01780922120240258,
+            0.0016219998217122911,
+            0.010559783293374071,
+            -0.0055135690971065965,
+            -0.005160232484458576,
+            0.006163424486940105,
+            0.020247627989655837,
+            0.015220964429515309,
+            0.04337432385613251,
+            -0.011915355138717245,
+            0.009617347907919507,
+            0.027812934121040484,
+            -0.030652481946156104,
+            0.01037470561447702,
+            0.02599512377987521,
+            0.0037340561802882417,
+            -0.0013375196064018082,
+            -0.010197691376747194,
+            0.004161692215514214,
+            -0.012254402844820636,
+            0.008715552098450461,
+            0.004697361473725416,
+            -0.027978648968732336,
+            0.017196771853396124,
+            0.019448218659009996,
+            0.0038720194629308794,
+            0.0006995062900842749,
+            -0.01527901684789497,
+            -0.011825577013200781,
+            0.019194236643713274,
+            0.027690065919038755,
+            0.005460028121722145,
+            -0.015512497062157167,
+            0.02370643287056251,
+            0.012916257803027072,
+            -0.004973882005506947,
+            0.0022332716531575122,
+            -0.016984605526823043,
+            -0.01443316577897661,
+            0.030942241604983316,
+            0.011563751997976407,
+            0.01908030186367601,
+            -0.021614212853976106,
+            -0.003982310717524619,
+            0.019931579069642925,
+            -0.021029366993395197,
+            0.013006879711144009,
+            -0.019497570222553013,
+            -0.014067925104548129,
+            -0.010687576618198498,
+            0.02884494104893037,
+            0.009229770955288097,
+            0.026701705044769222,
+            0.012715371448003833,
+            -0.021444875656527786,
+            -0.021050591467198086,
+            -0.0027703267406452336,
+            -0.02721606030674425,
+            0.01688256721881862,
+            -0.055990128879151335,
+            0.0016563858333845547,
+            0.010911810464298247,
+            -0.008604898990983188,
+            0.004039521277648332,
+            -0.007858378479197747,
+            0.010021929060776957,
+            0.020926274809879138,
+            -0.02956229623532218,
+            0.01256060127924585,
+            0.01156331569664224,
+            0.0218688589850221,
+            -0.018861065515394883,
+            0.015339087155577661,
+            -0.00841001512858443,
+            -0.027396412391138852,
+            0.008323898095545889,
+            0.01763697170382491,
+            0.014160184414353919,
+            -0.011992345081843988,
+            0.004485633614765869,
+            -0.04000932898876895,
+            0.027999519977069676,
+            -0.016823500188831896,
+            0.002010511519234648,
+            -0.004537991252901591,
+            -0.0067219583064939085,
+            -0.008523354091034399,
+            2.682477256914193e-05,
+            -0.0005371719103087346,
+            -0.030920653652724776,
+            0.0012939076000836422,
+            -0.04144943703666512,
+            -0.02260198516292792,
+            0.013327519374598822,
+            -0.0012897465780583367,
+            -0.005059170593504863,
+            0.02999548027478466,
+            -0.02863650640718448,
+            -0.027272295036911975,
+            -0.014376307736824656,
+            0.03534844385341626,
+            -0.015358480569131909,
+            -0.000540068394391221,
+            -0.009287193719601908,
+            -0.02236594406404711,
+            0.01091078016405303,
+            -0.006469935602381157,
+            -0.0019272207412132534,
+            -0.01387843655423569,
+            -0.03972775413987396,
+            -0.005948332199741248,
+            0.019500395452013777,
+            0.0006122786247500248,
+            0.003656719755271774,
+            0.014473976840529128,
+            -0.0072278373928449675,
+            -0.0036623750417377524,
+            -0.0038809301345520044,
+            0.026929584793529297,
+            -0.022225170775945557,
+            0.009744440210592443,
+            0.004704883568603746,
+            -0.0005991305039095727,
+            0.017317686588588826,
+            0.000351879037187266,
+            0.0024030356296394478,
+            0.0029060795825935717,
+            -0.0035301907818594274,
+            0.02922907976800865,
+            0.0064070048516525,
+            0.0016291092834589405,
+            0.0001988403049139903,
+            0.004746570638478093,
+            0.01395197992587789,
+            -0.01069245020264236,
+            0.0157940052093005,
+            0.02865126209030693,
+            0.00021287181382353683,
+            0.018345429962063935,
+            -0.015921361819192573,
+            -0.036059435316760584,
+            0.019807131219096722,
+            -0.029642240399677812,
+            -0.003504127677138975,
+            0.022473005924273325,
+            -0.012903852741467797,
+            -0.0031750649075827082,
+            -0.01107957248024579,
+            0.027243132530764567,
+            0.023512079029549708,
+            -0.014856101416774535,
+            -0.015142539116300868,
+            0.004337433267586975,
+            -0.010210607143984678,
+            0.010939153927790107,
+            0.029382995339774957,
+            -0.011230337698862454,
+            0.011717621080350227,
+            0.001161341258722717,
+            -0.003778338045364591,
+            0.019300005069271462,
+            0.0027681684372537606,
+            -0.007401677666787755,
+            0.025628069381915978,
+            -0.008115857591022265,
+            0.013043669211084909,
+            -0.004672358463640539,
+            -0.012902494254220482,
+            -0.003106794914564588,
+            -0.017558145644151636,
+            -0.01225024815282763,
+            0.0011522376202485873,
+            -0.005528597019035285,
+            0.06711658458916447,
+            0.010930891527455397,
+            -0.017888423424597618,
+            -0.0059190199704595775,
+            0.00955567741445378,
+            -0.000966061435587513,
+            -0.009466328431452424,
+            0.027063767833815658,
+            -0.016888493246695876,
+            -0.009681104485277698,
+            -3.814655691584766e-05,
+            -0.0233149747136092,
+            0.0030596902889784496,
+            0.010827987186229144,
+            0.0012052973350181351,
+            -0.013994504803884577,
+            0.005318095523120499,
+            -0.004843763992616558,
+            0.0034188401938723923,
+            0.003678626789714963,
+            -0.0025749484272345573,
+            -0.005333294277998365,
+            0.02307826358450567,
+            -0.012791914224263883,
+            -0.008519982879775168,
+            0.01173424692936401,
+            0.001777123104204978,
+            0.003219553788359312,
+            -0.004564745566711077,
+            -0.002015107048152335,
+            -0.017123545909339455,
+            -0.011278616226036531,
+            0.023713473402287875,
+            -0.02733523566514418,
+            -0.00150943777243135,
+            0.017328382962199716,
+            -0.005705544203971564,
+            0.016650187392502975,
+            9.750950361601131e-05,
+            -0.02100066506382883,
+            0.00010090567939930427,
+            -0.024778741347883073,
+            0.010484659495766551,
+            0.013963183459158656,
+            -0.009531952698151101,
+            -0.020878167882780043,
+            -0.027881362421494915,
+            -0.010190031350698889,
+            -0.027246821338460653,
+            -0.0280692647056191,
+            -0.02020582179742232,
+            0.009778340348478517,
+            0.005715364307462615,
+            0.021929057840860695,
+            -0.006380523858228711,
+            -0.027552858236741976,
+            0.015095163660340326,
+            -0.013269344817111162,
+            0.008282918921343354,
+            -0.019991891695114384,
+            0.01754833974582726,
+            -0.014720830704852575,
+            0.02112161088830394,
+            -0.00041409266970033855,
+            -0.004490472334793758,
+            -0.012076561942010787,
+            0.003976913901514849,
+            0.008101357042612863,
+            0.00176590819486174,
+            -0.008977295353115363,
+            0.026242190932756135,
+            0.01707634775369835,
+            -0.007385093173634561,
+            0.01604386075569055,
+            0.031855080733360075,
+            0.014762206220119051,
+            0.038860660002328094,
+            -0.007238177370595944,
+            -0.009237098188374397,
+            -0.02764495845127417,
+            -0.004407870047633689,
+            -0.006820430763682645,
+            -0.004516837004332235,
+            -0.015251921783911087,
+            0.03164122113689215,
+            -0.010371400055822811,
+            -0.01686972159902524,
+            0.03979035122593764,
+            -0.021846728842184655,
+            -0.007084905708666271,
+            0.008642771511425465,
+            -0.013996636040226265,
+            0.03176661229383477,
+            0.007275612529363987,
+            0.018452818352301472,
+            -0.015039890816023917,
+            -0.015988701055097997,
+            -0.0019911190487748244,
+            -0.002806138139429132,
+            -0.020828646560807865,
+            -0.0063794261154720665,
+            -0.01614570050728432,
+            -0.0026319562705484634,
+            -0.00570753504957681,
+            0.015917008930969468,
+            0.0013916917451795444,
+            -0.015420032290910069,
+            0.04488171544231248,
+            0.005033499525006087,
+            0.022768662319611596,
+            0.010777933460735837,
+            0.012372342667053587,
+            0.015819823672091563,
+            0.006759824420801962,
+            -0.0001668726099910387,
+            -0.02211654164646909,
+            0.004261169256891085,
+            -0.005641717035723983,
+            0.0041005490092982915,
+            0.02012094437568505,
+            -0.024436624754739972,
+            -0.029708548399946657,
+            0.024493078535528394,
+            -0.01833101179919427,
+            0.0030908990956393764,
+            -0.0031025537219982234,
+            -0.010059653447038065,
+            -0.011612233298907742,
+            -0.03293899346571458,
+            -0.0005929867206193162,
+            -0.027438115397399914,
+            -0.03635734152790657,
+            -0.006287687204976032,
+            -0.007850474352061472,
+            -0.003574235211373211,
+            0.02449669755605501,
+            0.004269460596739739,
+            -0.029959856511873897,
+            0.01782698044631415,
+            -0.013252733005984911,
+            -0.013342147906120882,
+            0.0029581361690999564,
+            0.004786570391076097,
+            -0.0027618259744494854,
+            0.010720003765859173,
+            0.007898965393358715,
+            0.007358694001293221,
+            -0.010710502803266644,
+            -0.004049671482838962,
+            0.0013263710826446039,
+            0.0030943253037901135,
+            -0.013325799375903166,
+            0.030553973934607993,
+            -0.018812559793353154,
+            -0.004819166852190956,
+            -0.0006499911473215254,
+            0.0034556893023524674,
+            -0.0110929170801761,
+            -0.014862886414702483,
+            0.021004665462647988,
+            0.0031088550726355926,
+            -0.012570481398485388,
+            0.0096952264731202,
+            -0.0018868240406799254,
+            -0.0024767709620225923,
+            0.01176108203579327,
+            -0.002080730326597438,
+            0.014924914157541781,
+            -0.011040818699356567,
+            0.01490073610670544,
+            0.0034649445207609085,
+            0.008645740731109337,
+            0.003603641937848282,
+            -0.01905231764119889,
+            0.009780124784430865,
+            0.016356631955067272,
+            0.013415470760506568,
+            -0.029496530287207723,
+            0.008881639987471632,
+            -0.005975399013716009,
+            -0.009870580036511721,
+            -0.024372445588075428,
+            0.002590423942920311,
+            0.028960961301291905,
+            0.012044179251559706,
+            0.007934516374398097,
+            -0.0077949340827692054,
+            -0.005480592942515702,
+            -0.019923500717513334,
+            0.009041031662948522,
+            -0.01095651774810384,
+            -0.025157908389819465,
+            -0.007578712371715301,
+            0.03430545904801765,
+            -0.010740874257824011,
+            0.01343875386664532,
+            -0.005526757815283282,
+            0.02436552949831661,
+            -0.0027538610388853284,
+            0.0013690344627799137,
+            0.008416180197734094,
+            -0.006091433734127276,
+            -0.001708607133233843,
+            0.013349212286052506,
+            0.041066250931277824,
+            0.0038375614886832564,
+            -0.0050950709988837216,
+            0.01149207527692021,
+            -0.00326389523728132,
+            -0.005712157453687712,
+            -0.00309579814793649,
+            -0.025491974096530076,
+            0.015133356727314608,
+            0.0010745378801251735,
+            -0.017162228172591873,
+            0.021785292616181687,
+            0.027891598422935242,
+            0.013262622112280296,
+            -0.0178471114368855,
+            0.014456217536546996,
+            -0.009278392107874806,
+            0.01629461595238019,
+            0.02423194076020779,
+            0.010067296472207578,
+            -0.011865648674106681,
+            -0.0070073298858753805,
+            0.008985354045706703,
+            0.008096016282907443,
+            0.0004236756235618935,
+            0.020370823466310733,
+            -0.028334250141960886,
+            0.01483270667238479,
+            0.007137078270349882,
+            0.03314918453195744,
+            -0.0006231629401653488,
+            -0.014397909717816884,
+            -0.008918359605259833,
+            -0.002915473990900902,
+            0.011759189078587877,
+            0.003523992368256188,
+            -0.0006853609664806012,
+            -0.004749148984168426,
+            -0.014181883848149463,
+            -0.006778789302277201,
+            0.012376777107762334,
+            -0.0031646329407784245,
+            0.0030514749941358095,
+            -0.0007326603315688041,
+            -0.007811941797535734,
+            0.015661953163847418,
+            0.001669500869963878,
+            -0.0007143628544145883,
+            0.0016729300148826723,
+            6.604560293656475e-06,
+            -0.00467413979302959,
+            -0.019114577369302277,
+            0.008922356418978664,
+            0.00867683974577237,
+            -0.006578536023513047,
+            -0.006467824108991175,
+            0.009324373505311057,
+            -0.01432602608910974,
+            0.01852478469282692,
+            -0.0061254590933158826,
+            -0.006603842196861297,
+            0.00014277641689784565,
+            -0.012328738339684425,
+            -0.003642202698645936,
+            0.0034895681343535914,
+            0.0051453312116433265,
+            -0.000908593523739809,
+            0.0036062764245630334,
+            0.003940522150829955,
+            0.01083964694261132,
+            0.002824079422783228,
+            0.0015266576108775834,
+            0.007280263953807289,
+            -0.01039293741283943,
+            -0.007403606843286745,
+            -0.004993622657038095,
+            -0.006341662331986892,
+            0.007288958567606623,
+            -0.004364975055335228,
+            0.010457360052582658,
+            -0.0062292104145735515,
+            0.020471836369529147,
+            -0.006133009423930811,
+            -0.0024055488697334888,
+            -0.017695183455939345,
+            -0.007234478249839231,
+            0.007156119488831356,
+            0.007878044165936736,
+            0.009359297231370758,
+            -0.04487830669749157,
+            0.005038301058510894,
+            -0.0054192943462008595,
+            0.02941616633783806,
+            -0.009828941844770915,
+            -0.004335130343692854,
+            0.016067824555917796,
+            -0.019100842323053376,
+            0.0037066021889614513,
+            -0.013965982913833292,
+            0.003695036285467973,
+            0.0034558765116311525,
+            0.014205693020177169,
+            0.031307058208919394,
+            -0.007482180869231769,
+            -0.009383819319265733,
+            -0.006384083847148052,
+            0.026967380724567013,
+            0.018643952476557566,
+            0.023548246269020395,
+            -0.011885146764471029,
+            0.00960518654541979,
+            0.001357650512536872,
+            0.018271473493526338,
+            -0.009663920325675169,
+            -0.011163336807225407,
+            -0.000516828412255004,
+            -0.010288703356943627,
+            -0.0032111641654394615,
+            -0.021092460896394025,
+            -0.0033456309643941965,
+            0.025876107752976232,
+            0.0038584058642743987,
+            -0.0036689325941524813,
+            -0.015624044740149702,
+            2.2644755460272697e-05,
+            -0.011931276845804481,
+            -0.0055870010742218364,
+            -0.015697330226950804,
+            -0.028927689083316704,
+            -0.016192017705678192,
+            -0.002914436546908901,
+            -0.017224654082473787,
+            -0.009460990414684924,
+            0.002071634636546831,
+            -0.015533259213802986,
+            0.004918377471991415,
+            -0.015669642231243106,
+            -0.010915412245035698,
+            -0.00046472075835247,
+            0.004672390433470177,
+            -0.004682173537415046,
+            -0.024109027987987815,
+            -0.013628794248618615,
+            -0.0022054473551170926,
+            0.006500170428351522,
+            0.010016298501187265,
+            0.008324628078950177,
+            -0.0069374343736907625,
+            0.011201167788973098,
+            -0.009746811993090597,
+            0.010155637867574787,
+            -0.02773759960959338,
+            -0.04317847702979315,
+            -0.000791583637601122,
+            0.009241551203900427,
+            0.001979688190287368,
+            -0.01884488538245213,
+            0.0038489955050695704,
+            -0.011280060242688173,
+            -0.00847814532468811,
+            -0.007522173019881079,
+            0.01573091287228822,
+            0.004138348969142638,
+            0.013463997539337106,
+            0.0042065270170965705,
+            0.0285467221116185,
+            0.012635383326761538,
+            0.01030078781909351,
+            -0.003092067773698843,
+            0.004763648809248413,
+            -0.0022835042548997662,
+            0.009077632103272405,
+            0.02331475506951857,
+            0.0076225652102231605,
+            -0.002326207128504976,
+            0.00409858249509217,
+            -0.0005698076375460636,
+            -0.005442088641415082,
+            0.018565262151290427,
+            0.02915691852605287,
+            -0.0069811288298562155,
+            0.008120076286611811,
+            -0.0009110258445641983,
+            -0.009900329477406562,
+            0.0038288023984988134,
+            0.005765264445222657,
+            -0.014230725839572923,
+            -0.0024175374911696653,
+            0.009367987848760041,
+            -0.024664375361024386,
+            -0.004113757123212052,
+            0.020371264792326226,
+            0.0024813232005061487,
+            -0.016599537494340785,
+            0.03488073638324099,
+            -0.003709931703179228,
+            0.003010315958426392,
+            -0.00944047118505417,
+            0.016362178907316286,
+            -0.017989464717541103,
+            0.0005390907615384354,
+            0.013131172782591922,
+            -0.005251614069791733,
+            0.005783823857695169,
+            0.003744510952053123,
+            -0.010417674399178015,
+            -0.010824994224858631,
+            -0.0033951133363820767,
+            0.01085129781083312,
+            0.002642227227069058,
+            -0.024446190958178605,
+            -0.017731767444262785,
+            0.019381370848622046,
+            -0.01001206050737818,
+            -0.008290790009659911,
+            0.0009331678418736739,
+            0.0029270629970877303,
+            -0.004637651319894003,
+            -0.004290917518172777,
+            -0.003574121539057569,
+            0.009121344354139133,
+            -0.009485343005168732,
+            0.01637203851901565,
+            0.004043708359084711,
+            0.01843096334879628,
+            -0.011588935298420727,
+            0.005808551475129831,
+            -0.001368952044937682,
+            -0.007242291047654279,
+            -0.019004856431577073,
+            0.0059537808323522675,
+            0.0073777483129045685,
+            0.016051495408653346,
+            0.011060627996179347,
+            0.018569088420830965,
+            0.011345955812555888,
+            0.0039209921807600205,
+            0.001180658155391396,
+            -0.011006781869601802,
+            -0.016371246428912994,
+            -0.014232536816028922,
+            -0.007416908142074357,
+            -0.019561375671062513,
+            -0.0013253236527051844,
+            0.017798265625542643,
+            0.014528038148955117,
+            0.017704367192118375,
+            -0.0048389172759564884,
+            0.016018290057587425,
+            -2.7789810195444745e-05,
+            0.0009215076153434622,
+            -0.005897137279173464,
+            0.001794778099552753,
+            0.016314258230521846,
+            -0.003313666704140227,
+            0.0063152813149371136,
+            0.0002935873451081568,
+            0.01645203621687452,
+            0.022650990579075773,
+            -0.018899997585540014,
+            -0.0014442758993479705,
+            -0.006467599303099793,
+            -0.0021225224281092863,
+            -0.019894752041664215,
+            0.020266472457121338,
+            0.014019662562577204,
+            -0.005002061958298289,
+            -0.00943116174529331,
+            -0.004354703314807638,
+            0.02678118874395904,
+            0.020206530983694557,
+            0.006044260997975214,
+            0.026222728100119382,
+            0.01976601108150248,
+            0.01984865298844998,
+            0.011918890024498122,
+            0.009052182076567042,
+            -0.013092243192366595,
+            0.018358346459726883,
+            -0.01681770553983246,
+            0.0008504895530001561,
+            -0.003223133403363917,
+            0.002743153478851782,
+            -0.007937179998086906,
+            0.013021430914653432,
+            -0.00030955362267617273,
+            -0.025935204965869492,
+            0.007680669831594167,
+            -0.013183173230049736,
+            0.001375652741144253,
+            6.07690288088255e-06,
+            -0.001857629339163609,
+            -0.017652365850892467,
+            -0.0082709435899296,
+            0.0008448530375262078,
+            -0.01435595518048752,
+            0.023866770609514233,
+            -0.026826645910954506,
+            -0.02140002089501948,
+            0.00988141127158139,
+            -0.004436844338114915,
+            -0.012632978351515892,
+            -0.006972217855086974,
+            0.021864041851582767,
+            -0.005139891275508354,
+            -0.016596866242234234,
+            0.014168148679282804,
+            0.0005869000756971533,
+            0.00412958763467087,
+            0.0012364712526909765,
+            -0.008001798306987228,
+            -0.0030416500521329983,
+            0.006860667487819997,
+            0.028693213258586023,
+            -0.011895509688811421,
+            0.0016632663352826491,
+            -0.013799461077520309,
+            -0.0037922015693749754,
+            -0.02974150200357857,
+            0.027752493864472517,
+            -0.011830981692069491,
+            0.01989293240244598,
+            -0.004540523332199857,
+            -0.007667738626842118,
+            -0.021173122202316107,
+            0.035703796333655495,
+            0.004594443089494466,
+            -0.0006664409717898689,
+            0.007256660335524136,
+            -0.047588498838074554,
+            0.01105469709139337,
+            0.006498789757276137,
+            -0.023261890747840144,
+            -0.000964111432535798,
+            -0.003995007813335293,
+            0.02143067593410374,
+            -0.019461700447829744,
+            -0.012777737727876326,
+            0.0021024453304811634,
+            -0.021925886491102078,
+            -0.007056889535073896,
+            -0.0340604475320119,
+            -0.006137802768771592,
+            0.006503135515471076,
+            0.0038385769709340417,
+            0.0020799029890387023,
+            -0.021308499975514292,
+            -0.0309544649831362,
+            0.013308972238482328,
+            0.01651668347077017,
+            0.010007831710070633,
+            0.01200701454477496,
+            0.0067050757442456005,
+            0.011425518484475204,
+            0.00865023114755134,
+            0.00974582806570851,
+            0.008884712478547614,
+            0.039109252942848706,
+            -0.016122498485808056,
+            0.008671479031657285,
+            0.0005699279397862279,
+            0.02156543966774511,
+            -0.02391743870774471,
+            -0.009877938786506814,
+            0.015475531666380014,
+            0.003045729682805746,
+            0.030465388674890067,
+            -0.01511217342930631,
+            0.005735960079359902,
+            0.0030955813611916066,
+            0.003343849304714193,
+            -0.017685941897160877,
+            -0.019795694199965495,
+            -0.015889472669342385,
+            0.0005064000414834533,
+            -0.0007719666713392906,
+            -0.012817517834744753,
+            0.01370266779873307,
+            0.0029457043321269045,
+            0.004380574233134056,
+            0.003997906783171028,
+            -0.02610528890228495,
+            0.0033844277218836307,
+            -0.008397899012048187,
+            0.013703609594527308,
+            -0.0012702392316682096,
+            2.036849934035612e-05,
+            0.007261982817787643,
+            -0.00406175072535136,
+            0.018353316401139956,
+            -0.007206833180919801,
+            -0.004174154024217347,
+            -0.004097578846173991,
+            -0.00573702992914836,
+            -0.0017184308802689716,
+            0.022375713882789285,
+            -0.008746034802967245,
+            0.032729322137817145,
+            -0.014835515993879445,
+            0.01049714579786462,
+            0.013344412726432523,
+            0.006468296817982878,
+            -0.0036641997047404426,
+            0.0003384623491490165,
+            -0.0035345323089769998,
+            -0.006579417470328358,
+            0.015682816949756224,
+            -0.018934655843627496,
+            0.006860259740057359,
+            0.03727162736659208,
+            -0.002536806673033066,
+            -0.00615871353663963,
+            -0.008853057427430264,
+            -0.012968501800661,
+            0.028268388127570685,
+            0.005935705363468247,
+            -0.0013092005585187737,
+            -0.03024974000885413,
+            0.015180604705329429,
+            -0.0037501687534873948,
+            0.0016387074669205704,
+            0.00860407823411825,
+            0.00044285387135976006,
+            0.020127437993717013,
+            -0.0011272951769363546,
+            0.021290480413673304,
+            0.010032359217391118,
+            -0.0010666180914844765,
+            0.011529468490024128,
+            -0.036558434912810016,
+            -0.008579049583792913,
+            0.0038048395503202733,
+            0.019444233855445248,
+            -0.0006623807457136332,
+            -0.015853103419331053,
+            -0.004242407392496502,
+            0.009029483146616182,
+            -0.01059695082306233,
+            -0.013492380798775181,
+            0.016259633969459952,
+            -0.019096544374819852,
+            0.011560512412214506,
+            0.008617334373339341,
+            0.02512750386847096,
+            0.0041202753709624245,
+            0.002613588241299414,
+            0.0035069606381226207,
+            0.006602772841623022,
+            -0.002625433099855357,
+            0.011789793979724633,
+            -0.011592113209370399,
+            0.02102479518464762,
+            -0.008294039267346376,
+            0.007068661518691129,
+            0.0032541671568032306,
+            -0.022166914870027622,
+            -0.018234159989683987,
+            -0.0023642465680105544,
+            0.021312775708470108,
+            0.006016177710817625,
+            -0.012618763344460982,
+            0.0055411096806088885,
+            0.004690341026678943,
+            -0.00547301082703577,
+            0.009118596747082881,
+            0.024398283179066196,
+            -0.0028098889343668772,
+            0.01068095200128235,
+            -0.007137857371825465,
+            -0.027832295014375272,
+            -0.007421428223393234,
+            0.021707007996690945,
+            0.00020946296332069377,
+            0.016793613517175475,
+            0.014396690762872469,
+            0.012889706504719996,
+            -0.00488698200749025,
+            -0.005354238088985329,
+            0.018317372626432005,
+            0.014340608961907827,
+            -0.0048105230366691145,
+            0.001495716612807014,
+            -0.0018529641162211795,
+            -0.017240719704808234,
+            0.026933447120216628,
+            -0.024232238634425658,
+            -0.003438477892192365,
+            0.00892792234782449,
+            -0.012586276728773793,
+            -0.004923824123059311,
+            -0.023468301644204742,
+            -0.02117100775850015,
+            0.014296432515879623,
+            -0.003314359969711106,
+            -0.01750062790930347,
+            -0.006856892040084997,
+            6.621834660421394e-05,
+            -0.003983545180909168,
+            -0.010826144842427663,
+            -0.0028995230412062357,
+            0.0018029406642546938,
+            -0.01286603500476852,
+            0.00937188453692113,
+            -0.00635685147895941,
+            0.010242542894487791,
+            0.0062482230941642404,
+            -0.022974002980633818,
+            0.005235079251883088,
+            0.020160136045340955,
+            -0.008953506137454955,
+            0.0007027144749750591,
+            -0.015318432934167311,
+            -0.006155069535076527,
+            -8.304773776972749e-05,
+            0.010665400866776814,
+            0.013561082526851403,
+            0.019453190615040503,
+            0.008724224096424808,
+            0.017716608184445207,
+            0.0035170519664773133,
+            -0.013169473712817174,
+            0.009862672596286218,
+            0.011645070824075182,
+            0.011745045118639314,
+            0.011761235635339118,
+            -0.0051506969158396755,
+            0.00628898514628358,
+            -0.0049857719357619505,
+            0.008525048536699957,
+            0.027507594807546505,
+            0.0013209667681023782,
+            0.017047984110417937,
+            0.016368796085341072,
+            0.020105811276523177,
+            -0.003759327296302425,
+            3.4626946806238876e-05,
+            0.00035688794161846735,
+            -0.001063984476108484,
+            0.003291497239446139,
+            -0.018279629509070614,
+            -0.01135428882896519,
+            0.017033095140582537,
+            -5.492833742539044e-05,
+            0.024685530931127835,
+            -0.0010090516352729089,
+            -0.0184210790691242,
+            -0.006072072044463014,
+            0.0019766255786695347,
+            -0.0143465314042477,
+            -0.00209268265697904,
+            0.008867797876246725,
+            0.02090777280180668,
+            0.007785784409274212,
+            -0.014694559759226521,
+            0.03266241399157856,
+            0.017596572028431588,
+            0.02301544904486291,
+            -0.02019106568899837,
+            -0.008116927224141509,
+            -0.007678137378694171,
+            0.020318874678520133,
+            -0.0068566512882342135,
+            -0.002510061555870801,
+            0.021473513752723834,
+            0.006645783892214728,
+            0.011211922100954015,
+            0.012286872605920514,
+            0.0003084092164612384,
+            0.0028492628853667647,
+            -0.015161921733689642,
+            0.011855756525919763,
+            -0.005289433905822296,
+            -0.006904878573181685,
+            0.021517401941638887,
+            -0.010444313612915583,
+            -0.009104739247715266,
+            0.005434542886627656,
+            -0.011460046711120444,
+            0.013324525803587403,
+            -0.0016717915349416276,
+            -0.013225789210352699,
+            -0.014868291792978007,
+            -0.004059908595020105,
+            -0.017086039690741954,
+            0.018935544362274584,
+            0.006810606440007176,
+            -0.0016747502037076753,
+            -0.005949836930379681,
+            0.004577809518378731,
+            -0.002387527459083855,
+            -0.0006415824062693734,
+            -0.015923707971787795,
+            0.0035047065077239213,
+            0.009700583796235219,
+            -0.0031324165789907116,
+            0.010643108490671052,
+            -0.013353561795075988,
+            0.033779753889958034,
+            -0.003236875179603599,
+            0.0033068852921624326,
+            0.005106828066670305,
+            -0.003034255184307467,
+            0.001481657575335246,
+            0.013519816427440722,
+            0.009066434673988428,
+            -0.008339461460990311,
+            -0.0009211583209818409,
+            0.013497039042975171,
+            -0.018668722901146233,
+            0.0013796912320793106,
+            0.013796986913155082,
+            -0.015849997808276967,
+            0.0049427770053857125,
+            0.011400118227502903,
+            -0.00639345486098034,
+            0.015340608222895147,
+            -0.01371015934577743,
+            0.007008681477695071,
+            0.009775206231977338,
+            0.028490233702896408,
+            -0.01966685875736091,
+            0.01928758485650401,
+            -0.025635008258084145,
+            0.031283028009357144,
+            0.012122242939206308,
+            -0.005796910058436286,
+            -0.025521126197222552,
+            -0.011593941553769327,
+            0.01283377684428954,
+            -0.021686338476615465,
+            0.0014883363235708173,
+            0.004930128822274494,
+            -0.013589466747020829,
+            -0.014766714233107183,
+            -0.01202283835955618,
+            0.011738578705966321,
+            0.0006399892787922201,
+            0.01589633486439762,
+            -0.001565909739326631,
+            0.006195453613357957,
+            0.00799709051068248,
+            -0.007262779704915372,
+            -0.006717513726969218,
+            -0.01982600498555257,
+            0.01452043414139846,
+            0.004256754663990435,
+            -0.005262312998149605,
+            -0.0224549938750886,
+            0.009397743096285409,
+            0.017707219262338467,
+            -0.005753255460862127,
+            0.013047488144162397,
+            -0.03383526087792666,
+            0.0009452367103428395,
+            -0.005822570832057675,
+            -0.0027621026437538738,
+            -0.011741384193082396,
+            0.005215022845011562,
+            -0.015301048137828595,
+            0.002594097201808009,
+            -0.006446736651739076,
+            -0.016963075831388622,
+            0.002297125878220545,
+            0.02099207084363147,
+            -0.0032941933998622765,
+            -0.0028818889706579017,
+            0.01784874229400347,
+            0.008499720521713671,
+            -0.01436426626087195,
+            -0.004000179664908013,
+            0.006397941984552902,
+            -0.018873859156915206,
+            0.0011599963694687036,
+            0.02198209878751397,
+            0.012652203334020664,
+            -0.026378312815268385,
+            -0.0016397712646032201,
+            0.013833745513458322,
+            0.008230921549345205,
+            0.008667892596847433,
+            0.0040942632709300665,
+            0.014593014365768302,
+            -0.006350891941908016,
+            -0.001374093276890415,
+            -6.3005496707736715e-06,
+            -0.006943874751919358,
+            0.011912879044321251,
+            0.003688355066059739,
+            0.040741622477833674,
+            -0.010639328654064204,
+            0.011006744363538067,
+            -0.01080331270092793,
+            -0.004477468739500037,
+            -0.004503151926350927,
+            0.0027311865257027917,
+            -0.013139063275489816,
+            -0.0001322676781706513,
+            0.0021051078165230533,
+            -0.0019788024041678294,
+            0.00649268452600941,
+            0.003305903531478554,
+            0.013176851471483939,
+            -0.017229299828998786,
+            -0.0024884474570636707,
+            -0.009910839696360717,
+            0.019347666346683084,
+            -0.010724923526200773,
+            0.01051871403061451,
+            0.03256893707453101,
+            0.013242686936629418,
+            0.00022445440868234016,
+            -0.014947304198470308,
+            -0.008836459838385262,
+            -0.008992977994567497,
+            -0.01739161854615299,
+            -0.012908166671408042,
+            -0.022654409684101404,
+            -0.00294050967213434,
+            0.006390611639719366,
+            -0.0010555036392130776,
+            -0.002785796833161689,
+            -0.027340941664129582,
+            -0.00792516100590229,
+            -0.03210910565574123,
+            -0.013679170743667392,
+            0.0056845041078574104,
+            -0.021558178294402857,
+            0.005195998493021035,
+            0.029445968279023734,
+            -0.008256723194649932,
+            0.018403154314213917,
+            0.006923789005793937,
+            0.011433558571124386,
+            -0.0022324062216246424,
+            -0.011870704203994723,
+            -0.011035736680889242,
+            0.00032924465177187114,
+            0.003455564411617082,
+            0.004938023824535148,
+            0.00916834019939669,
+            -0.01732811783799426,
+            -0.010720420451770168,
+            0.00025951769018448803,
+            0.01732173296763328,
+            -0.0036441011455210317,
+            -0.014654182169779525,
+            -0.009890543675136605,
+            0.00934755871928037,
+            -0.017560953324850772,
+            0.0028803276180087,
+            0.008144645056336303,
+            -0.006844751836577803,
+            0.009686357271083834,
+            0.00910269542748879,
+            -0.003634975911996508,
+            -0.01125116892179341,
+            0.015123292946443665,
+            0.011417498955214564,
+            -0.0005683452620824292,
+            0.009547451569266946,
+            0.0031907584166949686,
+            -0.004914343407341038,
+            0.0004946517266559977,
+            -0.013981094295278457,
+            -0.008438177812073363,
+            -0.009698303451884585,
+            -0.0004873949117926506,
+            0.008310275049300703,
+            -0.011244097579407173,
+            0.002589333020166005,
+            0.021677389796728952,
+            0.0016179416889548232,
+            -0.011231602157838225,
+            -0.0045296258349024725,
+            -0.03198651985714177,
+            -0.005752566699226788,
+            -0.017524199510403467,
+            0.014980719037744585,
+            -0.002529640898744345,
+            0.01028755110586937,
+            -0.0021848874276835985,
+            0.015817733088597956,
+            -0.005450667684071798,
+            0.012212046004369696,
+            -0.0008714643856749736,
+            -0.004560548606687623,
+            0.006746171236132563,
+            0.0034423786329218276,
+            -0.0009995673054631,
+            0.05643134636651925,
+            0.010296478797092773,
+            0.014377556149215558,
+            0.013825497434998718,
+            -0.017485595187900386,
+            0.013330532940316836,
+            0.013333976202885402,
+            0.012417102160127097,
+            0.004536479576179834,
+            -0.017256228281240734,
+            0.024337236913894427,
+            0.012755805964855683,
+            0.009622694631579495,
+            -0.0018516129725815202,
+            -0.0015877067248695194,
+            0.019827793457522933,
+            0.018266833138588977,
+            -0.009808363592736615,
+            0.011083663940891296,
+            0.004458465015627857,
+            -0.01068421391840819,
+            -0.0050807436529387556,
+            0.024563333063926475,
+            -0.01325209293057542,
+            -0.016539133573613984,
+            0.0059168302415277515,
+            -0.02488439856049784,
+            -0.029755917745609557,
+            0.001920092403971772,
+            0.006364691645397147,
+            0.002128416061225683,
+            -0.0021827593501562285,
+            0.00636889353893684,
+            0.01505350329221805,
+            -0.02223481012881443,
+            -0.01913380824474348,
+            0.004876437698740333,
+            -0.0031323018872010133,
+            0.015711985535422395,
+            -0.02142499388661488,
+            -0.011646214273628826,
+            0.003244037052013272,
+            -0.006889665577990765,
+            0.004421548112756594,
+            0.004704830748153097,
+            -0.004955793799612646,
+            -0.005239972361031197,
+            0.0007828188358756139,
+            0.019667585310240637,
+            0.017115475760556154,
+            0.02008255444898991,
+            0.012206187649127842,
+            -0.004915394518125827,
+            -0.00997481157928988,
+            0.002664275191354035,
+            -0.017453428141032505,
+            -0.005114569347643741,
+            0.006659824028894164,
+            0.013233844584497872,
+            -0.023794827807567007,
+            -0.007354308951883939,
+            0.012842257789962674,
+            0.011294252132532885,
+            -0.00824836528452984,
+            0.004256336058369505,
+            -0.011193345143979632,
+            0.009248664100690892,
+            0.010363170772580968,
+            0.010981447978991254,
+            -0.01250390246078834,
+            0.008163162620777558,
+            0.002528845973804195,
+            0.007003579605495149,
+            -0.005812703866552746,
+            0.008764871082692903,
+            0.013124123077340675,
+            -0.012212447674686537,
+            -0.0013940507123620205,
+            -0.001544873017903471,
+            -0.020493969663982807,
+            0.005782189204516662,
+            0.007033733269347455,
+            0.012878926833341835,
+            -0.008338295354334415,
+            0.015844568159504493,
+            -0.013423240193733334,
+            -0.00468920516522183,
+            0.0063319500880227525,
+            -0.004924778574104498,
+            -0.012053197667708071,
+            0.0017742020845283848,
+            -0.0035640186446405536,
+            -0.008666749197879684,
+            -0.0013894177719374007,
+            0.0016400693271668141,
+            0.004501038858023909,
+            0.02043221461176279,
+            0.001901506572332553,
+            -0.0056076433740056926,
+            0.019660750419391273,
+            0.003538773720733224,
+            -0.0018659146421562808,
+            0.005354994676719758,
+            0.0006789623595561136,
+            -0.0021516503031452773,
+            -0.0027566542393029298,
+            -0.010552067426241374,
+            0.01699044655141112,
+            -0.01883386136434379,
+            -0.006428123254736374,
+            -0.012514829515253224,
+            0.006923832110523631,
+            0.023065674483579466,
+            -0.0031719793455512423,
+            -0.006131653355136594,
+            -0.015640226612880566,
+            0.003589774262421098,
+            -0.07234225978543331,
+            -0.010636515019878534,
+            0.005436135654579842,
+            0.030362090167699224,
+            -0.008735845879763853,
+            0.0012226978854869956,
+            -0.00802187367908004,
+            0.010460901462092434,
+            -0.0024797678489701584,
+            0.005049311815982583,
+            0.0037643360764659196,
+            -0.003379308997651945,
+            -0.0029803517660911536,
+            0.0047612984744651015,
+            -0.010726918354479238,
+            -0.018870259480662628,
+            0.0028549213021684043,
+            0.025034678854420084,
+            0.009004274035878184,
+            0.004138885468102832,
+            0.009765532829693857,
+            -0.008537379113922283,
+            -0.0014902115840393893,
+            0.008462743183234337,
+            0.007328272099011593,
+            -0.009076904245691212,
+            -0.0008002047467975742,
+            0.008347212876126893,
+            -0.021799374855548816,
+            0.0020253943807330738,
+            0.012246389250647798,
+            -0.008912465890948629,
+            -0.006244789880293639,
+            0.002413038795382646,
+            0.004202974630810528,
+            -0.004339960146144246,
+            -0.009919301413657872,
+            -0.023862183602380987,
+            0.006400371798756263,
+            -0.007410957922309058,
+            -0.0001606112434129549,
+            0.0031662502228666104,
+            0.00012871401736865864,
+            -0.01687187870925491,
+            0.01374600947628664,
+            -0.008834671549515985,
+            -0.0033344289547729495,
+            0.012937173759658114,
+            -0.003267867578821276,
+            0.01864135643890992,
+            -0.014050228437162755,
+            0.007864998326137568,
+            -0.018988002335416778,
+            -0.003944450943320823,
+            -0.0032084001089808037,
+            -0.0023149926514803745,
+            -0.007750311931438483,
+            -0.045019596490520515,
+            0.013573650825957857,
+            -0.004374086208559833,
+            -0.005869005771503507,
+            -0.01250037134701792,
+            -0.016625210662158708,
+            0.011639132814162461,
+            0.004045997475412364,
+            -0.0014885281342871186,
+            -0.020321856013396592,
+            0.006222318025479057,
+            0.011121009394355636,
+            0.00904225037228108,
+            -0.011585148988670874,
+            -0.0018474735072673144,
+            -0.011325387916132704,
+            -0.0025890917877355725,
+            -0.009148717095934154,
+            0.018495348432506,
+            0.002404017871470538,
+            -0.010232069372010613,
+            0.022941087706292104,
+            -0.0024570420644715457,
+            0.020988341201251853,
+            -0.00442363926883869,
+            -0.004485898239018673,
+            -0.020447941631096866,
+            -0.004158168168969816,
+            -0.019358617104349624,
+            0.00036464612100595666,
+            0.0017749921060331022,
+            0.02078748568527368,
+            0.007256675717496499,
+            0.012612722550842472,
+            0.006032799184116686,
+            0.00631983142330159,
+            0.030151438614535002,
+            -0.010852408231267204,
+            0.0022451749744724233,
+            -0.017863828302314567,
+            0.010653806941450981,
+            0.028289026813870025,
+            0.009882042879445822,
+            -0.006920783006708277,
+            0.014053717954960316,
+            0.004932716665921658,
+            -0.00616100713292227,
+            0.025975267677767474,
+            -0.038836632659249616,
+            -0.021059874446788137,
+            -0.0061122622002863615,
+            0.001439949038430725,
+            -3.496872289060673e-05,
+            0.005030571386336924,
+            -0.011007922189367617,
+            0.009222155733548683,
+            0.0026655561130744807,
+            0.03101907459085608,
+            -0.002317540524509921,
+            -0.007348120500263025,
+            0.003987101796623003,
+            0.001511347369284844,
+            0.021229069065914295,
+            -0.01197457000398312,
+            -0.038429200853184124,
+            -0.0031917847963042626,
+            -0.0039488592790253305,
+            0.008371872140716302,
+            -0.012355524487988753,
+            -0.010839396359961,
+            -0.004574476927098761,
+            -0.01882180934875166,
+            -0.010292867074240718,
+            0.025595330740853118,
+            -0.007162452868646569,
+            0.01788200594893745,
+            -0.0076384034977797525,
+            -0.008112046063674713,
+            0.00683079906571602,
+            0.011572398977364099,
+            -0.01932794114157753,
+            0.0034055065356524052,
+            -0.0027824032743918262,
+            0.01627342501880651,
+            0.011718826886349412,
+            -0.013342112644751174,
+            0.006234666245077415,
+            0.01365078200475753,
+            -0.015476695744659024,
+            -0.0015378873781631448,
+            -0.0069121070205125585,
+            0.014866073481143516,
+            -0.003093423905641203,
+            -0.0033762696975023986,
+            0.011958295888528918,
+            -0.021982979614167224,
+            0.01697002117600754,
+            -0.02183323542526832,
+            0.022295922951583082,
+            -0.0006195453455557405,
+            -0.0006320640234089136,
+            0.008423668060539851,
+            0.008314680408345411,
+            -0.013378756052181496,
+            0.0016194432891316378,
+            -0.03416272433582183,
+            -0.009962172336151867,
+            -0.015051382650973002,
+            0.0038009542554177348,
+            -0.010106196779819915,
+            -0.01203081704994182,
+            -0.021869748479536547,
+            -0.013231917312580397,
+            0.009488867268928662,
+            0.02421155136770049,
+            -0.004349181861466944,
+            0.015665735941744613,
+            -0.008077879618601087,
+            -0.00346140349979159,
+            -0.019022501109520248,
+            -0.020158795455478176,
+            -0.011854460388465065,
+            -0.01570428154296937,
+            -0.007073290993912972,
+            0.004794071576762192,
+            -0.014492538032767992,
+            0.013391092874532436,
+            0.03615243857749714,
+            -0.017029836110560772,
+            0.008609740828901341,
+            0.019179133878629843,
+            0.0009923228421593012,
+            0.015393048602064972,
+            0.003790868078763006,
+            0.011710499244312703,
+            -0.014885526287290077,
+            -0.011829126494406653,
+            -0.0005627435441977781,
+            0.0020582596099707785,
+            0.003728052108395647,
+            0.027914733124690644,
+            0.0038604865752177283,
+            0.005302574635039923,
+            0.012293011832771443,
+            0.013320598642264739,
+            -0.010706185765391238,
+            0.015853648683536427,
+            0.004090821159159474,
+            -0.016980287717038522,
+            -0.0031197192883179663,
+            -0.006894138060845641,
+            -0.012739354473335233,
+            -0.002311031608856898,
+            0.0159372988694303,
+            -0.008091532959711314,
+            0.015169701976466646,
+            -0.004316674756425857,
+            -0.015090694840235596,
+            0.0007287957651103694,
+            -0.03810971752015664,
+            -0.0018729979445690102,
+            -0.01301867399306695,
+            -0.023136174288569147,
+            0.04241471786668668,
+            0.00415464531754789,
+            0.020836251111427635,
+            -0.006703690210920993,
+            0.006147438331654669,
+            -0.020673361951422737,
+            0.011783676068789637,
+            -0.0037516561524537765,
+            0.009702463136945385,
+            0.009366943837185086,
+            -0.018879598518030567,
+            -0.01132474738893142,
+            0.018484010977000243,
+            0.0008964474295952695,
+            0.001899853004269779,
+            0.014452586238316783,
+            0.019746940887134506,
+            0.01136527620507647,
+            -0.0020960934961233306,
+            0.02093114800466563,
+            0.018023169361765227,
+            -0.0002475129435970239,
+            -0.007118840223520255,
+            0.013527604253725194,
+            -0.02531530780276405,
+            0.014601163833111726,
+            0.03268673940622878,
+            0.013556815596615061,
+            0.020396267313433348,
+            -0.016161264086739215,
+            0.016259608464481445,
+            -0.006718110409512467,
+            0.017378727943854342,
+            0.001211924791251576,
+            0.01911265175070918,
+            0.014022657097776944,
+            0.01308266544853342,
+            0.006784756253744209,
+            -0.013367188766230505,
+            0.024435211011756285,
+            -0.008342072323415817,
+            -0.0004209982405190692,
+            -0.002823543536521797,
+            -0.023279946325307846,
+            0.01506543259962226,
+            0.0017095020186682082,
+            0.003411744637061027,
+            -0.0035317155649076414,
+            -0.001957702240702538,
+            0.008037874596341196,
+            0.008176809711854463,
+            -0.015780325305129808,
+            0.0031090160698441006,
+            -0.01924979849975192,
+            -0.008566500568783712,
+            -0.024111269290772643,
+            0.003704817310374187,
+            -0.015182531891766549,
+            -0.009971369164255839,
+            -0.019503345629597463,
+            -0.011275291123690537,
+            -0.010129027262987365,
+            0.00026549411615768147,
+            -0.0077553590287942975,
+            0.014297350661562049,
+            -0.017119422159171904,
+            -0.021341131823214652,
+            0.007945720599865192,
+            0.01439604029848557,
+            0.0037256533134793816,
+            0.005472351446140726,
+            0.008325407758477418,
+            0.009082034215555286,
+            -0.010830069811539129,
+            -0.010352768708989164,
+            0.001902868536460922,
+            0.0010307757364948558,
+            0.011572880679973516,
+            0.01480960939553474,
+            0.011328449881374384,
+            -0.013429607187664061,
+            0.0019326261581148884,
+            -0.014862631307250641,
+            0.004686987038977201,
+            -0.0011432874921856958,
+            -0.007026108142521568,
+            -0.0023122147402109927,
+            -0.01067305755615663,
+            0.008329587642130216,
+            0.0051774766682198365,
+            0.0014058609424645312,
+            0.0175095191841222,
+            -0.0016739751490535205,
+            -0.01295313808457633,
+            -0.012767766732175707,
+            -0.011653259281510757,
+            -0.015653450921388538,
+            0.006739771869002279,
+            -0.006908534840842188,
+            -0.006309735406143942,
+            0.00969748644524724,
+            -0.004300978285892996,
+            -0.0071082865141610775,
+            0.009870599453242536,
+            0.0003007194077865931,
+            -0.0011846789159672513,
+            0.03344372215649544,
+            -0.004482311849310301,
+            0.004153681423879411,
+            0.00022516282287549753,
+            -0.00096173441700326,
+            -0.01608715959652527,
+            -0.026963713856310283,
+            -0.0014323662880336336,
+            -0.02273696636049197,
+            0.017938234567205,
+            -0.0058065575851078514,
+            -0.001006759040376689,
+            0.002493842516193206,
+            -0.0042740143630253385,
+            -6.58295494211919e-05,
+            -0.011768915967276026,
+            0.0022431022211178383,
+            -0.029090994997079067,
+            0.009898341521414653,
+            -0.009544941198003441,
+            -0.02154260971685405,
+            0.01133866603878075,
+            0.040150135462722274,
+            -0.0034082812055093122,
+            0.0006477463440760081,
+            0.0010662318198694314,
+            0.010346430535548179,
+            0.014953186668393513,
+            -0.0019239187336010772,
+            -0.01603518108536549,
+            -0.010550241019652155,
+            0.007810013487665037,
+            0.0003366114893320438,
+            0.0030211673366186986,
+            0.025036691996012227,
+            -0.019097101590128545,
+            -0.016027314610013832,
+            -0.011102216330632247,
+            -0.011450582072890358,
+            0.009254748792585386,
+            0.019696746434919966,
+            0.013185751991632784,
+            -0.020709092762886617,
+            -0.017120964154502993,
+            0.00025737692261907426,
+            -0.02662183259737709,
+            -0.0015175233130328546,
+            0.12569047385271798,
+            -0.004250752280902458,
+            -0.009469510190778525,
+            -0.016304049953896647,
+            -0.00863767582150345,
+            0.02183050591695681,
+            -0.008697910752438377,
+            0.06936580824413162,
+            0.0006861269173917291,
+            -0.0029257096931296273,
+            -0.0020896063525880225,
+            -0.002993306834763656,
+            -0.013871927123029641,
+            0.001864949832879178,
+            -0.016847962023455045,
+            -0.01637518023758693,
+            -0.006332388047537449,
+            -0.013325712872379694,
+            -0.0017155737214029734,
+            -0.0061798136617018905,
+            0.004759205228314821,
+            -0.007830001529147273,
+            0.011495001353183024,
+            -0.026107152853762815,
+            -0.010130127689151806,
+            0.005962957535201191,
+            -0.011637257947784772,
+            0.0011307756612654312,
+            -0.030816554116329904,
+            -0.00441741097001083,
+            7.409036337997905e-05,
+            0.016501246370776126,
+            0.0015748302561177006,
+            -0.00940138978060357,
+            -0.011216292373500447,
+            0.006508185064247167,
+            0.021984022174755303,
+            0.02029850289834145,
+            0.020629050747183544,
+            0.009741513502990842,
+            0.017418812192334685,
+            -0.007031919478404256,
+            0.02625468226435525,
+            0.006196721544177288,
+            0.002206946290342702,
+            0.005596780938975919,
+            -0.01211153541496144,
+            -0.008534755574472074,
+            0.0074101096913398124,
+            -0.0219026467376419,
+            -0.0238583746438936,
+            -0.012260621283748488,
+            0.025739429552984337,
+            -0.009054977254128021,
+            -0.0019759724570709175,
+            -0.0030737356218219617,
+            0.024727831360321823,
+            -0.005149317224364814,
+            0.002332136352952536,
+            0.015587381251136284,
+            -0.028035348333895063,
+            0.008643405636450286,
+            0.007622365990511162,
+            0.02564089666732,
+            -0.0032690983288409517,
+            0.0030897564314706603,
+            -0.009711753474608342,
+            -0.009285462125807194,
+            -0.009613024258506932,
+            0.004240295765592783,
+            0.018961596475867332,
+            0.005797211858953558,
+            0.009922313409983964,
+            -0.006335322307636977,
+            -0.015248479856831972,
+            0.018517674622994848,
+            -0.018526204881631703,
+            -0.017805480495530986,
+            -0.0031014632675223346,
+            -0.023499333544416315,
+            0.007508363381003107,
+            0.006594465447638183,
+            -0.013304818586245387,
+            -0.0232989529893049,
+            0.020407788403667446,
+            0.003426820233878876,
+            -0.006520960224510217,
+            0.0028020004149410366,
+            -0.03282622678411901,
+            -0.018099884128291804,
+            0.004341732255376245,
+            0.012442902960644058,
+            -0.006914419410521592,
+            0.01684289728480369,
+            -0.0213314411022478,
+            0.021170682961259797,
+            -0.018444729425471068,
+            0.015400297173773751,
+            -0.005856807329212788,
+            0.01661347113847185,
+            0.011889835104373263,
+            0.0013585650928227913,
+            -0.019926344687386053,
+            0.02233564528528766,
+            0.0006518460244140545,
+            0.029977976063273147,
+            0.026669882018303386,
+            -0.005623183408671759,
+            -0.004497323576380724,
+            -0.017964594171161242,
+            0.011149712487916564,
+            0.0034380718823694635,
+            0.00295265329839474,
+            0.023835186931253683,
+            0.02803963164846445,
+            -0.028097235508129332,
+            0.017703040589712025,
+            -0.012893023698870347,
+            0.0089348477172039,
+            -0.011630331709306501,
+            0.0096768368423985,
+            0.00669273666692988,
+            0.003825890215203911,
+            0.008289398654583944,
+            0.011807275797649763,
+            0.024893949220958504,
+            0.02893733381281628,
+            0.004069052493622539,
+            0.012465630768347759,
+            -0.003600044227856378,
+            0.004954707230784128,
+            0.005985890100113585,
+            0.0014569711209781388,
+            -0.00648453455551435,
+            0.0031945554565833674,
+            -0.006090168119075504,
+            -0.003744222112779555,
+            0.015266602618825367,
+            -0.00028966361595811053,
+            0.015634624351841753,
+            0.012395295870300452,
+            -0.006577117978669073,
+            -0.003069372773552921,
+            -0.0029369830414480856,
+            -0.007598245840968154,
+            -0.017753359726409172,
+            0.025057809370078198,
+            -0.01606095813358513,
+            -0.019631204097884002,
+            0.0008782653868466668,
+            0.0031106969086903075,
+            -0.013675045629189149,
+            -0.008463196400259825,
+            0.008777824363912681,
+            0.014471929439779247,
+            -0.018478175346087953,
+            0.0017862044141924377,
+            -0.0004910793014860654,
+            0.013862986726603731,
+            -0.003641460118879092,
+            -0.00902817956483427,
+            -0.002076665354355939,
+            0.010393957588184604,
+            0.016413601414994997,
+            0.0038017609859922327,
+            -0.008644753777502864,
+            -0.0005484639867925445,
+            0.006452371249117759,
+            0.007391332924183688,
+            0.0036307379469783137,
+            -4.895894824382588e-05,
+            -0.008129811261078888,
+            -0.013655410823646597,
+            0.018829640339982075,
+            -0.007579460691893014,
+            0.016191139169828096,
+            -0.004984562798832502,
+            -0.011846989328030879,
+            0.004110855640416496,
+            0.0004775146965349116,
+            -0.014308885325290177,
+            0.002778951512640393,
+            -0.003583348080466357,
+            0.0017351729844533707,
+            0.006444402813802743,
+            -0.004945263366121632,
+            0.003803172980003619,
+            -0.008178062077842394,
+            0.02700102856098108,
+            0.015477045085093356,
+            0.005818362056942523,
+            -0.00876269298951217,
+            -0.024697543974427114,
+            0.008183329610246195,
+            0.00518473714350693,
+            0.018661120751406378,
+            0.014464207183253898,
+            0.0019868192486975638,
+            0.009827229842256728,
+            -0.013395371817208287,
+            -0.0022376892192351944,
+            0.009711801873474911,
+            0.017963581533215926,
+            -0.0059700736266576305,
+            0.019169874693002874,
+            -0.010316234316657931,
+            -0.0008125759237295765,
+            0.004003428870809722,
+            0.017109043212624506,
+            -0.004590883969173384,
+            0.013958072401225709
+          ],
+          "title": "F Pseudo Code for Switch Transformers 33",
+          "keyphrases": [
+            "Switch Transformers",
+            "Mixture-of-Experts",
+            "sparsely-activated expert model",
+            "natural language tasks",
+            "pre-training and fine-tuning"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": true,
+      "properties": {
+        "cosine_similarity": 0.8019698552544312
+      }
+    },
+    {
+      "id": "0923a51f-8a23-40b9-8166-e9197da04d06",
+      "type": "cosine_similarity",
+      "source": {
+        "id": "14339b1f-53f8-479b-9da7-9430b174b9ee",
+        "properties": {
+          "page_content": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer\n\nNoam Shazeer1, Azalia Mirhoseini∗†1, Krzysztof Maziarz∗2, Andy Davis1, Quoc Le1, Geoffrey Hinton1and Jeff Dean1 1Google Brain, {noam,azalia,andydavis,qvl,geoffhinton,jeff}@google.com 2Jagiellonian University, Cracow, krzysztof.maziarz@student.uj.edu.pl\n\nAbstract\n\nThe capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost.\n\n1 Introduction And Related Work 1.1 Conditional Computation\n\nExploiting scale in both training data and model size has been central to the success of deep learning. When datasets are sufficiently large, increasing the capacity (number of parameters) of neural networks can give much better prediction accuracy. This has been shown in domains such as text (Sutskever et al., 2014; Bahdanau et al., 2014; Jozefowicz et al., 2016; Wu et al., 2016), images (Krizhevsky et al., 2012; Le et al., 2012), and audio (Hinton et al., 2012; Amodei et al., 2015). For typical deep learning models, where the entire model is activated for every example, this leads to a roughly quadratic blow-up in training costs, as both the model size and the number of training examples increase. Unfortunately, the advances in computing power and distributed computation fall short of meeting such demand.\n\nVarious forms of conditional computation have been proposed as a way to increase model capacity without a proportional increase in computational costs (Davis & Arel, 2013; Bengio et al., 2013; Eigen et al., 2013; Ludovic Denoyer, 2014; Cho & Bengio, 2014; Bengio et al., 2015; Almahairi et al., 2015). In these schemes, large parts of a network are active or inactive on a per-example basis. The gating decisions may be binary or sparse and continuous, stochastic or deterministic. Various forms of reinforcement learning and back-propagation are proposed for trarining the gating decisions.\n\nWhile these ideas are promising in theory, no work to date has yet demonstrated massive improvements in model capacity, training time, or model quality. We blame this on a combination of the following challenges:\n\nModern computing devices, especially GPUs, are much faster at arithmetic than at branching. Most of the works above recognize this and propose turning on/off large chunks of the network with each gating decision.\n\nLarge batch sizes are critical for performance, as they amortize the costs of parameter transfers and updates. Conditional computation reduces the batch sizes for the conditionally active chunks of the network.\n\nNetwork bandwidth can be a bottleneck. A cluster of GPUs may have computational power thousands of times greater than the aggregate inter-device network bandwidth. To be computationally efficient, the relative computational versus network demands of an algorithm must exceed this ratio. Embedding layers, which can be seen as a form of conditional computation, are handicapped by this very problem. Since the embeddings generally need to be sent across the network, the number of (example, parameter) interactions is limited by network bandwidth instead of computational capacity.\n\nDepending on the scheme, loss terms may be necessary to achieve the desired level of sparsity per-chunk and/or per example. Bengio et al. (2015) use three such terms. These issues can affect both model quality and load-balancing.\n\nModel capacity is most critical for very large data sets. The existing literature on conditional computation deals with relatively small image recognition data sets consisting of up to 600,000 images. It is hard to imagine that the labels of these images provide a sufficient signal to adequately train a model with millions, let alone billions of parameters. In this work, we for the first time address all of the above challenges and finally realize the promise of conditional computation. We obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets.\n\n1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1). All parts of the network are trained jointly by back-propagation.\n\nWhile the introduced technique is generic, in this paper we focus on language modeling and machine translation tasks, which are known to benefit from very large models. In particular, we apply a MoE convolutionally between stacked LSTM layers (Hochreiter & Schmidhuber, 1997), as in Figure 1.\n\nThe MoE is called once for each position in the text, selecting a potentially different combination of experts at each position. The different experts tend to become highly specialized based on syntax and semantics (see Appendix E Table 9). On both language modeling and machine translation benchmarks, we improve on best published results at a fraction of the computational cost.\n\n1.3 Related Work On Mixtures Of Experts\n\nSince its introduction more than two decades ago (Jacobs et al., 1991; Jordan & Jacobs, 1994), the mixture-of-experts approach has been the subject of much research. Different types of expert architectures hae been proposed such as SVMs (Collobert et al., 2002), Gaussian Processes (Tresp, 2001; Theis & Bethge, 2015; Deisenroth & Ng, 2015), Dirichlet Processes (Shahbaba & Neal, 2009), and deep networks. Other work has focused on different expert configurations such as a hierarchical structure (Yao et al., 2009), infinite numbers of experts (Rasmussen & Ghahramani, 2002), and adding experts sequentially (Aljundi et al., 2016). Garmash & Monz (2016) suggest an ensemble model in the format of mixture of experts for machine translation. The gating network is trained on a pre-trained ensemble NMT model. The works above concern top-level mixtures of experts. The mixture of experts is the whole model. Eigen et al. (2013) introduce the idea of using multiple MoEs with their own gating networks as parts of a deep model. It is intuitive that the latter approach is more powerful, since complex problems may contain many sub-problems each requiring different experts. They also allude in their conclusion to the potential to introduce sparsity, turning MoEs into a vehicle for computational computation.\n\nOur work builds on this use of MoEs as a general purpose neural network component. While Eigen et al. (2013) uses two stacked MoEs allowing for two sets of gating decisions, our convolutional application of the MoE allows for different gating decisions at each position in the text. We also realize sparse gating and demonstrate its use as a practical way to massively increase model capacity.\n\n2 The Structure Of The Mixture-Of-Experts Layer\n\nThe Mixture-of-Experts (MoE) layer consists of a set of n \"expert networks\" E1, · · · , En, and a \"gating network\" G whose output is a sparse n-dimensional vector. Figure 1 shows an overview of the MoE module. The experts are themselves neural networks, each with their own parameters.\n\nAlthough in principle we only require that the experts accept the same sized inputs and produce the same-sized outputs, in our initial investigations in this paper, we restrict ourselves to the case where the models are feed-forward networks with identical architectures, but with separate parameters.\n\nLet us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows:\n\n$$y=\\sum_{i=1}^{n}G(x){i}E{i}(x)$$ $$(\\mathbf{l})$$ G(x)iEi(x) (1) We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example. If the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network. In the following we focus on ordinary MoEs. We provide more details on hierarchical MoEs in Appendix B. Our implementation is related to other models of conditional computation. A MoE whose experts are simple weight matrices is similar to the parameterized weight matrix proposed in (Cho & Bengio, 2014). A MoE whose experts have one hidden layer is similar to the block-wise dropout described in (Bengio et al., 2015), where the dropped-out layer is sandwiched between fully-activated layers.\n\n2.1 Gating Network\n\nSoftmax Gating: A simple choice of non-sparse gating function (Jordan & Jacobs, 1994) is to multiply the input by a trainable weight matrix Wg and then apply the Sof tmax function.\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ $\\eqref{eq:walpha}$. Gσ(x) = Sof tmax(x · Wg) (2) Noisy Top-K Gating: We add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to −∞ (which causes the corresponding gate values to equal 0). The sparsity serves to save computation, as described above. While this form of sparsity creates some theoretically scary discontinuities in the output of gating function, we have not yet observed this to be a problem in practice. The noise term helps with load balancing, as will be discussed in Appendix A. The amount of noise per component is controlled by a second trainable weight matrix Wnoise.\n\n$$G(x)=S o f t m a x(K e e p T o p K(H(x),k))$$ $$(4)$$ $$(S)$$ G(x) = Sof tmax(KeepT opK(H(x), k)) (3) $$H(x){i}=(x\\cdot W{g}){i}+S t a n d a r d N o r m a l()\\cdot S o f t p l u s((x\\cdot W{n o i s e}){i})$$ $KeepTopK(v,k){i}=\\begin{cases}v_{i}&\\text{if}v_{i}\\text{is in the top}k\\text{elements of}v.\\ -\\infty&\\text{otherwise.}\\end{cases}$ Training the Gating Network We train the gating network by simple back-propagation, along with the rest of the model. If we choose k > 1, the gate values for the top k experts have nonzero derivatives with respect to the weights of the gating network. This type of occasionally-sensitive behavior is described in (Bengio et al., 2013) with respect to noisy rectifiers. Gradients also backpropagate through the gating network to its inputs. Our method differs here from (Bengio et al., 2015) who use boolean gates and a REINFORCE-style approach to train the gating network.\n\n3 Addressing Performance Challenges 3.1 The Shrinking Batch Problem\n\nOn modern CPUs and GPUs, large batch sizes are necessary for computational efficiency, so as to amortize the overhead of parameter loads and updates. If the gating network chooses k out of n experts for each example, then for a batch of b examples, each expert receives a much smaller batch of approximately kb n b examples. This causes a naive MoE implementation to become very inefficient as the number of experts increases. The solution to this shrinking batch problem is to make the original batch size as large as possible. However, batch size tends to be limited by the memory necessary to store activations between the forwards and backwards passes. We propose the following techniques for increasing the batch size: Mixing Data Parallelism and Model Parallelism: In a conventional distributed training setting, multiple copies of the model on different devices asynchronously process distinct batches of data, and parameters are synchronized through a set of parameter servers. In our technique, these different batches run synchronously so that they can be combined for the MoE layer. We distribute the standard layers of the model and the gating network according to conventional data-parallel schemes, but keep only one shared copy of each expert. Each expert in the MoE layer receives a combined batch consisting of the relevant examples from all of the data-parallel input batches. The same set of devices function as data-parallel replicas (for the standard layers and the gating networks) and as model-parallel shards (each hosting a subset of the experts). If the model is distributed over d devices, and each device processes a batch of size b, each expert receives a batch of approximately kbd nexamples. Thus, we achieve a factor of d improvement in expert batch size.\n\nIn the case of a hierarchical MoE (Section B), the primary gating network employs data parallelism, and the secondary MoEs employ model parallelism. Each secondary MoE resides on one device.\n\nThis technique allows us to increase the number of experts (and hence the number of parameters) by proportionally increasing the number of devices in the training cluster. The total batch size increases, keeping the batch size per expert constant. The memory and bandwidth requirements per device also remain constant, as do the step times, as does the amount of time necessary to process a number of training examples equal to the number of parameters in the model. It is our goal to train a trillionparameter model on a trillion-word corpus. We have not scaled our systems this far as of the writing of this paper, but it should be possible by adding more hardware.\n\nTaking Advantage of Convolutionality: In our language models, we apply the same MoE to each time step of the previous layer. If we wait for the previous layer to finish, we can apply the MoE to all the time steps together as one big batch. Doing so increases the size of the input batch to the MoE layer by a factor of the number of unrolled time steps.\n\nIncreasing Batch Size for a Recurrent MoE: We suspect that even more powerful models may involve applying a MoE recurrently. For example, the weight matrices of a LSTM or other RNN could be replaced by a MoE. Sadly, such models break the convolutional trick from the last paragraph, since the input to the MoE at one timestep depends on the output of the MoE at the previous timestep. Gruslys et al. (2016) describe a technique for drastically reducing the number of stored activations in an unrolled RNN, at the cost of recomputing forward activations. This would allow for a large increase in batch size.\n\n3.2 Network Bandwidth\n\nAnother major performance concern in distributed computing is network bandwidth. Since the experts are stationary (see above) and the number of gating parameters is small, most of the communication involves sending the inputs and outputs of the experts across the network. To maintain computational efficiency, the ratio of an expert's computation to the size of its input and output must exceed the ratio of computational to network capacity of the computing device. For GPUs, this may be thousands to one. In our experiments, we use experts with one hidden layer containing thousands of RELU-activated units. Since the weight matrices in the expert have sizes input_size×hidden_size and hidden_size × output_size, the ratio of computation to input and output is equal to the size of the hidden layer. Conveniently, we can increase computational efficiency simply by using a larger hidden layer, or more hidden layers.\n\n4 Balancing Expert Utilization\n\nWe have observed that the gating network tends to converge to a state where it always produces large weights for the same few experts. This imbalance is self-reinforcing, as the favored experts are trained more rapidly and thus are selected even more by the gating network. Eigen et al. (2013) describe the same phenomenon, and use a hard constraint at the beginning of training to avoid this local minimum. Bengio et al. (2015) include a soft constraint on the batch-wise average of each gate.1 We take a soft constraint approach. We define the importance of an expert relative to a batch of training examples to be the batchwise sum of the gate values for that expert. We define an additional loss Limportance, which is added to the overall loss function for the model. This loss is equal to the square of the coefficient of variation of the set of importance values, multiplied by a hand-tuned scaling factor wimportance. This additional loss encourages all experts to have equal importance.\n\n$$I m p o r t a n c e(X)=\\sum_{x\\in X}G(x)$$ $$L_{importance}(X)=w_{importance}\\cdot CV(Importance(X))^{2}\\tag{7}$$ $$(6)$$\n\nWhile this loss function can ensure equal importance, experts may still receive very different numbers of examples. For example, one expert may receive a few examples with large weights, and another may receive many examples with small weights. This can cause memory and performance problems on distributed hardware. To solve this problem, we introduce a second loss function, Lload , which ensures balanced loads. Appendix A contains the definition of this function, along with experimental results.\n\n5 Experiments 5.1 1 Billion Word Language Modeling Benchmark\n\nDataset: This dataset, introduced by (Chelba et al., 2013) consists of shuffled unique sentences from news articles, totaling approximately 829 million words, with a vocabulary of 793,471 words.\n\nPrevious State-of-the-Art: The best previously published results (Jozefowicz et al., 2016) use models consisting of one or more stacked Long Short-Term Memory (LSTM) layers (Hochreiter & Schmidhuber, 1997; Gers et al., 2000). The number of parameters in the LSTM layers of these models vary from 2 million to 151 million. Quality increases greatly with parameter count, as do computational costs. Results for these models form the top line of Figure 2-right. MoE Models: Our models consist of two stacked LSTM layers with a MoE layer between them (see Figure 1). We vary the sizes of the layers and the number of experts. For full details on model architecture, training regimen, additional baselines and results, see Appendix C.\n\nLow Computation, Varied Capacity: To investigate the effects of adding capacity, we trained\n\na series of MoE models all with roughly equal computational costs: about 8 million multiply-andadds per training example per timestep in the forwards pass, excluding the softmax layer. We call this metric (ops/timestep). We trained models with flat MoEs containing 4, 32, and 256 experts, and models with hierarchical MoEs containing 256, 1024, and 4096 experts. Each expert had about 1 million parameters. For all the MoE layers, 4 experts were active per input.\n\nThe results of these models are shown in Figure 2-left. The model with 4 always-active experts performed (unsurprisingly) similarly to the computationally-matched baseline models, while the largest of the models (4096 experts) achieved an impressive 24% lower perplexity on the test set. Figure 2: Model comparison on 1-Billion-Word Language-Modeling Benchmark. On the left, we plot test perplexity as a function of model capacity for models with similar computational budgets of approximately 8-million-ops-per-timestep. On the right, we plot test perplexity as a function of computational budget. The top line represents the LSTM models from (Jozefowicz et al., 2016).\n\nThe bottom line represents 4-billion parameter MoE models with different computational budgets.\n\nVaried Computation, High Capacity: In addition to the largest model from the previous section, we trained two more MoE models with similarly high capacity (4 billion parameters), but higher computation budgets. These models had larger LSTMs, and fewer but larger and experts. Details\n\nTest Test #Parameters ops/timestep Training TFLOPS Perplexity Perplexity excluding embedding Time /GPU 10 epochs 100 epochs and softmax layers 10 epochs Best Published Results 34.7 30.6 151 million 151 million 59 hours, 32 k40s 1.09 Low-Budget MoE Model 34.1 4303 million 8.9 million 15 hours, 16 k40s 0.74 Medium-Budget MoE Model 31.3 4313 million 33.8 million 17 hours, 32 k40s 1.22 High-Budget MoE Model 28.0 4371 million 142.7 million 47 hours, 32 k40s 1.56\n\nTable 1: Summary of high-capacity MoE-augmented models with varying computational budgets, vs. best previously published results (Jozefowicz et al., 2016). Details in Appendix C. can be found in Appendix C.2. Results of these three models form the bottom line of Figure 2-right.\n\nTable 1 compares the results of these models to the best previously-published result on this dataset .\n\nEven the fastest of these models beats the best published result (when controlling for the number of training epochs), despite requiring only 6% of the computation.\n\nComputational Efficiency: We trained our models using TensorFlow (Abadi et al., 2016) on clusters containing 16-32 Tesla K40 GPUs. For each of our models, we determine computational efficiency in TFLOPS/GPU by dividing the number of floating point operations required to process one training batch by the observed step time and the number of GPUs in the cluster. The operation counts used here are higher than the ones we report in our ops/timestep numbers in that we include the backwards pass, we include the importance-sampling-based training of the softmax layer, and we count a multiply-and-add as two separate operations. For all of our MoE models, the floating point operations involved in the experts represent between 37% and 46% of the total.\n\nFor our baseline models wtih no MoE, observed computational efficiency ranged from 1.07-1.29 TFLOPS/GPU. For our low-computation MoE models, computation efficiency ranged from 0.740.90 TFLOPS/GPU, except for the 4-expert model which did not make full use of the available parallelism. Our highest-computation MoE model was more efficient at 1.56 TFLOPS/GPU, likely due to the larger matrices. These numbers represent a significant fraction of the theoretical maximum of 4.29 TFLOPS/GPU claimed by NVIDIA. Detailed results are in Appendix C, Table 7.\n\n5.2 100 BILLION WORD GOOGLE NEWS CORPUS\n\nOn the 1-billion-word corpus, adding additional capacity seems to produce diminishing returns as the number of parameters in the MoE layer exceeds 1 billion, as can be seen in Figure 2-left. We hypothesized that for a larger training set, even higher capacities would produce significant quality improvements. We constructed a similar training set consisting of shuffled unique sentences from Google's internal news corpus, totalling roughly 100 billion words. Similarly to the previous section, we tested a series of models with similar computational costs of about 8 million ops/timestep. In addition to a baseline LSTM model, we trained models augmented with MoE layers containing 32, 256, 1024, 4096, 16384, 65536, and 131072 experts. This corresponds to up to 137 billion parameters in the MoE layer. Details on architecture, training, and results are given in Appendix D.\n\nResults: Figure 3 shows test perplexity as a function of capacity after training on 10 billion words (top line) and 100 billion words (bottom line). When training over the full 100 billion words, test perplexity improves significantly up to 65536 experts (68 billion parameters), dropping 39% lower than the computationally matched baseline, but degrades at 131072 experts, possibly a result of too much sparsity. The widening gap between the two lines demonstrates (unsurprisingly) that increased model capacity helps more on larger training sets. Even at 65536 experts (99.994% layer sparsity), computational efficiency for the model stays at a respectable 0.72 TFLOPS/GPU.\n\n5.3 Machine Translation (Single Language Pair)\n\nModel Architecture: Our model was a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decreased the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We inserted MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). Each MoE layer contained up to 2048 experts each with about two million parameters, adding a total of about 8 billion parameters to the models. Further details on model architecture, testing procedure and results can be found in Appendix E.\n\nDatasets: We benchmarked our method on the WMT'14 En→Fr and En→De corpora, whose training sets have 36M sentence pairs and 5M sentence pairs, respectively. The experimental protocols were also similar to those in (Wu et al., 2016): newstest2014 was used as the test set to compare against previous work (Luong et al., 2015a; Zhou et al., 2016; Wu et al., 2016), while the combination of newstest2012 and newstest2013 was used as the development set. We also tested the same model on a Google's Production English to French data.\n\nTable 2: Results on WMT'14 En→ Fr newstest2014 (bold values represent best results). Model Test Test ops/timenstep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.69 40.35 85M 8.7B 3 days/64 k40s MoE with 2048 Experts (longer training) 2.63 40.56 85M 8.7B 6 days/64 k40s GNMT (Wu et al., 2016) 2.79 39.22 214M 278M 6 days/96 k80s GNMT+RL (Wu et al., 2016) 2.96 39.92 214M 278M 6 days/96 k80s PBMT (Durrani et al., 2014) 37.0 LSTM (6-layer) (Luong et al., 2015b) 31.5 LSTM (6-layer+PosUnk) (Luong et al., 2015b) 33.1 DeepAtt (Zhou et al., 2016) 37.7 DeepAtt+PosUnk (Zhou et al., 2016) 39.2\n\nTable 3: Results on WMT'14 En → De newstest2014 (bold values represent best results). Model Test Test ops/timestep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 4.64 26.03 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 5.25 24.91 214M 278M 1 day/96 k80s GNMT +RL (Wu et al., 2016) 8.08 24.66 214M 278M 1 day/96 k80s PBMT (Durrani et al., 2014) 20.7 DeepAtt (Zhou et al., 2016) 20.6\n\nTable 4: Results on the Google Production En→ Fr dataset (bold values represent best results). Model Eval Eval Test Test ops/timestep Total Training Perplexity BLEU Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.60 37.27 2.69 36.57 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 2.78 35.80 2.87 35.56 214M 278M 6 days/96 k80s\n\nResults: Tables 2, 3, and 4 show the results of our largest models, compared with published results. Our approach achieved BLEU scores of 40.56 and 26.03 on the WMT'14 En→Fr and En→De benchmarks. As our models did not use RL refinement, these results constitute significant gains of 1.34 and 1.12 BLEU score on top of the strong baselines in (Wu et al., 2016). The perplexity scores are also better.2 On the Google Production dataset, our model achieved 1.01 higher test BLEU score even after training for only one sixth of the time.\n\n5.4 Multilingual Machine Translation\n\nDataset: (Johnson et al., 2016) train a single GNMT (Wu et al., 2016) model on a very large combined dataset of twelve language pairs. Results are somewhat worse than those for 12 separately trained single-pair GNMT models. This is not surprising, given that the twelve models have 12 times the capacity and twelve times the aggregate training of the one model. We repeat this experiment with a single MoE-augmented model. See Appendix E for details on model architecture.\n\nWe train our model on the same dataset as (Johnson et al., 2016) and process the same number of training examples (about 3 billion sentence pairs). Our training time was shorter due to the lower computational budget of our model.\n\nResults: Results for the single-pair GNMT models, the multilingual GNMT model and the multilingual MoE model are given in Table 5. The MoE model achieves 19% lower perplexity on the dev set than the multilingual GNMT model. On BLEU score, the MoE model significantly beats the multilingual GNMT model on 11 of the 12 language pairs (by as much as 5.84 points), and even beats the monolingual GNMT models on 8 of 12 language pairs. The poor performance on English → Korean seems to be a result of severe overtraining, as for the rarer language pairs a small number of real examples were highly oversampled in the training corpus.\n\nTable 5: Multilingual Machine Translation (bold values represent best results). GNMT-Mono GNMT-Multi MoE-Multi MoE-Multi vs. GNMT-Multi Parameters 278M / model 278M 8.7B ops/timestep 212M 212M 102M training time, hardware various 21 days, 96 k20s 12 days, 64 k40s Perplexity (dev) 4.14 3.35 -19% French → English Test BLEU 36.47 34.40 37.46 +3.06 German → English Test BLEU 31.77 31.17 34.80 +3.63 Japanese → English Test BLEU 23.41 21.62 25.91 +4.29 Korean → English Test BLEU 25.42 22.87 28.71 +5.84 Portuguese → English Test BLEU 44.40 42.53 46.13 +3.60 Spanish → English Test BLEU 38.00 36.04 39.39 +3.35 English → French Test BLEU 35.37 34.00 36.59 +2.59 English → German Test BLEU 26.43 23.15 24.53 +1.38 English → Japanese Test BLEU 23.66 21.10 22.78 +1.68 English → Korean Test BLEU 19.75 18.41 16.62 -1.79 English → Portuguese Test BLEU 38.40 37.35 37.90 +0.55 English → Spanish Test BLEU 34.50 34.25 36.21 +1.96\n\n6 Conclusion\n\nThis work is the first to demonstrate major wins from conditional computation in deep networks.\n\nWe carefully identified the design considerations and challenges of conditional computing and addressed them with a combination of algorithmic and engineering solutions. While we focused on text, conditional computation may help in other domains as well, provided sufficiently large training sets. We look forward to seeing many novel implementations and applications of conditional computation in the years to come.\n\nAcknowledgments\n\nWe would like to thank all of the members of the Google Brain and Google Translate teams who helped us with this project, in particular Zhifeng Chen, Yonghui Wu, and Melvin Johnson. Thanks also to our anonymous ICLR reviewers for the helpful suggestions on making this paper better.\n\n2Reported perplexities relative to the tokenization used by both our models and GNMT.\n\nReferences\n\nMartín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Gordon Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda B. Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. CoRR, abs/1603.04467, 2016. URL http://arxiv.org/abs/1603.04467.\n\nRahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. CoRR, abs/1611.06194, 2016. URL http://arxiv.org/abs/1611.\n\n06194.\n\nA. Almahairi, N. Ballas, T. Cooijmans, Y. Zheng, H. Larochelle, and A. Courville. Dynamic Capacity Networks. ArXiv e-prints, November 2015.\n\nDario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan, Christopher Fougner, Tony Han, Awni Y. Hannun, Billy Jun, Patrick LeGresley, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595, 2015. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.\n\nEmmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297, 2015.\n\nYoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.\n\nCiprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling.\n\narXiv preprint arXiv:1312.3005, 2013.\n\nK. Cho and Y. Bengio. Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning. ArXiv e-prints, June 2014. Ronan Collobert, Samy Bengio, and Yoshua Bengio. A parallel mixture of SVMs for very large scale problems. Neural Computing, 2002.\n\nAndrew Davis and Itamar Arel. Low-rank approximations for conditional feedforward computation in deep neural networks. arXiv preprint arXiv:1312.4461, 2013. Marc Peter Deisenroth and Jun Wei Ng. Distributed Gaussian processes. In ICML, 2015.\n\nJohn Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization, 2010.\n\nNadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. Edinburgh's phrase-based machine translation systems for wmt-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation, 2014.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013. Ekaterina Garmash and Christof Monz. Ensemble learning for multi-source neural machine translation. In staff.science.uva.nl/c.monz, 2016.\n\nFelix A. Gers, Jürgen A. Schmidhuber, and Fred A. Cummins. Learning to forget: Continual prediction with lstm. Neural Computation, 2000.\n\nAudrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. Memory-efficient backpropagation through time. CoRR, abs/1606.03401, 2016. URL http://arxiv.org/ abs/1606.03401. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2015.\n\nGeoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 2012. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.\n\nSergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.\n\nRobert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computing, 1991.\n\nMelvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation.\n\nCoRR, abs/1611.04558, 2016. URL http://arxiv.org/abs/1611.04558.\n\nMichael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm.\n\nNeural Computing, 1994. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.\n\nDiederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.\n\nReinhard Kneser and Hermann. Ney. Improved backingoff for m-gram language modeling., 1995.\n\nAlex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.\n\nQuoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeffrey Dean, and Andrew Y. Ng. Building high-level features using large scale unsupervised learning. In ICML, 2012. Patrick Gallinari Ludovic Denoyer. Deep sequential neural network. arXiv preprint arXiv:1410.0510, 2014.\n\nMinh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attentionbased neural machine translation. EMNLP, 2015a.\n\nMinh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the rare word problem in neural machine translation. ACL, 2015b.\n\nCarl Edward Rasmussen and Zoubin Ghahramani. Infinite mixtures of Gaussian process experts.\n\nNIPS, 2002.\n\nHasim Sak, Andrew W Senior, and Françoise Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH, pp. 338–342, 2014.\n\nMike Schuster and Kaisuke Nakajima. Japanese and Korean voice search. ICASSP, 2012.\n\nBabak Shahbaba and Radford Neal. Nonlinear models using dirichlet process mixtures. JMLR, 2009.\n\nIlya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks.\n\nIn NIPS, 2014.\n\nLucas Theis and Matthias Bethge. Generative image modeling using spatial LSTMs. In NIPS, 2015. Volker Tresp. Mixtures of Gaussian Processes. In NIPS, 2001.\n\nYonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.\n\nBangpeng Yao, Dirk Walther, Diane Beck, and Li Fei-fei. Hierarchical mixture of classification experts uncovers interactions between brain regions. In NIPS. 2009.\n\nWojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization.\n\narXiv preprint arXiv:1409.2329, 2014.\n\nJie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with fast-forward connections for neural machine translation. arXiv preprint arXiv:1606.04199, 2016.\n\nAppendices A Load-Balancing Loss\n\nAs discussed in section 4, for load-balancing purposes, we want to define an additional loss function to encourage experts to receive roughly equal numbers of training examples. Unfortunately, the number of examples received by an expert is a discrete quantity, so it can not be used in backpropagation. Instead, we define a smooth estimator Load(X) of the number of examples assigned to each expert for a batch X of inputs. The smoothness allows us to back-propagate gradients through the estimator. This is the purpose of the noise term in the gating function. We define P(x, i) as the probability that G(x)iis nonzero, given a new random choice of noise on element i, but keeping the already-sampled choices of noise on the other elements. To compute P(x, i), we note that the G(x)iis nonzero if and only if H(x)iis greater than the k th-greatest element of H(x) excluding itself. The probability works out to be:\n\n$$P(x,i)=P r\\Big((x\\cdot W_{g}){i}+S t a n d x N o r a l()\\cdot S o f t p u s((x\\cdot W{n o i s e})_{i})$$ $$>k t h_e x c l u d i n(H(x),k,i)\\Big)$$ $$({\\boldsymbol{8}})$$\n\n$$(9)$$ $$(10)$$ (8) Where kth_excluding(v, k, i) means the kth highest component of v, excluding component i. Simplifying, we get:\n\n$$P(x,i)=\\Phi{\\Big(}{\\frac{(x\\cdot W_{g}){i}-k t h_c x c l u d i n g(H(x),k,i)}{S o f t p l u s((x\\cdot W{n o i s e})_{i})}}{\\Big)}$$\n\nWhere Φ is the CDF of the standard normal distribution.\n\n$$L o a d(X){i}=\\sum{x\\in X}P(x,i)$$\n\n$$(11)$$ P(x, i) (10) We can now define the load loss to be the square of the coefficient of variation of the load vector, multiplied by a hand-tuned scaling factor wload.\n\n$$L_{l o a d}(X)=w_{l o a d}\\cdot C V(L o a d(X))^{2}$$ Lload(X) = wload · CV (Load(X))2(11) Initial Load Imbalance: To avoid out-of-memory errors, we need to initialize the network in a state of approximately equal expert load (since the soft constraints need some time to work). To accomplish this, we initialize the matrices Wg and Wnoise to all zeros, which yields no signal and some noise.\n\nExperiments: We trained a set of models with identical architecture (the MoE-256 model described in Appendix C), using different values of wimportance and wload. We trained each model for 10 epochs, then measured perplexity on the test set. We also measured the coefficients of variation in Importance and Load, as well as ratio of the load on the most overloaded expert to the average load. This last value is significant for load balancing purposes on distributed hardware. All of these metrics were averaged over several training batches.\n\nwimportance wload Test Perplexity CV (Importance(X)) CV (Load(X)) max(Load(X)) mean(Load(X)) 0.0 0.0 39.8 3.04 3.01 17.80 0.2 0.0 35.6 0.06 0.17 1.47 0.0 0.2 35.7 0.22 0.04 1.15 0.1 0.1 35.6 0.06 0.05 1.14 0.01 0.01 35.7 0.48 0.11 1.37 1.0 1.0 35.7 0.03 0.02 1.07\n\nTable 6: Experiments with different combinations of losses.\n\n13 Results: Results are reported in Table 6. All the combinations containing at least one the two losses led to very similar model quality, where having no loss was much worse. Models with higher values of wload had lower loads on the most overloaded expert.\n\nB Hierachical Mixture Of Experts\n\nIf the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network.3If the hierarchical MoE consists of a groups of b experts each, we denote the primary gating network by Gprimary, the secondary gating networks by (G1, G2..Ga), and the expert networks by (E0,0, E0,1..Ea,b). The output of the MoE is given by:\n\n$$y_{H}=\\sum_{i=1}^{a}\\sum_{j=1}^{b}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}\\cdot E{i,j}(x)$$ $$(12)$$ $$(13)$$ $$(14)$$\n\nOur metrics of expert utilization change to the following:\n\n$$I m p o r t a n c e_{H}(X){i,j}=\\sum{x\\in X}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}$$ $$L o a d{H}(X){i,j}={\\frac{L o a d{p r i m a r y}(X){i}\\cdot L o a d{i}(X^{(i)})_{j}}{|X^{(i)}|}}$$\n\nLoadprimary and Loadi deonte the Load functions for the primary gating network and i th secondary gating network respectively. X(i) denotes the subset of X for which Gprimary(x)i > 0.\n\nIt would seem simpler to let LoadH(X)i,j = Loadi(Xi)j , but this would not have a gradient with respect to the primary gating network, so we use the formulation above.\n\nC 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS C.1 8-MILLION-OPERATIONS-PER-TIMESTEP MODELS Model Architecture: Our model consists of five layers: a word embedding layer, a recurrent Long Short-Term Memory (LSTM) layer (Hochreiter & Schmidhuber, 1997; Gers et al., 2000), a MoE layer, a second LSTM layer, and a softmax layer. The dimensionality of the embedding layer, the number of units in each LSTM layer, and the input and output dimensionality of the MoE layer are all equal to 512. For every layer other than the softmax, we apply drouput (Zaremba et al., 2014) to the layer output, dropping each activation with probability DropP rob, otherwise dividing by (1 − DropP rob). After dropout, the output of the previous layer is added to the layer output.\n\nThis residual connection encourages gradient flow (He et al., 2015).\n\nMoE Layer Architecture: Each expert in the MoE layer is a feed forward network with one ReLU-activated hidden layer of size 1024 and an output layer of size 512. Thus, each expert contains [512 ∗ 1024] + [1024 ∗ 512] = 1M parameters. The output of the MoE layer is passed through a sigmoid function before dropout. We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts.\n\nWe call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers. Thus, each example is processed by exactly 4 experts for a total of 4M ops/timestep. The two LSTM layers contribute 2M ops/timestep each for the desired total of 8M.\n\n3 We have not found the need for deeper hierarchies.\n\nComputationally-Matched Baselines: The MoE-4 model does not employ sparsity, since all 4 experts are always used. In addition, we trained four more computationally-matched baseline models with no sparsity:\n\nMoE-1-Wide: The MoE layer consists of a single \"expert\" containing one ReLU-activated hidden layer of size 4096.\n\nMoE-1-Deep: The MoE layer consists of a single \"expert\" containing four ReLU-activated hidden layers, each with size 1024.\n\n4xLSTM-512: We replace the MoE layer with two additional 512-unit LSTM layers.\n\nLSTM-2048-512: The model contains one 2048-unit LSTM layer (and no MoE). The output of the LSTM is projected down to 512 dimensions (Sak et al., 2014). The next timestep of the LSTM receives the projected output. This is identical to one of the models published in (Jozefowicz et al., 2016). We re-ran it to account for differences in training regimen, and obtained results very similar to the published ones. Training: The models were trained on a cluster of 16 K40 GPUs using the synchronous method described in Section 3. Each batch consisted of a set of sentences totaling roughly 300,000 words. In the interest of time, we limited training to 10 epochs, (27,000 steps). Training took 12-16 hours for all models, except for MoE-4, which took 18 hours (since all the expert computation was performed on only 4 of 16 GPUs). We used the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 1000 training steps, and decreased after that so as to be proportional to the inverse square root of the step number. The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016). For each model, we performed a hyper-parmeter search to find the best dropout probability, in increments of 0.1.\n\nTo ensure balanced expert utilization we set wimportance = 0.1 and wload = 0.1, as described in Section 4 and Appendix A.\n\nResults: We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol. Results are reported in Table 7. For each model, we report the test perplexity, the computational budget, the parameter counts, the value of DropP rob, and the computational efficiency.\n\nModel Test Test ops/timestep #Params excluding Total Drop- TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params P rob per GPU 10 epochs (final) (millions) (billions) (observed) Kneser-Ney 5-gram* 67.6 0.00001 1.8 LSTM-512-512* 54.1 2.4 2.4 0.8 0.1 LSTM-1024-512* 48.2 4.7 4.7 0.8 0.1 LSTM-2048-512* 45.0 43.7 9.4 9.4 0.8 0.1 0.61 LSTM-2048-512 44.7 9.4 9.4 0.8 0.1 1.21 4xLSTM-512 46.0 8.4 8.4 0.8 0.1 1.07 MoE-1-Wide 46.1 8.4 8.4 0.8 0.1 1.29 MoE-1-Deep 45.7 8.4 8.4 0.8 0.1 1.29 MoE-4 45.0 8.4 8.4 0.8 0.1 0.52 MoE-32 39.7 8.4 37.8 0.9 0.1 0.87 MoE-256 35.7 8.6 272.9 1.1 0.1 0.81 MoE-256-h 36.0 8.4 272.9 1.1 0.1 0.89 MoE-1024-h 34.6 8.5 1079.0 1.9 0.2 0.90 MoE-4096-h 34.1 8.9 4303.4 5.1 0.2 0.74 2xLSTM-8192-1024* 34.7 30.6 151.0 151.0 1.8 0.25 1.09 MoE-34M 31.3 33.8 4313.9 6.0 0.3 1.22 MoE-143M 28.0 142.7 4371.1 6.0 0.4 1.56\n\nC.2 More Expensive Models\n\nWe ran two additional models (MoE-34M and MoE-143M) to investigate the effects of adding more computation in the presence of a large MoE layer. These models have computation budgets of 34M and 143M ops/timestep. Similar to the models above, these models use a MoE layer between two LSTM layers. The dimensionality of the embedding layer, and the input and output dimensionality of the MoE layer are set to 1024 instead of 512. For MoE-34M, the LSTM layers have 1024 units.\n\nFor MoE-143M, the LSTM layers have 4096 units and an output projection of size 1024 (Sak et al., 2014). MoE-34M uses a hierarchical MoE layer with 1024 experts, each with a hidden layer of size 2048. MoE-143M uses a hierarchical MoE layer with 256 experts, each with a hidden layer of size 8192. Both models have 4B parameters in the MoE layers. We searched for the best DropP rob for each model, and trained each model for 10 epochs.\n\nThe two models achieved test perplexity of 31.3 and 28.0 respectively, showing that even in the presence of a large MoE, more computation is still useful. Results are reported at the bottom of Table 7. The larger of the two models has a similar computational budget to the best published model from the literature, and training times are similar. Comparing after 10 epochs, our model has a lower test perplexity by 18%.\n\nD 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS Model Architecture: The models are similar in structure to the 8-million-operations-per-timestep models described in the previous section. We vary the number of experts between models, using an ordinary MoE layer with 32 experts and hierarchical MoE layers with 256, 1024, 4096, 16384, 65536 and 131072 experts. For the hierarchical MoE layers, the first level branching factors are 32, 32, 64, 128, 256 and 256, respectively.\n\nTraining: Models are trained on a cluster of 32 Tesla K40 GPUs, except for the last two models, which are trained on clusters of 64 and 128 GPUs so as to have enough memory for all the parameters. For all models, training batch sizes are approximately 2.5 million words. Models are trained once-through over about 100 billion words. We implement several memory optimizations in order to fit up to 1 billion parameters per GPU. First, we do not store the activations of the hidden layers of the experts, but instead recompute them on the backwards pass. Secondly, we modify the optimizer on the expert parameters to require less auxiliary storage: The Adam optimizer (Kingma & Ba, 2015) keeps first and second moment estimates of the perparameter gradients. This triples the required memory. To avoid keeping a first-moment estimator, we set β1 = 0. To reduce the size of the second moment estimator, we replace it with a factored approximation. For a matrix of parameters, instead of maintaining a full matrix of second-moment estimators, we maintain vectors of row-wise and column-wise averages of that matrix. At each step, the matrix of estimators is taken to be the outer product of those two vectors divided by the mean of either one. This technique could similarly be applied to Adagrad (Duchi et al., 2010).\n\nModel Test Test ops/timestep #Params excluding Total TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params per GPU .1 epochs 1 epoch (millions) (billions) (observed) Kneser-Ney 5-gram 67.1 45.3 0.00001 76.0 4xLSTM-512 54.5 47.0 8.4 8.4 0.1 1.23 MoE-32 48.5 40.4 8.4 37.8 0.1 0.83 MoE-256-h 42.8 35.3 8.4 272.9 0.4 1.11 MoE-1024-h 40.3 32.7 8.5 1079.0 1.2 1.14 MoE-4096-h 38.9 30.9 8.6 4303.4 4.4 1.07 MoE-16384-h 38.2 29.7 8.8 17201.0 17.3 0.96 MoE-65536-h 38.2 28.9 9.2 68791.0 68.9 0.72 MoE-131072-h 39.8 29.2 9.7 137577.6 137.7 0.30\n\nResults: We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8. Perplexity after 100 billion training words is 39% lower for the 68-billion-parameter MoE model than for the baseline model. It is notable that the measured computational efficiency of the largest model (0.30 TFLOPS/GPU) is very low compared to the other models. This is likely a result of the fact that, for purposes of comparison to the other models, we did not increase the training batch size proportionally to the number of GPUs. For comparison, we include results for a computationally matched baseline model consisting of 4 LSTMs, and for an unpruned 5-gram model with Kneser-Ney smoothing (Kneser & Ney, 1995).4\n\nE Machine Translation - Experimental Details\n\nModel Architecture for Single Language Pair MoE Models: Our model is a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decrease the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We insert MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). We use an attention mechanism between the encoder and decoder, with the first decoder LSTM receiving output from and providing input for the attention 5. All of the layers in our model have input and output dimensionality of 512. Our LSTM layers have 2048 hidden units, with a 512-dimensional output projection. We add residual connections around all LSTM and MoE layers to encourage gradient flow (He et al., 2015). Similar to GNMT, to effectively deal with rare words, we used subword units (also known as \"wordpieces\") (Schuster & Nakajima, 2012) for inputs and outputs in our system.\n\nWe use a shared source and target vocabulary of 32K wordpieces. We also used the same beam search technique as proposed in (Wu et al., 2016).\n\nWe train models with different numbers of experts in the MoE layers. In addition to a baseline model with no MoE layers, we train models with flat MoE layers containing 32 experts, and models with hierarchical MoE layers containing 512 and 2048 experts. The flat MoE layers use k = 4 and the hierarchical MoE models use k = 2 at each level of the gating network. Thus, each input is processed by exactly 4 experts in each MoE layer. Each expert in the MoE layer is a feed forward network with one hidden layer of size 2048 and ReLU activation. Thus, each expert contains [512 ∗ 2048] + [2048 ∗ 512] = 2M parameters. The output of the MoE layer is passed through a sigmoid function. We use the strictly-balanced gating function described in Appendix F.\n\nModel Architecture for Multilingual MoE Model: We used the same model architecture as for the single-language-pair models, with the following exceptions: We used noisy-top-k gating as described in Section 2.1, not the scheme from Appendix F. The MoE layers in the encoder and decoder are non-hierarchical MoEs with n = 512 experts, and k = 2. Each expert has a larger hidden layer of size 8192. This doubles the amount of computation in the MoE layers, raising the computational budget of the entire model from 85M to 102M ops/timestep. Training: We trained our networks using the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 2000 training steps, held constant for an additional 8000 steps, and decreased after that so as to be proportional to the inverse square root of the step number. For the single-language-pair models, similarly to (Wu et al., 2016), we applied dropout (Zaremba et al., 2014) to the output of all embedding, LSTM and MoE layers, using DropP rob = 0.4. Training was done synchronously on a cluster of up to 64 GPUs as described in section 3. Each training batch consisted of a set of sentence pairs containing roughly 16000 words per GPU.\n\nTo ensure balanced expert utilization we set wimportance = 0.01 and wload = 0.01, as described in Section 4 and Appendix A. Metrics: We evaluated our models using the perplexity and the standard BLEU score metric. We reported tokenized BLEU score as computed by the multi-bleu.pl script, downloaded from the public implementation of Moses (on Github), which was also used in (Luong et al., 2015a). Results: Tables 2, 3 and 4 in Section 5.3 show comparisons of our results to other published methods. Figure 4 shows test perplexity as a function of number of words in the (training data's) source sentences processed for models with different numbers of experts. As can be seen from the Figure, as we increased the number of experts to approach 2048, the test perplexity of our model continued to improve.\n\nWe found that the experts indeed become highly specialized by syntax and/or semantics, as can be seen in Table 9. For example, one expert is used when the indefinite article \"a\" introduces the direct object in a verb phrase indicating importance or leadership.\n\nExpert 381 Expert 752 Expert 2004 ... with researchers , ... ... plays a core ... ... with rapidly growing ... ... to innovation . ... plays a critical ... ... under static conditions ... ... tics researchers . ... provides a legislative ... ... to swift ly ... ... the generation of ... ... play a leading ... ... to dras tically ... ... technology innovations is ... ... assume a leadership ... ... the rapid and ... ... technological innovations , ... ... plays a central ... ... the fast est ... ... support innovation throughout ... ... taken a leading ... ... the Quick Method ... ... role innovation will ... ... established a reconciliation ... ... rec urrent ) ... ... research scienti st ... ... played a vital ... ... provides quick access ... ... promoting innovation where ... ... have a central ... ... of volatile organic ... ... ... ...\n\nDue to some peculiarities in our infrastructure which have since been fixed, at the time we ran some of the machine translation experiments, our models ran faster if every expert received exactly the same batch size. To accommodate this, we used a different gating function which we describe below. Recall that we define the softmax gating function to be:\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ Gσ(x) = Sof tmax(x · Wg) (15) Sparse Gating (alternate formulation): To obtain a sparse gating vector, we multiply Gσ(x) component-wise with a sparse mask M(Gσ(x)) and normalize the output. The mask itself is a function of Gσ(x) and specifies which experts are assigned to each input example:\n\n$$(15)$$ $$G(x){i}=\\frac{G{\\sigma}(x){i}M(G{\\sigma}(x)){i}}{\\sum{j=1}^{n}G_{\\sigma}(x){j}M(G{\\sigma}(x))_{j}}\\tag{1}$$ $$(17)$$\n\n$$(16)$$\n\nTop-K Mask: To implement top-k gating in this formulation, we would let M(v) = T opK(v, k), where:\n\n$$T o p K(v,k){i}=\\begin{cases}1&{\\mathrm{if~}}v{i}{\\mathrm{~is~in~the~top~}}k{\\mathrm{~elements~of~}}v.\\ 0&{\\mathrm{otherwise.}}\\end{cases}$$ 0 otherwise. (17) Batchwise Mask: To force each expert to receive the exact same number of examples, we introduce an alternative mask function, Mbatchwise(X, m), which operates over batches of input vectors.\n\nInstead of keeping the top k values per example, we keep the top m values per expert across the training batch, where m = k|X| n, so that each example is sent to an average of k experts.\n\n$$M_{b a t c h w i s e}(X,m){j,i}=\\begin{cases}1&\\text{if}X{j,i}\\text{is in the top}m\\text{values for to expert}i\\ 0&\\text{otherwise}\\end{cases}$$\n\nAs our experiments suggest and also observed in (Ioffe & Szegedy, 2015), using a batchwise function during training (such as Mbatchwise) requires modifications to the inference when we may not have a large batch of examples. Our solution to this is to train a vector T of per-expert threshold values to approximate the effects of the batchwise mask. We use the following mask at inference time:\n\n$$(18)$$ $$M_{threshold}(x,T){i}=\\begin{cases}1&\\text{if}x{i}>T_{i}\\ 0&\\text{otherwise}\\end{cases}\\tag{1}$$ $$(19)$$\n\nTo learn the threshold values, we apply an additional loss at training time which is minimized when the batchwise mask and the threshold mask are identical.\n\n$$L_{batchwise}(X,T,m)=\\sum_{j=1}^{|X|}\\sum_{i=1}^{n}(M_{threshold}(x,T){i}-M{batchwise}(X,m){j,i})(X{j,i}-T_{i})\\tag{20}$$\n\nG ATTENTION FUNCTION The attention mechanism described in GNMT (Wu et al., 2016) involves a learned \"Attention Function\" A(xi, yj ) which takes a \"source vector\" xi and a \"target vector\" yj , and must be computed for every source time step i and target time step j. In GNMT, the attention function is implemented as a feed forward neural network with a hidden layer of size n. It can be expressed as:\n\n$$A_{G N M T}(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}t a n h((x_{i}U){d}+(y{j}W)_{d})$$\n\nWhere U and W are trainable weight matrices and V is a trainable weight vector.\n\nFor performance reasons, in our models, we used a slightly different attention function:\n\n$$A(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}tanh((x_{i}U){d})tanh((y{j}W)_{d})\\tag{22}$$\n\nWith our attention function, we can simultaneously compute the attention function on multiple source time steps and multiple target time steps using optimized matrix multiplications. We found little difference in quality between the two functions.\n\n$$(21)$$",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/1701.06538v1.md"
+          },
+          "headlines": {
+            "1 Introduction And Related Work": [
+              "1.1 Conditional Computation",
+              "1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER",
+              "1.3 Related Work On Mixtures Of Experts"
+            ],
+            "2 The Structure Of The Mixture-Of-Experts Layer": [
+              "2.1 Gating Network"
+            ],
+            "3 Addressing Performance Challenges": [
+              "3.1 The Shrinking Batch Problem",
+              "3.2 Network Bandwidth"
+            ],
+            "4 Balancing Expert Utilization": [],
+            "5 Experiments": [
+              "5.1 1 Billion Word Language Modeling Benchmark",
+              "5.2 100 BILLION WORD GOOGLE NEWS CORPUS",
+              "5.3 Machine Translation (Single Language Pair)",
+              "5.4 Multilingual Machine Translation"
+            ],
+            "6 Conclusion": [],
+            "Appendices": [
+              "A Load-Balancing Loss",
+              "B Hierachical Mixture Of Experts",
+              "C 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS",
+              "D 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS",
+              "E Machine Translation - Experimental Details"
+            ]
+          },
+          "summary": "The paper introduces a Sparsely-Gated Mixture-of-Experts (MoE) layer to enhance neural network capacity without a proportional increase in computation. This approach uses a trainable gating network to select a sparse combination of experts for each input, significantly improving model capacity and performance in language modeling and machine translation tasks. The MoE layer, with up to 137 billion parameters, achieves better results than state-of-the-art models at lower computational costs. The paper addresses challenges like batch size and network bandwidth, proposing solutions like mixing data and model parallelism. Experiments show that MoE models outperform traditional models in various benchmarks, demonstrating the potential of conditional computation in deep learning.",
+          "summary_embedding": [
+            -0.01644345559179783,
+            -0.016995344310998917,
+            -0.016372699290513992,
+            -0.014349107630550861,
+            0.02282555401325226,
+            0.045934125781059265,
+            -0.014646278694272041,
+            0.03563220426440239,
+            -0.004829026758670807,
+            0.005617944523692131,
+            -0.00932550523430109,
+            -0.008143897168338299,
+            0.005575491581112146,
+            0.01538213063031435,
+            -0.005653322208672762,
+            -0.020306676626205444,
+            -0.013832597061991692,
+            -0.023235931992530823,
+            0.028344441205263138,
+            -0.020080260932445526,
+            0.011823155917227268,
+            -0.020943470299243927,
+            -0.025868017226457596,
+            -0.016500059515237808,
+            0.025061409920454025,
+            0.024806693196296692,
+            0.02583971433341503,
+            0.003827843815088272,
+            -0.03492465242743492,
+            0.033651065081357956,
+            0.014674580655992031,
+            0.012488252483308315,
+            -0.031500112265348434,
+            -0.009162768721580505,
+            -0.05351905897259712,
+            -0.05227377265691757,
+            0.04143410921096802,
+            0.022245362401008606,
+            -0.016231190413236618,
+            0.023717066273093224,
+            0.017264213413000107,
+            0.005816058721393347,
+            -0.020632149651646614,
+            -0.020844414830207825,
+            0.003700484987348318,
+            -0.013103820383548737,
+            -0.01983969286084175,
+            -0.013528349809348583,
+            -0.01828308403491974,
+            -0.01577835902571678,
+            0.009551920928061008,
+            0.0002856731880456209,
+            0.0029186424799263477,
+            -0.003424540627747774,
+            -0.047830358147621155,
+            0.0248208437114954,
+            0.019853845238685608,
+            0.03492465242743492,
+            0.04633035510778427,
+            0.013011839240789413,
+            0.01516986545175314,
+            -0.014271277002990246,
+            -0.04021712392568588,
+            0.005497660953551531,
+            -0.00041656987741589546,
+            -0.005380915477871895,
+            -0.005819596350193024,
+            0.008575502783060074,
+            0.008724087849259377,
+            -0.01143400277942419,
+            -0.04202845320105553,
+            0.013952880166471004,
+            -0.020561393350362778,
+            0.00010143388499272987,
+            0.01354957651346922,
+            0.008292482234537601,
+            0.02677368000149727,
+            0.03195294365286827,
+            0.023971782997250557,
+            -0.008724087849259377,
+            -0.01340806670486927,
+            0.019924599677324295,
+            -0.0022676968947052956,
+            0.007018893025815487,
+            0.04570771008729935,
+            -0.04033033177256584,
+            0.02163686975836754,
+            -0.025726506486535072,
+            -0.015339677222073078,
+            -0.005349075887352228,
+            -0.022471778094768524,
+            0.02806142158806324,
+            -0.007344366051256657,
+            0.017801949754357338,
+            0.017094401642680168,
+            0.01594817079603672,
+            0.013330236077308655,
+            -0.001894464367069304,
+            -0.022938761860132217,
+            0.0018360915128141642,
+            -0.01541043259203434,
+            0.02114158496260643,
+            0.0015230007702484727,
+            0.005469358991831541,
+            0.027594437822699547,
+            0.031160488724708557,
+            -0.023490650579333305,
+            0.009502392262220383,
+            0.02422650158405304,
+            0.06967949122190475,
+            0.015424583107233047,
+            -0.006435164716094732,
+            -0.0024640418123453856,
+            0.04143410921096802,
+            0.0015513027319684625,
+            0.0038349193055182695,
+            -0.06826439499855042,
+            -0.03022652305662632,
+            -0.0011365016689524055,
+            -0.02254253253340721,
+            0.02227366529405117,
+            0.01002597901970148,
+            -0.009750034660100937,
+            -0.022457627579569817,
+            -0.04516997188329697,
+            0.05190584436058998,
+            -0.00545874610543251,
+            -0.016259491443634033,
+            -0.005554265342652798,
+            -0.05340585112571716,
+            0.00039136342820711434,
+            0.015042506158351898,
+            -0.018481198698282242,
+            -0.03670767694711685,
+            0.036141637712717056,
+            0.0007398316520266235,
+            0.03741522878408432,
+            0.012459950521588326,
+            -0.01708024926483631,
+            -0.020094411447644234,
+            0.022259512916207314,
+            -0.011639192700386047,
+            -0.0038313816767185926,
+            -0.027339721098542213,
+            0.0065236082300543785,
+            -0.028103873133659363,
+            -0.0014566679019480944,
+            0.06730213016271591,
+            -0.04816998168826103,
+            0.02946236915886402,
+            -0.029660483822226524,
+            0.023660462349653244,
+            -0.008575502783060074,
+            0.0278916098177433,
+            -0.010238244198262691,
+            0.005840822588652372,
+            -0.013457595370709896,
+            -0.0010586712742224336,
+            -0.020745357498526573,
+            0.0034210027661174536,
+            -0.016259491443634033,
+            -0.027750099077820778,
+            -0.007054270710796118,
+            0.031273696571588516,
+            0.01698119193315506,
+            0.025655752047896385,
+            0.025429336354136467,
+            0.01246702577918768,
+            -0.04935866594314575,
+            0.02595292218029499,
+            0.0018926954362541437,
+            0.013514199294149876,
+            0.013181651011109352,
+            -0.0010418669553473592,
+            0.025599148124456406,
+            0.012049571610987186,
+            0.025061409920454025,
+            0.037443529814481735,
+            0.0027682881336659193,
+            0.04737752676010132,
+            -0.015693452209234238,
+            -0.005292471963912249,
+            0.028528403490781784,
+            0.000505455827806145,
+            0.02465103194117546,
+            -0.001686621573753655,
+            0.013330236077308655,
+            0.000776978034991771,
+            0.03834919258952141,
+            -0.006339645478874445,
+            0.029518973082304,
+            0.021608568727970123,
+            0.028316138312220573,
+            -0.014759486541152,
+            0.023122724145650864,
+            -0.019754787907004356,
+            -0.02292460948228836,
+            0.035547297447919846,
+            0.004082561936229467,
+            0.01886327564716339,
+            -0.028457649052143097,
+            -0.030679354444146156,
+            0.015452885068953037,
+            -0.01855195313692093,
+            0.016556663438677788,
+            0.03639635816216469,
+            0.0008273909334093332,
+            -0.0025719432160258293,
+            0.026292547583580017,
+            -0.04245298355817795,
+            0.012856177985668182,
+            -0.01532552670687437,
+            -0.02855670638382435,
+            -0.019528372213244438,
+            -0.00875946506857872,
+            -0.0018095583654940128,
+            -0.013691086322069168,
+            -0.0037570889107882977,
+            -0.007563706487417221,
+            -0.02459442801773548,
+            0.00935380719602108,
+            0.013627407141029835,
+            0.009849091991782188,
+            -0.018665160983800888,
+            0.0019952901639044285,
+            -0.008257105015218258,
+            -0.0160047747194767,
+            -0.03752843663096428,
+            0.015268922783434391,
+            -0.0024888061452656984,
+            0.009191070683300495,
+            0.04078316316008568,
+            0.028485950082540512,
+            -0.005055442452430725,
+            0.0329718142747879,
+            -0.033707667142152786,
+            0.017915157601237297,
+            -0.008320784196257591,
+            0.009191070683300495,
+            0.002989397384226322,
+            -0.025542544201016426,
+            0.027424626052379608,
+            -0.03792466223239899,
+            -0.038151077926158905,
+            0.022995365783572197,
+            0.011858533136546612,
+            0.0017096169758588076,
+            -0.019386861473321915,
+            -0.017717044800519943,
+            0.0008499440737068653,
+            0.002752368338406086,
+            -0.018834972754120827,
+            -0.01019579079002142,
+            0.02227366529405117,
+            -0.025500090792775154,
+            0.036085035651922226,
+            0.023391593247652054,
+            -0.03520767390727997,
+            -0.00608492735773325,
+            -0.01903308741748333,
+            0.059037946164608,
+            -0.0011506526498124003,
+            -0.010301923379302025,
+            0.022938761860132217,
+            -0.027268964797258377,
+            -0.004524780437350273,
+            0.022301966324448586,
+            0.01828308403491974,
+            -0.008278331719338894,
+            0.003983504604548216,
+            -0.022287815809249878,
+            -0.05281151086091995,
+            -0.010577867738902569,
+            -0.0003305583668407053,
+            0.054368119686841965,
+            -0.012367968447506428,
+            0.0020695829298347235,
+            -0.006413938011974096,
+            0.004100250545889139,
+            0.03704730048775673,
+            0.002140337834134698,
+            0.0025082635693252087,
+            -0.013698162510991096,
+            0.0371888130903244,
+            0.022316116839647293,
+            0.005883275531232357,
+            -0.012162779457867146,
+            0.005377377849072218,
+            -0.04899074137210846,
+            -0.0009118546731770039,
+            0.005741765722632408,
+            0.007386818993836641,
+            -0.025910470634698868,
+            0.017717044800519943,
+            0.009205221198499203,
+            -0.0024693484883755445,
+            -0.009792488068342209,
+            0.01173117384314537,
+            0.025429336354136467,
+            -0.019938750192523003,
+            -0.05700020492076874,
+            0.0034705311991274357,
+            0.012014193460345268,
+            -0.004054259974509478,
+            -0.019401011988520622,
+            0.006792477332055569,
+            0.0021934041287750006,
+            0.022698193788528442,
+            -0.011271266266703606,
+            -0.0029221803415566683,
+            0.040585052222013474,
+            0.009516543708741665,
+            -0.0016813150141388178,
+            -0.017717044800519943,
+            -0.011794853024184704,
+            0.018070818856358528,
+            0.004740582779049873,
+            0.0035289041697978973,
+            -0.01751893013715744,
+            -0.029264256358146667,
+            0.02800481766462326,
+            0.019967053085565567,
+            -0.048905834555625916,
+            -0.008200501091778278,
+            -0.019273653626441956,
+            0.0019599127117544413,
+            0.008695785887539387,
+            0.04613224044442177,
+            -0.00590803986415267,
+            0.008313708938658237,
+            -0.020179318264126778,
+            -0.05003791302442551,
+            -0.02708500251173973,
+            0.03707560524344444,
+            -0.028684064745903015,
+            -0.04276430234313011,
+            -0.010252394713461399,
+            0.016740625724196434,
+            -0.009403335861861706,
+            -0.018679311498999596,
+            -0.011738249100744724,
+            -0.04969828948378563,
+            -0.003497064346447587,
+            0.0337359681725502,
+            0.017264213413000107,
+            -0.008441068232059479,
+            -0.033877480775117874,
+            0.019641580060124397,
+            -0.007160402834415436,
+            -0.006474079564213753,
+            0.0007862646016292274,
+            0.008794843219220638,
+            0.02400008589029312,
+            0.011879759840667248,
+            -0.02763689123094082,
+            -0.03427370637655258,
+            -0.0024888061452656984,
+            0.014929298311471939,
+            -0.00878776703029871,
+            0.0245378240942955,
+            0.024679332971572876,
+            -0.022995365783572197,
+            -0.01654251106083393,
+            0.015750056132674217,
+            -0.05779265984892845,
+            -0.010287772864103317,
+            -0.029235953465104103,
+            -0.006183984223753214,
+            0.0009613831643946469,
+            0.0061096916906535625,
+            0.00012304731353651732,
+            0.03846240043640137,
+            0.008801918476819992,
+            0.006014172453433275,
+            0.009191070683300495,
+            0.01913214474916458,
+            0.028471799567341805,
+            0.009155693463981152,
+            -0.03517937287688255,
+            0.02850010246038437,
+            -0.017051948234438896,
+            0.007811348885297775,
+            -0.025754809379577637,
+            -0.013033065013587475,
+            0.00443633645772934,
+            0.02725481428205967,
+            -0.000503244751598686,
+            0.018849123269319534,
+            -0.021919889375567436,
+            0.037500135600566864,
+            -0.029915200546383858,
+            -0.05278320610523224,
+            -0.02005195803940296,
+            -0.008285406976938248,
+            0.015764206647872925,
+            0.017278363928198814,
+            -0.019924599677324295,
+            -0.0286557637155056,
+            -0.021750077605247498,
+            0.006470541935414076,
+            0.010542490519583225,
+            -0.021099131554365158,
+            -0.010627396404743195,
+            0.01502835564315319,
+            -0.00404364662244916,
+            -0.01886327564716339,
+            0.013684011064469814,
+            0.015693452209234238,
+            0.03169822692871094,
+            -0.03308502212166786,
+            0.004287751391530037,
+            -0.025967074558138847,
+            -0.04327373951673508,
+            0.004096712917089462,
+            0.03166992589831352,
+            0.027509532868862152,
+            -0.018636859953403473,
+            0.010754755698144436,
+            0.003990580327808857,
+            0.004234685096889734,
+            -0.0033803186379373074,
+            0.023603858426213264,
+            0.03291521221399307,
+            0.02763689123094082,
+            -0.011653343215584755,
+            0.004921008367091417,
+            -0.01559439580887556,
+            -0.0011108530452474952,
+            0.01865101046860218,
+            -0.017816102132201195,
+            0.033877480775117874,
+            -0.029405765235424042,
+            0.02104252763092518,
+            -0.035122767090797424,
+            0.006141531281173229,
+            -0.012099100276827812,
+            -0.00404364662244916,
+            0.04833979532122612,
+            -0.017646290361881256,
+            -0.0019546060357242823,
+            -0.008101443760097027,
+            -0.014575524255633354,
+            -0.02158026583492756,
+            0.008766540326178074,
+            -0.04033033177256584,
+            -0.012367968447506428,
+            -0.028429346159100533,
+            0.003424540627747774,
+            0.019910449162125587,
+            -0.06277380883693695,
+            -0.06237758323550224,
+            0.012275987304747105,
+            -0.04918885603547096,
+            -0.026419905945658684,
+            0.02963218092918396,
+            -0.002984090941026807,
+            0.0012780115939676762,
+            -0.0024003623984754086,
+            0.0068101659417152405,
+            -0.029207652434706688,
+            0.00425944896414876,
+            -0.000887974863871932,
+            0.03271709755063057,
+            -0.021056678146123886,
+            -0.008122670464217663,
+            0.004878554958850145,
+            -0.06860402226448059,
+            -0.028344441205263138,
+            0.011271266266703606,
+            -0.009459939785301685,
+            -0.0066615804098546505,
+            -0.017915157601237297,
+            -0.004383270163089037,
+            0.0008950503543019295,
+            0.0032830305863171816,
+            0.05801907554268837,
+            -0.01983969286084175,
+            -0.053773775696754456,
+            0.03633975237607956,
+            0.00887267291545868,
+            0.029603879898786545,
+            -0.025556694716215134,
+            -0.017533080652356148,
+            -0.01855195313692093,
+            -0.020504789426922798,
+            -0.04123599827289581,
+            0.00404364662244916,
+            -0.005918653216212988,
+            0.009212297387421131,
+            -0.0009693430620245636,
+            0.02763689123094082,
+            0.029235953465104103,
+            0.023108573630452156,
+            0.012750045396387577,
+            0.01079013291746378,
+            -0.022372720763087273,
+            -0.021594416350126266,
+            0.0012222920777276158,
+            0.04683978855609894,
+            0.024636881425976753,
+            -0.02675952948629856,
+            -0.011504758149385452,
+            0.03953787684440613,
+            0.0419718474149704,
+            -0.01974063739180565,
+            -0.02276895008981228,
+            0.00764861237257719,
+            0.03266049548983574,
+            -0.04477374628186226,
+            0.005932804197072983,
+            0.03410389646887779,
+            0.014405711553990841,
+            -0.005356151144951582,
+            -0.010294848121702671,
+            0.022570835426449776,
+            0.03152841329574585,
+            -0.008724087849259377,
+            0.01876421831548214,
+            0.016910437494516373,
+            0.029349161311984062,
+            -0.0067889392375946045,
+            0.0005766529939137399,
+            -0.024735936895012856,
+            0.055188875645399094,
+            -0.03059444949030876,
+            0.02909444272518158,
+            0.010316074825823307,
+            -0.00593634182587266,
+            0.039056744426488876,
+            -0.016641568392515182,
+            -0.0431322306394577,
+            0.01913214474916458,
+            -0.006979977712035179,
+            -0.0055613406002521515,
+            -0.03444352000951767,
+            -0.007592008449137211,
+            -0.007507102098315954,
+            -0.006714646704494953,
+            -0.022556684911251068,
+            -0.02606613002717495,
+            0.004415110219269991,
+            0.00973588414490223,
+            -0.006484692916274071,
+            0.00698705343529582,
+            0.02092931978404522,
+            -0.013507124036550522,
+            0.02525952458381653,
+            -0.046047333627939224,
+            -0.0008366775000467896,
+            -0.0038596836384385824,
+            -0.025712355971336365,
+            -0.0345001220703125,
+            -0.0005770951975136995,
+            -0.0024251264985650778,
+            -0.024622729048132896,
+            -0.00914154201745987,
+            -0.008752389810979366,
+            -0.0023119186516851187,
+            -0.004938696976751089,
+            -0.055188875645399094,
+            0.008695785887539387,
+            0.010846736840903759,
+            -0.002713453024625778,
+            0.0055507272481918335,
+            -0.010613245889544487,
+            -0.03039633482694626,
+            0.03039633482694626,
+            0.04129260033369064,
+            0.033707667142152786,
+            0.014334957115352154,
+            -0.0074646491557359695,
+            -0.017165156081318855,
+            0.006060163024812937,
+            0.021721776574850082,
+            0.00902833417057991,
+            0.007846726104617119,
+            0.03877372294664383,
+            0.00011331850691931322,
+            -0.008349086157977581,
+            -0.009764186106622219,
+            0.028740668669342995,
+            -0.001085204305127263,
+            0.023603858426213264,
+            0.050802066922187805,
+            0.030424637719988823,
+            0.020589696243405342,
+            -0.01681138016283512,
+            -0.02010856196284294,
+            -0.00851889792829752,
+            -0.02855670638382435,
+            0.0364246591925621,
+            0.012127402238547802,
+            -0.005232329946011305,
+            -0.011540135368704796,
+            0.019061388447880745,
+            -0.033339742571115494,
+            -0.0037358624394983053,
+            0.00048422932741232216,
+            0.013924578204751015,
+            -0.03379257395863533,
+            0.0002480846014805138,
+            0.031613320112228394,
+            0.014957600273191929,
+            -0.016995344310998917,
+            -0.0188774261623621,
+            0.010549565777182579,
+            0.020462337881326675,
+            0.018948180601000786,
+            0.00037743354914709926,
+            0.00897173024713993,
+            -0.00022121983056422323,
+            -0.0005319889169186354,
+            0.01590571738779545,
+            -0.007952858693897724,
+            0.036254845559597015,
+            -0.003525366308167577,
+            -0.01325240544974804,
+            -0.0037500134203583,
+            -0.021438755095005035,
+            -0.024467067793011665,
+            0.043556760996580124,
+            0.023971782997250557,
+            0.04661337286233902,
+            -0.005660397466272116,
+            -0.004245297983288765,
+            0.025825563818216324,
+            0.008221727795898914,
+            -0.03288691118359566,
+            0.000265110022155568,
+            0.005281858611851931,
+            -0.02892463095486164,
+            0.016641568392515182,
+            -0.02476423978805542,
+            -0.016896286979317665,
+            -0.006771250627934933,
+            -0.0015557248843833804,
+            -0.028188779950141907,
+            0.01703779585659504,
+            0.0031309074256569147,
+            -0.0027099153958261013,
+            -0.039764292538166046,
+            0.006880920846015215,
+            0.020419884473085403,
+            0.015311375260353088,
+            0.0037358624394983053,
+            0.011525984853506088,
+            -0.016613267362117767,
+            0.002126186853274703,
+            0.03379257395863533,
+            0.004878554958850145,
+            0.02005195803940296,
+            0.0027647505048662424,
+            0.023773670196533203,
+            0.03633975237607956,
+            -0.01681138016283512,
+            -0.02189158834517002,
+            0.01519816741347313,
+            -0.02233026921749115,
+            0.029943503439426422,
+            0.004889168310910463,
+            -0.012283062562346458,
+            0.022033097222447395,
+            -0.009827865287661552,
+            -0.0027346794959157705,
+            0.01659911684691906,
+            -0.022740647196769714,
+            -0.017717044800519943,
+            0.03183973580598831,
+            -0.039452970027923584,
+            0.012856177985668182,
+            -0.008547200821340084,
+            0.015863263979554176,
+            -0.023066120222210884,
+            0.011172209866344929,
+            0.007100261282175779,
+            -0.01573590561747551,
+            0.010797208175063133,
+            0.009693430736660957,
+            0.00545874610543251,
+            0.0018926954362541437,
+            0.02179253101348877,
+            -0.0006049550138413906,
+            -0.011257115751504898,
+            -0.018155725672841072,
+            -0.004694592207670212,
+            -0.002614396158605814,
+            -0.016556663438677788,
+            0.01865101046860218,
+            -0.041066184639930725,
+            -0.013280707411468029,
+            -0.008879749104380608,
+            -0.013988257385790348,
+            -0.010011828504502773,
+            0.024184048175811768,
+            0.025669902563095093,
+            -0.002361447084695101,
+            -0.031217092648148537,
+            0.01790100708603859,
+            0.024198200553655624,
+            0.003654494183138013,
+            -0.023646311834454536,
+            0.0302831269800663,
+            -0.008936353027820587,
+            0.014971751719713211,
+            0.026136886328458786,
+            0.014094389975070953,
+            0.004595535341650248,
+            -0.010500037111341953,
+            -0.02114158496260643,
+            0.01481609046459198,
+            0.010507113300263882,
+            0.033537857234478,
+            0.014745336025953293,
+            0.02071705460548401,
+            0.01594817079603672,
+            -0.023320838809013367,
+            -0.011214662343263626,
+            -0.003482913365587592,
+            -0.022514231503009796,
+            0.03093407303094864,
+            0.0006129149114713073,
+            0.02114158496260643,
+            0.026618018746376038,
+            -0.03319823369383812,
+            -0.01187268365174532,
+            0.032688796520233154,
+            0.02779255248606205,
+            0.05108508840203285,
+            0.025316128507256508,
+            0.015396281145513058,
+            0.027226513251662254,
+            -0.03585861995816231,
+            -0.036368053406476974,
+            0.001548649393953383,
+            0.04256619140505791,
+            -0.008426916785538197,
+            0.00967928022146225,
+            0.03900013864040375,
+            -0.006781863979995251,
+            -0.01611798256635666,
+            0.019457615911960602,
+            0.0026710000820457935,
+            0.002448122017085552,
+            -0.0033113325480371714,
+            -0.01335146278142929,
+            0.03175482898950577,
+            -0.0051474240608513355,
+            0.013570803217589855,
+            -0.019273653626441956,
+            0.027113303542137146,
+            0.0001799829478841275,
+            -0.003229964291676879,
+            0.028344441205263138,
+            -0.002660386962816119,
+            -0.051990751177072525,
+            0.024212351068854332,
+            0.019811391830444336,
+            -0.011561362072825432,
+            -0.0036332677118480206,
+            -0.003792466362938285,
+            0.04567940905690193,
+            0.034698236733675,
+            0.029009537771344185,
+            -0.0020501252729445696,
+            0.0168679840862751,
+            0.018792519345879555,
+            0.0053667644970119,
+            0.014278353191912174,
+            -0.0017025414854288101,
+            -0.007853801362216473,
+            0.021056678146123886,
+            -0.020787810906767845,
+            0.03226426616311073,
+            -0.016570813953876495,
+            -0.020122714340686798,
+            0.008738238364458084,
+            -0.011172209866344929,
+            -0.0009083169279620051,
+            -0.006311343517154455,
+            -0.0008813415770418942,
+            -0.0029664020985364914,
+            0.016938740387558937,
+            -0.004446949809789658,
+            0.04661337286233902,
+            -0.013931653462350368,
+            -0.008808993734419346,
+            0.006972902454435825,
+            -0.025981225073337555,
+            0.0013346156338229775,
+            0.0028726516757160425,
+            0.027000095695257187,
+            -0.01989629678428173,
+            -0.00970758218318224,
+            0.01606137864291668,
+            0.06039644405245781,
+            -0.008639181964099407,
+            -0.0026267783250659704,
+            0.027226513251662254,
+            -0.002787745790556073,
+            -0.03551899641752243,
+            0.012919857166707516,
+            -0.04494355618953705,
+            0.017589686438441277,
+            0.026349151507019997,
+            -0.04986810311675072,
+            -0.022089701145887375,
+            -0.02347649820148945,
+            -0.02043403498828411,
+            0.023434046655893326,
+            -0.009976450353860855,
+            0.013867974281311035,
+            -0.017801949754357338,
+            -0.009014182724058628,
+            0.020518941804766655,
+            0.009176919236779213,
+            0.03735862299799919,
+            -0.027551984414458275,
+            -0.016783079132437706,
+            -0.027820853516459465,
+            -0.04304732382297516,
+            0.009558996185660362,
+            -0.0278916098177433,
+            0.0027099153958261013,
+            -0.0016291331266984344,
+            -0.040302030742168427,
+            -0.037981268018484116,
+            0.005278320983052254,
+            -0.017985913902521133,
+            0.01516986545175314,
+            0.020915169268846512,
+            0.02731141820549965,
+            -0.0020253609400242567,
+            -0.030198222026228905,
+            0.029235953465104103,
+            -0.007344366051256657,
+            0.03223596513271332,
+            -0.00689507182687521,
+            0.005066055804491043,
+            -0.0070436573587358,
+            0.031160488724708557,
+            -0.023009516298770905,
+            0.018792519345879555,
+            -0.026801982894539833,
+            -0.004086099565029144,
+            -0.018990634009242058,
+            -0.01774534583091736,
+            0.004224071744829416,
+            0.004521242808550596,
+            7.888073741924018e-05,
+            0.0048750173300504684,
+            0.005777143407613039,
+            0.024806693196296692,
+            0.0021421066485345364,
+            0.02589631825685501,
+            -0.027877457439899445,
+            -0.025316128507256508,
+            0.01719345711171627,
+            -0.02077365852892399,
+            0.005678086541593075,
+            -0.006261814851313829,
+            -0.0302831269800663,
+            -0.01202834490686655,
+            0.023391593247652054,
+            0.02162271924316883,
+            -0.005490585695952177,
+            0.002644466934725642,
+            0.02206140011548996,
+            -0.026023678481578827,
+            0.00437973253428936,
+            -0.03781145438551903,
+            -0.018396291881799698,
+            0.029773691669106483,
+            -0.03195294365286827,
+            0.020745357498526573,
+            0.003185742534697056,
+            0.03483974561095238,
+            -0.017292514443397522,
+            -0.01947176828980446,
+            0.01654251106083393,
+            -0.01008258294314146,
+            0.004142703488469124,
+            0.01606137864291668,
+            0.01616043597459793,
+            -0.0015070808585733175,
+            0.05380208045244217,
+            0.0008167776977643371,
+            0.006909222807735205,
+            -0.049075648188591,
+            0.004496478475630283,
+            -0.044858649373054504,
+            0.0002768288250081241,
+            -0.013337312266230583,
+            -0.02514631673693657,
+            0.0014027173165231943,
+            0.001001182827167213,
+            0.001963450340554118,
+            -0.01913214474916458,
+            -0.01989629678428173,
+            0.011596739292144775,
+            -0.015863263979554176,
+            0.004928083624690771,
+            -0.06418890506029129,
+            -0.01855195313692093,
+            -0.004974074196070433,
+            0.0043337419629096985,
+            0.0007699025445617735,
+            0.03342464938759804,
+            -0.01478778850287199,
+            0.010117961093783379,
+            -0.01549533847719431,
+            -0.026363302022218704,
+            0.04010391607880592,
+            0.018750067800283432,
+            -0.023773670196533203,
+            0.024693485349416733,
+            0.012106175534427166,
+            0.03099067695438862,
+            -0.0318114347755909,
+            0.020349130034446716,
+            0.018523650243878365,
+            -0.0029823218937963247,
+            0.006551910191774368,
+            0.008575502783060074,
+            0.01832553744316101,
+            -0.015764206647872925,
+            -0.0003632825391832739,
+            0.011419852264225483,
+            0.010033054277300835,
+            0.031924642622470856,
+            0.01403071079403162,
+            -0.03183973580598831,
+            0.019047237932682037,
+            0.01616043597459793,
+            0.02298121340572834,
+            0.033820874989032745,
+            -0.027481229975819588,
+            0.038575608283281326,
+            -0.010386829264461994,
+            -0.024042539298534393,
+            0.020801961421966553,
+            -0.032688796520233154,
+            -0.006679269019514322,
+            0.030735958367586136,
+            0.04398128762841225,
+            -0.0015566092915832996,
+            -0.044632233679294586,
+            0.01535382866859436,
+            0.01316749956458807,
+            0.0027806703001260757,
+            0.028811423107981682,
+            0.011264191009104252,
+            0.021820832043886185,
+            0.025273675099015236,
+            0.011419852264225483,
+            -0.0037252490874379873,
+            0.024962352588772774,
+            -0.008101443760097027,
+            -0.004446949809789658,
+            -0.0007035697344690561,
+            0.03206615149974823,
+            0.014702882617712021,
+            -0.017603836953639984,
+            -0.036962397396564484,
+            -0.007620310410857201,
+            0.009452863596379757,
+            -0.007903330028057098,
+            -0.027127455919981003,
+            -0.03950957581400871,
+            -0.026787832379341125,
+            -0.005667473189532757,
+            0.01505665760487318,
+            -0.06724552065134048,
+            -0.007620310410857201,
+            -0.012233533896505833,
+            0.017717044800519943,
+            0.0022906921803951263,
+            0.0181132722645998,
+            0.001277127186767757,
+            -0.009403335861861706,
+            0.004209920763969421,
+            0.050802066922187805,
+            -0.03059444949030876,
+            0.014242975041270256,
+            0.011292492970824242,
+            0.0032105068676173687,
+            0.002787745790556073,
+            0.038688816130161285,
+            -0.010988246649503708,
+            -0.030424637719988823,
+            0.011575513519346714,
+            -0.017292514443397522,
+            0.003383856499567628,
+            0.05006621778011322,
+            -0.009481165558099747,
+            -0.027495380491018295,
+            0.020915169268846512,
+            -0.009148617275059223,
+            -0.015523640438914299,
+            -0.0027258351910859346,
+            -0.006035399157553911,
+            0.0006283925613388419,
+            -0.03832089155912399,
+            -0.015750056132674217,
+            0.03212275728583336,
+            0.008617955259978771,
+            0.014603826217353344,
+            0.025344429537653923,
+            0.010209942236542702,
+            -0.030481241643428802,
+            0.027481229975819588,
+            -0.0005678086308762431,
+            0.005221716593950987,
+            0.013323160819709301,
+            0.03424540534615517,
+            -0.014264201745390892,
+            -0.020264223217964172,
+            -0.040245428681373596,
+            -0.013853822834789753,
+            -0.0463869571685791,
+            0.001943992800079286,
+            0.006919836159795523,
+            -0.012919857166707516,
+            -0.04366996884346008,
+            -0.0329718142747879,
+            0.021127434447407722,
+            0.03178313374519348,
+            -0.017023645341396332,
+            0.0025294902734458447,
+            -0.008483520708978176,
+            -0.007882103323936462,
+            -0.0017520699184387922,
+            -0.013217028230428696,
+            -0.006378560326993465,
+            -0.006229975260794163,
+            0.015452885068953037,
+            -0.004652139265090227,
+            -0.0036934094969183207,
+            -0.0498398020863533,
+            -0.018778368830680847,
+            0.004082561936229467,
+            -0.008221727795898914,
+            0.007323139347136021,
+            -0.006955213379114866,
+            -0.002727604005485773,
+            -0.012474101036787033,
+            -0.03829259052872658,
+            0.024184048175811768,
+            0.017278363928198814,
+            0.0035501306410878897,
+            -0.008922201581299305,
+            -0.018367990851402283,
+            0.03294351324439049,
+            0.012523629702627659,
+            -0.017547233030200005,
+            -0.018580254167318344,
+            0.025981225073337555,
+            0.026957644149661064,
+            0.008122670464217663,
+            -0.006137993652373552,
+            0.029886899515986443,
+            0.005695775151252747,
+            -0.022938761860132217,
+            -0.003983504604548216,
+            -0.005055442452430725,
+            -0.0038844477385282516,
+            0.020561393350362778,
+            -0.010556641966104507,
+            0.0300001073628664,
+            0.038717120885849,
+            0.012870329432189465,
+            0.0039198254235088825,
+            -0.0033714743331074715,
+            -0.008922201581299305,
+            0.015976471826434135,
+            0.006481155287474394,
+            0.008525974117219448,
+            0.014759486541152,
+            -0.016358548775315285,
+            -0.020405733957886696,
+            0.006215824279934168,
+            -0.007107336539775133,
+            0.020122714340686798,
+            0.01708024926483631,
+            0.016429303213953972,
+            0.0007610581233166158,
+            -0.028287837281823158,
+            -0.0018024828750640154,
+            0.02411329373717308,
+            0.02010856196284294,
+            -0.004655676893889904,
+            -0.023264234885573387,
+            0.014398636296391487,
+            -0.015806660056114197,
+            -0.009969375096261501,
+            0.0033431723713874817,
+            0.006229975260794163,
+            -0.003083147807046771,
+            0.0571134127676487,
+            -0.020193468779325485,
+            -0.011419852264225483,
+            -0.010287772864103317,
+            -0.010867963545024395,
+            0.0012072566896677017,
+            -0.01828308403491974,
+            0.008964654989540577,
+            0.025613298639655113,
+            -0.009764186106622219,
+            0.03568880632519722,
+            0.0228114016354084,
+            0.006912760436534882,
+            0.005260631907731295,
+            -0.027070851996541023,
+            0.012304289266467094,
+            -0.016047226265072823,
+            0.005660397466272116,
+            0.018467046320438385,
+            0.017320817336440086,
+            0.0076132346875965595,
+            0.026306698098778725,
+            0.019386861473321915,
+            0.016684021800756454,
+            -0.022783100605010986,
+            0.025457637384533882,
+            -0.007832575589418411,
+            0.027877457439899445,
+            -0.01611798256635666,
+            -0.002253545681014657,
+            -0.012990612536668777,
+            0.008681634441018105,
+            0.002416282193735242,
+            -0.0019351483788341284,
+            0.00036129256477579474,
+            0.016783079132437706,
+            -0.014589674770832062,
+            0.01062032114714384,
+            0.013316085562109947,
+            -0.027919910848140717,
+            -0.010372678749263287,
+            0.016344398260116577,
+            -0.00668988237157464,
+            0.006877383217215538,
+            -0.011865608394145966,
+            -0.0017202302115038037,
+            0.011009473353624344,
+            -0.0436982698738575,
+            0.0104929618537426,
+            0.006279503460973501,
+            -0.015155714936554432,
+            0.014801939949393272,
+            -0.009799563325941563,
+            -0.008235878311097622,
+            -0.013592029921710491,
+            0.003792466362938285,
+            -0.00030225637601688504,
+            -0.0018290160223841667,
+            -0.00022995365725364536,
+            0.00389859895221889,
+            0.010047205723822117,
+            -0.0045990729704499245,
+            -0.010450509376823902,
+            -0.0028638073708862066,
+            -0.02265574224293232,
+            -0.0010047205723822117,
+            0.018735915422439575,
+            -0.004591997712850571,
+            0.0005222601466812193,
+            0.0145189194008708,
+            -0.005830209702253342,
+            0.010556641966104507,
+            -0.005221716593950987,
+            -0.008320784196257591,
+            -0.010768906213343143,
+            -0.033764272928237915,
+            0.0026710000820457935,
+            -0.015254771336913109,
+            0.008865597657859325,
+            -0.012544856406748295,
+            0.04027372971177101,
+            0.01022409275174141,
+            -0.00837738811969757,
+            -0.002796590095385909,
+            -0.0228114016354084,
+            -0.004475251771509647,
+            -0.016938740387558937,
+            -0.01184438169002533,
+            0.008342010900378227,
+            0.012629762291908264,
+            0.005614406894892454,
+            -0.018410442396998405,
+            0.003665107535198331,
+            0.0030955299735069275,
+            -0.005575491581112146,
+            -0.008370312862098217,
+            0.010853812098503113,
+            -0.0006248548161238432,
+            -0.01800006441771984,
+            -0.01957082375884056,
+            -0.027424626052379608,
+            0.0150000536814332,
+            0.014589674770832062,
+            -0.01011088490486145,
+            -0.022754797711968422,
+            -0.007471724878996611,
+            0.007107336539775133,
+            -0.02064630016684532,
+            0.0006779210525564849,
+            0.006891534198075533,
+            0.008717012591660023,
+            0.004117939155548811,
+            0.009636826813220978,
+            0.0007699025445617735,
+            0.0121698547154665,
+            0.018566103652119637,
+            0.03271709755063057,
+            -0.0396793857216835,
+            -0.009537769481539726,
+            0.0057028504088521,
+            -0.001465512323193252,
+            -0.013853822834789753,
+            0.009672204032540321,
+            0.0024375086650252342,
+            -0.0026426981203258038,
+            0.024424616247415543,
+            -0.008044839836657047,
+            -0.01505665760487318,
+            0.01184438169002533,
+            0.00887267291545868,
+            -0.019075538963079453,
+            0.02746707946062088,
+            0.006169833242893219,
+            -0.01011088490486145,
+            0.02259913645684719,
+            -0.012360893189907074,
+            0.012658064253628254,
+            -0.005522425286471844,
+            0.023660462349653244,
+            -0.019004784524440765,
+            0.0006306036957539618,
+            0.015665151178836823,
+            0.03883032873272896,
+            -0.00437973253428936,
+            -0.004567233379930258,
+            -0.03421710431575775,
+            0.005115584470331669,
+            0.028853876516222954,
+            0.02666047215461731,
+            -0.0012346742441877723,
+            -0.005596718285232782,
+            -0.0019864458590745926,
+            0.010768906213343143,
+            0.011589664034545422,
+            0.017561383545398712,
+            -0.029547275975346565,
+            0.00022011429246049374,
+            -0.016938740387558937,
+            0.025500090792775154,
+            0.014632128179073334,
+            0.006951675750315189,
+            0.04072656109929085,
+            0.0027789014857262373,
+            0.003841994795948267,
+            0.027452928945422173,
+            -0.00779719790443778,
+            0.003546592779457569,
+            -0.015566092915832996,
+            0.0003175129240844399,
+            0.022853855043649673,
+            0.0016370931407436728,
+            0.0020784272346645594,
+            0.013337312266230583,
+            0.029320860281586647,
+            -0.022471778094768524,
+            -0.023929331451654434,
+            0.003482913365587592,
+            0.007185167167335749,
+            -0.010570792481303215,
+            0.0038738346192985773,
+            0.006275965832173824,
+            -0.006007096730172634,
+            0.008900974877178669,
+            0.006332569755613804,
+            -0.0003166284877806902,
+            0.0039198254235088825,
+            -0.00308845448307693,
+            0.012962310574948788,
+            -0.02702839858829975,
+            0.036849189549684525,
+            0.004439874552190304,
+            0.010075507685542107,
+            0.015268922783434391,
+            0.001627364312298596,
+            0.004917470272630453,
+            0.005869124550372362,
+            0.010188715532422066,
+            0.018636859953403473,
+            -0.00956607237458229,
+            -0.027481229975819588,
+            -0.03166992589831352,
+            0.020363280549645424,
+            -0.01173117384314537,
+            0.0035872769076377153,
+            0.0074646491557359695,
+            -0.01828308403491974,
+            -0.007507102098315954,
+            -0.009976450353860855,
+            0.015764206647872925,
+            -0.03189634159207344,
+            0.012990612536668777,
+            -0.0043231286108493805,
+            -0.028669914230704308,
+            0.00851889792829752,
+            -0.010181640274822712,
+            -0.0045884596183896065,
+            -0.0035094465129077435,
+            -0.029264256358146667,
+            -0.007931631989777088,
+            0.03908504545688629,
+            -0.02141045406460762,
+            0.024184048175811768,
+            0.008257105015218258,
+            -0.00014449491573031992,
+            -0.0040365708991885185,
+            -0.013563727959990501,
+            0.005331386812031269,
+            -0.01778779923915863,
+            0.025457637384533882,
+            0.010294848121702671,
+            -0.013924578204751015,
+            0.0322076641023159,
+            -0.011179285123944283,
+            0.027438776567578316,
+            -0.018580254167318344,
+            0.036198243498802185,
+            0.007422196213155985,
+            0.00674648629501462,
+            0.01974063739180565,
+            -0.004025958012789488,
+            0.008228803053498268,
+            0.04559450224041939,
+            -0.018523650243878365,
+            0.02460857853293419,
+            0.020349130034446716,
+            -0.00022685811563860625,
+            0.017915157601237297,
+            0.0028390432707965374,
+            -0.023165177553892136,
+            0.009495317004621029,
+            -0.01135617308318615,
+            -0.009955224581062794,
+            0.01583496294915676,
+            -0.019316107034683228,
+            -0.018721764907240868,
+            -0.025613298639655113,
+            0.0029416377656161785,
+            -0.003350247861817479,
+            -0.0077618202194571495,
+            -0.0019616815261542797,
+            0.01454722136259079,
+            0.008724087849259377,
+            0.025669902563095093,
+            0.0008326975512318313,
+            0.011971740983426571,
+            0.01511326152831316,
+            -0.00035576484515331686,
+            0.02313687466084957,
+            -0.03229256719350815,
+            0.01246702577918768,
+            -0.004517704714089632,
+            -0.00509789539501071,
+            0.023858575150370598,
+            0.0003747802402358502,
+            -0.0004877670726273209,
+            -0.0024888061452656984,
+            0.0005496777012012899,
+            -0.014589674770832062,
+            -0.0057099261321127415,
+            0.000907432462554425,
+            -0.005175726022571325,
+            0.04299071803689003,
+            -0.00701535539701581,
+            -0.004772422835230827,
+            -0.039169952273368835,
+            -0.01918874867260456,
+            0.00623351288959384,
+            -0.01583496294915676,
+            -0.018778368830680847,
+            -0.006307805422693491,
+            0.005801907740533352,
+            -0.04893413558602333,
+            0.008342010900378227,
+            0.03336804360151291,
+            -0.012219383381307125,
+            0.00259493850171566,
+            -0.010981171391904354,
+            -0.006969364359974861,
+            -0.0029115669894963503,
+            -7.766463386360556e-05,
+            -0.002412744564935565,
+            -0.020830262452363968,
+            -0.005617944523692131,
+            0.003364398842677474,
+            -0.0008795727044343948,
+            -0.025047259405255318,
+            0.0006177793256938457,
+            0.018849123269319534,
+            -0.003979966975748539,
+            -0.0011152751976624131,
+            -0.001825478277169168,
+            0.006399787031114101,
+            -0.01508495956659317,
+            0.008009462617337704,
+            -0.0076344613917171955,
+            0.003841994795948267,
+            -0.0030495391692966223,
+            -0.002214630600064993,
+            0.02844349853694439,
+            0.021594416350126266,
+            -0.04808507859706879,
+            -0.01405901275575161,
+            0.023009516298770905,
+            0.007280686404556036,
+            -0.004740582779049873,
+            0.01178777776658535,
+            -0.031613320112228394,
+            -0.00028943203506059945,
+            -0.026787832379341125,
+            0.02200479619204998,
+            -0.009205221198499203,
+            -0.007386818993836641,
+            0.016967041417956352,
+            -0.026632171124219894,
+            0.016344398260116577,
+            0.00794578343629837,
+            -0.00030181417241692543,
+            -0.000493515923153609,
+            0.0063856360502541065,
+            0.0008972614887170494,
+            -0.01659911684691906,
+            0.006442239973694086,
+            -0.0048502529971301556,
+            0.007959933951497078,
+            -0.005087282508611679,
+            -0.05527378246188164,
+            -0.009933997876942158,
+            -0.005497660953551531,
+            -0.008936353027820587,
+            -0.012820800766348839,
+            0.008136821910738945,
+            0.02827368676662445,
+            0.01606137864291668,
+            -0.009863242506980896,
+            -4.8312376748071983e-05,
+            0.033000119030475616,
+            -0.0008300442132167518,
+            0.009955224581062794,
+            -0.011349096894264221,
+            0.00015709814033471048,
+            -0.0006911876262165606,
+            -0.014660430140793324,
+            -0.019061388447880745,
+            0.024042539298534393,
+            -0.028344441205263138,
+            0.02378782071173191,
+            -0.01843874529004097,
+            -0.0130825936794281,
+            0.01757553406059742,
+            -0.032575588673353195,
+            -0.005108508747071028,
+            0.007450498174875975,
+            0.0090566361322999,
+            0.021608568727970123,
+            0.0032794929575175047,
+            0.020589696243405342,
+            -0.03716050833463669,
+            -0.009134466759860516,
+            -0.010160413570702076,
+            -0.00794578343629837,
+            -0.010231168940663338,
+            -0.011596739292144775,
+            0.021863285452127457,
+            -0.015240620821714401,
+            -0.016952890902757645,
+            0.007054270710796118,
+            0.0030212372075766325,
+            -0.011313719674944878,
+            0.015750056132674217,
+            0.00017810352437663823,
+            0.004712280817329884,
+            0.013011839240789413,
+            -0.0008923970744945109,
+            0.04086806997656822,
+            0.0253019779920578,
+            -0.0030212372075766325,
+            -0.008568426594138145,
+            0.021438755095005035,
+            0.0004285097820684314,
+            0.0150000536814332,
+            0.009375033900141716,
+            0.023264234885573387,
+            -0.02092931978404522,
+            0.009997677057981491,
+            -0.005501199048012495,
+            0.014603826217353344,
+            -0.009495317004621029,
+            -0.015339677222073078,
+            0.00479718716815114,
+            -0.01577835902571678,
+            0.010153338313102722,
+            -0.010860888287425041,
+            0.008221727795898914,
+            0.01913214474916458,
+            0.002467579673975706,
+            -0.00941041111946106,
+            0.01146230474114418,
+            0.02075950801372528,
+            -0.025641601532697678,
+            0.008851447142660618,
+            -0.0031998935155570507,
+            -0.0021916350815445185,
+            0.021127434447407722,
+            0.010301923379302025,
+            -0.01594817079603672,
+            0.01573590561747551,
+            -0.008879749104380608,
+            -0.0015053119277581573,
+            -0.01882082223892212,
+            -0.03291521221399307,
+            -0.0015283073298633099,
+            0.012396270409226418,
+            0.004800724796950817,
+            0.0150000536814332,
+            0.011264191009104252,
+            -0.0008950503543019295,
+            -0.006601438857614994,
+            0.0008433107868768275,
+            -0.013485897332429886,
+            -0.004963461309671402,
+            0.010231168940663338,
+            0.017051948234438896,
+            -0.0037217114586383104,
+            0.015650998800992966,
+            0.0088160689920187,
+            0.006137993652373552,
+            -0.0010480580385774374,
+            0.0036138100549578667,
+            -0.002060738392174244,
+            -0.007783046457916498,
+            0.019938750192523003,
+            0.019075538963079453,
+            -0.018509499728679657,
+            -0.0024870370980352163,
+            -0.02141045406460762,
+            0.025882167741656303,
+            0.022188758477568626,
+            0.011165134608745575,
+            -0.04222656786441803,
+            0.00012050455552525818,
+            0.00752832880243659,
+            -0.0013744152383878827,
+            -0.031103884801268578,
+            0.008172199130058289,
+            0.005221716593950987,
+            0.004330204334110022,
+            0.020844414830207825,
+            -0.018424594774842262,
+            0.010853812098503113,
+            0.0070861103013157845,
+            -0.0005766529939137399,
+            0.01019579079002142,
+            -0.0073372903279960155,
+            0.008688710629940033,
+            0.038717120885849,
+            -0.018410442396998405,
+            0.005303084850311279,
+            0.0062370505183935165,
+            0.023320838809013367,
+            -0.011228813789784908,
+            -0.00031485961517319083,
+            0.00800238735973835,
+            0.021608568727970123,
+            -0.012934008613228798,
+            0.005883275531232357,
+            0.067075714468956,
+            -0.012912781909108162,
+            0.031613320112228394,
+            0.020533092319965363,
+            -0.004970536567270756,
+            -0.0104929618537426,
+            -0.012445799075067043,
+            -0.0181132722645998,
+            0.021283095702528954,
+            0.01175947580486536,
+            0.003419233951717615,
+            0.01941516436636448,
+            -0.01610383205115795,
+            0.006636816076934338,
+            0.017547233030200005,
+            0.012601460330188274,
+            -0.007379743270576,
+            -0.0036173479165881872,
+            0.019004784524440765,
+            0.018509499728679657,
+            -0.0030849166214466095,
+            0.00019424450874794275,
+            0.01508495956659317,
+            -0.022075550630688667,
+            -0.012530704960227013,
+            0.01644345559179783,
+            -0.0037358624394983053,
+            0.017773648723959923,
+            -0.02227366529405117,
+            0.024198200553655624,
+            -0.004531855694949627,
+            -0.017179306596517563,
+            0.004698129836469889,
+            -0.007514177821576595,
+            0.022033097222447395,
+            -0.001578720286488533,
+            0.004857328720390797,
+            0.000821199850179255,
+            -0.0018015984678640962,
+            0.01349297259002924,
+            0.002826661104336381,
+            -0.0005120891146361828,
+            0.010507113300263882,
+            -0.0009702275274321437,
+            -0.0013496510218828917,
+            0.024509521201252937,
+            1.9374700059415773e-05,
+            0.005989408120512962,
+            0.0032936439383774996,
+            -0.009261825121939182,
+            -0.004429261200129986,
+            -0.004443412180989981,
+            0.0011842612875625491,
+            0.003877372248098254,
+            -0.007029506377875805,
+            -0.02384442463517189,
+            0.022471778094768524,
+            -0.008540124632418156,
+            0.01175947580486536,
+            0.011349096894264221,
+            0.003665107535198331,
+            0.006824316922575235,
+            -0.009771261364221573,
+            -0.017320817336440086,
+            0.0006633278680965304,
+            -0.01617458648979664,
+            0.006912760436534882,
+            -0.001656550681218505,
+            0.0020235921256244183,
+            0.009445788338780403,
+            -0.002570174168795347,
+            0.0015123875346034765,
+            0.011306644417345524,
+            0.011497682891786098,
+            0.003254728624597192,
+            0.004577846731990576,
+            -0.018721764907240868,
+            -0.020915169268846512,
+            0.005288933869451284,
+            0.013924578204751015,
+            0.004273600410670042,
+            0.0015309606678783894,
+            0.017603836953639984,
+            -0.006166295614093542,
+            -0.022967062890529633,
+            -0.010117961093783379,
+            -0.01584911346435547,
+            0.039396367967128754,
+            0.01800006441771984,
+            -0.02525952458381653,
+            -0.000530220044311136,
+            -0.009728807955980301,
+            0.010450509376823902,
+            -0.00404364662244916,
+            0.014575524255633354,
+            0.005476434715092182,
+            -0.02426895499229431,
+            -0.016797229647636414,
+            -0.010662773624062538,
+            -0.005257094278931618,
+            0.020080260932445526,
+            0.027014248073101044,
+            -0.004956385586410761,
+            0.0061026159673929214,
+            0.019584976136684418,
+            -0.010938717983663082,
+            0.009849091991782188,
+            0.020957622677087784,
+            0.0031698227394372225,
+            -0.007570781745016575,
+            -0.004553082399070263,
+            -0.016514210030436516,
+            0.024467067793011665,
+            0.014250051230192184,
+            -0.005104971118271351,
+            -0.02465103194117546,
+            0.004110863897949457,
+            -0.02428310550749302,
+            -0.01616043597459793,
+            -0.007301913108676672,
+            -0.014143918640911579,
+            -0.004931621253490448,
+            0.0027594438288360834,
+            -0.004836102016270161,
+            0.011278342455625534,
+            -0.0031592093873769045,
+            -0.004531855694949627,
+            -0.0041851564310491085,
+            -0.025584997609257698,
+            -0.02524537406861782,
+            0.007691065315157175,
+            -0.003005317412316799,
+            0.005596718285232782,
+            0.000650945701636374,
+            -0.013400991447269917,
+            -0.011589664034545422,
+            0.00970758218318224,
+            -0.007988235913217068,
+            0.012813725508749485,
+            0.0031149876303970814,
+            -0.009183995425701141,
+            0.0017193458043038845,
+            -0.0012311364989727736,
+            -0.0059752571396529675,
+            0.019924599677324295,
+            0.00916984397917986,
+            0.011724098585546017,
+            -0.03118879161775112,
+            -0.0021633331198245287,
+            0.008349086157977581,
+            0.01828308403491974,
+            -0.02244347706437111,
+            -0.008313708938658237,
+            0.005179263651371002,
+            -0.011228813789784908,
+            -0.013507124036550522,
+            -0.028952933847904205,
+            -0.0030460015404969454,
+            8.175515540642664e-05,
+            -0.018636859953403473,
+            -0.029886899515986443,
+            0.0009038947173394263,
+            -0.007860877551138401,
+            0.016882136464118958,
+            -0.006643891800194979,
+            0.01790100708603859,
+            0.00870286114513874,
+            0.0036173479165881872,
+            0.011554286815226078,
+            0.0025082635693252087,
+            -0.013266556896269321,
+            0.008426916785538197,
+            -0.02671707607805729,
+            -0.01516986545175314,
+            0.009792488068342209,
+            0.014214673079550266,
+            -0.0026126273442059755,
+            5.074458022136241e-05,
+            0.0058372849598526955,
+            0.024254804477095604,
+            0.013584954664111137,
+            0.012474101036787033,
+            0.003042463678866625,
+            -0.005303084850311279,
+            0.005568416323512793,
+            0.0227123461663723,
+            0.008603804744780064,
+            0.004011806566268206,
+            0.007592008449137211,
+            0.00389859895221889,
+            0.0009932229295372963,
+            -0.005943417549133301,
+            -0.00843399204313755,
+            -0.021537812426686287,
+            -0.008908051066100597,
+            0.019938750192523003,
+            -0.010422207415103912,
+            -0.03005671128630638,
+            0.0006023016758263111,
+            -0.0032777239102870226,
+            0.007988235913217068,
+            0.02595292218029499,
+            -0.014172220602631569,
+            0.004970536567270756,
+            0.005536576267331839,
+            -0.007910405285656452,
+            -0.00614860700443387,
+            0.022967062890529633,
+            0.02639160305261612,
+            -0.0034404604230076075,
+            -0.04800017178058624,
+            -0.01671232469379902,
+            0.004305440001189709,
+            -0.007768895477056503,
+            -0.034047290682792664,
+            -0.015424583107233047,
+            0.002757675014436245,
+            -0.03206615149974823,
+            0.010068432427942753,
+            -0.012948159128427505,
+            -0.01214155275374651,
+            0.020264223217964172,
+            0.03223596513271332,
+            0.005154499784111977,
+            0.00014847487909719348,
+            -0.0214670579880476,
+            -0.0021686397958546877,
+            -0.021382151171565056,
+            0.010174565017223358,
+            -0.008801918476819992,
+            -0.010584943927824497,
+            0.0059611061587929726,
+            -0.012191081419587135,
+            0.013266556896269321,
+            0.010797208175063133,
+            0.024778390303254128,
+            0.01584911346435547,
+            -0.01876421831548214,
+            -0.0024746551644057035,
+            -0.01108730398118496,
+            -0.0016450530383735895,
+            -0.009820790030062199,
+            0.006360871717333794,
+            -0.0031592093873769045,
+            0.011264191009104252,
+            0.015184016898274422,
+            0.004432798828929663,
+            0.00046786724124103785,
+            0.038151077926158905,
+            -2.4487851987942122e-05,
+            -0.0003418349369894713,
+            0.006803090218454599,
+            -0.015750056132674217,
+            0.008837295696139336,
+            0.012191081419587135,
+            0.005051904823631048,
+            -0.007698140572756529,
+            0.006739411037415266,
+            0.03099067695438862,
+            -0.01516986545175314,
+            -0.009014182724058628,
+            -0.023504801094532013,
+            0.007549555506557226,
+            -0.0004656561359297484,
+            0.029971804469823837,
+            0.020533092319965363,
+            -0.027509532868862152,
+            -0.016018925234675407,
+            -0.02179253101348877,
+            0.01267929095774889,
+            -0.011497682891786098,
+            -0.003704022616147995,
+            0.012240610085427761,
+            0.02367461286485195,
+            0.0018838511314243078,
+            -0.014455240219831467,
+            -0.008950503543019295,
+            0.008943428285419941,
+            0.014617976732552052,
+            0.003497064346447587,
+            0.007570781745016575,
+            0.00743634719401598,
+            0.008886824361979961,
+            -0.014801939949393272,
+            -0.012254760600626469,
+            0.00767691433429718,
+            -0.010202866978943348,
+            0.01794346049427986,
+            -0.009403335861861706,
+            0.016768928617239,
+            -0.004645063541829586,
+            -0.011964665725827217,
+            -0.0320095494389534,
+            -0.002292460994794965,
+            0.005037753842771053,
+            0.004627374932169914,
+            0.0034705311991274357,
+            -0.0022217060904949903,
+            -0.013160424306988716,
+            0.004673365969210863,
+            0.005073131527751684,
+            1.875283123808913e-05,
+            -0.008313708938658237,
+            -0.00479718716815114,
+            0.004252373706549406,
+            -0.001031253719702363,
+            -0.0073372903279960155,
+            -0.0059858704917132854,
+            0.018481198698282242,
+            0.0066509670577943325,
+            0.00800238735973835,
+            0.03234917297959328,
+            0.02974538877606392,
+            -0.0016353242099285126,
+            -0.004606148693710566,
+            -0.00023304918431676924,
+            -0.01589156687259674,
+            -0.0057205394841730595,
+            -0.023320838809013367,
+            0.022259512916207314,
+            -0.009665128774940968,
+            0.021820832043886185,
+            0.019457615911960602,
+            0.0039304387755692005,
+            -0.008391539566218853,
+            -0.0011586126638576388,
+            -0.007308988366276026,
+            -0.003560743760317564,
+            0.016641568392515182,
+            0.006742948666214943,
+            0.01226891204714775,
+            0.00030933189555071294,
+            -0.028401045128703117,
+            0.026405755430459976,
+            0.000579748535528779,
+            0.0034015451092272997,
+            0.008908051066100597,
+            0.014129767194390297,
+            0.013209952972829342,
+            -0.01235381793230772,
+            0.01768874190747738,
+            -0.03280200436711311,
+            -0.0033484790474176407,
+            -0.016500059515237808,
+            0.002076658420264721,
+            -0.0018520113080739975,
+            -0.0008145665633492172,
+            -0.05354736000299454,
+            0.0003842879377771169,
+            0.005313698202371597,
+            -0.015339677222073078,
+            0.029349161311984062,
+            -0.020589696243405342,
+            -0.017165156081318855,
+            -0.032858606427907944,
+            -0.010238244198262691,
+            0.018099121749401093,
+            0.005617944523692131,
+            0.00035178486723452806,
+            -0.02660386823117733,
+            0.015750056132674217,
+            -0.0047476585023105145,
+            -0.00743634719401598,
+            -0.029858596622943878,
+            -0.010938717983663082,
+            0.009403335861861706,
+            0.004485865123569965,
+            -0.0130825936794281,
+            0.009898620657622814,
+            -0.002890340518206358,
+            0.010867963545024395,
+            -0.0138467475771904,
+            -0.00420284504070878,
+            0.0010020672343671322,
+            -0.005363226868212223,
+            0.012615610845386982,
+            -0.030537845566868782,
+            0.011391550302505493,
+            -0.01352127455174923,
+            0.0008548084879294038,
+            -0.017858553677797318,
+            -0.027707645669579506,
+            0.0020288988016545773,
+            -0.016132133081555367,
+            0.039990708231925964,
+            0.010188715532422066,
+            -0.01481609046459198,
+            0.016995344310998917,
+            -0.002080196049064398,
+            0.017377421259880066,
+            0.0014955832157284021,
+            0.017023645341396332,
+            -0.011971740983426571,
+            0.021056678146123886,
+            -0.0041745430789887905,
+            0.02156611531972885,
+            0.00600002147257328,
+            0.034047290682792664,
+            -0.02033497765660286,
+            0.0070825726725161076,
+            0.003792466362938285,
+            0.005798369646072388,
+            -0.0026816134341061115,
+            0.021424604579806328,
+            0.017985913902521133,
+            0.019698183983564377,
+            0.0037110981065779924,
+            -0.024084990844130516,
+            -0.006137993652373552,
+            0.004867942072451115,
+            0.023858575150370598,
+            0.011150983162224293,
+            -0.016415152698755264,
+            -0.021056678146123886,
+            0.015226469375193119,
+            -0.007860877551138401,
+            0.006771250627934933,
+            -0.00851889792829752,
+            0.00037544354563578963,
+            -0.013259481638669968,
+            0.012233533896505833,
+            -0.012523629702627659,
+            0.0032335021533071995,
+            0.01043635793030262,
+            -0.027806703001260757,
+            0.01511326152831316,
+            -0.00371817359700799,
+            -0.014313730411231518,
+            0.023320838809013367,
+            -0.015877414494752884,
+            -0.013988257385790348,
+            0.005501199048012495,
+            0.008610880002379417,
+            0.01746232621371746,
+            -0.002761212643235922,
+            0.002669231267645955,
+            -0.020504789426922798,
+            -0.012445799075067043,
+            0.004591997712850571,
+            0.02210385352373123,
+            -0.015820810571312904,
+            0.012431648559868336,
+            -0.018099121749401093,
+            0.01364863384515047,
+            0.016202887520194054,
+            0.007149789482355118,
+            0.00573115237057209,
+            -0.007726442534476519,
+            -0.0011471149045974016,
+            0.032151058316230774,
+            -0.0029876285698264837,
+            0.02704254910349846,
+            -0.014915147796273232,
+            0.009615600109100342,
+            0.00518987700343132,
+            -0.013089669868350029,
+            -0.014589674770832062,
+            0.011108530685305595,
+            0.02428310550749302,
+            -0.009530694223940372,
+            -0.023660462349653244,
+            0.0018484735628589988,
+            0.014264201745390892,
+            0.010337301529943943,
+            0.009403335861861706,
+            -0.001997058978304267,
+            -0.0037358624394983053,
+            0.01484439242631197,
+            -0.0032918748911470175,
+            -0.0286557637155056,
+            -0.01633024774491787,
+            -0.004400959238409996,
+            -0.017490629106760025,
+            0.020745357498526573,
+            0.02367461286485195,
+            0.010507113300263882,
+            0.0037570889107882977,
+            0.005540114361792803,
+            0.018297234550118446,
+            0.01843874529004097,
+            0.013330236077308655,
+            0.009304278530180454,
+            0.012700516730546951,
+            -0.004319590982049704,
+            0.02800481766462326,
+            -0.010860888287425041,
+            -0.01046465989202261,
+            -0.015707602724432945,
+            0.0016636261716485023,
+            -0.005193414632230997,
+            -0.012431648559868336,
+            -0.0069481381215155125,
+            0.004369119182229042,
+            0.013330236077308655,
+            -0.012240610085427761,
+            -0.007149789482355118,
+            0.01016748882830143,
+            0.011971740983426571,
+            0.01403071079403162,
+            0.01719345711171627,
+            -0.01633024774491787,
+            -0.011278342455625534,
+            0.0009834941010922194,
+            -0.02844349853694439,
+            -0.0160047747194767,
+            -0.01767459139227867,
+            0.009580222889780998,
+            -0.014469391666352749,
+            0.03724541515111923,
+            0.028471799567341805,
+            0.0073514413088560104,
+            0.031613320112228394,
+            -0.028401045128703117,
+            0.00970758218318224,
+            -0.014398636296391487,
+            0.007132100872695446,
+            -0.004188694059848785,
+            0.003902136581018567,
+            0.002448122017085552,
+            0.020419884473085403,
+            0.00791748147457838,
+            0.014264201745390892,
+            0.008150972425937653,
+            -0.021934039890766144,
+            -0.00662620272487402,
+            -0.012559006921947002,
+            -0.016401002183556557,
+            -0.02595292218029499,
+            0.013719388283789158,
+            0.005087282508611679,
+            -0.003074303502216935,
+            -0.0002704166399780661,
+            0.006566061172634363,
+            -0.0013195802457630634,
+            0.010443433187901974,
+            -0.027608588337898254,
+            -0.01311089564114809,
+            0.0069269114173948765,
+            0.01449061743915081,
+            -0.02114158496260643,
+            0.0014637433923780918,
+            0.008900974877178669,
+            -0.004602610599249601,
+            0.035660505294799805,
+            -0.014674580655992031,
+            -0.008610880002379417,
+            0.0027806703001260757,
+            0.005062518175691366,
+            -0.017660440877079964,
+            -0.02319347858428955,
+            0.0070648835971951485,
+            0.011575513519346714,
+            0.010867963545024395,
+            -0.028570856899023056,
+            0.02087271586060524,
+            -0.0054339817725121975,
+            0.03832089155912399,
+            0.00401534466072917,
+            -0.013337312266230583,
+            0.003962278366088867,
+            0.0011860302183777094,
+            -0.012360893189907074,
+            0.002060738392174244,
+            -0.000340508297085762,
+            0.003700484987348318,
+            0.011228813789784908,
+            -0.014801939949393272,
+            -0.016967041417956352,
+            0.016910437494516373,
+            -0.011476456187665462,
+            -0.0035395172890275717,
+            -0.00943871308118105,
+            -0.016669871285557747,
+            0.007308988366276026,
+            0.025047259405255318,
+            0.0041745430789887905,
+            0.001242634141817689,
+            -0.011632117442786694,
+            -0.0038596836384385824,
+            -0.008575502783060074,
+            0.0011718792375177145,
+            -0.0207029040902853,
+            0.027396325021982193,
+            -0.01801421493291855,
+            0.002410975517705083,
+            0.021099131554365158,
+            0.008709936402738094,
+            -0.003944589756429195,
+            0.016316095367074013,
+            0.026363302022218704,
+            0.01205664686858654,
+            0.0031539027113467455,
+            -0.00902833417057991,
+            1.648646139074117e-05,
+            0.014957600273191929,
+            0.0015123875346034765,
+            -0.0015663381200283766,
+            0.0032140444964170456,
+            -0.006583749782294035,
+            -0.001859086798503995,
+            -0.017561383545398712,
+            0.005564878229051828,
+            0.02125479280948639,
+            -0.004132090136408806,
+            0.020476488396525383,
+            0.015509488992393017,
+            0.011625041253864765,
+            0.010726453736424446,
+            -0.0012390963966026902,
+            -0.002150951186195016,
+            -0.0021668709814548492,
+            -0.007393894251435995,
+            -0.018297234550118446,
+            0.006470541935414076,
+            0.0034493047278374434,
+            0.01822648011147976,
+            0.020504789426922798,
+            -0.02308027073740959,
+            0.010146263055503368,
+            0.014985902234911919,
+            -0.013011839240789413,
+            0.0235755555331707,
+            -0.02006610855460167,
+            0.0024605041835457087,
+            0.02156611531972885,
+            0.013337312266230583,
+            -0.014271277002990246,
+            -0.009219372645020485,
+            0.004662752617150545,
+            -0.01543873455375433,
+            0.000872939417604357,
+            -0.01478778850287199,
+            -0.010783057659864426,
+            0.014533070847392082,
+            -0.01490099634975195,
+            0.01229721400886774,
+            -0.006969364359974861,
+            0.0070967236533761024,
+            -0.017165156081318855,
+            0.015014204196631908,
+            0.008632106706500053,
+            0.01757553406059742,
+            0.0021279556676745415,
+            -0.02518877014517784,
+            -0.006290116813033819,
+            0.021169887855648994,
+            0.009509467519819736,
+            -0.02497650496661663,
+            0.019613277167081833,
+            0.0020094411447644234,
+            -0.019528372213244438,
+            -0.022401023656129837,
+            -0.012686366215348244,
+            -0.005055442452430725,
+            -0.017264213413000107,
+            -0.0011276573641225696,
+            -0.024792540818452835,
+            0.003972891718149185,
+            0.009106164798140526,
+            -0.011978816241025925,
+            -0.019273653626441956,
+            -0.0002315014135092497,
+            -0.01040805596858263,
+            0.006219361908733845,
+            -0.01132787112146616,
+            -0.019853845238685608,
+            0.0070436573587358,
+            0.012127402238547802,
+            -0.0027647505048662424,
+            -0.005646246485412121,
+            -0.02562745101749897,
+            -0.006919836159795523,
+            0.020179318264126778,
+            -0.003994117956608534,
+            -0.004974074196070433,
+            -0.01703779585659504,
+            0.01108730398118496,
+            0.011165134608745575,
+            0.020901018753647804,
+            0.010931642726063728,
+            0.005674548447132111,
+            0.013634482398629189,
+            -0.0029416377656161785,
+            -0.007156865205615759,
+            -9.645892532716971e-06,
+            0.004284213297069073,
+            0.0069233737885952,
+            0.0016636261716485023,
+            0.020235922187566757,
+            -0.01692458800971508,
+            0.018934030085802078,
+            -0.012290137819945812,
+            0.008794843219220638,
+            -0.01855195313692093,
+            -0.0003155229496769607,
+            -0.01067692507058382,
+            0.00015002264990471303,
+            -0.0038455326575785875,
+            -0.009544845670461655,
+            1.9595809135353193e-05,
+            0.00143986358307302,
+            0.008511822670698166,
+            -0.016783079132437706,
+            0.009615600109100342,
+            -0.010839661583304405,
+            -0.004308977629989386,
+            -0.019542522728443146,
+            0.017448175698518753,
+            0.013344387523829937,
+            0.007308988366276026,
+            -0.008278331719338894,
+            -0.006300730165094137,
+            -0.02282555401325226,
+            0.008582578040659428,
+            -0.0014637433923780918,
+            0.002214630600064993,
+            -0.028301987797021866,
+            -0.004358506295830011,
+            0.0024587351363152266,
+            0.020306676626205444,
+            0.0033661676570773125,
+            -0.010712302289903164,
+            0.008978805504739285,
+            -0.023377442732453346,
+            -0.0030159305315464735,
+            -0.023349139839410782,
+            -0.004075486212968826,
+            -0.005112046375870705,
+            0.010103809647262096,
+            0.008101443760097027,
+            0.01549533847719431,
+            -0.009898620657622814,
+            0.01478778850287199,
+            -0.004793649073690176,
+            0.012049571610987186,
+            -0.0012152165872976184,
+            0.01337976474314928,
+            -0.0003111007681582123,
+            -0.01505665760487318,
+            0.02158026583492756,
+            -0.008738238364458084,
+            0.001741456682793796,
+            -0.0052853962406516075,
+            0.0027346794959157705,
+            -0.017702894285321236,
+            -0.004553082399070263,
+            -0.014455240219831467,
+            0.01957082375884056,
+            -0.022584985941648483,
+            0.004153316840529442,
+            -0.005055442452430725,
+            -0.016075529158115387,
+            0.029434068128466606,
+            0.022896308451890945,
+            -0.0035483618266880512,
+            0.015070809051394463,
+            0.0017989451298490167,
+            0.0015061964513733983,
+            0.016202887520194054,
+            -0.015806660056114197,
+            -0.013025989755988121,
+            0.035066161304712296,
+            0.0176321379840374,
+            0.0004313841927796602,
+            -0.0023260696325451136,
+            -0.0037323245778679848,
+            0.0032193511724472046,
+            0.008023613132536411,
+            -0.010450509376823902,
+            -0.01141277700662613,
+            -0.017051948234438896,
+            0.009997677057981491,
+            -0.006155682262033224,
+            0.004446949809789658,
+            -0.00025891896802932024,
+            -7.617214578203857e-05,
+            -0.01516986545175314,
+            0.022584985941648483,
+            -0.017717044800519943,
+            -0.00602832343429327,
+            0.004687516950070858,
+            0.0207029040902853,
+            -0.02541518583893776,
+            0.01616043597459793,
+            -0.007107336539775133,
+            -0.013429293408989906,
+            0.012573158368468285,
+            -0.015127412974834442,
+            -0.0107335289940238,
+            -0.003175129182636738,
+            -0.008009462617337704,
+            -0.022754797711968422,
+            0.017929309979081154,
+            -0.013627407141029835,
+            0.015650998800992966,
+            -0.00287972716614604,
+            0.000562502013053745,
+            0.002057200763374567,
+            0.0032582662533968687,
+            0.02250008098781109,
+            0.030735958367586136,
+            0.010415131226181984,
+            -0.005465821363031864,
+            -0.021000074222683907,
+            0.012983537279069424,
+            0.0020589695777744055,
+            0.013202877715229988,
+            -0.002253545681014657,
+            -0.002135031158104539,
+            -0.016613267362117767,
+            -0.023122724145650864,
+            -0.0013010069960728288,
+            -0.01761798746883869,
+            -0.02552839368581772,
+            -0.01774534583091736,
+            -0.007047194987535477,
+            0.011349096894264221,
+            0.020405733957886696,
+            0.0009755341452546418,
+            -0.005384453106671572,
+            0.01559439580887556,
+            -0.006941062398254871,
+            0.002080196049064398,
+            -0.006594363134354353,
+            -0.002032436430454254,
+            -0.008200501091778278,
+            -0.021863285452127457,
+            0.029660483822226524,
+            -0.010103809647262096,
+            0.01141277700662613,
+            -0.005472897086292505,
+            -0.00965097825974226,
+            0.005575491581112146,
+            0.00900003220885992,
+            0.024849144741892815,
+            0.006452853325754404,
+            0.03141520544886589,
+            0.020151015371084213,
+            -0.018891576677560806,
+            -0.0006071660900488496,
+            0.022684043273329735,
+            -0.0037570889107882977,
+            -0.004889168310910463,
+            -0.0031786670442670584,
+            -0.017985913902521133,
+            -0.012007118202745914,
+            0.007839650847017765,
+            -0.013471745885908604,
+            -0.006636816076934338,
+            -0.00837738811969757,
+            0.018070818856358528,
+            -0.017773648723959923,
+            0.025698205456137657,
+            -0.006488230545073748,
+            0.005154499784111977,
+            0.031924642622470856,
+            0.014363259077072144,
+            0.019698183983564377,
+            -0.011794853024184704,
+            -0.031075581908226013,
+            -0.00683846790343523,
+            -0.007882103323936462,
+            0.009558996185660362,
+            -0.01062032114714384,
+            0.005204027984291315,
+            -0.02006610855460167,
+            -0.018948180601000786,
+            0.012863253243267536,
+            -0.017476476728916168,
+            0.013924578204751015,
+            -0.009955224581062794,
+            0.027226513251662254,
+            -0.001228483160957694,
+            0.0121698547154665,
+            -0.002614396158605814,
+            -0.017801949754357338,
+            0.01205664686858654,
+            0.004089637193828821,
+            -0.023943481966853142,
+            0.010337301529943943,
+            -0.001864393474534154,
+            -0.014037786051630974,
+            -0.018410442396998405,
+            -0.001228483160957694,
+            -0.005200490355491638,
+            -0.009658053517341614,
+            0.025457637384533882,
+            0.014101465232670307,
+            0.016231190413236618,
+            0.0035359796602278948,
+            0.0016335553955286741,
+            -0.011073152534663677,
+            0.009573147632181644,
+            -0.006803090218454599,
+            0.000612030504271388,
+            -0.0034404604230076075,
+            0.011893910355865955,
+            -0.004252373706549406,
+            -0.01543873455375433,
+            0.002324300818145275,
+            0.008561351336538792,
+            0.0143278818577528,
+            5.947839599684812e-05,
+            0.009587298147380352,
+            0.0009790719486773014,
+            -0.009764186106622219,
+            -0.00046388726332224905,
+            -0.0060884649865329266,
+            0.0069375247694551945,
+            0.02400008589029312,
+            -0.012516554445028305,
+            -0.01681138016283512,
+            0.014504768885672092,
+            0.004033033270388842,
+            -0.017589686438441277,
+            0.018367990851402283,
+            0.004103788174688816,
+            -0.022358570247888565,
+            0.0035412863362580538,
+            0.010662773624062538,
+            -0.009523618966341019,
+            -0.014469391666352749,
+            -0.0010339069413021207,
+            -0.015509488992393017,
+            0.03130199760198593,
+            0.00025073791039176285,
+            0.016910437494516373,
+            -0.01062032114714384,
+            -0.0025967073161154985,
+            -0.013309009373188019,
+            0.013103820383548737,
+            -0.015750056132674217,
+            -0.007273610681295395,
+            0.00035620704875327647,
+            -0.02298121340572834,
+            0.01741987280547619,
+            0.0013425755314528942,
+            0.00794578343629837,
+            0.003436922561377287,
+            -0.010556641966104507,
+            0.009820790030062199,
+            0.0018237093463540077,
+            0.006668656133115292,
+            -0.014915147796273232,
+            0.023618008941411972,
+            0.006413938011974096,
+            0.011235889047384262,
+            -0.01352127455174923,
+            -0.01546703651547432,
+            0.002253545681014657,
+            0.04134920611977577,
+            -0.01306136790663004,
+            0.0002593612007331103,
+            0.01562269777059555,
+            7.081024523358792e-05,
+            0.004156854469329119,
+            -0.0023755980655550957,
+            -0.0034493047278374434,
+            -0.034585028886795044,
+            0.005370302125811577,
+            -0.0006328147719614208,
+            0.008073141798377037,
+            -0.008992956951260567,
+            -0.005430444143712521,
+            -0.0040684109553694725,
+            -0.008625030517578125,
+            -0.00314859626814723,
+            -0.01478778850287199,
+            0.008363237604498863,
+            -0.014533070847392082,
+            -0.028910480439662933,
+            0.03166992589831352,
+            -0.006887996103614569,
+            -0.00533846253529191,
+            -0.021396303549408913,
+            0.007570781745016575,
+            -0.01370523776859045,
+            0.002663924591615796,
+            0.010967019945383072,
+            0.008865597657859325,
+            0.008738238364458084,
+            0.002762981690466404,
+            0.0009516543359495699,
+            -0.004255911335349083,
+            0.010273621417582035,
+            -0.00835616234689951,
+            -0.0005943417199887335,
+            -0.0014186371117830276,
+            -0.02725481428205967,
+            -0.01405901275575161,
+            -0.010592019185423851,
+            -0.007811348885297775,
+            -0.00435143057256937,
+            0.012608535587787628,
+            0.007174553815275431,
+            -0.0017485321732237935,
+            -0.022938761860132217,
+            -0.014745336025953293,
+            -0.0034103896468877792,
+            0.008349086157977581,
+            -0.01838214136660099,
+            0.01064862310886383,
+            0.0016830838285386562,
+            0.009976450353860855,
+            0.014292503707110882,
+            0.00302831269800663,
+            0.0016742395237088203,
+            0.003458149265497923,
+            0.014886845834553242,
+            0.012601460330188274,
+            -0.005801907740533352,
+            -0.021382151171565056,
+            -0.0024605041835457087,
+            -0.007068421691656113,
+            -0.0034687623847275972,
+            -0.012332591228187084,
+            0.03039633482694626,
+            0.0029805530793964863,
+            0.0176321379840374,
+            0.007832575589418411,
+            0.009573147632181644,
+            0.010507113300263882,
+            0.029377464205026627,
+            0.006049550138413906,
+            -0.0029929352458566427,
+            -0.008603804744780064,
+            -0.02141045406460762,
+            -0.013146273791790009,
+            0.009898620657622814,
+            0.013294858857989311,
+            0.03597182780504227,
+            -0.034896351397037506,
+            0.012162779457867146,
+            -0.011101454496383667,
+            -0.008264180272817612,
+            -0.008533049374818802,
+            0.0044717141427099705,
+            0.010068432427942753,
+            0.014370334334671497,
+            -0.0023402206134051085,
+            -0.01175947580486536,
+            0.009070787578821182,
+            0.023547254502773285,
+            -0.012983537279069424,
+            0.03016991913318634,
+            0.027325568720698357,
+            0.023066120222210884,
+            0.01761798746883869,
+            -0.012537780217826366,
+            0.026462359353899956,
+            -6.484030018327758e-05,
+            0.0006261814851313829,
+            -0.018084971234202385,
+            -0.00587973790243268,
+            -0.004294826649129391,
+            -0.004362043924629688,
+            0.013316085562109947,
+            -0.0025471788831055164,
+            -0.0008468485320918262,
+            0.015099111013114452,
+            -0.0037323245778679848,
+            -3.927453508367762e-05,
+            -0.0059752571396529675,
+            0.015127412974834442,
+            0.0002699744363781065,
+            0.019457615911960602,
+            -0.04270770028233528,
+            0.019726485013961792,
+            -0.021396303549408913,
+            -0.003102605463936925,
+            -0.0020890405867248774,
+            0.009523618966341019,
+            0.00764861237257719,
+            -0.007142714224755764,
+            -0.014334957115352154,
+            0.004524780437350273,
+            0.014066088013350964,
+            -0.0006726144347339869,
+            0.005080206785351038,
+            -0.0027541371528059244,
+            0.0017228835495188832,
+            -0.010089659132063389,
+            -0.005862049292773008,
+            5.842812606715597e-05,
+            -0.010570792481303215,
+            -0.008646257221698761,
+            -0.03232087194919586,
+            0.01062032114714384,
+            0.010209942236542702,
+            0.007294837385416031,
+            0.004252373706549406,
+            -0.022018946707248688,
+            -0.012318440712988377,
+            0.012948159128427505,
+            -0.01132787112146616,
+            -0.002570174168795347,
+            -0.0020005968399345875,
+            0.005387990735471249,
+            0.0004705205501522869,
+            0.010160413570702076,
+            -0.003472300246357918,
+            0.010938717983663082,
+            -0.014603826217353344,
+            0.018084971234202385,
+            0.03229256719350815,
+            -0.00533846253529191,
+            -0.019967053085565567,
+            0.000907432462554425,
+            -0.009077862836420536,
+            0.003813692834228277,
+            0.007641536649316549,
+            -0.021962342783808708,
+            -0.02173592709004879,
+            0.005087282508611679,
+            -0.013153349049389362,
+            -0.007783046457916498,
+            0.014830241911113262,
+            0.005221716593950987,
+            0.010839661583304405,
+            0.01751893013715744,
+            0.010889190249145031,
+            0.00840569008141756,
+            -0.026533113792538643,
+            0.0006734988419339061,
+            -0.012686366215348244,
+            -0.00590803986415267,
+            0.002239394700154662,
+            -0.0017591454088687897,
+            0.004591997712850571,
+            -0.00010115749319083989,
+            -0.011200511828064919,
+            -0.0007057808106765151,
+            0.010903340764343739,
+            0.027919910848140717,
+            -0.004146241117268801,
+            -0.0022570835426449776,
+            -0.014257126487791538,
+            0.0010100272484123707,
+            0.036481261253356934,
+            -0.012275987304747105,
+            -0.017434025183320045,
+            0.01757553406059742,
+            0.027452928945422173,
+            0.005954030901193619,
+            -0.0032529598101973534,
+            -0.0027364485431462526,
+            0.00039600671152584255,
+            0.011978816241025925,
+            0.007719367276877165,
+            -0.002692226553335786,
+            -0.002131493529304862,
+            0.004761809483170509,
+            0.035773713141679764,
+            -0.0015203474322333932,
+            -0.010330225341022015,
+            -0.024240652099251747,
+            0.010974096134305,
+            0.00277182599529624,
+            0.004754733759909868,
+            0.002313687466084957,
+            0.023504801094532013,
+            0.019655730575323105,
+            0.014858543872833252,
+            0.00470520555973053,
+            -0.009771261364221573,
+            0.03005671128630638,
+            -0.008172199130058289,
+            -0.004542469047009945,
+            -0.02838689461350441,
+            -0.005087282508611679,
+            -0.015650998800992966,
+            0.008929276838898659,
+            -0.0002221042668679729,
+            -0.005016527138650417,
+            -0.013422218151390553,
+            0.019726485013961792,
+            0.01064862310886383,
+            -0.011073152534663677,
+            0.012608535587787628,
+            -0.027070851996541023,
+            0.02346234768629074,
+            -0.03826428949832916,
+            -0.0071993181481957436,
+            -0.006421013735234737,
+            -0.010514188557863235,
+            -0.02627839520573616,
+            -0.0030353881884366274,
+            0.007372668012976646,
+            0.014250051230192184,
+            0.0022570835426449776,
+            0.0022818478755652905,
+            0.02282555401325226,
+            0.0016415152931585908,
+            -0.01508495956659317,
+            0.017985913902521133,
+            0.03492465242743492,
+            0.012926933355629444,
+            0.00155837822239846,
+            0.015990622341632843,
+            0.0027789014857262373,
+            0.0021633331198245287,
+            0.00788917951285839,
+            -0.0034493047278374434,
+            -0.007938707247376442,
+            0.008582578040659428,
+            0.006990591064095497,
+            -0.02763689123094082,
+            0.003824306186288595,
+            -0.00010408719390397891,
+            -0.009898620657622814,
+            -0.022429324686527252,
+            0.011568437330424786,
+            0.01913214474916458,
+            -0.004224071744829416,
+            -0.006771250627934933,
+            -0.0043337419629096985,
+            0.0006836698739789426,
+            0.006785401608794928,
+            -0.002412744564935565,
+            -0.000515184598043561,
+            -0.0015176940942183137,
+            -0.02368876338005066,
+            -0.010316074825823307,
+            0.013570803217589855,
+            -0.005175726022571325,
+            0.010358527302742004,
+            0.005242943298071623,
+            -0.018792519345879555,
+            -0.016938740387558937,
+            0.006852618884295225,
+            -0.016188737004995346,
+            0.01364863384515047,
+            0.00027948213391937315,
+            -0.0023172253277152777,
+            0.024962352588772774,
+            0.03594352304935455,
+            -0.013188726268708706,
+            0.005119122099131346,
+            -0.019386861473321915,
+            -0.0245378240942955,
+            -0.019004784524440765,
+            0.022613288834691048,
+            0.008306633681058884,
+            -0.018792519345879555,
+            0.012304289266467094,
+            -0.002752368338406086,
+            -0.008865597657859325,
+            -0.0066721937619149685,
+            -0.006845543161034584,
+            -0.008547200821340084,
+            0.020533092319965363,
+            -0.004935159347951412,
+            0.0038950610905885696,
+            0.019047237932682037,
+            0.04075486212968826,
+            -0.008292482234537601,
+            -0.01349297259002924,
+            0.0054339817725121975,
+            0.004878554958850145,
+            0.0018626245437189937,
+            -0.004814875777810812,
+            -0.008016537874937057,
+            -0.0185944065451622,
+            -0.01740572229027748,
+            -0.009615600109100342,
+            -0.0020041344687342644,
+            0.020122714340686798,
+            -0.007768895477056503,
+            -0.01354957651346922,
+            -0.0245378240942955,
+            -0.028188779950141907,
+            -0.006580212153494358,
+            0.02319347858428955,
+            0.017179306596517563,
+            -0.00013200224202591926,
+            -4.3337418901501223e-05,
+            -0.012042495422065258,
+            -0.01968403346836567,
+            0.015509488992393017,
+            0.00037876018905080855,
+            -0.01773119531571865,
+            -0.0065448349341750145,
+            -0.015919867902994156,
+            0.005787756759673357,
+            -0.0051368107087910175,
+            0.004574308637529612,
+            0.05057565122842789,
+            0.0005271245026960969,
+            0.008441068232059479,
+            -0.026575567200779915,
+            0.009106164798140526,
+            -0.01583496294915676,
+            -0.017377421259880066,
+            -0.00012702727690339088,
+            -0.022301966324448586,
+            -0.0030955299735069275,
+            -0.018792519345879555,
+            -0.007386818993836641,
+            0.008568426594138145,
+            0.009799563325941563,
+            -0.013896276243031025,
+            -0.013669860549271107,
+            -0.024693485349416733,
+            -0.0035872769076377153,
+            -0.001539805089123547,
+            0.009311353787779808,
+            0.007825499400496483,
+            -0.03693409264087677,
+            0.002536565763875842,
+            -0.0009507699287496507,
+            0.0005779796629212797,
+            0.01208494883030653,
+            -0.008667483925819397,
+            -0.003063690150156617,
+            -0.0214670579880476,
+            0.006452853325754404,
+            0.007103798910975456,
+            -0.013500047847628593,
+            0.012375044636428356,
+            -0.012863253243267536,
+            -0.007669838611036539,
+            -0.0005426021525636315,
+            0.03379257395863533,
+            0.011794853024184704,
+            -0.0025153392925858498,
+            -0.00654129683971405,
+            -0.011150983162224293,
+            0.02654726430773735,
+            -0.011349096894264221,
+            -0.026405755430459976,
+            0.028910480439662933,
+            0.013245330192148685,
+            0.0005974372616037726,
+            0.020151015371084213,
+            -0.0017936384538188577,
+            -0.01397410687059164,
+            0.009601449593901634,
+            0.004266524687409401,
+            0.012424572370946407,
+            -0.00014460546663030982,
+            0.0214670579880476,
+            0.0018555490532889962,
+            0.0027399861719459295,
+            -0.0036226543597877026,
+            0.004687516950070858,
+            -0.03093407303094864,
+            -0.0006549257086589932,
+            -0.003102605463936925,
+            -0.008398614823818207,
+            0.011207587085664272,
+            -0.002421588869765401,
+            0.006558985915035009,
+            -0.022188758477568626,
+            -0.028485950082540512,
+            0.011342021636664867,
+            -0.016231190413236618,
+            -0.021396303549408913,
+            0.015339677222073078,
+            -0.0007641536649316549,
+            0.00614860700443387,
+            -0.007443422917276621,
+            -0.023278385400772095,
+            -0.019386861473321915,
+            0.02064630016684532,
+            0.012212308123707771,
+            -0.01190098561346531,
+            0.012750045396387577,
+            -0.021113283932209015,
+            -0.012948159128427505,
+            -0.001894464367069304,
+            0.0090566361322999,
+            -0.0019864458590745926,
+            0.01325240544974804,
+            -0.012198156677186489,
+            -0.00334494118578732,
+            0.0031592093873769045,
+            -0.011589664034545422,
+            0.008158047683537006,
+            0.01978308893740177,
+            0.008193425834178925,
+            -0.009339655749499798,
+            0.003422771580517292,
+            -0.011922212317585945,
+            0.013500047847628593,
+            0.015537790954113007,
+            0.018311386927962303,
+            -0.006498843897134066,
+            0.0020412809681147337,
+            -0.01370523776859045,
+            0.0063962494023144245,
+            -0.009827865287661552,
+            -0.004878554958850145,
+            -0.0005333155859261751,
+            0.004921008367091417,
+            -0.011179285123944283,
+            -0.0025772498920559883,
+            -0.026830283924937248,
+            -0.0013823752524331212,
+            -0.020533092319965363,
+            0.005303084850311279,
+            0.023207630962133408,
+            -0.007514177821576595,
+            0.0006045127520337701,
+            -0.0039092120714485645,
+            0.014476466923952103,
+            0.020688753575086594,
+            0.00870286114513874,
+            0.00482548912987113,
+            -0.006081389728933573,
+            0.02179253101348877,
+            0.011603815481066704,
+            -0.017816102132201195,
+            0.00164239970035851,
+            -0.001509734196588397,
+            -0.0014336726162582636,
+            0.011625041253864765,
+            0.004443412180989981,
+            -0.020844414830207825,
+            0.0028602697420865297,
+            -0.010089659132063389,
+            0.007818424142897129,
+            0.014702882617712021,
+            -0.00031906069489195943,
+            -0.02486329711973667,
+            -0.009827865287661552,
+            0.021919889375567436,
+            -0.015070809051394463,
+            -0.0008291598060168326,
+            0.00764861237257719,
+            0.023264234885573387,
+            -0.024792540818452835,
+            -0.006831392180174589,
+            -0.0023455272894352674,
+            -0.003933976404368877,
+            -0.005193414632230997,
+            0.0010489424457773566,
+            0.0214670579880476,
+            0.004415110219269991,
+            -0.004234685096889734,
+            0.005724077112972736,
+            0.006470541935414076,
+            0.02325008250772953,
+            0.0040577976033091545,
+            0.028047269210219383,
+            -0.00943871308118105,
+            -0.0023809047415852547,
+            -0.0019988277927041054,
+            -4.775960405822843e-05,
+            0.0160047747194767,
+            -0.005430444143712521,
+            0.020320827141404152,
+            0.009445788338780403,
+            0.0036934094969183207,
+            0.016514210030436516,
+            0.02319347858428955,
+            -0.006767712999135256,
+            -0.004793649073690176,
+            0.019160445779561996,
+            0.0053667644970119,
+            -0.008858522400259972,
+            -0.02194819226861,
+            0.01141277700662613,
+            0.0014124460285529494,
+            -0.0006956097786314785,
+            0.012106175534427166,
+            0.0024375086650252342,
+            0.006300730165094137,
+            0.008172199130058289,
+            -0.016684021800756454,
+            0.004241760354489088,
+            -0.004698129836469889,
+            -0.01860855706036091,
+            0.010910416021943092,
+            -0.012092024087905884,
+            -5.201595922699198e-05,
+            -0.0010002984199672937,
+            0.014165145345032215,
+            -0.01703779585659504,
+            -0.0019103841623291373,
+            -0.0070719593204557896,
+            -1.3024719009990804e-05,
+            -0.0009162768837995827,
+            0.008320784196257591,
+            -0.006254739128053188,
+            -0.010938717983663082,
+            -0.010938717983663082,
+            0.027509532868862152,
+            -0.004991763271391392,
+            0.003916287794709206,
+            0.019330257549881935
+          ],
+          "embedding": [
+            0.007204997251390758,
+            -0.03173324616100502,
+            -0.020505790884701024,
+            0.0037451608231038273,
+            0.009817085391002229,
+            0.026883177166072393,
+            -0.016477826654817607,
+            0.06949311853773875,
+            -0.01224876667472692,
+            0.031952566873802865,
+            0.013633671951171586,
+            -0.005041437355104989,
+            0.02663392750069342,
+            -0.022099132865267587,
+            0.006665702000169926,
+            0.0030467129715919033,
+            0.007931802508415542,
+            0.009984477355413895,
+            -0.010362836464183985,
+            -0.03180848181251013,
+            -0.013542377333474449,
+            -0.013572345022577053,
+            -0.03979985300694044,
+            -0.007698412130292187,
+            -0.005953452234356532,
+            0.018890228917932694,
+            0.02217446994660065,
+            0.005158357501784182,
+            -0.05468694715835175,
+            0.00789994048345594,
+            0.017130184174786147,
+            -0.0026591475669416114,
+            -0.02081141846859738,
+            -0.008403928318906045,
+            -0.030925346724945978,
+            -0.03855023691059801,
+            0.038853354785097736,
+            0.01612011653434478,
+            -0.035635702395196575,
+            0.026266623757525934,
+            0.043450373922427316,
+            -0.007254334835408145,
+            -0.015376804869420366,
+            -0.0017276202867766907,
+            0.004650414207094434,
+            -0.002365514361535186,
+            0.011906139905776183,
+            -0.0005664194302009289,
+            -0.0191723447249157,
+            -0.025546124518037998,
+            0.01025005257280377,
+            -0.014436532079436437,
+            0.029277529346694118,
+            -0.0045414983847733344,
+            -0.02260315456913697,
+            0.01717918656248626,
+            0.013666132349359846,
+            0.057804090189845364,
+            0.056600576093100735,
+            0.051208022172527734,
+            0.03553760896972431,
+            0.02094599510567085,
+            -0.02438674472832137,
+            -0.0027797634437333432,
+            0.03614248430646102,
+            -0.026729308357113124,
+            -0.002251394534452815,
+            0.018929251842788906,
+            0.00689697483318378,
+            -0.026603606434115974,
+            -0.0541443728344337,
+            0.027164630314707807,
+            -0.004335272389193875,
+            -0.013967321193992958,
+            0.003323121377857836,
+            -0.0245295956412367,
+            0.024489335129134787,
+            0.06212059558640156,
+            0.009214058207162371,
+            -0.002082293435905329,
+            0.013659934201796686,
+            -0.003344037927706573,
+            -0.00689041317919184,
+            0.020235785390418023,
+            0.0212739457082623,
+            0.0016782346585238381,
+            -0.013484221664059555,
+            -0.06465582506604173,
+            0.011882797526607326,
+            -0.032970655964890155,
+            -0.04027700439305653,
+            0.025396847912528105,
+            0.013861157517716707,
+            0.00604596157583048,
+            -0.00744611407420936,
+            0.04038084298991919,
+            0.021507532290328625,
+            0.005983822806623904,
+            -0.012711110369915742,
+            0.02410138142963901,
+            -0.018372791517880082,
+            0.021016674128298246,
+            0.013387285956415756,
+            0.02039383525127615,
+            0.010700292312227496,
+            0.0053934660497349735,
+            -0.014807678014449691,
+            0.009422805142740376,
+            0.0414357625490679,
+            0.12206398267002624,
+            -0.010877509834324155,
+            -0.011079302503786522,
+            -0.04107714853756596,
+            0.006631397986165343,
+            -0.031914267104222384,
+            -0.0179965295457466,
+            -0.057530036803978266,
+            -0.0320488164246567,
+            0.008208202363320509,
+            -0.0020203090915064162,
+            0.03502055780919601,
+            0.04464599164916666,
+            0.019332152079287082,
+            0.017133943087431935,
+            -0.039885565008919065,
+            0.04703059988373572,
+            -0.029370696964304416,
+            0.0008494086835971101,
+            0.0014082189719987074,
+            -0.030973762582071615,
+            -0.006561157852024518,
+            0.011634653995510726,
+            -0.019896826887411713,
+            -0.034275921680252924,
+            0.026274465530039866,
+            -0.007728234125236442,
+            0.04814514040690998,
+            0.06550407296280117,
+            -0.02749051703615343,
+            -0.010041099502187074,
+            0.02019601967763537,
+            -0.01871701432558228,
+            0.008532829759864928,
+            -0.020164138072967613,
+            -0.045196925901010855,
+            -0.004523517988516156,
+            -0.002113469149506399,
+            0.07929650663710812,
+            -0.01111269708042164,
+            0.03358254444720164,
+            -0.019973382759721527,
+            -0.0007145592863325417,
+            -0.0011740862928390678,
+            0.02166182480535788,
+            0.004543228023398081,
+            -0.013377688428761516,
+            -0.01758250073647576,
+            0.009548292014596834,
+            -0.04753617258004735,
+            -0.007125744019792479,
+            -0.009930358894299303,
+            -0.007931672044235996,
+            -0.024561936537429774,
+            0.012979195012881556,
+            0.009209145250061479,
+            0.04586756662401364,
+            0.04311807301838248,
+            -0.0036241105681042675,
+            -0.06579256720896123,
+            0.04967191755416882,
+            0.015475886231466817,
+            0.03157651656768371,
+            0.02515941109300714,
+            0.01773554572598468,
+            0.005824418191900522,
+            0.016317635547562007,
+            0.007650872352957098,
+            0.010015166465492963,
+            -0.033203307142867844,
+            0.040272188258443496,
+            -0.013408423788025047,
+            0.01036004294342104,
+            0.012391622448703885,
+            0.04214857862444935,
+            0.03418032388760535,
+            0.015263018079889338,
+            0.018786704580603317,
+            0.02704685343985817,
+            0.015348111464412869,
+            0.0013031600434754704,
+            0.03241427528588285,
+            0.011689924942195664,
+            0.04544406114742012,
+            0.016186615914968734,
+            0.011624500626103958,
+            -0.008272953078864385,
+            -0.02363080189189375,
+            0.02784298119317616,
+            -0.010585980875564109,
+            0.01455013604305903,
+            0.01749228557615982,
+            -0.054742056843690476,
+            -0.005876213780171779,
+            -0.010556605098730201,
+            0.003967456397644932,
+            0.014663888851767762,
+            0.0015548476559947807,
+            -0.003858367677774835,
+            0.012504286824484335,
+            -0.025806882498628832,
+            -0.014216299453397403,
+            0.0031620924516417956,
+            -0.006448390918604307,
+            -0.02785577502346934,
+            -0.00766402687921159,
+            0.005275000635711884,
+            -0.0090448751942,
+            0.019702662630533,
+            0.007009091837467867,
+            -0.01847307100043082,
+            -0.0006774105206839772,
+            0.032952656101827446,
+            0.015900033044760432,
+            -0.02170121261931376,
+            -0.017964879081469612,
+            -0.005062849745084668,
+            -0.03171847232054206,
+            -0.038177398521606544,
+            0.029426291950046455,
+            -0.025687201683643784,
+            0.030158867757101797,
+            0.015780411479509624,
+            -0.007011484049169482,
+            -0.012114023533118393,
+            0.020661407942701628,
+            -0.025067530380881376,
+            0.009777190971413269,
+            -0.004160771288889298,
+            -0.028825369485727344,
+            0.008307290529469729,
+            -0.014415903546305744,
+            -0.007561235629544945,
+            -0.03201704135271719,
+            -0.015182869388110948,
+            -0.00501561450194676,
+            -0.0003863746169672397,
+            -0.01073403445816663,
+            -0.03321103354401307,
+            -0.002304357889348034,
+            0.017001213960120047,
+            0.00028222146729658097,
+            -0.04002882852650754,
+            0.014509588052058002,
+            -0.004150541038999649,
+            -0.034333837380045466,
+            0.0564137596224324,
+            0.03584898691608747,
+            -0.006830698606679202,
+            0.006121456445207273,
+            -0.04505674213649008,
+            0.04358467937307492,
+            0.010847859073780811,
+            -0.04918983713702805,
+            0.049190442935189385,
+            -0.00528960443684769,
+            0.01419195371129309,
+            0.040316024601793406,
+            0.004600843996974165,
+            -0.006780970176765864,
+            0.005206651850992678,
+            0.03397517915308201,
+            -0.010037223444627569,
+            -0.027377913828275806,
+            -0.03328136706915221,
+            0.02562568440972934,
+            -0.03485770844248482,
+            -0.020069979230687374,
+            -0.0052041408068332524,
+            0.029818953377223505,
+            0.028789460147370508,
+            0.004071274130619537,
+            0.01928375754300958,
+            0.0014970947734202088,
+            0.010502599795027616,
+            0.016914145702515263,
+            0.011942576738188502,
+            -0.014909953528043828,
+            -0.003432489584767786,
+            -0.05733768990013957,
+            0.0030710780388681002,
+            -0.02627280695622545,
+            0.007930906480371135,
+            -0.031227917075485834,
+            0.013132797714872345,
+            0.002514351081823354,
+            -0.007152021475569949,
+            -0.0066245205132327055,
+            -0.005279224195782313,
+            0.028512607222275155,
+            -0.00921960825771865,
+            -0.0359704105589433,
+            -0.021347597388353182,
+            0.011200044645812517,
+            -0.0004171294784927543,
+            -0.016784468873218786,
+            -0.0018746544944553878,
+            0.007749103417240419,
+            0.02390272097796833,
+            -0.005128540108845805,
+            0.006397849195755672,
+            0.01214922062611769,
+            0.01163371266250103,
+            0.012307186722130844,
+            -0.004982589662882946,
+            -0.016343724553037307,
+            0.019806887143115643,
+            0.013139657463054087,
+            0.0205801609265216,
+            -0.03767676007629612,
+            -0.0249879011550972,
+            0.05547392586363707,
+            0.005903313549319051,
+            -0.05568806298774435,
+            0.018827693208840233,
+            -0.007931178941307645,
+            0.008183992600768387,
+            -0.010319022946041053,
+            0.05111352384595183,
+            -0.024897203290807753,
+            -0.02096678205188722,
+            0.006436523335212348,
+            -0.04917084523640266,
+            -0.010459284852442706,
+            -0.011138558192296276,
+            0.014789370043397917,
+            -0.03105725978752438,
+            -0.004934745004818259,
+            0.015595801375817122,
+            -4.687585727052446e-05,
+            0.0028886067775750254,
+            -0.00850352807499054,
+            -0.0034885579171087815,
+            0.016339681487281293,
+            -0.0034050317046826813,
+            -0.008309570448487812,
+            -0.003615165868022203,
+            -0.027638409919916757,
+            0.023352910161586186,
+            0.026364511529991978,
+            0.001249936082895606,
+            0.001987948967948376,
+            0.01756629246122769,
+            0.004656075485833687,
+            -0.006752579977941239,
+            -0.013681608638716132,
+            -0.03405936202918687,
+            -0.0401867651078737,
+            0.022991180183813725,
+            -0.02318908140203705,
+            0.018299512436825923,
+            0.00904081847012441,
+            0.019697091729447287,
+            -0.019236130984371835,
+            0.015159917756323988,
+            -0.0387297415849285,
+            -0.018793498297161176,
+            -0.005256288213225733,
+            -0.010414697711744128,
+            0.0007039894599589104,
+            -0.01739645696111503,
+            -0.013728845328097506,
+            0.022709522955937343,
+            -0.022789243441781803,
+            -0.0036763802589970026,
+            -0.01429671717544519,
+            0.0023674126354206932,
+            0.049448298344058644,
+            -0.009926033487593551,
+            -0.023371285193090733,
+            0.020846792916650956,
+            -0.026996519120163966,
+            0.011246908457705283,
+            0.04091365159640201,
+            0.005469162850197613,
+            0.005640919201612091,
+            0.029575753750229278,
+            -0.0007860452837141593,
+            -0.0052805237594885516,
+            -0.009782745443933828,
+            0.021178503581190618,
+            -0.0009176512672832309,
+            -0.05245650420332918,
+            -0.007698364493410058,
+            -0.039027418406703554,
+            0.005448840777995117,
+            0.017581625031764185,
+            0.025522517097715556,
+            -0.020775492472580457,
+            -0.015906320383873178,
+            -0.000643669839811819,
+            -0.0086896084429604,
+            0.0029399233482444638,
+            0.014381240985647907,
+            0.021086693776922936,
+            0.002436575081260609,
+            -0.004913514868101904,
+            -0.011582101006213432,
+            0.014400297663309155,
+            0.0467181039341492,
+            -0.01722692180137712,
+            0.0011622224560709315,
+            -0.013036540368441797,
+            -0.06208454638917413,
+            -0.019160212331432766,
+            0.0351496273293468,
+            0.02429814055116254,
+            -0.009896662169762772,
+            -0.01647101145477158,
+            -0.002915821099802562,
+            -0.03557863734177486,
+            -0.01786433999615293,
+            -0.010699364747311715,
+            0.02390901423299708,
+            0.007213809581960646,
+            0.0005601364043388203,
+            0.00916614601011734,
+            0.014737491634516238,
+            0.019303599465763203,
+            0.008864535607492504,
+            -0.0575839606069681,
+            0.018051837786406608,
+            0.006315830080225406,
+            0.02525619013082567,
+            -0.015671924543113065,
+            0.003988943108210209,
+            -0.03179878281917948,
+            -0.029440641244429775,
+            0.040586322748158446,
+            -0.03418722657593859,
+            -0.030650542761032924,
+            0.006566745611226466,
+            0.009571089937153852,
+            0.004491158558315032,
+            -0.011352813042040581,
+            -0.02535505987721779,
+            -0.013381948772812133,
+            -0.030918676056735445,
+            -0.02192708443081135,
+            -0.005232667170255381,
+            -0.03342408726436631,
+            -0.07498634115550039,
+            -0.006245638452568239,
+            -0.049946896465756174,
+            -0.03900385398225758,
+            0.04458524125677807,
+            -0.006106489377959481,
+            0.02398410968928338,
+            -0.04615183642824994,
+            -0.01165382599292015,
+            -0.04890013373371541,
+            0.013624324077017377,
+            0.01476398305411351,
+            0.020202554298470666,
+            0.00014553149242532853,
+            0.036224316512987846,
+            0.00021498496267774312,
+            -0.05055208658013767,
+            -0.057482169168904106,
+            -0.003960451575823462,
+            -0.010217078509789989,
+            -0.005593738972150311,
+            -0.034090261748104665,
+            0.05835341009022349,
+            0.0116359661127526,
+            0.04203678184748103,
+            0.03447818335186531,
+            -0.006747564131379555,
+            -0.025756962754841885,
+            -0.028123416424745064,
+            -0.008073625379614702,
+            0.047544241458556825,
+            -0.029605671083240564,
+            0.008171081332993893,
+            -0.026007452205474395,
+            0.0067084589781962805,
+            -0.0070645448617940995,
+            -0.0012298101311792947,
+            0.012372286141738867,
+            0.006377319686763927,
+            -0.015024537877377194,
+            0.007934745664790507,
+            0.026041755081605937,
+            -0.007303731324541215,
+            0.0407000170257006,
+            -0.012990703097937777,
+            -0.033189084013863494,
+            -0.022525679964848666,
+            0.01657717698191603,
+            0.03440284885162039,
+            0.010230633738702899,
+            -0.008707808754455952,
+            -0.0363777020540216,
+            0.011184076269109602,
+            0.031774012431843524,
+            0.01504703222643543,
+            -0.022509833303978505,
+            0.025230974050270107,
+            0.017502662694813216,
+            -0.018700579139537456,
+            -0.03511734747709348,
+            0.015162183119327901,
+            -0.017711638132862063,
+            0.018190335050681287,
+            -0.01590174153215615,
+            0.011103110869700727,
+            0.0025189370696524936,
+            -0.00920305384736575,
+            0.01929652151514404,
+            -0.006196715938358087,
+            0.011426611271994685,
+            0.007058186102245829,
+            -0.018191352599117545,
+            0.0027202185072220937,
+            0.057310566886380025,
+            -0.000531639834386418,
+            0.01494975681909879,
+            0.03845692752400366,
+            -0.001971953300066856,
+            0.045554909803324765,
+            -0.02532332999916258,
+            -0.042616337063845953,
+            0.02633963974172818,
+            -0.02970705577683783,
+            -0.01624154998005974,
+            -0.02620993449678864,
+            -0.021490817420853802,
+            -0.004038424045768319,
+            0.009940887982818171,
+            -0.02692933845395842,
+            -0.023801570856681796,
+            -0.001948460540662095,
+            0.02290911497874141,
+            -0.03300532378029689,
+            0.006525488380791647,
+            0.007039306647666766,
+            -0.003685585408710752,
+            0.03183735852538168,
+            -0.027197064244321705,
+            0.006861928286688311,
+            0.011209869849682412,
+            -0.001094897323854524,
+            -0.005328676992017907,
+            -0.004007474089818478,
+            0.010796946231051235,
+            -0.030543173497906445,
+            -0.012879972141706173,
+            0.00935894527375688,
+            0.0017573071057611829,
+            -0.010050882212831222,
+            -0.03102132152986303,
+            -0.002915141291985554,
+            0.007634137769426993,
+            -0.011094093612593696,
+            0.01199609810193786,
+            -0.010458016758814742,
+            -0.04929347607956551,
+            0.010973132159729983,
+            0.028148117094907874,
+            -0.025504019123994758,
+            0.012470127357818321,
+            0.009182729379865986,
+            -0.016020133756226275,
+            0.016320111225714855,
+            -0.012051271446461976,
+            0.024098195821827578,
+            0.04317620443481784,
+            0.024592886889234396,
+            0.007727787037954994,
+            0.00771746776156926,
+            -0.0003147538723468374,
+            0.022572868970938624,
+            -0.016683340264320543,
+            0.013672085891597506,
+            0.020827021901755415,
+            0.02092309219721837,
+            0.0057130785714458055,
+            -0.003901728981013356,
+            -0.0036968415923910314,
+            0.002227024825579113,
+            -0.02106678364798051,
+            0.014578546755457675,
+            0.03128101864424301,
+            -0.005234627629617124,
+            -0.022394744692442837,
+            0.002430617645746389,
+            -0.015780389432069438,
+            4.7277885648917e-05,
+            0.013357803020920043,
+            0.013777651543690907,
+            -0.027713499926132385,
+            -0.0018999112228756628,
+            0.028364630573666852,
+            0.009146793705085586,
+            -0.009891297866910313,
+            -0.01644269527910728,
+            -0.0003903768506971414,
+            0.006512217314869576,
+            0.02408785204739448,
+            0.0018406499545367782,
+            -0.007360473750031044,
+            0.01484497577088404,
+            0.016723277095327158,
+            0.003629911330658209,
+            -0.0004820567356447713,
+            0.02394406451074397,
+            0.023794019615473026,
+            0.001632266615911801,
+            -0.02064379152040184,
+            -0.02755693521684878,
+            0.027335938400284726,
+            0.02769248863677192,
+            -0.006832353240623308,
+            0.027324973360715134,
+            0.003937207909967385,
+            0.013580154030133622,
+            0.02076818768224654,
+            -0.01784151382870951,
+            -0.017617903734311997,
+            0.0017316476451235418,
+            -0.009605294810027804,
+            -0.009829196696575069,
+            0.004891780008779742,
+            -0.011655402383771472,
+            -0.01124379942906224,
+            0.043205454954628145,
+            0.004713701851783307,
+            0.003560551284131962,
+            0.007894840447501912,
+            -0.018881214526749082,
+            -0.00945538175144614,
+            0.0019040259741796817,
+            0.029505863720301916,
+            -0.001400002868254808,
+            0.03315866433970851,
+            -0.00830542045504805,
+            0.008602408964961391,
+            -0.03202191233287069,
+            -0.006978289755114341,
+            0.0326499698728755,
+            0.015314305362524091,
+            0.015104436300213667,
+            0.0021257042489266787,
+            -0.0010840650756005028,
+            0.02434348355847526,
+            -0.007174433778817674,
+            -0.04839373063710409,
+            0.022696994450018916,
+            -0.004779511451775743,
+            0.023588216956749154,
+            0.0382733980620746,
+            -0.004232286553944853,
+            0.003815507826545929,
+            0.023531223314846965,
+            -0.01887650634213847,
+            0.015055403379782985,
+            -0.01918944132635701,
+            -0.017597900721414636,
+            0.02176630010543207,
+            -0.008896608357798277,
+            0.01732387427588561,
+            0.01764363756727683,
+            -0.012180043444768312,
+            -0.019003360499321047,
+            0.01229139850966224,
+            0.0025710091163655626,
+            -0.017075658234265562,
+            0.01797412486853335,
+            0.018372010308966135,
+            0.006723179570342314,
+            0.015417677746651683,
+            -0.00047634684109332134,
+            0.017608296628666153,
+            -0.001049984114818522,
+            -0.008416990841821452,
+            0.01827827907328822,
+            -0.009568717277044265,
+            -0.01658089563620308,
+            0.009808478422454319,
+            -0.05193977747210182,
+            0.010644335931025957,
+            0.0016781339270411392,
+            -0.01562285100294917,
+            0.00405036330963539,
+            0.007102724536527262,
+            0.005195069570261247,
+            0.0028773630901681767,
+            0.004610474981005765,
+            0.042921486858787584,
+            0.013213672397112759,
+            -0.0064030584930089415,
+            -0.010711457508461704,
+            -0.017480309425763254,
+            -0.027151012983864797,
+            -0.005204425551873271,
+            0.009927884476991038,
+            0.022601230152836282,
+            0.008046297278661014,
+            -0.0031858480011267704,
+            -0.02589628103025927,
+            0.019723430185137954,
+            0.03604835658149313,
+            0.062423099201768716,
+            0.010930268386637612,
+            0.02925564269324381,
+            0.02439517070384333,
+            0.023930432096006114,
+            0.013036828091946605,
+            0.006910311087503408,
+            -0.032995841694293995,
+            0.012813650252889403,
+            0.01333714655191133,
+            0.03678966264802025,
+            0.012571138353467183,
+            -0.05610151838497368,
+            3.0777966587107344e-05,
+            0.009791774904454618,
+            0.041959488802722524,
+            0.01956296954085397,
+            0.000155217569370113,
+            -0.007285602454264675,
+            0.031547870838711856,
+            -0.026518385135879206,
+            -0.012299593280363843,
+            -0.0026551378470968594,
+            -0.006190117573235779,
+            -0.003807907001161504,
+            -0.004991957475076049,
+            0.036983035940964804,
+            -0.015648565314743963,
+            -0.019256463332683493,
+            0.0004300362961407231,
+            -0.0015686065121077472,
+            0.0011311524136286674,
+            0.0032236641835276464,
+            0.013795345278976164,
+            0.015385029265915279,
+            -0.004800448639353721,
+            0.026514926384796363,
+            -0.008699453553651681,
+            0.022186817580012775,
+            -0.013427553742493343,
+            -0.011927259975371873,
+            0.017776228476138103,
+            -0.021243797085501972,
+            -0.01667280589187595,
+            -0.003688045116580565,
+            0.006189970874536939,
+            0.0020911968266404467,
+            0.0006097761986763103,
+            -0.004685468039463817,
+            -0.01011357666327168,
+            0.022631082517271542,
+            0.012398000826637277,
+            -0.006721588925828951,
+            0.018142966317821054,
+            -0.0021330638043572654,
+            -0.011869184647489572,
+            0.025073455250141923,
+            -0.0015392945266926425,
+            -0.0337085890598252,
+            0.01267302113143573,
+            -0.013438601165482187,
+            0.02673290895281225,
+            -0.009586762706902736,
+            -0.011208527168019762,
+            0.019144928964011168,
+            -0.0017591316941248897,
+            -0.005459150067129732,
+            -0.029230079356228025,
+            0.004679497471366242,
+            -0.011723195942933286,
+            0.015899377439148873,
+            0.0022054234777254627,
+            0.029187759252864872,
+            -0.018290694895057157,
+            0.00792533824593637,
+            0.011719198071061566,
+            0.014612375997487872,
+            0.005419943379355563,
+            0.010515265723144554,
+            -0.006893314445458116,
+            -0.04424495426064596,
+            5.8241472764594054e-05,
+            -0.005719143565673274,
+            0.0038220021965914754,
+            0.013106940982933836,
+            -0.01203955335201393,
+            0.011260626096963215,
+            -0.0279896016489392,
+            -0.0295228261396555,
+            0.036474811902902415,
+            -0.04900747926861378,
+            0.019261089742811332,
+            0.00014104646541594166,
+            -0.03057197731717424,
+            -0.04331886560073797,
+            -0.03232957159604667,
+            -0.019401402365464362,
+            0.004102283356333861,
+            0.0002610070704924167,
+            0.0012280372705500212,
+            -0.014025012323929194,
+            -0.014135990809256075,
+            0.013269444858490211,
+            0.00992155580941576,
+            0.011225162609266308,
+            -0.006545044517282357,
+            0.002368862850663672,
+            -0.02063183105457681,
+            -0.026981383275694142,
+            -0.02774194980145225,
+            0.013337303400678109,
+            -0.0025902548982556788,
+            -0.02794339093668326,
+            -0.025606606546251152,
+            -0.0029852498102770587,
+            -0.01906873442473352,
+            0.006508845592362198,
+            0.03268861529625085,
+            0.0355975143313428,
+            0.010066906387952277,
+            0.01690250467284985,
+            -0.017201544758038183,
+            0.048345309114140714,
+            -0.019031171948182872,
+            0.03188329748476941,
+            -0.0005845169448621133,
+            -0.024140202991373587,
+            0.00011597263700677706,
+            0.03203623528903764,
+            -0.019285565857567896,
+            -0.01332755507465857,
+            -0.010914187315481855,
+            -0.003374912271519852,
+            0.02419326154642231,
+            0.02443825785057301,
+            -0.0037923303809811746,
+            0.00022800399604730175,
+            0.015893265651968885,
+            0.003154527829977343,
+            0.007688529734165207,
+            0.030623379985627763,
+            0.014052282574575454,
+            0.01127235345371133,
+            -0.04164707563955819,
+            -0.011811587719246128,
+            -0.003763086694358232,
+            0.005909410979841885,
+            0.002913220159138154,
+            -0.002671613084376487,
+            0.0045765194162907885,
+            -0.007241703279856166,
+            -0.005855949422953439,
+            0.0016973541436777935,
+            -0.001576326424468957,
+            0.008393873711385013,
+            -0.0037217094191171083,
+            -0.03498879678492834,
+            -0.026096985039773694,
+            -0.021280075788049788,
+            0.006437497348154639,
+            0.0070214556591898375,
+            -0.00972721128392027,
+            0.01949553762684561,
+            -0.0029404319702066146,
+            0.000271037702827666,
+            -0.010734867909488861,
+            -0.0029784341515646075,
+            -0.01206464664516104,
+            -0.019544778160755956,
+            0.00035139045159739705,
+            0.013457601448003152,
+            0.00025947376767512006,
+            -0.03165522602526639,
+            0.011979811705146604,
+            0.006642849985850598,
+            0.013181162886675883,
+            -0.024499404540711422,
+            -0.02310714308893576,
+            -0.028028912907092744,
+            0.010570122662701183,
+            -0.0066866321204157725,
+            -0.024617571284817428,
+            -0.004097807033448707,
+            -0.0074576460628928256,
+            -0.0020417256533259887,
+            -0.02470953739335365,
+            -0.011408461668188081,
+            0.023537349682260345,
+            -0.006373652338136973,
+            0.01936338676879376,
+            -0.017636435070085086,
+            -0.017036091876814843,
+            -0.002198685044404603,
+            0.015584758061011048,
+            -0.038943923931820196,
+            -0.002076529978164001,
+            -0.008452513321945455,
+            0.006551055965075869,
+            -0.028840587857625097,
+            -0.007128279504360441,
+            0.0199397499488538,
+            0.012424319366271584,
+            -0.03518875279748495,
+            0.013596027106451115,
+            0.022191403935861113,
+            0.01218507466930854,
+            -0.0038551718784437783,
+            0.03910075115429626,
+            -0.0025215641203030336,
+            -0.005422688956956598,
+            0.016759888275451905,
+            0.04462538468871483,
+            0.023708097134660096,
+            -0.03152743757744915,
+            -0.008668774531128668,
+            -0.015224825785549947,
+            0.004102648962663789,
+            -0.025867795193966987,
+            0.004676262922216584,
+            -0.011739753544166856,
+            0.01645951757031186,
+            0.011253400919230348,
+            0.013958707917471253,
+            0.0376596076542735,
+            -0.005077354977296842,
+            0.02668910096392324,
+            0.0010332284250006117,
+            -0.00856723931523911,
+            0.0014431148305642308,
+            -0.01412579094241589,
+            0.029892393553325205,
+            0.02269599036836208,
+            0.014421409701746036,
+            0.029248195108770955,
+            -0.03588867295960255,
+            0.02458655544632528,
+            -0.0010013539341895672,
+            0.008368648583431747,
+            0.018150734970019018,
+            0.02628022565029391,
+            0.015423867373844644,
+            0.012477080028047823,
+            -0.02513018750675568,
+            -0.022180001014682045,
+            -0.0027550350630288997,
+            -0.031944843677988374,
+            -0.0014668093006914267,
+            -0.009789309083948147,
+            0.026386027599666254,
+            0.0046988411391149,
+            -0.019803160322638583,
+            -0.003931806845108628,
+            0.004482097428470392,
+            -0.00244660100774939,
+            -0.010441732346892935,
+            -0.004649318716095894,
+            -0.01869464992812615,
+            -0.017838643679294407,
+            0.023677878122153204,
+            -0.023958455640817666,
+            -0.025238725541146534,
+            0.014582798102040157,
+            0.0013223590277629197,
+            -0.002295641913710489,
+            0.0008238518461567685,
+            -0.02115533275282518,
+            -0.04214275933987015,
+            -0.027817083427786927,
+            0.0037588803464689412,
+            0.03565056002940463,
+            -0.026788865866928923,
+            0.014162550819567465,
+            -0.011336583458519,
+            0.019175861037658344,
+            -0.005127229083828615,
+            0.015531116410874209,
+            -0.018251290727445846,
+            -0.044630374763679266,
+            0.021833307809708984,
+            -0.012087009688208446,
+            0.02720742725750304,
+            0.021497809277370574,
+            -0.026937653159100156,
+            -0.013572733244188407,
+            0.01358214612023832,
+            0.0020408253706382332,
+            -0.009652962552750464,
+            -0.0063411929391088626,
+            -0.010533309167065193,
+            0.003901271588567942,
+            -0.04496345296706673,
+            -0.008761188928422701,
+            0.06948704056954039,
+            0.005916180375965137,
+            0.03157905519638486,
+            0.014668818318138927,
+            -0.028448496686381258,
+            -0.04008949536631331,
+            -0.00462253589823598,
+            -0.0003701522833020314,
+            -0.00024410762276269235,
+            -0.01850585249309217,
+            0.017699615637510956,
+            -0.002712069912576265,
+            -0.009767338815933488,
+            -0.013119464222297351,
+            -0.013743610418034631,
+            -0.030375365173543157,
+            -0.0030316327877227783,
+            -0.008632689470358149,
+            -0.006565457264917239,
+            -0.0328089275944709,
+            -0.011713996290006943,
+            0.029773910045828737,
+            0.02740100904353622,
+            -0.0061926915497348915,
+            0.00638737826122608,
+            -0.010244154924811975,
+            0.0010396203272677069,
+            -0.001264454445070139,
+            -0.004059969507786729,
+            0.008715730726831364,
+            0.0014767015415289076,
+            0.031074767756160294,
+            0.004761858344125113,
+            0.00987870172044607,
+            -0.059926117793699066,
+            -0.009294021263133636,
+            0.021188432871397475,
+            0.016602826531924377,
+            -0.009125784136292983,
+            0.013946188378030194,
+            -0.0017670794602549428,
+            -0.006837334179368295,
+            -0.028290055353601393,
+            0.024556541126233806,
+            0.04543639597357669,
+            0.0031311343860416473,
+            -0.011199639773247286,
+            0.014965677855595444,
+            0.0018248533506196013,
+            0.015507721118783542,
+            -0.011409402378400194,
+            -0.01338737990904486,
+            0.03063410115541065,
+            0.009305362935335817,
+            0.0025578631174065385,
+            0.012727648343848194,
+            -0.0038680554900795763,
+            -0.01173215289303824,
+            -0.01568862495293563,
+            -0.004101703129244437,
+            -0.03649200408657064,
+            -0.0022107849490954137,
+            0.009659780679763728,
+            0.0034091693454195806,
+            0.008848380026939594,
+            0.01842435611499587,
+            0.03546926214224266,
+            0.013478278853234764,
+            0.005970293629431872,
+            -0.0069197533761813455,
+            0.023196445250565058,
+            0.010403914879205351,
+            0.010017272564548626,
+            0.015076807790462658,
+            -0.029649260409273275,
+            -0.012737152920662164,
+            0.00373276777994811,
+            0.013239909450830258,
+            0.0001682544254742693,
+            0.02030652051051447,
+            0.014143961923579586,
+            0.0010147733405793957,
+            -0.012509446388074923,
+            0.0020437769941487626,
+            0.007405578085797006,
+            0.013458276946120546,
+            0.0006779114540819997,
+            -0.004250745498730309,
+            0.0009149988500358458,
+            -0.008355906072311008,
+            0.040391939952386505,
+            -0.01831218885700067,
+            -0.016793844108223176,
+            0.035064882870542116,
+            0.11416006426189942,
+            0.0010630856112440689,
+            -0.015660014600661857,
+            -0.017142534874931403,
+            -0.0086899013413391,
+            0.016611825572132997,
+            0.01609650977226868,
+            0.009780106186629793,
+            0.009622367309076686,
+            -0.000672926017804395,
+            0.024310949465350565,
+            0.02944698490225121,
+            -0.0037169531700178868,
+            -0.010990838766452194,
+            -0.036819646475041955,
+            0.022923962979125287,
+            -0.004596366586768123,
+            -0.007108041562635549,
+            -0.002604794313552557,
+            0.012641223813290853,
+            0.013668775645501733,
+            0.005125100040230003,
+            0.0077993841846437815,
+            -0.011500851674334447,
+            -0.017334458072700107,
+            0.0056819751688881774,
+            -0.015982234644378358,
+            -0.0076533013358874935,
+            -0.013127172654362694,
+            -0.011837346554842119,
+            -0.006562312362371076,
+            0.0029877420008976425,
+            -0.008572723854050838,
+            0.00854083969164562,
+            -0.010939639783598493,
+            0.016185642759084087,
+            -0.008052282332943615,
+            -0.0011992212936609377,
+            0.0010378646897070758,
+            -0.0025399515501741718,
+            0.010712865639618352,
+            0.01451357949543628,
+            -0.02521401163091652,
+            0.020836357346214075,
+            -0.008252813367155418,
+            0.0006409807249997337,
+            0.015338809580802425,
+            -0.04672526649678653,
+            0.022728876784937357,
+            0.005129225753457052,
+            -0.018372222092794564,
+            0.0066182736895667805,
+            -0.02048375780196863,
+            -0.00896382274306482,
+            0.0021738974344880794,
+            0.004491167150658828,
+            0.0034280799511763956,
+            0.015611133773752265,
+            0.018126362497548308,
+            0.00014383671280220997,
+            0.007927708693242186,
+            -0.006973076292062532,
+            0.008118313055163714,
+            -0.006791836664375275,
+            -0.01164567445668625,
+            -0.010894577039363014,
+            0.00014731542306100506,
+            -0.008865132658571428,
+            -0.011415696755798914,
+            -0.017922318360382588,
+            0.00358538001727405,
+            0.00289327637128695,
+            0.010406054711003452,
+            -0.007757607207626101,
+            -0.010994775479337082,
+            -0.011716096880261664,
+            -0.0013333474316266438,
+            -0.01231144419953554,
+            0.009792603083441917,
+            -0.02884953407490196,
+            0.0029097409399144306,
+            0.025277274938032477,
+            0.009263530315933738,
+            -0.028037993274175277,
+            -0.02532252134134532,
+            -0.0018596440554143638,
+            -0.013847737486669237,
+            0.009910107581119272,
+            0.013952962280235523,
+            -0.011807495289410774,
+            0.003896325037406752,
+            -0.02144921729633576,
+            -0.01628545945817568,
+            -0.0040148671532777005,
+            -0.016032493124255323,
+            0.004580041957001551,
+            -0.011375180364988195,
+            -0.007789996450583554,
+            -0.022041665680945066,
+            -0.005890065942573431,
+            -0.050455058269193866,
+            0.014401045842891128,
+            0.030219548461217365,
+            -0.004580747668649223,
+            -0.028684680969792875,
+            -0.015525870112797619,
+            0.011121401958547706,
+            -0.008879749268609245,
+            -0.019454243014778435,
+            -0.007839434838464604,
+            0.018908534797107714,
+            -0.0019283067432864216,
+            0.0024769961730655835,
+            -0.006732653593696035,
+            0.027604805021030646,
+            0.01366593691107021,
+            0.03740746059041736,
+            -0.014662268336336578,
+            0.02539666271785517,
+            0.02371809783638648,
+            -0.036126848124425154,
+            -0.007627445211958312,
+            0.03657481892016626,
+            0.01092515982671624,
+            0.0011046573579274704,
+            0.012350047525582187,
+            -0.006340802546498992,
+            -0.031330519599469066,
+            -0.0013691360532301728,
+            -0.0014676630169621005,
+            -0.023111265345539388,
+            0.027359543575867225,
+            0.018091458372134273,
+            0.014237156751820934,
+            0.0030993618578507172,
+            -0.0014037082734049385,
+            0.013257269860435359,
+            0.01021315143964527,
+            0.030885417052806473,
+            -0.003225065956381245,
+            -0.012825341172770057,
+            0.013476039084668826,
+            0.01852164843904835,
+            -0.009965495459579544,
+            0.012541079912270015,
+            -0.00824122613985912,
+            -0.018124134280975132,
+            0.03192263766222747,
+            0.0001484308749572035,
+            0.010888733498667343,
+            -0.010649515858317311,
+            -0.013082335214246402,
+            0.032614733822945205,
+            -0.00999075761269065,
+            0.004257643407198955,
+            -0.030723533031261387,
+            -0.003820776505926347,
+            -0.010471792928758869,
+            0.01737608180148319,
+            0.004597677757231315,
+            0.02095148515295301,
+            0.029253627700524396,
+            -0.011604450820008121,
+            -0.01996325509217509,
+            -0.001158626158150122,
+            -0.020583793738536985,
+            0.004168757459099133,
+            -0.03632338333261702,
+            -0.019373660230711246,
+            0.022259216603413172,
+            -0.010364620831688015,
+            -0.004751193403783363,
+            0.00444187509644351,
+            0.0008572202661850934,
+            0.014943779567410408,
+            -0.025408263453313398,
+            0.017755758370543148,
+            0.004627392860417132,
+            0.007402073071525438,
+            -0.024913329140362466,
+            0.013760771254234878,
+            -0.006087346663657887,
+            -0.03780469033982951,
+            0.005509388120163784,
+            0.03756141956381493,
+            0.02381019168406692,
+            -0.024580249929652097,
+            0.01807584361080568,
+            -0.03111039245576191,
+            0.023245542501125073,
+            -0.004868090951368273,
+            5.1544460403943004e-05,
+            0.00640235757127173,
+            -0.004698373089469779,
+            -0.00621383035264185,
+            -0.0025132415458431587,
+            -0.008746634331769936,
+            -0.01611801559493664,
+            -0.01193535623883546,
+            -0.03381789115349148,
+            -0.024119754887378317,
+            0.013881102852385332,
+            0.0019232668966222687,
+            -0.0095263729740736,
+            0.018652966857927805,
+            -0.03775912497511837,
+            -0.012022352025516115,
+            -0.023289377270709736,
+            0.023579672990954228,
+            -0.028605627815260902,
+            0.019235397553197357,
+            -0.025838655372560505,
+            -0.021458526449310157,
+            0.01508157885177254,
+            -0.010064396230734452,
+            -0.0016708624560165368,
+            -0.015042071642751917,
+            -0.03670785416972153,
+            -0.006267614265977843,
+            0.02298763967146357,
+            -0.012506444728541897,
+            0.00029871598584066644,
+            0.02671390618094565,
+            -0.003059865833779805,
+            -0.011103452976820622,
+            -0.0017382684168257506,
+            0.010923055147701782,
+            -0.02938686260440486,
+            0.014300097108092281,
+            0.0038571851420598074,
+            -0.0037614499297117365,
+            0.0010991440540809893,
+            -0.010108413486243776,
+            0.02127952161226619,
+            0.009142545227414532,
+            -0.0030260119510809205,
+            0.040589877142650374,
+            0.008677453889680373,
+            0.0017339508605300814,
+            0.008732950895530873,
+            0.01063352447960556,
+            0.006618635029823979,
+            -0.014986307792626314,
+            0.016905861015681957,
+            0.013101237900847744,
+            0.0002162791516784468,
+            0.019309343995079022,
+            0.0021263376627204313,
+            -0.024516267132902416,
+            0.013339486637573757,
+            -0.02311659751478777,
+            0.0020575982953961246,
+            0.02391695586495566,
+            -0.020440172706421535,
+            -0.001656132876701879,
+            -0.006304133667169861,
+            0.000543457661904495,
+            0.022135707414160034,
+            -0.0230903871986617,
+            -0.023444763940153112,
+            0.009604600033024832,
+            0.01721387169484476,
+            0.01827905276638783,
+            0.02824773106648473,
+            -0.007100625424382998,
+            0.01348510178676837,
+            0.0002604386361122316,
+            -0.011802178189607182,
+            0.007664157334475143,
+            0.0074175377458313594,
+            -0.0044008460682833875,
+            0.025491035291032547,
+            -0.0017015687016376205,
+            -0.006134010425966364,
+            -0.005379657499404915,
+            -0.01584573519366957,
+            0.0034386456145185213,
+            -0.013860742817569676,
+            -0.013355238174583658,
+            0.0040342831827141845,
+            -0.0208160950322342,
+            0.10904592104153767,
+            -0.0037038089210379766,
+            -0.019472012964045247,
+            0.0063554319446707044,
+            0.003716719884684827,
+            -0.003692786470980806,
+            -0.003353843210051379,
+            0.007788605566526459,
+            -0.011254480693190834,
+            -0.0045751956417398095,
+            -0.013976039724381875,
+            -0.001989262807706565,
+            0.010067374586433014,
+            0.009201983771978006,
+            -0.005902980676079493,
+            -0.02029714318363332,
+            0.013031676901335606,
+            -0.016970144750162643,
+            0.008874670945038486,
+            0.0008311255074953139,
+            0.00884167159995975,
+            0.001716603195908536,
+            0.01854951250316076,
+            -0.009310881405486074,
+            -0.01325242738760595,
+            0.029667589863813217,
+            -0.005616511284128333,
+            -0.0038179509972925954,
+            -0.0004690716809300012,
+            -0.005068974756744239,
+            0.0021204231125051274,
+            -0.027958274497127258,
+            0.022127088236172387,
+            -0.017271838492631593,
+            0.010267709087809265,
+            0.01316800560257366,
+            0.005599367128954229,
+            0.01817806970002991,
+            -0.007115594036834107,
+            -0.02627643123342057,
+            0.006542742528003751,
+            -0.003433874053082844,
+            0.006631427284537971,
+            0.02316295746074422,
+            -0.014573351469966297,
+            -0.033520864104177005,
+            -0.013707022143307006,
+            -0.01877574599842729,
+            -0.005992004412448576,
+            -0.0043698881826095435,
+            -0.01481352026460129,
+            0.006424121402963946,
+            0.01382938882387991,
+            0.018136021761515388,
+            0.0012697483790911433,
+            -0.011531136974861559,
+            0.00609187274277241,
+            0.01076133059240578,
+            0.008400887242930113,
+            -0.01533504932731406,
+            0.011439227832642668,
+            -0.028718487721012004,
+            0.01522697877548034,
+            -0.012452665599305595,
+            -0.024938748080573947,
+            -0.0022082450217489366,
+            0.017589399637694264,
+            0.0037824832255756763,
+            -0.004355445812154662,
+            -0.018778668474337727,
+            0.019825087149714933,
+            0.024692878851503508,
+            0.003883497401229232,
+            0.019470881143244197,
+            0.041834635328088364,
+            -0.003544509491805351,
+            0.027139164223766474,
+            -0.02129723143323512,
+            0.01461354860170749,
+            -0.03441729954693155,
+            0.001893713808284104,
+            -0.004222657393633789,
+            0.0013677766871517654,
+            -0.003811928806805752,
+            0.020142937759360904,
+            -0.009158202577174012,
+            0.0030638058587375543,
+            0.024854053046139163,
+            -0.018665378030958756,
+            0.00805903734779081,
+            0.010625429315145887,
+            -0.012856237251902233,
+            0.028535299711292306,
+            -0.006335094045840286,
+            0.00305422126851784,
+            -0.0297883488914759,
+            -0.0029192872993071216,
+            0.0026067044379754047,
+            -0.006097749614529602,
+            -0.00968023644800927,
+            -0.007433916539502931,
+            -0.006202274026852405,
+            0.0038354607448269555,
+            -0.021471361566683354,
+            0.003195646827154078,
+            -0.004390751282171024,
+            -0.003323447423321121,
+            0.02518759723554948,
+            0.025857746688368054,
+            0.016893707217894103,
+            0.012739170140289531,
+            0.008553625305650129,
+            0.019805351588748434,
+            0.01654964927067909,
+            0.011549329129728739,
+            -0.020026013396046632,
+            0.01150644479272541,
+            -0.004159611921476011,
+            -0.0022937297602724912,
+            0.014815466072709131,
+            -0.01572171840394285,
+            -0.02276187015864222,
+            0.024693009795978448,
+            -0.019725315346836404,
+            0.00998768979020729,
+            -1.1702210281215188e-06,
+            -0.011129745730507545,
+            -0.002591293237053776,
+            -0.018034357254990988,
+            0.00012070821244376275,
+            -0.023051045150566527,
+            -0.028465270110388716,
+            -0.00048603008831669926,
+            -0.000574546300981984,
+            -0.012992623897880395,
+            0.02064747165046032,
+            0.014820638761258362,
+            -0.024820174379739057,
+            0.02666518374464728,
+            -0.017430076811109155,
+            -0.01495051200726725,
+            -0.005861181182847845,
+            0.00100881727574559,
+            -0.0055223282819321295,
+            0.0034719859452138017,
+            -0.010180752654175722,
+            -0.009961678082468036,
+            -0.012134423047419183,
+            -0.028073672819317354,
+            0.011574683159318163,
+            0.016120674453124317,
+            -0.015043147648505721,
+            0.031834693135959095,
+            -0.008802805725050891,
+            -0.0018698230152007877,
+            0.016179742266521146,
+            0.005096232332051617,
+            -0.017187077376166815,
+            -0.018999569809271876,
+            0.012251308792871571,
+            0.013279090498325945,
+            -0.009588297039279112,
+            0.006630431200804809,
+            -0.014130176783207386,
+            -0.005092662672039885,
+            0.007020236623084732,
+            -0.005049363863784624,
+            0.0055051712211635915,
+            -0.00995314895304919,
+            0.0052743456988306485,
+            0.005841370337169303,
+            -0.0029434616355351975,
+            0.004677049987338193,
+            -0.018568941264514256,
+            0.020031253265629777,
+            0.0018725390936655962,
+            0.01332532735283085,
+            -0.03518855189467458,
+            -0.0056298723325832025,
+            -0.005369475318146738,
+            -0.004650364775945294,
+            -0.020833510091412353,
+            0.020445280915606132,
+            0.0159991245532085,
+            0.013202409908533162,
+            0.023094507005426774,
+            -0.014905666713082246,
+            0.01572052456975675,
+            -0.020945766915121455,
+            0.013114000985348841,
+            0.003349101658930482,
+            -0.019829605346106578,
+            -0.001423750749031516,
+            0.04557459007627153,
+            -0.005847231973329081,
+            0.011986110554869603,
+            0.0012709842140249691,
+            0.026758922819943853,
+            -0.0001635536762342011,
+            0.0065419519929442995,
+            0.005997692702992206,
+            -0.009935128108338126,
+            -0.0008057798606777003,
+            0.00865712443881613,
+            0.046254320605750854,
+            -0.01181132866321511,
+            0.008170688314239364,
+            0.013099051984057827,
+            0.0034971494510865624,
+            -0.0008386400616584293,
+            -0.015346506644390764,
+            -0.03067678541405639,
+            0.019305962963428017,
+            0.007736916644077731,
+            -0.012756525606221703,
+            0.008734673666457216,
+            0.015137771675476126,
+            0.012891824153867332,
+            0.003895886821685762,
+            0.02155154331885366,
+            -0.012761690789748027,
+            0.018528313792219413,
+            0.024663453412919027,
+            0.009077230086690327,
+            9.184526542175535e-05,
+            -0.000751529849753638,
+            0.006791927267880182,
+            -0.0004433479107716552,
+            -0.009832230053513483,
+            0.015858232096630655,
+            -0.014861341148074698,
+            0.017790924034108806,
+            2.7216806238946785e-05,
+            0.03744841066552559,
+            -0.015216207585985075,
+            -0.010138319613387346,
+            -0.007774325714088762,
+            -0.0016997034352120362,
+            0.003971564314593291,
+            0.0018686537276849653,
+            0.006051669054361943,
+            -0.0008319624938748146,
+            -0.017326561560776794,
+            0.01667737942301944,
+            0.005719808546053257,
+            -0.0007405637872110778,
+            0.013750752065593804,
+            0.0038732323799683453,
+            -0.00367645612814479,
+            0.023580906347714314,
+            0.006238541579764762,
+            -0.005235795826199542,
+            0.0028182323816636243,
+            -0.006443210286549713,
+            -0.016255209299456314,
+            -0.003061441395209219,
+            0.004597970789372988,
+            -0.0016590920570710628,
+            -0.011138261849069419,
+            -0.0019423173609729848,
+            0.020355534503272887,
+            -0.004612410336124759,
+            0.013346750536413418,
+            -0.005400060374036387,
+            0.006252154576727367,
+            0.010601459925435238,
+            -0.012346569079682625,
+            0.0014154620309235822,
+            0.010406076751218612,
+            0.0013983417564564663,
+            -0.007140180075225315,
+            0.0010544410919327675,
+            0.007757599672903709,
+            0.006505350216372801,
+            0.00483063937962671,
+            0.00836630436955631,
+            0.013284451514874498,
+            0.0033749897293287438,
+            -0.006243293993633622,
+            0.0013949831763016042,
+            -0.013269921385800992,
+            -0.006308639321247727,
+            0.0010906169976769569,
+            -0.0010152835709869353,
+            -0.02017142197515583,
+            0.010787630901414118,
+            0.005355784839945777,
+            0.007838485470991034,
+            -0.01706932891696025,
+            0.009576030158827565,
+            -0.0008144311254998525,
+            0.02829832828245172,
+            0.007407953684956539,
+            -0.02769403281764041,
+            0.014266001222243134,
+            -0.010492199740754951,
+            0.022575782293320284,
+            -0.0016349316593207844,
+            -0.005656505606143177,
+            0.008025514398322551,
+            -0.009634281579228592,
+            -0.005991026158213869,
+            -0.022366083313133788,
+            -0.025825209755160946,
+            0.016871087849192924,
+            0.013583173973358155,
+            0.021125252303971605,
+            0.00012555627355712715,
+            0.015347535950809997,
+            0.0020128011217859444,
+            0.015164099887550086,
+            0.0024544903332736167,
+            0.012507136277293888,
+            -0.011798050869794805,
+            0.0104828761281283,
+            -0.0137805761114781,
+            0.014620357150314097,
+            -0.005246639228547017,
+            -0.006986408834884282,
+            -0.008849580196616098,
+            -0.005442165099441538,
+            -0.020891551850452786,
+            -0.009155930286949747,
+            -0.0013590151948670868,
+            0.010665872513236615,
+            -0.0005386423407222223,
+            -0.0027671778465775897,
+            -0.03237311596935737,
+            -0.004875062533405715,
+            -0.006670982068843501,
+            -0.01942244110096111,
+            -0.0045975373348006885,
+            -0.026137783023267008,
+            -0.01657173843643047,
+            -0.0007784046498232684,
+            -0.015037306676062236,
+            -0.006889680260149007,
+            -0.02009705080543122,
+            -0.027192867101205106,
+            3.0161964469866572e-05,
+            -0.009647644858034833,
+            -0.005380165957043753,
+            0.013815092649296373,
+            0.011902366689169847,
+            -0.01572592170758115,
+            -0.022321702398924333,
+            -0.01088937403483412,
+            -0.0165754067023749,
+            0.01646435743868707,
+            0.009914938007210463,
+            0.018834863207293908,
+            -0.003724245912768107,
+            0.005643505694722323,
+            0.0009581443359000331,
+            0.02327704258596552,
+            -0.02532422120223976,
+            -0.028694220894404807,
+            0.0019298778426469338,
+            0.0025970175688368323,
+            0.006086164866310174,
+            -0.009414665394188936,
+            0.001200390869257535,
+            -0.007782899559317819,
+            -0.0228264209954853,
+            -0.00898519580978142,
+            0.008156835270668449,
+            -0.010036630164522355,
+            0.02230664068547531,
+            0.012332230358869292,
+            0.013699197379189934,
+            0.010593587938184292,
+            0.01576174456497367,
+            -0.0001654241831584687,
+            0.014471905257729428,
+            -0.008675915624727887,
+            -0.0031070346099204425,
+            0.0038358404385932777,
+            0.005878417037187274,
+            -0.00625022998235275,
+            0.028077114529659795,
+            -0.0011836453716817525,
+            0.0002962695480296182,
+            0.014405782572641941,
+            0.02697667559474498,
+            -0.004710313427099278,
+            0.0210512043261309,
+            -0.002393165025068831,
+            -0.006135816286212752,
+            -0.008429776018235104,
+            0.008607617299702318,
+            -0.009529291136733401,
+            0.011133497451942684,
+            0.019864404836361926,
+            -0.014041315457579166,
+            -0.007351413712676263,
+            0.02950254974910368,
+            -0.014564388091367897,
+            -0.01825317106331622,
+            0.02325964952838705,
+            -0.007414295676409368,
+            -0.0039042587057605484,
+            0.00377144546378926,
+            0.007169856503233239,
+            -0.004036744966949995,
+            -0.012928341573802416,
+            0.016090527447479415,
+            -0.00011159893712484579,
+            0.014146137810890391,
+            -0.0004655839804281017,
+            -0.006372231942983921,
+            -0.010047382825720416,
+            0.002373224247019062,
+            0.026818813126567614,
+            0.013483627627077692,
+            -0.017895710143841333,
+            -0.013785856764479817,
+            0.010826610341182479,
+            -0.01538893818197323,
+            -0.004425206001945227,
+            0.02155595194812746,
+            0.0028768153668128553,
+            -0.020543027467698483,
+            0.008028975884363618,
+            0.0003835165718138906,
+            0.00037046947047892953,
+            -0.00029363296913641716,
+            0.030135739208689217,
+            0.001343516577762509,
+            0.00018948658255495327,
+            -0.00875053830853609,
+            0.008606573235959595,
+            -0.01280305089741873,
+            0.012786478836532904,
+            -0.02437993139337839,
+            0.007648181775166282,
+            -0.00629541989541625,
+            0.004579605315599212,
+            0.016658386927051592,
+            0.01218098627022323,
+            0.015031618813778413,
+            0.008237200519537417,
+            -0.008942965899756683,
+            0.0009093265012901892,
+            -0.025302175262520477,
+            -0.001333639155622843,
+            -0.017971876771137367,
+            -0.02036433247634014,
+            0.0002463842837918767,
+            0.022144712689630985,
+            0.018559482073124734,
+            0.011579840028464503,
+            -0.008657014440886656,
+            0.0166696911919226,
+            -2.9834620942882283e-05,
+            0.011845827764518911,
+            -0.00037901176785495324,
+            -0.006138374154553732,
+            0.004711058953619512,
+            -0.003622694588720417,
+            0.013542405259906128,
+            -0.014730567462694827,
+            0.012076044672004906,
+            0.01690223095263866,
+            -0.009108632851807137,
+            -0.008946251274440141,
+            -0.013992423874475428,
+            0.00048381889145447367,
+            -0.015909599902931202,
+            0.035489252674454214,
+            0.02252772349379229,
+            -0.01210464970923921,
+            -0.010469120693943236,
+            -0.014164999313775756,
+            0.019496048806752753,
+            0.010427539351244087,
+            0.0028018908451425276,
+            0.017091768406027965,
+            0.021395638416767112,
+            0.008817247593404623,
+            -0.006081063711583319,
+            -0.0021216503501144983,
+            -0.003781810594530787,
+            0.020143098218482827,
+            -0.003078172339607037,
+            -0.011350171884507555,
+            -0.012516379645909518,
+            0.005766102930704529,
+            -0.01626433668691071,
+            0.0019294643928566657,
+            0.008641558521764407,
+            -0.015057688533773374,
+            0.019407986696657772,
+            -0.028126197703647868,
+            0.012364374199657305,
+            -0.0044278434183731225,
+            -0.007251147925369614,
+            -0.03496220945653279,
+            0.006389489026974697,
+            0.008941259259632024,
+            0.003941752277454314,
+            0.023003838655784314,
+            -0.024528514847470928,
+            -0.018361931455105486,
+            0.0037791061321962543,
+            0.00047162060848090104,
+            -0.00711894577052754,
+            0.0010138871688747137,
+            0.004128392769213672,
+            -0.00674546985015164,
+            0.0017307097443508704,
+            0.0033354800298095546,
+            -0.002829898077850263,
+            0.013252947102333348,
+            0.0081724269619999,
+            -0.021449979263734438,
+            0.001368791856860285,
+            0.01590656734732547,
+            0.01452008402429858,
+            -0.02391847163018108,
+            0.006331397860689212,
+            -0.02064812934794864,
+            -0.000631340242718475,
+            -0.031272866000888556,
+            0.016859793097564958,
+            -0.0069008624795683276,
+            0.01899718801899956,
+            -0.0015377436942771224,
+            -0.0065810685260109004,
+            -0.012822620655323715,
+            0.02137604937338179,
+            -0.0019121137078344135,
+            -0.0025598772127157477,
+            0.015678293057066646,
+            -0.025700865215693987,
+            0.017825186616033165,
+            -0.0026230848396158204,
+            -0.017342967698019966,
+            0.017199139867162275,
+            0.004536019178633225,
+            0.007224666811309087,
+            -0.023505589665315567,
+            0.0007208041198255396,
+            0.003900164485347562,
+            -0.017250324025622685,
+            0.004470862192961212,
+            -0.030807400122398566,
+            0.0006579147990327597,
+            -0.013222907940559794,
+            -0.0020514994691204847,
+            -0.0046763235724575445,
+            -0.014050035576183385,
+            -0.039181147615786484,
+            -0.005429639279537522,
+            0.016065742005148733,
+            -0.0009029636981539927,
+            0.01800715198737081,
+            -0.002730931639143803,
+            0.002947680833843522,
+            0.004912692811020822,
+            0.00818378920043427,
+            0.006915895602313347,
+            0.02274116906914758,
+            -0.0057854922886970365,
+            0.00471999900678522,
+            0.007058014277137233,
+            0.014638081356210031,
+            -0.01719565951201012,
+            -0.015170377598372723,
+            -0.016015921063369496,
+            -0.006351213642269936,
+            0.01739665344303718,
+            -0.017488885259028475,
+            0.013303332445465686,
+            -0.00429428898520701,
+            0.014825338166443021,
+            -0.008686427208690176,
+            -0.01575945086724703,
+            0.0028564767084934386,
+            -0.007570417028925067,
+            0.001007670647768344,
+            -0.02323908008484089,
+            0.02019729564706863,
+            -0.00847581038203716,
+            0.010508144473197669,
+            0.01729824052022313,
+            -0.021364529179605813,
+            -0.009742851959432839,
+            -0.014342281441152413,
+            0.029896210049029176,
+            0.013925163184699139,
+            -0.009392252140772913,
+            0.003927230832659267,
+            0.008873561269230905,
+            0.011529536139797032,
+            -0.012209070214071445,
+            -0.0027679974125029362,
+            0.002411914349172495,
+            -0.0062056207925691356,
+            -0.002025687569076897,
+            0.03090134105002816,
+            0.0038285457265343245,
+            0.024877474647546086,
+            -0.013214997316123852,
+            -0.0033763842539861907,
+            0.00023379106197363714,
+            0.005134764372494098,
+            0.01183867975321486,
+            -0.0017966997954747142,
+            -0.008029791503028367,
+            -0.01179500476055557,
+            0.02599911330072515,
+            -0.021852083993691557,
+            0.007024505421035273,
+            0.026784024996764876,
+            0.001390202993990949,
+            0.0029956956045771046,
+            -0.013131691460546865,
+            -0.006298736864509989,
+            0.02523640664202355,
+            0.010030824059858138,
+            0.0041523782603032225,
+            -0.026465819482580182,
+            0.0032128899272377593,
+            -0.0061901409332948115,
+            -0.008681758126710213,
+            0.009866461769956222,
+            -0.0028095454857224164,
+            0.01188012041754054,
+            -0.012191996383095982,
+            0.028459939543805705,
+            0.010711502132045176,
+            0.01047154118931583,
+            0.025888652452698974,
+            -0.02079614305315134,
+            0.0042203106489994615,
+            -0.010829796140534066,
+            0.0006773493086316061,
+            0.005596640511307504,
+            -0.027510562413905437,
+            0.001969898603871003,
+            0.010390866563404346,
+            -0.005642706366801436,
+            -0.01882994179544761,
+            0.01187970346127799,
+            -0.019814385265412387,
+            0.024479579275617887,
+            0.003231341626501863,
+            0.020056914356101588,
+            -0.0053276448188067154,
+            0.009834685132275868,
+            0.0002742942778577165,
+            0.0031361331890882127,
+            -0.0008074598092128465,
+            0.009009012303347016,
+            0.005786777410144894,
+            0.013719989127068968,
+            -0.025528370602188686,
+            0.006821572597883499,
+            -0.0029197127345352626,
+            -0.030539872624857384,
+            -0.013383097680839732,
+            0.010291485622655026,
+            0.012218533188117704,
+            -0.009144425807890594,
+            -0.010378679518611277,
+            0.0030042579827305603,
+            0.005748243116238633,
+            -0.006692403885450407,
+            0.006913672151000981,
+            0.01629516912493545,
+            -0.002533517074771728,
+            -0.0019083178803585312,
+            -0.0035007340321378893,
+            -0.01154609935927537,
+            -0.010016998439072824,
+            0.023193275321990937,
+            -0.00482990342915347,
+            0.017943025772239603,
+            0.025873224580389702,
+            0.0039876567755322,
+            -0.0057577932736143506,
+            -0.005460363755206188,
+            0.01478435999841998,
+            0.014195457012279476,
+            0.004669295894868759,
+            -0.009702220679239768,
+            0.007974305230005956,
+            -0.007760086293456636,
+            0.008375077126777163,
+            -0.020260917110786147,
+            0.002199401871514877,
+            0.0026191373298767766,
+            -0.0022089262577237776,
+            -0.015213807046175918,
+            -0.01956788452420682,
+            -0.021898368819136302,
+            0.012634659463647929,
+            0.00034643102239488474,
+            -0.009241772775582378,
+            -0.013645250997151429,
+            0.0016665106990117016,
+            0.0025630500346914592,
+            0.0002095521560180936,
+            -0.007783022628694597,
+            0.009546147813008391,
+            -0.01265287317178631,
+            0.007445223311736028,
+            -0.004007029181436962,
+            0.0051846833906574515,
+            -0.0016916014415925766,
+            -0.016413938859402438,
+            -0.0210868129634149,
+            0.01920794362224444,
+            -0.012897042227841011,
+            0.02269303737601583,
+            -0.0014935878147854765,
+            3.062680855559304e-05,
+            0.0033132626755401546,
+            0.007274487651476732,
+            0.012110042834251237,
+            0.011426424896797977,
+            0.004307086698938411,
+            0.008206932254416481,
+            0.004673427176334148,
+            -0.013124571552764586,
+            0.004324963759928715,
+            0.0039216033705804084,
+            -0.010449887922072297,
+            0.005298323150690876,
+            -0.0010196552573491,
+            0.006128327172621373,
+            0.012167492341907742,
+            0.01923525204293908,
+            0.01908107550907106,
+            -0.0012031208519043073,
+            -0.0010601912677170903,
+            0.014474374858125607,
+            0.004540954805526089,
+            -0.004883537463424679,
+            -0.002772988629590886,
+            -0.013363713078069854,
+            -0.002572403646338979,
+            0.011541515690646252,
+            -0.02376161747414617,
+            -0.018215231053002656,
+            0.021831219299143226,
+            -0.004991882316041097,
+            0.033796174834149546,
+            -0.013140970283091536,
+            -0.016197177724633614,
+            -0.00464190076219332,
+            0.0026389653040644607,
+            -0.013788049032378498,
+            -0.010300859744205445,
+            0.010545541278382187,
+            0.013848738866656793,
+            0.009903590151120531,
+            -0.0005984085581408575,
+            0.02922816576443645,
+            0.0019900932217491907,
+            0.02245395592422119,
+            0.0018978407151245295,
+            -0.007029294098689854,
+            -0.01440575120349887,
+            0.010840472832356266,
+            0.005529335652955532,
+            0.009050748012161991,
+            0.017791269808484802,
+            0.014502057454389247,
+            0.006265567405788131,
+            0.013928705812292101,
+            0.0012903002372833693,
+            0.0010858377120770652,
+            -0.002355229463060164,
+            -0.01692711767136341,
+            -0.01456786846097011,
+            -0.01627640642218236,
+            0.003873864312554098,
+            0.012620350492241247,
+            0.00525890570023481,
+            0.0007393789113866752,
+            -0.007153210751899716,
+            0.009254566395058412,
+            0.0065087881512835375,
+            -0.012279152412209152,
+            -0.016530068283509593,
+            0.004060671955726967,
+            -0.02210570272789274,
+            0.007849467899009437,
+            0.01554233481902349,
+            0.00219885032769439,
+            -0.002655449524219968,
+            0.0006843524304842362,
+            0.006272944169764281,
+            0.005531927368102191,
+            -0.0036710328198476905,
+            -0.00544113514775027,
+            0.011264661763762068,
+            -0.006379654234440906,
+            0.009537150372938296,
+            -0.00290031548037575,
+            0.03718190949929003,
+            -0.011976080887998194,
+            0.012120408392610773,
+            -0.001751472085435711,
+            0.008481703962951994,
+            0.008067669857949423,
+            0.010772687289903202,
+            -0.003805089040741351,
+            -0.0022164543131735167,
+            -0.008753625032781523,
+            0.006751953442322185,
+            -0.01582177621255226,
+            0.013631301238130705,
+            -0.00030685726238985963,
+            -0.010728958775974582,
+            0.01055472059121978,
+            0.006788316621149835,
+            0.004413382316882087,
+            -0.003262024961860459,
+            -0.01972283203505511,
+            0.014937729522976615,
+            0.0018799533869990667,
+            0.019247008681353865,
+            -0.023276106189004046,
+            0.029043727846427652,
+            -0.02019024213343764,
+            0.023083643395685142,
+            0.01865875504942157,
+            0.001179436167200093,
+            -0.02070329911663634,
+            -0.007618131527214554,
+            0.014871709122115308,
+            -0.02659818889735726,
+            -0.006234719259825804,
+            0.005986829551040766,
+            -0.014639912579831785,
+            0.00037813423546252645,
+            -0.01924149909676064,
+            0.00010832498610532006,
+            0.011013130149723431,
+            -0.011782407678373089,
+            0.00438725832178584,
+            0.01608349750921335,
+            0.019347883652009713,
+            -0.009760017612545174,
+            0.003869246694835168,
+            -0.024240324890794748,
+            0.010958789926998691,
+            0.020511684636649225,
+            0.008883830392748497,
+            -0.01761944522796127,
+            0.021817973960311446,
+            0.010652131228027521,
+            -0.012575679152857703,
+            -0.0012753095453828826,
+            -0.028191274775392804,
+            -0.012421170815054556,
+            -0.02188248722244861,
+            -0.0011670175621882146,
+            -0.014436683528976771,
+            -0.0029783880907228383,
+            0.0009506986717516226,
+            -0.004719165363484719,
+            -0.01179163390107033,
+            -0.019543732196676634,
+            0.016606805875498898,
+            0.0227313413004497,
+            -0.006738735201626204,
+            -0.014547240981002825,
+            0.013866974327185123,
+            0.031694077725890536,
+            -0.016189209297529324,
+            -0.00542110554672736,
+            0.0028061798782555557,
+            -0.012964876141500577,
+            0.018946046750284537,
+            0.0039419760434645495,
+            0.000322115192590907,
+            -0.01991851369210666,
+            0.002489725351392778,
+            0.010835458511052207,
+            0.01684964045708685,
+            0.0005541314524621196,
+            0.010855852688653808,
+            -0.007549882748861744,
+            0.010096800951420057,
+            -0.011832260836468588,
+            -3.6179995603517093e-06,
+            -0.007800366796516945,
+            0.007001049318404058,
+            -0.004424776169203575,
+            0.034356203288485945,
+            -0.02817071276766299,
+            0.004989631403681216,
+            -0.014539644354369607,
+            -0.010888763939716889,
+            -0.0027322086509797843,
+            0.0004865534340981866,
+            -0.010775238981581461,
+            -0.005162220412866082,
+            -0.0011263271110752205,
+            0.00913081242511139,
+            0.005844327320514489,
+            -0.013886529519306775,
+            0.019171827605726503,
+            -0.030984919318656618,
+            0.004469458202753611,
+            -0.008553175891755315,
+            0.010511707988936336,
+            -0.0151185694875566,
+            0.013830711686954931,
+            0.01301959345025159,
+            0.008205984085944315,
+            0.007740367849216328,
+            -0.01532123052721834,
+            -0.01678314222392063,
+            0.001376468747213867,
+            0.0036953018347776244,
+            -0.00931855481896886,
+            -0.033391028922085425,
+            -0.007072913446166969,
+            0.005331350534425606,
+            0.011614453225043,
+            0.008156128603003278,
+            -0.02345206873048703,
+            0.005003870024809975,
+            -0.024178720646704597,
+            -0.012115112684614331,
+            -0.007750557665424973,
+            -0.011580152272614108,
+            -0.006117298954192534,
+            0.026449765548206453,
+            0.007989237291215073,
+            0.016691517078382376,
+            0.004954133565782356,
+            0.02742809127725913,
+            0.011670893621991404,
+            -0.011736840457474895,
+            -0.003349151747880166,
+            0.00035419034843196817,
+            0.005507651806218584,
+            -0.008706590052024251,
+            0.011793241980257321,
+            -0.0013295035978611142,
+            -0.0065061682758755625,
+            -0.004398354347597199,
+            0.009372816666628725,
+            0.014188676762501037,
+            -0.003007980559117429,
+            -0.008724255229333351,
+            0.015749776067989826,
+            -0.030899793617096834,
+            -0.015215033379100091,
+            0.0016152308157299517,
+            -0.007813175908301852,
+            0.017806770680528907,
+            0.01169056345888971,
+            -0.026835412610223673,
+            -0.004538235170332766,
+            0.010532316940606419,
+            0.004755930024691789,
+            0.0095408889025506,
+            0.00954782703036282,
+            0.012079064562163901,
+            -0.007829549014214643,
+            0.01046722658021748,
+            -0.011519369840999348,
+            -0.007092961453184232,
+            -0.007241799314569802,
+            -0.008077934089061052,
+            0.021467547268987214,
+            -0.025816030568238717,
+            -0.006558853543772633,
+            0.01134670818701549,
+            0.013027491377727365,
+            -0.002301991242932548,
+            -0.006299073927206214,
+            -0.017357989759336386,
+            0.008311464809363397,
+            -0.013922568511594478,
+            0.025633907424998367,
+            -0.004854655498817219,
+            -0.005336807899100369,
+            -0.005228056880188717,
+            0.01958682721272445,
+            -0.016774409737540904,
+            0.02072573220949266,
+            -0.014449250679704904,
+            -0.007857351393427215,
+            0.001673929749689194,
+            0.006890008607998747,
+            -0.002948835017234416,
+            0.03734128725793149,
+            -0.00556474133167756,
+            -0.0014544201137417158,
+            0.02209084868209573,
+            -0.01883610080898967,
+            0.017901421953179442,
+            0.01563604483998435,
+            0.012327159100148019,
+            0.0018161034827768648,
+            -0.010253034391420751,
+            0.019979728662282904,
+            0.020823925491027635,
+            0.0139490143219451,
+            0.009577521724445478,
+            -0.013478761809038993,
+            0.014934933784421556,
+            -0.0003725422088504494,
+            -0.0024362115214204105,
+            0.012785534912074068,
+            0.016441944143888537,
+            -0.02164522371903646,
+            -0.013152102018736346,
+            0.0063548333449661805,
+            -0.016547629104776748,
+            -0.024875443975589356,
+            -0.008991095813132973,
+            -0.021885127280977718,
+            -0.023336714382596175,
+            0.00688008599534903,
+            0.016482610587564443,
+            0.0035797836729533367,
+            0.013111012464332918,
+            0.006820471349350837,
+            0.018274678506489214,
+            -0.00980248993077149,
+            -0.01582838461385605,
+            -0.0009454586621769173,
+            -0.0004906493597104733,
+            0.009787688665732045,
+            -0.01835583040075359,
+            -0.013977939077760832,
+            0.004340902272034388,
+            -0.020118484730757877,
+            0.01421736173162507,
+            -0.006428386730058143,
+            0.016756376129345597,
+            -0.0004630315831872392,
+            0.011701948921076007,
+            0.02621773278421472,
+            -0.001391567406453007,
+            0.01262026995055899,
+            0.004696906079976175,
+            -0.0030577364576645635,
+            -0.010624066224426196,
+            0.0009420498021845824,
+            -0.0044079288776429516,
+            -0.0004845606666642234,
+            0.006481113327122954,
+            0.020269778435314,
+            -0.03321232649491074,
+            -0.009425498978427855,
+            0.01246754422779043,
+            -0.001658574527323383,
+            -0.005816985985181387,
+            0.013050062223188965,
+            0.00011195028631856881,
+            0.016573827662919836,
+            -0.0017453203044868104,
+            0.01417371165305189,
+            -0.008165910519539772,
+            -0.0007461950113024975,
+            0.0016492309267666983,
+            0.005076015595439873,
+            0.005829299318055953,
+            0.012146800191035666,
+            0.01148884639506138,
+            -0.021051327863689618,
+            -0.0026357096198729156,
+            0.0036146148489310443,
+            -0.010673282438301611,
+            0.014967444393862347,
+            0.005737933867764028,
+            0.01391411712868596,
+            0.0029556189893790237,
+            0.01167806258173173,
+            -0.024440974650111174,
+            -0.01029944007931845,
+            0.01385509328958837,
+            0.00869671728965227,
+            -0.00998315356917989,
+            -0.016580053751364735,
+            -0.007061005072623325,
+            -0.00477175200214453,
+            -0.0021182980051220428,
+            -0.009871524928312118,
+            -0.0005933941177237234,
+            0.006284920058476413,
+            5.466820734762947e-05,
+            -0.012842295882002122,
+            0.015810530885294202,
+            0.004572385791133295,
+            0.003685966629168175,
+            0.005093714133656017,
+            0.011877541830396259,
+            -0.013130170365831912,
+            0.01258190257102377,
+            0.011433121961378686,
+            0.01142665981860032,
+            -0.013710025169815697,
+            -0.001774030746825879,
+            -0.01109881340948764,
+            0.0065684584345163735,
+            0.009983753894796107,
+            0.005346443477816739,
+            -0.002062387245572686,
+            0.004950044847352647,
+            0.006105199826576752,
+            -0.06391185159544624,
+            -0.02131511418723256,
+            0.0027165737784844413,
+            0.033389688791557785,
+            -0.007167593552251769,
+            -0.0037646833460996797,
+            0.002883510089742145,
+            0.01356887065340832,
+            0.009730171586767032,
+            0.00418566245984083,
+            0.020320934773473808,
+            -0.008814826860651366,
+            -0.003014193747732543,
+            0.001559573791453607,
+            0.003499420226345028,
+            -0.008703837291821111,
+            0.0044844701302276645,
+            0.0053963630238108035,
+            0.027764369548110287,
+            0.001469968540767627,
+            0.015793086297019008,
+            -0.008505290826588508,
+            -0.004075888558568453,
+            -0.009166710762668385,
+            -0.002860047211143503,
+            -0.011127820563172684,
+            0.003937110717404271,
+            0.02054186721680431,
+            -0.006126180963243227,
+            0.009346433819831456,
+            0.01057504086030076,
+            0.0001650851447316893,
+            -0.012033423322524951,
+            0.0049587335923558,
+            -0.0032240390628760535,
+            -0.005568976687946761,
+            -0.0024667040429312737,
+            -0.03474027725033016,
+            0.026855947684394643,
+            -0.004369626674503285,
+            0.009252512430957932,
+            -0.005081834388513357,
+            -0.0037896610731020337,
+            -0.014201609270756739,
+            0.017871088029615106,
+            -0.008249444723433136,
+            -0.020092257756971717,
+            0.009532994544485572,
+            -0.004991989149115383,
+            0.006464417738670472,
+            -0.02643078108339741,
+            0.0029138922065812495,
+            -0.017232410749655366,
+            -0.003999293090834831,
+            -0.006924488971625995,
+            0.01318082244178942,
+            -0.007232695119856389,
+            -0.02660711037908311,
+            0.01979251106803667,
+            -0.019485860712345603,
+            -0.011977904020170568,
+            -0.0177530973032872,
+            -0.014524003566945937,
+            0.004874952064727108,
+            -0.015326044325348929,
+            0.002047771741041311,
+            -0.009697340332342246,
+            -0.003652304998155628,
+            -0.0008878698629674222,
+            0.0017782435347417623,
+            -0.013017776298713044,
+            -0.018272697230002275,
+            -0.01353020425912225,
+            -0.00757239449706798,
+            -0.0008250000619957538,
+            0.01299142281772714,
+            -0.001679606555140765,
+            -0.007329941421303076,
+            0.0044605462460153116,
+            0.010231192277666706,
+            0.0006558644644452857,
+            0.0024231733856020414,
+            -0.010728844680699885,
+            -0.014137434553169443,
+            0.005230053181525096,
+            -0.012086468475542818,
+            -0.009459717916037003,
+            -0.0028185227600210556,
+            0.014984527039312847,
+            0.008189464334213754,
+            -0.009139240774931914,
+            -0.005639157579163322,
+            0.014325949891859377,
+            0.009020303519690971,
+            -0.013310205617445609,
+            0.01567574314514385,
+            -0.025369802629268523,
+            0.021964786021408878,
+            0.030536905527039065,
+            0.004945288795196981,
+            -0.01168303908203322,
+            0.02281128628230144,
+            -0.004072286832145419,
+            0.0028278384348551146,
+            0.01740522162288968,
+            -0.042558465579514926,
+            -0.013146562547084278,
+            0.002903743316339479,
+            -0.015297177099602701,
+            -0.010963312697387696,
+            0.008804938803398296,
+            -0.0026727734429658817,
+            0.005273238046953793,
+            0.004257613402172282,
+            0.03512246556126453,
+            0.002258612643297096,
+            -0.0002885708728599208,
+            -0.0019720190285930487,
+            0.012312613129996533,
+            0.010451003939611476,
+            -0.015556620910136185,
+            -0.02387856865043814,
+            -0.004583090523619507,
+            -0.0037828776396525835,
+            -0.0014588238692793338,
+            -0.004361834167147271,
+            -0.008159679676329573,
+            0.012243319920992457,
+            -0.017279964797769668,
+            -0.013748032758693542,
+            0.018394195628121487,
+            -0.006411113522842359,
+            0.019002611495040334,
+            -0.014970362556522149,
+            -0.0077437581301661946,
+            0.0017311115715410025,
+            0.024439691379769795,
+            -0.025013956798520212,
+            0.007359134924497539,
+            -0.0031406470488477937,
+            0.017574824561755627,
+            0.00419409250132525,
+            -0.0046803792187546656,
+            0.010556972371044426,
+            0.015229789583480183,
+            -0.004608293492221483,
+            -0.014449487985966966,
+            -0.01172850291078532,
+            0.016823024209386785,
+            -0.012304887597699588,
+            0.0011773351576246157,
+            0.002262670226244277,
+            -0.0023810578015456716,
+            0.03477119000959688,
+            -0.03442178291363675,
+            0.004062693040714608,
+            0.0029249583507755944,
+            0.003291349443702718,
+            0.006461525089528308,
+            0.006047777261425826,
+            -0.020642802018978404,
+            0.008916088381876748,
+            -0.03316762661930299,
+            -0.006367254689957291,
+            -0.005404276207844267,
+            0.0009189006423526284,
+            0.009401330264376041,
+            -0.011106443887384533,
+            -0.02259405372588916,
+            -0.006010439054323966,
+            0.0070744551115721095,
+            0.010262607009446879,
+            -0.002856616377476421,
+            0.019719451109535182,
+            0.008426509638139982,
+            0.0015877648339862691,
+            -0.010655792508751512,
+            -0.015527851374834499,
+            -0.015460974859638219,
+            -0.017847462632188386,
+            -0.02236964311096428,
+            0.011841749854626384,
+            -0.00576688153869993,
+            0.009054902130717224,
+            0.014317707346504314,
+            -0.015366657284925976,
+            0.002910211008094564,
+            0.017761685988205086,
+            0.007974806171830457,
+            0.002692237107055498,
+            0.0032405477366987947,
+            0.02685230091638845,
+            -0.0009116884421301454,
+            -0.0032228977180251673,
+            0.003158213042682977,
+            0.015872199488167846,
+            0.0004966905858799325,
+            0.019981401695749582,
+            0.015201538351780685,
+            0.005359075814444414,
+            -0.004668738193285921,
+            0.019853988551405383,
+            -0.009844394286458195,
+            0.013476144827747423,
+            -0.00411789677405284,
+            -0.00830656434595337,
+            -0.008496132520586954,
+            0.006442581751999304,
+            -0.008138786054348571,
+            0.006977337946948795,
+            0.02419128791801387,
+            -0.0005734827808485311,
+            -0.0028842142002271044,
+            0.005503033191779872,
+            -0.01696639779775455,
+            -0.00389835761601045,
+            -0.03366077608692946,
+            -0.009942613075976742,
+            -0.01708630689497001,
+            -0.02910756562213045,
+            0.026536268897199546,
+            0.00219827495662445,
+            0.02586503862171508,
+            -0.01905122645609761,
+            0.008935637454859214,
+            -0.01385806889332683,
+            0.029747093931541325,
+            0.009163959012862627,
+            0.0008611776492791918,
+            0.003598831885031635,
+            -0.02103045281939351,
+            -0.01150941825152158,
+            0.027341774725604333,
+            -0.0006700608710615124,
+            -0.012089044078073347,
+            0.022396400579038037,
+            0.017491565611764764,
+            0.0051760367056280536,
+            -0.00559536785318233,
+            0.01979262595019399,
+            0.002306380546194741,
+            0.011973863166872444,
+            -0.0028136750606972223,
+            0.012911235736067303,
+            -0.01510421118359169,
+            -0.003338602157155174,
+            0.039758808572572274,
+            0.007290642519242007,
+            -0.001802921446521437,
+            -0.013696774990705774,
+            0.003941198181491921,
+            -0.0008308293948698925,
+            0.0200351513103389,
+            -0.00014327587644797734,
+            0.028291423013030337,
+            -0.0001085859276231927,
+            -0.010723064585989723,
+            0.020443848393732245,
+            -0.017860296930396913,
+            0.04540901629084378,
+            -0.018671578423261854,
+            0.000852031295629291,
+            -0.010639853151638835,
+            -0.019017194262730474,
+            0.003861296738239369,
+            0.0010977143012920602,
+            0.0008387482016061222,
+            0.0060155816314618,
+            -0.010780239332444652,
+            0.002152214497259421,
+            0.010556338812873054,
+            -0.027636192450621728,
+            0.0010938992111792307,
+            -0.014385202899929138,
+            -0.01353192238925814,
+            -0.026667778226211785,
+            -0.00017403398236549974,
+            -0.015357674131635848,
+            -0.013956722505869091,
+            -0.01985038526848704,
+            -0.008927903879720431,
+            -0.009580214158077466,
+            -0.0020314441852913386,
+            -0.004755584878514854,
+            -0.00018388548107372635,
+            -0.002918855567842974,
+            -0.0015941852262316128,
+            -0.014230272061190995,
+            0.02133992212037431,
+            0.015283345362445234,
+            0.004642765060980717,
+            0.012829570028541471,
+            0.01708952142208596,
+            -0.004770647543823141,
+            0.006604762788847952,
+            -0.0025369836845081078,
+            0.008368000846861623,
+            0.014785031726639733,
+            0.012074738887486398,
+            0.02124518119032885,
+            -0.015459818988572564,
+            -0.004992865135357168,
+            -0.012722391036055767,
+            -0.012210202603282509,
+            -0.017850065946244383,
+            -0.004300429148015694,
+            -0.0014387298144714717,
+            -0.023128300275416423,
+            0.004468952290723408,
+            -0.0026602931688877443,
+            -0.005768852559887139,
+            0.007815212168719318,
+            0.0005862972228454734,
+            -0.002265719947629544,
+            -0.00019323737674204774,
+            -0.02735820445266249,
+            0.00024009122317215594,
+            -0.00018701993387558173,
+            -0.014663710240203917,
+            -0.00450797561624596,
+            0.010584731989847334,
+            -0.011966527748437568,
+            -0.00602367292291343,
+            0.0111155226518017,
+            0.0015565179432852256,
+            0.006062862961835896,
+            0.016952559697728433,
+            0.0005012157272316869,
+            0.011160668242334889,
+            0.014993782560797772,
+            -0.01591303023837098,
+            -0.015356412848444082,
+            -0.0236025426434855,
+            -0.004514538154596937,
+            -0.009753334017065119,
+            0.007553951535816575,
+            -0.004616595532996391,
+            -0.013023153809236325,
+            0.006040285197946957,
+            -0.0006343451398642784,
+            0.008338564073158257,
+            -0.017608246105292328,
+            -8.346168119689692e-05,
+            -0.020944870308705715,
+            0.009230123321316848,
+            -0.00531237224627929,
+            -0.00445985089300067,
+            0.01410675485522911,
+            0.02895945580213926,
+            -0.007799830351886844,
+            -0.009455128322575888,
+            -0.005663072244883114,
+            0.012817229945992764,
+            0.012474431030896143,
+            -0.004423141749110039,
+            -0.0018294512146253298,
+            -0.012817101093394574,
+            0.0004755435377918658,
+            -0.01129485262081729,
+            0.0062747111125005155,
+            0.017199374798326412,
+            -0.021965117839794063,
+            -0.01296097993683748,
+            -0.016765996091978336,
+            -0.016882855472266523,
+            0.004372970343291638,
+            0.017290565252244258,
+            0.022670788364378462,
+            -0.008456547449078897,
+            -0.019564049850942417,
+            0.006898353991539946,
+            -0.01620125883985687,
+            0.0037108638122592746,
+            0.12887455743463638,
+            -0.0028864852421660366,
+            -0.011487804449328495,
+            -0.022683334123299446,
+            -0.005467192611880829,
+            0.01210557696712196,
+            -0.01555159294496261,
+            0.06709672179195036,
+            -0.015071089552631105,
+            0.004364902817496082,
+            -0.004899872946473657,
+            0.0006310685609937571,
+            -0.017610935588191347,
+            -0.005962991715597585,
+            -0.007768819050529025,
+            -0.02688607975562596,
+            0.000883763319535686,
+            -0.009935313912803587,
+            -0.012883104064962595,
+            0.002126368813790577,
+            -0.0013697265269800346,
+            -0.016131186838419288,
+            0.01575514418037863,
+            -0.020677990768697592,
+            -0.007251917191831732,
+            -0.0029325623780992118,
+            -0.00600710212646933,
+            0.014573093764769918,
+            -0.03253336117394778,
+            0.002973725493330224,
+            -0.013601417220439602,
+            0.011905788192118131,
+            0.001542117126832946,
+            -0.010789486980776022,
+            -0.013862139885514524,
+            0.007086571810314664,
+            0.015658913897173155,
+            0.017039624764452546,
+            0.01901374401756777,
+            0.004797954569087795,
+            0.003928999581093576,
+            -0.008046722893614655,
+            0.021974371139905104,
+            -0.00697335955527197,
+            0.0008975473383295958,
+            -0.00970589295353857,
+            -0.004346869073499034,
+            0.0031279458769925416,
+            0.02303983493541673,
+            -0.012560380384977343,
+            -0.01536378830925542,
+            0.0012035987157572355,
+            0.033313215063595594,
+            -0.010845084209743817,
+            0.013587126023659577,
+            0.0029867620644300073,
+            0.020679673435988446,
+            -0.0006640787253300063,
+            0.0025071780490108306,
+            0.022554193713122565,
+            -0.018082906254599396,
+            0.018230000220500718,
+            -0.004451773302171552,
+            0.01915083025790679,
+            -0.004183521043835777,
+            0.015019320101074747,
+            -0.027014055979204272,
+            -0.005362044589023088,
+            -0.012355559881051253,
+            0.003995309003198048,
+            0.020680455787871383,
+            0.0054391755324872375,
+            0.009865676762244514,
+            -0.01645386866657314,
+            -0.01760657627715638,
+            0.022969828924328126,
+            -0.021566687194482292,
+            -0.015826424849757402,
+            0.00012804774676298237,
+            -0.015021970627042762,
+            0.006000610438452896,
+            0.010126179023323158,
+            -0.02912121211344025,
+            -0.02046150879033127,
+            0.004881850039543051,
+            -0.0023469840669773042,
+            -0.00915786433296249,
+            0.006547377770403066,
+            -0.022329063637464074,
+            -0.015702777677743352,
+            -0.005692904224111613,
+            0.01207999389934878,
+            -0.008429160038192635,
+            0.010075061963131637,
+            -0.020106712664936504,
+            0.0036787045650556907,
+            -0.002764965946892891,
+            0.010307702251974629,
+            0.004697169247387421,
+            0.011192285826515207,
+            0.0030113901179382946,
+            0.0014774681536380047,
+            -0.02778270000997313,
+            0.010043261430740039,
+            -5.178406450222847e-05,
+            0.04216636255860709,
+            0.030977096295679525,
+            0.004479857028809617,
+            -5.798020745008712e-05,
+            -0.009996471945562616,
+            0.012476009017187798,
+            -0.019306907391548302,
+            -0.005092322343599383,
+            0.01856463597985508,
+            0.02195467141594795,
+            -0.02236232264139305,
+            0.0052217593371508145,
+            -0.016766955100332014,
+            0.016133299654723637,
+            -0.011208763137897244,
+            0.006248024409576809,
+            0.011601988603146743,
+            0.0022330878390492527,
+            0.010317781892717903,
+            -0.005895142237225119,
+            0.019670196605183973,
+            0.01186677200787666,
+            -0.002339367025714548,
+            0.02367408853110795,
+            0.006858134385159451,
+            0.004479683673962342,
+            0.004798719889930319,
+            -0.009604173975576537,
+            -0.009045602998213713,
+            0.009230969488015385,
+            0.004083616606774546,
+            -0.007944628589259331,
+            0.011836323287097144,
+            0.015100350195476968,
+            0.01885317701149666,
+            -0.0011707847499298092,
+            -0.00684548433712413,
+            -0.006550059418141426,
+            0.001899758393077906,
+            -0.024577381680573596,
+            -0.02855511854789239,
+            0.021238935710272536,
+            -0.010427135883958709,
+            -0.019187769881105637,
+            -0.0015864640188683347,
+            0.016360037306061392,
+            -0.008853972319021563,
+            -0.01627167079143546,
+            0.001270414822216867,
+            0.02255755055961612,
+            -0.005959411125391464,
+            0.006009654905468478,
+            0.005909233773418581,
+            0.005874361519568704,
+            -0.0027859463941992753,
+            -0.024599789577567636,
+            0.0036612374968223453,
+            0.03221346865585554,
+            0.016461563205582293,
+            0.016139782455492073,
+            -0.007158168059808459,
+            0.001838455290310879,
+            0.007530954381521755,
+            0.005438791523412295,
+            0.015674000912615536,
+            0.008604106101542998,
+            -0.006990462920227135,
+            0.00244736628387359,
+            0.012213048255418866,
+            -0.007329299946651028,
+            0.017424711328323798,
+            0.001973882532418776,
+            -0.0133178791530314,
+            -0.008373452013951822,
+            -0.010462150707776556,
+            -0.014575483571750984,
+            0.0027958704799249897,
+            -0.011401916623878957,
+            0.0008016041012669959,
+            0.003953958717263787,
+            0.010260753041492368,
+            0.013961649189650642,
+            -0.0019275791443041316,
+            0.017019168629298462,
+            0.011534594243860177,
+            -0.0017859096307142257,
+            -0.002970739203219076,
+            -0.03142778833370656,
+            0.010207789755749909,
+            -0.014730553506704018,
+            -0.0037611416875830746,
+            0.01825790347981639,
+            -0.007766890847082858,
+            -0.0032778433418881666,
+            -0.021003405429807113,
+            0.0024512381432602736,
+            0.008768616692363321,
+            0.01558713989712387,
+            -0.012819542714293536,
+            0.009201155584074711,
+            -0.0012254337002053017,
+            -0.0057451574939563045,
+            0.00977251392462179,
+            0.0042356831792011935,
+            -0.0033433823868435786,
+            0.011499764125503874
+          ],
+          "title": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer",
+          "keyphrases": [
+            "Sparsely-Gated Mixture-of-Experts Layer",
+            "conditional computation",
+            "neural network",
+            "language modeling",
+            "machine translation"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "87b82083-05f3-4d69-9ea0-97d4f551aa37",
+        "properties": {
+          "page_content": "5 Experiments 5.1 1 Billion Word Language Modeling Benchmark\n\nDataset: This dataset, introduced by (Chelba et al., 2013) consists of shuffled unique sentences from news articles, totaling approximately 829 million words, with a vocabulary of 793,471 words.\n\nPrevious State-of-the-Art: The best previously published results (Jozefowicz et al., 2016) use models consisting of one or more stacked Long Short-Term Memory (LSTM) layers (Hochreiter & Schmidhuber, 1997; Gers et al., 2000). The number of parameters in the LSTM layers of these models vary from 2 million to 151 million. Quality increases greatly with parameter count, as do computational costs. Results for these models form the top line of Figure 2-right. MoE Models: Our models consist of two stacked LSTM layers with a MoE layer between them (see Figure 1). We vary the sizes of the layers and the number of experts. For full details on model architecture, training regimen, additional baselines and results, see Appendix C.\n\nLow Computation, Varied Capacity: To investigate the effects of adding capacity, we trained\n\na series of MoE models all with roughly equal computational costs: about 8 million multiply-andadds per training example per timestep in the forwards pass, excluding the softmax layer. We call this metric (ops/timestep). We trained models with flat MoEs containing 4, 32, and 256 experts, and models with hierarchical MoEs containing 256, 1024, and 4096 experts. Each expert had about 1 million parameters. For all the MoE layers, 4 experts were active per input.\n\nThe results of these models are shown in Figure 2-left. The model with 4 always-active experts performed (unsurprisingly) similarly to the computationally-matched baseline models, while the largest of the models (4096 experts) achieved an impressive 24% lower perplexity on the test set. Figure 2: Model comparison on 1-Billion-Word Language-Modeling Benchmark. On the left, we plot test perplexity as a function of model capacity for models with similar computational budgets of approximately 8-million-ops-per-timestep. On the right, we plot test perplexity as a function of computational budget. The top line represents the LSTM models from (Jozefowicz et al., 2016).\n\nThe bottom line represents 4-billion parameter MoE models with different computational budgets.\n\nVaried Computation, High Capacity: In addition to the largest model from the previous section, we trained two more MoE models with similarly high capacity (4 billion parameters), but higher computation budgets. These models had larger LSTMs, and fewer but larger and experts. Details\n\nTest Test #Parameters ops/timestep Training TFLOPS Perplexity Perplexity excluding embedding Time /GPU 10 epochs 100 epochs and softmax layers 10 epochs Best Published Results 34.7 30.6 151 million 151 million 59 hours, 32 k40s 1.09 Low-Budget MoE Model 34.1 4303 million 8.9 million 15 hours, 16 k40s 0.74 Medium-Budget MoE Model 31.3 4313 million 33.8 million 17 hours, 32 k40s 1.22 High-Budget MoE Model 28.0 4371 million 142.7 million 47 hours, 32 k40s 1.56\n\nTable 1: Summary of high-capacity MoE-augmented models with varying computational budgets, vs. best previously published results (Jozefowicz et al., 2016). Details in Appendix C. can be found in Appendix C.2. Results of these three models form the bottom line of Figure 2-right.\n\nTable 1 compares the results of these models to the best previously-published result on this dataset .\n\nEven the fastest of these models beats the best published result (when controlling for the number of training epochs), despite requiring only 6% of the computation.\n\nComputational Efficiency: We trained our models using TensorFlow (Abadi et al., 2016) on clusters containing 16-32 Tesla K40 GPUs. For each of our models, we determine computational efficiency in TFLOPS/GPU by dividing the number of floating point operations required to process one training batch by the observed step time and the number of GPUs in the cluster. The operation counts used here are higher than the ones we report in our ops/timestep numbers in that we include the backwards pass, we include the importance-sampling-based training of the softmax layer, and we count a multiply-and-add as two separate operations. For all of our MoE models, the floating point operations involved in the experts represent between 37% and 46% of the total.\n\nFor our baseline models wtih no MoE, observed computational efficiency ranged from 1.07-1.29 TFLOPS/GPU. For our low-computation MoE models, computation efficiency ranged from 0.740.90 TFLOPS/GPU, except for the 4-expert model which did not make full use of the available parallelism. Our highest-computation MoE model was more efficient at 1.56 TFLOPS/GPU, likely due to the larger matrices. These numbers represent a significant fraction of the theoretical maximum of 4.29 TFLOPS/GPU claimed by NVIDIA. Detailed results are in Appendix C, Table 7.\n\n5.2 100 BILLION WORD GOOGLE NEWS CORPUS\n\nOn the 1-billion-word corpus, adding additional capacity seems to produce diminishing returns as the number of parameters in the MoE layer exceeds 1 billion, as can be seen in Figure 2-left. We hypothesized that for a larger training set, even higher capacities would produce significant quality improvements. We constructed a similar training set consisting of shuffled unique sentences from Google's internal news corpus, totalling roughly 100 billion words. Similarly to the previous section, we tested a series of models with similar computational costs of about 8 million ops/timestep. In addition to a baseline LSTM model, we trained models augmented with MoE layers containing 32, 256, 1024, 4096, 16384, 65536, and 131072 experts. This corresponds to up to 137 billion parameters in the MoE layer. Details on architecture, training, and results are given in Appendix D.\n\nResults: Figure 3 shows test perplexity as a function of capacity after training on 10 billion words (top line) and 100 billion words (bottom line). When training over the full 100 billion words, test perplexity improves significantly up to 65536 experts (68 billion parameters), dropping 39% lower than the computationally matched baseline, but degrades at 131072 experts, possibly a result of too much sparsity. The widening gap between the two lines demonstrates (unsurprisingly) that increased model capacity helps more on larger training sets. Even at 65536 experts (99.994% layer sparsity), computational efficiency for the model stays at a respectable 0.72 TFLOPS/GPU.\n\n5.3 Machine Translation (Single Language Pair)\n\nModel Architecture: Our model was a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decreased the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We inserted MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). Each MoE layer contained up to 2048 experts each with about two million parameters, adding a total of about 8 billion parameters to the models. Further details on model architecture, testing procedure and results can be found in Appendix E.\n\nDatasets: We benchmarked our method on the WMT'14 En→Fr and En→De corpora, whose training sets have 36M sentence pairs and 5M sentence pairs, respectively. The experimental protocols were also similar to those in (Wu et al., 2016): newstest2014 was used as the test set to compare against previous work (Luong et al., 2015a; Zhou et al., 2016; Wu et al., 2016), while the combination of newstest2012 and newstest2013 was used as the development set. We also tested the same model on a Google's Production English to French data.\n\nTable 2: Results on WMT'14 En→ Fr newstest2014 (bold values represent best results). Model Test Test ops/timenstep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.69 40.35 85M 8.7B 3 days/64 k40s MoE with 2048 Experts (longer training) 2.63 40.56 85M 8.7B 6 days/64 k40s GNMT (Wu et al., 2016) 2.79 39.22 214M 278M 6 days/96 k80s GNMT+RL (Wu et al., 2016) 2.96 39.92 214M 278M 6 days/96 k80s PBMT (Durrani et al., 2014) 37.0 LSTM (6-layer) (Luong et al., 2015b) 31.5 LSTM (6-layer+PosUnk) (Luong et al., 2015b) 33.1 DeepAtt (Zhou et al., 2016) 37.7 DeepAtt+PosUnk (Zhou et al., 2016) 39.2\n\nTable 3: Results on WMT'14 En → De newstest2014 (bold values represent best results). Model Test Test ops/timestep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 4.64 26.03 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 5.25 24.91 214M 278M 1 day/96 k80s GNMT +RL (Wu et al., 2016) 8.08 24.66 214M 278M 1 day/96 k80s PBMT (Durrani et al., 2014) 20.7 DeepAtt (Zhou et al., 2016) 20.6\n\nTable 4: Results on the Google Production En→ Fr dataset (bold values represent best results). Model Eval Eval Test Test ops/timestep Total Training Perplexity BLEU Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.60 37.27 2.69 36.57 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 2.78 35.80 2.87 35.56 214M 278M 6 days/96 k80s\n\nResults: Tables 2, 3, and 4 show the results of our largest models, compared with published results. Our approach achieved BLEU scores of 40.56 and 26.03 on the WMT'14 En→Fr and En→De benchmarks. As our models did not use RL refinement, these results constitute significant gains of 1.34 and 1.12 BLEU score on top of the strong baselines in (Wu et al., 2016). The perplexity scores are also better.2 On the Google Production dataset, our model achieved 1.01 higher test BLEU score even after training for only one sixth of the time.\n\n5.4 Multilingual Machine Translation\n\nDataset: (Johnson et al., 2016) train a single GNMT (Wu et al., 2016) model on a very large combined dataset of twelve language pairs. Results are somewhat worse than those for 12 separately trained single-pair GNMT models. This is not surprising, given that the twelve models have 12 times the capacity and twelve times the aggregate training of the one model. We repeat this experiment with a single MoE-augmented model. See Appendix E for details on model architecture.\n\nWe train our model on the same dataset as (Johnson et al., 2016) and process the same number of training examples (about 3 billion sentence pairs). Our training time was shorter due to the lower computational budget of our model.\n\nResults: Results for the single-pair GNMT models, the multilingual GNMT model and the multilingual MoE model are given in Table 5. The MoE model achieves 19% lower perplexity on the dev set than the multilingual GNMT model. On BLEU score, the MoE model significantly beats the multilingual GNMT model on 11 of the 12 language pairs (by as much as 5.84 points), and even beats the monolingual GNMT models on 8 of 12 language pairs. The poor performance on English → Korean seems to be a result of severe overtraining, as for the rarer language pairs a small number of real examples were highly oversampled in the training corpus.\n\nTable 5: Multilingual Machine Translation (bold values represent best results). GNMT-Mono GNMT-Multi MoE-Multi MoE-Multi vs. GNMT-Multi Parameters 278M / model 278M 8.7B ops/timestep 212M 212M 102M training time, hardware various 21 days, 96 k20s 12 days, 64 k40s Perplexity (dev) 4.14 3.35 -19% French → English Test BLEU 36.47 34.40 37.46 +3.06 German → English Test BLEU 31.77 31.17 34.80 +3.63 Japanese → English Test BLEU 23.41 21.62 25.91 +4.29 Korean → English Test BLEU 25.42 22.87 28.71 +5.84 Portuguese → English Test BLEU 44.40 42.53 46.13 +3.60 Spanish → English Test BLEU 38.00 36.04 39.39 +3.35 English → French Test BLEU 35.37 34.00 36.59 +2.59 English → German Test BLEU 26.43 23.15 24.53 +1.38 English → Japanese Test BLEU 23.66 21.10 22.78 +1.68 English → Korean Test BLEU 19.75 18.41 16.62 -1.79 English → Portuguese Test BLEU 38.40 37.35 37.90 +0.55 English → Spanish Test BLEU 34.50 34.25 36.21 +1.96\n\n",
+          "embedding": [
+            -0.009427943266928196,
+            0.0009052514214999974,
+            -0.031882621347904205,
+            0.02155720256268978,
+            -0.0010102161904796958,
+            0.017834007740020752,
+            -0.019673390313982964,
+            0.04869475215673447,
+            -0.022961175069212914,
+            0.03497491776943207,
+            0.036041226238012314,
+            -0.002160385251045227,
+            0.013355513103306293,
+            -0.01921132393181324,
+            -0.013746492564678192,
+            0.007433058228343725,
+            -0.00785069540143013,
+            0.004056415054947138,
+            0.003800945123657584,
+            -0.0150793781504035,
+            -0.03134946897625923,
+            -0.014439593069255352,
+            -0.04698865860700607,
+            -0.015088263899087906,
+            -0.016625525429844856,
+            0.005407072138041258,
+            0.02432960458099842,
+            -0.0009768940508365631,
+            -0.040199827402830124,
+            0.01593242585659027,
+            0.004467388149350882,
+            0.0005736961611546576,
+            0.009525688365101814,
+            -0.008481594733893871,
+            -0.0005428731674328446,
+            -0.017078706994652748,
+            0.07435724139213562,
+            -0.004718414973467588,
+            -0.03138501197099686,
+            0.03760514408349991,
+            0.028381574898958206,
+            -0.006029085721820593,
+            -0.0007786272908560932,
+            0.0035210391506552696,
+            0.0162967462092638,
+            0.003805388230830431,
+            -0.01993996649980545,
+            0.007339756470173597,
+            -0.01571027748286724,
+            -0.037356339395046234,
+            -0.006606669165194035,
+            -0.03152718394994736,
+            0.0029923280235379934,
+            -0.02411634288728237,
+            -0.0463133305311203,
+            0.014768371358513832,
+            0.028292717412114143,
+            0.02548477053642273,
+            0.06902569532394409,
+            0.034797199070453644,
+            0.01878480054438114,
+            0.0006881021545268595,
+            -0.024222973734140396,
+            -0.02600015327334404,
+            0.03351762890815735,
+            -0.012244774959981441,
+            -0.02527150884270668,
+            0.005011649336665869,
+            0.016243431717157364,
+            -0.049334537237882614,
+            -0.03643220290541649,
+            0.037498511373996735,
+            0.009730064310133457,
+            -0.005127166397869587,
+            0.015026062726974487,
+            -0.022961175069212914,
+            0.04986768960952759,
+            0.045104846358299255,
+            -0.012777929194271564,
+            -0.0024103012401610613,
+            0.013133365660905838,
+            0.017505230382084846,
+            -0.008437165059149265,
+            -0.004611784126609564,
+            0.0228723157197237,
+            -0.022374704480171204,
+            -0.014164130203425884,
+            -0.07983095943927765,
+            0.0133999427780509,
+            -0.028097227215766907,
+            -0.03525926545262337,
+            0.015585875138640404,
+            0.015976853668689728,
+            -0.0007325316546484828,
+            -0.002772401785477996,
+            0.031171750277280807,
+            -0.024080798029899597,
+            -0.003567690262570977,
+            0.001345103606581688,
+            0.04762844368815422,
+            -0.01612791419029236,
+            2.6570933187031187e-05,
+            0.04848149046301842,
+            0.01478614378720522,
+            0.027048690244555473,
+            -0.012058171443641186,
+            -0.032824527472257614,
+            0.006117944605648518,
+            0.023796450346708298,
+            0.12625092267990112,
+            0.007517474237829447,
+            -0.004080851096659899,
+            -0.03270012512803078,
+            -0.01029876247048378,
+            -0.03732079640030861,
+            0.002994549460709095,
+            -0.030158756300807,
+            -0.02031317539513111,
+            0.021361712366342545,
+            -0.017620746046304703,
+            0.03206034004688263,
+            0.02752852812409401,
+            0.019246866926550865,
+            -0.020757470279932022,
+            -0.03646774962544441,
+            0.04862366244196892,
+            -0.004369643051177263,
+            0.012031513266265392,
+            0.0041452739387750626,
+            -0.024436235427856445,
+            -0.01317779440432787,
+            0.014368506148457527,
+            -0.004580683074891567,
+            -0.02411634288728237,
+            0.03417518362402916,
+            -0.020064370706677437,
+            0.026195643469691277,
+            0.04286559671163559,
+            -0.01928240992128849,
+            -0.015434814617037773,
+            -0.002169271232560277,
+            -0.013426600024104118,
+            0.0041008442640304565,
+            -0.018535993993282318,
+            -0.05061410740017891,
+            -0.006962105166167021,
+            -0.007939554750919342,
+            0.0924844816327095,
+            -0.012831244617700577,
+            0.023245522752404213,
+            -0.02607124112546444,
+            3.4328742913203314e-05,
+            -0.015274868346750736,
+            0.0298033207654953,
+            -0.013568774797022343,
+            -0.0069754342548549175,
+            -0.01673215627670288,
+            0.02038426324725151,
+            -0.04506930336356163,
+            0.015665847808122635,
+            -0.01008550077676773,
+            -0.010938547551631927,
+            -0.008441608399152756,
+            0.02219698764383793,
+            0.0029367911629378796,
+            0.035170406103134155,
+            0.05530586466193199,
+            -0.003258905140683055,
+            -0.06625329703092575,
+            0.04009319469332695,
+            0.004449616186320782,
+            0.009005863219499588,
+            -0.00478505901992321,
+            0.001390643883496523,
+            0.002912354888394475,
+            0.027155321091413498,
+            -0.004562911577522755,
+            0.016341175884008408,
+            -0.046846482902765274,
+            0.04155048355460167,
+            0.001919355127029121,
+            0.02104181982576847,
+            0.03643220290541649,
+            0.037143077701330185,
+            0.017149792984128,
+            0.006753286812454462,
+            0.03156273066997528,
+            0.02264128252863884,
+            0.006522253155708313,
+            0.015203780494630337,
+            0.006531138904392719,
+            0.011631648056209087,
+            0.0467398539185524,
+            0.004736186470836401,
+            0.0028012811671942472,
+            -0.008152816444635391,
+            -0.014608425088226795,
+            0.060957297682762146,
+            -0.0039808847941458225,
+            0.0034966031089425087,
+            0.011924882419407368,
+            -0.04453614726662636,
+            0.006513367407023907,
+            -0.010334305465221405,
+            0.02148611471056938,
+            0.017647404223680496,
+            0.015585875138640404,
+            0.032895613461732864,
+            0.021237310022115707,
+            -0.01701650582253933,
+            -0.030798541381955147,
+            -0.02534259669482708,
+            -0.02900358848273754,
+            -0.018011726438999176,
+            -0.004891689866781235,
+            0.023831993341445923,
+            -0.008308319374918938,
+            0.020490894094109535,
+            -0.008143930695950985,
+            -0.05004540830850601,
+            0.010680856183171272,
+            0.024347376078367233,
+            0.0011462816037237644,
+            -0.03884917125105858,
+            -0.03179376199841499,
+            -0.01540815643966198,
+            -0.025360368192195892,
+            -0.009223568253219128,
+            0.029110219329595566,
+            -0.01116958074271679,
+            0.02555585838854313,
+            0.02571580372750759,
+            -0.02038426324725151,
+            0.012555781751871109,
+            0.04592235013842583,
+            -0.016270089894533157,
+            0.0072997696697711945,
+            -0.00013884223881177604,
+            -0.025147106498479843,
+            0.009321312420070171,
+            -0.0197977926582098,
+            -0.019673390313982964,
+            -0.0338730625808239,
+            -0.017140908166766167,
+            -0.016909874975681305,
+            -0.02219698764383793,
+            -0.012466922402381897,
+            0.010494251735508442,
+            0.013702062889933586,
+            0.022694597020745277,
+            0.012031513266265392,
+            -0.03536589443683624,
+            0.015665847808122635,
+            0.007130937650799751,
+            -0.022730141878128052,
+            0.06554242223501205,
+            0.045460283756256104,
+            -0.004136388190090656,
+            -0.01041427906602621,
+            -0.046704307198524475,
+            0.0391690619289875,
+            0.032095883041620255,
+            -0.057296305894851685,
+            0.03236246109008789,
+            -0.017913982272148132,
+            0.042403530329465866,
+            0.02841711975634098,
+            0.025680260732769966,
+            -0.014377391897141933,
+            0.01803838461637497,
+            -0.005087179597467184,
+            -0.04777061566710472,
+            -0.034441761672496796,
+            -0.03433513268828392,
+            0.0076729776337742805,
+            -0.003112287726253271,
+            -0.012760157696902752,
+            -0.02900358848273754,
+            0.04133722186088562,
+            0.03214919939637184,
+            0.03563247248530388,
+            0.02104181982576847,
+            0.0030700797215104103,
+            0.00974783580750227,
+            0.016616638749837875,
+            -0.01890920288860798,
+            -0.014812801033258438,
+            -0.00617126002907753,
+            -0.03870699554681778,
+            -0.01540815643966198,
+            -0.017345283180475235,
+            0.002443623496219516,
+            -0.03756960108876228,
+            0.014155244454741478,
+            0.006144602317363024,
+            0.0014572881627827883,
+            0.011214010417461395,
+            0.002150388667359948,
+            0.03664546459913254,
+            -0.0016416707076132298,
+            -0.026906516402959824,
+            0.009943326003849506,
+            0.010405393317341805,
+            0.00527822645381093,
+            -0.029892180114984512,
+            -0.013799807988107204,
+            -0.006024642381817102,
+            0.027812877669930458,
+            -0.026266731321811676,
+            -0.009436829946935177,
+            -0.0020393149461597204,
+            0.0025813549291342497,
+            0.0016538887284696102,
+            -0.02024208754301071,
+            -0.016554437577724457,
+            0.014626197516918182,
+            0.0052115824073553085,
+            0.006637769751250744,
+            -0.04396745190024376,
+            -0.0198688805103302,
+            0.05573238804936409,
+            0.01586133800446987,
+            -0.06284110993146896,
+            0.023725362494587898,
+            -0.01658109575510025,
+            -0.011853795498609543,
+            0.006913233082741499,
+            0.02504047565162182,
+            -0.008406064473092556,
+            -0.01062754075974226,
+            0.0030611937399953604,
+            -0.052995529025793076,
+            -0.019389040768146515,
+            0.0035099319647997618,
+            0.004220804199576378,
+            -0.049263447523117065,
+            -0.019833335652947426,
+            -0.007597447372972965,
+            -0.015719164162874222,
+            0.003709864802658558,
+            0.009658977389335632,
+            -0.017416371032595634,
+            -0.010423164814710617,
+            0.012928989715874195,
+            0.018020613119006157,
+            -0.02299671806395054,
+            -0.03636111691594124,
+            0.027350811287760735,
+            -0.0019337947014719248,
+            -0.008099501021206379,
+            0.0016805464401841164,
+            0.012573553249239922,
+            0.0016838787123560905,
+            0.017771806567907333,
+            -0.005153824109584093,
+            -0.032682351768016815,
+            -0.02818608656525612,
+            0.02752852812409401,
+            -0.021734919399023056,
+            0.004147495608776808,
+            0.02278345637023449,
+            0.008303876966238022,
+            -0.011187352240085602,
+            0.020046597346663475,
+            -0.0527467243373394,
+            -0.008135044947266579,
+            0.0006581122288480401,
+            -0.006269005127251148,
+            0.025307053700089455,
+            -0.016829900443553925,
+            -0.014821686781942844,
+            0.019833335652947426,
+            0.009005863219499588,
+            0.0021626066882163286,
+            0.004014207050204277,
+            0.005966884084045887,
+            0.05512814596295357,
+            0.016154572367668152,
+            -0.01197819784283638,
+            0.017860665917396545,
+            -0.024596180766820908,
+            0.03102957457304001,
+            0.021290624514222145,
+            0.00529599841684103,
+            0.001194043317809701,
+            0.04581571742892265,
+            0.01008550077676773,
+            0.008757057599723339,
+            -0.011507244780659676,
+            0.011516130529344082,
+            -0.007401957642287016,
+            -0.05594564974308014,
+            -0.00806395709514618,
+            -0.022534651681780815,
+            0.002414744347333908,
+            0.0233699269592762,
+            0.031438328325748444,
+            -0.018376048654317856,
+            -0.00924133975058794,
+            -0.0009363520657643676,
+            -0.012102600187063217,
+            0.0010907446267083287,
+            -0.004687313921749592,
+            0.014359620399773121,
+            -0.0027457440737634897,
+            0.007686306722462177,
+            0.0018615968292579055,
+            -0.003058972302824259,
+            0.021343940868973732,
+            -0.03257572278380394,
+            -0.011853795498609543,
+            -0.03788949176669121,
+            -0.059784360229969025,
+            -0.03449507802724838,
+            0.030709682032465935,
+            0.023725362494587898,
+            -0.002539146924391389,
+            -0.004589569289237261,
+            -0.014324076473712921,
+            -0.031438328325748444,
+            -0.021592745557427406,
+            0.0006164595251902938,
+            0.04894355684518814,
+            -0.007868467830121517,
+            -0.02578689157962799,
+            0.01761186122894287,
+            -0.006939890794456005,
+            -0.004389636218547821,
+            -0.004927233327180147,
+            -0.03636111691594124,
+            0.023405469954013824,
+            0.010174359194934368,
+            0.03053196519613266,
+            -0.022143671289086342,
+            0.005575904622673988,
+            -0.006095729768276215,
+            -0.014377391897141933,
+            0.029963266104459763,
+            -0.008859246037900448,
+            0.004727300722151995,
+            0.0058869109489023685,
+            -0.003043422009795904,
+            0.0001384951319778338,
+            -0.005260454956442118,
+            -0.032540179789066315,
+            -0.026017924770712852,
+            -0.041230592876672745,
+            0.002370314672589302,
+            -0.02825717255473137,
+            -0.027137549594044685,
+            -0.06650210171937943,
+            0.024880530312657356,
+            -0.07098059356212616,
+            -0.02498716115951538,
+            0.038529276847839355,
+            -0.008885903283953667,
+            0.01768294721841812,
+            -0.04279451072216034,
+            -0.0006664427346549928,
+            -0.05107617378234863,
+            -0.0009580114856362343,
+            0.035241492092609406,
+            0.0029234623070806265,
+            -0.006393407471477985,
+            0.014368506148457527,
+            -0.00013120591756887734,
+            -0.03389083594083786,
+            -0.04229690134525299,
+            0.0038276028353720903,
+            0.005207139533013105,
+            -0.02301448956131935,
+            -0.014288532547652721,
+            0.04339875280857086,
+            0.00908583588898182,
+            -0.0006070183007977903,
+            0.0300876684486866,
+            0.007939554750919342,
+            -0.027137549594044685,
+            0.019246866926550865,
+            0.02564471773803234,
+            0.038884714245796204,
+            -0.02768847532570362,
+            0.021130679175257683,
+            -0.03200702369213104,
+            -0.005629220046103001,
+            0.011569445952773094,
+            0.0023636503610759974,
+            -0.006011313758790493,
+            0.014457364566624165,
+            -0.0015683620003983378,
+            0.0034077439922839403,
+            0.006842145696282387,
+            -0.006304548587650061,
+            0.032984472811222076,
+            -0.0004387414955999702,
+            -0.016821015626192093,
+            -0.021148450672626495,
+            0.009694520384073257,
+            0.031082890927791595,
+            0.028737012296915054,
+            -0.018304960802197456,
+            -0.014635083265602589,
+            -0.003976441919803619,
+            0.029056904837489128,
+            0.016083484515547752,
+            -0.01384423766285181,
+            0.0067444005981087685,
+            0.02431183122098446,
+            -0.023618731647729874,
+            -0.01586133800446987,
+            0.017398599535226822,
+            -0.006424508057534695,
+            0.012564667500555515,
+            -0.012644641101360321,
+            0.00200932496227324,
+            0.017931753769516945,
+            -0.005367085803300142,
+            0.026693254709243774,
+            -0.017034277319908142,
+            0.010529795661568642,
+            -0.007459715940058231,
+            -0.008681527338922024,
+            0.007917339913547039,
+            0.06948776543140411,
+            -0.020615296438336372,
+            0.022659054026007652,
+            0.018464908003807068,
+            -0.0018671504221856594,
+            0.034726109355688095,
+            -0.04457169026136398,
+            -0.046704307198524475,
+            0.012182573787868023,
+            -0.04314994812011719,
+            -0.0020493115298449993,
+            -0.022410249337553978,
+            -0.023032261058688164,
+            0.008321648463606834,
+            0.011915996670722961,
+            -0.024880530312657356,
+            -0.008717071264982224,
+            0.020864101126790047,
+            0.01577247865498066,
+            -0.02169937640428543,
+            0.019922195002436638,
+            0.00920579582452774,
+            -0.018749255686998367,
+            0.032682351768016815,
+            -0.02205481193959713,
+            -0.0003795946831814945,
+            -0.007099837064743042,
+            -0.0024947174824774265,
+            -0.00668219942599535,
+            -0.004285227041691542,
+            0.008868131786584854,
+            -0.02893250249326229,
+            -0.01622566021978855,
+            0.01680324412882328,
+            -0.00772185018286109,
+            -0.010058842599391937,
+            -0.042403530329465866,
+            0.0032611265778541565,
+            0.013782036490738392,
+            -0.023281067609786987,
+            0.01132064126431942,
+            -0.019477900117635727,
+            -0.032895613461732864,
+            0.01993996649980545,
+            0.03508154675364494,
+            -0.032984472811222076,
+            0.0016561102820560336,
+            0.00873040035367012,
+            -0.009028078056871891,
+            -0.0024502878077328205,
+            -0.01766517572104931,
+            0.007157595362514257,
+            0.03271789476275444,
+            0.014741714112460613,
+            -0.013257768005132675,
+            -0.024222973734140396,
+            -0.017807351425290108,
+            0.021379483863711357,
+            -0.0059313406236469746,
+            0.005162709858268499,
+            0.02562694624066353,
+            0.041514940559864044,
+            0.012173688039183617,
+            0.0008830366423353553,
+            -0.013666519895195961,
+            0.014803915284574032,
+            -0.015887996181845665,
+            0.021646060049533844,
+            0.022072583436965942,
+            -0.01746080070734024,
+            -0.03803166747093201,
+            0.018429363146424294,
+            0.00022339716088026762,
+            0.004554025363177061,
+            0.008112830109894276,
+            0.02768847532570362,
+            -0.016616638749837875,
+            -0.00549593148753047,
+            0.03934678062796593,
+            0.016314519569277763,
+            -0.014892774634063244,
+            -0.01622566021978855,
+            0.016838787123560905,
+            0.0074419439770281315,
+            0.006060186307877302,
+            -0.0018293854082003236,
+            -0.010369849391281605,
+            -0.0016127914423123002,
+            0.009365742094814777,
+            -0.008481594733893871,
+            0.008992534130811691,
+            0.014839459210634232,
+            0.005438172724097967,
+            0.0011035181814804673,
+            -0.01737194135785103,
+            -0.01263575442135334,
+            0.015799136832356453,
+            0.010538681410253048,
+            -0.010023298673331738,
+            0.00927688367664814,
+            0.007961769588291645,
+            0.014013069681823254,
+            0.032166969031095505,
+            -0.0031278380192816257,
+            -0.01671438477933407,
+            -0.0026657709386199713,
+            -0.0006958773010410368,
+            -0.025769120082259178,
+            0.007601890247315168,
+            -0.014892774634063244,
+            -0.02898581698536873,
+            0.018429363146424294,
+            0.02249910682439804,
+            -0.011418386362493038,
+            -0.01802949793636799,
+            -0.0030878514517098665,
+            -0.0043563139624893665,
+            -0.006264561787247658,
+            0.007406400516629219,
+            -0.011276211589574814,
+            0.042616792023181915,
+            -0.001704982714727521,
+            0.0023614289239048958,
+            -0.02703091874718666,
+            -0.0023503215052187443,
+            0.04332766681909561,
+            0.025253737345337868,
+            0.0030545294284820557,
+            0.009116937406361103,
+            0.008010641671717167,
+            0.0467398539185524,
+            -0.013764264062047005,
+            -0.04133722186088562,
+            0.008601554669439793,
+            -0.015319298021495342,
+            0.012173688039183617,
+            0.03266458213329315,
+            -0.009436829946935177,
+            0.010014412924647331,
+            0.010547567158937454,
+            -0.03147387132048607,
+            0.027350811287760735,
+            -0.023796450346708298,
+            -0.022605737671256065,
+            0.030994031578302383,
+            -0.01723865233361721,
+            0.01074305735528469,
+            0.028737012296915054,
+            0.010911889374256134,
+            -0.029838863760232925,
+            0.0013817580183967948,
+            -0.010423164814710617,
+            -0.025218194350600243,
+            0.025911293923854828,
+            0.02863038145005703,
+            0.018642624840140343,
+            0.0063001057133078575,
+            0.0015339291421696544,
+            0.004896132741123438,
+            0.0033699788618832827,
+            -0.01007661409676075,
+            0.013204452581703663,
+            -0.010165473446249962,
+            -0.01768294721841812,
+            -0.0015283754328265786,
+            -0.013568774797022343,
+            -0.00927688367664814,
+            0.0021748249419033527,
+            -0.02607124112546444,
+            0.009134708903729916,
+            0.012351405806839466,
+            0.010458708740770817,
+            -0.018962517380714417,
+            -0.01862485334277153,
+            0.05534140765666962,
+            0.013799807988107204,
+            -0.0032700125593692064,
+            0.004087515641003847,
+            -0.017585203051567078,
+            -0.04318549111485481,
+            0.004571797326207161,
+            0.011418386362493038,
+            0.005562575533986092,
+            0.00020270967797841877,
+            -0.030496420338749886,
+            0.008108386769890785,
+            0.032326918095350266,
+            0.008046185597777367,
+            0.04446506127715111,
+            -0.0012251439038664103,
+            0.030123213306069374,
+            -0.008414950221776962,
+            0.003574354574084282,
+            0.019513443112373352,
+            0.0041230591014027596,
+            -0.019033605232834816,
+            0.03038978949189186,
+            0.01834050565958023,
+            0.018304960802197456,
+            0.006979877129197121,
+            -0.04514038935303688,
+            -0.009872239083051682,
+            0.010236560367047787,
+            0.03182930499315262,
+            0.020259859040379524,
+            -0.017176451161503792,
+            -0.007437501102685928,
+            0.03296670317649841,
+            -0.022232530638575554,
+            -0.004431844223290682,
+            0.0116138756275177,
+            -0.008597111329436302,
+            0.002483610063791275,
+            -0.004687313921749592,
+            0.02249910682439804,
+            -0.039844390004873276,
+            -0.004687313921749592,
+            0.0019349055364727974,
+            0.0057225218042731285,
+            -0.002215922111645341,
+            -0.006939890794456005,
+            0.010405393317341805,
+            0.014261875301599503,
+            0.009076950140297413,
+            0.02943011187016964,
+            -0.0201532281935215,
+            0.015034948475658894,
+            -0.015168237499892712,
+            -0.003134502563625574,
+            0.005749179515987635,
+            -0.03138501197099686,
+            -0.030283158645033836,
+            0.007619662210345268,
+            0.03175821900367737,
+            0.013080050237476826,
+            0.0014817244373261929,
+            -0.004589569289237261,
+            -0.011516130529344082,
+            0.026853200048208237,
+            0.02109513431787491,
+            0.02111290581524372,
+            0.023831993341445923,
+            -0.005087179597467184,
+            -0.0004323547473177314,
+            0.012475808151066303,
+            -0.008650426752865314,
+            -0.024791670963168144,
+            0.008930332958698273,
+            -0.011427272111177444,
+            0.0270842332392931,
+            -0.01724753901362419,
+            -0.003258905140683055,
+            0.005646991543471813,
+            -0.012404721230268478,
+            -0.01128509733825922,
+            -0.039702218025922775,
+            0.025893522426486015,
+            -0.02548477053642273,
+            0.0036343345418572426,
+            -0.019193550571799278,
+            0.01490166038274765,
+            -0.03870699554681778,
+            0.000813615508377552,
+            -0.0018393819918856025,
+            0.010538681410253048,
+            -0.010529795661568642,
+            0.0021770463790744543,
+            -0.03193593770265579,
+            -0.03788949176669121,
+            -0.0116849634796381,
+            0.005073850974440575,
+            0.007899568416178226,
+            -0.0009330198518000543,
+            0.006913233082741499,
+            0.015967968851327896,
+            -0.025093792006373405,
+            -0.03369534760713577,
+            0.018820343539118767,
+            -0.03440621867775917,
+            0.020419806241989136,
+            0.0004870585980825126,
+            -0.02804391086101532,
+            -0.021965952590107918,
+            -0.016492236405611038,
+            -0.015248210169374943,
+            0.021006274968385696,
+            0.022552423179149628,
+            0.017798464745283127,
+            -0.016847673803567886,
+            0.0025124892126768827,
+            0.014039727859199047,
+            0.008121715858578682,
+            0.011880452744662762,
+            0.008481594733893871,
+            0.01825164631009102,
+            -0.016847673803567886,
+            -0.015887996181845665,
+            -0.012244774959981441,
+            -0.002794616622850299,
+            -0.013799807988107204,
+            0.0011179577559232712,
+            -0.038813624531030655,
+            -0.00737529993057251,
+            0.01116958074271679,
+            -0.0020048818551003933,
+            0.06038859859108925,
+            0.0527467243373394,
+            0.013160022906959057,
+            0.007384185679256916,
+            -0.01876702904701233,
+            0.0461711548268795,
+            -0.025804663076996803,
+            0.04986768960952759,
+            0.0034521734341979027,
+            -0.013142251409590244,
+            -0.013462143950164318,
+            0.013426600024104118,
+            -0.029199078679084778,
+            -0.00719313882291317,
+            -0.005424844101071358,
+            -0.009005863219499588,
+            0.018002839758992195,
+            0.00739307189360261,
+            -0.009347970597445965,
+            0.003696535946801305,
+            0.009383514523506165,
+            -0.011258440092206001,
+            0.013222224079072475,
+            0.01612791419029236,
+            0.0017349726986140013,
+            0.015070492401719093,
+            -0.04208363965153694,
+            -0.011231781914830208,
+            -0.00041291682282462716,
+            0.009685634635388851,
+            -0.006220132578164339,
+            0.013168908655643463,
+            -0.006428950931876898,
+            -0.0044229584746062756,
+            -0.007015420589596033,
+            -0.02169937640428543,
+            0.022534651681780815,
+            0.036041226238012314,
+            -0.0016938752960413694,
+            -0.048979099839925766,
+            -0.02614232711493969,
+            -0.050223127007484436,
+            -0.012928989715874195,
+            0.024063026532530785,
+            -0.0050249784253537655,
+            0.015976853668689728,
+            0.004087515641003847,
+            0.0048828041180968285,
+            -0.0030456434469670057,
+            -0.017567431554198265,
+            0.01613680087029934,
+            -0.0232277512550354,
+            0.002739079762250185,
+            0.0003854260721709579,
+            0.007144266273826361,
+            -0.03543698415160179,
+            0.007757393643260002,
+            0.00023422685626428574,
+            0.012093714438378811,
+            -0.03660992160439491,
+            -0.038244929164648056,
+            -0.017398599535226822,
+            0.015941310673952103,
+            -0.002383643528446555,
+            -0.012244774959981441,
+            0.0029501200187951326,
+            0.005033864174038172,
+            0.014741714112460613,
+            -0.017567431554198265,
+            -0.006229018326848745,
+            0.02162828855216503,
+            -0.0032433548476547003,
+            0.007206467911601067,
+            -0.035099320113658905,
+            -0.01555921696126461,
+            -0.01278681494295597,
+            0.0116494195535779,
+            -0.0133999427780509,
+            0.004905018489807844,
+            0.00017799575289245695,
+            0.014848344959318638,
+            -0.015505901537835598,
+            -0.027013147249817848,
+            -0.011231781914830208,
+            0.020099913701415062,
+            -0.03511708974838257,
+            0.036112312227487564,
+            0.002739079762250185,
+            0.02118399366736412,
+            -0.014404049143195152,
+            0.025609172880649567,
+            -0.004036421421915293,
+            -0.015301525592803955,
+            -0.002151499269530177,
+            0.022001497447490692,
+            0.03332213684916496,
+            -0.021361712366342545,
+            0.0013228888856247067,
+            -0.019015833735466003,
+            0.003327770857140422,
+            -0.02351210080087185,
+            0.017345283180475235,
+            -0.004007542505860329,
+            0.01234252005815506,
+            0.017105363309383392,
+            0.013204452581703663,
+            0.05996207520365715,
+            -0.017194222658872604,
+            0.02557362988591194,
+            0.012324748560786247,
+            -0.017265310510993004,
+            -0.007935111410915852,
+            -0.032095883041620255,
+            -0.027333039790391922,
+            0.03133169561624527,
+            0.002228140365332365,
+            0.0015561438631266356,
+            -0.03104734607040882,
+            0.015879109501838684,
+            0.01832273229956627,
+            0.019460128620266914,
+            -0.014288532547652721,
+            -0.015274868346750736,
+            0.016030170023441315,
+            0.026248957961797714,
+            -0.008135044947266579,
+            0.003865367965772748,
+            0.012769043445587158,
+            -0.04076852649450302,
+            0.005282669793814421,
+            0.0029634488746523857,
+            0.012857901863753796,
+            -0.0021548315417021513,
+            0.010183244943618774,
+            -0.03214919939637184,
+            -0.0055137029848992825,
+            -0.01457288209348917,
+            0.02315666526556015,
+            -0.015665847808122635,
+            -0.047806162387132645,
+            -0.03161604329943657,
+            0.025147106498479843,
+            -0.030194299295544624,
+            -0.020988503471016884,
+            0.01311559323221445,
+            0.0022170329466462135,
+            0.015337069518864155,
+            -0.01471505593508482,
+            -0.02747521363198757,
+            -0.038955800235271454,
+            -0.021965952590107918,
+            -0.0006886575138196349,
+            0.0394534096121788,
+            -0.03010544180870056,
+            0.015159351751208305,
+            0.005309327505528927,
+            0.00686436053365469,
+            -0.004869475029408932,
+            0.002365871798247099,
+            -0.029447883367538452,
+            -0.04133722186088562,
+            0.02351210080087185,
+            -0.00494944816455245,
+            -0.0015150465769693255,
+            0.014803915284574032,
+            -0.033055562525987625,
+            -0.023103348910808563,
+            0.003154495730996132,
+            0.00737529993057251,
+            -0.00785069540143013,
+            -0.002352542942389846,
+            0.0013395498972386122,
+            -0.01215591561049223,
+            -0.038600362837314606,
+            -0.004416293930262327,
+            0.03397969529032707,
+            0.02177046425640583,
+            0.031740445643663406,
+            0.011525017209351063,
+            0.00010468705295352265,
+            -0.02886141464114189,
+            -0.013595432043075562,
+            -0.006895461119711399,
+            -0.0009818923426792026,
+            -0.0025769120547920465,
+            0.01848267950117588,
+            0.005291555542498827,
+            0.0034077439922839403,
+            -0.02169937640428543,
+            -0.02783064916729927,
+            -0.03511708974838257,
+            -0.01928240992128849,
+            -0.010049956850707531,
+            -0.00949014537036419,
+            -0.03899134323000908,
+            -0.016741042956709862,
+            0.04485604166984558,
+            0.026995373889803886,
+            -0.009152480401098728,
+            -0.00018729818111751229,
+            -0.0012240331852808595,
+            -0.012973419390618801,
+            0.019886652007699013,
+            -0.013826466165482998,
+            0.0059091257862746716,
+            0.00031656029750593007,
+            0.018607081845402718,
+            0.0017538551473990083,
+            0.0010013302089646459,
+            -0.03419295698404312,
+            -0.009401286020874977,
+            0.031171750277280807,
+            0.00701097771525383,
+            -0.019531216472387314,
+            0.0163678340613842,
+            0.009259111247956753,
+            -0.014466251246631145,
+            -0.047806162387132645,
+            0.026782112196087837,
+            0.04567354544997215,
+            0.02212589979171753,
+            -0.005940226372331381,
+            0.022179214283823967,
+            0.024933844804763794,
+            -0.0012462479062378407,
+            -0.0007541910745203495,
+            -0.004945005290210247,
+            0.026977602392435074,
+            0.006535581778734922,
+            -0.000127387756947428,
+            0.00939240027219057,
+            0.01810947060585022,
+            0.005838038399815559,
+            -0.0007697413675487041,
+            0.007077622227370739,
+            -0.020793013274669647,
+            -0.004913904704153538,
+            -0.013853123411536217,
+            0.010974090546369553,
+            0.00920579582452774,
+            0.041088417172431946,
+            0.008774830028414726,
+            0.013559889048337936,
+            -0.0043563139624893665,
+            -0.010174359194934368,
+            0.028648152947425842,
+            0.014039727859199047,
+            0.001577247865498066,
+            0.008499366231262684,
+            -0.030834084376692772,
+            -0.01586133800446987,
+            -0.013950868509709835,
+            0.01384423766285181,
+            0.0037520728074014187,
+            0.04286559671163559,
+            0.006873246282339096,
+            0.0016794357215985656,
+            -0.02943011187016964,
+            0.0026235629338771105,
+            0.008232789114117622,
+            0.02285454422235489,
+            -0.013266653753817081,
+            -2.6171763238380663e-05,
+            0.0073664141818881035,
+            0.005638105794787407,
+            0.026497764512896538,
+            -0.026195643469691277,
+            -0.017052048817276955,
+            0.019300181418657303,
+            0.09511470794677734,
+            -0.009330199100077152,
+            -0.0233699269592762,
+            -0.026053469628095627,
+            -0.02104181982576847,
+            0.008241675794124603,
+            0.017585203051567078,
+            0.005429286975413561,
+            0.017443029209971428,
+            0.010218788869678974,
+            0.01788732409477234,
+            0.015603646636009216,
+            -0.015976853668689728,
+            -0.007419729605317116,
+            -0.03225582838058472,
+            0.0327356681227684,
+            0.00014911657490301877,
+            -0.0047228578478097916,
+            0.00739307189360261,
+            0.007135380525141954,
+            0.019744478166103363,
+            0.018091699108481407,
+            0.015887996181845665,
+            -0.007979541085660458,
+            -0.011924882419407368,
+            0.020135456696152687,
+            -0.013142251409590244,
+            -0.0034810525830835104,
+            -0.01899806223809719,
+            -0.021592745557427406,
+            -0.005651434417814016,
+            0.014643969014286995,
+            -0.006468937732279301,
+            0.014848344959318638,
+            -0.01132064126431942,
+            0.028310488909482956,
+            -0.021965952590107918,
+            0.011160694994032383,
+            -0.00737529993057251,
+            -0.01526598259806633,
+            0.005216025281697512,
+            0.010245447047054768,
+            -0.007677420508116484,
+            0.010716399177908897,
+            -0.014324076473712921,
+            0.014821686781942844,
+            0.019477900117635727,
+            -0.03650329262018204,
+            0.03328659385442734,
+            0.008126158267259598,
+            -0.009836695156991482,
+            0.00924133975058794,
+            -0.00582470977678895,
+            0.012529123574495316,
+            0.007268669083714485,
+            0.00806395709514618,
+            0.008352749049663544,
+            0.00817947369068861,
+            -0.007659649010747671,
+            0.014883887954056263,
+            0.012138144113123417,
+            0.0007353085093200207,
+            0.010636426508426666,
+            0.014279646798968315,
+            -0.009730064310133457,
+            -0.004696200136095285,
+            0.0038364888168871403,
+            -0.010325419716536999,
+            0.007317541632801294,
+            -0.009108050726354122,
+            0.0060912868939340115,
+            0.024507321417331696,
+            -0.0017083149868994951,
+            -0.016492236405611038,
+            -0.016625525429844856,
+            -0.007935111410915852,
+            -0.021237310022115707,
+            -0.015434814617037773,
+            0.00514938123524189,
+            -0.030834084376692772,
+            0.013053392060101032,
+            0.028665924444794655,
+            0.0233699269592762,
+            -0.006180145777761936,
+            -0.024720583111047745,
+            -0.012413606978952885,
+            -0.0018582645570859313,
+            0.011276211589574814,
+            0.005162709858268499,
+            -0.02454286627471447,
+            0.002643556334078312,
+            -0.011356184259057045,
+            -0.021432798355817795,
+            -0.008277218788862228,
+            -0.01651000790297985,
+            0.003458837978541851,
+            0.0030456434469670057,
+            -0.008441608399152756,
+            -0.010707513429224491,
+            0.003541032550856471,
+            -0.03685872629284859,
+            0.014048613607883453,
+            0.03207810968160629,
+            -0.000934130628593266,
+            -0.03255794942378998,
+            -0.015221552923321724,
+            -0.003250019159168005,
+            -0.017194222658872604,
+            -0.02863038145005703,
+            0.008934776298701763,
+            0.006104615516960621,
+            -0.008454937487840652,
+            0.01252023782581091,
+            -0.004036421421915293,
+            0.02001105435192585,
+            0.0009130265680141747,
+            0.01869594119489193,
+            -0.0408751554787159,
+            0.012564667500555515,
+            0.019158007577061653,
+            -0.019122464582324028,
+            0.006904346868395805,
+            0.023458784446120262,
+            -0.008383849635720253,
+            0.0034699453972280025,
+            0.035756874829530716,
+            0.010369849391281605,
+            -0.042474620044231415,
+            0.01905137673020363,
+            0.006082401145249605,
+            -0.00668219942599535,
+            0.016323404386639595,
+            0.019673390313982964,
+            -0.000788623932749033,
+            0.020757470279932022,
+            -0.006446722894906998,
+            -0.018233874812722206,
+            0.00218260008841753,
+            0.020846329629421234,
+            -0.009330199100077152,
+            -0.015345955267548561,
+            0.01092077512294054,
+            0.0027235294692218304,
+            0.004758401308208704,
+            0.01738082803785801,
+            -0.02607124112546444,
+            -0.018642624840140343,
+            0.019513443112373352,
+            0.012831244617700577,
+            0.017336398363113403,
+            -0.011214010417461395,
+            -0.01402195543050766,
+            0.02219698764383793,
+            -0.005464830435812473,
+            0.010707513429224491,
+            -0.031082890927791595,
+            0.008748171851038933,
+            -0.0019904423970729113,
+            0.01848267950117588,
+            -0.004918347578495741,
+            0.008752615191042423,
+            0.03255794942378998,
+            -0.015514787286520004,
+            -0.025964610278606415,
+            0.009543460793793201,
+            -0.033624257892370224,
+            -0.010094386525452137,
+            -0.029607830569148064,
+            -0.03920460492372513,
+            0.03170490264892578,
+            -0.008046185597777367,
+            -0.0049583339132368565,
+            0.026746569201350212,
+            0.007353085093200207,
+            0.0020270966924726963,
+            -0.03213142603635788,
+            0.021574974060058594,
+            0.006642212625592947,
+            0.007784051354974508,
+            -0.018873659893870354,
+            0.025147106498479843,
+            -0.0030922943260520697,
+            -0.021166222169995308,
+            0.013462143950164318,
+            0.012182573787868023,
+            0.008126158267259598,
+            -0.004534032195806503,
+            0.001586133730597794,
+            -0.03682318329811096,
+            -0.005775837227702141,
+            0.008246118202805519,
+            0.005522589199244976,
+            -0.003236690303310752,
+            0.018802572041749954,
+            -0.016909874975681305,
+            -0.002843489171937108,
+            -0.005629220046103001,
+            -0.022747913375496864,
+            0.0033855291549116373,
+            -0.03248686343431473,
+            -0.020686382427811623,
+            0.019780021160840988,
+            -0.003989770542830229,
+            -0.01825164631009102,
+            0.014581767842173576,
+            -0.03980884701013565,
+            -0.011178466491401196,
+            0.007304212544113398,
+            0.027777334675192833,
+            -0.03447730466723442,
+            0.015719164162874222,
+            -0.01958453096449375,
+            -0.0023769792169332504,
+            0.023618731647729874,
+            -0.011809365823864937,
+            -0.00010725563333835453,
+            -0.012564667500555515,
+            -0.031882621347904205,
+            0.003963112831115723,
+            0.028239401057362556,
+            -0.029234621673822403,
+            0.0009446826297789812,
+            0.031136205419898033,
+            -0.000653669296298176,
+            0.004616227000951767,
+            0.0007586340070702136,
+            0.01113403681665659,
+            -0.015186008997261524,
+            0.015106036327779293,
+            0.005838038399815559,
+            0.00582470977678895,
+            -0.01372872106730938,
+            -0.005202696658670902,
+            0.02710200473666191,
+            0.001230697613209486,
+            -0.008819258771836758,
+            0.0047006430104374886,
+            0.004058636259287596,
+            0.010174359194934368,
+            -0.0016627747099846601,
+            0.014457364566624165,
+            0.018100585788488388,
+            -0.028168313205242157,
+            0.012884560041129589,
+            0.009836695156991482,
+            7.844586798455566e-05,
+            0.013995298184454441,
+            -0.017514115199446678,
+            -0.034352902323007584,
+            0.02161051705479622,
+            -0.022019268944859505,
+            -0.016234545037150383,
+            0.0270842332392931,
+            -0.010867459699511528,
+            -0.006064629182219505,
+            -0.009774493984878063,
+            -0.001900472561828792,
+            0.010609768331050873,
+            -0.009978869929909706,
+            -0.009463487192988396,
+            0.023760905489325523,
+            0.018145015463232994,
+            0.020917417481541634,
+            0.029856635257601738,
+            0.0012118150480091572,
+            0.01570139080286026,
+            -0.0005223245243541896,
+            0.008974762633442879,
+            0.005171595606952906,
+            -0.0018627075478434563,
+            -0.01738971285521984,
+            0.025982381775975227,
+            -0.001804949133656919,
+            0.006264561787247658,
+            0.0063223205506801605,
+            -0.014368506148457527,
+            -0.010867459699511528,
+            -0.026479993015527725,
+            -0.004887246992439032,
+            -0.006580011453479528,
+            -0.016234545037150383,
+            0.0789068192243576,
+            -0.0099344402551651,
+            -0.016554437577724457,
+            0.010494251735508442,
+            -0.004856146406382322,
+            0.0015528115909546614,
+            -0.016847673803567886,
+            0.003736522514373064,
+            -0.008783715777099133,
+            -0.000988556770607829,
+            -0.030176527798175812,
+            -0.0015616975724697113,
+            0.0167943574488163,
+            0.015283754095435143,
+            -0.006557796616107225,
+            -0.018731484189629555,
+            0.008223903365433216,
+            -0.023209979757666588,
+            0.02036648988723755,
+            0.0012418050318956375,
+            0.018447136506438255,
+            0.005087179597467184,
+            0.005082736723124981,
+            -0.014946090057492256,
+            -0.016749927774071693,
+            0.004434065893292427,
+            0.012315861880779266,
+            0.002247022930532694,
+            -0.0090236347168684,
+            -0.009152480401098728,
+            -0.011009634472429752,
+            -0.021859321743249893,
+            0.020544208586215973,
+            -0.01296453271061182,
+            -0.0039142402820289135,
+            -0.003983106464147568,
+            -0.0020726369693875313,
+            0.006331206299364567,
+            0.011356184259057045,
+            -0.02358318865299225,
+            0.0018849222688004375,
+            -0.016972076147794724,
+            0.006913233082741499,
+            0.02504047565162182,
+            0.00981892365962267,
+            -0.04720192030072212,
+            -0.00651781028136611,
+            -0.02754630148410797,
+            0.013746492564678192,
+            -0.011747164651751518,
+            -0.009445715695619583,
+            0.0133999427780509,
+            0.008072842843830585,
+            0.028719240799546242,
+            0.010280990041792393,
+            0.009881124831736088,
+            0.015372613444924355,
+            -0.00032850075513124466,
+            -0.004440730437636375,
+            -0.017647404223680496,
+            0.013568774797022343,
+            -0.023316610604524612,
+            0.020348718389868736,
+            -0.0006103505147621036,
+            -0.017034277319908142,
+            0.004194146487861872,
+            0.002821274334564805,
+            0.013853123411536217,
+            -0.021806007251143456,
+            -0.022481335327029228,
+            0.022730141878128052,
+            0.018216101452708244,
+            0.012600211426615715,
+            0.015123807825148106,
+            0.03182930499315262,
+            0.02139725536108017,
+            0.024294059723615646,
+            -0.025324825197458267,
+            0.0014428485883399844,
+            -0.033855292946100235,
+            -0.005087179597467184,
+            -0.026551079005002975,
+            0.004038643091917038,
+            -0.012973419390618801,
+            0.005638105794787407,
+            -0.022001497447490692,
+            0.001768294838257134,
+            0.017549660056829453,
+            -0.030691910535097122,
+            0.010965204797685146,
+            0.015754707157611847,
+            0.00788623932749033,
+            0.017993954941630363,
+            0.006944333668798208,
+            -0.012831244617700577,
+            -0.01737194135785103,
+            -0.008717071264982224,
+            -0.013782036490738392,
+            0.014812801033258438,
+            0.0020082141272723675,
+            -0.0027679589111357927,
+            -0.00027990597300231457,
+            0.0001258604897884652,
+            -0.014466251246631145,
+            -0.001746080000884831,
+            -0.005180481821298599,
+            -0.013710948638617992,
+            0.005926897749304771,
+            0.0014928317395970225,
+            0.013222224079072475,
+            0.008917003870010376,
+            -0.007286441046744585,
+            0.011293983086943626,
+            0.02578689157962799,
+            0.0005528698093257844,
+            -0.018358277156949043,
+            0.00721091078594327,
+            0.008290547877550125,
+            0.007215353660285473,
+            0.006073514930903912,
+            0.0015172680141404271,
+            -0.012253660708665848,
+            0.027333039790391922,
+            -0.023938624188303947,
+            0.02205481193959713,
+            0.004856146406382322,
+            -0.0008075064979493618,
+            0.0003068413643632084,
+            -0.03518817946314812,
+            0.002401415491476655,
+            -0.024222973734140396,
+            -0.04112396016716957,
+            -0.02411634288728237,
+            -0.01547924429178238,
+            0.0007264225860126317,
+            0.022765684872865677,
+            0.0012862344738095999,
+            -0.027297494933009148,
+            0.022907858714461327,
+            -0.001922687399201095,
+            -0.015959082171320915,
+            0.00420081103220582,
+            -0.001362875453196466,
+            0.0019515665480867028,
+            0.0028745897579938173,
+            -0.0029567843303084373,
+            -0.0081839170306921,
+            -0.017407484352588654,
+            -0.02635558880865574,
+            0.003163381712511182,
+            0.009223568253219128,
+            -0.019815564155578613,
+            0.02249910682439804,
+            -0.009978869929909706,
+            -0.015879109501838684,
+            0.026497764512896538,
+            -0.009312426671385765,
+            -0.021343940868973732,
+            -0.021308396011590958,
+            0.02095296047627926,
+            0.004669542424380779,
+            -0.024418462067842484,
+            0.01643003523349762,
+            0.005562575533986092,
+            -0.017638517543673515,
+            -0.002251465804874897,
+            -0.012040399014949799,
+            0.012351405806839466,
+            -0.01782512292265892,
+            0.0016472243005409837,
+            0.010503137484192848,
+            0.0017738484311848879,
+            0.0030278717167675495,
+            -0.01095631904900074,
+            0.019993282854557037,
+            0.01825164631009102,
+            0.008046185597777367,
+            -0.03200702369213104,
+            0.008819258771836758,
+            -0.007588561624288559,
+            -0.0007630769396200776,
+            -0.020419806241989136,
+            0.029821092262864113,
+            0.013435485772788525,
+            0.007975098676979542,
+            0.01055645290762186,
+            -0.006491152569651604,
+            0.006282333750277758,
+            -0.009685634635388851,
+            -0.002785730641335249,
+            -0.013319969177246094,
+            -0.029678918421268463,
+            -0.007593004498630762,
+            0.023760905489325523,
+            -0.006451165769249201,
+            0.01095631904900074,
+            -0.004316327627748251,
+            0.029270166531205177,
+            -0.015106036327779293,
+            0.000624234729912132,
+            0.02477389946579933,
+            -0.0022370261140167713,
+            0.0058646961115300655,
+            0.011533902958035469,
+            0.06394296139478683,
+            -0.003012321190908551,
+            0.013239996507763863,
+            0.0013817580183967948,
+            0.002856818027794361,
+            0.007948440499603748,
+            -0.015070492401719093,
+            -0.03190039470791817,
+            0.001449513016268611,
+            0.0043718647211790085,
+            -0.013435485772788525,
+            -0.0016072377329692245,
+            -0.0018793685594573617,
+            0.001987110124900937,
+            -0.004909461829811335,
+            0.018153900280594826,
+            -0.020935188978910446,
+            0.028737012296915054,
+            0.019851109012961388,
+            0.012884560041129589,
+            -0.010387620888650417,
+            0.015941310673952103,
+            0.025253737345337868,
+            -0.008121715858578682,
+            -0.0038498176727443933,
+            0.007401957642287016,
+            0.0053937435150146484,
+            0.023760905489325523,
+            0.0027079791761934757,
+            0.029981037601828575,
+            -0.0035343680065125227,
+            0.003134502563625574,
+            -0.014270761050283909,
+            -0.001956009538844228,
+            -0.0033988580107688904,
+            0.005904682911932468,
+            0.003414408303797245,
+            0.0014061941765248775,
+            -0.016403377056121826,
+            0.013648747466504574,
+            0.004176374524831772,
+            0.003367757424712181,
+            0.01456399541348219,
+            -0.0006292330217547715,
+            -0.002503603231161833,
+            0.013444371521472931,
+            -0.002197039546445012,
+            -0.0013151137391105294,
+            0.008143930695950985,
+            -0.012315861880779266,
+            -0.016332291066646576,
+            -0.008255003951489925,
+            0.000503719667904079,
+            -0.006344534922391176,
+            -0.01958453096449375,
+            -0.004847260192036629,
+            0.022232530638575554,
+            0.0022259189281612635,
+            0.031011803075671196,
+            -0.003445509122684598,
+            0.008277218788862228,
+            0.016980960965156555,
+            -0.011489473283290863,
+            -0.0005217691650614142,
+            0.009312426671385765,
+            -0.0034566165413707495,
+            -0.0016960968496277928,
+            -0.004984991624951363,
+            0.00048372638411819935,
+            -0.009001419879496098,
+            3.720972017617896e-05,
+            0.0167232695966959,
+            0.011036291718482971,
+            0.0025458112359046936,
+            -0.005913568660616875,
+            -0.0017027612775564194,
+            -0.011844909749925137,
+            -0.006979877129197121,
+            -0.004487381316721439,
+            0.0017105364240705967,
+            -0.0011201791930943727,
+            0.01182713732123375,
+            0.00040319786057807505,
+            0.010218788869678974,
+            -0.010982976295053959,
+            -0.0011357294861227274,
+            -0.006548910867422819,
+            0.020419806241989136,
+            0.0053937435150146484,
+            -0.0043496498838067055,
+            0.007566346786916256,
+            -0.011027405969798565,
+            0.021379483863711357,
+            0.005291555542498827,
+            -0.00226479466073215,
+            0.0039075762033462524,
+            -0.0017560767009854317,
+            0.01149835903197527,
+            -0.0010962983360514045,
+            -0.013239996507763863,
+            0.02891472913324833,
+            0.017638517543673515,
+            0.027404125779867172,
+            0.0017405262915417552,
+            0.005735850892961025,
+            -0.00888146087527275,
+            0.027510756626725197,
+            0.01585245132446289,
+            -0.0017427478451281786,
+            -0.028150541707873344,
+            0.004243019036948681,
+            -0.012067057192325592,
+            -0.001978224376216531,
+            0.009614547714591026,
+            -0.01095631904900074,
+            -0.009552346542477608,
+            -0.01607459969818592,
+            -0.03138501197099686,
+            -0.026782112196087837,
+            0.0038164956495165825,
+            0.007068736013025045,
+            -0.008312762714922428,
+            0.0008847027784213424,
+            -0.015301525592803955,
+            -0.009543460793793201,
+            -0.025591401383280754,
+            0.002565804636105895,
+            -0.007313098758459091,
+            -0.037356339395046234,
+            -0.02381422184407711,
+            -0.009330199100077152,
+            -0.007166481111198664,
+            0.006611112039536238,
+            -0.0036898714024573565,
+            -0.02461395226418972,
+            0.011729392223060131,
+            -0.00974783580750227,
+            0.011542788706719875,
+            0.017700720578432083,
+            0.007535246200859547,
+            -0.010547567158937454,
+            -0.013559889048337936,
+            -0.009481258690357208,
+            -0.01628786139190197,
+            0.02052643708884716,
+            -0.010360963642597198,
+            0.006215689703822136,
+            -0.0058646961115300655,
+            -0.0018138349987566471,
+            0.0014184123137965798,
+            0.01832273229956627,
+            -0.03796058148145676,
+            -0.027190864086151123,
+            -0.00651781028136611,
+            0.011382842436432838,
+            0.0004651214985642582,
+            -0.012600211426615715,
+            0.014261875301599503,
+            -0.006406736560165882,
+            -0.02031317539513111,
+            -0.014519566670060158,
+            -0.001221811748109758,
+            -0.011054064147174358,
+            0.00840162206441164,
+            -0.011569445952773094,
+            0.01555921696126461,
+            0.0035299251321703196,
+            0.005349313840270042,
+            -0.00651781028136611,
+            0.015967968851327896,
+            0.008921447210013866,
+            -0.021343940868973732,
+            -0.003747629700228572,
+            -0.009463487192988396,
+            0.0017660732846707106,
+            0.01622566021978855,
+            0.007966212928295135,
+            -0.019815564155578613,
+            0.01738082803785801,
+            0.02687097154557705,
+            0.01315113715827465,
+            0.02118399366736412,
+            0.002925683744251728,
+            0.00752636045217514,
+            -0.001412858604453504,
+            0.011062949895858765,
+            -0.004811716731637716,
+            0.0002893472264986485,
+            0.00702874967828393,
+            -0.01628786139190197,
+            -0.011604989878833294,
+            0.0359523668885231,
+            0.000578694452997297,
+            -0.021219536662101746,
+            0.018153900280594826,
+            -0.005753622390329838,
+            -0.009076950140297413,
+            0.005900240037590265,
+            -0.0020581972785294056,
+            0.002081522950902581,
+            -0.010156587697565556,
+            0.009854466654360294,
+            -0.007961769588291645,
+            0.0057003069669008255,
+            0.006557796616107225,
+            0.004651770461350679,
+            -0.016190115362405777,
+            0.01658109575510025,
+            0.029092447832226753,
+            0.02616010047495365,
+            -0.042616792023181915,
+            -0.006753286812454462,
+            0.005424844101071358,
+            -0.018944745883345604,
+            -0.013675405643880367,
+            0.026017924770712852,
+            -0.0022559086792171,
+            -0.011658305302262306,
+            0.014039727859199047,
+            -0.009472372941672802,
+            -0.015434814617037773,
+            -0.0014961640117689967,
+            0.023938624188303947,
+            0.008623769506812096,
+            0.006402293220162392,
+            -0.007939554750919342,
+            0.005220468156039715,
+            -0.021077362820506096,
+            0.006242347415536642,
+            -0.006095729768276215,
+            0.009481258690357208,
+            -0.005349313840270042,
+            -0.005487045273184776,
+            0.012946761213243008,
+            0.005442616064101458,
+            -0.004847260192036629,
+            0.018589310348033905,
+            -0.00038292689714580774,
+            0.001768294838257134,
+            -0.012422492727637291,
+            0.0201532281935215,
+            -0.000900253071449697,
+            -0.009952211752533913,
+            -0.005424844101071358,
+            0.0197267048060894,
+            0.012511352077126503,
+            -0.0030878514517098665,
+            -0.0029145763255655766,
+            0.006055743433535099,
+            -3.049322731385473e-05,
+            0.000538985594175756,
+            -0.003632113104686141,
+            -0.008348306640982628,
+            -0.004402965307235718,
+            -0.007779608480632305,
+            0.015461471863090992,
+            -0.0028101669158786535,
+            -0.012742385268211365,
+            0.011427272111177444,
+            -0.003472166834399104,
+            -0.00031905947253108025,
+            -0.016607753932476044,
+            0.0011729393154382706,
+            0.005815824028104544,
+            0.026017924770712852,
+            0.012760157696902752,
+            -0.014519566670060158,
+            -0.025164877995848656,
+            -0.014484022743999958,
+            0.03270012512803078,
+            0.014377391897141933,
+            -0.004289669916033745,
+            0.01694541797041893,
+            0.005913568660616875,
+            0.021059591323137283,
+            0.005020535551011562,
+            -0.003063415177166462,
+            0.012102600187063217,
+            0.028132770210504532,
+            0.00480283098295331,
+            -0.01577247865498066,
+            -0.0042830053716897964,
+            -0.0017694055568426847,
+            -0.014128586277365685,
+            -0.008930332958698273,
+            -0.006495595443993807,
+            -0.02438291907310486,
+            0.01620788872241974,
+            -0.013497686944901943,
+            0.013666519895195961,
+            -0.014555109664797783,
+            -0.004509596154093742,
+            -0.027261951938271523,
+            0.002392529509961605,
+            0.021450571715831757,
+            -0.0035899050999432802,
+            0.012689069844782352,
+            -0.02738635428249836,
+            -0.020846329629421234,
+            0.019371269270777702,
+            -0.019922195002436638,
+            -0.00027823986602015793,
+            -0.0022559086792171,
+            -0.0034988245461136103,
+            -0.0051449378952383995,
+            -0.0010701960418373346,
+            -0.005375971551984549,
+            -0.0002896249061450362,
+            0.021432798355817795,
+            -0.004505153279751539,
+            0.0051671527326107025,
+            0.00011628037464106455,
+            0.015168237499892712,
+            0.014119700528681278,
+            -0.006544467993080616,
+            -0.007726293057203293,
+            -0.019762249663472176,
+            0.0037720659747719765,
+            -0.02249910682439804,
+            0.022321389988064766,
+            0.005011649336665869,
+            0.016403377056121826,
+            -0.007153152488172054,
+            -0.004263012204319239,
+            -0.006388964597135782,
+            -0.01047648023813963,
+            0.018660398200154305,
+            -0.010325419716536999,
+            0.007059850264340639,
+            -0.008277218788862228,
+            0.03403301164507866,
+            0.005269340705126524,
+            -0.013355513103306293,
+            0.02454286627471447,
+            0.005264897830784321,
+            0.011054064147174358,
+            -0.016741042956709862,
+            0.009427943266928196,
+            0.01022767461836338,
+            -0.016829900443553925,
+            0.01296453271061182,
+            -0.02921685017645359,
+            0.006251233164221048,
+            -0.024720583111047745,
+            -0.013071163557469845,
+            -0.017505230382084846,
+            -0.01914023607969284,
+            -0.04243907332420349,
+            -0.001449513016268611,
+            0.030691910535097122,
+            -0.005420401226729155,
+            0.017505230382084846,
+            -0.009738950058817863,
+            0.011373956687748432,
+            0.00821501761674881,
+            -0.005100508686155081,
+            0.020864101126790047,
+            0.02790173701941967,
+            -0.02052643708884716,
+            -0.006357864011079073,
+            0.0006603337242268026,
+            0.008983648382127285,
+            -0.006073514930903912,
+            -0.02921685017645359,
+            0.0050694081000983715,
+            -0.002747965743765235,
+            0.012644641101360321,
+            -0.01702539063990116,
+            0.0062778908759355545,
+            0.0007464158697985113,
+            0.011427272111177444,
+            -0.008339419960975647,
+            -0.014252989552915096,
+            -0.0180472694337368,
+            0.0016783250030130148,
+            0.0069976490922272205,
+            -0.022232530638575554,
+            0.016039054840803146,
+            -0.023903081193566322,
+            0.0014472915790975094,
+            0.015941310673952103,
+            -0.01956675946712494,
+            -0.006993206217885017,
+            -0.005438172724097967,
+            0.036929816007614136,
+            -0.0023414355237036943,
+            -0.002870146883651614,
+            0.011951540596783161,
+            0.0082194609567523,
+            0.010494251735508442,
+            -0.002934569725766778,
+            0.0051449378952383995,
+            0.0026280060410499573,
+            -0.004336320795118809,
+            0.0017427478451281786,
+            0.02761738747358322,
+            0.002912354888394475,
+            0.02125508151948452,
+            -0.0032966702710837126,
+            -0.002921240869909525,
+            0.0019349055364727974,
+            -0.007170923985540867,
+            0.009810036979615688,
+            7.4350020440761e-05,
+            -0.004203032236546278,
+            -0.017327511683106422,
+            0.015159351751208305,
+            -0.015168237499892712,
+            0.010121043771505356,
+            0.020473120734095573,
+            0.01788732409477234,
+            0.006771058309823275,
+            -0.004878360778093338,
+            -0.010547567158937454,
+            0.014928317628800869,
+            0.017078706994652748,
+            -0.00018424366135150194,
+            -0.020579751580953598,
+            0.0010801926255226135,
+            -0.00869485642760992,
+            0.003989770542830229,
+            -0.0116849634796381,
+            0.009516802616417408,
+            0.008805930614471436,
+            -0.003201146610081196,
+            0.015425928868353367,
+            0.008779272437095642,
+            0.014706170186400414,
+            0.01321333833038807,
+            -0.03317996487021446,
+            -0.0030145428609102964,
+            -0.008459379896521568,
+            -0.0020393149461597204,
+            -0.006384521722793579,
+            -0.017558544874191284,
+            -0.0004920568899251521,
+            0.016927646473050117,
+            -0.005993541795760393,
+            -0.006086844019591808,
+            0.01942458562552929,
+            -0.01420855987817049,
+            0.013853123411536217,
+            0.006180145777761936,
+            0.016305632889270782,
+            -0.02017100155353546,
+            0.003740965388715267,
+            -0.0012007077457383275,
+            -0.0037654016632586718,
+            -0.004905018489807844,
+            0.015665847808122635,
+            0.006224575452506542,
+            0.02475612796843052,
+            -0.02651553601026535,
+            0.006060186307877302,
+            -0.007779608480632305,
+            -0.011542788706719875,
+            -0.018011726438999176,
+            0.011009634472429752,
+            0.00021798232046421617,
+            -0.015372613444924355,
+            -0.01315113715827465,
+            0.006851031444966793,
+            0.011525017209351063,
+            -0.0014006405835971236,
+            0.013195566833019257,
+            0.009685634635388851,
+            0.0018615968292579055,
+            0.009738950058817863,
+            0.008605997078120708,
+            -0.02242802083492279,
+            -0.008059514686465263,
+            0.016554437577724457,
+            -0.0029056903440505266,
+            -0.010014412924647331,
+            0.018873659893870354,
+            0.006548910867422819,
+            -0.011880452744662762,
+            0.006891018245369196,
+            0.01701650582253933,
+            0.007988426834344864,
+            0.006011313758790493,
+            -0.007126494776457548,
+            0.003241133177652955,
+            0.002292563207447529,
+            0.0004895577440038323,
+            -0.020615296438336372,
+            -0.008530467748641968,
+            0.008921447210013866,
+            -0.01092077512294054,
+            0.008570454083383083,
+            -0.005282669793814421,
+            -0.023352153599262238,
+            0.01570139080286026,
+            0.0024325160775333643,
+            -0.008268333040177822,
+            -0.008015085011720657,
+            0.008090615272521973,
+            0.004487381316721439,
+            -0.008281662128865719,
+            0.0009963319171220064,
+            -0.0007880685734562576,
+            -0.012360291555523872,
+            0.022676825523376465,
+            0.00800175592303276,
+            0.0010085500543937087,
+            -0.006428950931876898,
+            -0.009116937406361103,
+            -0.0133643988519907,
+            0.031438328325748444,
+            0.013355513103306293,
+            0.023067805916070938,
+            0.01201374176889658,
+            -0.0033699788618832827,
+            -0.0062556760385632515,
+            -0.002807945478707552,
+            0.01559476088732481,
+            0.012111486867070198,
+            -0.00020451462478376925,
+            0.010831916704773903,
+            0.003245576284825802,
+            0.0037298579700291157,
+            0.006033528596162796,
+            0.005238240119069815,
+            -0.003241133177652955,
+            0.008139487355947495,
+            0.0016305632889270782,
+            0.004811716731637716,
+            0.002556918654590845,
+            0.011729392223060131,
+            0.02541368454694748,
+            -0.006544467993080616,
+            -0.008086171932518482,
+            0.01029876247048378,
+            0.012609097175300121,
+            0.004651770461350679,
+            -0.01569250598549843,
+            0.0066022262908518314,
+            -0.011391728185117245,
+            0.011276211589574814,
+            -0.02877255529165268,
+            -0.006877689156681299,
+            0.009952211752533913,
+            -0.021148450672626495,
+            0.03163381665945053,
+            -0.007419729605317116,
+            -0.024596180766820908,
+            0.007766279857605696,
+            -0.00941905751824379,
+            -0.013302197679877281,
+            -0.018429363146424294,
+            0.020402034744620323,
+            0.017505230382084846,
+            0.020419806241989136,
+            -0.012466922402381897,
+            0.01862485334277153,
+            0.004851703066378832,
+            0.022587966173887253,
+            -0.0037542942445725203,
+            -0.014439593069255352,
+            -0.010574225336313248,
+            0.013391056098043919,
+            0.009614547714591026,
+            -0.007495259400457144,
+            0.029465656727552414,
+            0.014368506148457527,
+            0.005838038399815559,
+            -0.0013339963043108582,
+            -0.00737529993057251,
+            0.004143052268773317,
+            -0.002876811195164919,
+            0.004416293930262327,
+            -0.009161366149783134,
+            -0.02111290581524372,
+            -0.0008302765781991184,
+            0.017265310510993004,
+            0.008619326166808605,
+            0.024933844804763794,
+            -0.015106036327779293,
+            0.0081839170306921,
+            0.002747965743765235,
+            -0.01044093631207943,
+            -0.02095296047627926,
+            0.002816831460222602,
+            -0.010201017372310162,
+            0.023192208260297775,
+            0.007468601688742638,
+            -0.007490816526114941,
+            -0.012022627517580986,
+            0.004451837856322527,
+            0.0034943814389407635,
+            0.024454006925225258,
+            -0.005944669246673584,
+            -0.01029876247048378,
+            0.01420855987817049,
+            0.004776173271238804,
+            0.0015372612979263067,
+            0.004038643091917038,
+            0.013506573624908924,
+            0.00668219942599535,
+            -0.0011362849036231637,
+            -0.001868261257186532,
+            0.0037298579700291157,
+            0.007544131949543953,
+            -0.0010435383301228285,
+            0.0011285097571089864,
+            0.001358432462438941,
+            0.004580683074891567,
+            0.012475808151066303,
+            -0.004098623059689999,
+            0.0027768448926508427,
+            -0.0041230591014027596,
+            -0.00912582315504551,
+            0.0014894995838403702,
+            0.0005156600964255631,
+            -0.012626868672668934,
+            -0.0009224678506143391,
+            -0.00912582315504551,
+            -0.0020126572344452143,
+            0.01417301595211029,
+            0.03367757424712181,
+            -0.010538681410253048,
+            0.022321389988064766,
+            -0.011720506474375725,
+            0.02146834321320057,
+            0.013373284600675106,
+            0.005469273775815964,
+            -0.02228584513068199,
+            -0.015088263899087906,
+            0.01405749935656786,
+            -0.017940638586878777,
+            0.004209696780890226,
+            -0.0052337972447276115,
+            -0.03024761565029621,
+            0.013053392060101032,
+            -0.03170490264892578,
+            -0.0030478648841381073,
+            0.005287112668156624,
+            -0.014332962222397327,
+            -0.009259111247956753,
+            -0.0031789320055395365,
+            -0.002585797803476453,
+            -0.0027368583250790834,
+            0.0064200651831924915,
+            -0.028026139363646507,
+            -0.0133999427780509,
+            0.005540360696613789,
+            0.009996641427278519,
+            -0.01620788872241974,
+            0.010538681410253048,
+            0.004589569289237261,
+            -0.0012617983156815171,
+            0.00462066987529397,
+            -0.018153900280594826,
+            -0.007033192552626133,
+            -0.02512933500111103,
+            -0.003809831105172634,
+            -0.00803729984909296,
+            0.015417042188346386,
+            -0.00045679096365347505,
+            -0.005953555461019278,
+            -0.0009407950565218925,
+            -0.011187352240085602,
+            0.0030834085773676634,
+            0.01055645290762186,
+            0.002639113226905465,
+            -0.008170587942004204,
+            0.01723865233361721,
+            0.02278345637023449,
+            -0.01300007663667202,
+            -0.025662489235401154,
+            0.01921132393181324,
+            -0.01577247865498066,
+            0.021574974060058594,
+            0.0005645325290970504,
+            -0.001508382149040699,
+            -0.0150793781504035,
+            0.015150465071201324,
+            0.0010563117684796453,
+            0.01216480229049921,
+            -0.014795029535889626,
+            0.02825717255473137,
+            -0.0010374292032793164,
+            0.003809831105172634,
+            -0.0073664141818881035,
+            -8.213386536226608e-06,
+            0.00888146087527275,
+            0.0019904423970729113,
+            -0.010369849391281605,
+            0.020508665591478348,
+            -0.012182573787868023,
+            0.0045473612844944,
+            -0.008446050807833672,
+            0.00020104357099626213,
+            -0.008974762633442879,
+            0.0034543948713690042,
+            -0.0017838451312854886,
+            0.012573553249239922,
+            0.012546896003186703,
+            0.015274868346750736,
+            -0.010023298673331738,
+            -0.008948104456067085,
+            0.013782036490738392,
+            -0.02498716115951538,
+            0.0013828687369823456,
+            0.0006231239531189203,
+            0.011951540596783161,
+            -0.02118399366736412,
+            0.006237904075533152,
+            0.026391133666038513,
+            0.005318213254213333,
+            0.009836695156991482,
+            -0.010716399177908897,
+            -0.018002839758992195,
+            -0.007815152406692505,
+            0.007917339913547039,
+            -0.014457364566624165,
+            -0.02059752494096756,
+            -0.013702062889933586,
+            -0.00020479230443015695,
+            -0.0021315061021596193,
+            0.004918347578495741,
+            -0.013630975969135761,
+            -0.002365871798247099,
+            -0.01664329692721367,
+            -0.0024502878077328205,
+            -0.014039727859199047,
+            -0.007042078301310539,
+            0.008090615272521973,
+            0.02855929359793663,
+            0.008077286183834076,
+            0.02402748353779316,
+            0.01237806398421526,
+            0.020917417481541634,
+            0.017834007740020752,
+            -0.004567354451864958,
+            -0.005695864092558622,
+            -0.0002508879406377673,
+            0.018660398200154305,
+            -0.03321550786495209,
+            0.01438627764582634,
+            -0.0077174073085188866,
+            -0.004138609394431114,
+            -0.0054914881475269794,
+            -0.0019304625457152724,
+            0.010796372778713703,
+            0.003976441919803619,
+            -0.004167488776147366,
+            0.010929660871624947,
+            -0.024596180766820908,
+            -0.007135380525141954,
+            -0.00735752796754241,
+            -0.01723865233361721,
+            -0.0013362177414819598,
+            0.02409856952726841,
+            -0.005424844101071358,
+            0.002994549460709095,
+            0.006984320003539324,
+            0.008437165059149265,
+            0.016492236405611038,
+            0.015808021649718285,
+            -0.0016438921447843313,
+            -0.0057669514790177345,
+            0.005398186389356852,
+            -0.004887246992439032,
+            -0.03081631287932396,
+            -0.006899903994053602,
+            0.008379407227039337,
+            0.011729392223060131,
+            -0.025289282202720642,
+            -0.011116265319287777,
+            0.0017338618636131287,
+            0.01540815643966198,
+            0.02081078663468361,
+            -0.0039297910407185555,
+            -0.016083484515547752,
+            -0.004147495608776808,
+            -0.020579751580953598,
+            0.021806007251143456,
+            -0.012804586440324783,
+            0.0005670317332260311,
+            -0.009232454001903534,
+            0.00939240027219057,
+            -0.012564667500555515,
+            0.03317996487021446,
+            -0.014590653590857983,
+            0.002401415491476655,
+            -0.0010685299057513475,
+            0.011098493821918964,
+            0.00476728705689311,
+            0.024134114384651184,
+            -0.01722976751625538,
+            -0.003796502249315381,
+            0.028683695942163467,
+            -0.033624257892370224,
+            0.016696613281965256,
+            0.008743729442358017,
+            0.01084080245345831,
+            -0.009303540922701359,
+            -0.008743729442358017,
+            0.027955051511526108,
+            0.020793013274669647,
+            0.006535581778734922,
+            -0.001412858604453504,
+            -0.019264638423919678,
+            0.023689819499850273,
+            -0.0011829358991235495,
+            0.002683542901650071,
+            -0.002794616622850299,
+            0.013453258201479912,
+            -0.034886058419942856,
+            -0.01657220907509327,
+            0.005562575533986092,
+            -0.008663755841553211,
+            -0.01892697438597679,
+            0.0005864696577191353,
+            -0.0116494195535779,
+            -0.02132616750895977,
+            -0.0032655694521963596,
+            0.01965561881661415,
+            -0.002612455515190959,
+            0.003072301158681512,
+            0.006686642300337553,
+            0.004411851055920124,
+            -0.008605997078120708,
+            -0.024525092914700508,
+            -0.01099186297506094,
+            0.003709864802658558,
+            0.007348642218858004,
+            -0.013017848134040833,
+            -0.012982305139303207,
+            -0.01077860128134489,
+            -0.0030100997537374496,
+            0.013471029698848724,
+            0.002305892063304782,
+            0.02168160490691662,
+            -0.004474052228033543,
+            0.01435073371976614,
+            0.013906438834965229,
+            -0.006851031444966793,
+            -0.008685970678925514,
+            0.013248882256448269,
+            -0.0027457440737634897,
+            -0.007504145614802837,
+            -0.007384185679256916,
+            -0.0016205665888264775,
+            -0.006584454327821732,
+            0.002288120100274682,
+            0.01956675946712494,
+            -0.01970893330872059,
+            0.004820602480322123,
+            0.0017249759985134006,
+            -0.01212925836443901,
+            0.005629220046103001,
+            0.008308319374918938,
+            -0.0011884896084666252,
+            0.011196237988770008,
+            0.003774287411943078,
+            -0.002202593255788088,
+            0.00226479466073215,
+            -0.013071163557469845,
+            0.006575568579137325,
+            0.015417042188346386,
+            0.014190787449479103,
+            0.01077860128134489,
+            0.006366749759763479,
+            -0.034299585968256,
+            -0.006415622308850288,
+            -0.003118952037766576,
+            -0.02374313399195671,
+            0.0014106371672824025,
+            0.0026280060410499573,
+            0.020259859040379524,
+            -0.003076744033023715,
+            0.014937203377485275,
+            -0.005882468074560165,
+            0.0006458940915763378,
+            0.016821015626192093,
+            0.009907782077789307,
+            -0.03573910519480705,
+            -0.010663083754479885,
+            -0.01523043867200613,
+            -0.007979541085660458,
+            -0.015177123248577118,
+            -0.014377391897141933,
+            0.008077286183834076,
+            0.017291968688368797,
+            -0.013648747466504574,
+            -0.012200345285236835,
+            0.011711620725691319,
+            0.013888667337596416,
+            0.007370857056230307,
+            0.006144602317363024,
+            0.016314519569277763,
+            -0.0018127242801710963,
+            0.0006347867310978472,
+            0.00026338372845202684,
+            0.018873659893870354,
+            -0.013026734814047813,
+            -0.0023392140865325928,
+            -0.011418386362493038,
+            0.011676076799631119,
+            0.020864101126790047,
+            -0.01076971460133791,
+            0.014448478817939758,
+            -0.007059850264340639,
+            -0.006779944524168968,
+            -0.047379638999700546,
+            -0.01321333833038807,
+            -0.009436829946935177,
+            0.028132770210504532,
+            -0.004807273857295513,
+            -0.005833595525473356,
+            0.0055803474970161915,
+            0.019602302461862564,
+            0.0063001057133078575,
+            0.008166145533323288,
+            0.01818055845797062,
+            -0.00839273538440466,
+            -0.012804586440324783,
+            0.004160824231803417,
+            0.007859582081437111,
+            -0.00927688367664814,
+            0.0027546300552785397,
+            0.0099344402551651,
+            0.032753441482782364,
+            -0.0057669514790177345,
+            0.0075085884891450405,
+            -0.009108050726354122,
+            0.0022303618025034666,
+            -0.016758814454078674,
+            0.001700539723969996,
+            -0.005473716650158167,
+            -0.006815487984567881,
+            0.014439593069255352,
+            -0.021788235753774643,
+            -0.0014795028837397695,
+            -0.0028968045953661203,
+            0.008406064473092556,
+            -0.01318668108433485,
+            0.011045178398489952,
+            -0.0041230591014027596,
+            -0.004271897953003645,
+            0.014128586277365685,
+            -0.011915996670722961,
+            0.027581844478845596,
+            -0.002074858406558633,
+            0.009969983249902725,
+            -0.007970655336976051,
+            -0.012866788543760777,
+            -0.017807351425290108,
+            0.0025369254872202873,
+            -0.0021348383743315935,
+            -0.008534910157322884,
+            0.008850360289216042,
+            -0.022463563829660416,
+            0.0036543277092278004,
+            -0.004873917903751135,
+            0.0015339291421696544,
+            -0.02592906542122364,
+            0.00015036614786367863,
+            -0.002412522677332163,
+            0.011080721393227577,
+            -0.008810373023152351,
+            -0.013160022906959057,
+            0.01176493614912033,
+            -0.010734171606600285,
+            -0.013888667337596416,
+            -0.01318668108433485,
+            0.008410507813096046,
+            0.004038643091917038,
+            -0.002434737514704466,
+            0.014457364566624165,
+            -0.011009634472429752,
+            0.0008480484248138964,
+            -0.006184588652104139,
+            0.00924133975058794,
+            -0.00047900574281811714,
+            -0.016989847645163536,
+            -4.682454527937807e-05,
+            -0.0002667159424163401,
+            0.012440265156328678,
+            0.012982305139303207,
+            0.018447136506438255,
+            -0.008441608399152756,
+            0.0026168986223638058,
+            0.0034766097087413073,
+            -0.0037809519562870264,
+            0.011187352240085602,
+            0.005775837227702141,
+            -0.013888667337596416,
+            0.0013650968903675675,
+            -0.013684291392564774,
+            0.00854379590600729,
+            -0.00891256146132946,
+            0.010209903120994568,
+            0.009108050726354122,
+            -0.0024769455194473267,
+            -0.010343191213905811,
+            0.011729392223060131,
+            0.01657220907509327,
+            -0.012333634309470654,
+            0.00032655693939886987,
+            -0.007228682283312082,
+            0.025431456044316292,
+            0.016749927774071693,
+            0.010583111084997654,
+            -0.013355513103306293,
+            0.013826466165482998,
+            -0.00025310940691269934,
+            0.0037254150956869125,
+            0.00957011803984642,
+            -0.026408905163407326,
+            -0.02491607330739498,
+            0.003736522514373064,
+            -0.0069310045801103115,
+            -0.0060468572191894054,
+            0.023316610604524612,
+            -0.012138144113123417,
+            0.006184588652104139,
+            -0.011569445952773094,
+            0.028168313205242157,
+            0.01621677353978157,
+            -0.0009307984146289527,
+            -0.015834679827094078,
+            0.0002776844776235521,
+            0.01011215802282095,
+            -0.0013417714508250356,
+            -0.007766279857605696,
+            -0.0022681269329041243,
+            -0.012538010254502296,
+            0.004398522432893515,
+            -0.003783173393458128,
+            0.004776173271238804,
+            0.004402965307235718,
+            -0.012697956524789333,
+            -0.011311755515635014,
+            0.028221629559993744,
+            0.0008724846411496401,
+            0.012973419390618801,
+            -0.004971663001924753,
+            0.0017494122730568051,
+            0.005957998335361481,
+            0.02233916148543358,
+            -0.024276288226246834,
+            0.01905137673020363,
+            0.006739957723766565,
+            0.017771806567907333,
+            0.008903675712645054,
+            -0.00839717872440815,
+            0.01249358057975769,
+            -0.006157930940389633,
+            -0.007415286265313625,
+            -0.010280990041792393,
+            -0.0014750600093975663,
+            -0.0016050162957981229,
+            -0.0076285479590296745,
+            0.009579003788530827,
+            -0.0011290650581941009,
+            -0.004905018489807844,
+            0.01420855987817049,
+            -0.022890087217092514,
+            0.014723941683769226,
+            -0.0013239996042102575,
+            0.01124955341219902,
+            -0.0021781569812446833,
+            0.011125151067972183,
+            -0.02278345637023449,
+            0.0038675894029438496,
+            -0.01797618344426155,
+            -0.0012118150480091572,
+            -0.0040941801853477955,
+            -0.001176271471194923,
+            -0.0030323145911097527,
+            0.007979541085660458,
+            -0.034139640629291534,
+            -0.010067728348076344,
+            0.015488130040466785,
+            0.013933097012341022,
+            0.0026768783573061228,
+            0.013026734814047813,
+            0.006246790289878845,
+            0.0012362513225525618,
+            -0.00615348806604743,
+            -0.009294655174016953,
+            -0.020259859040379524,
+            -0.02219698764383793,
+            -0.013053392060101032,
+            -0.0022725698072463274,
+            -0.0008230567909777164,
+            -0.005242682993412018,
+            0.008286104537546635,
+            -0.018287189304828644,
+            -0.0007325316546484828,
+            -0.0005164931644685566,
+            0.0004479050694499165,
+            0.001358432462438941,
+            0.0018538215663284063,
+            0.016341175884008408,
+            0.008765943348407745,
+            -0.0042363544926047325,
+            -0.007370857056230307,
+            0.025182651355862617,
+            -0.0030345360282808542,
+            0.012147029861807823,
+            0.020028825849294662,
+            0.0025080463383346796,
+            -0.014688398689031601,
+            0.013835351914167404,
+            -0.008290547877550125,
+            0.018642624840140343,
+            -0.012333634309470654,
+            -0.025680260732769966,
+            -0.014857230708003044,
+            0.007610776461660862,
+            -0.011871566995978355,
+            -0.009596775285899639,
+            0.01365763321518898,
+            -0.002821274334564805,
+            0.0001021878924802877,
+            0.007135380525141954,
+            -0.005762508604675531,
+            0.0025369254872202873,
+            -0.03497491776943207,
+            -0.000194934502360411,
+            -0.03001658245921135,
+            -0.030567508190870285,
+            0.014670626260340214,
+            0.006855474319308996,
+            0.021077362820506096,
+            -0.006948776543140411,
+            0.013977525755763054,
+            -0.009099164977669716,
+            0.026053469628095627,
+            0.014946090057492256,
+            -0.0006936558638699353,
+            0.0031789320055395365,
+            -0.007250897120684385,
+            -0.006371192634105682,
+            0.027155321091413498,
+            -0.008081729523837566,
+            -0.0059313406236469746,
+            0.02475612796843052,
+            0.026657709851861,
+            0.01738082803785801,
+            -0.018944745883345604,
+            0.019460128620266914,
+            0.0013017848832532763,
+            0.01789620891213417,
+            -0.016199002042412758,
+            -0.00032850075513124466,
+            -0.015665847808122635,
+            -0.008161702193319798,
+            0.05530586466193199,
+            0.01723865233361721,
+            -0.0007297547999769449,
+            -0.0202243160456419,
+            0.015017176978290081,
+            -0.0029789991676807404,
+            -0.002028207527473569,
+            0.00981892365962267,
+            0.0302298441529274,
+            -0.00920579582452774,
+            -0.013799807988107204,
+            0.013293311931192875,
+            -0.005247125867754221,
+            0.026746569201350212,
+            -0.002260351786389947,
+            0.0020904086995869875,
+            -0.015221552923321724,
+            -0.013133365660905838,
+            0.0013395498972386122,
+            0.009756721556186676,
+            -0.007939554750919342,
+            -0.0011018520453944802,
+            -0.006944333668798208,
+            0.012769043445587158,
+            0.006815487984567881,
+            -0.020277632400393486,
+            -0.003112287726253271,
+            -0.030922943726181984,
+            -0.00480283098295331,
+            -0.035525843501091,
+            -0.005944669246673584,
+            -0.005269340705126524,
+            -0.014626197516918182,
+            -0.032611265778541565,
+            -0.02073969878256321,
+            -0.009996641427278519,
+            0.004660656210035086,
+            0.002425851533189416,
+            0.010041071102023125,
+            0.002474724082276225,
+            0.0005764730158261955,
+            0.0010274326195940375,
+            0.0041230591014027596,
+            0.01303562056273222,
+            0.001622788142412901,
+            0.011729392223060131,
+            0.02233916148543358,
+            -0.018198329955339432,
+            -0.033410996198654175,
+            -0.014484022743999958,
+            0.0050249784253537655,
+            0.005686978343874216,
+            -0.0018749256851151586,
+            0.008601554669439793,
+            0.0015561438631266356,
+            0.011231781914830208,
+            -0.014324076473712921,
+            -0.005020535551011562,
+            -0.009774493984878063,
+            -0.0011773821897804737,
+            0.0026191200595349073,
+            0.003896468784660101,
+            0.0026946503203362226,
+            0.0029412340372800827,
+            -0.011711620725691319,
+            0.01014770194888115,
+            -0.005380414426326752,
+            -0.0009291322785429657,
+            -0.0009024745668284595,
+            -0.012538010254502296,
+            0.004927233327180147,
+            -0.007935111410915852,
+            -0.00753968907520175,
+            -0.008534910157322884,
+            0.0034521734341979027,
+            -0.01600351184606552,
+            -0.009036963805556297,
+            0.01716756634414196,
+            -0.006424508057534695,
+            0.007757393643260002,
+            0.012831244617700577,
+            -0.0013650968903675675,
+            -0.004562911577522755,
+            0.019229095429182053,
+            -0.012742385268211365,
+            -0.01417301595211029,
+            -0.024454006925225258,
+            -0.006446722894906998,
+            0.0005456500221043825,
+            0.015452586114406586,
+            0.0042830053716897964,
+            -0.010903003625571728,
+            -0.0034810525830835104,
+            -0.002579133491963148,
+            0.003554361406713724,
+            -0.012324748560786247,
+            0.005966884084045887,
+            -0.02388530783355236,
+            0.015959082171320915,
+            -0.006677756551653147,
+            -0.009303540922701359,
+            0.018713712692260742,
+            0.04535365104675293,
+            -0.007606333587318659,
+            -0.0177629217505455,
+            0.009014748968183994,
+            0.022836772724986076,
+            0.022676825523376465,
+            -0.007086507976055145,
+            0.013053392060101032,
+            -0.02001105435192585,
+            -0.011880452744662762,
+            -0.005082736723124981,
+            -0.002314777811989188,
+            0.022516880184412003,
+            -0.03104734607040882,
+            -0.019246866926550865,
+            -0.007784051354974508,
+            -0.03241577744483948,
+            -0.006948776543140411,
+            0.028470434248447418,
+            0.004611784126609564,
+            -0.005620333831757307,
+            -0.004143052268773317,
+            -0.002716864924877882,
+            -0.01753188669681549,
+            0.020259859040379524,
+            0.08686859160661697,
+            -0.014635083265602589,
+            -0.008294991217553616,
+            -0.004131945315748453,
+            0.00635342113673687,
+            0.005784722976386547,
+            -0.005056079011410475,
+            0.06181034445762634,
+            -0.023121120408177376,
+            0.00033433211501687765,
+            0.003609898267313838,
+            0.015816908329725266,
+            -0.007170923985540867,
+            -0.0002644944761414081,
+            0.0012107043294236064,
+            -0.005953555461019278,
+            -0.010129929520189762,
+            -0.017105363309383392,
+            -0.0045295893214643,
+            -0.0017949525499716401,
+            -0.00701097771525383,
+            -0.01620788872241974,
+            0.019335726276040077,
+            -0.023849764838814735,
+            -0.006615554913878441,
+            -0.0023347712121903896,
+            -0.006513367407023907,
+            0.009188024327158928,
+            -0.019833335652947426,
+            0.002570247510448098,
+            -0.0011773821897804737,
+            0.005380414426326752,
+            -0.005158266983926296,
+            -0.005575904622673988,
+            -0.0028234957717359066,
+            0.0009313537739217281,
+            -0.0098988963291049,
+            0.01826941780745983,
+            0.013790922239422798,
+            0.009152480401098728,
+            0.010058842599391937,
+            -0.0055803474970161915,
+            0.023121120408177376,
+            -0.005784722976386547,
+            0.0057891663163900375,
+            -0.0067888302728533745,
+            0.0014161908766254783,
+            -0.004789501894265413,
+            0.01839382015168667,
+            -0.020633067935705185,
+            -0.013071163557469845,
+            -0.014875002205371857,
+            0.035170406103134155,
+            -0.02097073197364807,
+            0.014972747303545475,
+            0.0010307647753506899,
+            0.011356184259057045,
+            0.010485365986824036,
+            0.0013051170390099287,
+            0.01702539063990116,
+            -0.012093714438378811,
+            0.016776585951447487,
+            0.002699093194678426,
+            0.015363726764917374,
+            -0.008255003951489925,
+            0.012573553249239922,
+            0.00010274325904902071,
+            0.003809831105172634,
+            -0.008637098595499992,
+            0.0044696093536913395,
+            0.011676076799631119,
+            -0.005780280102044344,
+            0.03541921079158783,
+            -0.0013728720368817449,
+            -0.02226807363331318,
+            0.016741042956709862,
+            -0.015106036327779293,
+            -0.02082855813205242,
+            0.0011740500340238214,
+            -0.014652854762971401,
+            -0.0030278717167675495,
+            0.012315861880779266,
+            -0.028381574898958206,
+            -0.020988503471016884,
+            0.005247125867754221,
+            0.001098519773222506,
+            0.0016572210006415844,
+            0.0006486709462478757,
+            -0.014439593069255352,
+            -0.0018138349987566471,
+            -0.0030656366143375635,
+            0.007957326248288155,
+            -0.014777257107198238,
+            0.006437837146222591,
+            -0.026266731321811676,
+            -0.0031389454379677773,
+            -0.003132280893623829,
+            0.023689819499850273,
+            -0.002907912014052272,
+            0.010432050563395023,
+            -0.005993541795760393,
+            -0.0025502543430775404,
+            -0.016252316534519196,
+            -0.0033522071316838264,
+            -0.003258905140683055,
+            0.018518222495913506,
+            0.026746569201350212,
+            -0.0016761035658419132,
+            -0.0025235966313630342,
+            -6.3693878473714e-05,
+            0.015461471863090992,
+            -0.00858378317207098,
+            -0.01022767461836338,
+            0.01666106842458248,
+            0.014404049143195152,
+            -0.016554437577724457,
+            0.011862681247293949,
+            -0.0022125900723040104,
+            0.011658305302262306,
+            -0.012253660708665848,
+            0.016536666080355644,
+            0.020490894094109535,
+            -0.005282669793814421,
+            0.019193550571799278,
+            -0.01095631904900074,
+            0.015390384942293167,
+            0.018731484189629555,
+            0.006948776543140411,
+            0.014155244454741478,
+            0.012662412598729134,
+            -0.01180048007518053,
+            0.0005836928030475974,
+            -0.005758065264672041,
+            -0.005322656128555536,
+            0.005127166397869587,
+            -0.002612455515190959,
+            -0.019406812265515327,
+            0.0036121197044849396,
+            0.0012462479062378407,
+            0.008508252911269665,
+            0.0034899385645985603,
+            -0.00617126002907753,
+            -0.0008730400004424155,
+            0.013462143950164318,
+            -0.02095296047627926,
+            -0.006077957805246115,
+            0.023938624188303947,
+            -0.006015756633132696,
+            -0.005673649255186319,
+            0.0011407278943806887,
+            0.012093714438378811,
+            -0.01092077512294054,
+            -0.01651889458298683,
+            -0.020508665591478348,
+            0.02841711975634098,
+            0.004509596154093742,
+            0.00887701753526926,
+            0.006762172561138868,
+            -0.008610440418124199,
+            -0.002639113226905465,
+            -0.015950197353959084,
+            -0.001235140603967011,
+            0.018233874812722206,
+            0.01723865233361721,
+            0.011027405969798565,
+            -0.019780021160840988,
+            0.008317206054925919,
+            -0.008486038073897362,
+            3.512708644848317e-05,
+            0.014004183933138847,
+            0.01622566021978855,
+            0.0063223205506801605,
+            -0.0027079791761934757,
+            0.016989847645163536,
+            0.001412858604453504,
+            0.029678918421268463,
+            -0.0006458940915763378,
+            -0.01767406240105629,
+            0.0003993102873209864,
+            0.001091300044208765,
+            -0.01620788872241974,
+            -0.003932012245059013,
+            0.0024769455194473267,
+            0.038813624531030655,
+            -0.008139487355947495,
+            0.007148709148168564,
+            -0.003058972302824259,
+            -0.012422492727637291,
+            0.0025724689476191998,
+            -0.0015883552841842175,
+            -0.002055975841358304,
+            0.02177046425640583,
+            -0.017567431554198265,
+            0.012004856020212173,
+            -0.012591325677931309,
+            0.0056780921295285225,
+            0.0012740164529532194,
+            0.007761836517602205,
+            -0.011356184259057045,
+            -0.03067413903772831,
+            -0.013790922239422798,
+            0.02635558880865574,
+            0.018642624840140343,
+            -0.007135380525141954,
+            0.018145015463232994,
+            -0.014821686781942844,
+            -0.00941905751824379,
+            0.008303876966238022,
+            -0.010982976295053959,
+            0.0009352413471788168,
+            0.007233125623315573
+          ],
+          "keyphrases": [
+            "1 Billion Word Language Modeling Benchmark",
+            "Long Short-Term Memory (LSTM) layers",
+            "Mixture of Experts (MoE) models",
+            "computational efficiency",
+            "Machine Translation"
+          ],
+          "title": "5 Experiments 5.1 1 Billion Word Language Modeling Benchmark"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": true,
+      "properties": {
+        "cosine_similarity": 0.8014434674248749
+      }
+    },
+    {
+      "id": "4d338bc1-d825-418a-99c3-96612f243371",
+      "type": "cosine_similarity",
+      "source": {
+        "id": "14339b1f-53f8-479b-9da7-9430b174b9ee",
+        "properties": {
+          "page_content": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer\n\nNoam Shazeer1, Azalia Mirhoseini∗†1, Krzysztof Maziarz∗2, Andy Davis1, Quoc Le1, Geoffrey Hinton1and Jeff Dean1 1Google Brain, {noam,azalia,andydavis,qvl,geoffhinton,jeff}@google.com 2Jagiellonian University, Cracow, krzysztof.maziarz@student.uj.edu.pl\n\nAbstract\n\nThe capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost.\n\n1 Introduction And Related Work 1.1 Conditional Computation\n\nExploiting scale in both training data and model size has been central to the success of deep learning. When datasets are sufficiently large, increasing the capacity (number of parameters) of neural networks can give much better prediction accuracy. This has been shown in domains such as text (Sutskever et al., 2014; Bahdanau et al., 2014; Jozefowicz et al., 2016; Wu et al., 2016), images (Krizhevsky et al., 2012; Le et al., 2012), and audio (Hinton et al., 2012; Amodei et al., 2015). For typical deep learning models, where the entire model is activated for every example, this leads to a roughly quadratic blow-up in training costs, as both the model size and the number of training examples increase. Unfortunately, the advances in computing power and distributed computation fall short of meeting such demand.\n\nVarious forms of conditional computation have been proposed as a way to increase model capacity without a proportional increase in computational costs (Davis & Arel, 2013; Bengio et al., 2013; Eigen et al., 2013; Ludovic Denoyer, 2014; Cho & Bengio, 2014; Bengio et al., 2015; Almahairi et al., 2015). In these schemes, large parts of a network are active or inactive on a per-example basis. The gating decisions may be binary or sparse and continuous, stochastic or deterministic. Various forms of reinforcement learning and back-propagation are proposed for trarining the gating decisions.\n\nWhile these ideas are promising in theory, no work to date has yet demonstrated massive improvements in model capacity, training time, or model quality. We blame this on a combination of the following challenges:\n\nModern computing devices, especially GPUs, are much faster at arithmetic than at branching. Most of the works above recognize this and propose turning on/off large chunks of the network with each gating decision.\n\nLarge batch sizes are critical for performance, as they amortize the costs of parameter transfers and updates. Conditional computation reduces the batch sizes for the conditionally active chunks of the network.\n\nNetwork bandwidth can be a bottleneck. A cluster of GPUs may have computational power thousands of times greater than the aggregate inter-device network bandwidth. To be computationally efficient, the relative computational versus network demands of an algorithm must exceed this ratio. Embedding layers, which can be seen as a form of conditional computation, are handicapped by this very problem. Since the embeddings generally need to be sent across the network, the number of (example, parameter) interactions is limited by network bandwidth instead of computational capacity.\n\nDepending on the scheme, loss terms may be necessary to achieve the desired level of sparsity per-chunk and/or per example. Bengio et al. (2015) use three such terms. These issues can affect both model quality and load-balancing.\n\nModel capacity is most critical for very large data sets. The existing literature on conditional computation deals with relatively small image recognition data sets consisting of up to 600,000 images. It is hard to imagine that the labels of these images provide a sufficient signal to adequately train a model with millions, let alone billions of parameters. In this work, we for the first time address all of the above challenges and finally realize the promise of conditional computation. We obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets.\n\n1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1). All parts of the network are trained jointly by back-propagation.\n\nWhile the introduced technique is generic, in this paper we focus on language modeling and machine translation tasks, which are known to benefit from very large models. In particular, we apply a MoE convolutionally between stacked LSTM layers (Hochreiter & Schmidhuber, 1997), as in Figure 1.\n\nThe MoE is called once for each position in the text, selecting a potentially different combination of experts at each position. The different experts tend to become highly specialized based on syntax and semantics (see Appendix E Table 9). On both language modeling and machine translation benchmarks, we improve on best published results at a fraction of the computational cost.\n\n1.3 Related Work On Mixtures Of Experts\n\nSince its introduction more than two decades ago (Jacobs et al., 1991; Jordan & Jacobs, 1994), the mixture-of-experts approach has been the subject of much research. Different types of expert architectures hae been proposed such as SVMs (Collobert et al., 2002), Gaussian Processes (Tresp, 2001; Theis & Bethge, 2015; Deisenroth & Ng, 2015), Dirichlet Processes (Shahbaba & Neal, 2009), and deep networks. Other work has focused on different expert configurations such as a hierarchical structure (Yao et al., 2009), infinite numbers of experts (Rasmussen & Ghahramani, 2002), and adding experts sequentially (Aljundi et al., 2016). Garmash & Monz (2016) suggest an ensemble model in the format of mixture of experts for machine translation. The gating network is trained on a pre-trained ensemble NMT model. The works above concern top-level mixtures of experts. The mixture of experts is the whole model. Eigen et al. (2013) introduce the idea of using multiple MoEs with their own gating networks as parts of a deep model. It is intuitive that the latter approach is more powerful, since complex problems may contain many sub-problems each requiring different experts. They also allude in their conclusion to the potential to introduce sparsity, turning MoEs into a vehicle for computational computation.\n\nOur work builds on this use of MoEs as a general purpose neural network component. While Eigen et al. (2013) uses two stacked MoEs allowing for two sets of gating decisions, our convolutional application of the MoE allows for different gating decisions at each position in the text. We also realize sparse gating and demonstrate its use as a practical way to massively increase model capacity.\n\n2 The Structure Of The Mixture-Of-Experts Layer\n\nThe Mixture-of-Experts (MoE) layer consists of a set of n \"expert networks\" E1, · · · , En, and a \"gating network\" G whose output is a sparse n-dimensional vector. Figure 1 shows an overview of the MoE module. The experts are themselves neural networks, each with their own parameters.\n\nAlthough in principle we only require that the experts accept the same sized inputs and produce the same-sized outputs, in our initial investigations in this paper, we restrict ourselves to the case where the models are feed-forward networks with identical architectures, but with separate parameters.\n\nLet us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows:\n\n$$y=\\sum_{i=1}^{n}G(x){i}E{i}(x)$$ $$(\\mathbf{l})$$ G(x)iEi(x) (1) We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example. If the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network. In the following we focus on ordinary MoEs. We provide more details on hierarchical MoEs in Appendix B. Our implementation is related to other models of conditional computation. A MoE whose experts are simple weight matrices is similar to the parameterized weight matrix proposed in (Cho & Bengio, 2014). A MoE whose experts have one hidden layer is similar to the block-wise dropout described in (Bengio et al., 2015), where the dropped-out layer is sandwiched between fully-activated layers.\n\n2.1 Gating Network\n\nSoftmax Gating: A simple choice of non-sparse gating function (Jordan & Jacobs, 1994) is to multiply the input by a trainable weight matrix Wg and then apply the Sof tmax function.\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ $\\eqref{eq:walpha}$. Gσ(x) = Sof tmax(x · Wg) (2) Noisy Top-K Gating: We add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to −∞ (which causes the corresponding gate values to equal 0). The sparsity serves to save computation, as described above. While this form of sparsity creates some theoretically scary discontinuities in the output of gating function, we have not yet observed this to be a problem in practice. The noise term helps with load balancing, as will be discussed in Appendix A. The amount of noise per component is controlled by a second trainable weight matrix Wnoise.\n\n$$G(x)=S o f t m a x(K e e p T o p K(H(x),k))$$ $$(4)$$ $$(S)$$ G(x) = Sof tmax(KeepT opK(H(x), k)) (3) $$H(x){i}=(x\\cdot W{g}){i}+S t a n d a r d N o r m a l()\\cdot S o f t p l u s((x\\cdot W{n o i s e}){i})$$ $KeepTopK(v,k){i}=\\begin{cases}v_{i}&\\text{if}v_{i}\\text{is in the top}k\\text{elements of}v.\\ -\\infty&\\text{otherwise.}\\end{cases}$ Training the Gating Network We train the gating network by simple back-propagation, along with the rest of the model. If we choose k > 1, the gate values for the top k experts have nonzero derivatives with respect to the weights of the gating network. This type of occasionally-sensitive behavior is described in (Bengio et al., 2013) with respect to noisy rectifiers. Gradients also backpropagate through the gating network to its inputs. Our method differs here from (Bengio et al., 2015) who use boolean gates and a REINFORCE-style approach to train the gating network.\n\n3 Addressing Performance Challenges 3.1 The Shrinking Batch Problem\n\nOn modern CPUs and GPUs, large batch sizes are necessary for computational efficiency, so as to amortize the overhead of parameter loads and updates. If the gating network chooses k out of n experts for each example, then for a batch of b examples, each expert receives a much smaller batch of approximately kb n b examples. This causes a naive MoE implementation to become very inefficient as the number of experts increases. The solution to this shrinking batch problem is to make the original batch size as large as possible. However, batch size tends to be limited by the memory necessary to store activations between the forwards and backwards passes. We propose the following techniques for increasing the batch size: Mixing Data Parallelism and Model Parallelism: In a conventional distributed training setting, multiple copies of the model on different devices asynchronously process distinct batches of data, and parameters are synchronized through a set of parameter servers. In our technique, these different batches run synchronously so that they can be combined for the MoE layer. We distribute the standard layers of the model and the gating network according to conventional data-parallel schemes, but keep only one shared copy of each expert. Each expert in the MoE layer receives a combined batch consisting of the relevant examples from all of the data-parallel input batches. The same set of devices function as data-parallel replicas (for the standard layers and the gating networks) and as model-parallel shards (each hosting a subset of the experts). If the model is distributed over d devices, and each device processes a batch of size b, each expert receives a batch of approximately kbd nexamples. Thus, we achieve a factor of d improvement in expert batch size.\n\nIn the case of a hierarchical MoE (Section B), the primary gating network employs data parallelism, and the secondary MoEs employ model parallelism. Each secondary MoE resides on one device.\n\nThis technique allows us to increase the number of experts (and hence the number of parameters) by proportionally increasing the number of devices in the training cluster. The total batch size increases, keeping the batch size per expert constant. The memory and bandwidth requirements per device also remain constant, as do the step times, as does the amount of time necessary to process a number of training examples equal to the number of parameters in the model. It is our goal to train a trillionparameter model on a trillion-word corpus. We have not scaled our systems this far as of the writing of this paper, but it should be possible by adding more hardware.\n\nTaking Advantage of Convolutionality: In our language models, we apply the same MoE to each time step of the previous layer. If we wait for the previous layer to finish, we can apply the MoE to all the time steps together as one big batch. Doing so increases the size of the input batch to the MoE layer by a factor of the number of unrolled time steps.\n\nIncreasing Batch Size for a Recurrent MoE: We suspect that even more powerful models may involve applying a MoE recurrently. For example, the weight matrices of a LSTM or other RNN could be replaced by a MoE. Sadly, such models break the convolutional trick from the last paragraph, since the input to the MoE at one timestep depends on the output of the MoE at the previous timestep. Gruslys et al. (2016) describe a technique for drastically reducing the number of stored activations in an unrolled RNN, at the cost of recomputing forward activations. This would allow for a large increase in batch size.\n\n3.2 Network Bandwidth\n\nAnother major performance concern in distributed computing is network bandwidth. Since the experts are stationary (see above) and the number of gating parameters is small, most of the communication involves sending the inputs and outputs of the experts across the network. To maintain computational efficiency, the ratio of an expert's computation to the size of its input and output must exceed the ratio of computational to network capacity of the computing device. For GPUs, this may be thousands to one. In our experiments, we use experts with one hidden layer containing thousands of RELU-activated units. Since the weight matrices in the expert have sizes input_size×hidden_size and hidden_size × output_size, the ratio of computation to input and output is equal to the size of the hidden layer. Conveniently, we can increase computational efficiency simply by using a larger hidden layer, or more hidden layers.\n\n4 Balancing Expert Utilization\n\nWe have observed that the gating network tends to converge to a state where it always produces large weights for the same few experts. This imbalance is self-reinforcing, as the favored experts are trained more rapidly and thus are selected even more by the gating network. Eigen et al. (2013) describe the same phenomenon, and use a hard constraint at the beginning of training to avoid this local minimum. Bengio et al. (2015) include a soft constraint on the batch-wise average of each gate.1 We take a soft constraint approach. We define the importance of an expert relative to a batch of training examples to be the batchwise sum of the gate values for that expert. We define an additional loss Limportance, which is added to the overall loss function for the model. This loss is equal to the square of the coefficient of variation of the set of importance values, multiplied by a hand-tuned scaling factor wimportance. This additional loss encourages all experts to have equal importance.\n\n$$I m p o r t a n c e(X)=\\sum_{x\\in X}G(x)$$ $$L_{importance}(X)=w_{importance}\\cdot CV(Importance(X))^{2}\\tag{7}$$ $$(6)$$\n\nWhile this loss function can ensure equal importance, experts may still receive very different numbers of examples. For example, one expert may receive a few examples with large weights, and another may receive many examples with small weights. This can cause memory and performance problems on distributed hardware. To solve this problem, we introduce a second loss function, Lload , which ensures balanced loads. Appendix A contains the definition of this function, along with experimental results.\n\n5 Experiments 5.1 1 Billion Word Language Modeling Benchmark\n\nDataset: This dataset, introduced by (Chelba et al., 2013) consists of shuffled unique sentences from news articles, totaling approximately 829 million words, with a vocabulary of 793,471 words.\n\nPrevious State-of-the-Art: The best previously published results (Jozefowicz et al., 2016) use models consisting of one or more stacked Long Short-Term Memory (LSTM) layers (Hochreiter & Schmidhuber, 1997; Gers et al., 2000). The number of parameters in the LSTM layers of these models vary from 2 million to 151 million. Quality increases greatly with parameter count, as do computational costs. Results for these models form the top line of Figure 2-right. MoE Models: Our models consist of two stacked LSTM layers with a MoE layer between them (see Figure 1). We vary the sizes of the layers and the number of experts. For full details on model architecture, training regimen, additional baselines and results, see Appendix C.\n\nLow Computation, Varied Capacity: To investigate the effects of adding capacity, we trained\n\na series of MoE models all with roughly equal computational costs: about 8 million multiply-andadds per training example per timestep in the forwards pass, excluding the softmax layer. We call this metric (ops/timestep). We trained models with flat MoEs containing 4, 32, and 256 experts, and models with hierarchical MoEs containing 256, 1024, and 4096 experts. Each expert had about 1 million parameters. For all the MoE layers, 4 experts were active per input.\n\nThe results of these models are shown in Figure 2-left. The model with 4 always-active experts performed (unsurprisingly) similarly to the computationally-matched baseline models, while the largest of the models (4096 experts) achieved an impressive 24% lower perplexity on the test set. Figure 2: Model comparison on 1-Billion-Word Language-Modeling Benchmark. On the left, we plot test perplexity as a function of model capacity for models with similar computational budgets of approximately 8-million-ops-per-timestep. On the right, we plot test perplexity as a function of computational budget. The top line represents the LSTM models from (Jozefowicz et al., 2016).\n\nThe bottom line represents 4-billion parameter MoE models with different computational budgets.\n\nVaried Computation, High Capacity: In addition to the largest model from the previous section, we trained two more MoE models with similarly high capacity (4 billion parameters), but higher computation budgets. These models had larger LSTMs, and fewer but larger and experts. Details\n\nTest Test #Parameters ops/timestep Training TFLOPS Perplexity Perplexity excluding embedding Time /GPU 10 epochs 100 epochs and softmax layers 10 epochs Best Published Results 34.7 30.6 151 million 151 million 59 hours, 32 k40s 1.09 Low-Budget MoE Model 34.1 4303 million 8.9 million 15 hours, 16 k40s 0.74 Medium-Budget MoE Model 31.3 4313 million 33.8 million 17 hours, 32 k40s 1.22 High-Budget MoE Model 28.0 4371 million 142.7 million 47 hours, 32 k40s 1.56\n\nTable 1: Summary of high-capacity MoE-augmented models with varying computational budgets, vs. best previously published results (Jozefowicz et al., 2016). Details in Appendix C. can be found in Appendix C.2. Results of these three models form the bottom line of Figure 2-right.\n\nTable 1 compares the results of these models to the best previously-published result on this dataset .\n\nEven the fastest of these models beats the best published result (when controlling for the number of training epochs), despite requiring only 6% of the computation.\n\nComputational Efficiency: We trained our models using TensorFlow (Abadi et al., 2016) on clusters containing 16-32 Tesla K40 GPUs. For each of our models, we determine computational efficiency in TFLOPS/GPU by dividing the number of floating point operations required to process one training batch by the observed step time and the number of GPUs in the cluster. The operation counts used here are higher than the ones we report in our ops/timestep numbers in that we include the backwards pass, we include the importance-sampling-based training of the softmax layer, and we count a multiply-and-add as two separate operations. For all of our MoE models, the floating point operations involved in the experts represent between 37% and 46% of the total.\n\nFor our baseline models wtih no MoE, observed computational efficiency ranged from 1.07-1.29 TFLOPS/GPU. For our low-computation MoE models, computation efficiency ranged from 0.740.90 TFLOPS/GPU, except for the 4-expert model which did not make full use of the available parallelism. Our highest-computation MoE model was more efficient at 1.56 TFLOPS/GPU, likely due to the larger matrices. These numbers represent a significant fraction of the theoretical maximum of 4.29 TFLOPS/GPU claimed by NVIDIA. Detailed results are in Appendix C, Table 7.\n\n5.2 100 BILLION WORD GOOGLE NEWS CORPUS\n\nOn the 1-billion-word corpus, adding additional capacity seems to produce diminishing returns as the number of parameters in the MoE layer exceeds 1 billion, as can be seen in Figure 2-left. We hypothesized that for a larger training set, even higher capacities would produce significant quality improvements. We constructed a similar training set consisting of shuffled unique sentences from Google's internal news corpus, totalling roughly 100 billion words. Similarly to the previous section, we tested a series of models with similar computational costs of about 8 million ops/timestep. In addition to a baseline LSTM model, we trained models augmented with MoE layers containing 32, 256, 1024, 4096, 16384, 65536, and 131072 experts. This corresponds to up to 137 billion parameters in the MoE layer. Details on architecture, training, and results are given in Appendix D.\n\nResults: Figure 3 shows test perplexity as a function of capacity after training on 10 billion words (top line) and 100 billion words (bottom line). When training over the full 100 billion words, test perplexity improves significantly up to 65536 experts (68 billion parameters), dropping 39% lower than the computationally matched baseline, but degrades at 131072 experts, possibly a result of too much sparsity. The widening gap between the two lines demonstrates (unsurprisingly) that increased model capacity helps more on larger training sets. Even at 65536 experts (99.994% layer sparsity), computational efficiency for the model stays at a respectable 0.72 TFLOPS/GPU.\n\n5.3 Machine Translation (Single Language Pair)\n\nModel Architecture: Our model was a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decreased the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We inserted MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). Each MoE layer contained up to 2048 experts each with about two million parameters, adding a total of about 8 billion parameters to the models. Further details on model architecture, testing procedure and results can be found in Appendix E.\n\nDatasets: We benchmarked our method on the WMT'14 En→Fr and En→De corpora, whose training sets have 36M sentence pairs and 5M sentence pairs, respectively. The experimental protocols were also similar to those in (Wu et al., 2016): newstest2014 was used as the test set to compare against previous work (Luong et al., 2015a; Zhou et al., 2016; Wu et al., 2016), while the combination of newstest2012 and newstest2013 was used as the development set. We also tested the same model on a Google's Production English to French data.\n\nTable 2: Results on WMT'14 En→ Fr newstest2014 (bold values represent best results). Model Test Test ops/timenstep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.69 40.35 85M 8.7B 3 days/64 k40s MoE with 2048 Experts (longer training) 2.63 40.56 85M 8.7B 6 days/64 k40s GNMT (Wu et al., 2016) 2.79 39.22 214M 278M 6 days/96 k80s GNMT+RL (Wu et al., 2016) 2.96 39.92 214M 278M 6 days/96 k80s PBMT (Durrani et al., 2014) 37.0 LSTM (6-layer) (Luong et al., 2015b) 31.5 LSTM (6-layer+PosUnk) (Luong et al., 2015b) 33.1 DeepAtt (Zhou et al., 2016) 37.7 DeepAtt+PosUnk (Zhou et al., 2016) 39.2\n\nTable 3: Results on WMT'14 En → De newstest2014 (bold values represent best results). Model Test Test ops/timestep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 4.64 26.03 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 5.25 24.91 214M 278M 1 day/96 k80s GNMT +RL (Wu et al., 2016) 8.08 24.66 214M 278M 1 day/96 k80s PBMT (Durrani et al., 2014) 20.7 DeepAtt (Zhou et al., 2016) 20.6\n\nTable 4: Results on the Google Production En→ Fr dataset (bold values represent best results). Model Eval Eval Test Test ops/timestep Total Training Perplexity BLEU Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.60 37.27 2.69 36.57 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 2.78 35.80 2.87 35.56 214M 278M 6 days/96 k80s\n\nResults: Tables 2, 3, and 4 show the results of our largest models, compared with published results. Our approach achieved BLEU scores of 40.56 and 26.03 on the WMT'14 En→Fr and En→De benchmarks. As our models did not use RL refinement, these results constitute significant gains of 1.34 and 1.12 BLEU score on top of the strong baselines in (Wu et al., 2016). The perplexity scores are also better.2 On the Google Production dataset, our model achieved 1.01 higher test BLEU score even after training for only one sixth of the time.\n\n5.4 Multilingual Machine Translation\n\nDataset: (Johnson et al., 2016) train a single GNMT (Wu et al., 2016) model on a very large combined dataset of twelve language pairs. Results are somewhat worse than those for 12 separately trained single-pair GNMT models. This is not surprising, given that the twelve models have 12 times the capacity and twelve times the aggregate training of the one model. We repeat this experiment with a single MoE-augmented model. See Appendix E for details on model architecture.\n\nWe train our model on the same dataset as (Johnson et al., 2016) and process the same number of training examples (about 3 billion sentence pairs). Our training time was shorter due to the lower computational budget of our model.\n\nResults: Results for the single-pair GNMT models, the multilingual GNMT model and the multilingual MoE model are given in Table 5. The MoE model achieves 19% lower perplexity on the dev set than the multilingual GNMT model. On BLEU score, the MoE model significantly beats the multilingual GNMT model on 11 of the 12 language pairs (by as much as 5.84 points), and even beats the monolingual GNMT models on 8 of 12 language pairs. The poor performance on English → Korean seems to be a result of severe overtraining, as for the rarer language pairs a small number of real examples were highly oversampled in the training corpus.\n\nTable 5: Multilingual Machine Translation (bold values represent best results). GNMT-Mono GNMT-Multi MoE-Multi MoE-Multi vs. GNMT-Multi Parameters 278M / model 278M 8.7B ops/timestep 212M 212M 102M training time, hardware various 21 days, 96 k20s 12 days, 64 k40s Perplexity (dev) 4.14 3.35 -19% French → English Test BLEU 36.47 34.40 37.46 +3.06 German → English Test BLEU 31.77 31.17 34.80 +3.63 Japanese → English Test BLEU 23.41 21.62 25.91 +4.29 Korean → English Test BLEU 25.42 22.87 28.71 +5.84 Portuguese → English Test BLEU 44.40 42.53 46.13 +3.60 Spanish → English Test BLEU 38.00 36.04 39.39 +3.35 English → French Test BLEU 35.37 34.00 36.59 +2.59 English → German Test BLEU 26.43 23.15 24.53 +1.38 English → Japanese Test BLEU 23.66 21.10 22.78 +1.68 English → Korean Test BLEU 19.75 18.41 16.62 -1.79 English → Portuguese Test BLEU 38.40 37.35 37.90 +0.55 English → Spanish Test BLEU 34.50 34.25 36.21 +1.96\n\n6 Conclusion\n\nThis work is the first to demonstrate major wins from conditional computation in deep networks.\n\nWe carefully identified the design considerations and challenges of conditional computing and addressed them with a combination of algorithmic and engineering solutions. While we focused on text, conditional computation may help in other domains as well, provided sufficiently large training sets. We look forward to seeing many novel implementations and applications of conditional computation in the years to come.\n\nAcknowledgments\n\nWe would like to thank all of the members of the Google Brain and Google Translate teams who helped us with this project, in particular Zhifeng Chen, Yonghui Wu, and Melvin Johnson. Thanks also to our anonymous ICLR reviewers for the helpful suggestions on making this paper better.\n\n2Reported perplexities relative to the tokenization used by both our models and GNMT.\n\nReferences\n\nMartín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Gordon Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda B. Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. CoRR, abs/1603.04467, 2016. URL http://arxiv.org/abs/1603.04467.\n\nRahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. CoRR, abs/1611.06194, 2016. URL http://arxiv.org/abs/1611.\n\n06194.\n\nA. Almahairi, N. Ballas, T. Cooijmans, Y. Zheng, H. Larochelle, and A. Courville. Dynamic Capacity Networks. ArXiv e-prints, November 2015.\n\nDario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan, Christopher Fougner, Tony Han, Awni Y. Hannun, Billy Jun, Patrick LeGresley, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595, 2015. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.\n\nEmmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297, 2015.\n\nYoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.\n\nCiprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling.\n\narXiv preprint arXiv:1312.3005, 2013.\n\nK. Cho and Y. Bengio. Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning. ArXiv e-prints, June 2014. Ronan Collobert, Samy Bengio, and Yoshua Bengio. A parallel mixture of SVMs for very large scale problems. Neural Computing, 2002.\n\nAndrew Davis and Itamar Arel. Low-rank approximations for conditional feedforward computation in deep neural networks. arXiv preprint arXiv:1312.4461, 2013. Marc Peter Deisenroth and Jun Wei Ng. Distributed Gaussian processes. In ICML, 2015.\n\nJohn Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization, 2010.\n\nNadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. Edinburgh's phrase-based machine translation systems for wmt-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation, 2014.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013. Ekaterina Garmash and Christof Monz. Ensemble learning for multi-source neural machine translation. In staff.science.uva.nl/c.monz, 2016.\n\nFelix A. Gers, Jürgen A. Schmidhuber, and Fred A. Cummins. Learning to forget: Continual prediction with lstm. Neural Computation, 2000.\n\nAudrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. Memory-efficient backpropagation through time. CoRR, abs/1606.03401, 2016. URL http://arxiv.org/ abs/1606.03401. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2015.\n\nGeoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 2012. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.\n\nSergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.\n\nRobert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computing, 1991.\n\nMelvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation.\n\nCoRR, abs/1611.04558, 2016. URL http://arxiv.org/abs/1611.04558.\n\nMichael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm.\n\nNeural Computing, 1994. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.\n\nDiederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.\n\nReinhard Kneser and Hermann. Ney. Improved backingoff for m-gram language modeling., 1995.\n\nAlex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.\n\nQuoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeffrey Dean, and Andrew Y. Ng. Building high-level features using large scale unsupervised learning. In ICML, 2012. Patrick Gallinari Ludovic Denoyer. Deep sequential neural network. arXiv preprint arXiv:1410.0510, 2014.\n\nMinh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attentionbased neural machine translation. EMNLP, 2015a.\n\nMinh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the rare word problem in neural machine translation. ACL, 2015b.\n\nCarl Edward Rasmussen and Zoubin Ghahramani. Infinite mixtures of Gaussian process experts.\n\nNIPS, 2002.\n\nHasim Sak, Andrew W Senior, and Françoise Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH, pp. 338–342, 2014.\n\nMike Schuster and Kaisuke Nakajima. Japanese and Korean voice search. ICASSP, 2012.\n\nBabak Shahbaba and Radford Neal. Nonlinear models using dirichlet process mixtures. JMLR, 2009.\n\nIlya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks.\n\nIn NIPS, 2014.\n\nLucas Theis and Matthias Bethge. Generative image modeling using spatial LSTMs. In NIPS, 2015. Volker Tresp. Mixtures of Gaussian Processes. In NIPS, 2001.\n\nYonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.\n\nBangpeng Yao, Dirk Walther, Diane Beck, and Li Fei-fei. Hierarchical mixture of classification experts uncovers interactions between brain regions. In NIPS. 2009.\n\nWojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization.\n\narXiv preprint arXiv:1409.2329, 2014.\n\nJie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with fast-forward connections for neural machine translation. arXiv preprint arXiv:1606.04199, 2016.\n\nAppendices A Load-Balancing Loss\n\nAs discussed in section 4, for load-balancing purposes, we want to define an additional loss function to encourage experts to receive roughly equal numbers of training examples. Unfortunately, the number of examples received by an expert is a discrete quantity, so it can not be used in backpropagation. Instead, we define a smooth estimator Load(X) of the number of examples assigned to each expert for a batch X of inputs. The smoothness allows us to back-propagate gradients through the estimator. This is the purpose of the noise term in the gating function. We define P(x, i) as the probability that G(x)iis nonzero, given a new random choice of noise on element i, but keeping the already-sampled choices of noise on the other elements. To compute P(x, i), we note that the G(x)iis nonzero if and only if H(x)iis greater than the k th-greatest element of H(x) excluding itself. The probability works out to be:\n\n$$P(x,i)=P r\\Big((x\\cdot W_{g}){i}+S t a n d x N o r a l()\\cdot S o f t p u s((x\\cdot W{n o i s e})_{i})$$ $$>k t h_e x c l u d i n(H(x),k,i)\\Big)$$ $$({\\boldsymbol{8}})$$\n\n$$(9)$$ $$(10)$$ (8) Where kth_excluding(v, k, i) means the kth highest component of v, excluding component i. Simplifying, we get:\n\n$$P(x,i)=\\Phi{\\Big(}{\\frac{(x\\cdot W_{g}){i}-k t h_c x c l u d i n g(H(x),k,i)}{S o f t p l u s((x\\cdot W{n o i s e})_{i})}}{\\Big)}$$\n\nWhere Φ is the CDF of the standard normal distribution.\n\n$$L o a d(X){i}=\\sum{x\\in X}P(x,i)$$\n\n$$(11)$$ P(x, i) (10) We can now define the load loss to be the square of the coefficient of variation of the load vector, multiplied by a hand-tuned scaling factor wload.\n\n$$L_{l o a d}(X)=w_{l o a d}\\cdot C V(L o a d(X))^{2}$$ Lload(X) = wload · CV (Load(X))2(11) Initial Load Imbalance: To avoid out-of-memory errors, we need to initialize the network in a state of approximately equal expert load (since the soft constraints need some time to work). To accomplish this, we initialize the matrices Wg and Wnoise to all zeros, which yields no signal and some noise.\n\nExperiments: We trained a set of models with identical architecture (the MoE-256 model described in Appendix C), using different values of wimportance and wload. We trained each model for 10 epochs, then measured perplexity on the test set. We also measured the coefficients of variation in Importance and Load, as well as ratio of the load on the most overloaded expert to the average load. This last value is significant for load balancing purposes on distributed hardware. All of these metrics were averaged over several training batches.\n\nwimportance wload Test Perplexity CV (Importance(X)) CV (Load(X)) max(Load(X)) mean(Load(X)) 0.0 0.0 39.8 3.04 3.01 17.80 0.2 0.0 35.6 0.06 0.17 1.47 0.0 0.2 35.7 0.22 0.04 1.15 0.1 0.1 35.6 0.06 0.05 1.14 0.01 0.01 35.7 0.48 0.11 1.37 1.0 1.0 35.7 0.03 0.02 1.07\n\nTable 6: Experiments with different combinations of losses.\n\n13 Results: Results are reported in Table 6. All the combinations containing at least one the two losses led to very similar model quality, where having no loss was much worse. Models with higher values of wload had lower loads on the most overloaded expert.\n\nB Hierachical Mixture Of Experts\n\nIf the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network.3If the hierarchical MoE consists of a groups of b experts each, we denote the primary gating network by Gprimary, the secondary gating networks by (G1, G2..Ga), and the expert networks by (E0,0, E0,1..Ea,b). The output of the MoE is given by:\n\n$$y_{H}=\\sum_{i=1}^{a}\\sum_{j=1}^{b}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}\\cdot E{i,j}(x)$$ $$(12)$$ $$(13)$$ $$(14)$$\n\nOur metrics of expert utilization change to the following:\n\n$$I m p o r t a n c e_{H}(X){i,j}=\\sum{x\\in X}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}$$ $$L o a d{H}(X){i,j}={\\frac{L o a d{p r i m a r y}(X){i}\\cdot L o a d{i}(X^{(i)})_{j}}{|X^{(i)}|}}$$\n\nLoadprimary and Loadi deonte the Load functions for the primary gating network and i th secondary gating network respectively. X(i) denotes the subset of X for which Gprimary(x)i > 0.\n\nIt would seem simpler to let LoadH(X)i,j = Loadi(Xi)j , but this would not have a gradient with respect to the primary gating network, so we use the formulation above.\n\nC 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS C.1 8-MILLION-OPERATIONS-PER-TIMESTEP MODELS Model Architecture: Our model consists of five layers: a word embedding layer, a recurrent Long Short-Term Memory (LSTM) layer (Hochreiter & Schmidhuber, 1997; Gers et al., 2000), a MoE layer, a second LSTM layer, and a softmax layer. The dimensionality of the embedding layer, the number of units in each LSTM layer, and the input and output dimensionality of the MoE layer are all equal to 512. For every layer other than the softmax, we apply drouput (Zaremba et al., 2014) to the layer output, dropping each activation with probability DropP rob, otherwise dividing by (1 − DropP rob). After dropout, the output of the previous layer is added to the layer output.\n\nThis residual connection encourages gradient flow (He et al., 2015).\n\nMoE Layer Architecture: Each expert in the MoE layer is a feed forward network with one ReLU-activated hidden layer of size 1024 and an output layer of size 512. Thus, each expert contains [512 ∗ 1024] + [1024 ∗ 512] = 1M parameters. The output of the MoE layer is passed through a sigmoid function before dropout. We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts.\n\nWe call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers. Thus, each example is processed by exactly 4 experts for a total of 4M ops/timestep. The two LSTM layers contribute 2M ops/timestep each for the desired total of 8M.\n\n3 We have not found the need for deeper hierarchies.\n\nComputationally-Matched Baselines: The MoE-4 model does not employ sparsity, since all 4 experts are always used. In addition, we trained four more computationally-matched baseline models with no sparsity:\n\nMoE-1-Wide: The MoE layer consists of a single \"expert\" containing one ReLU-activated hidden layer of size 4096.\n\nMoE-1-Deep: The MoE layer consists of a single \"expert\" containing four ReLU-activated hidden layers, each with size 1024.\n\n4xLSTM-512: We replace the MoE layer with two additional 512-unit LSTM layers.\n\nLSTM-2048-512: The model contains one 2048-unit LSTM layer (and no MoE). The output of the LSTM is projected down to 512 dimensions (Sak et al., 2014). The next timestep of the LSTM receives the projected output. This is identical to one of the models published in (Jozefowicz et al., 2016). We re-ran it to account for differences in training regimen, and obtained results very similar to the published ones. Training: The models were trained on a cluster of 16 K40 GPUs using the synchronous method described in Section 3. Each batch consisted of a set of sentences totaling roughly 300,000 words. In the interest of time, we limited training to 10 epochs, (27,000 steps). Training took 12-16 hours for all models, except for MoE-4, which took 18 hours (since all the expert computation was performed on only 4 of 16 GPUs). We used the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 1000 training steps, and decreased after that so as to be proportional to the inverse square root of the step number. The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016). For each model, we performed a hyper-parmeter search to find the best dropout probability, in increments of 0.1.\n\nTo ensure balanced expert utilization we set wimportance = 0.1 and wload = 0.1, as described in Section 4 and Appendix A.\n\nResults: We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol. Results are reported in Table 7. For each model, we report the test perplexity, the computational budget, the parameter counts, the value of DropP rob, and the computational efficiency.\n\nModel Test Test ops/timestep #Params excluding Total Drop- TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params P rob per GPU 10 epochs (final) (millions) (billions) (observed) Kneser-Ney 5-gram* 67.6 0.00001 1.8 LSTM-512-512* 54.1 2.4 2.4 0.8 0.1 LSTM-1024-512* 48.2 4.7 4.7 0.8 0.1 LSTM-2048-512* 45.0 43.7 9.4 9.4 0.8 0.1 0.61 LSTM-2048-512 44.7 9.4 9.4 0.8 0.1 1.21 4xLSTM-512 46.0 8.4 8.4 0.8 0.1 1.07 MoE-1-Wide 46.1 8.4 8.4 0.8 0.1 1.29 MoE-1-Deep 45.7 8.4 8.4 0.8 0.1 1.29 MoE-4 45.0 8.4 8.4 0.8 0.1 0.52 MoE-32 39.7 8.4 37.8 0.9 0.1 0.87 MoE-256 35.7 8.6 272.9 1.1 0.1 0.81 MoE-256-h 36.0 8.4 272.9 1.1 0.1 0.89 MoE-1024-h 34.6 8.5 1079.0 1.9 0.2 0.90 MoE-4096-h 34.1 8.9 4303.4 5.1 0.2 0.74 2xLSTM-8192-1024* 34.7 30.6 151.0 151.0 1.8 0.25 1.09 MoE-34M 31.3 33.8 4313.9 6.0 0.3 1.22 MoE-143M 28.0 142.7 4371.1 6.0 0.4 1.56\n\nC.2 More Expensive Models\n\nWe ran two additional models (MoE-34M and MoE-143M) to investigate the effects of adding more computation in the presence of a large MoE layer. These models have computation budgets of 34M and 143M ops/timestep. Similar to the models above, these models use a MoE layer between two LSTM layers. The dimensionality of the embedding layer, and the input and output dimensionality of the MoE layer are set to 1024 instead of 512. For MoE-34M, the LSTM layers have 1024 units.\n\nFor MoE-143M, the LSTM layers have 4096 units and an output projection of size 1024 (Sak et al., 2014). MoE-34M uses a hierarchical MoE layer with 1024 experts, each with a hidden layer of size 2048. MoE-143M uses a hierarchical MoE layer with 256 experts, each with a hidden layer of size 8192. Both models have 4B parameters in the MoE layers. We searched for the best DropP rob for each model, and trained each model for 10 epochs.\n\nThe two models achieved test perplexity of 31.3 and 28.0 respectively, showing that even in the presence of a large MoE, more computation is still useful. Results are reported at the bottom of Table 7. The larger of the two models has a similar computational budget to the best published model from the literature, and training times are similar. Comparing after 10 epochs, our model has a lower test perplexity by 18%.\n\nD 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS Model Architecture: The models are similar in structure to the 8-million-operations-per-timestep models described in the previous section. We vary the number of experts between models, using an ordinary MoE layer with 32 experts and hierarchical MoE layers with 256, 1024, 4096, 16384, 65536 and 131072 experts. For the hierarchical MoE layers, the first level branching factors are 32, 32, 64, 128, 256 and 256, respectively.\n\nTraining: Models are trained on a cluster of 32 Tesla K40 GPUs, except for the last two models, which are trained on clusters of 64 and 128 GPUs so as to have enough memory for all the parameters. For all models, training batch sizes are approximately 2.5 million words. Models are trained once-through over about 100 billion words. We implement several memory optimizations in order to fit up to 1 billion parameters per GPU. First, we do not store the activations of the hidden layers of the experts, but instead recompute them on the backwards pass. Secondly, we modify the optimizer on the expert parameters to require less auxiliary storage: The Adam optimizer (Kingma & Ba, 2015) keeps first and second moment estimates of the perparameter gradients. This triples the required memory. To avoid keeping a first-moment estimator, we set β1 = 0. To reduce the size of the second moment estimator, we replace it with a factored approximation. For a matrix of parameters, instead of maintaining a full matrix of second-moment estimators, we maintain vectors of row-wise and column-wise averages of that matrix. At each step, the matrix of estimators is taken to be the outer product of those two vectors divided by the mean of either one. This technique could similarly be applied to Adagrad (Duchi et al., 2010).\n\nModel Test Test ops/timestep #Params excluding Total TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params per GPU .1 epochs 1 epoch (millions) (billions) (observed) Kneser-Ney 5-gram 67.1 45.3 0.00001 76.0 4xLSTM-512 54.5 47.0 8.4 8.4 0.1 1.23 MoE-32 48.5 40.4 8.4 37.8 0.1 0.83 MoE-256-h 42.8 35.3 8.4 272.9 0.4 1.11 MoE-1024-h 40.3 32.7 8.5 1079.0 1.2 1.14 MoE-4096-h 38.9 30.9 8.6 4303.4 4.4 1.07 MoE-16384-h 38.2 29.7 8.8 17201.0 17.3 0.96 MoE-65536-h 38.2 28.9 9.2 68791.0 68.9 0.72 MoE-131072-h 39.8 29.2 9.7 137577.6 137.7 0.30\n\nResults: We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8. Perplexity after 100 billion training words is 39% lower for the 68-billion-parameter MoE model than for the baseline model. It is notable that the measured computational efficiency of the largest model (0.30 TFLOPS/GPU) is very low compared to the other models. This is likely a result of the fact that, for purposes of comparison to the other models, we did not increase the training batch size proportionally to the number of GPUs. For comparison, we include results for a computationally matched baseline model consisting of 4 LSTMs, and for an unpruned 5-gram model with Kneser-Ney smoothing (Kneser & Ney, 1995).4\n\nE Machine Translation - Experimental Details\n\nModel Architecture for Single Language Pair MoE Models: Our model is a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decrease the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We insert MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). We use an attention mechanism between the encoder and decoder, with the first decoder LSTM receiving output from and providing input for the attention 5. All of the layers in our model have input and output dimensionality of 512. Our LSTM layers have 2048 hidden units, with a 512-dimensional output projection. We add residual connections around all LSTM and MoE layers to encourage gradient flow (He et al., 2015). Similar to GNMT, to effectively deal with rare words, we used subword units (also known as \"wordpieces\") (Schuster & Nakajima, 2012) for inputs and outputs in our system.\n\nWe use a shared source and target vocabulary of 32K wordpieces. We also used the same beam search technique as proposed in (Wu et al., 2016).\n\nWe train models with different numbers of experts in the MoE layers. In addition to a baseline model with no MoE layers, we train models with flat MoE layers containing 32 experts, and models with hierarchical MoE layers containing 512 and 2048 experts. The flat MoE layers use k = 4 and the hierarchical MoE models use k = 2 at each level of the gating network. Thus, each input is processed by exactly 4 experts in each MoE layer. Each expert in the MoE layer is a feed forward network with one hidden layer of size 2048 and ReLU activation. Thus, each expert contains [512 ∗ 2048] + [2048 ∗ 512] = 2M parameters. The output of the MoE layer is passed through a sigmoid function. We use the strictly-balanced gating function described in Appendix F.\n\nModel Architecture for Multilingual MoE Model: We used the same model architecture as for the single-language-pair models, with the following exceptions: We used noisy-top-k gating as described in Section 2.1, not the scheme from Appendix F. The MoE layers in the encoder and decoder are non-hierarchical MoEs with n = 512 experts, and k = 2. Each expert has a larger hidden layer of size 8192. This doubles the amount of computation in the MoE layers, raising the computational budget of the entire model from 85M to 102M ops/timestep. Training: We trained our networks using the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 2000 training steps, held constant for an additional 8000 steps, and decreased after that so as to be proportional to the inverse square root of the step number. For the single-language-pair models, similarly to (Wu et al., 2016), we applied dropout (Zaremba et al., 2014) to the output of all embedding, LSTM and MoE layers, using DropP rob = 0.4. Training was done synchronously on a cluster of up to 64 GPUs as described in section 3. Each training batch consisted of a set of sentence pairs containing roughly 16000 words per GPU.\n\nTo ensure balanced expert utilization we set wimportance = 0.01 and wload = 0.01, as described in Section 4 and Appendix A. Metrics: We evaluated our models using the perplexity and the standard BLEU score metric. We reported tokenized BLEU score as computed by the multi-bleu.pl script, downloaded from the public implementation of Moses (on Github), which was also used in (Luong et al., 2015a). Results: Tables 2, 3 and 4 in Section 5.3 show comparisons of our results to other published methods. Figure 4 shows test perplexity as a function of number of words in the (training data's) source sentences processed for models with different numbers of experts. As can be seen from the Figure, as we increased the number of experts to approach 2048, the test perplexity of our model continued to improve.\n\nWe found that the experts indeed become highly specialized by syntax and/or semantics, as can be seen in Table 9. For example, one expert is used when the indefinite article \"a\" introduces the direct object in a verb phrase indicating importance or leadership.\n\nExpert 381 Expert 752 Expert 2004 ... with researchers , ... ... plays a core ... ... with rapidly growing ... ... to innovation . ... plays a critical ... ... under static conditions ... ... tics researchers . ... provides a legislative ... ... to swift ly ... ... the generation of ... ... play a leading ... ... to dras tically ... ... technology innovations is ... ... assume a leadership ... ... the rapid and ... ... technological innovations , ... ... plays a central ... ... the fast est ... ... support innovation throughout ... ... taken a leading ... ... the Quick Method ... ... role innovation will ... ... established a reconciliation ... ... rec urrent ) ... ... research scienti st ... ... played a vital ... ... provides quick access ... ... promoting innovation where ... ... have a central ... ... of volatile organic ... ... ... ...\n\nDue to some peculiarities in our infrastructure which have since been fixed, at the time we ran some of the machine translation experiments, our models ran faster if every expert received exactly the same batch size. To accommodate this, we used a different gating function which we describe below. Recall that we define the softmax gating function to be:\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ Gσ(x) = Sof tmax(x · Wg) (15) Sparse Gating (alternate formulation): To obtain a sparse gating vector, we multiply Gσ(x) component-wise with a sparse mask M(Gσ(x)) and normalize the output. The mask itself is a function of Gσ(x) and specifies which experts are assigned to each input example:\n\n$$(15)$$ $$G(x){i}=\\frac{G{\\sigma}(x){i}M(G{\\sigma}(x)){i}}{\\sum{j=1}^{n}G_{\\sigma}(x){j}M(G{\\sigma}(x))_{j}}\\tag{1}$$ $$(17)$$\n\n$$(16)$$\n\nTop-K Mask: To implement top-k gating in this formulation, we would let M(v) = T opK(v, k), where:\n\n$$T o p K(v,k){i}=\\begin{cases}1&{\\mathrm{if~}}v{i}{\\mathrm{~is~in~the~top~}}k{\\mathrm{~elements~of~}}v.\\ 0&{\\mathrm{otherwise.}}\\end{cases}$$ 0 otherwise. (17) Batchwise Mask: To force each expert to receive the exact same number of examples, we introduce an alternative mask function, Mbatchwise(X, m), which operates over batches of input vectors.\n\nInstead of keeping the top k values per example, we keep the top m values per expert across the training batch, where m = k|X| n, so that each example is sent to an average of k experts.\n\n$$M_{b a t c h w i s e}(X,m){j,i}=\\begin{cases}1&\\text{if}X{j,i}\\text{is in the top}m\\text{values for to expert}i\\ 0&\\text{otherwise}\\end{cases}$$\n\nAs our experiments suggest and also observed in (Ioffe & Szegedy, 2015), using a batchwise function during training (such as Mbatchwise) requires modifications to the inference when we may not have a large batch of examples. Our solution to this is to train a vector T of per-expert threshold values to approximate the effects of the batchwise mask. We use the following mask at inference time:\n\n$$(18)$$ $$M_{threshold}(x,T){i}=\\begin{cases}1&\\text{if}x{i}>T_{i}\\ 0&\\text{otherwise}\\end{cases}\\tag{1}$$ $$(19)$$\n\nTo learn the threshold values, we apply an additional loss at training time which is minimized when the batchwise mask and the threshold mask are identical.\n\n$$L_{batchwise}(X,T,m)=\\sum_{j=1}^{|X|}\\sum_{i=1}^{n}(M_{threshold}(x,T){i}-M{batchwise}(X,m){j,i})(X{j,i}-T_{i})\\tag{20}$$\n\nG ATTENTION FUNCTION The attention mechanism described in GNMT (Wu et al., 2016) involves a learned \"Attention Function\" A(xi, yj ) which takes a \"source vector\" xi and a \"target vector\" yj , and must be computed for every source time step i and target time step j. In GNMT, the attention function is implemented as a feed forward neural network with a hidden layer of size n. It can be expressed as:\n\n$$A_{G N M T}(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}t a n h((x_{i}U){d}+(y{j}W)_{d})$$\n\nWhere U and W are trainable weight matrices and V is a trainable weight vector.\n\nFor performance reasons, in our models, we used a slightly different attention function:\n\n$$A(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}tanh((x_{i}U){d})tanh((y{j}W)_{d})\\tag{22}$$\n\nWith our attention function, we can simultaneously compute the attention function on multiple source time steps and multiple target time steps using optimized matrix multiplications. We found little difference in quality between the two functions.\n\n$$(21)$$",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/1701.06538v1.md"
+          },
+          "headlines": {
+            "1 Introduction And Related Work": [
+              "1.1 Conditional Computation",
+              "1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER",
+              "1.3 Related Work On Mixtures Of Experts"
+            ],
+            "2 The Structure Of The Mixture-Of-Experts Layer": [
+              "2.1 Gating Network"
+            ],
+            "3 Addressing Performance Challenges": [
+              "3.1 The Shrinking Batch Problem",
+              "3.2 Network Bandwidth"
+            ],
+            "4 Balancing Expert Utilization": [],
+            "5 Experiments": [
+              "5.1 1 Billion Word Language Modeling Benchmark",
+              "5.2 100 BILLION WORD GOOGLE NEWS CORPUS",
+              "5.3 Machine Translation (Single Language Pair)",
+              "5.4 Multilingual Machine Translation"
+            ],
+            "6 Conclusion": [],
+            "Appendices": [
+              "A Load-Balancing Loss",
+              "B Hierachical Mixture Of Experts",
+              "C 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS",
+              "D 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS",
+              "E Machine Translation - Experimental Details"
+            ]
+          },
+          "summary": "The paper introduces a Sparsely-Gated Mixture-of-Experts (MoE) layer to enhance neural network capacity without a proportional increase in computation. This approach uses a trainable gating network to select a sparse combination of experts for each input, significantly improving model capacity and performance in language modeling and machine translation tasks. The MoE layer, with up to 137 billion parameters, achieves better results than state-of-the-art models at lower computational costs. The paper addresses challenges like batch size and network bandwidth, proposing solutions like mixing data and model parallelism. Experiments show that MoE models outperform traditional models in various benchmarks, demonstrating the potential of conditional computation in deep learning.",
+          "summary_embedding": [
+            -0.01644345559179783,
+            -0.016995344310998917,
+            -0.016372699290513992,
+            -0.014349107630550861,
+            0.02282555401325226,
+            0.045934125781059265,
+            -0.014646278694272041,
+            0.03563220426440239,
+            -0.004829026758670807,
+            0.005617944523692131,
+            -0.00932550523430109,
+            -0.008143897168338299,
+            0.005575491581112146,
+            0.01538213063031435,
+            -0.005653322208672762,
+            -0.020306676626205444,
+            -0.013832597061991692,
+            -0.023235931992530823,
+            0.028344441205263138,
+            -0.020080260932445526,
+            0.011823155917227268,
+            -0.020943470299243927,
+            -0.025868017226457596,
+            -0.016500059515237808,
+            0.025061409920454025,
+            0.024806693196296692,
+            0.02583971433341503,
+            0.003827843815088272,
+            -0.03492465242743492,
+            0.033651065081357956,
+            0.014674580655992031,
+            0.012488252483308315,
+            -0.031500112265348434,
+            -0.009162768721580505,
+            -0.05351905897259712,
+            -0.05227377265691757,
+            0.04143410921096802,
+            0.022245362401008606,
+            -0.016231190413236618,
+            0.023717066273093224,
+            0.017264213413000107,
+            0.005816058721393347,
+            -0.020632149651646614,
+            -0.020844414830207825,
+            0.003700484987348318,
+            -0.013103820383548737,
+            -0.01983969286084175,
+            -0.013528349809348583,
+            -0.01828308403491974,
+            -0.01577835902571678,
+            0.009551920928061008,
+            0.0002856731880456209,
+            0.0029186424799263477,
+            -0.003424540627747774,
+            -0.047830358147621155,
+            0.0248208437114954,
+            0.019853845238685608,
+            0.03492465242743492,
+            0.04633035510778427,
+            0.013011839240789413,
+            0.01516986545175314,
+            -0.014271277002990246,
+            -0.04021712392568588,
+            0.005497660953551531,
+            -0.00041656987741589546,
+            -0.005380915477871895,
+            -0.005819596350193024,
+            0.008575502783060074,
+            0.008724087849259377,
+            -0.01143400277942419,
+            -0.04202845320105553,
+            0.013952880166471004,
+            -0.020561393350362778,
+            0.00010143388499272987,
+            0.01354957651346922,
+            0.008292482234537601,
+            0.02677368000149727,
+            0.03195294365286827,
+            0.023971782997250557,
+            -0.008724087849259377,
+            -0.01340806670486927,
+            0.019924599677324295,
+            -0.0022676968947052956,
+            0.007018893025815487,
+            0.04570771008729935,
+            -0.04033033177256584,
+            0.02163686975836754,
+            -0.025726506486535072,
+            -0.015339677222073078,
+            -0.005349075887352228,
+            -0.022471778094768524,
+            0.02806142158806324,
+            -0.007344366051256657,
+            0.017801949754357338,
+            0.017094401642680168,
+            0.01594817079603672,
+            0.013330236077308655,
+            -0.001894464367069304,
+            -0.022938761860132217,
+            0.0018360915128141642,
+            -0.01541043259203434,
+            0.02114158496260643,
+            0.0015230007702484727,
+            0.005469358991831541,
+            0.027594437822699547,
+            0.031160488724708557,
+            -0.023490650579333305,
+            0.009502392262220383,
+            0.02422650158405304,
+            0.06967949122190475,
+            0.015424583107233047,
+            -0.006435164716094732,
+            -0.0024640418123453856,
+            0.04143410921096802,
+            0.0015513027319684625,
+            0.0038349193055182695,
+            -0.06826439499855042,
+            -0.03022652305662632,
+            -0.0011365016689524055,
+            -0.02254253253340721,
+            0.02227366529405117,
+            0.01002597901970148,
+            -0.009750034660100937,
+            -0.022457627579569817,
+            -0.04516997188329697,
+            0.05190584436058998,
+            -0.00545874610543251,
+            -0.016259491443634033,
+            -0.005554265342652798,
+            -0.05340585112571716,
+            0.00039136342820711434,
+            0.015042506158351898,
+            -0.018481198698282242,
+            -0.03670767694711685,
+            0.036141637712717056,
+            0.0007398316520266235,
+            0.03741522878408432,
+            0.012459950521588326,
+            -0.01708024926483631,
+            -0.020094411447644234,
+            0.022259512916207314,
+            -0.011639192700386047,
+            -0.0038313816767185926,
+            -0.027339721098542213,
+            0.0065236082300543785,
+            -0.028103873133659363,
+            -0.0014566679019480944,
+            0.06730213016271591,
+            -0.04816998168826103,
+            0.02946236915886402,
+            -0.029660483822226524,
+            0.023660462349653244,
+            -0.008575502783060074,
+            0.0278916098177433,
+            -0.010238244198262691,
+            0.005840822588652372,
+            -0.013457595370709896,
+            -0.0010586712742224336,
+            -0.020745357498526573,
+            0.0034210027661174536,
+            -0.016259491443634033,
+            -0.027750099077820778,
+            -0.007054270710796118,
+            0.031273696571588516,
+            0.01698119193315506,
+            0.025655752047896385,
+            0.025429336354136467,
+            0.01246702577918768,
+            -0.04935866594314575,
+            0.02595292218029499,
+            0.0018926954362541437,
+            0.013514199294149876,
+            0.013181651011109352,
+            -0.0010418669553473592,
+            0.025599148124456406,
+            0.012049571610987186,
+            0.025061409920454025,
+            0.037443529814481735,
+            0.0027682881336659193,
+            0.04737752676010132,
+            -0.015693452209234238,
+            -0.005292471963912249,
+            0.028528403490781784,
+            0.000505455827806145,
+            0.02465103194117546,
+            -0.001686621573753655,
+            0.013330236077308655,
+            0.000776978034991771,
+            0.03834919258952141,
+            -0.006339645478874445,
+            0.029518973082304,
+            0.021608568727970123,
+            0.028316138312220573,
+            -0.014759486541152,
+            0.023122724145650864,
+            -0.019754787907004356,
+            -0.02292460948228836,
+            0.035547297447919846,
+            0.004082561936229467,
+            0.01886327564716339,
+            -0.028457649052143097,
+            -0.030679354444146156,
+            0.015452885068953037,
+            -0.01855195313692093,
+            0.016556663438677788,
+            0.03639635816216469,
+            0.0008273909334093332,
+            -0.0025719432160258293,
+            0.026292547583580017,
+            -0.04245298355817795,
+            0.012856177985668182,
+            -0.01532552670687437,
+            -0.02855670638382435,
+            -0.019528372213244438,
+            -0.00875946506857872,
+            -0.0018095583654940128,
+            -0.013691086322069168,
+            -0.0037570889107882977,
+            -0.007563706487417221,
+            -0.02459442801773548,
+            0.00935380719602108,
+            0.013627407141029835,
+            0.009849091991782188,
+            -0.018665160983800888,
+            0.0019952901639044285,
+            -0.008257105015218258,
+            -0.0160047747194767,
+            -0.03752843663096428,
+            0.015268922783434391,
+            -0.0024888061452656984,
+            0.009191070683300495,
+            0.04078316316008568,
+            0.028485950082540512,
+            -0.005055442452430725,
+            0.0329718142747879,
+            -0.033707667142152786,
+            0.017915157601237297,
+            -0.008320784196257591,
+            0.009191070683300495,
+            0.002989397384226322,
+            -0.025542544201016426,
+            0.027424626052379608,
+            -0.03792466223239899,
+            -0.038151077926158905,
+            0.022995365783572197,
+            0.011858533136546612,
+            0.0017096169758588076,
+            -0.019386861473321915,
+            -0.017717044800519943,
+            0.0008499440737068653,
+            0.002752368338406086,
+            -0.018834972754120827,
+            -0.01019579079002142,
+            0.02227366529405117,
+            -0.025500090792775154,
+            0.036085035651922226,
+            0.023391593247652054,
+            -0.03520767390727997,
+            -0.00608492735773325,
+            -0.01903308741748333,
+            0.059037946164608,
+            -0.0011506526498124003,
+            -0.010301923379302025,
+            0.022938761860132217,
+            -0.027268964797258377,
+            -0.004524780437350273,
+            0.022301966324448586,
+            0.01828308403491974,
+            -0.008278331719338894,
+            0.003983504604548216,
+            -0.022287815809249878,
+            -0.05281151086091995,
+            -0.010577867738902569,
+            -0.0003305583668407053,
+            0.054368119686841965,
+            -0.012367968447506428,
+            0.0020695829298347235,
+            -0.006413938011974096,
+            0.004100250545889139,
+            0.03704730048775673,
+            0.002140337834134698,
+            0.0025082635693252087,
+            -0.013698162510991096,
+            0.0371888130903244,
+            0.022316116839647293,
+            0.005883275531232357,
+            -0.012162779457867146,
+            0.005377377849072218,
+            -0.04899074137210846,
+            -0.0009118546731770039,
+            0.005741765722632408,
+            0.007386818993836641,
+            -0.025910470634698868,
+            0.017717044800519943,
+            0.009205221198499203,
+            -0.0024693484883755445,
+            -0.009792488068342209,
+            0.01173117384314537,
+            0.025429336354136467,
+            -0.019938750192523003,
+            -0.05700020492076874,
+            0.0034705311991274357,
+            0.012014193460345268,
+            -0.004054259974509478,
+            -0.019401011988520622,
+            0.006792477332055569,
+            0.0021934041287750006,
+            0.022698193788528442,
+            -0.011271266266703606,
+            -0.0029221803415566683,
+            0.040585052222013474,
+            0.009516543708741665,
+            -0.0016813150141388178,
+            -0.017717044800519943,
+            -0.011794853024184704,
+            0.018070818856358528,
+            0.004740582779049873,
+            0.0035289041697978973,
+            -0.01751893013715744,
+            -0.029264256358146667,
+            0.02800481766462326,
+            0.019967053085565567,
+            -0.048905834555625916,
+            -0.008200501091778278,
+            -0.019273653626441956,
+            0.0019599127117544413,
+            0.008695785887539387,
+            0.04613224044442177,
+            -0.00590803986415267,
+            0.008313708938658237,
+            -0.020179318264126778,
+            -0.05003791302442551,
+            -0.02708500251173973,
+            0.03707560524344444,
+            -0.028684064745903015,
+            -0.04276430234313011,
+            -0.010252394713461399,
+            0.016740625724196434,
+            -0.009403335861861706,
+            -0.018679311498999596,
+            -0.011738249100744724,
+            -0.04969828948378563,
+            -0.003497064346447587,
+            0.0337359681725502,
+            0.017264213413000107,
+            -0.008441068232059479,
+            -0.033877480775117874,
+            0.019641580060124397,
+            -0.007160402834415436,
+            -0.006474079564213753,
+            0.0007862646016292274,
+            0.008794843219220638,
+            0.02400008589029312,
+            0.011879759840667248,
+            -0.02763689123094082,
+            -0.03427370637655258,
+            -0.0024888061452656984,
+            0.014929298311471939,
+            -0.00878776703029871,
+            0.0245378240942955,
+            0.024679332971572876,
+            -0.022995365783572197,
+            -0.01654251106083393,
+            0.015750056132674217,
+            -0.05779265984892845,
+            -0.010287772864103317,
+            -0.029235953465104103,
+            -0.006183984223753214,
+            0.0009613831643946469,
+            0.0061096916906535625,
+            0.00012304731353651732,
+            0.03846240043640137,
+            0.008801918476819992,
+            0.006014172453433275,
+            0.009191070683300495,
+            0.01913214474916458,
+            0.028471799567341805,
+            0.009155693463981152,
+            -0.03517937287688255,
+            0.02850010246038437,
+            -0.017051948234438896,
+            0.007811348885297775,
+            -0.025754809379577637,
+            -0.013033065013587475,
+            0.00443633645772934,
+            0.02725481428205967,
+            -0.000503244751598686,
+            0.018849123269319534,
+            -0.021919889375567436,
+            0.037500135600566864,
+            -0.029915200546383858,
+            -0.05278320610523224,
+            -0.02005195803940296,
+            -0.008285406976938248,
+            0.015764206647872925,
+            0.017278363928198814,
+            -0.019924599677324295,
+            -0.0286557637155056,
+            -0.021750077605247498,
+            0.006470541935414076,
+            0.010542490519583225,
+            -0.021099131554365158,
+            -0.010627396404743195,
+            0.01502835564315319,
+            -0.00404364662244916,
+            -0.01886327564716339,
+            0.013684011064469814,
+            0.015693452209234238,
+            0.03169822692871094,
+            -0.03308502212166786,
+            0.004287751391530037,
+            -0.025967074558138847,
+            -0.04327373951673508,
+            0.004096712917089462,
+            0.03166992589831352,
+            0.027509532868862152,
+            -0.018636859953403473,
+            0.010754755698144436,
+            0.003990580327808857,
+            0.004234685096889734,
+            -0.0033803186379373074,
+            0.023603858426213264,
+            0.03291521221399307,
+            0.02763689123094082,
+            -0.011653343215584755,
+            0.004921008367091417,
+            -0.01559439580887556,
+            -0.0011108530452474952,
+            0.01865101046860218,
+            -0.017816102132201195,
+            0.033877480775117874,
+            -0.029405765235424042,
+            0.02104252763092518,
+            -0.035122767090797424,
+            0.006141531281173229,
+            -0.012099100276827812,
+            -0.00404364662244916,
+            0.04833979532122612,
+            -0.017646290361881256,
+            -0.0019546060357242823,
+            -0.008101443760097027,
+            -0.014575524255633354,
+            -0.02158026583492756,
+            0.008766540326178074,
+            -0.04033033177256584,
+            -0.012367968447506428,
+            -0.028429346159100533,
+            0.003424540627747774,
+            0.019910449162125587,
+            -0.06277380883693695,
+            -0.06237758323550224,
+            0.012275987304747105,
+            -0.04918885603547096,
+            -0.026419905945658684,
+            0.02963218092918396,
+            -0.002984090941026807,
+            0.0012780115939676762,
+            -0.0024003623984754086,
+            0.0068101659417152405,
+            -0.029207652434706688,
+            0.00425944896414876,
+            -0.000887974863871932,
+            0.03271709755063057,
+            -0.021056678146123886,
+            -0.008122670464217663,
+            0.004878554958850145,
+            -0.06860402226448059,
+            -0.028344441205263138,
+            0.011271266266703606,
+            -0.009459939785301685,
+            -0.0066615804098546505,
+            -0.017915157601237297,
+            -0.004383270163089037,
+            0.0008950503543019295,
+            0.0032830305863171816,
+            0.05801907554268837,
+            -0.01983969286084175,
+            -0.053773775696754456,
+            0.03633975237607956,
+            0.00887267291545868,
+            0.029603879898786545,
+            -0.025556694716215134,
+            -0.017533080652356148,
+            -0.01855195313692093,
+            -0.020504789426922798,
+            -0.04123599827289581,
+            0.00404364662244916,
+            -0.005918653216212988,
+            0.009212297387421131,
+            -0.0009693430620245636,
+            0.02763689123094082,
+            0.029235953465104103,
+            0.023108573630452156,
+            0.012750045396387577,
+            0.01079013291746378,
+            -0.022372720763087273,
+            -0.021594416350126266,
+            0.0012222920777276158,
+            0.04683978855609894,
+            0.024636881425976753,
+            -0.02675952948629856,
+            -0.011504758149385452,
+            0.03953787684440613,
+            0.0419718474149704,
+            -0.01974063739180565,
+            -0.02276895008981228,
+            0.00764861237257719,
+            0.03266049548983574,
+            -0.04477374628186226,
+            0.005932804197072983,
+            0.03410389646887779,
+            0.014405711553990841,
+            -0.005356151144951582,
+            -0.010294848121702671,
+            0.022570835426449776,
+            0.03152841329574585,
+            -0.008724087849259377,
+            0.01876421831548214,
+            0.016910437494516373,
+            0.029349161311984062,
+            -0.0067889392375946045,
+            0.0005766529939137399,
+            -0.024735936895012856,
+            0.055188875645399094,
+            -0.03059444949030876,
+            0.02909444272518158,
+            0.010316074825823307,
+            -0.00593634182587266,
+            0.039056744426488876,
+            -0.016641568392515182,
+            -0.0431322306394577,
+            0.01913214474916458,
+            -0.006979977712035179,
+            -0.0055613406002521515,
+            -0.03444352000951767,
+            -0.007592008449137211,
+            -0.007507102098315954,
+            -0.006714646704494953,
+            -0.022556684911251068,
+            -0.02606613002717495,
+            0.004415110219269991,
+            0.00973588414490223,
+            -0.006484692916274071,
+            0.00698705343529582,
+            0.02092931978404522,
+            -0.013507124036550522,
+            0.02525952458381653,
+            -0.046047333627939224,
+            -0.0008366775000467896,
+            -0.0038596836384385824,
+            -0.025712355971336365,
+            -0.0345001220703125,
+            -0.0005770951975136995,
+            -0.0024251264985650778,
+            -0.024622729048132896,
+            -0.00914154201745987,
+            -0.008752389810979366,
+            -0.0023119186516851187,
+            -0.004938696976751089,
+            -0.055188875645399094,
+            0.008695785887539387,
+            0.010846736840903759,
+            -0.002713453024625778,
+            0.0055507272481918335,
+            -0.010613245889544487,
+            -0.03039633482694626,
+            0.03039633482694626,
+            0.04129260033369064,
+            0.033707667142152786,
+            0.014334957115352154,
+            -0.0074646491557359695,
+            -0.017165156081318855,
+            0.006060163024812937,
+            0.021721776574850082,
+            0.00902833417057991,
+            0.007846726104617119,
+            0.03877372294664383,
+            0.00011331850691931322,
+            -0.008349086157977581,
+            -0.009764186106622219,
+            0.028740668669342995,
+            -0.001085204305127263,
+            0.023603858426213264,
+            0.050802066922187805,
+            0.030424637719988823,
+            0.020589696243405342,
+            -0.01681138016283512,
+            -0.02010856196284294,
+            -0.00851889792829752,
+            -0.02855670638382435,
+            0.0364246591925621,
+            0.012127402238547802,
+            -0.005232329946011305,
+            -0.011540135368704796,
+            0.019061388447880745,
+            -0.033339742571115494,
+            -0.0037358624394983053,
+            0.00048422932741232216,
+            0.013924578204751015,
+            -0.03379257395863533,
+            0.0002480846014805138,
+            0.031613320112228394,
+            0.014957600273191929,
+            -0.016995344310998917,
+            -0.0188774261623621,
+            0.010549565777182579,
+            0.020462337881326675,
+            0.018948180601000786,
+            0.00037743354914709926,
+            0.00897173024713993,
+            -0.00022121983056422323,
+            -0.0005319889169186354,
+            0.01590571738779545,
+            -0.007952858693897724,
+            0.036254845559597015,
+            -0.003525366308167577,
+            -0.01325240544974804,
+            -0.0037500134203583,
+            -0.021438755095005035,
+            -0.024467067793011665,
+            0.043556760996580124,
+            0.023971782997250557,
+            0.04661337286233902,
+            -0.005660397466272116,
+            -0.004245297983288765,
+            0.025825563818216324,
+            0.008221727795898914,
+            -0.03288691118359566,
+            0.000265110022155568,
+            0.005281858611851931,
+            -0.02892463095486164,
+            0.016641568392515182,
+            -0.02476423978805542,
+            -0.016896286979317665,
+            -0.006771250627934933,
+            -0.0015557248843833804,
+            -0.028188779950141907,
+            0.01703779585659504,
+            0.0031309074256569147,
+            -0.0027099153958261013,
+            -0.039764292538166046,
+            0.006880920846015215,
+            0.020419884473085403,
+            0.015311375260353088,
+            0.0037358624394983053,
+            0.011525984853506088,
+            -0.016613267362117767,
+            0.002126186853274703,
+            0.03379257395863533,
+            0.004878554958850145,
+            0.02005195803940296,
+            0.0027647505048662424,
+            0.023773670196533203,
+            0.03633975237607956,
+            -0.01681138016283512,
+            -0.02189158834517002,
+            0.01519816741347313,
+            -0.02233026921749115,
+            0.029943503439426422,
+            0.004889168310910463,
+            -0.012283062562346458,
+            0.022033097222447395,
+            -0.009827865287661552,
+            -0.0027346794959157705,
+            0.01659911684691906,
+            -0.022740647196769714,
+            -0.017717044800519943,
+            0.03183973580598831,
+            -0.039452970027923584,
+            0.012856177985668182,
+            -0.008547200821340084,
+            0.015863263979554176,
+            -0.023066120222210884,
+            0.011172209866344929,
+            0.007100261282175779,
+            -0.01573590561747551,
+            0.010797208175063133,
+            0.009693430736660957,
+            0.00545874610543251,
+            0.0018926954362541437,
+            0.02179253101348877,
+            -0.0006049550138413906,
+            -0.011257115751504898,
+            -0.018155725672841072,
+            -0.004694592207670212,
+            -0.002614396158605814,
+            -0.016556663438677788,
+            0.01865101046860218,
+            -0.041066184639930725,
+            -0.013280707411468029,
+            -0.008879749104380608,
+            -0.013988257385790348,
+            -0.010011828504502773,
+            0.024184048175811768,
+            0.025669902563095093,
+            -0.002361447084695101,
+            -0.031217092648148537,
+            0.01790100708603859,
+            0.024198200553655624,
+            0.003654494183138013,
+            -0.023646311834454536,
+            0.0302831269800663,
+            -0.008936353027820587,
+            0.014971751719713211,
+            0.026136886328458786,
+            0.014094389975070953,
+            0.004595535341650248,
+            -0.010500037111341953,
+            -0.02114158496260643,
+            0.01481609046459198,
+            0.010507113300263882,
+            0.033537857234478,
+            0.014745336025953293,
+            0.02071705460548401,
+            0.01594817079603672,
+            -0.023320838809013367,
+            -0.011214662343263626,
+            -0.003482913365587592,
+            -0.022514231503009796,
+            0.03093407303094864,
+            0.0006129149114713073,
+            0.02114158496260643,
+            0.026618018746376038,
+            -0.03319823369383812,
+            -0.01187268365174532,
+            0.032688796520233154,
+            0.02779255248606205,
+            0.05108508840203285,
+            0.025316128507256508,
+            0.015396281145513058,
+            0.027226513251662254,
+            -0.03585861995816231,
+            -0.036368053406476974,
+            0.001548649393953383,
+            0.04256619140505791,
+            -0.008426916785538197,
+            0.00967928022146225,
+            0.03900013864040375,
+            -0.006781863979995251,
+            -0.01611798256635666,
+            0.019457615911960602,
+            0.0026710000820457935,
+            0.002448122017085552,
+            -0.0033113325480371714,
+            -0.01335146278142929,
+            0.03175482898950577,
+            -0.0051474240608513355,
+            0.013570803217589855,
+            -0.019273653626441956,
+            0.027113303542137146,
+            0.0001799829478841275,
+            -0.003229964291676879,
+            0.028344441205263138,
+            -0.002660386962816119,
+            -0.051990751177072525,
+            0.024212351068854332,
+            0.019811391830444336,
+            -0.011561362072825432,
+            -0.0036332677118480206,
+            -0.003792466362938285,
+            0.04567940905690193,
+            0.034698236733675,
+            0.029009537771344185,
+            -0.0020501252729445696,
+            0.0168679840862751,
+            0.018792519345879555,
+            0.0053667644970119,
+            0.014278353191912174,
+            -0.0017025414854288101,
+            -0.007853801362216473,
+            0.021056678146123886,
+            -0.020787810906767845,
+            0.03226426616311073,
+            -0.016570813953876495,
+            -0.020122714340686798,
+            0.008738238364458084,
+            -0.011172209866344929,
+            -0.0009083169279620051,
+            -0.006311343517154455,
+            -0.0008813415770418942,
+            -0.0029664020985364914,
+            0.016938740387558937,
+            -0.004446949809789658,
+            0.04661337286233902,
+            -0.013931653462350368,
+            -0.008808993734419346,
+            0.006972902454435825,
+            -0.025981225073337555,
+            0.0013346156338229775,
+            0.0028726516757160425,
+            0.027000095695257187,
+            -0.01989629678428173,
+            -0.00970758218318224,
+            0.01606137864291668,
+            0.06039644405245781,
+            -0.008639181964099407,
+            -0.0026267783250659704,
+            0.027226513251662254,
+            -0.002787745790556073,
+            -0.03551899641752243,
+            0.012919857166707516,
+            -0.04494355618953705,
+            0.017589686438441277,
+            0.026349151507019997,
+            -0.04986810311675072,
+            -0.022089701145887375,
+            -0.02347649820148945,
+            -0.02043403498828411,
+            0.023434046655893326,
+            -0.009976450353860855,
+            0.013867974281311035,
+            -0.017801949754357338,
+            -0.009014182724058628,
+            0.020518941804766655,
+            0.009176919236779213,
+            0.03735862299799919,
+            -0.027551984414458275,
+            -0.016783079132437706,
+            -0.027820853516459465,
+            -0.04304732382297516,
+            0.009558996185660362,
+            -0.0278916098177433,
+            0.0027099153958261013,
+            -0.0016291331266984344,
+            -0.040302030742168427,
+            -0.037981268018484116,
+            0.005278320983052254,
+            -0.017985913902521133,
+            0.01516986545175314,
+            0.020915169268846512,
+            0.02731141820549965,
+            -0.0020253609400242567,
+            -0.030198222026228905,
+            0.029235953465104103,
+            -0.007344366051256657,
+            0.03223596513271332,
+            -0.00689507182687521,
+            0.005066055804491043,
+            -0.0070436573587358,
+            0.031160488724708557,
+            -0.023009516298770905,
+            0.018792519345879555,
+            -0.026801982894539833,
+            -0.004086099565029144,
+            -0.018990634009242058,
+            -0.01774534583091736,
+            0.004224071744829416,
+            0.004521242808550596,
+            7.888073741924018e-05,
+            0.0048750173300504684,
+            0.005777143407613039,
+            0.024806693196296692,
+            0.0021421066485345364,
+            0.02589631825685501,
+            -0.027877457439899445,
+            -0.025316128507256508,
+            0.01719345711171627,
+            -0.02077365852892399,
+            0.005678086541593075,
+            -0.006261814851313829,
+            -0.0302831269800663,
+            -0.01202834490686655,
+            0.023391593247652054,
+            0.02162271924316883,
+            -0.005490585695952177,
+            0.002644466934725642,
+            0.02206140011548996,
+            -0.026023678481578827,
+            0.00437973253428936,
+            -0.03781145438551903,
+            -0.018396291881799698,
+            0.029773691669106483,
+            -0.03195294365286827,
+            0.020745357498526573,
+            0.003185742534697056,
+            0.03483974561095238,
+            -0.017292514443397522,
+            -0.01947176828980446,
+            0.01654251106083393,
+            -0.01008258294314146,
+            0.004142703488469124,
+            0.01606137864291668,
+            0.01616043597459793,
+            -0.0015070808585733175,
+            0.05380208045244217,
+            0.0008167776977643371,
+            0.006909222807735205,
+            -0.049075648188591,
+            0.004496478475630283,
+            -0.044858649373054504,
+            0.0002768288250081241,
+            -0.013337312266230583,
+            -0.02514631673693657,
+            0.0014027173165231943,
+            0.001001182827167213,
+            0.001963450340554118,
+            -0.01913214474916458,
+            -0.01989629678428173,
+            0.011596739292144775,
+            -0.015863263979554176,
+            0.004928083624690771,
+            -0.06418890506029129,
+            -0.01855195313692093,
+            -0.004974074196070433,
+            0.0043337419629096985,
+            0.0007699025445617735,
+            0.03342464938759804,
+            -0.01478778850287199,
+            0.010117961093783379,
+            -0.01549533847719431,
+            -0.026363302022218704,
+            0.04010391607880592,
+            0.018750067800283432,
+            -0.023773670196533203,
+            0.024693485349416733,
+            0.012106175534427166,
+            0.03099067695438862,
+            -0.0318114347755909,
+            0.020349130034446716,
+            0.018523650243878365,
+            -0.0029823218937963247,
+            0.006551910191774368,
+            0.008575502783060074,
+            0.01832553744316101,
+            -0.015764206647872925,
+            -0.0003632825391832739,
+            0.011419852264225483,
+            0.010033054277300835,
+            0.031924642622470856,
+            0.01403071079403162,
+            -0.03183973580598831,
+            0.019047237932682037,
+            0.01616043597459793,
+            0.02298121340572834,
+            0.033820874989032745,
+            -0.027481229975819588,
+            0.038575608283281326,
+            -0.010386829264461994,
+            -0.024042539298534393,
+            0.020801961421966553,
+            -0.032688796520233154,
+            -0.006679269019514322,
+            0.030735958367586136,
+            0.04398128762841225,
+            -0.0015566092915832996,
+            -0.044632233679294586,
+            0.01535382866859436,
+            0.01316749956458807,
+            0.0027806703001260757,
+            0.028811423107981682,
+            0.011264191009104252,
+            0.021820832043886185,
+            0.025273675099015236,
+            0.011419852264225483,
+            -0.0037252490874379873,
+            0.024962352588772774,
+            -0.008101443760097027,
+            -0.004446949809789658,
+            -0.0007035697344690561,
+            0.03206615149974823,
+            0.014702882617712021,
+            -0.017603836953639984,
+            -0.036962397396564484,
+            -0.007620310410857201,
+            0.009452863596379757,
+            -0.007903330028057098,
+            -0.027127455919981003,
+            -0.03950957581400871,
+            -0.026787832379341125,
+            -0.005667473189532757,
+            0.01505665760487318,
+            -0.06724552065134048,
+            -0.007620310410857201,
+            -0.012233533896505833,
+            0.017717044800519943,
+            0.0022906921803951263,
+            0.0181132722645998,
+            0.001277127186767757,
+            -0.009403335861861706,
+            0.004209920763969421,
+            0.050802066922187805,
+            -0.03059444949030876,
+            0.014242975041270256,
+            0.011292492970824242,
+            0.0032105068676173687,
+            0.002787745790556073,
+            0.038688816130161285,
+            -0.010988246649503708,
+            -0.030424637719988823,
+            0.011575513519346714,
+            -0.017292514443397522,
+            0.003383856499567628,
+            0.05006621778011322,
+            -0.009481165558099747,
+            -0.027495380491018295,
+            0.020915169268846512,
+            -0.009148617275059223,
+            -0.015523640438914299,
+            -0.0027258351910859346,
+            -0.006035399157553911,
+            0.0006283925613388419,
+            -0.03832089155912399,
+            -0.015750056132674217,
+            0.03212275728583336,
+            0.008617955259978771,
+            0.014603826217353344,
+            0.025344429537653923,
+            0.010209942236542702,
+            -0.030481241643428802,
+            0.027481229975819588,
+            -0.0005678086308762431,
+            0.005221716593950987,
+            0.013323160819709301,
+            0.03424540534615517,
+            -0.014264201745390892,
+            -0.020264223217964172,
+            -0.040245428681373596,
+            -0.013853822834789753,
+            -0.0463869571685791,
+            0.001943992800079286,
+            0.006919836159795523,
+            -0.012919857166707516,
+            -0.04366996884346008,
+            -0.0329718142747879,
+            0.021127434447407722,
+            0.03178313374519348,
+            -0.017023645341396332,
+            0.0025294902734458447,
+            -0.008483520708978176,
+            -0.007882103323936462,
+            -0.0017520699184387922,
+            -0.013217028230428696,
+            -0.006378560326993465,
+            -0.006229975260794163,
+            0.015452885068953037,
+            -0.004652139265090227,
+            -0.0036934094969183207,
+            -0.0498398020863533,
+            -0.018778368830680847,
+            0.004082561936229467,
+            -0.008221727795898914,
+            0.007323139347136021,
+            -0.006955213379114866,
+            -0.002727604005485773,
+            -0.012474101036787033,
+            -0.03829259052872658,
+            0.024184048175811768,
+            0.017278363928198814,
+            0.0035501306410878897,
+            -0.008922201581299305,
+            -0.018367990851402283,
+            0.03294351324439049,
+            0.012523629702627659,
+            -0.017547233030200005,
+            -0.018580254167318344,
+            0.025981225073337555,
+            0.026957644149661064,
+            0.008122670464217663,
+            -0.006137993652373552,
+            0.029886899515986443,
+            0.005695775151252747,
+            -0.022938761860132217,
+            -0.003983504604548216,
+            -0.005055442452430725,
+            -0.0038844477385282516,
+            0.020561393350362778,
+            -0.010556641966104507,
+            0.0300001073628664,
+            0.038717120885849,
+            0.012870329432189465,
+            0.0039198254235088825,
+            -0.0033714743331074715,
+            -0.008922201581299305,
+            0.015976471826434135,
+            0.006481155287474394,
+            0.008525974117219448,
+            0.014759486541152,
+            -0.016358548775315285,
+            -0.020405733957886696,
+            0.006215824279934168,
+            -0.007107336539775133,
+            0.020122714340686798,
+            0.01708024926483631,
+            0.016429303213953972,
+            0.0007610581233166158,
+            -0.028287837281823158,
+            -0.0018024828750640154,
+            0.02411329373717308,
+            0.02010856196284294,
+            -0.004655676893889904,
+            -0.023264234885573387,
+            0.014398636296391487,
+            -0.015806660056114197,
+            -0.009969375096261501,
+            0.0033431723713874817,
+            0.006229975260794163,
+            -0.003083147807046771,
+            0.0571134127676487,
+            -0.020193468779325485,
+            -0.011419852264225483,
+            -0.010287772864103317,
+            -0.010867963545024395,
+            0.0012072566896677017,
+            -0.01828308403491974,
+            0.008964654989540577,
+            0.025613298639655113,
+            -0.009764186106622219,
+            0.03568880632519722,
+            0.0228114016354084,
+            0.006912760436534882,
+            0.005260631907731295,
+            -0.027070851996541023,
+            0.012304289266467094,
+            -0.016047226265072823,
+            0.005660397466272116,
+            0.018467046320438385,
+            0.017320817336440086,
+            0.0076132346875965595,
+            0.026306698098778725,
+            0.019386861473321915,
+            0.016684021800756454,
+            -0.022783100605010986,
+            0.025457637384533882,
+            -0.007832575589418411,
+            0.027877457439899445,
+            -0.01611798256635666,
+            -0.002253545681014657,
+            -0.012990612536668777,
+            0.008681634441018105,
+            0.002416282193735242,
+            -0.0019351483788341284,
+            0.00036129256477579474,
+            0.016783079132437706,
+            -0.014589674770832062,
+            0.01062032114714384,
+            0.013316085562109947,
+            -0.027919910848140717,
+            -0.010372678749263287,
+            0.016344398260116577,
+            -0.00668988237157464,
+            0.006877383217215538,
+            -0.011865608394145966,
+            -0.0017202302115038037,
+            0.011009473353624344,
+            -0.0436982698738575,
+            0.0104929618537426,
+            0.006279503460973501,
+            -0.015155714936554432,
+            0.014801939949393272,
+            -0.009799563325941563,
+            -0.008235878311097622,
+            -0.013592029921710491,
+            0.003792466362938285,
+            -0.00030225637601688504,
+            -0.0018290160223841667,
+            -0.00022995365725364536,
+            0.00389859895221889,
+            0.010047205723822117,
+            -0.0045990729704499245,
+            -0.010450509376823902,
+            -0.0028638073708862066,
+            -0.02265574224293232,
+            -0.0010047205723822117,
+            0.018735915422439575,
+            -0.004591997712850571,
+            0.0005222601466812193,
+            0.0145189194008708,
+            -0.005830209702253342,
+            0.010556641966104507,
+            -0.005221716593950987,
+            -0.008320784196257591,
+            -0.010768906213343143,
+            -0.033764272928237915,
+            0.0026710000820457935,
+            -0.015254771336913109,
+            0.008865597657859325,
+            -0.012544856406748295,
+            0.04027372971177101,
+            0.01022409275174141,
+            -0.00837738811969757,
+            -0.002796590095385909,
+            -0.0228114016354084,
+            -0.004475251771509647,
+            -0.016938740387558937,
+            -0.01184438169002533,
+            0.008342010900378227,
+            0.012629762291908264,
+            0.005614406894892454,
+            -0.018410442396998405,
+            0.003665107535198331,
+            0.0030955299735069275,
+            -0.005575491581112146,
+            -0.008370312862098217,
+            0.010853812098503113,
+            -0.0006248548161238432,
+            -0.01800006441771984,
+            -0.01957082375884056,
+            -0.027424626052379608,
+            0.0150000536814332,
+            0.014589674770832062,
+            -0.01011088490486145,
+            -0.022754797711968422,
+            -0.007471724878996611,
+            0.007107336539775133,
+            -0.02064630016684532,
+            0.0006779210525564849,
+            0.006891534198075533,
+            0.008717012591660023,
+            0.004117939155548811,
+            0.009636826813220978,
+            0.0007699025445617735,
+            0.0121698547154665,
+            0.018566103652119637,
+            0.03271709755063057,
+            -0.0396793857216835,
+            -0.009537769481539726,
+            0.0057028504088521,
+            -0.001465512323193252,
+            -0.013853822834789753,
+            0.009672204032540321,
+            0.0024375086650252342,
+            -0.0026426981203258038,
+            0.024424616247415543,
+            -0.008044839836657047,
+            -0.01505665760487318,
+            0.01184438169002533,
+            0.00887267291545868,
+            -0.019075538963079453,
+            0.02746707946062088,
+            0.006169833242893219,
+            -0.01011088490486145,
+            0.02259913645684719,
+            -0.012360893189907074,
+            0.012658064253628254,
+            -0.005522425286471844,
+            0.023660462349653244,
+            -0.019004784524440765,
+            0.0006306036957539618,
+            0.015665151178836823,
+            0.03883032873272896,
+            -0.00437973253428936,
+            -0.004567233379930258,
+            -0.03421710431575775,
+            0.005115584470331669,
+            0.028853876516222954,
+            0.02666047215461731,
+            -0.0012346742441877723,
+            -0.005596718285232782,
+            -0.0019864458590745926,
+            0.010768906213343143,
+            0.011589664034545422,
+            0.017561383545398712,
+            -0.029547275975346565,
+            0.00022011429246049374,
+            -0.016938740387558937,
+            0.025500090792775154,
+            0.014632128179073334,
+            0.006951675750315189,
+            0.04072656109929085,
+            0.0027789014857262373,
+            0.003841994795948267,
+            0.027452928945422173,
+            -0.00779719790443778,
+            0.003546592779457569,
+            -0.015566092915832996,
+            0.0003175129240844399,
+            0.022853855043649673,
+            0.0016370931407436728,
+            0.0020784272346645594,
+            0.013337312266230583,
+            0.029320860281586647,
+            -0.022471778094768524,
+            -0.023929331451654434,
+            0.003482913365587592,
+            0.007185167167335749,
+            -0.010570792481303215,
+            0.0038738346192985773,
+            0.006275965832173824,
+            -0.006007096730172634,
+            0.008900974877178669,
+            0.006332569755613804,
+            -0.0003166284877806902,
+            0.0039198254235088825,
+            -0.00308845448307693,
+            0.012962310574948788,
+            -0.02702839858829975,
+            0.036849189549684525,
+            0.004439874552190304,
+            0.010075507685542107,
+            0.015268922783434391,
+            0.001627364312298596,
+            0.004917470272630453,
+            0.005869124550372362,
+            0.010188715532422066,
+            0.018636859953403473,
+            -0.00956607237458229,
+            -0.027481229975819588,
+            -0.03166992589831352,
+            0.020363280549645424,
+            -0.01173117384314537,
+            0.0035872769076377153,
+            0.0074646491557359695,
+            -0.01828308403491974,
+            -0.007507102098315954,
+            -0.009976450353860855,
+            0.015764206647872925,
+            -0.03189634159207344,
+            0.012990612536668777,
+            -0.0043231286108493805,
+            -0.028669914230704308,
+            0.00851889792829752,
+            -0.010181640274822712,
+            -0.0045884596183896065,
+            -0.0035094465129077435,
+            -0.029264256358146667,
+            -0.007931631989777088,
+            0.03908504545688629,
+            -0.02141045406460762,
+            0.024184048175811768,
+            0.008257105015218258,
+            -0.00014449491573031992,
+            -0.0040365708991885185,
+            -0.013563727959990501,
+            0.005331386812031269,
+            -0.01778779923915863,
+            0.025457637384533882,
+            0.010294848121702671,
+            -0.013924578204751015,
+            0.0322076641023159,
+            -0.011179285123944283,
+            0.027438776567578316,
+            -0.018580254167318344,
+            0.036198243498802185,
+            0.007422196213155985,
+            0.00674648629501462,
+            0.01974063739180565,
+            -0.004025958012789488,
+            0.008228803053498268,
+            0.04559450224041939,
+            -0.018523650243878365,
+            0.02460857853293419,
+            0.020349130034446716,
+            -0.00022685811563860625,
+            0.017915157601237297,
+            0.0028390432707965374,
+            -0.023165177553892136,
+            0.009495317004621029,
+            -0.01135617308318615,
+            -0.009955224581062794,
+            0.01583496294915676,
+            -0.019316107034683228,
+            -0.018721764907240868,
+            -0.025613298639655113,
+            0.0029416377656161785,
+            -0.003350247861817479,
+            -0.0077618202194571495,
+            -0.0019616815261542797,
+            0.01454722136259079,
+            0.008724087849259377,
+            0.025669902563095093,
+            0.0008326975512318313,
+            0.011971740983426571,
+            0.01511326152831316,
+            -0.00035576484515331686,
+            0.02313687466084957,
+            -0.03229256719350815,
+            0.01246702577918768,
+            -0.004517704714089632,
+            -0.00509789539501071,
+            0.023858575150370598,
+            0.0003747802402358502,
+            -0.0004877670726273209,
+            -0.0024888061452656984,
+            0.0005496777012012899,
+            -0.014589674770832062,
+            -0.0057099261321127415,
+            0.000907432462554425,
+            -0.005175726022571325,
+            0.04299071803689003,
+            -0.00701535539701581,
+            -0.004772422835230827,
+            -0.039169952273368835,
+            -0.01918874867260456,
+            0.00623351288959384,
+            -0.01583496294915676,
+            -0.018778368830680847,
+            -0.006307805422693491,
+            0.005801907740533352,
+            -0.04893413558602333,
+            0.008342010900378227,
+            0.03336804360151291,
+            -0.012219383381307125,
+            0.00259493850171566,
+            -0.010981171391904354,
+            -0.006969364359974861,
+            -0.0029115669894963503,
+            -7.766463386360556e-05,
+            -0.002412744564935565,
+            -0.020830262452363968,
+            -0.005617944523692131,
+            0.003364398842677474,
+            -0.0008795727044343948,
+            -0.025047259405255318,
+            0.0006177793256938457,
+            0.018849123269319534,
+            -0.003979966975748539,
+            -0.0011152751976624131,
+            -0.001825478277169168,
+            0.006399787031114101,
+            -0.01508495956659317,
+            0.008009462617337704,
+            -0.0076344613917171955,
+            0.003841994795948267,
+            -0.0030495391692966223,
+            -0.002214630600064993,
+            0.02844349853694439,
+            0.021594416350126266,
+            -0.04808507859706879,
+            -0.01405901275575161,
+            0.023009516298770905,
+            0.007280686404556036,
+            -0.004740582779049873,
+            0.01178777776658535,
+            -0.031613320112228394,
+            -0.00028943203506059945,
+            -0.026787832379341125,
+            0.02200479619204998,
+            -0.009205221198499203,
+            -0.007386818993836641,
+            0.016967041417956352,
+            -0.026632171124219894,
+            0.016344398260116577,
+            0.00794578343629837,
+            -0.00030181417241692543,
+            -0.000493515923153609,
+            0.0063856360502541065,
+            0.0008972614887170494,
+            -0.01659911684691906,
+            0.006442239973694086,
+            -0.0048502529971301556,
+            0.007959933951497078,
+            -0.005087282508611679,
+            -0.05527378246188164,
+            -0.009933997876942158,
+            -0.005497660953551531,
+            -0.008936353027820587,
+            -0.012820800766348839,
+            0.008136821910738945,
+            0.02827368676662445,
+            0.01606137864291668,
+            -0.009863242506980896,
+            -4.8312376748071983e-05,
+            0.033000119030475616,
+            -0.0008300442132167518,
+            0.009955224581062794,
+            -0.011349096894264221,
+            0.00015709814033471048,
+            -0.0006911876262165606,
+            -0.014660430140793324,
+            -0.019061388447880745,
+            0.024042539298534393,
+            -0.028344441205263138,
+            0.02378782071173191,
+            -0.01843874529004097,
+            -0.0130825936794281,
+            0.01757553406059742,
+            -0.032575588673353195,
+            -0.005108508747071028,
+            0.007450498174875975,
+            0.0090566361322999,
+            0.021608568727970123,
+            0.0032794929575175047,
+            0.020589696243405342,
+            -0.03716050833463669,
+            -0.009134466759860516,
+            -0.010160413570702076,
+            -0.00794578343629837,
+            -0.010231168940663338,
+            -0.011596739292144775,
+            0.021863285452127457,
+            -0.015240620821714401,
+            -0.016952890902757645,
+            0.007054270710796118,
+            0.0030212372075766325,
+            -0.011313719674944878,
+            0.015750056132674217,
+            0.00017810352437663823,
+            0.004712280817329884,
+            0.013011839240789413,
+            -0.0008923970744945109,
+            0.04086806997656822,
+            0.0253019779920578,
+            -0.0030212372075766325,
+            -0.008568426594138145,
+            0.021438755095005035,
+            0.0004285097820684314,
+            0.0150000536814332,
+            0.009375033900141716,
+            0.023264234885573387,
+            -0.02092931978404522,
+            0.009997677057981491,
+            -0.005501199048012495,
+            0.014603826217353344,
+            -0.009495317004621029,
+            -0.015339677222073078,
+            0.00479718716815114,
+            -0.01577835902571678,
+            0.010153338313102722,
+            -0.010860888287425041,
+            0.008221727795898914,
+            0.01913214474916458,
+            0.002467579673975706,
+            -0.00941041111946106,
+            0.01146230474114418,
+            0.02075950801372528,
+            -0.025641601532697678,
+            0.008851447142660618,
+            -0.0031998935155570507,
+            -0.0021916350815445185,
+            0.021127434447407722,
+            0.010301923379302025,
+            -0.01594817079603672,
+            0.01573590561747551,
+            -0.008879749104380608,
+            -0.0015053119277581573,
+            -0.01882082223892212,
+            -0.03291521221399307,
+            -0.0015283073298633099,
+            0.012396270409226418,
+            0.004800724796950817,
+            0.0150000536814332,
+            0.011264191009104252,
+            -0.0008950503543019295,
+            -0.006601438857614994,
+            0.0008433107868768275,
+            -0.013485897332429886,
+            -0.004963461309671402,
+            0.010231168940663338,
+            0.017051948234438896,
+            -0.0037217114586383104,
+            0.015650998800992966,
+            0.0088160689920187,
+            0.006137993652373552,
+            -0.0010480580385774374,
+            0.0036138100549578667,
+            -0.002060738392174244,
+            -0.007783046457916498,
+            0.019938750192523003,
+            0.019075538963079453,
+            -0.018509499728679657,
+            -0.0024870370980352163,
+            -0.02141045406460762,
+            0.025882167741656303,
+            0.022188758477568626,
+            0.011165134608745575,
+            -0.04222656786441803,
+            0.00012050455552525818,
+            0.00752832880243659,
+            -0.0013744152383878827,
+            -0.031103884801268578,
+            0.008172199130058289,
+            0.005221716593950987,
+            0.004330204334110022,
+            0.020844414830207825,
+            -0.018424594774842262,
+            0.010853812098503113,
+            0.0070861103013157845,
+            -0.0005766529939137399,
+            0.01019579079002142,
+            -0.0073372903279960155,
+            0.008688710629940033,
+            0.038717120885849,
+            -0.018410442396998405,
+            0.005303084850311279,
+            0.0062370505183935165,
+            0.023320838809013367,
+            -0.011228813789784908,
+            -0.00031485961517319083,
+            0.00800238735973835,
+            0.021608568727970123,
+            -0.012934008613228798,
+            0.005883275531232357,
+            0.067075714468956,
+            -0.012912781909108162,
+            0.031613320112228394,
+            0.020533092319965363,
+            -0.004970536567270756,
+            -0.0104929618537426,
+            -0.012445799075067043,
+            -0.0181132722645998,
+            0.021283095702528954,
+            0.01175947580486536,
+            0.003419233951717615,
+            0.01941516436636448,
+            -0.01610383205115795,
+            0.006636816076934338,
+            0.017547233030200005,
+            0.012601460330188274,
+            -0.007379743270576,
+            -0.0036173479165881872,
+            0.019004784524440765,
+            0.018509499728679657,
+            -0.0030849166214466095,
+            0.00019424450874794275,
+            0.01508495956659317,
+            -0.022075550630688667,
+            -0.012530704960227013,
+            0.01644345559179783,
+            -0.0037358624394983053,
+            0.017773648723959923,
+            -0.02227366529405117,
+            0.024198200553655624,
+            -0.004531855694949627,
+            -0.017179306596517563,
+            0.004698129836469889,
+            -0.007514177821576595,
+            0.022033097222447395,
+            -0.001578720286488533,
+            0.004857328720390797,
+            0.000821199850179255,
+            -0.0018015984678640962,
+            0.01349297259002924,
+            0.002826661104336381,
+            -0.0005120891146361828,
+            0.010507113300263882,
+            -0.0009702275274321437,
+            -0.0013496510218828917,
+            0.024509521201252937,
+            1.9374700059415773e-05,
+            0.005989408120512962,
+            0.0032936439383774996,
+            -0.009261825121939182,
+            -0.004429261200129986,
+            -0.004443412180989981,
+            0.0011842612875625491,
+            0.003877372248098254,
+            -0.007029506377875805,
+            -0.02384442463517189,
+            0.022471778094768524,
+            -0.008540124632418156,
+            0.01175947580486536,
+            0.011349096894264221,
+            0.003665107535198331,
+            0.006824316922575235,
+            -0.009771261364221573,
+            -0.017320817336440086,
+            0.0006633278680965304,
+            -0.01617458648979664,
+            0.006912760436534882,
+            -0.001656550681218505,
+            0.0020235921256244183,
+            0.009445788338780403,
+            -0.002570174168795347,
+            0.0015123875346034765,
+            0.011306644417345524,
+            0.011497682891786098,
+            0.003254728624597192,
+            0.004577846731990576,
+            -0.018721764907240868,
+            -0.020915169268846512,
+            0.005288933869451284,
+            0.013924578204751015,
+            0.004273600410670042,
+            0.0015309606678783894,
+            0.017603836953639984,
+            -0.006166295614093542,
+            -0.022967062890529633,
+            -0.010117961093783379,
+            -0.01584911346435547,
+            0.039396367967128754,
+            0.01800006441771984,
+            -0.02525952458381653,
+            -0.000530220044311136,
+            -0.009728807955980301,
+            0.010450509376823902,
+            -0.00404364662244916,
+            0.014575524255633354,
+            0.005476434715092182,
+            -0.02426895499229431,
+            -0.016797229647636414,
+            -0.010662773624062538,
+            -0.005257094278931618,
+            0.020080260932445526,
+            0.027014248073101044,
+            -0.004956385586410761,
+            0.0061026159673929214,
+            0.019584976136684418,
+            -0.010938717983663082,
+            0.009849091991782188,
+            0.020957622677087784,
+            0.0031698227394372225,
+            -0.007570781745016575,
+            -0.004553082399070263,
+            -0.016514210030436516,
+            0.024467067793011665,
+            0.014250051230192184,
+            -0.005104971118271351,
+            -0.02465103194117546,
+            0.004110863897949457,
+            -0.02428310550749302,
+            -0.01616043597459793,
+            -0.007301913108676672,
+            -0.014143918640911579,
+            -0.004931621253490448,
+            0.0027594438288360834,
+            -0.004836102016270161,
+            0.011278342455625534,
+            -0.0031592093873769045,
+            -0.004531855694949627,
+            -0.0041851564310491085,
+            -0.025584997609257698,
+            -0.02524537406861782,
+            0.007691065315157175,
+            -0.003005317412316799,
+            0.005596718285232782,
+            0.000650945701636374,
+            -0.013400991447269917,
+            -0.011589664034545422,
+            0.00970758218318224,
+            -0.007988235913217068,
+            0.012813725508749485,
+            0.0031149876303970814,
+            -0.009183995425701141,
+            0.0017193458043038845,
+            -0.0012311364989727736,
+            -0.0059752571396529675,
+            0.019924599677324295,
+            0.00916984397917986,
+            0.011724098585546017,
+            -0.03118879161775112,
+            -0.0021633331198245287,
+            0.008349086157977581,
+            0.01828308403491974,
+            -0.02244347706437111,
+            -0.008313708938658237,
+            0.005179263651371002,
+            -0.011228813789784908,
+            -0.013507124036550522,
+            -0.028952933847904205,
+            -0.0030460015404969454,
+            8.175515540642664e-05,
+            -0.018636859953403473,
+            -0.029886899515986443,
+            0.0009038947173394263,
+            -0.007860877551138401,
+            0.016882136464118958,
+            -0.006643891800194979,
+            0.01790100708603859,
+            0.00870286114513874,
+            0.0036173479165881872,
+            0.011554286815226078,
+            0.0025082635693252087,
+            -0.013266556896269321,
+            0.008426916785538197,
+            -0.02671707607805729,
+            -0.01516986545175314,
+            0.009792488068342209,
+            0.014214673079550266,
+            -0.0026126273442059755,
+            5.074458022136241e-05,
+            0.0058372849598526955,
+            0.024254804477095604,
+            0.013584954664111137,
+            0.012474101036787033,
+            0.003042463678866625,
+            -0.005303084850311279,
+            0.005568416323512793,
+            0.0227123461663723,
+            0.008603804744780064,
+            0.004011806566268206,
+            0.007592008449137211,
+            0.00389859895221889,
+            0.0009932229295372963,
+            -0.005943417549133301,
+            -0.00843399204313755,
+            -0.021537812426686287,
+            -0.008908051066100597,
+            0.019938750192523003,
+            -0.010422207415103912,
+            -0.03005671128630638,
+            0.0006023016758263111,
+            -0.0032777239102870226,
+            0.007988235913217068,
+            0.02595292218029499,
+            -0.014172220602631569,
+            0.004970536567270756,
+            0.005536576267331839,
+            -0.007910405285656452,
+            -0.00614860700443387,
+            0.022967062890529633,
+            0.02639160305261612,
+            -0.0034404604230076075,
+            -0.04800017178058624,
+            -0.01671232469379902,
+            0.004305440001189709,
+            -0.007768895477056503,
+            -0.034047290682792664,
+            -0.015424583107233047,
+            0.002757675014436245,
+            -0.03206615149974823,
+            0.010068432427942753,
+            -0.012948159128427505,
+            -0.01214155275374651,
+            0.020264223217964172,
+            0.03223596513271332,
+            0.005154499784111977,
+            0.00014847487909719348,
+            -0.0214670579880476,
+            -0.0021686397958546877,
+            -0.021382151171565056,
+            0.010174565017223358,
+            -0.008801918476819992,
+            -0.010584943927824497,
+            0.0059611061587929726,
+            -0.012191081419587135,
+            0.013266556896269321,
+            0.010797208175063133,
+            0.024778390303254128,
+            0.01584911346435547,
+            -0.01876421831548214,
+            -0.0024746551644057035,
+            -0.01108730398118496,
+            -0.0016450530383735895,
+            -0.009820790030062199,
+            0.006360871717333794,
+            -0.0031592093873769045,
+            0.011264191009104252,
+            0.015184016898274422,
+            0.004432798828929663,
+            0.00046786724124103785,
+            0.038151077926158905,
+            -2.4487851987942122e-05,
+            -0.0003418349369894713,
+            0.006803090218454599,
+            -0.015750056132674217,
+            0.008837295696139336,
+            0.012191081419587135,
+            0.005051904823631048,
+            -0.007698140572756529,
+            0.006739411037415266,
+            0.03099067695438862,
+            -0.01516986545175314,
+            -0.009014182724058628,
+            -0.023504801094532013,
+            0.007549555506557226,
+            -0.0004656561359297484,
+            0.029971804469823837,
+            0.020533092319965363,
+            -0.027509532868862152,
+            -0.016018925234675407,
+            -0.02179253101348877,
+            0.01267929095774889,
+            -0.011497682891786098,
+            -0.003704022616147995,
+            0.012240610085427761,
+            0.02367461286485195,
+            0.0018838511314243078,
+            -0.014455240219831467,
+            -0.008950503543019295,
+            0.008943428285419941,
+            0.014617976732552052,
+            0.003497064346447587,
+            0.007570781745016575,
+            0.00743634719401598,
+            0.008886824361979961,
+            -0.014801939949393272,
+            -0.012254760600626469,
+            0.00767691433429718,
+            -0.010202866978943348,
+            0.01794346049427986,
+            -0.009403335861861706,
+            0.016768928617239,
+            -0.004645063541829586,
+            -0.011964665725827217,
+            -0.0320095494389534,
+            -0.002292460994794965,
+            0.005037753842771053,
+            0.004627374932169914,
+            0.0034705311991274357,
+            -0.0022217060904949903,
+            -0.013160424306988716,
+            0.004673365969210863,
+            0.005073131527751684,
+            1.875283123808913e-05,
+            -0.008313708938658237,
+            -0.00479718716815114,
+            0.004252373706549406,
+            -0.001031253719702363,
+            -0.0073372903279960155,
+            -0.0059858704917132854,
+            0.018481198698282242,
+            0.0066509670577943325,
+            0.00800238735973835,
+            0.03234917297959328,
+            0.02974538877606392,
+            -0.0016353242099285126,
+            -0.004606148693710566,
+            -0.00023304918431676924,
+            -0.01589156687259674,
+            -0.0057205394841730595,
+            -0.023320838809013367,
+            0.022259512916207314,
+            -0.009665128774940968,
+            0.021820832043886185,
+            0.019457615911960602,
+            0.0039304387755692005,
+            -0.008391539566218853,
+            -0.0011586126638576388,
+            -0.007308988366276026,
+            -0.003560743760317564,
+            0.016641568392515182,
+            0.006742948666214943,
+            0.01226891204714775,
+            0.00030933189555071294,
+            -0.028401045128703117,
+            0.026405755430459976,
+            0.000579748535528779,
+            0.0034015451092272997,
+            0.008908051066100597,
+            0.014129767194390297,
+            0.013209952972829342,
+            -0.01235381793230772,
+            0.01768874190747738,
+            -0.03280200436711311,
+            -0.0033484790474176407,
+            -0.016500059515237808,
+            0.002076658420264721,
+            -0.0018520113080739975,
+            -0.0008145665633492172,
+            -0.05354736000299454,
+            0.0003842879377771169,
+            0.005313698202371597,
+            -0.015339677222073078,
+            0.029349161311984062,
+            -0.020589696243405342,
+            -0.017165156081318855,
+            -0.032858606427907944,
+            -0.010238244198262691,
+            0.018099121749401093,
+            0.005617944523692131,
+            0.00035178486723452806,
+            -0.02660386823117733,
+            0.015750056132674217,
+            -0.0047476585023105145,
+            -0.00743634719401598,
+            -0.029858596622943878,
+            -0.010938717983663082,
+            0.009403335861861706,
+            0.004485865123569965,
+            -0.0130825936794281,
+            0.009898620657622814,
+            -0.002890340518206358,
+            0.010867963545024395,
+            -0.0138467475771904,
+            -0.00420284504070878,
+            0.0010020672343671322,
+            -0.005363226868212223,
+            0.012615610845386982,
+            -0.030537845566868782,
+            0.011391550302505493,
+            -0.01352127455174923,
+            0.0008548084879294038,
+            -0.017858553677797318,
+            -0.027707645669579506,
+            0.0020288988016545773,
+            -0.016132133081555367,
+            0.039990708231925964,
+            0.010188715532422066,
+            -0.01481609046459198,
+            0.016995344310998917,
+            -0.002080196049064398,
+            0.017377421259880066,
+            0.0014955832157284021,
+            0.017023645341396332,
+            -0.011971740983426571,
+            0.021056678146123886,
+            -0.0041745430789887905,
+            0.02156611531972885,
+            0.00600002147257328,
+            0.034047290682792664,
+            -0.02033497765660286,
+            0.0070825726725161076,
+            0.003792466362938285,
+            0.005798369646072388,
+            -0.0026816134341061115,
+            0.021424604579806328,
+            0.017985913902521133,
+            0.019698183983564377,
+            0.0037110981065779924,
+            -0.024084990844130516,
+            -0.006137993652373552,
+            0.004867942072451115,
+            0.023858575150370598,
+            0.011150983162224293,
+            -0.016415152698755264,
+            -0.021056678146123886,
+            0.015226469375193119,
+            -0.007860877551138401,
+            0.006771250627934933,
+            -0.00851889792829752,
+            0.00037544354563578963,
+            -0.013259481638669968,
+            0.012233533896505833,
+            -0.012523629702627659,
+            0.0032335021533071995,
+            0.01043635793030262,
+            -0.027806703001260757,
+            0.01511326152831316,
+            -0.00371817359700799,
+            -0.014313730411231518,
+            0.023320838809013367,
+            -0.015877414494752884,
+            -0.013988257385790348,
+            0.005501199048012495,
+            0.008610880002379417,
+            0.01746232621371746,
+            -0.002761212643235922,
+            0.002669231267645955,
+            -0.020504789426922798,
+            -0.012445799075067043,
+            0.004591997712850571,
+            0.02210385352373123,
+            -0.015820810571312904,
+            0.012431648559868336,
+            -0.018099121749401093,
+            0.01364863384515047,
+            0.016202887520194054,
+            0.007149789482355118,
+            0.00573115237057209,
+            -0.007726442534476519,
+            -0.0011471149045974016,
+            0.032151058316230774,
+            -0.0029876285698264837,
+            0.02704254910349846,
+            -0.014915147796273232,
+            0.009615600109100342,
+            0.00518987700343132,
+            -0.013089669868350029,
+            -0.014589674770832062,
+            0.011108530685305595,
+            0.02428310550749302,
+            -0.009530694223940372,
+            -0.023660462349653244,
+            0.0018484735628589988,
+            0.014264201745390892,
+            0.010337301529943943,
+            0.009403335861861706,
+            -0.001997058978304267,
+            -0.0037358624394983053,
+            0.01484439242631197,
+            -0.0032918748911470175,
+            -0.0286557637155056,
+            -0.01633024774491787,
+            -0.004400959238409996,
+            -0.017490629106760025,
+            0.020745357498526573,
+            0.02367461286485195,
+            0.010507113300263882,
+            0.0037570889107882977,
+            0.005540114361792803,
+            0.018297234550118446,
+            0.01843874529004097,
+            0.013330236077308655,
+            0.009304278530180454,
+            0.012700516730546951,
+            -0.004319590982049704,
+            0.02800481766462326,
+            -0.010860888287425041,
+            -0.01046465989202261,
+            -0.015707602724432945,
+            0.0016636261716485023,
+            -0.005193414632230997,
+            -0.012431648559868336,
+            -0.0069481381215155125,
+            0.004369119182229042,
+            0.013330236077308655,
+            -0.012240610085427761,
+            -0.007149789482355118,
+            0.01016748882830143,
+            0.011971740983426571,
+            0.01403071079403162,
+            0.01719345711171627,
+            -0.01633024774491787,
+            -0.011278342455625534,
+            0.0009834941010922194,
+            -0.02844349853694439,
+            -0.0160047747194767,
+            -0.01767459139227867,
+            0.009580222889780998,
+            -0.014469391666352749,
+            0.03724541515111923,
+            0.028471799567341805,
+            0.0073514413088560104,
+            0.031613320112228394,
+            -0.028401045128703117,
+            0.00970758218318224,
+            -0.014398636296391487,
+            0.007132100872695446,
+            -0.004188694059848785,
+            0.003902136581018567,
+            0.002448122017085552,
+            0.020419884473085403,
+            0.00791748147457838,
+            0.014264201745390892,
+            0.008150972425937653,
+            -0.021934039890766144,
+            -0.00662620272487402,
+            -0.012559006921947002,
+            -0.016401002183556557,
+            -0.02595292218029499,
+            0.013719388283789158,
+            0.005087282508611679,
+            -0.003074303502216935,
+            -0.0002704166399780661,
+            0.006566061172634363,
+            -0.0013195802457630634,
+            0.010443433187901974,
+            -0.027608588337898254,
+            -0.01311089564114809,
+            0.0069269114173948765,
+            0.01449061743915081,
+            -0.02114158496260643,
+            0.0014637433923780918,
+            0.008900974877178669,
+            -0.004602610599249601,
+            0.035660505294799805,
+            -0.014674580655992031,
+            -0.008610880002379417,
+            0.0027806703001260757,
+            0.005062518175691366,
+            -0.017660440877079964,
+            -0.02319347858428955,
+            0.0070648835971951485,
+            0.011575513519346714,
+            0.010867963545024395,
+            -0.028570856899023056,
+            0.02087271586060524,
+            -0.0054339817725121975,
+            0.03832089155912399,
+            0.00401534466072917,
+            -0.013337312266230583,
+            0.003962278366088867,
+            0.0011860302183777094,
+            -0.012360893189907074,
+            0.002060738392174244,
+            -0.000340508297085762,
+            0.003700484987348318,
+            0.011228813789784908,
+            -0.014801939949393272,
+            -0.016967041417956352,
+            0.016910437494516373,
+            -0.011476456187665462,
+            -0.0035395172890275717,
+            -0.00943871308118105,
+            -0.016669871285557747,
+            0.007308988366276026,
+            0.025047259405255318,
+            0.0041745430789887905,
+            0.001242634141817689,
+            -0.011632117442786694,
+            -0.0038596836384385824,
+            -0.008575502783060074,
+            0.0011718792375177145,
+            -0.0207029040902853,
+            0.027396325021982193,
+            -0.01801421493291855,
+            0.002410975517705083,
+            0.021099131554365158,
+            0.008709936402738094,
+            -0.003944589756429195,
+            0.016316095367074013,
+            0.026363302022218704,
+            0.01205664686858654,
+            0.0031539027113467455,
+            -0.00902833417057991,
+            1.648646139074117e-05,
+            0.014957600273191929,
+            0.0015123875346034765,
+            -0.0015663381200283766,
+            0.0032140444964170456,
+            -0.006583749782294035,
+            -0.001859086798503995,
+            -0.017561383545398712,
+            0.005564878229051828,
+            0.02125479280948639,
+            -0.004132090136408806,
+            0.020476488396525383,
+            0.015509488992393017,
+            0.011625041253864765,
+            0.010726453736424446,
+            -0.0012390963966026902,
+            -0.002150951186195016,
+            -0.0021668709814548492,
+            -0.007393894251435995,
+            -0.018297234550118446,
+            0.006470541935414076,
+            0.0034493047278374434,
+            0.01822648011147976,
+            0.020504789426922798,
+            -0.02308027073740959,
+            0.010146263055503368,
+            0.014985902234911919,
+            -0.013011839240789413,
+            0.0235755555331707,
+            -0.02006610855460167,
+            0.0024605041835457087,
+            0.02156611531972885,
+            0.013337312266230583,
+            -0.014271277002990246,
+            -0.009219372645020485,
+            0.004662752617150545,
+            -0.01543873455375433,
+            0.000872939417604357,
+            -0.01478778850287199,
+            -0.010783057659864426,
+            0.014533070847392082,
+            -0.01490099634975195,
+            0.01229721400886774,
+            -0.006969364359974861,
+            0.0070967236533761024,
+            -0.017165156081318855,
+            0.015014204196631908,
+            0.008632106706500053,
+            0.01757553406059742,
+            0.0021279556676745415,
+            -0.02518877014517784,
+            -0.006290116813033819,
+            0.021169887855648994,
+            0.009509467519819736,
+            -0.02497650496661663,
+            0.019613277167081833,
+            0.0020094411447644234,
+            -0.019528372213244438,
+            -0.022401023656129837,
+            -0.012686366215348244,
+            -0.005055442452430725,
+            -0.017264213413000107,
+            -0.0011276573641225696,
+            -0.024792540818452835,
+            0.003972891718149185,
+            0.009106164798140526,
+            -0.011978816241025925,
+            -0.019273653626441956,
+            -0.0002315014135092497,
+            -0.01040805596858263,
+            0.006219361908733845,
+            -0.01132787112146616,
+            -0.019853845238685608,
+            0.0070436573587358,
+            0.012127402238547802,
+            -0.0027647505048662424,
+            -0.005646246485412121,
+            -0.02562745101749897,
+            -0.006919836159795523,
+            0.020179318264126778,
+            -0.003994117956608534,
+            -0.004974074196070433,
+            -0.01703779585659504,
+            0.01108730398118496,
+            0.011165134608745575,
+            0.020901018753647804,
+            0.010931642726063728,
+            0.005674548447132111,
+            0.013634482398629189,
+            -0.0029416377656161785,
+            -0.007156865205615759,
+            -9.645892532716971e-06,
+            0.004284213297069073,
+            0.0069233737885952,
+            0.0016636261716485023,
+            0.020235922187566757,
+            -0.01692458800971508,
+            0.018934030085802078,
+            -0.012290137819945812,
+            0.008794843219220638,
+            -0.01855195313692093,
+            -0.0003155229496769607,
+            -0.01067692507058382,
+            0.00015002264990471303,
+            -0.0038455326575785875,
+            -0.009544845670461655,
+            1.9595809135353193e-05,
+            0.00143986358307302,
+            0.008511822670698166,
+            -0.016783079132437706,
+            0.009615600109100342,
+            -0.010839661583304405,
+            -0.004308977629989386,
+            -0.019542522728443146,
+            0.017448175698518753,
+            0.013344387523829937,
+            0.007308988366276026,
+            -0.008278331719338894,
+            -0.006300730165094137,
+            -0.02282555401325226,
+            0.008582578040659428,
+            -0.0014637433923780918,
+            0.002214630600064993,
+            -0.028301987797021866,
+            -0.004358506295830011,
+            0.0024587351363152266,
+            0.020306676626205444,
+            0.0033661676570773125,
+            -0.010712302289903164,
+            0.008978805504739285,
+            -0.023377442732453346,
+            -0.0030159305315464735,
+            -0.023349139839410782,
+            -0.004075486212968826,
+            -0.005112046375870705,
+            0.010103809647262096,
+            0.008101443760097027,
+            0.01549533847719431,
+            -0.009898620657622814,
+            0.01478778850287199,
+            -0.004793649073690176,
+            0.012049571610987186,
+            -0.0012152165872976184,
+            0.01337976474314928,
+            -0.0003111007681582123,
+            -0.01505665760487318,
+            0.02158026583492756,
+            -0.008738238364458084,
+            0.001741456682793796,
+            -0.0052853962406516075,
+            0.0027346794959157705,
+            -0.017702894285321236,
+            -0.004553082399070263,
+            -0.014455240219831467,
+            0.01957082375884056,
+            -0.022584985941648483,
+            0.004153316840529442,
+            -0.005055442452430725,
+            -0.016075529158115387,
+            0.029434068128466606,
+            0.022896308451890945,
+            -0.0035483618266880512,
+            0.015070809051394463,
+            0.0017989451298490167,
+            0.0015061964513733983,
+            0.016202887520194054,
+            -0.015806660056114197,
+            -0.013025989755988121,
+            0.035066161304712296,
+            0.0176321379840374,
+            0.0004313841927796602,
+            -0.0023260696325451136,
+            -0.0037323245778679848,
+            0.0032193511724472046,
+            0.008023613132536411,
+            -0.010450509376823902,
+            -0.01141277700662613,
+            -0.017051948234438896,
+            0.009997677057981491,
+            -0.006155682262033224,
+            0.004446949809789658,
+            -0.00025891896802932024,
+            -7.617214578203857e-05,
+            -0.01516986545175314,
+            0.022584985941648483,
+            -0.017717044800519943,
+            -0.00602832343429327,
+            0.004687516950070858,
+            0.0207029040902853,
+            -0.02541518583893776,
+            0.01616043597459793,
+            -0.007107336539775133,
+            -0.013429293408989906,
+            0.012573158368468285,
+            -0.015127412974834442,
+            -0.0107335289940238,
+            -0.003175129182636738,
+            -0.008009462617337704,
+            -0.022754797711968422,
+            0.017929309979081154,
+            -0.013627407141029835,
+            0.015650998800992966,
+            -0.00287972716614604,
+            0.000562502013053745,
+            0.002057200763374567,
+            0.0032582662533968687,
+            0.02250008098781109,
+            0.030735958367586136,
+            0.010415131226181984,
+            -0.005465821363031864,
+            -0.021000074222683907,
+            0.012983537279069424,
+            0.0020589695777744055,
+            0.013202877715229988,
+            -0.002253545681014657,
+            -0.002135031158104539,
+            -0.016613267362117767,
+            -0.023122724145650864,
+            -0.0013010069960728288,
+            -0.01761798746883869,
+            -0.02552839368581772,
+            -0.01774534583091736,
+            -0.007047194987535477,
+            0.011349096894264221,
+            0.020405733957886696,
+            0.0009755341452546418,
+            -0.005384453106671572,
+            0.01559439580887556,
+            -0.006941062398254871,
+            0.002080196049064398,
+            -0.006594363134354353,
+            -0.002032436430454254,
+            -0.008200501091778278,
+            -0.021863285452127457,
+            0.029660483822226524,
+            -0.010103809647262096,
+            0.01141277700662613,
+            -0.005472897086292505,
+            -0.00965097825974226,
+            0.005575491581112146,
+            0.00900003220885992,
+            0.024849144741892815,
+            0.006452853325754404,
+            0.03141520544886589,
+            0.020151015371084213,
+            -0.018891576677560806,
+            -0.0006071660900488496,
+            0.022684043273329735,
+            -0.0037570889107882977,
+            -0.004889168310910463,
+            -0.0031786670442670584,
+            -0.017985913902521133,
+            -0.012007118202745914,
+            0.007839650847017765,
+            -0.013471745885908604,
+            -0.006636816076934338,
+            -0.00837738811969757,
+            0.018070818856358528,
+            -0.017773648723959923,
+            0.025698205456137657,
+            -0.006488230545073748,
+            0.005154499784111977,
+            0.031924642622470856,
+            0.014363259077072144,
+            0.019698183983564377,
+            -0.011794853024184704,
+            -0.031075581908226013,
+            -0.00683846790343523,
+            -0.007882103323936462,
+            0.009558996185660362,
+            -0.01062032114714384,
+            0.005204027984291315,
+            -0.02006610855460167,
+            -0.018948180601000786,
+            0.012863253243267536,
+            -0.017476476728916168,
+            0.013924578204751015,
+            -0.009955224581062794,
+            0.027226513251662254,
+            -0.001228483160957694,
+            0.0121698547154665,
+            -0.002614396158605814,
+            -0.017801949754357338,
+            0.01205664686858654,
+            0.004089637193828821,
+            -0.023943481966853142,
+            0.010337301529943943,
+            -0.001864393474534154,
+            -0.014037786051630974,
+            -0.018410442396998405,
+            -0.001228483160957694,
+            -0.005200490355491638,
+            -0.009658053517341614,
+            0.025457637384533882,
+            0.014101465232670307,
+            0.016231190413236618,
+            0.0035359796602278948,
+            0.0016335553955286741,
+            -0.011073152534663677,
+            0.009573147632181644,
+            -0.006803090218454599,
+            0.000612030504271388,
+            -0.0034404604230076075,
+            0.011893910355865955,
+            -0.004252373706549406,
+            -0.01543873455375433,
+            0.002324300818145275,
+            0.008561351336538792,
+            0.0143278818577528,
+            5.947839599684812e-05,
+            0.009587298147380352,
+            0.0009790719486773014,
+            -0.009764186106622219,
+            -0.00046388726332224905,
+            -0.0060884649865329266,
+            0.0069375247694551945,
+            0.02400008589029312,
+            -0.012516554445028305,
+            -0.01681138016283512,
+            0.014504768885672092,
+            0.004033033270388842,
+            -0.017589686438441277,
+            0.018367990851402283,
+            0.004103788174688816,
+            -0.022358570247888565,
+            0.0035412863362580538,
+            0.010662773624062538,
+            -0.009523618966341019,
+            -0.014469391666352749,
+            -0.0010339069413021207,
+            -0.015509488992393017,
+            0.03130199760198593,
+            0.00025073791039176285,
+            0.016910437494516373,
+            -0.01062032114714384,
+            -0.0025967073161154985,
+            -0.013309009373188019,
+            0.013103820383548737,
+            -0.015750056132674217,
+            -0.007273610681295395,
+            0.00035620704875327647,
+            -0.02298121340572834,
+            0.01741987280547619,
+            0.0013425755314528942,
+            0.00794578343629837,
+            0.003436922561377287,
+            -0.010556641966104507,
+            0.009820790030062199,
+            0.0018237093463540077,
+            0.006668656133115292,
+            -0.014915147796273232,
+            0.023618008941411972,
+            0.006413938011974096,
+            0.011235889047384262,
+            -0.01352127455174923,
+            -0.01546703651547432,
+            0.002253545681014657,
+            0.04134920611977577,
+            -0.01306136790663004,
+            0.0002593612007331103,
+            0.01562269777059555,
+            7.081024523358792e-05,
+            0.004156854469329119,
+            -0.0023755980655550957,
+            -0.0034493047278374434,
+            -0.034585028886795044,
+            0.005370302125811577,
+            -0.0006328147719614208,
+            0.008073141798377037,
+            -0.008992956951260567,
+            -0.005430444143712521,
+            -0.0040684109553694725,
+            -0.008625030517578125,
+            -0.00314859626814723,
+            -0.01478778850287199,
+            0.008363237604498863,
+            -0.014533070847392082,
+            -0.028910480439662933,
+            0.03166992589831352,
+            -0.006887996103614569,
+            -0.00533846253529191,
+            -0.021396303549408913,
+            0.007570781745016575,
+            -0.01370523776859045,
+            0.002663924591615796,
+            0.010967019945383072,
+            0.008865597657859325,
+            0.008738238364458084,
+            0.002762981690466404,
+            0.0009516543359495699,
+            -0.004255911335349083,
+            0.010273621417582035,
+            -0.00835616234689951,
+            -0.0005943417199887335,
+            -0.0014186371117830276,
+            -0.02725481428205967,
+            -0.01405901275575161,
+            -0.010592019185423851,
+            -0.007811348885297775,
+            -0.00435143057256937,
+            0.012608535587787628,
+            0.007174553815275431,
+            -0.0017485321732237935,
+            -0.022938761860132217,
+            -0.014745336025953293,
+            -0.0034103896468877792,
+            0.008349086157977581,
+            -0.01838214136660099,
+            0.01064862310886383,
+            0.0016830838285386562,
+            0.009976450353860855,
+            0.014292503707110882,
+            0.00302831269800663,
+            0.0016742395237088203,
+            0.003458149265497923,
+            0.014886845834553242,
+            0.012601460330188274,
+            -0.005801907740533352,
+            -0.021382151171565056,
+            -0.0024605041835457087,
+            -0.007068421691656113,
+            -0.0034687623847275972,
+            -0.012332591228187084,
+            0.03039633482694626,
+            0.0029805530793964863,
+            0.0176321379840374,
+            0.007832575589418411,
+            0.009573147632181644,
+            0.010507113300263882,
+            0.029377464205026627,
+            0.006049550138413906,
+            -0.0029929352458566427,
+            -0.008603804744780064,
+            -0.02141045406460762,
+            -0.013146273791790009,
+            0.009898620657622814,
+            0.013294858857989311,
+            0.03597182780504227,
+            -0.034896351397037506,
+            0.012162779457867146,
+            -0.011101454496383667,
+            -0.008264180272817612,
+            -0.008533049374818802,
+            0.0044717141427099705,
+            0.010068432427942753,
+            0.014370334334671497,
+            -0.0023402206134051085,
+            -0.01175947580486536,
+            0.009070787578821182,
+            0.023547254502773285,
+            -0.012983537279069424,
+            0.03016991913318634,
+            0.027325568720698357,
+            0.023066120222210884,
+            0.01761798746883869,
+            -0.012537780217826366,
+            0.026462359353899956,
+            -6.484030018327758e-05,
+            0.0006261814851313829,
+            -0.018084971234202385,
+            -0.00587973790243268,
+            -0.004294826649129391,
+            -0.004362043924629688,
+            0.013316085562109947,
+            -0.0025471788831055164,
+            -0.0008468485320918262,
+            0.015099111013114452,
+            -0.0037323245778679848,
+            -3.927453508367762e-05,
+            -0.0059752571396529675,
+            0.015127412974834442,
+            0.0002699744363781065,
+            0.019457615911960602,
+            -0.04270770028233528,
+            0.019726485013961792,
+            -0.021396303549408913,
+            -0.003102605463936925,
+            -0.0020890405867248774,
+            0.009523618966341019,
+            0.00764861237257719,
+            -0.007142714224755764,
+            -0.014334957115352154,
+            0.004524780437350273,
+            0.014066088013350964,
+            -0.0006726144347339869,
+            0.005080206785351038,
+            -0.0027541371528059244,
+            0.0017228835495188832,
+            -0.010089659132063389,
+            -0.005862049292773008,
+            5.842812606715597e-05,
+            -0.010570792481303215,
+            -0.008646257221698761,
+            -0.03232087194919586,
+            0.01062032114714384,
+            0.010209942236542702,
+            0.007294837385416031,
+            0.004252373706549406,
+            -0.022018946707248688,
+            -0.012318440712988377,
+            0.012948159128427505,
+            -0.01132787112146616,
+            -0.002570174168795347,
+            -0.0020005968399345875,
+            0.005387990735471249,
+            0.0004705205501522869,
+            0.010160413570702076,
+            -0.003472300246357918,
+            0.010938717983663082,
+            -0.014603826217353344,
+            0.018084971234202385,
+            0.03229256719350815,
+            -0.00533846253529191,
+            -0.019967053085565567,
+            0.000907432462554425,
+            -0.009077862836420536,
+            0.003813692834228277,
+            0.007641536649316549,
+            -0.021962342783808708,
+            -0.02173592709004879,
+            0.005087282508611679,
+            -0.013153349049389362,
+            -0.007783046457916498,
+            0.014830241911113262,
+            0.005221716593950987,
+            0.010839661583304405,
+            0.01751893013715744,
+            0.010889190249145031,
+            0.00840569008141756,
+            -0.026533113792538643,
+            0.0006734988419339061,
+            -0.012686366215348244,
+            -0.00590803986415267,
+            0.002239394700154662,
+            -0.0017591454088687897,
+            0.004591997712850571,
+            -0.00010115749319083989,
+            -0.011200511828064919,
+            -0.0007057808106765151,
+            0.010903340764343739,
+            0.027919910848140717,
+            -0.004146241117268801,
+            -0.0022570835426449776,
+            -0.014257126487791538,
+            0.0010100272484123707,
+            0.036481261253356934,
+            -0.012275987304747105,
+            -0.017434025183320045,
+            0.01757553406059742,
+            0.027452928945422173,
+            0.005954030901193619,
+            -0.0032529598101973534,
+            -0.0027364485431462526,
+            0.00039600671152584255,
+            0.011978816241025925,
+            0.007719367276877165,
+            -0.002692226553335786,
+            -0.002131493529304862,
+            0.004761809483170509,
+            0.035773713141679764,
+            -0.0015203474322333932,
+            -0.010330225341022015,
+            -0.024240652099251747,
+            0.010974096134305,
+            0.00277182599529624,
+            0.004754733759909868,
+            0.002313687466084957,
+            0.023504801094532013,
+            0.019655730575323105,
+            0.014858543872833252,
+            0.00470520555973053,
+            -0.009771261364221573,
+            0.03005671128630638,
+            -0.008172199130058289,
+            -0.004542469047009945,
+            -0.02838689461350441,
+            -0.005087282508611679,
+            -0.015650998800992966,
+            0.008929276838898659,
+            -0.0002221042668679729,
+            -0.005016527138650417,
+            -0.013422218151390553,
+            0.019726485013961792,
+            0.01064862310886383,
+            -0.011073152534663677,
+            0.012608535587787628,
+            -0.027070851996541023,
+            0.02346234768629074,
+            -0.03826428949832916,
+            -0.0071993181481957436,
+            -0.006421013735234737,
+            -0.010514188557863235,
+            -0.02627839520573616,
+            -0.0030353881884366274,
+            0.007372668012976646,
+            0.014250051230192184,
+            0.0022570835426449776,
+            0.0022818478755652905,
+            0.02282555401325226,
+            0.0016415152931585908,
+            -0.01508495956659317,
+            0.017985913902521133,
+            0.03492465242743492,
+            0.012926933355629444,
+            0.00155837822239846,
+            0.015990622341632843,
+            0.0027789014857262373,
+            0.0021633331198245287,
+            0.00788917951285839,
+            -0.0034493047278374434,
+            -0.007938707247376442,
+            0.008582578040659428,
+            0.006990591064095497,
+            -0.02763689123094082,
+            0.003824306186288595,
+            -0.00010408719390397891,
+            -0.009898620657622814,
+            -0.022429324686527252,
+            0.011568437330424786,
+            0.01913214474916458,
+            -0.004224071744829416,
+            -0.006771250627934933,
+            -0.0043337419629096985,
+            0.0006836698739789426,
+            0.006785401608794928,
+            -0.002412744564935565,
+            -0.000515184598043561,
+            -0.0015176940942183137,
+            -0.02368876338005066,
+            -0.010316074825823307,
+            0.013570803217589855,
+            -0.005175726022571325,
+            0.010358527302742004,
+            0.005242943298071623,
+            -0.018792519345879555,
+            -0.016938740387558937,
+            0.006852618884295225,
+            -0.016188737004995346,
+            0.01364863384515047,
+            0.00027948213391937315,
+            -0.0023172253277152777,
+            0.024962352588772774,
+            0.03594352304935455,
+            -0.013188726268708706,
+            0.005119122099131346,
+            -0.019386861473321915,
+            -0.0245378240942955,
+            -0.019004784524440765,
+            0.022613288834691048,
+            0.008306633681058884,
+            -0.018792519345879555,
+            0.012304289266467094,
+            -0.002752368338406086,
+            -0.008865597657859325,
+            -0.0066721937619149685,
+            -0.006845543161034584,
+            -0.008547200821340084,
+            0.020533092319965363,
+            -0.004935159347951412,
+            0.0038950610905885696,
+            0.019047237932682037,
+            0.04075486212968826,
+            -0.008292482234537601,
+            -0.01349297259002924,
+            0.0054339817725121975,
+            0.004878554958850145,
+            0.0018626245437189937,
+            -0.004814875777810812,
+            -0.008016537874937057,
+            -0.0185944065451622,
+            -0.01740572229027748,
+            -0.009615600109100342,
+            -0.0020041344687342644,
+            0.020122714340686798,
+            -0.007768895477056503,
+            -0.01354957651346922,
+            -0.0245378240942955,
+            -0.028188779950141907,
+            -0.006580212153494358,
+            0.02319347858428955,
+            0.017179306596517563,
+            -0.00013200224202591926,
+            -4.3337418901501223e-05,
+            -0.012042495422065258,
+            -0.01968403346836567,
+            0.015509488992393017,
+            0.00037876018905080855,
+            -0.01773119531571865,
+            -0.0065448349341750145,
+            -0.015919867902994156,
+            0.005787756759673357,
+            -0.0051368107087910175,
+            0.004574308637529612,
+            0.05057565122842789,
+            0.0005271245026960969,
+            0.008441068232059479,
+            -0.026575567200779915,
+            0.009106164798140526,
+            -0.01583496294915676,
+            -0.017377421259880066,
+            -0.00012702727690339088,
+            -0.022301966324448586,
+            -0.0030955299735069275,
+            -0.018792519345879555,
+            -0.007386818993836641,
+            0.008568426594138145,
+            0.009799563325941563,
+            -0.013896276243031025,
+            -0.013669860549271107,
+            -0.024693485349416733,
+            -0.0035872769076377153,
+            -0.001539805089123547,
+            0.009311353787779808,
+            0.007825499400496483,
+            -0.03693409264087677,
+            0.002536565763875842,
+            -0.0009507699287496507,
+            0.0005779796629212797,
+            0.01208494883030653,
+            -0.008667483925819397,
+            -0.003063690150156617,
+            -0.0214670579880476,
+            0.006452853325754404,
+            0.007103798910975456,
+            -0.013500047847628593,
+            0.012375044636428356,
+            -0.012863253243267536,
+            -0.007669838611036539,
+            -0.0005426021525636315,
+            0.03379257395863533,
+            0.011794853024184704,
+            -0.0025153392925858498,
+            -0.00654129683971405,
+            -0.011150983162224293,
+            0.02654726430773735,
+            -0.011349096894264221,
+            -0.026405755430459976,
+            0.028910480439662933,
+            0.013245330192148685,
+            0.0005974372616037726,
+            0.020151015371084213,
+            -0.0017936384538188577,
+            -0.01397410687059164,
+            0.009601449593901634,
+            0.004266524687409401,
+            0.012424572370946407,
+            -0.00014460546663030982,
+            0.0214670579880476,
+            0.0018555490532889962,
+            0.0027399861719459295,
+            -0.0036226543597877026,
+            0.004687516950070858,
+            -0.03093407303094864,
+            -0.0006549257086589932,
+            -0.003102605463936925,
+            -0.008398614823818207,
+            0.011207587085664272,
+            -0.002421588869765401,
+            0.006558985915035009,
+            -0.022188758477568626,
+            -0.028485950082540512,
+            0.011342021636664867,
+            -0.016231190413236618,
+            -0.021396303549408913,
+            0.015339677222073078,
+            -0.0007641536649316549,
+            0.00614860700443387,
+            -0.007443422917276621,
+            -0.023278385400772095,
+            -0.019386861473321915,
+            0.02064630016684532,
+            0.012212308123707771,
+            -0.01190098561346531,
+            0.012750045396387577,
+            -0.021113283932209015,
+            -0.012948159128427505,
+            -0.001894464367069304,
+            0.0090566361322999,
+            -0.0019864458590745926,
+            0.01325240544974804,
+            -0.012198156677186489,
+            -0.00334494118578732,
+            0.0031592093873769045,
+            -0.011589664034545422,
+            0.008158047683537006,
+            0.01978308893740177,
+            0.008193425834178925,
+            -0.009339655749499798,
+            0.003422771580517292,
+            -0.011922212317585945,
+            0.013500047847628593,
+            0.015537790954113007,
+            0.018311386927962303,
+            -0.006498843897134066,
+            0.0020412809681147337,
+            -0.01370523776859045,
+            0.0063962494023144245,
+            -0.009827865287661552,
+            -0.004878554958850145,
+            -0.0005333155859261751,
+            0.004921008367091417,
+            -0.011179285123944283,
+            -0.0025772498920559883,
+            -0.026830283924937248,
+            -0.0013823752524331212,
+            -0.020533092319965363,
+            0.005303084850311279,
+            0.023207630962133408,
+            -0.007514177821576595,
+            0.0006045127520337701,
+            -0.0039092120714485645,
+            0.014476466923952103,
+            0.020688753575086594,
+            0.00870286114513874,
+            0.00482548912987113,
+            -0.006081389728933573,
+            0.02179253101348877,
+            0.011603815481066704,
+            -0.017816102132201195,
+            0.00164239970035851,
+            -0.001509734196588397,
+            -0.0014336726162582636,
+            0.011625041253864765,
+            0.004443412180989981,
+            -0.020844414830207825,
+            0.0028602697420865297,
+            -0.010089659132063389,
+            0.007818424142897129,
+            0.014702882617712021,
+            -0.00031906069489195943,
+            -0.02486329711973667,
+            -0.009827865287661552,
+            0.021919889375567436,
+            -0.015070809051394463,
+            -0.0008291598060168326,
+            0.00764861237257719,
+            0.023264234885573387,
+            -0.024792540818452835,
+            -0.006831392180174589,
+            -0.0023455272894352674,
+            -0.003933976404368877,
+            -0.005193414632230997,
+            0.0010489424457773566,
+            0.0214670579880476,
+            0.004415110219269991,
+            -0.004234685096889734,
+            0.005724077112972736,
+            0.006470541935414076,
+            0.02325008250772953,
+            0.0040577976033091545,
+            0.028047269210219383,
+            -0.00943871308118105,
+            -0.0023809047415852547,
+            -0.0019988277927041054,
+            -4.775960405822843e-05,
+            0.0160047747194767,
+            -0.005430444143712521,
+            0.020320827141404152,
+            0.009445788338780403,
+            0.0036934094969183207,
+            0.016514210030436516,
+            0.02319347858428955,
+            -0.006767712999135256,
+            -0.004793649073690176,
+            0.019160445779561996,
+            0.0053667644970119,
+            -0.008858522400259972,
+            -0.02194819226861,
+            0.01141277700662613,
+            0.0014124460285529494,
+            -0.0006956097786314785,
+            0.012106175534427166,
+            0.0024375086650252342,
+            0.006300730165094137,
+            0.008172199130058289,
+            -0.016684021800756454,
+            0.004241760354489088,
+            -0.004698129836469889,
+            -0.01860855706036091,
+            0.010910416021943092,
+            -0.012092024087905884,
+            -5.201595922699198e-05,
+            -0.0010002984199672937,
+            0.014165145345032215,
+            -0.01703779585659504,
+            -0.0019103841623291373,
+            -0.0070719593204557896,
+            -1.3024719009990804e-05,
+            -0.0009162768837995827,
+            0.008320784196257591,
+            -0.006254739128053188,
+            -0.010938717983663082,
+            -0.010938717983663082,
+            0.027509532868862152,
+            -0.004991763271391392,
+            0.003916287794709206,
+            0.019330257549881935
+          ],
+          "embedding": [
+            0.007204997251390758,
+            -0.03173324616100502,
+            -0.020505790884701024,
+            0.0037451608231038273,
+            0.009817085391002229,
+            0.026883177166072393,
+            -0.016477826654817607,
+            0.06949311853773875,
+            -0.01224876667472692,
+            0.031952566873802865,
+            0.013633671951171586,
+            -0.005041437355104989,
+            0.02663392750069342,
+            -0.022099132865267587,
+            0.006665702000169926,
+            0.0030467129715919033,
+            0.007931802508415542,
+            0.009984477355413895,
+            -0.010362836464183985,
+            -0.03180848181251013,
+            -0.013542377333474449,
+            -0.013572345022577053,
+            -0.03979985300694044,
+            -0.007698412130292187,
+            -0.005953452234356532,
+            0.018890228917932694,
+            0.02217446994660065,
+            0.005158357501784182,
+            -0.05468694715835175,
+            0.00789994048345594,
+            0.017130184174786147,
+            -0.0026591475669416114,
+            -0.02081141846859738,
+            -0.008403928318906045,
+            -0.030925346724945978,
+            -0.03855023691059801,
+            0.038853354785097736,
+            0.01612011653434478,
+            -0.035635702395196575,
+            0.026266623757525934,
+            0.043450373922427316,
+            -0.007254334835408145,
+            -0.015376804869420366,
+            -0.0017276202867766907,
+            0.004650414207094434,
+            -0.002365514361535186,
+            0.011906139905776183,
+            -0.0005664194302009289,
+            -0.0191723447249157,
+            -0.025546124518037998,
+            0.01025005257280377,
+            -0.014436532079436437,
+            0.029277529346694118,
+            -0.0045414983847733344,
+            -0.02260315456913697,
+            0.01717918656248626,
+            0.013666132349359846,
+            0.057804090189845364,
+            0.056600576093100735,
+            0.051208022172527734,
+            0.03553760896972431,
+            0.02094599510567085,
+            -0.02438674472832137,
+            -0.0027797634437333432,
+            0.03614248430646102,
+            -0.026729308357113124,
+            -0.002251394534452815,
+            0.018929251842788906,
+            0.00689697483318378,
+            -0.026603606434115974,
+            -0.0541443728344337,
+            0.027164630314707807,
+            -0.004335272389193875,
+            -0.013967321193992958,
+            0.003323121377857836,
+            -0.0245295956412367,
+            0.024489335129134787,
+            0.06212059558640156,
+            0.009214058207162371,
+            -0.002082293435905329,
+            0.013659934201796686,
+            -0.003344037927706573,
+            -0.00689041317919184,
+            0.020235785390418023,
+            0.0212739457082623,
+            0.0016782346585238381,
+            -0.013484221664059555,
+            -0.06465582506604173,
+            0.011882797526607326,
+            -0.032970655964890155,
+            -0.04027700439305653,
+            0.025396847912528105,
+            0.013861157517716707,
+            0.00604596157583048,
+            -0.00744611407420936,
+            0.04038084298991919,
+            0.021507532290328625,
+            0.005983822806623904,
+            -0.012711110369915742,
+            0.02410138142963901,
+            -0.018372791517880082,
+            0.021016674128298246,
+            0.013387285956415756,
+            0.02039383525127615,
+            0.010700292312227496,
+            0.0053934660497349735,
+            -0.014807678014449691,
+            0.009422805142740376,
+            0.0414357625490679,
+            0.12206398267002624,
+            -0.010877509834324155,
+            -0.011079302503786522,
+            -0.04107714853756596,
+            0.006631397986165343,
+            -0.031914267104222384,
+            -0.0179965295457466,
+            -0.057530036803978266,
+            -0.0320488164246567,
+            0.008208202363320509,
+            -0.0020203090915064162,
+            0.03502055780919601,
+            0.04464599164916666,
+            0.019332152079287082,
+            0.017133943087431935,
+            -0.039885565008919065,
+            0.04703059988373572,
+            -0.029370696964304416,
+            0.0008494086835971101,
+            0.0014082189719987074,
+            -0.030973762582071615,
+            -0.006561157852024518,
+            0.011634653995510726,
+            -0.019896826887411713,
+            -0.034275921680252924,
+            0.026274465530039866,
+            -0.007728234125236442,
+            0.04814514040690998,
+            0.06550407296280117,
+            -0.02749051703615343,
+            -0.010041099502187074,
+            0.02019601967763537,
+            -0.01871701432558228,
+            0.008532829759864928,
+            -0.020164138072967613,
+            -0.045196925901010855,
+            -0.004523517988516156,
+            -0.002113469149506399,
+            0.07929650663710812,
+            -0.01111269708042164,
+            0.03358254444720164,
+            -0.019973382759721527,
+            -0.0007145592863325417,
+            -0.0011740862928390678,
+            0.02166182480535788,
+            0.004543228023398081,
+            -0.013377688428761516,
+            -0.01758250073647576,
+            0.009548292014596834,
+            -0.04753617258004735,
+            -0.007125744019792479,
+            -0.009930358894299303,
+            -0.007931672044235996,
+            -0.024561936537429774,
+            0.012979195012881556,
+            0.009209145250061479,
+            0.04586756662401364,
+            0.04311807301838248,
+            -0.0036241105681042675,
+            -0.06579256720896123,
+            0.04967191755416882,
+            0.015475886231466817,
+            0.03157651656768371,
+            0.02515941109300714,
+            0.01773554572598468,
+            0.005824418191900522,
+            0.016317635547562007,
+            0.007650872352957098,
+            0.010015166465492963,
+            -0.033203307142867844,
+            0.040272188258443496,
+            -0.013408423788025047,
+            0.01036004294342104,
+            0.012391622448703885,
+            0.04214857862444935,
+            0.03418032388760535,
+            0.015263018079889338,
+            0.018786704580603317,
+            0.02704685343985817,
+            0.015348111464412869,
+            0.0013031600434754704,
+            0.03241427528588285,
+            0.011689924942195664,
+            0.04544406114742012,
+            0.016186615914968734,
+            0.011624500626103958,
+            -0.008272953078864385,
+            -0.02363080189189375,
+            0.02784298119317616,
+            -0.010585980875564109,
+            0.01455013604305903,
+            0.01749228557615982,
+            -0.054742056843690476,
+            -0.005876213780171779,
+            -0.010556605098730201,
+            0.003967456397644932,
+            0.014663888851767762,
+            0.0015548476559947807,
+            -0.003858367677774835,
+            0.012504286824484335,
+            -0.025806882498628832,
+            -0.014216299453397403,
+            0.0031620924516417956,
+            -0.006448390918604307,
+            -0.02785577502346934,
+            -0.00766402687921159,
+            0.005275000635711884,
+            -0.0090448751942,
+            0.019702662630533,
+            0.007009091837467867,
+            -0.01847307100043082,
+            -0.0006774105206839772,
+            0.032952656101827446,
+            0.015900033044760432,
+            -0.02170121261931376,
+            -0.017964879081469612,
+            -0.005062849745084668,
+            -0.03171847232054206,
+            -0.038177398521606544,
+            0.029426291950046455,
+            -0.025687201683643784,
+            0.030158867757101797,
+            0.015780411479509624,
+            -0.007011484049169482,
+            -0.012114023533118393,
+            0.020661407942701628,
+            -0.025067530380881376,
+            0.009777190971413269,
+            -0.004160771288889298,
+            -0.028825369485727344,
+            0.008307290529469729,
+            -0.014415903546305744,
+            -0.007561235629544945,
+            -0.03201704135271719,
+            -0.015182869388110948,
+            -0.00501561450194676,
+            -0.0003863746169672397,
+            -0.01073403445816663,
+            -0.03321103354401307,
+            -0.002304357889348034,
+            0.017001213960120047,
+            0.00028222146729658097,
+            -0.04002882852650754,
+            0.014509588052058002,
+            -0.004150541038999649,
+            -0.034333837380045466,
+            0.0564137596224324,
+            0.03584898691608747,
+            -0.006830698606679202,
+            0.006121456445207273,
+            -0.04505674213649008,
+            0.04358467937307492,
+            0.010847859073780811,
+            -0.04918983713702805,
+            0.049190442935189385,
+            -0.00528960443684769,
+            0.01419195371129309,
+            0.040316024601793406,
+            0.004600843996974165,
+            -0.006780970176765864,
+            0.005206651850992678,
+            0.03397517915308201,
+            -0.010037223444627569,
+            -0.027377913828275806,
+            -0.03328136706915221,
+            0.02562568440972934,
+            -0.03485770844248482,
+            -0.020069979230687374,
+            -0.0052041408068332524,
+            0.029818953377223505,
+            0.028789460147370508,
+            0.004071274130619537,
+            0.01928375754300958,
+            0.0014970947734202088,
+            0.010502599795027616,
+            0.016914145702515263,
+            0.011942576738188502,
+            -0.014909953528043828,
+            -0.003432489584767786,
+            -0.05733768990013957,
+            0.0030710780388681002,
+            -0.02627280695622545,
+            0.007930906480371135,
+            -0.031227917075485834,
+            0.013132797714872345,
+            0.002514351081823354,
+            -0.007152021475569949,
+            -0.0066245205132327055,
+            -0.005279224195782313,
+            0.028512607222275155,
+            -0.00921960825771865,
+            -0.0359704105589433,
+            -0.021347597388353182,
+            0.011200044645812517,
+            -0.0004171294784927543,
+            -0.016784468873218786,
+            -0.0018746544944553878,
+            0.007749103417240419,
+            0.02390272097796833,
+            -0.005128540108845805,
+            0.006397849195755672,
+            0.01214922062611769,
+            0.01163371266250103,
+            0.012307186722130844,
+            -0.004982589662882946,
+            -0.016343724553037307,
+            0.019806887143115643,
+            0.013139657463054087,
+            0.0205801609265216,
+            -0.03767676007629612,
+            -0.0249879011550972,
+            0.05547392586363707,
+            0.005903313549319051,
+            -0.05568806298774435,
+            0.018827693208840233,
+            -0.007931178941307645,
+            0.008183992600768387,
+            -0.010319022946041053,
+            0.05111352384595183,
+            -0.024897203290807753,
+            -0.02096678205188722,
+            0.006436523335212348,
+            -0.04917084523640266,
+            -0.010459284852442706,
+            -0.011138558192296276,
+            0.014789370043397917,
+            -0.03105725978752438,
+            -0.004934745004818259,
+            0.015595801375817122,
+            -4.687585727052446e-05,
+            0.0028886067775750254,
+            -0.00850352807499054,
+            -0.0034885579171087815,
+            0.016339681487281293,
+            -0.0034050317046826813,
+            -0.008309570448487812,
+            -0.003615165868022203,
+            -0.027638409919916757,
+            0.023352910161586186,
+            0.026364511529991978,
+            0.001249936082895606,
+            0.001987948967948376,
+            0.01756629246122769,
+            0.004656075485833687,
+            -0.006752579977941239,
+            -0.013681608638716132,
+            -0.03405936202918687,
+            -0.0401867651078737,
+            0.022991180183813725,
+            -0.02318908140203705,
+            0.018299512436825923,
+            0.00904081847012441,
+            0.019697091729447287,
+            -0.019236130984371835,
+            0.015159917756323988,
+            -0.0387297415849285,
+            -0.018793498297161176,
+            -0.005256288213225733,
+            -0.010414697711744128,
+            0.0007039894599589104,
+            -0.01739645696111503,
+            -0.013728845328097506,
+            0.022709522955937343,
+            -0.022789243441781803,
+            -0.0036763802589970026,
+            -0.01429671717544519,
+            0.0023674126354206932,
+            0.049448298344058644,
+            -0.009926033487593551,
+            -0.023371285193090733,
+            0.020846792916650956,
+            -0.026996519120163966,
+            0.011246908457705283,
+            0.04091365159640201,
+            0.005469162850197613,
+            0.005640919201612091,
+            0.029575753750229278,
+            -0.0007860452837141593,
+            -0.0052805237594885516,
+            -0.009782745443933828,
+            0.021178503581190618,
+            -0.0009176512672832309,
+            -0.05245650420332918,
+            -0.007698364493410058,
+            -0.039027418406703554,
+            0.005448840777995117,
+            0.017581625031764185,
+            0.025522517097715556,
+            -0.020775492472580457,
+            -0.015906320383873178,
+            -0.000643669839811819,
+            -0.0086896084429604,
+            0.0029399233482444638,
+            0.014381240985647907,
+            0.021086693776922936,
+            0.002436575081260609,
+            -0.004913514868101904,
+            -0.011582101006213432,
+            0.014400297663309155,
+            0.0467181039341492,
+            -0.01722692180137712,
+            0.0011622224560709315,
+            -0.013036540368441797,
+            -0.06208454638917413,
+            -0.019160212331432766,
+            0.0351496273293468,
+            0.02429814055116254,
+            -0.009896662169762772,
+            -0.01647101145477158,
+            -0.002915821099802562,
+            -0.03557863734177486,
+            -0.01786433999615293,
+            -0.010699364747311715,
+            0.02390901423299708,
+            0.007213809581960646,
+            0.0005601364043388203,
+            0.00916614601011734,
+            0.014737491634516238,
+            0.019303599465763203,
+            0.008864535607492504,
+            -0.0575839606069681,
+            0.018051837786406608,
+            0.006315830080225406,
+            0.02525619013082567,
+            -0.015671924543113065,
+            0.003988943108210209,
+            -0.03179878281917948,
+            -0.029440641244429775,
+            0.040586322748158446,
+            -0.03418722657593859,
+            -0.030650542761032924,
+            0.006566745611226466,
+            0.009571089937153852,
+            0.004491158558315032,
+            -0.011352813042040581,
+            -0.02535505987721779,
+            -0.013381948772812133,
+            -0.030918676056735445,
+            -0.02192708443081135,
+            -0.005232667170255381,
+            -0.03342408726436631,
+            -0.07498634115550039,
+            -0.006245638452568239,
+            -0.049946896465756174,
+            -0.03900385398225758,
+            0.04458524125677807,
+            -0.006106489377959481,
+            0.02398410968928338,
+            -0.04615183642824994,
+            -0.01165382599292015,
+            -0.04890013373371541,
+            0.013624324077017377,
+            0.01476398305411351,
+            0.020202554298470666,
+            0.00014553149242532853,
+            0.036224316512987846,
+            0.00021498496267774312,
+            -0.05055208658013767,
+            -0.057482169168904106,
+            -0.003960451575823462,
+            -0.010217078509789989,
+            -0.005593738972150311,
+            -0.034090261748104665,
+            0.05835341009022349,
+            0.0116359661127526,
+            0.04203678184748103,
+            0.03447818335186531,
+            -0.006747564131379555,
+            -0.025756962754841885,
+            -0.028123416424745064,
+            -0.008073625379614702,
+            0.047544241458556825,
+            -0.029605671083240564,
+            0.008171081332993893,
+            -0.026007452205474395,
+            0.0067084589781962805,
+            -0.0070645448617940995,
+            -0.0012298101311792947,
+            0.012372286141738867,
+            0.006377319686763927,
+            -0.015024537877377194,
+            0.007934745664790507,
+            0.026041755081605937,
+            -0.007303731324541215,
+            0.0407000170257006,
+            -0.012990703097937777,
+            -0.033189084013863494,
+            -0.022525679964848666,
+            0.01657717698191603,
+            0.03440284885162039,
+            0.010230633738702899,
+            -0.008707808754455952,
+            -0.0363777020540216,
+            0.011184076269109602,
+            0.031774012431843524,
+            0.01504703222643543,
+            -0.022509833303978505,
+            0.025230974050270107,
+            0.017502662694813216,
+            -0.018700579139537456,
+            -0.03511734747709348,
+            0.015162183119327901,
+            -0.017711638132862063,
+            0.018190335050681287,
+            -0.01590174153215615,
+            0.011103110869700727,
+            0.0025189370696524936,
+            -0.00920305384736575,
+            0.01929652151514404,
+            -0.006196715938358087,
+            0.011426611271994685,
+            0.007058186102245829,
+            -0.018191352599117545,
+            0.0027202185072220937,
+            0.057310566886380025,
+            -0.000531639834386418,
+            0.01494975681909879,
+            0.03845692752400366,
+            -0.001971953300066856,
+            0.045554909803324765,
+            -0.02532332999916258,
+            -0.042616337063845953,
+            0.02633963974172818,
+            -0.02970705577683783,
+            -0.01624154998005974,
+            -0.02620993449678864,
+            -0.021490817420853802,
+            -0.004038424045768319,
+            0.009940887982818171,
+            -0.02692933845395842,
+            -0.023801570856681796,
+            -0.001948460540662095,
+            0.02290911497874141,
+            -0.03300532378029689,
+            0.006525488380791647,
+            0.007039306647666766,
+            -0.003685585408710752,
+            0.03183735852538168,
+            -0.027197064244321705,
+            0.006861928286688311,
+            0.011209869849682412,
+            -0.001094897323854524,
+            -0.005328676992017907,
+            -0.004007474089818478,
+            0.010796946231051235,
+            -0.030543173497906445,
+            -0.012879972141706173,
+            0.00935894527375688,
+            0.0017573071057611829,
+            -0.010050882212831222,
+            -0.03102132152986303,
+            -0.002915141291985554,
+            0.007634137769426993,
+            -0.011094093612593696,
+            0.01199609810193786,
+            -0.010458016758814742,
+            -0.04929347607956551,
+            0.010973132159729983,
+            0.028148117094907874,
+            -0.025504019123994758,
+            0.012470127357818321,
+            0.009182729379865986,
+            -0.016020133756226275,
+            0.016320111225714855,
+            -0.012051271446461976,
+            0.024098195821827578,
+            0.04317620443481784,
+            0.024592886889234396,
+            0.007727787037954994,
+            0.00771746776156926,
+            -0.0003147538723468374,
+            0.022572868970938624,
+            -0.016683340264320543,
+            0.013672085891597506,
+            0.020827021901755415,
+            0.02092309219721837,
+            0.0057130785714458055,
+            -0.003901728981013356,
+            -0.0036968415923910314,
+            0.002227024825579113,
+            -0.02106678364798051,
+            0.014578546755457675,
+            0.03128101864424301,
+            -0.005234627629617124,
+            -0.022394744692442837,
+            0.002430617645746389,
+            -0.015780389432069438,
+            4.7277885648917e-05,
+            0.013357803020920043,
+            0.013777651543690907,
+            -0.027713499926132385,
+            -0.0018999112228756628,
+            0.028364630573666852,
+            0.009146793705085586,
+            -0.009891297866910313,
+            -0.01644269527910728,
+            -0.0003903768506971414,
+            0.006512217314869576,
+            0.02408785204739448,
+            0.0018406499545367782,
+            -0.007360473750031044,
+            0.01484497577088404,
+            0.016723277095327158,
+            0.003629911330658209,
+            -0.0004820567356447713,
+            0.02394406451074397,
+            0.023794019615473026,
+            0.001632266615911801,
+            -0.02064379152040184,
+            -0.02755693521684878,
+            0.027335938400284726,
+            0.02769248863677192,
+            -0.006832353240623308,
+            0.027324973360715134,
+            0.003937207909967385,
+            0.013580154030133622,
+            0.02076818768224654,
+            -0.01784151382870951,
+            -0.017617903734311997,
+            0.0017316476451235418,
+            -0.009605294810027804,
+            -0.009829196696575069,
+            0.004891780008779742,
+            -0.011655402383771472,
+            -0.01124379942906224,
+            0.043205454954628145,
+            0.004713701851783307,
+            0.003560551284131962,
+            0.007894840447501912,
+            -0.018881214526749082,
+            -0.00945538175144614,
+            0.0019040259741796817,
+            0.029505863720301916,
+            -0.001400002868254808,
+            0.03315866433970851,
+            -0.00830542045504805,
+            0.008602408964961391,
+            -0.03202191233287069,
+            -0.006978289755114341,
+            0.0326499698728755,
+            0.015314305362524091,
+            0.015104436300213667,
+            0.0021257042489266787,
+            -0.0010840650756005028,
+            0.02434348355847526,
+            -0.007174433778817674,
+            -0.04839373063710409,
+            0.022696994450018916,
+            -0.004779511451775743,
+            0.023588216956749154,
+            0.0382733980620746,
+            -0.004232286553944853,
+            0.003815507826545929,
+            0.023531223314846965,
+            -0.01887650634213847,
+            0.015055403379782985,
+            -0.01918944132635701,
+            -0.017597900721414636,
+            0.02176630010543207,
+            -0.008896608357798277,
+            0.01732387427588561,
+            0.01764363756727683,
+            -0.012180043444768312,
+            -0.019003360499321047,
+            0.01229139850966224,
+            0.0025710091163655626,
+            -0.017075658234265562,
+            0.01797412486853335,
+            0.018372010308966135,
+            0.006723179570342314,
+            0.015417677746651683,
+            -0.00047634684109332134,
+            0.017608296628666153,
+            -0.001049984114818522,
+            -0.008416990841821452,
+            0.01827827907328822,
+            -0.009568717277044265,
+            -0.01658089563620308,
+            0.009808478422454319,
+            -0.05193977747210182,
+            0.010644335931025957,
+            0.0016781339270411392,
+            -0.01562285100294917,
+            0.00405036330963539,
+            0.007102724536527262,
+            0.005195069570261247,
+            0.0028773630901681767,
+            0.004610474981005765,
+            0.042921486858787584,
+            0.013213672397112759,
+            -0.0064030584930089415,
+            -0.010711457508461704,
+            -0.017480309425763254,
+            -0.027151012983864797,
+            -0.005204425551873271,
+            0.009927884476991038,
+            0.022601230152836282,
+            0.008046297278661014,
+            -0.0031858480011267704,
+            -0.02589628103025927,
+            0.019723430185137954,
+            0.03604835658149313,
+            0.062423099201768716,
+            0.010930268386637612,
+            0.02925564269324381,
+            0.02439517070384333,
+            0.023930432096006114,
+            0.013036828091946605,
+            0.006910311087503408,
+            -0.032995841694293995,
+            0.012813650252889403,
+            0.01333714655191133,
+            0.03678966264802025,
+            0.012571138353467183,
+            -0.05610151838497368,
+            3.0777966587107344e-05,
+            0.009791774904454618,
+            0.041959488802722524,
+            0.01956296954085397,
+            0.000155217569370113,
+            -0.007285602454264675,
+            0.031547870838711856,
+            -0.026518385135879206,
+            -0.012299593280363843,
+            -0.0026551378470968594,
+            -0.006190117573235779,
+            -0.003807907001161504,
+            -0.004991957475076049,
+            0.036983035940964804,
+            -0.015648565314743963,
+            -0.019256463332683493,
+            0.0004300362961407231,
+            -0.0015686065121077472,
+            0.0011311524136286674,
+            0.0032236641835276464,
+            0.013795345278976164,
+            0.015385029265915279,
+            -0.004800448639353721,
+            0.026514926384796363,
+            -0.008699453553651681,
+            0.022186817580012775,
+            -0.013427553742493343,
+            -0.011927259975371873,
+            0.017776228476138103,
+            -0.021243797085501972,
+            -0.01667280589187595,
+            -0.003688045116580565,
+            0.006189970874536939,
+            0.0020911968266404467,
+            0.0006097761986763103,
+            -0.004685468039463817,
+            -0.01011357666327168,
+            0.022631082517271542,
+            0.012398000826637277,
+            -0.006721588925828951,
+            0.018142966317821054,
+            -0.0021330638043572654,
+            -0.011869184647489572,
+            0.025073455250141923,
+            -0.0015392945266926425,
+            -0.0337085890598252,
+            0.01267302113143573,
+            -0.013438601165482187,
+            0.02673290895281225,
+            -0.009586762706902736,
+            -0.011208527168019762,
+            0.019144928964011168,
+            -0.0017591316941248897,
+            -0.005459150067129732,
+            -0.029230079356228025,
+            0.004679497471366242,
+            -0.011723195942933286,
+            0.015899377439148873,
+            0.0022054234777254627,
+            0.029187759252864872,
+            -0.018290694895057157,
+            0.00792533824593637,
+            0.011719198071061566,
+            0.014612375997487872,
+            0.005419943379355563,
+            0.010515265723144554,
+            -0.006893314445458116,
+            -0.04424495426064596,
+            5.8241472764594054e-05,
+            -0.005719143565673274,
+            0.0038220021965914754,
+            0.013106940982933836,
+            -0.01203955335201393,
+            0.011260626096963215,
+            -0.0279896016489392,
+            -0.0295228261396555,
+            0.036474811902902415,
+            -0.04900747926861378,
+            0.019261089742811332,
+            0.00014104646541594166,
+            -0.03057197731717424,
+            -0.04331886560073797,
+            -0.03232957159604667,
+            -0.019401402365464362,
+            0.004102283356333861,
+            0.0002610070704924167,
+            0.0012280372705500212,
+            -0.014025012323929194,
+            -0.014135990809256075,
+            0.013269444858490211,
+            0.00992155580941576,
+            0.011225162609266308,
+            -0.006545044517282357,
+            0.002368862850663672,
+            -0.02063183105457681,
+            -0.026981383275694142,
+            -0.02774194980145225,
+            0.013337303400678109,
+            -0.0025902548982556788,
+            -0.02794339093668326,
+            -0.025606606546251152,
+            -0.0029852498102770587,
+            -0.01906873442473352,
+            0.006508845592362198,
+            0.03268861529625085,
+            0.0355975143313428,
+            0.010066906387952277,
+            0.01690250467284985,
+            -0.017201544758038183,
+            0.048345309114140714,
+            -0.019031171948182872,
+            0.03188329748476941,
+            -0.0005845169448621133,
+            -0.024140202991373587,
+            0.00011597263700677706,
+            0.03203623528903764,
+            -0.019285565857567896,
+            -0.01332755507465857,
+            -0.010914187315481855,
+            -0.003374912271519852,
+            0.02419326154642231,
+            0.02443825785057301,
+            -0.0037923303809811746,
+            0.00022800399604730175,
+            0.015893265651968885,
+            0.003154527829977343,
+            0.007688529734165207,
+            0.030623379985627763,
+            0.014052282574575454,
+            0.01127235345371133,
+            -0.04164707563955819,
+            -0.011811587719246128,
+            -0.003763086694358232,
+            0.005909410979841885,
+            0.002913220159138154,
+            -0.002671613084376487,
+            0.0045765194162907885,
+            -0.007241703279856166,
+            -0.005855949422953439,
+            0.0016973541436777935,
+            -0.001576326424468957,
+            0.008393873711385013,
+            -0.0037217094191171083,
+            -0.03498879678492834,
+            -0.026096985039773694,
+            -0.021280075788049788,
+            0.006437497348154639,
+            0.0070214556591898375,
+            -0.00972721128392027,
+            0.01949553762684561,
+            -0.0029404319702066146,
+            0.000271037702827666,
+            -0.010734867909488861,
+            -0.0029784341515646075,
+            -0.01206464664516104,
+            -0.019544778160755956,
+            0.00035139045159739705,
+            0.013457601448003152,
+            0.00025947376767512006,
+            -0.03165522602526639,
+            0.011979811705146604,
+            0.006642849985850598,
+            0.013181162886675883,
+            -0.024499404540711422,
+            -0.02310714308893576,
+            -0.028028912907092744,
+            0.010570122662701183,
+            -0.0066866321204157725,
+            -0.024617571284817428,
+            -0.004097807033448707,
+            -0.0074576460628928256,
+            -0.0020417256533259887,
+            -0.02470953739335365,
+            -0.011408461668188081,
+            0.023537349682260345,
+            -0.006373652338136973,
+            0.01936338676879376,
+            -0.017636435070085086,
+            -0.017036091876814843,
+            -0.002198685044404603,
+            0.015584758061011048,
+            -0.038943923931820196,
+            -0.002076529978164001,
+            -0.008452513321945455,
+            0.006551055965075869,
+            -0.028840587857625097,
+            -0.007128279504360441,
+            0.0199397499488538,
+            0.012424319366271584,
+            -0.03518875279748495,
+            0.013596027106451115,
+            0.022191403935861113,
+            0.01218507466930854,
+            -0.0038551718784437783,
+            0.03910075115429626,
+            -0.0025215641203030336,
+            -0.005422688956956598,
+            0.016759888275451905,
+            0.04462538468871483,
+            0.023708097134660096,
+            -0.03152743757744915,
+            -0.008668774531128668,
+            -0.015224825785549947,
+            0.004102648962663789,
+            -0.025867795193966987,
+            0.004676262922216584,
+            -0.011739753544166856,
+            0.01645951757031186,
+            0.011253400919230348,
+            0.013958707917471253,
+            0.0376596076542735,
+            -0.005077354977296842,
+            0.02668910096392324,
+            0.0010332284250006117,
+            -0.00856723931523911,
+            0.0014431148305642308,
+            -0.01412579094241589,
+            0.029892393553325205,
+            0.02269599036836208,
+            0.014421409701746036,
+            0.029248195108770955,
+            -0.03588867295960255,
+            0.02458655544632528,
+            -0.0010013539341895672,
+            0.008368648583431747,
+            0.018150734970019018,
+            0.02628022565029391,
+            0.015423867373844644,
+            0.012477080028047823,
+            -0.02513018750675568,
+            -0.022180001014682045,
+            -0.0027550350630288997,
+            -0.031944843677988374,
+            -0.0014668093006914267,
+            -0.009789309083948147,
+            0.026386027599666254,
+            0.0046988411391149,
+            -0.019803160322638583,
+            -0.003931806845108628,
+            0.004482097428470392,
+            -0.00244660100774939,
+            -0.010441732346892935,
+            -0.004649318716095894,
+            -0.01869464992812615,
+            -0.017838643679294407,
+            0.023677878122153204,
+            -0.023958455640817666,
+            -0.025238725541146534,
+            0.014582798102040157,
+            0.0013223590277629197,
+            -0.002295641913710489,
+            0.0008238518461567685,
+            -0.02115533275282518,
+            -0.04214275933987015,
+            -0.027817083427786927,
+            0.0037588803464689412,
+            0.03565056002940463,
+            -0.026788865866928923,
+            0.014162550819567465,
+            -0.011336583458519,
+            0.019175861037658344,
+            -0.005127229083828615,
+            0.015531116410874209,
+            -0.018251290727445846,
+            -0.044630374763679266,
+            0.021833307809708984,
+            -0.012087009688208446,
+            0.02720742725750304,
+            0.021497809277370574,
+            -0.026937653159100156,
+            -0.013572733244188407,
+            0.01358214612023832,
+            0.0020408253706382332,
+            -0.009652962552750464,
+            -0.0063411929391088626,
+            -0.010533309167065193,
+            0.003901271588567942,
+            -0.04496345296706673,
+            -0.008761188928422701,
+            0.06948704056954039,
+            0.005916180375965137,
+            0.03157905519638486,
+            0.014668818318138927,
+            -0.028448496686381258,
+            -0.04008949536631331,
+            -0.00462253589823598,
+            -0.0003701522833020314,
+            -0.00024410762276269235,
+            -0.01850585249309217,
+            0.017699615637510956,
+            -0.002712069912576265,
+            -0.009767338815933488,
+            -0.013119464222297351,
+            -0.013743610418034631,
+            -0.030375365173543157,
+            -0.0030316327877227783,
+            -0.008632689470358149,
+            -0.006565457264917239,
+            -0.0328089275944709,
+            -0.011713996290006943,
+            0.029773910045828737,
+            0.02740100904353622,
+            -0.0061926915497348915,
+            0.00638737826122608,
+            -0.010244154924811975,
+            0.0010396203272677069,
+            -0.001264454445070139,
+            -0.004059969507786729,
+            0.008715730726831364,
+            0.0014767015415289076,
+            0.031074767756160294,
+            0.004761858344125113,
+            0.00987870172044607,
+            -0.059926117793699066,
+            -0.009294021263133636,
+            0.021188432871397475,
+            0.016602826531924377,
+            -0.009125784136292983,
+            0.013946188378030194,
+            -0.0017670794602549428,
+            -0.006837334179368295,
+            -0.028290055353601393,
+            0.024556541126233806,
+            0.04543639597357669,
+            0.0031311343860416473,
+            -0.011199639773247286,
+            0.014965677855595444,
+            0.0018248533506196013,
+            0.015507721118783542,
+            -0.011409402378400194,
+            -0.01338737990904486,
+            0.03063410115541065,
+            0.009305362935335817,
+            0.0025578631174065385,
+            0.012727648343848194,
+            -0.0038680554900795763,
+            -0.01173215289303824,
+            -0.01568862495293563,
+            -0.004101703129244437,
+            -0.03649200408657064,
+            -0.0022107849490954137,
+            0.009659780679763728,
+            0.0034091693454195806,
+            0.008848380026939594,
+            0.01842435611499587,
+            0.03546926214224266,
+            0.013478278853234764,
+            0.005970293629431872,
+            -0.0069197533761813455,
+            0.023196445250565058,
+            0.010403914879205351,
+            0.010017272564548626,
+            0.015076807790462658,
+            -0.029649260409273275,
+            -0.012737152920662164,
+            0.00373276777994811,
+            0.013239909450830258,
+            0.0001682544254742693,
+            0.02030652051051447,
+            0.014143961923579586,
+            0.0010147733405793957,
+            -0.012509446388074923,
+            0.0020437769941487626,
+            0.007405578085797006,
+            0.013458276946120546,
+            0.0006779114540819997,
+            -0.004250745498730309,
+            0.0009149988500358458,
+            -0.008355906072311008,
+            0.040391939952386505,
+            -0.01831218885700067,
+            -0.016793844108223176,
+            0.035064882870542116,
+            0.11416006426189942,
+            0.0010630856112440689,
+            -0.015660014600661857,
+            -0.017142534874931403,
+            -0.0086899013413391,
+            0.016611825572132997,
+            0.01609650977226868,
+            0.009780106186629793,
+            0.009622367309076686,
+            -0.000672926017804395,
+            0.024310949465350565,
+            0.02944698490225121,
+            -0.0037169531700178868,
+            -0.010990838766452194,
+            -0.036819646475041955,
+            0.022923962979125287,
+            -0.004596366586768123,
+            -0.007108041562635549,
+            -0.002604794313552557,
+            0.012641223813290853,
+            0.013668775645501733,
+            0.005125100040230003,
+            0.0077993841846437815,
+            -0.011500851674334447,
+            -0.017334458072700107,
+            0.0056819751688881774,
+            -0.015982234644378358,
+            -0.0076533013358874935,
+            -0.013127172654362694,
+            -0.011837346554842119,
+            -0.006562312362371076,
+            0.0029877420008976425,
+            -0.008572723854050838,
+            0.00854083969164562,
+            -0.010939639783598493,
+            0.016185642759084087,
+            -0.008052282332943615,
+            -0.0011992212936609377,
+            0.0010378646897070758,
+            -0.0025399515501741718,
+            0.010712865639618352,
+            0.01451357949543628,
+            -0.02521401163091652,
+            0.020836357346214075,
+            -0.008252813367155418,
+            0.0006409807249997337,
+            0.015338809580802425,
+            -0.04672526649678653,
+            0.022728876784937357,
+            0.005129225753457052,
+            -0.018372222092794564,
+            0.0066182736895667805,
+            -0.02048375780196863,
+            -0.00896382274306482,
+            0.0021738974344880794,
+            0.004491167150658828,
+            0.0034280799511763956,
+            0.015611133773752265,
+            0.018126362497548308,
+            0.00014383671280220997,
+            0.007927708693242186,
+            -0.006973076292062532,
+            0.008118313055163714,
+            -0.006791836664375275,
+            -0.01164567445668625,
+            -0.010894577039363014,
+            0.00014731542306100506,
+            -0.008865132658571428,
+            -0.011415696755798914,
+            -0.017922318360382588,
+            0.00358538001727405,
+            0.00289327637128695,
+            0.010406054711003452,
+            -0.007757607207626101,
+            -0.010994775479337082,
+            -0.011716096880261664,
+            -0.0013333474316266438,
+            -0.01231144419953554,
+            0.009792603083441917,
+            -0.02884953407490196,
+            0.0029097409399144306,
+            0.025277274938032477,
+            0.009263530315933738,
+            -0.028037993274175277,
+            -0.02532252134134532,
+            -0.0018596440554143638,
+            -0.013847737486669237,
+            0.009910107581119272,
+            0.013952962280235523,
+            -0.011807495289410774,
+            0.003896325037406752,
+            -0.02144921729633576,
+            -0.01628545945817568,
+            -0.0040148671532777005,
+            -0.016032493124255323,
+            0.004580041957001551,
+            -0.011375180364988195,
+            -0.007789996450583554,
+            -0.022041665680945066,
+            -0.005890065942573431,
+            -0.050455058269193866,
+            0.014401045842891128,
+            0.030219548461217365,
+            -0.004580747668649223,
+            -0.028684680969792875,
+            -0.015525870112797619,
+            0.011121401958547706,
+            -0.008879749268609245,
+            -0.019454243014778435,
+            -0.007839434838464604,
+            0.018908534797107714,
+            -0.0019283067432864216,
+            0.0024769961730655835,
+            -0.006732653593696035,
+            0.027604805021030646,
+            0.01366593691107021,
+            0.03740746059041736,
+            -0.014662268336336578,
+            0.02539666271785517,
+            0.02371809783638648,
+            -0.036126848124425154,
+            -0.007627445211958312,
+            0.03657481892016626,
+            0.01092515982671624,
+            0.0011046573579274704,
+            0.012350047525582187,
+            -0.006340802546498992,
+            -0.031330519599469066,
+            -0.0013691360532301728,
+            -0.0014676630169621005,
+            -0.023111265345539388,
+            0.027359543575867225,
+            0.018091458372134273,
+            0.014237156751820934,
+            0.0030993618578507172,
+            -0.0014037082734049385,
+            0.013257269860435359,
+            0.01021315143964527,
+            0.030885417052806473,
+            -0.003225065956381245,
+            -0.012825341172770057,
+            0.013476039084668826,
+            0.01852164843904835,
+            -0.009965495459579544,
+            0.012541079912270015,
+            -0.00824122613985912,
+            -0.018124134280975132,
+            0.03192263766222747,
+            0.0001484308749572035,
+            0.010888733498667343,
+            -0.010649515858317311,
+            -0.013082335214246402,
+            0.032614733822945205,
+            -0.00999075761269065,
+            0.004257643407198955,
+            -0.030723533031261387,
+            -0.003820776505926347,
+            -0.010471792928758869,
+            0.01737608180148319,
+            0.004597677757231315,
+            0.02095148515295301,
+            0.029253627700524396,
+            -0.011604450820008121,
+            -0.01996325509217509,
+            -0.001158626158150122,
+            -0.020583793738536985,
+            0.004168757459099133,
+            -0.03632338333261702,
+            -0.019373660230711246,
+            0.022259216603413172,
+            -0.010364620831688015,
+            -0.004751193403783363,
+            0.00444187509644351,
+            0.0008572202661850934,
+            0.014943779567410408,
+            -0.025408263453313398,
+            0.017755758370543148,
+            0.004627392860417132,
+            0.007402073071525438,
+            -0.024913329140362466,
+            0.013760771254234878,
+            -0.006087346663657887,
+            -0.03780469033982951,
+            0.005509388120163784,
+            0.03756141956381493,
+            0.02381019168406692,
+            -0.024580249929652097,
+            0.01807584361080568,
+            -0.03111039245576191,
+            0.023245542501125073,
+            -0.004868090951368273,
+            5.1544460403943004e-05,
+            0.00640235757127173,
+            -0.004698373089469779,
+            -0.00621383035264185,
+            -0.0025132415458431587,
+            -0.008746634331769936,
+            -0.01611801559493664,
+            -0.01193535623883546,
+            -0.03381789115349148,
+            -0.024119754887378317,
+            0.013881102852385332,
+            0.0019232668966222687,
+            -0.0095263729740736,
+            0.018652966857927805,
+            -0.03775912497511837,
+            -0.012022352025516115,
+            -0.023289377270709736,
+            0.023579672990954228,
+            -0.028605627815260902,
+            0.019235397553197357,
+            -0.025838655372560505,
+            -0.021458526449310157,
+            0.01508157885177254,
+            -0.010064396230734452,
+            -0.0016708624560165368,
+            -0.015042071642751917,
+            -0.03670785416972153,
+            -0.006267614265977843,
+            0.02298763967146357,
+            -0.012506444728541897,
+            0.00029871598584066644,
+            0.02671390618094565,
+            -0.003059865833779805,
+            -0.011103452976820622,
+            -0.0017382684168257506,
+            0.010923055147701782,
+            -0.02938686260440486,
+            0.014300097108092281,
+            0.0038571851420598074,
+            -0.0037614499297117365,
+            0.0010991440540809893,
+            -0.010108413486243776,
+            0.02127952161226619,
+            0.009142545227414532,
+            -0.0030260119510809205,
+            0.040589877142650374,
+            0.008677453889680373,
+            0.0017339508605300814,
+            0.008732950895530873,
+            0.01063352447960556,
+            0.006618635029823979,
+            -0.014986307792626314,
+            0.016905861015681957,
+            0.013101237900847744,
+            0.0002162791516784468,
+            0.019309343995079022,
+            0.0021263376627204313,
+            -0.024516267132902416,
+            0.013339486637573757,
+            -0.02311659751478777,
+            0.0020575982953961246,
+            0.02391695586495566,
+            -0.020440172706421535,
+            -0.001656132876701879,
+            -0.006304133667169861,
+            0.000543457661904495,
+            0.022135707414160034,
+            -0.0230903871986617,
+            -0.023444763940153112,
+            0.009604600033024832,
+            0.01721387169484476,
+            0.01827905276638783,
+            0.02824773106648473,
+            -0.007100625424382998,
+            0.01348510178676837,
+            0.0002604386361122316,
+            -0.011802178189607182,
+            0.007664157334475143,
+            0.0074175377458313594,
+            -0.0044008460682833875,
+            0.025491035291032547,
+            -0.0017015687016376205,
+            -0.006134010425966364,
+            -0.005379657499404915,
+            -0.01584573519366957,
+            0.0034386456145185213,
+            -0.013860742817569676,
+            -0.013355238174583658,
+            0.0040342831827141845,
+            -0.0208160950322342,
+            0.10904592104153767,
+            -0.0037038089210379766,
+            -0.019472012964045247,
+            0.0063554319446707044,
+            0.003716719884684827,
+            -0.003692786470980806,
+            -0.003353843210051379,
+            0.007788605566526459,
+            -0.011254480693190834,
+            -0.0045751956417398095,
+            -0.013976039724381875,
+            -0.001989262807706565,
+            0.010067374586433014,
+            0.009201983771978006,
+            -0.005902980676079493,
+            -0.02029714318363332,
+            0.013031676901335606,
+            -0.016970144750162643,
+            0.008874670945038486,
+            0.0008311255074953139,
+            0.00884167159995975,
+            0.001716603195908536,
+            0.01854951250316076,
+            -0.009310881405486074,
+            -0.01325242738760595,
+            0.029667589863813217,
+            -0.005616511284128333,
+            -0.0038179509972925954,
+            -0.0004690716809300012,
+            -0.005068974756744239,
+            0.0021204231125051274,
+            -0.027958274497127258,
+            0.022127088236172387,
+            -0.017271838492631593,
+            0.010267709087809265,
+            0.01316800560257366,
+            0.005599367128954229,
+            0.01817806970002991,
+            -0.007115594036834107,
+            -0.02627643123342057,
+            0.006542742528003751,
+            -0.003433874053082844,
+            0.006631427284537971,
+            0.02316295746074422,
+            -0.014573351469966297,
+            -0.033520864104177005,
+            -0.013707022143307006,
+            -0.01877574599842729,
+            -0.005992004412448576,
+            -0.0043698881826095435,
+            -0.01481352026460129,
+            0.006424121402963946,
+            0.01382938882387991,
+            0.018136021761515388,
+            0.0012697483790911433,
+            -0.011531136974861559,
+            0.00609187274277241,
+            0.01076133059240578,
+            0.008400887242930113,
+            -0.01533504932731406,
+            0.011439227832642668,
+            -0.028718487721012004,
+            0.01522697877548034,
+            -0.012452665599305595,
+            -0.024938748080573947,
+            -0.0022082450217489366,
+            0.017589399637694264,
+            0.0037824832255756763,
+            -0.004355445812154662,
+            -0.018778668474337727,
+            0.019825087149714933,
+            0.024692878851503508,
+            0.003883497401229232,
+            0.019470881143244197,
+            0.041834635328088364,
+            -0.003544509491805351,
+            0.027139164223766474,
+            -0.02129723143323512,
+            0.01461354860170749,
+            -0.03441729954693155,
+            0.001893713808284104,
+            -0.004222657393633789,
+            0.0013677766871517654,
+            -0.003811928806805752,
+            0.020142937759360904,
+            -0.009158202577174012,
+            0.0030638058587375543,
+            0.024854053046139163,
+            -0.018665378030958756,
+            0.00805903734779081,
+            0.010625429315145887,
+            -0.012856237251902233,
+            0.028535299711292306,
+            -0.006335094045840286,
+            0.00305422126851784,
+            -0.0297883488914759,
+            -0.0029192872993071216,
+            0.0026067044379754047,
+            -0.006097749614529602,
+            -0.00968023644800927,
+            -0.007433916539502931,
+            -0.006202274026852405,
+            0.0038354607448269555,
+            -0.021471361566683354,
+            0.003195646827154078,
+            -0.004390751282171024,
+            -0.003323447423321121,
+            0.02518759723554948,
+            0.025857746688368054,
+            0.016893707217894103,
+            0.012739170140289531,
+            0.008553625305650129,
+            0.019805351588748434,
+            0.01654964927067909,
+            0.011549329129728739,
+            -0.020026013396046632,
+            0.01150644479272541,
+            -0.004159611921476011,
+            -0.0022937297602724912,
+            0.014815466072709131,
+            -0.01572171840394285,
+            -0.02276187015864222,
+            0.024693009795978448,
+            -0.019725315346836404,
+            0.00998768979020729,
+            -1.1702210281215188e-06,
+            -0.011129745730507545,
+            -0.002591293237053776,
+            -0.018034357254990988,
+            0.00012070821244376275,
+            -0.023051045150566527,
+            -0.028465270110388716,
+            -0.00048603008831669926,
+            -0.000574546300981984,
+            -0.012992623897880395,
+            0.02064747165046032,
+            0.014820638761258362,
+            -0.024820174379739057,
+            0.02666518374464728,
+            -0.017430076811109155,
+            -0.01495051200726725,
+            -0.005861181182847845,
+            0.00100881727574559,
+            -0.0055223282819321295,
+            0.0034719859452138017,
+            -0.010180752654175722,
+            -0.009961678082468036,
+            -0.012134423047419183,
+            -0.028073672819317354,
+            0.011574683159318163,
+            0.016120674453124317,
+            -0.015043147648505721,
+            0.031834693135959095,
+            -0.008802805725050891,
+            -0.0018698230152007877,
+            0.016179742266521146,
+            0.005096232332051617,
+            -0.017187077376166815,
+            -0.018999569809271876,
+            0.012251308792871571,
+            0.013279090498325945,
+            -0.009588297039279112,
+            0.006630431200804809,
+            -0.014130176783207386,
+            -0.005092662672039885,
+            0.007020236623084732,
+            -0.005049363863784624,
+            0.0055051712211635915,
+            -0.00995314895304919,
+            0.0052743456988306485,
+            0.005841370337169303,
+            -0.0029434616355351975,
+            0.004677049987338193,
+            -0.018568941264514256,
+            0.020031253265629777,
+            0.0018725390936655962,
+            0.01332532735283085,
+            -0.03518855189467458,
+            -0.0056298723325832025,
+            -0.005369475318146738,
+            -0.004650364775945294,
+            -0.020833510091412353,
+            0.020445280915606132,
+            0.0159991245532085,
+            0.013202409908533162,
+            0.023094507005426774,
+            -0.014905666713082246,
+            0.01572052456975675,
+            -0.020945766915121455,
+            0.013114000985348841,
+            0.003349101658930482,
+            -0.019829605346106578,
+            -0.001423750749031516,
+            0.04557459007627153,
+            -0.005847231973329081,
+            0.011986110554869603,
+            0.0012709842140249691,
+            0.026758922819943853,
+            -0.0001635536762342011,
+            0.0065419519929442995,
+            0.005997692702992206,
+            -0.009935128108338126,
+            -0.0008057798606777003,
+            0.00865712443881613,
+            0.046254320605750854,
+            -0.01181132866321511,
+            0.008170688314239364,
+            0.013099051984057827,
+            0.0034971494510865624,
+            -0.0008386400616584293,
+            -0.015346506644390764,
+            -0.03067678541405639,
+            0.019305962963428017,
+            0.007736916644077731,
+            -0.012756525606221703,
+            0.008734673666457216,
+            0.015137771675476126,
+            0.012891824153867332,
+            0.003895886821685762,
+            0.02155154331885366,
+            -0.012761690789748027,
+            0.018528313792219413,
+            0.024663453412919027,
+            0.009077230086690327,
+            9.184526542175535e-05,
+            -0.000751529849753638,
+            0.006791927267880182,
+            -0.0004433479107716552,
+            -0.009832230053513483,
+            0.015858232096630655,
+            -0.014861341148074698,
+            0.017790924034108806,
+            2.7216806238946785e-05,
+            0.03744841066552559,
+            -0.015216207585985075,
+            -0.010138319613387346,
+            -0.007774325714088762,
+            -0.0016997034352120362,
+            0.003971564314593291,
+            0.0018686537276849653,
+            0.006051669054361943,
+            -0.0008319624938748146,
+            -0.017326561560776794,
+            0.01667737942301944,
+            0.005719808546053257,
+            -0.0007405637872110778,
+            0.013750752065593804,
+            0.0038732323799683453,
+            -0.00367645612814479,
+            0.023580906347714314,
+            0.006238541579764762,
+            -0.005235795826199542,
+            0.0028182323816636243,
+            -0.006443210286549713,
+            -0.016255209299456314,
+            -0.003061441395209219,
+            0.004597970789372988,
+            -0.0016590920570710628,
+            -0.011138261849069419,
+            -0.0019423173609729848,
+            0.020355534503272887,
+            -0.004612410336124759,
+            0.013346750536413418,
+            -0.005400060374036387,
+            0.006252154576727367,
+            0.010601459925435238,
+            -0.012346569079682625,
+            0.0014154620309235822,
+            0.010406076751218612,
+            0.0013983417564564663,
+            -0.007140180075225315,
+            0.0010544410919327675,
+            0.007757599672903709,
+            0.006505350216372801,
+            0.00483063937962671,
+            0.00836630436955631,
+            0.013284451514874498,
+            0.0033749897293287438,
+            -0.006243293993633622,
+            0.0013949831763016042,
+            -0.013269921385800992,
+            -0.006308639321247727,
+            0.0010906169976769569,
+            -0.0010152835709869353,
+            -0.02017142197515583,
+            0.010787630901414118,
+            0.005355784839945777,
+            0.007838485470991034,
+            -0.01706932891696025,
+            0.009576030158827565,
+            -0.0008144311254998525,
+            0.02829832828245172,
+            0.007407953684956539,
+            -0.02769403281764041,
+            0.014266001222243134,
+            -0.010492199740754951,
+            0.022575782293320284,
+            -0.0016349316593207844,
+            -0.005656505606143177,
+            0.008025514398322551,
+            -0.009634281579228592,
+            -0.005991026158213869,
+            -0.022366083313133788,
+            -0.025825209755160946,
+            0.016871087849192924,
+            0.013583173973358155,
+            0.021125252303971605,
+            0.00012555627355712715,
+            0.015347535950809997,
+            0.0020128011217859444,
+            0.015164099887550086,
+            0.0024544903332736167,
+            0.012507136277293888,
+            -0.011798050869794805,
+            0.0104828761281283,
+            -0.0137805761114781,
+            0.014620357150314097,
+            -0.005246639228547017,
+            -0.006986408834884282,
+            -0.008849580196616098,
+            -0.005442165099441538,
+            -0.020891551850452786,
+            -0.009155930286949747,
+            -0.0013590151948670868,
+            0.010665872513236615,
+            -0.0005386423407222223,
+            -0.0027671778465775897,
+            -0.03237311596935737,
+            -0.004875062533405715,
+            -0.006670982068843501,
+            -0.01942244110096111,
+            -0.0045975373348006885,
+            -0.026137783023267008,
+            -0.01657173843643047,
+            -0.0007784046498232684,
+            -0.015037306676062236,
+            -0.006889680260149007,
+            -0.02009705080543122,
+            -0.027192867101205106,
+            3.0161964469866572e-05,
+            -0.009647644858034833,
+            -0.005380165957043753,
+            0.013815092649296373,
+            0.011902366689169847,
+            -0.01572592170758115,
+            -0.022321702398924333,
+            -0.01088937403483412,
+            -0.0165754067023749,
+            0.01646435743868707,
+            0.009914938007210463,
+            0.018834863207293908,
+            -0.003724245912768107,
+            0.005643505694722323,
+            0.0009581443359000331,
+            0.02327704258596552,
+            -0.02532422120223976,
+            -0.028694220894404807,
+            0.0019298778426469338,
+            0.0025970175688368323,
+            0.006086164866310174,
+            -0.009414665394188936,
+            0.001200390869257535,
+            -0.007782899559317819,
+            -0.0228264209954853,
+            -0.00898519580978142,
+            0.008156835270668449,
+            -0.010036630164522355,
+            0.02230664068547531,
+            0.012332230358869292,
+            0.013699197379189934,
+            0.010593587938184292,
+            0.01576174456497367,
+            -0.0001654241831584687,
+            0.014471905257729428,
+            -0.008675915624727887,
+            -0.0031070346099204425,
+            0.0038358404385932777,
+            0.005878417037187274,
+            -0.00625022998235275,
+            0.028077114529659795,
+            -0.0011836453716817525,
+            0.0002962695480296182,
+            0.014405782572641941,
+            0.02697667559474498,
+            -0.004710313427099278,
+            0.0210512043261309,
+            -0.002393165025068831,
+            -0.006135816286212752,
+            -0.008429776018235104,
+            0.008607617299702318,
+            -0.009529291136733401,
+            0.011133497451942684,
+            0.019864404836361926,
+            -0.014041315457579166,
+            -0.007351413712676263,
+            0.02950254974910368,
+            -0.014564388091367897,
+            -0.01825317106331622,
+            0.02325964952838705,
+            -0.007414295676409368,
+            -0.0039042587057605484,
+            0.00377144546378926,
+            0.007169856503233239,
+            -0.004036744966949995,
+            -0.012928341573802416,
+            0.016090527447479415,
+            -0.00011159893712484579,
+            0.014146137810890391,
+            -0.0004655839804281017,
+            -0.006372231942983921,
+            -0.010047382825720416,
+            0.002373224247019062,
+            0.026818813126567614,
+            0.013483627627077692,
+            -0.017895710143841333,
+            -0.013785856764479817,
+            0.010826610341182479,
+            -0.01538893818197323,
+            -0.004425206001945227,
+            0.02155595194812746,
+            0.0028768153668128553,
+            -0.020543027467698483,
+            0.008028975884363618,
+            0.0003835165718138906,
+            0.00037046947047892953,
+            -0.00029363296913641716,
+            0.030135739208689217,
+            0.001343516577762509,
+            0.00018948658255495327,
+            -0.00875053830853609,
+            0.008606573235959595,
+            -0.01280305089741873,
+            0.012786478836532904,
+            -0.02437993139337839,
+            0.007648181775166282,
+            -0.00629541989541625,
+            0.004579605315599212,
+            0.016658386927051592,
+            0.01218098627022323,
+            0.015031618813778413,
+            0.008237200519537417,
+            -0.008942965899756683,
+            0.0009093265012901892,
+            -0.025302175262520477,
+            -0.001333639155622843,
+            -0.017971876771137367,
+            -0.02036433247634014,
+            0.0002463842837918767,
+            0.022144712689630985,
+            0.018559482073124734,
+            0.011579840028464503,
+            -0.008657014440886656,
+            0.0166696911919226,
+            -2.9834620942882283e-05,
+            0.011845827764518911,
+            -0.00037901176785495324,
+            -0.006138374154553732,
+            0.004711058953619512,
+            -0.003622694588720417,
+            0.013542405259906128,
+            -0.014730567462694827,
+            0.012076044672004906,
+            0.01690223095263866,
+            -0.009108632851807137,
+            -0.008946251274440141,
+            -0.013992423874475428,
+            0.00048381889145447367,
+            -0.015909599902931202,
+            0.035489252674454214,
+            0.02252772349379229,
+            -0.01210464970923921,
+            -0.010469120693943236,
+            -0.014164999313775756,
+            0.019496048806752753,
+            0.010427539351244087,
+            0.0028018908451425276,
+            0.017091768406027965,
+            0.021395638416767112,
+            0.008817247593404623,
+            -0.006081063711583319,
+            -0.0021216503501144983,
+            -0.003781810594530787,
+            0.020143098218482827,
+            -0.003078172339607037,
+            -0.011350171884507555,
+            -0.012516379645909518,
+            0.005766102930704529,
+            -0.01626433668691071,
+            0.0019294643928566657,
+            0.008641558521764407,
+            -0.015057688533773374,
+            0.019407986696657772,
+            -0.028126197703647868,
+            0.012364374199657305,
+            -0.0044278434183731225,
+            -0.007251147925369614,
+            -0.03496220945653279,
+            0.006389489026974697,
+            0.008941259259632024,
+            0.003941752277454314,
+            0.023003838655784314,
+            -0.024528514847470928,
+            -0.018361931455105486,
+            0.0037791061321962543,
+            0.00047162060848090104,
+            -0.00711894577052754,
+            0.0010138871688747137,
+            0.004128392769213672,
+            -0.00674546985015164,
+            0.0017307097443508704,
+            0.0033354800298095546,
+            -0.002829898077850263,
+            0.013252947102333348,
+            0.0081724269619999,
+            -0.021449979263734438,
+            0.001368791856860285,
+            0.01590656734732547,
+            0.01452008402429858,
+            -0.02391847163018108,
+            0.006331397860689212,
+            -0.02064812934794864,
+            -0.000631340242718475,
+            -0.031272866000888556,
+            0.016859793097564958,
+            -0.0069008624795683276,
+            0.01899718801899956,
+            -0.0015377436942771224,
+            -0.0065810685260109004,
+            -0.012822620655323715,
+            0.02137604937338179,
+            -0.0019121137078344135,
+            -0.0025598772127157477,
+            0.015678293057066646,
+            -0.025700865215693987,
+            0.017825186616033165,
+            -0.0026230848396158204,
+            -0.017342967698019966,
+            0.017199139867162275,
+            0.004536019178633225,
+            0.007224666811309087,
+            -0.023505589665315567,
+            0.0007208041198255396,
+            0.003900164485347562,
+            -0.017250324025622685,
+            0.004470862192961212,
+            -0.030807400122398566,
+            0.0006579147990327597,
+            -0.013222907940559794,
+            -0.0020514994691204847,
+            -0.0046763235724575445,
+            -0.014050035576183385,
+            -0.039181147615786484,
+            -0.005429639279537522,
+            0.016065742005148733,
+            -0.0009029636981539927,
+            0.01800715198737081,
+            -0.002730931639143803,
+            0.002947680833843522,
+            0.004912692811020822,
+            0.00818378920043427,
+            0.006915895602313347,
+            0.02274116906914758,
+            -0.0057854922886970365,
+            0.00471999900678522,
+            0.007058014277137233,
+            0.014638081356210031,
+            -0.01719565951201012,
+            -0.015170377598372723,
+            -0.016015921063369496,
+            -0.006351213642269936,
+            0.01739665344303718,
+            -0.017488885259028475,
+            0.013303332445465686,
+            -0.00429428898520701,
+            0.014825338166443021,
+            -0.008686427208690176,
+            -0.01575945086724703,
+            0.0028564767084934386,
+            -0.007570417028925067,
+            0.001007670647768344,
+            -0.02323908008484089,
+            0.02019729564706863,
+            -0.00847581038203716,
+            0.010508144473197669,
+            0.01729824052022313,
+            -0.021364529179605813,
+            -0.009742851959432839,
+            -0.014342281441152413,
+            0.029896210049029176,
+            0.013925163184699139,
+            -0.009392252140772913,
+            0.003927230832659267,
+            0.008873561269230905,
+            0.011529536139797032,
+            -0.012209070214071445,
+            -0.0027679974125029362,
+            0.002411914349172495,
+            -0.0062056207925691356,
+            -0.002025687569076897,
+            0.03090134105002816,
+            0.0038285457265343245,
+            0.024877474647546086,
+            -0.013214997316123852,
+            -0.0033763842539861907,
+            0.00023379106197363714,
+            0.005134764372494098,
+            0.01183867975321486,
+            -0.0017966997954747142,
+            -0.008029791503028367,
+            -0.01179500476055557,
+            0.02599911330072515,
+            -0.021852083993691557,
+            0.007024505421035273,
+            0.026784024996764876,
+            0.001390202993990949,
+            0.0029956956045771046,
+            -0.013131691460546865,
+            -0.006298736864509989,
+            0.02523640664202355,
+            0.010030824059858138,
+            0.0041523782603032225,
+            -0.026465819482580182,
+            0.0032128899272377593,
+            -0.0061901409332948115,
+            -0.008681758126710213,
+            0.009866461769956222,
+            -0.0028095454857224164,
+            0.01188012041754054,
+            -0.012191996383095982,
+            0.028459939543805705,
+            0.010711502132045176,
+            0.01047154118931583,
+            0.025888652452698974,
+            -0.02079614305315134,
+            0.0042203106489994615,
+            -0.010829796140534066,
+            0.0006773493086316061,
+            0.005596640511307504,
+            -0.027510562413905437,
+            0.001969898603871003,
+            0.010390866563404346,
+            -0.005642706366801436,
+            -0.01882994179544761,
+            0.01187970346127799,
+            -0.019814385265412387,
+            0.024479579275617887,
+            0.003231341626501863,
+            0.020056914356101588,
+            -0.0053276448188067154,
+            0.009834685132275868,
+            0.0002742942778577165,
+            0.0031361331890882127,
+            -0.0008074598092128465,
+            0.009009012303347016,
+            0.005786777410144894,
+            0.013719989127068968,
+            -0.025528370602188686,
+            0.006821572597883499,
+            -0.0029197127345352626,
+            -0.030539872624857384,
+            -0.013383097680839732,
+            0.010291485622655026,
+            0.012218533188117704,
+            -0.009144425807890594,
+            -0.010378679518611277,
+            0.0030042579827305603,
+            0.005748243116238633,
+            -0.006692403885450407,
+            0.006913672151000981,
+            0.01629516912493545,
+            -0.002533517074771728,
+            -0.0019083178803585312,
+            -0.0035007340321378893,
+            -0.01154609935927537,
+            -0.010016998439072824,
+            0.023193275321990937,
+            -0.00482990342915347,
+            0.017943025772239603,
+            0.025873224580389702,
+            0.0039876567755322,
+            -0.0057577932736143506,
+            -0.005460363755206188,
+            0.01478435999841998,
+            0.014195457012279476,
+            0.004669295894868759,
+            -0.009702220679239768,
+            0.007974305230005956,
+            -0.007760086293456636,
+            0.008375077126777163,
+            -0.020260917110786147,
+            0.002199401871514877,
+            0.0026191373298767766,
+            -0.0022089262577237776,
+            -0.015213807046175918,
+            -0.01956788452420682,
+            -0.021898368819136302,
+            0.012634659463647929,
+            0.00034643102239488474,
+            -0.009241772775582378,
+            -0.013645250997151429,
+            0.0016665106990117016,
+            0.0025630500346914592,
+            0.0002095521560180936,
+            -0.007783022628694597,
+            0.009546147813008391,
+            -0.01265287317178631,
+            0.007445223311736028,
+            -0.004007029181436962,
+            0.0051846833906574515,
+            -0.0016916014415925766,
+            -0.016413938859402438,
+            -0.0210868129634149,
+            0.01920794362224444,
+            -0.012897042227841011,
+            0.02269303737601583,
+            -0.0014935878147854765,
+            3.062680855559304e-05,
+            0.0033132626755401546,
+            0.007274487651476732,
+            0.012110042834251237,
+            0.011426424896797977,
+            0.004307086698938411,
+            0.008206932254416481,
+            0.004673427176334148,
+            -0.013124571552764586,
+            0.004324963759928715,
+            0.0039216033705804084,
+            -0.010449887922072297,
+            0.005298323150690876,
+            -0.0010196552573491,
+            0.006128327172621373,
+            0.012167492341907742,
+            0.01923525204293908,
+            0.01908107550907106,
+            -0.0012031208519043073,
+            -0.0010601912677170903,
+            0.014474374858125607,
+            0.004540954805526089,
+            -0.004883537463424679,
+            -0.002772988629590886,
+            -0.013363713078069854,
+            -0.002572403646338979,
+            0.011541515690646252,
+            -0.02376161747414617,
+            -0.018215231053002656,
+            0.021831219299143226,
+            -0.004991882316041097,
+            0.033796174834149546,
+            -0.013140970283091536,
+            -0.016197177724633614,
+            -0.00464190076219332,
+            0.0026389653040644607,
+            -0.013788049032378498,
+            -0.010300859744205445,
+            0.010545541278382187,
+            0.013848738866656793,
+            0.009903590151120531,
+            -0.0005984085581408575,
+            0.02922816576443645,
+            0.0019900932217491907,
+            0.02245395592422119,
+            0.0018978407151245295,
+            -0.007029294098689854,
+            -0.01440575120349887,
+            0.010840472832356266,
+            0.005529335652955532,
+            0.009050748012161991,
+            0.017791269808484802,
+            0.014502057454389247,
+            0.006265567405788131,
+            0.013928705812292101,
+            0.0012903002372833693,
+            0.0010858377120770652,
+            -0.002355229463060164,
+            -0.01692711767136341,
+            -0.01456786846097011,
+            -0.01627640642218236,
+            0.003873864312554098,
+            0.012620350492241247,
+            0.00525890570023481,
+            0.0007393789113866752,
+            -0.007153210751899716,
+            0.009254566395058412,
+            0.0065087881512835375,
+            -0.012279152412209152,
+            -0.016530068283509593,
+            0.004060671955726967,
+            -0.02210570272789274,
+            0.007849467899009437,
+            0.01554233481902349,
+            0.00219885032769439,
+            -0.002655449524219968,
+            0.0006843524304842362,
+            0.006272944169764281,
+            0.005531927368102191,
+            -0.0036710328198476905,
+            -0.00544113514775027,
+            0.011264661763762068,
+            -0.006379654234440906,
+            0.009537150372938296,
+            -0.00290031548037575,
+            0.03718190949929003,
+            -0.011976080887998194,
+            0.012120408392610773,
+            -0.001751472085435711,
+            0.008481703962951994,
+            0.008067669857949423,
+            0.010772687289903202,
+            -0.003805089040741351,
+            -0.0022164543131735167,
+            -0.008753625032781523,
+            0.006751953442322185,
+            -0.01582177621255226,
+            0.013631301238130705,
+            -0.00030685726238985963,
+            -0.010728958775974582,
+            0.01055472059121978,
+            0.006788316621149835,
+            0.004413382316882087,
+            -0.003262024961860459,
+            -0.01972283203505511,
+            0.014937729522976615,
+            0.0018799533869990667,
+            0.019247008681353865,
+            -0.023276106189004046,
+            0.029043727846427652,
+            -0.02019024213343764,
+            0.023083643395685142,
+            0.01865875504942157,
+            0.001179436167200093,
+            -0.02070329911663634,
+            -0.007618131527214554,
+            0.014871709122115308,
+            -0.02659818889735726,
+            -0.006234719259825804,
+            0.005986829551040766,
+            -0.014639912579831785,
+            0.00037813423546252645,
+            -0.01924149909676064,
+            0.00010832498610532006,
+            0.011013130149723431,
+            -0.011782407678373089,
+            0.00438725832178584,
+            0.01608349750921335,
+            0.019347883652009713,
+            -0.009760017612545174,
+            0.003869246694835168,
+            -0.024240324890794748,
+            0.010958789926998691,
+            0.020511684636649225,
+            0.008883830392748497,
+            -0.01761944522796127,
+            0.021817973960311446,
+            0.010652131228027521,
+            -0.012575679152857703,
+            -0.0012753095453828826,
+            -0.028191274775392804,
+            -0.012421170815054556,
+            -0.02188248722244861,
+            -0.0011670175621882146,
+            -0.014436683528976771,
+            -0.0029783880907228383,
+            0.0009506986717516226,
+            -0.004719165363484719,
+            -0.01179163390107033,
+            -0.019543732196676634,
+            0.016606805875498898,
+            0.0227313413004497,
+            -0.006738735201626204,
+            -0.014547240981002825,
+            0.013866974327185123,
+            0.031694077725890536,
+            -0.016189209297529324,
+            -0.00542110554672736,
+            0.0028061798782555557,
+            -0.012964876141500577,
+            0.018946046750284537,
+            0.0039419760434645495,
+            0.000322115192590907,
+            -0.01991851369210666,
+            0.002489725351392778,
+            0.010835458511052207,
+            0.01684964045708685,
+            0.0005541314524621196,
+            0.010855852688653808,
+            -0.007549882748861744,
+            0.010096800951420057,
+            -0.011832260836468588,
+            -3.6179995603517093e-06,
+            -0.007800366796516945,
+            0.007001049318404058,
+            -0.004424776169203575,
+            0.034356203288485945,
+            -0.02817071276766299,
+            0.004989631403681216,
+            -0.014539644354369607,
+            -0.010888763939716889,
+            -0.0027322086509797843,
+            0.0004865534340981866,
+            -0.010775238981581461,
+            -0.005162220412866082,
+            -0.0011263271110752205,
+            0.00913081242511139,
+            0.005844327320514489,
+            -0.013886529519306775,
+            0.019171827605726503,
+            -0.030984919318656618,
+            0.004469458202753611,
+            -0.008553175891755315,
+            0.010511707988936336,
+            -0.0151185694875566,
+            0.013830711686954931,
+            0.01301959345025159,
+            0.008205984085944315,
+            0.007740367849216328,
+            -0.01532123052721834,
+            -0.01678314222392063,
+            0.001376468747213867,
+            0.0036953018347776244,
+            -0.00931855481896886,
+            -0.033391028922085425,
+            -0.007072913446166969,
+            0.005331350534425606,
+            0.011614453225043,
+            0.008156128603003278,
+            -0.02345206873048703,
+            0.005003870024809975,
+            -0.024178720646704597,
+            -0.012115112684614331,
+            -0.007750557665424973,
+            -0.011580152272614108,
+            -0.006117298954192534,
+            0.026449765548206453,
+            0.007989237291215073,
+            0.016691517078382376,
+            0.004954133565782356,
+            0.02742809127725913,
+            0.011670893621991404,
+            -0.011736840457474895,
+            -0.003349151747880166,
+            0.00035419034843196817,
+            0.005507651806218584,
+            -0.008706590052024251,
+            0.011793241980257321,
+            -0.0013295035978611142,
+            -0.0065061682758755625,
+            -0.004398354347597199,
+            0.009372816666628725,
+            0.014188676762501037,
+            -0.003007980559117429,
+            -0.008724255229333351,
+            0.015749776067989826,
+            -0.030899793617096834,
+            -0.015215033379100091,
+            0.0016152308157299517,
+            -0.007813175908301852,
+            0.017806770680528907,
+            0.01169056345888971,
+            -0.026835412610223673,
+            -0.004538235170332766,
+            0.010532316940606419,
+            0.004755930024691789,
+            0.0095408889025506,
+            0.00954782703036282,
+            0.012079064562163901,
+            -0.007829549014214643,
+            0.01046722658021748,
+            -0.011519369840999348,
+            -0.007092961453184232,
+            -0.007241799314569802,
+            -0.008077934089061052,
+            0.021467547268987214,
+            -0.025816030568238717,
+            -0.006558853543772633,
+            0.01134670818701549,
+            0.013027491377727365,
+            -0.002301991242932548,
+            -0.006299073927206214,
+            -0.017357989759336386,
+            0.008311464809363397,
+            -0.013922568511594478,
+            0.025633907424998367,
+            -0.004854655498817219,
+            -0.005336807899100369,
+            -0.005228056880188717,
+            0.01958682721272445,
+            -0.016774409737540904,
+            0.02072573220949266,
+            -0.014449250679704904,
+            -0.007857351393427215,
+            0.001673929749689194,
+            0.006890008607998747,
+            -0.002948835017234416,
+            0.03734128725793149,
+            -0.00556474133167756,
+            -0.0014544201137417158,
+            0.02209084868209573,
+            -0.01883610080898967,
+            0.017901421953179442,
+            0.01563604483998435,
+            0.012327159100148019,
+            0.0018161034827768648,
+            -0.010253034391420751,
+            0.019979728662282904,
+            0.020823925491027635,
+            0.0139490143219451,
+            0.009577521724445478,
+            -0.013478761809038993,
+            0.014934933784421556,
+            -0.0003725422088504494,
+            -0.0024362115214204105,
+            0.012785534912074068,
+            0.016441944143888537,
+            -0.02164522371903646,
+            -0.013152102018736346,
+            0.0063548333449661805,
+            -0.016547629104776748,
+            -0.024875443975589356,
+            -0.008991095813132973,
+            -0.021885127280977718,
+            -0.023336714382596175,
+            0.00688008599534903,
+            0.016482610587564443,
+            0.0035797836729533367,
+            0.013111012464332918,
+            0.006820471349350837,
+            0.018274678506489214,
+            -0.00980248993077149,
+            -0.01582838461385605,
+            -0.0009454586621769173,
+            -0.0004906493597104733,
+            0.009787688665732045,
+            -0.01835583040075359,
+            -0.013977939077760832,
+            0.004340902272034388,
+            -0.020118484730757877,
+            0.01421736173162507,
+            -0.006428386730058143,
+            0.016756376129345597,
+            -0.0004630315831872392,
+            0.011701948921076007,
+            0.02621773278421472,
+            -0.001391567406453007,
+            0.01262026995055899,
+            0.004696906079976175,
+            -0.0030577364576645635,
+            -0.010624066224426196,
+            0.0009420498021845824,
+            -0.0044079288776429516,
+            -0.0004845606666642234,
+            0.006481113327122954,
+            0.020269778435314,
+            -0.03321232649491074,
+            -0.009425498978427855,
+            0.01246754422779043,
+            -0.001658574527323383,
+            -0.005816985985181387,
+            0.013050062223188965,
+            0.00011195028631856881,
+            0.016573827662919836,
+            -0.0017453203044868104,
+            0.01417371165305189,
+            -0.008165910519539772,
+            -0.0007461950113024975,
+            0.0016492309267666983,
+            0.005076015595439873,
+            0.005829299318055953,
+            0.012146800191035666,
+            0.01148884639506138,
+            -0.021051327863689618,
+            -0.0026357096198729156,
+            0.0036146148489310443,
+            -0.010673282438301611,
+            0.014967444393862347,
+            0.005737933867764028,
+            0.01391411712868596,
+            0.0029556189893790237,
+            0.01167806258173173,
+            -0.024440974650111174,
+            -0.01029944007931845,
+            0.01385509328958837,
+            0.00869671728965227,
+            -0.00998315356917989,
+            -0.016580053751364735,
+            -0.007061005072623325,
+            -0.00477175200214453,
+            -0.0021182980051220428,
+            -0.009871524928312118,
+            -0.0005933941177237234,
+            0.006284920058476413,
+            5.466820734762947e-05,
+            -0.012842295882002122,
+            0.015810530885294202,
+            0.004572385791133295,
+            0.003685966629168175,
+            0.005093714133656017,
+            0.011877541830396259,
+            -0.013130170365831912,
+            0.01258190257102377,
+            0.011433121961378686,
+            0.01142665981860032,
+            -0.013710025169815697,
+            -0.001774030746825879,
+            -0.01109881340948764,
+            0.0065684584345163735,
+            0.009983753894796107,
+            0.005346443477816739,
+            -0.002062387245572686,
+            0.004950044847352647,
+            0.006105199826576752,
+            -0.06391185159544624,
+            -0.02131511418723256,
+            0.0027165737784844413,
+            0.033389688791557785,
+            -0.007167593552251769,
+            -0.0037646833460996797,
+            0.002883510089742145,
+            0.01356887065340832,
+            0.009730171586767032,
+            0.00418566245984083,
+            0.020320934773473808,
+            -0.008814826860651366,
+            -0.003014193747732543,
+            0.001559573791453607,
+            0.003499420226345028,
+            -0.008703837291821111,
+            0.0044844701302276645,
+            0.0053963630238108035,
+            0.027764369548110287,
+            0.001469968540767627,
+            0.015793086297019008,
+            -0.008505290826588508,
+            -0.004075888558568453,
+            -0.009166710762668385,
+            -0.002860047211143503,
+            -0.011127820563172684,
+            0.003937110717404271,
+            0.02054186721680431,
+            -0.006126180963243227,
+            0.009346433819831456,
+            0.01057504086030076,
+            0.0001650851447316893,
+            -0.012033423322524951,
+            0.0049587335923558,
+            -0.0032240390628760535,
+            -0.005568976687946761,
+            -0.0024667040429312737,
+            -0.03474027725033016,
+            0.026855947684394643,
+            -0.004369626674503285,
+            0.009252512430957932,
+            -0.005081834388513357,
+            -0.0037896610731020337,
+            -0.014201609270756739,
+            0.017871088029615106,
+            -0.008249444723433136,
+            -0.020092257756971717,
+            0.009532994544485572,
+            -0.004991989149115383,
+            0.006464417738670472,
+            -0.02643078108339741,
+            0.0029138922065812495,
+            -0.017232410749655366,
+            -0.003999293090834831,
+            -0.006924488971625995,
+            0.01318082244178942,
+            -0.007232695119856389,
+            -0.02660711037908311,
+            0.01979251106803667,
+            -0.019485860712345603,
+            -0.011977904020170568,
+            -0.0177530973032872,
+            -0.014524003566945937,
+            0.004874952064727108,
+            -0.015326044325348929,
+            0.002047771741041311,
+            -0.009697340332342246,
+            -0.003652304998155628,
+            -0.0008878698629674222,
+            0.0017782435347417623,
+            -0.013017776298713044,
+            -0.018272697230002275,
+            -0.01353020425912225,
+            -0.00757239449706798,
+            -0.0008250000619957538,
+            0.01299142281772714,
+            -0.001679606555140765,
+            -0.007329941421303076,
+            0.0044605462460153116,
+            0.010231192277666706,
+            0.0006558644644452857,
+            0.0024231733856020414,
+            -0.010728844680699885,
+            -0.014137434553169443,
+            0.005230053181525096,
+            -0.012086468475542818,
+            -0.009459717916037003,
+            -0.0028185227600210556,
+            0.014984527039312847,
+            0.008189464334213754,
+            -0.009139240774931914,
+            -0.005639157579163322,
+            0.014325949891859377,
+            0.009020303519690971,
+            -0.013310205617445609,
+            0.01567574314514385,
+            -0.025369802629268523,
+            0.021964786021408878,
+            0.030536905527039065,
+            0.004945288795196981,
+            -0.01168303908203322,
+            0.02281128628230144,
+            -0.004072286832145419,
+            0.0028278384348551146,
+            0.01740522162288968,
+            -0.042558465579514926,
+            -0.013146562547084278,
+            0.002903743316339479,
+            -0.015297177099602701,
+            -0.010963312697387696,
+            0.008804938803398296,
+            -0.0026727734429658817,
+            0.005273238046953793,
+            0.004257613402172282,
+            0.03512246556126453,
+            0.002258612643297096,
+            -0.0002885708728599208,
+            -0.0019720190285930487,
+            0.012312613129996533,
+            0.010451003939611476,
+            -0.015556620910136185,
+            -0.02387856865043814,
+            -0.004583090523619507,
+            -0.0037828776396525835,
+            -0.0014588238692793338,
+            -0.004361834167147271,
+            -0.008159679676329573,
+            0.012243319920992457,
+            -0.017279964797769668,
+            -0.013748032758693542,
+            0.018394195628121487,
+            -0.006411113522842359,
+            0.019002611495040334,
+            -0.014970362556522149,
+            -0.0077437581301661946,
+            0.0017311115715410025,
+            0.024439691379769795,
+            -0.025013956798520212,
+            0.007359134924497539,
+            -0.0031406470488477937,
+            0.017574824561755627,
+            0.00419409250132525,
+            -0.0046803792187546656,
+            0.010556972371044426,
+            0.015229789583480183,
+            -0.004608293492221483,
+            -0.014449487985966966,
+            -0.01172850291078532,
+            0.016823024209386785,
+            -0.012304887597699588,
+            0.0011773351576246157,
+            0.002262670226244277,
+            -0.0023810578015456716,
+            0.03477119000959688,
+            -0.03442178291363675,
+            0.004062693040714608,
+            0.0029249583507755944,
+            0.003291349443702718,
+            0.006461525089528308,
+            0.006047777261425826,
+            -0.020642802018978404,
+            0.008916088381876748,
+            -0.03316762661930299,
+            -0.006367254689957291,
+            -0.005404276207844267,
+            0.0009189006423526284,
+            0.009401330264376041,
+            -0.011106443887384533,
+            -0.02259405372588916,
+            -0.006010439054323966,
+            0.0070744551115721095,
+            0.010262607009446879,
+            -0.002856616377476421,
+            0.019719451109535182,
+            0.008426509638139982,
+            0.0015877648339862691,
+            -0.010655792508751512,
+            -0.015527851374834499,
+            -0.015460974859638219,
+            -0.017847462632188386,
+            -0.02236964311096428,
+            0.011841749854626384,
+            -0.00576688153869993,
+            0.009054902130717224,
+            0.014317707346504314,
+            -0.015366657284925976,
+            0.002910211008094564,
+            0.017761685988205086,
+            0.007974806171830457,
+            0.002692237107055498,
+            0.0032405477366987947,
+            0.02685230091638845,
+            -0.0009116884421301454,
+            -0.0032228977180251673,
+            0.003158213042682977,
+            0.015872199488167846,
+            0.0004966905858799325,
+            0.019981401695749582,
+            0.015201538351780685,
+            0.005359075814444414,
+            -0.004668738193285921,
+            0.019853988551405383,
+            -0.009844394286458195,
+            0.013476144827747423,
+            -0.00411789677405284,
+            -0.00830656434595337,
+            -0.008496132520586954,
+            0.006442581751999304,
+            -0.008138786054348571,
+            0.006977337946948795,
+            0.02419128791801387,
+            -0.0005734827808485311,
+            -0.0028842142002271044,
+            0.005503033191779872,
+            -0.01696639779775455,
+            -0.00389835761601045,
+            -0.03366077608692946,
+            -0.009942613075976742,
+            -0.01708630689497001,
+            -0.02910756562213045,
+            0.026536268897199546,
+            0.00219827495662445,
+            0.02586503862171508,
+            -0.01905122645609761,
+            0.008935637454859214,
+            -0.01385806889332683,
+            0.029747093931541325,
+            0.009163959012862627,
+            0.0008611776492791918,
+            0.003598831885031635,
+            -0.02103045281939351,
+            -0.01150941825152158,
+            0.027341774725604333,
+            -0.0006700608710615124,
+            -0.012089044078073347,
+            0.022396400579038037,
+            0.017491565611764764,
+            0.0051760367056280536,
+            -0.00559536785318233,
+            0.01979262595019399,
+            0.002306380546194741,
+            0.011973863166872444,
+            -0.0028136750606972223,
+            0.012911235736067303,
+            -0.01510421118359169,
+            -0.003338602157155174,
+            0.039758808572572274,
+            0.007290642519242007,
+            -0.001802921446521437,
+            -0.013696774990705774,
+            0.003941198181491921,
+            -0.0008308293948698925,
+            0.0200351513103389,
+            -0.00014327587644797734,
+            0.028291423013030337,
+            -0.0001085859276231927,
+            -0.010723064585989723,
+            0.020443848393732245,
+            -0.017860296930396913,
+            0.04540901629084378,
+            -0.018671578423261854,
+            0.000852031295629291,
+            -0.010639853151638835,
+            -0.019017194262730474,
+            0.003861296738239369,
+            0.0010977143012920602,
+            0.0008387482016061222,
+            0.0060155816314618,
+            -0.010780239332444652,
+            0.002152214497259421,
+            0.010556338812873054,
+            -0.027636192450621728,
+            0.0010938992111792307,
+            -0.014385202899929138,
+            -0.01353192238925814,
+            -0.026667778226211785,
+            -0.00017403398236549974,
+            -0.015357674131635848,
+            -0.013956722505869091,
+            -0.01985038526848704,
+            -0.008927903879720431,
+            -0.009580214158077466,
+            -0.0020314441852913386,
+            -0.004755584878514854,
+            -0.00018388548107372635,
+            -0.002918855567842974,
+            -0.0015941852262316128,
+            -0.014230272061190995,
+            0.02133992212037431,
+            0.015283345362445234,
+            0.004642765060980717,
+            0.012829570028541471,
+            0.01708952142208596,
+            -0.004770647543823141,
+            0.006604762788847952,
+            -0.0025369836845081078,
+            0.008368000846861623,
+            0.014785031726639733,
+            0.012074738887486398,
+            0.02124518119032885,
+            -0.015459818988572564,
+            -0.004992865135357168,
+            -0.012722391036055767,
+            -0.012210202603282509,
+            -0.017850065946244383,
+            -0.004300429148015694,
+            -0.0014387298144714717,
+            -0.023128300275416423,
+            0.004468952290723408,
+            -0.0026602931688877443,
+            -0.005768852559887139,
+            0.007815212168719318,
+            0.0005862972228454734,
+            -0.002265719947629544,
+            -0.00019323737674204774,
+            -0.02735820445266249,
+            0.00024009122317215594,
+            -0.00018701993387558173,
+            -0.014663710240203917,
+            -0.00450797561624596,
+            0.010584731989847334,
+            -0.011966527748437568,
+            -0.00602367292291343,
+            0.0111155226518017,
+            0.0015565179432852256,
+            0.006062862961835896,
+            0.016952559697728433,
+            0.0005012157272316869,
+            0.011160668242334889,
+            0.014993782560797772,
+            -0.01591303023837098,
+            -0.015356412848444082,
+            -0.0236025426434855,
+            -0.004514538154596937,
+            -0.009753334017065119,
+            0.007553951535816575,
+            -0.004616595532996391,
+            -0.013023153809236325,
+            0.006040285197946957,
+            -0.0006343451398642784,
+            0.008338564073158257,
+            -0.017608246105292328,
+            -8.346168119689692e-05,
+            -0.020944870308705715,
+            0.009230123321316848,
+            -0.00531237224627929,
+            -0.00445985089300067,
+            0.01410675485522911,
+            0.02895945580213926,
+            -0.007799830351886844,
+            -0.009455128322575888,
+            -0.005663072244883114,
+            0.012817229945992764,
+            0.012474431030896143,
+            -0.004423141749110039,
+            -0.0018294512146253298,
+            -0.012817101093394574,
+            0.0004755435377918658,
+            -0.01129485262081729,
+            0.0062747111125005155,
+            0.017199374798326412,
+            -0.021965117839794063,
+            -0.01296097993683748,
+            -0.016765996091978336,
+            -0.016882855472266523,
+            0.004372970343291638,
+            0.017290565252244258,
+            0.022670788364378462,
+            -0.008456547449078897,
+            -0.019564049850942417,
+            0.006898353991539946,
+            -0.01620125883985687,
+            0.0037108638122592746,
+            0.12887455743463638,
+            -0.0028864852421660366,
+            -0.011487804449328495,
+            -0.022683334123299446,
+            -0.005467192611880829,
+            0.01210557696712196,
+            -0.01555159294496261,
+            0.06709672179195036,
+            -0.015071089552631105,
+            0.004364902817496082,
+            -0.004899872946473657,
+            0.0006310685609937571,
+            -0.017610935588191347,
+            -0.005962991715597585,
+            -0.007768819050529025,
+            -0.02688607975562596,
+            0.000883763319535686,
+            -0.009935313912803587,
+            -0.012883104064962595,
+            0.002126368813790577,
+            -0.0013697265269800346,
+            -0.016131186838419288,
+            0.01575514418037863,
+            -0.020677990768697592,
+            -0.007251917191831732,
+            -0.0029325623780992118,
+            -0.00600710212646933,
+            0.014573093764769918,
+            -0.03253336117394778,
+            0.002973725493330224,
+            -0.013601417220439602,
+            0.011905788192118131,
+            0.001542117126832946,
+            -0.010789486980776022,
+            -0.013862139885514524,
+            0.007086571810314664,
+            0.015658913897173155,
+            0.017039624764452546,
+            0.01901374401756777,
+            0.004797954569087795,
+            0.003928999581093576,
+            -0.008046722893614655,
+            0.021974371139905104,
+            -0.00697335955527197,
+            0.0008975473383295958,
+            -0.00970589295353857,
+            -0.004346869073499034,
+            0.0031279458769925416,
+            0.02303983493541673,
+            -0.012560380384977343,
+            -0.01536378830925542,
+            0.0012035987157572355,
+            0.033313215063595594,
+            -0.010845084209743817,
+            0.013587126023659577,
+            0.0029867620644300073,
+            0.020679673435988446,
+            -0.0006640787253300063,
+            0.0025071780490108306,
+            0.022554193713122565,
+            -0.018082906254599396,
+            0.018230000220500718,
+            -0.004451773302171552,
+            0.01915083025790679,
+            -0.004183521043835777,
+            0.015019320101074747,
+            -0.027014055979204272,
+            -0.005362044589023088,
+            -0.012355559881051253,
+            0.003995309003198048,
+            0.020680455787871383,
+            0.0054391755324872375,
+            0.009865676762244514,
+            -0.01645386866657314,
+            -0.01760657627715638,
+            0.022969828924328126,
+            -0.021566687194482292,
+            -0.015826424849757402,
+            0.00012804774676298237,
+            -0.015021970627042762,
+            0.006000610438452896,
+            0.010126179023323158,
+            -0.02912121211344025,
+            -0.02046150879033127,
+            0.004881850039543051,
+            -0.0023469840669773042,
+            -0.00915786433296249,
+            0.006547377770403066,
+            -0.022329063637464074,
+            -0.015702777677743352,
+            -0.005692904224111613,
+            0.01207999389934878,
+            -0.008429160038192635,
+            0.010075061963131637,
+            -0.020106712664936504,
+            0.0036787045650556907,
+            -0.002764965946892891,
+            0.010307702251974629,
+            0.004697169247387421,
+            0.011192285826515207,
+            0.0030113901179382946,
+            0.0014774681536380047,
+            -0.02778270000997313,
+            0.010043261430740039,
+            -5.178406450222847e-05,
+            0.04216636255860709,
+            0.030977096295679525,
+            0.004479857028809617,
+            -5.798020745008712e-05,
+            -0.009996471945562616,
+            0.012476009017187798,
+            -0.019306907391548302,
+            -0.005092322343599383,
+            0.01856463597985508,
+            0.02195467141594795,
+            -0.02236232264139305,
+            0.0052217593371508145,
+            -0.016766955100332014,
+            0.016133299654723637,
+            -0.011208763137897244,
+            0.006248024409576809,
+            0.011601988603146743,
+            0.0022330878390492527,
+            0.010317781892717903,
+            -0.005895142237225119,
+            0.019670196605183973,
+            0.01186677200787666,
+            -0.002339367025714548,
+            0.02367408853110795,
+            0.006858134385159451,
+            0.004479683673962342,
+            0.004798719889930319,
+            -0.009604173975576537,
+            -0.009045602998213713,
+            0.009230969488015385,
+            0.004083616606774546,
+            -0.007944628589259331,
+            0.011836323287097144,
+            0.015100350195476968,
+            0.01885317701149666,
+            -0.0011707847499298092,
+            -0.00684548433712413,
+            -0.006550059418141426,
+            0.001899758393077906,
+            -0.024577381680573596,
+            -0.02855511854789239,
+            0.021238935710272536,
+            -0.010427135883958709,
+            -0.019187769881105637,
+            -0.0015864640188683347,
+            0.016360037306061392,
+            -0.008853972319021563,
+            -0.01627167079143546,
+            0.001270414822216867,
+            0.02255755055961612,
+            -0.005959411125391464,
+            0.006009654905468478,
+            0.005909233773418581,
+            0.005874361519568704,
+            -0.0027859463941992753,
+            -0.024599789577567636,
+            0.0036612374968223453,
+            0.03221346865585554,
+            0.016461563205582293,
+            0.016139782455492073,
+            -0.007158168059808459,
+            0.001838455290310879,
+            0.007530954381521755,
+            0.005438791523412295,
+            0.015674000912615536,
+            0.008604106101542998,
+            -0.006990462920227135,
+            0.00244736628387359,
+            0.012213048255418866,
+            -0.007329299946651028,
+            0.017424711328323798,
+            0.001973882532418776,
+            -0.0133178791530314,
+            -0.008373452013951822,
+            -0.010462150707776556,
+            -0.014575483571750984,
+            0.0027958704799249897,
+            -0.011401916623878957,
+            0.0008016041012669959,
+            0.003953958717263787,
+            0.010260753041492368,
+            0.013961649189650642,
+            -0.0019275791443041316,
+            0.017019168629298462,
+            0.011534594243860177,
+            -0.0017859096307142257,
+            -0.002970739203219076,
+            -0.03142778833370656,
+            0.010207789755749909,
+            -0.014730553506704018,
+            -0.0037611416875830746,
+            0.01825790347981639,
+            -0.007766890847082858,
+            -0.0032778433418881666,
+            -0.021003405429807113,
+            0.0024512381432602736,
+            0.008768616692363321,
+            0.01558713989712387,
+            -0.012819542714293536,
+            0.009201155584074711,
+            -0.0012254337002053017,
+            -0.0057451574939563045,
+            0.00977251392462179,
+            0.0042356831792011935,
+            -0.0033433823868435786,
+            0.011499764125503874
+          ],
+          "title": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer",
+          "keyphrases": [
+            "Sparsely-Gated Mixture-of-Experts Layer",
+            "conditional computation",
+            "neural network",
+            "language modeling",
+            "machine translation"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "7f92225a-9c2d-4745-b7b1-2273bcee2b14",
+        "properties": {
+          "page_content": "Appendices A Load-Balancing Loss\n\nAs discussed in section 4, for load-balancing purposes, we want to define an additional loss function to encourage experts to receive roughly equal numbers of training examples. Unfortunately, the number of examples received by an expert is a discrete quantity, so it can not be used in backpropagation. Instead, we define a smooth estimator Load(X) of the number of examples assigned to each expert for a batch X of inputs. The smoothness allows us to back-propagate gradients through the estimator. This is the purpose of the noise term in the gating function. We define P(x, i) as the probability that G(x)iis nonzero, given a new random choice of noise on element i, but keeping the already-sampled choices of noise on the other elements. To compute P(x, i), we note that the G(x)iis nonzero if and only if H(x)iis greater than the k th-greatest element of H(x) excluding itself. The probability works out to be:\n\n$$P(x,i)=P r\\Big((x\\cdot W_{g}){i}+S t a n d x N o r a l()\\cdot S o f t p u s((x\\cdot W{n o i s e})_{i})$$ $$>k t h_e x c l u d i n(H(x),k,i)\\Big)$$ $$({\\boldsymbol{8}})$$\n\n$$(9)$$ $$(10)$$ (8) Where kth_excluding(v, k, i) means the kth highest component of v, excluding component i. Simplifying, we get:\n\n$$P(x,i)=\\Phi{\\Big(}{\\frac{(x\\cdot W_{g}){i}-k t h_c x c l u d i n g(H(x),k,i)}{S o f t p l u s((x\\cdot W{n o i s e})_{i})}}{\\Big)}$$\n\nWhere Φ is the CDF of the standard normal distribution.\n\n$$L o a d(X){i}=\\sum{x\\in X}P(x,i)$$\n\n$$(11)$$ P(x, i) (10) We can now define the load loss to be the square of the coefficient of variation of the load vector, multiplied by a hand-tuned scaling factor wload.\n\n$$L_{l o a d}(X)=w_{l o a d}\\cdot C V(L o a d(X))^{2}$$ Lload(X) = wload · CV (Load(X))2(11) Initial Load Imbalance: To avoid out-of-memory errors, we need to initialize the network in a state of approximately equal expert load (since the soft constraints need some time to work). To accomplish this, we initialize the matrices Wg and Wnoise to all zeros, which yields no signal and some noise.\n\nExperiments: We trained a set of models with identical architecture (the MoE-256 model described in Appendix C), using different values of wimportance and wload. We trained each model for 10 epochs, then measured perplexity on the test set. We also measured the coefficients of variation in Importance and Load, as well as ratio of the load on the most overloaded expert to the average load. This last value is significant for load balancing purposes on distributed hardware. All of these metrics were averaged over several training batches.\n\nwimportance wload Test Perplexity CV (Importance(X)) CV (Load(X)) max(Load(X)) mean(Load(X)) 0.0 0.0 39.8 3.04 3.01 17.80 0.2 0.0 35.6 0.06 0.17 1.47 0.0 0.2 35.7 0.22 0.04 1.15 0.1 0.1 35.6 0.06 0.05 1.14 0.01 0.01 35.7 0.48 0.11 1.37 1.0 1.0 35.7 0.03 0.02 1.07\n\nTable 6: Experiments with different combinations of losses.\n\n13 Results: Results are reported in Table 6. All the combinations containing at least one the two losses led to very similar model quality, where having no loss was much worse. Models with higher values of wload had lower loads on the most overloaded expert.\n\nB Hierachical Mixture Of Experts\n\nIf the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network.3If the hierarchical MoE consists of a groups of b experts each, we denote the primary gating network by Gprimary, the secondary gating networks by (G1, G2..Ga), and the expert networks by (E0,0, E0,1..Ea,b). The output of the MoE is given by:\n\n$$y_{H}=\\sum_{i=1}^{a}\\sum_{j=1}^{b}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}\\cdot E{i,j}(x)$$ $$(12)$$ $$(13)$$ $$(14)$$\n\nOur metrics of expert utilization change to the following:\n\n$$I m p o r t a n c e_{H}(X){i,j}=\\sum{x\\in X}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}$$ $$L o a d{H}(X){i,j}={\\frac{L o a d{p r i m a r y}(X){i}\\cdot L o a d{i}(X^{(i)})_{j}}{|X^{(i)}|}}$$\n\nLoadprimary and Loadi deonte the Load functions for the primary gating network and i th secondary gating network respectively. X(i) denotes the subset of X for which Gprimary(x)i > 0.\n\nIt would seem simpler to let LoadH(X)i,j = Loadi(Xi)j , but this would not have a gradient with respect to the primary gating network, so we use the formulation above.\n\nC 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS C.1 8-MILLION-OPERATIONS-PER-TIMESTEP MODELS Model Architecture: Our model consists of five layers: a word embedding layer, a recurrent Long Short-Term Memory (LSTM) layer (Hochreiter & Schmidhuber, 1997; Gers et al., 2000), a MoE layer, a second LSTM layer, and a softmax layer. The dimensionality of the embedding layer, the number of units in each LSTM layer, and the input and output dimensionality of the MoE layer are all equal to 512. For every layer other than the softmax, we apply drouput (Zaremba et al., 2014) to the layer output, dropping each activation with probability DropP rob, otherwise dividing by (1 − DropP rob). After dropout, the output of the previous layer is added to the layer output.\n\nThis residual connection encourages gradient flow (He et al., 2015).\n\nMoE Layer Architecture: Each expert in the MoE layer is a feed forward network with one ReLU-activated hidden layer of size 1024 and an output layer of size 512. Thus, each expert contains [512 ∗ 1024] + [1024 ∗ 512] = 1M parameters. The output of the MoE layer is passed through a sigmoid function before dropout. We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts.\n\nWe call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers. Thus, each example is processed by exactly 4 experts for a total of 4M ops/timestep. The two LSTM layers contribute 2M ops/timestep each for the desired total of 8M.\n\n3 We have not found the need for deeper hierarchies.\n\nComputationally-Matched Baselines: The MoE-4 model does not employ sparsity, since all 4 experts are always used. In addition, we trained four more computationally-matched baseline models with no sparsity:\n\nMoE-1-Wide: The MoE layer consists of a single \"expert\" containing one ReLU-activated hidden layer of size 4096.\n\nMoE-1-Deep: The MoE layer consists of a single \"expert\" containing four ReLU-activated hidden layers, each with size 1024.\n\n4xLSTM-512: We replace the MoE layer with two additional 512-unit LSTM layers.\n\nLSTM-2048-512: The model contains one 2048-unit LSTM layer (and no MoE). The output of the LSTM is projected down to 512 dimensions (Sak et al., 2014). The next timestep of the LSTM receives the projected output. This is identical to one of the models published in (Jozefowicz et al., 2016). We re-ran it to account for differences in training regimen, and obtained results very similar to the published ones. Training: The models were trained on a cluster of 16 K40 GPUs using the synchronous method described in Section 3. Each batch consisted of a set of sentences totaling roughly 300,000 words. In the interest of time, we limited training to 10 epochs, (27,000 steps). Training took 12-16 hours for all models, except for MoE-4, which took 18 hours (since all the expert computation was performed on only 4 of 16 GPUs). We used the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 1000 training steps, and decreased after that so as to be proportional to the inverse square root of the step number. The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016). For each model, we performed a hyper-parmeter search to find the best dropout probability, in increments of 0.1.\n\nTo ensure balanced expert utilization we set wimportance = 0.1 and wload = 0.1, as described in Section 4 and Appendix A.\n\nResults: We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol. Results are reported in Table 7. For each model, we report the test perplexity, the computational budget, the parameter counts, the value of DropP rob, and the computational efficiency.\n\nModel Test Test ops/timestep #Params excluding Total Drop- TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params P rob per GPU 10 epochs (final) (millions) (billions) (observed) Kneser-Ney 5-gram* 67.6 0.00001 1.8 LSTM-512-512* 54.1 2.4 2.4 0.8 0.1 LSTM-1024-512* 48.2 4.7 4.7 0.8 0.1 LSTM-2048-512* 45.0 43.7 9.4 9.4 0.8 0.1 0.61 LSTM-2048-512 44.7 9.4 9.4 0.8 0.1 1.21 4xLSTM-512 46.0 8.4 8.4 0.8 0.1 1.07 MoE-1-Wide 46.1 8.4 8.4 0.8 0.1 1.29 MoE-1-Deep 45.7 8.4 8.4 0.8 0.1 1.29 MoE-4 45.0 8.4 8.4 0.8 0.1 0.52 MoE-32 39.7 8.4 37.8 0.9 0.1 0.87 MoE-256 35.7 8.6 272.9 1.1 0.1 0.81 MoE-256-h 36.0 8.4 272.9 1.1 0.1 0.89 MoE-1024-h 34.6 8.5 1079.0 1.9 0.2 0.90 MoE-4096-h 34.1 8.9 4303.4 5.1 0.2 0.74 2xLSTM-8192-1024* 34.7 30.6 151.0 151.0 1.8 0.25 1.09 MoE-34M 31.3 33.8 4313.9 6.0 0.3 1.22 MoE-143M 28.0 142.7 4371.1 6.0 0.4 1.56\n\nC.2 More Expensive Models\n\nWe ran two additional models (MoE-34M and MoE-143M) to investigate the effects of adding more computation in the presence of a large MoE layer. These models have computation budgets of 34M and 143M ops/timestep. Similar to the models above, these models use a MoE layer between two LSTM layers. The dimensionality of the embedding layer, and the input and output dimensionality of the MoE layer are set to 1024 instead of 512. For MoE-34M, the LSTM layers have 1024 units.\n\nFor MoE-143M, the LSTM layers have 4096 units and an output projection of size 1024 (Sak et al., 2014). MoE-34M uses a hierarchical MoE layer with 1024 experts, each with a hidden layer of size 2048. MoE-143M uses a hierarchical MoE layer with 256 experts, each with a hidden layer of size 8192. Both models have 4B parameters in the MoE layers. We searched for the best DropP rob for each model, and trained each model for 10 epochs.\n\nThe two models achieved test perplexity of 31.3 and 28.0 respectively, showing that even in the presence of a large MoE, more computation is still useful. Results are reported at the bottom of Table 7. The larger of the two models has a similar computational budget to the best published model from the literature, and training times are similar. Comparing after 10 epochs, our model has a lower test perplexity by 18%.\n\nD 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS Model Architecture: The models are similar in structure to the 8-million-operations-per-timestep models described in the previous section. We vary the number of experts between models, using an ordinary MoE layer with 32 experts and hierarchical MoE layers with 256, 1024, 4096, 16384, 65536 and 131072 experts. For the hierarchical MoE layers, the first level branching factors are 32, 32, 64, 128, 256 and 256, respectively.\n\nTraining: Models are trained on a cluster of 32 Tesla K40 GPUs, except for the last two models, which are trained on clusters of 64 and 128 GPUs so as to have enough memory for all the parameters. For all models, training batch sizes are approximately 2.5 million words. Models are trained once-through over about 100 billion words. We implement several memory optimizations in order to fit up to 1 billion parameters per GPU. First, we do not store the activations of the hidden layers of the experts, but instead recompute them on the backwards pass. Secondly, we modify the optimizer on the expert parameters to require less auxiliary storage: The Adam optimizer (Kingma & Ba, 2015) keeps first and second moment estimates of the perparameter gradients. This triples the required memory. To avoid keeping a first-moment estimator, we set β1 = 0. To reduce the size of the second moment estimator, we replace it with a factored approximation. For a matrix of parameters, instead of maintaining a full matrix of second-moment estimators, we maintain vectors of row-wise and column-wise averages of that matrix. At each step, the matrix of estimators is taken to be the outer product of those two vectors divided by the mean of either one. This technique could similarly be applied to Adagrad (Duchi et al., 2010).\n\nModel Test Test ops/timestep #Params excluding Total TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params per GPU .1 epochs 1 epoch (millions) (billions) (observed) Kneser-Ney 5-gram 67.1 45.3 0.00001 76.0 4xLSTM-512 54.5 47.0 8.4 8.4 0.1 1.23 MoE-32 48.5 40.4 8.4 37.8 0.1 0.83 MoE-256-h 42.8 35.3 8.4 272.9 0.4 1.11 MoE-1024-h 40.3 32.7 8.5 1079.0 1.2 1.14 MoE-4096-h 38.9 30.9 8.6 4303.4 4.4 1.07 MoE-16384-h 38.2 29.7 8.8 17201.0 17.3 0.96 MoE-65536-h 38.2 28.9 9.2 68791.0 68.9 0.72 MoE-131072-h 39.8 29.2 9.7 137577.6 137.7 0.30\n\nResults: We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8. Perplexity after 100 billion training words is 39% lower for the 68-billion-parameter MoE model than for the baseline model. It is notable that the measured computational efficiency of the largest model (0.30 TFLOPS/GPU) is very low compared to the other models. This is likely a result of the fact that, for purposes of comparison to the other models, we did not increase the training batch size proportionally to the number of GPUs. For comparison, we include results for a computationally matched baseline model consisting of 4 LSTMs, and for an unpruned 5-gram model with Kneser-Ney smoothing (Kneser & Ney, 1995).4\n\nE Machine Translation - Experimental Details\n\nModel Architecture for Single Language Pair MoE Models: Our model is a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decrease the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We insert MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). We use an attention mechanism between the encoder and decoder, with the first decoder LSTM receiving output from and providing input for the attention 5. All of the layers in our model have input and output dimensionality of 512. Our LSTM layers have 2048 hidden units, with a 512-dimensional output projection. We add residual connections around all LSTM and MoE layers to encourage gradient flow (He et al., 2015). Similar to GNMT, to effectively deal with rare words, we used subword units (also known as \"wordpieces\") (Schuster & Nakajima, 2012) for inputs and outputs in our system.\n\nWe use a shared source and target vocabulary of 32K wordpieces. We also used the same beam search technique as proposed in (Wu et al., 2016).\n\nWe train models with different numbers of experts in the MoE layers. In addition to a baseline model with no MoE layers, we train models with flat MoE layers containing 32 experts, and models with hierarchical MoE layers containing 512 and 2048 experts. The flat MoE layers use k = 4 and the hierarchical MoE models use k = 2 at each level of the gating network. Thus, each input is processed by exactly 4 experts in each MoE layer. Each expert in the MoE layer is a feed forward network with one hidden layer of size 2048 and ReLU activation. Thus, each expert contains [512 ∗ 2048] + [2048 ∗ 512] = 2M parameters. The output of the MoE layer is passed through a sigmoid function. We use the strictly-balanced gating function described in Appendix F.\n\nModel Architecture for Multilingual MoE Model: We used the same model architecture as for the single-language-pair models, with the following exceptions: We used noisy-top-k gating as described in Section 2.1, not the scheme from Appendix F. The MoE layers in the encoder and decoder are non-hierarchical MoEs with n = 512 experts, and k = 2. Each expert has a larger hidden layer of size 8192. This doubles the amount of computation in the MoE layers, raising the computational budget of the entire model from 85M to 102M ops/timestep. Training: We trained our networks using the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 2000 training steps, held constant for an additional 8000 steps, and decreased after that so as to be proportional to the inverse square root of the step number. For the single-language-pair models, similarly to (Wu et al., 2016), we applied dropout (Zaremba et al., 2014) to the output of all embedding, LSTM and MoE layers, using DropP rob = 0.4. Training was done synchronously on a cluster of up to 64 GPUs as described in section 3. Each training batch consisted of a set of sentence pairs containing roughly 16000 words per GPU.\n\nTo ensure balanced expert utilization we set wimportance = 0.01 and wload = 0.01, as described in Section 4 and Appendix A. Metrics: We evaluated our models using the perplexity and the standard BLEU score metric. We reported tokenized BLEU score as computed by the multi-bleu.pl script, downloaded from the public implementation of Moses (on Github), which was also used in (Luong et al., 2015a). Results: Tables 2, 3 and 4 in Section 5.3 show comparisons of our results to other published methods. Figure 4 shows test perplexity as a function of number of words in the (training data's) source sentences processed for models with different numbers of experts. As can be seen from the Figure, as we increased the number of experts to approach 2048, the test perplexity of our model continued to improve.\n\nWe found that the experts indeed become highly specialized by syntax and/or semantics, as can be seen in Table 9. For example, one expert is used when the indefinite article \"a\" introduces the direct object in a verb phrase indicating importance or leadership.\n\nExpert 381 Expert 752 Expert 2004 ... with researchers , ... ... plays a core ... ... with rapidly growing ... ... to innovation . ... plays a critical ... ... under static conditions ... ... tics researchers . ... provides a legislative ... ... to swift ly ... ... the generation of ... ... play a leading ... ... to dras tically ... ... technology innovations is ... ... assume a leadership ... ... the rapid and ... ... technological innovations , ... ... plays a central ... ... the fast est ... ... support innovation throughout ... ... taken a leading ... ... the Quick Method ... ... role innovation will ... ... established a reconciliation ... ... rec urrent ) ... ... research scienti st ... ... played a vital ... ... provides quick access ... ... promoting innovation where ... ... have a central ... ... of volatile organic ... ... ... ...\n\nDue to some peculiarities in our infrastructure which have since been fixed, at the time we ran some of the machine translation experiments, our models ran faster if every expert received exactly the same batch size. To accommodate this, we used a different gating function which we describe below. Recall that we define the softmax gating function to be:\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ Gσ(x) = Sof tmax(x · Wg) (15) Sparse Gating (alternate formulation): To obtain a sparse gating vector, we multiply Gσ(x) component-wise with a sparse mask M(Gσ(x)) and normalize the output. The mask itself is a function of Gσ(x) and specifies which experts are assigned to each input example:\n\n$$(15)$$ $$G(x){i}=\\frac{G{\\sigma}(x){i}M(G{\\sigma}(x)){i}}{\\sum{j=1}^{n}G_{\\sigma}(x){j}M(G{\\sigma}(x))_{j}}\\tag{1}$$ $$(17)$$\n\n$$(16)$$\n\nTop-K Mask: To implement top-k gating in this formulation, we would let M(v) = T opK(v, k), where:\n\n$$T o p K(v,k){i}=\\begin{cases}1&{\\mathrm{if~}}v{i}{\\mathrm{~is~in~the~top~}}k{\\mathrm{~elements~of~}}v.\\ 0&{\\mathrm{otherwise.}}\\end{cases}$$ 0 otherwise. (17) Batchwise Mask: To force each expert to receive the exact same number of examples, we introduce an alternative mask function, Mbatchwise(X, m), which operates over batches of input vectors.\n\nInstead of keeping the top k values per example, we keep the top m values per expert across the training batch, where m = k|X| n, so that each example is sent to an average of k experts.\n\n$$M_{b a t c h w i s e}(X,m){j,i}=\\begin{cases}1&\\text{if}X{j,i}\\text{is in the top}m\\text{values for to expert}i\\ 0&\\text{otherwise}\\end{cases}$$\n\nAs our experiments suggest and also observed in (Ioffe & Szegedy, 2015), using a batchwise function during training (such as Mbatchwise) requires modifications to the inference when we may not have a large batch of examples. Our solution to this is to train a vector T of per-expert threshold values to approximate the effects of the batchwise mask. We use the following mask at inference time:\n\n$$(18)$$ $$M_{threshold}(x,T){i}=\\begin{cases}1&\\text{if}x{i}>T_{i}\\ 0&\\text{otherwise}\\end{cases}\\tag{1}$$ $$(19)$$\n\nTo learn the threshold values, we apply an additional loss at training time which is minimized when the batchwise mask and the threshold mask are identical.\n\n$$L_{batchwise}(X,T,m)=\\sum_{j=1}^{|X|}\\sum_{i=1}^{n}(M_{threshold}(x,T){i}-M{batchwise}(X,m){j,i})(X{j,i}-T_{i})\\tag{20}$$\n\nG ATTENTION FUNCTION The attention mechanism described in GNMT (Wu et al., 2016) involves a learned \"Attention Function\" A(xi, yj ) which takes a \"source vector\" xi and a \"target vector\" yj , and must be computed for every source time step i and target time step j. In GNMT, the attention function is implemented as a feed forward neural network with a hidden layer of size n. It can be expressed as:\n\n$$A_{G N M T}(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}t a n h((x_{i}U){d}+(y{j}W)_{d})$$\n\nWhere U and W are trainable weight matrices and V is a trainable weight vector.\n\nFor performance reasons, in our models, we used a slightly different attention function:\n\n$$A(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}tanh((x_{i}U){d})tanh((y{j}W)_{d})\\tag{22}$$\n\nWith our attention function, we can simultaneously compute the attention function on multiple source time steps and multiple target time steps using optimized matrix multiplications. We found little difference in quality between the two functions.\n\n$$(21)$$",
+          "embedding": [
+            0.0011764472583308816,
+            -0.018546752631664276,
+            -0.01668562740087509,
+            -0.003858148353174329,
+            0.009172027930617332,
+            0.015837986022233963,
+            -0.013212142512202263,
+            0.06935913860797882,
+            -0.011065399274230003,
+            0.035029683262109756,
+            0.016455290839076042,
+            0.002998990472406149,
+            0.020509224385023117,
+            -0.022867878898978233,
+            0.0126132657751441,
+            -0.002063821302726865,
+            0.02303372137248516,
+            0.015008772723376751,
+            -0.015994615852832794,
+            -0.019071921706199646,
+            -0.008485622704029083,
+            -0.004452418070286512,
+            -0.029298892244696617,
+            -0.004053934942930937,
+            -0.01113910786807537,
+            0.010531017556786537,
+            0.020085405558347702,
+            0.015220683068037033,
+            -0.0586714930832386,
+            -0.0015317423967644572,
+            -0.002752529690042138,
+            -0.0007854498689994216,
+            -0.015248322859406471,
+            -0.012097310274839401,
+            -0.027824735268950462,
+            -0.03810698911547661,
+            0.047799576073884964,
+            -0.00880348775535822,
+            -0.03169440105557442,
+            0.020896190777420998,
+            0.0326526015996933,
+            0.009411578066647053,
+            -0.009347083978354931,
+            -0.00776697089895606,
+            -0.009227308444678783,
+            0.001612360472790897,
+            0.012263152748346329,
+            0.010208545252680779,
+            -0.01241978257894516,
+            -0.031841814517974854,
+            -0.0024001137353479862,
+            -0.020306527614593506,
+            0.015008772723376751,
+            0.0039019123651087284,
+            -0.015156188048422337,
+            -0.008172363974153996,
+            0.0126132657751441,
+            0.048573512583971024,
+            0.05200092867016792,
+            0.04765216261148453,
+            0.038364965468645096,
+            0.01730293035507202,
+            -0.0017505628056824207,
+            -0.00107452308293432,
+            0.031086310744285583,
+            -0.020195966586470604,
+            -0.006030227988958359,
+            0.006214498076587915,
+            -0.016768548637628555,
+            -0.034440018236637115,
+            -0.06047733873128891,
+            0.02084091119468212,
+            0.007868318818509579,
+            -0.015764279291033745,
+            -0.00650011608377099,
+            -0.028635522350668907,
+            0.01830720156431198,
+            0.07252857834100723,
+            0.013009445741772652,
+            0.005095059052109718,
+            0.017883380874991417,
+            0.010816635563969612,
+            -0.015036413446068764,
+            0.01986428163945675,
+            0.024415744468569756,
+            -0.005647868383675814,
+            -0.032431479543447495,
+            -0.06198835000395775,
+            0.008844949305057526,
+            -0.011083826422691345,
+            -0.039323169738054276,
+            0.03440316393971443,
+            0.0019210122991353273,
+            -0.01265933271497488,
+            -0.0008666437352076173,
+            0.03983912244439125,
+            0.03114159032702446,
+            0.0012944950722157955,
+            -0.011793265119194984,
+            0.022093946114182472,
+            -0.002939102705568075,
+            0.0032569682225584984,
+            0.005164160393178463,
+            0.025171250104904175,
+            0.02139372006058693,
+            -0.005099665839225054,
+            -0.007439891807734966,
+            -0.007467532530426979,
+            0.039212606847286224,
+            0.11468950659036636,
+            -0.019606303423643112,
+            -0.0036969122011214495,
+            -0.03038608469069004,
+            -0.0014557312242686749,
+            -0.023678665980696678,
+            -0.015764279291033745,
+            -0.056386549025774,
+            -0.022204507142305374,
+            0.02006697840988636,
+            0.0007134695188142359,
+            0.030109679326415062,
+            0.051153287291526794,
+            0.031473275274038315,
+            -0.0012956467689946294,
+            -0.02800900489091873,
+            0.06519464403390884,
+            -0.02168855257332325,
+            0.025853049010038376,
+            -0.006477082148194313,
+            -0.04636227339506149,
+            -0.0012069669319316745,
+            0.018279561772942543,
+            -0.013875513337552547,
+            -0.031510129570961,
+            0.014760008081793785,
+            0.006771914195269346,
+            0.02799057774245739,
+            0.05008452385663986,
+            -0.029685860499739647,
+            -0.010733714327216148,
+            -0.0009679921204224229,
+            -0.02004855126142502,
+            -0.003448148025199771,
+            -0.012640906497836113,
+            -0.03899148106575012,
+            0.0012334557250142097,
+            0.015543155372142792,
+            0.07444498687982559,
+            -0.022296641021966934,
+            0.029354173690080643,
+            -0.01696203276515007,
+            -0.016519784927368164,
+            0.01671326719224453,
+            0.018887652084231377,
+            -0.008181577548384666,
+            -0.01625259406864643,
+            -0.022130798548460007,
+            0.009333264082670212,
+            -0.04415103793144226,
+            0.0015858716797083616,
+            -0.00802494864910841,
+            -0.008881802670657635,
+            -0.018344055861234665,
+            0.01060472521930933,
+            0.0071035996079444885,
+            0.033481817692518234,
+            0.04308227077126503,
+            -0.006698206067085266,
+            -0.058413516730070114,
+            0.05148497223854065,
+            0.005809104535728693,
+            0.04463013634085655,
+            0.030072826892137527,
+            0.03891777619719505,
+            -0.0029068556614220142,
+            0.01007034257054329,
+            0.010687646456062794,
+            0.008918656967580318,
+            -0.03639327734708786,
+            0.04595687985420227,
+            -0.023181136697530746,
+            0.0117564108222723,
+            0.016556639224290848,
+            0.04440901428461075,
+            0.03666968271136284,
+            0.01958787627518177,
+            0.009793938137590885,
+            0.019716864451766014,
+            -0.011719557456672192,
+            0.010770567692816257,
+            0.02760361135005951,
+            0.017579335719347,
+            0.04927373677492142,
+            0.017081808298826218,
+            0.00880348775535822,
+            -0.00950832013040781,
+            -0.015091693960130215,
+            0.020214393734931946,
+            -0.010365174151957035,
+            0.0126132657751441,
+            0.026387430727481842,
+            -0.03106788359582424,
+            -0.004744946490973234,
+            -0.014068996533751488,
+            -0.003989440388977528,
+            0.00300820404663682,
+            -0.011876186355948448,
+            -0.016694841906428337,
+            0.03066249005496502,
+            -0.01771753840148449,
+            -0.021854395046830177,
+            -0.0038051707670092583,
+            0.00054647505749017,
+            -0.017653044313192368,
+            -0.0017010403098538518,
+            0.01538652554154396,
+            -0.009517533704638481,
+            0.019035067409276962,
+            0.01853753812611103,
+            -0.0024876417592167854,
+            -0.0067212400026619434,
+            0.03226563706994057,
+            0.012631692923605442,
+            -0.003680788679048419,
+            -0.022978439927101135,
+            -0.010236185975372791,
+            -0.03145484998822212,
+            -0.04444586858153343,
+            0.017643829807639122,
+            -0.027179790660738945,
+            0.017579335719347,
+            0.024968553334474564,
+            -0.011102253571152687,
+            -0.012677759863436222,
+            0.02036180905997753,
+            -0.03563777357339859,
+            0.02331012487411499,
+            0.0023482879623770714,
+            -0.023862935602664948,
+            -0.009600454941391945,
+            -0.024913271889090538,
+            -0.01956944912672043,
+            -0.027437768876552582,
+            -0.002099523553624749,
+            -0.0045491596683859825,
+            -0.0019878100138157606,
+            -0.01796630211174488,
+            -0.015856413170695305,
+            -0.011968321166932583,
+            0.013064726255834103,
+            0.024397317320108414,
+            -0.04297171160578728,
+            0.012576411478221416,
+            -0.0023770800326019526,
+            -0.030883613973855972,
+            0.06814295798540115,
+            0.0442984513938427,
+            0.00950832013040781,
+            0.015469446778297424,
+            -0.047541599720716476,
+            0.03315013274550438,
+            0.024765856564044952,
+            -0.05155868083238602,
+            0.047799576073884964,
+            -0.001612360472790897,
+            -0.005776857025921345,
+            0.052885424345731735,
+            -0.004975283518433571,
+            0.0018519111908972263,
+            0.012475063093006611,
+            0.02846967987716198,
+            -0.014833716675639153,
+            -0.03521395102143288,
+            -0.03162069246172905,
+            0.02769574522972107,
+            -0.017312144860625267,
+            -0.03217350319027901,
+            0.0005597193958237767,
+            0.04440901428461075,
+            0.01614203117787838,
+            0.002754833083599806,
+            0.013654389418661594,
+            0.015174615196883678,
+            0.02120945043861866,
+            0.024986980482935905,
+            0.01589326746761799,
+            -0.014004502445459366,
+            9.407835750607774e-05,
+            -0.05056362599134445,
+            0.008273712359368801,
+            -0.020417090505361557,
+            -0.008563937619328499,
+            -0.014704727567732334,
+            -0.005085845477879047,
+            0.015672143548727036,
+            -0.003065788187086582,
+            -0.008149330504238605,
+            0.011949894018471241,
+            0.03455058112740517,
+            -0.03066249005496502,
+            -0.026590127497911453,
+            -0.008250678889453411,
+            -0.0036001706030219793,
+            0.0102730393409729,
+            -0.025558216497302055,
+            0.0033145523630082607,
+            0.004162193275988102,
+            0.029777994379401207,
+            -0.004226687829941511,
+            -0.0012323040282353759,
+            0.02084091119468212,
+            0.006725846324115992,
+            0.008872589096426964,
+            -0.0073661841452121735,
+            -0.016556639224290848,
+            0.008872589096426964,
+            0.0007664470467716455,
+            0.020877765491604805,
+            -0.02331012487411499,
+            -0.024231474846601486,
+            0.037222493439912796,
+            0.015027199871838093,
+            -0.04470384493470192,
+            0.02091461792588234,
+            -0.014336188323795795,
+            0.010328320786356926,
+            -0.012180231511592865,
+            0.0410921573638916,
+            -0.014907424338161945,
+            -0.024028778076171875,
+            0.003388260258361697,
+            -0.04142384231090546,
+            -0.02463686838746071,
+            -0.027253499254584312,
+            0.006085508968681097,
+            -0.02397349663078785,
+            0.005845958366990089,
+            0.0005499300896190107,
+            0.0009127111989073455,
+            -0.004892362281680107,
+            -0.01694360561668873,
+            0.0160959642380476,
+            0.016833042725920677,
+            -0.009052252396941185,
+            -0.013608322478830814,
+            0.009056858718395233,
+            -0.024913271889090538,
+            0.03388721123337746,
+            0.035306088626384735,
+            0.008370454423129559,
+            0.001825422397814691,
+            0.003524159314110875,
+            0.0060578687116503716,
+            -0.01534045860171318,
+            -0.010678432881832123,
+            -0.03687237948179245,
+            -0.04193979874253273,
+            0.013396412134170532,
+            -0.02837754413485527,
+            0.02817484736442566,
+            0.01318450178951025,
+            0.03751732409000397,
+            -0.020675068721175194,
+            0.0005908149178139865,
+            -0.050121378153562546,
+            -0.01739506609737873,
+            -0.0131292212754488,
+            -0.026295294985175133,
+            0.01566293090581894,
+            -0.031491704285144806,
+            -0.017247650772333145,
+            0.026442712172865868,
+            -0.007485959213227034,
+            -0.013479333370923996,
+            -0.01728450506925583,
+            0.003307642415165901,
+            0.05376992002129555,
+            -0.017588550224900246,
+            -0.025650352239608765,
+            0.02463686838746071,
+            -0.03305799514055252,
+            0.0005672053666785359,
+            0.02301529422402382,
+            0.008762027136981487,
+            0.0011442000977694988,
+            0.03316855803132057,
+            -0.0034619681537151337,
+            -0.018758662045001984,
+            0.00160084362141788,
+            0.027751026675105095,
+            -0.00611314969137311,
+            -0.036540694534778595,
+            -0.0026650014333426952,
+            -0.03226563706994057,
+            0.004079272039234638,
+            0.00807101558893919,
+            0.013110794126987457,
+            -0.019090348854660988,
+            -0.005288542248308659,
+            0.008084836415946484,
+            -0.01021775882691145,
+            -0.0018288774881511927,
+            0.011968321166932583,
+            0.032910581678152084,
+            -0.004533036146312952,
+            6.323476554825902e-05,
+            -0.02714293636381626,
+            0.016446076333522797,
+            0.04153440520167351,
+            -0.017183156684041023,
+            0.011636636219918728,
+            -0.012171017937362194,
+            -0.059371720999479294,
+            -0.0513744130730629,
+            0.04098159819841385,
+            0.014833716675639153,
+            -0.017708323895931244,
+            -0.011645849794149399,
+            -0.004203653894364834,
+            -0.037222493439912796,
+            -0.006730453111231327,
+            0.004611351061612368,
+            0.00632966635748744,
+            0.01519304234534502,
+            -0.012788321822881699,
+            0.028230128809809685,
+            0.017026526853442192,
+            0.018251920118927956,
+            0.008001914247870445,
+            -0.05491239205002785,
+            0.016648773103952408,
+            -0.003132586134597659,
+            0.024618441238999367,
+            -0.019809000194072723,
+            -0.005979553796350956,
+            -0.03237619996070862,
+            -0.034716423600912094,
+            0.022757316008210182,
+            -0.026111025363206863,
+            -0.04289800301194191,
+            -0.008757420815527439,
+            0.0358404703438282,
+            -0.0010618545347824693,
+            0.004005563911050558,
+            -0.01921933703124523,
+            -0.021467428654432297,
+            -0.019164055585861206,
+            -0.010503376834094524,
+            -0.013654389418661594,
+            -0.0235128216445446,
+            -0.06497351825237274,
+            -0.008808095008134842,
+            -0.0442984513938427,
+            -0.01725686341524124,
+            0.029114622622728348,
+            -0.010383601300418377,
+            0.028598668053746223,
+            -0.052111491560935974,
+            -0.00685483543202281,
+            -0.04153440520167351,
+            0.008084836415946484,
+            0.025908328592777252,
+            0.023715518414974213,
+            -0.0075274198316037655,
+            0.04319283366203308,
+            0.012134164571762085,
+            -0.030644062906503677,
+            -0.031215298920869827,
+            -0.00568472221493721,
+            -0.013626749627292156,
+            -0.00014914335042703897,
+            -0.024784283712506294,
+            0.06335194408893585,
+            0.02883821912109852,
+            0.05122699588537216,
+            0.02876451052725315,
+            -0.018076865002512932,
+            -0.02616630680859089,
+            -0.028340689837932587,
+            0.006449441891163588,
+            0.05447014421224594,
+            -0.021559562534093857,
+            -0.003945676609873772,
+            -0.013608322478830814,
+            0.0029874735046178102,
+            -0.012327647767961025,
+            0.0146125927567482,
+            0.03449530154466629,
+            -0.012253939174115658,
+            -0.017266077920794487,
+            0.004417867865413427,
+            0.01947731524705887,
+            -0.020232820883393288,
+            0.03545350208878517,
+            -0.005740003194659948,
+            -0.04488811641931534,
+            -0.009545174427330494,
+            0.020011696964502335,
+            0.028193274512887,
+            0.01910877414047718,
+            -0.0073892176151275635,
+            -0.026037318632006645,
+            0.011959107592701912,
+            0.023346979171037674,
+            0.004510002676397562,
+            -0.022775743156671524,
+            0.013774164952337742,
+            0.017597762867808342,
+            -0.006799554452300072,
+            -0.04956856742501259,
+            0.022886306047439575,
+            -0.02894878014922142,
+            0.02600046433508396,
+            -0.003966406919062138,
+            0.01899821311235428,
+            0.001621573930606246,
+            -0.01853753812611103,
+            0.022996867075562477,
+            -0.006675172131508589,
+            0.00989528652280569,
+            -0.0034919120371341705,
+            -0.012134164571762085,
+            -0.0012472759699448943,
+            0.05664452537894249,
+            0.0038535415660589933,
+            0.0047910138964653015,
+            0.0410921573638916,
+            0.013488546945154667,
+            0.013811019249260426,
+            -0.015460233204066753,
+            -0.039028335362672806,
+            0.025576643645763397,
+            -0.019053494557738304,
+            -0.013811019249260426,
+            -0.026903385296463966,
+            -0.028635522350668907,
+            -0.011415512301027775,
+            0.021320011466741562,
+            -0.027106082066893578,
+            -0.014842930249869823,
+            -0.0005844806437380612,
+            0.0168975368142128,
+            -0.0363011434674263,
+            0.009282589890062809,
+            0.019827427342534065,
+            -0.0022642146795988083,
+            0.02426832914352417,
+            -0.03217350319027901,
+            0.0007566577405668795,
+            0.02139372006058693,
+            0.008656072430312634,
+            -0.0001934832544066012,
+            -0.001268006395548582,
+            0.020896190777420998,
+            -0.02893035300076008,
+            -0.00601640809327364,
+            0.0059426999650895596,
+            0.010549443773925304,
+            -0.009379331022500992,
+            -0.01637236960232258,
+            0.009388544596731663,
+            0.0020960685797035694,
+            -0.01022697240114212,
+            0.010190118104219437,
+            -0.020619787275791168,
+            -0.03895463049411774,
+            0.0053944974206388,
+            0.008011127822101116,
+            -0.018979785963892937,
+            0.0069976444356143475,
+            0.0018887651385739446,
+            -0.0028469678945839405,
+            0.026866532862186432,
+            -0.005458991974592209,
+            0.032247208058834076,
+            0.04249260947108269,
+            0.01611439138650894,
+            0.008301353082060814,
+            0.00578146381303668,
+            -0.0067442734725773335,
+            0.02963057905435562,
+            -0.008292139507830143,
+            0.0016273324145004153,
+            0.026018891483545303,
+            0.010881129652261734,
+            -0.006384947337210178,
+            -0.007025284692645073,
+            -0.018251920118927956,
+            -0.0035287661012262106,
+            -0.014750794507563114,
+            0.017892595380544662,
+            0.041129011660814285,
+            -0.00030836393125355244,
+            -0.022867878898978233,
+            0.013470119796693325,
+            -0.02205709181725979,
+            -0.00439483392983675,
+            0.012309220619499683,
+            0.01847304403781891,
+            -0.039875976741313934,
+            -0.010577084496617317,
+            0.023623384535312653,
+            0.021320011466741562,
+            -0.012585625052452087,
+            -0.02876451052725315,
+            0.014022929593920708,
+            0.009185847826302052,
+            0.0167777631431818,
+            -0.002157107926905155,
+            -0.007485959213227034,
+            0.008038768544793129,
+            0.006380340550094843,
+            -0.007928206585347652,
+            -0.011820905841886997,
+            0.018251920118927956,
+            0.01979057304561138,
+            0.0063204532489180565,
+            -0.015644503757357597,
+            -0.023531248793005943,
+            0.024010350927710533,
+            0.03497440367937088,
+            -0.016888324171304703,
+            0.0194036066532135,
+            0.0036347212735563517,
+            0.011056185699999332,
+            0.00874360091984272,
+            -0.009821578860282898,
+            -0.009968994185328484,
+            -0.0004307305789552629,
+            -0.0037061257753521204,
+            -0.011194388382136822,
+            -0.005302362609654665,
+            -0.016860684379935265,
+            -0.012263152748346329,
+            0.053069692105054855,
+            0.010392814874649048,
+            0.014824503101408482,
+            -0.0009841157589107752,
+            -0.019237764179706573,
+            -0.007559667341411114,
+            0.011784051544964314,
+            0.02647956646978855,
+            0.004072362091392279,
+            0.03162069246172905,
+            -0.005058205220848322,
+            0.013479333370923996,
+            -0.024600014090538025,
+            0.0019382876344025135,
+            0.020011696964502335,
+            0.016409222036600113,
+            -0.0023678664583712816,
+            -0.0027502262964844704,
+            0.004544553346931934,
+            0.020896190777420998,
+            -0.007297082804143429,
+            -0.046583399176597595,
+            0.018316414207220078,
+            -0.015745852142572403,
+            0.02664540894329548,
+            0.04234519228339195,
+            0.01513776183128357,
+            -0.000702528515830636,
+            0.030883613973855972,
+            -0.021559562534093857,
+            0.015856413170695305,
+            -0.02577934041619301,
+            -0.027880016714334488,
+            0.014824503101408482,
+            -0.0067442734725773335,
+            0.013921581208705902,
+            0.01074292790144682,
+            -0.018758662045001984,
+            -0.02559507079422474,
+            0.015128548257052898,
+            0.007707083132117987,
+            -0.029685860499739647,
+            0.008421128615736961,
+            0.025060689076781273,
+            0.0011384416138753295,
+            0.01026382576674223,
+            0.004095395561307669,
+            0.01680540293455124,
+            -0.01189461350440979,
+            -0.007601127959787846,
+            0.029114622622728348,
+            -0.010973264463245869,
+            -0.013230569660663605,
+            0.000723258825019002,
+            -0.04853665828704834,
+            -0.007988094352185726,
+            -0.0020649731159210205,
+            -0.02091461792588234,
+            -0.001933680847287178,
+            0.011857759207487106,
+            0.006200677715241909,
+            0.010180904529988766,
+            0.015294390730559826,
+            0.043708790093660355,
+            -0.0009547477238811553,
+            -0.002020057290792465,
+            -0.0018231190042570233,
+            -0.030736196786165237,
+            -0.01744113303720951,
+            -0.011572141200304031,
+            -0.002694945316761732,
+            0.019348325207829475,
+            -0.0029183723963797092,
+            -0.011341804638504982,
+            -0.02292315848171711,
+            0.016657987609505653,
+            0.029372600838541985,
+            0.056902505457401276,
+            0.0031602266244590282,
+            0.043524518609046936,
+            0.0007048318511806428,
+            0.02828541025519371,
+            0.019274618476629257,
+            0.008660678751766682,
+            -0.046104297041893005,
+            0.01223551295697689,
+            0.022130798548460007,
+            0.03217350319027901,
+            0.01805843785405159,
+            -0.047357331961393356,
+            -0.004242811352014542,
+            0.006366520654410124,
+            0.03735148161649704,
+            0.027179790660738945,
+            0.013967648148536682,
+            -0.0034619681537151337,
+            0.022757316008210182,
+            -0.029593724757432938,
+            -0.021080462262034416,
+            -0.0005450354074127972,
+            -0.011931467801332474,
+            0.00437180045992136,
+            -0.018638886511325836,
+            0.029022488743066788,
+            -0.02111731469631195,
+            -0.0356193445622921,
+            -0.006255958694964647,
+            -0.008702139370143414,
+            -0.001766686444170773,
+            -0.002782473573461175,
+            0.009066072292625904,
+            0.028782937675714493,
+            0.004316519480198622,
+            0.027013948187232018,
+            -0.01118517480790615,
+            0.02559507079422474,
+            0.004715002607554197,
+            -0.00549123901873827,
+            0.008296746760606766,
+            -0.018325628712773323,
+            -0.006610678043216467,
+            -0.009351690299808979,
+            -0.007937420159578323,
+            0.00712202675640583,
+            0.006864049006253481,
+            -0.009254949167370796,
+            -0.010927197523415089,
+            0.019551021978259087,
+            0.0032892152667045593,
+            0.010005848482251167,
+            0.026332149282097816,
+            -0.010475736111402512,
+            -0.0011493826750665903,
+            0.024028778076171875,
+            0.002002781955525279,
+            -0.02349439635872841,
+            0.005924272816628218,
+            0.0013843266060575843,
+            0.037222493439912796,
+            -0.021246304735541344,
+            -0.008338207378983498,
+            0.020951472222805023,
+            0.000956475269049406,
+            -0.0028561814688146114,
+            -0.03913889825344086,
+            0.0062052845023572445,
+            -0.013285850174725056,
+            0.01022697240114212,
+            0.007974274456501007,
+            0.016400009393692017,
+            -0.01389394048601389,
+            0.006186857353895903,
+            0.004413261078298092,
+            0.00858236476778984,
+            -0.0075043863616883755,
+            0.018178213387727737,
+            -0.0030151139944791794,
+            -0.05074789375066757,
+            -0.006656745448708534,
+            -0.014971918426454067,
+            0.020509224385023117,
+            0.012825176119804382,
+            -0.004035507794469595,
+            0.020988326519727707,
+            -0.02531866542994976,
+            -0.030128106474876404,
+            0.03499282896518707,
+            -0.03283687308430672,
+            0.014916637912392616,
+            -0.0005496421945281327,
+            -0.032818444073200226,
+            -0.040834181010723114,
+            -0.018961358815431595,
+            -0.00040280219400301576,
+            -0.0052977558225393295,
+            0.004065451677888632,
+            -0.013718884438276291,
+            -0.017109448090195656,
+            -0.02616630680859089,
+            0.02109888754785061,
+            -0.0020039337687194347,
+            0.013682030141353607,
+            0.002022360684350133,
+            -0.012898883782327175,
+            -0.014962704852223396,
+            -0.01801236905157566,
+            -0.012382928282022476,
+            0.01810450479388237,
+            -0.016427649185061455,
+            -0.014262479729950428,
+            -0.017864953726530075,
+            -0.004896969068795443,
+            -0.012935738079249859,
+            0.0010215454967692494,
+            0.03637485206127167,
+            0.03565619885921478,
+            0.009079893119633198,
+            0.018942931666970253,
+            -0.020232820883393288,
+            0.045993734151124954,
+            -0.023936642333865166,
+            0.034532155841588974,
+            0.0001733287499519065,
+            -0.020859338343143463,
+            -0.005606407765299082,
+            0.024194620549678802,
+            -0.016519784927368164,
+            -0.01357146818190813,
+            0.004461631644517183,
+            -0.002443877747282386,
+            0.023549675941467285,
+            0.02159641683101654,
+            0.01447439007461071,
+            -0.004507699050009251,
+            0.02577934041619301,
+            0.004634384531527758,
+            -0.0006092418916523457,
+            0.0303676575422287,
+            0.02493169903755188,
+            0.008642252534627914,
+            -0.030920466408133507,
+            -0.013811019249260426,
+            -0.01584720052778721,
+            0.0071358466520905495,
+            0.012143378145992756,
+            0.001317528891377151,
+            0.0056985425762832165,
+            -0.014354614540934563,
+            0.004228990990668535,
+            -0.008315172977745533,
+            0.008633038960397243,
+            0.02244405820965767,
+            -0.010171690955758095,
+            -0.03093889355659485,
+            -0.028395971283316612,
+            -0.01851911097764969,
+            -0.003362923162057996,
+            -0.00018139055464416742,
+            -0.01666720025241375,
+            0.020582932978868484,
+            -0.0003740100364666432,
+            0.005620228126645088,
+            -0.014032143168151379,
+            0.0027225858066231012,
+            0.005993374157696962,
+            -0.012447422370314598,
+            0.0015064053004607558,
+            0.0055833738297224045,
+            -0.0015156188746914268,
+            -0.041239574551582336,
+            -0.004615957848727703,
+            0.004438598174601793,
+            0.004832474514842033,
+            -0.03455058112740517,
+            -0.03331597521901131,
+            -0.03270788490772247,
+            0.006767307408154011,
+            -0.003961800131946802,
+            -0.023936642333865166,
+            0.0131292212754488,
+            -0.01156292762607336,
+            -0.007416858337819576,
+            -0.023641811683773994,
+            -0.01151686068624258,
+            0.01938517950475216,
+            0.0014292424311861396,
+            0.01685146987438202,
+            -0.008200004696846008,
+            -0.00261663063429296,
+            -0.0204355176538229,
+            0.006993037648499012,
+            -0.03068091720342636,
+            0.0012023602612316608,
+            -0.009913713671267033,
+            -0.0005608710926026106,
+            -0.026921812444925308,
+            0.0022503945510834455,
+            0.0044109574519097805,
+            0.013746525160968304,
+            -0.03534294292330742,
+            0.0065646106377244,
+            0.011194388382136822,
+            0.016630345955491066,
+            -0.00720955478027463,
+            0.03244990482926369,
+            0.01094562467187643,
+            -0.005482025444507599,
+            0.03132586181163788,
+            0.05682879686355591,
+            0.022480912506580353,
+            -0.03364766016602516,
+            -0.01394922100007534,
+            -0.012198658660054207,
+            0.013811019249260426,
+            -0.014723154716193676,
+            -0.010770567692816257,
+            -0.008771240711212158,
+            0.023457542061805725,
+            -0.005362250376492739,
+            0.0075274198316037655,
+            0.04020766168832779,
+            -0.00880348775535822,
+            0.009591241367161274,
+            0.006417194847017527,
+            -0.012327647767961025,
+            0.00601640809327364,
+            -0.00658764410763979,
+            0.01710023358464241,
+            0.030920466408133507,
+            0.013534614816308022,
+            0.023328552022576332,
+            -0.04205036163330078,
+            0.0204355176538229,
+            0.010503376834094524,
+            0.006960790604352951,
+            0.022407203912734985,
+            0.028801364824175835,
+            0.016086751595139503,
+            -0.007195734418928623,
+            -0.026442712172865868,
+            -0.036337997764348984,
+            0.0020718832965940237,
+            -0.03373979404568672,
+            -0.0034435412380844355,
+            -0.014262479729950428,
+            0.023144282400608063,
+            0.004795620683580637,
+            -0.019458888098597527,
+            -0.013534614816308022,
+            0.006960790604352951,
+            -0.02091461792588234,
+            -0.004961463622748852,
+            -0.009867645800113678,
+            -0.021522708237171173,
+            -0.024102484807372093,
+            0.02120945043861866,
+            -0.009351690299808979,
+            -0.029298892244696617,
+            0.010125624015927315,
+            0.001757472869940102,
+            0.0003642207011580467,
+            -0.0007860257173888385,
+            -0.020509224385023117,
+            -0.03943372890353203,
+            -0.02082248404622078,
+            0.0025935969315469265,
+            0.032910581678152084,
+            -0.026313722133636475,
+            0.03554563969373703,
+            -0.007559667341411114,
+            0.007508993148803711,
+            -0.0023701698519289494,
+            0.009185847826302052,
+            -0.01634472794830799,
+            -0.038549236953258514,
+            0.01785574108362198,
+            -0.0010491859866306186,
+            0.026111025363206863,
+            0.006334273144602776,
+            -0.027474623173475266,
+            -0.0058275312185287476,
+            0.025742486119270325,
+            0.005048991646617651,
+            -0.013737311586737633,
+            -0.007375397253781557,
+            -0.005320789758116007,
+            0.011526074260473251,
+            -0.04157125949859619,
+            -0.003858148353174329,
+            0.06368362903594971,
+            -0.013276636600494385,
+            0.02732720598578453,
+            0.019090348854660988,
+            -0.03283687308430672,
+            -0.04824182391166687,
+            -0.021946528926491737,
+            -0.001739045954309404,
+            -0.008467195555567741,
+            -0.024342035874724388,
+            0.019661584869027138,
+            -0.002150197746232152,
+            -0.001446517650038004,
+            0.005910452920943499,
+            -0.0042727552354335785,
+            -0.026240015402436256,
+            0.0004845719086006284,
+            -0.015275963582098484,
+            -0.0006691296002827585,
+            -0.03466114401817322,
+            -0.02874608337879181,
+            0.02319956384599209,
+            0.014935065060853958,
+            -0.0055833738297224045,
+            -0.0008309414843097329,
+            -0.013995288871228695,
+            -0.0037521931808441877,
+            0.017579335719347,
+            -0.0010486101964488626,
+            0.012014389038085938,
+            0.00955438707023859,
+            0.031768109649419785,
+            0.0035517998039722443,
+            0.003777530277147889,
+            -0.049236882477998734,
+            -0.011673489585518837,
+            0.02760361135005951,
+            0.003952586557716131,
+            -0.017957089468836784,
+            0.0008326690294779837,
+            -0.00437180045992136,
+            -0.009517533704638481,
+            -0.038070134818553925,
+            0.030146533623337746,
+            0.046583399176597595,
+            -0.0022319676354527473,
+            -0.003123372560366988,
+            0.019256191328167915,
+            5.466009952215245e-06,
+            0.010300680063664913,
+            -0.005417531356215477,
+            -0.012336861342191696,
+            0.030902039259672165,
+            0.005905846133828163,
+            -0.00028720172122120857,
+            0.014778435230255127,
+            -0.009402364492416382,
+            -0.0024576981086283922,
+            -0.02561349794268608,
+            -0.0004474300367292017,
+            -0.028340689837932587,
+            0.006601464468985796,
+            0.0006265172269195318,
+            0.006467869039624929,
+            0.010660005733370781,
+            0.029391027987003326,
+            0.03668811172246933,
+            0.016768548637628555,
+            -0.00218590022996068,
+            -0.010917983949184418,
+            0.029980691149830818,
+            0.022794170305132866,
+            0.009922927245497704,
+            0.022812597453594208,
+            -0.031970806419849396,
+            -0.010825849138200283,
+            0.01065079215914011,
+            0.0015294391196221113,
+            -0.0008384274551644921,
+            0.008324386551976204,
+            0.009812365286052227,
+            0.0049061826430261135,
+            -0.005882812198251486,
+            0.016105176880955696,
+            0.007347756996750832,
+            0.014953491277992725,
+            -0.0020557595416903496,
+            -0.004404047504067421,
+            0.009738657623529434,
+            -0.012336861342191696,
+            0.03517710044980049,
+            -0.018712595105171204,
+            -0.014253266155719757,
+            0.031123163178563118,
+            0.10658163577318192,
+            0.0036278110928833485,
+            -0.014741581864655018,
+            -0.02034338191151619,
+            -0.0010428517125546932,
+            0.0235128216445446,
+            0.009922927245497704,
+            0.02972271293401718,
+            0.009204274974763393,
+            0.0026488779112696648,
+            0.02426832914352417,
+            0.029649006202816963,
+            0.0031072490382939577,
+            0.0006656745099462569,
+            -0.024950126186013222,
+            0.008715960197150707,
+            0.002158259740099311,
+            -0.005979553796350956,
+            -0.005380677059292793,
+            0.010365174151957035,
+            0.017551695927977562,
+            0.005431351251900196,
+            -0.004035507794469595,
+            -0.011820905841886997,
+            -0.016353942453861237,
+            0.005458991974592209,
+            -0.01881394349038601,
+            -0.009411578066647053,
+            -0.017984729260206223,
+            -0.009784724563360214,
+            0.00013971391308587044,
+            -0.003088821889832616,
+            -0.005947306752204895,
+            0.013580681756138802,
+            -0.019698437303304672,
+            0.01977214589715004,
+            -0.008522477000951767,
+            -0.00026863077073358,
+            0.0012587928213179111,
+            -0.010051915422081947,
+            0.017772819846868515,
+            0.0021167988888919353,
+            -0.029925409704446793,
+            0.013635963201522827,
+            0.008204611018300056,
+            -0.005772250704467297,
+            0.02052765153348446,
+            -0.03344496339559555,
+            0.019809000194072723,
+            -0.0013981468509882689,
+            -0.015561581589281559,
+            0.0040309010073542595,
+            -0.018408549949526787,
+            0.005997980944812298,
+            0.006831801496446133,
+            0.006361913867294788,
+            -0.0014879783848300576,
+            0.009075285866856575,
+            0.005634048022329807,
+            -0.0016054504085332155,
+            -0.002223905874416232,
+            0.006767307408154011,
+            -0.0011056186631321907,
+            -0.00016728240007068962,
+            -0.014824503101408482,
+            -0.011811692267656326,
+            -0.0002376792108407244,
+            -0.00412073265761137,
+            -0.0037452830001711845,
+            -0.01739506609737873,
+            0.003547193016856909,
+            0.00501674460247159,
+            0.013488546945154667,
+            -0.002655788091942668,
+            -0.010733714327216148,
+            0.0019117988413199782,
+            0.007799217943102121,
+            -0.014216412790119648,
+            -0.005099665839225054,
+            -0.030902039259672165,
+            -0.008646858856081963,
+            0.02196495607495308,
+            0.005882812198251486,
+            -0.025152822956442833,
+            -0.021062035113573074,
+            0.004450114909559488,
+            -0.016446076333522797,
+            0.006725846324115992,
+            0.007163487374782562,
+            -0.013046300038695335,
+            0.005578767042607069,
+            -0.010881129652261734,
+            -0.0031602266244590282,
+            -0.0029644398018717766,
+            -0.00865146517753601,
+            0.000947261753026396,
+            -0.010770567692816257,
+            -0.0020465462002903223,
+            -0.012410569004714489,
+            -0.014437536709010601,
+            -0.05399104207754135,
+            0.008223038166761398,
+            0.01753326877951622,
+            -0.0027686532121151686,
+            -0.024802710860967636,
+            -0.013718884438276291,
+            0.013820232823491096,
+            -0.0038788786623626947,
+            -0.01867574080824852,
+            0.005067418795078993,
+            0.01657506637275219,
+            -0.008780454285442829,
+            -0.006108542904257774,
+            -0.014013716019690037,
+            0.027179790660738945,
+            0.018076865002512932,
+            0.02874608337879181,
+            -0.018795516341924667,
+            0.021909674629569054,
+            0.02284945175051689,
+            -0.026866532862186432,
+            -0.010291466489434242,
+            0.03385035693645477,
+            0.01045730896294117,
+            0.004989103879779577,
+            0.006753487046808004,
+            0.0011989051708951592,
+            -0.0346979983150959,
+            -0.007002251222729683,
+            0.0020373326260596514,
+            -0.031196871772408485,
+            0.022407203912734985,
+            0.014898210763931274,
+            0.006449441891163588,
+            0.01332270447164774,
+            -0.014511244371533394,
+            0.010531017556786537,
+            0.015368098393082619,
+            0.021191023290157318,
+            -0.0071266330778598785,
+            -0.010696860030293465,
+            0.013101580552756786,
+            0.01737663894891739,
+            -0.013672816567122936,
+            0.004544553346931934,
+            0.014750794507563114,
+            -0.018482258543372154,
+            0.03764631226658821,
+            -0.010153263807296753,
+            0.006923936307430267,
+            -0.013460906222462654,
+            -0.019698437303304672,
+            0.03283687308430672,
+            -0.0032362379133701324,
+            0.000820000481326133,
+            -0.024507878348231316,
+            0.009241129271686077,
+            0.005002924241125584,
+            0.019753718748688698,
+            0.014603379182517529,
+            0.02646113932132721,
+            0.023162709549069405,
+            0.004090788774192333,
+            -0.02867237664759159,
+            -0.0013992985477671027,
+            -0.020232820883393288,
+            0.00740303797647357,
+            -0.06250430643558502,
+            -0.026516418904066086,
+            0.017708323895931244,
+            -0.01956944912672043,
+            -0.006094722542911768,
+            -0.008955510333180428,
+            0.0022780350409448147,
+            0.025373946875333786,
+            -0.017127875238656998,
+            0.014962704852223396,
+            6.658185156993568e-05,
+            0.011719557456672192,
+            -0.013202928937971592,
+            0.011093039996922016,
+            -0.014326974749565125,
+            -0.04050249606370926,
+            0.01332270447164774,
+            0.035582493990659714,
+            0.02963057905435562,
+            -0.016565851867198944,
+            0.02360495738685131,
+            -0.01899821311235428,
+            0.013276636600494385,
+            0.005283935461193323,
+            -0.009033825248479843,
+            -0.0034020806197077036,
+            0.0004123036051169038,
+            -0.009545174427330494,
+            -0.013046300038695335,
+            -0.006099329330027103,
+            -0.027179790660738945,
+            0.003427417716011405,
+            -0.034145187586545944,
+            -0.019090348854660988,
+            0.018942931666970253,
+            0.0012230905704200268,
+            -0.009683376178145409,
+            0.027345633134245872,
+            -0.04164496809244156,
+            0.005288542248308659,
+            -0.026424285024404526,
+            0.02657170034945011,
+            -0.025834621861577034,
+            0.008877196349203587,
+            -0.016750121489167213,
+            -0.011480006389319897,
+            0.020656641572713852,
+            -0.0010278797708451748,
+            0.0063434867188334465,
+            -0.011618209071457386,
+            -0.03274473920464516,
+            -0.0045583732426166534,
+            0.02979642152786255,
+            -0.010917983949184418,
+            0.005915059708058834,
+            0.021835967898368835,
+            -0.005283935461193323,
+            -0.01428090687841177,
+            -0.00246000150218606,
+            0.007011464796960354,
+            -0.04186609014868736,
+            0.012769894674420357,
+            0.01899821311235428,
+            0.01204202976077795,
+            0.008642252534627914,
+            0.0056985425762832165,
+            0.013626749627292156,
+            0.006085508968681097,
+            -0.006219104863703251,
+            0.048684071749448776,
+            0.0002988625201396644,
+            -0.00037055497523397207,
+            -0.004710395820438862,
+            0.0043902271427214146,
+            0.0002775563334580511,
+            -0.009102926589548588,
+            0.016427649185061455,
+            0.0021617147140204906,
+            0.00016742636216804385,
+            0.007117419969290495,
+            0.006062475498765707,
+            -0.027953723445534706,
+            0.023254845291376114,
+            -0.029980691149830818,
+            0.00029051280580461025,
+            0.03611687570810318,
+            -0.016105176880955696,
+            0.000605211011134088,
+            0.007200341206043959,
+            -0.0022573047317564487,
+            0.016722481697797775,
+            -0.017616190016269684,
+            -0.024673722684383392,
+            0.02120945043861866,
+            0.022186079993844032,
+            0.013663602992892265,
+            0.03326069191098213,
+            -0.020656641572713852,
+            0.02082248404622078,
+            -0.011940681375563145,
+            -0.009420791640877724,
+            0.004238204564899206,
+            0.0032753951381891966,
+            -0.01591169461607933,
+            0.018270347267389297,
+            -0.017708323895931244,
+            -0.0023909001611173153,
+            -0.004590620752424002,
+            -0.016040682792663574,
+            -0.0009904500329867005,
+            -0.010319107212126255,
+            -0.009370117448270321,
+            -0.0026028105057775974,
+            -0.013921581208705902,
+            0.10665534436702728,
+            0.0005424441187642515,
+            -0.0137557378038764,
+            0.014824503101408482,
+            -0.0033352829050272703,
+            -0.014492817223072052,
+            -0.004192137159407139,
+            0.001406208612024784,
+            -0.00640337448567152,
+            -0.007882139645516872,
+            -0.013608322478830814,
+            -0.0004266996693331748,
+            0.001249579363502562,
+            0.01026382576674223,
+            -0.009139780886471272,
+            -0.029188331216573715,
+            0.009420791640877724,
+            -0.02970428578555584,
+            0.01204202976077795,
+            0.005081238690763712,
+            0.015091693960130215,
+            0.006647531874477863,
+            0.009978207759559155,
+            -0.007177307270467281,
+            -0.013737311586737633,
+            0.031178444623947144,
+            -0.007098992820829153,
+            0.0022791866213083267,
+            -0.0024254508316516876,
+            -0.010835062712430954,
+            -0.012640906497836113,
+            -0.02196495607495308,
+            0.024950126186013222,
+            -0.01616045832633972,
+            0.007181914057582617,
+            0.006338879931718111,
+            0.00218590022996068,
+            0.030164960771799088,
+            0.0008608853095211089,
+            -0.02006697840988636,
+            0.00620989128947258,
+            -0.035766761749982834,
+            -0.005260901991277933,
+            0.019937988370656967,
+            -0.014345401898026466,
+            -0.04079732671380043,
+            -0.008923263289034367,
+            -0.009116746485233307,
+            0.0030335411429405212,
+            0.00068352569360286,
+            -0.014686300419270992,
+            0.003503429004922509,
+            -0.003699215594679117,
+            0.018657313659787178,
+            0.008826522156596184,
+            -0.0077485437504947186,
+            0.002402417128905654,
+            -0.0002808674471452832,
+            0.006454048678278923,
+            -0.013313490897417068,
+            0.005776857025921345,
+            -0.019827427342534065,
+            0.018509898334741592,
+            -0.008144724182784557,
+            -0.0204355176538229,
+            0.005095059052109718,
+            0.014253266155719757,
+            0.006592250894755125,
+            -0.003673878498375416,
+            -0.008200004696846008,
+            0.014068996533751488,
+            0.023678665980696678,
+            0.014124277979135513,
+            0.018657313659787178,
+            0.03781215474009514,
+            0.001152837765403092,
+            0.032247208058834076,
+            -0.02255461923778057,
+            0.018592819571495056,
+            -0.04116586595773697,
+            -0.0009167420794256032,
+            -0.009932140819728374,
+            -0.005638654809445143,
+            -0.0016238773241639137,
+            0.02301529422402382,
+            -0.014013716019690037,
+            0.0029644398018717766,
+            0.015985403209924698,
+            -0.010503376834094524,
+            0.012410569004714489,
+            0.01132337749004364,
+            -0.01828877441585064,
+            0.013479333370923996,
+            -0.0068502286449074745,
+            0.004818654619157314,
+            -0.027916869148612022,
+            0.005832138005644083,
+            -0.004450114909559488,
+            -0.005108879413455725,
+            -0.0015386525774374604,
+            -0.0016803100006654859,
+            -0.008273712359368801,
+            -0.009282589890062809,
+            -0.035011254251003265,
+            -0.005270115565508604,
+            -5.72963799641002e-05,
+            -0.006970004178583622,
+            0.024821138009428978,
+            0.029096195474267006,
+            0.011811692267656326,
+            0.00695157703012228,
+            0.000879312283359468,
+            0.012815962545573711,
+            0.005578767042607069,
+            0.0234022606164217,
+            -0.02579776756465435,
+            0.00412764260545373,
+            -0.007140453439205885,
+            -0.013672816567122936,
+            0.013728098012506962,
+            -0.004323429428040981,
+            -0.013350344263017178,
+            0.027769453823566437,
+            -0.02026967518031597,
+            0.007605734746903181,
+            -0.00029713500407524407,
+            -0.002948316279798746,
+            -0.0031302827410399914,
+            -0.01794787496328354,
+            0.0015881750732660294,
+            -0.00242314743809402,
+            -0.026387430727481842,
+            -0.0030128106009215117,
+            -0.008540904149413109,
+            -0.02244405820965767,
+            0.02310742810368538,
+            0.01703573949635029,
+            -0.01623416692018509,
+            0.0262584425508976,
+            -0.006808768026530743,
+            -0.013700457289814949,
+            -0.01228157989680767,
+            -0.0017805066891014576,
+            -0.004086181987076998,
+            0.009720230475068092,
+            -0.019937988370656967,
+            -0.011461579240858555,
+            0.0015121637843549252,
+            -0.02426832914352417,
+            0.015598435886204243,
+            0.017588550224900246,
+            -0.013433266431093216,
+            0.029483163729310036,
+            -0.0046320813708007336,
+            0.008287533186376095,
+            0.017800459638237953,
+            0.009738657623529434,
+            -0.019735291600227356,
+            -0.022609900683164597,
+            0.012788321822881699,
+            0.0051687671802937984,
+            -0.0065646106377244,
+            0.00027035828679800034,
+            -0.02472900226712227,
+            0.0009691438172012568,
+            0.007711689919233322,
+            0.001163778710179031,
+            0.0027133722323924303,
+            0.0036485414020717144,
+            0.0014119670959189534,
+            0.0021525011397898197,
+            -0.007587307598441839,
+            0.006131576374173164,
+            -0.02826698310673237,
+            0.013414839282631874,
+            0.004325732588768005,
+            0.002100675366818905,
+            -0.03475327789783478,
+            -0.001816208939999342,
+            -0.0011989051708951592,
+            -0.00821843184530735,
+            -0.012106523849070072,
+            0.023899788036942482,
+            0.021264731884002686,
+            0.015036413446068764,
+            0.025871476158499718,
+            -0.010816635563969612,
+            0.013746525160968304,
+            -0.011212815530598164,
+            0.018500683829188347,
+            -0.0041253394447267056,
+            -0.010328320786356926,
+            -0.008292139507830143,
+            0.03777530416846275,
+            -0.010171690955758095,
+            0.01260405220091343,
+            0.0024830352049320936,
+            0.023733945563435555,
+            0.0015098603907972574,
+            0.0005044384743086994,
+            0.002237726002931595,
+            -0.008697533048689365,
+            -0.0059196664951741695,
+            0.007513599935919046,
+            0.06983824074268341,
+            -0.011028545908629894,
+            0.005744609981775284,
+            0.010448095388710499,
+            -0.007637981791049242,
+            -0.006034834776073694,
+            -0.005763037130236626,
+            -0.030422938987612724,
+            0.01685146987438202,
+            -0.0017344391671940684,
+            -0.01920090988278389,
+            0.013580681756138802,
+            0.017505627125501633,
+            0.018030796200037003,
+            0.000927107292227447,
+            0.017726751044392586,
+            -0.005472812335938215,
+            0.0155800087377429,
+            0.02703237533569336,
+            0.009411578066647053,
+            -0.008153936825692654,
+            -0.009545174427330494,
+            0.007016071584075689,
+            0.0010779781732708216,
+            -0.01228157989680767,
+            0.003489608643576503,
+            -0.015229896642267704,
+            0.026866532862186432,
+            -0.006025621201843023,
+            0.04182923585176468,
+            -0.012097310274839401,
+            -0.00623292475938797,
+            -0.0011344107333570719,
+            -0.007260228972882032,
+            0.011885399930179119,
+            0.005629441235214472,
+            0.008748207241296768,
+            -0.0021835968364030123,
+            -0.014262479729950428,
+            0.018040010705590248,
+            0.002409327309578657,
+            -0.00040021090535447,
+            0.004966070409864187,
+            -0.00466893520206213,
+            -0.00474955327808857,
+            0.012622479349374771,
+            0.009913713671267033,
+            -0.008614611811935902,
+            0.005399104207754135,
+            -0.007149667013436556,
+            -0.015561581589281559,
+            -0.01065079215914011,
+            0.002605113899335265,
+            -0.008734387345612049,
+            -0.007370790932327509,
+            0.0038604517467319965,
+            0.020472371950745583,
+            0.007168094161897898,
+            0.012134164571762085,
+            -0.020122257992625237,
+            -0.0006063627079129219,
+            0.017266077920794487,
+            -0.010448095388710499,
+            0.001591630163602531,
+            0.004058541730046272,
+            0.011820905841886997,
+            -0.013147647492587566,
+            0.006200677715241909,
+            0.010144051164388657,
+            0.0037153391167521477,
+            0.01618809998035431,
+            0.006154610309749842,
+            0.003945676609873772,
+            0.004721913021057844,
+            -0.0014246356440708041,
+            0.0003927249344997108,
+            -0.009264162741601467,
+            1.6843408957356587e-05,
+            -0.005025957711040974,
+            -0.0023102823179215193,
+            -0.015451019629836082,
+            0.009738657623529434,
+            0.012217085808515549,
+            0.01680540293455124,
+            -0.026626981794834137,
+            0.011959107592701912,
+            -0.010503376834094524,
+            0.02741934172809124,
+            0.018021583557128906,
+            -0.02618473395705223,
+            0.016105176880955696,
+            -0.01079820841550827,
+            0.015626076608896255,
+            0.0018208156106993556,
+            -0.01675933599472046,
+            0.004528429359197617,
+            -0.004590620752424002,
+            0.004249721299856901,
+            -0.01833484135568142,
+            -0.024305181577801704,
+            0.021651698276400566,
+            0.0011275006690993905,
+            0.0155800087377429,
+            0.005205621011555195,
+            0.013700457289814949,
+            0.0007929358398541808,
+            0.01787416823208332,
+            0.007333936635404825,
+            0.005366857163608074,
+            -0.011397085152566433,
+            0.008909443393349648,
+            -0.0047910138964653015,
+            -0.007969667203724384,
+            -0.006785734090954065,
+            -0.010567870922386646,
+            -0.0005983008886687458,
+            -0.01065079215914011,
+            -0.024986980482935905,
+            -0.018565179780125618,
+            -0.007826858200132847,
+            0.02378922700881958,
+            -0.000312682765070349,
+            -0.0018092987593263388,
+            -0.03547193109989166,
+            -0.001534045790322125,
+            -0.003911125939339399,
+            -0.01997484266757965,
+            -0.003017417388036847,
+            -0.0205460786819458,
+            -0.017570123076438904,
+            -0.00611314969137311,
+            -0.02399192377924919,
+            -0.005035171285271645,
+            -0.018122931942343712,
+            -0.01428090687841177,
+            0.006435621529817581,
+            0.005002924241125584,
+            0.01265011914074421,
+            0.028617095202207565,
+            0.01150764711201191,
+            -0.007347756996750832,
+            -0.014686300419270992,
+            -0.01299101859331131,
+            -0.014197985641658306,
+            0.011258882470428944,
+            0.0096465228125453,
+            0.013995288871228695,
+            -1.3226394912635442e-05,
+            0.010844276286661625,
+            0.00018714898033067584,
+            0.02026967518031597,
+            -0.02924361266195774,
+            -0.03038608469069004,
+            0.0026511813048273325,
+            -0.000183118085260503,
+            -0.004219777882099152,
+            -0.0033053390216082335,
+            -0.005500452592968941,
+            -0.00932405050843954,
+            -0.015165401622653008,
+            -0.006292812526226044,
+            2.4365357603528537e-05,
+            -0.006638318300247192,
+            0.024526305496692657,
+            0.020103830844163895,
+            0.006007194519042969,
+            0.0016423043562099338,
+            0.01881394349038601,
+            0.006804161239415407,
+            0.00805719569325447,
+            -0.00289073190651834,
+            0.0043050022795796394,
+            0.0062283179722726345,
+            0.005131912883371115,
+            -0.004648204892873764,
+            0.02905934303998947,
+            0.0014799166237935424,
+            0.011995961889624596,
+            0.014446749351918697,
+            0.03256046772003174,
+            -0.0013359558070078492,
+            0.025963610038161278,
+            0.012161804363131523,
+            -0.009655735455453396,
+            -0.01611439138650894,
+            0.008301353082060814,
+            -0.008135510608553886,
+            0.006251351907849312,
+            0.01632630079984665,
+            -0.02071192115545273,
+            -0.007200341206043959,
+            0.03394249081611633,
+            -0.01389394048601389,
+            -0.012788321822881699,
+            0.017745178192853928,
+            0.007370790932327509,
+            -0.003231631126254797,
+            -0.0008505201549269259,
+            0.004609047435224056,
+            0.001758624566718936,
+            -0.004152979701757431,
+            0.019348325207829475,
+            0.004309609066694975,
+            0.003989440388977528,
+            0.012530344538390636,
+            -0.0019970235880464315,
+            -0.0076103415340185165,
+            -0.0016745515167713165,
+            0.01662113331258297,
+            0.004222081042826176,
+            -0.017791246995329857,
+            -0.006887082476168871,
+            0.013811019249260426,
+            -0.0041575864888727665,
+            -0.004590620752424002,
+            0.025152822956442833,
+            -0.002501462120562792,
+            -0.012576411478221416,
+            0.0011338349431753159,
+            0.006624497938901186,
+            -0.003643934614956379,
+            0.011986748315393925,
+            0.023439114913344383,
+            -0.003038147697225213,
+            -0.0018116021528840065,
+            -0.009056858718395233,
+            0.007923600263893604,
+            -0.008513263426721096,
+            0.013424052856862545,
+            -0.029372600838541985,
+            0.01002427563071251,
+            -0.012327647767961025,
+            0.005095059052109718,
+            0.012318434193730354,
+            0.01768989861011505,
+            0.0204355176538229,
+            0.013801805675029755,
+            -0.007043711841106415,
+            -0.011737984605133533,
+            -0.012917310930788517,
+            -0.0013843266060575843,
+            -0.009172027930617332,
+            -0.01901664026081562,
+            0.004970677196979523,
+            0.019937988370656967,
+            0.017127875238656998,
+            0.009259555488824844,
+            -0.015027199871838093,
+            0.019993269816040993,
+            -2.9781880584778264e-05,
+            0.007711689919233322,
+            0.0028976420871913433,
+            -0.01280674897134304,
+            -0.0020476977806538343,
+            -0.015828773379325867,
+            0.016400009393692017,
+            -0.014326974749565125,
+            0.01628023386001587,
+            0.030349230393767357,
+            -0.01337798498570919,
+            -0.013967648148536682,
+            -0.014262479729950428,
+            -0.00429118238389492,
+            -0.019035067409276962,
+            0.026442712172865868,
+            0.018076865002512932,
+            -0.016980459913611412,
+            -0.014741581864655018,
+            -0.019937988370656967,
+            0.01920090988278389,
+            0.015045627020299435,
+            0.007108206395059824,
+            0.01921933703124523,
+            0.02026967518031597,
+            0.006182250566780567,
+            0.0036761818919330835,
+            -0.01094562467187643,
+            0.0010889191180467606,
+            0.017339784651994705,
+            0.0016388492658734322,
+            -0.011111467145383358,
+            -0.017044954001903534,
+            0.0039963508024811745,
+            -0.02109888754785061,
+            0.008831128478050232,
+            0.006366520654410124,
+            -0.011480006389319897,
+            0.016040682792663574,
+            -0.024544732645154,
+            0.0073892176151275635,
+            -0.012023602612316608,
+            0.0029736533761024475,
+            -0.036724966019392014,
+            0.0002545226307120174,
+            0.003602473996579647,
+            0.006246745120733976,
+            0.023365406319499016,
+            -0.02990698255598545,
+            -0.01733057200908661,
+            0.003339889459311962,
+            -0.006435621529817581,
+            0.003643934614956379,
+            -0.014916637912392616,
+            0.003385957097634673,
+            -0.00484629487618804,
+            -0.0011177113046869636,
+            0.0036900020204484463,
+            -0.003945676609873772,
+            0.009319443255662918,
+            0.015128548257052898,
+            -0.017026526853442192,
+            -0.00016339545254595578,
+            0.017643829807639122,
+            0.014446749351918697,
+            -0.02255461923778057,
+            0.006688992492854595,
+            -0.013626749627292156,
+            0.006454048678278923,
+            -0.027069229632616043,
+            0.011222029104828835,
+            -0.005302362609654665,
+            0.01481528952717781,
+            -0.0031302827410399914,
+            -0.004351069685071707,
+            -0.0076425885781645775,
+            0.014713941141963005,
+            0.004574496764689684,
+            0.007568880915641785,
+            0.017201581969857216,
+            -0.018297987058758736,
+            0.012364501133561134,
+            -0.007158880587667227,
+            -0.016473717987537384,
+            0.006665959022939205,
+            -0.00675809383392334,
+            0.002443877747282386,
+            -0.02331012487411499,
+            -0.005440564826130867,
+            -0.0011453517945483327,
+            -0.019643157720565796,
+            0.008849555626511574,
+            -0.02999911829829216,
+            0.0001314361725235358,
+            -0.01700809970498085,
+            -0.0031072490382939577,
+            -0.007591914385557175,
+            -0.011655062437057495,
+            -0.02915147691965103,
+            -0.009406971745193005,
+            0.023328552022576332,
+            -0.009729444049298763,
+            0.011774837970733643,
+            0.0009916017297655344,
+            0.007914386689662933,
+            0.00585517194122076,
+            -0.0007952392334118485,
+            0.008522477000951767,
+            0.016704054549336433,
+            -0.014400682412087917,
+            -0.0012645513052120805,
+            0.005353036802262068,
+            0.024600014090538025,
+            -0.019422033801674843,
+            -0.01318450178951025,
+            -0.00865146517753601,
+            0.0019198606023564935,
+            0.017643829807639122,
+            -0.02102518081665039,
+            -0.002324102446436882,
+            -0.001895675202831626,
+            0.005486632231622934,
+            -0.006841015070676804,
+            -0.01441910956054926,
+            -0.009609668515622616,
+            -0.00790056586265564,
+            -0.0009622336947359145,
+            -0.019329898059368134,
+            0.01910877414047718,
+            -0.020490799099206924,
+            0.003613990731537342,
+            0.008821914903819561,
+            -0.010890343226492405,
+            -0.005348430015146732,
+            -0.016676414757966995,
+            0.024120911955833435,
+            0.011885399930179119,
+            -0.002489945152774453,
+            0.005002924241125584,
+            -0.00041950165177695453,
+            0.013221356086432934,
+            -0.006338879931718111,
+            0.003779833670705557,
+            0.0003184411907568574,
+            -0.01150764711201191,
+            -0.005707756150513887,
+            0.023328552022576332,
+            -0.007587307598441839,
+            0.021172596141695976,
+            -0.0018023886950686574,
+            0.005417531356215477,
+            0.004599833860993385,
+            -0.004627474583685398,
+            0.0080479821190238,
+            -0.003604777390137315,
+            -0.008577757515013218,
+            -0.019716864451766014,
+            0.02472900226712227,
+            -0.022573046386241913,
+            -0.0025820801965892315,
+            0.025171250104904175,
+            0.0016941301291808486,
+            -0.001815057243220508,
+            -0.012263152748346329,
+            -0.011949894018471241,
+            0.025337092578411102,
+            0.011378658004105091,
+            0.004256631713360548,
+            -0.014502030797302723,
+            -0.007269442547112703,
+            0.0017148605547845364,
+            -0.013506974093616009,
+            0.01222629938274622,
+            -0.012152590788900852,
+            0.021670125424861908,
+            -0.013875513337552547,
+            0.024231474846601486,
+            0.010641579516232014,
+            0.01586562767624855,
+            0.016031470149755478,
+            -0.019053494557738304,
+            0.0022711248602718115,
+            -0.006421801634132862,
+            0.008181577548384666,
+            0.0009587786626070738,
+            -0.026866532862186432,
+            -0.0029229791834950447,
+            0.009453038685023785,
+            0.0037268560845404863,
+            -0.01805843785405159,
+            0.00776697089895606,
+            -0.015322031453251839,
+            0.019716864451766014,
+            0.007301689591258764,
+            -0.002683428581804037,
+            -0.010917983949184418,
+            0.0028653948102146387,
+            -0.001710253767669201,
+            0.008568543940782547,
+            0.0005744034424424171,
+            0.008738993667066097,
+            0.013230569660663605,
+            0.01890607737004757,
+            -0.02883821912109852,
+            -0.0006230621365830302,
+            -0.005790677387267351,
+            -0.027364060282707214,
+            -0.008305959403514862,
+            0.009471465833485126,
+            0.016704054549336433,
+            -0.0007359273731708527,
+            -0.0004033780423924327,
+            -0.004583710338920355,
+            0.0028193274047225714,
+            0.0007566577405668795,
+            0.0004030901181977242,
+            0.013396412134170532,
+            -0.010954837314784527,
+            0.011102253571152687,
+            0.00021853242651559412,
+            -0.0061361831612885,
+            -0.016169672831892967,
+            0.02588990330696106,
+            -0.006122363265603781,
+            0.019993269816040993,
+            0.02447102591395378,
+            0.012576411478221416,
+            0.004205957520753145,
+            -0.014704727567732334,
+            0.019274618476629257,
+            0.012742254883050919,
+            -0.0029943836852908134,
+            -0.017339784651994705,
+            0.0024669114500284195,
+            0.0042612385004758835,
+            0.0005887994775548577,
+            -0.019993269816040993,
+            0.007016071584075689,
+            0.0008320931810885668,
+            -0.008278319612145424,
+            -0.020030124112963676,
+            -0.016842257231473923,
+            -0.024047205224633217,
+            0.02244405820965767,
+            0.0001376984582748264,
+            -0.014575738459825516,
+            -0.012244726531207561,
+            0.00013928202679380774,
+            0.006444835104048252,
+            -0.006887082476168871,
+            -0.005532699637115002,
+            0.009075285866856575,
+            -0.013442480005323887,
+            -0.00310494564473629,
+            0.0066291047260165215,
+            0.009070679545402527,
+            -0.0072418018244206905,
+            -0.01098247803747654,
+            -0.01785574108362198,
+            0.025742486119270325,
+            -0.009259555488824844,
+            0.01648293063044548,
+            0.0009034976828843355,
+            -0.0011361383367329836,
+            0.00893708411604166,
+            0.0072418018244206905,
+            0.005592587403953075,
+            0.021430574357509613,
+            0.010521803982555866,
+            0.01150764711201191,
+            0.0020384842064231634,
+            -0.022112371399998665,
+            0.014115064404904842,
+            -0.007269442547112703,
+            -0.009218094870448112,
+            -0.006182250566780567,
+            -0.0015225289389491081,
+            0.004470845218747854,
+            0.011046973057091236,
+            0.018979785963892937,
+            0.025355519726872444,
+            -0.006652138661593199,
+            -0.004754160065203905,
+            0.01228157989680767,
+            -0.0004039538907818496,
+            0.008660678751766682,
+            -0.0034435412380844355,
+            -0.002821630798280239,
+            -0.004199047107249498,
+            0.0061960709281265736,
+            -0.019127201288938522,
+            -0.005514272954314947,
+            0.008835735730826855,
+            -0.007297082804143429,
+            0.03232091665267944,
+            -0.0140966372564435,
+            -0.02417619340121746,
+            -0.00136705138720572,
+            0.002978260163217783,
+            -0.017496414482593536,
+            -0.006970004178583622,
+            0.016132818534970284,
+            0.015607649460434914,
+            0.01041124202311039,
+            -0.0027133722323924303,
+            0.024286756291985512,
+            0.005790677387267351,
+            0.025926755741238594,
+            -0.00041460696957074106,
+            -0.004364890046417713,
+            -0.022573046386241913,
+            0.017431920394301414,
+            -0.0003259271616116166,
+            0.01357146818190813,
+            0.027069229632616043,
+            0.008292139507830143,
+            0.006357307080179453,
+            0.00783146545290947,
+            0.012253939174115658,
+            -0.000859157822560519,
+            0.00014316897431854159,
+            -0.01663956046104431,
+            -0.013460906222462654,
+            -0.01568135805428028,
+            -0.002111040521413088,
+            0.010908770374953747,
+            0.011820905841886997,
+            0.004827867727726698,
+            0.0028976420871913433,
+            0.012143378145992756,
+            0.008550116792321205,
+            -0.007955847308039665,
+            -0.024802710860967636,
+            -0.00040740895201452076,
+            -0.026221588253974915,
+            0.009950567036867142,
+            0.007476745638996363,
+            -0.0007117419736459851,
+            0.0013705063611268997,
+            0.0015248323325067759,
+            0.003427417716011405,
+            -0.0005240171449258924,
+            -0.0034135973546653986,
+            -0.004279665183275938,
+            0.011102253571152687,
+            -0.006887082476168871,
+            -0.003547193016856909,
+            -0.00171946722548455,
+            0.03639327734708786,
+            -0.019422033801674843,
+            0.006960790604352951,
+            -0.011286523193120956,
+            0.009904500097036362,
+            0.005081238690763712,
+            0.014704727567732334,
+            -0.004265845287591219,
+            -0.009277982637286186,
+            -0.016022255644202232,
+            0.003584047080948949,
+            -0.016860684379935265,
+            0.014299334026873112,
+            0.0008084836299531162,
+            -0.018804728984832764,
+            0.01742270588874817,
+            -0.003095732070505619,
+            0.007720903493463993,
+            -0.0006224862881936133,
+            -0.01003348920494318,
+            0.013838659971952438,
+            0.008144724182784557,
+            0.03552721068263054,
+            -0.01714630238711834,
+            0.023715518414974213,
+            -0.015220683068037033,
+            0.020011696964502335,
+            0.020030124112963676,
+            -0.0013014052528887987,
+            -0.006504722870886326,
+            0.0013877816963940859,
+            0.013940008357167244,
+            -0.03057035431265831,
+            -0.008319780230522156,
+            0.010208545252680779,
+            -0.015091693960130215,
+            -0.011065399274230003,
+            -0.015837986022233963,
+            0.0012138771126046777,
+            0.00650011608377099,
+            -0.011028545908629894,
+            -0.0007267139153555036,
+            0.004288878757506609,
+            0.01742270588874817,
+            -0.008992364630103111,
+            0.003961800131946802,
+            -0.024194620549678802,
+            0.005085845477879047,
+            0.02283102460205555,
+            0.007011464796960354,
+            -0.006868655793368816,
+            0.019827427342534065,
+            0.012954164296388626,
+            -0.007154273800551891,
+            -0.006278992164880037,
+            -0.02273888885974884,
+            -0.015460233204066753,
+            -0.0240840595215559,
+            -0.0005948458565399051,
+            -0.01744113303720951,
+            -0.003581743687391281,
+            0.00019549870921764523,
+            0.0033721367362886667,
+            -0.0007203796412795782,
+            -0.01997484266757965,
+            0.018159786239266396,
+            0.01552472822368145,
+            0.002715675625950098,
+            -0.012557984329760075,
+            0.015626076608896255,
+            0.0234022606164217,
+            -0.01796630211174488,
+            -0.0004338977159932256,
+            0.0029874735046178102,
+            -0.017984729260206223,
+            0.01313843484967947,
+            0.004966070409864187,
+            -0.001163778710179031,
+            -0.022683609277009964,
+            -0.0006858290289528668,
+            0.009821578860282898,
+            0.015358884818851948,
+            0.004360283259302378,
+            0.005187193863093853,
+            -0.007168094161897898,
+            -0.000966840423643589,
+            -0.00818618480116129,
+            -5.551486538024619e-06,
+            -0.013553041033446789,
+            0.007025284692645073,
+            -0.008844949305057526,
+            0.022130798548460007,
+            -0.01958787627518177,
+            0.004318822640925646,
+            -0.02600046433508396,
+            -0.0011966017773374915,
+            0.003358316607773304,
+            0.009913713671267033,
+            -0.009738657623529434,
+            -0.009655735455453396,
+            -0.003991744015365839,
+            0.004171406850218773,
+            0.011599781922996044,
+            -0.009148993529379368,
+            0.010319107212126255,
+            -0.02915147691965103,
+            0.005025957711040974,
+            -0.0040078675374388695,
+            0.008762027136981487,
+            -0.023955069482326508,
+            0.009453038685023785,
+            -0.0010797056602314115,
+            -3.174334779032506e-05,
+            0.008932476863265038,
+            -0.01332270447164774,
+            -0.023770799860358238,
+            0.0043994407169520855,
+            -0.0016849166713654995,
+            -0.013589895330369473,
+            -0.03777530416846275,
+            -0.010871916078031063,
+            0.0010428517125546932,
+            0.0075043863616883755,
+            0.01170113030821085,
+            -0.028709229081869125,
+            -0.0003699791559483856,
+            -0.026332149282097816,
+            -0.006288205739110708,
+            -0.0187218077480793,
+            -0.011461579240858555,
+            0.0037452830001711845,
+            0.01931147091090679,
+            0.004984497092664242,
+            0.011922254227101803,
+            0.009959780611097813,
+            0.03009125404059887,
+            0.008444162085652351,
+            -0.0043994407169520855,
+            0.004109215922653675,
+            -0.005744609981775284,
+            -0.004947643261402845,
+            -0.013645176775753498,
+            0.003895002184435725,
+            -0.005145733244717121,
+            -0.0016238773241639137,
+            -0.003293822053819895,
+            0.01074292790144682,
+            0.011083826422691345,
+            0.00013122022210154682,
+            -0.01441910956054926,
+            0.019698437303304672,
+            -0.014225626364350319,
+            -0.010577084496617317,
+            -0.0021340742241591215,
+            -0.013820232823491096,
+            0.015110121108591557,
+            0.015469446778297424,
+            -0.026055745780467987,
+            -0.005730789620429277,
+            0.013543828390538692,
+            0.01279753539711237,
+            0.0054129245691001415,
+            0.009287196211516857,
+            0.01204202976077795,
+            -0.014068996533751488,
+            -0.0045053958892822266,
+            -0.01537731196731329,
+            -0.006527756340801716,
+            -0.0034435412380844355,
+            -0.0004523246898315847,
+            0.013838659971952438,
+            -0.02102518081665039,
+            -0.004643598105758429,
+            0.003116462379693985,
+            0.008296746760606766,
+            -0.0031878668814897537,
+            -0.0023379225749522448,
+            -0.02025124803185463,
+            0.007591914385557175,
+            -0.012640906497836113,
+            0.016906751319766045,
+            -0.0007146212155930698,
+            -0.0019267707830294967,
+            -0.01189461350440979,
+            0.013672816567122936,
+            -0.021154168993234634,
+            0.030035972595214844,
+            -0.011802478693425655,
+            0.002077641664072871,
+            -0.002655788091942668,
+            -0.001952107879333198,
+            0.00790056586265564,
+            0.03256046772003174,
+            -0.004786407109349966,
+            -0.004164496902376413,
+            0.015497087500989437,
+            -0.023549675941467285,
+            0.02541080117225647,
+            0.007974274456501007,
+            0.013433266431093216,
+            0.0006869807257317007,
+            -0.016013043001294136,
+            0.018942931666970253,
+            0.017883380874991417,
+            0.005288542248308659,
+            0.011046973057091236,
+            -0.01570899784564972,
+            0.016105176880955696,
+            0.007158880587667227,
+            -0.001377416541799903,
+            0.011903827078640461,
+            0.019532594829797745,
+            -0.02848810702562332,
+            -0.016842257231473923,
+            0.014391468837857246,
+            -0.018758662045001984,
+            -0.01995641551911831,
+            -0.012576411478221416,
+            -0.011968321166932583,
+            -0.028635522350668907,
+            0.012567197903990746,
+            0.015607649460434914,
+            0.0002802915987558663,
+            0.015229896642267704,
+            0.0030035972595214844,
+            0.017579335719347,
+            -0.01685146987438202,
+            -0.012244726531207561,
+            -0.0032085974235087633,
+            0.008301353082060814,
+            0.006771914195269346,
+            -0.015220683068037033,
+            -0.00785449892282486,
+            0.00334219285286963,
+            -0.027198217809200287,
+            0.008333600126206875,
+            -0.009664949029684067,
+            0.01910877414047718,
+            0.003918035887181759,
+            0.00503977807238698,
+            0.012742254883050919,
+            0.018620459362864494,
+            0.018500683829188347,
+            0.0050443848595023155,
+            -0.0021363776177167892,
+            0.004026294220238924,
+            -0.0035402828361839056,
+            -0.00210873712785542,
+            0.004005563911050558,
+            0.009236522018909454,
+            0.023918215185403824,
+            -0.0328737273812294,
+            -0.0026903385296463966,
+            0.0072418018244206905,
+            -0.0013992985477671027,
+            -0.0042842719703912735,
+            0.018417762592434883,
+            -0.002002781955525279,
+            0.010448095388710499,
+            0.005177980288863182,
+            0.012880456633865833,
+            -0.009231915697455406,
+            0.003026630962267518,
+            0.015589222311973572,
+            0.014769221656024456,
+            0.014548097737133503,
+            0.011802478693425655,
+            0.0009990877006202936,
+            -0.017597762867808342,
+            0.0075043863616883755,
+            0.004067755304276943,
+            -0.009245735593140125,
+            0.004666632041335106,
+            0.010899556800723076,
+            0.014824503101408482,
+            0.004139159806072712,
+            0.010964050889015198,
+            -0.01890607737004757,
+            -0.009010791778564453,
+            0.01429012045264244,
+            0.005970340687781572,
+            -0.0102730393409729,
+            -0.009775510989129543,
+            -0.004189833998680115,
+            0.0031072490382939577,
+            -0.0028354511596262455,
+            -0.009029218927025795,
+            0.005652475170791149,
+            0.002283793408423662,
+            -0.003855844959616661,
+            -0.015810346230864525,
+            0.014981132000684738,
+            0.004643598105758429,
+            0.006186857353895903,
+            0.007803824730217457,
+            0.007684049196541309,
+            -0.007536633405834436,
+            0.018546752631664276,
+            0.0038166875019669533,
+            0.004613654222339392,
+            -0.012447422370314598,
+            -0.00558798061683774,
+            -0.011572141200304031,
+            0.011747198179364204,
+            0.006647531874477863,
+            0.008964723907411098,
+            0.0024047205224633217,
+            -0.001717163948342204,
+            0.007688655983656645,
+            -0.07857263088226318,
+            -0.014713941141963005,
+            0.006583037320524454,
+            0.03526923432946205,
+            -0.0012610962148755789,
+            -0.005118092987686396,
+            0.0045146094635128975,
+            0.009406971745193005,
+            -0.004072362091392279,
+            -0.0013808716321364045,
+            0.02045394480228424,
+            -0.009305623359978199,
+            -0.013442480005323887,
+            -0.002022360684350133,
+            -0.000278132181847468,
+            -0.00224578776396811,
+            0.002814720617607236,
+            0.0028746083844453096,
+            0.028156420215964317,
+            7.521949737565592e-05,
+            0.017220009118318558,
+            -0.006675172131508589,
+            -0.004781800322234631,
+            -0.007559667341411114,
+            -0.008600790984928608,
+            -0.016261806711554527,
+            0.003406687406823039,
+            0.019845854490995407,
+            -0.011240456253290176,
+            0.017026526853442192,
+            0.0056893290020525455,
+            0.0036830920726060867,
+            -0.0019106471445411444,
+            -0.008863375522196293,
+            -0.011793265119194984,
+            -0.005090452264994383,
+            -0.009379331022500992,
+            -0.031104736030101776,
+            0.02073034830391407,
+            -0.002413933863863349,
+            0.010475736111402512,
+            -0.015358884818851948,
+            0.0033928670454770327,
+            -0.024010350927710533,
+            0.025392374023795128,
+            -0.019827427342534065,
+            -0.019182482734322548,
+            0.0067212400026619434,
+            -0.0019866584334522486,
+            0.006730453111231327,
+            -0.01839933544397354,
+            -0.0006184554076753557,
+            -0.02216765284538269,
+            -0.009245735593140125,
+            -0.0126132657751441,
+            0.012862029485404491,
+            -0.013488546945154667,
+            -0.01968001201748848,
+            0.01968001201748848,
+            -0.014594165608286858,
+            -0.015257536433637142,
+            -0.025816194713115692,
+            -0.016123604029417038,
+            0.0055603403598070145,
+            -0.008421128615736961,
+            0.0023183440789580345,
+            -0.008794275112450123,
+            0.005643261596560478,
+            -0.0036692717112600803,
+            -0.002842361107468605,
+            -0.004910789430141449,
+            -0.018482258543372154,
+            -0.0075274198316037655,
+            -0.00802494864910841,
+            -0.008550116792321205,
+            0.010687646456062794,
+            -0.005196407437324524,
+            -0.003959496505558491,
+            -0.004166800063103437,
+            0.010245399549603462,
+            0.002411630703136325,
+            -0.004613654222339392,
+            -0.009411578066647053,
+            -0.012438209727406502,
+            0.006656745448708534,
+            -0.011111467145383358,
+            -0.015699783340096474,
+            -0.018159786239266396,
+            0.015874840319156647,
+            0.016178885474801064,
+            -0.008001914247870445,
+            -0.005426744464784861,
+            0.008808095008134842,
+            0.0013532311422750354,
+            -0.029649006202816963,
+            0.0215779896825552,
+            -0.02073034830391407,
+            0.019661584869027138,
+            0.021338438615202904,
+            0.0036001706030219793,
+            -0.002823934191837907,
+            0.03268945589661598,
+            -0.008172363974153996,
+            0.0012380625121295452,
+            0.013046300038695335,
+            -0.03620900958776474,
+            -0.011608995497226715,
+            -0.0006720087840221822,
+            -0.027087656781077385,
+            -0.015598435886204243,
+            0.018095292150974274,
+            0.0006242138333618641,
+            0.004360283259302378,
+            0.003075001761317253,
+            0.03136271610856056,
+            0.0036462380085140467,
+            -0.006219104863703251,
+            0.005353036802262068,
+            0.00365314818918705,
+            0.010319107212126255,
+            -0.02120945043861866,
+            -0.02972271293401718,
+            -0.007315509952604771,
+            0.0019290741765871644,
+            -0.0026926419232040644,
+            -0.011848545633256435,
+            -0.0004975283518433571,
+            0.008338207378983498,
+            -0.022038664668798447,
+            -0.012539558112621307,
+            0.02474742941558361,
+            -0.01336877141147852,
+            0.018049223348498344,
+            -0.009784724563360214,
+            -0.009968994185328484,
+            0.006472475826740265,
+            0.023918215185403824,
+            -0.016317088156938553,
+            0.001357837812975049,
+            0.0019071920542046428,
+            0.03027552366256714,
+            0.0013267423491925001,
+            -0.017699111253023148,
+            0.013083153404295444,
+            0.012714614160358906,
+            -0.010623152367770672,
+            -0.015966976061463356,
+            -0.008213824592530727,
+            0.017127875238656998,
+            -0.015165401622653008,
+            0.006366520654410124,
+            -0.0054451716132462025,
+            0.0020269674714654684,
+            0.017044954001903534,
+            -0.03068091720342636,
+            0.005629441235214472,
+            0.010254612192511559,
+            -0.00041950165177695453,
+            0.002570563228800893,
+            0.006923936307430267,
+            -0.028045859187841415,
+            0.005403710994869471,
+            -0.035858895629644394,
+            -0.004445508122444153,
+            0.008886409923434258,
+            -0.009618882089853287,
+            0.01842697709798813,
+            -0.00916742067784071,
+            -0.020122257992625237,
+            -0.005201014224439859,
+            0.008757420815527439,
+            0.013248995877802372,
+            -0.006817981600761414,
+            0.014032143168151379,
+            0.011913040652871132,
+            0.0025912935379892588,
+            -0.003731462871655822,
+            -0.012290793471038342,
+            -0.005205621011555195,
+            -0.02227821573615074,
+            -0.014216412790119648,
+            0.006679778918623924,
+            0.006034834776073694,
+            -0.0010117562487721443,
+            0.009563600644469261,
+            -0.022609900683164597,
+            -0.00016785824846010655,
+            0.01988270878791809,
+            0.005901239346712828,
+            0.0011401692172512412,
+            -0.0016676414525136352,
+            0.028985634446144104,
+            0.008734387345612049,
+            -0.008135510608553886,
+            0.0010992843890562654,
+            0.012751467525959015,
+            0.0017206189222633839,
+            0.01447439007461071,
+            0.01424405351281166,
+            0.006444835104048252,
+            -0.009033825248479843,
+            0.0034803953021764755,
+            -0.003321462543681264,
+            0.00697461050003767,
+            0.002355197910219431,
+            -0.01538652554154396,
+            -0.013497760519385338,
+            0.012917310930788517,
+            -0.009462252259254456,
+            0.002360956510528922,
+            0.01862967386841774,
+            -0.006546183489263058,
+            -0.0028193274047225714,
+            0.0020684280898422003,
+            -0.025926755741238594,
+            -0.0016492144204676151,
+            -0.03191552311182022,
+            -0.010125624015927315,
+            -0.014511244371533394,
+            -0.026534846052527428,
+            0.03335282951593399,
+            -0.0021525011397898197,
+            0.02905934303998947,
+            -0.017524054273962975,
+            0.007739330176264048,
+            -0.015220683068037033,
+            0.024950126186013222,
+            0.005067418795078993,
+            0.00015432592772413045,
+            0.01604989729821682,
+            -0.027916869148612022,
+            -0.01566293090581894,
+            0.02550293505191803,
+            0.009609668515622616,
+            -0.01002427563071251,
+            0.036172155290842056,
+            0.02082248404622078,
+            0.0077485437504947186,
+            -0.006168430671095848,
+            0.023439114913344383,
+            0.0010319106513634324,
+            0.007260228972882032,
+            -0.010660005733370781,
+            0.013847872614860535,
+            -0.012686973437666893,
+            -0.0030151139944791794,
+            0.046767666935920715,
+            0.004477755166590214,
+            -0.0008827673736959696,
+            -0.013801805675029755,
+            0.010844276286661625,
+            -0.0060578687116503716,
+            0.015976188704371452,
+            -0.00862843170762062,
+            0.031583838164806366,
+            -0.005288542248308659,
+            -0.016980459913611412,
+            0.019753718748688698,
+            -0.01442832313477993,
+            0.04374564439058304,
+            -0.02561349794268608,
+            0.003968710079789162,
+            -0.012493490241467953,
+            -0.024544732645154,
+            0.013672816567122936,
+            -0.007707083132117987,
+            -0.003770620096474886,
+            0.008338207378983498,
+            -0.010733714327216148,
+            -0.010715287178754807,
+            0.011341804638504982,
+            -0.010116410441696644,
+            -0.006675172131508589,
+            -0.017275290563702583,
+            -0.009858432225883007,
+            -0.018528325483202934,
+            -0.002324102446436882,
+            -0.007725509814918041,
+            -0.00027582881739363074,
+            -0.00900618452578783,
+            -0.008490229956805706,
+            0.004222081042826176,
+            -0.00027122205938212574,
+            -0.004804834257811308,
+            0.0006040593143552542,
+            -0.003789047012105584,
+            0.00511348620057106,
+            -0.01486135646700859,
+            0.01639079675078392,
+            0.015423379838466644,
+            0.017477987334132195,
+            0.005680115427821875,
+            0.010577084496617317,
+            -0.006103936117142439,
+            -0.016409222036600113,
+            0.0006558852037414908,
+            0.01698967255651951,
+            0.01151686068624258,
+            0.012014389038085938,
+            0.019237764179706573,
+            -0.001894523506052792,
+            0.0007935116882435977,
+            -0.004219777882099152,
+            -0.005832138005644083,
+            -0.01223551295697689,
+            -0.009563600644469261,
+            -0.007587307598441839,
+            -0.02319956384599209,
+            0.0036969122011214495,
+            -0.0010906467214226723,
+            0.0013405625941231847,
+            0.0030427544843405485,
+            0.004763373639434576,
+            0.0072187683545053005,
+            0.001698736916296184,
+            -0.01685146987438202,
+            -0.003547193016856909,
+            -0.002957529854029417,
+            -0.019514167681336403,
+            -0.004247418139129877,
+            0.008702139370143414,
+            -0.010107196867465973,
+            -0.0016250290209427476,
+            0.012382928282022476,
+            0.0025521363131701946,
+            0.016934391111135483,
+            0.012640906497836113,
+            -0.006513936445116997,
+            -0.0006904358160682023,
+            0.009208881296217442,
+            -0.01462180633097887,
+            -0.012309220619499683,
+            -0.025355519726872444,
+            -0.017063381150364876,
+            -0.002549832919612527,
+            0.018749449402093887,
+            -0.00759652117267251,
+            -0.00880348775535822,
+            0.003574833506718278,
+            -0.009425398893654346,
+            0.00511348620057106,
+            -0.021559562534093857,
+            -0.0007624161662533879,
+            -0.020785629749298096,
+            0.005634048022329807,
+            -0.007052925415337086,
+            0.0036393278278410435,
+            0.009821578860282898,
+            0.034347884356975555,
+            0.0023678664583712816,
+            -0.013470119796693325,
+            -0.01265011914074421,
+            0.010881129652261734,
+            0.013212142512202263,
+            -0.013811019249260426,
+            -0.0020949169993400574,
+            -0.008324386551976204,
+            -0.002998990472406149,
+            0.0010509134735912085,
+            0.014492817223072052,
+            0.026203161105513573,
+            -0.01947731524705887,
+            -0.014713941141963005,
+            -0.018270347267389297,
+            -0.024323608726263046,
+            0.003708429168909788,
+            0.009637309238314629,
+            0.02082248404622078,
+            -0.00788674596697092,
+            -0.009148993529379368,
+            -0.005707756150513887,
+            -0.009047646075487137,
+            0.007545846980065107,
+            0.11542658507823944,
+            0.005772250704467297,
+            -0.001601995318196714,
+            -0.01462180633097887,
+            -0.004254328086972237,
+            0.010595511645078659,
+            -0.012760681100189686,
+            0.0664476752281189,
+            -0.013064726255834103,
+            0.005730789620429277,
+            -0.007467532530426979,
+            -0.0006184554076753557,
+            -0.004480058792978525,
+            -0.010973264463245869,
+            0.00079120829468593,
+            -0.018482258543372154,
+            0.000761840317863971,
+            -0.011295736767351627,
+            -0.0063204532489180565,
+            -0.004611351061612368,
+            -0.011885399930179119,
+            -0.016786975786089897,
+            0.01614203117787838,
+            -0.03038608469069004,
+            -0.001484523294493556,
+            0.0005410045268945396,
+            -0.020804056897759438,
+            0.003203990636393428,
+            -0.027013948187232018,
+            0.005726182833313942,
+            -0.0008816156769171357,
+            0.013811019249260426,
+            -0.005647868383675814,
+            -0.0204355176538229,
+            -0.013212142512202263,
+            0.008946297690272331,
+            0.009978207759559155,
+            0.01668562740087509,
+            0.017800459638237953,
+            0.005620228126645088,
+            -0.000780843140091747,
+            -0.011295736767351627,
+            0.010558657348155975,
+            -0.01065079215914011,
+            0.0003250633890274912,
+            -0.010890343226492405,
+            0.0018588212551549077,
+            -0.0021755348425358534,
+            0.02342068776488304,
+            -0.011811692267656326,
+            -0.021670125424861908,
+            0.001484523294493556,
+            0.02940945513546467,
+            -0.0034020806197077036,
+            0.017091020941734314,
+            7.133255712687969e-05,
+            0.009231915697455406,
+            -0.003399777226150036,
+            0.0004618261009454727,
+            0.026829678565263748,
+            -0.018454616889357567,
+            0.00623292475938797,
+            0.004625170957297087,
+            0.013672816567122936,
+            -0.008361240848898888,
+            0.01332270447164774,
+            -0.029943836852908134,
+            -0.005095059052109718,
+            -0.020195966586470604,
+            0.009089106693863869,
+            0.027566757053136826,
+            0.0051825870759785175,
+            0.012198658660054207,
+            -0.01899821311235428,
+            -0.012115737423300743,
+            0.0337950736284256,
+            -0.02006697840988636,
+            -0.010558657348155975,
+            -0.010871916078031063,
+            -0.024342035874724388,
+            0.00790977943688631,
+            0.011019332334399223,
+            -0.023457542061805725,
+            -0.015884054824709892,
+            0.004841688089072704,
+            -0.009664949029684067,
+            -0.017597762867808342,
+            0.0017160122515633702,
+            -0.015073266811668873,
+            -0.014409895986318588,
+            -0.005850565154105425,
+            -0.001894523506052792,
+            -0.011940681375563145,
+            0.00858236476778984,
+            -0.02159641683101654,
+            -0.0016446076333522797,
+            -0.0008729780092835426,
+            0.0024553947150707245,
+            0.0030634847935289145,
+            0.00623292475938797,
+            -0.0023770800326019526,
+            -0.005979553796350956,
+            -0.025171250104904175,
+            0.013064726255834103,
+            -0.0036577549763023853,
+            0.034937549382448196,
+            0.019256191328167915,
+            0.004056238103657961,
+            -0.004758766852319241,
+            -0.00841191504150629,
+            0.019993269816040993,
+            -0.02139372006058693,
+            -0.019145628437399864,
+            0.007776184007525444,
+            0.020398663356900215,
+            -0.02331012487411499,
+            0.00395028293132782,
+            -0.009024611674249172,
+            0.008665286004543304,
+            -0.010236185975372791,
+            -0.005247081629931927,
+            0.007587307598441839,
+            0.00011257731239311397,
+            0.0029045522678643465,
+            -0.006647531874477863,
+            0.01785574108362198,
+            0.013193715363740921,
+            -0.01242899615317583,
+            0.02310742810368538,
+            -0.0021064337342977524,
+            0.0019325291505083442,
+            0.0072510153986513615,
+            -0.009351690299808979,
+            -0.00769326277077198,
+            0.012521130964159966,
+            0.00484629487618804,
+            -0.02045394480228424,
+            0.018777089193463326,
+            0.007803824730217457,
+            0.02828541025519371,
+            -0.0014016019413247705,
+            -0.01662113331258297,
+            -0.0068594422191381454,
+            0.004173710476607084,
+            -0.02168855257332325,
+            -0.027953723445534706,
+            0.017072593793272972,
+            -0.012382928282022476,
+            -0.019182482734322548,
+            -0.00895090401172638,
+            0.010558657348155975,
+            -0.020472371950745583,
+            -0.0068502286449074745,
+            0.0029344961512833834,
+            0.0337950736284256,
+            -0.0038650583010166883,
+            0.002782473573461175,
+            0.011673489585518837,
+            0.004118429496884346,
+            0.001210422022268176,
+            -0.021430574357509613,
+            0.0016549729043617845,
+            0.02905934303998947,
+            0.025631925091147423,
+            0.018565179780125618,
+            -0.018841583281755447,
+            0.0005994525854475796,
+            0.009959780611097813,
+            0.009775510989129543,
+            0.006983824074268341,
+            0.008066409267485142,
+            -0.013405625708401203,
+            -0.0031187657732516527,
+            0.009411578066647053,
+            -0.011922254227101803,
+            0.014170344918966293,
+            -0.0013014052528887987,
+            -0.012180231511592865,
+            -0.004152979701757431,
+            -0.020656641572713852,
+            -0.02187282219529152,
+            -0.001980900065973401,
+            -0.005744609981775284,
+            0.0035978672094643116,
+            3.9805148844607174e-05,
+            0.005362250376492739,
+            0.01593933440744877,
+            -0.013884726911783218,
+            0.018076865002512932,
+            0.009932140819728374,
+            0.0033053390216082335,
+            -0.010917983949184418,
+            -0.0363011434674263,
+            0.01785574108362198,
+            -0.01513776183128357,
+            -0.012336861342191696,
+            0.021743832156062126,
+            -0.006679778918623924,
+            0.013783378526568413,
+            -0.021246304735541344,
+            0.0003751617332454771,
+            0.00769326277077198,
+            0.024581586942076683,
+            -0.017643829807639122,
+            0.00568472221493721,
+            0.0062283179722726345,
+            0.0030012938659638166,
+            0.00974787026643753,
+            0.008706746622920036,
+            0.0035287661012262106,
+            0.015515514649450779
+          ],
+          "title": "Appendices A Load-Balancing Loss",
+          "keyphrases": [
+            "Load-Balancing Loss",
+            "hierarchical MoE",
+            "experts",
+            "perplexity",
+            "attention mechanism"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": true,
+      "properties": {
+        "cosine_similarity": 0.9166148659377193
+      }
+    },
+    {
+      "id": "efca763c-d61e-4cce-a449-723c2e48a20e",
+      "type": "cosine_similarity",
+      "source": {
+        "id": "1e687808-8db8-4320-805d-0282a8d679bc",
+        "properties": {
+          "page_content": "Deepseekmoe: Towards Ultimate Expert Specialization In\n\nMixture-Of-Experts Language Models\n\nDamai Dai∗1,2, Chengqi Deng1, Chenggang Zhao∗1,3, R.X. Xu1, Huazuo Gao1, Deli Chen1, Jiashi Li1, Wangding Zeng1, Xingkai Yu∗1,4, Y. Wu1, Zhenda Xie1, Y.K. Li1, Panpan Huang1, Fuli Luo1, Chong Ruan1, Zhifang Sui2, Wenfeng Liang1 1DeepSeek-AI 2National Key Laboratory for Multimedia Information Processing, Peking University 3Institute for Interdisciplinary Information Sciences, Tsinghua University 4National Key Laboratory for Novel Software Technology, Nanjing University {daidamai, szf}@pku.edu.cn, {wenfeng.liang}@deepseek.com https://github.com/deepseek-ai/DeepSeek-MoE\n\nAbstract\n\nIn the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top- out of experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into ones and activating from them, allowing for a more flexible combination of activated experts; (2) isolating experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5× expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations.\n\nFurther, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\n1. Introduction\n\nRecent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023a). It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017) has emerged as a popular solution. It can\n\nenable parameter scaling, while concurrently keeping computational costs at a modest level.\n\nRecent applications of MoE architectures in Transformers (Vaswani et al., 2017) have yielded successful attempts at scaling language models to a substantial size (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022), accompanied with remarkable performance. These achievements underscore the considerable potential and promise of MoE language models.\n\nDespite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts.\n\nThis architecture manifests two potential issues: (1) Knowledge Hybridity: existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. (2) Knowledge Redundancy: tokens assigned to different experts may require common knowledge. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters.\n\nThese issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models.\n\nIn response to the aforementioned issues, we introduce DeepSeekMoE, an innovative MoE architecture specifically designed towards ultimate expert specialization. Our architecture involves two principal strategies: (1) Fine-Grained Expert Segmentation: while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. (2) Shared Expert Isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.\n\nThrough compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized.\n\nStarting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B (Lepikhin et al., 2021) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5× expert parameters and computation. Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE.\n\nThese studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization.\n\nLeveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B (DeepSeek-AI, 2024), a dense model trained on the same 2T corpus. We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B (Touvron et al., 2023b), which has approximately 2.5 times the activated parameters. Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard1. Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. The experimental results still validate its substantial advantages over the GShard architecture consistently. In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nOur contributions are summarized as follows: - Architectural Innovation. We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation.\n\nEmpirical Validation. We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. Experimental results validate the high 1https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models\n\nScalability. We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B.\n\nAlignment for MoE. We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B.\n\nPublic Release. In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization.\n\n2. Preliminaries: Mixture-Of-Experts For Transformers\n\nWe first introduce a generic MoE architecture commonly used in Transformer language models. A standard Transformer language model is constructed by stacking layers of standard Transformer blocks, where each block can be represented as follows:\n\n$$\\mathbf{u}{1:T}^{l}=\\text{Self-Att}\\left(\\mathbf{h}{1:T}^{l-1}\\right)+\\mathbf{h}{1:T}^{l-1},\\tag{1}$$ $$\\mathbf{h}{t}^{l}=\\text{FFN}\\left(\\mathbf{u}{t}^{l}\\right)+\\mathbf{u}{t}^{l},\\tag{2}$$\n\nwhere denotes the sequence length, Self-Att(·) denotes the self-attention module, FFN(·) denotes the Feed-Forward Network (FFN), u 1: ∈ R× are the hidden states of all tokens after the -th attention module, and h\n\n∈ Ris the output hidden state of the -th token after the -th Transformer block. For brevity, we omit the layer normalization in the above formulations.\n\nA typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022). An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. Then, each token will be assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts. If the -th FFN is substituted with an MoE layer, the computation for its output hidden state h\n\nis expressed as: where denotes the total number of experts, FFN(·) is the -th expert FFN, , denotes the gate value for the -th expert, , denotes the token-to-expert affinity, Topk(·, ) denotes the set comprising highest affinity scores among those calculated for the -th token and all experts, and e\n\nis the centroid of the -th expert in the -th layer. Note that ,is sparse, indicating that only out of gate values are nonzero. This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only experts. Also, in the above formulations, we omit the layer normalization operation for brevity.\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{N}\\left(g_{i,t}\\operatorname{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\operatorname{Topk}({s_{j,t}|1\\leqslant j\\leqslant N},K),\\ 0,&\\text{otherwise},\\end{cases}$$ $$s_{i,t}=\\operatorname{Softmax}{i}\\left(\\mathbf{u}{t}^{l}\\,\\mathbf{e}_{i}^{l}\\right),$$\n\n$$(3)$$\n\n(4) $\\binom{4}{5}$ .\n\n3. Deepseekmoe Architecture\n\nOn top of the generic MoE architecture outlined in Section 2, we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. As illustrated in Figure 2, our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. Both of these strategies are designed to elevate the level of expert specialization.\n\n3.1. Fine-Grained Expert Segmentation\n\nIn scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts.\n\nIn pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. The finer expert segmentation enables a more flexible and adaptable combination of activated experts. To be specific, on top of a typical MoE architecture shown in Figure 2(a), we segment each expert FFN into smaller experts by reducing the FFN intermediate hidden dimension to 1 times its original size. Since each expert becomes smaller, in response, we also increase the number of activated experts to times to keep the same computation cost, as illustrated in Figure 2(b). With the fine-grained expert segmentation, the output of an MoE layer can be expressed as:\n\n$$\\mathbf{h}{t}^{i}=\\sum{i=1}^{\\text{mN}}\\left(g_{i,t}\\text{FFN}{i}\\left(\\mathbf{u}{t}^{i}\\right)\\right)+\\mathbf{u}{t}^{i},\\tag{6}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\text{Tokr}(s_{i,t}|1\\leqslant j\\leqslant m\\text{N}),m\\text{K}),\\ 0,&\\text{otherwise},\\end{cases}$$ (7) $$s_{i,t}=\\text{Softmax}{i}\\left(\\mathbf{u}{t}^{T}\\mathbf{e}_{t}^{i}\\right),\\tag{8}$$ where the total number of expert parameters is equal to $N$ times the number of parameters in a\n\n$$(6)$$\n\n(7) (8) $\\frac{1}{2}$\n\nstandard FFN, and denotes the total number of fine-grained experts. With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to .\n\nFrom a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. As an illustrative example, we consider the case where = 16. A typical top-2 routing strategy can yield 16 2 = 120 possible combinations. By contrast, if each expert is split into 4 smaller experts, the fine-grained routing strategy can yield 64 8 = 4, 426, 165, 368 potential combinations. The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition.\n\n3.2. Shared Expert Isolation\n\nWith a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. This alleviation of redundancy will contribute to a more parameterefficient model with more specialized experts.\n\nTowards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate experts to serve as shared experts. Regardless of the router module, each token will be deterministically assigned to these shared experts. In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by , as depicted in Figure 2(c). With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows:\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{K_{s}}\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)+\\sum_{i=K_{s}+1}^{mN}\\left(g_{i,t}\\;\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},\\tag{9}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\mathrm{Topk}{s_{j,t}|K_{s}+1\\leqslant j\\leqslant mN},mK-K_{s}},\\ 0,&\\text{otherwise},\\end{cases}$$ (10) $$s_{i,t}=\\mathrm{Softmax}{i}\\left(\\mathbf{u}{t}^{l,T}e_{i}^{l}\\right).\\tag{11}$$ $\\mathbf{h}\\mathbf{M}\\mathbf{E}$ the number of closed-compact is $K_{s}$ the total number of routed currents.\n\nFinally, in DeepSeekMoE, the number of shared expert is , the total number of routed experts is − , and the number of nonzero gates is − .\n\nIt is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022). The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint.\n\n3.3. Load Balance Consideration\n\nAutomatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. Firstly, there is a risk of routing collapse (Shazeer et al., 2017), i.e., the model always selects only a few experts, preventing other experts from sufficient training.\n\nSecondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.\n\nExpert-Level Balance Loss. In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. The computation of the balance loss is as follows:\n\n$$\\begin{array}{l}{{{\\mathcal L}{\\mathrm{ExpBal}}=\\alpha{1}\\sum_{i=1}^{N^{\\prime}}f_{i}P_{i},}}\\ {{f_{i}=\\frac{N^{\\prime}}{K^{T}}\\sum_{t=1}^{T}\\mathds{1}\\,(\\mathrm{Token}\\;t\\;\\mathrm{s}\\mathrm{lects}\\;\\mathrm{Expert}\\;i),}}\\ {{P_{i}=\\frac{1}{T}\\sum_{t=1}^{T}s_{i,t},}}\\end{array}$$\n\n(12) $$\\begin{array}{l}\\small\\text{(13)}\\end{array}$$ = (14) $$\\begin{array}{l}\\small\\text{(14)}\\end{array}$$ .\n\n7 where 1 is a hyper-parameter called expert-level balance factor, ′is equal to ( − ) and ′ is equal to ( − ) for brevity. 1(·) denotes the indicator function.\n\nDevice-Level Balance Loss. In addition to the expert-level balance loss, we introduce a devicelevel balance loss. When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. Instead, our primary objective is to ensure balanced computation across the devices. If we partition all routed experts into groups {E1, E2, ..., E}, and deploy each group on a single device, the device-level balance loss is computed as follows:\n\n$$\\mathcal{L}{\\text{DevBal}}=\\alpha{2}\\sum_{i=1}^{D}f_{i}^{\\prime}P_{i}^{\\prime},$$ $$f_{i}^{\\prime}=\\frac{1}{|\\mathcal{E}{i}|}\\sum{j\\in\\mathcal{E}{i}}f{j},$$ $$P_{i}^{\\prime}=\\sum_{j\\in\\mathcal{E}{i}}P{j},$$\n\n(15) $$\\begin{array}{l}\\small\\text{(16)}\\end{array}$$ = (17) .\n\nwhere 2 is a hyper-parameter called device-level balance factor. In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices.\n\n4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n6. Alignment For Deepseekmoe 16B\n\nPrevious research indicates that MoE models typically do not emerge significant gains from fine-tuning (Artetxe et al., 2022; Fedus et al., 2021). However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B.\n\n6.1. Experimental Setup\n\nTraining Data. For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios.\n\nHyper-Parameters. During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).\n\nWe employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10−5 without incorporating any learning rate scheduling strategy.\n\nEvaluation Benchmarks. For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3, with the following adjustments: (1) We exclude Pile (Gao et al., 2020) since chat models are seldom employed for pure language modeling. (2) We exclude\n\nMetric # Shot LLaMA2 DeepSeek DeepSeekMoE SFT 7B Chat 7B Chat 16B # Total Params N/A 6.7B 6.9B 16.4B # Activated Params N/A 6.7B 6.9B 2.8B FLOPs per 4K Tokens N/A 187.9T 183.5T 74.4T HellaSwag (Acc.) 0-shot 67.9 71.0 72.2 PIQA (Acc.) 0-shot 76.9 78.4 79.7 ARC-easy (Acc.) 0-shot 69.7 70.2 69.9 ARC-challenge (Acc.) 0-shot 50.8 50.2 50.0 BBH (EM) 3-shot 39.3 43.1 42.2 RACE-middle (Acc.) 5-shot 63.9 66.1 64.8 RACE-high (Acc.) 5-shot 49.6 50.8 50.6 DROP (EM) 1-shot 40.0 41.7 33.8 GSM8K (EM) 0-shot 63.4 62.6 62.2 MATH (EM) 4-shot 13.5 14.7 15.2 HumanEval (Pass@1) 0-shot 35.4 45.1 45.7 MBPP (Pass@1) 3-shot 27.8 39.0 46.2 TriviaQA (EM) 5-shot 60.1 59.5 63.3 NaturalQuestions (EM) 0-shot 35.2 32.7 35.1 MMLU (Acc.) 0-shot 50.0 49.7 47.2 WinoGrande (Acc.) 0-shot 65.1 68.4 69.0 CLUEWSC (EM) 5-shot 48.4 66.2 68.2 CEval (Acc.) 0-shot 35.1 44.7 40.0 CMMLU (Acc.) 0-shot 36.9 51.2 49.3\n\nCHID (Zheng et al., 2019) due to the observed instability of results, hindering the derivation of solid conclusions. (3) We additionally include BBH (Suzgun et al., 2022) to provide a more comprehensive assessment of the reasoning ability of the chat models.\n\nTable 5 | Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations.\n\n6.2. Evaluations\n\nBaselines. In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. Correspondingly, we construct three chat models, including LLaMA2 SFT 7B3, DeepSeek Chat 7B, and DeepSeekMoE Chat 16B.\n\nSubsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks.\n\nResults. The evaluation results are presented in Table 5. Our key observations include: (1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intensive tasks (TriviaQA, NaturalQuestions). (2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. In addition, it also surpasses DeepSeek Chat 7B. (3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1). However, it is worth noting that, after supervised finetuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. (4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations.\n\n7. Deepseekmoe 145B Ongoing\n\nEncouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available.\n\n7.1. Experimental Setup\n\nTraining Data and Tokenization. For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study.\n\nModel Settings. For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps.\n\nSubsequently, the learning rate keeps constant during the remaining training process. The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. Also, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse.\n\nEvaluation Benchmarks. We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3).\n\n7.2. Evaluations\n\nBaselines. Apart from DeepSeekMoE 145B, we consider three additional models for comparison. DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters.\n\nResults. From the evaluation results presented in Table 6, we have the following observations: (1) Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. (2) On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. (3) At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5.\n\n8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\nAppendices A. Overview Of Hyper-Parameters\n\nWe present the overview of hyper-parameters for DeepSeekMoE across various sizes in Table 7.\n\n# Params # Layers Hidden # Attn # Shared # Routed Relative Sequence Batch Size Learning Size Heads Experts Experts Expert Size Length (Sequence) Rate 2.0B 9 1280 10 1 63 (7 activated) 0.25 2048 2048 1.08e-3 16.4B 28 2048 16 2 64 (6 activated) 0.25 4096 4608 4.2e-4 144.6B 62 4096 32 4 128 (12 activated) 0.125 4096 4608 3.0e-4\n\nTable 7 | Overview of hyper-parameters for DeepSeekMoE across various sizes. The relative expert size is in comparison to a standard FFN.\n\nB. Comparing Deepseekmoe With Larger Models\n\nComparisons among DeepSeekMoE, GShard×1.2, and GShard×1.5 are shown in Table 8. Comparisons among DeepSeekMoE, Dense×4, and Dense×16 are shown in Table 9.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 2.3B 2.8B 1.9B # Activated Expert Params N/A 0.28B 0.35B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.824 1.808 1.808 HellaSwag (Acc.) 0-shot 53.7 54.4 54.8 PIQA (Acc.) 0-shot 71.8 71.1 72.3 ARC-easy (Acc.) 0-shot 46.8 47.3 49.4 ARC-challenge (Acc.) 0-shot 31.7 34.1 34.3 RACE-middle (Acc.) 5-shot 43.7 46.4 44.0 RACE-high (Acc.) 5-shot 31.9 32.4 31.7 HumanEval (Pass@1) 0-shot 3.7 3.0 4.9 MBPP (Pass@1) 3-shot 2.4 2.6 2.2 TriviaQA (EM) 5-shot 15.2 15.7 16.6 NaturalQuestions (EM) 5-shot 4.5 4.7 5.7\n\nTable 8 | Comparison between DeepSeekMoE and larger GShard models.\n\nAt a larger scale of 13B total parameters, we also compare DeepSeekMoE with GShard×1.2 and GShard×1.5, and show results in Table 10. At a larger scale, DeepSeekMoE even outperforms GShard×1.5 distinctly.\n\nMetric # Shot Dense×4 Dense×16 DeepSeekMoE Relative Expert Size N/A 1 1 0.25 # Experts N/A 4 + 0 16 + 0 1 + 63 # Activated Experts N/A 4 + 0 16 + 0 1 + 7 # Total Expert Params N/A 0.47B 1.89B 1.89B # Activated Expert Params N/A 0.47B 1.89B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.908 1.806 1.808 HellaSwag (Acc.) 0-shot 47.6 55.1 54.8 PIQA (Acc.) 0-shot 70.0 71.9 72.3 ARC-easy (Acc.) 0-shot 43.9 51.9 49.4 ARC-challenge (Acc.) 0-shot 30.5 33.8 34.3 RACE-middle (Acc.) 5-shot 42.4 46.3 44.0 RACE-high (Acc.) 5-shot 30.7 33.0 31.7 HumanEval (Pass@1) 0-shot 1.8 4.3 4.9 MBPP (Pass@1) 3-shot 0.2 2.2 2.2 TriviaQA (EM) 5-shot 9.9 16.5 16.6 NaturalQuestions (EM) 5-shot 3.0 6.3 5.7\n\nTable 9 | Comparison between DeepSeekMoE and larger dense baselines.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 15.9B 19.8B 13.3B # Activated Expert Params N/A 2.37B 2.82B 2.05B # Training Tokens N/A 100B 100B 100B HellaSwag (Acc.) 0-shot 66.6 67.7 69.1 PIQA (Acc.) 0-shot 75.6 76.0 75.7 ARC-easy (Acc.) 0-shot 56.8 56.8 58.8 ARC-challenge (Acc.) 0-shot 39.9 37.6 38.5 RACE-middle (Acc.) 5-shot 51.6 50.6 52.4 RACE-high (Acc.) 5-shot 37.4 36.3 38.5 HumanEval (Pass@1) 0-shot 6.1 6.1 9.8 MBPP (Pass@1) 3-shot 7.0 11.6 10.6 TriviaQA (EM) 5-shot 36.5 36.7 38.2 NaturalQuestions (EM) 5-shot 12.6 12.1 13.7\n\nTable 10 | Comparison between DeepSeekMoE and larger GShard models at a larger scale.\n\nC. Training Benchmark Curves Of Deepseekmoe 16B\n\nWe present the benchmark curves during training of DeepSeekMoE 16B and DeepSeek 7B (Dense) in Figure 7 for reference.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/DeepSeekMoE_2.md"
+          },
+          "summary": "DeepSeekMoE is an innovative Mixture-of-Experts (MoE) architecture designed to enhance expert specialization in language models. It employs fine-grained expert segmentation and shared expert isolation to improve performance and reduce computational costs. Starting with a model of 2B parameters, DeepSeekMoE demonstrates comparable performance to larger models like GShard 2.9B and LLaMA2 7B, using significantly fewer computations. The architecture scales up to 16B and 145B parameters, maintaining advantages over traditional MoE architectures. DeepSeekMoE 16B achieves similar results to dense models with only 40% of the computations, and its chat model version shows adaptability in bilingual scenarios. The architecture's public release aims to contribute to advancements in large-scale language models.",
+          "headlines": {
+            "1. Introduction": [],
+            "2. Preliminaries: Mixture-Of-Experts For Transformers": [],
+            "3. Deepseekmoe Architecture": [
+              "3.1. Fine-Grained Expert Segmentation",
+              "3.2. Shared Expert Isolation",
+              "3.3. Load Balance Consideration"
+            ],
+            "4. Validation Experiments": [
+              "4.1. Experimental Setup",
+              "4.1.1. Training Data And Tokenization",
+              "4.1.2. Infrastructures",
+              "4.1.3. Hyper-Parameters",
+              "4.1.4. Evaluation Benchmarks",
+              "4.2. Evaluations",
+              "4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models",
+              "4.4. Ablation Studies",
+              "4.5. Analysis On Expert Specialization"
+            ],
+            "5. Scaling Up To Deepseekmoe 16B": [
+              "5.1. Experimental Setup",
+              "5.1.1. Training Data And Tokenization",
+              "5.1.2. Hyper-Parameters",
+              "5.1.3. Evaluation Benchmarks",
+              "5.2. Evaluations",
+              "5.2.1. Internal Comparison With Deepseek 7B",
+              "5.2.2. Comparison With Open Source Models"
+            ],
+            "6. Alignment For Deepseekmoe 16B": [
+              "6.1. Experimental Setup",
+              "6.2. Evaluations"
+            ],
+            "7. Deepseekmoe 145B Ongoing": [
+              "7.1. Experimental Setup",
+              "7.2. Evaluations"
+            ],
+            "8. Related Work": [],
+            "9. Conclusion": [],
+            "Appendices": [
+              "A. Overview Of Hyper-Parameters",
+              "B. Comparing Deepseekmoe With Larger Models",
+              "C. Training Benchmark Curves Of Deepseekmoe 16B"
+            ]
+          },
+          "summary_embedding": [
+            -0.012465757317841053,
+            -0.022808987647294998,
+            -0.018792515620589256,
+            -0.02308654971420765,
+            0.01797616109251976,
+            0.007200260180979967,
+            0.006490030791610479,
+            0.042613785713911057,
+            -0.014049487188458443,
+            0.03187054023146629,
+            -0.021486490964889526,
+            -0.023592690005898476,
+            0.009045224636793137,
+            -0.014441338367760181,
+            0.005865518003702164,
+            -0.007363531738519669,
+            -0.008743173442780972,
+            -0.005163452122360468,
+            0.018841497600078583,
+            -0.03100520372390747,
+            0.011110605672001839,
+            -0.046075135469436646,
+            -0.020017050206661224,
+            -0.021845687180757523,
+            0.016425084322690964,
+            0.037389107048511505,
+            0.022547753527760506,
+            -0.0028062239289283752,
+            -0.06844329088926315,
+            0.034678805619478226,
+            0.035168617963790894,
+            0.02052319049835205,
+            0.0005438972148112953,
+            -0.00031302153365686536,
+            -0.024833550676703453,
+            -0.028556134551763535,
+            0.03347059711813927,
+            -0.004379750229418278,
+            -0.01453930139541626,
+            0.015976088121533394,
+            0.0414055772125721,
+            0.012106560170650482,
+            -0.02754385396838188,
+            0.0014245413476601243,
+            0.015151568688452244,
+            0.0055961208418011665,
+            -0.01678428053855896,
+            -0.0022960014175623655,
+            -0.002424577483907342,
+            -0.021225256845355034,
+            -0.01400050614029169,
+            0.018825169652700424,
+            0.001296985661610961,
+            -0.008865626528859138,
+            -0.0542386956512928,
+            0.007330877240747213,
+            0.03840138763189316,
+            0.05623060464859009,
+            0.044148534536361694,
+            -0.020180322229862213,
+            0.03369917720556259,
+            -0.030025575309991837,
+            -0.02501314878463745,
+            0.011028970591723919,
+            0.003983817528933287,
+            -0.018531281501054764,
+            0.01784554310142994,
+            -0.005636938381940126,
+            -0.0006556359585374594,
+            0.01751900091767311,
+            -0.03941367194056511,
+            0.020751770585775375,
+            -0.01887415163218975,
+            0.031266435980796814,
+            0.02173139713704586,
+            0.0073798587545752525,
+            0.01872720755636692,
+            0.01678428053855896,
+            0.0036858476232737303,
+            -0.04107903689146042,
+            0.009118696674704552,
+            0.016523046419024467,
+            0.014196431264281273,
+            0.016269976273179054,
+            0.06553706526756287,
+            -0.05391215533018112,
+            -0.004485876299440861,
+            -0.05169166624546051,
+            0.0012561678886413574,
+            -0.004971608519554138,
+            -0.03670336678624153,
+            0.01048201136291027,
+            0.028425518423318863,
+            0.021372200921177864,
+            -0.01063711941242218,
+            0.024392718449234962,
+            -0.022743679583072662,
+            0.0032001156359910965,
+            -0.04062187671661377,
+            0.0019653772469609976,
+            0.004518530797213316,
+            0.016531210392713547,
+            0.0070778070949018,
+            0.004273623693734407,
+            -0.016702644526958466,
+            0.006045116577297449,
+            -0.01905374974012375,
+            0.03291547670960426,
+            -0.008465612307190895,
+            0.08653374016284943,
+            0.008751336485147476,
+            0.004738946910947561,
+            0.004281787667423487,
+            0.02398454025387764,
+            0.012131051160395145,
+            -0.00810233410447836,
+            -0.04718538001179695,
+            -0.015690363943576813,
+            0.011061624623835087,
+            -0.0383034273982048,
+            0.018335357308387756,
+            -0.008759500458836555,
+            0.01282495353370905,
+            -0.04107903689146042,
+            -0.06344719231128693,
+            0.01498013362288475,
+            0.001784758409485221,
+            -0.004065453074872494,
+            -0.0002903166168835014,
+            -0.018923133611679077,
+            0.003930754493921995,
+            -0.004200151655822992,
+            -0.010645282454788685,
+            -0.021502818912267685,
+            0.03159298002719879,
+            0.03657275065779686,
+            0.06527583301067352,
+            0.036801330745220184,
+            -0.0030164355412125587,
+            0.005775718949735165,
+            0.019690508022904396,
+            -0.020784424617886543,
+            0.003481758525595069,
+            -0.01214737817645073,
+            -0.021992631256580353,
+            -0.03820546343922615,
+            0.010971825569868088,
+            0.07379858940839767,
+            -0.029029620811343193,
+            0.013437220826745033,
+            -0.017616963014006615,
+            0.04205866530537605,
+            -0.004567512311041355,
+            0.026727497577667236,
+            -0.016669990494847298,
+            0.004059330560266972,
+            -0.025388672947883606,
+            0.01620466820895672,
+            -0.024964168667793274,
+            0.012571883387863636,
+            -0.000885746325366199,
+            -0.04222193360328674,
+            0.00011116473615402356,
+            0.026890767738223076,
+            -0.010971825569868088,
+            0.02261306345462799,
+            0.0004949158756062388,
+            -0.006302268709987402,
+            -0.05097327008843422,
+            0.038727931678295135,
+            -0.003912386484444141,
+            0.018645571544766426,
+            -0.0018959869630634785,
+            0.0163107942789793,
+            -0.0016408757073804736,
+            0.013216804713010788,
+            0.009543201886117458,
+            0.007114543113857508,
+            -0.033503253012895584,
+            0.02697240374982357,
+            -0.02504580281674862,
+            0.028049994260072708,
+            0.03435226157307625,
+            -0.0030184765346348286,
+            0.03255628049373627,
+            0.019249675795435905,
+            0.03840138763189316,
+            0.0327032245695591,
+            0.030335791409015656,
+            -0.00824927818030119,
+            0.013878053054213524,
+            0.008200296200811863,
+            0.03608293831348419,
+            0.013429056853055954,
+            0.019723162055015564,
+            -0.009730963967740536,
+            -0.014041324146091938,
+            0.048785436898469925,
+            0.007318632211536169,
+            -0.0179598331451416,
+            -0.005441013257950544,
+            -0.03510330989956856,
+            -0.002832755446434021,
+            -0.014425011351704597,
+            0.0323440283536911,
+            0.02158445306122303,
+            -0.01647406443953514,
+            0.035135965794324875,
+            0.024980494752526283,
+            -0.03751972317695618,
+            -0.005232842173427343,
+            -0.02026195637881756,
+            -0.023298801854252815,
+            -0.04215662553906441,
+            0.011624909937381744,
+            0.012833117507398129,
+            -0.013976015150547028,
+            -0.007469657808542252,
+            0.020457882434129715,
+            -0.023625344038009644,
+            -0.0018265966791659594,
+            -0.01356783788651228,
+            0.006547175347805023,
+            -0.020898714661598206,
+            -0.03667071461677551,
+            -0.01048201136291027,
+            0.025862159207463264,
+            0.012637191452085972,
+            0.04362606629729271,
+            -0.013518855907022953,
+            -0.005367541220039129,
+            0.033193036913871765,
+            0.012759645469486713,
+            0.010767736472189426,
+            0.03699725493788719,
+            -0.03027048148214817,
+            0.024621298536658287,
+            -0.020294612273573875,
+            -0.0032878739293664694,
+            0.01707816869020462,
+            -0.0030062312725931406,
+            0.01962519995868206,
+            -0.04535674303770065,
+            -0.024196794256567955,
+            0.04395261034369469,
+            -0.0007107399869710207,
+            0.016302630305290222,
+            -0.005959399044513702,
+            -0.012106560170650482,
+            0.0035899258218705654,
+            0.0027715289033949375,
+            -0.01438419334590435,
+            -0.008620720356702805,
+            0.007473739795386791,
+            -0.01962519995868206,
+            0.03660540655255318,
+            0.03794422745704651,
+            -0.004132802598178387,
+            -0.014131123200058937,
+            -0.03324201703071594,
+            0.010016689077019691,
+            -0.017878197133541107,
+            -0.023951886221766472,
+            0.04078514873981476,
+            -0.032115448266267776,
+            -0.00029312283731997013,
+            0.039478980004787445,
+            0.018808843567967415,
+            -0.014898497611284256,
+            0.00868602842092514,
+            -0.01709449477493763,
+            -0.026302991434931755,
+            -0.017159804701805115,
+            0.03356856107711792,
+            0.024800896644592285,
+            -0.010743245482444763,
+            0.010245268233120441,
+            0.005330805201083422,
+            0.02893165871500969,
+            0.024376392364501953,
+            0.00648594880476594,
+            -0.013812744058668613,
+            0.015029114671051502,
+            0.027641816064715385,
+            -0.0028021421749144793,
+            -7.66609373386018e-05,
+            -0.004010349046438932,
+            -0.007796200457960367,
+            -0.006453294772654772,
+            -0.013069860637187958,
+            -0.005089980084449053,
+            0.0009184005321003497,
+            -0.032295044511556625,
+            0.015967924147844315,
+            -0.005890008993446827,
+            0.010359558276832104,
+            -0.011535110883414745,
+            0.01288209855556488,
+            0.01722511276602745,
+            -0.013494365848600864,
+            -0.0716434046626091,
+            0.014612773433327675,
+            0.02458864450454712,
+            -0.006706364918500185,
+            -0.017943505197763443,
+            0.008481939323246479,
+            0.00016148542636074126,
+            0.038597315549850464,
+            -0.014180104248225689,
+            -0.031103165820240974,
+            0.02680913358926773,
+            -0.017437364906072617,
+            -0.015241367742419243,
+            -0.010661610402166843,
+            -0.00580837344750762,
+            0.04649963974952698,
+            0.01889047957956791,
+            0.011420820839703083,
+            -0.04042595252394676,
+            -0.007449248805642128,
+            0.015078096650540829,
+            0.014506647363305092,
+            -0.04705476388335228,
+            0.0052083516493439674,
+            0.015486273914575577,
+            0.019723162055015564,
+            0.024800896644592285,
+            0.04274440184235573,
+            -0.0021531390957534313,
+            0.01661284640431404,
+            -0.015053605660796165,
+            -0.05136512219905853,
+            -0.03340528905391693,
+            0.029943939298391342,
+            -0.017731253057718277,
+            -0.02906227484345436,
+            -0.01693122461438179,
+            -0.006151242647320032,
+            0.0017939424142241478,
+            0.007126788143068552,
+            -0.01916803978383541,
+            -0.03464614972472191,
+            -0.03970756009221077,
+            0.02277633361518383,
+            0.007983962073922157,
+            0.004269542172551155,
+            -0.042777057737112045,
+            0.0013459670590236783,
+            0.009184005670249462,
+            0.012073906138539314,
+            0.0009954441338777542,
+            -0.019984396174550056,
+            0.015494437888264656,
+            0.0032001156359910965,
+            -0.031070511788129807,
+            -0.035593122243881226,
+            -0.03595232218503952,
+            0.0023919232189655304,
+            0.003781769424676895,
+            0.06279410421848297,
+            0.012988224625587463,
+            -0.03840138763189316,
+            -0.04522612690925598,
+            0.029176564887166023,
+            -0.02052319049835205,
+            -0.016678154468536377,
+            0.0059430720284581184,
+            0.012898425571620464,
+            -0.009877908043563366,
+            0.013192313723266125,
+            0.023560035973787308,
+            0.041601505130529404,
+            -0.005987971555441618,
+            0.012914752587676048,
+            0.023037567734718323,
+            0.012955570593476295,
+            0.0362788625061512,
+            -0.00936360377818346,
+            0.0035409443080425262,
+            0.039936136454343796,
+            -0.012269831262528896,
+            0.03175625205039978,
+            -0.02189466916024685,
+            0.0020398697815835476,
+            0.02249877341091633,
+            0.027184655889868736,
+            -0.0006479826406575739,
+            0.02594379521906376,
+            -0.031543996185064316,
+            0.018368011340498924,
+            -0.014498483389616013,
+            -0.04888340085744858,
+            -0.01557607389986515,
+            0.013102514669299126,
+            0.030384771525859833,
+            0.02756018005311489,
+            -0.0041348435916006565,
+            -0.019723162055015564,
+            -0.01155143789947033,
+            -0.010596301406621933,
+            0.0018419033149257302,
+            0.03285016864538193,
+            0.010506502352654934,
+            0.010890189558267593,
+            -0.011935125105082989,
+            -0.004104230087250471,
+            0.03025415539741516,
+            -0.021192602813243866,
+            0.01650671847164631,
+            0.0041511706076562405,
+            -0.002029665280133486,
+            -0.04320156201720238,
+            -0.035266581922769547,
+            0.007845181971788406,
+            0.011747363954782486,
+            0.014874007552862167,
+            0.0368339866399765,
+            0.025094784796237946,
+            -0.01662100851535797,
+            0.030629679560661316,
+            -0.01905374974012375,
+            0.027984686195850372,
+            0.039054471999406815,
+            0.011624909937381744,
+            -0.018351683393120766,
+            0.012237177230417728,
+            0.002424577483907342,
+            -0.025127438828349113,
+            0.01680060662329197,
+            -0.032735876739025116,
+            0.01738838478922844,
+            -0.02354370802640915,
+            0.025143766775727272,
+            -0.02579685114324093,
+            -0.009290131740272045,
+            -0.004428731743246317,
+            -0.006130834110081196,
+            0.04888340085744858,
+            -0.026727497577667236,
+            -0.0142454132437706,
+            -0.0024694770108908415,
+            -0.0024531499948352575,
+            -0.008718682453036308,
+            0.010000362060964108,
+            -0.045781247317790985,
+            -0.01844964735209942,
+            -0.04016471654176712,
+            0.007355368230491877,
+            0.005412440747022629,
+            -0.030907239764928818,
+            -0.05694899708032608,
+            0.025421326979994774,
+            -0.06331657618284225,
+            -0.019886434078216553,
+            0.051626358181238174,
+            -0.018400665372610092,
+            -0.008212542161345482,
+            0.009641164913773537,
+            0.006375740747898817,
+            -0.029290854930877686,
+            0.0041348435916006565,
+            0.02504580281674862,
+            0.04418119043111801,
+            -0.0045797573402523994,
+            -0.02865409664809704,
+            0.0028674507047981024,
+            -0.027788760140538216,
+            -0.013461710885167122,
+            0.005261414684355259,
+            0.016449574381113052,
+            0.024539662525057793,
+            0.007628847379237413,
+            0.03585435822606087,
+            0.007294141221791506,
+            0.006318595726042986,
+            0.044279150664806366,
+            -0.025666235014796257,
+            -0.027380581945180893,
+            0.02519274689257145,
+            0.04045860469341278,
+            0.023151857778429985,
+            -0.018253721296787262,
+            -0.0142454132437706,
+            0.0053063142113387585,
+            0.0009296254720538855,
+            -0.022139575332403183,
+            0.002808264922350645,
+            -0.013902543112635612,
+            0.005143043119460344,
+            0.03892385587096214,
+            -0.001205145614221692,
+            0.014196431264281273,
+            0.007302304729819298,
+            0.006845145486295223,
+            0.004334850702434778,
+            -0.011731036007404327,
+            -0.023592690005898476,
+            0.00032220553839579225,
+            0.034972693771123886,
+            0.015404638834297657,
+            -0.030515389516949654,
+            -0.00995954405516386,
+            0.034711457788944244,
+            -0.00795130804181099,
+            -0.009118696674704552,
+            -0.009771781973540783,
+            0.003385836724191904,
+            0.02638462744653225,
+            -0.01606588624417782,
+            0.006649219896644354,
+            0.03716052696108818,
+            -0.019870106130838394,
+            0.005543057806789875,
+            -0.025405000895261765,
+            0.0018061877926811576,
+            0.02442537248134613,
+            -0.006216551177203655,
+            0.028098974376916885,
+            5.6315813708351925e-05,
+            0.03025415539741516,
+            -0.004677720367908478,
+            0.008432958275079727,
+            -0.020327266305685043,
+            0.03987082839012146,
+            -0.017633290961384773,
+            0.03402572125196457,
+            0.02067013457417488,
+            -0.01258004643023014,
+            0.04783846437931061,
+            -0.036801330745220184,
+            -0.020784424617886543,
+            0.00584919098764658,
+            0.012743317522108555,
+            0.0028919412288814783,
+            -0.0191517136991024,
+            -0.018400665372610092,
+            -0.010996315628290176,
+            0.02380494214594364,
+            -0.022547753527760506,
+            -0.016865916550159454,
+            -0.007441085297614336,
+            -0.01890680566430092,
+            -0.009673818945884705,
+            -0.007988044060766697,
+            -0.01709449477493763,
+            -0.016547536477446556,
+            0.044573038816452026,
+            -0.06334923207759857,
+            0.016996532678604126,
+            0.017159804701805115,
+            0.011477965861558914,
+            -0.034678805619478226,
+            -0.019510909914970398,
+            -0.017633290961384773,
+            -0.002347023691982031,
+            -0.030743969604372978,
+            0.001702102366834879,
+            0.01725776679813862,
+            0.01138000376522541,
+            -0.05022222548723221,
+            0.012335140258073807,
+            0.02772345207631588,
+            -0.012065742164850235,
+            0.0014949520118534565,
+            -0.023135529831051826,
+            -0.017176130786538124,
+            0.0031776658724993467,
+            0.022123249247670174,
+            0.005975726060569286,
+            -0.008188051171600819,
+            -0.016865916550159454,
+            -0.005241005681455135,
+            -0.006045116577297449,
+            -0.002359268954023719,
+            0.005094061605632305,
+            0.011780017986893654,
+            0.019690508022904396,
+            -0.0024633544962853193,
+            -0.004436895251274109,
+            -0.006396149750798941,
+            0.027364253997802734,
+            -0.001450052484869957,
+            -0.008416631259024143,
+            0.050907962024211884,
+            0.03556047007441521,
+            0.006094098091125488,
+            -0.01108611561357975,
+            -0.03990348428487778,
+            0.010661610402166843,
+            -0.022711025550961494,
+            0.0401320643723011,
+            0.01905374974012375,
+            -0.008298259228467941,
+            -0.028866350650787354,
+            0.010294250212609768,
+            -0.02290695160627365,
+            -0.009975871071219444,
+            0.009469729848206043,
+            0.00726556871086359,
+            -0.03263791650533676,
+            -0.013543346896767616,
+            0.03510330989956856,
+            0.011437147855758667,
+            -0.006841063499450684,
+            -0.036050282418727875,
+            0.008588065393269062,
+            0.01646590232849121,
+            0.011869817040860653,
+            -0.019233347848057747,
+            0.004449140280485153,
+            -0.012041252106428146,
+            0.017470018938183784,
+            0.008849299512803555,
+            -0.017796561121940613,
+            0.018237393349409103,
+            0.004710374400019646,
+            -0.008490103296935558,
+            -0.003912386484444141,
+            -0.0163107942789793,
+            -0.02006603218615055,
+            0.03840138763189316,
+            0.024490682408213615,
+            0.026335645467042923,
+            0.005457340274006128,
+            0.02008235827088356,
+            0.03227871656417847,
+            0.011282040737569332,
+            -0.03134807199239731,
+            0.027935704216361046,
+            0.016996532678604126,
+            -0.04604247957468033,
+            0.021551799029111862,
+            -0.02218855731189251,
+            -0.028670424595475197,
+            0.009649328887462616,
+            -0.0017367975087836385,
+            0.011192241683602333,
+            0.0034837995190173388,
+            -0.003642988856881857,
+            -0.03307874873280525,
+            -0.006236960180103779,
+            0.015771998092532158,
+            0.015502601861953735,
+            -0.0048695639707148075,
+            -0.003296037670224905,
+            -0.011886144056916237,
+            0.00487364549189806,
+            0.009796272963285446,
+            0.039054471999406815,
+            0.001381682581268251,
+            -0.004751192405819893,
+            -0.0018765984568744898,
+            0.031854212284088135,
+            0.0271193478256464,
+            -0.017274094745516777,
+            -0.026286665350198746,
+            0.00580837344750762,
+            -0.014253576286137104,
+            0.026156047359108925,
+            -0.00487364549189806,
+            0.011877980083227158,
+            0.008718682453036308,
+            -0.0007908449042588472,
+            -0.001373519073240459,
+            0.025584599003195763,
+            -0.02710302174091339,
+            -0.021764053031802177,
+            0.031233781948685646,
+            -0.025731543079018593,
+            0.028735732659697533,
+            0.013061696663498878,
+            -0.01557607389986515,
+            -0.015225040726363659,
+            -0.010302413254976273,
+            0.009943217039108276,
+            -0.012277995236217976,
+            0.023853924125432968,
+            0.003608293831348419,
+            0.006392067763954401,
+            0.0003584313380997628,
+            0.02770712412893772,
+            0.024833550676703453,
+            0.007820690982043743,
+            -0.018074123188853264,
+            -0.0016816934803500772,
+            -0.02320083975791931,
+            -0.01680060662329197,
+            0.023772288113832474,
+            -0.018090449273586273,
+            -0.03268689662218094,
+            -0.028556134551763535,
+            0.012114724144339561,
+            0.007392104249447584,
+            0.010816717520356178,
+            0.03461349755525589,
+            -0.030384771525859833,
+            -0.0344502255320549,
+            0.01766594499349594,
+            0.026270337402820587,
+            -0.005551221314817667,
+            -0.021535472944378853,
+            0.03085825778543949,
+            -0.005828781984746456,
+            0.017600636929273605,
+            0.0404912605881691,
+            -0.0016225076979026198,
+            0.010228941217064857,
+            -0.042026009410619736,
+            -0.014057651162147522,
+            -0.0066614653915166855,
+            0.006261450704187155,
+            0.034417569637298584,
+            0.01048201136291027,
+            0.01535565685480833,
+            0.03448288142681122,
+            -0.021192602813243866,
+            -0.0002903166168835014,
+            -0.0022449791431427,
+            -0.02818061038851738,
+            0.03347059711813927,
+            0.025715215131640434,
+            0.011918798089027405,
+            0.019706835970282555,
+            -0.03794422745704651,
+            -0.013037205673754215,
+            0.027070365846157074,
+            0.015053605660796165,
+            0.01273515447974205,
+            0.016694480553269386,
+            0.011249386705458164,
+            0.00868602842092514,
+            -0.021633435040712357,
+            -0.05169166624546051,
+            0.01214737817645073,
+            0.01993541419506073,
+            -0.016449574381113052,
+            0.01057181041687727,
+            0.024980494752526283,
+            0.007620683871209621,
+            -0.007045152597129345,
+            0.012808626517653465,
+            -0.02682545967400074,
+            0.002259265398606658,
+            0.0019010890973731875,
+            -0.014482156373560429,
+            0.038270771503448486,
+            -0.016367938369512558,
+            0.011363675817847252,
+            -0.018204739317297935,
+            0.023870250210165977,
+            0.00011531029304023832,
+            0.0007964573451317847,
+            0.014963806606829166,
+            -0.011347348801791668,
+            -0.04235255345702171,
+            0.0021613026037812233,
+            0.03342161700129509,
+            -0.015200549736618996,
+            0.026449935510754585,
+            -0.0020674217958003283,
+            0.032458316534757614,
+            0.042777057737112045,
+            0.017731253057718277,
+            0.0003497575526125729,
+            0.025682561099529266,
+            0.0036144163459539413,
+            0.0073880222626030445,
+            0.012400448322296143,
+            0.009322785772383213,
+            0.007245160173624754,
+            0.030809277668595314,
+            -0.010261595249176025,
+            0.02173139713704586,
+            -0.04760988429188728,
+            -0.014629100449383259,
+            0.012024925090372562,
+            -0.0013316808035597205,
+            -0.012449429370462894,
+            -0.033029764890670776,
+            0.012563719414174557,
+            -0.03503800183534622,
+            0.022074267268180847,
+            -0.0004959363141097128,
+            0.04333217814564705,
+            -0.02187834307551384,
+            -0.005518566817045212,
+            0.016694480553269386,
+            0.0016510800924152136,
+            0.023282473906874657,
+            -0.0012153501156717539,
+            -0.016416920349001884,
+            -0.0067798369564116,
+            -0.009257477708160877,
+            0.004138925112783909,
+            0.04375668615102768,
+            -0.003722583642229438,
+            -0.0001682458823779598,
+            0.01409030519425869,
+            -0.011959616094827652,
+            -0.03250729665160179,
+            0.045193471014499664,
+            -0.0030837850645184517,
+            0.02484987862408161,
+            0.04979771748185158,
+            -0.06410028040409088,
+            -0.019543563947081566,
+            0.013061696663498878,
+            -0.010441194288432598,
+            0.011363675817847252,
+            0.005547139327973127,
+            0.016114868223667145,
+            0.017894525080919266,
+            -0.005485912784934044,
+            0.01619650423526764,
+            0.005106307100504637,
+            0.028098974376916885,
+            -0.02336410991847515,
+            -0.03296445682644844,
+            -0.02636830136179924,
+            -0.017747581005096436,
+            0.0014480114914476871,
+            -0.015796490013599396,
+            -3.563585551091819e-06,
+            -0.0022204883862286806,
+            -0.04388730227947235,
+            -0.025274382904171944,
+            0.040687184780836105,
+            0.006706364918500185,
+            0.02458864450454712,
+            0.0077268099412322044,
+            0.03418898954987526,
+            0.005885927006602287,
+            -0.009037061594426632,
+            0.030058229342103004,
+            -0.020049704238772392,
+            0.03833607956767082,
+            0.022106921300292015,
+            -0.002832755446434021,
+            0.005865518003702164,
+            0.031086837872862816,
+            -0.033797141164541245,
+            0.008086007088422775,
+            -0.034384917467832565,
+            0.0046695563942193985,
+            -0.008620720356702805,
+            -0.015094423666596413,
+            -0.009526874870061874,
+            -0.02504580281674862,
+            0.011216731742024422,
+            -0.021535472944378853,
+            0.01590261608362198,
+            0.010416703298687935,
+            -0.002916431985795498,
+            0.03448288142681122,
+            -0.014204595237970352,
+            -0.0027796924114227295,
+            0.0024531499948352575,
+            -0.027070365846157074,
+            -0.006343086715787649,
+            -0.003377673216164112,
+            -0.03689929470419884,
+            -0.004502203781157732,
+            0.010988152585923672,
+            -0.00024184548237826675,
+            0.016898570582270622,
+            0.02787039615213871,
+            0.0005092020728625357,
+            -0.022302847355604172,
+            0.002340900944545865,
+            -0.04199335351586342,
+            0.0011877981014549732,
+            0.01483318954706192,
+            -0.017600636929273605,
+            -0.008612556383013725,
+            0.033046092838048935,
+            0.023298801854252815,
+            -0.009396257810294628,
+            -0.035005345940589905,
+            0.017796561121940613,
+            -0.011853490024805069,
+            0.027511198073625565,
+            -0.0002630197268445045,
+            0.019298657774925232,
+            -0.0076941559091210365,
+            0.04493223875761032,
+            0.001369437319226563,
+            0.008645210415124893,
+            -0.021959977224469185,
+            0.007020662073045969,
+            -0.020278284326195717,
+            -0.002406209474429488,
+            -0.006171651650220156,
+            -0.012751481495797634,
+            -0.010669773444533348,
+            0.0032735876739025116,
+            0.007730891928076744,
+            -0.0176822729408741,
+            0.0055961208418011665,
+            -0.015959760174155235,
+            -0.02741323597729206,
+            0.007543129846453667,
+            -0.05965929850935936,
+            -0.0179598331451416,
+            -0.03373183310031891,
+            -0.009477893821895123,
+            0.02008235827088356,
+            0.03510330989956856,
+            -0.012759645469486713,
+            0.00497977202758193,
+            -0.014898497611284256,
+            0.004477712791413069,
+            0.03490738570690155,
+            0.044442422688007355,
+            -0.014759717509150505,
+            0.003381754970178008,
+            -0.010955498553812504,
+            0.033029764890670776,
+            -0.03347059711813927,
+            0.0030062312725931406,
+            0.031086837872862816,
+            -0.005057325586676598,
+            0.007959471084177494,
+            0.011624909937381744,
+            0.020474210381507874,
+            -0.019396619871258736,
+            0.00877582747489214,
+            0.002100075827911496,
+            0.019233347848057747,
+            0.02171507105231285,
+            0.020343592390418053,
+            -0.03539719805121422,
+            0.028409190475940704,
+            0.018384337425231934,
+            0.02444170042872429,
+            0.044442422688007355,
+            -0.03974021226167679,
+            -0.004147088620811701,
+            -0.0032225653994828463,
+            -0.02997659333050251,
+            0.02008235827088356,
+            -0.007530884351581335,
+            -0.01841699331998825,
+            -0.007069643586874008,
+            0.03327467292547226,
+            0.014571955427527428,
+            -0.05051611363887787,
+            0.01980479806661606,
+            0.01740471087396145,
+            0.02055584453046322,
+            0.024474354460835457,
+            -0.01123305968940258,
+            -0.0019072118448093534,
+            0.008996243588626385,
+            0.006804327480494976,
+            0.019266001880168915,
+            0.015257694758474827,
+            0.002012317767366767,
+            -0.0022082431241869926,
+            -0.011837163008749485,
+            0.005224678665399551,
+            -0.009077879600226879,
+            0.010808554477989674,
+            -0.02458864450454712,
+            -0.009698309935629368,
+            0.021388528868556023,
+            0.0031164393294602633,
+            -0.013665799982845783,
+            -0.03729114308953285,
+            -0.045944519340991974,
+            0.0033837957307696342,
+            0.004543021321296692,
+            -0.05365091934800148,
+            0.005322641227394342,
+            -0.026547899469733238,
+            -0.002108239568769932,
+            -0.005502239800989628,
+            0.0019214981002733111,
+            0.005636938381940126,
+            0.0167679525911808,
+            -0.013494365848600864,
+            0.02248244546353817,
+            -0.015502601861953735,
+            0.03165828809142113,
+            0.0055961208418011665,
+            0.007375776767730713,
+            -0.0027796924114227295,
+            0.05590406060218811,
+            -0.003712379140779376,
+            -0.03699725493788719,
+            0.023282473906874657,
+            -0.01311067771166563,
+            -0.0001747512142173946,
+            0.030090883374214172,
+            0.00884113647043705,
+            -0.015257694758474827,
+            0.013437220826745033,
+            -0.030613351613283157,
+            -0.03249097242951393,
+            0.010996315628290176,
+            -0.013755599036812782,
+            -0.0015776080545037985,
+            -0.04568328335881233,
+            -0.009339112788438797,
+            0.004269542172551155,
+            -0.018335357308387756,
+            0.010775899514555931,
+            0.01631895825266838,
+            0.013992343097925186,
+            -0.024245774373412132,
+            0.009053388610482216,
+            0.006865554489195347,
+            0.013453547842800617,
+            0.0007331897504627705,
+            0.014049487188458443,
+            -0.008677864447236061,
+            -0.011306531727313995,
+            -0.018319029361009598,
+            0.0032450153958052397,
+            -0.026792805641889572,
+            0.005155288614332676,
+            -0.003155216109007597,
+            -0.03771565109491348,
+            -0.06521052122116089,
+            -0.0344502255320549,
+            0.016294466331601143,
+            0.03396041318774223,
+            -0.009045224636793137,
+            0.0035144127905368805,
+            -0.00023648813657928258,
+            -0.01093917153775692,
+            0.024523336440324783,
+            -0.005930826533585787,
+            -0.0004829256213270128,
+            0.0035389035474509,
+            0.02050686441361904,
+            -0.009722800925374031,
+            0.005845109466463327,
+            -0.051952898502349854,
+            -0.01586179807782173,
+            0.01812310516834259,
+            0.006306350696831942,
+            0.014629100449383259,
+            -0.009690145961940289,
+            0.01693122461438179,
+            -0.030074557289481163,
+            -0.028147956356406212,
+            0.04130761697888374,
+            0.008277850225567818,
+            0.011486129835247993,
+            -0.003055212553590536,
+            -0.01680060662329197,
+            0.02504580281674862,
+            -0.0020633398089557886,
+            -0.006461458280682564,
+            -0.00817988719791174,
+            0.015820980072021484,
+            0.044279150664806366,
+            0.003255219664424658,
+            0.0009821783751249313,
+            0.017045514658093452,
+            -0.002663361607119441,
+            -0.011575928889214993,
+            0.007677828427404165,
+            0.013592327944934368,
+            -0.016384266316890717,
+            0.010375885292887688,
+            -0.025862159207463264,
+            0.0332583449780941,
+            0.03131541982293129,
+            0.011575928889214993,
+            -0.015045441687107086,
+            0.006130834110081196,
+            0.005330805201083422,
+            0.008086007088422775,
+            -0.001369437319226563,
+            -0.0017133272485807538,
+            0.020719116553664207,
+            -0.03276853263378143,
+            -0.01693122461438179,
+            -0.005347132217139006,
+            0.003932795487344265,
+            0.005277741700410843,
+            0.03396041318774223,
+            -0.0033266509417444468,
+            0.017927179113030434,
+            -0.011657563969492912,
+            0.010579974390566349,
+            0.021764053031802177,
+            0.037095218896865845,
+            -0.024996822699904442,
+            -0.029862305149435997,
+            0.012914752587676048,
+            -0.001296985661610961,
+            0.014065815135836601,
+            0.0010173837654292583,
+            0.014139287173748016,
+            0.0029409227427095175,
+            0.060475654900074005,
+            -0.017763907089829445,
+            0.004628738854080439,
+            0.0004035860183648765,
+            -0.01858026348054409,
+            -0.0013531101867556572,
+            0.004261378664523363,
+            -0.010979988612234592,
+            0.03608293831348419,
+            0.002344982698559761,
+            0.039185091853141785,
+            0.013918871060013771,
+            0.009290131740272045,
+            0.0015143405180424452,
+            -0.013592327944934368,
+            0.008355404250323772,
+            0.0015592400450259447,
+            -0.001132694073021412,
+            0.009461566805839539,
+            0.019298657774925232,
+            0.009714636951684952,
+            0.025274382904171944,
+            0.009788108989596367,
+            0.012857607565820217,
+            -0.029470453038811684,
+            0.03510330989956856,
+            -0.0006224715034477413,
+            0.012996388599276543,
+            -0.02040890045464039,
+            -0.003136848099529743,
+            -0.0008240094175562263,
+            0.015649545937776566,
+            -0.015616890974342823,
+            -0.001201063860207796,
+            -0.01297189760953188,
+            0.040099408477544785,
+            -0.018988441675901413,
+            0.013976015150547028,
+            0.009673818945884705,
+            -0.02865409664809704,
+            -0.007596192881464958,
+            0.01995174214243889,
+            0.014294394291937351,
+            -0.0053144777193665504,
+            -0.020490536466240883,
+            0.005832863971590996,
+            0.015380147844552994,
+            -0.028572462499141693,
+            1.3584674888988957e-05,
+            0.014751553535461426,
+            -0.0164903923869133,
+            -0.00025409081717953086,
+            -0.004702210891991854,
+            0.003975654020905495,
+            -0.009265640750527382,
+            0.007028825581073761,
+            0.016057724133133888,
+            -0.024507008492946625,
+            -0.020049704238772392,
+            0.005722655914723873,
+            0.006571666337549686,
+            0.01444950234144926,
+            0.007739055436104536,
+            0.004355259705334902,
+            -0.020033378154039383,
+            -0.01799248717725277,
+            0.0032674651592969894,
+            -0.01649855636060238,
+            -0.0048614004626870155,
+            0.0020531355403363705,
+            -0.019363965839147568,
+            0.01875986158847809,
+            -0.010743245482444763,
+            0.00017538899555802345,
+            -0.003226647386327386,
+            -0.020800752565264702,
+            0.006204306147992611,
+            -0.004865481983870268,
+            0.019870106130838394,
+            0.003555230563506484,
+            0.02785406820476055,
+            0.02310287579894066,
+            -0.004428731743246317,
+            0.017600636929273605,
+            -0.016155686229467392,
+            0.007983962073922157,
+            -0.031135819852352142,
+            -0.001782717532478273,
+            -0.010408539324998856,
+            0.0036103345919400454,
+            -0.015943434089422226,
+            -0.009126860648393631,
+            -0.0048777274787425995,
+            -0.0006678812787868083,
+            -0.004820582456886768,
+            0.0004275664687156677,
+            0.0027817331720143557,
+            0.02231917530298233,
+            -0.013208640739321709,
+            -0.02803366631269455,
+            -0.017437364906072617,
+            0.006045116577297449,
+            0.019396619871258736,
+            -0.026025431230664253,
+            -0.009184005670249462,
+            -0.005012426059693098,
+            -0.004326687194406986,
+            -0.021404854953289032,
+            -0.015167895704507828,
+            -0.005632856860756874,
+            -0.008045189082622528,
+            -0.010522829368710518,
+            0.0046532293781638145,
+            -0.0035715578123927116,
+            -0.011739199981093407,
+            0.021666089072823524,
+            0.018057795241475105,
+            -0.019837452098727226,
+            -0.019837452098727226,
+            0.013641309924423695,
+            -0.006159406621009111,
+            0.0018878233386203647,
+            0.02008235827088356,
+            0.00029286774224601686,
+            0.005824700463563204,
+            0.020882388576865196,
+            -0.0035225762985646725,
+            -0.01797616109251976,
+            0.003577680326998234,
+            -0.0018888438353314996,
+            -0.012122887186706066,
+            0.003212361130863428,
+            0.02773977816104889,
+            0.0008076822850853205,
+            0.023429417982697487,
+            -0.03056436963379383,
+            -0.011412657797336578,
+            0.006061443593353033,
+            0.02369065210223198,
+            0.00211844383738935,
+            -0.0021694661118090153,
+            0.020131340250372887,
+            0.03941367194056511,
+            0.005775718949735165,
+            0.0015755671774968505,
+            -0.04003410041332245,
+            0.01027792226523161,
+            0.009208496659994125,
+            0.024621298536658287,
+            0.008339077234268188,
+            0.0014653591206297278,
+            -0.012906589545309544,
+            -0.009755454957485199,
+            -0.0017939424142241478,
+            0.014702572487294674,
+            -0.023870250210165977,
+            0.0024755997583270073,
+            -0.03157665207982063,
+            0.005094061605632305,
+            0.0021959978621453047,
+            -0.009559528902173042,
+            0.0335359051823616,
+            0.00980443600565195,
+            -0.02065380848944187,
+            0.015151568688452244,
+            0.005767555441707373,
+            0.007330877240747213,
+            0.022449791431427002,
+            -0.008228869177401066,
+            0.030450081452727318,
+            -0.005049162078648806,
+            0.01213921420276165,
+            0.02742956392467022,
+            0.032588932663202286,
+            -0.013102514669299126,
+            -0.003959327004849911,
+            0.0006097159348428249,
+            -0.0002561317232903093,
+            0.0007995187188498676,
+            -0.008204378187656403,
+            0.0037389106582850218,
+            -0.014800535514950752,
+            0.012710663489997387,
+            -0.019543563947081566,
+            -0.0034286954905837774,
+            0.0038532004691660404,
+            0.01991908811032772,
+            0.0016265894519165158,
+            -0.033633869141340256,
+            0.02055584453046322,
+            -0.015420965850353241,
+            0.0135596739128232,
+            0.001290863030590117,
+            -0.011110605672001839,
+            -0.00029210239881649613,
+            0.020588500425219536,
+            0.010947334580123425,
+            -0.007845181971788406,
+            -0.0135596739128232,
+            -0.028376536443829536,
+            -0.021535472944378853,
+            0.00823703221976757,
+            0.009445239789783955,
+            -0.0028837777208536863,
+            0.0021306893322616816,
+            -0.011869817040860653,
+            0.0006520643946714699,
+            -0.0008122742874547839,
+            0.01740471087396145,
+            -0.0335359051823616,
+            0.021943651139736176,
+            -0.019527235999703407,
+            -0.019543563947081566,
+            0.017045514658093452,
+            0.008049271069467068,
+            0.009445239789783955,
+            0.0033552234526723623,
+            -0.01995174214243889,
+            -0.012326976284384727,
+            0.0338951013982296,
+            -0.01138000376522541,
+            0.006298186723142862,
+            0.004408322740346193,
+            0.012914752587676048,
+            -0.008914608508348465,
+            -0.011926962062716484,
+            0.01784554310142994,
+            -0.009861581027507782,
+            0.013951525092124939,
+            0.0010010566329583526,
+            -0.0254703089594841,
+            0.02739690989255905,
+            -0.019233347848057747,
+            0.03487472981214523,
+            -0.021617108955979347,
+            0.00711862463504076,
+            0.0066614653915166855,
+            0.008792154490947723,
+            -0.006690037902444601,
+            -0.013224967755377293,
+            0.0012786176521331072,
+            0.03807484731078148,
+            -0.02623768337070942,
+            0.024180466309189796,
+            0.022400809451937675,
+            -0.00040766780148260295,
+            0.02065380848944187,
+            -0.008367649279534817,
+            -0.026711169630289078,
+            0.0179598331451416,
+            -0.012596373446285725,
+            -0.026123393326997757,
+            0.013257622718811035,
+            -0.022678371518850327,
+            -0.020278284326195717,
+            -0.026302991434931755,
+            0.004947117529809475,
+            -0.0025694805663079023,
+            -0.005151206627488136,
+            0.009004407562315464,
+            0.02561725303530693,
+            0.020915042608976364,
+            0.019576217979192734,
+            0.02111096680164337,
+            0.01161674689501524,
+            0.022449791431427002,
+            -0.007559456862509251,
+            0.04401791840791702,
+            -0.004947117529809475,
+            0.004632820375263691,
+            -0.0015980169409886003,
+            -0.015657709911465645,
+            0.009771781973540783,
+            0.0119106350466609,
+            0.0029572497587651014,
+            -0.029176564887166023,
+            -0.009061552584171295,
+            -0.03660540655255318,
+            0.005053244065493345,
+            0.0020664012990891933,
+            0.0176822729408741,
+            0.032115448266267776,
+            -0.005865518003702164,
+            0.004592002835124731,
+            -0.005102225113660097,
+            -0.013927034102380276,
+            -0.009420748800039291,
+            -0.02739690989255905,
+            -0.025437654927372932,
+            -0.007653337903320789,
+            -0.013739272020757198,
+            -0.030368445441126823,
+            -0.011722872965037823,
+            0.01551892887800932,
+            -0.007645174395292997,
+            0.011918798089027405,
+            -0.016702644526958466,
+            0.01385356206446886,
+            -0.004110352601855993,
+            -0.004763437435030937,
+            -0.004530776292085648,
+            -0.017600636929273605,
+            -0.006420640274882317,
+            0.014367866329848766,
+            -0.009477893821895123,
+            -0.01237595733255148,
+            -0.02215590327978134,
+            -0.0015459742862731218,
+            -0.01574750803411007,
+            0.003489922033622861,
+            -0.014596446417272091,
+            -0.006771673448383808,
+            -0.003908304497599602,
+            -0.018514955416321754,
+            0.01766594499349594,
+            0.014816862531006336,
+            -0.019380291923880577,
+            -0.0090697156265378,
+            0.02638462744653225,
+            0.042777057737112045,
+            -0.03218075633049011,
+            -0.008200296200811863,
+            0.0020378287881612778,
+            0.018090449273586273,
+            0.010171796195209026,
+            0.016702644526958466,
+            -0.024653952568769455,
+            0.0022286521270871162,
+            -0.021649762988090515,
+            -0.001220452249981463,
+            -0.023739634081721306,
+            0.010775899514555931,
+            0.025715215131640434,
+            -0.010971825569868088,
+            0.025976449251174927,
+            0.01634344831109047,
+            -0.01812310516834259,
+            0.012016761116683483,
+            0.007183933164924383,
+            0.014645427465438843,
+            -0.019755816087126732,
+            0.0167679525911808,
+            -0.015796490013599396,
+            0.031233781948685646,
+            0.00892277155071497,
+            -0.04829562455415726,
+            -0.03503800183534622,
+            -0.013388238847255707,
+            0.0013388239312916994,
+            -0.010702427476644516,
+            0.015037278644740582,
+            0.015967924147844315,
+            0.013959688134491444,
+            -0.019380291923880577,
+            -0.015820980072021484,
+            0.011984107084572315,
+            0.010065670125186443,
+            0.027462217956781387,
+            -0.006649219896644354,
+            -0.004212397150695324,
+            -0.00960034690797329,
+            -0.02249877341091633,
+            -0.016098542138934135,
+            0.0209803506731987,
+            -0.018041469156742096,
+            0.015935270115733147,
+            -0.026139721274375916,
+            -0.012449429370462894,
+            0.017453692853450775,
+            -0.05211617052555084,
+            -0.01453113742172718,
+            0.0007612520130351186,
+            0.03012353740632534,
+            0.004751192405819893,
+            0.0035184945445507765,
+            0.01841699331998825,
+            -0.01993541419506073,
+            -0.0039776950143277645,
+            -0.02099667675793171,
+            0.006832899991422892,
+            -0.011853490024805069,
+            -0.013984179124236107,
+            0.014482156373560429,
+            -0.013845399022102356,
+            -0.01601690612733364,
+            0.0034368589986115694,
+            0.007183933164924383,
+            -0.021094640716910362,
+            0.011951452121138573,
+            0.003945040516555309,
+            0.012171869166195393,
+            0.012171869166195393,
+            0.01199227012693882,
+            0.021551799029111862,
+            0.02295593172311783,
+            -0.007334959227591753,
+            -0.004767519421875477,
+            -0.01004117913544178,
+            -0.0015398516552522779,
+            0.008898280560970306,
+            0.0033797139767557383,
+            -0.002989904023706913,
+            0.007837017998099327,
+            0.010033016093075275,
+            -0.01498013362288475,
+            0.033193036913871765,
+            -0.0035878848284482956,
+            -0.04134026914834976,
+            -0.003238892648369074,
+            -0.03800953924655914,
+            0.020033378154039383,
+            -0.023298801854252815,
+            -0.016996532678604126,
+            0.0032776694279164076,
+            -0.00629410520195961,
+            -0.019902760162949562,
+            0.013608654960989952,
+            0.011404493823647499,
+            0.00258580781519413,
+            0.02021297626197338,
+            0.012269831262528896,
+            -0.005134879611432552,
+            0.026172375306487083,
+            -0.014718899503350258,
+            -0.008751336485147476,
+            0.015567909926176071,
+            0.006853308994323015,
+            0.0022347746416926384,
+            0.0011031010653823614,
+            -0.018825169652700424,
+            0.005098143592476845,
+            0.0164903923869133,
+            0.007216587662696838,
+            -0.000566347036510706,
+            0.014049487188458443,
+            0.012514738366007805,
+            -0.015135240741074085,
+            -0.0002213090192526579,
+            -0.026564225554466248,
+            0.0032817514147609472,
+            0.004567512311041355,
+            0.007506393827497959,
+            -0.006771673448383808,
+            0.006281859707087278,
+            0.00658799335360527,
+            0.014759717509150505,
+            -0.010155469179153442,
+            -0.016980206593871117,
+            -0.002510294783860445,
+            0.004285869188606739,
+            0.016694480553269386,
+            0.03157665207982063,
+            -0.010310577228665352,
+            -0.005734901409596205,
+            -0.0386626236140728,
+            0.02070278860628605,
+            0.03332365304231644,
+            0.011175914667546749,
+            -0.028213264420628548,
+            0.012302485294640064,
+            0.0006515541463159025,
+            0.0019276207312941551,
+            -0.026302991434931755,
+            0.02787039615213871,
+            0.00974729098379612,
+            0.0066614653915166855,
+            0.017861871048808098,
+            0.0022306928876787424,
+            0.0029266364872455597,
+            -0.016302630305290222,
+            0.01018812321126461,
+            0.009086042642593384,
+            -0.017306748777627945,
+            0.0009745250572450459,
+            0.02324981987476349,
+            -0.01423724927008152,
+            -0.003208279376849532,
+            0.0018133309204131365,
+            0.02651524543762207,
+            -0.013820908032357693,
+            -0.014294394291937351,
+            0.0005714492290280759,
+            0.01751900091767311,
+            -0.0007979880319908261,
+            0.012098397128283978,
+            0.04898136481642723,
+            -0.012424939312040806,
+            0.019657853990793228,
+            0.005461421795189381,
+            -0.015290348790585995,
+            -0.032605260610580444,
+            -0.004922627005726099,
+            -0.01830270327627659,
+            0.011200404725968838,
+            0.014343376271426678,
+            -0.015763835981488228,
+            0.0007525782566517591,
+            -0.03144603595137596,
+            0.007743136957287788,
+            0.014735226519405842,
+            0.018776189535856247,
+            -0.002440904499962926,
+            0.006134915631264448,
+            0.027674470096826553,
+            0.02458864450454712,
+            -0.007392104249447584,
+            0.013804581016302109,
+            0.02320083975791931,
+            -0.025062130764126778,
+            -0.003069498809054494,
+            0.006820654962211847,
+            0.004104230087250471,
+            0.0053063142113387585,
+            -0.016963878646492958,
+            0.03915243595838547,
+            -0.005922663025557995,
+            -0.01506176870316267,
+            0.0030266400426626205,
+            -0.009028897620737553,
+            0.0063390047289431095,
+            -0.012563719414174557,
+            0.012637191452085972,
+            -0.008114579133689404,
+            0.00022883480414748192,
+            -0.0059430720284581184,
+            -0.011110605672001839,
+            -0.012653518468141556,
+            -0.0007949267164804041,
+            -0.016531210392713547,
+            0.007432921789586544,
+            0.00981259997934103,
+            0.005045080557465553,
+            0.01025343220680952,
+            -0.0036797248758375645,
+            -0.003136848099529743,
+            -0.016751626506447792,
+            -0.009984034113585949,
+            0.0073716952465474606,
+            0.01872720755636692,
+            -0.028588788583874702,
+            -0.018547609448432922,
+            0.009290131740272045,
+            0.0022898786701261997,
+            0.018776189535856247,
+            0.0158944521099329,
+            -0.0002595757250674069,
+            0.012318813242018223,
+            -0.01964152604341507,
+            -0.009184005670249462,
+            0.009690145961940289,
+            -0.003569516818970442,
+            0.01048201136291027,
+            -0.014759717509150505,
+            0.010033016093075275,
+            -0.011159587651491165,
+            -0.011053460650146008,
+            0.01385356206446886,
+            0.01738838478922844,
+            0.008873790502548218,
+            -0.001294944784604013,
+            3.383413059054874e-05,
+            -0.010506502352654934,
+            -0.024033522233366966,
+            0.004771600943058729,
+            0.027021385729312897,
+            -0.013273949734866619,
+            0.003883813973516226,
+            0.008702355436980724,
+            -0.017502672970294952,
+            -0.019821124151349068,
+            -0.012653518468141556,
+            -0.022106921300292015,
+            0.020572172477841377,
+            0.023413091897964478,
+            -0.02026195637881756,
+            0.00877582747489214,
+            -0.004620575346052647,
+            0.014049487188458443,
+            -0.01400050614029169,
+            0.010612628422677517,
+            0.0317889042198658,
+            -0.020000724121928215,
+            -0.0028654097113758326,
+            0.0036633978597819805,
+            -0.015347493812441826,
+            0.018629245460033417,
+            0.03268689662218094,
+            -0.00176945177372545,
+            0.014041324146091938,
+            0.0008138049161061645,
+            -0.009118696674704552,
+            0.01905374974012375,
+            0.011094278655946255,
+            0.005347132217139006,
+            -0.0042491331696510315,
+            -0.012212686240673065,
+            -0.03010721132159233,
+            0.016702644526958466,
+            0.0129637336358428,
+            -0.0037674831692129374,
+            -0.0011551438365131617,
+            -0.006343086715787649,
+            -0.015404638834297657,
+            0.0035123717971146107,
+            -0.013078023679554462,
+            -0.00803294312208891,
+            -0.008759500458836555,
+            0.004314441699534655,
+            -0.0034960447810590267,
+            -0.009053388610482216,
+            0.007236996199935675,
+            0.0009510547970421612,
+            -0.0010602424154058099,
+            -0.01902109570801258,
+            -0.025078458711504936,
+            -0.00995954405516386,
+            -0.016849588602781296,
+            0.0038287099450826645,
+            0.0005234883283264935,
+            -0.027184655889868736,
+            0.0053961132653057575,
+            -0.008139070123434067,
+            -0.008134988136589527,
+            0.007126788143068552,
+            0.006881881505250931,
+            -0.007437003776431084,
+            -0.010816717520356178,
+            -0.00786150898784399,
+            -0.006865554489195347,
+            0.020033378154039383,
+            0.0034348180051892996,
+            0.012082069180905819,
+            -0.010498338378965855,
+            -0.018025141209363937,
+            -0.016963878646492958,
+            0.016718972474336624,
+            -0.008971752598881721,
+            -0.02083340659737587,
+            -0.01444950234144926,
+            -0.008424794301390648,
+            -0.007665583398193121,
+            -0.01527402177453041,
+            -0.021159948781132698,
+            -0.0032103201374411583,
+            -0.011347348801791668,
+            -0.0067431009374558926,
+            -0.0028960229828953743,
+            0.0013571919407695532,
+            0.00951871182769537,
+            0.00546958576887846,
+            0.00438791373744607,
+            0.021666089072823524,
+            0.010122815147042274,
+            -0.005461421795189381,
+            0.016441410407423973,
+            -0.006183897145092487,
+            -0.004440976772457361,
+            -0.01498013362288475,
+            -0.009918726049363613,
+            0.018547609448432922,
+            -0.0055226488038897514,
+            0.008457448333501816,
+            -0.0009536059224046767,
+            -0.00576347392052412,
+            0.015674035996198654,
+            0.0129637336358428,
+            0.016718972474336624,
+            0.004771600943058729,
+            -0.00012168807734269649,
+            0.012416775338351727,
+            -0.0015745466807857156,
+            0.012555556371808052,
+            -0.0038429962005466223,
+            0.014506647363305092,
+            -0.009722800925374031,
+            -0.018988441675901413,
+            0.004157293122261763,
+            0.006265532691031694,
+            -0.012531065382063389,
+            0.009404421783983707,
+            0.007743136957287788,
+            -0.013747435994446278,
+            -0.02261306345462799,
+            -0.000106317616882734,
+            0.00014362763613462448,
+            0.008122743107378483,
+            0.008971752598881721,
+            -0.008073761127889156,
+            0.012890261597931385,
+            0.004616493359208107,
+            -0.007179851643741131,
+            -0.019657853990793228,
+            0.026009103283286095,
+            0.015608727931976318,
+            0.012269831262528896,
+            -0.052508022636175156,
+            -0.0030980713199824095,
+            0.010743245482444763,
+            -0.015951598063111305,
+            -0.026792805641889572,
+            -0.006404313258826733,
+            -0.008759500458836555,
+            -0.04290767386555672,
+            -0.009624837897717953,
+            -0.0005724696675315499,
+            -0.011388166807591915,
+            0.010400376282632351,
+            0.01573934406042099,
+            0.014612773433327675,
+            0.005489994306117296,
+            -0.007232914678752422,
+            -0.017633290961384773,
+            -0.02788672223687172,
+            0.010057506151497364,
+            -0.013380075804889202,
+            -0.006412476766854525,
+            -0.002263347152620554,
+            -0.007690073922276497,
+            0.0016112826997414231,
+            0.01691489666700363,
+            0.042483169585466385,
+            0.023037567734718323,
+            -0.012335140258073807,
+            -0.020131340250372887,
+            -0.012800462543964386,
+            -0.009192168712615967,
+            0.00633492274209857,
+            0.011592255905270576,
+            -0.004738946910947561,
+            0.0096656559035182,
+            -0.004294032696634531,
+            0.012302485294640064,
+            0.012767808511853218,
+            0.01680060662329197,
+            -2.3023791072773747e-05,
+            0.005579793825745583,
+            -0.0024694770108908415,
+            -0.01738838478922844,
+            -0.004379750229418278,
+            0.008261523209512234,
+            -0.0008872769540175796,
+            0.0019317024853080511,
+            0.014204595237970352,
+            0.019102731719613075,
+            -0.009094206616282463,
+            -0.014612773433327675,
+            -0.016098542138934135,
+            -0.005277741700410843,
+            -0.0012622905196622014,
+            0.029927613213658333,
+            0.02037624642252922,
+            -0.019266001880168915,
+            -0.018237393349409103,
+            -0.018351683393120766,
+            0.00922482367604971,
+            -0.003563394071534276,
+            -0.0064369672909379005,
+            0.01905374974012375,
+            0.023021239787340164,
+            0.017910851165652275,
+            -0.003642988856881857,
+            -0.014865843579173088,
+            0.003804219188168645,
+            0.02847449854016304,
+            0.008367649279534817,
+            0.01078406348824501,
+            0.008726846426725388,
+            -0.0011908593587577343,
+            -0.032441988587379456,
+            -0.0113146947696805,
+            0.00951871182769537,
+            -0.008481939323246479,
+            0.026564225554466248,
+            -0.016531210392713547,
+            0.009388094767928123,
+            -0.005077734589576721,
+            0.0019286411115899682,
+            -0.02365799807012081,
+            0.0014929111348465085,
+            0.010122815147042274,
+            -0.024947840720415115,
+            0.00336950970813632,
+            -0.023592690005898476,
+            -0.00013763252354692668,
+            0.008367649279534817,
+            0.0022837561555206776,
+            0.01650671847164631,
+            -0.0113146947696805,
+            0.0051838611252605915,
+            0.02099667675793171,
+            -0.002832755446434021,
+            -0.01512707769870758,
+            -0.0007842120248824358,
+            0.0005092020728625357,
+            -0.001286781276576221,
+            0.02563357912003994,
+            0.015812816098332405,
+            -0.0036756431218236685,
+            0.020882388576865196,
+            0.008677864447236061,
+            0.0019133344758301973,
+            -0.021764053031802177,
+            0.0023796779569238424,
+            -0.032899148762226105,
+            0.017437364906072617,
+            -0.011126932688057423,
+            0.012490247376263142,
+            0.00868602842092514,
+            -0.01341272983700037,
+            -0.0007388022495433688,
+            -0.0107187544927001,
+            -0.0015725058037787676,
+            -0.007934981025755405,
+            0.009779945947229862,
+            -0.004363423213362694,
+            0.008339077234268188,
+            -0.00024579971795901656,
+            -0.01476788055151701,
+            0.018694553524255753,
+            9.678283277025912e-06,
+            -0.00010925139940809458,
+            0.005441013257950544,
+            0.009935053065419197,
+            0.00786559097468853,
+            0.002009256277233362,
+            0.023870250210165977,
+            -0.031119491904973984,
+            -0.016082214191555977,
+            -0.010204450227320194,
+            0.005543057806789875,
+            -0.007922735065221786,
+            -0.018531281501054764,
+            -0.03257260471582413,
+            0.01318415068089962,
+            0.001293924287892878,
+            -0.0009806477464735508,
+            0.01258004643023014,
+            -0.00942891277372837,
+            -0.0013878053287044168,
+            -0.01606588624417782,
+            -0.014808698557317257,
+            0.006416558753699064,
+            0.0033633869607001543,
+            -0.01222901325672865,
+            -0.023739634081721306,
+            0.014759717509150505,
+            0.0007418635650537908,
+            -0.016980206593871117,
+            -0.017502672970294952,
+            0.00018457300029695034,
+            -0.004804255440831184,
+            0.02204161323606968,
+            -0.016678154468536377,
+            0.003373591462150216,
+            0.0007219648687168956,
+            0.009232986718416214,
+            -0.02889900468289852,
+            0.00716352416202426,
+            0.0006107363733462989,
+            0.0035164537839591503,
+            -0.006518602836877108,
+            -0.021013004705309868,
+            0.011371839791536331,
+            -0.0020674217958003283,
+            -0.014751553535461426,
+            -0.020751770585775375,
+            -0.0020796670578420162,
+            -0.007420676294714212,
+            -0.02724996581673622,
+            0.017910851165652275,
+            0.0009985055075958371,
+            -0.01738838478922844,
+            0.008077843114733696,
+            0.006596156861633062,
+            0.009355440735816956,
+            0.02455599047243595,
+            0.010147306136786938,
+            -0.018776189535856247,
+            -0.0008087027235887945,
+            0.006979844067245722,
+            0.009306458756327629,
+            0.0036450298503041267,
+            0.04222193360328674,
+            -0.008130906149744987,
+            -0.006579829845577478,
+            0.0066614653915166855,
+            -0.008449285291135311,
+            -0.013747435994446278,
+            0.026727497577667236,
+            0.005796127952635288,
+            0.015984252095222473,
+            0.016123032197356224,
+            -0.00945340283215046,
+            -0.005008344538509846,
+            -0.0018786393338814378,
+            0.02065380848944187,
+            0.0073798587545752525,
+            -0.005502239800989628,
+            -0.009714636951684952,
+            0.005881845485419035,
+            -0.00898808054625988,
+            0.0021653843577951193,
+            -0.009526874870061874,
+            -0.005824700463563204,
+            -0.008069679141044617,
+            0.016400592401623726,
+            -0.0003523086488712579,
+            0.027315273880958557,
+            0.020751770585775375,
+            -0.008563575334846973,
+            -0.0005683879135176539,
+            0.017208784818649292,
+            -0.005820618476718664,
+            0.015608727931976318,
+            -0.011192241683602333,
+            -0.007706400938332081,
+            0.006408394780009985,
+            0.024017194285988808,
+            0.012237177230417728,
+            -0.0056042843498289585,
+            0.0002885308349505067,
+            -0.0030348035506904125,
+            -0.010612628422677517,
+            0.013388238847255707,
+            0.015323002822697163,
+            -0.01947825588285923,
+            -0.004722619894891977,
+            -0.0016592437168583274,
+            -0.003300119424238801,
+            0.00945340283215046,
+            0.02305389568209648,
+            0.000171817431692034,
+            0.009184005670249462,
+            0.0012765767751261592,
+            0.03862996771931648,
+            -0.0053144777193665504,
+            0.03102152980864048,
+            -0.0010423846542835236,
+            -0.0059430720284581184,
+            -0.009094206616282463,
+            -0.020441556349396706,
+            -0.011486129835247993,
+            -0.00041481092921458185,
+            -0.005040998570621014,
+            -0.0004956811899319291,
+            -0.020327266305685043,
+            0.0026960158720612526,
+            0.014637264423072338,
+            0.007661501411348581,
+            0.017927179113030434,
+            -0.002889900468289852,
+            0.017437364906072617,
+            0.019690508022904396,
+            -0.01400050614029169,
+            -0.029862305149435997,
+            -0.0006515541463159025,
+            0.01498013362288475,
+            -0.01661284640431404,
+            0.019445601850748062,
+            0.004085862077772617,
+            0.0029531680047512054,
+            0.004779764451086521,
+            -0.006049198564141989,
+            0.018482301384210587,
+            0.02039257436990738,
+            0.01453930139541626,
+            -0.013167822733521461,
+            0.014057651162147522,
+            0.009771781973540783,
+            0.006600238848477602,
+            -0.01646590232849121,
+            -0.013535182923078537,
+            0.002555194543674588,
+            0.0009245232213288546,
+            0.024392718449234962,
+            -0.020588500425219536,
+            0.0016602640971541405,
+            0.010588138364255428,
+            0.01769859902560711,
+            -0.004302196204662323,
+            0.008906444534659386,
+            0.0008740111952647567,
+            0.007134951651096344,
+            -0.0035919665824621916,
+            -0.006457376293838024,
+            -0.006673710886389017,
+            -0.010392212308943272,
+            -0.0020959940738976,
+            -0.02070278860628605,
+            0.002198038622736931,
+            -0.002830714685842395,
+            0.018237393349409103,
+            -0.01018812321126461,
+            0.038140155375003815,
+            0.03399306535720825,
+            -0.0007025764207355678,
+            0.011102442629635334,
+            -0.01709449477493763,
+            -0.003945040516555309,
+            -0.021959977224469185,
+            0.01844964735209942,
+            0.016392430290579796,
+            0.012269831262528896,
+            0.005302232690155506,
+            0.017796561121940613,
+            -0.0006301248213276267,
+            0.014490320347249508,
+            0.0029654132667928934,
+            0.007971717044711113,
+            -0.002265388146042824,
+            0.003938918001949787,
+            -0.031103165820240974,
+            -0.017029186710715294,
+            -0.01138000376522541,
+            0.004796091932803392,
+            -0.006375740747898817,
+            -0.0035205355379730463,
+            0.0084737753495574,
+            -0.005902254022657871,
+            0.0004270562494639307,
+            -0.025992777198553085,
+            -0.0157148540019989,
+            -0.010318740271031857,
+            -0.004040962550789118,
+            -0.01169838197529316,
+            -0.0027694879099726677,
+            -0.0022082431241869926,
+            0.0029552089981734753,
+            0.020947696641087532,
+            -0.012808626517653465,
+            -0.016294466331601143,
+            0.004379750229418278,
+            -0.002263347152620554,
+            -0.014016833156347275,
+            -0.016898570582270622,
+            0.006665547378361225,
+            0.012098397128283978,
+            -0.0017470018938183784,
+            -0.013608654960989952,
+            0.011739199981093407,
+            -0.006628811359405518,
+            0.04663025587797165,
+            -0.013584164902567863,
+            -0.0078002819791436195,
+            0.003720542648807168,
+            0.004698128905147314,
+            -0.009396257810294628,
+            0.007330877240747213,
+            0.013020878657698631,
+            0.01830270327627659,
+            0.0095758568495512,
+            -0.008465612307190895,
+            -0.0024388637393712997,
+            0.007796200457960367,
+            -0.016147522255778313,
+            0.006404313258826733,
+            0.0008525818702764809,
+            -0.014400520361959934,
+            0.006110425107181072,
+            0.028980640694499016,
+            0.013151495717465878,
+            0.01288209855556488,
+            -0.022351829335093498,
+            -0.004094025585800409,
+            -0.009355440735816956,
+            -0.004943036008626223,
+            -0.010090161114931107,
+            0.027070365846157074,
+            -0.018155759200453758,
+            0.02651524543762207,
+            0.00898808054625988,
+            0.0017684312770143151,
+            -0.01176369097083807,
+            0.006343086715787649,
+            0.02832755446434021,
+            0.0028592869639396667,
+            -0.020000724121928215,
+            -0.013592327944934368,
+            0.006314514204859734,
+            0.01707816869020462,
+            -0.002647034591063857,
+            -0.005428767763078213,
+            0.019657853990793228,
+            0.01784554310142994,
+            -0.004334850702434778,
+            -0.00936360377818346,
+            0.0008337036124430597,
+            0.00928196869790554,
+            0.016963878646492958,
+            0.015020951628684998,
+            0.009086042642593384,
+            -0.002647034591063857,
+            0.008037025108933449,
+            -0.008800318464636803,
+            -0.016449574381113052,
+            0.007220669183880091,
+            -0.008432958275079727,
+            -0.016963878646492958,
+            0.01872720755636692,
+            0.015257694758474827,
+            0.0181394312530756,
+            0.0024592725094407797,
+            -0.016735298559069633,
+            0.022302847355604172,
+            0.005755309946835041,
+            0.002998067531734705,
+            0.02160078100860119,
+            -0.022221211344003677,
+            0.013012715615332127,
+            0.02037624642252922,
+            0.027609162032604218,
+            -0.01737205684185028,
+            -0.010906516574323177,
+            -0.004857318475842476,
+            -0.006853308994323015,
+            -0.00176945177372545,
+            -0.008253359235823154,
+            -0.012620864436030388,
+            0.0028735732194036245,
+            -0.030466407537460327,
+            0.018106777220964432,
+            -0.005102225113660097,
+            0.010669773444533348,
+            -0.019266001880168915,
+            0.011363675817847252,
+            -0.0035838030744343996,
+            0.011135096661746502,
+            0.0019612954929471016,
+            -0.030041903257369995,
+            -0.008571738377213478,
+            0.015388311818242073,
+            -0.0030368445441126823,
+            -0.013641309924423695,
+            0.02006603218615055,
+            0.00018942011229228228,
+            -0.022270193323493004,
+            -0.004314441699534655,
+            -0.006934944540262222,
+            0.002193956868723035,
+            -0.0029613315127789974,
+            -0.0052981507033109665,
+            -0.030188847333192825,
+            0.020033378154039383,
+            0.009559528902173042,
+            -0.03072764165699482,
+            -0.003255219664424658,
+            -0.014996460638940334,
+            -0.0025184585247188807,
+            0.008881953544914722,
+            -0.004000144544988871,
+            -0.015551582910120487,
+            0.01784554310142994,
+            0.022711025550961494,
+            -0.0023164101876318455,
+            -0.016057724133133888,
+            -0.0090697156265378,
+            -0.027494871988892555,
+            0.030825603753328323,
+            -0.0066696288995444775,
+            0.0022878379095345736,
+            -0.005322641227394342,
+            0.009388094767928123,
+            0.004228724166750908,
+            0.0157148540019989,
+            0.006081852596253157,
+            -0.002647034591063857,
+            0.026596879586577415,
+            -0.01964152604341507,
+            0.009567692875862122,
+            -1.588864870427642e-05,
+            -0.003308282932266593,
+            0.01890680566430092,
+            0.0007939062779769301,
+            0.021649762988090515,
+            -0.008571738377213478,
+            -0.0001475818717153743,
+            -0.0034388997592031956,
+            0.010898353531956673,
+            -0.0028450009413063526,
+            -0.003465431509539485,
+            -0.001531688030809164,
+            0.011118769645690918,
+            0.01013097818940878,
+            0.003804219188168645,
+            -0.01737205684185028,
+            0.003142970846965909,
+            -0.008547247387468815,
+            -0.007028825581073761,
+            0.01302904263138771,
+            -0.011192241683602333,
+            -0.0018990482203662395,
+            -0.03160930797457695,
+            0.016286302357912064,
+            0.0533243790268898,
+            0.008547247387468815,
+            -0.014433175325393677,
+            -0.0018806802108883858,
+            -0.019755816087126732,
+            0.01535565685480833,
+            0.011004479601979256,
+            -0.016816934570670128,
+            -0.011984107084572315,
+            -0.012310649268329144,
+            0.007649256382137537,
+            0.021306892856955528,
+            -0.006522684823721647,
+            -0.008579902350902557,
+            0.025372346863150597,
+            -0.011788181029260159,
+            -0.010882026515901089,
+            -0.01661284640431404,
+            -0.01635977439582348,
+            0.00241641397587955,
+            0.019902760162949562,
+            -0.0169475506991148,
+            0.021176276728510857,
+            0.00936360377818346,
+            0.0010133018950000405,
+            0.003624620847404003,
+            0.008588065393269062,
+            0.0024878450203686953,
+            0.017878197133541107,
+            0.003883813973516226,
+            -0.01439235731959343,
+            0.0021408938337117434,
+            -0.0030286810360848904,
+            0.004624656867235899,
+            3.4320437407586724e-06,
+            -0.002749078907072544,
+            -0.0065635028295218945,
+            0.003710338147357106,
+            -0.021306892856955528,
+            0.021502818912267685,
+            -0.022678371518850327,
+            -0.004245051648467779,
+            0.005555302836000919,
+            -0.005118552595376968,
+            0.013469874858856201,
+            0.018792515620589256,
+            0.01423724927008152,
+            0.008702355436980724,
+            0.01462093647569418,
+            -0.011331021785736084,
+            0.02323349379003048,
+            -0.013282112777233124,
+            0.0005816536722704768,
+            0.012939243577420712,
+            0.01755165494978428,
+            0.018792515620589256,
+            -0.006245123688131571,
+            -0.010384049266576767,
+            -0.0002946535241790116,
+            0.0129637336358428,
+            -0.022286519408226013,
+            -0.012269831262528896,
+            -0.01386172603815794,
+            0.011486129835247993,
+            -0.004743028432130814,
+            0.025453981012105942,
+            -0.00633492274209857,
+            -0.014653591439127922,
+            -0.015918942168354988,
+            0.014196431264281273,
+            -0.048752784729003906,
+            0.0030348035506904125,
+            -0.0024919267743825912,
+            0.03043375350534916,
+            -0.027021385729312897,
+            0.01738838478922844,
+            -0.0028490826953202486,
+            -0.009355440735816956,
+            0.015429128892719746,
+            -0.015992414206266403,
+            -0.0046613928861916065,
+            -0.0016806729836389422,
+            -0.007539048325270414,
+            -0.00112248957157135,
+            0.017616963014006615,
+            -0.015845470130443573,
+            0.01282495353370905,
+            -0.00989423505961895,
+            0.0027021386194974184,
+            -0.004526694305241108,
+            -0.007751300930976868,
+            0.0196088720113039,
+            0.034548189491033554,
+            -0.007053316105157137,
+            0.0014990337658673525,
+            -0.022547753527760506,
+            0.013192313723266125,
+            -0.006730855442583561,
+            0.02351105399429798,
+            -0.005432849284261465,
+            -0.016898570582270622,
+            -0.013976015150547028,
+            -0.009739127941429615,
+            0.00839214026927948,
+            -0.022433463484048843,
+            -0.011110605672001839,
+            -0.00922482367604971,
+            -0.002573562553152442,
+            -0.018694553524255753,
+            0.004865481983870268,
+            -0.002426618244498968,
+            -0.012506574392318726,
+            -0.008816645480692387,
+            0.01048201136291027,
+            -0.009347276762127876,
+            0.003077662317082286,
+            -0.0011929002357646823,
+            -0.011453475803136826,
+            -0.02369065210223198,
+            0.0158944521099329,
+            0.0001974561164388433,
+            -0.010359558276832104,
+            0.008408467285335064,
+            -0.0018276170594617724,
+            0.001211268245242536,
+            0.016963878646492958,
+            -0.0020664012990891933,
+            0.003467472270131111,
+            0.023478399962186813,
+            0.004057289566844702,
+            -0.010971825569868088,
+            -0.00025906547671183944,
+            0.013535182923078537,
+            -0.005118552595376968,
+            -0.010008525103330612,
+            1.6646010408294387e-05,
+            -0.02984597720205784,
+            -0.008351322263479233,
+            0.006094098091125488,
+            0.018629245460033417,
+            -0.010620792396366596,
+            -0.015853634104132652,
+            0.026319319382309914,
+            -0.006359413731843233,
+            0.019070077687501907,
+            0.002889900468289852,
+            0.0025694805663079023,
+            0.01724144071340561,
+            0.013461710885167122,
+            0.00786150898784399,
+            -0.013927034102380276,
+            -0.02429475635290146,
+            0.0016398552106693387,
+            -0.018090449273586273,
+            0.010579974390566349,
+            -0.011028970591723919,
+            0.0032756286673247814,
+            -0.02398454025387764,
+            -0.012237177230417728,
+            0.004934872500598431,
+            -0.020343592390418053,
+            -0.00040588201954960823,
+            -0.0056940834037959576,
+            0.022237539291381836,
+            -0.005310396198183298,
+            0.008669701404869556,
+            0.0018255761824548244,
+            -0.012237177230417728,
+            0.0009877908742055297,
+            0.009469729848206043,
+            -0.029192892834544182,
+            -0.0007127808639779687,
+            -0.003138889092952013,
+            0.0011418780777603388,
+            0.005351213738322258,
+            0.009755454957485199,
+            0.007322713732719421,
+            0.0027062203735113144,
+            0.007253323681652546,
+            0.010449357330799103,
+            -0.00022041612828616053,
+            0.0197884701192379,
+            0.018172085285186768,
+            -0.007351286243647337,
+            0.008710519410669804,
+            0.024817224591970444,
+            -0.013159659691154957,
+            -0.008718682453036308,
+            0.023266147822141647,
+            -0.006461458280682564,
+            -0.0011122850701212883,
+            -0.01483318954706192,
+            0.00815947912633419,
+            0.0017367975087836385,
+            0.028131630271673203,
+            0.019657853990793228,
+            0.005820618476718664,
+            0.000639819074422121,
+            0.015788326039910316,
+            -0.013020878657698631,
+            0.0035286990460008383,
+            0.015461783856153488,
+            -0.003940958995372057,
+            -0.019821124151349068,
+            0.013878053054213524,
+            0.009502384811639786,
+            -0.003228688146919012,
+            0.01551892887800932,
+            0.011412657797336578,
+            -0.030841931700706482,
+            -0.0011898389784619212,
+            0.010498338378965855,
+            -0.0021449755877256393,
+            -0.008751336485147476,
+            -0.00014273475972004235,
+            -0.012384121306240559,
+            0.021388528868556023,
+            -0.004408322740346193,
+            0.012277995236217976,
+            -0.00815947912633419,
+            -0.017127150669693947,
+            -0.011559601873159409,
+            0.007914572022855282,
+            -0.010547320358455181,
+            -0.0006714528426527977,
+            -0.004022594541311264,
+            -0.005779800936579704,
+            0.017747581005096436,
+            0.004706292413175106,
+            -0.0008964610169641674,
+            -0.0018398624379187822,
+            0.007983962073922157,
+            -0.006979844067245722,
+            -0.008726846426725388,
+            0.004816500470042229,
+            -0.012833117507398129,
+            0.003981776535511017,
+            0.005543057806789875,
+            0.01506176870316267,
+            -0.014857680536806583,
+            -0.021159948781132698,
+            0.007020662073045969,
+            0.02695607766509056,
+            -0.02516009286046028,
+            -0.005232842173427343,
+            0.006245123688131571,
+            0.002093953313305974,
+            0.006914535537362099,
+            -0.006934944540262222,
+            -0.007208424154669046,
+            -0.03657275065779686,
+            -0.0020745648071169853,
+            0.0009005427709780633,
+            0.008963589556515217,
+            -0.008049271069467068,
+            -0.0136494729667902,
+            -0.011135096661746502,
+            0.002410291228443384,
+            0.007134951651096344,
+            -0.02204161323606968,
+            0.0049593630246818066,
+            0.009608510881662369,
+            -0.019559891894459724,
+            0.025127438828349113,
+            -0.010890189558267593,
+            -6.964537897147238e-05,
+            -0.00995954405516386,
+            0.021633435040712357,
+            -0.013486201874911785,
+            -0.018368011340498924,
+            -0.0006898208521306515,
+            0.008355404250323772,
+            0.011926962062716484,
+            -0.0017663904000073671,
+            -0.011371839791536331,
+            0.0003752686607185751,
+            0.01646590232849121,
+            0.019135385751724243,
+            0.002744997153058648,
+            -0.0024837632663547993,
+            -0.01019628718495369,
+            -0.009592183865606785,
+            -0.0073961857706308365,
+            -0.0007449248805642128,
+            0.002091912319883704,
+            0.02035992033779621,
+            0.013469874858856201,
+            0.0024837632663547993,
+            -0.015135240741074085,
+            -0.005257333163172007,
+            0.008939098566770554,
+            0.009412584826350212,
+            -0.011690218932926655,
+            -0.00898808054625988,
+            0.002659279853105545,
+            0.02249877341091633,
+            0.017633290961384773,
+            -0.0014000505907461047,
+            0.0032878739293664694,
+            -0.0016592437168583274,
+            0.012506574392318726,
+            -0.007555375341325998,
+            -0.010269759222865105,
+            -0.006943108048290014,
+            -0.014645427465438843,
+            -0.010800390504300594,
+            -0.0023062059190124273,
+            -0.010179960168898106,
+            0.022629389539361,
+            0.007988044060766697,
+            0.009779945947229862,
+            0.008265605196356773,
+            0.010269759222865105,
+            0.002087830565869808,
+            0.025715215131640434,
+            -0.004424649756401777,
+            -0.011984107084572315,
+            0.007236996199935675,
+            -0.022368155419826508,
+            -0.021535472944378853,
+            0.008257441222667694,
+            0.006367577239871025,
+            0.03595232218503952,
+            -0.026890767738223076,
+            0.0052981507033109665,
+            -0.009779945947229862,
+            -0.010400376282632351,
+            0.007996207103133202,
+            -0.0015439334092661738,
+            -0.004245051648467779,
+            0.022368155419826508,
+            0.00726556871086359,
+            0.0035797213204205036,
+            0.0032817514147609472,
+            0.019837452098727226,
+            -0.005685919895768166,
+            0.01573934406042099,
+            0.0259274672716856,
+            0.024621298536658287,
+            0.009388094767928123,
+            -0.009469729848206043,
+            0.02785406820476055,
+            0.0002895512734539807,
+            -0.0034327772445976734,
+            0.007220669183880091,
+            -0.00305113079957664,
+            -0.0009556467994116247,
+            -0.010033016093075275,
+            0.021764053031802177,
+            -0.001455154619179666,
+            0.004359341226518154,
+            0.02081707864999771,
+            -7.774515688652173e-05,
+            0.006220633164048195,
+            0.01057181041687727,
+            0.002181711606681347,
+            0.003396041225641966,
+            0.010775899514555931,
+            -0.039315707981586456,
+            -0.0006571666453965008,
+            -0.025600925087928772,
+            -0.0018286375561729074,
+            0.0007357408758252859,
+            -0.01266984548419714,
+            0.013600491918623447,
+            -0.002015379024669528,
+            -0.010751409456133842,
+            -0.00029337796149775386,
+            0.021649762988090515,
+            0.005416522268205881,
+            0.004071575589478016,
+            -0.004469549283385277,
+            0.018792515620589256,
+            -0.006955353543162346,
+            -0.017861871048808098,
+            -0.012759645469486713,
+            -0.012367794290184975,
+            -0.01918436773121357,
+            -0.01214737817645073,
+            -0.0024551907554268837,
+            -0.002667443361133337,
+            0.004738946910947561,
+            0.002481722505763173,
+            -0.003228688146919012,
+            -0.02070278860628605,
+            0.015037278644740582,
+            -0.01430255826562643,
+            0.012873934581875801,
+            -0.011053460650146008,
+            0.005963481031358242,
+            -0.004122598096728325,
+            0.012726990506052971,
+            -0.021404854953289032,
+            0.02070278860628605,
+            -0.01536382082849741,
+            0.0038532004691660404,
+            0.019592545926570892,
+            0.006551257334649563,
+            -0.008118661120533943,
+            -0.013404566794633865,
+            -0.022074267268180847,
+            0.021633435040712357,
+            -0.011633073911070824,
+            -0.018237393349409103,
+            -0.004608329851180315,
+            0.009649328887462616,
+            -0.017470018938183784,
+            -0.007367613259702921,
+            0.01707816869020462,
+            -0.0018949664663523436,
+            0.012980061583220959,
+            0.023004913702607155,
+            -0.0052981507033109665,
+            0.0036572751123458147,
+            -0.009167678654193878,
+            0.0010286086471751332,
+            -0.02470293454825878,
+            0.013527019880712032,
+            0.008971752598881721,
+            0.006269614677876234,
+            -0.013935198076069355,
+            -0.013306603766977787,
+            0.0036531933583319187,
+            0.004498121794313192,
+            0.011437147855758667,
+            0.009420748800039291,
+            0.004220560658723116,
+            -0.013918871060013771,
+            -0.027788760140538216,
+            -0.027070365846157074,
+            0.03595232218503952,
+            -0.0005002731923013926,
+            -0.020947696641087532,
+            0.03399306535720825,
+            0.016114868223667145,
+            0.0022980421781539917,
+            -0.012449429370462894,
+            -0.004314441699534655,
+            0.024621298536658287,
+            0.016400592401623726,
+            0.010465684346854687,
+            -0.0004742518358398229,
+            0.005571629852056503,
+            0.0016469983384013176,
+            0.03012353740632534,
+            0.015323002822697163,
+            0.0009459525463171303,
+            -0.038597315549850464,
+            0.011943289078772068,
+            -0.007547211833298206,
+            0.0013408648082986474,
+            0.006702283397316933,
+            0.017780235037207603,
+            0.011935125105082989,
+            0.015567909926176071,
+            0.014865843579173088,
+            -0.003065417055040598,
+            0.0029511270113289356,
+            0.0006163488142192364,
+            -0.009624837897717953,
+            -0.001767410896718502,
+            -0.008881953544914722,
+            -0.005714492406696081,
+            -0.002920513739809394,
+            0.005592038854956627,
+            0.007657419890165329,
+            -0.003983817528933287,
+            0.028049994260072708,
+            -0.0006433906382881105,
+            -0.017600636929273605,
+            -0.009061552584171295,
+            -0.03611559048295021,
+            0.00434301421046257,
+            -0.030939893797039986,
+            -0.011257549747824669,
+            0.0033674687147140503,
+            -0.020457882434129715,
+            -0.024768242612481117,
+            0.0010306495241820812,
+            0.014335212297737598,
+            0.007392104249447584,
+            0.0033327736891806126,
+            -0.01004934310913086,
+            0.003816464450210333,
+            -0.001855169073678553,
+            -0.006461458280682564,
+            0.028833696618676186,
+            0.019559891894459724,
+            0.020457882434129715,
+            -0.01190247107297182,
+            0.019298657774925232,
+            0.005845109466463327,
+            0.02035992033779621,
+            0.010465684346854687,
+            -0.0048695639707148075,
+            0.0074859848245978355,
+            0.009567692875862122,
+            -0.006155324634164572,
+            -0.021519144997000694,
+            0.013192313723266125,
+            -0.0010837126756086946,
+            -0.0052083516493439674,
+            -0.016269976273179054,
+            -0.003534821793437004,
+            -0.015812816098332405,
+            0.010547320358455181,
+            -0.017159804701805115,
+            0.012498411349952221,
+            -0.000802069844212383,
+            0.00168475485406816,
+            -0.015469946898519993,
+            -0.013951525092124939,
+            -0.0012439225101843476,
+            -0.01244126632809639,
+            -0.008971752598881721,
+            0.006163488142192364,
+            -0.007539048325270414,
+            -0.008318668231368065,
+            -0.00535937724635005,
+            -0.011437147855758667,
+            -0.010882026515901089,
+            0.004367504734545946,
+            -0.0010653446661308408,
+            0.020931368693709373,
+            0.00011671340325847268,
+            -0.005824700463563204,
+            0.01190247107297182,
+            0.02787039615213871,
+            0.0016704685986042023,
+            0.004583839327096939,
+            -0.014261740259826183,
+            -0.011118769645690918,
+            -0.02279266156256199,
+            0.016572028398513794,
+            0.0049675265327095985,
+            -0.0344502255320549,
+            0.01438419334590435,
+            0.00824927818030119,
+            0.0015102586476132274,
+            -0.006698201410472393,
+            -0.005334886722266674,
+            -0.005608365871012211,
+            0.006306350696831942,
+            -0.0022837561555206776,
+            0.0005107327597215772,
+            0.0035919665824621916,
+            0.034678805619478226,
+            -0.0040674940682947636,
+            -0.007228832691907883,
+            0.00981259997934103,
+            0.0005130287609063089,
+            0.020196648314595222,
+            -0.005518566817045212,
+            0.0037572786677628756,
+            -0.019282329827547073,
+            -0.016563864424824715,
+            0.0018429238116368651,
+            -0.01722511276602745,
+            0.023282473906874657,
+            -0.005392031744122505,
+            -0.01347803883254528,
+            -0.006906372029334307,
+            -0.020115012302994728,
+            0.012637191452085972,
+            0.02310287579894066,
+            0.007016580086201429,
+            0.01033506728708744,
+            0.012131051160395145,
+            -0.02444170042872429,
+            -0.020457882434129715,
+            0.012171869166195393,
+            0.0013571919407695532,
+            -0.008449285291135311,
+            -0.005404277238994837,
+            -0.001783737912774086,
+            0.007783954963088036,
+            -0.0032633834052830935,
+            0.007881917990744114,
+            0.06491663306951523,
+            -0.01093100756406784,
+            -0.002987863030284643,
+            -0.02966637909412384,
+            -0.004396077245473862,
+            -0.006628811359405518,
+            0.0056287748739123344,
+            0.009151351638138294,
+            -0.023135529831051826,
+            0.009232986718416214,
+            -0.030025575309991837,
+            -0.010628955438733101,
+            0.00824111420661211,
+            0.0028490826953202486,
+            -0.01220452319830656,
+            -0.0009592183632776141,
+            -0.013665799982845783,
+            0.0006107363733462989,
+            -0.009412584826350212,
+            -0.00576347392052412,
+            0.012718827463686466,
+            -0.02321716584265232,
+            0.006298186723142862,
+            0.007592111360281706,
+            -0.006514521315693855,
+            -0.013722945004701614,
+            -0.017861871048808098,
+            -0.011967780068516731,
+            -0.013314766809344292,
+            0.002996026771143079,
+            0.01190247107297182,
+            -0.02607441134750843,
+            0.012922916561365128,
+            -0.0037389106582850218,
+            0.00945340283215046,
+            0.004404240753501654,
+            0.013469874858856201,
+            0.018074123188853264,
+            0.009355440735816956,
+            0.0021429345943033695,
+            -0.008024780079722404,
+            0.012702500447630882,
+            -0.01993541419506073,
+            -0.01635161228477955,
+            0.013837235048413277,
+            0.012016761116683483,
+            0.003061335301026702,
+            0.009828926995396614,
+            0.0020959940738976,
+            0.011886144056916237,
+            0.010775899514555931,
+            -0.019739490002393723,
+            -0.0006648199632763863,
+            0.0007326795603148639,
+            0.03280118480324745,
+            -0.0005938990507274866,
+            0.008857463486492634,
+            0.004420568235218525,
+            0.013159659691154957,
+            -0.012784135527908802,
+            -0.01769859902560711,
+            0.0008087027235887945,
+            -0.01057181041687727,
+            0.021257910877466202,
+            -0.008192133158445358,
+            0.006216551177203655,
+            -0.018612917512655258,
+            -0.027086693793535233,
+            0.006392067763954401,
+            -0.016261812299489975,
+            -0.018221067264676094,
+            0.01317598670721054,
+            0.0077268099412322044,
+            0.0017572063952684402,
+            -0.005098143592476845,
+            -0.020457882434129715,
+            -0.009837090037763119,
+            0.028441844508051872,
+            0.011102442629635334,
+            -0.004120557103306055,
+            0.009584019891917706,
+            -0.009094206616282463,
+            -0.012490247376263142,
+            -0.0017541450215503573,
+            0.0036715613678097725,
+            0.008261523209512234,
+            0.005645101889967918,
+            -0.01386172603815794,
+            -0.00029873527819290757,
+            -0.012400448322296143,
+            -0.021225256845355034,
+            -0.008906444534659386,
+            0.016751626506447792,
+            0.007820690982043743,
+            -0.0038960592355579138,
+            0.015812816098332405,
+            -0.004404240753501654,
+            0.002189875114709139,
+            0.014294394291937351,
+            0.005334886722266674,
+            -0.004285869188606739,
+            -0.0095758568495512,
+            -0.004612411838024855,
+            0.004043003544211388,
+            -0.0025960123166441917,
+            0.006457376293838024,
+            0.017192458733916283,
+            0.0072900597006082535,
+            -0.013755599036812782,
+            0.013135168701410294,
+            0.0007791098323650658,
+            0.0049675265327095985,
+            -0.03699725493788719,
+            -0.003247056156396866,
+            0.015796490013599396,
+            -0.013020878657698631,
+            0.02501314878463745,
+            0.005400195252150297,
+            0.022743679583072662,
+            0.015233203768730164,
+            0.008669701404869556,
+            -0.001542912912555039,
+            0.00464098434895277,
+            0.02759283408522606,
+            -0.003151134354993701,
+            -0.008637047372758389,
+            -0.0020602785516530275,
+            0.009241150692105293,
+            0.0002063850115519017,
+            0.003000108525156975,
+            0.008281932212412357,
+            -0.03562577813863754,
+            0.011706545948982239,
+            -0.019576217979192734,
+            0.0024694770108908415,
+            0.003459308762103319,
+            -0.007183933164924383,
+            -0.014865843579173088,
+            -0.005857354495674372,
+            0.022841641679406166,
+            -0.003224606392905116,
+            0.021078312769532204,
+            0.002885818714275956,
+            0.016106704249978065,
+            -0.011575928889214993,
+            -0.009208496659994125,
+            -0.0004130251472815871,
+            -0.005351213738322258,
+            -0.007539048325270414,
+            0.010596301406621933,
+            0.0034409407526254654,
+            0.005534893833100796,
+            0.002265388146042824,
+            0.0036654386203736067,
+            -0.0036878883838653564,
+            0.02773977816104889,
+            0.003563394071534276,
+            0.007996207103133202,
+            -0.011567764915525913,
+            0.014082142151892185,
+            -0.017029186710715294,
+            0.0070778070949018,
+            0.005575711838901043,
+            -0.026156047359108925,
+            0.016849588602781296,
+            0.004775682929903269,
+            0.006628811359405518,
+            0.013812744058668613,
+            0.01618834026157856,
+            -0.010408539324998856,
+            -0.014816862531006336,
+            0.024049850180745125,
+            -0.0032041973900049925,
+            -0.015225040726363659,
+            -0.006775755435228348,
+            4.438935866346583e-05,
+            0.006559420842677355,
+            0.00373074715025723,
+            0.0013020879123359919,
+            -0.0006204306264407933,
+            -0.010743245482444763,
+            0.004645065870136023,
+            -0.01934763789176941,
+            -0.0007163524278439581,
+            -0.016449574381113052,
+            -0.015633217990398407,
+            0.018678225576877594,
+            -0.01830270327627659,
+            0.009633000940084457,
+            0.0012275953777134418,
+            0.028588788583874702,
+            -0.011706545948982239,
+            -0.012155541218817234,
+            0.0015102586476132274,
+            0.010849371552467346,
+            0.0067512644454836845,
+            -0.010033016093075275,
+            -0.007583947852253914,
+            -0.014180104248225689,
+            -0.010163633152842522,
+            0.015502601861953735,
+            -0.014498483389616013,
+            -0.014343376271426678,
+            0.008914608508348465
+          ],
+          "embedding": [
+            -0.006359657085039302,
+            -0.024594786241399234,
+            -0.02553538924848095,
+            0.005360188465088291,
+            0.012801870977589839,
+            0.021680505862506104,
+            -0.016668048966001273,
+            0.06860852474581301,
+            -0.007122272841157724,
+            0.03471775000705488,
+            -0.009004297848861675,
+            0.01059974144678391,
+            0.02565604599426148,
+            -0.022544127646041287,
+            0.010616674976540785,
+            0.013658194823359737,
+            0.016400329743363844,
+            0.02858546964917478,
+            0.0067612676434693686,
+            -0.04174737193606879,
+            0.03184155110488616,
+            -0.02223105127585514,
+            -0.05506239667974977,
+            -0.0015307551410555752,
+            -0.013169788751379287,
+            0.017681749722384007,
+            0.01682706317229353,
+            0.019652295823816928,
+            -0.05623980369271269,
+            0.030698582851533197,
+            0.02286327124296613,
+            0.016019032539689178,
+            0.005508113353872315,
+            8.58985091215149e-05,
+            -0.019206292853495906,
+            -0.018439290483495144,
+            0.03475023176950151,
+            0.01018612942141759,
+            -0.0252551158857426,
+            0.0334562130666827,
+            0.020462153537485785,
+            0.02260508249714211,
+            -0.03323524989392951,
+            -0.002094251009530707,
+            0.015177442347686773,
+            -0.00018698464350038317,
+            -0.0006788648344273403,
+            0.0007777616893581039,
+            0.013352398615059647,
+            -0.030751111528744813,
+            -0.0317733818445192,
+            -0.006729064081710058,
+            0.009413204899986514,
+            -0.005712894407943592,
+            -0.038961826880527066,
+            -0.002827692206530345,
+            0.017863300537973137,
+            0.07918455388563496,
+            0.048967158919816366,
+            0.018002566325350088,
+            0.040264962620950695,
+            -0.019715100359659787,
+            -0.02211350500879408,
+            0.004918148608475574,
+            0.01062585305866606,
+            -0.014478993521696027,
+            0.0034983131226364672,
+            0.000766228629514787,
+            0.009667268771502067,
+            -0.01558439890697323,
+            -0.04984718709870083,
+            0.029613955432821887,
+            -0.009686854114630624,
+            -0.015279319238124568,
+            0.007460629676325322,
+            -0.006235979350809034,
+            0.012514387473717085,
+            0.02800435263741497,
+            -0.008688925946672778,
+            -0.01236301543374469,
+            0.029755705020188972,
+            0.01158037219446542,
+            -0.005242392206024511,
+            0.009982679027728814,
+            0.052414361714483264,
+            -0.018788500381726227,
+            -0.020335831196027204,
+            -0.08583959943356621,
+            0.029309228216812346,
+            -0.0249285282010673,
+            -0.018929396555426307,
+            0.007710663193334574,
+            0.03096798035958051,
+            -0.017089977064591828,
+            -0.028015107605405817,
+            0.06740703271128606,
+            0.009398795412354399,
+            0.01118769022826835,
+            -0.03041417460855722,
+            0.014776002883150835,
+            0.004319194067673161,
+            0.022507636639041203,
+            0.0053562788646786845,
+            0.007548072258943124,
+            0.011123374308563128,
+            -0.006924362115585165,
+            -0.011118507147092024,
+            0.009563232643295304,
+            0.00657178473678069,
+            0.08313027447380966,
+            -0.004239370156003264,
+            -0.00930406163946907,
+            -0.039202672821898946,
+            0.009761029930556298,
+            -0.02827698465424993,
+            -0.037914533246900244,
+            -0.04496050911855815,
+            -0.017711307900236353,
+            0.01934435542287117,
+            0.004335173027873153,
+            0.03800509969714242,
+            0.031198127357716314,
+            0.031311207941680276,
+            -0.012184553894588813,
+            -0.058985863670856876,
+            0.041442017057707635,
+            -0.012456189292721711,
+            -0.0040966351720394135,
+            0.010849179519959421,
+            0.005243123049347636,
+            0.0011495799963862647,
+            0.003534183676934487,
+            -0.03139340512003631,
+            -0.023876261883905306,
+            0.008382957569752911,
+            -0.004778283217011976,
+            0.03703067589888386,
+            0.06888630688839406,
+            -0.03993532296478411,
+            -0.009472167678222372,
+            0.025246872942105057,
+            -0.01524567912700368,
+            0.01595917939878582,
+            -0.003736723164442467,
+            -0.030836797784967753,
+            0.0026246446163368533,
+            0.008894300082148884,
+            0.08342856279154155,
+            -0.0007476580521704578,
+            0.025149116238550588,
+            0.004972542633631468,
+            0.034495632546556576,
+            -0.009173329764686424,
+            0.014009381543772768,
+            0.004427261668151028,
+            -0.013426138607902904,
+            -0.022234568880321808,
+            0.01976282364689966,
+            -0.03592433789490043,
+            -0.010918278220831188,
+            0.013527180289641174,
+            -0.013023113052242677,
+            -0.008304770302495363,
+            0.003677063799167178,
+            0.0013981822663219421,
+            0.047001163242600175,
+            0.01828332714887986,
+            0.002483375662343689,
+            -0.06820498187523893,
+            0.05295312038995555,
+            0.025038753504342162,
+            0.025286870198436735,
+            -0.0029544808216316115,
+            0.029982752196635428,
+            0.0005600481523874767,
+            0.0128290258663539,
+            0.009023805227424919,
+            -0.0019219166367750596,
+            -0.04482843936478531,
+            0.04523479691409549,
+            -0.004678574027241186,
+            0.010486850804218904,
+            0.006736229865018014,
+            0.02134813656476443,
+            0.028083972673690036,
+            0.016857413440125445,
+            0.0189082447820146,
+            0.03843690613820022,
+            -0.011922027011229613,
+            0.016193433524453224,
+            0.03828207911467065,
+            -0.0013055128404331705,
+            0.043198163443700956,
+            0.024025562893260542,
+            -0.001257851946749882,
+            -0.011726713433348844,
+            -0.021907984099443396,
+            0.02752198420597673,
+            0.004431866336999851,
+            0.015798806714413912,
+            -0.012910782191382367,
+            -0.03180549180206364,
+            0.004550247372715741,
+            -0.022778272841118514,
+            0.031242218240017532,
+            -0.006106984949693483,
+            -0.006259993190497421,
+            0.02187449047033339,
+            0.009334146847221406,
+            -0.009381278960016145,
+            -0.020888853823931134,
+            -0.007502374908776156,
+            -0.010390018386728482,
+            -0.04316693435221676,
+            -0.006274513730720055,
+            0.019152443937064258,
+            -0.01668984824845791,
+            0.018573194036505578,
+            0.0035035262183249925,
+            -0.019779911381789077,
+            -0.0027495798165085993,
+            0.02425481236315913,
+            0.006857723545270935,
+            -0.014733535699087731,
+            -0.026326146471941727,
+            -0.019906660231056833,
+            -0.001033294559612807,
+            0.0159154063026895,
+            0.035649534442431415,
+            -0.026210592096240003,
+            0.014397127229517285,
+            0.005114311681443673,
+            -0.0033104236005207077,
+            0.00324988235167305,
+            0.03764004154553922,
+            -0.017822437760737605,
+            -0.008579701063708334,
+            -0.00032410984200965487,
+            0.018450351071676443,
+            0.017283379764846195,
+            0.006384411611332382,
+            -0.005115090309123598,
+            -0.030590047062583125,
+            -0.011776211359193503,
+            0.01183623914060493,
+            0.0016564847796208331,
+            -0.009530766634361575,
+            -0.004810598258154945,
+            0.03876242515526699,
+            0.027076785774401667,
+            0.005324763136165989,
+            -0.03416412480913939,
+            -0.007943016733586697,
+            0.002114307650456928,
+            -0.02961449347300628,
+            0.05142732770518828,
+            0.05566026433842563,
+            -0.0030729309664974704,
+            -0.00938186948393074,
+            -0.04547908981236149,
+            0.0651216359692231,
+            -0.01813722445944094,
+            -0.0482650853560993,
+            0.044045155892545554,
+            -0.014272653003441165,
+            0.013740652306746851,
+            0.04563732439184395,
+            0.02126513340521316,
+            -0.0054845632439384905,
+            -0.011635625101262898,
+            0.030992082069685376,
+            -0.010051935721068779,
+            -0.050967258971854425,
+            -0.009827942095033494,
+            0.013373878556654464,
+            -0.013017627321534534,
+            -0.009361001542607812,
+            -0.01925867665425919,
+            0.020329052915436022,
+            0.023020741670660188,
+            0.019387043947695172,
+            0.005628632318152202,
+            0.018451555161952665,
+            0.016226845400848455,
+            -0.01384033112836216,
+            0.004762589697426251,
+            -0.012718476561725138,
+            -0.024222765127494463,
+            -0.037487144314010595,
+            -0.019971524166659278,
+            -0.015034588610516958,
+            0.009483936496215338,
+            -0.014301924283553591,
+            -0.0024139665139412125,
+            0.013361045353529458,
+            -0.0005870798721107379,
+            -0.011192319340879789,
+            0.013927695434391159,
+            0.021337539669342293,
+            -0.027300952495815437,
+            -0.04765670695199488,
+            -0.00561706224251562,
+            0.0082776639906139,
+            -0.0030984973749633724,
+            -0.026269016057688548,
+            0.007040524859505573,
+            0.0017859008072463442,
+            0.020091542517503808,
+            -0.018716761130867725,
+            0.0035283051960672963,
+            0.008519383548064053,
+            -0.015247417050336866,
+            -0.016917626868166634,
+            -0.018580550690103854,
+            -0.024697180580294884,
+            0.034202028316737125,
+            0.0204395003463227,
+            0.030392895534267717,
+            -0.04527669147753553,
+            -0.018359428092075348,
+            0.05494107499558624,
+            0.015234754907198422,
+            -0.05066008106884831,
+            0.031139824660661085,
+            0.005016615710206006,
+            0.0019000453624929544,
+            -0.006094364589229031,
+            0.0637324238017742,
+            -0.02711381489292883,
+            -0.02187418906973867,
+            0.004601744864532875,
+            -0.04141121277496072,
+            -0.026689585608019588,
+            -0.005568100750837504,
+            0.01291019036002459,
+            -0.031853041426355644,
+            -0.02815800259372657,
+            -0.007964861641790374,
+            -0.010933515363060793,
+            0.013726639077745378,
+            -0.023956800669647334,
+            -0.015672058590455337,
+            -0.023642025374517057,
+            -0.01288435863544158,
+            -0.0042516881552192385,
+            -0.011162214160460695,
+            -0.02412733256710621,
+            0.010088528028211243,
+            0.03310175458095482,
+            0.023064360903748363,
+            0.001996159684818467,
+            0.009478229572933101,
+            0.014478772877286076,
+            0.00371783700962415,
+            0.007864103610320116,
+            -0.048511515124609,
+            -0.031398754206534776,
+            0.020227872561659397,
+            0.012159811587094092,
+            0.0172163448580163,
+            0.009578952796095119,
+            -0.04154545961112031,
+            -0.05095080393682759,
+            0.02968486144855133,
+            -0.03932395023720674,
+            -0.02063219739061335,
+            0.007512423249865194,
+            -0.011911709468952548,
+            -0.00810951734088105,
+            -0.0008079327452253181,
+            -0.01600701450676789,
+            0.023984909975645632,
+            -0.02418072541081482,
+            0.0033984265728439398,
+            -0.0104343782811971,
+            0.01588977364765743,
+            0.05109529246167992,
+            -0.027800269136366997,
+            -0.027889034481533438,
+            0.03706813210665186,
+            -0.02058120652429308,
+            0.030801396468451574,
+            0.03246127847621613,
+            0.008355517405358826,
+            -0.008605467586247932,
+            0.017693344563968573,
+            0.014499487024471321,
+            0.012431277080478217,
+            -0.040103870071440256,
+            0.017328848567062113,
+            0.02723900253727976,
+            -0.054909074068976875,
+            -0.008547596270650966,
+            -0.00777964838164648,
+            0.027134666572242665,
+            0.02765036594970255,
+            -0.002207173084941375,
+            -0.016838740117653725,
+            -0.028289233739722936,
+            -0.019433376533829132,
+            -0.011053380458946764,
+            0.03247641283550457,
+            0.0007514989009289282,
+            0.007919574219000797,
+            0.001316042436488591,
+            -0.0017934342041762042,
+            0.01924145734197619,
+            0.015260199553572459,
+            0.048259602293134723,
+            -0.03383050622747848,
+            0.0021712225589426266,
+            -0.03508567715696004,
+            -0.04578731401250471,
+            -0.008768228935292693,
+            0.010203318819548585,
+            0.024135236371743855,
+            -0.007467140460133838,
+            0.0028907962385387818,
+            -0.007582083854422456,
+            -0.005384477718912037,
+            -0.01867978482547249,
+            0.01641191108381631,
+            0.037168089880543244,
+            0.0031992495663980464,
+            -0.008533890994695162,
+            0.000778843942291118,
+            -0.004620258538699103,
+            0.010572690335163528,
+            0.033233014627100904,
+            -0.030497362482602697,
+            0.02974571152880676,
+            -0.004274442492343003,
+            0.025341347125910928,
+            -0.04307748954707389,
+            -0.018069609916024403,
+            -0.018491122727274337,
+            -0.02069307802429807,
+            0.05372967226264594,
+            -0.030768978301976772,
+            -0.03662045886993282,
+            -0.003205991669059688,
+            0.016815511161004534,
+            -0.0014152644009276566,
+            -0.0016015439520090104,
+            -0.03984716970293867,
+            -0.010828260334550217,
+            -0.05894985435251326,
+            -0.01638035185618134,
+            -0.021819442096404948,
+            -0.005315102328745982,
+            -0.06251579337157112,
+            0.0033713876515840233,
+            -0.07815974854883057,
+            0.001484860071649606,
+            0.04539427129131382,
+            -0.023530943197353476,
+            0.05142250021536681,
+            -0.024746600229063383,
+            -0.0005885740455257141,
+            -0.044698536284421976,
+            0.0026481001040031785,
+            0.02389178603305878,
+            0.051387657872255674,
+            -0.002949198236157656,
+            0.010674377196609455,
+            0.013322075519401089,
+            -0.026245090721950647,
+            -0.01036724558050635,
+            -0.005321884180696542,
+            0.005958798671856245,
+            0.013755178085972732,
+            -0.0074734329537091675,
+            0.038368016046563674,
+            0.010941421833174982,
+            0.03443115781919206,
+            0.034623617136788874,
+            -0.004361926531008214,
+            -0.03348681614127998,
+            -0.00632870804659212,
+            0.024433331840566692,
+            0.0321998807702602,
+            -0.02899983280320611,
+            0.005958471722097962,
+            -0.015602447981516634,
+            0.015633217539747047,
+            0.006321931116948646,
+            -0.011827541320878442,
+            0.02372204564999124,
+            0.015673426811623124,
+            0.006103300725621424,
+            -0.005165006670545588,
+            0.013589526056189892,
+            -0.025411864292353578,
+            0.03041791947375232,
+            -0.004456865298584878,
+            -0.04276058054583262,
+            -0.0024736078733106825,
+            -0.020059309867945227,
+            0.024883721794000956,
+            0.015578237413840115,
+            -0.00536523508503184,
+            -0.005290645798641171,
+            0.008242987948511412,
+            0.020469675635821673,
+            0.006668354647245447,
+            0.00017729517736814364,
+            0.02373604785121342,
+            0.029915687122900485,
+            -0.02432635861789771,
+            -0.029540533943782886,
+            0.008151297444985344,
+            -0.005730733861939129,
+            0.011318816579801282,
+            -0.02644855229221747,
+            0.00033932872948409137,
+            0.0035505684154028996,
+            -0.03570104343808711,
+            0.03408366016010711,
+            -0.009199700348630527,
+            0.005857598530788056,
+            -0.009538177736472163,
+            -0.0017768806148210118,
+            0.0027410153099351694,
+            0.03898108970435115,
+            -0.007367623392751247,
+            0.009827254226889936,
+            0.015465895035342481,
+            0.000981604396724216,
+            0.05143083558797524,
+            -0.027350877485564314,
+            -0.02183036962301852,
+            0.01885468378168636,
+            -0.007285239007489837,
+            -0.016846837665812908,
+            -0.020546383007536074,
+            -0.011863943814684079,
+            -0.016293192750336218,
+            0.01976516340568883,
+            -0.02387328798681798,
+            -0.011892336568452852,
+            0.007170651391639504,
+            0.0013725014050721622,
+            -0.009809669068563384,
+            0.0016617183704652932,
+            -0.006307590349798603,
+            0.02037693835615797,
+            0.031753656615094446,
+            -0.024892675105483313,
+            0.03577919637821846,
+            0.03178303188315581,
+            0.017260898395251155,
+            -0.02102289690701718,
+            -0.009661035772073294,
+            -0.00048542689442823136,
+            -0.014694631967051945,
+            -0.0326379186158631,
+            0.025443206465204822,
+            0.017194626456982255,
+            -0.011381421003142453,
+            -0.03680783761649673,
+            0.0017216115811536721,
+            0.022819859205829785,
+            -0.013600989058799166,
+            0.028141657788496834,
+            -0.01869047298692401,
+            -0.01366833755869951,
+            -0.00782959271979894,
+            0.031227914441058244,
+            -0.05252392878125891,
+            0.016957842026348195,
+            0.00020880696574616416,
+            0.001154370575234767,
+            0.010892067708194324,
+            -0.008368819756628046,
+            0.012224564683325483,
+            0.031700389750709,
+            0.018063405869147564,
+            0.0010794115307202266,
+            -0.002783668216641561,
+            0.01418234057248712,
+            0.017773269840982066,
+            -0.015653443355569053,
+            -0.000497740485657824,
+            0.019890882604100452,
+            0.022672054781919238,
+            -0.00521797370595221,
+            -0.016686942830774276,
+            -0.018443538351123584,
+            0.025270057639201107,
+            -0.010354431070738671,
+            0.03484047753830186,
+            0.04594785214094975,
+            0.0036441959528283004,
+            -0.05695476763071743,
+            -0.009412209619190693,
+            0.0010371110229966086,
+            -0.01099343927053564,
+            0.01606863921935641,
+            0.012470660511773616,
+            -0.03237187923799263,
+            0.006388672213167589,
+            0.012946570716567364,
+            0.022152554488829627,
+            -0.02540317621005867,
+            -0.023183252921885226,
+            -0.0030984019855787655,
+            -0.007890467768631811,
+            0.018648760402624687,
+            -0.008653405980661855,
+            0.0006228315253506687,
+            0.009672526897422602,
+            0.0005436494014569835,
+            0.011632326145022935,
+            0.005655942007108228,
+            0.021736827172951,
+            -0.001256608167230203,
+            0.013124413886287987,
+            -0.01986808920858981,
+            -0.0270450339809516,
+            0.006500423802582728,
+            0.04438242119211055,
+            -0.014813399694814117,
+            0.030621709833336064,
+            0.0092171224268983,
+            0.016433384739642534,
+            0.024875072913970443,
+            -0.00891470467241686,
+            -0.014124009014664287,
+            0.00845777541223868,
+            0.009019007848927453,
+            -0.04147358041382793,
+            0.012513030150663913,
+            -0.01710243399313324,
+            -0.033144939719206276,
+            0.01904674924133127,
+            -0.00534122554342279,
+            0.012367320751926314,
+            0.004270315800911569,
+            -0.002886460711688153,
+            -0.025696286516581048,
+            0.014137901708875033,
+            0.028135087538950946,
+            0.021797869775098042,
+            0.0296173052503884,
+            -0.012757078368361444,
+            0.0005460601780756187,
+            -0.008748930817524899,
+            0.005470975782187157,
+            0.04187083128854477,
+            0.0008687182830616019,
+            -0.0008032802520818105,
+            0.01507726552487313,
+            0.024945744517309208,
+            0.0376130540370386,
+            -0.015415560911023553,
+            -0.03407353556457147,
+            0.01564969972264396,
+            -0.010104443707865262,
+            0.018855792875436674,
+            0.005503259932317811,
+            0.0049634097823351596,
+            0.0038600156268000702,
+            0.019312419743287703,
+            -0.006635193789316492,
+            0.018798045806429395,
+            -0.02500983683871332,
+            -0.028785757874583755,
+            0.01829411751138161,
+            -0.005277972443258226,
+            0.03338623299668019,
+            0.016225885095995672,
+            -0.029522280454919586,
+            -0.01712097344863975,
+            0.01553706187924322,
+            -0.008465766956343768,
+            -0.024842700926275538,
+            0.01895228828512998,
+            0.007076862934523977,
+            0.008956105593078965,
+            0.0024373779389833218,
+            -0.01106503696703573,
+            0.01125466547367417,
+            -0.000818223040067914,
+            -0.015322790791698395,
+            0.014157131555941659,
+            -0.017853386731281715,
+            -0.014994640991761465,
+            0.01775991780643474,
+            -0.037498078532902435,
+            0.009570197712419421,
+            -0.007687297543935123,
+            0.009050494195378254,
+            -0.004222041469820445,
+            0.007460383234429904,
+            -0.0010804379186060264,
+            -0.014987857934522812,
+            -0.014903749199810126,
+            0.03228485430019625,
+            0.022096121825475306,
+            -0.000410816936759558,
+            0.006728733918931589,
+            -3.971636461500232e-05,
+            -0.026719848032785824,
+            -0.011253751790407687,
+            0.02245467425830765,
+            -0.005664635254398481,
+            0.005134496095445943,
+            -0.01109590855846966,
+            -0.0149616794609307,
+            0.0019726017255571673,
+            0.032668637270108894,
+            0.06066635618119076,
+            0.012001732730659747,
+            0.024422093585231484,
+            0.0189499396761026,
+            0.02119276033547528,
+            -0.008489818917377516,
+            0.006413901834269739,
+            -0.03097413287789443,
+            0.03509225573986143,
+            0.005392594635384506,
+            0.03108937318306421,
+            0.026664331248813712,
+            -0.04722525921137681,
+            -0.0058539171830594774,
+            0.017087531454837677,
+            0.022673712717885323,
+            0.0159020681868943,
+            0.0008382498574190522,
+            -0.008938174268517353,
+            0.035650940044285984,
+            -0.03315250299348084,
+            -0.023647111263155,
+            0.005059382024092534,
+            -0.01439537467317451,
+            -0.010488359956731476,
+            -0.006785295580877329,
+            0.02776449681317373,
+            -0.005584052419598926,
+            -0.004582019433008779,
+            0.009697064252638535,
+            -0.014400206558190682,
+            0.008165676294736836,
+            0.0007624778531368436,
+            0.008681263897172326,
+            0.039249866981334126,
+            0.0005918063050375113,
+            0.024275699182767528,
+            -0.004138313072843555,
+            0.011079529739650594,
+            -0.0158972266904151,
+            -0.016985058543605,
+            0.02276982363172991,
+            0.00351101120496393,
+            -0.03244126813432563,
+            -0.00768819222024939,
+            0.00928180979884261,
+            -0.0013793404539641824,
+            0.005266058566444064,
+            -0.00503433670609399,
+            0.003747448261046857,
+            0.03286230666764075,
+            0.0022383592721685153,
+            0.008606190020480513,
+            0.013593947677257625,
+            0.00030951915677929204,
+            -0.01490307680740127,
+            0.013254591138113545,
+            -2.1379308136707722e-05,
+            -0.012900073369514622,
+            0.03357519186606058,
+            0.006705902094864628,
+            0.04243347200485736,
+            -0.03302953674711988,
+            0.009031534935013593,
+            0.01822227379789576,
+            -0.008491770335368761,
+            -0.02725412856390648,
+            -0.03630788717786838,
+            0.008259145747740535,
+            -0.019003260334340084,
+            0.015781275447792516,
+            -0.010039277287105153,
+            0.029665124310477384,
+            -0.03563370794521629,
+            0.004714858110268169,
+            0.004053913355055313,
+            0.0009503673086552651,
+            0.02068360255395961,
+            0.018524093113914366,
+            -0.002785698934204024,
+            -0.031443835341186485,
+            -0.02066365433385339,
+            0.006869321907853112,
+            0.02431260052207595,
+            -0.010367048093621498,
+            -0.005612149485859478,
+            0.0006316058139530722,
+            -0.024369562712568606,
+            -0.016845124984022828,
+            0.030359419843284823,
+            -0.040974427176153246,
+            0.005252911449822132,
+            0.04272936053727608,
+            -0.04599092715016919,
+            -0.03802288716943514,
+            -0.002314988728575789,
+            -0.024124014894379466,
+            -0.006360771882090754,
+            -0.002068030981218464,
+            0.016669991113175926,
+            0.0036232707799735537,
+            -0.028763764013482795,
+            0.00943401978066451,
+            -0.005173988601101547,
+            0.003971420424839974,
+            -0.0031669592494977393,
+            -0.02216336363004636,
+            -0.01657772032717298,
+            -0.02531962163186398,
+            -0.035174102848865116,
+            -0.0015903434876311088,
+            -0.005158703150821607,
+            -0.028703075279214722,
+            -0.03941940849540492,
+            -0.021696000925262537,
+            -0.0160221488026189,
+            -0.00843213135542359,
+            0.040641882367348774,
+            0.021319060666253945,
+            0.019336844555345886,
+            0.0025252167889382917,
+            -0.020346900609125973,
+            0.049440028814233025,
+            -0.007642182460707816,
+            0.035628691886108926,
+            -0.0017882888000877158,
+            -0.013461669667950893,
+            0.00455558389906464,
+            0.03825428026120858,
+            -0.014248434654950971,
+            -0.0009902062564027718,
+            0.00027811655483232545,
+            -0.0011866253757890136,
+            0.0008535155950871286,
+            0.015433650396777696,
+            0.001307651096343185,
+            -0.007112199543314309,
+            0.030380673552683906,
+            -0.012803247194895328,
+            0.01637501752192592,
+            0.018350814869962304,
+            -0.0013359719137099667,
+            0.008064546833360553,
+            -0.021430659268086814,
+            -0.00731572118780415,
+            0.0044578576331236325,
+            0.008422210247967642,
+            -0.015096885054883586,
+            -0.009964588397149773,
+            -0.02359494467299634,
+            -0.01174145797396426,
+            0.0023855749166744366,
+            -0.004559202896782688,
+            0.004902138778592298,
+            0.034595817216889944,
+            -0.016068744500427584,
+            -0.035106033294403614,
+            0.009997207702122002,
+            -0.02663088875167486,
+            0.02222306406130687,
+            3.983384942889326e-05,
+            0.006838221769859366,
+            -0.0023434342406307616,
+            0.001765445386886839,
+            -0.00169484278667461,
+            0.002862331185439823,
+            -0.018217975204395134,
+            0.0031859734246840257,
+            -0.0032119988065673565,
+            0.0009052950782312657,
+            -0.0012459653052875638,
+            0.00687339892509406,
+            0.006332799551164718,
+            -0.0008019538898592668,
+            -0.002968451740267742,
+            -8.541086355097327e-06,
+            -0.020561816931299912,
+            -0.021642512405352372,
+            -0.028259049261410905,
+            -0.000989887278779396,
+            -0.012359733017834338,
+            -0.027804394906433664,
+            0.011443385424045703,
+            -0.006872684999639775,
+            0.008327844771213648,
+            -0.008969527138034904,
+            0.006219994886600349,
+            -0.008415106130309062,
+            -0.0004143996206881767,
+            0.013119286511794273,
+            -0.02925209282917943,
+            -0.02096907906398536,
+            -0.03096513357712367,
+            0.01762223841790055,
+            -0.00925741670612824,
+            -0.002656382260603827,
+            0.0025223527960181835,
+            0.005092659166535706,
+            -0.033606542487570154,
+            -0.006653438998497412,
+            0.007313682256676618,
+            0.009547290751154595,
+            -0.03287788897629887,
+            0.02446728061355753,
+            0.017023222076526946,
+            0.01874233066565012,
+            -0.02417566312664911,
+            0.04049827067897005,
+            0.014426879325647042,
+            -0.021740606525480655,
+            0.016712907564723732,
+            0.027385150577780248,
+            0.03177163221499527,
+            -0.03469098547555545,
+            0.004481376248622418,
+            -0.01518200063132508,
+            0.018918680020516044,
+            -0.025284921244460287,
+            0.020625170445627613,
+            -0.016707629602243063,
+            0.021278719363178687,
+            0.03520334001264928,
+            -0.005524369131116028,
+            0.03608133014180745,
+            -0.019234548780647943,
+            0.0059985801736444555,
+            0.004041778461022664,
+            -0.01645503683802124,
+            0.01024694092614158,
+            -0.0042700351372843386,
+            0.0006059159967286885,
+            0.009406090340935747,
+            0.019133229156115282,
+            0.019007219188041775,
+            -0.04442208547053466,
+            0.028588965123434923,
+            0.00804120101594811,
+            0.021472086070876265,
+            0.014177820346835521,
+            -0.02708807970982664,
+            0.004742981711172693,
+            0.002072673190279628,
+            -0.004659713164250794,
+            0.0061167904730676514,
+            -0.013796604518930328,
+            -0.034540857808309226,
+            0.006338792319814448,
+            -0.018918703778187926,
+            0.015036937364271984,
+            -0.009026670935442384,
+            0.0011667525636770179,
+            -0.01152652811400482,
+            -0.011938679695598347,
+            0.0006170592505446954,
+            -0.005495151258143897,
+            -0.0008075736908949407,
+            -0.03171721592827769,
+            -0.04427658539412448,
+            0.02036132271908089,
+            -0.0021221965958423804,
+            -0.04320396743189005,
+            0.004111179168317842,
+            -0.025352428814134187,
+            -0.01865918186335911,
+            -0.011237699157960728,
+            -0.025047923887906474,
+            -0.04219630696740606,
+            -0.011108496740166623,
+            0.007785006066683012,
+            0.016810642610918525,
+            -0.009081815651299525,
+            0.040965574143584864,
+            0.0073944364393447965,
+            0.015313576815177263,
+            -0.017731360025694955,
+            0.026137777790243826,
+            -0.019314392060123,
+            -0.03108486330987073,
+            0.02676829566273746,
+            -6.259161089416372e-05,
+            0.003053748984540415,
+            0.014358356168574803,
+            2.8186924071913628e-05,
+            -0.006181789640967465,
+            0.01947730374003864,
+            -0.028739300217145797,
+            -0.023460503178221397,
+            0.0012460072421725688,
+            -0.008850395570726385,
+            -0.016271242375881602,
+            -0.035896509136022194,
+            0.005227881053956234,
+            0.03324337105109496,
+            0.004970419987430122,
+            0.008273497228454482,
+            0.01814500594735864,
+            -0.02229593571615308,
+            -0.024230675455380678,
+            0.000639112529814716,
+            -0.010266640429960994,
+            -0.010257247303470066,
+            -0.021302483296731352,
+            0.019150757066356872,
+            0.01651375496287087,
+            -0.00390189952706486,
+            -0.006971692709239087,
+            -0.010621003777733253,
+            -0.01918206725910585,
+            -0.02020345113211036,
+            -0.013827322378214754,
+            -0.017980206764311026,
+            -0.034153741666571154,
+            -0.032195001867229935,
+            0.03225205339201753,
+            0.026354817588919596,
+            0.0014076409241492998,
+            -0.007959081015805435,
+            -0.023070419006792155,
+            -0.004483328183581872,
+            0.02879516595345542,
+            0.002549934782150634,
+            -0.0001541849716285838,
+            -0.00830090759167373,
+            0.015341095409025125,
+            0.00835695019244165,
+            0.005107844984733648,
+            -0.06676974053142512,
+            -0.006027540973808174,
+            0.01655283585235268,
+            0.01351155740684989,
+            -0.002359114145814062,
+            0.02591301169926259,
+            0.01108099264396926,
+            -0.015361038608114896,
+            -0.019765743462262668,
+            0.035195761515422364,
+            0.02408910733949848,
+            0.023468767772473746,
+            -0.0034492999385544727,
+            -0.004472311490618405,
+            0.007030833258457338,
+            -0.010684252427867153,
+            -0.013326964647936429,
+            -0.004558513842504658,
+            0.026925146859337397,
+            0.016099443021514896,
+            -0.0064331198280825575,
+            0.010848931851939124,
+            -0.0008320209125061849,
+            -0.019444491995041537,
+            -0.001841330829156697,
+            0.008730275423516445,
+            0.0012974082030866743,
+            -0.014611902493400719,
+            -0.005911807768916134,
+            -0.019130604101278263,
+            0.024719468317569562,
+            0.0209456329723249,
+            0.03381727436368338,
+            -0.013237902591972769,
+            -0.0003695142331921278,
+            0.005410360755058108,
+            0.0002841632253252816,
+            0.0023352645111704958,
+            0.004801123870325242,
+            0.02623649813035263,
+            -0.029010710171522985,
+            -0.020741774066460415,
+            0.0068426980682519505,
+            0.013445270043414486,
+            0.0022744032984779294,
+            0.033840485942696216,
+            0.035040080489222324,
+            0.004166066423508316,
+            -0.010046666684988869,
+            -0.006148765564658713,
+            0.007071740742464309,
+            0.01865897335451031,
+            -0.023267411753900076,
+            -0.0005936657905496225,
+            0.004833030519903764,
+            -0.004229582648167378,
+            0.03620610659071198,
+            -0.009961238880412731,
+            -0.007197670122096852,
+            0.022212791947347127,
+            0.09819205436926473,
+            0.00047695621444943017,
+            0.0025354521225913395,
+            -0.029358210234742176,
+            -0.023614016211567963,
+            -0.0012564198366457442,
+            0.00242680708457405,
+            0.00830167274308547,
+            0.028076260517106013,
+            0.0004852400049405906,
+            0.011579274037419117,
+            0.02773972145320728,
+            -0.017217305534161797,
+            0.006752070301141707,
+            -0.011376051474975524,
+            0.024577071515205502,
+            -0.012694034026935636,
+            -0.0019809719792138673,
+            -0.016512649783050146,
+            0.015907402258634816,
+            0.004556574687833768,
+            0.004907510649166062,
+            0.011347630984408937,
+            0.009824484201337644,
+            -0.02650731900265866,
+            0.01756773113723491,
+            -0.005671986127099093,
+            0.004449611508541289,
+            -0.027841346572384462,
+            -0.020124233970502267,
+            0.0025986391930124847,
+            -0.0014440758308989809,
+            -0.009579444248630085,
+            0.006005274872617889,
+            -0.0024984235450999127,
+            0.03765664613203687,
+            -0.03765496603611941,
+            0.01409350051861866,
+            0.0066453175608372555,
+            -0.0015390841089599882,
+            0.011806653942640485,
+            0.017306914776093522,
+            -0.008206256939290151,
+            0.02589800448884443,
+            -0.00798213351353373,
+            0.009846274212000523,
+            0.015230070803055296,
+            -0.03449531102975745,
+            0.009174739184806312,
+            0.010145762060174144,
+            -0.024165660737765746,
+            0.012146877603301336,
+            -0.013784371016298891,
+            -0.004818759788054113,
+            0.005930446700613062,
+            0.007349837554764034,
+            0.010733731974181452,
+            -0.02295571464086674,
+            0.016176424826629898,
+            -0.010947841926210105,
+            0.0037766665765835445,
+            0.013559846998199008,
+            0.024892849808979323,
+            0.0012426907454129122,
+            0.0005950515136086193,
+            -0.01379469431504576,
+            0.005534724429523084,
+            -0.015667467221535825,
+            -0.022401586867694584,
+            -0.014192041775234079,
+            -0.006366846766730035,
+            0.01029598687444803,
+            -0.003972386708834858,
+            0.016971476088808737,
+            -0.010309648640697812,
+            -0.010645860303528099,
+            0.0037818310327362103,
+            0.01040806357838676,
+            0.008316605962234849,
+            -0.02855914079113881,
+            -0.005676310556150579,
+            0.004961084138163161,
+            0.00029285971503001265,
+            0.009592497954559435,
+            -0.014556073011576487,
+            0.013689526487160009,
+            -0.013290113354287095,
+            -0.0010726537498214612,
+            0.013201860397060236,
+            -0.012443552043930505,
+            -0.004012305744087929,
+            -0.01592905426965635,
+            -0.028487066798154617,
+            0.008462143660968396,
+            -0.01365614159523485,
+            0.001076882584445294,
+            0.0019503240125505646,
+            0.004730085646042709,
+            -0.016919069570642004,
+            -0.007860046623166481,
+            -0.033515448866521046,
+            0.011065416478687504,
+            0.02314589687882036,
+            -0.006435733825418645,
+            -0.030351727858789472,
+            -0.009546333526852862,
+            0.011327489786114963,
+            -0.0071765122743172715,
+            -0.01676736383051139,
+            -0.0029390038145944096,
+            0.007617910650439855,
+            -0.008174620754097493,
+            -0.00508943241317381,
+            -0.0025173969010414098,
+            0.0217328037301181,
+            0.016094975605203832,
+            0.01769642142712967,
+            -0.021400862603034036,
+            -0.00976992221747187,
+            0.026043879119776864,
+            -0.03275550610951898,
+            0.012441465975914529,
+            0.024587446266013183,
+            0.003446527051428522,
+            0.005492228054838714,
+            0.0030377554109693714,
+            0.005459159163772825,
+            -0.006985179226831048,
+            0.019925583798328282,
+            0.010870421035240845,
+            -0.019709062313471467,
+            0.005385443078244229,
+            0.02086085539375208,
+            0.0037248737640193097,
+            0.012892299156840998,
+            -0.017546870090541213,
+            -0.0020341807891543716,
+            0.015594779287017958,
+            0.04816214672670215,
+            -0.0032851797178091685,
+            -0.010450846098750195,
+            0.029513742590677183,
+            0.023526004023637855,
+            0.005373455940309873,
+            0.0012120777808635214,
+            -0.02525630456877349,
+            -0.013075130701774785,
+            0.01962235086017781,
+            0.016944870494393874,
+            0.013484925533584295,
+            -0.0023255656480479004,
+            -0.019236081983457366,
+            0.021520156296467717,
+            -0.01701733319731161,
+            0.012422659010697477,
+            -0.023764941037007197,
+            -0.001490790833263327,
+            -0.0035419008954878724,
+            0.021998949056452454,
+            0.0021819635638621092,
+            0.008891505304514552,
+            0.024354552127458608,
+            -0.006978548952551799,
+            -0.019526615985982355,
+            -0.0051799020400150305,
+            -0.014578953088901798,
+            0.022506635912517146,
+            -0.024117708614271965,
+            -0.007008641019892289,
+            0.011379795968360019,
+            -0.012362093847309038,
+            -0.0102534051092702,
+            0.001164934138876502,
+            0.017686807394319725,
+            0.0009423993435129515,
+            -0.014784508280754078,
+            0.01944445593807992,
+            -0.0011359271135469456,
+            0.010410915597949927,
+            -0.01921419010977189,
+            0.009390638453648025,
+            -0.016682442516992777,
+            -0.007849184048209236,
+            0.010343008873053859,
+            0.0012604521199837046,
+            0.003948300703645506,
+            0.002592490121754231,
+            0.007773437307969315,
+            -0.03355658426684504,
+            0.022695505487314516,
+            -0.010751606023585792,
+            -0.003538523688786008,
+            -0.0014125330167969218,
+            -0.01678692701672328,
+            -0.0006918162525881059,
+            0.005816502852598842,
+            0.010878358093847386,
+            -0.016045147059003256,
+            -0.00873270513444376,
+            -0.04920244395446931,
+            -0.030314009649638693,
+            0.023156333024481057,
+            0.013733657599720879,
+            -0.010971506342695331,
+            0.018852159415605117,
+            -0.023760853159532587,
+            -0.010587249404421732,
+            0.0011379869499042905,
+            0.023319856329056813,
+            -0.03539290744576506,
+            0.01215369703816749,
+            -0.014493682360886549,
+            -0.02039288669534806,
+            0.009070984596830142,
+            0.011445354735283254,
+            0.0007259852045513916,
+            -0.0011514627555568814,
+            -0.029350945123925453,
+            -0.010246845821727087,
+            0.02391068177425878,
+            -0.00861718589055971,
+            -0.004723044986826333,
+            0.024716520302409263,
+            0.0033721801437248873,
+            -0.012524663490691517,
+            -0.017955758068233754,
+            0.021612671655463236,
+            -0.026077203994996058,
+            0.007645944309444457,
+            0.010032183011253845,
+            -0.0173991263874487,
+            0.021649599230588117,
+            -0.007529701797485131,
+            0.020327763440441377,
+            0.004227390938169872,
+            -0.011249233530251522,
+            0.020352283974471443,
+            0.013644443758970054,
+            0.004216958800094986,
+            -0.004081669404784771,
+            0.006869353098014179,
+            0.02212732880780841,
+            -0.011021739614266063,
+            0.02219580717345023,
+            0.028777213295471067,
+            0.00017328191188631918,
+            0.03260179554743322,
+            -0.021287655520443285,
+            -0.03159728983004563,
+            0.021740163175540124,
+            -0.01035146858119169,
+            -0.021564906188712388,
+            0.028914245422221707,
+            -0.013674731836508768,
+            -0.00786476052825481,
+            -0.0166718429993922,
+            0.00533027311342322,
+            0.016680336311088843,
+            -0.005246190451159961,
+            -0.014360934066628332,
+            0.0029628103834216345,
+            0.019809487418628386,
+            0.014587234544007958,
+            0.029780009648869975,
+            -0.010778199815920887,
+            0.035180242185168895,
+            -0.002320576872571577,
+            0.008990157317473909,
+            0.012415465232073404,
+            0.006489435152972723,
+            -0.011251531176923445,
+            0.013501894212117511,
+            -0.003446759011266526,
+            0.008230548711588447,
+            -0.001449023474280409,
+            -0.035111679577202855,
+            -0.013709382199848678,
+            -0.04709294504969336,
+            -0.0023752976043235697,
+            -0.0025098360125255215,
+            0.0030975417740913154,
+            0.09736815869389318,
+            -0.005723049647039619,
+            -0.011988817571623457,
+            0.001625059666070763,
+            0.009611697638805556,
+            -0.0009896494420056814,
+            -0.007606836952841873,
+            0.015973152482488318,
+            -0.01343292635775299,
+            -0.02140928016038583,
+            -0.010532701190806147,
+            -0.01288594454368977,
+            0.012079198190522715,
+            0.015929716343161652,
+            -0.0005947359502228821,
+            -0.015207331663761012,
+            0.006524010655124294,
+            -0.004566385899752684,
+            0.0064717704997400245,
+            0.008647099975176308,
+            -0.010414668624343924,
+            -0.0071698879102925555,
+            0.030899143952119196,
+            -0.025301269549419674,
+            -0.024361748354052457,
+            -0.0010579931222861095,
+            -0.009055796737486008,
+            -0.010335459737012047,
+            -0.002692653633719985,
+            -0.0017679501322462056,
+            -0.010519994985692081,
+            -0.010019865263564572,
+            0.006537076627706128,
+            -0.015472130358273033,
+            0.004812857776601667,
+            -0.0047457011269495634,
+            -0.013296069496526007,
+            0.017054968086538724,
+            0.022563785357424853,
+            -0.02841491200807702,
+            0.0006755428946168761,
+            -0.017422510783648978,
+            0.012941211381006863,
+            0.008866069010589792,
+            0.006049618782357251,
+            -0.022451528209097505,
+            -0.023622906582403243,
+            -0.015019272754405791,
+            -0.01927535750568903,
+            -0.024973490129133732,
+            -0.009073381957684092,
+            0.0067692803507600945,
+            -0.001267339788153158,
+            0.021918281111033645,
+            0.000816538714279555,
+            -0.01812545613159305,
+            0.0061084913570894905,
+            0.000568371220952269,
+            0.018292497018668842,
+            -0.027468677288882186,
+            0.0033806422195878524,
+            -0.02307582511848465,
+            0.033508234091571124,
+            0.0039195050746594975,
+            -0.026680111514794824,
+            -0.021558668725755275,
+            -0.003311830287529699,
+            0.018994577319066473,
+            -0.0025745912019675805,
+            -0.00724049447539589,
+            0.014775207617625293,
+            0.0155334760323838,
+            -0.004693118176196483,
+            0.002015625406296403,
+            0.027968079002437102,
+            0.01077998866180723,
+            0.0313892866933904,
+            -0.015236169297590675,
+            0.004945828495617267,
+            -0.027215016713379062,
+            0.010214419950212732,
+            -0.01503548137805401,
+            0.008859554812646242,
+            -0.01290890682557532,
+            0.03531384210182567,
+            -0.012005613861257432,
+            -0.004250698950192301,
+            0.03253288796441852,
+            -0.029150986393345252,
+            -0.0035020051536906334,
+            -0.009255437891304041,
+            -0.004142204892160768,
+            0.013240580426083011,
+            -0.001198093874819309,
+            0.008485141356876356,
+            -0.01854370462655459,
+            9.726096864073609e-05,
+            -0.003221852304410318,
+            0.005404595442127598,
+            -0.02005458688172967,
+            -0.003314118809778734,
+            0.0073427568419969814,
+            0.004259518237708634,
+            0.002408630356035373,
+            0.016420646142725346,
+            0.006662121128241274,
+            -0.0029357701493526907,
+            0.03424224243607692,
+            0.008171895850772245,
+            0.0152521424877615,
+            -0.0018638264193202728,
+            0.0030234211261443034,
+            0.019287963516760636,
+            0.019912956661192856,
+            -0.00048443399729935116,
+            -0.01058935153883094,
+            0.00195113175754497,
+            -0.001140732137433033,
+            0.011048741602327437,
+            0.024396046054912645,
+            -0.021158025221644738,
+            -0.02419562092713724,
+            0.016377661005279716,
+            -0.015076145951740961,
+            0.01903712233378994,
+            0.010541463112889985,
+            -0.010099351302095792,
+            -0.00031558616108876247,
+            -0.042316016201379245,
+            0.010964989031040692,
+            -0.02068939391415103,
+            -0.04004120028915868,
+            -0.00244383573783919,
+            0.0009542507389088937,
+            -0.029439456243505052,
+            0.02822814855547978,
+            0.0023755032182188156,
+            -0.017280221740813364,
+            0.018223709137876153,
+            -0.005548654680726505,
+            -0.0024040301239229064,
+            0.005397711787238679,
+            -0.01092026090471121,
+            -0.010648485086769927,
+            0.008235250252660972,
+            0.010957715075632535,
+            0.005425683656535705,
+            -0.005698919020608095,
+            -0.011284911899894062,
+            0.009975882651638084,
+            0.012318901214538077,
+            -0.0019922884373083155,
+            0.010050520101815593,
+            0.0031720752428247446,
+            0.007352859750323051,
+            -0.0073483049386141005,
+            0.0046483587496626965,
+            -0.005093220128411125,
+            -0.026080057085901446,
+            0.012863580653903103,
+            0.0052434400313520774,
+            -0.011863360599637747,
+            0.0103209502958491,
+            -0.01048162756158923,
+            0.005787157766911021,
+            0.001318945460232246,
+            -0.0059333230422061395,
+            0.004821320238893602,
+            -0.0038922239931512784,
+            0.011267634571983924,
+            0.013256611037442042,
+            0.004592529900805719,
+            -0.013076954459595086,
+            -0.02177318635989856,
+            0.009001291224623467,
+            0.01959767163356408,
+            0.024161754192119852,
+            -0.030416109293488064,
+            0.003784634588325551,
+            -0.0014915849060829743,
+            -0.0020550564975080885,
+            -0.015632785947151753,
+            0.01587790981601866,
+            0.01716742056401084,
+            0.012721563202528083,
+            0.01893393006967701,
+            -0.004009617703771589,
+            0.010539100665843704,
+            -0.02080068888791658,
+            0.013642451766699067,
+            0.0008609869904066794,
+            -0.013629067231663643,
+            0.0014620655123815298,
+            0.027046304278766087,
+            -0.011056783825898616,
+            0.006206969699250696,
+            0.008097527175511791,
+            0.022111870736907183,
+            -0.012509479100313492,
+            -0.003735868296238069,
+            0.01516254968167132,
+            0.0014878674796313539,
+            0.0035887079417890352,
+            0.016533149527222273,
+            0.04729052276110659,
+            0.00102751186013391,
+            0.0016713773449738153,
+            0.0192177920520243,
+            -0.005563519063161466,
+            -0.01114821419109643,
+            -0.003549999863325982,
+            -0.021807893906469106,
+            0.010998558936481785,
+            0.00431279785115251,
+            -0.02186324212032519,
+            0.01024093366826206,
+            0.0125495473550584,
+            0.005268320135120931,
+            -0.0031679212410026603,
+            0.02574575247827441,
+            -0.019807879421996258,
+            0.014935295894965128,
+            0.03329450628384787,
+            0.00493356778223407,
+            -0.011423949333937285,
+            0.007263264249871384,
+            0.0027354459678742013,
+            -0.004494406041906684,
+            -0.00606745413660583,
+            0.010497391290734088,
+            -0.008191702034391382,
+            0.009870458191510176,
+            -0.009580130479693,
+            0.039202091799313755,
+            -0.009619018331262844,
+            -0.010782138248435458,
+            -0.004558645539551881,
+            -0.006241773108460662,
+            0.003478414182938609,
+            -0.003395165694642073,
+            0.005812225135633631,
+            -0.0069045407353789945,
+            -0.014607146220271209,
+            0.0008522982757210454,
+            0.00020175973815948303,
+            0.014001922919116725,
+            0.0011804904415274088,
+            -0.004871874532755353,
+            -0.007877217154706205,
+            0.015243277056007988,
+            -0.006047906208304912,
+            0.0006531937033317192,
+            -0.0011653523616313422,
+            -0.0013662556399359335,
+            -0.007401050012179761,
+            -0.015166105795085528,
+            0.00974609244984173,
+            0.016253733908242134,
+            -0.013857834499211056,
+            -0.005142936208363415,
+            0.012274716212832219,
+            0.008540229125193836,
+            0.011742606168187876,
+            0.0007451338650642946,
+            -0.009255813682572937,
+            0.008292531177374192,
+            -0.015020523035432158,
+            -0.008781317965985481,
+            0.0026250469418136603,
+            0.002381147382977664,
+            0.005839723981046825,
+            0.0006240641828668715,
+            0.004502463508605272,
+            0.005798912705218846,
+            -0.00497936644919367,
+            0.01112666823857836,
+            0.008198424171006548,
+            -0.002452143482105151,
+            -0.01017569410837814,
+            -0.001127589984272638,
+            -0.015733298167661346,
+            -0.016758315411028353,
+            -0.0009286266498936771,
+            0.015175988113750882,
+            -0.02352862898718829,
+            0.018996352975391644,
+            -0.0014524363611708201,
+            -0.0033802546171857704,
+            -0.021392428456030312,
+            -0.0025923983911570183,
+            0.0034214188668518796,
+            0.01255768230394476,
+            0.01858853630745928,
+            -0.02729543519465384,
+            0.010575894388563116,
+            -0.007204725245442141,
+            0.030394929424014717,
+            -0.018328782140005044,
+            -0.004022941273610227,
+            0.026915618079621992,
+            -0.02200018306828758,
+            0.0008791185377513978,
+            -0.011454606955158956,
+            -0.008103032693853504,
+            0.019287674837688177,
+            0.023747720480736777,
+            0.02038287242167783,
+            0.01364489294869211,
+            0.0007758044919789085,
+            0.004768030222445397,
+            0.03572705648994403,
+            0.010568241270862248,
+            0.014828087092859878,
+            -0.007542489523110974,
+            9.457808640264811e-05,
+            -0.02554033312739386,
+            0.01635899434369978,
+            2.587383534566272e-05,
+            -0.012370727506722627,
+            0.007162422186621379,
+            -0.013370882403931942,
+            -0.011066483600541058,
+            -0.007151077450730025,
+            -0.01721910018182534,
+            0.02152239849400241,
+            -0.012322440280680164,
+            0.004587729883596212,
+            -0.009079747696669277,
+            -0.013272707286450417,
+            -0.012953682633400515,
+            -0.014202214807187545,
+            -0.01313141988034687,
+            -0.026041403779538307,
+            -0.027109731260915763,
+            -0.006573285295895077,
+            -0.023780366346334524,
+            -0.013799072648505526,
+            0.005588622594136253,
+            -0.022617902666316757,
+            0.012262325666428604,
+            -0.012689827812482627,
+            -0.01226264872059864,
+            0.012805734807835008,
+            0.020274844315616616,
+            -0.011765231803328348,
+            -0.028494489703400308,
+            -0.018583843821043854,
+            -0.019970998470704358,
+            0.012576201137505969,
+            0.002266288538944273,
+            0.013165889075444961,
+            -0.003018437854660793,
+            0.003227412667880239,
+            -0.014895831590298957,
+            0.018570624675927183,
+            -0.03442390703780963,
+            -0.03108003806963536,
+            -0.013611306765348361,
+            0.005182777066476028,
+            -0.0015782033203578553,
+            -0.011429046504256945,
+            -0.0018606648572920801,
+            -0.00433805271182878,
+            -0.025109572024446088,
+            -0.010134856632190067,
+            0.011010662054484161,
+            -0.015205629927469414,
+            0.0072933565255675036,
+            0.0009018832564038863,
+            0.013870902834260887,
+            0.009587432944616442,
+            0.013393301893492882,
+            -0.006095063698384775,
+            0.004356485794113208,
+            -0.0026476612747324386,
+            0.0004894024149759108,
+            0.009336366919474478,
+            0.00839591561027103,
+            0.010551161369079399,
+            0.002260707777260338,
+            0.007830059223001696,
+            -0.0060927446216633265,
+            0.018065002687169148,
+            0.015436944875538481,
+            -0.0002520689442496089,
+            0.013323098485251487,
+            0.0003769862591044255,
+            0.00022293304921127018,
+            -0.0029970303106340466,
+            0.007888437018607987,
+            -0.011785957933642163,
+            -0.001311704406355858,
+            0.015138190893483804,
+            -0.01707247730569978,
+            -0.013045187198628258,
+            0.03107034781283638,
+            0.0018583638488268697,
+            -0.02495564203304796,
+            0.031146554085787762,
+            0.004220047610942943,
+            -0.005835199594223787,
+            -0.013205086624848782,
+            0.0014059047822222418,
+            -0.011570624844100298,
+            0.003584232809362274,
+            0.014295032061446826,
+            -0.02231984223610487,
+            0.010521094801839404,
+            0.008446892441796151,
+            -0.009681535919980559,
+            -0.012337659795586223,
+            0.009847073709209602,
+            0.016809328445200358,
+            0.004472485504034416,
+            -0.03539093810210523,
+            -0.011957694102753411,
+            0.02145470732008716,
+            -0.029237428225009913,
+            -0.00810041956259673,
+            0.007452962400105787,
+            0.0065154832040426065,
+            -0.019353109017609053,
+            -0.008580955020412362,
+            0.005725794605897509,
+            -0.004940390374637084,
+            -0.003892210172017758,
+            0.017541623128534682,
+            0.00485047043626186,
+            0.013698341792101025,
+            -0.01689738578307474,
+            0.009771458057400557,
+            -0.022222817828602557,
+            0.004021882074676753,
+            -0.023373700500461992,
+            -0.009873403167702603,
+            0.002364143609079849,
+            0.012454271200139762,
+            0.014328556467632212,
+            0.027839363143659326,
+            0.030056404238348527,
+            0.013995696971422429,
+            0.007833677585250953,
+            -0.016265199386953717,
+            -0.018134327237616175,
+            -0.006409513840075933,
+            -0.004033173350204784,
+            -0.008714798059218596,
+            0.0066373547199279235,
+            0.01858354212009877,
+            0.004499712802472377,
+            0.02609078206670711,
+            -0.006227189596272876,
+            0.009039744622014585,
+            -3.127526911951914e-05,
+            0.0008833930131827993,
+            -0.004226083827636182,
+            -0.01971685546584527,
+            0.00481003211084742,
+            -0.0013721482162825508,
+            0.025874758880591167,
+            0.00940496860916206,
+            0.0024915013399541847,
+            0.01615978314811047,
+            -0.006378952757397484,
+            -0.01174653607178726,
+            -0.010540107733238588,
+            -0.0061926268398085176,
+            -0.021752582127981104,
+            0.029059100503124387,
+            0.021195595568253612,
+            -0.010800765264123782,
+            -0.013693582166598082,
+            -0.00903369526059487,
+            0.02795767990556847,
+            0.004080965688836953,
+            -0.0013050212744006933,
+            0.016313311850308505,
+            0.012084555958999215,
+            0.012329325712491252,
+            0.009794698207689061,
+            -0.00036157148861933123,
+            0.0015364709122308352,
+            0.03722522141440073,
+            -0.007427028510548186,
+            0.0026463292682538495,
+            -0.005239324411651515,
+            0.0025987932868461313,
+            -0.009705817520422487,
+            0.007242122655628283,
+            -0.004012676385296197,
+            -0.0158760051470016,
+            0.015003444174308607,
+            -0.022443331305169994,
+            -0.0016749787204471238,
+            -0.008150635190952799,
+            0.0016696640640947303,
+            -0.02781023337840043,
+            -0.0025731206607371845,
+            0.006695065701630532,
+            -0.015547340828489069,
+            0.011816010501093885,
+            -0.01740673229051557,
+            -0.011553604323149105,
+            0.007914961618894248,
+            -0.012034157288065447,
+            0.0016684130271324446,
+            -0.0008268957621923278,
+            0.004460864332388434,
+            0.013426055079012884,
+            -0.011008808570221245,
+            0.012166046631826273,
+            -0.005816284310138776,
+            -0.007872091214834688,
+            -0.0016055129487028885,
+            -0.0030049472973942043,
+            -0.0025600746761171543,
+            -0.0009966231170649917,
+            0.034355910234323155,
+            -0.004446570896048543,
+            0.003761242147499512,
+            -0.01362639743248326,
+            0.0019833052855181806,
+            -0.036176376394999044,
+            0.028344464817309477,
+            -0.008642835889481802,
+            0.011013502221008498,
+            -0.0017508365480865596,
+            -0.013243706268954915,
+            -0.01456517309747086,
+            0.015247299501245178,
+            0.007602596572394442,
+            -0.005854531057349665,
+            0.013558248087347848,
+            -0.019724373614448298,
+            0.008530500822910953,
+            0.004463694543554244,
+            -0.015561397976987216,
+            0.004668780190328835,
+            -0.007479912658259757,
+            0.012657201279639194,
+            -0.01774730033362932,
+            0.004314594593150765,
+            0.00958571617187387,
+            -0.01892876084898881,
+            0.009901041800918215,
+            -0.028675879564512306,
+            -0.004631920242208613,
+            -0.012323455524366223,
+            0.0053804608827285865,
+            -0.000660864620708182,
+            -0.02389088252276507,
+            -0.028944280660230762,
+            -0.0002099667627277945,
+            0.00458533068247518,
+            0.0018878351997997833,
+            0.011621407953503755,
+            0.005970807015151215,
+            0.008542958629199003,
+            0.01023896054756221,
+            -0.00205207247588841,
+            0.009402752124876478,
+            0.04169706217414899,
+            -0.022899912321317197,
+            -0.011574970753549647,
+            0.005321872977369877,
+            0.011499271505460519,
+            -0.02647653483798284,
+            -0.011039791758347888,
+            0.0033424345263339144,
+            -0.005219877979550397,
+            0.03609469691834737,
+            -0.02448746405881972,
+            0.019020618652020673,
+            -0.011392297116547224,
+            0.007625475937968714,
+            -0.021465240076078376,
+            -0.003921088264697311,
+            -0.014265745991522477,
+            -0.00011221615798127438,
+            -0.0031478236827989703,
+            -0.0076053733863387755,
+            0.007485039021296698,
+            -0.001907678661303243,
+            0.0020348246022161925,
+            0.006291436821831794,
+            -0.02173305455485444,
+            -0.0019125499969493854,
+            -0.016432460924913166,
+            0.030367263349759286,
+            -0.016242095671439496,
+            0.005072805486919414,
+            0.0053258139728101506,
+            -0.002810639419737258,
+            0.012368771814660942,
+            0.004165967191536186,
+            0.00513754440091394,
+            -0.004199767241827414,
+            -0.005142642360546972,
+            -0.0053886448467479215,
+            0.017868874549760376,
+            -0.013280048598849825,
+            0.035024701466710065,
+            -0.0100959186045929,
+            -0.007094001877261568,
+            0.0019733042325232155,
+            0.010380430710282744,
+            4.2084202709296443e-05,
+            0.01053565670076933,
+            0.0034270489770133685,
+            -0.0006311078385359693,
+            0.0019774779328212756,
+            -0.017162448473652843,
+            0.003089758682621882,
+            0.01682656561693936,
+            0.005463677441664502,
+            0.004469261499451007,
+            -0.013332130891003062,
+            -0.012050233723282738,
+            0.004538268704442613,
+            -0.007533940408255447,
+            0.0003675737140813486,
+            -0.018816953644773076,
+            0.013972696748575243,
+            -0.0011096683686631858,
+            0.0002715683052236706,
+            0.00524678369365015,
+            0.023056514997908218,
+            0.01813167860994747,
+            -0.002553352602551003,
+            0.03155164978556312,
+            0.0174738879054593,
+            0.0029268039949260673,
+            0.010637295711777793,
+            -0.033222781195984666,
+            -0.013301873512972214,
+            -0.0013721703891642354,
+            0.02238421933149517,
+            -0.01530189703335162,
+            -0.014059029663525449,
+            -0.0037393534403749947,
+            0.0010394601933309767,
+            -0.005177359933828505,
+            0.0042489306011644495,
+            0.011039889511148341,
+            -0.014697595240068863,
+            0.007597149702441915,
+            0.0026780496822907634,
+            0.02092734730183164,
+            0.0004688144432243786,
+            0.017181410417231698,
+            -0.0005083970646221274,
+            0.00749639055342427,
+            0.007319420826337891,
+            0.017541463125355265,
+            -0.01002811389296805,
+            0.023582113845081534,
+            -0.008823933397729605,
+            0.0019889258902730535,
+            -0.01257106158422629,
+            -0.028159594067544737,
+            -0.012624750220149823,
+            0.015247564545279736,
+            0.005765911899842925,
+            0.006478299226263732,
+            -0.016537090297144265,
+            0.006276997440048981,
+            0.015647122171767974,
+            -0.002251683151059454,
+            0.010295358869874306,
+            0.009730378299068289,
+            0.00012816405890877098,
+            0.010181120688239766,
+            -0.00801256937783534,
+            -0.00734257767723293,
+            -0.004497520124843868,
+            0.023982754195729983,
+            -0.005443550092842393,
+            0.022434177122972028,
+            0.012647958478747348,
+            0.00616791857573897,
+            -0.010853355357954727,
+            -0.003340111254357573,
+            0.023389634440057473,
+            0.011863246790550376,
+            0.00029642699874398923,
+            -0.009846034060495678,
+            0.0019458876579524617,
+            0.0054010373765486005,
+            0.012983913038716568,
+            -0.02049427270240938,
+            -0.004344571352918878,
+            0.008461947538684664,
+            -0.008472919299762712,
+            0.004318309005722874,
+            -0.030156444522730416,
+            -0.005812303237222271,
+            0.021354352236942434,
+            0.008392281015122372,
+            -0.010232039722494646,
+            -0.004395682831498155,
+            -0.0006527587802367882,
+            0.0040601850269714945,
+            -0.0024323167053212176,
+            -0.0023729138545912176,
+            -0.008541233856441836,
+            -0.006470933419694742,
+            0.014035696617797647,
+            -0.0052691760528871924,
+            0.008597459268477037,
+            0.00012084859681905111,
+            -0.0017255958688825231,
+            -0.007323529232751411,
+            0.03382792341957058,
+            0.008087732655845718,
+            0.005980666974986827,
+            0.0013613625176327891,
+            0.0042244330945601215,
+            0.000616791811672998,
+            -0.0028374856646453707,
+            0.018075360591135542,
+            0.0156738939725117,
+            0.005054696911814988,
+            0.009038525980341447,
+            0.014672270369917403,
+            -0.019700405219230957,
+            0.012625337070556061,
+            0.012184185186281482,
+            0.019382321278525114,
+            0.0007219498737851579,
+            0.002564970959687783,
+            -0.010601873823737926,
+            -0.0029963537053764736,
+            -0.002070567089566093,
+            0.029563028293936092,
+            0.0005903976560069006,
+            -0.010509758568838935,
+            0.010992421945232951,
+            0.020603868018231348,
+            -0.02101157589451638,
+            -0.001971974816402967,
+            0.005963911670920596,
+            -0.010704207628592966,
+            -0.014466113540150018,
+            -0.021269176952661614,
+            -0.013331247081394942,
+            0.0071597345125287174,
+            -0.008162102923896894,
+            0.02367549407145414,
+            -0.014256771777502395,
+            -0.03960796112042958,
+            -0.014803626428839152,
+            -0.00864320347698291,
+            -0.027531337207277483,
+            -0.007305441146142331,
+            0.00965056093889974,
+            0.016698589600806273,
+            -0.003330867073501157,
+            -0.025198703196464594,
+            0.030483790406960656,
+            0.008872821868559751,
+            0.023708422767352917,
+            -0.021464374508407488,
+            -0.0018617153194555344,
+            -0.00886376784514331,
+            0.014505918224188001,
+            -0.017588761818663975,
+            -0.011756112651591795,
+            0.020014632992762026,
+            0.014356419813769714,
+            0.007246557500044434,
+            -0.004355132555323843,
+            0.0067299398462013936,
+            0.007649499235136303,
+            -0.012515181210363267,
+            0.012239792251154642,
+            -0.013499277313916029,
+            -0.024858295126228433,
+            0.014430561986516884,
+            0.002229494935179828,
+            0.009563004428064055,
+            -0.0015984171085659323,
+            -0.016197659608768088,
+            0.010078159490026602,
+            0.00808729175515505,
+            -0.017508840440512225,
+            -0.004793934415922065,
+            0.007133527615305415,
+            -0.020818869260580315,
+            0.031234686259160864,
+            0.011872765152279094,
+            -0.010692949770503467,
+            -0.001989039629661815,
+            0.004431079028880923,
+            0.016511544608975148,
+            0.007893700385963683,
+            -0.01997314388229276,
+            -0.003708006454566117,
+            0.01530128306203401,
+            0.00822588305477249,
+            0.004861703074728471,
+            -0.010836328783594584,
+            0.03434874763691911,
+            0.006082873924472328,
+            4.192481378506604e-05,
+            -0.0052997095110967115,
+            -0.0076108924981838355,
+            0.004196112457289409,
+            0.018905391806961686,
+            0.013330078288599763,
+            -0.004200590783114919,
+            -0.006630107527854746,
+            0.007778448128358027,
+            -0.01706991160178345,
+            0.007329056462125937,
+            0.009390327499494826,
+            -0.012870674959022891,
+            -0.0017571189976160295,
+            0.016462608870101713,
+            0.00209846661139665,
+            0.004942714850185376,
+            -0.011305050739130339,
+            0.004252776146827917,
+            0.015841821944109317,
+            0.018727567181082646,
+            -0.008399642221255733,
+            0.021448055799259867,
+            -0.020972459569228434,
+            0.027243639465535856,
+            0.028499400850980086,
+            0.016573144403275928,
+            -0.01847096443459798,
+            -0.0022409373140116146,
+            -0.0041737366139895414,
+            -0.014761716560267328,
+            0.0034546020161337894,
+            0.005125870074458691,
+            -0.01174186608185822,
+            -0.009577921232380895,
+            -0.024331993794416026,
+            0.011075959943215117,
+            0.007918147902741738,
+            -0.0015045185909419564,
+            -0.011791155919533844,
+            0.016019077577552835,
+            0.010141868950981925,
+            -0.010938402325619172,
+            -0.005635351352447256,
+            -0.019967361901734836,
+            0.009750427567651833,
+            0.009128294017175602,
+            -0.00300556760891084,
+            -0.011496103417643132,
+            0.012977890524465716,
+            0.0014080059843484115,
+            -0.01292380658817521,
+            0.010841954636959441,
+            -0.03142011651894961,
+            -0.005510537354374977,
+            -0.011027026734432255,
+            -0.003620482919145167,
+            -0.014415606143566519,
+            0.003608572766363589,
+            0.0005034611321129689,
+            -0.011470488351158329,
+            -0.0035005774696278,
+            -0.016207453351921627,
+            0.004676327282262429,
+            0.015021513761697039,
+            0.001679614645967629,
+            -0.006917375094634678,
+            0.021994274901326064,
+            0.018518524483306403,
+            -0.021903538417919984,
+            -0.020116013379669034,
+            0.004708084983787958,
+            -0.03261690161800251,
+            0.015019348187107588,
+            0.00966124471577622,
+            0.011676937880751252,
+            -0.028768229226408037,
+            0.007114534979415504,
+            0.00900420165933673,
+            0.016398927481511824,
+            0.0050879301160439245,
+            0.009453255486450082,
+            0.01222758288108498,
+            -0.0104657614414275,
+            -0.003164502778525433,
+            -5.410382753651127e-06,
+            -0.005618225986727351,
+            0.015607620507415484,
+            0.0008527697935056332,
+            0.03603666488542239,
+            -0.005277207843139309,
+            -0.0009799879252482288,
+            -0.011735731235560851,
+            0.00812046801749312,
+            -0.011713349784345638,
+            0.006334172347602515,
+            -0.0024643499727789167,
+            0.0001240009282616789,
+            0.002965067818077541,
+            0.009442572075213767,
+            -0.01086095783353607,
+            -0.007595379345224076,
+            0.007444897256325813,
+            -0.01482814340478338,
+            -0.00019984717819268013,
+            -0.009339343730980451,
+            0.0019680804106275156,
+            -0.02241031895876346,
+            0.006629838013824249,
+            0.040935644507527846,
+            0.02140216465404127,
+            -0.005481040712269146,
+            -0.012431659209559418,
+            -0.004680230194268222,
+            -0.007229300034633229,
+            -0.006482401275991039,
+            -0.004794698160419071,
+            -0.022762405726589375,
+            -0.006928553505112732,
+            0.004320321244630314,
+            0.013300138724212552,
+            0.009741018117644266,
+            -0.008842135469729511,
+            0.004531760574809748,
+            -0.02662650543210291,
+            -0.016578081411548973,
+            -0.0060021425889846495,
+            -0.015840130031005147,
+            0.006125301421471048,
+            0.027521841635711787,
+            -0.0063432924325983525,
+            0.01789356321527744,
+            0.005142029893259461,
+            0.013034833065163448,
+            -0.005338869351330258,
+            0.00035250243782976347,
+            -0.006010938630154189,
+            0.0032958656042048292,
+            0.003074861351445954,
+            -0.0015947217852680714,
+            0.004151590621941346,
+            -0.007139285822575166,
+            -0.0026091937868967284,
+            0.0011629380311555433,
+            0.0117841884590861,
+            0.006877419488576261,
+            -0.00016252653621991807,
+            -0.01761285866765538,
+            0.022221104524808997,
+            -0.018933508559881868,
+            -0.008840161102111973,
+            0.011092684029552064,
+            -0.009660487856185668,
+            0.00702597023411946,
+            0.020387386977378873,
+            -0.006854213562767843,
+            -0.009583981919303215,
+            0.013028181087649146,
+            -0.0036507672754997286,
+            0.004464368543493108,
+            0.009867428978560914,
+            0.006036303973414791,
+            0.002482944923100156,
+            0.013244650094892797,
+            0.004530294330688855,
+            -0.0032625482538717137,
+            -0.020238533752799943,
+            -0.005186498720507101,
+            0.019015752262933585,
+            -0.02252458872825473,
+            -0.0027641215988942563,
+            0.0044506144202026456,
+            -0.0030317840657635457,
+            -0.01733962766050954,
+            0.004205861489436576,
+            -0.018770275720867473,
+            -0.0033883519277721433,
+            -0.008244668812709337,
+            0.009948090119054079,
+            -0.023700732627507225,
+            0.009109315916173436,
+            -0.011739192106106307,
+            0.02356509175177699,
+            -0.003925851077921686,
+            0.024734433836185024,
+            0.0015142645677087268,
+            -0.013482933030628261,
+            0.0055323884713736554,
+            -0.0008930661843169134,
+            -0.004477731568313757,
+            0.029338490404300612,
+            -0.005418258701133154,
+            0.002249996261871466,
+            0.013491124734151694,
+            -0.021332541202417246,
+            0.021773687217263173,
+            0.0046662696076723836,
+            0.01628697326539432,
+            -0.00986238951830334,
+            -0.013799209313714441,
+            0.025178874553620743,
+            0.03324555035648896,
+            0.008398796308070151,
+            0.011261509865513953,
+            -0.01226890254177634,
+            0.018966175575083295,
+            -0.0038328997984309597,
+            0.004344837325292706,
+            0.00878668880458693,
+            -0.007200842513742962,
+            -0.011500562922303248,
+            -0.004533014670236755,
+            0.027639894892383943,
+            -0.021838848944317424,
+            -0.008920629621611768,
+            -0.005739181836252042,
+            -0.01593272996094732,
+            -0.038167534267554606,
+            -0.004371701247332816,
+            0.019604405563120094,
+            -0.005974358141737469,
+            -0.0018341151594086288,
+            0.014334266690716505,
+            0.00402620072221496,
+            -0.002852569009185304,
+            -0.008195218707838913,
+            -0.0027944153624309856,
+            -0.0059390492900516225,
+            0.016651206007022145,
+            -0.004961864528562493,
+            -0.007929830585906699,
+            0.01145431100211762,
+            -0.005731957927601062,
+            0.012130694936714224,
+            0.0004848418056781251,
+            8.392048424584131e-05,
+            -0.0015174120840537315,
+            0.00822932723013239,
+            0.018620869354789082,
+            -0.0055803454666417465,
+            0.025682778991091174,
+            0.006956936033581638,
+            -0.003495709501259274,
+            -0.005489431577052735,
+            -0.0044154293119170445,
+            -0.024561016942073285,
+            -0.0020864636079347357,
+            0.006593060072668424,
+            0.024333813203596122,
+            -0.02515989187381735,
+            -0.004953266000463339,
+            0.020957806164031402,
+            -0.0012192724005613,
+            -0.011978722479596745,
+            0.014690559986694138,
+            0.0041346025614047875,
+            0.009881584250499358,
+            0.012176393203318002,
+            0.015764530703333238,
+            -0.004923548157246672,
+            -0.0015192775000207833,
+            0.012766482928380375,
+            0.0005745973040874939,
+            0.00997809510696241,
+            0.016495444272181515,
+            0.010467514633877887,
+            -0.021108281220544853,
+            0.0015078843789125156,
+            0.00522593579521476,
+            -0.017481430635971396,
+            0.0034362821987740755,
+            0.005397819166037461,
+            0.011626527721895694,
+            0.004048632382026344,
+            0.013207335578216437,
+            -0.015768003434275563,
+            -0.005916168437120099,
+            0.008153798523776654,
+            0.004319451296835019,
+            -0.02599474886669325,
+            -0.01485009085106089,
+            -0.0006781964116146299,
+            -0.004444144921332929,
+            0.013556265879295017,
+            0.01285768162554067,
+            0.02033626001285354,
+            0.024637349990579514,
+            -0.013833060730799217,
+            -0.00842819657792762,
+            0.009436072455719896,
+            0.015560357465711765,
+            0.005798181411384387,
+            0.007874051721372273,
+            0.009297440071118126,
+            0.0077320823375042,
+            -0.004073209576661068,
+            -0.002973928843567532,
+            0.020205003097225747,
+            -0.019725382393637726,
+            0.0006746659838642147,
+            -0.018417443693472958,
+            0.007530796896254817,
+            0.01140696723506382,
+            0.01129515042745398,
+            0.002816883643872417,
+            -0.005274757308627299,
+            0.005716917047812153,
+            -0.06254973656832136,
+            -0.009191453047502128,
+            0.003384751551032943,
+            0.027256466708746898,
+            -0.010885336140129142,
+            -0.003415707910562931,
+            0.007309577090125959,
+            0.005511970708769456,
+            0.0005572239548625122,
+            0.003186427278034852,
+            0.019698716902282903,
+            -0.02052581021967601,
+            -0.010793968505805883,
+            0.004219269922138548,
+            0.0033111837923746746,
+            -0.007269521534951044,
+            0.004805737092302134,
+            0.014143978208501908,
+            0.01986656063456352,
+            -0.0024048854610842748,
+            0.01629335842915328,
+            -0.0015498603704589015,
+            -0.014496994444493684,
+            -0.002671064325909516,
+            0.0037875833047555925,
+            -0.022466433279066646,
+            -0.005980897597195858,
+            0.0043262697442044435,
+            -0.011226121648007667,
+            0.006978495397818936,
+            0.008452177557316863,
+            -0.003018545465557218,
+            -0.0008303307644324117,
+            0.017796789945838845,
+            0.006591364879905198,
+            -0.013464486184025026,
+            0.0028668426952326957,
+            -0.023377605222213174,
+            0.004325185794796361,
+            0.001416245546006914,
+            0.007809449561523174,
+            -0.008431335139468173,
+            -0.014774499666285603,
+            -0.005933810712528226,
+            0.017746847020610673,
+            -0.01965188730040385,
+            -0.002406875192894708,
+            0.01683009662984443,
+            -0.0002518220110313279,
+            0.005456812494894182,
+            -0.02011241622618092,
+            -0.0009782974741561381,
+            -0.02933296756078644,
+            -0.01171823509606186,
+            0.001162345317872531,
+            0.00892221099991475,
+            -0.006551308022163148,
+            -0.03934356275100932,
+            0.00427623835776202,
+            -0.013111664387776261,
+            -0.008752857644809231,
+            -0.010214546496134218,
+            -0.00518087722852466,
+            0.015240297918667867,
+            -0.012498100033235827,
+            0.01086490299272505,
+            -0.00917777212602538,
+            0.005895055740265625,
+            0.0090554651297836,
+            0.014212841798768113,
+            -0.005544047915369086,
+            -0.0033589999440643385,
+            -0.007924830667307956,
+            0.0022291797354234005,
+            -0.0032976226056924502,
+            0.009615674261248298,
+            -0.0051932155872883,
+            -0.0065636389655770455,
+            0.02426106498596339,
+            0.011445591067574932,
+            0.013312596099532533,
+            0.0055118624272110965,
+            -0.007220620072349783,
+            -0.01916551187613823,
+            -0.003977616240384473,
+            -0.013204506708246246,
+            -0.006315966975649096,
+            0.01434099854765068,
+            0.012588249778961223,
+            0.006357972880218866,
+            0.0007747002406996095,
+            -0.004794439624135275,
+            0.006975593520375694,
+            0.0262335496080837,
+            -0.016481909971520166,
+            -0.0013084662210086592,
+            -0.01369313900978409,
+            0.01921748474212319,
+            0.03333761672336437,
+            -0.0008918419419301244,
+            -0.007699525186523558,
+            0.013757216065348164,
+            0.00819045303047094,
+            -0.007411380297164911,
+            0.02453611080891859,
+            -0.025045462880463647,
+            -0.015989319900023054,
+            -0.011650886878708265,
+            -0.011407253985176972,
+            -0.008386367959227559,
+            0.016331961716602287,
+            -0.00141225438918211,
+            -0.0018505754298268654,
+            0.009557291617446645,
+            0.03357924917013024,
+            0.006916860901290756,
+            0.007030308953738567,
+            -0.002739345982346728,
+            -0.009376522438071562,
+            0.011482637968411055,
+            -0.007259054391879614,
+            -0.04075661435928125,
+            0.004250195164583401,
+            -0.005160505473755424,
+            0.02405623169060588,
+            -0.02197586561375289,
+            -0.00619175761282745,
+            -0.006736129049831973,
+            -0.028200015908139763,
+            -0.005867722104196303,
+            0.02805071895018145,
+            -0.015402198567775478,
+            0.014773294766214671,
+            -0.009502185346558072,
+            -0.0013419858580438616,
+            0.004443749010136947,
+            0.013008783570355656,
+            -0.02289662952366337,
+            0.0013834888740540108,
+            -0.00048549197205067054,
+            0.027969842651517314,
+            0.010437729090975158,
+            -0.0047705656249623744,
+            0.014202218020412936,
+            0.008240861154198411,
+            -0.018637315941614406,
+            0.007314019317343154,
+            -0.005747966765101792,
+            0.014833896739762147,
+            -0.01314847940891154,
+            0.007762100494499689,
+            0.00705026393499352,
+            -0.01278023669088385,
+            0.03016342404000438,
+            -0.015204129985039077,
+            0.007776946006562982,
+            0.009278778571285689,
+            0.009656723862179986,
+            0.01257686993982689,
+            0.004362439504889069,
+            -0.02898418193036562,
+            0.0002144431781454467,
+            -0.03828100004144633,
+            -0.00833748467467171,
+            -0.002611937107022567,
+            -0.002145329666219461,
+            0.007050630951707937,
+            -0.009383645162261629,
+            -0.033976442057557676,
+            0.0006429978367640709,
+            0.01603027598131823,
+            0.017501067413108286,
+            0.0002196997337451293,
+            0.013075379412332973,
+            -0.006197537539554383,
+            -0.021148120787551232,
+            -0.016986448660412497,
+            -0.010207705210051413,
+            -0.02122194744719993,
+            -0.020550470201191853,
+            -0.00908564461857315,
+            0.012090079127547193,
+            -0.009927390005879807,
+            0.013888458967497627,
+            0.029513250508042935,
+            -0.024529175134769883,
+            0.0011144447737101862,
+            0.021977677894839922,
+            -0.00198308986575084,
+            0.014108189311291326,
+            0.001696775006906642,
+            0.014122064251288995,
+            -0.0004166140484018266,
+            -0.010556016685519523,
+            -0.009729362492661684,
+            0.017487929296614076,
+            -0.0026043257634797124,
+            0.016715798948897194,
+            0.012183487338360088,
+            0.003771721868999568,
+            -0.0030887189945795796,
+            0.010667776697461199,
+            -0.026471157050402344,
+            0.014249996582064706,
+            -0.006690961709923224,
+            -0.0181382691440242,
+            -0.008729104191096461,
+            0.003882731567105777,
+            -0.005609384377642956,
+            -0.007794812949125115,
+            0.012967089730551055,
+            -0.00944374111323783,
+            0.02088525981049842,
+            0.0023400121055514958,
+            -0.012196925240774426,
+            0.003648688547842322,
+            -0.03322041209304968,
+            0.0015191765655963997,
+            -0.018402382776013956,
+            -0.024659631472836927,
+            0.03840173429666435,
+            0.011991452465253198,
+            0.006639702644606365,
+            -0.01675838115087676,
+            0.007953102302963526,
+            0.0004257479565099406,
+            0.017484338819046517,
+            0.011068864013925452,
+            0.006466915839538296,
+            0.008364454017976605,
+            -0.023316137291380285,
+            -0.010898421318668592,
+            0.016204569093730564,
+            0.012099473359791148,
+            -0.01017229369980751,
+            0.019431669546223253,
+            0.017955119269367787,
+            0.007300372071724306,
+            -0.006898619924677706,
+            0.01647927363640854,
+            0.021012103522831126,
+            0.010291016618557653,
+            -0.00047955552103474145,
+            0.0008684297441778696,
+            -0.010534247928997246,
+            0.005037930235889528,
+            0.035417626740307205,
+            0.014712595692093886,
+            0.010650306363963653,
+            -0.03626584475133921,
+            0.021764456453493054,
+            -0.011961521673661574,
+            0.009092981192874177,
+            0.006089104887183293,
+            0.017982801101846038,
+            0.006916969397519662,
+            0.014196254958707752,
+            0.011745069030657836,
+            -0.006689305450954901,
+            0.025205280281804708,
+            -0.001601414806100461,
+            -0.011992525142592405,
+            0.003386930324498371,
+            -0.01859193251438437,
+            0.005303261319773131,
+            -0.0041710536965100615,
+            -0.004459393120257307,
+            0.0032461828144552565,
+            -0.0018650956576455287,
+            0.015678225251799887,
+            0.003894142628084471,
+            -0.02355212701026675,
+            -0.0014873755115206368,
+            -0.024284295126348117,
+            0.0027260592623277375,
+            -0.022185571998195344,
+            0.00022883605560765203,
+            -0.005557568353382136,
+            -0.009749382813201403,
+            -0.025385857431333838,
+            -0.009853073898348951,
+            -0.0035843255959309307,
+            0.007661137888845632,
+            -0.010521037433807653,
+            -0.004246698084422667,
+            -0.0019920499321964815,
+            -0.019465113363548715,
+            0.009014255846983305,
+            0.027122533756959888,
+            0.016848221859975445,
+            0.027170883580103264,
+            -0.00019554112591107156,
+            0.013614675849485926,
+            -0.002066822813103284,
+            -0.01603063005944364,
+            0.008824663429777616,
+            0.0075653522381218495,
+            -0.00531780740880296,
+            0.014643348459785112,
+            0.0072634484937344704,
+            -0.012800149015319017,
+            0.009859503660384916,
+            -0.007161712176469609,
+            -0.006437857349742842,
+            -0.009764856421740622,
+            -0.02058868349990357,
+            -0.014866137971603676,
+            0.0012450207572602557,
+            -0.002123265406747006,
+            0.006000473862248991,
+            -0.004054016790850226,
+            0.009584879012937315,
+            -0.00608816807208866,
+            -0.00925801057439376,
+            -0.013962683072792106,
+            -0.008924356118158748,
+            -0.004210064145949253,
+            -0.004381709838058668,
+            -0.011732699976353612,
+            -0.005479523097469261,
+            0.0010680258077037758,
+            -0.0019401640391035755,
+            -0.01397652201595301,
+            0.011650839243603377,
+            0.0058731838291845645,
+            -0.002462743134993977,
+            0.02219008688925881,
+            0.01097316136828334,
+            0.0018083001685932406,
+            0.01366095798812196,
+            0.0003942601242216818,
+            -0.0064063532958756175,
+            -0.03343312148366622,
+            -0.0005083355836126291,
+            -0.02815586367527057,
+            0.025770078162165123,
+            -0.0069114949102666145,
+            -0.010322087105114614,
+            -0.002519047562374146,
+            -0.006063669644021698,
+            0.005873609611264377,
+            -0.002722940652283699,
+            0.0021229525800901114,
+            -0.023673778230768945,
+            0.0028629715379660113,
+            -0.003953653437464966,
+            -0.0036366957271968823,
+            0.007469664864329332,
+            0.0346520690389124,
+            -0.004054282838167587,
+            -0.0034765253405897676,
+            0.005159466707114532,
+            0.011642913651743502,
+            0.027845571220425726,
+            -0.004445196354520585,
+            -0.0032038508043339413,
+            -0.02233838584152591,
+            0.0054112576531374175,
+            -0.00019544838850471866,
+            -0.004733192299951088,
+            0.016664217348411003,
+            -0.016719175418727594,
+            -0.01631658832047864,
+            -0.00545969677717507,
+            -0.014570848053102338,
+            -0.00022620258237138677,
+            0.014454101124666536,
+            0.011056312437637014,
+            -0.004035526363810078,
+            -0.012459378363360557,
+            -0.0022595493558020706,
+            -0.030370952502579162,
+            0.012766010284884204,
+            0.1141689619126009,
+            -0.00766197047210299,
+            -0.00896655658602649,
+            -0.009481112393590025,
+            0.001435269335646347,
+            0.001338755472877713,
+            -0.011739143028767118,
+            0.07166096084053437,
+            -0.0077855767070652395,
+            0.0002139580430342728,
+            -0.011312198831181402,
+            0.001844005266872301,
+            -0.001825412693163496,
+            0.01371932421024605,
+            -0.005025814503687022,
+            -0.018213360991353643,
+            -0.0033612481616769416,
+            -0.02941403830641238,
+            -0.0041137865872933515,
+            -0.005683630256743215,
+            -0.004288198570156578,
+            -0.014982999363445122,
+            0.0067149665791261335,
+            -0.019661973497372313,
+            -0.012688556129040908,
+            0.001124391650175233,
+            -0.0033623453063430534,
+            0.00996944383866679,
+            -0.020496336814603385,
+            -0.0044354321069609536,
+            0.0019875840782527733,
+            0.009706675240563883,
+            0.0011600364428219214,
+            -0.012755178230821602,
+            -0.0026514442584739412,
+            0.004056350166031111,
+            0.01909951327913368,
+            0.014360281326789544,
+            0.003732344102045803,
+            0.008623457695718206,
+            0.014461868085333023,
+            -0.001692149528228451,
+            0.01948035376572433,
+            0.01172550239477637,
+            0.0031966531144076173,
+            0.0004779413449255626,
+            0.0030912562273611517,
+            -0.004597633235071031,
+            0.01582547400214653,
+            -0.014811177637844808,
+            -0.02411202772171917,
+            -0.0071272238837422185,
+            0.018015541137524113,
+            -0.013081287852028074,
+            1.8837689271256874e-05,
+            0.004062803284646432,
+            0.027569726868200254,
+            0.0010564850664255057,
+            -0.01179755720247972,
+            0.0028876134800690334,
+            -0.01638215925131053,
+            0.022944183106912663,
+            0.0026043713064243836,
+            0.03279173899194346,
+            -0.005103903273968361,
+            0.00578213669463383,
+            -0.020744194169889544,
+            -0.007588793365216579,
+            -0.00819862611489168,
+            0.0017401442433921013,
+            0.010993500174794084,
+            0.005411238350013134,
+            0.01620925462604661,
+            -0.00641615069699833,
+            -0.020751730783437995,
+            0.014776878596707538,
+            -0.009552193304486048,
+            -0.03393319554458277,
+            0.008881987295131347,
+            -0.029102297123590998,
+            -0.0045267103602711,
+            0.00368971742585232,
+            -0.014269341759285646,
+            -0.016264809810970175,
+            0.012837302095591106,
+            -0.0011890334180728564,
+            -0.008937992768323412,
+            0.0014444613725248846,
+            -0.017110187133760626,
+            -0.02385668037818897,
+            0.006017405364919834,
+            0.00676670672597817,
+            -0.0077994500447595,
+            0.018312452858615755,
+            -0.027015660037024585,
+            0.006123912473734307,
+            -0.012973676426738582,
+            0.009041974762537394,
+            -0.01580183887345168,
+            0.01163516332546347,
+            0.010887662470499361,
+            -0.001503467664709027,
+            -0.0076234124970223326,
+            0.01231202541338883,
+            0.004852720134894811,
+            0.033760365210093034,
+            0.028658326381800395,
+            0.000995999783627898,
+            -0.0025605367058226737,
+            -0.0020356794893366384,
+            0.011667525096213252,
+            -0.014049700287539817,
+            0.011218246246280464,
+            0.02658113044439717,
+            0.017568631325838108,
+            -0.019133062654008885,
+            0.018050678772708717,
+            -0.010356118256625172,
+            0.012872052420999813,
+            -0.025620154415048252,
+            -0.005135768485044442,
+            0.004936490920616527,
+            0.005163008729256001,
+            0.007425278892536464,
+            0.001410212306439329,
+            0.021891387289347516,
+            0.014999616600766989,
+            0.00027613811156650413,
+            0.00558422822386443,
+            0.01477009526981092,
+            0.010497782599293994,
+            -0.0005567894749233204,
+            -0.00467776506604011,
+            -0.016998035358377343,
+            0.00957271797745408,
+            -0.006065116857898469,
+            0.005138070401846389,
+            0.01698914855850691,
+            -0.01230257809343985,
+            0.011109844802663414,
+            0.007518553660463051,
+            -0.015504773927595823,
+            -0.017169933945839305,
+            0.005552829003043926,
+            -0.02252633790984703,
+            -0.01306722099657118,
+            0.02715689072730398,
+            -0.010513941399990838,
+            -0.02790582732974509,
+            -0.0009847768069205477,
+            0.009797865241108237,
+            -0.002772952073769185,
+            -0.0027968741526000576,
+            0.0037541733651479613,
+            0.010801137722284016,
+            -0.014658156970099418,
+            0.002075415233345957,
+            0.004735264309137368,
+            0.01930804315461151,
+            -0.0014151522239496115,
+            -0.015046118414100617,
+            -0.002288495894227231,
+            0.021592829677937263,
+            0.017174609713578173,
+            0.00265560378681856,
+            -0.016499825201963983,
+            -3.965296928410772e-05,
+            -0.0074142747014567635,
+            0.00021360696629503726,
+            0.00027431174904249525,
+            -0.0056391089282546054,
+            -0.018931681700770225,
+            -0.003888248121485032,
+            0.011181675419027736,
+            -0.00247252920175839,
+            0.01977860312223846,
+            -0.0011527575128223392,
+            -0.015763587433200915,
+            0.001960734212846797,
+            -0.005280860341480037,
+            -0.010372829570478649,
+            0.01351900743232294,
+            -0.009359942374603997,
+            0.006788746813046239,
+            -0.003212015225284119,
+            0.004611143085183367,
+            -0.0003308099698088422,
+            -0.017017265064185255,
+            0.02924142709831441,
+            -0.0018238549856433466,
+            -0.004769274041546295,
+            -0.0014058363976586562,
+            -0.033393331166741945,
+            0.015017165991303838,
+            -0.014574579250749552,
+            0.006745699409061803,
+            0.002705853802154516,
+            0.01272741296341735,
+            0.003253095246601883,
+            -0.008898616461779179,
+            0.0027485606928249326,
+            0.017040360927723885,
+            0.013756402798365262,
+            -0.007292864293090161,
+            0.008694855886257706,
+            -0.0028934389922887676,
+            -0.0037483208383542015,
+            0.01363963339139356,
+            0.004170611039133969,
+            -0.006282657671540625,
+            -0.0009784372731140945
+          ],
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "Mixture-of-Experts (MoE)",
+            "fine-grained expert segmentation",
+            "shared expert isolation"
+          ],
+          "title": "Deepseekmoe: Towards Ultimate Expert Specialization In Mixture-Of-Experts Language Models"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "65742957-6bda-446d-a61a-0fe20313c760",
+        "properties": {
+          "page_content": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity\n\nWilliam Fedus∗ liamfedus@google.com Barret Zoph∗ barretzoph@google.com Noam Shazeer noam@google.com Google, Mountain View, CA 94043, USA Editor: Alexander Clark\n\nAbstract\n\nIn deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) models defy this and instead select different parameters for each incoming example. The result is a sparsely-activated model—with an outrageous number of parameters—but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs, and training instability. We address these with the introduction of the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques mitigate the instabilities, and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large (Raffel et al., 2019) to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the \"Colossal Clean Crawled Corpus\", and achieve a 4x speedup over the T5-XXL model.12 Keywords: mixture-of-experts, natural language processing, sparsity, large-scale machine learning, distributed computing arXiv:2101.03961v3 [cs.LG] 16 Jun 2022 Contents\n\n1 Introduction 3 2 Switch Transformer 4 2.1 Simplifying Sparse Routing 5 2.2 Efficient Sparse Routing 6 2.3 Putting It All Together: The Switch Transformer 8 2.4 Improved Training and Fine-Tuning Techniques 8 3 Scaling Properties 11 3.1 Scaling Results on a Step-Basis 12 3.2 Scaling Results on a Time-Basis 13 3.3 Scaling Versus a Larger Dense Model 13 4 Downstream Results 14 4.1 Fine-Tuning 14 4.2 Distillation 16 4.3 Multilingual Learning 17 5 Designing Models with Data, Model, and Expert-Parallelism 18 5.1 Data Parallelism 20 5.2 Model Parallelism 20 5.3 Model and Data Parallelism 21 5.4 Expert and Data Parallelism 22 5.5 Expert, Model and Data Parallelism 22 5.6 Towards Trillion Parameter Models 22 6 Related Work 24 7 Discussion 25 8 Future Work 26 9 Conclusion 27 A Switch for Attention 27 B Preventing Token Dropping with No-Token-Left-Behind 29\n\nC Encouraging Exploration Across Experts 29 D Switch Transformers in Lower Compute Regimes 29 E Relation of Upstream to Downstream Model Performance 32 F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/switch_transformers.md"
+          },
+          "summary": "The Switch Transformer is a sparsely-activated model that simplifies the Mixture of Experts (MoE) approach, allowing for efficient scaling to trillion-parameter models with reduced communication and computational costs. It achieves significant speedups in pre-training and fine-tuning by activating only a subset of neural network weights for each input, maintaining a constant computational cost. The model demonstrates superior scaling properties and improved performance across various natural language processing tasks, including multilingual settings. It also allows for effective distillation into smaller dense models, preserving a portion of the quality gains. The architecture is designed to maximize parameter count while keeping floating point operations per example constant, making it suitable for both large-scale and smaller-scale applications.",
+          "headlines": {
+            "1 Introduction": [],
+            "2 Switch Transformer": [
+              "2.1 Simplifying Sparse Routing",
+              "2.2 Efficient Sparse Routing",
+              "2.3 Putting It All Together: The Switch Transformer",
+              "2.4 Improved Training and Fine-Tuning Techniques"
+            ],
+            "3 Scaling Properties": [
+              "3.1 Scaling Results on a Step-Basis",
+              "3.2 Scaling Results on a Time-Basis",
+              "3.3 Scaling Versus a Larger Dense Model"
+            ],
+            "4 Downstream Results": [
+              "4.1 Fine-Tuning",
+              "4.2 Distillation",
+              "4.3 Multilingual Learning"
+            ],
+            "5 Designing Models with Data, Model, and Expert-Parallelism": [
+              "5.1 Data Parallelism",
+              "5.2 Model Parallelism",
+              "5.3 Model and Data Parallelism",
+              "5.4 Expert and Data Parallelism",
+              "5.5 Expert, Model and Data Parallelism",
+              "5.6 Towards Trillion Parameter Models"
+            ],
+            "6 Related Work": [],
+            "7 Discussion": [],
+            "8 Future Work": [],
+            "9 Conclusion": [],
+            "A Switch for Attention": [],
+            "B Preventing Token Dropping with No-Token-Left-Behind": [],
+            "C Encouraging Exploration Across Experts": [],
+            "D Switch Transformers in Lower Compute Regimes": [],
+            "E Relation of Upstream to Downstream Model Performance": [],
+            "F Pseudo Code for Switch Transformers": []
+          },
+          "summary_embedding": [
+            -0.0060447100549936295,
+            -0.00390331051312387,
+            -0.01784370467066765,
+            -0.0021820589900016785,
+            0.013227885589003563,
+            0.011338187381625175,
+            0.0165580902248621,
+            0.024240797385573387,
+            -0.005986624862998724,
+            0.03596621751785278,
+            -0.004852031823247671,
+            -0.0068811336532235146,
+            -0.003574162255972624,
+            0.014242436736822128,
+            0.011578272096812725,
+            -0.033302053809165955,
+            -0.024519605562090874,
+            0.01050176378339529,
+            0.016093410551548004,
+            -0.006795942317694426,
+            0.007473600562661886,
+            -0.030513975769281387,
+            -0.06124480068683624,
+            -0.006838538218289614,
+            -0.0026099516544491053,
+            0.01029265858232975,
+            0.0010329445358365774,
+            -0.010873507708311081,
+            -0.053469158709049225,
+            0.023311438038945198,
+            0.025294071063399315,
+            0.005092116538435221,
+            -0.029584616422653198,
+            0.0009361362317577004,
+            -0.05161044001579285,
+            -0.04448534920811653,
+            0.0379488542675972,
+            0.012561844661831856,
+            -0.04364892467856407,
+            0.039249956607818604,
+            -0.004875265993177891,
+            -0.0022440163884311914,
+            -0.021499186754226685,
+            -0.0034909071400761604,
+            -0.007039899472147226,
+            -0.009851212613284588,
+            -0.02874819189310074,
+            0.0014037203509360552,
+            0.021886419504880905,
+            -0.0056962003000080585,
+            0.023264970630407333,
+            -0.024271776899695396,
+            0.01936165988445282,
+            0.04240977764129639,
+            -0.06957805901765823,
+            0.024659009650349617,
+            0.017549408599734306,
+            0.05120771750807762,
+            0.06511713564395905,
+            0.018339363858103752,
+            0.021669568493962288,
+            -0.014141756109893322,
+            -0.017766257748007774,
+            -0.011872569099068642,
+            0.004673904739320278,
+            -0.014970434829592705,
+            -0.01891246810555458,
+            0.008449427783489227,
+            -0.020461400970816612,
+            0.008209343999624252,
+            -0.05576157942414284,
+            -0.014730350114405155,
+            -0.034200433641672134,
+            -0.01140014547854662,
+            0.017642343416810036,
+            -0.018215449526906013,
+            0.030126741155982018,
+            0.02997184917330742,
+            0.002638994250446558,
+            0.0007822110201232135,
+            0.021049996837973595,
+            0.006509389728307724,
+            -0.0053438181057572365,
+            0.012321759946644306,
+            0.028159597888588905,
+            -0.01450575515627861,
+            -0.010106786154210567,
+            -0.03454120084643364,
+            0.01861817203462124,
+            0.01589204929769039,
+            -0.04194509983062744,
+            -0.012569589540362358,
+            -0.013824224472045898,
+            0.032341714948415756,
+            -0.009549170732498169,
+            0.04098476096987724,
+            -0.010083552449941635,
+            0.001383390510454774,
+            -0.031149037182331085,
+            -0.009463978931307793,
+            0.017069239169359207,
+            0.006172496825456619,
+            -0.016589069738984108,
+            -0.0012846461031585932,
+            0.016170857474207878,
+            0.006617815233767033,
+            -0.030142230913043022,
+            0.04488807171583176,
+            0.023497309535741806,
+            0.055296897888183594,
+            0.05579255893826485,
+            -0.037391237914562225,
+            -0.002445377642288804,
+            0.0344172865152359,
+            -0.0228312686085701,
+            -0.015040136873722076,
+            -0.034014564007520676,
+            -0.01768881268799305,
+            0.035935238003730774,
+            -0.0318460576236248,
+            0.01556677371263504,
+            0.028066661208868027,
+            -0.025712283328175545,
+            0.003488970920443535,
+            -0.03943582624197006,
+            0.003622566582635045,
+            0.01220559049397707,
+            0.006095050368458033,
+            -0.0026293133851140738,
+            -0.013615119270980358,
+            0.008906363509595394,
+            0.003055269829928875,
+            0.00800798274576664,
+            -0.04615819454193115,
+            0.018478767946362495,
+            -0.00454224506393075,
+            0.04033420979976654,
+            0.042502716183662415,
+            -0.02730768360197544,
+            -0.016743963584303856,
+            0.008557853288948536,
+            0.012104909867048264,
+            0.04188314080238342,
+            -0.007287728600203991,
+            -0.00121784838847816,
+            -0.0007836631848476827,
+            0.007322579622268677,
+            0.047552235424518585,
+            0.004646798130124807,
+            0.0020697612781077623,
+            -0.027230238541960716,
+            0.014978179708123207,
+            -0.019733402878046036,
+            0.027648448944091797,
+            0.023373395204544067,
+            0.02942972257733345,
+            -0.04048910364508629,
+            0.017518429085612297,
+            -0.011818356812000275,
+            -0.005568413063883781,
+            -0.002726121572777629,
+            -0.02856232039630413,
+            -0.0074426219798624516,
+            0.033983584493398666,
+            0.015365412458777428,
+            0.012956822291016579,
+            0.03271345794200897,
+            0.0012052633101120591,
+            -0.042502716183662415,
+            0.03804178908467293,
+            0.024550583213567734,
+            0.04640602692961693,
+            0.023466331884264946,
+            0.010796061716973782,
+            0.006571347359567881,
+            0.013375034555792809,
+            -0.019377149641513824,
+            0.015907539054751396,
+            0.02560385875403881,
+            0.009750531986355782,
+            -0.03167567402124405,
+            -0.0009501734166406095,
+            0.01178737822920084,
+            -0.004546117503196001,
+            0.03701949119567871,
+            0.04104671627283096,
+            0.020569827407598495,
+            0.05030933395028114,
+            0.0020155487582087517,
+            0.017936641350388527,
+            0.005936284549534321,
+            0.02820606529712677,
+            0.030467506498098373,
+            -0.005285732913762331,
+            0.005076627247035503,
+            -0.01136916596442461,
+            -0.03385967016220093,
+            0.014110777527093887,
+            0.01561324205249548,
+            -0.008077683858573437,
+            -0.01697630248963833,
+            0.006033093202859163,
+            0.009169681929051876,
+            -0.01795213110744953,
+            -0.017456471920013428,
+            0.005971135571599007,
+            0.018866000697016716,
+            -0.013576395809650421,
+            0.024689987301826477,
+            -0.04442339017987251,
+            0.029119934886693954,
+            -0.033209118992090225,
+            -0.003384418087080121,
+            -0.010641167871654034,
+            -0.0030843124259263277,
+            0.0041666291654109955,
+            0.011314953677356243,
+            0.013406013138592243,
+            0.002042655134573579,
+            -0.013909416273236275,
+            0.024519605562090874,
+            0.018447788432240486,
+            0.012073930352926254,
+            -0.03148980438709259,
+            -0.014397329650819302,
+            -0.0015673263696953654,
+            -0.008743725717067719,
+            -0.034138478338718414,
+            0.026966920122504234,
+            0.014048820361495018,
+            -0.03844451159238815,
+            0.017797237262129784,
+            0.029538147151470184,
+            0.014420564286410809,
+            -0.004790074657648802,
+            -0.0463130883872509,
+            0.04045812413096428,
+            -0.030436528846621513,
+            -0.033333033323287964,
+            -0.010478530079126358,
+            -0.0003146269591525197,
+            0.02894955314695835,
+            -0.0064358157105743885,
+            -0.050835974514484406,
+            0.01184933539479971,
+            0.020631784573197365,
+            0.00010763872705865651,
+            0.005731051322072744,
+            -0.019593998789787292,
+            0.00028631053282879293,
+            -0.03395260497927666,
+            0.0090844901278615,
+            0.026750069111585617,
+            0.008867640048265457,
+            -0.016418687999248505,
+            0.03351890295743942,
+            -0.0004121613164898008,
+            -0.018695618957281113,
+            -0.003889757441356778,
+            0.006885006092488766,
+            0.034912943840026855,
+            0.004836542531847954,
+            -0.02946070022881031,
+            0.05665995925664902,
+            -0.02100352756679058,
+            0.021344292908906937,
+            0.05631919577717781,
+            -0.019439106807112694,
+            -0.008472662419080734,
+            -0.002065889071673155,
+            0.0026196325197815895,
+            -0.0260530486702919,
+            -0.008937342092394829,
+            0.032341714948415756,
+            0.041728246957063675,
+            -0.018742086365818977,
+            -0.001398879918269813,
+            0.006780453026294708,
+            0.009425255469977856,
+            0.004948840010911226,
+            0.02692045085132122,
+            -0.012128143571317196,
+            -0.014559967443346977,
+            0.05198218300938606,
+            0.028314489871263504,
+            -0.012081675231456757,
+            -0.0164031982421875,
+            0.019888296723365784,
+            -0.024844881147146225,
+            -0.020244551822543144,
+            -0.02431824430823326,
+            -0.020414933562278748,
+            -0.0042518205009400845,
+            0.009525936096906662,
+            0.037484172731637955,
+            -0.03968365862965584,
+            0.012747716158628464,
+            0.021700548008084297,
+            0.02981695532798767,
+            -0.019532041624188423,
+            -0.05910727381706238,
+            -0.0016128262504935265,
+            0.03296129032969475,
+            -0.015458349138498306,
+            -0.049379974603652954,
+            0.015164051204919815,
+            0.0260530486702919,
+            0.05957195535302162,
+            -0.001461805310100317,
+            -0.014954946003854275,
+            0.061926331371068954,
+            0.03026614524424076,
+            -0.004383479710668325,
+            -0.009494957514107227,
+            -0.0127864396199584,
+            0.003396035172045231,
+            0.00465454300865531,
+            -0.03364282101392746,
+            -0.041170634329319,
+            -0.011841590516269207,
+            -0.010284913703799248,
+            0.008108663372695446,
+            -0.040922801941633224,
+            0.03531566634774208,
+            -0.021917399019002914,
+            0.010950954630970955,
+            0.015473837964236736,
+            0.037391237914562225,
+            -0.021344292908906937,
+            -0.01284839678555727,
+            -0.01572941243648529,
+            -0.024968795478343964,
+            -0.014010096900165081,
+            0.021917399019002914,
+            0.007682706229388714,
+            3.4699722164077684e-05,
+            0.010075807571411133,
+            -0.024659009650349617,
+            -0.051052823662757874,
+            0.0291044469922781,
+            0.0015605497173964977,
+            -0.015938518568873405,
+            -0.004623564425855875,
+            -0.01977987214922905,
+            -0.01122201792895794,
+            -0.012608313001692295,
+            -0.039249956607818604,
+            0.021111954003572464,
+            -0.01135367713868618,
+            0.015876561403274536,
+            0.0009250032599084079,
+            0.003659353591501713,
+            0.028624277561903,
+            0.01116006076335907,
+            0.0051811798475682735,
+            -0.03255856782197952,
+            -0.018277406692504883,
+            0.010122275911271572,
+            -0.00561100896447897,
+            -0.018339363858103752,
+            -0.013351799920201302,
+            -0.05037129297852516,
+            -0.03305422514677048,
+            -0.008054450154304504,
+            -0.05415068939328194,
+            0.0020697612781077623,
+            -0.014296649023890495,
+            -0.02232012152671814,
+            -0.005990497302263975,
+            0.03107159025967121,
+            -0.003235333366319537,
+            0.025294071063399315,
+            0.0076594725251197815,
+            0.037360258400440216,
+            0.040117356926202774,
+            0.008426194079220295,
+            0.03065337985754013,
+            -0.0029991210903972387,
+            -0.016867877915501595,
+            0.05486319959163666,
+            -0.01881953328847885,
+            0.013529927469789982,
+            -0.008604321628808975,
+            0.008604321628808975,
+            0.03299226611852646,
+            -0.003794885240495205,
+            0.02007416822016239,
+            -0.0311955064535141,
+            -0.07998688519001007,
+            0.01637221872806549,
+            0.028159597888588905,
+            -0.06418777257204056,
+            0.014691627584397793,
+            -0.020275529474020004,
+            0.02052335813641548,
+            0.02411688305437565,
+            -0.0009114501299336553,
+            -0.014064309187233448,
+            0.005595519673079252,
+            0.0020833145827054977,
+            -0.01456771232187748,
+            -0.002691270550712943,
+            -0.009417511522769928,
+            0.008774704299867153,
+            -0.017007282003760338,
+            -0.013406013138592243,
+            -0.034231413155794144,
+            0.010873507708311081,
+            0.01233724970370531,
+            -0.003969140350818634,
+            0.0011500825639814138,
+            -0.03277541697025299,
+            -0.007876322604715824,
+            -0.02145271934568882,
+            0.03382869064807892,
+            0.012058441527187824,
+            -0.030699847266077995,
+            -0.007814365439116955,
+            0.015264731831848621,
+            -0.007876322604715824,
+            0.0008906363509595394,
+            0.024132372811436653,
+            0.02103450708091259,
+            0.0028364830650389194,
+            -0.0009700191440060735,
+            0.02438020147383213,
+            0.007427132688462734,
+            -0.01720864325761795,
+            0.028794659301638603,
+            0.014288905076682568,
+            0.060160547494888306,
+            -0.03271345794200897,
+            0.001297231181524694,
+            -0.04135650396347046,
+            -0.031118059530854225,
+            0.006408709101378918,
+            0.014350862242281437,
+            0.007597515359520912,
+            -0.028252532705664635,
+            0.005978880450129509,
+            0.02004319056868553,
+            0.01469937153160572,
+            0.009704063646495342,
+            0.0037580979987978935,
+            -0.036740683019161224,
+            -0.000691695255227387,
+            -0.011214273050427437,
+            -0.0038781403563916683,
+            0.035656433552503586,
+            -0.008186109364032745,
+            -0.05151750147342682,
+            -0.0006418389966711402,
+            -0.03255856782197952,
+            -0.0353776253759861,
+            0.021313315257430077,
+            -0.00839521549642086,
+            -0.0039207362569868565,
+            -0.013986863195896149,
+            -0.000537770101800561,
+            -0.010524998418986797,
+            0.027509044855833054,
+            0.028345469385385513,
+            0.022490503266453743,
+            -0.05384090170264244,
+            0.009479468688368797,
+            0.00023766436788719147,
+            -0.05817791447043419,
+            0.0009709871956147254,
+            -0.007117345929145813,
+            0.011191039346158504,
+            -0.020213572308421135,
+            -0.02414786070585251,
+            0.011500826105475426,
+            0.03392162546515465,
+            0.014304393902420998,
+            0.0321558453142643,
+            -0.013367289677262306,
+            -0.09250226616859436,
+            0.018292896449565887,
+            -0.008689512498676777,
+            0.04126356914639473,
+            0.00865078903734684,
+            -0.004278926644474268,
+            -0.015845581889152527,
+            -0.00865078903734684,
+            -0.006943091284483671,
+            0.01283290795981884,
+            0.004530628211796284,
+            -0.005626498255878687,
+            0.0016631665639579296,
+            0.013708055019378662,
+            -0.005967263597995043,
+            0.007481344975531101,
+            0.003651608945801854,
+            -0.013189162127673626,
+            -0.06059424951672554,
+            -0.04055105894804001,
+            -0.018540725111961365,
+            0.050835974514484406,
+            -0.0069159846752882,
+            -0.046684831380844116,
+            -0.010850274004042149,
+            0.05058814212679863,
+            0.012259802781045437,
+            -0.0334569476544857,
+            -0.0007841471815481782,
+            0.0050882440991699696,
+            0.015458349138498306,
+            0.004546117503196001,
+            -0.027292195707559586,
+            0.02820606529712677,
+            0.0008412640891037881,
+            0.02300165221095085,
+            -0.018339363858103752,
+            -0.005312839522957802,
+            0.035780347883701324,
+            -0.024844881147146225,
+            0.023791607469320297,
+            0.0055219451896846294,
+            0.031629208475351334,
+            0.002952652983367443,
+            0.010060317814350128,
+            0.004704883322119713,
+            0.0186491496860981,
+            0.004275054205209017,
+            0.0276639387011528,
+            0.011462102644145489,
+            0.006428070832043886,
+            0.021623101085424423,
+            -0.009378788061439991,
+            -0.014513500034809113,
+            0.01885051093995571,
+            -0.0014695499558001757,
+            0.025232113897800446,
+            -0.020538847893476486,
+            0.002726121572777629,
+            -0.0016883367206901312,
+            -0.017967619001865387,
+            -0.040922801941633224,
+            -0.0007405834621749818,
+            0.021747015416622162,
+            -0.011671207845211029,
+            -0.0228312686085701,
+            -0.013715799897909164,
+            -0.01585332676768303,
+            0.0038490979932248592,
+            0.034169457852840424,
+            -0.0398695282638073,
+            0.00963436160236597,
+            -0.000830615172162652,
+            -0.03125746175646782,
+            -0.0239619892090559,
+            -0.01093546487390995,
+            0.014614180661737919,
+            -0.01652711257338524,
+            -0.021173911169171333,
+            -0.028469383716583252,
+            -0.008410705253481865,
+            0.006048582494258881,
+            -0.05065010115504265,
+            -0.004054331686347723,
+            0.019965743646025658,
+            0.0276639387011528,
+            -0.003099801717326045,
+            -0.02177799493074417,
+            0.0012468908680602908,
+            0.022351099178195,
+            0.02936776541173458,
+            0.014226946979761124,
+            0.027586491778492928,
+            -0.002178186783567071,
+            -0.0011675080750137568,
+            -0.0073148347437381744,
+            0.01307299267500639,
+            0.0011287847300991416,
+            0.036616768687963486,
+            0.0122210793197155,
+            -0.020306508988142014,
+            0.0013959755888208747,
+            -0.006431943271309137,
+            0.029181893914937973,
+            -0.02270735427737236,
+            -9.832092473516241e-05,
+            0.03531566634774208,
+            0.027741385623812675,
+            0.004933350719511509,
+            -0.0143121387809515,
+            -0.05142456665635109,
+            0.021592123433947563,
+            -0.01765783317387104,
+            0.031025122851133347,
+            0.016666516661643982,
+            0.003397971158847213,
+            -0.0034212053287774324,
+            0.012368228286504745,
+            -0.022087780758738518,
+            -0.004406713880598545,
+            -0.021142931655049324,
+            0.04389675334095955,
+            -0.043184246867895126,
+            0.008472662419080734,
+            0.029445212334394455,
+            0.011926782317459583,
+            -0.019082851707935333,
+            -0.005425137002021074,
+            0.029212871566414833,
+            -0.00162637943867594,
+            0.01308848150074482,
+            0.0058665829710662365,
+            -0.026935940608382225,
+            0.01202746294438839,
+            -0.031660184264183044,
+            0.02177799493074417,
+            -0.013436991721391678,
+            0.018447788432240486,
+            0.03342596814036369,
+            -0.0030397805385291576,
+            -0.0038239278364926577,
+            -0.016263794153928757,
+            -0.00969631876796484,
+            0.0662323608994484,
+            0.01054823212325573,
+            0.03726732358336449,
+            0.0018974426202476025,
+            -0.009773765690624714,
+            0.007969259284436703,
+            0.0063041565008461475,
+            -0.012329504825174809,
+            0.010494019836187363,
+            0.024178840219974518,
+            0.0012730291346088052,
+            -0.002412462839856744,
+            -0.019408127292990685,
+            -0.03438630700111389,
+            0.003727119415998459,
+            -0.030932186171412468,
+            0.022691864520311356,
+            0.009208405390381813,
+            0.017456471920013428,
+            0.008240322582423687,
+            -0.015907539054751396,
+            0.03200095146894455,
+            0.05461536720395088,
+            0.0019768255297094584,
+            0.013375034555792809,
+            0.007938280701637268,
+            0.007036027032881975,
+            -0.011338187381625175,
+            0.04702559858560562,
+            0.000662652775645256,
+            0.005169562995433807,
+            0.022536970674991608,
+            0.02145271934568882,
+            0.022939693182706833,
+            0.002304037567228079,
+            -0.011694442480802536,
+            0.010478530079126358,
+            -0.0006302220281213522,
+            0.004445436876267195,
+            -0.010695381090044975,
+            -0.027865299955010414,
+            0.007492962293326855,
+            0.01302652433514595,
+            0.022598929703235626,
+            0.021592123433947563,
+            -0.03178409859538078,
+            -0.01855621486902237,
+            0.03364282101392746,
+            -0.02997184917330742,
+            0.03612111136317253,
+            -0.0014424435794353485,
+            -0.023807097226381302,
+            -0.0058356039226055145,
+            0.016929835081100464,
+            0.02656419761478901,
+            -0.029181893914937973,
+            0.01804506592452526,
+            0.029693040996789932,
+            -0.0074426219798624516,
+            0.0005939188995398581,
+            0.012112653814256191,
+            0.02447313815355301,
+            -0.009293596260249615,
+            -0.0024027819745242596,
+            -0.0013320822035893798,
+            -0.011926782317459583,
+            -0.018370341509580612,
+            -0.0044725434854626656,
+            -0.022629907354712486,
+            0.005564541090279818,
+            -0.007365175057202578,
+            -0.003481226274743676,
+            -0.008581086993217468,
+            0.01154729351401329,
+            0.001979729626327753,
+            -0.019857317209243774,
+            -0.022335609421133995,
+            0.01652711257338524,
+            0.03887821361422539,
+            -0.018215449526906013,
+            -0.02106548473238945,
+            0.024442158639431,
+            -0.01759587600827217,
+            -0.01601596549153328,
+            0.036678727716207504,
+            0.013739033602178097,
+            0.007736918982118368,
+            -0.03358086198568344,
+            -0.030049296095967293,
+            -0.03274443745613098,
+            0.006296411622315645,
+            0.03785591572523117,
+            -0.0028752065263688564,
+            0.02569679357111454,
+            0.03701949119567871,
+            0.0057891360484063625,
+            -0.028934063389897346,
+            -0.014823286794126034,
+            -0.045104920864105225,
+            0.060160547494888306,
+            0.0011481463443487883,
+            0.05161044001579285,
+            0.016294771805405617,
+            -0.01456771232187748,
+            0.00419760774821043,
+            -0.013251119293272495,
+            0.026192452758550644,
+            0.02280029095709324,
+            0.025929134339094162,
+            0.006582964211702347,
+            0.022505993023514748,
+            -0.0037522895727306604,
+            -0.012197845615446568,
+            0.007307090330868959,
+            0.014536733739078045,
+            0.0002497654058970511,
+            -0.011392400600016117,
+            0.015783624723553658,
+            -0.01307299267500639,
+            -0.011361422017216682,
+            0.023295948281884193,
+            0.01368482131510973,
+            -0.012344993650913239,
+            0.014064309187233448,
+            -0.008565598167479038,
+            0.01239920686930418,
+            -0.023915521800518036,
+            0.022691864520311356,
+            -0.03262052312493324,
+            0.009587894193828106,
+            0.0015053689712658525,
+            -0.0073148347437381744,
+            0.024550583213567734,
+            -0.0027454833034425974,
+            -0.05071205645799637,
+            0.03550153970718384,
+            -0.008015726692974567,
+            0.0037213109899312258,
+            0.006381602957844734,
+            0.011392400600016117,
+            -0.0012623801594600081,
+            0.03807276859879494,
+            0.01239920686930418,
+            0.015969496220350266,
+            0.02199484594166279,
+            0.011911292560398579,
+            -0.008433938957750797,
+            -0.004096927121281624,
+            -0.0016137943603098392,
+            -0.01784370467066765,
+            0.025944624096155167,
+            0.0018674320308491588,
+            0.03962169960141182,
+            -0.018168980255723,
+            -0.02300165221095085,
+            0.028066661208868027,
+            0.007597515359520912,
+            -0.02351279929280281,
+            -0.0017377089243382215,
+            0.01202746294438839,
+            -0.009185170754790306,
+            0.027385130524635315,
+            0.01196550577878952,
+            0.019113829359412193,
+            -0.018060555681586266,
+            -0.004340883810073137,
+            -0.005072754807770252,
+            -0.015938518568873405,
+            0.02238207869231701,
+            0.020771188661456108,
+            0.011152315884828568,
+            -0.011880313977599144,
+            -0.01307299267500639,
+            0.020600805059075356,
+            0.035036858171224594,
+            -0.0002744515077210963,
+            0.005672966130077839,
+            0.017487451434135437,
+            0.002373739378526807,
+            -0.04470219835639,
+            0.016232814639806747,
+            -0.014025585725903511,
+            0.011214273050427437,
+            0.019593998789787292,
+            -0.021917399019002914,
+            -0.03196997195482254,
+            -0.011539548635482788,
+            -0.0414804182946682,
+            0.00493722315877676,
+            -0.0069159846752882,
+            0.013576395809650421,
+            -0.010176488198339939,
+            -0.0014831031439825892,
+            0.016666516661643982,
+            0.0038103745318949223,
+            0.02177799493074417,
+            -0.011678952723741531,
+            -0.012073930352926254,
+            -0.020213572308421135,
+            -0.020786678418517113,
+            0.004278926644474268,
+            -0.030405549332499504,
+            -0.004488032776862383,
+            0.021375272423028946,
+            -0.04783104360103607,
+            -0.03689557686448097,
+            0.02898053266108036,
+            -0.008503641001880169,
+            0.020755698904395103,
+            0.022955182939767838,
+            -0.015094350092113018,
+            -0.0008717587334103882,
+            -0.018695618957281113,
+            0.02258343994617462,
+            -0.007837600074708462,
+            0.026300877332687378,
+            -0.0027551641687750816,
+            0.0006292539183050394,
+            -0.02659517526626587,
+            0.020306508988142014,
+            0.005227648187428713,
+            0.012275291606783867,
+            -0.02390003204345703,
+            0.006815304048359394,
+            -0.03133491054177284,
+            -0.01784370467066765,
+            0.0180140882730484,
+            -0.011152315884828568,
+            0.027989214286208153,
+            -0.006060199346393347,
+            0.028438406065106392,
+            0.022722844034433365,
+            0.024751944467425346,
+            0.02676555886864662,
+            -0.02363671362400055,
+            0.008495896123349667,
+            0.03726732358336449,
+            0.00865078903734684,
+            -0.015876561403274536,
+            0.008170620538294315,
+            -0.04863648861646652,
+            -0.008062195032835007,
+            0.05030933395028114,
+            0.002127846470102668,
+            0.023141054436564445,
+            0.016294771805405617,
+            -0.002873270306736231,
+            -0.005688455421477556,
+            0.010269423946738243,
+            -0.04296739399433136,
+            -0.023172033950686455,
+            0.022629907354712486,
+            -0.021561143919825554,
+            6.54212328754511e-07,
+            0.03122648410499096,
+            0.012120398692786694,
+            -0.0009109660750254989,
+            -0.05108380317687988,
+            0.02100352756679058,
+            0.01178737822920084,
+            0.014443797990679741,
+            -0.010300402529537678,
+            0.026068538427352905,
+            0.01733255758881569,
+            0.038413532078266144,
+            0.012755461037158966,
+            -0.002679653698578477,
+            -0.03782493621110916,
+            0.009177426807582378,
+            -0.021685058251023293,
+            0.0011471783509477973,
+            0.0074503663927316666,
+            -0.02273833192884922,
+            -0.024782923981547356,
+            0.0043912241235375404,
+            0.004588713403791189,
+            0.011880313977599144,
+            0.022087780758738518,
+            0.009417511522769928,
+            -0.021204888820648193,
+            0.015404135920107365,
+            -0.05446047708392143,
+            -0.015450604259967804,
+            -0.023884542286396027,
+            0.02566581591963768,
+            0.020012211054563522,
+            0.010184233076870441,
+            0.005545179359614849,
+            0.010904486291110516,
+            -0.020988039672374725,
+            -0.009239383973181248,
+            0.020631784573197365,
+            0.011074868962168694,
+            -0.012840652838349342,
+            0.04113965481519699,
+            -0.00011126903700642288,
+            0.031210994347929955,
+            -0.02280029095709324,
+            0.002941036131232977,
+            -0.001241082325577736,
+            -0.03649285435676575,
+            0.009130958467721939,
+            0.01981084980070591,
+            0.02151467651128769,
+            -0.01861817203462124,
+            0.0024143988266587257,
+            0.01048627495765686,
+            0.004766840487718582,
+            0.038351576775312424,
+            0.027121812105178833,
+            -0.044206541031599045,
+            0.011663463898003101,
+            0.012995545752346516,
+            -0.009363298304378986,
+            0.048419639468193054,
+            -0.017518429085612297,
+            0.017673322930932045,
+            0.008240322582423687,
+            -0.01798310875892639,
+            0.02318752370774746,
+            -0.01469937153160572,
+            -0.013282098807394505,
+            0.00650551775470376,
+            0.009192915633320808,
+            -0.0001770381786627695,
+            -0.037577107548713684,
+            -0.017859194427728653,
+            -0.005719434004276991,
+            0.0038297362625598907,
+            0.01562873087823391,
+            0.013498948886990547,
+            0.009967382065951824,
+            0.005161818582564592,
+            -0.00945623405277729,
+            -0.0016089539276435971,
+            0.008124152198433876,
+            -0.0005532593932002783,
+            0.0019216446671634912,
+            -0.01926872320473194,
+            0.02698240801692009,
+            0.02572777308523655,
+            -0.0018354853382334113,
+            -0.04652994126081467,
+            -0.0036709706764668226,
+            0.0037077576853334904,
+            -0.017735280096530914,
+            -0.0035315665882080793,
+            -0.04519785940647125,
+            -0.01810702309012413,
+            0.0143121387809515,
+            -0.014614180661737919,
+            -0.0460033044219017,
+            0.009076746180653572,
+            -0.013886182568967342,
+            0.0063970922492444515,
+            -0.00932457484304905,
+            0.005762029904872179,
+            0.003651608945801854,
+            0.005680711008608341,
+            -0.005909178406000137,
+            0.05108380317687988,
+            -0.03466511517763138,
+            0.028051171451807022,
+            0.008565598167479038,
+            -0.012437930330634117,
+            -0.02261441759765148,
+            0.00887538492679596,
+            -0.044981006532907486,
+            -0.033209118992090225,
+            0.016743963584303856,
+            -0.01781272701919079,
+            0.011074868962168694,
+            0.06161654368042946,
+            0.007330324035137892,
+            -0.023342415690422058,
+            -0.01233724970370531,
+            -0.021049996837973595,
+            -0.04191412031650543,
+            0.003442503046244383,
+            0.005463860463351011,
+            -0.004948840010911226,
+            -0.03807276859879494,
+            -0.021716037765145302,
+            0.04615819454193115,
+            -0.012004229240119457,
+            0.012422440573573112,
+            0.01080380566418171,
+            -0.0019410063978284597,
+            -0.04860550910234451,
+            -0.021019017323851585,
+            0.0395597442984581,
+            0.021886419504880905,
+            -0.008426194079220295,
+            0.0028364830650389194,
+            -0.018277406692504883,
+            -0.003289545886218548,
+            0.006296411622315645,
+            -0.009107724763453007,
+            -0.03562545403838158,
+            0.02067825198173523,
+            0.0054832217283546925,
+            -0.023373395204544067,
+            -0.025805220007896423,
+            -0.03296129032969475,
+            0.009944148361682892,
+            0.02453509531915188,
+            -0.01968693546950817,
+            0.005823987070471048,
+            0.00027203132049180567,
+            -0.000881923595443368,
+            0.00042740863864310086,
+            -0.009959637187421322,
+            -0.019764382392168045,
+            -0.020151615142822266,
+            0.03268248215317726,
+            -0.0060059865936636925,
+            0.0074387495405972,
+            -0.03630698472261429,
+            -0.010974188335239887,
+            0.011322698555886745,
+            -0.002395037328824401,
+            -0.013452480547130108,
+            -0.009092235006392002,
+            -0.01739451475441456,
+            -0.009882191196084023,
+            -0.007876322604715824,
+            0.022660886868834496,
+            0.006168624851852655,
+            0.005409647710621357,
+            0.0023795480374246836,
+            -0.008581086993217468,
+            0.02898053266108036,
+            0.017874684184789658,
+            -0.014188224449753761,
+            -0.00920066051185131,
+            0.03451022133231163,
+            0.02665713243186474,
+            -0.009115469641983509,
+            0.01798310875892639,
+            0.023264970630407333,
+            -0.0010484338272362947,
+            -0.01196550577878952,
+            0.001922612776979804,
+            -0.0009550138493068516,
+            -0.0032972905319184065,
+            0.00845717266201973,
+            -0.015365412458777428,
+            0.0315207801759243,
+            0.04287445917725563,
+            -0.002145271748304367,
+            0.009587894193828106,
+            -0.003727119415998459,
+            -0.008472662419080734,
+            0.017069239169359207,
+            0.009316830895841122,
+            0.04625113308429718,
+            0.01098193321377039,
+            0.005680711008608341,
+            -0.033983584493398666,
+            -0.001229465357027948,
+            -0.01184933539479971,
+            0.03308520466089249,
+            0.02061629481613636,
+            0.00889087375253439,
+            -0.008379725739359856,
+            0.01344473659992218,
+            0.008341003209352493,
+            0.00661007035523653,
+            0.020569827407598495,
+            -0.016542602330446243,
+            -0.017286090180277824,
+            0.030467506498098373,
+            -0.00672236829996109,
+            0.01289486512541771,
+            0.02029101923108101,
+            -0.002089123008772731,
+            0.010842529125511646,
+            0.07100307941436768,
+            -0.006184114143252373,
+            0.015713922679424286,
+            -0.019082851707935333,
+            0.0015789433382451534,
+            -0.007640110794454813,
+            -0.03370477631688118,
+            0.018246427178382874,
+            0.0039962464943528175,
+            0.00865078903734684,
+            0.01198099460452795,
+            0.009053511545062065,
+            -0.007946024648845196,
+            -0.00963436160236597,
+            0.015195030719041824,
+            0.0019080914789810777,
+            0.00033568276558071375,
+            0.001336922636255622,
+            -0.0065210070461034775,
+            -0.004011735785752535,
+            -0.021948376670479774,
+            0.02763296104967594,
+            0.007140580099076033,
+            0.030606910586357117,
+            -0.055296897888183594,
+            0.030095763504505157,
+            -0.00789568480104208,
+            0.02554190158843994,
+            -0.016201836988329887,
+            -0.011090358719229698,
+            0.001609921921044588,
+            0.018153492361307144,
+            0.004491904750466347,
+            0.006462921854108572,
+            0.011888058856129646,
+            0.023233991116285324,
+            -0.03444826602935791,
+            0.017053749412298203,
+            0.023110076785087585,
+            0.0048675211146473885,
+            0.0003535923024173826,
+            0.005165690556168556,
+            0.015458349138498306,
+            0.007198664825409651,
+            -0.023048119619488716,
+            -0.0025692922063171864,
+            0.017611365765333176,
+            -0.02306360937654972,
+            0.0036980770528316498,
+            -0.006923729553818703,
+            0.004623564425855875,
+            -0.0014705180656164885,
+            -0.03509881719946861,
+            -0.00100099784322083,
+            -0.025743262842297554,
+            0.01923774555325508,
+            0.009556914679706097,
+            -0.025867177173495293,
+            0.003932353109121323,
+            0.025650326162576675,
+            -0.0003695656778290868,
+            0.00821708794683218,
+            0.0004683101433329284,
+            0.00869725737720728,
+            -0.00974278710782528,
+            -0.01493945624679327,
+            0.004801691509783268,
+            0.01531120017170906,
+            -0.01493945624679327,
+            -0.007431004662066698,
+            -0.009510447271168232,
+            0.030560443177819252,
+            0.0006674932083114982,
+            0.004189862869679928,
+            -0.005045648664236069,
+            -0.01858719252049923,
+            -0.0032237162813544273,
+            -0.021142931655049324,
+            0.004367990419268608,
+            -0.0009046735358424485,
+            0.04197607934474945,
+            0.0038161829579621553,
+            -0.02029101923108101,
+            0.0032972905319184065,
+            -0.03136589005589485,
+            0.0005000148667022586,
+            -0.007276111748069525,
+            -0.02004319056868553,
+            0.0015005286550149322,
+            0.0016525175888091326,
+            0.012631546705961227,
+            -0.03255856782197952,
+            -0.009301341138780117,
+            0.018416810780763626,
+            -0.00020680672605521977,
+            -0.010687636211514473,
+            -0.017828216776251793,
+            0.00035189816844649613,
+            -0.011338187381625175,
+            -0.027973726391792297,
+            -0.02850036323070526,
+            -0.0026719090528786182,
+            0.013491204008460045,
+            -0.018277406692504883,
+            -0.014993669465184212,
+            0.0031249718740582466,
+            0.021592123433947563,
+            -0.009781510569155216,
+            0.008302279748022556,
+            0.02241305634379387,
+            -0.008255811408162117,
+            -0.005514200776815414,
+            -0.000941460661124438,
+            -0.009130958467721939,
+            0.004267309792339802,
+            0.006300284061580896,
+            0.043339136987924576,
+            -0.032403673976659775,
+            -0.0018238683696836233,
+            0.014815541915595531,
+            0.0034153966698795557,
+            0.0025905899237841368,
+            0.002503462601453066,
+            -0.0007841471815481782,
+            0.01344473659992218,
+            0.01178737822920084,
+            0.015466093085706234,
+            0.00100099784322083,
+            -0.0033011629711836576,
+            0.006408709101378918,
+            -0.021173911169171333,
+            0.015721667557954788,
+            0.027958236634731293,
+            -0.0033999073784798384,
+            0.005525817628949881,
+            -0.013731288723647594,
+            -0.008759214542806149,
+            0.013134949840605259,
+            0.0026641644071787596,
+            -0.005068882368505001,
+            -0.009394276887178421,
+            0.017967619001865387,
+            0.019082851707935333,
+            0.006668155547231436,
+            -0.037453193217515945,
+            -0.02898053266108036,
+            -0.009959637187421322,
+            0.020569827407598495,
+            0.014916222542524338,
+            0.0010339126456528902,
+            -0.010540487244725227,
+            -0.009595638141036034,
+            -0.011903548613190651,
+            0.010989678092300892,
+            0.02721474878489971,
+            -0.032372694462537766,
+            0.0008214184199459851,
+            -0.029383255168795586,
+            0.03068435750901699,
+            0.0020000594668090343,
+            0.011446612887084484,
+            -0.0008185141487047076,
+            -0.015140817500650883,
+            -0.029181893914937973,
+            0.013824224472045898,
+            -0.004395096562802792,
+            0.001024231780320406,
+            -0.0031753121875226498,
+            0.0201051477342844,
+            0.012887120246887207,
+            -0.0014472840121015906,
+            0.01968693546950817,
+            0.008178364485502243,
+            0.016480645164847374,
+            -0.010153254494071007,
+            -0.04135650396347046,
+            0.019965743646025658,
+            0.007562664337456226,
+            0.014219203032553196,
+            0.004739734344184399,
+            0.004588713403791189,
+            -0.020771188661456108,
+            0.002724185585975647,
+            0.004333139397203922,
+            0.014405074529349804,
+            0.011028401553630829,
+            0.0025615475606173277,
+            0.011678952723741531,
+            -0.02228914201259613,
+            0.01929970271885395,
+            -0.01631026156246662,
+            0.01006806269288063,
+            0.004445436876267195,
+            -0.026873983442783356,
+            0.007547175046056509,
+            0.007822110317647457,
+            -0.0019187404541298747,
+            -0.017549408599734306,
+            0.02160761132836342,
+            -0.0034831624943763018,
+            -0.03509881719946861,
+            0.0360281765460968,
+            -0.0043447562493383884,
+            0.019346170127391815,
+            0.03209388628602028,
+            -0.01771979033946991,
+            -0.01993476413190365,
+            0.00906900130212307,
+            0.010408828034996986,
+            -0.002873270306736231,
+            0.017859194427728653,
+            -0.002919738180935383,
+            -0.02232012152671814,
+            0.017766257748007774,
+            -0.028918573632836342,
+            -0.014234691858291626,
+            0.020337486639618874,
+            -0.03624502569437027,
+            -0.0021026763133704662,
+            0.03175312280654907,
+            -0.024070415645837784,
+            0.005734923295676708,
+            0.005196669604629278,
+            -0.0019351978553459048,
+            0.0012236569309607148,
+            -0.006451305001974106,
+            0.03801080957055092,
+            -0.010865762829780579,
+            0.0201051477342844,
+            0.01929970271885395,
+            -0.023388884961605072,
+            0.01977987214922905,
+            -0.001109423115849495,
+            0.01675945147871971,
+            -0.003940097521990538,
+            0.03466511517763138,
+            -0.006017603911459446,
+            -0.011330443434417248,
+            0.006431943271309137,
+            0.008488151244819164,
+            -0.009959637187421322,
+            0.008681767620146275,
+            -0.018122512847185135,
+            0.027152791619300842,
+            0.02091059274971485,
+            -0.0002691270783543587,
+            0.013800990767776966,
+            0.0004421719058882445,
+            -0.020833145827054977,
+            0.00789568480104208,
+            -0.027121812105178833,
+            -0.009967382065951824,
+            0.01646515540778637,
+            -0.005762029904872179,
+            0.007380664348602295,
+            -0.02064727433025837,
+            0.028004704043269157,
+            0.0077524082735180855,
+            -0.013591884635388851,
+            -0.0008824076503515244,
+            0.023760627955198288,
+            -0.005432881414890289,
+            0.009355553425848484,
+            0.02070923149585724,
+            0.015713922679424286,
+            0.013615119270980358,
+            0.016263794153928757,
+            0.03351890295743942,
+            0.005328328814357519,
+            0.005568413063883781,
+            0.003519949736073613,
+            0.016991792246699333,
+            -0.008557853288948536,
+            0.018029578030109406,
+            0.0008093173382803798,
+            -0.013576395809650421,
+            0.021499186754226685,
+            -0.0018684001406654716,
+            -0.013622863218188286,
+            -0.0040891822427511215,
+            -0.004337011836469173,
+            0.03271345794200897,
+            0.022041313350200653,
+            -0.004468671046197414,
+            -0.024442158639431,
+            -0.00394784240052104,
+            0.016682006418704987,
+            -0.003167567541822791,
+            0.011136826127767563,
+            0.0007952801533974707,
+            -0.01586107164621353,
+            -0.009595638141036034,
+            0.0063583687879145145,
+            0.010586955584585667,
+            -0.008921852335333824,
+            0.010346870869398117,
+            -0.0228312686085701,
+            -0.004011735785752535,
+            0.00046613195445388556,
+            -0.004844287410378456,
+            0.0053438181057572365,
+            -0.011314953677356243,
+            0.004948840010911226,
+            0.016062432900071144,
+            0.01026167906820774,
+            0.004832670092582703,
+            0.006989559158682823,
+            0.012066186405718327,
+            0.009161937050521374,
+            -0.01627928391098976,
+            0.00428667152300477,
+            -0.005072754807770252,
+            -0.02345084212720394,
+            0.005831731949001551,
+            0.00902253296226263,
+            -0.01122201792895794,
+            0.009386532939970493,
+            -0.010958699509501457,
+            0.03296129032969475,
+            0.01030814740806818,
+            -0.037391237914562225,
+            -0.0033379499800503254,
+            -0.0016244432190433145,
+            0.014885243959724903,
+            0.012298526242375374,
+            0.008550108410418034,
+            -0.017518429085612297,
+            -0.0047823297791182995,
+            -0.009835722856223583,
+            0.0012827098835259676,
+            -0.012306271120905876,
+            0.017456471920013428,
+            0.022567950189113617,
+            -0.036337960511446,
+            0.01968693546950817,
+            -0.010432061739265919,
+            -0.023652203381061554,
+            0.020476890727877617,
+            0.011500826105475426,
+            0.01942361705005169,
+            -0.009402021765708923,
+            0.018896980211138725,
+            -0.0001331114035565406,
+            0.025510922074317932,
+            -0.004228586331009865,
+            -0.022087780758738518,
+            -0.012499887496232986,
+            -0.03209388628602028,
+            -0.0051811798475682735,
+            0.006819176487624645,
+            -0.00045524101005867124,
+            0.013111716136336327,
+            0.014226946979761124,
+            -0.025712283328175545,
+            0.007295473478734493,
+            0.032589543610811234,
+            0.014637414366006851,
+            0.023311438038945198,
+            -0.011276230216026306,
+            -0.0010794125264510512,
+            -0.0026060794480144978,
+            -0.013770012184977531,
+            -0.03212486580014229,
+            -0.0017493258928880095,
+            -0.036585792899131775,
+            0.005680711008608341,
+            -0.016387708485126495,
+            -0.013096226379275322,
+            0.023264970630407333,
+            -0.034138478338718414,
+            0.0014608372002840042,
+            0.013708055019378662,
+            0.010540487244725227,
+            0.012391461990773678,
+            -0.00162637943867594,
+            0.015032391995191574,
+            -0.029243851080536842,
+            -0.010284913703799248,
+            -0.004491904750466347,
+            -0.01981084980070591,
+            -0.007338068913668394,
+            -0.018060555681586266,
+            -0.013127204962074757,
+            -0.003566417610272765,
+            -0.010060317814350128,
+            0.002247888594865799,
+            0.0032062907703220844,
+            -0.014714861288666725,
+            0.03206290677189827,
+            0.003157886676490307,
+            0.018958937376737595,
+            0.0033418224193155766,
+            0.004449309315532446,
+            0.018788553774356842,
+            -0.0026331855915486813,
+            -0.0058278595097362995,
+            0.030963165685534477,
+            -0.00032672801171429455,
+            0.008759214542806149,
+            0.019082851707935333,
+            0.004724245052784681,
+            0.0038529701996594667,
+            -0.0260530486702919,
+            0.012887120246887207,
+            -0.027447087690234184,
+            0.005080499220639467,
+            -0.0077175572514534,
+            -0.024302754551172256,
+            -0.028128618374466896,
+            -0.006571347359567881,
+            0.012058441527187824,
+            -0.02898053266108036,
+            -0.00613377382978797,
+            0.002315654419362545,
+            -0.00400399137288332,
+            -0.002673845272511244,
+            0.007241260726004839,
+            0.029150914400815964,
+            -0.033209118992090225,
+            0.008751469664275646,
+            0.005595519673079252,
+            -0.0005053392960689962,
+            0.013297587633132935,
+            0.008038961328566074,
+            0.003343758638948202,
+            0.020848635584115982,
+            0.006637176964432001,
+            0.002458930714055896,
+            -0.0025576751213520765,
+            -0.014242436736822128,
+            0.004000118933618069,
+            0.02154565416276455,
+            0.007562664337456226,
+            0.018385831266641617,
+            -0.008914108388125896,
+            0.002621568739414215,
+            -0.01455222349613905,
+            0.005552923772484064,
+            -0.002867461647838354,
+            0.004309905227273703,
+            -0.012236569076776505,
+            0.008093173615634441,
+            0.0028771425131708384,
+            -0.0043447562493383884,
+            0.018060555681586266,
+            -0.002319526858627796,
+            0.016480645164847374,
+            0.004015608225017786,
+            0.015620986931025982,
+            -0.0035412474535405636,
+            0.01178737822920084,
+            0.0005731051205657423,
+            -0.017905661836266518,
+            -0.017022771760821342,
+            -0.039249956607818604,
+            0.007702067960053682,
+            0.03531566634774208,
+            -0.0009564659558236599,
+            -0.03444826602935791,
+            0.010145509615540504,
+            -0.011942271143198013,
+            -0.009975126944482327,
+            -0.022862248122692108,
+            0.013862947933375835,
+            0.012430185452103615,
+            0.007415515370666981,
+            0.012252057902514935,
+            -0.006997303571552038,
+            0.01026167906820774,
+            -0.005216030869632959,
+            0.008356492035090923,
+            -0.0010823167394846678,
+            -0.037422213703393936,
+            0.012344993650913239,
+            0.02216522768139839,
+            -0.002635121811181307,
+            0.0016089539276435971,
+            -0.006145390681922436,
+            0.022103270515799522,
+            -0.013971373438835144,
+            -0.00650551775470376,
+            -0.00046443779137916863,
+            0.00797700323164463,
+            -0.01795213110744953,
+            0.004778457339853048,
+            0.0755879208445549,
+            -0.007775642443448305,
+            0.006122156511992216,
+            0.009270362555980682,
+            -0.015148562379181385,
+            -0.01637221872806549,
+            -0.004561606794595718,
+            -0.020260039716959,
+            0.008844406343996525,
+            -0.0018819533288478851,
+            -0.0110981035977602,
+            0.012468908913433552,
+            0.0053050946444272995,
+            0.002031038049608469,
+            -0.008379725739359856,
+            0.010811550542712212,
+            0.009525936096906662,
+            0.010432061739265919,
+            0.02360573597252369,
+            0.011477591469883919,
+            -0.00858883187174797,
+            -0.011795123107731342,
+            0.01129171997308731,
+            0.0028577810153365135,
+            0.016697494313120842,
+            0.007907302118837833,
+            0.0010648912284523249,
+            0.019501063972711563,
+            -0.002309845993295312,
+            0.02180897258222103,
+            -2.2689444449497387e-05,
+            -0.009301341138780117,
+            -0.004236331209540367,
+            -0.015349923633038998,
+            0.024271776899695396,
+            -0.0020155487582087517,
+            0.0015450604259967804,
+            -0.020213572308421135,
+            0.0013979118084535003,
+            0.013243375346064568,
+            0.011957760900259018,
+            -0.0020407189149409533,
+            -0.0051347119733691216,
+            0.0007154133054427803,
+            -0.016356728971004486,
+            0.032806396484375,
+            0.007578153628855944,
+            0.0007681738352403045,
+            0.0008838597568683326,
+            0.0009288755827583373,
+            -0.0032972905319184065,
+            0.0024608669336885214,
+            -0.004766840487718582,
+            0.009549170732498169,
+            -0.010943209752440453,
+            -0.010796061716973782,
+            0.010075807571411133,
+            -0.02402394637465477,
+            0.003566417610272765,
+            -0.00558777479454875,
+            -0.008062195032835007,
+            0.0076207490637898445,
+            0.00015537731815129519,
+            -0.007392281666398048,
+            -0.0030939930584281683,
+            0.0023408245760947466,
+            0.013429246842861176,
+            0.004406713880598545,
+            0.010850274004042149,
+            0.012956822291016579,
+            0.005359307397156954,
+            -0.006629432085901499,
+            0.015512561425566673,
+            -0.006946963258087635,
+            -0.004228586331009865,
+            -0.0012468908680602908,
+            -0.019996721297502518,
+            -0.0009419447160325944,
+            -0.004708755761384964,
+            0.0015092412941157818,
+            0.0001705036120256409,
+            -8.749050175538287e-05,
+            -0.0018490385264158249,
+            -0.011988739483058453,
+            -0.001831613015383482,
+            -0.011663463898003101,
+            -0.014498010277748108,
+            0.010253935120999813,
+            -0.0041201612912118435,
+            -0.022939693182706833,
+            -0.015349923633038998,
+            -0.020662762224674225,
+            0.02145271934568882,
+            -0.004387352149933577,
+            -0.005800753366202116,
+            0.006733985152095556,
+            -0.010215211659669876,
+            -0.015706177800893784,
+            -0.008069939911365509,
+            -0.0205853171646595,
+            0.010703125037252903,
+            0.003214035416021943,
+            0.0006791101768612862,
+            0.00882891658693552,
+            0.014397329650819302,
+            -0.01589204929769039,
+            0.0111755495890975,
+            0.012182355858385563,
+            0.011361422017216682,
+            -0.006854027509689331,
+            0.0008911203476600349,
+            -0.014103032648563385,
+            0.02596011385321617,
+            0.021623101085424423,
+            0.005169562995433807,
+            -0.04132552444934845,
+            -0.005862710531800985,
+            -0.013344055972993374,
+            -0.010524998418986797,
+            -0.012120398692786694,
+            0.011771888472139835,
+            -0.0038820127956569195,
+            0.0013117524795234203,
+            0.006288666743785143,
+            0.0015063370810821652,
+            0.004367990419268608,
+            -0.0022382077295333147,
+            -0.007709812838584185,
+            -0.03212486580014229,
+            -0.02817508764564991,
+            -0.00863530021160841,
+            0.020461400970816612,
+            0.002838419284671545,
+            -0.0009453330421820283,
+            -0.0036380558740347624,
+            0.0027048238553106785,
+            -0.005882072262465954,
+            -0.005018542055040598,
+            -0.014753584749996662,
+            -0.012739972211420536,
+            0.0101300198584795,
+            0.00802347157150507,
+            0.014195968396961689,
+            -0.011950016021728516,
+            0.0028190575540065765,
+            0.018788553774356842,
+            0.007690451107919216,
+            -0.023357905447483063,
+            0.01135367713868618,
+            -0.010912231169641018,
+            -0.009270362555980682,
+            -0.009433000348508358,
+            -0.009471723809838295,
+            0.0016980175860226154,
+            -0.008317768573760986,
+            0.003945906180888414,
+            -0.03615209087729454,
+            -0.011330443434417248,
+            -0.0013901671627536416,
+            -0.010346870869398117,
+            0.0009864765452221036,
+            0.02524760365486145,
+            0.006815304048359394,
+            0.01432762760668993,
+            -0.0006733016925863922,
+            0.017502939328551292,
+            -0.0018984107300639153,
+            0.018726596608757973,
+            0.012763205915689468,
+            -0.00797700323164463,
+            0.011717676185071468,
+            0.014738094992935658,
+            -0.0033863543067127466,
+            0.015620986931025982,
+            -0.010168743319809437,
+            0.005285732913762331,
+            -0.009556914679706097,
+            -0.003214035416021943,
+            0.002495717955753207,
+            0.030901208519935608,
+            -0.003618694143369794,
+            0.00858883187174797,
+            0.01098193321377039,
+            -0.015241498127579689,
+            0.0026951429899781942,
+            0.014962689951062202,
+            0.01141563430428505,
+            0.007760153152048588,
+            0.0024434414226561785,
+            -0.024225307628512383,
+            0.004193735308945179,
+            0.005037903785705566,
+            0.0031520782504230738,
+            -0.014366351068019867,
+            0.0021859314292669296,
+            0.015675200149416924,
+            0.008550108410418034,
+            -0.025325050577521324,
+            0.016418687999248505,
+            0.0009138703462667763,
+            -0.003909118939191103,
+            0.029119934886693954,
+            -0.011268485337495804,
+            -0.005723306443542242,
+            0.009463978931307793,
+            0.0009980935137718916,
+            0.0030029932968318462,
+            0.008937342092394829,
+            0.0228312686085701,
+            -0.003988501615822315,
+            -0.032341714948415756,
+            -0.022459525614976883,
+            0.015706177800893784,
+            -0.0047823297791182995,
+            -0.022939693182706833,
+            -0.026703599840402603,
+            -0.003113354789093137,
+            -0.02228914201259613,
+            -0.0022323993034660816,
+            -0.020058678463101387,
+            -0.014056564308702946,
+            0.001268188701942563,
+            0.029987338930368423,
+            0.016201836988329887,
+            0.008224832825362682,
+            -0.011051635257899761,
+            -0.029956359416246414,
+            -0.024271776899695396,
+            0.01810702309012413,
+            -0.007055388763546944,
+            -0.005541306920349598,
+            0.00591692328453064,
+            -0.009951893240213394,
+            -0.008480406366288662,
+            0.011802867986261845,
+            0.027137301862239838,
+            -0.002747419523075223,
+            -0.029987338930368423,
+            -0.02688947319984436,
+            -0.014335372485220432,
+            -0.01840132102370262,
+            -0.017704300582408905,
+            -0.008627555333077908,
+            0.0045035220682621,
+            0.00030300996149890125,
+            0.016186347231268883,
+            0.014962689951062202,
+            -0.000480653194244951,
+            0.0028597170021384954,
+            -2.7711375878425315e-05,
+            -0.001763847190886736,
+            0.006888878531754017,
+            -0.004933350719511509,
+            0.026997897773981094,
+            0.014614180661737919,
+            -0.006594581063836813,
+            0.018711106851696968,
+            0.028314489871263504,
+            0.020430423319339752,
+            0.003157886676490307,
+            -0.010695381090044975,
+            -0.014606435783207417,
+            0.009727297350764275,
+            -0.013421501964330673,
+            0.023264970630407333,
+            0.022552460432052612,
+            -0.01480779703706503,
+            -0.014885243959724903,
+            -0.017905661836266518,
+            0.015210519544780254,
+            -0.007562664337456226,
+            -0.008224832825362682,
+            0.012368228286504745,
+            0.022335609421133995,
+            -0.007024410180747509,
+            0.002371803391724825,
+            -0.0010232636705040932,
+            -0.021731525659561157,
+            -0.006122156511992216,
+            -0.004596457816660404,
+            -0.008666278794407845,
+            0.008681767620146275,
+            0.008131897076964378,
+            -0.001928421319462359,
+            -0.006319645792245865,
+            0.005471604876220226,
+            -0.013367289677262306,
+            0.01301103550940752,
+            -0.00789568480104208,
+            0.03379771113395691,
+            -0.009138703346252441,
+            -0.0006733016925863922,
+            -0.022598929703235626,
+            -0.011446612887084484,
+            0.00845717266201973,
+            0.0017967619933187962,
+            0.01682141050696373,
+            -0.0011626676423475146,
+            -0.005688455421477556,
+            0.009913169778883457,
+            -0.0010687635513022542,
+            -0.001450188341550529,
+            -0.006892750971019268,
+            -0.010401083156466484,
+            0.0002884887217078358,
+            0.013258864171802998,
+            -0.006025348324328661,
+            0.0018054747488349676,
+            0.006095050368458033,
+            -0.0175339188426733,
+            0.029027000069618225,
+            0.023884542286396027,
+            0.017255110666155815,
+            0.014660648070275784,
+            0.0006394187803380191,
+            0.011647974140942097,
+            -0.003572226269170642,
+            0.0038820127956569195,
+            -0.020337486639618874,
+            0.021080974489450455,
+            -0.010811550542712212,
+            0.01846327818930149,
+            0.011655719019472599,
+            -0.015233753249049187,
+            -0.005022414494305849,
+            0.021406250074505806,
+            0.009967382065951824,
+            0.00889087375253439,
+            0.0017541663255542517,
+            -0.02470547705888748,
+            0.017766257748007774,
+            0.010099041275680065,
+            -0.020399443805217743,
+            0.019547531381249428,
+            -0.009471723809838295,
+            0.01233724970370531,
+            -0.0005300253978930414,
+            0.004418330732733011,
+            0.0058743273839354515,
+            0.021499186754226685,
+            0.011570527218282223,
+            -0.03884723410010338,
+            -0.013615119270980358,
+            -0.0017357728211209178,
+            0.006857899948954582,
+            -0.0036303112283349037,
+            -0.015233753249049187,
+            -0.041635312139987946,
+            0.01604694314301014,
+            0.009998360648751259,
+            -0.01643417589366436,
+            0.004987563472241163,
+            -0.002427952131256461,
+            -0.0042053526267409325,
+            -0.008170620538294315,
+            0.003186929039657116,
+            -0.004170501604676247,
+            0.02324948087334633,
+            0.0018984107300639153,
+            -0.006946963258087635,
+            0.013258864171802998,
+            0.01344473659992218,
+            -0.021948376670479774,
+            0.0009874446550384164,
+            0.0021375271026045084,
+            0.000636998622212559,
+            0.023683181032538414,
+            -0.014707116410136223,
+            -0.0017657832941040397,
+            -0.01685238815844059,
+            -0.0013069120468571782,
+            -0.021592123433947563,
+            -0.001188805908896029,
+            -0.015187285840511322,
+            -0.008511385880410671,
+            0.025030752643942833,
+            -0.011005166918039322,
+            0.010997422970831394,
+            -0.006687517277896404,
+            0.006943091284483671,
+            -0.006904367823153734,
+            -0.01936165988445282,
+            -0.0008267428493127227,
+            0.009975126944482327,
+            -0.000512115890160203,
+            0.002941036131232977,
+            8.482827252009884e-05,
+            0.002464739140123129,
+            0.008867640048265457,
+            0.006234454456716776,
+            -0.0075278133153915405,
+            0.02489134855568409,
+            -0.012073930352926254,
+            0.0016399325104430318,
+            0.011911292560398579,
+            0.0004743606550619006,
+            -0.010424317792057991,
+            0.025588368996977806,
+            -0.02139076218008995,
+            -0.00800798274576664,
+            -0.00014799568452872336,
+            -0.0016544538084417582,
+            -0.006513262167572975,
+            0.006718495860695839,
+            0.02946070022881031,
+            0.0062654330395162106,
+            0.010447551496326923,
+            -0.0281441081315279,
+            -0.011493081226944923,
+            0.012538610957562923,
+            -0.022273652255535126,
+            -0.0007739823195151985,
+            -0.0004322490422055125,
+            -0.00619960343465209,
+            0.026378324255347252,
+            -0.02022906206548214,
+            -0.01093546487390995,
+            -0.018664639443159103,
+            0.005405775271356106,
+            -0.011121337302029133,
+            0.008294534869492054,
+            -0.0009777637897059321,
+            0.006540368776768446,
+            0.016805920749902725,
+            -0.019563021138310432,
+            0.016031453385949135,
+            0.005119222681969404,
+            -0.016217326745390892,
+            0.010222956538200378,
+            -0.017580386251211166,
+            -0.0008199662552215159,
+            0.020275529474020004,
+            -0.00400399137288332,
+            0.008751469664275646,
+            -0.003363120136782527,
+            0.005208286456763744,
+            -0.006513262167572975,
+            0.0001492057926952839,
+            -0.007249005138874054,
+            0.014428308233618736,
+            -0.016805920749902725,
+            -0.017828216776251793,
+            -0.008821171708405018,
+            0.0016709112096577883,
+            0.021297825500369072,
+            0.011136826127767563,
+            -0.010044828988611698,
+            -0.011059380136430264,
+            -0.011694442480802536,
+            0.041604332625865936,
+            0.005762029904872179,
+            0.027230238541960716,
+            -0.02261441759765148,
+            0.00545998802408576,
+            0.025170156732201576,
+            -0.014242436736822128,
+            0.0012788375606760383,
+            -0.0022788674104958773,
+            0.0053050946444272995,
+            0.011562783271074295,
+            -0.01819995976984501,
+            0.0049139889888465405,
+            -0.00019216447253711522,
+            -0.00782985519617796,
+            -0.000656844291370362,
+            0.003442503046244383,
+            0.00020196630794089288,
+            0.016325751319527626,
+            -0.004027225077152252,
+            -0.016511622816324234,
+            -0.004429947584867477,
+            -0.0003216455806978047,
+            -0.027896279469132423,
+            0.013514438644051552,
+            0.016589069738984108,
+            0.01566745527088642,
+            0.027989214286208153,
+            -0.014513500034809113,
+            0.016031453385949135,
+            0.009533680975437164,
+            0.013901671394705772,
+            -0.010610189288854599,
+            0.008952830918133259,
+            -0.01595400646328926,
+            0.007446494419127703,
+            -0.03454120084643364,
+            -0.007887939922511578,
+            -0.00802347157150507,
+            0.01984182931482792,
+            0.028763681650161743,
+            0.002923610620200634,
+            0.006493900436908007,
+            0.000196883876924403,
+            0.0024202074855566025,
+            -0.01714668609201908,
+            -0.020182594656944275,
+            -0.0003337466041557491,
+            0.0029332914855331182,
+            0.0006950835813768208,
+            -0.004495777189731598,
+            0.00472037261351943,
+            -0.012391461990773678,
+            0.0026835259050130844,
+            -0.013134949840605259,
+            0.005971135571599007,
+            -0.0059982421807944775,
+            0.009603383019566536,
+            0.0019429425010457635,
+            0.020864123478531837,
+            0.02300165221095085,
+            0.004100799560546875,
+            0.0046971384435892105,
+            -0.024333734065294266,
+            0.00920066051185131,
+            -0.013847459107637405,
+            0.012329504825174809,
+            0.019408127292990685,
+            0.008248066529631615,
+            0.006571347359567881,
+            0.01233724970370531,
+            0.012918098829686642,
+            -0.0031772484071552753,
+            0.024287264794111252,
+            0.005014669615775347,
+            -0.0009070937521755695,
+            0.002052335999906063,
+            -0.014954946003854275,
+            -0.0070979841984808445,
+            -0.0015286030247807503,
+            0.0018848575418815017,
+            -0.004731989465653896,
+            0.01906736195087433,
+            0.015256987884640694,
+            -0.01050176378339529,
+            0.017874684184789658,
+            -0.03937387093901634,
+            0.017379024997353554,
+            -0.0027377386577427387,
+            -0.0060911779291927814,
+            -0.00545998802408576,
+            0.00665653869509697,
+            0.004515138920396566,
+            0.003669034456834197,
+            0.03773200139403343,
+            -0.010408828034996986,
+            0.003963331691920757,
+            -0.005847221240401268,
+            0.007121218368411064,
+            -0.020864123478531837,
+            0.008302279748022556,
+            0.005336073227226734,
+            0.02160761132836342,
+            -0.0056497324258089066,
+            -0.0014482521219179034,
+            0.02328045852482319,
+            0.008859895169734955,
+            0.03469609469175339,
+            -0.01566745527088642,
+            -0.004418330732733011,
+            0.01048627495765686,
+            0.0207402091473341,
+            -0.00580462533980608,
+            -0.008976065553724766,
+            0.007903429679572582,
+            0.0035586729645729065,
+            0.021824462339282036,
+            -0.00515407370403409,
+            -0.0006650729919783771,
+            0.0054367538541555405,
+            -0.0060911779291927814,
+            0.009735042229294777,
+            0.011252996511757374,
+            -0.009076746180653572,
+            0.007539430167526007,
+            0.021530164405703545,
+            -0.009975126944482327,
+            -0.008658533915877342,
+            -0.009649851359426975,
+            0.0010658593382686377,
+            -0.03068435750901699,
+            0.001649613375775516,
+            0.005359307397156954,
+            0.015187285840511322,
+            -0.0004603234410751611,
+            -0.009014789015054703,
+            0.021019017323851585,
+            0.017255110666155815,
+            -0.005936284549534321,
+            -0.0028132491279393435,
+            0.0023795480374246836,
+            -0.0017154429806396365,
+            -0.006160879973322153,
+            -0.0006248975405469537,
+            -0.002656419761478901,
+            0.014730350114405155,
+            -0.0015469965292140841,
+            -0.006617815233767033,
+            -0.00756653631106019,
+            0.0033302053343504667,
+            0.010834784246981144,
+            -0.005661349277943373,
+            0.013057502917945385,
+            0.016805920749902725,
+            -0.00031704717548564076,
+            0.019671445712447166,
+            0.01037010457366705,
+            -0.010447551496326923,
+            0.015938518568873405,
+            -0.007984748110175133,
+            0.006563602481037378,
+            0.0164031982421875,
+            -0.01277095079421997,
+            -0.01246116403490305,
+            0.007210282143205404,
+            0.00665653869509697,
+            0.028918573632836342,
+            0.014792308211326599,
+            -0.02109646424651146,
+            0.014908477663993835,
+            0.015768134966492653,
+            -0.013018779456615448,
+            0.003537375247105956,
+            -0.01784370467066765,
+            0.020260039716959,
+            0.019051872193813324,
+            -0.015535795129835606,
+            -0.017224133014678955,
+            -0.014947201125323772,
+            -0.005092116538435221,
+            0.006331262644380331,
+            0.0033263331279158592,
+            -0.00617636926472187,
+            -0.027989214286208153,
+            -0.0049139889888465405,
+            -0.018633661791682243,
+            0.013870692811906338,
+            -0.008674023672938347,
+            0.0007715621031820774,
+            -0.004511266481131315,
+            0.007984748110175133,
+            -0.0033882902935147285,
+            0.032465629279613495,
+            -0.0051811798475682735,
+            -0.02832997962832451,
+            0.004398969002068043,
+            0.016356728971004486,
+            0.012623801827430725,
+            -0.029243851080536842,
+            0.010463040322065353,
+            0.004805563949048519,
+            -0.012530866079032421,
+            0.005990497302263975,
+            -0.02258343994617462,
+            -0.007632365915924311,
+            -0.0009395244996994734,
+            -0.016743963584303856,
+            -0.027121812105178833,
+            -0.008433938957750797,
+            -0.006993431597948074,
+            -0.005401902832090855,
+            -0.009525936096906662,
+            -0.008720491081476212,
+            -0.011477591469883919,
+            0.02303262986242771,
+            -0.007899557240307331,
+            -0.001866463921032846,
+            -0.0014637414133176208,
+            0.00930908601731062,
+            -0.023776117712259293,
+            -0.015605497173964977,
+            0.006389347370713949,
+            -0.002594462363049388,
+            0.022815778851509094,
+            -0.012747716158628464,
+            0.005940156988799572,
+            -0.009316830895841122,
+            0.009851212613284588,
+            0.015241498127579689,
+            0.004956584889441729,
+            0.017440982162952423,
+            -0.009889936074614525,
+            0.0021026763133704662,
+            -0.008627555333077908,
+            -0.00018986526993103325,
+            -1.027075541060185e-05,
+            0.014428308233618736,
+            0.007624621503055096,
+            0.014141756109893322,
+            0.017487451434135437,
+            -0.013553161174058914,
+            0.005003052763640881,
+            -0.0003274540649726987,
+            -0.017255110666155815,
+            0.0027725896798074245,
+            -0.007984748110175133,
+            -0.0008896682411432266,
+            0.001758038648404181,
+            6.395397940650582e-05,
+            -0.02112744189798832,
+            -0.0009124181815423071,
+            0.0018809852190315723,
+            0.0010222955606877804,
+            -0.025201136246323586,
+            0.00889087375253439,
+            -0.00889087375253439,
+            -0.0014918158994987607,
+            -0.019113829359412193,
+            0.019795360043644905,
+            0.026300877332687378,
+            -0.004619691986590624,
+            0.00036206303047947586,
+            -0.013537672348320484,
+            -0.02049238048493862,
+            0.0001489637652412057,
+            -0.011314953677356243,
+            -0.011345932260155678,
+            -0.0022169100120663643,
+            -0.003214035416021943,
+            0.008805682882666588,
+            0.0004189379105810076,
+            -0.009433000348508358,
+            -0.004956584889441729,
+            -0.011764144524931908,
+            -0.017549408599734306,
+            -0.010401083156466484,
+            0.01933068037033081,
+            -0.0018016024259850383,
+            0.0020794421434402466,
+            -0.0032508226577192545,
+            0.0005271211848594248,
+            0.011748654767870903,
+            0.013475715182721615,
+            0.013344055972993374,
+            -0.023915521800518036,
+            -0.017890173941850662,
+            -0.013320821337401867,
+            0.0029487807769328356,
+            -0.007187047973275185,
+            0.004844287410378456,
+            0.026362836360931396,
+            -0.0009695350890979171,
+            -0.01091997604817152,
+            0.002327271504327655,
+            0.019051872193813324,
+            -0.031272951513528824,
+            -0.013390523381531239,
+            -0.025077221915125847,
+            0.02225816436111927,
+            -0.013351799920201302,
+            0.011516314931213856,
+            0.010191977955400944,
+            -0.025944624096155167,
+            0.006768836174160242,
+            -0.007686578668653965,
+            0.026037558913230896,
+            0.01634124107658863,
+            0.0005934348446317017,
+            -0.00944074522703886,
+            0.013622863218188286,
+            -0.0008722427301108837,
+            -0.013576395809650421,
+            0.002201420720666647,
+            0.014645159244537354,
+            -0.014691627584397793,
+            -0.009510447271168232,
+            -0.004507394507527351,
+            0.004313777666538954,
+            0.012763205915689468,
+            -0.0030339721124619246,
+            -0.004031097516417503,
+            -0.011624740436673164,
+            0.013344055972993374,
+            -0.003930416889488697,
+            0.012383717112243176,
+            0.007380664348602295,
+            -0.014056564308702946,
+            -0.019532041624188423,
+            0.033333033323287964,
+            0.00400399137288332,
+            0.00338054564781487,
+            -0.010997422970831394,
+            0.02740062028169632,
+            -0.009897680021822453,
+            0.010571465827524662,
+            -0.006408709101378918,
+            0.0031985461246222258,
+            0.012321759946644306,
+            -0.012104909867048264,
+            -0.01981084980070591,
+            0.021468207240104675,
+            -0.00364192808046937,
+            -0.0010193913476541638,
+            0.008310024626553059,
+            -0.020182594656944275,
+            -0.00663330452516675,
+            0.002180122770369053,
+            0.009541425853967667,
+            -0.011818356812000275,
+            0.006369986105710268,
+            0.017905661836266518,
+            0.003912991378456354,
+            -0.0007672057254239917,
+            0.019996721297502518,
+            -0.01672847382724285,
+            0.011717676185071468,
+            -0.010850274004042149,
+            0.0016825281782075763,
+            0.0006065039779059589,
+            -0.0048597767017781734,
+            0.0058743273839354515,
+            -0.014536733739078045,
+            0.011988739483058453,
+            -6.244135147426277e-05,
+            -0.013979118317365646,
+            0.007601387333124876,
+            -0.013359544798731804,
+            -0.016836898401379585,
+            0.041542377322912216,
+            -0.005301222205162048,
+            -0.0038665232714265585,
+            -0.0056962003000080585,
+            0.01350669376552105,
+            0.02823704481124878,
+            -0.005336073227226734,
+            -0.01153180468827486,
+            -0.010780571959912777,
+            -0.016573579981923103,
+            0.015907539054751396,
+            -0.01810702309012413,
+            0.002495717955753207,
+            -0.010532742366194725,
+            -0.007903429679572582,
+            0.004302160814404488,
+            -0.003516077296808362,
+            0.014722606167197227,
+            0.00950270239263773,
+            0.023357905447483063,
+            0.00797700323164463,
+            -0.00027348342700861394,
+            0.010393339209258556,
+            0.011663463898003101,
+            0.003243078012019396,
+            -0.010509508661925793,
+            -0.00011949774489039555,
+            -0.02894955314695835,
+            -0.008449427783489227,
+            -0.004863648675382137,
+            -0.001485039247199893,
+            -0.00428667152300477,
+            -0.01239920686930418,
+            0.006617815233767033,
+            -0.0022052929271012545,
+            0.013909416273236275,
+            -0.009254872798919678,
+            -0.017766257748007774,
+            0.03389064967632294,
+            0.004495777189731598,
+            0.006466794293373823,
+            -0.008286789990961552,
+            -0.03705047070980072,
+            -0.0032566310837864876,
+            -0.008116407319903374,
+            -0.013529927469789982,
+            0.004538373090326786,
+            0.01283290795981884,
+            -0.014831030741333961,
+            -0.028097640722990036,
+            0.0019390701781958342,
+            -0.006683644838631153,
+            0.02978597767651081,
+            -0.007798876613378525,
+            0.00419760774821043,
+            0.0037251831963658333,
+            0.0011558911064639688,
+            0.01239920686930418,
+            -0.00845717266201973,
+            -0.0022536972537636757,
+            0.016387708485126495,
+            -0.010858018882572651,
+            0.011051635257899761,
+            -0.010966443456709385,
+            -0.02554190158843994,
+            -0.01456771232187748,
+            0.021344292908906937,
+            -0.0004823473282158375,
+            -0.00024903935263864696,
+            0.014823286794126034,
+            0.02354377694427967,
+            0.011260741390287876,
+            0.00969631876796484,
+            0.006408709101378918,
+            -0.001035848748870194,
+            0.005975008010864258,
+            0.006095050368458033,
+            -0.0024163350462913513,
+            -0.013398268260061741,
+            0.017564896494150162,
+            0.009773765690624714,
+            -0.01810702309012413,
+            0.0008514289511367679,
+            0.0036612898111343384,
+            0.0168833676725626,
+            0.006935346405953169,
+            0.009316830895841122,
+            -0.0143121387809515,
+            -0.023760627955198288,
+            -0.00014932679187040776,
+            0.002873270306736231,
+            0.015721667557954788,
+            0.010579210706055164,
+            -0.014319883659482002,
+            0.0005963391158729792,
+            0.008712747134268284,
+            0.020213572308421135,
+            -0.024209819734096527,
+            0.019702425226569176,
+            -0.004673904739320278,
+            -0.0021084847394376993,
+            0.007946024648845196,
+            0.011632485315203667,
+            -0.011872569099068642,
+            -0.00650551775470376,
+            0.006249943748116493,
+            0.003157886676490307,
+            0.03172214329242706,
+            -0.0017260919557884336,
+            0.009665340185165405,
+            -0.010997422970831394,
+            -0.0002405686245765537,
+            0.0006728176376782358,
+            -0.0006820144481025636,
+            -0.002884887158870697,
+            -0.011020656675100327,
+            0.0018974426202476025,
+            -0.020306508988142014,
+            0.0024473138619214296,
+            0.00463130883872509,
+            -0.005684582982212305,
+            -0.0032972905319184065,
+            -0.019222255796194077,
+            -0.0022672503255307674,
+            0.003293418325483799,
+            0.014335372485220432,
+            -0.031691163778305054,
+            0.004972074180841446,
+            0.010672146454453468,
+            -0.002309845993295312,
+            -0.010416572913527489,
+            0.011423379182815552,
+            -0.0026060794480144978,
+            0.011988739483058453,
+            -0.0038142469711601734,
+            -0.0004905760288238525,
+            0.025681305676698685,
+            -0.005308967083692551,
+            0.014707116410136223,
+            -0.01026167906820774,
+            -0.002993312431499362,
+            -0.028252532705664635,
+            0.005235392600297928,
+            -0.007403898518532515,
+            -0.0012081675231456757,
+            -0.004402841441333294,
+            -0.01130720879882574,
+            -0.0026990154292434454,
+            0.002122037811204791,
+            0.008000237867236137,
+            0.012910354882478714,
+            -0.0007725302129983902,
+            -0.010075807571411133,
+            0.004832670092582703,
+            0.0075355577282607555,
+            -0.015520306304097176,
+            -0.0070437719114124775,
+            -0.020306508988142014,
+            0.029662061482667923,
+            -0.013731288723647594,
+            -0.0011123273288831115,
+            -0.020864123478531837,
+            0.010160998441278934,
+            -0.0063119009137153625,
+            -0.002782270545139909,
+            0.004708755761384964,
+            -0.015109838917851448,
+            0.01344473659992218,
+            0.008751469664275646,
+            0.01450575515627861,
+            -0.02151467651128769,
+            -0.01762685552239418,
+            -0.0044725434854626656,
+            -0.002735802438110113,
+            -0.010648912750184536,
+            -0.0005334137240424752,
+            0.002013612538576126,
+            0.021251358091831207,
+            -0.00032503384863957763,
+            0.0042053526267409325,
+            -0.003214035416021943,
+            0.017224133014678955,
+            -0.010137764737010002,
+            -0.013057502917945385,
+            0.0024260159116238356,
+            0.005994369741529226,
+            0.016186347231268883,
+            0.03158273920416832,
+            -0.020693741738796234,
+            -0.00272999401204288,
+            0.0032740565948188305,
+            0.010656657628715038,
+            -0.004085310269147158,
+            6.304640555754304e-05,
+            -0.027803342789411545,
+            -0.04464024305343628,
+            -0.002422143705189228,
+            0.013057502917945385,
+            -0.015636475756764412,
+            0.012670270167291164,
+            0.008743725717067719,
+            0.010021595284342766,
+            -0.005196669604629278,
+            0.02620794251561165,
+            -0.004793947096914053,
+            0.014304393902420998,
+            0.011733165942132473,
+            0.011911292560398579,
+            0.0009874446550384164,
+            0.0007459079497493804,
+            -0.018354853615164757,
+            -0.008255811408162117,
+            -0.0015954007394611835,
+            0.01765783317387104,
+            -0.027478067204356194,
+            0.008526874706149101,
+            -0.011516314931213856,
+            0.013839714229106903,
+            0.0003666614356916398,
+            0.005661349277943373,
+            -0.01771979033946991,
+            0.012445674277842045,
+            -0.0009337160154245794,
+            -0.025325050577521324,
+            0.004805563949048519,
+            0.011276230216026306,
+            -0.010362359695136547,
+            0.006877261213958263,
+            0.019950253888964653,
+            0.0023079097736626863,
+            0.008495896123349667,
+            -0.014126266352832317,
+            0.010703125037252903,
+            0.009649851359426975,
+            -0.021080974489450455,
+            -0.01233724970370531,
+            -0.007059261202812195,
+            -0.006335135083645582,
+            -0.0035470561124384403,
+            -0.00013105422840453684,
+            0.010176488198339939,
+            -0.016000475734472275,
+            0.010099041275680065,
+            -0.006559730041772127,
+            0.006400964688509703,
+            0.00944074522703886,
+            0.001115231541916728,
+            -0.013576395809650421,
+            0.012716737575829029,
+            -0.0218709297478199,
+            0.008426194079220295,
+            -0.04287445917725563,
+            -0.005847221240401268,
+            -0.015078860335052013,
+            0.005204414017498493,
+            -0.003798757679760456,
+            -0.004453181754797697,
+            -0.006203475873917341,
+            -0.005061137955635786,
+            0.00920066051185131,
+            0.021747015416622162,
+            0.012135888449847698,
+            -0.016620049253106117,
+            -0.007551047019660473,
+            0.005425137002021074,
+            0.0004854935978073627,
+            -0.013336311094462872,
+            0.009316830895841122,
+            -0.01576039008796215,
+            0.0009259713697247207,
+            0.006339007057249546,
+            -0.01072635967284441,
+            0.013452480547130108,
+            0.004731989465653896,
+            -0.003401843598112464,
+            0.013390523381531239,
+            -0.0050417762249708176,
+            -0.0057891360484063625,
+            -0.007272239308804274,
+            -0.013739033602178097,
+            -0.003818119177594781,
+            -0.003072695340961218,
+            0.012747716158628464,
+            0.001529571134597063,
+            -0.0003243077953811735,
+            -0.00782985519617796,
+            0.009154192171990871,
+            0.002220782218500972,
+            -0.0013146566925570369,
+            0.005773646757006645,
+            -0.02300165221095085,
+            0.0001287550403503701,
+            0.004755223635584116,
+            -0.0009487213101238012,
+            -0.006246071308851242,
+            -0.012151377275586128,
+            -0.011903548613190651,
+            -0.038165703415870667,
+            0.009363298304378986,
+            0.0019197085639461875,
+            -0.0076207490637898445,
+            0.0064822835847735405,
+            0.017255110666155815,
+            0.01456771232187748,
+            0.016743963584303856,
+            -0.020693741738796234,
+            -0.02303262986242771,
+            0.005556796211749315,
+            -0.009402021765708923,
+            0.02261441759765148,
+            0.004046586807817221,
+            -0.005847221240401268,
+            0.0013243375578895211,
+            -0.013258864171802998,
+            0.008929597213864326,
+            0.005723306443542242,
+            -0.001592496526427567,
+            0.005425137002021074,
+            -0.013692565262317657,
+            -0.010904486291110516,
+            0.01048627495765686,
+            0.032496608793735504,
+            -0.0004261985304765403,
+            0.0024105266202241182,
+            0.023295948281884193,
+            0.015466093085706234,
+            0.016356728971004486,
+            -0.021111954003572464,
+            0.011167804710566998,
+            0.010060317814350128,
+            -0.0018645278178155422,
+            -0.014892988838255405,
+            0.00513083953410387,
+            -0.012716737575829029,
+            -0.010664401575922966,
+            0.014993669465184212,
+            0.007853088900446892,
+            0.017735280096530914,
+            -0.02369867078959942,
+            0.00789568480104208,
+            0.00042765066609717906,
+            0.00013710475468542427,
+            0.011524059809744358,
+            0.012956822291016579,
+            0.03370477631688118,
+            0.027152791619300842,
+            0.020662762224674225,
+            -0.009285852313041687,
+            0.014366351068019867,
+            -0.009192915633320808,
+            -0.011090358719229698,
+            -0.019470084458589554,
+            -0.015249243006110191,
+            -0.011136826127767563,
+            0.017998598515987396,
+            0.008790193125605583,
+            -0.009355553425848484,
+            -0.005947901867330074,
+            0.01586107164621353,
+            -0.003909118939191103,
+            -0.004635181277990341,
+            0.01646515540778637,
+            -0.026269899681210518,
+            0.016836898401379585,
+            -0.000745423894841224,
+            -0.014668392948806286,
+            -0.014691627584397793,
+            -0.004182118456810713,
+            -0.012670270167291164,
+            0.0003964299685321748,
+            -0.020817656069993973,
+            0.013181418180465698,
+            0.024597052484750748,
+            -0.0030920570716261864,
+            -0.004100799560546875,
+            -0.011299464851617813,
+            -0.0005111477803438902,
+            0.018385831266641617,
+            0.0034773540683090687,
+            0.009192915633320808,
+            -0.0023872926831245422,
+            0.013924905098974705,
+            0.00524313747882843,
+            0.0027958236169070005,
+            -0.0007565568666905165,
+            -0.016263794153928757,
+            0.012197845615446568,
+            0.0026835259050130844,
+            0.012135888449847698,
+            -0.022242674604058266,
+            0.007938280701637268,
+            -0.011764144524931908,
+            -0.0023698671720921993,
+            -0.02228914201259613,
+            0.0189744271337986,
+            -0.01610890030860901,
+            0.001046497724018991,
+            0.003841353114694357,
+            0.0006224773242138326,
+            0.009580149315297604,
+            0.01610890030860901,
+            -0.023388884961605072,
+            -0.0003460896550677717,
+            -0.014908477663993835,
+            -0.021917399019002914,
+            -0.01607792265713215,
+            0.004728117026388645,
+            -0.019501063972711563,
+            0.005444498732686043,
+            -0.0025557391345500946,
+            -0.015125328674912453,
+            0.018478767946362495,
+            0.014691627584397793,
+            -0.004379607271403074,
+            0.002412462839856744,
+            0.005425137002021074,
+            -0.0014405074762180448,
+            0.01971791312098503,
+            0.00013274837692733854,
+            -0.0010048701660707593,
+            -0.009425255469977856,
+            -0.0168833676725626,
+            -0.0065752193331718445,
+            -0.035005878657102585,
+            0.0260530486702919,
+            -0.004670032300055027,
+            -0.016511622816324234,
+            0.014319883659482002,
+            0.009115469641983509,
+            -0.014629669487476349,
+            -0.007059261202812195,
+            0.016294771805405617,
+            -0.024287264794111252,
+            0.02244403585791588,
+            -0.009611127898097038,
+            -0.008774704299867153,
+            0.00784921646118164,
+            0.05154848098754883,
+            0.009146448224782944,
+            0.003492843359708786,
+            0.011036145500838757,
+            0.005839476361870766,
+            -0.0013165927957743406,
+            -0.023683181032538414,
+            0.005568413063883781,
+            -0.007190920412540436,
+            -0.003601268632337451,
+            0.002536377403885126,
+            -0.004801691509783268,
+            0.031149037182331085,
+            -0.004379607271403074,
+            -0.005289605353027582,
+            0.004484160337597132,
+            -0.02196386642754078,
+            -0.0001332324172835797,
+            0.023203013464808464,
+            -2.3128106477088295e-05,
+            -0.023590246215462685,
+            -0.006571347359567881,
+            -0.015071115456521511,
+            -0.013770012184977531,
+            -0.018989915028214455,
+            0.005231520161032677,
+            -0.012755461037158966,
+            -0.0033940989524126053,
+            0.0016660707769915462,
+            -0.005940156988799572,
+            -0.002582845278084278,
+            0.009603383019566536,
+            0.05362405255436897,
+            0.014103032648563385,
+            -0.006335135083645582,
+            -0.005854965653270483,
+            0.009192915633320808,
+            -0.010524998418986797,
+            -0.019795360043644905,
+            -0.009820234030485153,
+            -0.029507169499993324,
+            -0.012747716158628464,
+            -0.0143121387809515,
+            -0.010191977955400944,
+            -0.010277168825268745,
+            0.00963436160236597,
+            -0.018122512847185135,
+            -0.007628493942320347,
+            -0.020337486639618874,
+            0.002941036131232977,
+            -0.011268485337495804,
+            -0.014714861288666725,
+            -0.008054450154304504,
+            -0.04807887226343155,
+            0.005796880926936865,
+            0.019593998789787292,
+            0.01263929158449173,
+            0.002515079453587532,
+            0.004383479710668325,
+            -0.01993476413190365,
+            -0.030792783945798874,
+            0.013839714229106903,
+            0.011283975094556808,
+            0.011733165942132473,
+            0.03283737599849701,
+            -0.011709931306540966,
+            -0.00813964195549488,
+            -0.004693266469985247,
+            -0.0017251238459721208,
+            0.012066186405718327,
+            0.00969631876796484,
+            -0.004917861428111792,
+            -0.012375972233712673,
+            0.016217326745390892,
+            -0.01981084980070591,
+            -0.023683181032538414,
+            -0.01672847382724285,
+            0.006157007534056902,
+            0.00813964195549488,
+            -0.008054450154304504,
+            0.002973950933665037,
+            0.0056962003000080585,
+            -0.007159941829741001,
+            -0.00022423222253564745,
+            0.004890755284577608,
+            -0.023048119619488716,
+            0.0008490087930113077,
+            -0.002257569460198283,
+            0.0061376458033919334,
+            0.021266845986247063,
+            -0.010168743319809437,
+            -0.0014075926737859845,
+            -0.0053902859799563885,
+            -0.0028984404634684324,
+            -0.006764963734894991,
+            0.02620794251561165,
+            0.009154192171990871,
+            -0.009882191196084023,
+            -0.01011453103274107,
+            -0.01129171997308731,
+            -0.0011278166202828288,
+            -0.017255110666155815,
+            -0.016186347231268883,
+            0.0015566773945465684,
+            -0.016712984070181847,
+            0.013971373438835144,
+            -0.00876695942133665,
+            0.00400399137288332,
+            -0.014110777527093887,
+            0.02019808255136013,
+            0.01929970271885395,
+            -0.009711808525025845,
+            0.023776117712259293,
+            -0.007709812838584185,
+            -0.038382552564144135,
+            -0.00800798274576664,
+            0.01843230053782463,
+            -0.006118284538388252,
+            0.010037084110081196,
+            -0.011314953677356243,
+            0.008209343999624252,
+            0.00015319912927225232,
+            0.0018122512847185135,
+            -0.0003112386621069163,
+            0.027555514127016068,
+            0.015535795129835606,
+            -0.005560668651014566,
+            -0.005494839046150446,
+            0.004255692940205336,
+            0.009541425853967667,
+            0.017580386251211166,
+            0.012151377275586128,
+            -0.021019017323851585,
+            -0.014257925562560558,
+            -0.016263794153928757,
+            0.011012911796569824,
+            0.015125328674912453,
+            -0.0040891822427511215,
+            0.01691434532403946,
+            0.014126266352832317,
+            -0.019098341464996338,
+            0.0019884423818439245,
+            -0.008193854242563248,
+            -0.00158184755127877,
+            -0.03029712475836277,
+            0.011043890379369259,
+            0.0074387495405972,
+            -0.0072218989953398705,
+            0.036616768687963486,
+            0.009858956560492516,
+            0.037577107548713684,
+            0.018680129200220108,
+            0.0008974128868430853,
+            0.009045767597854137,
+            -0.018370341509580612,
+            0.013065247796475887,
+            0.01500141341239214,
+            -0.012724482454359531,
+            -0.005734923295676708,
+            0.0013446672819554806,
+            0.0007551047019660473,
+            0.012631546705961227,
+            0.02004319056868553,
+            -0.01795213110744953,
+            0.009773765690624714,
+            -0.015032391995191574,
+            -0.005250881891697645,
+            0.006555858068168163,
+            -0.013932649977505207,
+            -0.008495896123349667,
+            -0.00963436160236597,
+            0.019408127292990685,
+            -0.012600568123161793,
+            0.016573579981923103,
+            -0.009595638141036034,
+            0.015520306304097176,
+            -0.029724018648266792,
+            -0.003669034456834197,
+            -0.006679772399365902,
+            -0.019593998789787292,
+            -0.012887120246887207,
+            0.009820234030485153,
+            0.010633422993123531,
+            -0.009301341138780117,
+            -0.011562783271074295,
+            0.020368466153740883,
+            -0.004185990896075964,
+            0.013529927469789982,
+            0.021189400926232338,
+            0.00737291993573308,
+            0.024132372811436653,
+            0.0005605200421996415,
+            0.014164989814162254,
+            0.005463860463351011,
+            0.010362359695136547,
+            -0.004677776712924242,
+            0.005161818582564592,
+            0.00932457484304905,
+            0.014203713275492191,
+            0.017766257748007774,
+            0.013196907006204128,
+            -0.00598275288939476,
+            0.007419387809932232,
+            0.02016710489988327,
+            0.0016254113288596272,
+            -0.030312614515423775,
+            -0.009719553403556347,
+            0.0001736498816171661,
+            0.00924712885171175,
+            0.009185170754790306,
+            3.817877222900279e-05,
+            -0.01093546487390995,
+            0.011144571006298065,
+            0.00969631876796484,
+            0.002286612056195736,
+            0.006280922330915928,
+            -0.01456771232187748,
+            -0.020957060158252716,
+            0.006106667220592499,
+            0.01798310875892639,
+            0.013917161151766777,
+            -0.014722606167197227,
+            0.022784801200032234,
+            0.006145390681922436,
+            -0.021406250074505806,
+            -0.01840132102370262,
+            -0.0009559819591231644,
+            0.0004542729293461889,
+            -0.002265314105898142,
+            0.03720536455512047,
+            -0.0011539548868313432,
+            -0.005955646280199289,
+            -0.0054754773154854774,
+            0.02775687538087368,
+            0.02049238048493862,
+            0.009177426807582378
+          ],
+          "embedding": [
+            0.0008215984936325879,
+            -0.019349418284688353,
+            -0.02322660633029795,
+            0.009747820845194852,
+            0.007353771220903641,
+            0.015252908783199839,
+            -0.012176175829980202,
+            0.051898051094966605,
+            -0.0068130353886146165,
+            0.008189212816890595,
+            0.008102287831931905,
+            0.01682801136011614,
+            0.02005557996049711,
+            -0.020779879580497183,
+            0.002332165268329781,
+            0.02337485065616377,
+            0.006074992895540845,
+            0.02576575146535689,
+            0.0007517286429767504,
+            -0.03763909942314134,
+            0.0011204729720314833,
+            0.00041212613589632806,
+            -0.06089176800314795,
+            0.012852035873395388,
+            -0.008901127092058543,
+            0.01702347055484216,
+            0.005803446395466281,
+            0.017889570275475296,
+            -0.05375047382413552,
+            0.021623778774252724,
+            0.014784317465883424,
+            -0.0018881880857885708,
+            0.0004735328554290247,
+            0.012393802850407656,
+            -0.019234442968055952,
+            -0.019566283611638122,
+            0.04407789373296535,
+            -0.0053565873209102995,
+            -0.026240866282480922,
+            0.048107902270426675,
+            -0.0054681314939387235,
+            -0.0009441397948041965,
+            -0.015205079626883548,
+            0.006885981841893229,
+            0.01403724256866746,
+            -0.026086548246792754,
+            0.0034778946051893067,
+            0.00044587533948775473,
+            0.032851328435891634,
+            -0.03882097333255334,
+            -0.011126344604254894,
+            -0.01648943336384904,
+            0.035810456555802166,
+            -0.019384589526165754,
+            -0.019430702768170285,
+            0.014297440272239852,
+            -0.007883453204663151,
+            0.050996900793086844,
+            0.06007149434153715,
+            0.03406832053483379,
+            0.025193439934671852,
+            0.00027785129663604897,
+            -0.008014391056168083,
+            -0.018806730908490114,
+            0.027193788903098558,
+            -0.006195006670328595,
+            -0.020466000344352826,
+            0.0068939884219589185,
+            0.017252276167086602,
+            -0.045073444648704136,
+            -0.049242269917908754,
+            0.023021140866008313,
+            -0.015676872472071106,
+            -0.024497517908898602,
+            0.005873633718973869,
+            -0.022828585938203683,
+            0.011748341004940436,
+            0.04624205727927451,
+            -0.015442165499674641,
+            0.018565551071491767,
+            0.015246505395507096,
+            0.004119269796224144,
+            0.004862735250819252,
+            -0.000145323321960046,
+            0.05155838952887077,
+            0.0027052423345323936,
+            -0.016218063670586664,
+            -0.07462470920944148,
+            0.043695974746295135,
+            -0.027608273720377708,
+            -0.03860912774695494,
+            0.002823110565150435,
+            0.026471469849508766,
+            -0.0026063577189074103,
+            -0.029230652014151735,
+            0.05684164470477843,
+            0.00022182868300572677,
+            0.015590845552956364,
+            -0.024939617549529242,
+            0.01035803286203917,
+            -0.010225298012384327,
+            0.019571847648759157,
+            0.01841613041441054,
+            0.0156981470538354,
+            0.021764297694793343,
+            -0.018831230007033416,
+            -0.014419529638789993,
+            -0.0029965475899652273,
+            0.028881185012095884,
+            0.09163526460163937,
+            0.022644302207811818,
+            -0.017933909936330133,
+            -0.04663928709016869,
+            0.004962951444105128,
+            -0.041333041174526026,
+            -0.027545628007617527,
+            -0.037142394579894954,
+            -0.013358143162788402,
+            0.02719686432396677,
+            -0.0034971800937504205,
+            0.04413635817371732,
+            0.04903857110328917,
+            0.03186067155630788,
+            0.007831051050269414,
+            -0.025257538623313574,
+            0.023517513773109414,
+            -0.0016527403209505964,
+            -0.0016101860707429347,
+            0.014448515185903624,
+            0.009905018312477684,
+            0.0024114582235163955,
+            0.00172043851384822,
+            -0.03818316221102173,
+            -0.02304537008110578,
+            0.01024969233570089,
+            -0.01901116356327339,
+            0.019791121460848986,
+            0.07985067340821365,
+            -0.051903191272236665,
+            -0.025924646976556942,
+            0.014766511532849252,
+            -0.002073086076894999,
+            0.030959269926800852,
+            0.006394417986617089,
+            -0.028177586693713442,
+            0.010725285336674526,
+            0.0009614052276924192,
+            0.07500253788978564,
+            -0.00036794182149960157,
+            0.004058458332826519,
+            -0.013218269166951234,
+            0.027613539268683245,
+            -0.012995170442480582,
+            0.023388707018488407,
+            0.026389423313105165,
+            -0.003068639956061266,
+            -0.02661967649632026,
+            0.026145274570635913,
+            -0.03300591322174331,
+            -0.004556072066733024,
+            0.020045722084112896,
+            0.0033437343914784865,
+            -0.010224220905483085,
+            -0.0025025498684079703,
+            0.03460103245395426,
+            0.04514362357278504,
+            0.03797031988058962,
+            -0.004586662475576956,
+            -0.06351723722596923,
+            0.06037983484369874,
+            0.019415753124363902,
+            0.028850618647994476,
+            -0.0034810237045849006,
+            0.02957938154808314,
+            0.02642652523231673,
+            0.013187653724133275,
+            -0.007856320804109569,
+            -0.007065882659482447,
+            -0.016672784625197085,
+            0.029020161154209707,
+            -0.013243058921970134,
+            -0.014900932834241732,
+            0.006001769901896033,
+            0.008648456186944271,
+            0.03484763413464497,
+            0.012279368984306564,
+            0.02379643508687133,
+            0.04933044959399941,
+            -0.021047723935647383,
+            -0.007264823701706426,
+            0.0007058774795716792,
+            -0.010512071832195085,
+            0.027489671668749157,
+            0.009731157570361245,
+            -0.03195919640758579,
+            -0.0033626031972691915,
+            -0.04282419484651207,
+            0.01873977910485268,
+            -0.014398285316695653,
+            0.012854631447494207,
+            -0.0017372959978092432,
+            -0.037331867825735776,
+            0.001088209292238142,
+            -0.012345684323522151,
+            0.037096063326414734,
+            0.005050219690024652,
+            0.0008906051515353004,
+            -0.003697709449630797,
+            0.006548479142737107,
+            -0.010727663712299397,
+            -0.007486226350610755,
+            -0.02704618481069485,
+            -0.019752552016312926,
+            -0.014632581296303486,
+            -0.0037501138119724717,
+            0.014588411795703245,
+            -0.007119137071719376,
+            0.016440258608998535,
+            -0.03686645352111013,
+            -0.023546402448713415,
+            0.005628703432152666,
+            0.024988145751166143,
+            0.011971199595808573,
+            -0.010868778329190568,
+            -0.03270046433624425,
+            -0.021950559285357267,
+            -0.011889914022657738,
+            -0.005123971523701009,
+            0.02619713302280345,
+            -0.021990810305904974,
+            0.01165257474485436,
+            -0.009117901804315863,
+            -0.007492082755249399,
+            0.010725902070291232,
+            0.01862811996181951,
+            -0.026694281868322685,
+            -0.01623776415010199,
+            -0.009403997538684658,
+            -0.02207329027097839,
+            0.025939540497196416,
+            0.014178803271471762,
+            -0.001141821318145744,
+            -0.006281187017652108,
+            -0.035993365198554964,
+            0.0026192776111449633,
+            -0.015254976771874396,
+            -0.0116872453009606,
+            -0.015780016491993135,
+            0.021421328283124968,
+            0.02889432250124136,
+            -0.01871009416296692,
+            -0.026540472640514536,
+            0.015703370682110186,
+            -0.0002849235245164438,
+            -0.02594930889995505,
+            0.04491828744809679,
+            0.044063522197410734,
+            -0.037441864491547405,
+            -0.007005339187980476,
+            -0.030381794393931814,
+            0.06901083845539985,
+            -0.017604082400603037,
+            -0.04060591381445593,
+            0.06044850935418443,
+            -0.01756490171206357,
+            0.025539717592705295,
+            0.03452018789549939,
+            0.0016217577522523927,
+            0.00018825054264059277,
+            -0.002563606092068712,
+            0.04770277039047993,
+            -0.0032010143255574255,
+            -0.03425328601475849,
+            -0.02105909767072799,
+            -0.00840881022563103,
+            -0.028699447677549002,
+            -0.015925160861589006,
+            -0.0055750742281166,
+            0.017513987082479483,
+            0.010399718005952185,
+            0.02267201252273823,
+            0.01258511878601025,
+            0.0015081978484714903,
+            0.01874981572091931,
+            0.011158889421057756,
+            -0.002607810686437211,
+            -0.003732023411827874,
+            -0.007787189972165309,
+            -0.05524012485122728,
+            -0.025110689689279583,
+            -0.03946383236703947,
+            0.004443572529325475,
+            -0.018745077598477586,
+            0.005926160888274727,
+            0.039430331582190524,
+            -0.0049500324516682,
+            0.004352795787899792,
+            0.015566139991780173,
+            0.013148156356408315,
+            -0.015279560337042242,
+            -0.03316846614871165,
+            0.0001592209771257332,
+            0.018362411161743347,
+            -0.012706582418607377,
+            -0.03424477956681259,
+            -0.001991022841226589,
+            0.0033467286047492175,
+            0.03357019284647071,
+            -0.024359110796680374,
+            0.00359117029951049,
+            0.005061818038334802,
+            0.011359582931760315,
+            -0.026793992746244076,
+            -0.018170136079817746,
+            -0.017421603453826397,
+            0.03996608926471779,
+            0.02587013058343799,
+            0.015763410145648683,
+            -0.04512536449983358,
+            -0.02293473890636517,
+            0.048575083783480195,
+            -0.009589860756342604,
+            -0.06486262416944208,
+            0.03447666009581266,
+            0.016105654870253654,
+            0.003526582643722421,
+            -0.01354754623151814,
+            0.05401575248587799,
+            -0.03563190295559527,
+            -0.017365166993914202,
+            -0.0046288709698227735,
+            -0.033657212157212645,
+            -0.007686252059880836,
+            -0.001385273642139782,
+            0.048229062789530236,
+            -0.029083855744728694,
+            -0.030465559177859865,
+            0.0024368661249980416,
+            -0.024144031154493057,
+            0.021693278579520123,
+            -0.016405088055634124,
+            -0.018439935250344958,
+            0.006948603359995389,
+            -0.033943037395199695,
+            -0.003007796423804453,
+            -0.016783195426536633,
+            -0.02270589491206155,
+            0.020874438753828814,
+            0.016724666903417947,
+            0.027033184180344852,
+            0.0019847647852695785,
+            0.017749524526371133,
+            0.0204661893602285,
+            -0.005604929968008821,
+            0.028462234633072944,
+            -0.020790459496727066,
+            -0.04183816355941877,
+            0.03946487892663695,
+            0.01192861758684541,
+            -0.0018024841275380242,
+            0.01077082591611702,
+            -0.01475239020295694,
+            -0.03952356245715099,
+            0.03683950018557526,
+            -0.056767672022551764,
+            -0.007507600090017693,
+            0.009530544352246083,
+            -0.00788856051420074,
+            -0.011204694027562521,
+            -0.010587701466318598,
+            -0.028604940377315504,
+            0.023749537635431462,
+            -0.013672419168235691,
+            0.01953449229707597,
+            -0.011449939491554354,
+            -0.012238180529977605,
+            0.05795536247491174,
+            -0.021518581774137322,
+            -0.040672955795925836,
+            0.026066659238628543,
+            -0.014061106927896723,
+            0.03328357230532424,
+            0.03957107406164427,
+            0.030670947051333917,
+            -0.004278577327978564,
+            0.025885946540364826,
+            0.0319106101668716,
+            -0.003462160036198642,
+            -0.04686868330851868,
+            0.00555303933159587,
+            0.02156143971867743,
+            -0.03739529489075782,
+            -0.01025626126223168,
+            -0.02141970191890338,
+            0.022134910980548407,
+            0.017807484142010496,
+            -0.007232643256393455,
+            0.011318041192005274,
+            -0.010489012862455913,
+            -0.013003217163997554,
+            -0.00847279880392288,
+            0.02357274819910594,
+            -0.0016485066139744577,
+            -0.005873113634992993,
+            0.006424249429163587,
+            0.004008703814301658,
+            0.0054639858216457724,
+            0.010198079500408441,
+            0.017694720520798957,
+            -0.017087021465208255,
+            -0.0025558869389290554,
+            -0.016503579212015177,
+            -0.0420082404922411,
+            -0.003563173424359332,
+            0.022924161700566468,
+            0.012345933179990547,
+            -0.00729908668365632,
+            -0.004234683369642856,
+            -0.0022962545569187934,
+            -0.020666205489143392,
+            -0.014796149043465308,
+            0.024562653223873056,
+            0.030217494971086438,
+            0.007810961433007407,
+            0.007781263640449456,
+            -0.0030348559388780655,
+            0.007310876971216526,
+            0.016099591640649783,
+            0.02311929597780174,
+            -0.04160162038353416,
+            0.027152100845897007,
+            -0.00025396315502635146,
+            0.023717123370023598,
+            -0.04186918759097667,
+            -0.017988531857836703,
+            -0.03106663560912813,
+            -0.023595079699181436,
+            0.02437134274097885,
+            -0.02317559303912261,
+            -0.012327263775257306,
+            0.000989566600374836,
+            0.00954921216572729,
+            -0.005155079395685017,
+            -0.017508404061379164,
+            -0.04582321461886958,
+            -0.0014367264231457263,
+            -0.030417229142190933,
+            -0.017829710557332023,
+            -0.001056472801554354,
+            0.005016912084754242,
+            -0.06276644565618253,
+            0.0109965681725148,
+            -0.07270669271977365,
+            -0.004542796125637244,
+            0.040139998232966566,
+            -0.017890441783630643,
+            0.055909102081208445,
+            -0.05029104308417623,
+            0.01673962668706085,
+            -0.04507293803055318,
+            0.015385506084493484,
+            0.025677372277539627,
+            0.030235171586316813,
+            -0.022189978375852563,
+            0.008461188938299745,
+            0.008033707483259569,
+            -0.045089505755429415,
+            -0.04144698743458133,
+            -0.013640256640978024,
+            0.01036782197518379,
+            0.0012696958402094232,
+            -0.016090728932978384,
+            0.023838437877040163,
+            0.011946482002767135,
+            0.03342604921859589,
+            0.04211801335157251,
+            0.002708015395637621,
+            -0.047156646610976634,
+            -0.013104464812562618,
+            -0.003933116891120681,
+            0.03155321708116329,
+            -0.018866109579446184,
+            0.015376812010630058,
+            -0.01955557376085726,
+            0.016255217383093355,
+            0.017868332102274922,
+            -0.020325179506828335,
+            0.013434364256590097,
+            0.004176904701252156,
+            -0.03877060944639854,
+            0.0006475542749369515,
+            0.00042468300294954125,
+            -0.02116992978038693,
+            0.0032654387271108293,
+            -0.00575003786159731,
+            -0.05149954071159492,
+            -0.031382926943128234,
+            -0.015730563640253065,
+            0.0382312502769352,
+            0.004297939848910064,
+            0.019724983545557365,
+            -0.017907869879607064,
+            -0.005165628493894463,
+            0.03698976707579373,
+            0.008501001373266799,
+            -0.01725527821298066,
+            0.006879620077191753,
+            0.02455792200929138,
+            -0.03224193972301503,
+            -0.039181328884413266,
+            -0.0008206986106407542,
+            -0.023306758893533952,
+            0.01672946960876128,
+            -0.02658587698869307,
+            0.005873620623736886,
+            0.0025331170979801232,
+            -0.03490809296591325,
+            0.02456764314036355,
+            0.003291113462353163,
+            -0.004303423672882112,
+            -0.020359544091059775,
+            -0.016892415165120203,
+            0.017174797487220006,
+            0.04391489511478065,
+            0.007327321684948843,
+            -0.003943713832544148,
+            0.00862684231313155,
+            -0.006361172785753631,
+            0.05826688726534012,
+            -0.02284956969636637,
+            -0.026680023999604643,
+            0.027527436347931256,
+            -0.022597669279069723,
+            -0.006277863878949772,
+            -0.017343446085471857,
+            -0.012065192152203348,
+            -0.0004183712490442143,
+            0.019339311388446943,
+            -0.019423562920897294,
+            -0.010212527447243049,
+            0.004783006556748757,
+            0.012824133870627983,
+            -0.004855945108275706,
+            0.006239949458562236,
+            0.006847581690471813,
+            0.03094005096378827,
+            0.026993131409761086,
+            -0.011331881651539968,
+            0.03297014291876675,
+            0.010265600643742144,
+            4.666113318251278e-05,
+            0.001970879281172325,
+            0.002523664335535017,
+            0.0034107811520368574,
+            -0.03143890864030425,
+            -0.01624584278940617,
+            0.013468729048885906,
+            0.005233309338223403,
+            -0.004550544967856448,
+            -0.023783653541634683,
+            -0.009402494239255264,
+            0.03334923712760375,
+            -0.003260424721959948,
+            0.02748395114281811,
+            -0.0058374897745283065,
+            -0.01985840930273332,
+            0.017595418271610163,
+            0.02881743246631328,
+            -0.05928552440605384,
+            0.017062540894682354,
+            -0.0011508547522925793,
+            0.008953761538519997,
+            0.01610561914042553,
+            -0.008894160513648953,
+            0.013301991014533684,
+            0.04080863883897929,
+            0.010738051773755976,
+            -0.023345002072886145,
+            0.004215726310838674,
+            0.01541749448570608,
+            0.01291813299324752,
+            -0.022747942296528025,
+            0.0008640791861357522,
+            0.011008522066277018,
+            0.005893793007012561,
+            0.0026106450193140156,
+            -0.006278394080583205,
+            -0.025659188287146268,
+            0.024180550333596625,
+            -0.0044491292740074925,
+            0.026967973346698617,
+            0.04390909273429254,
+            0.009596855239692042,
+            -0.03686020608700941,
+            -0.013109951664921729,
+            0.0002799680359670127,
+            -0.0038537577602593366,
+            -0.0037552094562213916,
+            0.011105295862811114,
+            -0.04185615640149602,
+            0.019950890919369135,
+            0.006453241355231516,
+            0.015314645427201533,
+            -0.020031193483734364,
+            -0.024530173488182337,
+            0.0031949828520724337,
+            -0.01492645564652531,
+            0.012875827757040941,
+            0.005057170413681063,
+            -0.004319834710426384,
+            0.005386862713494232,
+            0.00030848054013413157,
+            0.008384941028326983,
+            -0.010811526993301901,
+            0.03175068297908541,
+            0.004680987487229685,
+            0.009264547472281849,
+            -0.02856314241700808,
+            -0.021083721348429348,
+            0.020113396771629096,
+            0.03572366771051171,
+            -0.01708529918264109,
+            0.03346921724975398,
+            -0.009042046660366718,
+            0.0033219155387035227,
+            0.0151720168318272,
+            -0.015805318210244414,
+            -0.008057236707694303,
+            -0.0005483009439015931,
+            -0.002917527004423299,
+            -0.014538511124345527,
+            0.007302979537905532,
+            -0.0208529958115422,
+            -0.02965227242147972,
+            0.020064178921373875,
+            -0.00011892622932349096,
+            0.016455188521316146,
+            0.010335044657747514,
+            -0.007816155992852236,
+            -0.0052262330850905165,
+            0.006959344748883874,
+            0.04432147882994085,
+            0.026734484553994495,
+            0.05547661378166294,
+            -0.005787997952888261,
+            -0.004559444854138438,
+            -0.0082295581123363,
+            -0.006588172755707144,
+            0.04524009609473585,
+            0.015915003261638767,
+            -0.004910285095602864,
+            0.017495607570191165,
+            0.01557886571266685,
+            0.03858259260136037,
+            0.0027648525902558316,
+            -0.016912796336835408,
+            0.04014755669075221,
+            -0.004077405782937414,
+            0.008616827127381418,
+            0.011102692199778469,
+            0.0024185742218798562,
+            0.00807141794538637,
+            0.02996274720815659,
+            -0.012429186019907761,
+            0.01680942269506771,
+            -0.04571916592313758,
+            -0.02775530246832996,
+            0.014370280587419414,
+            -0.004931905540824582,
+            0.03419959665162407,
+            0.01668521021731861,
+            -0.023105305446763272,
+            -0.024761385385514877,
+            0.021529428247380984,
+            -0.002603042335561961,
+            -0.024135326707477325,
+            -0.015906833215844473,
+            0.003286605488908195,
+            -0.0023437997044735617,
+            0.0060251848371355166,
+            -0.013481652076194643,
+            0.0002908426939460218,
+            0.01953362396253743,
+            0.0038544889405965057,
+            0.020760460182873287,
+            -0.016224396121239458,
+            -0.00259233446682179,
+            -0.009282861237653619,
+            -0.02952130765234087,
+            0.017988370145301916,
+            0.006138236653154911,
+            -0.007607182955958238,
+            -0.009516767050028553,
+            0.0048347778733230365,
+            -0.005819209642901871,
+            -0.0025851701689263706,
+            -0.01073014664868326,
+            0.012433465684360061,
+            0.025784014942833097,
+            -0.016875624785552303,
+            0.007944349714123104,
+            8.651589872318479e-05,
+            -0.022306044741692753,
+            -0.022127016745066307,
+            0.020143472307319017,
+            0.002792646249762062,
+            0.008768813346304779,
+            -0.004074214276300631,
+            -0.00015024341785520542,
+            0.002990550758627699,
+            0.01951746553838439,
+            0.03896451381932467,
+            0.004232673242619471,
+            0.027876586543280867,
+            0.0145860303663199,
+            0.03994234200854897,
+            0.009657017763624678,
+            0.00696607158576712,
+            -0.026055387694656488,
+            0.02819890357011778,
+            0.00018297502510753014,
+            0.043190622821775236,
+            0.022488402221902314,
+            -0.0407730101161195,
+            -0.0021946518523759066,
+            0.01669520746276111,
+            0.013212623257794862,
+            0.017009844136059122,
+            -0.004537512994041156,
+            -0.014041213424740049,
+            0.028135404528975894,
+            -0.022859586430516003,
+            0.0008637006862481223,
+            -0.004486332934188017,
+            -0.025031795510001887,
+            0.007351370254175368,
+            -0.004567816916440959,
+            0.030025413528233364,
+            0.0014976740344892735,
+            -0.022285096307825804,
+            0.022837170656498543,
+            -0.009551269792616045,
+            0.011045530488684207,
+            0.008809047185288755,
+            0.020065001354321994,
+            0.031993362733068575,
+            0.009143210434415064,
+            0.02133640814803879,
+            0.012241777474889248,
+            0.014771538127546474,
+            -0.008804192497858033,
+            -0.02341427260922065,
+            0.023167424359504862,
+            -0.0011598649565273595,
+            -0.029627604902846688,
+            0.0021991474725787303,
+            0.003261878079344105,
+            -0.0005935713331682358,
+            -0.009144689470851402,
+            -0.005580557862988033,
+            -0.007240836451875213,
+            0.019631377011298905,
+            -0.007085213886352692,
+            0.007017573363709754,
+            -0.004074171791476129,
+            -0.00822871835665378,
+            -0.0421101573155962,
+            0.004210914075286222,
+            0.003408427700622687,
+            -0.01752179330665445,
+            0.006384440404174512,
+            0.011121481025573962,
+            0.043020575455262985,
+            -0.022809979913698254,
+            -0.0005737138409049003,
+            0.019206623212867476,
+            0.0009562551889820241,
+            -0.023427467810802533,
+            -0.038931688376706186,
+            0.007172345672967499,
+            -0.03467050750542982,
+            0.023185301620612612,
+            -0.0007075596339842658,
+            0.02387977204565927,
+            -0.03043468414449851,
+            0.013909304572370437,
+            0.004784668158421625,
+            0.012524879174804915,
+            0.021654512058639427,
+            0.03142146922870715,
+            -0.02053473475540882,
+            -0.050409862638621725,
+            -0.005406667316025549,
+            0.00035138762584085976,
+            0.022018487016014856,
+            -0.026014580970974616,
+            0.00025633004699302783,
+            -0.003349094027291921,
+            -0.02401366769212778,
+            -0.01735996745614562,
+            0.01962340791446025,
+            -0.050011480783403936,
+            0.015042913576492952,
+            0.0262737424555697,
+            -0.02507473901367252,
+            -0.05408889321034101,
+            -0.018988587082325706,
+            -0.028217893770659686,
+            -0.009285770094319318,
+            -0.00661159230987788,
+            0.028597405154740078,
+            0.003525576256054959,
+            -0.011462219358952466,
+            0.025401668390029348,
+            0.006155837251830549,
+            0.0026050693705924593,
+            0.015985722392086222,
+            0.007289910674212967,
+            -0.011452511453623235,
+            -0.014599577191293152,
+            -0.04558584556100621,
+            0.010197536827229795,
+            -0.0045431962637690155,
+            -0.019372598050508005,
+            -0.030147301333718016,
+            -0.0137828704002156,
+            -0.027060515748528666,
+            -0.0017291966950107505,
+            0.053783684195381654,
+            0.03292494290301388,
+            0.005560172296221419,
+            -0.0026085670565688026,
+            -0.010466890489356092,
+            0.036247483457995895,
+            -0.006789746401727701,
+            0.03662690472838588,
+            0.001784522073210422,
+            -0.0242673253380092,
+            0.01169382884548301,
+            0.02892627181542449,
+            0.009858451662166387,
+            -0.0005592283645355213,
+            -0.006160132931009508,
+            -0.008782830470079547,
+            0.0031207630972208394,
+            0.025613233596604418,
+            0.003268170392795548,
+            -0.010062991425127038,
+            0.018106967407336815,
+            0.004480291699976396,
+            0.02010374270557836,
+            0.03555357587371324,
+            0.023489837757983054,
+            -0.021782173652026497,
+            -0.026581762796336082,
+            -0.0013561486781077014,
+            0.0072266106510934505,
+            0.03551967295942857,
+            -0.01651315514034218,
+            0.0001274396319698452,
+            -0.008272735507120127,
+            -0.00894931703949637,
+            0.011149952898504355,
+            -0.0029671593121986116,
+            -0.0030752010285951576,
+            0.01662711184497578,
+            -0.0011672263615879709,
+            -0.033448222499147764,
+            0.0078288560210653,
+            -0.023496331346917084,
+            0.0051040606404415545,
+            -0.004113104164416575,
+            -0.007076525381595441,
+            -0.002157548492138289,
+            0.006302962169276006,
+            -0.008065841090139872,
+            -0.00035242603988955366,
+            -0.01586406637763306,
+            0.008271894439780969,
+            0.007438312999409211,
+            -0.0069617481460406095,
+            0.011402932583388205,
+            0.002666895909532698,
+            0.0005219881149353855,
+            -0.012159425670629751,
+            -0.012042145306711322,
+            -0.00957276588761916,
+            -0.03401863879157551,
+            -0.02604353712472079,
+            -0.024949533021906844,
+            -0.0014209149651479556,
+            -0.006566685925666972,
+            -0.03187796739029736,
+            -0.01209152137926458,
+            -0.011110763124407752,
+            0.007756961097537267,
+            -0.013260548545917543,
+            0.005433285079114047,
+            -6.399098520041351e-05,
+            -0.0044960488826674555,
+            0.016615636438440516,
+            -0.003643291599919672,
+            -0.019488010211707343,
+            -0.03927818878149417,
+            0.008209336177358258,
+            -0.02805593125673831,
+            -0.0038816771550190788,
+            0.007607119219721711,
+            -0.00528681677614909,
+            -0.029502897838743516,
+            -0.009149794737430706,
+            0.008767001199311985,
+            0.003507217008280066,
+            -0.01773349756494436,
+            0.03579670773605949,
+            0.026971617790626075,
+            0.011875179879209696,
+            -0.019891342676443626,
+            0.0508929813275565,
+            -0.0047010735337056275,
+            -0.012544935684845649,
+            0.018297503869683027,
+            0.02355022268131519,
+            0.03273167043015095,
+            -0.024830243320788693,
+            0.0009370814985214882,
+            -0.021937793653233454,
+            0.00535662615596573,
+            -0.035009915581788176,
+            0.0028595659314725012,
+            -0.014395027018699446,
+            0.015596459769560532,
+            0.03397372881171701,
+            -0.013254226072981457,
+            0.011195747181764677,
+            -0.016299384375589934,
+            0.00730244731485402,
+            0.00569944090461748,
+            -0.012916284280345044,
+            -0.005812311882399539,
+            -0.022902815152116493,
+            0.020140276669979015,
+            0.005809799950176385,
+            -0.0008130655714656685,
+            0.0104085826652653,
+            -0.044723432238900435,
+            0.03286429597337671,
+            0.008730225944501239,
+            0.017599189944921396,
+            0.0029762446259779998,
+            -0.0003068671951232415,
+            0.011611551606224214,
+            0.011228401175698422,
+            -0.016029727902740706,
+            0.004345913009102136,
+            -0.004613061137378433,
+            -0.03850533909474302,
+            0.009363351681220008,
+            -0.03631184057978666,
+            0.014427784677340159,
+            0.016026872002233575,
+            -0.004170582842643461,
+            -0.025166189571409366,
+            -0.005428471012333673,
+            0.010827910811085384,
+            -0.009783069664327864,
+            -0.009987805234276443,
+            -0.04163358183060102,
+            -0.025236440364963034,
+            0.034097508113604576,
+            -0.002119474613596975,
+            -0.025826403511707147,
+            -0.010830738112107807,
+            -0.011351132636491575,
+            0.020207687645133,
+            -0.006542026283946858,
+            -0.0006263975804936843,
+            -0.04150011545311013,
+            -0.0073396962917864515,
+            0.011192225907726928,
+            0.027746213794734322,
+            -0.035970017302005566,
+            0.02708126943501373,
+            0.002105633542220187,
+            0.014262674845755191,
+            -0.01626664983003942,
+            -0.005631805361903626,
+            -0.013990539604372344,
+            -0.035877093361375685,
+            0.016487723013906077,
+            -0.008607758248055054,
+            0.004714436224135091,
+            0.04458009880881818,
+            -0.0036872068506712727,
+            -0.009466125348499464,
+            0.007600705817195143,
+            -0.0034590883553048063,
+            -0.010351576085287273,
+            -0.0028564505793999295,
+            0.001959661298507,
+            -0.007080697863126256,
+            -0.04183476693971616,
+            -0.00499888976530217,
+            0.04684391782633148,
+            0.006361744381879857,
+            0.013276793765092465,
+            0.013516220521120684,
+            -0.018335746557932003,
+            -0.045577810515431215,
+            -0.0071616111758423025,
+            -0.0032763880176386808,
+            -0.022708028392758567,
+            -0.0240284945459861,
+            -0.008735040856657758,
+            0.005844145402902923,
+            0.006359368165686724,
+            -0.01120763599008222,
+            -0.022676131361171057,
+            -0.027085968700452136,
+            -0.010526103176253857,
+            -0.02123944716136792,
+            -0.010792145796733375,
+            -0.027142270190771887,
+            -0.028742234538796804,
+            0.02880426953386151,
+            0.007575182232859564,
+            -0.004084088301176117,
+            -0.010005765476121966,
+            -0.01255640546813973,
+            -0.0004774761617900028,
+            -0.0025307994917461232,
+            -0.005822065709999144,
+            0.0038614286313282164,
+            -0.011390482620811868,
+            0.023152086963976554,
+            0.0021220701260546895,
+            0.018329162172128138,
+            -0.05772324571231113,
+            -0.002709083851090071,
+            0.021497896395795905,
+            0.0033025410488410913,
+            -0.005412445141910314,
+            0.029352650459792132,
+            0.011223947499453582,
+            -0.006539460282439711,
+            -0.031355428613718234,
+            0.020942976176820262,
+            0.024122993515785947,
+            0.00817615634503138,
+            -0.012994160510030828,
+            0.01221095914384204,
+            0.01629778384790159,
+            -0.004494985018246821,
+            -0.011161098655072933,
+            0.005586857967039181,
+            0.020941039642372112,
+            0.013580622629911925,
+            0.0013398071867077221,
+            0.030905022925366163,
+            0.0006862084391508121,
+            -0.03521231612862609,
+            -0.01908191995763771,
+            -0.01275995164030872,
+            -0.003584331305986364,
+            -0.015264425022763126,
+            0.005294329654809353,
+            -0.002355465010876059,
+            0.006335052149475905,
+            0.022655379849785256,
+            0.03593231939948758,
+            0.0146937743203089,
+            0.00937330328448886,
+            0.003167462724350819,
+            -0.0013235176101076284,
+            0.012102872918129403,
+            0.008638011773075119,
+            0.0350701240268404,
+            -0.014653221697693864,
+            -0.011003994230902835,
+            -0.0037767288519369537,
+            0.010100131061186566,
+            -0.004113527518662366,
+            0.020414393256136372,
+            0.03274709615649469,
+            -0.0037222949335916876,
+            -0.005604552180246868,
+            -0.006160118231975786,
+            0.0041467454414452565,
+            0.01807879127455274,
+            -0.01252134965321004,
+            0.004364858987501461,
+            0.005549300665159658,
+            -0.0071897162833861,
+            0.018567059255813558,
+            0.0002622074720860951,
+            -0.012983550838417323,
+            0.03266376564402191,
+            0.11449509387240629,
+            0.0017251916835047628,
+            0.007125215166916139,
+            -0.040229997815548584,
+            -0.020537987527148888,
+            -0.00245728735683473,
+            0.005779972828836593,
+            0.007004732428236457,
+            0.02225914982152608,
+            0.006505827715591914,
+            0.0006708045351142873,
+            0.031708522931495536,
+            -0.017334083511709092,
+            -0.002522861743921599,
+            -0.01745656559688129,
+            0.013890451996979048,
+            -0.014312605700272691,
+            -0.013784963572164409,
+            -0.01572417853955075,
+            0.004507425871748977,
+            -0.000601158156253668,
+            0.0016158265715634729,
+            0.01791303234734911,
+            0.0011926917387973374,
+            -0.026393351252156246,
+            0.0006694008039871878,
+            0.001026031515214626,
+            -0.004514419527575117,
+            -0.011320929719992686,
+            -0.02223001723942815,
+            0.016235841442653918,
+            0.014188311619681008,
+            -0.004895218832118094,
+            0.009244697376301004,
+            0.010558796153959677,
+            0.02707485418319703,
+            -0.028380342401606758,
+            0.0040204430418629405,
+            0.007023041225985294,
+            0.01606556581447933,
+            0.017617862031190843,
+            0.0034870847111552862,
+            -0.011253315705053224,
+            0.03238289399766116,
+            -0.010563282687352038,
+            0.0065093536270233385,
+            0.014966802208644462,
+            -0.03541611540945151,
+            -5.821682910348165e-05,
+            0.0061985444938402745,
+            -0.011716201661363997,
+            0.009580035961315467,
+            -0.03120692534097433,
+            -0.0039291243987978845,
+            0.009317468331974374,
+            0.017611312558635673,
+            0.008804996982574765,
+            -0.0034129169537151925,
+            0.010822226698822738,
+            0.012510661458054741,
+            -0.0027147975086409966,
+            -0.008855213644027594,
+            0.01238502686635956,
+            -0.0007687053446954701,
+            0.00450108863711973,
+            -0.00799170686059026,
+            0.0029755595812552293,
+            0.002290650917244691,
+            -0.01554856300008412,
+            -0.011779430686394945,
+            0.00438393126679289,
+            0.0036931891075561957,
+            0.001535662168147833,
+            0.012014408636737167,
+            -0.006087255045485002,
+            -0.0077816220325470226,
+            0.0040573724156794284,
+            0.003136943055301706,
+            -5.505014316363865e-05,
+            -0.030625245024561508,
+            0.01585923084309149,
+            0.0067595435845396405,
+            0.010013308598626031,
+            -0.005315626260228517,
+            -0.00903142414091241,
+            -0.007240920015018893,
+            -0.01228521172101564,
+            -0.010657894223173453,
+            0.007658605592936815,
+            -0.00724924993432376,
+            0.011124174211132038,
+            -0.03128027787357322,
+            -0.013988715435985316,
+            0.01231792993245938,
+            -0.01528161242159377,
+            0.0007675821323979293,
+            -0.008389914312201402,
+            0.0009417242487395995,
+            -0.005119304379761705,
+            0.007887436211143328,
+            -0.03731503169547854,
+            0.007878001990271541,
+            0.02456170736765724,
+            -0.00330924625934981,
+            -0.020259771366167557,
+            -0.009613930457931786,
+            0.0241314385136084,
+            -0.008267831641886344,
+            -0.01645888287387047,
+            0.008421304862524506,
+            0.010140813149093993,
+            -0.0006309118552875393,
+            0.0008237784055549769,
+            -0.003726984191590917,
+            0.01669125388798323,
+            0.02054777616420997,
+            0.041930770964107515,
+            -0.014789610432004559,
+            0.01092349703703152,
+            0.02866938974792608,
+            -0.03378576980722847,
+            0.014728113846952417,
+            0.02518409645155824,
+            0.014590531327650454,
+            -0.0004446956315190408,
+            0.001609777186372437,
+            0.0068939321280085475,
+            -0.014273492115311677,
+            0.009247676157551494,
+            0.013380104440744367,
+            -0.018407746260686045,
+            0.020444430509441124,
+            0.03140071554008003,
+            0.0025839998101985736,
+            0.01355205268903058,
+            -0.006268939830271642,
+            -0.013860692186849113,
+            0.014717364068116991,
+            0.031429975488060896,
+            0.0006919565473572909,
+            -0.018561500446336806,
+            0.019128244642842268,
+            0.006988095806264863,
+            -0.003678529269242034,
+            -0.0021993078588142664,
+            -0.021334919445200388,
+            -0.021003586791369228,
+            0.017304350704792804,
+            0.013508779408333964,
+            0.027794485858413365,
+            -0.010594702782952029,
+            -0.007818089330662161,
+            0.014293688127901867,
+            -0.022368568887004094,
+            0.015426482139881204,
+            -0.02160820059598751,
+            -0.00571499813703566,
+            -0.0077402210272634235,
+            0.03274360648629609,
+            0.001441304806200403,
+            0.02734331677589297,
+            0.016457027764064002,
+            -0.033609420134045206,
+            -0.029790535553657092,
+            -0.00712106584238484,
+            -0.02307933297713104,
+            0.010356561171963577,
+            -0.03901183838032092,
+            -0.00071735150189382,
+            0.008251646656432826,
+            -0.008079917288471352,
+            -0.0007087526632966007,
+            -0.006419191399519291,
+            0.013989344185652506,
+            0.013661633160336152,
+            -0.0228103143826663,
+            0.023644639659577426,
+            0.008343481940133684,
+            0.025998483906975782,
+            -0.02076055376006461,
+            0.019148568238437927,
+            -0.017705363646416224,
+            -0.02777313933338929,
+            0.014358130220942483,
+            0.023585731219788467,
+            0.01635364038245971,
+            -0.007122853965024528,
+            0.007110003583832299,
+            -0.03556761320536796,
+            0.027585457849964405,
+            -0.013064062623866305,
+            -0.012504161289952458,
+            -0.0032379276562094555,
+            -0.011931827532360686,
+            -0.002574161005800279,
+            0.0008533907465694845,
+            0.0034345232246102806,
+            -0.03131462673429805,
+            -0.0002450959360520896,
+            -0.03727416416766574,
+            -0.028944814604815072,
+            0.01360126708805471,
+            -0.00036361284498363273,
+            -0.0049097813612814616,
+            0.02933242705521636,
+            -0.026339136251046315,
+            -0.02693295650849956,
+            -0.009590247544093352,
+            0.02853096456887074,
+            -0.0220673798697511,
+            -0.004519050393279848,
+            0.0015034014871316069,
+            -0.01828226786700525,
+            0.004436383319946364,
+            -0.0012748481042235446,
+            -0.010416065158876917,
+            -0.01105146139270597,
+            -0.03229479099927737,
+            -0.005822133182096844,
+            0.02106215163620738,
+            -0.006957125617688484,
+            0.002746241228040847,
+            0.013312248331456934,
+            -0.010968629118224582,
+            -0.0029946900704643812,
+            0.0017875284441794235,
+            0.023304175718480834,
+            -0.021033941064173063,
+            0.007714334541806199,
+            0.005955442031111047,
+            0.0016405586860131207,
+            0.018406562522853148,
+            -0.0054922197063461825,
+            0.0037690571959124617,
+            0.0162126534419016,
+            -0.009969304596173302,
+            0.017959307638726878,
+            0.007691549802475525,
+            0.006681837561745192,
+            0.002775700618222622,
+            0.007712770283579111,
+            0.009752652973700739,
+            -0.013506020126078112,
+            0.016911141425375933,
+            0.022468715726349706,
+            0.00023233555209372766,
+            0.021066580799163234,
+            -0.02302516292580905,
+            -0.028618544628453015,
+            0.018297878810036087,
+            -0.0225078941299981,
+            -0.012735208680311652,
+            0.016459925496937065,
+            -0.0043032724578332855,
+            -0.00797609922711709,
+            -0.014986057073351055,
+            0.02260618561608301,
+            0.02132930921464508,
+            -0.020249788476562527,
+            -0.012385936327068884,
+            0.0072304431443959505,
+            -0.004373475447051833,
+            0.007086656081865733,
+            0.03195810137269375,
+            -0.015982210897392764,
+            0.006818890093901413,
+            -0.0038306138017521626,
+            0.0030753642449792215,
+            0.017830546304667783,
+            0.009470018737705222,
+            -0.0019815241970236146,
+            0.030100238416495066,
+            -0.001330614013444443,
+            0.016392226282459177,
+            0.0004888028477871524,
+            -0.010948833692356176,
+            0.006343890495626752,
+            -0.026652806306173643,
+            -0.008627570055007363,
+            -0.001203530223345183,
+            -0.009581494451831351,
+            0.07598616930784373,
+            0.003930153552069478,
+            -0.01295957554290434,
+            -0.001673007115291578,
+            -0.001368287102181597,
+            -0.006897179139643672,
+            -0.00960679451603541,
+            0.0287166325268306,
+            -0.016613613331623346,
+            -0.01894527592131826,
+            0.0008370014132002703,
+            -0.006785618888421608,
+            0.00919924251249592,
+            0.01816363936999796,
+            -0.010225961928154075,
+            -0.01604759761423727,
+            0.001577138242530474,
+            -0.003520564936698765,
+            -0.004395434431595494,
+            0.0052399043934902014,
+            -0.010127122101423486,
+            0.006456662777583096,
+            0.026560491731327886,
+            -0.013684536319659296,
+            -0.01425196172663986,
+            0.004084982328707292,
+            -0.0010682084059972982,
+            0.009758641913073978,
+            -0.00864879062601467,
+            -0.00481222050200678,
+            -0.014375330954464725,
+            -0.011378890549403524,
+            0.02505641780042516,
+            -0.02861164504722546,
+            0.0034060847953066354,
+            0.014937227531980433,
+            -0.00016904224334342915,
+            0.01541381851648916,
+            0.0033028855218525644,
+            -0.03615935765718852,
+            -0.013677774380513626,
+            -0.02073839691250734,
+            0.019821074228011413,
+            0.01688422012576042,
+            -0.0033731840788147442,
+            -0.013840993586441808,
+            -0.026149880178874887,
+            -0.007430563876740068,
+            -0.02466234818567362,
+            -0.020708282040996014,
+            -0.019276991448222817,
+            0.006137002799644838,
+            0.005162892426268572,
+            0.022881759407670583,
+            0.0025854880108157857,
+            -0.015058889448404231,
+            0.0029948608544278386,
+            -0.006722836436935071,
+            0.011845298212977357,
+            -0.02747944486818378,
+            0.015311272959213262,
+            -0.01610493464983465,
+            0.025239768880017402,
+            0.007425208902253236,
+            0.0010569008485112994,
+            -0.008822681490988575,
+            0.00010704167529046774,
+            0.014883787023445807,
+            -0.004186968199973372,
+            -0.0007781259002896265,
+            0.02787555889813697,
+            0.02087373813833492,
+            -0.0017783425135865238,
+            0.008743718448136432,
+            0.028026471206552342,
+            0.019081083283793984,
+            0.0325261942213039,
+            -0.013815526479760766,
+            0.005113676604826083,
+            -0.028873600114081546,
+            0.010510475843707614,
+            -0.010871039729515013,
+            -0.010201824683340651,
+            -0.0067946599549165735,
+            0.027756887504389347,
+            -0.007962601205446472,
+            -0.0006687976124826922,
+            0.03469894503563926,
+            -0.014185294847328063,
+            -0.00010543064621374546,
+            0.0007386418834840544,
+            -0.009759481309152999,
+            0.02742722730386853,
+            -0.0073590367291510046,
+            0.018456032721630843,
+            -0.019560882612141943,
+            -0.013705917411254969,
+            -0.011894560208199602,
+            -0.001142768182039817,
+            -0.02576754877141608,
+            -0.008755240066826174,
+            -0.02249287216087102,
+            0.00859172262856532,
+            -0.0046012388553094375,
+            0.018845920253446127,
+            0.0011617844393195566,
+            -0.005275799428018491,
+            0.029069367036565145,
+            0.0088080893110627,
+            0.018100411389202663,
+            -0.0004971948011851863,
+            0.01216566778850985,
+            0.02369448391321832,
+            0.002637001729688794,
+            0.005012948614448268,
+            -0.024213704079796478,
+            0.003420184574622472,
+            -0.0004359900462023086,
+            8.265488264269535e-05,
+            0.016267948861008616,
+            -0.019572032453100303,
+            -0.02216264534756073,
+            0.021871880626992544,
+            -0.012742926070754964,
+            0.011676769226089007,
+            0.0035362870996336194,
+            -0.010732229460038356,
+            0.0006838777488770268,
+            -0.03083438636386314,
+            0.008820746379709355,
+            -0.03216685858285627,
+            -0.032230808828686675,
+            -0.013547681710236494,
+            -0.009994106571421729,
+            0.0008758088338659712,
+            0.028447187234357855,
+            -0.00795317029726568,
+            -0.015797892954136448,
+            0.005363565557248085,
+            -0.011199252697624111,
+            -0.00817162221789202,
+            -0.0007867892785855576,
+            0.005856939999700809,
+            0.009461959355310273,
+            0.005997804620690314,
+            0.011648335019189696,
+            0.007491946658046298,
+            -0.008028800679169293,
+            -0.008837725220736191,
+            0.015889923814605387,
+            -0.0014409716087425222,
+            -0.01176424391171685,
+            0.030430130082762363,
+            -0.013300620185884817,
+            -0.0005592926632214984,
+            -0.0021975669766032472,
+            -0.0015687208316663597,
+            -0.0021652076181411244,
+            -0.014225289143358926,
+            0.01870568404189425,
+            8.334501002782455e-05,
+            -0.014288806218067678,
+            0.0032841743905975634,
+            0.003841780428153241,
+            -0.0036498339483411074,
+            0.008388922960269383,
+            -0.008147275785908849,
+            0.017348147498892,
+            -0.015113395971380741,
+            0.008740452410210708,
+            0.0038335161236289234,
+            0.0043482078469644385,
+            -0.0065755348640655865,
+            -0.01592809184384243,
+            0.009642593996701513,
+            0.012051872087663658,
+            0.02173264012652684,
+            -0.02525660485859878,
+            0.00876161653901566,
+            -0.005374533920781607,
+            -0.014534179652013885,
+            -0.016127866637369738,
+            0.00982303248417706,
+            0.022322910435388785,
+            0.0063086346898091296,
+            0.012139263756190026,
+            0.00013383660542132047,
+            0.005848259030010402,
+            -0.014767198851314148,
+            0.010570158368602513,
+            -0.011355774405066537,
+            -0.018439261453084426,
+            -0.005243587266408389,
+            0.028894120979627323,
+            -0.005492024421080143,
+            0.017938747701972086,
+            0.0021350711250093858,
+            0.029007356490801176,
+            -0.008686781541806358,
+            0.0014883465513564527,
+            0.0039866022469891155,
+            -0.01295271525889559,
+            -0.004272008629260297,
+            0.01976273843646234,
+            0.04882868638601926,
+            0.010204475902027572,
+            -0.0009155109545661691,
+            0.010069071220988797,
+            0.005626418787776309,
+            0.002148101019776153,
+            -0.00805781810841286,
+            -0.030431071067211577,
+            0.008883707378543252,
+            0.005628218215212441,
+            -0.01997825125961566,
+            0.019838234890736068,
+            0.026483365072033395,
+            0.023040623182975964,
+            -0.011585643598171132,
+            0.016539930468533123,
+            -0.010092529089936188,
+            0.0177008735686489,
+            0.028665424114962996,
+            0.013810061902648515,
+            -0.010487150104822674,
+            -0.0020847870330548534,
+            0.006329342599564054,
+            0.004186981876004016,
+            0.00026239408223817384,
+            0.015190614519766204,
+            -0.019155180136775043,
+            0.01005728016778505,
+            0.014725826697964637,
+            0.03322436221319799,
+            8.049930133627564e-05,
+            -0.01645191835557271,
+            0.0016249281996128196,
+            -0.009976155662939693,
+            0.00856399492362911,
+            0.000632427127231252,
+            0.0032242260692672124,
+            -0.009688119824276678,
+            -0.012682779503915694,
+            -0.002851918183031152,
+            0.009403205508474241,
+            0.0021188716361101698,
+            0.002759004011231831,
+            0.007641925150177999,
+            0.0022830062031146204,
+            0.01713024474384216,
+            -0.0011181980841546123,
+            -0.00029141889484139217,
+            0.004601090256674874,
+            0.010509199590505558,
+            -0.01649852835819883,
+            -0.017548369964263514,
+            0.004337237475053179,
+            0.012217494210546133,
+            -0.013159179948224596,
+            -0.013241981632059755,
+            0.015704543608959117,
+            -0.01913575130892795,
+            0.024934102868065366,
+            -0.00942381892138331,
+            -0.004543611336526587,
+            0.000343200128934031,
+            -0.00727538880611703,
+            -0.0038751118497522673,
+            -0.0030064279504724797,
+            0.003984715814505561,
+            0.0019261818110367147,
+            0.007472350699291607,
+            0.008472369788801792,
+            0.014568547873264247,
+            0.006043895352236181,
+            0.0032580830408778395,
+            0.009109104498825864,
+            -0.014523846294121576,
+            -0.014800649887278973,
+            -0.002704791598011488,
+            -0.008895394038700809,
+            0.00902016867266151,
+            -0.006607182154874112,
+            0.00845408382182461,
+            -0.015501867835442603,
+            0.015573642225745032,
+            -0.012200552219016517,
+            -0.0008900215712478553,
+            -0.009666233545144905,
+            -2.7470277092642562e-05,
+            0.008191697904514857,
+            0.014605287653054787,
+            0.0070552942843382215,
+            -0.031921405225964206,
+            0.0017403427534633655,
+            -0.008846568352320488,
+            0.03446319360336241,
+            0.002612606662691094,
+            -0.01060863123202008,
+            0.020352181356568008,
+            -0.01592124132155158,
+            -0.00027704668715660693,
+            -0.01602075523652062,
+            -0.004797557413047212,
+            0.00707939446796679,
+            0.016399017978990946,
+            0.02759049512664933,
+            -0.005697020356865688,
+            -0.007995305872076221,
+            0.006384047872862793,
+            0.029590696670334232,
+            0.012246955508890425,
+            0.024748351124415785,
+            -0.007633712004672087,
+            0.014348228716100414,
+            0.007773260552660029,
+            0.0076045488653348975,
+            0.004321689250440021,
+            -0.002591801797145718,
+            -0.011698574606499644,
+            -0.014974971814360001,
+            -0.0037563183330623384,
+            -0.016987915346053498,
+            0.00012247833530362867,
+            0.021344400276501117,
+            -0.00475351423959752,
+            -0.0033847161875730026,
+            -0.01874186437694691,
+            0.0033461670218611346,
+            -0.010029445302549983,
+            -0.00367421246775698,
+            -0.013133469163793961,
+            -0.0254122035745965,
+            -0.018556369080030412,
+            -0.002155244024999317,
+            -0.00929309186646207,
+            -0.015194269781263309,
+            0.0070354690219415535,
+            -0.016305117309271634,
+            0.0080837166473439,
+            -0.00448473417024141,
+            -0.016508775226236883,
+            -0.00958925666995396,
+            0.01100335153658195,
+            -0.0041819993634961685,
+            -0.01929493648733792,
+            -0.01282300765645703,
+            -0.011300260564838231,
+            0.0051733704121938884,
+            0.005996780699975213,
+            -0.0038111648304178544,
+            -0.0028279639348479987,
+            0.016640121845366295,
+            -0.010197365811240557,
+            0.009878730312873492,
+            -0.03335474019383245,
+            -0.03551826040406959,
+            0.004497847402931096,
+            0.01507007013703286,
+            0.0013085881889541252,
+            -0.02078680101065954,
+            0.010806315988966003,
+            -0.016499186313125556,
+            -0.00858013792516617,
+            -0.008522534127498645,
+            0.009649521088411411,
+            -0.007283003380127847,
+            0.01299542385123007,
+            -5.3226845309105124e-05,
+            0.030477683597373687,
+            0.0074153455615142755,
+            0.004161881195438522,
+            -0.00679081048132963,
+            0.008354912600159931,
+            -0.001621776416108105,
+            0.004080191054849395,
+            0.02120524874226696,
+            0.012440848502230418,
+            -0.00218041171518331,
+            0.006458415098595872,
+            0.005678055538564355,
+            -0.017186537377121235,
+            0.02507960402044456,
+            0.024678178106806115,
+            -0.008205288425409454,
+            0.004108402466376321,
+            0.005783335302169668,
+            -0.008846663924040241,
+            -0.006897707231869481,
+            0.006527795262026734,
+            -0.018474181903007085,
+            -0.0008721381678008483,
+            0.010598084358464908,
+            -0.02255441909004843,
+            -0.004840511019512733,
+            0.021745651362109403,
+            0.004631567631335321,
+            -0.018558972627958834,
+            0.02764788406651861,
+            -0.004942361238163609,
+            -0.0027290999934532494,
+            -0.010876135202650048,
+            0.01671497143541799,
+            -0.012173527207375091,
+            0.008443317346655462,
+            0.011021541861818043,
+            -0.006295639322287843,
+            0.003083685676540999,
+            0.008277206401503182,
+            -0.004504302913715969,
+            -0.0068075424927125676,
+            -0.0010508416962215848,
+            0.010167160518175114,
+            0.0036980815941666336,
+            -0.022704963865525812,
+            -0.006020073261002279,
+            0.02934227992246887,
+            -0.015082922782242507,
+            -0.0050916398557337705,
+            0.005376294716056329,
+            0.007709342013963906,
+            -0.003476236561444464,
+            -0.0023399991233097337,
+            -0.00787486645740744,
+            0.010356414979428607,
+            -0.015950083845400274,
+            0.02076360898078935,
+            0.025608263494550438,
+            0.024504236990840842,
+            -0.014814235990238182,
+            0.010828584133720797,
+            -0.011703992394890898,
+            -0.011047470905668532,
+            -0.015542835222173864,
+            -0.001658296303013357,
+            0.001970208852236451,
+            0.013107754130616248,
+            0.009666887546609378,
+            0.0235853683657408,
+            0.006473544089109194,
+            0.008973300312205858,
+            0.014118601565410305,
+            -0.003902202854013708,
+            -0.018303438244719996,
+            -0.014584226585460755,
+            -0.011977613285028946,
+            -0.01756345491020528,
+            0.0004563602312336091,
+            0.020461659996271687,
+            0.012417744207683839,
+            0.024136626592375227,
+            0.002119213701333967,
+            0.013759902247467922,
+            -2.9525392347047554e-05,
+            -0.000892319792935443,
+            0.00045783825646242434,
+            -0.005238483120312843,
+            0.019255756091120846,
+            -0.004599752813817219,
+            0.01656589522110913,
+            0.0006443773300684925,
+            0.007133940816771615,
+            0.013554761340256566,
+            -0.01704388669741371,
+            -0.003443100892108368,
+            -0.010029620309829285,
+            0.000941391966877521,
+            -0.019633111750884034,
+            0.01934571074127927,
+            0.010273330977233852,
+            0.0017121345993533963,
+            -0.018327189396014198,
+            -0.0029706385099949693,
+            0.03448635131998772,
+            0.019458999453011265,
+            0.0050734354363614525,
+            0.02632443373898792,
+            0.013018762352749236,
+            0.014443611411502168,
+            0.010153370950346075,
+            0.02356766253713842,
+            -0.023479811300626968,
+            0.02095255383200973,
+            -0.011590347850340786,
+            -0.007414136601134069,
+            -0.0035774888741646705,
+            0.0025141508183687115,
+            -0.013588691733097813,
+            0.00982877622088977,
+            -0.0017916539399655826,
+            -0.025386211797192578,
+            0.009164316443206016,
+            -0.008811052979979271,
+            0.010464442616928296,
+            -0.007774569252594923,
+            -0.0002806136360286729,
+            -0.0127084026190163,
+            -0.015184317235990752,
+            0.001259201880504347,
+            -0.009158429132281081,
+            0.023908150915412617,
+            -0.021210848953875233,
+            -0.01321384014436847,
+            0.013997502713857512,
+            -0.006869443694669701,
+            -0.011364239609894453,
+            -0.00374449980963666,
+            0.023673050288879224,
+            -0.009243341446512421,
+            -0.01359367519009039,
+            0.014464166953530989,
+            0.003834735113400469,
+            0.0059713317014342395,
+            -0.007508862694272452,
+            -0.008428705474135382,
+            -0.003280788036976103,
+            0.005998840043285335,
+            0.026615077617284676,
+            -0.014059930972065399,
+            -0.0032833264512892022,
+            -0.009189999799498163,
+            0.004118204537632102,
+            -0.02359889635281631,
+            0.02771075351384059,
+            -0.008871395303200375,
+            0.021705177758560575,
+            -0.0015924449308442096,
+            -0.006239188158271628,
+            -0.024231036074349822,
+            0.027522945599928003,
+            0.003683870966132199,
+            -0.010221180899173243,
+            0.0012768725460662862,
+            -0.052005292736885915,
+            0.016636109796947025,
+            0.008429895913163985,
+            -0.019507994429834353,
+            0.003697519981610208,
+            -0.004773584856005853,
+            0.018983847580558853,
+            -0.022966993228850697,
+            -0.014078986007109447,
+            0.008283766047449966,
+            -0.01153285124362746,
+            0.0031439180465649996,
+            -0.02527749501821793,
+            -0.005754537200570427,
+            0.0011760345167229269,
+            -0.005276936074280004,
+            -0.001705933750221904,
+            -0.020022761608132548,
+            -0.025972227850795115,
+            0.017666017379157135,
+            0.015820875579600418,
+            -0.002278262448951789,
+            0.01320818011561616,
+            0.008709899932124187,
+            0.00836573261091119,
+            0.016034263394940008,
+            0.007124509234103267,
+            0.0081473711608784,
+            0.045498533239850446,
+            -0.019846323044145485,
+            0.007371629414298193,
+            -0.0014854149174929534,
+            0.013951741898629716,
+            -0.030467924817555014,
+            -0.0055144480998344705,
+            0.0062802744565489945,
+            -0.0049373689340857915,
+            0.03510454786147091,
+            -0.017326094216003073,
+            0.01694160526735264,
+            -0.0026075809870445238,
+            0.0007420394265014909,
+            -0.02042003877666893,
+            -0.026156937726134134,
+            -0.014294938857971834,
+            0.0011857406889443987,
+            -0.0010047941261608975,
+            -0.011946583993069242,
+            0.006797001710574735,
+            0.004917161312925866,
+            -0.0020713379451245515,
+            0.015652523103715316,
+            -0.02952061566240878,
+            0.007356800041369564,
+            -0.010733578620243498,
+            0.020008504830359524,
+            -0.00990331048284697,
+            -0.0013319400098308093,
+            0.007236960885181047,
+            -0.004122868383988748,
+            0.013216043472071927,
+            -0.0038337251451538677,
+            0.004227115653354611,
+            -0.005221321656766545,
+            -0.01794343569722877,
+            -0.0016435261495179767,
+            0.021429008432348098,
+            -0.008349684096540855,
+            0.031864497493526205,
+            -0.01264743878037828,
+            0.009174982784686633,
+            0.010692990337462762,
+            0.0006628831036334152,
+            -0.005286285851702483,
+            0.009052971814211658,
+            0.0021431970274545893,
+            -0.008690520448278274,
+            0.006934310986093342,
+            -0.018349449937310143,
+            0.008494451785507706,
+            0.03394632739506833,
+            0.0018511328780472266,
+            -0.0007519661670070526,
+            -0.007266663179613392,
+            -0.01335844201823745,
+            0.018540713029383066,
+            0.005488273906733518,
+            -0.002546531152114063,
+            -0.027536660869590543,
+            0.02007746621157705,
+            0.0012265798861291223,
+            -0.004755325069578531,
+            0.012466557901411086,
+            0.0020495867632654885,
+            0.019989531151046736,
+            -0.010171678639113316,
+            0.030757957899447096,
+            0.010050036220124393,
+            0.002959833051227505,
+            0.0038556928367700113,
+            -0.036655749074851536,
+            -0.01652152778069314,
+            -0.00019357502472740266,
+            0.014271581054865031,
+            -0.012430647252826173,
+            -0.022432580814844155,
+            0.0010346169306977334,
+            0.01148004051449691,
+            -0.0033898642762881046,
+            -0.009650665305271706,
+            0.01560757541276278,
+            -0.020342213902670575,
+            0.006701302394408393,
+            0.007693672045981921,
+            0.0287100214318104,
+            -0.0020735384554022153,
+            -0.0012324838849990056,
+            0.000626416803904792,
+            -0.0055669860801192935,
+            -0.0008037366126870824,
+            0.017251369734470945,
+            -0.013661316947002466,
+            0.0175926211217534,
+            -0.012735241490550407,
+            0.0064713656749838585,
+            -0.0021092378432037563,
+            -0.010629579860050812,
+            -0.020114807971202787,
+            0.0029609495764307357,
+            0.009378003163375899,
+            0.006920144698428806,
+            -0.010438984881449953,
+            0.010715409812532968,
+            0.012197673689706352,
+            -0.00753962861484439,
+            0.014519905132928041,
+            0.022173989895384192,
+            -0.007687503607783572,
+            0.013633321414009471,
+            -0.0041446334728745095,
+            -0.014056264816568229,
+            -0.009985463824440833,
+            0.023929750617160402,
+            -0.0005563806942667597,
+            0.008756997373618809,
+            0.00388125051641897,
+            0.0007856447663040696,
+            -0.011226530517316183,
+            -0.001653447552601681,
+            0.019294448360333633,
+            0.011831934675073843,
+            -0.0013888534822008192,
+            -0.0010525764356699885,
+            -0.0014566936525653407,
+            -0.009427557739483988,
+            0.02049188741459728,
+            -0.022030783927622997,
+            -0.0019604940727076573,
+            0.00461018148439991,
+            -0.022589816803875618,
+            0.0001863117949313752,
+            -0.024570217962989905,
+            -0.01569952241187498,
+            0.011763033781122343,
+            -0.0022530631823737554,
+            -0.016028810867854907,
+            -0.0003242701040942633,
+            7.424127301806556e-05,
+            -0.007815789940508527,
+            -0.004135500768181258,
+            0.006115909992703774,
+            -0.00034193217510010117,
+            -0.0032262189374874842,
+            0.013430154310506884,
+            -0.009996622196870008,
+            0.015431528911575157,
+            0.0067029980830614145,
+            -0.015702167721800483,
+            0.0021406610839557003,
+            0.018500219544264753,
+            -0.0058910087864916085,
+            0.00787021425357441,
+            -0.014681907249972176,
+            -0.0037495551894426303,
+            -0.005787661351944086,
+            0.006051351977821761,
+            0.024849195780885833,
+            0.012969434450643372,
+            0.01139340445077682,
+            0.017405361843075933,
+            0.004325000715054734,
+            -0.01572297405297993,
+            0.0007416798945675923,
+            0.01682727189828363,
+            0.010464036837245312,
+            0.009804326468113703,
+            0.0008975556667684365,
+            0.0043704860361487635,
+            -0.005142176468800175,
+            0.009317064072855551,
+            0.02915482449687158,
+            0.0013434689612997423,
+            0.008696141010760192,
+            0.017276730345147134,
+            0.027038741396131265,
+            -0.0035088997194497725,
+            -0.007469614319788499,
+            0.008739942310300076,
+            -0.009657991604173173,
+            -0.001081629174976152,
+            -0.017000351183542026,
+            -0.012310275736750868,
+            0.016510505259358398,
+            -0.006084627062230492,
+            0.02848913486719956,
+            -0.0010655806791907225,
+            -0.02026498848361482,
+            -0.01565505608194376,
+            0.001404549479440637,
+            -0.01362132895080652,
+            0.005287566256051299,
+            0.014187265685427415,
+            0.01917046513020831,
+            0.004687191682875024,
+            -0.022258337721085656,
+            0.037364962197399806,
+            0.01757108963349427,
+            0.019377870795760123,
+            -0.019422840506328382,
+            -0.006159311237412833,
+            -0.009077095160676276,
+            0.02116697012944498,
+            -0.00880692095488465,
+            -0.009875674856043954,
+            0.024014332004451508,
+            0.017812606845656484,
+            0.005679806286464216,
+            0.006736609904079586,
+            0.0020048663439319585,
+            0.0025414630253661444,
+            -0.015831593851012047,
+            0.017957902498542212,
+            -0.005858125698347523,
+            -0.0058068554681761945,
+            0.017529514573571298,
+            -0.005989581792993925,
+            -0.0030306021203732347,
+            0.00015839997806822045,
+            -0.019697052145531482,
+            0.011618112886207201,
+            0.008256007601576028,
+            -0.017229774429089857,
+            -0.0014757560402567304,
+            -0.0002404480326945295,
+            -0.014118694740134205,
+            0.02004885146995733,
+            0.01694893356262835,
+            -0.0013248884139405398,
+            -0.0051752815372013315,
+            0.008223868147393114,
+            0.007000679132597393,
+            0.005256194890343006,
+            -0.019496098876977206,
+            0.005230597149342019,
+            0.005103909294824105,
+            -0.0014716649591412774,
+            0.010274346323051634,
+            -0.009030594825985526,
+            0.033740380586893826,
+            -0.006505788197604522,
+            0.013065658941136721,
+            -0.0043634731684193745,
+            -0.0055529072007985125,
+            0.001816020676232658,
+            0.013054468478598695,
+            0.009753501030700457,
+            -0.00970446155762939,
+            -0.004172354962129865,
+            0.018697982092775737,
+            -0.011983156146516074,
+            0.0013300791598046338,
+            0.01564072865804035,
+            -0.004858814041438713,
+            0.004708235083775013,
+            0.008635204790345423,
+            -0.004489505237783172,
+            0.01801512858970922,
+            -0.016482572999490366,
+            0.0023953436660054026,
+            0.010694869684962088,
+            0.02390864159830096,
+            -0.019152187046410703,
+            0.021786596486708207,
+            -0.017951050280323393,
+            0.026538582634436314,
+            0.014484894489694262,
+            -0.0035222259457311444,
+            -0.01946200885147006,
+            -0.00850690913394181,
+            0.011654565129063417,
+            -0.021423803876241532,
+            -0.0007985948078290641,
+            -0.0007290222944686444,
+            -0.021367628543092133,
+            -0.011189364017231132,
+            -0.009961779129778331,
+            0.010701637745810138,
+            0.001918633898887013,
+            0.010380983868185359,
+            0.008002182827196827,
+            0.010784760574518087,
+            0.014405640442760822,
+            0.0008958107769384412,
+            -0.010743001445440941,
+            -0.023795589003655273,
+            0.016987708698679597,
+            0.00438463274382093,
+            0.0007862212695139549,
+            -0.01627468994515667,
+            0.011936651269737248,
+            0.013250643815768723,
+            -0.009385605548926286,
+            0.009121882469425302,
+            -0.03716901103204463,
+            -0.004552808410398574,
+            -0.0035704811925430366,
+            -0.0020309197365173475,
+            -0.00908678158053104,
+            0.01559445415882485,
+            -0.013682507836005851,
+            -0.0009590922904844632,
+            0.00011696985057536548,
+            -0.0074756526903548915,
+            -1.5467890432254378e-05,
+            0.020649928837031005,
+            0.008438785939908428,
+            -0.0037382105257999757,
+            0.010376213224654074,
+            0.008826031303751312,
+            -0.013900359449287676,
+            -0.01527474712786084,
+            0.019035366035884273,
+            -0.018776918681908793,
+            0.0063098418544296165,
+            0.01677916431403883,
+            0.020778813096283988,
+            -0.02891862501830075,
+            -0.0007231469961304435,
+            0.022093939332021185,
+            -0.0012807635799950273,
+            0.006370841109790851,
+            0.008781853138377092,
+            0.006754082980602544,
+            0.0029102159062355374,
+            -0.003711173531879644,
+            -4.843770941622283e-06,
+            -0.003940474588075707,
+            0.007155539022862352,
+            0.004366433868349658,
+            0.036453728071874844,
+            -0.013661129348165794,
+            0.002541308805735001,
+            -0.0058838191638835155,
+            -0.00347401304572066,
+            -0.00593469688967835,
+            0.004407665856950772,
+            -0.010055727994798024,
+            0.004330827357536078,
+            -0.001570861535288093,
+            -0.0003731129422346693,
+            0.004883969598300302,
+            -0.001701673379646408,
+            0.007036892564844683,
+            -0.011863650867472625,
+            -0.0003979727750011258,
+            -0.020174851412609312,
+            0.021117284557875267,
+            0.00011666551282792189,
+            0.003662462447326642,
+            0.03723741246290771,
+            0.00476064510662738,
+            -0.00238355131307445,
+            -0.012880057949435798,
+            -0.013551338766321525,
+            -0.012986444917747435,
+            -0.020703821629987552,
+            -0.0036665673548525388,
+            -0.021006311026247164,
+            -0.0052061406234528915,
+            0.010362862594785442,
+            -0.011540184420787572,
+            -0.004023659825622843,
+            -0.015881589631889473,
+            -0.006531981517815174,
+            -0.03114211633687585,
+            -0.01124607488650473,
+            0.007493228366452009,
+            -0.019149285949049886,
+            -0.0039932870926865805,
+            0.026850121590606017,
+            0.001095360635760988,
+            0.01387696424607248,
+            0.0038058248676879625,
+            0.008117686785954085,
+            -0.0045937472017383955,
+            -0.0005669025036141417,
+            -0.01928244877716143,
+            0.003391980330529026,
+            0.008525354854576925,
+            -0.00361507909530282,
+            0.0070008404454106905,
+            -0.008835727374865623,
+            -0.016950017602980382,
+            0.0021654956715219517,
+            0.009334379529385142,
+            -0.006639140003365139,
+            -0.017587472740268683,
+            -0.011049929670847685,
+            0.010401189523318744,
+            -0.02283429470064187,
+            0.006257145380266984,
+            0.016439581169238607,
+            -0.009363030375624585,
+            0.004651575425738153,
+            0.019809183749755425,
+            -0.011810674576781702,
+            -0.005811328501528397,
+            0.008471992070998505,
+            0.013891259999721588,
+            0.0095606134736748,
+            0.02954498207274139,
+            0.0026596809804008996,
+            0.000561870602466584,
+            0.004889263753490895,
+            -0.004277061142504552,
+            -0.017423200644235665,
+            -0.006243787897080996,
+            0.0038507494787521856,
+            0.012873996671952184,
+            -0.009879417960879032,
+            -0.00048575277472139055,
+            0.01682604801673206,
+            0.004530939990187663,
+            -0.0033901248079550284,
+            -0.011868399508738274,
+            -0.020080331108085103,
+            0.0015580259874809898,
+            -0.01362200460317948,
+            0.015354347040143093,
+            0.003390601069331352,
+            0.0015284610287850575,
+            -0.004239880336806429,
+            0.01828645373058079,
+            -0.005013418527935815,
+            0.012749833447770729,
+            -0.006148890994661865,
+            -0.004587831231448011,
+            0.007986533783001116,
+            0.009166578151006778,
+            0.0004939857995863664,
+            0.05421019830078428,
+            0.002838056604877719,
+            0.01333205648597435,
+            0.013419127695838135,
+            -0.014523733369484934,
+            0.014086525147734823,
+            0.012688569853552613,
+            0.018497344547821693,
+            0.013037819487160016,
+            -0.012353776077273462,
+            0.017354801119939964,
+            0.0024084723134891487,
+            0.003909681065813095,
+            0.0022202438064449653,
+            -0.004528576079628071,
+            0.019594683950225115,
+            0.004070222328547378,
+            -0.0074362443729372105,
+            0.004453618452623641,
+            0.005007543168224267,
+            -0.013105524222311019,
+            -0.0112958915514782,
+            0.027105208848863218,
+            -0.01219417557128091,
+            -0.018749971743148304,
+            0.006706594269933445,
+            -0.02008458683853808,
+            -0.03763686535076864,
+            -0.0021332074289244973,
+            0.009992361098878196,
+            0.002795183524495193,
+            -0.00253695604043486,
+            0.01614298064601032,
+            0.016730443047719683,
+            -0.016985296937992726,
+            -0.018245902728910347,
+            0.0025257780765481825,
+            0.0016333325293570182,
+            0.01861807236979066,
+            -0.023168663546030196,
+            -0.006378864850313035,
+            -1.3795465857380962e-05,
+            -0.006829008132963306,
+            0.004850306884033312,
+            0.004703280112191218,
+            -0.0071714759384722965,
+            -0.0027773519096287217,
+            0.002256846645341137,
+            0.01975383140755998,
+            0.013198754584066554,
+            0.01572971337890945,
+            0.002600741959577911,
+            -0.005033508062449482,
+            -0.016817359058724685,
+            -0.005450011991586895,
+            -0.013478330855299777,
+            -0.009091854345715177,
+            -0.003042052092498012,
+            0.008217376329142566,
+            -0.025053469059115735,
+            -0.011965083373220982,
+            0.0119294363235832,
+            0.00391377331459168,
+            0.0003099203772539202,
+            0.007607672728136279,
+            -0.003830132495364688,
+            0.010331053246451333,
+            0.013823890473448383,
+            0.0076112887952959766,
+            -0.004723377809559611,
+            0.009600411526173948,
+            0.011460781436250457,
+            0.006015477068316837,
+            -0.005072711638895421,
+            0.023859899048048694,
+            0.01821920335673539,
+            -0.01680365156728093,
+            -0.002149107383573353,
+            -0.0027829676356423997,
+            -0.01884308738652242,
+            0.009665830801645276,
+            0.00880322398336952,
+            0.008308268605449346,
+            0.003906369351868086,
+            0.00974531700889854,
+            -0.013875904819275925,
+            -0.0050570811397453235,
+            0.002870120865445989,
+            -0.005729556771404781,
+            -0.01901337075686543,
+            0.000297242372701423,
+            -0.011987192261104132,
+            -0.011327157550660895,
+            -0.01409710922165158,
+            -0.003221240464411861,
+            0.005283472511900512,
+            0.021472187640986972,
+            -0.004410881709548286,
+            -0.001855852749039012,
+            0.01351697599517034,
+            0.010060125186506398,
+            0.0021658738823899905,
+            0.0002206474541239167,
+            0.005418828324605768,
+            -0.0027919130944783397,
+            0.0008094042430809068,
+            -0.0015826797905407809,
+            0.021089276765434876,
+            -0.02016611086095107,
+            -0.009413977569213423,
+            -0.009288078098692112,
+            0.005842197094339927,
+            0.028793032462603636,
+            -0.007165634502373454,
+            -0.010104002083017078,
+            -0.01994315293556678,
+            0.0003786527922888834,
+            -0.07810609175343784,
+            -0.005210636756258548,
+            0.001870513618910238,
+            0.03126380113105983,
+            -0.010817888476436834,
+            0.0033673768058629823,
+            -0.002204487854177595,
+            0.014542299655712928,
+            -0.001760107965206874,
+            -0.0013283746982194259,
+            0.0071954806737858056,
+            -0.005640400818010697,
+            -0.006210989097047217,
+            0.005904163661619133,
+            -9.945941028848348e-05,
+            -0.010266938705916303,
+            0.00057263080955709,
+            0.02209498395848068,
+            0.0159398346608872,
+            -0.006575439759804175,
+            0.01816748552021211,
+            -0.00719294553484007,
+            -0.009838738592324234,
+            0.005639288780502127,
+            -0.0019007683459126327,
+            -0.015369619176147573,
+            -0.011475977270803784,
+            0.009862808084641291,
+            -0.030069890644150284,
+            -0.00010725753918890802,
+            0.00943698424646706,
+            -0.013540245857270492,
+            -0.008293042205008774,
+            0.007212974960781723,
+            0.006708343474532583,
+            -0.0012017968151025369,
+            -0.0078026000239507145,
+            -0.02527526731750743,
+            0.0027578400032925625,
+            -0.002446811971857853,
+            -0.004899702349784173,
+            0.0019960465173086183,
+            0.001479503476855694,
+            -0.012645905858466798,
+            0.01216748847452763,
+            -0.013378877368040743,
+            -0.012053251571418766,
+            0.012508842412556306,
+            -0.00027167241745418584,
+            0.01682680930386839,
+            -0.020607299598265426,
+            0.0010702151063620052,
+            -0.028075439824932304,
+            -0.007671848153932823,
+            -0.015486630894165054,
+            -0.0009776136330459065,
+            -0.0012939921569540803,
+            -0.03481880531471923,
+            0.012889447665794033,
+            -0.005200149829198305,
+            -0.006094702500154477,
+            -0.00459778303079732,
+            -0.024583504817899775,
+            0.006072281238752663,
+            0.00607350167867964,
+            -0.006506508970816635,
+            -0.016816439635445445,
+            0.008428048142588516,
+            -0.0011040593013238872,
+            0.004873670535200424,
+            -0.007475800310154908,
+            -0.0005638920132076605,
+            -0.013047135093134946,
+            -0.005463454269764494,
+            -0.0008522562324122834,
+            0.01827123304871433,
+            0.0044055851550317835,
+            -0.014526381222031981,
+            0.016862445236481526,
+            0.013250692156820124,
+            0.01602714293656341,
+            -0.0071089293427753115,
+            -0.0004265739315250755,
+            -0.02750630108787212,
+            -0.00327225092954903,
+            -0.023687644154808972,
+            0.00041213204365507744,
+            0.009447660230507965,
+            0.0203905264889617,
+            0.012694421477479161,
+            0.004620617778996077,
+            -0.001557511764988156,
+            0.0066751090724502,
+            0.02696959560423252,
+            -0.01099183126742224,
+            -0.006969111598423376,
+            -0.01884504192789845,
+            0.01113023717919988,
+            0.025135748541199255,
+            0.003381303139273845,
+            -0.011317957771935042,
+            0.008112598754655146,
+            -0.0009247845526290428,
+            -0.010302980598863016,
+            0.02889587382004902,
+            -0.0391134481322538,
+            -0.015599591108049343,
+            -0.004605824192962023,
+            0.007406520608110251,
+            -0.015596917115435033,
+            0.00770468362805995,
+            -0.012614339656105264,
+            0.001699049329817941,
+            -0.005851477164035241,
+            0.03535087340659861,
+            -0.0031950960727572925,
+            0.0006909856233673865,
+            -0.003395233050301427,
+            0.003531135044131077,
+            0.012355948349937309,
+            -0.004444221317023453,
+            -0.030726300006909638,
+            -0.0022983766869702346,
+            -0.011198252640110788,
+            0.009274666307515955,
+            -0.016207714291658314,
+            -0.017342435976846468,
+            -0.009424737744066259,
+            -0.021827243671837953,
+            -0.012403084982789827,
+            0.029747453318486338,
+            -0.005824001042844527,
+            0.01747569398142269,
+            -0.005219060548639168,
+            -0.008365003057800934,
+            0.00021272411948388247,
+            0.01344335658390732,
+            -0.028498240040284714,
+            0.00041963499785238845,
+            -0.002322931702641349,
+            0.0126393550661767,
+            -0.001595134597082556,
+            -0.00983073906977773,
+            0.006903313245694627,
+            0.0010522208685011098,
+            -0.015648940549713764,
+            -0.0007573245570947632,
+            -0.008533504878052644,
+            0.020475329445149498,
+            0.00027791387213409814,
+            0.001074213629646005,
+            0.013864331671732542,
+            -0.02102352993295736,
+            0.01576938071930521,
+            -0.021722775924283996,
+            0.01893267442161547,
+            -0.006577103101038035,
+            0.005772016664149475,
+            0.008283223343661555,
+            0.014068789332250016,
+            -0.014030042849491258,
+            -0.00663752723080222,
+            -0.034436220972634726,
+            -0.0122621879114314,
+            -0.001946072976826902,
+            -0.005308941654794638,
+            -0.014773123927909993,
+            -0.0011054822266082056,
+            -0.026205461509573584,
+            -0.006460373065662098,
+            0.012822188046808264,
+            0.019900312098359022,
+            -0.008409070094265829,
+            0.014163132434889265,
+            -0.006743711872055091,
+            -0.0028230771810882492,
+            -0.012652021749331811,
+            -0.018061180380009395,
+            -0.01083672324723357,
+            -0.01890966636827258,
+            -0.0018635408070944168,
+            0.004709222372611061,
+            -0.015760618199235518,
+            0.011435131737642627,
+            0.03198874888333405,
+            -0.01567271472650171,
+            0.008243406188694909,
+            0.016411496328333205,
+            0.0034236971031973503,
+            0.011728079139322837,
+            0.0038311415233052795,
+            0.004621789116267879,
+            -0.011109450058055078,
+            -0.011473158133621661,
+            0.006524490579223986,
+            0.00954511918454185,
+            -0.004913142601134428,
+            0.024347072255007807,
+            0.0008745419206472714,
+            0.005045794664611216,
+            0.005304506902446091,
+            0.011488187043525824,
+            -0.013135271908307747,
+            0.019746484188715552,
+            -0.002055210086078267,
+            -0.015571665869889405,
+            -0.01535499169234956,
+            -0.01105419264826655,
+            -0.018626399757822328,
+            0.0017108083455869296,
+            0.0181439079855921,
+            -0.0012493542718277685,
+            0.012137753522284387,
+            -0.00924460959150499,
+            -0.005019437549649743,
+            -0.0027156377206998474,
+            -0.038211954658821295,
+            -0.0021878970332906457,
+            -0.018809487178512628,
+            -0.02498945889484935,
+            0.03505216568400238,
+            -4.26477081213836e-05,
+            0.00997643255528116,
+            -0.010310197159232386,
+            -0.0002454902098986129,
+            -0.011065403358802515,
+            0.014954998647970473,
+            0.004624123403100598,
+            0.008872923678983478,
+            0.008560280683876702,
+            -0.016419500515801093,
+            -0.002997557798375716,
+            0.007472390947008284,
+            -0.00374822278027057,
+            0.012680407710694952,
+            0.015353684671120731,
+            0.020886474047003733,
+            0.011161262397502876,
+            0.002045862158732102,
+            0.00887271411363814,
+            0.009563361150297699,
+            -0.0003513237086717377,
+            -0.010459621030424415,
+            0.015419039131459582,
+            -0.02204848739950703,
+            0.012172257374410352,
+            0.04346634265794738,
+            0.019197433209794286,
+            0.029488039318675732,
+            -0.02272810439246353,
+            0.01448092574459596,
+            -0.003801043409441206,
+            0.009476229966122152,
+            0.0049309968883087685,
+            0.02346191840355209,
+            0.003973840862678076,
+            0.0073546443568171405,
+            0.010647581332452525,
+            -0.00972051412996416,
+            0.025644022498404723,
+            -0.00846669916801108,
+            5.1518540791212095e-05,
+            -0.004407265804145216,
+            -0.026400029105519137,
+            0.01615193008014211,
+            0.0081722536401398,
+            0.0010702314763197463,
+            0.0010263623479474385,
+            -0.0019566604229607643,
+            0.007982949510899019,
+            0.007079322685771456,
+            -0.015274148711633536,
+            -0.003886053737590683,
+            -0.018527455865296014,
+            0.0012864569064749438,
+            -0.024337133883771334,
+            0.010466651250597847,
+            -0.017568706814132484,
+            -0.014822354159709854,
+            -0.021666681078831483,
+            -0.013232016867264097,
+            -0.02011653224168991,
+            0.0035858355517491515,
+            -0.008632755352062642,
+            0.013025728891203198,
+            -0.016962939287139683,
+            -0.021494971929779273,
+            0.008430887589398292,
+            0.00870802359711456,
+            0.007481282677955534,
+            0.009728368603458082,
+            0.014996157358749115,
+            0.01736292711410622,
+            -0.022712386143092265,
+            -0.023163001611599246,
+            -0.006682209717364464,
+            -0.003531495447167166,
+            0.006762379833125297,
+            0.00933681196321128,
+            0.012884185500343249,
+            -0.007077895849501418,
+            -0.007554762054641417,
+            -0.013283067570021503,
+            0.006197966297581871,
+            -0.003993067576677973,
+            -0.009464789349849612,
+            -0.0015662739825834614,
+            -0.0061389795593180686,
+            0.002753664732282926,
+            -0.0001791088888074743,
+            0.0007595261237141144,
+            0.017122834676182343,
+            -0.008176319370751018,
+            -0.009787747261447871,
+            -0.02027228453943018,
+            -0.010683273300243025,
+            -0.015604745521156111,
+            -0.0016279406232527138,
+            -0.006506713191861036,
+            -0.014420323628589382,
+            0.010888206569219648,
+            -0.011930708571093279,
+            -0.00887210117616784,
+            0.010807147705360856,
+            0.0073900910661282935,
+            -0.003431679006289002,
+            0.037804020313161775,
+            0.0011945597874820288,
+            0.00778778394048267,
+            -0.003366543306746973,
+            0.005296727102519836,
+            -0.017761251541598876,
+            -0.026382400396890007,
+            0.0010357716464853195,
+            -0.01775777668381489,
+            0.010807713750049491,
+            -0.008515368407459745,
+            0.0016051389544388124,
+            -0.00034958600424912995,
+            -0.009585854918521076,
+            0.0036549720706351473,
+            -0.008110927907950148,
+            0.004212219431447508,
+            -0.03507890639477636,
+            0.011409377585617926,
+            -0.01819120811524444,
+            -0.008962242411035718,
+            0.01710177594046791,
+            0.037560936862121835,
+            0.0028513388998398104,
+            0.00030058353811717895,
+            0.0014792184333546447,
+            0.004427921639326168,
+            0.021438127785303025,
+            0.006339483221420334,
+            -0.001187747087595264,
+            -0.02237982599829809,
+            0.0062435650294483695,
+            0.002200578793697352,
+            0.010092828474934744,
+            0.023105023225115266,
+            -0.016797326021070008,
+            -0.015378520287395565,
+            -0.012732527281000097,
+            -0.010797463778182385,
+            0.009881499418260473,
+            0.02801767062785128,
+            0.01811139137849365,
+            -0.017358098300028903,
+            -0.020207173948076473,
+            0.0016242901745373126,
+            -0.026331540602868762,
+            0.007269751661479859,
+            0.11940631114242901,
+            -0.0045382135734049305,
+            -0.005994407567723939,
+            -0.008679712231646732,
+            -0.004482553208064535,
+            0.016709571294385293,
+            -0.013387043582971358,
+            0.069179848974036,
+            -0.007221545718220735,
+            0.0037392229364642915,
+            0.0028916224964057385,
+            0.00462729158956839,
+            -0.009727339449635306,
+            0.011570145487622916,
+            -0.011287365791681587,
+            -0.012991994493845187,
+            -0.013076194601468383,
+            -0.0180830261130804,
+            -0.007272768410218907,
+            -0.004155564814758654,
+            0.004204993170487392,
+            -0.005525575983182816,
+            0.008051108079520902,
+            -0.017551645696802648,
+            -0.010800070994485792,
+            0.013007176804761548,
+            -0.0023627485852619817,
+            0.005390986239687555,
+            -0.029668390278675554,
+            -0.0007650391259819867,
+            -0.0002446394046947245,
+            0.021280292985495608,
+            0.003137771886236958,
+            -0.0019077298815862946,
+            -0.010810159675564049,
+            0.007040107942856957,
+            0.023835993694726953,
+            0.021875366566457236,
+            0.0180511566130312,
+            0.012135351541044354,
+            0.020483975362807574,
+            -0.008667079096167644,
+            0.026324312888512866,
+            -0.001552207146726937,
+            -0.00397810465151493,
+            0.0077577136352257004,
+            -0.009849600842081875,
+            -0.00816392268697345,
+            0.00677353736462267,
+            -0.017288900282504148,
+            -0.020418071729964713,
+            -0.020383762873109595,
+            0.025741408097358233,
+            -0.011755094311186617,
+            0.002737129201159793,
+            -0.008266210469668866,
+            0.027287436144516414,
+            -0.006283892307996516,
+            -0.0029440642954701632,
+            0.009428876742171362,
+            -0.03788574758692254,
+            0.008834605607057885,
+            0.006463342392514337,
+            0.026711196829464153,
+            -3.9827332346130654e-05,
+            0.005565561841469247,
+            -0.0006738085601962865,
+            -0.006104320995853506,
+            -0.010792830169647851,
+            2.4565453208679464e-05,
+            0.017983908748172493,
+            0.0035181716362611077,
+            0.016970206528324653,
+            -0.005449808667944701,
+            -0.023171388778833197,
+            0.007092509502738752,
+            -0.01555320330471131,
+            -0.023306285898839978,
+            -0.002028783456704567,
+            -0.03196436161255752,
+            0.0033412618908711163,
+            0.007527633574188688,
+            -0.01890570959143866,
+            -0.017859735533232564,
+            0.010329340599904344,
+            0.004382396539605985,
+            -0.0016746397858953155,
+            0.006643785064338525,
+            -0.03307812909454471,
+            -0.01927475592253288,
+            0.004686970010943116,
+            0.012733525168279355,
+            -0.012993441126970916,
+            0.015291832952199082,
+            -0.020491167895268936,
+            0.016451976170508816,
+            -0.012259387189535237,
+            0.018393274368854132,
+            -0.006997668664226754,
+            0.011843450716200625,
+            0.011457415642860215,
+            0.009560877634897626,
+            -0.01790001115302499,
+            0.0214879415944361,
+            0.0066829070506954115,
+            0.03900547021123392,
+            0.02512484499576633,
+            -0.012651294056575184,
+            -0.0018425680226742782,
+            -0.00011361458320146007,
+            0.014982647266052349,
+            0.0061874424115559,
+            -0.003689464024837091,
+            0.02705902205734107,
+            0.018239469373522537,
+            -0.021070704644859155,
+            0.020037508334843183,
+            -0.00973804771680803,
+            0.018876283979591423,
+            -0.013303080480396818,
+            0.010592424494628194,
+            0.004717870260467274,
+            0.01070131238869639,
+            0.009087656015197565,
+            0.006708673669681232,
+            0.025928065153829043,
+            0.03663367449576964,
+            -0.00511073108390551,
+            0.002161381031392467,
+            0.004864092591944181,
+            0.012814518637357098,
+            0.001556833001747549,
+            0.007915766323215207,
+            -0.007012785514562176,
+            0.005834786334750152,
+            -0.002311533032619821,
+            -0.00122162928193159,
+            0.019525999280692766,
+            0.000824163362963678,
+            0.012875575199417254,
+            0.013259376059185821,
+            -0.002390133687709065,
+            -0.005682644446302278,
+            0.0019262218417782657,
+            -0.018193478045923098,
+            -0.015131696847666434,
+            0.029463723011600933,
+            -0.011447039651934148,
+            -0.008083114389964213,
+            0.0020642857121204753,
+            0.012160202510990566,
+            -0.00866944494225225,
+            -0.005378300635997713,
+            0.0038280364584418335,
+            0.014427318067525318,
+            -0.01799358399064332,
+            0.003381165206832161,
+            0.005252552821854107,
+            0.019433554824232185,
+            -0.00028791985674147885,
+            -0.013487678738391956,
+            0.0005317922815671783,
+            0.014982152801567637,
+            0.011902348165369078,
+            0.0016056390299304837,
+            -0.006935777197137747,
+            -0.00446584737370504,
+            0.004586814866182051,
+            0.006269466956110759,
+            0.0027516492952993445,
+            0.008157662045793032,
+            -0.0025001885993292637,
+            -0.016648747436203294,
+            0.017735757469839637,
+            -0.0030669873770852725,
+            0.018058248627079793,
+            -0.0038238312452176745,
+            -0.0147172470474118,
+            0.0043328417868095355,
+            -0.001888507803054499,
+            -0.018556943283494755,
+            0.009407930200720534,
+            0.002560949424273885,
+            0.010411159836451958,
+            0.0052571230238294825,
+            -0.007499470181994448,
+            -6.145889479758644e-05,
+            -0.015163819825329975,
+            0.022836682800304944,
+            0.016390944816242627,
+            0.001713572907204073,
+            -0.0035687505973373878,
+            -0.031210299824387475,
+            0.0029076511444359886,
+            -0.007822231980296341,
+            0.017821806420465356,
+            0.008250605126658402,
+            0.007908773724009905,
+            0.008351355849444162,
+            -0.00743970642051451,
+            -0.002411981861500092,
+            0.012543476434788293,
+            0.0202407777715137,
+            -0.0024635818448627447,
+            0.02535771413579989,
+            -0.012533449594824386,
+            -0.0022489292647974992,
+            0.0012362917358624392,
+            0.01578939075186126,
+            -0.0008001783766371555,
+            0.006311837812042848
+          ],
+          "keyphrases": [
+            "Switch Transformers",
+            "Mixture of Experts",
+            "sparsely-activated model",
+            "training instability",
+            "multilingual settings"
+          ],
+          "title": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity"
+        },
+        "type": "document"
+      },
+      "bidirectional": true,
+      "properties": {
+        "cosine_similarity": 0.8425598372328582
+      }
+    },
+    {
+      "id": "874d635e-6476-40dd-9e0a-25a93ddd2f20",
+      "type": "cosine_similarity",
+      "source": {
+        "id": "1e687808-8db8-4320-805d-0282a8d679bc",
+        "properties": {
+          "page_content": "Deepseekmoe: Towards Ultimate Expert Specialization In\n\nMixture-Of-Experts Language Models\n\nDamai Dai∗1,2, Chengqi Deng1, Chenggang Zhao∗1,3, R.X. Xu1, Huazuo Gao1, Deli Chen1, Jiashi Li1, Wangding Zeng1, Xingkai Yu∗1,4, Y. Wu1, Zhenda Xie1, Y.K. Li1, Panpan Huang1, Fuli Luo1, Chong Ruan1, Zhifang Sui2, Wenfeng Liang1 1DeepSeek-AI 2National Key Laboratory for Multimedia Information Processing, Peking University 3Institute for Interdisciplinary Information Sciences, Tsinghua University 4National Key Laboratory for Novel Software Technology, Nanjing University {daidamai, szf}@pku.edu.cn, {wenfeng.liang}@deepseek.com https://github.com/deepseek-ai/DeepSeek-MoE\n\nAbstract\n\nIn the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top- out of experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into ones and activating from them, allowing for a more flexible combination of activated experts; (2) isolating experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5× expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations.\n\nFurther, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\n1. Introduction\n\nRecent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023a). It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017) has emerged as a popular solution. It can\n\nenable parameter scaling, while concurrently keeping computational costs at a modest level.\n\nRecent applications of MoE architectures in Transformers (Vaswani et al., 2017) have yielded successful attempts at scaling language models to a substantial size (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022), accompanied with remarkable performance. These achievements underscore the considerable potential and promise of MoE language models.\n\nDespite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts.\n\nThis architecture manifests two potential issues: (1) Knowledge Hybridity: existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. (2) Knowledge Redundancy: tokens assigned to different experts may require common knowledge. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters.\n\nThese issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models.\n\nIn response to the aforementioned issues, we introduce DeepSeekMoE, an innovative MoE architecture specifically designed towards ultimate expert specialization. Our architecture involves two principal strategies: (1) Fine-Grained Expert Segmentation: while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. (2) Shared Expert Isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.\n\nThrough compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized.\n\nStarting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B (Lepikhin et al., 2021) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5× expert parameters and computation. Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE.\n\nThese studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization.\n\nLeveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B (DeepSeek-AI, 2024), a dense model trained on the same 2T corpus. We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B (Touvron et al., 2023b), which has approximately 2.5 times the activated parameters. Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard1. Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. The experimental results still validate its substantial advantages over the GShard architecture consistently. In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nOur contributions are summarized as follows: - Architectural Innovation. We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation.\n\nEmpirical Validation. We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. Experimental results validate the high 1https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models\n\nScalability. We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B.\n\nAlignment for MoE. We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B.\n\nPublic Release. In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization.\n\n2. Preliminaries: Mixture-Of-Experts For Transformers\n\nWe first introduce a generic MoE architecture commonly used in Transformer language models. A standard Transformer language model is constructed by stacking layers of standard Transformer blocks, where each block can be represented as follows:\n\n$$\\mathbf{u}{1:T}^{l}=\\text{Self-Att}\\left(\\mathbf{h}{1:T}^{l-1}\\right)+\\mathbf{h}{1:T}^{l-1},\\tag{1}$$ $$\\mathbf{h}{t}^{l}=\\text{FFN}\\left(\\mathbf{u}{t}^{l}\\right)+\\mathbf{u}{t}^{l},\\tag{2}$$\n\nwhere denotes the sequence length, Self-Att(·) denotes the self-attention module, FFN(·) denotes the Feed-Forward Network (FFN), u 1: ∈ R× are the hidden states of all tokens after the -th attention module, and h\n\n∈ Ris the output hidden state of the -th token after the -th Transformer block. For brevity, we omit the layer normalization in the above formulations.\n\nA typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022). An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. Then, each token will be assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts. If the -th FFN is substituted with an MoE layer, the computation for its output hidden state h\n\nis expressed as: where denotes the total number of experts, FFN(·) is the -th expert FFN, , denotes the gate value for the -th expert, , denotes the token-to-expert affinity, Topk(·, ) denotes the set comprising highest affinity scores among those calculated for the -th token and all experts, and e\n\nis the centroid of the -th expert in the -th layer. Note that ,is sparse, indicating that only out of gate values are nonzero. This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only experts. Also, in the above formulations, we omit the layer normalization operation for brevity.\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{N}\\left(g_{i,t}\\operatorname{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\operatorname{Topk}({s_{j,t}|1\\leqslant j\\leqslant N},K),\\ 0,&\\text{otherwise},\\end{cases}$$ $$s_{i,t}=\\operatorname{Softmax}{i}\\left(\\mathbf{u}{t}^{l}\\,\\mathbf{e}_{i}^{l}\\right),$$\n\n$$(3)$$\n\n(4) $\\binom{4}{5}$ .\n\n3. Deepseekmoe Architecture\n\nOn top of the generic MoE architecture outlined in Section 2, we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. As illustrated in Figure 2, our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. Both of these strategies are designed to elevate the level of expert specialization.\n\n3.1. Fine-Grained Expert Segmentation\n\nIn scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts.\n\nIn pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. The finer expert segmentation enables a more flexible and adaptable combination of activated experts. To be specific, on top of a typical MoE architecture shown in Figure 2(a), we segment each expert FFN into smaller experts by reducing the FFN intermediate hidden dimension to 1 times its original size. Since each expert becomes smaller, in response, we also increase the number of activated experts to times to keep the same computation cost, as illustrated in Figure 2(b). With the fine-grained expert segmentation, the output of an MoE layer can be expressed as:\n\n$$\\mathbf{h}{t}^{i}=\\sum{i=1}^{\\text{mN}}\\left(g_{i,t}\\text{FFN}{i}\\left(\\mathbf{u}{t}^{i}\\right)\\right)+\\mathbf{u}{t}^{i},\\tag{6}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\text{Tokr}(s_{i,t}|1\\leqslant j\\leqslant m\\text{N}),m\\text{K}),\\ 0,&\\text{otherwise},\\end{cases}$$ (7) $$s_{i,t}=\\text{Softmax}{i}\\left(\\mathbf{u}{t}^{T}\\mathbf{e}_{t}^{i}\\right),\\tag{8}$$ where the total number of expert parameters is equal to $N$ times the number of parameters in a\n\n$$(6)$$\n\n(7) (8) $\\frac{1}{2}$\n\nstandard FFN, and denotes the total number of fine-grained experts. With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to .\n\nFrom a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. As an illustrative example, we consider the case where = 16. A typical top-2 routing strategy can yield 16 2 = 120 possible combinations. By contrast, if each expert is split into 4 smaller experts, the fine-grained routing strategy can yield 64 8 = 4, 426, 165, 368 potential combinations. The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition.\n\n3.2. Shared Expert Isolation\n\nWith a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. This alleviation of redundancy will contribute to a more parameterefficient model with more specialized experts.\n\nTowards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate experts to serve as shared experts. Regardless of the router module, each token will be deterministically assigned to these shared experts. In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by , as depicted in Figure 2(c). With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows:\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{K_{s}}\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)+\\sum_{i=K_{s}+1}^{mN}\\left(g_{i,t}\\;\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},\\tag{9}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\mathrm{Topk}{s_{j,t}|K_{s}+1\\leqslant j\\leqslant mN},mK-K_{s}},\\ 0,&\\text{otherwise},\\end{cases}$$ (10) $$s_{i,t}=\\mathrm{Softmax}{i}\\left(\\mathbf{u}{t}^{l,T}e_{i}^{l}\\right).\\tag{11}$$ $\\mathbf{h}\\mathbf{M}\\mathbf{E}$ the number of closed-compact is $K_{s}$ the total number of routed currents.\n\nFinally, in DeepSeekMoE, the number of shared expert is , the total number of routed experts is − , and the number of nonzero gates is − .\n\nIt is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022). The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint.\n\n3.3. Load Balance Consideration\n\nAutomatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. Firstly, there is a risk of routing collapse (Shazeer et al., 2017), i.e., the model always selects only a few experts, preventing other experts from sufficient training.\n\nSecondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.\n\nExpert-Level Balance Loss. In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. The computation of the balance loss is as follows:\n\n$$\\begin{array}{l}{{{\\mathcal L}{\\mathrm{ExpBal}}=\\alpha{1}\\sum_{i=1}^{N^{\\prime}}f_{i}P_{i},}}\\ {{f_{i}=\\frac{N^{\\prime}}{K^{T}}\\sum_{t=1}^{T}\\mathds{1}\\,(\\mathrm{Token}\\;t\\;\\mathrm{s}\\mathrm{lects}\\;\\mathrm{Expert}\\;i),}}\\ {{P_{i}=\\frac{1}{T}\\sum_{t=1}^{T}s_{i,t},}}\\end{array}$$\n\n(12) $$\\begin{array}{l}\\small\\text{(13)}\\end{array}$$ = (14) $$\\begin{array}{l}\\small\\text{(14)}\\end{array}$$ .\n\n7 where 1 is a hyper-parameter called expert-level balance factor, ′is equal to ( − ) and ′ is equal to ( − ) for brevity. 1(·) denotes the indicator function.\n\nDevice-Level Balance Loss. In addition to the expert-level balance loss, we introduce a devicelevel balance loss. When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. Instead, our primary objective is to ensure balanced computation across the devices. If we partition all routed experts into groups {E1, E2, ..., E}, and deploy each group on a single device, the device-level balance loss is computed as follows:\n\n$$\\mathcal{L}{\\text{DevBal}}=\\alpha{2}\\sum_{i=1}^{D}f_{i}^{\\prime}P_{i}^{\\prime},$$ $$f_{i}^{\\prime}=\\frac{1}{|\\mathcal{E}{i}|}\\sum{j\\in\\mathcal{E}{i}}f{j},$$ $$P_{i}^{\\prime}=\\sum_{j\\in\\mathcal{E}{i}}P{j},$$\n\n(15) $$\\begin{array}{l}\\small\\text{(16)}\\end{array}$$ = (17) .\n\nwhere 2 is a hyper-parameter called device-level balance factor. In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices.\n\n4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n6. Alignment For Deepseekmoe 16B\n\nPrevious research indicates that MoE models typically do not emerge significant gains from fine-tuning (Artetxe et al., 2022; Fedus et al., 2021). However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B.\n\n6.1. Experimental Setup\n\nTraining Data. For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios.\n\nHyper-Parameters. During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).\n\nWe employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10−5 without incorporating any learning rate scheduling strategy.\n\nEvaluation Benchmarks. For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3, with the following adjustments: (1) We exclude Pile (Gao et al., 2020) since chat models are seldom employed for pure language modeling. (2) We exclude\n\nMetric # Shot LLaMA2 DeepSeek DeepSeekMoE SFT 7B Chat 7B Chat 16B # Total Params N/A 6.7B 6.9B 16.4B # Activated Params N/A 6.7B 6.9B 2.8B FLOPs per 4K Tokens N/A 187.9T 183.5T 74.4T HellaSwag (Acc.) 0-shot 67.9 71.0 72.2 PIQA (Acc.) 0-shot 76.9 78.4 79.7 ARC-easy (Acc.) 0-shot 69.7 70.2 69.9 ARC-challenge (Acc.) 0-shot 50.8 50.2 50.0 BBH (EM) 3-shot 39.3 43.1 42.2 RACE-middle (Acc.) 5-shot 63.9 66.1 64.8 RACE-high (Acc.) 5-shot 49.6 50.8 50.6 DROP (EM) 1-shot 40.0 41.7 33.8 GSM8K (EM) 0-shot 63.4 62.6 62.2 MATH (EM) 4-shot 13.5 14.7 15.2 HumanEval (Pass@1) 0-shot 35.4 45.1 45.7 MBPP (Pass@1) 3-shot 27.8 39.0 46.2 TriviaQA (EM) 5-shot 60.1 59.5 63.3 NaturalQuestions (EM) 0-shot 35.2 32.7 35.1 MMLU (Acc.) 0-shot 50.0 49.7 47.2 WinoGrande (Acc.) 0-shot 65.1 68.4 69.0 CLUEWSC (EM) 5-shot 48.4 66.2 68.2 CEval (Acc.) 0-shot 35.1 44.7 40.0 CMMLU (Acc.) 0-shot 36.9 51.2 49.3\n\nCHID (Zheng et al., 2019) due to the observed instability of results, hindering the derivation of solid conclusions. (3) We additionally include BBH (Suzgun et al., 2022) to provide a more comprehensive assessment of the reasoning ability of the chat models.\n\nTable 5 | Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations.\n\n6.2. Evaluations\n\nBaselines. In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. Correspondingly, we construct three chat models, including LLaMA2 SFT 7B3, DeepSeek Chat 7B, and DeepSeekMoE Chat 16B.\n\nSubsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks.\n\nResults. The evaluation results are presented in Table 5. Our key observations include: (1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intensive tasks (TriviaQA, NaturalQuestions). (2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. In addition, it also surpasses DeepSeek Chat 7B. (3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1). However, it is worth noting that, after supervised finetuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. (4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations.\n\n7. Deepseekmoe 145B Ongoing\n\nEncouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available.\n\n7.1. Experimental Setup\n\nTraining Data and Tokenization. For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study.\n\nModel Settings. For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps.\n\nSubsequently, the learning rate keeps constant during the remaining training process. The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. Also, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse.\n\nEvaluation Benchmarks. We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3).\n\n7.2. Evaluations\n\nBaselines. Apart from DeepSeekMoE 145B, we consider three additional models for comparison. DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters.\n\nResults. From the evaluation results presented in Table 6, we have the following observations: (1) Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. (2) On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. (3) At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5.\n\n8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\nAppendices A. Overview Of Hyper-Parameters\n\nWe present the overview of hyper-parameters for DeepSeekMoE across various sizes in Table 7.\n\n# Params # Layers Hidden # Attn # Shared # Routed Relative Sequence Batch Size Learning Size Heads Experts Experts Expert Size Length (Sequence) Rate 2.0B 9 1280 10 1 63 (7 activated) 0.25 2048 2048 1.08e-3 16.4B 28 2048 16 2 64 (6 activated) 0.25 4096 4608 4.2e-4 144.6B 62 4096 32 4 128 (12 activated) 0.125 4096 4608 3.0e-4\n\nTable 7 | Overview of hyper-parameters for DeepSeekMoE across various sizes. The relative expert size is in comparison to a standard FFN.\n\nB. Comparing Deepseekmoe With Larger Models\n\nComparisons among DeepSeekMoE, GShard×1.2, and GShard×1.5 are shown in Table 8. Comparisons among DeepSeekMoE, Dense×4, and Dense×16 are shown in Table 9.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 2.3B 2.8B 1.9B # Activated Expert Params N/A 0.28B 0.35B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.824 1.808 1.808 HellaSwag (Acc.) 0-shot 53.7 54.4 54.8 PIQA (Acc.) 0-shot 71.8 71.1 72.3 ARC-easy (Acc.) 0-shot 46.8 47.3 49.4 ARC-challenge (Acc.) 0-shot 31.7 34.1 34.3 RACE-middle (Acc.) 5-shot 43.7 46.4 44.0 RACE-high (Acc.) 5-shot 31.9 32.4 31.7 HumanEval (Pass@1) 0-shot 3.7 3.0 4.9 MBPP (Pass@1) 3-shot 2.4 2.6 2.2 TriviaQA (EM) 5-shot 15.2 15.7 16.6 NaturalQuestions (EM) 5-shot 4.5 4.7 5.7\n\nTable 8 | Comparison between DeepSeekMoE and larger GShard models.\n\nAt a larger scale of 13B total parameters, we also compare DeepSeekMoE with GShard×1.2 and GShard×1.5, and show results in Table 10. At a larger scale, DeepSeekMoE even outperforms GShard×1.5 distinctly.\n\nMetric # Shot Dense×4 Dense×16 DeepSeekMoE Relative Expert Size N/A 1 1 0.25 # Experts N/A 4 + 0 16 + 0 1 + 63 # Activated Experts N/A 4 + 0 16 + 0 1 + 7 # Total Expert Params N/A 0.47B 1.89B 1.89B # Activated Expert Params N/A 0.47B 1.89B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.908 1.806 1.808 HellaSwag (Acc.) 0-shot 47.6 55.1 54.8 PIQA (Acc.) 0-shot 70.0 71.9 72.3 ARC-easy (Acc.) 0-shot 43.9 51.9 49.4 ARC-challenge (Acc.) 0-shot 30.5 33.8 34.3 RACE-middle (Acc.) 5-shot 42.4 46.3 44.0 RACE-high (Acc.) 5-shot 30.7 33.0 31.7 HumanEval (Pass@1) 0-shot 1.8 4.3 4.9 MBPP (Pass@1) 3-shot 0.2 2.2 2.2 TriviaQA (EM) 5-shot 9.9 16.5 16.6 NaturalQuestions (EM) 5-shot 3.0 6.3 5.7\n\nTable 9 | Comparison between DeepSeekMoE and larger dense baselines.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 15.9B 19.8B 13.3B # Activated Expert Params N/A 2.37B 2.82B 2.05B # Training Tokens N/A 100B 100B 100B HellaSwag (Acc.) 0-shot 66.6 67.7 69.1 PIQA (Acc.) 0-shot 75.6 76.0 75.7 ARC-easy (Acc.) 0-shot 56.8 56.8 58.8 ARC-challenge (Acc.) 0-shot 39.9 37.6 38.5 RACE-middle (Acc.) 5-shot 51.6 50.6 52.4 RACE-high (Acc.) 5-shot 37.4 36.3 38.5 HumanEval (Pass@1) 0-shot 6.1 6.1 9.8 MBPP (Pass@1) 3-shot 7.0 11.6 10.6 TriviaQA (EM) 5-shot 36.5 36.7 38.2 NaturalQuestions (EM) 5-shot 12.6 12.1 13.7\n\nTable 10 | Comparison between DeepSeekMoE and larger GShard models at a larger scale.\n\nC. Training Benchmark Curves Of Deepseekmoe 16B\n\nWe present the benchmark curves during training of DeepSeekMoE 16B and DeepSeek 7B (Dense) in Figure 7 for reference.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/DeepSeekMoE_2.md"
+          },
+          "summary": "DeepSeekMoE is an innovative Mixture-of-Experts (MoE) architecture designed to enhance expert specialization in language models. It employs fine-grained expert segmentation and shared expert isolation to improve performance and reduce computational costs. Starting with a model of 2B parameters, DeepSeekMoE demonstrates comparable performance to larger models like GShard 2.9B and LLaMA2 7B, using significantly fewer computations. The architecture scales up to 16B and 145B parameters, maintaining advantages over traditional MoE architectures. DeepSeekMoE 16B achieves similar results to dense models with only 40% of the computations, and its chat model version shows adaptability in bilingual scenarios. The architecture's public release aims to contribute to advancements in large-scale language models.",
+          "headlines": {
+            "1. Introduction": [],
+            "2. Preliminaries: Mixture-Of-Experts For Transformers": [],
+            "3. Deepseekmoe Architecture": [
+              "3.1. Fine-Grained Expert Segmentation",
+              "3.2. Shared Expert Isolation",
+              "3.3. Load Balance Consideration"
+            ],
+            "4. Validation Experiments": [
+              "4.1. Experimental Setup",
+              "4.1.1. Training Data And Tokenization",
+              "4.1.2. Infrastructures",
+              "4.1.3. Hyper-Parameters",
+              "4.1.4. Evaluation Benchmarks",
+              "4.2. Evaluations",
+              "4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models",
+              "4.4. Ablation Studies",
+              "4.5. Analysis On Expert Specialization"
+            ],
+            "5. Scaling Up To Deepseekmoe 16B": [
+              "5.1. Experimental Setup",
+              "5.1.1. Training Data And Tokenization",
+              "5.1.2. Hyper-Parameters",
+              "5.1.3. Evaluation Benchmarks",
+              "5.2. Evaluations",
+              "5.2.1. Internal Comparison With Deepseek 7B",
+              "5.2.2. Comparison With Open Source Models"
+            ],
+            "6. Alignment For Deepseekmoe 16B": [
+              "6.1. Experimental Setup",
+              "6.2. Evaluations"
+            ],
+            "7. Deepseekmoe 145B Ongoing": [
+              "7.1. Experimental Setup",
+              "7.2. Evaluations"
+            ],
+            "8. Related Work": [],
+            "9. Conclusion": [],
+            "Appendices": [
+              "A. Overview Of Hyper-Parameters",
+              "B. Comparing Deepseekmoe With Larger Models",
+              "C. Training Benchmark Curves Of Deepseekmoe 16B"
+            ]
+          },
+          "summary_embedding": [
+            -0.012465757317841053,
+            -0.022808987647294998,
+            -0.018792515620589256,
+            -0.02308654971420765,
+            0.01797616109251976,
+            0.007200260180979967,
+            0.006490030791610479,
+            0.042613785713911057,
+            -0.014049487188458443,
+            0.03187054023146629,
+            -0.021486490964889526,
+            -0.023592690005898476,
+            0.009045224636793137,
+            -0.014441338367760181,
+            0.005865518003702164,
+            -0.007363531738519669,
+            -0.008743173442780972,
+            -0.005163452122360468,
+            0.018841497600078583,
+            -0.03100520372390747,
+            0.011110605672001839,
+            -0.046075135469436646,
+            -0.020017050206661224,
+            -0.021845687180757523,
+            0.016425084322690964,
+            0.037389107048511505,
+            0.022547753527760506,
+            -0.0028062239289283752,
+            -0.06844329088926315,
+            0.034678805619478226,
+            0.035168617963790894,
+            0.02052319049835205,
+            0.0005438972148112953,
+            -0.00031302153365686536,
+            -0.024833550676703453,
+            -0.028556134551763535,
+            0.03347059711813927,
+            -0.004379750229418278,
+            -0.01453930139541626,
+            0.015976088121533394,
+            0.0414055772125721,
+            0.012106560170650482,
+            -0.02754385396838188,
+            0.0014245413476601243,
+            0.015151568688452244,
+            0.0055961208418011665,
+            -0.01678428053855896,
+            -0.0022960014175623655,
+            -0.002424577483907342,
+            -0.021225256845355034,
+            -0.01400050614029169,
+            0.018825169652700424,
+            0.001296985661610961,
+            -0.008865626528859138,
+            -0.0542386956512928,
+            0.007330877240747213,
+            0.03840138763189316,
+            0.05623060464859009,
+            0.044148534536361694,
+            -0.020180322229862213,
+            0.03369917720556259,
+            -0.030025575309991837,
+            -0.02501314878463745,
+            0.011028970591723919,
+            0.003983817528933287,
+            -0.018531281501054764,
+            0.01784554310142994,
+            -0.005636938381940126,
+            -0.0006556359585374594,
+            0.01751900091767311,
+            -0.03941367194056511,
+            0.020751770585775375,
+            -0.01887415163218975,
+            0.031266435980796814,
+            0.02173139713704586,
+            0.0073798587545752525,
+            0.01872720755636692,
+            0.01678428053855896,
+            0.0036858476232737303,
+            -0.04107903689146042,
+            0.009118696674704552,
+            0.016523046419024467,
+            0.014196431264281273,
+            0.016269976273179054,
+            0.06553706526756287,
+            -0.05391215533018112,
+            -0.004485876299440861,
+            -0.05169166624546051,
+            0.0012561678886413574,
+            -0.004971608519554138,
+            -0.03670336678624153,
+            0.01048201136291027,
+            0.028425518423318863,
+            0.021372200921177864,
+            -0.01063711941242218,
+            0.024392718449234962,
+            -0.022743679583072662,
+            0.0032001156359910965,
+            -0.04062187671661377,
+            0.0019653772469609976,
+            0.004518530797213316,
+            0.016531210392713547,
+            0.0070778070949018,
+            0.004273623693734407,
+            -0.016702644526958466,
+            0.006045116577297449,
+            -0.01905374974012375,
+            0.03291547670960426,
+            -0.008465612307190895,
+            0.08653374016284943,
+            0.008751336485147476,
+            0.004738946910947561,
+            0.004281787667423487,
+            0.02398454025387764,
+            0.012131051160395145,
+            -0.00810233410447836,
+            -0.04718538001179695,
+            -0.015690363943576813,
+            0.011061624623835087,
+            -0.0383034273982048,
+            0.018335357308387756,
+            -0.008759500458836555,
+            0.01282495353370905,
+            -0.04107903689146042,
+            -0.06344719231128693,
+            0.01498013362288475,
+            0.001784758409485221,
+            -0.004065453074872494,
+            -0.0002903166168835014,
+            -0.018923133611679077,
+            0.003930754493921995,
+            -0.004200151655822992,
+            -0.010645282454788685,
+            -0.021502818912267685,
+            0.03159298002719879,
+            0.03657275065779686,
+            0.06527583301067352,
+            0.036801330745220184,
+            -0.0030164355412125587,
+            0.005775718949735165,
+            0.019690508022904396,
+            -0.020784424617886543,
+            0.003481758525595069,
+            -0.01214737817645073,
+            -0.021992631256580353,
+            -0.03820546343922615,
+            0.010971825569868088,
+            0.07379858940839767,
+            -0.029029620811343193,
+            0.013437220826745033,
+            -0.017616963014006615,
+            0.04205866530537605,
+            -0.004567512311041355,
+            0.026727497577667236,
+            -0.016669990494847298,
+            0.004059330560266972,
+            -0.025388672947883606,
+            0.01620466820895672,
+            -0.024964168667793274,
+            0.012571883387863636,
+            -0.000885746325366199,
+            -0.04222193360328674,
+            0.00011116473615402356,
+            0.026890767738223076,
+            -0.010971825569868088,
+            0.02261306345462799,
+            0.0004949158756062388,
+            -0.006302268709987402,
+            -0.05097327008843422,
+            0.038727931678295135,
+            -0.003912386484444141,
+            0.018645571544766426,
+            -0.0018959869630634785,
+            0.0163107942789793,
+            -0.0016408757073804736,
+            0.013216804713010788,
+            0.009543201886117458,
+            0.007114543113857508,
+            -0.033503253012895584,
+            0.02697240374982357,
+            -0.02504580281674862,
+            0.028049994260072708,
+            0.03435226157307625,
+            -0.0030184765346348286,
+            0.03255628049373627,
+            0.019249675795435905,
+            0.03840138763189316,
+            0.0327032245695591,
+            0.030335791409015656,
+            -0.00824927818030119,
+            0.013878053054213524,
+            0.008200296200811863,
+            0.03608293831348419,
+            0.013429056853055954,
+            0.019723162055015564,
+            -0.009730963967740536,
+            -0.014041324146091938,
+            0.048785436898469925,
+            0.007318632211536169,
+            -0.0179598331451416,
+            -0.005441013257950544,
+            -0.03510330989956856,
+            -0.002832755446434021,
+            -0.014425011351704597,
+            0.0323440283536911,
+            0.02158445306122303,
+            -0.01647406443953514,
+            0.035135965794324875,
+            0.024980494752526283,
+            -0.03751972317695618,
+            -0.005232842173427343,
+            -0.02026195637881756,
+            -0.023298801854252815,
+            -0.04215662553906441,
+            0.011624909937381744,
+            0.012833117507398129,
+            -0.013976015150547028,
+            -0.007469657808542252,
+            0.020457882434129715,
+            -0.023625344038009644,
+            -0.0018265966791659594,
+            -0.01356783788651228,
+            0.006547175347805023,
+            -0.020898714661598206,
+            -0.03667071461677551,
+            -0.01048201136291027,
+            0.025862159207463264,
+            0.012637191452085972,
+            0.04362606629729271,
+            -0.013518855907022953,
+            -0.005367541220039129,
+            0.033193036913871765,
+            0.012759645469486713,
+            0.010767736472189426,
+            0.03699725493788719,
+            -0.03027048148214817,
+            0.024621298536658287,
+            -0.020294612273573875,
+            -0.0032878739293664694,
+            0.01707816869020462,
+            -0.0030062312725931406,
+            0.01962519995868206,
+            -0.04535674303770065,
+            -0.024196794256567955,
+            0.04395261034369469,
+            -0.0007107399869710207,
+            0.016302630305290222,
+            -0.005959399044513702,
+            -0.012106560170650482,
+            0.0035899258218705654,
+            0.0027715289033949375,
+            -0.01438419334590435,
+            -0.008620720356702805,
+            0.007473739795386791,
+            -0.01962519995868206,
+            0.03660540655255318,
+            0.03794422745704651,
+            -0.004132802598178387,
+            -0.014131123200058937,
+            -0.03324201703071594,
+            0.010016689077019691,
+            -0.017878197133541107,
+            -0.023951886221766472,
+            0.04078514873981476,
+            -0.032115448266267776,
+            -0.00029312283731997013,
+            0.039478980004787445,
+            0.018808843567967415,
+            -0.014898497611284256,
+            0.00868602842092514,
+            -0.01709449477493763,
+            -0.026302991434931755,
+            -0.017159804701805115,
+            0.03356856107711792,
+            0.024800896644592285,
+            -0.010743245482444763,
+            0.010245268233120441,
+            0.005330805201083422,
+            0.02893165871500969,
+            0.024376392364501953,
+            0.00648594880476594,
+            -0.013812744058668613,
+            0.015029114671051502,
+            0.027641816064715385,
+            -0.0028021421749144793,
+            -7.66609373386018e-05,
+            -0.004010349046438932,
+            -0.007796200457960367,
+            -0.006453294772654772,
+            -0.013069860637187958,
+            -0.005089980084449053,
+            0.0009184005321003497,
+            -0.032295044511556625,
+            0.015967924147844315,
+            -0.005890008993446827,
+            0.010359558276832104,
+            -0.011535110883414745,
+            0.01288209855556488,
+            0.01722511276602745,
+            -0.013494365848600864,
+            -0.0716434046626091,
+            0.014612773433327675,
+            0.02458864450454712,
+            -0.006706364918500185,
+            -0.017943505197763443,
+            0.008481939323246479,
+            0.00016148542636074126,
+            0.038597315549850464,
+            -0.014180104248225689,
+            -0.031103165820240974,
+            0.02680913358926773,
+            -0.017437364906072617,
+            -0.015241367742419243,
+            -0.010661610402166843,
+            -0.00580837344750762,
+            0.04649963974952698,
+            0.01889047957956791,
+            0.011420820839703083,
+            -0.04042595252394676,
+            -0.007449248805642128,
+            0.015078096650540829,
+            0.014506647363305092,
+            -0.04705476388335228,
+            0.0052083516493439674,
+            0.015486273914575577,
+            0.019723162055015564,
+            0.024800896644592285,
+            0.04274440184235573,
+            -0.0021531390957534313,
+            0.01661284640431404,
+            -0.015053605660796165,
+            -0.05136512219905853,
+            -0.03340528905391693,
+            0.029943939298391342,
+            -0.017731253057718277,
+            -0.02906227484345436,
+            -0.01693122461438179,
+            -0.006151242647320032,
+            0.0017939424142241478,
+            0.007126788143068552,
+            -0.01916803978383541,
+            -0.03464614972472191,
+            -0.03970756009221077,
+            0.02277633361518383,
+            0.007983962073922157,
+            0.004269542172551155,
+            -0.042777057737112045,
+            0.0013459670590236783,
+            0.009184005670249462,
+            0.012073906138539314,
+            0.0009954441338777542,
+            -0.019984396174550056,
+            0.015494437888264656,
+            0.0032001156359910965,
+            -0.031070511788129807,
+            -0.035593122243881226,
+            -0.03595232218503952,
+            0.0023919232189655304,
+            0.003781769424676895,
+            0.06279410421848297,
+            0.012988224625587463,
+            -0.03840138763189316,
+            -0.04522612690925598,
+            0.029176564887166023,
+            -0.02052319049835205,
+            -0.016678154468536377,
+            0.0059430720284581184,
+            0.012898425571620464,
+            -0.009877908043563366,
+            0.013192313723266125,
+            0.023560035973787308,
+            0.041601505130529404,
+            -0.005987971555441618,
+            0.012914752587676048,
+            0.023037567734718323,
+            0.012955570593476295,
+            0.0362788625061512,
+            -0.00936360377818346,
+            0.0035409443080425262,
+            0.039936136454343796,
+            -0.012269831262528896,
+            0.03175625205039978,
+            -0.02189466916024685,
+            0.0020398697815835476,
+            0.02249877341091633,
+            0.027184655889868736,
+            -0.0006479826406575739,
+            0.02594379521906376,
+            -0.031543996185064316,
+            0.018368011340498924,
+            -0.014498483389616013,
+            -0.04888340085744858,
+            -0.01557607389986515,
+            0.013102514669299126,
+            0.030384771525859833,
+            0.02756018005311489,
+            -0.0041348435916006565,
+            -0.019723162055015564,
+            -0.01155143789947033,
+            -0.010596301406621933,
+            0.0018419033149257302,
+            0.03285016864538193,
+            0.010506502352654934,
+            0.010890189558267593,
+            -0.011935125105082989,
+            -0.004104230087250471,
+            0.03025415539741516,
+            -0.021192602813243866,
+            0.01650671847164631,
+            0.0041511706076562405,
+            -0.002029665280133486,
+            -0.04320156201720238,
+            -0.035266581922769547,
+            0.007845181971788406,
+            0.011747363954782486,
+            0.014874007552862167,
+            0.0368339866399765,
+            0.025094784796237946,
+            -0.01662100851535797,
+            0.030629679560661316,
+            -0.01905374974012375,
+            0.027984686195850372,
+            0.039054471999406815,
+            0.011624909937381744,
+            -0.018351683393120766,
+            0.012237177230417728,
+            0.002424577483907342,
+            -0.025127438828349113,
+            0.01680060662329197,
+            -0.032735876739025116,
+            0.01738838478922844,
+            -0.02354370802640915,
+            0.025143766775727272,
+            -0.02579685114324093,
+            -0.009290131740272045,
+            -0.004428731743246317,
+            -0.006130834110081196,
+            0.04888340085744858,
+            -0.026727497577667236,
+            -0.0142454132437706,
+            -0.0024694770108908415,
+            -0.0024531499948352575,
+            -0.008718682453036308,
+            0.010000362060964108,
+            -0.045781247317790985,
+            -0.01844964735209942,
+            -0.04016471654176712,
+            0.007355368230491877,
+            0.005412440747022629,
+            -0.030907239764928818,
+            -0.05694899708032608,
+            0.025421326979994774,
+            -0.06331657618284225,
+            -0.019886434078216553,
+            0.051626358181238174,
+            -0.018400665372610092,
+            -0.008212542161345482,
+            0.009641164913773537,
+            0.006375740747898817,
+            -0.029290854930877686,
+            0.0041348435916006565,
+            0.02504580281674862,
+            0.04418119043111801,
+            -0.0045797573402523994,
+            -0.02865409664809704,
+            0.0028674507047981024,
+            -0.027788760140538216,
+            -0.013461710885167122,
+            0.005261414684355259,
+            0.016449574381113052,
+            0.024539662525057793,
+            0.007628847379237413,
+            0.03585435822606087,
+            0.007294141221791506,
+            0.006318595726042986,
+            0.044279150664806366,
+            -0.025666235014796257,
+            -0.027380581945180893,
+            0.02519274689257145,
+            0.04045860469341278,
+            0.023151857778429985,
+            -0.018253721296787262,
+            -0.0142454132437706,
+            0.0053063142113387585,
+            0.0009296254720538855,
+            -0.022139575332403183,
+            0.002808264922350645,
+            -0.013902543112635612,
+            0.005143043119460344,
+            0.03892385587096214,
+            -0.001205145614221692,
+            0.014196431264281273,
+            0.007302304729819298,
+            0.006845145486295223,
+            0.004334850702434778,
+            -0.011731036007404327,
+            -0.023592690005898476,
+            0.00032220553839579225,
+            0.034972693771123886,
+            0.015404638834297657,
+            -0.030515389516949654,
+            -0.00995954405516386,
+            0.034711457788944244,
+            -0.00795130804181099,
+            -0.009118696674704552,
+            -0.009771781973540783,
+            0.003385836724191904,
+            0.02638462744653225,
+            -0.01606588624417782,
+            0.006649219896644354,
+            0.03716052696108818,
+            -0.019870106130838394,
+            0.005543057806789875,
+            -0.025405000895261765,
+            0.0018061877926811576,
+            0.02442537248134613,
+            -0.006216551177203655,
+            0.028098974376916885,
+            5.6315813708351925e-05,
+            0.03025415539741516,
+            -0.004677720367908478,
+            0.008432958275079727,
+            -0.020327266305685043,
+            0.03987082839012146,
+            -0.017633290961384773,
+            0.03402572125196457,
+            0.02067013457417488,
+            -0.01258004643023014,
+            0.04783846437931061,
+            -0.036801330745220184,
+            -0.020784424617886543,
+            0.00584919098764658,
+            0.012743317522108555,
+            0.0028919412288814783,
+            -0.0191517136991024,
+            -0.018400665372610092,
+            -0.010996315628290176,
+            0.02380494214594364,
+            -0.022547753527760506,
+            -0.016865916550159454,
+            -0.007441085297614336,
+            -0.01890680566430092,
+            -0.009673818945884705,
+            -0.007988044060766697,
+            -0.01709449477493763,
+            -0.016547536477446556,
+            0.044573038816452026,
+            -0.06334923207759857,
+            0.016996532678604126,
+            0.017159804701805115,
+            0.011477965861558914,
+            -0.034678805619478226,
+            -0.019510909914970398,
+            -0.017633290961384773,
+            -0.002347023691982031,
+            -0.030743969604372978,
+            0.001702102366834879,
+            0.01725776679813862,
+            0.01138000376522541,
+            -0.05022222548723221,
+            0.012335140258073807,
+            0.02772345207631588,
+            -0.012065742164850235,
+            0.0014949520118534565,
+            -0.023135529831051826,
+            -0.017176130786538124,
+            0.0031776658724993467,
+            0.022123249247670174,
+            0.005975726060569286,
+            -0.008188051171600819,
+            -0.016865916550159454,
+            -0.005241005681455135,
+            -0.006045116577297449,
+            -0.002359268954023719,
+            0.005094061605632305,
+            0.011780017986893654,
+            0.019690508022904396,
+            -0.0024633544962853193,
+            -0.004436895251274109,
+            -0.006396149750798941,
+            0.027364253997802734,
+            -0.001450052484869957,
+            -0.008416631259024143,
+            0.050907962024211884,
+            0.03556047007441521,
+            0.006094098091125488,
+            -0.01108611561357975,
+            -0.03990348428487778,
+            0.010661610402166843,
+            -0.022711025550961494,
+            0.0401320643723011,
+            0.01905374974012375,
+            -0.008298259228467941,
+            -0.028866350650787354,
+            0.010294250212609768,
+            -0.02290695160627365,
+            -0.009975871071219444,
+            0.009469729848206043,
+            0.00726556871086359,
+            -0.03263791650533676,
+            -0.013543346896767616,
+            0.03510330989956856,
+            0.011437147855758667,
+            -0.006841063499450684,
+            -0.036050282418727875,
+            0.008588065393269062,
+            0.01646590232849121,
+            0.011869817040860653,
+            -0.019233347848057747,
+            0.004449140280485153,
+            -0.012041252106428146,
+            0.017470018938183784,
+            0.008849299512803555,
+            -0.017796561121940613,
+            0.018237393349409103,
+            0.004710374400019646,
+            -0.008490103296935558,
+            -0.003912386484444141,
+            -0.0163107942789793,
+            -0.02006603218615055,
+            0.03840138763189316,
+            0.024490682408213615,
+            0.026335645467042923,
+            0.005457340274006128,
+            0.02008235827088356,
+            0.03227871656417847,
+            0.011282040737569332,
+            -0.03134807199239731,
+            0.027935704216361046,
+            0.016996532678604126,
+            -0.04604247957468033,
+            0.021551799029111862,
+            -0.02218855731189251,
+            -0.028670424595475197,
+            0.009649328887462616,
+            -0.0017367975087836385,
+            0.011192241683602333,
+            0.0034837995190173388,
+            -0.003642988856881857,
+            -0.03307874873280525,
+            -0.006236960180103779,
+            0.015771998092532158,
+            0.015502601861953735,
+            -0.0048695639707148075,
+            -0.003296037670224905,
+            -0.011886144056916237,
+            0.00487364549189806,
+            0.009796272963285446,
+            0.039054471999406815,
+            0.001381682581268251,
+            -0.004751192405819893,
+            -0.0018765984568744898,
+            0.031854212284088135,
+            0.0271193478256464,
+            -0.017274094745516777,
+            -0.026286665350198746,
+            0.00580837344750762,
+            -0.014253576286137104,
+            0.026156047359108925,
+            -0.00487364549189806,
+            0.011877980083227158,
+            0.008718682453036308,
+            -0.0007908449042588472,
+            -0.001373519073240459,
+            0.025584599003195763,
+            -0.02710302174091339,
+            -0.021764053031802177,
+            0.031233781948685646,
+            -0.025731543079018593,
+            0.028735732659697533,
+            0.013061696663498878,
+            -0.01557607389986515,
+            -0.015225040726363659,
+            -0.010302413254976273,
+            0.009943217039108276,
+            -0.012277995236217976,
+            0.023853924125432968,
+            0.003608293831348419,
+            0.006392067763954401,
+            0.0003584313380997628,
+            0.02770712412893772,
+            0.024833550676703453,
+            0.007820690982043743,
+            -0.018074123188853264,
+            -0.0016816934803500772,
+            -0.02320083975791931,
+            -0.01680060662329197,
+            0.023772288113832474,
+            -0.018090449273586273,
+            -0.03268689662218094,
+            -0.028556134551763535,
+            0.012114724144339561,
+            0.007392104249447584,
+            0.010816717520356178,
+            0.03461349755525589,
+            -0.030384771525859833,
+            -0.0344502255320549,
+            0.01766594499349594,
+            0.026270337402820587,
+            -0.005551221314817667,
+            -0.021535472944378853,
+            0.03085825778543949,
+            -0.005828781984746456,
+            0.017600636929273605,
+            0.0404912605881691,
+            -0.0016225076979026198,
+            0.010228941217064857,
+            -0.042026009410619736,
+            -0.014057651162147522,
+            -0.0066614653915166855,
+            0.006261450704187155,
+            0.034417569637298584,
+            0.01048201136291027,
+            0.01535565685480833,
+            0.03448288142681122,
+            -0.021192602813243866,
+            -0.0002903166168835014,
+            -0.0022449791431427,
+            -0.02818061038851738,
+            0.03347059711813927,
+            0.025715215131640434,
+            0.011918798089027405,
+            0.019706835970282555,
+            -0.03794422745704651,
+            -0.013037205673754215,
+            0.027070365846157074,
+            0.015053605660796165,
+            0.01273515447974205,
+            0.016694480553269386,
+            0.011249386705458164,
+            0.00868602842092514,
+            -0.021633435040712357,
+            -0.05169166624546051,
+            0.01214737817645073,
+            0.01993541419506073,
+            -0.016449574381113052,
+            0.01057181041687727,
+            0.024980494752526283,
+            0.007620683871209621,
+            -0.007045152597129345,
+            0.012808626517653465,
+            -0.02682545967400074,
+            0.002259265398606658,
+            0.0019010890973731875,
+            -0.014482156373560429,
+            0.038270771503448486,
+            -0.016367938369512558,
+            0.011363675817847252,
+            -0.018204739317297935,
+            0.023870250210165977,
+            0.00011531029304023832,
+            0.0007964573451317847,
+            0.014963806606829166,
+            -0.011347348801791668,
+            -0.04235255345702171,
+            0.0021613026037812233,
+            0.03342161700129509,
+            -0.015200549736618996,
+            0.026449935510754585,
+            -0.0020674217958003283,
+            0.032458316534757614,
+            0.042777057737112045,
+            0.017731253057718277,
+            0.0003497575526125729,
+            0.025682561099529266,
+            0.0036144163459539413,
+            0.0073880222626030445,
+            0.012400448322296143,
+            0.009322785772383213,
+            0.007245160173624754,
+            0.030809277668595314,
+            -0.010261595249176025,
+            0.02173139713704586,
+            -0.04760988429188728,
+            -0.014629100449383259,
+            0.012024925090372562,
+            -0.0013316808035597205,
+            -0.012449429370462894,
+            -0.033029764890670776,
+            0.012563719414174557,
+            -0.03503800183534622,
+            0.022074267268180847,
+            -0.0004959363141097128,
+            0.04333217814564705,
+            -0.02187834307551384,
+            -0.005518566817045212,
+            0.016694480553269386,
+            0.0016510800924152136,
+            0.023282473906874657,
+            -0.0012153501156717539,
+            -0.016416920349001884,
+            -0.0067798369564116,
+            -0.009257477708160877,
+            0.004138925112783909,
+            0.04375668615102768,
+            -0.003722583642229438,
+            -0.0001682458823779598,
+            0.01409030519425869,
+            -0.011959616094827652,
+            -0.03250729665160179,
+            0.045193471014499664,
+            -0.0030837850645184517,
+            0.02484987862408161,
+            0.04979771748185158,
+            -0.06410028040409088,
+            -0.019543563947081566,
+            0.013061696663498878,
+            -0.010441194288432598,
+            0.011363675817847252,
+            0.005547139327973127,
+            0.016114868223667145,
+            0.017894525080919266,
+            -0.005485912784934044,
+            0.01619650423526764,
+            0.005106307100504637,
+            0.028098974376916885,
+            -0.02336410991847515,
+            -0.03296445682644844,
+            -0.02636830136179924,
+            -0.017747581005096436,
+            0.0014480114914476871,
+            -0.015796490013599396,
+            -3.563585551091819e-06,
+            -0.0022204883862286806,
+            -0.04388730227947235,
+            -0.025274382904171944,
+            0.040687184780836105,
+            0.006706364918500185,
+            0.02458864450454712,
+            0.0077268099412322044,
+            0.03418898954987526,
+            0.005885927006602287,
+            -0.009037061594426632,
+            0.030058229342103004,
+            -0.020049704238772392,
+            0.03833607956767082,
+            0.022106921300292015,
+            -0.002832755446434021,
+            0.005865518003702164,
+            0.031086837872862816,
+            -0.033797141164541245,
+            0.008086007088422775,
+            -0.034384917467832565,
+            0.0046695563942193985,
+            -0.008620720356702805,
+            -0.015094423666596413,
+            -0.009526874870061874,
+            -0.02504580281674862,
+            0.011216731742024422,
+            -0.021535472944378853,
+            0.01590261608362198,
+            0.010416703298687935,
+            -0.002916431985795498,
+            0.03448288142681122,
+            -0.014204595237970352,
+            -0.0027796924114227295,
+            0.0024531499948352575,
+            -0.027070365846157074,
+            -0.006343086715787649,
+            -0.003377673216164112,
+            -0.03689929470419884,
+            -0.004502203781157732,
+            0.010988152585923672,
+            -0.00024184548237826675,
+            0.016898570582270622,
+            0.02787039615213871,
+            0.0005092020728625357,
+            -0.022302847355604172,
+            0.002340900944545865,
+            -0.04199335351586342,
+            0.0011877981014549732,
+            0.01483318954706192,
+            -0.017600636929273605,
+            -0.008612556383013725,
+            0.033046092838048935,
+            0.023298801854252815,
+            -0.009396257810294628,
+            -0.035005345940589905,
+            0.017796561121940613,
+            -0.011853490024805069,
+            0.027511198073625565,
+            -0.0002630197268445045,
+            0.019298657774925232,
+            -0.0076941559091210365,
+            0.04493223875761032,
+            0.001369437319226563,
+            0.008645210415124893,
+            -0.021959977224469185,
+            0.007020662073045969,
+            -0.020278284326195717,
+            -0.002406209474429488,
+            -0.006171651650220156,
+            -0.012751481495797634,
+            -0.010669773444533348,
+            0.0032735876739025116,
+            0.007730891928076744,
+            -0.0176822729408741,
+            0.0055961208418011665,
+            -0.015959760174155235,
+            -0.02741323597729206,
+            0.007543129846453667,
+            -0.05965929850935936,
+            -0.0179598331451416,
+            -0.03373183310031891,
+            -0.009477893821895123,
+            0.02008235827088356,
+            0.03510330989956856,
+            -0.012759645469486713,
+            0.00497977202758193,
+            -0.014898497611284256,
+            0.004477712791413069,
+            0.03490738570690155,
+            0.044442422688007355,
+            -0.014759717509150505,
+            0.003381754970178008,
+            -0.010955498553812504,
+            0.033029764890670776,
+            -0.03347059711813927,
+            0.0030062312725931406,
+            0.031086837872862816,
+            -0.005057325586676598,
+            0.007959471084177494,
+            0.011624909937381744,
+            0.020474210381507874,
+            -0.019396619871258736,
+            0.00877582747489214,
+            0.002100075827911496,
+            0.019233347848057747,
+            0.02171507105231285,
+            0.020343592390418053,
+            -0.03539719805121422,
+            0.028409190475940704,
+            0.018384337425231934,
+            0.02444170042872429,
+            0.044442422688007355,
+            -0.03974021226167679,
+            -0.004147088620811701,
+            -0.0032225653994828463,
+            -0.02997659333050251,
+            0.02008235827088356,
+            -0.007530884351581335,
+            -0.01841699331998825,
+            -0.007069643586874008,
+            0.03327467292547226,
+            0.014571955427527428,
+            -0.05051611363887787,
+            0.01980479806661606,
+            0.01740471087396145,
+            0.02055584453046322,
+            0.024474354460835457,
+            -0.01123305968940258,
+            -0.0019072118448093534,
+            0.008996243588626385,
+            0.006804327480494976,
+            0.019266001880168915,
+            0.015257694758474827,
+            0.002012317767366767,
+            -0.0022082431241869926,
+            -0.011837163008749485,
+            0.005224678665399551,
+            -0.009077879600226879,
+            0.010808554477989674,
+            -0.02458864450454712,
+            -0.009698309935629368,
+            0.021388528868556023,
+            0.0031164393294602633,
+            -0.013665799982845783,
+            -0.03729114308953285,
+            -0.045944519340991974,
+            0.0033837957307696342,
+            0.004543021321296692,
+            -0.05365091934800148,
+            0.005322641227394342,
+            -0.026547899469733238,
+            -0.002108239568769932,
+            -0.005502239800989628,
+            0.0019214981002733111,
+            0.005636938381940126,
+            0.0167679525911808,
+            -0.013494365848600864,
+            0.02248244546353817,
+            -0.015502601861953735,
+            0.03165828809142113,
+            0.0055961208418011665,
+            0.007375776767730713,
+            -0.0027796924114227295,
+            0.05590406060218811,
+            -0.003712379140779376,
+            -0.03699725493788719,
+            0.023282473906874657,
+            -0.01311067771166563,
+            -0.0001747512142173946,
+            0.030090883374214172,
+            0.00884113647043705,
+            -0.015257694758474827,
+            0.013437220826745033,
+            -0.030613351613283157,
+            -0.03249097242951393,
+            0.010996315628290176,
+            -0.013755599036812782,
+            -0.0015776080545037985,
+            -0.04568328335881233,
+            -0.009339112788438797,
+            0.004269542172551155,
+            -0.018335357308387756,
+            0.010775899514555931,
+            0.01631895825266838,
+            0.013992343097925186,
+            -0.024245774373412132,
+            0.009053388610482216,
+            0.006865554489195347,
+            0.013453547842800617,
+            0.0007331897504627705,
+            0.014049487188458443,
+            -0.008677864447236061,
+            -0.011306531727313995,
+            -0.018319029361009598,
+            0.0032450153958052397,
+            -0.026792805641889572,
+            0.005155288614332676,
+            -0.003155216109007597,
+            -0.03771565109491348,
+            -0.06521052122116089,
+            -0.0344502255320549,
+            0.016294466331601143,
+            0.03396041318774223,
+            -0.009045224636793137,
+            0.0035144127905368805,
+            -0.00023648813657928258,
+            -0.01093917153775692,
+            0.024523336440324783,
+            -0.005930826533585787,
+            -0.0004829256213270128,
+            0.0035389035474509,
+            0.02050686441361904,
+            -0.009722800925374031,
+            0.005845109466463327,
+            -0.051952898502349854,
+            -0.01586179807782173,
+            0.01812310516834259,
+            0.006306350696831942,
+            0.014629100449383259,
+            -0.009690145961940289,
+            0.01693122461438179,
+            -0.030074557289481163,
+            -0.028147956356406212,
+            0.04130761697888374,
+            0.008277850225567818,
+            0.011486129835247993,
+            -0.003055212553590536,
+            -0.01680060662329197,
+            0.02504580281674862,
+            -0.0020633398089557886,
+            -0.006461458280682564,
+            -0.00817988719791174,
+            0.015820980072021484,
+            0.044279150664806366,
+            0.003255219664424658,
+            0.0009821783751249313,
+            0.017045514658093452,
+            -0.002663361607119441,
+            -0.011575928889214993,
+            0.007677828427404165,
+            0.013592327944934368,
+            -0.016384266316890717,
+            0.010375885292887688,
+            -0.025862159207463264,
+            0.0332583449780941,
+            0.03131541982293129,
+            0.011575928889214993,
+            -0.015045441687107086,
+            0.006130834110081196,
+            0.005330805201083422,
+            0.008086007088422775,
+            -0.001369437319226563,
+            -0.0017133272485807538,
+            0.020719116553664207,
+            -0.03276853263378143,
+            -0.01693122461438179,
+            -0.005347132217139006,
+            0.003932795487344265,
+            0.005277741700410843,
+            0.03396041318774223,
+            -0.0033266509417444468,
+            0.017927179113030434,
+            -0.011657563969492912,
+            0.010579974390566349,
+            0.021764053031802177,
+            0.037095218896865845,
+            -0.024996822699904442,
+            -0.029862305149435997,
+            0.012914752587676048,
+            -0.001296985661610961,
+            0.014065815135836601,
+            0.0010173837654292583,
+            0.014139287173748016,
+            0.0029409227427095175,
+            0.060475654900074005,
+            -0.017763907089829445,
+            0.004628738854080439,
+            0.0004035860183648765,
+            -0.01858026348054409,
+            -0.0013531101867556572,
+            0.004261378664523363,
+            -0.010979988612234592,
+            0.03608293831348419,
+            0.002344982698559761,
+            0.039185091853141785,
+            0.013918871060013771,
+            0.009290131740272045,
+            0.0015143405180424452,
+            -0.013592327944934368,
+            0.008355404250323772,
+            0.0015592400450259447,
+            -0.001132694073021412,
+            0.009461566805839539,
+            0.019298657774925232,
+            0.009714636951684952,
+            0.025274382904171944,
+            0.009788108989596367,
+            0.012857607565820217,
+            -0.029470453038811684,
+            0.03510330989956856,
+            -0.0006224715034477413,
+            0.012996388599276543,
+            -0.02040890045464039,
+            -0.003136848099529743,
+            -0.0008240094175562263,
+            0.015649545937776566,
+            -0.015616890974342823,
+            -0.001201063860207796,
+            -0.01297189760953188,
+            0.040099408477544785,
+            -0.018988441675901413,
+            0.013976015150547028,
+            0.009673818945884705,
+            -0.02865409664809704,
+            -0.007596192881464958,
+            0.01995174214243889,
+            0.014294394291937351,
+            -0.0053144777193665504,
+            -0.020490536466240883,
+            0.005832863971590996,
+            0.015380147844552994,
+            -0.028572462499141693,
+            1.3584674888988957e-05,
+            0.014751553535461426,
+            -0.0164903923869133,
+            -0.00025409081717953086,
+            -0.004702210891991854,
+            0.003975654020905495,
+            -0.009265640750527382,
+            0.007028825581073761,
+            0.016057724133133888,
+            -0.024507008492946625,
+            -0.020049704238772392,
+            0.005722655914723873,
+            0.006571666337549686,
+            0.01444950234144926,
+            0.007739055436104536,
+            0.004355259705334902,
+            -0.020033378154039383,
+            -0.01799248717725277,
+            0.0032674651592969894,
+            -0.01649855636060238,
+            -0.0048614004626870155,
+            0.0020531355403363705,
+            -0.019363965839147568,
+            0.01875986158847809,
+            -0.010743245482444763,
+            0.00017538899555802345,
+            -0.003226647386327386,
+            -0.020800752565264702,
+            0.006204306147992611,
+            -0.004865481983870268,
+            0.019870106130838394,
+            0.003555230563506484,
+            0.02785406820476055,
+            0.02310287579894066,
+            -0.004428731743246317,
+            0.017600636929273605,
+            -0.016155686229467392,
+            0.007983962073922157,
+            -0.031135819852352142,
+            -0.001782717532478273,
+            -0.010408539324998856,
+            0.0036103345919400454,
+            -0.015943434089422226,
+            -0.009126860648393631,
+            -0.0048777274787425995,
+            -0.0006678812787868083,
+            -0.004820582456886768,
+            0.0004275664687156677,
+            0.0027817331720143557,
+            0.02231917530298233,
+            -0.013208640739321709,
+            -0.02803366631269455,
+            -0.017437364906072617,
+            0.006045116577297449,
+            0.019396619871258736,
+            -0.026025431230664253,
+            -0.009184005670249462,
+            -0.005012426059693098,
+            -0.004326687194406986,
+            -0.021404854953289032,
+            -0.015167895704507828,
+            -0.005632856860756874,
+            -0.008045189082622528,
+            -0.010522829368710518,
+            0.0046532293781638145,
+            -0.0035715578123927116,
+            -0.011739199981093407,
+            0.021666089072823524,
+            0.018057795241475105,
+            -0.019837452098727226,
+            -0.019837452098727226,
+            0.013641309924423695,
+            -0.006159406621009111,
+            0.0018878233386203647,
+            0.02008235827088356,
+            0.00029286774224601686,
+            0.005824700463563204,
+            0.020882388576865196,
+            -0.0035225762985646725,
+            -0.01797616109251976,
+            0.003577680326998234,
+            -0.0018888438353314996,
+            -0.012122887186706066,
+            0.003212361130863428,
+            0.02773977816104889,
+            0.0008076822850853205,
+            0.023429417982697487,
+            -0.03056436963379383,
+            -0.011412657797336578,
+            0.006061443593353033,
+            0.02369065210223198,
+            0.00211844383738935,
+            -0.0021694661118090153,
+            0.020131340250372887,
+            0.03941367194056511,
+            0.005775718949735165,
+            0.0015755671774968505,
+            -0.04003410041332245,
+            0.01027792226523161,
+            0.009208496659994125,
+            0.024621298536658287,
+            0.008339077234268188,
+            0.0014653591206297278,
+            -0.012906589545309544,
+            -0.009755454957485199,
+            -0.0017939424142241478,
+            0.014702572487294674,
+            -0.023870250210165977,
+            0.0024755997583270073,
+            -0.03157665207982063,
+            0.005094061605632305,
+            0.0021959978621453047,
+            -0.009559528902173042,
+            0.0335359051823616,
+            0.00980443600565195,
+            -0.02065380848944187,
+            0.015151568688452244,
+            0.005767555441707373,
+            0.007330877240747213,
+            0.022449791431427002,
+            -0.008228869177401066,
+            0.030450081452727318,
+            -0.005049162078648806,
+            0.01213921420276165,
+            0.02742956392467022,
+            0.032588932663202286,
+            -0.013102514669299126,
+            -0.003959327004849911,
+            0.0006097159348428249,
+            -0.0002561317232903093,
+            0.0007995187188498676,
+            -0.008204378187656403,
+            0.0037389106582850218,
+            -0.014800535514950752,
+            0.012710663489997387,
+            -0.019543563947081566,
+            -0.0034286954905837774,
+            0.0038532004691660404,
+            0.01991908811032772,
+            0.0016265894519165158,
+            -0.033633869141340256,
+            0.02055584453046322,
+            -0.015420965850353241,
+            0.0135596739128232,
+            0.001290863030590117,
+            -0.011110605672001839,
+            -0.00029210239881649613,
+            0.020588500425219536,
+            0.010947334580123425,
+            -0.007845181971788406,
+            -0.0135596739128232,
+            -0.028376536443829536,
+            -0.021535472944378853,
+            0.00823703221976757,
+            0.009445239789783955,
+            -0.0028837777208536863,
+            0.0021306893322616816,
+            -0.011869817040860653,
+            0.0006520643946714699,
+            -0.0008122742874547839,
+            0.01740471087396145,
+            -0.0335359051823616,
+            0.021943651139736176,
+            -0.019527235999703407,
+            -0.019543563947081566,
+            0.017045514658093452,
+            0.008049271069467068,
+            0.009445239789783955,
+            0.0033552234526723623,
+            -0.01995174214243889,
+            -0.012326976284384727,
+            0.0338951013982296,
+            -0.01138000376522541,
+            0.006298186723142862,
+            0.004408322740346193,
+            0.012914752587676048,
+            -0.008914608508348465,
+            -0.011926962062716484,
+            0.01784554310142994,
+            -0.009861581027507782,
+            0.013951525092124939,
+            0.0010010566329583526,
+            -0.0254703089594841,
+            0.02739690989255905,
+            -0.019233347848057747,
+            0.03487472981214523,
+            -0.021617108955979347,
+            0.00711862463504076,
+            0.0066614653915166855,
+            0.008792154490947723,
+            -0.006690037902444601,
+            -0.013224967755377293,
+            0.0012786176521331072,
+            0.03807484731078148,
+            -0.02623768337070942,
+            0.024180466309189796,
+            0.022400809451937675,
+            -0.00040766780148260295,
+            0.02065380848944187,
+            -0.008367649279534817,
+            -0.026711169630289078,
+            0.0179598331451416,
+            -0.012596373446285725,
+            -0.026123393326997757,
+            0.013257622718811035,
+            -0.022678371518850327,
+            -0.020278284326195717,
+            -0.026302991434931755,
+            0.004947117529809475,
+            -0.0025694805663079023,
+            -0.005151206627488136,
+            0.009004407562315464,
+            0.02561725303530693,
+            0.020915042608976364,
+            0.019576217979192734,
+            0.02111096680164337,
+            0.01161674689501524,
+            0.022449791431427002,
+            -0.007559456862509251,
+            0.04401791840791702,
+            -0.004947117529809475,
+            0.004632820375263691,
+            -0.0015980169409886003,
+            -0.015657709911465645,
+            0.009771781973540783,
+            0.0119106350466609,
+            0.0029572497587651014,
+            -0.029176564887166023,
+            -0.009061552584171295,
+            -0.03660540655255318,
+            0.005053244065493345,
+            0.0020664012990891933,
+            0.0176822729408741,
+            0.032115448266267776,
+            -0.005865518003702164,
+            0.004592002835124731,
+            -0.005102225113660097,
+            -0.013927034102380276,
+            -0.009420748800039291,
+            -0.02739690989255905,
+            -0.025437654927372932,
+            -0.007653337903320789,
+            -0.013739272020757198,
+            -0.030368445441126823,
+            -0.011722872965037823,
+            0.01551892887800932,
+            -0.007645174395292997,
+            0.011918798089027405,
+            -0.016702644526958466,
+            0.01385356206446886,
+            -0.004110352601855993,
+            -0.004763437435030937,
+            -0.004530776292085648,
+            -0.017600636929273605,
+            -0.006420640274882317,
+            0.014367866329848766,
+            -0.009477893821895123,
+            -0.01237595733255148,
+            -0.02215590327978134,
+            -0.0015459742862731218,
+            -0.01574750803411007,
+            0.003489922033622861,
+            -0.014596446417272091,
+            -0.006771673448383808,
+            -0.003908304497599602,
+            -0.018514955416321754,
+            0.01766594499349594,
+            0.014816862531006336,
+            -0.019380291923880577,
+            -0.0090697156265378,
+            0.02638462744653225,
+            0.042777057737112045,
+            -0.03218075633049011,
+            -0.008200296200811863,
+            0.0020378287881612778,
+            0.018090449273586273,
+            0.010171796195209026,
+            0.016702644526958466,
+            -0.024653952568769455,
+            0.0022286521270871162,
+            -0.021649762988090515,
+            -0.001220452249981463,
+            -0.023739634081721306,
+            0.010775899514555931,
+            0.025715215131640434,
+            -0.010971825569868088,
+            0.025976449251174927,
+            0.01634344831109047,
+            -0.01812310516834259,
+            0.012016761116683483,
+            0.007183933164924383,
+            0.014645427465438843,
+            -0.019755816087126732,
+            0.0167679525911808,
+            -0.015796490013599396,
+            0.031233781948685646,
+            0.00892277155071497,
+            -0.04829562455415726,
+            -0.03503800183534622,
+            -0.013388238847255707,
+            0.0013388239312916994,
+            -0.010702427476644516,
+            0.015037278644740582,
+            0.015967924147844315,
+            0.013959688134491444,
+            -0.019380291923880577,
+            -0.015820980072021484,
+            0.011984107084572315,
+            0.010065670125186443,
+            0.027462217956781387,
+            -0.006649219896644354,
+            -0.004212397150695324,
+            -0.00960034690797329,
+            -0.02249877341091633,
+            -0.016098542138934135,
+            0.0209803506731987,
+            -0.018041469156742096,
+            0.015935270115733147,
+            -0.026139721274375916,
+            -0.012449429370462894,
+            0.017453692853450775,
+            -0.05211617052555084,
+            -0.01453113742172718,
+            0.0007612520130351186,
+            0.03012353740632534,
+            0.004751192405819893,
+            0.0035184945445507765,
+            0.01841699331998825,
+            -0.01993541419506073,
+            -0.0039776950143277645,
+            -0.02099667675793171,
+            0.006832899991422892,
+            -0.011853490024805069,
+            -0.013984179124236107,
+            0.014482156373560429,
+            -0.013845399022102356,
+            -0.01601690612733364,
+            0.0034368589986115694,
+            0.007183933164924383,
+            -0.021094640716910362,
+            0.011951452121138573,
+            0.003945040516555309,
+            0.012171869166195393,
+            0.012171869166195393,
+            0.01199227012693882,
+            0.021551799029111862,
+            0.02295593172311783,
+            -0.007334959227591753,
+            -0.004767519421875477,
+            -0.01004117913544178,
+            -0.0015398516552522779,
+            0.008898280560970306,
+            0.0033797139767557383,
+            -0.002989904023706913,
+            0.007837017998099327,
+            0.010033016093075275,
+            -0.01498013362288475,
+            0.033193036913871765,
+            -0.0035878848284482956,
+            -0.04134026914834976,
+            -0.003238892648369074,
+            -0.03800953924655914,
+            0.020033378154039383,
+            -0.023298801854252815,
+            -0.016996532678604126,
+            0.0032776694279164076,
+            -0.00629410520195961,
+            -0.019902760162949562,
+            0.013608654960989952,
+            0.011404493823647499,
+            0.00258580781519413,
+            0.02021297626197338,
+            0.012269831262528896,
+            -0.005134879611432552,
+            0.026172375306487083,
+            -0.014718899503350258,
+            -0.008751336485147476,
+            0.015567909926176071,
+            0.006853308994323015,
+            0.0022347746416926384,
+            0.0011031010653823614,
+            -0.018825169652700424,
+            0.005098143592476845,
+            0.0164903923869133,
+            0.007216587662696838,
+            -0.000566347036510706,
+            0.014049487188458443,
+            0.012514738366007805,
+            -0.015135240741074085,
+            -0.0002213090192526579,
+            -0.026564225554466248,
+            0.0032817514147609472,
+            0.004567512311041355,
+            0.007506393827497959,
+            -0.006771673448383808,
+            0.006281859707087278,
+            0.00658799335360527,
+            0.014759717509150505,
+            -0.010155469179153442,
+            -0.016980206593871117,
+            -0.002510294783860445,
+            0.004285869188606739,
+            0.016694480553269386,
+            0.03157665207982063,
+            -0.010310577228665352,
+            -0.005734901409596205,
+            -0.0386626236140728,
+            0.02070278860628605,
+            0.03332365304231644,
+            0.011175914667546749,
+            -0.028213264420628548,
+            0.012302485294640064,
+            0.0006515541463159025,
+            0.0019276207312941551,
+            -0.026302991434931755,
+            0.02787039615213871,
+            0.00974729098379612,
+            0.0066614653915166855,
+            0.017861871048808098,
+            0.0022306928876787424,
+            0.0029266364872455597,
+            -0.016302630305290222,
+            0.01018812321126461,
+            0.009086042642593384,
+            -0.017306748777627945,
+            0.0009745250572450459,
+            0.02324981987476349,
+            -0.01423724927008152,
+            -0.003208279376849532,
+            0.0018133309204131365,
+            0.02651524543762207,
+            -0.013820908032357693,
+            -0.014294394291937351,
+            0.0005714492290280759,
+            0.01751900091767311,
+            -0.0007979880319908261,
+            0.012098397128283978,
+            0.04898136481642723,
+            -0.012424939312040806,
+            0.019657853990793228,
+            0.005461421795189381,
+            -0.015290348790585995,
+            -0.032605260610580444,
+            -0.004922627005726099,
+            -0.01830270327627659,
+            0.011200404725968838,
+            0.014343376271426678,
+            -0.015763835981488228,
+            0.0007525782566517591,
+            -0.03144603595137596,
+            0.007743136957287788,
+            0.014735226519405842,
+            0.018776189535856247,
+            -0.002440904499962926,
+            0.006134915631264448,
+            0.027674470096826553,
+            0.02458864450454712,
+            -0.007392104249447584,
+            0.013804581016302109,
+            0.02320083975791931,
+            -0.025062130764126778,
+            -0.003069498809054494,
+            0.006820654962211847,
+            0.004104230087250471,
+            0.0053063142113387585,
+            -0.016963878646492958,
+            0.03915243595838547,
+            -0.005922663025557995,
+            -0.01506176870316267,
+            0.0030266400426626205,
+            -0.009028897620737553,
+            0.0063390047289431095,
+            -0.012563719414174557,
+            0.012637191452085972,
+            -0.008114579133689404,
+            0.00022883480414748192,
+            -0.0059430720284581184,
+            -0.011110605672001839,
+            -0.012653518468141556,
+            -0.0007949267164804041,
+            -0.016531210392713547,
+            0.007432921789586544,
+            0.00981259997934103,
+            0.005045080557465553,
+            0.01025343220680952,
+            -0.0036797248758375645,
+            -0.003136848099529743,
+            -0.016751626506447792,
+            -0.009984034113585949,
+            0.0073716952465474606,
+            0.01872720755636692,
+            -0.028588788583874702,
+            -0.018547609448432922,
+            0.009290131740272045,
+            0.0022898786701261997,
+            0.018776189535856247,
+            0.0158944521099329,
+            -0.0002595757250674069,
+            0.012318813242018223,
+            -0.01964152604341507,
+            -0.009184005670249462,
+            0.009690145961940289,
+            -0.003569516818970442,
+            0.01048201136291027,
+            -0.014759717509150505,
+            0.010033016093075275,
+            -0.011159587651491165,
+            -0.011053460650146008,
+            0.01385356206446886,
+            0.01738838478922844,
+            0.008873790502548218,
+            -0.001294944784604013,
+            3.383413059054874e-05,
+            -0.010506502352654934,
+            -0.024033522233366966,
+            0.004771600943058729,
+            0.027021385729312897,
+            -0.013273949734866619,
+            0.003883813973516226,
+            0.008702355436980724,
+            -0.017502672970294952,
+            -0.019821124151349068,
+            -0.012653518468141556,
+            -0.022106921300292015,
+            0.020572172477841377,
+            0.023413091897964478,
+            -0.02026195637881756,
+            0.00877582747489214,
+            -0.004620575346052647,
+            0.014049487188458443,
+            -0.01400050614029169,
+            0.010612628422677517,
+            0.0317889042198658,
+            -0.020000724121928215,
+            -0.0028654097113758326,
+            0.0036633978597819805,
+            -0.015347493812441826,
+            0.018629245460033417,
+            0.03268689662218094,
+            -0.00176945177372545,
+            0.014041324146091938,
+            0.0008138049161061645,
+            -0.009118696674704552,
+            0.01905374974012375,
+            0.011094278655946255,
+            0.005347132217139006,
+            -0.0042491331696510315,
+            -0.012212686240673065,
+            -0.03010721132159233,
+            0.016702644526958466,
+            0.0129637336358428,
+            -0.0037674831692129374,
+            -0.0011551438365131617,
+            -0.006343086715787649,
+            -0.015404638834297657,
+            0.0035123717971146107,
+            -0.013078023679554462,
+            -0.00803294312208891,
+            -0.008759500458836555,
+            0.004314441699534655,
+            -0.0034960447810590267,
+            -0.009053388610482216,
+            0.007236996199935675,
+            0.0009510547970421612,
+            -0.0010602424154058099,
+            -0.01902109570801258,
+            -0.025078458711504936,
+            -0.00995954405516386,
+            -0.016849588602781296,
+            0.0038287099450826645,
+            0.0005234883283264935,
+            -0.027184655889868736,
+            0.0053961132653057575,
+            -0.008139070123434067,
+            -0.008134988136589527,
+            0.007126788143068552,
+            0.006881881505250931,
+            -0.007437003776431084,
+            -0.010816717520356178,
+            -0.00786150898784399,
+            -0.006865554489195347,
+            0.020033378154039383,
+            0.0034348180051892996,
+            0.012082069180905819,
+            -0.010498338378965855,
+            -0.018025141209363937,
+            -0.016963878646492958,
+            0.016718972474336624,
+            -0.008971752598881721,
+            -0.02083340659737587,
+            -0.01444950234144926,
+            -0.008424794301390648,
+            -0.007665583398193121,
+            -0.01527402177453041,
+            -0.021159948781132698,
+            -0.0032103201374411583,
+            -0.011347348801791668,
+            -0.0067431009374558926,
+            -0.0028960229828953743,
+            0.0013571919407695532,
+            0.00951871182769537,
+            0.00546958576887846,
+            0.00438791373744607,
+            0.021666089072823524,
+            0.010122815147042274,
+            -0.005461421795189381,
+            0.016441410407423973,
+            -0.006183897145092487,
+            -0.004440976772457361,
+            -0.01498013362288475,
+            -0.009918726049363613,
+            0.018547609448432922,
+            -0.0055226488038897514,
+            0.008457448333501816,
+            -0.0009536059224046767,
+            -0.00576347392052412,
+            0.015674035996198654,
+            0.0129637336358428,
+            0.016718972474336624,
+            0.004771600943058729,
+            -0.00012168807734269649,
+            0.012416775338351727,
+            -0.0015745466807857156,
+            0.012555556371808052,
+            -0.0038429962005466223,
+            0.014506647363305092,
+            -0.009722800925374031,
+            -0.018988441675901413,
+            0.004157293122261763,
+            0.006265532691031694,
+            -0.012531065382063389,
+            0.009404421783983707,
+            0.007743136957287788,
+            -0.013747435994446278,
+            -0.02261306345462799,
+            -0.000106317616882734,
+            0.00014362763613462448,
+            0.008122743107378483,
+            0.008971752598881721,
+            -0.008073761127889156,
+            0.012890261597931385,
+            0.004616493359208107,
+            -0.007179851643741131,
+            -0.019657853990793228,
+            0.026009103283286095,
+            0.015608727931976318,
+            0.012269831262528896,
+            -0.052508022636175156,
+            -0.0030980713199824095,
+            0.010743245482444763,
+            -0.015951598063111305,
+            -0.026792805641889572,
+            -0.006404313258826733,
+            -0.008759500458836555,
+            -0.04290767386555672,
+            -0.009624837897717953,
+            -0.0005724696675315499,
+            -0.011388166807591915,
+            0.010400376282632351,
+            0.01573934406042099,
+            0.014612773433327675,
+            0.005489994306117296,
+            -0.007232914678752422,
+            -0.017633290961384773,
+            -0.02788672223687172,
+            0.010057506151497364,
+            -0.013380075804889202,
+            -0.006412476766854525,
+            -0.002263347152620554,
+            -0.007690073922276497,
+            0.0016112826997414231,
+            0.01691489666700363,
+            0.042483169585466385,
+            0.023037567734718323,
+            -0.012335140258073807,
+            -0.020131340250372887,
+            -0.012800462543964386,
+            -0.009192168712615967,
+            0.00633492274209857,
+            0.011592255905270576,
+            -0.004738946910947561,
+            0.0096656559035182,
+            -0.004294032696634531,
+            0.012302485294640064,
+            0.012767808511853218,
+            0.01680060662329197,
+            -2.3023791072773747e-05,
+            0.005579793825745583,
+            -0.0024694770108908415,
+            -0.01738838478922844,
+            -0.004379750229418278,
+            0.008261523209512234,
+            -0.0008872769540175796,
+            0.0019317024853080511,
+            0.014204595237970352,
+            0.019102731719613075,
+            -0.009094206616282463,
+            -0.014612773433327675,
+            -0.016098542138934135,
+            -0.005277741700410843,
+            -0.0012622905196622014,
+            0.029927613213658333,
+            0.02037624642252922,
+            -0.019266001880168915,
+            -0.018237393349409103,
+            -0.018351683393120766,
+            0.00922482367604971,
+            -0.003563394071534276,
+            -0.0064369672909379005,
+            0.01905374974012375,
+            0.023021239787340164,
+            0.017910851165652275,
+            -0.003642988856881857,
+            -0.014865843579173088,
+            0.003804219188168645,
+            0.02847449854016304,
+            0.008367649279534817,
+            0.01078406348824501,
+            0.008726846426725388,
+            -0.0011908593587577343,
+            -0.032441988587379456,
+            -0.0113146947696805,
+            0.00951871182769537,
+            -0.008481939323246479,
+            0.026564225554466248,
+            -0.016531210392713547,
+            0.009388094767928123,
+            -0.005077734589576721,
+            0.0019286411115899682,
+            -0.02365799807012081,
+            0.0014929111348465085,
+            0.010122815147042274,
+            -0.024947840720415115,
+            0.00336950970813632,
+            -0.023592690005898476,
+            -0.00013763252354692668,
+            0.008367649279534817,
+            0.0022837561555206776,
+            0.01650671847164631,
+            -0.0113146947696805,
+            0.0051838611252605915,
+            0.02099667675793171,
+            -0.002832755446434021,
+            -0.01512707769870758,
+            -0.0007842120248824358,
+            0.0005092020728625357,
+            -0.001286781276576221,
+            0.02563357912003994,
+            0.015812816098332405,
+            -0.0036756431218236685,
+            0.020882388576865196,
+            0.008677864447236061,
+            0.0019133344758301973,
+            -0.021764053031802177,
+            0.0023796779569238424,
+            -0.032899148762226105,
+            0.017437364906072617,
+            -0.011126932688057423,
+            0.012490247376263142,
+            0.00868602842092514,
+            -0.01341272983700037,
+            -0.0007388022495433688,
+            -0.0107187544927001,
+            -0.0015725058037787676,
+            -0.007934981025755405,
+            0.009779945947229862,
+            -0.004363423213362694,
+            0.008339077234268188,
+            -0.00024579971795901656,
+            -0.01476788055151701,
+            0.018694553524255753,
+            9.678283277025912e-06,
+            -0.00010925139940809458,
+            0.005441013257950544,
+            0.009935053065419197,
+            0.00786559097468853,
+            0.002009256277233362,
+            0.023870250210165977,
+            -0.031119491904973984,
+            -0.016082214191555977,
+            -0.010204450227320194,
+            0.005543057806789875,
+            -0.007922735065221786,
+            -0.018531281501054764,
+            -0.03257260471582413,
+            0.01318415068089962,
+            0.001293924287892878,
+            -0.0009806477464735508,
+            0.01258004643023014,
+            -0.00942891277372837,
+            -0.0013878053287044168,
+            -0.01606588624417782,
+            -0.014808698557317257,
+            0.006416558753699064,
+            0.0033633869607001543,
+            -0.01222901325672865,
+            -0.023739634081721306,
+            0.014759717509150505,
+            0.0007418635650537908,
+            -0.016980206593871117,
+            -0.017502672970294952,
+            0.00018457300029695034,
+            -0.004804255440831184,
+            0.02204161323606968,
+            -0.016678154468536377,
+            0.003373591462150216,
+            0.0007219648687168956,
+            0.009232986718416214,
+            -0.02889900468289852,
+            0.00716352416202426,
+            0.0006107363733462989,
+            0.0035164537839591503,
+            -0.006518602836877108,
+            -0.021013004705309868,
+            0.011371839791536331,
+            -0.0020674217958003283,
+            -0.014751553535461426,
+            -0.020751770585775375,
+            -0.0020796670578420162,
+            -0.007420676294714212,
+            -0.02724996581673622,
+            0.017910851165652275,
+            0.0009985055075958371,
+            -0.01738838478922844,
+            0.008077843114733696,
+            0.006596156861633062,
+            0.009355440735816956,
+            0.02455599047243595,
+            0.010147306136786938,
+            -0.018776189535856247,
+            -0.0008087027235887945,
+            0.006979844067245722,
+            0.009306458756327629,
+            0.0036450298503041267,
+            0.04222193360328674,
+            -0.008130906149744987,
+            -0.006579829845577478,
+            0.0066614653915166855,
+            -0.008449285291135311,
+            -0.013747435994446278,
+            0.026727497577667236,
+            0.005796127952635288,
+            0.015984252095222473,
+            0.016123032197356224,
+            -0.00945340283215046,
+            -0.005008344538509846,
+            -0.0018786393338814378,
+            0.02065380848944187,
+            0.0073798587545752525,
+            -0.005502239800989628,
+            -0.009714636951684952,
+            0.005881845485419035,
+            -0.00898808054625988,
+            0.0021653843577951193,
+            -0.009526874870061874,
+            -0.005824700463563204,
+            -0.008069679141044617,
+            0.016400592401623726,
+            -0.0003523086488712579,
+            0.027315273880958557,
+            0.020751770585775375,
+            -0.008563575334846973,
+            -0.0005683879135176539,
+            0.017208784818649292,
+            -0.005820618476718664,
+            0.015608727931976318,
+            -0.011192241683602333,
+            -0.007706400938332081,
+            0.006408394780009985,
+            0.024017194285988808,
+            0.012237177230417728,
+            -0.0056042843498289585,
+            0.0002885308349505067,
+            -0.0030348035506904125,
+            -0.010612628422677517,
+            0.013388238847255707,
+            0.015323002822697163,
+            -0.01947825588285923,
+            -0.004722619894891977,
+            -0.0016592437168583274,
+            -0.003300119424238801,
+            0.00945340283215046,
+            0.02305389568209648,
+            0.000171817431692034,
+            0.009184005670249462,
+            0.0012765767751261592,
+            0.03862996771931648,
+            -0.0053144777193665504,
+            0.03102152980864048,
+            -0.0010423846542835236,
+            -0.0059430720284581184,
+            -0.009094206616282463,
+            -0.020441556349396706,
+            -0.011486129835247993,
+            -0.00041481092921458185,
+            -0.005040998570621014,
+            -0.0004956811899319291,
+            -0.020327266305685043,
+            0.0026960158720612526,
+            0.014637264423072338,
+            0.007661501411348581,
+            0.017927179113030434,
+            -0.002889900468289852,
+            0.017437364906072617,
+            0.019690508022904396,
+            -0.01400050614029169,
+            -0.029862305149435997,
+            -0.0006515541463159025,
+            0.01498013362288475,
+            -0.01661284640431404,
+            0.019445601850748062,
+            0.004085862077772617,
+            0.0029531680047512054,
+            0.004779764451086521,
+            -0.006049198564141989,
+            0.018482301384210587,
+            0.02039257436990738,
+            0.01453930139541626,
+            -0.013167822733521461,
+            0.014057651162147522,
+            0.009771781973540783,
+            0.006600238848477602,
+            -0.01646590232849121,
+            -0.013535182923078537,
+            0.002555194543674588,
+            0.0009245232213288546,
+            0.024392718449234962,
+            -0.020588500425219536,
+            0.0016602640971541405,
+            0.010588138364255428,
+            0.01769859902560711,
+            -0.004302196204662323,
+            0.008906444534659386,
+            0.0008740111952647567,
+            0.007134951651096344,
+            -0.0035919665824621916,
+            -0.006457376293838024,
+            -0.006673710886389017,
+            -0.010392212308943272,
+            -0.0020959940738976,
+            -0.02070278860628605,
+            0.002198038622736931,
+            -0.002830714685842395,
+            0.018237393349409103,
+            -0.01018812321126461,
+            0.038140155375003815,
+            0.03399306535720825,
+            -0.0007025764207355678,
+            0.011102442629635334,
+            -0.01709449477493763,
+            -0.003945040516555309,
+            -0.021959977224469185,
+            0.01844964735209942,
+            0.016392430290579796,
+            0.012269831262528896,
+            0.005302232690155506,
+            0.017796561121940613,
+            -0.0006301248213276267,
+            0.014490320347249508,
+            0.0029654132667928934,
+            0.007971717044711113,
+            -0.002265388146042824,
+            0.003938918001949787,
+            -0.031103165820240974,
+            -0.017029186710715294,
+            -0.01138000376522541,
+            0.004796091932803392,
+            -0.006375740747898817,
+            -0.0035205355379730463,
+            0.0084737753495574,
+            -0.005902254022657871,
+            0.0004270562494639307,
+            -0.025992777198553085,
+            -0.0157148540019989,
+            -0.010318740271031857,
+            -0.004040962550789118,
+            -0.01169838197529316,
+            -0.0027694879099726677,
+            -0.0022082431241869926,
+            0.0029552089981734753,
+            0.020947696641087532,
+            -0.012808626517653465,
+            -0.016294466331601143,
+            0.004379750229418278,
+            -0.002263347152620554,
+            -0.014016833156347275,
+            -0.016898570582270622,
+            0.006665547378361225,
+            0.012098397128283978,
+            -0.0017470018938183784,
+            -0.013608654960989952,
+            0.011739199981093407,
+            -0.006628811359405518,
+            0.04663025587797165,
+            -0.013584164902567863,
+            -0.0078002819791436195,
+            0.003720542648807168,
+            0.004698128905147314,
+            -0.009396257810294628,
+            0.007330877240747213,
+            0.013020878657698631,
+            0.01830270327627659,
+            0.0095758568495512,
+            -0.008465612307190895,
+            -0.0024388637393712997,
+            0.007796200457960367,
+            -0.016147522255778313,
+            0.006404313258826733,
+            0.0008525818702764809,
+            -0.014400520361959934,
+            0.006110425107181072,
+            0.028980640694499016,
+            0.013151495717465878,
+            0.01288209855556488,
+            -0.022351829335093498,
+            -0.004094025585800409,
+            -0.009355440735816956,
+            -0.004943036008626223,
+            -0.010090161114931107,
+            0.027070365846157074,
+            -0.018155759200453758,
+            0.02651524543762207,
+            0.00898808054625988,
+            0.0017684312770143151,
+            -0.01176369097083807,
+            0.006343086715787649,
+            0.02832755446434021,
+            0.0028592869639396667,
+            -0.020000724121928215,
+            -0.013592327944934368,
+            0.006314514204859734,
+            0.01707816869020462,
+            -0.002647034591063857,
+            -0.005428767763078213,
+            0.019657853990793228,
+            0.01784554310142994,
+            -0.004334850702434778,
+            -0.00936360377818346,
+            0.0008337036124430597,
+            0.00928196869790554,
+            0.016963878646492958,
+            0.015020951628684998,
+            0.009086042642593384,
+            -0.002647034591063857,
+            0.008037025108933449,
+            -0.008800318464636803,
+            -0.016449574381113052,
+            0.007220669183880091,
+            -0.008432958275079727,
+            -0.016963878646492958,
+            0.01872720755636692,
+            0.015257694758474827,
+            0.0181394312530756,
+            0.0024592725094407797,
+            -0.016735298559069633,
+            0.022302847355604172,
+            0.005755309946835041,
+            0.002998067531734705,
+            0.02160078100860119,
+            -0.022221211344003677,
+            0.013012715615332127,
+            0.02037624642252922,
+            0.027609162032604218,
+            -0.01737205684185028,
+            -0.010906516574323177,
+            -0.004857318475842476,
+            -0.006853308994323015,
+            -0.00176945177372545,
+            -0.008253359235823154,
+            -0.012620864436030388,
+            0.0028735732194036245,
+            -0.030466407537460327,
+            0.018106777220964432,
+            -0.005102225113660097,
+            0.010669773444533348,
+            -0.019266001880168915,
+            0.011363675817847252,
+            -0.0035838030744343996,
+            0.011135096661746502,
+            0.0019612954929471016,
+            -0.030041903257369995,
+            -0.008571738377213478,
+            0.015388311818242073,
+            -0.0030368445441126823,
+            -0.013641309924423695,
+            0.02006603218615055,
+            0.00018942011229228228,
+            -0.022270193323493004,
+            -0.004314441699534655,
+            -0.006934944540262222,
+            0.002193956868723035,
+            -0.0029613315127789974,
+            -0.0052981507033109665,
+            -0.030188847333192825,
+            0.020033378154039383,
+            0.009559528902173042,
+            -0.03072764165699482,
+            -0.003255219664424658,
+            -0.014996460638940334,
+            -0.0025184585247188807,
+            0.008881953544914722,
+            -0.004000144544988871,
+            -0.015551582910120487,
+            0.01784554310142994,
+            0.022711025550961494,
+            -0.0023164101876318455,
+            -0.016057724133133888,
+            -0.0090697156265378,
+            -0.027494871988892555,
+            0.030825603753328323,
+            -0.0066696288995444775,
+            0.0022878379095345736,
+            -0.005322641227394342,
+            0.009388094767928123,
+            0.004228724166750908,
+            0.0157148540019989,
+            0.006081852596253157,
+            -0.002647034591063857,
+            0.026596879586577415,
+            -0.01964152604341507,
+            0.009567692875862122,
+            -1.588864870427642e-05,
+            -0.003308282932266593,
+            0.01890680566430092,
+            0.0007939062779769301,
+            0.021649762988090515,
+            -0.008571738377213478,
+            -0.0001475818717153743,
+            -0.0034388997592031956,
+            0.010898353531956673,
+            -0.0028450009413063526,
+            -0.003465431509539485,
+            -0.001531688030809164,
+            0.011118769645690918,
+            0.01013097818940878,
+            0.003804219188168645,
+            -0.01737205684185028,
+            0.003142970846965909,
+            -0.008547247387468815,
+            -0.007028825581073761,
+            0.01302904263138771,
+            -0.011192241683602333,
+            -0.0018990482203662395,
+            -0.03160930797457695,
+            0.016286302357912064,
+            0.0533243790268898,
+            0.008547247387468815,
+            -0.014433175325393677,
+            -0.0018806802108883858,
+            -0.019755816087126732,
+            0.01535565685480833,
+            0.011004479601979256,
+            -0.016816934570670128,
+            -0.011984107084572315,
+            -0.012310649268329144,
+            0.007649256382137537,
+            0.021306892856955528,
+            -0.006522684823721647,
+            -0.008579902350902557,
+            0.025372346863150597,
+            -0.011788181029260159,
+            -0.010882026515901089,
+            -0.01661284640431404,
+            -0.01635977439582348,
+            0.00241641397587955,
+            0.019902760162949562,
+            -0.0169475506991148,
+            0.021176276728510857,
+            0.00936360377818346,
+            0.0010133018950000405,
+            0.003624620847404003,
+            0.008588065393269062,
+            0.0024878450203686953,
+            0.017878197133541107,
+            0.003883813973516226,
+            -0.01439235731959343,
+            0.0021408938337117434,
+            -0.0030286810360848904,
+            0.004624656867235899,
+            3.4320437407586724e-06,
+            -0.002749078907072544,
+            -0.0065635028295218945,
+            0.003710338147357106,
+            -0.021306892856955528,
+            0.021502818912267685,
+            -0.022678371518850327,
+            -0.004245051648467779,
+            0.005555302836000919,
+            -0.005118552595376968,
+            0.013469874858856201,
+            0.018792515620589256,
+            0.01423724927008152,
+            0.008702355436980724,
+            0.01462093647569418,
+            -0.011331021785736084,
+            0.02323349379003048,
+            -0.013282112777233124,
+            0.0005816536722704768,
+            0.012939243577420712,
+            0.01755165494978428,
+            0.018792515620589256,
+            -0.006245123688131571,
+            -0.010384049266576767,
+            -0.0002946535241790116,
+            0.0129637336358428,
+            -0.022286519408226013,
+            -0.012269831262528896,
+            -0.01386172603815794,
+            0.011486129835247993,
+            -0.004743028432130814,
+            0.025453981012105942,
+            -0.00633492274209857,
+            -0.014653591439127922,
+            -0.015918942168354988,
+            0.014196431264281273,
+            -0.048752784729003906,
+            0.0030348035506904125,
+            -0.0024919267743825912,
+            0.03043375350534916,
+            -0.027021385729312897,
+            0.01738838478922844,
+            -0.0028490826953202486,
+            -0.009355440735816956,
+            0.015429128892719746,
+            -0.015992414206266403,
+            -0.0046613928861916065,
+            -0.0016806729836389422,
+            -0.007539048325270414,
+            -0.00112248957157135,
+            0.017616963014006615,
+            -0.015845470130443573,
+            0.01282495353370905,
+            -0.00989423505961895,
+            0.0027021386194974184,
+            -0.004526694305241108,
+            -0.007751300930976868,
+            0.0196088720113039,
+            0.034548189491033554,
+            -0.007053316105157137,
+            0.0014990337658673525,
+            -0.022547753527760506,
+            0.013192313723266125,
+            -0.006730855442583561,
+            0.02351105399429798,
+            -0.005432849284261465,
+            -0.016898570582270622,
+            -0.013976015150547028,
+            -0.009739127941429615,
+            0.00839214026927948,
+            -0.022433463484048843,
+            -0.011110605672001839,
+            -0.00922482367604971,
+            -0.002573562553152442,
+            -0.018694553524255753,
+            0.004865481983870268,
+            -0.002426618244498968,
+            -0.012506574392318726,
+            -0.008816645480692387,
+            0.01048201136291027,
+            -0.009347276762127876,
+            0.003077662317082286,
+            -0.0011929002357646823,
+            -0.011453475803136826,
+            -0.02369065210223198,
+            0.0158944521099329,
+            0.0001974561164388433,
+            -0.010359558276832104,
+            0.008408467285335064,
+            -0.0018276170594617724,
+            0.001211268245242536,
+            0.016963878646492958,
+            -0.0020664012990891933,
+            0.003467472270131111,
+            0.023478399962186813,
+            0.004057289566844702,
+            -0.010971825569868088,
+            -0.00025906547671183944,
+            0.013535182923078537,
+            -0.005118552595376968,
+            -0.010008525103330612,
+            1.6646010408294387e-05,
+            -0.02984597720205784,
+            -0.008351322263479233,
+            0.006094098091125488,
+            0.018629245460033417,
+            -0.010620792396366596,
+            -0.015853634104132652,
+            0.026319319382309914,
+            -0.006359413731843233,
+            0.019070077687501907,
+            0.002889900468289852,
+            0.0025694805663079023,
+            0.01724144071340561,
+            0.013461710885167122,
+            0.00786150898784399,
+            -0.013927034102380276,
+            -0.02429475635290146,
+            0.0016398552106693387,
+            -0.018090449273586273,
+            0.010579974390566349,
+            -0.011028970591723919,
+            0.0032756286673247814,
+            -0.02398454025387764,
+            -0.012237177230417728,
+            0.004934872500598431,
+            -0.020343592390418053,
+            -0.00040588201954960823,
+            -0.0056940834037959576,
+            0.022237539291381836,
+            -0.005310396198183298,
+            0.008669701404869556,
+            0.0018255761824548244,
+            -0.012237177230417728,
+            0.0009877908742055297,
+            0.009469729848206043,
+            -0.029192892834544182,
+            -0.0007127808639779687,
+            -0.003138889092952013,
+            0.0011418780777603388,
+            0.005351213738322258,
+            0.009755454957485199,
+            0.007322713732719421,
+            0.0027062203735113144,
+            0.007253323681652546,
+            0.010449357330799103,
+            -0.00022041612828616053,
+            0.0197884701192379,
+            0.018172085285186768,
+            -0.007351286243647337,
+            0.008710519410669804,
+            0.024817224591970444,
+            -0.013159659691154957,
+            -0.008718682453036308,
+            0.023266147822141647,
+            -0.006461458280682564,
+            -0.0011122850701212883,
+            -0.01483318954706192,
+            0.00815947912633419,
+            0.0017367975087836385,
+            0.028131630271673203,
+            0.019657853990793228,
+            0.005820618476718664,
+            0.000639819074422121,
+            0.015788326039910316,
+            -0.013020878657698631,
+            0.0035286990460008383,
+            0.015461783856153488,
+            -0.003940958995372057,
+            -0.019821124151349068,
+            0.013878053054213524,
+            0.009502384811639786,
+            -0.003228688146919012,
+            0.01551892887800932,
+            0.011412657797336578,
+            -0.030841931700706482,
+            -0.0011898389784619212,
+            0.010498338378965855,
+            -0.0021449755877256393,
+            -0.008751336485147476,
+            -0.00014273475972004235,
+            -0.012384121306240559,
+            0.021388528868556023,
+            -0.004408322740346193,
+            0.012277995236217976,
+            -0.00815947912633419,
+            -0.017127150669693947,
+            -0.011559601873159409,
+            0.007914572022855282,
+            -0.010547320358455181,
+            -0.0006714528426527977,
+            -0.004022594541311264,
+            -0.005779800936579704,
+            0.017747581005096436,
+            0.004706292413175106,
+            -0.0008964610169641674,
+            -0.0018398624379187822,
+            0.007983962073922157,
+            -0.006979844067245722,
+            -0.008726846426725388,
+            0.004816500470042229,
+            -0.012833117507398129,
+            0.003981776535511017,
+            0.005543057806789875,
+            0.01506176870316267,
+            -0.014857680536806583,
+            -0.021159948781132698,
+            0.007020662073045969,
+            0.02695607766509056,
+            -0.02516009286046028,
+            -0.005232842173427343,
+            0.006245123688131571,
+            0.002093953313305974,
+            0.006914535537362099,
+            -0.006934944540262222,
+            -0.007208424154669046,
+            -0.03657275065779686,
+            -0.0020745648071169853,
+            0.0009005427709780633,
+            0.008963589556515217,
+            -0.008049271069467068,
+            -0.0136494729667902,
+            -0.011135096661746502,
+            0.002410291228443384,
+            0.007134951651096344,
+            -0.02204161323606968,
+            0.0049593630246818066,
+            0.009608510881662369,
+            -0.019559891894459724,
+            0.025127438828349113,
+            -0.010890189558267593,
+            -6.964537897147238e-05,
+            -0.00995954405516386,
+            0.021633435040712357,
+            -0.013486201874911785,
+            -0.018368011340498924,
+            -0.0006898208521306515,
+            0.008355404250323772,
+            0.011926962062716484,
+            -0.0017663904000073671,
+            -0.011371839791536331,
+            0.0003752686607185751,
+            0.01646590232849121,
+            0.019135385751724243,
+            0.002744997153058648,
+            -0.0024837632663547993,
+            -0.01019628718495369,
+            -0.009592183865606785,
+            -0.0073961857706308365,
+            -0.0007449248805642128,
+            0.002091912319883704,
+            0.02035992033779621,
+            0.013469874858856201,
+            0.0024837632663547993,
+            -0.015135240741074085,
+            -0.005257333163172007,
+            0.008939098566770554,
+            0.009412584826350212,
+            -0.011690218932926655,
+            -0.00898808054625988,
+            0.002659279853105545,
+            0.02249877341091633,
+            0.017633290961384773,
+            -0.0014000505907461047,
+            0.0032878739293664694,
+            -0.0016592437168583274,
+            0.012506574392318726,
+            -0.007555375341325998,
+            -0.010269759222865105,
+            -0.006943108048290014,
+            -0.014645427465438843,
+            -0.010800390504300594,
+            -0.0023062059190124273,
+            -0.010179960168898106,
+            0.022629389539361,
+            0.007988044060766697,
+            0.009779945947229862,
+            0.008265605196356773,
+            0.010269759222865105,
+            0.002087830565869808,
+            0.025715215131640434,
+            -0.004424649756401777,
+            -0.011984107084572315,
+            0.007236996199935675,
+            -0.022368155419826508,
+            -0.021535472944378853,
+            0.008257441222667694,
+            0.006367577239871025,
+            0.03595232218503952,
+            -0.026890767738223076,
+            0.0052981507033109665,
+            -0.009779945947229862,
+            -0.010400376282632351,
+            0.007996207103133202,
+            -0.0015439334092661738,
+            -0.004245051648467779,
+            0.022368155419826508,
+            0.00726556871086359,
+            0.0035797213204205036,
+            0.0032817514147609472,
+            0.019837452098727226,
+            -0.005685919895768166,
+            0.01573934406042099,
+            0.0259274672716856,
+            0.024621298536658287,
+            0.009388094767928123,
+            -0.009469729848206043,
+            0.02785406820476055,
+            0.0002895512734539807,
+            -0.0034327772445976734,
+            0.007220669183880091,
+            -0.00305113079957664,
+            -0.0009556467994116247,
+            -0.010033016093075275,
+            0.021764053031802177,
+            -0.001455154619179666,
+            0.004359341226518154,
+            0.02081707864999771,
+            -7.774515688652173e-05,
+            0.006220633164048195,
+            0.01057181041687727,
+            0.002181711606681347,
+            0.003396041225641966,
+            0.010775899514555931,
+            -0.039315707981586456,
+            -0.0006571666453965008,
+            -0.025600925087928772,
+            -0.0018286375561729074,
+            0.0007357408758252859,
+            -0.01266984548419714,
+            0.013600491918623447,
+            -0.002015379024669528,
+            -0.010751409456133842,
+            -0.00029337796149775386,
+            0.021649762988090515,
+            0.005416522268205881,
+            0.004071575589478016,
+            -0.004469549283385277,
+            0.018792515620589256,
+            -0.006955353543162346,
+            -0.017861871048808098,
+            -0.012759645469486713,
+            -0.012367794290184975,
+            -0.01918436773121357,
+            -0.01214737817645073,
+            -0.0024551907554268837,
+            -0.002667443361133337,
+            0.004738946910947561,
+            0.002481722505763173,
+            -0.003228688146919012,
+            -0.02070278860628605,
+            0.015037278644740582,
+            -0.01430255826562643,
+            0.012873934581875801,
+            -0.011053460650146008,
+            0.005963481031358242,
+            -0.004122598096728325,
+            0.012726990506052971,
+            -0.021404854953289032,
+            0.02070278860628605,
+            -0.01536382082849741,
+            0.0038532004691660404,
+            0.019592545926570892,
+            0.006551257334649563,
+            -0.008118661120533943,
+            -0.013404566794633865,
+            -0.022074267268180847,
+            0.021633435040712357,
+            -0.011633073911070824,
+            -0.018237393349409103,
+            -0.004608329851180315,
+            0.009649328887462616,
+            -0.017470018938183784,
+            -0.007367613259702921,
+            0.01707816869020462,
+            -0.0018949664663523436,
+            0.012980061583220959,
+            0.023004913702607155,
+            -0.0052981507033109665,
+            0.0036572751123458147,
+            -0.009167678654193878,
+            0.0010286086471751332,
+            -0.02470293454825878,
+            0.013527019880712032,
+            0.008971752598881721,
+            0.006269614677876234,
+            -0.013935198076069355,
+            -0.013306603766977787,
+            0.0036531933583319187,
+            0.004498121794313192,
+            0.011437147855758667,
+            0.009420748800039291,
+            0.004220560658723116,
+            -0.013918871060013771,
+            -0.027788760140538216,
+            -0.027070365846157074,
+            0.03595232218503952,
+            -0.0005002731923013926,
+            -0.020947696641087532,
+            0.03399306535720825,
+            0.016114868223667145,
+            0.0022980421781539917,
+            -0.012449429370462894,
+            -0.004314441699534655,
+            0.024621298536658287,
+            0.016400592401623726,
+            0.010465684346854687,
+            -0.0004742518358398229,
+            0.005571629852056503,
+            0.0016469983384013176,
+            0.03012353740632534,
+            0.015323002822697163,
+            0.0009459525463171303,
+            -0.038597315549850464,
+            0.011943289078772068,
+            -0.007547211833298206,
+            0.0013408648082986474,
+            0.006702283397316933,
+            0.017780235037207603,
+            0.011935125105082989,
+            0.015567909926176071,
+            0.014865843579173088,
+            -0.003065417055040598,
+            0.0029511270113289356,
+            0.0006163488142192364,
+            -0.009624837897717953,
+            -0.001767410896718502,
+            -0.008881953544914722,
+            -0.005714492406696081,
+            -0.002920513739809394,
+            0.005592038854956627,
+            0.007657419890165329,
+            -0.003983817528933287,
+            0.028049994260072708,
+            -0.0006433906382881105,
+            -0.017600636929273605,
+            -0.009061552584171295,
+            -0.03611559048295021,
+            0.00434301421046257,
+            -0.030939893797039986,
+            -0.011257549747824669,
+            0.0033674687147140503,
+            -0.020457882434129715,
+            -0.024768242612481117,
+            0.0010306495241820812,
+            0.014335212297737598,
+            0.007392104249447584,
+            0.0033327736891806126,
+            -0.01004934310913086,
+            0.003816464450210333,
+            -0.001855169073678553,
+            -0.006461458280682564,
+            0.028833696618676186,
+            0.019559891894459724,
+            0.020457882434129715,
+            -0.01190247107297182,
+            0.019298657774925232,
+            0.005845109466463327,
+            0.02035992033779621,
+            0.010465684346854687,
+            -0.0048695639707148075,
+            0.0074859848245978355,
+            0.009567692875862122,
+            -0.006155324634164572,
+            -0.021519144997000694,
+            0.013192313723266125,
+            -0.0010837126756086946,
+            -0.0052083516493439674,
+            -0.016269976273179054,
+            -0.003534821793437004,
+            -0.015812816098332405,
+            0.010547320358455181,
+            -0.017159804701805115,
+            0.012498411349952221,
+            -0.000802069844212383,
+            0.00168475485406816,
+            -0.015469946898519993,
+            -0.013951525092124939,
+            -0.0012439225101843476,
+            -0.01244126632809639,
+            -0.008971752598881721,
+            0.006163488142192364,
+            -0.007539048325270414,
+            -0.008318668231368065,
+            -0.00535937724635005,
+            -0.011437147855758667,
+            -0.010882026515901089,
+            0.004367504734545946,
+            -0.0010653446661308408,
+            0.020931368693709373,
+            0.00011671340325847268,
+            -0.005824700463563204,
+            0.01190247107297182,
+            0.02787039615213871,
+            0.0016704685986042023,
+            0.004583839327096939,
+            -0.014261740259826183,
+            -0.011118769645690918,
+            -0.02279266156256199,
+            0.016572028398513794,
+            0.0049675265327095985,
+            -0.0344502255320549,
+            0.01438419334590435,
+            0.00824927818030119,
+            0.0015102586476132274,
+            -0.006698201410472393,
+            -0.005334886722266674,
+            -0.005608365871012211,
+            0.006306350696831942,
+            -0.0022837561555206776,
+            0.0005107327597215772,
+            0.0035919665824621916,
+            0.034678805619478226,
+            -0.0040674940682947636,
+            -0.007228832691907883,
+            0.00981259997934103,
+            0.0005130287609063089,
+            0.020196648314595222,
+            -0.005518566817045212,
+            0.0037572786677628756,
+            -0.019282329827547073,
+            -0.016563864424824715,
+            0.0018429238116368651,
+            -0.01722511276602745,
+            0.023282473906874657,
+            -0.005392031744122505,
+            -0.01347803883254528,
+            -0.006906372029334307,
+            -0.020115012302994728,
+            0.012637191452085972,
+            0.02310287579894066,
+            0.007016580086201429,
+            0.01033506728708744,
+            0.012131051160395145,
+            -0.02444170042872429,
+            -0.020457882434129715,
+            0.012171869166195393,
+            0.0013571919407695532,
+            -0.008449285291135311,
+            -0.005404277238994837,
+            -0.001783737912774086,
+            0.007783954963088036,
+            -0.0032633834052830935,
+            0.007881917990744114,
+            0.06491663306951523,
+            -0.01093100756406784,
+            -0.002987863030284643,
+            -0.02966637909412384,
+            -0.004396077245473862,
+            -0.006628811359405518,
+            0.0056287748739123344,
+            0.009151351638138294,
+            -0.023135529831051826,
+            0.009232986718416214,
+            -0.030025575309991837,
+            -0.010628955438733101,
+            0.00824111420661211,
+            0.0028490826953202486,
+            -0.01220452319830656,
+            -0.0009592183632776141,
+            -0.013665799982845783,
+            0.0006107363733462989,
+            -0.009412584826350212,
+            -0.00576347392052412,
+            0.012718827463686466,
+            -0.02321716584265232,
+            0.006298186723142862,
+            0.007592111360281706,
+            -0.006514521315693855,
+            -0.013722945004701614,
+            -0.017861871048808098,
+            -0.011967780068516731,
+            -0.013314766809344292,
+            0.002996026771143079,
+            0.01190247107297182,
+            -0.02607441134750843,
+            0.012922916561365128,
+            -0.0037389106582850218,
+            0.00945340283215046,
+            0.004404240753501654,
+            0.013469874858856201,
+            0.018074123188853264,
+            0.009355440735816956,
+            0.0021429345943033695,
+            -0.008024780079722404,
+            0.012702500447630882,
+            -0.01993541419506073,
+            -0.01635161228477955,
+            0.013837235048413277,
+            0.012016761116683483,
+            0.003061335301026702,
+            0.009828926995396614,
+            0.0020959940738976,
+            0.011886144056916237,
+            0.010775899514555931,
+            -0.019739490002393723,
+            -0.0006648199632763863,
+            0.0007326795603148639,
+            0.03280118480324745,
+            -0.0005938990507274866,
+            0.008857463486492634,
+            0.004420568235218525,
+            0.013159659691154957,
+            -0.012784135527908802,
+            -0.01769859902560711,
+            0.0008087027235887945,
+            -0.01057181041687727,
+            0.021257910877466202,
+            -0.008192133158445358,
+            0.006216551177203655,
+            -0.018612917512655258,
+            -0.027086693793535233,
+            0.006392067763954401,
+            -0.016261812299489975,
+            -0.018221067264676094,
+            0.01317598670721054,
+            0.0077268099412322044,
+            0.0017572063952684402,
+            -0.005098143592476845,
+            -0.020457882434129715,
+            -0.009837090037763119,
+            0.028441844508051872,
+            0.011102442629635334,
+            -0.004120557103306055,
+            0.009584019891917706,
+            -0.009094206616282463,
+            -0.012490247376263142,
+            -0.0017541450215503573,
+            0.0036715613678097725,
+            0.008261523209512234,
+            0.005645101889967918,
+            -0.01386172603815794,
+            -0.00029873527819290757,
+            -0.012400448322296143,
+            -0.021225256845355034,
+            -0.008906444534659386,
+            0.016751626506447792,
+            0.007820690982043743,
+            -0.0038960592355579138,
+            0.015812816098332405,
+            -0.004404240753501654,
+            0.002189875114709139,
+            0.014294394291937351,
+            0.005334886722266674,
+            -0.004285869188606739,
+            -0.0095758568495512,
+            -0.004612411838024855,
+            0.004043003544211388,
+            -0.0025960123166441917,
+            0.006457376293838024,
+            0.017192458733916283,
+            0.0072900597006082535,
+            -0.013755599036812782,
+            0.013135168701410294,
+            0.0007791098323650658,
+            0.0049675265327095985,
+            -0.03699725493788719,
+            -0.003247056156396866,
+            0.015796490013599396,
+            -0.013020878657698631,
+            0.02501314878463745,
+            0.005400195252150297,
+            0.022743679583072662,
+            0.015233203768730164,
+            0.008669701404869556,
+            -0.001542912912555039,
+            0.00464098434895277,
+            0.02759283408522606,
+            -0.003151134354993701,
+            -0.008637047372758389,
+            -0.0020602785516530275,
+            0.009241150692105293,
+            0.0002063850115519017,
+            0.003000108525156975,
+            0.008281932212412357,
+            -0.03562577813863754,
+            0.011706545948982239,
+            -0.019576217979192734,
+            0.0024694770108908415,
+            0.003459308762103319,
+            -0.007183933164924383,
+            -0.014865843579173088,
+            -0.005857354495674372,
+            0.022841641679406166,
+            -0.003224606392905116,
+            0.021078312769532204,
+            0.002885818714275956,
+            0.016106704249978065,
+            -0.011575928889214993,
+            -0.009208496659994125,
+            -0.0004130251472815871,
+            -0.005351213738322258,
+            -0.007539048325270414,
+            0.010596301406621933,
+            0.0034409407526254654,
+            0.005534893833100796,
+            0.002265388146042824,
+            0.0036654386203736067,
+            -0.0036878883838653564,
+            0.02773977816104889,
+            0.003563394071534276,
+            0.007996207103133202,
+            -0.011567764915525913,
+            0.014082142151892185,
+            -0.017029186710715294,
+            0.0070778070949018,
+            0.005575711838901043,
+            -0.026156047359108925,
+            0.016849588602781296,
+            0.004775682929903269,
+            0.006628811359405518,
+            0.013812744058668613,
+            0.01618834026157856,
+            -0.010408539324998856,
+            -0.014816862531006336,
+            0.024049850180745125,
+            -0.0032041973900049925,
+            -0.015225040726363659,
+            -0.006775755435228348,
+            4.438935866346583e-05,
+            0.006559420842677355,
+            0.00373074715025723,
+            0.0013020879123359919,
+            -0.0006204306264407933,
+            -0.010743245482444763,
+            0.004645065870136023,
+            -0.01934763789176941,
+            -0.0007163524278439581,
+            -0.016449574381113052,
+            -0.015633217990398407,
+            0.018678225576877594,
+            -0.01830270327627659,
+            0.009633000940084457,
+            0.0012275953777134418,
+            0.028588788583874702,
+            -0.011706545948982239,
+            -0.012155541218817234,
+            0.0015102586476132274,
+            0.010849371552467346,
+            0.0067512644454836845,
+            -0.010033016093075275,
+            -0.007583947852253914,
+            -0.014180104248225689,
+            -0.010163633152842522,
+            0.015502601861953735,
+            -0.014498483389616013,
+            -0.014343376271426678,
+            0.008914608508348465
+          ],
+          "embedding": [
+            -0.006359657085039302,
+            -0.024594786241399234,
+            -0.02553538924848095,
+            0.005360188465088291,
+            0.012801870977589839,
+            0.021680505862506104,
+            -0.016668048966001273,
+            0.06860852474581301,
+            -0.007122272841157724,
+            0.03471775000705488,
+            -0.009004297848861675,
+            0.01059974144678391,
+            0.02565604599426148,
+            -0.022544127646041287,
+            0.010616674976540785,
+            0.013658194823359737,
+            0.016400329743363844,
+            0.02858546964917478,
+            0.0067612676434693686,
+            -0.04174737193606879,
+            0.03184155110488616,
+            -0.02223105127585514,
+            -0.05506239667974977,
+            -0.0015307551410555752,
+            -0.013169788751379287,
+            0.017681749722384007,
+            0.01682706317229353,
+            0.019652295823816928,
+            -0.05623980369271269,
+            0.030698582851533197,
+            0.02286327124296613,
+            0.016019032539689178,
+            0.005508113353872315,
+            8.58985091215149e-05,
+            -0.019206292853495906,
+            -0.018439290483495144,
+            0.03475023176950151,
+            0.01018612942141759,
+            -0.0252551158857426,
+            0.0334562130666827,
+            0.020462153537485785,
+            0.02260508249714211,
+            -0.03323524989392951,
+            -0.002094251009530707,
+            0.015177442347686773,
+            -0.00018698464350038317,
+            -0.0006788648344273403,
+            0.0007777616893581039,
+            0.013352398615059647,
+            -0.030751111528744813,
+            -0.0317733818445192,
+            -0.006729064081710058,
+            0.009413204899986514,
+            -0.005712894407943592,
+            -0.038961826880527066,
+            -0.002827692206530345,
+            0.017863300537973137,
+            0.07918455388563496,
+            0.048967158919816366,
+            0.018002566325350088,
+            0.040264962620950695,
+            -0.019715100359659787,
+            -0.02211350500879408,
+            0.004918148608475574,
+            0.01062585305866606,
+            -0.014478993521696027,
+            0.0034983131226364672,
+            0.000766228629514787,
+            0.009667268771502067,
+            -0.01558439890697323,
+            -0.04984718709870083,
+            0.029613955432821887,
+            -0.009686854114630624,
+            -0.015279319238124568,
+            0.007460629676325322,
+            -0.006235979350809034,
+            0.012514387473717085,
+            0.02800435263741497,
+            -0.008688925946672778,
+            -0.01236301543374469,
+            0.029755705020188972,
+            0.01158037219446542,
+            -0.005242392206024511,
+            0.009982679027728814,
+            0.052414361714483264,
+            -0.018788500381726227,
+            -0.020335831196027204,
+            -0.08583959943356621,
+            0.029309228216812346,
+            -0.0249285282010673,
+            -0.018929396555426307,
+            0.007710663193334574,
+            0.03096798035958051,
+            -0.017089977064591828,
+            -0.028015107605405817,
+            0.06740703271128606,
+            0.009398795412354399,
+            0.01118769022826835,
+            -0.03041417460855722,
+            0.014776002883150835,
+            0.004319194067673161,
+            0.022507636639041203,
+            0.0053562788646786845,
+            0.007548072258943124,
+            0.011123374308563128,
+            -0.006924362115585165,
+            -0.011118507147092024,
+            0.009563232643295304,
+            0.00657178473678069,
+            0.08313027447380966,
+            -0.004239370156003264,
+            -0.00930406163946907,
+            -0.039202672821898946,
+            0.009761029930556298,
+            -0.02827698465424993,
+            -0.037914533246900244,
+            -0.04496050911855815,
+            -0.017711307900236353,
+            0.01934435542287117,
+            0.004335173027873153,
+            0.03800509969714242,
+            0.031198127357716314,
+            0.031311207941680276,
+            -0.012184553894588813,
+            -0.058985863670856876,
+            0.041442017057707635,
+            -0.012456189292721711,
+            -0.0040966351720394135,
+            0.010849179519959421,
+            0.005243123049347636,
+            0.0011495799963862647,
+            0.003534183676934487,
+            -0.03139340512003631,
+            -0.023876261883905306,
+            0.008382957569752911,
+            -0.004778283217011976,
+            0.03703067589888386,
+            0.06888630688839406,
+            -0.03993532296478411,
+            -0.009472167678222372,
+            0.025246872942105057,
+            -0.01524567912700368,
+            0.01595917939878582,
+            -0.003736723164442467,
+            -0.030836797784967753,
+            0.0026246446163368533,
+            0.008894300082148884,
+            0.08342856279154155,
+            -0.0007476580521704578,
+            0.025149116238550588,
+            0.004972542633631468,
+            0.034495632546556576,
+            -0.009173329764686424,
+            0.014009381543772768,
+            0.004427261668151028,
+            -0.013426138607902904,
+            -0.022234568880321808,
+            0.01976282364689966,
+            -0.03592433789490043,
+            -0.010918278220831188,
+            0.013527180289641174,
+            -0.013023113052242677,
+            -0.008304770302495363,
+            0.003677063799167178,
+            0.0013981822663219421,
+            0.047001163242600175,
+            0.01828332714887986,
+            0.002483375662343689,
+            -0.06820498187523893,
+            0.05295312038995555,
+            0.025038753504342162,
+            0.025286870198436735,
+            -0.0029544808216316115,
+            0.029982752196635428,
+            0.0005600481523874767,
+            0.0128290258663539,
+            0.009023805227424919,
+            -0.0019219166367750596,
+            -0.04482843936478531,
+            0.04523479691409549,
+            -0.004678574027241186,
+            0.010486850804218904,
+            0.006736229865018014,
+            0.02134813656476443,
+            0.028083972673690036,
+            0.016857413440125445,
+            0.0189082447820146,
+            0.03843690613820022,
+            -0.011922027011229613,
+            0.016193433524453224,
+            0.03828207911467065,
+            -0.0013055128404331705,
+            0.043198163443700956,
+            0.024025562893260542,
+            -0.001257851946749882,
+            -0.011726713433348844,
+            -0.021907984099443396,
+            0.02752198420597673,
+            0.004431866336999851,
+            0.015798806714413912,
+            -0.012910782191382367,
+            -0.03180549180206364,
+            0.004550247372715741,
+            -0.022778272841118514,
+            0.031242218240017532,
+            -0.006106984949693483,
+            -0.006259993190497421,
+            0.02187449047033339,
+            0.009334146847221406,
+            -0.009381278960016145,
+            -0.020888853823931134,
+            -0.007502374908776156,
+            -0.010390018386728482,
+            -0.04316693435221676,
+            -0.006274513730720055,
+            0.019152443937064258,
+            -0.01668984824845791,
+            0.018573194036505578,
+            0.0035035262183249925,
+            -0.019779911381789077,
+            -0.0027495798165085993,
+            0.02425481236315913,
+            0.006857723545270935,
+            -0.014733535699087731,
+            -0.026326146471941727,
+            -0.019906660231056833,
+            -0.001033294559612807,
+            0.0159154063026895,
+            0.035649534442431415,
+            -0.026210592096240003,
+            0.014397127229517285,
+            0.005114311681443673,
+            -0.0033104236005207077,
+            0.00324988235167305,
+            0.03764004154553922,
+            -0.017822437760737605,
+            -0.008579701063708334,
+            -0.00032410984200965487,
+            0.018450351071676443,
+            0.017283379764846195,
+            0.006384411611332382,
+            -0.005115090309123598,
+            -0.030590047062583125,
+            -0.011776211359193503,
+            0.01183623914060493,
+            0.0016564847796208331,
+            -0.009530766634361575,
+            -0.004810598258154945,
+            0.03876242515526699,
+            0.027076785774401667,
+            0.005324763136165989,
+            -0.03416412480913939,
+            -0.007943016733586697,
+            0.002114307650456928,
+            -0.02961449347300628,
+            0.05142732770518828,
+            0.05566026433842563,
+            -0.0030729309664974704,
+            -0.00938186948393074,
+            -0.04547908981236149,
+            0.0651216359692231,
+            -0.01813722445944094,
+            -0.0482650853560993,
+            0.044045155892545554,
+            -0.014272653003441165,
+            0.013740652306746851,
+            0.04563732439184395,
+            0.02126513340521316,
+            -0.0054845632439384905,
+            -0.011635625101262898,
+            0.030992082069685376,
+            -0.010051935721068779,
+            -0.050967258971854425,
+            -0.009827942095033494,
+            0.013373878556654464,
+            -0.013017627321534534,
+            -0.009361001542607812,
+            -0.01925867665425919,
+            0.020329052915436022,
+            0.023020741670660188,
+            0.019387043947695172,
+            0.005628632318152202,
+            0.018451555161952665,
+            0.016226845400848455,
+            -0.01384033112836216,
+            0.004762589697426251,
+            -0.012718476561725138,
+            -0.024222765127494463,
+            -0.037487144314010595,
+            -0.019971524166659278,
+            -0.015034588610516958,
+            0.009483936496215338,
+            -0.014301924283553591,
+            -0.0024139665139412125,
+            0.013361045353529458,
+            -0.0005870798721107379,
+            -0.011192319340879789,
+            0.013927695434391159,
+            0.021337539669342293,
+            -0.027300952495815437,
+            -0.04765670695199488,
+            -0.00561706224251562,
+            0.0082776639906139,
+            -0.0030984973749633724,
+            -0.026269016057688548,
+            0.007040524859505573,
+            0.0017859008072463442,
+            0.020091542517503808,
+            -0.018716761130867725,
+            0.0035283051960672963,
+            0.008519383548064053,
+            -0.015247417050336866,
+            -0.016917626868166634,
+            -0.018580550690103854,
+            -0.024697180580294884,
+            0.034202028316737125,
+            0.0204395003463227,
+            0.030392895534267717,
+            -0.04527669147753553,
+            -0.018359428092075348,
+            0.05494107499558624,
+            0.015234754907198422,
+            -0.05066008106884831,
+            0.031139824660661085,
+            0.005016615710206006,
+            0.0019000453624929544,
+            -0.006094364589229031,
+            0.0637324238017742,
+            -0.02711381489292883,
+            -0.02187418906973867,
+            0.004601744864532875,
+            -0.04141121277496072,
+            -0.026689585608019588,
+            -0.005568100750837504,
+            0.01291019036002459,
+            -0.031853041426355644,
+            -0.02815800259372657,
+            -0.007964861641790374,
+            -0.010933515363060793,
+            0.013726639077745378,
+            -0.023956800669647334,
+            -0.015672058590455337,
+            -0.023642025374517057,
+            -0.01288435863544158,
+            -0.0042516881552192385,
+            -0.011162214160460695,
+            -0.02412733256710621,
+            0.010088528028211243,
+            0.03310175458095482,
+            0.023064360903748363,
+            0.001996159684818467,
+            0.009478229572933101,
+            0.014478772877286076,
+            0.00371783700962415,
+            0.007864103610320116,
+            -0.048511515124609,
+            -0.031398754206534776,
+            0.020227872561659397,
+            0.012159811587094092,
+            0.0172163448580163,
+            0.009578952796095119,
+            -0.04154545961112031,
+            -0.05095080393682759,
+            0.02968486144855133,
+            -0.03932395023720674,
+            -0.02063219739061335,
+            0.007512423249865194,
+            -0.011911709468952548,
+            -0.00810951734088105,
+            -0.0008079327452253181,
+            -0.01600701450676789,
+            0.023984909975645632,
+            -0.02418072541081482,
+            0.0033984265728439398,
+            -0.0104343782811971,
+            0.01588977364765743,
+            0.05109529246167992,
+            -0.027800269136366997,
+            -0.027889034481533438,
+            0.03706813210665186,
+            -0.02058120652429308,
+            0.030801396468451574,
+            0.03246127847621613,
+            0.008355517405358826,
+            -0.008605467586247932,
+            0.017693344563968573,
+            0.014499487024471321,
+            0.012431277080478217,
+            -0.040103870071440256,
+            0.017328848567062113,
+            0.02723900253727976,
+            -0.054909074068976875,
+            -0.008547596270650966,
+            -0.00777964838164648,
+            0.027134666572242665,
+            0.02765036594970255,
+            -0.002207173084941375,
+            -0.016838740117653725,
+            -0.028289233739722936,
+            -0.019433376533829132,
+            -0.011053380458946764,
+            0.03247641283550457,
+            0.0007514989009289282,
+            0.007919574219000797,
+            0.001316042436488591,
+            -0.0017934342041762042,
+            0.01924145734197619,
+            0.015260199553572459,
+            0.048259602293134723,
+            -0.03383050622747848,
+            0.0021712225589426266,
+            -0.03508567715696004,
+            -0.04578731401250471,
+            -0.008768228935292693,
+            0.010203318819548585,
+            0.024135236371743855,
+            -0.007467140460133838,
+            0.0028907962385387818,
+            -0.007582083854422456,
+            -0.005384477718912037,
+            -0.01867978482547249,
+            0.01641191108381631,
+            0.037168089880543244,
+            0.0031992495663980464,
+            -0.008533890994695162,
+            0.000778843942291118,
+            -0.004620258538699103,
+            0.010572690335163528,
+            0.033233014627100904,
+            -0.030497362482602697,
+            0.02974571152880676,
+            -0.004274442492343003,
+            0.025341347125910928,
+            -0.04307748954707389,
+            -0.018069609916024403,
+            -0.018491122727274337,
+            -0.02069307802429807,
+            0.05372967226264594,
+            -0.030768978301976772,
+            -0.03662045886993282,
+            -0.003205991669059688,
+            0.016815511161004534,
+            -0.0014152644009276566,
+            -0.0016015439520090104,
+            -0.03984716970293867,
+            -0.010828260334550217,
+            -0.05894985435251326,
+            -0.01638035185618134,
+            -0.021819442096404948,
+            -0.005315102328745982,
+            -0.06251579337157112,
+            0.0033713876515840233,
+            -0.07815974854883057,
+            0.001484860071649606,
+            0.04539427129131382,
+            -0.023530943197353476,
+            0.05142250021536681,
+            -0.024746600229063383,
+            -0.0005885740455257141,
+            -0.044698536284421976,
+            0.0026481001040031785,
+            0.02389178603305878,
+            0.051387657872255674,
+            -0.002949198236157656,
+            0.010674377196609455,
+            0.013322075519401089,
+            -0.026245090721950647,
+            -0.01036724558050635,
+            -0.005321884180696542,
+            0.005958798671856245,
+            0.013755178085972732,
+            -0.0074734329537091675,
+            0.038368016046563674,
+            0.010941421833174982,
+            0.03443115781919206,
+            0.034623617136788874,
+            -0.004361926531008214,
+            -0.03348681614127998,
+            -0.00632870804659212,
+            0.024433331840566692,
+            0.0321998807702602,
+            -0.02899983280320611,
+            0.005958471722097962,
+            -0.015602447981516634,
+            0.015633217539747047,
+            0.006321931116948646,
+            -0.011827541320878442,
+            0.02372204564999124,
+            0.015673426811623124,
+            0.006103300725621424,
+            -0.005165006670545588,
+            0.013589526056189892,
+            -0.025411864292353578,
+            0.03041791947375232,
+            -0.004456865298584878,
+            -0.04276058054583262,
+            -0.0024736078733106825,
+            -0.020059309867945227,
+            0.024883721794000956,
+            0.015578237413840115,
+            -0.00536523508503184,
+            -0.005290645798641171,
+            0.008242987948511412,
+            0.020469675635821673,
+            0.006668354647245447,
+            0.00017729517736814364,
+            0.02373604785121342,
+            0.029915687122900485,
+            -0.02432635861789771,
+            -0.029540533943782886,
+            0.008151297444985344,
+            -0.005730733861939129,
+            0.011318816579801282,
+            -0.02644855229221747,
+            0.00033932872948409137,
+            0.0035505684154028996,
+            -0.03570104343808711,
+            0.03408366016010711,
+            -0.009199700348630527,
+            0.005857598530788056,
+            -0.009538177736472163,
+            -0.0017768806148210118,
+            0.0027410153099351694,
+            0.03898108970435115,
+            -0.007367623392751247,
+            0.009827254226889936,
+            0.015465895035342481,
+            0.000981604396724216,
+            0.05143083558797524,
+            -0.027350877485564314,
+            -0.02183036962301852,
+            0.01885468378168636,
+            -0.007285239007489837,
+            -0.016846837665812908,
+            -0.020546383007536074,
+            -0.011863943814684079,
+            -0.016293192750336218,
+            0.01976516340568883,
+            -0.02387328798681798,
+            -0.011892336568452852,
+            0.007170651391639504,
+            0.0013725014050721622,
+            -0.009809669068563384,
+            0.0016617183704652932,
+            -0.006307590349798603,
+            0.02037693835615797,
+            0.031753656615094446,
+            -0.024892675105483313,
+            0.03577919637821846,
+            0.03178303188315581,
+            0.017260898395251155,
+            -0.02102289690701718,
+            -0.009661035772073294,
+            -0.00048542689442823136,
+            -0.014694631967051945,
+            -0.0326379186158631,
+            0.025443206465204822,
+            0.017194626456982255,
+            -0.011381421003142453,
+            -0.03680783761649673,
+            0.0017216115811536721,
+            0.022819859205829785,
+            -0.013600989058799166,
+            0.028141657788496834,
+            -0.01869047298692401,
+            -0.01366833755869951,
+            -0.00782959271979894,
+            0.031227914441058244,
+            -0.05252392878125891,
+            0.016957842026348195,
+            0.00020880696574616416,
+            0.001154370575234767,
+            0.010892067708194324,
+            -0.008368819756628046,
+            0.012224564683325483,
+            0.031700389750709,
+            0.018063405869147564,
+            0.0010794115307202266,
+            -0.002783668216641561,
+            0.01418234057248712,
+            0.017773269840982066,
+            -0.015653443355569053,
+            -0.000497740485657824,
+            0.019890882604100452,
+            0.022672054781919238,
+            -0.00521797370595221,
+            -0.016686942830774276,
+            -0.018443538351123584,
+            0.025270057639201107,
+            -0.010354431070738671,
+            0.03484047753830186,
+            0.04594785214094975,
+            0.0036441959528283004,
+            -0.05695476763071743,
+            -0.009412209619190693,
+            0.0010371110229966086,
+            -0.01099343927053564,
+            0.01606863921935641,
+            0.012470660511773616,
+            -0.03237187923799263,
+            0.006388672213167589,
+            0.012946570716567364,
+            0.022152554488829627,
+            -0.02540317621005867,
+            -0.023183252921885226,
+            -0.0030984019855787655,
+            -0.007890467768631811,
+            0.018648760402624687,
+            -0.008653405980661855,
+            0.0006228315253506687,
+            0.009672526897422602,
+            0.0005436494014569835,
+            0.011632326145022935,
+            0.005655942007108228,
+            0.021736827172951,
+            -0.001256608167230203,
+            0.013124413886287987,
+            -0.01986808920858981,
+            -0.0270450339809516,
+            0.006500423802582728,
+            0.04438242119211055,
+            -0.014813399694814117,
+            0.030621709833336064,
+            0.0092171224268983,
+            0.016433384739642534,
+            0.024875072913970443,
+            -0.00891470467241686,
+            -0.014124009014664287,
+            0.00845777541223868,
+            0.009019007848927453,
+            -0.04147358041382793,
+            0.012513030150663913,
+            -0.01710243399313324,
+            -0.033144939719206276,
+            0.01904674924133127,
+            -0.00534122554342279,
+            0.012367320751926314,
+            0.004270315800911569,
+            -0.002886460711688153,
+            -0.025696286516581048,
+            0.014137901708875033,
+            0.028135087538950946,
+            0.021797869775098042,
+            0.0296173052503884,
+            -0.012757078368361444,
+            0.0005460601780756187,
+            -0.008748930817524899,
+            0.005470975782187157,
+            0.04187083128854477,
+            0.0008687182830616019,
+            -0.0008032802520818105,
+            0.01507726552487313,
+            0.024945744517309208,
+            0.0376130540370386,
+            -0.015415560911023553,
+            -0.03407353556457147,
+            0.01564969972264396,
+            -0.010104443707865262,
+            0.018855792875436674,
+            0.005503259932317811,
+            0.0049634097823351596,
+            0.0038600156268000702,
+            0.019312419743287703,
+            -0.006635193789316492,
+            0.018798045806429395,
+            -0.02500983683871332,
+            -0.028785757874583755,
+            0.01829411751138161,
+            -0.005277972443258226,
+            0.03338623299668019,
+            0.016225885095995672,
+            -0.029522280454919586,
+            -0.01712097344863975,
+            0.01553706187924322,
+            -0.008465766956343768,
+            -0.024842700926275538,
+            0.01895228828512998,
+            0.007076862934523977,
+            0.008956105593078965,
+            0.0024373779389833218,
+            -0.01106503696703573,
+            0.01125466547367417,
+            -0.000818223040067914,
+            -0.015322790791698395,
+            0.014157131555941659,
+            -0.017853386731281715,
+            -0.014994640991761465,
+            0.01775991780643474,
+            -0.037498078532902435,
+            0.009570197712419421,
+            -0.007687297543935123,
+            0.009050494195378254,
+            -0.004222041469820445,
+            0.007460383234429904,
+            -0.0010804379186060264,
+            -0.014987857934522812,
+            -0.014903749199810126,
+            0.03228485430019625,
+            0.022096121825475306,
+            -0.000410816936759558,
+            0.006728733918931589,
+            -3.971636461500232e-05,
+            -0.026719848032785824,
+            -0.011253751790407687,
+            0.02245467425830765,
+            -0.005664635254398481,
+            0.005134496095445943,
+            -0.01109590855846966,
+            -0.0149616794609307,
+            0.0019726017255571673,
+            0.032668637270108894,
+            0.06066635618119076,
+            0.012001732730659747,
+            0.024422093585231484,
+            0.0189499396761026,
+            0.02119276033547528,
+            -0.008489818917377516,
+            0.006413901834269739,
+            -0.03097413287789443,
+            0.03509225573986143,
+            0.005392594635384506,
+            0.03108937318306421,
+            0.026664331248813712,
+            -0.04722525921137681,
+            -0.0058539171830594774,
+            0.017087531454837677,
+            0.022673712717885323,
+            0.0159020681868943,
+            0.0008382498574190522,
+            -0.008938174268517353,
+            0.035650940044285984,
+            -0.03315250299348084,
+            -0.023647111263155,
+            0.005059382024092534,
+            -0.01439537467317451,
+            -0.010488359956731476,
+            -0.006785295580877329,
+            0.02776449681317373,
+            -0.005584052419598926,
+            -0.004582019433008779,
+            0.009697064252638535,
+            -0.014400206558190682,
+            0.008165676294736836,
+            0.0007624778531368436,
+            0.008681263897172326,
+            0.039249866981334126,
+            0.0005918063050375113,
+            0.024275699182767528,
+            -0.004138313072843555,
+            0.011079529739650594,
+            -0.0158972266904151,
+            -0.016985058543605,
+            0.02276982363172991,
+            0.00351101120496393,
+            -0.03244126813432563,
+            -0.00768819222024939,
+            0.00928180979884261,
+            -0.0013793404539641824,
+            0.005266058566444064,
+            -0.00503433670609399,
+            0.003747448261046857,
+            0.03286230666764075,
+            0.0022383592721685153,
+            0.008606190020480513,
+            0.013593947677257625,
+            0.00030951915677929204,
+            -0.01490307680740127,
+            0.013254591138113545,
+            -2.1379308136707722e-05,
+            -0.012900073369514622,
+            0.03357519186606058,
+            0.006705902094864628,
+            0.04243347200485736,
+            -0.03302953674711988,
+            0.009031534935013593,
+            0.01822227379789576,
+            -0.008491770335368761,
+            -0.02725412856390648,
+            -0.03630788717786838,
+            0.008259145747740535,
+            -0.019003260334340084,
+            0.015781275447792516,
+            -0.010039277287105153,
+            0.029665124310477384,
+            -0.03563370794521629,
+            0.004714858110268169,
+            0.004053913355055313,
+            0.0009503673086552651,
+            0.02068360255395961,
+            0.018524093113914366,
+            -0.002785698934204024,
+            -0.031443835341186485,
+            -0.02066365433385339,
+            0.006869321907853112,
+            0.02431260052207595,
+            -0.010367048093621498,
+            -0.005612149485859478,
+            0.0006316058139530722,
+            -0.024369562712568606,
+            -0.016845124984022828,
+            0.030359419843284823,
+            -0.040974427176153246,
+            0.005252911449822132,
+            0.04272936053727608,
+            -0.04599092715016919,
+            -0.03802288716943514,
+            -0.002314988728575789,
+            -0.024124014894379466,
+            -0.006360771882090754,
+            -0.002068030981218464,
+            0.016669991113175926,
+            0.0036232707799735537,
+            -0.028763764013482795,
+            0.00943401978066451,
+            -0.005173988601101547,
+            0.003971420424839974,
+            -0.0031669592494977393,
+            -0.02216336363004636,
+            -0.01657772032717298,
+            -0.02531962163186398,
+            -0.035174102848865116,
+            -0.0015903434876311088,
+            -0.005158703150821607,
+            -0.028703075279214722,
+            -0.03941940849540492,
+            -0.021696000925262537,
+            -0.0160221488026189,
+            -0.00843213135542359,
+            0.040641882367348774,
+            0.021319060666253945,
+            0.019336844555345886,
+            0.0025252167889382917,
+            -0.020346900609125973,
+            0.049440028814233025,
+            -0.007642182460707816,
+            0.035628691886108926,
+            -0.0017882888000877158,
+            -0.013461669667950893,
+            0.00455558389906464,
+            0.03825428026120858,
+            -0.014248434654950971,
+            -0.0009902062564027718,
+            0.00027811655483232545,
+            -0.0011866253757890136,
+            0.0008535155950871286,
+            0.015433650396777696,
+            0.001307651096343185,
+            -0.007112199543314309,
+            0.030380673552683906,
+            -0.012803247194895328,
+            0.01637501752192592,
+            0.018350814869962304,
+            -0.0013359719137099667,
+            0.008064546833360553,
+            -0.021430659268086814,
+            -0.00731572118780415,
+            0.0044578576331236325,
+            0.008422210247967642,
+            -0.015096885054883586,
+            -0.009964588397149773,
+            -0.02359494467299634,
+            -0.01174145797396426,
+            0.0023855749166744366,
+            -0.004559202896782688,
+            0.004902138778592298,
+            0.034595817216889944,
+            -0.016068744500427584,
+            -0.035106033294403614,
+            0.009997207702122002,
+            -0.02663088875167486,
+            0.02222306406130687,
+            3.983384942889326e-05,
+            0.006838221769859366,
+            -0.0023434342406307616,
+            0.001765445386886839,
+            -0.00169484278667461,
+            0.002862331185439823,
+            -0.018217975204395134,
+            0.0031859734246840257,
+            -0.0032119988065673565,
+            0.0009052950782312657,
+            -0.0012459653052875638,
+            0.00687339892509406,
+            0.006332799551164718,
+            -0.0008019538898592668,
+            -0.002968451740267742,
+            -8.541086355097327e-06,
+            -0.020561816931299912,
+            -0.021642512405352372,
+            -0.028259049261410905,
+            -0.000989887278779396,
+            -0.012359733017834338,
+            -0.027804394906433664,
+            0.011443385424045703,
+            -0.006872684999639775,
+            0.008327844771213648,
+            -0.008969527138034904,
+            0.006219994886600349,
+            -0.008415106130309062,
+            -0.0004143996206881767,
+            0.013119286511794273,
+            -0.02925209282917943,
+            -0.02096907906398536,
+            -0.03096513357712367,
+            0.01762223841790055,
+            -0.00925741670612824,
+            -0.002656382260603827,
+            0.0025223527960181835,
+            0.005092659166535706,
+            -0.033606542487570154,
+            -0.006653438998497412,
+            0.007313682256676618,
+            0.009547290751154595,
+            -0.03287788897629887,
+            0.02446728061355753,
+            0.017023222076526946,
+            0.01874233066565012,
+            -0.02417566312664911,
+            0.04049827067897005,
+            0.014426879325647042,
+            -0.021740606525480655,
+            0.016712907564723732,
+            0.027385150577780248,
+            0.03177163221499527,
+            -0.03469098547555545,
+            0.004481376248622418,
+            -0.01518200063132508,
+            0.018918680020516044,
+            -0.025284921244460287,
+            0.020625170445627613,
+            -0.016707629602243063,
+            0.021278719363178687,
+            0.03520334001264928,
+            -0.005524369131116028,
+            0.03608133014180745,
+            -0.019234548780647943,
+            0.0059985801736444555,
+            0.004041778461022664,
+            -0.01645503683802124,
+            0.01024694092614158,
+            -0.0042700351372843386,
+            0.0006059159967286885,
+            0.009406090340935747,
+            0.019133229156115282,
+            0.019007219188041775,
+            -0.04442208547053466,
+            0.028588965123434923,
+            0.00804120101594811,
+            0.021472086070876265,
+            0.014177820346835521,
+            -0.02708807970982664,
+            0.004742981711172693,
+            0.002072673190279628,
+            -0.004659713164250794,
+            0.0061167904730676514,
+            -0.013796604518930328,
+            -0.034540857808309226,
+            0.006338792319814448,
+            -0.018918703778187926,
+            0.015036937364271984,
+            -0.009026670935442384,
+            0.0011667525636770179,
+            -0.01152652811400482,
+            -0.011938679695598347,
+            0.0006170592505446954,
+            -0.005495151258143897,
+            -0.0008075736908949407,
+            -0.03171721592827769,
+            -0.04427658539412448,
+            0.02036132271908089,
+            -0.0021221965958423804,
+            -0.04320396743189005,
+            0.004111179168317842,
+            -0.025352428814134187,
+            -0.01865918186335911,
+            -0.011237699157960728,
+            -0.025047923887906474,
+            -0.04219630696740606,
+            -0.011108496740166623,
+            0.007785006066683012,
+            0.016810642610918525,
+            -0.009081815651299525,
+            0.040965574143584864,
+            0.0073944364393447965,
+            0.015313576815177263,
+            -0.017731360025694955,
+            0.026137777790243826,
+            -0.019314392060123,
+            -0.03108486330987073,
+            0.02676829566273746,
+            -6.259161089416372e-05,
+            0.003053748984540415,
+            0.014358356168574803,
+            2.8186924071913628e-05,
+            -0.006181789640967465,
+            0.01947730374003864,
+            -0.028739300217145797,
+            -0.023460503178221397,
+            0.0012460072421725688,
+            -0.008850395570726385,
+            -0.016271242375881602,
+            -0.035896509136022194,
+            0.005227881053956234,
+            0.03324337105109496,
+            0.004970419987430122,
+            0.008273497228454482,
+            0.01814500594735864,
+            -0.02229593571615308,
+            -0.024230675455380678,
+            0.000639112529814716,
+            -0.010266640429960994,
+            -0.010257247303470066,
+            -0.021302483296731352,
+            0.019150757066356872,
+            0.01651375496287087,
+            -0.00390189952706486,
+            -0.006971692709239087,
+            -0.010621003777733253,
+            -0.01918206725910585,
+            -0.02020345113211036,
+            -0.013827322378214754,
+            -0.017980206764311026,
+            -0.034153741666571154,
+            -0.032195001867229935,
+            0.03225205339201753,
+            0.026354817588919596,
+            0.0014076409241492998,
+            -0.007959081015805435,
+            -0.023070419006792155,
+            -0.004483328183581872,
+            0.02879516595345542,
+            0.002549934782150634,
+            -0.0001541849716285838,
+            -0.00830090759167373,
+            0.015341095409025125,
+            0.00835695019244165,
+            0.005107844984733648,
+            -0.06676974053142512,
+            -0.006027540973808174,
+            0.01655283585235268,
+            0.01351155740684989,
+            -0.002359114145814062,
+            0.02591301169926259,
+            0.01108099264396926,
+            -0.015361038608114896,
+            -0.019765743462262668,
+            0.035195761515422364,
+            0.02408910733949848,
+            0.023468767772473746,
+            -0.0034492999385544727,
+            -0.004472311490618405,
+            0.007030833258457338,
+            -0.010684252427867153,
+            -0.013326964647936429,
+            -0.004558513842504658,
+            0.026925146859337397,
+            0.016099443021514896,
+            -0.0064331198280825575,
+            0.010848931851939124,
+            -0.0008320209125061849,
+            -0.019444491995041537,
+            -0.001841330829156697,
+            0.008730275423516445,
+            0.0012974082030866743,
+            -0.014611902493400719,
+            -0.005911807768916134,
+            -0.019130604101278263,
+            0.024719468317569562,
+            0.0209456329723249,
+            0.03381727436368338,
+            -0.013237902591972769,
+            -0.0003695142331921278,
+            0.005410360755058108,
+            0.0002841632253252816,
+            0.0023352645111704958,
+            0.004801123870325242,
+            0.02623649813035263,
+            -0.029010710171522985,
+            -0.020741774066460415,
+            0.0068426980682519505,
+            0.013445270043414486,
+            0.0022744032984779294,
+            0.033840485942696216,
+            0.035040080489222324,
+            0.004166066423508316,
+            -0.010046666684988869,
+            -0.006148765564658713,
+            0.007071740742464309,
+            0.01865897335451031,
+            -0.023267411753900076,
+            -0.0005936657905496225,
+            0.004833030519903764,
+            -0.004229582648167378,
+            0.03620610659071198,
+            -0.009961238880412731,
+            -0.007197670122096852,
+            0.022212791947347127,
+            0.09819205436926473,
+            0.00047695621444943017,
+            0.0025354521225913395,
+            -0.029358210234742176,
+            -0.023614016211567963,
+            -0.0012564198366457442,
+            0.00242680708457405,
+            0.00830167274308547,
+            0.028076260517106013,
+            0.0004852400049405906,
+            0.011579274037419117,
+            0.02773972145320728,
+            -0.017217305534161797,
+            0.006752070301141707,
+            -0.011376051474975524,
+            0.024577071515205502,
+            -0.012694034026935636,
+            -0.0019809719792138673,
+            -0.016512649783050146,
+            0.015907402258634816,
+            0.004556574687833768,
+            0.004907510649166062,
+            0.011347630984408937,
+            0.009824484201337644,
+            -0.02650731900265866,
+            0.01756773113723491,
+            -0.005671986127099093,
+            0.004449611508541289,
+            -0.027841346572384462,
+            -0.020124233970502267,
+            0.0025986391930124847,
+            -0.0014440758308989809,
+            -0.009579444248630085,
+            0.006005274872617889,
+            -0.0024984235450999127,
+            0.03765664613203687,
+            -0.03765496603611941,
+            0.01409350051861866,
+            0.0066453175608372555,
+            -0.0015390841089599882,
+            0.011806653942640485,
+            0.017306914776093522,
+            -0.008206256939290151,
+            0.02589800448884443,
+            -0.00798213351353373,
+            0.009846274212000523,
+            0.015230070803055296,
+            -0.03449531102975745,
+            0.009174739184806312,
+            0.010145762060174144,
+            -0.024165660737765746,
+            0.012146877603301336,
+            -0.013784371016298891,
+            -0.004818759788054113,
+            0.005930446700613062,
+            0.007349837554764034,
+            0.010733731974181452,
+            -0.02295571464086674,
+            0.016176424826629898,
+            -0.010947841926210105,
+            0.0037766665765835445,
+            0.013559846998199008,
+            0.024892849808979323,
+            0.0012426907454129122,
+            0.0005950515136086193,
+            -0.01379469431504576,
+            0.005534724429523084,
+            -0.015667467221535825,
+            -0.022401586867694584,
+            -0.014192041775234079,
+            -0.006366846766730035,
+            0.01029598687444803,
+            -0.003972386708834858,
+            0.016971476088808737,
+            -0.010309648640697812,
+            -0.010645860303528099,
+            0.0037818310327362103,
+            0.01040806357838676,
+            0.008316605962234849,
+            -0.02855914079113881,
+            -0.005676310556150579,
+            0.004961084138163161,
+            0.00029285971503001265,
+            0.009592497954559435,
+            -0.014556073011576487,
+            0.013689526487160009,
+            -0.013290113354287095,
+            -0.0010726537498214612,
+            0.013201860397060236,
+            -0.012443552043930505,
+            -0.004012305744087929,
+            -0.01592905426965635,
+            -0.028487066798154617,
+            0.008462143660968396,
+            -0.01365614159523485,
+            0.001076882584445294,
+            0.0019503240125505646,
+            0.004730085646042709,
+            -0.016919069570642004,
+            -0.007860046623166481,
+            -0.033515448866521046,
+            0.011065416478687504,
+            0.02314589687882036,
+            -0.006435733825418645,
+            -0.030351727858789472,
+            -0.009546333526852862,
+            0.011327489786114963,
+            -0.0071765122743172715,
+            -0.01676736383051139,
+            -0.0029390038145944096,
+            0.007617910650439855,
+            -0.008174620754097493,
+            -0.00508943241317381,
+            -0.0025173969010414098,
+            0.0217328037301181,
+            0.016094975605203832,
+            0.01769642142712967,
+            -0.021400862603034036,
+            -0.00976992221747187,
+            0.026043879119776864,
+            -0.03275550610951898,
+            0.012441465975914529,
+            0.024587446266013183,
+            0.003446527051428522,
+            0.005492228054838714,
+            0.0030377554109693714,
+            0.005459159163772825,
+            -0.006985179226831048,
+            0.019925583798328282,
+            0.010870421035240845,
+            -0.019709062313471467,
+            0.005385443078244229,
+            0.02086085539375208,
+            0.0037248737640193097,
+            0.012892299156840998,
+            -0.017546870090541213,
+            -0.0020341807891543716,
+            0.015594779287017958,
+            0.04816214672670215,
+            -0.0032851797178091685,
+            -0.010450846098750195,
+            0.029513742590677183,
+            0.023526004023637855,
+            0.005373455940309873,
+            0.0012120777808635214,
+            -0.02525630456877349,
+            -0.013075130701774785,
+            0.01962235086017781,
+            0.016944870494393874,
+            0.013484925533584295,
+            -0.0023255656480479004,
+            -0.019236081983457366,
+            0.021520156296467717,
+            -0.01701733319731161,
+            0.012422659010697477,
+            -0.023764941037007197,
+            -0.001490790833263327,
+            -0.0035419008954878724,
+            0.021998949056452454,
+            0.0021819635638621092,
+            0.008891505304514552,
+            0.024354552127458608,
+            -0.006978548952551799,
+            -0.019526615985982355,
+            -0.0051799020400150305,
+            -0.014578953088901798,
+            0.022506635912517146,
+            -0.024117708614271965,
+            -0.007008641019892289,
+            0.011379795968360019,
+            -0.012362093847309038,
+            -0.0102534051092702,
+            0.001164934138876502,
+            0.017686807394319725,
+            0.0009423993435129515,
+            -0.014784508280754078,
+            0.01944445593807992,
+            -0.0011359271135469456,
+            0.010410915597949927,
+            -0.01921419010977189,
+            0.009390638453648025,
+            -0.016682442516992777,
+            -0.007849184048209236,
+            0.010343008873053859,
+            0.0012604521199837046,
+            0.003948300703645506,
+            0.002592490121754231,
+            0.007773437307969315,
+            -0.03355658426684504,
+            0.022695505487314516,
+            -0.010751606023585792,
+            -0.003538523688786008,
+            -0.0014125330167969218,
+            -0.01678692701672328,
+            -0.0006918162525881059,
+            0.005816502852598842,
+            0.010878358093847386,
+            -0.016045147059003256,
+            -0.00873270513444376,
+            -0.04920244395446931,
+            -0.030314009649638693,
+            0.023156333024481057,
+            0.013733657599720879,
+            -0.010971506342695331,
+            0.018852159415605117,
+            -0.023760853159532587,
+            -0.010587249404421732,
+            0.0011379869499042905,
+            0.023319856329056813,
+            -0.03539290744576506,
+            0.01215369703816749,
+            -0.014493682360886549,
+            -0.02039288669534806,
+            0.009070984596830142,
+            0.011445354735283254,
+            0.0007259852045513916,
+            -0.0011514627555568814,
+            -0.029350945123925453,
+            -0.010246845821727087,
+            0.02391068177425878,
+            -0.00861718589055971,
+            -0.004723044986826333,
+            0.024716520302409263,
+            0.0033721801437248873,
+            -0.012524663490691517,
+            -0.017955758068233754,
+            0.021612671655463236,
+            -0.026077203994996058,
+            0.007645944309444457,
+            0.010032183011253845,
+            -0.0173991263874487,
+            0.021649599230588117,
+            -0.007529701797485131,
+            0.020327763440441377,
+            0.004227390938169872,
+            -0.011249233530251522,
+            0.020352283974471443,
+            0.013644443758970054,
+            0.004216958800094986,
+            -0.004081669404784771,
+            0.006869353098014179,
+            0.02212732880780841,
+            -0.011021739614266063,
+            0.02219580717345023,
+            0.028777213295471067,
+            0.00017328191188631918,
+            0.03260179554743322,
+            -0.021287655520443285,
+            -0.03159728983004563,
+            0.021740163175540124,
+            -0.01035146858119169,
+            -0.021564906188712388,
+            0.028914245422221707,
+            -0.013674731836508768,
+            -0.00786476052825481,
+            -0.0166718429993922,
+            0.00533027311342322,
+            0.016680336311088843,
+            -0.005246190451159961,
+            -0.014360934066628332,
+            0.0029628103834216345,
+            0.019809487418628386,
+            0.014587234544007958,
+            0.029780009648869975,
+            -0.010778199815920887,
+            0.035180242185168895,
+            -0.002320576872571577,
+            0.008990157317473909,
+            0.012415465232073404,
+            0.006489435152972723,
+            -0.011251531176923445,
+            0.013501894212117511,
+            -0.003446759011266526,
+            0.008230548711588447,
+            -0.001449023474280409,
+            -0.035111679577202855,
+            -0.013709382199848678,
+            -0.04709294504969336,
+            -0.0023752976043235697,
+            -0.0025098360125255215,
+            0.0030975417740913154,
+            0.09736815869389318,
+            -0.005723049647039619,
+            -0.011988817571623457,
+            0.001625059666070763,
+            0.009611697638805556,
+            -0.0009896494420056814,
+            -0.007606836952841873,
+            0.015973152482488318,
+            -0.01343292635775299,
+            -0.02140928016038583,
+            -0.010532701190806147,
+            -0.01288594454368977,
+            0.012079198190522715,
+            0.015929716343161652,
+            -0.0005947359502228821,
+            -0.015207331663761012,
+            0.006524010655124294,
+            -0.004566385899752684,
+            0.0064717704997400245,
+            0.008647099975176308,
+            -0.010414668624343924,
+            -0.0071698879102925555,
+            0.030899143952119196,
+            -0.025301269549419674,
+            -0.024361748354052457,
+            -0.0010579931222861095,
+            -0.009055796737486008,
+            -0.010335459737012047,
+            -0.002692653633719985,
+            -0.0017679501322462056,
+            -0.010519994985692081,
+            -0.010019865263564572,
+            0.006537076627706128,
+            -0.015472130358273033,
+            0.004812857776601667,
+            -0.0047457011269495634,
+            -0.013296069496526007,
+            0.017054968086538724,
+            0.022563785357424853,
+            -0.02841491200807702,
+            0.0006755428946168761,
+            -0.017422510783648978,
+            0.012941211381006863,
+            0.008866069010589792,
+            0.006049618782357251,
+            -0.022451528209097505,
+            -0.023622906582403243,
+            -0.015019272754405791,
+            -0.01927535750568903,
+            -0.024973490129133732,
+            -0.009073381957684092,
+            0.0067692803507600945,
+            -0.001267339788153158,
+            0.021918281111033645,
+            0.000816538714279555,
+            -0.01812545613159305,
+            0.0061084913570894905,
+            0.000568371220952269,
+            0.018292497018668842,
+            -0.027468677288882186,
+            0.0033806422195878524,
+            -0.02307582511848465,
+            0.033508234091571124,
+            0.0039195050746594975,
+            -0.026680111514794824,
+            -0.021558668725755275,
+            -0.003311830287529699,
+            0.018994577319066473,
+            -0.0025745912019675805,
+            -0.00724049447539589,
+            0.014775207617625293,
+            0.0155334760323838,
+            -0.004693118176196483,
+            0.002015625406296403,
+            0.027968079002437102,
+            0.01077998866180723,
+            0.0313892866933904,
+            -0.015236169297590675,
+            0.004945828495617267,
+            -0.027215016713379062,
+            0.010214419950212732,
+            -0.01503548137805401,
+            0.008859554812646242,
+            -0.01290890682557532,
+            0.03531384210182567,
+            -0.012005613861257432,
+            -0.004250698950192301,
+            0.03253288796441852,
+            -0.029150986393345252,
+            -0.0035020051536906334,
+            -0.009255437891304041,
+            -0.004142204892160768,
+            0.013240580426083011,
+            -0.001198093874819309,
+            0.008485141356876356,
+            -0.01854370462655459,
+            9.726096864073609e-05,
+            -0.003221852304410318,
+            0.005404595442127598,
+            -0.02005458688172967,
+            -0.003314118809778734,
+            0.0073427568419969814,
+            0.004259518237708634,
+            0.002408630356035373,
+            0.016420646142725346,
+            0.006662121128241274,
+            -0.0029357701493526907,
+            0.03424224243607692,
+            0.008171895850772245,
+            0.0152521424877615,
+            -0.0018638264193202728,
+            0.0030234211261443034,
+            0.019287963516760636,
+            0.019912956661192856,
+            -0.00048443399729935116,
+            -0.01058935153883094,
+            0.00195113175754497,
+            -0.001140732137433033,
+            0.011048741602327437,
+            0.024396046054912645,
+            -0.021158025221644738,
+            -0.02419562092713724,
+            0.016377661005279716,
+            -0.015076145951740961,
+            0.01903712233378994,
+            0.010541463112889985,
+            -0.010099351302095792,
+            -0.00031558616108876247,
+            -0.042316016201379245,
+            0.010964989031040692,
+            -0.02068939391415103,
+            -0.04004120028915868,
+            -0.00244383573783919,
+            0.0009542507389088937,
+            -0.029439456243505052,
+            0.02822814855547978,
+            0.0023755032182188156,
+            -0.017280221740813364,
+            0.018223709137876153,
+            -0.005548654680726505,
+            -0.0024040301239229064,
+            0.005397711787238679,
+            -0.01092026090471121,
+            -0.010648485086769927,
+            0.008235250252660972,
+            0.010957715075632535,
+            0.005425683656535705,
+            -0.005698919020608095,
+            -0.011284911899894062,
+            0.009975882651638084,
+            0.012318901214538077,
+            -0.0019922884373083155,
+            0.010050520101815593,
+            0.0031720752428247446,
+            0.007352859750323051,
+            -0.0073483049386141005,
+            0.0046483587496626965,
+            -0.005093220128411125,
+            -0.026080057085901446,
+            0.012863580653903103,
+            0.0052434400313520774,
+            -0.011863360599637747,
+            0.0103209502958491,
+            -0.01048162756158923,
+            0.005787157766911021,
+            0.001318945460232246,
+            -0.0059333230422061395,
+            0.004821320238893602,
+            -0.0038922239931512784,
+            0.011267634571983924,
+            0.013256611037442042,
+            0.004592529900805719,
+            -0.013076954459595086,
+            -0.02177318635989856,
+            0.009001291224623467,
+            0.01959767163356408,
+            0.024161754192119852,
+            -0.030416109293488064,
+            0.003784634588325551,
+            -0.0014915849060829743,
+            -0.0020550564975080885,
+            -0.015632785947151753,
+            0.01587790981601866,
+            0.01716742056401084,
+            0.012721563202528083,
+            0.01893393006967701,
+            -0.004009617703771589,
+            0.010539100665843704,
+            -0.02080068888791658,
+            0.013642451766699067,
+            0.0008609869904066794,
+            -0.013629067231663643,
+            0.0014620655123815298,
+            0.027046304278766087,
+            -0.011056783825898616,
+            0.006206969699250696,
+            0.008097527175511791,
+            0.022111870736907183,
+            -0.012509479100313492,
+            -0.003735868296238069,
+            0.01516254968167132,
+            0.0014878674796313539,
+            0.0035887079417890352,
+            0.016533149527222273,
+            0.04729052276110659,
+            0.00102751186013391,
+            0.0016713773449738153,
+            0.0192177920520243,
+            -0.005563519063161466,
+            -0.01114821419109643,
+            -0.003549999863325982,
+            -0.021807893906469106,
+            0.010998558936481785,
+            0.00431279785115251,
+            -0.02186324212032519,
+            0.01024093366826206,
+            0.0125495473550584,
+            0.005268320135120931,
+            -0.0031679212410026603,
+            0.02574575247827441,
+            -0.019807879421996258,
+            0.014935295894965128,
+            0.03329450628384787,
+            0.00493356778223407,
+            -0.011423949333937285,
+            0.007263264249871384,
+            0.0027354459678742013,
+            -0.004494406041906684,
+            -0.00606745413660583,
+            0.010497391290734088,
+            -0.008191702034391382,
+            0.009870458191510176,
+            -0.009580130479693,
+            0.039202091799313755,
+            -0.009619018331262844,
+            -0.010782138248435458,
+            -0.004558645539551881,
+            -0.006241773108460662,
+            0.003478414182938609,
+            -0.003395165694642073,
+            0.005812225135633631,
+            -0.0069045407353789945,
+            -0.014607146220271209,
+            0.0008522982757210454,
+            0.00020175973815948303,
+            0.014001922919116725,
+            0.0011804904415274088,
+            -0.004871874532755353,
+            -0.007877217154706205,
+            0.015243277056007988,
+            -0.006047906208304912,
+            0.0006531937033317192,
+            -0.0011653523616313422,
+            -0.0013662556399359335,
+            -0.007401050012179761,
+            -0.015166105795085528,
+            0.00974609244984173,
+            0.016253733908242134,
+            -0.013857834499211056,
+            -0.005142936208363415,
+            0.012274716212832219,
+            0.008540229125193836,
+            0.011742606168187876,
+            0.0007451338650642946,
+            -0.009255813682572937,
+            0.008292531177374192,
+            -0.015020523035432158,
+            -0.008781317965985481,
+            0.0026250469418136603,
+            0.002381147382977664,
+            0.005839723981046825,
+            0.0006240641828668715,
+            0.004502463508605272,
+            0.005798912705218846,
+            -0.00497936644919367,
+            0.01112666823857836,
+            0.008198424171006548,
+            -0.002452143482105151,
+            -0.01017569410837814,
+            -0.001127589984272638,
+            -0.015733298167661346,
+            -0.016758315411028353,
+            -0.0009286266498936771,
+            0.015175988113750882,
+            -0.02352862898718829,
+            0.018996352975391644,
+            -0.0014524363611708201,
+            -0.0033802546171857704,
+            -0.021392428456030312,
+            -0.0025923983911570183,
+            0.0034214188668518796,
+            0.01255768230394476,
+            0.01858853630745928,
+            -0.02729543519465384,
+            0.010575894388563116,
+            -0.007204725245442141,
+            0.030394929424014717,
+            -0.018328782140005044,
+            -0.004022941273610227,
+            0.026915618079621992,
+            -0.02200018306828758,
+            0.0008791185377513978,
+            -0.011454606955158956,
+            -0.008103032693853504,
+            0.019287674837688177,
+            0.023747720480736777,
+            0.02038287242167783,
+            0.01364489294869211,
+            0.0007758044919789085,
+            0.004768030222445397,
+            0.03572705648994403,
+            0.010568241270862248,
+            0.014828087092859878,
+            -0.007542489523110974,
+            9.457808640264811e-05,
+            -0.02554033312739386,
+            0.01635899434369978,
+            2.587383534566272e-05,
+            -0.012370727506722627,
+            0.007162422186621379,
+            -0.013370882403931942,
+            -0.011066483600541058,
+            -0.007151077450730025,
+            -0.01721910018182534,
+            0.02152239849400241,
+            -0.012322440280680164,
+            0.004587729883596212,
+            -0.009079747696669277,
+            -0.013272707286450417,
+            -0.012953682633400515,
+            -0.014202214807187545,
+            -0.01313141988034687,
+            -0.026041403779538307,
+            -0.027109731260915763,
+            -0.006573285295895077,
+            -0.023780366346334524,
+            -0.013799072648505526,
+            0.005588622594136253,
+            -0.022617902666316757,
+            0.012262325666428604,
+            -0.012689827812482627,
+            -0.01226264872059864,
+            0.012805734807835008,
+            0.020274844315616616,
+            -0.011765231803328348,
+            -0.028494489703400308,
+            -0.018583843821043854,
+            -0.019970998470704358,
+            0.012576201137505969,
+            0.002266288538944273,
+            0.013165889075444961,
+            -0.003018437854660793,
+            0.003227412667880239,
+            -0.014895831590298957,
+            0.018570624675927183,
+            -0.03442390703780963,
+            -0.03108003806963536,
+            -0.013611306765348361,
+            0.005182777066476028,
+            -0.0015782033203578553,
+            -0.011429046504256945,
+            -0.0018606648572920801,
+            -0.00433805271182878,
+            -0.025109572024446088,
+            -0.010134856632190067,
+            0.011010662054484161,
+            -0.015205629927469414,
+            0.0072933565255675036,
+            0.0009018832564038863,
+            0.013870902834260887,
+            0.009587432944616442,
+            0.013393301893492882,
+            -0.006095063698384775,
+            0.004356485794113208,
+            -0.0026476612747324386,
+            0.0004894024149759108,
+            0.009336366919474478,
+            0.00839591561027103,
+            0.010551161369079399,
+            0.002260707777260338,
+            0.007830059223001696,
+            -0.0060927446216633265,
+            0.018065002687169148,
+            0.015436944875538481,
+            -0.0002520689442496089,
+            0.013323098485251487,
+            0.0003769862591044255,
+            0.00022293304921127018,
+            -0.0029970303106340466,
+            0.007888437018607987,
+            -0.011785957933642163,
+            -0.001311704406355858,
+            0.015138190893483804,
+            -0.01707247730569978,
+            -0.013045187198628258,
+            0.03107034781283638,
+            0.0018583638488268697,
+            -0.02495564203304796,
+            0.031146554085787762,
+            0.004220047610942943,
+            -0.005835199594223787,
+            -0.013205086624848782,
+            0.0014059047822222418,
+            -0.011570624844100298,
+            0.003584232809362274,
+            0.014295032061446826,
+            -0.02231984223610487,
+            0.010521094801839404,
+            0.008446892441796151,
+            -0.009681535919980559,
+            -0.012337659795586223,
+            0.009847073709209602,
+            0.016809328445200358,
+            0.004472485504034416,
+            -0.03539093810210523,
+            -0.011957694102753411,
+            0.02145470732008716,
+            -0.029237428225009913,
+            -0.00810041956259673,
+            0.007452962400105787,
+            0.0065154832040426065,
+            -0.019353109017609053,
+            -0.008580955020412362,
+            0.005725794605897509,
+            -0.004940390374637084,
+            -0.003892210172017758,
+            0.017541623128534682,
+            0.00485047043626186,
+            0.013698341792101025,
+            -0.01689738578307474,
+            0.009771458057400557,
+            -0.022222817828602557,
+            0.004021882074676753,
+            -0.023373700500461992,
+            -0.009873403167702603,
+            0.002364143609079849,
+            0.012454271200139762,
+            0.014328556467632212,
+            0.027839363143659326,
+            0.030056404238348527,
+            0.013995696971422429,
+            0.007833677585250953,
+            -0.016265199386953717,
+            -0.018134327237616175,
+            -0.006409513840075933,
+            -0.004033173350204784,
+            -0.008714798059218596,
+            0.0066373547199279235,
+            0.01858354212009877,
+            0.004499712802472377,
+            0.02609078206670711,
+            -0.006227189596272876,
+            0.009039744622014585,
+            -3.127526911951914e-05,
+            0.0008833930131827993,
+            -0.004226083827636182,
+            -0.01971685546584527,
+            0.00481003211084742,
+            -0.0013721482162825508,
+            0.025874758880591167,
+            0.00940496860916206,
+            0.0024915013399541847,
+            0.01615978314811047,
+            -0.006378952757397484,
+            -0.01174653607178726,
+            -0.010540107733238588,
+            -0.0061926268398085176,
+            -0.021752582127981104,
+            0.029059100503124387,
+            0.021195595568253612,
+            -0.010800765264123782,
+            -0.013693582166598082,
+            -0.00903369526059487,
+            0.02795767990556847,
+            0.004080965688836953,
+            -0.0013050212744006933,
+            0.016313311850308505,
+            0.012084555958999215,
+            0.012329325712491252,
+            0.009794698207689061,
+            -0.00036157148861933123,
+            0.0015364709122308352,
+            0.03722522141440073,
+            -0.007427028510548186,
+            0.0026463292682538495,
+            -0.005239324411651515,
+            0.0025987932868461313,
+            -0.009705817520422487,
+            0.007242122655628283,
+            -0.004012676385296197,
+            -0.0158760051470016,
+            0.015003444174308607,
+            -0.022443331305169994,
+            -0.0016749787204471238,
+            -0.008150635190952799,
+            0.0016696640640947303,
+            -0.02781023337840043,
+            -0.0025731206607371845,
+            0.006695065701630532,
+            -0.015547340828489069,
+            0.011816010501093885,
+            -0.01740673229051557,
+            -0.011553604323149105,
+            0.007914961618894248,
+            -0.012034157288065447,
+            0.0016684130271324446,
+            -0.0008268957621923278,
+            0.004460864332388434,
+            0.013426055079012884,
+            -0.011008808570221245,
+            0.012166046631826273,
+            -0.005816284310138776,
+            -0.007872091214834688,
+            -0.0016055129487028885,
+            -0.0030049472973942043,
+            -0.0025600746761171543,
+            -0.0009966231170649917,
+            0.034355910234323155,
+            -0.004446570896048543,
+            0.003761242147499512,
+            -0.01362639743248326,
+            0.0019833052855181806,
+            -0.036176376394999044,
+            0.028344464817309477,
+            -0.008642835889481802,
+            0.011013502221008498,
+            -0.0017508365480865596,
+            -0.013243706268954915,
+            -0.01456517309747086,
+            0.015247299501245178,
+            0.007602596572394442,
+            -0.005854531057349665,
+            0.013558248087347848,
+            -0.019724373614448298,
+            0.008530500822910953,
+            0.004463694543554244,
+            -0.015561397976987216,
+            0.004668780190328835,
+            -0.007479912658259757,
+            0.012657201279639194,
+            -0.01774730033362932,
+            0.004314594593150765,
+            0.00958571617187387,
+            -0.01892876084898881,
+            0.009901041800918215,
+            -0.028675879564512306,
+            -0.004631920242208613,
+            -0.012323455524366223,
+            0.0053804608827285865,
+            -0.000660864620708182,
+            -0.02389088252276507,
+            -0.028944280660230762,
+            -0.0002099667627277945,
+            0.00458533068247518,
+            0.0018878351997997833,
+            0.011621407953503755,
+            0.005970807015151215,
+            0.008542958629199003,
+            0.01023896054756221,
+            -0.00205207247588841,
+            0.009402752124876478,
+            0.04169706217414899,
+            -0.022899912321317197,
+            -0.011574970753549647,
+            0.005321872977369877,
+            0.011499271505460519,
+            -0.02647653483798284,
+            -0.011039791758347888,
+            0.0033424345263339144,
+            -0.005219877979550397,
+            0.03609469691834737,
+            -0.02448746405881972,
+            0.019020618652020673,
+            -0.011392297116547224,
+            0.007625475937968714,
+            -0.021465240076078376,
+            -0.003921088264697311,
+            -0.014265745991522477,
+            -0.00011221615798127438,
+            -0.0031478236827989703,
+            -0.0076053733863387755,
+            0.007485039021296698,
+            -0.001907678661303243,
+            0.0020348246022161925,
+            0.006291436821831794,
+            -0.02173305455485444,
+            -0.0019125499969493854,
+            -0.016432460924913166,
+            0.030367263349759286,
+            -0.016242095671439496,
+            0.005072805486919414,
+            0.0053258139728101506,
+            -0.002810639419737258,
+            0.012368771814660942,
+            0.004165967191536186,
+            0.00513754440091394,
+            -0.004199767241827414,
+            -0.005142642360546972,
+            -0.0053886448467479215,
+            0.017868874549760376,
+            -0.013280048598849825,
+            0.035024701466710065,
+            -0.0100959186045929,
+            -0.007094001877261568,
+            0.0019733042325232155,
+            0.010380430710282744,
+            4.2084202709296443e-05,
+            0.01053565670076933,
+            0.0034270489770133685,
+            -0.0006311078385359693,
+            0.0019774779328212756,
+            -0.017162448473652843,
+            0.003089758682621882,
+            0.01682656561693936,
+            0.005463677441664502,
+            0.004469261499451007,
+            -0.013332130891003062,
+            -0.012050233723282738,
+            0.004538268704442613,
+            -0.007533940408255447,
+            0.0003675737140813486,
+            -0.018816953644773076,
+            0.013972696748575243,
+            -0.0011096683686631858,
+            0.0002715683052236706,
+            0.00524678369365015,
+            0.023056514997908218,
+            0.01813167860994747,
+            -0.002553352602551003,
+            0.03155164978556312,
+            0.0174738879054593,
+            0.0029268039949260673,
+            0.010637295711777793,
+            -0.033222781195984666,
+            -0.013301873512972214,
+            -0.0013721703891642354,
+            0.02238421933149517,
+            -0.01530189703335162,
+            -0.014059029663525449,
+            -0.0037393534403749947,
+            0.0010394601933309767,
+            -0.005177359933828505,
+            0.0042489306011644495,
+            0.011039889511148341,
+            -0.014697595240068863,
+            0.007597149702441915,
+            0.0026780496822907634,
+            0.02092734730183164,
+            0.0004688144432243786,
+            0.017181410417231698,
+            -0.0005083970646221274,
+            0.00749639055342427,
+            0.007319420826337891,
+            0.017541463125355265,
+            -0.01002811389296805,
+            0.023582113845081534,
+            -0.008823933397729605,
+            0.0019889258902730535,
+            -0.01257106158422629,
+            -0.028159594067544737,
+            -0.012624750220149823,
+            0.015247564545279736,
+            0.005765911899842925,
+            0.006478299226263732,
+            -0.016537090297144265,
+            0.006276997440048981,
+            0.015647122171767974,
+            -0.002251683151059454,
+            0.010295358869874306,
+            0.009730378299068289,
+            0.00012816405890877098,
+            0.010181120688239766,
+            -0.00801256937783534,
+            -0.00734257767723293,
+            -0.004497520124843868,
+            0.023982754195729983,
+            -0.005443550092842393,
+            0.022434177122972028,
+            0.012647958478747348,
+            0.00616791857573897,
+            -0.010853355357954727,
+            -0.003340111254357573,
+            0.023389634440057473,
+            0.011863246790550376,
+            0.00029642699874398923,
+            -0.009846034060495678,
+            0.0019458876579524617,
+            0.0054010373765486005,
+            0.012983913038716568,
+            -0.02049427270240938,
+            -0.004344571352918878,
+            0.008461947538684664,
+            -0.008472919299762712,
+            0.004318309005722874,
+            -0.030156444522730416,
+            -0.005812303237222271,
+            0.021354352236942434,
+            0.008392281015122372,
+            -0.010232039722494646,
+            -0.004395682831498155,
+            -0.0006527587802367882,
+            0.0040601850269714945,
+            -0.0024323167053212176,
+            -0.0023729138545912176,
+            -0.008541233856441836,
+            -0.006470933419694742,
+            0.014035696617797647,
+            -0.0052691760528871924,
+            0.008597459268477037,
+            0.00012084859681905111,
+            -0.0017255958688825231,
+            -0.007323529232751411,
+            0.03382792341957058,
+            0.008087732655845718,
+            0.005980666974986827,
+            0.0013613625176327891,
+            0.0042244330945601215,
+            0.000616791811672998,
+            -0.0028374856646453707,
+            0.018075360591135542,
+            0.0156738939725117,
+            0.005054696911814988,
+            0.009038525980341447,
+            0.014672270369917403,
+            -0.019700405219230957,
+            0.012625337070556061,
+            0.012184185186281482,
+            0.019382321278525114,
+            0.0007219498737851579,
+            0.002564970959687783,
+            -0.010601873823737926,
+            -0.0029963537053764736,
+            -0.002070567089566093,
+            0.029563028293936092,
+            0.0005903976560069006,
+            -0.010509758568838935,
+            0.010992421945232951,
+            0.020603868018231348,
+            -0.02101157589451638,
+            -0.001971974816402967,
+            0.005963911670920596,
+            -0.010704207628592966,
+            -0.014466113540150018,
+            -0.021269176952661614,
+            -0.013331247081394942,
+            0.0071597345125287174,
+            -0.008162102923896894,
+            0.02367549407145414,
+            -0.014256771777502395,
+            -0.03960796112042958,
+            -0.014803626428839152,
+            -0.00864320347698291,
+            -0.027531337207277483,
+            -0.007305441146142331,
+            0.00965056093889974,
+            0.016698589600806273,
+            -0.003330867073501157,
+            -0.025198703196464594,
+            0.030483790406960656,
+            0.008872821868559751,
+            0.023708422767352917,
+            -0.021464374508407488,
+            -0.0018617153194555344,
+            -0.00886376784514331,
+            0.014505918224188001,
+            -0.017588761818663975,
+            -0.011756112651591795,
+            0.020014632992762026,
+            0.014356419813769714,
+            0.007246557500044434,
+            -0.004355132555323843,
+            0.0067299398462013936,
+            0.007649499235136303,
+            -0.012515181210363267,
+            0.012239792251154642,
+            -0.013499277313916029,
+            -0.024858295126228433,
+            0.014430561986516884,
+            0.002229494935179828,
+            0.009563004428064055,
+            -0.0015984171085659323,
+            -0.016197659608768088,
+            0.010078159490026602,
+            0.00808729175515505,
+            -0.017508840440512225,
+            -0.004793934415922065,
+            0.007133527615305415,
+            -0.020818869260580315,
+            0.031234686259160864,
+            0.011872765152279094,
+            -0.010692949770503467,
+            -0.001989039629661815,
+            0.004431079028880923,
+            0.016511544608975148,
+            0.007893700385963683,
+            -0.01997314388229276,
+            -0.003708006454566117,
+            0.01530128306203401,
+            0.00822588305477249,
+            0.004861703074728471,
+            -0.010836328783594584,
+            0.03434874763691911,
+            0.006082873924472328,
+            4.192481378506604e-05,
+            -0.0052997095110967115,
+            -0.0076108924981838355,
+            0.004196112457289409,
+            0.018905391806961686,
+            0.013330078288599763,
+            -0.004200590783114919,
+            -0.006630107527854746,
+            0.007778448128358027,
+            -0.01706991160178345,
+            0.007329056462125937,
+            0.009390327499494826,
+            -0.012870674959022891,
+            -0.0017571189976160295,
+            0.016462608870101713,
+            0.00209846661139665,
+            0.004942714850185376,
+            -0.011305050739130339,
+            0.004252776146827917,
+            0.015841821944109317,
+            0.018727567181082646,
+            -0.008399642221255733,
+            0.021448055799259867,
+            -0.020972459569228434,
+            0.027243639465535856,
+            0.028499400850980086,
+            0.016573144403275928,
+            -0.01847096443459798,
+            -0.0022409373140116146,
+            -0.0041737366139895414,
+            -0.014761716560267328,
+            0.0034546020161337894,
+            0.005125870074458691,
+            -0.01174186608185822,
+            -0.009577921232380895,
+            -0.024331993794416026,
+            0.011075959943215117,
+            0.007918147902741738,
+            -0.0015045185909419564,
+            -0.011791155919533844,
+            0.016019077577552835,
+            0.010141868950981925,
+            -0.010938402325619172,
+            -0.005635351352447256,
+            -0.019967361901734836,
+            0.009750427567651833,
+            0.009128294017175602,
+            -0.00300556760891084,
+            -0.011496103417643132,
+            0.012977890524465716,
+            0.0014080059843484115,
+            -0.01292380658817521,
+            0.010841954636959441,
+            -0.03142011651894961,
+            -0.005510537354374977,
+            -0.011027026734432255,
+            -0.003620482919145167,
+            -0.014415606143566519,
+            0.003608572766363589,
+            0.0005034611321129689,
+            -0.011470488351158329,
+            -0.0035005774696278,
+            -0.016207453351921627,
+            0.004676327282262429,
+            0.015021513761697039,
+            0.001679614645967629,
+            -0.006917375094634678,
+            0.021994274901326064,
+            0.018518524483306403,
+            -0.021903538417919984,
+            -0.020116013379669034,
+            0.004708084983787958,
+            -0.03261690161800251,
+            0.015019348187107588,
+            0.00966124471577622,
+            0.011676937880751252,
+            -0.028768229226408037,
+            0.007114534979415504,
+            0.00900420165933673,
+            0.016398927481511824,
+            0.0050879301160439245,
+            0.009453255486450082,
+            0.01222758288108498,
+            -0.0104657614414275,
+            -0.003164502778525433,
+            -5.410382753651127e-06,
+            -0.005618225986727351,
+            0.015607620507415484,
+            0.0008527697935056332,
+            0.03603666488542239,
+            -0.005277207843139309,
+            -0.0009799879252482288,
+            -0.011735731235560851,
+            0.00812046801749312,
+            -0.011713349784345638,
+            0.006334172347602515,
+            -0.0024643499727789167,
+            0.0001240009282616789,
+            0.002965067818077541,
+            0.009442572075213767,
+            -0.01086095783353607,
+            -0.007595379345224076,
+            0.007444897256325813,
+            -0.01482814340478338,
+            -0.00019984717819268013,
+            -0.009339343730980451,
+            0.0019680804106275156,
+            -0.02241031895876346,
+            0.006629838013824249,
+            0.040935644507527846,
+            0.02140216465404127,
+            -0.005481040712269146,
+            -0.012431659209559418,
+            -0.004680230194268222,
+            -0.007229300034633229,
+            -0.006482401275991039,
+            -0.004794698160419071,
+            -0.022762405726589375,
+            -0.006928553505112732,
+            0.004320321244630314,
+            0.013300138724212552,
+            0.009741018117644266,
+            -0.008842135469729511,
+            0.004531760574809748,
+            -0.02662650543210291,
+            -0.016578081411548973,
+            -0.0060021425889846495,
+            -0.015840130031005147,
+            0.006125301421471048,
+            0.027521841635711787,
+            -0.0063432924325983525,
+            0.01789356321527744,
+            0.005142029893259461,
+            0.013034833065163448,
+            -0.005338869351330258,
+            0.00035250243782976347,
+            -0.006010938630154189,
+            0.0032958656042048292,
+            0.003074861351445954,
+            -0.0015947217852680714,
+            0.004151590621941346,
+            -0.007139285822575166,
+            -0.0026091937868967284,
+            0.0011629380311555433,
+            0.0117841884590861,
+            0.006877419488576261,
+            -0.00016252653621991807,
+            -0.01761285866765538,
+            0.022221104524808997,
+            -0.018933508559881868,
+            -0.008840161102111973,
+            0.011092684029552064,
+            -0.009660487856185668,
+            0.00702597023411946,
+            0.020387386977378873,
+            -0.006854213562767843,
+            -0.009583981919303215,
+            0.013028181087649146,
+            -0.0036507672754997286,
+            0.004464368543493108,
+            0.009867428978560914,
+            0.006036303973414791,
+            0.002482944923100156,
+            0.013244650094892797,
+            0.004530294330688855,
+            -0.0032625482538717137,
+            -0.020238533752799943,
+            -0.005186498720507101,
+            0.019015752262933585,
+            -0.02252458872825473,
+            -0.0027641215988942563,
+            0.0044506144202026456,
+            -0.0030317840657635457,
+            -0.01733962766050954,
+            0.004205861489436576,
+            -0.018770275720867473,
+            -0.0033883519277721433,
+            -0.008244668812709337,
+            0.009948090119054079,
+            -0.023700732627507225,
+            0.009109315916173436,
+            -0.011739192106106307,
+            0.02356509175177699,
+            -0.003925851077921686,
+            0.024734433836185024,
+            0.0015142645677087268,
+            -0.013482933030628261,
+            0.0055323884713736554,
+            -0.0008930661843169134,
+            -0.004477731568313757,
+            0.029338490404300612,
+            -0.005418258701133154,
+            0.002249996261871466,
+            0.013491124734151694,
+            -0.021332541202417246,
+            0.021773687217263173,
+            0.0046662696076723836,
+            0.01628697326539432,
+            -0.00986238951830334,
+            -0.013799209313714441,
+            0.025178874553620743,
+            0.03324555035648896,
+            0.008398796308070151,
+            0.011261509865513953,
+            -0.01226890254177634,
+            0.018966175575083295,
+            -0.0038328997984309597,
+            0.004344837325292706,
+            0.00878668880458693,
+            -0.007200842513742962,
+            -0.011500562922303248,
+            -0.004533014670236755,
+            0.027639894892383943,
+            -0.021838848944317424,
+            -0.008920629621611768,
+            -0.005739181836252042,
+            -0.01593272996094732,
+            -0.038167534267554606,
+            -0.004371701247332816,
+            0.019604405563120094,
+            -0.005974358141737469,
+            -0.0018341151594086288,
+            0.014334266690716505,
+            0.00402620072221496,
+            -0.002852569009185304,
+            -0.008195218707838913,
+            -0.0027944153624309856,
+            -0.0059390492900516225,
+            0.016651206007022145,
+            -0.004961864528562493,
+            -0.007929830585906699,
+            0.01145431100211762,
+            -0.005731957927601062,
+            0.012130694936714224,
+            0.0004848418056781251,
+            8.392048424584131e-05,
+            -0.0015174120840537315,
+            0.00822932723013239,
+            0.018620869354789082,
+            -0.0055803454666417465,
+            0.025682778991091174,
+            0.006956936033581638,
+            -0.003495709501259274,
+            -0.005489431577052735,
+            -0.0044154293119170445,
+            -0.024561016942073285,
+            -0.0020864636079347357,
+            0.006593060072668424,
+            0.024333813203596122,
+            -0.02515989187381735,
+            -0.004953266000463339,
+            0.020957806164031402,
+            -0.0012192724005613,
+            -0.011978722479596745,
+            0.014690559986694138,
+            0.0041346025614047875,
+            0.009881584250499358,
+            0.012176393203318002,
+            0.015764530703333238,
+            -0.004923548157246672,
+            -0.0015192775000207833,
+            0.012766482928380375,
+            0.0005745973040874939,
+            0.00997809510696241,
+            0.016495444272181515,
+            0.010467514633877887,
+            -0.021108281220544853,
+            0.0015078843789125156,
+            0.00522593579521476,
+            -0.017481430635971396,
+            0.0034362821987740755,
+            0.005397819166037461,
+            0.011626527721895694,
+            0.004048632382026344,
+            0.013207335578216437,
+            -0.015768003434275563,
+            -0.005916168437120099,
+            0.008153798523776654,
+            0.004319451296835019,
+            -0.02599474886669325,
+            -0.01485009085106089,
+            -0.0006781964116146299,
+            -0.004444144921332929,
+            0.013556265879295017,
+            0.01285768162554067,
+            0.02033626001285354,
+            0.024637349990579514,
+            -0.013833060730799217,
+            -0.00842819657792762,
+            0.009436072455719896,
+            0.015560357465711765,
+            0.005798181411384387,
+            0.007874051721372273,
+            0.009297440071118126,
+            0.0077320823375042,
+            -0.004073209576661068,
+            -0.002973928843567532,
+            0.020205003097225747,
+            -0.019725382393637726,
+            0.0006746659838642147,
+            -0.018417443693472958,
+            0.007530796896254817,
+            0.01140696723506382,
+            0.01129515042745398,
+            0.002816883643872417,
+            -0.005274757308627299,
+            0.005716917047812153,
+            -0.06254973656832136,
+            -0.009191453047502128,
+            0.003384751551032943,
+            0.027256466708746898,
+            -0.010885336140129142,
+            -0.003415707910562931,
+            0.007309577090125959,
+            0.005511970708769456,
+            0.0005572239548625122,
+            0.003186427278034852,
+            0.019698716902282903,
+            -0.02052581021967601,
+            -0.010793968505805883,
+            0.004219269922138548,
+            0.0033111837923746746,
+            -0.007269521534951044,
+            0.004805737092302134,
+            0.014143978208501908,
+            0.01986656063456352,
+            -0.0024048854610842748,
+            0.01629335842915328,
+            -0.0015498603704589015,
+            -0.014496994444493684,
+            -0.002671064325909516,
+            0.0037875833047555925,
+            -0.022466433279066646,
+            -0.005980897597195858,
+            0.0043262697442044435,
+            -0.011226121648007667,
+            0.006978495397818936,
+            0.008452177557316863,
+            -0.003018545465557218,
+            -0.0008303307644324117,
+            0.017796789945838845,
+            0.006591364879905198,
+            -0.013464486184025026,
+            0.0028668426952326957,
+            -0.023377605222213174,
+            0.004325185794796361,
+            0.001416245546006914,
+            0.007809449561523174,
+            -0.008431335139468173,
+            -0.014774499666285603,
+            -0.005933810712528226,
+            0.017746847020610673,
+            -0.01965188730040385,
+            -0.002406875192894708,
+            0.01683009662984443,
+            -0.0002518220110313279,
+            0.005456812494894182,
+            -0.02011241622618092,
+            -0.0009782974741561381,
+            -0.02933296756078644,
+            -0.01171823509606186,
+            0.001162345317872531,
+            0.00892221099991475,
+            -0.006551308022163148,
+            -0.03934356275100932,
+            0.00427623835776202,
+            -0.013111664387776261,
+            -0.008752857644809231,
+            -0.010214546496134218,
+            -0.00518087722852466,
+            0.015240297918667867,
+            -0.012498100033235827,
+            0.01086490299272505,
+            -0.00917777212602538,
+            0.005895055740265625,
+            0.0090554651297836,
+            0.014212841798768113,
+            -0.005544047915369086,
+            -0.0033589999440643385,
+            -0.007924830667307956,
+            0.0022291797354234005,
+            -0.0032976226056924502,
+            0.009615674261248298,
+            -0.0051932155872883,
+            -0.0065636389655770455,
+            0.02426106498596339,
+            0.011445591067574932,
+            0.013312596099532533,
+            0.0055118624272110965,
+            -0.007220620072349783,
+            -0.01916551187613823,
+            -0.003977616240384473,
+            -0.013204506708246246,
+            -0.006315966975649096,
+            0.01434099854765068,
+            0.012588249778961223,
+            0.006357972880218866,
+            0.0007747002406996095,
+            -0.004794439624135275,
+            0.006975593520375694,
+            0.0262335496080837,
+            -0.016481909971520166,
+            -0.0013084662210086592,
+            -0.01369313900978409,
+            0.01921748474212319,
+            0.03333761672336437,
+            -0.0008918419419301244,
+            -0.007699525186523558,
+            0.013757216065348164,
+            0.00819045303047094,
+            -0.007411380297164911,
+            0.02453611080891859,
+            -0.025045462880463647,
+            -0.015989319900023054,
+            -0.011650886878708265,
+            -0.011407253985176972,
+            -0.008386367959227559,
+            0.016331961716602287,
+            -0.00141225438918211,
+            -0.0018505754298268654,
+            0.009557291617446645,
+            0.03357924917013024,
+            0.006916860901290756,
+            0.007030308953738567,
+            -0.002739345982346728,
+            -0.009376522438071562,
+            0.011482637968411055,
+            -0.007259054391879614,
+            -0.04075661435928125,
+            0.004250195164583401,
+            -0.005160505473755424,
+            0.02405623169060588,
+            -0.02197586561375289,
+            -0.00619175761282745,
+            -0.006736129049831973,
+            -0.028200015908139763,
+            -0.005867722104196303,
+            0.02805071895018145,
+            -0.015402198567775478,
+            0.014773294766214671,
+            -0.009502185346558072,
+            -0.0013419858580438616,
+            0.004443749010136947,
+            0.013008783570355656,
+            -0.02289662952366337,
+            0.0013834888740540108,
+            -0.00048549197205067054,
+            0.027969842651517314,
+            0.010437729090975158,
+            -0.0047705656249623744,
+            0.014202218020412936,
+            0.008240861154198411,
+            -0.018637315941614406,
+            0.007314019317343154,
+            -0.005747966765101792,
+            0.014833896739762147,
+            -0.01314847940891154,
+            0.007762100494499689,
+            0.00705026393499352,
+            -0.01278023669088385,
+            0.03016342404000438,
+            -0.015204129985039077,
+            0.007776946006562982,
+            0.009278778571285689,
+            0.009656723862179986,
+            0.01257686993982689,
+            0.004362439504889069,
+            -0.02898418193036562,
+            0.0002144431781454467,
+            -0.03828100004144633,
+            -0.00833748467467171,
+            -0.002611937107022567,
+            -0.002145329666219461,
+            0.007050630951707937,
+            -0.009383645162261629,
+            -0.033976442057557676,
+            0.0006429978367640709,
+            0.01603027598131823,
+            0.017501067413108286,
+            0.0002196997337451293,
+            0.013075379412332973,
+            -0.006197537539554383,
+            -0.021148120787551232,
+            -0.016986448660412497,
+            -0.010207705210051413,
+            -0.02122194744719993,
+            -0.020550470201191853,
+            -0.00908564461857315,
+            0.012090079127547193,
+            -0.009927390005879807,
+            0.013888458967497627,
+            0.029513250508042935,
+            -0.024529175134769883,
+            0.0011144447737101862,
+            0.021977677894839922,
+            -0.00198308986575084,
+            0.014108189311291326,
+            0.001696775006906642,
+            0.014122064251288995,
+            -0.0004166140484018266,
+            -0.010556016685519523,
+            -0.009729362492661684,
+            0.017487929296614076,
+            -0.0026043257634797124,
+            0.016715798948897194,
+            0.012183487338360088,
+            0.003771721868999568,
+            -0.0030887189945795796,
+            0.010667776697461199,
+            -0.026471157050402344,
+            0.014249996582064706,
+            -0.006690961709923224,
+            -0.0181382691440242,
+            -0.008729104191096461,
+            0.003882731567105777,
+            -0.005609384377642956,
+            -0.007794812949125115,
+            0.012967089730551055,
+            -0.00944374111323783,
+            0.02088525981049842,
+            0.0023400121055514958,
+            -0.012196925240774426,
+            0.003648688547842322,
+            -0.03322041209304968,
+            0.0015191765655963997,
+            -0.018402382776013956,
+            -0.024659631472836927,
+            0.03840173429666435,
+            0.011991452465253198,
+            0.006639702644606365,
+            -0.01675838115087676,
+            0.007953102302963526,
+            0.0004257479565099406,
+            0.017484338819046517,
+            0.011068864013925452,
+            0.006466915839538296,
+            0.008364454017976605,
+            -0.023316137291380285,
+            -0.010898421318668592,
+            0.016204569093730564,
+            0.012099473359791148,
+            -0.01017229369980751,
+            0.019431669546223253,
+            0.017955119269367787,
+            0.007300372071724306,
+            -0.006898619924677706,
+            0.01647927363640854,
+            0.021012103522831126,
+            0.010291016618557653,
+            -0.00047955552103474145,
+            0.0008684297441778696,
+            -0.010534247928997246,
+            0.005037930235889528,
+            0.035417626740307205,
+            0.014712595692093886,
+            0.010650306363963653,
+            -0.03626584475133921,
+            0.021764456453493054,
+            -0.011961521673661574,
+            0.009092981192874177,
+            0.006089104887183293,
+            0.017982801101846038,
+            0.006916969397519662,
+            0.014196254958707752,
+            0.011745069030657836,
+            -0.006689305450954901,
+            0.025205280281804708,
+            -0.001601414806100461,
+            -0.011992525142592405,
+            0.003386930324498371,
+            -0.01859193251438437,
+            0.005303261319773131,
+            -0.0041710536965100615,
+            -0.004459393120257307,
+            0.0032461828144552565,
+            -0.0018650956576455287,
+            0.015678225251799887,
+            0.003894142628084471,
+            -0.02355212701026675,
+            -0.0014873755115206368,
+            -0.024284295126348117,
+            0.0027260592623277375,
+            -0.022185571998195344,
+            0.00022883605560765203,
+            -0.005557568353382136,
+            -0.009749382813201403,
+            -0.025385857431333838,
+            -0.009853073898348951,
+            -0.0035843255959309307,
+            0.007661137888845632,
+            -0.010521037433807653,
+            -0.004246698084422667,
+            -0.0019920499321964815,
+            -0.019465113363548715,
+            0.009014255846983305,
+            0.027122533756959888,
+            0.016848221859975445,
+            0.027170883580103264,
+            -0.00019554112591107156,
+            0.013614675849485926,
+            -0.002066822813103284,
+            -0.01603063005944364,
+            0.008824663429777616,
+            0.0075653522381218495,
+            -0.00531780740880296,
+            0.014643348459785112,
+            0.0072634484937344704,
+            -0.012800149015319017,
+            0.009859503660384916,
+            -0.007161712176469609,
+            -0.006437857349742842,
+            -0.009764856421740622,
+            -0.02058868349990357,
+            -0.014866137971603676,
+            0.0012450207572602557,
+            -0.002123265406747006,
+            0.006000473862248991,
+            -0.004054016790850226,
+            0.009584879012937315,
+            -0.00608816807208866,
+            -0.00925801057439376,
+            -0.013962683072792106,
+            -0.008924356118158748,
+            -0.004210064145949253,
+            -0.004381709838058668,
+            -0.011732699976353612,
+            -0.005479523097469261,
+            0.0010680258077037758,
+            -0.0019401640391035755,
+            -0.01397652201595301,
+            0.011650839243603377,
+            0.0058731838291845645,
+            -0.002462743134993977,
+            0.02219008688925881,
+            0.01097316136828334,
+            0.0018083001685932406,
+            0.01366095798812196,
+            0.0003942601242216818,
+            -0.0064063532958756175,
+            -0.03343312148366622,
+            -0.0005083355836126291,
+            -0.02815586367527057,
+            0.025770078162165123,
+            -0.0069114949102666145,
+            -0.010322087105114614,
+            -0.002519047562374146,
+            -0.006063669644021698,
+            0.005873609611264377,
+            -0.002722940652283699,
+            0.0021229525800901114,
+            -0.023673778230768945,
+            0.0028629715379660113,
+            -0.003953653437464966,
+            -0.0036366957271968823,
+            0.007469664864329332,
+            0.0346520690389124,
+            -0.004054282838167587,
+            -0.0034765253405897676,
+            0.005159466707114532,
+            0.011642913651743502,
+            0.027845571220425726,
+            -0.004445196354520585,
+            -0.0032038508043339413,
+            -0.02233838584152591,
+            0.0054112576531374175,
+            -0.00019544838850471866,
+            -0.004733192299951088,
+            0.016664217348411003,
+            -0.016719175418727594,
+            -0.01631658832047864,
+            -0.00545969677717507,
+            -0.014570848053102338,
+            -0.00022620258237138677,
+            0.014454101124666536,
+            0.011056312437637014,
+            -0.004035526363810078,
+            -0.012459378363360557,
+            -0.0022595493558020706,
+            -0.030370952502579162,
+            0.012766010284884204,
+            0.1141689619126009,
+            -0.00766197047210299,
+            -0.00896655658602649,
+            -0.009481112393590025,
+            0.001435269335646347,
+            0.001338755472877713,
+            -0.011739143028767118,
+            0.07166096084053437,
+            -0.0077855767070652395,
+            0.0002139580430342728,
+            -0.011312198831181402,
+            0.001844005266872301,
+            -0.001825412693163496,
+            0.01371932421024605,
+            -0.005025814503687022,
+            -0.018213360991353643,
+            -0.0033612481616769416,
+            -0.02941403830641238,
+            -0.0041137865872933515,
+            -0.005683630256743215,
+            -0.004288198570156578,
+            -0.014982999363445122,
+            0.0067149665791261335,
+            -0.019661973497372313,
+            -0.012688556129040908,
+            0.001124391650175233,
+            -0.0033623453063430534,
+            0.00996944383866679,
+            -0.020496336814603385,
+            -0.0044354321069609536,
+            0.0019875840782527733,
+            0.009706675240563883,
+            0.0011600364428219214,
+            -0.012755178230821602,
+            -0.0026514442584739412,
+            0.004056350166031111,
+            0.01909951327913368,
+            0.014360281326789544,
+            0.003732344102045803,
+            0.008623457695718206,
+            0.014461868085333023,
+            -0.001692149528228451,
+            0.01948035376572433,
+            0.01172550239477637,
+            0.0031966531144076173,
+            0.0004779413449255626,
+            0.0030912562273611517,
+            -0.004597633235071031,
+            0.01582547400214653,
+            -0.014811177637844808,
+            -0.02411202772171917,
+            -0.0071272238837422185,
+            0.018015541137524113,
+            -0.013081287852028074,
+            1.8837689271256874e-05,
+            0.004062803284646432,
+            0.027569726868200254,
+            0.0010564850664255057,
+            -0.01179755720247972,
+            0.0028876134800690334,
+            -0.01638215925131053,
+            0.022944183106912663,
+            0.0026043713064243836,
+            0.03279173899194346,
+            -0.005103903273968361,
+            0.00578213669463383,
+            -0.020744194169889544,
+            -0.007588793365216579,
+            -0.00819862611489168,
+            0.0017401442433921013,
+            0.010993500174794084,
+            0.005411238350013134,
+            0.01620925462604661,
+            -0.00641615069699833,
+            -0.020751730783437995,
+            0.014776878596707538,
+            -0.009552193304486048,
+            -0.03393319554458277,
+            0.008881987295131347,
+            -0.029102297123590998,
+            -0.0045267103602711,
+            0.00368971742585232,
+            -0.014269341759285646,
+            -0.016264809810970175,
+            0.012837302095591106,
+            -0.0011890334180728564,
+            -0.008937992768323412,
+            0.0014444613725248846,
+            -0.017110187133760626,
+            -0.02385668037818897,
+            0.006017405364919834,
+            0.00676670672597817,
+            -0.0077994500447595,
+            0.018312452858615755,
+            -0.027015660037024585,
+            0.006123912473734307,
+            -0.012973676426738582,
+            0.009041974762537394,
+            -0.01580183887345168,
+            0.01163516332546347,
+            0.010887662470499361,
+            -0.001503467664709027,
+            -0.0076234124970223326,
+            0.01231202541338883,
+            0.004852720134894811,
+            0.033760365210093034,
+            0.028658326381800395,
+            0.000995999783627898,
+            -0.0025605367058226737,
+            -0.0020356794893366384,
+            0.011667525096213252,
+            -0.014049700287539817,
+            0.011218246246280464,
+            0.02658113044439717,
+            0.017568631325838108,
+            -0.019133062654008885,
+            0.018050678772708717,
+            -0.010356118256625172,
+            0.012872052420999813,
+            -0.025620154415048252,
+            -0.005135768485044442,
+            0.004936490920616527,
+            0.005163008729256001,
+            0.007425278892536464,
+            0.001410212306439329,
+            0.021891387289347516,
+            0.014999616600766989,
+            0.00027613811156650413,
+            0.00558422822386443,
+            0.01477009526981092,
+            0.010497782599293994,
+            -0.0005567894749233204,
+            -0.00467776506604011,
+            -0.016998035358377343,
+            0.00957271797745408,
+            -0.006065116857898469,
+            0.005138070401846389,
+            0.01698914855850691,
+            -0.01230257809343985,
+            0.011109844802663414,
+            0.007518553660463051,
+            -0.015504773927595823,
+            -0.017169933945839305,
+            0.005552829003043926,
+            -0.02252633790984703,
+            -0.01306722099657118,
+            0.02715689072730398,
+            -0.010513941399990838,
+            -0.02790582732974509,
+            -0.0009847768069205477,
+            0.009797865241108237,
+            -0.002772952073769185,
+            -0.0027968741526000576,
+            0.0037541733651479613,
+            0.010801137722284016,
+            -0.014658156970099418,
+            0.002075415233345957,
+            0.004735264309137368,
+            0.01930804315461151,
+            -0.0014151522239496115,
+            -0.015046118414100617,
+            -0.002288495894227231,
+            0.021592829677937263,
+            0.017174609713578173,
+            0.00265560378681856,
+            -0.016499825201963983,
+            -3.965296928410772e-05,
+            -0.0074142747014567635,
+            0.00021360696629503726,
+            0.00027431174904249525,
+            -0.0056391089282546054,
+            -0.018931681700770225,
+            -0.003888248121485032,
+            0.011181675419027736,
+            -0.00247252920175839,
+            0.01977860312223846,
+            -0.0011527575128223392,
+            -0.015763587433200915,
+            0.001960734212846797,
+            -0.005280860341480037,
+            -0.010372829570478649,
+            0.01351900743232294,
+            -0.009359942374603997,
+            0.006788746813046239,
+            -0.003212015225284119,
+            0.004611143085183367,
+            -0.0003308099698088422,
+            -0.017017265064185255,
+            0.02924142709831441,
+            -0.0018238549856433466,
+            -0.004769274041546295,
+            -0.0014058363976586562,
+            -0.033393331166741945,
+            0.015017165991303838,
+            -0.014574579250749552,
+            0.006745699409061803,
+            0.002705853802154516,
+            0.01272741296341735,
+            0.003253095246601883,
+            -0.008898616461779179,
+            0.0027485606928249326,
+            0.017040360927723885,
+            0.013756402798365262,
+            -0.007292864293090161,
+            0.008694855886257706,
+            -0.0028934389922887676,
+            -0.0037483208383542015,
+            0.01363963339139356,
+            0.004170611039133969,
+            -0.006282657671540625,
+            -0.0009784372731140945
+          ],
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "Mixture-of-Experts (MoE)",
+            "fine-grained expert segmentation",
+            "shared expert isolation"
+          ],
+          "title": "Deepseekmoe: Towards Ultimate Expert Specialization In Mixture-Of-Experts Language Models"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "86d0cb2e-1f6d-4e38-a077-7b7de2fe9e4a",
+        "properties": {
+          "page_content": "3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n",
+          "embedding": [
+            0.010104401037096977,
+            -0.020574966445565224,
+            -0.02167346142232418,
+            -0.031542472541332245,
+            0.03588413819670677,
+            0.03464615345001221,
+            -0.009755672886967659,
+            0.046869076788425446,
+            0.004803731571882963,
+            0.0364421010017395,
+            0.016547156497836113,
+            -0.007223033811897039,
+            0.009546436369419098,
+            -0.0017371026333421469,
+            0.005762734450399876,
+            0.04048734903335571,
+            0.011603932827711105,
+            0.01784616895020008,
+            0.011150586418807507,
+            -0.02915368229150772,
+            0.005383492447435856,
+            -0.01708768494427204,
+            -0.04317255690693855,
+            -0.006194285582751036,
+            0.009546436369419098,
+            0.005871712230145931,
+            0.027340294793248177,
+            0.0036812627222388983,
+            -0.055901139974594116,
+            0.023800702765583992,
+            0.021708333864808083,
+            0.010984940454363823,
+            -0.01007824670523405,
+            -0.010601338930428028,
+            -0.026573093608021736,
+            -0.021969880908727646,
+            0.04634598642587662,
+            0.007685098797082901,
+            -0.006399163510650396,
+            0.042475100606679916,
+            0.03703493997454643,
+            0.012728581205010414,
+            -0.012772172689437866,
+            -0.014175803400576115,
+            -0.003430614247918129,
+            -0.0012979229213669896,
+            -0.004712190479040146,
+            0.0065735275857150555,
+            0.009311044588685036,
+            -0.026172054931521416,
+            -0.001842810888774693,
+            -0.021900134161114693,
+            0.02798544242978096,
+            0.020609840750694275,
+            -0.049379922449588776,
+            0.028072625398635864,
+            -0.004890913609415293,
+            0.06461934745311737,
+            0.047461915761232376,
+            0.014602995477616787,
+            0.03794163465499878,
+            -0.034035876393318176,
+            -0.029275737702846527,
+            -0.003177786245942116,
+            0.03234454616904259,
+            0.007654585409909487,
+            -0.01033979281783104,
+            0.0010679803090170026,
+            0.011717269197106361,
+            -0.005706066265702248,
+            -0.044532597064971924,
+            0.010496720671653748,
+            -0.014925569295883179,
+            0.002881367225199938,
+            0.004034350160509348,
+            -0.020017001777887344,
+            0.03030448593199253,
+            0.0268520750105381,
+            0.014899414964020252,
+            0.010740830563008785,
+            0.010653648525476456,
+            0.03204812481999397,
+            -0.0026961052790284157,
+            0.009555154480040073,
+            0.0364421010017395,
+            -0.029415227472782135,
+            -0.01360040158033371,
+            -0.062178246676921844,
+            0.010540311224758625,
+            0.0028072623535990715,
+            -0.02815980650484562,
+            0.0018340926617383957,
+            -0.006569168530404568,
+            0.0036986989434808493,
+            -0.033094312995672226,
+            0.0727447122335434,
+            0.010653648525476456,
+            0.00023280335881281644,
+            -0.024044813588261604,
+            0.006394804455339909,
+            -0.015762517228722572,
+            0.021969880908727646,
+            0.0006560450419783592,
+            0.004008195362985134,
+            0.0229811929166317,
+            -0.02859571762382984,
+            -0.004690395202487707,
+            0.03815086930990219,
+            0.02148166112601757,
+            0.08676359057426453,
+            0.01007824670523405,
+            -0.017593340948224068,
+            -0.011211613193154335,
+            0.018482597544789314,
+            -0.04278895631432533,
+            -0.017689241096377373,
+            -0.03965040296316147,
+            -0.03215274587273598,
+            0.0114818774163723,
+            -0.025038689374923706,
+            0.01996469311416149,
+            0.02566639892756939,
+            -0.0011344566009938717,
+            -0.025422289967536926,
+            -0.07155904173851013,
+            0.04292844980955124,
+            -0.023696085438132286,
+            0.01780257746577263,
+            0.001439593848772347,
+            -0.0022144243121147156,
+            -0.00422179140150547,
+            0.0037183149252086878,
+            -0.020261110737919807,
+            -0.041882261633872986,
+            0.01856978051364422,
+            0.010479284450411797,
+            0.046834204345941544,
+            0.0633290484547615,
+            -0.021132932975888252,
+            -0.0072448295541107655,
+            0.017375385388731956,
+            -0.022894009947776794,
+            0.003243172774091363,
+            -0.0038142153061926365,
+            -0.029450101777911186,
+            0.013975284993648529,
+            0.028508534654974937,
+            0.08578714728355408,
+            -0.01956365443766117,
+            0.012022406794130802,
+            -0.033286113291978836,
+            0.037662651389837265,
+            -0.009799264371395111,
+            0.023417102172970772,
+            -0.025404853746294975,
+            -0.021708333864808083,
+            -0.025352545082569122,
+            0.010705957189202309,
+            -0.014995315112173557,
+            0.0059632533229887486,
+            -0.005889148451387882,
+            -0.0062945447862148285,
+            -0.018691834062337875,
+            0.010470565408468246,
+            0.007685098797082901,
+            0.047985006123781204,
+            0.029816266149282455,
+            0.005858634598553181,
+            -0.07769665867090225,
+            0.038325235247612,
+            0.017514877021312714,
+            0.036546722054481506,
+            0.01543994341045618,
+            0.033443041145801544,
+            -0.026538221165537834,
+            0.017680522054433823,
+            0.0032475318294018507,
+            0.01094134896993637,
+            -0.03197837993502617,
+            0.044916197657585144,
+            -0.00843050517141819,
+            0.002720080316066742,
+            0.01972058229148388,
+            0.013312701135873795,
+            0.04062684252858162,
+            0.027915697544813156,
+            0.034210242331027985,
+            0.03815086930990219,
+            -0.01332141924649477,
+            0.010453129187226295,
+            0.029380355030298233,
+            0.03515180945396423,
+            0.02796800620853901,
+            0.0027636713348329067,
+            0.0041890982538461685,
+            -0.026590529829263687,
+            -0.006721737328916788,
+            0.009964910335838795,
+            0.007039951626211405,
+            0.016390228644013405,
+            -0.012780890800058842,
+            -0.028124934062361717,
+            0.0220221895724535,
+            -0.028648026287555695,
+            0.030740395188331604,
+            0.008626664988696575,
+            -0.01083673071116209,
+            0.013208082877099514,
+            0.009895164519548416,
+            -0.029833702370524406,
+            0.004243587143719196,
+            -0.006250953767448664,
+            -0.004468080587685108,
+            -0.025021253153681755,
+            -0.01933698169887066,
+            0.00797280017286539,
+            0.0029532923363149166,
+            0.005021686665713787,
+            0.009380790404975414,
+            -0.022545281797647476,
+            0.004284998401999474,
+            0.02127242274582386,
+            0.022876573726534843,
+            -0.018325669690966606,
+            -0.018901070579886436,
+            -0.011438286863267422,
+            -0.01914518140256405,
+            -0.011865478940308094,
+            0.044393107295036316,
+            -0.012711144983768463,
+            -0.009694645181298256,
+            0.020574966445565224,
+            0.006774046458303928,
+            -0.005043482407927513,
+            0.017872322350740433,
+            -0.029484974220395088,
+            -0.010932630859315395,
+            -0.0035592077765613794,
+            0.011072122491896152,
+            0.010243892669677734,
+            -0.00997362844645977,
+            0.01550968922674656,
+            -0.050879452377557755,
+            -0.010932630859315395,
+            0.007881258614361286,
+            0.015082497149705887,
+            -0.0110285310074687,
+            0.02339966595172882,
+            0.012161897495388985,
+            0.016224581748247147,
+            -0.003081885864958167,
+            -0.014829669147729874,
+            0.008221268653869629,
+            0.017384104430675507,
+            0.007881258614361286,
+            0.05712168663740158,
+            0.02338222973048687,
+            -0.03947603702545166,
+            0.006538654677569866,
+            -0.029973194003105164,
+            0.05579652264714241,
+            0.007994594983756542,
+            -0.029833702370524406,
+            0.053215932101011276,
+            -0.023434538394212723,
+            -0.0024193022400140762,
+            0.060853078961372375,
+            -0.0007007258245721459,
+            -0.013548092916607857,
+            0.0013011923292651772,
+            0.02435866929590702,
+            -0.056075502187013626,
+            -0.04899631813168526,
+            -0.02148166112601757,
+            0.02857828140258789,
+            -0.02453303337097168,
+            0.0077156126499176025,
+            -0.015405070967972279,
+            0.04735729843378067,
+            0.036721084266901016,
+            0.018116433173418045,
+            -0.008731283247470856,
+            0.02008674666285515,
+            0.04749678820371628,
+            -0.007977158762514591,
+            -0.004742704331874847,
+            -0.02833417057991028,
+            0.005152460187673569,
+            -0.033216364681720734,
+            -0.009485408663749695,
+            -0.027462350204586983,
+            -0.011133149266242981,
+            -0.02125498652458191,
+            -0.0030317562632262707,
+            0.022336045280098915,
+            0.0041389684192836285,
+            5.1253518904559314e-05,
+            -0.0020389705896377563,
+            0.040522221475839615,
+            -0.024951506406068802,
+            -0.04470696300268173,
+            -0.00825178250670433,
+            0.01704409345984459,
+            0.0038556267973035574,
+            -0.05475033447146416,
+            -0.006250953767448664,
+            -0.0029707287903875113,
+            0.01304243691265583,
+            -0.023539157584309578,
+            -0.00863538309931755,
+            0.016573309898376465,
+            0.002870469354093075,
+            0.009807982482016087,
+            -0.02800287865102291,
+            -0.014629150740802288,
+            0.007406116463243961,
+            0.01841285265982151,
+            0.02817724272608757,
+            -0.0555872842669487,
+            -0.03225736320018768,
+            0.02067958563566208,
+            -0.009040780365467072,
+            -0.06409624963998795,
+            0.007833308540284634,
+            -0.001987751107662916,
+            0.01799437776207924,
+            0.016948193311691284,
+            0.04160328209400177,
+            -0.019581090658903122,
+            -0.022283734753727913,
+            -0.016599465161561966,
+            -0.0364072285592556,
+            -0.028700334951281548,
+            -0.009712082333862782,
+            -0.008299732580780983,
+            -0.028857262805104256,
+            -0.0017839629435911775,
+            0.01342603750526905,
+            -0.03759290650486946,
+            0.006176849361509085,
+            -0.021603714674711227,
+            0.0028486738447099924,
+            -0.03703493997454643,
+            0.007842026650905609,
+            -0.0030339357908815145,
+            -0.01613740064203739,
+            -0.02125498652458191,
+            -0.01561430748552084,
+            0.014725050888955593,
+            -0.0002036518562817946,
+            0.0017142173601314425,
+            0.0229637548327446,
+            0.007484580390155315,
+            0.009328480809926987,
+            -0.008535124361515045,
+            -0.029798829928040504,
+            -0.013382446952164173,
+            0.006586604751646519,
+            0.018534906208515167,
+            0.019999565556645393,
+            0.005501188337802887,
+            -0.04958915710449219,
+            -0.03584926575422287,
+            0.021621152758598328,
+            -0.05890020355582237,
+            0.006394804455339909,
+            -0.0017425514524802566,
+            -0.008622306399047375,
+            0.0033150978852063417,
+            -0.005832480266690254,
+            -0.026294110342860222,
+            0.015570716932415962,
+            0.0086135882884264,
+            0.012275234796106815,
+            0.007488939445465803,
+            0.015980472788214684,
+            0.02852597087621689,
+            -0.02451559714972973,
+            -0.05788889154791832,
+            0.03427998721599579,
+            -0.019075434654951096,
+            0.014210676774382591,
+            0.0440792515873909,
+            0.005475033540278673,
+            -0.006551732309162617,
+            0.025003815069794655,
+            0.013399883173406124,
+            -0.010034655220806599,
+            -0.03407074883580208,
+            0.04376539587974548,
+            -0.003430614247918129,
+            -0.07121030986309052,
+            0.005479392595589161,
+            0.0014810053398832679,
+            0.013103464618325233,
+            0.015283015556633472,
+            -0.003430614247918129,
+            -0.023643774911761284,
+            -0.01956365443766117,
+            -0.014088621363043785,
+            -0.018168741837143898,
+            0.029484974220395088,
+            0.003539591794833541,
+            0.005004250444471836,
+            -0.0006091846735216677,
+            -0.0014341449132189155,
+            0.004677317570894957,
+            0.014027593657374382,
+            0.04080120474100113,
+            -0.0364769771695137,
+            0.0025958458427339792,
+            -0.04275408387184143,
+            -0.03874370828270912,
+            -0.0046380856074392796,
+            0.018360542133450508,
+            0.026485910639166832,
+            -0.025910509750247,
+            0.02320786565542221,
+            0.0100608104839921,
+            -0.015675334259867668,
+            -0.005008609499782324,
+            0.028090061619877815,
+            0.025369981303811073,
+            0.01285063661634922,
+            -0.004533467348664999,
+            0.014690177515149117,
+            -0.012623962946236134,
+            0.0026045641861855984,
+            0.01574508100748062,
+            -0.00010305192699888721,
+            0.03490769863128662,
+            0.001937621389515698,
+            0.018918508663773537,
+            -0.04693882167339325,
+            -0.02699156664311886,
+            -0.027689022943377495,
+            -0.0172620490193367,
+            0.03874370828270912,
+            -0.006503781769424677,
+            -0.012728581205010414,
+            0.020261110737919807,
+            0.004282819107174873,
+            0.0017044093692675233,
+            -0.04066171497106552,
+            -0.028630590066313744,
+            -0.022283734753727913,
+            -0.044916197657585144,
+            -0.005618884228169918,
+            -0.02183038927614689,
+            -0.019057998433709145,
+            -0.05342517048120499,
+            0.0010527233825996518,
+            -0.07811512798070908,
+            -0.005370415281504393,
+            0.04048734903335571,
+            0.004686036147177219,
+            0.009424380958080292,
+            -0.028682898730039597,
+            0.01025261078029871,
+            -0.036337483674287796,
+            -0.02568383514881134,
+            0.017785141244530678,
+            0.04387001320719719,
+            -0.011813169345259666,
+            0.0019997386261820793,
+            0.019057998433709145,
+            -0.05056559666991234,
+            -0.005370415281504393,
+            -0.0031799657735973597,
+            0.007641507778316736,
+            0.012920381501317024,
+            -0.016974348574876785,
+            0.05457597225904465,
+            0.011054685339331627,
+            0.02223142609000206,
+            0.02606743760406971,
+            -0.00020024630066473037,
+            -0.038046251982450485,
+            0.017584621906280518,
+            0.008535124361515045,
+            0.02796800620853901,
+            0.0015681873774155974,
+            0.0073014977388083935,
+            -0.019807765260338783,
+            0.008979752659797668,
+            -0.009485408663749695,
+            0.01646869257092476,
+            0.018866198137402534,
+            0.02645103819668293,
+            -0.008644101209938526,
+            -0.002175192581489682,
+            0.0033434322103857994,
+            -0.0034785643219947815,
+            0.028700334951281548,
+            0.0076371487230062485,
+            -0.038848329335451126,
+            0.006813278421759605,
+            -0.002189359627664089,
+            0.023120682686567307,
+            0.030374230816960335,
+            -0.04108018800616264,
+            -0.02338222973048687,
+            -0.002796364715322852,
+            0.016974348574876785,
+            0.011019812896847725,
+            -0.020662149414420128,
+            -0.0031668883748352528,
+            0.044602345675230026,
+            -0.010479284450411797,
+            -0.032484035938978195,
+            0.01293781865388155,
+            -0.0018504393519833684,
+            0.0059806895442306995,
+            0.0024476363323628902,
+            0.012772172689437866,
+            0.013469628989696503,
+            -0.019598528742790222,
+            0.04087095335125923,
+            -0.004965018481016159,
+            0.013208082877099514,
+            -0.012031124904751778,
+            -0.010052092373371124,
+            -0.015195833519101143,
+            0.03926680237054825,
+            -0.0047906544059515,
+            0.02013905718922615,
+            0.024864325299859047,
+            0.006338136270642281,
+            0.05216974765062332,
+            -0.04125455394387245,
+            -0.03724417835474014,
+            0.005971971433609724,
+            0.0016215863870456815,
+            0.005871712230145931,
+            -0.011272640898823738,
+            -0.030932195484638214,
+            -0.012074715457856655,
+            0.004768858663737774,
+            -0.015535843558609486,
+            -0.015684053301811218,
+            0.00019043832435272634,
+            0.022144244983792305,
+            0.0011257383739575744,
+            0.005475033540278673,
+            0.00287700816988945,
+            0.014585559256374836,
+            0.03581439331173897,
+            -0.02589307352900505,
+            0.010758266784250736,
+            0.03147272393107414,
+            -0.009136680513620377,
+            -0.026764893904328346,
+            -0.027305422350764275,
+            0.015073779039084911,
+            -0.018918508663773537,
+            -0.028874699026346207,
+            3.807404209510423e-05,
+            -0.0034872826654464006,
+            0.008391273207962513,
+            -0.03584926575422287,
+            -0.0076763806864619255,
+            0.0021283321548253298,
+            -0.00196377607062459,
+            0.0007078094058670104,
+            -0.022318609058856964,
+            -0.013591683469712734,
+            0.01741897687315941,
+            0.024219177663326263,
+            0.005727861542254686,
+            0.03028704971075058,
+            -0.010243892669677734,
+            -0.00016768925706855953,
+            0.006525577511638403,
+            0.01589328981935978,
+            0.023050937801599503,
+            0.023312482982873917,
+            0.026573093608021736,
+            0.0009344827267341316,
+            -0.012327543459832668,
+            0.004252305254340172,
+            0.032484035938978195,
+            -0.016058936715126038,
+            -0.0030993223190307617,
+            0.014812232926487923,
+            0.03274558484554291,
+            0.010627493262290955,
+            -0.012188052758574486,
+            -0.02528279833495617,
+            0.015431225299835205,
+            -0.022283734753727913,
+            0.012510626576840878,
+            0.04777577146887779,
+            0.004393976181745529,
+            -0.029240863397717476,
+            0.02430635876953602,
+            -0.009546436369419098,
+            -0.024044813588261604,
+            0.027165930718183517,
+            0.003757546888664365,
+            -0.03271070867776871,
+            -6.98818766977638e-05,
+            -0.00018321855168323964,
+            0.014794795773923397,
+            -0.026015127077698708,
+            -0.015579435043036938,
+            0.02322530187666416,
+            0.009293608367443085,
+            0.0032627887558192015,
+            -0.021603714674711227,
+            0.001829733606427908,
+            0.015344043262302876,
+            0.0018177460879087448,
+            0.009424380958080292,
+            0.009145398624241352,
+            0.01505634281784296,
+            0.006939692422747612,
+            -0.013190646655857563,
+            -0.0032344546634703875,
+            -0.02720080316066742,
+            -0.006098385434597731,
+            0.0652819275856018,
+            -0.007292779628187418,
+            0.03686057776212692,
+            0.0015213270671665668,
+            0.012868072837591171,
+            0.025788454338908195,
+            -0.025160742923617363,
+            -0.02437610551714897,
+            -0.005047841463238001,
+            0.006961487699300051,
+            -0.04292844980955124,
+            -0.013269110582768917,
+            -0.023818138986825943,
+            -0.030217302963137627,
+            0.01933698169887066,
+            0.0005187332862988114,
+            0.0019267236348241568,
+            0.024829450994729996,
+            0.0051742554642260075,
+            -0.015038905665278435,
+            -0.017009221017360687,
+            0.0383601076900959,
+            0.016625620424747467,
+            0.028229551389813423,
+            -0.015989189967513084,
+            0.01625945419073105,
+            -0.008099214173853397,
+            0.003171247662976384,
+            0.03236198052763939,
+            -0.0026982848066836596,
+            -0.0030186788644641638,
+            0.009197707287967205,
+            0.01881388947367668,
+            0.040731459856033325,
+            -0.01745384931564331,
+            -0.030827578157186508,
+            -0.008914366364479065,
+            -0.014236831106245518,
+            0.021969880908727646,
+            -0.0009491947130300105,
+            -0.00041084547410719097,
+            -0.003934090491384268,
+            0.04352128505706787,
+            -0.029572155326604843,
+            0.014428631402552128,
+            -0.024445850402116776,
+            -0.039022691547870636,
+            0.017706677317619324,
+            -0.006102744489908218,
+            0.032693274319171906,
+            0.006883023772388697,
+            -0.007833308540284634,
+            -0.032867636531591415,
+            0.0016749853966757655,
+            0.0086179468780756,
+            -0.03194350749254227,
+            0.020418038591742516,
+            0.02472483366727829,
+            -0.00651250034570694,
+            0.0033063797745853662,
+            -0.016782546415925026,
+            0.008142804726958275,
+            -0.013844511471688747,
+            -0.022946318611502647,
+            0.022737082093954086,
+            -0.004298075567930937,
+            -0.028055187314748764,
+            0.006686864420771599,
+            -0.022458098828792572,
+            0.004707831423729658,
+            -0.0016771649243310094,
+            -0.00017014124023262411,
+            -0.00797280017286539,
+            0.010461847297847271,
+            -0.007571762427687645,
+            -0.01073211245238781,
+            -0.022353481501340866,
+            0.009991064667701721,
+            0.025631526485085487,
+            0.0017588981427252293,
+            -0.004672958515584469,
+            0.006848151329904795,
+            -0.03658159449696541,
+            0.00574093870818615,
+            0.02413199469447136,
+            -0.012205488979816437,
+            -0.008713847026228905,
+            -0.018255924805998802,
+            -0.029380355030298233,
+            -0.002366993110626936,
+            0.01130751334130764,
+            0.03584926575422287,
+            0.025788454338908195,
+            0.030409103259444237,
+            0.03391382470726967,
+            0.010810576379299164,
+            -0.00398204056546092,
+            0.009127962402999401,
+            -0.029031626880168915,
+            0.04006887599825859,
+            0.0040823002345860004,
+            0.02608487382531166,
+            0.016172273084521294,
+            -0.0459623858332634,
+            -0.00968592707067728,
+            0.014681459404528141,
+            0.027898261323571205,
+            0.044148996472358704,
+            0.025038689374923706,
+            -0.0010232995264232159,
+            0.027654150500893593,
+            -0.022196553647518158,
+            -0.02878751792013645,
+            0.01956365443766117,
+            -0.005235282704234123,
+            -0.00918898917734623,
+            -0.0013131798477843404,
+            0.021533969789743423,
+            -0.011385977268218994,
+            0.016058936715126038,
+            0.03415793180465698,
+            -0.008461019024252892,
+            0.012135743163526058,
+            0.0005590550135821104,
+            -0.008286654949188232,
+            0.026764893904328346,
+            -0.007214315701276064,
+            0.03485538810491562,
+            -0.01650356501340866,
+            -0.0018656961619853973,
+            0.000780279457103461,
+            -0.010766984894871712,
+            0.028020314872264862,
+            0.007270984351634979,
+            -0.06172490119934082,
+            0.00873564276844263,
+            0.011063403449952602,
+            0.013835793361067772,
+            -0.001278307056054473,
+            -0.0033957413397729397,
+            -0.025788454338908195,
+            0.019040562212467194,
+            0.008129727095365524,
+            0.02472483366727829,
+            0.010679802857339382,
+            0.019493909552693367,
+            -0.016643056645989418,
+            0.016599465161561966,
+            -0.0026590528432279825,
+            -0.02854340709745884,
+            0.007266624830663204,
+            0.013181927613914013,
+            0.0402083694934845,
+            -0.026538221165537834,
+            -0.002582768676802516,
+            0.010889039374887943,
+            -0.006970205809921026,
+            -0.011194176971912384,
+            -0.030443977564573288,
+            -0.0010756086558103561,
+            -0.028421353548765182,
+            0.02416686899960041,
+            -0.005139382556080818,
+            0.019406726583838463,
+            -0.027357731014490128,
+            0.0063337767496705055,
+            0.00043264098349027336,
+            -0.027654150500893593,
+            0.006390445400029421,
+            0.01782001368701458,
+            0.0031189383007586002,
+            -0.031176306307315826,
+            -0.006006844341754913,
+            0.023242738097906113,
+            0.03619799390435219,
+            0.006713018752634525,
+            -0.006848151329904795,
+            0.021533969789743423,
+            -0.015579435043036938,
+            -0.006150694563984871,
+            0.036546722054481506,
+            -0.025910509750247,
+            0.012231643311679363,
+            0.03759290650486946,
+            -0.03543078899383545,
+            -0.03196094557642937,
+            -0.0004021272761747241,
+            -0.04142891615629196,
+            0.001680434332229197,
+            -0.002975087845697999,
+            0.00835204217582941,
+            -0.0009715351043269038,
+            -0.027113622054457664,
+            0.024812014773488045,
+            -0.0003577189054340124,
+            0.013373728841543198,
+            -0.006848151329904795,
+            0.0021065366454422474,
+            -0.025143306702375412,
+            -0.01862208917737007,
+            -0.010793139226734638,
+            0.00015651904686819762,
+            0.0023059656377881765,
+            -0.008112290874123573,
+            -0.03944116458296776,
+            -0.020976005122065544,
+            0.007257906720042229,
+            0.00045770584256388247,
+            0.03584926575422287,
+            0.018482597544789314,
+            0.012188052758574486,
+            -0.02165602520108223,
+            -0.01898825354874134,
+            0.04756653308868408,
+            -0.026416165754199028,
+            0.0402432419359684,
+            -0.012135743163526058,
+            -0.017488721758127213,
+            0.002946753753349185,
+            0.02625923790037632,
+            -0.020923694595694542,
+            0.016817420721054077,
+            -0.01561430748552084,
+            0.005719143431633711,
+            -0.01409733947366476,
+            -0.004755781497806311,
+            0.009642336517572403,
+            -0.0017796038882806897,
+            0.018255924805998802,
+            0.0009072383400052786,
+            -0.0031123997177928686,
+            0.03183889016509056,
+            0.013652711175382137,
+            0.007763562723994255,
+            -0.016773829236626625,
+            -0.012222925201058388,
+            -0.010766984894871712,
+            0.007087901700288057,
+            -0.015073779039084911,
+            -0.007066106423735619,
+            -0.029659338295459747,
+            0.001694601378403604,
+            0.0007154378108680248,
+            -0.015256861224770546,
+            0.01235369872301817,
+            0.02706131339073181,
+            -0.013295264914631844,
+            -0.03157734498381615,
+            -0.006229158490896225,
+            -0.03944116458296776,
+            -0.021045750007033348,
+            0.019511345773935318,
+            -0.0005190056981518865,
+            0.009127962402999401,
+            0.004250125493854284,
+            0.00043509298120625317,
+            -0.022946318611502647,
+            -0.017976941540837288,
+            0.00714892940595746,
+            -0.009572590701282024,
+            0.00107724335975945,
+            0.00131862866692245,
+            0.01822105050086975,
+            -0.010156710632145405,
+            0.02800287865102291,
+            -0.017497440800070763,
+            0.015265579335391521,
+            -0.029851138591766357,
+            -0.02760184183716774,
+            -0.03199581801891327,
+            0.01304243691265583,
+            -0.014611713588237762,
+            -0.008195114322006702,
+            -0.010113119147717953,
+            0.0004999346565455198,
+            0.007567403372377157,
+            -0.009441817179322243,
+            0.00842614658176899,
+            0.011603932827711105,
+            0.013303983025252819,
+            0.007406116463243961,
+            -0.03244916349649429,
+            -0.02029598504304886,
+            -0.02397506684064865,
+            0.043974634259939194,
+            -0.030513722449541092,
+            0.01703537441790104,
+            -0.017392821609973907,
+            0.005823761690407991,
+            -0.0363723561167717,
+            -0.019075434654951096,
+            0.008953598327934742,
+            0.020749330520629883,
+            -0.025910509750247,
+            0.02741004154086113,
+            0.02934548258781433,
+            0.03013012185692787,
+            -0.036546722054481506,
+            0.025352545082569122,
+            -0.010505438782274723,
+            -0.02512587048113346,
+            0.027357731014490128,
+            0.03731392323970795,
+            0.03679082915186882,
+            -0.013460910879075527,
+            -0.012589089572429657,
+            -0.014690177515149117,
+            0.012318825349211693,
+            -0.0021555765997618437,
+            0.005801966413855553,
+            -0.037069812417030334,
+            0.01862208917737007,
+            0.009895164519548416,
+            0.005121946334838867,
+            0.03773239627480507,
+            -0.01898825354874134,
+            0.02054009400308132,
+            0.007471502758562565,
+            -0.01839541457593441,
+            0.006974565330892801,
+            0.000942111189942807,
+            -0.0004040343628730625,
+            0.003602798795327544,
+            0.017584621906280518,
+            0.005553497467190027,
+            -0.054541099816560745,
+            0.006582245696336031,
+            0.027898261323571205,
+            0.020418038591742516,
+            0.005235282704234123,
+            -0.026398729532957077,
+            0.014629150740802288,
+            0.015823544934391975,
+            0.002135960618034005,
+            0.007035592570900917,
+            0.016538437455892563,
+            -0.011080840602517128,
+            0.00042283302173018456,
+            0.0030339357908815145,
+            0.03720930591225624,
+            0.01475120522081852,
+            -0.013190646655857563,
+            -0.025056125596165657,
+            -0.016364073380827904,
+            0.0066519915126264095,
+            -0.005915303248912096,
+            -0.0004182014672551304,
+            -0.037488289177417755,
+            -0.03598875552415848,
+            0.013521937653422356,
+            -0.01818617805838585,
+            -0.03815086930990219,
+            -0.0042741005308926105,
+            -0.03222249075770378,
+            0.005200410261750221,
+            -0.014890696853399277,
+            -0.019319545477628708,
+            -0.03926680237054825,
+            -0.0038970382884144783,
+            0.00012893410166725516,
+            0.04045247659087181,
+            0.00037842465098947287,
+            0.03476820886135101,
+            -0.007253547664731741,
+            -0.0013676686212420464,
+            -0.004143327474594116,
+            0.022876573726534843,
+            -0.03147272393107414,
+            -0.04529980197548866,
+            0.026573093608021736,
+            0.005832480266690254,
+            0.016024062409996986,
+            0.035744644701480865,
+            -0.013870666734874249,
+            -0.010200301185250282,
+            0.024864325299859047,
+            -0.016747673973441124,
+            -0.010984940454363823,
+            -0.0022231426555663347,
+            0.0025980256032198668,
+            -0.012231643311679363,
+            -0.013521937653422356,
+            -0.011656241491436958,
+            0.05157690867781639,
+            0.005575293209403753,
+            0.027845950797200203,
+            0.04038273170590401,
+            0.0010271136416122317,
+            -0.04059197008609772,
+            0.0032889433205127716,
+            -0.0091018071398139,
+            -0.009642336517572403,
+            -0.004625008441507816,
+            0.033024564385414124,
+            0.001587803359143436,
+            0.017872322350740433,
+            -0.0042174323461949825,
+            -0.016416382044553757,
+            -0.020418038591742516,
+            -0.012318825349211693,
+            -0.00761535344645381,
+            -0.004145507235080004,
+            -0.03373945876955986,
+            -0.01256293524056673,
+            0.031716834753751755,
+            0.025335106998682022,
+            -0.022388353943824768,
+            -0.004925786517560482,
+            -0.01782001368701458,
+            -0.007672021631151438,
+            0.010479284450411797,
+            -0.002959831152111292,
+            -0.00416948227211833,
+            0.009250016883015633,
+            0.018133869394659996,
+            0.01007824670523405,
+            0.00766766257584095,
+            -0.05715656280517578,
+            -0.0019398009171709418,
+            0.011351104825735092,
+            0.010400820523500443,
+            -0.003968963399529457,
+            0.014541968703269958,
+            -0.012580371461808681,
+            -0.01265011727809906,
+            -0.01975545659661293,
+            0.0326235294342041,
+            0.006355572491884232,
+            0.030426539480686188,
+            -0.0005541510181501508,
+            0.009075652807950974,
+            0.02149909734725952,
+            0.004101916216313839,
+            0.012807045131921768,
+            0.00199102028273046,
+            0.019807765260338783,
+            0.00986029114574194,
+            -0.03142041712999344,
+            0.012414725497364998,
+            0.0010429153917357326,
+            -0.0024519956205040216,
+            -0.017872322350740433,
+            0.006922255735844374,
+            0.0023757112212479115,
+            -0.013949130661785603,
+            0.0041215321980416775,
+            -0.022370917722582817,
+            0.005954534746706486,
+            0.031089123338460922,
+            0.033233802765607834,
+            0.009206426329910755,
+            -0.015169679187238216,
+            -0.010810576379299164,
+            0.002057496691122651,
+            0.02008674666285515,
+            0.025404853746294975,
+            0.025003815069794655,
+            -0.0268172025680542,
+            -0.014393758960068226,
+            -0.003729212796315551,
+            -0.001253242138773203,
+            -0.010932630859315395,
+            0.03046141378581524,
+            0.02833417057991028,
+            -0.0008914365898817778,
+            0.0028835467528551817,
+            -0.0051916916854679585,
+            0.01523070689290762,
+            0.01898825354874134,
+            -0.017340512946248055,
+            0.0017665266059339046,
+            0.022283734753727913,
+            -0.020191365852952003,
+            0.022946318611502647,
+            -0.014672741293907166,
+            -0.011429568752646446,
+            0.02013905718922615,
+            0.09366840869188309,
+            -0.005248360335826874,
+            0.01120289508253336,
+            -0.028700334951281548,
+            -0.019214926287531853,
+            -0.008474096655845642,
+            -0.0035330532118678093,
+            0.016765110194683075,
+            0.026503346860408783,
+            0.009511562995612621,
+            0.021359605714678764,
+            0.015466097742319107,
+            -0.01666049286723137,
+            0.014411195181310177,
+            -0.009694645181298256,
+            0.016198428347706795,
+            -0.01362655684351921,
+            -0.004267562180757523,
+            0.010069528594613075,
+            -0.001332795829512179,
+            0.009816700592637062,
+            0.018674397841095924,
+            0.016799984499812126,
+            0.013190646655857563,
+            -0.024864325299859047,
+            0.00949412677437067,
+            0.002284170128405094,
+            -0.007763562723994255,
+            -0.02013905718922615,
+            -0.02322530187666416,
+            0.004999891389161348,
+            0.004428848624229431,
+            0.005296310409903526,
+            0.010331074707210064,
+            -0.0032889433205127716,
+            0.03811599686741829,
+            -0.03319893032312393,
+            0.01496044173836708,
+            0.006917896680533886,
+            -0.013103464618325233,
+            0.011935224756598473,
+            0.016198428347706795,
+            0.004049607086926699,
+            0.012249079532921314,
+            -0.011569059453904629,
+            0.014245549216866493,
+            0.02052265778183937,
+            -0.029781393706798553,
+            -0.00266123260371387,
+            -0.016372792422771454,
+            -0.008474096655845642,
+            0.0026677711866796017,
+            -0.016172273084521294,
+            -0.006006844341754913,
+            -0.00546195637434721,
+            0.01169983297586441,
+            0.0024825092405080795,
+            -0.0325886569917202,
+            0.005060918629169464,
+            0.00261110276915133,
+            0.005928380414843559,
+            0.018168741837143898,
+            0.01632048189640045,
+            0.01765436865389347,
+            -0.011525468900799751,
+            -0.01975545659661293,
+            0.006285826675593853,
+            0.0045029534958302975,
+            -0.01073211245238781,
+            0.0023473771288990974,
+            0.0024912275839596987,
+            0.01588457264006138,
+            -0.011185458861291409,
+            -0.005823761690407991,
+            -0.004141148179769516,
+            0.011664960533380508,
+            0.0031516316812485456,
+            -0.014541968703269958,
+            0.003940629307180643,
+            -0.021150369197130203,
+            0.020644713193178177,
+            0.011577778495848179,
+            -0.0019932000432163477,
+            -0.02088882215321064,
+            -0.02873520739376545,
+            0.01332141924649477,
+            -0.020836513489484787,
+            0.0031952227000147104,
+            0.015771236270666122,
+            -0.011673678644001484,
+            0.013260391540825367,
+            -0.018255924805998802,
+            -0.018866198137402534,
+            -0.0114818774163723,
+            -0.00473834527656436,
+            -0.005226564593613148,
+            -0.007576121482998133,
+            0.007502016611397266,
+            -0.01823848858475685,
+            -0.0306706503033638,
+            -0.033634841442108154,
+            0.019023125991225243,
+            0.01670408435165882,
+            -0.009467972442507744,
+            -0.015823544934391975,
+            -0.004642445128411055,
+            0.02319042943418026,
+            -0.0034415118861943483,
+            0.0007110786973498762,
+            -0.0026830281130969524,
+            0.013391165062785149,
+            -0.007610994391143322,
+            0.00016441992192994803,
+            -0.02284170128405094,
+            0.00044217653339728713,
+            0.0011725988006219268,
+            0.013391165062785149,
+            -0.027357731014490128,
+            -0.003380484413355589,
+            0.0003002332232426852,
+            -0.018098996952176094,
+            -0.004476799163967371,
+            0.04254484549164772,
+            -0.013417319394648075,
+            0.008979752659797668,
+            0.021551406010985374,
+            0.010104401037096977,
+            -0.00977310910820961,
+            0.010095682926476002,
+            0.007850744761526585,
+            -0.0028857262805104256,
+            0.026538221165537834,
+            0.022911446169018745,
+            0.01914518140256405,
+            0.01670408435165882,
+            -0.022161681205034256,
+            0.012528062798082829,
+            0.021969880908727646,
+            0.03752316161990166,
+            -0.017331793904304504,
+            -0.021952442824840546,
+            0.019860073924064636,
+            0.04676445946097374,
+            0.0021032672375440598,
+            -0.0006200824282132089,
+            -0.028648026287555695,
+            -0.017201021313667297,
+            0.024689961224794388,
+            0.020557530224323273,
+            0.016224581748247147,
+            -0.010121837258338928,
+            -0.031507596373558044,
+            0.014280421659350395,
+            -0.00309278373606503,
+            0.006865587551146746,
+            -0.01996469311416149,
+            0.01688716560602188,
+            -0.015004033222794533,
+            0.04257972165942192,
+            -0.012658835388720036,
+            0.004829886369407177,
+            0.017148712649941444,
+            0.011150586418807507,
+            -0.022318609058856964,
+            0.0009263094398193061,
+            -0.01666049286723137,
+            0.027933133766055107,
+            -0.018883634358644485,
+            0.003618055721744895,
+            0.01914518140256405,
+            -0.016608182340860367,
+            -0.014533249661326408,
+            0.004191277548670769,
+            0.019459037110209465,
+            0.006107103545218706,
+            -0.025247925892472267,
+            0.018098996952176094,
+            0.0019190951716154814,
+            0.009171552956104279,
+            -0.009930036962032318,
+            0.023870449513196945,
+            -0.027636714279651642,
+            -0.015387633815407753,
+            0.025003815069794655,
+            0.0041607641614973545,
+            0.013565529137849808,
+            -0.026311546564102173,
+            0.009380790404975414,
+            -0.029049063101410866,
+            0.02336479350924492,
+            0.0032584297005087137,
+            0.014280421659350395,
+            0.0015147883677855134,
+            0.002866110298782587,
+            -0.0015115190763026476,
+            0.014446067623794079,
+            -0.000550881668459624,
+            -0.022527845576405525,
+            -0.01569277234375477,
+            -0.03696519508957863,
+            -0.022737082093954086,
+            0.023818138986825943,
+            -0.002715721260756254,
+            0.003528693923726678,
+            0.02430635876953602,
+            -0.027462350204586983,
+            -0.01435888558626175,
+            0.009572590701282024,
+            0.02549203485250473,
+            -0.043625906109809875,
+            4.093470488442108e-05,
+            0.0013829254312440753,
+            -0.019232362508773804,
+            0.010627493262290955,
+            -0.005440160632133484,
+            0.015500971116125584,
+            0.0018776836805045605,
+            -0.037662651389837265,
+            0.011586496606469154,
+            0.03762777894735336,
+            -0.017924632877111435,
+            0.0028225192800164223,
+            0.017706677317619324,
+            0.012493189424276352,
+            -0.0012390750925987959,
+            -0.0017033196054399014,
+            0.01877901703119278,
+            -0.025945382192730904,
+            0.026102310046553612,
+            0.017026657238602638,
+            -0.014881978742778301,
+            0.007842026650905609,
+            -0.008753078989684582,
+            0.020731894299387932,
+            -0.0013208083109930158,
+            -0.013155773282051086,
+            0.017375385388731956,
+            0.002195898210629821,
+            0.01349578332155943,
+            -0.001912556472234428,
+            0.004304614383727312,
+            0.03107168711721897,
+            -0.01609380915760994,
+            0.02280682697892189,
+            0.01994725689291954,
+            3.6405323044164106e-05,
+            0.015405070967972279,
+            -0.019877510145306587,
+            -0.03312918543815613,
+            0.02221398986876011,
+            -0.013591683469712734,
+            -0.007310215849429369,
+            0.023347357288002968,
+            -0.010453129187226295,
+            -0.006250953767448664,
+            -0.004282819107174873,
+            0.004077941179275513,
+            0.012406007386744022,
+            -0.0230160653591156,
+            -0.015631744638085365,
+            0.022318609058856964,
+            0.017131276428699493,
+            0.022179117426276207,
+            0.023800702765583992,
+            -0.015378915704786777,
+            0.03182145208120346,
+            0.009223862551152706,
+            0.0065735275857150555,
+            0.0076371487230062485,
+            0.008395632728934288,
+            -0.005871712230145931,
+            0.01933698169887066,
+            0.014114775694906712,
+            0.007336370646953583,
+            -0.003565746359527111,
+            -0.02029598504304886,
+            -0.011447004973888397,
+            -0.027532095089554787,
+            -0.00045825072447769344,
+            -0.013914257287979126,
+            0.00704866973683238,
+            0.07797563821077347,
+            -0.006682505365461111,
+            0.0018144766800105572,
+            -0.007868180982768536,
+            0.000739412906114012,
+            0.0016771649243310094,
+            -0.005209128372371197,
+            0.016538437455892563,
+            -0.020662149414420128,
+            -0.020749330520629883,
+            -0.006163771729916334,
+            -0.010967503301799297,
+            0.013460910879075527,
+            0.01741897687315941,
+            0.0030034221708774567,
+            -0.02277195453643799,
+            0.011141867376863956,
+            -0.016355354338884354,
+            -0.011106994934380054,
+            0.010348510928452015,
+            -0.007231752388179302,
+            -0.011656241491436958,
+            0.03295481950044632,
+            -0.009206426329910755,
+            -0.03147272393107414,
+            0.015640461817383766,
+            -0.0024541751481592655,
+            -0.005976330488920212,
+            0.00470347236841917,
+            0.00853076484054327,
+            -0.004986814223229885,
+            0.0005544234300032258,
+            0.013504501432180405,
+            -0.002068394562229514,
+            0.005414006300270557,
+            0.005318105686455965,
+            -0.013565529137849808,
+            0.017296921461820602,
+            0.022527845576405525,
+            -0.033477913588285446,
+            -0.004755781497806311,
+            -0.013940411619842052,
+            0.010479284450411797,
+            0.018709270283579826,
+            0.015535843558609486,
+            -0.022580154240131378,
+            0.0025871277321130037,
+            -0.009154116734862328,
+            -0.012702426873147488,
+            -0.02741004154086113,
+            -0.018255924805998802,
+            0.00723611144348979,
+            -0.00851768720895052,
+            0.018656961619853973,
+            0.007907412946224213,
+            -0.005745298229157925,
+            0.005012968555092812,
+            -0.005178614519536495,
+            0.015518407337367535,
+            -0.0326235294342041,
+            0.010461847297847271,
+            -0.009337198920547962,
+            0.009581308811903,
+            0.01055774837732315,
+            -0.022388353943824768,
+            -0.014759923331439495,
+            -0.023417102172970772,
+            0.016555873677134514,
+            -0.010086964815855026,
+            -0.0020934592466801405,
+            0.02008674666285515,
+            0.0009088730439543724,
+            0.015413789078593254,
+            -0.001491903094574809,
+            0.023242738097906113,
+            0.014140930958092213,
+            0.016590746119618416,
+            -0.01215317938476801,
+            -0.001640112604945898,
+            -0.011124431155622005,
+            0.012719863094389439,
+            -0.02303350158035755,
+            -0.006813278421759605,
+            -0.026015127077698708,
+            0.03598875552415848,
+            -0.007602275814861059,
+            -0.01588457264006138,
+            0.02857828140258789,
+            -0.012414725497364998,
+            -0.0037052377592772245,
+            0.0058847893960773945,
+            -0.015431225299835205,
+            0.023451974615454674,
+            0.0004955755430273712,
+            0.011499314568936825,
+            -0.02549203485250473,
+            0.00863102450966835,
+            -0.007087901700288057,
+            0.006076589692384005,
+            -0.02128986082971096,
+            -0.002423661295324564,
+            0.012135743163526058,
+            -0.010426974855363369,
+            0.00411717314273119,
+            0.021185241639614105,
+            0.005331183318048716,
+            -0.01744513027369976,
+            0.03951091319322586,
+            -0.004025631584227085,
+            0.010653648525476456,
+            0.005475033540278673,
+            -0.0009028792264871299,
+            0.016799984499812126,
+            0.01742769405245781,
+            -0.004925786517560482,
+            -0.016547156497836113,
+            0.0032235567923635244,
+            -0.015021469444036484,
+            0.01561430748552084,
+            0.016233300790190697,
+            -0.02125498652458191,
+            -0.032658401876688004,
+            0.01779385842382908,
+            0.002061855746433139,
+            0.028072625398635864,
+            0.006983283441513777,
+            -0.025997690856456757,
+            -0.005296310409903526,
+            -0.03815086930990219,
+            0.012885509058833122,
+            -0.01801181398332119,
+            -0.02437610551714897,
+            0.002785466844215989,
+            -0.00986029114574194,
+            -0.01856978051364422,
+            0.02512587048113346,
+            0.013652711175382137,
+            -0.02552690915763378,
+            0.010889039374887943,
+            0.0013011923292651772,
+            -0.017549749463796616,
+            0.010052092373371124,
+            0.004350385162979364,
+            -0.012493189424276352,
+            0.007410475518554449,
+            0.005130664445459843,
+            -0.006978924386203289,
+            -0.02890957146883011,
+            -0.01670408435165882,
+            0.01972058229148388,
+            0.010993658564984798,
+            -0.0058455574326217175,
+            0.011290077120065689,
+            -0.00997362844645977,
+            0.0033456117380410433,
+            0.0035221553407609463,
+            0.00901462510228157,
+            -0.013051155023276806,
+            -0.007371243555098772,
+            -0.007162006571888924,
+            0.01973801851272583,
+            -0.016024062409996986,
+            0.007990236394107342,
+            -0.013443474657833576,
+            -0.007981518283486366,
+            0.011429568752646446,
+            -0.0013774766121059656,
+            0.014986597001552582,
+            -0.018744144588708878,
+            0.0073407297022640705,
+            0.01189163327217102,
+            -0.008883852511644363,
+            -0.010671084746718407,
+            -0.02437610551714897,
+            0.019110308960080147,
+            0.01438504084944725,
+            0.004694754257798195,
+            -0.03679082915186882,
+            0.003951527178287506,
+            -0.010453129187226295,
+            -0.0009775288635864854,
+            -0.008796670474112034,
+            0.019249800592660904,
+            0.018831325694918633,
+            0.007985876873135567,
+            0.013277828693389893,
+            -0.0023081451654434204,
+            0.013548092916607857,
+            -0.012414725497364998,
+            0.017593340948224068,
+            -0.004036529455333948,
+            -0.017575904726982117,
+            -0.005714784376323223,
+            0.019406726583838463,
+            -0.001491903094574809,
+            0.007903054356575012,
+            0.004372180439531803,
+            0.010226456448435783,
+            -0.012275234796106815,
+            -0.007493298500776291,
+            0.014045030809938908,
+            0.008378196507692337,
+            -0.0004288267809897661,
+            0.009250016883015633,
+            0.06758353859186172,
+            -0.00870512891560793,
+            0.023853013291954994,
+            0.006311981473118067,
+            -0.009345917031168938,
+            -0.008112290874123573,
+            0.0019932000432163477,
+            -0.02951984666287899,
+            0.023329921066761017,
+            0.011970097199082375,
+            -0.009180271066725254,
+            0.00036126066697761416,
+            -0.006726096384227276,
+            0.018325669690966606,
+            -0.009694645181298256,
+            0.016948193311691284,
+            -0.010052092373371124,
+            0.02435866929590702,
+            0.019162617623806,
+            0.005640679504722357,
+            0.005536061245948076,
+            0.0028203397523611784,
+            0.019459037110209465,
+            0.012868072837591171,
+            -0.002165384590625763,
+            -0.003123297356069088,
+            -0.009877728298306465,
+            0.019389290362596512,
+            -0.001910376944579184,
+            0.030792705714702606,
+            -0.0022144243121147156,
+            -0.005182973574846983,
+            -0.009842854924499989,
+            0.0041607641614973545,
+            0.011569059453904629,
+            0.009520281106233597,
+            0.012876790948212147,
+            -0.0013567708665505052,
+            -0.010679802857339382,
+            0.011909069493412971,
+            0.00863102450966835,
+            0.005326824262738228,
+            -0.0032518908847123384,
+            0.0045247492380440235,
+            -0.008404350839555264,
+            0.014001439325511456,
+            -0.0024781501851975918,
+            0.0015354941133409739,
+            -0.008774874731898308,
+            0.004426669329404831,
+            -0.01312961895018816,
+            0.0005348074482753873,
+            0.009607463143765926,
+            0.012868072837591171,
+            0.0015213270671665668,
+            -0.002273272257298231,
+            0.015405070967972279,
+            0.0025980256032198668,
+            0.0073014977388083935,
+            0.0009486498311161995,
+            -0.009537717327475548,
+            0.019057998433709145,
+            -0.01033979281783104,
+            -0.0019212746992707253,
+            0.010705957189202309,
+            0.006067871581763029,
+            0.008260500617325306,
+            0.01496916078031063,
+            0.009354635141789913,
+            0.003968963399529457,
+            0.0013545913388952613,
+            -0.00565375667065382,
+            -0.003535232739523053,
+            0.0036267738323658705,
+            0.0016771649243310094,
+            0.01646869257092476,
+            -0.019459037110209465,
+            -0.007039951626211405,
+            0.010679802857339382,
+            0.011438286863267422,
+            0.0017839629435911775,
+            0.015684053301811218,
+            0.014428631402552128,
+            0.01994725689291954,
+            -0.022562718018889427,
+            0.003831651760265231,
+            -0.002920599188655615,
+            0.019057998433709145,
+            0.008770515210926533,
+            -0.024114558473229408,
+            0.0003645300166681409,
+            -0.01063621137291193,
+            0.012676272541284561,
+            -0.0067086596973240376,
+            0.0023473771288990974,
+            0.01665177382528782,
+            -0.010121837258338928,
+            -0.009930036962032318,
+            -0.019424164667725563,
+            -0.016983065754175186,
+            0.010976221412420273,
+            0.014437349513173103,
+            0.0014188881032168865,
+            0.019232362508773804,
+            -0.0012412546202540398,
+            -0.008474096655845642,
+            0.030827578157186508,
+            0.016765110194683075,
+            -0.0004157494695391506,
+            -0.008962316438555717,
+            -0.0006544103380292654,
+            -0.015082497149705887,
+            0.01074954867362976,
+            0.004426669329404831,
+            0.005108869168907404,
+            0.003387023229151964,
+            9.174550359603018e-05,
+            -0.016712801530957222,
+            -0.014620431698858738,
+            -0.004668599460273981,
+            0.011045967228710651,
+            -0.003751008305698633,
+            0.009197707287967205,
+            -0.01609380915760994,
+            0.003121117828413844,
+            -0.008173318579792976,
+            -0.010714675299823284,
+            -0.008356400765478611,
+            -0.039406292140483856,
+            -0.02067958563566208,
+            0.0003173972072545439,
+            -0.00473834527656436,
+            -0.002358274767175317,
+            0.009223862551152706,
+            -0.003672544378787279,
+            0.013670147396624088,
+            -0.0015551100950688124,
+            0.0047906544059515,
+            0.008657178841531277,
+            0.01561430748552084,
+            -0.013504501432180405,
+            -0.0006631285650655627,
+            -0.014071185141801834,
+            -0.026782330125570297,
+            0.01707896590232849,
+            0.003199581755325198,
+            0.014446067623794079,
+            -0.015187115408480167,
+            0.009528999216854572,
+            -0.012589089572429657,
+            0.014463504776358604,
+            -0.0421612448990345,
+            -0.009520281106233597,
+            0.007031233515590429,
+            -0.0075979167595505714,
+            -0.005496829282492399,
+            -0.025369981303811073,
+            -0.0012205488746985793,
+            -0.011054685339331627,
+            -0.01666049286723137,
+            -0.019459037110209465,
+            0.00574093870818615,
+            -0.009311044588685036,
+            0.008591792546212673,
+            -0.006120180711150169,
+            0.02761927805840969,
+            0.00532246520742774,
+            -0.001346962875686586,
+            -0.006128899287432432,
+            0.009206426329910755,
+            -0.0007503106608055532,
+            0.012885509058833122,
+            0.008648460730910301,
+            0.005823761690407991,
+            0.023434538394212723,
+            0.0009339378448203206,
+            0.004690395202487707,
+            -0.006141976453363895,
+            0.014541968703269958,
+            0.024689961224794388,
+            0.011342386715114117,
+            0.011455723084509373,
+            0.005967612378299236,
+            0.0005950175691395998,
+            0.011429568752646446,
+            0.017392821609973907,
+            -0.003990759141743183,
+            -0.009171552956104279,
+            0.004869118332862854,
+            -0.01761949434876442,
+            -0.0033347138669341803,
+            0.012885509058833122,
+            -0.006128899287432432,
+            -0.02706131339073181,
+            0.01285063661634922,
+            0.004317691549658775,
+            -0.006673786789178848,
+            -0.016948193311691284,
+            -0.00661711860448122,
+            0.004646804183721542,
+            0.004729627165943384,
+            0.024236613884568214,
+            -0.022144244983792305,
+            -0.008644101209938526,
+            0.011917788535356522,
+            -0.014446067623794079,
+            -0.01063621137291193,
+            0.009790545329451561,
+            0.029484974220395088,
+            0.003496000776067376,
+            -0.05307644233107567,
+            -0.012467035092413425,
+            0.014236831106245518,
+            -0.01503018755465746,
+            -0.025980254635214806,
+            -0.0014428631402552128,
+            0.0038578063249588013,
+            -0.014010157436132431,
+            0.006625836715102196,
+            0.0011606112821027637,
+            -0.011708551086485386,
+            0.017139993607997894,
+            0.028874699026346207,
+            0.01312961895018816,
+            0.0114818774163723,
+            -0.025213053449988365,
+            -0.0008952508214861155,
+            -0.01914518140256405,
+            0.011752142570912838,
+            -0.030566031113266945,
+            -0.013984003104269505,
+            -0.0023342997301369905,
+            0.004141148179769516,
+            0.027444913983345032,
+            0.014376322738826275,
+            0.025962818413972855,
+            0.006778405513614416,
+            0.0037989583797752857,
+            -0.02125498652458191,
+            -0.018447725102305412,
+            0.0033456117380410433,
+            -0.0008304091752506793,
+            -0.002329940674826503,
+            -0.008216909132897854,
+            0.012275234796106815,
+            0.012039843015372753,
+            0.026172054931521416,
+            0.005087073426693678,
+            0.005553497467190027,
+            -2.538836997700855e-05,
+            -0.00494322320446372,
+            0.007593557704240084,
+            -0.019232362508773804,
+            -0.0029925242997705936,
+            -0.004559622146189213,
+            0.015405070967972279,
+            0.0045029534958302975,
+            0.0022275017108768225,
+            0.017357949167490005,
+            -0.0030099607538431883,
+            0.003729212796315551,
+            -0.019982129335403442,
+            -0.014672741293907166,
+            -0.0032518908847123384,
+            0.03525642678141594,
+            0.003971143160015345,
+            -0.02761927805840969,
+            -0.016224581748247147,
+            -0.0153353251516819,
+            0.030792705714702606,
+            -0.009581308811903,
+            0.004315512254834175,
+            0.02683463878929615,
+            0.01593688130378723,
+            0.012388571165502071,
+            0.006399163510650396,
+            -0.01822105050086975,
+            0.005021686665713787,
+            0.02931061014533043,
+            -0.0030317562632262707,
+            -0.010086964815855026,
+            -0.0066912234760820866,
+            0.007144570350646973,
+            -0.007798435632139444,
+            0.007632789667695761,
+            -0.0035068984143435955,
+            -0.013417319394648075,
+            0.029554719105362892,
+            -0.00968592707067728,
+            0.022370917722582817,
+            -0.010130555368959904,
+            0.00268738716840744,
+            -0.038848329335451126,
+            -0.002004097681492567,
+            0.0011453543556854129,
+            0.0019692247733473778,
+            0.004555262625217438,
+            -0.01881388947367668,
+            -0.008753078989684582,
+            0.011839324608445168,
+            -0.01276345457881689,
+            -0.007885618135333061,
+            -0.006137617398053408,
+            0.002983806189149618,
+            0.01977289281785488,
+            -0.00947669055312872,
+            0.0009857022669166327,
+            -0.0210980586707592,
+            0.01256293524056673,
+            -0.00282687833532691,
+            0.0017567186150699854,
+            0.010453129187226295,
+            0.007009437773376703,
+            0.03502975404262543,
+            0.01094134896993637,
+            0.005531701724976301,
+            -0.0032780456822365522,
+            -0.014053748920559883,
+            -0.03218761831521988,
+            0.013295264914631844,
+            -0.010453129187226295,
+            0.023800702765583992,
+            0.0036703648511320353,
+            -0.008103572763502598,
+            -0.0051960512064397335,
+            0.0037139558698982,
+            0.009590026922523975,
+            -0.0041106343269348145,
+            0.013478347100317478,
+            -0.010243892669677734,
+            0.021167805418372154,
+            0.0031298361718654633,
+            -0.005121946334838867,
+            0.012571653351187706,
+            -0.017689241096377373,
+            0.01055774837732315,
+            -0.01744513027369976,
+            0.0028225192800164223,
+            0.01360040158033371,
+            -0.010296201333403587,
+            0.0056276023387908936,
+            -0.052588220685720444,
+            -0.001197663601487875,
+            -0.016521001234650612,
+            -0.00675660977140069,
+            -0.008456660434603691,
+            -0.021394478157162666,
+            -0.03319893032312393,
+            -0.0036660057958215475,
+            0.015370197594165802,
+            0.006551732309162617,
+            0.023260174319148064,
+            0.0009895163821056485,
+            0.012257798574864864,
+            0.012807045131921768,
+            -0.0047906544059515,
+            0.012519344687461853,
+            0.033861514180898666,
+            -0.027741333469748497,
+            -0.019685709848999977,
+            0.005257078446447849,
+            -0.00028552126605063677,
+            -0.015788672491908073,
+            -0.013103464618325233,
+            -0.0017665266059339046,
+            0.0024955866392701864,
+            0.030443977564573288,
+            -0.020191365852952003,
+            -0.005579652264714241,
+            -0.007053028792142868,
+            0.004433208145201206,
+            -0.01918005384504795,
+            -0.0031450928654521704,
+            -0.011708551086485386,
+            -0.012780890800058842,
+            0.016041500493884087,
+            -0.011089558713138103,
+            0.011159304529428482,
+            -0.019511345773935318,
+            0.008626664988696575,
+            -0.015378915704786777,
+            -0.010627493262290955,
+            -0.0031690681353211403,
+            -0.010461847297847271,
+            0.030164994299411774,
+            -0.0041890982538461685,
+            0.0048386044800281525,
+            0.01670408435165882,
+            -0.00853076484054327,
+            0.011603932827711105,
+            -0.0006124540232121944,
+            0.007898694835603237,
+            -0.002704823622480035,
+            -0.0086135882884264,
+            0.0015278656501322985,
+            0.009537717327475548,
+            -0.010950067080557346,
+            0.03846472501754761,
+            -0.013731175102293491,
+            -0.008879492990672588,
+            -0.003859985852614045,
+            -0.004511671606451273,
+            0.002425840822979808,
+            0.0021762822289019823,
+            0.010984940454363823,
+            -0.0012870251666754484,
+            0.006835073698312044,
+            -0.024219177663326263,
+            -0.011717269197106361,
+            0.0013164491392672062,
+            0.006468909326940775,
+            0.003321636701002717,
+            0.008016390725970268,
+            -0.025579217821359634,
+            -0.0020782025530934334,
+            -0.012859354726970196,
+            0.005475033540278673,
+            -0.011499314568936825,
+            0.007332011591643095,
+            -0.01215317938476801,
+            -0.0013295264216139913,
+            0.009337198920547962,
+            0.012946536764502525,
+            0.013861947692930698,
+            -0.006608400493860245,
+            0.026573093608021736,
+            0.014899414964020252,
+            0.008709488436579704,
+            0.005405288189649582,
+            -0.033216364681720734,
+            -0.007802794687449932,
+            0.010365947149693966,
+            0.007519452832639217,
+            0.009232580661773682,
+            -0.017689241096377373,
+            -0.004176021087914705,
+            0.0029402151703834534,
+            -0.007039951626211405,
+            4.651980270864442e-05,
+            0.011080840602517128,
+            -0.010627493262290955,
+            0.005505547393113375,
+            -0.00031521765049546957,
+            0.0027549532242119312,
+            -0.006460190750658512,
+            0.012188052758574486,
+            -0.013286546804010868,
+            -0.008168959058821201,
+            0.0211154967546463,
+            0.02760184183716774,
+            -0.005531701724976301,
+            0.023696085438132286,
+            -0.021917570382356644,
+            0.008487174287438393,
+            -0.005043482407927513,
+            -0.014149649068713188,
+            -0.015300451777875423,
+            0.006883023772388697,
+            0.014777359552681446,
+            0.004760140553116798,
+            -0.013155773282051086,
+            -0.003027397207915783,
+            0.016983065754175186,
+            -0.0010952246375381947,
+            0.006216081324964762,
+            0.01593688130378723,
+            -0.00026971951592713594,
+            0.00017667989595793188,
+            -0.006002485286444426,
+            -0.01876158080995083,
+            -0.006290185730904341,
+            0.017924632877111435,
+            -0.0073799616657197475,
+            0.014262985438108444,
+            0.02320786565542221,
+            0.0005429807933978736,
+            0.011351104825735092,
+            -0.0038469084538519382,
+            0.01351321954280138,
+            0.01505634281784296,
+            -0.005235282704234123,
+            -0.0030600903555750847,
+            -0.0036965194158256054,
+            0.00011858122888952494,
+            0.012022406794130802,
+            -0.01741897687315941,
+            -0.011333668604493141,
+            0.0114818774163723,
+            -0.002121793571859598,
+            -0.008404350839555264,
+            -0.023486848920583725,
+            0.0016313943779096007,
+            0.0077156126499176025,
+            0.011447004973888397,
+            -0.016425101086497307,
+            -0.0005239096935838461,
+            0.000875089957844466,
+            0.010801857337355614,
+            -0.007497657556086779,
+            0.01159521471709013,
+            -0.007907412946224213,
+            -0.00863102450966835,
+            0.006381727289408445,
+            0.0015507509233430028,
+            0.007414834573864937,
+            -0.009999782778322697,
+            0.0015987011138349771,
+            -0.008025108836591244,
+            0.026590529829263687,
+            0.008317168802022934,
+            -0.0013011923292651772,
+            -0.005640679504722357,
+            0.004995532333850861,
+            0.008421787060797215,
+            -0.0100608104839921,
+            0.012815763242542744,
+            0.006107103545218706,
+            -0.002933676354587078,
+            0.010592620819807053,
+            0.015283015556633472,
+            0.0025631526950746775,
+            0.009328480809926987,
+            0.003443691413849592,
+            0.0059022256173193455,
+            -0.007022515404969454,
+            0.0017861425876617432,
+            -0.01550968922674656,
+            -0.020801641047000885,
+            -0.0035831828135997057,
+            0.016991784796118736,
+            -0.010583902709186077,
+            -0.001182406791485846,
+            0.017924632877111435,
+            0.01371373888105154,
+            -0.005034764297306538,
+            0.00551426550373435,
+            0.002217693720012903,
+            -0.0015529305674135685,
+            0.003323816228657961,
+            -0.014280421659350395,
+            -0.012493189424276352,
+            0.008726924657821655,
+            -0.02087138593196869,
+            0.01952878199517727,
+            -0.02165602520108223,
+            -0.035361044108867645,
+            -0.008831542916595936,
+            -0.009197707287967205,
+            -0.01707896590232849,
+            -0.013574247248470783,
+            0.023661212995648384,
+            0.016207145527005196,
+            -0.009354635141789913,
+            -0.034384604543447495,
+            0.03155990689992905,
+            0.00918898917734623,
+            0.027933133766055107,
+            -0.015544561669230461,
+            0.004343846347182989,
+            -0.0015213270671665668,
+            0.014707613736391068,
+            -0.006163771729916334,
+            -0.006686864420771599,
+            0.0091018071398139,
+            0.014620431698858738,
+            0.010897758416831493,
+            0.003245352301746607,
+            -0.003160349791869521,
+            0.007279702462255955,
+            0.002530459314584732,
+            0.0022285915911197662,
+            -0.008583074435591698,
+            -0.02282426506280899,
+            -0.009781827218830585,
+            0.017942069098353386,
+            0.022266298532485962,
+            0.012746017426252365,
+            -0.017131276428699493,
+            0.0020520477555692196,
+            -0.00484296353533864,
+            -0.014053748920559883,
+            -0.009991064667701721,
+            0.009642336517572403,
+            -0.033617403358221054,
+            0.02317299321293831,
+            0.004760140553116798,
+            -0.00627274950966239,
+            -0.0015006213216111064,
+            0.003836010815575719,
+            0.005679911468178034,
+            -0.004313332494348288,
+            0.0002961465797852725,
+            -0.002532638842239976,
+            0.010897758416831493,
+            0.016599465161561966,
+            0.005100150592625141,
+            -0.010400820523500443,
+            0.012388571165502071,
+            -0.0031407338101416826,
+            -0.002761491807177663,
+            -0.013914257287979126,
+            -0.006011203397065401,
+            -0.009502844884991646,
+            0.010261328890919685,
+            0.01483838725835085,
+            0.0054924702271819115,
+            0.006791482679545879,
+            0.004335128236562014,
+            -0.00013084120291750878,
+            -0.0002569146454334259,
+            0.008469737134873867,
+            -0.0005070181796327233,
+            -0.0045639812014997005,
+            0.0010821473551914096,
+            -0.0014679280575364828,
+            0.02320786565542221,
+            -0.00532246520742774,
+            -0.010950067080557346,
+            0.007196879480034113,
+            0.02033085748553276,
+            -0.022370917722582817,
+            0.022091934457421303,
+            -0.022179117426276207,
+            0.023853013291954994,
+            0.03482051566243172,
+            0.013975284993648529,
+            -0.011717269197106361,
+            -0.007096619810909033,
+            0.012815763242542744,
+            -0.012510626576840878,
+            -0.0009600924677215517,
+            0.0018079380970448256,
+            -0.022580154240131378,
+            0.015178397297859192,
+            -0.023347357288002968,
+            0.014280421659350395,
+            0.005121946334838867,
+            -0.00852640625089407,
+            -0.0073232934810221195,
+            -0.0009764390997588634,
+            0.018133869394659996,
+            -0.003990759141743183,
+            -0.0006255313055589795,
+            -0.03028704971075058,
+            -0.006930973846465349,
+            0.02163858897984028,
+            -0.0045247492380440235,
+            -0.013949130661785603,
+            0.007253547664731741,
+            -8.973622243502177e-06,
+            -0.016154836863279343,
+            -0.005876071285456419,
+            -0.03166452422738075,
+            -0.008142804726958275,
+            -0.02264990098774433,
+            -0.0075979167595505714,
+            -0.011577778495848179,
+            -0.012292671017348766,
+            -0.0038512677419930696,
+            -0.016276890411973,
+            0.001949608908034861,
+            -0.001063076313585043,
+            0.0033020207192748785,
+            0.0027091826777905226,
+            -0.009563872590661049,
+            -0.003968963399529457,
+            0.021324733272194862,
+            0.010705957189202309,
+            -0.016041500493884087,
+            -0.026206929236650467,
+            0.011638805270195007,
+            -0.012205488979816437,
+            -0.007593557704240084,
+            0.001547481631860137,
+            -0.005915303248912096,
+            -0.019267236813902855,
+            -0.01860465295612812,
+            0.005967612378299236,
+            0.0003718859807122499,
+            0.0051175872795283794,
+            0.013582965359091759,
+            0.014341449365019798,
+            -0.0019180054077878594,
+            -0.006734814494848251,
+            -1.330718441749923e-05,
+            -0.002195898210629821,
+            0.014027593657374382,
+            0.004348205402493477,
+            0.024445850402116776,
+            -0.008744360879063606,
+            0.0028944446239620447,
+            -0.017924632877111435,
+            0.00449859444051981,
+            -0.007393038831651211,
+            -0.005679911468178034,
+            -0.004064863547682762,
+            -0.011089558713138103,
+            0.005993766710162163,
+            -0.0011126610916107893,
+            -0.00247161160223186,
+            -0.007911772467195988,
+            -0.0020912797190248966,
+            -0.01707024872303009,
+            -0.0029794471338391304,
+            -0.020557530224323273,
+            -0.0013905538944527507,
+            -0.014916851185262203,
+            0.017000501975417137,
+            0.026172054931521416,
+            -0.00513502350077033,
+            -0.0006091846735216677,
+            -0.011229049414396286,
+            -4.08665910072159e-05,
+            -0.011909069493412971,
+            -0.00637736776843667,
+            -0.016416382044553757,
+            -0.0325189083814621,
+            -0.016904601827263832,
+            0.009049498476088047,
+            0.024288922548294067,
+            0.0019398009171709418,
+            -0.024986378848552704,
+            0.0029511128086596727,
+            -0.026607966050505638,
+            -0.014228112995624542,
+            -0.01496916078031063,
+            -0.006529936566948891,
+            -0.009136680513620377,
+            0.017392821609973907,
+            -0.0026350778061896563,
+            0.01571020856499672,
+            0.01747128553688526,
+            0.010906476527452469,
+            0.007497657556086779,
+            0.010880321264266968,
+            -0.009528999216854572,
+            -0.00016932391736190766,
+            0.009869009256362915,
+            -0.006311981473118067,
+            0.011839324608445168,
+            -0.00815152283757925,
+            -0.0011933045461773872,
+            0.01879645325243473,
+            0.008016390725970268,
+            -0.006255313288420439,
+            -0.0026699507143348455,
+            -0.02742747776210308,
+            0.017506157979369164,
+            -0.018552342429757118,
+            -0.00258494820445776,
+            -0.002944574225693941,
+            -0.016738956794142723,
+            0.02319042943418026,
+            0.017323076725006104,
+            -0.005021686665713787,
+            -0.0027549532242119312,
+            0.017889760434627533,
+            -0.006451472640037537,
+            0.00398204056546092,
+            0.0025762300938367844,
+            -0.0033281752839684486,
+            0.009790545329451561,
+            0.0027636713348329067,
+            -0.004577058367431164,
+            -0.0025587936397641897,
+            0.012196770869195461,
+            0.003901397343724966,
+            0.013530656695365906,
+            -0.027131058275699615,
+            -0.010566466487944126,
+            -0.0010303830495104194,
+            0.01935441792011261,
+            -0.005405288189649582,
+            -0.0007906323298811913,
+            -0.013312701135873795,
+            -0.0008854428306221962,
+            -0.005719143431633711,
+            0.023818138986825943,
+            -0.01992982067167759,
+            0.006120180711150169,
+            -0.008395632728934288,
+            0.013861947692930698,
+            -0.0032191977370530367,
+            0.010801857337355614,
+            -0.0007955363253131509,
+            -1.3375295566220302e-05,
+            0.009964910335838795,
+            0.012981409206986427,
+            -0.0022024367935955524,
+            0.024027377367019653,
+            -0.0004737800336442888,
+            -0.006403522565960884,
+            0.0027745692059397697,
+            -0.01895338110625744,
+            0.0229637548327446,
+            0.003402280155569315,
+            0.004315512254834175,
+            -0.0058629936538636684,
+            -0.008199472911655903,
+            0.039789892733097076,
+            0.021516533568501472,
+            0.010392102412879467,
+            -0.000592293159570545,
+            -0.01494300551712513,
+            0.028229551389813423,
+            0.007519452832639217,
+            0.00930232647806406,
+            -0.004012554418295622,
+            0.0027113622054457664,
+            -0.015178397297859192,
+            -0.010976221412420273,
+            0.019807765260338783,
+            -0.019999565556645393,
+            -0.008731283247470856,
+            -0.0026328982785344124,
+            -0.019703146070241928,
+            -0.029293173924088478,
+            0.015474816784262657,
+            0.009764390997588634,
+            0.0028443147893995047,
+            0.008495892398059368,
+            0.0037335718516260386,
+            -0.0015028008492663503,
+            -0.011647523380815983,
+            -0.003986400086432695,
+            -0.008574356324970722,
+            -0.01044441107660532,
+            0.006848151329904795,
+            -0.012004970572888851,
+            0.006717377807945013,
+            -0.008256141096353531,
+            -0.00024533577379770577,
+            0.01285063661634922,
+            0.005091432482004166,
+            0.0062771085649728775,
+            0.003443691413849592,
+            0.005078355316072702,
+            0.01591944508254528,
+            -0.013757329434156418,
+            0.016346637159585953,
+            0.017113838344812393,
+            0.0062945447862148285,
+            -0.0034131777938455343,
+            0.0023103246930986643,
+            -0.029188554733991623,
+            -0.00022381270537152886,
+            -0.0016869729151949286,
+            0.012632681056857109,
+            -0.02163858897984028,
+            0.004733986221253872,
+            0.021568842232227325,
+            -0.0016172273317351937,
+            0.009816700592637062,
+            0.014864541590213776,
+            -0.0001531135058030486,
+            0.028037751093506813,
+            0.011342386715114117,
+            0.02416686899960041,
+            -0.009459254331886768,
+            -0.02852597087621689,
+            0.006368649657815695,
+            -0.0038904997054487467,
+            0.014376322738826275,
+            0.0091018071398139,
+            0.0201216209679842,
+            -0.01369630265980959,
+            -0.006695582531392574,
+            0.007789717521518469,
+            0.0018918508430942893,
+            0.014306576922535896,
+            0.003217018209397793,
+            0.019302109256386757,
+            0.009119244292378426,
+            -0.0008625575574114919,
+            -0.007737408392131329,
+            8.588795753894374e-05,
+            0.006250953767448664,
+            -0.003500359831377864,
+            -0.03047885000705719,
+            0.0016390228411182761,
+            -0.004834245424717665,
+            0.004978095646947622,
+            -0.014245549216866493,
+            0.010409538634121418,
+            -0.0010015039006248116,
+            0.022196553647518158,
+            -0.0041498662903904915,
+            0.004064863547682762,
+            0.01708768494427204,
+            0.00938950851559639,
+            0.004991173278540373,
+            0.015300451777875423,
+            0.005658115725964308,
+            0.004511671606451273,
+            -0.008038186468183994,
+            -0.019493909552693367,
+            0.019162617623806,
+            -0.027462350204586983,
+            0.010653648525476456,
+            -0.00873564276844263,
+            0.004317691549658775,
+            0.003925372380763292,
+            -0.006887382827699184,
+            -0.002567511750385165,
+            0.003607157850638032,
+            0.001491903094574809,
+            -0.059144310653209686,
+            0.0037706242874264717,
+            0.0007388679659925401,
+            0.020452912896871567,
+            -0.00881846621632576,
+            0.0007590288296341896,
+            0.0026394370943307877,
+            0.00390357687138021,
+            -0.013347573578357697,
+            -0.0023604542948305607,
+            0.01571892574429512,
+            -0.01858721673488617,
+            -0.0059632533229887486,
+            0.004038709215819836,
+            -0.005557856522500515,
+            0.0035940806847065687,
+            -0.008779234252870083,
+            -0.002622000640258193,
+            0.02838647924363613,
+            -0.001655369414947927,
+            0.03140297904610634,
+            0.002506484277546406,
+            -0.0004756871494464576,
+            -0.01666049286723137,
+            0.007410475518554449,
+            -0.016738956794142723,
+            -0.00425884360447526,
+            0.008927443064749241,
+            -0.018726706504821777,
+            -0.00123580580111593,
+            0.001693511614575982,
+            -0.0013382446486502886,
+            -0.0077722808346152306,
+            -0.001763257198035717,
+            0.0010429153917357326,
+            -0.02048778533935547,
+            0.015997909009456635,
+            -0.023853013291954994,
+            0.01625945419073105,
+            -0.0034371528308838606,
+            -0.002946753753349185,
+            -0.0013033718569204211,
+            -0.0038098562508821487,
+            0.005666834302246571,
+            0.02531767077744007,
+            -0.015448661521077156,
+            -0.00522220553830266,
+            0.008365118876099586,
+            -0.003567925887182355,
+            0.002730978187173605,
+            -0.008761797100305557,
+            0.006656350567936897,
+            -0.0325886569917202,
+            -0.008112290874123573,
+            0.004413592163473368,
+            0.0031908636447042227,
+            -0.001236895564943552,
+            -0.02758440561592579,
+            0.003960245288908482,
+            -0.00854384247213602,
+            -0.024201741442084312,
+            -0.00027380615938454866,
+            -0.020592402666807175,
+            0.006717377807945013,
+            -0.00881846621632576,
+            0.01761949434876442,
+            -0.014568123035132885,
+            0.011098276823759079,
+            -0.014367603696882725,
+            0.016006626188755035,
+            -0.007406116463243961,
+            -0.009398226626217365,
+            -0.0069266147911548615,
+            -0.006338136270642281,
+            0.01044441107660532,
+            -0.014916851185262203,
+            -0.009511562995612621,
+            -0.007641507778316736,
+            0.005161178298294544,
+            0.011045967228710651,
+            0.013574247248470783,
+            0.0011333668371662498,
+            -0.00880538858473301,
+            -0.016547156497836113,
+            -0.005392210558056831,
+            -0.014881978742778301,
+            -0.0009301236714236438,
+            0.0042174323461949825,
+            0.020853949710726738,
+            0.008160240948200226,
+            -0.003753187833353877,
+            0.011586496606469154,
+            0.020801641047000885,
+            0.014132212847471237,
+            -0.014777359552681446,
+            0.0021697436459362507,
+            -0.010095682926476002,
+            0.013417319394648075,
+            0.022144244983792305,
+            0.002098908182233572,
+            0.002796364715322852,
+            0.006085308268666267,
+            0.007951004430651665,
+            -0.004396155476570129,
+            0.008291014470160007,
+            -0.025457162410020828,
+            -0.012667553499341011,
+            -0.006569168530404568,
+            0.005387851502746344,
+            -0.015378915704786777,
+            0.029973194003105164,
+            -0.003609337378293276,
+            0.004472440108656883,
+            -0.002182820811867714,
+            0.0326409637928009,
+            0.007196879480034113,
+            0.00473834527656436,
+            0.002874828642234206,
+            9.467427298659459e-05,
+            0.009136680513620377,
+            -0.01761077716946602,
+            -0.03888320177793503,
+            0.003254070645198226,
+            -0.000592293159570545,
+            0.01766308583319187,
+            -0.01514352485537529,
+            0.006878664717078209,
+            -0.0036899808328598738,
+            -0.01282448135316372,
+            -0.009633618406951427,
+            0.03138554468750954,
+            -0.005466315429657698,
+            0.03387894853949547,
+            -0.007558684796094894,
+            -0.010104401037096977,
+            0.021725770086050034,
+            0.021900134161114693,
+            -0.014210676774382591,
+            0.005531701724976301,
+            0.015762517228722572,
+            0.020278548821806908,
+            0.005209128372371197,
+            -0.00806434080004692,
+            0.02627667412161827,
+            0.012946536764502525,
+            -0.012807045131921768,
+            -0.0026350778061896563,
+            -0.009572590701282024,
+            0.006726096384227276,
+            -0.012301389127969742,
+            0.005387851502746344,
+            0.006434036418795586,
+            0.004986814223229885,
+            0.012963972985744476,
+            -0.01879645325243473,
+            -0.003831651760265231,
+            0.0073014977388083935,
+            0.013966566883027554,
+            0.00651250034570694,
+            0.017558468505740166,
+            -0.023643774911761284,
+            0.0028966241516172886,
+            -0.02493407018482685,
+            -0.008103572763502598,
+            0.0029968833550810814,
+            0.001748000388033688,
+            0.008875134401023388,
+            -0.01592816226184368,
+            -0.029484974220395088,
+            0.011516750790178776,
+            0.01708768494427204,
+            0.02568383514881134,
+            0.0011126610916107893,
+            0.008914366364479065,
+            0.00560144754126668,
+            -0.013739893212914467,
+            -0.00805998221039772,
+            -0.00997362844645977,
+            -0.01073211245238781,
+            -0.011159304529428482,
+            -0.002761491807177663,
+            0.009119244292378426,
+            -0.010592620819807053,
+            0.0006821996648795903,
+            0.016346637159585953,
+            -0.011717269197106361,
+            -0.0051960512064397335,
+            0.015535843558609486,
+            -0.005008609499782324,
+            0.0007977159111760557,
+            0.0016891524428501725,
+            0.013086027465760708,
+            0.0034480507019907236,
+            -0.0063730087131261826,
+            -0.01591944508254528,
+            0.01215317938476801,
+            -0.00417166156694293,
+            0.0016074193408712745,
+            0.00708354264497757,
+            -0.007933568209409714,
+            -0.0047819362953305244,
+            0.013373728841543198,
+            -0.00854384247213602,
+            0.0016259454423561692,
+            -0.007606634870171547,
+            -0.017549749463796616,
+            -0.0076938169077038765,
+            0.006394804455339909,
+            -0.026730021461844444,
+            -0.013478347100317478,
+            0.016233300790190697,
+            -0.003570105414837599,
+            0.001546391868032515,
+            -0.0005138292908668518,
+            -0.011124431155622005,
+            0.003160349791869521,
+            -0.019598528742790222,
+            -0.01064493041485548,
+            -0.004864759277552366,
+            -0.02662540227174759,
+            0.017549749463796616,
+            0.015091215260326862,
+            0.0033172776456922293,
+            -0.00699636060744524,
+            -0.005592729430645704,
+            0.002567511750385165,
+            0.010348510928452015,
+            0.01763693243265152,
+            0.009311044588685036,
+            -0.005592729430645704,
+            -0.019685709848999977,
+            -0.003443691413849592,
+            0.021324733272194862,
+            -6.375188240781426e-05,
+            -0.016590746119618416,
+            0.020609840750694275,
+            0.007109697442501783,
+            0.014306576922535896,
+            0.0055883703753352165,
+            0.0023451976012438536,
+            0.005383492447435856,
+            0.014114775694906712,
+            -0.0001486181718064472,
+            -0.01783744990825653,
+            -0.013984003104269505,
+            0.0004890369018539786,
+            0.03140297904610634,
+            0.00708354264497757,
+            0.017139993607997894,
+            -0.03623286634683609,
+            0.0006642183288931847,
+            -0.005335542373359203,
+            -0.0038490882143378258,
+            -0.0045988536439836025,
+            0.027671586722135544,
+            0.010766984894871712,
+            -0.0001184450084110722,
+            0.015605589374899864,
+            -0.010043373331427574,
+            0.0422309935092926,
+            -0.015378915704786777,
+            -0.00737560261040926,
+            -0.010052092373371124,
+            -0.01685229316353798,
+            -0.003940629307180643,
+            -0.0011769578559324145,
+            -0.01689588464796543,
+            -0.006647632457315922,
+            -0.006320699583739042,
+            0.021411914378404617,
+            0.0046380856074392796,
+            -0.01665177382528782,
+            0.0017065888969227672,
+            -0.023835577070713043,
+            -0.0010946798138320446,
+            -0.022475536912679672,
+            -0.006403522565960884,
+            -0.012327543459832668,
+            -0.01349578332155943,
+            -0.020993441343307495,
+            -0.02031342126429081,
+            -0.004908350296318531,
+            0.016547156497836113,
+            0.00532246520742774,
+            -0.0069091785699129105,
+            -0.0021043571177870035,
+            -0.014402477070689201,
+            -0.010278765112161636,
+            0.02144678682088852,
+            0.015605589374899864,
+            0.012432162649929523,
+            0.01332141924649477,
+            0.01685229316353798,
+            0.0037597264163196087,
+            -0.034593842923641205,
+            0.006128899287432432,
+            0.0041607641614973545,
+            -0.005435801576822996,
+            0.01782873272895813,
+            0.000511104823090136,
+            -0.010793139226734638,
+            0.013399883173406124,
+            -0.007593557704240084,
+            0.018098996952176094,
+            -0.003212659154087305,
+            -0.004184739198535681,
+            -0.005536061245948076,
+            -0.008740002289414406,
+            -0.007371243555098772,
+            0.00455090356990695,
+            0.0021043571177870035,
+            0.013643993064761162,
+            0.0005775811732746661,
+            -0.004625008441507816,
+            -0.011333668604493141,
+            0.0014602995943278074,
+            -0.0032257363200187683,
+            -0.001628125086426735,
+            -0.023835577070713043,
+            -0.012170616537332535,
+            0.0017120378324761987,
+            -0.0086179468780756,
+            -0.01235369872301817,
+            0.02856084331870079,
+            -0.0017011400777846575,
+            -0.0008009852026589215,
+            0.010688520967960358,
+            0.015675334259867668,
+            0.001266319421119988,
+            0.01837797835469246,
+            -0.012170616537332535,
+            -0.011185458861291409,
+            -0.01649484597146511,
+            -0.01282448135316372,
+            -0.01122033130377531,
+            0.02090625837445259,
+            0.0033979208674281836,
+            -0.00708354264497757,
+            0.010418256744742393,
+            -0.011106994934380054,
+            -0.005056559573858976,
+            -0.0091018071398139,
+            0.011708551086485386,
+            -0.02969421073794365,
+            0.02069702185690403,
+            -0.012711144983768463,
+            0.006024280562996864,
+            0.018639525398612022,
+            0.039231929928064346,
+            -0.006608400493860245,
+            -0.006869946606457233,
+            0.010034655220806599,
+            -0.003068808699026704,
+            0.029205990955233574,
+            -0.0059414575807750225,
+            0.004237048327922821,
+            -0.021551406010985374,
+            0.0018449904164299369,
+            -0.01074954867362976,
+            0.0012815763475373387,
+            0.01991238258779049,
+            -0.0191974900662899,
+            -0.013722456991672516,
+            -0.00675660977140069,
+            -0.016058936715126038,
+            -0.00823870487511158,
+            0.022283734753727913,
+            0.011377259157598019,
+            -0.006111462600529194,
+            -0.008273578248918056,
+            -0.005736579652875662,
+            -0.024201741442084312,
+            0.014716332778334618,
+            0.08174190670251846,
+            -0.009982346557080746,
+            -0.0001858067698776722,
+            0.0003367407189216465,
+            0.0017806936521083117,
+            0.007907412946224213,
+            -0.00031821453012526035,
+            0.05956278741359711,
+            -0.003509078174829483,
+            0.003899217816069722,
+            -0.012998845428228378,
+            0.01083673071116209,
+            -0.01438504084944725,
+            -0.005684270523488522,
+            -0.009712082333862782,
+            -0.01197881530970335,
+            -0.0008167869527824223,
+            -0.02203962579369545,
+            -0.001063076313585043,
+            -0.011996251530945301,
+            -0.005021686665713787,
+            -0.021324733272194862,
+            0.017514877021312714,
+            -0.02087138593196869,
+            -0.01648612879216671,
+            -0.004594494588673115,
+            -0.010705957189202309,
+            0.012118306942284107,
+            -0.016555873677134514,
+            0.002380070276558399,
+            0.007270984351634979,
+            0.014228112995624542,
+            0.007475861813873053,
+            -0.008199472911655903,
+            -0.008408710360527039,
+            -0.0076197125017642975,
+            0.004986814223229885,
+            0.0069876424968242645,
+            -0.0011088468600064516,
+            0.014637868851423264,
+            -0.02088882215321064,
+            -0.016058936715126038,
+            0.007589198648929596,
+            0.0017817834159359336,
+            0.009502844884991646,
+            -0.007820230908691883,
+            0.007449707482010126,
+            0.00020882829267065972,
+            0.022754518315196037,
+            -0.010156710632145405,
+            -0.0268520750105381,
+            -0.007933568209409714,
+            0.008644101209938526,
+            -0.013138337060809135,
+            -0.006102744489908218,
+            0.005331183318048716,
+            0.009807982482016087,
+            0.008081777021288872,
+            -0.009877728298306465,
+            0.0007110786973498762,
+            -0.018918508663773537,
+            0.013373728841543198,
+            0.003238813718780875,
+            0.022196553647518158,
+            -0.012667553499341011,
+            0.010880321264266968,
+            -0.02683463878929615,
+            -0.002574050333350897,
+            -0.014419913291931152,
+            0.0008592882077209651,
+            0.008434864692389965,
+            0.004742704331874847,
+            0.01684357412159443,
+            -0.017192302271723747,
+            -0.017096402123570442,
+            0.009311044588685036,
+            -0.01782873272895813,
+            -0.02376583032310009,
+            -0.0018471699440851808,
+            -0.02624180167913437,
+            -0.0038163948338478804,
+            0.007532530464231968,
+            -0.019389290362596512,
+            -0.01369630265980959,
+            0.0065953233279287815,
+            -0.0033412526827305555,
+            -0.014611713588237762,
+            0.017183585092425346,
+            -0.011970097199082375,
+            -0.03544822707772255,
+            0.003727033268660307,
+            0.008382555097341537,
+            -0.004729627165943384,
+            0.006747891660779715,
+            -0.024410977959632874,
+            0.016015345230698586,
+            -0.002397506730630994,
+            0.004293716512620449,
+            0.008029468357563019,
+            0.02606743760406971,
+            0.0010309278732165694,
+            0.0034545892849564552,
+            -0.005544779356569052,
+            0.014411195181310177,
+            0.005801966413855553,
+            0.023504285141825676,
+            0.026416165754199028,
+            0.0015529305674135685,
+            -0.010958785191178322,
+            -0.01081929448992014,
+            0.01592816226184368,
+            -0.011586496606469154,
+            -0.0048386044800281525,
+            0.022737082093954086,
+            0.013827075250446796,
+            -0.02203962579369545,
+            0.00017259323794860393,
+            -0.02149909734725952,
+            0.0069876424968242645,
+            -0.017148712649941444,
+            0.009467972442507744,
+            0.008731283247470856,
+            -0.005073996260762215,
+            0.02069702185690403,
+            0.010095682926476002,
+            0.030984506011009216,
+            0.01991238258779049,
+            0.002920599188655615,
+            0.011412132531404495,
+            -0.006451472640037537,
+            0.010470565408468246,
+            0.004537826403975487,
+            0.0022275017108768225,
+            -0.006608400493860245,
+            0.006930973846465349,
+            0.0023451976012438536,
+            -0.001170419156551361,
+            0.008583074435591698,
+            -0.007471502758562565,
+            0.009921318851411343,
+            0.0005939278053119779,
+            0.0055273426696658134,
+            -0.004287178162485361,
+            -0.010575184598565102,
+            -0.030339358374476433,
+            -0.003132015699520707,
+            0.020958568900823593,
+            -0.023539157584309578,
+            -0.008003313094377518,
+            0.005932739470154047,
+            0.016983065754175186,
+            -0.008282296359539032,
+            -0.012798327021300793,
+            -0.01708768494427204,
+            0.003105861134827137,
+            -0.0004893093137070537,
+            -0.010313638485968113,
+            0.01747128553688526,
+            0.0009546435903757811,
+            -0.004951941315084696,
+            0.008953598327934742,
+            -0.00680456031113863,
+            0.028212115168571472,
+            0.01505634281784296,
+            0.011150586418807507,
+            -0.012100870721042156,
+            0.014201958663761616,
+            -0.012711144983768463,
+            0.007680739741772413,
+            0.009799264371395111,
+            -0.006416599731892347,
+            -0.021220114082098007,
+            -0.017558468505740166,
+            0.023451974615454674,
+            0.006848151329904795,
+            0.012222925201058388,
+            -0.011987533420324326,
+            -0.0012815763475373387,
+            0.007475861813873053,
+            0.004834245424717665,
+            -0.01611996442079544,
+            -0.00048631246318109334,
+            0.0061986446380615234,
+            0.038604218512773514,
+            0.005087073426693678,
+            0.01072339341044426,
+            -0.0035984397400170565,
+            -0.0002066487359115854,
+            0.019162617623806,
+            -0.01952878199517727,
+            -0.006503781769424677,
+            -0.005832480266690254,
+            -0.02739260531961918,
+            0.016573309898376465,
+            -0.01685229316353798,
+            -0.016451256349682808,
+            0.010976221412420273,
+            -0.00041356991278007627,
+            -0.004422310274094343,
+            -0.01284191757440567,
+            -0.008766156621277332,
+            0.00794228632003069,
+            0.010810576379299164,
+            -0.009136680513620377,
+            0.00532246520742774,
+            0.00357446470297873,
+            -0.0035984397400170565,
+            0.02127242274582386,
+            0.0006467819330282509,
+            -0.002837776206433773,
+            0.007358166389167309
+          ],
+          "title": "Method",
+          "keyphrases": [
+            "mixture-of-experts (MoE) models",
+            "token-choice routing",
+            "load imbalance",
+            "expert choice method",
+            "Transformer architecture"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": true,
+      "properties": {
+        "cosine_similarity": 0.810423577096145
+      }
+    },
+    {
+      "id": "b561ecfd-aaef-4c88-8047-17399ffd276f",
+      "type": "cosine_similarity",
+      "source": {
+        "id": "1e687808-8db8-4320-805d-0282a8d679bc",
+        "properties": {
+          "page_content": "Deepseekmoe: Towards Ultimate Expert Specialization In\n\nMixture-Of-Experts Language Models\n\nDamai Dai∗1,2, Chengqi Deng1, Chenggang Zhao∗1,3, R.X. Xu1, Huazuo Gao1, Deli Chen1, Jiashi Li1, Wangding Zeng1, Xingkai Yu∗1,4, Y. Wu1, Zhenda Xie1, Y.K. Li1, Panpan Huang1, Fuli Luo1, Chong Ruan1, Zhifang Sui2, Wenfeng Liang1 1DeepSeek-AI 2National Key Laboratory for Multimedia Information Processing, Peking University 3Institute for Interdisciplinary Information Sciences, Tsinghua University 4National Key Laboratory for Novel Software Technology, Nanjing University {daidamai, szf}@pku.edu.cn, {wenfeng.liang}@deepseek.com https://github.com/deepseek-ai/DeepSeek-MoE\n\nAbstract\n\nIn the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top- out of experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into ones and activating from them, allowing for a more flexible combination of activated experts; (2) isolating experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5× expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations.\n\nFurther, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\n1. Introduction\n\nRecent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023a). It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017) has emerged as a popular solution. It can\n\nenable parameter scaling, while concurrently keeping computational costs at a modest level.\n\nRecent applications of MoE architectures in Transformers (Vaswani et al., 2017) have yielded successful attempts at scaling language models to a substantial size (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022), accompanied with remarkable performance. These achievements underscore the considerable potential and promise of MoE language models.\n\nDespite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts.\n\nThis architecture manifests two potential issues: (1) Knowledge Hybridity: existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. (2) Knowledge Redundancy: tokens assigned to different experts may require common knowledge. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters.\n\nThese issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models.\n\nIn response to the aforementioned issues, we introduce DeepSeekMoE, an innovative MoE architecture specifically designed towards ultimate expert specialization. Our architecture involves two principal strategies: (1) Fine-Grained Expert Segmentation: while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. (2) Shared Expert Isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.\n\nThrough compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized.\n\nStarting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B (Lepikhin et al., 2021) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5× expert parameters and computation. Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE.\n\nThese studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization.\n\nLeveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B (DeepSeek-AI, 2024), a dense model trained on the same 2T corpus. We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B (Touvron et al., 2023b), which has approximately 2.5 times the activated parameters. Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard1. Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. The experimental results still validate its substantial advantages over the GShard architecture consistently. In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nOur contributions are summarized as follows: - Architectural Innovation. We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation.\n\nEmpirical Validation. We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. Experimental results validate the high 1https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models\n\nScalability. We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B.\n\nAlignment for MoE. We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B.\n\nPublic Release. In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization.\n\n2. Preliminaries: Mixture-Of-Experts For Transformers\n\nWe first introduce a generic MoE architecture commonly used in Transformer language models. A standard Transformer language model is constructed by stacking layers of standard Transformer blocks, where each block can be represented as follows:\n\n$$\\mathbf{u}{1:T}^{l}=\\text{Self-Att}\\left(\\mathbf{h}{1:T}^{l-1}\\right)+\\mathbf{h}{1:T}^{l-1},\\tag{1}$$ $$\\mathbf{h}{t}^{l}=\\text{FFN}\\left(\\mathbf{u}{t}^{l}\\right)+\\mathbf{u}{t}^{l},\\tag{2}$$\n\nwhere denotes the sequence length, Self-Att(·) denotes the self-attention module, FFN(·) denotes the Feed-Forward Network (FFN), u 1: ∈ R× are the hidden states of all tokens after the -th attention module, and h\n\n∈ Ris the output hidden state of the -th token after the -th Transformer block. For brevity, we omit the layer normalization in the above formulations.\n\nA typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022). An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. Then, each token will be assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts. If the -th FFN is substituted with an MoE layer, the computation for its output hidden state h\n\nis expressed as: where denotes the total number of experts, FFN(·) is the -th expert FFN, , denotes the gate value for the -th expert, , denotes the token-to-expert affinity, Topk(·, ) denotes the set comprising highest affinity scores among those calculated for the -th token and all experts, and e\n\nis the centroid of the -th expert in the -th layer. Note that ,is sparse, indicating that only out of gate values are nonzero. This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only experts. Also, in the above formulations, we omit the layer normalization operation for brevity.\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{N}\\left(g_{i,t}\\operatorname{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\operatorname{Topk}({s_{j,t}|1\\leqslant j\\leqslant N},K),\\ 0,&\\text{otherwise},\\end{cases}$$ $$s_{i,t}=\\operatorname{Softmax}{i}\\left(\\mathbf{u}{t}^{l}\\,\\mathbf{e}_{i}^{l}\\right),$$\n\n$$(3)$$\n\n(4) $\\binom{4}{5}$ .\n\n3. Deepseekmoe Architecture\n\nOn top of the generic MoE architecture outlined in Section 2, we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. As illustrated in Figure 2, our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. Both of these strategies are designed to elevate the level of expert specialization.\n\n3.1. Fine-Grained Expert Segmentation\n\nIn scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts.\n\nIn pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. The finer expert segmentation enables a more flexible and adaptable combination of activated experts. To be specific, on top of a typical MoE architecture shown in Figure 2(a), we segment each expert FFN into smaller experts by reducing the FFN intermediate hidden dimension to 1 times its original size. Since each expert becomes smaller, in response, we also increase the number of activated experts to times to keep the same computation cost, as illustrated in Figure 2(b). With the fine-grained expert segmentation, the output of an MoE layer can be expressed as:\n\n$$\\mathbf{h}{t}^{i}=\\sum{i=1}^{\\text{mN}}\\left(g_{i,t}\\text{FFN}{i}\\left(\\mathbf{u}{t}^{i}\\right)\\right)+\\mathbf{u}{t}^{i},\\tag{6}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\text{Tokr}(s_{i,t}|1\\leqslant j\\leqslant m\\text{N}),m\\text{K}),\\ 0,&\\text{otherwise},\\end{cases}$$ (7) $$s_{i,t}=\\text{Softmax}{i}\\left(\\mathbf{u}{t}^{T}\\mathbf{e}_{t}^{i}\\right),\\tag{8}$$ where the total number of expert parameters is equal to $N$ times the number of parameters in a\n\n$$(6)$$\n\n(7) (8) $\\frac{1}{2}$\n\nstandard FFN, and denotes the total number of fine-grained experts. With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to .\n\nFrom a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. As an illustrative example, we consider the case where = 16. A typical top-2 routing strategy can yield 16 2 = 120 possible combinations. By contrast, if each expert is split into 4 smaller experts, the fine-grained routing strategy can yield 64 8 = 4, 426, 165, 368 potential combinations. The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition.\n\n3.2. Shared Expert Isolation\n\nWith a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. This alleviation of redundancy will contribute to a more parameterefficient model with more specialized experts.\n\nTowards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate experts to serve as shared experts. Regardless of the router module, each token will be deterministically assigned to these shared experts. In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by , as depicted in Figure 2(c). With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows:\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{K_{s}}\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)+\\sum_{i=K_{s}+1}^{mN}\\left(g_{i,t}\\;\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},\\tag{9}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\mathrm{Topk}{s_{j,t}|K_{s}+1\\leqslant j\\leqslant mN},mK-K_{s}},\\ 0,&\\text{otherwise},\\end{cases}$$ (10) $$s_{i,t}=\\mathrm{Softmax}{i}\\left(\\mathbf{u}{t}^{l,T}e_{i}^{l}\\right).\\tag{11}$$ $\\mathbf{h}\\mathbf{M}\\mathbf{E}$ the number of closed-compact is $K_{s}$ the total number of routed currents.\n\nFinally, in DeepSeekMoE, the number of shared expert is , the total number of routed experts is − , and the number of nonzero gates is − .\n\nIt is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022). The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint.\n\n3.3. Load Balance Consideration\n\nAutomatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. Firstly, there is a risk of routing collapse (Shazeer et al., 2017), i.e., the model always selects only a few experts, preventing other experts from sufficient training.\n\nSecondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.\n\nExpert-Level Balance Loss. In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. The computation of the balance loss is as follows:\n\n$$\\begin{array}{l}{{{\\mathcal L}{\\mathrm{ExpBal}}=\\alpha{1}\\sum_{i=1}^{N^{\\prime}}f_{i}P_{i},}}\\ {{f_{i}=\\frac{N^{\\prime}}{K^{T}}\\sum_{t=1}^{T}\\mathds{1}\\,(\\mathrm{Token}\\;t\\;\\mathrm{s}\\mathrm{lects}\\;\\mathrm{Expert}\\;i),}}\\ {{P_{i}=\\frac{1}{T}\\sum_{t=1}^{T}s_{i,t},}}\\end{array}$$\n\n(12) $$\\begin{array}{l}\\small\\text{(13)}\\end{array}$$ = (14) $$\\begin{array}{l}\\small\\text{(14)}\\end{array}$$ .\n\n7 where 1 is a hyper-parameter called expert-level balance factor, ′is equal to ( − ) and ′ is equal to ( − ) for brevity. 1(·) denotes the indicator function.\n\nDevice-Level Balance Loss. In addition to the expert-level balance loss, we introduce a devicelevel balance loss. When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. Instead, our primary objective is to ensure balanced computation across the devices. If we partition all routed experts into groups {E1, E2, ..., E}, and deploy each group on a single device, the device-level balance loss is computed as follows:\n\n$$\\mathcal{L}{\\text{DevBal}}=\\alpha{2}\\sum_{i=1}^{D}f_{i}^{\\prime}P_{i}^{\\prime},$$ $$f_{i}^{\\prime}=\\frac{1}{|\\mathcal{E}{i}|}\\sum{j\\in\\mathcal{E}{i}}f{j},$$ $$P_{i}^{\\prime}=\\sum_{j\\in\\mathcal{E}{i}}P{j},$$\n\n(15) $$\\begin{array}{l}\\small\\text{(16)}\\end{array}$$ = (17) .\n\nwhere 2 is a hyper-parameter called device-level balance factor. In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices.\n\n4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n6. Alignment For Deepseekmoe 16B\n\nPrevious research indicates that MoE models typically do not emerge significant gains from fine-tuning (Artetxe et al., 2022; Fedus et al., 2021). However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B.\n\n6.1. Experimental Setup\n\nTraining Data. For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios.\n\nHyper-Parameters. During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).\n\nWe employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10−5 without incorporating any learning rate scheduling strategy.\n\nEvaluation Benchmarks. For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3, with the following adjustments: (1) We exclude Pile (Gao et al., 2020) since chat models are seldom employed for pure language modeling. (2) We exclude\n\nMetric # Shot LLaMA2 DeepSeek DeepSeekMoE SFT 7B Chat 7B Chat 16B # Total Params N/A 6.7B 6.9B 16.4B # Activated Params N/A 6.7B 6.9B 2.8B FLOPs per 4K Tokens N/A 187.9T 183.5T 74.4T HellaSwag (Acc.) 0-shot 67.9 71.0 72.2 PIQA (Acc.) 0-shot 76.9 78.4 79.7 ARC-easy (Acc.) 0-shot 69.7 70.2 69.9 ARC-challenge (Acc.) 0-shot 50.8 50.2 50.0 BBH (EM) 3-shot 39.3 43.1 42.2 RACE-middle (Acc.) 5-shot 63.9 66.1 64.8 RACE-high (Acc.) 5-shot 49.6 50.8 50.6 DROP (EM) 1-shot 40.0 41.7 33.8 GSM8K (EM) 0-shot 63.4 62.6 62.2 MATH (EM) 4-shot 13.5 14.7 15.2 HumanEval (Pass@1) 0-shot 35.4 45.1 45.7 MBPP (Pass@1) 3-shot 27.8 39.0 46.2 TriviaQA (EM) 5-shot 60.1 59.5 63.3 NaturalQuestions (EM) 0-shot 35.2 32.7 35.1 MMLU (Acc.) 0-shot 50.0 49.7 47.2 WinoGrande (Acc.) 0-shot 65.1 68.4 69.0 CLUEWSC (EM) 5-shot 48.4 66.2 68.2 CEval (Acc.) 0-shot 35.1 44.7 40.0 CMMLU (Acc.) 0-shot 36.9 51.2 49.3\n\nCHID (Zheng et al., 2019) due to the observed instability of results, hindering the derivation of solid conclusions. (3) We additionally include BBH (Suzgun et al., 2022) to provide a more comprehensive assessment of the reasoning ability of the chat models.\n\nTable 5 | Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations.\n\n6.2. Evaluations\n\nBaselines. In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. Correspondingly, we construct three chat models, including LLaMA2 SFT 7B3, DeepSeek Chat 7B, and DeepSeekMoE Chat 16B.\n\nSubsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks.\n\nResults. The evaluation results are presented in Table 5. Our key observations include: (1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intensive tasks (TriviaQA, NaturalQuestions). (2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. In addition, it also surpasses DeepSeek Chat 7B. (3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1). However, it is worth noting that, after supervised finetuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. (4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations.\n\n7. Deepseekmoe 145B Ongoing\n\nEncouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available.\n\n7.1. Experimental Setup\n\nTraining Data and Tokenization. For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study.\n\nModel Settings. For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps.\n\nSubsequently, the learning rate keeps constant during the remaining training process. The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. Also, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse.\n\nEvaluation Benchmarks. We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3).\n\n7.2. Evaluations\n\nBaselines. Apart from DeepSeekMoE 145B, we consider three additional models for comparison. DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters.\n\nResults. From the evaluation results presented in Table 6, we have the following observations: (1) Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. (2) On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. (3) At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5.\n\n8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\nAppendices A. Overview Of Hyper-Parameters\n\nWe present the overview of hyper-parameters for DeepSeekMoE across various sizes in Table 7.\n\n# Params # Layers Hidden # Attn # Shared # Routed Relative Sequence Batch Size Learning Size Heads Experts Experts Expert Size Length (Sequence) Rate 2.0B 9 1280 10 1 63 (7 activated) 0.25 2048 2048 1.08e-3 16.4B 28 2048 16 2 64 (6 activated) 0.25 4096 4608 4.2e-4 144.6B 62 4096 32 4 128 (12 activated) 0.125 4096 4608 3.0e-4\n\nTable 7 | Overview of hyper-parameters for DeepSeekMoE across various sizes. The relative expert size is in comparison to a standard FFN.\n\nB. Comparing Deepseekmoe With Larger Models\n\nComparisons among DeepSeekMoE, GShard×1.2, and GShard×1.5 are shown in Table 8. Comparisons among DeepSeekMoE, Dense×4, and Dense×16 are shown in Table 9.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 2.3B 2.8B 1.9B # Activated Expert Params N/A 0.28B 0.35B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.824 1.808 1.808 HellaSwag (Acc.) 0-shot 53.7 54.4 54.8 PIQA (Acc.) 0-shot 71.8 71.1 72.3 ARC-easy (Acc.) 0-shot 46.8 47.3 49.4 ARC-challenge (Acc.) 0-shot 31.7 34.1 34.3 RACE-middle (Acc.) 5-shot 43.7 46.4 44.0 RACE-high (Acc.) 5-shot 31.9 32.4 31.7 HumanEval (Pass@1) 0-shot 3.7 3.0 4.9 MBPP (Pass@1) 3-shot 2.4 2.6 2.2 TriviaQA (EM) 5-shot 15.2 15.7 16.6 NaturalQuestions (EM) 5-shot 4.5 4.7 5.7\n\nTable 8 | Comparison between DeepSeekMoE and larger GShard models.\n\nAt a larger scale of 13B total parameters, we also compare DeepSeekMoE with GShard×1.2 and GShard×1.5, and show results in Table 10. At a larger scale, DeepSeekMoE even outperforms GShard×1.5 distinctly.\n\nMetric # Shot Dense×4 Dense×16 DeepSeekMoE Relative Expert Size N/A 1 1 0.25 # Experts N/A 4 + 0 16 + 0 1 + 63 # Activated Experts N/A 4 + 0 16 + 0 1 + 7 # Total Expert Params N/A 0.47B 1.89B 1.89B # Activated Expert Params N/A 0.47B 1.89B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.908 1.806 1.808 HellaSwag (Acc.) 0-shot 47.6 55.1 54.8 PIQA (Acc.) 0-shot 70.0 71.9 72.3 ARC-easy (Acc.) 0-shot 43.9 51.9 49.4 ARC-challenge (Acc.) 0-shot 30.5 33.8 34.3 RACE-middle (Acc.) 5-shot 42.4 46.3 44.0 RACE-high (Acc.) 5-shot 30.7 33.0 31.7 HumanEval (Pass@1) 0-shot 1.8 4.3 4.9 MBPP (Pass@1) 3-shot 0.2 2.2 2.2 TriviaQA (EM) 5-shot 9.9 16.5 16.6 NaturalQuestions (EM) 5-shot 3.0 6.3 5.7\n\nTable 9 | Comparison between DeepSeekMoE and larger dense baselines.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 15.9B 19.8B 13.3B # Activated Expert Params N/A 2.37B 2.82B 2.05B # Training Tokens N/A 100B 100B 100B HellaSwag (Acc.) 0-shot 66.6 67.7 69.1 PIQA (Acc.) 0-shot 75.6 76.0 75.7 ARC-easy (Acc.) 0-shot 56.8 56.8 58.8 ARC-challenge (Acc.) 0-shot 39.9 37.6 38.5 RACE-middle (Acc.) 5-shot 51.6 50.6 52.4 RACE-high (Acc.) 5-shot 37.4 36.3 38.5 HumanEval (Pass@1) 0-shot 6.1 6.1 9.8 MBPP (Pass@1) 3-shot 7.0 11.6 10.6 TriviaQA (EM) 5-shot 36.5 36.7 38.2 NaturalQuestions (EM) 5-shot 12.6 12.1 13.7\n\nTable 10 | Comparison between DeepSeekMoE and larger GShard models at a larger scale.\n\nC. Training Benchmark Curves Of Deepseekmoe 16B\n\nWe present the benchmark curves during training of DeepSeekMoE 16B and DeepSeek 7B (Dense) in Figure 7 for reference.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/DeepSeekMoE_2.md"
+          },
+          "summary": "DeepSeekMoE is an innovative Mixture-of-Experts (MoE) architecture designed to enhance expert specialization in language models. It employs fine-grained expert segmentation and shared expert isolation to improve performance and reduce computational costs. Starting with a model of 2B parameters, DeepSeekMoE demonstrates comparable performance to larger models like GShard 2.9B and LLaMA2 7B, using significantly fewer computations. The architecture scales up to 16B and 145B parameters, maintaining advantages over traditional MoE architectures. DeepSeekMoE 16B achieves similar results to dense models with only 40% of the computations, and its chat model version shows adaptability in bilingual scenarios. The architecture's public release aims to contribute to advancements in large-scale language models.",
+          "headlines": {
+            "1. Introduction": [],
+            "2. Preliminaries: Mixture-Of-Experts For Transformers": [],
+            "3. Deepseekmoe Architecture": [
+              "3.1. Fine-Grained Expert Segmentation",
+              "3.2. Shared Expert Isolation",
+              "3.3. Load Balance Consideration"
+            ],
+            "4. Validation Experiments": [
+              "4.1. Experimental Setup",
+              "4.1.1. Training Data And Tokenization",
+              "4.1.2. Infrastructures",
+              "4.1.3. Hyper-Parameters",
+              "4.1.4. Evaluation Benchmarks",
+              "4.2. Evaluations",
+              "4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models",
+              "4.4. Ablation Studies",
+              "4.5. Analysis On Expert Specialization"
+            ],
+            "5. Scaling Up To Deepseekmoe 16B": [
+              "5.1. Experimental Setup",
+              "5.1.1. Training Data And Tokenization",
+              "5.1.2. Hyper-Parameters",
+              "5.1.3. Evaluation Benchmarks",
+              "5.2. Evaluations",
+              "5.2.1. Internal Comparison With Deepseek 7B",
+              "5.2.2. Comparison With Open Source Models"
+            ],
+            "6. Alignment For Deepseekmoe 16B": [
+              "6.1. Experimental Setup",
+              "6.2. Evaluations"
+            ],
+            "7. Deepseekmoe 145B Ongoing": [
+              "7.1. Experimental Setup",
+              "7.2. Evaluations"
+            ],
+            "8. Related Work": [],
+            "9. Conclusion": [],
+            "Appendices": [
+              "A. Overview Of Hyper-Parameters",
+              "B. Comparing Deepseekmoe With Larger Models",
+              "C. Training Benchmark Curves Of Deepseekmoe 16B"
+            ]
+          },
+          "summary_embedding": [
+            -0.012465757317841053,
+            -0.022808987647294998,
+            -0.018792515620589256,
+            -0.02308654971420765,
+            0.01797616109251976,
+            0.007200260180979967,
+            0.006490030791610479,
+            0.042613785713911057,
+            -0.014049487188458443,
+            0.03187054023146629,
+            -0.021486490964889526,
+            -0.023592690005898476,
+            0.009045224636793137,
+            -0.014441338367760181,
+            0.005865518003702164,
+            -0.007363531738519669,
+            -0.008743173442780972,
+            -0.005163452122360468,
+            0.018841497600078583,
+            -0.03100520372390747,
+            0.011110605672001839,
+            -0.046075135469436646,
+            -0.020017050206661224,
+            -0.021845687180757523,
+            0.016425084322690964,
+            0.037389107048511505,
+            0.022547753527760506,
+            -0.0028062239289283752,
+            -0.06844329088926315,
+            0.034678805619478226,
+            0.035168617963790894,
+            0.02052319049835205,
+            0.0005438972148112953,
+            -0.00031302153365686536,
+            -0.024833550676703453,
+            -0.028556134551763535,
+            0.03347059711813927,
+            -0.004379750229418278,
+            -0.01453930139541626,
+            0.015976088121533394,
+            0.0414055772125721,
+            0.012106560170650482,
+            -0.02754385396838188,
+            0.0014245413476601243,
+            0.015151568688452244,
+            0.0055961208418011665,
+            -0.01678428053855896,
+            -0.0022960014175623655,
+            -0.002424577483907342,
+            -0.021225256845355034,
+            -0.01400050614029169,
+            0.018825169652700424,
+            0.001296985661610961,
+            -0.008865626528859138,
+            -0.0542386956512928,
+            0.007330877240747213,
+            0.03840138763189316,
+            0.05623060464859009,
+            0.044148534536361694,
+            -0.020180322229862213,
+            0.03369917720556259,
+            -0.030025575309991837,
+            -0.02501314878463745,
+            0.011028970591723919,
+            0.003983817528933287,
+            -0.018531281501054764,
+            0.01784554310142994,
+            -0.005636938381940126,
+            -0.0006556359585374594,
+            0.01751900091767311,
+            -0.03941367194056511,
+            0.020751770585775375,
+            -0.01887415163218975,
+            0.031266435980796814,
+            0.02173139713704586,
+            0.0073798587545752525,
+            0.01872720755636692,
+            0.01678428053855896,
+            0.0036858476232737303,
+            -0.04107903689146042,
+            0.009118696674704552,
+            0.016523046419024467,
+            0.014196431264281273,
+            0.016269976273179054,
+            0.06553706526756287,
+            -0.05391215533018112,
+            -0.004485876299440861,
+            -0.05169166624546051,
+            0.0012561678886413574,
+            -0.004971608519554138,
+            -0.03670336678624153,
+            0.01048201136291027,
+            0.028425518423318863,
+            0.021372200921177864,
+            -0.01063711941242218,
+            0.024392718449234962,
+            -0.022743679583072662,
+            0.0032001156359910965,
+            -0.04062187671661377,
+            0.0019653772469609976,
+            0.004518530797213316,
+            0.016531210392713547,
+            0.0070778070949018,
+            0.004273623693734407,
+            -0.016702644526958466,
+            0.006045116577297449,
+            -0.01905374974012375,
+            0.03291547670960426,
+            -0.008465612307190895,
+            0.08653374016284943,
+            0.008751336485147476,
+            0.004738946910947561,
+            0.004281787667423487,
+            0.02398454025387764,
+            0.012131051160395145,
+            -0.00810233410447836,
+            -0.04718538001179695,
+            -0.015690363943576813,
+            0.011061624623835087,
+            -0.0383034273982048,
+            0.018335357308387756,
+            -0.008759500458836555,
+            0.01282495353370905,
+            -0.04107903689146042,
+            -0.06344719231128693,
+            0.01498013362288475,
+            0.001784758409485221,
+            -0.004065453074872494,
+            -0.0002903166168835014,
+            -0.018923133611679077,
+            0.003930754493921995,
+            -0.004200151655822992,
+            -0.010645282454788685,
+            -0.021502818912267685,
+            0.03159298002719879,
+            0.03657275065779686,
+            0.06527583301067352,
+            0.036801330745220184,
+            -0.0030164355412125587,
+            0.005775718949735165,
+            0.019690508022904396,
+            -0.020784424617886543,
+            0.003481758525595069,
+            -0.01214737817645073,
+            -0.021992631256580353,
+            -0.03820546343922615,
+            0.010971825569868088,
+            0.07379858940839767,
+            -0.029029620811343193,
+            0.013437220826745033,
+            -0.017616963014006615,
+            0.04205866530537605,
+            -0.004567512311041355,
+            0.026727497577667236,
+            -0.016669990494847298,
+            0.004059330560266972,
+            -0.025388672947883606,
+            0.01620466820895672,
+            -0.024964168667793274,
+            0.012571883387863636,
+            -0.000885746325366199,
+            -0.04222193360328674,
+            0.00011116473615402356,
+            0.026890767738223076,
+            -0.010971825569868088,
+            0.02261306345462799,
+            0.0004949158756062388,
+            -0.006302268709987402,
+            -0.05097327008843422,
+            0.038727931678295135,
+            -0.003912386484444141,
+            0.018645571544766426,
+            -0.0018959869630634785,
+            0.0163107942789793,
+            -0.0016408757073804736,
+            0.013216804713010788,
+            0.009543201886117458,
+            0.007114543113857508,
+            -0.033503253012895584,
+            0.02697240374982357,
+            -0.02504580281674862,
+            0.028049994260072708,
+            0.03435226157307625,
+            -0.0030184765346348286,
+            0.03255628049373627,
+            0.019249675795435905,
+            0.03840138763189316,
+            0.0327032245695591,
+            0.030335791409015656,
+            -0.00824927818030119,
+            0.013878053054213524,
+            0.008200296200811863,
+            0.03608293831348419,
+            0.013429056853055954,
+            0.019723162055015564,
+            -0.009730963967740536,
+            -0.014041324146091938,
+            0.048785436898469925,
+            0.007318632211536169,
+            -0.0179598331451416,
+            -0.005441013257950544,
+            -0.03510330989956856,
+            -0.002832755446434021,
+            -0.014425011351704597,
+            0.0323440283536911,
+            0.02158445306122303,
+            -0.01647406443953514,
+            0.035135965794324875,
+            0.024980494752526283,
+            -0.03751972317695618,
+            -0.005232842173427343,
+            -0.02026195637881756,
+            -0.023298801854252815,
+            -0.04215662553906441,
+            0.011624909937381744,
+            0.012833117507398129,
+            -0.013976015150547028,
+            -0.007469657808542252,
+            0.020457882434129715,
+            -0.023625344038009644,
+            -0.0018265966791659594,
+            -0.01356783788651228,
+            0.006547175347805023,
+            -0.020898714661598206,
+            -0.03667071461677551,
+            -0.01048201136291027,
+            0.025862159207463264,
+            0.012637191452085972,
+            0.04362606629729271,
+            -0.013518855907022953,
+            -0.005367541220039129,
+            0.033193036913871765,
+            0.012759645469486713,
+            0.010767736472189426,
+            0.03699725493788719,
+            -0.03027048148214817,
+            0.024621298536658287,
+            -0.020294612273573875,
+            -0.0032878739293664694,
+            0.01707816869020462,
+            -0.0030062312725931406,
+            0.01962519995868206,
+            -0.04535674303770065,
+            -0.024196794256567955,
+            0.04395261034369469,
+            -0.0007107399869710207,
+            0.016302630305290222,
+            -0.005959399044513702,
+            -0.012106560170650482,
+            0.0035899258218705654,
+            0.0027715289033949375,
+            -0.01438419334590435,
+            -0.008620720356702805,
+            0.007473739795386791,
+            -0.01962519995868206,
+            0.03660540655255318,
+            0.03794422745704651,
+            -0.004132802598178387,
+            -0.014131123200058937,
+            -0.03324201703071594,
+            0.010016689077019691,
+            -0.017878197133541107,
+            -0.023951886221766472,
+            0.04078514873981476,
+            -0.032115448266267776,
+            -0.00029312283731997013,
+            0.039478980004787445,
+            0.018808843567967415,
+            -0.014898497611284256,
+            0.00868602842092514,
+            -0.01709449477493763,
+            -0.026302991434931755,
+            -0.017159804701805115,
+            0.03356856107711792,
+            0.024800896644592285,
+            -0.010743245482444763,
+            0.010245268233120441,
+            0.005330805201083422,
+            0.02893165871500969,
+            0.024376392364501953,
+            0.00648594880476594,
+            -0.013812744058668613,
+            0.015029114671051502,
+            0.027641816064715385,
+            -0.0028021421749144793,
+            -7.66609373386018e-05,
+            -0.004010349046438932,
+            -0.007796200457960367,
+            -0.006453294772654772,
+            -0.013069860637187958,
+            -0.005089980084449053,
+            0.0009184005321003497,
+            -0.032295044511556625,
+            0.015967924147844315,
+            -0.005890008993446827,
+            0.010359558276832104,
+            -0.011535110883414745,
+            0.01288209855556488,
+            0.01722511276602745,
+            -0.013494365848600864,
+            -0.0716434046626091,
+            0.014612773433327675,
+            0.02458864450454712,
+            -0.006706364918500185,
+            -0.017943505197763443,
+            0.008481939323246479,
+            0.00016148542636074126,
+            0.038597315549850464,
+            -0.014180104248225689,
+            -0.031103165820240974,
+            0.02680913358926773,
+            -0.017437364906072617,
+            -0.015241367742419243,
+            -0.010661610402166843,
+            -0.00580837344750762,
+            0.04649963974952698,
+            0.01889047957956791,
+            0.011420820839703083,
+            -0.04042595252394676,
+            -0.007449248805642128,
+            0.015078096650540829,
+            0.014506647363305092,
+            -0.04705476388335228,
+            0.0052083516493439674,
+            0.015486273914575577,
+            0.019723162055015564,
+            0.024800896644592285,
+            0.04274440184235573,
+            -0.0021531390957534313,
+            0.01661284640431404,
+            -0.015053605660796165,
+            -0.05136512219905853,
+            -0.03340528905391693,
+            0.029943939298391342,
+            -0.017731253057718277,
+            -0.02906227484345436,
+            -0.01693122461438179,
+            -0.006151242647320032,
+            0.0017939424142241478,
+            0.007126788143068552,
+            -0.01916803978383541,
+            -0.03464614972472191,
+            -0.03970756009221077,
+            0.02277633361518383,
+            0.007983962073922157,
+            0.004269542172551155,
+            -0.042777057737112045,
+            0.0013459670590236783,
+            0.009184005670249462,
+            0.012073906138539314,
+            0.0009954441338777542,
+            -0.019984396174550056,
+            0.015494437888264656,
+            0.0032001156359910965,
+            -0.031070511788129807,
+            -0.035593122243881226,
+            -0.03595232218503952,
+            0.0023919232189655304,
+            0.003781769424676895,
+            0.06279410421848297,
+            0.012988224625587463,
+            -0.03840138763189316,
+            -0.04522612690925598,
+            0.029176564887166023,
+            -0.02052319049835205,
+            -0.016678154468536377,
+            0.0059430720284581184,
+            0.012898425571620464,
+            -0.009877908043563366,
+            0.013192313723266125,
+            0.023560035973787308,
+            0.041601505130529404,
+            -0.005987971555441618,
+            0.012914752587676048,
+            0.023037567734718323,
+            0.012955570593476295,
+            0.0362788625061512,
+            -0.00936360377818346,
+            0.0035409443080425262,
+            0.039936136454343796,
+            -0.012269831262528896,
+            0.03175625205039978,
+            -0.02189466916024685,
+            0.0020398697815835476,
+            0.02249877341091633,
+            0.027184655889868736,
+            -0.0006479826406575739,
+            0.02594379521906376,
+            -0.031543996185064316,
+            0.018368011340498924,
+            -0.014498483389616013,
+            -0.04888340085744858,
+            -0.01557607389986515,
+            0.013102514669299126,
+            0.030384771525859833,
+            0.02756018005311489,
+            -0.0041348435916006565,
+            -0.019723162055015564,
+            -0.01155143789947033,
+            -0.010596301406621933,
+            0.0018419033149257302,
+            0.03285016864538193,
+            0.010506502352654934,
+            0.010890189558267593,
+            -0.011935125105082989,
+            -0.004104230087250471,
+            0.03025415539741516,
+            -0.021192602813243866,
+            0.01650671847164631,
+            0.0041511706076562405,
+            -0.002029665280133486,
+            -0.04320156201720238,
+            -0.035266581922769547,
+            0.007845181971788406,
+            0.011747363954782486,
+            0.014874007552862167,
+            0.0368339866399765,
+            0.025094784796237946,
+            -0.01662100851535797,
+            0.030629679560661316,
+            -0.01905374974012375,
+            0.027984686195850372,
+            0.039054471999406815,
+            0.011624909937381744,
+            -0.018351683393120766,
+            0.012237177230417728,
+            0.002424577483907342,
+            -0.025127438828349113,
+            0.01680060662329197,
+            -0.032735876739025116,
+            0.01738838478922844,
+            -0.02354370802640915,
+            0.025143766775727272,
+            -0.02579685114324093,
+            -0.009290131740272045,
+            -0.004428731743246317,
+            -0.006130834110081196,
+            0.04888340085744858,
+            -0.026727497577667236,
+            -0.0142454132437706,
+            -0.0024694770108908415,
+            -0.0024531499948352575,
+            -0.008718682453036308,
+            0.010000362060964108,
+            -0.045781247317790985,
+            -0.01844964735209942,
+            -0.04016471654176712,
+            0.007355368230491877,
+            0.005412440747022629,
+            -0.030907239764928818,
+            -0.05694899708032608,
+            0.025421326979994774,
+            -0.06331657618284225,
+            -0.019886434078216553,
+            0.051626358181238174,
+            -0.018400665372610092,
+            -0.008212542161345482,
+            0.009641164913773537,
+            0.006375740747898817,
+            -0.029290854930877686,
+            0.0041348435916006565,
+            0.02504580281674862,
+            0.04418119043111801,
+            -0.0045797573402523994,
+            -0.02865409664809704,
+            0.0028674507047981024,
+            -0.027788760140538216,
+            -0.013461710885167122,
+            0.005261414684355259,
+            0.016449574381113052,
+            0.024539662525057793,
+            0.007628847379237413,
+            0.03585435822606087,
+            0.007294141221791506,
+            0.006318595726042986,
+            0.044279150664806366,
+            -0.025666235014796257,
+            -0.027380581945180893,
+            0.02519274689257145,
+            0.04045860469341278,
+            0.023151857778429985,
+            -0.018253721296787262,
+            -0.0142454132437706,
+            0.0053063142113387585,
+            0.0009296254720538855,
+            -0.022139575332403183,
+            0.002808264922350645,
+            -0.013902543112635612,
+            0.005143043119460344,
+            0.03892385587096214,
+            -0.001205145614221692,
+            0.014196431264281273,
+            0.007302304729819298,
+            0.006845145486295223,
+            0.004334850702434778,
+            -0.011731036007404327,
+            -0.023592690005898476,
+            0.00032220553839579225,
+            0.034972693771123886,
+            0.015404638834297657,
+            -0.030515389516949654,
+            -0.00995954405516386,
+            0.034711457788944244,
+            -0.00795130804181099,
+            -0.009118696674704552,
+            -0.009771781973540783,
+            0.003385836724191904,
+            0.02638462744653225,
+            -0.01606588624417782,
+            0.006649219896644354,
+            0.03716052696108818,
+            -0.019870106130838394,
+            0.005543057806789875,
+            -0.025405000895261765,
+            0.0018061877926811576,
+            0.02442537248134613,
+            -0.006216551177203655,
+            0.028098974376916885,
+            5.6315813708351925e-05,
+            0.03025415539741516,
+            -0.004677720367908478,
+            0.008432958275079727,
+            -0.020327266305685043,
+            0.03987082839012146,
+            -0.017633290961384773,
+            0.03402572125196457,
+            0.02067013457417488,
+            -0.01258004643023014,
+            0.04783846437931061,
+            -0.036801330745220184,
+            -0.020784424617886543,
+            0.00584919098764658,
+            0.012743317522108555,
+            0.0028919412288814783,
+            -0.0191517136991024,
+            -0.018400665372610092,
+            -0.010996315628290176,
+            0.02380494214594364,
+            -0.022547753527760506,
+            -0.016865916550159454,
+            -0.007441085297614336,
+            -0.01890680566430092,
+            -0.009673818945884705,
+            -0.007988044060766697,
+            -0.01709449477493763,
+            -0.016547536477446556,
+            0.044573038816452026,
+            -0.06334923207759857,
+            0.016996532678604126,
+            0.017159804701805115,
+            0.011477965861558914,
+            -0.034678805619478226,
+            -0.019510909914970398,
+            -0.017633290961384773,
+            -0.002347023691982031,
+            -0.030743969604372978,
+            0.001702102366834879,
+            0.01725776679813862,
+            0.01138000376522541,
+            -0.05022222548723221,
+            0.012335140258073807,
+            0.02772345207631588,
+            -0.012065742164850235,
+            0.0014949520118534565,
+            -0.023135529831051826,
+            -0.017176130786538124,
+            0.0031776658724993467,
+            0.022123249247670174,
+            0.005975726060569286,
+            -0.008188051171600819,
+            -0.016865916550159454,
+            -0.005241005681455135,
+            -0.006045116577297449,
+            -0.002359268954023719,
+            0.005094061605632305,
+            0.011780017986893654,
+            0.019690508022904396,
+            -0.0024633544962853193,
+            -0.004436895251274109,
+            -0.006396149750798941,
+            0.027364253997802734,
+            -0.001450052484869957,
+            -0.008416631259024143,
+            0.050907962024211884,
+            0.03556047007441521,
+            0.006094098091125488,
+            -0.01108611561357975,
+            -0.03990348428487778,
+            0.010661610402166843,
+            -0.022711025550961494,
+            0.0401320643723011,
+            0.01905374974012375,
+            -0.008298259228467941,
+            -0.028866350650787354,
+            0.010294250212609768,
+            -0.02290695160627365,
+            -0.009975871071219444,
+            0.009469729848206043,
+            0.00726556871086359,
+            -0.03263791650533676,
+            -0.013543346896767616,
+            0.03510330989956856,
+            0.011437147855758667,
+            -0.006841063499450684,
+            -0.036050282418727875,
+            0.008588065393269062,
+            0.01646590232849121,
+            0.011869817040860653,
+            -0.019233347848057747,
+            0.004449140280485153,
+            -0.012041252106428146,
+            0.017470018938183784,
+            0.008849299512803555,
+            -0.017796561121940613,
+            0.018237393349409103,
+            0.004710374400019646,
+            -0.008490103296935558,
+            -0.003912386484444141,
+            -0.0163107942789793,
+            -0.02006603218615055,
+            0.03840138763189316,
+            0.024490682408213615,
+            0.026335645467042923,
+            0.005457340274006128,
+            0.02008235827088356,
+            0.03227871656417847,
+            0.011282040737569332,
+            -0.03134807199239731,
+            0.027935704216361046,
+            0.016996532678604126,
+            -0.04604247957468033,
+            0.021551799029111862,
+            -0.02218855731189251,
+            -0.028670424595475197,
+            0.009649328887462616,
+            -0.0017367975087836385,
+            0.011192241683602333,
+            0.0034837995190173388,
+            -0.003642988856881857,
+            -0.03307874873280525,
+            -0.006236960180103779,
+            0.015771998092532158,
+            0.015502601861953735,
+            -0.0048695639707148075,
+            -0.003296037670224905,
+            -0.011886144056916237,
+            0.00487364549189806,
+            0.009796272963285446,
+            0.039054471999406815,
+            0.001381682581268251,
+            -0.004751192405819893,
+            -0.0018765984568744898,
+            0.031854212284088135,
+            0.0271193478256464,
+            -0.017274094745516777,
+            -0.026286665350198746,
+            0.00580837344750762,
+            -0.014253576286137104,
+            0.026156047359108925,
+            -0.00487364549189806,
+            0.011877980083227158,
+            0.008718682453036308,
+            -0.0007908449042588472,
+            -0.001373519073240459,
+            0.025584599003195763,
+            -0.02710302174091339,
+            -0.021764053031802177,
+            0.031233781948685646,
+            -0.025731543079018593,
+            0.028735732659697533,
+            0.013061696663498878,
+            -0.01557607389986515,
+            -0.015225040726363659,
+            -0.010302413254976273,
+            0.009943217039108276,
+            -0.012277995236217976,
+            0.023853924125432968,
+            0.003608293831348419,
+            0.006392067763954401,
+            0.0003584313380997628,
+            0.02770712412893772,
+            0.024833550676703453,
+            0.007820690982043743,
+            -0.018074123188853264,
+            -0.0016816934803500772,
+            -0.02320083975791931,
+            -0.01680060662329197,
+            0.023772288113832474,
+            -0.018090449273586273,
+            -0.03268689662218094,
+            -0.028556134551763535,
+            0.012114724144339561,
+            0.007392104249447584,
+            0.010816717520356178,
+            0.03461349755525589,
+            -0.030384771525859833,
+            -0.0344502255320549,
+            0.01766594499349594,
+            0.026270337402820587,
+            -0.005551221314817667,
+            -0.021535472944378853,
+            0.03085825778543949,
+            -0.005828781984746456,
+            0.017600636929273605,
+            0.0404912605881691,
+            -0.0016225076979026198,
+            0.010228941217064857,
+            -0.042026009410619736,
+            -0.014057651162147522,
+            -0.0066614653915166855,
+            0.006261450704187155,
+            0.034417569637298584,
+            0.01048201136291027,
+            0.01535565685480833,
+            0.03448288142681122,
+            -0.021192602813243866,
+            -0.0002903166168835014,
+            -0.0022449791431427,
+            -0.02818061038851738,
+            0.03347059711813927,
+            0.025715215131640434,
+            0.011918798089027405,
+            0.019706835970282555,
+            -0.03794422745704651,
+            -0.013037205673754215,
+            0.027070365846157074,
+            0.015053605660796165,
+            0.01273515447974205,
+            0.016694480553269386,
+            0.011249386705458164,
+            0.00868602842092514,
+            -0.021633435040712357,
+            -0.05169166624546051,
+            0.01214737817645073,
+            0.01993541419506073,
+            -0.016449574381113052,
+            0.01057181041687727,
+            0.024980494752526283,
+            0.007620683871209621,
+            -0.007045152597129345,
+            0.012808626517653465,
+            -0.02682545967400074,
+            0.002259265398606658,
+            0.0019010890973731875,
+            -0.014482156373560429,
+            0.038270771503448486,
+            -0.016367938369512558,
+            0.011363675817847252,
+            -0.018204739317297935,
+            0.023870250210165977,
+            0.00011531029304023832,
+            0.0007964573451317847,
+            0.014963806606829166,
+            -0.011347348801791668,
+            -0.04235255345702171,
+            0.0021613026037812233,
+            0.03342161700129509,
+            -0.015200549736618996,
+            0.026449935510754585,
+            -0.0020674217958003283,
+            0.032458316534757614,
+            0.042777057737112045,
+            0.017731253057718277,
+            0.0003497575526125729,
+            0.025682561099529266,
+            0.0036144163459539413,
+            0.0073880222626030445,
+            0.012400448322296143,
+            0.009322785772383213,
+            0.007245160173624754,
+            0.030809277668595314,
+            -0.010261595249176025,
+            0.02173139713704586,
+            -0.04760988429188728,
+            -0.014629100449383259,
+            0.012024925090372562,
+            -0.0013316808035597205,
+            -0.012449429370462894,
+            -0.033029764890670776,
+            0.012563719414174557,
+            -0.03503800183534622,
+            0.022074267268180847,
+            -0.0004959363141097128,
+            0.04333217814564705,
+            -0.02187834307551384,
+            -0.005518566817045212,
+            0.016694480553269386,
+            0.0016510800924152136,
+            0.023282473906874657,
+            -0.0012153501156717539,
+            -0.016416920349001884,
+            -0.0067798369564116,
+            -0.009257477708160877,
+            0.004138925112783909,
+            0.04375668615102768,
+            -0.003722583642229438,
+            -0.0001682458823779598,
+            0.01409030519425869,
+            -0.011959616094827652,
+            -0.03250729665160179,
+            0.045193471014499664,
+            -0.0030837850645184517,
+            0.02484987862408161,
+            0.04979771748185158,
+            -0.06410028040409088,
+            -0.019543563947081566,
+            0.013061696663498878,
+            -0.010441194288432598,
+            0.011363675817847252,
+            0.005547139327973127,
+            0.016114868223667145,
+            0.017894525080919266,
+            -0.005485912784934044,
+            0.01619650423526764,
+            0.005106307100504637,
+            0.028098974376916885,
+            -0.02336410991847515,
+            -0.03296445682644844,
+            -0.02636830136179924,
+            -0.017747581005096436,
+            0.0014480114914476871,
+            -0.015796490013599396,
+            -3.563585551091819e-06,
+            -0.0022204883862286806,
+            -0.04388730227947235,
+            -0.025274382904171944,
+            0.040687184780836105,
+            0.006706364918500185,
+            0.02458864450454712,
+            0.0077268099412322044,
+            0.03418898954987526,
+            0.005885927006602287,
+            -0.009037061594426632,
+            0.030058229342103004,
+            -0.020049704238772392,
+            0.03833607956767082,
+            0.022106921300292015,
+            -0.002832755446434021,
+            0.005865518003702164,
+            0.031086837872862816,
+            -0.033797141164541245,
+            0.008086007088422775,
+            -0.034384917467832565,
+            0.0046695563942193985,
+            -0.008620720356702805,
+            -0.015094423666596413,
+            -0.009526874870061874,
+            -0.02504580281674862,
+            0.011216731742024422,
+            -0.021535472944378853,
+            0.01590261608362198,
+            0.010416703298687935,
+            -0.002916431985795498,
+            0.03448288142681122,
+            -0.014204595237970352,
+            -0.0027796924114227295,
+            0.0024531499948352575,
+            -0.027070365846157074,
+            -0.006343086715787649,
+            -0.003377673216164112,
+            -0.03689929470419884,
+            -0.004502203781157732,
+            0.010988152585923672,
+            -0.00024184548237826675,
+            0.016898570582270622,
+            0.02787039615213871,
+            0.0005092020728625357,
+            -0.022302847355604172,
+            0.002340900944545865,
+            -0.04199335351586342,
+            0.0011877981014549732,
+            0.01483318954706192,
+            -0.017600636929273605,
+            -0.008612556383013725,
+            0.033046092838048935,
+            0.023298801854252815,
+            -0.009396257810294628,
+            -0.035005345940589905,
+            0.017796561121940613,
+            -0.011853490024805069,
+            0.027511198073625565,
+            -0.0002630197268445045,
+            0.019298657774925232,
+            -0.0076941559091210365,
+            0.04493223875761032,
+            0.001369437319226563,
+            0.008645210415124893,
+            -0.021959977224469185,
+            0.007020662073045969,
+            -0.020278284326195717,
+            -0.002406209474429488,
+            -0.006171651650220156,
+            -0.012751481495797634,
+            -0.010669773444533348,
+            0.0032735876739025116,
+            0.007730891928076744,
+            -0.0176822729408741,
+            0.0055961208418011665,
+            -0.015959760174155235,
+            -0.02741323597729206,
+            0.007543129846453667,
+            -0.05965929850935936,
+            -0.0179598331451416,
+            -0.03373183310031891,
+            -0.009477893821895123,
+            0.02008235827088356,
+            0.03510330989956856,
+            -0.012759645469486713,
+            0.00497977202758193,
+            -0.014898497611284256,
+            0.004477712791413069,
+            0.03490738570690155,
+            0.044442422688007355,
+            -0.014759717509150505,
+            0.003381754970178008,
+            -0.010955498553812504,
+            0.033029764890670776,
+            -0.03347059711813927,
+            0.0030062312725931406,
+            0.031086837872862816,
+            -0.005057325586676598,
+            0.007959471084177494,
+            0.011624909937381744,
+            0.020474210381507874,
+            -0.019396619871258736,
+            0.00877582747489214,
+            0.002100075827911496,
+            0.019233347848057747,
+            0.02171507105231285,
+            0.020343592390418053,
+            -0.03539719805121422,
+            0.028409190475940704,
+            0.018384337425231934,
+            0.02444170042872429,
+            0.044442422688007355,
+            -0.03974021226167679,
+            -0.004147088620811701,
+            -0.0032225653994828463,
+            -0.02997659333050251,
+            0.02008235827088356,
+            -0.007530884351581335,
+            -0.01841699331998825,
+            -0.007069643586874008,
+            0.03327467292547226,
+            0.014571955427527428,
+            -0.05051611363887787,
+            0.01980479806661606,
+            0.01740471087396145,
+            0.02055584453046322,
+            0.024474354460835457,
+            -0.01123305968940258,
+            -0.0019072118448093534,
+            0.008996243588626385,
+            0.006804327480494976,
+            0.019266001880168915,
+            0.015257694758474827,
+            0.002012317767366767,
+            -0.0022082431241869926,
+            -0.011837163008749485,
+            0.005224678665399551,
+            -0.009077879600226879,
+            0.010808554477989674,
+            -0.02458864450454712,
+            -0.009698309935629368,
+            0.021388528868556023,
+            0.0031164393294602633,
+            -0.013665799982845783,
+            -0.03729114308953285,
+            -0.045944519340991974,
+            0.0033837957307696342,
+            0.004543021321296692,
+            -0.05365091934800148,
+            0.005322641227394342,
+            -0.026547899469733238,
+            -0.002108239568769932,
+            -0.005502239800989628,
+            0.0019214981002733111,
+            0.005636938381940126,
+            0.0167679525911808,
+            -0.013494365848600864,
+            0.02248244546353817,
+            -0.015502601861953735,
+            0.03165828809142113,
+            0.0055961208418011665,
+            0.007375776767730713,
+            -0.0027796924114227295,
+            0.05590406060218811,
+            -0.003712379140779376,
+            -0.03699725493788719,
+            0.023282473906874657,
+            -0.01311067771166563,
+            -0.0001747512142173946,
+            0.030090883374214172,
+            0.00884113647043705,
+            -0.015257694758474827,
+            0.013437220826745033,
+            -0.030613351613283157,
+            -0.03249097242951393,
+            0.010996315628290176,
+            -0.013755599036812782,
+            -0.0015776080545037985,
+            -0.04568328335881233,
+            -0.009339112788438797,
+            0.004269542172551155,
+            -0.018335357308387756,
+            0.010775899514555931,
+            0.01631895825266838,
+            0.013992343097925186,
+            -0.024245774373412132,
+            0.009053388610482216,
+            0.006865554489195347,
+            0.013453547842800617,
+            0.0007331897504627705,
+            0.014049487188458443,
+            -0.008677864447236061,
+            -0.011306531727313995,
+            -0.018319029361009598,
+            0.0032450153958052397,
+            -0.026792805641889572,
+            0.005155288614332676,
+            -0.003155216109007597,
+            -0.03771565109491348,
+            -0.06521052122116089,
+            -0.0344502255320549,
+            0.016294466331601143,
+            0.03396041318774223,
+            -0.009045224636793137,
+            0.0035144127905368805,
+            -0.00023648813657928258,
+            -0.01093917153775692,
+            0.024523336440324783,
+            -0.005930826533585787,
+            -0.0004829256213270128,
+            0.0035389035474509,
+            0.02050686441361904,
+            -0.009722800925374031,
+            0.005845109466463327,
+            -0.051952898502349854,
+            -0.01586179807782173,
+            0.01812310516834259,
+            0.006306350696831942,
+            0.014629100449383259,
+            -0.009690145961940289,
+            0.01693122461438179,
+            -0.030074557289481163,
+            -0.028147956356406212,
+            0.04130761697888374,
+            0.008277850225567818,
+            0.011486129835247993,
+            -0.003055212553590536,
+            -0.01680060662329197,
+            0.02504580281674862,
+            -0.0020633398089557886,
+            -0.006461458280682564,
+            -0.00817988719791174,
+            0.015820980072021484,
+            0.044279150664806366,
+            0.003255219664424658,
+            0.0009821783751249313,
+            0.017045514658093452,
+            -0.002663361607119441,
+            -0.011575928889214993,
+            0.007677828427404165,
+            0.013592327944934368,
+            -0.016384266316890717,
+            0.010375885292887688,
+            -0.025862159207463264,
+            0.0332583449780941,
+            0.03131541982293129,
+            0.011575928889214993,
+            -0.015045441687107086,
+            0.006130834110081196,
+            0.005330805201083422,
+            0.008086007088422775,
+            -0.001369437319226563,
+            -0.0017133272485807538,
+            0.020719116553664207,
+            -0.03276853263378143,
+            -0.01693122461438179,
+            -0.005347132217139006,
+            0.003932795487344265,
+            0.005277741700410843,
+            0.03396041318774223,
+            -0.0033266509417444468,
+            0.017927179113030434,
+            -0.011657563969492912,
+            0.010579974390566349,
+            0.021764053031802177,
+            0.037095218896865845,
+            -0.024996822699904442,
+            -0.029862305149435997,
+            0.012914752587676048,
+            -0.001296985661610961,
+            0.014065815135836601,
+            0.0010173837654292583,
+            0.014139287173748016,
+            0.0029409227427095175,
+            0.060475654900074005,
+            -0.017763907089829445,
+            0.004628738854080439,
+            0.0004035860183648765,
+            -0.01858026348054409,
+            -0.0013531101867556572,
+            0.004261378664523363,
+            -0.010979988612234592,
+            0.03608293831348419,
+            0.002344982698559761,
+            0.039185091853141785,
+            0.013918871060013771,
+            0.009290131740272045,
+            0.0015143405180424452,
+            -0.013592327944934368,
+            0.008355404250323772,
+            0.0015592400450259447,
+            -0.001132694073021412,
+            0.009461566805839539,
+            0.019298657774925232,
+            0.009714636951684952,
+            0.025274382904171944,
+            0.009788108989596367,
+            0.012857607565820217,
+            -0.029470453038811684,
+            0.03510330989956856,
+            -0.0006224715034477413,
+            0.012996388599276543,
+            -0.02040890045464039,
+            -0.003136848099529743,
+            -0.0008240094175562263,
+            0.015649545937776566,
+            -0.015616890974342823,
+            -0.001201063860207796,
+            -0.01297189760953188,
+            0.040099408477544785,
+            -0.018988441675901413,
+            0.013976015150547028,
+            0.009673818945884705,
+            -0.02865409664809704,
+            -0.007596192881464958,
+            0.01995174214243889,
+            0.014294394291937351,
+            -0.0053144777193665504,
+            -0.020490536466240883,
+            0.005832863971590996,
+            0.015380147844552994,
+            -0.028572462499141693,
+            1.3584674888988957e-05,
+            0.014751553535461426,
+            -0.0164903923869133,
+            -0.00025409081717953086,
+            -0.004702210891991854,
+            0.003975654020905495,
+            -0.009265640750527382,
+            0.007028825581073761,
+            0.016057724133133888,
+            -0.024507008492946625,
+            -0.020049704238772392,
+            0.005722655914723873,
+            0.006571666337549686,
+            0.01444950234144926,
+            0.007739055436104536,
+            0.004355259705334902,
+            -0.020033378154039383,
+            -0.01799248717725277,
+            0.0032674651592969894,
+            -0.01649855636060238,
+            -0.0048614004626870155,
+            0.0020531355403363705,
+            -0.019363965839147568,
+            0.01875986158847809,
+            -0.010743245482444763,
+            0.00017538899555802345,
+            -0.003226647386327386,
+            -0.020800752565264702,
+            0.006204306147992611,
+            -0.004865481983870268,
+            0.019870106130838394,
+            0.003555230563506484,
+            0.02785406820476055,
+            0.02310287579894066,
+            -0.004428731743246317,
+            0.017600636929273605,
+            -0.016155686229467392,
+            0.007983962073922157,
+            -0.031135819852352142,
+            -0.001782717532478273,
+            -0.010408539324998856,
+            0.0036103345919400454,
+            -0.015943434089422226,
+            -0.009126860648393631,
+            -0.0048777274787425995,
+            -0.0006678812787868083,
+            -0.004820582456886768,
+            0.0004275664687156677,
+            0.0027817331720143557,
+            0.02231917530298233,
+            -0.013208640739321709,
+            -0.02803366631269455,
+            -0.017437364906072617,
+            0.006045116577297449,
+            0.019396619871258736,
+            -0.026025431230664253,
+            -0.009184005670249462,
+            -0.005012426059693098,
+            -0.004326687194406986,
+            -0.021404854953289032,
+            -0.015167895704507828,
+            -0.005632856860756874,
+            -0.008045189082622528,
+            -0.010522829368710518,
+            0.0046532293781638145,
+            -0.0035715578123927116,
+            -0.011739199981093407,
+            0.021666089072823524,
+            0.018057795241475105,
+            -0.019837452098727226,
+            -0.019837452098727226,
+            0.013641309924423695,
+            -0.006159406621009111,
+            0.0018878233386203647,
+            0.02008235827088356,
+            0.00029286774224601686,
+            0.005824700463563204,
+            0.020882388576865196,
+            -0.0035225762985646725,
+            -0.01797616109251976,
+            0.003577680326998234,
+            -0.0018888438353314996,
+            -0.012122887186706066,
+            0.003212361130863428,
+            0.02773977816104889,
+            0.0008076822850853205,
+            0.023429417982697487,
+            -0.03056436963379383,
+            -0.011412657797336578,
+            0.006061443593353033,
+            0.02369065210223198,
+            0.00211844383738935,
+            -0.0021694661118090153,
+            0.020131340250372887,
+            0.03941367194056511,
+            0.005775718949735165,
+            0.0015755671774968505,
+            -0.04003410041332245,
+            0.01027792226523161,
+            0.009208496659994125,
+            0.024621298536658287,
+            0.008339077234268188,
+            0.0014653591206297278,
+            -0.012906589545309544,
+            -0.009755454957485199,
+            -0.0017939424142241478,
+            0.014702572487294674,
+            -0.023870250210165977,
+            0.0024755997583270073,
+            -0.03157665207982063,
+            0.005094061605632305,
+            0.0021959978621453047,
+            -0.009559528902173042,
+            0.0335359051823616,
+            0.00980443600565195,
+            -0.02065380848944187,
+            0.015151568688452244,
+            0.005767555441707373,
+            0.007330877240747213,
+            0.022449791431427002,
+            -0.008228869177401066,
+            0.030450081452727318,
+            -0.005049162078648806,
+            0.01213921420276165,
+            0.02742956392467022,
+            0.032588932663202286,
+            -0.013102514669299126,
+            -0.003959327004849911,
+            0.0006097159348428249,
+            -0.0002561317232903093,
+            0.0007995187188498676,
+            -0.008204378187656403,
+            0.0037389106582850218,
+            -0.014800535514950752,
+            0.012710663489997387,
+            -0.019543563947081566,
+            -0.0034286954905837774,
+            0.0038532004691660404,
+            0.01991908811032772,
+            0.0016265894519165158,
+            -0.033633869141340256,
+            0.02055584453046322,
+            -0.015420965850353241,
+            0.0135596739128232,
+            0.001290863030590117,
+            -0.011110605672001839,
+            -0.00029210239881649613,
+            0.020588500425219536,
+            0.010947334580123425,
+            -0.007845181971788406,
+            -0.0135596739128232,
+            -0.028376536443829536,
+            -0.021535472944378853,
+            0.00823703221976757,
+            0.009445239789783955,
+            -0.0028837777208536863,
+            0.0021306893322616816,
+            -0.011869817040860653,
+            0.0006520643946714699,
+            -0.0008122742874547839,
+            0.01740471087396145,
+            -0.0335359051823616,
+            0.021943651139736176,
+            -0.019527235999703407,
+            -0.019543563947081566,
+            0.017045514658093452,
+            0.008049271069467068,
+            0.009445239789783955,
+            0.0033552234526723623,
+            -0.01995174214243889,
+            -0.012326976284384727,
+            0.0338951013982296,
+            -0.01138000376522541,
+            0.006298186723142862,
+            0.004408322740346193,
+            0.012914752587676048,
+            -0.008914608508348465,
+            -0.011926962062716484,
+            0.01784554310142994,
+            -0.009861581027507782,
+            0.013951525092124939,
+            0.0010010566329583526,
+            -0.0254703089594841,
+            0.02739690989255905,
+            -0.019233347848057747,
+            0.03487472981214523,
+            -0.021617108955979347,
+            0.00711862463504076,
+            0.0066614653915166855,
+            0.008792154490947723,
+            -0.006690037902444601,
+            -0.013224967755377293,
+            0.0012786176521331072,
+            0.03807484731078148,
+            -0.02623768337070942,
+            0.024180466309189796,
+            0.022400809451937675,
+            -0.00040766780148260295,
+            0.02065380848944187,
+            -0.008367649279534817,
+            -0.026711169630289078,
+            0.0179598331451416,
+            -0.012596373446285725,
+            -0.026123393326997757,
+            0.013257622718811035,
+            -0.022678371518850327,
+            -0.020278284326195717,
+            -0.026302991434931755,
+            0.004947117529809475,
+            -0.0025694805663079023,
+            -0.005151206627488136,
+            0.009004407562315464,
+            0.02561725303530693,
+            0.020915042608976364,
+            0.019576217979192734,
+            0.02111096680164337,
+            0.01161674689501524,
+            0.022449791431427002,
+            -0.007559456862509251,
+            0.04401791840791702,
+            -0.004947117529809475,
+            0.004632820375263691,
+            -0.0015980169409886003,
+            -0.015657709911465645,
+            0.009771781973540783,
+            0.0119106350466609,
+            0.0029572497587651014,
+            -0.029176564887166023,
+            -0.009061552584171295,
+            -0.03660540655255318,
+            0.005053244065493345,
+            0.0020664012990891933,
+            0.0176822729408741,
+            0.032115448266267776,
+            -0.005865518003702164,
+            0.004592002835124731,
+            -0.005102225113660097,
+            -0.013927034102380276,
+            -0.009420748800039291,
+            -0.02739690989255905,
+            -0.025437654927372932,
+            -0.007653337903320789,
+            -0.013739272020757198,
+            -0.030368445441126823,
+            -0.011722872965037823,
+            0.01551892887800932,
+            -0.007645174395292997,
+            0.011918798089027405,
+            -0.016702644526958466,
+            0.01385356206446886,
+            -0.004110352601855993,
+            -0.004763437435030937,
+            -0.004530776292085648,
+            -0.017600636929273605,
+            -0.006420640274882317,
+            0.014367866329848766,
+            -0.009477893821895123,
+            -0.01237595733255148,
+            -0.02215590327978134,
+            -0.0015459742862731218,
+            -0.01574750803411007,
+            0.003489922033622861,
+            -0.014596446417272091,
+            -0.006771673448383808,
+            -0.003908304497599602,
+            -0.018514955416321754,
+            0.01766594499349594,
+            0.014816862531006336,
+            -0.019380291923880577,
+            -0.0090697156265378,
+            0.02638462744653225,
+            0.042777057737112045,
+            -0.03218075633049011,
+            -0.008200296200811863,
+            0.0020378287881612778,
+            0.018090449273586273,
+            0.010171796195209026,
+            0.016702644526958466,
+            -0.024653952568769455,
+            0.0022286521270871162,
+            -0.021649762988090515,
+            -0.001220452249981463,
+            -0.023739634081721306,
+            0.010775899514555931,
+            0.025715215131640434,
+            -0.010971825569868088,
+            0.025976449251174927,
+            0.01634344831109047,
+            -0.01812310516834259,
+            0.012016761116683483,
+            0.007183933164924383,
+            0.014645427465438843,
+            -0.019755816087126732,
+            0.0167679525911808,
+            -0.015796490013599396,
+            0.031233781948685646,
+            0.00892277155071497,
+            -0.04829562455415726,
+            -0.03503800183534622,
+            -0.013388238847255707,
+            0.0013388239312916994,
+            -0.010702427476644516,
+            0.015037278644740582,
+            0.015967924147844315,
+            0.013959688134491444,
+            -0.019380291923880577,
+            -0.015820980072021484,
+            0.011984107084572315,
+            0.010065670125186443,
+            0.027462217956781387,
+            -0.006649219896644354,
+            -0.004212397150695324,
+            -0.00960034690797329,
+            -0.02249877341091633,
+            -0.016098542138934135,
+            0.0209803506731987,
+            -0.018041469156742096,
+            0.015935270115733147,
+            -0.026139721274375916,
+            -0.012449429370462894,
+            0.017453692853450775,
+            -0.05211617052555084,
+            -0.01453113742172718,
+            0.0007612520130351186,
+            0.03012353740632534,
+            0.004751192405819893,
+            0.0035184945445507765,
+            0.01841699331998825,
+            -0.01993541419506073,
+            -0.0039776950143277645,
+            -0.02099667675793171,
+            0.006832899991422892,
+            -0.011853490024805069,
+            -0.013984179124236107,
+            0.014482156373560429,
+            -0.013845399022102356,
+            -0.01601690612733364,
+            0.0034368589986115694,
+            0.007183933164924383,
+            -0.021094640716910362,
+            0.011951452121138573,
+            0.003945040516555309,
+            0.012171869166195393,
+            0.012171869166195393,
+            0.01199227012693882,
+            0.021551799029111862,
+            0.02295593172311783,
+            -0.007334959227591753,
+            -0.004767519421875477,
+            -0.01004117913544178,
+            -0.0015398516552522779,
+            0.008898280560970306,
+            0.0033797139767557383,
+            -0.002989904023706913,
+            0.007837017998099327,
+            0.010033016093075275,
+            -0.01498013362288475,
+            0.033193036913871765,
+            -0.0035878848284482956,
+            -0.04134026914834976,
+            -0.003238892648369074,
+            -0.03800953924655914,
+            0.020033378154039383,
+            -0.023298801854252815,
+            -0.016996532678604126,
+            0.0032776694279164076,
+            -0.00629410520195961,
+            -0.019902760162949562,
+            0.013608654960989952,
+            0.011404493823647499,
+            0.00258580781519413,
+            0.02021297626197338,
+            0.012269831262528896,
+            -0.005134879611432552,
+            0.026172375306487083,
+            -0.014718899503350258,
+            -0.008751336485147476,
+            0.015567909926176071,
+            0.006853308994323015,
+            0.0022347746416926384,
+            0.0011031010653823614,
+            -0.018825169652700424,
+            0.005098143592476845,
+            0.0164903923869133,
+            0.007216587662696838,
+            -0.000566347036510706,
+            0.014049487188458443,
+            0.012514738366007805,
+            -0.015135240741074085,
+            -0.0002213090192526579,
+            -0.026564225554466248,
+            0.0032817514147609472,
+            0.004567512311041355,
+            0.007506393827497959,
+            -0.006771673448383808,
+            0.006281859707087278,
+            0.00658799335360527,
+            0.014759717509150505,
+            -0.010155469179153442,
+            -0.016980206593871117,
+            -0.002510294783860445,
+            0.004285869188606739,
+            0.016694480553269386,
+            0.03157665207982063,
+            -0.010310577228665352,
+            -0.005734901409596205,
+            -0.0386626236140728,
+            0.02070278860628605,
+            0.03332365304231644,
+            0.011175914667546749,
+            -0.028213264420628548,
+            0.012302485294640064,
+            0.0006515541463159025,
+            0.0019276207312941551,
+            -0.026302991434931755,
+            0.02787039615213871,
+            0.00974729098379612,
+            0.0066614653915166855,
+            0.017861871048808098,
+            0.0022306928876787424,
+            0.0029266364872455597,
+            -0.016302630305290222,
+            0.01018812321126461,
+            0.009086042642593384,
+            -0.017306748777627945,
+            0.0009745250572450459,
+            0.02324981987476349,
+            -0.01423724927008152,
+            -0.003208279376849532,
+            0.0018133309204131365,
+            0.02651524543762207,
+            -0.013820908032357693,
+            -0.014294394291937351,
+            0.0005714492290280759,
+            0.01751900091767311,
+            -0.0007979880319908261,
+            0.012098397128283978,
+            0.04898136481642723,
+            -0.012424939312040806,
+            0.019657853990793228,
+            0.005461421795189381,
+            -0.015290348790585995,
+            -0.032605260610580444,
+            -0.004922627005726099,
+            -0.01830270327627659,
+            0.011200404725968838,
+            0.014343376271426678,
+            -0.015763835981488228,
+            0.0007525782566517591,
+            -0.03144603595137596,
+            0.007743136957287788,
+            0.014735226519405842,
+            0.018776189535856247,
+            -0.002440904499962926,
+            0.006134915631264448,
+            0.027674470096826553,
+            0.02458864450454712,
+            -0.007392104249447584,
+            0.013804581016302109,
+            0.02320083975791931,
+            -0.025062130764126778,
+            -0.003069498809054494,
+            0.006820654962211847,
+            0.004104230087250471,
+            0.0053063142113387585,
+            -0.016963878646492958,
+            0.03915243595838547,
+            -0.005922663025557995,
+            -0.01506176870316267,
+            0.0030266400426626205,
+            -0.009028897620737553,
+            0.0063390047289431095,
+            -0.012563719414174557,
+            0.012637191452085972,
+            -0.008114579133689404,
+            0.00022883480414748192,
+            -0.0059430720284581184,
+            -0.011110605672001839,
+            -0.012653518468141556,
+            -0.0007949267164804041,
+            -0.016531210392713547,
+            0.007432921789586544,
+            0.00981259997934103,
+            0.005045080557465553,
+            0.01025343220680952,
+            -0.0036797248758375645,
+            -0.003136848099529743,
+            -0.016751626506447792,
+            -0.009984034113585949,
+            0.0073716952465474606,
+            0.01872720755636692,
+            -0.028588788583874702,
+            -0.018547609448432922,
+            0.009290131740272045,
+            0.0022898786701261997,
+            0.018776189535856247,
+            0.0158944521099329,
+            -0.0002595757250674069,
+            0.012318813242018223,
+            -0.01964152604341507,
+            -0.009184005670249462,
+            0.009690145961940289,
+            -0.003569516818970442,
+            0.01048201136291027,
+            -0.014759717509150505,
+            0.010033016093075275,
+            -0.011159587651491165,
+            -0.011053460650146008,
+            0.01385356206446886,
+            0.01738838478922844,
+            0.008873790502548218,
+            -0.001294944784604013,
+            3.383413059054874e-05,
+            -0.010506502352654934,
+            -0.024033522233366966,
+            0.004771600943058729,
+            0.027021385729312897,
+            -0.013273949734866619,
+            0.003883813973516226,
+            0.008702355436980724,
+            -0.017502672970294952,
+            -0.019821124151349068,
+            -0.012653518468141556,
+            -0.022106921300292015,
+            0.020572172477841377,
+            0.023413091897964478,
+            -0.02026195637881756,
+            0.00877582747489214,
+            -0.004620575346052647,
+            0.014049487188458443,
+            -0.01400050614029169,
+            0.010612628422677517,
+            0.0317889042198658,
+            -0.020000724121928215,
+            -0.0028654097113758326,
+            0.0036633978597819805,
+            -0.015347493812441826,
+            0.018629245460033417,
+            0.03268689662218094,
+            -0.00176945177372545,
+            0.014041324146091938,
+            0.0008138049161061645,
+            -0.009118696674704552,
+            0.01905374974012375,
+            0.011094278655946255,
+            0.005347132217139006,
+            -0.0042491331696510315,
+            -0.012212686240673065,
+            -0.03010721132159233,
+            0.016702644526958466,
+            0.0129637336358428,
+            -0.0037674831692129374,
+            -0.0011551438365131617,
+            -0.006343086715787649,
+            -0.015404638834297657,
+            0.0035123717971146107,
+            -0.013078023679554462,
+            -0.00803294312208891,
+            -0.008759500458836555,
+            0.004314441699534655,
+            -0.0034960447810590267,
+            -0.009053388610482216,
+            0.007236996199935675,
+            0.0009510547970421612,
+            -0.0010602424154058099,
+            -0.01902109570801258,
+            -0.025078458711504936,
+            -0.00995954405516386,
+            -0.016849588602781296,
+            0.0038287099450826645,
+            0.0005234883283264935,
+            -0.027184655889868736,
+            0.0053961132653057575,
+            -0.008139070123434067,
+            -0.008134988136589527,
+            0.007126788143068552,
+            0.006881881505250931,
+            -0.007437003776431084,
+            -0.010816717520356178,
+            -0.00786150898784399,
+            -0.006865554489195347,
+            0.020033378154039383,
+            0.0034348180051892996,
+            0.012082069180905819,
+            -0.010498338378965855,
+            -0.018025141209363937,
+            -0.016963878646492958,
+            0.016718972474336624,
+            -0.008971752598881721,
+            -0.02083340659737587,
+            -0.01444950234144926,
+            -0.008424794301390648,
+            -0.007665583398193121,
+            -0.01527402177453041,
+            -0.021159948781132698,
+            -0.0032103201374411583,
+            -0.011347348801791668,
+            -0.0067431009374558926,
+            -0.0028960229828953743,
+            0.0013571919407695532,
+            0.00951871182769537,
+            0.00546958576887846,
+            0.00438791373744607,
+            0.021666089072823524,
+            0.010122815147042274,
+            -0.005461421795189381,
+            0.016441410407423973,
+            -0.006183897145092487,
+            -0.004440976772457361,
+            -0.01498013362288475,
+            -0.009918726049363613,
+            0.018547609448432922,
+            -0.0055226488038897514,
+            0.008457448333501816,
+            -0.0009536059224046767,
+            -0.00576347392052412,
+            0.015674035996198654,
+            0.0129637336358428,
+            0.016718972474336624,
+            0.004771600943058729,
+            -0.00012168807734269649,
+            0.012416775338351727,
+            -0.0015745466807857156,
+            0.012555556371808052,
+            -0.0038429962005466223,
+            0.014506647363305092,
+            -0.009722800925374031,
+            -0.018988441675901413,
+            0.004157293122261763,
+            0.006265532691031694,
+            -0.012531065382063389,
+            0.009404421783983707,
+            0.007743136957287788,
+            -0.013747435994446278,
+            -0.02261306345462799,
+            -0.000106317616882734,
+            0.00014362763613462448,
+            0.008122743107378483,
+            0.008971752598881721,
+            -0.008073761127889156,
+            0.012890261597931385,
+            0.004616493359208107,
+            -0.007179851643741131,
+            -0.019657853990793228,
+            0.026009103283286095,
+            0.015608727931976318,
+            0.012269831262528896,
+            -0.052508022636175156,
+            -0.0030980713199824095,
+            0.010743245482444763,
+            -0.015951598063111305,
+            -0.026792805641889572,
+            -0.006404313258826733,
+            -0.008759500458836555,
+            -0.04290767386555672,
+            -0.009624837897717953,
+            -0.0005724696675315499,
+            -0.011388166807591915,
+            0.010400376282632351,
+            0.01573934406042099,
+            0.014612773433327675,
+            0.005489994306117296,
+            -0.007232914678752422,
+            -0.017633290961384773,
+            -0.02788672223687172,
+            0.010057506151497364,
+            -0.013380075804889202,
+            -0.006412476766854525,
+            -0.002263347152620554,
+            -0.007690073922276497,
+            0.0016112826997414231,
+            0.01691489666700363,
+            0.042483169585466385,
+            0.023037567734718323,
+            -0.012335140258073807,
+            -0.020131340250372887,
+            -0.012800462543964386,
+            -0.009192168712615967,
+            0.00633492274209857,
+            0.011592255905270576,
+            -0.004738946910947561,
+            0.0096656559035182,
+            -0.004294032696634531,
+            0.012302485294640064,
+            0.012767808511853218,
+            0.01680060662329197,
+            -2.3023791072773747e-05,
+            0.005579793825745583,
+            -0.0024694770108908415,
+            -0.01738838478922844,
+            -0.004379750229418278,
+            0.008261523209512234,
+            -0.0008872769540175796,
+            0.0019317024853080511,
+            0.014204595237970352,
+            0.019102731719613075,
+            -0.009094206616282463,
+            -0.014612773433327675,
+            -0.016098542138934135,
+            -0.005277741700410843,
+            -0.0012622905196622014,
+            0.029927613213658333,
+            0.02037624642252922,
+            -0.019266001880168915,
+            -0.018237393349409103,
+            -0.018351683393120766,
+            0.00922482367604971,
+            -0.003563394071534276,
+            -0.0064369672909379005,
+            0.01905374974012375,
+            0.023021239787340164,
+            0.017910851165652275,
+            -0.003642988856881857,
+            -0.014865843579173088,
+            0.003804219188168645,
+            0.02847449854016304,
+            0.008367649279534817,
+            0.01078406348824501,
+            0.008726846426725388,
+            -0.0011908593587577343,
+            -0.032441988587379456,
+            -0.0113146947696805,
+            0.00951871182769537,
+            -0.008481939323246479,
+            0.026564225554466248,
+            -0.016531210392713547,
+            0.009388094767928123,
+            -0.005077734589576721,
+            0.0019286411115899682,
+            -0.02365799807012081,
+            0.0014929111348465085,
+            0.010122815147042274,
+            -0.024947840720415115,
+            0.00336950970813632,
+            -0.023592690005898476,
+            -0.00013763252354692668,
+            0.008367649279534817,
+            0.0022837561555206776,
+            0.01650671847164631,
+            -0.0113146947696805,
+            0.0051838611252605915,
+            0.02099667675793171,
+            -0.002832755446434021,
+            -0.01512707769870758,
+            -0.0007842120248824358,
+            0.0005092020728625357,
+            -0.001286781276576221,
+            0.02563357912003994,
+            0.015812816098332405,
+            -0.0036756431218236685,
+            0.020882388576865196,
+            0.008677864447236061,
+            0.0019133344758301973,
+            -0.021764053031802177,
+            0.0023796779569238424,
+            -0.032899148762226105,
+            0.017437364906072617,
+            -0.011126932688057423,
+            0.012490247376263142,
+            0.00868602842092514,
+            -0.01341272983700037,
+            -0.0007388022495433688,
+            -0.0107187544927001,
+            -0.0015725058037787676,
+            -0.007934981025755405,
+            0.009779945947229862,
+            -0.004363423213362694,
+            0.008339077234268188,
+            -0.00024579971795901656,
+            -0.01476788055151701,
+            0.018694553524255753,
+            9.678283277025912e-06,
+            -0.00010925139940809458,
+            0.005441013257950544,
+            0.009935053065419197,
+            0.00786559097468853,
+            0.002009256277233362,
+            0.023870250210165977,
+            -0.031119491904973984,
+            -0.016082214191555977,
+            -0.010204450227320194,
+            0.005543057806789875,
+            -0.007922735065221786,
+            -0.018531281501054764,
+            -0.03257260471582413,
+            0.01318415068089962,
+            0.001293924287892878,
+            -0.0009806477464735508,
+            0.01258004643023014,
+            -0.00942891277372837,
+            -0.0013878053287044168,
+            -0.01606588624417782,
+            -0.014808698557317257,
+            0.006416558753699064,
+            0.0033633869607001543,
+            -0.01222901325672865,
+            -0.023739634081721306,
+            0.014759717509150505,
+            0.0007418635650537908,
+            -0.016980206593871117,
+            -0.017502672970294952,
+            0.00018457300029695034,
+            -0.004804255440831184,
+            0.02204161323606968,
+            -0.016678154468536377,
+            0.003373591462150216,
+            0.0007219648687168956,
+            0.009232986718416214,
+            -0.02889900468289852,
+            0.00716352416202426,
+            0.0006107363733462989,
+            0.0035164537839591503,
+            -0.006518602836877108,
+            -0.021013004705309868,
+            0.011371839791536331,
+            -0.0020674217958003283,
+            -0.014751553535461426,
+            -0.020751770585775375,
+            -0.0020796670578420162,
+            -0.007420676294714212,
+            -0.02724996581673622,
+            0.017910851165652275,
+            0.0009985055075958371,
+            -0.01738838478922844,
+            0.008077843114733696,
+            0.006596156861633062,
+            0.009355440735816956,
+            0.02455599047243595,
+            0.010147306136786938,
+            -0.018776189535856247,
+            -0.0008087027235887945,
+            0.006979844067245722,
+            0.009306458756327629,
+            0.0036450298503041267,
+            0.04222193360328674,
+            -0.008130906149744987,
+            -0.006579829845577478,
+            0.0066614653915166855,
+            -0.008449285291135311,
+            -0.013747435994446278,
+            0.026727497577667236,
+            0.005796127952635288,
+            0.015984252095222473,
+            0.016123032197356224,
+            -0.00945340283215046,
+            -0.005008344538509846,
+            -0.0018786393338814378,
+            0.02065380848944187,
+            0.0073798587545752525,
+            -0.005502239800989628,
+            -0.009714636951684952,
+            0.005881845485419035,
+            -0.00898808054625988,
+            0.0021653843577951193,
+            -0.009526874870061874,
+            -0.005824700463563204,
+            -0.008069679141044617,
+            0.016400592401623726,
+            -0.0003523086488712579,
+            0.027315273880958557,
+            0.020751770585775375,
+            -0.008563575334846973,
+            -0.0005683879135176539,
+            0.017208784818649292,
+            -0.005820618476718664,
+            0.015608727931976318,
+            -0.011192241683602333,
+            -0.007706400938332081,
+            0.006408394780009985,
+            0.024017194285988808,
+            0.012237177230417728,
+            -0.0056042843498289585,
+            0.0002885308349505067,
+            -0.0030348035506904125,
+            -0.010612628422677517,
+            0.013388238847255707,
+            0.015323002822697163,
+            -0.01947825588285923,
+            -0.004722619894891977,
+            -0.0016592437168583274,
+            -0.003300119424238801,
+            0.00945340283215046,
+            0.02305389568209648,
+            0.000171817431692034,
+            0.009184005670249462,
+            0.0012765767751261592,
+            0.03862996771931648,
+            -0.0053144777193665504,
+            0.03102152980864048,
+            -0.0010423846542835236,
+            -0.0059430720284581184,
+            -0.009094206616282463,
+            -0.020441556349396706,
+            -0.011486129835247993,
+            -0.00041481092921458185,
+            -0.005040998570621014,
+            -0.0004956811899319291,
+            -0.020327266305685043,
+            0.0026960158720612526,
+            0.014637264423072338,
+            0.007661501411348581,
+            0.017927179113030434,
+            -0.002889900468289852,
+            0.017437364906072617,
+            0.019690508022904396,
+            -0.01400050614029169,
+            -0.029862305149435997,
+            -0.0006515541463159025,
+            0.01498013362288475,
+            -0.01661284640431404,
+            0.019445601850748062,
+            0.004085862077772617,
+            0.0029531680047512054,
+            0.004779764451086521,
+            -0.006049198564141989,
+            0.018482301384210587,
+            0.02039257436990738,
+            0.01453930139541626,
+            -0.013167822733521461,
+            0.014057651162147522,
+            0.009771781973540783,
+            0.006600238848477602,
+            -0.01646590232849121,
+            -0.013535182923078537,
+            0.002555194543674588,
+            0.0009245232213288546,
+            0.024392718449234962,
+            -0.020588500425219536,
+            0.0016602640971541405,
+            0.010588138364255428,
+            0.01769859902560711,
+            -0.004302196204662323,
+            0.008906444534659386,
+            0.0008740111952647567,
+            0.007134951651096344,
+            -0.0035919665824621916,
+            -0.006457376293838024,
+            -0.006673710886389017,
+            -0.010392212308943272,
+            -0.0020959940738976,
+            -0.02070278860628605,
+            0.002198038622736931,
+            -0.002830714685842395,
+            0.018237393349409103,
+            -0.01018812321126461,
+            0.038140155375003815,
+            0.03399306535720825,
+            -0.0007025764207355678,
+            0.011102442629635334,
+            -0.01709449477493763,
+            -0.003945040516555309,
+            -0.021959977224469185,
+            0.01844964735209942,
+            0.016392430290579796,
+            0.012269831262528896,
+            0.005302232690155506,
+            0.017796561121940613,
+            -0.0006301248213276267,
+            0.014490320347249508,
+            0.0029654132667928934,
+            0.007971717044711113,
+            -0.002265388146042824,
+            0.003938918001949787,
+            -0.031103165820240974,
+            -0.017029186710715294,
+            -0.01138000376522541,
+            0.004796091932803392,
+            -0.006375740747898817,
+            -0.0035205355379730463,
+            0.0084737753495574,
+            -0.005902254022657871,
+            0.0004270562494639307,
+            -0.025992777198553085,
+            -0.0157148540019989,
+            -0.010318740271031857,
+            -0.004040962550789118,
+            -0.01169838197529316,
+            -0.0027694879099726677,
+            -0.0022082431241869926,
+            0.0029552089981734753,
+            0.020947696641087532,
+            -0.012808626517653465,
+            -0.016294466331601143,
+            0.004379750229418278,
+            -0.002263347152620554,
+            -0.014016833156347275,
+            -0.016898570582270622,
+            0.006665547378361225,
+            0.012098397128283978,
+            -0.0017470018938183784,
+            -0.013608654960989952,
+            0.011739199981093407,
+            -0.006628811359405518,
+            0.04663025587797165,
+            -0.013584164902567863,
+            -0.0078002819791436195,
+            0.003720542648807168,
+            0.004698128905147314,
+            -0.009396257810294628,
+            0.007330877240747213,
+            0.013020878657698631,
+            0.01830270327627659,
+            0.0095758568495512,
+            -0.008465612307190895,
+            -0.0024388637393712997,
+            0.007796200457960367,
+            -0.016147522255778313,
+            0.006404313258826733,
+            0.0008525818702764809,
+            -0.014400520361959934,
+            0.006110425107181072,
+            0.028980640694499016,
+            0.013151495717465878,
+            0.01288209855556488,
+            -0.022351829335093498,
+            -0.004094025585800409,
+            -0.009355440735816956,
+            -0.004943036008626223,
+            -0.010090161114931107,
+            0.027070365846157074,
+            -0.018155759200453758,
+            0.02651524543762207,
+            0.00898808054625988,
+            0.0017684312770143151,
+            -0.01176369097083807,
+            0.006343086715787649,
+            0.02832755446434021,
+            0.0028592869639396667,
+            -0.020000724121928215,
+            -0.013592327944934368,
+            0.006314514204859734,
+            0.01707816869020462,
+            -0.002647034591063857,
+            -0.005428767763078213,
+            0.019657853990793228,
+            0.01784554310142994,
+            -0.004334850702434778,
+            -0.00936360377818346,
+            0.0008337036124430597,
+            0.00928196869790554,
+            0.016963878646492958,
+            0.015020951628684998,
+            0.009086042642593384,
+            -0.002647034591063857,
+            0.008037025108933449,
+            -0.008800318464636803,
+            -0.016449574381113052,
+            0.007220669183880091,
+            -0.008432958275079727,
+            -0.016963878646492958,
+            0.01872720755636692,
+            0.015257694758474827,
+            0.0181394312530756,
+            0.0024592725094407797,
+            -0.016735298559069633,
+            0.022302847355604172,
+            0.005755309946835041,
+            0.002998067531734705,
+            0.02160078100860119,
+            -0.022221211344003677,
+            0.013012715615332127,
+            0.02037624642252922,
+            0.027609162032604218,
+            -0.01737205684185028,
+            -0.010906516574323177,
+            -0.004857318475842476,
+            -0.006853308994323015,
+            -0.00176945177372545,
+            -0.008253359235823154,
+            -0.012620864436030388,
+            0.0028735732194036245,
+            -0.030466407537460327,
+            0.018106777220964432,
+            -0.005102225113660097,
+            0.010669773444533348,
+            -0.019266001880168915,
+            0.011363675817847252,
+            -0.0035838030744343996,
+            0.011135096661746502,
+            0.0019612954929471016,
+            -0.030041903257369995,
+            -0.008571738377213478,
+            0.015388311818242073,
+            -0.0030368445441126823,
+            -0.013641309924423695,
+            0.02006603218615055,
+            0.00018942011229228228,
+            -0.022270193323493004,
+            -0.004314441699534655,
+            -0.006934944540262222,
+            0.002193956868723035,
+            -0.0029613315127789974,
+            -0.0052981507033109665,
+            -0.030188847333192825,
+            0.020033378154039383,
+            0.009559528902173042,
+            -0.03072764165699482,
+            -0.003255219664424658,
+            -0.014996460638940334,
+            -0.0025184585247188807,
+            0.008881953544914722,
+            -0.004000144544988871,
+            -0.015551582910120487,
+            0.01784554310142994,
+            0.022711025550961494,
+            -0.0023164101876318455,
+            -0.016057724133133888,
+            -0.0090697156265378,
+            -0.027494871988892555,
+            0.030825603753328323,
+            -0.0066696288995444775,
+            0.0022878379095345736,
+            -0.005322641227394342,
+            0.009388094767928123,
+            0.004228724166750908,
+            0.0157148540019989,
+            0.006081852596253157,
+            -0.002647034591063857,
+            0.026596879586577415,
+            -0.01964152604341507,
+            0.009567692875862122,
+            -1.588864870427642e-05,
+            -0.003308282932266593,
+            0.01890680566430092,
+            0.0007939062779769301,
+            0.021649762988090515,
+            -0.008571738377213478,
+            -0.0001475818717153743,
+            -0.0034388997592031956,
+            0.010898353531956673,
+            -0.0028450009413063526,
+            -0.003465431509539485,
+            -0.001531688030809164,
+            0.011118769645690918,
+            0.01013097818940878,
+            0.003804219188168645,
+            -0.01737205684185028,
+            0.003142970846965909,
+            -0.008547247387468815,
+            -0.007028825581073761,
+            0.01302904263138771,
+            -0.011192241683602333,
+            -0.0018990482203662395,
+            -0.03160930797457695,
+            0.016286302357912064,
+            0.0533243790268898,
+            0.008547247387468815,
+            -0.014433175325393677,
+            -0.0018806802108883858,
+            -0.019755816087126732,
+            0.01535565685480833,
+            0.011004479601979256,
+            -0.016816934570670128,
+            -0.011984107084572315,
+            -0.012310649268329144,
+            0.007649256382137537,
+            0.021306892856955528,
+            -0.006522684823721647,
+            -0.008579902350902557,
+            0.025372346863150597,
+            -0.011788181029260159,
+            -0.010882026515901089,
+            -0.01661284640431404,
+            -0.01635977439582348,
+            0.00241641397587955,
+            0.019902760162949562,
+            -0.0169475506991148,
+            0.021176276728510857,
+            0.00936360377818346,
+            0.0010133018950000405,
+            0.003624620847404003,
+            0.008588065393269062,
+            0.0024878450203686953,
+            0.017878197133541107,
+            0.003883813973516226,
+            -0.01439235731959343,
+            0.0021408938337117434,
+            -0.0030286810360848904,
+            0.004624656867235899,
+            3.4320437407586724e-06,
+            -0.002749078907072544,
+            -0.0065635028295218945,
+            0.003710338147357106,
+            -0.021306892856955528,
+            0.021502818912267685,
+            -0.022678371518850327,
+            -0.004245051648467779,
+            0.005555302836000919,
+            -0.005118552595376968,
+            0.013469874858856201,
+            0.018792515620589256,
+            0.01423724927008152,
+            0.008702355436980724,
+            0.01462093647569418,
+            -0.011331021785736084,
+            0.02323349379003048,
+            -0.013282112777233124,
+            0.0005816536722704768,
+            0.012939243577420712,
+            0.01755165494978428,
+            0.018792515620589256,
+            -0.006245123688131571,
+            -0.010384049266576767,
+            -0.0002946535241790116,
+            0.0129637336358428,
+            -0.022286519408226013,
+            -0.012269831262528896,
+            -0.01386172603815794,
+            0.011486129835247993,
+            -0.004743028432130814,
+            0.025453981012105942,
+            -0.00633492274209857,
+            -0.014653591439127922,
+            -0.015918942168354988,
+            0.014196431264281273,
+            -0.048752784729003906,
+            0.0030348035506904125,
+            -0.0024919267743825912,
+            0.03043375350534916,
+            -0.027021385729312897,
+            0.01738838478922844,
+            -0.0028490826953202486,
+            -0.009355440735816956,
+            0.015429128892719746,
+            -0.015992414206266403,
+            -0.0046613928861916065,
+            -0.0016806729836389422,
+            -0.007539048325270414,
+            -0.00112248957157135,
+            0.017616963014006615,
+            -0.015845470130443573,
+            0.01282495353370905,
+            -0.00989423505961895,
+            0.0027021386194974184,
+            -0.004526694305241108,
+            -0.007751300930976868,
+            0.0196088720113039,
+            0.034548189491033554,
+            -0.007053316105157137,
+            0.0014990337658673525,
+            -0.022547753527760506,
+            0.013192313723266125,
+            -0.006730855442583561,
+            0.02351105399429798,
+            -0.005432849284261465,
+            -0.016898570582270622,
+            -0.013976015150547028,
+            -0.009739127941429615,
+            0.00839214026927948,
+            -0.022433463484048843,
+            -0.011110605672001839,
+            -0.00922482367604971,
+            -0.002573562553152442,
+            -0.018694553524255753,
+            0.004865481983870268,
+            -0.002426618244498968,
+            -0.012506574392318726,
+            -0.008816645480692387,
+            0.01048201136291027,
+            -0.009347276762127876,
+            0.003077662317082286,
+            -0.0011929002357646823,
+            -0.011453475803136826,
+            -0.02369065210223198,
+            0.0158944521099329,
+            0.0001974561164388433,
+            -0.010359558276832104,
+            0.008408467285335064,
+            -0.0018276170594617724,
+            0.001211268245242536,
+            0.016963878646492958,
+            -0.0020664012990891933,
+            0.003467472270131111,
+            0.023478399962186813,
+            0.004057289566844702,
+            -0.010971825569868088,
+            -0.00025906547671183944,
+            0.013535182923078537,
+            -0.005118552595376968,
+            -0.010008525103330612,
+            1.6646010408294387e-05,
+            -0.02984597720205784,
+            -0.008351322263479233,
+            0.006094098091125488,
+            0.018629245460033417,
+            -0.010620792396366596,
+            -0.015853634104132652,
+            0.026319319382309914,
+            -0.006359413731843233,
+            0.019070077687501907,
+            0.002889900468289852,
+            0.0025694805663079023,
+            0.01724144071340561,
+            0.013461710885167122,
+            0.00786150898784399,
+            -0.013927034102380276,
+            -0.02429475635290146,
+            0.0016398552106693387,
+            -0.018090449273586273,
+            0.010579974390566349,
+            -0.011028970591723919,
+            0.0032756286673247814,
+            -0.02398454025387764,
+            -0.012237177230417728,
+            0.004934872500598431,
+            -0.020343592390418053,
+            -0.00040588201954960823,
+            -0.0056940834037959576,
+            0.022237539291381836,
+            -0.005310396198183298,
+            0.008669701404869556,
+            0.0018255761824548244,
+            -0.012237177230417728,
+            0.0009877908742055297,
+            0.009469729848206043,
+            -0.029192892834544182,
+            -0.0007127808639779687,
+            -0.003138889092952013,
+            0.0011418780777603388,
+            0.005351213738322258,
+            0.009755454957485199,
+            0.007322713732719421,
+            0.0027062203735113144,
+            0.007253323681652546,
+            0.010449357330799103,
+            -0.00022041612828616053,
+            0.0197884701192379,
+            0.018172085285186768,
+            -0.007351286243647337,
+            0.008710519410669804,
+            0.024817224591970444,
+            -0.013159659691154957,
+            -0.008718682453036308,
+            0.023266147822141647,
+            -0.006461458280682564,
+            -0.0011122850701212883,
+            -0.01483318954706192,
+            0.00815947912633419,
+            0.0017367975087836385,
+            0.028131630271673203,
+            0.019657853990793228,
+            0.005820618476718664,
+            0.000639819074422121,
+            0.015788326039910316,
+            -0.013020878657698631,
+            0.0035286990460008383,
+            0.015461783856153488,
+            -0.003940958995372057,
+            -0.019821124151349068,
+            0.013878053054213524,
+            0.009502384811639786,
+            -0.003228688146919012,
+            0.01551892887800932,
+            0.011412657797336578,
+            -0.030841931700706482,
+            -0.0011898389784619212,
+            0.010498338378965855,
+            -0.0021449755877256393,
+            -0.008751336485147476,
+            -0.00014273475972004235,
+            -0.012384121306240559,
+            0.021388528868556023,
+            -0.004408322740346193,
+            0.012277995236217976,
+            -0.00815947912633419,
+            -0.017127150669693947,
+            -0.011559601873159409,
+            0.007914572022855282,
+            -0.010547320358455181,
+            -0.0006714528426527977,
+            -0.004022594541311264,
+            -0.005779800936579704,
+            0.017747581005096436,
+            0.004706292413175106,
+            -0.0008964610169641674,
+            -0.0018398624379187822,
+            0.007983962073922157,
+            -0.006979844067245722,
+            -0.008726846426725388,
+            0.004816500470042229,
+            -0.012833117507398129,
+            0.003981776535511017,
+            0.005543057806789875,
+            0.01506176870316267,
+            -0.014857680536806583,
+            -0.021159948781132698,
+            0.007020662073045969,
+            0.02695607766509056,
+            -0.02516009286046028,
+            -0.005232842173427343,
+            0.006245123688131571,
+            0.002093953313305974,
+            0.006914535537362099,
+            -0.006934944540262222,
+            -0.007208424154669046,
+            -0.03657275065779686,
+            -0.0020745648071169853,
+            0.0009005427709780633,
+            0.008963589556515217,
+            -0.008049271069467068,
+            -0.0136494729667902,
+            -0.011135096661746502,
+            0.002410291228443384,
+            0.007134951651096344,
+            -0.02204161323606968,
+            0.0049593630246818066,
+            0.009608510881662369,
+            -0.019559891894459724,
+            0.025127438828349113,
+            -0.010890189558267593,
+            -6.964537897147238e-05,
+            -0.00995954405516386,
+            0.021633435040712357,
+            -0.013486201874911785,
+            -0.018368011340498924,
+            -0.0006898208521306515,
+            0.008355404250323772,
+            0.011926962062716484,
+            -0.0017663904000073671,
+            -0.011371839791536331,
+            0.0003752686607185751,
+            0.01646590232849121,
+            0.019135385751724243,
+            0.002744997153058648,
+            -0.0024837632663547993,
+            -0.01019628718495369,
+            -0.009592183865606785,
+            -0.0073961857706308365,
+            -0.0007449248805642128,
+            0.002091912319883704,
+            0.02035992033779621,
+            0.013469874858856201,
+            0.0024837632663547993,
+            -0.015135240741074085,
+            -0.005257333163172007,
+            0.008939098566770554,
+            0.009412584826350212,
+            -0.011690218932926655,
+            -0.00898808054625988,
+            0.002659279853105545,
+            0.02249877341091633,
+            0.017633290961384773,
+            -0.0014000505907461047,
+            0.0032878739293664694,
+            -0.0016592437168583274,
+            0.012506574392318726,
+            -0.007555375341325998,
+            -0.010269759222865105,
+            -0.006943108048290014,
+            -0.014645427465438843,
+            -0.010800390504300594,
+            -0.0023062059190124273,
+            -0.010179960168898106,
+            0.022629389539361,
+            0.007988044060766697,
+            0.009779945947229862,
+            0.008265605196356773,
+            0.010269759222865105,
+            0.002087830565869808,
+            0.025715215131640434,
+            -0.004424649756401777,
+            -0.011984107084572315,
+            0.007236996199935675,
+            -0.022368155419826508,
+            -0.021535472944378853,
+            0.008257441222667694,
+            0.006367577239871025,
+            0.03595232218503952,
+            -0.026890767738223076,
+            0.0052981507033109665,
+            -0.009779945947229862,
+            -0.010400376282632351,
+            0.007996207103133202,
+            -0.0015439334092661738,
+            -0.004245051648467779,
+            0.022368155419826508,
+            0.00726556871086359,
+            0.0035797213204205036,
+            0.0032817514147609472,
+            0.019837452098727226,
+            -0.005685919895768166,
+            0.01573934406042099,
+            0.0259274672716856,
+            0.024621298536658287,
+            0.009388094767928123,
+            -0.009469729848206043,
+            0.02785406820476055,
+            0.0002895512734539807,
+            -0.0034327772445976734,
+            0.007220669183880091,
+            -0.00305113079957664,
+            -0.0009556467994116247,
+            -0.010033016093075275,
+            0.021764053031802177,
+            -0.001455154619179666,
+            0.004359341226518154,
+            0.02081707864999771,
+            -7.774515688652173e-05,
+            0.006220633164048195,
+            0.01057181041687727,
+            0.002181711606681347,
+            0.003396041225641966,
+            0.010775899514555931,
+            -0.039315707981586456,
+            -0.0006571666453965008,
+            -0.025600925087928772,
+            -0.0018286375561729074,
+            0.0007357408758252859,
+            -0.01266984548419714,
+            0.013600491918623447,
+            -0.002015379024669528,
+            -0.010751409456133842,
+            -0.00029337796149775386,
+            0.021649762988090515,
+            0.005416522268205881,
+            0.004071575589478016,
+            -0.004469549283385277,
+            0.018792515620589256,
+            -0.006955353543162346,
+            -0.017861871048808098,
+            -0.012759645469486713,
+            -0.012367794290184975,
+            -0.01918436773121357,
+            -0.01214737817645073,
+            -0.0024551907554268837,
+            -0.002667443361133337,
+            0.004738946910947561,
+            0.002481722505763173,
+            -0.003228688146919012,
+            -0.02070278860628605,
+            0.015037278644740582,
+            -0.01430255826562643,
+            0.012873934581875801,
+            -0.011053460650146008,
+            0.005963481031358242,
+            -0.004122598096728325,
+            0.012726990506052971,
+            -0.021404854953289032,
+            0.02070278860628605,
+            -0.01536382082849741,
+            0.0038532004691660404,
+            0.019592545926570892,
+            0.006551257334649563,
+            -0.008118661120533943,
+            -0.013404566794633865,
+            -0.022074267268180847,
+            0.021633435040712357,
+            -0.011633073911070824,
+            -0.018237393349409103,
+            -0.004608329851180315,
+            0.009649328887462616,
+            -0.017470018938183784,
+            -0.007367613259702921,
+            0.01707816869020462,
+            -0.0018949664663523436,
+            0.012980061583220959,
+            0.023004913702607155,
+            -0.0052981507033109665,
+            0.0036572751123458147,
+            -0.009167678654193878,
+            0.0010286086471751332,
+            -0.02470293454825878,
+            0.013527019880712032,
+            0.008971752598881721,
+            0.006269614677876234,
+            -0.013935198076069355,
+            -0.013306603766977787,
+            0.0036531933583319187,
+            0.004498121794313192,
+            0.011437147855758667,
+            0.009420748800039291,
+            0.004220560658723116,
+            -0.013918871060013771,
+            -0.027788760140538216,
+            -0.027070365846157074,
+            0.03595232218503952,
+            -0.0005002731923013926,
+            -0.020947696641087532,
+            0.03399306535720825,
+            0.016114868223667145,
+            0.0022980421781539917,
+            -0.012449429370462894,
+            -0.004314441699534655,
+            0.024621298536658287,
+            0.016400592401623726,
+            0.010465684346854687,
+            -0.0004742518358398229,
+            0.005571629852056503,
+            0.0016469983384013176,
+            0.03012353740632534,
+            0.015323002822697163,
+            0.0009459525463171303,
+            -0.038597315549850464,
+            0.011943289078772068,
+            -0.007547211833298206,
+            0.0013408648082986474,
+            0.006702283397316933,
+            0.017780235037207603,
+            0.011935125105082989,
+            0.015567909926176071,
+            0.014865843579173088,
+            -0.003065417055040598,
+            0.0029511270113289356,
+            0.0006163488142192364,
+            -0.009624837897717953,
+            -0.001767410896718502,
+            -0.008881953544914722,
+            -0.005714492406696081,
+            -0.002920513739809394,
+            0.005592038854956627,
+            0.007657419890165329,
+            -0.003983817528933287,
+            0.028049994260072708,
+            -0.0006433906382881105,
+            -0.017600636929273605,
+            -0.009061552584171295,
+            -0.03611559048295021,
+            0.00434301421046257,
+            -0.030939893797039986,
+            -0.011257549747824669,
+            0.0033674687147140503,
+            -0.020457882434129715,
+            -0.024768242612481117,
+            0.0010306495241820812,
+            0.014335212297737598,
+            0.007392104249447584,
+            0.0033327736891806126,
+            -0.01004934310913086,
+            0.003816464450210333,
+            -0.001855169073678553,
+            -0.006461458280682564,
+            0.028833696618676186,
+            0.019559891894459724,
+            0.020457882434129715,
+            -0.01190247107297182,
+            0.019298657774925232,
+            0.005845109466463327,
+            0.02035992033779621,
+            0.010465684346854687,
+            -0.0048695639707148075,
+            0.0074859848245978355,
+            0.009567692875862122,
+            -0.006155324634164572,
+            -0.021519144997000694,
+            0.013192313723266125,
+            -0.0010837126756086946,
+            -0.0052083516493439674,
+            -0.016269976273179054,
+            -0.003534821793437004,
+            -0.015812816098332405,
+            0.010547320358455181,
+            -0.017159804701805115,
+            0.012498411349952221,
+            -0.000802069844212383,
+            0.00168475485406816,
+            -0.015469946898519993,
+            -0.013951525092124939,
+            -0.0012439225101843476,
+            -0.01244126632809639,
+            -0.008971752598881721,
+            0.006163488142192364,
+            -0.007539048325270414,
+            -0.008318668231368065,
+            -0.00535937724635005,
+            -0.011437147855758667,
+            -0.010882026515901089,
+            0.004367504734545946,
+            -0.0010653446661308408,
+            0.020931368693709373,
+            0.00011671340325847268,
+            -0.005824700463563204,
+            0.01190247107297182,
+            0.02787039615213871,
+            0.0016704685986042023,
+            0.004583839327096939,
+            -0.014261740259826183,
+            -0.011118769645690918,
+            -0.02279266156256199,
+            0.016572028398513794,
+            0.0049675265327095985,
+            -0.0344502255320549,
+            0.01438419334590435,
+            0.00824927818030119,
+            0.0015102586476132274,
+            -0.006698201410472393,
+            -0.005334886722266674,
+            -0.005608365871012211,
+            0.006306350696831942,
+            -0.0022837561555206776,
+            0.0005107327597215772,
+            0.0035919665824621916,
+            0.034678805619478226,
+            -0.0040674940682947636,
+            -0.007228832691907883,
+            0.00981259997934103,
+            0.0005130287609063089,
+            0.020196648314595222,
+            -0.005518566817045212,
+            0.0037572786677628756,
+            -0.019282329827547073,
+            -0.016563864424824715,
+            0.0018429238116368651,
+            -0.01722511276602745,
+            0.023282473906874657,
+            -0.005392031744122505,
+            -0.01347803883254528,
+            -0.006906372029334307,
+            -0.020115012302994728,
+            0.012637191452085972,
+            0.02310287579894066,
+            0.007016580086201429,
+            0.01033506728708744,
+            0.012131051160395145,
+            -0.02444170042872429,
+            -0.020457882434129715,
+            0.012171869166195393,
+            0.0013571919407695532,
+            -0.008449285291135311,
+            -0.005404277238994837,
+            -0.001783737912774086,
+            0.007783954963088036,
+            -0.0032633834052830935,
+            0.007881917990744114,
+            0.06491663306951523,
+            -0.01093100756406784,
+            -0.002987863030284643,
+            -0.02966637909412384,
+            -0.004396077245473862,
+            -0.006628811359405518,
+            0.0056287748739123344,
+            0.009151351638138294,
+            -0.023135529831051826,
+            0.009232986718416214,
+            -0.030025575309991837,
+            -0.010628955438733101,
+            0.00824111420661211,
+            0.0028490826953202486,
+            -0.01220452319830656,
+            -0.0009592183632776141,
+            -0.013665799982845783,
+            0.0006107363733462989,
+            -0.009412584826350212,
+            -0.00576347392052412,
+            0.012718827463686466,
+            -0.02321716584265232,
+            0.006298186723142862,
+            0.007592111360281706,
+            -0.006514521315693855,
+            -0.013722945004701614,
+            -0.017861871048808098,
+            -0.011967780068516731,
+            -0.013314766809344292,
+            0.002996026771143079,
+            0.01190247107297182,
+            -0.02607441134750843,
+            0.012922916561365128,
+            -0.0037389106582850218,
+            0.00945340283215046,
+            0.004404240753501654,
+            0.013469874858856201,
+            0.018074123188853264,
+            0.009355440735816956,
+            0.0021429345943033695,
+            -0.008024780079722404,
+            0.012702500447630882,
+            -0.01993541419506073,
+            -0.01635161228477955,
+            0.013837235048413277,
+            0.012016761116683483,
+            0.003061335301026702,
+            0.009828926995396614,
+            0.0020959940738976,
+            0.011886144056916237,
+            0.010775899514555931,
+            -0.019739490002393723,
+            -0.0006648199632763863,
+            0.0007326795603148639,
+            0.03280118480324745,
+            -0.0005938990507274866,
+            0.008857463486492634,
+            0.004420568235218525,
+            0.013159659691154957,
+            -0.012784135527908802,
+            -0.01769859902560711,
+            0.0008087027235887945,
+            -0.01057181041687727,
+            0.021257910877466202,
+            -0.008192133158445358,
+            0.006216551177203655,
+            -0.018612917512655258,
+            -0.027086693793535233,
+            0.006392067763954401,
+            -0.016261812299489975,
+            -0.018221067264676094,
+            0.01317598670721054,
+            0.0077268099412322044,
+            0.0017572063952684402,
+            -0.005098143592476845,
+            -0.020457882434129715,
+            -0.009837090037763119,
+            0.028441844508051872,
+            0.011102442629635334,
+            -0.004120557103306055,
+            0.009584019891917706,
+            -0.009094206616282463,
+            -0.012490247376263142,
+            -0.0017541450215503573,
+            0.0036715613678097725,
+            0.008261523209512234,
+            0.005645101889967918,
+            -0.01386172603815794,
+            -0.00029873527819290757,
+            -0.012400448322296143,
+            -0.021225256845355034,
+            -0.008906444534659386,
+            0.016751626506447792,
+            0.007820690982043743,
+            -0.0038960592355579138,
+            0.015812816098332405,
+            -0.004404240753501654,
+            0.002189875114709139,
+            0.014294394291937351,
+            0.005334886722266674,
+            -0.004285869188606739,
+            -0.0095758568495512,
+            -0.004612411838024855,
+            0.004043003544211388,
+            -0.0025960123166441917,
+            0.006457376293838024,
+            0.017192458733916283,
+            0.0072900597006082535,
+            -0.013755599036812782,
+            0.013135168701410294,
+            0.0007791098323650658,
+            0.0049675265327095985,
+            -0.03699725493788719,
+            -0.003247056156396866,
+            0.015796490013599396,
+            -0.013020878657698631,
+            0.02501314878463745,
+            0.005400195252150297,
+            0.022743679583072662,
+            0.015233203768730164,
+            0.008669701404869556,
+            -0.001542912912555039,
+            0.00464098434895277,
+            0.02759283408522606,
+            -0.003151134354993701,
+            -0.008637047372758389,
+            -0.0020602785516530275,
+            0.009241150692105293,
+            0.0002063850115519017,
+            0.003000108525156975,
+            0.008281932212412357,
+            -0.03562577813863754,
+            0.011706545948982239,
+            -0.019576217979192734,
+            0.0024694770108908415,
+            0.003459308762103319,
+            -0.007183933164924383,
+            -0.014865843579173088,
+            -0.005857354495674372,
+            0.022841641679406166,
+            -0.003224606392905116,
+            0.021078312769532204,
+            0.002885818714275956,
+            0.016106704249978065,
+            -0.011575928889214993,
+            -0.009208496659994125,
+            -0.0004130251472815871,
+            -0.005351213738322258,
+            -0.007539048325270414,
+            0.010596301406621933,
+            0.0034409407526254654,
+            0.005534893833100796,
+            0.002265388146042824,
+            0.0036654386203736067,
+            -0.0036878883838653564,
+            0.02773977816104889,
+            0.003563394071534276,
+            0.007996207103133202,
+            -0.011567764915525913,
+            0.014082142151892185,
+            -0.017029186710715294,
+            0.0070778070949018,
+            0.005575711838901043,
+            -0.026156047359108925,
+            0.016849588602781296,
+            0.004775682929903269,
+            0.006628811359405518,
+            0.013812744058668613,
+            0.01618834026157856,
+            -0.010408539324998856,
+            -0.014816862531006336,
+            0.024049850180745125,
+            -0.0032041973900049925,
+            -0.015225040726363659,
+            -0.006775755435228348,
+            4.438935866346583e-05,
+            0.006559420842677355,
+            0.00373074715025723,
+            0.0013020879123359919,
+            -0.0006204306264407933,
+            -0.010743245482444763,
+            0.004645065870136023,
+            -0.01934763789176941,
+            -0.0007163524278439581,
+            -0.016449574381113052,
+            -0.015633217990398407,
+            0.018678225576877594,
+            -0.01830270327627659,
+            0.009633000940084457,
+            0.0012275953777134418,
+            0.028588788583874702,
+            -0.011706545948982239,
+            -0.012155541218817234,
+            0.0015102586476132274,
+            0.010849371552467346,
+            0.0067512644454836845,
+            -0.010033016093075275,
+            -0.007583947852253914,
+            -0.014180104248225689,
+            -0.010163633152842522,
+            0.015502601861953735,
+            -0.014498483389616013,
+            -0.014343376271426678,
+            0.008914608508348465
+          ],
+          "embedding": [
+            -0.006359657085039302,
+            -0.024594786241399234,
+            -0.02553538924848095,
+            0.005360188465088291,
+            0.012801870977589839,
+            0.021680505862506104,
+            -0.016668048966001273,
+            0.06860852474581301,
+            -0.007122272841157724,
+            0.03471775000705488,
+            -0.009004297848861675,
+            0.01059974144678391,
+            0.02565604599426148,
+            -0.022544127646041287,
+            0.010616674976540785,
+            0.013658194823359737,
+            0.016400329743363844,
+            0.02858546964917478,
+            0.0067612676434693686,
+            -0.04174737193606879,
+            0.03184155110488616,
+            -0.02223105127585514,
+            -0.05506239667974977,
+            -0.0015307551410555752,
+            -0.013169788751379287,
+            0.017681749722384007,
+            0.01682706317229353,
+            0.019652295823816928,
+            -0.05623980369271269,
+            0.030698582851533197,
+            0.02286327124296613,
+            0.016019032539689178,
+            0.005508113353872315,
+            8.58985091215149e-05,
+            -0.019206292853495906,
+            -0.018439290483495144,
+            0.03475023176950151,
+            0.01018612942141759,
+            -0.0252551158857426,
+            0.0334562130666827,
+            0.020462153537485785,
+            0.02260508249714211,
+            -0.03323524989392951,
+            -0.002094251009530707,
+            0.015177442347686773,
+            -0.00018698464350038317,
+            -0.0006788648344273403,
+            0.0007777616893581039,
+            0.013352398615059647,
+            -0.030751111528744813,
+            -0.0317733818445192,
+            -0.006729064081710058,
+            0.009413204899986514,
+            -0.005712894407943592,
+            -0.038961826880527066,
+            -0.002827692206530345,
+            0.017863300537973137,
+            0.07918455388563496,
+            0.048967158919816366,
+            0.018002566325350088,
+            0.040264962620950695,
+            -0.019715100359659787,
+            -0.02211350500879408,
+            0.004918148608475574,
+            0.01062585305866606,
+            -0.014478993521696027,
+            0.0034983131226364672,
+            0.000766228629514787,
+            0.009667268771502067,
+            -0.01558439890697323,
+            -0.04984718709870083,
+            0.029613955432821887,
+            -0.009686854114630624,
+            -0.015279319238124568,
+            0.007460629676325322,
+            -0.006235979350809034,
+            0.012514387473717085,
+            0.02800435263741497,
+            -0.008688925946672778,
+            -0.01236301543374469,
+            0.029755705020188972,
+            0.01158037219446542,
+            -0.005242392206024511,
+            0.009982679027728814,
+            0.052414361714483264,
+            -0.018788500381726227,
+            -0.020335831196027204,
+            -0.08583959943356621,
+            0.029309228216812346,
+            -0.0249285282010673,
+            -0.018929396555426307,
+            0.007710663193334574,
+            0.03096798035958051,
+            -0.017089977064591828,
+            -0.028015107605405817,
+            0.06740703271128606,
+            0.009398795412354399,
+            0.01118769022826835,
+            -0.03041417460855722,
+            0.014776002883150835,
+            0.004319194067673161,
+            0.022507636639041203,
+            0.0053562788646786845,
+            0.007548072258943124,
+            0.011123374308563128,
+            -0.006924362115585165,
+            -0.011118507147092024,
+            0.009563232643295304,
+            0.00657178473678069,
+            0.08313027447380966,
+            -0.004239370156003264,
+            -0.00930406163946907,
+            -0.039202672821898946,
+            0.009761029930556298,
+            -0.02827698465424993,
+            -0.037914533246900244,
+            -0.04496050911855815,
+            -0.017711307900236353,
+            0.01934435542287117,
+            0.004335173027873153,
+            0.03800509969714242,
+            0.031198127357716314,
+            0.031311207941680276,
+            -0.012184553894588813,
+            -0.058985863670856876,
+            0.041442017057707635,
+            -0.012456189292721711,
+            -0.0040966351720394135,
+            0.010849179519959421,
+            0.005243123049347636,
+            0.0011495799963862647,
+            0.003534183676934487,
+            -0.03139340512003631,
+            -0.023876261883905306,
+            0.008382957569752911,
+            -0.004778283217011976,
+            0.03703067589888386,
+            0.06888630688839406,
+            -0.03993532296478411,
+            -0.009472167678222372,
+            0.025246872942105057,
+            -0.01524567912700368,
+            0.01595917939878582,
+            -0.003736723164442467,
+            -0.030836797784967753,
+            0.0026246446163368533,
+            0.008894300082148884,
+            0.08342856279154155,
+            -0.0007476580521704578,
+            0.025149116238550588,
+            0.004972542633631468,
+            0.034495632546556576,
+            -0.009173329764686424,
+            0.014009381543772768,
+            0.004427261668151028,
+            -0.013426138607902904,
+            -0.022234568880321808,
+            0.01976282364689966,
+            -0.03592433789490043,
+            -0.010918278220831188,
+            0.013527180289641174,
+            -0.013023113052242677,
+            -0.008304770302495363,
+            0.003677063799167178,
+            0.0013981822663219421,
+            0.047001163242600175,
+            0.01828332714887986,
+            0.002483375662343689,
+            -0.06820498187523893,
+            0.05295312038995555,
+            0.025038753504342162,
+            0.025286870198436735,
+            -0.0029544808216316115,
+            0.029982752196635428,
+            0.0005600481523874767,
+            0.0128290258663539,
+            0.009023805227424919,
+            -0.0019219166367750596,
+            -0.04482843936478531,
+            0.04523479691409549,
+            -0.004678574027241186,
+            0.010486850804218904,
+            0.006736229865018014,
+            0.02134813656476443,
+            0.028083972673690036,
+            0.016857413440125445,
+            0.0189082447820146,
+            0.03843690613820022,
+            -0.011922027011229613,
+            0.016193433524453224,
+            0.03828207911467065,
+            -0.0013055128404331705,
+            0.043198163443700956,
+            0.024025562893260542,
+            -0.001257851946749882,
+            -0.011726713433348844,
+            -0.021907984099443396,
+            0.02752198420597673,
+            0.004431866336999851,
+            0.015798806714413912,
+            -0.012910782191382367,
+            -0.03180549180206364,
+            0.004550247372715741,
+            -0.022778272841118514,
+            0.031242218240017532,
+            -0.006106984949693483,
+            -0.006259993190497421,
+            0.02187449047033339,
+            0.009334146847221406,
+            -0.009381278960016145,
+            -0.020888853823931134,
+            -0.007502374908776156,
+            -0.010390018386728482,
+            -0.04316693435221676,
+            -0.006274513730720055,
+            0.019152443937064258,
+            -0.01668984824845791,
+            0.018573194036505578,
+            0.0035035262183249925,
+            -0.019779911381789077,
+            -0.0027495798165085993,
+            0.02425481236315913,
+            0.006857723545270935,
+            -0.014733535699087731,
+            -0.026326146471941727,
+            -0.019906660231056833,
+            -0.001033294559612807,
+            0.0159154063026895,
+            0.035649534442431415,
+            -0.026210592096240003,
+            0.014397127229517285,
+            0.005114311681443673,
+            -0.0033104236005207077,
+            0.00324988235167305,
+            0.03764004154553922,
+            -0.017822437760737605,
+            -0.008579701063708334,
+            -0.00032410984200965487,
+            0.018450351071676443,
+            0.017283379764846195,
+            0.006384411611332382,
+            -0.005115090309123598,
+            -0.030590047062583125,
+            -0.011776211359193503,
+            0.01183623914060493,
+            0.0016564847796208331,
+            -0.009530766634361575,
+            -0.004810598258154945,
+            0.03876242515526699,
+            0.027076785774401667,
+            0.005324763136165989,
+            -0.03416412480913939,
+            -0.007943016733586697,
+            0.002114307650456928,
+            -0.02961449347300628,
+            0.05142732770518828,
+            0.05566026433842563,
+            -0.0030729309664974704,
+            -0.00938186948393074,
+            -0.04547908981236149,
+            0.0651216359692231,
+            -0.01813722445944094,
+            -0.0482650853560993,
+            0.044045155892545554,
+            -0.014272653003441165,
+            0.013740652306746851,
+            0.04563732439184395,
+            0.02126513340521316,
+            -0.0054845632439384905,
+            -0.011635625101262898,
+            0.030992082069685376,
+            -0.010051935721068779,
+            -0.050967258971854425,
+            -0.009827942095033494,
+            0.013373878556654464,
+            -0.013017627321534534,
+            -0.009361001542607812,
+            -0.01925867665425919,
+            0.020329052915436022,
+            0.023020741670660188,
+            0.019387043947695172,
+            0.005628632318152202,
+            0.018451555161952665,
+            0.016226845400848455,
+            -0.01384033112836216,
+            0.004762589697426251,
+            -0.012718476561725138,
+            -0.024222765127494463,
+            -0.037487144314010595,
+            -0.019971524166659278,
+            -0.015034588610516958,
+            0.009483936496215338,
+            -0.014301924283553591,
+            -0.0024139665139412125,
+            0.013361045353529458,
+            -0.0005870798721107379,
+            -0.011192319340879789,
+            0.013927695434391159,
+            0.021337539669342293,
+            -0.027300952495815437,
+            -0.04765670695199488,
+            -0.00561706224251562,
+            0.0082776639906139,
+            -0.0030984973749633724,
+            -0.026269016057688548,
+            0.007040524859505573,
+            0.0017859008072463442,
+            0.020091542517503808,
+            -0.018716761130867725,
+            0.0035283051960672963,
+            0.008519383548064053,
+            -0.015247417050336866,
+            -0.016917626868166634,
+            -0.018580550690103854,
+            -0.024697180580294884,
+            0.034202028316737125,
+            0.0204395003463227,
+            0.030392895534267717,
+            -0.04527669147753553,
+            -0.018359428092075348,
+            0.05494107499558624,
+            0.015234754907198422,
+            -0.05066008106884831,
+            0.031139824660661085,
+            0.005016615710206006,
+            0.0019000453624929544,
+            -0.006094364589229031,
+            0.0637324238017742,
+            -0.02711381489292883,
+            -0.02187418906973867,
+            0.004601744864532875,
+            -0.04141121277496072,
+            -0.026689585608019588,
+            -0.005568100750837504,
+            0.01291019036002459,
+            -0.031853041426355644,
+            -0.02815800259372657,
+            -0.007964861641790374,
+            -0.010933515363060793,
+            0.013726639077745378,
+            -0.023956800669647334,
+            -0.015672058590455337,
+            -0.023642025374517057,
+            -0.01288435863544158,
+            -0.0042516881552192385,
+            -0.011162214160460695,
+            -0.02412733256710621,
+            0.010088528028211243,
+            0.03310175458095482,
+            0.023064360903748363,
+            0.001996159684818467,
+            0.009478229572933101,
+            0.014478772877286076,
+            0.00371783700962415,
+            0.007864103610320116,
+            -0.048511515124609,
+            -0.031398754206534776,
+            0.020227872561659397,
+            0.012159811587094092,
+            0.0172163448580163,
+            0.009578952796095119,
+            -0.04154545961112031,
+            -0.05095080393682759,
+            0.02968486144855133,
+            -0.03932395023720674,
+            -0.02063219739061335,
+            0.007512423249865194,
+            -0.011911709468952548,
+            -0.00810951734088105,
+            -0.0008079327452253181,
+            -0.01600701450676789,
+            0.023984909975645632,
+            -0.02418072541081482,
+            0.0033984265728439398,
+            -0.0104343782811971,
+            0.01588977364765743,
+            0.05109529246167992,
+            -0.027800269136366997,
+            -0.027889034481533438,
+            0.03706813210665186,
+            -0.02058120652429308,
+            0.030801396468451574,
+            0.03246127847621613,
+            0.008355517405358826,
+            -0.008605467586247932,
+            0.017693344563968573,
+            0.014499487024471321,
+            0.012431277080478217,
+            -0.040103870071440256,
+            0.017328848567062113,
+            0.02723900253727976,
+            -0.054909074068976875,
+            -0.008547596270650966,
+            -0.00777964838164648,
+            0.027134666572242665,
+            0.02765036594970255,
+            -0.002207173084941375,
+            -0.016838740117653725,
+            -0.028289233739722936,
+            -0.019433376533829132,
+            -0.011053380458946764,
+            0.03247641283550457,
+            0.0007514989009289282,
+            0.007919574219000797,
+            0.001316042436488591,
+            -0.0017934342041762042,
+            0.01924145734197619,
+            0.015260199553572459,
+            0.048259602293134723,
+            -0.03383050622747848,
+            0.0021712225589426266,
+            -0.03508567715696004,
+            -0.04578731401250471,
+            -0.008768228935292693,
+            0.010203318819548585,
+            0.024135236371743855,
+            -0.007467140460133838,
+            0.0028907962385387818,
+            -0.007582083854422456,
+            -0.005384477718912037,
+            -0.01867978482547249,
+            0.01641191108381631,
+            0.037168089880543244,
+            0.0031992495663980464,
+            -0.008533890994695162,
+            0.000778843942291118,
+            -0.004620258538699103,
+            0.010572690335163528,
+            0.033233014627100904,
+            -0.030497362482602697,
+            0.02974571152880676,
+            -0.004274442492343003,
+            0.025341347125910928,
+            -0.04307748954707389,
+            -0.018069609916024403,
+            -0.018491122727274337,
+            -0.02069307802429807,
+            0.05372967226264594,
+            -0.030768978301976772,
+            -0.03662045886993282,
+            -0.003205991669059688,
+            0.016815511161004534,
+            -0.0014152644009276566,
+            -0.0016015439520090104,
+            -0.03984716970293867,
+            -0.010828260334550217,
+            -0.05894985435251326,
+            -0.01638035185618134,
+            -0.021819442096404948,
+            -0.005315102328745982,
+            -0.06251579337157112,
+            0.0033713876515840233,
+            -0.07815974854883057,
+            0.001484860071649606,
+            0.04539427129131382,
+            -0.023530943197353476,
+            0.05142250021536681,
+            -0.024746600229063383,
+            -0.0005885740455257141,
+            -0.044698536284421976,
+            0.0026481001040031785,
+            0.02389178603305878,
+            0.051387657872255674,
+            -0.002949198236157656,
+            0.010674377196609455,
+            0.013322075519401089,
+            -0.026245090721950647,
+            -0.01036724558050635,
+            -0.005321884180696542,
+            0.005958798671856245,
+            0.013755178085972732,
+            -0.0074734329537091675,
+            0.038368016046563674,
+            0.010941421833174982,
+            0.03443115781919206,
+            0.034623617136788874,
+            -0.004361926531008214,
+            -0.03348681614127998,
+            -0.00632870804659212,
+            0.024433331840566692,
+            0.0321998807702602,
+            -0.02899983280320611,
+            0.005958471722097962,
+            -0.015602447981516634,
+            0.015633217539747047,
+            0.006321931116948646,
+            -0.011827541320878442,
+            0.02372204564999124,
+            0.015673426811623124,
+            0.006103300725621424,
+            -0.005165006670545588,
+            0.013589526056189892,
+            -0.025411864292353578,
+            0.03041791947375232,
+            -0.004456865298584878,
+            -0.04276058054583262,
+            -0.0024736078733106825,
+            -0.020059309867945227,
+            0.024883721794000956,
+            0.015578237413840115,
+            -0.00536523508503184,
+            -0.005290645798641171,
+            0.008242987948511412,
+            0.020469675635821673,
+            0.006668354647245447,
+            0.00017729517736814364,
+            0.02373604785121342,
+            0.029915687122900485,
+            -0.02432635861789771,
+            -0.029540533943782886,
+            0.008151297444985344,
+            -0.005730733861939129,
+            0.011318816579801282,
+            -0.02644855229221747,
+            0.00033932872948409137,
+            0.0035505684154028996,
+            -0.03570104343808711,
+            0.03408366016010711,
+            -0.009199700348630527,
+            0.005857598530788056,
+            -0.009538177736472163,
+            -0.0017768806148210118,
+            0.0027410153099351694,
+            0.03898108970435115,
+            -0.007367623392751247,
+            0.009827254226889936,
+            0.015465895035342481,
+            0.000981604396724216,
+            0.05143083558797524,
+            -0.027350877485564314,
+            -0.02183036962301852,
+            0.01885468378168636,
+            -0.007285239007489837,
+            -0.016846837665812908,
+            -0.020546383007536074,
+            -0.011863943814684079,
+            -0.016293192750336218,
+            0.01976516340568883,
+            -0.02387328798681798,
+            -0.011892336568452852,
+            0.007170651391639504,
+            0.0013725014050721622,
+            -0.009809669068563384,
+            0.0016617183704652932,
+            -0.006307590349798603,
+            0.02037693835615797,
+            0.031753656615094446,
+            -0.024892675105483313,
+            0.03577919637821846,
+            0.03178303188315581,
+            0.017260898395251155,
+            -0.02102289690701718,
+            -0.009661035772073294,
+            -0.00048542689442823136,
+            -0.014694631967051945,
+            -0.0326379186158631,
+            0.025443206465204822,
+            0.017194626456982255,
+            -0.011381421003142453,
+            -0.03680783761649673,
+            0.0017216115811536721,
+            0.022819859205829785,
+            -0.013600989058799166,
+            0.028141657788496834,
+            -0.01869047298692401,
+            -0.01366833755869951,
+            -0.00782959271979894,
+            0.031227914441058244,
+            -0.05252392878125891,
+            0.016957842026348195,
+            0.00020880696574616416,
+            0.001154370575234767,
+            0.010892067708194324,
+            -0.008368819756628046,
+            0.012224564683325483,
+            0.031700389750709,
+            0.018063405869147564,
+            0.0010794115307202266,
+            -0.002783668216641561,
+            0.01418234057248712,
+            0.017773269840982066,
+            -0.015653443355569053,
+            -0.000497740485657824,
+            0.019890882604100452,
+            0.022672054781919238,
+            -0.00521797370595221,
+            -0.016686942830774276,
+            -0.018443538351123584,
+            0.025270057639201107,
+            -0.010354431070738671,
+            0.03484047753830186,
+            0.04594785214094975,
+            0.0036441959528283004,
+            -0.05695476763071743,
+            -0.009412209619190693,
+            0.0010371110229966086,
+            -0.01099343927053564,
+            0.01606863921935641,
+            0.012470660511773616,
+            -0.03237187923799263,
+            0.006388672213167589,
+            0.012946570716567364,
+            0.022152554488829627,
+            -0.02540317621005867,
+            -0.023183252921885226,
+            -0.0030984019855787655,
+            -0.007890467768631811,
+            0.018648760402624687,
+            -0.008653405980661855,
+            0.0006228315253506687,
+            0.009672526897422602,
+            0.0005436494014569835,
+            0.011632326145022935,
+            0.005655942007108228,
+            0.021736827172951,
+            -0.001256608167230203,
+            0.013124413886287987,
+            -0.01986808920858981,
+            -0.0270450339809516,
+            0.006500423802582728,
+            0.04438242119211055,
+            -0.014813399694814117,
+            0.030621709833336064,
+            0.0092171224268983,
+            0.016433384739642534,
+            0.024875072913970443,
+            -0.00891470467241686,
+            -0.014124009014664287,
+            0.00845777541223868,
+            0.009019007848927453,
+            -0.04147358041382793,
+            0.012513030150663913,
+            -0.01710243399313324,
+            -0.033144939719206276,
+            0.01904674924133127,
+            -0.00534122554342279,
+            0.012367320751926314,
+            0.004270315800911569,
+            -0.002886460711688153,
+            -0.025696286516581048,
+            0.014137901708875033,
+            0.028135087538950946,
+            0.021797869775098042,
+            0.0296173052503884,
+            -0.012757078368361444,
+            0.0005460601780756187,
+            -0.008748930817524899,
+            0.005470975782187157,
+            0.04187083128854477,
+            0.0008687182830616019,
+            -0.0008032802520818105,
+            0.01507726552487313,
+            0.024945744517309208,
+            0.0376130540370386,
+            -0.015415560911023553,
+            -0.03407353556457147,
+            0.01564969972264396,
+            -0.010104443707865262,
+            0.018855792875436674,
+            0.005503259932317811,
+            0.0049634097823351596,
+            0.0038600156268000702,
+            0.019312419743287703,
+            -0.006635193789316492,
+            0.018798045806429395,
+            -0.02500983683871332,
+            -0.028785757874583755,
+            0.01829411751138161,
+            -0.005277972443258226,
+            0.03338623299668019,
+            0.016225885095995672,
+            -0.029522280454919586,
+            -0.01712097344863975,
+            0.01553706187924322,
+            -0.008465766956343768,
+            -0.024842700926275538,
+            0.01895228828512998,
+            0.007076862934523977,
+            0.008956105593078965,
+            0.0024373779389833218,
+            -0.01106503696703573,
+            0.01125466547367417,
+            -0.000818223040067914,
+            -0.015322790791698395,
+            0.014157131555941659,
+            -0.017853386731281715,
+            -0.014994640991761465,
+            0.01775991780643474,
+            -0.037498078532902435,
+            0.009570197712419421,
+            -0.007687297543935123,
+            0.009050494195378254,
+            -0.004222041469820445,
+            0.007460383234429904,
+            -0.0010804379186060264,
+            -0.014987857934522812,
+            -0.014903749199810126,
+            0.03228485430019625,
+            0.022096121825475306,
+            -0.000410816936759558,
+            0.006728733918931589,
+            -3.971636461500232e-05,
+            -0.026719848032785824,
+            -0.011253751790407687,
+            0.02245467425830765,
+            -0.005664635254398481,
+            0.005134496095445943,
+            -0.01109590855846966,
+            -0.0149616794609307,
+            0.0019726017255571673,
+            0.032668637270108894,
+            0.06066635618119076,
+            0.012001732730659747,
+            0.024422093585231484,
+            0.0189499396761026,
+            0.02119276033547528,
+            -0.008489818917377516,
+            0.006413901834269739,
+            -0.03097413287789443,
+            0.03509225573986143,
+            0.005392594635384506,
+            0.03108937318306421,
+            0.026664331248813712,
+            -0.04722525921137681,
+            -0.0058539171830594774,
+            0.017087531454837677,
+            0.022673712717885323,
+            0.0159020681868943,
+            0.0008382498574190522,
+            -0.008938174268517353,
+            0.035650940044285984,
+            -0.03315250299348084,
+            -0.023647111263155,
+            0.005059382024092534,
+            -0.01439537467317451,
+            -0.010488359956731476,
+            -0.006785295580877329,
+            0.02776449681317373,
+            -0.005584052419598926,
+            -0.004582019433008779,
+            0.009697064252638535,
+            -0.014400206558190682,
+            0.008165676294736836,
+            0.0007624778531368436,
+            0.008681263897172326,
+            0.039249866981334126,
+            0.0005918063050375113,
+            0.024275699182767528,
+            -0.004138313072843555,
+            0.011079529739650594,
+            -0.0158972266904151,
+            -0.016985058543605,
+            0.02276982363172991,
+            0.00351101120496393,
+            -0.03244126813432563,
+            -0.00768819222024939,
+            0.00928180979884261,
+            -0.0013793404539641824,
+            0.005266058566444064,
+            -0.00503433670609399,
+            0.003747448261046857,
+            0.03286230666764075,
+            0.0022383592721685153,
+            0.008606190020480513,
+            0.013593947677257625,
+            0.00030951915677929204,
+            -0.01490307680740127,
+            0.013254591138113545,
+            -2.1379308136707722e-05,
+            -0.012900073369514622,
+            0.03357519186606058,
+            0.006705902094864628,
+            0.04243347200485736,
+            -0.03302953674711988,
+            0.009031534935013593,
+            0.01822227379789576,
+            -0.008491770335368761,
+            -0.02725412856390648,
+            -0.03630788717786838,
+            0.008259145747740535,
+            -0.019003260334340084,
+            0.015781275447792516,
+            -0.010039277287105153,
+            0.029665124310477384,
+            -0.03563370794521629,
+            0.004714858110268169,
+            0.004053913355055313,
+            0.0009503673086552651,
+            0.02068360255395961,
+            0.018524093113914366,
+            -0.002785698934204024,
+            -0.031443835341186485,
+            -0.02066365433385339,
+            0.006869321907853112,
+            0.02431260052207595,
+            -0.010367048093621498,
+            -0.005612149485859478,
+            0.0006316058139530722,
+            -0.024369562712568606,
+            -0.016845124984022828,
+            0.030359419843284823,
+            -0.040974427176153246,
+            0.005252911449822132,
+            0.04272936053727608,
+            -0.04599092715016919,
+            -0.03802288716943514,
+            -0.002314988728575789,
+            -0.024124014894379466,
+            -0.006360771882090754,
+            -0.002068030981218464,
+            0.016669991113175926,
+            0.0036232707799735537,
+            -0.028763764013482795,
+            0.00943401978066451,
+            -0.005173988601101547,
+            0.003971420424839974,
+            -0.0031669592494977393,
+            -0.02216336363004636,
+            -0.01657772032717298,
+            -0.02531962163186398,
+            -0.035174102848865116,
+            -0.0015903434876311088,
+            -0.005158703150821607,
+            -0.028703075279214722,
+            -0.03941940849540492,
+            -0.021696000925262537,
+            -0.0160221488026189,
+            -0.00843213135542359,
+            0.040641882367348774,
+            0.021319060666253945,
+            0.019336844555345886,
+            0.0025252167889382917,
+            -0.020346900609125973,
+            0.049440028814233025,
+            -0.007642182460707816,
+            0.035628691886108926,
+            -0.0017882888000877158,
+            -0.013461669667950893,
+            0.00455558389906464,
+            0.03825428026120858,
+            -0.014248434654950971,
+            -0.0009902062564027718,
+            0.00027811655483232545,
+            -0.0011866253757890136,
+            0.0008535155950871286,
+            0.015433650396777696,
+            0.001307651096343185,
+            -0.007112199543314309,
+            0.030380673552683906,
+            -0.012803247194895328,
+            0.01637501752192592,
+            0.018350814869962304,
+            -0.0013359719137099667,
+            0.008064546833360553,
+            -0.021430659268086814,
+            -0.00731572118780415,
+            0.0044578576331236325,
+            0.008422210247967642,
+            -0.015096885054883586,
+            -0.009964588397149773,
+            -0.02359494467299634,
+            -0.01174145797396426,
+            0.0023855749166744366,
+            -0.004559202896782688,
+            0.004902138778592298,
+            0.034595817216889944,
+            -0.016068744500427584,
+            -0.035106033294403614,
+            0.009997207702122002,
+            -0.02663088875167486,
+            0.02222306406130687,
+            3.983384942889326e-05,
+            0.006838221769859366,
+            -0.0023434342406307616,
+            0.001765445386886839,
+            -0.00169484278667461,
+            0.002862331185439823,
+            -0.018217975204395134,
+            0.0031859734246840257,
+            -0.0032119988065673565,
+            0.0009052950782312657,
+            -0.0012459653052875638,
+            0.00687339892509406,
+            0.006332799551164718,
+            -0.0008019538898592668,
+            -0.002968451740267742,
+            -8.541086355097327e-06,
+            -0.020561816931299912,
+            -0.021642512405352372,
+            -0.028259049261410905,
+            -0.000989887278779396,
+            -0.012359733017834338,
+            -0.027804394906433664,
+            0.011443385424045703,
+            -0.006872684999639775,
+            0.008327844771213648,
+            -0.008969527138034904,
+            0.006219994886600349,
+            -0.008415106130309062,
+            -0.0004143996206881767,
+            0.013119286511794273,
+            -0.02925209282917943,
+            -0.02096907906398536,
+            -0.03096513357712367,
+            0.01762223841790055,
+            -0.00925741670612824,
+            -0.002656382260603827,
+            0.0025223527960181835,
+            0.005092659166535706,
+            -0.033606542487570154,
+            -0.006653438998497412,
+            0.007313682256676618,
+            0.009547290751154595,
+            -0.03287788897629887,
+            0.02446728061355753,
+            0.017023222076526946,
+            0.01874233066565012,
+            -0.02417566312664911,
+            0.04049827067897005,
+            0.014426879325647042,
+            -0.021740606525480655,
+            0.016712907564723732,
+            0.027385150577780248,
+            0.03177163221499527,
+            -0.03469098547555545,
+            0.004481376248622418,
+            -0.01518200063132508,
+            0.018918680020516044,
+            -0.025284921244460287,
+            0.020625170445627613,
+            -0.016707629602243063,
+            0.021278719363178687,
+            0.03520334001264928,
+            -0.005524369131116028,
+            0.03608133014180745,
+            -0.019234548780647943,
+            0.0059985801736444555,
+            0.004041778461022664,
+            -0.01645503683802124,
+            0.01024694092614158,
+            -0.0042700351372843386,
+            0.0006059159967286885,
+            0.009406090340935747,
+            0.019133229156115282,
+            0.019007219188041775,
+            -0.04442208547053466,
+            0.028588965123434923,
+            0.00804120101594811,
+            0.021472086070876265,
+            0.014177820346835521,
+            -0.02708807970982664,
+            0.004742981711172693,
+            0.002072673190279628,
+            -0.004659713164250794,
+            0.0061167904730676514,
+            -0.013796604518930328,
+            -0.034540857808309226,
+            0.006338792319814448,
+            -0.018918703778187926,
+            0.015036937364271984,
+            -0.009026670935442384,
+            0.0011667525636770179,
+            -0.01152652811400482,
+            -0.011938679695598347,
+            0.0006170592505446954,
+            -0.005495151258143897,
+            -0.0008075736908949407,
+            -0.03171721592827769,
+            -0.04427658539412448,
+            0.02036132271908089,
+            -0.0021221965958423804,
+            -0.04320396743189005,
+            0.004111179168317842,
+            -0.025352428814134187,
+            -0.01865918186335911,
+            -0.011237699157960728,
+            -0.025047923887906474,
+            -0.04219630696740606,
+            -0.011108496740166623,
+            0.007785006066683012,
+            0.016810642610918525,
+            -0.009081815651299525,
+            0.040965574143584864,
+            0.0073944364393447965,
+            0.015313576815177263,
+            -0.017731360025694955,
+            0.026137777790243826,
+            -0.019314392060123,
+            -0.03108486330987073,
+            0.02676829566273746,
+            -6.259161089416372e-05,
+            0.003053748984540415,
+            0.014358356168574803,
+            2.8186924071913628e-05,
+            -0.006181789640967465,
+            0.01947730374003864,
+            -0.028739300217145797,
+            -0.023460503178221397,
+            0.0012460072421725688,
+            -0.008850395570726385,
+            -0.016271242375881602,
+            -0.035896509136022194,
+            0.005227881053956234,
+            0.03324337105109496,
+            0.004970419987430122,
+            0.008273497228454482,
+            0.01814500594735864,
+            -0.02229593571615308,
+            -0.024230675455380678,
+            0.000639112529814716,
+            -0.010266640429960994,
+            -0.010257247303470066,
+            -0.021302483296731352,
+            0.019150757066356872,
+            0.01651375496287087,
+            -0.00390189952706486,
+            -0.006971692709239087,
+            -0.010621003777733253,
+            -0.01918206725910585,
+            -0.02020345113211036,
+            -0.013827322378214754,
+            -0.017980206764311026,
+            -0.034153741666571154,
+            -0.032195001867229935,
+            0.03225205339201753,
+            0.026354817588919596,
+            0.0014076409241492998,
+            -0.007959081015805435,
+            -0.023070419006792155,
+            -0.004483328183581872,
+            0.02879516595345542,
+            0.002549934782150634,
+            -0.0001541849716285838,
+            -0.00830090759167373,
+            0.015341095409025125,
+            0.00835695019244165,
+            0.005107844984733648,
+            -0.06676974053142512,
+            -0.006027540973808174,
+            0.01655283585235268,
+            0.01351155740684989,
+            -0.002359114145814062,
+            0.02591301169926259,
+            0.01108099264396926,
+            -0.015361038608114896,
+            -0.019765743462262668,
+            0.035195761515422364,
+            0.02408910733949848,
+            0.023468767772473746,
+            -0.0034492999385544727,
+            -0.004472311490618405,
+            0.007030833258457338,
+            -0.010684252427867153,
+            -0.013326964647936429,
+            -0.004558513842504658,
+            0.026925146859337397,
+            0.016099443021514896,
+            -0.0064331198280825575,
+            0.010848931851939124,
+            -0.0008320209125061849,
+            -0.019444491995041537,
+            -0.001841330829156697,
+            0.008730275423516445,
+            0.0012974082030866743,
+            -0.014611902493400719,
+            -0.005911807768916134,
+            -0.019130604101278263,
+            0.024719468317569562,
+            0.0209456329723249,
+            0.03381727436368338,
+            -0.013237902591972769,
+            -0.0003695142331921278,
+            0.005410360755058108,
+            0.0002841632253252816,
+            0.0023352645111704958,
+            0.004801123870325242,
+            0.02623649813035263,
+            -0.029010710171522985,
+            -0.020741774066460415,
+            0.0068426980682519505,
+            0.013445270043414486,
+            0.0022744032984779294,
+            0.033840485942696216,
+            0.035040080489222324,
+            0.004166066423508316,
+            -0.010046666684988869,
+            -0.006148765564658713,
+            0.007071740742464309,
+            0.01865897335451031,
+            -0.023267411753900076,
+            -0.0005936657905496225,
+            0.004833030519903764,
+            -0.004229582648167378,
+            0.03620610659071198,
+            -0.009961238880412731,
+            -0.007197670122096852,
+            0.022212791947347127,
+            0.09819205436926473,
+            0.00047695621444943017,
+            0.0025354521225913395,
+            -0.029358210234742176,
+            -0.023614016211567963,
+            -0.0012564198366457442,
+            0.00242680708457405,
+            0.00830167274308547,
+            0.028076260517106013,
+            0.0004852400049405906,
+            0.011579274037419117,
+            0.02773972145320728,
+            -0.017217305534161797,
+            0.006752070301141707,
+            -0.011376051474975524,
+            0.024577071515205502,
+            -0.012694034026935636,
+            -0.0019809719792138673,
+            -0.016512649783050146,
+            0.015907402258634816,
+            0.004556574687833768,
+            0.004907510649166062,
+            0.011347630984408937,
+            0.009824484201337644,
+            -0.02650731900265866,
+            0.01756773113723491,
+            -0.005671986127099093,
+            0.004449611508541289,
+            -0.027841346572384462,
+            -0.020124233970502267,
+            0.0025986391930124847,
+            -0.0014440758308989809,
+            -0.009579444248630085,
+            0.006005274872617889,
+            -0.0024984235450999127,
+            0.03765664613203687,
+            -0.03765496603611941,
+            0.01409350051861866,
+            0.0066453175608372555,
+            -0.0015390841089599882,
+            0.011806653942640485,
+            0.017306914776093522,
+            -0.008206256939290151,
+            0.02589800448884443,
+            -0.00798213351353373,
+            0.009846274212000523,
+            0.015230070803055296,
+            -0.03449531102975745,
+            0.009174739184806312,
+            0.010145762060174144,
+            -0.024165660737765746,
+            0.012146877603301336,
+            -0.013784371016298891,
+            -0.004818759788054113,
+            0.005930446700613062,
+            0.007349837554764034,
+            0.010733731974181452,
+            -0.02295571464086674,
+            0.016176424826629898,
+            -0.010947841926210105,
+            0.0037766665765835445,
+            0.013559846998199008,
+            0.024892849808979323,
+            0.0012426907454129122,
+            0.0005950515136086193,
+            -0.01379469431504576,
+            0.005534724429523084,
+            -0.015667467221535825,
+            -0.022401586867694584,
+            -0.014192041775234079,
+            -0.006366846766730035,
+            0.01029598687444803,
+            -0.003972386708834858,
+            0.016971476088808737,
+            -0.010309648640697812,
+            -0.010645860303528099,
+            0.0037818310327362103,
+            0.01040806357838676,
+            0.008316605962234849,
+            -0.02855914079113881,
+            -0.005676310556150579,
+            0.004961084138163161,
+            0.00029285971503001265,
+            0.009592497954559435,
+            -0.014556073011576487,
+            0.013689526487160009,
+            -0.013290113354287095,
+            -0.0010726537498214612,
+            0.013201860397060236,
+            -0.012443552043930505,
+            -0.004012305744087929,
+            -0.01592905426965635,
+            -0.028487066798154617,
+            0.008462143660968396,
+            -0.01365614159523485,
+            0.001076882584445294,
+            0.0019503240125505646,
+            0.004730085646042709,
+            -0.016919069570642004,
+            -0.007860046623166481,
+            -0.033515448866521046,
+            0.011065416478687504,
+            0.02314589687882036,
+            -0.006435733825418645,
+            -0.030351727858789472,
+            -0.009546333526852862,
+            0.011327489786114963,
+            -0.0071765122743172715,
+            -0.01676736383051139,
+            -0.0029390038145944096,
+            0.007617910650439855,
+            -0.008174620754097493,
+            -0.00508943241317381,
+            -0.0025173969010414098,
+            0.0217328037301181,
+            0.016094975605203832,
+            0.01769642142712967,
+            -0.021400862603034036,
+            -0.00976992221747187,
+            0.026043879119776864,
+            -0.03275550610951898,
+            0.012441465975914529,
+            0.024587446266013183,
+            0.003446527051428522,
+            0.005492228054838714,
+            0.0030377554109693714,
+            0.005459159163772825,
+            -0.006985179226831048,
+            0.019925583798328282,
+            0.010870421035240845,
+            -0.019709062313471467,
+            0.005385443078244229,
+            0.02086085539375208,
+            0.0037248737640193097,
+            0.012892299156840998,
+            -0.017546870090541213,
+            -0.0020341807891543716,
+            0.015594779287017958,
+            0.04816214672670215,
+            -0.0032851797178091685,
+            -0.010450846098750195,
+            0.029513742590677183,
+            0.023526004023637855,
+            0.005373455940309873,
+            0.0012120777808635214,
+            -0.02525630456877349,
+            -0.013075130701774785,
+            0.01962235086017781,
+            0.016944870494393874,
+            0.013484925533584295,
+            -0.0023255656480479004,
+            -0.019236081983457366,
+            0.021520156296467717,
+            -0.01701733319731161,
+            0.012422659010697477,
+            -0.023764941037007197,
+            -0.001490790833263327,
+            -0.0035419008954878724,
+            0.021998949056452454,
+            0.0021819635638621092,
+            0.008891505304514552,
+            0.024354552127458608,
+            -0.006978548952551799,
+            -0.019526615985982355,
+            -0.0051799020400150305,
+            -0.014578953088901798,
+            0.022506635912517146,
+            -0.024117708614271965,
+            -0.007008641019892289,
+            0.011379795968360019,
+            -0.012362093847309038,
+            -0.0102534051092702,
+            0.001164934138876502,
+            0.017686807394319725,
+            0.0009423993435129515,
+            -0.014784508280754078,
+            0.01944445593807992,
+            -0.0011359271135469456,
+            0.010410915597949927,
+            -0.01921419010977189,
+            0.009390638453648025,
+            -0.016682442516992777,
+            -0.007849184048209236,
+            0.010343008873053859,
+            0.0012604521199837046,
+            0.003948300703645506,
+            0.002592490121754231,
+            0.007773437307969315,
+            -0.03355658426684504,
+            0.022695505487314516,
+            -0.010751606023585792,
+            -0.003538523688786008,
+            -0.0014125330167969218,
+            -0.01678692701672328,
+            -0.0006918162525881059,
+            0.005816502852598842,
+            0.010878358093847386,
+            -0.016045147059003256,
+            -0.00873270513444376,
+            -0.04920244395446931,
+            -0.030314009649638693,
+            0.023156333024481057,
+            0.013733657599720879,
+            -0.010971506342695331,
+            0.018852159415605117,
+            -0.023760853159532587,
+            -0.010587249404421732,
+            0.0011379869499042905,
+            0.023319856329056813,
+            -0.03539290744576506,
+            0.01215369703816749,
+            -0.014493682360886549,
+            -0.02039288669534806,
+            0.009070984596830142,
+            0.011445354735283254,
+            0.0007259852045513916,
+            -0.0011514627555568814,
+            -0.029350945123925453,
+            -0.010246845821727087,
+            0.02391068177425878,
+            -0.00861718589055971,
+            -0.004723044986826333,
+            0.024716520302409263,
+            0.0033721801437248873,
+            -0.012524663490691517,
+            -0.017955758068233754,
+            0.021612671655463236,
+            -0.026077203994996058,
+            0.007645944309444457,
+            0.010032183011253845,
+            -0.0173991263874487,
+            0.021649599230588117,
+            -0.007529701797485131,
+            0.020327763440441377,
+            0.004227390938169872,
+            -0.011249233530251522,
+            0.020352283974471443,
+            0.013644443758970054,
+            0.004216958800094986,
+            -0.004081669404784771,
+            0.006869353098014179,
+            0.02212732880780841,
+            -0.011021739614266063,
+            0.02219580717345023,
+            0.028777213295471067,
+            0.00017328191188631918,
+            0.03260179554743322,
+            -0.021287655520443285,
+            -0.03159728983004563,
+            0.021740163175540124,
+            -0.01035146858119169,
+            -0.021564906188712388,
+            0.028914245422221707,
+            -0.013674731836508768,
+            -0.00786476052825481,
+            -0.0166718429993922,
+            0.00533027311342322,
+            0.016680336311088843,
+            -0.005246190451159961,
+            -0.014360934066628332,
+            0.0029628103834216345,
+            0.019809487418628386,
+            0.014587234544007958,
+            0.029780009648869975,
+            -0.010778199815920887,
+            0.035180242185168895,
+            -0.002320576872571577,
+            0.008990157317473909,
+            0.012415465232073404,
+            0.006489435152972723,
+            -0.011251531176923445,
+            0.013501894212117511,
+            -0.003446759011266526,
+            0.008230548711588447,
+            -0.001449023474280409,
+            -0.035111679577202855,
+            -0.013709382199848678,
+            -0.04709294504969336,
+            -0.0023752976043235697,
+            -0.0025098360125255215,
+            0.0030975417740913154,
+            0.09736815869389318,
+            -0.005723049647039619,
+            -0.011988817571623457,
+            0.001625059666070763,
+            0.009611697638805556,
+            -0.0009896494420056814,
+            -0.007606836952841873,
+            0.015973152482488318,
+            -0.01343292635775299,
+            -0.02140928016038583,
+            -0.010532701190806147,
+            -0.01288594454368977,
+            0.012079198190522715,
+            0.015929716343161652,
+            -0.0005947359502228821,
+            -0.015207331663761012,
+            0.006524010655124294,
+            -0.004566385899752684,
+            0.0064717704997400245,
+            0.008647099975176308,
+            -0.010414668624343924,
+            -0.0071698879102925555,
+            0.030899143952119196,
+            -0.025301269549419674,
+            -0.024361748354052457,
+            -0.0010579931222861095,
+            -0.009055796737486008,
+            -0.010335459737012047,
+            -0.002692653633719985,
+            -0.0017679501322462056,
+            -0.010519994985692081,
+            -0.010019865263564572,
+            0.006537076627706128,
+            -0.015472130358273033,
+            0.004812857776601667,
+            -0.0047457011269495634,
+            -0.013296069496526007,
+            0.017054968086538724,
+            0.022563785357424853,
+            -0.02841491200807702,
+            0.0006755428946168761,
+            -0.017422510783648978,
+            0.012941211381006863,
+            0.008866069010589792,
+            0.006049618782357251,
+            -0.022451528209097505,
+            -0.023622906582403243,
+            -0.015019272754405791,
+            -0.01927535750568903,
+            -0.024973490129133732,
+            -0.009073381957684092,
+            0.0067692803507600945,
+            -0.001267339788153158,
+            0.021918281111033645,
+            0.000816538714279555,
+            -0.01812545613159305,
+            0.0061084913570894905,
+            0.000568371220952269,
+            0.018292497018668842,
+            -0.027468677288882186,
+            0.0033806422195878524,
+            -0.02307582511848465,
+            0.033508234091571124,
+            0.0039195050746594975,
+            -0.026680111514794824,
+            -0.021558668725755275,
+            -0.003311830287529699,
+            0.018994577319066473,
+            -0.0025745912019675805,
+            -0.00724049447539589,
+            0.014775207617625293,
+            0.0155334760323838,
+            -0.004693118176196483,
+            0.002015625406296403,
+            0.027968079002437102,
+            0.01077998866180723,
+            0.0313892866933904,
+            -0.015236169297590675,
+            0.004945828495617267,
+            -0.027215016713379062,
+            0.010214419950212732,
+            -0.01503548137805401,
+            0.008859554812646242,
+            -0.01290890682557532,
+            0.03531384210182567,
+            -0.012005613861257432,
+            -0.004250698950192301,
+            0.03253288796441852,
+            -0.029150986393345252,
+            -0.0035020051536906334,
+            -0.009255437891304041,
+            -0.004142204892160768,
+            0.013240580426083011,
+            -0.001198093874819309,
+            0.008485141356876356,
+            -0.01854370462655459,
+            9.726096864073609e-05,
+            -0.003221852304410318,
+            0.005404595442127598,
+            -0.02005458688172967,
+            -0.003314118809778734,
+            0.0073427568419969814,
+            0.004259518237708634,
+            0.002408630356035373,
+            0.016420646142725346,
+            0.006662121128241274,
+            -0.0029357701493526907,
+            0.03424224243607692,
+            0.008171895850772245,
+            0.0152521424877615,
+            -0.0018638264193202728,
+            0.0030234211261443034,
+            0.019287963516760636,
+            0.019912956661192856,
+            -0.00048443399729935116,
+            -0.01058935153883094,
+            0.00195113175754497,
+            -0.001140732137433033,
+            0.011048741602327437,
+            0.024396046054912645,
+            -0.021158025221644738,
+            -0.02419562092713724,
+            0.016377661005279716,
+            -0.015076145951740961,
+            0.01903712233378994,
+            0.010541463112889985,
+            -0.010099351302095792,
+            -0.00031558616108876247,
+            -0.042316016201379245,
+            0.010964989031040692,
+            -0.02068939391415103,
+            -0.04004120028915868,
+            -0.00244383573783919,
+            0.0009542507389088937,
+            -0.029439456243505052,
+            0.02822814855547978,
+            0.0023755032182188156,
+            -0.017280221740813364,
+            0.018223709137876153,
+            -0.005548654680726505,
+            -0.0024040301239229064,
+            0.005397711787238679,
+            -0.01092026090471121,
+            -0.010648485086769927,
+            0.008235250252660972,
+            0.010957715075632535,
+            0.005425683656535705,
+            -0.005698919020608095,
+            -0.011284911899894062,
+            0.009975882651638084,
+            0.012318901214538077,
+            -0.0019922884373083155,
+            0.010050520101815593,
+            0.0031720752428247446,
+            0.007352859750323051,
+            -0.0073483049386141005,
+            0.0046483587496626965,
+            -0.005093220128411125,
+            -0.026080057085901446,
+            0.012863580653903103,
+            0.0052434400313520774,
+            -0.011863360599637747,
+            0.0103209502958491,
+            -0.01048162756158923,
+            0.005787157766911021,
+            0.001318945460232246,
+            -0.0059333230422061395,
+            0.004821320238893602,
+            -0.0038922239931512784,
+            0.011267634571983924,
+            0.013256611037442042,
+            0.004592529900805719,
+            -0.013076954459595086,
+            -0.02177318635989856,
+            0.009001291224623467,
+            0.01959767163356408,
+            0.024161754192119852,
+            -0.030416109293488064,
+            0.003784634588325551,
+            -0.0014915849060829743,
+            -0.0020550564975080885,
+            -0.015632785947151753,
+            0.01587790981601866,
+            0.01716742056401084,
+            0.012721563202528083,
+            0.01893393006967701,
+            -0.004009617703771589,
+            0.010539100665843704,
+            -0.02080068888791658,
+            0.013642451766699067,
+            0.0008609869904066794,
+            -0.013629067231663643,
+            0.0014620655123815298,
+            0.027046304278766087,
+            -0.011056783825898616,
+            0.006206969699250696,
+            0.008097527175511791,
+            0.022111870736907183,
+            -0.012509479100313492,
+            -0.003735868296238069,
+            0.01516254968167132,
+            0.0014878674796313539,
+            0.0035887079417890352,
+            0.016533149527222273,
+            0.04729052276110659,
+            0.00102751186013391,
+            0.0016713773449738153,
+            0.0192177920520243,
+            -0.005563519063161466,
+            -0.01114821419109643,
+            -0.003549999863325982,
+            -0.021807893906469106,
+            0.010998558936481785,
+            0.00431279785115251,
+            -0.02186324212032519,
+            0.01024093366826206,
+            0.0125495473550584,
+            0.005268320135120931,
+            -0.0031679212410026603,
+            0.02574575247827441,
+            -0.019807879421996258,
+            0.014935295894965128,
+            0.03329450628384787,
+            0.00493356778223407,
+            -0.011423949333937285,
+            0.007263264249871384,
+            0.0027354459678742013,
+            -0.004494406041906684,
+            -0.00606745413660583,
+            0.010497391290734088,
+            -0.008191702034391382,
+            0.009870458191510176,
+            -0.009580130479693,
+            0.039202091799313755,
+            -0.009619018331262844,
+            -0.010782138248435458,
+            -0.004558645539551881,
+            -0.006241773108460662,
+            0.003478414182938609,
+            -0.003395165694642073,
+            0.005812225135633631,
+            -0.0069045407353789945,
+            -0.014607146220271209,
+            0.0008522982757210454,
+            0.00020175973815948303,
+            0.014001922919116725,
+            0.0011804904415274088,
+            -0.004871874532755353,
+            -0.007877217154706205,
+            0.015243277056007988,
+            -0.006047906208304912,
+            0.0006531937033317192,
+            -0.0011653523616313422,
+            -0.0013662556399359335,
+            -0.007401050012179761,
+            -0.015166105795085528,
+            0.00974609244984173,
+            0.016253733908242134,
+            -0.013857834499211056,
+            -0.005142936208363415,
+            0.012274716212832219,
+            0.008540229125193836,
+            0.011742606168187876,
+            0.0007451338650642946,
+            -0.009255813682572937,
+            0.008292531177374192,
+            -0.015020523035432158,
+            -0.008781317965985481,
+            0.0026250469418136603,
+            0.002381147382977664,
+            0.005839723981046825,
+            0.0006240641828668715,
+            0.004502463508605272,
+            0.005798912705218846,
+            -0.00497936644919367,
+            0.01112666823857836,
+            0.008198424171006548,
+            -0.002452143482105151,
+            -0.01017569410837814,
+            -0.001127589984272638,
+            -0.015733298167661346,
+            -0.016758315411028353,
+            -0.0009286266498936771,
+            0.015175988113750882,
+            -0.02352862898718829,
+            0.018996352975391644,
+            -0.0014524363611708201,
+            -0.0033802546171857704,
+            -0.021392428456030312,
+            -0.0025923983911570183,
+            0.0034214188668518796,
+            0.01255768230394476,
+            0.01858853630745928,
+            -0.02729543519465384,
+            0.010575894388563116,
+            -0.007204725245442141,
+            0.030394929424014717,
+            -0.018328782140005044,
+            -0.004022941273610227,
+            0.026915618079621992,
+            -0.02200018306828758,
+            0.0008791185377513978,
+            -0.011454606955158956,
+            -0.008103032693853504,
+            0.019287674837688177,
+            0.023747720480736777,
+            0.02038287242167783,
+            0.01364489294869211,
+            0.0007758044919789085,
+            0.004768030222445397,
+            0.03572705648994403,
+            0.010568241270862248,
+            0.014828087092859878,
+            -0.007542489523110974,
+            9.457808640264811e-05,
+            -0.02554033312739386,
+            0.01635899434369978,
+            2.587383534566272e-05,
+            -0.012370727506722627,
+            0.007162422186621379,
+            -0.013370882403931942,
+            -0.011066483600541058,
+            -0.007151077450730025,
+            -0.01721910018182534,
+            0.02152239849400241,
+            -0.012322440280680164,
+            0.004587729883596212,
+            -0.009079747696669277,
+            -0.013272707286450417,
+            -0.012953682633400515,
+            -0.014202214807187545,
+            -0.01313141988034687,
+            -0.026041403779538307,
+            -0.027109731260915763,
+            -0.006573285295895077,
+            -0.023780366346334524,
+            -0.013799072648505526,
+            0.005588622594136253,
+            -0.022617902666316757,
+            0.012262325666428604,
+            -0.012689827812482627,
+            -0.01226264872059864,
+            0.012805734807835008,
+            0.020274844315616616,
+            -0.011765231803328348,
+            -0.028494489703400308,
+            -0.018583843821043854,
+            -0.019970998470704358,
+            0.012576201137505969,
+            0.002266288538944273,
+            0.013165889075444961,
+            -0.003018437854660793,
+            0.003227412667880239,
+            -0.014895831590298957,
+            0.018570624675927183,
+            -0.03442390703780963,
+            -0.03108003806963536,
+            -0.013611306765348361,
+            0.005182777066476028,
+            -0.0015782033203578553,
+            -0.011429046504256945,
+            -0.0018606648572920801,
+            -0.00433805271182878,
+            -0.025109572024446088,
+            -0.010134856632190067,
+            0.011010662054484161,
+            -0.015205629927469414,
+            0.0072933565255675036,
+            0.0009018832564038863,
+            0.013870902834260887,
+            0.009587432944616442,
+            0.013393301893492882,
+            -0.006095063698384775,
+            0.004356485794113208,
+            -0.0026476612747324386,
+            0.0004894024149759108,
+            0.009336366919474478,
+            0.00839591561027103,
+            0.010551161369079399,
+            0.002260707777260338,
+            0.007830059223001696,
+            -0.0060927446216633265,
+            0.018065002687169148,
+            0.015436944875538481,
+            -0.0002520689442496089,
+            0.013323098485251487,
+            0.0003769862591044255,
+            0.00022293304921127018,
+            -0.0029970303106340466,
+            0.007888437018607987,
+            -0.011785957933642163,
+            -0.001311704406355858,
+            0.015138190893483804,
+            -0.01707247730569978,
+            -0.013045187198628258,
+            0.03107034781283638,
+            0.0018583638488268697,
+            -0.02495564203304796,
+            0.031146554085787762,
+            0.004220047610942943,
+            -0.005835199594223787,
+            -0.013205086624848782,
+            0.0014059047822222418,
+            -0.011570624844100298,
+            0.003584232809362274,
+            0.014295032061446826,
+            -0.02231984223610487,
+            0.010521094801839404,
+            0.008446892441796151,
+            -0.009681535919980559,
+            -0.012337659795586223,
+            0.009847073709209602,
+            0.016809328445200358,
+            0.004472485504034416,
+            -0.03539093810210523,
+            -0.011957694102753411,
+            0.02145470732008716,
+            -0.029237428225009913,
+            -0.00810041956259673,
+            0.007452962400105787,
+            0.0065154832040426065,
+            -0.019353109017609053,
+            -0.008580955020412362,
+            0.005725794605897509,
+            -0.004940390374637084,
+            -0.003892210172017758,
+            0.017541623128534682,
+            0.00485047043626186,
+            0.013698341792101025,
+            -0.01689738578307474,
+            0.009771458057400557,
+            -0.022222817828602557,
+            0.004021882074676753,
+            -0.023373700500461992,
+            -0.009873403167702603,
+            0.002364143609079849,
+            0.012454271200139762,
+            0.014328556467632212,
+            0.027839363143659326,
+            0.030056404238348527,
+            0.013995696971422429,
+            0.007833677585250953,
+            -0.016265199386953717,
+            -0.018134327237616175,
+            -0.006409513840075933,
+            -0.004033173350204784,
+            -0.008714798059218596,
+            0.0066373547199279235,
+            0.01858354212009877,
+            0.004499712802472377,
+            0.02609078206670711,
+            -0.006227189596272876,
+            0.009039744622014585,
+            -3.127526911951914e-05,
+            0.0008833930131827993,
+            -0.004226083827636182,
+            -0.01971685546584527,
+            0.00481003211084742,
+            -0.0013721482162825508,
+            0.025874758880591167,
+            0.00940496860916206,
+            0.0024915013399541847,
+            0.01615978314811047,
+            -0.006378952757397484,
+            -0.01174653607178726,
+            -0.010540107733238588,
+            -0.0061926268398085176,
+            -0.021752582127981104,
+            0.029059100503124387,
+            0.021195595568253612,
+            -0.010800765264123782,
+            -0.013693582166598082,
+            -0.00903369526059487,
+            0.02795767990556847,
+            0.004080965688836953,
+            -0.0013050212744006933,
+            0.016313311850308505,
+            0.012084555958999215,
+            0.012329325712491252,
+            0.009794698207689061,
+            -0.00036157148861933123,
+            0.0015364709122308352,
+            0.03722522141440073,
+            -0.007427028510548186,
+            0.0026463292682538495,
+            -0.005239324411651515,
+            0.0025987932868461313,
+            -0.009705817520422487,
+            0.007242122655628283,
+            -0.004012676385296197,
+            -0.0158760051470016,
+            0.015003444174308607,
+            -0.022443331305169994,
+            -0.0016749787204471238,
+            -0.008150635190952799,
+            0.0016696640640947303,
+            -0.02781023337840043,
+            -0.0025731206607371845,
+            0.006695065701630532,
+            -0.015547340828489069,
+            0.011816010501093885,
+            -0.01740673229051557,
+            -0.011553604323149105,
+            0.007914961618894248,
+            -0.012034157288065447,
+            0.0016684130271324446,
+            -0.0008268957621923278,
+            0.004460864332388434,
+            0.013426055079012884,
+            -0.011008808570221245,
+            0.012166046631826273,
+            -0.005816284310138776,
+            -0.007872091214834688,
+            -0.0016055129487028885,
+            -0.0030049472973942043,
+            -0.0025600746761171543,
+            -0.0009966231170649917,
+            0.034355910234323155,
+            -0.004446570896048543,
+            0.003761242147499512,
+            -0.01362639743248326,
+            0.0019833052855181806,
+            -0.036176376394999044,
+            0.028344464817309477,
+            -0.008642835889481802,
+            0.011013502221008498,
+            -0.0017508365480865596,
+            -0.013243706268954915,
+            -0.01456517309747086,
+            0.015247299501245178,
+            0.007602596572394442,
+            -0.005854531057349665,
+            0.013558248087347848,
+            -0.019724373614448298,
+            0.008530500822910953,
+            0.004463694543554244,
+            -0.015561397976987216,
+            0.004668780190328835,
+            -0.007479912658259757,
+            0.012657201279639194,
+            -0.01774730033362932,
+            0.004314594593150765,
+            0.00958571617187387,
+            -0.01892876084898881,
+            0.009901041800918215,
+            -0.028675879564512306,
+            -0.004631920242208613,
+            -0.012323455524366223,
+            0.0053804608827285865,
+            -0.000660864620708182,
+            -0.02389088252276507,
+            -0.028944280660230762,
+            -0.0002099667627277945,
+            0.00458533068247518,
+            0.0018878351997997833,
+            0.011621407953503755,
+            0.005970807015151215,
+            0.008542958629199003,
+            0.01023896054756221,
+            -0.00205207247588841,
+            0.009402752124876478,
+            0.04169706217414899,
+            -0.022899912321317197,
+            -0.011574970753549647,
+            0.005321872977369877,
+            0.011499271505460519,
+            -0.02647653483798284,
+            -0.011039791758347888,
+            0.0033424345263339144,
+            -0.005219877979550397,
+            0.03609469691834737,
+            -0.02448746405881972,
+            0.019020618652020673,
+            -0.011392297116547224,
+            0.007625475937968714,
+            -0.021465240076078376,
+            -0.003921088264697311,
+            -0.014265745991522477,
+            -0.00011221615798127438,
+            -0.0031478236827989703,
+            -0.0076053733863387755,
+            0.007485039021296698,
+            -0.001907678661303243,
+            0.0020348246022161925,
+            0.006291436821831794,
+            -0.02173305455485444,
+            -0.0019125499969493854,
+            -0.016432460924913166,
+            0.030367263349759286,
+            -0.016242095671439496,
+            0.005072805486919414,
+            0.0053258139728101506,
+            -0.002810639419737258,
+            0.012368771814660942,
+            0.004165967191536186,
+            0.00513754440091394,
+            -0.004199767241827414,
+            -0.005142642360546972,
+            -0.0053886448467479215,
+            0.017868874549760376,
+            -0.013280048598849825,
+            0.035024701466710065,
+            -0.0100959186045929,
+            -0.007094001877261568,
+            0.0019733042325232155,
+            0.010380430710282744,
+            4.2084202709296443e-05,
+            0.01053565670076933,
+            0.0034270489770133685,
+            -0.0006311078385359693,
+            0.0019774779328212756,
+            -0.017162448473652843,
+            0.003089758682621882,
+            0.01682656561693936,
+            0.005463677441664502,
+            0.004469261499451007,
+            -0.013332130891003062,
+            -0.012050233723282738,
+            0.004538268704442613,
+            -0.007533940408255447,
+            0.0003675737140813486,
+            -0.018816953644773076,
+            0.013972696748575243,
+            -0.0011096683686631858,
+            0.0002715683052236706,
+            0.00524678369365015,
+            0.023056514997908218,
+            0.01813167860994747,
+            -0.002553352602551003,
+            0.03155164978556312,
+            0.0174738879054593,
+            0.0029268039949260673,
+            0.010637295711777793,
+            -0.033222781195984666,
+            -0.013301873512972214,
+            -0.0013721703891642354,
+            0.02238421933149517,
+            -0.01530189703335162,
+            -0.014059029663525449,
+            -0.0037393534403749947,
+            0.0010394601933309767,
+            -0.005177359933828505,
+            0.0042489306011644495,
+            0.011039889511148341,
+            -0.014697595240068863,
+            0.007597149702441915,
+            0.0026780496822907634,
+            0.02092734730183164,
+            0.0004688144432243786,
+            0.017181410417231698,
+            -0.0005083970646221274,
+            0.00749639055342427,
+            0.007319420826337891,
+            0.017541463125355265,
+            -0.01002811389296805,
+            0.023582113845081534,
+            -0.008823933397729605,
+            0.0019889258902730535,
+            -0.01257106158422629,
+            -0.028159594067544737,
+            -0.012624750220149823,
+            0.015247564545279736,
+            0.005765911899842925,
+            0.006478299226263732,
+            -0.016537090297144265,
+            0.006276997440048981,
+            0.015647122171767974,
+            -0.002251683151059454,
+            0.010295358869874306,
+            0.009730378299068289,
+            0.00012816405890877098,
+            0.010181120688239766,
+            -0.00801256937783534,
+            -0.00734257767723293,
+            -0.004497520124843868,
+            0.023982754195729983,
+            -0.005443550092842393,
+            0.022434177122972028,
+            0.012647958478747348,
+            0.00616791857573897,
+            -0.010853355357954727,
+            -0.003340111254357573,
+            0.023389634440057473,
+            0.011863246790550376,
+            0.00029642699874398923,
+            -0.009846034060495678,
+            0.0019458876579524617,
+            0.0054010373765486005,
+            0.012983913038716568,
+            -0.02049427270240938,
+            -0.004344571352918878,
+            0.008461947538684664,
+            -0.008472919299762712,
+            0.004318309005722874,
+            -0.030156444522730416,
+            -0.005812303237222271,
+            0.021354352236942434,
+            0.008392281015122372,
+            -0.010232039722494646,
+            -0.004395682831498155,
+            -0.0006527587802367882,
+            0.0040601850269714945,
+            -0.0024323167053212176,
+            -0.0023729138545912176,
+            -0.008541233856441836,
+            -0.006470933419694742,
+            0.014035696617797647,
+            -0.0052691760528871924,
+            0.008597459268477037,
+            0.00012084859681905111,
+            -0.0017255958688825231,
+            -0.007323529232751411,
+            0.03382792341957058,
+            0.008087732655845718,
+            0.005980666974986827,
+            0.0013613625176327891,
+            0.0042244330945601215,
+            0.000616791811672998,
+            -0.0028374856646453707,
+            0.018075360591135542,
+            0.0156738939725117,
+            0.005054696911814988,
+            0.009038525980341447,
+            0.014672270369917403,
+            -0.019700405219230957,
+            0.012625337070556061,
+            0.012184185186281482,
+            0.019382321278525114,
+            0.0007219498737851579,
+            0.002564970959687783,
+            -0.010601873823737926,
+            -0.0029963537053764736,
+            -0.002070567089566093,
+            0.029563028293936092,
+            0.0005903976560069006,
+            -0.010509758568838935,
+            0.010992421945232951,
+            0.020603868018231348,
+            -0.02101157589451638,
+            -0.001971974816402967,
+            0.005963911670920596,
+            -0.010704207628592966,
+            -0.014466113540150018,
+            -0.021269176952661614,
+            -0.013331247081394942,
+            0.0071597345125287174,
+            -0.008162102923896894,
+            0.02367549407145414,
+            -0.014256771777502395,
+            -0.03960796112042958,
+            -0.014803626428839152,
+            -0.00864320347698291,
+            -0.027531337207277483,
+            -0.007305441146142331,
+            0.00965056093889974,
+            0.016698589600806273,
+            -0.003330867073501157,
+            -0.025198703196464594,
+            0.030483790406960656,
+            0.008872821868559751,
+            0.023708422767352917,
+            -0.021464374508407488,
+            -0.0018617153194555344,
+            -0.00886376784514331,
+            0.014505918224188001,
+            -0.017588761818663975,
+            -0.011756112651591795,
+            0.020014632992762026,
+            0.014356419813769714,
+            0.007246557500044434,
+            -0.004355132555323843,
+            0.0067299398462013936,
+            0.007649499235136303,
+            -0.012515181210363267,
+            0.012239792251154642,
+            -0.013499277313916029,
+            -0.024858295126228433,
+            0.014430561986516884,
+            0.002229494935179828,
+            0.009563004428064055,
+            -0.0015984171085659323,
+            -0.016197659608768088,
+            0.010078159490026602,
+            0.00808729175515505,
+            -0.017508840440512225,
+            -0.004793934415922065,
+            0.007133527615305415,
+            -0.020818869260580315,
+            0.031234686259160864,
+            0.011872765152279094,
+            -0.010692949770503467,
+            -0.001989039629661815,
+            0.004431079028880923,
+            0.016511544608975148,
+            0.007893700385963683,
+            -0.01997314388229276,
+            -0.003708006454566117,
+            0.01530128306203401,
+            0.00822588305477249,
+            0.004861703074728471,
+            -0.010836328783594584,
+            0.03434874763691911,
+            0.006082873924472328,
+            4.192481378506604e-05,
+            -0.0052997095110967115,
+            -0.0076108924981838355,
+            0.004196112457289409,
+            0.018905391806961686,
+            0.013330078288599763,
+            -0.004200590783114919,
+            -0.006630107527854746,
+            0.007778448128358027,
+            -0.01706991160178345,
+            0.007329056462125937,
+            0.009390327499494826,
+            -0.012870674959022891,
+            -0.0017571189976160295,
+            0.016462608870101713,
+            0.00209846661139665,
+            0.004942714850185376,
+            -0.011305050739130339,
+            0.004252776146827917,
+            0.015841821944109317,
+            0.018727567181082646,
+            -0.008399642221255733,
+            0.021448055799259867,
+            -0.020972459569228434,
+            0.027243639465535856,
+            0.028499400850980086,
+            0.016573144403275928,
+            -0.01847096443459798,
+            -0.0022409373140116146,
+            -0.0041737366139895414,
+            -0.014761716560267328,
+            0.0034546020161337894,
+            0.005125870074458691,
+            -0.01174186608185822,
+            -0.009577921232380895,
+            -0.024331993794416026,
+            0.011075959943215117,
+            0.007918147902741738,
+            -0.0015045185909419564,
+            -0.011791155919533844,
+            0.016019077577552835,
+            0.010141868950981925,
+            -0.010938402325619172,
+            -0.005635351352447256,
+            -0.019967361901734836,
+            0.009750427567651833,
+            0.009128294017175602,
+            -0.00300556760891084,
+            -0.011496103417643132,
+            0.012977890524465716,
+            0.0014080059843484115,
+            -0.01292380658817521,
+            0.010841954636959441,
+            -0.03142011651894961,
+            -0.005510537354374977,
+            -0.011027026734432255,
+            -0.003620482919145167,
+            -0.014415606143566519,
+            0.003608572766363589,
+            0.0005034611321129689,
+            -0.011470488351158329,
+            -0.0035005774696278,
+            -0.016207453351921627,
+            0.004676327282262429,
+            0.015021513761697039,
+            0.001679614645967629,
+            -0.006917375094634678,
+            0.021994274901326064,
+            0.018518524483306403,
+            -0.021903538417919984,
+            -0.020116013379669034,
+            0.004708084983787958,
+            -0.03261690161800251,
+            0.015019348187107588,
+            0.00966124471577622,
+            0.011676937880751252,
+            -0.028768229226408037,
+            0.007114534979415504,
+            0.00900420165933673,
+            0.016398927481511824,
+            0.0050879301160439245,
+            0.009453255486450082,
+            0.01222758288108498,
+            -0.0104657614414275,
+            -0.003164502778525433,
+            -5.410382753651127e-06,
+            -0.005618225986727351,
+            0.015607620507415484,
+            0.0008527697935056332,
+            0.03603666488542239,
+            -0.005277207843139309,
+            -0.0009799879252482288,
+            -0.011735731235560851,
+            0.00812046801749312,
+            -0.011713349784345638,
+            0.006334172347602515,
+            -0.0024643499727789167,
+            0.0001240009282616789,
+            0.002965067818077541,
+            0.009442572075213767,
+            -0.01086095783353607,
+            -0.007595379345224076,
+            0.007444897256325813,
+            -0.01482814340478338,
+            -0.00019984717819268013,
+            -0.009339343730980451,
+            0.0019680804106275156,
+            -0.02241031895876346,
+            0.006629838013824249,
+            0.040935644507527846,
+            0.02140216465404127,
+            -0.005481040712269146,
+            -0.012431659209559418,
+            -0.004680230194268222,
+            -0.007229300034633229,
+            -0.006482401275991039,
+            -0.004794698160419071,
+            -0.022762405726589375,
+            -0.006928553505112732,
+            0.004320321244630314,
+            0.013300138724212552,
+            0.009741018117644266,
+            -0.008842135469729511,
+            0.004531760574809748,
+            -0.02662650543210291,
+            -0.016578081411548973,
+            -0.0060021425889846495,
+            -0.015840130031005147,
+            0.006125301421471048,
+            0.027521841635711787,
+            -0.0063432924325983525,
+            0.01789356321527744,
+            0.005142029893259461,
+            0.013034833065163448,
+            -0.005338869351330258,
+            0.00035250243782976347,
+            -0.006010938630154189,
+            0.0032958656042048292,
+            0.003074861351445954,
+            -0.0015947217852680714,
+            0.004151590621941346,
+            -0.007139285822575166,
+            -0.0026091937868967284,
+            0.0011629380311555433,
+            0.0117841884590861,
+            0.006877419488576261,
+            -0.00016252653621991807,
+            -0.01761285866765538,
+            0.022221104524808997,
+            -0.018933508559881868,
+            -0.008840161102111973,
+            0.011092684029552064,
+            -0.009660487856185668,
+            0.00702597023411946,
+            0.020387386977378873,
+            -0.006854213562767843,
+            -0.009583981919303215,
+            0.013028181087649146,
+            -0.0036507672754997286,
+            0.004464368543493108,
+            0.009867428978560914,
+            0.006036303973414791,
+            0.002482944923100156,
+            0.013244650094892797,
+            0.004530294330688855,
+            -0.0032625482538717137,
+            -0.020238533752799943,
+            -0.005186498720507101,
+            0.019015752262933585,
+            -0.02252458872825473,
+            -0.0027641215988942563,
+            0.0044506144202026456,
+            -0.0030317840657635457,
+            -0.01733962766050954,
+            0.004205861489436576,
+            -0.018770275720867473,
+            -0.0033883519277721433,
+            -0.008244668812709337,
+            0.009948090119054079,
+            -0.023700732627507225,
+            0.009109315916173436,
+            -0.011739192106106307,
+            0.02356509175177699,
+            -0.003925851077921686,
+            0.024734433836185024,
+            0.0015142645677087268,
+            -0.013482933030628261,
+            0.0055323884713736554,
+            -0.0008930661843169134,
+            -0.004477731568313757,
+            0.029338490404300612,
+            -0.005418258701133154,
+            0.002249996261871466,
+            0.013491124734151694,
+            -0.021332541202417246,
+            0.021773687217263173,
+            0.0046662696076723836,
+            0.01628697326539432,
+            -0.00986238951830334,
+            -0.013799209313714441,
+            0.025178874553620743,
+            0.03324555035648896,
+            0.008398796308070151,
+            0.011261509865513953,
+            -0.01226890254177634,
+            0.018966175575083295,
+            -0.0038328997984309597,
+            0.004344837325292706,
+            0.00878668880458693,
+            -0.007200842513742962,
+            -0.011500562922303248,
+            -0.004533014670236755,
+            0.027639894892383943,
+            -0.021838848944317424,
+            -0.008920629621611768,
+            -0.005739181836252042,
+            -0.01593272996094732,
+            -0.038167534267554606,
+            -0.004371701247332816,
+            0.019604405563120094,
+            -0.005974358141737469,
+            -0.0018341151594086288,
+            0.014334266690716505,
+            0.00402620072221496,
+            -0.002852569009185304,
+            -0.008195218707838913,
+            -0.0027944153624309856,
+            -0.0059390492900516225,
+            0.016651206007022145,
+            -0.004961864528562493,
+            -0.007929830585906699,
+            0.01145431100211762,
+            -0.005731957927601062,
+            0.012130694936714224,
+            0.0004848418056781251,
+            8.392048424584131e-05,
+            -0.0015174120840537315,
+            0.00822932723013239,
+            0.018620869354789082,
+            -0.0055803454666417465,
+            0.025682778991091174,
+            0.006956936033581638,
+            -0.003495709501259274,
+            -0.005489431577052735,
+            -0.0044154293119170445,
+            -0.024561016942073285,
+            -0.0020864636079347357,
+            0.006593060072668424,
+            0.024333813203596122,
+            -0.02515989187381735,
+            -0.004953266000463339,
+            0.020957806164031402,
+            -0.0012192724005613,
+            -0.011978722479596745,
+            0.014690559986694138,
+            0.0041346025614047875,
+            0.009881584250499358,
+            0.012176393203318002,
+            0.015764530703333238,
+            -0.004923548157246672,
+            -0.0015192775000207833,
+            0.012766482928380375,
+            0.0005745973040874939,
+            0.00997809510696241,
+            0.016495444272181515,
+            0.010467514633877887,
+            -0.021108281220544853,
+            0.0015078843789125156,
+            0.00522593579521476,
+            -0.017481430635971396,
+            0.0034362821987740755,
+            0.005397819166037461,
+            0.011626527721895694,
+            0.004048632382026344,
+            0.013207335578216437,
+            -0.015768003434275563,
+            -0.005916168437120099,
+            0.008153798523776654,
+            0.004319451296835019,
+            -0.02599474886669325,
+            -0.01485009085106089,
+            -0.0006781964116146299,
+            -0.004444144921332929,
+            0.013556265879295017,
+            0.01285768162554067,
+            0.02033626001285354,
+            0.024637349990579514,
+            -0.013833060730799217,
+            -0.00842819657792762,
+            0.009436072455719896,
+            0.015560357465711765,
+            0.005798181411384387,
+            0.007874051721372273,
+            0.009297440071118126,
+            0.0077320823375042,
+            -0.004073209576661068,
+            -0.002973928843567532,
+            0.020205003097225747,
+            -0.019725382393637726,
+            0.0006746659838642147,
+            -0.018417443693472958,
+            0.007530796896254817,
+            0.01140696723506382,
+            0.01129515042745398,
+            0.002816883643872417,
+            -0.005274757308627299,
+            0.005716917047812153,
+            -0.06254973656832136,
+            -0.009191453047502128,
+            0.003384751551032943,
+            0.027256466708746898,
+            -0.010885336140129142,
+            -0.003415707910562931,
+            0.007309577090125959,
+            0.005511970708769456,
+            0.0005572239548625122,
+            0.003186427278034852,
+            0.019698716902282903,
+            -0.02052581021967601,
+            -0.010793968505805883,
+            0.004219269922138548,
+            0.0033111837923746746,
+            -0.007269521534951044,
+            0.004805737092302134,
+            0.014143978208501908,
+            0.01986656063456352,
+            -0.0024048854610842748,
+            0.01629335842915328,
+            -0.0015498603704589015,
+            -0.014496994444493684,
+            -0.002671064325909516,
+            0.0037875833047555925,
+            -0.022466433279066646,
+            -0.005980897597195858,
+            0.0043262697442044435,
+            -0.011226121648007667,
+            0.006978495397818936,
+            0.008452177557316863,
+            -0.003018545465557218,
+            -0.0008303307644324117,
+            0.017796789945838845,
+            0.006591364879905198,
+            -0.013464486184025026,
+            0.0028668426952326957,
+            -0.023377605222213174,
+            0.004325185794796361,
+            0.001416245546006914,
+            0.007809449561523174,
+            -0.008431335139468173,
+            -0.014774499666285603,
+            -0.005933810712528226,
+            0.017746847020610673,
+            -0.01965188730040385,
+            -0.002406875192894708,
+            0.01683009662984443,
+            -0.0002518220110313279,
+            0.005456812494894182,
+            -0.02011241622618092,
+            -0.0009782974741561381,
+            -0.02933296756078644,
+            -0.01171823509606186,
+            0.001162345317872531,
+            0.00892221099991475,
+            -0.006551308022163148,
+            -0.03934356275100932,
+            0.00427623835776202,
+            -0.013111664387776261,
+            -0.008752857644809231,
+            -0.010214546496134218,
+            -0.00518087722852466,
+            0.015240297918667867,
+            -0.012498100033235827,
+            0.01086490299272505,
+            -0.00917777212602538,
+            0.005895055740265625,
+            0.0090554651297836,
+            0.014212841798768113,
+            -0.005544047915369086,
+            -0.0033589999440643385,
+            -0.007924830667307956,
+            0.0022291797354234005,
+            -0.0032976226056924502,
+            0.009615674261248298,
+            -0.0051932155872883,
+            -0.0065636389655770455,
+            0.02426106498596339,
+            0.011445591067574932,
+            0.013312596099532533,
+            0.0055118624272110965,
+            -0.007220620072349783,
+            -0.01916551187613823,
+            -0.003977616240384473,
+            -0.013204506708246246,
+            -0.006315966975649096,
+            0.01434099854765068,
+            0.012588249778961223,
+            0.006357972880218866,
+            0.0007747002406996095,
+            -0.004794439624135275,
+            0.006975593520375694,
+            0.0262335496080837,
+            -0.016481909971520166,
+            -0.0013084662210086592,
+            -0.01369313900978409,
+            0.01921748474212319,
+            0.03333761672336437,
+            -0.0008918419419301244,
+            -0.007699525186523558,
+            0.013757216065348164,
+            0.00819045303047094,
+            -0.007411380297164911,
+            0.02453611080891859,
+            -0.025045462880463647,
+            -0.015989319900023054,
+            -0.011650886878708265,
+            -0.011407253985176972,
+            -0.008386367959227559,
+            0.016331961716602287,
+            -0.00141225438918211,
+            -0.0018505754298268654,
+            0.009557291617446645,
+            0.03357924917013024,
+            0.006916860901290756,
+            0.007030308953738567,
+            -0.002739345982346728,
+            -0.009376522438071562,
+            0.011482637968411055,
+            -0.007259054391879614,
+            -0.04075661435928125,
+            0.004250195164583401,
+            -0.005160505473755424,
+            0.02405623169060588,
+            -0.02197586561375289,
+            -0.00619175761282745,
+            -0.006736129049831973,
+            -0.028200015908139763,
+            -0.005867722104196303,
+            0.02805071895018145,
+            -0.015402198567775478,
+            0.014773294766214671,
+            -0.009502185346558072,
+            -0.0013419858580438616,
+            0.004443749010136947,
+            0.013008783570355656,
+            -0.02289662952366337,
+            0.0013834888740540108,
+            -0.00048549197205067054,
+            0.027969842651517314,
+            0.010437729090975158,
+            -0.0047705656249623744,
+            0.014202218020412936,
+            0.008240861154198411,
+            -0.018637315941614406,
+            0.007314019317343154,
+            -0.005747966765101792,
+            0.014833896739762147,
+            -0.01314847940891154,
+            0.007762100494499689,
+            0.00705026393499352,
+            -0.01278023669088385,
+            0.03016342404000438,
+            -0.015204129985039077,
+            0.007776946006562982,
+            0.009278778571285689,
+            0.009656723862179986,
+            0.01257686993982689,
+            0.004362439504889069,
+            -0.02898418193036562,
+            0.0002144431781454467,
+            -0.03828100004144633,
+            -0.00833748467467171,
+            -0.002611937107022567,
+            -0.002145329666219461,
+            0.007050630951707937,
+            -0.009383645162261629,
+            -0.033976442057557676,
+            0.0006429978367640709,
+            0.01603027598131823,
+            0.017501067413108286,
+            0.0002196997337451293,
+            0.013075379412332973,
+            -0.006197537539554383,
+            -0.021148120787551232,
+            -0.016986448660412497,
+            -0.010207705210051413,
+            -0.02122194744719993,
+            -0.020550470201191853,
+            -0.00908564461857315,
+            0.012090079127547193,
+            -0.009927390005879807,
+            0.013888458967497627,
+            0.029513250508042935,
+            -0.024529175134769883,
+            0.0011144447737101862,
+            0.021977677894839922,
+            -0.00198308986575084,
+            0.014108189311291326,
+            0.001696775006906642,
+            0.014122064251288995,
+            -0.0004166140484018266,
+            -0.010556016685519523,
+            -0.009729362492661684,
+            0.017487929296614076,
+            -0.0026043257634797124,
+            0.016715798948897194,
+            0.012183487338360088,
+            0.003771721868999568,
+            -0.0030887189945795796,
+            0.010667776697461199,
+            -0.026471157050402344,
+            0.014249996582064706,
+            -0.006690961709923224,
+            -0.0181382691440242,
+            -0.008729104191096461,
+            0.003882731567105777,
+            -0.005609384377642956,
+            -0.007794812949125115,
+            0.012967089730551055,
+            -0.00944374111323783,
+            0.02088525981049842,
+            0.0023400121055514958,
+            -0.012196925240774426,
+            0.003648688547842322,
+            -0.03322041209304968,
+            0.0015191765655963997,
+            -0.018402382776013956,
+            -0.024659631472836927,
+            0.03840173429666435,
+            0.011991452465253198,
+            0.006639702644606365,
+            -0.01675838115087676,
+            0.007953102302963526,
+            0.0004257479565099406,
+            0.017484338819046517,
+            0.011068864013925452,
+            0.006466915839538296,
+            0.008364454017976605,
+            -0.023316137291380285,
+            -0.010898421318668592,
+            0.016204569093730564,
+            0.012099473359791148,
+            -0.01017229369980751,
+            0.019431669546223253,
+            0.017955119269367787,
+            0.007300372071724306,
+            -0.006898619924677706,
+            0.01647927363640854,
+            0.021012103522831126,
+            0.010291016618557653,
+            -0.00047955552103474145,
+            0.0008684297441778696,
+            -0.010534247928997246,
+            0.005037930235889528,
+            0.035417626740307205,
+            0.014712595692093886,
+            0.010650306363963653,
+            -0.03626584475133921,
+            0.021764456453493054,
+            -0.011961521673661574,
+            0.009092981192874177,
+            0.006089104887183293,
+            0.017982801101846038,
+            0.006916969397519662,
+            0.014196254958707752,
+            0.011745069030657836,
+            -0.006689305450954901,
+            0.025205280281804708,
+            -0.001601414806100461,
+            -0.011992525142592405,
+            0.003386930324498371,
+            -0.01859193251438437,
+            0.005303261319773131,
+            -0.0041710536965100615,
+            -0.004459393120257307,
+            0.0032461828144552565,
+            -0.0018650956576455287,
+            0.015678225251799887,
+            0.003894142628084471,
+            -0.02355212701026675,
+            -0.0014873755115206368,
+            -0.024284295126348117,
+            0.0027260592623277375,
+            -0.022185571998195344,
+            0.00022883605560765203,
+            -0.005557568353382136,
+            -0.009749382813201403,
+            -0.025385857431333838,
+            -0.009853073898348951,
+            -0.0035843255959309307,
+            0.007661137888845632,
+            -0.010521037433807653,
+            -0.004246698084422667,
+            -0.0019920499321964815,
+            -0.019465113363548715,
+            0.009014255846983305,
+            0.027122533756959888,
+            0.016848221859975445,
+            0.027170883580103264,
+            -0.00019554112591107156,
+            0.013614675849485926,
+            -0.002066822813103284,
+            -0.01603063005944364,
+            0.008824663429777616,
+            0.0075653522381218495,
+            -0.00531780740880296,
+            0.014643348459785112,
+            0.0072634484937344704,
+            -0.012800149015319017,
+            0.009859503660384916,
+            -0.007161712176469609,
+            -0.006437857349742842,
+            -0.009764856421740622,
+            -0.02058868349990357,
+            -0.014866137971603676,
+            0.0012450207572602557,
+            -0.002123265406747006,
+            0.006000473862248991,
+            -0.004054016790850226,
+            0.009584879012937315,
+            -0.00608816807208866,
+            -0.00925801057439376,
+            -0.013962683072792106,
+            -0.008924356118158748,
+            -0.004210064145949253,
+            -0.004381709838058668,
+            -0.011732699976353612,
+            -0.005479523097469261,
+            0.0010680258077037758,
+            -0.0019401640391035755,
+            -0.01397652201595301,
+            0.011650839243603377,
+            0.0058731838291845645,
+            -0.002462743134993977,
+            0.02219008688925881,
+            0.01097316136828334,
+            0.0018083001685932406,
+            0.01366095798812196,
+            0.0003942601242216818,
+            -0.0064063532958756175,
+            -0.03343312148366622,
+            -0.0005083355836126291,
+            -0.02815586367527057,
+            0.025770078162165123,
+            -0.0069114949102666145,
+            -0.010322087105114614,
+            -0.002519047562374146,
+            -0.006063669644021698,
+            0.005873609611264377,
+            -0.002722940652283699,
+            0.0021229525800901114,
+            -0.023673778230768945,
+            0.0028629715379660113,
+            -0.003953653437464966,
+            -0.0036366957271968823,
+            0.007469664864329332,
+            0.0346520690389124,
+            -0.004054282838167587,
+            -0.0034765253405897676,
+            0.005159466707114532,
+            0.011642913651743502,
+            0.027845571220425726,
+            -0.004445196354520585,
+            -0.0032038508043339413,
+            -0.02233838584152591,
+            0.0054112576531374175,
+            -0.00019544838850471866,
+            -0.004733192299951088,
+            0.016664217348411003,
+            -0.016719175418727594,
+            -0.01631658832047864,
+            -0.00545969677717507,
+            -0.014570848053102338,
+            -0.00022620258237138677,
+            0.014454101124666536,
+            0.011056312437637014,
+            -0.004035526363810078,
+            -0.012459378363360557,
+            -0.0022595493558020706,
+            -0.030370952502579162,
+            0.012766010284884204,
+            0.1141689619126009,
+            -0.00766197047210299,
+            -0.00896655658602649,
+            -0.009481112393590025,
+            0.001435269335646347,
+            0.001338755472877713,
+            -0.011739143028767118,
+            0.07166096084053437,
+            -0.0077855767070652395,
+            0.0002139580430342728,
+            -0.011312198831181402,
+            0.001844005266872301,
+            -0.001825412693163496,
+            0.01371932421024605,
+            -0.005025814503687022,
+            -0.018213360991353643,
+            -0.0033612481616769416,
+            -0.02941403830641238,
+            -0.0041137865872933515,
+            -0.005683630256743215,
+            -0.004288198570156578,
+            -0.014982999363445122,
+            0.0067149665791261335,
+            -0.019661973497372313,
+            -0.012688556129040908,
+            0.001124391650175233,
+            -0.0033623453063430534,
+            0.00996944383866679,
+            -0.020496336814603385,
+            -0.0044354321069609536,
+            0.0019875840782527733,
+            0.009706675240563883,
+            0.0011600364428219214,
+            -0.012755178230821602,
+            -0.0026514442584739412,
+            0.004056350166031111,
+            0.01909951327913368,
+            0.014360281326789544,
+            0.003732344102045803,
+            0.008623457695718206,
+            0.014461868085333023,
+            -0.001692149528228451,
+            0.01948035376572433,
+            0.01172550239477637,
+            0.0031966531144076173,
+            0.0004779413449255626,
+            0.0030912562273611517,
+            -0.004597633235071031,
+            0.01582547400214653,
+            -0.014811177637844808,
+            -0.02411202772171917,
+            -0.0071272238837422185,
+            0.018015541137524113,
+            -0.013081287852028074,
+            1.8837689271256874e-05,
+            0.004062803284646432,
+            0.027569726868200254,
+            0.0010564850664255057,
+            -0.01179755720247972,
+            0.0028876134800690334,
+            -0.01638215925131053,
+            0.022944183106912663,
+            0.0026043713064243836,
+            0.03279173899194346,
+            -0.005103903273968361,
+            0.00578213669463383,
+            -0.020744194169889544,
+            -0.007588793365216579,
+            -0.00819862611489168,
+            0.0017401442433921013,
+            0.010993500174794084,
+            0.005411238350013134,
+            0.01620925462604661,
+            -0.00641615069699833,
+            -0.020751730783437995,
+            0.014776878596707538,
+            -0.009552193304486048,
+            -0.03393319554458277,
+            0.008881987295131347,
+            -0.029102297123590998,
+            -0.0045267103602711,
+            0.00368971742585232,
+            -0.014269341759285646,
+            -0.016264809810970175,
+            0.012837302095591106,
+            -0.0011890334180728564,
+            -0.008937992768323412,
+            0.0014444613725248846,
+            -0.017110187133760626,
+            -0.02385668037818897,
+            0.006017405364919834,
+            0.00676670672597817,
+            -0.0077994500447595,
+            0.018312452858615755,
+            -0.027015660037024585,
+            0.006123912473734307,
+            -0.012973676426738582,
+            0.009041974762537394,
+            -0.01580183887345168,
+            0.01163516332546347,
+            0.010887662470499361,
+            -0.001503467664709027,
+            -0.0076234124970223326,
+            0.01231202541338883,
+            0.004852720134894811,
+            0.033760365210093034,
+            0.028658326381800395,
+            0.000995999783627898,
+            -0.0025605367058226737,
+            -0.0020356794893366384,
+            0.011667525096213252,
+            -0.014049700287539817,
+            0.011218246246280464,
+            0.02658113044439717,
+            0.017568631325838108,
+            -0.019133062654008885,
+            0.018050678772708717,
+            -0.010356118256625172,
+            0.012872052420999813,
+            -0.025620154415048252,
+            -0.005135768485044442,
+            0.004936490920616527,
+            0.005163008729256001,
+            0.007425278892536464,
+            0.001410212306439329,
+            0.021891387289347516,
+            0.014999616600766989,
+            0.00027613811156650413,
+            0.00558422822386443,
+            0.01477009526981092,
+            0.010497782599293994,
+            -0.0005567894749233204,
+            -0.00467776506604011,
+            -0.016998035358377343,
+            0.00957271797745408,
+            -0.006065116857898469,
+            0.005138070401846389,
+            0.01698914855850691,
+            -0.01230257809343985,
+            0.011109844802663414,
+            0.007518553660463051,
+            -0.015504773927595823,
+            -0.017169933945839305,
+            0.005552829003043926,
+            -0.02252633790984703,
+            -0.01306722099657118,
+            0.02715689072730398,
+            -0.010513941399990838,
+            -0.02790582732974509,
+            -0.0009847768069205477,
+            0.009797865241108237,
+            -0.002772952073769185,
+            -0.0027968741526000576,
+            0.0037541733651479613,
+            0.010801137722284016,
+            -0.014658156970099418,
+            0.002075415233345957,
+            0.004735264309137368,
+            0.01930804315461151,
+            -0.0014151522239496115,
+            -0.015046118414100617,
+            -0.002288495894227231,
+            0.021592829677937263,
+            0.017174609713578173,
+            0.00265560378681856,
+            -0.016499825201963983,
+            -3.965296928410772e-05,
+            -0.0074142747014567635,
+            0.00021360696629503726,
+            0.00027431174904249525,
+            -0.0056391089282546054,
+            -0.018931681700770225,
+            -0.003888248121485032,
+            0.011181675419027736,
+            -0.00247252920175839,
+            0.01977860312223846,
+            -0.0011527575128223392,
+            -0.015763587433200915,
+            0.001960734212846797,
+            -0.005280860341480037,
+            -0.010372829570478649,
+            0.01351900743232294,
+            -0.009359942374603997,
+            0.006788746813046239,
+            -0.003212015225284119,
+            0.004611143085183367,
+            -0.0003308099698088422,
+            -0.017017265064185255,
+            0.02924142709831441,
+            -0.0018238549856433466,
+            -0.004769274041546295,
+            -0.0014058363976586562,
+            -0.033393331166741945,
+            0.015017165991303838,
+            -0.014574579250749552,
+            0.006745699409061803,
+            0.002705853802154516,
+            0.01272741296341735,
+            0.003253095246601883,
+            -0.008898616461779179,
+            0.0027485606928249326,
+            0.017040360927723885,
+            0.013756402798365262,
+            -0.007292864293090161,
+            0.008694855886257706,
+            -0.0028934389922887676,
+            -0.0037483208383542015,
+            0.01363963339139356,
+            0.004170611039133969,
+            -0.006282657671540625,
+            -0.0009784372731140945
+          ],
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "Mixture-of-Experts (MoE)",
+            "fine-grained expert segmentation",
+            "shared expert isolation"
+          ],
+          "title": "Deepseekmoe: Towards Ultimate Expert Specialization In Mixture-Of-Experts Language Models"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "33412f87-7418-4ce2-9116-377a27fd2732",
+        "properties": {
+          "page_content": "4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n",
+          "title": "4 Experiments",
+          "embedding": [
+            0.009938324801623821,
+            -0.009144282899796963,
+            -0.026211904361844063,
+            -0.022352691739797592,
+            0.016043050214648247,
+            0.016247965395450592,
+            -0.02783413790166378,
+            0.0537557490170002,
+            -0.02827811799943447,
+            -0.01755429059267044,
+            0.010237157344818115,
+            0.001297788112424314,
+            0.014762339182198048,
+            -0.01001516729593277,
+            0.007487895432859659,
+            0.015283161774277687,
+            0.019347287714481354,
+            0.019910801202058792,
+            0.02535809576511383,
+            -0.04251963645219803,
+            0.0020213902462273836,
+            -0.032034873962402344,
+            -0.058332156389951706,
+            0.0034600566141307354,
+            0.001006959704682231,
+            0.01755429059267044,
+            0.011645940132439137,
+            -0.0015795447397977114,
+            -0.053584985435009,
+            0.04794985428452492,
+            0.016555335372686386,
+            0.0044910297729074955,
+            -0.007833687588572502,
+            -0.004798400681465864,
+            -0.04204150289297104,
+            -0.028961164876818657,
+            0.05720513314008713,
+            0.005165537819266319,
+            -0.005519868340343237,
+            0.05481446906924248,
+            0.020252324640750885,
+            0.010245694778859615,
+            -0.03961668908596039,
+            -0.009904172271490097,
+            0.010134699754416943,
+            0.001490962109528482,
+            -0.0010427129454910755,
+            -0.007479357533156872,
+            0.02295035682618618,
+            -0.04146091267466545,
+            -0.005549751687794924,
+            -0.03811398521065712,
+            0.01863008923828602,
+            0.013780459761619568,
+            -0.04426140338182449,
+            0.023223575204610825,
+            -0.0052978782914578915,
+            0.03527934476733208,
+            0.057854026556015015,
+            0.02783413790166378,
+            0.024162763729691505,
+            -0.02590453252196312,
+            -0.04340759664773941,
+            -0.0036180110182613134,
+            0.010143238119781017,
+            0.008119713515043259,
+            -0.01344747468829155,
+            -0.010527451522648335,
+            0.046173933893442154,
+            -0.023701708763837814,
+            -0.05327761545777321,
+            0.027680452913045883,
+            -0.013584083877503872,
+            -0.008081291802227497,
+            0.0036372218746691942,
+            -0.03517688810825348,
+            0.012807118706405163,
+            0.03548425808548927,
+            0.01004078146070242,
+            0.015018481761217117,
+            0.001224147155880928,
+            0.008418546058237553,
+            -0.0014920294051989913,
+            0.017912890762090683,
+            0.028517184779047966,
+            -0.01417321152985096,
+            -0.006309640593826771,
+            -0.0686461552977562,
+            0.025767924264073372,
+            -0.007368362508714199,
+            -0.020115714520215988,
+            0.00026281276950612664,
+            -0.007099412847310305,
+            0.0033597343135625124,
+            -0.020781684666872025,
+            0.07472527027130127,
+            -0.013105952180922031,
+            0.014386664144694805,
+            -0.02096952311694622,
+            0.007300057914108038,
+            -0.031932417303323746,
+            0.02236976847052574,
+            0.01146664097905159,
+            0.005916888825595379,
+            0.038318898528814316,
+            -0.02834642305970192,
+            -0.005869929678738117,
+            0.0122009152546525,
+            0.0292856115847826,
+            0.08387809246778488,
+            0.01419028826057911,
+            -0.014352511614561081,
+            -0.025563009083271027,
+            -0.016538260504603386,
+            -0.046105626970529556,
+            -0.0292856115847826,
+            -0.025170257315039635,
+            -0.027099864557385445,
+            0.036338064819574356,
+            -0.0010613900376483798,
+            0.012388752773404121,
+            0.05338007211685181,
+            0.015863751992583275,
+            -0.00874299369752407,
+            -0.039241012185811996,
+            0.0438857264816761,
+            -0.006198645569384098,
+            0.010809208266437054,
+            0.002330895746126771,
+            -0.0146428057923913,
+            -0.006185838486999273,
+            0.011833777651190758,
+            -0.013763383962213993,
+            -0.029456373304128647,
+            0.031949494034051895,
+            -0.01727253384888172,
+            0.03944592550396919,
+            0.06239628419280052,
+            -0.03562086820602417,
+            -0.02001325786113739,
+            0.03189826384186745,
+            -0.035825781524181366,
+            0.018288565799593925,
+            -0.0005250918911769986,
+            -0.030993228778243065,
+            0.026246055960655212,
+            0.01562468521296978,
+            0.06092773377895355,
+            -0.024196917191147804,
+            0.028705023229122162,
+            -0.019176525995135307,
+            0.01628211699426174,
+            -0.020627999678254128,
+            0.03596239164471626,
+            0.0023351646959781647,
+            -0.024265222251415253,
+            -0.015752756968140602,
+            0.014694035053253174,
+            -0.014275669120252132,
+            -0.022830823436379433,
+            0.019740039482712746,
+            0.02100367471575737,
+            -0.010450609028339386,
+            -0.0049051265232264996,
+            0.012013077735900879,
+            0.04989653453230858,
+            0.016205275431275368,
+            -0.014130521565675735,
+            -0.07479357719421387,
+            0.04801815748214722,
+            0.021703798323869705,
+            0.01958635449409485,
+            0.023104043677449226,
+            0.035825781524181366,
+            -0.001056053675711155,
+            0.030566323548555374,
+            0.0019007899099960923,
+            -0.013601160608232021,
+            -0.02044016122817993,
+            0.043646663427352905,
+            -0.0039019023533910513,
+            -0.0019274713704362512,
+            0.0076373121701180935,
+            0.018288565799593925,
+            0.03123229369521141,
+            0.005430218297988176,
+            0.010399380698800087,
+            0.039719145745038986,
+            -0.011586173437535763,
+            0.01414759736508131,
+            0.009460192173719406,
+            0.02091829478740692,
+            0.04299776628613472,
+            0.02245514839887619,
+            -0.008670419454574585,
+            -0.02194286324083805,
+            0.00032551426556892693,
+            0.0219599399715662,
+            0.01811780408024788,
+            0.02199409157037735,
+            -0.012013077735900879,
+            -0.0220111683011055,
+            0.009383349679410458,
+            -0.02199409157037735,
+            0.01682855375111103,
+            -0.00167666538618505,
+            0.01859593763947487,
+            0.013865840621292591,
+            -0.006258412264287472,
+            -0.01677732542157173,
+            0.0002502724528312683,
+            -0.018186109140515327,
+            -0.008380125276744366,
+            -0.010945817455649376,
+            -0.016495568677783012,
+            0.03364003449678421,
+            -0.020747533068060875,
+            0.006599935237318277,
+            -0.01810072734951973,
+            -0.011885006912052631,
+            -0.005835777148604393,
+            0.03596239164471626,
+            0.012209453620016575,
+            -0.01851055584847927,
+            -0.026058217510581017,
+            -0.011108040809631348,
+            -0.018817927688360214,
+            -0.006463326048105955,
+            0.0342206209897995,
+            -0.01750306226313114,
+            0.006659701932221651,
+            -0.004211407620459795,
+            -0.002409872831776738,
+            0.004258367232978344,
+            0.033435117453336716,
+            -0.03082246705889702,
+            -0.01676025055348873,
+            0.023582175374031067,
+            0.008059946820139885,
+            0.029046546667814255,
+            -0.016888320446014404,
+            -0.006275488529354334,
+            -0.024231068789958954,
+            -0.016973702237010002,
+            -0.01643580198287964,
+            -0.005118578672409058,
+            -0.02199409157037735,
+            -0.016025975346565247,
+            0.02790244296193123,
+            0.013686541467905045,
+            -0.00790199264883995,
+            -0.019313136115670204,
+            -0.002333030104637146,
+            0.02542640082538128,
+            -0.013592622242867947,
+            0.04064125940203667,
+            0.050545431673526764,
+            -0.05713682621717453,
+            -0.0122521435841918,
+            -0.03661128506064415,
+            0.07684271782636642,
+            -0.015377080999314785,
+            -0.03032725863158703,
+            0.033964481204748154,
+            -0.029388070106506348,
+            -0.008990597911179066,
+            0.026997406035661697,
+            0.025272715836763382,
+            -0.014719649218022823,
+            0.01123611256480217,
+            0.03756754845380783,
+            -0.030139420181512833,
+            -0.0439881831407547,
+            -0.03777246177196503,
+            0.019859572872519493,
+            -0.025289790704846382,
+            -0.01419028826057911,
+            -0.0110311983153224,
+            0.023530947044491768,
+            0.01801534742116928,
+            0.010664060711860657,
+            0.007547662127763033,
+            0.006843270733952522,
+            0.021157361567020416,
+            0.02144765481352806,
+            0.007073798682540655,
+            -0.03237639740109444,
+            0.0003433908859733492,
+            -0.03944592550396919,
+            -0.009093054570257664,
+            -0.025289790704846382,
+            -0.02095244638621807,
+            -0.023821242153644562,
+            0.00949434470385313,
+            0.03275207430124283,
+            -0.0008143192972056568,
+            0.0067365444265306,
+            0.015044095925986767,
+            0.015488076023757458,
+            -0.03807983547449112,
+            -0.03555256128311157,
+            0.006348061840981245,
+            0.01682855375111103,
+            -0.0055625587701797485,
+            -0.059220120310783386,
+            -0.00042770442087203264,
+            -0.003165493020787835,
+            -0.005016121547669172,
+            -0.028909936547279358,
+            0.010476223193109035,
+            0.013345018029212952,
+            0.010237157344818115,
+            0.0110311983153224,
+            -0.027526767924427986,
+            -0.014651344157755375,
+            -0.0036073385272175074,
+            0.006019345950335264,
+            0.02100367471575737,
+            -0.040709562599658966,
+            -0.022830823436379433,
+            0.03818229213356972,
+            0.0003810117777902633,
+            -0.06598227471113205,
+            -0.005387527868151665,
+            -0.010757979936897755,
+            0.015436847694218159,
+            -0.0005656477296724916,
+            0.034476764500141144,
+            -0.03121521882712841,
+            -0.019296059384942055,
+            -0.020815838128328323,
+            -0.051979828625917435,
+            -0.03531349450349808,
+            -0.015436847694218159,
+            0.007389707490801811,
+            -0.038353051990270615,
+            -0.024265222251415253,
+            0.004111085087060928,
+            -0.0268949493765831,
+            0.007398245856165886,
+            -0.013567008078098297,
+            0.015633223578333855,
+            -0.03852381557226181,
+            -0.010092009790241718,
+            0.004262635949999094,
+            -0.01861301250755787,
+            -0.005507061257958412,
+            0.014369587413966656,
+            0.013925607316195965,
+            0.014719649218022823,
+            0.0017268265364691615,
+            0.015266085974872112,
+            0.0062071834690868855,
+            -0.00769280968233943,
+            0.007833687588572502,
+            -0.04091447591781616,
+            0.0014461372047662735,
+            0.018852079287171364,
+            0.011432488448917866,
+            -0.011082426644861698,
+            0.004151640925556421,
+            -0.010373766534030437,
+            -0.03034433349967003,
+            0.029917430132627487,
+            -0.05419972911477089,
+            0.0024141420144587755,
+            0.005554020404815674,
+            -0.016410188749432564,
+            -0.008384393528103828,
+            -0.009690719656646252,
+            -0.03708941861987114,
+            -0.01248267199844122,
+            0.0039019023533910513,
+            0.00019050588889513165,
+            0.009349197149276733,
+            0.002870929194614291,
+            0.05471201241016388,
+            -0.003165493020787835,
+            -0.04484199360013008,
+            0.023308956995606422,
+            -0.03671374171972275,
+            0.018749622628092766,
+            0.04245132952928543,
+            0.020713381469249725,
+            -0.017212767153978348,
+            0.04791570082306862,
+            0.0195692777633667,
+            0.000224524803343229,
+            -0.016999315470457077,
+            0.025153182446956635,
+            -0.007099412847310305,
+            -0.061303410679101944,
+            -0.004448339343070984,
+            -0.018749622628092766,
+            0.018664240837097168,
+            0.016674868762493134,
+            -0.002407738473266363,
+            -0.010032243095338345,
+            0.009426039643585682,
+            -0.01510386262089014,
+            -0.017443295568227768,
+            0.01705908216536045,
+            -0.011458102613687515,
+            -0.004525181837379932,
+            0.007526317145675421,
+            0.013148642145097256,
+            0.01271320041269064,
+            0.008853988721966743,
+            0.023992003872990608,
+            -0.045627497136592865,
+            0.009127207100391388,
+            -0.01997910626232624,
+            -0.036918655037879944,
+            -0.002990462351590395,
+            0.01318279467523098,
+            0.04183658957481384,
+            -0.02779998630285263,
+            0.0019253368955105543,
+            0.005007583647966385,
+            -0.013515779748558998,
+            -0.021106131374835968,
+            0.00341309723444283,
+            0.04600317031145096,
+            -0.0015763428527861834,
+            0.021618416532874107,
+            -0.011304416693747044,
+            -0.03562086820602417,
+            0.008811297826468945,
+            0.04142676293849945,
+            -0.005028928630053997,
+            0.01663217879831791,
+            0.009434578008949757,
+            0.03548425808548927,
+            -0.04590071365237236,
+            -0.007833687588572502,
+            -0.02344556525349617,
+            -0.01813488081097603,
+            0.04105108603835106,
+            -0.03075416199862957,
+            -0.01365238893777132,
+            0.03524519130587578,
+            -0.005306416191160679,
+            0.02441890724003315,
+            -0.018271489068865776,
+            -0.02588745579123497,
+            0.0014600115828216076,
+            -0.03712356835603714,
+            -0.00708660576492548,
+            -0.01806657575070858,
+            -0.006160224322229624,
+            -0.06557244807481766,
+            0.00010459146869834512,
+            -0.07431544363498688,
+            -0.002902946900576353,
+            0.025136105716228485,
+            -0.006552976090461016,
+            0.04429555684328079,
+            -0.026228979229927063,
+            0.02098659984767437,
+            -0.05553166940808296,
+            -0.01367800310254097,
+            0.00978463888168335,
+            0.03375956416130066,
+            0.007611697539687157,
+            -0.001995776081457734,
+            0.0015507286880165339,
+            -0.026997406035661697,
+            -0.0018303507240489125,
+            -0.007282981649041176,
+            -0.001387437921948731,
+            4.89605481561739e-05,
+            -0.018852079287171364,
+            0.00950288213789463,
+            0.000838332693092525,
+            0.01610281690955162,
+            0.031454283744096756,
+            0.022711291909217834,
+            -0.03070293366909027,
+            -0.008862526156008244,
+            0.005195421166718006,
+            0.02587038092315197,
+            -0.02978082001209259,
+            0.003897633170709014,
+            -0.03464752435684204,
+            0.014864795841276646,
+            5.636466448777355e-05,
+            0.016657792031764984,
+            0.008008718490600586,
+            0.017016392201185226,
+            -0.040197279304265976,
+            0.0030160765163600445,
+            -0.0016254368238151073,
+            -0.027202321216464043,
+            0.020867066457867622,
+            0.002305281348526478,
+            -0.022147778421640396,
+            0.0054472945630550385,
+            -0.004207138437777758,
+            0.024213993921875954,
+            0.04473953694105148,
+            -0.0010592554463073611,
+            -0.02930268831551075,
+            -0.012704662047326565,
+            0.042212262749671936,
+            0.02633143588900566,
+            -0.028483033180236816,
+            0.017118848860263824,
+            0.02103782817721367,
+            -0.02341141365468502,
+            -0.021089056506752968,
+            0.013925607316195965,
+            -0.016179660335183144,
+            -0.0007694943924434483,
+            -0.005468640010803938,
+            0.010399380698800087,
+            0.01240582950413227,
+            -0.029883278533816338,
+            0.05676115304231644,
+            -0.016820017248392105,
+            -0.006834732368588448,
+            -0.03034433349967003,
+            -0.006168762221932411,
+            -0.014531810767948627,
+            0.05553166940808296,
+            0.0028175662737339735,
+            0.02728770114481449,
+            0.01658095046877861,
+            0.002264725510030985,
+            0.05628301948308945,
+            -0.03678204491734505,
+            -0.03451091796159744,
+            0.018681317567825317,
+            -0.0055625587701797485,
+            -0.00671093026176095,
+            -0.022728366777300835,
+            -0.014207364059984684,
+            -0.013353556394577026,
+            0.016965163871645927,
+            -0.0268778745085001,
+            -0.012260681949555874,
+            0.019740039482712746,
+            0.017690900713205338,
+            -2.0469710761972237e-06,
+            0.02293328195810318,
+            0.012644895352423191,
+            0.014284206554293633,
+            0.017878737300634384,
+            -0.020867066457867622,
+            0.019142374396324158,
+            0.024111535400152206,
+            0.009946862235665321,
+            -0.01763967238366604,
+            -0.017878737300634384,
+            0.007347017526626587,
+            -0.0038613462820649147,
+            -0.007125027012079954,
+            0.01437812577933073,
+            0.007043915335088968,
+            -0.004047049675136805,
+            -0.027151092886924744,
+            -0.018356870859861374,
+            0.02093537151813507,
+            -0.0023757205344736576,
+            0.01539415679872036,
+            -0.00928943045437336,
+            -0.010612832382321358,
+            0.03763585537672043,
+            0.01861301250755787,
+            -0.046139780431985855,
+            0.026280207559466362,
+            0.01051037572324276,
+            -0.006228528916835785,
+            -0.013729231432080269,
+            0.005626594182103872,
+            0.009101592935621738,
+            0.010220080614089966,
+            0.016572412103414536,
+            -0.005097233224660158,
+            -0.03328143432736397,
+            0.01564176194369793,
+            0.026211904361844063,
+            -0.020798761397600174,
+            0.012021616101264954,
+            0.00017663151083979756,
+            0.022762520238757133,
+            -0.005028928630053997,
+            -0.012311910279095173,
+            -0.01562468521296978,
+            0.009741948917508125,
+            -0.012235067784786224,
+            0.01123611256480217,
+            0.037396788597106934,
+            0.015914980322122574,
+            -0.03565501794219017,
+            0.015735680237412453,
+            -0.01686270721256733,
+            -0.015684451907873154,
+            0.03261546418070793,
+            0.008264861069619656,
+            -0.02636558935046196,
+            -0.007142103277146816,
+            -0.006527361460030079,
+            0.02001325786113739,
+            -0.02102075144648552,
+            -0.008508196100592613,
+            0.004418455995619297,
+            0.0035646480973809958,
+            0.009938324801623821,
+            0.006745082791894674,
+            0.005016121547669172,
+            0.019159449264407158,
+            0.013806073926389217,
+            -0.0005186883499845862,
+            0.012730276212096214,
+            0.011577636003494263,
+            0.002902946900576353,
+            0.0075391242280602455,
+            -0.010997045785188675,
+            -0.02295035682618618,
+            0.004734364803880453,
+            0.05945918336510658,
+            -0.005852853413671255,
+            0.03876287862658501,
+            0.010407919064164162,
+            0.028961164876818657,
+            0.021686721593141556,
+            -0.024299373850226402,
+            -0.01681147888302803,
+            -0.018151957541704178,
+            -0.021737949922680855,
+            -0.04453462362289429,
+            -0.01510386262089014,
+            -0.01629919372498989,
+            -0.013404784724116325,
+            0.005694898776710033,
+            -0.00013474156730808318,
+            -0.0013116624904796481,
+            0.011355645954608917,
+            -0.002003246918320656,
+            -0.012491210363805294,
+            -0.003355465130880475,
+            0.05324346199631691,
+            -0.0011056812945753336,
+            0.017912890762090683,
+            -0.004576410632580519,
+            0.015539304353296757,
+            -0.007300057914108038,
+            -0.016111355274915695,
+            0.029131926596164703,
+            0.0035561099648475647,
+            -0.0007540191290900111,
+            0.012106996960937977,
+            0.004469684325158596,
+            0.04149506613612175,
+            -0.005887005943804979,
+            -0.026058217510581017,
+            0.011910621076822281,
+            -0.017187153920531273,
+            0.020371858030557632,
+            0.00769707839936018,
+            -0.007607428822666407,
+            0.001932807732373476,
+            0.012909576296806335,
+            -0.027116939425468445,
+            0.019705887883901596,
+            -0.020645076408982277,
+            -0.045149363577365875,
+            0.009280892089009285,
+            -0.01175693515688181,
+            0.023035738617181778,
+            0.001117421081289649,
+            -0.010800670832395554,
+            -0.03507443144917488,
+            0.0021387888118624687,
+            0.0039040367119014263,
+            -0.03372541442513466,
+            0.006548706907778978,
+            0.01863008923828602,
+            0.015539304353296757,
+            0.009485806338489056,
+            -0.012465596199035645,
+            0.003758889390155673,
+            0.0031847036443650723,
+            -0.012371676973998547,
+            0.02776583470404148,
+            -0.004742903169244528,
+            -0.031983643770217896,
+            0.035859934985637665,
+            -0.02733892947435379,
+            0.009468729607760906,
+            -0.020730456337332726,
+            -0.012346062809228897,
+            -0.01608574204146862,
+            0.00441418681293726,
+            -0.02148180827498436,
+            -0.002561423694714904,
+            -0.02983205020427704,
+            0.027577996253967285,
+            0.009383349679410458,
+            -0.015249010175466537,
+            -0.004862436093389988,
+            0.009101592935621738,
+            -0.025255639106035233,
+            0.007846495136618614,
+            0.012585128657519817,
+            -0.010646984912455082,
+            -0.01367800310254097,
+            -0.00706099160015583,
+            -0.0014685495989397168,
+            0.005972386337816715,
+            0.0158210601657629,
+            0.04672037065029144,
+            0.004431263078004122,
+            0.03712356835603714,
+            0.02337726205587387,
+            0.013532855547964573,
+            0.0037012575194239616,
+            0.0047898623161017895,
+            -0.0067322757095098495,
+            0.04012897238135338,
+            0.006249873898923397,
+            0.0219599399715662,
+            0.008247784338891506,
+            -0.044671230018138885,
+            0.008503926917910576,
+            0.03264961391687393,
+            0.026621730998158455,
+            0.018715469166636467,
+            0.009588262997567654,
+            -0.0024013349320739508,
+            0.021191513165831566,
+            -0.020252324640750885,
+            -0.01705908216536045,
+            0.0035902622621506453,
+            -0.007530585862696171,
+            -0.0056650154292583466,
+            -0.009716334752738476,
+            0.03387909755110741,
+            -0.003229528432711959,
+            -0.008862526156008244,
+            0.018186109140515327,
+            -0.007432397920638323,
+            0.00475997943431139,
+            0.006245605181902647,
+            0.0011163539020344615,
+            0.02739015780389309,
+            0.008683227002620697,
+            0.027680452913045883,
+            0.005391797050833702,
+            0.007573276292532682,
+            0.0058571225963532925,
+            -0.009451653808355331,
+            0.018920384347438812,
+            -0.0097590247169137,
+            -0.033366814255714417,
+            -0.009443115442991257,
+            0.0016393112018704414,
+            0.014070754870772362,
+            0.0004023569927085191,
+            -0.014062216505408287,
+            -0.011893544346094131,
+            0.007859301753342152,
+            0.012533900327980518,
+            0.02107197977602482,
+            0.018169032409787178,
+            -0.005016121547669172,
+            -0.024726277217268944,
+            0.012593667022883892,
+            0.011056812480092049,
+            -0.01861301250755787,
+            0.008465506136417389,
+            0.012542438693344593,
+            0.057410046458244324,
+            -0.044705383479595184,
+            0.016538260504603386,
+            0.02151595987379551,
+            -0.004896588623523712,
+            -0.015249010175466537,
+            -0.010851899161934853,
+            0.0031633584294468164,
+            -0.017477449029684067,
+            0.014975790865719318,
+            -0.01798119582235813,
+            0.011193421669304371,
+            -0.014745263382792473,
+            0.00743666710332036,
+            0.012866885401308537,
+            -0.009682182222604752,
+            0.003942457959055901,
+            0.020371858030557632,
+            0.0028068935498595238,
+            -0.0077867284417152405,
+            0.005387527868151665,
+            0.01390853151679039,
+            0.020354781299829483,
+            0.006066305562853813,
+            -0.0005933965439908206,
+            0.019876649603247643,
+            -0.028175661340355873,
+            -0.025170257315039635,
+            0.01909114606678486,
+            -0.03551841154694557,
+            0.005472908727824688,
+            0.02790244296193123,
+            -0.04296361654996872,
+            -0.05020390823483467,
+            0.0027407235465943813,
+            -0.039719145745038986,
+            0.0004191663465462625,
+            -0.0023287611547857523,
+            0.01536854263395071,
+            -0.004057722166180611,
+            -0.017212767153978348,
+            0.0146428057923913,
+            -0.012397291138768196,
+            -0.0067322757095098495,
+            0.009835867211222649,
+            0.009016212075948715,
+            -0.024572592228651047,
+            -0.0016222350532189012,
+            -0.030890770256519318,
+            0.009169897064566612,
+            -0.017844585701823235,
+            -0.019142374396324158,
+            -0.025528857484459877,
+            -0.0005656477296724916,
+            -0.010638446547091007,
+            -0.001638244022615254,
+            0.051501695066690445,
+            0.021652569994330406,
+            0.020166942849755287,
+            -0.007726961746811867,
+            -0.023650480434298515,
+            0.05457540228962898,
+            0.0018132745753973722,
+            0.03814813867211342,
+            -0.012380215339362621,
+            -0.0002006448630709201,
+            -0.0008852921309880912,
+            0.022113624960184097,
+            -0.017101772129535675,
+            -0.0011547752656042576,
+            -0.011022660881280899,
+            -0.00586566049605608,
+            -0.0070780678652226925,
+            0.014045140706002712,
+            0.015974747017025948,
+            -0.002610517665743828,
+            0.014156135730445385,
+            0.01221799198538065,
+            0.022694215178489685,
+            0.03130060061812401,
+            0.016230888664722443,
+            -0.0034664601553231478,
+            -0.022199006751179695,
+            -0.012918113730847836,
+            -0.008200825192034245,
+            0.008999135345220566,
+            -0.008525271899998188,
+            -0.0031847036443650723,
+            -0.02631436102092266,
+            -0.020730456337332726,
+            -0.011312955059111118,
+            -0.01292665209621191,
+            -0.0031462821643799543,
+            0.02641681768000126,
+            -0.002469639526680112,
+            -0.03763585537672043,
+            -0.01511240005493164,
+            -0.03524519130587578,
+            -0.015607609413564205,
+            -0.0017876604106277227,
+            -0.01748598739504814,
+            0.016538260504603386,
+            -0.003780234605073929,
+            -0.007833687588572502,
+            -0.0008964983280748129,
+            -0.018169032409787178,
+            0.009690719656646252,
+            -0.011048275046050549,
+            -0.000543768925126642,
+            -0.0017503062263131142,
+            0.013345018029212952,
+            -0.008205094374716282,
+            0.008973521180450916,
+            -0.01412198320031166,
+            0.006868884898722172,
+            -0.048120614141225815,
+            -0.027731681242585182,
+            -0.027099864557385445,
+            0.022557605057954788,
+            -0.015214857645332813,
+            -0.023189423605799675,
+            0.008734455332159996,
+            -0.013686541467905045,
+            0.008939368650317192,
+            -0.012021616101264954,
+            0.008388662710785866,
+            0.019210679456591606,
+            0.009024749509990215,
+            -0.002614786848425865,
+            -0.013669464737176895,
+            -0.02390662208199501,
+            -0.02585330419242382,
+            0.023650480434298515,
+            -0.026485122740268707,
+            0.01071528997272253,
+            -0.012021616101264954,
+            0.008585038594901562,
+            -0.029917430132627487,
+            -0.020730456337332726,
+            0.0044910297729074955,
+            0.011347107589244843,
+            -0.021140284836292267,
+            0.044671230018138885,
+            0.04043634235858917,
+            0.029985735192894936,
+            -0.020252324640750885,
+            0.05139923840761185,
+            -0.022796671837568283,
+            0.007312864996492863,
+            0.036440521478652954,
+            0.030446790158748627,
+            0.026092370972037315,
+            -0.015581995248794556,
+            -0.012294834479689598,
+            -0.015496614389121532,
+            0.013404784724116325,
+            -0.024333525449037552,
+            0.002821835223585367,
+            -0.02042308636009693,
+            0.013643850572407246,
+            0.03451091796159744,
+            -0.004917933605611324,
+            0.029934506863355637,
+            -0.013720693998038769,
+            0.030361410230398178,
+            0.0017652479000389576,
+            -0.026587579399347305,
+            0.015667375177145004,
+            -0.014232978224754333,
+            0.004418455995619297,
+            0.017964119091629982,
+            -0.006087650544941425,
+            0.013575546443462372,
+            -0.061337560415267944,
+            0.028141509741544724,
+            0.01758844405412674,
+            0.01123611256480217,
+            -0.002180411946028471,
+            -0.024726277217268944,
+            -0.0021142419427633286,
+            0.009340658783912659,
+            -0.0016745307948440313,
+            0.011859391815960407,
+            0.01193623524159193,
+            -0.033418044447898865,
+            0.010578680783510208,
+            -0.013635313138365746,
+            0.024299373850226402,
+            0.006258412264287472,
+            0.0007476155878975987,
+            -0.03961668908596039,
+            -0.00560524920001626,
+            0.014830644242465496,
+            0.01095435582101345,
+            -0.005989462602883577,
+            -0.04624223709106445,
+            -0.030105268582701683,
+            0.025563009083271027,
+            0.005110040307044983,
+            -0.05167245492339134,
+            0.014557424932718277,
+            -0.015334390103816986,
+            0.018766699358820915,
+            -0.008708841167390347,
+            -0.025972837582230568,
+            -0.029985735192894936,
+            -0.017187153920531273,
+            0.016726097092032433,
+            0.048564594238996506,
+            0.004207138437777758,
+            0.013345018029212952,
+            -0.0048538981936872005,
+            0.006651164032518864,
+            -0.024760430678725243,
+            0.012030153535306454,
+            -0.027151092886924744,
+            -0.02971251681447029,
+            0.029012393206357956,
+            0.009981014765799046,
+            -0.011330030858516693,
+            0.019330210983753204,
+            -0.017084697261452675,
+            -0.0014141193823888898,
+            0.02240392006933689,
+            -0.0006163425859995186,
+            -0.007295788731426001,
+            -0.010382304899394512,
+            0.013618236407637596,
+            -0.0014354645973071456,
+            -0.01762259565293789,
+            -0.009571187198162079,
+            0.04241717979311943,
+            0.010092009790241718,
+            0.018835002556443214,
+            0.03261546418070793,
+            -0.0018933190731331706,
+            -0.02400907874107361,
+            -0.0027215126901865005,
+            -0.026638807728886604,
+            -0.017033468931913376,
+            -0.014548887498676777,
+            0.016674868762493134,
+            0.023530947044491768,
+            0.015249010175466537,
+            -0.027526767924427986,
+            -0.030019886791706085,
+            -0.01950097270309925,
+            -0.022199006751179695,
+            -0.0034493841230869293,
+            -0.0004213008505757898,
+            -0.026502197608351707,
+            0.007005494087934494,
+            0.0341181643307209,
+            0.03357172757387161,
+            -0.0010870042024180293,
+            -0.009016212075948715,
+            -0.010638446547091007,
+            0.0007721625734120607,
+            0.015231933444738388,
+            -0.015018481761217117,
+            0.0012326851719990373,
+            0.005028928630053997,
+            0.005383259151130915,
+            0.0029285612981766462,
+            -0.007095144130289555,
+            -0.05747834965586662,
+            0.003961668815463781,
+            0.02544347569346428,
+            0.005084426142275333,
+            -0.0008377990452572703,
+            0.02629728429019451,
+            0.0008276600856333971,
+            0.008123982697725296,
+            -0.03142013028264046,
+            0.011108040809631348,
+            0.01803242415189743,
+            0.015163629315793514,
+            -0.018151957541704178,
+            0.0122521435841918,
+            0.0064206356182694435,
+            0.004811207763850689,
+            -0.00030950538348406553,
+            0.010143238119781017,
+            0.028636718168854713,
+            0.019620506092905998,
+            -0.014736725017428398,
+            0.023821242153644562,
+            0.0006435577524825931,
+            -0.012072844430804253,
+            -0.024333525449037552,
+            -0.001040044822730124,
+            0.0029883277602493763,
+            -0.011176345869898796,
+            0.014702572487294674,
+            -0.0023949311580508947,
+            0.0007956422632560134,
+            0.04139260947704315,
+            0.0437491200864315,
+            0.013481627218425274,
+            -0.011295879259705544,
+            0.003304236801341176,
+            0.026502197608351707,
+            0.020132791250944138,
+            0.004862436093389988,
+            0.011458102613687515,
+            -0.03331558406352997,
+            -0.019620506092905998,
+            -0.002981924219056964,
+            0.0014290610561147332,
+            -0.0026788224931806326,
+            0.03876287862658501,
+            0.029524678364396095,
+            0.009682182222604752,
+            -0.005741858389228582,
+            -0.011364183388650417,
+            0.013643850572407246,
+            0.020201096311211586,
+            -0.023599252104759216,
+            -0.0038015798199921846,
+            0.0016745307948440313,
+            -0.017170077189803123,
+            0.02288205362856388,
+            -0.0070311082527041435,
+            -0.00928943045437336,
+            0.02645096927881241,
+            0.09186973422765732,
+            -0.00464471522718668,
+            0.012141148559749126,
+            -0.03948007896542549,
+            -0.021823331713676453,
+            -0.006975610740482807,
+            -0.0036671049892902374,
+            0.015402695164084435,
+            0.02739015780389309,
+            0.009135744534432888,
+            0.007133565377444029,
+            0.017878737300634384,
+            -0.004606293980032206,
+            0.012508286163210869,
+            -0.02826104313135147,
+            0.020166942849755287,
+            -0.005007583647966385,
+            -0.0038335975259542465,
+            -0.011389797553420067,
+            -0.005989462602883577,
+            0.011765473522245884,
+            0.005468640010803938,
+            0.017016392201185226,
+            0.0019893725402653217,
+            -0.0024760430678725243,
+            0.0014333301223814487,
+            0.003031018190085888,
+            -0.010877513326704502,
+            -0.009383349679410458,
+            -0.021396426483988762,
+            0.003827193984761834,
+            0.0009914844995364547,
+            -0.007884915918111801,
+            0.012755890376865864,
+            0.005502792075276375,
+            0.024043232202529907,
+            -0.028175661340355873,
+            -0.0030609015375375748,
+            0.00633525475859642,
+            -0.008956445381045341,
+            0.010390842333436012,
+            0.012380215339362621,
+            -0.008640536107122898,
+            0.022830823436379433,
+            -0.01466841995716095,
+            0.014762339182198048,
+            0.01170570682734251,
+            -0.021601341664791107,
+            -0.004155910108238459,
+            -0.009938324801623821,
+            -0.02192578837275505,
+            0.020252324640750885,
+            -0.018715469166636467,
+            0.005033197812736034,
+            0.00073320756200701,
+            0.006774966139346361,
+            -0.0010496501345187426,
+            -0.01740914396941662,
+            0.010348152369260788,
+            -0.008785683661699295,
+            0.006839001551270485,
+            -0.005058811977505684,
+            0.018254414200782776,
+            0.012166762724518776,
+            0.0028090281412005424,
+            -0.0011622461024671793,
+            0.014600115828216076,
+            -0.00372900627553463,
+            -0.018886230885982513,
+            -0.0036713741719722748,
+            0.01633334532380104,
+            0.01051037572324276,
+            0.0002606782363727689,
+            -0.010305461473762989,
+            -0.0146342683583498,
+            0.014139058999717236,
+            0.000977076473645866,
+            -0.0018122073961421847,
+            -0.0005285604856908321,
+            -0.03760170191526413,
+            0.002606248715892434,
+            0.013532855547964573,
+            0.012704662047326565,
+            -0.005818700883537531,
+            -0.017477449029684067,
+            0.007983104325830936,
+            -0.004190062638372183,
+            -0.0003204447857569903,
+            0.01661510206758976,
+            -0.019774191081523895,
+            0.01002370472997427,
+            -0.014779414981603622,
+            -0.03278622403740883,
+            -0.0029776552692055702,
+            -0.008380125276744366,
+            0.005822970066219568,
+            -0.005592442117631435,
+            -0.0071805245243012905,
+            -0.01414759736508131,
+            -0.005741858389228582,
+            -0.02979789674282074,
+            0.008947907015681267,
+            0.0317445807158947,
+            0.004473953507840633,
+            -0.028978241607546806,
+            -0.012158225290477276,
+            0.025580085813999176,
+            -0.0024931191001087427,
+            -0.006915844511240721,
+            0.003009672975167632,
+            0.0012572321575134993,
+            -0.005784548819065094,
+            -0.009656568057835102,
+            -0.012935190461575985,
+            0.013063261285424232,
+            0.01320840884000063,
+            0.011244650930166245,
+            -0.03357172757387161,
+            0.010664060711860657,
+            0.01813488081097603,
+            -0.02042308636009693,
+            0.011748396791517735,
+            0.028585489839315414,
+            0.0029435029719024897,
+            0.010399380698800087,
+            0.007334209978580475,
+            0.008286206051707268,
+            -0.017153002321720123,
+            0.016683407127857208,
+            0.009186973795294762,
+            -0.012559514492750168,
+            0.004563603550195694,
+            0.007381169591099024,
+            0.008542348630726337,
+            0.01960342936217785,
+            -0.011603250168263912,
+            0.013609698973596096,
+            -0.0068816919811069965,
+            0.03321312740445137,
+            -0.011919158510863781,
+            -0.014881872572004795,
+            0.03229101747274399,
+            0.027560919523239136,
+            -0.004563603550195694,
+            0.017246920615434647,
+            -0.01943266950547695,
+            -0.01267050951719284,
+            0.019296059384942055,
+            0.0005907283630222082,
+            0.019313136115670204,
+            -0.011304416693747044,
+            -0.02595576085150242,
+            0.029012393206357956,
+            -0.008106905966997147,
+            -0.001259366748854518,
+            -0.015829598531126976,
+            0.0043458822183310986,
+            -0.0005789885181002319,
+            0.037328481674194336,
+            0.012653433717787266,
+            0.0011985328746959567,
+            0.026570502668619156,
+            -0.00769280968233943,
+            -0.025067800655961037,
+            0.001990439835935831,
+            -0.015214857645332813,
+            0.013336479663848877,
+            -0.027970748022198677,
+            -0.00560524920001626,
+            0.014506196603178978,
+            0.003026749240234494,
+            -0.002644670195877552,
+            0.01666633039712906,
+            -0.007513510063290596,
+            0.011791087687015533,
+            -0.019244831055402756,
+            0.016717558726668358,
+            -0.011517869308590889,
+            0.004473953507840633,
+            -0.011910621076822281,
+            0.026092370972037315,
+            -0.025563009083271027,
+            -0.00999809056520462,
+            0.013660927303135395,
+            -0.008354511111974716,
+            0.0006376878009177744,
+            -0.02252345345914364,
+            -0.008089830167591572,
+            -0.020371858030557632,
+            0.01679440215229988,
+            -0.005682091694325209,
+            0.011902082711458206,
+            -0.00306303589604795,
+            0.0025528857950121164,
+            -0.008585038594901562,
+            0.010322538204491138,
+            -0.0009199780761264265,
+            -0.017417682334780693,
+            -0.005968117620795965,
+            -0.03025895357131958,
+            -0.030361410230398178,
+            0.03072001039981842,
+            -0.00880275946110487,
+            0.003923247568309307,
+            0.02728770114481449,
+            -0.02540932409465313,
+            -0.005485715810209513,
+            0.001918933354318142,
+            0.022728366777300835,
+            -0.030651705339550972,
+            0.007914799265563488,
+            -0.009391887113451958,
+            -0.016452878713607788,
+            0.007184793706983328,
+            -0.0008874266059137881,
+            0.015880826860666275,
+            -0.0025635582860559225,
+            -0.04924764111638069,
+            -0.0007081269286572933,
+            0.017135925590991974,
+            -0.013806073926389217,
+            0.008081291802227497,
+            0.02151595987379551,
+            -0.009630953893065453,
+            -0.008439891040325165,
+            -0.0011334300506860018,
+            0.025989914312958717,
+            -0.022625910118222237,
+            0.030019886791706085,
+            0.005767472553998232,
+            0.004713019821792841,
+            0.005340568721294403,
+            0.002962713595479727,
+            0.01730668731033802,
+            0.002080089645460248,
+            0.004258367232978344,
+            0.025238562375307083,
+            0.013148642145097256,
+            0.0037418133579194546,
+            -0.0032978332601487637,
+            -0.0030352871399372816,
+            0.016538260504603386,
+            -0.018442250788211823,
+            0.02100367471575737,
+            -0.0004714620590675622,
+            0.00013560871593654156,
+            0.016256503760814667,
+            -0.01484772004187107,
+            -0.027219397947192192,
+            0.03075416199862957,
+            -0.022335615009069443,
+            0.01757136732339859,
+            0.023770013824105263,
+            -0.01339624635875225,
+            -0.004922202788293362,
+            -0.007073798682540655,
+            0.001534719718620181,
+            0.018339794129133224,
+            -0.0134218605235219,
+            -0.023308956995606422,
+            -0.00030790449818596244,
+            0.0220111683011055,
+            0.02293328195810318,
+            0.028175661340355873,
+            -0.021140284836292267,
+            0.01412198320031166,
+            0.007466550450772047,
+            -0.008311820216476917,
+            0.016469955444335938,
+            0.008452698588371277,
+            -0.0025272713974118233,
+            0.021293969824910164,
+            0.0075903525575995445,
+            0.004576410632580519,
+            -0.008896678686141968,
+            -0.013242561370134354,
+            -0.01267050951719284,
+            -0.006749351508915424,
+            -0.006164493504911661,
+            -0.008264861069619656,
+            -0.01709323562681675,
+            0.09548988193273544,
+            -0.011867930181324482,
+            -0.0028410458471626043,
+            -0.009195511229336262,
+            0.0026041141245514154,
+            -0.006949996575713158,
+            0.0013479492627084255,
+            0.023086966946721077,
+            -0.022728366777300835,
+            -0.03671374171972275,
+            0.011167807504534721,
+            -0.004070529248565435,
+            0.006288295611739159,
+            0.03712356835603714,
+            0.00038314631092362106,
+            -0.016589488834142685,
+            -6.773765198886395e-05,
+            -0.003372541395947337,
+            0.01295226626098156,
+            0.0028197006322443485,
+            -0.0036158766597509384,
+            0.0109799699857831,
+            0.029200231656432152,
+            -0.017810434103012085,
+            -0.02144765481352806,
+            -0.012525362893939018,
+            0.0032615463715046644,
+            -0.002578499959781766,
+            0.003707661060616374,
+            0.0051996903494000435,
+            -0.014053679071366787,
+            -0.0134218605235219,
+            0.018817927688360214,
+            -0.016094280406832695,
+            0.0008025794522836804,
+            -0.012627819553017616,
+            -0.008222170174121857,
+            0.01753721572458744,
+            0.01757136732339859,
+            -0.03406693786382675,
+            -0.009921248070895672,
+            -0.03855796530842781,
+            0.018749622628092766,
+            0.01643580198287964,
+            0.014540349133312702,
+            -0.020884141325950623,
+            -0.012644895352423191,
+            -0.004520913120359182,
+            -0.02981497347354889,
+            -0.02243807166814804,
+            -0.026536351069808006,
+            0.009605339728295803,
+            0.00365856708958745,
+            0.027595072984695435,
+            0.005259457044303417,
+            -0.006262681446969509,
+            0.00801725685596466,
+            -0.01025423314422369,
+            0.00755620002746582,
+            -0.02146473154425621,
+            0.007107951212674379,
+            -0.0011409008875489235,
+            0.0012529630912467837,
+            0.005592442117631435,
+            -0.008853988721966743,
+            -0.01735791563987732,
+            -0.0079276068136096,
+            0.01466841995716095,
+            -0.015752756968140602,
+            -0.00821790099143982,
+            0.020798761397600174,
+            0.009383349679410458,
+            0.00931504461914301,
+            0.015616146847605705,
+            0.045661646872758865,
+            0.008273398503661156,
+            0.00783795677125454,
+            -0.011056812480092049,
+            0.009648029692471027,
+            -0.01813488081097603,
+            0.00745801255106926,
+            -0.019330210983753204,
+            -0.006292564328759909,
+            -0.019722962751984596,
+            0.04012897238135338,
+            -0.009366272948682308,
+            -0.01635042205452919,
+            0.033435117453336716,
+            -0.011808163486421108,
+            0.01910822093486786,
+            0.0035475720651447773,
+            -0.022830823436379433,
+            0.03352050110697746,
+            0.003876287955790758,
+            0.012918113730847836,
+            -0.025682542473077774,
+            0.0035859933122992516,
+            -0.00877714529633522,
+            0.006032153032720089,
+            -0.011586173437535763,
+            0.003494208911433816,
+            0.0023885276168584824,
+            -0.004841091111302376,
+            -0.008760069496929646,
+            0.018698394298553467,
+            0.006595666520297527,
+            0.0002183880569646135,
+            0.028056127950549126,
+            -0.000954130373429507,
+            0.018220260739326477,
+            6.110063259257004e-05,
+            0.006010808050632477,
+            0.027680452913045883,
+            0.002821835223585367,
+            0.006809118203818798,
+            -0.01714446395635605,
+            -0.0009199780761264265,
+            -0.009485806338489056,
+            0.009596801362931728,
+            0.02639974094927311,
+            -0.010612832382321358,
+            -0.031078608706593513,
+            0.022574681788682938,
+            -0.007048184517771006,
+            0.015966208651661873,
+            0.002999000484123826,
+            -0.014062216505408287,
+            -0.0023437028285115957,
+            -0.027680452913045883,
+            0.014454968273639679,
+            -0.017349377274513245,
+            -0.024265222251415253,
+            -0.007231753319501877,
+            -0.01045914739370346,
+            -0.004845359828323126,
+            0.038353051990270615,
+            0.012474133633077145,
+            -0.024572592228651047,
+            0.006817656569182873,
+            -0.003974475897848606,
+            -0.005989462602883577,
+            0.0057119750417768955,
+            0.005976655520498753,
+            0.004098278004676104,
+            0.010945817455649376,
+            -0.0065700518898665905,
+            -0.01320840884000063,
+            -0.03324728086590767,
+            -0.02240392006933689,
+            0.014941639266908169,
+            0.00028068936080671847,
+            -0.01315718051046133,
+            0.028483033180236816,
+            -0.01002370472997427,
+            -0.005575365852564573,
+            0.0071292961947619915,
+            0.00013887719251215458,
+            -0.0022796671837568283,
+            -0.017844585701823235,
+            0.002273263642564416,
+            0.006245605181902647,
+            -0.014950176700949669,
+            0.009844405576586723,
+            0.0032124524004757404,
+            -0.014540349133312702,
+            0.005635132547467947,
+            -0.0015027020126581192,
+            -0.003562513506039977,
+            -0.02481165900826454,
+            0.0059595792554318905,
+            0.00708233704790473,
+            -0.0018036692636087537,
+            0.002390662208199501,
+            -0.018169032409787178,
+            0.01801534742116928,
+            0.001955220242962241,
+            0.0071762558072805405,
+            -0.029166078194975853,
+            0.007282981649041176,
+            -0.006740813609212637,
+            0.0015080382581800222,
+            -0.0195692777633667,
+            0.009161358699202538,
+            0.020798761397600174,
+            0.012491210363805294,
+            -0.0005250918911769986,
+            0.010442070662975311,
+            0.008175211027264595,
+            -0.012704662047326565,
+            0.003950996324419975,
+            -0.02050846628844738,
+            -0.007308595813810825,
+            -0.010057857260107994,
+            0.011372721754014492,
+            -0.004219945520162582,
+            0.015983285382390022,
+            0.009511420503258705,
+            0.005600980017334223,
+            -0.009699258022010326,
+            0.00743666710332036,
+            0.01292665209621191,
+            0.005874198395758867,
+            -0.0011099503608420491,
+            0.017827508971095085,
+            0.07363239675760269,
+            0.005336299538612366,
+            0.013618236407637596,
+            6.870485958643258e-05,
+            -0.013071799650788307,
+            0.0015624684747308493,
+            -0.004939279053360224,
+            -0.01748598739504814,
+            0.026126522570848465,
+            0.026587579399347305,
+            -0.006382214371114969,
+            0.0004917400074191391,
+            0.009016212075948715,
+            0.005558289587497711,
+            0.005485715810209513,
+            0.022216081619262695,
+            -0.0077867284417152405,
+            0.01753721572458744,
+            0.024179840460419655,
+            0.005541213322430849,
+            -0.0019093280425295234,
+            -0.0022305732127279043,
+            0.009152821265161037,
+            0.004209273029118776,
+            -0.009050363674759865,
+            0.01150079257786274,
+            -0.01437812577933073,
+            0.01909114606678486,
+            0.011389797553420067,
+            0.02146473154425621,
+            0.007073798682540655,
+            -0.01704200729727745,
+            -0.018356870859861374,
+            0.0041281613521277905,
+            -0.0013479492627084255,
+            0.0059083509258925915,
+            0.0063224476762115955,
+            0.005477177910506725,
+            -0.011611787602305412,
+            -0.0041793896816670895,
+            0.010356690734624863,
+            -0.002471773885190487,
+            -0.0008500725380145013,
+            0.01757136732339859,
+            -0.01268758624792099,
+            0.01168009266257286,
+            -0.001839956152252853,
+            -0.0009984216885641217,
+            0.009682182222604752,
+            0.0014354645973071456,
+            -0.011825240217149258,
+            -0.004166582599282265,
+            0.0021003675647079945,
+            0.01217530108988285,
+            -0.013259637169539928,
+            -0.01048476155847311,
+            0.011825240217149258,
+            -0.006168762221932411,
+            0.009169897064566612,
+            -0.009579724632203579,
+            -0.0025315405800938606,
+            -0.00039648707024753094,
+            -0.008085560984909534,
+            -0.0014578771078959107,
+            0.007820880971848965,
+            0.0008324627415277064,
+            0.010518914088606834,
+            0.011560559272766113,
+            0.01150079257786274,
+            0.003560379147529602,
+            0.004002224653959274,
+            -0.002964847953990102,
+            -0.0064206356182694435,
+            -0.011432488448917866,
+            8.15119783510454e-05,
+            0.005776010919362307,
+            -0.02242099680006504,
+            -0.00024800453684292734,
+            0.008465506136417389,
+            0.005046004895120859,
+            -0.0007710952777415514,
+            0.020849989727139473,
+            0.001264702994376421,
+            0.007820880971848965,
+            -0.025221485644578934,
+            0.0011238247388973832,
+            0.006890229880809784,
+            0.007718423847109079,
+            0.012798581272363663,
+            -0.028636718168854713,
+            -0.0038528083823621273,
+            -0.0038634808734059334,
+            0.0171273872256279,
+            -0.00011606451880652457,
+            -0.0033106403425335884,
+            0.021686721593141556,
+            -0.0007833687704987824,
+            -0.003430173499509692,
+            -0.00755620002746582,
+            -0.005579635035246611,
+            0.008004449307918549,
+            0.012994957156479359,
+            0.024623820558190346,
+            -0.0019968433771282434,
+            -0.0055967108346521854,
+            -0.00809409935027361,
+            0.0244530588388443,
+            0.015249010175466537,
+            0.01863008923828602,
+            -0.0219087116420269,
+            0.010638446547091007,
+            -0.013865840621292591,
+            0.009067440405488014,
+            0.011714245192706585,
+            -0.010911665856838226,
+            0.0004290384822525084,
+            -0.010757979936897755,
+            -0.017135925590991974,
+            -0.022711291909217834,
+            0.00502039073035121,
+            0.02839765138924122,
+            -0.0002676154254004359,
+            0.007193332072347403,
+            -0.016358960419893265,
+            -0.0018442251021042466,
+            -0.011312955059111118,
+            -0.025750847533345222,
+            -0.016222350299358368,
+            -0.021772101521492004,
+            -0.02141350321471691,
+            -0.0038592119235545397,
+            -0.002131317974999547,
+            -0.01907406933605671,
+            0.00013153978215996176,
+            -0.0012817791430279613,
+            0.01902284100651741,
+            -0.0014888276346027851,
+            0.0015624684747308493,
+            0.012337524443864822,
+            0.012943727895617485,
+            -0.012824195437133312,
+            -0.004228483885526657,
+            -0.019261907786130905,
+            -0.024179840460419655,
+            0.0032487392891198397,
+            -0.007726961746811867,
+            0.013874378986656666,
+            -0.008234977722167969,
+            0.004260501824319363,
+            -0.017187153920531273,
+            0.014702572487294674,
+            -0.03579162806272507,
+            -0.024128612130880356,
+            0.0006531630642712116,
+            -0.0040299734100699425,
+            -0.009605339728295803,
+            -0.02585330419242382,
+            0.01198746357113123,
+            -0.019620506092905998,
+            0.002104636514559388,
+            -0.018442250788211823,
+            0.011398335918784142,
+            -0.015283161774277687,
+            0.018681317567825317,
+            -0.00025520852068439126,
+            0.03179580718278885,
+            0.004405648913234472,
+            0.007978835143148899,
+            -0.004525181837379932,
+            0.003327716374769807,
+            -0.005468640010803938,
+            0.006365138106048107,
+            0.006988417822867632,
+            0.008055677637457848,
+            0.007825150154531002,
+            -0.004768517334014177,
+            0.004061991348862648,
+            0.005110040307044983,
+            0.030515095219016075,
+            0.028021976351737976,
+            0.0027706066612154245,
+            0.0014674824196845293,
+            0.0034685947466641665,
+            0.004576410632580519,
+            -0.00038021133514121175,
+            0.008862526156008244,
+            -0.011287340894341469,
+            -0.005391797050833702,
+            0.01417321152985096,
+            -0.012448519468307495,
+            -0.0037695621140301228,
+            0.015377080999314785,
+            -0.015086785890161991,
+            -0.026092370972037315,
+            0.021823331713676453,
+            -0.006552976090461016,
+            -0.002450428670272231,
+            -0.006689585279673338,
+            0.008115444332361221,
+            -0.005946772173047066,
+            0.004194331355392933,
+            0.028107358142733574,
+            -0.005737589206546545,
+            -0.010945817455649376,
+            0.012636357918381691,
+            -0.002330895746126771,
+            -0.007948951795697212,
+            0.02291620522737503,
+            0.035894084721803665,
+            0.00999809056520462,
+            -0.026041142642498016,
+            -0.007381169591099024,
+            0.02144765481352806,
+            -0.00973341055214405,
+            -0.021755026653409004,
+            0.008273398503661156,
+            0.007522047962993383,
+            -0.005421680398285389,
+            0.018937459215521812,
+            0.004239156376570463,
+            -0.010792132467031479,
+            0.009041826240718365,
+            0.022711291909217834,
+            0.021686721593141556,
+            0.008947907015681267,
+            -0.023548023775219917,
+            0.01025423314422369,
+            -0.026228979229927063,
+            -0.013831689022481441,
+            -0.023821242153644562,
+            0.003581724362447858,
+            0.003857077332213521,
+            -0.003257277188822627,
+            0.026536351069808006,
+            0.011876468546688557,
+            0.011645940132439137,
+            0.0030950538348406553,
+            0.023052813485264778,
+            -0.015223395079374313,
+            -0.028107358142733574,
+            -0.0009733410552144051,
+            -0.015479537658393383,
+            -0.0018282162491232157,
+            0.002166537567973137,
+            0.007150641642510891,
+            0.005852853413671255,
+            0.01953512616455555,
+            0.000376209121895954,
+            0.0019584218971431255,
+            -2.4263486920972355e-05,
+            0.010100548155605793,
+            -0.015727141872048378,
+            -0.019705887883901596,
+            -0.008922292850911617,
+            6.470263178925961e-05,
+            0.011637401767075062,
+            -0.005208228249102831,
+            -3.448583447607234e-05,
+            0.007846495136618614,
+            -0.015897903591394424,
+            -0.0097590247169137,
+            -0.014412278309464455,
+            -0.0029306956566870213,
+            -0.005071619059890509,
+            0.03136890381574631,
+            0.014437892474234104,
+            -0.004747172351926565,
+            -0.011543483473360538,
+            -0.005741858389228582,
+            0.03370833769440651,
+            0.0017204229952767491,
+            0.012422905303537846,
+            0.0146342683583498,
+            0.015684451907873154,
+            0.027014482766389847,
+            0.012807118706405163,
+            0.003018211107701063,
+            0.002516598906368017,
+            0.02346264198422432,
+            -0.013951221480965614,
+            0.008730186149477959,
+            -0.011645940132439137,
+            -0.003521957667544484,
+            -0.010937280021607876,
+            0.015078248456120491,
+            -0.013003494590520859,
+            -0.011278802528977394,
+            0.023069890215992928,
+            -0.0004381102044135332,
+            0.013276712968945503,
+            -0.0004242358263581991,
+            -0.003953130915760994,
+            -0.02534101903438568,
+            0.0015464596217498183,
+            0.0006531630642712116,
+            -0.004401379730552435,
+            0.010578680783510208,
+            -0.027475539594888687,
+            0.0013660927070304751,
+            0.015240471810102463,
+            -0.007116489112377167,
+            0.00038981667603366077,
+            -0.010826284997165203,
+            0.005904081743210554,
+            0.00011606451880652457,
+            -0.02385539375245571,
+            0.015488076023757458,
+            -0.006698123179376125,
+            0.01243144366890192,
+            -0.0018548977095633745,
+            -0.013788998126983643,
+            0.008687496185302734,
+            0.019296059384942055,
+            0.027560919523239136,
+            -0.007825150154531002,
+            0.0005528406472876668,
+            -0.0001855698210420087,
+            -0.006027883850038052,
+            -0.02737308293581009,
+            0.02054261974990368,
+            -0.007863570936024189,
+            0.01953512616455555,
+            -0.009682182222604752,
+            -0.018186109140515327,
+            -0.014856258407235146,
+            0.011927696876227856,
+            0.007603159639984369,
+            0.0010304395109415054,
+            0.01489894837141037,
+            -0.024077383801341057,
+            0.020867066457867622,
+            0.0033383890986442566,
+            -0.0064718639478087425,
+            0.008303281851112843,
+            0.004230618476867676,
+            0.004695943556725979,
+            -0.008384393528103828,
+            0.007658657152205706,
+            0.007569007575511932,
+            -0.007253098301589489,
+            0.004326671827584505,
+            -0.04057295247912407,
+            0.010732365772128105,
+            -0.011090965010225773,
+            0.005878467578440905,
+            -0.004742903169244528,
+            -0.022745443508028984,
+            -0.035825781524181366,
+            0.004567872267216444,
+            0.019313136115670204,
+            -0.020235247910022736,
+            0.018408099189400673,
+            0.012124072760343552,
+            0.014574501663446426,
+            0.01859593763947487,
+            0.008853988721966743,
+            0.01461719162762165,
+            0.0438857264816761,
+            -0.024231068789958954,
+            6.483604374807328e-05,
+            0.008734455332159996,
+            0.011381260119378567,
+            -0.013404784724116325,
+            -0.010126162320375443,
+            0.005784548819065094,
+            -0.0015197781613096595,
+            0.04067540913820267,
+            -0.019210679456591606,
+            0.010886050760746002,
+            -0.010962894186377525,
+            0.008371586911380291,
+            -0.009562648832798004,
+            -0.0036158766597509384,
+            -0.012559514492750168,
+            -0.011893544346094131,
+            0.014745263382792473,
+            -0.019671734422445297,
+            -0.005460101645439863,
+            -0.007282981649041176,
+            0.01048476155847311,
+            -0.006096188444644213,
+            -0.020730456337332726,
+            0.018647165969014168,
+            -0.0018079383298754692,
+            0.029968658462166786,
+            -0.012755890376865864,
+            0.00574612757191062,
+            0.022591758519411087,
+            -0.010604294948279858,
+            0.018732545897364616,
+            -0.004362958483397961,
+            0.008853988721966743,
+            0.0055625587701797485,
+            -0.0065188235603272915,
+            -0.0014685495989397168,
+            0.01389145478606224,
+            -0.0027385889552533627,
+            0.029934506863355637,
+            -0.017400605604052544,
+            -0.009818791411817074,
+            0.009707796387374401,
+            0.0073769004084169865,
+            0.0042370217852294445,
+            0.003878422547131777,
+            0.009579724632203579,
+            -0.01071528997272253,
+            0.0009728074073791504,
+            -0.021686721593141556,
+            -0.015453923493623734,
+            -0.0011483717244118452,
+            -0.004683136474341154,
+            -0.0018730411538854241,
+            -0.002467504935339093,
+            -0.02044016122817993,
+            0.004627638962119818,
+            -0.0021579996682703495,
+            -8.451364556094632e-05,
+            -0.024487212300300598,
+            -0.00769280968233943,
+            -0.010057857260107994,
+            -0.005161269102245569,
+            -0.003139878623187542,
+            0.018271489068865776,
+            0.0034856710117310286,
+            -0.004862436093389988,
+            0.0293197650462389,
+            0.005404604133218527,
+            0.01536854263395071,
+            0.00280262460000813,
+            -0.05334591865539551,
+            -0.011014122515916824,
+            -0.004563603550195694,
+            -0.009912709705531597,
+            -0.007261636666953564,
+            -0.008456967771053314,
+            -0.010681137442588806,
+            -0.0019061262719333172,
+            0.0032914294861257076,
+            -0.004170851781964302,
+            0.0122521435841918,
+            -0.01700785383582115,
+            0.025494705885648727,
+            -0.004017166327685118,
+            0.015744218602776527,
+            -0.011722782626748085,
+            0.011654478497803211,
+            -0.006194376386702061,
+            -0.010851899161934853,
+            0.010544528253376484,
+            0.0146342683583498,
+            -0.0030544979963451624,
+            0.020371858030557632,
+            -0.014736725017428398,
+            0.003197510726749897,
+            -0.008303281851112843,
+            -0.02143057994544506,
+            -0.015991821885108948,
+            0.011295879259705544,
+            0.03130060061812401,
+            0.010066395625472069,
+            -0.0037524858489632607,
+            0.004072663839906454,
+            0.021669644862413406,
+            -0.004352285992354155,
+            0.0063181789591908455,
+            0.03025895357131958,
+            -0.0010982104577124119,
+            0.017742129042744637,
+            0.0001935742620844394,
+            -0.005028928630053997,
+            -0.002362913452088833,
+            0.01808365248143673,
+            -0.002766337711364031,
+            0.0057119750417768955,
+            0.02441890724003315,
+            0.010997045785188675,
+            -0.0064718639478087425,
+            0.008230708539485931,
+            0.014309820719063282,
+            0.013874378986656666,
+            -0.015351466834545135,
+            -0.0122009152546525,
+            -0.011534945107996464,
+            -0.015778370201587677,
+            0.007291519548743963,
+            -0.019261907786130905,
+            -0.004049184266477823,
+            0.007829418405890465,
+            -0.015138015151023865,
+            -0.012807118706405163,
+            -0.023548023775219917,
+            -0.015914980322122574,
+            0.01863008923828602,
+            0.005357644986361265,
+            -0.01465988252311945,
+            -0.004369362257421017,
+            0.0013714289525523782,
+            0.006181569304317236,
+            -0.0054985228925943375,
+            0.021840406581759453,
+            -0.004098278004676104,
+            -0.011125117540359497,
+            0.019330210983753204,
+            0.0017897948855534196,
+            0.005097233224660158,
+            -0.0064206356182694435,
+            0.004550796467810869,
+            -0.011364183388650417,
+            0.022096550092101097,
+            -0.003470729338005185,
+            0.012636357918381691,
+            -0.0037033918779343367,
+            0.004631908144801855,
+            -0.001978699816390872,
+            0.00821790099143982,
+            0.004017166327685118,
+            0.010629909113049507,
+            -0.005656477529555559,
+            0.004525181837379932,
+            0.012491210363805294,
+            -0.01641872711479664,
+            0.007265905383974314,
+            0.004559334367513657,
+            -4.365760105429217e-05,
+            0.005780279636383057,
+            -0.007645850069820881,
+            -0.005417411215603352,
+            -0.015488076023757458,
+            0.010100548155605793,
+            0.030873695388436317,
+            -0.0016649254830554128,
+            -0.007240291219204664,
+            0.01704200729727745,
+            0.015735680237412453,
+            -0.003816521493718028,
+            0.0061218030750751495,
+            0.009280892089009285,
+            -0.003148416755720973,
+            0.01370361726731062,
+            -0.024487212300300598,
+            -0.029524678364396095,
+            0.01763967238366604,
+            -0.020867066457867622,
+            0.022779595106840134,
+            -0.019210679456591606,
+            -0.027731681242585182,
+            -0.01847640424966812,
+            -0.0032530082389712334,
+            -0.019842496141791344,
+            -0.01611989364027977,
+            0.02532394416630268,
+            0.00685180863365531,
+            -0.006949996575713158,
+            -0.031112760305404663,
+            0.04050464928150177,
+            0.014489120803773403,
+            0.038318898528814316,
+            -0.011603250168263912,
+            -2.554753518779762e-05,
+            -0.007504971697926521,
+            0.012064306065440178,
+            -0.01146664097905159,
+            -0.014258592389523983,
+            0.007509240880608559,
+            0.028175661340355873,
+            0.015231933444738388,
+            -0.015223395079374313,
+            -0.007944682613015175,
+            0.015991821885108948,
+            -0.007043915335088968,
+            -0.0005480379913933575,
+            -0.008836911991238594,
+            -0.014830644242465496,
+            -0.010066395625472069,
+            -0.0005312286084517837,
+            0.012243606150150299,
+            0.016179660335183144,
+            -0.0007257367251440883,
+            0.01390853151679039,
+            -0.009955400601029396,
+            -0.014420815743505955,
+            -0.004602024797350168,
+            0.005327761638909578,
+            -0.02252345345914364,
+            0.02105490304529667,
+            0.002292474266141653,
+            -0.009537034668028355,
+            0.0030865157023072243,
+            -0.004572141449898481,
+            0.012602205388247967,
+            0.009391887113451958,
+            -0.0038421356584876776,
+            -0.0036137420684099197,
+            0.00572051340714097,
+            -0.001715086749754846,
+            0.015001405030488968,
+            -0.011714245192706585,
+            0.011842316016554832,
+            -0.007432397920638323,
+            -0.00019931078713852912,
+            -0.009648029692471027,
+            -0.005238111596554518,
+            -0.010493299923837185,
+            0.0028794673271477222,
+            0.012943727895617485,
+            -0.014548887498676777,
+            0.012807118706405163,
+            -0.0008132520597428083,
+            -0.00949434470385313,
+            -0.015138015151023865,
+            -0.0055155991576612,
+            -0.011364183388650417,
+            -0.016683407127857208,
+            0.007069529499858618,
+            0.0007855033036321402,
+            0.019347287714481354,
+            0.0030054040253162384,
+            -0.004572141449898481,
+            -0.00017262929759453982,
+            0.018903307616710663,
+            -0.02580207586288452,
+            0.03403278440237045,
+            -0.00010045584349427372,
+            0.022711291909217834,
+            0.04251963645219803,
+            0.00954557303339243,
+            -0.006663971114903688,
+            -0.013754845596849918,
+            0.005801625084131956,
+            -0.013609698973596096,
+            0.009981014765799046,
+            0.014335434883832932,
+            -0.03235932067036629,
+            -0.003944592550396919,
+            -0.013976835645735264,
+            0.011586173437535763,
+            -0.007957490161061287,
+            0.0032850259449332952,
+            -0.010587218217551708,
+            0.011765473522245884,
+            0.01679440215229988,
+            -0.009075978770852089,
+            -0.004311730153858662,
+            -0.027492616325616837,
+            0.00032524747075513005,
+            0.016674868762493134,
+            -0.010911665856838226,
+            -0.01684563048183918,
+            0.010945817455649376,
+            -0.007218946237117052,
+            -0.015453923493623734,
+            -0.006898768246173859,
+            -0.03278622403740883,
+            -0.00020504731219261885,
+            -0.01076651830226183,
+            -0.003927516285330057,
+            -0.0028794673271477222,
+            0.0011024795239791274,
+            -0.0033917520195245743,
+            -0.0067066615447402,
+            -0.0025891726836562157,
+            0.0067664277739822865,
+            0.005011852364987135,
+            0.002080089645460248,
+            0.0005987327895127237,
+            -0.00023506399884354323,
+            0.009255277924239635,
+            0.010168852284550667,
+            -0.007419590838253498,
+            -0.017212767153978348,
+            0.010834822431206703,
+            -0.020730456337332726,
+            0.005938234273344278,
+            0.00783795677125454,
+            0.011039736680686474,
+            -0.0220111683011055,
+            -0.010868974961340427,
+            0.009818791411817074,
+            0.003835732117295265,
+            0.01902284100651741,
+            0.010886050760746002,
+            0.0022455148864537477,
+            -0.0035646480973809958,
+            0.0028495839796960354,
+            -1.0947752343781758e-05,
+            -0.0012508286163210869,
+            0.015308775939047337,
+            0.0006504949415102601,
+            0.02974666841328144,
+            -0.012055767700076103,
+            0.011201960034668446,
+            -0.009340658783912659,
+            0.0013842361513525248,
+            -0.009741948917508125,
+            -0.001973363570868969,
+            -0.004194331355392933,
+            -0.006484671495854855,
+            0.013148642145097256,
+            0.007714154664427042,
+            -0.004239156376570463,
+            -0.01416467409580946,
+            0.01194477267563343,
+            -0.023035738617181778,
+            -0.012269220314919949,
+            -0.00881983619183302,
+            0.012935190461575985,
+            -0.0053960662335157394,
+            0.01170570682734251,
+            0.03457922115921974,
+            -0.0018559650052338839,
+            0.0007860369514673948,
+            -0.013746308162808418,
+            -0.008311820216476917,
+            -0.02728770114481449,
+            -0.01510386262089014,
+            -0.008922292850911617,
+            -0.03454506769776344,
+            -0.011919158510863781,
+            0.008239246904850006,
+            0.008111175149679184,
+            0.0014237246941775084,
+            -0.019176525995135307,
+            -0.008888140320777893,
+            -0.01492456253618002,
+            -0.017460372298955917,
+            -0.00924673955887556,
+            -0.01175693515688181,
+            -0.0037119300104677677,
+            0.020884141325950623,
+            -0.004162313882261515,
+            0.014267130754888058,
+            0.016939548775553703,
+            1.3132299500284716e-05,
+            0.015334390103816986,
+            0.009665105491876602,
+            -0.009127207100391388,
+            -0.004725826904177666,
+            0.006885961163789034,
+            0.0027706066612154245,
+            0.018749622628092766,
+            -0.006702392362058163,
+            -0.014796491712331772,
+            0.004298923071473837,
+            0.005571096669882536,
+            -0.012346062809228897,
+            0.010655523277819157,
+            -0.00476424815133214,
+            0.02054261974990368,
+            -0.018254414200782776,
+            -0.011885006912052631,
+            0.014472044073045254,
+            -0.008862526156008244,
+            0.02394077368080616,
+            0.013951221480965614,
+            -0.015291700139641762,
+            0.007167717441916466,
+            0.013029108755290508,
+            0.007615966722369194,
+            0.0018495614640414715,
+            0.022147778421640396,
+            -0.0011846584966406226,
+            0.0031590894795954227,
+            0.010237157344818115,
+            -0.018835002556443214,
+            -0.02042308636009693,
+            -0.003118533408269286,
+            0.02247222512960434,
+            0.007961759343743324,
+            -0.016222350299358368,
+            0.0037930416874587536,
+            0.012849809601902962,
+            0.012072844430804253,
+            0.0039787450805306435,
+            -0.001204936415888369,
+            -0.02248930186033249,
+            0.0035155541263520718,
+            0.0001910395221784711,
+            0.018664240837097168,
+            -0.011116579174995422,
+            0.001461078878492117,
+            -0.011389797553420067,
+            0.012747352942824364,
+            -0.005221035331487656,
+            0.01193623524159193,
+            0.0011761204805225134,
+            -0.005353375803679228,
+            0.0001780989987310022,
+            0.014941639266908169,
+            0.014070754870772362,
+            0.037738312035799026,
+            -0.005916888825595379,
+            -0.006480402313172817,
+            0.017170077189803123,
+            -0.012388752773404121,
+            0.018715469166636467,
+            0.006497478578239679,
+            0.0170249305665493,
+            -0.011090965010225773,
+            0.0011344973463565111,
+            0.04115354269742966,
+            0.015949131920933723,
+            -6.583659705938771e-05,
+            -0.008760069496929646,
+            -0.00805140845477581,
+            0.0244530588388443,
+            0.009067440405488014,
+            -2.7882166250492446e-05,
+            0.015744218602776527,
+            0.006096188444644213,
+            -0.016324806958436966,
+            -0.012918113730847836,
+            0.03555256128311157,
+            -0.007718423847109079,
+            -0.022045321762561798,
+            -0.0009311842732131481,
+            -0.009383349679410458,
+            -0.02884163148701191,
+            0.013959759846329689,
+            0.013788998126983643,
+            -0.008162403479218483,
+            0.006992687005549669,
+            0.008439891040325165,
+            -0.004117488861083984,
+            -0.004435532260686159,
+            0.0005346972029656172,
+            -0.0007049251580610871,
+            0.006177300587296486,
+            0.005092964507639408,
+            -0.014463506639003754,
+            -0.006787773221731186,
+            -0.005669284611940384,
+            -0.004064125940203667,
+            0.009016212075948715,
+            0.005058811977505684,
+            0.004870974458754063,
+            -0.008482581935822964,
+            0.018339794129133224,
+            0.02298451028764248,
+            -0.0122009152546525,
+            0.012115534394979477,
+            0.005233842879533768,
+            -0.0146428057923913,
+            -0.004614831879734993,
+            -0.01121903583407402,
+            -0.026570502668619156,
+            -0.006113264709711075,
+            0.0033490615896880627,
+            0.015428309328854084,
+            -0.022796671837568283,
+            0.0008500725380145013,
+            0.007790997624397278,
+            2.7365211280994117e-05,
+            -0.008879602886736393,
+            0.007573276292532682,
+            -0.00044157879892736673,
+            0.022301463410258293,
+            0.019808344542980194,
+            0.025631314143538475,
+            -0.008414276875555515,
+            -0.00499904528260231,
+            0.014301283285021782,
+            0.00433947890996933,
+            0.009374811314046383,
+            0.027543844655156136,
+            0.022164853289723396,
+            -0.013857303187251091,
+            0.006429173983633518,
+            0.000640355923678726,
+            -0.014250054024159908,
+            0.0066682398319244385,
+            0.013054722920060158,
+            0.012414366938173771,
+            0.0028410458471626043,
+            -0.0013415457215160131,
+            -0.015957670286297798,
+            0.004824014846235514,
+            0.018408099189400673,
+            -0.017118848860263824,
+            -0.03471583127975464,
+            -0.011842316016554832,
+            -0.012158225290477276,
+            -0.00624133599922061,
+            -0.012875423766672611,
+            0.01440373994410038,
+            0.004418455995619297,
+            0.019705887883901596,
+            0.0002309283590875566,
+            -0.01022861897945404,
+            0.014250054024159908,
+            0.005110040307044983,
+            -0.0016638581873849034,
+            0.0005352308508008718,
+            0.007231753319501877,
+            -0.0030331527814269066,
+            -0.0022476494777947664,
+            -0.00027055040118284523,
+            0.016239427030086517,
+            -0.02532394416630268,
+            0.008947907015681267,
+            -0.017067620530724525,
+            0.0017951311310753226,
+            0.015249010175466537,
+            -0.003775965655222535,
+            -0.016025975346565247,
+            0.0011216901475563645,
+            -0.008619191125035286,
+            -0.07288104295730591,
+            -0.005882736761122942,
+            0.004512374754995108,
+            0.023992003872990608,
+            -0.013336479663848877,
+            -0.0024824466090649366,
+            0.0018474268727004528,
+            0.008657612837851048,
+            -0.010134699754416943,
+            0.003995821345597506,
+            0.010390842333436012,
+            -0.003319178242236376,
+            -0.009221125394105911,
+            0.002508060773834586,
+            -0.008106905966997147,
+            -0.0038805571384727955,
+            -0.00783795677125454,
+            0.003402424743399024,
+            0.025221485644578934,
+            -0.010868974961340427,
+            0.024657974019646645,
+            0.009699258022010326,
+            -0.0075433929450809956,
+            -0.0027428579051047564,
+            -0.007761114276945591,
+            -0.027543844655156136,
+            -0.012397291138768196,
+            -0.003396020969375968,
+            -0.01292665209621191,
+            0.0029264267068356276,
+            0.001692674239166081,
+            -0.007167717441916466,
+            -0.004348016809672117,
+            0.0021547977812588215,
+            0.00030336863710545003,
+            -0.022147778421640396,
+            -0.0009407896432094276,
+            -0.014267130754888058,
+            0.02105490304529667,
+            -0.012371676973998547,
+            -0.004461146425455809,
+            0.0030544979963451624,
+            -0.010109085589647293,
+            0.004580679349601269,
+            0.019313136115670204,
+            -0.007308595813810825,
+            -0.009682182222604752,
+            0.014045140706002712,
+            -0.003690584795549512,
+            0.00185916677583009,
+            -0.01798119582235813,
+            0.010032243095338345,
+            -0.03232516720890999,
+            -0.003583858720958233,
+            -0.009981014765799046,
+            -0.0031847036443650723,
+            -0.0067151994444429874,
+            -0.03162504732608795,
+            0.015138015151023865,
+            -0.007039646618068218,
+            -0.02346264198422432,
+            -0.0011793222511187196,
+            -0.01859593763947487,
+            0.01419028826057911,
+            -0.01487333420664072,
+            0.01393414568156004,
+            -0.020645076408982277,
+            0.012533900327980518,
+            0.012798581272363663,
+            0.021123208105564117,
+            0.006493209395557642,
+            -0.005302147474139929,
+            0.002676687901839614,
+            0.0018730411538854241,
+            0.014352511614561081,
+            -0.008905217051506042,
+            0.00462336977943778,
+            -0.002390662208199501,
+            0.005737589206546545,
+            0.009426039643585682,
+            0.02107197977602482,
+            -0.0005896611255593598,
+            -0.0037823691964149475,
+            -0.017468910664319992,
+            0.00757754547521472,
+            -0.01559907104820013,
+            0.005485715810209513,
+            0.0025251370389014482,
+            0.023787088692188263,
+            0.010886050760746002,
+            0.0035859933122992516,
+            0.004900857340544462,
+            0.004520913120359182,
+            0.01861301250755787,
+            -0.006890229880809784,
+            -0.004747172351926565,
+            -0.018186109140515327,
+            0.019688811153173447,
+            0.007705616764724255,
+            -0.005306416191160679,
+            -0.011902082711458206,
+            0.006010808050632477,
+            0.002134519862011075,
+            0.0017577770631760359,
+            0.02288205362856388,
+            -0.02974666841328144,
+            -0.008964983746409416,
+            -0.007808073423802853,
+            -0.009682182222604752,
+            -0.017844585701823235,
+            0.017084697261452675,
+            -0.008012987673282623,
+            0.00461910106241703,
+            -0.012790042906999588,
+            0.033418044447898865,
+            0.002969117136672139,
+            0.00513992365449667,
+            0.013729231432080269,
+            -0.00024373548512812704,
+            0.017844585701823235,
+            -0.008367317728698254,
+            -0.04036803916096687,
+            -0.0031911071855574846,
+            0.0034515184815973043,
+            0.007645850069820881,
+            -0.010903127491474152,
+            0.004862436093389988,
+            -0.006339523941278458,
+            -0.01854470930993557,
+            -0.014232978224754333,
+            0.037840768694877625,
+            -0.005225304514169693,
+            0.017947042360901833,
+            -0.016145508736371994,
+            -0.0033682722132653,
+            0.007107951212674379,
+            0.022762520238757133,
+            -0.012602205388247967,
+            0.00791053008288145,
+            0.009571187198162079,
+            0.02199409157037735,
+            0.00978463888168335,
+            -0.01565883681178093,
+            0.023735860362648964,
+            0.005741858389228582,
+            -0.0023565099108964205,
+            0.005276532843708992,
+            -0.0030352871399372816,
+            0.016000360250473022,
+            -0.014813567511737347,
+            -0.011876468546688557,
+            0.002303146990016103,
+            -0.010168852284550667,
+            0.0037055264692753553,
+            -0.024589668959379196,
+            0.00037540867924690247,
+            -0.009844405576586723,
+            0.021089056506752968,
+            0.01027130987495184,
+            0.00877714529633522,
+            -0.019637582823634148,
+            -0.0048026698641479015,
+            -0.02645096927881241,
+            -0.014574501663446426,
+            -0.01022861897945404,
+            -0.0011451698374003172,
+            -0.004431263078004122,
+            -0.007765382993966341,
+            -0.03133475035429001,
+            -0.004202869720757008,
+            0.013541393913328648,
+            0.014975790865719318,
+            -0.0013714289525523782,
+            0.012089920230209827,
+            -0.00019544197130016983,
+            -0.008106905966997147,
+            -0.02236976847052574,
+            -0.010407919064164162,
+            -0.02091829478740692,
+            -0.03123229369521141,
+            -0.022199006751179695,
+            0.008670419454574585,
+            -0.012021616101264954,
+            -0.002941368380561471,
+            0.028961164876818657,
+            -0.022130701690912247,
+            0.007513510063290596,
+            0.005400335416197777,
+            0.0028666602447628975,
+            0.014856258407235146,
+            0.0054472945630550385,
+            0.0003634020104072988,
+            0.0037652929313480854,
+            -0.008341703563928604,
+            -0.008081291802227497,
+            0.01799827069044113,
+            -0.003331985557451844,
+            0.005097233224660158,
+            0.010809208266437054,
+            -0.006749351508915424,
+            -0.005438756663352251,
+            0.02392369881272316,
+            -0.015718603506684303,
+            0.002044870052486658,
+            -0.008610652759671211,
+            -0.014079293236136436,
+            -0.00949434470385313,
+            -0.005793086718767881,
+            -0.01813488081097603,
+            -0.015240471810102463,
+            0.01689685881137848,
+            -0.0004028906114399433,
+            0.016025975346565247,
+            -0.005289340391755104,
+            -0.005007583647966385,
+            0.014437892474234104,
+            -0.03558671474456787,
+            -0.003402424743399024,
+            0.006377945188432932,
+            -0.026211904361844063,
+            0.02484581060707569,
+            0.005276532843708992,
+            0.01605158858001232,
+            -0.013780459761619568,
+            0.0026318628806620836,
+            -0.006365138106048107,
+            0.004358689300715923,
+            0.018903307616710663,
+            0.007308595813810825,
+            0.004858167376369238,
+            -0.006689585279673338,
+            -0.008047140203416348,
+            0.016077203676104546,
+            -0.007415322121232748,
+            0.0018719738582149148,
+            0.016785863786935806,
+            0.018425175920128822,
+            0.012004539370536804,
+            0.0059553105384111404,
+            0.013071799650788307,
+            0.01750306226313114,
+            -0.007043915335088968,
+            -0.005835777148604393,
+            -0.013481627218425274,
+            -0.00802579429000616,
+            -0.0013105951948091388,
+            0.03859211876988411,
+            0.018920384347438812,
+            0.007385438773781061,
+            -0.02054261974990368,
+            0.013456013053655624,
+            0.0043928418308496475,
+            6.841302820248529e-06,
+            -0.0027385889552533627,
+            0.02293328195810318,
+            0.0024952536914497614,
+            0.007125027012079954,
+            0.006258412264287472,
+            -0.017212767153978348,
+            0.03309359401464462,
+            -0.020713381469249725,
+            0.0008532743086107075,
+            -0.013319403864443302,
+            -0.012166762724518776,
+            0.015530765987932682,
+            0.002104636514559388,
+            -0.004164448007941246,
+            -0.006296833511441946,
+            -0.0033575997222214937,
+            0.013515779748558998,
+            0.008499657735228539,
+            -0.012875423766672611,
+            0.004516643937677145,
+            -0.02542640082538128,
+            0.004315999336540699,
+            -0.015334390103816986,
+            0.021755026653409004,
+            -0.004559334367513657,
+            -0.004674598574638367,
+            -0.019261907786130905,
+            -0.017067620530724525,
+            -0.00999809056520462,
+            0.002403469290584326,
+            -0.004755710251629353,
+            0.004422725178301334,
+            -0.009468729607760906,
+            -0.008499657735228539,
+            0.0027599341701716185,
+            0.0036628360394388437,
+            0.012465596199035645,
+            0.013669464737176895,
+            0.019893724471330643,
+            0.017759205773472786,
+            -0.01536000519990921,
+            -0.0341864712536335,
+            0.0006056699785403907,
+            0.005084426142275333,
+            -0.0036436254158616066,
+            0.018664240837097168,
+            -0.001938143977895379,
+            -0.0009968207450583577,
+            -0.0007150641758926213,
+            0.0005419012159109116,
+            0.02004740945994854,
+            -0.011014122515916824,
+            -0.016657792031764984,
+            0.00021545309573411942,
+            0.008853988721966743,
+            0.00502039073035121,
+            -0.0025934416335076094,
+            -0.010296924039721489,
+            -0.0009455922991037369,
+            0.0020341973286122084,
+            0.006032153032720089,
+            -0.009101592935621738,
+            -0.0010496501345187426,
+            -0.0013671600027009845,
+            -0.0030544979963451624,
+            -0.006254143081605434,
+            -0.009844405576586723,
+            0.008222170174121857,
+            -0.012602205388247967,
+            -0.01123611256480217,
+            0.016222350299358368,
+            -0.015086785890161991,
+            0.001136631821282208,
+            0.023821242153644562,
+            0.022557605057954788,
+            -0.0008842248353175819,
+            0.006774966139346361,
+            -0.007936145178973675,
+            -0.011517869308590889,
+            -0.02785121463239193,
+            -0.019637582823634148,
+            -0.00806848518550396,
+            0.017947042360901833,
+            -0.002053408185020089,
+            -0.004049184266477823,
+            -0.0018869155319407582,
+            -0.008580769412219524,
+            0.008444160223007202,
+            -0.013814612291753292,
+            0.011620325967669487,
+            -0.034425534307956696,
+            0.013088875450193882,
+            -0.006702392362058163,
+            0.0010485828388482332,
+            0.015009943395853043,
+            0.04351005330681801,
+            -0.007351286243647337,
+            0.0029541754629462957,
+            0.001540056080557406,
+            0.01389145478606224,
+            0.028209814801812172,
+            0.004369362257421017,
+            0.0006136744632385671,
+            -0.01765674725174904,
+            0.004091874696314335,
+            0.002264725510030985,
+            -0.013456013053655624,
+            0.009579724632203579,
+            -0.023035738617181778,
+            -0.023155272006988525,
+            -0.009067440405488014,
+            -0.021123208105564117,
+            -0.002608383307233453,
+            0.002011785050854087,
+            0.012883962132036686,
+            -0.009878558106720448,
+            -0.012098458595573902,
+            -0.0005656477296724916,
+            -0.032991137355566025,
+            0.016709022223949432,
+            0.09665105491876602,
+            0.00875153113156557,
+            -0.003908305894583464,
+            -0.0021515958942472935,
+            0.002499522641301155,
+            0.009562648832798004,
+            -0.004768517334014177,
+            0.055360905826091766,
+            -0.018852079287171364,
+            0.008311820216476917,
+            -0.0008105839369818568,
+            0.008064216002821922,
+            -0.007223214954137802,
+            0.015530765987932682,
+            -0.012209453620016575,
+            -0.005951041355729103,
+            -0.009460192173719406,
+            -0.01123611256480217,
+            -0.0017246920615434647,
+            -0.016640717163681984,
+            -0.004986238200217485,
+            -0.020645076408982277,
+            0.013336479663848877,
+            -0.03234224393963814,
+            -0.011509330943226814,
+            -0.012474133633077145,
+            -0.0063224476762115955,
+            0.006066305562853813,
+            -0.015684451907873154,
+            -0.002595576224848628,
+            -0.002099300269037485,
+            0.015402695164084435,
+            0.0013863706262782216,
+            -0.00232235761359334,
+            -0.008328896015882492,
+            0.0005405671545304358,
+            0.005024659913033247,
+            0.003940323833376169,
+            0.006557244807481766,
+            0.006792041938751936,
+            -0.005848584230989218,
+            -0.008811297826468945,
+            0.010655523277819157,
+            -0.002510195365175605,
+            -0.0027919518761336803,
+            -0.008871064521372318,
+            0.010382304899394512,
+            -0.013481627218425274,
+            0.018356870859861374,
+            -0.013840226456522942,
+            -0.029507601633667946,
+            -0.015547842718660831,
+            0.02100367471575737,
+            -0.01996202953159809,
+            -0.0039381892420351505,
+            0.010553065687417984,
+            0.02829519473016262,
+            0.004113219678401947,
+            -0.001529383473098278,
+            -0.004909395705908537,
+            -0.017204230651259422,
+            0.009511420503258705,
+            0.008196556009352207,
+            0.018442250788211823,
+            -0.012371676973998547,
+            0.016487032175064087,
+            -0.018288565799593925,
+            0.006177300587296486,
+            -0.01663217879831791,
+            0.0006083381595090032,
+            0.00011866596469189972,
+            -0.0041793896816670895,
+            0.01608574204146862,
+            -0.01534292846918106,
+            -0.026263132691383362,
+            0.01900576427578926,
+            -0.013635313138365746,
+            -0.03307652100920677,
+            0.003374675754457712,
+            -0.034937821328639984,
+            -0.0025827689096331596,
+            0.009716334752738476,
+            -0.02149888314306736,
+            -0.018305642530322075,
+            0.007487895432859659,
+            -0.010066395625472069,
+            -0.008871064521372318,
+            0.00502039073035121,
+            -0.01689685881137848,
+            -0.012149686925113201,
+            -0.001152640674263239,
+            0.006753620691597462,
+            -0.00499477656558156,
+            0.0057632033713161945,
+            -0.026075294241309166,
+            0.014010988175868988,
+            -0.0170249305665493,
+            0.01144102681428194,
+            -0.010587218217551708,
+            0.022762520238757133,
+            0.011748396791517735,
+            -0.010356690734624863,
+            -0.0008303282083943486,
+            0.006036422215402126,
+            0.0005264259525574744,
+            0.026126522570848465,
+            0.03237639740109444,
+            -0.0011889275629073381,
+            0.008469774387776852,
+            0.007586083374917507,
+            0.003921112976968288,
+            -0.011014122515916824,
+            0.0015923518221825361,
+            0.014830644242465496,
+            0.01992787793278694,
+            -0.013729231432080269,
+            0.008879602886736393,
+            -0.026672959327697754,
+            0.02103782817721367,
+            -0.022096550092101097,
+            -0.000982412719167769,
+            0.010066395625472069,
+            -0.004777055233716965,
+            0.007090874947607517,
+            0.0066682398319244385,
+            0.021362274885177612,
+            0.03375956416130066,
+            0.00807702261954546,
+            0.008879602886736393,
+            0.007257367484271526,
+            -0.004798400681465864,
+            0.002044870052486658,
+            0.009135744534432888,
+            -0.005869929678738117,
+            -0.0040940092876553535,
+            -0.010083471424877644,
+            -0.006685316096991301,
+            0.016205275431275368,
+            0.0008079157560132444,
+            0.012602205388247967,
+            0.007018301170319319,
+            -0.007428129203617573,
+            -0.005669284611940384,
+            0.007014031987637281,
+            -0.023787088692188263,
+            -0.00950288213789463,
+            0.025221485644578934,
+            -0.013814612291753292,
+            -0.043202683329582214,
+            -0.0024333526380360126,
+            0.008853988721966743,
+            -0.017964119091629982,
+            -0.01245705783367157,
+            -0.004772786516696215,
+            0.010117623955011368,
+            0.0007241358398459852,
+            -0.007569007575511932,
+            0.028175661340355873,
+            0.00502039073035121,
+            -0.005169807001948357,
+            0.003571051638573408,
+            -2.8232359909452498e-05,
+            0.019244831055402756,
+            0.01536000519990921,
+            0.008294744417071342,
+            -0.03266669064760208,
+            -0.007304327096790075,
+            -0.005780279636383057,
+            0.013114489614963531,
+            0.002813297091051936,
+            0.003423769725486636,
+            -0.019740039482712746,
+            -0.020798761397600174,
+            0.019210679456591606,
+            0.004935009870678186,
+            0.019825419411063194,
+            -0.00867895781993866,
+            -0.006548706907778978,
+            -0.004841091111302376,
+            -0.004742903169244528,
+            0.00499477656558156,
+            0.0076415808871388435,
+            0.0038933642208576202,
+            0.03377664089202881,
+            -0.001280711847357452,
+            0.004042780492454767,
+            -0.0036350872833281755,
+            -0.0002441357064526528,
+            0.014787953346967697,
+            -0.00790199264883995,
+            0.008905217051506042,
+            0.018800850957632065,
+            -0.02887578494846821,
+            0.011398335918784142,
+            -0.01679440215229988,
+            -0.00828193686902523,
+            0.016000360250473022,
+            -0.0005320290802046657,
+            -0.004922202788293362,
+            -0.014583039097487926,
+            0.0008127184119075537,
+            0.007752575911581516,
+            0.01996202953159809,
+            -0.004091874696314335,
+            0.011287340894341469,
+            -0.009648029692471027,
+            -0.004305326379835606,
+            0.010083471424877644,
+            0.006087650544941425,
+            -0.0033213128335773945,
+            0.006232798099517822
+          ],
+          "keyphrases": [
+            "MoE models",
+            "expert choice",
+            "training efficiency",
+            "fine-tuning on GLUE and SuperGLUE",
+            "capacity factor"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": true,
+      "properties": {
+        "cosine_similarity": 0.8020664672125095
+      }
+    },
+    {
+      "id": "8a0f9319-7c71-4296-8697-1bf9875e99b2",
+      "type": "cosine_similarity",
+      "source": {
+        "id": "1e687808-8db8-4320-805d-0282a8d679bc",
+        "properties": {
+          "page_content": "Deepseekmoe: Towards Ultimate Expert Specialization In\n\nMixture-Of-Experts Language Models\n\nDamai Dai∗1,2, Chengqi Deng1, Chenggang Zhao∗1,3, R.X. Xu1, Huazuo Gao1, Deli Chen1, Jiashi Li1, Wangding Zeng1, Xingkai Yu∗1,4, Y. Wu1, Zhenda Xie1, Y.K. Li1, Panpan Huang1, Fuli Luo1, Chong Ruan1, Zhifang Sui2, Wenfeng Liang1 1DeepSeek-AI 2National Key Laboratory for Multimedia Information Processing, Peking University 3Institute for Interdisciplinary Information Sciences, Tsinghua University 4National Key Laboratory for Novel Software Technology, Nanjing University {daidamai, szf}@pku.edu.cn, {wenfeng.liang}@deepseek.com https://github.com/deepseek-ai/DeepSeek-MoE\n\nAbstract\n\nIn the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top- out of experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into ones and activating from them, allowing for a more flexible combination of activated experts; (2) isolating experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5× expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations.\n\nFurther, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\n1. Introduction\n\nRecent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023a). It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017) has emerged as a popular solution. It can\n\nenable parameter scaling, while concurrently keeping computational costs at a modest level.\n\nRecent applications of MoE architectures in Transformers (Vaswani et al., 2017) have yielded successful attempts at scaling language models to a substantial size (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022), accompanied with remarkable performance. These achievements underscore the considerable potential and promise of MoE language models.\n\nDespite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts.\n\nThis architecture manifests two potential issues: (1) Knowledge Hybridity: existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. (2) Knowledge Redundancy: tokens assigned to different experts may require common knowledge. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters.\n\nThese issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models.\n\nIn response to the aforementioned issues, we introduce DeepSeekMoE, an innovative MoE architecture specifically designed towards ultimate expert specialization. Our architecture involves two principal strategies: (1) Fine-Grained Expert Segmentation: while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. (2) Shared Expert Isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.\n\nThrough compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized.\n\nStarting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B (Lepikhin et al., 2021) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5× expert parameters and computation. Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE.\n\nThese studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization.\n\nLeveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B (DeepSeek-AI, 2024), a dense model trained on the same 2T corpus. We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B (Touvron et al., 2023b), which has approximately 2.5 times the activated parameters. Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard1. Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. The experimental results still validate its substantial advantages over the GShard architecture consistently. In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nOur contributions are summarized as follows: - Architectural Innovation. We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation.\n\nEmpirical Validation. We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. Experimental results validate the high 1https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models\n\nScalability. We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B.\n\nAlignment for MoE. We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B.\n\nPublic Release. In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization.\n\n2. Preliminaries: Mixture-Of-Experts For Transformers\n\nWe first introduce a generic MoE architecture commonly used in Transformer language models. A standard Transformer language model is constructed by stacking layers of standard Transformer blocks, where each block can be represented as follows:\n\n$$\\mathbf{u}{1:T}^{l}=\\text{Self-Att}\\left(\\mathbf{h}{1:T}^{l-1}\\right)+\\mathbf{h}{1:T}^{l-1},\\tag{1}$$ $$\\mathbf{h}{t}^{l}=\\text{FFN}\\left(\\mathbf{u}{t}^{l}\\right)+\\mathbf{u}{t}^{l},\\tag{2}$$\n\nwhere denotes the sequence length, Self-Att(·) denotes the self-attention module, FFN(·) denotes the Feed-Forward Network (FFN), u 1: ∈ R× are the hidden states of all tokens after the -th attention module, and h\n\n∈ Ris the output hidden state of the -th token after the -th Transformer block. For brevity, we omit the layer normalization in the above formulations.\n\nA typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022). An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. Then, each token will be assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts. If the -th FFN is substituted with an MoE layer, the computation for its output hidden state h\n\nis expressed as: where denotes the total number of experts, FFN(·) is the -th expert FFN, , denotes the gate value for the -th expert, , denotes the token-to-expert affinity, Topk(·, ) denotes the set comprising highest affinity scores among those calculated for the -th token and all experts, and e\n\nis the centroid of the -th expert in the -th layer. Note that ,is sparse, indicating that only out of gate values are nonzero. This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only experts. Also, in the above formulations, we omit the layer normalization operation for brevity.\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{N}\\left(g_{i,t}\\operatorname{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\operatorname{Topk}({s_{j,t}|1\\leqslant j\\leqslant N},K),\\ 0,&\\text{otherwise},\\end{cases}$$ $$s_{i,t}=\\operatorname{Softmax}{i}\\left(\\mathbf{u}{t}^{l}\\,\\mathbf{e}_{i}^{l}\\right),$$\n\n$$(3)$$\n\n(4) $\\binom{4}{5}$ .\n\n3. Deepseekmoe Architecture\n\nOn top of the generic MoE architecture outlined in Section 2, we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. As illustrated in Figure 2, our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. Both of these strategies are designed to elevate the level of expert specialization.\n\n3.1. Fine-Grained Expert Segmentation\n\nIn scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts.\n\nIn pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. The finer expert segmentation enables a more flexible and adaptable combination of activated experts. To be specific, on top of a typical MoE architecture shown in Figure 2(a), we segment each expert FFN into smaller experts by reducing the FFN intermediate hidden dimension to 1 times its original size. Since each expert becomes smaller, in response, we also increase the number of activated experts to times to keep the same computation cost, as illustrated in Figure 2(b). With the fine-grained expert segmentation, the output of an MoE layer can be expressed as:\n\n$$\\mathbf{h}{t}^{i}=\\sum{i=1}^{\\text{mN}}\\left(g_{i,t}\\text{FFN}{i}\\left(\\mathbf{u}{t}^{i}\\right)\\right)+\\mathbf{u}{t}^{i},\\tag{6}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\text{Tokr}(s_{i,t}|1\\leqslant j\\leqslant m\\text{N}),m\\text{K}),\\ 0,&\\text{otherwise},\\end{cases}$$ (7) $$s_{i,t}=\\text{Softmax}{i}\\left(\\mathbf{u}{t}^{T}\\mathbf{e}_{t}^{i}\\right),\\tag{8}$$ where the total number of expert parameters is equal to $N$ times the number of parameters in a\n\n$$(6)$$\n\n(7) (8) $\\frac{1}{2}$\n\nstandard FFN, and denotes the total number of fine-grained experts. With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to .\n\nFrom a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. As an illustrative example, we consider the case where = 16. A typical top-2 routing strategy can yield 16 2 = 120 possible combinations. By contrast, if each expert is split into 4 smaller experts, the fine-grained routing strategy can yield 64 8 = 4, 426, 165, 368 potential combinations. The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition.\n\n3.2. Shared Expert Isolation\n\nWith a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. This alleviation of redundancy will contribute to a more parameterefficient model with more specialized experts.\n\nTowards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate experts to serve as shared experts. Regardless of the router module, each token will be deterministically assigned to these shared experts. In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by , as depicted in Figure 2(c). With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows:\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{K_{s}}\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)+\\sum_{i=K_{s}+1}^{mN}\\left(g_{i,t}\\;\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},\\tag{9}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\mathrm{Topk}{s_{j,t}|K_{s}+1\\leqslant j\\leqslant mN},mK-K_{s}},\\ 0,&\\text{otherwise},\\end{cases}$$ (10) $$s_{i,t}=\\mathrm{Softmax}{i}\\left(\\mathbf{u}{t}^{l,T}e_{i}^{l}\\right).\\tag{11}$$ $\\mathbf{h}\\mathbf{M}\\mathbf{E}$ the number of closed-compact is $K_{s}$ the total number of routed currents.\n\nFinally, in DeepSeekMoE, the number of shared expert is , the total number of routed experts is − , and the number of nonzero gates is − .\n\nIt is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022). The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint.\n\n3.3. Load Balance Consideration\n\nAutomatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. Firstly, there is a risk of routing collapse (Shazeer et al., 2017), i.e., the model always selects only a few experts, preventing other experts from sufficient training.\n\nSecondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.\n\nExpert-Level Balance Loss. In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. The computation of the balance loss is as follows:\n\n$$\\begin{array}{l}{{{\\mathcal L}{\\mathrm{ExpBal}}=\\alpha{1}\\sum_{i=1}^{N^{\\prime}}f_{i}P_{i},}}\\ {{f_{i}=\\frac{N^{\\prime}}{K^{T}}\\sum_{t=1}^{T}\\mathds{1}\\,(\\mathrm{Token}\\;t\\;\\mathrm{s}\\mathrm{lects}\\;\\mathrm{Expert}\\;i),}}\\ {{P_{i}=\\frac{1}{T}\\sum_{t=1}^{T}s_{i,t},}}\\end{array}$$\n\n(12) $$\\begin{array}{l}\\small\\text{(13)}\\end{array}$$ = (14) $$\\begin{array}{l}\\small\\text{(14)}\\end{array}$$ .\n\n7 where 1 is a hyper-parameter called expert-level balance factor, ′is equal to ( − ) and ′ is equal to ( − ) for brevity. 1(·) denotes the indicator function.\n\nDevice-Level Balance Loss. In addition to the expert-level balance loss, we introduce a devicelevel balance loss. When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. Instead, our primary objective is to ensure balanced computation across the devices. If we partition all routed experts into groups {E1, E2, ..., E}, and deploy each group on a single device, the device-level balance loss is computed as follows:\n\n$$\\mathcal{L}{\\text{DevBal}}=\\alpha{2}\\sum_{i=1}^{D}f_{i}^{\\prime}P_{i}^{\\prime},$$ $$f_{i}^{\\prime}=\\frac{1}{|\\mathcal{E}{i}|}\\sum{j\\in\\mathcal{E}{i}}f{j},$$ $$P_{i}^{\\prime}=\\sum_{j\\in\\mathcal{E}{i}}P{j},$$\n\n(15) $$\\begin{array}{l}\\small\\text{(16)}\\end{array}$$ = (17) .\n\nwhere 2 is a hyper-parameter called device-level balance factor. In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices.\n\n4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n6. Alignment For Deepseekmoe 16B\n\nPrevious research indicates that MoE models typically do not emerge significant gains from fine-tuning (Artetxe et al., 2022; Fedus et al., 2021). However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B.\n\n6.1. Experimental Setup\n\nTraining Data. For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios.\n\nHyper-Parameters. During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).\n\nWe employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10−5 without incorporating any learning rate scheduling strategy.\n\nEvaluation Benchmarks. For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3, with the following adjustments: (1) We exclude Pile (Gao et al., 2020) since chat models are seldom employed for pure language modeling. (2) We exclude\n\nMetric # Shot LLaMA2 DeepSeek DeepSeekMoE SFT 7B Chat 7B Chat 16B # Total Params N/A 6.7B 6.9B 16.4B # Activated Params N/A 6.7B 6.9B 2.8B FLOPs per 4K Tokens N/A 187.9T 183.5T 74.4T HellaSwag (Acc.) 0-shot 67.9 71.0 72.2 PIQA (Acc.) 0-shot 76.9 78.4 79.7 ARC-easy (Acc.) 0-shot 69.7 70.2 69.9 ARC-challenge (Acc.) 0-shot 50.8 50.2 50.0 BBH (EM) 3-shot 39.3 43.1 42.2 RACE-middle (Acc.) 5-shot 63.9 66.1 64.8 RACE-high (Acc.) 5-shot 49.6 50.8 50.6 DROP (EM) 1-shot 40.0 41.7 33.8 GSM8K (EM) 0-shot 63.4 62.6 62.2 MATH (EM) 4-shot 13.5 14.7 15.2 HumanEval (Pass@1) 0-shot 35.4 45.1 45.7 MBPP (Pass@1) 3-shot 27.8 39.0 46.2 TriviaQA (EM) 5-shot 60.1 59.5 63.3 NaturalQuestions (EM) 0-shot 35.2 32.7 35.1 MMLU (Acc.) 0-shot 50.0 49.7 47.2 WinoGrande (Acc.) 0-shot 65.1 68.4 69.0 CLUEWSC (EM) 5-shot 48.4 66.2 68.2 CEval (Acc.) 0-shot 35.1 44.7 40.0 CMMLU (Acc.) 0-shot 36.9 51.2 49.3\n\nCHID (Zheng et al., 2019) due to the observed instability of results, hindering the derivation of solid conclusions. (3) We additionally include BBH (Suzgun et al., 2022) to provide a more comprehensive assessment of the reasoning ability of the chat models.\n\nTable 5 | Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations.\n\n6.2. Evaluations\n\nBaselines. In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. Correspondingly, we construct three chat models, including LLaMA2 SFT 7B3, DeepSeek Chat 7B, and DeepSeekMoE Chat 16B.\n\nSubsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks.\n\nResults. The evaluation results are presented in Table 5. Our key observations include: (1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intensive tasks (TriviaQA, NaturalQuestions). (2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. In addition, it also surpasses DeepSeek Chat 7B. (3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1). However, it is worth noting that, after supervised finetuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. (4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations.\n\n7. Deepseekmoe 145B Ongoing\n\nEncouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available.\n\n7.1. Experimental Setup\n\nTraining Data and Tokenization. For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study.\n\nModel Settings. For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps.\n\nSubsequently, the learning rate keeps constant during the remaining training process. The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. Also, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse.\n\nEvaluation Benchmarks. We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3).\n\n7.2. Evaluations\n\nBaselines. Apart from DeepSeekMoE 145B, we consider three additional models for comparison. DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters.\n\nResults. From the evaluation results presented in Table 6, we have the following observations: (1) Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. (2) On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. (3) At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5.\n\n8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\nAppendices A. Overview Of Hyper-Parameters\n\nWe present the overview of hyper-parameters for DeepSeekMoE across various sizes in Table 7.\n\n# Params # Layers Hidden # Attn # Shared # Routed Relative Sequence Batch Size Learning Size Heads Experts Experts Expert Size Length (Sequence) Rate 2.0B 9 1280 10 1 63 (7 activated) 0.25 2048 2048 1.08e-3 16.4B 28 2048 16 2 64 (6 activated) 0.25 4096 4608 4.2e-4 144.6B 62 4096 32 4 128 (12 activated) 0.125 4096 4608 3.0e-4\n\nTable 7 | Overview of hyper-parameters for DeepSeekMoE across various sizes. The relative expert size is in comparison to a standard FFN.\n\nB. Comparing Deepseekmoe With Larger Models\n\nComparisons among DeepSeekMoE, GShard×1.2, and GShard×1.5 are shown in Table 8. Comparisons among DeepSeekMoE, Dense×4, and Dense×16 are shown in Table 9.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 2.3B 2.8B 1.9B # Activated Expert Params N/A 0.28B 0.35B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.824 1.808 1.808 HellaSwag (Acc.) 0-shot 53.7 54.4 54.8 PIQA (Acc.) 0-shot 71.8 71.1 72.3 ARC-easy (Acc.) 0-shot 46.8 47.3 49.4 ARC-challenge (Acc.) 0-shot 31.7 34.1 34.3 RACE-middle (Acc.) 5-shot 43.7 46.4 44.0 RACE-high (Acc.) 5-shot 31.9 32.4 31.7 HumanEval (Pass@1) 0-shot 3.7 3.0 4.9 MBPP (Pass@1) 3-shot 2.4 2.6 2.2 TriviaQA (EM) 5-shot 15.2 15.7 16.6 NaturalQuestions (EM) 5-shot 4.5 4.7 5.7\n\nTable 8 | Comparison between DeepSeekMoE and larger GShard models.\n\nAt a larger scale of 13B total parameters, we also compare DeepSeekMoE with GShard×1.2 and GShard×1.5, and show results in Table 10. At a larger scale, DeepSeekMoE even outperforms GShard×1.5 distinctly.\n\nMetric # Shot Dense×4 Dense×16 DeepSeekMoE Relative Expert Size N/A 1 1 0.25 # Experts N/A 4 + 0 16 + 0 1 + 63 # Activated Experts N/A 4 + 0 16 + 0 1 + 7 # Total Expert Params N/A 0.47B 1.89B 1.89B # Activated Expert Params N/A 0.47B 1.89B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.908 1.806 1.808 HellaSwag (Acc.) 0-shot 47.6 55.1 54.8 PIQA (Acc.) 0-shot 70.0 71.9 72.3 ARC-easy (Acc.) 0-shot 43.9 51.9 49.4 ARC-challenge (Acc.) 0-shot 30.5 33.8 34.3 RACE-middle (Acc.) 5-shot 42.4 46.3 44.0 RACE-high (Acc.) 5-shot 30.7 33.0 31.7 HumanEval (Pass@1) 0-shot 1.8 4.3 4.9 MBPP (Pass@1) 3-shot 0.2 2.2 2.2 TriviaQA (EM) 5-shot 9.9 16.5 16.6 NaturalQuestions (EM) 5-shot 3.0 6.3 5.7\n\nTable 9 | Comparison between DeepSeekMoE and larger dense baselines.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 15.9B 19.8B 13.3B # Activated Expert Params N/A 2.37B 2.82B 2.05B # Training Tokens N/A 100B 100B 100B HellaSwag (Acc.) 0-shot 66.6 67.7 69.1 PIQA (Acc.) 0-shot 75.6 76.0 75.7 ARC-easy (Acc.) 0-shot 56.8 56.8 58.8 ARC-challenge (Acc.) 0-shot 39.9 37.6 38.5 RACE-middle (Acc.) 5-shot 51.6 50.6 52.4 RACE-high (Acc.) 5-shot 37.4 36.3 38.5 HumanEval (Pass@1) 0-shot 6.1 6.1 9.8 MBPP (Pass@1) 3-shot 7.0 11.6 10.6 TriviaQA (EM) 5-shot 36.5 36.7 38.2 NaturalQuestions (EM) 5-shot 12.6 12.1 13.7\n\nTable 10 | Comparison between DeepSeekMoE and larger GShard models at a larger scale.\n\nC. Training Benchmark Curves Of Deepseekmoe 16B\n\nWe present the benchmark curves during training of DeepSeekMoE 16B and DeepSeek 7B (Dense) in Figure 7 for reference.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/DeepSeekMoE_2.md"
+          },
+          "summary": "DeepSeekMoE is an innovative Mixture-of-Experts (MoE) architecture designed to enhance expert specialization in language models. It employs fine-grained expert segmentation and shared expert isolation to improve performance and reduce computational costs. Starting with a model of 2B parameters, DeepSeekMoE demonstrates comparable performance to larger models like GShard 2.9B and LLaMA2 7B, using significantly fewer computations. The architecture scales up to 16B and 145B parameters, maintaining advantages over traditional MoE architectures. DeepSeekMoE 16B achieves similar results to dense models with only 40% of the computations, and its chat model version shows adaptability in bilingual scenarios. The architecture's public release aims to contribute to advancements in large-scale language models.",
+          "headlines": {
+            "1. Introduction": [],
+            "2. Preliminaries: Mixture-Of-Experts For Transformers": [],
+            "3. Deepseekmoe Architecture": [
+              "3.1. Fine-Grained Expert Segmentation",
+              "3.2. Shared Expert Isolation",
+              "3.3. Load Balance Consideration"
+            ],
+            "4. Validation Experiments": [
+              "4.1. Experimental Setup",
+              "4.1.1. Training Data And Tokenization",
+              "4.1.2. Infrastructures",
+              "4.1.3. Hyper-Parameters",
+              "4.1.4. Evaluation Benchmarks",
+              "4.2. Evaluations",
+              "4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models",
+              "4.4. Ablation Studies",
+              "4.5. Analysis On Expert Specialization"
+            ],
+            "5. Scaling Up To Deepseekmoe 16B": [
+              "5.1. Experimental Setup",
+              "5.1.1. Training Data And Tokenization",
+              "5.1.2. Hyper-Parameters",
+              "5.1.3. Evaluation Benchmarks",
+              "5.2. Evaluations",
+              "5.2.1. Internal Comparison With Deepseek 7B",
+              "5.2.2. Comparison With Open Source Models"
+            ],
+            "6. Alignment For Deepseekmoe 16B": [
+              "6.1. Experimental Setup",
+              "6.2. Evaluations"
+            ],
+            "7. Deepseekmoe 145B Ongoing": [
+              "7.1. Experimental Setup",
+              "7.2. Evaluations"
+            ],
+            "8. Related Work": [],
+            "9. Conclusion": [],
+            "Appendices": [
+              "A. Overview Of Hyper-Parameters",
+              "B. Comparing Deepseekmoe With Larger Models",
+              "C. Training Benchmark Curves Of Deepseekmoe 16B"
+            ]
+          },
+          "summary_embedding": [
+            -0.012465757317841053,
+            -0.022808987647294998,
+            -0.018792515620589256,
+            -0.02308654971420765,
+            0.01797616109251976,
+            0.007200260180979967,
+            0.006490030791610479,
+            0.042613785713911057,
+            -0.014049487188458443,
+            0.03187054023146629,
+            -0.021486490964889526,
+            -0.023592690005898476,
+            0.009045224636793137,
+            -0.014441338367760181,
+            0.005865518003702164,
+            -0.007363531738519669,
+            -0.008743173442780972,
+            -0.005163452122360468,
+            0.018841497600078583,
+            -0.03100520372390747,
+            0.011110605672001839,
+            -0.046075135469436646,
+            -0.020017050206661224,
+            -0.021845687180757523,
+            0.016425084322690964,
+            0.037389107048511505,
+            0.022547753527760506,
+            -0.0028062239289283752,
+            -0.06844329088926315,
+            0.034678805619478226,
+            0.035168617963790894,
+            0.02052319049835205,
+            0.0005438972148112953,
+            -0.00031302153365686536,
+            -0.024833550676703453,
+            -0.028556134551763535,
+            0.03347059711813927,
+            -0.004379750229418278,
+            -0.01453930139541626,
+            0.015976088121533394,
+            0.0414055772125721,
+            0.012106560170650482,
+            -0.02754385396838188,
+            0.0014245413476601243,
+            0.015151568688452244,
+            0.0055961208418011665,
+            -0.01678428053855896,
+            -0.0022960014175623655,
+            -0.002424577483907342,
+            -0.021225256845355034,
+            -0.01400050614029169,
+            0.018825169652700424,
+            0.001296985661610961,
+            -0.008865626528859138,
+            -0.0542386956512928,
+            0.007330877240747213,
+            0.03840138763189316,
+            0.05623060464859009,
+            0.044148534536361694,
+            -0.020180322229862213,
+            0.03369917720556259,
+            -0.030025575309991837,
+            -0.02501314878463745,
+            0.011028970591723919,
+            0.003983817528933287,
+            -0.018531281501054764,
+            0.01784554310142994,
+            -0.005636938381940126,
+            -0.0006556359585374594,
+            0.01751900091767311,
+            -0.03941367194056511,
+            0.020751770585775375,
+            -0.01887415163218975,
+            0.031266435980796814,
+            0.02173139713704586,
+            0.0073798587545752525,
+            0.01872720755636692,
+            0.01678428053855896,
+            0.0036858476232737303,
+            -0.04107903689146042,
+            0.009118696674704552,
+            0.016523046419024467,
+            0.014196431264281273,
+            0.016269976273179054,
+            0.06553706526756287,
+            -0.05391215533018112,
+            -0.004485876299440861,
+            -0.05169166624546051,
+            0.0012561678886413574,
+            -0.004971608519554138,
+            -0.03670336678624153,
+            0.01048201136291027,
+            0.028425518423318863,
+            0.021372200921177864,
+            -0.01063711941242218,
+            0.024392718449234962,
+            -0.022743679583072662,
+            0.0032001156359910965,
+            -0.04062187671661377,
+            0.0019653772469609976,
+            0.004518530797213316,
+            0.016531210392713547,
+            0.0070778070949018,
+            0.004273623693734407,
+            -0.016702644526958466,
+            0.006045116577297449,
+            -0.01905374974012375,
+            0.03291547670960426,
+            -0.008465612307190895,
+            0.08653374016284943,
+            0.008751336485147476,
+            0.004738946910947561,
+            0.004281787667423487,
+            0.02398454025387764,
+            0.012131051160395145,
+            -0.00810233410447836,
+            -0.04718538001179695,
+            -0.015690363943576813,
+            0.011061624623835087,
+            -0.0383034273982048,
+            0.018335357308387756,
+            -0.008759500458836555,
+            0.01282495353370905,
+            -0.04107903689146042,
+            -0.06344719231128693,
+            0.01498013362288475,
+            0.001784758409485221,
+            -0.004065453074872494,
+            -0.0002903166168835014,
+            -0.018923133611679077,
+            0.003930754493921995,
+            -0.004200151655822992,
+            -0.010645282454788685,
+            -0.021502818912267685,
+            0.03159298002719879,
+            0.03657275065779686,
+            0.06527583301067352,
+            0.036801330745220184,
+            -0.0030164355412125587,
+            0.005775718949735165,
+            0.019690508022904396,
+            -0.020784424617886543,
+            0.003481758525595069,
+            -0.01214737817645073,
+            -0.021992631256580353,
+            -0.03820546343922615,
+            0.010971825569868088,
+            0.07379858940839767,
+            -0.029029620811343193,
+            0.013437220826745033,
+            -0.017616963014006615,
+            0.04205866530537605,
+            -0.004567512311041355,
+            0.026727497577667236,
+            -0.016669990494847298,
+            0.004059330560266972,
+            -0.025388672947883606,
+            0.01620466820895672,
+            -0.024964168667793274,
+            0.012571883387863636,
+            -0.000885746325366199,
+            -0.04222193360328674,
+            0.00011116473615402356,
+            0.026890767738223076,
+            -0.010971825569868088,
+            0.02261306345462799,
+            0.0004949158756062388,
+            -0.006302268709987402,
+            -0.05097327008843422,
+            0.038727931678295135,
+            -0.003912386484444141,
+            0.018645571544766426,
+            -0.0018959869630634785,
+            0.0163107942789793,
+            -0.0016408757073804736,
+            0.013216804713010788,
+            0.009543201886117458,
+            0.007114543113857508,
+            -0.033503253012895584,
+            0.02697240374982357,
+            -0.02504580281674862,
+            0.028049994260072708,
+            0.03435226157307625,
+            -0.0030184765346348286,
+            0.03255628049373627,
+            0.019249675795435905,
+            0.03840138763189316,
+            0.0327032245695591,
+            0.030335791409015656,
+            -0.00824927818030119,
+            0.013878053054213524,
+            0.008200296200811863,
+            0.03608293831348419,
+            0.013429056853055954,
+            0.019723162055015564,
+            -0.009730963967740536,
+            -0.014041324146091938,
+            0.048785436898469925,
+            0.007318632211536169,
+            -0.0179598331451416,
+            -0.005441013257950544,
+            -0.03510330989956856,
+            -0.002832755446434021,
+            -0.014425011351704597,
+            0.0323440283536911,
+            0.02158445306122303,
+            -0.01647406443953514,
+            0.035135965794324875,
+            0.024980494752526283,
+            -0.03751972317695618,
+            -0.005232842173427343,
+            -0.02026195637881756,
+            -0.023298801854252815,
+            -0.04215662553906441,
+            0.011624909937381744,
+            0.012833117507398129,
+            -0.013976015150547028,
+            -0.007469657808542252,
+            0.020457882434129715,
+            -0.023625344038009644,
+            -0.0018265966791659594,
+            -0.01356783788651228,
+            0.006547175347805023,
+            -0.020898714661598206,
+            -0.03667071461677551,
+            -0.01048201136291027,
+            0.025862159207463264,
+            0.012637191452085972,
+            0.04362606629729271,
+            -0.013518855907022953,
+            -0.005367541220039129,
+            0.033193036913871765,
+            0.012759645469486713,
+            0.010767736472189426,
+            0.03699725493788719,
+            -0.03027048148214817,
+            0.024621298536658287,
+            -0.020294612273573875,
+            -0.0032878739293664694,
+            0.01707816869020462,
+            -0.0030062312725931406,
+            0.01962519995868206,
+            -0.04535674303770065,
+            -0.024196794256567955,
+            0.04395261034369469,
+            -0.0007107399869710207,
+            0.016302630305290222,
+            -0.005959399044513702,
+            -0.012106560170650482,
+            0.0035899258218705654,
+            0.0027715289033949375,
+            -0.01438419334590435,
+            -0.008620720356702805,
+            0.007473739795386791,
+            -0.01962519995868206,
+            0.03660540655255318,
+            0.03794422745704651,
+            -0.004132802598178387,
+            -0.014131123200058937,
+            -0.03324201703071594,
+            0.010016689077019691,
+            -0.017878197133541107,
+            -0.023951886221766472,
+            0.04078514873981476,
+            -0.032115448266267776,
+            -0.00029312283731997013,
+            0.039478980004787445,
+            0.018808843567967415,
+            -0.014898497611284256,
+            0.00868602842092514,
+            -0.01709449477493763,
+            -0.026302991434931755,
+            -0.017159804701805115,
+            0.03356856107711792,
+            0.024800896644592285,
+            -0.010743245482444763,
+            0.010245268233120441,
+            0.005330805201083422,
+            0.02893165871500969,
+            0.024376392364501953,
+            0.00648594880476594,
+            -0.013812744058668613,
+            0.015029114671051502,
+            0.027641816064715385,
+            -0.0028021421749144793,
+            -7.66609373386018e-05,
+            -0.004010349046438932,
+            -0.007796200457960367,
+            -0.006453294772654772,
+            -0.013069860637187958,
+            -0.005089980084449053,
+            0.0009184005321003497,
+            -0.032295044511556625,
+            0.015967924147844315,
+            -0.005890008993446827,
+            0.010359558276832104,
+            -0.011535110883414745,
+            0.01288209855556488,
+            0.01722511276602745,
+            -0.013494365848600864,
+            -0.0716434046626091,
+            0.014612773433327675,
+            0.02458864450454712,
+            -0.006706364918500185,
+            -0.017943505197763443,
+            0.008481939323246479,
+            0.00016148542636074126,
+            0.038597315549850464,
+            -0.014180104248225689,
+            -0.031103165820240974,
+            0.02680913358926773,
+            -0.017437364906072617,
+            -0.015241367742419243,
+            -0.010661610402166843,
+            -0.00580837344750762,
+            0.04649963974952698,
+            0.01889047957956791,
+            0.011420820839703083,
+            -0.04042595252394676,
+            -0.007449248805642128,
+            0.015078096650540829,
+            0.014506647363305092,
+            -0.04705476388335228,
+            0.0052083516493439674,
+            0.015486273914575577,
+            0.019723162055015564,
+            0.024800896644592285,
+            0.04274440184235573,
+            -0.0021531390957534313,
+            0.01661284640431404,
+            -0.015053605660796165,
+            -0.05136512219905853,
+            -0.03340528905391693,
+            0.029943939298391342,
+            -0.017731253057718277,
+            -0.02906227484345436,
+            -0.01693122461438179,
+            -0.006151242647320032,
+            0.0017939424142241478,
+            0.007126788143068552,
+            -0.01916803978383541,
+            -0.03464614972472191,
+            -0.03970756009221077,
+            0.02277633361518383,
+            0.007983962073922157,
+            0.004269542172551155,
+            -0.042777057737112045,
+            0.0013459670590236783,
+            0.009184005670249462,
+            0.012073906138539314,
+            0.0009954441338777542,
+            -0.019984396174550056,
+            0.015494437888264656,
+            0.0032001156359910965,
+            -0.031070511788129807,
+            -0.035593122243881226,
+            -0.03595232218503952,
+            0.0023919232189655304,
+            0.003781769424676895,
+            0.06279410421848297,
+            0.012988224625587463,
+            -0.03840138763189316,
+            -0.04522612690925598,
+            0.029176564887166023,
+            -0.02052319049835205,
+            -0.016678154468536377,
+            0.0059430720284581184,
+            0.012898425571620464,
+            -0.009877908043563366,
+            0.013192313723266125,
+            0.023560035973787308,
+            0.041601505130529404,
+            -0.005987971555441618,
+            0.012914752587676048,
+            0.023037567734718323,
+            0.012955570593476295,
+            0.0362788625061512,
+            -0.00936360377818346,
+            0.0035409443080425262,
+            0.039936136454343796,
+            -0.012269831262528896,
+            0.03175625205039978,
+            -0.02189466916024685,
+            0.0020398697815835476,
+            0.02249877341091633,
+            0.027184655889868736,
+            -0.0006479826406575739,
+            0.02594379521906376,
+            -0.031543996185064316,
+            0.018368011340498924,
+            -0.014498483389616013,
+            -0.04888340085744858,
+            -0.01557607389986515,
+            0.013102514669299126,
+            0.030384771525859833,
+            0.02756018005311489,
+            -0.0041348435916006565,
+            -0.019723162055015564,
+            -0.01155143789947033,
+            -0.010596301406621933,
+            0.0018419033149257302,
+            0.03285016864538193,
+            0.010506502352654934,
+            0.010890189558267593,
+            -0.011935125105082989,
+            -0.004104230087250471,
+            0.03025415539741516,
+            -0.021192602813243866,
+            0.01650671847164631,
+            0.0041511706076562405,
+            -0.002029665280133486,
+            -0.04320156201720238,
+            -0.035266581922769547,
+            0.007845181971788406,
+            0.011747363954782486,
+            0.014874007552862167,
+            0.0368339866399765,
+            0.025094784796237946,
+            -0.01662100851535797,
+            0.030629679560661316,
+            -0.01905374974012375,
+            0.027984686195850372,
+            0.039054471999406815,
+            0.011624909937381744,
+            -0.018351683393120766,
+            0.012237177230417728,
+            0.002424577483907342,
+            -0.025127438828349113,
+            0.01680060662329197,
+            -0.032735876739025116,
+            0.01738838478922844,
+            -0.02354370802640915,
+            0.025143766775727272,
+            -0.02579685114324093,
+            -0.009290131740272045,
+            -0.004428731743246317,
+            -0.006130834110081196,
+            0.04888340085744858,
+            -0.026727497577667236,
+            -0.0142454132437706,
+            -0.0024694770108908415,
+            -0.0024531499948352575,
+            -0.008718682453036308,
+            0.010000362060964108,
+            -0.045781247317790985,
+            -0.01844964735209942,
+            -0.04016471654176712,
+            0.007355368230491877,
+            0.005412440747022629,
+            -0.030907239764928818,
+            -0.05694899708032608,
+            0.025421326979994774,
+            -0.06331657618284225,
+            -0.019886434078216553,
+            0.051626358181238174,
+            -0.018400665372610092,
+            -0.008212542161345482,
+            0.009641164913773537,
+            0.006375740747898817,
+            -0.029290854930877686,
+            0.0041348435916006565,
+            0.02504580281674862,
+            0.04418119043111801,
+            -0.0045797573402523994,
+            -0.02865409664809704,
+            0.0028674507047981024,
+            -0.027788760140538216,
+            -0.013461710885167122,
+            0.005261414684355259,
+            0.016449574381113052,
+            0.024539662525057793,
+            0.007628847379237413,
+            0.03585435822606087,
+            0.007294141221791506,
+            0.006318595726042986,
+            0.044279150664806366,
+            -0.025666235014796257,
+            -0.027380581945180893,
+            0.02519274689257145,
+            0.04045860469341278,
+            0.023151857778429985,
+            -0.018253721296787262,
+            -0.0142454132437706,
+            0.0053063142113387585,
+            0.0009296254720538855,
+            -0.022139575332403183,
+            0.002808264922350645,
+            -0.013902543112635612,
+            0.005143043119460344,
+            0.03892385587096214,
+            -0.001205145614221692,
+            0.014196431264281273,
+            0.007302304729819298,
+            0.006845145486295223,
+            0.004334850702434778,
+            -0.011731036007404327,
+            -0.023592690005898476,
+            0.00032220553839579225,
+            0.034972693771123886,
+            0.015404638834297657,
+            -0.030515389516949654,
+            -0.00995954405516386,
+            0.034711457788944244,
+            -0.00795130804181099,
+            -0.009118696674704552,
+            -0.009771781973540783,
+            0.003385836724191904,
+            0.02638462744653225,
+            -0.01606588624417782,
+            0.006649219896644354,
+            0.03716052696108818,
+            -0.019870106130838394,
+            0.005543057806789875,
+            -0.025405000895261765,
+            0.0018061877926811576,
+            0.02442537248134613,
+            -0.006216551177203655,
+            0.028098974376916885,
+            5.6315813708351925e-05,
+            0.03025415539741516,
+            -0.004677720367908478,
+            0.008432958275079727,
+            -0.020327266305685043,
+            0.03987082839012146,
+            -0.017633290961384773,
+            0.03402572125196457,
+            0.02067013457417488,
+            -0.01258004643023014,
+            0.04783846437931061,
+            -0.036801330745220184,
+            -0.020784424617886543,
+            0.00584919098764658,
+            0.012743317522108555,
+            0.0028919412288814783,
+            -0.0191517136991024,
+            -0.018400665372610092,
+            -0.010996315628290176,
+            0.02380494214594364,
+            -0.022547753527760506,
+            -0.016865916550159454,
+            -0.007441085297614336,
+            -0.01890680566430092,
+            -0.009673818945884705,
+            -0.007988044060766697,
+            -0.01709449477493763,
+            -0.016547536477446556,
+            0.044573038816452026,
+            -0.06334923207759857,
+            0.016996532678604126,
+            0.017159804701805115,
+            0.011477965861558914,
+            -0.034678805619478226,
+            -0.019510909914970398,
+            -0.017633290961384773,
+            -0.002347023691982031,
+            -0.030743969604372978,
+            0.001702102366834879,
+            0.01725776679813862,
+            0.01138000376522541,
+            -0.05022222548723221,
+            0.012335140258073807,
+            0.02772345207631588,
+            -0.012065742164850235,
+            0.0014949520118534565,
+            -0.023135529831051826,
+            -0.017176130786538124,
+            0.0031776658724993467,
+            0.022123249247670174,
+            0.005975726060569286,
+            -0.008188051171600819,
+            -0.016865916550159454,
+            -0.005241005681455135,
+            -0.006045116577297449,
+            -0.002359268954023719,
+            0.005094061605632305,
+            0.011780017986893654,
+            0.019690508022904396,
+            -0.0024633544962853193,
+            -0.004436895251274109,
+            -0.006396149750798941,
+            0.027364253997802734,
+            -0.001450052484869957,
+            -0.008416631259024143,
+            0.050907962024211884,
+            0.03556047007441521,
+            0.006094098091125488,
+            -0.01108611561357975,
+            -0.03990348428487778,
+            0.010661610402166843,
+            -0.022711025550961494,
+            0.0401320643723011,
+            0.01905374974012375,
+            -0.008298259228467941,
+            -0.028866350650787354,
+            0.010294250212609768,
+            -0.02290695160627365,
+            -0.009975871071219444,
+            0.009469729848206043,
+            0.00726556871086359,
+            -0.03263791650533676,
+            -0.013543346896767616,
+            0.03510330989956856,
+            0.011437147855758667,
+            -0.006841063499450684,
+            -0.036050282418727875,
+            0.008588065393269062,
+            0.01646590232849121,
+            0.011869817040860653,
+            -0.019233347848057747,
+            0.004449140280485153,
+            -0.012041252106428146,
+            0.017470018938183784,
+            0.008849299512803555,
+            -0.017796561121940613,
+            0.018237393349409103,
+            0.004710374400019646,
+            -0.008490103296935558,
+            -0.003912386484444141,
+            -0.0163107942789793,
+            -0.02006603218615055,
+            0.03840138763189316,
+            0.024490682408213615,
+            0.026335645467042923,
+            0.005457340274006128,
+            0.02008235827088356,
+            0.03227871656417847,
+            0.011282040737569332,
+            -0.03134807199239731,
+            0.027935704216361046,
+            0.016996532678604126,
+            -0.04604247957468033,
+            0.021551799029111862,
+            -0.02218855731189251,
+            -0.028670424595475197,
+            0.009649328887462616,
+            -0.0017367975087836385,
+            0.011192241683602333,
+            0.0034837995190173388,
+            -0.003642988856881857,
+            -0.03307874873280525,
+            -0.006236960180103779,
+            0.015771998092532158,
+            0.015502601861953735,
+            -0.0048695639707148075,
+            -0.003296037670224905,
+            -0.011886144056916237,
+            0.00487364549189806,
+            0.009796272963285446,
+            0.039054471999406815,
+            0.001381682581268251,
+            -0.004751192405819893,
+            -0.0018765984568744898,
+            0.031854212284088135,
+            0.0271193478256464,
+            -0.017274094745516777,
+            -0.026286665350198746,
+            0.00580837344750762,
+            -0.014253576286137104,
+            0.026156047359108925,
+            -0.00487364549189806,
+            0.011877980083227158,
+            0.008718682453036308,
+            -0.0007908449042588472,
+            -0.001373519073240459,
+            0.025584599003195763,
+            -0.02710302174091339,
+            -0.021764053031802177,
+            0.031233781948685646,
+            -0.025731543079018593,
+            0.028735732659697533,
+            0.013061696663498878,
+            -0.01557607389986515,
+            -0.015225040726363659,
+            -0.010302413254976273,
+            0.009943217039108276,
+            -0.012277995236217976,
+            0.023853924125432968,
+            0.003608293831348419,
+            0.006392067763954401,
+            0.0003584313380997628,
+            0.02770712412893772,
+            0.024833550676703453,
+            0.007820690982043743,
+            -0.018074123188853264,
+            -0.0016816934803500772,
+            -0.02320083975791931,
+            -0.01680060662329197,
+            0.023772288113832474,
+            -0.018090449273586273,
+            -0.03268689662218094,
+            -0.028556134551763535,
+            0.012114724144339561,
+            0.007392104249447584,
+            0.010816717520356178,
+            0.03461349755525589,
+            -0.030384771525859833,
+            -0.0344502255320549,
+            0.01766594499349594,
+            0.026270337402820587,
+            -0.005551221314817667,
+            -0.021535472944378853,
+            0.03085825778543949,
+            -0.005828781984746456,
+            0.017600636929273605,
+            0.0404912605881691,
+            -0.0016225076979026198,
+            0.010228941217064857,
+            -0.042026009410619736,
+            -0.014057651162147522,
+            -0.0066614653915166855,
+            0.006261450704187155,
+            0.034417569637298584,
+            0.01048201136291027,
+            0.01535565685480833,
+            0.03448288142681122,
+            -0.021192602813243866,
+            -0.0002903166168835014,
+            -0.0022449791431427,
+            -0.02818061038851738,
+            0.03347059711813927,
+            0.025715215131640434,
+            0.011918798089027405,
+            0.019706835970282555,
+            -0.03794422745704651,
+            -0.013037205673754215,
+            0.027070365846157074,
+            0.015053605660796165,
+            0.01273515447974205,
+            0.016694480553269386,
+            0.011249386705458164,
+            0.00868602842092514,
+            -0.021633435040712357,
+            -0.05169166624546051,
+            0.01214737817645073,
+            0.01993541419506073,
+            -0.016449574381113052,
+            0.01057181041687727,
+            0.024980494752526283,
+            0.007620683871209621,
+            -0.007045152597129345,
+            0.012808626517653465,
+            -0.02682545967400074,
+            0.002259265398606658,
+            0.0019010890973731875,
+            -0.014482156373560429,
+            0.038270771503448486,
+            -0.016367938369512558,
+            0.011363675817847252,
+            -0.018204739317297935,
+            0.023870250210165977,
+            0.00011531029304023832,
+            0.0007964573451317847,
+            0.014963806606829166,
+            -0.011347348801791668,
+            -0.04235255345702171,
+            0.0021613026037812233,
+            0.03342161700129509,
+            -0.015200549736618996,
+            0.026449935510754585,
+            -0.0020674217958003283,
+            0.032458316534757614,
+            0.042777057737112045,
+            0.017731253057718277,
+            0.0003497575526125729,
+            0.025682561099529266,
+            0.0036144163459539413,
+            0.0073880222626030445,
+            0.012400448322296143,
+            0.009322785772383213,
+            0.007245160173624754,
+            0.030809277668595314,
+            -0.010261595249176025,
+            0.02173139713704586,
+            -0.04760988429188728,
+            -0.014629100449383259,
+            0.012024925090372562,
+            -0.0013316808035597205,
+            -0.012449429370462894,
+            -0.033029764890670776,
+            0.012563719414174557,
+            -0.03503800183534622,
+            0.022074267268180847,
+            -0.0004959363141097128,
+            0.04333217814564705,
+            -0.02187834307551384,
+            -0.005518566817045212,
+            0.016694480553269386,
+            0.0016510800924152136,
+            0.023282473906874657,
+            -0.0012153501156717539,
+            -0.016416920349001884,
+            -0.0067798369564116,
+            -0.009257477708160877,
+            0.004138925112783909,
+            0.04375668615102768,
+            -0.003722583642229438,
+            -0.0001682458823779598,
+            0.01409030519425869,
+            -0.011959616094827652,
+            -0.03250729665160179,
+            0.045193471014499664,
+            -0.0030837850645184517,
+            0.02484987862408161,
+            0.04979771748185158,
+            -0.06410028040409088,
+            -0.019543563947081566,
+            0.013061696663498878,
+            -0.010441194288432598,
+            0.011363675817847252,
+            0.005547139327973127,
+            0.016114868223667145,
+            0.017894525080919266,
+            -0.005485912784934044,
+            0.01619650423526764,
+            0.005106307100504637,
+            0.028098974376916885,
+            -0.02336410991847515,
+            -0.03296445682644844,
+            -0.02636830136179924,
+            -0.017747581005096436,
+            0.0014480114914476871,
+            -0.015796490013599396,
+            -3.563585551091819e-06,
+            -0.0022204883862286806,
+            -0.04388730227947235,
+            -0.025274382904171944,
+            0.040687184780836105,
+            0.006706364918500185,
+            0.02458864450454712,
+            0.0077268099412322044,
+            0.03418898954987526,
+            0.005885927006602287,
+            -0.009037061594426632,
+            0.030058229342103004,
+            -0.020049704238772392,
+            0.03833607956767082,
+            0.022106921300292015,
+            -0.002832755446434021,
+            0.005865518003702164,
+            0.031086837872862816,
+            -0.033797141164541245,
+            0.008086007088422775,
+            -0.034384917467832565,
+            0.0046695563942193985,
+            -0.008620720356702805,
+            -0.015094423666596413,
+            -0.009526874870061874,
+            -0.02504580281674862,
+            0.011216731742024422,
+            -0.021535472944378853,
+            0.01590261608362198,
+            0.010416703298687935,
+            -0.002916431985795498,
+            0.03448288142681122,
+            -0.014204595237970352,
+            -0.0027796924114227295,
+            0.0024531499948352575,
+            -0.027070365846157074,
+            -0.006343086715787649,
+            -0.003377673216164112,
+            -0.03689929470419884,
+            -0.004502203781157732,
+            0.010988152585923672,
+            -0.00024184548237826675,
+            0.016898570582270622,
+            0.02787039615213871,
+            0.0005092020728625357,
+            -0.022302847355604172,
+            0.002340900944545865,
+            -0.04199335351586342,
+            0.0011877981014549732,
+            0.01483318954706192,
+            -0.017600636929273605,
+            -0.008612556383013725,
+            0.033046092838048935,
+            0.023298801854252815,
+            -0.009396257810294628,
+            -0.035005345940589905,
+            0.017796561121940613,
+            -0.011853490024805069,
+            0.027511198073625565,
+            -0.0002630197268445045,
+            0.019298657774925232,
+            -0.0076941559091210365,
+            0.04493223875761032,
+            0.001369437319226563,
+            0.008645210415124893,
+            -0.021959977224469185,
+            0.007020662073045969,
+            -0.020278284326195717,
+            -0.002406209474429488,
+            -0.006171651650220156,
+            -0.012751481495797634,
+            -0.010669773444533348,
+            0.0032735876739025116,
+            0.007730891928076744,
+            -0.0176822729408741,
+            0.0055961208418011665,
+            -0.015959760174155235,
+            -0.02741323597729206,
+            0.007543129846453667,
+            -0.05965929850935936,
+            -0.0179598331451416,
+            -0.03373183310031891,
+            -0.009477893821895123,
+            0.02008235827088356,
+            0.03510330989956856,
+            -0.012759645469486713,
+            0.00497977202758193,
+            -0.014898497611284256,
+            0.004477712791413069,
+            0.03490738570690155,
+            0.044442422688007355,
+            -0.014759717509150505,
+            0.003381754970178008,
+            -0.010955498553812504,
+            0.033029764890670776,
+            -0.03347059711813927,
+            0.0030062312725931406,
+            0.031086837872862816,
+            -0.005057325586676598,
+            0.007959471084177494,
+            0.011624909937381744,
+            0.020474210381507874,
+            -0.019396619871258736,
+            0.00877582747489214,
+            0.002100075827911496,
+            0.019233347848057747,
+            0.02171507105231285,
+            0.020343592390418053,
+            -0.03539719805121422,
+            0.028409190475940704,
+            0.018384337425231934,
+            0.02444170042872429,
+            0.044442422688007355,
+            -0.03974021226167679,
+            -0.004147088620811701,
+            -0.0032225653994828463,
+            -0.02997659333050251,
+            0.02008235827088356,
+            -0.007530884351581335,
+            -0.01841699331998825,
+            -0.007069643586874008,
+            0.03327467292547226,
+            0.014571955427527428,
+            -0.05051611363887787,
+            0.01980479806661606,
+            0.01740471087396145,
+            0.02055584453046322,
+            0.024474354460835457,
+            -0.01123305968940258,
+            -0.0019072118448093534,
+            0.008996243588626385,
+            0.006804327480494976,
+            0.019266001880168915,
+            0.015257694758474827,
+            0.002012317767366767,
+            -0.0022082431241869926,
+            -0.011837163008749485,
+            0.005224678665399551,
+            -0.009077879600226879,
+            0.010808554477989674,
+            -0.02458864450454712,
+            -0.009698309935629368,
+            0.021388528868556023,
+            0.0031164393294602633,
+            -0.013665799982845783,
+            -0.03729114308953285,
+            -0.045944519340991974,
+            0.0033837957307696342,
+            0.004543021321296692,
+            -0.05365091934800148,
+            0.005322641227394342,
+            -0.026547899469733238,
+            -0.002108239568769932,
+            -0.005502239800989628,
+            0.0019214981002733111,
+            0.005636938381940126,
+            0.0167679525911808,
+            -0.013494365848600864,
+            0.02248244546353817,
+            -0.015502601861953735,
+            0.03165828809142113,
+            0.0055961208418011665,
+            0.007375776767730713,
+            -0.0027796924114227295,
+            0.05590406060218811,
+            -0.003712379140779376,
+            -0.03699725493788719,
+            0.023282473906874657,
+            -0.01311067771166563,
+            -0.0001747512142173946,
+            0.030090883374214172,
+            0.00884113647043705,
+            -0.015257694758474827,
+            0.013437220826745033,
+            -0.030613351613283157,
+            -0.03249097242951393,
+            0.010996315628290176,
+            -0.013755599036812782,
+            -0.0015776080545037985,
+            -0.04568328335881233,
+            -0.009339112788438797,
+            0.004269542172551155,
+            -0.018335357308387756,
+            0.010775899514555931,
+            0.01631895825266838,
+            0.013992343097925186,
+            -0.024245774373412132,
+            0.009053388610482216,
+            0.006865554489195347,
+            0.013453547842800617,
+            0.0007331897504627705,
+            0.014049487188458443,
+            -0.008677864447236061,
+            -0.011306531727313995,
+            -0.018319029361009598,
+            0.0032450153958052397,
+            -0.026792805641889572,
+            0.005155288614332676,
+            -0.003155216109007597,
+            -0.03771565109491348,
+            -0.06521052122116089,
+            -0.0344502255320549,
+            0.016294466331601143,
+            0.03396041318774223,
+            -0.009045224636793137,
+            0.0035144127905368805,
+            -0.00023648813657928258,
+            -0.01093917153775692,
+            0.024523336440324783,
+            -0.005930826533585787,
+            -0.0004829256213270128,
+            0.0035389035474509,
+            0.02050686441361904,
+            -0.009722800925374031,
+            0.005845109466463327,
+            -0.051952898502349854,
+            -0.01586179807782173,
+            0.01812310516834259,
+            0.006306350696831942,
+            0.014629100449383259,
+            -0.009690145961940289,
+            0.01693122461438179,
+            -0.030074557289481163,
+            -0.028147956356406212,
+            0.04130761697888374,
+            0.008277850225567818,
+            0.011486129835247993,
+            -0.003055212553590536,
+            -0.01680060662329197,
+            0.02504580281674862,
+            -0.0020633398089557886,
+            -0.006461458280682564,
+            -0.00817988719791174,
+            0.015820980072021484,
+            0.044279150664806366,
+            0.003255219664424658,
+            0.0009821783751249313,
+            0.017045514658093452,
+            -0.002663361607119441,
+            -0.011575928889214993,
+            0.007677828427404165,
+            0.013592327944934368,
+            -0.016384266316890717,
+            0.010375885292887688,
+            -0.025862159207463264,
+            0.0332583449780941,
+            0.03131541982293129,
+            0.011575928889214993,
+            -0.015045441687107086,
+            0.006130834110081196,
+            0.005330805201083422,
+            0.008086007088422775,
+            -0.001369437319226563,
+            -0.0017133272485807538,
+            0.020719116553664207,
+            -0.03276853263378143,
+            -0.01693122461438179,
+            -0.005347132217139006,
+            0.003932795487344265,
+            0.005277741700410843,
+            0.03396041318774223,
+            -0.0033266509417444468,
+            0.017927179113030434,
+            -0.011657563969492912,
+            0.010579974390566349,
+            0.021764053031802177,
+            0.037095218896865845,
+            -0.024996822699904442,
+            -0.029862305149435997,
+            0.012914752587676048,
+            -0.001296985661610961,
+            0.014065815135836601,
+            0.0010173837654292583,
+            0.014139287173748016,
+            0.0029409227427095175,
+            0.060475654900074005,
+            -0.017763907089829445,
+            0.004628738854080439,
+            0.0004035860183648765,
+            -0.01858026348054409,
+            -0.0013531101867556572,
+            0.004261378664523363,
+            -0.010979988612234592,
+            0.03608293831348419,
+            0.002344982698559761,
+            0.039185091853141785,
+            0.013918871060013771,
+            0.009290131740272045,
+            0.0015143405180424452,
+            -0.013592327944934368,
+            0.008355404250323772,
+            0.0015592400450259447,
+            -0.001132694073021412,
+            0.009461566805839539,
+            0.019298657774925232,
+            0.009714636951684952,
+            0.025274382904171944,
+            0.009788108989596367,
+            0.012857607565820217,
+            -0.029470453038811684,
+            0.03510330989956856,
+            -0.0006224715034477413,
+            0.012996388599276543,
+            -0.02040890045464039,
+            -0.003136848099529743,
+            -0.0008240094175562263,
+            0.015649545937776566,
+            -0.015616890974342823,
+            -0.001201063860207796,
+            -0.01297189760953188,
+            0.040099408477544785,
+            -0.018988441675901413,
+            0.013976015150547028,
+            0.009673818945884705,
+            -0.02865409664809704,
+            -0.007596192881464958,
+            0.01995174214243889,
+            0.014294394291937351,
+            -0.0053144777193665504,
+            -0.020490536466240883,
+            0.005832863971590996,
+            0.015380147844552994,
+            -0.028572462499141693,
+            1.3584674888988957e-05,
+            0.014751553535461426,
+            -0.0164903923869133,
+            -0.00025409081717953086,
+            -0.004702210891991854,
+            0.003975654020905495,
+            -0.009265640750527382,
+            0.007028825581073761,
+            0.016057724133133888,
+            -0.024507008492946625,
+            -0.020049704238772392,
+            0.005722655914723873,
+            0.006571666337549686,
+            0.01444950234144926,
+            0.007739055436104536,
+            0.004355259705334902,
+            -0.020033378154039383,
+            -0.01799248717725277,
+            0.0032674651592969894,
+            -0.01649855636060238,
+            -0.0048614004626870155,
+            0.0020531355403363705,
+            -0.019363965839147568,
+            0.01875986158847809,
+            -0.010743245482444763,
+            0.00017538899555802345,
+            -0.003226647386327386,
+            -0.020800752565264702,
+            0.006204306147992611,
+            -0.004865481983870268,
+            0.019870106130838394,
+            0.003555230563506484,
+            0.02785406820476055,
+            0.02310287579894066,
+            -0.004428731743246317,
+            0.017600636929273605,
+            -0.016155686229467392,
+            0.007983962073922157,
+            -0.031135819852352142,
+            -0.001782717532478273,
+            -0.010408539324998856,
+            0.0036103345919400454,
+            -0.015943434089422226,
+            -0.009126860648393631,
+            -0.0048777274787425995,
+            -0.0006678812787868083,
+            -0.004820582456886768,
+            0.0004275664687156677,
+            0.0027817331720143557,
+            0.02231917530298233,
+            -0.013208640739321709,
+            -0.02803366631269455,
+            -0.017437364906072617,
+            0.006045116577297449,
+            0.019396619871258736,
+            -0.026025431230664253,
+            -0.009184005670249462,
+            -0.005012426059693098,
+            -0.004326687194406986,
+            -0.021404854953289032,
+            -0.015167895704507828,
+            -0.005632856860756874,
+            -0.008045189082622528,
+            -0.010522829368710518,
+            0.0046532293781638145,
+            -0.0035715578123927116,
+            -0.011739199981093407,
+            0.021666089072823524,
+            0.018057795241475105,
+            -0.019837452098727226,
+            -0.019837452098727226,
+            0.013641309924423695,
+            -0.006159406621009111,
+            0.0018878233386203647,
+            0.02008235827088356,
+            0.00029286774224601686,
+            0.005824700463563204,
+            0.020882388576865196,
+            -0.0035225762985646725,
+            -0.01797616109251976,
+            0.003577680326998234,
+            -0.0018888438353314996,
+            -0.012122887186706066,
+            0.003212361130863428,
+            0.02773977816104889,
+            0.0008076822850853205,
+            0.023429417982697487,
+            -0.03056436963379383,
+            -0.011412657797336578,
+            0.006061443593353033,
+            0.02369065210223198,
+            0.00211844383738935,
+            -0.0021694661118090153,
+            0.020131340250372887,
+            0.03941367194056511,
+            0.005775718949735165,
+            0.0015755671774968505,
+            -0.04003410041332245,
+            0.01027792226523161,
+            0.009208496659994125,
+            0.024621298536658287,
+            0.008339077234268188,
+            0.0014653591206297278,
+            -0.012906589545309544,
+            -0.009755454957485199,
+            -0.0017939424142241478,
+            0.014702572487294674,
+            -0.023870250210165977,
+            0.0024755997583270073,
+            -0.03157665207982063,
+            0.005094061605632305,
+            0.0021959978621453047,
+            -0.009559528902173042,
+            0.0335359051823616,
+            0.00980443600565195,
+            -0.02065380848944187,
+            0.015151568688452244,
+            0.005767555441707373,
+            0.007330877240747213,
+            0.022449791431427002,
+            -0.008228869177401066,
+            0.030450081452727318,
+            -0.005049162078648806,
+            0.01213921420276165,
+            0.02742956392467022,
+            0.032588932663202286,
+            -0.013102514669299126,
+            -0.003959327004849911,
+            0.0006097159348428249,
+            -0.0002561317232903093,
+            0.0007995187188498676,
+            -0.008204378187656403,
+            0.0037389106582850218,
+            -0.014800535514950752,
+            0.012710663489997387,
+            -0.019543563947081566,
+            -0.0034286954905837774,
+            0.0038532004691660404,
+            0.01991908811032772,
+            0.0016265894519165158,
+            -0.033633869141340256,
+            0.02055584453046322,
+            -0.015420965850353241,
+            0.0135596739128232,
+            0.001290863030590117,
+            -0.011110605672001839,
+            -0.00029210239881649613,
+            0.020588500425219536,
+            0.010947334580123425,
+            -0.007845181971788406,
+            -0.0135596739128232,
+            -0.028376536443829536,
+            -0.021535472944378853,
+            0.00823703221976757,
+            0.009445239789783955,
+            -0.0028837777208536863,
+            0.0021306893322616816,
+            -0.011869817040860653,
+            0.0006520643946714699,
+            -0.0008122742874547839,
+            0.01740471087396145,
+            -0.0335359051823616,
+            0.021943651139736176,
+            -0.019527235999703407,
+            -0.019543563947081566,
+            0.017045514658093452,
+            0.008049271069467068,
+            0.009445239789783955,
+            0.0033552234526723623,
+            -0.01995174214243889,
+            -0.012326976284384727,
+            0.0338951013982296,
+            -0.01138000376522541,
+            0.006298186723142862,
+            0.004408322740346193,
+            0.012914752587676048,
+            -0.008914608508348465,
+            -0.011926962062716484,
+            0.01784554310142994,
+            -0.009861581027507782,
+            0.013951525092124939,
+            0.0010010566329583526,
+            -0.0254703089594841,
+            0.02739690989255905,
+            -0.019233347848057747,
+            0.03487472981214523,
+            -0.021617108955979347,
+            0.00711862463504076,
+            0.0066614653915166855,
+            0.008792154490947723,
+            -0.006690037902444601,
+            -0.013224967755377293,
+            0.0012786176521331072,
+            0.03807484731078148,
+            -0.02623768337070942,
+            0.024180466309189796,
+            0.022400809451937675,
+            -0.00040766780148260295,
+            0.02065380848944187,
+            -0.008367649279534817,
+            -0.026711169630289078,
+            0.0179598331451416,
+            -0.012596373446285725,
+            -0.026123393326997757,
+            0.013257622718811035,
+            -0.022678371518850327,
+            -0.020278284326195717,
+            -0.026302991434931755,
+            0.004947117529809475,
+            -0.0025694805663079023,
+            -0.005151206627488136,
+            0.009004407562315464,
+            0.02561725303530693,
+            0.020915042608976364,
+            0.019576217979192734,
+            0.02111096680164337,
+            0.01161674689501524,
+            0.022449791431427002,
+            -0.007559456862509251,
+            0.04401791840791702,
+            -0.004947117529809475,
+            0.004632820375263691,
+            -0.0015980169409886003,
+            -0.015657709911465645,
+            0.009771781973540783,
+            0.0119106350466609,
+            0.0029572497587651014,
+            -0.029176564887166023,
+            -0.009061552584171295,
+            -0.03660540655255318,
+            0.005053244065493345,
+            0.0020664012990891933,
+            0.0176822729408741,
+            0.032115448266267776,
+            -0.005865518003702164,
+            0.004592002835124731,
+            -0.005102225113660097,
+            -0.013927034102380276,
+            -0.009420748800039291,
+            -0.02739690989255905,
+            -0.025437654927372932,
+            -0.007653337903320789,
+            -0.013739272020757198,
+            -0.030368445441126823,
+            -0.011722872965037823,
+            0.01551892887800932,
+            -0.007645174395292997,
+            0.011918798089027405,
+            -0.016702644526958466,
+            0.01385356206446886,
+            -0.004110352601855993,
+            -0.004763437435030937,
+            -0.004530776292085648,
+            -0.017600636929273605,
+            -0.006420640274882317,
+            0.014367866329848766,
+            -0.009477893821895123,
+            -0.01237595733255148,
+            -0.02215590327978134,
+            -0.0015459742862731218,
+            -0.01574750803411007,
+            0.003489922033622861,
+            -0.014596446417272091,
+            -0.006771673448383808,
+            -0.003908304497599602,
+            -0.018514955416321754,
+            0.01766594499349594,
+            0.014816862531006336,
+            -0.019380291923880577,
+            -0.0090697156265378,
+            0.02638462744653225,
+            0.042777057737112045,
+            -0.03218075633049011,
+            -0.008200296200811863,
+            0.0020378287881612778,
+            0.018090449273586273,
+            0.010171796195209026,
+            0.016702644526958466,
+            -0.024653952568769455,
+            0.0022286521270871162,
+            -0.021649762988090515,
+            -0.001220452249981463,
+            -0.023739634081721306,
+            0.010775899514555931,
+            0.025715215131640434,
+            -0.010971825569868088,
+            0.025976449251174927,
+            0.01634344831109047,
+            -0.01812310516834259,
+            0.012016761116683483,
+            0.007183933164924383,
+            0.014645427465438843,
+            -0.019755816087126732,
+            0.0167679525911808,
+            -0.015796490013599396,
+            0.031233781948685646,
+            0.00892277155071497,
+            -0.04829562455415726,
+            -0.03503800183534622,
+            -0.013388238847255707,
+            0.0013388239312916994,
+            -0.010702427476644516,
+            0.015037278644740582,
+            0.015967924147844315,
+            0.013959688134491444,
+            -0.019380291923880577,
+            -0.015820980072021484,
+            0.011984107084572315,
+            0.010065670125186443,
+            0.027462217956781387,
+            -0.006649219896644354,
+            -0.004212397150695324,
+            -0.00960034690797329,
+            -0.02249877341091633,
+            -0.016098542138934135,
+            0.0209803506731987,
+            -0.018041469156742096,
+            0.015935270115733147,
+            -0.026139721274375916,
+            -0.012449429370462894,
+            0.017453692853450775,
+            -0.05211617052555084,
+            -0.01453113742172718,
+            0.0007612520130351186,
+            0.03012353740632534,
+            0.004751192405819893,
+            0.0035184945445507765,
+            0.01841699331998825,
+            -0.01993541419506073,
+            -0.0039776950143277645,
+            -0.02099667675793171,
+            0.006832899991422892,
+            -0.011853490024805069,
+            -0.013984179124236107,
+            0.014482156373560429,
+            -0.013845399022102356,
+            -0.01601690612733364,
+            0.0034368589986115694,
+            0.007183933164924383,
+            -0.021094640716910362,
+            0.011951452121138573,
+            0.003945040516555309,
+            0.012171869166195393,
+            0.012171869166195393,
+            0.01199227012693882,
+            0.021551799029111862,
+            0.02295593172311783,
+            -0.007334959227591753,
+            -0.004767519421875477,
+            -0.01004117913544178,
+            -0.0015398516552522779,
+            0.008898280560970306,
+            0.0033797139767557383,
+            -0.002989904023706913,
+            0.007837017998099327,
+            0.010033016093075275,
+            -0.01498013362288475,
+            0.033193036913871765,
+            -0.0035878848284482956,
+            -0.04134026914834976,
+            -0.003238892648369074,
+            -0.03800953924655914,
+            0.020033378154039383,
+            -0.023298801854252815,
+            -0.016996532678604126,
+            0.0032776694279164076,
+            -0.00629410520195961,
+            -0.019902760162949562,
+            0.013608654960989952,
+            0.011404493823647499,
+            0.00258580781519413,
+            0.02021297626197338,
+            0.012269831262528896,
+            -0.005134879611432552,
+            0.026172375306487083,
+            -0.014718899503350258,
+            -0.008751336485147476,
+            0.015567909926176071,
+            0.006853308994323015,
+            0.0022347746416926384,
+            0.0011031010653823614,
+            -0.018825169652700424,
+            0.005098143592476845,
+            0.0164903923869133,
+            0.007216587662696838,
+            -0.000566347036510706,
+            0.014049487188458443,
+            0.012514738366007805,
+            -0.015135240741074085,
+            -0.0002213090192526579,
+            -0.026564225554466248,
+            0.0032817514147609472,
+            0.004567512311041355,
+            0.007506393827497959,
+            -0.006771673448383808,
+            0.006281859707087278,
+            0.00658799335360527,
+            0.014759717509150505,
+            -0.010155469179153442,
+            -0.016980206593871117,
+            -0.002510294783860445,
+            0.004285869188606739,
+            0.016694480553269386,
+            0.03157665207982063,
+            -0.010310577228665352,
+            -0.005734901409596205,
+            -0.0386626236140728,
+            0.02070278860628605,
+            0.03332365304231644,
+            0.011175914667546749,
+            -0.028213264420628548,
+            0.012302485294640064,
+            0.0006515541463159025,
+            0.0019276207312941551,
+            -0.026302991434931755,
+            0.02787039615213871,
+            0.00974729098379612,
+            0.0066614653915166855,
+            0.017861871048808098,
+            0.0022306928876787424,
+            0.0029266364872455597,
+            -0.016302630305290222,
+            0.01018812321126461,
+            0.009086042642593384,
+            -0.017306748777627945,
+            0.0009745250572450459,
+            0.02324981987476349,
+            -0.01423724927008152,
+            -0.003208279376849532,
+            0.0018133309204131365,
+            0.02651524543762207,
+            -0.013820908032357693,
+            -0.014294394291937351,
+            0.0005714492290280759,
+            0.01751900091767311,
+            -0.0007979880319908261,
+            0.012098397128283978,
+            0.04898136481642723,
+            -0.012424939312040806,
+            0.019657853990793228,
+            0.005461421795189381,
+            -0.015290348790585995,
+            -0.032605260610580444,
+            -0.004922627005726099,
+            -0.01830270327627659,
+            0.011200404725968838,
+            0.014343376271426678,
+            -0.015763835981488228,
+            0.0007525782566517591,
+            -0.03144603595137596,
+            0.007743136957287788,
+            0.014735226519405842,
+            0.018776189535856247,
+            -0.002440904499962926,
+            0.006134915631264448,
+            0.027674470096826553,
+            0.02458864450454712,
+            -0.007392104249447584,
+            0.013804581016302109,
+            0.02320083975791931,
+            -0.025062130764126778,
+            -0.003069498809054494,
+            0.006820654962211847,
+            0.004104230087250471,
+            0.0053063142113387585,
+            -0.016963878646492958,
+            0.03915243595838547,
+            -0.005922663025557995,
+            -0.01506176870316267,
+            0.0030266400426626205,
+            -0.009028897620737553,
+            0.0063390047289431095,
+            -0.012563719414174557,
+            0.012637191452085972,
+            -0.008114579133689404,
+            0.00022883480414748192,
+            -0.0059430720284581184,
+            -0.011110605672001839,
+            -0.012653518468141556,
+            -0.0007949267164804041,
+            -0.016531210392713547,
+            0.007432921789586544,
+            0.00981259997934103,
+            0.005045080557465553,
+            0.01025343220680952,
+            -0.0036797248758375645,
+            -0.003136848099529743,
+            -0.016751626506447792,
+            -0.009984034113585949,
+            0.0073716952465474606,
+            0.01872720755636692,
+            -0.028588788583874702,
+            -0.018547609448432922,
+            0.009290131740272045,
+            0.0022898786701261997,
+            0.018776189535856247,
+            0.0158944521099329,
+            -0.0002595757250674069,
+            0.012318813242018223,
+            -0.01964152604341507,
+            -0.009184005670249462,
+            0.009690145961940289,
+            -0.003569516818970442,
+            0.01048201136291027,
+            -0.014759717509150505,
+            0.010033016093075275,
+            -0.011159587651491165,
+            -0.011053460650146008,
+            0.01385356206446886,
+            0.01738838478922844,
+            0.008873790502548218,
+            -0.001294944784604013,
+            3.383413059054874e-05,
+            -0.010506502352654934,
+            -0.024033522233366966,
+            0.004771600943058729,
+            0.027021385729312897,
+            -0.013273949734866619,
+            0.003883813973516226,
+            0.008702355436980724,
+            -0.017502672970294952,
+            -0.019821124151349068,
+            -0.012653518468141556,
+            -0.022106921300292015,
+            0.020572172477841377,
+            0.023413091897964478,
+            -0.02026195637881756,
+            0.00877582747489214,
+            -0.004620575346052647,
+            0.014049487188458443,
+            -0.01400050614029169,
+            0.010612628422677517,
+            0.0317889042198658,
+            -0.020000724121928215,
+            -0.0028654097113758326,
+            0.0036633978597819805,
+            -0.015347493812441826,
+            0.018629245460033417,
+            0.03268689662218094,
+            -0.00176945177372545,
+            0.014041324146091938,
+            0.0008138049161061645,
+            -0.009118696674704552,
+            0.01905374974012375,
+            0.011094278655946255,
+            0.005347132217139006,
+            -0.0042491331696510315,
+            -0.012212686240673065,
+            -0.03010721132159233,
+            0.016702644526958466,
+            0.0129637336358428,
+            -0.0037674831692129374,
+            -0.0011551438365131617,
+            -0.006343086715787649,
+            -0.015404638834297657,
+            0.0035123717971146107,
+            -0.013078023679554462,
+            -0.00803294312208891,
+            -0.008759500458836555,
+            0.004314441699534655,
+            -0.0034960447810590267,
+            -0.009053388610482216,
+            0.007236996199935675,
+            0.0009510547970421612,
+            -0.0010602424154058099,
+            -0.01902109570801258,
+            -0.025078458711504936,
+            -0.00995954405516386,
+            -0.016849588602781296,
+            0.0038287099450826645,
+            0.0005234883283264935,
+            -0.027184655889868736,
+            0.0053961132653057575,
+            -0.008139070123434067,
+            -0.008134988136589527,
+            0.007126788143068552,
+            0.006881881505250931,
+            -0.007437003776431084,
+            -0.010816717520356178,
+            -0.00786150898784399,
+            -0.006865554489195347,
+            0.020033378154039383,
+            0.0034348180051892996,
+            0.012082069180905819,
+            -0.010498338378965855,
+            -0.018025141209363937,
+            -0.016963878646492958,
+            0.016718972474336624,
+            -0.008971752598881721,
+            -0.02083340659737587,
+            -0.01444950234144926,
+            -0.008424794301390648,
+            -0.007665583398193121,
+            -0.01527402177453041,
+            -0.021159948781132698,
+            -0.0032103201374411583,
+            -0.011347348801791668,
+            -0.0067431009374558926,
+            -0.0028960229828953743,
+            0.0013571919407695532,
+            0.00951871182769537,
+            0.00546958576887846,
+            0.00438791373744607,
+            0.021666089072823524,
+            0.010122815147042274,
+            -0.005461421795189381,
+            0.016441410407423973,
+            -0.006183897145092487,
+            -0.004440976772457361,
+            -0.01498013362288475,
+            -0.009918726049363613,
+            0.018547609448432922,
+            -0.0055226488038897514,
+            0.008457448333501816,
+            -0.0009536059224046767,
+            -0.00576347392052412,
+            0.015674035996198654,
+            0.0129637336358428,
+            0.016718972474336624,
+            0.004771600943058729,
+            -0.00012168807734269649,
+            0.012416775338351727,
+            -0.0015745466807857156,
+            0.012555556371808052,
+            -0.0038429962005466223,
+            0.014506647363305092,
+            -0.009722800925374031,
+            -0.018988441675901413,
+            0.004157293122261763,
+            0.006265532691031694,
+            -0.012531065382063389,
+            0.009404421783983707,
+            0.007743136957287788,
+            -0.013747435994446278,
+            -0.02261306345462799,
+            -0.000106317616882734,
+            0.00014362763613462448,
+            0.008122743107378483,
+            0.008971752598881721,
+            -0.008073761127889156,
+            0.012890261597931385,
+            0.004616493359208107,
+            -0.007179851643741131,
+            -0.019657853990793228,
+            0.026009103283286095,
+            0.015608727931976318,
+            0.012269831262528896,
+            -0.052508022636175156,
+            -0.0030980713199824095,
+            0.010743245482444763,
+            -0.015951598063111305,
+            -0.026792805641889572,
+            -0.006404313258826733,
+            -0.008759500458836555,
+            -0.04290767386555672,
+            -0.009624837897717953,
+            -0.0005724696675315499,
+            -0.011388166807591915,
+            0.010400376282632351,
+            0.01573934406042099,
+            0.014612773433327675,
+            0.005489994306117296,
+            -0.007232914678752422,
+            -0.017633290961384773,
+            -0.02788672223687172,
+            0.010057506151497364,
+            -0.013380075804889202,
+            -0.006412476766854525,
+            -0.002263347152620554,
+            -0.007690073922276497,
+            0.0016112826997414231,
+            0.01691489666700363,
+            0.042483169585466385,
+            0.023037567734718323,
+            -0.012335140258073807,
+            -0.020131340250372887,
+            -0.012800462543964386,
+            -0.009192168712615967,
+            0.00633492274209857,
+            0.011592255905270576,
+            -0.004738946910947561,
+            0.0096656559035182,
+            -0.004294032696634531,
+            0.012302485294640064,
+            0.012767808511853218,
+            0.01680060662329197,
+            -2.3023791072773747e-05,
+            0.005579793825745583,
+            -0.0024694770108908415,
+            -0.01738838478922844,
+            -0.004379750229418278,
+            0.008261523209512234,
+            -0.0008872769540175796,
+            0.0019317024853080511,
+            0.014204595237970352,
+            0.019102731719613075,
+            -0.009094206616282463,
+            -0.014612773433327675,
+            -0.016098542138934135,
+            -0.005277741700410843,
+            -0.0012622905196622014,
+            0.029927613213658333,
+            0.02037624642252922,
+            -0.019266001880168915,
+            -0.018237393349409103,
+            -0.018351683393120766,
+            0.00922482367604971,
+            -0.003563394071534276,
+            -0.0064369672909379005,
+            0.01905374974012375,
+            0.023021239787340164,
+            0.017910851165652275,
+            -0.003642988856881857,
+            -0.014865843579173088,
+            0.003804219188168645,
+            0.02847449854016304,
+            0.008367649279534817,
+            0.01078406348824501,
+            0.008726846426725388,
+            -0.0011908593587577343,
+            -0.032441988587379456,
+            -0.0113146947696805,
+            0.00951871182769537,
+            -0.008481939323246479,
+            0.026564225554466248,
+            -0.016531210392713547,
+            0.009388094767928123,
+            -0.005077734589576721,
+            0.0019286411115899682,
+            -0.02365799807012081,
+            0.0014929111348465085,
+            0.010122815147042274,
+            -0.024947840720415115,
+            0.00336950970813632,
+            -0.023592690005898476,
+            -0.00013763252354692668,
+            0.008367649279534817,
+            0.0022837561555206776,
+            0.01650671847164631,
+            -0.0113146947696805,
+            0.0051838611252605915,
+            0.02099667675793171,
+            -0.002832755446434021,
+            -0.01512707769870758,
+            -0.0007842120248824358,
+            0.0005092020728625357,
+            -0.001286781276576221,
+            0.02563357912003994,
+            0.015812816098332405,
+            -0.0036756431218236685,
+            0.020882388576865196,
+            0.008677864447236061,
+            0.0019133344758301973,
+            -0.021764053031802177,
+            0.0023796779569238424,
+            -0.032899148762226105,
+            0.017437364906072617,
+            -0.011126932688057423,
+            0.012490247376263142,
+            0.00868602842092514,
+            -0.01341272983700037,
+            -0.0007388022495433688,
+            -0.0107187544927001,
+            -0.0015725058037787676,
+            -0.007934981025755405,
+            0.009779945947229862,
+            -0.004363423213362694,
+            0.008339077234268188,
+            -0.00024579971795901656,
+            -0.01476788055151701,
+            0.018694553524255753,
+            9.678283277025912e-06,
+            -0.00010925139940809458,
+            0.005441013257950544,
+            0.009935053065419197,
+            0.00786559097468853,
+            0.002009256277233362,
+            0.023870250210165977,
+            -0.031119491904973984,
+            -0.016082214191555977,
+            -0.010204450227320194,
+            0.005543057806789875,
+            -0.007922735065221786,
+            -0.018531281501054764,
+            -0.03257260471582413,
+            0.01318415068089962,
+            0.001293924287892878,
+            -0.0009806477464735508,
+            0.01258004643023014,
+            -0.00942891277372837,
+            -0.0013878053287044168,
+            -0.01606588624417782,
+            -0.014808698557317257,
+            0.006416558753699064,
+            0.0033633869607001543,
+            -0.01222901325672865,
+            -0.023739634081721306,
+            0.014759717509150505,
+            0.0007418635650537908,
+            -0.016980206593871117,
+            -0.017502672970294952,
+            0.00018457300029695034,
+            -0.004804255440831184,
+            0.02204161323606968,
+            -0.016678154468536377,
+            0.003373591462150216,
+            0.0007219648687168956,
+            0.009232986718416214,
+            -0.02889900468289852,
+            0.00716352416202426,
+            0.0006107363733462989,
+            0.0035164537839591503,
+            -0.006518602836877108,
+            -0.021013004705309868,
+            0.011371839791536331,
+            -0.0020674217958003283,
+            -0.014751553535461426,
+            -0.020751770585775375,
+            -0.0020796670578420162,
+            -0.007420676294714212,
+            -0.02724996581673622,
+            0.017910851165652275,
+            0.0009985055075958371,
+            -0.01738838478922844,
+            0.008077843114733696,
+            0.006596156861633062,
+            0.009355440735816956,
+            0.02455599047243595,
+            0.010147306136786938,
+            -0.018776189535856247,
+            -0.0008087027235887945,
+            0.006979844067245722,
+            0.009306458756327629,
+            0.0036450298503041267,
+            0.04222193360328674,
+            -0.008130906149744987,
+            -0.006579829845577478,
+            0.0066614653915166855,
+            -0.008449285291135311,
+            -0.013747435994446278,
+            0.026727497577667236,
+            0.005796127952635288,
+            0.015984252095222473,
+            0.016123032197356224,
+            -0.00945340283215046,
+            -0.005008344538509846,
+            -0.0018786393338814378,
+            0.02065380848944187,
+            0.0073798587545752525,
+            -0.005502239800989628,
+            -0.009714636951684952,
+            0.005881845485419035,
+            -0.00898808054625988,
+            0.0021653843577951193,
+            -0.009526874870061874,
+            -0.005824700463563204,
+            -0.008069679141044617,
+            0.016400592401623726,
+            -0.0003523086488712579,
+            0.027315273880958557,
+            0.020751770585775375,
+            -0.008563575334846973,
+            -0.0005683879135176539,
+            0.017208784818649292,
+            -0.005820618476718664,
+            0.015608727931976318,
+            -0.011192241683602333,
+            -0.007706400938332081,
+            0.006408394780009985,
+            0.024017194285988808,
+            0.012237177230417728,
+            -0.0056042843498289585,
+            0.0002885308349505067,
+            -0.0030348035506904125,
+            -0.010612628422677517,
+            0.013388238847255707,
+            0.015323002822697163,
+            -0.01947825588285923,
+            -0.004722619894891977,
+            -0.0016592437168583274,
+            -0.003300119424238801,
+            0.00945340283215046,
+            0.02305389568209648,
+            0.000171817431692034,
+            0.009184005670249462,
+            0.0012765767751261592,
+            0.03862996771931648,
+            -0.0053144777193665504,
+            0.03102152980864048,
+            -0.0010423846542835236,
+            -0.0059430720284581184,
+            -0.009094206616282463,
+            -0.020441556349396706,
+            -0.011486129835247993,
+            -0.00041481092921458185,
+            -0.005040998570621014,
+            -0.0004956811899319291,
+            -0.020327266305685043,
+            0.0026960158720612526,
+            0.014637264423072338,
+            0.007661501411348581,
+            0.017927179113030434,
+            -0.002889900468289852,
+            0.017437364906072617,
+            0.019690508022904396,
+            -0.01400050614029169,
+            -0.029862305149435997,
+            -0.0006515541463159025,
+            0.01498013362288475,
+            -0.01661284640431404,
+            0.019445601850748062,
+            0.004085862077772617,
+            0.0029531680047512054,
+            0.004779764451086521,
+            -0.006049198564141989,
+            0.018482301384210587,
+            0.02039257436990738,
+            0.01453930139541626,
+            -0.013167822733521461,
+            0.014057651162147522,
+            0.009771781973540783,
+            0.006600238848477602,
+            -0.01646590232849121,
+            -0.013535182923078537,
+            0.002555194543674588,
+            0.0009245232213288546,
+            0.024392718449234962,
+            -0.020588500425219536,
+            0.0016602640971541405,
+            0.010588138364255428,
+            0.01769859902560711,
+            -0.004302196204662323,
+            0.008906444534659386,
+            0.0008740111952647567,
+            0.007134951651096344,
+            -0.0035919665824621916,
+            -0.006457376293838024,
+            -0.006673710886389017,
+            -0.010392212308943272,
+            -0.0020959940738976,
+            -0.02070278860628605,
+            0.002198038622736931,
+            -0.002830714685842395,
+            0.018237393349409103,
+            -0.01018812321126461,
+            0.038140155375003815,
+            0.03399306535720825,
+            -0.0007025764207355678,
+            0.011102442629635334,
+            -0.01709449477493763,
+            -0.003945040516555309,
+            -0.021959977224469185,
+            0.01844964735209942,
+            0.016392430290579796,
+            0.012269831262528896,
+            0.005302232690155506,
+            0.017796561121940613,
+            -0.0006301248213276267,
+            0.014490320347249508,
+            0.0029654132667928934,
+            0.007971717044711113,
+            -0.002265388146042824,
+            0.003938918001949787,
+            -0.031103165820240974,
+            -0.017029186710715294,
+            -0.01138000376522541,
+            0.004796091932803392,
+            -0.006375740747898817,
+            -0.0035205355379730463,
+            0.0084737753495574,
+            -0.005902254022657871,
+            0.0004270562494639307,
+            -0.025992777198553085,
+            -0.0157148540019989,
+            -0.010318740271031857,
+            -0.004040962550789118,
+            -0.01169838197529316,
+            -0.0027694879099726677,
+            -0.0022082431241869926,
+            0.0029552089981734753,
+            0.020947696641087532,
+            -0.012808626517653465,
+            -0.016294466331601143,
+            0.004379750229418278,
+            -0.002263347152620554,
+            -0.014016833156347275,
+            -0.016898570582270622,
+            0.006665547378361225,
+            0.012098397128283978,
+            -0.0017470018938183784,
+            -0.013608654960989952,
+            0.011739199981093407,
+            -0.006628811359405518,
+            0.04663025587797165,
+            -0.013584164902567863,
+            -0.0078002819791436195,
+            0.003720542648807168,
+            0.004698128905147314,
+            -0.009396257810294628,
+            0.007330877240747213,
+            0.013020878657698631,
+            0.01830270327627659,
+            0.0095758568495512,
+            -0.008465612307190895,
+            -0.0024388637393712997,
+            0.007796200457960367,
+            -0.016147522255778313,
+            0.006404313258826733,
+            0.0008525818702764809,
+            -0.014400520361959934,
+            0.006110425107181072,
+            0.028980640694499016,
+            0.013151495717465878,
+            0.01288209855556488,
+            -0.022351829335093498,
+            -0.004094025585800409,
+            -0.009355440735816956,
+            -0.004943036008626223,
+            -0.010090161114931107,
+            0.027070365846157074,
+            -0.018155759200453758,
+            0.02651524543762207,
+            0.00898808054625988,
+            0.0017684312770143151,
+            -0.01176369097083807,
+            0.006343086715787649,
+            0.02832755446434021,
+            0.0028592869639396667,
+            -0.020000724121928215,
+            -0.013592327944934368,
+            0.006314514204859734,
+            0.01707816869020462,
+            -0.002647034591063857,
+            -0.005428767763078213,
+            0.019657853990793228,
+            0.01784554310142994,
+            -0.004334850702434778,
+            -0.00936360377818346,
+            0.0008337036124430597,
+            0.00928196869790554,
+            0.016963878646492958,
+            0.015020951628684998,
+            0.009086042642593384,
+            -0.002647034591063857,
+            0.008037025108933449,
+            -0.008800318464636803,
+            -0.016449574381113052,
+            0.007220669183880091,
+            -0.008432958275079727,
+            -0.016963878646492958,
+            0.01872720755636692,
+            0.015257694758474827,
+            0.0181394312530756,
+            0.0024592725094407797,
+            -0.016735298559069633,
+            0.022302847355604172,
+            0.005755309946835041,
+            0.002998067531734705,
+            0.02160078100860119,
+            -0.022221211344003677,
+            0.013012715615332127,
+            0.02037624642252922,
+            0.027609162032604218,
+            -0.01737205684185028,
+            -0.010906516574323177,
+            -0.004857318475842476,
+            -0.006853308994323015,
+            -0.00176945177372545,
+            -0.008253359235823154,
+            -0.012620864436030388,
+            0.0028735732194036245,
+            -0.030466407537460327,
+            0.018106777220964432,
+            -0.005102225113660097,
+            0.010669773444533348,
+            -0.019266001880168915,
+            0.011363675817847252,
+            -0.0035838030744343996,
+            0.011135096661746502,
+            0.0019612954929471016,
+            -0.030041903257369995,
+            -0.008571738377213478,
+            0.015388311818242073,
+            -0.0030368445441126823,
+            -0.013641309924423695,
+            0.02006603218615055,
+            0.00018942011229228228,
+            -0.022270193323493004,
+            -0.004314441699534655,
+            -0.006934944540262222,
+            0.002193956868723035,
+            -0.0029613315127789974,
+            -0.0052981507033109665,
+            -0.030188847333192825,
+            0.020033378154039383,
+            0.009559528902173042,
+            -0.03072764165699482,
+            -0.003255219664424658,
+            -0.014996460638940334,
+            -0.0025184585247188807,
+            0.008881953544914722,
+            -0.004000144544988871,
+            -0.015551582910120487,
+            0.01784554310142994,
+            0.022711025550961494,
+            -0.0023164101876318455,
+            -0.016057724133133888,
+            -0.0090697156265378,
+            -0.027494871988892555,
+            0.030825603753328323,
+            -0.0066696288995444775,
+            0.0022878379095345736,
+            -0.005322641227394342,
+            0.009388094767928123,
+            0.004228724166750908,
+            0.0157148540019989,
+            0.006081852596253157,
+            -0.002647034591063857,
+            0.026596879586577415,
+            -0.01964152604341507,
+            0.009567692875862122,
+            -1.588864870427642e-05,
+            -0.003308282932266593,
+            0.01890680566430092,
+            0.0007939062779769301,
+            0.021649762988090515,
+            -0.008571738377213478,
+            -0.0001475818717153743,
+            -0.0034388997592031956,
+            0.010898353531956673,
+            -0.0028450009413063526,
+            -0.003465431509539485,
+            -0.001531688030809164,
+            0.011118769645690918,
+            0.01013097818940878,
+            0.003804219188168645,
+            -0.01737205684185028,
+            0.003142970846965909,
+            -0.008547247387468815,
+            -0.007028825581073761,
+            0.01302904263138771,
+            -0.011192241683602333,
+            -0.0018990482203662395,
+            -0.03160930797457695,
+            0.016286302357912064,
+            0.0533243790268898,
+            0.008547247387468815,
+            -0.014433175325393677,
+            -0.0018806802108883858,
+            -0.019755816087126732,
+            0.01535565685480833,
+            0.011004479601979256,
+            -0.016816934570670128,
+            -0.011984107084572315,
+            -0.012310649268329144,
+            0.007649256382137537,
+            0.021306892856955528,
+            -0.006522684823721647,
+            -0.008579902350902557,
+            0.025372346863150597,
+            -0.011788181029260159,
+            -0.010882026515901089,
+            -0.01661284640431404,
+            -0.01635977439582348,
+            0.00241641397587955,
+            0.019902760162949562,
+            -0.0169475506991148,
+            0.021176276728510857,
+            0.00936360377818346,
+            0.0010133018950000405,
+            0.003624620847404003,
+            0.008588065393269062,
+            0.0024878450203686953,
+            0.017878197133541107,
+            0.003883813973516226,
+            -0.01439235731959343,
+            0.0021408938337117434,
+            -0.0030286810360848904,
+            0.004624656867235899,
+            3.4320437407586724e-06,
+            -0.002749078907072544,
+            -0.0065635028295218945,
+            0.003710338147357106,
+            -0.021306892856955528,
+            0.021502818912267685,
+            -0.022678371518850327,
+            -0.004245051648467779,
+            0.005555302836000919,
+            -0.005118552595376968,
+            0.013469874858856201,
+            0.018792515620589256,
+            0.01423724927008152,
+            0.008702355436980724,
+            0.01462093647569418,
+            -0.011331021785736084,
+            0.02323349379003048,
+            -0.013282112777233124,
+            0.0005816536722704768,
+            0.012939243577420712,
+            0.01755165494978428,
+            0.018792515620589256,
+            -0.006245123688131571,
+            -0.010384049266576767,
+            -0.0002946535241790116,
+            0.0129637336358428,
+            -0.022286519408226013,
+            -0.012269831262528896,
+            -0.01386172603815794,
+            0.011486129835247993,
+            -0.004743028432130814,
+            0.025453981012105942,
+            -0.00633492274209857,
+            -0.014653591439127922,
+            -0.015918942168354988,
+            0.014196431264281273,
+            -0.048752784729003906,
+            0.0030348035506904125,
+            -0.0024919267743825912,
+            0.03043375350534916,
+            -0.027021385729312897,
+            0.01738838478922844,
+            -0.0028490826953202486,
+            -0.009355440735816956,
+            0.015429128892719746,
+            -0.015992414206266403,
+            -0.0046613928861916065,
+            -0.0016806729836389422,
+            -0.007539048325270414,
+            -0.00112248957157135,
+            0.017616963014006615,
+            -0.015845470130443573,
+            0.01282495353370905,
+            -0.00989423505961895,
+            0.0027021386194974184,
+            -0.004526694305241108,
+            -0.007751300930976868,
+            0.0196088720113039,
+            0.034548189491033554,
+            -0.007053316105157137,
+            0.0014990337658673525,
+            -0.022547753527760506,
+            0.013192313723266125,
+            -0.006730855442583561,
+            0.02351105399429798,
+            -0.005432849284261465,
+            -0.016898570582270622,
+            -0.013976015150547028,
+            -0.009739127941429615,
+            0.00839214026927948,
+            -0.022433463484048843,
+            -0.011110605672001839,
+            -0.00922482367604971,
+            -0.002573562553152442,
+            -0.018694553524255753,
+            0.004865481983870268,
+            -0.002426618244498968,
+            -0.012506574392318726,
+            -0.008816645480692387,
+            0.01048201136291027,
+            -0.009347276762127876,
+            0.003077662317082286,
+            -0.0011929002357646823,
+            -0.011453475803136826,
+            -0.02369065210223198,
+            0.0158944521099329,
+            0.0001974561164388433,
+            -0.010359558276832104,
+            0.008408467285335064,
+            -0.0018276170594617724,
+            0.001211268245242536,
+            0.016963878646492958,
+            -0.0020664012990891933,
+            0.003467472270131111,
+            0.023478399962186813,
+            0.004057289566844702,
+            -0.010971825569868088,
+            -0.00025906547671183944,
+            0.013535182923078537,
+            -0.005118552595376968,
+            -0.010008525103330612,
+            1.6646010408294387e-05,
+            -0.02984597720205784,
+            -0.008351322263479233,
+            0.006094098091125488,
+            0.018629245460033417,
+            -0.010620792396366596,
+            -0.015853634104132652,
+            0.026319319382309914,
+            -0.006359413731843233,
+            0.019070077687501907,
+            0.002889900468289852,
+            0.0025694805663079023,
+            0.01724144071340561,
+            0.013461710885167122,
+            0.00786150898784399,
+            -0.013927034102380276,
+            -0.02429475635290146,
+            0.0016398552106693387,
+            -0.018090449273586273,
+            0.010579974390566349,
+            -0.011028970591723919,
+            0.0032756286673247814,
+            -0.02398454025387764,
+            -0.012237177230417728,
+            0.004934872500598431,
+            -0.020343592390418053,
+            -0.00040588201954960823,
+            -0.0056940834037959576,
+            0.022237539291381836,
+            -0.005310396198183298,
+            0.008669701404869556,
+            0.0018255761824548244,
+            -0.012237177230417728,
+            0.0009877908742055297,
+            0.009469729848206043,
+            -0.029192892834544182,
+            -0.0007127808639779687,
+            -0.003138889092952013,
+            0.0011418780777603388,
+            0.005351213738322258,
+            0.009755454957485199,
+            0.007322713732719421,
+            0.0027062203735113144,
+            0.007253323681652546,
+            0.010449357330799103,
+            -0.00022041612828616053,
+            0.0197884701192379,
+            0.018172085285186768,
+            -0.007351286243647337,
+            0.008710519410669804,
+            0.024817224591970444,
+            -0.013159659691154957,
+            -0.008718682453036308,
+            0.023266147822141647,
+            -0.006461458280682564,
+            -0.0011122850701212883,
+            -0.01483318954706192,
+            0.00815947912633419,
+            0.0017367975087836385,
+            0.028131630271673203,
+            0.019657853990793228,
+            0.005820618476718664,
+            0.000639819074422121,
+            0.015788326039910316,
+            -0.013020878657698631,
+            0.0035286990460008383,
+            0.015461783856153488,
+            -0.003940958995372057,
+            -0.019821124151349068,
+            0.013878053054213524,
+            0.009502384811639786,
+            -0.003228688146919012,
+            0.01551892887800932,
+            0.011412657797336578,
+            -0.030841931700706482,
+            -0.0011898389784619212,
+            0.010498338378965855,
+            -0.0021449755877256393,
+            -0.008751336485147476,
+            -0.00014273475972004235,
+            -0.012384121306240559,
+            0.021388528868556023,
+            -0.004408322740346193,
+            0.012277995236217976,
+            -0.00815947912633419,
+            -0.017127150669693947,
+            -0.011559601873159409,
+            0.007914572022855282,
+            -0.010547320358455181,
+            -0.0006714528426527977,
+            -0.004022594541311264,
+            -0.005779800936579704,
+            0.017747581005096436,
+            0.004706292413175106,
+            -0.0008964610169641674,
+            -0.0018398624379187822,
+            0.007983962073922157,
+            -0.006979844067245722,
+            -0.008726846426725388,
+            0.004816500470042229,
+            -0.012833117507398129,
+            0.003981776535511017,
+            0.005543057806789875,
+            0.01506176870316267,
+            -0.014857680536806583,
+            -0.021159948781132698,
+            0.007020662073045969,
+            0.02695607766509056,
+            -0.02516009286046028,
+            -0.005232842173427343,
+            0.006245123688131571,
+            0.002093953313305974,
+            0.006914535537362099,
+            -0.006934944540262222,
+            -0.007208424154669046,
+            -0.03657275065779686,
+            -0.0020745648071169853,
+            0.0009005427709780633,
+            0.008963589556515217,
+            -0.008049271069467068,
+            -0.0136494729667902,
+            -0.011135096661746502,
+            0.002410291228443384,
+            0.007134951651096344,
+            -0.02204161323606968,
+            0.0049593630246818066,
+            0.009608510881662369,
+            -0.019559891894459724,
+            0.025127438828349113,
+            -0.010890189558267593,
+            -6.964537897147238e-05,
+            -0.00995954405516386,
+            0.021633435040712357,
+            -0.013486201874911785,
+            -0.018368011340498924,
+            -0.0006898208521306515,
+            0.008355404250323772,
+            0.011926962062716484,
+            -0.0017663904000073671,
+            -0.011371839791536331,
+            0.0003752686607185751,
+            0.01646590232849121,
+            0.019135385751724243,
+            0.002744997153058648,
+            -0.0024837632663547993,
+            -0.01019628718495369,
+            -0.009592183865606785,
+            -0.0073961857706308365,
+            -0.0007449248805642128,
+            0.002091912319883704,
+            0.02035992033779621,
+            0.013469874858856201,
+            0.0024837632663547993,
+            -0.015135240741074085,
+            -0.005257333163172007,
+            0.008939098566770554,
+            0.009412584826350212,
+            -0.011690218932926655,
+            -0.00898808054625988,
+            0.002659279853105545,
+            0.02249877341091633,
+            0.017633290961384773,
+            -0.0014000505907461047,
+            0.0032878739293664694,
+            -0.0016592437168583274,
+            0.012506574392318726,
+            -0.007555375341325998,
+            -0.010269759222865105,
+            -0.006943108048290014,
+            -0.014645427465438843,
+            -0.010800390504300594,
+            -0.0023062059190124273,
+            -0.010179960168898106,
+            0.022629389539361,
+            0.007988044060766697,
+            0.009779945947229862,
+            0.008265605196356773,
+            0.010269759222865105,
+            0.002087830565869808,
+            0.025715215131640434,
+            -0.004424649756401777,
+            -0.011984107084572315,
+            0.007236996199935675,
+            -0.022368155419826508,
+            -0.021535472944378853,
+            0.008257441222667694,
+            0.006367577239871025,
+            0.03595232218503952,
+            -0.026890767738223076,
+            0.0052981507033109665,
+            -0.009779945947229862,
+            -0.010400376282632351,
+            0.007996207103133202,
+            -0.0015439334092661738,
+            -0.004245051648467779,
+            0.022368155419826508,
+            0.00726556871086359,
+            0.0035797213204205036,
+            0.0032817514147609472,
+            0.019837452098727226,
+            -0.005685919895768166,
+            0.01573934406042099,
+            0.0259274672716856,
+            0.024621298536658287,
+            0.009388094767928123,
+            -0.009469729848206043,
+            0.02785406820476055,
+            0.0002895512734539807,
+            -0.0034327772445976734,
+            0.007220669183880091,
+            -0.00305113079957664,
+            -0.0009556467994116247,
+            -0.010033016093075275,
+            0.021764053031802177,
+            -0.001455154619179666,
+            0.004359341226518154,
+            0.02081707864999771,
+            -7.774515688652173e-05,
+            0.006220633164048195,
+            0.01057181041687727,
+            0.002181711606681347,
+            0.003396041225641966,
+            0.010775899514555931,
+            -0.039315707981586456,
+            -0.0006571666453965008,
+            -0.025600925087928772,
+            -0.0018286375561729074,
+            0.0007357408758252859,
+            -0.01266984548419714,
+            0.013600491918623447,
+            -0.002015379024669528,
+            -0.010751409456133842,
+            -0.00029337796149775386,
+            0.021649762988090515,
+            0.005416522268205881,
+            0.004071575589478016,
+            -0.004469549283385277,
+            0.018792515620589256,
+            -0.006955353543162346,
+            -0.017861871048808098,
+            -0.012759645469486713,
+            -0.012367794290184975,
+            -0.01918436773121357,
+            -0.01214737817645073,
+            -0.0024551907554268837,
+            -0.002667443361133337,
+            0.004738946910947561,
+            0.002481722505763173,
+            -0.003228688146919012,
+            -0.02070278860628605,
+            0.015037278644740582,
+            -0.01430255826562643,
+            0.012873934581875801,
+            -0.011053460650146008,
+            0.005963481031358242,
+            -0.004122598096728325,
+            0.012726990506052971,
+            -0.021404854953289032,
+            0.02070278860628605,
+            -0.01536382082849741,
+            0.0038532004691660404,
+            0.019592545926570892,
+            0.006551257334649563,
+            -0.008118661120533943,
+            -0.013404566794633865,
+            -0.022074267268180847,
+            0.021633435040712357,
+            -0.011633073911070824,
+            -0.018237393349409103,
+            -0.004608329851180315,
+            0.009649328887462616,
+            -0.017470018938183784,
+            -0.007367613259702921,
+            0.01707816869020462,
+            -0.0018949664663523436,
+            0.012980061583220959,
+            0.023004913702607155,
+            -0.0052981507033109665,
+            0.0036572751123458147,
+            -0.009167678654193878,
+            0.0010286086471751332,
+            -0.02470293454825878,
+            0.013527019880712032,
+            0.008971752598881721,
+            0.006269614677876234,
+            -0.013935198076069355,
+            -0.013306603766977787,
+            0.0036531933583319187,
+            0.004498121794313192,
+            0.011437147855758667,
+            0.009420748800039291,
+            0.004220560658723116,
+            -0.013918871060013771,
+            -0.027788760140538216,
+            -0.027070365846157074,
+            0.03595232218503952,
+            -0.0005002731923013926,
+            -0.020947696641087532,
+            0.03399306535720825,
+            0.016114868223667145,
+            0.0022980421781539917,
+            -0.012449429370462894,
+            -0.004314441699534655,
+            0.024621298536658287,
+            0.016400592401623726,
+            0.010465684346854687,
+            -0.0004742518358398229,
+            0.005571629852056503,
+            0.0016469983384013176,
+            0.03012353740632534,
+            0.015323002822697163,
+            0.0009459525463171303,
+            -0.038597315549850464,
+            0.011943289078772068,
+            -0.007547211833298206,
+            0.0013408648082986474,
+            0.006702283397316933,
+            0.017780235037207603,
+            0.011935125105082989,
+            0.015567909926176071,
+            0.014865843579173088,
+            -0.003065417055040598,
+            0.0029511270113289356,
+            0.0006163488142192364,
+            -0.009624837897717953,
+            -0.001767410896718502,
+            -0.008881953544914722,
+            -0.005714492406696081,
+            -0.002920513739809394,
+            0.005592038854956627,
+            0.007657419890165329,
+            -0.003983817528933287,
+            0.028049994260072708,
+            -0.0006433906382881105,
+            -0.017600636929273605,
+            -0.009061552584171295,
+            -0.03611559048295021,
+            0.00434301421046257,
+            -0.030939893797039986,
+            -0.011257549747824669,
+            0.0033674687147140503,
+            -0.020457882434129715,
+            -0.024768242612481117,
+            0.0010306495241820812,
+            0.014335212297737598,
+            0.007392104249447584,
+            0.0033327736891806126,
+            -0.01004934310913086,
+            0.003816464450210333,
+            -0.001855169073678553,
+            -0.006461458280682564,
+            0.028833696618676186,
+            0.019559891894459724,
+            0.020457882434129715,
+            -0.01190247107297182,
+            0.019298657774925232,
+            0.005845109466463327,
+            0.02035992033779621,
+            0.010465684346854687,
+            -0.0048695639707148075,
+            0.0074859848245978355,
+            0.009567692875862122,
+            -0.006155324634164572,
+            -0.021519144997000694,
+            0.013192313723266125,
+            -0.0010837126756086946,
+            -0.0052083516493439674,
+            -0.016269976273179054,
+            -0.003534821793437004,
+            -0.015812816098332405,
+            0.010547320358455181,
+            -0.017159804701805115,
+            0.012498411349952221,
+            -0.000802069844212383,
+            0.00168475485406816,
+            -0.015469946898519993,
+            -0.013951525092124939,
+            -0.0012439225101843476,
+            -0.01244126632809639,
+            -0.008971752598881721,
+            0.006163488142192364,
+            -0.007539048325270414,
+            -0.008318668231368065,
+            -0.00535937724635005,
+            -0.011437147855758667,
+            -0.010882026515901089,
+            0.004367504734545946,
+            -0.0010653446661308408,
+            0.020931368693709373,
+            0.00011671340325847268,
+            -0.005824700463563204,
+            0.01190247107297182,
+            0.02787039615213871,
+            0.0016704685986042023,
+            0.004583839327096939,
+            -0.014261740259826183,
+            -0.011118769645690918,
+            -0.02279266156256199,
+            0.016572028398513794,
+            0.0049675265327095985,
+            -0.0344502255320549,
+            0.01438419334590435,
+            0.00824927818030119,
+            0.0015102586476132274,
+            -0.006698201410472393,
+            -0.005334886722266674,
+            -0.005608365871012211,
+            0.006306350696831942,
+            -0.0022837561555206776,
+            0.0005107327597215772,
+            0.0035919665824621916,
+            0.034678805619478226,
+            -0.0040674940682947636,
+            -0.007228832691907883,
+            0.00981259997934103,
+            0.0005130287609063089,
+            0.020196648314595222,
+            -0.005518566817045212,
+            0.0037572786677628756,
+            -0.019282329827547073,
+            -0.016563864424824715,
+            0.0018429238116368651,
+            -0.01722511276602745,
+            0.023282473906874657,
+            -0.005392031744122505,
+            -0.01347803883254528,
+            -0.006906372029334307,
+            -0.020115012302994728,
+            0.012637191452085972,
+            0.02310287579894066,
+            0.007016580086201429,
+            0.01033506728708744,
+            0.012131051160395145,
+            -0.02444170042872429,
+            -0.020457882434129715,
+            0.012171869166195393,
+            0.0013571919407695532,
+            -0.008449285291135311,
+            -0.005404277238994837,
+            -0.001783737912774086,
+            0.007783954963088036,
+            -0.0032633834052830935,
+            0.007881917990744114,
+            0.06491663306951523,
+            -0.01093100756406784,
+            -0.002987863030284643,
+            -0.02966637909412384,
+            -0.004396077245473862,
+            -0.006628811359405518,
+            0.0056287748739123344,
+            0.009151351638138294,
+            -0.023135529831051826,
+            0.009232986718416214,
+            -0.030025575309991837,
+            -0.010628955438733101,
+            0.00824111420661211,
+            0.0028490826953202486,
+            -0.01220452319830656,
+            -0.0009592183632776141,
+            -0.013665799982845783,
+            0.0006107363733462989,
+            -0.009412584826350212,
+            -0.00576347392052412,
+            0.012718827463686466,
+            -0.02321716584265232,
+            0.006298186723142862,
+            0.007592111360281706,
+            -0.006514521315693855,
+            -0.013722945004701614,
+            -0.017861871048808098,
+            -0.011967780068516731,
+            -0.013314766809344292,
+            0.002996026771143079,
+            0.01190247107297182,
+            -0.02607441134750843,
+            0.012922916561365128,
+            -0.0037389106582850218,
+            0.00945340283215046,
+            0.004404240753501654,
+            0.013469874858856201,
+            0.018074123188853264,
+            0.009355440735816956,
+            0.0021429345943033695,
+            -0.008024780079722404,
+            0.012702500447630882,
+            -0.01993541419506073,
+            -0.01635161228477955,
+            0.013837235048413277,
+            0.012016761116683483,
+            0.003061335301026702,
+            0.009828926995396614,
+            0.0020959940738976,
+            0.011886144056916237,
+            0.010775899514555931,
+            -0.019739490002393723,
+            -0.0006648199632763863,
+            0.0007326795603148639,
+            0.03280118480324745,
+            -0.0005938990507274866,
+            0.008857463486492634,
+            0.004420568235218525,
+            0.013159659691154957,
+            -0.012784135527908802,
+            -0.01769859902560711,
+            0.0008087027235887945,
+            -0.01057181041687727,
+            0.021257910877466202,
+            -0.008192133158445358,
+            0.006216551177203655,
+            -0.018612917512655258,
+            -0.027086693793535233,
+            0.006392067763954401,
+            -0.016261812299489975,
+            -0.018221067264676094,
+            0.01317598670721054,
+            0.0077268099412322044,
+            0.0017572063952684402,
+            -0.005098143592476845,
+            -0.020457882434129715,
+            -0.009837090037763119,
+            0.028441844508051872,
+            0.011102442629635334,
+            -0.004120557103306055,
+            0.009584019891917706,
+            -0.009094206616282463,
+            -0.012490247376263142,
+            -0.0017541450215503573,
+            0.0036715613678097725,
+            0.008261523209512234,
+            0.005645101889967918,
+            -0.01386172603815794,
+            -0.00029873527819290757,
+            -0.012400448322296143,
+            -0.021225256845355034,
+            -0.008906444534659386,
+            0.016751626506447792,
+            0.007820690982043743,
+            -0.0038960592355579138,
+            0.015812816098332405,
+            -0.004404240753501654,
+            0.002189875114709139,
+            0.014294394291937351,
+            0.005334886722266674,
+            -0.004285869188606739,
+            -0.0095758568495512,
+            -0.004612411838024855,
+            0.004043003544211388,
+            -0.0025960123166441917,
+            0.006457376293838024,
+            0.017192458733916283,
+            0.0072900597006082535,
+            -0.013755599036812782,
+            0.013135168701410294,
+            0.0007791098323650658,
+            0.0049675265327095985,
+            -0.03699725493788719,
+            -0.003247056156396866,
+            0.015796490013599396,
+            -0.013020878657698631,
+            0.02501314878463745,
+            0.005400195252150297,
+            0.022743679583072662,
+            0.015233203768730164,
+            0.008669701404869556,
+            -0.001542912912555039,
+            0.00464098434895277,
+            0.02759283408522606,
+            -0.003151134354993701,
+            -0.008637047372758389,
+            -0.0020602785516530275,
+            0.009241150692105293,
+            0.0002063850115519017,
+            0.003000108525156975,
+            0.008281932212412357,
+            -0.03562577813863754,
+            0.011706545948982239,
+            -0.019576217979192734,
+            0.0024694770108908415,
+            0.003459308762103319,
+            -0.007183933164924383,
+            -0.014865843579173088,
+            -0.005857354495674372,
+            0.022841641679406166,
+            -0.003224606392905116,
+            0.021078312769532204,
+            0.002885818714275956,
+            0.016106704249978065,
+            -0.011575928889214993,
+            -0.009208496659994125,
+            -0.0004130251472815871,
+            -0.005351213738322258,
+            -0.007539048325270414,
+            0.010596301406621933,
+            0.0034409407526254654,
+            0.005534893833100796,
+            0.002265388146042824,
+            0.0036654386203736067,
+            -0.0036878883838653564,
+            0.02773977816104889,
+            0.003563394071534276,
+            0.007996207103133202,
+            -0.011567764915525913,
+            0.014082142151892185,
+            -0.017029186710715294,
+            0.0070778070949018,
+            0.005575711838901043,
+            -0.026156047359108925,
+            0.016849588602781296,
+            0.004775682929903269,
+            0.006628811359405518,
+            0.013812744058668613,
+            0.01618834026157856,
+            -0.010408539324998856,
+            -0.014816862531006336,
+            0.024049850180745125,
+            -0.0032041973900049925,
+            -0.015225040726363659,
+            -0.006775755435228348,
+            4.438935866346583e-05,
+            0.006559420842677355,
+            0.00373074715025723,
+            0.0013020879123359919,
+            -0.0006204306264407933,
+            -0.010743245482444763,
+            0.004645065870136023,
+            -0.01934763789176941,
+            -0.0007163524278439581,
+            -0.016449574381113052,
+            -0.015633217990398407,
+            0.018678225576877594,
+            -0.01830270327627659,
+            0.009633000940084457,
+            0.0012275953777134418,
+            0.028588788583874702,
+            -0.011706545948982239,
+            -0.012155541218817234,
+            0.0015102586476132274,
+            0.010849371552467346,
+            0.0067512644454836845,
+            -0.010033016093075275,
+            -0.007583947852253914,
+            -0.014180104248225689,
+            -0.010163633152842522,
+            0.015502601861953735,
+            -0.014498483389616013,
+            -0.014343376271426678,
+            0.008914608508348465
+          ],
+          "embedding": [
+            -0.006359657085039302,
+            -0.024594786241399234,
+            -0.02553538924848095,
+            0.005360188465088291,
+            0.012801870977589839,
+            0.021680505862506104,
+            -0.016668048966001273,
+            0.06860852474581301,
+            -0.007122272841157724,
+            0.03471775000705488,
+            -0.009004297848861675,
+            0.01059974144678391,
+            0.02565604599426148,
+            -0.022544127646041287,
+            0.010616674976540785,
+            0.013658194823359737,
+            0.016400329743363844,
+            0.02858546964917478,
+            0.0067612676434693686,
+            -0.04174737193606879,
+            0.03184155110488616,
+            -0.02223105127585514,
+            -0.05506239667974977,
+            -0.0015307551410555752,
+            -0.013169788751379287,
+            0.017681749722384007,
+            0.01682706317229353,
+            0.019652295823816928,
+            -0.05623980369271269,
+            0.030698582851533197,
+            0.02286327124296613,
+            0.016019032539689178,
+            0.005508113353872315,
+            8.58985091215149e-05,
+            -0.019206292853495906,
+            -0.018439290483495144,
+            0.03475023176950151,
+            0.01018612942141759,
+            -0.0252551158857426,
+            0.0334562130666827,
+            0.020462153537485785,
+            0.02260508249714211,
+            -0.03323524989392951,
+            -0.002094251009530707,
+            0.015177442347686773,
+            -0.00018698464350038317,
+            -0.0006788648344273403,
+            0.0007777616893581039,
+            0.013352398615059647,
+            -0.030751111528744813,
+            -0.0317733818445192,
+            -0.006729064081710058,
+            0.009413204899986514,
+            -0.005712894407943592,
+            -0.038961826880527066,
+            -0.002827692206530345,
+            0.017863300537973137,
+            0.07918455388563496,
+            0.048967158919816366,
+            0.018002566325350088,
+            0.040264962620950695,
+            -0.019715100359659787,
+            -0.02211350500879408,
+            0.004918148608475574,
+            0.01062585305866606,
+            -0.014478993521696027,
+            0.0034983131226364672,
+            0.000766228629514787,
+            0.009667268771502067,
+            -0.01558439890697323,
+            -0.04984718709870083,
+            0.029613955432821887,
+            -0.009686854114630624,
+            -0.015279319238124568,
+            0.007460629676325322,
+            -0.006235979350809034,
+            0.012514387473717085,
+            0.02800435263741497,
+            -0.008688925946672778,
+            -0.01236301543374469,
+            0.029755705020188972,
+            0.01158037219446542,
+            -0.005242392206024511,
+            0.009982679027728814,
+            0.052414361714483264,
+            -0.018788500381726227,
+            -0.020335831196027204,
+            -0.08583959943356621,
+            0.029309228216812346,
+            -0.0249285282010673,
+            -0.018929396555426307,
+            0.007710663193334574,
+            0.03096798035958051,
+            -0.017089977064591828,
+            -0.028015107605405817,
+            0.06740703271128606,
+            0.009398795412354399,
+            0.01118769022826835,
+            -0.03041417460855722,
+            0.014776002883150835,
+            0.004319194067673161,
+            0.022507636639041203,
+            0.0053562788646786845,
+            0.007548072258943124,
+            0.011123374308563128,
+            -0.006924362115585165,
+            -0.011118507147092024,
+            0.009563232643295304,
+            0.00657178473678069,
+            0.08313027447380966,
+            -0.004239370156003264,
+            -0.00930406163946907,
+            -0.039202672821898946,
+            0.009761029930556298,
+            -0.02827698465424993,
+            -0.037914533246900244,
+            -0.04496050911855815,
+            -0.017711307900236353,
+            0.01934435542287117,
+            0.004335173027873153,
+            0.03800509969714242,
+            0.031198127357716314,
+            0.031311207941680276,
+            -0.012184553894588813,
+            -0.058985863670856876,
+            0.041442017057707635,
+            -0.012456189292721711,
+            -0.0040966351720394135,
+            0.010849179519959421,
+            0.005243123049347636,
+            0.0011495799963862647,
+            0.003534183676934487,
+            -0.03139340512003631,
+            -0.023876261883905306,
+            0.008382957569752911,
+            -0.004778283217011976,
+            0.03703067589888386,
+            0.06888630688839406,
+            -0.03993532296478411,
+            -0.009472167678222372,
+            0.025246872942105057,
+            -0.01524567912700368,
+            0.01595917939878582,
+            -0.003736723164442467,
+            -0.030836797784967753,
+            0.0026246446163368533,
+            0.008894300082148884,
+            0.08342856279154155,
+            -0.0007476580521704578,
+            0.025149116238550588,
+            0.004972542633631468,
+            0.034495632546556576,
+            -0.009173329764686424,
+            0.014009381543772768,
+            0.004427261668151028,
+            -0.013426138607902904,
+            -0.022234568880321808,
+            0.01976282364689966,
+            -0.03592433789490043,
+            -0.010918278220831188,
+            0.013527180289641174,
+            -0.013023113052242677,
+            -0.008304770302495363,
+            0.003677063799167178,
+            0.0013981822663219421,
+            0.047001163242600175,
+            0.01828332714887986,
+            0.002483375662343689,
+            -0.06820498187523893,
+            0.05295312038995555,
+            0.025038753504342162,
+            0.025286870198436735,
+            -0.0029544808216316115,
+            0.029982752196635428,
+            0.0005600481523874767,
+            0.0128290258663539,
+            0.009023805227424919,
+            -0.0019219166367750596,
+            -0.04482843936478531,
+            0.04523479691409549,
+            -0.004678574027241186,
+            0.010486850804218904,
+            0.006736229865018014,
+            0.02134813656476443,
+            0.028083972673690036,
+            0.016857413440125445,
+            0.0189082447820146,
+            0.03843690613820022,
+            -0.011922027011229613,
+            0.016193433524453224,
+            0.03828207911467065,
+            -0.0013055128404331705,
+            0.043198163443700956,
+            0.024025562893260542,
+            -0.001257851946749882,
+            -0.011726713433348844,
+            -0.021907984099443396,
+            0.02752198420597673,
+            0.004431866336999851,
+            0.015798806714413912,
+            -0.012910782191382367,
+            -0.03180549180206364,
+            0.004550247372715741,
+            -0.022778272841118514,
+            0.031242218240017532,
+            -0.006106984949693483,
+            -0.006259993190497421,
+            0.02187449047033339,
+            0.009334146847221406,
+            -0.009381278960016145,
+            -0.020888853823931134,
+            -0.007502374908776156,
+            -0.010390018386728482,
+            -0.04316693435221676,
+            -0.006274513730720055,
+            0.019152443937064258,
+            -0.01668984824845791,
+            0.018573194036505578,
+            0.0035035262183249925,
+            -0.019779911381789077,
+            -0.0027495798165085993,
+            0.02425481236315913,
+            0.006857723545270935,
+            -0.014733535699087731,
+            -0.026326146471941727,
+            -0.019906660231056833,
+            -0.001033294559612807,
+            0.0159154063026895,
+            0.035649534442431415,
+            -0.026210592096240003,
+            0.014397127229517285,
+            0.005114311681443673,
+            -0.0033104236005207077,
+            0.00324988235167305,
+            0.03764004154553922,
+            -0.017822437760737605,
+            -0.008579701063708334,
+            -0.00032410984200965487,
+            0.018450351071676443,
+            0.017283379764846195,
+            0.006384411611332382,
+            -0.005115090309123598,
+            -0.030590047062583125,
+            -0.011776211359193503,
+            0.01183623914060493,
+            0.0016564847796208331,
+            -0.009530766634361575,
+            -0.004810598258154945,
+            0.03876242515526699,
+            0.027076785774401667,
+            0.005324763136165989,
+            -0.03416412480913939,
+            -0.007943016733586697,
+            0.002114307650456928,
+            -0.02961449347300628,
+            0.05142732770518828,
+            0.05566026433842563,
+            -0.0030729309664974704,
+            -0.00938186948393074,
+            -0.04547908981236149,
+            0.0651216359692231,
+            -0.01813722445944094,
+            -0.0482650853560993,
+            0.044045155892545554,
+            -0.014272653003441165,
+            0.013740652306746851,
+            0.04563732439184395,
+            0.02126513340521316,
+            -0.0054845632439384905,
+            -0.011635625101262898,
+            0.030992082069685376,
+            -0.010051935721068779,
+            -0.050967258971854425,
+            -0.009827942095033494,
+            0.013373878556654464,
+            -0.013017627321534534,
+            -0.009361001542607812,
+            -0.01925867665425919,
+            0.020329052915436022,
+            0.023020741670660188,
+            0.019387043947695172,
+            0.005628632318152202,
+            0.018451555161952665,
+            0.016226845400848455,
+            -0.01384033112836216,
+            0.004762589697426251,
+            -0.012718476561725138,
+            -0.024222765127494463,
+            -0.037487144314010595,
+            -0.019971524166659278,
+            -0.015034588610516958,
+            0.009483936496215338,
+            -0.014301924283553591,
+            -0.0024139665139412125,
+            0.013361045353529458,
+            -0.0005870798721107379,
+            -0.011192319340879789,
+            0.013927695434391159,
+            0.021337539669342293,
+            -0.027300952495815437,
+            -0.04765670695199488,
+            -0.00561706224251562,
+            0.0082776639906139,
+            -0.0030984973749633724,
+            -0.026269016057688548,
+            0.007040524859505573,
+            0.0017859008072463442,
+            0.020091542517503808,
+            -0.018716761130867725,
+            0.0035283051960672963,
+            0.008519383548064053,
+            -0.015247417050336866,
+            -0.016917626868166634,
+            -0.018580550690103854,
+            -0.024697180580294884,
+            0.034202028316737125,
+            0.0204395003463227,
+            0.030392895534267717,
+            -0.04527669147753553,
+            -0.018359428092075348,
+            0.05494107499558624,
+            0.015234754907198422,
+            -0.05066008106884831,
+            0.031139824660661085,
+            0.005016615710206006,
+            0.0019000453624929544,
+            -0.006094364589229031,
+            0.0637324238017742,
+            -0.02711381489292883,
+            -0.02187418906973867,
+            0.004601744864532875,
+            -0.04141121277496072,
+            -0.026689585608019588,
+            -0.005568100750837504,
+            0.01291019036002459,
+            -0.031853041426355644,
+            -0.02815800259372657,
+            -0.007964861641790374,
+            -0.010933515363060793,
+            0.013726639077745378,
+            -0.023956800669647334,
+            -0.015672058590455337,
+            -0.023642025374517057,
+            -0.01288435863544158,
+            -0.0042516881552192385,
+            -0.011162214160460695,
+            -0.02412733256710621,
+            0.010088528028211243,
+            0.03310175458095482,
+            0.023064360903748363,
+            0.001996159684818467,
+            0.009478229572933101,
+            0.014478772877286076,
+            0.00371783700962415,
+            0.007864103610320116,
+            -0.048511515124609,
+            -0.031398754206534776,
+            0.020227872561659397,
+            0.012159811587094092,
+            0.0172163448580163,
+            0.009578952796095119,
+            -0.04154545961112031,
+            -0.05095080393682759,
+            0.02968486144855133,
+            -0.03932395023720674,
+            -0.02063219739061335,
+            0.007512423249865194,
+            -0.011911709468952548,
+            -0.00810951734088105,
+            -0.0008079327452253181,
+            -0.01600701450676789,
+            0.023984909975645632,
+            -0.02418072541081482,
+            0.0033984265728439398,
+            -0.0104343782811971,
+            0.01588977364765743,
+            0.05109529246167992,
+            -0.027800269136366997,
+            -0.027889034481533438,
+            0.03706813210665186,
+            -0.02058120652429308,
+            0.030801396468451574,
+            0.03246127847621613,
+            0.008355517405358826,
+            -0.008605467586247932,
+            0.017693344563968573,
+            0.014499487024471321,
+            0.012431277080478217,
+            -0.040103870071440256,
+            0.017328848567062113,
+            0.02723900253727976,
+            -0.054909074068976875,
+            -0.008547596270650966,
+            -0.00777964838164648,
+            0.027134666572242665,
+            0.02765036594970255,
+            -0.002207173084941375,
+            -0.016838740117653725,
+            -0.028289233739722936,
+            -0.019433376533829132,
+            -0.011053380458946764,
+            0.03247641283550457,
+            0.0007514989009289282,
+            0.007919574219000797,
+            0.001316042436488591,
+            -0.0017934342041762042,
+            0.01924145734197619,
+            0.015260199553572459,
+            0.048259602293134723,
+            -0.03383050622747848,
+            0.0021712225589426266,
+            -0.03508567715696004,
+            -0.04578731401250471,
+            -0.008768228935292693,
+            0.010203318819548585,
+            0.024135236371743855,
+            -0.007467140460133838,
+            0.0028907962385387818,
+            -0.007582083854422456,
+            -0.005384477718912037,
+            -0.01867978482547249,
+            0.01641191108381631,
+            0.037168089880543244,
+            0.0031992495663980464,
+            -0.008533890994695162,
+            0.000778843942291118,
+            -0.004620258538699103,
+            0.010572690335163528,
+            0.033233014627100904,
+            -0.030497362482602697,
+            0.02974571152880676,
+            -0.004274442492343003,
+            0.025341347125910928,
+            -0.04307748954707389,
+            -0.018069609916024403,
+            -0.018491122727274337,
+            -0.02069307802429807,
+            0.05372967226264594,
+            -0.030768978301976772,
+            -0.03662045886993282,
+            -0.003205991669059688,
+            0.016815511161004534,
+            -0.0014152644009276566,
+            -0.0016015439520090104,
+            -0.03984716970293867,
+            -0.010828260334550217,
+            -0.05894985435251326,
+            -0.01638035185618134,
+            -0.021819442096404948,
+            -0.005315102328745982,
+            -0.06251579337157112,
+            0.0033713876515840233,
+            -0.07815974854883057,
+            0.001484860071649606,
+            0.04539427129131382,
+            -0.023530943197353476,
+            0.05142250021536681,
+            -0.024746600229063383,
+            -0.0005885740455257141,
+            -0.044698536284421976,
+            0.0026481001040031785,
+            0.02389178603305878,
+            0.051387657872255674,
+            -0.002949198236157656,
+            0.010674377196609455,
+            0.013322075519401089,
+            -0.026245090721950647,
+            -0.01036724558050635,
+            -0.005321884180696542,
+            0.005958798671856245,
+            0.013755178085972732,
+            -0.0074734329537091675,
+            0.038368016046563674,
+            0.010941421833174982,
+            0.03443115781919206,
+            0.034623617136788874,
+            -0.004361926531008214,
+            -0.03348681614127998,
+            -0.00632870804659212,
+            0.024433331840566692,
+            0.0321998807702602,
+            -0.02899983280320611,
+            0.005958471722097962,
+            -0.015602447981516634,
+            0.015633217539747047,
+            0.006321931116948646,
+            -0.011827541320878442,
+            0.02372204564999124,
+            0.015673426811623124,
+            0.006103300725621424,
+            -0.005165006670545588,
+            0.013589526056189892,
+            -0.025411864292353578,
+            0.03041791947375232,
+            -0.004456865298584878,
+            -0.04276058054583262,
+            -0.0024736078733106825,
+            -0.020059309867945227,
+            0.024883721794000956,
+            0.015578237413840115,
+            -0.00536523508503184,
+            -0.005290645798641171,
+            0.008242987948511412,
+            0.020469675635821673,
+            0.006668354647245447,
+            0.00017729517736814364,
+            0.02373604785121342,
+            0.029915687122900485,
+            -0.02432635861789771,
+            -0.029540533943782886,
+            0.008151297444985344,
+            -0.005730733861939129,
+            0.011318816579801282,
+            -0.02644855229221747,
+            0.00033932872948409137,
+            0.0035505684154028996,
+            -0.03570104343808711,
+            0.03408366016010711,
+            -0.009199700348630527,
+            0.005857598530788056,
+            -0.009538177736472163,
+            -0.0017768806148210118,
+            0.0027410153099351694,
+            0.03898108970435115,
+            -0.007367623392751247,
+            0.009827254226889936,
+            0.015465895035342481,
+            0.000981604396724216,
+            0.05143083558797524,
+            -0.027350877485564314,
+            -0.02183036962301852,
+            0.01885468378168636,
+            -0.007285239007489837,
+            -0.016846837665812908,
+            -0.020546383007536074,
+            -0.011863943814684079,
+            -0.016293192750336218,
+            0.01976516340568883,
+            -0.02387328798681798,
+            -0.011892336568452852,
+            0.007170651391639504,
+            0.0013725014050721622,
+            -0.009809669068563384,
+            0.0016617183704652932,
+            -0.006307590349798603,
+            0.02037693835615797,
+            0.031753656615094446,
+            -0.024892675105483313,
+            0.03577919637821846,
+            0.03178303188315581,
+            0.017260898395251155,
+            -0.02102289690701718,
+            -0.009661035772073294,
+            -0.00048542689442823136,
+            -0.014694631967051945,
+            -0.0326379186158631,
+            0.025443206465204822,
+            0.017194626456982255,
+            -0.011381421003142453,
+            -0.03680783761649673,
+            0.0017216115811536721,
+            0.022819859205829785,
+            -0.013600989058799166,
+            0.028141657788496834,
+            -0.01869047298692401,
+            -0.01366833755869951,
+            -0.00782959271979894,
+            0.031227914441058244,
+            -0.05252392878125891,
+            0.016957842026348195,
+            0.00020880696574616416,
+            0.001154370575234767,
+            0.010892067708194324,
+            -0.008368819756628046,
+            0.012224564683325483,
+            0.031700389750709,
+            0.018063405869147564,
+            0.0010794115307202266,
+            -0.002783668216641561,
+            0.01418234057248712,
+            0.017773269840982066,
+            -0.015653443355569053,
+            -0.000497740485657824,
+            0.019890882604100452,
+            0.022672054781919238,
+            -0.00521797370595221,
+            -0.016686942830774276,
+            -0.018443538351123584,
+            0.025270057639201107,
+            -0.010354431070738671,
+            0.03484047753830186,
+            0.04594785214094975,
+            0.0036441959528283004,
+            -0.05695476763071743,
+            -0.009412209619190693,
+            0.0010371110229966086,
+            -0.01099343927053564,
+            0.01606863921935641,
+            0.012470660511773616,
+            -0.03237187923799263,
+            0.006388672213167589,
+            0.012946570716567364,
+            0.022152554488829627,
+            -0.02540317621005867,
+            -0.023183252921885226,
+            -0.0030984019855787655,
+            -0.007890467768631811,
+            0.018648760402624687,
+            -0.008653405980661855,
+            0.0006228315253506687,
+            0.009672526897422602,
+            0.0005436494014569835,
+            0.011632326145022935,
+            0.005655942007108228,
+            0.021736827172951,
+            -0.001256608167230203,
+            0.013124413886287987,
+            -0.01986808920858981,
+            -0.0270450339809516,
+            0.006500423802582728,
+            0.04438242119211055,
+            -0.014813399694814117,
+            0.030621709833336064,
+            0.0092171224268983,
+            0.016433384739642534,
+            0.024875072913970443,
+            -0.00891470467241686,
+            -0.014124009014664287,
+            0.00845777541223868,
+            0.009019007848927453,
+            -0.04147358041382793,
+            0.012513030150663913,
+            -0.01710243399313324,
+            -0.033144939719206276,
+            0.01904674924133127,
+            -0.00534122554342279,
+            0.012367320751926314,
+            0.004270315800911569,
+            -0.002886460711688153,
+            -0.025696286516581048,
+            0.014137901708875033,
+            0.028135087538950946,
+            0.021797869775098042,
+            0.0296173052503884,
+            -0.012757078368361444,
+            0.0005460601780756187,
+            -0.008748930817524899,
+            0.005470975782187157,
+            0.04187083128854477,
+            0.0008687182830616019,
+            -0.0008032802520818105,
+            0.01507726552487313,
+            0.024945744517309208,
+            0.0376130540370386,
+            -0.015415560911023553,
+            -0.03407353556457147,
+            0.01564969972264396,
+            -0.010104443707865262,
+            0.018855792875436674,
+            0.005503259932317811,
+            0.0049634097823351596,
+            0.0038600156268000702,
+            0.019312419743287703,
+            -0.006635193789316492,
+            0.018798045806429395,
+            -0.02500983683871332,
+            -0.028785757874583755,
+            0.01829411751138161,
+            -0.005277972443258226,
+            0.03338623299668019,
+            0.016225885095995672,
+            -0.029522280454919586,
+            -0.01712097344863975,
+            0.01553706187924322,
+            -0.008465766956343768,
+            -0.024842700926275538,
+            0.01895228828512998,
+            0.007076862934523977,
+            0.008956105593078965,
+            0.0024373779389833218,
+            -0.01106503696703573,
+            0.01125466547367417,
+            -0.000818223040067914,
+            -0.015322790791698395,
+            0.014157131555941659,
+            -0.017853386731281715,
+            -0.014994640991761465,
+            0.01775991780643474,
+            -0.037498078532902435,
+            0.009570197712419421,
+            -0.007687297543935123,
+            0.009050494195378254,
+            -0.004222041469820445,
+            0.007460383234429904,
+            -0.0010804379186060264,
+            -0.014987857934522812,
+            -0.014903749199810126,
+            0.03228485430019625,
+            0.022096121825475306,
+            -0.000410816936759558,
+            0.006728733918931589,
+            -3.971636461500232e-05,
+            -0.026719848032785824,
+            -0.011253751790407687,
+            0.02245467425830765,
+            -0.005664635254398481,
+            0.005134496095445943,
+            -0.01109590855846966,
+            -0.0149616794609307,
+            0.0019726017255571673,
+            0.032668637270108894,
+            0.06066635618119076,
+            0.012001732730659747,
+            0.024422093585231484,
+            0.0189499396761026,
+            0.02119276033547528,
+            -0.008489818917377516,
+            0.006413901834269739,
+            -0.03097413287789443,
+            0.03509225573986143,
+            0.005392594635384506,
+            0.03108937318306421,
+            0.026664331248813712,
+            -0.04722525921137681,
+            -0.0058539171830594774,
+            0.017087531454837677,
+            0.022673712717885323,
+            0.0159020681868943,
+            0.0008382498574190522,
+            -0.008938174268517353,
+            0.035650940044285984,
+            -0.03315250299348084,
+            -0.023647111263155,
+            0.005059382024092534,
+            -0.01439537467317451,
+            -0.010488359956731476,
+            -0.006785295580877329,
+            0.02776449681317373,
+            -0.005584052419598926,
+            -0.004582019433008779,
+            0.009697064252638535,
+            -0.014400206558190682,
+            0.008165676294736836,
+            0.0007624778531368436,
+            0.008681263897172326,
+            0.039249866981334126,
+            0.0005918063050375113,
+            0.024275699182767528,
+            -0.004138313072843555,
+            0.011079529739650594,
+            -0.0158972266904151,
+            -0.016985058543605,
+            0.02276982363172991,
+            0.00351101120496393,
+            -0.03244126813432563,
+            -0.00768819222024939,
+            0.00928180979884261,
+            -0.0013793404539641824,
+            0.005266058566444064,
+            -0.00503433670609399,
+            0.003747448261046857,
+            0.03286230666764075,
+            0.0022383592721685153,
+            0.008606190020480513,
+            0.013593947677257625,
+            0.00030951915677929204,
+            -0.01490307680740127,
+            0.013254591138113545,
+            -2.1379308136707722e-05,
+            -0.012900073369514622,
+            0.03357519186606058,
+            0.006705902094864628,
+            0.04243347200485736,
+            -0.03302953674711988,
+            0.009031534935013593,
+            0.01822227379789576,
+            -0.008491770335368761,
+            -0.02725412856390648,
+            -0.03630788717786838,
+            0.008259145747740535,
+            -0.019003260334340084,
+            0.015781275447792516,
+            -0.010039277287105153,
+            0.029665124310477384,
+            -0.03563370794521629,
+            0.004714858110268169,
+            0.004053913355055313,
+            0.0009503673086552651,
+            0.02068360255395961,
+            0.018524093113914366,
+            -0.002785698934204024,
+            -0.031443835341186485,
+            -0.02066365433385339,
+            0.006869321907853112,
+            0.02431260052207595,
+            -0.010367048093621498,
+            -0.005612149485859478,
+            0.0006316058139530722,
+            -0.024369562712568606,
+            -0.016845124984022828,
+            0.030359419843284823,
+            -0.040974427176153246,
+            0.005252911449822132,
+            0.04272936053727608,
+            -0.04599092715016919,
+            -0.03802288716943514,
+            -0.002314988728575789,
+            -0.024124014894379466,
+            -0.006360771882090754,
+            -0.002068030981218464,
+            0.016669991113175926,
+            0.0036232707799735537,
+            -0.028763764013482795,
+            0.00943401978066451,
+            -0.005173988601101547,
+            0.003971420424839974,
+            -0.0031669592494977393,
+            -0.02216336363004636,
+            -0.01657772032717298,
+            -0.02531962163186398,
+            -0.035174102848865116,
+            -0.0015903434876311088,
+            -0.005158703150821607,
+            -0.028703075279214722,
+            -0.03941940849540492,
+            -0.021696000925262537,
+            -0.0160221488026189,
+            -0.00843213135542359,
+            0.040641882367348774,
+            0.021319060666253945,
+            0.019336844555345886,
+            0.0025252167889382917,
+            -0.020346900609125973,
+            0.049440028814233025,
+            -0.007642182460707816,
+            0.035628691886108926,
+            -0.0017882888000877158,
+            -0.013461669667950893,
+            0.00455558389906464,
+            0.03825428026120858,
+            -0.014248434654950971,
+            -0.0009902062564027718,
+            0.00027811655483232545,
+            -0.0011866253757890136,
+            0.0008535155950871286,
+            0.015433650396777696,
+            0.001307651096343185,
+            -0.007112199543314309,
+            0.030380673552683906,
+            -0.012803247194895328,
+            0.01637501752192592,
+            0.018350814869962304,
+            -0.0013359719137099667,
+            0.008064546833360553,
+            -0.021430659268086814,
+            -0.00731572118780415,
+            0.0044578576331236325,
+            0.008422210247967642,
+            -0.015096885054883586,
+            -0.009964588397149773,
+            -0.02359494467299634,
+            -0.01174145797396426,
+            0.0023855749166744366,
+            -0.004559202896782688,
+            0.004902138778592298,
+            0.034595817216889944,
+            -0.016068744500427584,
+            -0.035106033294403614,
+            0.009997207702122002,
+            -0.02663088875167486,
+            0.02222306406130687,
+            3.983384942889326e-05,
+            0.006838221769859366,
+            -0.0023434342406307616,
+            0.001765445386886839,
+            -0.00169484278667461,
+            0.002862331185439823,
+            -0.018217975204395134,
+            0.0031859734246840257,
+            -0.0032119988065673565,
+            0.0009052950782312657,
+            -0.0012459653052875638,
+            0.00687339892509406,
+            0.006332799551164718,
+            -0.0008019538898592668,
+            -0.002968451740267742,
+            -8.541086355097327e-06,
+            -0.020561816931299912,
+            -0.021642512405352372,
+            -0.028259049261410905,
+            -0.000989887278779396,
+            -0.012359733017834338,
+            -0.027804394906433664,
+            0.011443385424045703,
+            -0.006872684999639775,
+            0.008327844771213648,
+            -0.008969527138034904,
+            0.006219994886600349,
+            -0.008415106130309062,
+            -0.0004143996206881767,
+            0.013119286511794273,
+            -0.02925209282917943,
+            -0.02096907906398536,
+            -0.03096513357712367,
+            0.01762223841790055,
+            -0.00925741670612824,
+            -0.002656382260603827,
+            0.0025223527960181835,
+            0.005092659166535706,
+            -0.033606542487570154,
+            -0.006653438998497412,
+            0.007313682256676618,
+            0.009547290751154595,
+            -0.03287788897629887,
+            0.02446728061355753,
+            0.017023222076526946,
+            0.01874233066565012,
+            -0.02417566312664911,
+            0.04049827067897005,
+            0.014426879325647042,
+            -0.021740606525480655,
+            0.016712907564723732,
+            0.027385150577780248,
+            0.03177163221499527,
+            -0.03469098547555545,
+            0.004481376248622418,
+            -0.01518200063132508,
+            0.018918680020516044,
+            -0.025284921244460287,
+            0.020625170445627613,
+            -0.016707629602243063,
+            0.021278719363178687,
+            0.03520334001264928,
+            -0.005524369131116028,
+            0.03608133014180745,
+            -0.019234548780647943,
+            0.0059985801736444555,
+            0.004041778461022664,
+            -0.01645503683802124,
+            0.01024694092614158,
+            -0.0042700351372843386,
+            0.0006059159967286885,
+            0.009406090340935747,
+            0.019133229156115282,
+            0.019007219188041775,
+            -0.04442208547053466,
+            0.028588965123434923,
+            0.00804120101594811,
+            0.021472086070876265,
+            0.014177820346835521,
+            -0.02708807970982664,
+            0.004742981711172693,
+            0.002072673190279628,
+            -0.004659713164250794,
+            0.0061167904730676514,
+            -0.013796604518930328,
+            -0.034540857808309226,
+            0.006338792319814448,
+            -0.018918703778187926,
+            0.015036937364271984,
+            -0.009026670935442384,
+            0.0011667525636770179,
+            -0.01152652811400482,
+            -0.011938679695598347,
+            0.0006170592505446954,
+            -0.005495151258143897,
+            -0.0008075736908949407,
+            -0.03171721592827769,
+            -0.04427658539412448,
+            0.02036132271908089,
+            -0.0021221965958423804,
+            -0.04320396743189005,
+            0.004111179168317842,
+            -0.025352428814134187,
+            -0.01865918186335911,
+            -0.011237699157960728,
+            -0.025047923887906474,
+            -0.04219630696740606,
+            -0.011108496740166623,
+            0.007785006066683012,
+            0.016810642610918525,
+            -0.009081815651299525,
+            0.040965574143584864,
+            0.0073944364393447965,
+            0.015313576815177263,
+            -0.017731360025694955,
+            0.026137777790243826,
+            -0.019314392060123,
+            -0.03108486330987073,
+            0.02676829566273746,
+            -6.259161089416372e-05,
+            0.003053748984540415,
+            0.014358356168574803,
+            2.8186924071913628e-05,
+            -0.006181789640967465,
+            0.01947730374003864,
+            -0.028739300217145797,
+            -0.023460503178221397,
+            0.0012460072421725688,
+            -0.008850395570726385,
+            -0.016271242375881602,
+            -0.035896509136022194,
+            0.005227881053956234,
+            0.03324337105109496,
+            0.004970419987430122,
+            0.008273497228454482,
+            0.01814500594735864,
+            -0.02229593571615308,
+            -0.024230675455380678,
+            0.000639112529814716,
+            -0.010266640429960994,
+            -0.010257247303470066,
+            -0.021302483296731352,
+            0.019150757066356872,
+            0.01651375496287087,
+            -0.00390189952706486,
+            -0.006971692709239087,
+            -0.010621003777733253,
+            -0.01918206725910585,
+            -0.02020345113211036,
+            -0.013827322378214754,
+            -0.017980206764311026,
+            -0.034153741666571154,
+            -0.032195001867229935,
+            0.03225205339201753,
+            0.026354817588919596,
+            0.0014076409241492998,
+            -0.007959081015805435,
+            -0.023070419006792155,
+            -0.004483328183581872,
+            0.02879516595345542,
+            0.002549934782150634,
+            -0.0001541849716285838,
+            -0.00830090759167373,
+            0.015341095409025125,
+            0.00835695019244165,
+            0.005107844984733648,
+            -0.06676974053142512,
+            -0.006027540973808174,
+            0.01655283585235268,
+            0.01351155740684989,
+            -0.002359114145814062,
+            0.02591301169926259,
+            0.01108099264396926,
+            -0.015361038608114896,
+            -0.019765743462262668,
+            0.035195761515422364,
+            0.02408910733949848,
+            0.023468767772473746,
+            -0.0034492999385544727,
+            -0.004472311490618405,
+            0.007030833258457338,
+            -0.010684252427867153,
+            -0.013326964647936429,
+            -0.004558513842504658,
+            0.026925146859337397,
+            0.016099443021514896,
+            -0.0064331198280825575,
+            0.010848931851939124,
+            -0.0008320209125061849,
+            -0.019444491995041537,
+            -0.001841330829156697,
+            0.008730275423516445,
+            0.0012974082030866743,
+            -0.014611902493400719,
+            -0.005911807768916134,
+            -0.019130604101278263,
+            0.024719468317569562,
+            0.0209456329723249,
+            0.03381727436368338,
+            -0.013237902591972769,
+            -0.0003695142331921278,
+            0.005410360755058108,
+            0.0002841632253252816,
+            0.0023352645111704958,
+            0.004801123870325242,
+            0.02623649813035263,
+            -0.029010710171522985,
+            -0.020741774066460415,
+            0.0068426980682519505,
+            0.013445270043414486,
+            0.0022744032984779294,
+            0.033840485942696216,
+            0.035040080489222324,
+            0.004166066423508316,
+            -0.010046666684988869,
+            -0.006148765564658713,
+            0.007071740742464309,
+            0.01865897335451031,
+            -0.023267411753900076,
+            -0.0005936657905496225,
+            0.004833030519903764,
+            -0.004229582648167378,
+            0.03620610659071198,
+            -0.009961238880412731,
+            -0.007197670122096852,
+            0.022212791947347127,
+            0.09819205436926473,
+            0.00047695621444943017,
+            0.0025354521225913395,
+            -0.029358210234742176,
+            -0.023614016211567963,
+            -0.0012564198366457442,
+            0.00242680708457405,
+            0.00830167274308547,
+            0.028076260517106013,
+            0.0004852400049405906,
+            0.011579274037419117,
+            0.02773972145320728,
+            -0.017217305534161797,
+            0.006752070301141707,
+            -0.011376051474975524,
+            0.024577071515205502,
+            -0.012694034026935636,
+            -0.0019809719792138673,
+            -0.016512649783050146,
+            0.015907402258634816,
+            0.004556574687833768,
+            0.004907510649166062,
+            0.011347630984408937,
+            0.009824484201337644,
+            -0.02650731900265866,
+            0.01756773113723491,
+            -0.005671986127099093,
+            0.004449611508541289,
+            -0.027841346572384462,
+            -0.020124233970502267,
+            0.0025986391930124847,
+            -0.0014440758308989809,
+            -0.009579444248630085,
+            0.006005274872617889,
+            -0.0024984235450999127,
+            0.03765664613203687,
+            -0.03765496603611941,
+            0.01409350051861866,
+            0.0066453175608372555,
+            -0.0015390841089599882,
+            0.011806653942640485,
+            0.017306914776093522,
+            -0.008206256939290151,
+            0.02589800448884443,
+            -0.00798213351353373,
+            0.009846274212000523,
+            0.015230070803055296,
+            -0.03449531102975745,
+            0.009174739184806312,
+            0.010145762060174144,
+            -0.024165660737765746,
+            0.012146877603301336,
+            -0.013784371016298891,
+            -0.004818759788054113,
+            0.005930446700613062,
+            0.007349837554764034,
+            0.010733731974181452,
+            -0.02295571464086674,
+            0.016176424826629898,
+            -0.010947841926210105,
+            0.0037766665765835445,
+            0.013559846998199008,
+            0.024892849808979323,
+            0.0012426907454129122,
+            0.0005950515136086193,
+            -0.01379469431504576,
+            0.005534724429523084,
+            -0.015667467221535825,
+            -0.022401586867694584,
+            -0.014192041775234079,
+            -0.006366846766730035,
+            0.01029598687444803,
+            -0.003972386708834858,
+            0.016971476088808737,
+            -0.010309648640697812,
+            -0.010645860303528099,
+            0.0037818310327362103,
+            0.01040806357838676,
+            0.008316605962234849,
+            -0.02855914079113881,
+            -0.005676310556150579,
+            0.004961084138163161,
+            0.00029285971503001265,
+            0.009592497954559435,
+            -0.014556073011576487,
+            0.013689526487160009,
+            -0.013290113354287095,
+            -0.0010726537498214612,
+            0.013201860397060236,
+            -0.012443552043930505,
+            -0.004012305744087929,
+            -0.01592905426965635,
+            -0.028487066798154617,
+            0.008462143660968396,
+            -0.01365614159523485,
+            0.001076882584445294,
+            0.0019503240125505646,
+            0.004730085646042709,
+            -0.016919069570642004,
+            -0.007860046623166481,
+            -0.033515448866521046,
+            0.011065416478687504,
+            0.02314589687882036,
+            -0.006435733825418645,
+            -0.030351727858789472,
+            -0.009546333526852862,
+            0.011327489786114963,
+            -0.0071765122743172715,
+            -0.01676736383051139,
+            -0.0029390038145944096,
+            0.007617910650439855,
+            -0.008174620754097493,
+            -0.00508943241317381,
+            -0.0025173969010414098,
+            0.0217328037301181,
+            0.016094975605203832,
+            0.01769642142712967,
+            -0.021400862603034036,
+            -0.00976992221747187,
+            0.026043879119776864,
+            -0.03275550610951898,
+            0.012441465975914529,
+            0.024587446266013183,
+            0.003446527051428522,
+            0.005492228054838714,
+            0.0030377554109693714,
+            0.005459159163772825,
+            -0.006985179226831048,
+            0.019925583798328282,
+            0.010870421035240845,
+            -0.019709062313471467,
+            0.005385443078244229,
+            0.02086085539375208,
+            0.0037248737640193097,
+            0.012892299156840998,
+            -0.017546870090541213,
+            -0.0020341807891543716,
+            0.015594779287017958,
+            0.04816214672670215,
+            -0.0032851797178091685,
+            -0.010450846098750195,
+            0.029513742590677183,
+            0.023526004023637855,
+            0.005373455940309873,
+            0.0012120777808635214,
+            -0.02525630456877349,
+            -0.013075130701774785,
+            0.01962235086017781,
+            0.016944870494393874,
+            0.013484925533584295,
+            -0.0023255656480479004,
+            -0.019236081983457366,
+            0.021520156296467717,
+            -0.01701733319731161,
+            0.012422659010697477,
+            -0.023764941037007197,
+            -0.001490790833263327,
+            -0.0035419008954878724,
+            0.021998949056452454,
+            0.0021819635638621092,
+            0.008891505304514552,
+            0.024354552127458608,
+            -0.006978548952551799,
+            -0.019526615985982355,
+            -0.0051799020400150305,
+            -0.014578953088901798,
+            0.022506635912517146,
+            -0.024117708614271965,
+            -0.007008641019892289,
+            0.011379795968360019,
+            -0.012362093847309038,
+            -0.0102534051092702,
+            0.001164934138876502,
+            0.017686807394319725,
+            0.0009423993435129515,
+            -0.014784508280754078,
+            0.01944445593807992,
+            -0.0011359271135469456,
+            0.010410915597949927,
+            -0.01921419010977189,
+            0.009390638453648025,
+            -0.016682442516992777,
+            -0.007849184048209236,
+            0.010343008873053859,
+            0.0012604521199837046,
+            0.003948300703645506,
+            0.002592490121754231,
+            0.007773437307969315,
+            -0.03355658426684504,
+            0.022695505487314516,
+            -0.010751606023585792,
+            -0.003538523688786008,
+            -0.0014125330167969218,
+            -0.01678692701672328,
+            -0.0006918162525881059,
+            0.005816502852598842,
+            0.010878358093847386,
+            -0.016045147059003256,
+            -0.00873270513444376,
+            -0.04920244395446931,
+            -0.030314009649638693,
+            0.023156333024481057,
+            0.013733657599720879,
+            -0.010971506342695331,
+            0.018852159415605117,
+            -0.023760853159532587,
+            -0.010587249404421732,
+            0.0011379869499042905,
+            0.023319856329056813,
+            -0.03539290744576506,
+            0.01215369703816749,
+            -0.014493682360886549,
+            -0.02039288669534806,
+            0.009070984596830142,
+            0.011445354735283254,
+            0.0007259852045513916,
+            -0.0011514627555568814,
+            -0.029350945123925453,
+            -0.010246845821727087,
+            0.02391068177425878,
+            -0.00861718589055971,
+            -0.004723044986826333,
+            0.024716520302409263,
+            0.0033721801437248873,
+            -0.012524663490691517,
+            -0.017955758068233754,
+            0.021612671655463236,
+            -0.026077203994996058,
+            0.007645944309444457,
+            0.010032183011253845,
+            -0.0173991263874487,
+            0.021649599230588117,
+            -0.007529701797485131,
+            0.020327763440441377,
+            0.004227390938169872,
+            -0.011249233530251522,
+            0.020352283974471443,
+            0.013644443758970054,
+            0.004216958800094986,
+            -0.004081669404784771,
+            0.006869353098014179,
+            0.02212732880780841,
+            -0.011021739614266063,
+            0.02219580717345023,
+            0.028777213295471067,
+            0.00017328191188631918,
+            0.03260179554743322,
+            -0.021287655520443285,
+            -0.03159728983004563,
+            0.021740163175540124,
+            -0.01035146858119169,
+            -0.021564906188712388,
+            0.028914245422221707,
+            -0.013674731836508768,
+            -0.00786476052825481,
+            -0.0166718429993922,
+            0.00533027311342322,
+            0.016680336311088843,
+            -0.005246190451159961,
+            -0.014360934066628332,
+            0.0029628103834216345,
+            0.019809487418628386,
+            0.014587234544007958,
+            0.029780009648869975,
+            -0.010778199815920887,
+            0.035180242185168895,
+            -0.002320576872571577,
+            0.008990157317473909,
+            0.012415465232073404,
+            0.006489435152972723,
+            -0.011251531176923445,
+            0.013501894212117511,
+            -0.003446759011266526,
+            0.008230548711588447,
+            -0.001449023474280409,
+            -0.035111679577202855,
+            -0.013709382199848678,
+            -0.04709294504969336,
+            -0.0023752976043235697,
+            -0.0025098360125255215,
+            0.0030975417740913154,
+            0.09736815869389318,
+            -0.005723049647039619,
+            -0.011988817571623457,
+            0.001625059666070763,
+            0.009611697638805556,
+            -0.0009896494420056814,
+            -0.007606836952841873,
+            0.015973152482488318,
+            -0.01343292635775299,
+            -0.02140928016038583,
+            -0.010532701190806147,
+            -0.01288594454368977,
+            0.012079198190522715,
+            0.015929716343161652,
+            -0.0005947359502228821,
+            -0.015207331663761012,
+            0.006524010655124294,
+            -0.004566385899752684,
+            0.0064717704997400245,
+            0.008647099975176308,
+            -0.010414668624343924,
+            -0.0071698879102925555,
+            0.030899143952119196,
+            -0.025301269549419674,
+            -0.024361748354052457,
+            -0.0010579931222861095,
+            -0.009055796737486008,
+            -0.010335459737012047,
+            -0.002692653633719985,
+            -0.0017679501322462056,
+            -0.010519994985692081,
+            -0.010019865263564572,
+            0.006537076627706128,
+            -0.015472130358273033,
+            0.004812857776601667,
+            -0.0047457011269495634,
+            -0.013296069496526007,
+            0.017054968086538724,
+            0.022563785357424853,
+            -0.02841491200807702,
+            0.0006755428946168761,
+            -0.017422510783648978,
+            0.012941211381006863,
+            0.008866069010589792,
+            0.006049618782357251,
+            -0.022451528209097505,
+            -0.023622906582403243,
+            -0.015019272754405791,
+            -0.01927535750568903,
+            -0.024973490129133732,
+            -0.009073381957684092,
+            0.0067692803507600945,
+            -0.001267339788153158,
+            0.021918281111033645,
+            0.000816538714279555,
+            -0.01812545613159305,
+            0.0061084913570894905,
+            0.000568371220952269,
+            0.018292497018668842,
+            -0.027468677288882186,
+            0.0033806422195878524,
+            -0.02307582511848465,
+            0.033508234091571124,
+            0.0039195050746594975,
+            -0.026680111514794824,
+            -0.021558668725755275,
+            -0.003311830287529699,
+            0.018994577319066473,
+            -0.0025745912019675805,
+            -0.00724049447539589,
+            0.014775207617625293,
+            0.0155334760323838,
+            -0.004693118176196483,
+            0.002015625406296403,
+            0.027968079002437102,
+            0.01077998866180723,
+            0.0313892866933904,
+            -0.015236169297590675,
+            0.004945828495617267,
+            -0.027215016713379062,
+            0.010214419950212732,
+            -0.01503548137805401,
+            0.008859554812646242,
+            -0.01290890682557532,
+            0.03531384210182567,
+            -0.012005613861257432,
+            -0.004250698950192301,
+            0.03253288796441852,
+            -0.029150986393345252,
+            -0.0035020051536906334,
+            -0.009255437891304041,
+            -0.004142204892160768,
+            0.013240580426083011,
+            -0.001198093874819309,
+            0.008485141356876356,
+            -0.01854370462655459,
+            9.726096864073609e-05,
+            -0.003221852304410318,
+            0.005404595442127598,
+            -0.02005458688172967,
+            -0.003314118809778734,
+            0.0073427568419969814,
+            0.004259518237708634,
+            0.002408630356035373,
+            0.016420646142725346,
+            0.006662121128241274,
+            -0.0029357701493526907,
+            0.03424224243607692,
+            0.008171895850772245,
+            0.0152521424877615,
+            -0.0018638264193202728,
+            0.0030234211261443034,
+            0.019287963516760636,
+            0.019912956661192856,
+            -0.00048443399729935116,
+            -0.01058935153883094,
+            0.00195113175754497,
+            -0.001140732137433033,
+            0.011048741602327437,
+            0.024396046054912645,
+            -0.021158025221644738,
+            -0.02419562092713724,
+            0.016377661005279716,
+            -0.015076145951740961,
+            0.01903712233378994,
+            0.010541463112889985,
+            -0.010099351302095792,
+            -0.00031558616108876247,
+            -0.042316016201379245,
+            0.010964989031040692,
+            -0.02068939391415103,
+            -0.04004120028915868,
+            -0.00244383573783919,
+            0.0009542507389088937,
+            -0.029439456243505052,
+            0.02822814855547978,
+            0.0023755032182188156,
+            -0.017280221740813364,
+            0.018223709137876153,
+            -0.005548654680726505,
+            -0.0024040301239229064,
+            0.005397711787238679,
+            -0.01092026090471121,
+            -0.010648485086769927,
+            0.008235250252660972,
+            0.010957715075632535,
+            0.005425683656535705,
+            -0.005698919020608095,
+            -0.011284911899894062,
+            0.009975882651638084,
+            0.012318901214538077,
+            -0.0019922884373083155,
+            0.010050520101815593,
+            0.0031720752428247446,
+            0.007352859750323051,
+            -0.0073483049386141005,
+            0.0046483587496626965,
+            -0.005093220128411125,
+            -0.026080057085901446,
+            0.012863580653903103,
+            0.0052434400313520774,
+            -0.011863360599637747,
+            0.0103209502958491,
+            -0.01048162756158923,
+            0.005787157766911021,
+            0.001318945460232246,
+            -0.0059333230422061395,
+            0.004821320238893602,
+            -0.0038922239931512784,
+            0.011267634571983924,
+            0.013256611037442042,
+            0.004592529900805719,
+            -0.013076954459595086,
+            -0.02177318635989856,
+            0.009001291224623467,
+            0.01959767163356408,
+            0.024161754192119852,
+            -0.030416109293488064,
+            0.003784634588325551,
+            -0.0014915849060829743,
+            -0.0020550564975080885,
+            -0.015632785947151753,
+            0.01587790981601866,
+            0.01716742056401084,
+            0.012721563202528083,
+            0.01893393006967701,
+            -0.004009617703771589,
+            0.010539100665843704,
+            -0.02080068888791658,
+            0.013642451766699067,
+            0.0008609869904066794,
+            -0.013629067231663643,
+            0.0014620655123815298,
+            0.027046304278766087,
+            -0.011056783825898616,
+            0.006206969699250696,
+            0.008097527175511791,
+            0.022111870736907183,
+            -0.012509479100313492,
+            -0.003735868296238069,
+            0.01516254968167132,
+            0.0014878674796313539,
+            0.0035887079417890352,
+            0.016533149527222273,
+            0.04729052276110659,
+            0.00102751186013391,
+            0.0016713773449738153,
+            0.0192177920520243,
+            -0.005563519063161466,
+            -0.01114821419109643,
+            -0.003549999863325982,
+            -0.021807893906469106,
+            0.010998558936481785,
+            0.00431279785115251,
+            -0.02186324212032519,
+            0.01024093366826206,
+            0.0125495473550584,
+            0.005268320135120931,
+            -0.0031679212410026603,
+            0.02574575247827441,
+            -0.019807879421996258,
+            0.014935295894965128,
+            0.03329450628384787,
+            0.00493356778223407,
+            -0.011423949333937285,
+            0.007263264249871384,
+            0.0027354459678742013,
+            -0.004494406041906684,
+            -0.00606745413660583,
+            0.010497391290734088,
+            -0.008191702034391382,
+            0.009870458191510176,
+            -0.009580130479693,
+            0.039202091799313755,
+            -0.009619018331262844,
+            -0.010782138248435458,
+            -0.004558645539551881,
+            -0.006241773108460662,
+            0.003478414182938609,
+            -0.003395165694642073,
+            0.005812225135633631,
+            -0.0069045407353789945,
+            -0.014607146220271209,
+            0.0008522982757210454,
+            0.00020175973815948303,
+            0.014001922919116725,
+            0.0011804904415274088,
+            -0.004871874532755353,
+            -0.007877217154706205,
+            0.015243277056007988,
+            -0.006047906208304912,
+            0.0006531937033317192,
+            -0.0011653523616313422,
+            -0.0013662556399359335,
+            -0.007401050012179761,
+            -0.015166105795085528,
+            0.00974609244984173,
+            0.016253733908242134,
+            -0.013857834499211056,
+            -0.005142936208363415,
+            0.012274716212832219,
+            0.008540229125193836,
+            0.011742606168187876,
+            0.0007451338650642946,
+            -0.009255813682572937,
+            0.008292531177374192,
+            -0.015020523035432158,
+            -0.008781317965985481,
+            0.0026250469418136603,
+            0.002381147382977664,
+            0.005839723981046825,
+            0.0006240641828668715,
+            0.004502463508605272,
+            0.005798912705218846,
+            -0.00497936644919367,
+            0.01112666823857836,
+            0.008198424171006548,
+            -0.002452143482105151,
+            -0.01017569410837814,
+            -0.001127589984272638,
+            -0.015733298167661346,
+            -0.016758315411028353,
+            -0.0009286266498936771,
+            0.015175988113750882,
+            -0.02352862898718829,
+            0.018996352975391644,
+            -0.0014524363611708201,
+            -0.0033802546171857704,
+            -0.021392428456030312,
+            -0.0025923983911570183,
+            0.0034214188668518796,
+            0.01255768230394476,
+            0.01858853630745928,
+            -0.02729543519465384,
+            0.010575894388563116,
+            -0.007204725245442141,
+            0.030394929424014717,
+            -0.018328782140005044,
+            -0.004022941273610227,
+            0.026915618079621992,
+            -0.02200018306828758,
+            0.0008791185377513978,
+            -0.011454606955158956,
+            -0.008103032693853504,
+            0.019287674837688177,
+            0.023747720480736777,
+            0.02038287242167783,
+            0.01364489294869211,
+            0.0007758044919789085,
+            0.004768030222445397,
+            0.03572705648994403,
+            0.010568241270862248,
+            0.014828087092859878,
+            -0.007542489523110974,
+            9.457808640264811e-05,
+            -0.02554033312739386,
+            0.01635899434369978,
+            2.587383534566272e-05,
+            -0.012370727506722627,
+            0.007162422186621379,
+            -0.013370882403931942,
+            -0.011066483600541058,
+            -0.007151077450730025,
+            -0.01721910018182534,
+            0.02152239849400241,
+            -0.012322440280680164,
+            0.004587729883596212,
+            -0.009079747696669277,
+            -0.013272707286450417,
+            -0.012953682633400515,
+            -0.014202214807187545,
+            -0.01313141988034687,
+            -0.026041403779538307,
+            -0.027109731260915763,
+            -0.006573285295895077,
+            -0.023780366346334524,
+            -0.013799072648505526,
+            0.005588622594136253,
+            -0.022617902666316757,
+            0.012262325666428604,
+            -0.012689827812482627,
+            -0.01226264872059864,
+            0.012805734807835008,
+            0.020274844315616616,
+            -0.011765231803328348,
+            -0.028494489703400308,
+            -0.018583843821043854,
+            -0.019970998470704358,
+            0.012576201137505969,
+            0.002266288538944273,
+            0.013165889075444961,
+            -0.003018437854660793,
+            0.003227412667880239,
+            -0.014895831590298957,
+            0.018570624675927183,
+            -0.03442390703780963,
+            -0.03108003806963536,
+            -0.013611306765348361,
+            0.005182777066476028,
+            -0.0015782033203578553,
+            -0.011429046504256945,
+            -0.0018606648572920801,
+            -0.00433805271182878,
+            -0.025109572024446088,
+            -0.010134856632190067,
+            0.011010662054484161,
+            -0.015205629927469414,
+            0.0072933565255675036,
+            0.0009018832564038863,
+            0.013870902834260887,
+            0.009587432944616442,
+            0.013393301893492882,
+            -0.006095063698384775,
+            0.004356485794113208,
+            -0.0026476612747324386,
+            0.0004894024149759108,
+            0.009336366919474478,
+            0.00839591561027103,
+            0.010551161369079399,
+            0.002260707777260338,
+            0.007830059223001696,
+            -0.0060927446216633265,
+            0.018065002687169148,
+            0.015436944875538481,
+            -0.0002520689442496089,
+            0.013323098485251487,
+            0.0003769862591044255,
+            0.00022293304921127018,
+            -0.0029970303106340466,
+            0.007888437018607987,
+            -0.011785957933642163,
+            -0.001311704406355858,
+            0.015138190893483804,
+            -0.01707247730569978,
+            -0.013045187198628258,
+            0.03107034781283638,
+            0.0018583638488268697,
+            -0.02495564203304796,
+            0.031146554085787762,
+            0.004220047610942943,
+            -0.005835199594223787,
+            -0.013205086624848782,
+            0.0014059047822222418,
+            -0.011570624844100298,
+            0.003584232809362274,
+            0.014295032061446826,
+            -0.02231984223610487,
+            0.010521094801839404,
+            0.008446892441796151,
+            -0.009681535919980559,
+            -0.012337659795586223,
+            0.009847073709209602,
+            0.016809328445200358,
+            0.004472485504034416,
+            -0.03539093810210523,
+            -0.011957694102753411,
+            0.02145470732008716,
+            -0.029237428225009913,
+            -0.00810041956259673,
+            0.007452962400105787,
+            0.0065154832040426065,
+            -0.019353109017609053,
+            -0.008580955020412362,
+            0.005725794605897509,
+            -0.004940390374637084,
+            -0.003892210172017758,
+            0.017541623128534682,
+            0.00485047043626186,
+            0.013698341792101025,
+            -0.01689738578307474,
+            0.009771458057400557,
+            -0.022222817828602557,
+            0.004021882074676753,
+            -0.023373700500461992,
+            -0.009873403167702603,
+            0.002364143609079849,
+            0.012454271200139762,
+            0.014328556467632212,
+            0.027839363143659326,
+            0.030056404238348527,
+            0.013995696971422429,
+            0.007833677585250953,
+            -0.016265199386953717,
+            -0.018134327237616175,
+            -0.006409513840075933,
+            -0.004033173350204784,
+            -0.008714798059218596,
+            0.0066373547199279235,
+            0.01858354212009877,
+            0.004499712802472377,
+            0.02609078206670711,
+            -0.006227189596272876,
+            0.009039744622014585,
+            -3.127526911951914e-05,
+            0.0008833930131827993,
+            -0.004226083827636182,
+            -0.01971685546584527,
+            0.00481003211084742,
+            -0.0013721482162825508,
+            0.025874758880591167,
+            0.00940496860916206,
+            0.0024915013399541847,
+            0.01615978314811047,
+            -0.006378952757397484,
+            -0.01174653607178726,
+            -0.010540107733238588,
+            -0.0061926268398085176,
+            -0.021752582127981104,
+            0.029059100503124387,
+            0.021195595568253612,
+            -0.010800765264123782,
+            -0.013693582166598082,
+            -0.00903369526059487,
+            0.02795767990556847,
+            0.004080965688836953,
+            -0.0013050212744006933,
+            0.016313311850308505,
+            0.012084555958999215,
+            0.012329325712491252,
+            0.009794698207689061,
+            -0.00036157148861933123,
+            0.0015364709122308352,
+            0.03722522141440073,
+            -0.007427028510548186,
+            0.0026463292682538495,
+            -0.005239324411651515,
+            0.0025987932868461313,
+            -0.009705817520422487,
+            0.007242122655628283,
+            -0.004012676385296197,
+            -0.0158760051470016,
+            0.015003444174308607,
+            -0.022443331305169994,
+            -0.0016749787204471238,
+            -0.008150635190952799,
+            0.0016696640640947303,
+            -0.02781023337840043,
+            -0.0025731206607371845,
+            0.006695065701630532,
+            -0.015547340828489069,
+            0.011816010501093885,
+            -0.01740673229051557,
+            -0.011553604323149105,
+            0.007914961618894248,
+            -0.012034157288065447,
+            0.0016684130271324446,
+            -0.0008268957621923278,
+            0.004460864332388434,
+            0.013426055079012884,
+            -0.011008808570221245,
+            0.012166046631826273,
+            -0.005816284310138776,
+            -0.007872091214834688,
+            -0.0016055129487028885,
+            -0.0030049472973942043,
+            -0.0025600746761171543,
+            -0.0009966231170649917,
+            0.034355910234323155,
+            -0.004446570896048543,
+            0.003761242147499512,
+            -0.01362639743248326,
+            0.0019833052855181806,
+            -0.036176376394999044,
+            0.028344464817309477,
+            -0.008642835889481802,
+            0.011013502221008498,
+            -0.0017508365480865596,
+            -0.013243706268954915,
+            -0.01456517309747086,
+            0.015247299501245178,
+            0.007602596572394442,
+            -0.005854531057349665,
+            0.013558248087347848,
+            -0.019724373614448298,
+            0.008530500822910953,
+            0.004463694543554244,
+            -0.015561397976987216,
+            0.004668780190328835,
+            -0.007479912658259757,
+            0.012657201279639194,
+            -0.01774730033362932,
+            0.004314594593150765,
+            0.00958571617187387,
+            -0.01892876084898881,
+            0.009901041800918215,
+            -0.028675879564512306,
+            -0.004631920242208613,
+            -0.012323455524366223,
+            0.0053804608827285865,
+            -0.000660864620708182,
+            -0.02389088252276507,
+            -0.028944280660230762,
+            -0.0002099667627277945,
+            0.00458533068247518,
+            0.0018878351997997833,
+            0.011621407953503755,
+            0.005970807015151215,
+            0.008542958629199003,
+            0.01023896054756221,
+            -0.00205207247588841,
+            0.009402752124876478,
+            0.04169706217414899,
+            -0.022899912321317197,
+            -0.011574970753549647,
+            0.005321872977369877,
+            0.011499271505460519,
+            -0.02647653483798284,
+            -0.011039791758347888,
+            0.0033424345263339144,
+            -0.005219877979550397,
+            0.03609469691834737,
+            -0.02448746405881972,
+            0.019020618652020673,
+            -0.011392297116547224,
+            0.007625475937968714,
+            -0.021465240076078376,
+            -0.003921088264697311,
+            -0.014265745991522477,
+            -0.00011221615798127438,
+            -0.0031478236827989703,
+            -0.0076053733863387755,
+            0.007485039021296698,
+            -0.001907678661303243,
+            0.0020348246022161925,
+            0.006291436821831794,
+            -0.02173305455485444,
+            -0.0019125499969493854,
+            -0.016432460924913166,
+            0.030367263349759286,
+            -0.016242095671439496,
+            0.005072805486919414,
+            0.0053258139728101506,
+            -0.002810639419737258,
+            0.012368771814660942,
+            0.004165967191536186,
+            0.00513754440091394,
+            -0.004199767241827414,
+            -0.005142642360546972,
+            -0.0053886448467479215,
+            0.017868874549760376,
+            -0.013280048598849825,
+            0.035024701466710065,
+            -0.0100959186045929,
+            -0.007094001877261568,
+            0.0019733042325232155,
+            0.010380430710282744,
+            4.2084202709296443e-05,
+            0.01053565670076933,
+            0.0034270489770133685,
+            -0.0006311078385359693,
+            0.0019774779328212756,
+            -0.017162448473652843,
+            0.003089758682621882,
+            0.01682656561693936,
+            0.005463677441664502,
+            0.004469261499451007,
+            -0.013332130891003062,
+            -0.012050233723282738,
+            0.004538268704442613,
+            -0.007533940408255447,
+            0.0003675737140813486,
+            -0.018816953644773076,
+            0.013972696748575243,
+            -0.0011096683686631858,
+            0.0002715683052236706,
+            0.00524678369365015,
+            0.023056514997908218,
+            0.01813167860994747,
+            -0.002553352602551003,
+            0.03155164978556312,
+            0.0174738879054593,
+            0.0029268039949260673,
+            0.010637295711777793,
+            -0.033222781195984666,
+            -0.013301873512972214,
+            -0.0013721703891642354,
+            0.02238421933149517,
+            -0.01530189703335162,
+            -0.014059029663525449,
+            -0.0037393534403749947,
+            0.0010394601933309767,
+            -0.005177359933828505,
+            0.0042489306011644495,
+            0.011039889511148341,
+            -0.014697595240068863,
+            0.007597149702441915,
+            0.0026780496822907634,
+            0.02092734730183164,
+            0.0004688144432243786,
+            0.017181410417231698,
+            -0.0005083970646221274,
+            0.00749639055342427,
+            0.007319420826337891,
+            0.017541463125355265,
+            -0.01002811389296805,
+            0.023582113845081534,
+            -0.008823933397729605,
+            0.0019889258902730535,
+            -0.01257106158422629,
+            -0.028159594067544737,
+            -0.012624750220149823,
+            0.015247564545279736,
+            0.005765911899842925,
+            0.006478299226263732,
+            -0.016537090297144265,
+            0.006276997440048981,
+            0.015647122171767974,
+            -0.002251683151059454,
+            0.010295358869874306,
+            0.009730378299068289,
+            0.00012816405890877098,
+            0.010181120688239766,
+            -0.00801256937783534,
+            -0.00734257767723293,
+            -0.004497520124843868,
+            0.023982754195729983,
+            -0.005443550092842393,
+            0.022434177122972028,
+            0.012647958478747348,
+            0.00616791857573897,
+            -0.010853355357954727,
+            -0.003340111254357573,
+            0.023389634440057473,
+            0.011863246790550376,
+            0.00029642699874398923,
+            -0.009846034060495678,
+            0.0019458876579524617,
+            0.0054010373765486005,
+            0.012983913038716568,
+            -0.02049427270240938,
+            -0.004344571352918878,
+            0.008461947538684664,
+            -0.008472919299762712,
+            0.004318309005722874,
+            -0.030156444522730416,
+            -0.005812303237222271,
+            0.021354352236942434,
+            0.008392281015122372,
+            -0.010232039722494646,
+            -0.004395682831498155,
+            -0.0006527587802367882,
+            0.0040601850269714945,
+            -0.0024323167053212176,
+            -0.0023729138545912176,
+            -0.008541233856441836,
+            -0.006470933419694742,
+            0.014035696617797647,
+            -0.0052691760528871924,
+            0.008597459268477037,
+            0.00012084859681905111,
+            -0.0017255958688825231,
+            -0.007323529232751411,
+            0.03382792341957058,
+            0.008087732655845718,
+            0.005980666974986827,
+            0.0013613625176327891,
+            0.0042244330945601215,
+            0.000616791811672998,
+            -0.0028374856646453707,
+            0.018075360591135542,
+            0.0156738939725117,
+            0.005054696911814988,
+            0.009038525980341447,
+            0.014672270369917403,
+            -0.019700405219230957,
+            0.012625337070556061,
+            0.012184185186281482,
+            0.019382321278525114,
+            0.0007219498737851579,
+            0.002564970959687783,
+            -0.010601873823737926,
+            -0.0029963537053764736,
+            -0.002070567089566093,
+            0.029563028293936092,
+            0.0005903976560069006,
+            -0.010509758568838935,
+            0.010992421945232951,
+            0.020603868018231348,
+            -0.02101157589451638,
+            -0.001971974816402967,
+            0.005963911670920596,
+            -0.010704207628592966,
+            -0.014466113540150018,
+            -0.021269176952661614,
+            -0.013331247081394942,
+            0.0071597345125287174,
+            -0.008162102923896894,
+            0.02367549407145414,
+            -0.014256771777502395,
+            -0.03960796112042958,
+            -0.014803626428839152,
+            -0.00864320347698291,
+            -0.027531337207277483,
+            -0.007305441146142331,
+            0.00965056093889974,
+            0.016698589600806273,
+            -0.003330867073501157,
+            -0.025198703196464594,
+            0.030483790406960656,
+            0.008872821868559751,
+            0.023708422767352917,
+            -0.021464374508407488,
+            -0.0018617153194555344,
+            -0.00886376784514331,
+            0.014505918224188001,
+            -0.017588761818663975,
+            -0.011756112651591795,
+            0.020014632992762026,
+            0.014356419813769714,
+            0.007246557500044434,
+            -0.004355132555323843,
+            0.0067299398462013936,
+            0.007649499235136303,
+            -0.012515181210363267,
+            0.012239792251154642,
+            -0.013499277313916029,
+            -0.024858295126228433,
+            0.014430561986516884,
+            0.002229494935179828,
+            0.009563004428064055,
+            -0.0015984171085659323,
+            -0.016197659608768088,
+            0.010078159490026602,
+            0.00808729175515505,
+            -0.017508840440512225,
+            -0.004793934415922065,
+            0.007133527615305415,
+            -0.020818869260580315,
+            0.031234686259160864,
+            0.011872765152279094,
+            -0.010692949770503467,
+            -0.001989039629661815,
+            0.004431079028880923,
+            0.016511544608975148,
+            0.007893700385963683,
+            -0.01997314388229276,
+            -0.003708006454566117,
+            0.01530128306203401,
+            0.00822588305477249,
+            0.004861703074728471,
+            -0.010836328783594584,
+            0.03434874763691911,
+            0.006082873924472328,
+            4.192481378506604e-05,
+            -0.0052997095110967115,
+            -0.0076108924981838355,
+            0.004196112457289409,
+            0.018905391806961686,
+            0.013330078288599763,
+            -0.004200590783114919,
+            -0.006630107527854746,
+            0.007778448128358027,
+            -0.01706991160178345,
+            0.007329056462125937,
+            0.009390327499494826,
+            -0.012870674959022891,
+            -0.0017571189976160295,
+            0.016462608870101713,
+            0.00209846661139665,
+            0.004942714850185376,
+            -0.011305050739130339,
+            0.004252776146827917,
+            0.015841821944109317,
+            0.018727567181082646,
+            -0.008399642221255733,
+            0.021448055799259867,
+            -0.020972459569228434,
+            0.027243639465535856,
+            0.028499400850980086,
+            0.016573144403275928,
+            -0.01847096443459798,
+            -0.0022409373140116146,
+            -0.0041737366139895414,
+            -0.014761716560267328,
+            0.0034546020161337894,
+            0.005125870074458691,
+            -0.01174186608185822,
+            -0.009577921232380895,
+            -0.024331993794416026,
+            0.011075959943215117,
+            0.007918147902741738,
+            -0.0015045185909419564,
+            -0.011791155919533844,
+            0.016019077577552835,
+            0.010141868950981925,
+            -0.010938402325619172,
+            -0.005635351352447256,
+            -0.019967361901734836,
+            0.009750427567651833,
+            0.009128294017175602,
+            -0.00300556760891084,
+            -0.011496103417643132,
+            0.012977890524465716,
+            0.0014080059843484115,
+            -0.01292380658817521,
+            0.010841954636959441,
+            -0.03142011651894961,
+            -0.005510537354374977,
+            -0.011027026734432255,
+            -0.003620482919145167,
+            -0.014415606143566519,
+            0.003608572766363589,
+            0.0005034611321129689,
+            -0.011470488351158329,
+            -0.0035005774696278,
+            -0.016207453351921627,
+            0.004676327282262429,
+            0.015021513761697039,
+            0.001679614645967629,
+            -0.006917375094634678,
+            0.021994274901326064,
+            0.018518524483306403,
+            -0.021903538417919984,
+            -0.020116013379669034,
+            0.004708084983787958,
+            -0.03261690161800251,
+            0.015019348187107588,
+            0.00966124471577622,
+            0.011676937880751252,
+            -0.028768229226408037,
+            0.007114534979415504,
+            0.00900420165933673,
+            0.016398927481511824,
+            0.0050879301160439245,
+            0.009453255486450082,
+            0.01222758288108498,
+            -0.0104657614414275,
+            -0.003164502778525433,
+            -5.410382753651127e-06,
+            -0.005618225986727351,
+            0.015607620507415484,
+            0.0008527697935056332,
+            0.03603666488542239,
+            -0.005277207843139309,
+            -0.0009799879252482288,
+            -0.011735731235560851,
+            0.00812046801749312,
+            -0.011713349784345638,
+            0.006334172347602515,
+            -0.0024643499727789167,
+            0.0001240009282616789,
+            0.002965067818077541,
+            0.009442572075213767,
+            -0.01086095783353607,
+            -0.007595379345224076,
+            0.007444897256325813,
+            -0.01482814340478338,
+            -0.00019984717819268013,
+            -0.009339343730980451,
+            0.0019680804106275156,
+            -0.02241031895876346,
+            0.006629838013824249,
+            0.040935644507527846,
+            0.02140216465404127,
+            -0.005481040712269146,
+            -0.012431659209559418,
+            -0.004680230194268222,
+            -0.007229300034633229,
+            -0.006482401275991039,
+            -0.004794698160419071,
+            -0.022762405726589375,
+            -0.006928553505112732,
+            0.004320321244630314,
+            0.013300138724212552,
+            0.009741018117644266,
+            -0.008842135469729511,
+            0.004531760574809748,
+            -0.02662650543210291,
+            -0.016578081411548973,
+            -0.0060021425889846495,
+            -0.015840130031005147,
+            0.006125301421471048,
+            0.027521841635711787,
+            -0.0063432924325983525,
+            0.01789356321527744,
+            0.005142029893259461,
+            0.013034833065163448,
+            -0.005338869351330258,
+            0.00035250243782976347,
+            -0.006010938630154189,
+            0.0032958656042048292,
+            0.003074861351445954,
+            -0.0015947217852680714,
+            0.004151590621941346,
+            -0.007139285822575166,
+            -0.0026091937868967284,
+            0.0011629380311555433,
+            0.0117841884590861,
+            0.006877419488576261,
+            -0.00016252653621991807,
+            -0.01761285866765538,
+            0.022221104524808997,
+            -0.018933508559881868,
+            -0.008840161102111973,
+            0.011092684029552064,
+            -0.009660487856185668,
+            0.00702597023411946,
+            0.020387386977378873,
+            -0.006854213562767843,
+            -0.009583981919303215,
+            0.013028181087649146,
+            -0.0036507672754997286,
+            0.004464368543493108,
+            0.009867428978560914,
+            0.006036303973414791,
+            0.002482944923100156,
+            0.013244650094892797,
+            0.004530294330688855,
+            -0.0032625482538717137,
+            -0.020238533752799943,
+            -0.005186498720507101,
+            0.019015752262933585,
+            -0.02252458872825473,
+            -0.0027641215988942563,
+            0.0044506144202026456,
+            -0.0030317840657635457,
+            -0.01733962766050954,
+            0.004205861489436576,
+            -0.018770275720867473,
+            -0.0033883519277721433,
+            -0.008244668812709337,
+            0.009948090119054079,
+            -0.023700732627507225,
+            0.009109315916173436,
+            -0.011739192106106307,
+            0.02356509175177699,
+            -0.003925851077921686,
+            0.024734433836185024,
+            0.0015142645677087268,
+            -0.013482933030628261,
+            0.0055323884713736554,
+            -0.0008930661843169134,
+            -0.004477731568313757,
+            0.029338490404300612,
+            -0.005418258701133154,
+            0.002249996261871466,
+            0.013491124734151694,
+            -0.021332541202417246,
+            0.021773687217263173,
+            0.0046662696076723836,
+            0.01628697326539432,
+            -0.00986238951830334,
+            -0.013799209313714441,
+            0.025178874553620743,
+            0.03324555035648896,
+            0.008398796308070151,
+            0.011261509865513953,
+            -0.01226890254177634,
+            0.018966175575083295,
+            -0.0038328997984309597,
+            0.004344837325292706,
+            0.00878668880458693,
+            -0.007200842513742962,
+            -0.011500562922303248,
+            -0.004533014670236755,
+            0.027639894892383943,
+            -0.021838848944317424,
+            -0.008920629621611768,
+            -0.005739181836252042,
+            -0.01593272996094732,
+            -0.038167534267554606,
+            -0.004371701247332816,
+            0.019604405563120094,
+            -0.005974358141737469,
+            -0.0018341151594086288,
+            0.014334266690716505,
+            0.00402620072221496,
+            -0.002852569009185304,
+            -0.008195218707838913,
+            -0.0027944153624309856,
+            -0.0059390492900516225,
+            0.016651206007022145,
+            -0.004961864528562493,
+            -0.007929830585906699,
+            0.01145431100211762,
+            -0.005731957927601062,
+            0.012130694936714224,
+            0.0004848418056781251,
+            8.392048424584131e-05,
+            -0.0015174120840537315,
+            0.00822932723013239,
+            0.018620869354789082,
+            -0.0055803454666417465,
+            0.025682778991091174,
+            0.006956936033581638,
+            -0.003495709501259274,
+            -0.005489431577052735,
+            -0.0044154293119170445,
+            -0.024561016942073285,
+            -0.0020864636079347357,
+            0.006593060072668424,
+            0.024333813203596122,
+            -0.02515989187381735,
+            -0.004953266000463339,
+            0.020957806164031402,
+            -0.0012192724005613,
+            -0.011978722479596745,
+            0.014690559986694138,
+            0.0041346025614047875,
+            0.009881584250499358,
+            0.012176393203318002,
+            0.015764530703333238,
+            -0.004923548157246672,
+            -0.0015192775000207833,
+            0.012766482928380375,
+            0.0005745973040874939,
+            0.00997809510696241,
+            0.016495444272181515,
+            0.010467514633877887,
+            -0.021108281220544853,
+            0.0015078843789125156,
+            0.00522593579521476,
+            -0.017481430635971396,
+            0.0034362821987740755,
+            0.005397819166037461,
+            0.011626527721895694,
+            0.004048632382026344,
+            0.013207335578216437,
+            -0.015768003434275563,
+            -0.005916168437120099,
+            0.008153798523776654,
+            0.004319451296835019,
+            -0.02599474886669325,
+            -0.01485009085106089,
+            -0.0006781964116146299,
+            -0.004444144921332929,
+            0.013556265879295017,
+            0.01285768162554067,
+            0.02033626001285354,
+            0.024637349990579514,
+            -0.013833060730799217,
+            -0.00842819657792762,
+            0.009436072455719896,
+            0.015560357465711765,
+            0.005798181411384387,
+            0.007874051721372273,
+            0.009297440071118126,
+            0.0077320823375042,
+            -0.004073209576661068,
+            -0.002973928843567532,
+            0.020205003097225747,
+            -0.019725382393637726,
+            0.0006746659838642147,
+            -0.018417443693472958,
+            0.007530796896254817,
+            0.01140696723506382,
+            0.01129515042745398,
+            0.002816883643872417,
+            -0.005274757308627299,
+            0.005716917047812153,
+            -0.06254973656832136,
+            -0.009191453047502128,
+            0.003384751551032943,
+            0.027256466708746898,
+            -0.010885336140129142,
+            -0.003415707910562931,
+            0.007309577090125959,
+            0.005511970708769456,
+            0.0005572239548625122,
+            0.003186427278034852,
+            0.019698716902282903,
+            -0.02052581021967601,
+            -0.010793968505805883,
+            0.004219269922138548,
+            0.0033111837923746746,
+            -0.007269521534951044,
+            0.004805737092302134,
+            0.014143978208501908,
+            0.01986656063456352,
+            -0.0024048854610842748,
+            0.01629335842915328,
+            -0.0015498603704589015,
+            -0.014496994444493684,
+            -0.002671064325909516,
+            0.0037875833047555925,
+            -0.022466433279066646,
+            -0.005980897597195858,
+            0.0043262697442044435,
+            -0.011226121648007667,
+            0.006978495397818936,
+            0.008452177557316863,
+            -0.003018545465557218,
+            -0.0008303307644324117,
+            0.017796789945838845,
+            0.006591364879905198,
+            -0.013464486184025026,
+            0.0028668426952326957,
+            -0.023377605222213174,
+            0.004325185794796361,
+            0.001416245546006914,
+            0.007809449561523174,
+            -0.008431335139468173,
+            -0.014774499666285603,
+            -0.005933810712528226,
+            0.017746847020610673,
+            -0.01965188730040385,
+            -0.002406875192894708,
+            0.01683009662984443,
+            -0.0002518220110313279,
+            0.005456812494894182,
+            -0.02011241622618092,
+            -0.0009782974741561381,
+            -0.02933296756078644,
+            -0.01171823509606186,
+            0.001162345317872531,
+            0.00892221099991475,
+            -0.006551308022163148,
+            -0.03934356275100932,
+            0.00427623835776202,
+            -0.013111664387776261,
+            -0.008752857644809231,
+            -0.010214546496134218,
+            -0.00518087722852466,
+            0.015240297918667867,
+            -0.012498100033235827,
+            0.01086490299272505,
+            -0.00917777212602538,
+            0.005895055740265625,
+            0.0090554651297836,
+            0.014212841798768113,
+            -0.005544047915369086,
+            -0.0033589999440643385,
+            -0.007924830667307956,
+            0.0022291797354234005,
+            -0.0032976226056924502,
+            0.009615674261248298,
+            -0.0051932155872883,
+            -0.0065636389655770455,
+            0.02426106498596339,
+            0.011445591067574932,
+            0.013312596099532533,
+            0.0055118624272110965,
+            -0.007220620072349783,
+            -0.01916551187613823,
+            -0.003977616240384473,
+            -0.013204506708246246,
+            -0.006315966975649096,
+            0.01434099854765068,
+            0.012588249778961223,
+            0.006357972880218866,
+            0.0007747002406996095,
+            -0.004794439624135275,
+            0.006975593520375694,
+            0.0262335496080837,
+            -0.016481909971520166,
+            -0.0013084662210086592,
+            -0.01369313900978409,
+            0.01921748474212319,
+            0.03333761672336437,
+            -0.0008918419419301244,
+            -0.007699525186523558,
+            0.013757216065348164,
+            0.00819045303047094,
+            -0.007411380297164911,
+            0.02453611080891859,
+            -0.025045462880463647,
+            -0.015989319900023054,
+            -0.011650886878708265,
+            -0.011407253985176972,
+            -0.008386367959227559,
+            0.016331961716602287,
+            -0.00141225438918211,
+            -0.0018505754298268654,
+            0.009557291617446645,
+            0.03357924917013024,
+            0.006916860901290756,
+            0.007030308953738567,
+            -0.002739345982346728,
+            -0.009376522438071562,
+            0.011482637968411055,
+            -0.007259054391879614,
+            -0.04075661435928125,
+            0.004250195164583401,
+            -0.005160505473755424,
+            0.02405623169060588,
+            -0.02197586561375289,
+            -0.00619175761282745,
+            -0.006736129049831973,
+            -0.028200015908139763,
+            -0.005867722104196303,
+            0.02805071895018145,
+            -0.015402198567775478,
+            0.014773294766214671,
+            -0.009502185346558072,
+            -0.0013419858580438616,
+            0.004443749010136947,
+            0.013008783570355656,
+            -0.02289662952366337,
+            0.0013834888740540108,
+            -0.00048549197205067054,
+            0.027969842651517314,
+            0.010437729090975158,
+            -0.0047705656249623744,
+            0.014202218020412936,
+            0.008240861154198411,
+            -0.018637315941614406,
+            0.007314019317343154,
+            -0.005747966765101792,
+            0.014833896739762147,
+            -0.01314847940891154,
+            0.007762100494499689,
+            0.00705026393499352,
+            -0.01278023669088385,
+            0.03016342404000438,
+            -0.015204129985039077,
+            0.007776946006562982,
+            0.009278778571285689,
+            0.009656723862179986,
+            0.01257686993982689,
+            0.004362439504889069,
+            -0.02898418193036562,
+            0.0002144431781454467,
+            -0.03828100004144633,
+            -0.00833748467467171,
+            -0.002611937107022567,
+            -0.002145329666219461,
+            0.007050630951707937,
+            -0.009383645162261629,
+            -0.033976442057557676,
+            0.0006429978367640709,
+            0.01603027598131823,
+            0.017501067413108286,
+            0.0002196997337451293,
+            0.013075379412332973,
+            -0.006197537539554383,
+            -0.021148120787551232,
+            -0.016986448660412497,
+            -0.010207705210051413,
+            -0.02122194744719993,
+            -0.020550470201191853,
+            -0.00908564461857315,
+            0.012090079127547193,
+            -0.009927390005879807,
+            0.013888458967497627,
+            0.029513250508042935,
+            -0.024529175134769883,
+            0.0011144447737101862,
+            0.021977677894839922,
+            -0.00198308986575084,
+            0.014108189311291326,
+            0.001696775006906642,
+            0.014122064251288995,
+            -0.0004166140484018266,
+            -0.010556016685519523,
+            -0.009729362492661684,
+            0.017487929296614076,
+            -0.0026043257634797124,
+            0.016715798948897194,
+            0.012183487338360088,
+            0.003771721868999568,
+            -0.0030887189945795796,
+            0.010667776697461199,
+            -0.026471157050402344,
+            0.014249996582064706,
+            -0.006690961709923224,
+            -0.0181382691440242,
+            -0.008729104191096461,
+            0.003882731567105777,
+            -0.005609384377642956,
+            -0.007794812949125115,
+            0.012967089730551055,
+            -0.00944374111323783,
+            0.02088525981049842,
+            0.0023400121055514958,
+            -0.012196925240774426,
+            0.003648688547842322,
+            -0.03322041209304968,
+            0.0015191765655963997,
+            -0.018402382776013956,
+            -0.024659631472836927,
+            0.03840173429666435,
+            0.011991452465253198,
+            0.006639702644606365,
+            -0.01675838115087676,
+            0.007953102302963526,
+            0.0004257479565099406,
+            0.017484338819046517,
+            0.011068864013925452,
+            0.006466915839538296,
+            0.008364454017976605,
+            -0.023316137291380285,
+            -0.010898421318668592,
+            0.016204569093730564,
+            0.012099473359791148,
+            -0.01017229369980751,
+            0.019431669546223253,
+            0.017955119269367787,
+            0.007300372071724306,
+            -0.006898619924677706,
+            0.01647927363640854,
+            0.021012103522831126,
+            0.010291016618557653,
+            -0.00047955552103474145,
+            0.0008684297441778696,
+            -0.010534247928997246,
+            0.005037930235889528,
+            0.035417626740307205,
+            0.014712595692093886,
+            0.010650306363963653,
+            -0.03626584475133921,
+            0.021764456453493054,
+            -0.011961521673661574,
+            0.009092981192874177,
+            0.006089104887183293,
+            0.017982801101846038,
+            0.006916969397519662,
+            0.014196254958707752,
+            0.011745069030657836,
+            -0.006689305450954901,
+            0.025205280281804708,
+            -0.001601414806100461,
+            -0.011992525142592405,
+            0.003386930324498371,
+            -0.01859193251438437,
+            0.005303261319773131,
+            -0.0041710536965100615,
+            -0.004459393120257307,
+            0.0032461828144552565,
+            -0.0018650956576455287,
+            0.015678225251799887,
+            0.003894142628084471,
+            -0.02355212701026675,
+            -0.0014873755115206368,
+            -0.024284295126348117,
+            0.0027260592623277375,
+            -0.022185571998195344,
+            0.00022883605560765203,
+            -0.005557568353382136,
+            -0.009749382813201403,
+            -0.025385857431333838,
+            -0.009853073898348951,
+            -0.0035843255959309307,
+            0.007661137888845632,
+            -0.010521037433807653,
+            -0.004246698084422667,
+            -0.0019920499321964815,
+            -0.019465113363548715,
+            0.009014255846983305,
+            0.027122533756959888,
+            0.016848221859975445,
+            0.027170883580103264,
+            -0.00019554112591107156,
+            0.013614675849485926,
+            -0.002066822813103284,
+            -0.01603063005944364,
+            0.008824663429777616,
+            0.0075653522381218495,
+            -0.00531780740880296,
+            0.014643348459785112,
+            0.0072634484937344704,
+            -0.012800149015319017,
+            0.009859503660384916,
+            -0.007161712176469609,
+            -0.006437857349742842,
+            -0.009764856421740622,
+            -0.02058868349990357,
+            -0.014866137971603676,
+            0.0012450207572602557,
+            -0.002123265406747006,
+            0.006000473862248991,
+            -0.004054016790850226,
+            0.009584879012937315,
+            -0.00608816807208866,
+            -0.00925801057439376,
+            -0.013962683072792106,
+            -0.008924356118158748,
+            -0.004210064145949253,
+            -0.004381709838058668,
+            -0.011732699976353612,
+            -0.005479523097469261,
+            0.0010680258077037758,
+            -0.0019401640391035755,
+            -0.01397652201595301,
+            0.011650839243603377,
+            0.0058731838291845645,
+            -0.002462743134993977,
+            0.02219008688925881,
+            0.01097316136828334,
+            0.0018083001685932406,
+            0.01366095798812196,
+            0.0003942601242216818,
+            -0.0064063532958756175,
+            -0.03343312148366622,
+            -0.0005083355836126291,
+            -0.02815586367527057,
+            0.025770078162165123,
+            -0.0069114949102666145,
+            -0.010322087105114614,
+            -0.002519047562374146,
+            -0.006063669644021698,
+            0.005873609611264377,
+            -0.002722940652283699,
+            0.0021229525800901114,
+            -0.023673778230768945,
+            0.0028629715379660113,
+            -0.003953653437464966,
+            -0.0036366957271968823,
+            0.007469664864329332,
+            0.0346520690389124,
+            -0.004054282838167587,
+            -0.0034765253405897676,
+            0.005159466707114532,
+            0.011642913651743502,
+            0.027845571220425726,
+            -0.004445196354520585,
+            -0.0032038508043339413,
+            -0.02233838584152591,
+            0.0054112576531374175,
+            -0.00019544838850471866,
+            -0.004733192299951088,
+            0.016664217348411003,
+            -0.016719175418727594,
+            -0.01631658832047864,
+            -0.00545969677717507,
+            -0.014570848053102338,
+            -0.00022620258237138677,
+            0.014454101124666536,
+            0.011056312437637014,
+            -0.004035526363810078,
+            -0.012459378363360557,
+            -0.0022595493558020706,
+            -0.030370952502579162,
+            0.012766010284884204,
+            0.1141689619126009,
+            -0.00766197047210299,
+            -0.00896655658602649,
+            -0.009481112393590025,
+            0.001435269335646347,
+            0.001338755472877713,
+            -0.011739143028767118,
+            0.07166096084053437,
+            -0.0077855767070652395,
+            0.0002139580430342728,
+            -0.011312198831181402,
+            0.001844005266872301,
+            -0.001825412693163496,
+            0.01371932421024605,
+            -0.005025814503687022,
+            -0.018213360991353643,
+            -0.0033612481616769416,
+            -0.02941403830641238,
+            -0.0041137865872933515,
+            -0.005683630256743215,
+            -0.004288198570156578,
+            -0.014982999363445122,
+            0.0067149665791261335,
+            -0.019661973497372313,
+            -0.012688556129040908,
+            0.001124391650175233,
+            -0.0033623453063430534,
+            0.00996944383866679,
+            -0.020496336814603385,
+            -0.0044354321069609536,
+            0.0019875840782527733,
+            0.009706675240563883,
+            0.0011600364428219214,
+            -0.012755178230821602,
+            -0.0026514442584739412,
+            0.004056350166031111,
+            0.01909951327913368,
+            0.014360281326789544,
+            0.003732344102045803,
+            0.008623457695718206,
+            0.014461868085333023,
+            -0.001692149528228451,
+            0.01948035376572433,
+            0.01172550239477637,
+            0.0031966531144076173,
+            0.0004779413449255626,
+            0.0030912562273611517,
+            -0.004597633235071031,
+            0.01582547400214653,
+            -0.014811177637844808,
+            -0.02411202772171917,
+            -0.0071272238837422185,
+            0.018015541137524113,
+            -0.013081287852028074,
+            1.8837689271256874e-05,
+            0.004062803284646432,
+            0.027569726868200254,
+            0.0010564850664255057,
+            -0.01179755720247972,
+            0.0028876134800690334,
+            -0.01638215925131053,
+            0.022944183106912663,
+            0.0026043713064243836,
+            0.03279173899194346,
+            -0.005103903273968361,
+            0.00578213669463383,
+            -0.020744194169889544,
+            -0.007588793365216579,
+            -0.00819862611489168,
+            0.0017401442433921013,
+            0.010993500174794084,
+            0.005411238350013134,
+            0.01620925462604661,
+            -0.00641615069699833,
+            -0.020751730783437995,
+            0.014776878596707538,
+            -0.009552193304486048,
+            -0.03393319554458277,
+            0.008881987295131347,
+            -0.029102297123590998,
+            -0.0045267103602711,
+            0.00368971742585232,
+            -0.014269341759285646,
+            -0.016264809810970175,
+            0.012837302095591106,
+            -0.0011890334180728564,
+            -0.008937992768323412,
+            0.0014444613725248846,
+            -0.017110187133760626,
+            -0.02385668037818897,
+            0.006017405364919834,
+            0.00676670672597817,
+            -0.0077994500447595,
+            0.018312452858615755,
+            -0.027015660037024585,
+            0.006123912473734307,
+            -0.012973676426738582,
+            0.009041974762537394,
+            -0.01580183887345168,
+            0.01163516332546347,
+            0.010887662470499361,
+            -0.001503467664709027,
+            -0.0076234124970223326,
+            0.01231202541338883,
+            0.004852720134894811,
+            0.033760365210093034,
+            0.028658326381800395,
+            0.000995999783627898,
+            -0.0025605367058226737,
+            -0.0020356794893366384,
+            0.011667525096213252,
+            -0.014049700287539817,
+            0.011218246246280464,
+            0.02658113044439717,
+            0.017568631325838108,
+            -0.019133062654008885,
+            0.018050678772708717,
+            -0.010356118256625172,
+            0.012872052420999813,
+            -0.025620154415048252,
+            -0.005135768485044442,
+            0.004936490920616527,
+            0.005163008729256001,
+            0.007425278892536464,
+            0.001410212306439329,
+            0.021891387289347516,
+            0.014999616600766989,
+            0.00027613811156650413,
+            0.00558422822386443,
+            0.01477009526981092,
+            0.010497782599293994,
+            -0.0005567894749233204,
+            -0.00467776506604011,
+            -0.016998035358377343,
+            0.00957271797745408,
+            -0.006065116857898469,
+            0.005138070401846389,
+            0.01698914855850691,
+            -0.01230257809343985,
+            0.011109844802663414,
+            0.007518553660463051,
+            -0.015504773927595823,
+            -0.017169933945839305,
+            0.005552829003043926,
+            -0.02252633790984703,
+            -0.01306722099657118,
+            0.02715689072730398,
+            -0.010513941399990838,
+            -0.02790582732974509,
+            -0.0009847768069205477,
+            0.009797865241108237,
+            -0.002772952073769185,
+            -0.0027968741526000576,
+            0.0037541733651479613,
+            0.010801137722284016,
+            -0.014658156970099418,
+            0.002075415233345957,
+            0.004735264309137368,
+            0.01930804315461151,
+            -0.0014151522239496115,
+            -0.015046118414100617,
+            -0.002288495894227231,
+            0.021592829677937263,
+            0.017174609713578173,
+            0.00265560378681856,
+            -0.016499825201963983,
+            -3.965296928410772e-05,
+            -0.0074142747014567635,
+            0.00021360696629503726,
+            0.00027431174904249525,
+            -0.0056391089282546054,
+            -0.018931681700770225,
+            -0.003888248121485032,
+            0.011181675419027736,
+            -0.00247252920175839,
+            0.01977860312223846,
+            -0.0011527575128223392,
+            -0.015763587433200915,
+            0.001960734212846797,
+            -0.005280860341480037,
+            -0.010372829570478649,
+            0.01351900743232294,
+            -0.009359942374603997,
+            0.006788746813046239,
+            -0.003212015225284119,
+            0.004611143085183367,
+            -0.0003308099698088422,
+            -0.017017265064185255,
+            0.02924142709831441,
+            -0.0018238549856433466,
+            -0.004769274041546295,
+            -0.0014058363976586562,
+            -0.033393331166741945,
+            0.015017165991303838,
+            -0.014574579250749552,
+            0.006745699409061803,
+            0.002705853802154516,
+            0.01272741296341735,
+            0.003253095246601883,
+            -0.008898616461779179,
+            0.0027485606928249326,
+            0.017040360927723885,
+            0.013756402798365262,
+            -0.007292864293090161,
+            0.008694855886257706,
+            -0.0028934389922887676,
+            -0.0037483208383542015,
+            0.01363963339139356,
+            0.004170611039133969,
+            -0.006282657671540625,
+            -0.0009784372731140945
+          ],
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "Mixture-of-Experts (MoE)",
+            "fine-grained expert segmentation",
+            "shared expert isolation"
+          ],
+          "title": "Deepseekmoe: Towards Ultimate Expert Specialization In Mixture-Of-Experts Language Models"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "addd7c61-bdd3-470e-8c4f-439af824f89e",
+        "properties": {
+          "page_content": "F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+          "embedding": [
+            0.0002445570513886185,
+            -0.025035678422212698,
+            -0.026056660824720954,
+            0.0021234386221176942,
+            -0.007526403638487191,
+            0.018963965001103752,
+            -0.007728070932730228,
+            0.062213823247851435,
+            -0.00501600735659763,
+            0.031461940272958254,
+            0.014585783857839997,
+            0.010118033107401152,
+            0.009983318452435566,
+            -0.03282616973330728,
+            0.015192449242879527,
+            0.01480596701302375,
+            0.019405922018728892,
+            0.022192910412596127,
+            0.008048967251594976,
+            -0.0367043822293015,
+            0.017737679976556845,
+            -0.022757028540067756,
+            -0.06290354368907101,
+            0.016842087557557506,
+            -0.0050319451760917995,
+            0.019311667659179976,
+            0.008548884527537198,
+            0.02344214718766695,
+            -0.042417710048062586,
+            0.01950972309054391,
+            0.021637767293446648,
+            0.010581067128107188,
+            -0.014214247204390872,
+            0.01016365063627519,
+            -0.025782209417014867,
+            -0.02611397356963557,
+            0.034251067799009986,
+            0.0071402676991737055,
+            -0.020147891665364803,
+            0.056472902950962155,
+            -0.006027534100459245,
+            0.005473493744726429,
+            -0.03139313966858068,
+            0.00043634965151753214,
+            0.013171990459657583,
+            -0.016474119463665594,
+            0.0075792906007691105,
+            -0.0032442837467945356,
+            0.02492845005795393,
+            -0.04366675539501486,
+            -0.006033838756045725,
+            -0.018660330752604155,
+            0.02088611082654532,
+            -0.019387265112916365,
+            -0.01222746328548615,
+            0.005977009213829137,
+            0.0012690574665308455,
+            0.05430319716489232,
+            0.058075720286678886,
+            0.02554562736016709,
+            0.02997758641056458,
+            0.00014178556329408164,
+            0.004139467932725944,
+            -0.004454862143349584,
+            0.031094774362479788,
+            0.0017837745263941452,
+            -0.013542834385584962,
+            0.01983846902748998,
+            0.010136662804503968,
+            -0.0387179389182337,
+            -0.06156798393873734,
+            0.015844092498186847,
+            -0.020820695961906926,
+            -0.02678200120240037,
+            -4.570339833017677e-05,
+            -0.015544616932001905,
+            0.005434882475939711,
+            0.050835936350804696,
+            -0.004985250023269435,
+            0.0016509410141492726,
+            0.022801366331672433,
+            -0.004080282248725238,
+            0.0078367492683837,
+            -0.00034876063727630653,
+            0.039772801289415724,
+            0.0288510926430687,
+            -0.022609202992981695,
+            -0.07091181813750308,
+            0.045194222785472735,
+            -0.03225547853169315,
+            -0.032585463285183205,
+            0.01670145480899693,
+            0.03286138231600811,
+            -0.009733112712376757,
+            -0.030197857075781466,
+            0.0731500616714208,
+            -0.005545147701735842,
+            0.017072915518860705,
+            -0.021002529816498382,
+            0.010525361666209598,
+            -0.012745643075817463,
+            0.025352505498532282,
+            -0.003102906957075941,
+            0.016391641977220053,
+            0.02534354270624064,
+            -0.008108690979345334,
+            -0.006451703723989414,
+            0.0065097623653779324,
+            0.04453924610212246,
+            0.0889545538715994,
+            0.019774398083206815,
+            -0.003981259900744729,
+            -0.046615274629371053,
+            0.00732791197790897,
+            -0.02916202133853672,
+            -0.04303144138453202,
+            -0.04367640508681853,
+            -0.010910501188273782,
+            0.023755744360759886,
+            -0.003399898430475389,
+            0.03910227884474275,
+            0.04137187598162877,
+            0.03363334616934253,
+            0.00916121492790106,
+            -0.031686589382074935,
+            0.026907320660055866,
+            -0.01693827759642556,
+            0.002542422168436765,
+            0.00734783450287368,
+            0.008189080458799796,
+            -0.006352648760821389,
+            -0.002333605867785783,
+            -0.03831808898525347,
+            -0.036762698962852246,
+            0.012773386000855496,
+            -0.008349610778914014,
+            0.027936454763857345,
+            0.07015560257394027,
+            -0.03713244149832974,
+            -0.0066245405611366675,
+            0.02802618921440782,
+            0.0017169880395367697,
+            0.023346396749133396,
+            0.007605522879978981,
+            -0.023484167590494896,
+            0.008594274917753499,
+            -0.0071765874950466295,
+            0.0677537727828362,
+            -0.008126000179550069,
+            0.011046754551416857,
+            -0.019516727128214784,
+            0.031654663874166704,
+            0.0007057013357055938,
+            0.021535694192642306,
+            0.03313372731162823,
+            -0.005794425715079591,
+            -0.03865395674172617,
+            0.032271538528276474,
+            -0.029790294001121286,
+            0.0013647166052139265,
+            0.014194415049797637,
+            -0.004899479294176156,
+            -0.015920769574443834,
+            0.0013708196727143433,
+            0.029757298661028133,
+            0.045717552753231847,
+            0.042911032378569906,
+            0.0016019404554248433,
+            -0.06844613496660937,
+            0.06365820914938357,
+            0.012895947255192629,
+            0.03208498337942363,
+            0.009724409704032419,
+            0.042361427690196844,
+            0.008120380964731038,
+            -0.0004384537671641681,
+            -0.005540416382489673,
+            -0.005125920468958917,
+            -0.021186731704155592,
+            0.03258917022091572,
+            -0.01443327736487225,
+            -0.007107908179884635,
+            -0.00206346150761894,
+            0.014042865018508809,
+            0.028517557440008168,
+            0.017283745982594707,
+            0.0029564774390786045,
+            0.04378212980825259,
+            -0.019454732028746872,
+            -0.009437148551161395,
+            0.018820938021162286,
+            0.0008903198503990673,
+            0.025297040557527045,
+            -0.0011429799732923295,
+            -0.01670583390791835,
+            -0.009421860087826363,
+            -0.0328764813310764,
+            0.009563088287577815,
+            -0.0015347427243049344,
+            0.019228666906882602,
+            -0.007174098839673089,
+            -0.04068735475327182,
+            0.008903031576514094,
+            -0.011472954071988202,
+            0.015735403376626513,
+            -0.007084760348074423,
+            0.008729329609340102,
+            -0.00638454270428521,
+            0.016210007209816298,
+            -0.02460475066044075,
+            -0.009047036293783832,
+            -0.022598561172568143,
+            -0.020144302113939427,
+            -0.010886272544512816,
+            -0.010886770387412563,
+            0.008591903125597681,
+            -0.012216526799178626,
+            0.023956058440750026,
+            -0.017402060516065108,
+            -0.018328740623422317,
+            -0.00046940812181371226,
+            0.022921229116282273,
+            0.009026508358954174,
+            -0.00446208425714566,
+            -0.04421346673289319,
+            -0.01214251475197077,
+            -0.016860336137191526,
+            0.005985059770480569,
+            0.02804423359971541,
+            -0.03510963462663318,
+            0.012588036039507566,
+            -0.0018909139908325232,
+            0.0019847602212350957,
+            0.011374423039138836,
+            0.015486762163219968,
+            -0.027040748338117617,
+            -0.016329845569406535,
+            -0.012600824227928017,
+            -0.01765236203915568,
+            0.01673827406092187,
+            0.0016641690585652725,
+            0.003997011410859195,
+            -0.00499922800986537,
+            -0.02782499674212141,
+            -0.008774389172636544,
+            -0.011506812701874534,
+            -0.009783555199790996,
+            -0.015740884114025612,
+            0.013493654174963239,
+            0.014254624157312108,
+            -0.033270488877124596,
+            -0.018446763439664365,
+            0.02247061755496804,
+            -0.0035396859846893343,
+            -0.03759408156279411,
+            0.04544560120093284,
+            0.052893430652542234,
+            -0.02220682841881151,
+            0.002828355769475809,
+            -0.03469141962159207,
+            0.05950248906327849,
+            -0.03444371252359212,
+            -0.03378776392860449,
+            0.06521072243320349,
+            -0.009713026070923203,
+            0.015249223153809483,
+            0.03131206469454657,
+            0.0034865285730464384,
+            -0.0025997752599464445,
+            0.00912836988165194,
+            0.033031075123542926,
+            -0.014878258361522118,
+            -0.013101723481793925,
+            -0.01797681408884685,
+            -0.002397237505725697,
+            -0.03438738791694146,
+            -0.021713511748922477,
+            0.00836495266415799,
+            0.013064910474172053,
+            0.017110113503472666,
+            0.017514586931181946,
+            0.016195079102773242,
+            -0.0024109742901844617,
+            0.01545592172528996,
+            0.001046760697009891,
+            -8.863738247355372e-05,
+            -0.010979717992873124,
+            -0.017762086340346584,
+            -0.052667758929128256,
+            -0.018996338856872796,
+            -0.04421628171000429,
+            -0.005982494173649518,
+            0.0012420734363470585,
+            0.009886466834005627,
+            0.04949340056515024,
+            -0.01295176306623405,
+            0.003508142432105508,
+            0.010930632194984236,
+            0.01019261004578912,
+            -0.006556730312565122,
+            -0.032765993356945154,
+            -0.006485728779853869,
+            0.025709969056450885,
+            -0.022932446214570953,
+            -0.021792413124305354,
+            -0.007047777916806282,
+            0.0038659532636633486,
+            0.02321123092409853,
+            -0.026003302747277822,
+            0.017643847938685852,
+            0.010129447679208158,
+            0.009551458127273026,
+            -0.017747795509094742,
+            -0.02891499441428287,
+            -0.030321565228144824,
+            0.039350178167852384,
+            0.02899873253152933,
+            0.011338359735211436,
+            -0.033634061116901,
+            -0.013629902520109783,
+            0.0376352218933715,
+            -0.0019251015746424632,
+            -0.07267257827375968,
+            0.036685408572352814,
+            0.0050526642351482985,
+            -0.00019289056191322955,
+            -0.013581714265389013,
+            0.05300479823846402,
+            -0.04139717386779843,
+            -0.03335524761191713,
+            -0.00920987783097114,
+            -0.022616612319255443,
+            -0.005274093640024995,
+            -0.0012667939280239298,
+            0.032502321377997974,
+            -0.020089629343721754,
+            -0.024354491736044894,
+            -0.010206744478678104,
+            -0.02465036256948816,
+            0.017966455981144307,
+            -0.012995995315432267,
+            -0.02135780149071486,
+            0.010967719068923269,
+            -0.03402154331207373,
+            -2.0626689329697705e-05,
+            -0.01092309016851862,
+            -0.018683564965672157,
+            0.01571041043301007,
+            0.016259448102797114,
+            0.015403085986247105,
+            0.0020047641027155683,
+            0.015172380396260792,
+            0.008099191720720747,
+            0.004074377809743208,
+            0.01893667379047588,
+            -0.03384452762579117,
+            -0.038822883991989264,
+            0.050315952008617104,
+            0.0008458969404873289,
+            0.0012049589464131246,
+            0.006406225290437734,
+            -0.0042817559898658255,
+            -0.0411169553122232,
+            0.012933323148113837,
+            -0.05784927886369837,
+            0.006262727192870561,
+            0.012928199517490538,
+            -0.01786898028943474,
+            -0.02291183998443847,
+            0.00046953379111417065,
+            -0.024864892714864046,
+            0.02151263275001755,
+            -0.017247234755598105,
+            0.020405349906524557,
+            -0.023068655052252782,
+            -0.006872857342813033,
+            0.05842934123173711,
+            -0.02308642746390288,
+            -0.04925938088613823,
+            0.032829595473385986,
+            -0.007184802367095134,
+            0.03878513331161335,
+            0.055971017386540985,
+            0.016337316575743225,
+            0.006439501224944926,
+            0.026084366083114555,
+            0.013204436213767278,
+            -0.012028383848550473,
+            -0.0383295105300241,
+            0.015805232298071025,
+            0.030751727443425712,
+            -0.04685643586863969,
+            -0.010258856888449854,
+            -0.02541648783174773,
+            0.01599307149086869,
+            0.020046184897703115,
+            -0.0044598427609889935,
+            -0.0012270315730225135,
+            -0.009664326045898607,
+            -0.01520444928791987,
+            -0.02631191309701645,
+            0.02193635473056903,
+            -0.013740214173212537,
+            0.0007504924364311022,
+            -0.001111971939163715,
+            0.0012797636039507028,
+            0.008478066024469051,
+            0.01501219347281274,
+            0.027440349509054086,
+            -0.022728634568050663,
+            0.006916913421268421,
+            -0.008499438828073956,
+            -0.03788213665656746,
+            -0.006718262893819488,
+            0.029198375723488434,
+            0.006169843558767785,
+            -0.016427547826608675,
+            -0.0049009468913267205,
+            0.0022509753971959087,
+            -0.016307639896946134,
+            0.00372456949880952,
+            0.010347360137213142,
+            0.035374109073612686,
+            0.01856811791368892,
+            0.019247903053478954,
+            0.0035591273657627456,
+            0.010292179556783772,
+            0.004040044337368795,
+            0.025869815038962875,
+            -0.04526297785801751,
+            0.030173313035569235,
+            0.01521363974209499,
+            0.009059160353158267,
+            -0.03914219183565228,
+            -0.02367109515190293,
+            -0.025737979702884085,
+            -0.021654077593318197,
+            0.02346511753268162,
+            -0.02730833269330862,
+            -0.006289840626554075,
+            0.006531934540756005,
+            -0.0006804032161296996,
+            0.003928945341546481,
+            -0.0033962179480786624,
+            -0.037975452991752895,
+            -0.009840510213526213,
+            -0.04023861493405766,
+            -0.0241245373830213,
+            0.003822063151525241,
+            0.003249755356115233,
+            -0.06594891648887229,
+            -0.001768385222533634,
+            -0.06222620899240395,
+            -0.013594921028316139,
+            0.03441942883093182,
+            -0.024438842229617237,
+            0.04820486752403123,
+            -0.05486229161412509,
+            0.018219866569279347,
+            -0.046562594596398506,
+            0.011354841468658838,
+            0.021738373879790916,
+            0.04347115691988908,
+            -0.007572536447300832,
+            0.014626550834376365,
+            0.009906079088662038,
+            -0.04530647508884267,
+            -0.04520033458696902,
+            -0.011400230794947595,
+            0.011239630851440557,
+            0.003287355301854498,
+            -0.012304101523783504,
+            0.02623659451177029,
+            0.015903981896266153,
+            0.04300392046214319,
+            0.03414364576071245,
+            -0.0005691335103219632,
+            -0.049506948352089984,
+            -0.027336474868358992,
+            -0.014910043127825028,
+            0.02552137904202476,
+            -0.03055258097953423,
+            0.003401887357920115,
+            -0.03228200232009397,
+            0.013338682663233296,
+            0.018207379090305786,
+            -0.02224210534533673,
+            0.032569850137027664,
+            0.014446434357859929,
+            -0.040301668913492676,
+            0.0012399866793165108,
+            0.0002450513991178417,
+            -0.038300625347349834,
+            0.008119751344998158,
+            -0.013712452991626932,
+            -0.05381257202269606,
+            -0.022199278450525977,
+            -0.014717114803771204,
+            0.02206967996899982,
+            -0.0023470596291977116,
+            0.017222681546831476,
+            -0.029965631581610256,
+            0.0053103260868707786,
+            0.039279027237044066,
+            -0.001286692479487674,
+            -0.01148541970963179,
+            -0.0008706049086180766,
+            0.022795196067681996,
+            -0.01968201411368477,
+            -0.045534614872904305,
+            0.009877856016973188,
+            -0.020566606587457816,
+            0.016002665913994348,
+            -0.015445999456355206,
+            -0.0052940252527317235,
+            0.010360748741468617,
+            -0.03170100753615497,
+            0.02004511444257144,
+            0.009722541379665509,
+            0.006411072442536516,
+            -0.017460683536795665,
+            -0.0033423349847155888,
+            0.015743901543629382,
+            0.047005401368949463,
+            0.0007329338628268773,
+            0.006153613674047577,
+            0.02501606022399451,
+            -0.013054974002495664,
+            0.05036387123988784,
+            -0.02807526639433245,
+            -0.03099731580589117,
+            0.04344605263104773,
+            -0.019489740746149623,
+            -0.017636713524640496,
+            -0.017960525325193457,
+            -0.009792405274000215,
+            -0.0062801692104839674,
+            0.02801421653619339,
+            -0.00712206060099242,
+            -0.005749470909187103,
+            0.020504942834135256,
+            0.013170883165927672,
+            -0.011218763497707175,
+            0.008523945766715354,
+            0.01511514654097483,
+            0.027931571407773977,
+            0.03214582418497515,
+            -0.02345593941074465,
+            0.0333746849861944,
+            0.009324947617983663,
+            -0.004234113006464689,
+            0.005131158074842622,
+            0.0020437263732432953,
+            0.00176803337042029,
+            -0.029372739581870692,
+            -0.01539406767336474,
+            0.015958225909710817,
+            0.007396588215584237,
+            -0.003291618431912445,
+            -0.0316834931316845,
+            -0.0028368107695447525,
+            0.02614797456533501,
+            0.003832488818435695,
+            0.026426519154286857,
+            -0.015605455255333021,
+            -0.02018445776455915,
+            0.008876565653043164,
+            0.023669947107607083,
+            -0.055401884612295824,
+            0.01882053182190406,
+            0.011046506385076857,
+            -2.6003594081234644e-05,
+            0.01591257026228522,
+            -0.0046432737555961385,
+            0.005094314300671912,
+            0.0482159962414822,
+            0.015126643838581288,
+            -0.013318945197936635,
+            0.0020418779929576307,
+            0.015136735547940301,
+            0.001727083579122356,
+            -0.020116332736417462,
+            0.002896210161586164,
+            0.012858057228188658,
+            0.0092589465483404,
+            -0.012984552574495295,
+            -0.005651788746934606,
+            -0.016856929103895944,
+            0.03249171052801259,
+            -0.005843029913406538,
+            0.02336833175997056,
+            0.04800276014595125,
+            0.02261889070736121,
+            -0.03578655846417886,
+            -0.011451513080556048,
+            0.0077334037670644135,
+            0.009415939321950421,
+            -0.012399032514032437,
+            0.006084821107593102,
+            -0.034655634244304866,
+            0.013104268092544735,
+            -0.0016781301605750378,
+            0.012249966178417542,
+            -0.025567673456449492,
+            -0.023488408901029257,
+            0.007732021927617865,
+            -0.006986715221001125,
+            0.01461433573263053,
+            0.00011845838735106802,
+            -0.005506714145045683,
+            -0.006827219009979501,
+            0.0005242424465709722,
+            0.002885486381705293,
+            -0.0042669493792680474,
+            0.021618593454103052,
+            0.00010034640948597114,
+            0.011602625527297699,
+            -0.023477647311478233,
+            -0.0162254255990209,
+            0.031352657169434826,
+            0.03529846802317018,
+            -0.0007059281329032357,
+            0.02568807107901636,
+            -0.00277252926066966,
+            0.010828396988798257,
+            0.00810821945842862,
+            -0.010307381937016405,
+            0.0003407672690264756,
+            -0.003768030765932326,
+            -0.0027367413382233317,
+            -0.012887542072043335,
+            0.0031930180018598015,
+            -0.014086667286395425,
+            -0.01504545079945656,
+            0.02010182276528891,
+            -0.002519095978589373,
+            0.011477055159845283,
+            0.006165264376334778,
+            -0.01139265727997407,
+            -0.016539552549751473,
+            0.01574258576985793,
+            0.05233886430069725,
+            0.022428487725773775,
+            0.044334901638531425,
+            -0.00706941862193002,
+            0.011585028247041065,
+            -0.016760013805097753,
+            0.001211362507398395,
+            0.048283299647824234,
+            0.015403290676022106,
+            -0.013446978976128664,
+            0.024015090204073844,
+            0.013753743307879434,
+            0.0398341544586196,
+            0.0034302789804899806,
+            -0.020152697770903245,
+            0.041614050886963766,
+            0.00392650113407049,
+            0.013465009394885544,
+            0.020487015678247938,
+            0.002430045518615804,
+            0.0035516244267459565,
+            0.02938659684630997,
+            -0.010142701862398358,
+            0.010335745659719125,
+            -0.033187389861872627,
+            -0.021792906328508686,
+            0.02532137764121119,
+            0.0005114146307176893,
+            0.033795722059780535,
+            0.007069260093915293,
+            -0.017628598898476375,
+            -0.017395163521872672,
+            0.021985468266342154,
+            -0.007987351575783343,
+            -0.025037973555114328,
+            0.002627649768366858,
+            0.008700784263932844,
+            -0.004480915144455101,
+            0.006798551384053197,
+            -0.014580806303786968,
+            -0.009741248878656092,
+            0.025066050175629597,
+            0.007330922027951371,
+            0.010472464570407973,
+            -0.024286928122882124,
+            -0.011159235410800293,
+            0.006107141408585794,
+            -0.04181412282877586,
+            0.008647396691893692,
+            0.00528510107024571,
+            -0.006647986735406744,
+            -0.002548771993161688,
+            -0.0011322362644692566,
+            0.00513401828632921,
+            -0.0035963203271502263,
+            -0.014021183815351008,
+            0.022993860073007393,
+            0.02875036422479212,
+            -0.012996574913859303,
+            0.014373647963439562,
+            -0.0093541571449835,
+            -0.019474535598737786,
+            -0.0233122546137944,
+            0.014269723193443946,
+            0.008784376430480408,
+            0.004317715767550436,
+            -0.008273153355003107,
+            0.0010203136295909875,
+            0.001909823045834306,
+            0.02406426043795662,
+            0.049377235716785324,
+            0.005265955549193235,
+            0.029166436381966566,
+            0.017136545810044906,
+            0.03882940870384413,
+            0.0007533463090350569,
+            0.016233986042907378,
+            -0.03311355052072762,
+            0.03082356484543373,
+            -0.0009431717656252606,
+            0.04077734731808,
+            0.0369941736537998,
+            -0.05477277258937169,
+            -0.014658721668336077,
+            0.028327060073149418,
+            0.010896122046930724,
+            0.01436699901738847,
+            -0.003307011398652852,
+            -0.01163199508062231,
+            0.029659278016495517,
+            -0.023936970669211955,
+            -0.010748756889752312,
+            -0.00767811402724157,
+            -0.010785978668072403,
+            0.009408061520238972,
+            -0.0038320809092061126,
+            0.038383104318077226,
+            0.0070413219383093,
+            -0.009684310528333753,
+            0.019154521368080882,
+            -0.006292471963504016,
+            0.01596901434992771,
+            0.0037167338451907566,
+            0.0019879266696342887,
+            0.03104390318244226,
+            -0.003561483519923973,
+            0.028949607591091022,
+            0.015156617705082505,
+            0.017473301120296506,
+            -0.0029321911929339165,
+            -0.017573294769065502,
+            0.027284173131869906,
+            0.009113069584708108,
+            -0.01971507323375489,
+            0.003667023499937692,
+            -0.0025339283534597176,
+            -0.0077697455666997945,
+            -0.010491575421861018,
+            -9.407001004405342e-05,
+            -0.00211520268689493,
+            0.02087980536494742,
+            -0.005582301812064227,
+            -0.0016376727058019977,
+            -0.0064225805186745605,
+            0.0052835419390265485,
+            -0.019662445004865352,
+            0.015117937189193789,
+            0.007575238218238001,
+            -0.018537762525347407,
+            0.014673223139419801,
+            0.019165766004277593,
+            0.0343151018851732,
+            -0.023949650834192267,
+            -0.0009279394158895472,
+            0.01730410035639405,
+            0.005928527001253011,
+            -0.018975612465080345,
+            -0.033873819454780706,
+            0.00825263613547798,
+            -0.01745801647450136,
+            0.027401724356781083,
+            0.008830481774058282,
+            0.0347823637819511,
+            -0.02309575756624861,
+            0.020559439488958053,
+            -1.2525886083543457e-05,
+            0.01593239963830551,
+            0.01455178268663934,
+            0.027993448238508672,
+            -0.017573784980205873,
+            -0.0379374065982424,
+            -0.0030450459335320053,
+            -0.0029968228865394214,
+            0.016851016821808874,
+            -0.013002468483615099,
+            -0.003337198735398279,
+            -0.0019044899095842511,
+            -0.029686441224288696,
+            -0.021487593939979652,
+            0.027796807698461906,
+            -0.04637191752173605,
+            0.0030176823995715597,
+            0.03281049040516897,
+            -0.030047693785808505,
+            -0.04246000386774685,
+            -0.0014603539776430917,
+            -0.03560639883119047,
+            0.0010602583020630973,
+            -0.01222013383414755,
+            0.029128085483749913,
+            0.007388529661657186,
+            -0.020518571527741426,
+            0.018184513437823,
+            0.0076594003031722,
+            -0.004161472492195519,
+            0.006100954284405263,
+            -0.005239699097592758,
+            -0.028136388241670696,
+            -0.015254134018585082,
+            -0.03037392857155718,
+            -1.6317930920712043e-06,
+            -0.002478230556191563,
+            -0.012845180214134591,
+            -0.024383682688014996,
+            -0.000484419407302015,
+            -0.014810859291916117,
+            3.958274758340876e-06,
+            0.05136363130801806,
+            0.02120810926469503,
+            0.005060247818396135,
+            0.012869784532979883,
+            -0.019665446839172217,
+            0.04743686110578455,
+            0.003514534022065111,
+            0.03583012040901262,
+            0.002062991366066978,
+            -0.02173965243546218,
+            -0.0005002960506349255,
+            0.0319890423992129,
+            -0.0016894669649440992,
+            -0.017249094049398075,
+            0.006136324758346181,
+            -0.009220681391996718,
+            0.00837047468537671,
+            0.012871598576797918,
+            0.01579543926540845,
+            -0.0061999233512262265,
+            0.014725777187858035,
+            -0.003927118678273384,
+            0.009336164668870259,
+            0.03122006739418818,
+            0.019771411252821138,
+            -0.024716950288097537,
+            -0.037736399088695305,
+            -0.012523788001500705,
+            0.001234987976751515,
+            0.034477269953960235,
+            -0.013836458205716058,
+            0.003837278407556882,
+            -0.011027628278972294,
+            -0.0054689562005572675,
+            0.005739209890610896,
+            -0.006985434270987544,
+            0.0015806421600885957,
+            0.012196603223058041,
+            -0.006859901421089723,
+            -0.025670287745499374,
+            -0.004011122451643543,
+            -0.023258896941014143,
+            0.012997299613183496,
+            -0.010144866692641839,
+            -0.006816231069049567,
+            -0.005213359047878286,
+            -0.0014167278420750282,
+            -0.0062535968890290545,
+            0.0006662897611595949,
+            -0.014236807237438571,
+            0.01012846684096945,
+            -0.00899852930003816,
+            0.0016620397335915663,
+            0.007838787681725669,
+            0.007725632848979133,
+            0.0033110678307931205,
+            -0.008206916937522372,
+            -0.008777095103563134,
+            -0.01631820698659747,
+            -0.032412484976539334,
+            -0.03000207979123948,
+            -0.02276516767125653,
+            -0.003870969029785218,
+            0.0034978865240734053,
+            -0.032594545526009384,
+            -0.020635972136797354,
+            -0.002367523739258502,
+            0.0030103817989158062,
+            -0.014587341442812854,
+            0.0014779462725289245,
+            0.007969995140846214,
+            -0.0029319878689507335,
+            0.019355924479361485,
+            -0.005448177691636249,
+            -0.021203447333791942,
+            -0.028956667864566195,
+            0.017851274292380093,
+            -0.015891501231440428,
+            -0.010070179095955473,
+            0.0064388768290910715,
+            -0.005633920123057214,
+            -0.03957042012546069,
+            -0.008945181709739484,
+            0.010992989857329521,
+            0.0035378401849368002,
+            -0.025936727040303652,
+            0.031903561853958325,
+            0.030132441728296204,
+            0.01845519141895959,
+            -0.017902241158092712,
+            0.04710441323879975,
+            -0.006739916600200837,
+            -0.012103058303430145,
+            0.016276540022040643,
+            0.023963592487909288,
+            0.028774476787161263,
+            -0.02643829153200483,
+            -0.005349178006882811,
+            -0.010545703050010008,
+            0.005354643441490569,
+            -0.02271588009302058,
+            0.008468528451368553,
+            -0.013601178278798316,
+            0.011576067943282815,
+            0.019227032134009135,
+            -0.013197207810056759,
+            0.02629307658052319,
+            0.0045263515787155985,
+            -6.067227645639678e-05,
+            -0.0037770406201685248,
+            -0.016287414859552752,
+            -0.012319957965328086,
+            -0.034674880061192896,
+            0.020297690806102293,
+            0.01889209267823917,
+            0.003953684662429981,
+            0.021845046751532583,
+            -0.049434011185276405,
+            0.02806744160678342,
+            0.0017849117483741536,
+            0.021763434864703356,
+            0.013889518910180847,
+            -0.001843068102127134,
+            0.013886691089114936,
+            -0.0032117229736863266,
+            -0.019971320584891997,
+            0.006079730961157539,
+            -0.006352535944493006,
+            -0.03316544192496317,
+            0.014736816574462923,
+            -0.03268321622529288,
+            0.013639134163846678,
+            0.015454170145821088,
+            -0.010170396823729307,
+            -0.021708948733883182,
+            -0.008303375382721087,
+            0.0010570623547459114,
+            -0.0014149688171089135,
+            -0.006849845444850982,
+            -0.04346483266888068,
+            -0.03020319096411844,
+            0.03124837430439466,
+            -0.0016720020681567616,
+            -0.03404132565569305,
+            -0.007209824644083462,
+            -0.02711263625315296,
+            0.004780610743996515,
+            -0.004212647822173262,
+            0.0025676935176963222,
+            -0.048750272936498695,
+            -0.01018434342955251,
+            0.011179560412168381,
+            0.031648611061279724,
+            -0.029298960703078986,
+            0.021370266671953315,
+            3.770954396468699e-05,
+            0.006629905234129844,
+            -0.00943632381701963,
+            -0.00484754585233744,
+            -0.012115337995959813,
+            -0.02607251387584821,
+            0.021868809148577698,
+            -0.0020587964818437345,
+            -0.0013502026980372474,
+            0.0341522429571705,
+            -0.00669289309893371,
+            -0.010113192232031375,
+            0.0005934017402521538,
+            -0.011327797122243258,
+            -0.017269539623150844,
+            -0.010467173785198161,
+            0.0005779464354975677,
+            0.0030406502310922306,
+            -0.024755136158113783,
+            0.004943446717510779,
+            0.04566273647510376,
+            0.013607936503018211,
+            0.02534200532143803,
+            0.017104750405207106,
+            -0.03792550084708064,
+            -0.04394052120951292,
+            0.0012600504352102988,
+            -0.005807815271334456,
+            -0.01944685298932453,
+            -0.01732108197375621,
+            0.0011677414026425442,
+            0.00790730598557849,
+            0.0037240231183698074,
+            -0.011427532982681041,
+            -0.02086964367350831,
+            -0.032539146090370744,
+            -0.0136797717535217,
+            -0.009464326388205681,
+            -0.007302204556505735,
+            -0.032703238152349344,
+            -0.015929357859286307,
+            0.026848214129378756,
+            0.01012098852614113,
+            -0.005646819862636667,
+            -0.0107564788588665,
+            -0.024118155140225247,
+            0.007122339033521904,
+            -0.0027183981945039814,
+            -0.009219030314883272,
+            -0.004672057262936543,
+            -0.016450690196817726,
+            0.02348093022809639,
+            0.009139210149358477,
+            0.024997129282118148,
+            -0.058533911093319214,
+            -0.014411932474734742,
+            0.014256038602455438,
+            0.0020188651815054508,
+            0.001417136751974749,
+            0.029836680491427268,
+            0.0028786535004858044,
+            -0.006526872714072965,
+            -0.02967220833325222,
+            0.028817295504169067,
+            0.027449694157944637,
+            0.01800200378294509,
+            -0.010560094796033294,
+            0.0033996190760546383,
+            0.006004691351249025,
+            -0.005679195734863024,
+            -0.011927786207717698,
+            0.0021260244673648285,
+            0.018998574211603905,
+            0.017174632765532803,
+            0.014659925822127794,
+            0.02455156987384756,
+            -0.005882590537857664,
+            -0.03911277764764406,
+            -0.009781110378157883,
+            -0.0119630929326396,
+            -0.004815125964940769,
+            -0.0026631770014339242,
+            0.008903827530107938,
+            -0.011824767329332145,
+            0.013088406716464123,
+            0.030264448923574067,
+            0.042592268021664714,
+            0.021292401951863188,
+            0.006150997237661769,
+            -0.005463801502425317,
+            0.0007399518998122848,
+            0.0038649546282459707,
+            0.011355675550395135,
+            0.035762872771750435,
+            -0.018147218572765546,
+            -0.019704502923991365,
+            0.0015558311499702044,
+            0.023543487977413267,
+            -0.007150197116420024,
+            0.02608209797565159,
+            0.023985156585794257,
+            -0.0003434024317322991,
+            -0.01271338268931938,
+            0.000985282363050309,
+            0.006710364567868783,
+            0.014540943712513688,
+            -0.02154205198718209,
+            0.004039619025717627,
+            0.001865104196024055,
+            -0.009756600776161738,
+            0.029738871754072076,
+            0.00039154552800346406,
+            -0.012981811693538832,
+            0.03825344802886,
+            0.11025963334575106,
+            0.001476339401950677,
+            0.0031187927813794386,
+            -0.031522639386920355,
+            -0.015113380587160163,
+            0.0021786482865071505,
+            -0.005948756393737655,
+            0.020082591515778573,
+            0.012751062800862873,
+            0.0055847042526916265,
+            0.010100961720464159,
+            0.038084609873482715,
+            -0.009203643420546616,
+            0.009804311800033327,
+            -0.01878272400403416,
+            0.017185477997947226,
+            -0.00972716681599207,
+            -0.00857276597428136,
+            -0.007641511960016827,
+            0.004167416313632853,
+            0.00984005584819001,
+            -0.004596999098606572,
+            0.018810908413859465,
+            0.002496310438422421,
+            -0.01613133688770545,
+            0.011505030137914962,
+            0.007437924384075958,
+            -0.005386885126779478,
+            -0.031755982499042136,
+            -0.013811586180238287,
+            0.01190667400285204,
+            -0.0013786018401773159,
+            0.0007177632889203917,
+            0.012342640512536284,
+            0.01205074763420124,
+            0.031217897495485192,
+            -0.0244324065837541,
+            0.0025071378510626696,
+            0.0070486374045588465,
+            0.0159322020159265,
+            0.01098799814774876,
+            0.00029844505852147526,
+            -0.015456787478044924,
+            0.027139764983358832,
+            -0.011978684728746428,
+            0.010378658252224036,
+            0.01146607671795495,
+            -0.04131658683485702,
+            -0.0009967932006190922,
+            -0.005916133731248287,
+            -0.015698754902727707,
+            0.013905118384587169,
+            -0.040163447416240027,
+            -0.005682084326086952,
+            0.004297114532998809,
+            0.010724201884417054,
+            0.0058749102353125955,
+            -0.017715270733111323,
+            0.009962853888582856,
+            0.00955216765250444,
+            0.0025392532243413593,
+            -0.006356691980285544,
+            -0.006218169970979339,
+            -0.008606942887730427,
+            -0.004280171880086814,
+            -0.015669721147884084,
+            0.0057481312044666884,
+            0.0077227094511494055,
+            -0.018751123856436715,
+            -0.01585864749751703,
+            0.009183099781375234,
+            0.006837438250956817,
+            -0.001445622524955126,
+            0.004828557942941401,
+            -0.010034012200008746,
+            -0.004543834537450213,
+            0.01097540631691429,
+            -0.009334158551770033,
+            -0.004804430123454511,
+            -0.02609985520241612,
+            0.013342299390862539,
+            0.011103136832190223,
+            -0.008261332966534102,
+            -0.006944131083745896,
+            -0.016931502665633764,
+            -0.0031318730206304824,
+            -0.0026508016484082988,
+            0.0016110884325639078,
+            0.017071266355203386,
+            -0.009906320666640836,
+            0.010877416311741906,
+            -0.03496707406301918,
+            -0.015357787514985604,
+            0.01622830705930841,
+            -0.01290003882907745,
+            -0.005625420341281716,
+            -0.009838333350984609,
+            0.0006911383780218238,
+            -0.007545081680638308,
+            0.006074752194220675,
+            -0.041320607445282646,
+            0.0035102345636816475,
+            0.018825308640738208,
+            -0.009723969339534365,
+            -0.020829421583878845,
+            -0.009055117268112308,
+            0.010487235049013567,
+            -0.015831829161845813,
+            -0.01780922120240258,
+            0.0016219998217122911,
+            0.010559783293374071,
+            -0.0055135690971065965,
+            -0.005160232484458576,
+            0.006163424486940105,
+            0.020247627989655837,
+            0.015220964429515309,
+            0.04337432385613251,
+            -0.011915355138717245,
+            0.009617347907919507,
+            0.027812934121040484,
+            -0.030652481946156104,
+            0.01037470561447702,
+            0.02599512377987521,
+            0.0037340561802882417,
+            -0.0013375196064018082,
+            -0.010197691376747194,
+            0.004161692215514214,
+            -0.012254402844820636,
+            0.008715552098450461,
+            0.004697361473725416,
+            -0.027978648968732336,
+            0.017196771853396124,
+            0.019448218659009996,
+            0.0038720194629308794,
+            0.0006995062900842749,
+            -0.01527901684789497,
+            -0.011825577013200781,
+            0.019194236643713274,
+            0.027690065919038755,
+            0.005460028121722145,
+            -0.015512497062157167,
+            0.02370643287056251,
+            0.012916257803027072,
+            -0.004973882005506947,
+            0.0022332716531575122,
+            -0.016984605526823043,
+            -0.01443316577897661,
+            0.030942241604983316,
+            0.011563751997976407,
+            0.01908030186367601,
+            -0.021614212853976106,
+            -0.003982310717524619,
+            0.019931579069642925,
+            -0.021029366993395197,
+            0.013006879711144009,
+            -0.019497570222553013,
+            -0.014067925104548129,
+            -0.010687576618198498,
+            0.02884494104893037,
+            0.009229770955288097,
+            0.026701705044769222,
+            0.012715371448003833,
+            -0.021444875656527786,
+            -0.021050591467198086,
+            -0.0027703267406452336,
+            -0.02721606030674425,
+            0.01688256721881862,
+            -0.055990128879151335,
+            0.0016563858333845547,
+            0.010911810464298247,
+            -0.008604898990983188,
+            0.004039521277648332,
+            -0.007858378479197747,
+            0.010021929060776957,
+            0.020926274809879138,
+            -0.02956229623532218,
+            0.01256060127924585,
+            0.01156331569664224,
+            0.0218688589850221,
+            -0.018861065515394883,
+            0.015339087155577661,
+            -0.00841001512858443,
+            -0.027396412391138852,
+            0.008323898095545889,
+            0.01763697170382491,
+            0.014160184414353919,
+            -0.011992345081843988,
+            0.004485633614765869,
+            -0.04000932898876895,
+            0.027999519977069676,
+            -0.016823500188831896,
+            0.002010511519234648,
+            -0.004537991252901591,
+            -0.0067219583064939085,
+            -0.008523354091034399,
+            2.682477256914193e-05,
+            -0.0005371719103087346,
+            -0.030920653652724776,
+            0.0012939076000836422,
+            -0.04144943703666512,
+            -0.02260198516292792,
+            0.013327519374598822,
+            -0.0012897465780583367,
+            -0.005059170593504863,
+            0.02999548027478466,
+            -0.02863650640718448,
+            -0.027272295036911975,
+            -0.014376307736824656,
+            0.03534844385341626,
+            -0.015358480569131909,
+            -0.000540068394391221,
+            -0.009287193719601908,
+            -0.02236594406404711,
+            0.01091078016405303,
+            -0.006469935602381157,
+            -0.0019272207412132534,
+            -0.01387843655423569,
+            -0.03972775413987396,
+            -0.005948332199741248,
+            0.019500395452013777,
+            0.0006122786247500248,
+            0.003656719755271774,
+            0.014473976840529128,
+            -0.0072278373928449675,
+            -0.0036623750417377524,
+            -0.0038809301345520044,
+            0.026929584793529297,
+            -0.022225170775945557,
+            0.009744440210592443,
+            0.004704883568603746,
+            -0.0005991305039095727,
+            0.017317686588588826,
+            0.000351879037187266,
+            0.0024030356296394478,
+            0.0029060795825935717,
+            -0.0035301907818594274,
+            0.02922907976800865,
+            0.0064070048516525,
+            0.0016291092834589405,
+            0.0001988403049139903,
+            0.004746570638478093,
+            0.01395197992587789,
+            -0.01069245020264236,
+            0.0157940052093005,
+            0.02865126209030693,
+            0.00021287181382353683,
+            0.018345429962063935,
+            -0.015921361819192573,
+            -0.036059435316760584,
+            0.019807131219096722,
+            -0.029642240399677812,
+            -0.003504127677138975,
+            0.022473005924273325,
+            -0.012903852741467797,
+            -0.0031750649075827082,
+            -0.01107957248024579,
+            0.027243132530764567,
+            0.023512079029549708,
+            -0.014856101416774535,
+            -0.015142539116300868,
+            0.004337433267586975,
+            -0.010210607143984678,
+            0.010939153927790107,
+            0.029382995339774957,
+            -0.011230337698862454,
+            0.011717621080350227,
+            0.001161341258722717,
+            -0.003778338045364591,
+            0.019300005069271462,
+            0.0027681684372537606,
+            -0.007401677666787755,
+            0.025628069381915978,
+            -0.008115857591022265,
+            0.013043669211084909,
+            -0.004672358463640539,
+            -0.012902494254220482,
+            -0.003106794914564588,
+            -0.017558145644151636,
+            -0.01225024815282763,
+            0.0011522376202485873,
+            -0.005528597019035285,
+            0.06711658458916447,
+            0.010930891527455397,
+            -0.017888423424597618,
+            -0.0059190199704595775,
+            0.00955567741445378,
+            -0.000966061435587513,
+            -0.009466328431452424,
+            0.027063767833815658,
+            -0.016888493246695876,
+            -0.009681104485277698,
+            -3.814655691584766e-05,
+            -0.0233149747136092,
+            0.0030596902889784496,
+            0.010827987186229144,
+            0.0012052973350181351,
+            -0.013994504803884577,
+            0.005318095523120499,
+            -0.004843763992616558,
+            0.0034188401938723923,
+            0.003678626789714963,
+            -0.0025749484272345573,
+            -0.005333294277998365,
+            0.02307826358450567,
+            -0.012791914224263883,
+            -0.008519982879775168,
+            0.01173424692936401,
+            0.001777123104204978,
+            0.003219553788359312,
+            -0.004564745566711077,
+            -0.002015107048152335,
+            -0.017123545909339455,
+            -0.011278616226036531,
+            0.023713473402287875,
+            -0.02733523566514418,
+            -0.00150943777243135,
+            0.017328382962199716,
+            -0.005705544203971564,
+            0.016650187392502975,
+            9.750950361601131e-05,
+            -0.02100066506382883,
+            0.00010090567939930427,
+            -0.024778741347883073,
+            0.010484659495766551,
+            0.013963183459158656,
+            -0.009531952698151101,
+            -0.020878167882780043,
+            -0.027881362421494915,
+            -0.010190031350698889,
+            -0.027246821338460653,
+            -0.0280692647056191,
+            -0.02020582179742232,
+            0.009778340348478517,
+            0.005715364307462615,
+            0.021929057840860695,
+            -0.006380523858228711,
+            -0.027552858236741976,
+            0.015095163660340326,
+            -0.013269344817111162,
+            0.008282918921343354,
+            -0.019991891695114384,
+            0.01754833974582726,
+            -0.014720830704852575,
+            0.02112161088830394,
+            -0.00041409266970033855,
+            -0.004490472334793758,
+            -0.012076561942010787,
+            0.003976913901514849,
+            0.008101357042612863,
+            0.00176590819486174,
+            -0.008977295353115363,
+            0.026242190932756135,
+            0.01707634775369835,
+            -0.007385093173634561,
+            0.01604386075569055,
+            0.031855080733360075,
+            0.014762206220119051,
+            0.038860660002328094,
+            -0.007238177370595944,
+            -0.009237098188374397,
+            -0.02764495845127417,
+            -0.004407870047633689,
+            -0.006820430763682645,
+            -0.004516837004332235,
+            -0.015251921783911087,
+            0.03164122113689215,
+            -0.010371400055822811,
+            -0.01686972159902524,
+            0.03979035122593764,
+            -0.021846728842184655,
+            -0.007084905708666271,
+            0.008642771511425465,
+            -0.013996636040226265,
+            0.03176661229383477,
+            0.007275612529363987,
+            0.018452818352301472,
+            -0.015039890816023917,
+            -0.015988701055097997,
+            -0.0019911190487748244,
+            -0.002806138139429132,
+            -0.020828646560807865,
+            -0.0063794261154720665,
+            -0.01614570050728432,
+            -0.0026319562705484634,
+            -0.00570753504957681,
+            0.015917008930969468,
+            0.0013916917451795444,
+            -0.015420032290910069,
+            0.04488171544231248,
+            0.005033499525006087,
+            0.022768662319611596,
+            0.010777933460735837,
+            0.012372342667053587,
+            0.015819823672091563,
+            0.006759824420801962,
+            -0.0001668726099910387,
+            -0.02211654164646909,
+            0.004261169256891085,
+            -0.005641717035723983,
+            0.0041005490092982915,
+            0.02012094437568505,
+            -0.024436624754739972,
+            -0.029708548399946657,
+            0.024493078535528394,
+            -0.01833101179919427,
+            0.0030908990956393764,
+            -0.0031025537219982234,
+            -0.010059653447038065,
+            -0.011612233298907742,
+            -0.03293899346571458,
+            -0.0005929867206193162,
+            -0.027438115397399914,
+            -0.03635734152790657,
+            -0.006287687204976032,
+            -0.007850474352061472,
+            -0.003574235211373211,
+            0.02449669755605501,
+            0.004269460596739739,
+            -0.029959856511873897,
+            0.01782698044631415,
+            -0.013252733005984911,
+            -0.013342147906120882,
+            0.0029581361690999564,
+            0.004786570391076097,
+            -0.0027618259744494854,
+            0.010720003765859173,
+            0.007898965393358715,
+            0.007358694001293221,
+            -0.010710502803266644,
+            -0.004049671482838962,
+            0.0013263710826446039,
+            0.0030943253037901135,
+            -0.013325799375903166,
+            0.030553973934607993,
+            -0.018812559793353154,
+            -0.004819166852190956,
+            -0.0006499911473215254,
+            0.0034556893023524674,
+            -0.0110929170801761,
+            -0.014862886414702483,
+            0.021004665462647988,
+            0.0031088550726355926,
+            -0.012570481398485388,
+            0.0096952264731202,
+            -0.0018868240406799254,
+            -0.0024767709620225923,
+            0.01176108203579327,
+            -0.002080730326597438,
+            0.014924914157541781,
+            -0.011040818699356567,
+            0.01490073610670544,
+            0.0034649445207609085,
+            0.008645740731109337,
+            0.003603641937848282,
+            -0.01905231764119889,
+            0.009780124784430865,
+            0.016356631955067272,
+            0.013415470760506568,
+            -0.029496530287207723,
+            0.008881639987471632,
+            -0.005975399013716009,
+            -0.009870580036511721,
+            -0.024372445588075428,
+            0.002590423942920311,
+            0.028960961301291905,
+            0.012044179251559706,
+            0.007934516374398097,
+            -0.0077949340827692054,
+            -0.005480592942515702,
+            -0.019923500717513334,
+            0.009041031662948522,
+            -0.01095651774810384,
+            -0.025157908389819465,
+            -0.007578712371715301,
+            0.03430545904801765,
+            -0.010740874257824011,
+            0.01343875386664532,
+            -0.005526757815283282,
+            0.02436552949831661,
+            -0.0027538610388853284,
+            0.0013690344627799137,
+            0.008416180197734094,
+            -0.006091433734127276,
+            -0.001708607133233843,
+            0.013349212286052506,
+            0.041066250931277824,
+            0.0038375614886832564,
+            -0.0050950709988837216,
+            0.01149207527692021,
+            -0.00326389523728132,
+            -0.005712157453687712,
+            -0.00309579814793649,
+            -0.025491974096530076,
+            0.015133356727314608,
+            0.0010745378801251735,
+            -0.017162228172591873,
+            0.021785292616181687,
+            0.027891598422935242,
+            0.013262622112280296,
+            -0.0178471114368855,
+            0.014456217536546996,
+            -0.009278392107874806,
+            0.01629461595238019,
+            0.02423194076020779,
+            0.010067296472207578,
+            -0.011865648674106681,
+            -0.0070073298858753805,
+            0.008985354045706703,
+            0.008096016282907443,
+            0.0004236756235618935,
+            0.020370823466310733,
+            -0.028334250141960886,
+            0.01483270667238479,
+            0.007137078270349882,
+            0.03314918453195744,
+            -0.0006231629401653488,
+            -0.014397909717816884,
+            -0.008918359605259833,
+            -0.002915473990900902,
+            0.011759189078587877,
+            0.003523992368256188,
+            -0.0006853609664806012,
+            -0.004749148984168426,
+            -0.014181883848149463,
+            -0.006778789302277201,
+            0.012376777107762334,
+            -0.0031646329407784245,
+            0.0030514749941358095,
+            -0.0007326603315688041,
+            -0.007811941797535734,
+            0.015661953163847418,
+            0.001669500869963878,
+            -0.0007143628544145883,
+            0.0016729300148826723,
+            6.604560293656475e-06,
+            -0.00467413979302959,
+            -0.019114577369302277,
+            0.008922356418978664,
+            0.00867683974577237,
+            -0.006578536023513047,
+            -0.006467824108991175,
+            0.009324373505311057,
+            -0.01432602608910974,
+            0.01852478469282692,
+            -0.0061254590933158826,
+            -0.006603842196861297,
+            0.00014277641689784565,
+            -0.012328738339684425,
+            -0.003642202698645936,
+            0.0034895681343535914,
+            0.0051453312116433265,
+            -0.000908593523739809,
+            0.0036062764245630334,
+            0.003940522150829955,
+            0.01083964694261132,
+            0.002824079422783228,
+            0.0015266576108775834,
+            0.007280263953807289,
+            -0.01039293741283943,
+            -0.007403606843286745,
+            -0.004993622657038095,
+            -0.006341662331986892,
+            0.007288958567606623,
+            -0.004364975055335228,
+            0.010457360052582658,
+            -0.0062292104145735515,
+            0.020471836369529147,
+            -0.006133009423930811,
+            -0.0024055488697334888,
+            -0.017695183455939345,
+            -0.007234478249839231,
+            0.007156119488831356,
+            0.007878044165936736,
+            0.009359297231370758,
+            -0.04487830669749157,
+            0.005038301058510894,
+            -0.0054192943462008595,
+            0.02941616633783806,
+            -0.009828941844770915,
+            -0.004335130343692854,
+            0.016067824555917796,
+            -0.019100842323053376,
+            0.0037066021889614513,
+            -0.013965982913833292,
+            0.003695036285467973,
+            0.0034558765116311525,
+            0.014205693020177169,
+            0.031307058208919394,
+            -0.007482180869231769,
+            -0.009383819319265733,
+            -0.006384083847148052,
+            0.026967380724567013,
+            0.018643952476557566,
+            0.023548246269020395,
+            -0.011885146764471029,
+            0.00960518654541979,
+            0.001357650512536872,
+            0.018271473493526338,
+            -0.009663920325675169,
+            -0.011163336807225407,
+            -0.000516828412255004,
+            -0.010288703356943627,
+            -0.0032111641654394615,
+            -0.021092460896394025,
+            -0.0033456309643941965,
+            0.025876107752976232,
+            0.0038584058642743987,
+            -0.0036689325941524813,
+            -0.015624044740149702,
+            2.2644755460272697e-05,
+            -0.011931276845804481,
+            -0.0055870010742218364,
+            -0.015697330226950804,
+            -0.028927689083316704,
+            -0.016192017705678192,
+            -0.002914436546908901,
+            -0.017224654082473787,
+            -0.009460990414684924,
+            0.002071634636546831,
+            -0.015533259213802986,
+            0.004918377471991415,
+            -0.015669642231243106,
+            -0.010915412245035698,
+            -0.00046472075835247,
+            0.004672390433470177,
+            -0.004682173537415046,
+            -0.024109027987987815,
+            -0.013628794248618615,
+            -0.0022054473551170926,
+            0.006500170428351522,
+            0.010016298501187265,
+            0.008324628078950177,
+            -0.0069374343736907625,
+            0.011201167788973098,
+            -0.009746811993090597,
+            0.010155637867574787,
+            -0.02773759960959338,
+            -0.04317847702979315,
+            -0.000791583637601122,
+            0.009241551203900427,
+            0.001979688190287368,
+            -0.01884488538245213,
+            0.0038489955050695704,
+            -0.011280060242688173,
+            -0.00847814532468811,
+            -0.007522173019881079,
+            0.01573091287228822,
+            0.004138348969142638,
+            0.013463997539337106,
+            0.0042065270170965705,
+            0.0285467221116185,
+            0.012635383326761538,
+            0.01030078781909351,
+            -0.003092067773698843,
+            0.004763648809248413,
+            -0.0022835042548997662,
+            0.009077632103272405,
+            0.02331475506951857,
+            0.0076225652102231605,
+            -0.002326207128504976,
+            0.00409858249509217,
+            -0.0005698076375460636,
+            -0.005442088641415082,
+            0.018565262151290427,
+            0.02915691852605287,
+            -0.0069811288298562155,
+            0.008120076286611811,
+            -0.0009110258445641983,
+            -0.009900329477406562,
+            0.0038288023984988134,
+            0.005765264445222657,
+            -0.014230725839572923,
+            -0.0024175374911696653,
+            0.009367987848760041,
+            -0.024664375361024386,
+            -0.004113757123212052,
+            0.020371264792326226,
+            0.0024813232005061487,
+            -0.016599537494340785,
+            0.03488073638324099,
+            -0.003709931703179228,
+            0.003010315958426392,
+            -0.00944047118505417,
+            0.016362178907316286,
+            -0.017989464717541103,
+            0.0005390907615384354,
+            0.013131172782591922,
+            -0.005251614069791733,
+            0.005783823857695169,
+            0.003744510952053123,
+            -0.010417674399178015,
+            -0.010824994224858631,
+            -0.0033951133363820767,
+            0.01085129781083312,
+            0.002642227227069058,
+            -0.024446190958178605,
+            -0.017731767444262785,
+            0.019381370848622046,
+            -0.01001206050737818,
+            -0.008290790009659911,
+            0.0009331678418736739,
+            0.0029270629970877303,
+            -0.004637651319894003,
+            -0.004290917518172777,
+            -0.003574121539057569,
+            0.009121344354139133,
+            -0.009485343005168732,
+            0.01637203851901565,
+            0.004043708359084711,
+            0.01843096334879628,
+            -0.011588935298420727,
+            0.005808551475129831,
+            -0.001368952044937682,
+            -0.007242291047654279,
+            -0.019004856431577073,
+            0.0059537808323522675,
+            0.0073777483129045685,
+            0.016051495408653346,
+            0.011060627996179347,
+            0.018569088420830965,
+            0.011345955812555888,
+            0.0039209921807600205,
+            0.001180658155391396,
+            -0.011006781869601802,
+            -0.016371246428912994,
+            -0.014232536816028922,
+            -0.007416908142074357,
+            -0.019561375671062513,
+            -0.0013253236527051844,
+            0.017798265625542643,
+            0.014528038148955117,
+            0.017704367192118375,
+            -0.0048389172759564884,
+            0.016018290057587425,
+            -2.7789810195444745e-05,
+            0.0009215076153434622,
+            -0.005897137279173464,
+            0.001794778099552753,
+            0.016314258230521846,
+            -0.003313666704140227,
+            0.0063152813149371136,
+            0.0002935873451081568,
+            0.01645203621687452,
+            0.022650990579075773,
+            -0.018899997585540014,
+            -0.0014442758993479705,
+            -0.006467599303099793,
+            -0.0021225224281092863,
+            -0.019894752041664215,
+            0.020266472457121338,
+            0.014019662562577204,
+            -0.005002061958298289,
+            -0.00943116174529331,
+            -0.004354703314807638,
+            0.02678118874395904,
+            0.020206530983694557,
+            0.006044260997975214,
+            0.026222728100119382,
+            0.01976601108150248,
+            0.01984865298844998,
+            0.011918890024498122,
+            0.009052182076567042,
+            -0.013092243192366595,
+            0.018358346459726883,
+            -0.01681770553983246,
+            0.0008504895530001561,
+            -0.003223133403363917,
+            0.002743153478851782,
+            -0.007937179998086906,
+            0.013021430914653432,
+            -0.00030955362267617273,
+            -0.025935204965869492,
+            0.007680669831594167,
+            -0.013183173230049736,
+            0.001375652741144253,
+            6.07690288088255e-06,
+            -0.001857629339163609,
+            -0.017652365850892467,
+            -0.0082709435899296,
+            0.0008448530375262078,
+            -0.01435595518048752,
+            0.023866770609514233,
+            -0.026826645910954506,
+            -0.02140002089501948,
+            0.00988141127158139,
+            -0.004436844338114915,
+            -0.012632978351515892,
+            -0.006972217855086974,
+            0.021864041851582767,
+            -0.005139891275508354,
+            -0.016596866242234234,
+            0.014168148679282804,
+            0.0005869000756971533,
+            0.00412958763467087,
+            0.0012364712526909765,
+            -0.008001798306987228,
+            -0.0030416500521329983,
+            0.006860667487819997,
+            0.028693213258586023,
+            -0.011895509688811421,
+            0.0016632663352826491,
+            -0.013799461077520309,
+            -0.0037922015693749754,
+            -0.02974150200357857,
+            0.027752493864472517,
+            -0.011830981692069491,
+            0.01989293240244598,
+            -0.004540523332199857,
+            -0.007667738626842118,
+            -0.021173122202316107,
+            0.035703796333655495,
+            0.004594443089494466,
+            -0.0006664409717898689,
+            0.007256660335524136,
+            -0.047588498838074554,
+            0.01105469709139337,
+            0.006498789757276137,
+            -0.023261890747840144,
+            -0.000964111432535798,
+            -0.003995007813335293,
+            0.02143067593410374,
+            -0.019461700447829744,
+            -0.012777737727876326,
+            0.0021024453304811634,
+            -0.021925886491102078,
+            -0.007056889535073896,
+            -0.0340604475320119,
+            -0.006137802768771592,
+            0.006503135515471076,
+            0.0038385769709340417,
+            0.0020799029890387023,
+            -0.021308499975514292,
+            -0.0309544649831362,
+            0.013308972238482328,
+            0.01651668347077017,
+            0.010007831710070633,
+            0.01200701454477496,
+            0.0067050757442456005,
+            0.011425518484475204,
+            0.00865023114755134,
+            0.00974582806570851,
+            0.008884712478547614,
+            0.039109252942848706,
+            -0.016122498485808056,
+            0.008671479031657285,
+            0.0005699279397862279,
+            0.02156543966774511,
+            -0.02391743870774471,
+            -0.009877938786506814,
+            0.015475531666380014,
+            0.003045729682805746,
+            0.030465388674890067,
+            -0.01511217342930631,
+            0.005735960079359902,
+            0.0030955813611916066,
+            0.003343849304714193,
+            -0.017685941897160877,
+            -0.019795694199965495,
+            -0.015889472669342385,
+            0.0005064000414834533,
+            -0.0007719666713392906,
+            -0.012817517834744753,
+            0.01370266779873307,
+            0.0029457043321269045,
+            0.004380574233134056,
+            0.003997906783171028,
+            -0.02610528890228495,
+            0.0033844277218836307,
+            -0.008397899012048187,
+            0.013703609594527308,
+            -0.0012702392316682096,
+            2.036849934035612e-05,
+            0.007261982817787643,
+            -0.00406175072535136,
+            0.018353316401139956,
+            -0.007206833180919801,
+            -0.004174154024217347,
+            -0.004097578846173991,
+            -0.00573702992914836,
+            -0.0017184308802689716,
+            0.022375713882789285,
+            -0.008746034802967245,
+            0.032729322137817145,
+            -0.014835515993879445,
+            0.01049714579786462,
+            0.013344412726432523,
+            0.006468296817982878,
+            -0.0036641997047404426,
+            0.0003384623491490165,
+            -0.0035345323089769998,
+            -0.006579417470328358,
+            0.015682816949756224,
+            -0.018934655843627496,
+            0.006860259740057359,
+            0.03727162736659208,
+            -0.002536806673033066,
+            -0.00615871353663963,
+            -0.008853057427430264,
+            -0.012968501800661,
+            0.028268388127570685,
+            0.005935705363468247,
+            -0.0013092005585187737,
+            -0.03024974000885413,
+            0.015180604705329429,
+            -0.0037501687534873948,
+            0.0016387074669205704,
+            0.00860407823411825,
+            0.00044285387135976006,
+            0.020127437993717013,
+            -0.0011272951769363546,
+            0.021290480413673304,
+            0.010032359217391118,
+            -0.0010666180914844765,
+            0.011529468490024128,
+            -0.036558434912810016,
+            -0.008579049583792913,
+            0.0038048395503202733,
+            0.019444233855445248,
+            -0.0006623807457136332,
+            -0.015853103419331053,
+            -0.004242407392496502,
+            0.009029483146616182,
+            -0.01059695082306233,
+            -0.013492380798775181,
+            0.016259633969459952,
+            -0.019096544374819852,
+            0.011560512412214506,
+            0.008617334373339341,
+            0.02512750386847096,
+            0.0041202753709624245,
+            0.002613588241299414,
+            0.0035069606381226207,
+            0.006602772841623022,
+            -0.002625433099855357,
+            0.011789793979724633,
+            -0.011592113209370399,
+            0.02102479518464762,
+            -0.008294039267346376,
+            0.007068661518691129,
+            0.0032541671568032306,
+            -0.022166914870027622,
+            -0.018234159989683987,
+            -0.0023642465680105544,
+            0.021312775708470108,
+            0.006016177710817625,
+            -0.012618763344460982,
+            0.0055411096806088885,
+            0.004690341026678943,
+            -0.00547301082703577,
+            0.009118596747082881,
+            0.024398283179066196,
+            -0.0028098889343668772,
+            0.01068095200128235,
+            -0.007137857371825465,
+            -0.027832295014375272,
+            -0.007421428223393234,
+            0.021707007996690945,
+            0.00020946296332069377,
+            0.016793613517175475,
+            0.014396690762872469,
+            0.012889706504719996,
+            -0.00488698200749025,
+            -0.005354238088985329,
+            0.018317372626432005,
+            0.014340608961907827,
+            -0.0048105230366691145,
+            0.001495716612807014,
+            -0.0018529641162211795,
+            -0.017240719704808234,
+            0.026933447120216628,
+            -0.024232238634425658,
+            -0.003438477892192365,
+            0.00892792234782449,
+            -0.012586276728773793,
+            -0.004923824123059311,
+            -0.023468301644204742,
+            -0.02117100775850015,
+            0.014296432515879623,
+            -0.003314359969711106,
+            -0.01750062790930347,
+            -0.006856892040084997,
+            6.621834660421394e-05,
+            -0.003983545180909168,
+            -0.010826144842427663,
+            -0.0028995230412062357,
+            0.0018029406642546938,
+            -0.01286603500476852,
+            0.00937188453692113,
+            -0.00635685147895941,
+            0.010242542894487791,
+            0.0062482230941642404,
+            -0.022974002980633818,
+            0.005235079251883088,
+            0.020160136045340955,
+            -0.008953506137454955,
+            0.0007027144749750591,
+            -0.015318432934167311,
+            -0.006155069535076527,
+            -8.304773776972749e-05,
+            0.010665400866776814,
+            0.013561082526851403,
+            0.019453190615040503,
+            0.008724224096424808,
+            0.017716608184445207,
+            0.0035170519664773133,
+            -0.013169473712817174,
+            0.009862672596286218,
+            0.011645070824075182,
+            0.011745045118639314,
+            0.011761235635339118,
+            -0.0051506969158396755,
+            0.00628898514628358,
+            -0.0049857719357619505,
+            0.008525048536699957,
+            0.027507594807546505,
+            0.0013209667681023782,
+            0.017047984110417937,
+            0.016368796085341072,
+            0.020105811276523177,
+            -0.003759327296302425,
+            3.4626946806238876e-05,
+            0.00035688794161846735,
+            -0.001063984476108484,
+            0.003291497239446139,
+            -0.018279629509070614,
+            -0.01135428882896519,
+            0.017033095140582537,
+            -5.492833742539044e-05,
+            0.024685530931127835,
+            -0.0010090516352729089,
+            -0.0184210790691242,
+            -0.006072072044463014,
+            0.0019766255786695347,
+            -0.0143465314042477,
+            -0.00209268265697904,
+            0.008867797876246725,
+            0.02090777280180668,
+            0.007785784409274212,
+            -0.014694559759226521,
+            0.03266241399157856,
+            0.017596572028431588,
+            0.02301544904486291,
+            -0.02019106568899837,
+            -0.008116927224141509,
+            -0.007678137378694171,
+            0.020318874678520133,
+            -0.0068566512882342135,
+            -0.002510061555870801,
+            0.021473513752723834,
+            0.006645783892214728,
+            0.011211922100954015,
+            0.012286872605920514,
+            0.0003084092164612384,
+            0.0028492628853667647,
+            -0.015161921733689642,
+            0.011855756525919763,
+            -0.005289433905822296,
+            -0.006904878573181685,
+            0.021517401941638887,
+            -0.010444313612915583,
+            -0.009104739247715266,
+            0.005434542886627656,
+            -0.011460046711120444,
+            0.013324525803587403,
+            -0.0016717915349416276,
+            -0.013225789210352699,
+            -0.014868291792978007,
+            -0.004059908595020105,
+            -0.017086039690741954,
+            0.018935544362274584,
+            0.006810606440007176,
+            -0.0016747502037076753,
+            -0.005949836930379681,
+            0.004577809518378731,
+            -0.002387527459083855,
+            -0.0006415824062693734,
+            -0.015923707971787795,
+            0.0035047065077239213,
+            0.009700583796235219,
+            -0.0031324165789907116,
+            0.010643108490671052,
+            -0.013353561795075988,
+            0.033779753889958034,
+            -0.003236875179603599,
+            0.0033068852921624326,
+            0.005106828066670305,
+            -0.003034255184307467,
+            0.001481657575335246,
+            0.013519816427440722,
+            0.009066434673988428,
+            -0.008339461460990311,
+            -0.0009211583209818409,
+            0.013497039042975171,
+            -0.018668722901146233,
+            0.0013796912320793106,
+            0.013796986913155082,
+            -0.015849997808276967,
+            0.0049427770053857125,
+            0.011400118227502903,
+            -0.00639345486098034,
+            0.015340608222895147,
+            -0.01371015934577743,
+            0.007008681477695071,
+            0.009775206231977338,
+            0.028490233702896408,
+            -0.01966685875736091,
+            0.01928758485650401,
+            -0.025635008258084145,
+            0.031283028009357144,
+            0.012122242939206308,
+            -0.005796910058436286,
+            -0.025521126197222552,
+            -0.011593941553769327,
+            0.01283377684428954,
+            -0.021686338476615465,
+            0.0014883363235708173,
+            0.004930128822274494,
+            -0.013589466747020829,
+            -0.014766714233107183,
+            -0.01202283835955618,
+            0.011738578705966321,
+            0.0006399892787922201,
+            0.01589633486439762,
+            -0.001565909739326631,
+            0.006195453613357957,
+            0.00799709051068248,
+            -0.007262779704915372,
+            -0.006717513726969218,
+            -0.01982600498555257,
+            0.01452043414139846,
+            0.004256754663990435,
+            -0.005262312998149605,
+            -0.0224549938750886,
+            0.009397743096285409,
+            0.017707219262338467,
+            -0.005753255460862127,
+            0.013047488144162397,
+            -0.03383526087792666,
+            0.0009452367103428395,
+            -0.005822570832057675,
+            -0.0027621026437538738,
+            -0.011741384193082396,
+            0.005215022845011562,
+            -0.015301048137828595,
+            0.002594097201808009,
+            -0.006446736651739076,
+            -0.016963075831388622,
+            0.002297125878220545,
+            0.02099207084363147,
+            -0.0032941933998622765,
+            -0.0028818889706579017,
+            0.01784874229400347,
+            0.008499720521713671,
+            -0.01436426626087195,
+            -0.004000179664908013,
+            0.006397941984552902,
+            -0.018873859156915206,
+            0.0011599963694687036,
+            0.02198209878751397,
+            0.012652203334020664,
+            -0.026378312815268385,
+            -0.0016397712646032201,
+            0.013833745513458322,
+            0.008230921549345205,
+            0.008667892596847433,
+            0.0040942632709300665,
+            0.014593014365768302,
+            -0.006350891941908016,
+            -0.001374093276890415,
+            -6.3005496707736715e-06,
+            -0.006943874751919358,
+            0.011912879044321251,
+            0.003688355066059739,
+            0.040741622477833674,
+            -0.010639328654064204,
+            0.011006744363538067,
+            -0.01080331270092793,
+            -0.004477468739500037,
+            -0.004503151926350927,
+            0.0027311865257027917,
+            -0.013139063275489816,
+            -0.0001322676781706513,
+            0.0021051078165230533,
+            -0.0019788024041678294,
+            0.00649268452600941,
+            0.003305903531478554,
+            0.013176851471483939,
+            -0.017229299828998786,
+            -0.0024884474570636707,
+            -0.009910839696360717,
+            0.019347666346683084,
+            -0.010724923526200773,
+            0.01051871403061451,
+            0.03256893707453101,
+            0.013242686936629418,
+            0.00022445440868234016,
+            -0.014947304198470308,
+            -0.008836459838385262,
+            -0.008992977994567497,
+            -0.01739161854615299,
+            -0.012908166671408042,
+            -0.022654409684101404,
+            -0.00294050967213434,
+            0.006390611639719366,
+            -0.0010555036392130776,
+            -0.002785796833161689,
+            -0.027340941664129582,
+            -0.00792516100590229,
+            -0.03210910565574123,
+            -0.013679170743667392,
+            0.0056845041078574104,
+            -0.021558178294402857,
+            0.005195998493021035,
+            0.029445968279023734,
+            -0.008256723194649932,
+            0.018403154314213917,
+            0.006923789005793937,
+            0.011433558571124386,
+            -0.0022324062216246424,
+            -0.011870704203994723,
+            -0.011035736680889242,
+            0.00032924465177187114,
+            0.003455564411617082,
+            0.004938023824535148,
+            0.00916834019939669,
+            -0.01732811783799426,
+            -0.010720420451770168,
+            0.00025951769018448803,
+            0.01732173296763328,
+            -0.0036441011455210317,
+            -0.014654182169779525,
+            -0.009890543675136605,
+            0.00934755871928037,
+            -0.017560953324850772,
+            0.0028803276180087,
+            0.008144645056336303,
+            -0.006844751836577803,
+            0.009686357271083834,
+            0.00910269542748879,
+            -0.003634975911996508,
+            -0.01125116892179341,
+            0.015123292946443665,
+            0.011417498955214564,
+            -0.0005683452620824292,
+            0.009547451569266946,
+            0.0031907584166949686,
+            -0.004914343407341038,
+            0.0004946517266559977,
+            -0.013981094295278457,
+            -0.008438177812073363,
+            -0.009698303451884585,
+            -0.0004873949117926506,
+            0.008310275049300703,
+            -0.011244097579407173,
+            0.002589333020166005,
+            0.021677389796728952,
+            0.0016179416889548232,
+            -0.011231602157838225,
+            -0.0045296258349024725,
+            -0.03198651985714177,
+            -0.005752566699226788,
+            -0.017524199510403467,
+            0.014980719037744585,
+            -0.002529640898744345,
+            0.01028755110586937,
+            -0.0021848874276835985,
+            0.015817733088597956,
+            -0.005450667684071798,
+            0.012212046004369696,
+            -0.0008714643856749736,
+            -0.004560548606687623,
+            0.006746171236132563,
+            0.0034423786329218276,
+            -0.0009995673054631,
+            0.05643134636651925,
+            0.010296478797092773,
+            0.014377556149215558,
+            0.013825497434998718,
+            -0.017485595187900386,
+            0.013330532940316836,
+            0.013333976202885402,
+            0.012417102160127097,
+            0.004536479576179834,
+            -0.017256228281240734,
+            0.024337236913894427,
+            0.012755805964855683,
+            0.009622694631579495,
+            -0.0018516129725815202,
+            -0.0015877067248695194,
+            0.019827793457522933,
+            0.018266833138588977,
+            -0.009808363592736615,
+            0.011083663940891296,
+            0.004458465015627857,
+            -0.01068421391840819,
+            -0.0050807436529387556,
+            0.024563333063926475,
+            -0.01325209293057542,
+            -0.016539133573613984,
+            0.0059168302415277515,
+            -0.02488439856049784,
+            -0.029755917745609557,
+            0.001920092403971772,
+            0.006364691645397147,
+            0.002128416061225683,
+            -0.0021827593501562285,
+            0.00636889353893684,
+            0.01505350329221805,
+            -0.02223481012881443,
+            -0.01913380824474348,
+            0.004876437698740333,
+            -0.0031323018872010133,
+            0.015711985535422395,
+            -0.02142499388661488,
+            -0.011646214273628826,
+            0.003244037052013272,
+            -0.006889665577990765,
+            0.004421548112756594,
+            0.004704830748153097,
+            -0.004955793799612646,
+            -0.005239972361031197,
+            0.0007828188358756139,
+            0.019667585310240637,
+            0.017115475760556154,
+            0.02008255444898991,
+            0.012206187649127842,
+            -0.004915394518125827,
+            -0.00997481157928988,
+            0.002664275191354035,
+            -0.017453428141032505,
+            -0.005114569347643741,
+            0.006659824028894164,
+            0.013233844584497872,
+            -0.023794827807567007,
+            -0.007354308951883939,
+            0.012842257789962674,
+            0.011294252132532885,
+            -0.00824836528452984,
+            0.004256336058369505,
+            -0.011193345143979632,
+            0.009248664100690892,
+            0.010363170772580968,
+            0.010981447978991254,
+            -0.01250390246078834,
+            0.008163162620777558,
+            0.002528845973804195,
+            0.007003579605495149,
+            -0.005812703866552746,
+            0.008764871082692903,
+            0.013124123077340675,
+            -0.012212447674686537,
+            -0.0013940507123620205,
+            -0.001544873017903471,
+            -0.020493969663982807,
+            0.005782189204516662,
+            0.007033733269347455,
+            0.012878926833341835,
+            -0.008338295354334415,
+            0.015844568159504493,
+            -0.013423240193733334,
+            -0.00468920516522183,
+            0.0063319500880227525,
+            -0.004924778574104498,
+            -0.012053197667708071,
+            0.0017742020845283848,
+            -0.0035640186446405536,
+            -0.008666749197879684,
+            -0.0013894177719374007,
+            0.0016400693271668141,
+            0.004501038858023909,
+            0.02043221461176279,
+            0.001901506572332553,
+            -0.0056076433740056926,
+            0.019660750419391273,
+            0.003538773720733224,
+            -0.0018659146421562808,
+            0.005354994676719758,
+            0.0006789623595561136,
+            -0.0021516503031452773,
+            -0.0027566542393029298,
+            -0.010552067426241374,
+            0.01699044655141112,
+            -0.01883386136434379,
+            -0.006428123254736374,
+            -0.012514829515253224,
+            0.006923832110523631,
+            0.023065674483579466,
+            -0.0031719793455512423,
+            -0.006131653355136594,
+            -0.015640226612880566,
+            0.003589774262421098,
+            -0.07234225978543331,
+            -0.010636515019878534,
+            0.005436135654579842,
+            0.030362090167699224,
+            -0.008735845879763853,
+            0.0012226978854869956,
+            -0.00802187367908004,
+            0.010460901462092434,
+            -0.0024797678489701584,
+            0.005049311815982583,
+            0.0037643360764659196,
+            -0.003379308997651945,
+            -0.0029803517660911536,
+            0.0047612984744651015,
+            -0.010726918354479238,
+            -0.018870259480662628,
+            0.0028549213021684043,
+            0.025034678854420084,
+            0.009004274035878184,
+            0.004138885468102832,
+            0.009765532829693857,
+            -0.008537379113922283,
+            -0.0014902115840393893,
+            0.008462743183234337,
+            0.007328272099011593,
+            -0.009076904245691212,
+            -0.0008002047467975742,
+            0.008347212876126893,
+            -0.021799374855548816,
+            0.0020253943807330738,
+            0.012246389250647798,
+            -0.008912465890948629,
+            -0.006244789880293639,
+            0.002413038795382646,
+            0.004202974630810528,
+            -0.004339960146144246,
+            -0.009919301413657872,
+            -0.023862183602380987,
+            0.006400371798756263,
+            -0.007410957922309058,
+            -0.0001606112434129549,
+            0.0031662502228666104,
+            0.00012871401736865864,
+            -0.01687187870925491,
+            0.01374600947628664,
+            -0.008834671549515985,
+            -0.0033344289547729495,
+            0.012937173759658114,
+            -0.003267867578821276,
+            0.01864135643890992,
+            -0.014050228437162755,
+            0.007864998326137568,
+            -0.018988002335416778,
+            -0.003944450943320823,
+            -0.0032084001089808037,
+            -0.0023149926514803745,
+            -0.007750311931438483,
+            -0.045019596490520515,
+            0.013573650825957857,
+            -0.004374086208559833,
+            -0.005869005771503507,
+            -0.01250037134701792,
+            -0.016625210662158708,
+            0.011639132814162461,
+            0.004045997475412364,
+            -0.0014885281342871186,
+            -0.020321856013396592,
+            0.006222318025479057,
+            0.011121009394355636,
+            0.00904225037228108,
+            -0.011585148988670874,
+            -0.0018474735072673144,
+            -0.011325387916132704,
+            -0.0025890917877355725,
+            -0.009148717095934154,
+            0.018495348432506,
+            0.002404017871470538,
+            -0.010232069372010613,
+            0.022941087706292104,
+            -0.0024570420644715457,
+            0.020988341201251853,
+            -0.00442363926883869,
+            -0.004485898239018673,
+            -0.020447941631096866,
+            -0.004158168168969816,
+            -0.019358617104349624,
+            0.00036464612100595666,
+            0.0017749921060331022,
+            0.02078748568527368,
+            0.007256675717496499,
+            0.012612722550842472,
+            0.006032799184116686,
+            0.00631983142330159,
+            0.030151438614535002,
+            -0.010852408231267204,
+            0.0022451749744724233,
+            -0.017863828302314567,
+            0.010653806941450981,
+            0.028289026813870025,
+            0.009882042879445822,
+            -0.006920783006708277,
+            0.014053717954960316,
+            0.004932716665921658,
+            -0.00616100713292227,
+            0.025975267677767474,
+            -0.038836632659249616,
+            -0.021059874446788137,
+            -0.0061122622002863615,
+            0.001439949038430725,
+            -3.496872289060673e-05,
+            0.005030571386336924,
+            -0.011007922189367617,
+            0.009222155733548683,
+            0.0026655561130744807,
+            0.03101907459085608,
+            -0.002317540524509921,
+            -0.007348120500263025,
+            0.003987101796623003,
+            0.001511347369284844,
+            0.021229069065914295,
+            -0.01197457000398312,
+            -0.038429200853184124,
+            -0.0031917847963042626,
+            -0.0039488592790253305,
+            0.008371872140716302,
+            -0.012355524487988753,
+            -0.010839396359961,
+            -0.004574476927098761,
+            -0.01882180934875166,
+            -0.010292867074240718,
+            0.025595330740853118,
+            -0.007162452868646569,
+            0.01788200594893745,
+            -0.0076384034977797525,
+            -0.008112046063674713,
+            0.00683079906571602,
+            0.011572398977364099,
+            -0.01932794114157753,
+            0.0034055065356524052,
+            -0.0027824032743918262,
+            0.01627342501880651,
+            0.011718826886349412,
+            -0.013342112644751174,
+            0.006234666245077415,
+            0.01365078200475753,
+            -0.015476695744659024,
+            -0.0015378873781631448,
+            -0.0069121070205125585,
+            0.014866073481143516,
+            -0.003093423905641203,
+            -0.0033762696975023986,
+            0.011958295888528918,
+            -0.021982979614167224,
+            0.01697002117600754,
+            -0.02183323542526832,
+            0.022295922951583082,
+            -0.0006195453455557405,
+            -0.0006320640234089136,
+            0.008423668060539851,
+            0.008314680408345411,
+            -0.013378756052181496,
+            0.0016194432891316378,
+            -0.03416272433582183,
+            -0.009962172336151867,
+            -0.015051382650973002,
+            0.0038009542554177348,
+            -0.010106196779819915,
+            -0.01203081704994182,
+            -0.021869748479536547,
+            -0.013231917312580397,
+            0.009488867268928662,
+            0.02421155136770049,
+            -0.004349181861466944,
+            0.015665735941744613,
+            -0.008077879618601087,
+            -0.00346140349979159,
+            -0.019022501109520248,
+            -0.020158795455478176,
+            -0.011854460388465065,
+            -0.01570428154296937,
+            -0.007073290993912972,
+            0.004794071576762192,
+            -0.014492538032767992,
+            0.013391092874532436,
+            0.03615243857749714,
+            -0.017029836110560772,
+            0.008609740828901341,
+            0.019179133878629843,
+            0.0009923228421593012,
+            0.015393048602064972,
+            0.003790868078763006,
+            0.011710499244312703,
+            -0.014885526287290077,
+            -0.011829126494406653,
+            -0.0005627435441977781,
+            0.0020582596099707785,
+            0.003728052108395647,
+            0.027914733124690644,
+            0.0038604865752177283,
+            0.005302574635039923,
+            0.012293011832771443,
+            0.013320598642264739,
+            -0.010706185765391238,
+            0.015853648683536427,
+            0.004090821159159474,
+            -0.016980287717038522,
+            -0.0031197192883179663,
+            -0.006894138060845641,
+            -0.012739354473335233,
+            -0.002311031608856898,
+            0.0159372988694303,
+            -0.008091532959711314,
+            0.015169701976466646,
+            -0.004316674756425857,
+            -0.015090694840235596,
+            0.0007287957651103694,
+            -0.03810971752015664,
+            -0.0018729979445690102,
+            -0.01301867399306695,
+            -0.023136174288569147,
+            0.04241471786668668,
+            0.00415464531754789,
+            0.020836251111427635,
+            -0.006703690210920993,
+            0.006147438331654669,
+            -0.020673361951422737,
+            0.011783676068789637,
+            -0.0037516561524537765,
+            0.009702463136945385,
+            0.009366943837185086,
+            -0.018879598518030567,
+            -0.01132474738893142,
+            0.018484010977000243,
+            0.0008964474295952695,
+            0.001899853004269779,
+            0.014452586238316783,
+            0.019746940887134506,
+            0.01136527620507647,
+            -0.0020960934961233306,
+            0.02093114800466563,
+            0.018023169361765227,
+            -0.0002475129435970239,
+            -0.007118840223520255,
+            0.013527604253725194,
+            -0.02531530780276405,
+            0.014601163833111726,
+            0.03268673940622878,
+            0.013556815596615061,
+            0.020396267313433348,
+            -0.016161264086739215,
+            0.016259608464481445,
+            -0.006718110409512467,
+            0.017378727943854342,
+            0.001211924791251576,
+            0.01911265175070918,
+            0.014022657097776944,
+            0.01308266544853342,
+            0.006784756253744209,
+            -0.013367188766230505,
+            0.024435211011756285,
+            -0.008342072323415817,
+            -0.0004209982405190692,
+            -0.002823543536521797,
+            -0.023279946325307846,
+            0.01506543259962226,
+            0.0017095020186682082,
+            0.003411744637061027,
+            -0.0035317155649076414,
+            -0.001957702240702538,
+            0.008037874596341196,
+            0.008176809711854463,
+            -0.015780325305129808,
+            0.0031090160698441006,
+            -0.01924979849975192,
+            -0.008566500568783712,
+            -0.024111269290772643,
+            0.003704817310374187,
+            -0.015182531891766549,
+            -0.009971369164255839,
+            -0.019503345629597463,
+            -0.011275291123690537,
+            -0.010129027262987365,
+            0.00026549411615768147,
+            -0.0077553590287942975,
+            0.014297350661562049,
+            -0.017119422159171904,
+            -0.021341131823214652,
+            0.007945720599865192,
+            0.01439604029848557,
+            0.0037256533134793816,
+            0.005472351446140726,
+            0.008325407758477418,
+            0.009082034215555286,
+            -0.010830069811539129,
+            -0.010352768708989164,
+            0.001902868536460922,
+            0.0010307757364948558,
+            0.011572880679973516,
+            0.01480960939553474,
+            0.011328449881374384,
+            -0.013429607187664061,
+            0.0019326261581148884,
+            -0.014862631307250641,
+            0.004686987038977201,
+            -0.0011432874921856958,
+            -0.007026108142521568,
+            -0.0023122147402109927,
+            -0.01067305755615663,
+            0.008329587642130216,
+            0.0051774766682198365,
+            0.0014058609424645312,
+            0.0175095191841222,
+            -0.0016739751490535205,
+            -0.01295313808457633,
+            -0.012767766732175707,
+            -0.011653259281510757,
+            -0.015653450921388538,
+            0.006739771869002279,
+            -0.006908534840842188,
+            -0.006309735406143942,
+            0.00969748644524724,
+            -0.004300978285892996,
+            -0.0071082865141610775,
+            0.009870599453242536,
+            0.0003007194077865931,
+            -0.0011846789159672513,
+            0.03344372215649544,
+            -0.004482311849310301,
+            0.004153681423879411,
+            0.00022516282287549753,
+            -0.00096173441700326,
+            -0.01608715959652527,
+            -0.026963713856310283,
+            -0.0014323662880336336,
+            -0.02273696636049197,
+            0.017938234567205,
+            -0.0058065575851078514,
+            -0.001006759040376689,
+            0.002493842516193206,
+            -0.0042740143630253385,
+            -6.58295494211919e-05,
+            -0.011768915967276026,
+            0.0022431022211178383,
+            -0.029090994997079067,
+            0.009898341521414653,
+            -0.009544941198003441,
+            -0.02154260971685405,
+            0.01133866603878075,
+            0.040150135462722274,
+            -0.0034082812055093122,
+            0.0006477463440760081,
+            0.0010662318198694314,
+            0.010346430535548179,
+            0.014953186668393513,
+            -0.0019239187336010772,
+            -0.01603518108536549,
+            -0.010550241019652155,
+            0.007810013487665037,
+            0.0003366114893320438,
+            0.0030211673366186986,
+            0.025036691996012227,
+            -0.019097101590128545,
+            -0.016027314610013832,
+            -0.011102216330632247,
+            -0.011450582072890358,
+            0.009254748792585386,
+            0.019696746434919966,
+            0.013185751991632784,
+            -0.020709092762886617,
+            -0.017120964154502993,
+            0.00025737692261907426,
+            -0.02662183259737709,
+            -0.0015175233130328546,
+            0.12569047385271798,
+            -0.004250752280902458,
+            -0.009469510190778525,
+            -0.016304049953896647,
+            -0.00863767582150345,
+            0.02183050591695681,
+            -0.008697910752438377,
+            0.06936580824413162,
+            0.0006861269173917291,
+            -0.0029257096931296273,
+            -0.0020896063525880225,
+            -0.002993306834763656,
+            -0.013871927123029641,
+            0.001864949832879178,
+            -0.016847962023455045,
+            -0.01637518023758693,
+            -0.006332388047537449,
+            -0.013325712872379694,
+            -0.0017155737214029734,
+            -0.0061798136617018905,
+            0.004759205228314821,
+            -0.007830001529147273,
+            0.011495001353183024,
+            -0.026107152853762815,
+            -0.010130127689151806,
+            0.005962957535201191,
+            -0.011637257947784772,
+            0.0011307756612654312,
+            -0.030816554116329904,
+            -0.00441741097001083,
+            7.409036337997905e-05,
+            0.016501246370776126,
+            0.0015748302561177006,
+            -0.00940138978060357,
+            -0.011216292373500447,
+            0.006508185064247167,
+            0.021984022174755303,
+            0.02029850289834145,
+            0.020629050747183544,
+            0.009741513502990842,
+            0.017418812192334685,
+            -0.007031919478404256,
+            0.02625468226435525,
+            0.006196721544177288,
+            0.002206946290342702,
+            0.005596780938975919,
+            -0.01211153541496144,
+            -0.008534755574472074,
+            0.0074101096913398124,
+            -0.0219026467376419,
+            -0.0238583746438936,
+            -0.012260621283748488,
+            0.025739429552984337,
+            -0.009054977254128021,
+            -0.0019759724570709175,
+            -0.0030737356218219617,
+            0.024727831360321823,
+            -0.005149317224364814,
+            0.002332136352952536,
+            0.015587381251136284,
+            -0.028035348333895063,
+            0.008643405636450286,
+            0.007622365990511162,
+            0.02564089666732,
+            -0.0032690983288409517,
+            0.0030897564314706603,
+            -0.009711753474608342,
+            -0.009285462125807194,
+            -0.009613024258506932,
+            0.004240295765592783,
+            0.018961596475867332,
+            0.005797211858953558,
+            0.009922313409983964,
+            -0.006335322307636977,
+            -0.015248479856831972,
+            0.018517674622994848,
+            -0.018526204881631703,
+            -0.017805480495530986,
+            -0.0031014632675223346,
+            -0.023499333544416315,
+            0.007508363381003107,
+            0.006594465447638183,
+            -0.013304818586245387,
+            -0.0232989529893049,
+            0.020407788403667446,
+            0.003426820233878876,
+            -0.006520960224510217,
+            0.0028020004149410366,
+            -0.03282622678411901,
+            -0.018099884128291804,
+            0.004341732255376245,
+            0.012442902960644058,
+            -0.006914419410521592,
+            0.01684289728480369,
+            -0.0213314411022478,
+            0.021170682961259797,
+            -0.018444729425471068,
+            0.015400297173773751,
+            -0.005856807329212788,
+            0.01661347113847185,
+            0.011889835104373263,
+            0.0013585650928227913,
+            -0.019926344687386053,
+            0.02233564528528766,
+            0.0006518460244140545,
+            0.029977976063273147,
+            0.026669882018303386,
+            -0.005623183408671759,
+            -0.004497323576380724,
+            -0.017964594171161242,
+            0.011149712487916564,
+            0.0034380718823694635,
+            0.00295265329839474,
+            0.023835186931253683,
+            0.02803963164846445,
+            -0.028097235508129332,
+            0.017703040589712025,
+            -0.012893023698870347,
+            0.0089348477172039,
+            -0.011630331709306501,
+            0.0096768368423985,
+            0.00669273666692988,
+            0.003825890215203911,
+            0.008289398654583944,
+            0.011807275797649763,
+            0.024893949220958504,
+            0.02893733381281628,
+            0.004069052493622539,
+            0.012465630768347759,
+            -0.003600044227856378,
+            0.004954707230784128,
+            0.005985890100113585,
+            0.0014569711209781388,
+            -0.00648453455551435,
+            0.0031945554565833674,
+            -0.006090168119075504,
+            -0.003744222112779555,
+            0.015266602618825367,
+            -0.00028966361595811053,
+            0.015634624351841753,
+            0.012395295870300452,
+            -0.006577117978669073,
+            -0.003069372773552921,
+            -0.0029369830414480856,
+            -0.007598245840968154,
+            -0.017753359726409172,
+            0.025057809370078198,
+            -0.01606095813358513,
+            -0.019631204097884002,
+            0.0008782653868466668,
+            0.0031106969086903075,
+            -0.013675045629189149,
+            -0.008463196400259825,
+            0.008777824363912681,
+            0.014471929439779247,
+            -0.018478175346087953,
+            0.0017862044141924377,
+            -0.0004910793014860654,
+            0.013862986726603731,
+            -0.003641460118879092,
+            -0.00902817956483427,
+            -0.002076665354355939,
+            0.010393957588184604,
+            0.016413601414994997,
+            0.0038017609859922327,
+            -0.008644753777502864,
+            -0.0005484639867925445,
+            0.006452371249117759,
+            0.007391332924183688,
+            0.0036307379469783137,
+            -4.895894824382588e-05,
+            -0.008129811261078888,
+            -0.013655410823646597,
+            0.018829640339982075,
+            -0.007579460691893014,
+            0.016191139169828096,
+            -0.004984562798832502,
+            -0.011846989328030879,
+            0.004110855640416496,
+            0.0004775146965349116,
+            -0.014308885325290177,
+            0.002778951512640393,
+            -0.003583348080466357,
+            0.0017351729844533707,
+            0.006444402813802743,
+            -0.004945263366121632,
+            0.003803172980003619,
+            -0.008178062077842394,
+            0.02700102856098108,
+            0.015477045085093356,
+            0.005818362056942523,
+            -0.00876269298951217,
+            -0.024697543974427114,
+            0.008183329610246195,
+            0.00518473714350693,
+            0.018661120751406378,
+            0.014464207183253898,
+            0.0019868192486975638,
+            0.009827229842256728,
+            -0.013395371817208287,
+            -0.0022376892192351944,
+            0.009711801873474911,
+            0.017963581533215926,
+            -0.0059700736266576305,
+            0.019169874693002874,
+            -0.010316234316657931,
+            -0.0008125759237295765,
+            0.004003428870809722,
+            0.017109043212624506,
+            -0.004590883969173384,
+            0.013958072401225709
+          ],
+          "title": "F Pseudo Code for Switch Transformers 33",
+          "keyphrases": [
+            "Switch Transformers",
+            "Mixture-of-Experts",
+            "sparsely-activated expert model",
+            "natural language tasks",
+            "pre-training and fine-tuning"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": true,
+      "properties": {
+        "cosine_similarity": 0.8399206539153038
+      }
+    },
+    {
+      "id": "a02cce15-93bb-42c6-be38-408cee00e56f",
+      "type": "cosine_similarity",
+      "source": {
+        "id": "1e687808-8db8-4320-805d-0282a8d679bc",
+        "properties": {
+          "page_content": "Deepseekmoe: Towards Ultimate Expert Specialization In\n\nMixture-Of-Experts Language Models\n\nDamai Dai∗1,2, Chengqi Deng1, Chenggang Zhao∗1,3, R.X. Xu1, Huazuo Gao1, Deli Chen1, Jiashi Li1, Wangding Zeng1, Xingkai Yu∗1,4, Y. Wu1, Zhenda Xie1, Y.K. Li1, Panpan Huang1, Fuli Luo1, Chong Ruan1, Zhifang Sui2, Wenfeng Liang1 1DeepSeek-AI 2National Key Laboratory for Multimedia Information Processing, Peking University 3Institute for Interdisciplinary Information Sciences, Tsinghua University 4National Key Laboratory for Novel Software Technology, Nanjing University {daidamai, szf}@pku.edu.cn, {wenfeng.liang}@deepseek.com https://github.com/deepseek-ai/DeepSeek-MoE\n\nAbstract\n\nIn the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top- out of experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into ones and activating from them, allowing for a more flexible combination of activated experts; (2) isolating experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5× expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations.\n\nFurther, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\n1. Introduction\n\nRecent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023a). It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017) has emerged as a popular solution. It can\n\nenable parameter scaling, while concurrently keeping computational costs at a modest level.\n\nRecent applications of MoE architectures in Transformers (Vaswani et al., 2017) have yielded successful attempts at scaling language models to a substantial size (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022), accompanied with remarkable performance. These achievements underscore the considerable potential and promise of MoE language models.\n\nDespite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts.\n\nThis architecture manifests two potential issues: (1) Knowledge Hybridity: existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. (2) Knowledge Redundancy: tokens assigned to different experts may require common knowledge. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters.\n\nThese issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models.\n\nIn response to the aforementioned issues, we introduce DeepSeekMoE, an innovative MoE architecture specifically designed towards ultimate expert specialization. Our architecture involves two principal strategies: (1) Fine-Grained Expert Segmentation: while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. (2) Shared Expert Isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.\n\nThrough compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized.\n\nStarting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B (Lepikhin et al., 2021) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5× expert parameters and computation. Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE.\n\nThese studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization.\n\nLeveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B (DeepSeek-AI, 2024), a dense model trained on the same 2T corpus. We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B (Touvron et al., 2023b), which has approximately 2.5 times the activated parameters. Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard1. Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. The experimental results still validate its substantial advantages over the GShard architecture consistently. In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nOur contributions are summarized as follows: - Architectural Innovation. We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation.\n\nEmpirical Validation. We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. Experimental results validate the high 1https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models\n\nScalability. We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B.\n\nAlignment for MoE. We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B.\n\nPublic Release. In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization.\n\n2. Preliminaries: Mixture-Of-Experts For Transformers\n\nWe first introduce a generic MoE architecture commonly used in Transformer language models. A standard Transformer language model is constructed by stacking layers of standard Transformer blocks, where each block can be represented as follows:\n\n$$\\mathbf{u}{1:T}^{l}=\\text{Self-Att}\\left(\\mathbf{h}{1:T}^{l-1}\\right)+\\mathbf{h}{1:T}^{l-1},\\tag{1}$$ $$\\mathbf{h}{t}^{l}=\\text{FFN}\\left(\\mathbf{u}{t}^{l}\\right)+\\mathbf{u}{t}^{l},\\tag{2}$$\n\nwhere denotes the sequence length, Self-Att(·) denotes the self-attention module, FFN(·) denotes the Feed-Forward Network (FFN), u 1: ∈ R× are the hidden states of all tokens after the -th attention module, and h\n\n∈ Ris the output hidden state of the -th token after the -th Transformer block. For brevity, we omit the layer normalization in the above formulations.\n\nA typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022). An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. Then, each token will be assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts. If the -th FFN is substituted with an MoE layer, the computation for its output hidden state h\n\nis expressed as: where denotes the total number of experts, FFN(·) is the -th expert FFN, , denotes the gate value for the -th expert, , denotes the token-to-expert affinity, Topk(·, ) denotes the set comprising highest affinity scores among those calculated for the -th token and all experts, and e\n\nis the centroid of the -th expert in the -th layer. Note that ,is sparse, indicating that only out of gate values are nonzero. This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only experts. Also, in the above formulations, we omit the layer normalization operation for brevity.\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{N}\\left(g_{i,t}\\operatorname{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\operatorname{Topk}({s_{j,t}|1\\leqslant j\\leqslant N},K),\\ 0,&\\text{otherwise},\\end{cases}$$ $$s_{i,t}=\\operatorname{Softmax}{i}\\left(\\mathbf{u}{t}^{l}\\,\\mathbf{e}_{i}^{l}\\right),$$\n\n$$(3)$$\n\n(4) $\\binom{4}{5}$ .\n\n3. Deepseekmoe Architecture\n\nOn top of the generic MoE architecture outlined in Section 2, we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. As illustrated in Figure 2, our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. Both of these strategies are designed to elevate the level of expert specialization.\n\n3.1. Fine-Grained Expert Segmentation\n\nIn scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts.\n\nIn pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. The finer expert segmentation enables a more flexible and adaptable combination of activated experts. To be specific, on top of a typical MoE architecture shown in Figure 2(a), we segment each expert FFN into smaller experts by reducing the FFN intermediate hidden dimension to 1 times its original size. Since each expert becomes smaller, in response, we also increase the number of activated experts to times to keep the same computation cost, as illustrated in Figure 2(b). With the fine-grained expert segmentation, the output of an MoE layer can be expressed as:\n\n$$\\mathbf{h}{t}^{i}=\\sum{i=1}^{\\text{mN}}\\left(g_{i,t}\\text{FFN}{i}\\left(\\mathbf{u}{t}^{i}\\right)\\right)+\\mathbf{u}{t}^{i},\\tag{6}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\text{Tokr}(s_{i,t}|1\\leqslant j\\leqslant m\\text{N}),m\\text{K}),\\ 0,&\\text{otherwise},\\end{cases}$$ (7) $$s_{i,t}=\\text{Softmax}{i}\\left(\\mathbf{u}{t}^{T}\\mathbf{e}_{t}^{i}\\right),\\tag{8}$$ where the total number of expert parameters is equal to $N$ times the number of parameters in a\n\n$$(6)$$\n\n(7) (8) $\\frac{1}{2}$\n\nstandard FFN, and denotes the total number of fine-grained experts. With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to .\n\nFrom a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. As an illustrative example, we consider the case where = 16. A typical top-2 routing strategy can yield 16 2 = 120 possible combinations. By contrast, if each expert is split into 4 smaller experts, the fine-grained routing strategy can yield 64 8 = 4, 426, 165, 368 potential combinations. The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition.\n\n3.2. Shared Expert Isolation\n\nWith a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. This alleviation of redundancy will contribute to a more parameterefficient model with more specialized experts.\n\nTowards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate experts to serve as shared experts. Regardless of the router module, each token will be deterministically assigned to these shared experts. In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by , as depicted in Figure 2(c). With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows:\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{K_{s}}\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)+\\sum_{i=K_{s}+1}^{mN}\\left(g_{i,t}\\;\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},\\tag{9}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\mathrm{Topk}{s_{j,t}|K_{s}+1\\leqslant j\\leqslant mN},mK-K_{s}},\\ 0,&\\text{otherwise},\\end{cases}$$ (10) $$s_{i,t}=\\mathrm{Softmax}{i}\\left(\\mathbf{u}{t}^{l,T}e_{i}^{l}\\right).\\tag{11}$$ $\\mathbf{h}\\mathbf{M}\\mathbf{E}$ the number of closed-compact is $K_{s}$ the total number of routed currents.\n\nFinally, in DeepSeekMoE, the number of shared expert is , the total number of routed experts is − , and the number of nonzero gates is − .\n\nIt is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022). The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint.\n\n3.3. Load Balance Consideration\n\nAutomatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. Firstly, there is a risk of routing collapse (Shazeer et al., 2017), i.e., the model always selects only a few experts, preventing other experts from sufficient training.\n\nSecondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.\n\nExpert-Level Balance Loss. In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. The computation of the balance loss is as follows:\n\n$$\\begin{array}{l}{{{\\mathcal L}{\\mathrm{ExpBal}}=\\alpha{1}\\sum_{i=1}^{N^{\\prime}}f_{i}P_{i},}}\\ {{f_{i}=\\frac{N^{\\prime}}{K^{T}}\\sum_{t=1}^{T}\\mathds{1}\\,(\\mathrm{Token}\\;t\\;\\mathrm{s}\\mathrm{lects}\\;\\mathrm{Expert}\\;i),}}\\ {{P_{i}=\\frac{1}{T}\\sum_{t=1}^{T}s_{i,t},}}\\end{array}$$\n\n(12) $$\\begin{array}{l}\\small\\text{(13)}\\end{array}$$ = (14) $$\\begin{array}{l}\\small\\text{(14)}\\end{array}$$ .\n\n7 where 1 is a hyper-parameter called expert-level balance factor, ′is equal to ( − ) and ′ is equal to ( − ) for brevity. 1(·) denotes the indicator function.\n\nDevice-Level Balance Loss. In addition to the expert-level balance loss, we introduce a devicelevel balance loss. When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. Instead, our primary objective is to ensure balanced computation across the devices. If we partition all routed experts into groups {E1, E2, ..., E}, and deploy each group on a single device, the device-level balance loss is computed as follows:\n\n$$\\mathcal{L}{\\text{DevBal}}=\\alpha{2}\\sum_{i=1}^{D}f_{i}^{\\prime}P_{i}^{\\prime},$$ $$f_{i}^{\\prime}=\\frac{1}{|\\mathcal{E}{i}|}\\sum{j\\in\\mathcal{E}{i}}f{j},$$ $$P_{i}^{\\prime}=\\sum_{j\\in\\mathcal{E}{i}}P{j},$$\n\n(15) $$\\begin{array}{l}\\small\\text{(16)}\\end{array}$$ = (17) .\n\nwhere 2 is a hyper-parameter called device-level balance factor. In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices.\n\n4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n6. Alignment For Deepseekmoe 16B\n\nPrevious research indicates that MoE models typically do not emerge significant gains from fine-tuning (Artetxe et al., 2022; Fedus et al., 2021). However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B.\n\n6.1. Experimental Setup\n\nTraining Data. For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios.\n\nHyper-Parameters. During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).\n\nWe employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10−5 without incorporating any learning rate scheduling strategy.\n\nEvaluation Benchmarks. For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3, with the following adjustments: (1) We exclude Pile (Gao et al., 2020) since chat models are seldom employed for pure language modeling. (2) We exclude\n\nMetric # Shot LLaMA2 DeepSeek DeepSeekMoE SFT 7B Chat 7B Chat 16B # Total Params N/A 6.7B 6.9B 16.4B # Activated Params N/A 6.7B 6.9B 2.8B FLOPs per 4K Tokens N/A 187.9T 183.5T 74.4T HellaSwag (Acc.) 0-shot 67.9 71.0 72.2 PIQA (Acc.) 0-shot 76.9 78.4 79.7 ARC-easy (Acc.) 0-shot 69.7 70.2 69.9 ARC-challenge (Acc.) 0-shot 50.8 50.2 50.0 BBH (EM) 3-shot 39.3 43.1 42.2 RACE-middle (Acc.) 5-shot 63.9 66.1 64.8 RACE-high (Acc.) 5-shot 49.6 50.8 50.6 DROP (EM) 1-shot 40.0 41.7 33.8 GSM8K (EM) 0-shot 63.4 62.6 62.2 MATH (EM) 4-shot 13.5 14.7 15.2 HumanEval (Pass@1) 0-shot 35.4 45.1 45.7 MBPP (Pass@1) 3-shot 27.8 39.0 46.2 TriviaQA (EM) 5-shot 60.1 59.5 63.3 NaturalQuestions (EM) 0-shot 35.2 32.7 35.1 MMLU (Acc.) 0-shot 50.0 49.7 47.2 WinoGrande (Acc.) 0-shot 65.1 68.4 69.0 CLUEWSC (EM) 5-shot 48.4 66.2 68.2 CEval (Acc.) 0-shot 35.1 44.7 40.0 CMMLU (Acc.) 0-shot 36.9 51.2 49.3\n\nCHID (Zheng et al., 2019) due to the observed instability of results, hindering the derivation of solid conclusions. (3) We additionally include BBH (Suzgun et al., 2022) to provide a more comprehensive assessment of the reasoning ability of the chat models.\n\nTable 5 | Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations.\n\n6.2. Evaluations\n\nBaselines. In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. Correspondingly, we construct three chat models, including LLaMA2 SFT 7B3, DeepSeek Chat 7B, and DeepSeekMoE Chat 16B.\n\nSubsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks.\n\nResults. The evaluation results are presented in Table 5. Our key observations include: (1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intensive tasks (TriviaQA, NaturalQuestions). (2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. In addition, it also surpasses DeepSeek Chat 7B. (3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1). However, it is worth noting that, after supervised finetuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. (4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations.\n\n7. Deepseekmoe 145B Ongoing\n\nEncouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available.\n\n7.1. Experimental Setup\n\nTraining Data and Tokenization. For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study.\n\nModel Settings. For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps.\n\nSubsequently, the learning rate keeps constant during the remaining training process. The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. Also, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse.\n\nEvaluation Benchmarks. We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3).\n\n7.2. Evaluations\n\nBaselines. Apart from DeepSeekMoE 145B, we consider three additional models for comparison. DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters.\n\nResults. From the evaluation results presented in Table 6, we have the following observations: (1) Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. (2) On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. (3) At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5.\n\n8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\nAppendices A. Overview Of Hyper-Parameters\n\nWe present the overview of hyper-parameters for DeepSeekMoE across various sizes in Table 7.\n\n# Params # Layers Hidden # Attn # Shared # Routed Relative Sequence Batch Size Learning Size Heads Experts Experts Expert Size Length (Sequence) Rate 2.0B 9 1280 10 1 63 (7 activated) 0.25 2048 2048 1.08e-3 16.4B 28 2048 16 2 64 (6 activated) 0.25 4096 4608 4.2e-4 144.6B 62 4096 32 4 128 (12 activated) 0.125 4096 4608 3.0e-4\n\nTable 7 | Overview of hyper-parameters for DeepSeekMoE across various sizes. The relative expert size is in comparison to a standard FFN.\n\nB. Comparing Deepseekmoe With Larger Models\n\nComparisons among DeepSeekMoE, GShard×1.2, and GShard×1.5 are shown in Table 8. Comparisons among DeepSeekMoE, Dense×4, and Dense×16 are shown in Table 9.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 2.3B 2.8B 1.9B # Activated Expert Params N/A 0.28B 0.35B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.824 1.808 1.808 HellaSwag (Acc.) 0-shot 53.7 54.4 54.8 PIQA (Acc.) 0-shot 71.8 71.1 72.3 ARC-easy (Acc.) 0-shot 46.8 47.3 49.4 ARC-challenge (Acc.) 0-shot 31.7 34.1 34.3 RACE-middle (Acc.) 5-shot 43.7 46.4 44.0 RACE-high (Acc.) 5-shot 31.9 32.4 31.7 HumanEval (Pass@1) 0-shot 3.7 3.0 4.9 MBPP (Pass@1) 3-shot 2.4 2.6 2.2 TriviaQA (EM) 5-shot 15.2 15.7 16.6 NaturalQuestions (EM) 5-shot 4.5 4.7 5.7\n\nTable 8 | Comparison between DeepSeekMoE and larger GShard models.\n\nAt a larger scale of 13B total parameters, we also compare DeepSeekMoE with GShard×1.2 and GShard×1.5, and show results in Table 10. At a larger scale, DeepSeekMoE even outperforms GShard×1.5 distinctly.\n\nMetric # Shot Dense×4 Dense×16 DeepSeekMoE Relative Expert Size N/A 1 1 0.25 # Experts N/A 4 + 0 16 + 0 1 + 63 # Activated Experts N/A 4 + 0 16 + 0 1 + 7 # Total Expert Params N/A 0.47B 1.89B 1.89B # Activated Expert Params N/A 0.47B 1.89B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.908 1.806 1.808 HellaSwag (Acc.) 0-shot 47.6 55.1 54.8 PIQA (Acc.) 0-shot 70.0 71.9 72.3 ARC-easy (Acc.) 0-shot 43.9 51.9 49.4 ARC-challenge (Acc.) 0-shot 30.5 33.8 34.3 RACE-middle (Acc.) 5-shot 42.4 46.3 44.0 RACE-high (Acc.) 5-shot 30.7 33.0 31.7 HumanEval (Pass@1) 0-shot 1.8 4.3 4.9 MBPP (Pass@1) 3-shot 0.2 2.2 2.2 TriviaQA (EM) 5-shot 9.9 16.5 16.6 NaturalQuestions (EM) 5-shot 3.0 6.3 5.7\n\nTable 9 | Comparison between DeepSeekMoE and larger dense baselines.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 15.9B 19.8B 13.3B # Activated Expert Params N/A 2.37B 2.82B 2.05B # Training Tokens N/A 100B 100B 100B HellaSwag (Acc.) 0-shot 66.6 67.7 69.1 PIQA (Acc.) 0-shot 75.6 76.0 75.7 ARC-easy (Acc.) 0-shot 56.8 56.8 58.8 ARC-challenge (Acc.) 0-shot 39.9 37.6 38.5 RACE-middle (Acc.) 5-shot 51.6 50.6 52.4 RACE-high (Acc.) 5-shot 37.4 36.3 38.5 HumanEval (Pass@1) 0-shot 6.1 6.1 9.8 MBPP (Pass@1) 3-shot 7.0 11.6 10.6 TriviaQA (EM) 5-shot 36.5 36.7 38.2 NaturalQuestions (EM) 5-shot 12.6 12.1 13.7\n\nTable 10 | Comparison between DeepSeekMoE and larger GShard models at a larger scale.\n\nC. Training Benchmark Curves Of Deepseekmoe 16B\n\nWe present the benchmark curves during training of DeepSeekMoE 16B and DeepSeek 7B (Dense) in Figure 7 for reference.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/DeepSeekMoE_2.md"
+          },
+          "summary": "DeepSeekMoE is an innovative Mixture-of-Experts (MoE) architecture designed to enhance expert specialization in language models. It employs fine-grained expert segmentation and shared expert isolation to improve performance and reduce computational costs. Starting with a model of 2B parameters, DeepSeekMoE demonstrates comparable performance to larger models like GShard 2.9B and LLaMA2 7B, using significantly fewer computations. The architecture scales up to 16B and 145B parameters, maintaining advantages over traditional MoE architectures. DeepSeekMoE 16B achieves similar results to dense models with only 40% of the computations, and its chat model version shows adaptability in bilingual scenarios. The architecture's public release aims to contribute to advancements in large-scale language models.",
+          "headlines": {
+            "1. Introduction": [],
+            "2. Preliminaries: Mixture-Of-Experts For Transformers": [],
+            "3. Deepseekmoe Architecture": [
+              "3.1. Fine-Grained Expert Segmentation",
+              "3.2. Shared Expert Isolation",
+              "3.3. Load Balance Consideration"
+            ],
+            "4. Validation Experiments": [
+              "4.1. Experimental Setup",
+              "4.1.1. Training Data And Tokenization",
+              "4.1.2. Infrastructures",
+              "4.1.3. Hyper-Parameters",
+              "4.1.4. Evaluation Benchmarks",
+              "4.2. Evaluations",
+              "4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models",
+              "4.4. Ablation Studies",
+              "4.5. Analysis On Expert Specialization"
+            ],
+            "5. Scaling Up To Deepseekmoe 16B": [
+              "5.1. Experimental Setup",
+              "5.1.1. Training Data And Tokenization",
+              "5.1.2. Hyper-Parameters",
+              "5.1.3. Evaluation Benchmarks",
+              "5.2. Evaluations",
+              "5.2.1. Internal Comparison With Deepseek 7B",
+              "5.2.2. Comparison With Open Source Models"
+            ],
+            "6. Alignment For Deepseekmoe 16B": [
+              "6.1. Experimental Setup",
+              "6.2. Evaluations"
+            ],
+            "7. Deepseekmoe 145B Ongoing": [
+              "7.1. Experimental Setup",
+              "7.2. Evaluations"
+            ],
+            "8. Related Work": [],
+            "9. Conclusion": [],
+            "Appendices": [
+              "A. Overview Of Hyper-Parameters",
+              "B. Comparing Deepseekmoe With Larger Models",
+              "C. Training Benchmark Curves Of Deepseekmoe 16B"
+            ]
+          },
+          "summary_embedding": [
+            -0.012465757317841053,
+            -0.022808987647294998,
+            -0.018792515620589256,
+            -0.02308654971420765,
+            0.01797616109251976,
+            0.007200260180979967,
+            0.006490030791610479,
+            0.042613785713911057,
+            -0.014049487188458443,
+            0.03187054023146629,
+            -0.021486490964889526,
+            -0.023592690005898476,
+            0.009045224636793137,
+            -0.014441338367760181,
+            0.005865518003702164,
+            -0.007363531738519669,
+            -0.008743173442780972,
+            -0.005163452122360468,
+            0.018841497600078583,
+            -0.03100520372390747,
+            0.011110605672001839,
+            -0.046075135469436646,
+            -0.020017050206661224,
+            -0.021845687180757523,
+            0.016425084322690964,
+            0.037389107048511505,
+            0.022547753527760506,
+            -0.0028062239289283752,
+            -0.06844329088926315,
+            0.034678805619478226,
+            0.035168617963790894,
+            0.02052319049835205,
+            0.0005438972148112953,
+            -0.00031302153365686536,
+            -0.024833550676703453,
+            -0.028556134551763535,
+            0.03347059711813927,
+            -0.004379750229418278,
+            -0.01453930139541626,
+            0.015976088121533394,
+            0.0414055772125721,
+            0.012106560170650482,
+            -0.02754385396838188,
+            0.0014245413476601243,
+            0.015151568688452244,
+            0.0055961208418011665,
+            -0.01678428053855896,
+            -0.0022960014175623655,
+            -0.002424577483907342,
+            -0.021225256845355034,
+            -0.01400050614029169,
+            0.018825169652700424,
+            0.001296985661610961,
+            -0.008865626528859138,
+            -0.0542386956512928,
+            0.007330877240747213,
+            0.03840138763189316,
+            0.05623060464859009,
+            0.044148534536361694,
+            -0.020180322229862213,
+            0.03369917720556259,
+            -0.030025575309991837,
+            -0.02501314878463745,
+            0.011028970591723919,
+            0.003983817528933287,
+            -0.018531281501054764,
+            0.01784554310142994,
+            -0.005636938381940126,
+            -0.0006556359585374594,
+            0.01751900091767311,
+            -0.03941367194056511,
+            0.020751770585775375,
+            -0.01887415163218975,
+            0.031266435980796814,
+            0.02173139713704586,
+            0.0073798587545752525,
+            0.01872720755636692,
+            0.01678428053855896,
+            0.0036858476232737303,
+            -0.04107903689146042,
+            0.009118696674704552,
+            0.016523046419024467,
+            0.014196431264281273,
+            0.016269976273179054,
+            0.06553706526756287,
+            -0.05391215533018112,
+            -0.004485876299440861,
+            -0.05169166624546051,
+            0.0012561678886413574,
+            -0.004971608519554138,
+            -0.03670336678624153,
+            0.01048201136291027,
+            0.028425518423318863,
+            0.021372200921177864,
+            -0.01063711941242218,
+            0.024392718449234962,
+            -0.022743679583072662,
+            0.0032001156359910965,
+            -0.04062187671661377,
+            0.0019653772469609976,
+            0.004518530797213316,
+            0.016531210392713547,
+            0.0070778070949018,
+            0.004273623693734407,
+            -0.016702644526958466,
+            0.006045116577297449,
+            -0.01905374974012375,
+            0.03291547670960426,
+            -0.008465612307190895,
+            0.08653374016284943,
+            0.008751336485147476,
+            0.004738946910947561,
+            0.004281787667423487,
+            0.02398454025387764,
+            0.012131051160395145,
+            -0.00810233410447836,
+            -0.04718538001179695,
+            -0.015690363943576813,
+            0.011061624623835087,
+            -0.0383034273982048,
+            0.018335357308387756,
+            -0.008759500458836555,
+            0.01282495353370905,
+            -0.04107903689146042,
+            -0.06344719231128693,
+            0.01498013362288475,
+            0.001784758409485221,
+            -0.004065453074872494,
+            -0.0002903166168835014,
+            -0.018923133611679077,
+            0.003930754493921995,
+            -0.004200151655822992,
+            -0.010645282454788685,
+            -0.021502818912267685,
+            0.03159298002719879,
+            0.03657275065779686,
+            0.06527583301067352,
+            0.036801330745220184,
+            -0.0030164355412125587,
+            0.005775718949735165,
+            0.019690508022904396,
+            -0.020784424617886543,
+            0.003481758525595069,
+            -0.01214737817645073,
+            -0.021992631256580353,
+            -0.03820546343922615,
+            0.010971825569868088,
+            0.07379858940839767,
+            -0.029029620811343193,
+            0.013437220826745033,
+            -0.017616963014006615,
+            0.04205866530537605,
+            -0.004567512311041355,
+            0.026727497577667236,
+            -0.016669990494847298,
+            0.004059330560266972,
+            -0.025388672947883606,
+            0.01620466820895672,
+            -0.024964168667793274,
+            0.012571883387863636,
+            -0.000885746325366199,
+            -0.04222193360328674,
+            0.00011116473615402356,
+            0.026890767738223076,
+            -0.010971825569868088,
+            0.02261306345462799,
+            0.0004949158756062388,
+            -0.006302268709987402,
+            -0.05097327008843422,
+            0.038727931678295135,
+            -0.003912386484444141,
+            0.018645571544766426,
+            -0.0018959869630634785,
+            0.0163107942789793,
+            -0.0016408757073804736,
+            0.013216804713010788,
+            0.009543201886117458,
+            0.007114543113857508,
+            -0.033503253012895584,
+            0.02697240374982357,
+            -0.02504580281674862,
+            0.028049994260072708,
+            0.03435226157307625,
+            -0.0030184765346348286,
+            0.03255628049373627,
+            0.019249675795435905,
+            0.03840138763189316,
+            0.0327032245695591,
+            0.030335791409015656,
+            -0.00824927818030119,
+            0.013878053054213524,
+            0.008200296200811863,
+            0.03608293831348419,
+            0.013429056853055954,
+            0.019723162055015564,
+            -0.009730963967740536,
+            -0.014041324146091938,
+            0.048785436898469925,
+            0.007318632211536169,
+            -0.0179598331451416,
+            -0.005441013257950544,
+            -0.03510330989956856,
+            -0.002832755446434021,
+            -0.014425011351704597,
+            0.0323440283536911,
+            0.02158445306122303,
+            -0.01647406443953514,
+            0.035135965794324875,
+            0.024980494752526283,
+            -0.03751972317695618,
+            -0.005232842173427343,
+            -0.02026195637881756,
+            -0.023298801854252815,
+            -0.04215662553906441,
+            0.011624909937381744,
+            0.012833117507398129,
+            -0.013976015150547028,
+            -0.007469657808542252,
+            0.020457882434129715,
+            -0.023625344038009644,
+            -0.0018265966791659594,
+            -0.01356783788651228,
+            0.006547175347805023,
+            -0.020898714661598206,
+            -0.03667071461677551,
+            -0.01048201136291027,
+            0.025862159207463264,
+            0.012637191452085972,
+            0.04362606629729271,
+            -0.013518855907022953,
+            -0.005367541220039129,
+            0.033193036913871765,
+            0.012759645469486713,
+            0.010767736472189426,
+            0.03699725493788719,
+            -0.03027048148214817,
+            0.024621298536658287,
+            -0.020294612273573875,
+            -0.0032878739293664694,
+            0.01707816869020462,
+            -0.0030062312725931406,
+            0.01962519995868206,
+            -0.04535674303770065,
+            -0.024196794256567955,
+            0.04395261034369469,
+            -0.0007107399869710207,
+            0.016302630305290222,
+            -0.005959399044513702,
+            -0.012106560170650482,
+            0.0035899258218705654,
+            0.0027715289033949375,
+            -0.01438419334590435,
+            -0.008620720356702805,
+            0.007473739795386791,
+            -0.01962519995868206,
+            0.03660540655255318,
+            0.03794422745704651,
+            -0.004132802598178387,
+            -0.014131123200058937,
+            -0.03324201703071594,
+            0.010016689077019691,
+            -0.017878197133541107,
+            -0.023951886221766472,
+            0.04078514873981476,
+            -0.032115448266267776,
+            -0.00029312283731997013,
+            0.039478980004787445,
+            0.018808843567967415,
+            -0.014898497611284256,
+            0.00868602842092514,
+            -0.01709449477493763,
+            -0.026302991434931755,
+            -0.017159804701805115,
+            0.03356856107711792,
+            0.024800896644592285,
+            -0.010743245482444763,
+            0.010245268233120441,
+            0.005330805201083422,
+            0.02893165871500969,
+            0.024376392364501953,
+            0.00648594880476594,
+            -0.013812744058668613,
+            0.015029114671051502,
+            0.027641816064715385,
+            -0.0028021421749144793,
+            -7.66609373386018e-05,
+            -0.004010349046438932,
+            -0.007796200457960367,
+            -0.006453294772654772,
+            -0.013069860637187958,
+            -0.005089980084449053,
+            0.0009184005321003497,
+            -0.032295044511556625,
+            0.015967924147844315,
+            -0.005890008993446827,
+            0.010359558276832104,
+            -0.011535110883414745,
+            0.01288209855556488,
+            0.01722511276602745,
+            -0.013494365848600864,
+            -0.0716434046626091,
+            0.014612773433327675,
+            0.02458864450454712,
+            -0.006706364918500185,
+            -0.017943505197763443,
+            0.008481939323246479,
+            0.00016148542636074126,
+            0.038597315549850464,
+            -0.014180104248225689,
+            -0.031103165820240974,
+            0.02680913358926773,
+            -0.017437364906072617,
+            -0.015241367742419243,
+            -0.010661610402166843,
+            -0.00580837344750762,
+            0.04649963974952698,
+            0.01889047957956791,
+            0.011420820839703083,
+            -0.04042595252394676,
+            -0.007449248805642128,
+            0.015078096650540829,
+            0.014506647363305092,
+            -0.04705476388335228,
+            0.0052083516493439674,
+            0.015486273914575577,
+            0.019723162055015564,
+            0.024800896644592285,
+            0.04274440184235573,
+            -0.0021531390957534313,
+            0.01661284640431404,
+            -0.015053605660796165,
+            -0.05136512219905853,
+            -0.03340528905391693,
+            0.029943939298391342,
+            -0.017731253057718277,
+            -0.02906227484345436,
+            -0.01693122461438179,
+            -0.006151242647320032,
+            0.0017939424142241478,
+            0.007126788143068552,
+            -0.01916803978383541,
+            -0.03464614972472191,
+            -0.03970756009221077,
+            0.02277633361518383,
+            0.007983962073922157,
+            0.004269542172551155,
+            -0.042777057737112045,
+            0.0013459670590236783,
+            0.009184005670249462,
+            0.012073906138539314,
+            0.0009954441338777542,
+            -0.019984396174550056,
+            0.015494437888264656,
+            0.0032001156359910965,
+            -0.031070511788129807,
+            -0.035593122243881226,
+            -0.03595232218503952,
+            0.0023919232189655304,
+            0.003781769424676895,
+            0.06279410421848297,
+            0.012988224625587463,
+            -0.03840138763189316,
+            -0.04522612690925598,
+            0.029176564887166023,
+            -0.02052319049835205,
+            -0.016678154468536377,
+            0.0059430720284581184,
+            0.012898425571620464,
+            -0.009877908043563366,
+            0.013192313723266125,
+            0.023560035973787308,
+            0.041601505130529404,
+            -0.005987971555441618,
+            0.012914752587676048,
+            0.023037567734718323,
+            0.012955570593476295,
+            0.0362788625061512,
+            -0.00936360377818346,
+            0.0035409443080425262,
+            0.039936136454343796,
+            -0.012269831262528896,
+            0.03175625205039978,
+            -0.02189466916024685,
+            0.0020398697815835476,
+            0.02249877341091633,
+            0.027184655889868736,
+            -0.0006479826406575739,
+            0.02594379521906376,
+            -0.031543996185064316,
+            0.018368011340498924,
+            -0.014498483389616013,
+            -0.04888340085744858,
+            -0.01557607389986515,
+            0.013102514669299126,
+            0.030384771525859833,
+            0.02756018005311489,
+            -0.0041348435916006565,
+            -0.019723162055015564,
+            -0.01155143789947033,
+            -0.010596301406621933,
+            0.0018419033149257302,
+            0.03285016864538193,
+            0.010506502352654934,
+            0.010890189558267593,
+            -0.011935125105082989,
+            -0.004104230087250471,
+            0.03025415539741516,
+            -0.021192602813243866,
+            0.01650671847164631,
+            0.0041511706076562405,
+            -0.002029665280133486,
+            -0.04320156201720238,
+            -0.035266581922769547,
+            0.007845181971788406,
+            0.011747363954782486,
+            0.014874007552862167,
+            0.0368339866399765,
+            0.025094784796237946,
+            -0.01662100851535797,
+            0.030629679560661316,
+            -0.01905374974012375,
+            0.027984686195850372,
+            0.039054471999406815,
+            0.011624909937381744,
+            -0.018351683393120766,
+            0.012237177230417728,
+            0.002424577483907342,
+            -0.025127438828349113,
+            0.01680060662329197,
+            -0.032735876739025116,
+            0.01738838478922844,
+            -0.02354370802640915,
+            0.025143766775727272,
+            -0.02579685114324093,
+            -0.009290131740272045,
+            -0.004428731743246317,
+            -0.006130834110081196,
+            0.04888340085744858,
+            -0.026727497577667236,
+            -0.0142454132437706,
+            -0.0024694770108908415,
+            -0.0024531499948352575,
+            -0.008718682453036308,
+            0.010000362060964108,
+            -0.045781247317790985,
+            -0.01844964735209942,
+            -0.04016471654176712,
+            0.007355368230491877,
+            0.005412440747022629,
+            -0.030907239764928818,
+            -0.05694899708032608,
+            0.025421326979994774,
+            -0.06331657618284225,
+            -0.019886434078216553,
+            0.051626358181238174,
+            -0.018400665372610092,
+            -0.008212542161345482,
+            0.009641164913773537,
+            0.006375740747898817,
+            -0.029290854930877686,
+            0.0041348435916006565,
+            0.02504580281674862,
+            0.04418119043111801,
+            -0.0045797573402523994,
+            -0.02865409664809704,
+            0.0028674507047981024,
+            -0.027788760140538216,
+            -0.013461710885167122,
+            0.005261414684355259,
+            0.016449574381113052,
+            0.024539662525057793,
+            0.007628847379237413,
+            0.03585435822606087,
+            0.007294141221791506,
+            0.006318595726042986,
+            0.044279150664806366,
+            -0.025666235014796257,
+            -0.027380581945180893,
+            0.02519274689257145,
+            0.04045860469341278,
+            0.023151857778429985,
+            -0.018253721296787262,
+            -0.0142454132437706,
+            0.0053063142113387585,
+            0.0009296254720538855,
+            -0.022139575332403183,
+            0.002808264922350645,
+            -0.013902543112635612,
+            0.005143043119460344,
+            0.03892385587096214,
+            -0.001205145614221692,
+            0.014196431264281273,
+            0.007302304729819298,
+            0.006845145486295223,
+            0.004334850702434778,
+            -0.011731036007404327,
+            -0.023592690005898476,
+            0.00032220553839579225,
+            0.034972693771123886,
+            0.015404638834297657,
+            -0.030515389516949654,
+            -0.00995954405516386,
+            0.034711457788944244,
+            -0.00795130804181099,
+            -0.009118696674704552,
+            -0.009771781973540783,
+            0.003385836724191904,
+            0.02638462744653225,
+            -0.01606588624417782,
+            0.006649219896644354,
+            0.03716052696108818,
+            -0.019870106130838394,
+            0.005543057806789875,
+            -0.025405000895261765,
+            0.0018061877926811576,
+            0.02442537248134613,
+            -0.006216551177203655,
+            0.028098974376916885,
+            5.6315813708351925e-05,
+            0.03025415539741516,
+            -0.004677720367908478,
+            0.008432958275079727,
+            -0.020327266305685043,
+            0.03987082839012146,
+            -0.017633290961384773,
+            0.03402572125196457,
+            0.02067013457417488,
+            -0.01258004643023014,
+            0.04783846437931061,
+            -0.036801330745220184,
+            -0.020784424617886543,
+            0.00584919098764658,
+            0.012743317522108555,
+            0.0028919412288814783,
+            -0.0191517136991024,
+            -0.018400665372610092,
+            -0.010996315628290176,
+            0.02380494214594364,
+            -0.022547753527760506,
+            -0.016865916550159454,
+            -0.007441085297614336,
+            -0.01890680566430092,
+            -0.009673818945884705,
+            -0.007988044060766697,
+            -0.01709449477493763,
+            -0.016547536477446556,
+            0.044573038816452026,
+            -0.06334923207759857,
+            0.016996532678604126,
+            0.017159804701805115,
+            0.011477965861558914,
+            -0.034678805619478226,
+            -0.019510909914970398,
+            -0.017633290961384773,
+            -0.002347023691982031,
+            -0.030743969604372978,
+            0.001702102366834879,
+            0.01725776679813862,
+            0.01138000376522541,
+            -0.05022222548723221,
+            0.012335140258073807,
+            0.02772345207631588,
+            -0.012065742164850235,
+            0.0014949520118534565,
+            -0.023135529831051826,
+            -0.017176130786538124,
+            0.0031776658724993467,
+            0.022123249247670174,
+            0.005975726060569286,
+            -0.008188051171600819,
+            -0.016865916550159454,
+            -0.005241005681455135,
+            -0.006045116577297449,
+            -0.002359268954023719,
+            0.005094061605632305,
+            0.011780017986893654,
+            0.019690508022904396,
+            -0.0024633544962853193,
+            -0.004436895251274109,
+            -0.006396149750798941,
+            0.027364253997802734,
+            -0.001450052484869957,
+            -0.008416631259024143,
+            0.050907962024211884,
+            0.03556047007441521,
+            0.006094098091125488,
+            -0.01108611561357975,
+            -0.03990348428487778,
+            0.010661610402166843,
+            -0.022711025550961494,
+            0.0401320643723011,
+            0.01905374974012375,
+            -0.008298259228467941,
+            -0.028866350650787354,
+            0.010294250212609768,
+            -0.02290695160627365,
+            -0.009975871071219444,
+            0.009469729848206043,
+            0.00726556871086359,
+            -0.03263791650533676,
+            -0.013543346896767616,
+            0.03510330989956856,
+            0.011437147855758667,
+            -0.006841063499450684,
+            -0.036050282418727875,
+            0.008588065393269062,
+            0.01646590232849121,
+            0.011869817040860653,
+            -0.019233347848057747,
+            0.004449140280485153,
+            -0.012041252106428146,
+            0.017470018938183784,
+            0.008849299512803555,
+            -0.017796561121940613,
+            0.018237393349409103,
+            0.004710374400019646,
+            -0.008490103296935558,
+            -0.003912386484444141,
+            -0.0163107942789793,
+            -0.02006603218615055,
+            0.03840138763189316,
+            0.024490682408213615,
+            0.026335645467042923,
+            0.005457340274006128,
+            0.02008235827088356,
+            0.03227871656417847,
+            0.011282040737569332,
+            -0.03134807199239731,
+            0.027935704216361046,
+            0.016996532678604126,
+            -0.04604247957468033,
+            0.021551799029111862,
+            -0.02218855731189251,
+            -0.028670424595475197,
+            0.009649328887462616,
+            -0.0017367975087836385,
+            0.011192241683602333,
+            0.0034837995190173388,
+            -0.003642988856881857,
+            -0.03307874873280525,
+            -0.006236960180103779,
+            0.015771998092532158,
+            0.015502601861953735,
+            -0.0048695639707148075,
+            -0.003296037670224905,
+            -0.011886144056916237,
+            0.00487364549189806,
+            0.009796272963285446,
+            0.039054471999406815,
+            0.001381682581268251,
+            -0.004751192405819893,
+            -0.0018765984568744898,
+            0.031854212284088135,
+            0.0271193478256464,
+            -0.017274094745516777,
+            -0.026286665350198746,
+            0.00580837344750762,
+            -0.014253576286137104,
+            0.026156047359108925,
+            -0.00487364549189806,
+            0.011877980083227158,
+            0.008718682453036308,
+            -0.0007908449042588472,
+            -0.001373519073240459,
+            0.025584599003195763,
+            -0.02710302174091339,
+            -0.021764053031802177,
+            0.031233781948685646,
+            -0.025731543079018593,
+            0.028735732659697533,
+            0.013061696663498878,
+            -0.01557607389986515,
+            -0.015225040726363659,
+            -0.010302413254976273,
+            0.009943217039108276,
+            -0.012277995236217976,
+            0.023853924125432968,
+            0.003608293831348419,
+            0.006392067763954401,
+            0.0003584313380997628,
+            0.02770712412893772,
+            0.024833550676703453,
+            0.007820690982043743,
+            -0.018074123188853264,
+            -0.0016816934803500772,
+            -0.02320083975791931,
+            -0.01680060662329197,
+            0.023772288113832474,
+            -0.018090449273586273,
+            -0.03268689662218094,
+            -0.028556134551763535,
+            0.012114724144339561,
+            0.007392104249447584,
+            0.010816717520356178,
+            0.03461349755525589,
+            -0.030384771525859833,
+            -0.0344502255320549,
+            0.01766594499349594,
+            0.026270337402820587,
+            -0.005551221314817667,
+            -0.021535472944378853,
+            0.03085825778543949,
+            -0.005828781984746456,
+            0.017600636929273605,
+            0.0404912605881691,
+            -0.0016225076979026198,
+            0.010228941217064857,
+            -0.042026009410619736,
+            -0.014057651162147522,
+            -0.0066614653915166855,
+            0.006261450704187155,
+            0.034417569637298584,
+            0.01048201136291027,
+            0.01535565685480833,
+            0.03448288142681122,
+            -0.021192602813243866,
+            -0.0002903166168835014,
+            -0.0022449791431427,
+            -0.02818061038851738,
+            0.03347059711813927,
+            0.025715215131640434,
+            0.011918798089027405,
+            0.019706835970282555,
+            -0.03794422745704651,
+            -0.013037205673754215,
+            0.027070365846157074,
+            0.015053605660796165,
+            0.01273515447974205,
+            0.016694480553269386,
+            0.011249386705458164,
+            0.00868602842092514,
+            -0.021633435040712357,
+            -0.05169166624546051,
+            0.01214737817645073,
+            0.01993541419506073,
+            -0.016449574381113052,
+            0.01057181041687727,
+            0.024980494752526283,
+            0.007620683871209621,
+            -0.007045152597129345,
+            0.012808626517653465,
+            -0.02682545967400074,
+            0.002259265398606658,
+            0.0019010890973731875,
+            -0.014482156373560429,
+            0.038270771503448486,
+            -0.016367938369512558,
+            0.011363675817847252,
+            -0.018204739317297935,
+            0.023870250210165977,
+            0.00011531029304023832,
+            0.0007964573451317847,
+            0.014963806606829166,
+            -0.011347348801791668,
+            -0.04235255345702171,
+            0.0021613026037812233,
+            0.03342161700129509,
+            -0.015200549736618996,
+            0.026449935510754585,
+            -0.0020674217958003283,
+            0.032458316534757614,
+            0.042777057737112045,
+            0.017731253057718277,
+            0.0003497575526125729,
+            0.025682561099529266,
+            0.0036144163459539413,
+            0.0073880222626030445,
+            0.012400448322296143,
+            0.009322785772383213,
+            0.007245160173624754,
+            0.030809277668595314,
+            -0.010261595249176025,
+            0.02173139713704586,
+            -0.04760988429188728,
+            -0.014629100449383259,
+            0.012024925090372562,
+            -0.0013316808035597205,
+            -0.012449429370462894,
+            -0.033029764890670776,
+            0.012563719414174557,
+            -0.03503800183534622,
+            0.022074267268180847,
+            -0.0004959363141097128,
+            0.04333217814564705,
+            -0.02187834307551384,
+            -0.005518566817045212,
+            0.016694480553269386,
+            0.0016510800924152136,
+            0.023282473906874657,
+            -0.0012153501156717539,
+            -0.016416920349001884,
+            -0.0067798369564116,
+            -0.009257477708160877,
+            0.004138925112783909,
+            0.04375668615102768,
+            -0.003722583642229438,
+            -0.0001682458823779598,
+            0.01409030519425869,
+            -0.011959616094827652,
+            -0.03250729665160179,
+            0.045193471014499664,
+            -0.0030837850645184517,
+            0.02484987862408161,
+            0.04979771748185158,
+            -0.06410028040409088,
+            -0.019543563947081566,
+            0.013061696663498878,
+            -0.010441194288432598,
+            0.011363675817847252,
+            0.005547139327973127,
+            0.016114868223667145,
+            0.017894525080919266,
+            -0.005485912784934044,
+            0.01619650423526764,
+            0.005106307100504637,
+            0.028098974376916885,
+            -0.02336410991847515,
+            -0.03296445682644844,
+            -0.02636830136179924,
+            -0.017747581005096436,
+            0.0014480114914476871,
+            -0.015796490013599396,
+            -3.563585551091819e-06,
+            -0.0022204883862286806,
+            -0.04388730227947235,
+            -0.025274382904171944,
+            0.040687184780836105,
+            0.006706364918500185,
+            0.02458864450454712,
+            0.0077268099412322044,
+            0.03418898954987526,
+            0.005885927006602287,
+            -0.009037061594426632,
+            0.030058229342103004,
+            -0.020049704238772392,
+            0.03833607956767082,
+            0.022106921300292015,
+            -0.002832755446434021,
+            0.005865518003702164,
+            0.031086837872862816,
+            -0.033797141164541245,
+            0.008086007088422775,
+            -0.034384917467832565,
+            0.0046695563942193985,
+            -0.008620720356702805,
+            -0.015094423666596413,
+            -0.009526874870061874,
+            -0.02504580281674862,
+            0.011216731742024422,
+            -0.021535472944378853,
+            0.01590261608362198,
+            0.010416703298687935,
+            -0.002916431985795498,
+            0.03448288142681122,
+            -0.014204595237970352,
+            -0.0027796924114227295,
+            0.0024531499948352575,
+            -0.027070365846157074,
+            -0.006343086715787649,
+            -0.003377673216164112,
+            -0.03689929470419884,
+            -0.004502203781157732,
+            0.010988152585923672,
+            -0.00024184548237826675,
+            0.016898570582270622,
+            0.02787039615213871,
+            0.0005092020728625357,
+            -0.022302847355604172,
+            0.002340900944545865,
+            -0.04199335351586342,
+            0.0011877981014549732,
+            0.01483318954706192,
+            -0.017600636929273605,
+            -0.008612556383013725,
+            0.033046092838048935,
+            0.023298801854252815,
+            -0.009396257810294628,
+            -0.035005345940589905,
+            0.017796561121940613,
+            -0.011853490024805069,
+            0.027511198073625565,
+            -0.0002630197268445045,
+            0.019298657774925232,
+            -0.0076941559091210365,
+            0.04493223875761032,
+            0.001369437319226563,
+            0.008645210415124893,
+            -0.021959977224469185,
+            0.007020662073045969,
+            -0.020278284326195717,
+            -0.002406209474429488,
+            -0.006171651650220156,
+            -0.012751481495797634,
+            -0.010669773444533348,
+            0.0032735876739025116,
+            0.007730891928076744,
+            -0.0176822729408741,
+            0.0055961208418011665,
+            -0.015959760174155235,
+            -0.02741323597729206,
+            0.007543129846453667,
+            -0.05965929850935936,
+            -0.0179598331451416,
+            -0.03373183310031891,
+            -0.009477893821895123,
+            0.02008235827088356,
+            0.03510330989956856,
+            -0.012759645469486713,
+            0.00497977202758193,
+            -0.014898497611284256,
+            0.004477712791413069,
+            0.03490738570690155,
+            0.044442422688007355,
+            -0.014759717509150505,
+            0.003381754970178008,
+            -0.010955498553812504,
+            0.033029764890670776,
+            -0.03347059711813927,
+            0.0030062312725931406,
+            0.031086837872862816,
+            -0.005057325586676598,
+            0.007959471084177494,
+            0.011624909937381744,
+            0.020474210381507874,
+            -0.019396619871258736,
+            0.00877582747489214,
+            0.002100075827911496,
+            0.019233347848057747,
+            0.02171507105231285,
+            0.020343592390418053,
+            -0.03539719805121422,
+            0.028409190475940704,
+            0.018384337425231934,
+            0.02444170042872429,
+            0.044442422688007355,
+            -0.03974021226167679,
+            -0.004147088620811701,
+            -0.0032225653994828463,
+            -0.02997659333050251,
+            0.02008235827088356,
+            -0.007530884351581335,
+            -0.01841699331998825,
+            -0.007069643586874008,
+            0.03327467292547226,
+            0.014571955427527428,
+            -0.05051611363887787,
+            0.01980479806661606,
+            0.01740471087396145,
+            0.02055584453046322,
+            0.024474354460835457,
+            -0.01123305968940258,
+            -0.0019072118448093534,
+            0.008996243588626385,
+            0.006804327480494976,
+            0.019266001880168915,
+            0.015257694758474827,
+            0.002012317767366767,
+            -0.0022082431241869926,
+            -0.011837163008749485,
+            0.005224678665399551,
+            -0.009077879600226879,
+            0.010808554477989674,
+            -0.02458864450454712,
+            -0.009698309935629368,
+            0.021388528868556023,
+            0.0031164393294602633,
+            -0.013665799982845783,
+            -0.03729114308953285,
+            -0.045944519340991974,
+            0.0033837957307696342,
+            0.004543021321296692,
+            -0.05365091934800148,
+            0.005322641227394342,
+            -0.026547899469733238,
+            -0.002108239568769932,
+            -0.005502239800989628,
+            0.0019214981002733111,
+            0.005636938381940126,
+            0.0167679525911808,
+            -0.013494365848600864,
+            0.02248244546353817,
+            -0.015502601861953735,
+            0.03165828809142113,
+            0.0055961208418011665,
+            0.007375776767730713,
+            -0.0027796924114227295,
+            0.05590406060218811,
+            -0.003712379140779376,
+            -0.03699725493788719,
+            0.023282473906874657,
+            -0.01311067771166563,
+            -0.0001747512142173946,
+            0.030090883374214172,
+            0.00884113647043705,
+            -0.015257694758474827,
+            0.013437220826745033,
+            -0.030613351613283157,
+            -0.03249097242951393,
+            0.010996315628290176,
+            -0.013755599036812782,
+            -0.0015776080545037985,
+            -0.04568328335881233,
+            -0.009339112788438797,
+            0.004269542172551155,
+            -0.018335357308387756,
+            0.010775899514555931,
+            0.01631895825266838,
+            0.013992343097925186,
+            -0.024245774373412132,
+            0.009053388610482216,
+            0.006865554489195347,
+            0.013453547842800617,
+            0.0007331897504627705,
+            0.014049487188458443,
+            -0.008677864447236061,
+            -0.011306531727313995,
+            -0.018319029361009598,
+            0.0032450153958052397,
+            -0.026792805641889572,
+            0.005155288614332676,
+            -0.003155216109007597,
+            -0.03771565109491348,
+            -0.06521052122116089,
+            -0.0344502255320549,
+            0.016294466331601143,
+            0.03396041318774223,
+            -0.009045224636793137,
+            0.0035144127905368805,
+            -0.00023648813657928258,
+            -0.01093917153775692,
+            0.024523336440324783,
+            -0.005930826533585787,
+            -0.0004829256213270128,
+            0.0035389035474509,
+            0.02050686441361904,
+            -0.009722800925374031,
+            0.005845109466463327,
+            -0.051952898502349854,
+            -0.01586179807782173,
+            0.01812310516834259,
+            0.006306350696831942,
+            0.014629100449383259,
+            -0.009690145961940289,
+            0.01693122461438179,
+            -0.030074557289481163,
+            -0.028147956356406212,
+            0.04130761697888374,
+            0.008277850225567818,
+            0.011486129835247993,
+            -0.003055212553590536,
+            -0.01680060662329197,
+            0.02504580281674862,
+            -0.0020633398089557886,
+            -0.006461458280682564,
+            -0.00817988719791174,
+            0.015820980072021484,
+            0.044279150664806366,
+            0.003255219664424658,
+            0.0009821783751249313,
+            0.017045514658093452,
+            -0.002663361607119441,
+            -0.011575928889214993,
+            0.007677828427404165,
+            0.013592327944934368,
+            -0.016384266316890717,
+            0.010375885292887688,
+            -0.025862159207463264,
+            0.0332583449780941,
+            0.03131541982293129,
+            0.011575928889214993,
+            -0.015045441687107086,
+            0.006130834110081196,
+            0.005330805201083422,
+            0.008086007088422775,
+            -0.001369437319226563,
+            -0.0017133272485807538,
+            0.020719116553664207,
+            -0.03276853263378143,
+            -0.01693122461438179,
+            -0.005347132217139006,
+            0.003932795487344265,
+            0.005277741700410843,
+            0.03396041318774223,
+            -0.0033266509417444468,
+            0.017927179113030434,
+            -0.011657563969492912,
+            0.010579974390566349,
+            0.021764053031802177,
+            0.037095218896865845,
+            -0.024996822699904442,
+            -0.029862305149435997,
+            0.012914752587676048,
+            -0.001296985661610961,
+            0.014065815135836601,
+            0.0010173837654292583,
+            0.014139287173748016,
+            0.0029409227427095175,
+            0.060475654900074005,
+            -0.017763907089829445,
+            0.004628738854080439,
+            0.0004035860183648765,
+            -0.01858026348054409,
+            -0.0013531101867556572,
+            0.004261378664523363,
+            -0.010979988612234592,
+            0.03608293831348419,
+            0.002344982698559761,
+            0.039185091853141785,
+            0.013918871060013771,
+            0.009290131740272045,
+            0.0015143405180424452,
+            -0.013592327944934368,
+            0.008355404250323772,
+            0.0015592400450259447,
+            -0.001132694073021412,
+            0.009461566805839539,
+            0.019298657774925232,
+            0.009714636951684952,
+            0.025274382904171944,
+            0.009788108989596367,
+            0.012857607565820217,
+            -0.029470453038811684,
+            0.03510330989956856,
+            -0.0006224715034477413,
+            0.012996388599276543,
+            -0.02040890045464039,
+            -0.003136848099529743,
+            -0.0008240094175562263,
+            0.015649545937776566,
+            -0.015616890974342823,
+            -0.001201063860207796,
+            -0.01297189760953188,
+            0.040099408477544785,
+            -0.018988441675901413,
+            0.013976015150547028,
+            0.009673818945884705,
+            -0.02865409664809704,
+            -0.007596192881464958,
+            0.01995174214243889,
+            0.014294394291937351,
+            -0.0053144777193665504,
+            -0.020490536466240883,
+            0.005832863971590996,
+            0.015380147844552994,
+            -0.028572462499141693,
+            1.3584674888988957e-05,
+            0.014751553535461426,
+            -0.0164903923869133,
+            -0.00025409081717953086,
+            -0.004702210891991854,
+            0.003975654020905495,
+            -0.009265640750527382,
+            0.007028825581073761,
+            0.016057724133133888,
+            -0.024507008492946625,
+            -0.020049704238772392,
+            0.005722655914723873,
+            0.006571666337549686,
+            0.01444950234144926,
+            0.007739055436104536,
+            0.004355259705334902,
+            -0.020033378154039383,
+            -0.01799248717725277,
+            0.0032674651592969894,
+            -0.01649855636060238,
+            -0.0048614004626870155,
+            0.0020531355403363705,
+            -0.019363965839147568,
+            0.01875986158847809,
+            -0.010743245482444763,
+            0.00017538899555802345,
+            -0.003226647386327386,
+            -0.020800752565264702,
+            0.006204306147992611,
+            -0.004865481983870268,
+            0.019870106130838394,
+            0.003555230563506484,
+            0.02785406820476055,
+            0.02310287579894066,
+            -0.004428731743246317,
+            0.017600636929273605,
+            -0.016155686229467392,
+            0.007983962073922157,
+            -0.031135819852352142,
+            -0.001782717532478273,
+            -0.010408539324998856,
+            0.0036103345919400454,
+            -0.015943434089422226,
+            -0.009126860648393631,
+            -0.0048777274787425995,
+            -0.0006678812787868083,
+            -0.004820582456886768,
+            0.0004275664687156677,
+            0.0027817331720143557,
+            0.02231917530298233,
+            -0.013208640739321709,
+            -0.02803366631269455,
+            -0.017437364906072617,
+            0.006045116577297449,
+            0.019396619871258736,
+            -0.026025431230664253,
+            -0.009184005670249462,
+            -0.005012426059693098,
+            -0.004326687194406986,
+            -0.021404854953289032,
+            -0.015167895704507828,
+            -0.005632856860756874,
+            -0.008045189082622528,
+            -0.010522829368710518,
+            0.0046532293781638145,
+            -0.0035715578123927116,
+            -0.011739199981093407,
+            0.021666089072823524,
+            0.018057795241475105,
+            -0.019837452098727226,
+            -0.019837452098727226,
+            0.013641309924423695,
+            -0.006159406621009111,
+            0.0018878233386203647,
+            0.02008235827088356,
+            0.00029286774224601686,
+            0.005824700463563204,
+            0.020882388576865196,
+            -0.0035225762985646725,
+            -0.01797616109251976,
+            0.003577680326998234,
+            -0.0018888438353314996,
+            -0.012122887186706066,
+            0.003212361130863428,
+            0.02773977816104889,
+            0.0008076822850853205,
+            0.023429417982697487,
+            -0.03056436963379383,
+            -0.011412657797336578,
+            0.006061443593353033,
+            0.02369065210223198,
+            0.00211844383738935,
+            -0.0021694661118090153,
+            0.020131340250372887,
+            0.03941367194056511,
+            0.005775718949735165,
+            0.0015755671774968505,
+            -0.04003410041332245,
+            0.01027792226523161,
+            0.009208496659994125,
+            0.024621298536658287,
+            0.008339077234268188,
+            0.0014653591206297278,
+            -0.012906589545309544,
+            -0.009755454957485199,
+            -0.0017939424142241478,
+            0.014702572487294674,
+            -0.023870250210165977,
+            0.0024755997583270073,
+            -0.03157665207982063,
+            0.005094061605632305,
+            0.0021959978621453047,
+            -0.009559528902173042,
+            0.0335359051823616,
+            0.00980443600565195,
+            -0.02065380848944187,
+            0.015151568688452244,
+            0.005767555441707373,
+            0.007330877240747213,
+            0.022449791431427002,
+            -0.008228869177401066,
+            0.030450081452727318,
+            -0.005049162078648806,
+            0.01213921420276165,
+            0.02742956392467022,
+            0.032588932663202286,
+            -0.013102514669299126,
+            -0.003959327004849911,
+            0.0006097159348428249,
+            -0.0002561317232903093,
+            0.0007995187188498676,
+            -0.008204378187656403,
+            0.0037389106582850218,
+            -0.014800535514950752,
+            0.012710663489997387,
+            -0.019543563947081566,
+            -0.0034286954905837774,
+            0.0038532004691660404,
+            0.01991908811032772,
+            0.0016265894519165158,
+            -0.033633869141340256,
+            0.02055584453046322,
+            -0.015420965850353241,
+            0.0135596739128232,
+            0.001290863030590117,
+            -0.011110605672001839,
+            -0.00029210239881649613,
+            0.020588500425219536,
+            0.010947334580123425,
+            -0.007845181971788406,
+            -0.0135596739128232,
+            -0.028376536443829536,
+            -0.021535472944378853,
+            0.00823703221976757,
+            0.009445239789783955,
+            -0.0028837777208536863,
+            0.0021306893322616816,
+            -0.011869817040860653,
+            0.0006520643946714699,
+            -0.0008122742874547839,
+            0.01740471087396145,
+            -0.0335359051823616,
+            0.021943651139736176,
+            -0.019527235999703407,
+            -0.019543563947081566,
+            0.017045514658093452,
+            0.008049271069467068,
+            0.009445239789783955,
+            0.0033552234526723623,
+            -0.01995174214243889,
+            -0.012326976284384727,
+            0.0338951013982296,
+            -0.01138000376522541,
+            0.006298186723142862,
+            0.004408322740346193,
+            0.012914752587676048,
+            -0.008914608508348465,
+            -0.011926962062716484,
+            0.01784554310142994,
+            -0.009861581027507782,
+            0.013951525092124939,
+            0.0010010566329583526,
+            -0.0254703089594841,
+            0.02739690989255905,
+            -0.019233347848057747,
+            0.03487472981214523,
+            -0.021617108955979347,
+            0.00711862463504076,
+            0.0066614653915166855,
+            0.008792154490947723,
+            -0.006690037902444601,
+            -0.013224967755377293,
+            0.0012786176521331072,
+            0.03807484731078148,
+            -0.02623768337070942,
+            0.024180466309189796,
+            0.022400809451937675,
+            -0.00040766780148260295,
+            0.02065380848944187,
+            -0.008367649279534817,
+            -0.026711169630289078,
+            0.0179598331451416,
+            -0.012596373446285725,
+            -0.026123393326997757,
+            0.013257622718811035,
+            -0.022678371518850327,
+            -0.020278284326195717,
+            -0.026302991434931755,
+            0.004947117529809475,
+            -0.0025694805663079023,
+            -0.005151206627488136,
+            0.009004407562315464,
+            0.02561725303530693,
+            0.020915042608976364,
+            0.019576217979192734,
+            0.02111096680164337,
+            0.01161674689501524,
+            0.022449791431427002,
+            -0.007559456862509251,
+            0.04401791840791702,
+            -0.004947117529809475,
+            0.004632820375263691,
+            -0.0015980169409886003,
+            -0.015657709911465645,
+            0.009771781973540783,
+            0.0119106350466609,
+            0.0029572497587651014,
+            -0.029176564887166023,
+            -0.009061552584171295,
+            -0.03660540655255318,
+            0.005053244065493345,
+            0.0020664012990891933,
+            0.0176822729408741,
+            0.032115448266267776,
+            -0.005865518003702164,
+            0.004592002835124731,
+            -0.005102225113660097,
+            -0.013927034102380276,
+            -0.009420748800039291,
+            -0.02739690989255905,
+            -0.025437654927372932,
+            -0.007653337903320789,
+            -0.013739272020757198,
+            -0.030368445441126823,
+            -0.011722872965037823,
+            0.01551892887800932,
+            -0.007645174395292997,
+            0.011918798089027405,
+            -0.016702644526958466,
+            0.01385356206446886,
+            -0.004110352601855993,
+            -0.004763437435030937,
+            -0.004530776292085648,
+            -0.017600636929273605,
+            -0.006420640274882317,
+            0.014367866329848766,
+            -0.009477893821895123,
+            -0.01237595733255148,
+            -0.02215590327978134,
+            -0.0015459742862731218,
+            -0.01574750803411007,
+            0.003489922033622861,
+            -0.014596446417272091,
+            -0.006771673448383808,
+            -0.003908304497599602,
+            -0.018514955416321754,
+            0.01766594499349594,
+            0.014816862531006336,
+            -0.019380291923880577,
+            -0.0090697156265378,
+            0.02638462744653225,
+            0.042777057737112045,
+            -0.03218075633049011,
+            -0.008200296200811863,
+            0.0020378287881612778,
+            0.018090449273586273,
+            0.010171796195209026,
+            0.016702644526958466,
+            -0.024653952568769455,
+            0.0022286521270871162,
+            -0.021649762988090515,
+            -0.001220452249981463,
+            -0.023739634081721306,
+            0.010775899514555931,
+            0.025715215131640434,
+            -0.010971825569868088,
+            0.025976449251174927,
+            0.01634344831109047,
+            -0.01812310516834259,
+            0.012016761116683483,
+            0.007183933164924383,
+            0.014645427465438843,
+            -0.019755816087126732,
+            0.0167679525911808,
+            -0.015796490013599396,
+            0.031233781948685646,
+            0.00892277155071497,
+            -0.04829562455415726,
+            -0.03503800183534622,
+            -0.013388238847255707,
+            0.0013388239312916994,
+            -0.010702427476644516,
+            0.015037278644740582,
+            0.015967924147844315,
+            0.013959688134491444,
+            -0.019380291923880577,
+            -0.015820980072021484,
+            0.011984107084572315,
+            0.010065670125186443,
+            0.027462217956781387,
+            -0.006649219896644354,
+            -0.004212397150695324,
+            -0.00960034690797329,
+            -0.02249877341091633,
+            -0.016098542138934135,
+            0.0209803506731987,
+            -0.018041469156742096,
+            0.015935270115733147,
+            -0.026139721274375916,
+            -0.012449429370462894,
+            0.017453692853450775,
+            -0.05211617052555084,
+            -0.01453113742172718,
+            0.0007612520130351186,
+            0.03012353740632534,
+            0.004751192405819893,
+            0.0035184945445507765,
+            0.01841699331998825,
+            -0.01993541419506073,
+            -0.0039776950143277645,
+            -0.02099667675793171,
+            0.006832899991422892,
+            -0.011853490024805069,
+            -0.013984179124236107,
+            0.014482156373560429,
+            -0.013845399022102356,
+            -0.01601690612733364,
+            0.0034368589986115694,
+            0.007183933164924383,
+            -0.021094640716910362,
+            0.011951452121138573,
+            0.003945040516555309,
+            0.012171869166195393,
+            0.012171869166195393,
+            0.01199227012693882,
+            0.021551799029111862,
+            0.02295593172311783,
+            -0.007334959227591753,
+            -0.004767519421875477,
+            -0.01004117913544178,
+            -0.0015398516552522779,
+            0.008898280560970306,
+            0.0033797139767557383,
+            -0.002989904023706913,
+            0.007837017998099327,
+            0.010033016093075275,
+            -0.01498013362288475,
+            0.033193036913871765,
+            -0.0035878848284482956,
+            -0.04134026914834976,
+            -0.003238892648369074,
+            -0.03800953924655914,
+            0.020033378154039383,
+            -0.023298801854252815,
+            -0.016996532678604126,
+            0.0032776694279164076,
+            -0.00629410520195961,
+            -0.019902760162949562,
+            0.013608654960989952,
+            0.011404493823647499,
+            0.00258580781519413,
+            0.02021297626197338,
+            0.012269831262528896,
+            -0.005134879611432552,
+            0.026172375306487083,
+            -0.014718899503350258,
+            -0.008751336485147476,
+            0.015567909926176071,
+            0.006853308994323015,
+            0.0022347746416926384,
+            0.0011031010653823614,
+            -0.018825169652700424,
+            0.005098143592476845,
+            0.0164903923869133,
+            0.007216587662696838,
+            -0.000566347036510706,
+            0.014049487188458443,
+            0.012514738366007805,
+            -0.015135240741074085,
+            -0.0002213090192526579,
+            -0.026564225554466248,
+            0.0032817514147609472,
+            0.004567512311041355,
+            0.007506393827497959,
+            -0.006771673448383808,
+            0.006281859707087278,
+            0.00658799335360527,
+            0.014759717509150505,
+            -0.010155469179153442,
+            -0.016980206593871117,
+            -0.002510294783860445,
+            0.004285869188606739,
+            0.016694480553269386,
+            0.03157665207982063,
+            -0.010310577228665352,
+            -0.005734901409596205,
+            -0.0386626236140728,
+            0.02070278860628605,
+            0.03332365304231644,
+            0.011175914667546749,
+            -0.028213264420628548,
+            0.012302485294640064,
+            0.0006515541463159025,
+            0.0019276207312941551,
+            -0.026302991434931755,
+            0.02787039615213871,
+            0.00974729098379612,
+            0.0066614653915166855,
+            0.017861871048808098,
+            0.0022306928876787424,
+            0.0029266364872455597,
+            -0.016302630305290222,
+            0.01018812321126461,
+            0.009086042642593384,
+            -0.017306748777627945,
+            0.0009745250572450459,
+            0.02324981987476349,
+            -0.01423724927008152,
+            -0.003208279376849532,
+            0.0018133309204131365,
+            0.02651524543762207,
+            -0.013820908032357693,
+            -0.014294394291937351,
+            0.0005714492290280759,
+            0.01751900091767311,
+            -0.0007979880319908261,
+            0.012098397128283978,
+            0.04898136481642723,
+            -0.012424939312040806,
+            0.019657853990793228,
+            0.005461421795189381,
+            -0.015290348790585995,
+            -0.032605260610580444,
+            -0.004922627005726099,
+            -0.01830270327627659,
+            0.011200404725968838,
+            0.014343376271426678,
+            -0.015763835981488228,
+            0.0007525782566517591,
+            -0.03144603595137596,
+            0.007743136957287788,
+            0.014735226519405842,
+            0.018776189535856247,
+            -0.002440904499962926,
+            0.006134915631264448,
+            0.027674470096826553,
+            0.02458864450454712,
+            -0.007392104249447584,
+            0.013804581016302109,
+            0.02320083975791931,
+            -0.025062130764126778,
+            -0.003069498809054494,
+            0.006820654962211847,
+            0.004104230087250471,
+            0.0053063142113387585,
+            -0.016963878646492958,
+            0.03915243595838547,
+            -0.005922663025557995,
+            -0.01506176870316267,
+            0.0030266400426626205,
+            -0.009028897620737553,
+            0.0063390047289431095,
+            -0.012563719414174557,
+            0.012637191452085972,
+            -0.008114579133689404,
+            0.00022883480414748192,
+            -0.0059430720284581184,
+            -0.011110605672001839,
+            -0.012653518468141556,
+            -0.0007949267164804041,
+            -0.016531210392713547,
+            0.007432921789586544,
+            0.00981259997934103,
+            0.005045080557465553,
+            0.01025343220680952,
+            -0.0036797248758375645,
+            -0.003136848099529743,
+            -0.016751626506447792,
+            -0.009984034113585949,
+            0.0073716952465474606,
+            0.01872720755636692,
+            -0.028588788583874702,
+            -0.018547609448432922,
+            0.009290131740272045,
+            0.0022898786701261997,
+            0.018776189535856247,
+            0.0158944521099329,
+            -0.0002595757250674069,
+            0.012318813242018223,
+            -0.01964152604341507,
+            -0.009184005670249462,
+            0.009690145961940289,
+            -0.003569516818970442,
+            0.01048201136291027,
+            -0.014759717509150505,
+            0.010033016093075275,
+            -0.011159587651491165,
+            -0.011053460650146008,
+            0.01385356206446886,
+            0.01738838478922844,
+            0.008873790502548218,
+            -0.001294944784604013,
+            3.383413059054874e-05,
+            -0.010506502352654934,
+            -0.024033522233366966,
+            0.004771600943058729,
+            0.027021385729312897,
+            -0.013273949734866619,
+            0.003883813973516226,
+            0.008702355436980724,
+            -0.017502672970294952,
+            -0.019821124151349068,
+            -0.012653518468141556,
+            -0.022106921300292015,
+            0.020572172477841377,
+            0.023413091897964478,
+            -0.02026195637881756,
+            0.00877582747489214,
+            -0.004620575346052647,
+            0.014049487188458443,
+            -0.01400050614029169,
+            0.010612628422677517,
+            0.0317889042198658,
+            -0.020000724121928215,
+            -0.0028654097113758326,
+            0.0036633978597819805,
+            -0.015347493812441826,
+            0.018629245460033417,
+            0.03268689662218094,
+            -0.00176945177372545,
+            0.014041324146091938,
+            0.0008138049161061645,
+            -0.009118696674704552,
+            0.01905374974012375,
+            0.011094278655946255,
+            0.005347132217139006,
+            -0.0042491331696510315,
+            -0.012212686240673065,
+            -0.03010721132159233,
+            0.016702644526958466,
+            0.0129637336358428,
+            -0.0037674831692129374,
+            -0.0011551438365131617,
+            -0.006343086715787649,
+            -0.015404638834297657,
+            0.0035123717971146107,
+            -0.013078023679554462,
+            -0.00803294312208891,
+            -0.008759500458836555,
+            0.004314441699534655,
+            -0.0034960447810590267,
+            -0.009053388610482216,
+            0.007236996199935675,
+            0.0009510547970421612,
+            -0.0010602424154058099,
+            -0.01902109570801258,
+            -0.025078458711504936,
+            -0.00995954405516386,
+            -0.016849588602781296,
+            0.0038287099450826645,
+            0.0005234883283264935,
+            -0.027184655889868736,
+            0.0053961132653057575,
+            -0.008139070123434067,
+            -0.008134988136589527,
+            0.007126788143068552,
+            0.006881881505250931,
+            -0.007437003776431084,
+            -0.010816717520356178,
+            -0.00786150898784399,
+            -0.006865554489195347,
+            0.020033378154039383,
+            0.0034348180051892996,
+            0.012082069180905819,
+            -0.010498338378965855,
+            -0.018025141209363937,
+            -0.016963878646492958,
+            0.016718972474336624,
+            -0.008971752598881721,
+            -0.02083340659737587,
+            -0.01444950234144926,
+            -0.008424794301390648,
+            -0.007665583398193121,
+            -0.01527402177453041,
+            -0.021159948781132698,
+            -0.0032103201374411583,
+            -0.011347348801791668,
+            -0.0067431009374558926,
+            -0.0028960229828953743,
+            0.0013571919407695532,
+            0.00951871182769537,
+            0.00546958576887846,
+            0.00438791373744607,
+            0.021666089072823524,
+            0.010122815147042274,
+            -0.005461421795189381,
+            0.016441410407423973,
+            -0.006183897145092487,
+            -0.004440976772457361,
+            -0.01498013362288475,
+            -0.009918726049363613,
+            0.018547609448432922,
+            -0.0055226488038897514,
+            0.008457448333501816,
+            -0.0009536059224046767,
+            -0.00576347392052412,
+            0.015674035996198654,
+            0.0129637336358428,
+            0.016718972474336624,
+            0.004771600943058729,
+            -0.00012168807734269649,
+            0.012416775338351727,
+            -0.0015745466807857156,
+            0.012555556371808052,
+            -0.0038429962005466223,
+            0.014506647363305092,
+            -0.009722800925374031,
+            -0.018988441675901413,
+            0.004157293122261763,
+            0.006265532691031694,
+            -0.012531065382063389,
+            0.009404421783983707,
+            0.007743136957287788,
+            -0.013747435994446278,
+            -0.02261306345462799,
+            -0.000106317616882734,
+            0.00014362763613462448,
+            0.008122743107378483,
+            0.008971752598881721,
+            -0.008073761127889156,
+            0.012890261597931385,
+            0.004616493359208107,
+            -0.007179851643741131,
+            -0.019657853990793228,
+            0.026009103283286095,
+            0.015608727931976318,
+            0.012269831262528896,
+            -0.052508022636175156,
+            -0.0030980713199824095,
+            0.010743245482444763,
+            -0.015951598063111305,
+            -0.026792805641889572,
+            -0.006404313258826733,
+            -0.008759500458836555,
+            -0.04290767386555672,
+            -0.009624837897717953,
+            -0.0005724696675315499,
+            -0.011388166807591915,
+            0.010400376282632351,
+            0.01573934406042099,
+            0.014612773433327675,
+            0.005489994306117296,
+            -0.007232914678752422,
+            -0.017633290961384773,
+            -0.02788672223687172,
+            0.010057506151497364,
+            -0.013380075804889202,
+            -0.006412476766854525,
+            -0.002263347152620554,
+            -0.007690073922276497,
+            0.0016112826997414231,
+            0.01691489666700363,
+            0.042483169585466385,
+            0.023037567734718323,
+            -0.012335140258073807,
+            -0.020131340250372887,
+            -0.012800462543964386,
+            -0.009192168712615967,
+            0.00633492274209857,
+            0.011592255905270576,
+            -0.004738946910947561,
+            0.0096656559035182,
+            -0.004294032696634531,
+            0.012302485294640064,
+            0.012767808511853218,
+            0.01680060662329197,
+            -2.3023791072773747e-05,
+            0.005579793825745583,
+            -0.0024694770108908415,
+            -0.01738838478922844,
+            -0.004379750229418278,
+            0.008261523209512234,
+            -0.0008872769540175796,
+            0.0019317024853080511,
+            0.014204595237970352,
+            0.019102731719613075,
+            -0.009094206616282463,
+            -0.014612773433327675,
+            -0.016098542138934135,
+            -0.005277741700410843,
+            -0.0012622905196622014,
+            0.029927613213658333,
+            0.02037624642252922,
+            -0.019266001880168915,
+            -0.018237393349409103,
+            -0.018351683393120766,
+            0.00922482367604971,
+            -0.003563394071534276,
+            -0.0064369672909379005,
+            0.01905374974012375,
+            0.023021239787340164,
+            0.017910851165652275,
+            -0.003642988856881857,
+            -0.014865843579173088,
+            0.003804219188168645,
+            0.02847449854016304,
+            0.008367649279534817,
+            0.01078406348824501,
+            0.008726846426725388,
+            -0.0011908593587577343,
+            -0.032441988587379456,
+            -0.0113146947696805,
+            0.00951871182769537,
+            -0.008481939323246479,
+            0.026564225554466248,
+            -0.016531210392713547,
+            0.009388094767928123,
+            -0.005077734589576721,
+            0.0019286411115899682,
+            -0.02365799807012081,
+            0.0014929111348465085,
+            0.010122815147042274,
+            -0.024947840720415115,
+            0.00336950970813632,
+            -0.023592690005898476,
+            -0.00013763252354692668,
+            0.008367649279534817,
+            0.0022837561555206776,
+            0.01650671847164631,
+            -0.0113146947696805,
+            0.0051838611252605915,
+            0.02099667675793171,
+            -0.002832755446434021,
+            -0.01512707769870758,
+            -0.0007842120248824358,
+            0.0005092020728625357,
+            -0.001286781276576221,
+            0.02563357912003994,
+            0.015812816098332405,
+            -0.0036756431218236685,
+            0.020882388576865196,
+            0.008677864447236061,
+            0.0019133344758301973,
+            -0.021764053031802177,
+            0.0023796779569238424,
+            -0.032899148762226105,
+            0.017437364906072617,
+            -0.011126932688057423,
+            0.012490247376263142,
+            0.00868602842092514,
+            -0.01341272983700037,
+            -0.0007388022495433688,
+            -0.0107187544927001,
+            -0.0015725058037787676,
+            -0.007934981025755405,
+            0.009779945947229862,
+            -0.004363423213362694,
+            0.008339077234268188,
+            -0.00024579971795901656,
+            -0.01476788055151701,
+            0.018694553524255753,
+            9.678283277025912e-06,
+            -0.00010925139940809458,
+            0.005441013257950544,
+            0.009935053065419197,
+            0.00786559097468853,
+            0.002009256277233362,
+            0.023870250210165977,
+            -0.031119491904973984,
+            -0.016082214191555977,
+            -0.010204450227320194,
+            0.005543057806789875,
+            -0.007922735065221786,
+            -0.018531281501054764,
+            -0.03257260471582413,
+            0.01318415068089962,
+            0.001293924287892878,
+            -0.0009806477464735508,
+            0.01258004643023014,
+            -0.00942891277372837,
+            -0.0013878053287044168,
+            -0.01606588624417782,
+            -0.014808698557317257,
+            0.006416558753699064,
+            0.0033633869607001543,
+            -0.01222901325672865,
+            -0.023739634081721306,
+            0.014759717509150505,
+            0.0007418635650537908,
+            -0.016980206593871117,
+            -0.017502672970294952,
+            0.00018457300029695034,
+            -0.004804255440831184,
+            0.02204161323606968,
+            -0.016678154468536377,
+            0.003373591462150216,
+            0.0007219648687168956,
+            0.009232986718416214,
+            -0.02889900468289852,
+            0.00716352416202426,
+            0.0006107363733462989,
+            0.0035164537839591503,
+            -0.006518602836877108,
+            -0.021013004705309868,
+            0.011371839791536331,
+            -0.0020674217958003283,
+            -0.014751553535461426,
+            -0.020751770585775375,
+            -0.0020796670578420162,
+            -0.007420676294714212,
+            -0.02724996581673622,
+            0.017910851165652275,
+            0.0009985055075958371,
+            -0.01738838478922844,
+            0.008077843114733696,
+            0.006596156861633062,
+            0.009355440735816956,
+            0.02455599047243595,
+            0.010147306136786938,
+            -0.018776189535856247,
+            -0.0008087027235887945,
+            0.006979844067245722,
+            0.009306458756327629,
+            0.0036450298503041267,
+            0.04222193360328674,
+            -0.008130906149744987,
+            -0.006579829845577478,
+            0.0066614653915166855,
+            -0.008449285291135311,
+            -0.013747435994446278,
+            0.026727497577667236,
+            0.005796127952635288,
+            0.015984252095222473,
+            0.016123032197356224,
+            -0.00945340283215046,
+            -0.005008344538509846,
+            -0.0018786393338814378,
+            0.02065380848944187,
+            0.0073798587545752525,
+            -0.005502239800989628,
+            -0.009714636951684952,
+            0.005881845485419035,
+            -0.00898808054625988,
+            0.0021653843577951193,
+            -0.009526874870061874,
+            -0.005824700463563204,
+            -0.008069679141044617,
+            0.016400592401623726,
+            -0.0003523086488712579,
+            0.027315273880958557,
+            0.020751770585775375,
+            -0.008563575334846973,
+            -0.0005683879135176539,
+            0.017208784818649292,
+            -0.005820618476718664,
+            0.015608727931976318,
+            -0.011192241683602333,
+            -0.007706400938332081,
+            0.006408394780009985,
+            0.024017194285988808,
+            0.012237177230417728,
+            -0.0056042843498289585,
+            0.0002885308349505067,
+            -0.0030348035506904125,
+            -0.010612628422677517,
+            0.013388238847255707,
+            0.015323002822697163,
+            -0.01947825588285923,
+            -0.004722619894891977,
+            -0.0016592437168583274,
+            -0.003300119424238801,
+            0.00945340283215046,
+            0.02305389568209648,
+            0.000171817431692034,
+            0.009184005670249462,
+            0.0012765767751261592,
+            0.03862996771931648,
+            -0.0053144777193665504,
+            0.03102152980864048,
+            -0.0010423846542835236,
+            -0.0059430720284581184,
+            -0.009094206616282463,
+            -0.020441556349396706,
+            -0.011486129835247993,
+            -0.00041481092921458185,
+            -0.005040998570621014,
+            -0.0004956811899319291,
+            -0.020327266305685043,
+            0.0026960158720612526,
+            0.014637264423072338,
+            0.007661501411348581,
+            0.017927179113030434,
+            -0.002889900468289852,
+            0.017437364906072617,
+            0.019690508022904396,
+            -0.01400050614029169,
+            -0.029862305149435997,
+            -0.0006515541463159025,
+            0.01498013362288475,
+            -0.01661284640431404,
+            0.019445601850748062,
+            0.004085862077772617,
+            0.0029531680047512054,
+            0.004779764451086521,
+            -0.006049198564141989,
+            0.018482301384210587,
+            0.02039257436990738,
+            0.01453930139541626,
+            -0.013167822733521461,
+            0.014057651162147522,
+            0.009771781973540783,
+            0.006600238848477602,
+            -0.01646590232849121,
+            -0.013535182923078537,
+            0.002555194543674588,
+            0.0009245232213288546,
+            0.024392718449234962,
+            -0.020588500425219536,
+            0.0016602640971541405,
+            0.010588138364255428,
+            0.01769859902560711,
+            -0.004302196204662323,
+            0.008906444534659386,
+            0.0008740111952647567,
+            0.007134951651096344,
+            -0.0035919665824621916,
+            -0.006457376293838024,
+            -0.006673710886389017,
+            -0.010392212308943272,
+            -0.0020959940738976,
+            -0.02070278860628605,
+            0.002198038622736931,
+            -0.002830714685842395,
+            0.018237393349409103,
+            -0.01018812321126461,
+            0.038140155375003815,
+            0.03399306535720825,
+            -0.0007025764207355678,
+            0.011102442629635334,
+            -0.01709449477493763,
+            -0.003945040516555309,
+            -0.021959977224469185,
+            0.01844964735209942,
+            0.016392430290579796,
+            0.012269831262528896,
+            0.005302232690155506,
+            0.017796561121940613,
+            -0.0006301248213276267,
+            0.014490320347249508,
+            0.0029654132667928934,
+            0.007971717044711113,
+            -0.002265388146042824,
+            0.003938918001949787,
+            -0.031103165820240974,
+            -0.017029186710715294,
+            -0.01138000376522541,
+            0.004796091932803392,
+            -0.006375740747898817,
+            -0.0035205355379730463,
+            0.0084737753495574,
+            -0.005902254022657871,
+            0.0004270562494639307,
+            -0.025992777198553085,
+            -0.0157148540019989,
+            -0.010318740271031857,
+            -0.004040962550789118,
+            -0.01169838197529316,
+            -0.0027694879099726677,
+            -0.0022082431241869926,
+            0.0029552089981734753,
+            0.020947696641087532,
+            -0.012808626517653465,
+            -0.016294466331601143,
+            0.004379750229418278,
+            -0.002263347152620554,
+            -0.014016833156347275,
+            -0.016898570582270622,
+            0.006665547378361225,
+            0.012098397128283978,
+            -0.0017470018938183784,
+            -0.013608654960989952,
+            0.011739199981093407,
+            -0.006628811359405518,
+            0.04663025587797165,
+            -0.013584164902567863,
+            -0.0078002819791436195,
+            0.003720542648807168,
+            0.004698128905147314,
+            -0.009396257810294628,
+            0.007330877240747213,
+            0.013020878657698631,
+            0.01830270327627659,
+            0.0095758568495512,
+            -0.008465612307190895,
+            -0.0024388637393712997,
+            0.007796200457960367,
+            -0.016147522255778313,
+            0.006404313258826733,
+            0.0008525818702764809,
+            -0.014400520361959934,
+            0.006110425107181072,
+            0.028980640694499016,
+            0.013151495717465878,
+            0.01288209855556488,
+            -0.022351829335093498,
+            -0.004094025585800409,
+            -0.009355440735816956,
+            -0.004943036008626223,
+            -0.010090161114931107,
+            0.027070365846157074,
+            -0.018155759200453758,
+            0.02651524543762207,
+            0.00898808054625988,
+            0.0017684312770143151,
+            -0.01176369097083807,
+            0.006343086715787649,
+            0.02832755446434021,
+            0.0028592869639396667,
+            -0.020000724121928215,
+            -0.013592327944934368,
+            0.006314514204859734,
+            0.01707816869020462,
+            -0.002647034591063857,
+            -0.005428767763078213,
+            0.019657853990793228,
+            0.01784554310142994,
+            -0.004334850702434778,
+            -0.00936360377818346,
+            0.0008337036124430597,
+            0.00928196869790554,
+            0.016963878646492958,
+            0.015020951628684998,
+            0.009086042642593384,
+            -0.002647034591063857,
+            0.008037025108933449,
+            -0.008800318464636803,
+            -0.016449574381113052,
+            0.007220669183880091,
+            -0.008432958275079727,
+            -0.016963878646492958,
+            0.01872720755636692,
+            0.015257694758474827,
+            0.0181394312530756,
+            0.0024592725094407797,
+            -0.016735298559069633,
+            0.022302847355604172,
+            0.005755309946835041,
+            0.002998067531734705,
+            0.02160078100860119,
+            -0.022221211344003677,
+            0.013012715615332127,
+            0.02037624642252922,
+            0.027609162032604218,
+            -0.01737205684185028,
+            -0.010906516574323177,
+            -0.004857318475842476,
+            -0.006853308994323015,
+            -0.00176945177372545,
+            -0.008253359235823154,
+            -0.012620864436030388,
+            0.0028735732194036245,
+            -0.030466407537460327,
+            0.018106777220964432,
+            -0.005102225113660097,
+            0.010669773444533348,
+            -0.019266001880168915,
+            0.011363675817847252,
+            -0.0035838030744343996,
+            0.011135096661746502,
+            0.0019612954929471016,
+            -0.030041903257369995,
+            -0.008571738377213478,
+            0.015388311818242073,
+            -0.0030368445441126823,
+            -0.013641309924423695,
+            0.02006603218615055,
+            0.00018942011229228228,
+            -0.022270193323493004,
+            -0.004314441699534655,
+            -0.006934944540262222,
+            0.002193956868723035,
+            -0.0029613315127789974,
+            -0.0052981507033109665,
+            -0.030188847333192825,
+            0.020033378154039383,
+            0.009559528902173042,
+            -0.03072764165699482,
+            -0.003255219664424658,
+            -0.014996460638940334,
+            -0.0025184585247188807,
+            0.008881953544914722,
+            -0.004000144544988871,
+            -0.015551582910120487,
+            0.01784554310142994,
+            0.022711025550961494,
+            -0.0023164101876318455,
+            -0.016057724133133888,
+            -0.0090697156265378,
+            -0.027494871988892555,
+            0.030825603753328323,
+            -0.0066696288995444775,
+            0.0022878379095345736,
+            -0.005322641227394342,
+            0.009388094767928123,
+            0.004228724166750908,
+            0.0157148540019989,
+            0.006081852596253157,
+            -0.002647034591063857,
+            0.026596879586577415,
+            -0.01964152604341507,
+            0.009567692875862122,
+            -1.588864870427642e-05,
+            -0.003308282932266593,
+            0.01890680566430092,
+            0.0007939062779769301,
+            0.021649762988090515,
+            -0.008571738377213478,
+            -0.0001475818717153743,
+            -0.0034388997592031956,
+            0.010898353531956673,
+            -0.0028450009413063526,
+            -0.003465431509539485,
+            -0.001531688030809164,
+            0.011118769645690918,
+            0.01013097818940878,
+            0.003804219188168645,
+            -0.01737205684185028,
+            0.003142970846965909,
+            -0.008547247387468815,
+            -0.007028825581073761,
+            0.01302904263138771,
+            -0.011192241683602333,
+            -0.0018990482203662395,
+            -0.03160930797457695,
+            0.016286302357912064,
+            0.0533243790268898,
+            0.008547247387468815,
+            -0.014433175325393677,
+            -0.0018806802108883858,
+            -0.019755816087126732,
+            0.01535565685480833,
+            0.011004479601979256,
+            -0.016816934570670128,
+            -0.011984107084572315,
+            -0.012310649268329144,
+            0.007649256382137537,
+            0.021306892856955528,
+            -0.006522684823721647,
+            -0.008579902350902557,
+            0.025372346863150597,
+            -0.011788181029260159,
+            -0.010882026515901089,
+            -0.01661284640431404,
+            -0.01635977439582348,
+            0.00241641397587955,
+            0.019902760162949562,
+            -0.0169475506991148,
+            0.021176276728510857,
+            0.00936360377818346,
+            0.0010133018950000405,
+            0.003624620847404003,
+            0.008588065393269062,
+            0.0024878450203686953,
+            0.017878197133541107,
+            0.003883813973516226,
+            -0.01439235731959343,
+            0.0021408938337117434,
+            -0.0030286810360848904,
+            0.004624656867235899,
+            3.4320437407586724e-06,
+            -0.002749078907072544,
+            -0.0065635028295218945,
+            0.003710338147357106,
+            -0.021306892856955528,
+            0.021502818912267685,
+            -0.022678371518850327,
+            -0.004245051648467779,
+            0.005555302836000919,
+            -0.005118552595376968,
+            0.013469874858856201,
+            0.018792515620589256,
+            0.01423724927008152,
+            0.008702355436980724,
+            0.01462093647569418,
+            -0.011331021785736084,
+            0.02323349379003048,
+            -0.013282112777233124,
+            0.0005816536722704768,
+            0.012939243577420712,
+            0.01755165494978428,
+            0.018792515620589256,
+            -0.006245123688131571,
+            -0.010384049266576767,
+            -0.0002946535241790116,
+            0.0129637336358428,
+            -0.022286519408226013,
+            -0.012269831262528896,
+            -0.01386172603815794,
+            0.011486129835247993,
+            -0.004743028432130814,
+            0.025453981012105942,
+            -0.00633492274209857,
+            -0.014653591439127922,
+            -0.015918942168354988,
+            0.014196431264281273,
+            -0.048752784729003906,
+            0.0030348035506904125,
+            -0.0024919267743825912,
+            0.03043375350534916,
+            -0.027021385729312897,
+            0.01738838478922844,
+            -0.0028490826953202486,
+            -0.009355440735816956,
+            0.015429128892719746,
+            -0.015992414206266403,
+            -0.0046613928861916065,
+            -0.0016806729836389422,
+            -0.007539048325270414,
+            -0.00112248957157135,
+            0.017616963014006615,
+            -0.015845470130443573,
+            0.01282495353370905,
+            -0.00989423505961895,
+            0.0027021386194974184,
+            -0.004526694305241108,
+            -0.007751300930976868,
+            0.0196088720113039,
+            0.034548189491033554,
+            -0.007053316105157137,
+            0.0014990337658673525,
+            -0.022547753527760506,
+            0.013192313723266125,
+            -0.006730855442583561,
+            0.02351105399429798,
+            -0.005432849284261465,
+            -0.016898570582270622,
+            -0.013976015150547028,
+            -0.009739127941429615,
+            0.00839214026927948,
+            -0.022433463484048843,
+            -0.011110605672001839,
+            -0.00922482367604971,
+            -0.002573562553152442,
+            -0.018694553524255753,
+            0.004865481983870268,
+            -0.002426618244498968,
+            -0.012506574392318726,
+            -0.008816645480692387,
+            0.01048201136291027,
+            -0.009347276762127876,
+            0.003077662317082286,
+            -0.0011929002357646823,
+            -0.011453475803136826,
+            -0.02369065210223198,
+            0.0158944521099329,
+            0.0001974561164388433,
+            -0.010359558276832104,
+            0.008408467285335064,
+            -0.0018276170594617724,
+            0.001211268245242536,
+            0.016963878646492958,
+            -0.0020664012990891933,
+            0.003467472270131111,
+            0.023478399962186813,
+            0.004057289566844702,
+            -0.010971825569868088,
+            -0.00025906547671183944,
+            0.013535182923078537,
+            -0.005118552595376968,
+            -0.010008525103330612,
+            1.6646010408294387e-05,
+            -0.02984597720205784,
+            -0.008351322263479233,
+            0.006094098091125488,
+            0.018629245460033417,
+            -0.010620792396366596,
+            -0.015853634104132652,
+            0.026319319382309914,
+            -0.006359413731843233,
+            0.019070077687501907,
+            0.002889900468289852,
+            0.0025694805663079023,
+            0.01724144071340561,
+            0.013461710885167122,
+            0.00786150898784399,
+            -0.013927034102380276,
+            -0.02429475635290146,
+            0.0016398552106693387,
+            -0.018090449273586273,
+            0.010579974390566349,
+            -0.011028970591723919,
+            0.0032756286673247814,
+            -0.02398454025387764,
+            -0.012237177230417728,
+            0.004934872500598431,
+            -0.020343592390418053,
+            -0.00040588201954960823,
+            -0.0056940834037959576,
+            0.022237539291381836,
+            -0.005310396198183298,
+            0.008669701404869556,
+            0.0018255761824548244,
+            -0.012237177230417728,
+            0.0009877908742055297,
+            0.009469729848206043,
+            -0.029192892834544182,
+            -0.0007127808639779687,
+            -0.003138889092952013,
+            0.0011418780777603388,
+            0.005351213738322258,
+            0.009755454957485199,
+            0.007322713732719421,
+            0.0027062203735113144,
+            0.007253323681652546,
+            0.010449357330799103,
+            -0.00022041612828616053,
+            0.0197884701192379,
+            0.018172085285186768,
+            -0.007351286243647337,
+            0.008710519410669804,
+            0.024817224591970444,
+            -0.013159659691154957,
+            -0.008718682453036308,
+            0.023266147822141647,
+            -0.006461458280682564,
+            -0.0011122850701212883,
+            -0.01483318954706192,
+            0.00815947912633419,
+            0.0017367975087836385,
+            0.028131630271673203,
+            0.019657853990793228,
+            0.005820618476718664,
+            0.000639819074422121,
+            0.015788326039910316,
+            -0.013020878657698631,
+            0.0035286990460008383,
+            0.015461783856153488,
+            -0.003940958995372057,
+            -0.019821124151349068,
+            0.013878053054213524,
+            0.009502384811639786,
+            -0.003228688146919012,
+            0.01551892887800932,
+            0.011412657797336578,
+            -0.030841931700706482,
+            -0.0011898389784619212,
+            0.010498338378965855,
+            -0.0021449755877256393,
+            -0.008751336485147476,
+            -0.00014273475972004235,
+            -0.012384121306240559,
+            0.021388528868556023,
+            -0.004408322740346193,
+            0.012277995236217976,
+            -0.00815947912633419,
+            -0.017127150669693947,
+            -0.011559601873159409,
+            0.007914572022855282,
+            -0.010547320358455181,
+            -0.0006714528426527977,
+            -0.004022594541311264,
+            -0.005779800936579704,
+            0.017747581005096436,
+            0.004706292413175106,
+            -0.0008964610169641674,
+            -0.0018398624379187822,
+            0.007983962073922157,
+            -0.006979844067245722,
+            -0.008726846426725388,
+            0.004816500470042229,
+            -0.012833117507398129,
+            0.003981776535511017,
+            0.005543057806789875,
+            0.01506176870316267,
+            -0.014857680536806583,
+            -0.021159948781132698,
+            0.007020662073045969,
+            0.02695607766509056,
+            -0.02516009286046028,
+            -0.005232842173427343,
+            0.006245123688131571,
+            0.002093953313305974,
+            0.006914535537362099,
+            -0.006934944540262222,
+            -0.007208424154669046,
+            -0.03657275065779686,
+            -0.0020745648071169853,
+            0.0009005427709780633,
+            0.008963589556515217,
+            -0.008049271069467068,
+            -0.0136494729667902,
+            -0.011135096661746502,
+            0.002410291228443384,
+            0.007134951651096344,
+            -0.02204161323606968,
+            0.0049593630246818066,
+            0.009608510881662369,
+            -0.019559891894459724,
+            0.025127438828349113,
+            -0.010890189558267593,
+            -6.964537897147238e-05,
+            -0.00995954405516386,
+            0.021633435040712357,
+            -0.013486201874911785,
+            -0.018368011340498924,
+            -0.0006898208521306515,
+            0.008355404250323772,
+            0.011926962062716484,
+            -0.0017663904000073671,
+            -0.011371839791536331,
+            0.0003752686607185751,
+            0.01646590232849121,
+            0.019135385751724243,
+            0.002744997153058648,
+            -0.0024837632663547993,
+            -0.01019628718495369,
+            -0.009592183865606785,
+            -0.0073961857706308365,
+            -0.0007449248805642128,
+            0.002091912319883704,
+            0.02035992033779621,
+            0.013469874858856201,
+            0.0024837632663547993,
+            -0.015135240741074085,
+            -0.005257333163172007,
+            0.008939098566770554,
+            0.009412584826350212,
+            -0.011690218932926655,
+            -0.00898808054625988,
+            0.002659279853105545,
+            0.02249877341091633,
+            0.017633290961384773,
+            -0.0014000505907461047,
+            0.0032878739293664694,
+            -0.0016592437168583274,
+            0.012506574392318726,
+            -0.007555375341325998,
+            -0.010269759222865105,
+            -0.006943108048290014,
+            -0.014645427465438843,
+            -0.010800390504300594,
+            -0.0023062059190124273,
+            -0.010179960168898106,
+            0.022629389539361,
+            0.007988044060766697,
+            0.009779945947229862,
+            0.008265605196356773,
+            0.010269759222865105,
+            0.002087830565869808,
+            0.025715215131640434,
+            -0.004424649756401777,
+            -0.011984107084572315,
+            0.007236996199935675,
+            -0.022368155419826508,
+            -0.021535472944378853,
+            0.008257441222667694,
+            0.006367577239871025,
+            0.03595232218503952,
+            -0.026890767738223076,
+            0.0052981507033109665,
+            -0.009779945947229862,
+            -0.010400376282632351,
+            0.007996207103133202,
+            -0.0015439334092661738,
+            -0.004245051648467779,
+            0.022368155419826508,
+            0.00726556871086359,
+            0.0035797213204205036,
+            0.0032817514147609472,
+            0.019837452098727226,
+            -0.005685919895768166,
+            0.01573934406042099,
+            0.0259274672716856,
+            0.024621298536658287,
+            0.009388094767928123,
+            -0.009469729848206043,
+            0.02785406820476055,
+            0.0002895512734539807,
+            -0.0034327772445976734,
+            0.007220669183880091,
+            -0.00305113079957664,
+            -0.0009556467994116247,
+            -0.010033016093075275,
+            0.021764053031802177,
+            -0.001455154619179666,
+            0.004359341226518154,
+            0.02081707864999771,
+            -7.774515688652173e-05,
+            0.006220633164048195,
+            0.01057181041687727,
+            0.002181711606681347,
+            0.003396041225641966,
+            0.010775899514555931,
+            -0.039315707981586456,
+            -0.0006571666453965008,
+            -0.025600925087928772,
+            -0.0018286375561729074,
+            0.0007357408758252859,
+            -0.01266984548419714,
+            0.013600491918623447,
+            -0.002015379024669528,
+            -0.010751409456133842,
+            -0.00029337796149775386,
+            0.021649762988090515,
+            0.005416522268205881,
+            0.004071575589478016,
+            -0.004469549283385277,
+            0.018792515620589256,
+            -0.006955353543162346,
+            -0.017861871048808098,
+            -0.012759645469486713,
+            -0.012367794290184975,
+            -0.01918436773121357,
+            -0.01214737817645073,
+            -0.0024551907554268837,
+            -0.002667443361133337,
+            0.004738946910947561,
+            0.002481722505763173,
+            -0.003228688146919012,
+            -0.02070278860628605,
+            0.015037278644740582,
+            -0.01430255826562643,
+            0.012873934581875801,
+            -0.011053460650146008,
+            0.005963481031358242,
+            -0.004122598096728325,
+            0.012726990506052971,
+            -0.021404854953289032,
+            0.02070278860628605,
+            -0.01536382082849741,
+            0.0038532004691660404,
+            0.019592545926570892,
+            0.006551257334649563,
+            -0.008118661120533943,
+            -0.013404566794633865,
+            -0.022074267268180847,
+            0.021633435040712357,
+            -0.011633073911070824,
+            -0.018237393349409103,
+            -0.004608329851180315,
+            0.009649328887462616,
+            -0.017470018938183784,
+            -0.007367613259702921,
+            0.01707816869020462,
+            -0.0018949664663523436,
+            0.012980061583220959,
+            0.023004913702607155,
+            -0.0052981507033109665,
+            0.0036572751123458147,
+            -0.009167678654193878,
+            0.0010286086471751332,
+            -0.02470293454825878,
+            0.013527019880712032,
+            0.008971752598881721,
+            0.006269614677876234,
+            -0.013935198076069355,
+            -0.013306603766977787,
+            0.0036531933583319187,
+            0.004498121794313192,
+            0.011437147855758667,
+            0.009420748800039291,
+            0.004220560658723116,
+            -0.013918871060013771,
+            -0.027788760140538216,
+            -0.027070365846157074,
+            0.03595232218503952,
+            -0.0005002731923013926,
+            -0.020947696641087532,
+            0.03399306535720825,
+            0.016114868223667145,
+            0.0022980421781539917,
+            -0.012449429370462894,
+            -0.004314441699534655,
+            0.024621298536658287,
+            0.016400592401623726,
+            0.010465684346854687,
+            -0.0004742518358398229,
+            0.005571629852056503,
+            0.0016469983384013176,
+            0.03012353740632534,
+            0.015323002822697163,
+            0.0009459525463171303,
+            -0.038597315549850464,
+            0.011943289078772068,
+            -0.007547211833298206,
+            0.0013408648082986474,
+            0.006702283397316933,
+            0.017780235037207603,
+            0.011935125105082989,
+            0.015567909926176071,
+            0.014865843579173088,
+            -0.003065417055040598,
+            0.0029511270113289356,
+            0.0006163488142192364,
+            -0.009624837897717953,
+            -0.001767410896718502,
+            -0.008881953544914722,
+            -0.005714492406696081,
+            -0.002920513739809394,
+            0.005592038854956627,
+            0.007657419890165329,
+            -0.003983817528933287,
+            0.028049994260072708,
+            -0.0006433906382881105,
+            -0.017600636929273605,
+            -0.009061552584171295,
+            -0.03611559048295021,
+            0.00434301421046257,
+            -0.030939893797039986,
+            -0.011257549747824669,
+            0.0033674687147140503,
+            -0.020457882434129715,
+            -0.024768242612481117,
+            0.0010306495241820812,
+            0.014335212297737598,
+            0.007392104249447584,
+            0.0033327736891806126,
+            -0.01004934310913086,
+            0.003816464450210333,
+            -0.001855169073678553,
+            -0.006461458280682564,
+            0.028833696618676186,
+            0.019559891894459724,
+            0.020457882434129715,
+            -0.01190247107297182,
+            0.019298657774925232,
+            0.005845109466463327,
+            0.02035992033779621,
+            0.010465684346854687,
+            -0.0048695639707148075,
+            0.0074859848245978355,
+            0.009567692875862122,
+            -0.006155324634164572,
+            -0.021519144997000694,
+            0.013192313723266125,
+            -0.0010837126756086946,
+            -0.0052083516493439674,
+            -0.016269976273179054,
+            -0.003534821793437004,
+            -0.015812816098332405,
+            0.010547320358455181,
+            -0.017159804701805115,
+            0.012498411349952221,
+            -0.000802069844212383,
+            0.00168475485406816,
+            -0.015469946898519993,
+            -0.013951525092124939,
+            -0.0012439225101843476,
+            -0.01244126632809639,
+            -0.008971752598881721,
+            0.006163488142192364,
+            -0.007539048325270414,
+            -0.008318668231368065,
+            -0.00535937724635005,
+            -0.011437147855758667,
+            -0.010882026515901089,
+            0.004367504734545946,
+            -0.0010653446661308408,
+            0.020931368693709373,
+            0.00011671340325847268,
+            -0.005824700463563204,
+            0.01190247107297182,
+            0.02787039615213871,
+            0.0016704685986042023,
+            0.004583839327096939,
+            -0.014261740259826183,
+            -0.011118769645690918,
+            -0.02279266156256199,
+            0.016572028398513794,
+            0.0049675265327095985,
+            -0.0344502255320549,
+            0.01438419334590435,
+            0.00824927818030119,
+            0.0015102586476132274,
+            -0.006698201410472393,
+            -0.005334886722266674,
+            -0.005608365871012211,
+            0.006306350696831942,
+            -0.0022837561555206776,
+            0.0005107327597215772,
+            0.0035919665824621916,
+            0.034678805619478226,
+            -0.0040674940682947636,
+            -0.007228832691907883,
+            0.00981259997934103,
+            0.0005130287609063089,
+            0.020196648314595222,
+            -0.005518566817045212,
+            0.0037572786677628756,
+            -0.019282329827547073,
+            -0.016563864424824715,
+            0.0018429238116368651,
+            -0.01722511276602745,
+            0.023282473906874657,
+            -0.005392031744122505,
+            -0.01347803883254528,
+            -0.006906372029334307,
+            -0.020115012302994728,
+            0.012637191452085972,
+            0.02310287579894066,
+            0.007016580086201429,
+            0.01033506728708744,
+            0.012131051160395145,
+            -0.02444170042872429,
+            -0.020457882434129715,
+            0.012171869166195393,
+            0.0013571919407695532,
+            -0.008449285291135311,
+            -0.005404277238994837,
+            -0.001783737912774086,
+            0.007783954963088036,
+            -0.0032633834052830935,
+            0.007881917990744114,
+            0.06491663306951523,
+            -0.01093100756406784,
+            -0.002987863030284643,
+            -0.02966637909412384,
+            -0.004396077245473862,
+            -0.006628811359405518,
+            0.0056287748739123344,
+            0.009151351638138294,
+            -0.023135529831051826,
+            0.009232986718416214,
+            -0.030025575309991837,
+            -0.010628955438733101,
+            0.00824111420661211,
+            0.0028490826953202486,
+            -0.01220452319830656,
+            -0.0009592183632776141,
+            -0.013665799982845783,
+            0.0006107363733462989,
+            -0.009412584826350212,
+            -0.00576347392052412,
+            0.012718827463686466,
+            -0.02321716584265232,
+            0.006298186723142862,
+            0.007592111360281706,
+            -0.006514521315693855,
+            -0.013722945004701614,
+            -0.017861871048808098,
+            -0.011967780068516731,
+            -0.013314766809344292,
+            0.002996026771143079,
+            0.01190247107297182,
+            -0.02607441134750843,
+            0.012922916561365128,
+            -0.0037389106582850218,
+            0.00945340283215046,
+            0.004404240753501654,
+            0.013469874858856201,
+            0.018074123188853264,
+            0.009355440735816956,
+            0.0021429345943033695,
+            -0.008024780079722404,
+            0.012702500447630882,
+            -0.01993541419506073,
+            -0.01635161228477955,
+            0.013837235048413277,
+            0.012016761116683483,
+            0.003061335301026702,
+            0.009828926995396614,
+            0.0020959940738976,
+            0.011886144056916237,
+            0.010775899514555931,
+            -0.019739490002393723,
+            -0.0006648199632763863,
+            0.0007326795603148639,
+            0.03280118480324745,
+            -0.0005938990507274866,
+            0.008857463486492634,
+            0.004420568235218525,
+            0.013159659691154957,
+            -0.012784135527908802,
+            -0.01769859902560711,
+            0.0008087027235887945,
+            -0.01057181041687727,
+            0.021257910877466202,
+            -0.008192133158445358,
+            0.006216551177203655,
+            -0.018612917512655258,
+            -0.027086693793535233,
+            0.006392067763954401,
+            -0.016261812299489975,
+            -0.018221067264676094,
+            0.01317598670721054,
+            0.0077268099412322044,
+            0.0017572063952684402,
+            -0.005098143592476845,
+            -0.020457882434129715,
+            -0.009837090037763119,
+            0.028441844508051872,
+            0.011102442629635334,
+            -0.004120557103306055,
+            0.009584019891917706,
+            -0.009094206616282463,
+            -0.012490247376263142,
+            -0.0017541450215503573,
+            0.0036715613678097725,
+            0.008261523209512234,
+            0.005645101889967918,
+            -0.01386172603815794,
+            -0.00029873527819290757,
+            -0.012400448322296143,
+            -0.021225256845355034,
+            -0.008906444534659386,
+            0.016751626506447792,
+            0.007820690982043743,
+            -0.0038960592355579138,
+            0.015812816098332405,
+            -0.004404240753501654,
+            0.002189875114709139,
+            0.014294394291937351,
+            0.005334886722266674,
+            -0.004285869188606739,
+            -0.0095758568495512,
+            -0.004612411838024855,
+            0.004043003544211388,
+            -0.0025960123166441917,
+            0.006457376293838024,
+            0.017192458733916283,
+            0.0072900597006082535,
+            -0.013755599036812782,
+            0.013135168701410294,
+            0.0007791098323650658,
+            0.0049675265327095985,
+            -0.03699725493788719,
+            -0.003247056156396866,
+            0.015796490013599396,
+            -0.013020878657698631,
+            0.02501314878463745,
+            0.005400195252150297,
+            0.022743679583072662,
+            0.015233203768730164,
+            0.008669701404869556,
+            -0.001542912912555039,
+            0.00464098434895277,
+            0.02759283408522606,
+            -0.003151134354993701,
+            -0.008637047372758389,
+            -0.0020602785516530275,
+            0.009241150692105293,
+            0.0002063850115519017,
+            0.003000108525156975,
+            0.008281932212412357,
+            -0.03562577813863754,
+            0.011706545948982239,
+            -0.019576217979192734,
+            0.0024694770108908415,
+            0.003459308762103319,
+            -0.007183933164924383,
+            -0.014865843579173088,
+            -0.005857354495674372,
+            0.022841641679406166,
+            -0.003224606392905116,
+            0.021078312769532204,
+            0.002885818714275956,
+            0.016106704249978065,
+            -0.011575928889214993,
+            -0.009208496659994125,
+            -0.0004130251472815871,
+            -0.005351213738322258,
+            -0.007539048325270414,
+            0.010596301406621933,
+            0.0034409407526254654,
+            0.005534893833100796,
+            0.002265388146042824,
+            0.0036654386203736067,
+            -0.0036878883838653564,
+            0.02773977816104889,
+            0.003563394071534276,
+            0.007996207103133202,
+            -0.011567764915525913,
+            0.014082142151892185,
+            -0.017029186710715294,
+            0.0070778070949018,
+            0.005575711838901043,
+            -0.026156047359108925,
+            0.016849588602781296,
+            0.004775682929903269,
+            0.006628811359405518,
+            0.013812744058668613,
+            0.01618834026157856,
+            -0.010408539324998856,
+            -0.014816862531006336,
+            0.024049850180745125,
+            -0.0032041973900049925,
+            -0.015225040726363659,
+            -0.006775755435228348,
+            4.438935866346583e-05,
+            0.006559420842677355,
+            0.00373074715025723,
+            0.0013020879123359919,
+            -0.0006204306264407933,
+            -0.010743245482444763,
+            0.004645065870136023,
+            -0.01934763789176941,
+            -0.0007163524278439581,
+            -0.016449574381113052,
+            -0.015633217990398407,
+            0.018678225576877594,
+            -0.01830270327627659,
+            0.009633000940084457,
+            0.0012275953777134418,
+            0.028588788583874702,
+            -0.011706545948982239,
+            -0.012155541218817234,
+            0.0015102586476132274,
+            0.010849371552467346,
+            0.0067512644454836845,
+            -0.010033016093075275,
+            -0.007583947852253914,
+            -0.014180104248225689,
+            -0.010163633152842522,
+            0.015502601861953735,
+            -0.014498483389616013,
+            -0.014343376271426678,
+            0.008914608508348465
+          ],
+          "embedding": [
+            -0.006359657085039302,
+            -0.024594786241399234,
+            -0.02553538924848095,
+            0.005360188465088291,
+            0.012801870977589839,
+            0.021680505862506104,
+            -0.016668048966001273,
+            0.06860852474581301,
+            -0.007122272841157724,
+            0.03471775000705488,
+            -0.009004297848861675,
+            0.01059974144678391,
+            0.02565604599426148,
+            -0.022544127646041287,
+            0.010616674976540785,
+            0.013658194823359737,
+            0.016400329743363844,
+            0.02858546964917478,
+            0.0067612676434693686,
+            -0.04174737193606879,
+            0.03184155110488616,
+            -0.02223105127585514,
+            -0.05506239667974977,
+            -0.0015307551410555752,
+            -0.013169788751379287,
+            0.017681749722384007,
+            0.01682706317229353,
+            0.019652295823816928,
+            -0.05623980369271269,
+            0.030698582851533197,
+            0.02286327124296613,
+            0.016019032539689178,
+            0.005508113353872315,
+            8.58985091215149e-05,
+            -0.019206292853495906,
+            -0.018439290483495144,
+            0.03475023176950151,
+            0.01018612942141759,
+            -0.0252551158857426,
+            0.0334562130666827,
+            0.020462153537485785,
+            0.02260508249714211,
+            -0.03323524989392951,
+            -0.002094251009530707,
+            0.015177442347686773,
+            -0.00018698464350038317,
+            -0.0006788648344273403,
+            0.0007777616893581039,
+            0.013352398615059647,
+            -0.030751111528744813,
+            -0.0317733818445192,
+            -0.006729064081710058,
+            0.009413204899986514,
+            -0.005712894407943592,
+            -0.038961826880527066,
+            -0.002827692206530345,
+            0.017863300537973137,
+            0.07918455388563496,
+            0.048967158919816366,
+            0.018002566325350088,
+            0.040264962620950695,
+            -0.019715100359659787,
+            -0.02211350500879408,
+            0.004918148608475574,
+            0.01062585305866606,
+            -0.014478993521696027,
+            0.0034983131226364672,
+            0.000766228629514787,
+            0.009667268771502067,
+            -0.01558439890697323,
+            -0.04984718709870083,
+            0.029613955432821887,
+            -0.009686854114630624,
+            -0.015279319238124568,
+            0.007460629676325322,
+            -0.006235979350809034,
+            0.012514387473717085,
+            0.02800435263741497,
+            -0.008688925946672778,
+            -0.01236301543374469,
+            0.029755705020188972,
+            0.01158037219446542,
+            -0.005242392206024511,
+            0.009982679027728814,
+            0.052414361714483264,
+            -0.018788500381726227,
+            -0.020335831196027204,
+            -0.08583959943356621,
+            0.029309228216812346,
+            -0.0249285282010673,
+            -0.018929396555426307,
+            0.007710663193334574,
+            0.03096798035958051,
+            -0.017089977064591828,
+            -0.028015107605405817,
+            0.06740703271128606,
+            0.009398795412354399,
+            0.01118769022826835,
+            -0.03041417460855722,
+            0.014776002883150835,
+            0.004319194067673161,
+            0.022507636639041203,
+            0.0053562788646786845,
+            0.007548072258943124,
+            0.011123374308563128,
+            -0.006924362115585165,
+            -0.011118507147092024,
+            0.009563232643295304,
+            0.00657178473678069,
+            0.08313027447380966,
+            -0.004239370156003264,
+            -0.00930406163946907,
+            -0.039202672821898946,
+            0.009761029930556298,
+            -0.02827698465424993,
+            -0.037914533246900244,
+            -0.04496050911855815,
+            -0.017711307900236353,
+            0.01934435542287117,
+            0.004335173027873153,
+            0.03800509969714242,
+            0.031198127357716314,
+            0.031311207941680276,
+            -0.012184553894588813,
+            -0.058985863670856876,
+            0.041442017057707635,
+            -0.012456189292721711,
+            -0.0040966351720394135,
+            0.010849179519959421,
+            0.005243123049347636,
+            0.0011495799963862647,
+            0.003534183676934487,
+            -0.03139340512003631,
+            -0.023876261883905306,
+            0.008382957569752911,
+            -0.004778283217011976,
+            0.03703067589888386,
+            0.06888630688839406,
+            -0.03993532296478411,
+            -0.009472167678222372,
+            0.025246872942105057,
+            -0.01524567912700368,
+            0.01595917939878582,
+            -0.003736723164442467,
+            -0.030836797784967753,
+            0.0026246446163368533,
+            0.008894300082148884,
+            0.08342856279154155,
+            -0.0007476580521704578,
+            0.025149116238550588,
+            0.004972542633631468,
+            0.034495632546556576,
+            -0.009173329764686424,
+            0.014009381543772768,
+            0.004427261668151028,
+            -0.013426138607902904,
+            -0.022234568880321808,
+            0.01976282364689966,
+            -0.03592433789490043,
+            -0.010918278220831188,
+            0.013527180289641174,
+            -0.013023113052242677,
+            -0.008304770302495363,
+            0.003677063799167178,
+            0.0013981822663219421,
+            0.047001163242600175,
+            0.01828332714887986,
+            0.002483375662343689,
+            -0.06820498187523893,
+            0.05295312038995555,
+            0.025038753504342162,
+            0.025286870198436735,
+            -0.0029544808216316115,
+            0.029982752196635428,
+            0.0005600481523874767,
+            0.0128290258663539,
+            0.009023805227424919,
+            -0.0019219166367750596,
+            -0.04482843936478531,
+            0.04523479691409549,
+            -0.004678574027241186,
+            0.010486850804218904,
+            0.006736229865018014,
+            0.02134813656476443,
+            0.028083972673690036,
+            0.016857413440125445,
+            0.0189082447820146,
+            0.03843690613820022,
+            -0.011922027011229613,
+            0.016193433524453224,
+            0.03828207911467065,
+            -0.0013055128404331705,
+            0.043198163443700956,
+            0.024025562893260542,
+            -0.001257851946749882,
+            -0.011726713433348844,
+            -0.021907984099443396,
+            0.02752198420597673,
+            0.004431866336999851,
+            0.015798806714413912,
+            -0.012910782191382367,
+            -0.03180549180206364,
+            0.004550247372715741,
+            -0.022778272841118514,
+            0.031242218240017532,
+            -0.006106984949693483,
+            -0.006259993190497421,
+            0.02187449047033339,
+            0.009334146847221406,
+            -0.009381278960016145,
+            -0.020888853823931134,
+            -0.007502374908776156,
+            -0.010390018386728482,
+            -0.04316693435221676,
+            -0.006274513730720055,
+            0.019152443937064258,
+            -0.01668984824845791,
+            0.018573194036505578,
+            0.0035035262183249925,
+            -0.019779911381789077,
+            -0.0027495798165085993,
+            0.02425481236315913,
+            0.006857723545270935,
+            -0.014733535699087731,
+            -0.026326146471941727,
+            -0.019906660231056833,
+            -0.001033294559612807,
+            0.0159154063026895,
+            0.035649534442431415,
+            -0.026210592096240003,
+            0.014397127229517285,
+            0.005114311681443673,
+            -0.0033104236005207077,
+            0.00324988235167305,
+            0.03764004154553922,
+            -0.017822437760737605,
+            -0.008579701063708334,
+            -0.00032410984200965487,
+            0.018450351071676443,
+            0.017283379764846195,
+            0.006384411611332382,
+            -0.005115090309123598,
+            -0.030590047062583125,
+            -0.011776211359193503,
+            0.01183623914060493,
+            0.0016564847796208331,
+            -0.009530766634361575,
+            -0.004810598258154945,
+            0.03876242515526699,
+            0.027076785774401667,
+            0.005324763136165989,
+            -0.03416412480913939,
+            -0.007943016733586697,
+            0.002114307650456928,
+            -0.02961449347300628,
+            0.05142732770518828,
+            0.05566026433842563,
+            -0.0030729309664974704,
+            -0.00938186948393074,
+            -0.04547908981236149,
+            0.0651216359692231,
+            -0.01813722445944094,
+            -0.0482650853560993,
+            0.044045155892545554,
+            -0.014272653003441165,
+            0.013740652306746851,
+            0.04563732439184395,
+            0.02126513340521316,
+            -0.0054845632439384905,
+            -0.011635625101262898,
+            0.030992082069685376,
+            -0.010051935721068779,
+            -0.050967258971854425,
+            -0.009827942095033494,
+            0.013373878556654464,
+            -0.013017627321534534,
+            -0.009361001542607812,
+            -0.01925867665425919,
+            0.020329052915436022,
+            0.023020741670660188,
+            0.019387043947695172,
+            0.005628632318152202,
+            0.018451555161952665,
+            0.016226845400848455,
+            -0.01384033112836216,
+            0.004762589697426251,
+            -0.012718476561725138,
+            -0.024222765127494463,
+            -0.037487144314010595,
+            -0.019971524166659278,
+            -0.015034588610516958,
+            0.009483936496215338,
+            -0.014301924283553591,
+            -0.0024139665139412125,
+            0.013361045353529458,
+            -0.0005870798721107379,
+            -0.011192319340879789,
+            0.013927695434391159,
+            0.021337539669342293,
+            -0.027300952495815437,
+            -0.04765670695199488,
+            -0.00561706224251562,
+            0.0082776639906139,
+            -0.0030984973749633724,
+            -0.026269016057688548,
+            0.007040524859505573,
+            0.0017859008072463442,
+            0.020091542517503808,
+            -0.018716761130867725,
+            0.0035283051960672963,
+            0.008519383548064053,
+            -0.015247417050336866,
+            -0.016917626868166634,
+            -0.018580550690103854,
+            -0.024697180580294884,
+            0.034202028316737125,
+            0.0204395003463227,
+            0.030392895534267717,
+            -0.04527669147753553,
+            -0.018359428092075348,
+            0.05494107499558624,
+            0.015234754907198422,
+            -0.05066008106884831,
+            0.031139824660661085,
+            0.005016615710206006,
+            0.0019000453624929544,
+            -0.006094364589229031,
+            0.0637324238017742,
+            -0.02711381489292883,
+            -0.02187418906973867,
+            0.004601744864532875,
+            -0.04141121277496072,
+            -0.026689585608019588,
+            -0.005568100750837504,
+            0.01291019036002459,
+            -0.031853041426355644,
+            -0.02815800259372657,
+            -0.007964861641790374,
+            -0.010933515363060793,
+            0.013726639077745378,
+            -0.023956800669647334,
+            -0.015672058590455337,
+            -0.023642025374517057,
+            -0.01288435863544158,
+            -0.0042516881552192385,
+            -0.011162214160460695,
+            -0.02412733256710621,
+            0.010088528028211243,
+            0.03310175458095482,
+            0.023064360903748363,
+            0.001996159684818467,
+            0.009478229572933101,
+            0.014478772877286076,
+            0.00371783700962415,
+            0.007864103610320116,
+            -0.048511515124609,
+            -0.031398754206534776,
+            0.020227872561659397,
+            0.012159811587094092,
+            0.0172163448580163,
+            0.009578952796095119,
+            -0.04154545961112031,
+            -0.05095080393682759,
+            0.02968486144855133,
+            -0.03932395023720674,
+            -0.02063219739061335,
+            0.007512423249865194,
+            -0.011911709468952548,
+            -0.00810951734088105,
+            -0.0008079327452253181,
+            -0.01600701450676789,
+            0.023984909975645632,
+            -0.02418072541081482,
+            0.0033984265728439398,
+            -0.0104343782811971,
+            0.01588977364765743,
+            0.05109529246167992,
+            -0.027800269136366997,
+            -0.027889034481533438,
+            0.03706813210665186,
+            -0.02058120652429308,
+            0.030801396468451574,
+            0.03246127847621613,
+            0.008355517405358826,
+            -0.008605467586247932,
+            0.017693344563968573,
+            0.014499487024471321,
+            0.012431277080478217,
+            -0.040103870071440256,
+            0.017328848567062113,
+            0.02723900253727976,
+            -0.054909074068976875,
+            -0.008547596270650966,
+            -0.00777964838164648,
+            0.027134666572242665,
+            0.02765036594970255,
+            -0.002207173084941375,
+            -0.016838740117653725,
+            -0.028289233739722936,
+            -0.019433376533829132,
+            -0.011053380458946764,
+            0.03247641283550457,
+            0.0007514989009289282,
+            0.007919574219000797,
+            0.001316042436488591,
+            -0.0017934342041762042,
+            0.01924145734197619,
+            0.015260199553572459,
+            0.048259602293134723,
+            -0.03383050622747848,
+            0.0021712225589426266,
+            -0.03508567715696004,
+            -0.04578731401250471,
+            -0.008768228935292693,
+            0.010203318819548585,
+            0.024135236371743855,
+            -0.007467140460133838,
+            0.0028907962385387818,
+            -0.007582083854422456,
+            -0.005384477718912037,
+            -0.01867978482547249,
+            0.01641191108381631,
+            0.037168089880543244,
+            0.0031992495663980464,
+            -0.008533890994695162,
+            0.000778843942291118,
+            -0.004620258538699103,
+            0.010572690335163528,
+            0.033233014627100904,
+            -0.030497362482602697,
+            0.02974571152880676,
+            -0.004274442492343003,
+            0.025341347125910928,
+            -0.04307748954707389,
+            -0.018069609916024403,
+            -0.018491122727274337,
+            -0.02069307802429807,
+            0.05372967226264594,
+            -0.030768978301976772,
+            -0.03662045886993282,
+            -0.003205991669059688,
+            0.016815511161004534,
+            -0.0014152644009276566,
+            -0.0016015439520090104,
+            -0.03984716970293867,
+            -0.010828260334550217,
+            -0.05894985435251326,
+            -0.01638035185618134,
+            -0.021819442096404948,
+            -0.005315102328745982,
+            -0.06251579337157112,
+            0.0033713876515840233,
+            -0.07815974854883057,
+            0.001484860071649606,
+            0.04539427129131382,
+            -0.023530943197353476,
+            0.05142250021536681,
+            -0.024746600229063383,
+            -0.0005885740455257141,
+            -0.044698536284421976,
+            0.0026481001040031785,
+            0.02389178603305878,
+            0.051387657872255674,
+            -0.002949198236157656,
+            0.010674377196609455,
+            0.013322075519401089,
+            -0.026245090721950647,
+            -0.01036724558050635,
+            -0.005321884180696542,
+            0.005958798671856245,
+            0.013755178085972732,
+            -0.0074734329537091675,
+            0.038368016046563674,
+            0.010941421833174982,
+            0.03443115781919206,
+            0.034623617136788874,
+            -0.004361926531008214,
+            -0.03348681614127998,
+            -0.00632870804659212,
+            0.024433331840566692,
+            0.0321998807702602,
+            -0.02899983280320611,
+            0.005958471722097962,
+            -0.015602447981516634,
+            0.015633217539747047,
+            0.006321931116948646,
+            -0.011827541320878442,
+            0.02372204564999124,
+            0.015673426811623124,
+            0.006103300725621424,
+            -0.005165006670545588,
+            0.013589526056189892,
+            -0.025411864292353578,
+            0.03041791947375232,
+            -0.004456865298584878,
+            -0.04276058054583262,
+            -0.0024736078733106825,
+            -0.020059309867945227,
+            0.024883721794000956,
+            0.015578237413840115,
+            -0.00536523508503184,
+            -0.005290645798641171,
+            0.008242987948511412,
+            0.020469675635821673,
+            0.006668354647245447,
+            0.00017729517736814364,
+            0.02373604785121342,
+            0.029915687122900485,
+            -0.02432635861789771,
+            -0.029540533943782886,
+            0.008151297444985344,
+            -0.005730733861939129,
+            0.011318816579801282,
+            -0.02644855229221747,
+            0.00033932872948409137,
+            0.0035505684154028996,
+            -0.03570104343808711,
+            0.03408366016010711,
+            -0.009199700348630527,
+            0.005857598530788056,
+            -0.009538177736472163,
+            -0.0017768806148210118,
+            0.0027410153099351694,
+            0.03898108970435115,
+            -0.007367623392751247,
+            0.009827254226889936,
+            0.015465895035342481,
+            0.000981604396724216,
+            0.05143083558797524,
+            -0.027350877485564314,
+            -0.02183036962301852,
+            0.01885468378168636,
+            -0.007285239007489837,
+            -0.016846837665812908,
+            -0.020546383007536074,
+            -0.011863943814684079,
+            -0.016293192750336218,
+            0.01976516340568883,
+            -0.02387328798681798,
+            -0.011892336568452852,
+            0.007170651391639504,
+            0.0013725014050721622,
+            -0.009809669068563384,
+            0.0016617183704652932,
+            -0.006307590349798603,
+            0.02037693835615797,
+            0.031753656615094446,
+            -0.024892675105483313,
+            0.03577919637821846,
+            0.03178303188315581,
+            0.017260898395251155,
+            -0.02102289690701718,
+            -0.009661035772073294,
+            -0.00048542689442823136,
+            -0.014694631967051945,
+            -0.0326379186158631,
+            0.025443206465204822,
+            0.017194626456982255,
+            -0.011381421003142453,
+            -0.03680783761649673,
+            0.0017216115811536721,
+            0.022819859205829785,
+            -0.013600989058799166,
+            0.028141657788496834,
+            -0.01869047298692401,
+            -0.01366833755869951,
+            -0.00782959271979894,
+            0.031227914441058244,
+            -0.05252392878125891,
+            0.016957842026348195,
+            0.00020880696574616416,
+            0.001154370575234767,
+            0.010892067708194324,
+            -0.008368819756628046,
+            0.012224564683325483,
+            0.031700389750709,
+            0.018063405869147564,
+            0.0010794115307202266,
+            -0.002783668216641561,
+            0.01418234057248712,
+            0.017773269840982066,
+            -0.015653443355569053,
+            -0.000497740485657824,
+            0.019890882604100452,
+            0.022672054781919238,
+            -0.00521797370595221,
+            -0.016686942830774276,
+            -0.018443538351123584,
+            0.025270057639201107,
+            -0.010354431070738671,
+            0.03484047753830186,
+            0.04594785214094975,
+            0.0036441959528283004,
+            -0.05695476763071743,
+            -0.009412209619190693,
+            0.0010371110229966086,
+            -0.01099343927053564,
+            0.01606863921935641,
+            0.012470660511773616,
+            -0.03237187923799263,
+            0.006388672213167589,
+            0.012946570716567364,
+            0.022152554488829627,
+            -0.02540317621005867,
+            -0.023183252921885226,
+            -0.0030984019855787655,
+            -0.007890467768631811,
+            0.018648760402624687,
+            -0.008653405980661855,
+            0.0006228315253506687,
+            0.009672526897422602,
+            0.0005436494014569835,
+            0.011632326145022935,
+            0.005655942007108228,
+            0.021736827172951,
+            -0.001256608167230203,
+            0.013124413886287987,
+            -0.01986808920858981,
+            -0.0270450339809516,
+            0.006500423802582728,
+            0.04438242119211055,
+            -0.014813399694814117,
+            0.030621709833336064,
+            0.0092171224268983,
+            0.016433384739642534,
+            0.024875072913970443,
+            -0.00891470467241686,
+            -0.014124009014664287,
+            0.00845777541223868,
+            0.009019007848927453,
+            -0.04147358041382793,
+            0.012513030150663913,
+            -0.01710243399313324,
+            -0.033144939719206276,
+            0.01904674924133127,
+            -0.00534122554342279,
+            0.012367320751926314,
+            0.004270315800911569,
+            -0.002886460711688153,
+            -0.025696286516581048,
+            0.014137901708875033,
+            0.028135087538950946,
+            0.021797869775098042,
+            0.0296173052503884,
+            -0.012757078368361444,
+            0.0005460601780756187,
+            -0.008748930817524899,
+            0.005470975782187157,
+            0.04187083128854477,
+            0.0008687182830616019,
+            -0.0008032802520818105,
+            0.01507726552487313,
+            0.024945744517309208,
+            0.0376130540370386,
+            -0.015415560911023553,
+            -0.03407353556457147,
+            0.01564969972264396,
+            -0.010104443707865262,
+            0.018855792875436674,
+            0.005503259932317811,
+            0.0049634097823351596,
+            0.0038600156268000702,
+            0.019312419743287703,
+            -0.006635193789316492,
+            0.018798045806429395,
+            -0.02500983683871332,
+            -0.028785757874583755,
+            0.01829411751138161,
+            -0.005277972443258226,
+            0.03338623299668019,
+            0.016225885095995672,
+            -0.029522280454919586,
+            -0.01712097344863975,
+            0.01553706187924322,
+            -0.008465766956343768,
+            -0.024842700926275538,
+            0.01895228828512998,
+            0.007076862934523977,
+            0.008956105593078965,
+            0.0024373779389833218,
+            -0.01106503696703573,
+            0.01125466547367417,
+            -0.000818223040067914,
+            -0.015322790791698395,
+            0.014157131555941659,
+            -0.017853386731281715,
+            -0.014994640991761465,
+            0.01775991780643474,
+            -0.037498078532902435,
+            0.009570197712419421,
+            -0.007687297543935123,
+            0.009050494195378254,
+            -0.004222041469820445,
+            0.007460383234429904,
+            -0.0010804379186060264,
+            -0.014987857934522812,
+            -0.014903749199810126,
+            0.03228485430019625,
+            0.022096121825475306,
+            -0.000410816936759558,
+            0.006728733918931589,
+            -3.971636461500232e-05,
+            -0.026719848032785824,
+            -0.011253751790407687,
+            0.02245467425830765,
+            -0.005664635254398481,
+            0.005134496095445943,
+            -0.01109590855846966,
+            -0.0149616794609307,
+            0.0019726017255571673,
+            0.032668637270108894,
+            0.06066635618119076,
+            0.012001732730659747,
+            0.024422093585231484,
+            0.0189499396761026,
+            0.02119276033547528,
+            -0.008489818917377516,
+            0.006413901834269739,
+            -0.03097413287789443,
+            0.03509225573986143,
+            0.005392594635384506,
+            0.03108937318306421,
+            0.026664331248813712,
+            -0.04722525921137681,
+            -0.0058539171830594774,
+            0.017087531454837677,
+            0.022673712717885323,
+            0.0159020681868943,
+            0.0008382498574190522,
+            -0.008938174268517353,
+            0.035650940044285984,
+            -0.03315250299348084,
+            -0.023647111263155,
+            0.005059382024092534,
+            -0.01439537467317451,
+            -0.010488359956731476,
+            -0.006785295580877329,
+            0.02776449681317373,
+            -0.005584052419598926,
+            -0.004582019433008779,
+            0.009697064252638535,
+            -0.014400206558190682,
+            0.008165676294736836,
+            0.0007624778531368436,
+            0.008681263897172326,
+            0.039249866981334126,
+            0.0005918063050375113,
+            0.024275699182767528,
+            -0.004138313072843555,
+            0.011079529739650594,
+            -0.0158972266904151,
+            -0.016985058543605,
+            0.02276982363172991,
+            0.00351101120496393,
+            -0.03244126813432563,
+            -0.00768819222024939,
+            0.00928180979884261,
+            -0.0013793404539641824,
+            0.005266058566444064,
+            -0.00503433670609399,
+            0.003747448261046857,
+            0.03286230666764075,
+            0.0022383592721685153,
+            0.008606190020480513,
+            0.013593947677257625,
+            0.00030951915677929204,
+            -0.01490307680740127,
+            0.013254591138113545,
+            -2.1379308136707722e-05,
+            -0.012900073369514622,
+            0.03357519186606058,
+            0.006705902094864628,
+            0.04243347200485736,
+            -0.03302953674711988,
+            0.009031534935013593,
+            0.01822227379789576,
+            -0.008491770335368761,
+            -0.02725412856390648,
+            -0.03630788717786838,
+            0.008259145747740535,
+            -0.019003260334340084,
+            0.015781275447792516,
+            -0.010039277287105153,
+            0.029665124310477384,
+            -0.03563370794521629,
+            0.004714858110268169,
+            0.004053913355055313,
+            0.0009503673086552651,
+            0.02068360255395961,
+            0.018524093113914366,
+            -0.002785698934204024,
+            -0.031443835341186485,
+            -0.02066365433385339,
+            0.006869321907853112,
+            0.02431260052207595,
+            -0.010367048093621498,
+            -0.005612149485859478,
+            0.0006316058139530722,
+            -0.024369562712568606,
+            -0.016845124984022828,
+            0.030359419843284823,
+            -0.040974427176153246,
+            0.005252911449822132,
+            0.04272936053727608,
+            -0.04599092715016919,
+            -0.03802288716943514,
+            -0.002314988728575789,
+            -0.024124014894379466,
+            -0.006360771882090754,
+            -0.002068030981218464,
+            0.016669991113175926,
+            0.0036232707799735537,
+            -0.028763764013482795,
+            0.00943401978066451,
+            -0.005173988601101547,
+            0.003971420424839974,
+            -0.0031669592494977393,
+            -0.02216336363004636,
+            -0.01657772032717298,
+            -0.02531962163186398,
+            -0.035174102848865116,
+            -0.0015903434876311088,
+            -0.005158703150821607,
+            -0.028703075279214722,
+            -0.03941940849540492,
+            -0.021696000925262537,
+            -0.0160221488026189,
+            -0.00843213135542359,
+            0.040641882367348774,
+            0.021319060666253945,
+            0.019336844555345886,
+            0.0025252167889382917,
+            -0.020346900609125973,
+            0.049440028814233025,
+            -0.007642182460707816,
+            0.035628691886108926,
+            -0.0017882888000877158,
+            -0.013461669667950893,
+            0.00455558389906464,
+            0.03825428026120858,
+            -0.014248434654950971,
+            -0.0009902062564027718,
+            0.00027811655483232545,
+            -0.0011866253757890136,
+            0.0008535155950871286,
+            0.015433650396777696,
+            0.001307651096343185,
+            -0.007112199543314309,
+            0.030380673552683906,
+            -0.012803247194895328,
+            0.01637501752192592,
+            0.018350814869962304,
+            -0.0013359719137099667,
+            0.008064546833360553,
+            -0.021430659268086814,
+            -0.00731572118780415,
+            0.0044578576331236325,
+            0.008422210247967642,
+            -0.015096885054883586,
+            -0.009964588397149773,
+            -0.02359494467299634,
+            -0.01174145797396426,
+            0.0023855749166744366,
+            -0.004559202896782688,
+            0.004902138778592298,
+            0.034595817216889944,
+            -0.016068744500427584,
+            -0.035106033294403614,
+            0.009997207702122002,
+            -0.02663088875167486,
+            0.02222306406130687,
+            3.983384942889326e-05,
+            0.006838221769859366,
+            -0.0023434342406307616,
+            0.001765445386886839,
+            -0.00169484278667461,
+            0.002862331185439823,
+            -0.018217975204395134,
+            0.0031859734246840257,
+            -0.0032119988065673565,
+            0.0009052950782312657,
+            -0.0012459653052875638,
+            0.00687339892509406,
+            0.006332799551164718,
+            -0.0008019538898592668,
+            -0.002968451740267742,
+            -8.541086355097327e-06,
+            -0.020561816931299912,
+            -0.021642512405352372,
+            -0.028259049261410905,
+            -0.000989887278779396,
+            -0.012359733017834338,
+            -0.027804394906433664,
+            0.011443385424045703,
+            -0.006872684999639775,
+            0.008327844771213648,
+            -0.008969527138034904,
+            0.006219994886600349,
+            -0.008415106130309062,
+            -0.0004143996206881767,
+            0.013119286511794273,
+            -0.02925209282917943,
+            -0.02096907906398536,
+            -0.03096513357712367,
+            0.01762223841790055,
+            -0.00925741670612824,
+            -0.002656382260603827,
+            0.0025223527960181835,
+            0.005092659166535706,
+            -0.033606542487570154,
+            -0.006653438998497412,
+            0.007313682256676618,
+            0.009547290751154595,
+            -0.03287788897629887,
+            0.02446728061355753,
+            0.017023222076526946,
+            0.01874233066565012,
+            -0.02417566312664911,
+            0.04049827067897005,
+            0.014426879325647042,
+            -0.021740606525480655,
+            0.016712907564723732,
+            0.027385150577780248,
+            0.03177163221499527,
+            -0.03469098547555545,
+            0.004481376248622418,
+            -0.01518200063132508,
+            0.018918680020516044,
+            -0.025284921244460287,
+            0.020625170445627613,
+            -0.016707629602243063,
+            0.021278719363178687,
+            0.03520334001264928,
+            -0.005524369131116028,
+            0.03608133014180745,
+            -0.019234548780647943,
+            0.0059985801736444555,
+            0.004041778461022664,
+            -0.01645503683802124,
+            0.01024694092614158,
+            -0.0042700351372843386,
+            0.0006059159967286885,
+            0.009406090340935747,
+            0.019133229156115282,
+            0.019007219188041775,
+            -0.04442208547053466,
+            0.028588965123434923,
+            0.00804120101594811,
+            0.021472086070876265,
+            0.014177820346835521,
+            -0.02708807970982664,
+            0.004742981711172693,
+            0.002072673190279628,
+            -0.004659713164250794,
+            0.0061167904730676514,
+            -0.013796604518930328,
+            -0.034540857808309226,
+            0.006338792319814448,
+            -0.018918703778187926,
+            0.015036937364271984,
+            -0.009026670935442384,
+            0.0011667525636770179,
+            -0.01152652811400482,
+            -0.011938679695598347,
+            0.0006170592505446954,
+            -0.005495151258143897,
+            -0.0008075736908949407,
+            -0.03171721592827769,
+            -0.04427658539412448,
+            0.02036132271908089,
+            -0.0021221965958423804,
+            -0.04320396743189005,
+            0.004111179168317842,
+            -0.025352428814134187,
+            -0.01865918186335911,
+            -0.011237699157960728,
+            -0.025047923887906474,
+            -0.04219630696740606,
+            -0.011108496740166623,
+            0.007785006066683012,
+            0.016810642610918525,
+            -0.009081815651299525,
+            0.040965574143584864,
+            0.0073944364393447965,
+            0.015313576815177263,
+            -0.017731360025694955,
+            0.026137777790243826,
+            -0.019314392060123,
+            -0.03108486330987073,
+            0.02676829566273746,
+            -6.259161089416372e-05,
+            0.003053748984540415,
+            0.014358356168574803,
+            2.8186924071913628e-05,
+            -0.006181789640967465,
+            0.01947730374003864,
+            -0.028739300217145797,
+            -0.023460503178221397,
+            0.0012460072421725688,
+            -0.008850395570726385,
+            -0.016271242375881602,
+            -0.035896509136022194,
+            0.005227881053956234,
+            0.03324337105109496,
+            0.004970419987430122,
+            0.008273497228454482,
+            0.01814500594735864,
+            -0.02229593571615308,
+            -0.024230675455380678,
+            0.000639112529814716,
+            -0.010266640429960994,
+            -0.010257247303470066,
+            -0.021302483296731352,
+            0.019150757066356872,
+            0.01651375496287087,
+            -0.00390189952706486,
+            -0.006971692709239087,
+            -0.010621003777733253,
+            -0.01918206725910585,
+            -0.02020345113211036,
+            -0.013827322378214754,
+            -0.017980206764311026,
+            -0.034153741666571154,
+            -0.032195001867229935,
+            0.03225205339201753,
+            0.026354817588919596,
+            0.0014076409241492998,
+            -0.007959081015805435,
+            -0.023070419006792155,
+            -0.004483328183581872,
+            0.02879516595345542,
+            0.002549934782150634,
+            -0.0001541849716285838,
+            -0.00830090759167373,
+            0.015341095409025125,
+            0.00835695019244165,
+            0.005107844984733648,
+            -0.06676974053142512,
+            -0.006027540973808174,
+            0.01655283585235268,
+            0.01351155740684989,
+            -0.002359114145814062,
+            0.02591301169926259,
+            0.01108099264396926,
+            -0.015361038608114896,
+            -0.019765743462262668,
+            0.035195761515422364,
+            0.02408910733949848,
+            0.023468767772473746,
+            -0.0034492999385544727,
+            -0.004472311490618405,
+            0.007030833258457338,
+            -0.010684252427867153,
+            -0.013326964647936429,
+            -0.004558513842504658,
+            0.026925146859337397,
+            0.016099443021514896,
+            -0.0064331198280825575,
+            0.010848931851939124,
+            -0.0008320209125061849,
+            -0.019444491995041537,
+            -0.001841330829156697,
+            0.008730275423516445,
+            0.0012974082030866743,
+            -0.014611902493400719,
+            -0.005911807768916134,
+            -0.019130604101278263,
+            0.024719468317569562,
+            0.0209456329723249,
+            0.03381727436368338,
+            -0.013237902591972769,
+            -0.0003695142331921278,
+            0.005410360755058108,
+            0.0002841632253252816,
+            0.0023352645111704958,
+            0.004801123870325242,
+            0.02623649813035263,
+            -0.029010710171522985,
+            -0.020741774066460415,
+            0.0068426980682519505,
+            0.013445270043414486,
+            0.0022744032984779294,
+            0.033840485942696216,
+            0.035040080489222324,
+            0.004166066423508316,
+            -0.010046666684988869,
+            -0.006148765564658713,
+            0.007071740742464309,
+            0.01865897335451031,
+            -0.023267411753900076,
+            -0.0005936657905496225,
+            0.004833030519903764,
+            -0.004229582648167378,
+            0.03620610659071198,
+            -0.009961238880412731,
+            -0.007197670122096852,
+            0.022212791947347127,
+            0.09819205436926473,
+            0.00047695621444943017,
+            0.0025354521225913395,
+            -0.029358210234742176,
+            -0.023614016211567963,
+            -0.0012564198366457442,
+            0.00242680708457405,
+            0.00830167274308547,
+            0.028076260517106013,
+            0.0004852400049405906,
+            0.011579274037419117,
+            0.02773972145320728,
+            -0.017217305534161797,
+            0.006752070301141707,
+            -0.011376051474975524,
+            0.024577071515205502,
+            -0.012694034026935636,
+            -0.0019809719792138673,
+            -0.016512649783050146,
+            0.015907402258634816,
+            0.004556574687833768,
+            0.004907510649166062,
+            0.011347630984408937,
+            0.009824484201337644,
+            -0.02650731900265866,
+            0.01756773113723491,
+            -0.005671986127099093,
+            0.004449611508541289,
+            -0.027841346572384462,
+            -0.020124233970502267,
+            0.0025986391930124847,
+            -0.0014440758308989809,
+            -0.009579444248630085,
+            0.006005274872617889,
+            -0.0024984235450999127,
+            0.03765664613203687,
+            -0.03765496603611941,
+            0.01409350051861866,
+            0.0066453175608372555,
+            -0.0015390841089599882,
+            0.011806653942640485,
+            0.017306914776093522,
+            -0.008206256939290151,
+            0.02589800448884443,
+            -0.00798213351353373,
+            0.009846274212000523,
+            0.015230070803055296,
+            -0.03449531102975745,
+            0.009174739184806312,
+            0.010145762060174144,
+            -0.024165660737765746,
+            0.012146877603301336,
+            -0.013784371016298891,
+            -0.004818759788054113,
+            0.005930446700613062,
+            0.007349837554764034,
+            0.010733731974181452,
+            -0.02295571464086674,
+            0.016176424826629898,
+            -0.010947841926210105,
+            0.0037766665765835445,
+            0.013559846998199008,
+            0.024892849808979323,
+            0.0012426907454129122,
+            0.0005950515136086193,
+            -0.01379469431504576,
+            0.005534724429523084,
+            -0.015667467221535825,
+            -0.022401586867694584,
+            -0.014192041775234079,
+            -0.006366846766730035,
+            0.01029598687444803,
+            -0.003972386708834858,
+            0.016971476088808737,
+            -0.010309648640697812,
+            -0.010645860303528099,
+            0.0037818310327362103,
+            0.01040806357838676,
+            0.008316605962234849,
+            -0.02855914079113881,
+            -0.005676310556150579,
+            0.004961084138163161,
+            0.00029285971503001265,
+            0.009592497954559435,
+            -0.014556073011576487,
+            0.013689526487160009,
+            -0.013290113354287095,
+            -0.0010726537498214612,
+            0.013201860397060236,
+            -0.012443552043930505,
+            -0.004012305744087929,
+            -0.01592905426965635,
+            -0.028487066798154617,
+            0.008462143660968396,
+            -0.01365614159523485,
+            0.001076882584445294,
+            0.0019503240125505646,
+            0.004730085646042709,
+            -0.016919069570642004,
+            -0.007860046623166481,
+            -0.033515448866521046,
+            0.011065416478687504,
+            0.02314589687882036,
+            -0.006435733825418645,
+            -0.030351727858789472,
+            -0.009546333526852862,
+            0.011327489786114963,
+            -0.0071765122743172715,
+            -0.01676736383051139,
+            -0.0029390038145944096,
+            0.007617910650439855,
+            -0.008174620754097493,
+            -0.00508943241317381,
+            -0.0025173969010414098,
+            0.0217328037301181,
+            0.016094975605203832,
+            0.01769642142712967,
+            -0.021400862603034036,
+            -0.00976992221747187,
+            0.026043879119776864,
+            -0.03275550610951898,
+            0.012441465975914529,
+            0.024587446266013183,
+            0.003446527051428522,
+            0.005492228054838714,
+            0.0030377554109693714,
+            0.005459159163772825,
+            -0.006985179226831048,
+            0.019925583798328282,
+            0.010870421035240845,
+            -0.019709062313471467,
+            0.005385443078244229,
+            0.02086085539375208,
+            0.0037248737640193097,
+            0.012892299156840998,
+            -0.017546870090541213,
+            -0.0020341807891543716,
+            0.015594779287017958,
+            0.04816214672670215,
+            -0.0032851797178091685,
+            -0.010450846098750195,
+            0.029513742590677183,
+            0.023526004023637855,
+            0.005373455940309873,
+            0.0012120777808635214,
+            -0.02525630456877349,
+            -0.013075130701774785,
+            0.01962235086017781,
+            0.016944870494393874,
+            0.013484925533584295,
+            -0.0023255656480479004,
+            -0.019236081983457366,
+            0.021520156296467717,
+            -0.01701733319731161,
+            0.012422659010697477,
+            -0.023764941037007197,
+            -0.001490790833263327,
+            -0.0035419008954878724,
+            0.021998949056452454,
+            0.0021819635638621092,
+            0.008891505304514552,
+            0.024354552127458608,
+            -0.006978548952551799,
+            -0.019526615985982355,
+            -0.0051799020400150305,
+            -0.014578953088901798,
+            0.022506635912517146,
+            -0.024117708614271965,
+            -0.007008641019892289,
+            0.011379795968360019,
+            -0.012362093847309038,
+            -0.0102534051092702,
+            0.001164934138876502,
+            0.017686807394319725,
+            0.0009423993435129515,
+            -0.014784508280754078,
+            0.01944445593807992,
+            -0.0011359271135469456,
+            0.010410915597949927,
+            -0.01921419010977189,
+            0.009390638453648025,
+            -0.016682442516992777,
+            -0.007849184048209236,
+            0.010343008873053859,
+            0.0012604521199837046,
+            0.003948300703645506,
+            0.002592490121754231,
+            0.007773437307969315,
+            -0.03355658426684504,
+            0.022695505487314516,
+            -0.010751606023585792,
+            -0.003538523688786008,
+            -0.0014125330167969218,
+            -0.01678692701672328,
+            -0.0006918162525881059,
+            0.005816502852598842,
+            0.010878358093847386,
+            -0.016045147059003256,
+            -0.00873270513444376,
+            -0.04920244395446931,
+            -0.030314009649638693,
+            0.023156333024481057,
+            0.013733657599720879,
+            -0.010971506342695331,
+            0.018852159415605117,
+            -0.023760853159532587,
+            -0.010587249404421732,
+            0.0011379869499042905,
+            0.023319856329056813,
+            -0.03539290744576506,
+            0.01215369703816749,
+            -0.014493682360886549,
+            -0.02039288669534806,
+            0.009070984596830142,
+            0.011445354735283254,
+            0.0007259852045513916,
+            -0.0011514627555568814,
+            -0.029350945123925453,
+            -0.010246845821727087,
+            0.02391068177425878,
+            -0.00861718589055971,
+            -0.004723044986826333,
+            0.024716520302409263,
+            0.0033721801437248873,
+            -0.012524663490691517,
+            -0.017955758068233754,
+            0.021612671655463236,
+            -0.026077203994996058,
+            0.007645944309444457,
+            0.010032183011253845,
+            -0.0173991263874487,
+            0.021649599230588117,
+            -0.007529701797485131,
+            0.020327763440441377,
+            0.004227390938169872,
+            -0.011249233530251522,
+            0.020352283974471443,
+            0.013644443758970054,
+            0.004216958800094986,
+            -0.004081669404784771,
+            0.006869353098014179,
+            0.02212732880780841,
+            -0.011021739614266063,
+            0.02219580717345023,
+            0.028777213295471067,
+            0.00017328191188631918,
+            0.03260179554743322,
+            -0.021287655520443285,
+            -0.03159728983004563,
+            0.021740163175540124,
+            -0.01035146858119169,
+            -0.021564906188712388,
+            0.028914245422221707,
+            -0.013674731836508768,
+            -0.00786476052825481,
+            -0.0166718429993922,
+            0.00533027311342322,
+            0.016680336311088843,
+            -0.005246190451159961,
+            -0.014360934066628332,
+            0.0029628103834216345,
+            0.019809487418628386,
+            0.014587234544007958,
+            0.029780009648869975,
+            -0.010778199815920887,
+            0.035180242185168895,
+            -0.002320576872571577,
+            0.008990157317473909,
+            0.012415465232073404,
+            0.006489435152972723,
+            -0.011251531176923445,
+            0.013501894212117511,
+            -0.003446759011266526,
+            0.008230548711588447,
+            -0.001449023474280409,
+            -0.035111679577202855,
+            -0.013709382199848678,
+            -0.04709294504969336,
+            -0.0023752976043235697,
+            -0.0025098360125255215,
+            0.0030975417740913154,
+            0.09736815869389318,
+            -0.005723049647039619,
+            -0.011988817571623457,
+            0.001625059666070763,
+            0.009611697638805556,
+            -0.0009896494420056814,
+            -0.007606836952841873,
+            0.015973152482488318,
+            -0.01343292635775299,
+            -0.02140928016038583,
+            -0.010532701190806147,
+            -0.01288594454368977,
+            0.012079198190522715,
+            0.015929716343161652,
+            -0.0005947359502228821,
+            -0.015207331663761012,
+            0.006524010655124294,
+            -0.004566385899752684,
+            0.0064717704997400245,
+            0.008647099975176308,
+            -0.010414668624343924,
+            -0.0071698879102925555,
+            0.030899143952119196,
+            -0.025301269549419674,
+            -0.024361748354052457,
+            -0.0010579931222861095,
+            -0.009055796737486008,
+            -0.010335459737012047,
+            -0.002692653633719985,
+            -0.0017679501322462056,
+            -0.010519994985692081,
+            -0.010019865263564572,
+            0.006537076627706128,
+            -0.015472130358273033,
+            0.004812857776601667,
+            -0.0047457011269495634,
+            -0.013296069496526007,
+            0.017054968086538724,
+            0.022563785357424853,
+            -0.02841491200807702,
+            0.0006755428946168761,
+            -0.017422510783648978,
+            0.012941211381006863,
+            0.008866069010589792,
+            0.006049618782357251,
+            -0.022451528209097505,
+            -0.023622906582403243,
+            -0.015019272754405791,
+            -0.01927535750568903,
+            -0.024973490129133732,
+            -0.009073381957684092,
+            0.0067692803507600945,
+            -0.001267339788153158,
+            0.021918281111033645,
+            0.000816538714279555,
+            -0.01812545613159305,
+            0.0061084913570894905,
+            0.000568371220952269,
+            0.018292497018668842,
+            -0.027468677288882186,
+            0.0033806422195878524,
+            -0.02307582511848465,
+            0.033508234091571124,
+            0.0039195050746594975,
+            -0.026680111514794824,
+            -0.021558668725755275,
+            -0.003311830287529699,
+            0.018994577319066473,
+            -0.0025745912019675805,
+            -0.00724049447539589,
+            0.014775207617625293,
+            0.0155334760323838,
+            -0.004693118176196483,
+            0.002015625406296403,
+            0.027968079002437102,
+            0.01077998866180723,
+            0.0313892866933904,
+            -0.015236169297590675,
+            0.004945828495617267,
+            -0.027215016713379062,
+            0.010214419950212732,
+            -0.01503548137805401,
+            0.008859554812646242,
+            -0.01290890682557532,
+            0.03531384210182567,
+            -0.012005613861257432,
+            -0.004250698950192301,
+            0.03253288796441852,
+            -0.029150986393345252,
+            -0.0035020051536906334,
+            -0.009255437891304041,
+            -0.004142204892160768,
+            0.013240580426083011,
+            -0.001198093874819309,
+            0.008485141356876356,
+            -0.01854370462655459,
+            9.726096864073609e-05,
+            -0.003221852304410318,
+            0.005404595442127598,
+            -0.02005458688172967,
+            -0.003314118809778734,
+            0.0073427568419969814,
+            0.004259518237708634,
+            0.002408630356035373,
+            0.016420646142725346,
+            0.006662121128241274,
+            -0.0029357701493526907,
+            0.03424224243607692,
+            0.008171895850772245,
+            0.0152521424877615,
+            -0.0018638264193202728,
+            0.0030234211261443034,
+            0.019287963516760636,
+            0.019912956661192856,
+            -0.00048443399729935116,
+            -0.01058935153883094,
+            0.00195113175754497,
+            -0.001140732137433033,
+            0.011048741602327437,
+            0.024396046054912645,
+            -0.021158025221644738,
+            -0.02419562092713724,
+            0.016377661005279716,
+            -0.015076145951740961,
+            0.01903712233378994,
+            0.010541463112889985,
+            -0.010099351302095792,
+            -0.00031558616108876247,
+            -0.042316016201379245,
+            0.010964989031040692,
+            -0.02068939391415103,
+            -0.04004120028915868,
+            -0.00244383573783919,
+            0.0009542507389088937,
+            -0.029439456243505052,
+            0.02822814855547978,
+            0.0023755032182188156,
+            -0.017280221740813364,
+            0.018223709137876153,
+            -0.005548654680726505,
+            -0.0024040301239229064,
+            0.005397711787238679,
+            -0.01092026090471121,
+            -0.010648485086769927,
+            0.008235250252660972,
+            0.010957715075632535,
+            0.005425683656535705,
+            -0.005698919020608095,
+            -0.011284911899894062,
+            0.009975882651638084,
+            0.012318901214538077,
+            -0.0019922884373083155,
+            0.010050520101815593,
+            0.0031720752428247446,
+            0.007352859750323051,
+            -0.0073483049386141005,
+            0.0046483587496626965,
+            -0.005093220128411125,
+            -0.026080057085901446,
+            0.012863580653903103,
+            0.0052434400313520774,
+            -0.011863360599637747,
+            0.0103209502958491,
+            -0.01048162756158923,
+            0.005787157766911021,
+            0.001318945460232246,
+            -0.0059333230422061395,
+            0.004821320238893602,
+            -0.0038922239931512784,
+            0.011267634571983924,
+            0.013256611037442042,
+            0.004592529900805719,
+            -0.013076954459595086,
+            -0.02177318635989856,
+            0.009001291224623467,
+            0.01959767163356408,
+            0.024161754192119852,
+            -0.030416109293488064,
+            0.003784634588325551,
+            -0.0014915849060829743,
+            -0.0020550564975080885,
+            -0.015632785947151753,
+            0.01587790981601866,
+            0.01716742056401084,
+            0.012721563202528083,
+            0.01893393006967701,
+            -0.004009617703771589,
+            0.010539100665843704,
+            -0.02080068888791658,
+            0.013642451766699067,
+            0.0008609869904066794,
+            -0.013629067231663643,
+            0.0014620655123815298,
+            0.027046304278766087,
+            -0.011056783825898616,
+            0.006206969699250696,
+            0.008097527175511791,
+            0.022111870736907183,
+            -0.012509479100313492,
+            -0.003735868296238069,
+            0.01516254968167132,
+            0.0014878674796313539,
+            0.0035887079417890352,
+            0.016533149527222273,
+            0.04729052276110659,
+            0.00102751186013391,
+            0.0016713773449738153,
+            0.0192177920520243,
+            -0.005563519063161466,
+            -0.01114821419109643,
+            -0.003549999863325982,
+            -0.021807893906469106,
+            0.010998558936481785,
+            0.00431279785115251,
+            -0.02186324212032519,
+            0.01024093366826206,
+            0.0125495473550584,
+            0.005268320135120931,
+            -0.0031679212410026603,
+            0.02574575247827441,
+            -0.019807879421996258,
+            0.014935295894965128,
+            0.03329450628384787,
+            0.00493356778223407,
+            -0.011423949333937285,
+            0.007263264249871384,
+            0.0027354459678742013,
+            -0.004494406041906684,
+            -0.00606745413660583,
+            0.010497391290734088,
+            -0.008191702034391382,
+            0.009870458191510176,
+            -0.009580130479693,
+            0.039202091799313755,
+            -0.009619018331262844,
+            -0.010782138248435458,
+            -0.004558645539551881,
+            -0.006241773108460662,
+            0.003478414182938609,
+            -0.003395165694642073,
+            0.005812225135633631,
+            -0.0069045407353789945,
+            -0.014607146220271209,
+            0.0008522982757210454,
+            0.00020175973815948303,
+            0.014001922919116725,
+            0.0011804904415274088,
+            -0.004871874532755353,
+            -0.007877217154706205,
+            0.015243277056007988,
+            -0.006047906208304912,
+            0.0006531937033317192,
+            -0.0011653523616313422,
+            -0.0013662556399359335,
+            -0.007401050012179761,
+            -0.015166105795085528,
+            0.00974609244984173,
+            0.016253733908242134,
+            -0.013857834499211056,
+            -0.005142936208363415,
+            0.012274716212832219,
+            0.008540229125193836,
+            0.011742606168187876,
+            0.0007451338650642946,
+            -0.009255813682572937,
+            0.008292531177374192,
+            -0.015020523035432158,
+            -0.008781317965985481,
+            0.0026250469418136603,
+            0.002381147382977664,
+            0.005839723981046825,
+            0.0006240641828668715,
+            0.004502463508605272,
+            0.005798912705218846,
+            -0.00497936644919367,
+            0.01112666823857836,
+            0.008198424171006548,
+            -0.002452143482105151,
+            -0.01017569410837814,
+            -0.001127589984272638,
+            -0.015733298167661346,
+            -0.016758315411028353,
+            -0.0009286266498936771,
+            0.015175988113750882,
+            -0.02352862898718829,
+            0.018996352975391644,
+            -0.0014524363611708201,
+            -0.0033802546171857704,
+            -0.021392428456030312,
+            -0.0025923983911570183,
+            0.0034214188668518796,
+            0.01255768230394476,
+            0.01858853630745928,
+            -0.02729543519465384,
+            0.010575894388563116,
+            -0.007204725245442141,
+            0.030394929424014717,
+            -0.018328782140005044,
+            -0.004022941273610227,
+            0.026915618079621992,
+            -0.02200018306828758,
+            0.0008791185377513978,
+            -0.011454606955158956,
+            -0.008103032693853504,
+            0.019287674837688177,
+            0.023747720480736777,
+            0.02038287242167783,
+            0.01364489294869211,
+            0.0007758044919789085,
+            0.004768030222445397,
+            0.03572705648994403,
+            0.010568241270862248,
+            0.014828087092859878,
+            -0.007542489523110974,
+            9.457808640264811e-05,
+            -0.02554033312739386,
+            0.01635899434369978,
+            2.587383534566272e-05,
+            -0.012370727506722627,
+            0.007162422186621379,
+            -0.013370882403931942,
+            -0.011066483600541058,
+            -0.007151077450730025,
+            -0.01721910018182534,
+            0.02152239849400241,
+            -0.012322440280680164,
+            0.004587729883596212,
+            -0.009079747696669277,
+            -0.013272707286450417,
+            -0.012953682633400515,
+            -0.014202214807187545,
+            -0.01313141988034687,
+            -0.026041403779538307,
+            -0.027109731260915763,
+            -0.006573285295895077,
+            -0.023780366346334524,
+            -0.013799072648505526,
+            0.005588622594136253,
+            -0.022617902666316757,
+            0.012262325666428604,
+            -0.012689827812482627,
+            -0.01226264872059864,
+            0.012805734807835008,
+            0.020274844315616616,
+            -0.011765231803328348,
+            -0.028494489703400308,
+            -0.018583843821043854,
+            -0.019970998470704358,
+            0.012576201137505969,
+            0.002266288538944273,
+            0.013165889075444961,
+            -0.003018437854660793,
+            0.003227412667880239,
+            -0.014895831590298957,
+            0.018570624675927183,
+            -0.03442390703780963,
+            -0.03108003806963536,
+            -0.013611306765348361,
+            0.005182777066476028,
+            -0.0015782033203578553,
+            -0.011429046504256945,
+            -0.0018606648572920801,
+            -0.00433805271182878,
+            -0.025109572024446088,
+            -0.010134856632190067,
+            0.011010662054484161,
+            -0.015205629927469414,
+            0.0072933565255675036,
+            0.0009018832564038863,
+            0.013870902834260887,
+            0.009587432944616442,
+            0.013393301893492882,
+            -0.006095063698384775,
+            0.004356485794113208,
+            -0.0026476612747324386,
+            0.0004894024149759108,
+            0.009336366919474478,
+            0.00839591561027103,
+            0.010551161369079399,
+            0.002260707777260338,
+            0.007830059223001696,
+            -0.0060927446216633265,
+            0.018065002687169148,
+            0.015436944875538481,
+            -0.0002520689442496089,
+            0.013323098485251487,
+            0.0003769862591044255,
+            0.00022293304921127018,
+            -0.0029970303106340466,
+            0.007888437018607987,
+            -0.011785957933642163,
+            -0.001311704406355858,
+            0.015138190893483804,
+            -0.01707247730569978,
+            -0.013045187198628258,
+            0.03107034781283638,
+            0.0018583638488268697,
+            -0.02495564203304796,
+            0.031146554085787762,
+            0.004220047610942943,
+            -0.005835199594223787,
+            -0.013205086624848782,
+            0.0014059047822222418,
+            -0.011570624844100298,
+            0.003584232809362274,
+            0.014295032061446826,
+            -0.02231984223610487,
+            0.010521094801839404,
+            0.008446892441796151,
+            -0.009681535919980559,
+            -0.012337659795586223,
+            0.009847073709209602,
+            0.016809328445200358,
+            0.004472485504034416,
+            -0.03539093810210523,
+            -0.011957694102753411,
+            0.02145470732008716,
+            -0.029237428225009913,
+            -0.00810041956259673,
+            0.007452962400105787,
+            0.0065154832040426065,
+            -0.019353109017609053,
+            -0.008580955020412362,
+            0.005725794605897509,
+            -0.004940390374637084,
+            -0.003892210172017758,
+            0.017541623128534682,
+            0.00485047043626186,
+            0.013698341792101025,
+            -0.01689738578307474,
+            0.009771458057400557,
+            -0.022222817828602557,
+            0.004021882074676753,
+            -0.023373700500461992,
+            -0.009873403167702603,
+            0.002364143609079849,
+            0.012454271200139762,
+            0.014328556467632212,
+            0.027839363143659326,
+            0.030056404238348527,
+            0.013995696971422429,
+            0.007833677585250953,
+            -0.016265199386953717,
+            -0.018134327237616175,
+            -0.006409513840075933,
+            -0.004033173350204784,
+            -0.008714798059218596,
+            0.0066373547199279235,
+            0.01858354212009877,
+            0.004499712802472377,
+            0.02609078206670711,
+            -0.006227189596272876,
+            0.009039744622014585,
+            -3.127526911951914e-05,
+            0.0008833930131827993,
+            -0.004226083827636182,
+            -0.01971685546584527,
+            0.00481003211084742,
+            -0.0013721482162825508,
+            0.025874758880591167,
+            0.00940496860916206,
+            0.0024915013399541847,
+            0.01615978314811047,
+            -0.006378952757397484,
+            -0.01174653607178726,
+            -0.010540107733238588,
+            -0.0061926268398085176,
+            -0.021752582127981104,
+            0.029059100503124387,
+            0.021195595568253612,
+            -0.010800765264123782,
+            -0.013693582166598082,
+            -0.00903369526059487,
+            0.02795767990556847,
+            0.004080965688836953,
+            -0.0013050212744006933,
+            0.016313311850308505,
+            0.012084555958999215,
+            0.012329325712491252,
+            0.009794698207689061,
+            -0.00036157148861933123,
+            0.0015364709122308352,
+            0.03722522141440073,
+            -0.007427028510548186,
+            0.0026463292682538495,
+            -0.005239324411651515,
+            0.0025987932868461313,
+            -0.009705817520422487,
+            0.007242122655628283,
+            -0.004012676385296197,
+            -0.0158760051470016,
+            0.015003444174308607,
+            -0.022443331305169994,
+            -0.0016749787204471238,
+            -0.008150635190952799,
+            0.0016696640640947303,
+            -0.02781023337840043,
+            -0.0025731206607371845,
+            0.006695065701630532,
+            -0.015547340828489069,
+            0.011816010501093885,
+            -0.01740673229051557,
+            -0.011553604323149105,
+            0.007914961618894248,
+            -0.012034157288065447,
+            0.0016684130271324446,
+            -0.0008268957621923278,
+            0.004460864332388434,
+            0.013426055079012884,
+            -0.011008808570221245,
+            0.012166046631826273,
+            -0.005816284310138776,
+            -0.007872091214834688,
+            -0.0016055129487028885,
+            -0.0030049472973942043,
+            -0.0025600746761171543,
+            -0.0009966231170649917,
+            0.034355910234323155,
+            -0.004446570896048543,
+            0.003761242147499512,
+            -0.01362639743248326,
+            0.0019833052855181806,
+            -0.036176376394999044,
+            0.028344464817309477,
+            -0.008642835889481802,
+            0.011013502221008498,
+            -0.0017508365480865596,
+            -0.013243706268954915,
+            -0.01456517309747086,
+            0.015247299501245178,
+            0.007602596572394442,
+            -0.005854531057349665,
+            0.013558248087347848,
+            -0.019724373614448298,
+            0.008530500822910953,
+            0.004463694543554244,
+            -0.015561397976987216,
+            0.004668780190328835,
+            -0.007479912658259757,
+            0.012657201279639194,
+            -0.01774730033362932,
+            0.004314594593150765,
+            0.00958571617187387,
+            -0.01892876084898881,
+            0.009901041800918215,
+            -0.028675879564512306,
+            -0.004631920242208613,
+            -0.012323455524366223,
+            0.0053804608827285865,
+            -0.000660864620708182,
+            -0.02389088252276507,
+            -0.028944280660230762,
+            -0.0002099667627277945,
+            0.00458533068247518,
+            0.0018878351997997833,
+            0.011621407953503755,
+            0.005970807015151215,
+            0.008542958629199003,
+            0.01023896054756221,
+            -0.00205207247588841,
+            0.009402752124876478,
+            0.04169706217414899,
+            -0.022899912321317197,
+            -0.011574970753549647,
+            0.005321872977369877,
+            0.011499271505460519,
+            -0.02647653483798284,
+            -0.011039791758347888,
+            0.0033424345263339144,
+            -0.005219877979550397,
+            0.03609469691834737,
+            -0.02448746405881972,
+            0.019020618652020673,
+            -0.011392297116547224,
+            0.007625475937968714,
+            -0.021465240076078376,
+            -0.003921088264697311,
+            -0.014265745991522477,
+            -0.00011221615798127438,
+            -0.0031478236827989703,
+            -0.0076053733863387755,
+            0.007485039021296698,
+            -0.001907678661303243,
+            0.0020348246022161925,
+            0.006291436821831794,
+            -0.02173305455485444,
+            -0.0019125499969493854,
+            -0.016432460924913166,
+            0.030367263349759286,
+            -0.016242095671439496,
+            0.005072805486919414,
+            0.0053258139728101506,
+            -0.002810639419737258,
+            0.012368771814660942,
+            0.004165967191536186,
+            0.00513754440091394,
+            -0.004199767241827414,
+            -0.005142642360546972,
+            -0.0053886448467479215,
+            0.017868874549760376,
+            -0.013280048598849825,
+            0.035024701466710065,
+            -0.0100959186045929,
+            -0.007094001877261568,
+            0.0019733042325232155,
+            0.010380430710282744,
+            4.2084202709296443e-05,
+            0.01053565670076933,
+            0.0034270489770133685,
+            -0.0006311078385359693,
+            0.0019774779328212756,
+            -0.017162448473652843,
+            0.003089758682621882,
+            0.01682656561693936,
+            0.005463677441664502,
+            0.004469261499451007,
+            -0.013332130891003062,
+            -0.012050233723282738,
+            0.004538268704442613,
+            -0.007533940408255447,
+            0.0003675737140813486,
+            -0.018816953644773076,
+            0.013972696748575243,
+            -0.0011096683686631858,
+            0.0002715683052236706,
+            0.00524678369365015,
+            0.023056514997908218,
+            0.01813167860994747,
+            -0.002553352602551003,
+            0.03155164978556312,
+            0.0174738879054593,
+            0.0029268039949260673,
+            0.010637295711777793,
+            -0.033222781195984666,
+            -0.013301873512972214,
+            -0.0013721703891642354,
+            0.02238421933149517,
+            -0.01530189703335162,
+            -0.014059029663525449,
+            -0.0037393534403749947,
+            0.0010394601933309767,
+            -0.005177359933828505,
+            0.0042489306011644495,
+            0.011039889511148341,
+            -0.014697595240068863,
+            0.007597149702441915,
+            0.0026780496822907634,
+            0.02092734730183164,
+            0.0004688144432243786,
+            0.017181410417231698,
+            -0.0005083970646221274,
+            0.00749639055342427,
+            0.007319420826337891,
+            0.017541463125355265,
+            -0.01002811389296805,
+            0.023582113845081534,
+            -0.008823933397729605,
+            0.0019889258902730535,
+            -0.01257106158422629,
+            -0.028159594067544737,
+            -0.012624750220149823,
+            0.015247564545279736,
+            0.005765911899842925,
+            0.006478299226263732,
+            -0.016537090297144265,
+            0.006276997440048981,
+            0.015647122171767974,
+            -0.002251683151059454,
+            0.010295358869874306,
+            0.009730378299068289,
+            0.00012816405890877098,
+            0.010181120688239766,
+            -0.00801256937783534,
+            -0.00734257767723293,
+            -0.004497520124843868,
+            0.023982754195729983,
+            -0.005443550092842393,
+            0.022434177122972028,
+            0.012647958478747348,
+            0.00616791857573897,
+            -0.010853355357954727,
+            -0.003340111254357573,
+            0.023389634440057473,
+            0.011863246790550376,
+            0.00029642699874398923,
+            -0.009846034060495678,
+            0.0019458876579524617,
+            0.0054010373765486005,
+            0.012983913038716568,
+            -0.02049427270240938,
+            -0.004344571352918878,
+            0.008461947538684664,
+            -0.008472919299762712,
+            0.004318309005722874,
+            -0.030156444522730416,
+            -0.005812303237222271,
+            0.021354352236942434,
+            0.008392281015122372,
+            -0.010232039722494646,
+            -0.004395682831498155,
+            -0.0006527587802367882,
+            0.0040601850269714945,
+            -0.0024323167053212176,
+            -0.0023729138545912176,
+            -0.008541233856441836,
+            -0.006470933419694742,
+            0.014035696617797647,
+            -0.0052691760528871924,
+            0.008597459268477037,
+            0.00012084859681905111,
+            -0.0017255958688825231,
+            -0.007323529232751411,
+            0.03382792341957058,
+            0.008087732655845718,
+            0.005980666974986827,
+            0.0013613625176327891,
+            0.0042244330945601215,
+            0.000616791811672998,
+            -0.0028374856646453707,
+            0.018075360591135542,
+            0.0156738939725117,
+            0.005054696911814988,
+            0.009038525980341447,
+            0.014672270369917403,
+            -0.019700405219230957,
+            0.012625337070556061,
+            0.012184185186281482,
+            0.019382321278525114,
+            0.0007219498737851579,
+            0.002564970959687783,
+            -0.010601873823737926,
+            -0.0029963537053764736,
+            -0.002070567089566093,
+            0.029563028293936092,
+            0.0005903976560069006,
+            -0.010509758568838935,
+            0.010992421945232951,
+            0.020603868018231348,
+            -0.02101157589451638,
+            -0.001971974816402967,
+            0.005963911670920596,
+            -0.010704207628592966,
+            -0.014466113540150018,
+            -0.021269176952661614,
+            -0.013331247081394942,
+            0.0071597345125287174,
+            -0.008162102923896894,
+            0.02367549407145414,
+            -0.014256771777502395,
+            -0.03960796112042958,
+            -0.014803626428839152,
+            -0.00864320347698291,
+            -0.027531337207277483,
+            -0.007305441146142331,
+            0.00965056093889974,
+            0.016698589600806273,
+            -0.003330867073501157,
+            -0.025198703196464594,
+            0.030483790406960656,
+            0.008872821868559751,
+            0.023708422767352917,
+            -0.021464374508407488,
+            -0.0018617153194555344,
+            -0.00886376784514331,
+            0.014505918224188001,
+            -0.017588761818663975,
+            -0.011756112651591795,
+            0.020014632992762026,
+            0.014356419813769714,
+            0.007246557500044434,
+            -0.004355132555323843,
+            0.0067299398462013936,
+            0.007649499235136303,
+            -0.012515181210363267,
+            0.012239792251154642,
+            -0.013499277313916029,
+            -0.024858295126228433,
+            0.014430561986516884,
+            0.002229494935179828,
+            0.009563004428064055,
+            -0.0015984171085659323,
+            -0.016197659608768088,
+            0.010078159490026602,
+            0.00808729175515505,
+            -0.017508840440512225,
+            -0.004793934415922065,
+            0.007133527615305415,
+            -0.020818869260580315,
+            0.031234686259160864,
+            0.011872765152279094,
+            -0.010692949770503467,
+            -0.001989039629661815,
+            0.004431079028880923,
+            0.016511544608975148,
+            0.007893700385963683,
+            -0.01997314388229276,
+            -0.003708006454566117,
+            0.01530128306203401,
+            0.00822588305477249,
+            0.004861703074728471,
+            -0.010836328783594584,
+            0.03434874763691911,
+            0.006082873924472328,
+            4.192481378506604e-05,
+            -0.0052997095110967115,
+            -0.0076108924981838355,
+            0.004196112457289409,
+            0.018905391806961686,
+            0.013330078288599763,
+            -0.004200590783114919,
+            -0.006630107527854746,
+            0.007778448128358027,
+            -0.01706991160178345,
+            0.007329056462125937,
+            0.009390327499494826,
+            -0.012870674959022891,
+            -0.0017571189976160295,
+            0.016462608870101713,
+            0.00209846661139665,
+            0.004942714850185376,
+            -0.011305050739130339,
+            0.004252776146827917,
+            0.015841821944109317,
+            0.018727567181082646,
+            -0.008399642221255733,
+            0.021448055799259867,
+            -0.020972459569228434,
+            0.027243639465535856,
+            0.028499400850980086,
+            0.016573144403275928,
+            -0.01847096443459798,
+            -0.0022409373140116146,
+            -0.0041737366139895414,
+            -0.014761716560267328,
+            0.0034546020161337894,
+            0.005125870074458691,
+            -0.01174186608185822,
+            -0.009577921232380895,
+            -0.024331993794416026,
+            0.011075959943215117,
+            0.007918147902741738,
+            -0.0015045185909419564,
+            -0.011791155919533844,
+            0.016019077577552835,
+            0.010141868950981925,
+            -0.010938402325619172,
+            -0.005635351352447256,
+            -0.019967361901734836,
+            0.009750427567651833,
+            0.009128294017175602,
+            -0.00300556760891084,
+            -0.011496103417643132,
+            0.012977890524465716,
+            0.0014080059843484115,
+            -0.01292380658817521,
+            0.010841954636959441,
+            -0.03142011651894961,
+            -0.005510537354374977,
+            -0.011027026734432255,
+            -0.003620482919145167,
+            -0.014415606143566519,
+            0.003608572766363589,
+            0.0005034611321129689,
+            -0.011470488351158329,
+            -0.0035005774696278,
+            -0.016207453351921627,
+            0.004676327282262429,
+            0.015021513761697039,
+            0.001679614645967629,
+            -0.006917375094634678,
+            0.021994274901326064,
+            0.018518524483306403,
+            -0.021903538417919984,
+            -0.020116013379669034,
+            0.004708084983787958,
+            -0.03261690161800251,
+            0.015019348187107588,
+            0.00966124471577622,
+            0.011676937880751252,
+            -0.028768229226408037,
+            0.007114534979415504,
+            0.00900420165933673,
+            0.016398927481511824,
+            0.0050879301160439245,
+            0.009453255486450082,
+            0.01222758288108498,
+            -0.0104657614414275,
+            -0.003164502778525433,
+            -5.410382753651127e-06,
+            -0.005618225986727351,
+            0.015607620507415484,
+            0.0008527697935056332,
+            0.03603666488542239,
+            -0.005277207843139309,
+            -0.0009799879252482288,
+            -0.011735731235560851,
+            0.00812046801749312,
+            -0.011713349784345638,
+            0.006334172347602515,
+            -0.0024643499727789167,
+            0.0001240009282616789,
+            0.002965067818077541,
+            0.009442572075213767,
+            -0.01086095783353607,
+            -0.007595379345224076,
+            0.007444897256325813,
+            -0.01482814340478338,
+            -0.00019984717819268013,
+            -0.009339343730980451,
+            0.0019680804106275156,
+            -0.02241031895876346,
+            0.006629838013824249,
+            0.040935644507527846,
+            0.02140216465404127,
+            -0.005481040712269146,
+            -0.012431659209559418,
+            -0.004680230194268222,
+            -0.007229300034633229,
+            -0.006482401275991039,
+            -0.004794698160419071,
+            -0.022762405726589375,
+            -0.006928553505112732,
+            0.004320321244630314,
+            0.013300138724212552,
+            0.009741018117644266,
+            -0.008842135469729511,
+            0.004531760574809748,
+            -0.02662650543210291,
+            -0.016578081411548973,
+            -0.0060021425889846495,
+            -0.015840130031005147,
+            0.006125301421471048,
+            0.027521841635711787,
+            -0.0063432924325983525,
+            0.01789356321527744,
+            0.005142029893259461,
+            0.013034833065163448,
+            -0.005338869351330258,
+            0.00035250243782976347,
+            -0.006010938630154189,
+            0.0032958656042048292,
+            0.003074861351445954,
+            -0.0015947217852680714,
+            0.004151590621941346,
+            -0.007139285822575166,
+            -0.0026091937868967284,
+            0.0011629380311555433,
+            0.0117841884590861,
+            0.006877419488576261,
+            -0.00016252653621991807,
+            -0.01761285866765538,
+            0.022221104524808997,
+            -0.018933508559881868,
+            -0.008840161102111973,
+            0.011092684029552064,
+            -0.009660487856185668,
+            0.00702597023411946,
+            0.020387386977378873,
+            -0.006854213562767843,
+            -0.009583981919303215,
+            0.013028181087649146,
+            -0.0036507672754997286,
+            0.004464368543493108,
+            0.009867428978560914,
+            0.006036303973414791,
+            0.002482944923100156,
+            0.013244650094892797,
+            0.004530294330688855,
+            -0.0032625482538717137,
+            -0.020238533752799943,
+            -0.005186498720507101,
+            0.019015752262933585,
+            -0.02252458872825473,
+            -0.0027641215988942563,
+            0.0044506144202026456,
+            -0.0030317840657635457,
+            -0.01733962766050954,
+            0.004205861489436576,
+            -0.018770275720867473,
+            -0.0033883519277721433,
+            -0.008244668812709337,
+            0.009948090119054079,
+            -0.023700732627507225,
+            0.009109315916173436,
+            -0.011739192106106307,
+            0.02356509175177699,
+            -0.003925851077921686,
+            0.024734433836185024,
+            0.0015142645677087268,
+            -0.013482933030628261,
+            0.0055323884713736554,
+            -0.0008930661843169134,
+            -0.004477731568313757,
+            0.029338490404300612,
+            -0.005418258701133154,
+            0.002249996261871466,
+            0.013491124734151694,
+            -0.021332541202417246,
+            0.021773687217263173,
+            0.0046662696076723836,
+            0.01628697326539432,
+            -0.00986238951830334,
+            -0.013799209313714441,
+            0.025178874553620743,
+            0.03324555035648896,
+            0.008398796308070151,
+            0.011261509865513953,
+            -0.01226890254177634,
+            0.018966175575083295,
+            -0.0038328997984309597,
+            0.004344837325292706,
+            0.00878668880458693,
+            -0.007200842513742962,
+            -0.011500562922303248,
+            -0.004533014670236755,
+            0.027639894892383943,
+            -0.021838848944317424,
+            -0.008920629621611768,
+            -0.005739181836252042,
+            -0.01593272996094732,
+            -0.038167534267554606,
+            -0.004371701247332816,
+            0.019604405563120094,
+            -0.005974358141737469,
+            -0.0018341151594086288,
+            0.014334266690716505,
+            0.00402620072221496,
+            -0.002852569009185304,
+            -0.008195218707838913,
+            -0.0027944153624309856,
+            -0.0059390492900516225,
+            0.016651206007022145,
+            -0.004961864528562493,
+            -0.007929830585906699,
+            0.01145431100211762,
+            -0.005731957927601062,
+            0.012130694936714224,
+            0.0004848418056781251,
+            8.392048424584131e-05,
+            -0.0015174120840537315,
+            0.00822932723013239,
+            0.018620869354789082,
+            -0.0055803454666417465,
+            0.025682778991091174,
+            0.006956936033581638,
+            -0.003495709501259274,
+            -0.005489431577052735,
+            -0.0044154293119170445,
+            -0.024561016942073285,
+            -0.0020864636079347357,
+            0.006593060072668424,
+            0.024333813203596122,
+            -0.02515989187381735,
+            -0.004953266000463339,
+            0.020957806164031402,
+            -0.0012192724005613,
+            -0.011978722479596745,
+            0.014690559986694138,
+            0.0041346025614047875,
+            0.009881584250499358,
+            0.012176393203318002,
+            0.015764530703333238,
+            -0.004923548157246672,
+            -0.0015192775000207833,
+            0.012766482928380375,
+            0.0005745973040874939,
+            0.00997809510696241,
+            0.016495444272181515,
+            0.010467514633877887,
+            -0.021108281220544853,
+            0.0015078843789125156,
+            0.00522593579521476,
+            -0.017481430635971396,
+            0.0034362821987740755,
+            0.005397819166037461,
+            0.011626527721895694,
+            0.004048632382026344,
+            0.013207335578216437,
+            -0.015768003434275563,
+            -0.005916168437120099,
+            0.008153798523776654,
+            0.004319451296835019,
+            -0.02599474886669325,
+            -0.01485009085106089,
+            -0.0006781964116146299,
+            -0.004444144921332929,
+            0.013556265879295017,
+            0.01285768162554067,
+            0.02033626001285354,
+            0.024637349990579514,
+            -0.013833060730799217,
+            -0.00842819657792762,
+            0.009436072455719896,
+            0.015560357465711765,
+            0.005798181411384387,
+            0.007874051721372273,
+            0.009297440071118126,
+            0.0077320823375042,
+            -0.004073209576661068,
+            -0.002973928843567532,
+            0.020205003097225747,
+            -0.019725382393637726,
+            0.0006746659838642147,
+            -0.018417443693472958,
+            0.007530796896254817,
+            0.01140696723506382,
+            0.01129515042745398,
+            0.002816883643872417,
+            -0.005274757308627299,
+            0.005716917047812153,
+            -0.06254973656832136,
+            -0.009191453047502128,
+            0.003384751551032943,
+            0.027256466708746898,
+            -0.010885336140129142,
+            -0.003415707910562931,
+            0.007309577090125959,
+            0.005511970708769456,
+            0.0005572239548625122,
+            0.003186427278034852,
+            0.019698716902282903,
+            -0.02052581021967601,
+            -0.010793968505805883,
+            0.004219269922138548,
+            0.0033111837923746746,
+            -0.007269521534951044,
+            0.004805737092302134,
+            0.014143978208501908,
+            0.01986656063456352,
+            -0.0024048854610842748,
+            0.01629335842915328,
+            -0.0015498603704589015,
+            -0.014496994444493684,
+            -0.002671064325909516,
+            0.0037875833047555925,
+            -0.022466433279066646,
+            -0.005980897597195858,
+            0.0043262697442044435,
+            -0.011226121648007667,
+            0.006978495397818936,
+            0.008452177557316863,
+            -0.003018545465557218,
+            -0.0008303307644324117,
+            0.017796789945838845,
+            0.006591364879905198,
+            -0.013464486184025026,
+            0.0028668426952326957,
+            -0.023377605222213174,
+            0.004325185794796361,
+            0.001416245546006914,
+            0.007809449561523174,
+            -0.008431335139468173,
+            -0.014774499666285603,
+            -0.005933810712528226,
+            0.017746847020610673,
+            -0.01965188730040385,
+            -0.002406875192894708,
+            0.01683009662984443,
+            -0.0002518220110313279,
+            0.005456812494894182,
+            -0.02011241622618092,
+            -0.0009782974741561381,
+            -0.02933296756078644,
+            -0.01171823509606186,
+            0.001162345317872531,
+            0.00892221099991475,
+            -0.006551308022163148,
+            -0.03934356275100932,
+            0.00427623835776202,
+            -0.013111664387776261,
+            -0.008752857644809231,
+            -0.010214546496134218,
+            -0.00518087722852466,
+            0.015240297918667867,
+            -0.012498100033235827,
+            0.01086490299272505,
+            -0.00917777212602538,
+            0.005895055740265625,
+            0.0090554651297836,
+            0.014212841798768113,
+            -0.005544047915369086,
+            -0.0033589999440643385,
+            -0.007924830667307956,
+            0.0022291797354234005,
+            -0.0032976226056924502,
+            0.009615674261248298,
+            -0.0051932155872883,
+            -0.0065636389655770455,
+            0.02426106498596339,
+            0.011445591067574932,
+            0.013312596099532533,
+            0.0055118624272110965,
+            -0.007220620072349783,
+            -0.01916551187613823,
+            -0.003977616240384473,
+            -0.013204506708246246,
+            -0.006315966975649096,
+            0.01434099854765068,
+            0.012588249778961223,
+            0.006357972880218866,
+            0.0007747002406996095,
+            -0.004794439624135275,
+            0.006975593520375694,
+            0.0262335496080837,
+            -0.016481909971520166,
+            -0.0013084662210086592,
+            -0.01369313900978409,
+            0.01921748474212319,
+            0.03333761672336437,
+            -0.0008918419419301244,
+            -0.007699525186523558,
+            0.013757216065348164,
+            0.00819045303047094,
+            -0.007411380297164911,
+            0.02453611080891859,
+            -0.025045462880463647,
+            -0.015989319900023054,
+            -0.011650886878708265,
+            -0.011407253985176972,
+            -0.008386367959227559,
+            0.016331961716602287,
+            -0.00141225438918211,
+            -0.0018505754298268654,
+            0.009557291617446645,
+            0.03357924917013024,
+            0.006916860901290756,
+            0.007030308953738567,
+            -0.002739345982346728,
+            -0.009376522438071562,
+            0.011482637968411055,
+            -0.007259054391879614,
+            -0.04075661435928125,
+            0.004250195164583401,
+            -0.005160505473755424,
+            0.02405623169060588,
+            -0.02197586561375289,
+            -0.00619175761282745,
+            -0.006736129049831973,
+            -0.028200015908139763,
+            -0.005867722104196303,
+            0.02805071895018145,
+            -0.015402198567775478,
+            0.014773294766214671,
+            -0.009502185346558072,
+            -0.0013419858580438616,
+            0.004443749010136947,
+            0.013008783570355656,
+            -0.02289662952366337,
+            0.0013834888740540108,
+            -0.00048549197205067054,
+            0.027969842651517314,
+            0.010437729090975158,
+            -0.0047705656249623744,
+            0.014202218020412936,
+            0.008240861154198411,
+            -0.018637315941614406,
+            0.007314019317343154,
+            -0.005747966765101792,
+            0.014833896739762147,
+            -0.01314847940891154,
+            0.007762100494499689,
+            0.00705026393499352,
+            -0.01278023669088385,
+            0.03016342404000438,
+            -0.015204129985039077,
+            0.007776946006562982,
+            0.009278778571285689,
+            0.009656723862179986,
+            0.01257686993982689,
+            0.004362439504889069,
+            -0.02898418193036562,
+            0.0002144431781454467,
+            -0.03828100004144633,
+            -0.00833748467467171,
+            -0.002611937107022567,
+            -0.002145329666219461,
+            0.007050630951707937,
+            -0.009383645162261629,
+            -0.033976442057557676,
+            0.0006429978367640709,
+            0.01603027598131823,
+            0.017501067413108286,
+            0.0002196997337451293,
+            0.013075379412332973,
+            -0.006197537539554383,
+            -0.021148120787551232,
+            -0.016986448660412497,
+            -0.010207705210051413,
+            -0.02122194744719993,
+            -0.020550470201191853,
+            -0.00908564461857315,
+            0.012090079127547193,
+            -0.009927390005879807,
+            0.013888458967497627,
+            0.029513250508042935,
+            -0.024529175134769883,
+            0.0011144447737101862,
+            0.021977677894839922,
+            -0.00198308986575084,
+            0.014108189311291326,
+            0.001696775006906642,
+            0.014122064251288995,
+            -0.0004166140484018266,
+            -0.010556016685519523,
+            -0.009729362492661684,
+            0.017487929296614076,
+            -0.0026043257634797124,
+            0.016715798948897194,
+            0.012183487338360088,
+            0.003771721868999568,
+            -0.0030887189945795796,
+            0.010667776697461199,
+            -0.026471157050402344,
+            0.014249996582064706,
+            -0.006690961709923224,
+            -0.0181382691440242,
+            -0.008729104191096461,
+            0.003882731567105777,
+            -0.005609384377642956,
+            -0.007794812949125115,
+            0.012967089730551055,
+            -0.00944374111323783,
+            0.02088525981049842,
+            0.0023400121055514958,
+            -0.012196925240774426,
+            0.003648688547842322,
+            -0.03322041209304968,
+            0.0015191765655963997,
+            -0.018402382776013956,
+            -0.024659631472836927,
+            0.03840173429666435,
+            0.011991452465253198,
+            0.006639702644606365,
+            -0.01675838115087676,
+            0.007953102302963526,
+            0.0004257479565099406,
+            0.017484338819046517,
+            0.011068864013925452,
+            0.006466915839538296,
+            0.008364454017976605,
+            -0.023316137291380285,
+            -0.010898421318668592,
+            0.016204569093730564,
+            0.012099473359791148,
+            -0.01017229369980751,
+            0.019431669546223253,
+            0.017955119269367787,
+            0.007300372071724306,
+            -0.006898619924677706,
+            0.01647927363640854,
+            0.021012103522831126,
+            0.010291016618557653,
+            -0.00047955552103474145,
+            0.0008684297441778696,
+            -0.010534247928997246,
+            0.005037930235889528,
+            0.035417626740307205,
+            0.014712595692093886,
+            0.010650306363963653,
+            -0.03626584475133921,
+            0.021764456453493054,
+            -0.011961521673661574,
+            0.009092981192874177,
+            0.006089104887183293,
+            0.017982801101846038,
+            0.006916969397519662,
+            0.014196254958707752,
+            0.011745069030657836,
+            -0.006689305450954901,
+            0.025205280281804708,
+            -0.001601414806100461,
+            -0.011992525142592405,
+            0.003386930324498371,
+            -0.01859193251438437,
+            0.005303261319773131,
+            -0.0041710536965100615,
+            -0.004459393120257307,
+            0.0032461828144552565,
+            -0.0018650956576455287,
+            0.015678225251799887,
+            0.003894142628084471,
+            -0.02355212701026675,
+            -0.0014873755115206368,
+            -0.024284295126348117,
+            0.0027260592623277375,
+            -0.022185571998195344,
+            0.00022883605560765203,
+            -0.005557568353382136,
+            -0.009749382813201403,
+            -0.025385857431333838,
+            -0.009853073898348951,
+            -0.0035843255959309307,
+            0.007661137888845632,
+            -0.010521037433807653,
+            -0.004246698084422667,
+            -0.0019920499321964815,
+            -0.019465113363548715,
+            0.009014255846983305,
+            0.027122533756959888,
+            0.016848221859975445,
+            0.027170883580103264,
+            -0.00019554112591107156,
+            0.013614675849485926,
+            -0.002066822813103284,
+            -0.01603063005944364,
+            0.008824663429777616,
+            0.0075653522381218495,
+            -0.00531780740880296,
+            0.014643348459785112,
+            0.0072634484937344704,
+            -0.012800149015319017,
+            0.009859503660384916,
+            -0.007161712176469609,
+            -0.006437857349742842,
+            -0.009764856421740622,
+            -0.02058868349990357,
+            -0.014866137971603676,
+            0.0012450207572602557,
+            -0.002123265406747006,
+            0.006000473862248991,
+            -0.004054016790850226,
+            0.009584879012937315,
+            -0.00608816807208866,
+            -0.00925801057439376,
+            -0.013962683072792106,
+            -0.008924356118158748,
+            -0.004210064145949253,
+            -0.004381709838058668,
+            -0.011732699976353612,
+            -0.005479523097469261,
+            0.0010680258077037758,
+            -0.0019401640391035755,
+            -0.01397652201595301,
+            0.011650839243603377,
+            0.0058731838291845645,
+            -0.002462743134993977,
+            0.02219008688925881,
+            0.01097316136828334,
+            0.0018083001685932406,
+            0.01366095798812196,
+            0.0003942601242216818,
+            -0.0064063532958756175,
+            -0.03343312148366622,
+            -0.0005083355836126291,
+            -0.02815586367527057,
+            0.025770078162165123,
+            -0.0069114949102666145,
+            -0.010322087105114614,
+            -0.002519047562374146,
+            -0.006063669644021698,
+            0.005873609611264377,
+            -0.002722940652283699,
+            0.0021229525800901114,
+            -0.023673778230768945,
+            0.0028629715379660113,
+            -0.003953653437464966,
+            -0.0036366957271968823,
+            0.007469664864329332,
+            0.0346520690389124,
+            -0.004054282838167587,
+            -0.0034765253405897676,
+            0.005159466707114532,
+            0.011642913651743502,
+            0.027845571220425726,
+            -0.004445196354520585,
+            -0.0032038508043339413,
+            -0.02233838584152591,
+            0.0054112576531374175,
+            -0.00019544838850471866,
+            -0.004733192299951088,
+            0.016664217348411003,
+            -0.016719175418727594,
+            -0.01631658832047864,
+            -0.00545969677717507,
+            -0.014570848053102338,
+            -0.00022620258237138677,
+            0.014454101124666536,
+            0.011056312437637014,
+            -0.004035526363810078,
+            -0.012459378363360557,
+            -0.0022595493558020706,
+            -0.030370952502579162,
+            0.012766010284884204,
+            0.1141689619126009,
+            -0.00766197047210299,
+            -0.00896655658602649,
+            -0.009481112393590025,
+            0.001435269335646347,
+            0.001338755472877713,
+            -0.011739143028767118,
+            0.07166096084053437,
+            -0.0077855767070652395,
+            0.0002139580430342728,
+            -0.011312198831181402,
+            0.001844005266872301,
+            -0.001825412693163496,
+            0.01371932421024605,
+            -0.005025814503687022,
+            -0.018213360991353643,
+            -0.0033612481616769416,
+            -0.02941403830641238,
+            -0.0041137865872933515,
+            -0.005683630256743215,
+            -0.004288198570156578,
+            -0.014982999363445122,
+            0.0067149665791261335,
+            -0.019661973497372313,
+            -0.012688556129040908,
+            0.001124391650175233,
+            -0.0033623453063430534,
+            0.00996944383866679,
+            -0.020496336814603385,
+            -0.0044354321069609536,
+            0.0019875840782527733,
+            0.009706675240563883,
+            0.0011600364428219214,
+            -0.012755178230821602,
+            -0.0026514442584739412,
+            0.004056350166031111,
+            0.01909951327913368,
+            0.014360281326789544,
+            0.003732344102045803,
+            0.008623457695718206,
+            0.014461868085333023,
+            -0.001692149528228451,
+            0.01948035376572433,
+            0.01172550239477637,
+            0.0031966531144076173,
+            0.0004779413449255626,
+            0.0030912562273611517,
+            -0.004597633235071031,
+            0.01582547400214653,
+            -0.014811177637844808,
+            -0.02411202772171917,
+            -0.0071272238837422185,
+            0.018015541137524113,
+            -0.013081287852028074,
+            1.8837689271256874e-05,
+            0.004062803284646432,
+            0.027569726868200254,
+            0.0010564850664255057,
+            -0.01179755720247972,
+            0.0028876134800690334,
+            -0.01638215925131053,
+            0.022944183106912663,
+            0.0026043713064243836,
+            0.03279173899194346,
+            -0.005103903273968361,
+            0.00578213669463383,
+            -0.020744194169889544,
+            -0.007588793365216579,
+            -0.00819862611489168,
+            0.0017401442433921013,
+            0.010993500174794084,
+            0.005411238350013134,
+            0.01620925462604661,
+            -0.00641615069699833,
+            -0.020751730783437995,
+            0.014776878596707538,
+            -0.009552193304486048,
+            -0.03393319554458277,
+            0.008881987295131347,
+            -0.029102297123590998,
+            -0.0045267103602711,
+            0.00368971742585232,
+            -0.014269341759285646,
+            -0.016264809810970175,
+            0.012837302095591106,
+            -0.0011890334180728564,
+            -0.008937992768323412,
+            0.0014444613725248846,
+            -0.017110187133760626,
+            -0.02385668037818897,
+            0.006017405364919834,
+            0.00676670672597817,
+            -0.0077994500447595,
+            0.018312452858615755,
+            -0.027015660037024585,
+            0.006123912473734307,
+            -0.012973676426738582,
+            0.009041974762537394,
+            -0.01580183887345168,
+            0.01163516332546347,
+            0.010887662470499361,
+            -0.001503467664709027,
+            -0.0076234124970223326,
+            0.01231202541338883,
+            0.004852720134894811,
+            0.033760365210093034,
+            0.028658326381800395,
+            0.000995999783627898,
+            -0.0025605367058226737,
+            -0.0020356794893366384,
+            0.011667525096213252,
+            -0.014049700287539817,
+            0.011218246246280464,
+            0.02658113044439717,
+            0.017568631325838108,
+            -0.019133062654008885,
+            0.018050678772708717,
+            -0.010356118256625172,
+            0.012872052420999813,
+            -0.025620154415048252,
+            -0.005135768485044442,
+            0.004936490920616527,
+            0.005163008729256001,
+            0.007425278892536464,
+            0.001410212306439329,
+            0.021891387289347516,
+            0.014999616600766989,
+            0.00027613811156650413,
+            0.00558422822386443,
+            0.01477009526981092,
+            0.010497782599293994,
+            -0.0005567894749233204,
+            -0.00467776506604011,
+            -0.016998035358377343,
+            0.00957271797745408,
+            -0.006065116857898469,
+            0.005138070401846389,
+            0.01698914855850691,
+            -0.01230257809343985,
+            0.011109844802663414,
+            0.007518553660463051,
+            -0.015504773927595823,
+            -0.017169933945839305,
+            0.005552829003043926,
+            -0.02252633790984703,
+            -0.01306722099657118,
+            0.02715689072730398,
+            -0.010513941399990838,
+            -0.02790582732974509,
+            -0.0009847768069205477,
+            0.009797865241108237,
+            -0.002772952073769185,
+            -0.0027968741526000576,
+            0.0037541733651479613,
+            0.010801137722284016,
+            -0.014658156970099418,
+            0.002075415233345957,
+            0.004735264309137368,
+            0.01930804315461151,
+            -0.0014151522239496115,
+            -0.015046118414100617,
+            -0.002288495894227231,
+            0.021592829677937263,
+            0.017174609713578173,
+            0.00265560378681856,
+            -0.016499825201963983,
+            -3.965296928410772e-05,
+            -0.0074142747014567635,
+            0.00021360696629503726,
+            0.00027431174904249525,
+            -0.0056391089282546054,
+            -0.018931681700770225,
+            -0.003888248121485032,
+            0.011181675419027736,
+            -0.00247252920175839,
+            0.01977860312223846,
+            -0.0011527575128223392,
+            -0.015763587433200915,
+            0.001960734212846797,
+            -0.005280860341480037,
+            -0.010372829570478649,
+            0.01351900743232294,
+            -0.009359942374603997,
+            0.006788746813046239,
+            -0.003212015225284119,
+            0.004611143085183367,
+            -0.0003308099698088422,
+            -0.017017265064185255,
+            0.02924142709831441,
+            -0.0018238549856433466,
+            -0.004769274041546295,
+            -0.0014058363976586562,
+            -0.033393331166741945,
+            0.015017165991303838,
+            -0.014574579250749552,
+            0.006745699409061803,
+            0.002705853802154516,
+            0.01272741296341735,
+            0.003253095246601883,
+            -0.008898616461779179,
+            0.0027485606928249326,
+            0.017040360927723885,
+            0.013756402798365262,
+            -0.007292864293090161,
+            0.008694855886257706,
+            -0.0028934389922887676,
+            -0.0037483208383542015,
+            0.01363963339139356,
+            0.004170611039133969,
+            -0.006282657671540625,
+            -0.0009784372731140945
+          ],
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "Mixture-of-Experts (MoE)",
+            "fine-grained expert segmentation",
+            "shared expert isolation"
+          ],
+          "title": "Deepseekmoe: Towards Ultimate Expert Specialization In Mixture-Of-Experts Language Models"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "1ea6fc21-365a-41fe-9b07-1e5411ca0bc9",
+        "properties": {
+          "page_content": "4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n",
+          "title": "Validation Experiments",
+          "embedding": [
+            -0.010754532180726528,
+            -0.023697586730122566,
+            -0.02424471639096737,
+            -0.004748920910060406,
+            0.0029044931288808584,
+            0.014695581048727036,
+            -0.013233717530965805,
+            0.03884626179933548,
+            -0.0167387705296278,
+            0.032195206731557846,
+            -0.028296900913119316,
+            0.003022040706127882,
+            0.033374954015016556,
+            -0.032622650265693665,
+            -0.007941939868032932,
+            0.013165325857698917,
+            0.035289913415908813,
+            0.03330656513571739,
+            0.011327309533953667,
+            -0.037751998752355576,
+            0.04247099906206131,
+            -0.036520954221487045,
+            -0.060047563165426254,
+            -0.011079391464591026,
+            -0.019577009603381157,
+            0.02089354395866394,
+            0.00764272827655077,
+            0.016584891825914383,
+            -0.0636381059885025,
+            0.03330656513571739,
+            0.00491989916190505,
+            0.023150455206632614,
+            0.0002941360289696604,
+            0.008754086680710316,
+            -0.011498288251459599,
+            -0.03205842152237892,
+            0.022381054237484932,
+            -0.0009312971960753202,
+            -0.015362396836280823,
+            0.042607780545949936,
+            0.03381950035691261,
+            0.01822628267109394,
+            -0.023372728377580643,
+            -0.004340710584074259,
+            0.016832809895277023,
+            0.00703148078173399,
+            -0.013173874467611313,
+            -0.010062070563435555,
+            0.015704352408647537,
+            -0.04240260645747185,
+            -0.03546088933944702,
+            -0.011600874364376068,
+            0.008172760717570782,
+            -0.01200267393141985,
+            -0.03761521726846695,
+            0.007856450974941254,
+            0.016140347346663475,
+            0.026159673929214478,
+            0.04093219339847565,
+            0.015439336188137531,
+            0.027305228635668755,
+            -0.018602434545755386,
+            -0.019765086472034454,
+            0.010164657607674599,
+            0.0010231980122625828,
+            -0.03443502262234688,
+            0.0016926847165450454,
+            -0.011720559559762478,
+            0.017046531662344933,
+            -0.034383729100227356,
+            -0.04520665109157562,
+            0.02380017377436161,
+            -0.009292668662965298,
+            -0.010993901640176773,
+            -0.0009270227164961398,
+            -0.010198852978646755,
+            0.031203532591462135,
+            0.027544597163796425,
+            -0.0026907704304903746,
+            0.000909390626475215,
+            0.014806716702878475,
+            0.02405664138495922,
+            -0.023714683949947357,
+            0.0028040434699505568,
+            0.04373623803257942,
+            -0.0030434129294008017,
+            -0.02814302034676075,
+            -0.06527949869632721,
+            0.06415104120969772,
+            -0.0167815163731575,
+            -0.013644064776599407,
+            0.0002978761913254857,
+            0.023646293208003044,
+            -0.019577009603381157,
+            -0.029237281531095505,
+            0.05844036862254143,
+            -0.007394809741526842,
+            0.0007074225577525795,
+            -0.018619531765580177,
+            0.014601543545722961,
+            -0.009771407581865788,
+            0.010865667834877968,
+            -0.008617304265499115,
+            0.014422016218304634,
+            0.032622650265693665,
+            -0.024877335876226425,
+            -0.014447662979364395,
+            0.012831917963922024,
+            0.013712456449866295,
+            0.10628008842468262,
+            -0.016217287629842758,
+            -0.005920122377574444,
+            -0.03257135674357414,
+            0.0011006725253537297,
+            -0.009745760820806026,
+            -0.05006243288516998,
+            -0.026826487854123116,
+            -0.025373173877596855,
+            0.031237727031111717,
+            -0.008035978302359581,
+            0.033836595714092255,
+            0.032434575259685516,
+            0.045035671442747116,
+            -0.0132422661408782,
+            -0.04804489016532898,
+            0.046950630843639374,
+            -0.0051549943163990974,
+            0.000823367154225707,
+            -0.0025347527116537094,
+            -0.003041275776922703,
+            -0.0017194000538438559,
+            -0.007373437285423279,
+            -0.01822628267109394,
+            -0.01443056482821703,
+            0.011831695213913918,
+            0.0043449848890304565,
+            0.034298237413167953,
+            0.053003259003162384,
+            -0.04373623803257942,
+            -0.009685918688774109,
+            0.026792293414473534,
+            -0.029938293620944023,
+            0.010985353030264378,
+            0.006091100629419088,
+            -0.02403954230248928,
+            0.015011890791356564,
+            0.0010638053063303232,
+            0.055465348064899445,
+            0.0003606572572607547,
+            0.02374888025224209,
+            0.0011113586369901896,
+            0.04274456575512886,
+            -0.008360836654901505,
+            0.04363365098834038,
+            0.010592103004455566,
+            0.0010258695110678673,
+            -0.020927738398313522,
+            0.018602434545755386,
+            -0.039427585899829865,
+            -0.000900841667316854,
+            0.03171646595001221,
+            -0.0211500097066164,
+            -0.009899641387164593,
+            0.012660940177738667,
+            0.004398415796458721,
+            0.06869906187057495,
+            -0.01395182590931654,
+            -0.0074888477101922035,
+            -0.0781712606549263,
+            0.04982306435704231,
+            0.023971151560544968,
+            0.00814711395651102,
+            0.005907298997044563,
+            0.04182128235697746,
+            0.003832050133496523,
+            0.01309693418443203,
+            0.012438667938113213,
+            -0.0098397983238101,
+            -0.05026760697364807,
+            0.03207552060484886,
+            -0.0026587119791656733,
+            0.012908858247101307,
+            -0.009130238555371761,
+            0.026860684156417847,
+            0.05505499988794327,
+            0.022021999582648277,
+            0.01165216788649559,
+            0.027476206421852112,
+            -0.020671270787715912,
+            0.01749962382018566,
+            0.037033889442682266,
+            -0.0011412798194214702,
+            0.03744423761963844,
+            0.026980368420481682,
+            -0.005979964509606361,
+            -0.013105483725667,
+            -0.028348194435238838,
+            0.013797945342957973,
+            0.006086825858801603,
+            0.03405886888504028,
+            -0.02034641243517399,
+            -0.03525571525096893,
+            0.007664100266993046,
+            -0.0034281141124665737,
+            0.04079541191458702,
+            -0.00934396218508482,
+            0.007822255603969097,
+            0.023099161684513092,
+            0.010942608118057251,
+            -0.004847233649343252,
+            -0.024210521951317787,
+            -0.008540363982319832,
+            -0.01188298873603344,
+            -0.0402824766933918,
+            -0.011891537345945835,
+            0.03157968446612358,
+            -0.015353847295045853,
+            0.01938893459737301,
+            -0.005915847606956959,
+            -0.007800883147865534,
+            -0.023543706163764,
+            0.018072402104735374,
+            0.0032464496325701475,
+            -0.008027428761124611,
+            -0.0069075217470526695,
+            0.003357585519552231,
+            -0.0019352601375430822,
+            0.003267821855843067,
+            0.048181671649217606,
+            -0.02402244508266449,
+            0.012113809585571289,
+            0.008527540601789951,
+            0.0011316622840240598,
+            0.02380017377436161,
+            0.03346044570207596,
+            -0.035563476383686066,
+            -0.012934505008161068,
+            -0.0014618640998378396,
+            0.009514939971268177,
+            0.020876444876194,
+            -0.009873994626104832,
+            -0.0097030159085989,
+            -0.03378530219197273,
+            -0.0005765173118561506,
+            0.00013043770741205662,
+            -0.015712901949882507,
+            -0.008467698469758034,
+            0.017166217789053917,
+            0.042641978710889816,
+            0.032434575259685516,
+            0.02034641243517399,
+            -0.035289913415908813,
+            -0.00907894503325224,
+            0.027082955464720726,
+            -0.014610092155635357,
+            0.034315336495637894,
+            0.0504043884575367,
+            -0.029459552839398384,
+            -0.018773412331938744,
+            -0.03980373963713646,
+            0.07112695276737213,
+            -0.023731783032417297,
+            -0.02964762970805168,
+            0.042368412017822266,
+            -0.01408860832452774,
+            0.0021778354421257973,
+            0.03187034651637077,
+            0.035324107855558395,
+            -0.0035307009238749743,
+            0.012053967453539371,
+            0.03499924764037132,
+            -0.0194060318171978,
+            -0.04691643267869949,
+            -0.02523639053106308,
+            -0.025116706266999245,
+            -0.016619086265563965,
+            -0.010275793261826038,
+            -0.017311548814177513,
+            0.010087717324495316,
+            0.020278021693229675,
+            0.013635516166687012,
+            -0.010882765986025333,
+            0.007022931706160307,
+            0.009985130280256271,
+            -0.009976581670343876,
+            -0.000917939527425915,
+            -0.027561694383621216,
+            -0.0071939099580049515,
+            -0.028467880561947823,
+            -0.012053967453539371,
+            -0.014977695420384407,
+            -0.006685249973088503,
+            -0.025783522054553032,
+            0.01889309659600258,
+            0.023167554289102554,
+            -0.002190658822655678,
+            -0.008459148928523064,
+            0.01238737441599369,
+            0.03472568467259407,
+            -0.028108825907111168,
+            -0.03812815248966217,
+            0.008916515856981277,
+            -0.0034558980260044336,
+            0.009762858040630817,
+            -0.015037537552416325,
+            0.010677591897547245,
+            -0.011395701207220554,
+            0.010540809482336044,
+            -0.010703238658607006,
+            -0.010002228431403637,
+            -0.0006646779947914183,
+            0.021953608840703964,
+            0.00801460538059473,
+            -0.014592994004487991,
+            -0.017593663185834885,
+            0.027715574949979782,
+            0.014293782413005829,
+            0.0334775410592556,
+            -0.029545042663812637,
+            -0.023099161684513092,
+            0.042573586106300354,
+            0.004015851765871048,
+            -0.043086521327495575,
+            0.016148896887898445,
+            0.01671312376856804,
+            0.011797499842941761,
+            0.01942313089966774,
+            0.048181671649217606,
+            -0.033870793879032135,
+            -0.038196541368961334,
+            0.005420010536909103,
+            -0.031938739120960236,
+            -0.030929965898394585,
+            0.014883656986057758,
+            -0.006227883044630289,
+            -0.05125928297638893,
+            -0.031203532591462135,
+            -0.014020216651260853,
+            -0.019337641075253487,
+            0.0022098938934504986,
+            -0.015046087093651295,
+            0.004941271618008614,
+            -0.03741004317998886,
+            0.0025689483154565096,
+            0.0004648471367545426,
+            -0.018328869715332985,
+            -0.028724348172545433,
+            0.0027548871003091335,
+            0.004488179460167885,
+            0.03542669489979744,
+            0.0017097825184464455,
+            0.01128456462174654,
+            -0.012951603159308434,
+            -0.000579188868869096,
+            0.003056236309930682,
+            -0.05016501992940903,
+            -0.019474424421787262,
+            0.038983043283224106,
+            0.009027652442455292,
+            0.006048355717211962,
+            0.011874440126121044,
+            -0.028211412951350212,
+            -0.0592268668115139,
+            0.029716020449995995,
+            -0.04643769562244415,
+            0.015268358401954174,
+            0.004415513481944799,
+            -0.0010344184702262282,
+            -0.001160514890216291,
+            -0.0007758138235658407,
+            -0.0067707388661801815,
+            0.023167554289102554,
+            -0.007407633122056723,
+            0.012498511001467705,
+            -0.0028788463678210974,
+            0.03669193387031555,
+            0.0358712375164032,
+            -0.00673226872459054,
+            -0.038948848843574524,
+            0.02956213988363743,
+            -0.008668597787618637,
+            0.025612542405724525,
+            0.01889309659600258,
+            0.02961343340575695,
+            -0.017918521538376808,
+            0.032417476177215576,
+            0.002891669748350978,
+            0.024125032126903534,
+            -0.03877786919474602,
+            0.023167554289102554,
+            0.0052191112190485,
+            -0.054268497973680496,
+            -0.00860020611435175,
+            -0.0073777115903794765,
+            0.022244270890951157,
+            0.008822478353977203,
+            -0.02477474883198738,
+            -0.021970706060528755,
+            0.0009409147314727306,
+            -0.02817721664905548,
+            -0.007574336603283882,
+            0.04568539187312126,
+            -0.012558353133499622,
+            0.0014875108608976007,
+            -0.017414135858416557,
+            0.014174097217619419,
+            0.02886112965643406,
+            0.027510400861501694,
+            0.03354593366384506,
+            -0.013986021280288696,
+            0.020021554082632065,
+            -0.022278467193245888,
+            -0.037033889442682266,
+            -0.02595449984073639,
+            0.0073050460778176785,
+            0.03939339146018028,
+            0.009728662669658661,
+            0.006039807107299566,
+            -0.0013913356233388186,
+            0.00790346972644329,
+            -0.015370945446193218,
+            0.027989141643047333,
+            0.044796302914619446,
+            -0.002494145417585969,
+            -0.011464091949164867,
+            0.007505945395678282,
+            0.015046087093651295,
+            -0.018619531765580177,
+            0.027373619377613068,
+            -0.025424467399716377,
+            0.02255203202366829,
+            0.004744646605104208,
+            0.02065417356789112,
+            -0.047702934592962265,
+            -0.004556570667773485,
+            -0.0023082063999027014,
+            -0.03445211797952652,
+            0.05946623906493187,
+            -0.028690151870250702,
+            -0.013524380512535572,
+            0.0029386887326836586,
+            -0.0011316622840240598,
+            -0.01313113048672676,
+            -0.026980368420481682,
+            -0.014849461615085602,
+            -0.0061894129030406475,
+            -0.04554860666394234,
+            -0.029801510274410248,
+            -0.014815266244113445,
+            -0.01213945634663105,
+            -0.06240706518292427,
+            0.004058596212416887,
+            -0.0741361752152443,
+            0.007206733338534832,
+            0.027715574949979782,
+            -0.016661832109093666,
+            0.03963275998830795,
+            -0.022637521848082542,
+            0.02374888025224209,
+            -0.04076121747493744,
+            -0.0157299991697073,
+            0.03258845582604408,
+            0.047463566064834595,
+            0.010848570615053177,
+            0.011498288251459599,
+            0.021047422662377357,
+            -0.03199003264307976,
+            -0.0026758096646517515,
+            -0.00860875565558672,
+            -0.0018219870980829,
+            0.03332366421818733,
+            -0.008553187362849712,
+            0.021765531972050667,
+            0.0051549943163990974,
+            0.024142129346728325,
+            0.031442902982234955,
+            -0.002782671246677637,
+            -0.023868564516305923,
+            0.012857564724981785,
+            0.025544151663780212,
+            0.013054190203547478,
+            -0.014251037500798702,
+            0.012686586938798428,
+            -0.01579839177429676,
+            0.007719668094068766,
+            0.008484795689582825,
+            0.009908189997076988,
+            0.034315336495637894,
+            0.024142129346728325,
+            -0.013105483725667,
+            0.0029814334120601416,
+            0.009865445084869862,
+            -0.03645256534218788,
+            0.042641978710889816,
+            0.0051849158480763435,
+            -0.03953017294406891,
+            -0.0005487333401106298,
+            -0.007617081515491009,
+            0.020551586523652077,
+            0.02156035788357258,
+            0.00936105940490961,
+            -0.022671716287732124,
+            -0.00020944836433045566,
+            0.016841357573866844,
+            0.008660049177706242,
+            -0.012182201258838177,
+            0.02450118400156498,
+            0.03833332657814026,
+            -0.020260922610759735,
+            -0.030639303848147392,
+            0.00922427698969841,
+            -0.010489515960216522,
+            0.0007736766128800809,
+            -0.02715134806931019,
+            -0.004394141025841236,
+            0.018072402104735374,
+            -0.044762108474969864,
+            0.05177221819758415,
+            -0.013575674034655094,
+            0.008685695938766003,
+            -0.012943054549396038,
+            -0.002703593811020255,
+            -0.02138938009738922,
+            0.034588903188705444,
+            0.014131353236734867,
+            0.02867305465042591,
+            0.016106151044368744,
+            0.010318538174033165,
+            0.028040433302521706,
+            -0.020278021693229675,
+            -0.023099161684513092,
+            0.014422016218304634,
+            -0.007980410009622574,
+            -0.00467198109254241,
+            -0.02959633618593216,
+            -0.024107934907078743,
+            -0.01892729289829731,
+            0.02527058683335781,
+            -0.01530255377292633,
+            -0.008566010743379593,
+            0.010951157659292221,
+            0.0014981969725340605,
+            -0.002131885150447488,
+            0.013943277299404144,
+            -0.007638453505933285,
+            0.008707067929208279,
+            0.04192386940121651,
+            -0.013934727758169174,
+            0.02159455418586731,
+            0.03611060976982117,
+            0.011472641490399837,
+            -0.018516944721341133,
+            -0.0064800758846104145,
+            -0.0007293291273526847,
+            -0.01066904328763485,
+            -0.03570026159286499,
+            0.035289913415908813,
+            0.020996129140257835,
+            -0.0021842471323907375,
+            -0.037239063531160355,
+            -0.012583999894559383,
+            0.014635738916695118,
+            -0.014037314802408218,
+            0.027732674032449722,
+            -0.015943722799420357,
+            -0.015653058886528015,
+            0.00474037230014801,
+            0.021970706060528755,
+            -0.049857258796691895,
+            -0.012977249920368195,
+            0.010899864137172699,
+            0.002707868115976453,
+            0.008057350292801857,
+            -0.006779287941753864,
+            0.013455988839268684,
+            0.020568683743476868,
+            0.004945545922964811,
+            0.0036439741961658,
+            -0.0038085407577455044,
+            0.01647375524044037,
+            0.013455988839268684,
+            -0.017593663185834885,
+            -0.009241375140845776,
+            0.02210748940706253,
+            0.0230820644646883,
+            -0.030211858451366425,
+            -0.023919858038425446,
+            -0.00583035871386528,
+            0.021902315318584442,
+            -0.004753195680677891,
+            0.03689710795879364,
+            0.0388120636343956,
+            0.011438445188105106,
+            -0.04318910837173462,
+            -0.014516053721308708,
+            0.0017290175892412663,
+            -0.010660494677722454,
+            0.016431011259555817,
+            -0.0012438668636605144,
+            -0.042812954634428024,
+            0.002143639838322997,
+            -0.0036055040545761585,
+            0.028450781479477882,
+            -0.040214087814092636,
+            -0.011959929019212723,
+            0.009617527015507221,
+            -0.012951603159308434,
+            0.003302017692476511,
+            0.012182201258838177,
+            0.008484795689582825,
+            0.021269695833325386,
+            0.012703684158623219,
+            -0.005526872351765633,
+            -0.0014607955235987902,
+            0.016550695523619652,
+            0.004535198211669922,
+            0.026604216545820236,
+            -0.018328869715332985,
+            -0.017345745116472244,
+            0.01844855397939682,
+            0.03860688954591751,
+            -0.012327532283961773,
+            0.04076121747493744,
+            0.007514494471251965,
+            0.012062516063451767,
+            0.012088162824511528,
+            -0.02453538030385971,
+            -0.01598646678030491,
+            -0.011951380409300327,
+            0.006257804110646248,
+            -0.037512630224227905,
+            0.014858010224997997,
+            -0.02380017377436161,
+            -0.027134248986840248,
+            0.02619386836886406,
+            -0.006014160346239805,
+            0.010686141438782215,
+            0.022244270890951157,
+            -0.012891761027276516,
+            -0.027271032333374023,
+            0.03369981423020363,
+            0.034349530935287476,
+            0.008433502167463303,
+            0.04677965119481087,
+            -0.010395478457212448,
+            0.006779287941753864,
+            0.005702124908566475,
+            0.0024920080322772264,
+            0.044762108474969864,
+            -0.008236877620220184,
+            -0.01200267393141985,
+            0.007535866927355528,
+            0.009779956191778183,
+            0.04206065088510513,
+            -0.015413690358400345,
+            -0.049412716180086136,
+            0.019508618861436844,
+            -0.014507505111396313,
+            0.009856896474957466,
+            -0.006655328441411257,
+            0.008198407478630543,
+            -0.005890200845897198,
+            0.010053521953523159,
+            -0.004522374831140041,
+            0.010113364085555077,
+            -0.005843182094395161,
+            -0.014327977783977985,
+            0.015174319967627525,
+            -0.01018175482749939,
+            0.030280249193310738,
+            0.009557684883475304,
+            -0.02503121644258499,
+            -0.026501629501581192,
+            0.01746542938053608,
+            0.0019277798710390925,
+            -0.028724348172545433,
+            0.009882543236017227,
+            0.0023872838355600834,
+            0.010104815475642681,
+            0.004381317645311356,
+            -0.012242043390870094,
+            0.015926625579595566,
+            0.01676441729068756,
+            -0.022210076451301575,
+            0.03911982476711273,
+            -0.014285233803093433,
+            -0.014644287526607513,
+            0.019235054031014442,
+            -0.03730745613574982,
+            0.008424953557550907,
+            -0.012524157762527466,
+            0.010831472463905811,
+            0.00491989916190505,
+            0.004667706321924925,
+            -0.011352956295013428,
+            -0.0026715353596955538,
+            -0.02352660894393921,
+            0.03281072899699211,
+            0.0272197388112545,
+            -0.0012834055814892054,
+            0.005958592053502798,
+            -0.007535866927355528,
+            -0.012900309637188911,
+            0.0024984197225421667,
+            0.026142574846744537,
+            -0.024381499737501144,
+            -0.0017899286467581987,
+            -0.0006310166209004819,
+            0.007694021798670292,
+            0.00551832327619195,
+            0.022432347759604454,
+            0.04900236800312996,
+            -0.0038192267529666424,
+            0.028980813920497894,
+            0.035050541162490845,
+            0.024142129346728325,
+            -0.003259273013100028,
+            0.004761744290590286,
+            -0.025150902569293976,
+            0.03645256534218788,
+            0.008950712159276009,
+            0.016670379787683487,
+            0.017798837274312973,
+            -0.04312071576714516,
+            -0.009634625166654587,
+            0.019525716081261635,
+            0.02205619588494301,
+            0.020089944824576378,
+            -0.003881206503137946,
+            -0.0035884061362594366,
+            0.03298170492053032,
+            -0.025920303538441658,
+            -0.019354738295078278,
+            0.004154771566390991,
+            -0.0053174239583313465,
+            -0.005732045974582434,
+            -0.0069032469764351845,
+            0.01865372806787491,
+            -0.014584445394575596,
+            -0.004458257928490639,
+            0.016867004334926605,
+            0.001722605898976326,
+            0.012113809585571289,
+            0.016165994107723236,
+            -0.00024137321452144533,
+            0.022415248677134514,
+            -0.004932722542434931,
+            0.021509064361453056,
+            0.005330247338861227,
+            -0.003663209266960621,
+            -0.006911796052008867,
+            -0.010275793261826038,
+            0.029015010222792625,
+            -0.006980187259614468,
+            -0.034640196710824966,
+            -0.01360986940562725,
+            -0.009284119121730328,
+            0.004156908951699734,
+            0.0011391426669433713,
+            0.0050524077378213406,
+            0.01744833029806614,
+            0.026501629501581192,
+            -0.00046751866466365755,
+            0.004280867986381054,
+            0.014336527325212955,
+            -0.0024193422868847847,
+            -0.010506614111363888,
+            0.019559912383556366,
+            0.008779733441770077,
+            -0.01624293439090252,
+            0.018756315112113953,
+            0.0030861576087772846,
+            0.04688223823904991,
+            -0.044522739946842194,
+            0.018209183588624,
+            0.007805157452821732,
+            0.0006956678116694093,
+            -0.014960597269237041,
+            -0.03327237069606781,
+            0.0036097783595323563,
+            -0.014122803695499897,
+            -0.0033597226720303297,
+            -0.024689260870218277,
+            0.00786499958485365,
+            -0.018089499324560165,
+            0.014045863412320614,
+            0.011600874364376068,
+            -0.0016349796205759048,
+            -0.002028229646384716,
+            0.017542369663715363,
+            -0.001940603251568973,
+            0.00787782296538353,
+            -0.017362842336297035,
+            0.010942608118057251,
+            0.022381054237484932,
+            -0.010677591897547245,
+            0.008638676255941391,
+            0.0073306928388774395,
+            -0.02793784812092781,
+            -0.028536271303892136,
+            0.013336303643882275,
+            -0.03546088933944702,
+            0.007681198418140411,
+            0.031203532591462135,
+            -0.032434575259685516,
+            -0.028519174084067345,
+            0.0019085448002442718,
+            -0.021406477317214012,
+            -0.017867228016257286,
+            0.011446994729340076,
+            0.018021108582615852,
+            -0.004216751083731651,
+            -0.030451226979494095,
+            0.013173874467611313,
+            -0.007809431757777929,
+            0.00698873633518815,
+            0.00015388043539132923,
+            -0.02569803223013878,
+            -0.02569803223013878,
+            0.004984016064554453,
+            -0.028211412951350212,
+            -0.007138342130929232,
+            -0.0005297654424794018,
+            -0.024466989561915398,
+            -0.043804630637168884,
+            -0.015242711640894413,
+            -0.013507282361388206,
+            -0.010156108066439629,
+            0.0480106957256794,
+            0.0196795966476202,
+            0.026894880458712578,
+            -0.0003211185394320637,
+            -0.018106596544384956,
+            0.04773712903261185,
+            0.0009954140987247229,
+            0.041752889752388,
+            0.013387597166001797,
+            -0.008335189893841743,
+            -0.006693798583000898,
+            0.03392208740115166,
+            -0.015584668144583702,
+            -0.007775236386805773,
+            -0.010651945136487484,
+            -0.0015836860984563828,
+            0.0025091059505939484,
+            0.022449444979429245,
+            0.0019085448002442718,
+            0.004037224221974611,
+            0.027100054547190666,
+            -0.023201748728752136,
+            0.002739926567301154,
+            0.01238737441599369,
+            -0.006937442813068628,
+            0.008215505629777908,
+            -0.02838239073753357,
+            -0.008300994522869587,
+            0.019081173464655876,
+            0.006044081412255764,
+            -0.007437554188072681,
+            -0.0022440897300839424,
+            -0.027134248986840248,
+            -0.0014052275801077485,
+            0.0038106779102236032,
+            -0.005826083943247795,
+            0.005744869355112314,
+            0.04459112882614136,
+            -0.004817312583327293,
+            -0.039940521121025085,
+            -0.00029493749025277793,
+            -0.03634997829794884,
+            0.0001610935723874718,
+            0.00010799681331263855,
+            -0.0033191153779625893,
+            0.005488402210175991,
+            0.0018422907451167703,
+            -0.012472864240407944,
+            0.012130907736718655,
+            -0.028741445392370224,
+            0.019782183691859245,
+            0.004817312583327293,
+            -0.007574336603283882,
+            -0.005056682042777538,
+            0.01988477073609829,
+            0.008151388727128506,
+            -0.013148227706551552,
+            -0.0014960597036406398,
+            0.0028852580580860376,
+            -0.026125477626919746,
+            -0.020773857831954956,
+            -0.03369981423020363,
+            -0.0022505014203488827,
+            -0.009583331644535065,
+            -0.03010927140712738,
+            -0.006937442813068628,
+            -0.0003617258626036346,
+            0.007775236386805773,
+            -0.020295118913054466,
+            0.0006956678116694093,
+            0.005552519112825394,
+            0.012199298478662968,
+            0.00624070642516017,
+            -0.007339241914451122,
+            -0.004263770300894976,
+            -0.03033154271543026,
+            0.003584131831303239,
+            -0.0015345298452302814,
+            0.0038619714323431253,
+            -0.016063407063484192,
+            0.006351842079311609,
+            -0.016183091327548027,
+            -0.0035264266189187765,
+            0.011335858143866062,
+            0.01800401136279106,
+            -0.028741445392370224,
+            0.036726128309965134,
+            0.02889532595872879,
+            0.017320098355412483,
+            -0.038914650678634644,
+            0.028587564826011658,
+            -0.0012331806356087327,
+            0.001522775157354772,
+            0.0030177661683410406,
+            0.021731335669755936,
+            0.026039987802505493,
+            -0.016431011259555817,
+            -0.006531369406729937,
+            0.010224499739706516,
+            0.028194313868880272,
+            -0.031545490026474,
+            -0.007266575936228037,
+            1.9886239897459745e-05,
+            0.02378307469189167,
+            0.027783967554569244,
+            0.023116260766983032,
+            0.04271036759018898,
+            -0.0053131491877138615,
+            0.002891669748350978,
+            -0.005082328803837299,
+            -0.013926179148256779,
+            0.0024727729614824057,
+            -0.0028767092153429985,
+            -0.0017952717607840896,
+            0.011489738710224628,
+            0.012695135548710823,
+            0.017593663185834885,
+            -0.07126373797655106,
+            0.038743674755096436,
+            0.026570022106170654,
+            0.012105260975658894,
+            0.008399306796491146,
+            -0.024706358090043068,
+            -0.0017909972229972482,
+            -0.0022205801215022802,
+            0.0018604571232572198,
+            0.013438890688121319,
+            -0.010874217376112938,
+            -0.02475765161216259,
+            0.016567792743444443,
+            -0.019457325339317322,
+            0.017149118706583977,
+            0.0050267609767615795,
+            0.00503958435729146,
+            -0.024603771045804024,
+            -0.014721227809786797,
+            0.006830581463873386,
+            0.008065898902714252,
+            0.008100095205008984,
+            -0.03857269510626793,
+            -0.044727910310029984,
+            0.01750817336142063,
+            0.008035978302359581,
+            -0.04862621799111366,
+            -0.006586937233805656,
+            -0.017388489097356796,
+            -0.004210339393466711,
+            -0.006326195318251848,
+            -0.01767915114760399,
+            -0.040897998958826065,
+            0.002064562402665615,
+            0.017302999272942543,
+            0.018534043803811073,
+            5.349749335437082e-05,
+            0.028826933354139328,
+            0.0017183314776048064,
+            -0.0022355406545102596,
+            -0.030434129759669304,
+            0.026860684156417847,
+            -0.007326418533921242,
+            -0.022278467193245888,
+            0.024142129346728325,
+            0.01628567837178707,
+            -0.019970260560512543,
+            -0.0038769319653511047,
+            0.0014928538585081697,
+            -0.008446325547993183,
+            0.034554705023765564,
+            -0.0017129883635789156,
+            -0.008933614008128643,
+            0.0026907704304903746,
+            -0.0020998266991227865,
+            -0.00959188025444746,
+            -0.02479184791445732,
+            0.00145438383333385,
+            0.025869010016322136,
+            -0.007749589625746012,
+            0.006437331438064575,
+            0.04202645644545555,
+            -0.012353179045021534,
+            -0.019850576296448708,
+            0.00043145293602719903,
+            -0.02839948795735836,
+            0.010318538174033165,
+            -0.006685249973088503,
+            0.030280249193310738,
+            0.004206065088510513,
+            0.013002896681427956,
+            -0.016644733026623726,
+            -0.01066904328763485,
+            -0.021970706060528755,
+            -0.0641852393746376,
+            -0.012122358195483685,
+            -0.005561067722737789,
+            -0.024894434958696365,
+            -0.02398824878036976,
+            0.03234908729791641,
+            0.022398151457309723,
+            0.010609201155602932,
+            -0.002107307082042098,
+            -0.01891019567847252,
+            -0.004458257928490639,
+            0.03009217418730259,
+            -0.005458480678498745,
+            0.017371391877532005,
+            -0.011617972515523434,
+            0.01627713069319725,
+            -0.006578388623893261,
+            0.0051336223259568214,
+            -0.053960736840963364,
+            -0.005167817696928978,
+            0.02983570657670498,
+            0.01993606425821781,
+            0.007240929175168276,
+            0.010019325651228428,
+            0.013293559662997723,
+            -0.007501671090722084,
+            -0.02764718420803547,
+            0.02525348775088787,
+            0.017132021486759186,
+            0.01991896703839302,
+            -0.015935173258185387,
+            0.007587159983813763,
+            0.011788951233029366,
+            0.00288098375312984,
+            0.000775279535446316,
+            0.0008281759219244123,
+            0.025133803486824036,
+            0.003549935994669795,
+            -0.009275570511817932,
+            0.015721451491117477,
+            -0.007163988891988993,
+            -0.021269695833325386,
+            -0.004233849234879017,
+            0.008959260769188404,
+            0.004035086836665869,
+            0.001731154858134687,
+            -0.0015922350576147437,
+            -0.008984907530248165,
+            0.02886112965643406,
+            0.03539250046014786,
+            0.03183615207672119,
+            -0.0057876138016581535,
+            -0.005137896630913019,
+            0.0016456657322123647,
+            0.003513603238388896,
+            -0.007659825962036848,
+            -0.0018294673645868897,
+            0.019799282774329185,
+            -0.030006684362888336,
+            -0.011276016011834145,
+            -0.009318315424025059,
+            0.016362618654966354,
+            0.003522152081131935,
+            0.04411239176988602,
+            0.02111581526696682,
+            0.006522820331156254,
+            -0.00557389110326767,
+            0.0037209142465144396,
+            0.004389866720885038,
+            0.029237281531095505,
+            -0.02282559685409069,
+            9.023110033012927e-05,
+            -0.0034217024222016335,
+            -0.017901424318552017,
+            0.019046977162361145,
+            -0.007300771772861481,
+            0.009540586732327938,
+            0.02205619588494301,
+            0.09212308377027512,
+            -0.0065527418628335,
+            0.022449444979429245,
+            -0.027852358296513557,
+            -0.024090835824608803,
+            0.00291517935693264,
+            0.006313371937721968,
+            0.002265461953356862,
+            0.022039096802473068,
+            0.0054499320685863495,
+            0.02525348775088787,
+            0.0272197388112545,
+            0.0009067190694622695,
+            0.006698073353618383,
+            -0.010292891412973404,
+            0.004118438810110092,
+            -0.013182424008846283,
+            -0.009549135342240334,
+            -0.009437999688088894,
+            0.008779733441770077,
+            0.019012782722711563,
+            0.00911314133554697,
+            0.007531592156738043,
+            -0.004084242973476648,
+            -0.015482081100344658,
+            0.026518728584051132,
+            0.010292891412973404,
+            -0.004475356079638004,
+            -0.015601766295731068,
+            -0.015114477835595608,
+            0.012404472567141056,
+            -0.006886149290949106,
+            -0.005894475616514683,
+            0.01721751131117344,
+            0.005069505423307419,
+            0.02865595556795597,
+            -0.03369981423020363,
+            0.008890869095921516,
+            0.014259587042033672,
+            -0.0010857119923457503,
+            0.007245203480124474,
+            0.021423576399683952,
+            -0.0022633245680481195,
+            0.0242789126932621,
+            -0.01468703243881464,
+            0.00874553807079792,
+            0.010959706269204617,
+            -0.021201303228735924,
+            0.0036055040545761585,
+            0.01990186981856823,
+            -0.003381095128133893,
+            0.015413690358400345,
+            -0.0052191112190485,
+            -0.001976936124265194,
+            0.0025903205387294292,
+            0.006582662928849459,
+            0.00363756250590086,
+            -0.01988477073609829,
+            0.01188298873603344,
+            -0.01872211880981922,
+            0.009172983467578888,
+            -0.004565119277685881,
+            0.012190749868750572,
+            -0.008424953557550907,
+            0.011370054446160793,
+            -0.006121021695435047,
+            0.006847679149359465,
+            -0.0105750048533082,
+            -0.016559245064854622,
+            -0.012515608221292496,
+            0.012960151769220829,
+            0.007249478250741959,
+            -0.013558575883507729,
+            0.008309543132781982,
+            -0.008228329010307789,
+            -0.002067768247798085,
+            0.014960597269237041,
+            -0.011746206320822239,
+            0.00407355697825551,
+            -0.039461780339479446,
+            -0.004535198211669922,
+            0.018089499324560165,
+            -0.001959838205948472,
+            0.0027719850186258554,
+            -0.007352065294981003,
+            0.011199075728654861,
+            -0.008288171142339706,
+            -0.0024086562916636467,
+            0.0254757609218359,
+            -0.019782183691859245,
+            -0.017328646034002304,
+            -0.032913316041231155,
+            -0.05009663105010986,
+            -0.002131885150447488,
+            -0.009326864033937454,
+            0.005864554084837437,
+            -0.00790346972644329,
+            0.017815934494137764,
+            -0.0040179891511797905,
+            -0.007095597684383392,
+            -0.024125032126903534,
+            0.010395478457212448,
+            0.028211412951350212,
+            0.0008682489860802889,
+            -0.03079318441450596,
+            -0.012344630435109138,
+            0.010463869199156761,
+            0.0006646779947914183,
+            -0.029049206525087357,
+            0.0083693852648139,
+            0.0006181932403706014,
+            -0.014473309740424156,
+            -0.008882320486009121,
+            -0.005971415434032679,
+            0.021440673619508743,
+            0.024193422868847847,
+            0.005753418430685997,
+            -0.019782183691859245,
+            -0.004225300159305334,
+            0.03258845582604408,
+            -0.018294673413038254,
+            0.003994479309767485,
+            0.023868564516305923,
+            -0.00264375121332705,
+            0.00982270110398531,
+            0.01054935809224844,
+            0.018072402104735374,
+            -0.008352288044989109,
+            0.012660940177738667,
+            0.007561513222754002,
+            -0.015328200533986092,
+            0.008548912592232227,
+            0.006668151821941137,
+            0.016405364498496056,
+            0.006890423595905304,
+            -0.022244270890951157,
+            -0.004411239176988602,
+            0.015832586213946342,
+            0.053276825696229935,
+            -0.006706621963530779,
+            -0.008506168611347675,
+            0.03617899864912033,
+            0.024415696039795876,
+            0.007416181731969118,
+            0.005937220063060522,
+            -0.03231488913297653,
+            -0.004646334331482649,
+            0.0008709204848855734,
+            0.005236208904534578,
+            0.0132422661408782,
+            0.0035029170103371143,
+            -0.007770961616188288,
+            0.018978586420416832,
+            -0.001043501659296453,
+            0.014293782413005829,
+            -0.020107043907046318,
+            0.009514939971268177,
+            -0.00618513859808445,
+            0.03161387890577316,
+            0.01042112521827221,
+            -0.0010632710764184594,
+            0.03318687900900841,
+            0.006719445344060659,
+            -0.03344334661960602,
+            0.006125296000391245,
+            -0.007638453505933285,
+            0.011352956295013428,
+            -0.02260332554578781,
+            -0.007894921116530895,
+            0.009899641387164593,
+            0.0012289062142372131,
+            -0.018123695626854897,
+            0.011900086887180805,
+            0.007142616901546717,
+            0.009378157556056976,
+            -0.0005217508296482265,
+            0.006356116849929094,
+            0.008553187362849712,
+            -0.004522374831140041,
+            -0.008168485946953297,
+            0.005599537864327431,
+            -0.016336971893906593,
+            -0.0040179891511797905,
+            0.029425358399748802,
+            -0.0006726926076225936,
+            0.010309988632798195,
+            0.0004135536728426814,
+            -0.0004116835771128535,
+            -0.018978586420416832,
+            0.009207178838551044,
+            -0.024569574743509293,
+            -0.006415958981961012,
+            0.005595263559371233,
+            -0.01225059200078249,
+            -0.006142393685877323,
+            0.014413466677069664,
+            0.0022633245680481195,
+            -0.021047422662377357,
+            -0.008061625063419342,
+            -0.030485423281788826,
+            -0.021218402311205864,
+            0.031904544681310654,
+            0.006518546026200056,
+            -0.005646557081490755,
+            0.015379494056105614,
+            -0.022876890376210213,
+            -0.0054969508200883865,
+            -0.005035309586673975,
+            0.016636185348033905,
+            -0.020021554082632065,
+            0.004398415796458721,
+            -0.010634847916662693,
+            -0.016431011259555817,
+            0.023099161684513092,
+            0.008326641283929348,
+            0.0011220448650419712,
+            0.006133845075964928,
+            -0.02497992292046547,
+            -0.014285233803093433,
+            0.01798691228032112,
+            0.005702124908566475,
+            -0.008228329010307789,
+            0.01795271784067154,
+            -0.0012609646655619144,
+            -0.007215282414108515,
+            -0.013780847191810608,
+            0.01841435767710209,
+            -0.02183392271399498,
+            0.026535825803875923,
+            0.012156554497778416,
+            -0.010472417809069157,
+            0.008822478353977203,
+            0.004992565140128136,
+            0.01287466287612915,
+            -0.0003889755462296307,
+            -0.015370945446193218,
+            0.013421793468296528,
+            0.013712456449866295,
+            0.002194933360442519,
+            -0.0021778354421257973,
+            0.0037572472356259823,
+            0.005826083943247795,
+            -0.0054456572979688644,
+            0.02333853207528591,
+            0.023594999685883522,
+            0.0002186651690863073,
+            0.015080282464623451,
+            -0.02084225043654442,
+            -0.019611205905675888,
+            0.02378307469189167,
+            -0.004595040809363127,
+            -0.0314599983394146,
+            0.038709476590156555,
+            -0.010292891412973404,
+            -0.007236654870212078,
+            -0.01993606425821781,
+            0.00771111948415637,
+            0.020448999479413033,
+            -0.021885216236114502,
+            -0.010002228431403637,
+            -0.00110708421561867,
+            0.008848125115036964,
+            0.027852358296513557,
+            0.020568683743476868,
+            -0.015507727861404419,
+            0.02788655459880829,
+            0.009668820537626743,
+            0.008044526912271976,
+            0.011874440126121044,
+            0.005561067722737789,
+            -0.010840021073818207,
+            0.010044972412288189,
+            -0.0005241552134975791,
+            0.005296051502227783,
+            -0.00516354339197278,
+            -0.02231266349554062,
+            -0.015644511207938194,
+            -0.04178708791732788,
+            -0.0036589347291737795,
+            -0.008604480884969234,
+            0.01553337462246418,
+            0.08624143153429031,
+            -0.014037314802408218,
+            -0.009309765882790089,
+            -0.0020271609537303448,
+            0.009976581670343876,
+            -0.01720896176993847,
+            0.012566901743412018,
+            0.024466989561915398,
+            -0.013626967556774616,
+            -0.0329304113984108,
+            0.0025518503971397877,
+            -0.012994347140192986,
+            0.007805157452821732,
+            0.017542369663715363,
+            0.0019245740259066224,
+            -0.023680489510297775,
+            -0.0018433593213558197,
+            -0.020004456862807274,
+            0.001239592325873673,
+            0.010369831696152687,
+            -0.016131797805428505,
+            -0.010771630331873894,
+            0.023902760818600655,
+            -0.02017543464899063,
+            -0.025167999789118767,
+            -0.012925956398248672,
+            -0.007745314855128527,
+            -0.011891537345945835,
+            0.017815934494137764,
+            0.008647225797176361,
+            -0.0186708252876997,
+            -0.008595932275056839,
+            0.005915847606956959,
+            -0.011378603056073189,
+            8.963001164374873e-05,
+            0.011925733648240566,
+            -0.0033169782254844904,
+            0.008788282051682472,
+            0.01482381485402584,
+            -0.03400757536292076,
+            -0.003128902055323124,
+            -0.014704130589962006,
+            0.01360986940562725,
+            0.006557016167789698,
+            0.028057532384991646,
+            -0.028279803693294525,
+            -0.021765531972050667,
+            -0.00981415156275034,
+            -0.01892729289829731,
+            -0.03082738071680069,
+            -0.02692907489836216,
+            0.0058175353333354,
+            0.00636466545984149,
+            0.021526163443922997,
+            0.014729776419699192,
+            -0.014396369457244873,
+            0.010096265934407711,
+            -0.0033981928136199713,
+            0.008984907530248165,
+            -0.014404918067157269,
+            -0.0018454965902492404,
+            -0.015550472773611546,
+            0.015131575986742973,
+            0.017662053927779198,
+            -0.002583908848464489,
+            -0.018380163237452507,
+            -0.019730890169739723,
+            0.010429673828184605,
+            -0.007604258134961128,
+            -0.004637785255908966,
+            0.02668970637023449,
+            0.003182332729920745,
+            0.0019096133764833212,
+            0.007527317851781845,
+            0.023406922817230225,
+            0.008664323017001152,
+            0.03546088933944702,
+            -0.023406922817230225,
+            0.001999377040192485,
+            -0.0189443901181221,
+            0.0031866072677075863,
+            -0.023218847811222076,
+            0.005112249869853258,
+            -0.01942313089966774,
+            0.03417855501174927,
+            -0.006133845075964928,
+            -0.007852176204323769,
+            0.030468326061964035,
+            -0.02624516189098358,
+            0.002979296026751399,
+            -0.009754309430718422,
+            0.004650608636438847,
+            0.022141683846712112,
+            -0.0038256384432315826,
+            -0.017285902053117752,
+            -0.03925660625100136,
+            0.003522152081131935,
+            -0.009899641387164593,
+            -0.006091100629419088,
+            -0.011318760924041271,
+            0.01773044466972351,
+            0.005065230652689934,
+            0.005569616798311472,
+            -0.0015633824514225125,
+            0.016892651095986366,
+            0.0013913356233388186,
+            0.0035456616897135973,
+            0.029955390840768814,
+            -0.001200053608044982,
+            0.012558353133499622,
+            0.0033533109817653894,
+            0.005689301528036594,
+            0.023492412641644478,
+            -0.007745314855128527,
+            -0.0015206378884613514,
+            -0.017362842336297035,
+            0.0067237201146781445,
+            -0.0007555101765319705,
+            0.02520219422876835,
+            0.01539659220725298,
+            -0.01771334744989872,
+            -0.040658630430698395,
+            0.02159455418586731,
+            -0.011352956295013428,
+            0.027373619377613068,
+            -0.009497841820120811,
+            -0.006569839548319578,
+            -0.0003144397051073611,
+            -0.03754682466387749,
+            0.003182332729920745,
+            -0.009720114059746265,
+            -0.052798084914684296,
+            -0.018089499324560165,
+            -0.007548689842224121,
+            -0.026296455413103104,
+            0.03342624753713608,
+            -0.0014650699449703097,
+            -0.026775194332003593,
+            0.024706358090043068,
+            -0.0034516234882175922,
+            -0.00698873633518815,
+            0.012233494780957699,
+            -0.016704576089978218,
+            -0.009378157556056976,
+            -0.0032443124800920486,
+            -0.005116524174809456,
+            0.0006069728406146169,
+            -0.012421570718288422,
+            -0.019235054031014442,
+            0.015866782516241074,
+            -0.006980187259614468,
+            0.0008538226829841733,
+            0.015866782516241074,
+            -0.0020496018696576357,
+            0.025065412744879723,
+            -0.014122803695499897,
+            0.005385815165936947,
+            0.007612806744873524,
+            -0.013883434236049652,
+            0.012301885522902012,
+            0.003847010899335146,
+            -0.015430787578225136,
+            0.01580693945288658,
+            0.009215728379786015,
+            0.00848907046020031,
+            0.0041611832566559315,
+            -0.014533151872456074,
+            0.0022889713291078806,
+            -0.011421347968280315,
+            0.006646779831498861,
+            0.013909080997109413,
+            0.005304600577801466,
+            -0.011276016011834145,
+            -0.016653282567858696,
+            0.013857787474989891,
+            0.007501671090722084,
+            0.020004456862807274,
+            -0.03180195763707161,
+            0.007386260665953159,
+            -0.004676255397498608,
+            0.006941717118024826,
+            -0.01227623876184225,
+            0.025287684053182602,
+            0.022620422765612602,
+            0.02111581526696682,
+            0.006356116849929094,
+            0.0028767092153429985,
+            0.010515162721276283,
+            -0.01966249942779541,
+            -0.0033618600573390722,
+            -0.0204148031771183,
+            -0.0006417027907446027,
+            -0.0008324504015035927,
+            0.01743123307824135,
+            -0.012318983674049377,
+            0.013105483725667,
+            0.006522820331156254,
+            0.015447885729372501,
+            -0.02887822687625885,
+            0.0039047158788889647,
+            0.0070400298573076725,
+            0.0013421793701127172,
+            0.0066211330704391,
+            0.0031224903650581837,
+            0.08371095359325409,
+            0.010472417809069157,
+            -0.0070870486088097095,
+            0.003968833014369011,
+            -0.009437999688088894,
+            -0.0077538639307022095,
+            0.015251260250806808,
+            -0.01081437524408102,
+            0.012233494780957699,
+            0.007347790524363518,
+            -0.013148227706551552,
+            0.004582217428833246,
+            0.0073563395999372005,
+            0.008108643814921379,
+            -0.01623438484966755,
+            0.011190527118742466,
+            -0.005253307055681944,
+            0.0038961670361459255,
+            0.025133803486824036,
+            0.01652504876255989,
+            -0.005723497364670038,
+            0.00971156544983387,
+            0.006467252504080534,
+            0.0009863307932391763,
+            -0.012404472567141056,
+            0.006390312220901251,
+            -0.004152634646743536,
+            0.023013673722743988,
+            -0.016208738088607788,
+            0.0344863161444664,
+            -0.0005706399097107351,
+            -0.003855559742078185,
+            -0.010882765986025333,
+            -0.003383232280611992,
+            0.007501671090722084,
+            -0.008322366513311863,
+            0.01114778220653534,
+            -0.013712456449866295,
+            0.0005973553052172065,
+            -0.013045640662312508,
+            -0.008189858868718147,
+            0.004058596212416887,
+            -0.007540141232311726,
+            -0.0073777115903794765,
+            -0.0009211453725583851,
+            0.001756801619194448,
+            -4.190970867057331e-05,
+            -0.0012331806356087327,
+            -0.0012919544242322445,
+            -0.009437999688088894,
+            -0.004513826221227646,
+            -0.013601320795714855,
+            -1.5010688912298065e-05,
+            0.02137228287756443,
+            -0.003682444104924798,
+            -0.006809209007769823,
+            0.01044677197933197,
+            0.003128902055323124,
+            0.02429600991308689,
+            -0.006095374934375286,
+            -0.014558798633515835,
+            -0.0021564632188528776,
+            -0.007005834020674229,
+            -0.007390534970909357,
+            0.00033928497578017414,
+            0.0038790691178292036,
+            0.006180863827466965,
+            0.003287056926637888,
+            0.012438667938113213,
+            0.006018434651196003,
+            -0.008831026963889599,
+            0.011763304471969604,
+            0.003120353212580085,
+            -0.0009190081618726254,
+            -0.01891019567847252,
+            -0.003242175094783306,
+            -0.013550027273595333,
+            -0.01199412439018488,
+            -0.011455543339252472,
+            0.017345745116472244,
+            -0.01723460853099823,
+            0.019765086472034454,
+            -0.006339018698781729,
+            0.00765555165708065,
+            -0.02549285814166069,
+            0.002449263585731387,
+            0.008899418637156487,
+            0.011250369250774384,
+            0.006702347658574581,
+            -0.012857564724981785,
+            -0.002013268880546093,
+            -0.0010862462222576141,
+            0.02963053248822689,
+            -0.008702793158590794,
+            -0.010010777041316032,
+            0.02816011942923069,
+            -0.014413466677069664,
+            -0.01019030436873436,
+            0.0014362173387780786,
+            -0.0013090522261336446,
+            0.02552705444395542,
+            0.011455543339252472,
+            0.017328646034002304,
+            0.0054456572979688644,
+            0.002840376226231456,
+            0.0013379048323258758,
+            0.023936955258250237,
+            0.016071956604719162,
+            0.010908412747085094,
+            -0.01418264675885439,
+            -0.008446325547993183,
+            -0.01942313089966774,
+            0.014216842129826546,
+            0.0033746834378689528,
+            -0.01530255377292633,
+            0.011224722489714622,
+            -0.021953608840703964,
+            -0.019064076244831085,
+            -0.023697586730122566,
+            -0.010386928915977478,
+            0.013438890688121319,
+            -0.011686364188790321,
+            -0.005505499895662069,
+            -0.017089277505874634,
+            -0.005454206373542547,
+            -0.010455320589244366,
+            -0.015148673206567764,
+            -0.01673022285103798,
+            -0.02598869428038597,
+            -0.022688815370202065,
+            -0.0098397983238101,
+            -0.017294451594352722,
+            -0.008300994522869587,
+            0.012797722592949867,
+            -0.020448999479413033,
+            0.017662053927779198,
+            -0.005065230652689934,
+            0.0037893056869506836,
+            0.01188298873603344,
+            0.0216116514056921,
+            -0.01748252660036087,
+            -0.031904544681310654,
+            -0.02402244508266449,
+            -0.02135518379509449,
+            0.022705912590026855,
+            0.0021927962079644203,
+            0.024142129346728325,
+            -0.007125518750399351,
+            0.016157444566488266,
+            -0.018363064154982567,
+            0.019303444772958755,
+            -0.04202645644545555,
+            -0.03222940117120743,
+            -0.007168263662606478,
+            0.00034836819395422935,
+            -0.004362082574516535,
+            -0.0010199921671301126,
+            -0.015661608427762985,
+            -0.010993901640176773,
+            -0.007168263662606478,
+            -0.006210785359144211,
+            -0.009685918688774109,
+            -0.01516577135771513,
+            0.01007916871458292,
+            0.004370631650090218,
+            0.016165994107723236,
+            0.001985484967008233,
+            0.015559021383523941,
+            -0.01516577135771513,
+            0.006945991888642311,
+            -0.0022269918117672205,
+            -0.005753418430685997,
+            0.004646334331482649,
+            0.003804266219958663,
+            0.0044924537651240826,
+            0.0002427089784760028,
+            0.0011017411015927792,
+            -0.00029386888490989804,
+            0.017362842336297035,
+            0.013361950404942036,
+            -0.007146891206502914,
+            0.01773044466972351,
+            6.485151970991865e-05,
+            0.0016787927597761154,
+            -0.010429673828184605,
+            -0.003214391181245446,
+            -0.00911314133554697,
+            -0.008882320486009121,
+            0.015362396836280823,
+            -0.019730890169739723,
+            -0.010626298375427723,
+            0.026843586936593056,
+            -0.0018348104786127806,
+            -0.03443502262234688,
+            0.02742491289973259,
+            7.86015789344674e-06,
+            -0.0024364402052015066,
+            -0.01369535829871893,
+            -0.007078499998897314,
+            -0.0006363596767187119,
+            -0.0005978895933367312,
+            0.017337195575237274,
+            -0.018038205802440643,
+            -0.005249032285064459,
+            0.00630909763276577,
+            -0.0040436359122395515,
+            -0.0021254734601825476,
+            0.006497173570096493,
+            0.014028766192495823,
+            -0.00047820480540394783,
+            -0.03501634672284126,
+            -0.012190749868750572,
+            0.020568683743476868,
+            -0.017106374725699425,
+            -0.014994793571531773,
+            0.011233271099627018,
+            -0.001976936124265194,
+            -0.012601097114384174,
+            -0.003911127801984549,
+            -0.005223385989665985,
+            -0.010993901640176773,
+            0.008442051708698273,
+            0.010224499739706516,
+            0.008339464664459229,
+            0.02210748940706253,
+            -0.014447662979364395,
+            -0.005757692735642195,
+            -0.015251260250806808,
+            0.0031075298320502043,
+            -0.022962380200624466,
+            -0.00015214394079521298,
+            -0.009745760820806026,
+            0.0016766554908826947,
+            0.012575451284646988,
+            0.01749962382018566,
+            0.028724348172545433,
+            0.0089678093791008,
+            0.0072879483923316,
+            -0.010848570615053177,
+            -0.01456734724342823,
+            0.004680529702454805,
+            0.0031417254358530045,
+            -0.00558671448379755,
+            0.006655328441411257,
+            0.019286347553133965,
+            0.0013261501444503665,
+            0.019303444772958755,
+            0.0025133804883807898,
+            0.0037978545296937227,
+            -2.8652215405600145e-05,
+            -0.010455320589244366,
+            -0.006706621963530779,
+            -0.02374888025224209,
+            0.0007453583530150354,
+            0.005779065191745758,
+            0.01889309659600258,
+            0.0005540763959288597,
+            0.0030370012391358614,
+            0.026416141539812088,
+            -0.0029771588742733,
+            -0.014345075935125351,
+            -0.012592548504471779,
+            -0.011942830868065357,
+            -0.00874553807079792,
+            0.034332435578107834,
+            0.01454170048236847,
+            -0.01454170048236847,
+            -0.015234163030982018,
+            -0.011566678993403912,
+            0.03033154271543026,
+            0.00442833686247468,
+            -0.0004928982816636562,
+            0.021902315318584442,
+            0.018294673413038254,
+            0.017345745116472244,
+            0.016362618654966354,
+            0.009420901536941528,
+            0.0027912200894206762,
+            0.020466096699237823,
+            0.0019480835180729628,
+            0.015088831074535847,
+            -0.008172760717570782,
+            0.008642951026558876,
+            -0.005864554084837437,
+            0.012712233699858189,
+            -0.010677591897547245,
+            -0.011797499842941761,
+            0.008433502167463303,
+            -0.014003119431436062,
+            0.002117993077263236,
+            -0.007373437285423279,
+            0.00948074460029602,
+            -0.02017543464899063,
+            -0.007599983364343643,
+            0.012190749868750572,
+            -0.014533151872456074,
+            0.0024043817538768053,
+            -0.022893989458680153,
+            -0.0009761789697222412,
+            0.014695581048727036,
+            -0.0025368898641318083,
+            0.0066125839948654175,
+            0.005321698263287544,
+            0.00012549536768347025,
+            0.010104815475642681,
+            -0.014789619483053684,
+            0.004030812531709671,
+            -0.005377266090363264,
+            -0.004834410268813372,
+            -0.00637748884037137,
+            -0.0011712011182680726,
+            0.0009542724001221359,
+            -0.005556793417781591,
+            0.01991896703839302,
+            0.008091545663774014,
+            0.00435994565486908,
+            -0.005270404741168022,
+            -0.012831917963922024,
+            -0.026159673929214478,
+            0.0145758967846632,
+            -0.015447885729372501,
+            0.01528545655310154,
+            0.0013550026342272758,
+            -0.003092569299042225,
+            -0.014370722696185112,
+            0.010643396526575089,
+            0.0056337337009608746,
+            0.00474037230014801,
+            0.015114477835595608,
+            -0.011481190100312233,
+            0.009583331644535065,
+            0.009147336706519127,
+            -0.009267021901905537,
+            -0.0067493668757379055,
+            0.0005936151137575507,
+            0.006279176566749811,
+            -0.0232701413333416,
+            -0.0016862730262801051,
+            0.008292445912957191,
+            -0.01962830312550068,
+            0.008762635290622711,
+            -0.03730745613574982,
+            0.0020068571902811527,
+            -0.012088162824511528,
+            0.011258917860686779,
+            -0.002776259556412697,
+            -0.033374954015016556,
+            -0.026518728584051132,
+            0.0016745183384045959,
+            0.0040179891511797905,
+            -0.009993678890168667,
+            0.009010554291307926,
+            0.009275570511817932,
+            0.011532483622431755,
+            0.01443056482821703,
+            0.0098397983238101,
+            0.010113364085555077,
+            0.04223163053393364,
+            -0.04151352122426033,
+            -0.007146891206502914,
+            0.0070656766183674335,
+            0.02209039032459259,
+            -0.010267244651913643,
+            -0.011361504904925823,
+            0.008685695938766003,
+            -0.003859834047034383,
+            0.034349530935287476,
+            -0.024877335876226425,
+            0.014994793571531773,
+            -0.0040543219074606895,
+            -0.00010605995339574292,
+            -0.024159228429198265,
+            -0.0020325039513409138,
+            -0.01969669573009014,
+            0.0032357636373490095,
+            -0.002079523168504238,
+            -0.017191864550113678,
+            0.018021108582615852,
+            -0.004595040809363127,
+            0.012789173983037472,
+            0.007813706062734127,
+            -0.01625148393213749,
+            0.004526649136096239,
+            -0.015593216754496098,
+            0.032896216958761215,
+            -0.012105260975658894,
+            0.005911573302000761,
+            0.00873698852956295,
+            -0.006779287941753864,
+            0.016439558938145638,
+            0.009352510794997215,
+            -0.006044081412255764,
+            -0.0051592690870165825,
+            -0.002759161638095975,
+            -0.0023680487647652626,
+            0.008493345230817795,
+            -0.021474869921803474,
+            0.027493303641676903,
+            -0.006894698366522789,
+            -0.014037314802408218,
+            0.006266353186219931,
+            -0.001722605898976326,
+            0.002626653527840972,
+            0.004406964406371117,
+            -0.0018059578724205494,
+            -0.0007939802599139512,
+            -0.004958369303494692,
+            -0.017798837274312973,
+            -0.010130461305379868,
+            0.0010467075044289231,
+            -0.0016114701284095645,
+            0.013105483725667,
+            -0.010540809482336044,
+            -0.016601989045739174,
+            0.005112249869853258,
+            0.00026929075829684734,
+            -0.003975244704633951,
+            -0.016807163134217262,
+            0.007215282414108515,
+            -0.007544415537267923,
+            0.005372991785407066,
+            -0.008168485946953297,
+            0.016448108479380608,
+            0.008313817903399467,
+            -0.0003630616411101073,
+            0.01767915114760399,
+            0.024706358090043068,
+            0.02061997726559639,
+            0.01152393501251936,
+            -0.040214087814092636,
+            -0.0189443901181221,
+            -0.0025091059505939484,
+            0.018807608634233475,
+            -0.0008981701685115695,
+            -0.006420233752578497,
+            -0.0017632133094593883,
+            0.0029835705645382404,
+            -0.003084020223468542,
+            -0.0007474955637007952,
+            0.016918297857046127,
+            -0.020192531868815422,
+            0.0032742335461080074,
+            -0.012720782309770584,
+            0.014404918067157269,
+            0.0006860502762719989,
+            -0.0006737611838616431,
+            5.767176844528876e-05,
+            0.00945509783923626,
+            0.014900755137205124,
+            0.014661385677754879,
+            0.0014159136917442083,
+            0.019064076244831085,
+            -0.003663209266960621,
+            -0.0016510088462382555,
+            -0.018636630848050117,
+            -0.01696104370057583,
+            -0.0034494863357394934,
+            0.01298579853028059,
+            -0.0015601766062900424,
+            0.011575227603316307,
+            -0.014191195368766785,
+            0.01649085246026516,
+            0.025680935010313988,
+            -0.012840467505156994,
+            0.005069505423307419,
+            0.002564673777669668,
+            -0.001211808412335813,
+            0.0064800758846104145,
+            -0.007381986360996962,
+            -0.004030812531709671,
+            -0.012438667938113213,
+            0.01868792437016964,
+            -0.0032208028715103865,
+            0.018277576193213463,
+            0.013635516166687012,
+            0.011968477629125118,
+            -0.00032993461354635656,
+            0.008305269293487072,
+            0.018636630848050117,
+            0.014413466677069664,
+            -0.0005158734857104719,
+            0.0011615835828706622,
+            -0.01284901611506939,
+            0.0023594999220222235,
+            0.016311325132846832,
+            -0.02715134806931019,
+            -0.007685472723096609,
+            0.011840244755148888,
+            -0.016567792743444443,
+            0.0025603994727134705,
+            -0.02597159706056118,
+            -0.011472641490399837,
+            0.012053967453539371,
+            0.010233048349618912,
+            -0.011053744703531265,
+            0.001883966731838882,
+            0.001705508097074926,
+            -0.0008377934573218226,
+            -0.00936105940490961,
+            0.0167815163731575,
+            -0.005868828855454922,
+            -0.011258917860686779,
+            0.011686364188790321,
+            -0.00981415156275034,
+            0.012695135548710823,
+            0.004603589419275522,
+            0.011746206320822239,
+            -0.0024578124284744263,
+            0.026877781376242638,
+            0.01287466287612915,
+            0.007236654870212078,
+            0.0004386660875752568,
+            0.005937220063060522,
+            -0.004847233649343252,
+            0.007407633122056723,
+            0.026570022106170654,
+            -0.00048728802357800305,
+            -0.0116350706666708,
+            -0.0013197384541854262,
+            0.010951157659292221,
+            -0.01988477073609829,
+            0.0023124809376895428,
+            0.005676478147506714,
+            0.028211412951350212,
+            0.014327977783977985,
+            0.0025796345435082912,
+            -0.0204148031771183,
+            -0.00023709874949418008,
+            -0.00041595802758820355,
+            0.04106897860765457,
+            0.0026116929948329926,
+            -0.012532706372439861,
+            0.0021489830687642097,
+            0.013849238865077496,
+            -0.005590988788753748,
+            -0.004834410268813372,
+            -0.014310880564153194,
+            -0.008027428761124611,
+            -0.006018434651196003,
+            -0.0198676735162735,
+            0.001279131043702364,
+            0.005800437182188034,
+            -0.013319206424057484,
+            0.007471750024706125,
+            -0.006403135601431131,
+            -0.03031444549560547,
+            -0.005932945758104324,
+            -0.0034131535794585943,
+            -0.023731783032417297,
+            -0.007125518750399351,
+            0.0089678093791008,
+            0.005915847606956959,
+            -0.014122803695499897,
+            -0.025338977575302124,
+            0.020996129140257835,
+            0.007668375037610531,
+            0.03392208740115166,
+            -0.012669488787651062,
+            -0.007185361348092556,
+            -0.00948074460029602,
+            0.008557462133467197,
+            -0.023150455206632614,
+            -0.008715616539120674,
+            0.009061847813427448,
+            0.01767915114760399,
+            0.012404472567141056,
+            -0.011370054446160793,
+            0.0015163634670898318,
+            0.012122358195483685,
+            -0.00010085045505547896,
+            0.02764718420803547,
+            -0.009104591794312,
+            -0.014994793571531773,
+            0.010754532180726528,
+            -0.003259273013100028,
+            0.01916666328907013,
+            -0.0006203305092640221,
+            -0.0025924579240381718,
+            0.012917407788336277,
+            0.0038341875188052654,
+            -0.012934505008161068,
+            -0.005240483675152063,
+            0.008647225797176361,
+            -0.028553368523716927,
+            0.02792074903845787,
+            0.015909526497125626,
+            -0.02816011942923069,
+            -0.008305269293487072,
+            -0.0025091059505939484,
+            0.017140571027994156,
+            0.020089944824576378,
+            -0.011446994729340076,
+            -0.009728662669658661,
+            0.014960597269237041,
+            0.0034217024222016335,
+            0.0077538639307022095,
+            -0.0026565745938569307,
+            0.01896148920059204,
+            -0.004370631650090218,
+            -0.007800883147865534,
+            -0.003408879041671753,
+            -0.012789173983037472,
+            -0.0003505054337438196,
+            0.015618863515555859,
+            0.012977249920368195,
+            -0.005984238814562559,
+            -0.008771184831857681,
+            0.015208516269922256,
+            -0.022432347759604454,
+            0.0010841090697795153,
+            0.010908412747085094,
+            -0.01745687983930111,
+            -0.007561513222754002,
+            0.01767915114760399,
+            -0.0031395882833749056,
+            0.011258917860686779,
+            -0.007381986360996962,
+            0.007856450974941254,
+            0.0067237201146781445,
+            0.031135139986872673,
+            -0.007476024329662323,
+            0.03525571525096893,
+            -0.009309765882790089,
+            0.01991896703839302,
+            0.037991367280483246,
+            0.01455025002360344,
+            -0.0031118043698370457,
+            -0.008335189893841743,
+            0.009198630228638649,
+            -0.010096265934407711,
+            0.002415067981928587,
+            0.013524380512535572,
+            -0.026809390634298325,
+            -0.01418264675885439,
+            -0.011378603056073189,
+            0.013233717530965805,
+            0.0034580351784825325,
+            -0.013661162927746773,
+            -0.011575227603316307,
+            0.019508618861436844,
+            0.010352733545005322,
+            -0.0038705202750861645,
+            -0.007869274355471134,
+            -0.02157745696604252,
+            0.0020688369404524565,
+            0.012088162824511528,
+            -0.009164434857666492,
+            -0.011532483622431755,
+            0.016670379787683487,
+            -0.01261819526553154,
+            0.0027228286489844322,
+            -0.008300994522869587,
+            -0.027989141643047333,
+            -0.005800437182188034,
+            -0.00491989916190505,
+            -0.0019844165071845055,
+            -0.005565342493355274,
+            0.006471527274698019,
+            0.005227660294622183,
+            -0.01824337989091873,
+            0.0024278913624584675,
+            -0.008788282051682472,
+            0.00861302949488163,
+            0.00436635734513402,
+            0.006249255500733852,
+            -0.0019384659826755524,
+            0.014225390739738941,
+            0.013661162927746773,
+            -0.005377266090363264,
+            -0.020773857831954956,
+            0.012772075831890106,
+            -0.03634997829794884,
+            0.020722564309835434,
+            0.008694244548678398,
+            0.0025411644019186497,
+            -0.03086157515645027,
+            0.01018175482749939,
+            0.01115633174777031,
+            0.0160035640001297,
+            0.011370054446160793,
+            0.0016766554908826947,
+            0.013190972618758678,
+            -0.01114778220653534,
+            0.006766464561223984,
+            -7.192273642431246e-06,
+            -0.01482381485402584,
+            0.020688369870185852,
+            0.011814597994089127,
+            0.02913469448685646,
+            0.002951512113213539,
+            -0.0011209761723876,
+            0.001963044051080942,
+            0.00899345614016056,
+            -0.007685472723096609,
+            0.0030904319137334824,
+            0.004954094998538494,
+            0.0058175353333354,
+            0.013190972618758678,
+            0.007745314855128527,
+            -0.00860875565558672,
+            0.007839352823793888,
+            0.0026736725121736526,
+            -0.014020216651260853,
+            -0.000775279535446316,
+            -0.006997285410761833,
+            0.007552964612841606,
+            -0.01505463570356369,
+            0.004560844972729683,
+            0.04342847689986229,
+            0.012413021177053452,
+            -0.0034323884174227715,
+            -0.017354292795062065,
+            -0.011079391464591026,
+            -0.01273788046091795,
+            -0.01272933091968298,
+            -0.0053644427098333836,
+            -0.02303077094256878,
+            -0.003434525802731514,
+            0.0065912120044231415,
+            0.02453538030385971,
+            0.01092551089823246,
+            -0.017191864550113678,
+            -0.005813260562717915,
+            -0.02380017377436161,
+            -0.010566456243395805,
+            -0.016191640868782997,
+            -0.016319874674081802,
+            -0.008061625063419342,
+            0.018978586420416832,
+            -0.008839575573801994,
+            0.01408860832452774,
+            0.002968610031530261,
+            -0.0002393695613136515,
+            -0.0020431901793926954,
+            -0.000852754048537463,
+            -0.0051549943163990974,
+            0.0015911663649603724,
+            0.003152411663904786,
+            -0.0005375128821469843,
+            0.020722564309835434,
+            -0.007800883147865534,
+            0.003866245737299323,
+            -0.0026929075829684734,
+            0.01542223896831274,
+            0.007108421064913273,
+            0.0028681603725999594,
+            -0.0238172709941864,
+            0.02304787002503872,
+            -0.024381499737501144,
+            -0.014156999997794628,
+            0.02284269593656063,
+            -0.008108643814921379,
+            0.004804489202797413,
+            0.024107934907078743,
+            0.00029493749025277793,
+            -0.0009697672794573009,
+            0.015123026445508003,
+            0.004045772831887007,
+            0.011258917860686779,
+            0.015242711640894413,
+            0.007082774303853512,
+            -0.0052875024266541,
+            0.019098270684480667,
+            0.0073306928388774395,
+            -0.006403135601431131,
+            -0.01626003161072731,
+            0.0014244626509025693,
+            0.004238123539835215,
+            -0.010044972412288189,
+            -0.004137673880904913,
+            0.008630127646028996,
+            0.004453983623534441,
+            -0.007843627594411373,
+            -0.0012972975382581353,
+            -0.024706358090043068,
+            -0.005821809638291597,
+            -0.01530255377292633,
+            -0.006039807107299566,
+            -0.023406922817230225,
+            0.005676478147506714,
+            -0.009053298272192478,
+            0.01672167330980301,
+            0.00016403225890826434,
+            0.024347303435206413,
+            0.0020154062658548355,
+            -0.0007266575703397393,
+            -0.0051592690870165825,
+            0.009318315424025059,
+            -0.0007266575703397393,
+            0.027236836031079292,
+            -0.00011026761785615236,
+            -0.005911573302000761,
+            0.01032708678394556,
+            -0.01769625023007393,
+            0.020876444876194,
+            0.006356116849929094,
+            0.01454170048236847,
+            -0.015610314905643463,
+            -0.0073563395999372005,
+            0.035289913415908813,
+            0.024843141436576843,
+            0.0035029170103371143,
+            0.019816379994153976,
+            -0.010455320589244366,
+            0.016311325132846832,
+            0.003013491863384843,
+            -0.002389421220868826,
+            0.011010999791324139,
+            -0.005420010536909103,
+            -0.014627190306782722,
+            -0.006997285410761833,
+            0.03911982476711273,
+            -0.014832363463938236,
+            -6.972707342356443e-05,
+            -0.005060956347733736,
+            -0.010104815475642681,
+            -0.030929965898394585,
+            0.011609423905611038,
+            0.011951380409300327,
+            -0.024706358090043068,
+            -0.011566678993403912,
+            0.0032699592411518097,
+            -0.00026448199059814215,
+            -0.0030348640866577625,
+            -0.005706399213522673,
+            0.006509996950626373,
+            -0.0017375665483996272,
+            0.009275570511817932,
+            0.005569616798311472,
+            -0.0009617527248337865,
+            0.00013798479631077498,
+            0.005539695732295513,
+            0.011780401691794395,
+            0.006206510588526726,
+            0.0005498019745573401,
+            -0.007381986360996962,
+            0.00558671448379755,
+            0.010745983570814133,
+            -0.015088831074535847,
+            0.020739663392305374,
+            0.013823592104017735,
+            -0.016576342284679413,
+            0.0038149524480104446,
+            -0.019132466986775398,
+            -0.0329304113984108,
+            0.004787391051650047,
+            -0.0009029789362102747,
+            0.021748434752225876,
+            -0.018311770632863045,
+            -0.00995948351919651,
+            0.00442833686247468,
+            0.003908990416675806,
+            -0.011421347968280315,
+            0.019371837377548218,
+            0.007215282414108515,
+            -0.0017664191545918584,
+            0.01745687983930111,
+            0.016593439504504204,
+            -0.006403135601431131,
+            -0.00907894503325224,
+            0.018824705854058266,
+            -0.00437276903539896,
+            0.02744201011955738,
+            0.026621313765645027,
+            0.009207178838551044,
+            -0.018602434545755386,
+            0.0018262615194544196,
+            0.0080231549218297,
+            -0.02036350965499878,
+            -0.005590988788753748,
+            0.015670157968997955,
+            0.005402912851423025,
+            -0.0003841667785309255,
+            -0.0050737797282636166,
+            -0.022175880149006844,
+            0.007407633122056723,
+            0.011840244755148888,
+            -0.005189190153032541,
+            -0.028809836134314537,
+            -0.0011199075961485505,
+            -0.01431942917406559,
+            -0.0007694021333009005,
+            0.0010648739989846945,
+            0.01236172765493393,
+            0.009036201052367687,
+            0.028074629604816437,
+            -0.009865445084869862,
+            -0.00885667372494936,
+            0.010173206217586994,
+            0.009797054342925549,
+            0.012583999894559383,
+            0.012327532283961773,
+            0.013986021280288696,
+            0.007292222697287798,
+            -0.015960820019245148,
+            -0.004966918379068375,
+            0.007527317851781845,
+            -0.00899345614016056,
+            0.003985930699855089,
+            -0.01358422264456749,
+            0.020278021693229675,
+            0.015259809792041779,
+            0.0008260387112386525,
+            0.004804489202797413,
+            -0.004753195680677891,
+            -0.0012684449320659041,
+            -0.06483495235443115,
+            -0.013532929122447968,
+            0.00861302949488163,
+            0.021252596750855446,
+            -0.016396814957261086,
+            0.00046244275290519,
+            0.0014287370722740889,
+            0.008788282051682472,
+            -0.011481190100312233,
+            0.01020740158855915,
+            0.014524603262543678,
+            -0.005223385989665985,
+            -0.0035349754616618156,
+            0.005099426489323378,
+            -0.0016403226181864738,
+            -0.00013371034583542496,
+            0.004958369303494692,
+            0.014900755137205124,
+            0.021406477317214012,
+            -0.004385592415928841,
+            0.018585337325930595,
+            0.004112027119845152,
+            -0.015080282464623451,
+            -0.0027014564257115126,
+            0.013575674034655094,
+            -0.02328723855316639,
+            -0.004607864189893007,
+            0.007480298634618521,
+            -0.006980187259614468,
+            0.010908412747085094,
+            -0.004069282673299313,
+            -0.004406964406371117,
+            0.008677146397531033,
+            0.01420829351991415,
+            0.002276147948578,
+            -0.026775194332003593,
+            0.0034665842540562153,
+            -0.011233271099627018,
+            0.0035093287006020546,
+            0.01020740158855915,
+            -0.008219779469072819,
+            1.5344630810432136e-05,
+            -0.015328200533986092,
+            -0.002739926567301154,
+            0.024860238656401634,
+            -0.004289417061954737,
+            0.0067707388661801815,
+            0.017302999272942543,
+            -0.007018657401204109,
+            0.002389421220868826,
+            -0.02862176112830639,
+            0.007796608377248049,
+            -0.0369655005633831,
+            -0.0004579011583700776,
+            0.004984016064554453,
+            0.004097066354006529,
+            -0.010104815475642681,
+            -0.030211858451366425,
+            0.008412130177021027,
+            -0.00474037230014801,
+            -0.006753641180694103,
+            -0.022723009809851646,
+            -0.007937666028738022,
+            0.008429228328168392,
+            -0.006941717118024826,
+            0.010147559456527233,
+            -0.017662053927779198,
+            0.017815934494137764,
+            0.015439336188137531,
+            0.013302108272910118,
+            0.003968833014369011,
+            0.0032379007898271084,
+            0.005129347555339336,
+            0.007946214638650417,
+            0.004701902158558369,
+            0.004697627853602171,
+            -0.01019030436873436,
+            -0.011464091949164867,
+            0.01575564593076706,
+            0.004255221225321293,
+            0.00860875565558672,
+            -0.0018604571232572198,
+            0.002479184651747346,
+            -0.0073050460778176785,
+            -0.00045629823580384254,
+            -0.014396369457244873,
+            0.0021596690639853477,
+            0.01199412439018488,
+            0.015499179251492023,
+            0.0014650699449703097,
+            -0.00625352980569005,
+            -0.007634179200977087,
+            -0.00667670089751482,
+            0.020209630951285362,
+            -0.017149118706583977,
+            -0.006698073353618383,
+            -0.012635293416678905,
+            0.021526163443922997,
+            0.01599501632153988,
+            0.0035563476849347353,
+            -0.006826307158917189,
+            -0.004092792049050331,
+            0.0097030159085989,
+            0.008570285513997078,
+            0.006766464561223984,
+            -0.023663390427827835,
+            -0.005377266090363264,
+            -0.019081173464655876,
+            -0.01539659220725298,
+            -0.01406296156346798,
+            0.021218402311205864,
+            -0.004924173932522535,
+            -0.007471750024706125,
+            0.008454875089228153,
+            0.024655064567923546,
+            0.007916293106973171,
+            0.007715393789112568,
+            0.00503958435729146,
+            -0.01726025529205799,
+            0.012071064673364162,
+            -0.010139010846614838,
+            -0.03665773943066597,
+            0.0065912120044231415,
+            -0.00971156544983387,
+            0.02598869428038597,
+            -0.017183315008878708,
+            -0.0011167017510160804,
+            -0.002635202370584011,
+            -0.030878672376275063,
+            -0.005638008005917072,
+            0.02961343340575695,
+            -0.010737434960901737,
+            0.011669266037642956,
+            -0.010062070563435555,
+            -0.009950934909284115,
+            0.01019030436873436,
+            0.013062738813459873,
+            -0.011720559559762478,
+            -0.002940826117992401,
+            0.014721227809786797,
+            0.02453538030385971,
+            0.0035093287006020546,
+            -0.004548021592199802,
+            0.009429451078176498,
+            0.0123702771961689,
+            -0.006804934702813625,
+            0.019559912383556366,
+            -0.007561513222754002,
+            -0.0015078145079314709,
+            -0.00656556524336338,
+            0.0012235631002113223,
+            0.006193687207996845,
+            -0.019577009603381157,
+            0.017320098355412483,
+            -0.009916738606989384,
+            0.008873771876096725,
+            0.0028702975250780582,
+            0.01726025529205799,
+            0.011267467401921749,
+            0.01896148920059204,
+            -0.02891242317855358,
+            -0.001095329411327839,
+            -0.0276813805103302,
+            -0.015772745013237,
+            -0.010703238658607006,
+            -0.00873698852956295,
+            -0.002485596342012286,
+            -0.007796608377248049,
+            -0.023868564516305923,
+            -0.005625184625387192,
+            0.0007939802599139512,
+            0.0036781697999686003,
+            -0.017362842336297035,
+            0.01007061917334795,
+            1.5286190318875015e-05,
+            -0.016593439504504204,
+            -0.020825151354074478,
+            -0.002087003318592906,
+            -0.022021999582648277,
+            -0.012464314699172974,
+            -0.005005388520658016,
+            -0.0022248546592891216,
+            -0.0012139456812292337,
+            -0.0005172092351131141,
+            0.027544597163796425,
+            -0.019457325339317322,
+            0.004381317645311356,
+            0.02744201011955738,
+            -0.0019352601375430822,
+            0.005035309586673975,
+            -0.006668151821941137,
+            0.012532706372439861,
+            0.0032913314644247293,
+            -0.003973107319325209,
+            -0.022688815370202065,
+            0.02521929331123829,
+            0.0005311011918820441,
+            0.013669711537659168,
+            0.010139010846614838,
+            -0.008065898902714252,
+            -0.005783339496701956,
+            0.00485150795429945,
+            -0.03253716230392456,
+            0.0028531996067613363,
+            -0.013011445291340351,
+            -0.01865372806787491,
+            -0.005539695732295513,
+            -0.004586491733789444,
+            -0.006561290472745895,
+            -0.0071939099580049515,
+            0.017285902053117752,
+            -0.004377043340355158,
+            0.02376597747206688,
+            -0.007146891206502914,
+            -0.00453947251662612,
+            0.010908412747085094,
+            -0.04301812872290611,
+            -0.007240929175168276,
+            -0.006129570305347443,
+            -0.01843145675957203,
+            0.019799282774329185,
+            0.012173651717603207,
+            -0.001380649395287037,
+            -0.020243825390934944,
+            0.019816379994153976,
+            -0.0030028056353330612,
+            0.02552705444395542,
+            0.010634847916662693,
+            0.010173206217586994,
+            0.004364219959825277,
+            -0.010840021073818207,
+            -0.010113364085555077,
+            0.018311770632863045,
+            0.006304823327809572,
+            -0.011609423905611038,
+            0.01873921789228916,
+            0.004445434547960758,
+            -0.00186686881352216,
+            -0.003041275776922703,
+            0.018756315112113953,
+            0.024603771045804024,
+            0.013002896681427956,
+            0.002274010796099901,
+            -0.016174543648958206,
+            -0.006475801579654217,
+            0.0037038165610283613,
+            0.036418367177248,
+            0.012036869302392006,
+            0.012994347140192986,
+            -0.008202682249248028,
+            0.027373619377613068,
+            -0.009181532077491283,
+            0.009044749662280083,
+            0.00863440241664648,
+            0.017174765467643738,
+            0.0035328383091837168,
+            0.020329315215349197,
+            0.010857119224965572,
+            -0.00764272827655077,
+            0.02405664138495922,
+            0.007625630125403404,
+            0.003767933463677764,
+            0.006625407375395298,
+            -0.01844855397939682,
+            0.012626743875443935,
+            -0.011609423905611038,
+            -0.005950043443590403,
+            0.004505277145653963,
+            0.002058150712400675,
+            0.02374888025224209,
+            0.008027428761124611,
+            -0.03135741129517555,
+            -0.0017867228016257286,
+            -0.023612096905708313,
+            0.004340710584074259,
+            -0.0167387705296278,
+            0.012447217479348183,
+            0.004092792049050331,
+            -0.01774754375219345,
+            -0.018790511414408684,
+            -0.010728885419666767,
+            -0.01942313089966774,
+            -0.004924173932522535,
+            -0.012173651717603207,
+            -0.010840021073818207,
+            0.006954540498554707,
+            -0.0025069687981158495,
+            0.0024813220370560884,
+            0.006963089574128389,
+            0.019046977162361145,
+            0.020260922610759735,
+            0.0035819944459944963,
+            0.018824705854058266,
+            -0.0009019103017635643,
+            -0.055431149899959564,
+            0.0036589347291737795,
+            0.009309765882790089,
+            -0.0002863885893020779,
+            0.026108380407094955,
+            -0.004041498526930809,
+            0.00042557556298561394,
+            0.007223831489682198,
+            0.006937442813068628,
+            0.013276461511850357,
+            -0.004210339393466711,
+            -0.025886109098792076,
+            -0.019337641075253487,
+            0.0040393611416220665,
+            -0.006155217066407204,
+            0.008100095205008984,
+            -0.006685249973088503,
+            0.004761744290590286,
+            0.002353088231757283,
+            -0.009745760820806026,
+            -0.006595486309379339,
+            0.009617527015507221,
+            0.0014469034504145384,
+            -0.010386928915977478,
+            -0.013173874467611313,
+            -0.0018690060824155807,
+            0.01824337989091873,
+            -0.009019102901220322,
+            -0.024364402517676353,
+            0.009660271927714348,
+            -0.010010777041316032,
+            -0.0014821677468717098,
+            0.025869010016322136,
+            0.02429600991308689,
+            -0.003970969934016466,
+            0.019371837377548218,
+            -0.0021190617699176073,
+            -0.016405364498496056,
+            -0.03029734641313553,
+            -0.006873325910419226,
+            -0.028997913002967834,
+            0.027253935113549232,
+            -0.006424508057534695,
+            -0.0013176011852920055,
+            0.0035285637713968754,
+            -0.00413553649559617,
+            0.003434525802731514,
+            -0.009788504801690578,
+            -0.007672649342566729,
+            -0.02716844528913498,
+            0.007527317851781845,
+            -0.0069075217470526695,
+            -0.008172760717570782,
+            0.00031550831045024097,
+            0.039701152592897415,
+            -0.004556570667773485,
+            0.00850189384073019,
+            0.006232157349586487,
+            0.011865891516208649,
+            0.024193422868847847,
+            -0.008463423699140549,
+            -0.012096711434423923,
+            -0.015046087093651295,
+            -0.0020154062658548355,
+            0.00014052275219000876,
+            -0.003975244704633951,
+            0.022876890376210213,
+            -0.013019993901252747,
+            -0.015191418118774891,
+            -0.00034489520476199687,
+            -0.023389825597405434,
+            -0.011737657710909843,
+            0.009968032129108906,
+            0.012908858247101307,
+            0.007535866927355528,
+            -0.014003119431436062,
+            -0.014268135651946068,
+            -0.022945282980799675,
+            0.01800401136279106,
+            0.0976627841591835,
+            0.0012545529752969742,
+            -0.009070396423339844,
+            0.0026758096646517515,
+            0.010771630331873894,
+            -0.0008372591692022979,
+            -0.017559466883540154,
+            0.05919267237186432,
+            -0.013567124493420124,
+            0.005856005474925041,
+            -0.009010554291307926,
+            0.01007916871458292,
+            0.005031035281717777,
+            -0.00013257493264973164,
+            0.005928670987486839,
+            -0.008796831592917442,
+            0.006129570305347443,
+            -0.03645256534218788,
+            -0.00186686881352216,
+            -0.011344407685101032,
+            -0.006381763610988855,
+            -0.016824260354042053,
+            0.009275570511817932,
+            -0.02911759726703167,
+            -0.00922427698969841,
+            0.0020880720112472773,
+            -0.0059543177485466,
+            0.012336080893874168,
+            -0.01628567837178707,
+            -0.0017995461821556091,
+            0.005368717480450869,
+            0.005120798945426941,
+            -0.0036867186427116394,
+            -0.0074632009491324425,
+            -0.013310656882822514,
+            0.005065230652689934,
+            0.015576119534671307,
+            0.005548244342207909,
+            -0.008228329010307789,
+            0.00959188025444746,
+            0.016388265416026115,
+            0.0009510665549896657,
+            0.013661162927746773,
+            0.009412352927029133,
+            0.0007950488943606615,
+            0.0007875685696490109,
+            0.019611205905675888,
+            -0.016345521435141563,
+            0.02137228287756443,
+            -0.015883879736065865,
+            -0.02843368425965309,
+            -0.006236432120203972,
+            0.006411684677004814,
+            -0.014413466677069664,
+            0.0010910549899563193,
+            0.006582662928849459,
+            0.04100058600306511,
+            0.00863440241664648,
+            -0.0034281141124665737,
+            -0.0009259541402570903,
+            -0.013669711537659168,
+            0.023954054340720177,
+            0.01091696135699749,
+            0.030502520501613617,
+            0.007604258134961128,
+            0.00373373762704432,
+            -0.007001559715718031,
+            -0.009557684883475304,
+            -0.003487956477329135,
+            0.00474037230014801,
+            0.0025069687981158495,
+            0.004415513481944799,
+            0.008523265831172466,
+            -0.014498956501483917,
+            -0.02036350965499878,
+            0.017610760405659676,
+            -0.00474037230014801,
+            -0.023663390427827835,
+            0.012609646655619144,
+            -0.0316309779882431,
+            0.0047831167466938496,
+            0.0010344184702262282,
+            -0.005877377465367317,
+            -0.013960374519228935,
+            0.006300548557192087,
+            0.01151538547128439,
+            -0.009378157556056976,
+            -0.015482081100344658,
+            -0.008454875089228153,
+            -0.010592103004455566,
+            0.0001227570464834571,
+            0.0037657960783690214,
+            -0.011583777144551277,
+            -0.00394318625330925,
+            -0.01796981506049633,
+            0.007142616901546717,
+            -0.003013491863384843,
+            0.004654882941395044,
+            -0.011122135445475578,
+            0.011558130383491516,
+            0.020226728171110153,
+            -0.009788504801690578,
+            -0.007069950923323631,
+            0.013105483725667,
+            0.003400330198928714,
+            0.016089053824543953,
+            0.03368271514773369,
+            -0.0033704089000821114,
+            0.0033511738292872906,
+            0.003849148051813245,
+            0.0058731031604111195,
+            0.0008874839986674488,
+            0.015909526497125626,
+            0.03621319308876991,
+            0.01942313089966774,
+            -0.0250141192227602,
+            0.02186811901628971,
+            -0.019713792949914932,
+            0.021936509758234024,
+            -0.02231266349554062,
+            -0.0030797459185123444,
+            0.007005834020674229,
+            -0.007394809741526842,
+            0.004146222956478596,
+            0.00624070642516017,
+            0.017627857625484467,
+            0.022363955155014992,
+            0.000617124664131552,
+            -0.0003163097717333585,
+            0.004860057029873133,
+            0.021953608840703964,
+            -0.0025411644019186497,
+            0.007125518750399351,
+            -0.013421793468296528,
+            0.004411239176988602,
+            -0.007202459033578634,
+            0.0021767669823020697,
+            0.008395032025873661,
+            -0.014952048659324646,
+            0.013344853185117245,
+            0.012891761027276516,
+            -0.017302999272942543,
+            0.0043342988938093185,
+            0.0025945950765162706,
+            -0.018568238243460655,
+            -0.018978586420416832,
+            0.023372728377580643,
+            -0.013635516166687012,
+            -0.012660940177738667,
+            -0.009891091845929623,
+            -0.01152393501251936,
+            -0.0201412383466959,
+            -0.005056682042777538,
+            -0.0017471840837970376,
+            0.005753418430685997,
+            -0.0024449890479445457,
+            0.006467252504080534,
+            0.01573854871094227,
+            0.005612361244857311,
+            -0.006437331438064575,
+            -0.005561067722737789,
+            -0.0015163634670898318,
+            0.020107043907046318,
+            0.005013937596231699,
+            0.0026458885986357927,
+            -0.04031667485833168,
+            0.0013143953401595354,
+            -0.010540809482336044,
+            0.006428782362490892,
+            0.0031417254358530045,
+            -0.004971192684024572,
+            -0.01942313089966774,
+            -0.014755423180758953,
+            0.01872211880981922,
+            -0.0103014400228858,
+            0.024637967348098755,
+            4.564986011246219e-05,
+            -0.012053967453539371,
+            -0.0018999958410859108,
+            0.003960283938795328,
+            0.0016029211692512035,
+            0.004415513481944799,
+            -0.00800178200006485,
+            0.016174543648958206,
+            -0.011694912798702717,
+            0.010908412747085094,
+            -0.00401157746091485,
+            -0.024706358090043068,
+            0.010891314595937729,
+            -0.01744833029806614,
+            0.0004752661334350705,
+            -0.00666387751698494,
+            -0.030040880665183067,
+            0.018551141023635864,
+            -0.023731783032417297,
+            -0.0033233899157494307,
+            0.009643173776566982,
+            0.018841803073883057,
+            0.0017461155075579882,
+            -0.010976804420351982,
+            0.00630909763276577,
+            0.017362842336297035,
+            0.013481635600328445,
+            -0.004062870983034372,
+            0.012438667938113213,
+            -0.004971192684024572,
+            0.0025603994727134705,
+            0.005172092467546463,
+            0.008399306796491146,
+            -0.006804934702813625,
+            0.007805157452821732
+          ],
+          "keyphrases": [
+            "DeepSeekMoE",
+            "multilingual corpus",
+            "HuggingFace Tokenizer",
+            "MoE architectures",
+            "evaluation benchmarks"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": true,
+      "properties": {
+        "cosine_similarity": 0.8725451451557741
+      }
+    },
+    {
+      "id": "e0986b19-b408-4e2b-994d-58878543d654",
+      "type": "cosine_similarity",
+      "source": {
+        "id": "1e687808-8db8-4320-805d-0282a8d679bc",
+        "properties": {
+          "page_content": "Deepseekmoe: Towards Ultimate Expert Specialization In\n\nMixture-Of-Experts Language Models\n\nDamai Dai∗1,2, Chengqi Deng1, Chenggang Zhao∗1,3, R.X. Xu1, Huazuo Gao1, Deli Chen1, Jiashi Li1, Wangding Zeng1, Xingkai Yu∗1,4, Y. Wu1, Zhenda Xie1, Y.K. Li1, Panpan Huang1, Fuli Luo1, Chong Ruan1, Zhifang Sui2, Wenfeng Liang1 1DeepSeek-AI 2National Key Laboratory for Multimedia Information Processing, Peking University 3Institute for Interdisciplinary Information Sciences, Tsinghua University 4National Key Laboratory for Novel Software Technology, Nanjing University {daidamai, szf}@pku.edu.cn, {wenfeng.liang}@deepseek.com https://github.com/deepseek-ai/DeepSeek-MoE\n\nAbstract\n\nIn the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top- out of experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into ones and activating from them, allowing for a more flexible combination of activated experts; (2) isolating experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5× expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations.\n\nFurther, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\n1. Introduction\n\nRecent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023a). It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017) has emerged as a popular solution. It can\n\nenable parameter scaling, while concurrently keeping computational costs at a modest level.\n\nRecent applications of MoE architectures in Transformers (Vaswani et al., 2017) have yielded successful attempts at scaling language models to a substantial size (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022), accompanied with remarkable performance. These achievements underscore the considerable potential and promise of MoE language models.\n\nDespite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts.\n\nThis architecture manifests two potential issues: (1) Knowledge Hybridity: existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. (2) Knowledge Redundancy: tokens assigned to different experts may require common knowledge. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters.\n\nThese issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models.\n\nIn response to the aforementioned issues, we introduce DeepSeekMoE, an innovative MoE architecture specifically designed towards ultimate expert specialization. Our architecture involves two principal strategies: (1) Fine-Grained Expert Segmentation: while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. (2) Shared Expert Isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.\n\nThrough compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized.\n\nStarting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B (Lepikhin et al., 2021) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5× expert parameters and computation. Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE.\n\nThese studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization.\n\nLeveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B (DeepSeek-AI, 2024), a dense model trained on the same 2T corpus. We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B (Touvron et al., 2023b), which has approximately 2.5 times the activated parameters. Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard1. Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. The experimental results still validate its substantial advantages over the GShard architecture consistently. In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nOur contributions are summarized as follows: - Architectural Innovation. We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation.\n\nEmpirical Validation. We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. Experimental results validate the high 1https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models\n\nScalability. We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B.\n\nAlignment for MoE. We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B.\n\nPublic Release. In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization.\n\n2. Preliminaries: Mixture-Of-Experts For Transformers\n\nWe first introduce a generic MoE architecture commonly used in Transformer language models. A standard Transformer language model is constructed by stacking layers of standard Transformer blocks, where each block can be represented as follows:\n\n$$\\mathbf{u}{1:T}^{l}=\\text{Self-Att}\\left(\\mathbf{h}{1:T}^{l-1}\\right)+\\mathbf{h}{1:T}^{l-1},\\tag{1}$$ $$\\mathbf{h}{t}^{l}=\\text{FFN}\\left(\\mathbf{u}{t}^{l}\\right)+\\mathbf{u}{t}^{l},\\tag{2}$$\n\nwhere denotes the sequence length, Self-Att(·) denotes the self-attention module, FFN(·) denotes the Feed-Forward Network (FFN), u 1: ∈ R× are the hidden states of all tokens after the -th attention module, and h\n\n∈ Ris the output hidden state of the -th token after the -th Transformer block. For brevity, we omit the layer normalization in the above formulations.\n\nA typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022). An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. Then, each token will be assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts. If the -th FFN is substituted with an MoE layer, the computation for its output hidden state h\n\nis expressed as: where denotes the total number of experts, FFN(·) is the -th expert FFN, , denotes the gate value for the -th expert, , denotes the token-to-expert affinity, Topk(·, ) denotes the set comprising highest affinity scores among those calculated for the -th token and all experts, and e\n\nis the centroid of the -th expert in the -th layer. Note that ,is sparse, indicating that only out of gate values are nonzero. This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only experts. Also, in the above formulations, we omit the layer normalization operation for brevity.\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{N}\\left(g_{i,t}\\operatorname{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\operatorname{Topk}({s_{j,t}|1\\leqslant j\\leqslant N},K),\\ 0,&\\text{otherwise},\\end{cases}$$ $$s_{i,t}=\\operatorname{Softmax}{i}\\left(\\mathbf{u}{t}^{l}\\,\\mathbf{e}_{i}^{l}\\right),$$\n\n$$(3)$$\n\n(4) $\\binom{4}{5}$ .\n\n3. Deepseekmoe Architecture\n\nOn top of the generic MoE architecture outlined in Section 2, we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. As illustrated in Figure 2, our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. Both of these strategies are designed to elevate the level of expert specialization.\n\n3.1. Fine-Grained Expert Segmentation\n\nIn scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts.\n\nIn pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. The finer expert segmentation enables a more flexible and adaptable combination of activated experts. To be specific, on top of a typical MoE architecture shown in Figure 2(a), we segment each expert FFN into smaller experts by reducing the FFN intermediate hidden dimension to 1 times its original size. Since each expert becomes smaller, in response, we also increase the number of activated experts to times to keep the same computation cost, as illustrated in Figure 2(b). With the fine-grained expert segmentation, the output of an MoE layer can be expressed as:\n\n$$\\mathbf{h}{t}^{i}=\\sum{i=1}^{\\text{mN}}\\left(g_{i,t}\\text{FFN}{i}\\left(\\mathbf{u}{t}^{i}\\right)\\right)+\\mathbf{u}{t}^{i},\\tag{6}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\text{Tokr}(s_{i,t}|1\\leqslant j\\leqslant m\\text{N}),m\\text{K}),\\ 0,&\\text{otherwise},\\end{cases}$$ (7) $$s_{i,t}=\\text{Softmax}{i}\\left(\\mathbf{u}{t}^{T}\\mathbf{e}_{t}^{i}\\right),\\tag{8}$$ where the total number of expert parameters is equal to $N$ times the number of parameters in a\n\n$$(6)$$\n\n(7) (8) $\\frac{1}{2}$\n\nstandard FFN, and denotes the total number of fine-grained experts. With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to .\n\nFrom a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. As an illustrative example, we consider the case where = 16. A typical top-2 routing strategy can yield 16 2 = 120 possible combinations. By contrast, if each expert is split into 4 smaller experts, the fine-grained routing strategy can yield 64 8 = 4, 426, 165, 368 potential combinations. The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition.\n\n3.2. Shared Expert Isolation\n\nWith a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. This alleviation of redundancy will contribute to a more parameterefficient model with more specialized experts.\n\nTowards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate experts to serve as shared experts. Regardless of the router module, each token will be deterministically assigned to these shared experts. In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by , as depicted in Figure 2(c). With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows:\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{K_{s}}\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)+\\sum_{i=K_{s}+1}^{mN}\\left(g_{i,t}\\;\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},\\tag{9}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\mathrm{Topk}{s_{j,t}|K_{s}+1\\leqslant j\\leqslant mN},mK-K_{s}},\\ 0,&\\text{otherwise},\\end{cases}$$ (10) $$s_{i,t}=\\mathrm{Softmax}{i}\\left(\\mathbf{u}{t}^{l,T}e_{i}^{l}\\right).\\tag{11}$$ $\\mathbf{h}\\mathbf{M}\\mathbf{E}$ the number of closed-compact is $K_{s}$ the total number of routed currents.\n\nFinally, in DeepSeekMoE, the number of shared expert is , the total number of routed experts is − , and the number of nonzero gates is − .\n\nIt is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022). The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint.\n\n3.3. Load Balance Consideration\n\nAutomatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. Firstly, there is a risk of routing collapse (Shazeer et al., 2017), i.e., the model always selects only a few experts, preventing other experts from sufficient training.\n\nSecondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.\n\nExpert-Level Balance Loss. In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. The computation of the balance loss is as follows:\n\n$$\\begin{array}{l}{{{\\mathcal L}{\\mathrm{ExpBal}}=\\alpha{1}\\sum_{i=1}^{N^{\\prime}}f_{i}P_{i},}}\\ {{f_{i}=\\frac{N^{\\prime}}{K^{T}}\\sum_{t=1}^{T}\\mathds{1}\\,(\\mathrm{Token}\\;t\\;\\mathrm{s}\\mathrm{lects}\\;\\mathrm{Expert}\\;i),}}\\ {{P_{i}=\\frac{1}{T}\\sum_{t=1}^{T}s_{i,t},}}\\end{array}$$\n\n(12) $$\\begin{array}{l}\\small\\text{(13)}\\end{array}$$ = (14) $$\\begin{array}{l}\\small\\text{(14)}\\end{array}$$ .\n\n7 where 1 is a hyper-parameter called expert-level balance factor, ′is equal to ( − ) and ′ is equal to ( − ) for brevity. 1(·) denotes the indicator function.\n\nDevice-Level Balance Loss. In addition to the expert-level balance loss, we introduce a devicelevel balance loss. When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. Instead, our primary objective is to ensure balanced computation across the devices. If we partition all routed experts into groups {E1, E2, ..., E}, and deploy each group on a single device, the device-level balance loss is computed as follows:\n\n$$\\mathcal{L}{\\text{DevBal}}=\\alpha{2}\\sum_{i=1}^{D}f_{i}^{\\prime}P_{i}^{\\prime},$$ $$f_{i}^{\\prime}=\\frac{1}{|\\mathcal{E}{i}|}\\sum{j\\in\\mathcal{E}{i}}f{j},$$ $$P_{i}^{\\prime}=\\sum_{j\\in\\mathcal{E}{i}}P{j},$$\n\n(15) $$\\begin{array}{l}\\small\\text{(16)}\\end{array}$$ = (17) .\n\nwhere 2 is a hyper-parameter called device-level balance factor. In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices.\n\n4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n6. Alignment For Deepseekmoe 16B\n\nPrevious research indicates that MoE models typically do not emerge significant gains from fine-tuning (Artetxe et al., 2022; Fedus et al., 2021). However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B.\n\n6.1. Experimental Setup\n\nTraining Data. For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios.\n\nHyper-Parameters. During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).\n\nWe employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10−5 without incorporating any learning rate scheduling strategy.\n\nEvaluation Benchmarks. For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3, with the following adjustments: (1) We exclude Pile (Gao et al., 2020) since chat models are seldom employed for pure language modeling. (2) We exclude\n\nMetric # Shot LLaMA2 DeepSeek DeepSeekMoE SFT 7B Chat 7B Chat 16B # Total Params N/A 6.7B 6.9B 16.4B # Activated Params N/A 6.7B 6.9B 2.8B FLOPs per 4K Tokens N/A 187.9T 183.5T 74.4T HellaSwag (Acc.) 0-shot 67.9 71.0 72.2 PIQA (Acc.) 0-shot 76.9 78.4 79.7 ARC-easy (Acc.) 0-shot 69.7 70.2 69.9 ARC-challenge (Acc.) 0-shot 50.8 50.2 50.0 BBH (EM) 3-shot 39.3 43.1 42.2 RACE-middle (Acc.) 5-shot 63.9 66.1 64.8 RACE-high (Acc.) 5-shot 49.6 50.8 50.6 DROP (EM) 1-shot 40.0 41.7 33.8 GSM8K (EM) 0-shot 63.4 62.6 62.2 MATH (EM) 4-shot 13.5 14.7 15.2 HumanEval (Pass@1) 0-shot 35.4 45.1 45.7 MBPP (Pass@1) 3-shot 27.8 39.0 46.2 TriviaQA (EM) 5-shot 60.1 59.5 63.3 NaturalQuestions (EM) 0-shot 35.2 32.7 35.1 MMLU (Acc.) 0-shot 50.0 49.7 47.2 WinoGrande (Acc.) 0-shot 65.1 68.4 69.0 CLUEWSC (EM) 5-shot 48.4 66.2 68.2 CEval (Acc.) 0-shot 35.1 44.7 40.0 CMMLU (Acc.) 0-shot 36.9 51.2 49.3\n\nCHID (Zheng et al., 2019) due to the observed instability of results, hindering the derivation of solid conclusions. (3) We additionally include BBH (Suzgun et al., 2022) to provide a more comprehensive assessment of the reasoning ability of the chat models.\n\nTable 5 | Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations.\n\n6.2. Evaluations\n\nBaselines. In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. Correspondingly, we construct three chat models, including LLaMA2 SFT 7B3, DeepSeek Chat 7B, and DeepSeekMoE Chat 16B.\n\nSubsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks.\n\nResults. The evaluation results are presented in Table 5. Our key observations include: (1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intensive tasks (TriviaQA, NaturalQuestions). (2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. In addition, it also surpasses DeepSeek Chat 7B. (3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1). However, it is worth noting that, after supervised finetuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. (4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations.\n\n7. Deepseekmoe 145B Ongoing\n\nEncouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available.\n\n7.1. Experimental Setup\n\nTraining Data and Tokenization. For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study.\n\nModel Settings. For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps.\n\nSubsequently, the learning rate keeps constant during the remaining training process. The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. Also, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse.\n\nEvaluation Benchmarks. We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3).\n\n7.2. Evaluations\n\nBaselines. Apart from DeepSeekMoE 145B, we consider three additional models for comparison. DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters.\n\nResults. From the evaluation results presented in Table 6, we have the following observations: (1) Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. (2) On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. (3) At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5.\n\n8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\nAppendices A. Overview Of Hyper-Parameters\n\nWe present the overview of hyper-parameters for DeepSeekMoE across various sizes in Table 7.\n\n# Params # Layers Hidden # Attn # Shared # Routed Relative Sequence Batch Size Learning Size Heads Experts Experts Expert Size Length (Sequence) Rate 2.0B 9 1280 10 1 63 (7 activated) 0.25 2048 2048 1.08e-3 16.4B 28 2048 16 2 64 (6 activated) 0.25 4096 4608 4.2e-4 144.6B 62 4096 32 4 128 (12 activated) 0.125 4096 4608 3.0e-4\n\nTable 7 | Overview of hyper-parameters for DeepSeekMoE across various sizes. The relative expert size is in comparison to a standard FFN.\n\nB. Comparing Deepseekmoe With Larger Models\n\nComparisons among DeepSeekMoE, GShard×1.2, and GShard×1.5 are shown in Table 8. Comparisons among DeepSeekMoE, Dense×4, and Dense×16 are shown in Table 9.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 2.3B 2.8B 1.9B # Activated Expert Params N/A 0.28B 0.35B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.824 1.808 1.808 HellaSwag (Acc.) 0-shot 53.7 54.4 54.8 PIQA (Acc.) 0-shot 71.8 71.1 72.3 ARC-easy (Acc.) 0-shot 46.8 47.3 49.4 ARC-challenge (Acc.) 0-shot 31.7 34.1 34.3 RACE-middle (Acc.) 5-shot 43.7 46.4 44.0 RACE-high (Acc.) 5-shot 31.9 32.4 31.7 HumanEval (Pass@1) 0-shot 3.7 3.0 4.9 MBPP (Pass@1) 3-shot 2.4 2.6 2.2 TriviaQA (EM) 5-shot 15.2 15.7 16.6 NaturalQuestions (EM) 5-shot 4.5 4.7 5.7\n\nTable 8 | Comparison between DeepSeekMoE and larger GShard models.\n\nAt a larger scale of 13B total parameters, we also compare DeepSeekMoE with GShard×1.2 and GShard×1.5, and show results in Table 10. At a larger scale, DeepSeekMoE even outperforms GShard×1.5 distinctly.\n\nMetric # Shot Dense×4 Dense×16 DeepSeekMoE Relative Expert Size N/A 1 1 0.25 # Experts N/A 4 + 0 16 + 0 1 + 63 # Activated Experts N/A 4 + 0 16 + 0 1 + 7 # Total Expert Params N/A 0.47B 1.89B 1.89B # Activated Expert Params N/A 0.47B 1.89B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.908 1.806 1.808 HellaSwag (Acc.) 0-shot 47.6 55.1 54.8 PIQA (Acc.) 0-shot 70.0 71.9 72.3 ARC-easy (Acc.) 0-shot 43.9 51.9 49.4 ARC-challenge (Acc.) 0-shot 30.5 33.8 34.3 RACE-middle (Acc.) 5-shot 42.4 46.3 44.0 RACE-high (Acc.) 5-shot 30.7 33.0 31.7 HumanEval (Pass@1) 0-shot 1.8 4.3 4.9 MBPP (Pass@1) 3-shot 0.2 2.2 2.2 TriviaQA (EM) 5-shot 9.9 16.5 16.6 NaturalQuestions (EM) 5-shot 3.0 6.3 5.7\n\nTable 9 | Comparison between DeepSeekMoE and larger dense baselines.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 15.9B 19.8B 13.3B # Activated Expert Params N/A 2.37B 2.82B 2.05B # Training Tokens N/A 100B 100B 100B HellaSwag (Acc.) 0-shot 66.6 67.7 69.1 PIQA (Acc.) 0-shot 75.6 76.0 75.7 ARC-easy (Acc.) 0-shot 56.8 56.8 58.8 ARC-challenge (Acc.) 0-shot 39.9 37.6 38.5 RACE-middle (Acc.) 5-shot 51.6 50.6 52.4 RACE-high (Acc.) 5-shot 37.4 36.3 38.5 HumanEval (Pass@1) 0-shot 6.1 6.1 9.8 MBPP (Pass@1) 3-shot 7.0 11.6 10.6 TriviaQA (EM) 5-shot 36.5 36.7 38.2 NaturalQuestions (EM) 5-shot 12.6 12.1 13.7\n\nTable 10 | Comparison between DeepSeekMoE and larger GShard models at a larger scale.\n\nC. Training Benchmark Curves Of Deepseekmoe 16B\n\nWe present the benchmark curves during training of DeepSeekMoE 16B and DeepSeek 7B (Dense) in Figure 7 for reference.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/DeepSeekMoE_2.md"
+          },
+          "summary": "DeepSeekMoE is an innovative Mixture-of-Experts (MoE) architecture designed to enhance expert specialization in language models. It employs fine-grained expert segmentation and shared expert isolation to improve performance and reduce computational costs. Starting with a model of 2B parameters, DeepSeekMoE demonstrates comparable performance to larger models like GShard 2.9B and LLaMA2 7B, using significantly fewer computations. The architecture scales up to 16B and 145B parameters, maintaining advantages over traditional MoE architectures. DeepSeekMoE 16B achieves similar results to dense models with only 40% of the computations, and its chat model version shows adaptability in bilingual scenarios. The architecture's public release aims to contribute to advancements in large-scale language models.",
+          "headlines": {
+            "1. Introduction": [],
+            "2. Preliminaries: Mixture-Of-Experts For Transformers": [],
+            "3. Deepseekmoe Architecture": [
+              "3.1. Fine-Grained Expert Segmentation",
+              "3.2. Shared Expert Isolation",
+              "3.3. Load Balance Consideration"
+            ],
+            "4. Validation Experiments": [
+              "4.1. Experimental Setup",
+              "4.1.1. Training Data And Tokenization",
+              "4.1.2. Infrastructures",
+              "4.1.3. Hyper-Parameters",
+              "4.1.4. Evaluation Benchmarks",
+              "4.2. Evaluations",
+              "4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models",
+              "4.4. Ablation Studies",
+              "4.5. Analysis On Expert Specialization"
+            ],
+            "5. Scaling Up To Deepseekmoe 16B": [
+              "5.1. Experimental Setup",
+              "5.1.1. Training Data And Tokenization",
+              "5.1.2. Hyper-Parameters",
+              "5.1.3. Evaluation Benchmarks",
+              "5.2. Evaluations",
+              "5.2.1. Internal Comparison With Deepseek 7B",
+              "5.2.2. Comparison With Open Source Models"
+            ],
+            "6. Alignment For Deepseekmoe 16B": [
+              "6.1. Experimental Setup",
+              "6.2. Evaluations"
+            ],
+            "7. Deepseekmoe 145B Ongoing": [
+              "7.1. Experimental Setup",
+              "7.2. Evaluations"
+            ],
+            "8. Related Work": [],
+            "9. Conclusion": [],
+            "Appendices": [
+              "A. Overview Of Hyper-Parameters",
+              "B. Comparing Deepseekmoe With Larger Models",
+              "C. Training Benchmark Curves Of Deepseekmoe 16B"
+            ]
+          },
+          "summary_embedding": [
+            -0.012465757317841053,
+            -0.022808987647294998,
+            -0.018792515620589256,
+            -0.02308654971420765,
+            0.01797616109251976,
+            0.007200260180979967,
+            0.006490030791610479,
+            0.042613785713911057,
+            -0.014049487188458443,
+            0.03187054023146629,
+            -0.021486490964889526,
+            -0.023592690005898476,
+            0.009045224636793137,
+            -0.014441338367760181,
+            0.005865518003702164,
+            -0.007363531738519669,
+            -0.008743173442780972,
+            -0.005163452122360468,
+            0.018841497600078583,
+            -0.03100520372390747,
+            0.011110605672001839,
+            -0.046075135469436646,
+            -0.020017050206661224,
+            -0.021845687180757523,
+            0.016425084322690964,
+            0.037389107048511505,
+            0.022547753527760506,
+            -0.0028062239289283752,
+            -0.06844329088926315,
+            0.034678805619478226,
+            0.035168617963790894,
+            0.02052319049835205,
+            0.0005438972148112953,
+            -0.00031302153365686536,
+            -0.024833550676703453,
+            -0.028556134551763535,
+            0.03347059711813927,
+            -0.004379750229418278,
+            -0.01453930139541626,
+            0.015976088121533394,
+            0.0414055772125721,
+            0.012106560170650482,
+            -0.02754385396838188,
+            0.0014245413476601243,
+            0.015151568688452244,
+            0.0055961208418011665,
+            -0.01678428053855896,
+            -0.0022960014175623655,
+            -0.002424577483907342,
+            -0.021225256845355034,
+            -0.01400050614029169,
+            0.018825169652700424,
+            0.001296985661610961,
+            -0.008865626528859138,
+            -0.0542386956512928,
+            0.007330877240747213,
+            0.03840138763189316,
+            0.05623060464859009,
+            0.044148534536361694,
+            -0.020180322229862213,
+            0.03369917720556259,
+            -0.030025575309991837,
+            -0.02501314878463745,
+            0.011028970591723919,
+            0.003983817528933287,
+            -0.018531281501054764,
+            0.01784554310142994,
+            -0.005636938381940126,
+            -0.0006556359585374594,
+            0.01751900091767311,
+            -0.03941367194056511,
+            0.020751770585775375,
+            -0.01887415163218975,
+            0.031266435980796814,
+            0.02173139713704586,
+            0.0073798587545752525,
+            0.01872720755636692,
+            0.01678428053855896,
+            0.0036858476232737303,
+            -0.04107903689146042,
+            0.009118696674704552,
+            0.016523046419024467,
+            0.014196431264281273,
+            0.016269976273179054,
+            0.06553706526756287,
+            -0.05391215533018112,
+            -0.004485876299440861,
+            -0.05169166624546051,
+            0.0012561678886413574,
+            -0.004971608519554138,
+            -0.03670336678624153,
+            0.01048201136291027,
+            0.028425518423318863,
+            0.021372200921177864,
+            -0.01063711941242218,
+            0.024392718449234962,
+            -0.022743679583072662,
+            0.0032001156359910965,
+            -0.04062187671661377,
+            0.0019653772469609976,
+            0.004518530797213316,
+            0.016531210392713547,
+            0.0070778070949018,
+            0.004273623693734407,
+            -0.016702644526958466,
+            0.006045116577297449,
+            -0.01905374974012375,
+            0.03291547670960426,
+            -0.008465612307190895,
+            0.08653374016284943,
+            0.008751336485147476,
+            0.004738946910947561,
+            0.004281787667423487,
+            0.02398454025387764,
+            0.012131051160395145,
+            -0.00810233410447836,
+            -0.04718538001179695,
+            -0.015690363943576813,
+            0.011061624623835087,
+            -0.0383034273982048,
+            0.018335357308387756,
+            -0.008759500458836555,
+            0.01282495353370905,
+            -0.04107903689146042,
+            -0.06344719231128693,
+            0.01498013362288475,
+            0.001784758409485221,
+            -0.004065453074872494,
+            -0.0002903166168835014,
+            -0.018923133611679077,
+            0.003930754493921995,
+            -0.004200151655822992,
+            -0.010645282454788685,
+            -0.021502818912267685,
+            0.03159298002719879,
+            0.03657275065779686,
+            0.06527583301067352,
+            0.036801330745220184,
+            -0.0030164355412125587,
+            0.005775718949735165,
+            0.019690508022904396,
+            -0.020784424617886543,
+            0.003481758525595069,
+            -0.01214737817645073,
+            -0.021992631256580353,
+            -0.03820546343922615,
+            0.010971825569868088,
+            0.07379858940839767,
+            -0.029029620811343193,
+            0.013437220826745033,
+            -0.017616963014006615,
+            0.04205866530537605,
+            -0.004567512311041355,
+            0.026727497577667236,
+            -0.016669990494847298,
+            0.004059330560266972,
+            -0.025388672947883606,
+            0.01620466820895672,
+            -0.024964168667793274,
+            0.012571883387863636,
+            -0.000885746325366199,
+            -0.04222193360328674,
+            0.00011116473615402356,
+            0.026890767738223076,
+            -0.010971825569868088,
+            0.02261306345462799,
+            0.0004949158756062388,
+            -0.006302268709987402,
+            -0.05097327008843422,
+            0.038727931678295135,
+            -0.003912386484444141,
+            0.018645571544766426,
+            -0.0018959869630634785,
+            0.0163107942789793,
+            -0.0016408757073804736,
+            0.013216804713010788,
+            0.009543201886117458,
+            0.007114543113857508,
+            -0.033503253012895584,
+            0.02697240374982357,
+            -0.02504580281674862,
+            0.028049994260072708,
+            0.03435226157307625,
+            -0.0030184765346348286,
+            0.03255628049373627,
+            0.019249675795435905,
+            0.03840138763189316,
+            0.0327032245695591,
+            0.030335791409015656,
+            -0.00824927818030119,
+            0.013878053054213524,
+            0.008200296200811863,
+            0.03608293831348419,
+            0.013429056853055954,
+            0.019723162055015564,
+            -0.009730963967740536,
+            -0.014041324146091938,
+            0.048785436898469925,
+            0.007318632211536169,
+            -0.0179598331451416,
+            -0.005441013257950544,
+            -0.03510330989956856,
+            -0.002832755446434021,
+            -0.014425011351704597,
+            0.0323440283536911,
+            0.02158445306122303,
+            -0.01647406443953514,
+            0.035135965794324875,
+            0.024980494752526283,
+            -0.03751972317695618,
+            -0.005232842173427343,
+            -0.02026195637881756,
+            -0.023298801854252815,
+            -0.04215662553906441,
+            0.011624909937381744,
+            0.012833117507398129,
+            -0.013976015150547028,
+            -0.007469657808542252,
+            0.020457882434129715,
+            -0.023625344038009644,
+            -0.0018265966791659594,
+            -0.01356783788651228,
+            0.006547175347805023,
+            -0.020898714661598206,
+            -0.03667071461677551,
+            -0.01048201136291027,
+            0.025862159207463264,
+            0.012637191452085972,
+            0.04362606629729271,
+            -0.013518855907022953,
+            -0.005367541220039129,
+            0.033193036913871765,
+            0.012759645469486713,
+            0.010767736472189426,
+            0.03699725493788719,
+            -0.03027048148214817,
+            0.024621298536658287,
+            -0.020294612273573875,
+            -0.0032878739293664694,
+            0.01707816869020462,
+            -0.0030062312725931406,
+            0.01962519995868206,
+            -0.04535674303770065,
+            -0.024196794256567955,
+            0.04395261034369469,
+            -0.0007107399869710207,
+            0.016302630305290222,
+            -0.005959399044513702,
+            -0.012106560170650482,
+            0.0035899258218705654,
+            0.0027715289033949375,
+            -0.01438419334590435,
+            -0.008620720356702805,
+            0.007473739795386791,
+            -0.01962519995868206,
+            0.03660540655255318,
+            0.03794422745704651,
+            -0.004132802598178387,
+            -0.014131123200058937,
+            -0.03324201703071594,
+            0.010016689077019691,
+            -0.017878197133541107,
+            -0.023951886221766472,
+            0.04078514873981476,
+            -0.032115448266267776,
+            -0.00029312283731997013,
+            0.039478980004787445,
+            0.018808843567967415,
+            -0.014898497611284256,
+            0.00868602842092514,
+            -0.01709449477493763,
+            -0.026302991434931755,
+            -0.017159804701805115,
+            0.03356856107711792,
+            0.024800896644592285,
+            -0.010743245482444763,
+            0.010245268233120441,
+            0.005330805201083422,
+            0.02893165871500969,
+            0.024376392364501953,
+            0.00648594880476594,
+            -0.013812744058668613,
+            0.015029114671051502,
+            0.027641816064715385,
+            -0.0028021421749144793,
+            -7.66609373386018e-05,
+            -0.004010349046438932,
+            -0.007796200457960367,
+            -0.006453294772654772,
+            -0.013069860637187958,
+            -0.005089980084449053,
+            0.0009184005321003497,
+            -0.032295044511556625,
+            0.015967924147844315,
+            -0.005890008993446827,
+            0.010359558276832104,
+            -0.011535110883414745,
+            0.01288209855556488,
+            0.01722511276602745,
+            -0.013494365848600864,
+            -0.0716434046626091,
+            0.014612773433327675,
+            0.02458864450454712,
+            -0.006706364918500185,
+            -0.017943505197763443,
+            0.008481939323246479,
+            0.00016148542636074126,
+            0.038597315549850464,
+            -0.014180104248225689,
+            -0.031103165820240974,
+            0.02680913358926773,
+            -0.017437364906072617,
+            -0.015241367742419243,
+            -0.010661610402166843,
+            -0.00580837344750762,
+            0.04649963974952698,
+            0.01889047957956791,
+            0.011420820839703083,
+            -0.04042595252394676,
+            -0.007449248805642128,
+            0.015078096650540829,
+            0.014506647363305092,
+            -0.04705476388335228,
+            0.0052083516493439674,
+            0.015486273914575577,
+            0.019723162055015564,
+            0.024800896644592285,
+            0.04274440184235573,
+            -0.0021531390957534313,
+            0.01661284640431404,
+            -0.015053605660796165,
+            -0.05136512219905853,
+            -0.03340528905391693,
+            0.029943939298391342,
+            -0.017731253057718277,
+            -0.02906227484345436,
+            -0.01693122461438179,
+            -0.006151242647320032,
+            0.0017939424142241478,
+            0.007126788143068552,
+            -0.01916803978383541,
+            -0.03464614972472191,
+            -0.03970756009221077,
+            0.02277633361518383,
+            0.007983962073922157,
+            0.004269542172551155,
+            -0.042777057737112045,
+            0.0013459670590236783,
+            0.009184005670249462,
+            0.012073906138539314,
+            0.0009954441338777542,
+            -0.019984396174550056,
+            0.015494437888264656,
+            0.0032001156359910965,
+            -0.031070511788129807,
+            -0.035593122243881226,
+            -0.03595232218503952,
+            0.0023919232189655304,
+            0.003781769424676895,
+            0.06279410421848297,
+            0.012988224625587463,
+            -0.03840138763189316,
+            -0.04522612690925598,
+            0.029176564887166023,
+            -0.02052319049835205,
+            -0.016678154468536377,
+            0.0059430720284581184,
+            0.012898425571620464,
+            -0.009877908043563366,
+            0.013192313723266125,
+            0.023560035973787308,
+            0.041601505130529404,
+            -0.005987971555441618,
+            0.012914752587676048,
+            0.023037567734718323,
+            0.012955570593476295,
+            0.0362788625061512,
+            -0.00936360377818346,
+            0.0035409443080425262,
+            0.039936136454343796,
+            -0.012269831262528896,
+            0.03175625205039978,
+            -0.02189466916024685,
+            0.0020398697815835476,
+            0.02249877341091633,
+            0.027184655889868736,
+            -0.0006479826406575739,
+            0.02594379521906376,
+            -0.031543996185064316,
+            0.018368011340498924,
+            -0.014498483389616013,
+            -0.04888340085744858,
+            -0.01557607389986515,
+            0.013102514669299126,
+            0.030384771525859833,
+            0.02756018005311489,
+            -0.0041348435916006565,
+            -0.019723162055015564,
+            -0.01155143789947033,
+            -0.010596301406621933,
+            0.0018419033149257302,
+            0.03285016864538193,
+            0.010506502352654934,
+            0.010890189558267593,
+            -0.011935125105082989,
+            -0.004104230087250471,
+            0.03025415539741516,
+            -0.021192602813243866,
+            0.01650671847164631,
+            0.0041511706076562405,
+            -0.002029665280133486,
+            -0.04320156201720238,
+            -0.035266581922769547,
+            0.007845181971788406,
+            0.011747363954782486,
+            0.014874007552862167,
+            0.0368339866399765,
+            0.025094784796237946,
+            -0.01662100851535797,
+            0.030629679560661316,
+            -0.01905374974012375,
+            0.027984686195850372,
+            0.039054471999406815,
+            0.011624909937381744,
+            -0.018351683393120766,
+            0.012237177230417728,
+            0.002424577483907342,
+            -0.025127438828349113,
+            0.01680060662329197,
+            -0.032735876739025116,
+            0.01738838478922844,
+            -0.02354370802640915,
+            0.025143766775727272,
+            -0.02579685114324093,
+            -0.009290131740272045,
+            -0.004428731743246317,
+            -0.006130834110081196,
+            0.04888340085744858,
+            -0.026727497577667236,
+            -0.0142454132437706,
+            -0.0024694770108908415,
+            -0.0024531499948352575,
+            -0.008718682453036308,
+            0.010000362060964108,
+            -0.045781247317790985,
+            -0.01844964735209942,
+            -0.04016471654176712,
+            0.007355368230491877,
+            0.005412440747022629,
+            -0.030907239764928818,
+            -0.05694899708032608,
+            0.025421326979994774,
+            -0.06331657618284225,
+            -0.019886434078216553,
+            0.051626358181238174,
+            -0.018400665372610092,
+            -0.008212542161345482,
+            0.009641164913773537,
+            0.006375740747898817,
+            -0.029290854930877686,
+            0.0041348435916006565,
+            0.02504580281674862,
+            0.04418119043111801,
+            -0.0045797573402523994,
+            -0.02865409664809704,
+            0.0028674507047981024,
+            -0.027788760140538216,
+            -0.013461710885167122,
+            0.005261414684355259,
+            0.016449574381113052,
+            0.024539662525057793,
+            0.007628847379237413,
+            0.03585435822606087,
+            0.007294141221791506,
+            0.006318595726042986,
+            0.044279150664806366,
+            -0.025666235014796257,
+            -0.027380581945180893,
+            0.02519274689257145,
+            0.04045860469341278,
+            0.023151857778429985,
+            -0.018253721296787262,
+            -0.0142454132437706,
+            0.0053063142113387585,
+            0.0009296254720538855,
+            -0.022139575332403183,
+            0.002808264922350645,
+            -0.013902543112635612,
+            0.005143043119460344,
+            0.03892385587096214,
+            -0.001205145614221692,
+            0.014196431264281273,
+            0.007302304729819298,
+            0.006845145486295223,
+            0.004334850702434778,
+            -0.011731036007404327,
+            -0.023592690005898476,
+            0.00032220553839579225,
+            0.034972693771123886,
+            0.015404638834297657,
+            -0.030515389516949654,
+            -0.00995954405516386,
+            0.034711457788944244,
+            -0.00795130804181099,
+            -0.009118696674704552,
+            -0.009771781973540783,
+            0.003385836724191904,
+            0.02638462744653225,
+            -0.01606588624417782,
+            0.006649219896644354,
+            0.03716052696108818,
+            -0.019870106130838394,
+            0.005543057806789875,
+            -0.025405000895261765,
+            0.0018061877926811576,
+            0.02442537248134613,
+            -0.006216551177203655,
+            0.028098974376916885,
+            5.6315813708351925e-05,
+            0.03025415539741516,
+            -0.004677720367908478,
+            0.008432958275079727,
+            -0.020327266305685043,
+            0.03987082839012146,
+            -0.017633290961384773,
+            0.03402572125196457,
+            0.02067013457417488,
+            -0.01258004643023014,
+            0.04783846437931061,
+            -0.036801330745220184,
+            -0.020784424617886543,
+            0.00584919098764658,
+            0.012743317522108555,
+            0.0028919412288814783,
+            -0.0191517136991024,
+            -0.018400665372610092,
+            -0.010996315628290176,
+            0.02380494214594364,
+            -0.022547753527760506,
+            -0.016865916550159454,
+            -0.007441085297614336,
+            -0.01890680566430092,
+            -0.009673818945884705,
+            -0.007988044060766697,
+            -0.01709449477493763,
+            -0.016547536477446556,
+            0.044573038816452026,
+            -0.06334923207759857,
+            0.016996532678604126,
+            0.017159804701805115,
+            0.011477965861558914,
+            -0.034678805619478226,
+            -0.019510909914970398,
+            -0.017633290961384773,
+            -0.002347023691982031,
+            -0.030743969604372978,
+            0.001702102366834879,
+            0.01725776679813862,
+            0.01138000376522541,
+            -0.05022222548723221,
+            0.012335140258073807,
+            0.02772345207631588,
+            -0.012065742164850235,
+            0.0014949520118534565,
+            -0.023135529831051826,
+            -0.017176130786538124,
+            0.0031776658724993467,
+            0.022123249247670174,
+            0.005975726060569286,
+            -0.008188051171600819,
+            -0.016865916550159454,
+            -0.005241005681455135,
+            -0.006045116577297449,
+            -0.002359268954023719,
+            0.005094061605632305,
+            0.011780017986893654,
+            0.019690508022904396,
+            -0.0024633544962853193,
+            -0.004436895251274109,
+            -0.006396149750798941,
+            0.027364253997802734,
+            -0.001450052484869957,
+            -0.008416631259024143,
+            0.050907962024211884,
+            0.03556047007441521,
+            0.006094098091125488,
+            -0.01108611561357975,
+            -0.03990348428487778,
+            0.010661610402166843,
+            -0.022711025550961494,
+            0.0401320643723011,
+            0.01905374974012375,
+            -0.008298259228467941,
+            -0.028866350650787354,
+            0.010294250212609768,
+            -0.02290695160627365,
+            -0.009975871071219444,
+            0.009469729848206043,
+            0.00726556871086359,
+            -0.03263791650533676,
+            -0.013543346896767616,
+            0.03510330989956856,
+            0.011437147855758667,
+            -0.006841063499450684,
+            -0.036050282418727875,
+            0.008588065393269062,
+            0.01646590232849121,
+            0.011869817040860653,
+            -0.019233347848057747,
+            0.004449140280485153,
+            -0.012041252106428146,
+            0.017470018938183784,
+            0.008849299512803555,
+            -0.017796561121940613,
+            0.018237393349409103,
+            0.004710374400019646,
+            -0.008490103296935558,
+            -0.003912386484444141,
+            -0.0163107942789793,
+            -0.02006603218615055,
+            0.03840138763189316,
+            0.024490682408213615,
+            0.026335645467042923,
+            0.005457340274006128,
+            0.02008235827088356,
+            0.03227871656417847,
+            0.011282040737569332,
+            -0.03134807199239731,
+            0.027935704216361046,
+            0.016996532678604126,
+            -0.04604247957468033,
+            0.021551799029111862,
+            -0.02218855731189251,
+            -0.028670424595475197,
+            0.009649328887462616,
+            -0.0017367975087836385,
+            0.011192241683602333,
+            0.0034837995190173388,
+            -0.003642988856881857,
+            -0.03307874873280525,
+            -0.006236960180103779,
+            0.015771998092532158,
+            0.015502601861953735,
+            -0.0048695639707148075,
+            -0.003296037670224905,
+            -0.011886144056916237,
+            0.00487364549189806,
+            0.009796272963285446,
+            0.039054471999406815,
+            0.001381682581268251,
+            -0.004751192405819893,
+            -0.0018765984568744898,
+            0.031854212284088135,
+            0.0271193478256464,
+            -0.017274094745516777,
+            -0.026286665350198746,
+            0.00580837344750762,
+            -0.014253576286137104,
+            0.026156047359108925,
+            -0.00487364549189806,
+            0.011877980083227158,
+            0.008718682453036308,
+            -0.0007908449042588472,
+            -0.001373519073240459,
+            0.025584599003195763,
+            -0.02710302174091339,
+            -0.021764053031802177,
+            0.031233781948685646,
+            -0.025731543079018593,
+            0.028735732659697533,
+            0.013061696663498878,
+            -0.01557607389986515,
+            -0.015225040726363659,
+            -0.010302413254976273,
+            0.009943217039108276,
+            -0.012277995236217976,
+            0.023853924125432968,
+            0.003608293831348419,
+            0.006392067763954401,
+            0.0003584313380997628,
+            0.02770712412893772,
+            0.024833550676703453,
+            0.007820690982043743,
+            -0.018074123188853264,
+            -0.0016816934803500772,
+            -0.02320083975791931,
+            -0.01680060662329197,
+            0.023772288113832474,
+            -0.018090449273586273,
+            -0.03268689662218094,
+            -0.028556134551763535,
+            0.012114724144339561,
+            0.007392104249447584,
+            0.010816717520356178,
+            0.03461349755525589,
+            -0.030384771525859833,
+            -0.0344502255320549,
+            0.01766594499349594,
+            0.026270337402820587,
+            -0.005551221314817667,
+            -0.021535472944378853,
+            0.03085825778543949,
+            -0.005828781984746456,
+            0.017600636929273605,
+            0.0404912605881691,
+            -0.0016225076979026198,
+            0.010228941217064857,
+            -0.042026009410619736,
+            -0.014057651162147522,
+            -0.0066614653915166855,
+            0.006261450704187155,
+            0.034417569637298584,
+            0.01048201136291027,
+            0.01535565685480833,
+            0.03448288142681122,
+            -0.021192602813243866,
+            -0.0002903166168835014,
+            -0.0022449791431427,
+            -0.02818061038851738,
+            0.03347059711813927,
+            0.025715215131640434,
+            0.011918798089027405,
+            0.019706835970282555,
+            -0.03794422745704651,
+            -0.013037205673754215,
+            0.027070365846157074,
+            0.015053605660796165,
+            0.01273515447974205,
+            0.016694480553269386,
+            0.011249386705458164,
+            0.00868602842092514,
+            -0.021633435040712357,
+            -0.05169166624546051,
+            0.01214737817645073,
+            0.01993541419506073,
+            -0.016449574381113052,
+            0.01057181041687727,
+            0.024980494752526283,
+            0.007620683871209621,
+            -0.007045152597129345,
+            0.012808626517653465,
+            -0.02682545967400074,
+            0.002259265398606658,
+            0.0019010890973731875,
+            -0.014482156373560429,
+            0.038270771503448486,
+            -0.016367938369512558,
+            0.011363675817847252,
+            -0.018204739317297935,
+            0.023870250210165977,
+            0.00011531029304023832,
+            0.0007964573451317847,
+            0.014963806606829166,
+            -0.011347348801791668,
+            -0.04235255345702171,
+            0.0021613026037812233,
+            0.03342161700129509,
+            -0.015200549736618996,
+            0.026449935510754585,
+            -0.0020674217958003283,
+            0.032458316534757614,
+            0.042777057737112045,
+            0.017731253057718277,
+            0.0003497575526125729,
+            0.025682561099529266,
+            0.0036144163459539413,
+            0.0073880222626030445,
+            0.012400448322296143,
+            0.009322785772383213,
+            0.007245160173624754,
+            0.030809277668595314,
+            -0.010261595249176025,
+            0.02173139713704586,
+            -0.04760988429188728,
+            -0.014629100449383259,
+            0.012024925090372562,
+            -0.0013316808035597205,
+            -0.012449429370462894,
+            -0.033029764890670776,
+            0.012563719414174557,
+            -0.03503800183534622,
+            0.022074267268180847,
+            -0.0004959363141097128,
+            0.04333217814564705,
+            -0.02187834307551384,
+            -0.005518566817045212,
+            0.016694480553269386,
+            0.0016510800924152136,
+            0.023282473906874657,
+            -0.0012153501156717539,
+            -0.016416920349001884,
+            -0.0067798369564116,
+            -0.009257477708160877,
+            0.004138925112783909,
+            0.04375668615102768,
+            -0.003722583642229438,
+            -0.0001682458823779598,
+            0.01409030519425869,
+            -0.011959616094827652,
+            -0.03250729665160179,
+            0.045193471014499664,
+            -0.0030837850645184517,
+            0.02484987862408161,
+            0.04979771748185158,
+            -0.06410028040409088,
+            -0.019543563947081566,
+            0.013061696663498878,
+            -0.010441194288432598,
+            0.011363675817847252,
+            0.005547139327973127,
+            0.016114868223667145,
+            0.017894525080919266,
+            -0.005485912784934044,
+            0.01619650423526764,
+            0.005106307100504637,
+            0.028098974376916885,
+            -0.02336410991847515,
+            -0.03296445682644844,
+            -0.02636830136179924,
+            -0.017747581005096436,
+            0.0014480114914476871,
+            -0.015796490013599396,
+            -3.563585551091819e-06,
+            -0.0022204883862286806,
+            -0.04388730227947235,
+            -0.025274382904171944,
+            0.040687184780836105,
+            0.006706364918500185,
+            0.02458864450454712,
+            0.0077268099412322044,
+            0.03418898954987526,
+            0.005885927006602287,
+            -0.009037061594426632,
+            0.030058229342103004,
+            -0.020049704238772392,
+            0.03833607956767082,
+            0.022106921300292015,
+            -0.002832755446434021,
+            0.005865518003702164,
+            0.031086837872862816,
+            -0.033797141164541245,
+            0.008086007088422775,
+            -0.034384917467832565,
+            0.0046695563942193985,
+            -0.008620720356702805,
+            -0.015094423666596413,
+            -0.009526874870061874,
+            -0.02504580281674862,
+            0.011216731742024422,
+            -0.021535472944378853,
+            0.01590261608362198,
+            0.010416703298687935,
+            -0.002916431985795498,
+            0.03448288142681122,
+            -0.014204595237970352,
+            -0.0027796924114227295,
+            0.0024531499948352575,
+            -0.027070365846157074,
+            -0.006343086715787649,
+            -0.003377673216164112,
+            -0.03689929470419884,
+            -0.004502203781157732,
+            0.010988152585923672,
+            -0.00024184548237826675,
+            0.016898570582270622,
+            0.02787039615213871,
+            0.0005092020728625357,
+            -0.022302847355604172,
+            0.002340900944545865,
+            -0.04199335351586342,
+            0.0011877981014549732,
+            0.01483318954706192,
+            -0.017600636929273605,
+            -0.008612556383013725,
+            0.033046092838048935,
+            0.023298801854252815,
+            -0.009396257810294628,
+            -0.035005345940589905,
+            0.017796561121940613,
+            -0.011853490024805069,
+            0.027511198073625565,
+            -0.0002630197268445045,
+            0.019298657774925232,
+            -0.0076941559091210365,
+            0.04493223875761032,
+            0.001369437319226563,
+            0.008645210415124893,
+            -0.021959977224469185,
+            0.007020662073045969,
+            -0.020278284326195717,
+            -0.002406209474429488,
+            -0.006171651650220156,
+            -0.012751481495797634,
+            -0.010669773444533348,
+            0.0032735876739025116,
+            0.007730891928076744,
+            -0.0176822729408741,
+            0.0055961208418011665,
+            -0.015959760174155235,
+            -0.02741323597729206,
+            0.007543129846453667,
+            -0.05965929850935936,
+            -0.0179598331451416,
+            -0.03373183310031891,
+            -0.009477893821895123,
+            0.02008235827088356,
+            0.03510330989956856,
+            -0.012759645469486713,
+            0.00497977202758193,
+            -0.014898497611284256,
+            0.004477712791413069,
+            0.03490738570690155,
+            0.044442422688007355,
+            -0.014759717509150505,
+            0.003381754970178008,
+            -0.010955498553812504,
+            0.033029764890670776,
+            -0.03347059711813927,
+            0.0030062312725931406,
+            0.031086837872862816,
+            -0.005057325586676598,
+            0.007959471084177494,
+            0.011624909937381744,
+            0.020474210381507874,
+            -0.019396619871258736,
+            0.00877582747489214,
+            0.002100075827911496,
+            0.019233347848057747,
+            0.02171507105231285,
+            0.020343592390418053,
+            -0.03539719805121422,
+            0.028409190475940704,
+            0.018384337425231934,
+            0.02444170042872429,
+            0.044442422688007355,
+            -0.03974021226167679,
+            -0.004147088620811701,
+            -0.0032225653994828463,
+            -0.02997659333050251,
+            0.02008235827088356,
+            -0.007530884351581335,
+            -0.01841699331998825,
+            -0.007069643586874008,
+            0.03327467292547226,
+            0.014571955427527428,
+            -0.05051611363887787,
+            0.01980479806661606,
+            0.01740471087396145,
+            0.02055584453046322,
+            0.024474354460835457,
+            -0.01123305968940258,
+            -0.0019072118448093534,
+            0.008996243588626385,
+            0.006804327480494976,
+            0.019266001880168915,
+            0.015257694758474827,
+            0.002012317767366767,
+            -0.0022082431241869926,
+            -0.011837163008749485,
+            0.005224678665399551,
+            -0.009077879600226879,
+            0.010808554477989674,
+            -0.02458864450454712,
+            -0.009698309935629368,
+            0.021388528868556023,
+            0.0031164393294602633,
+            -0.013665799982845783,
+            -0.03729114308953285,
+            -0.045944519340991974,
+            0.0033837957307696342,
+            0.004543021321296692,
+            -0.05365091934800148,
+            0.005322641227394342,
+            -0.026547899469733238,
+            -0.002108239568769932,
+            -0.005502239800989628,
+            0.0019214981002733111,
+            0.005636938381940126,
+            0.0167679525911808,
+            -0.013494365848600864,
+            0.02248244546353817,
+            -0.015502601861953735,
+            0.03165828809142113,
+            0.0055961208418011665,
+            0.007375776767730713,
+            -0.0027796924114227295,
+            0.05590406060218811,
+            -0.003712379140779376,
+            -0.03699725493788719,
+            0.023282473906874657,
+            -0.01311067771166563,
+            -0.0001747512142173946,
+            0.030090883374214172,
+            0.00884113647043705,
+            -0.015257694758474827,
+            0.013437220826745033,
+            -0.030613351613283157,
+            -0.03249097242951393,
+            0.010996315628290176,
+            -0.013755599036812782,
+            -0.0015776080545037985,
+            -0.04568328335881233,
+            -0.009339112788438797,
+            0.004269542172551155,
+            -0.018335357308387756,
+            0.010775899514555931,
+            0.01631895825266838,
+            0.013992343097925186,
+            -0.024245774373412132,
+            0.009053388610482216,
+            0.006865554489195347,
+            0.013453547842800617,
+            0.0007331897504627705,
+            0.014049487188458443,
+            -0.008677864447236061,
+            -0.011306531727313995,
+            -0.018319029361009598,
+            0.0032450153958052397,
+            -0.026792805641889572,
+            0.005155288614332676,
+            -0.003155216109007597,
+            -0.03771565109491348,
+            -0.06521052122116089,
+            -0.0344502255320549,
+            0.016294466331601143,
+            0.03396041318774223,
+            -0.009045224636793137,
+            0.0035144127905368805,
+            -0.00023648813657928258,
+            -0.01093917153775692,
+            0.024523336440324783,
+            -0.005930826533585787,
+            -0.0004829256213270128,
+            0.0035389035474509,
+            0.02050686441361904,
+            -0.009722800925374031,
+            0.005845109466463327,
+            -0.051952898502349854,
+            -0.01586179807782173,
+            0.01812310516834259,
+            0.006306350696831942,
+            0.014629100449383259,
+            -0.009690145961940289,
+            0.01693122461438179,
+            -0.030074557289481163,
+            -0.028147956356406212,
+            0.04130761697888374,
+            0.008277850225567818,
+            0.011486129835247993,
+            -0.003055212553590536,
+            -0.01680060662329197,
+            0.02504580281674862,
+            -0.0020633398089557886,
+            -0.006461458280682564,
+            -0.00817988719791174,
+            0.015820980072021484,
+            0.044279150664806366,
+            0.003255219664424658,
+            0.0009821783751249313,
+            0.017045514658093452,
+            -0.002663361607119441,
+            -0.011575928889214993,
+            0.007677828427404165,
+            0.013592327944934368,
+            -0.016384266316890717,
+            0.010375885292887688,
+            -0.025862159207463264,
+            0.0332583449780941,
+            0.03131541982293129,
+            0.011575928889214993,
+            -0.015045441687107086,
+            0.006130834110081196,
+            0.005330805201083422,
+            0.008086007088422775,
+            -0.001369437319226563,
+            -0.0017133272485807538,
+            0.020719116553664207,
+            -0.03276853263378143,
+            -0.01693122461438179,
+            -0.005347132217139006,
+            0.003932795487344265,
+            0.005277741700410843,
+            0.03396041318774223,
+            -0.0033266509417444468,
+            0.017927179113030434,
+            -0.011657563969492912,
+            0.010579974390566349,
+            0.021764053031802177,
+            0.037095218896865845,
+            -0.024996822699904442,
+            -0.029862305149435997,
+            0.012914752587676048,
+            -0.001296985661610961,
+            0.014065815135836601,
+            0.0010173837654292583,
+            0.014139287173748016,
+            0.0029409227427095175,
+            0.060475654900074005,
+            -0.017763907089829445,
+            0.004628738854080439,
+            0.0004035860183648765,
+            -0.01858026348054409,
+            -0.0013531101867556572,
+            0.004261378664523363,
+            -0.010979988612234592,
+            0.03608293831348419,
+            0.002344982698559761,
+            0.039185091853141785,
+            0.013918871060013771,
+            0.009290131740272045,
+            0.0015143405180424452,
+            -0.013592327944934368,
+            0.008355404250323772,
+            0.0015592400450259447,
+            -0.001132694073021412,
+            0.009461566805839539,
+            0.019298657774925232,
+            0.009714636951684952,
+            0.025274382904171944,
+            0.009788108989596367,
+            0.012857607565820217,
+            -0.029470453038811684,
+            0.03510330989956856,
+            -0.0006224715034477413,
+            0.012996388599276543,
+            -0.02040890045464039,
+            -0.003136848099529743,
+            -0.0008240094175562263,
+            0.015649545937776566,
+            -0.015616890974342823,
+            -0.001201063860207796,
+            -0.01297189760953188,
+            0.040099408477544785,
+            -0.018988441675901413,
+            0.013976015150547028,
+            0.009673818945884705,
+            -0.02865409664809704,
+            -0.007596192881464958,
+            0.01995174214243889,
+            0.014294394291937351,
+            -0.0053144777193665504,
+            -0.020490536466240883,
+            0.005832863971590996,
+            0.015380147844552994,
+            -0.028572462499141693,
+            1.3584674888988957e-05,
+            0.014751553535461426,
+            -0.0164903923869133,
+            -0.00025409081717953086,
+            -0.004702210891991854,
+            0.003975654020905495,
+            -0.009265640750527382,
+            0.007028825581073761,
+            0.016057724133133888,
+            -0.024507008492946625,
+            -0.020049704238772392,
+            0.005722655914723873,
+            0.006571666337549686,
+            0.01444950234144926,
+            0.007739055436104536,
+            0.004355259705334902,
+            -0.020033378154039383,
+            -0.01799248717725277,
+            0.0032674651592969894,
+            -0.01649855636060238,
+            -0.0048614004626870155,
+            0.0020531355403363705,
+            -0.019363965839147568,
+            0.01875986158847809,
+            -0.010743245482444763,
+            0.00017538899555802345,
+            -0.003226647386327386,
+            -0.020800752565264702,
+            0.006204306147992611,
+            -0.004865481983870268,
+            0.019870106130838394,
+            0.003555230563506484,
+            0.02785406820476055,
+            0.02310287579894066,
+            -0.004428731743246317,
+            0.017600636929273605,
+            -0.016155686229467392,
+            0.007983962073922157,
+            -0.031135819852352142,
+            -0.001782717532478273,
+            -0.010408539324998856,
+            0.0036103345919400454,
+            -0.015943434089422226,
+            -0.009126860648393631,
+            -0.0048777274787425995,
+            -0.0006678812787868083,
+            -0.004820582456886768,
+            0.0004275664687156677,
+            0.0027817331720143557,
+            0.02231917530298233,
+            -0.013208640739321709,
+            -0.02803366631269455,
+            -0.017437364906072617,
+            0.006045116577297449,
+            0.019396619871258736,
+            -0.026025431230664253,
+            -0.009184005670249462,
+            -0.005012426059693098,
+            -0.004326687194406986,
+            -0.021404854953289032,
+            -0.015167895704507828,
+            -0.005632856860756874,
+            -0.008045189082622528,
+            -0.010522829368710518,
+            0.0046532293781638145,
+            -0.0035715578123927116,
+            -0.011739199981093407,
+            0.021666089072823524,
+            0.018057795241475105,
+            -0.019837452098727226,
+            -0.019837452098727226,
+            0.013641309924423695,
+            -0.006159406621009111,
+            0.0018878233386203647,
+            0.02008235827088356,
+            0.00029286774224601686,
+            0.005824700463563204,
+            0.020882388576865196,
+            -0.0035225762985646725,
+            -0.01797616109251976,
+            0.003577680326998234,
+            -0.0018888438353314996,
+            -0.012122887186706066,
+            0.003212361130863428,
+            0.02773977816104889,
+            0.0008076822850853205,
+            0.023429417982697487,
+            -0.03056436963379383,
+            -0.011412657797336578,
+            0.006061443593353033,
+            0.02369065210223198,
+            0.00211844383738935,
+            -0.0021694661118090153,
+            0.020131340250372887,
+            0.03941367194056511,
+            0.005775718949735165,
+            0.0015755671774968505,
+            -0.04003410041332245,
+            0.01027792226523161,
+            0.009208496659994125,
+            0.024621298536658287,
+            0.008339077234268188,
+            0.0014653591206297278,
+            -0.012906589545309544,
+            -0.009755454957485199,
+            -0.0017939424142241478,
+            0.014702572487294674,
+            -0.023870250210165977,
+            0.0024755997583270073,
+            -0.03157665207982063,
+            0.005094061605632305,
+            0.0021959978621453047,
+            -0.009559528902173042,
+            0.0335359051823616,
+            0.00980443600565195,
+            -0.02065380848944187,
+            0.015151568688452244,
+            0.005767555441707373,
+            0.007330877240747213,
+            0.022449791431427002,
+            -0.008228869177401066,
+            0.030450081452727318,
+            -0.005049162078648806,
+            0.01213921420276165,
+            0.02742956392467022,
+            0.032588932663202286,
+            -0.013102514669299126,
+            -0.003959327004849911,
+            0.0006097159348428249,
+            -0.0002561317232903093,
+            0.0007995187188498676,
+            -0.008204378187656403,
+            0.0037389106582850218,
+            -0.014800535514950752,
+            0.012710663489997387,
+            -0.019543563947081566,
+            -0.0034286954905837774,
+            0.0038532004691660404,
+            0.01991908811032772,
+            0.0016265894519165158,
+            -0.033633869141340256,
+            0.02055584453046322,
+            -0.015420965850353241,
+            0.0135596739128232,
+            0.001290863030590117,
+            -0.011110605672001839,
+            -0.00029210239881649613,
+            0.020588500425219536,
+            0.010947334580123425,
+            -0.007845181971788406,
+            -0.0135596739128232,
+            -0.028376536443829536,
+            -0.021535472944378853,
+            0.00823703221976757,
+            0.009445239789783955,
+            -0.0028837777208536863,
+            0.0021306893322616816,
+            -0.011869817040860653,
+            0.0006520643946714699,
+            -0.0008122742874547839,
+            0.01740471087396145,
+            -0.0335359051823616,
+            0.021943651139736176,
+            -0.019527235999703407,
+            -0.019543563947081566,
+            0.017045514658093452,
+            0.008049271069467068,
+            0.009445239789783955,
+            0.0033552234526723623,
+            -0.01995174214243889,
+            -0.012326976284384727,
+            0.0338951013982296,
+            -0.01138000376522541,
+            0.006298186723142862,
+            0.004408322740346193,
+            0.012914752587676048,
+            -0.008914608508348465,
+            -0.011926962062716484,
+            0.01784554310142994,
+            -0.009861581027507782,
+            0.013951525092124939,
+            0.0010010566329583526,
+            -0.0254703089594841,
+            0.02739690989255905,
+            -0.019233347848057747,
+            0.03487472981214523,
+            -0.021617108955979347,
+            0.00711862463504076,
+            0.0066614653915166855,
+            0.008792154490947723,
+            -0.006690037902444601,
+            -0.013224967755377293,
+            0.0012786176521331072,
+            0.03807484731078148,
+            -0.02623768337070942,
+            0.024180466309189796,
+            0.022400809451937675,
+            -0.00040766780148260295,
+            0.02065380848944187,
+            -0.008367649279534817,
+            -0.026711169630289078,
+            0.0179598331451416,
+            -0.012596373446285725,
+            -0.026123393326997757,
+            0.013257622718811035,
+            -0.022678371518850327,
+            -0.020278284326195717,
+            -0.026302991434931755,
+            0.004947117529809475,
+            -0.0025694805663079023,
+            -0.005151206627488136,
+            0.009004407562315464,
+            0.02561725303530693,
+            0.020915042608976364,
+            0.019576217979192734,
+            0.02111096680164337,
+            0.01161674689501524,
+            0.022449791431427002,
+            -0.007559456862509251,
+            0.04401791840791702,
+            -0.004947117529809475,
+            0.004632820375263691,
+            -0.0015980169409886003,
+            -0.015657709911465645,
+            0.009771781973540783,
+            0.0119106350466609,
+            0.0029572497587651014,
+            -0.029176564887166023,
+            -0.009061552584171295,
+            -0.03660540655255318,
+            0.005053244065493345,
+            0.0020664012990891933,
+            0.0176822729408741,
+            0.032115448266267776,
+            -0.005865518003702164,
+            0.004592002835124731,
+            -0.005102225113660097,
+            -0.013927034102380276,
+            -0.009420748800039291,
+            -0.02739690989255905,
+            -0.025437654927372932,
+            -0.007653337903320789,
+            -0.013739272020757198,
+            -0.030368445441126823,
+            -0.011722872965037823,
+            0.01551892887800932,
+            -0.007645174395292997,
+            0.011918798089027405,
+            -0.016702644526958466,
+            0.01385356206446886,
+            -0.004110352601855993,
+            -0.004763437435030937,
+            -0.004530776292085648,
+            -0.017600636929273605,
+            -0.006420640274882317,
+            0.014367866329848766,
+            -0.009477893821895123,
+            -0.01237595733255148,
+            -0.02215590327978134,
+            -0.0015459742862731218,
+            -0.01574750803411007,
+            0.003489922033622861,
+            -0.014596446417272091,
+            -0.006771673448383808,
+            -0.003908304497599602,
+            -0.018514955416321754,
+            0.01766594499349594,
+            0.014816862531006336,
+            -0.019380291923880577,
+            -0.0090697156265378,
+            0.02638462744653225,
+            0.042777057737112045,
+            -0.03218075633049011,
+            -0.008200296200811863,
+            0.0020378287881612778,
+            0.018090449273586273,
+            0.010171796195209026,
+            0.016702644526958466,
+            -0.024653952568769455,
+            0.0022286521270871162,
+            -0.021649762988090515,
+            -0.001220452249981463,
+            -0.023739634081721306,
+            0.010775899514555931,
+            0.025715215131640434,
+            -0.010971825569868088,
+            0.025976449251174927,
+            0.01634344831109047,
+            -0.01812310516834259,
+            0.012016761116683483,
+            0.007183933164924383,
+            0.014645427465438843,
+            -0.019755816087126732,
+            0.0167679525911808,
+            -0.015796490013599396,
+            0.031233781948685646,
+            0.00892277155071497,
+            -0.04829562455415726,
+            -0.03503800183534622,
+            -0.013388238847255707,
+            0.0013388239312916994,
+            -0.010702427476644516,
+            0.015037278644740582,
+            0.015967924147844315,
+            0.013959688134491444,
+            -0.019380291923880577,
+            -0.015820980072021484,
+            0.011984107084572315,
+            0.010065670125186443,
+            0.027462217956781387,
+            -0.006649219896644354,
+            -0.004212397150695324,
+            -0.00960034690797329,
+            -0.02249877341091633,
+            -0.016098542138934135,
+            0.0209803506731987,
+            -0.018041469156742096,
+            0.015935270115733147,
+            -0.026139721274375916,
+            -0.012449429370462894,
+            0.017453692853450775,
+            -0.05211617052555084,
+            -0.01453113742172718,
+            0.0007612520130351186,
+            0.03012353740632534,
+            0.004751192405819893,
+            0.0035184945445507765,
+            0.01841699331998825,
+            -0.01993541419506073,
+            -0.0039776950143277645,
+            -0.02099667675793171,
+            0.006832899991422892,
+            -0.011853490024805069,
+            -0.013984179124236107,
+            0.014482156373560429,
+            -0.013845399022102356,
+            -0.01601690612733364,
+            0.0034368589986115694,
+            0.007183933164924383,
+            -0.021094640716910362,
+            0.011951452121138573,
+            0.003945040516555309,
+            0.012171869166195393,
+            0.012171869166195393,
+            0.01199227012693882,
+            0.021551799029111862,
+            0.02295593172311783,
+            -0.007334959227591753,
+            -0.004767519421875477,
+            -0.01004117913544178,
+            -0.0015398516552522779,
+            0.008898280560970306,
+            0.0033797139767557383,
+            -0.002989904023706913,
+            0.007837017998099327,
+            0.010033016093075275,
+            -0.01498013362288475,
+            0.033193036913871765,
+            -0.0035878848284482956,
+            -0.04134026914834976,
+            -0.003238892648369074,
+            -0.03800953924655914,
+            0.020033378154039383,
+            -0.023298801854252815,
+            -0.016996532678604126,
+            0.0032776694279164076,
+            -0.00629410520195961,
+            -0.019902760162949562,
+            0.013608654960989952,
+            0.011404493823647499,
+            0.00258580781519413,
+            0.02021297626197338,
+            0.012269831262528896,
+            -0.005134879611432552,
+            0.026172375306487083,
+            -0.014718899503350258,
+            -0.008751336485147476,
+            0.015567909926176071,
+            0.006853308994323015,
+            0.0022347746416926384,
+            0.0011031010653823614,
+            -0.018825169652700424,
+            0.005098143592476845,
+            0.0164903923869133,
+            0.007216587662696838,
+            -0.000566347036510706,
+            0.014049487188458443,
+            0.012514738366007805,
+            -0.015135240741074085,
+            -0.0002213090192526579,
+            -0.026564225554466248,
+            0.0032817514147609472,
+            0.004567512311041355,
+            0.007506393827497959,
+            -0.006771673448383808,
+            0.006281859707087278,
+            0.00658799335360527,
+            0.014759717509150505,
+            -0.010155469179153442,
+            -0.016980206593871117,
+            -0.002510294783860445,
+            0.004285869188606739,
+            0.016694480553269386,
+            0.03157665207982063,
+            -0.010310577228665352,
+            -0.005734901409596205,
+            -0.0386626236140728,
+            0.02070278860628605,
+            0.03332365304231644,
+            0.011175914667546749,
+            -0.028213264420628548,
+            0.012302485294640064,
+            0.0006515541463159025,
+            0.0019276207312941551,
+            -0.026302991434931755,
+            0.02787039615213871,
+            0.00974729098379612,
+            0.0066614653915166855,
+            0.017861871048808098,
+            0.0022306928876787424,
+            0.0029266364872455597,
+            -0.016302630305290222,
+            0.01018812321126461,
+            0.009086042642593384,
+            -0.017306748777627945,
+            0.0009745250572450459,
+            0.02324981987476349,
+            -0.01423724927008152,
+            -0.003208279376849532,
+            0.0018133309204131365,
+            0.02651524543762207,
+            -0.013820908032357693,
+            -0.014294394291937351,
+            0.0005714492290280759,
+            0.01751900091767311,
+            -0.0007979880319908261,
+            0.012098397128283978,
+            0.04898136481642723,
+            -0.012424939312040806,
+            0.019657853990793228,
+            0.005461421795189381,
+            -0.015290348790585995,
+            -0.032605260610580444,
+            -0.004922627005726099,
+            -0.01830270327627659,
+            0.011200404725968838,
+            0.014343376271426678,
+            -0.015763835981488228,
+            0.0007525782566517591,
+            -0.03144603595137596,
+            0.007743136957287788,
+            0.014735226519405842,
+            0.018776189535856247,
+            -0.002440904499962926,
+            0.006134915631264448,
+            0.027674470096826553,
+            0.02458864450454712,
+            -0.007392104249447584,
+            0.013804581016302109,
+            0.02320083975791931,
+            -0.025062130764126778,
+            -0.003069498809054494,
+            0.006820654962211847,
+            0.004104230087250471,
+            0.0053063142113387585,
+            -0.016963878646492958,
+            0.03915243595838547,
+            -0.005922663025557995,
+            -0.01506176870316267,
+            0.0030266400426626205,
+            -0.009028897620737553,
+            0.0063390047289431095,
+            -0.012563719414174557,
+            0.012637191452085972,
+            -0.008114579133689404,
+            0.00022883480414748192,
+            -0.0059430720284581184,
+            -0.011110605672001839,
+            -0.012653518468141556,
+            -0.0007949267164804041,
+            -0.016531210392713547,
+            0.007432921789586544,
+            0.00981259997934103,
+            0.005045080557465553,
+            0.01025343220680952,
+            -0.0036797248758375645,
+            -0.003136848099529743,
+            -0.016751626506447792,
+            -0.009984034113585949,
+            0.0073716952465474606,
+            0.01872720755636692,
+            -0.028588788583874702,
+            -0.018547609448432922,
+            0.009290131740272045,
+            0.0022898786701261997,
+            0.018776189535856247,
+            0.0158944521099329,
+            -0.0002595757250674069,
+            0.012318813242018223,
+            -0.01964152604341507,
+            -0.009184005670249462,
+            0.009690145961940289,
+            -0.003569516818970442,
+            0.01048201136291027,
+            -0.014759717509150505,
+            0.010033016093075275,
+            -0.011159587651491165,
+            -0.011053460650146008,
+            0.01385356206446886,
+            0.01738838478922844,
+            0.008873790502548218,
+            -0.001294944784604013,
+            3.383413059054874e-05,
+            -0.010506502352654934,
+            -0.024033522233366966,
+            0.004771600943058729,
+            0.027021385729312897,
+            -0.013273949734866619,
+            0.003883813973516226,
+            0.008702355436980724,
+            -0.017502672970294952,
+            -0.019821124151349068,
+            -0.012653518468141556,
+            -0.022106921300292015,
+            0.020572172477841377,
+            0.023413091897964478,
+            -0.02026195637881756,
+            0.00877582747489214,
+            -0.004620575346052647,
+            0.014049487188458443,
+            -0.01400050614029169,
+            0.010612628422677517,
+            0.0317889042198658,
+            -0.020000724121928215,
+            -0.0028654097113758326,
+            0.0036633978597819805,
+            -0.015347493812441826,
+            0.018629245460033417,
+            0.03268689662218094,
+            -0.00176945177372545,
+            0.014041324146091938,
+            0.0008138049161061645,
+            -0.009118696674704552,
+            0.01905374974012375,
+            0.011094278655946255,
+            0.005347132217139006,
+            -0.0042491331696510315,
+            -0.012212686240673065,
+            -0.03010721132159233,
+            0.016702644526958466,
+            0.0129637336358428,
+            -0.0037674831692129374,
+            -0.0011551438365131617,
+            -0.006343086715787649,
+            -0.015404638834297657,
+            0.0035123717971146107,
+            -0.013078023679554462,
+            -0.00803294312208891,
+            -0.008759500458836555,
+            0.004314441699534655,
+            -0.0034960447810590267,
+            -0.009053388610482216,
+            0.007236996199935675,
+            0.0009510547970421612,
+            -0.0010602424154058099,
+            -0.01902109570801258,
+            -0.025078458711504936,
+            -0.00995954405516386,
+            -0.016849588602781296,
+            0.0038287099450826645,
+            0.0005234883283264935,
+            -0.027184655889868736,
+            0.0053961132653057575,
+            -0.008139070123434067,
+            -0.008134988136589527,
+            0.007126788143068552,
+            0.006881881505250931,
+            -0.007437003776431084,
+            -0.010816717520356178,
+            -0.00786150898784399,
+            -0.006865554489195347,
+            0.020033378154039383,
+            0.0034348180051892996,
+            0.012082069180905819,
+            -0.010498338378965855,
+            -0.018025141209363937,
+            -0.016963878646492958,
+            0.016718972474336624,
+            -0.008971752598881721,
+            -0.02083340659737587,
+            -0.01444950234144926,
+            -0.008424794301390648,
+            -0.007665583398193121,
+            -0.01527402177453041,
+            -0.021159948781132698,
+            -0.0032103201374411583,
+            -0.011347348801791668,
+            -0.0067431009374558926,
+            -0.0028960229828953743,
+            0.0013571919407695532,
+            0.00951871182769537,
+            0.00546958576887846,
+            0.00438791373744607,
+            0.021666089072823524,
+            0.010122815147042274,
+            -0.005461421795189381,
+            0.016441410407423973,
+            -0.006183897145092487,
+            -0.004440976772457361,
+            -0.01498013362288475,
+            -0.009918726049363613,
+            0.018547609448432922,
+            -0.0055226488038897514,
+            0.008457448333501816,
+            -0.0009536059224046767,
+            -0.00576347392052412,
+            0.015674035996198654,
+            0.0129637336358428,
+            0.016718972474336624,
+            0.004771600943058729,
+            -0.00012168807734269649,
+            0.012416775338351727,
+            -0.0015745466807857156,
+            0.012555556371808052,
+            -0.0038429962005466223,
+            0.014506647363305092,
+            -0.009722800925374031,
+            -0.018988441675901413,
+            0.004157293122261763,
+            0.006265532691031694,
+            -0.012531065382063389,
+            0.009404421783983707,
+            0.007743136957287788,
+            -0.013747435994446278,
+            -0.02261306345462799,
+            -0.000106317616882734,
+            0.00014362763613462448,
+            0.008122743107378483,
+            0.008971752598881721,
+            -0.008073761127889156,
+            0.012890261597931385,
+            0.004616493359208107,
+            -0.007179851643741131,
+            -0.019657853990793228,
+            0.026009103283286095,
+            0.015608727931976318,
+            0.012269831262528896,
+            -0.052508022636175156,
+            -0.0030980713199824095,
+            0.010743245482444763,
+            -0.015951598063111305,
+            -0.026792805641889572,
+            -0.006404313258826733,
+            -0.008759500458836555,
+            -0.04290767386555672,
+            -0.009624837897717953,
+            -0.0005724696675315499,
+            -0.011388166807591915,
+            0.010400376282632351,
+            0.01573934406042099,
+            0.014612773433327675,
+            0.005489994306117296,
+            -0.007232914678752422,
+            -0.017633290961384773,
+            -0.02788672223687172,
+            0.010057506151497364,
+            -0.013380075804889202,
+            -0.006412476766854525,
+            -0.002263347152620554,
+            -0.007690073922276497,
+            0.0016112826997414231,
+            0.01691489666700363,
+            0.042483169585466385,
+            0.023037567734718323,
+            -0.012335140258073807,
+            -0.020131340250372887,
+            -0.012800462543964386,
+            -0.009192168712615967,
+            0.00633492274209857,
+            0.011592255905270576,
+            -0.004738946910947561,
+            0.0096656559035182,
+            -0.004294032696634531,
+            0.012302485294640064,
+            0.012767808511853218,
+            0.01680060662329197,
+            -2.3023791072773747e-05,
+            0.005579793825745583,
+            -0.0024694770108908415,
+            -0.01738838478922844,
+            -0.004379750229418278,
+            0.008261523209512234,
+            -0.0008872769540175796,
+            0.0019317024853080511,
+            0.014204595237970352,
+            0.019102731719613075,
+            -0.009094206616282463,
+            -0.014612773433327675,
+            -0.016098542138934135,
+            -0.005277741700410843,
+            -0.0012622905196622014,
+            0.029927613213658333,
+            0.02037624642252922,
+            -0.019266001880168915,
+            -0.018237393349409103,
+            -0.018351683393120766,
+            0.00922482367604971,
+            -0.003563394071534276,
+            -0.0064369672909379005,
+            0.01905374974012375,
+            0.023021239787340164,
+            0.017910851165652275,
+            -0.003642988856881857,
+            -0.014865843579173088,
+            0.003804219188168645,
+            0.02847449854016304,
+            0.008367649279534817,
+            0.01078406348824501,
+            0.008726846426725388,
+            -0.0011908593587577343,
+            -0.032441988587379456,
+            -0.0113146947696805,
+            0.00951871182769537,
+            -0.008481939323246479,
+            0.026564225554466248,
+            -0.016531210392713547,
+            0.009388094767928123,
+            -0.005077734589576721,
+            0.0019286411115899682,
+            -0.02365799807012081,
+            0.0014929111348465085,
+            0.010122815147042274,
+            -0.024947840720415115,
+            0.00336950970813632,
+            -0.023592690005898476,
+            -0.00013763252354692668,
+            0.008367649279534817,
+            0.0022837561555206776,
+            0.01650671847164631,
+            -0.0113146947696805,
+            0.0051838611252605915,
+            0.02099667675793171,
+            -0.002832755446434021,
+            -0.01512707769870758,
+            -0.0007842120248824358,
+            0.0005092020728625357,
+            -0.001286781276576221,
+            0.02563357912003994,
+            0.015812816098332405,
+            -0.0036756431218236685,
+            0.020882388576865196,
+            0.008677864447236061,
+            0.0019133344758301973,
+            -0.021764053031802177,
+            0.0023796779569238424,
+            -0.032899148762226105,
+            0.017437364906072617,
+            -0.011126932688057423,
+            0.012490247376263142,
+            0.00868602842092514,
+            -0.01341272983700037,
+            -0.0007388022495433688,
+            -0.0107187544927001,
+            -0.0015725058037787676,
+            -0.007934981025755405,
+            0.009779945947229862,
+            -0.004363423213362694,
+            0.008339077234268188,
+            -0.00024579971795901656,
+            -0.01476788055151701,
+            0.018694553524255753,
+            9.678283277025912e-06,
+            -0.00010925139940809458,
+            0.005441013257950544,
+            0.009935053065419197,
+            0.00786559097468853,
+            0.002009256277233362,
+            0.023870250210165977,
+            -0.031119491904973984,
+            -0.016082214191555977,
+            -0.010204450227320194,
+            0.005543057806789875,
+            -0.007922735065221786,
+            -0.018531281501054764,
+            -0.03257260471582413,
+            0.01318415068089962,
+            0.001293924287892878,
+            -0.0009806477464735508,
+            0.01258004643023014,
+            -0.00942891277372837,
+            -0.0013878053287044168,
+            -0.01606588624417782,
+            -0.014808698557317257,
+            0.006416558753699064,
+            0.0033633869607001543,
+            -0.01222901325672865,
+            -0.023739634081721306,
+            0.014759717509150505,
+            0.0007418635650537908,
+            -0.016980206593871117,
+            -0.017502672970294952,
+            0.00018457300029695034,
+            -0.004804255440831184,
+            0.02204161323606968,
+            -0.016678154468536377,
+            0.003373591462150216,
+            0.0007219648687168956,
+            0.009232986718416214,
+            -0.02889900468289852,
+            0.00716352416202426,
+            0.0006107363733462989,
+            0.0035164537839591503,
+            -0.006518602836877108,
+            -0.021013004705309868,
+            0.011371839791536331,
+            -0.0020674217958003283,
+            -0.014751553535461426,
+            -0.020751770585775375,
+            -0.0020796670578420162,
+            -0.007420676294714212,
+            -0.02724996581673622,
+            0.017910851165652275,
+            0.0009985055075958371,
+            -0.01738838478922844,
+            0.008077843114733696,
+            0.006596156861633062,
+            0.009355440735816956,
+            0.02455599047243595,
+            0.010147306136786938,
+            -0.018776189535856247,
+            -0.0008087027235887945,
+            0.006979844067245722,
+            0.009306458756327629,
+            0.0036450298503041267,
+            0.04222193360328674,
+            -0.008130906149744987,
+            -0.006579829845577478,
+            0.0066614653915166855,
+            -0.008449285291135311,
+            -0.013747435994446278,
+            0.026727497577667236,
+            0.005796127952635288,
+            0.015984252095222473,
+            0.016123032197356224,
+            -0.00945340283215046,
+            -0.005008344538509846,
+            -0.0018786393338814378,
+            0.02065380848944187,
+            0.0073798587545752525,
+            -0.005502239800989628,
+            -0.009714636951684952,
+            0.005881845485419035,
+            -0.00898808054625988,
+            0.0021653843577951193,
+            -0.009526874870061874,
+            -0.005824700463563204,
+            -0.008069679141044617,
+            0.016400592401623726,
+            -0.0003523086488712579,
+            0.027315273880958557,
+            0.020751770585775375,
+            -0.008563575334846973,
+            -0.0005683879135176539,
+            0.017208784818649292,
+            -0.005820618476718664,
+            0.015608727931976318,
+            -0.011192241683602333,
+            -0.007706400938332081,
+            0.006408394780009985,
+            0.024017194285988808,
+            0.012237177230417728,
+            -0.0056042843498289585,
+            0.0002885308349505067,
+            -0.0030348035506904125,
+            -0.010612628422677517,
+            0.013388238847255707,
+            0.015323002822697163,
+            -0.01947825588285923,
+            -0.004722619894891977,
+            -0.0016592437168583274,
+            -0.003300119424238801,
+            0.00945340283215046,
+            0.02305389568209648,
+            0.000171817431692034,
+            0.009184005670249462,
+            0.0012765767751261592,
+            0.03862996771931648,
+            -0.0053144777193665504,
+            0.03102152980864048,
+            -0.0010423846542835236,
+            -0.0059430720284581184,
+            -0.009094206616282463,
+            -0.020441556349396706,
+            -0.011486129835247993,
+            -0.00041481092921458185,
+            -0.005040998570621014,
+            -0.0004956811899319291,
+            -0.020327266305685043,
+            0.0026960158720612526,
+            0.014637264423072338,
+            0.007661501411348581,
+            0.017927179113030434,
+            -0.002889900468289852,
+            0.017437364906072617,
+            0.019690508022904396,
+            -0.01400050614029169,
+            -0.029862305149435997,
+            -0.0006515541463159025,
+            0.01498013362288475,
+            -0.01661284640431404,
+            0.019445601850748062,
+            0.004085862077772617,
+            0.0029531680047512054,
+            0.004779764451086521,
+            -0.006049198564141989,
+            0.018482301384210587,
+            0.02039257436990738,
+            0.01453930139541626,
+            -0.013167822733521461,
+            0.014057651162147522,
+            0.009771781973540783,
+            0.006600238848477602,
+            -0.01646590232849121,
+            -0.013535182923078537,
+            0.002555194543674588,
+            0.0009245232213288546,
+            0.024392718449234962,
+            -0.020588500425219536,
+            0.0016602640971541405,
+            0.010588138364255428,
+            0.01769859902560711,
+            -0.004302196204662323,
+            0.008906444534659386,
+            0.0008740111952647567,
+            0.007134951651096344,
+            -0.0035919665824621916,
+            -0.006457376293838024,
+            -0.006673710886389017,
+            -0.010392212308943272,
+            -0.0020959940738976,
+            -0.02070278860628605,
+            0.002198038622736931,
+            -0.002830714685842395,
+            0.018237393349409103,
+            -0.01018812321126461,
+            0.038140155375003815,
+            0.03399306535720825,
+            -0.0007025764207355678,
+            0.011102442629635334,
+            -0.01709449477493763,
+            -0.003945040516555309,
+            -0.021959977224469185,
+            0.01844964735209942,
+            0.016392430290579796,
+            0.012269831262528896,
+            0.005302232690155506,
+            0.017796561121940613,
+            -0.0006301248213276267,
+            0.014490320347249508,
+            0.0029654132667928934,
+            0.007971717044711113,
+            -0.002265388146042824,
+            0.003938918001949787,
+            -0.031103165820240974,
+            -0.017029186710715294,
+            -0.01138000376522541,
+            0.004796091932803392,
+            -0.006375740747898817,
+            -0.0035205355379730463,
+            0.0084737753495574,
+            -0.005902254022657871,
+            0.0004270562494639307,
+            -0.025992777198553085,
+            -0.0157148540019989,
+            -0.010318740271031857,
+            -0.004040962550789118,
+            -0.01169838197529316,
+            -0.0027694879099726677,
+            -0.0022082431241869926,
+            0.0029552089981734753,
+            0.020947696641087532,
+            -0.012808626517653465,
+            -0.016294466331601143,
+            0.004379750229418278,
+            -0.002263347152620554,
+            -0.014016833156347275,
+            -0.016898570582270622,
+            0.006665547378361225,
+            0.012098397128283978,
+            -0.0017470018938183784,
+            -0.013608654960989952,
+            0.011739199981093407,
+            -0.006628811359405518,
+            0.04663025587797165,
+            -0.013584164902567863,
+            -0.0078002819791436195,
+            0.003720542648807168,
+            0.004698128905147314,
+            -0.009396257810294628,
+            0.007330877240747213,
+            0.013020878657698631,
+            0.01830270327627659,
+            0.0095758568495512,
+            -0.008465612307190895,
+            -0.0024388637393712997,
+            0.007796200457960367,
+            -0.016147522255778313,
+            0.006404313258826733,
+            0.0008525818702764809,
+            -0.014400520361959934,
+            0.006110425107181072,
+            0.028980640694499016,
+            0.013151495717465878,
+            0.01288209855556488,
+            -0.022351829335093498,
+            -0.004094025585800409,
+            -0.009355440735816956,
+            -0.004943036008626223,
+            -0.010090161114931107,
+            0.027070365846157074,
+            -0.018155759200453758,
+            0.02651524543762207,
+            0.00898808054625988,
+            0.0017684312770143151,
+            -0.01176369097083807,
+            0.006343086715787649,
+            0.02832755446434021,
+            0.0028592869639396667,
+            -0.020000724121928215,
+            -0.013592327944934368,
+            0.006314514204859734,
+            0.01707816869020462,
+            -0.002647034591063857,
+            -0.005428767763078213,
+            0.019657853990793228,
+            0.01784554310142994,
+            -0.004334850702434778,
+            -0.00936360377818346,
+            0.0008337036124430597,
+            0.00928196869790554,
+            0.016963878646492958,
+            0.015020951628684998,
+            0.009086042642593384,
+            -0.002647034591063857,
+            0.008037025108933449,
+            -0.008800318464636803,
+            -0.016449574381113052,
+            0.007220669183880091,
+            -0.008432958275079727,
+            -0.016963878646492958,
+            0.01872720755636692,
+            0.015257694758474827,
+            0.0181394312530756,
+            0.0024592725094407797,
+            -0.016735298559069633,
+            0.022302847355604172,
+            0.005755309946835041,
+            0.002998067531734705,
+            0.02160078100860119,
+            -0.022221211344003677,
+            0.013012715615332127,
+            0.02037624642252922,
+            0.027609162032604218,
+            -0.01737205684185028,
+            -0.010906516574323177,
+            -0.004857318475842476,
+            -0.006853308994323015,
+            -0.00176945177372545,
+            -0.008253359235823154,
+            -0.012620864436030388,
+            0.0028735732194036245,
+            -0.030466407537460327,
+            0.018106777220964432,
+            -0.005102225113660097,
+            0.010669773444533348,
+            -0.019266001880168915,
+            0.011363675817847252,
+            -0.0035838030744343996,
+            0.011135096661746502,
+            0.0019612954929471016,
+            -0.030041903257369995,
+            -0.008571738377213478,
+            0.015388311818242073,
+            -0.0030368445441126823,
+            -0.013641309924423695,
+            0.02006603218615055,
+            0.00018942011229228228,
+            -0.022270193323493004,
+            -0.004314441699534655,
+            -0.006934944540262222,
+            0.002193956868723035,
+            -0.0029613315127789974,
+            -0.0052981507033109665,
+            -0.030188847333192825,
+            0.020033378154039383,
+            0.009559528902173042,
+            -0.03072764165699482,
+            -0.003255219664424658,
+            -0.014996460638940334,
+            -0.0025184585247188807,
+            0.008881953544914722,
+            -0.004000144544988871,
+            -0.015551582910120487,
+            0.01784554310142994,
+            0.022711025550961494,
+            -0.0023164101876318455,
+            -0.016057724133133888,
+            -0.0090697156265378,
+            -0.027494871988892555,
+            0.030825603753328323,
+            -0.0066696288995444775,
+            0.0022878379095345736,
+            -0.005322641227394342,
+            0.009388094767928123,
+            0.004228724166750908,
+            0.0157148540019989,
+            0.006081852596253157,
+            -0.002647034591063857,
+            0.026596879586577415,
+            -0.01964152604341507,
+            0.009567692875862122,
+            -1.588864870427642e-05,
+            -0.003308282932266593,
+            0.01890680566430092,
+            0.0007939062779769301,
+            0.021649762988090515,
+            -0.008571738377213478,
+            -0.0001475818717153743,
+            -0.0034388997592031956,
+            0.010898353531956673,
+            -0.0028450009413063526,
+            -0.003465431509539485,
+            -0.001531688030809164,
+            0.011118769645690918,
+            0.01013097818940878,
+            0.003804219188168645,
+            -0.01737205684185028,
+            0.003142970846965909,
+            -0.008547247387468815,
+            -0.007028825581073761,
+            0.01302904263138771,
+            -0.011192241683602333,
+            -0.0018990482203662395,
+            -0.03160930797457695,
+            0.016286302357912064,
+            0.0533243790268898,
+            0.008547247387468815,
+            -0.014433175325393677,
+            -0.0018806802108883858,
+            -0.019755816087126732,
+            0.01535565685480833,
+            0.011004479601979256,
+            -0.016816934570670128,
+            -0.011984107084572315,
+            -0.012310649268329144,
+            0.007649256382137537,
+            0.021306892856955528,
+            -0.006522684823721647,
+            -0.008579902350902557,
+            0.025372346863150597,
+            -0.011788181029260159,
+            -0.010882026515901089,
+            -0.01661284640431404,
+            -0.01635977439582348,
+            0.00241641397587955,
+            0.019902760162949562,
+            -0.0169475506991148,
+            0.021176276728510857,
+            0.00936360377818346,
+            0.0010133018950000405,
+            0.003624620847404003,
+            0.008588065393269062,
+            0.0024878450203686953,
+            0.017878197133541107,
+            0.003883813973516226,
+            -0.01439235731959343,
+            0.0021408938337117434,
+            -0.0030286810360848904,
+            0.004624656867235899,
+            3.4320437407586724e-06,
+            -0.002749078907072544,
+            -0.0065635028295218945,
+            0.003710338147357106,
+            -0.021306892856955528,
+            0.021502818912267685,
+            -0.022678371518850327,
+            -0.004245051648467779,
+            0.005555302836000919,
+            -0.005118552595376968,
+            0.013469874858856201,
+            0.018792515620589256,
+            0.01423724927008152,
+            0.008702355436980724,
+            0.01462093647569418,
+            -0.011331021785736084,
+            0.02323349379003048,
+            -0.013282112777233124,
+            0.0005816536722704768,
+            0.012939243577420712,
+            0.01755165494978428,
+            0.018792515620589256,
+            -0.006245123688131571,
+            -0.010384049266576767,
+            -0.0002946535241790116,
+            0.0129637336358428,
+            -0.022286519408226013,
+            -0.012269831262528896,
+            -0.01386172603815794,
+            0.011486129835247993,
+            -0.004743028432130814,
+            0.025453981012105942,
+            -0.00633492274209857,
+            -0.014653591439127922,
+            -0.015918942168354988,
+            0.014196431264281273,
+            -0.048752784729003906,
+            0.0030348035506904125,
+            -0.0024919267743825912,
+            0.03043375350534916,
+            -0.027021385729312897,
+            0.01738838478922844,
+            -0.0028490826953202486,
+            -0.009355440735816956,
+            0.015429128892719746,
+            -0.015992414206266403,
+            -0.0046613928861916065,
+            -0.0016806729836389422,
+            -0.007539048325270414,
+            -0.00112248957157135,
+            0.017616963014006615,
+            -0.015845470130443573,
+            0.01282495353370905,
+            -0.00989423505961895,
+            0.0027021386194974184,
+            -0.004526694305241108,
+            -0.007751300930976868,
+            0.0196088720113039,
+            0.034548189491033554,
+            -0.007053316105157137,
+            0.0014990337658673525,
+            -0.022547753527760506,
+            0.013192313723266125,
+            -0.006730855442583561,
+            0.02351105399429798,
+            -0.005432849284261465,
+            -0.016898570582270622,
+            -0.013976015150547028,
+            -0.009739127941429615,
+            0.00839214026927948,
+            -0.022433463484048843,
+            -0.011110605672001839,
+            -0.00922482367604971,
+            -0.002573562553152442,
+            -0.018694553524255753,
+            0.004865481983870268,
+            -0.002426618244498968,
+            -0.012506574392318726,
+            -0.008816645480692387,
+            0.01048201136291027,
+            -0.009347276762127876,
+            0.003077662317082286,
+            -0.0011929002357646823,
+            -0.011453475803136826,
+            -0.02369065210223198,
+            0.0158944521099329,
+            0.0001974561164388433,
+            -0.010359558276832104,
+            0.008408467285335064,
+            -0.0018276170594617724,
+            0.001211268245242536,
+            0.016963878646492958,
+            -0.0020664012990891933,
+            0.003467472270131111,
+            0.023478399962186813,
+            0.004057289566844702,
+            -0.010971825569868088,
+            -0.00025906547671183944,
+            0.013535182923078537,
+            -0.005118552595376968,
+            -0.010008525103330612,
+            1.6646010408294387e-05,
+            -0.02984597720205784,
+            -0.008351322263479233,
+            0.006094098091125488,
+            0.018629245460033417,
+            -0.010620792396366596,
+            -0.015853634104132652,
+            0.026319319382309914,
+            -0.006359413731843233,
+            0.019070077687501907,
+            0.002889900468289852,
+            0.0025694805663079023,
+            0.01724144071340561,
+            0.013461710885167122,
+            0.00786150898784399,
+            -0.013927034102380276,
+            -0.02429475635290146,
+            0.0016398552106693387,
+            -0.018090449273586273,
+            0.010579974390566349,
+            -0.011028970591723919,
+            0.0032756286673247814,
+            -0.02398454025387764,
+            -0.012237177230417728,
+            0.004934872500598431,
+            -0.020343592390418053,
+            -0.00040588201954960823,
+            -0.0056940834037959576,
+            0.022237539291381836,
+            -0.005310396198183298,
+            0.008669701404869556,
+            0.0018255761824548244,
+            -0.012237177230417728,
+            0.0009877908742055297,
+            0.009469729848206043,
+            -0.029192892834544182,
+            -0.0007127808639779687,
+            -0.003138889092952013,
+            0.0011418780777603388,
+            0.005351213738322258,
+            0.009755454957485199,
+            0.007322713732719421,
+            0.0027062203735113144,
+            0.007253323681652546,
+            0.010449357330799103,
+            -0.00022041612828616053,
+            0.0197884701192379,
+            0.018172085285186768,
+            -0.007351286243647337,
+            0.008710519410669804,
+            0.024817224591970444,
+            -0.013159659691154957,
+            -0.008718682453036308,
+            0.023266147822141647,
+            -0.006461458280682564,
+            -0.0011122850701212883,
+            -0.01483318954706192,
+            0.00815947912633419,
+            0.0017367975087836385,
+            0.028131630271673203,
+            0.019657853990793228,
+            0.005820618476718664,
+            0.000639819074422121,
+            0.015788326039910316,
+            -0.013020878657698631,
+            0.0035286990460008383,
+            0.015461783856153488,
+            -0.003940958995372057,
+            -0.019821124151349068,
+            0.013878053054213524,
+            0.009502384811639786,
+            -0.003228688146919012,
+            0.01551892887800932,
+            0.011412657797336578,
+            -0.030841931700706482,
+            -0.0011898389784619212,
+            0.010498338378965855,
+            -0.0021449755877256393,
+            -0.008751336485147476,
+            -0.00014273475972004235,
+            -0.012384121306240559,
+            0.021388528868556023,
+            -0.004408322740346193,
+            0.012277995236217976,
+            -0.00815947912633419,
+            -0.017127150669693947,
+            -0.011559601873159409,
+            0.007914572022855282,
+            -0.010547320358455181,
+            -0.0006714528426527977,
+            -0.004022594541311264,
+            -0.005779800936579704,
+            0.017747581005096436,
+            0.004706292413175106,
+            -0.0008964610169641674,
+            -0.0018398624379187822,
+            0.007983962073922157,
+            -0.006979844067245722,
+            -0.008726846426725388,
+            0.004816500470042229,
+            -0.012833117507398129,
+            0.003981776535511017,
+            0.005543057806789875,
+            0.01506176870316267,
+            -0.014857680536806583,
+            -0.021159948781132698,
+            0.007020662073045969,
+            0.02695607766509056,
+            -0.02516009286046028,
+            -0.005232842173427343,
+            0.006245123688131571,
+            0.002093953313305974,
+            0.006914535537362099,
+            -0.006934944540262222,
+            -0.007208424154669046,
+            -0.03657275065779686,
+            -0.0020745648071169853,
+            0.0009005427709780633,
+            0.008963589556515217,
+            -0.008049271069467068,
+            -0.0136494729667902,
+            -0.011135096661746502,
+            0.002410291228443384,
+            0.007134951651096344,
+            -0.02204161323606968,
+            0.0049593630246818066,
+            0.009608510881662369,
+            -0.019559891894459724,
+            0.025127438828349113,
+            -0.010890189558267593,
+            -6.964537897147238e-05,
+            -0.00995954405516386,
+            0.021633435040712357,
+            -0.013486201874911785,
+            -0.018368011340498924,
+            -0.0006898208521306515,
+            0.008355404250323772,
+            0.011926962062716484,
+            -0.0017663904000073671,
+            -0.011371839791536331,
+            0.0003752686607185751,
+            0.01646590232849121,
+            0.019135385751724243,
+            0.002744997153058648,
+            -0.0024837632663547993,
+            -0.01019628718495369,
+            -0.009592183865606785,
+            -0.0073961857706308365,
+            -0.0007449248805642128,
+            0.002091912319883704,
+            0.02035992033779621,
+            0.013469874858856201,
+            0.0024837632663547993,
+            -0.015135240741074085,
+            -0.005257333163172007,
+            0.008939098566770554,
+            0.009412584826350212,
+            -0.011690218932926655,
+            -0.00898808054625988,
+            0.002659279853105545,
+            0.02249877341091633,
+            0.017633290961384773,
+            -0.0014000505907461047,
+            0.0032878739293664694,
+            -0.0016592437168583274,
+            0.012506574392318726,
+            -0.007555375341325998,
+            -0.010269759222865105,
+            -0.006943108048290014,
+            -0.014645427465438843,
+            -0.010800390504300594,
+            -0.0023062059190124273,
+            -0.010179960168898106,
+            0.022629389539361,
+            0.007988044060766697,
+            0.009779945947229862,
+            0.008265605196356773,
+            0.010269759222865105,
+            0.002087830565869808,
+            0.025715215131640434,
+            -0.004424649756401777,
+            -0.011984107084572315,
+            0.007236996199935675,
+            -0.022368155419826508,
+            -0.021535472944378853,
+            0.008257441222667694,
+            0.006367577239871025,
+            0.03595232218503952,
+            -0.026890767738223076,
+            0.0052981507033109665,
+            -0.009779945947229862,
+            -0.010400376282632351,
+            0.007996207103133202,
+            -0.0015439334092661738,
+            -0.004245051648467779,
+            0.022368155419826508,
+            0.00726556871086359,
+            0.0035797213204205036,
+            0.0032817514147609472,
+            0.019837452098727226,
+            -0.005685919895768166,
+            0.01573934406042099,
+            0.0259274672716856,
+            0.024621298536658287,
+            0.009388094767928123,
+            -0.009469729848206043,
+            0.02785406820476055,
+            0.0002895512734539807,
+            -0.0034327772445976734,
+            0.007220669183880091,
+            -0.00305113079957664,
+            -0.0009556467994116247,
+            -0.010033016093075275,
+            0.021764053031802177,
+            -0.001455154619179666,
+            0.004359341226518154,
+            0.02081707864999771,
+            -7.774515688652173e-05,
+            0.006220633164048195,
+            0.01057181041687727,
+            0.002181711606681347,
+            0.003396041225641966,
+            0.010775899514555931,
+            -0.039315707981586456,
+            -0.0006571666453965008,
+            -0.025600925087928772,
+            -0.0018286375561729074,
+            0.0007357408758252859,
+            -0.01266984548419714,
+            0.013600491918623447,
+            -0.002015379024669528,
+            -0.010751409456133842,
+            -0.00029337796149775386,
+            0.021649762988090515,
+            0.005416522268205881,
+            0.004071575589478016,
+            -0.004469549283385277,
+            0.018792515620589256,
+            -0.006955353543162346,
+            -0.017861871048808098,
+            -0.012759645469486713,
+            -0.012367794290184975,
+            -0.01918436773121357,
+            -0.01214737817645073,
+            -0.0024551907554268837,
+            -0.002667443361133337,
+            0.004738946910947561,
+            0.002481722505763173,
+            -0.003228688146919012,
+            -0.02070278860628605,
+            0.015037278644740582,
+            -0.01430255826562643,
+            0.012873934581875801,
+            -0.011053460650146008,
+            0.005963481031358242,
+            -0.004122598096728325,
+            0.012726990506052971,
+            -0.021404854953289032,
+            0.02070278860628605,
+            -0.01536382082849741,
+            0.0038532004691660404,
+            0.019592545926570892,
+            0.006551257334649563,
+            -0.008118661120533943,
+            -0.013404566794633865,
+            -0.022074267268180847,
+            0.021633435040712357,
+            -0.011633073911070824,
+            -0.018237393349409103,
+            -0.004608329851180315,
+            0.009649328887462616,
+            -0.017470018938183784,
+            -0.007367613259702921,
+            0.01707816869020462,
+            -0.0018949664663523436,
+            0.012980061583220959,
+            0.023004913702607155,
+            -0.0052981507033109665,
+            0.0036572751123458147,
+            -0.009167678654193878,
+            0.0010286086471751332,
+            -0.02470293454825878,
+            0.013527019880712032,
+            0.008971752598881721,
+            0.006269614677876234,
+            -0.013935198076069355,
+            -0.013306603766977787,
+            0.0036531933583319187,
+            0.004498121794313192,
+            0.011437147855758667,
+            0.009420748800039291,
+            0.004220560658723116,
+            -0.013918871060013771,
+            -0.027788760140538216,
+            -0.027070365846157074,
+            0.03595232218503952,
+            -0.0005002731923013926,
+            -0.020947696641087532,
+            0.03399306535720825,
+            0.016114868223667145,
+            0.0022980421781539917,
+            -0.012449429370462894,
+            -0.004314441699534655,
+            0.024621298536658287,
+            0.016400592401623726,
+            0.010465684346854687,
+            -0.0004742518358398229,
+            0.005571629852056503,
+            0.0016469983384013176,
+            0.03012353740632534,
+            0.015323002822697163,
+            0.0009459525463171303,
+            -0.038597315549850464,
+            0.011943289078772068,
+            -0.007547211833298206,
+            0.0013408648082986474,
+            0.006702283397316933,
+            0.017780235037207603,
+            0.011935125105082989,
+            0.015567909926176071,
+            0.014865843579173088,
+            -0.003065417055040598,
+            0.0029511270113289356,
+            0.0006163488142192364,
+            -0.009624837897717953,
+            -0.001767410896718502,
+            -0.008881953544914722,
+            -0.005714492406696081,
+            -0.002920513739809394,
+            0.005592038854956627,
+            0.007657419890165329,
+            -0.003983817528933287,
+            0.028049994260072708,
+            -0.0006433906382881105,
+            -0.017600636929273605,
+            -0.009061552584171295,
+            -0.03611559048295021,
+            0.00434301421046257,
+            -0.030939893797039986,
+            -0.011257549747824669,
+            0.0033674687147140503,
+            -0.020457882434129715,
+            -0.024768242612481117,
+            0.0010306495241820812,
+            0.014335212297737598,
+            0.007392104249447584,
+            0.0033327736891806126,
+            -0.01004934310913086,
+            0.003816464450210333,
+            -0.001855169073678553,
+            -0.006461458280682564,
+            0.028833696618676186,
+            0.019559891894459724,
+            0.020457882434129715,
+            -0.01190247107297182,
+            0.019298657774925232,
+            0.005845109466463327,
+            0.02035992033779621,
+            0.010465684346854687,
+            -0.0048695639707148075,
+            0.0074859848245978355,
+            0.009567692875862122,
+            -0.006155324634164572,
+            -0.021519144997000694,
+            0.013192313723266125,
+            -0.0010837126756086946,
+            -0.0052083516493439674,
+            -0.016269976273179054,
+            -0.003534821793437004,
+            -0.015812816098332405,
+            0.010547320358455181,
+            -0.017159804701805115,
+            0.012498411349952221,
+            -0.000802069844212383,
+            0.00168475485406816,
+            -0.015469946898519993,
+            -0.013951525092124939,
+            -0.0012439225101843476,
+            -0.01244126632809639,
+            -0.008971752598881721,
+            0.006163488142192364,
+            -0.007539048325270414,
+            -0.008318668231368065,
+            -0.00535937724635005,
+            -0.011437147855758667,
+            -0.010882026515901089,
+            0.004367504734545946,
+            -0.0010653446661308408,
+            0.020931368693709373,
+            0.00011671340325847268,
+            -0.005824700463563204,
+            0.01190247107297182,
+            0.02787039615213871,
+            0.0016704685986042023,
+            0.004583839327096939,
+            -0.014261740259826183,
+            -0.011118769645690918,
+            -0.02279266156256199,
+            0.016572028398513794,
+            0.0049675265327095985,
+            -0.0344502255320549,
+            0.01438419334590435,
+            0.00824927818030119,
+            0.0015102586476132274,
+            -0.006698201410472393,
+            -0.005334886722266674,
+            -0.005608365871012211,
+            0.006306350696831942,
+            -0.0022837561555206776,
+            0.0005107327597215772,
+            0.0035919665824621916,
+            0.034678805619478226,
+            -0.0040674940682947636,
+            -0.007228832691907883,
+            0.00981259997934103,
+            0.0005130287609063089,
+            0.020196648314595222,
+            -0.005518566817045212,
+            0.0037572786677628756,
+            -0.019282329827547073,
+            -0.016563864424824715,
+            0.0018429238116368651,
+            -0.01722511276602745,
+            0.023282473906874657,
+            -0.005392031744122505,
+            -0.01347803883254528,
+            -0.006906372029334307,
+            -0.020115012302994728,
+            0.012637191452085972,
+            0.02310287579894066,
+            0.007016580086201429,
+            0.01033506728708744,
+            0.012131051160395145,
+            -0.02444170042872429,
+            -0.020457882434129715,
+            0.012171869166195393,
+            0.0013571919407695532,
+            -0.008449285291135311,
+            -0.005404277238994837,
+            -0.001783737912774086,
+            0.007783954963088036,
+            -0.0032633834052830935,
+            0.007881917990744114,
+            0.06491663306951523,
+            -0.01093100756406784,
+            -0.002987863030284643,
+            -0.02966637909412384,
+            -0.004396077245473862,
+            -0.006628811359405518,
+            0.0056287748739123344,
+            0.009151351638138294,
+            -0.023135529831051826,
+            0.009232986718416214,
+            -0.030025575309991837,
+            -0.010628955438733101,
+            0.00824111420661211,
+            0.0028490826953202486,
+            -0.01220452319830656,
+            -0.0009592183632776141,
+            -0.013665799982845783,
+            0.0006107363733462989,
+            -0.009412584826350212,
+            -0.00576347392052412,
+            0.012718827463686466,
+            -0.02321716584265232,
+            0.006298186723142862,
+            0.007592111360281706,
+            -0.006514521315693855,
+            -0.013722945004701614,
+            -0.017861871048808098,
+            -0.011967780068516731,
+            -0.013314766809344292,
+            0.002996026771143079,
+            0.01190247107297182,
+            -0.02607441134750843,
+            0.012922916561365128,
+            -0.0037389106582850218,
+            0.00945340283215046,
+            0.004404240753501654,
+            0.013469874858856201,
+            0.018074123188853264,
+            0.009355440735816956,
+            0.0021429345943033695,
+            -0.008024780079722404,
+            0.012702500447630882,
+            -0.01993541419506073,
+            -0.01635161228477955,
+            0.013837235048413277,
+            0.012016761116683483,
+            0.003061335301026702,
+            0.009828926995396614,
+            0.0020959940738976,
+            0.011886144056916237,
+            0.010775899514555931,
+            -0.019739490002393723,
+            -0.0006648199632763863,
+            0.0007326795603148639,
+            0.03280118480324745,
+            -0.0005938990507274866,
+            0.008857463486492634,
+            0.004420568235218525,
+            0.013159659691154957,
+            -0.012784135527908802,
+            -0.01769859902560711,
+            0.0008087027235887945,
+            -0.01057181041687727,
+            0.021257910877466202,
+            -0.008192133158445358,
+            0.006216551177203655,
+            -0.018612917512655258,
+            -0.027086693793535233,
+            0.006392067763954401,
+            -0.016261812299489975,
+            -0.018221067264676094,
+            0.01317598670721054,
+            0.0077268099412322044,
+            0.0017572063952684402,
+            -0.005098143592476845,
+            -0.020457882434129715,
+            -0.009837090037763119,
+            0.028441844508051872,
+            0.011102442629635334,
+            -0.004120557103306055,
+            0.009584019891917706,
+            -0.009094206616282463,
+            -0.012490247376263142,
+            -0.0017541450215503573,
+            0.0036715613678097725,
+            0.008261523209512234,
+            0.005645101889967918,
+            -0.01386172603815794,
+            -0.00029873527819290757,
+            -0.012400448322296143,
+            -0.021225256845355034,
+            -0.008906444534659386,
+            0.016751626506447792,
+            0.007820690982043743,
+            -0.0038960592355579138,
+            0.015812816098332405,
+            -0.004404240753501654,
+            0.002189875114709139,
+            0.014294394291937351,
+            0.005334886722266674,
+            -0.004285869188606739,
+            -0.0095758568495512,
+            -0.004612411838024855,
+            0.004043003544211388,
+            -0.0025960123166441917,
+            0.006457376293838024,
+            0.017192458733916283,
+            0.0072900597006082535,
+            -0.013755599036812782,
+            0.013135168701410294,
+            0.0007791098323650658,
+            0.0049675265327095985,
+            -0.03699725493788719,
+            -0.003247056156396866,
+            0.015796490013599396,
+            -0.013020878657698631,
+            0.02501314878463745,
+            0.005400195252150297,
+            0.022743679583072662,
+            0.015233203768730164,
+            0.008669701404869556,
+            -0.001542912912555039,
+            0.00464098434895277,
+            0.02759283408522606,
+            -0.003151134354993701,
+            -0.008637047372758389,
+            -0.0020602785516530275,
+            0.009241150692105293,
+            0.0002063850115519017,
+            0.003000108525156975,
+            0.008281932212412357,
+            -0.03562577813863754,
+            0.011706545948982239,
+            -0.019576217979192734,
+            0.0024694770108908415,
+            0.003459308762103319,
+            -0.007183933164924383,
+            -0.014865843579173088,
+            -0.005857354495674372,
+            0.022841641679406166,
+            -0.003224606392905116,
+            0.021078312769532204,
+            0.002885818714275956,
+            0.016106704249978065,
+            -0.011575928889214993,
+            -0.009208496659994125,
+            -0.0004130251472815871,
+            -0.005351213738322258,
+            -0.007539048325270414,
+            0.010596301406621933,
+            0.0034409407526254654,
+            0.005534893833100796,
+            0.002265388146042824,
+            0.0036654386203736067,
+            -0.0036878883838653564,
+            0.02773977816104889,
+            0.003563394071534276,
+            0.007996207103133202,
+            -0.011567764915525913,
+            0.014082142151892185,
+            -0.017029186710715294,
+            0.0070778070949018,
+            0.005575711838901043,
+            -0.026156047359108925,
+            0.016849588602781296,
+            0.004775682929903269,
+            0.006628811359405518,
+            0.013812744058668613,
+            0.01618834026157856,
+            -0.010408539324998856,
+            -0.014816862531006336,
+            0.024049850180745125,
+            -0.0032041973900049925,
+            -0.015225040726363659,
+            -0.006775755435228348,
+            4.438935866346583e-05,
+            0.006559420842677355,
+            0.00373074715025723,
+            0.0013020879123359919,
+            -0.0006204306264407933,
+            -0.010743245482444763,
+            0.004645065870136023,
+            -0.01934763789176941,
+            -0.0007163524278439581,
+            -0.016449574381113052,
+            -0.015633217990398407,
+            0.018678225576877594,
+            -0.01830270327627659,
+            0.009633000940084457,
+            0.0012275953777134418,
+            0.028588788583874702,
+            -0.011706545948982239,
+            -0.012155541218817234,
+            0.0015102586476132274,
+            0.010849371552467346,
+            0.0067512644454836845,
+            -0.010033016093075275,
+            -0.007583947852253914,
+            -0.014180104248225689,
+            -0.010163633152842522,
+            0.015502601861953735,
+            -0.014498483389616013,
+            -0.014343376271426678,
+            0.008914608508348465
+          ],
+          "embedding": [
+            -0.006359657085039302,
+            -0.024594786241399234,
+            -0.02553538924848095,
+            0.005360188465088291,
+            0.012801870977589839,
+            0.021680505862506104,
+            -0.016668048966001273,
+            0.06860852474581301,
+            -0.007122272841157724,
+            0.03471775000705488,
+            -0.009004297848861675,
+            0.01059974144678391,
+            0.02565604599426148,
+            -0.022544127646041287,
+            0.010616674976540785,
+            0.013658194823359737,
+            0.016400329743363844,
+            0.02858546964917478,
+            0.0067612676434693686,
+            -0.04174737193606879,
+            0.03184155110488616,
+            -0.02223105127585514,
+            -0.05506239667974977,
+            -0.0015307551410555752,
+            -0.013169788751379287,
+            0.017681749722384007,
+            0.01682706317229353,
+            0.019652295823816928,
+            -0.05623980369271269,
+            0.030698582851533197,
+            0.02286327124296613,
+            0.016019032539689178,
+            0.005508113353872315,
+            8.58985091215149e-05,
+            -0.019206292853495906,
+            -0.018439290483495144,
+            0.03475023176950151,
+            0.01018612942141759,
+            -0.0252551158857426,
+            0.0334562130666827,
+            0.020462153537485785,
+            0.02260508249714211,
+            -0.03323524989392951,
+            -0.002094251009530707,
+            0.015177442347686773,
+            -0.00018698464350038317,
+            -0.0006788648344273403,
+            0.0007777616893581039,
+            0.013352398615059647,
+            -0.030751111528744813,
+            -0.0317733818445192,
+            -0.006729064081710058,
+            0.009413204899986514,
+            -0.005712894407943592,
+            -0.038961826880527066,
+            -0.002827692206530345,
+            0.017863300537973137,
+            0.07918455388563496,
+            0.048967158919816366,
+            0.018002566325350088,
+            0.040264962620950695,
+            -0.019715100359659787,
+            -0.02211350500879408,
+            0.004918148608475574,
+            0.01062585305866606,
+            -0.014478993521696027,
+            0.0034983131226364672,
+            0.000766228629514787,
+            0.009667268771502067,
+            -0.01558439890697323,
+            -0.04984718709870083,
+            0.029613955432821887,
+            -0.009686854114630624,
+            -0.015279319238124568,
+            0.007460629676325322,
+            -0.006235979350809034,
+            0.012514387473717085,
+            0.02800435263741497,
+            -0.008688925946672778,
+            -0.01236301543374469,
+            0.029755705020188972,
+            0.01158037219446542,
+            -0.005242392206024511,
+            0.009982679027728814,
+            0.052414361714483264,
+            -0.018788500381726227,
+            -0.020335831196027204,
+            -0.08583959943356621,
+            0.029309228216812346,
+            -0.0249285282010673,
+            -0.018929396555426307,
+            0.007710663193334574,
+            0.03096798035958051,
+            -0.017089977064591828,
+            -0.028015107605405817,
+            0.06740703271128606,
+            0.009398795412354399,
+            0.01118769022826835,
+            -0.03041417460855722,
+            0.014776002883150835,
+            0.004319194067673161,
+            0.022507636639041203,
+            0.0053562788646786845,
+            0.007548072258943124,
+            0.011123374308563128,
+            -0.006924362115585165,
+            -0.011118507147092024,
+            0.009563232643295304,
+            0.00657178473678069,
+            0.08313027447380966,
+            -0.004239370156003264,
+            -0.00930406163946907,
+            -0.039202672821898946,
+            0.009761029930556298,
+            -0.02827698465424993,
+            -0.037914533246900244,
+            -0.04496050911855815,
+            -0.017711307900236353,
+            0.01934435542287117,
+            0.004335173027873153,
+            0.03800509969714242,
+            0.031198127357716314,
+            0.031311207941680276,
+            -0.012184553894588813,
+            -0.058985863670856876,
+            0.041442017057707635,
+            -0.012456189292721711,
+            -0.0040966351720394135,
+            0.010849179519959421,
+            0.005243123049347636,
+            0.0011495799963862647,
+            0.003534183676934487,
+            -0.03139340512003631,
+            -0.023876261883905306,
+            0.008382957569752911,
+            -0.004778283217011976,
+            0.03703067589888386,
+            0.06888630688839406,
+            -0.03993532296478411,
+            -0.009472167678222372,
+            0.025246872942105057,
+            -0.01524567912700368,
+            0.01595917939878582,
+            -0.003736723164442467,
+            -0.030836797784967753,
+            0.0026246446163368533,
+            0.008894300082148884,
+            0.08342856279154155,
+            -0.0007476580521704578,
+            0.025149116238550588,
+            0.004972542633631468,
+            0.034495632546556576,
+            -0.009173329764686424,
+            0.014009381543772768,
+            0.004427261668151028,
+            -0.013426138607902904,
+            -0.022234568880321808,
+            0.01976282364689966,
+            -0.03592433789490043,
+            -0.010918278220831188,
+            0.013527180289641174,
+            -0.013023113052242677,
+            -0.008304770302495363,
+            0.003677063799167178,
+            0.0013981822663219421,
+            0.047001163242600175,
+            0.01828332714887986,
+            0.002483375662343689,
+            -0.06820498187523893,
+            0.05295312038995555,
+            0.025038753504342162,
+            0.025286870198436735,
+            -0.0029544808216316115,
+            0.029982752196635428,
+            0.0005600481523874767,
+            0.0128290258663539,
+            0.009023805227424919,
+            -0.0019219166367750596,
+            -0.04482843936478531,
+            0.04523479691409549,
+            -0.004678574027241186,
+            0.010486850804218904,
+            0.006736229865018014,
+            0.02134813656476443,
+            0.028083972673690036,
+            0.016857413440125445,
+            0.0189082447820146,
+            0.03843690613820022,
+            -0.011922027011229613,
+            0.016193433524453224,
+            0.03828207911467065,
+            -0.0013055128404331705,
+            0.043198163443700956,
+            0.024025562893260542,
+            -0.001257851946749882,
+            -0.011726713433348844,
+            -0.021907984099443396,
+            0.02752198420597673,
+            0.004431866336999851,
+            0.015798806714413912,
+            -0.012910782191382367,
+            -0.03180549180206364,
+            0.004550247372715741,
+            -0.022778272841118514,
+            0.031242218240017532,
+            -0.006106984949693483,
+            -0.006259993190497421,
+            0.02187449047033339,
+            0.009334146847221406,
+            -0.009381278960016145,
+            -0.020888853823931134,
+            -0.007502374908776156,
+            -0.010390018386728482,
+            -0.04316693435221676,
+            -0.006274513730720055,
+            0.019152443937064258,
+            -0.01668984824845791,
+            0.018573194036505578,
+            0.0035035262183249925,
+            -0.019779911381789077,
+            -0.0027495798165085993,
+            0.02425481236315913,
+            0.006857723545270935,
+            -0.014733535699087731,
+            -0.026326146471941727,
+            -0.019906660231056833,
+            -0.001033294559612807,
+            0.0159154063026895,
+            0.035649534442431415,
+            -0.026210592096240003,
+            0.014397127229517285,
+            0.005114311681443673,
+            -0.0033104236005207077,
+            0.00324988235167305,
+            0.03764004154553922,
+            -0.017822437760737605,
+            -0.008579701063708334,
+            -0.00032410984200965487,
+            0.018450351071676443,
+            0.017283379764846195,
+            0.006384411611332382,
+            -0.005115090309123598,
+            -0.030590047062583125,
+            -0.011776211359193503,
+            0.01183623914060493,
+            0.0016564847796208331,
+            -0.009530766634361575,
+            -0.004810598258154945,
+            0.03876242515526699,
+            0.027076785774401667,
+            0.005324763136165989,
+            -0.03416412480913939,
+            -0.007943016733586697,
+            0.002114307650456928,
+            -0.02961449347300628,
+            0.05142732770518828,
+            0.05566026433842563,
+            -0.0030729309664974704,
+            -0.00938186948393074,
+            -0.04547908981236149,
+            0.0651216359692231,
+            -0.01813722445944094,
+            -0.0482650853560993,
+            0.044045155892545554,
+            -0.014272653003441165,
+            0.013740652306746851,
+            0.04563732439184395,
+            0.02126513340521316,
+            -0.0054845632439384905,
+            -0.011635625101262898,
+            0.030992082069685376,
+            -0.010051935721068779,
+            -0.050967258971854425,
+            -0.009827942095033494,
+            0.013373878556654464,
+            -0.013017627321534534,
+            -0.009361001542607812,
+            -0.01925867665425919,
+            0.020329052915436022,
+            0.023020741670660188,
+            0.019387043947695172,
+            0.005628632318152202,
+            0.018451555161952665,
+            0.016226845400848455,
+            -0.01384033112836216,
+            0.004762589697426251,
+            -0.012718476561725138,
+            -0.024222765127494463,
+            -0.037487144314010595,
+            -0.019971524166659278,
+            -0.015034588610516958,
+            0.009483936496215338,
+            -0.014301924283553591,
+            -0.0024139665139412125,
+            0.013361045353529458,
+            -0.0005870798721107379,
+            -0.011192319340879789,
+            0.013927695434391159,
+            0.021337539669342293,
+            -0.027300952495815437,
+            -0.04765670695199488,
+            -0.00561706224251562,
+            0.0082776639906139,
+            -0.0030984973749633724,
+            -0.026269016057688548,
+            0.007040524859505573,
+            0.0017859008072463442,
+            0.020091542517503808,
+            -0.018716761130867725,
+            0.0035283051960672963,
+            0.008519383548064053,
+            -0.015247417050336866,
+            -0.016917626868166634,
+            -0.018580550690103854,
+            -0.024697180580294884,
+            0.034202028316737125,
+            0.0204395003463227,
+            0.030392895534267717,
+            -0.04527669147753553,
+            -0.018359428092075348,
+            0.05494107499558624,
+            0.015234754907198422,
+            -0.05066008106884831,
+            0.031139824660661085,
+            0.005016615710206006,
+            0.0019000453624929544,
+            -0.006094364589229031,
+            0.0637324238017742,
+            -0.02711381489292883,
+            -0.02187418906973867,
+            0.004601744864532875,
+            -0.04141121277496072,
+            -0.026689585608019588,
+            -0.005568100750837504,
+            0.01291019036002459,
+            -0.031853041426355644,
+            -0.02815800259372657,
+            -0.007964861641790374,
+            -0.010933515363060793,
+            0.013726639077745378,
+            -0.023956800669647334,
+            -0.015672058590455337,
+            -0.023642025374517057,
+            -0.01288435863544158,
+            -0.0042516881552192385,
+            -0.011162214160460695,
+            -0.02412733256710621,
+            0.010088528028211243,
+            0.03310175458095482,
+            0.023064360903748363,
+            0.001996159684818467,
+            0.009478229572933101,
+            0.014478772877286076,
+            0.00371783700962415,
+            0.007864103610320116,
+            -0.048511515124609,
+            -0.031398754206534776,
+            0.020227872561659397,
+            0.012159811587094092,
+            0.0172163448580163,
+            0.009578952796095119,
+            -0.04154545961112031,
+            -0.05095080393682759,
+            0.02968486144855133,
+            -0.03932395023720674,
+            -0.02063219739061335,
+            0.007512423249865194,
+            -0.011911709468952548,
+            -0.00810951734088105,
+            -0.0008079327452253181,
+            -0.01600701450676789,
+            0.023984909975645632,
+            -0.02418072541081482,
+            0.0033984265728439398,
+            -0.0104343782811971,
+            0.01588977364765743,
+            0.05109529246167992,
+            -0.027800269136366997,
+            -0.027889034481533438,
+            0.03706813210665186,
+            -0.02058120652429308,
+            0.030801396468451574,
+            0.03246127847621613,
+            0.008355517405358826,
+            -0.008605467586247932,
+            0.017693344563968573,
+            0.014499487024471321,
+            0.012431277080478217,
+            -0.040103870071440256,
+            0.017328848567062113,
+            0.02723900253727976,
+            -0.054909074068976875,
+            -0.008547596270650966,
+            -0.00777964838164648,
+            0.027134666572242665,
+            0.02765036594970255,
+            -0.002207173084941375,
+            -0.016838740117653725,
+            -0.028289233739722936,
+            -0.019433376533829132,
+            -0.011053380458946764,
+            0.03247641283550457,
+            0.0007514989009289282,
+            0.007919574219000797,
+            0.001316042436488591,
+            -0.0017934342041762042,
+            0.01924145734197619,
+            0.015260199553572459,
+            0.048259602293134723,
+            -0.03383050622747848,
+            0.0021712225589426266,
+            -0.03508567715696004,
+            -0.04578731401250471,
+            -0.008768228935292693,
+            0.010203318819548585,
+            0.024135236371743855,
+            -0.007467140460133838,
+            0.0028907962385387818,
+            -0.007582083854422456,
+            -0.005384477718912037,
+            -0.01867978482547249,
+            0.01641191108381631,
+            0.037168089880543244,
+            0.0031992495663980464,
+            -0.008533890994695162,
+            0.000778843942291118,
+            -0.004620258538699103,
+            0.010572690335163528,
+            0.033233014627100904,
+            -0.030497362482602697,
+            0.02974571152880676,
+            -0.004274442492343003,
+            0.025341347125910928,
+            -0.04307748954707389,
+            -0.018069609916024403,
+            -0.018491122727274337,
+            -0.02069307802429807,
+            0.05372967226264594,
+            -0.030768978301976772,
+            -0.03662045886993282,
+            -0.003205991669059688,
+            0.016815511161004534,
+            -0.0014152644009276566,
+            -0.0016015439520090104,
+            -0.03984716970293867,
+            -0.010828260334550217,
+            -0.05894985435251326,
+            -0.01638035185618134,
+            -0.021819442096404948,
+            -0.005315102328745982,
+            -0.06251579337157112,
+            0.0033713876515840233,
+            -0.07815974854883057,
+            0.001484860071649606,
+            0.04539427129131382,
+            -0.023530943197353476,
+            0.05142250021536681,
+            -0.024746600229063383,
+            -0.0005885740455257141,
+            -0.044698536284421976,
+            0.0026481001040031785,
+            0.02389178603305878,
+            0.051387657872255674,
+            -0.002949198236157656,
+            0.010674377196609455,
+            0.013322075519401089,
+            -0.026245090721950647,
+            -0.01036724558050635,
+            -0.005321884180696542,
+            0.005958798671856245,
+            0.013755178085972732,
+            -0.0074734329537091675,
+            0.038368016046563674,
+            0.010941421833174982,
+            0.03443115781919206,
+            0.034623617136788874,
+            -0.004361926531008214,
+            -0.03348681614127998,
+            -0.00632870804659212,
+            0.024433331840566692,
+            0.0321998807702602,
+            -0.02899983280320611,
+            0.005958471722097962,
+            -0.015602447981516634,
+            0.015633217539747047,
+            0.006321931116948646,
+            -0.011827541320878442,
+            0.02372204564999124,
+            0.015673426811623124,
+            0.006103300725621424,
+            -0.005165006670545588,
+            0.013589526056189892,
+            -0.025411864292353578,
+            0.03041791947375232,
+            -0.004456865298584878,
+            -0.04276058054583262,
+            -0.0024736078733106825,
+            -0.020059309867945227,
+            0.024883721794000956,
+            0.015578237413840115,
+            -0.00536523508503184,
+            -0.005290645798641171,
+            0.008242987948511412,
+            0.020469675635821673,
+            0.006668354647245447,
+            0.00017729517736814364,
+            0.02373604785121342,
+            0.029915687122900485,
+            -0.02432635861789771,
+            -0.029540533943782886,
+            0.008151297444985344,
+            -0.005730733861939129,
+            0.011318816579801282,
+            -0.02644855229221747,
+            0.00033932872948409137,
+            0.0035505684154028996,
+            -0.03570104343808711,
+            0.03408366016010711,
+            -0.009199700348630527,
+            0.005857598530788056,
+            -0.009538177736472163,
+            -0.0017768806148210118,
+            0.0027410153099351694,
+            0.03898108970435115,
+            -0.007367623392751247,
+            0.009827254226889936,
+            0.015465895035342481,
+            0.000981604396724216,
+            0.05143083558797524,
+            -0.027350877485564314,
+            -0.02183036962301852,
+            0.01885468378168636,
+            -0.007285239007489837,
+            -0.016846837665812908,
+            -0.020546383007536074,
+            -0.011863943814684079,
+            -0.016293192750336218,
+            0.01976516340568883,
+            -0.02387328798681798,
+            -0.011892336568452852,
+            0.007170651391639504,
+            0.0013725014050721622,
+            -0.009809669068563384,
+            0.0016617183704652932,
+            -0.006307590349798603,
+            0.02037693835615797,
+            0.031753656615094446,
+            -0.024892675105483313,
+            0.03577919637821846,
+            0.03178303188315581,
+            0.017260898395251155,
+            -0.02102289690701718,
+            -0.009661035772073294,
+            -0.00048542689442823136,
+            -0.014694631967051945,
+            -0.0326379186158631,
+            0.025443206465204822,
+            0.017194626456982255,
+            -0.011381421003142453,
+            -0.03680783761649673,
+            0.0017216115811536721,
+            0.022819859205829785,
+            -0.013600989058799166,
+            0.028141657788496834,
+            -0.01869047298692401,
+            -0.01366833755869951,
+            -0.00782959271979894,
+            0.031227914441058244,
+            -0.05252392878125891,
+            0.016957842026348195,
+            0.00020880696574616416,
+            0.001154370575234767,
+            0.010892067708194324,
+            -0.008368819756628046,
+            0.012224564683325483,
+            0.031700389750709,
+            0.018063405869147564,
+            0.0010794115307202266,
+            -0.002783668216641561,
+            0.01418234057248712,
+            0.017773269840982066,
+            -0.015653443355569053,
+            -0.000497740485657824,
+            0.019890882604100452,
+            0.022672054781919238,
+            -0.00521797370595221,
+            -0.016686942830774276,
+            -0.018443538351123584,
+            0.025270057639201107,
+            -0.010354431070738671,
+            0.03484047753830186,
+            0.04594785214094975,
+            0.0036441959528283004,
+            -0.05695476763071743,
+            -0.009412209619190693,
+            0.0010371110229966086,
+            -0.01099343927053564,
+            0.01606863921935641,
+            0.012470660511773616,
+            -0.03237187923799263,
+            0.006388672213167589,
+            0.012946570716567364,
+            0.022152554488829627,
+            -0.02540317621005867,
+            -0.023183252921885226,
+            -0.0030984019855787655,
+            -0.007890467768631811,
+            0.018648760402624687,
+            -0.008653405980661855,
+            0.0006228315253506687,
+            0.009672526897422602,
+            0.0005436494014569835,
+            0.011632326145022935,
+            0.005655942007108228,
+            0.021736827172951,
+            -0.001256608167230203,
+            0.013124413886287987,
+            -0.01986808920858981,
+            -0.0270450339809516,
+            0.006500423802582728,
+            0.04438242119211055,
+            -0.014813399694814117,
+            0.030621709833336064,
+            0.0092171224268983,
+            0.016433384739642534,
+            0.024875072913970443,
+            -0.00891470467241686,
+            -0.014124009014664287,
+            0.00845777541223868,
+            0.009019007848927453,
+            -0.04147358041382793,
+            0.012513030150663913,
+            -0.01710243399313324,
+            -0.033144939719206276,
+            0.01904674924133127,
+            -0.00534122554342279,
+            0.012367320751926314,
+            0.004270315800911569,
+            -0.002886460711688153,
+            -0.025696286516581048,
+            0.014137901708875033,
+            0.028135087538950946,
+            0.021797869775098042,
+            0.0296173052503884,
+            -0.012757078368361444,
+            0.0005460601780756187,
+            -0.008748930817524899,
+            0.005470975782187157,
+            0.04187083128854477,
+            0.0008687182830616019,
+            -0.0008032802520818105,
+            0.01507726552487313,
+            0.024945744517309208,
+            0.0376130540370386,
+            -0.015415560911023553,
+            -0.03407353556457147,
+            0.01564969972264396,
+            -0.010104443707865262,
+            0.018855792875436674,
+            0.005503259932317811,
+            0.0049634097823351596,
+            0.0038600156268000702,
+            0.019312419743287703,
+            -0.006635193789316492,
+            0.018798045806429395,
+            -0.02500983683871332,
+            -0.028785757874583755,
+            0.01829411751138161,
+            -0.005277972443258226,
+            0.03338623299668019,
+            0.016225885095995672,
+            -0.029522280454919586,
+            -0.01712097344863975,
+            0.01553706187924322,
+            -0.008465766956343768,
+            -0.024842700926275538,
+            0.01895228828512998,
+            0.007076862934523977,
+            0.008956105593078965,
+            0.0024373779389833218,
+            -0.01106503696703573,
+            0.01125466547367417,
+            -0.000818223040067914,
+            -0.015322790791698395,
+            0.014157131555941659,
+            -0.017853386731281715,
+            -0.014994640991761465,
+            0.01775991780643474,
+            -0.037498078532902435,
+            0.009570197712419421,
+            -0.007687297543935123,
+            0.009050494195378254,
+            -0.004222041469820445,
+            0.007460383234429904,
+            -0.0010804379186060264,
+            -0.014987857934522812,
+            -0.014903749199810126,
+            0.03228485430019625,
+            0.022096121825475306,
+            -0.000410816936759558,
+            0.006728733918931589,
+            -3.971636461500232e-05,
+            -0.026719848032785824,
+            -0.011253751790407687,
+            0.02245467425830765,
+            -0.005664635254398481,
+            0.005134496095445943,
+            -0.01109590855846966,
+            -0.0149616794609307,
+            0.0019726017255571673,
+            0.032668637270108894,
+            0.06066635618119076,
+            0.012001732730659747,
+            0.024422093585231484,
+            0.0189499396761026,
+            0.02119276033547528,
+            -0.008489818917377516,
+            0.006413901834269739,
+            -0.03097413287789443,
+            0.03509225573986143,
+            0.005392594635384506,
+            0.03108937318306421,
+            0.026664331248813712,
+            -0.04722525921137681,
+            -0.0058539171830594774,
+            0.017087531454837677,
+            0.022673712717885323,
+            0.0159020681868943,
+            0.0008382498574190522,
+            -0.008938174268517353,
+            0.035650940044285984,
+            -0.03315250299348084,
+            -0.023647111263155,
+            0.005059382024092534,
+            -0.01439537467317451,
+            -0.010488359956731476,
+            -0.006785295580877329,
+            0.02776449681317373,
+            -0.005584052419598926,
+            -0.004582019433008779,
+            0.009697064252638535,
+            -0.014400206558190682,
+            0.008165676294736836,
+            0.0007624778531368436,
+            0.008681263897172326,
+            0.039249866981334126,
+            0.0005918063050375113,
+            0.024275699182767528,
+            -0.004138313072843555,
+            0.011079529739650594,
+            -0.0158972266904151,
+            -0.016985058543605,
+            0.02276982363172991,
+            0.00351101120496393,
+            -0.03244126813432563,
+            -0.00768819222024939,
+            0.00928180979884261,
+            -0.0013793404539641824,
+            0.005266058566444064,
+            -0.00503433670609399,
+            0.003747448261046857,
+            0.03286230666764075,
+            0.0022383592721685153,
+            0.008606190020480513,
+            0.013593947677257625,
+            0.00030951915677929204,
+            -0.01490307680740127,
+            0.013254591138113545,
+            -2.1379308136707722e-05,
+            -0.012900073369514622,
+            0.03357519186606058,
+            0.006705902094864628,
+            0.04243347200485736,
+            -0.03302953674711988,
+            0.009031534935013593,
+            0.01822227379789576,
+            -0.008491770335368761,
+            -0.02725412856390648,
+            -0.03630788717786838,
+            0.008259145747740535,
+            -0.019003260334340084,
+            0.015781275447792516,
+            -0.010039277287105153,
+            0.029665124310477384,
+            -0.03563370794521629,
+            0.004714858110268169,
+            0.004053913355055313,
+            0.0009503673086552651,
+            0.02068360255395961,
+            0.018524093113914366,
+            -0.002785698934204024,
+            -0.031443835341186485,
+            -0.02066365433385339,
+            0.006869321907853112,
+            0.02431260052207595,
+            -0.010367048093621498,
+            -0.005612149485859478,
+            0.0006316058139530722,
+            -0.024369562712568606,
+            -0.016845124984022828,
+            0.030359419843284823,
+            -0.040974427176153246,
+            0.005252911449822132,
+            0.04272936053727608,
+            -0.04599092715016919,
+            -0.03802288716943514,
+            -0.002314988728575789,
+            -0.024124014894379466,
+            -0.006360771882090754,
+            -0.002068030981218464,
+            0.016669991113175926,
+            0.0036232707799735537,
+            -0.028763764013482795,
+            0.00943401978066451,
+            -0.005173988601101547,
+            0.003971420424839974,
+            -0.0031669592494977393,
+            -0.02216336363004636,
+            -0.01657772032717298,
+            -0.02531962163186398,
+            -0.035174102848865116,
+            -0.0015903434876311088,
+            -0.005158703150821607,
+            -0.028703075279214722,
+            -0.03941940849540492,
+            -0.021696000925262537,
+            -0.0160221488026189,
+            -0.00843213135542359,
+            0.040641882367348774,
+            0.021319060666253945,
+            0.019336844555345886,
+            0.0025252167889382917,
+            -0.020346900609125973,
+            0.049440028814233025,
+            -0.007642182460707816,
+            0.035628691886108926,
+            -0.0017882888000877158,
+            -0.013461669667950893,
+            0.00455558389906464,
+            0.03825428026120858,
+            -0.014248434654950971,
+            -0.0009902062564027718,
+            0.00027811655483232545,
+            -0.0011866253757890136,
+            0.0008535155950871286,
+            0.015433650396777696,
+            0.001307651096343185,
+            -0.007112199543314309,
+            0.030380673552683906,
+            -0.012803247194895328,
+            0.01637501752192592,
+            0.018350814869962304,
+            -0.0013359719137099667,
+            0.008064546833360553,
+            -0.021430659268086814,
+            -0.00731572118780415,
+            0.0044578576331236325,
+            0.008422210247967642,
+            -0.015096885054883586,
+            -0.009964588397149773,
+            -0.02359494467299634,
+            -0.01174145797396426,
+            0.0023855749166744366,
+            -0.004559202896782688,
+            0.004902138778592298,
+            0.034595817216889944,
+            -0.016068744500427584,
+            -0.035106033294403614,
+            0.009997207702122002,
+            -0.02663088875167486,
+            0.02222306406130687,
+            3.983384942889326e-05,
+            0.006838221769859366,
+            -0.0023434342406307616,
+            0.001765445386886839,
+            -0.00169484278667461,
+            0.002862331185439823,
+            -0.018217975204395134,
+            0.0031859734246840257,
+            -0.0032119988065673565,
+            0.0009052950782312657,
+            -0.0012459653052875638,
+            0.00687339892509406,
+            0.006332799551164718,
+            -0.0008019538898592668,
+            -0.002968451740267742,
+            -8.541086355097327e-06,
+            -0.020561816931299912,
+            -0.021642512405352372,
+            -0.028259049261410905,
+            -0.000989887278779396,
+            -0.012359733017834338,
+            -0.027804394906433664,
+            0.011443385424045703,
+            -0.006872684999639775,
+            0.008327844771213648,
+            -0.008969527138034904,
+            0.006219994886600349,
+            -0.008415106130309062,
+            -0.0004143996206881767,
+            0.013119286511794273,
+            -0.02925209282917943,
+            -0.02096907906398536,
+            -0.03096513357712367,
+            0.01762223841790055,
+            -0.00925741670612824,
+            -0.002656382260603827,
+            0.0025223527960181835,
+            0.005092659166535706,
+            -0.033606542487570154,
+            -0.006653438998497412,
+            0.007313682256676618,
+            0.009547290751154595,
+            -0.03287788897629887,
+            0.02446728061355753,
+            0.017023222076526946,
+            0.01874233066565012,
+            -0.02417566312664911,
+            0.04049827067897005,
+            0.014426879325647042,
+            -0.021740606525480655,
+            0.016712907564723732,
+            0.027385150577780248,
+            0.03177163221499527,
+            -0.03469098547555545,
+            0.004481376248622418,
+            -0.01518200063132508,
+            0.018918680020516044,
+            -0.025284921244460287,
+            0.020625170445627613,
+            -0.016707629602243063,
+            0.021278719363178687,
+            0.03520334001264928,
+            -0.005524369131116028,
+            0.03608133014180745,
+            -0.019234548780647943,
+            0.0059985801736444555,
+            0.004041778461022664,
+            -0.01645503683802124,
+            0.01024694092614158,
+            -0.0042700351372843386,
+            0.0006059159967286885,
+            0.009406090340935747,
+            0.019133229156115282,
+            0.019007219188041775,
+            -0.04442208547053466,
+            0.028588965123434923,
+            0.00804120101594811,
+            0.021472086070876265,
+            0.014177820346835521,
+            -0.02708807970982664,
+            0.004742981711172693,
+            0.002072673190279628,
+            -0.004659713164250794,
+            0.0061167904730676514,
+            -0.013796604518930328,
+            -0.034540857808309226,
+            0.006338792319814448,
+            -0.018918703778187926,
+            0.015036937364271984,
+            -0.009026670935442384,
+            0.0011667525636770179,
+            -0.01152652811400482,
+            -0.011938679695598347,
+            0.0006170592505446954,
+            -0.005495151258143897,
+            -0.0008075736908949407,
+            -0.03171721592827769,
+            -0.04427658539412448,
+            0.02036132271908089,
+            -0.0021221965958423804,
+            -0.04320396743189005,
+            0.004111179168317842,
+            -0.025352428814134187,
+            -0.01865918186335911,
+            -0.011237699157960728,
+            -0.025047923887906474,
+            -0.04219630696740606,
+            -0.011108496740166623,
+            0.007785006066683012,
+            0.016810642610918525,
+            -0.009081815651299525,
+            0.040965574143584864,
+            0.0073944364393447965,
+            0.015313576815177263,
+            -0.017731360025694955,
+            0.026137777790243826,
+            -0.019314392060123,
+            -0.03108486330987073,
+            0.02676829566273746,
+            -6.259161089416372e-05,
+            0.003053748984540415,
+            0.014358356168574803,
+            2.8186924071913628e-05,
+            -0.006181789640967465,
+            0.01947730374003864,
+            -0.028739300217145797,
+            -0.023460503178221397,
+            0.0012460072421725688,
+            -0.008850395570726385,
+            -0.016271242375881602,
+            -0.035896509136022194,
+            0.005227881053956234,
+            0.03324337105109496,
+            0.004970419987430122,
+            0.008273497228454482,
+            0.01814500594735864,
+            -0.02229593571615308,
+            -0.024230675455380678,
+            0.000639112529814716,
+            -0.010266640429960994,
+            -0.010257247303470066,
+            -0.021302483296731352,
+            0.019150757066356872,
+            0.01651375496287087,
+            -0.00390189952706486,
+            -0.006971692709239087,
+            -0.010621003777733253,
+            -0.01918206725910585,
+            -0.02020345113211036,
+            -0.013827322378214754,
+            -0.017980206764311026,
+            -0.034153741666571154,
+            -0.032195001867229935,
+            0.03225205339201753,
+            0.026354817588919596,
+            0.0014076409241492998,
+            -0.007959081015805435,
+            -0.023070419006792155,
+            -0.004483328183581872,
+            0.02879516595345542,
+            0.002549934782150634,
+            -0.0001541849716285838,
+            -0.00830090759167373,
+            0.015341095409025125,
+            0.00835695019244165,
+            0.005107844984733648,
+            -0.06676974053142512,
+            -0.006027540973808174,
+            0.01655283585235268,
+            0.01351155740684989,
+            -0.002359114145814062,
+            0.02591301169926259,
+            0.01108099264396926,
+            -0.015361038608114896,
+            -0.019765743462262668,
+            0.035195761515422364,
+            0.02408910733949848,
+            0.023468767772473746,
+            -0.0034492999385544727,
+            -0.004472311490618405,
+            0.007030833258457338,
+            -0.010684252427867153,
+            -0.013326964647936429,
+            -0.004558513842504658,
+            0.026925146859337397,
+            0.016099443021514896,
+            -0.0064331198280825575,
+            0.010848931851939124,
+            -0.0008320209125061849,
+            -0.019444491995041537,
+            -0.001841330829156697,
+            0.008730275423516445,
+            0.0012974082030866743,
+            -0.014611902493400719,
+            -0.005911807768916134,
+            -0.019130604101278263,
+            0.024719468317569562,
+            0.0209456329723249,
+            0.03381727436368338,
+            -0.013237902591972769,
+            -0.0003695142331921278,
+            0.005410360755058108,
+            0.0002841632253252816,
+            0.0023352645111704958,
+            0.004801123870325242,
+            0.02623649813035263,
+            -0.029010710171522985,
+            -0.020741774066460415,
+            0.0068426980682519505,
+            0.013445270043414486,
+            0.0022744032984779294,
+            0.033840485942696216,
+            0.035040080489222324,
+            0.004166066423508316,
+            -0.010046666684988869,
+            -0.006148765564658713,
+            0.007071740742464309,
+            0.01865897335451031,
+            -0.023267411753900076,
+            -0.0005936657905496225,
+            0.004833030519903764,
+            -0.004229582648167378,
+            0.03620610659071198,
+            -0.009961238880412731,
+            -0.007197670122096852,
+            0.022212791947347127,
+            0.09819205436926473,
+            0.00047695621444943017,
+            0.0025354521225913395,
+            -0.029358210234742176,
+            -0.023614016211567963,
+            -0.0012564198366457442,
+            0.00242680708457405,
+            0.00830167274308547,
+            0.028076260517106013,
+            0.0004852400049405906,
+            0.011579274037419117,
+            0.02773972145320728,
+            -0.017217305534161797,
+            0.006752070301141707,
+            -0.011376051474975524,
+            0.024577071515205502,
+            -0.012694034026935636,
+            -0.0019809719792138673,
+            -0.016512649783050146,
+            0.015907402258634816,
+            0.004556574687833768,
+            0.004907510649166062,
+            0.011347630984408937,
+            0.009824484201337644,
+            -0.02650731900265866,
+            0.01756773113723491,
+            -0.005671986127099093,
+            0.004449611508541289,
+            -0.027841346572384462,
+            -0.020124233970502267,
+            0.0025986391930124847,
+            -0.0014440758308989809,
+            -0.009579444248630085,
+            0.006005274872617889,
+            -0.0024984235450999127,
+            0.03765664613203687,
+            -0.03765496603611941,
+            0.01409350051861866,
+            0.0066453175608372555,
+            -0.0015390841089599882,
+            0.011806653942640485,
+            0.017306914776093522,
+            -0.008206256939290151,
+            0.02589800448884443,
+            -0.00798213351353373,
+            0.009846274212000523,
+            0.015230070803055296,
+            -0.03449531102975745,
+            0.009174739184806312,
+            0.010145762060174144,
+            -0.024165660737765746,
+            0.012146877603301336,
+            -0.013784371016298891,
+            -0.004818759788054113,
+            0.005930446700613062,
+            0.007349837554764034,
+            0.010733731974181452,
+            -0.02295571464086674,
+            0.016176424826629898,
+            -0.010947841926210105,
+            0.0037766665765835445,
+            0.013559846998199008,
+            0.024892849808979323,
+            0.0012426907454129122,
+            0.0005950515136086193,
+            -0.01379469431504576,
+            0.005534724429523084,
+            -0.015667467221535825,
+            -0.022401586867694584,
+            -0.014192041775234079,
+            -0.006366846766730035,
+            0.01029598687444803,
+            -0.003972386708834858,
+            0.016971476088808737,
+            -0.010309648640697812,
+            -0.010645860303528099,
+            0.0037818310327362103,
+            0.01040806357838676,
+            0.008316605962234849,
+            -0.02855914079113881,
+            -0.005676310556150579,
+            0.004961084138163161,
+            0.00029285971503001265,
+            0.009592497954559435,
+            -0.014556073011576487,
+            0.013689526487160009,
+            -0.013290113354287095,
+            -0.0010726537498214612,
+            0.013201860397060236,
+            -0.012443552043930505,
+            -0.004012305744087929,
+            -0.01592905426965635,
+            -0.028487066798154617,
+            0.008462143660968396,
+            -0.01365614159523485,
+            0.001076882584445294,
+            0.0019503240125505646,
+            0.004730085646042709,
+            -0.016919069570642004,
+            -0.007860046623166481,
+            -0.033515448866521046,
+            0.011065416478687504,
+            0.02314589687882036,
+            -0.006435733825418645,
+            -0.030351727858789472,
+            -0.009546333526852862,
+            0.011327489786114963,
+            -0.0071765122743172715,
+            -0.01676736383051139,
+            -0.0029390038145944096,
+            0.007617910650439855,
+            -0.008174620754097493,
+            -0.00508943241317381,
+            -0.0025173969010414098,
+            0.0217328037301181,
+            0.016094975605203832,
+            0.01769642142712967,
+            -0.021400862603034036,
+            -0.00976992221747187,
+            0.026043879119776864,
+            -0.03275550610951898,
+            0.012441465975914529,
+            0.024587446266013183,
+            0.003446527051428522,
+            0.005492228054838714,
+            0.0030377554109693714,
+            0.005459159163772825,
+            -0.006985179226831048,
+            0.019925583798328282,
+            0.010870421035240845,
+            -0.019709062313471467,
+            0.005385443078244229,
+            0.02086085539375208,
+            0.0037248737640193097,
+            0.012892299156840998,
+            -0.017546870090541213,
+            -0.0020341807891543716,
+            0.015594779287017958,
+            0.04816214672670215,
+            -0.0032851797178091685,
+            -0.010450846098750195,
+            0.029513742590677183,
+            0.023526004023637855,
+            0.005373455940309873,
+            0.0012120777808635214,
+            -0.02525630456877349,
+            -0.013075130701774785,
+            0.01962235086017781,
+            0.016944870494393874,
+            0.013484925533584295,
+            -0.0023255656480479004,
+            -0.019236081983457366,
+            0.021520156296467717,
+            -0.01701733319731161,
+            0.012422659010697477,
+            -0.023764941037007197,
+            -0.001490790833263327,
+            -0.0035419008954878724,
+            0.021998949056452454,
+            0.0021819635638621092,
+            0.008891505304514552,
+            0.024354552127458608,
+            -0.006978548952551799,
+            -0.019526615985982355,
+            -0.0051799020400150305,
+            -0.014578953088901798,
+            0.022506635912517146,
+            -0.024117708614271965,
+            -0.007008641019892289,
+            0.011379795968360019,
+            -0.012362093847309038,
+            -0.0102534051092702,
+            0.001164934138876502,
+            0.017686807394319725,
+            0.0009423993435129515,
+            -0.014784508280754078,
+            0.01944445593807992,
+            -0.0011359271135469456,
+            0.010410915597949927,
+            -0.01921419010977189,
+            0.009390638453648025,
+            -0.016682442516992777,
+            -0.007849184048209236,
+            0.010343008873053859,
+            0.0012604521199837046,
+            0.003948300703645506,
+            0.002592490121754231,
+            0.007773437307969315,
+            -0.03355658426684504,
+            0.022695505487314516,
+            -0.010751606023585792,
+            -0.003538523688786008,
+            -0.0014125330167969218,
+            -0.01678692701672328,
+            -0.0006918162525881059,
+            0.005816502852598842,
+            0.010878358093847386,
+            -0.016045147059003256,
+            -0.00873270513444376,
+            -0.04920244395446931,
+            -0.030314009649638693,
+            0.023156333024481057,
+            0.013733657599720879,
+            -0.010971506342695331,
+            0.018852159415605117,
+            -0.023760853159532587,
+            -0.010587249404421732,
+            0.0011379869499042905,
+            0.023319856329056813,
+            -0.03539290744576506,
+            0.01215369703816749,
+            -0.014493682360886549,
+            -0.02039288669534806,
+            0.009070984596830142,
+            0.011445354735283254,
+            0.0007259852045513916,
+            -0.0011514627555568814,
+            -0.029350945123925453,
+            -0.010246845821727087,
+            0.02391068177425878,
+            -0.00861718589055971,
+            -0.004723044986826333,
+            0.024716520302409263,
+            0.0033721801437248873,
+            -0.012524663490691517,
+            -0.017955758068233754,
+            0.021612671655463236,
+            -0.026077203994996058,
+            0.007645944309444457,
+            0.010032183011253845,
+            -0.0173991263874487,
+            0.021649599230588117,
+            -0.007529701797485131,
+            0.020327763440441377,
+            0.004227390938169872,
+            -0.011249233530251522,
+            0.020352283974471443,
+            0.013644443758970054,
+            0.004216958800094986,
+            -0.004081669404784771,
+            0.006869353098014179,
+            0.02212732880780841,
+            -0.011021739614266063,
+            0.02219580717345023,
+            0.028777213295471067,
+            0.00017328191188631918,
+            0.03260179554743322,
+            -0.021287655520443285,
+            -0.03159728983004563,
+            0.021740163175540124,
+            -0.01035146858119169,
+            -0.021564906188712388,
+            0.028914245422221707,
+            -0.013674731836508768,
+            -0.00786476052825481,
+            -0.0166718429993922,
+            0.00533027311342322,
+            0.016680336311088843,
+            -0.005246190451159961,
+            -0.014360934066628332,
+            0.0029628103834216345,
+            0.019809487418628386,
+            0.014587234544007958,
+            0.029780009648869975,
+            -0.010778199815920887,
+            0.035180242185168895,
+            -0.002320576872571577,
+            0.008990157317473909,
+            0.012415465232073404,
+            0.006489435152972723,
+            -0.011251531176923445,
+            0.013501894212117511,
+            -0.003446759011266526,
+            0.008230548711588447,
+            -0.001449023474280409,
+            -0.035111679577202855,
+            -0.013709382199848678,
+            -0.04709294504969336,
+            -0.0023752976043235697,
+            -0.0025098360125255215,
+            0.0030975417740913154,
+            0.09736815869389318,
+            -0.005723049647039619,
+            -0.011988817571623457,
+            0.001625059666070763,
+            0.009611697638805556,
+            -0.0009896494420056814,
+            -0.007606836952841873,
+            0.015973152482488318,
+            -0.01343292635775299,
+            -0.02140928016038583,
+            -0.010532701190806147,
+            -0.01288594454368977,
+            0.012079198190522715,
+            0.015929716343161652,
+            -0.0005947359502228821,
+            -0.015207331663761012,
+            0.006524010655124294,
+            -0.004566385899752684,
+            0.0064717704997400245,
+            0.008647099975176308,
+            -0.010414668624343924,
+            -0.0071698879102925555,
+            0.030899143952119196,
+            -0.025301269549419674,
+            -0.024361748354052457,
+            -0.0010579931222861095,
+            -0.009055796737486008,
+            -0.010335459737012047,
+            -0.002692653633719985,
+            -0.0017679501322462056,
+            -0.010519994985692081,
+            -0.010019865263564572,
+            0.006537076627706128,
+            -0.015472130358273033,
+            0.004812857776601667,
+            -0.0047457011269495634,
+            -0.013296069496526007,
+            0.017054968086538724,
+            0.022563785357424853,
+            -0.02841491200807702,
+            0.0006755428946168761,
+            -0.017422510783648978,
+            0.012941211381006863,
+            0.008866069010589792,
+            0.006049618782357251,
+            -0.022451528209097505,
+            -0.023622906582403243,
+            -0.015019272754405791,
+            -0.01927535750568903,
+            -0.024973490129133732,
+            -0.009073381957684092,
+            0.0067692803507600945,
+            -0.001267339788153158,
+            0.021918281111033645,
+            0.000816538714279555,
+            -0.01812545613159305,
+            0.0061084913570894905,
+            0.000568371220952269,
+            0.018292497018668842,
+            -0.027468677288882186,
+            0.0033806422195878524,
+            -0.02307582511848465,
+            0.033508234091571124,
+            0.0039195050746594975,
+            -0.026680111514794824,
+            -0.021558668725755275,
+            -0.003311830287529699,
+            0.018994577319066473,
+            -0.0025745912019675805,
+            -0.00724049447539589,
+            0.014775207617625293,
+            0.0155334760323838,
+            -0.004693118176196483,
+            0.002015625406296403,
+            0.027968079002437102,
+            0.01077998866180723,
+            0.0313892866933904,
+            -0.015236169297590675,
+            0.004945828495617267,
+            -0.027215016713379062,
+            0.010214419950212732,
+            -0.01503548137805401,
+            0.008859554812646242,
+            -0.01290890682557532,
+            0.03531384210182567,
+            -0.012005613861257432,
+            -0.004250698950192301,
+            0.03253288796441852,
+            -0.029150986393345252,
+            -0.0035020051536906334,
+            -0.009255437891304041,
+            -0.004142204892160768,
+            0.013240580426083011,
+            -0.001198093874819309,
+            0.008485141356876356,
+            -0.01854370462655459,
+            9.726096864073609e-05,
+            -0.003221852304410318,
+            0.005404595442127598,
+            -0.02005458688172967,
+            -0.003314118809778734,
+            0.0073427568419969814,
+            0.004259518237708634,
+            0.002408630356035373,
+            0.016420646142725346,
+            0.006662121128241274,
+            -0.0029357701493526907,
+            0.03424224243607692,
+            0.008171895850772245,
+            0.0152521424877615,
+            -0.0018638264193202728,
+            0.0030234211261443034,
+            0.019287963516760636,
+            0.019912956661192856,
+            -0.00048443399729935116,
+            -0.01058935153883094,
+            0.00195113175754497,
+            -0.001140732137433033,
+            0.011048741602327437,
+            0.024396046054912645,
+            -0.021158025221644738,
+            -0.02419562092713724,
+            0.016377661005279716,
+            -0.015076145951740961,
+            0.01903712233378994,
+            0.010541463112889985,
+            -0.010099351302095792,
+            -0.00031558616108876247,
+            -0.042316016201379245,
+            0.010964989031040692,
+            -0.02068939391415103,
+            -0.04004120028915868,
+            -0.00244383573783919,
+            0.0009542507389088937,
+            -0.029439456243505052,
+            0.02822814855547978,
+            0.0023755032182188156,
+            -0.017280221740813364,
+            0.018223709137876153,
+            -0.005548654680726505,
+            -0.0024040301239229064,
+            0.005397711787238679,
+            -0.01092026090471121,
+            -0.010648485086769927,
+            0.008235250252660972,
+            0.010957715075632535,
+            0.005425683656535705,
+            -0.005698919020608095,
+            -0.011284911899894062,
+            0.009975882651638084,
+            0.012318901214538077,
+            -0.0019922884373083155,
+            0.010050520101815593,
+            0.0031720752428247446,
+            0.007352859750323051,
+            -0.0073483049386141005,
+            0.0046483587496626965,
+            -0.005093220128411125,
+            -0.026080057085901446,
+            0.012863580653903103,
+            0.0052434400313520774,
+            -0.011863360599637747,
+            0.0103209502958491,
+            -0.01048162756158923,
+            0.005787157766911021,
+            0.001318945460232246,
+            -0.0059333230422061395,
+            0.004821320238893602,
+            -0.0038922239931512784,
+            0.011267634571983924,
+            0.013256611037442042,
+            0.004592529900805719,
+            -0.013076954459595086,
+            -0.02177318635989856,
+            0.009001291224623467,
+            0.01959767163356408,
+            0.024161754192119852,
+            -0.030416109293488064,
+            0.003784634588325551,
+            -0.0014915849060829743,
+            -0.0020550564975080885,
+            -0.015632785947151753,
+            0.01587790981601866,
+            0.01716742056401084,
+            0.012721563202528083,
+            0.01893393006967701,
+            -0.004009617703771589,
+            0.010539100665843704,
+            -0.02080068888791658,
+            0.013642451766699067,
+            0.0008609869904066794,
+            -0.013629067231663643,
+            0.0014620655123815298,
+            0.027046304278766087,
+            -0.011056783825898616,
+            0.006206969699250696,
+            0.008097527175511791,
+            0.022111870736907183,
+            -0.012509479100313492,
+            -0.003735868296238069,
+            0.01516254968167132,
+            0.0014878674796313539,
+            0.0035887079417890352,
+            0.016533149527222273,
+            0.04729052276110659,
+            0.00102751186013391,
+            0.0016713773449738153,
+            0.0192177920520243,
+            -0.005563519063161466,
+            -0.01114821419109643,
+            -0.003549999863325982,
+            -0.021807893906469106,
+            0.010998558936481785,
+            0.00431279785115251,
+            -0.02186324212032519,
+            0.01024093366826206,
+            0.0125495473550584,
+            0.005268320135120931,
+            -0.0031679212410026603,
+            0.02574575247827441,
+            -0.019807879421996258,
+            0.014935295894965128,
+            0.03329450628384787,
+            0.00493356778223407,
+            -0.011423949333937285,
+            0.007263264249871384,
+            0.0027354459678742013,
+            -0.004494406041906684,
+            -0.00606745413660583,
+            0.010497391290734088,
+            -0.008191702034391382,
+            0.009870458191510176,
+            -0.009580130479693,
+            0.039202091799313755,
+            -0.009619018331262844,
+            -0.010782138248435458,
+            -0.004558645539551881,
+            -0.006241773108460662,
+            0.003478414182938609,
+            -0.003395165694642073,
+            0.005812225135633631,
+            -0.0069045407353789945,
+            -0.014607146220271209,
+            0.0008522982757210454,
+            0.00020175973815948303,
+            0.014001922919116725,
+            0.0011804904415274088,
+            -0.004871874532755353,
+            -0.007877217154706205,
+            0.015243277056007988,
+            -0.006047906208304912,
+            0.0006531937033317192,
+            -0.0011653523616313422,
+            -0.0013662556399359335,
+            -0.007401050012179761,
+            -0.015166105795085528,
+            0.00974609244984173,
+            0.016253733908242134,
+            -0.013857834499211056,
+            -0.005142936208363415,
+            0.012274716212832219,
+            0.008540229125193836,
+            0.011742606168187876,
+            0.0007451338650642946,
+            -0.009255813682572937,
+            0.008292531177374192,
+            -0.015020523035432158,
+            -0.008781317965985481,
+            0.0026250469418136603,
+            0.002381147382977664,
+            0.005839723981046825,
+            0.0006240641828668715,
+            0.004502463508605272,
+            0.005798912705218846,
+            -0.00497936644919367,
+            0.01112666823857836,
+            0.008198424171006548,
+            -0.002452143482105151,
+            -0.01017569410837814,
+            -0.001127589984272638,
+            -0.015733298167661346,
+            -0.016758315411028353,
+            -0.0009286266498936771,
+            0.015175988113750882,
+            -0.02352862898718829,
+            0.018996352975391644,
+            -0.0014524363611708201,
+            -0.0033802546171857704,
+            -0.021392428456030312,
+            -0.0025923983911570183,
+            0.0034214188668518796,
+            0.01255768230394476,
+            0.01858853630745928,
+            -0.02729543519465384,
+            0.010575894388563116,
+            -0.007204725245442141,
+            0.030394929424014717,
+            -0.018328782140005044,
+            -0.004022941273610227,
+            0.026915618079621992,
+            -0.02200018306828758,
+            0.0008791185377513978,
+            -0.011454606955158956,
+            -0.008103032693853504,
+            0.019287674837688177,
+            0.023747720480736777,
+            0.02038287242167783,
+            0.01364489294869211,
+            0.0007758044919789085,
+            0.004768030222445397,
+            0.03572705648994403,
+            0.010568241270862248,
+            0.014828087092859878,
+            -0.007542489523110974,
+            9.457808640264811e-05,
+            -0.02554033312739386,
+            0.01635899434369978,
+            2.587383534566272e-05,
+            -0.012370727506722627,
+            0.007162422186621379,
+            -0.013370882403931942,
+            -0.011066483600541058,
+            -0.007151077450730025,
+            -0.01721910018182534,
+            0.02152239849400241,
+            -0.012322440280680164,
+            0.004587729883596212,
+            -0.009079747696669277,
+            -0.013272707286450417,
+            -0.012953682633400515,
+            -0.014202214807187545,
+            -0.01313141988034687,
+            -0.026041403779538307,
+            -0.027109731260915763,
+            -0.006573285295895077,
+            -0.023780366346334524,
+            -0.013799072648505526,
+            0.005588622594136253,
+            -0.022617902666316757,
+            0.012262325666428604,
+            -0.012689827812482627,
+            -0.01226264872059864,
+            0.012805734807835008,
+            0.020274844315616616,
+            -0.011765231803328348,
+            -0.028494489703400308,
+            -0.018583843821043854,
+            -0.019970998470704358,
+            0.012576201137505969,
+            0.002266288538944273,
+            0.013165889075444961,
+            -0.003018437854660793,
+            0.003227412667880239,
+            -0.014895831590298957,
+            0.018570624675927183,
+            -0.03442390703780963,
+            -0.03108003806963536,
+            -0.013611306765348361,
+            0.005182777066476028,
+            -0.0015782033203578553,
+            -0.011429046504256945,
+            -0.0018606648572920801,
+            -0.00433805271182878,
+            -0.025109572024446088,
+            -0.010134856632190067,
+            0.011010662054484161,
+            -0.015205629927469414,
+            0.0072933565255675036,
+            0.0009018832564038863,
+            0.013870902834260887,
+            0.009587432944616442,
+            0.013393301893492882,
+            -0.006095063698384775,
+            0.004356485794113208,
+            -0.0026476612747324386,
+            0.0004894024149759108,
+            0.009336366919474478,
+            0.00839591561027103,
+            0.010551161369079399,
+            0.002260707777260338,
+            0.007830059223001696,
+            -0.0060927446216633265,
+            0.018065002687169148,
+            0.015436944875538481,
+            -0.0002520689442496089,
+            0.013323098485251487,
+            0.0003769862591044255,
+            0.00022293304921127018,
+            -0.0029970303106340466,
+            0.007888437018607987,
+            -0.011785957933642163,
+            -0.001311704406355858,
+            0.015138190893483804,
+            -0.01707247730569978,
+            -0.013045187198628258,
+            0.03107034781283638,
+            0.0018583638488268697,
+            -0.02495564203304796,
+            0.031146554085787762,
+            0.004220047610942943,
+            -0.005835199594223787,
+            -0.013205086624848782,
+            0.0014059047822222418,
+            -0.011570624844100298,
+            0.003584232809362274,
+            0.014295032061446826,
+            -0.02231984223610487,
+            0.010521094801839404,
+            0.008446892441796151,
+            -0.009681535919980559,
+            -0.012337659795586223,
+            0.009847073709209602,
+            0.016809328445200358,
+            0.004472485504034416,
+            -0.03539093810210523,
+            -0.011957694102753411,
+            0.02145470732008716,
+            -0.029237428225009913,
+            -0.00810041956259673,
+            0.007452962400105787,
+            0.0065154832040426065,
+            -0.019353109017609053,
+            -0.008580955020412362,
+            0.005725794605897509,
+            -0.004940390374637084,
+            -0.003892210172017758,
+            0.017541623128534682,
+            0.00485047043626186,
+            0.013698341792101025,
+            -0.01689738578307474,
+            0.009771458057400557,
+            -0.022222817828602557,
+            0.004021882074676753,
+            -0.023373700500461992,
+            -0.009873403167702603,
+            0.002364143609079849,
+            0.012454271200139762,
+            0.014328556467632212,
+            0.027839363143659326,
+            0.030056404238348527,
+            0.013995696971422429,
+            0.007833677585250953,
+            -0.016265199386953717,
+            -0.018134327237616175,
+            -0.006409513840075933,
+            -0.004033173350204784,
+            -0.008714798059218596,
+            0.0066373547199279235,
+            0.01858354212009877,
+            0.004499712802472377,
+            0.02609078206670711,
+            -0.006227189596272876,
+            0.009039744622014585,
+            -3.127526911951914e-05,
+            0.0008833930131827993,
+            -0.004226083827636182,
+            -0.01971685546584527,
+            0.00481003211084742,
+            -0.0013721482162825508,
+            0.025874758880591167,
+            0.00940496860916206,
+            0.0024915013399541847,
+            0.01615978314811047,
+            -0.006378952757397484,
+            -0.01174653607178726,
+            -0.010540107733238588,
+            -0.0061926268398085176,
+            -0.021752582127981104,
+            0.029059100503124387,
+            0.021195595568253612,
+            -0.010800765264123782,
+            -0.013693582166598082,
+            -0.00903369526059487,
+            0.02795767990556847,
+            0.004080965688836953,
+            -0.0013050212744006933,
+            0.016313311850308505,
+            0.012084555958999215,
+            0.012329325712491252,
+            0.009794698207689061,
+            -0.00036157148861933123,
+            0.0015364709122308352,
+            0.03722522141440073,
+            -0.007427028510548186,
+            0.0026463292682538495,
+            -0.005239324411651515,
+            0.0025987932868461313,
+            -0.009705817520422487,
+            0.007242122655628283,
+            -0.004012676385296197,
+            -0.0158760051470016,
+            0.015003444174308607,
+            -0.022443331305169994,
+            -0.0016749787204471238,
+            -0.008150635190952799,
+            0.0016696640640947303,
+            -0.02781023337840043,
+            -0.0025731206607371845,
+            0.006695065701630532,
+            -0.015547340828489069,
+            0.011816010501093885,
+            -0.01740673229051557,
+            -0.011553604323149105,
+            0.007914961618894248,
+            -0.012034157288065447,
+            0.0016684130271324446,
+            -0.0008268957621923278,
+            0.004460864332388434,
+            0.013426055079012884,
+            -0.011008808570221245,
+            0.012166046631826273,
+            -0.005816284310138776,
+            -0.007872091214834688,
+            -0.0016055129487028885,
+            -0.0030049472973942043,
+            -0.0025600746761171543,
+            -0.0009966231170649917,
+            0.034355910234323155,
+            -0.004446570896048543,
+            0.003761242147499512,
+            -0.01362639743248326,
+            0.0019833052855181806,
+            -0.036176376394999044,
+            0.028344464817309477,
+            -0.008642835889481802,
+            0.011013502221008498,
+            -0.0017508365480865596,
+            -0.013243706268954915,
+            -0.01456517309747086,
+            0.015247299501245178,
+            0.007602596572394442,
+            -0.005854531057349665,
+            0.013558248087347848,
+            -0.019724373614448298,
+            0.008530500822910953,
+            0.004463694543554244,
+            -0.015561397976987216,
+            0.004668780190328835,
+            -0.007479912658259757,
+            0.012657201279639194,
+            -0.01774730033362932,
+            0.004314594593150765,
+            0.00958571617187387,
+            -0.01892876084898881,
+            0.009901041800918215,
+            -0.028675879564512306,
+            -0.004631920242208613,
+            -0.012323455524366223,
+            0.0053804608827285865,
+            -0.000660864620708182,
+            -0.02389088252276507,
+            -0.028944280660230762,
+            -0.0002099667627277945,
+            0.00458533068247518,
+            0.0018878351997997833,
+            0.011621407953503755,
+            0.005970807015151215,
+            0.008542958629199003,
+            0.01023896054756221,
+            -0.00205207247588841,
+            0.009402752124876478,
+            0.04169706217414899,
+            -0.022899912321317197,
+            -0.011574970753549647,
+            0.005321872977369877,
+            0.011499271505460519,
+            -0.02647653483798284,
+            -0.011039791758347888,
+            0.0033424345263339144,
+            -0.005219877979550397,
+            0.03609469691834737,
+            -0.02448746405881972,
+            0.019020618652020673,
+            -0.011392297116547224,
+            0.007625475937968714,
+            -0.021465240076078376,
+            -0.003921088264697311,
+            -0.014265745991522477,
+            -0.00011221615798127438,
+            -0.0031478236827989703,
+            -0.0076053733863387755,
+            0.007485039021296698,
+            -0.001907678661303243,
+            0.0020348246022161925,
+            0.006291436821831794,
+            -0.02173305455485444,
+            -0.0019125499969493854,
+            -0.016432460924913166,
+            0.030367263349759286,
+            -0.016242095671439496,
+            0.005072805486919414,
+            0.0053258139728101506,
+            -0.002810639419737258,
+            0.012368771814660942,
+            0.004165967191536186,
+            0.00513754440091394,
+            -0.004199767241827414,
+            -0.005142642360546972,
+            -0.0053886448467479215,
+            0.017868874549760376,
+            -0.013280048598849825,
+            0.035024701466710065,
+            -0.0100959186045929,
+            -0.007094001877261568,
+            0.0019733042325232155,
+            0.010380430710282744,
+            4.2084202709296443e-05,
+            0.01053565670076933,
+            0.0034270489770133685,
+            -0.0006311078385359693,
+            0.0019774779328212756,
+            -0.017162448473652843,
+            0.003089758682621882,
+            0.01682656561693936,
+            0.005463677441664502,
+            0.004469261499451007,
+            -0.013332130891003062,
+            -0.012050233723282738,
+            0.004538268704442613,
+            -0.007533940408255447,
+            0.0003675737140813486,
+            -0.018816953644773076,
+            0.013972696748575243,
+            -0.0011096683686631858,
+            0.0002715683052236706,
+            0.00524678369365015,
+            0.023056514997908218,
+            0.01813167860994747,
+            -0.002553352602551003,
+            0.03155164978556312,
+            0.0174738879054593,
+            0.0029268039949260673,
+            0.010637295711777793,
+            -0.033222781195984666,
+            -0.013301873512972214,
+            -0.0013721703891642354,
+            0.02238421933149517,
+            -0.01530189703335162,
+            -0.014059029663525449,
+            -0.0037393534403749947,
+            0.0010394601933309767,
+            -0.005177359933828505,
+            0.0042489306011644495,
+            0.011039889511148341,
+            -0.014697595240068863,
+            0.007597149702441915,
+            0.0026780496822907634,
+            0.02092734730183164,
+            0.0004688144432243786,
+            0.017181410417231698,
+            -0.0005083970646221274,
+            0.00749639055342427,
+            0.007319420826337891,
+            0.017541463125355265,
+            -0.01002811389296805,
+            0.023582113845081534,
+            -0.008823933397729605,
+            0.0019889258902730535,
+            -0.01257106158422629,
+            -0.028159594067544737,
+            -0.012624750220149823,
+            0.015247564545279736,
+            0.005765911899842925,
+            0.006478299226263732,
+            -0.016537090297144265,
+            0.006276997440048981,
+            0.015647122171767974,
+            -0.002251683151059454,
+            0.010295358869874306,
+            0.009730378299068289,
+            0.00012816405890877098,
+            0.010181120688239766,
+            -0.00801256937783534,
+            -0.00734257767723293,
+            -0.004497520124843868,
+            0.023982754195729983,
+            -0.005443550092842393,
+            0.022434177122972028,
+            0.012647958478747348,
+            0.00616791857573897,
+            -0.010853355357954727,
+            -0.003340111254357573,
+            0.023389634440057473,
+            0.011863246790550376,
+            0.00029642699874398923,
+            -0.009846034060495678,
+            0.0019458876579524617,
+            0.0054010373765486005,
+            0.012983913038716568,
+            -0.02049427270240938,
+            -0.004344571352918878,
+            0.008461947538684664,
+            -0.008472919299762712,
+            0.004318309005722874,
+            -0.030156444522730416,
+            -0.005812303237222271,
+            0.021354352236942434,
+            0.008392281015122372,
+            -0.010232039722494646,
+            -0.004395682831498155,
+            -0.0006527587802367882,
+            0.0040601850269714945,
+            -0.0024323167053212176,
+            -0.0023729138545912176,
+            -0.008541233856441836,
+            -0.006470933419694742,
+            0.014035696617797647,
+            -0.0052691760528871924,
+            0.008597459268477037,
+            0.00012084859681905111,
+            -0.0017255958688825231,
+            -0.007323529232751411,
+            0.03382792341957058,
+            0.008087732655845718,
+            0.005980666974986827,
+            0.0013613625176327891,
+            0.0042244330945601215,
+            0.000616791811672998,
+            -0.0028374856646453707,
+            0.018075360591135542,
+            0.0156738939725117,
+            0.005054696911814988,
+            0.009038525980341447,
+            0.014672270369917403,
+            -0.019700405219230957,
+            0.012625337070556061,
+            0.012184185186281482,
+            0.019382321278525114,
+            0.0007219498737851579,
+            0.002564970959687783,
+            -0.010601873823737926,
+            -0.0029963537053764736,
+            -0.002070567089566093,
+            0.029563028293936092,
+            0.0005903976560069006,
+            -0.010509758568838935,
+            0.010992421945232951,
+            0.020603868018231348,
+            -0.02101157589451638,
+            -0.001971974816402967,
+            0.005963911670920596,
+            -0.010704207628592966,
+            -0.014466113540150018,
+            -0.021269176952661614,
+            -0.013331247081394942,
+            0.0071597345125287174,
+            -0.008162102923896894,
+            0.02367549407145414,
+            -0.014256771777502395,
+            -0.03960796112042958,
+            -0.014803626428839152,
+            -0.00864320347698291,
+            -0.027531337207277483,
+            -0.007305441146142331,
+            0.00965056093889974,
+            0.016698589600806273,
+            -0.003330867073501157,
+            -0.025198703196464594,
+            0.030483790406960656,
+            0.008872821868559751,
+            0.023708422767352917,
+            -0.021464374508407488,
+            -0.0018617153194555344,
+            -0.00886376784514331,
+            0.014505918224188001,
+            -0.017588761818663975,
+            -0.011756112651591795,
+            0.020014632992762026,
+            0.014356419813769714,
+            0.007246557500044434,
+            -0.004355132555323843,
+            0.0067299398462013936,
+            0.007649499235136303,
+            -0.012515181210363267,
+            0.012239792251154642,
+            -0.013499277313916029,
+            -0.024858295126228433,
+            0.014430561986516884,
+            0.002229494935179828,
+            0.009563004428064055,
+            -0.0015984171085659323,
+            -0.016197659608768088,
+            0.010078159490026602,
+            0.00808729175515505,
+            -0.017508840440512225,
+            -0.004793934415922065,
+            0.007133527615305415,
+            -0.020818869260580315,
+            0.031234686259160864,
+            0.011872765152279094,
+            -0.010692949770503467,
+            -0.001989039629661815,
+            0.004431079028880923,
+            0.016511544608975148,
+            0.007893700385963683,
+            -0.01997314388229276,
+            -0.003708006454566117,
+            0.01530128306203401,
+            0.00822588305477249,
+            0.004861703074728471,
+            -0.010836328783594584,
+            0.03434874763691911,
+            0.006082873924472328,
+            4.192481378506604e-05,
+            -0.0052997095110967115,
+            -0.0076108924981838355,
+            0.004196112457289409,
+            0.018905391806961686,
+            0.013330078288599763,
+            -0.004200590783114919,
+            -0.006630107527854746,
+            0.007778448128358027,
+            -0.01706991160178345,
+            0.007329056462125937,
+            0.009390327499494826,
+            -0.012870674959022891,
+            -0.0017571189976160295,
+            0.016462608870101713,
+            0.00209846661139665,
+            0.004942714850185376,
+            -0.011305050739130339,
+            0.004252776146827917,
+            0.015841821944109317,
+            0.018727567181082646,
+            -0.008399642221255733,
+            0.021448055799259867,
+            -0.020972459569228434,
+            0.027243639465535856,
+            0.028499400850980086,
+            0.016573144403275928,
+            -0.01847096443459798,
+            -0.0022409373140116146,
+            -0.0041737366139895414,
+            -0.014761716560267328,
+            0.0034546020161337894,
+            0.005125870074458691,
+            -0.01174186608185822,
+            -0.009577921232380895,
+            -0.024331993794416026,
+            0.011075959943215117,
+            0.007918147902741738,
+            -0.0015045185909419564,
+            -0.011791155919533844,
+            0.016019077577552835,
+            0.010141868950981925,
+            -0.010938402325619172,
+            -0.005635351352447256,
+            -0.019967361901734836,
+            0.009750427567651833,
+            0.009128294017175602,
+            -0.00300556760891084,
+            -0.011496103417643132,
+            0.012977890524465716,
+            0.0014080059843484115,
+            -0.01292380658817521,
+            0.010841954636959441,
+            -0.03142011651894961,
+            -0.005510537354374977,
+            -0.011027026734432255,
+            -0.003620482919145167,
+            -0.014415606143566519,
+            0.003608572766363589,
+            0.0005034611321129689,
+            -0.011470488351158329,
+            -0.0035005774696278,
+            -0.016207453351921627,
+            0.004676327282262429,
+            0.015021513761697039,
+            0.001679614645967629,
+            -0.006917375094634678,
+            0.021994274901326064,
+            0.018518524483306403,
+            -0.021903538417919984,
+            -0.020116013379669034,
+            0.004708084983787958,
+            -0.03261690161800251,
+            0.015019348187107588,
+            0.00966124471577622,
+            0.011676937880751252,
+            -0.028768229226408037,
+            0.007114534979415504,
+            0.00900420165933673,
+            0.016398927481511824,
+            0.0050879301160439245,
+            0.009453255486450082,
+            0.01222758288108498,
+            -0.0104657614414275,
+            -0.003164502778525433,
+            -5.410382753651127e-06,
+            -0.005618225986727351,
+            0.015607620507415484,
+            0.0008527697935056332,
+            0.03603666488542239,
+            -0.005277207843139309,
+            -0.0009799879252482288,
+            -0.011735731235560851,
+            0.00812046801749312,
+            -0.011713349784345638,
+            0.006334172347602515,
+            -0.0024643499727789167,
+            0.0001240009282616789,
+            0.002965067818077541,
+            0.009442572075213767,
+            -0.01086095783353607,
+            -0.007595379345224076,
+            0.007444897256325813,
+            -0.01482814340478338,
+            -0.00019984717819268013,
+            -0.009339343730980451,
+            0.0019680804106275156,
+            -0.02241031895876346,
+            0.006629838013824249,
+            0.040935644507527846,
+            0.02140216465404127,
+            -0.005481040712269146,
+            -0.012431659209559418,
+            -0.004680230194268222,
+            -0.007229300034633229,
+            -0.006482401275991039,
+            -0.004794698160419071,
+            -0.022762405726589375,
+            -0.006928553505112732,
+            0.004320321244630314,
+            0.013300138724212552,
+            0.009741018117644266,
+            -0.008842135469729511,
+            0.004531760574809748,
+            -0.02662650543210291,
+            -0.016578081411548973,
+            -0.0060021425889846495,
+            -0.015840130031005147,
+            0.006125301421471048,
+            0.027521841635711787,
+            -0.0063432924325983525,
+            0.01789356321527744,
+            0.005142029893259461,
+            0.013034833065163448,
+            -0.005338869351330258,
+            0.00035250243782976347,
+            -0.006010938630154189,
+            0.0032958656042048292,
+            0.003074861351445954,
+            -0.0015947217852680714,
+            0.004151590621941346,
+            -0.007139285822575166,
+            -0.0026091937868967284,
+            0.0011629380311555433,
+            0.0117841884590861,
+            0.006877419488576261,
+            -0.00016252653621991807,
+            -0.01761285866765538,
+            0.022221104524808997,
+            -0.018933508559881868,
+            -0.008840161102111973,
+            0.011092684029552064,
+            -0.009660487856185668,
+            0.00702597023411946,
+            0.020387386977378873,
+            -0.006854213562767843,
+            -0.009583981919303215,
+            0.013028181087649146,
+            -0.0036507672754997286,
+            0.004464368543493108,
+            0.009867428978560914,
+            0.006036303973414791,
+            0.002482944923100156,
+            0.013244650094892797,
+            0.004530294330688855,
+            -0.0032625482538717137,
+            -0.020238533752799943,
+            -0.005186498720507101,
+            0.019015752262933585,
+            -0.02252458872825473,
+            -0.0027641215988942563,
+            0.0044506144202026456,
+            -0.0030317840657635457,
+            -0.01733962766050954,
+            0.004205861489436576,
+            -0.018770275720867473,
+            -0.0033883519277721433,
+            -0.008244668812709337,
+            0.009948090119054079,
+            -0.023700732627507225,
+            0.009109315916173436,
+            -0.011739192106106307,
+            0.02356509175177699,
+            -0.003925851077921686,
+            0.024734433836185024,
+            0.0015142645677087268,
+            -0.013482933030628261,
+            0.0055323884713736554,
+            -0.0008930661843169134,
+            -0.004477731568313757,
+            0.029338490404300612,
+            -0.005418258701133154,
+            0.002249996261871466,
+            0.013491124734151694,
+            -0.021332541202417246,
+            0.021773687217263173,
+            0.0046662696076723836,
+            0.01628697326539432,
+            -0.00986238951830334,
+            -0.013799209313714441,
+            0.025178874553620743,
+            0.03324555035648896,
+            0.008398796308070151,
+            0.011261509865513953,
+            -0.01226890254177634,
+            0.018966175575083295,
+            -0.0038328997984309597,
+            0.004344837325292706,
+            0.00878668880458693,
+            -0.007200842513742962,
+            -0.011500562922303248,
+            -0.004533014670236755,
+            0.027639894892383943,
+            -0.021838848944317424,
+            -0.008920629621611768,
+            -0.005739181836252042,
+            -0.01593272996094732,
+            -0.038167534267554606,
+            -0.004371701247332816,
+            0.019604405563120094,
+            -0.005974358141737469,
+            -0.0018341151594086288,
+            0.014334266690716505,
+            0.00402620072221496,
+            -0.002852569009185304,
+            -0.008195218707838913,
+            -0.0027944153624309856,
+            -0.0059390492900516225,
+            0.016651206007022145,
+            -0.004961864528562493,
+            -0.007929830585906699,
+            0.01145431100211762,
+            -0.005731957927601062,
+            0.012130694936714224,
+            0.0004848418056781251,
+            8.392048424584131e-05,
+            -0.0015174120840537315,
+            0.00822932723013239,
+            0.018620869354789082,
+            -0.0055803454666417465,
+            0.025682778991091174,
+            0.006956936033581638,
+            -0.003495709501259274,
+            -0.005489431577052735,
+            -0.0044154293119170445,
+            -0.024561016942073285,
+            -0.0020864636079347357,
+            0.006593060072668424,
+            0.024333813203596122,
+            -0.02515989187381735,
+            -0.004953266000463339,
+            0.020957806164031402,
+            -0.0012192724005613,
+            -0.011978722479596745,
+            0.014690559986694138,
+            0.0041346025614047875,
+            0.009881584250499358,
+            0.012176393203318002,
+            0.015764530703333238,
+            -0.004923548157246672,
+            -0.0015192775000207833,
+            0.012766482928380375,
+            0.0005745973040874939,
+            0.00997809510696241,
+            0.016495444272181515,
+            0.010467514633877887,
+            -0.021108281220544853,
+            0.0015078843789125156,
+            0.00522593579521476,
+            -0.017481430635971396,
+            0.0034362821987740755,
+            0.005397819166037461,
+            0.011626527721895694,
+            0.004048632382026344,
+            0.013207335578216437,
+            -0.015768003434275563,
+            -0.005916168437120099,
+            0.008153798523776654,
+            0.004319451296835019,
+            -0.02599474886669325,
+            -0.01485009085106089,
+            -0.0006781964116146299,
+            -0.004444144921332929,
+            0.013556265879295017,
+            0.01285768162554067,
+            0.02033626001285354,
+            0.024637349990579514,
+            -0.013833060730799217,
+            -0.00842819657792762,
+            0.009436072455719896,
+            0.015560357465711765,
+            0.005798181411384387,
+            0.007874051721372273,
+            0.009297440071118126,
+            0.0077320823375042,
+            -0.004073209576661068,
+            -0.002973928843567532,
+            0.020205003097225747,
+            -0.019725382393637726,
+            0.0006746659838642147,
+            -0.018417443693472958,
+            0.007530796896254817,
+            0.01140696723506382,
+            0.01129515042745398,
+            0.002816883643872417,
+            -0.005274757308627299,
+            0.005716917047812153,
+            -0.06254973656832136,
+            -0.009191453047502128,
+            0.003384751551032943,
+            0.027256466708746898,
+            -0.010885336140129142,
+            -0.003415707910562931,
+            0.007309577090125959,
+            0.005511970708769456,
+            0.0005572239548625122,
+            0.003186427278034852,
+            0.019698716902282903,
+            -0.02052581021967601,
+            -0.010793968505805883,
+            0.004219269922138548,
+            0.0033111837923746746,
+            -0.007269521534951044,
+            0.004805737092302134,
+            0.014143978208501908,
+            0.01986656063456352,
+            -0.0024048854610842748,
+            0.01629335842915328,
+            -0.0015498603704589015,
+            -0.014496994444493684,
+            -0.002671064325909516,
+            0.0037875833047555925,
+            -0.022466433279066646,
+            -0.005980897597195858,
+            0.0043262697442044435,
+            -0.011226121648007667,
+            0.006978495397818936,
+            0.008452177557316863,
+            -0.003018545465557218,
+            -0.0008303307644324117,
+            0.017796789945838845,
+            0.006591364879905198,
+            -0.013464486184025026,
+            0.0028668426952326957,
+            -0.023377605222213174,
+            0.004325185794796361,
+            0.001416245546006914,
+            0.007809449561523174,
+            -0.008431335139468173,
+            -0.014774499666285603,
+            -0.005933810712528226,
+            0.017746847020610673,
+            -0.01965188730040385,
+            -0.002406875192894708,
+            0.01683009662984443,
+            -0.0002518220110313279,
+            0.005456812494894182,
+            -0.02011241622618092,
+            -0.0009782974741561381,
+            -0.02933296756078644,
+            -0.01171823509606186,
+            0.001162345317872531,
+            0.00892221099991475,
+            -0.006551308022163148,
+            -0.03934356275100932,
+            0.00427623835776202,
+            -0.013111664387776261,
+            -0.008752857644809231,
+            -0.010214546496134218,
+            -0.00518087722852466,
+            0.015240297918667867,
+            -0.012498100033235827,
+            0.01086490299272505,
+            -0.00917777212602538,
+            0.005895055740265625,
+            0.0090554651297836,
+            0.014212841798768113,
+            -0.005544047915369086,
+            -0.0033589999440643385,
+            -0.007924830667307956,
+            0.0022291797354234005,
+            -0.0032976226056924502,
+            0.009615674261248298,
+            -0.0051932155872883,
+            -0.0065636389655770455,
+            0.02426106498596339,
+            0.011445591067574932,
+            0.013312596099532533,
+            0.0055118624272110965,
+            -0.007220620072349783,
+            -0.01916551187613823,
+            -0.003977616240384473,
+            -0.013204506708246246,
+            -0.006315966975649096,
+            0.01434099854765068,
+            0.012588249778961223,
+            0.006357972880218866,
+            0.0007747002406996095,
+            -0.004794439624135275,
+            0.006975593520375694,
+            0.0262335496080837,
+            -0.016481909971520166,
+            -0.0013084662210086592,
+            -0.01369313900978409,
+            0.01921748474212319,
+            0.03333761672336437,
+            -0.0008918419419301244,
+            -0.007699525186523558,
+            0.013757216065348164,
+            0.00819045303047094,
+            -0.007411380297164911,
+            0.02453611080891859,
+            -0.025045462880463647,
+            -0.015989319900023054,
+            -0.011650886878708265,
+            -0.011407253985176972,
+            -0.008386367959227559,
+            0.016331961716602287,
+            -0.00141225438918211,
+            -0.0018505754298268654,
+            0.009557291617446645,
+            0.03357924917013024,
+            0.006916860901290756,
+            0.007030308953738567,
+            -0.002739345982346728,
+            -0.009376522438071562,
+            0.011482637968411055,
+            -0.007259054391879614,
+            -0.04075661435928125,
+            0.004250195164583401,
+            -0.005160505473755424,
+            0.02405623169060588,
+            -0.02197586561375289,
+            -0.00619175761282745,
+            -0.006736129049831973,
+            -0.028200015908139763,
+            -0.005867722104196303,
+            0.02805071895018145,
+            -0.015402198567775478,
+            0.014773294766214671,
+            -0.009502185346558072,
+            -0.0013419858580438616,
+            0.004443749010136947,
+            0.013008783570355656,
+            -0.02289662952366337,
+            0.0013834888740540108,
+            -0.00048549197205067054,
+            0.027969842651517314,
+            0.010437729090975158,
+            -0.0047705656249623744,
+            0.014202218020412936,
+            0.008240861154198411,
+            -0.018637315941614406,
+            0.007314019317343154,
+            -0.005747966765101792,
+            0.014833896739762147,
+            -0.01314847940891154,
+            0.007762100494499689,
+            0.00705026393499352,
+            -0.01278023669088385,
+            0.03016342404000438,
+            -0.015204129985039077,
+            0.007776946006562982,
+            0.009278778571285689,
+            0.009656723862179986,
+            0.01257686993982689,
+            0.004362439504889069,
+            -0.02898418193036562,
+            0.0002144431781454467,
+            -0.03828100004144633,
+            -0.00833748467467171,
+            -0.002611937107022567,
+            -0.002145329666219461,
+            0.007050630951707937,
+            -0.009383645162261629,
+            -0.033976442057557676,
+            0.0006429978367640709,
+            0.01603027598131823,
+            0.017501067413108286,
+            0.0002196997337451293,
+            0.013075379412332973,
+            -0.006197537539554383,
+            -0.021148120787551232,
+            -0.016986448660412497,
+            -0.010207705210051413,
+            -0.02122194744719993,
+            -0.020550470201191853,
+            -0.00908564461857315,
+            0.012090079127547193,
+            -0.009927390005879807,
+            0.013888458967497627,
+            0.029513250508042935,
+            -0.024529175134769883,
+            0.0011144447737101862,
+            0.021977677894839922,
+            -0.00198308986575084,
+            0.014108189311291326,
+            0.001696775006906642,
+            0.014122064251288995,
+            -0.0004166140484018266,
+            -0.010556016685519523,
+            -0.009729362492661684,
+            0.017487929296614076,
+            -0.0026043257634797124,
+            0.016715798948897194,
+            0.012183487338360088,
+            0.003771721868999568,
+            -0.0030887189945795796,
+            0.010667776697461199,
+            -0.026471157050402344,
+            0.014249996582064706,
+            -0.006690961709923224,
+            -0.0181382691440242,
+            -0.008729104191096461,
+            0.003882731567105777,
+            -0.005609384377642956,
+            -0.007794812949125115,
+            0.012967089730551055,
+            -0.00944374111323783,
+            0.02088525981049842,
+            0.0023400121055514958,
+            -0.012196925240774426,
+            0.003648688547842322,
+            -0.03322041209304968,
+            0.0015191765655963997,
+            -0.018402382776013956,
+            -0.024659631472836927,
+            0.03840173429666435,
+            0.011991452465253198,
+            0.006639702644606365,
+            -0.01675838115087676,
+            0.007953102302963526,
+            0.0004257479565099406,
+            0.017484338819046517,
+            0.011068864013925452,
+            0.006466915839538296,
+            0.008364454017976605,
+            -0.023316137291380285,
+            -0.010898421318668592,
+            0.016204569093730564,
+            0.012099473359791148,
+            -0.01017229369980751,
+            0.019431669546223253,
+            0.017955119269367787,
+            0.007300372071724306,
+            -0.006898619924677706,
+            0.01647927363640854,
+            0.021012103522831126,
+            0.010291016618557653,
+            -0.00047955552103474145,
+            0.0008684297441778696,
+            -0.010534247928997246,
+            0.005037930235889528,
+            0.035417626740307205,
+            0.014712595692093886,
+            0.010650306363963653,
+            -0.03626584475133921,
+            0.021764456453493054,
+            -0.011961521673661574,
+            0.009092981192874177,
+            0.006089104887183293,
+            0.017982801101846038,
+            0.006916969397519662,
+            0.014196254958707752,
+            0.011745069030657836,
+            -0.006689305450954901,
+            0.025205280281804708,
+            -0.001601414806100461,
+            -0.011992525142592405,
+            0.003386930324498371,
+            -0.01859193251438437,
+            0.005303261319773131,
+            -0.0041710536965100615,
+            -0.004459393120257307,
+            0.0032461828144552565,
+            -0.0018650956576455287,
+            0.015678225251799887,
+            0.003894142628084471,
+            -0.02355212701026675,
+            -0.0014873755115206368,
+            -0.024284295126348117,
+            0.0027260592623277375,
+            -0.022185571998195344,
+            0.00022883605560765203,
+            -0.005557568353382136,
+            -0.009749382813201403,
+            -0.025385857431333838,
+            -0.009853073898348951,
+            -0.0035843255959309307,
+            0.007661137888845632,
+            -0.010521037433807653,
+            -0.004246698084422667,
+            -0.0019920499321964815,
+            -0.019465113363548715,
+            0.009014255846983305,
+            0.027122533756959888,
+            0.016848221859975445,
+            0.027170883580103264,
+            -0.00019554112591107156,
+            0.013614675849485926,
+            -0.002066822813103284,
+            -0.01603063005944364,
+            0.008824663429777616,
+            0.0075653522381218495,
+            -0.00531780740880296,
+            0.014643348459785112,
+            0.0072634484937344704,
+            -0.012800149015319017,
+            0.009859503660384916,
+            -0.007161712176469609,
+            -0.006437857349742842,
+            -0.009764856421740622,
+            -0.02058868349990357,
+            -0.014866137971603676,
+            0.0012450207572602557,
+            -0.002123265406747006,
+            0.006000473862248991,
+            -0.004054016790850226,
+            0.009584879012937315,
+            -0.00608816807208866,
+            -0.00925801057439376,
+            -0.013962683072792106,
+            -0.008924356118158748,
+            -0.004210064145949253,
+            -0.004381709838058668,
+            -0.011732699976353612,
+            -0.005479523097469261,
+            0.0010680258077037758,
+            -0.0019401640391035755,
+            -0.01397652201595301,
+            0.011650839243603377,
+            0.0058731838291845645,
+            -0.002462743134993977,
+            0.02219008688925881,
+            0.01097316136828334,
+            0.0018083001685932406,
+            0.01366095798812196,
+            0.0003942601242216818,
+            -0.0064063532958756175,
+            -0.03343312148366622,
+            -0.0005083355836126291,
+            -0.02815586367527057,
+            0.025770078162165123,
+            -0.0069114949102666145,
+            -0.010322087105114614,
+            -0.002519047562374146,
+            -0.006063669644021698,
+            0.005873609611264377,
+            -0.002722940652283699,
+            0.0021229525800901114,
+            -0.023673778230768945,
+            0.0028629715379660113,
+            -0.003953653437464966,
+            -0.0036366957271968823,
+            0.007469664864329332,
+            0.0346520690389124,
+            -0.004054282838167587,
+            -0.0034765253405897676,
+            0.005159466707114532,
+            0.011642913651743502,
+            0.027845571220425726,
+            -0.004445196354520585,
+            -0.0032038508043339413,
+            -0.02233838584152591,
+            0.0054112576531374175,
+            -0.00019544838850471866,
+            -0.004733192299951088,
+            0.016664217348411003,
+            -0.016719175418727594,
+            -0.01631658832047864,
+            -0.00545969677717507,
+            -0.014570848053102338,
+            -0.00022620258237138677,
+            0.014454101124666536,
+            0.011056312437637014,
+            -0.004035526363810078,
+            -0.012459378363360557,
+            -0.0022595493558020706,
+            -0.030370952502579162,
+            0.012766010284884204,
+            0.1141689619126009,
+            -0.00766197047210299,
+            -0.00896655658602649,
+            -0.009481112393590025,
+            0.001435269335646347,
+            0.001338755472877713,
+            -0.011739143028767118,
+            0.07166096084053437,
+            -0.0077855767070652395,
+            0.0002139580430342728,
+            -0.011312198831181402,
+            0.001844005266872301,
+            -0.001825412693163496,
+            0.01371932421024605,
+            -0.005025814503687022,
+            -0.018213360991353643,
+            -0.0033612481616769416,
+            -0.02941403830641238,
+            -0.0041137865872933515,
+            -0.005683630256743215,
+            -0.004288198570156578,
+            -0.014982999363445122,
+            0.0067149665791261335,
+            -0.019661973497372313,
+            -0.012688556129040908,
+            0.001124391650175233,
+            -0.0033623453063430534,
+            0.00996944383866679,
+            -0.020496336814603385,
+            -0.0044354321069609536,
+            0.0019875840782527733,
+            0.009706675240563883,
+            0.0011600364428219214,
+            -0.012755178230821602,
+            -0.0026514442584739412,
+            0.004056350166031111,
+            0.01909951327913368,
+            0.014360281326789544,
+            0.003732344102045803,
+            0.008623457695718206,
+            0.014461868085333023,
+            -0.001692149528228451,
+            0.01948035376572433,
+            0.01172550239477637,
+            0.0031966531144076173,
+            0.0004779413449255626,
+            0.0030912562273611517,
+            -0.004597633235071031,
+            0.01582547400214653,
+            -0.014811177637844808,
+            -0.02411202772171917,
+            -0.0071272238837422185,
+            0.018015541137524113,
+            -0.013081287852028074,
+            1.8837689271256874e-05,
+            0.004062803284646432,
+            0.027569726868200254,
+            0.0010564850664255057,
+            -0.01179755720247972,
+            0.0028876134800690334,
+            -0.01638215925131053,
+            0.022944183106912663,
+            0.0026043713064243836,
+            0.03279173899194346,
+            -0.005103903273968361,
+            0.00578213669463383,
+            -0.020744194169889544,
+            -0.007588793365216579,
+            -0.00819862611489168,
+            0.0017401442433921013,
+            0.010993500174794084,
+            0.005411238350013134,
+            0.01620925462604661,
+            -0.00641615069699833,
+            -0.020751730783437995,
+            0.014776878596707538,
+            -0.009552193304486048,
+            -0.03393319554458277,
+            0.008881987295131347,
+            -0.029102297123590998,
+            -0.0045267103602711,
+            0.00368971742585232,
+            -0.014269341759285646,
+            -0.016264809810970175,
+            0.012837302095591106,
+            -0.0011890334180728564,
+            -0.008937992768323412,
+            0.0014444613725248846,
+            -0.017110187133760626,
+            -0.02385668037818897,
+            0.006017405364919834,
+            0.00676670672597817,
+            -0.0077994500447595,
+            0.018312452858615755,
+            -0.027015660037024585,
+            0.006123912473734307,
+            -0.012973676426738582,
+            0.009041974762537394,
+            -0.01580183887345168,
+            0.01163516332546347,
+            0.010887662470499361,
+            -0.001503467664709027,
+            -0.0076234124970223326,
+            0.01231202541338883,
+            0.004852720134894811,
+            0.033760365210093034,
+            0.028658326381800395,
+            0.000995999783627898,
+            -0.0025605367058226737,
+            -0.0020356794893366384,
+            0.011667525096213252,
+            -0.014049700287539817,
+            0.011218246246280464,
+            0.02658113044439717,
+            0.017568631325838108,
+            -0.019133062654008885,
+            0.018050678772708717,
+            -0.010356118256625172,
+            0.012872052420999813,
+            -0.025620154415048252,
+            -0.005135768485044442,
+            0.004936490920616527,
+            0.005163008729256001,
+            0.007425278892536464,
+            0.001410212306439329,
+            0.021891387289347516,
+            0.014999616600766989,
+            0.00027613811156650413,
+            0.00558422822386443,
+            0.01477009526981092,
+            0.010497782599293994,
+            -0.0005567894749233204,
+            -0.00467776506604011,
+            -0.016998035358377343,
+            0.00957271797745408,
+            -0.006065116857898469,
+            0.005138070401846389,
+            0.01698914855850691,
+            -0.01230257809343985,
+            0.011109844802663414,
+            0.007518553660463051,
+            -0.015504773927595823,
+            -0.017169933945839305,
+            0.005552829003043926,
+            -0.02252633790984703,
+            -0.01306722099657118,
+            0.02715689072730398,
+            -0.010513941399990838,
+            -0.02790582732974509,
+            -0.0009847768069205477,
+            0.009797865241108237,
+            -0.002772952073769185,
+            -0.0027968741526000576,
+            0.0037541733651479613,
+            0.010801137722284016,
+            -0.014658156970099418,
+            0.002075415233345957,
+            0.004735264309137368,
+            0.01930804315461151,
+            -0.0014151522239496115,
+            -0.015046118414100617,
+            -0.002288495894227231,
+            0.021592829677937263,
+            0.017174609713578173,
+            0.00265560378681856,
+            -0.016499825201963983,
+            -3.965296928410772e-05,
+            -0.0074142747014567635,
+            0.00021360696629503726,
+            0.00027431174904249525,
+            -0.0056391089282546054,
+            -0.018931681700770225,
+            -0.003888248121485032,
+            0.011181675419027736,
+            -0.00247252920175839,
+            0.01977860312223846,
+            -0.0011527575128223392,
+            -0.015763587433200915,
+            0.001960734212846797,
+            -0.005280860341480037,
+            -0.010372829570478649,
+            0.01351900743232294,
+            -0.009359942374603997,
+            0.006788746813046239,
+            -0.003212015225284119,
+            0.004611143085183367,
+            -0.0003308099698088422,
+            -0.017017265064185255,
+            0.02924142709831441,
+            -0.0018238549856433466,
+            -0.004769274041546295,
+            -0.0014058363976586562,
+            -0.033393331166741945,
+            0.015017165991303838,
+            -0.014574579250749552,
+            0.006745699409061803,
+            0.002705853802154516,
+            0.01272741296341735,
+            0.003253095246601883,
+            -0.008898616461779179,
+            0.0027485606928249326,
+            0.017040360927723885,
+            0.013756402798365262,
+            -0.007292864293090161,
+            0.008694855886257706,
+            -0.0028934389922887676,
+            -0.0037483208383542015,
+            0.01363963339139356,
+            0.004170611039133969,
+            -0.006282657671540625,
+            -0.0009784372731140945
+          ],
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "Mixture-of-Experts (MoE)",
+            "fine-grained expert segmentation",
+            "shared expert isolation"
+          ],
+          "title": "Deepseekmoe: Towards Ultimate Expert Specialization In Mixture-Of-Experts Language Models"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "95f27e84-0fb9-44ee-a783-45f42551401c",
+        "properties": {
+          "page_content": "5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n",
+          "title": "Scaling Up To Deepseekmoe 16B",
+          "keyphrases": [
+            "DeepSeekMoE 16B",
+            "MoE model",
+            "language modeling",
+            "evaluation benchmarks",
+            "Open LLM Leaderboard"
+          ],
+          "embedding": [
+            0.005968157202005386,
+            -0.015490485355257988,
+            -0.027613861486315727,
+            0.012274806387722492,
+            0.008261889219284058,
+            -0.01577553153038025,
+            -0.016158562153577805,
+            0.06238951161503792,
+            -0.002393943490460515,
+            0.048564765602350235,
+            0.008756265975534916,
+            -0.009914266876876354,
+            0.03532792627811432,
+            -0.018332039937376976,
+            -0.006921280641108751,
+            0.023961706086993217,
+            0.018617086112499237,
+            0.02068367227911949,
+            -0.00015922509192023426,
+            -0.02930632419884205,
+            -0.021538810804486275,
+            -0.03865940496325493,
+            -0.037447959184646606,
+            -0.023801365867257118,
+            -0.021129056811332703,
+            0.012862714938819408,
+            0.01521434634923935,
+            0.015909146517515182,
+            -0.041616760194301605,
+            0.008319789543747902,
+            0.016042763367295265,
+            0.021449735388159752,
+            0.004554060287773609,
+            0.009063581936061382,
+            0.012675653211772442,
+            -0.015294515527784824,
+            0.07097653299570084,
+            -0.0016189739108085632,
+            -0.009361989796161652,
+            0.010226036421954632,
+            0.018777426332235336,
+            0.010083513334393501,
+            -0.009317451156675816,
+            -0.010787221603095531,
+            0.019899794831871986,
+            0.012176821939647198,
+            -0.00043619878124445677,
+            -0.0028460093308240175,
+            0.00021837776876054704,
+            -0.021592257544398308,
+            -0.027720753103494644,
+            -0.013682222925126553,
+            0.0055272262543439865,
+            -0.006075049284845591,
+            -0.017797578126192093,
+            -0.005594033747911453,
+            0.03819620609283447,
+            0.0277029387652874,
+            0.03762611374258995,
+            -0.01240842230618,
+            0.03308318555355072,
+            -0.017494717612862587,
+            -0.006288834381848574,
+            -2.058165227936115e-05,
+            -0.006248749326914549,
+            -0.034597497433423996,
+            -0.005117471795529127,
+            0.001308317994698882,
+            0.005865718703716993,
+            -0.024068597704172134,
+            -0.04464537650346756,
+            0.02754260040819645,
+            -0.003171140095219016,
+            0.00048101565334945917,
+            0.02171696536242962,
+            0.0010444275103509426,
+            0.01798463985323906,
+            0.007918942719697952,
+            -0.02321345917880535,
+            -0.014279038645327091,
+            0.020505519583821297,
+            0.010386374779045582,
+            -0.006529341917484999,
+            0.025565091520547867,
+            0.06046544760465622,
+            -0.01011914387345314,
+            -0.024959366768598557,
+            -0.08444496989250183,
+            7.453237049048766e-05,
+            -0.0283442921936512,
+            -0.011490928940474987,
+            0.007825411856174469,
+            -0.004527336917817593,
+            -0.019062472507357597,
+            -0.0022169030271470547,
+            0.025547275319695473,
+            -0.005602941382676363,
+            -0.006342280190438032,
+            -0.017957916483283043,
+            0.03791115805506706,
+            -0.002776974579319358,
+            -0.004046321380883455,
+            0.030054569244384766,
+            0.0013561969390138984,
+            0.016888992860913277,
+            -0.0027858822140842676,
+            -0.02565416693687439,
+            0.029965493828058243,
+            -0.0067653958685696125,
+            0.08929075300693512,
+            -0.014564084820449352,
+            -0.009691574610769749,
+            -0.050453197211027145,
+            -0.011063359677791595,
+            -0.014083068817853928,
+            -0.003999555949121714,
+            -0.03424118831753731,
+            0.001817170181311667,
+            0.011588914319872856,
+            -0.014368114992976189,
+            0.013682222925126553,
+            0.021859489381313324,
+            0.03887319192290306,
+            -0.026990322396159172,
+            -0.04343393072485924,
+            0.06506182253360748,
+            0.021503180265426636,
+            -0.014804592356085777,
+            0.013824746012687683,
+            0.003099878551438451,
+            -0.01950785703957081,
+            0.011366221122443676,
+            -0.022393951192498207,
+            -0.013334822840988636,
+            0.017254209145903587,
+            -0.00778978131711483,
+            0.026152998208999634,
+            0.0797417014837265,
+            -0.018332039937376976,
+            0.014973838813602924,
+            0.022108905017375946,
+            -0.029217246919870377,
+            0.012372790835797787,
+            0.0028148323763161898,
+            -0.034989435225725174,
+            0.0058924416080117226,
+            0.008894335478544235,
+            0.05825633928179741,
+            0.003538582706823945,
+            0.010929744690656662,
+            0.010974282398819923,
+            0.011098990216851234,
+            -0.0009859707206487656,
+            0.023997336626052856,
+            0.008497943170368671,
+            0.0018372124759480357,
+            -0.026527121663093567,
+            0.01851019449532032,
+            -0.018813056871294975,
+            0.00869391206651926,
+            0.012167913839221,
+            -0.022661181166768074,
+            -0.012417329475283623,
+            -0.002204654971137643,
+            0.004934864118695259,
+            0.05184279754757881,
+            0.020024502649903297,
+            -0.010555621236562729,
+            -0.0979846715927124,
+            0.050132520496845245,
+            0.014145422726869583,
+            0.024923736229538918,
+            -0.04197306931018829,
+            0.017476901412010193,
+            0.009647035971283913,
+            0.028878754004836082,
+            0.0057677337899804115,
+            0.0020532242488116026,
+            -0.06816169619560242,
+            0.03545263409614563,
+            0.011419667862355709,
+            0.024994997307658195,
+            0.02485247515141964,
+            0.008840889669954777,
+            0.014065253548324108,
+            0.0004027949180454016,
+            0.04058346897363663,
+            0.029573554173111916,
+            -0.005389156751334667,
+            0.0357198640704155,
+            0.036236513406038284,
+            -0.0020799473859369755,
+            0.04428907111287117,
+            0.003500724909827113,
+            0.015089638531208038,
+            -0.006609511096030474,
+            -0.027328815311193466,
+            0.04963368922472,
+            -0.01895557902753353,
+            0.024157674983143806,
+            0.012069929391145706,
+            -0.03119475580751896,
+            -0.011125713586807251,
+            -0.01026166696101427,
+            0.028611524030566216,
+            -0.01782430149614811,
+            0.0019407644867897034,
+            0.02168133482336998,
+            0.009451067075133324,
+            -0.008163904771208763,
+            -0.026919061318039894,
+            -0.0066406880505383015,
+            -0.028967831283807755,
+            -0.04175928235054016,
+            0.0026010475121438503,
+            0.02702595293521881,
+            -0.012755822390317917,
+            -0.00048185072955675423,
+            -0.010742682963609695,
+            -0.030820632353425026,
+            -0.009932082146406174,
+            0.03388487920165062,
+            0.007901127450168133,
+            -0.011802698485553265,
+            -0.02971607819199562,
+            -0.014403746463358402,
+            0.01066251378506422,
+            -0.0017403413075953722,
+            0.05105891823768616,
+            -0.01533905416727066,
+            0.020648041740059853,
+            0.011437483131885529,
+            0.0065605188719928265,
+            0.02424675226211548,
+            0.02437146008014679,
+            -0.016559408977627754,
+            0.005843449383974075,
+            -0.003349294187501073,
+            -0.0055183181539177895,
+            0.043505191802978516,
+            -0.003075382439419627,
+            -0.029128169640898705,
+            -0.031586695462465286,
+            -0.012203545309603214,
+            0.010181497782468796,
+            -0.005081841256469488,
+            -0.01433248445391655,
+            0.02123595029115677,
+            0.02882530726492405,
+            0.04842223972082138,
+            0.01276472955942154,
+            -0.05725867673754692,
+            -0.013896007090806961,
+            0.0020732665434479713,
+            -0.02260773442685604,
+            0.032424017786979675,
+            0.060322925448417664,
+            -0.00506847957149148,
+            -0.020612411201000214,
+            -0.027150660753250122,
+            0.0659882202744484,
+            0.012328253127634525,
+            -0.04464537650346756,
+            0.04938427358865738,
+            -0.013655499555170536,
+            0.022340504452586174,
+            0.04517984017729759,
+            0.021859489381313324,
+            -0.0304286926984787,
+            0.017735224217176437,
+            0.024157674983143806,
+            -0.004836879670619965,
+            -0.06196194142103195,
+            -0.02260773442685604,
+            -0.002491928171366453,
+            -0.002779201604425907,
+            -0.013397176750004292,
+            -0.024585243314504623,
+            0.02485247515141964,
+            0.014537361450493336,
+            0.019098103046417236,
+            0.02120031975209713,
+            0.02324908971786499,
+            0.0164970550686121,
+            0.005259995348751545,
+            -0.007331034634262323,
+            -0.005153102800250053,
+            -0.023355981335043907,
+            -0.0269012451171875,
+            -0.02184167318046093,
+            -0.01982853375375271,
+            0.017574885860085487,
+            -0.04371897876262665,
+            0.016568316146731377,
+            0.0024251204449683428,
+            0.024389274418354034,
+            -0.023712290450930595,
+            0.004099767655134201,
+            0.028682785108685493,
+            -0.010591251775622368,
+            -0.04079725220799446,
+            0.025084074586629868,
+            0.011677990667521954,
+            -0.007241957820951939,
+            -0.007504734676331282,
+            0.0006981407641433179,
+            -0.008867612108588219,
+            0.031622324138879776,
+            -0.025280045345425606,
+            -0.01902684196829796,
+            -0.005375795066356659,
+            -0.02011357992887497,
+            0.0058924416080117226,
+            -0.03436589613556862,
+            -0.007464650087058544,
+            0.048600394278764725,
+            0.02132502757012844,
+            0.015098546631634235,
+            -0.04240063950419426,
+            -0.006631780415773392,
+            0.06142747774720192,
+            0.015392500907182693,
+            -0.07354194670915604,
+            0.044787902384996414,
+            0.0010861823102459311,
+            0.006698588375002146,
+            0.004685448482632637,
+            0.030054569244384766,
+            -0.015454854816198349,
+            -0.0159358698874712,
+            0.007673981133848429,
+            -0.030250540003180504,
+            -0.022091088816523552,
+            -0.005175372119992971,
+            0.021449735388159752,
+            -0.04521546885371208,
+            -0.03887319192290306,
+            -0.016791008412837982,
+            -0.0053802491165697575,
+            0.017129501327872276,
+            -0.011758159846067429,
+            -0.024923736229538918,
+            -0.030054569244384766,
+            -0.0005670305690728128,
+            -0.00188175099901855,
+            -0.04101103916764259,
+            -0.03067810833454132,
+            0.0068455650471150875,
+            0.021930750459432602,
+            0.02886093780398369,
+            0.0017392279114574194,
+            0.009459974244236946,
+            0.02504844404757023,
+            -0.00419552531093359,
+            0.029520107433199883,
+            -0.02766730636358261,
+            -0.032584354281425476,
+            0.012782545760273933,
+            -0.010911929421126842,
+            0.02116468921303749,
+            0.001672420185059309,
+            -0.021022165194153786,
+            -0.04689011722803116,
+            0.02706158347427845,
+            -0.0457855649292469,
+            -0.012087744660675526,
+            0.010974282398819923,
+            -0.0022937320172786713,
+            0.010288390330970287,
+            -0.00252533215098083,
+            0.0066451421007514,
+            0.02526222914457321,
+            -0.017601609230041504,
+            0.011909591034054756,
+            0.008012473583221436,
+            0.006306649651378393,
+            0.033991772681474686,
+            -0.008444496430456638,
+            -0.020505519583821297,
+            0.026242075487971306,
+            -0.03416992723941803,
+            0.01401180773973465,
+            0.0360405407845974,
+            0.013726761564612389,
+            0.010181497782468796,
+            0.011998668313026428,
+            0.011820513755083084,
+            0.0013862603809684515,
+            -0.02376573532819748,
+            0.01915154792368412,
+            0.01711168698966503,
+            -0.06866052746772766,
+            -0.020327365025877953,
+            -0.012702375650405884,
+            0.024068597704172134,
+            0.014412653632462025,
+            0.03340386599302292,
+            0.0033092093653976917,
+            -0.032459646463394165,
+            -0.025636352598667145,
+            -0.0075002810917794704,
+            0.04624876379966736,
+            -0.007437927182763815,
+            0.015962593257427216,
+            -0.00821289699524641,
+            0.021859489381313324,
+            0.012675653211772442,
+            -0.0016623989213258028,
+            0.026847798377275467,
+            -0.02597484551370144,
+            0.01240842230618,
+            -0.018688349053263664,
+            -0.04375460743904114,
+            -0.009771743789315224,
+            0.02360539697110653,
+            0.01886650174856186,
+            0.00418661767616868,
+            0.0078075965866446495,
+            -0.01634562388062477,
+            -0.0024986090138554573,
+            -0.03823183476924896,
+            0.010448728688061237,
+            0.027649492025375366,
+            0.0016223143320530653,
+            -0.005638572387397289,
+            -0.0038815289735794067,
+            0.0159358698874712,
+            0.0004142079269513488,
+            0.007469104137271643,
+            -0.034918174147605896,
+            0.040334053337574005,
+            0.0054158796556293964,
+            0.04582119360566139,
+            -0.04724642634391785,
+            -0.017450178042054176,
+            -0.0038904366083443165,
+            -0.007380026858299971,
+            0.042863838374614716,
+            -0.03573768213391304,
+            -0.02830866165459156,
+            -0.011464206501841545,
+            0.01947222650051117,
+            0.015668639913201332,
+            -0.02155662700533867,
+            -0.03976396098732948,
+            -0.002783655421808362,
+            -0.035274479538202286,
+            -0.010947559960186481,
+            -0.014412653632462025,
+            -0.0032802594359964132,
+            -0.049847472459077835,
+            -0.0017592702060937881,
+            -0.0835898295044899,
+            0.005081841256469488,
+            0.03755485266447067,
+            -0.009807374328374863,
+            0.04111792892217636,
+            -0.002641132101416588,
+            0.002984078601002693,
+            -0.047959040850400925,
+            0.01689790189266205,
+            0.026242075487971306,
+            0.04678322374820709,
+            -0.022180166095495224,
+            0.005242179613560438,
+            0.014679884538054466,
+            -0.011214790865778923,
+            -0.017610516399145126,
+            -0.010511082597076893,
+            0.02004231885075569,
+            -0.0051976414397358894,
+            0.0012392833596095443,
+            0.04528673365712166,
+            0.029537923634052277,
+            0.014697699807584286,
+            0.031943000853061676,
+            0.00030648044776171446,
+            -0.01814497821033001,
+            0.0025320129934698343,
+            0.043148886412382126,
+            0.04093977436423302,
+            -0.04375460743904114,
+            0.015543931163847446,
+            -0.03905134275555611,
+            0.025636352598667145,
+            0.013985084369778633,
+            0.007442380767315626,
+            0.03343949466943741,
+            0.006466988008469343,
+            -0.005803364794701338,
+            0.00020348520774859935,
+            0.024674320593476295,
+            -0.01662176288664341,
+            0.017922285944223404,
+            -0.004836879670619965,
+            -0.022661181166768074,
+            0.000857365841511637,
+            -0.015232161618769169,
+            0.017191855236887932,
+            0.02187730371952057,
+            -0.007188511546701193,
+            -0.0010806149803102016,
+            -0.0003092641127295792,
+            0.021948564797639847,
+            0.004395948722958565,
+            -0.0005812272429466248,
+            0.011428575031459332,
+            0.03500724956393242,
+            -0.03215678781270981,
+            -0.010644698515534401,
+            0.03212115541100502,
+            -0.033101003617048264,
+            0.009566866792738438,
+            -0.04104666784405708,
+            0.010609067045152187,
+            0.0022324915044009686,
+            -0.020986534655094147,
+            0.04076162353157997,
+            -0.010911929421126842,
+            0.001672420185059309,
+            -0.013317007571458817,
+            0.00856920424848795,
+            0.015561746433377266,
+            0.03762611374258995,
+            -0.0039015712682157755,
+            0.02882530726492405,
+            0.01489366963505745,
+            -0.020220471546053886,
+            0.04008463770151138,
+            -0.04268568381667137,
+            -0.010208221152424812,
+            0.015249977819621563,
+            -0.005981518421322107,
+            -0.012461868114769459,
+            -0.020576780661940575,
+            -0.021663518622517586,
+            -0.007001449819654226,
+            0.015143085271120071,
+            -0.02686561457812786,
+            0.006760942284017801,
+            0.006801026873290539,
+            -0.017842117697000504,
+            -0.007651711814105511,
+            0.006609511096030474,
+            -0.008016927167773247,
+            -0.0056296647526323795,
+            0.028201768174767494,
+            -0.017975732684135437,
+            0.01260439120233059,
+            0.02321345917880535,
+            0.028843123465776443,
+            -0.018162794411182404,
+            -0.0016189739108085632,
+            -0.0032490824814885855,
+            -0.0058479029685258865,
+            -0.03285158798098564,
+            0.026509307324886322,
+            0.009620312601327896,
+            -0.010056789964437485,
+            -0.024709951132535934,
+            -0.008836435154080391,
+            0.01879524067044258,
+            -0.0035185404121875763,
+            -0.005549495108425617,
+            -0.021984197199344635,
+            -0.02878967672586441,
+            0.0257788747549057,
+            0.01863490231335163,
+            -0.046711962670087814,
+            -0.0032112249173223972,
+            0.005411426071077585,
+            0.004783433396369219,
+            0.0027213015127927065,
+            -0.014849130995571613,
+            0.007388934958726168,
+            0.026972506195306778,
+            0.0054826876148581505,
+            -0.015543931163847446,
+            -0.012622206471860409,
+            -0.005099656525999308,
+            0.008613742887973785,
+            0.0069925421848893166,
+            0.009932082146406174,
+            0.02975170873105526,
+            0.04072599112987518,
+            0.0018628222169354558,
+            -0.01288943737745285,
+            -0.02561853639781475,
+            0.025315675884485245,
+            -0.0019630338065326214,
+            0.04165239259600639,
+            0.04564303904771805,
+            0.0025876860599964857,
+            -0.060572341084480286,
+            0.012479683384299278,
+            -0.02517315186560154,
+            -0.009843004867434502,
+            0.0066406880505383015,
+            0.0036209789104759693,
+            -0.04660507291555405,
+            -0.0031956364400684834,
+            0.0022536474280059338,
+            0.034152109175920486,
+            -0.007299857679754496,
+            -0.04436033219099045,
+            -0.008929966017603874,
+            -0.007406750228255987,
+            0.02574324421584606,
+            0.010110235773026943,
+            -0.007892219349741936,
+            -0.015294515527784824,
+            0.013361545279622078,
+            -0.0026567205786705017,
+            -0.006190849468111992,
+            0.017013700678944588,
+            -0.007691796403378248,
+            0.011170252226293087,
+            -0.012096652761101723,
+            -0.019739456474781036,
+            -0.017655055969953537,
+            0.03393832594156265,
+            -0.02706158347427845,
+            0.025565091520547867,
+            0.014947115443646908,
+            0.0026923513505607843,
+            0.039407651871442795,
+            -0.01535686943680048,
+            -0.027364445850253105,
+            -0.002897228579968214,
+            -0.0037078289315104485,
+            -0.03766174241900444,
+            0.01815388724207878,
+            -0.023516319692134857,
+            -0.022091088816523552,
+            0.013495161198079586,
+            0.00951342098414898,
+            0.0002751643187366426,
+            -0.009308543056249619,
+            -0.00851130485534668,
+            -0.0015465988544747233,
+            0.01956130377948284,
+            0.019258441403508186,
+            -0.007451288867741823,
+            0.043505191802978516,
+            -0.0004167132137808949,
+            0.0038369905669242144,
+            0.0005995993851684034,
+            0.018100440502166748,
+            0.060287293046712875,
+            0.020452072843909264,
+            0.0012481911107897758,
+            0.030482139438390732,
+            0.022500842809677124,
+            0.037162911146879196,
+            -0.03019709326326847,
+            -0.04564303904771805,
+            0.017895564436912537,
+            -0.007250865455716848,
+            0.009522328153252602,
+            0.022429581731557846,
+            0.0011813833843916655,
+            0.01418996136635542,
+            0.007063803728669882,
+            -0.010243851691484451,
+            0.023801365867257118,
+            -0.032905034720897675,
+            -0.02789890766143799,
+            0.027115030214190483,
+            0.010056789964437485,
+            0.019971057772636414,
+            0.0360405407845974,
+            -0.024870289489626884,
+            -0.033671095967292786,
+            0.004881417844444513,
+            0.006680772639811039,
+            -0.0283442921936512,
+            0.012942884117364883,
+            -0.013521884568035603,
+            0.015160900540649891,
+            0.0064001805149018764,
+            -0.012648929841816425,
+            -0.0027279823552817106,
+            0.013780207373201847,
+            -0.0141721460968256,
+            0.008698366582393646,
+            -0.015160900540649891,
+            -0.012452960945665836,
+            0.010626882314682007,
+            -0.030642477795481682,
+            0.0008445610292255878,
+            0.00507738720625639,
+            -0.0015131949912756681,
+            0.006230934057384729,
+            -0.0067297653295099735,
+            -0.004431579262018204,
+            -0.005865718703716993,
+            -0.03185392543673515,
+            0.02795235440135002,
+            0.02766730636358261,
+            -0.025119705125689507,
+            -0.0030553401447832584,
+            -0.0036900134291499853,
+            -0.014528454281389713,
+            -0.006346734240651131,
+            0.010876297950744629,
+            -0.014448285102844238,
+            0.0021411876659840345,
+            -0.016087301075458527,
+            -0.018367672339081764,
+            0.023961706086993217,
+            0.022518659010529518,
+            0.042863838374614716,
+            -0.0010037861065939069,
+            0.022750258445739746,
+            0.021307211369276047,
+            0.0011028843000531197,
+            0.0017982413992285728,
+            0.017494717612862587,
+            -0.017557071521878242,
+            0.028985645622015,
+            0.025707613676786423,
+            0.03099878504872322,
+            0.010849575512111187,
+            -0.03221023455262184,
+            -0.0024674320593476295,
+            0.018848687410354614,
+            0.017076054587960243,
+            0.0008317562169395387,
+            -0.013486253097653389,
+            -0.009771743789315224,
+            0.012533130124211311,
+            -0.02152099646627903,
+            -0.0065248883329331875,
+            0.013512976467609406,
+            -0.016514871269464493,
+            0.0032802594359964132,
+            -0.007825411856174469,
+            0.02472776733338833,
+            0.0028838668949902058,
+            -0.005193187389522791,
+            0.012773637659847736,
+            -0.00048185072955675423,
+            0.005237726029008627,
+            -0.005037302616983652,
+            0.02324908971786499,
+            0.03493598848581314,
+            0.012684560380876064,
+            0.04407528415322304,
+            0.01998887211084366,
+            -0.013423899188637733,
+            -0.009531236253678799,
+            -0.020612411201000214,
+            0.022358320653438568,
+            -0.030286170542240143,
+            -0.024709951132535934,
+            -0.022073272615671158,
+            0.01535686943680048,
+            0.00679657282307744,
+            0.015606285072863102,
+            -0.0033893787767738104,
+            -0.004447167739272118,
+            0.031372908502817154,
+            0.004179936833679676,
+            0.025511644780635834,
+            0.006907918956130743,
+            -0.007905581034719944,
+            -0.004012917634099722,
+            0.0024139860179275274,
+            0.008150543086230755,
+            -0.015241069719195366,
+            0.03826746717095375,
+            -0.0035630788188427687,
+            0.04813719540834427,
+            -0.052733566612005234,
+            -0.002003118395805359,
+            0.00635564187541604,
+            -0.01918718032538891,
+            -0.005669749341905117,
+            -0.03915823623538017,
+            0.02030954882502556,
+            -0.03377798572182655,
+            0.026669645681977272,
+            -0.02027391828596592,
+            0.026491491124033928,
+            -0.029929861426353455,
+            0.003576440503820777,
+            -0.0015243296511471272,
+            -0.0038035865873098373,
+            0.0164970550686121,
+            0.024282382801175117,
+            -0.002409531967714429,
+            -0.02590358257293701,
+            -0.0008083735010586679,
+            0.0037635019980371,
+            0.0066273268312215805,
+            -0.013717853464186192,
+            0.0054158796556293964,
+            -0.0012582122581079602,
+            -0.029680447652935982,
+            -0.033101003617048264,
+            0.012132283300161362,
+            -0.010911929421126842,
+            0.0337958037853241,
+            0.030375247821211815,
+            -0.026794353500008583,
+            -0.027043769136071205,
+            0.0034561865031719208,
+            -0.008676096796989441,
+            -0.009090305306017399,
+            0.020737119019031525,
+            0.04532236233353615,
+            0.00925509724766016,
+            -0.020790565758943558,
+            0.023943889886140823,
+            0.004734441172331572,
+            0.008582565933465958,
+            0.004422671627253294,
+            -0.021734781563282013,
+            -0.018492378294467926,
+            -0.01689790189266205,
+            -0.01726311631500721,
+            0.008177265524864197,
+            -0.01972164213657379,
+            -0.04012026637792587,
+            -0.04742458090186119,
+            -0.011179160326719284,
+            0.019383149221539497,
+            0.004144306294620037,
+            0.05052445828914642,
+            0.012452960945665836,
+            0.02478121407330036,
+            -0.0017191855004057288,
+            -0.005469325929880142,
+            0.04646254703402519,
+            -0.019899794831871986,
+            0.051700275391340256,
+            0.009700481779873371,
+            -0.0021634569857269526,
+            -0.01578443869948387,
+            0.03188955411314964,
+            -0.024905921891331673,
+            -0.0019240626133978367,
+            0.006342280190438032,
+            -0.0034984981175512075,
+            0.017298748716711998,
+            0.008644919842481613,
+            0.010128051973879337,
+            -0.02360539697110653,
+            0.028130507096648216,
+            -0.023641027510166168,
+            0.014590808190405369,
+            0.015926962718367577,
+            -0.002009799238294363,
+            0.0036098442506045103,
+            -0.0054826876148581505,
+            0.004574102349579334,
+            -0.002970716916024685,
+            0.011597821488976479,
+            -0.02510189078748226,
+            0.015009469352662563,
+            -0.02228705771267414,
+            -0.013317007571458817,
+            0.0004178266681265086,
+            -0.030945340171456337,
+            0.012399514205753803,
+            0.04108230024576187,
+            0.01037746760994196,
+            -0.02232268825173378,
+            0.005371341481804848,
+            -0.051450859755277634,
+            -0.008199535310268402,
+            0.0014809046406298876,
+            0.01271128375083208,
+            0.01346843782812357,
+            0.0007504734676331282,
+            0.009727205149829388,
+            -0.00955795869231224,
+            -0.03055340051651001,
+            0.033421680331230164,
+            -0.001038860180415213,
+            -0.0020298415329307318,
+            0.003928294405341148,
+            0.01851019449532032,
+            0.0025854590348899364,
+            0.004188844468444586,
+            0.012773637659847736,
+            -0.007709611672908068,
+            -0.03224586322903633,
+            -0.021449735388159752,
+            -0.0038926636334508657,
+            0.0019318568520247936,
+            -0.003694467479363084,
+            0.003318117232993245,
+            0.0055851261131465435,
+            -0.015535023994743824,
+            0.01082285214215517,
+            -0.011980853043496609,
+            0.010707051493227482,
+            -0.019169364124536514,
+            -0.004329140763729811,
+            0.006658503785729408,
+            -0.020808380097150803,
+            -0.018296409398317337,
+            -0.051450859755277634,
+            -0.010466543957591057,
+            0.002101103076711297,
+            0.021146873012185097,
+            0.006676319055259228,
+            0.008489035069942474,
+            -0.014394838362932205,
+            0.0068322038277983665,
+            0.003656609682366252,
+            0.02520878240466118,
+            -0.026829984039068222,
+            0.03119475580751896,
+            0.018723979592323303,
+            0.016951346769928932,
+            -0.03862377628684044,
+            0.03249527886509895,
+            0.016443608328700066,
+            -0.003759048180654645,
+            0.017521440982818604,
+            0.014002899639308453,
+            0.04753147065639496,
+            -0.030606847256422043,
+            0.004810156300663948,
+            -0.02754260040819645,
+            0.025084074586629868,
+            -0.03805368393659592,
+            0.028237400576472282,
+            0.0009163793292827904,
+            0.023943889886140823,
+            0.04428907111287117,
+            9.728875738801435e-05,
+            0.03851688280701637,
+            -0.022019827738404274,
+            -0.009045766666531563,
+            0.025636352598667145,
+            -0.017476901412010193,
+            0.015882423147559166,
+            0.00778978131711483,
+            -0.024282382801175117,
+            0.01838548667728901,
+            0.007709611672908068,
+            -0.0031956364400684834,
+            -0.04788777977228165,
+            0.02795235440135002,
+            0.02260773442685604,
+            0.02120031975209713,
+            0.0067297653295099735,
+            -0.012150098569691181,
+            -0.010234943591058254,
+            0.005611849017441273,
+            -0.009246189147233963,
+            0.016755377873778343,
+            -0.015846792608499527,
+            -0.030909709632396698,
+            0.00849348958581686,
+            -0.01927625574171543,
+            0.012648929841816425,
+            -0.0038904366083443165,
+            0.029092539101839066,
+            -0.017770856618881226,
+            -0.01068032905459404,
+            0.01101882103830576,
+            -0.0028237400110810995,
+            -0.016069484874606133,
+            -0.04842223972082138,
+            -0.038801927119493484,
+            0.008453404530882835,
+            -0.0013105449033901095,
+            -0.020184841006994247,
+            0.011909591034054756,
+            -0.010591251775622368,
+            -0.004792341031134129,
+            -0.01972164213657379,
+            -0.010742682963609695,
+            -0.023035304620862007,
+            -0.006760942284017801,
+            0.010074605233967304,
+            0.026277706027030945,
+            -0.018848687410354614,
+            0.03798241913318634,
+            0.029413215816020966,
+            0.013985084369778633,
+            -0.012506406754255295,
+            0.0257788747549057,
+            -0.009353081695735455,
+            -0.01489366963505745,
+            0.033671095967292786,
+            -0.0031466439832001925,
+            -0.009192743338644505,
+            0.00017439600196667016,
+            -0.024799028411507607,
+            -0.023070935159921646,
+            0.019579118117690086,
+            -0.022393951192498207,
+            -0.025280045345425606,
+            -0.004921502433717251,
+            -0.006141857244074345,
+            -0.02975170873105526,
+            -0.04753147065639496,
+            -0.011214790865778923,
+            0.03645029664039612,
+            0.0057632802054286,
+            0.018492378294467926,
+            0.01711168698966503,
+            -0.012230267748236656,
+            -0.03039306215941906,
+            0.004721079487353563,
+            -0.012105559930205345,
+            -0.008738450706005096,
+            -0.01287162210792303,
+            0.007700704038143158,
+            0.016951346769928932,
+            0.01101882103830576,
+            0.0011669083032757044,
+            -0.0052778106182813644,
+            -0.02123595029115677,
+            -0.007861042395234108,
+            -0.0176995936781168,
+            -0.028985645622015,
+            -0.0031466439832001925,
+            -0.033029742538928986,
+            0.03823183476924896,
+            0.018777426332235336,
+            0.005273356568068266,
+            -0.01982853375375271,
+            -0.012720191851258278,
+            -0.020790565758943558,
+            0.00927291251718998,
+            -0.00707716541364789,
+            0.009718297980725765,
+            -0.004614186938852072,
+            0.01726311631500721,
+            -0.006057234015315771,
+            0.00047684015589766204,
+            -0.04436033219099045,
+            0.009023496881127357,
+            0.01934751868247986,
+            0.003930521197617054,
+            -0.009878636337816715,
+            0.008880973793566227,
+            0.011606729589402676,
+            -0.021022165194153786,
+            -0.0236944742500782,
+            0.019222810864448547,
+            0.020576780661940575,
+            0.015909146517515182,
+            -0.009299635887145996,
+            -0.009299635887145996,
+            0.007130611687898636,
+            0.0007682888535782695,
+            -0.006315557286143303,
+            0.016550501808524132,
+            0.023623213171958923,
+            0.01375348400324583,
+            -0.00018177268793806434,
+            0.026206444948911667,
+            0.009174928069114685,
+            -0.012310437858104706,
+            -0.012114468030631542,
+            -0.007620534859597683,
+            0.0023694473784416914,
+            -0.01670193299651146,
+            -0.011989760212600231,
+            -0.010083513334393501,
+            0.025226598605513573,
+            0.009370896965265274,
+            0.011294960044324398,
+            -0.003090970916673541,
+            -0.006173034198582172,
+            0.005411426071077585,
+            0.009949897415935993,
+            0.0001479512866353616,
+            -0.001145752496086061,
+            0.007219688501209021,
+            -0.038766298443078995,
+            -0.028682785108685493,
+            0.0014196642441675067,
+            -0.002750251442193985,
+            0.0033515209797769785,
+            0.03506069630384445,
+            0.027596045285463333,
+            -0.007014811504632235,
+            -0.01918718032538891,
+            -0.0018227375112473965,
+            0.024282382801175117,
+            0.0001636789384065196,
+            -0.02308875136077404,
+            -0.009477789513766766,
+            0.009090305306017399,
+            0.00032874970929697156,
+            0.02927069365978241,
+            0.005700926296412945,
+            -0.008235165849328041,
+            0.019383149221539497,
+            0.10076387226581573,
+            0.007776419632136822,
+            0.006885650102049112,
+            -0.012150098569691181,
+            -0.031016601249575615,
+            0.0021033298689872026,
+            0.021699151024222374,
+            -0.019080286845564842,
+            0.04485916346311569,
+            0.011553282849490643,
+            0.013263560831546783,
+            0.018973395228385925,
+            -0.0021144645288586617,
+            -0.010974282398819923,
+            -0.016131838783621788,
+            0.030606847256422043,
+            0.0016323354793712497,
+            0.02184167318046093,
+            -0.020897457376122475,
+            0.0017414548201486468,
+            0.004591918084770441,
+            0.012800361029803753,
+            0.00606168806552887,
+            -0.005055117886513472,
+            -0.02510189078748226,
+            0.018777426332235336,
+            -0.018180610612034798,
+            0.007224142551422119,
+            -0.008858704939484596,
+            -0.022874966263771057,
+            0.008987866342067719,
+            -0.00014502844715025276,
+            -0.01244405284523964,
+            0.005513864569365978,
+            -0.014109792187809944,
+            0.0326734334230423,
+            -0.012738007120788097,
+            0.0015354643110185862,
+            0.003663290524855256,
+            -0.016390163451433182,
+            0.01202539075165987,
+            0.01746799424290657,
+            -0.01200757548213005,
+            0.030090201646089554,
+            -0.016710840165615082,
+            -0.004318006336688995,
+            0.012061022222042084,
+            -0.018002456054091454,
+            0.011214790865778923,
+            0.028486816212534904,
+            -0.009335266426205635,
+            0.005683110561221838,
+            -0.0075581809505820274,
+            0.009905358776450157,
+            0.009335266426205635,
+            0.026402413845062256,
+            0.023142196238040924,
+            -0.0005219354061409831,
+            0.004251198377460241,
+            -0.008880973793566227,
+            -0.0019708280451595783,
+            -0.0009464428294450045,
+            0.034312449395656586,
+            -0.018162794411182404,
+            -0.0043246871791779995,
+            -0.009326359257102013,
+            0.0006680772639811039,
+            -0.014724423177540302,
+            -0.002120031975209713,
+            -0.007437927182763815,
+            -0.0019351971568539739,
+            0.0076606194488704205,
+            -0.00036493720836006105,
+            0.016158562153577805,
+            -0.008257435634732246,
+            -0.004556287080049515,
+            0.004997218027710915,
+            -0.005224364344030619,
+            0.01657722517848015,
+            -0.02574324421584606,
+            0.0037612752057611942,
+            0.018456747755408287,
+            0.00869391206651926,
+            0.011508745141327381,
+            -0.01642579399049282,
+            0.0025854590348899364,
+            -0.002992986235767603,
+            -0.012577668763697147,
+            0.012577668763697147,
+            -0.01831422559916973,
+            -0.011499837040901184,
+            -0.021129056811332703,
+            -0.0297873392701149,
+            -0.002863824600353837,
+            -0.01851019449532032,
+            0.008146088570356369,
+            0.01918718032538891,
+            0.028647154569625854,
+            0.00018024168093688786,
+            -0.004970495123416185,
+            -0.01743236370384693,
+            0.015962593257427216,
+            0.045037318021059036,
+            -0.016292177140712738,
+            -0.02520878240466118,
+            -0.009914266876876354,
+            0.0127290990203619,
+            -0.008622650988399982,
+            -0.026046106591820717,
+            -0.003943882882595062,
+            0.00577218784019351,
+            -0.02911035344004631,
+            0.009379805065691471,
+            0.0005712060956284404,
+            0.024442721158266068,
+            0.004672087263315916,
+            0.020665857940912247,
+            -0.017521440982818604,
+            -0.0019708280451595783,
+            0.025476014241576195,
+            -0.029520107433199883,
+            0.0045874640345573425,
+            0.01802917942404747,
+            -0.01346843782812357,
+            0.010769405402243137,
+            0.030820632353425026,
+            0.025226598605513573,
+            -0.01963256485760212,
+            0.024335827678442,
+            0.014626438729465008,
+            -0.008720635436475277,
+            0.003284713253378868,
+            0.011660175397992134,
+            0.012622206471860409,
+            0.006698588375002146,
+            -0.0034584132954478264,
+            -0.005861264653503895,
+            0.009045766666531563,
+            0.031337279826402664,
+            0.0016623989213258028,
+            -0.012657837942242622,
+            0.028415553271770477,
+            0.02497718296945095,
+            -0.005429241340607405,
+            0.008426681160926819,
+            -0.02039862610399723,
+            -0.016443608328700066,
+            0.01730765588581562,
+            0.022465212270617485,
+            0.005910256877541542,
+            0.0032401748467236757,
+            -0.011366221122443676,
+            0.022839335724711418,
+            -0.008377688936889172,
+            0.01154437568038702,
+            -0.027168476954102516,
+            -0.002518651308491826,
+            -0.0036900134291499853,
+            0.02068367227911949,
+            -0.0008106004097498953,
+            -0.005032849032431841,
+            0.019062472507357597,
+            -0.01228371448814869,
+            -0.019775087013840675,
+            -0.006462534423917532,
+            -0.018118256703019142,
+            0.017637239769101143,
+            -0.004774525761604309,
+            -0.02030954882502556,
+            0.007549273315817118,
+            0.0005678656743839383,
+            -0.009041312150657177,
+            0.006801026873290539,
+            0.004144306294620037,
+            0.008057012222707272,
+            -0.02264336682856083,
+            0.010751590132713318,
+            -0.005464872345328331,
+            0.007901127450168133,
+            -0.008382143452763557,
+            0.014368114992976189,
+            -0.006738672964274883,
+            0.0015599604230374098,
+            0.0006012695375829935,
+            -0.006208664737641811,
+            0.000636900367680937,
+            0.013210115022957325,
+            0.0032624441664665937,
+            -0.016550501808524132,
+            0.015312331728637218,
+            -0.008662735112011433,
+            -0.004925956483930349,
+            -0.005028394982218742,
+            -0.004106448497623205,
+            -0.0011357313487678766,
+            0.012738007120788097,
+            0.00865382794290781,
+            -0.031123492866754532,
+            -0.006689680740237236,
+            -0.024959366768598557,
+            -0.009326359257102013,
+            0.010145867243409157,
+            0.012052114121615887,
+            -0.020576780661940575,
+            0.009001228027045727,
+            -0.027079399675130844,
+            -0.01226589921861887,
+            0.0269012451171875,
+            0.023961706086993217,
+            -0.02068367227911949,
+            0.012738007120788097,
+            -0.014804592356085777,
+            -0.02014921046793461,
+            0.015009469352662563,
+            0.021467549726366997,
+            0.01224808394908905,
+            0.005028394982218742,
+            -0.04400402307510376,
+            -0.010920836590230465,
+            0.019864164292812347,
+            -0.03140854090452194,
+            -0.009504512883722782,
+            0.027417892590165138,
+            -0.006827749777585268,
+            -0.01726311631500721,
+            -0.003516313387081027,
+            0.03522103279829025,
+            -0.02123595029115677,
+            0.015526115894317627,
+            0.011428575031459332,
+            -0.02152099646627903,
+            0.014760053716599941,
+            -0.012853806838393211,
+            0.008618197403848171,
+            0.0019296299433335662,
+            -0.022536473348736763,
+            -0.0006502618780359626,
+            0.018011363223195076,
+            -0.0005444829585030675,
+            -0.004050775431096554,
+            0.011455298401415348,
+            0.016639579087495804,
+            -0.018421117216348648,
+            0.013922730460762978,
+            0.006088410969823599,
+            9.90285407169722e-05,
+            0.016283269971609116,
+            -0.02052333392202854,
+            -0.01810934767127037,
+            0.01979290321469307,
+            -0.004179936833679676,
+            -0.02421111986041069,
+            0.02344505861401558,
+            -0.02376573532819748,
+            -0.020950904116034508,
+            -0.02340942807495594,
+            0.00032652277150191367,
+            0.00849348958581686,
+            -0.01734328642487526,
+            -0.026562752202153206,
+            0.007233050186187029,
+            0.01950785703957081,
+            0.04689011722803116,
+            0.018038086593151093,
+            -0.0023048666771501303,
+            0.023819182068109512,
+            -0.016595039516687393,
+            0.013263560831546783,
+            0.012105559930205345,
+            -0.0036543826572597027,
+            -0.015757715329527855,
+            0.0025743243750184774,
+            0.009602497331798077,
+            0.007152881007641554,
+            0.0030263899825513363,
+            -0.022251427173614502,
+            -0.02120031975209713,
+            -0.04101103916764259,
+            0.00957577396184206,
+            -0.007771965581923723,
+            0.0008968937327153981,
+            0.05790003016591072,
+            0.00126711989287287,
+            -0.033350419253110886,
+            0.01754816249012947,
+            0.00047767526120878756,
+            0.0007610513712279499,
+            -0.011633452028036118,
+            0.002986305393278599,
+            -0.02389044314622879,
+            -0.013949453830718994,
+            -0.0353991873562336,
+            -0.0036877866368740797,
+            0.01027948223054409,
+            0.005162010435014963,
+            0.005700926296412945,
+            -0.02805924601852894,
+            -0.002245853189378977,
+            -0.025867952033877373,
+            0.022304873913526535,
+            0.006475895643234253,
+            -0.0001588075392646715,
+            0.0023649935610592365,
+            0.027400076389312744,
+            -0.008622650988399982,
+            -0.01854582503437996,
+            -0.020790565758943558,
+            0.0011902910191565752,
+            -0.004248971585184336,
+            0.001448614289984107,
+            0.0009458860731683671,
+            -0.025832321494817734,
+            0.009486697614192963,
+            0.00028532466967590153,
+            -0.012595484033226967,
+            0.014100884087383747,
+            -0.013352638110518456,
+            -0.007001449819654226,
+            0.009157112799584866,
+            0.023160012438893318,
+            -0.027524784207344055,
+            -0.008506850339472294,
+            -0.024870289489626884,
+            0.009985528886318207,
+            0.012675653211772442,
+            0.0029484478291124105,
+            -0.019775087013840675,
+            -0.012925068847835064,
+            -0.02574324421584606,
+            -0.009851912967860699,
+            -0.02059459500014782,
+            -0.008012473583221436,
+            0.005041756667196751,
+            0.01657722517848015,
+            0.03244183212518692,
+            0.016799917444586754,
+            -0.005157556850463152,
+            0.0017548163887113333,
+            -0.002420666627585888,
+            0.008751812390983105,
+            -0.020060133188962936,
+            2.0077113731531426e-05,
+            -0.0159358698874712,
+            0.017957916483283043,
+            0.00999443605542183,
+            -0.013887099921703339,
+            -0.017761947587132454,
+            -0.02276807464659214,
+            0.019525671377778053,
+            -0.022429581731557846,
+            -0.017886655405163765,
+            0.01918718032538891,
+            -0.011036636307835579,
+            0.006409088149666786,
+            -0.013691130094230175,
+            0.037198543548583984,
+            0.010021159425377846,
+            0.025476014241576195,
+            -0.016612855717539787,
+            -0.005567310843616724,
+            -0.0272219218313694,
+            -0.004400402307510376,
+            -0.01682663895189762,
+            -0.00434695603325963,
+            -0.0012337160296738148,
+            0.009753928519785404,
+            -0.02526222914457321,
+            -0.005567310843616724,
+            0.02321345917880535,
+            -0.027560414746403694,
+            -0.0022124492097646,
+            -0.0021757050417363644,
+            0.006337826605886221,
+            0.007121704053133726,
+            0.0014397065388038754,
+            -0.002219130052253604,
+            -0.022180166095495224,
+            -0.008448950946331024,
+            -0.028807492926716805,
+            0.022091088816523552,
+            -0.010315113700926304,
+            -0.014653162099421024,
+            0.0001746743801049888,
+            0.021182503551244736,
+            0.0046899025328457355,
+            -0.010172589682042599,
+            0.0017403413075953722,
+            -0.007598265539854765,
+            0.0021645703818649054,
+            0.0038147212471812963,
+            0.012096652761101723,
+            -0.0020988760516047478,
+            0.014083068817853928,
+            0.02020265720784664,
+            -0.008560297079384327,
+            -0.0006975840078666806,
+            -0.0058122724294662476,
+            0.0037790904752910137,
+            0.011375129222869873,
+            0.01037746760994196,
+            0.016203101724386215,
+            0.003641021205112338,
+            -0.009486697614192963,
+            0.01670193299651146,
+            -0.014083068817853928,
+            0.02184167318046093,
+            -0.001927402918227017,
+            -0.01943659596145153,
+            0.002552055288106203,
+            -0.047923412173986435,
+            0.0009069149382412434,
+            -0.012310437858104706,
+            -0.022750258445739746,
+            0.006293287966400385,
+            -0.011838329024612904,
+            -0.008190627209842205,
+            0.019098103046417236,
+            -0.009103666059672832,
+            -0.02219798043370247,
+            0.011571098119020462,
+            0.0027591590769588947,
+            -0.009036858566105366,
+            0.006613965146243572,
+            -0.0016679662512615323,
+            0.008658281527459621,
+            0.004482798743993044,
+            0.005366887431591749,
+            0.001175816054455936,
+            -0.01565973088145256,
+            -0.018563641235232353,
+            0.020095765590667725,
+            0.008333150297403336,
+            -0.0016278816619887948,
+            0.010003344155848026,
+            0.002652266761288047,
+            0.011428575031459332,
+            -0.003694467479363084,
+            -0.004810156300663948,
+            0.00019415996212046593,
+            -0.014679884538054466,
+            0.014145422726869583,
+            -0.013085407204926014,
+            -0.032228048890829086,
+            0.013664407655596733,
+            0.006008241791278124,
+            0.00681438809260726,
+            -0.008119366131722927,
+            -0.011891775764524937,
+            -0.007161788642406464,
+            -0.012078837491571903,
+            0.007455742452293634,
+            0.015294515527784824,
+            -0.0038147212471812963,
+            -0.0008879860397428274,
+            -0.01418996136635542,
+            0.007575996220111847,
+            0.020184841006994247,
+            0.029965493828058243,
+            -0.02488810569047928,
+            0.00724641140550375,
+            -0.005344618111848831,
+            -0.011873960494995117,
+            -0.020291734486818314,
+            0.024264566600322723,
+            0.01457299292087555,
+            0.01626545563340187,
+            0.004206659737974405,
+            0.0036165250930935144,
+            0.000802249473053962,
+            -0.019365333020687103,
+            0.003191182389855385,
+            0.008137181401252747,
+            -0.013317007571458817,
+            -0.0020688127260655165,
+            0.02478121407330036,
+            -0.013780207373201847,
+            -0.0013484027003869414,
+            0.001965260598808527,
+            0.024032967165112495,
+            -0.0138514693826437,
+            0.001873956760391593,
+            0.004729987122118473,
+            0.0006508186343125999,
+            0.005522772204130888,
+            0.013219022192060947,
+            0.07496717572212219,
+            0.019579118117690086,
+            0.005393610801547766,
+            0.007446834817528725,
+            -0.006311103235930204,
+            -0.006008241791278124,
+            -0.009264005348086357,
+            -0.011553282849490643,
+            0.002538693603128195,
+            -0.0017325470689684153,
+            -0.02308875136077404,
+            -0.002113351132720709,
+            -0.005825634114444256,
+            0.00045429254532791674,
+            -0.0012982968473806977,
+            0.01734328642487526,
+            -0.012666745111346245,
+            0.013913823291659355,
+            0.03204989433288574,
+            0.007188511546701193,
+            -0.021770412102341652,
+            0.013504069298505783,
+            0.0025854590348899364,
+            -0.006484803277999163,
+            -0.003402740228921175,
+            -0.006609511096030474,
+            0.006996996235102415,
+            0.011669083498418331,
+            -0.010600159876048565,
+            0.03851688280701637,
+            0.0013951680157333612,
+            0.0014764508232474327,
+            -0.01298742275685072,
+            -0.008008019998669624,
+            -0.0038035865873098373,
+            0.0024852475617080927,
+            0.012354975566267967,
+            -0.0010978736681863666,
+            -0.01851019449532032,
+            0.003068701596930623,
+            0.004166575148701668,
+            0.021182503551244736,
+            0.0044182175770401955,
+            0.002165683778002858,
+            -0.009246189147233963,
+            0.01069814432412386,
+            -0.00923728197813034,
+            -0.004021825268864632,
+            0.009860821068286896,
+            -0.0033760173246264458,
+            -0.010457636788487434,
+            -0.016452517360448837,
+            -0.0003234607574995607,
+            0.008742904290556908,
+            -0.013691130094230175,
+            -0.014742238447070122,
+            0.022429581731557846,
+            -0.0014385931426659226,
+            0.023035304620862007,
+            -0.005406972020864487,
+            0.008622650988399982,
+            0.011473113670945168,
+            -0.0012159006437286735,
+            -0.00782986544072628,
+            0.019703825935721397,
+            -0.00027196313021704555,
+            0.004277921747416258,
+            -0.014679884538054466,
+            0.009531236253678799,
+            -0.01114352885633707,
+            -0.0023605397436767817,
+            0.016363440081477165,
+            0.0001629830221645534,
+            0.007037080824375153,
+            -0.002538693603128195,
+            -0.009032404981553555,
+            -0.031782664358615875,
+            -0.02353413589298725,
+            -0.0022057686001062393,
+            0.013094314374029636,
+            -0.008578112348914146,
+            0.0022592146415263414,
+            -0.0027480246499180794,
+            -0.004053002223372459,
+            -0.016194192692637444,
+            0.006507072597742081,
+            5.518596663023345e-05,
+            0.015971500426530838,
+            0.0011435255873948336,
+            -0.015276700258255005,
+            0.003774636657908559,
+            -0.0031399631407111883,
+            0.018937764689326286,
+            -0.013922730460762978,
+            -0.0029573554638773203,
+            0.0236944742500782,
+            -0.01618528552353382,
+            -0.0011691352119669318,
+            0.003202317049726844,
+            -0.013682222925126553,
+            0.031016601249575615,
+            0.013094314374029636,
+            0.023943889886140823,
+            0.015401408076286316,
+            0.01212337613105774,
+            0.001612293184734881,
+            0.01630108617246151,
+            0.013619869016110897,
+            0.012925068847835064,
+            -0.015989316627383232,
+            -0.007348849903792143,
+            -0.017280932515859604,
+            -0.004551833029836416,
+            0.012782545760273933,
+            -0.017031516879796982,
+            0.0053579797968268394,
+            -0.01559737790375948,
+            -0.013014145195484161,
+            -0.0167375635355711,
+            -0.006079503335058689,
+            0.005148648750036955,
+            -0.02216234989464283,
+            -0.012942884117364883,
+            -0.012862714938819408,
+            -0.009860821068286896,
+            -0.01879524067044258,
+            -0.011339498683810234,
+            -0.012016483582556248,
+            -0.019204994663596153,
+            -0.035559527575969696,
+            -0.014626438729465008,
+            -0.0037456867285072803,
+            -0.011615636758506298,
+            -0.0038414443843066692,
+            -0.03634340316057205,
+            0.012916160747408867,
+            6.475478585343808e-05,
+            -0.004371452145278454,
+            0.003963925410062075,
+            0.013397176750004292,
+            -0.00856920424848795,
+            -0.03237057104706764,
+            -0.028077062219381332,
+            -0.01972164213657379,
+            -0.00024579677847214043,
+            -0.00031984198722057045,
+            0.011259329505264759,
+            0.0004286829207558185,
+            0.017058240249753,
+            -0.01053780596703291,
+            0.017672870308160782,
+            -0.03586238995194435,
+            -0.03196081891655922,
+            -0.003759048180654645,
+            -0.008729543536901474,
+            -0.012043206952512264,
+            0.006409088149666786,
+            0.007037080824375153,
+            -0.0055851261131465435,
+            -0.012354975566267967,
+            -0.007117250002920628,
+            -0.016407977789640427,
+            -0.008925512433052063,
+            0.009210558608174324,
+            0.003333705710247159,
+            0.012809268198907375,
+            0.0150540079921484,
+            0.00868945848196745,
+            -0.01657722517848015,
+            0.01975727267563343,
+            -0.005362433847039938,
+            -0.003008574713021517,
+            0.013512976467609406,
+            0.010110235773026943,
+            0.0167375635355711,
+            0.01130386721342802,
+            0.004306871443986893,
+            -0.014964930713176727,
+            0.02911035344004631,
+            0.015846792608499527,
+            0.009174928069114685,
+            0.01543703954666853,
+            0.015151992440223694,
+            0.009081397205591202,
+            -0.01202539075165987,
+            -0.0007176263607107103,
+            -0.017655055969953537,
+            0.0008055898360908031,
+            0.007353303954005241,
+            -0.010992098599672318,
+            -0.006070595700293779,
+            0.033350419253110886,
+            0.0022703493013978004,
+            -0.015730993822216988,
+            0.013138853013515472,
+            -0.011250421404838562,
+            -0.006720857694745064,
+            -0.011214790865778923,
+            -0.0118472371250391,
+            -0.003019709372892976,
+            0.015303423628211021,
+            0.011366221122443676,
+            -0.02103998139500618,
+            0.011437483131885529,
+            0.01303196046501398,
+            -0.0057098339311778545,
+            -0.005335710477083921,
+            0.014528454281389713,
+            0.01870616339147091,
+            0.01433248445391655,
+            -0.03628995642066002,
+            -0.0023271357640624046,
+            0.012078837491571903,
+            -0.026527121663093567,
+            -0.019383149221539497,
+            0.013940545730292797,
+            0.0012548718368634582,
+            -0.019044656306505203,
+            0.01642579399049282,
+            -0.002663401421159506,
+            -0.022874966263771057,
+            -0.008542481809854507,
+            0.0034851364325731993,
+            0.019935425370931625,
+            0.021129056811332703,
+            -0.012542037293314934,
+            0.008217350579798222,
+            -0.014697699807584286,
+            0.004286829382181168,
+            -0.023355981335043907,
+            -0.0037456867285072803,
+            -0.008560297079384327,
+            -0.0021345068234950304,
+            0.017672870308160782,
+            0.008729543536901474,
+            0.02312438189983368,
+            0.019971057772636414,
+            0.0006380138220265508,
+            -0.016167471185326576,
+            -0.003306982573121786,
+            0.011232606135308743,
+            -0.008208442479372025,
+            -0.0002967376494780183,
+            -0.001585570047609508,
+            0.012907253578305244,
+            0.005393610801547766,
+            0.030642477795481682,
+            -0.002772520761936903,
+            0.0012994103599339724,
+            -3.0011284252395853e-05,
+            0.002788109239190817,
+            0.0012392833596095443,
+            -0.005972610786557198,
+            -0.0018917721463367343,
+            0.005153102800250053,
+            0.02552945911884308,
+            0.004939318168908358,
+            -0.0069123730063438416,
+            0.007976843044161797,
+            -0.0022169030271470547,
+            -0.011642360128462315,
+            -0.0038102674297988415,
+            -0.014840223826467991,
+            -0.01746799424290657,
+            0.01979290321469307,
+            0.0277029387652874,
+            -0.013762392103672028,
+            -0.026954691857099533,
+            -0.00011412987078074366,
+            0.04044094309210777,
+            0.002268122276291251,
+            0.0028727322351187468,
+            0.022946227341890335,
+            -0.0020465434063225985,
+            0.018563641235232353,
+            0.021823858842253685,
+            0.011027729138731956,
+            0.009442158974707127,
+            0.03221023455262184,
+            0.0017158451955765486,
+            -0.004694356583058834,
+            -0.0030286170076578856,
+            -0.0025320129934698343,
+            -0.012497498653829098,
+            0.0009325245628133416,
+            0.002859370782971382,
+            -0.02068367227911949,
+            0.00792785082012415,
+            -0.01959693431854248,
+            -0.0006819955888204277,
+            -0.003652155864983797,
+            0.0065382495522499084,
+            -0.025476014241576195,
+            -0.010047882795333862,
+            0.009967712685465813,
+            -0.01043982058763504,
+            0.0023739011958241463,
+            -0.01915154792368412,
+            -0.004184390883892775,
+            0.022055458277463913,
+            -0.004451621789485216,
+            0.013887099921703339,
+            -0.00841331947594881,
+            -0.0011318342294543982,
+            -0.008404412306845188,
+            -0.0066451421007514,
+            0.011196975596249104,
+            -0.0058122724294662476,
+            0.001120699685998261,
+            -0.010876297950744629,
+            0.008065919391810894,
+            -0.005634118337184191,
+            0.004805702716112137,
+            0.023712290450930595,
+            -0.0030442054849117994,
+            -0.0068901036866009235,
+            0.007424565497785807,
+            0.010038974694907665,
+            -0.022411765530705452,
+            0.02702595293521881,
+            -0.011348405852913857,
+            0.02059459500014782,
+            0.00296626309864223,
+            -0.019703825935721397,
+            -0.0027168476954102516,
+            -0.0010488813277333975,
+            0.010867390781641006,
+            -0.006649595685303211,
+            0.002970716916024685,
+            -0.01831422559916973,
+            0.015276700258255005,
+            0.005945887882262468,
+            -0.015330146998167038,
+            0.01979290321469307,
+            -0.006988088600337505,
+            0.0019663742277771235,
+            -0.017485808581113815,
+            0.00578109547495842,
+            0.007130611687898636,
+            -0.013664407655596733,
+            -0.003739005886018276,
+            -0.019703825935721397,
+            -0.0005723195499740541,
+            -0.01565973088145256,
+            0.0036098442506045103,
+            -0.009718297980725765,
+            -0.013646592386066914,
+            -0.036913495510816574,
+            -0.0001862265489762649,
+            0.004685448482632637,
+            -0.007197419181466103,
+            -0.001399621949531138,
+            0.00012637795589398593,
+            0.013958360999822617,
+            0.009290727786719799,
+            -0.0039572445675730705,
+            0.004979402758181095,
+            0.034348081797361374,
+            -0.04421781003475189,
+            -0.01138403732329607,
+            -0.00033654391882009804,
+            0.004008463583886623,
+            -0.020737119019031525,
+            -0.008275250904262066,
+            0.013121037743985653,
+            -0.0007081619114615023,
+            0.016274362802505493,
+            -0.016033854335546494,
+            0.010947559960186481,
+            0.003079836256802082,
+            0.008177265524864197,
+            -0.007135065272450447,
+            -0.0053579797968268394,
+            -0.012034298852086067,
+            -0.000576773367356509,
+            -0.001016590977087617,
+            -0.02827303111553192,
+            0.01734328642487526,
+            0.0016033854335546494,
+            -0.0056296647526323795,
+            0.0069480035454034805,
+            -0.02453179843723774,
+            -0.009148204699158669,
+            -0.012292621657252312,
+            0.023658843711018562,
+            -0.01738782413303852,
+            -0.00026625662576407194,
+            -0.004698810167610645,
+            -0.006444718688726425,
+            0.014724423177540302,
+            -0.0037234174087643623,
+            0.007322126999497414,
+            0.00037635021726600826,
+            -0.011339498683810234,
+            -0.002014253055676818,
+            0.0024785667192190886,
+            -0.010190405882894993,
+            0.031693585216999054,
+            -0.005959249567240477,
+            -0.0027702937368303537,
+            0.007433473132550716,
+            0.0010666967136785388,
+            -0.002318228129297495,
+            0.010270575061440468,
+            0.0017882202519103885,
+            0.0039572445675730705,
+            0.008894335478544235,
+            -0.0036989212967455387,
+            -0.006021603476256132,
+            0.016007130965590477,
+            0.020541150122880936,
+            0.005616303067654371,
+            -0.016033854335546494,
+            -0.011116806417703629,
+            0.011758159846067429,
+            0.0015733219916000962,
+            0.0006396839744411409,
+            -0.008057012222707272,
+            0.006324464920908213,
+            -0.002256987849250436,
+            0.012043206952512264,
+            0.011838329024612904,
+            0.017298748716711998,
+            0.012898345477879047,
+            -0.005723195616155863,
+            0.03144416958093643,
+            0.01614074781537056,
+            0.0009007908520288765,
+            0.015134177170693874,
+            -0.04368334636092186,
+            -0.01027948223054409,
+            -0.012078837491571903,
+            0.01754816249012947,
+            -0.014751146547496319,
+            -0.016488147899508476,
+            -0.013183391653001308,
+            0.010653605684638023,
+            -0.0032446286641061306,
+            0.005473779980093241,
+            0.0251375213265419,
+            -0.013414992019534111,
+            0.004393721465021372,
+            -0.002739116782322526,
+            0.023623213171958923,
+            0.0030263899825513363,
+            0.00907694362103939,
+            0.007495827041566372,
+            0.005104110576212406,
+            -0.0027057130355387926,
+            0.01963256485760212,
+            -0.009219466708600521,
+            0.036485929042100906,
+            -0.01053780596703291,
+            -0.009317451156675816,
+            -0.01582006923854351,
+            -0.017637239769101143,
+            -0.012354975566267967,
+            0.004752256441861391,
+            -0.000690346525516361,
+            0.0012570987455546856,
+            -0.00821289699524641,
+            0.013637684285640717,
+            0.03140854090452194,
+            -0.0013417218578979373,
+            0.015160900540649891,
+            0.007856588810682297,
+            -0.0006513753323815763,
+            0.01543703954666853,
+            0.003883755998685956,
+            -0.004489479120820761,
+            0.004729987122118473,
+            0.024513982236385345,
+            0.0004924286040477455,
+            0.00767843471840024,
+            -0.0010744909523054957,
+            -0.009602497331798077,
+            -0.01156219094991684,
+            0.010831759311258793,
+            0.027115030214190483,
+            0.017334379255771637,
+            0.001331700710579753,
+            -0.0118472371250391,
+            -0.00017815394676290452,
+            -0.0009375351364724338,
+            0.016791008412837982,
+            -0.010965375229716301,
+            -0.006689680740237236,
+            0.013121037743985653,
+            -0.009085850790143013,
+            0.007611627224832773,
+            -0.01662176288664341,
+            0.0013907141983509064,
+            0.00722859613597393,
+            0.014270130544900894,
+            -0.0045206560753285885,
+            0.006070595700293779,
+            -0.00707716541364789,
+            0.010555621236562729,
+            -0.0127290990203619,
+            0.011277144774794579,
+            -0.005723195616155863,
+            0.00046709735761396587,
+            0.028807492926716805,
+            -0.006475895643234253,
+            0.008195081725716591,
+            0.00011621761223068461,
+            -0.0034717749804258347,
+            -0.017494717612862587,
+            0.025761060416698456,
+            0.004084179177880287,
+            0.005914710927754641,
+            0.00020014481560792774,
+            0.010288390330970287,
+            -0.008301973342895508,
+            -0.001823851023800671,
+            0.022304873913526535,
+            0.011927406303584576,
+            0.0038703943137079477,
+            -0.0005873512709513307,
+            0.008003565482795238,
+            -0.012221360579133034,
+            0.015134177170693874,
+            0.01899120956659317,
+            0.008644919842481613,
+            0.0036543826572597027,
+            0.014599715359508991,
+            -0.013094314374029636,
+            0.008466766215860844,
+            -0.009072489105165005,
+            0.029163800179958344,
+            0.012640022672712803,
+            -0.016523778438568115,
+            0.019294071942567825,
+            0.030767185613512993,
+            -0.004930410534143448,
+            -0.010430913418531418,
+            0.0015844566514715552,
+            -0.017414547502994537,
+            -0.0021567761432379484,
+            -0.014920393005013466,
+            -0.014768961817026138,
+            0.005126379895955324,
+            -0.015312331728637218,
+            0.02497718296945095,
+            -0.017280932515859604,
+            -0.031657956540584564,
+            -0.004309098701924086,
+            -0.00825298111885786,
+            -0.02818395383656025,
+            -0.008301973342895508,
+            0.006190849468111992,
+            0.014849130995571613,
+            -0.0019930971320718527,
+            -0.0144928228110075,
+            0.013664407655596733,
+            0.009085850790143013,
+            0.01359314564615488,
+            -0.017316563054919243,
+            -0.01606057770550251,
+            -0.015615193173289299,
+            0.017316563054919243,
+            -0.010956467129290104,
+            -0.016470331698656082,
+            0.014270130544900894,
+            0.003759048180654645,
+            0.005464872345328331,
+            0.0017503624549135566,
+            -0.014519546180963516,
+            -0.016354531049728394,
+            -0.027845460921525955,
+            0.010626882314682007,
+            -0.0027970168739557266,
+            -0.010974282398819923,
+            0.0029194976668804884,
+            0.011437483131885529,
+            0.00939762033522129,
+            0.005749918520450592,
+            -0.018242964521050453,
+            0.011268236674368382,
+            0.01311213057488203,
+            -0.018528010696172714,
+            -0.007429019547998905,
+            0.005451510660350323,
+            -0.009727205149829388,
+            0.03798241913318634,
+            0.01345062255859375,
+            -0.015205439180135727,
+            -0.0030397516675293446,
+            0.004930410534143448,
+            0.01686227135360241,
+            0.010056789964437485,
+            -0.018848687410354614,
+            -0.001327246893197298,
+            0.021895119920372963,
+            -0.004549606237560511,
+            0.004366998560726643,
+            -3.483918590063695e-06,
+            0.042614422738552094,
+            0.0033426133450120687,
+            -0.0056875646114349365,
+            -0.013138853013515472,
+            0.002210222417488694,
+            0.00997662078589201,
+            0.013023053295910358,
+            0.012239175848662853,
+            -0.007326581049710512,
+            0.0025008360389620066,
+            0.02020265720784664,
+            -0.010395282879471779,
+            0.0020955358631908894,
+            0.0030352978501468897,
+            0.005936980247497559,
+            -0.016880085691809654,
+            0.026242075487971306,
+            -0.0009653716697357595,
+            0.00692573469132185,
+            0.0004620867839548737,
+            -0.009923174977302551,
+            0.014670977368950844,
+            0.024122044444084167,
+            0.0008417773642577231,
+            0.015624100342392921,
+            -0.012435144744813442,
+            0.01831422559916973,
+            0.007901127450168133,
+            0.00782986544072628,
+            -0.015330146998167038,
+            -0.010092420503497124,
+            0.006952457595616579,
+            -0.0028816401027143,
+            -0.002538693603128195,
+            -0.0039839674718678,
+            -0.012372790835797787,
+            -0.011936314404010773,
+            -0.021129056811332703,
+            0.004373679403215647,
+            0.00836878176778555,
+            -0.006132949609309435,
+            0.006850019097328186,
+            0.004309098701924086,
+            -0.0033671094570308924,
+            0.0005984858726151288,
+            -0.013682222925126553,
+            -0.008707273751497269,
+            0.0033470671623945236,
+            0.002696805400773883,
+            0.004191071726381779,
+            -0.007856588810682297,
+            0.020184841006994247,
+            0.006609511096030474,
+            -0.006110680289566517,
+            0.007874404080212116,
+            -0.019062472507357597,
+            -0.0028660516254603863,
+            -0.012105559930205345,
+            0.00011127662583021447,
+            0.0042200214229524136,
+            0.01638125441968441,
+            0.0031243746634572744,
+            -0.023747920989990234,
+            0.0068322038277983665,
+            -0.01202539075165987,
+            0.008965597487986088,
+            -0.003741232678294182,
+            -0.0005160897271707654,
+            -0.012693468481302261,
+            0.01594477705657482,
+            0.01535686943680048,
+            -0.011419667862355709,
+            -0.021093426272273064,
+            0.017637239769101143,
+            -0.03103441745042801,
+            0.022108905017375946,
+            0.010671420954167843,
+            0.0036855596117675304,
+            -0.019810717552900314,
+            0.017120594158768654,
+            0.012399514205753803,
+            0.01101882103830576,
+            -0.0032936211209744215,
+            0.011294960044324398,
+            0.021627888083457947,
+            -0.007941211573779583,
+            -0.0010410870891064405,
+            -9.525320820102934e-06,
+            -0.010430913418531418,
+            -0.0002900568943005055,
+            -0.007175149861723185,
+            0.014421561732888222,
+            0.00836878176778555,
+            -0.008836435154080391,
+            -0.013548607006669044,
+            0.01228371448814869,
+            -0.011482021771371365,
+            0.007117250002920628,
+            -0.005985972471535206,
+            0.007219688501209021,
+            0.004342502448707819,
+            0.01770850270986557,
+            -0.021057795733213425,
+            0.0005670305690728128,
+            0.005237726029008627,
+            -0.022946227341890335,
+            0.007041534408926964,
+            -0.024585243314504623,
+            0.014902577735483646,
+            -0.025351306423544884,
+            -0.005549495108425617,
+            0.02046988718211651,
+            0.014599715359508991,
+            0.0033092093653976917,
+            -0.018011363223195076,
+            -0.007059350144118071,
+            -0.013619869016110897,
+            0.002101103076711297,
+            -0.022429581731557846,
+            -0.013388268649578094,
+            -0.015303423628211021,
+            0.009531236253678799,
+            0.011027729138731956,
+            0.0024028513580560684,
+            -0.010582344606518745,
+            0.007535911630839109,
+            -0.02168133482336998,
+            -0.010617975145578384,
+            -0.014653162099421024,
+            -0.017681779339909554,
+            0.001418550731614232,
+            0.02661619894206524,
+            -0.0027413438074290752,
+            0.01879524067044258,
+            0.011954129673540592,
+            0.01471551600843668,
+            -0.0031733671203255653,
+            -0.010938651859760284,
+            -0.00923728197813034,
+            -0.0027925630565732718,
+            -0.003554171184077859,
+            -0.018777426332235336,
+            -0.012559852562844753,
+            0.00489923357963562,
+            -0.0019853028934448957,
+            0.003981740679591894,
+            0.007874404080212116,
+            0.00841331947594881,
+            -0.01100100576877594,
+            -0.010751590132713318,
+            0.016630670055747032,
+            -0.025671983137726784,
+            -0.017405640333890915,
+            0.018278595060110092,
+            -0.011232606135308743,
+            -0.0012348295422270894,
+            0.019864164292812347,
+            -0.010956467129290104,
+            -0.007491373457014561,
+            0.010493267327547073,
+            -0.003865940496325493,
+            0.010528897866606712,
+            0.022180166095495224,
+            0.006484803277999163,
+            -0.016087301075458527,
+            0.008595927618443966,
+            0.010609067045152187,
+            -0.014163237996399403,
+            -0.02734662964940071,
+            -0.0026010475121438503,
+            0.004560741130262613,
+            -0.020701488479971886,
+            -0.0026945783756673336,
+            0.0004078055208083242,
+            0.00722859613597393,
+            -0.0003855362592730671,
+            0.004925956483930349,
+            -0.018100440502166748,
+            -0.007139519322663546,
+            0.003429463366046548,
+            0.00868945848196745,
+            -0.022215796634554863,
+            0.0013873737771064043,
+            -0.013227930292487144,
+            0.02216234989464283,
+            -0.003257990116253495,
+            0.007415657863020897,
+            -0.0012548718368634582,
+            -0.0074201119132339954,
+            0.008769627660512924,
+            0.012069929391145706,
+            0.005972610786557198,
+            0.03295847773551941,
+            -0.003894890658557415,
+            -0.015009469352662563,
+            0.014234500005841255,
+            -0.025885768234729767,
+            -0.0007337715360336006,
+            -0.0038325367495417595,
+            0.010626882314682007,
+            -0.011036636307835579,
+            -0.011188067495822906,
+            0.02203764207661152,
+            0.02100434899330139,
+            -0.006930188275873661,
+            -0.0002297907485626638,
+            -0.01750362478196621,
+            0.01662176288664341,
+            -0.00824852753430605,
+            0.0015833431389182806,
+            0.007976843044161797,
+            -0.023320350795984268,
+            -0.012631114572286606,
+            -0.006021603476256132,
+            0.024870289489626884,
+            -0.01934751868247986,
+            -0.016559408977627754,
+            -0.012515314854681492,
+            0.00012282878742553294,
+            -0.02709721401333809,
+            -0.0075225504115223885,
+            0.007063803728669882,
+            -0.012925068847835064,
+            -0.007553727366030216,
+            0.007954573258757591,
+            0.009477789513766766,
+            -0.0021033298689872026,
+            -0.02520878240466118,
+            -0.011188067495822906,
+            0.013210115022957325,
+            0.003939428832381964,
+            0.006190849468111992,
+            -0.004364771768450737,
+            0.006168580148369074,
+            -0.006279926281422377,
+            0.0032513095065951347,
+            0.00899231992661953,
+            -0.0016490374691784382,
+            -0.006342280190438032,
+            0.02219798043370247,
+            0.005522772204130888,
+            -0.018180610612034798,
+            0.01433248445391655,
+            0.01570427045226097,
+            -0.012185730040073395,
+            -0.008489035069942474,
+            3.824046507361345e-05,
+            -0.013931638561189175,
+            -0.0008918831590563059,
+            -0.006475895643234253,
+            0.03196081891655922,
+            -0.022874966263771057,
+            0.003857032861560583,
+            0.008685004897415638,
+            -0.0002842112153302878,
+            -0.004903687164187431,
+            0.002912817057222128,
+            0.0054826876148581505,
+            -0.0012793679488822818,
+            0.00577218784019351,
+            0.00812827330082655,
+            -0.014216684736311436,
+            -0.006337826605886221,
+            0.013691130094230175,
+            0.002291504992172122,
+            0.017806487157940865,
+            0.016452517360448837,
+            1.9311608411953785e-05,
+            -0.03313663229346275,
+            0.004193298518657684,
+            0.0007877744501456618,
+            -0.021627888083457947,
+            -0.0038169482722878456,
+            -0.006106226239353418,
+            0.00967375934123993,
+            0.010324020870029926,
+            0.01614074781537056,
+            0.0029194976668804884,
+            0.007143972907215357,
+            0.006141857244074345,
+            0.010288390330970287,
+            -0.030927523970603943,
+            -0.006444718688726425,
+            -0.008333150297403336,
+            -0.006453626789152622,
+            0.012622206471860409,
+            0.012631114572286606,
+            0.029983308166265488,
+            0.028736229985952377,
+            -0.014751146547496319,
+            -0.019204994663596153,
+            0.004242290742695332,
+            0.011027729138731956,
+            0.008640466257929802,
+            -0.007290950044989586,
+            0.006462534423917532,
+            8.504067227477208e-05,
+            -0.02526222914457321,
+            0.0017392279114574194,
+            0.01375348400324583,
+            -0.017530348151922226,
+            -0.010617975145578384,
+            -0.018652718514204025,
+            0.014626438729465008,
+            0.02116468921303749,
+            -0.00392384035512805,
+            0.007188511546701193,
+            -0.010591251775622368,
+            -0.009317451156675816,
+            -0.04500168561935425,
+            0.0053000799380242825,
+            0.009054673835635185,
+            0.022215796634554863,
+            0.001312771812081337,
+            -0.004001782741397619,
+            -6.158837641123682e-05,
+            0.007001449819654226,
+            -0.0033960596192628145,
+            0.011464206501841545,
+            0.013210115022957325,
+            -0.018759610131382942,
+            -0.016042763367295265,
+            0.004150986671447754,
+            0.0023115472868084908,
+            -0.010172589682042599,
+            0.005968157202005386,
+            0.021253764629364014,
+            0.01586460880935192,
+            -0.007241957820951939,
+            0.004431579262018204,
+            0.005625210702419281,
+            -0.014225591905415058,
+            -0.012034298852086067,
+            -0.0045429253950715065,
+            -0.011517652310431004,
+            -0.024193305522203445,
+            0.017191855236887932,
+            -0.013824746012687683,
+            0.006395726464688778,
+            0.009869728237390518,
+            0.011223698034882545,
+            0.0018450067145749927,
+            0.023177828639745712,
+            0.00811045803129673,
+            -0.008738450706005096,
+            0.011036636307835579,
+            -0.026170814409852028,
+            0.012684560380876064,
+            -0.00023368786787614226,
+            0.011259329505264759,
+            0.0001613128260942176,
+            -0.03301192447543144,
+            -0.013512976467609406,
+            0.008172811940312386,
+            -0.006622872781008482,
+            0.009896451607346535,
+            0.014626438729465008,
+            -0.008288612589240074,
+            -0.00100879673846066,
+            -0.022340504452586174,
+            0.00445830263197422,
+            -0.03147980198264122,
+            0.00011357313633197919,
+            -0.004716625437140465,
+            0.010929744690656662,
+            -0.005362433847039938,
+            -0.0240151509642601,
+            0.0026789898984134197,
+            -0.012720191851258278,
+            -0.005825634114444256,
+            0.0010616861982271075,
+            -0.0005828974535688758,
+            0.010876297950744629,
+            -0.00593252619728446,
+            0.01375348400324583,
+            -0.007348849903792143,
+            0.005215456709265709,
+            -0.0007688456098549068,
+            0.006792119238525629,
+            -0.006658503785729408,
+            -0.012907253578305244,
+            0.00238948967307806,
+            -0.004845787305384874,
+            0.021342841908335686,
+            0.010243851691484451,
+            -0.0006380138220265508,
+            0.004362544510513544,
+            0.020291734486818314,
+            0.004672087263315916,
+            0.013165576383471489,
+            0.006676319055259228,
+            0.0032000902574509382,
+            -0.015285608358681202,
+            0.009736113250255585,
+            -0.007371119223535061,
+            0.009477789513766766,
+            0.002870505442842841,
+            -0.0010310659417882562,
+            0.014599715359508991,
+            0.008364327251911163,
+            -0.009905358776450157,
+            0.011767067946493626,
+            0.018581455573439598,
+            -0.008408865891397,
+            -0.003220132552087307,
+            -0.0127290990203619,
+            0.011294960044324398,
+            0.027204107493162155,
+            0.004260106012225151,
+            -0.01831422559916973,
+            0.014145422726869583,
+            0.00010097710037371144,
+            -0.011713622137904167,
+            0.026046106591820717,
+            -0.014751146547496319,
+            -0.025689799338579178,
+            -0.013379361480474472,
+            -0.007273134775459766,
+            -0.01750362478196621,
+            0.021271580830216408,
+            -0.011473113670945168,
+            0.012212452478706837,
+            0.004193298518657684,
+            0.03119475580751896,
+            0.010689236223697662,
+            0.01212337613105774,
+            -0.003015255555510521,
+            -0.025351306423544884,
+            0.009379805065691471,
+            0.0045874640345573425,
+            -0.04044094309210777,
+            0.0005979291745461524,
+            -0.003004120895639062,
+            0.029377585276961327,
+            0.0010550053557381034,
+            -0.0014898123918101192,
+            -0.0038971174508333206,
+            -0.020380811765789986,
+            -0.015686454251408577,
+            0.007322126999497414,
+            -0.028682785108685493,
+            0.013397176750004292,
+            -0.00507738720625639,
+            -0.012150098569691181,
+            0.002214676234871149,
+            0.010831759311258793,
+            -0.019044656306505203,
+            0.01879524067044258,
+            0.015846792608499527,
+            0.025030629709362984,
+            0.01069814432412386,
+            -0.008885428309440613,
+            0.020006688311696053,
+            0.0010210447944700718,
+            -0.006591695826500654,
+            0.0020688127260655165,
+            -1.8998447558260523e-05,
+            0.0015410316409543157,
+            -0.0033671094570308924,
+            0.003015255555510521,
+            0.01598040945827961,
+            -0.01738782413303852,
+            0.019418779760599136,
+            -0.01702260971069336,
+            0.016007130965590477,
+            0.0005347402184270322,
+            0.012925068847835064,
+            4.071095827384852e-05,
+            0.0011958583490923047,
+            -0.029840786010026932,
+            -0.0009653716697357595,
+            -0.03915823623538017,
+            -0.011312775313854218,
+            -0.0016679662512615323,
+            -0.018091533333063126,
+            -0.0037679558154195547,
+            0.009032404981553555,
+            -0.01487585436552763,
+            -0.011588914319872856,
+            0.016069484874606133,
+            0.007598265539854765,
+            0.007375573273748159,
+            0.001965260598808527,
+            -8.03084549261257e-05,
+            -0.005162010435014963,
+            -0.020808380097150803,
+            -0.012951791286468506,
+            -0.025155337527394295,
+            -0.025190968066453934,
+            -0.0035742134787142277,
+            0.007584904320538044,
+            -0.004832425620406866,
+            -0.008564750663936138,
+            0.017761947587132454,
+            -0.019044656306505203,
+            -0.00040140311466529965,
+            0.004242290742695332,
+            0.008172811940312386,
+            0.013103222474455833,
+            -0.007063803728669882,
+            0.007384480908513069,
+            0.0030709286220371723,
+            -0.0013361545279622078,
+            -0.0005183166358619928,
+            0.024104228243231773,
+            -0.008836435154080391,
+            0.011009913869202137,
+            0.007722973357886076,
+            0.011775976046919823,
+            -0.006217572372406721,
+            0.009045766666531563,
+            -0.024104228243231773,
+            0.007687342818826437,
+            -0.011045544408261776,
+            -0.026794353500008583,
+            -0.007148426957428455,
+            0.02071930281817913,
+            -0.0006536022992804646,
+            0.0008662735344842076,
+            -0.0018193970900028944,
+            -0.0004406526277307421,
+            0.023676658049225807,
+            0.01042200531810522,
+            0.0012147872475907207,
+            -0.007700704038143158,
+            -0.029769523069262505,
+            0.005985972471535206,
+            -0.021271580830216408,
+            -0.021699151024222374,
+            0.03288721665740013,
+            0.010368559509515762,
+            0.0027213015127927065,
+            -0.013691130094230175,
+            0.010038974694907665,
+            0.009442158974707127,
+            6.134480645414442e-05,
+            0.014804592356085777,
+            0.003088743891566992,
+            0.001969714416190982,
+            -0.009932082146406174,
+            -0.01918718032538891,
+            0.01614074781537056,
+            0.014581900089979172,
+            -0.01559737790375948,
+            0.036877866834402084,
+            0.016773194074630737,
+            0.012809268198907375,
+            -0.010288390330970287,
+            0.007714065723121166,
+            0.016995886340737343,
+            0.020060133188962936,
+            -0.0070860730484128,
+            0.0034339171834290028,
+            -0.010742682963609695,
+            -0.00023466214770451188,
+            0.03894445300102234,
+            0.03516758978366852,
+            0.008444496430456638,
+            -0.022785888984799385,
+            0.014581900089979172,
+            -0.009335266426205635,
+            -0.01287162210792303,
+            0.010306205600500107,
+            0.01606057770550251,
+            -0.0005923618446104228,
+            0.0014686565846204758,
+            0.009451067075133324,
+            0.0076606194488704205,
+            0.017441270872950554,
+            0.005353525746613741,
+            -0.002378355246037245,
+            0.003402740228921175,
+            -0.0034339171834290028,
+            0.00965594407171011,
+            -0.016808824613690376,
+            -0.005126379895955324,
+            -0.00339828641153872,
+            -0.0016969162970781326,
+            0.005251087248325348,
+            0.009540143422782421,
+            -0.036361221224069595,
+            -0.007415657863020897,
+            -0.033350419253110886,
+            0.004021825268864632,
+            -0.01995324157178402,
+            -0.0035430365242064,
+            -0.010965375229716301,
+            -0.01598040945827961,
+            -0.026562752202153206,
+            -0.016933532431721687,
+            -0.0031266016885638237,
+            0.008903243578970432,
+            -0.003888209816068411,
+            0.003215678734704852,
+            0.003436144208535552,
+            -0.021378472447395325,
+            0.005660841707140207,
+            0.013174484483897686,
+            0.009370896965265274,
+            0.021057795733213425,
+            0.0001496214681537822,
+            0.014368114992976189,
+            -0.014822407625615597,
+            -0.04029842093586922,
+            -0.0007649484905414283,
+            0.00031009921804070473,
+            -0.005175372119992971,
+            -0.00424674479290843,
+            0.004859148990362883,
+            -0.009379805065691471,
+            0.011098990216851234,
+            -0.0021745916455984116,
+            0.0009681553347036242,
+            -0.015722084790468216,
+            -0.010707051493227482,
+            -0.023587582632899284,
+            -0.00505066430196166,
+            -0.013121037743985653,
+            -2.176470479753334e-05,
+            -0.01288943737745285,
+            -0.004952679388225079,
+            -0.005398064386099577,
+            -0.0013461756752803922,
+            -0.006426903419196606,
+            -0.002723528305068612,
+            -0.001202539075165987,
+            -0.013263560831546783,
+            -0.012559852562844753,
+            -0.0013172257458791137,
+            0.004832425620406866,
+            0.0006942436448298395,
+            -0.01098319049924612,
+            0.013414992019534111,
+            -0.014911484904587269,
+            0.0018038086127489805,
+            0.02574324421584606,
+            0.013325914740562439,
+            -0.010484359227120876,
+            0.016479238867759705,
+            0.002199087757617235,
+            0.010128051973879337,
+            -0.02152099646627903,
+            -0.015722084790468216,
+            -0.010110235773026943,
+            0.026990322396159172,
+            0.0012682334054261446,
+            -0.013557515107095242,
+            -0.007206326816231012,
+            -0.01706714741885662,
+            0.004680994898080826,
+            -0.0077407886274158955,
+            0.011375129222869873,
+            -0.020630227401852608,
+            -0.007215234450995922,
+            -0.011473113670945168,
+            -0.007531458046287298,
+            0.0065827881917357445,
+            0.03794679045677185,
+            -0.00289054773747921,
+            -0.007852135226130486,
+            0.005963703151792288,
+            0.010849575512111187,
+            0.028237400576472282,
+            0.0010516649344936013,
+            -0.0014307989040389657,
+            -0.012043206952512264,
+            0.0034717749804258347,
+            0.004551833029836416,
+            -0.01051999069750309,
+            0.017031516879796982,
+            -0.009148204699158669,
+            -0.01934751868247986,
+            -0.004500614013522863,
+            -0.00868945848196745,
+            -0.007353303954005241,
+            0.01753925532102585,
+            0.0023226819466799498,
+            0.003208997892215848,
+            0.0008540254784747958,
+            -0.007250865455716848,
+            -0.008333150297403336,
+            0.011009913869202137,
+            0.08451622724533081,
+            0.00943325087428093,
+            -0.02228705771267414,
+            0.0012526449281722307,
+            0.004943771753460169,
+            0.001479791128076613,
+            -0.014795685186982155,
+            0.06591695547103882,
+            -0.014368114992976189,
+            0.005451510660350323,
+            0.0004428795655258,
+            0.004810156300663948,
+            -0.004447167739272118,
+            0.02136065810918808,
+            0.010199313051998615,
+            -0.01244405284523964,
+            0.002025387715548277,
+            -0.03541700541973114,
+            -0.017280932515859604,
+            0.0034984981175512075,
+            -0.0044404868967831135,
+            -0.004393721465021372,
+            0.013121037743985653,
+            -0.023106565698981285,
+            -0.008163904771208763,
+            0.010956467129290104,
+            -0.012052114121615887,
+            0.011473113670945168,
+            -0.019383149221539497,
+            -0.001312771812081337,
+            -0.0020198202691972256,
+            0.0014519545948132873,
+            -0.009727205149829388,
+            0.0036766519770026207,
+            -0.0027123938780277967,
+            0.0010705938329920173,
+            0.009477789513766766,
+            0.0077853272669017315,
+            -0.003741232678294182,
+            0.017877748236060143,
+            0.013441715389490128,
+            0.0035563979763537645,
+            0.017485808581113815,
+            0.0027480246499180794,
+            0.007651711814105511,
+            0.0003485136548988521,
+            0.01785993203520775,
+            -0.014483915641903877,
+            0.021022165194153786,
+            -0.01834985613822937,
+            -0.02059459500014782,
+            -0.013691130094230175,
+            0.009063581936061382,
+            -0.015561746433377266,
+            0.001760383602231741,
+            -0.011571098119020462,
+            0.029858600348234177,
+            0.010885206051170826,
+            -0.003088743891566992,
+            0.004380360245704651,
+            -0.007045988459140062,
+            0.03930075839161873,
+            0.007856588810682297,
+            0.033225711435079575,
+            0.003155551617965102,
+            0.017886655405163765,
+            -0.002788109239190817,
+            0.004001782741397619,
+            -0.013254653662443161,
+            -0.005821180064231157,
+            0.018278595060110092,
+            -0.009905358776450157,
+            0.019062472507357597,
+            0.004863602574914694,
+            -0.014439377002418041,
+            0.01770850270986557,
+            -0.015410316176712513,
+            -0.02292841300368309,
+            0.013842561282217503,
+            -0.02408641390502453,
+            0.0002925621811300516,
+            0.006230934057384729,
+            -0.0036276597529649734,
+            -0.01051999069750309,
+            0.021752595901489258,
+            -0.0005116358515806496,
+            -0.006907918956130743,
+            -0.010644698515534401,
+            -0.013245745562016964,
+            -0.014279038645327091,
+            0.00895668938755989,
+            -0.004574102349579334,
+            -0.01298742275685072,
+            0.011009913869202137,
+            -0.02526222914457321,
+            0.012862714938819408,
+            -0.012043206952512264,
+            0.023017490282654762,
+            -0.00795011967420578,
+            0.02075493521988392,
+            0.0022825973574072123,
+            -0.0002908919705078006,
+            0.0019853028934448957,
+            0.008863158524036407,
+            -0.003224586369469762,
+            0.016327809542417526,
+            0.004819063935428858,
+            0.004997218027710915,
+            0.004583009984344244,
+            -0.005041756667196751,
+            0.015472670085728168,
+            -0.011749252676963806,
+            -0.0009202764485962689,
+            0.01418996136635542,
+            0.009290727786719799,
+            -0.018207332119345665,
+            0.025850137695670128,
+            0.0038748481310904026,
+            0.015303423628211021,
+            -0.023266904056072235,
+            0.0058924416080117226,
+            0.012622206471860409,
+            0.002320455154404044,
+            0.01288943737745285,
+            -0.004729987122118473,
+            0.022946227341890335,
+            0.018902134150266647,
+            0.004734441172331572,
+            -0.0025854590348899364,
+            0.013040868565440178,
+            -0.002086627995595336,
+            -0.0016456970479339361,
+            0.011891775764524937,
+            -0.0034940443001687527,
+            0.004077498335391283,
+            0.001429685391485691,
+            -0.0006591695710085332,
+            0.016755377873778343,
+            -0.01399399247020483,
+            0.014216684736311436,
+            0.006030511111021042,
+            -0.011063359677791595,
+            -0.013896007090806961,
+            0.007099434733390808,
+            -0.015080731362104416,
+            0.0025097436737269163,
+            0.023177828639745712,
+            -0.0014775642193853855,
+            -0.00865382794290781,
+            0.0048502408899366856,
+            0.012381698936223984,
+            -0.0058479029685258865,
+            -0.010484359227120876,
+            0.006181941833347082,
+            0.008221804164350033,
+            0.00027766963467001915,
+            0.006596149876713753,
+            0.0068099345080554485,
+            0.007028173189610243,
+            -0.0007922283257357776,
+            -0.008195081725716591,
+            0.0030508863274008036,
+            0.020933087915182114,
+            -0.0008629331714473665,
+            -0.009210558608174324,
+            -0.03643248230218887,
+            -0.003191182389855385,
+            -0.014679884538054466,
+            0.003086517099291086,
+            0.00030870738555677235,
+            -0.0008350965799763799,
+            -0.004493933171033859,
+            -0.009638127870857716,
+            0.009099212475121021,
+            0.0008907697047106922,
+            0.024745581671595573,
+            -0.012693468481302261,
+            -0.015365777537226677,
+            0.003257990116253495,
+            -0.0055405874736607075,
+            -0.010769405402243137,
+            0.007451288867741823,
+            -0.004213340580463409,
+            0.03269124776124954,
+            0.00901013519614935,
+            0.0025052898563444614,
+            -0.014786777086555958,
+            -0.018813056871294975,
+            0.010208221152424812,
+            -0.00245852442458272,
+            -0.005598487798124552,
+            -0.003939428832381964,
+            -0.02994767762720585,
+            0.010324020870029926,
+            -0.00967375934123993,
+            0.0036744249518960714,
+            -0.0044182175770401955,
+            0.009522328153252602,
+            0.003327024867758155,
+            -0.01487585436552763,
+            -0.0030375246424227953,
+            0.013566422276198864,
+            0.015258884988725185,
+            -0.012435144744813442,
+            0.004451621789485216,
+            -0.005999334156513214,
+            -0.012185730040073395,
+            0.009326359257102013,
+            -0.00967375934123993,
+            -0.017655055969953537,
+            0.002447389764711261
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": true,
+      "properties": {
+        "cosine_similarity": 0.8308763913825133
+      }
+    },
+    {
+      "id": "9606dab1-ad7c-4c15-827c-b893506afcc9",
+      "type": "cosine_similarity",
+      "source": {
+        "id": "1e687808-8db8-4320-805d-0282a8d679bc",
+        "properties": {
+          "page_content": "Deepseekmoe: Towards Ultimate Expert Specialization In\n\nMixture-Of-Experts Language Models\n\nDamai Dai∗1,2, Chengqi Deng1, Chenggang Zhao∗1,3, R.X. Xu1, Huazuo Gao1, Deli Chen1, Jiashi Li1, Wangding Zeng1, Xingkai Yu∗1,4, Y. Wu1, Zhenda Xie1, Y.K. Li1, Panpan Huang1, Fuli Luo1, Chong Ruan1, Zhifang Sui2, Wenfeng Liang1 1DeepSeek-AI 2National Key Laboratory for Multimedia Information Processing, Peking University 3Institute for Interdisciplinary Information Sciences, Tsinghua University 4National Key Laboratory for Novel Software Technology, Nanjing University {daidamai, szf}@pku.edu.cn, {wenfeng.liang}@deepseek.com https://github.com/deepseek-ai/DeepSeek-MoE\n\nAbstract\n\nIn the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top- out of experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into ones and activating from them, allowing for a more flexible combination of activated experts; (2) isolating experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5× expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations.\n\nFurther, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\n1. Introduction\n\nRecent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023a). It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017) has emerged as a popular solution. It can\n\nenable parameter scaling, while concurrently keeping computational costs at a modest level.\n\nRecent applications of MoE architectures in Transformers (Vaswani et al., 2017) have yielded successful attempts at scaling language models to a substantial size (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022), accompanied with remarkable performance. These achievements underscore the considerable potential and promise of MoE language models.\n\nDespite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts.\n\nThis architecture manifests two potential issues: (1) Knowledge Hybridity: existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. (2) Knowledge Redundancy: tokens assigned to different experts may require common knowledge. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters.\n\nThese issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models.\n\nIn response to the aforementioned issues, we introduce DeepSeekMoE, an innovative MoE architecture specifically designed towards ultimate expert specialization. Our architecture involves two principal strategies: (1) Fine-Grained Expert Segmentation: while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. (2) Shared Expert Isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.\n\nThrough compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized.\n\nStarting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B (Lepikhin et al., 2021) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5× expert parameters and computation. Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE.\n\nThese studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization.\n\nLeveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B (DeepSeek-AI, 2024), a dense model trained on the same 2T corpus. We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B (Touvron et al., 2023b), which has approximately 2.5 times the activated parameters. Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard1. Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. The experimental results still validate its substantial advantages over the GShard architecture consistently. In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nOur contributions are summarized as follows: - Architectural Innovation. We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation.\n\nEmpirical Validation. We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. Experimental results validate the high 1https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models\n\nScalability. We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B.\n\nAlignment for MoE. We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B.\n\nPublic Release. In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization.\n\n2. Preliminaries: Mixture-Of-Experts For Transformers\n\nWe first introduce a generic MoE architecture commonly used in Transformer language models. A standard Transformer language model is constructed by stacking layers of standard Transformer blocks, where each block can be represented as follows:\n\n$$\\mathbf{u}{1:T}^{l}=\\text{Self-Att}\\left(\\mathbf{h}{1:T}^{l-1}\\right)+\\mathbf{h}{1:T}^{l-1},\\tag{1}$$ $$\\mathbf{h}{t}^{l}=\\text{FFN}\\left(\\mathbf{u}{t}^{l}\\right)+\\mathbf{u}{t}^{l},\\tag{2}$$\n\nwhere denotes the sequence length, Self-Att(·) denotes the self-attention module, FFN(·) denotes the Feed-Forward Network (FFN), u 1: ∈ R× are the hidden states of all tokens after the -th attention module, and h\n\n∈ Ris the output hidden state of the -th token after the -th Transformer block. For brevity, we omit the layer normalization in the above formulations.\n\nA typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022). An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. Then, each token will be assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts. If the -th FFN is substituted with an MoE layer, the computation for its output hidden state h\n\nis expressed as: where denotes the total number of experts, FFN(·) is the -th expert FFN, , denotes the gate value for the -th expert, , denotes the token-to-expert affinity, Topk(·, ) denotes the set comprising highest affinity scores among those calculated for the -th token and all experts, and e\n\nis the centroid of the -th expert in the -th layer. Note that ,is sparse, indicating that only out of gate values are nonzero. This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only experts. Also, in the above formulations, we omit the layer normalization operation for brevity.\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{N}\\left(g_{i,t}\\operatorname{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\operatorname{Topk}({s_{j,t}|1\\leqslant j\\leqslant N},K),\\ 0,&\\text{otherwise},\\end{cases}$$ $$s_{i,t}=\\operatorname{Softmax}{i}\\left(\\mathbf{u}{t}^{l}\\,\\mathbf{e}_{i}^{l}\\right),$$\n\n$$(3)$$\n\n(4) $\\binom{4}{5}$ .\n\n3. Deepseekmoe Architecture\n\nOn top of the generic MoE architecture outlined in Section 2, we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. As illustrated in Figure 2, our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. Both of these strategies are designed to elevate the level of expert specialization.\n\n3.1. Fine-Grained Expert Segmentation\n\nIn scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts.\n\nIn pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. The finer expert segmentation enables a more flexible and adaptable combination of activated experts. To be specific, on top of a typical MoE architecture shown in Figure 2(a), we segment each expert FFN into smaller experts by reducing the FFN intermediate hidden dimension to 1 times its original size. Since each expert becomes smaller, in response, we also increase the number of activated experts to times to keep the same computation cost, as illustrated in Figure 2(b). With the fine-grained expert segmentation, the output of an MoE layer can be expressed as:\n\n$$\\mathbf{h}{t}^{i}=\\sum{i=1}^{\\text{mN}}\\left(g_{i,t}\\text{FFN}{i}\\left(\\mathbf{u}{t}^{i}\\right)\\right)+\\mathbf{u}{t}^{i},\\tag{6}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\text{Tokr}(s_{i,t}|1\\leqslant j\\leqslant m\\text{N}),m\\text{K}),\\ 0,&\\text{otherwise},\\end{cases}$$ (7) $$s_{i,t}=\\text{Softmax}{i}\\left(\\mathbf{u}{t}^{T}\\mathbf{e}_{t}^{i}\\right),\\tag{8}$$ where the total number of expert parameters is equal to $N$ times the number of parameters in a\n\n$$(6)$$\n\n(7) (8) $\\frac{1}{2}$\n\nstandard FFN, and denotes the total number of fine-grained experts. With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to .\n\nFrom a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. As an illustrative example, we consider the case where = 16. A typical top-2 routing strategy can yield 16 2 = 120 possible combinations. By contrast, if each expert is split into 4 smaller experts, the fine-grained routing strategy can yield 64 8 = 4, 426, 165, 368 potential combinations. The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition.\n\n3.2. Shared Expert Isolation\n\nWith a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. This alleviation of redundancy will contribute to a more parameterefficient model with more specialized experts.\n\nTowards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate experts to serve as shared experts. Regardless of the router module, each token will be deterministically assigned to these shared experts. In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by , as depicted in Figure 2(c). With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows:\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{K_{s}}\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)+\\sum_{i=K_{s}+1}^{mN}\\left(g_{i,t}\\;\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},\\tag{9}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\mathrm{Topk}{s_{j,t}|K_{s}+1\\leqslant j\\leqslant mN},mK-K_{s}},\\ 0,&\\text{otherwise},\\end{cases}$$ (10) $$s_{i,t}=\\mathrm{Softmax}{i}\\left(\\mathbf{u}{t}^{l,T}e_{i}^{l}\\right).\\tag{11}$$ $\\mathbf{h}\\mathbf{M}\\mathbf{E}$ the number of closed-compact is $K_{s}$ the total number of routed currents.\n\nFinally, in DeepSeekMoE, the number of shared expert is , the total number of routed experts is − , and the number of nonzero gates is − .\n\nIt is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022). The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint.\n\n3.3. Load Balance Consideration\n\nAutomatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. Firstly, there is a risk of routing collapse (Shazeer et al., 2017), i.e., the model always selects only a few experts, preventing other experts from sufficient training.\n\nSecondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.\n\nExpert-Level Balance Loss. In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. The computation of the balance loss is as follows:\n\n$$\\begin{array}{l}{{{\\mathcal L}{\\mathrm{ExpBal}}=\\alpha{1}\\sum_{i=1}^{N^{\\prime}}f_{i}P_{i},}}\\ {{f_{i}=\\frac{N^{\\prime}}{K^{T}}\\sum_{t=1}^{T}\\mathds{1}\\,(\\mathrm{Token}\\;t\\;\\mathrm{s}\\mathrm{lects}\\;\\mathrm{Expert}\\;i),}}\\ {{P_{i}=\\frac{1}{T}\\sum_{t=1}^{T}s_{i,t},}}\\end{array}$$\n\n(12) $$\\begin{array}{l}\\small\\text{(13)}\\end{array}$$ = (14) $$\\begin{array}{l}\\small\\text{(14)}\\end{array}$$ .\n\n7 where 1 is a hyper-parameter called expert-level balance factor, ′is equal to ( − ) and ′ is equal to ( − ) for brevity. 1(·) denotes the indicator function.\n\nDevice-Level Balance Loss. In addition to the expert-level balance loss, we introduce a devicelevel balance loss. When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. Instead, our primary objective is to ensure balanced computation across the devices. If we partition all routed experts into groups {E1, E2, ..., E}, and deploy each group on a single device, the device-level balance loss is computed as follows:\n\n$$\\mathcal{L}{\\text{DevBal}}=\\alpha{2}\\sum_{i=1}^{D}f_{i}^{\\prime}P_{i}^{\\prime},$$ $$f_{i}^{\\prime}=\\frac{1}{|\\mathcal{E}{i}|}\\sum{j\\in\\mathcal{E}{i}}f{j},$$ $$P_{i}^{\\prime}=\\sum_{j\\in\\mathcal{E}{i}}P{j},$$\n\n(15) $$\\begin{array}{l}\\small\\text{(16)}\\end{array}$$ = (17) .\n\nwhere 2 is a hyper-parameter called device-level balance factor. In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices.\n\n4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n6. Alignment For Deepseekmoe 16B\n\nPrevious research indicates that MoE models typically do not emerge significant gains from fine-tuning (Artetxe et al., 2022; Fedus et al., 2021). However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B.\n\n6.1. Experimental Setup\n\nTraining Data. For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios.\n\nHyper-Parameters. During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).\n\nWe employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10−5 without incorporating any learning rate scheduling strategy.\n\nEvaluation Benchmarks. For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3, with the following adjustments: (1) We exclude Pile (Gao et al., 2020) since chat models are seldom employed for pure language modeling. (2) We exclude\n\nMetric # Shot LLaMA2 DeepSeek DeepSeekMoE SFT 7B Chat 7B Chat 16B # Total Params N/A 6.7B 6.9B 16.4B # Activated Params N/A 6.7B 6.9B 2.8B FLOPs per 4K Tokens N/A 187.9T 183.5T 74.4T HellaSwag (Acc.) 0-shot 67.9 71.0 72.2 PIQA (Acc.) 0-shot 76.9 78.4 79.7 ARC-easy (Acc.) 0-shot 69.7 70.2 69.9 ARC-challenge (Acc.) 0-shot 50.8 50.2 50.0 BBH (EM) 3-shot 39.3 43.1 42.2 RACE-middle (Acc.) 5-shot 63.9 66.1 64.8 RACE-high (Acc.) 5-shot 49.6 50.8 50.6 DROP (EM) 1-shot 40.0 41.7 33.8 GSM8K (EM) 0-shot 63.4 62.6 62.2 MATH (EM) 4-shot 13.5 14.7 15.2 HumanEval (Pass@1) 0-shot 35.4 45.1 45.7 MBPP (Pass@1) 3-shot 27.8 39.0 46.2 TriviaQA (EM) 5-shot 60.1 59.5 63.3 NaturalQuestions (EM) 0-shot 35.2 32.7 35.1 MMLU (Acc.) 0-shot 50.0 49.7 47.2 WinoGrande (Acc.) 0-shot 65.1 68.4 69.0 CLUEWSC (EM) 5-shot 48.4 66.2 68.2 CEval (Acc.) 0-shot 35.1 44.7 40.0 CMMLU (Acc.) 0-shot 36.9 51.2 49.3\n\nCHID (Zheng et al., 2019) due to the observed instability of results, hindering the derivation of solid conclusions. (3) We additionally include BBH (Suzgun et al., 2022) to provide a more comprehensive assessment of the reasoning ability of the chat models.\n\nTable 5 | Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations.\n\n6.2. Evaluations\n\nBaselines. In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. Correspondingly, we construct three chat models, including LLaMA2 SFT 7B3, DeepSeek Chat 7B, and DeepSeekMoE Chat 16B.\n\nSubsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks.\n\nResults. The evaluation results are presented in Table 5. Our key observations include: (1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intensive tasks (TriviaQA, NaturalQuestions). (2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. In addition, it also surpasses DeepSeek Chat 7B. (3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1). However, it is worth noting that, after supervised finetuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. (4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations.\n\n7. Deepseekmoe 145B Ongoing\n\nEncouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available.\n\n7.1. Experimental Setup\n\nTraining Data and Tokenization. For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study.\n\nModel Settings. For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps.\n\nSubsequently, the learning rate keeps constant during the remaining training process. The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. Also, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse.\n\nEvaluation Benchmarks. We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3).\n\n7.2. Evaluations\n\nBaselines. Apart from DeepSeekMoE 145B, we consider three additional models for comparison. DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters.\n\nResults. From the evaluation results presented in Table 6, we have the following observations: (1) Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. (2) On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. (3) At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5.\n\n8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\nAppendices A. Overview Of Hyper-Parameters\n\nWe present the overview of hyper-parameters for DeepSeekMoE across various sizes in Table 7.\n\n# Params # Layers Hidden # Attn # Shared # Routed Relative Sequence Batch Size Learning Size Heads Experts Experts Expert Size Length (Sequence) Rate 2.0B 9 1280 10 1 63 (7 activated) 0.25 2048 2048 1.08e-3 16.4B 28 2048 16 2 64 (6 activated) 0.25 4096 4608 4.2e-4 144.6B 62 4096 32 4 128 (12 activated) 0.125 4096 4608 3.0e-4\n\nTable 7 | Overview of hyper-parameters for DeepSeekMoE across various sizes. The relative expert size is in comparison to a standard FFN.\n\nB. Comparing Deepseekmoe With Larger Models\n\nComparisons among DeepSeekMoE, GShard×1.2, and GShard×1.5 are shown in Table 8. Comparisons among DeepSeekMoE, Dense×4, and Dense×16 are shown in Table 9.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 2.3B 2.8B 1.9B # Activated Expert Params N/A 0.28B 0.35B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.824 1.808 1.808 HellaSwag (Acc.) 0-shot 53.7 54.4 54.8 PIQA (Acc.) 0-shot 71.8 71.1 72.3 ARC-easy (Acc.) 0-shot 46.8 47.3 49.4 ARC-challenge (Acc.) 0-shot 31.7 34.1 34.3 RACE-middle (Acc.) 5-shot 43.7 46.4 44.0 RACE-high (Acc.) 5-shot 31.9 32.4 31.7 HumanEval (Pass@1) 0-shot 3.7 3.0 4.9 MBPP (Pass@1) 3-shot 2.4 2.6 2.2 TriviaQA (EM) 5-shot 15.2 15.7 16.6 NaturalQuestions (EM) 5-shot 4.5 4.7 5.7\n\nTable 8 | Comparison between DeepSeekMoE and larger GShard models.\n\nAt a larger scale of 13B total parameters, we also compare DeepSeekMoE with GShard×1.2 and GShard×1.5, and show results in Table 10. At a larger scale, DeepSeekMoE even outperforms GShard×1.5 distinctly.\n\nMetric # Shot Dense×4 Dense×16 DeepSeekMoE Relative Expert Size N/A 1 1 0.25 # Experts N/A 4 + 0 16 + 0 1 + 63 # Activated Experts N/A 4 + 0 16 + 0 1 + 7 # Total Expert Params N/A 0.47B 1.89B 1.89B # Activated Expert Params N/A 0.47B 1.89B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.908 1.806 1.808 HellaSwag (Acc.) 0-shot 47.6 55.1 54.8 PIQA (Acc.) 0-shot 70.0 71.9 72.3 ARC-easy (Acc.) 0-shot 43.9 51.9 49.4 ARC-challenge (Acc.) 0-shot 30.5 33.8 34.3 RACE-middle (Acc.) 5-shot 42.4 46.3 44.0 RACE-high (Acc.) 5-shot 30.7 33.0 31.7 HumanEval (Pass@1) 0-shot 1.8 4.3 4.9 MBPP (Pass@1) 3-shot 0.2 2.2 2.2 TriviaQA (EM) 5-shot 9.9 16.5 16.6 NaturalQuestions (EM) 5-shot 3.0 6.3 5.7\n\nTable 9 | Comparison between DeepSeekMoE and larger dense baselines.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 15.9B 19.8B 13.3B # Activated Expert Params N/A 2.37B 2.82B 2.05B # Training Tokens N/A 100B 100B 100B HellaSwag (Acc.) 0-shot 66.6 67.7 69.1 PIQA (Acc.) 0-shot 75.6 76.0 75.7 ARC-easy (Acc.) 0-shot 56.8 56.8 58.8 ARC-challenge (Acc.) 0-shot 39.9 37.6 38.5 RACE-middle (Acc.) 5-shot 51.6 50.6 52.4 RACE-high (Acc.) 5-shot 37.4 36.3 38.5 HumanEval (Pass@1) 0-shot 6.1 6.1 9.8 MBPP (Pass@1) 3-shot 7.0 11.6 10.6 TriviaQA (EM) 5-shot 36.5 36.7 38.2 NaturalQuestions (EM) 5-shot 12.6 12.1 13.7\n\nTable 10 | Comparison between DeepSeekMoE and larger GShard models at a larger scale.\n\nC. Training Benchmark Curves Of Deepseekmoe 16B\n\nWe present the benchmark curves during training of DeepSeekMoE 16B and DeepSeek 7B (Dense) in Figure 7 for reference.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/DeepSeekMoE_2.md"
+          },
+          "summary": "DeepSeekMoE is an innovative Mixture-of-Experts (MoE) architecture designed to enhance expert specialization in language models. It employs fine-grained expert segmentation and shared expert isolation to improve performance and reduce computational costs. Starting with a model of 2B parameters, DeepSeekMoE demonstrates comparable performance to larger models like GShard 2.9B and LLaMA2 7B, using significantly fewer computations. The architecture scales up to 16B and 145B parameters, maintaining advantages over traditional MoE architectures. DeepSeekMoE 16B achieves similar results to dense models with only 40% of the computations, and its chat model version shows adaptability in bilingual scenarios. The architecture's public release aims to contribute to advancements in large-scale language models.",
+          "headlines": {
+            "1. Introduction": [],
+            "2. Preliminaries: Mixture-Of-Experts For Transformers": [],
+            "3. Deepseekmoe Architecture": [
+              "3.1. Fine-Grained Expert Segmentation",
+              "3.2. Shared Expert Isolation",
+              "3.3. Load Balance Consideration"
+            ],
+            "4. Validation Experiments": [
+              "4.1. Experimental Setup",
+              "4.1.1. Training Data And Tokenization",
+              "4.1.2. Infrastructures",
+              "4.1.3. Hyper-Parameters",
+              "4.1.4. Evaluation Benchmarks",
+              "4.2. Evaluations",
+              "4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models",
+              "4.4. Ablation Studies",
+              "4.5. Analysis On Expert Specialization"
+            ],
+            "5. Scaling Up To Deepseekmoe 16B": [
+              "5.1. Experimental Setup",
+              "5.1.1. Training Data And Tokenization",
+              "5.1.2. Hyper-Parameters",
+              "5.1.3. Evaluation Benchmarks",
+              "5.2. Evaluations",
+              "5.2.1. Internal Comparison With Deepseek 7B",
+              "5.2.2. Comparison With Open Source Models"
+            ],
+            "6. Alignment For Deepseekmoe 16B": [
+              "6.1. Experimental Setup",
+              "6.2. Evaluations"
+            ],
+            "7. Deepseekmoe 145B Ongoing": [
+              "7.1. Experimental Setup",
+              "7.2. Evaluations"
+            ],
+            "8. Related Work": [],
+            "9. Conclusion": [],
+            "Appendices": [
+              "A. Overview Of Hyper-Parameters",
+              "B. Comparing Deepseekmoe With Larger Models",
+              "C. Training Benchmark Curves Of Deepseekmoe 16B"
+            ]
+          },
+          "summary_embedding": [
+            -0.012465757317841053,
+            -0.022808987647294998,
+            -0.018792515620589256,
+            -0.02308654971420765,
+            0.01797616109251976,
+            0.007200260180979967,
+            0.006490030791610479,
+            0.042613785713911057,
+            -0.014049487188458443,
+            0.03187054023146629,
+            -0.021486490964889526,
+            -0.023592690005898476,
+            0.009045224636793137,
+            -0.014441338367760181,
+            0.005865518003702164,
+            -0.007363531738519669,
+            -0.008743173442780972,
+            -0.005163452122360468,
+            0.018841497600078583,
+            -0.03100520372390747,
+            0.011110605672001839,
+            -0.046075135469436646,
+            -0.020017050206661224,
+            -0.021845687180757523,
+            0.016425084322690964,
+            0.037389107048511505,
+            0.022547753527760506,
+            -0.0028062239289283752,
+            -0.06844329088926315,
+            0.034678805619478226,
+            0.035168617963790894,
+            0.02052319049835205,
+            0.0005438972148112953,
+            -0.00031302153365686536,
+            -0.024833550676703453,
+            -0.028556134551763535,
+            0.03347059711813927,
+            -0.004379750229418278,
+            -0.01453930139541626,
+            0.015976088121533394,
+            0.0414055772125721,
+            0.012106560170650482,
+            -0.02754385396838188,
+            0.0014245413476601243,
+            0.015151568688452244,
+            0.0055961208418011665,
+            -0.01678428053855896,
+            -0.0022960014175623655,
+            -0.002424577483907342,
+            -0.021225256845355034,
+            -0.01400050614029169,
+            0.018825169652700424,
+            0.001296985661610961,
+            -0.008865626528859138,
+            -0.0542386956512928,
+            0.007330877240747213,
+            0.03840138763189316,
+            0.05623060464859009,
+            0.044148534536361694,
+            -0.020180322229862213,
+            0.03369917720556259,
+            -0.030025575309991837,
+            -0.02501314878463745,
+            0.011028970591723919,
+            0.003983817528933287,
+            -0.018531281501054764,
+            0.01784554310142994,
+            -0.005636938381940126,
+            -0.0006556359585374594,
+            0.01751900091767311,
+            -0.03941367194056511,
+            0.020751770585775375,
+            -0.01887415163218975,
+            0.031266435980796814,
+            0.02173139713704586,
+            0.0073798587545752525,
+            0.01872720755636692,
+            0.01678428053855896,
+            0.0036858476232737303,
+            -0.04107903689146042,
+            0.009118696674704552,
+            0.016523046419024467,
+            0.014196431264281273,
+            0.016269976273179054,
+            0.06553706526756287,
+            -0.05391215533018112,
+            -0.004485876299440861,
+            -0.05169166624546051,
+            0.0012561678886413574,
+            -0.004971608519554138,
+            -0.03670336678624153,
+            0.01048201136291027,
+            0.028425518423318863,
+            0.021372200921177864,
+            -0.01063711941242218,
+            0.024392718449234962,
+            -0.022743679583072662,
+            0.0032001156359910965,
+            -0.04062187671661377,
+            0.0019653772469609976,
+            0.004518530797213316,
+            0.016531210392713547,
+            0.0070778070949018,
+            0.004273623693734407,
+            -0.016702644526958466,
+            0.006045116577297449,
+            -0.01905374974012375,
+            0.03291547670960426,
+            -0.008465612307190895,
+            0.08653374016284943,
+            0.008751336485147476,
+            0.004738946910947561,
+            0.004281787667423487,
+            0.02398454025387764,
+            0.012131051160395145,
+            -0.00810233410447836,
+            -0.04718538001179695,
+            -0.015690363943576813,
+            0.011061624623835087,
+            -0.0383034273982048,
+            0.018335357308387756,
+            -0.008759500458836555,
+            0.01282495353370905,
+            -0.04107903689146042,
+            -0.06344719231128693,
+            0.01498013362288475,
+            0.001784758409485221,
+            -0.004065453074872494,
+            -0.0002903166168835014,
+            -0.018923133611679077,
+            0.003930754493921995,
+            -0.004200151655822992,
+            -0.010645282454788685,
+            -0.021502818912267685,
+            0.03159298002719879,
+            0.03657275065779686,
+            0.06527583301067352,
+            0.036801330745220184,
+            -0.0030164355412125587,
+            0.005775718949735165,
+            0.019690508022904396,
+            -0.020784424617886543,
+            0.003481758525595069,
+            -0.01214737817645073,
+            -0.021992631256580353,
+            -0.03820546343922615,
+            0.010971825569868088,
+            0.07379858940839767,
+            -0.029029620811343193,
+            0.013437220826745033,
+            -0.017616963014006615,
+            0.04205866530537605,
+            -0.004567512311041355,
+            0.026727497577667236,
+            -0.016669990494847298,
+            0.004059330560266972,
+            -0.025388672947883606,
+            0.01620466820895672,
+            -0.024964168667793274,
+            0.012571883387863636,
+            -0.000885746325366199,
+            -0.04222193360328674,
+            0.00011116473615402356,
+            0.026890767738223076,
+            -0.010971825569868088,
+            0.02261306345462799,
+            0.0004949158756062388,
+            -0.006302268709987402,
+            -0.05097327008843422,
+            0.038727931678295135,
+            -0.003912386484444141,
+            0.018645571544766426,
+            -0.0018959869630634785,
+            0.0163107942789793,
+            -0.0016408757073804736,
+            0.013216804713010788,
+            0.009543201886117458,
+            0.007114543113857508,
+            -0.033503253012895584,
+            0.02697240374982357,
+            -0.02504580281674862,
+            0.028049994260072708,
+            0.03435226157307625,
+            -0.0030184765346348286,
+            0.03255628049373627,
+            0.019249675795435905,
+            0.03840138763189316,
+            0.0327032245695591,
+            0.030335791409015656,
+            -0.00824927818030119,
+            0.013878053054213524,
+            0.008200296200811863,
+            0.03608293831348419,
+            0.013429056853055954,
+            0.019723162055015564,
+            -0.009730963967740536,
+            -0.014041324146091938,
+            0.048785436898469925,
+            0.007318632211536169,
+            -0.0179598331451416,
+            -0.005441013257950544,
+            -0.03510330989956856,
+            -0.002832755446434021,
+            -0.014425011351704597,
+            0.0323440283536911,
+            0.02158445306122303,
+            -0.01647406443953514,
+            0.035135965794324875,
+            0.024980494752526283,
+            -0.03751972317695618,
+            -0.005232842173427343,
+            -0.02026195637881756,
+            -0.023298801854252815,
+            -0.04215662553906441,
+            0.011624909937381744,
+            0.012833117507398129,
+            -0.013976015150547028,
+            -0.007469657808542252,
+            0.020457882434129715,
+            -0.023625344038009644,
+            -0.0018265966791659594,
+            -0.01356783788651228,
+            0.006547175347805023,
+            -0.020898714661598206,
+            -0.03667071461677551,
+            -0.01048201136291027,
+            0.025862159207463264,
+            0.012637191452085972,
+            0.04362606629729271,
+            -0.013518855907022953,
+            -0.005367541220039129,
+            0.033193036913871765,
+            0.012759645469486713,
+            0.010767736472189426,
+            0.03699725493788719,
+            -0.03027048148214817,
+            0.024621298536658287,
+            -0.020294612273573875,
+            -0.0032878739293664694,
+            0.01707816869020462,
+            -0.0030062312725931406,
+            0.01962519995868206,
+            -0.04535674303770065,
+            -0.024196794256567955,
+            0.04395261034369469,
+            -0.0007107399869710207,
+            0.016302630305290222,
+            -0.005959399044513702,
+            -0.012106560170650482,
+            0.0035899258218705654,
+            0.0027715289033949375,
+            -0.01438419334590435,
+            -0.008620720356702805,
+            0.007473739795386791,
+            -0.01962519995868206,
+            0.03660540655255318,
+            0.03794422745704651,
+            -0.004132802598178387,
+            -0.014131123200058937,
+            -0.03324201703071594,
+            0.010016689077019691,
+            -0.017878197133541107,
+            -0.023951886221766472,
+            0.04078514873981476,
+            -0.032115448266267776,
+            -0.00029312283731997013,
+            0.039478980004787445,
+            0.018808843567967415,
+            -0.014898497611284256,
+            0.00868602842092514,
+            -0.01709449477493763,
+            -0.026302991434931755,
+            -0.017159804701805115,
+            0.03356856107711792,
+            0.024800896644592285,
+            -0.010743245482444763,
+            0.010245268233120441,
+            0.005330805201083422,
+            0.02893165871500969,
+            0.024376392364501953,
+            0.00648594880476594,
+            -0.013812744058668613,
+            0.015029114671051502,
+            0.027641816064715385,
+            -0.0028021421749144793,
+            -7.66609373386018e-05,
+            -0.004010349046438932,
+            -0.007796200457960367,
+            -0.006453294772654772,
+            -0.013069860637187958,
+            -0.005089980084449053,
+            0.0009184005321003497,
+            -0.032295044511556625,
+            0.015967924147844315,
+            -0.005890008993446827,
+            0.010359558276832104,
+            -0.011535110883414745,
+            0.01288209855556488,
+            0.01722511276602745,
+            -0.013494365848600864,
+            -0.0716434046626091,
+            0.014612773433327675,
+            0.02458864450454712,
+            -0.006706364918500185,
+            -0.017943505197763443,
+            0.008481939323246479,
+            0.00016148542636074126,
+            0.038597315549850464,
+            -0.014180104248225689,
+            -0.031103165820240974,
+            0.02680913358926773,
+            -0.017437364906072617,
+            -0.015241367742419243,
+            -0.010661610402166843,
+            -0.00580837344750762,
+            0.04649963974952698,
+            0.01889047957956791,
+            0.011420820839703083,
+            -0.04042595252394676,
+            -0.007449248805642128,
+            0.015078096650540829,
+            0.014506647363305092,
+            -0.04705476388335228,
+            0.0052083516493439674,
+            0.015486273914575577,
+            0.019723162055015564,
+            0.024800896644592285,
+            0.04274440184235573,
+            -0.0021531390957534313,
+            0.01661284640431404,
+            -0.015053605660796165,
+            -0.05136512219905853,
+            -0.03340528905391693,
+            0.029943939298391342,
+            -0.017731253057718277,
+            -0.02906227484345436,
+            -0.01693122461438179,
+            -0.006151242647320032,
+            0.0017939424142241478,
+            0.007126788143068552,
+            -0.01916803978383541,
+            -0.03464614972472191,
+            -0.03970756009221077,
+            0.02277633361518383,
+            0.007983962073922157,
+            0.004269542172551155,
+            -0.042777057737112045,
+            0.0013459670590236783,
+            0.009184005670249462,
+            0.012073906138539314,
+            0.0009954441338777542,
+            -0.019984396174550056,
+            0.015494437888264656,
+            0.0032001156359910965,
+            -0.031070511788129807,
+            -0.035593122243881226,
+            -0.03595232218503952,
+            0.0023919232189655304,
+            0.003781769424676895,
+            0.06279410421848297,
+            0.012988224625587463,
+            -0.03840138763189316,
+            -0.04522612690925598,
+            0.029176564887166023,
+            -0.02052319049835205,
+            -0.016678154468536377,
+            0.0059430720284581184,
+            0.012898425571620464,
+            -0.009877908043563366,
+            0.013192313723266125,
+            0.023560035973787308,
+            0.041601505130529404,
+            -0.005987971555441618,
+            0.012914752587676048,
+            0.023037567734718323,
+            0.012955570593476295,
+            0.0362788625061512,
+            -0.00936360377818346,
+            0.0035409443080425262,
+            0.039936136454343796,
+            -0.012269831262528896,
+            0.03175625205039978,
+            -0.02189466916024685,
+            0.0020398697815835476,
+            0.02249877341091633,
+            0.027184655889868736,
+            -0.0006479826406575739,
+            0.02594379521906376,
+            -0.031543996185064316,
+            0.018368011340498924,
+            -0.014498483389616013,
+            -0.04888340085744858,
+            -0.01557607389986515,
+            0.013102514669299126,
+            0.030384771525859833,
+            0.02756018005311489,
+            -0.0041348435916006565,
+            -0.019723162055015564,
+            -0.01155143789947033,
+            -0.010596301406621933,
+            0.0018419033149257302,
+            0.03285016864538193,
+            0.010506502352654934,
+            0.010890189558267593,
+            -0.011935125105082989,
+            -0.004104230087250471,
+            0.03025415539741516,
+            -0.021192602813243866,
+            0.01650671847164631,
+            0.0041511706076562405,
+            -0.002029665280133486,
+            -0.04320156201720238,
+            -0.035266581922769547,
+            0.007845181971788406,
+            0.011747363954782486,
+            0.014874007552862167,
+            0.0368339866399765,
+            0.025094784796237946,
+            -0.01662100851535797,
+            0.030629679560661316,
+            -0.01905374974012375,
+            0.027984686195850372,
+            0.039054471999406815,
+            0.011624909937381744,
+            -0.018351683393120766,
+            0.012237177230417728,
+            0.002424577483907342,
+            -0.025127438828349113,
+            0.01680060662329197,
+            -0.032735876739025116,
+            0.01738838478922844,
+            -0.02354370802640915,
+            0.025143766775727272,
+            -0.02579685114324093,
+            -0.009290131740272045,
+            -0.004428731743246317,
+            -0.006130834110081196,
+            0.04888340085744858,
+            -0.026727497577667236,
+            -0.0142454132437706,
+            -0.0024694770108908415,
+            -0.0024531499948352575,
+            -0.008718682453036308,
+            0.010000362060964108,
+            -0.045781247317790985,
+            -0.01844964735209942,
+            -0.04016471654176712,
+            0.007355368230491877,
+            0.005412440747022629,
+            -0.030907239764928818,
+            -0.05694899708032608,
+            0.025421326979994774,
+            -0.06331657618284225,
+            -0.019886434078216553,
+            0.051626358181238174,
+            -0.018400665372610092,
+            -0.008212542161345482,
+            0.009641164913773537,
+            0.006375740747898817,
+            -0.029290854930877686,
+            0.0041348435916006565,
+            0.02504580281674862,
+            0.04418119043111801,
+            -0.0045797573402523994,
+            -0.02865409664809704,
+            0.0028674507047981024,
+            -0.027788760140538216,
+            -0.013461710885167122,
+            0.005261414684355259,
+            0.016449574381113052,
+            0.024539662525057793,
+            0.007628847379237413,
+            0.03585435822606087,
+            0.007294141221791506,
+            0.006318595726042986,
+            0.044279150664806366,
+            -0.025666235014796257,
+            -0.027380581945180893,
+            0.02519274689257145,
+            0.04045860469341278,
+            0.023151857778429985,
+            -0.018253721296787262,
+            -0.0142454132437706,
+            0.0053063142113387585,
+            0.0009296254720538855,
+            -0.022139575332403183,
+            0.002808264922350645,
+            -0.013902543112635612,
+            0.005143043119460344,
+            0.03892385587096214,
+            -0.001205145614221692,
+            0.014196431264281273,
+            0.007302304729819298,
+            0.006845145486295223,
+            0.004334850702434778,
+            -0.011731036007404327,
+            -0.023592690005898476,
+            0.00032220553839579225,
+            0.034972693771123886,
+            0.015404638834297657,
+            -0.030515389516949654,
+            -0.00995954405516386,
+            0.034711457788944244,
+            -0.00795130804181099,
+            -0.009118696674704552,
+            -0.009771781973540783,
+            0.003385836724191904,
+            0.02638462744653225,
+            -0.01606588624417782,
+            0.006649219896644354,
+            0.03716052696108818,
+            -0.019870106130838394,
+            0.005543057806789875,
+            -0.025405000895261765,
+            0.0018061877926811576,
+            0.02442537248134613,
+            -0.006216551177203655,
+            0.028098974376916885,
+            5.6315813708351925e-05,
+            0.03025415539741516,
+            -0.004677720367908478,
+            0.008432958275079727,
+            -0.020327266305685043,
+            0.03987082839012146,
+            -0.017633290961384773,
+            0.03402572125196457,
+            0.02067013457417488,
+            -0.01258004643023014,
+            0.04783846437931061,
+            -0.036801330745220184,
+            -0.020784424617886543,
+            0.00584919098764658,
+            0.012743317522108555,
+            0.0028919412288814783,
+            -0.0191517136991024,
+            -0.018400665372610092,
+            -0.010996315628290176,
+            0.02380494214594364,
+            -0.022547753527760506,
+            -0.016865916550159454,
+            -0.007441085297614336,
+            -0.01890680566430092,
+            -0.009673818945884705,
+            -0.007988044060766697,
+            -0.01709449477493763,
+            -0.016547536477446556,
+            0.044573038816452026,
+            -0.06334923207759857,
+            0.016996532678604126,
+            0.017159804701805115,
+            0.011477965861558914,
+            -0.034678805619478226,
+            -0.019510909914970398,
+            -0.017633290961384773,
+            -0.002347023691982031,
+            -0.030743969604372978,
+            0.001702102366834879,
+            0.01725776679813862,
+            0.01138000376522541,
+            -0.05022222548723221,
+            0.012335140258073807,
+            0.02772345207631588,
+            -0.012065742164850235,
+            0.0014949520118534565,
+            -0.023135529831051826,
+            -0.017176130786538124,
+            0.0031776658724993467,
+            0.022123249247670174,
+            0.005975726060569286,
+            -0.008188051171600819,
+            -0.016865916550159454,
+            -0.005241005681455135,
+            -0.006045116577297449,
+            -0.002359268954023719,
+            0.005094061605632305,
+            0.011780017986893654,
+            0.019690508022904396,
+            -0.0024633544962853193,
+            -0.004436895251274109,
+            -0.006396149750798941,
+            0.027364253997802734,
+            -0.001450052484869957,
+            -0.008416631259024143,
+            0.050907962024211884,
+            0.03556047007441521,
+            0.006094098091125488,
+            -0.01108611561357975,
+            -0.03990348428487778,
+            0.010661610402166843,
+            -0.022711025550961494,
+            0.0401320643723011,
+            0.01905374974012375,
+            -0.008298259228467941,
+            -0.028866350650787354,
+            0.010294250212609768,
+            -0.02290695160627365,
+            -0.009975871071219444,
+            0.009469729848206043,
+            0.00726556871086359,
+            -0.03263791650533676,
+            -0.013543346896767616,
+            0.03510330989956856,
+            0.011437147855758667,
+            -0.006841063499450684,
+            -0.036050282418727875,
+            0.008588065393269062,
+            0.01646590232849121,
+            0.011869817040860653,
+            -0.019233347848057747,
+            0.004449140280485153,
+            -0.012041252106428146,
+            0.017470018938183784,
+            0.008849299512803555,
+            -0.017796561121940613,
+            0.018237393349409103,
+            0.004710374400019646,
+            -0.008490103296935558,
+            -0.003912386484444141,
+            -0.0163107942789793,
+            -0.02006603218615055,
+            0.03840138763189316,
+            0.024490682408213615,
+            0.026335645467042923,
+            0.005457340274006128,
+            0.02008235827088356,
+            0.03227871656417847,
+            0.011282040737569332,
+            -0.03134807199239731,
+            0.027935704216361046,
+            0.016996532678604126,
+            -0.04604247957468033,
+            0.021551799029111862,
+            -0.02218855731189251,
+            -0.028670424595475197,
+            0.009649328887462616,
+            -0.0017367975087836385,
+            0.011192241683602333,
+            0.0034837995190173388,
+            -0.003642988856881857,
+            -0.03307874873280525,
+            -0.006236960180103779,
+            0.015771998092532158,
+            0.015502601861953735,
+            -0.0048695639707148075,
+            -0.003296037670224905,
+            -0.011886144056916237,
+            0.00487364549189806,
+            0.009796272963285446,
+            0.039054471999406815,
+            0.001381682581268251,
+            -0.004751192405819893,
+            -0.0018765984568744898,
+            0.031854212284088135,
+            0.0271193478256464,
+            -0.017274094745516777,
+            -0.026286665350198746,
+            0.00580837344750762,
+            -0.014253576286137104,
+            0.026156047359108925,
+            -0.00487364549189806,
+            0.011877980083227158,
+            0.008718682453036308,
+            -0.0007908449042588472,
+            -0.001373519073240459,
+            0.025584599003195763,
+            -0.02710302174091339,
+            -0.021764053031802177,
+            0.031233781948685646,
+            -0.025731543079018593,
+            0.028735732659697533,
+            0.013061696663498878,
+            -0.01557607389986515,
+            -0.015225040726363659,
+            -0.010302413254976273,
+            0.009943217039108276,
+            -0.012277995236217976,
+            0.023853924125432968,
+            0.003608293831348419,
+            0.006392067763954401,
+            0.0003584313380997628,
+            0.02770712412893772,
+            0.024833550676703453,
+            0.007820690982043743,
+            -0.018074123188853264,
+            -0.0016816934803500772,
+            -0.02320083975791931,
+            -0.01680060662329197,
+            0.023772288113832474,
+            -0.018090449273586273,
+            -0.03268689662218094,
+            -0.028556134551763535,
+            0.012114724144339561,
+            0.007392104249447584,
+            0.010816717520356178,
+            0.03461349755525589,
+            -0.030384771525859833,
+            -0.0344502255320549,
+            0.01766594499349594,
+            0.026270337402820587,
+            -0.005551221314817667,
+            -0.021535472944378853,
+            0.03085825778543949,
+            -0.005828781984746456,
+            0.017600636929273605,
+            0.0404912605881691,
+            -0.0016225076979026198,
+            0.010228941217064857,
+            -0.042026009410619736,
+            -0.014057651162147522,
+            -0.0066614653915166855,
+            0.006261450704187155,
+            0.034417569637298584,
+            0.01048201136291027,
+            0.01535565685480833,
+            0.03448288142681122,
+            -0.021192602813243866,
+            -0.0002903166168835014,
+            -0.0022449791431427,
+            -0.02818061038851738,
+            0.03347059711813927,
+            0.025715215131640434,
+            0.011918798089027405,
+            0.019706835970282555,
+            -0.03794422745704651,
+            -0.013037205673754215,
+            0.027070365846157074,
+            0.015053605660796165,
+            0.01273515447974205,
+            0.016694480553269386,
+            0.011249386705458164,
+            0.00868602842092514,
+            -0.021633435040712357,
+            -0.05169166624546051,
+            0.01214737817645073,
+            0.01993541419506073,
+            -0.016449574381113052,
+            0.01057181041687727,
+            0.024980494752526283,
+            0.007620683871209621,
+            -0.007045152597129345,
+            0.012808626517653465,
+            -0.02682545967400074,
+            0.002259265398606658,
+            0.0019010890973731875,
+            -0.014482156373560429,
+            0.038270771503448486,
+            -0.016367938369512558,
+            0.011363675817847252,
+            -0.018204739317297935,
+            0.023870250210165977,
+            0.00011531029304023832,
+            0.0007964573451317847,
+            0.014963806606829166,
+            -0.011347348801791668,
+            -0.04235255345702171,
+            0.0021613026037812233,
+            0.03342161700129509,
+            -0.015200549736618996,
+            0.026449935510754585,
+            -0.0020674217958003283,
+            0.032458316534757614,
+            0.042777057737112045,
+            0.017731253057718277,
+            0.0003497575526125729,
+            0.025682561099529266,
+            0.0036144163459539413,
+            0.0073880222626030445,
+            0.012400448322296143,
+            0.009322785772383213,
+            0.007245160173624754,
+            0.030809277668595314,
+            -0.010261595249176025,
+            0.02173139713704586,
+            -0.04760988429188728,
+            -0.014629100449383259,
+            0.012024925090372562,
+            -0.0013316808035597205,
+            -0.012449429370462894,
+            -0.033029764890670776,
+            0.012563719414174557,
+            -0.03503800183534622,
+            0.022074267268180847,
+            -0.0004959363141097128,
+            0.04333217814564705,
+            -0.02187834307551384,
+            -0.005518566817045212,
+            0.016694480553269386,
+            0.0016510800924152136,
+            0.023282473906874657,
+            -0.0012153501156717539,
+            -0.016416920349001884,
+            -0.0067798369564116,
+            -0.009257477708160877,
+            0.004138925112783909,
+            0.04375668615102768,
+            -0.003722583642229438,
+            -0.0001682458823779598,
+            0.01409030519425869,
+            -0.011959616094827652,
+            -0.03250729665160179,
+            0.045193471014499664,
+            -0.0030837850645184517,
+            0.02484987862408161,
+            0.04979771748185158,
+            -0.06410028040409088,
+            -0.019543563947081566,
+            0.013061696663498878,
+            -0.010441194288432598,
+            0.011363675817847252,
+            0.005547139327973127,
+            0.016114868223667145,
+            0.017894525080919266,
+            -0.005485912784934044,
+            0.01619650423526764,
+            0.005106307100504637,
+            0.028098974376916885,
+            -0.02336410991847515,
+            -0.03296445682644844,
+            -0.02636830136179924,
+            -0.017747581005096436,
+            0.0014480114914476871,
+            -0.015796490013599396,
+            -3.563585551091819e-06,
+            -0.0022204883862286806,
+            -0.04388730227947235,
+            -0.025274382904171944,
+            0.040687184780836105,
+            0.006706364918500185,
+            0.02458864450454712,
+            0.0077268099412322044,
+            0.03418898954987526,
+            0.005885927006602287,
+            -0.009037061594426632,
+            0.030058229342103004,
+            -0.020049704238772392,
+            0.03833607956767082,
+            0.022106921300292015,
+            -0.002832755446434021,
+            0.005865518003702164,
+            0.031086837872862816,
+            -0.033797141164541245,
+            0.008086007088422775,
+            -0.034384917467832565,
+            0.0046695563942193985,
+            -0.008620720356702805,
+            -0.015094423666596413,
+            -0.009526874870061874,
+            -0.02504580281674862,
+            0.011216731742024422,
+            -0.021535472944378853,
+            0.01590261608362198,
+            0.010416703298687935,
+            -0.002916431985795498,
+            0.03448288142681122,
+            -0.014204595237970352,
+            -0.0027796924114227295,
+            0.0024531499948352575,
+            -0.027070365846157074,
+            -0.006343086715787649,
+            -0.003377673216164112,
+            -0.03689929470419884,
+            -0.004502203781157732,
+            0.010988152585923672,
+            -0.00024184548237826675,
+            0.016898570582270622,
+            0.02787039615213871,
+            0.0005092020728625357,
+            -0.022302847355604172,
+            0.002340900944545865,
+            -0.04199335351586342,
+            0.0011877981014549732,
+            0.01483318954706192,
+            -0.017600636929273605,
+            -0.008612556383013725,
+            0.033046092838048935,
+            0.023298801854252815,
+            -0.009396257810294628,
+            -0.035005345940589905,
+            0.017796561121940613,
+            -0.011853490024805069,
+            0.027511198073625565,
+            -0.0002630197268445045,
+            0.019298657774925232,
+            -0.0076941559091210365,
+            0.04493223875761032,
+            0.001369437319226563,
+            0.008645210415124893,
+            -0.021959977224469185,
+            0.007020662073045969,
+            -0.020278284326195717,
+            -0.002406209474429488,
+            -0.006171651650220156,
+            -0.012751481495797634,
+            -0.010669773444533348,
+            0.0032735876739025116,
+            0.007730891928076744,
+            -0.0176822729408741,
+            0.0055961208418011665,
+            -0.015959760174155235,
+            -0.02741323597729206,
+            0.007543129846453667,
+            -0.05965929850935936,
+            -0.0179598331451416,
+            -0.03373183310031891,
+            -0.009477893821895123,
+            0.02008235827088356,
+            0.03510330989956856,
+            -0.012759645469486713,
+            0.00497977202758193,
+            -0.014898497611284256,
+            0.004477712791413069,
+            0.03490738570690155,
+            0.044442422688007355,
+            -0.014759717509150505,
+            0.003381754970178008,
+            -0.010955498553812504,
+            0.033029764890670776,
+            -0.03347059711813927,
+            0.0030062312725931406,
+            0.031086837872862816,
+            -0.005057325586676598,
+            0.007959471084177494,
+            0.011624909937381744,
+            0.020474210381507874,
+            -0.019396619871258736,
+            0.00877582747489214,
+            0.002100075827911496,
+            0.019233347848057747,
+            0.02171507105231285,
+            0.020343592390418053,
+            -0.03539719805121422,
+            0.028409190475940704,
+            0.018384337425231934,
+            0.02444170042872429,
+            0.044442422688007355,
+            -0.03974021226167679,
+            -0.004147088620811701,
+            -0.0032225653994828463,
+            -0.02997659333050251,
+            0.02008235827088356,
+            -0.007530884351581335,
+            -0.01841699331998825,
+            -0.007069643586874008,
+            0.03327467292547226,
+            0.014571955427527428,
+            -0.05051611363887787,
+            0.01980479806661606,
+            0.01740471087396145,
+            0.02055584453046322,
+            0.024474354460835457,
+            -0.01123305968940258,
+            -0.0019072118448093534,
+            0.008996243588626385,
+            0.006804327480494976,
+            0.019266001880168915,
+            0.015257694758474827,
+            0.002012317767366767,
+            -0.0022082431241869926,
+            -0.011837163008749485,
+            0.005224678665399551,
+            -0.009077879600226879,
+            0.010808554477989674,
+            -0.02458864450454712,
+            -0.009698309935629368,
+            0.021388528868556023,
+            0.0031164393294602633,
+            -0.013665799982845783,
+            -0.03729114308953285,
+            -0.045944519340991974,
+            0.0033837957307696342,
+            0.004543021321296692,
+            -0.05365091934800148,
+            0.005322641227394342,
+            -0.026547899469733238,
+            -0.002108239568769932,
+            -0.005502239800989628,
+            0.0019214981002733111,
+            0.005636938381940126,
+            0.0167679525911808,
+            -0.013494365848600864,
+            0.02248244546353817,
+            -0.015502601861953735,
+            0.03165828809142113,
+            0.0055961208418011665,
+            0.007375776767730713,
+            -0.0027796924114227295,
+            0.05590406060218811,
+            -0.003712379140779376,
+            -0.03699725493788719,
+            0.023282473906874657,
+            -0.01311067771166563,
+            -0.0001747512142173946,
+            0.030090883374214172,
+            0.00884113647043705,
+            -0.015257694758474827,
+            0.013437220826745033,
+            -0.030613351613283157,
+            -0.03249097242951393,
+            0.010996315628290176,
+            -0.013755599036812782,
+            -0.0015776080545037985,
+            -0.04568328335881233,
+            -0.009339112788438797,
+            0.004269542172551155,
+            -0.018335357308387756,
+            0.010775899514555931,
+            0.01631895825266838,
+            0.013992343097925186,
+            -0.024245774373412132,
+            0.009053388610482216,
+            0.006865554489195347,
+            0.013453547842800617,
+            0.0007331897504627705,
+            0.014049487188458443,
+            -0.008677864447236061,
+            -0.011306531727313995,
+            -0.018319029361009598,
+            0.0032450153958052397,
+            -0.026792805641889572,
+            0.005155288614332676,
+            -0.003155216109007597,
+            -0.03771565109491348,
+            -0.06521052122116089,
+            -0.0344502255320549,
+            0.016294466331601143,
+            0.03396041318774223,
+            -0.009045224636793137,
+            0.0035144127905368805,
+            -0.00023648813657928258,
+            -0.01093917153775692,
+            0.024523336440324783,
+            -0.005930826533585787,
+            -0.0004829256213270128,
+            0.0035389035474509,
+            0.02050686441361904,
+            -0.009722800925374031,
+            0.005845109466463327,
+            -0.051952898502349854,
+            -0.01586179807782173,
+            0.01812310516834259,
+            0.006306350696831942,
+            0.014629100449383259,
+            -0.009690145961940289,
+            0.01693122461438179,
+            -0.030074557289481163,
+            -0.028147956356406212,
+            0.04130761697888374,
+            0.008277850225567818,
+            0.011486129835247993,
+            -0.003055212553590536,
+            -0.01680060662329197,
+            0.02504580281674862,
+            -0.0020633398089557886,
+            -0.006461458280682564,
+            -0.00817988719791174,
+            0.015820980072021484,
+            0.044279150664806366,
+            0.003255219664424658,
+            0.0009821783751249313,
+            0.017045514658093452,
+            -0.002663361607119441,
+            -0.011575928889214993,
+            0.007677828427404165,
+            0.013592327944934368,
+            -0.016384266316890717,
+            0.010375885292887688,
+            -0.025862159207463264,
+            0.0332583449780941,
+            0.03131541982293129,
+            0.011575928889214993,
+            -0.015045441687107086,
+            0.006130834110081196,
+            0.005330805201083422,
+            0.008086007088422775,
+            -0.001369437319226563,
+            -0.0017133272485807538,
+            0.020719116553664207,
+            -0.03276853263378143,
+            -0.01693122461438179,
+            -0.005347132217139006,
+            0.003932795487344265,
+            0.005277741700410843,
+            0.03396041318774223,
+            -0.0033266509417444468,
+            0.017927179113030434,
+            -0.011657563969492912,
+            0.010579974390566349,
+            0.021764053031802177,
+            0.037095218896865845,
+            -0.024996822699904442,
+            -0.029862305149435997,
+            0.012914752587676048,
+            -0.001296985661610961,
+            0.014065815135836601,
+            0.0010173837654292583,
+            0.014139287173748016,
+            0.0029409227427095175,
+            0.060475654900074005,
+            -0.017763907089829445,
+            0.004628738854080439,
+            0.0004035860183648765,
+            -0.01858026348054409,
+            -0.0013531101867556572,
+            0.004261378664523363,
+            -0.010979988612234592,
+            0.03608293831348419,
+            0.002344982698559761,
+            0.039185091853141785,
+            0.013918871060013771,
+            0.009290131740272045,
+            0.0015143405180424452,
+            -0.013592327944934368,
+            0.008355404250323772,
+            0.0015592400450259447,
+            -0.001132694073021412,
+            0.009461566805839539,
+            0.019298657774925232,
+            0.009714636951684952,
+            0.025274382904171944,
+            0.009788108989596367,
+            0.012857607565820217,
+            -0.029470453038811684,
+            0.03510330989956856,
+            -0.0006224715034477413,
+            0.012996388599276543,
+            -0.02040890045464039,
+            -0.003136848099529743,
+            -0.0008240094175562263,
+            0.015649545937776566,
+            -0.015616890974342823,
+            -0.001201063860207796,
+            -0.01297189760953188,
+            0.040099408477544785,
+            -0.018988441675901413,
+            0.013976015150547028,
+            0.009673818945884705,
+            -0.02865409664809704,
+            -0.007596192881464958,
+            0.01995174214243889,
+            0.014294394291937351,
+            -0.0053144777193665504,
+            -0.020490536466240883,
+            0.005832863971590996,
+            0.015380147844552994,
+            -0.028572462499141693,
+            1.3584674888988957e-05,
+            0.014751553535461426,
+            -0.0164903923869133,
+            -0.00025409081717953086,
+            -0.004702210891991854,
+            0.003975654020905495,
+            -0.009265640750527382,
+            0.007028825581073761,
+            0.016057724133133888,
+            -0.024507008492946625,
+            -0.020049704238772392,
+            0.005722655914723873,
+            0.006571666337549686,
+            0.01444950234144926,
+            0.007739055436104536,
+            0.004355259705334902,
+            -0.020033378154039383,
+            -0.01799248717725277,
+            0.0032674651592969894,
+            -0.01649855636060238,
+            -0.0048614004626870155,
+            0.0020531355403363705,
+            -0.019363965839147568,
+            0.01875986158847809,
+            -0.010743245482444763,
+            0.00017538899555802345,
+            -0.003226647386327386,
+            -0.020800752565264702,
+            0.006204306147992611,
+            -0.004865481983870268,
+            0.019870106130838394,
+            0.003555230563506484,
+            0.02785406820476055,
+            0.02310287579894066,
+            -0.004428731743246317,
+            0.017600636929273605,
+            -0.016155686229467392,
+            0.007983962073922157,
+            -0.031135819852352142,
+            -0.001782717532478273,
+            -0.010408539324998856,
+            0.0036103345919400454,
+            -0.015943434089422226,
+            -0.009126860648393631,
+            -0.0048777274787425995,
+            -0.0006678812787868083,
+            -0.004820582456886768,
+            0.0004275664687156677,
+            0.0027817331720143557,
+            0.02231917530298233,
+            -0.013208640739321709,
+            -0.02803366631269455,
+            -0.017437364906072617,
+            0.006045116577297449,
+            0.019396619871258736,
+            -0.026025431230664253,
+            -0.009184005670249462,
+            -0.005012426059693098,
+            -0.004326687194406986,
+            -0.021404854953289032,
+            -0.015167895704507828,
+            -0.005632856860756874,
+            -0.008045189082622528,
+            -0.010522829368710518,
+            0.0046532293781638145,
+            -0.0035715578123927116,
+            -0.011739199981093407,
+            0.021666089072823524,
+            0.018057795241475105,
+            -0.019837452098727226,
+            -0.019837452098727226,
+            0.013641309924423695,
+            -0.006159406621009111,
+            0.0018878233386203647,
+            0.02008235827088356,
+            0.00029286774224601686,
+            0.005824700463563204,
+            0.020882388576865196,
+            -0.0035225762985646725,
+            -0.01797616109251976,
+            0.003577680326998234,
+            -0.0018888438353314996,
+            -0.012122887186706066,
+            0.003212361130863428,
+            0.02773977816104889,
+            0.0008076822850853205,
+            0.023429417982697487,
+            -0.03056436963379383,
+            -0.011412657797336578,
+            0.006061443593353033,
+            0.02369065210223198,
+            0.00211844383738935,
+            -0.0021694661118090153,
+            0.020131340250372887,
+            0.03941367194056511,
+            0.005775718949735165,
+            0.0015755671774968505,
+            -0.04003410041332245,
+            0.01027792226523161,
+            0.009208496659994125,
+            0.024621298536658287,
+            0.008339077234268188,
+            0.0014653591206297278,
+            -0.012906589545309544,
+            -0.009755454957485199,
+            -0.0017939424142241478,
+            0.014702572487294674,
+            -0.023870250210165977,
+            0.0024755997583270073,
+            -0.03157665207982063,
+            0.005094061605632305,
+            0.0021959978621453047,
+            -0.009559528902173042,
+            0.0335359051823616,
+            0.00980443600565195,
+            -0.02065380848944187,
+            0.015151568688452244,
+            0.005767555441707373,
+            0.007330877240747213,
+            0.022449791431427002,
+            -0.008228869177401066,
+            0.030450081452727318,
+            -0.005049162078648806,
+            0.01213921420276165,
+            0.02742956392467022,
+            0.032588932663202286,
+            -0.013102514669299126,
+            -0.003959327004849911,
+            0.0006097159348428249,
+            -0.0002561317232903093,
+            0.0007995187188498676,
+            -0.008204378187656403,
+            0.0037389106582850218,
+            -0.014800535514950752,
+            0.012710663489997387,
+            -0.019543563947081566,
+            -0.0034286954905837774,
+            0.0038532004691660404,
+            0.01991908811032772,
+            0.0016265894519165158,
+            -0.033633869141340256,
+            0.02055584453046322,
+            -0.015420965850353241,
+            0.0135596739128232,
+            0.001290863030590117,
+            -0.011110605672001839,
+            -0.00029210239881649613,
+            0.020588500425219536,
+            0.010947334580123425,
+            -0.007845181971788406,
+            -0.0135596739128232,
+            -0.028376536443829536,
+            -0.021535472944378853,
+            0.00823703221976757,
+            0.009445239789783955,
+            -0.0028837777208536863,
+            0.0021306893322616816,
+            -0.011869817040860653,
+            0.0006520643946714699,
+            -0.0008122742874547839,
+            0.01740471087396145,
+            -0.0335359051823616,
+            0.021943651139736176,
+            -0.019527235999703407,
+            -0.019543563947081566,
+            0.017045514658093452,
+            0.008049271069467068,
+            0.009445239789783955,
+            0.0033552234526723623,
+            -0.01995174214243889,
+            -0.012326976284384727,
+            0.0338951013982296,
+            -0.01138000376522541,
+            0.006298186723142862,
+            0.004408322740346193,
+            0.012914752587676048,
+            -0.008914608508348465,
+            -0.011926962062716484,
+            0.01784554310142994,
+            -0.009861581027507782,
+            0.013951525092124939,
+            0.0010010566329583526,
+            -0.0254703089594841,
+            0.02739690989255905,
+            -0.019233347848057747,
+            0.03487472981214523,
+            -0.021617108955979347,
+            0.00711862463504076,
+            0.0066614653915166855,
+            0.008792154490947723,
+            -0.006690037902444601,
+            -0.013224967755377293,
+            0.0012786176521331072,
+            0.03807484731078148,
+            -0.02623768337070942,
+            0.024180466309189796,
+            0.022400809451937675,
+            -0.00040766780148260295,
+            0.02065380848944187,
+            -0.008367649279534817,
+            -0.026711169630289078,
+            0.0179598331451416,
+            -0.012596373446285725,
+            -0.026123393326997757,
+            0.013257622718811035,
+            -0.022678371518850327,
+            -0.020278284326195717,
+            -0.026302991434931755,
+            0.004947117529809475,
+            -0.0025694805663079023,
+            -0.005151206627488136,
+            0.009004407562315464,
+            0.02561725303530693,
+            0.020915042608976364,
+            0.019576217979192734,
+            0.02111096680164337,
+            0.01161674689501524,
+            0.022449791431427002,
+            -0.007559456862509251,
+            0.04401791840791702,
+            -0.004947117529809475,
+            0.004632820375263691,
+            -0.0015980169409886003,
+            -0.015657709911465645,
+            0.009771781973540783,
+            0.0119106350466609,
+            0.0029572497587651014,
+            -0.029176564887166023,
+            -0.009061552584171295,
+            -0.03660540655255318,
+            0.005053244065493345,
+            0.0020664012990891933,
+            0.0176822729408741,
+            0.032115448266267776,
+            -0.005865518003702164,
+            0.004592002835124731,
+            -0.005102225113660097,
+            -0.013927034102380276,
+            -0.009420748800039291,
+            -0.02739690989255905,
+            -0.025437654927372932,
+            -0.007653337903320789,
+            -0.013739272020757198,
+            -0.030368445441126823,
+            -0.011722872965037823,
+            0.01551892887800932,
+            -0.007645174395292997,
+            0.011918798089027405,
+            -0.016702644526958466,
+            0.01385356206446886,
+            -0.004110352601855993,
+            -0.004763437435030937,
+            -0.004530776292085648,
+            -0.017600636929273605,
+            -0.006420640274882317,
+            0.014367866329848766,
+            -0.009477893821895123,
+            -0.01237595733255148,
+            -0.02215590327978134,
+            -0.0015459742862731218,
+            -0.01574750803411007,
+            0.003489922033622861,
+            -0.014596446417272091,
+            -0.006771673448383808,
+            -0.003908304497599602,
+            -0.018514955416321754,
+            0.01766594499349594,
+            0.014816862531006336,
+            -0.019380291923880577,
+            -0.0090697156265378,
+            0.02638462744653225,
+            0.042777057737112045,
+            -0.03218075633049011,
+            -0.008200296200811863,
+            0.0020378287881612778,
+            0.018090449273586273,
+            0.010171796195209026,
+            0.016702644526958466,
+            -0.024653952568769455,
+            0.0022286521270871162,
+            -0.021649762988090515,
+            -0.001220452249981463,
+            -0.023739634081721306,
+            0.010775899514555931,
+            0.025715215131640434,
+            -0.010971825569868088,
+            0.025976449251174927,
+            0.01634344831109047,
+            -0.01812310516834259,
+            0.012016761116683483,
+            0.007183933164924383,
+            0.014645427465438843,
+            -0.019755816087126732,
+            0.0167679525911808,
+            -0.015796490013599396,
+            0.031233781948685646,
+            0.00892277155071497,
+            -0.04829562455415726,
+            -0.03503800183534622,
+            -0.013388238847255707,
+            0.0013388239312916994,
+            -0.010702427476644516,
+            0.015037278644740582,
+            0.015967924147844315,
+            0.013959688134491444,
+            -0.019380291923880577,
+            -0.015820980072021484,
+            0.011984107084572315,
+            0.010065670125186443,
+            0.027462217956781387,
+            -0.006649219896644354,
+            -0.004212397150695324,
+            -0.00960034690797329,
+            -0.02249877341091633,
+            -0.016098542138934135,
+            0.0209803506731987,
+            -0.018041469156742096,
+            0.015935270115733147,
+            -0.026139721274375916,
+            -0.012449429370462894,
+            0.017453692853450775,
+            -0.05211617052555084,
+            -0.01453113742172718,
+            0.0007612520130351186,
+            0.03012353740632534,
+            0.004751192405819893,
+            0.0035184945445507765,
+            0.01841699331998825,
+            -0.01993541419506073,
+            -0.0039776950143277645,
+            -0.02099667675793171,
+            0.006832899991422892,
+            -0.011853490024805069,
+            -0.013984179124236107,
+            0.014482156373560429,
+            -0.013845399022102356,
+            -0.01601690612733364,
+            0.0034368589986115694,
+            0.007183933164924383,
+            -0.021094640716910362,
+            0.011951452121138573,
+            0.003945040516555309,
+            0.012171869166195393,
+            0.012171869166195393,
+            0.01199227012693882,
+            0.021551799029111862,
+            0.02295593172311783,
+            -0.007334959227591753,
+            -0.004767519421875477,
+            -0.01004117913544178,
+            -0.0015398516552522779,
+            0.008898280560970306,
+            0.0033797139767557383,
+            -0.002989904023706913,
+            0.007837017998099327,
+            0.010033016093075275,
+            -0.01498013362288475,
+            0.033193036913871765,
+            -0.0035878848284482956,
+            -0.04134026914834976,
+            -0.003238892648369074,
+            -0.03800953924655914,
+            0.020033378154039383,
+            -0.023298801854252815,
+            -0.016996532678604126,
+            0.0032776694279164076,
+            -0.00629410520195961,
+            -0.019902760162949562,
+            0.013608654960989952,
+            0.011404493823647499,
+            0.00258580781519413,
+            0.02021297626197338,
+            0.012269831262528896,
+            -0.005134879611432552,
+            0.026172375306487083,
+            -0.014718899503350258,
+            -0.008751336485147476,
+            0.015567909926176071,
+            0.006853308994323015,
+            0.0022347746416926384,
+            0.0011031010653823614,
+            -0.018825169652700424,
+            0.005098143592476845,
+            0.0164903923869133,
+            0.007216587662696838,
+            -0.000566347036510706,
+            0.014049487188458443,
+            0.012514738366007805,
+            -0.015135240741074085,
+            -0.0002213090192526579,
+            -0.026564225554466248,
+            0.0032817514147609472,
+            0.004567512311041355,
+            0.007506393827497959,
+            -0.006771673448383808,
+            0.006281859707087278,
+            0.00658799335360527,
+            0.014759717509150505,
+            -0.010155469179153442,
+            -0.016980206593871117,
+            -0.002510294783860445,
+            0.004285869188606739,
+            0.016694480553269386,
+            0.03157665207982063,
+            -0.010310577228665352,
+            -0.005734901409596205,
+            -0.0386626236140728,
+            0.02070278860628605,
+            0.03332365304231644,
+            0.011175914667546749,
+            -0.028213264420628548,
+            0.012302485294640064,
+            0.0006515541463159025,
+            0.0019276207312941551,
+            -0.026302991434931755,
+            0.02787039615213871,
+            0.00974729098379612,
+            0.0066614653915166855,
+            0.017861871048808098,
+            0.0022306928876787424,
+            0.0029266364872455597,
+            -0.016302630305290222,
+            0.01018812321126461,
+            0.009086042642593384,
+            -0.017306748777627945,
+            0.0009745250572450459,
+            0.02324981987476349,
+            -0.01423724927008152,
+            -0.003208279376849532,
+            0.0018133309204131365,
+            0.02651524543762207,
+            -0.013820908032357693,
+            -0.014294394291937351,
+            0.0005714492290280759,
+            0.01751900091767311,
+            -0.0007979880319908261,
+            0.012098397128283978,
+            0.04898136481642723,
+            -0.012424939312040806,
+            0.019657853990793228,
+            0.005461421795189381,
+            -0.015290348790585995,
+            -0.032605260610580444,
+            -0.004922627005726099,
+            -0.01830270327627659,
+            0.011200404725968838,
+            0.014343376271426678,
+            -0.015763835981488228,
+            0.0007525782566517591,
+            -0.03144603595137596,
+            0.007743136957287788,
+            0.014735226519405842,
+            0.018776189535856247,
+            -0.002440904499962926,
+            0.006134915631264448,
+            0.027674470096826553,
+            0.02458864450454712,
+            -0.007392104249447584,
+            0.013804581016302109,
+            0.02320083975791931,
+            -0.025062130764126778,
+            -0.003069498809054494,
+            0.006820654962211847,
+            0.004104230087250471,
+            0.0053063142113387585,
+            -0.016963878646492958,
+            0.03915243595838547,
+            -0.005922663025557995,
+            -0.01506176870316267,
+            0.0030266400426626205,
+            -0.009028897620737553,
+            0.0063390047289431095,
+            -0.012563719414174557,
+            0.012637191452085972,
+            -0.008114579133689404,
+            0.00022883480414748192,
+            -0.0059430720284581184,
+            -0.011110605672001839,
+            -0.012653518468141556,
+            -0.0007949267164804041,
+            -0.016531210392713547,
+            0.007432921789586544,
+            0.00981259997934103,
+            0.005045080557465553,
+            0.01025343220680952,
+            -0.0036797248758375645,
+            -0.003136848099529743,
+            -0.016751626506447792,
+            -0.009984034113585949,
+            0.0073716952465474606,
+            0.01872720755636692,
+            -0.028588788583874702,
+            -0.018547609448432922,
+            0.009290131740272045,
+            0.0022898786701261997,
+            0.018776189535856247,
+            0.0158944521099329,
+            -0.0002595757250674069,
+            0.012318813242018223,
+            -0.01964152604341507,
+            -0.009184005670249462,
+            0.009690145961940289,
+            -0.003569516818970442,
+            0.01048201136291027,
+            -0.014759717509150505,
+            0.010033016093075275,
+            -0.011159587651491165,
+            -0.011053460650146008,
+            0.01385356206446886,
+            0.01738838478922844,
+            0.008873790502548218,
+            -0.001294944784604013,
+            3.383413059054874e-05,
+            -0.010506502352654934,
+            -0.024033522233366966,
+            0.004771600943058729,
+            0.027021385729312897,
+            -0.013273949734866619,
+            0.003883813973516226,
+            0.008702355436980724,
+            -0.017502672970294952,
+            -0.019821124151349068,
+            -0.012653518468141556,
+            -0.022106921300292015,
+            0.020572172477841377,
+            0.023413091897964478,
+            -0.02026195637881756,
+            0.00877582747489214,
+            -0.004620575346052647,
+            0.014049487188458443,
+            -0.01400050614029169,
+            0.010612628422677517,
+            0.0317889042198658,
+            -0.020000724121928215,
+            -0.0028654097113758326,
+            0.0036633978597819805,
+            -0.015347493812441826,
+            0.018629245460033417,
+            0.03268689662218094,
+            -0.00176945177372545,
+            0.014041324146091938,
+            0.0008138049161061645,
+            -0.009118696674704552,
+            0.01905374974012375,
+            0.011094278655946255,
+            0.005347132217139006,
+            -0.0042491331696510315,
+            -0.012212686240673065,
+            -0.03010721132159233,
+            0.016702644526958466,
+            0.0129637336358428,
+            -0.0037674831692129374,
+            -0.0011551438365131617,
+            -0.006343086715787649,
+            -0.015404638834297657,
+            0.0035123717971146107,
+            -0.013078023679554462,
+            -0.00803294312208891,
+            -0.008759500458836555,
+            0.004314441699534655,
+            -0.0034960447810590267,
+            -0.009053388610482216,
+            0.007236996199935675,
+            0.0009510547970421612,
+            -0.0010602424154058099,
+            -0.01902109570801258,
+            -0.025078458711504936,
+            -0.00995954405516386,
+            -0.016849588602781296,
+            0.0038287099450826645,
+            0.0005234883283264935,
+            -0.027184655889868736,
+            0.0053961132653057575,
+            -0.008139070123434067,
+            -0.008134988136589527,
+            0.007126788143068552,
+            0.006881881505250931,
+            -0.007437003776431084,
+            -0.010816717520356178,
+            -0.00786150898784399,
+            -0.006865554489195347,
+            0.020033378154039383,
+            0.0034348180051892996,
+            0.012082069180905819,
+            -0.010498338378965855,
+            -0.018025141209363937,
+            -0.016963878646492958,
+            0.016718972474336624,
+            -0.008971752598881721,
+            -0.02083340659737587,
+            -0.01444950234144926,
+            -0.008424794301390648,
+            -0.007665583398193121,
+            -0.01527402177453041,
+            -0.021159948781132698,
+            -0.0032103201374411583,
+            -0.011347348801791668,
+            -0.0067431009374558926,
+            -0.0028960229828953743,
+            0.0013571919407695532,
+            0.00951871182769537,
+            0.00546958576887846,
+            0.00438791373744607,
+            0.021666089072823524,
+            0.010122815147042274,
+            -0.005461421795189381,
+            0.016441410407423973,
+            -0.006183897145092487,
+            -0.004440976772457361,
+            -0.01498013362288475,
+            -0.009918726049363613,
+            0.018547609448432922,
+            -0.0055226488038897514,
+            0.008457448333501816,
+            -0.0009536059224046767,
+            -0.00576347392052412,
+            0.015674035996198654,
+            0.0129637336358428,
+            0.016718972474336624,
+            0.004771600943058729,
+            -0.00012168807734269649,
+            0.012416775338351727,
+            -0.0015745466807857156,
+            0.012555556371808052,
+            -0.0038429962005466223,
+            0.014506647363305092,
+            -0.009722800925374031,
+            -0.018988441675901413,
+            0.004157293122261763,
+            0.006265532691031694,
+            -0.012531065382063389,
+            0.009404421783983707,
+            0.007743136957287788,
+            -0.013747435994446278,
+            -0.02261306345462799,
+            -0.000106317616882734,
+            0.00014362763613462448,
+            0.008122743107378483,
+            0.008971752598881721,
+            -0.008073761127889156,
+            0.012890261597931385,
+            0.004616493359208107,
+            -0.007179851643741131,
+            -0.019657853990793228,
+            0.026009103283286095,
+            0.015608727931976318,
+            0.012269831262528896,
+            -0.052508022636175156,
+            -0.0030980713199824095,
+            0.010743245482444763,
+            -0.015951598063111305,
+            -0.026792805641889572,
+            -0.006404313258826733,
+            -0.008759500458836555,
+            -0.04290767386555672,
+            -0.009624837897717953,
+            -0.0005724696675315499,
+            -0.011388166807591915,
+            0.010400376282632351,
+            0.01573934406042099,
+            0.014612773433327675,
+            0.005489994306117296,
+            -0.007232914678752422,
+            -0.017633290961384773,
+            -0.02788672223687172,
+            0.010057506151497364,
+            -0.013380075804889202,
+            -0.006412476766854525,
+            -0.002263347152620554,
+            -0.007690073922276497,
+            0.0016112826997414231,
+            0.01691489666700363,
+            0.042483169585466385,
+            0.023037567734718323,
+            -0.012335140258073807,
+            -0.020131340250372887,
+            -0.012800462543964386,
+            -0.009192168712615967,
+            0.00633492274209857,
+            0.011592255905270576,
+            -0.004738946910947561,
+            0.0096656559035182,
+            -0.004294032696634531,
+            0.012302485294640064,
+            0.012767808511853218,
+            0.01680060662329197,
+            -2.3023791072773747e-05,
+            0.005579793825745583,
+            -0.0024694770108908415,
+            -0.01738838478922844,
+            -0.004379750229418278,
+            0.008261523209512234,
+            -0.0008872769540175796,
+            0.0019317024853080511,
+            0.014204595237970352,
+            0.019102731719613075,
+            -0.009094206616282463,
+            -0.014612773433327675,
+            -0.016098542138934135,
+            -0.005277741700410843,
+            -0.0012622905196622014,
+            0.029927613213658333,
+            0.02037624642252922,
+            -0.019266001880168915,
+            -0.018237393349409103,
+            -0.018351683393120766,
+            0.00922482367604971,
+            -0.003563394071534276,
+            -0.0064369672909379005,
+            0.01905374974012375,
+            0.023021239787340164,
+            0.017910851165652275,
+            -0.003642988856881857,
+            -0.014865843579173088,
+            0.003804219188168645,
+            0.02847449854016304,
+            0.008367649279534817,
+            0.01078406348824501,
+            0.008726846426725388,
+            -0.0011908593587577343,
+            -0.032441988587379456,
+            -0.0113146947696805,
+            0.00951871182769537,
+            -0.008481939323246479,
+            0.026564225554466248,
+            -0.016531210392713547,
+            0.009388094767928123,
+            -0.005077734589576721,
+            0.0019286411115899682,
+            -0.02365799807012081,
+            0.0014929111348465085,
+            0.010122815147042274,
+            -0.024947840720415115,
+            0.00336950970813632,
+            -0.023592690005898476,
+            -0.00013763252354692668,
+            0.008367649279534817,
+            0.0022837561555206776,
+            0.01650671847164631,
+            -0.0113146947696805,
+            0.0051838611252605915,
+            0.02099667675793171,
+            -0.002832755446434021,
+            -0.01512707769870758,
+            -0.0007842120248824358,
+            0.0005092020728625357,
+            -0.001286781276576221,
+            0.02563357912003994,
+            0.015812816098332405,
+            -0.0036756431218236685,
+            0.020882388576865196,
+            0.008677864447236061,
+            0.0019133344758301973,
+            -0.021764053031802177,
+            0.0023796779569238424,
+            -0.032899148762226105,
+            0.017437364906072617,
+            -0.011126932688057423,
+            0.012490247376263142,
+            0.00868602842092514,
+            -0.01341272983700037,
+            -0.0007388022495433688,
+            -0.0107187544927001,
+            -0.0015725058037787676,
+            -0.007934981025755405,
+            0.009779945947229862,
+            -0.004363423213362694,
+            0.008339077234268188,
+            -0.00024579971795901656,
+            -0.01476788055151701,
+            0.018694553524255753,
+            9.678283277025912e-06,
+            -0.00010925139940809458,
+            0.005441013257950544,
+            0.009935053065419197,
+            0.00786559097468853,
+            0.002009256277233362,
+            0.023870250210165977,
+            -0.031119491904973984,
+            -0.016082214191555977,
+            -0.010204450227320194,
+            0.005543057806789875,
+            -0.007922735065221786,
+            -0.018531281501054764,
+            -0.03257260471582413,
+            0.01318415068089962,
+            0.001293924287892878,
+            -0.0009806477464735508,
+            0.01258004643023014,
+            -0.00942891277372837,
+            -0.0013878053287044168,
+            -0.01606588624417782,
+            -0.014808698557317257,
+            0.006416558753699064,
+            0.0033633869607001543,
+            -0.01222901325672865,
+            -0.023739634081721306,
+            0.014759717509150505,
+            0.0007418635650537908,
+            -0.016980206593871117,
+            -0.017502672970294952,
+            0.00018457300029695034,
+            -0.004804255440831184,
+            0.02204161323606968,
+            -0.016678154468536377,
+            0.003373591462150216,
+            0.0007219648687168956,
+            0.009232986718416214,
+            -0.02889900468289852,
+            0.00716352416202426,
+            0.0006107363733462989,
+            0.0035164537839591503,
+            -0.006518602836877108,
+            -0.021013004705309868,
+            0.011371839791536331,
+            -0.0020674217958003283,
+            -0.014751553535461426,
+            -0.020751770585775375,
+            -0.0020796670578420162,
+            -0.007420676294714212,
+            -0.02724996581673622,
+            0.017910851165652275,
+            0.0009985055075958371,
+            -0.01738838478922844,
+            0.008077843114733696,
+            0.006596156861633062,
+            0.009355440735816956,
+            0.02455599047243595,
+            0.010147306136786938,
+            -0.018776189535856247,
+            -0.0008087027235887945,
+            0.006979844067245722,
+            0.009306458756327629,
+            0.0036450298503041267,
+            0.04222193360328674,
+            -0.008130906149744987,
+            -0.006579829845577478,
+            0.0066614653915166855,
+            -0.008449285291135311,
+            -0.013747435994446278,
+            0.026727497577667236,
+            0.005796127952635288,
+            0.015984252095222473,
+            0.016123032197356224,
+            -0.00945340283215046,
+            -0.005008344538509846,
+            -0.0018786393338814378,
+            0.02065380848944187,
+            0.0073798587545752525,
+            -0.005502239800989628,
+            -0.009714636951684952,
+            0.005881845485419035,
+            -0.00898808054625988,
+            0.0021653843577951193,
+            -0.009526874870061874,
+            -0.005824700463563204,
+            -0.008069679141044617,
+            0.016400592401623726,
+            -0.0003523086488712579,
+            0.027315273880958557,
+            0.020751770585775375,
+            -0.008563575334846973,
+            -0.0005683879135176539,
+            0.017208784818649292,
+            -0.005820618476718664,
+            0.015608727931976318,
+            -0.011192241683602333,
+            -0.007706400938332081,
+            0.006408394780009985,
+            0.024017194285988808,
+            0.012237177230417728,
+            -0.0056042843498289585,
+            0.0002885308349505067,
+            -0.0030348035506904125,
+            -0.010612628422677517,
+            0.013388238847255707,
+            0.015323002822697163,
+            -0.01947825588285923,
+            -0.004722619894891977,
+            -0.0016592437168583274,
+            -0.003300119424238801,
+            0.00945340283215046,
+            0.02305389568209648,
+            0.000171817431692034,
+            0.009184005670249462,
+            0.0012765767751261592,
+            0.03862996771931648,
+            -0.0053144777193665504,
+            0.03102152980864048,
+            -0.0010423846542835236,
+            -0.0059430720284581184,
+            -0.009094206616282463,
+            -0.020441556349396706,
+            -0.011486129835247993,
+            -0.00041481092921458185,
+            -0.005040998570621014,
+            -0.0004956811899319291,
+            -0.020327266305685043,
+            0.0026960158720612526,
+            0.014637264423072338,
+            0.007661501411348581,
+            0.017927179113030434,
+            -0.002889900468289852,
+            0.017437364906072617,
+            0.019690508022904396,
+            -0.01400050614029169,
+            -0.029862305149435997,
+            -0.0006515541463159025,
+            0.01498013362288475,
+            -0.01661284640431404,
+            0.019445601850748062,
+            0.004085862077772617,
+            0.0029531680047512054,
+            0.004779764451086521,
+            -0.006049198564141989,
+            0.018482301384210587,
+            0.02039257436990738,
+            0.01453930139541626,
+            -0.013167822733521461,
+            0.014057651162147522,
+            0.009771781973540783,
+            0.006600238848477602,
+            -0.01646590232849121,
+            -0.013535182923078537,
+            0.002555194543674588,
+            0.0009245232213288546,
+            0.024392718449234962,
+            -0.020588500425219536,
+            0.0016602640971541405,
+            0.010588138364255428,
+            0.01769859902560711,
+            -0.004302196204662323,
+            0.008906444534659386,
+            0.0008740111952647567,
+            0.007134951651096344,
+            -0.0035919665824621916,
+            -0.006457376293838024,
+            -0.006673710886389017,
+            -0.010392212308943272,
+            -0.0020959940738976,
+            -0.02070278860628605,
+            0.002198038622736931,
+            -0.002830714685842395,
+            0.018237393349409103,
+            -0.01018812321126461,
+            0.038140155375003815,
+            0.03399306535720825,
+            -0.0007025764207355678,
+            0.011102442629635334,
+            -0.01709449477493763,
+            -0.003945040516555309,
+            -0.021959977224469185,
+            0.01844964735209942,
+            0.016392430290579796,
+            0.012269831262528896,
+            0.005302232690155506,
+            0.017796561121940613,
+            -0.0006301248213276267,
+            0.014490320347249508,
+            0.0029654132667928934,
+            0.007971717044711113,
+            -0.002265388146042824,
+            0.003938918001949787,
+            -0.031103165820240974,
+            -0.017029186710715294,
+            -0.01138000376522541,
+            0.004796091932803392,
+            -0.006375740747898817,
+            -0.0035205355379730463,
+            0.0084737753495574,
+            -0.005902254022657871,
+            0.0004270562494639307,
+            -0.025992777198553085,
+            -0.0157148540019989,
+            -0.010318740271031857,
+            -0.004040962550789118,
+            -0.01169838197529316,
+            -0.0027694879099726677,
+            -0.0022082431241869926,
+            0.0029552089981734753,
+            0.020947696641087532,
+            -0.012808626517653465,
+            -0.016294466331601143,
+            0.004379750229418278,
+            -0.002263347152620554,
+            -0.014016833156347275,
+            -0.016898570582270622,
+            0.006665547378361225,
+            0.012098397128283978,
+            -0.0017470018938183784,
+            -0.013608654960989952,
+            0.011739199981093407,
+            -0.006628811359405518,
+            0.04663025587797165,
+            -0.013584164902567863,
+            -0.0078002819791436195,
+            0.003720542648807168,
+            0.004698128905147314,
+            -0.009396257810294628,
+            0.007330877240747213,
+            0.013020878657698631,
+            0.01830270327627659,
+            0.0095758568495512,
+            -0.008465612307190895,
+            -0.0024388637393712997,
+            0.007796200457960367,
+            -0.016147522255778313,
+            0.006404313258826733,
+            0.0008525818702764809,
+            -0.014400520361959934,
+            0.006110425107181072,
+            0.028980640694499016,
+            0.013151495717465878,
+            0.01288209855556488,
+            -0.022351829335093498,
+            -0.004094025585800409,
+            -0.009355440735816956,
+            -0.004943036008626223,
+            -0.010090161114931107,
+            0.027070365846157074,
+            -0.018155759200453758,
+            0.02651524543762207,
+            0.00898808054625988,
+            0.0017684312770143151,
+            -0.01176369097083807,
+            0.006343086715787649,
+            0.02832755446434021,
+            0.0028592869639396667,
+            -0.020000724121928215,
+            -0.013592327944934368,
+            0.006314514204859734,
+            0.01707816869020462,
+            -0.002647034591063857,
+            -0.005428767763078213,
+            0.019657853990793228,
+            0.01784554310142994,
+            -0.004334850702434778,
+            -0.00936360377818346,
+            0.0008337036124430597,
+            0.00928196869790554,
+            0.016963878646492958,
+            0.015020951628684998,
+            0.009086042642593384,
+            -0.002647034591063857,
+            0.008037025108933449,
+            -0.008800318464636803,
+            -0.016449574381113052,
+            0.007220669183880091,
+            -0.008432958275079727,
+            -0.016963878646492958,
+            0.01872720755636692,
+            0.015257694758474827,
+            0.0181394312530756,
+            0.0024592725094407797,
+            -0.016735298559069633,
+            0.022302847355604172,
+            0.005755309946835041,
+            0.002998067531734705,
+            0.02160078100860119,
+            -0.022221211344003677,
+            0.013012715615332127,
+            0.02037624642252922,
+            0.027609162032604218,
+            -0.01737205684185028,
+            -0.010906516574323177,
+            -0.004857318475842476,
+            -0.006853308994323015,
+            -0.00176945177372545,
+            -0.008253359235823154,
+            -0.012620864436030388,
+            0.0028735732194036245,
+            -0.030466407537460327,
+            0.018106777220964432,
+            -0.005102225113660097,
+            0.010669773444533348,
+            -0.019266001880168915,
+            0.011363675817847252,
+            -0.0035838030744343996,
+            0.011135096661746502,
+            0.0019612954929471016,
+            -0.030041903257369995,
+            -0.008571738377213478,
+            0.015388311818242073,
+            -0.0030368445441126823,
+            -0.013641309924423695,
+            0.02006603218615055,
+            0.00018942011229228228,
+            -0.022270193323493004,
+            -0.004314441699534655,
+            -0.006934944540262222,
+            0.002193956868723035,
+            -0.0029613315127789974,
+            -0.0052981507033109665,
+            -0.030188847333192825,
+            0.020033378154039383,
+            0.009559528902173042,
+            -0.03072764165699482,
+            -0.003255219664424658,
+            -0.014996460638940334,
+            -0.0025184585247188807,
+            0.008881953544914722,
+            -0.004000144544988871,
+            -0.015551582910120487,
+            0.01784554310142994,
+            0.022711025550961494,
+            -0.0023164101876318455,
+            -0.016057724133133888,
+            -0.0090697156265378,
+            -0.027494871988892555,
+            0.030825603753328323,
+            -0.0066696288995444775,
+            0.0022878379095345736,
+            -0.005322641227394342,
+            0.009388094767928123,
+            0.004228724166750908,
+            0.0157148540019989,
+            0.006081852596253157,
+            -0.002647034591063857,
+            0.026596879586577415,
+            -0.01964152604341507,
+            0.009567692875862122,
+            -1.588864870427642e-05,
+            -0.003308282932266593,
+            0.01890680566430092,
+            0.0007939062779769301,
+            0.021649762988090515,
+            -0.008571738377213478,
+            -0.0001475818717153743,
+            -0.0034388997592031956,
+            0.010898353531956673,
+            -0.0028450009413063526,
+            -0.003465431509539485,
+            -0.001531688030809164,
+            0.011118769645690918,
+            0.01013097818940878,
+            0.003804219188168645,
+            -0.01737205684185028,
+            0.003142970846965909,
+            -0.008547247387468815,
+            -0.007028825581073761,
+            0.01302904263138771,
+            -0.011192241683602333,
+            -0.0018990482203662395,
+            -0.03160930797457695,
+            0.016286302357912064,
+            0.0533243790268898,
+            0.008547247387468815,
+            -0.014433175325393677,
+            -0.0018806802108883858,
+            -0.019755816087126732,
+            0.01535565685480833,
+            0.011004479601979256,
+            -0.016816934570670128,
+            -0.011984107084572315,
+            -0.012310649268329144,
+            0.007649256382137537,
+            0.021306892856955528,
+            -0.006522684823721647,
+            -0.008579902350902557,
+            0.025372346863150597,
+            -0.011788181029260159,
+            -0.010882026515901089,
+            -0.01661284640431404,
+            -0.01635977439582348,
+            0.00241641397587955,
+            0.019902760162949562,
+            -0.0169475506991148,
+            0.021176276728510857,
+            0.00936360377818346,
+            0.0010133018950000405,
+            0.003624620847404003,
+            0.008588065393269062,
+            0.0024878450203686953,
+            0.017878197133541107,
+            0.003883813973516226,
+            -0.01439235731959343,
+            0.0021408938337117434,
+            -0.0030286810360848904,
+            0.004624656867235899,
+            3.4320437407586724e-06,
+            -0.002749078907072544,
+            -0.0065635028295218945,
+            0.003710338147357106,
+            -0.021306892856955528,
+            0.021502818912267685,
+            -0.022678371518850327,
+            -0.004245051648467779,
+            0.005555302836000919,
+            -0.005118552595376968,
+            0.013469874858856201,
+            0.018792515620589256,
+            0.01423724927008152,
+            0.008702355436980724,
+            0.01462093647569418,
+            -0.011331021785736084,
+            0.02323349379003048,
+            -0.013282112777233124,
+            0.0005816536722704768,
+            0.012939243577420712,
+            0.01755165494978428,
+            0.018792515620589256,
+            -0.006245123688131571,
+            -0.010384049266576767,
+            -0.0002946535241790116,
+            0.0129637336358428,
+            -0.022286519408226013,
+            -0.012269831262528896,
+            -0.01386172603815794,
+            0.011486129835247993,
+            -0.004743028432130814,
+            0.025453981012105942,
+            -0.00633492274209857,
+            -0.014653591439127922,
+            -0.015918942168354988,
+            0.014196431264281273,
+            -0.048752784729003906,
+            0.0030348035506904125,
+            -0.0024919267743825912,
+            0.03043375350534916,
+            -0.027021385729312897,
+            0.01738838478922844,
+            -0.0028490826953202486,
+            -0.009355440735816956,
+            0.015429128892719746,
+            -0.015992414206266403,
+            -0.0046613928861916065,
+            -0.0016806729836389422,
+            -0.007539048325270414,
+            -0.00112248957157135,
+            0.017616963014006615,
+            -0.015845470130443573,
+            0.01282495353370905,
+            -0.00989423505961895,
+            0.0027021386194974184,
+            -0.004526694305241108,
+            -0.007751300930976868,
+            0.0196088720113039,
+            0.034548189491033554,
+            -0.007053316105157137,
+            0.0014990337658673525,
+            -0.022547753527760506,
+            0.013192313723266125,
+            -0.006730855442583561,
+            0.02351105399429798,
+            -0.005432849284261465,
+            -0.016898570582270622,
+            -0.013976015150547028,
+            -0.009739127941429615,
+            0.00839214026927948,
+            -0.022433463484048843,
+            -0.011110605672001839,
+            -0.00922482367604971,
+            -0.002573562553152442,
+            -0.018694553524255753,
+            0.004865481983870268,
+            -0.002426618244498968,
+            -0.012506574392318726,
+            -0.008816645480692387,
+            0.01048201136291027,
+            -0.009347276762127876,
+            0.003077662317082286,
+            -0.0011929002357646823,
+            -0.011453475803136826,
+            -0.02369065210223198,
+            0.0158944521099329,
+            0.0001974561164388433,
+            -0.010359558276832104,
+            0.008408467285335064,
+            -0.0018276170594617724,
+            0.001211268245242536,
+            0.016963878646492958,
+            -0.0020664012990891933,
+            0.003467472270131111,
+            0.023478399962186813,
+            0.004057289566844702,
+            -0.010971825569868088,
+            -0.00025906547671183944,
+            0.013535182923078537,
+            -0.005118552595376968,
+            -0.010008525103330612,
+            1.6646010408294387e-05,
+            -0.02984597720205784,
+            -0.008351322263479233,
+            0.006094098091125488,
+            0.018629245460033417,
+            -0.010620792396366596,
+            -0.015853634104132652,
+            0.026319319382309914,
+            -0.006359413731843233,
+            0.019070077687501907,
+            0.002889900468289852,
+            0.0025694805663079023,
+            0.01724144071340561,
+            0.013461710885167122,
+            0.00786150898784399,
+            -0.013927034102380276,
+            -0.02429475635290146,
+            0.0016398552106693387,
+            -0.018090449273586273,
+            0.010579974390566349,
+            -0.011028970591723919,
+            0.0032756286673247814,
+            -0.02398454025387764,
+            -0.012237177230417728,
+            0.004934872500598431,
+            -0.020343592390418053,
+            -0.00040588201954960823,
+            -0.0056940834037959576,
+            0.022237539291381836,
+            -0.005310396198183298,
+            0.008669701404869556,
+            0.0018255761824548244,
+            -0.012237177230417728,
+            0.0009877908742055297,
+            0.009469729848206043,
+            -0.029192892834544182,
+            -0.0007127808639779687,
+            -0.003138889092952013,
+            0.0011418780777603388,
+            0.005351213738322258,
+            0.009755454957485199,
+            0.007322713732719421,
+            0.0027062203735113144,
+            0.007253323681652546,
+            0.010449357330799103,
+            -0.00022041612828616053,
+            0.0197884701192379,
+            0.018172085285186768,
+            -0.007351286243647337,
+            0.008710519410669804,
+            0.024817224591970444,
+            -0.013159659691154957,
+            -0.008718682453036308,
+            0.023266147822141647,
+            -0.006461458280682564,
+            -0.0011122850701212883,
+            -0.01483318954706192,
+            0.00815947912633419,
+            0.0017367975087836385,
+            0.028131630271673203,
+            0.019657853990793228,
+            0.005820618476718664,
+            0.000639819074422121,
+            0.015788326039910316,
+            -0.013020878657698631,
+            0.0035286990460008383,
+            0.015461783856153488,
+            -0.003940958995372057,
+            -0.019821124151349068,
+            0.013878053054213524,
+            0.009502384811639786,
+            -0.003228688146919012,
+            0.01551892887800932,
+            0.011412657797336578,
+            -0.030841931700706482,
+            -0.0011898389784619212,
+            0.010498338378965855,
+            -0.0021449755877256393,
+            -0.008751336485147476,
+            -0.00014273475972004235,
+            -0.012384121306240559,
+            0.021388528868556023,
+            -0.004408322740346193,
+            0.012277995236217976,
+            -0.00815947912633419,
+            -0.017127150669693947,
+            -0.011559601873159409,
+            0.007914572022855282,
+            -0.010547320358455181,
+            -0.0006714528426527977,
+            -0.004022594541311264,
+            -0.005779800936579704,
+            0.017747581005096436,
+            0.004706292413175106,
+            -0.0008964610169641674,
+            -0.0018398624379187822,
+            0.007983962073922157,
+            -0.006979844067245722,
+            -0.008726846426725388,
+            0.004816500470042229,
+            -0.012833117507398129,
+            0.003981776535511017,
+            0.005543057806789875,
+            0.01506176870316267,
+            -0.014857680536806583,
+            -0.021159948781132698,
+            0.007020662073045969,
+            0.02695607766509056,
+            -0.02516009286046028,
+            -0.005232842173427343,
+            0.006245123688131571,
+            0.002093953313305974,
+            0.006914535537362099,
+            -0.006934944540262222,
+            -0.007208424154669046,
+            -0.03657275065779686,
+            -0.0020745648071169853,
+            0.0009005427709780633,
+            0.008963589556515217,
+            -0.008049271069467068,
+            -0.0136494729667902,
+            -0.011135096661746502,
+            0.002410291228443384,
+            0.007134951651096344,
+            -0.02204161323606968,
+            0.0049593630246818066,
+            0.009608510881662369,
+            -0.019559891894459724,
+            0.025127438828349113,
+            -0.010890189558267593,
+            -6.964537897147238e-05,
+            -0.00995954405516386,
+            0.021633435040712357,
+            -0.013486201874911785,
+            -0.018368011340498924,
+            -0.0006898208521306515,
+            0.008355404250323772,
+            0.011926962062716484,
+            -0.0017663904000073671,
+            -0.011371839791536331,
+            0.0003752686607185751,
+            0.01646590232849121,
+            0.019135385751724243,
+            0.002744997153058648,
+            -0.0024837632663547993,
+            -0.01019628718495369,
+            -0.009592183865606785,
+            -0.0073961857706308365,
+            -0.0007449248805642128,
+            0.002091912319883704,
+            0.02035992033779621,
+            0.013469874858856201,
+            0.0024837632663547993,
+            -0.015135240741074085,
+            -0.005257333163172007,
+            0.008939098566770554,
+            0.009412584826350212,
+            -0.011690218932926655,
+            -0.00898808054625988,
+            0.002659279853105545,
+            0.02249877341091633,
+            0.017633290961384773,
+            -0.0014000505907461047,
+            0.0032878739293664694,
+            -0.0016592437168583274,
+            0.012506574392318726,
+            -0.007555375341325998,
+            -0.010269759222865105,
+            -0.006943108048290014,
+            -0.014645427465438843,
+            -0.010800390504300594,
+            -0.0023062059190124273,
+            -0.010179960168898106,
+            0.022629389539361,
+            0.007988044060766697,
+            0.009779945947229862,
+            0.008265605196356773,
+            0.010269759222865105,
+            0.002087830565869808,
+            0.025715215131640434,
+            -0.004424649756401777,
+            -0.011984107084572315,
+            0.007236996199935675,
+            -0.022368155419826508,
+            -0.021535472944378853,
+            0.008257441222667694,
+            0.006367577239871025,
+            0.03595232218503952,
+            -0.026890767738223076,
+            0.0052981507033109665,
+            -0.009779945947229862,
+            -0.010400376282632351,
+            0.007996207103133202,
+            -0.0015439334092661738,
+            -0.004245051648467779,
+            0.022368155419826508,
+            0.00726556871086359,
+            0.0035797213204205036,
+            0.0032817514147609472,
+            0.019837452098727226,
+            -0.005685919895768166,
+            0.01573934406042099,
+            0.0259274672716856,
+            0.024621298536658287,
+            0.009388094767928123,
+            -0.009469729848206043,
+            0.02785406820476055,
+            0.0002895512734539807,
+            -0.0034327772445976734,
+            0.007220669183880091,
+            -0.00305113079957664,
+            -0.0009556467994116247,
+            -0.010033016093075275,
+            0.021764053031802177,
+            -0.001455154619179666,
+            0.004359341226518154,
+            0.02081707864999771,
+            -7.774515688652173e-05,
+            0.006220633164048195,
+            0.01057181041687727,
+            0.002181711606681347,
+            0.003396041225641966,
+            0.010775899514555931,
+            -0.039315707981586456,
+            -0.0006571666453965008,
+            -0.025600925087928772,
+            -0.0018286375561729074,
+            0.0007357408758252859,
+            -0.01266984548419714,
+            0.013600491918623447,
+            -0.002015379024669528,
+            -0.010751409456133842,
+            -0.00029337796149775386,
+            0.021649762988090515,
+            0.005416522268205881,
+            0.004071575589478016,
+            -0.004469549283385277,
+            0.018792515620589256,
+            -0.006955353543162346,
+            -0.017861871048808098,
+            -0.012759645469486713,
+            -0.012367794290184975,
+            -0.01918436773121357,
+            -0.01214737817645073,
+            -0.0024551907554268837,
+            -0.002667443361133337,
+            0.004738946910947561,
+            0.002481722505763173,
+            -0.003228688146919012,
+            -0.02070278860628605,
+            0.015037278644740582,
+            -0.01430255826562643,
+            0.012873934581875801,
+            -0.011053460650146008,
+            0.005963481031358242,
+            -0.004122598096728325,
+            0.012726990506052971,
+            -0.021404854953289032,
+            0.02070278860628605,
+            -0.01536382082849741,
+            0.0038532004691660404,
+            0.019592545926570892,
+            0.006551257334649563,
+            -0.008118661120533943,
+            -0.013404566794633865,
+            -0.022074267268180847,
+            0.021633435040712357,
+            -0.011633073911070824,
+            -0.018237393349409103,
+            -0.004608329851180315,
+            0.009649328887462616,
+            -0.017470018938183784,
+            -0.007367613259702921,
+            0.01707816869020462,
+            -0.0018949664663523436,
+            0.012980061583220959,
+            0.023004913702607155,
+            -0.0052981507033109665,
+            0.0036572751123458147,
+            -0.009167678654193878,
+            0.0010286086471751332,
+            -0.02470293454825878,
+            0.013527019880712032,
+            0.008971752598881721,
+            0.006269614677876234,
+            -0.013935198076069355,
+            -0.013306603766977787,
+            0.0036531933583319187,
+            0.004498121794313192,
+            0.011437147855758667,
+            0.009420748800039291,
+            0.004220560658723116,
+            -0.013918871060013771,
+            -0.027788760140538216,
+            -0.027070365846157074,
+            0.03595232218503952,
+            -0.0005002731923013926,
+            -0.020947696641087532,
+            0.03399306535720825,
+            0.016114868223667145,
+            0.0022980421781539917,
+            -0.012449429370462894,
+            -0.004314441699534655,
+            0.024621298536658287,
+            0.016400592401623726,
+            0.010465684346854687,
+            -0.0004742518358398229,
+            0.005571629852056503,
+            0.0016469983384013176,
+            0.03012353740632534,
+            0.015323002822697163,
+            0.0009459525463171303,
+            -0.038597315549850464,
+            0.011943289078772068,
+            -0.007547211833298206,
+            0.0013408648082986474,
+            0.006702283397316933,
+            0.017780235037207603,
+            0.011935125105082989,
+            0.015567909926176071,
+            0.014865843579173088,
+            -0.003065417055040598,
+            0.0029511270113289356,
+            0.0006163488142192364,
+            -0.009624837897717953,
+            -0.001767410896718502,
+            -0.008881953544914722,
+            -0.005714492406696081,
+            -0.002920513739809394,
+            0.005592038854956627,
+            0.007657419890165329,
+            -0.003983817528933287,
+            0.028049994260072708,
+            -0.0006433906382881105,
+            -0.017600636929273605,
+            -0.009061552584171295,
+            -0.03611559048295021,
+            0.00434301421046257,
+            -0.030939893797039986,
+            -0.011257549747824669,
+            0.0033674687147140503,
+            -0.020457882434129715,
+            -0.024768242612481117,
+            0.0010306495241820812,
+            0.014335212297737598,
+            0.007392104249447584,
+            0.0033327736891806126,
+            -0.01004934310913086,
+            0.003816464450210333,
+            -0.001855169073678553,
+            -0.006461458280682564,
+            0.028833696618676186,
+            0.019559891894459724,
+            0.020457882434129715,
+            -0.01190247107297182,
+            0.019298657774925232,
+            0.005845109466463327,
+            0.02035992033779621,
+            0.010465684346854687,
+            -0.0048695639707148075,
+            0.0074859848245978355,
+            0.009567692875862122,
+            -0.006155324634164572,
+            -0.021519144997000694,
+            0.013192313723266125,
+            -0.0010837126756086946,
+            -0.0052083516493439674,
+            -0.016269976273179054,
+            -0.003534821793437004,
+            -0.015812816098332405,
+            0.010547320358455181,
+            -0.017159804701805115,
+            0.012498411349952221,
+            -0.000802069844212383,
+            0.00168475485406816,
+            -0.015469946898519993,
+            -0.013951525092124939,
+            -0.0012439225101843476,
+            -0.01244126632809639,
+            -0.008971752598881721,
+            0.006163488142192364,
+            -0.007539048325270414,
+            -0.008318668231368065,
+            -0.00535937724635005,
+            -0.011437147855758667,
+            -0.010882026515901089,
+            0.004367504734545946,
+            -0.0010653446661308408,
+            0.020931368693709373,
+            0.00011671340325847268,
+            -0.005824700463563204,
+            0.01190247107297182,
+            0.02787039615213871,
+            0.0016704685986042023,
+            0.004583839327096939,
+            -0.014261740259826183,
+            -0.011118769645690918,
+            -0.02279266156256199,
+            0.016572028398513794,
+            0.0049675265327095985,
+            -0.0344502255320549,
+            0.01438419334590435,
+            0.00824927818030119,
+            0.0015102586476132274,
+            -0.006698201410472393,
+            -0.005334886722266674,
+            -0.005608365871012211,
+            0.006306350696831942,
+            -0.0022837561555206776,
+            0.0005107327597215772,
+            0.0035919665824621916,
+            0.034678805619478226,
+            -0.0040674940682947636,
+            -0.007228832691907883,
+            0.00981259997934103,
+            0.0005130287609063089,
+            0.020196648314595222,
+            -0.005518566817045212,
+            0.0037572786677628756,
+            -0.019282329827547073,
+            -0.016563864424824715,
+            0.0018429238116368651,
+            -0.01722511276602745,
+            0.023282473906874657,
+            -0.005392031744122505,
+            -0.01347803883254528,
+            -0.006906372029334307,
+            -0.020115012302994728,
+            0.012637191452085972,
+            0.02310287579894066,
+            0.007016580086201429,
+            0.01033506728708744,
+            0.012131051160395145,
+            -0.02444170042872429,
+            -0.020457882434129715,
+            0.012171869166195393,
+            0.0013571919407695532,
+            -0.008449285291135311,
+            -0.005404277238994837,
+            -0.001783737912774086,
+            0.007783954963088036,
+            -0.0032633834052830935,
+            0.007881917990744114,
+            0.06491663306951523,
+            -0.01093100756406784,
+            -0.002987863030284643,
+            -0.02966637909412384,
+            -0.004396077245473862,
+            -0.006628811359405518,
+            0.0056287748739123344,
+            0.009151351638138294,
+            -0.023135529831051826,
+            0.009232986718416214,
+            -0.030025575309991837,
+            -0.010628955438733101,
+            0.00824111420661211,
+            0.0028490826953202486,
+            -0.01220452319830656,
+            -0.0009592183632776141,
+            -0.013665799982845783,
+            0.0006107363733462989,
+            -0.009412584826350212,
+            -0.00576347392052412,
+            0.012718827463686466,
+            -0.02321716584265232,
+            0.006298186723142862,
+            0.007592111360281706,
+            -0.006514521315693855,
+            -0.013722945004701614,
+            -0.017861871048808098,
+            -0.011967780068516731,
+            -0.013314766809344292,
+            0.002996026771143079,
+            0.01190247107297182,
+            -0.02607441134750843,
+            0.012922916561365128,
+            -0.0037389106582850218,
+            0.00945340283215046,
+            0.004404240753501654,
+            0.013469874858856201,
+            0.018074123188853264,
+            0.009355440735816956,
+            0.0021429345943033695,
+            -0.008024780079722404,
+            0.012702500447630882,
+            -0.01993541419506073,
+            -0.01635161228477955,
+            0.013837235048413277,
+            0.012016761116683483,
+            0.003061335301026702,
+            0.009828926995396614,
+            0.0020959940738976,
+            0.011886144056916237,
+            0.010775899514555931,
+            -0.019739490002393723,
+            -0.0006648199632763863,
+            0.0007326795603148639,
+            0.03280118480324745,
+            -0.0005938990507274866,
+            0.008857463486492634,
+            0.004420568235218525,
+            0.013159659691154957,
+            -0.012784135527908802,
+            -0.01769859902560711,
+            0.0008087027235887945,
+            -0.01057181041687727,
+            0.021257910877466202,
+            -0.008192133158445358,
+            0.006216551177203655,
+            -0.018612917512655258,
+            -0.027086693793535233,
+            0.006392067763954401,
+            -0.016261812299489975,
+            -0.018221067264676094,
+            0.01317598670721054,
+            0.0077268099412322044,
+            0.0017572063952684402,
+            -0.005098143592476845,
+            -0.020457882434129715,
+            -0.009837090037763119,
+            0.028441844508051872,
+            0.011102442629635334,
+            -0.004120557103306055,
+            0.009584019891917706,
+            -0.009094206616282463,
+            -0.012490247376263142,
+            -0.0017541450215503573,
+            0.0036715613678097725,
+            0.008261523209512234,
+            0.005645101889967918,
+            -0.01386172603815794,
+            -0.00029873527819290757,
+            -0.012400448322296143,
+            -0.021225256845355034,
+            -0.008906444534659386,
+            0.016751626506447792,
+            0.007820690982043743,
+            -0.0038960592355579138,
+            0.015812816098332405,
+            -0.004404240753501654,
+            0.002189875114709139,
+            0.014294394291937351,
+            0.005334886722266674,
+            -0.004285869188606739,
+            -0.0095758568495512,
+            -0.004612411838024855,
+            0.004043003544211388,
+            -0.0025960123166441917,
+            0.006457376293838024,
+            0.017192458733916283,
+            0.0072900597006082535,
+            -0.013755599036812782,
+            0.013135168701410294,
+            0.0007791098323650658,
+            0.0049675265327095985,
+            -0.03699725493788719,
+            -0.003247056156396866,
+            0.015796490013599396,
+            -0.013020878657698631,
+            0.02501314878463745,
+            0.005400195252150297,
+            0.022743679583072662,
+            0.015233203768730164,
+            0.008669701404869556,
+            -0.001542912912555039,
+            0.00464098434895277,
+            0.02759283408522606,
+            -0.003151134354993701,
+            -0.008637047372758389,
+            -0.0020602785516530275,
+            0.009241150692105293,
+            0.0002063850115519017,
+            0.003000108525156975,
+            0.008281932212412357,
+            -0.03562577813863754,
+            0.011706545948982239,
+            -0.019576217979192734,
+            0.0024694770108908415,
+            0.003459308762103319,
+            -0.007183933164924383,
+            -0.014865843579173088,
+            -0.005857354495674372,
+            0.022841641679406166,
+            -0.003224606392905116,
+            0.021078312769532204,
+            0.002885818714275956,
+            0.016106704249978065,
+            -0.011575928889214993,
+            -0.009208496659994125,
+            -0.0004130251472815871,
+            -0.005351213738322258,
+            -0.007539048325270414,
+            0.010596301406621933,
+            0.0034409407526254654,
+            0.005534893833100796,
+            0.002265388146042824,
+            0.0036654386203736067,
+            -0.0036878883838653564,
+            0.02773977816104889,
+            0.003563394071534276,
+            0.007996207103133202,
+            -0.011567764915525913,
+            0.014082142151892185,
+            -0.017029186710715294,
+            0.0070778070949018,
+            0.005575711838901043,
+            -0.026156047359108925,
+            0.016849588602781296,
+            0.004775682929903269,
+            0.006628811359405518,
+            0.013812744058668613,
+            0.01618834026157856,
+            -0.010408539324998856,
+            -0.014816862531006336,
+            0.024049850180745125,
+            -0.0032041973900049925,
+            -0.015225040726363659,
+            -0.006775755435228348,
+            4.438935866346583e-05,
+            0.006559420842677355,
+            0.00373074715025723,
+            0.0013020879123359919,
+            -0.0006204306264407933,
+            -0.010743245482444763,
+            0.004645065870136023,
+            -0.01934763789176941,
+            -0.0007163524278439581,
+            -0.016449574381113052,
+            -0.015633217990398407,
+            0.018678225576877594,
+            -0.01830270327627659,
+            0.009633000940084457,
+            0.0012275953777134418,
+            0.028588788583874702,
+            -0.011706545948982239,
+            -0.012155541218817234,
+            0.0015102586476132274,
+            0.010849371552467346,
+            0.0067512644454836845,
+            -0.010033016093075275,
+            -0.007583947852253914,
+            -0.014180104248225689,
+            -0.010163633152842522,
+            0.015502601861953735,
+            -0.014498483389616013,
+            -0.014343376271426678,
+            0.008914608508348465
+          ],
+          "embedding": [
+            -0.006359657085039302,
+            -0.024594786241399234,
+            -0.02553538924848095,
+            0.005360188465088291,
+            0.012801870977589839,
+            0.021680505862506104,
+            -0.016668048966001273,
+            0.06860852474581301,
+            -0.007122272841157724,
+            0.03471775000705488,
+            -0.009004297848861675,
+            0.01059974144678391,
+            0.02565604599426148,
+            -0.022544127646041287,
+            0.010616674976540785,
+            0.013658194823359737,
+            0.016400329743363844,
+            0.02858546964917478,
+            0.0067612676434693686,
+            -0.04174737193606879,
+            0.03184155110488616,
+            -0.02223105127585514,
+            -0.05506239667974977,
+            -0.0015307551410555752,
+            -0.013169788751379287,
+            0.017681749722384007,
+            0.01682706317229353,
+            0.019652295823816928,
+            -0.05623980369271269,
+            0.030698582851533197,
+            0.02286327124296613,
+            0.016019032539689178,
+            0.005508113353872315,
+            8.58985091215149e-05,
+            -0.019206292853495906,
+            -0.018439290483495144,
+            0.03475023176950151,
+            0.01018612942141759,
+            -0.0252551158857426,
+            0.0334562130666827,
+            0.020462153537485785,
+            0.02260508249714211,
+            -0.03323524989392951,
+            -0.002094251009530707,
+            0.015177442347686773,
+            -0.00018698464350038317,
+            -0.0006788648344273403,
+            0.0007777616893581039,
+            0.013352398615059647,
+            -0.030751111528744813,
+            -0.0317733818445192,
+            -0.006729064081710058,
+            0.009413204899986514,
+            -0.005712894407943592,
+            -0.038961826880527066,
+            -0.002827692206530345,
+            0.017863300537973137,
+            0.07918455388563496,
+            0.048967158919816366,
+            0.018002566325350088,
+            0.040264962620950695,
+            -0.019715100359659787,
+            -0.02211350500879408,
+            0.004918148608475574,
+            0.01062585305866606,
+            -0.014478993521696027,
+            0.0034983131226364672,
+            0.000766228629514787,
+            0.009667268771502067,
+            -0.01558439890697323,
+            -0.04984718709870083,
+            0.029613955432821887,
+            -0.009686854114630624,
+            -0.015279319238124568,
+            0.007460629676325322,
+            -0.006235979350809034,
+            0.012514387473717085,
+            0.02800435263741497,
+            -0.008688925946672778,
+            -0.01236301543374469,
+            0.029755705020188972,
+            0.01158037219446542,
+            -0.005242392206024511,
+            0.009982679027728814,
+            0.052414361714483264,
+            -0.018788500381726227,
+            -0.020335831196027204,
+            -0.08583959943356621,
+            0.029309228216812346,
+            -0.0249285282010673,
+            -0.018929396555426307,
+            0.007710663193334574,
+            0.03096798035958051,
+            -0.017089977064591828,
+            -0.028015107605405817,
+            0.06740703271128606,
+            0.009398795412354399,
+            0.01118769022826835,
+            -0.03041417460855722,
+            0.014776002883150835,
+            0.004319194067673161,
+            0.022507636639041203,
+            0.0053562788646786845,
+            0.007548072258943124,
+            0.011123374308563128,
+            -0.006924362115585165,
+            -0.011118507147092024,
+            0.009563232643295304,
+            0.00657178473678069,
+            0.08313027447380966,
+            -0.004239370156003264,
+            -0.00930406163946907,
+            -0.039202672821898946,
+            0.009761029930556298,
+            -0.02827698465424993,
+            -0.037914533246900244,
+            -0.04496050911855815,
+            -0.017711307900236353,
+            0.01934435542287117,
+            0.004335173027873153,
+            0.03800509969714242,
+            0.031198127357716314,
+            0.031311207941680276,
+            -0.012184553894588813,
+            -0.058985863670856876,
+            0.041442017057707635,
+            -0.012456189292721711,
+            -0.0040966351720394135,
+            0.010849179519959421,
+            0.005243123049347636,
+            0.0011495799963862647,
+            0.003534183676934487,
+            -0.03139340512003631,
+            -0.023876261883905306,
+            0.008382957569752911,
+            -0.004778283217011976,
+            0.03703067589888386,
+            0.06888630688839406,
+            -0.03993532296478411,
+            -0.009472167678222372,
+            0.025246872942105057,
+            -0.01524567912700368,
+            0.01595917939878582,
+            -0.003736723164442467,
+            -0.030836797784967753,
+            0.0026246446163368533,
+            0.008894300082148884,
+            0.08342856279154155,
+            -0.0007476580521704578,
+            0.025149116238550588,
+            0.004972542633631468,
+            0.034495632546556576,
+            -0.009173329764686424,
+            0.014009381543772768,
+            0.004427261668151028,
+            -0.013426138607902904,
+            -0.022234568880321808,
+            0.01976282364689966,
+            -0.03592433789490043,
+            -0.010918278220831188,
+            0.013527180289641174,
+            -0.013023113052242677,
+            -0.008304770302495363,
+            0.003677063799167178,
+            0.0013981822663219421,
+            0.047001163242600175,
+            0.01828332714887986,
+            0.002483375662343689,
+            -0.06820498187523893,
+            0.05295312038995555,
+            0.025038753504342162,
+            0.025286870198436735,
+            -0.0029544808216316115,
+            0.029982752196635428,
+            0.0005600481523874767,
+            0.0128290258663539,
+            0.009023805227424919,
+            -0.0019219166367750596,
+            -0.04482843936478531,
+            0.04523479691409549,
+            -0.004678574027241186,
+            0.010486850804218904,
+            0.006736229865018014,
+            0.02134813656476443,
+            0.028083972673690036,
+            0.016857413440125445,
+            0.0189082447820146,
+            0.03843690613820022,
+            -0.011922027011229613,
+            0.016193433524453224,
+            0.03828207911467065,
+            -0.0013055128404331705,
+            0.043198163443700956,
+            0.024025562893260542,
+            -0.001257851946749882,
+            -0.011726713433348844,
+            -0.021907984099443396,
+            0.02752198420597673,
+            0.004431866336999851,
+            0.015798806714413912,
+            -0.012910782191382367,
+            -0.03180549180206364,
+            0.004550247372715741,
+            -0.022778272841118514,
+            0.031242218240017532,
+            -0.006106984949693483,
+            -0.006259993190497421,
+            0.02187449047033339,
+            0.009334146847221406,
+            -0.009381278960016145,
+            -0.020888853823931134,
+            -0.007502374908776156,
+            -0.010390018386728482,
+            -0.04316693435221676,
+            -0.006274513730720055,
+            0.019152443937064258,
+            -0.01668984824845791,
+            0.018573194036505578,
+            0.0035035262183249925,
+            -0.019779911381789077,
+            -0.0027495798165085993,
+            0.02425481236315913,
+            0.006857723545270935,
+            -0.014733535699087731,
+            -0.026326146471941727,
+            -0.019906660231056833,
+            -0.001033294559612807,
+            0.0159154063026895,
+            0.035649534442431415,
+            -0.026210592096240003,
+            0.014397127229517285,
+            0.005114311681443673,
+            -0.0033104236005207077,
+            0.00324988235167305,
+            0.03764004154553922,
+            -0.017822437760737605,
+            -0.008579701063708334,
+            -0.00032410984200965487,
+            0.018450351071676443,
+            0.017283379764846195,
+            0.006384411611332382,
+            -0.005115090309123598,
+            -0.030590047062583125,
+            -0.011776211359193503,
+            0.01183623914060493,
+            0.0016564847796208331,
+            -0.009530766634361575,
+            -0.004810598258154945,
+            0.03876242515526699,
+            0.027076785774401667,
+            0.005324763136165989,
+            -0.03416412480913939,
+            -0.007943016733586697,
+            0.002114307650456928,
+            -0.02961449347300628,
+            0.05142732770518828,
+            0.05566026433842563,
+            -0.0030729309664974704,
+            -0.00938186948393074,
+            -0.04547908981236149,
+            0.0651216359692231,
+            -0.01813722445944094,
+            -0.0482650853560993,
+            0.044045155892545554,
+            -0.014272653003441165,
+            0.013740652306746851,
+            0.04563732439184395,
+            0.02126513340521316,
+            -0.0054845632439384905,
+            -0.011635625101262898,
+            0.030992082069685376,
+            -0.010051935721068779,
+            -0.050967258971854425,
+            -0.009827942095033494,
+            0.013373878556654464,
+            -0.013017627321534534,
+            -0.009361001542607812,
+            -0.01925867665425919,
+            0.020329052915436022,
+            0.023020741670660188,
+            0.019387043947695172,
+            0.005628632318152202,
+            0.018451555161952665,
+            0.016226845400848455,
+            -0.01384033112836216,
+            0.004762589697426251,
+            -0.012718476561725138,
+            -0.024222765127494463,
+            -0.037487144314010595,
+            -0.019971524166659278,
+            -0.015034588610516958,
+            0.009483936496215338,
+            -0.014301924283553591,
+            -0.0024139665139412125,
+            0.013361045353529458,
+            -0.0005870798721107379,
+            -0.011192319340879789,
+            0.013927695434391159,
+            0.021337539669342293,
+            -0.027300952495815437,
+            -0.04765670695199488,
+            -0.00561706224251562,
+            0.0082776639906139,
+            -0.0030984973749633724,
+            -0.026269016057688548,
+            0.007040524859505573,
+            0.0017859008072463442,
+            0.020091542517503808,
+            -0.018716761130867725,
+            0.0035283051960672963,
+            0.008519383548064053,
+            -0.015247417050336866,
+            -0.016917626868166634,
+            -0.018580550690103854,
+            -0.024697180580294884,
+            0.034202028316737125,
+            0.0204395003463227,
+            0.030392895534267717,
+            -0.04527669147753553,
+            -0.018359428092075348,
+            0.05494107499558624,
+            0.015234754907198422,
+            -0.05066008106884831,
+            0.031139824660661085,
+            0.005016615710206006,
+            0.0019000453624929544,
+            -0.006094364589229031,
+            0.0637324238017742,
+            -0.02711381489292883,
+            -0.02187418906973867,
+            0.004601744864532875,
+            -0.04141121277496072,
+            -0.026689585608019588,
+            -0.005568100750837504,
+            0.01291019036002459,
+            -0.031853041426355644,
+            -0.02815800259372657,
+            -0.007964861641790374,
+            -0.010933515363060793,
+            0.013726639077745378,
+            -0.023956800669647334,
+            -0.015672058590455337,
+            -0.023642025374517057,
+            -0.01288435863544158,
+            -0.0042516881552192385,
+            -0.011162214160460695,
+            -0.02412733256710621,
+            0.010088528028211243,
+            0.03310175458095482,
+            0.023064360903748363,
+            0.001996159684818467,
+            0.009478229572933101,
+            0.014478772877286076,
+            0.00371783700962415,
+            0.007864103610320116,
+            -0.048511515124609,
+            -0.031398754206534776,
+            0.020227872561659397,
+            0.012159811587094092,
+            0.0172163448580163,
+            0.009578952796095119,
+            -0.04154545961112031,
+            -0.05095080393682759,
+            0.02968486144855133,
+            -0.03932395023720674,
+            -0.02063219739061335,
+            0.007512423249865194,
+            -0.011911709468952548,
+            -0.00810951734088105,
+            -0.0008079327452253181,
+            -0.01600701450676789,
+            0.023984909975645632,
+            -0.02418072541081482,
+            0.0033984265728439398,
+            -0.0104343782811971,
+            0.01588977364765743,
+            0.05109529246167992,
+            -0.027800269136366997,
+            -0.027889034481533438,
+            0.03706813210665186,
+            -0.02058120652429308,
+            0.030801396468451574,
+            0.03246127847621613,
+            0.008355517405358826,
+            -0.008605467586247932,
+            0.017693344563968573,
+            0.014499487024471321,
+            0.012431277080478217,
+            -0.040103870071440256,
+            0.017328848567062113,
+            0.02723900253727976,
+            -0.054909074068976875,
+            -0.008547596270650966,
+            -0.00777964838164648,
+            0.027134666572242665,
+            0.02765036594970255,
+            -0.002207173084941375,
+            -0.016838740117653725,
+            -0.028289233739722936,
+            -0.019433376533829132,
+            -0.011053380458946764,
+            0.03247641283550457,
+            0.0007514989009289282,
+            0.007919574219000797,
+            0.001316042436488591,
+            -0.0017934342041762042,
+            0.01924145734197619,
+            0.015260199553572459,
+            0.048259602293134723,
+            -0.03383050622747848,
+            0.0021712225589426266,
+            -0.03508567715696004,
+            -0.04578731401250471,
+            -0.008768228935292693,
+            0.010203318819548585,
+            0.024135236371743855,
+            -0.007467140460133838,
+            0.0028907962385387818,
+            -0.007582083854422456,
+            -0.005384477718912037,
+            -0.01867978482547249,
+            0.01641191108381631,
+            0.037168089880543244,
+            0.0031992495663980464,
+            -0.008533890994695162,
+            0.000778843942291118,
+            -0.004620258538699103,
+            0.010572690335163528,
+            0.033233014627100904,
+            -0.030497362482602697,
+            0.02974571152880676,
+            -0.004274442492343003,
+            0.025341347125910928,
+            -0.04307748954707389,
+            -0.018069609916024403,
+            -0.018491122727274337,
+            -0.02069307802429807,
+            0.05372967226264594,
+            -0.030768978301976772,
+            -0.03662045886993282,
+            -0.003205991669059688,
+            0.016815511161004534,
+            -0.0014152644009276566,
+            -0.0016015439520090104,
+            -0.03984716970293867,
+            -0.010828260334550217,
+            -0.05894985435251326,
+            -0.01638035185618134,
+            -0.021819442096404948,
+            -0.005315102328745982,
+            -0.06251579337157112,
+            0.0033713876515840233,
+            -0.07815974854883057,
+            0.001484860071649606,
+            0.04539427129131382,
+            -0.023530943197353476,
+            0.05142250021536681,
+            -0.024746600229063383,
+            -0.0005885740455257141,
+            -0.044698536284421976,
+            0.0026481001040031785,
+            0.02389178603305878,
+            0.051387657872255674,
+            -0.002949198236157656,
+            0.010674377196609455,
+            0.013322075519401089,
+            -0.026245090721950647,
+            -0.01036724558050635,
+            -0.005321884180696542,
+            0.005958798671856245,
+            0.013755178085972732,
+            -0.0074734329537091675,
+            0.038368016046563674,
+            0.010941421833174982,
+            0.03443115781919206,
+            0.034623617136788874,
+            -0.004361926531008214,
+            -0.03348681614127998,
+            -0.00632870804659212,
+            0.024433331840566692,
+            0.0321998807702602,
+            -0.02899983280320611,
+            0.005958471722097962,
+            -0.015602447981516634,
+            0.015633217539747047,
+            0.006321931116948646,
+            -0.011827541320878442,
+            0.02372204564999124,
+            0.015673426811623124,
+            0.006103300725621424,
+            -0.005165006670545588,
+            0.013589526056189892,
+            -0.025411864292353578,
+            0.03041791947375232,
+            -0.004456865298584878,
+            -0.04276058054583262,
+            -0.0024736078733106825,
+            -0.020059309867945227,
+            0.024883721794000956,
+            0.015578237413840115,
+            -0.00536523508503184,
+            -0.005290645798641171,
+            0.008242987948511412,
+            0.020469675635821673,
+            0.006668354647245447,
+            0.00017729517736814364,
+            0.02373604785121342,
+            0.029915687122900485,
+            -0.02432635861789771,
+            -0.029540533943782886,
+            0.008151297444985344,
+            -0.005730733861939129,
+            0.011318816579801282,
+            -0.02644855229221747,
+            0.00033932872948409137,
+            0.0035505684154028996,
+            -0.03570104343808711,
+            0.03408366016010711,
+            -0.009199700348630527,
+            0.005857598530788056,
+            -0.009538177736472163,
+            -0.0017768806148210118,
+            0.0027410153099351694,
+            0.03898108970435115,
+            -0.007367623392751247,
+            0.009827254226889936,
+            0.015465895035342481,
+            0.000981604396724216,
+            0.05143083558797524,
+            -0.027350877485564314,
+            -0.02183036962301852,
+            0.01885468378168636,
+            -0.007285239007489837,
+            -0.016846837665812908,
+            -0.020546383007536074,
+            -0.011863943814684079,
+            -0.016293192750336218,
+            0.01976516340568883,
+            -0.02387328798681798,
+            -0.011892336568452852,
+            0.007170651391639504,
+            0.0013725014050721622,
+            -0.009809669068563384,
+            0.0016617183704652932,
+            -0.006307590349798603,
+            0.02037693835615797,
+            0.031753656615094446,
+            -0.024892675105483313,
+            0.03577919637821846,
+            0.03178303188315581,
+            0.017260898395251155,
+            -0.02102289690701718,
+            -0.009661035772073294,
+            -0.00048542689442823136,
+            -0.014694631967051945,
+            -0.0326379186158631,
+            0.025443206465204822,
+            0.017194626456982255,
+            -0.011381421003142453,
+            -0.03680783761649673,
+            0.0017216115811536721,
+            0.022819859205829785,
+            -0.013600989058799166,
+            0.028141657788496834,
+            -0.01869047298692401,
+            -0.01366833755869951,
+            -0.00782959271979894,
+            0.031227914441058244,
+            -0.05252392878125891,
+            0.016957842026348195,
+            0.00020880696574616416,
+            0.001154370575234767,
+            0.010892067708194324,
+            -0.008368819756628046,
+            0.012224564683325483,
+            0.031700389750709,
+            0.018063405869147564,
+            0.0010794115307202266,
+            -0.002783668216641561,
+            0.01418234057248712,
+            0.017773269840982066,
+            -0.015653443355569053,
+            -0.000497740485657824,
+            0.019890882604100452,
+            0.022672054781919238,
+            -0.00521797370595221,
+            -0.016686942830774276,
+            -0.018443538351123584,
+            0.025270057639201107,
+            -0.010354431070738671,
+            0.03484047753830186,
+            0.04594785214094975,
+            0.0036441959528283004,
+            -0.05695476763071743,
+            -0.009412209619190693,
+            0.0010371110229966086,
+            -0.01099343927053564,
+            0.01606863921935641,
+            0.012470660511773616,
+            -0.03237187923799263,
+            0.006388672213167589,
+            0.012946570716567364,
+            0.022152554488829627,
+            -0.02540317621005867,
+            -0.023183252921885226,
+            -0.0030984019855787655,
+            -0.007890467768631811,
+            0.018648760402624687,
+            -0.008653405980661855,
+            0.0006228315253506687,
+            0.009672526897422602,
+            0.0005436494014569835,
+            0.011632326145022935,
+            0.005655942007108228,
+            0.021736827172951,
+            -0.001256608167230203,
+            0.013124413886287987,
+            -0.01986808920858981,
+            -0.0270450339809516,
+            0.006500423802582728,
+            0.04438242119211055,
+            -0.014813399694814117,
+            0.030621709833336064,
+            0.0092171224268983,
+            0.016433384739642534,
+            0.024875072913970443,
+            -0.00891470467241686,
+            -0.014124009014664287,
+            0.00845777541223868,
+            0.009019007848927453,
+            -0.04147358041382793,
+            0.012513030150663913,
+            -0.01710243399313324,
+            -0.033144939719206276,
+            0.01904674924133127,
+            -0.00534122554342279,
+            0.012367320751926314,
+            0.004270315800911569,
+            -0.002886460711688153,
+            -0.025696286516581048,
+            0.014137901708875033,
+            0.028135087538950946,
+            0.021797869775098042,
+            0.0296173052503884,
+            -0.012757078368361444,
+            0.0005460601780756187,
+            -0.008748930817524899,
+            0.005470975782187157,
+            0.04187083128854477,
+            0.0008687182830616019,
+            -0.0008032802520818105,
+            0.01507726552487313,
+            0.024945744517309208,
+            0.0376130540370386,
+            -0.015415560911023553,
+            -0.03407353556457147,
+            0.01564969972264396,
+            -0.010104443707865262,
+            0.018855792875436674,
+            0.005503259932317811,
+            0.0049634097823351596,
+            0.0038600156268000702,
+            0.019312419743287703,
+            -0.006635193789316492,
+            0.018798045806429395,
+            -0.02500983683871332,
+            -0.028785757874583755,
+            0.01829411751138161,
+            -0.005277972443258226,
+            0.03338623299668019,
+            0.016225885095995672,
+            -0.029522280454919586,
+            -0.01712097344863975,
+            0.01553706187924322,
+            -0.008465766956343768,
+            -0.024842700926275538,
+            0.01895228828512998,
+            0.007076862934523977,
+            0.008956105593078965,
+            0.0024373779389833218,
+            -0.01106503696703573,
+            0.01125466547367417,
+            -0.000818223040067914,
+            -0.015322790791698395,
+            0.014157131555941659,
+            -0.017853386731281715,
+            -0.014994640991761465,
+            0.01775991780643474,
+            -0.037498078532902435,
+            0.009570197712419421,
+            -0.007687297543935123,
+            0.009050494195378254,
+            -0.004222041469820445,
+            0.007460383234429904,
+            -0.0010804379186060264,
+            -0.014987857934522812,
+            -0.014903749199810126,
+            0.03228485430019625,
+            0.022096121825475306,
+            -0.000410816936759558,
+            0.006728733918931589,
+            -3.971636461500232e-05,
+            -0.026719848032785824,
+            -0.011253751790407687,
+            0.02245467425830765,
+            -0.005664635254398481,
+            0.005134496095445943,
+            -0.01109590855846966,
+            -0.0149616794609307,
+            0.0019726017255571673,
+            0.032668637270108894,
+            0.06066635618119076,
+            0.012001732730659747,
+            0.024422093585231484,
+            0.0189499396761026,
+            0.02119276033547528,
+            -0.008489818917377516,
+            0.006413901834269739,
+            -0.03097413287789443,
+            0.03509225573986143,
+            0.005392594635384506,
+            0.03108937318306421,
+            0.026664331248813712,
+            -0.04722525921137681,
+            -0.0058539171830594774,
+            0.017087531454837677,
+            0.022673712717885323,
+            0.0159020681868943,
+            0.0008382498574190522,
+            -0.008938174268517353,
+            0.035650940044285984,
+            -0.03315250299348084,
+            -0.023647111263155,
+            0.005059382024092534,
+            -0.01439537467317451,
+            -0.010488359956731476,
+            -0.006785295580877329,
+            0.02776449681317373,
+            -0.005584052419598926,
+            -0.004582019433008779,
+            0.009697064252638535,
+            -0.014400206558190682,
+            0.008165676294736836,
+            0.0007624778531368436,
+            0.008681263897172326,
+            0.039249866981334126,
+            0.0005918063050375113,
+            0.024275699182767528,
+            -0.004138313072843555,
+            0.011079529739650594,
+            -0.0158972266904151,
+            -0.016985058543605,
+            0.02276982363172991,
+            0.00351101120496393,
+            -0.03244126813432563,
+            -0.00768819222024939,
+            0.00928180979884261,
+            -0.0013793404539641824,
+            0.005266058566444064,
+            -0.00503433670609399,
+            0.003747448261046857,
+            0.03286230666764075,
+            0.0022383592721685153,
+            0.008606190020480513,
+            0.013593947677257625,
+            0.00030951915677929204,
+            -0.01490307680740127,
+            0.013254591138113545,
+            -2.1379308136707722e-05,
+            -0.012900073369514622,
+            0.03357519186606058,
+            0.006705902094864628,
+            0.04243347200485736,
+            -0.03302953674711988,
+            0.009031534935013593,
+            0.01822227379789576,
+            -0.008491770335368761,
+            -0.02725412856390648,
+            -0.03630788717786838,
+            0.008259145747740535,
+            -0.019003260334340084,
+            0.015781275447792516,
+            -0.010039277287105153,
+            0.029665124310477384,
+            -0.03563370794521629,
+            0.004714858110268169,
+            0.004053913355055313,
+            0.0009503673086552651,
+            0.02068360255395961,
+            0.018524093113914366,
+            -0.002785698934204024,
+            -0.031443835341186485,
+            -0.02066365433385339,
+            0.006869321907853112,
+            0.02431260052207595,
+            -0.010367048093621498,
+            -0.005612149485859478,
+            0.0006316058139530722,
+            -0.024369562712568606,
+            -0.016845124984022828,
+            0.030359419843284823,
+            -0.040974427176153246,
+            0.005252911449822132,
+            0.04272936053727608,
+            -0.04599092715016919,
+            -0.03802288716943514,
+            -0.002314988728575789,
+            -0.024124014894379466,
+            -0.006360771882090754,
+            -0.002068030981218464,
+            0.016669991113175926,
+            0.0036232707799735537,
+            -0.028763764013482795,
+            0.00943401978066451,
+            -0.005173988601101547,
+            0.003971420424839974,
+            -0.0031669592494977393,
+            -0.02216336363004636,
+            -0.01657772032717298,
+            -0.02531962163186398,
+            -0.035174102848865116,
+            -0.0015903434876311088,
+            -0.005158703150821607,
+            -0.028703075279214722,
+            -0.03941940849540492,
+            -0.021696000925262537,
+            -0.0160221488026189,
+            -0.00843213135542359,
+            0.040641882367348774,
+            0.021319060666253945,
+            0.019336844555345886,
+            0.0025252167889382917,
+            -0.020346900609125973,
+            0.049440028814233025,
+            -0.007642182460707816,
+            0.035628691886108926,
+            -0.0017882888000877158,
+            -0.013461669667950893,
+            0.00455558389906464,
+            0.03825428026120858,
+            -0.014248434654950971,
+            -0.0009902062564027718,
+            0.00027811655483232545,
+            -0.0011866253757890136,
+            0.0008535155950871286,
+            0.015433650396777696,
+            0.001307651096343185,
+            -0.007112199543314309,
+            0.030380673552683906,
+            -0.012803247194895328,
+            0.01637501752192592,
+            0.018350814869962304,
+            -0.0013359719137099667,
+            0.008064546833360553,
+            -0.021430659268086814,
+            -0.00731572118780415,
+            0.0044578576331236325,
+            0.008422210247967642,
+            -0.015096885054883586,
+            -0.009964588397149773,
+            -0.02359494467299634,
+            -0.01174145797396426,
+            0.0023855749166744366,
+            -0.004559202896782688,
+            0.004902138778592298,
+            0.034595817216889944,
+            -0.016068744500427584,
+            -0.035106033294403614,
+            0.009997207702122002,
+            -0.02663088875167486,
+            0.02222306406130687,
+            3.983384942889326e-05,
+            0.006838221769859366,
+            -0.0023434342406307616,
+            0.001765445386886839,
+            -0.00169484278667461,
+            0.002862331185439823,
+            -0.018217975204395134,
+            0.0031859734246840257,
+            -0.0032119988065673565,
+            0.0009052950782312657,
+            -0.0012459653052875638,
+            0.00687339892509406,
+            0.006332799551164718,
+            -0.0008019538898592668,
+            -0.002968451740267742,
+            -8.541086355097327e-06,
+            -0.020561816931299912,
+            -0.021642512405352372,
+            -0.028259049261410905,
+            -0.000989887278779396,
+            -0.012359733017834338,
+            -0.027804394906433664,
+            0.011443385424045703,
+            -0.006872684999639775,
+            0.008327844771213648,
+            -0.008969527138034904,
+            0.006219994886600349,
+            -0.008415106130309062,
+            -0.0004143996206881767,
+            0.013119286511794273,
+            -0.02925209282917943,
+            -0.02096907906398536,
+            -0.03096513357712367,
+            0.01762223841790055,
+            -0.00925741670612824,
+            -0.002656382260603827,
+            0.0025223527960181835,
+            0.005092659166535706,
+            -0.033606542487570154,
+            -0.006653438998497412,
+            0.007313682256676618,
+            0.009547290751154595,
+            -0.03287788897629887,
+            0.02446728061355753,
+            0.017023222076526946,
+            0.01874233066565012,
+            -0.02417566312664911,
+            0.04049827067897005,
+            0.014426879325647042,
+            -0.021740606525480655,
+            0.016712907564723732,
+            0.027385150577780248,
+            0.03177163221499527,
+            -0.03469098547555545,
+            0.004481376248622418,
+            -0.01518200063132508,
+            0.018918680020516044,
+            -0.025284921244460287,
+            0.020625170445627613,
+            -0.016707629602243063,
+            0.021278719363178687,
+            0.03520334001264928,
+            -0.005524369131116028,
+            0.03608133014180745,
+            -0.019234548780647943,
+            0.0059985801736444555,
+            0.004041778461022664,
+            -0.01645503683802124,
+            0.01024694092614158,
+            -0.0042700351372843386,
+            0.0006059159967286885,
+            0.009406090340935747,
+            0.019133229156115282,
+            0.019007219188041775,
+            -0.04442208547053466,
+            0.028588965123434923,
+            0.00804120101594811,
+            0.021472086070876265,
+            0.014177820346835521,
+            -0.02708807970982664,
+            0.004742981711172693,
+            0.002072673190279628,
+            -0.004659713164250794,
+            0.0061167904730676514,
+            -0.013796604518930328,
+            -0.034540857808309226,
+            0.006338792319814448,
+            -0.018918703778187926,
+            0.015036937364271984,
+            -0.009026670935442384,
+            0.0011667525636770179,
+            -0.01152652811400482,
+            -0.011938679695598347,
+            0.0006170592505446954,
+            -0.005495151258143897,
+            -0.0008075736908949407,
+            -0.03171721592827769,
+            -0.04427658539412448,
+            0.02036132271908089,
+            -0.0021221965958423804,
+            -0.04320396743189005,
+            0.004111179168317842,
+            -0.025352428814134187,
+            -0.01865918186335911,
+            -0.011237699157960728,
+            -0.025047923887906474,
+            -0.04219630696740606,
+            -0.011108496740166623,
+            0.007785006066683012,
+            0.016810642610918525,
+            -0.009081815651299525,
+            0.040965574143584864,
+            0.0073944364393447965,
+            0.015313576815177263,
+            -0.017731360025694955,
+            0.026137777790243826,
+            -0.019314392060123,
+            -0.03108486330987073,
+            0.02676829566273746,
+            -6.259161089416372e-05,
+            0.003053748984540415,
+            0.014358356168574803,
+            2.8186924071913628e-05,
+            -0.006181789640967465,
+            0.01947730374003864,
+            -0.028739300217145797,
+            -0.023460503178221397,
+            0.0012460072421725688,
+            -0.008850395570726385,
+            -0.016271242375881602,
+            -0.035896509136022194,
+            0.005227881053956234,
+            0.03324337105109496,
+            0.004970419987430122,
+            0.008273497228454482,
+            0.01814500594735864,
+            -0.02229593571615308,
+            -0.024230675455380678,
+            0.000639112529814716,
+            -0.010266640429960994,
+            -0.010257247303470066,
+            -0.021302483296731352,
+            0.019150757066356872,
+            0.01651375496287087,
+            -0.00390189952706486,
+            -0.006971692709239087,
+            -0.010621003777733253,
+            -0.01918206725910585,
+            -0.02020345113211036,
+            -0.013827322378214754,
+            -0.017980206764311026,
+            -0.034153741666571154,
+            -0.032195001867229935,
+            0.03225205339201753,
+            0.026354817588919596,
+            0.0014076409241492998,
+            -0.007959081015805435,
+            -0.023070419006792155,
+            -0.004483328183581872,
+            0.02879516595345542,
+            0.002549934782150634,
+            -0.0001541849716285838,
+            -0.00830090759167373,
+            0.015341095409025125,
+            0.00835695019244165,
+            0.005107844984733648,
+            -0.06676974053142512,
+            -0.006027540973808174,
+            0.01655283585235268,
+            0.01351155740684989,
+            -0.002359114145814062,
+            0.02591301169926259,
+            0.01108099264396926,
+            -0.015361038608114896,
+            -0.019765743462262668,
+            0.035195761515422364,
+            0.02408910733949848,
+            0.023468767772473746,
+            -0.0034492999385544727,
+            -0.004472311490618405,
+            0.007030833258457338,
+            -0.010684252427867153,
+            -0.013326964647936429,
+            -0.004558513842504658,
+            0.026925146859337397,
+            0.016099443021514896,
+            -0.0064331198280825575,
+            0.010848931851939124,
+            -0.0008320209125061849,
+            -0.019444491995041537,
+            -0.001841330829156697,
+            0.008730275423516445,
+            0.0012974082030866743,
+            -0.014611902493400719,
+            -0.005911807768916134,
+            -0.019130604101278263,
+            0.024719468317569562,
+            0.0209456329723249,
+            0.03381727436368338,
+            -0.013237902591972769,
+            -0.0003695142331921278,
+            0.005410360755058108,
+            0.0002841632253252816,
+            0.0023352645111704958,
+            0.004801123870325242,
+            0.02623649813035263,
+            -0.029010710171522985,
+            -0.020741774066460415,
+            0.0068426980682519505,
+            0.013445270043414486,
+            0.0022744032984779294,
+            0.033840485942696216,
+            0.035040080489222324,
+            0.004166066423508316,
+            -0.010046666684988869,
+            -0.006148765564658713,
+            0.007071740742464309,
+            0.01865897335451031,
+            -0.023267411753900076,
+            -0.0005936657905496225,
+            0.004833030519903764,
+            -0.004229582648167378,
+            0.03620610659071198,
+            -0.009961238880412731,
+            -0.007197670122096852,
+            0.022212791947347127,
+            0.09819205436926473,
+            0.00047695621444943017,
+            0.0025354521225913395,
+            -0.029358210234742176,
+            -0.023614016211567963,
+            -0.0012564198366457442,
+            0.00242680708457405,
+            0.00830167274308547,
+            0.028076260517106013,
+            0.0004852400049405906,
+            0.011579274037419117,
+            0.02773972145320728,
+            -0.017217305534161797,
+            0.006752070301141707,
+            -0.011376051474975524,
+            0.024577071515205502,
+            -0.012694034026935636,
+            -0.0019809719792138673,
+            -0.016512649783050146,
+            0.015907402258634816,
+            0.004556574687833768,
+            0.004907510649166062,
+            0.011347630984408937,
+            0.009824484201337644,
+            -0.02650731900265866,
+            0.01756773113723491,
+            -0.005671986127099093,
+            0.004449611508541289,
+            -0.027841346572384462,
+            -0.020124233970502267,
+            0.0025986391930124847,
+            -0.0014440758308989809,
+            -0.009579444248630085,
+            0.006005274872617889,
+            -0.0024984235450999127,
+            0.03765664613203687,
+            -0.03765496603611941,
+            0.01409350051861866,
+            0.0066453175608372555,
+            -0.0015390841089599882,
+            0.011806653942640485,
+            0.017306914776093522,
+            -0.008206256939290151,
+            0.02589800448884443,
+            -0.00798213351353373,
+            0.009846274212000523,
+            0.015230070803055296,
+            -0.03449531102975745,
+            0.009174739184806312,
+            0.010145762060174144,
+            -0.024165660737765746,
+            0.012146877603301336,
+            -0.013784371016298891,
+            -0.004818759788054113,
+            0.005930446700613062,
+            0.007349837554764034,
+            0.010733731974181452,
+            -0.02295571464086674,
+            0.016176424826629898,
+            -0.010947841926210105,
+            0.0037766665765835445,
+            0.013559846998199008,
+            0.024892849808979323,
+            0.0012426907454129122,
+            0.0005950515136086193,
+            -0.01379469431504576,
+            0.005534724429523084,
+            -0.015667467221535825,
+            -0.022401586867694584,
+            -0.014192041775234079,
+            -0.006366846766730035,
+            0.01029598687444803,
+            -0.003972386708834858,
+            0.016971476088808737,
+            -0.010309648640697812,
+            -0.010645860303528099,
+            0.0037818310327362103,
+            0.01040806357838676,
+            0.008316605962234849,
+            -0.02855914079113881,
+            -0.005676310556150579,
+            0.004961084138163161,
+            0.00029285971503001265,
+            0.009592497954559435,
+            -0.014556073011576487,
+            0.013689526487160009,
+            -0.013290113354287095,
+            -0.0010726537498214612,
+            0.013201860397060236,
+            -0.012443552043930505,
+            -0.004012305744087929,
+            -0.01592905426965635,
+            -0.028487066798154617,
+            0.008462143660968396,
+            -0.01365614159523485,
+            0.001076882584445294,
+            0.0019503240125505646,
+            0.004730085646042709,
+            -0.016919069570642004,
+            -0.007860046623166481,
+            -0.033515448866521046,
+            0.011065416478687504,
+            0.02314589687882036,
+            -0.006435733825418645,
+            -0.030351727858789472,
+            -0.009546333526852862,
+            0.011327489786114963,
+            -0.0071765122743172715,
+            -0.01676736383051139,
+            -0.0029390038145944096,
+            0.007617910650439855,
+            -0.008174620754097493,
+            -0.00508943241317381,
+            -0.0025173969010414098,
+            0.0217328037301181,
+            0.016094975605203832,
+            0.01769642142712967,
+            -0.021400862603034036,
+            -0.00976992221747187,
+            0.026043879119776864,
+            -0.03275550610951898,
+            0.012441465975914529,
+            0.024587446266013183,
+            0.003446527051428522,
+            0.005492228054838714,
+            0.0030377554109693714,
+            0.005459159163772825,
+            -0.006985179226831048,
+            0.019925583798328282,
+            0.010870421035240845,
+            -0.019709062313471467,
+            0.005385443078244229,
+            0.02086085539375208,
+            0.0037248737640193097,
+            0.012892299156840998,
+            -0.017546870090541213,
+            -0.0020341807891543716,
+            0.015594779287017958,
+            0.04816214672670215,
+            -0.0032851797178091685,
+            -0.010450846098750195,
+            0.029513742590677183,
+            0.023526004023637855,
+            0.005373455940309873,
+            0.0012120777808635214,
+            -0.02525630456877349,
+            -0.013075130701774785,
+            0.01962235086017781,
+            0.016944870494393874,
+            0.013484925533584295,
+            -0.0023255656480479004,
+            -0.019236081983457366,
+            0.021520156296467717,
+            -0.01701733319731161,
+            0.012422659010697477,
+            -0.023764941037007197,
+            -0.001490790833263327,
+            -0.0035419008954878724,
+            0.021998949056452454,
+            0.0021819635638621092,
+            0.008891505304514552,
+            0.024354552127458608,
+            -0.006978548952551799,
+            -0.019526615985982355,
+            -0.0051799020400150305,
+            -0.014578953088901798,
+            0.022506635912517146,
+            -0.024117708614271965,
+            -0.007008641019892289,
+            0.011379795968360019,
+            -0.012362093847309038,
+            -0.0102534051092702,
+            0.001164934138876502,
+            0.017686807394319725,
+            0.0009423993435129515,
+            -0.014784508280754078,
+            0.01944445593807992,
+            -0.0011359271135469456,
+            0.010410915597949927,
+            -0.01921419010977189,
+            0.009390638453648025,
+            -0.016682442516992777,
+            -0.007849184048209236,
+            0.010343008873053859,
+            0.0012604521199837046,
+            0.003948300703645506,
+            0.002592490121754231,
+            0.007773437307969315,
+            -0.03355658426684504,
+            0.022695505487314516,
+            -0.010751606023585792,
+            -0.003538523688786008,
+            -0.0014125330167969218,
+            -0.01678692701672328,
+            -0.0006918162525881059,
+            0.005816502852598842,
+            0.010878358093847386,
+            -0.016045147059003256,
+            -0.00873270513444376,
+            -0.04920244395446931,
+            -0.030314009649638693,
+            0.023156333024481057,
+            0.013733657599720879,
+            -0.010971506342695331,
+            0.018852159415605117,
+            -0.023760853159532587,
+            -0.010587249404421732,
+            0.0011379869499042905,
+            0.023319856329056813,
+            -0.03539290744576506,
+            0.01215369703816749,
+            -0.014493682360886549,
+            -0.02039288669534806,
+            0.009070984596830142,
+            0.011445354735283254,
+            0.0007259852045513916,
+            -0.0011514627555568814,
+            -0.029350945123925453,
+            -0.010246845821727087,
+            0.02391068177425878,
+            -0.00861718589055971,
+            -0.004723044986826333,
+            0.024716520302409263,
+            0.0033721801437248873,
+            -0.012524663490691517,
+            -0.017955758068233754,
+            0.021612671655463236,
+            -0.026077203994996058,
+            0.007645944309444457,
+            0.010032183011253845,
+            -0.0173991263874487,
+            0.021649599230588117,
+            -0.007529701797485131,
+            0.020327763440441377,
+            0.004227390938169872,
+            -0.011249233530251522,
+            0.020352283974471443,
+            0.013644443758970054,
+            0.004216958800094986,
+            -0.004081669404784771,
+            0.006869353098014179,
+            0.02212732880780841,
+            -0.011021739614266063,
+            0.02219580717345023,
+            0.028777213295471067,
+            0.00017328191188631918,
+            0.03260179554743322,
+            -0.021287655520443285,
+            -0.03159728983004563,
+            0.021740163175540124,
+            -0.01035146858119169,
+            -0.021564906188712388,
+            0.028914245422221707,
+            -0.013674731836508768,
+            -0.00786476052825481,
+            -0.0166718429993922,
+            0.00533027311342322,
+            0.016680336311088843,
+            -0.005246190451159961,
+            -0.014360934066628332,
+            0.0029628103834216345,
+            0.019809487418628386,
+            0.014587234544007958,
+            0.029780009648869975,
+            -0.010778199815920887,
+            0.035180242185168895,
+            -0.002320576872571577,
+            0.008990157317473909,
+            0.012415465232073404,
+            0.006489435152972723,
+            -0.011251531176923445,
+            0.013501894212117511,
+            -0.003446759011266526,
+            0.008230548711588447,
+            -0.001449023474280409,
+            -0.035111679577202855,
+            -0.013709382199848678,
+            -0.04709294504969336,
+            -0.0023752976043235697,
+            -0.0025098360125255215,
+            0.0030975417740913154,
+            0.09736815869389318,
+            -0.005723049647039619,
+            -0.011988817571623457,
+            0.001625059666070763,
+            0.009611697638805556,
+            -0.0009896494420056814,
+            -0.007606836952841873,
+            0.015973152482488318,
+            -0.01343292635775299,
+            -0.02140928016038583,
+            -0.010532701190806147,
+            -0.01288594454368977,
+            0.012079198190522715,
+            0.015929716343161652,
+            -0.0005947359502228821,
+            -0.015207331663761012,
+            0.006524010655124294,
+            -0.004566385899752684,
+            0.0064717704997400245,
+            0.008647099975176308,
+            -0.010414668624343924,
+            -0.0071698879102925555,
+            0.030899143952119196,
+            -0.025301269549419674,
+            -0.024361748354052457,
+            -0.0010579931222861095,
+            -0.009055796737486008,
+            -0.010335459737012047,
+            -0.002692653633719985,
+            -0.0017679501322462056,
+            -0.010519994985692081,
+            -0.010019865263564572,
+            0.006537076627706128,
+            -0.015472130358273033,
+            0.004812857776601667,
+            -0.0047457011269495634,
+            -0.013296069496526007,
+            0.017054968086538724,
+            0.022563785357424853,
+            -0.02841491200807702,
+            0.0006755428946168761,
+            -0.017422510783648978,
+            0.012941211381006863,
+            0.008866069010589792,
+            0.006049618782357251,
+            -0.022451528209097505,
+            -0.023622906582403243,
+            -0.015019272754405791,
+            -0.01927535750568903,
+            -0.024973490129133732,
+            -0.009073381957684092,
+            0.0067692803507600945,
+            -0.001267339788153158,
+            0.021918281111033645,
+            0.000816538714279555,
+            -0.01812545613159305,
+            0.0061084913570894905,
+            0.000568371220952269,
+            0.018292497018668842,
+            -0.027468677288882186,
+            0.0033806422195878524,
+            -0.02307582511848465,
+            0.033508234091571124,
+            0.0039195050746594975,
+            -0.026680111514794824,
+            -0.021558668725755275,
+            -0.003311830287529699,
+            0.018994577319066473,
+            -0.0025745912019675805,
+            -0.00724049447539589,
+            0.014775207617625293,
+            0.0155334760323838,
+            -0.004693118176196483,
+            0.002015625406296403,
+            0.027968079002437102,
+            0.01077998866180723,
+            0.0313892866933904,
+            -0.015236169297590675,
+            0.004945828495617267,
+            -0.027215016713379062,
+            0.010214419950212732,
+            -0.01503548137805401,
+            0.008859554812646242,
+            -0.01290890682557532,
+            0.03531384210182567,
+            -0.012005613861257432,
+            -0.004250698950192301,
+            0.03253288796441852,
+            -0.029150986393345252,
+            -0.0035020051536906334,
+            -0.009255437891304041,
+            -0.004142204892160768,
+            0.013240580426083011,
+            -0.001198093874819309,
+            0.008485141356876356,
+            -0.01854370462655459,
+            9.726096864073609e-05,
+            -0.003221852304410318,
+            0.005404595442127598,
+            -0.02005458688172967,
+            -0.003314118809778734,
+            0.0073427568419969814,
+            0.004259518237708634,
+            0.002408630356035373,
+            0.016420646142725346,
+            0.006662121128241274,
+            -0.0029357701493526907,
+            0.03424224243607692,
+            0.008171895850772245,
+            0.0152521424877615,
+            -0.0018638264193202728,
+            0.0030234211261443034,
+            0.019287963516760636,
+            0.019912956661192856,
+            -0.00048443399729935116,
+            -0.01058935153883094,
+            0.00195113175754497,
+            -0.001140732137433033,
+            0.011048741602327437,
+            0.024396046054912645,
+            -0.021158025221644738,
+            -0.02419562092713724,
+            0.016377661005279716,
+            -0.015076145951740961,
+            0.01903712233378994,
+            0.010541463112889985,
+            -0.010099351302095792,
+            -0.00031558616108876247,
+            -0.042316016201379245,
+            0.010964989031040692,
+            -0.02068939391415103,
+            -0.04004120028915868,
+            -0.00244383573783919,
+            0.0009542507389088937,
+            -0.029439456243505052,
+            0.02822814855547978,
+            0.0023755032182188156,
+            -0.017280221740813364,
+            0.018223709137876153,
+            -0.005548654680726505,
+            -0.0024040301239229064,
+            0.005397711787238679,
+            -0.01092026090471121,
+            -0.010648485086769927,
+            0.008235250252660972,
+            0.010957715075632535,
+            0.005425683656535705,
+            -0.005698919020608095,
+            -0.011284911899894062,
+            0.009975882651638084,
+            0.012318901214538077,
+            -0.0019922884373083155,
+            0.010050520101815593,
+            0.0031720752428247446,
+            0.007352859750323051,
+            -0.0073483049386141005,
+            0.0046483587496626965,
+            -0.005093220128411125,
+            -0.026080057085901446,
+            0.012863580653903103,
+            0.0052434400313520774,
+            -0.011863360599637747,
+            0.0103209502958491,
+            -0.01048162756158923,
+            0.005787157766911021,
+            0.001318945460232246,
+            -0.0059333230422061395,
+            0.004821320238893602,
+            -0.0038922239931512784,
+            0.011267634571983924,
+            0.013256611037442042,
+            0.004592529900805719,
+            -0.013076954459595086,
+            -0.02177318635989856,
+            0.009001291224623467,
+            0.01959767163356408,
+            0.024161754192119852,
+            -0.030416109293488064,
+            0.003784634588325551,
+            -0.0014915849060829743,
+            -0.0020550564975080885,
+            -0.015632785947151753,
+            0.01587790981601866,
+            0.01716742056401084,
+            0.012721563202528083,
+            0.01893393006967701,
+            -0.004009617703771589,
+            0.010539100665843704,
+            -0.02080068888791658,
+            0.013642451766699067,
+            0.0008609869904066794,
+            -0.013629067231663643,
+            0.0014620655123815298,
+            0.027046304278766087,
+            -0.011056783825898616,
+            0.006206969699250696,
+            0.008097527175511791,
+            0.022111870736907183,
+            -0.012509479100313492,
+            -0.003735868296238069,
+            0.01516254968167132,
+            0.0014878674796313539,
+            0.0035887079417890352,
+            0.016533149527222273,
+            0.04729052276110659,
+            0.00102751186013391,
+            0.0016713773449738153,
+            0.0192177920520243,
+            -0.005563519063161466,
+            -0.01114821419109643,
+            -0.003549999863325982,
+            -0.021807893906469106,
+            0.010998558936481785,
+            0.00431279785115251,
+            -0.02186324212032519,
+            0.01024093366826206,
+            0.0125495473550584,
+            0.005268320135120931,
+            -0.0031679212410026603,
+            0.02574575247827441,
+            -0.019807879421996258,
+            0.014935295894965128,
+            0.03329450628384787,
+            0.00493356778223407,
+            -0.011423949333937285,
+            0.007263264249871384,
+            0.0027354459678742013,
+            -0.004494406041906684,
+            -0.00606745413660583,
+            0.010497391290734088,
+            -0.008191702034391382,
+            0.009870458191510176,
+            -0.009580130479693,
+            0.039202091799313755,
+            -0.009619018331262844,
+            -0.010782138248435458,
+            -0.004558645539551881,
+            -0.006241773108460662,
+            0.003478414182938609,
+            -0.003395165694642073,
+            0.005812225135633631,
+            -0.0069045407353789945,
+            -0.014607146220271209,
+            0.0008522982757210454,
+            0.00020175973815948303,
+            0.014001922919116725,
+            0.0011804904415274088,
+            -0.004871874532755353,
+            -0.007877217154706205,
+            0.015243277056007988,
+            -0.006047906208304912,
+            0.0006531937033317192,
+            -0.0011653523616313422,
+            -0.0013662556399359335,
+            -0.007401050012179761,
+            -0.015166105795085528,
+            0.00974609244984173,
+            0.016253733908242134,
+            -0.013857834499211056,
+            -0.005142936208363415,
+            0.012274716212832219,
+            0.008540229125193836,
+            0.011742606168187876,
+            0.0007451338650642946,
+            -0.009255813682572937,
+            0.008292531177374192,
+            -0.015020523035432158,
+            -0.008781317965985481,
+            0.0026250469418136603,
+            0.002381147382977664,
+            0.005839723981046825,
+            0.0006240641828668715,
+            0.004502463508605272,
+            0.005798912705218846,
+            -0.00497936644919367,
+            0.01112666823857836,
+            0.008198424171006548,
+            -0.002452143482105151,
+            -0.01017569410837814,
+            -0.001127589984272638,
+            -0.015733298167661346,
+            -0.016758315411028353,
+            -0.0009286266498936771,
+            0.015175988113750882,
+            -0.02352862898718829,
+            0.018996352975391644,
+            -0.0014524363611708201,
+            -0.0033802546171857704,
+            -0.021392428456030312,
+            -0.0025923983911570183,
+            0.0034214188668518796,
+            0.01255768230394476,
+            0.01858853630745928,
+            -0.02729543519465384,
+            0.010575894388563116,
+            -0.007204725245442141,
+            0.030394929424014717,
+            -0.018328782140005044,
+            -0.004022941273610227,
+            0.026915618079621992,
+            -0.02200018306828758,
+            0.0008791185377513978,
+            -0.011454606955158956,
+            -0.008103032693853504,
+            0.019287674837688177,
+            0.023747720480736777,
+            0.02038287242167783,
+            0.01364489294869211,
+            0.0007758044919789085,
+            0.004768030222445397,
+            0.03572705648994403,
+            0.010568241270862248,
+            0.014828087092859878,
+            -0.007542489523110974,
+            9.457808640264811e-05,
+            -0.02554033312739386,
+            0.01635899434369978,
+            2.587383534566272e-05,
+            -0.012370727506722627,
+            0.007162422186621379,
+            -0.013370882403931942,
+            -0.011066483600541058,
+            -0.007151077450730025,
+            -0.01721910018182534,
+            0.02152239849400241,
+            -0.012322440280680164,
+            0.004587729883596212,
+            -0.009079747696669277,
+            -0.013272707286450417,
+            -0.012953682633400515,
+            -0.014202214807187545,
+            -0.01313141988034687,
+            -0.026041403779538307,
+            -0.027109731260915763,
+            -0.006573285295895077,
+            -0.023780366346334524,
+            -0.013799072648505526,
+            0.005588622594136253,
+            -0.022617902666316757,
+            0.012262325666428604,
+            -0.012689827812482627,
+            -0.01226264872059864,
+            0.012805734807835008,
+            0.020274844315616616,
+            -0.011765231803328348,
+            -0.028494489703400308,
+            -0.018583843821043854,
+            -0.019970998470704358,
+            0.012576201137505969,
+            0.002266288538944273,
+            0.013165889075444961,
+            -0.003018437854660793,
+            0.003227412667880239,
+            -0.014895831590298957,
+            0.018570624675927183,
+            -0.03442390703780963,
+            -0.03108003806963536,
+            -0.013611306765348361,
+            0.005182777066476028,
+            -0.0015782033203578553,
+            -0.011429046504256945,
+            -0.0018606648572920801,
+            -0.00433805271182878,
+            -0.025109572024446088,
+            -0.010134856632190067,
+            0.011010662054484161,
+            -0.015205629927469414,
+            0.0072933565255675036,
+            0.0009018832564038863,
+            0.013870902834260887,
+            0.009587432944616442,
+            0.013393301893492882,
+            -0.006095063698384775,
+            0.004356485794113208,
+            -0.0026476612747324386,
+            0.0004894024149759108,
+            0.009336366919474478,
+            0.00839591561027103,
+            0.010551161369079399,
+            0.002260707777260338,
+            0.007830059223001696,
+            -0.0060927446216633265,
+            0.018065002687169148,
+            0.015436944875538481,
+            -0.0002520689442496089,
+            0.013323098485251487,
+            0.0003769862591044255,
+            0.00022293304921127018,
+            -0.0029970303106340466,
+            0.007888437018607987,
+            -0.011785957933642163,
+            -0.001311704406355858,
+            0.015138190893483804,
+            -0.01707247730569978,
+            -0.013045187198628258,
+            0.03107034781283638,
+            0.0018583638488268697,
+            -0.02495564203304796,
+            0.031146554085787762,
+            0.004220047610942943,
+            -0.005835199594223787,
+            -0.013205086624848782,
+            0.0014059047822222418,
+            -0.011570624844100298,
+            0.003584232809362274,
+            0.014295032061446826,
+            -0.02231984223610487,
+            0.010521094801839404,
+            0.008446892441796151,
+            -0.009681535919980559,
+            -0.012337659795586223,
+            0.009847073709209602,
+            0.016809328445200358,
+            0.004472485504034416,
+            -0.03539093810210523,
+            -0.011957694102753411,
+            0.02145470732008716,
+            -0.029237428225009913,
+            -0.00810041956259673,
+            0.007452962400105787,
+            0.0065154832040426065,
+            -0.019353109017609053,
+            -0.008580955020412362,
+            0.005725794605897509,
+            -0.004940390374637084,
+            -0.003892210172017758,
+            0.017541623128534682,
+            0.00485047043626186,
+            0.013698341792101025,
+            -0.01689738578307474,
+            0.009771458057400557,
+            -0.022222817828602557,
+            0.004021882074676753,
+            -0.023373700500461992,
+            -0.009873403167702603,
+            0.002364143609079849,
+            0.012454271200139762,
+            0.014328556467632212,
+            0.027839363143659326,
+            0.030056404238348527,
+            0.013995696971422429,
+            0.007833677585250953,
+            -0.016265199386953717,
+            -0.018134327237616175,
+            -0.006409513840075933,
+            -0.004033173350204784,
+            -0.008714798059218596,
+            0.0066373547199279235,
+            0.01858354212009877,
+            0.004499712802472377,
+            0.02609078206670711,
+            -0.006227189596272876,
+            0.009039744622014585,
+            -3.127526911951914e-05,
+            0.0008833930131827993,
+            -0.004226083827636182,
+            -0.01971685546584527,
+            0.00481003211084742,
+            -0.0013721482162825508,
+            0.025874758880591167,
+            0.00940496860916206,
+            0.0024915013399541847,
+            0.01615978314811047,
+            -0.006378952757397484,
+            -0.01174653607178726,
+            -0.010540107733238588,
+            -0.0061926268398085176,
+            -0.021752582127981104,
+            0.029059100503124387,
+            0.021195595568253612,
+            -0.010800765264123782,
+            -0.013693582166598082,
+            -0.00903369526059487,
+            0.02795767990556847,
+            0.004080965688836953,
+            -0.0013050212744006933,
+            0.016313311850308505,
+            0.012084555958999215,
+            0.012329325712491252,
+            0.009794698207689061,
+            -0.00036157148861933123,
+            0.0015364709122308352,
+            0.03722522141440073,
+            -0.007427028510548186,
+            0.0026463292682538495,
+            -0.005239324411651515,
+            0.0025987932868461313,
+            -0.009705817520422487,
+            0.007242122655628283,
+            -0.004012676385296197,
+            -0.0158760051470016,
+            0.015003444174308607,
+            -0.022443331305169994,
+            -0.0016749787204471238,
+            -0.008150635190952799,
+            0.0016696640640947303,
+            -0.02781023337840043,
+            -0.0025731206607371845,
+            0.006695065701630532,
+            -0.015547340828489069,
+            0.011816010501093885,
+            -0.01740673229051557,
+            -0.011553604323149105,
+            0.007914961618894248,
+            -0.012034157288065447,
+            0.0016684130271324446,
+            -0.0008268957621923278,
+            0.004460864332388434,
+            0.013426055079012884,
+            -0.011008808570221245,
+            0.012166046631826273,
+            -0.005816284310138776,
+            -0.007872091214834688,
+            -0.0016055129487028885,
+            -0.0030049472973942043,
+            -0.0025600746761171543,
+            -0.0009966231170649917,
+            0.034355910234323155,
+            -0.004446570896048543,
+            0.003761242147499512,
+            -0.01362639743248326,
+            0.0019833052855181806,
+            -0.036176376394999044,
+            0.028344464817309477,
+            -0.008642835889481802,
+            0.011013502221008498,
+            -0.0017508365480865596,
+            -0.013243706268954915,
+            -0.01456517309747086,
+            0.015247299501245178,
+            0.007602596572394442,
+            -0.005854531057349665,
+            0.013558248087347848,
+            -0.019724373614448298,
+            0.008530500822910953,
+            0.004463694543554244,
+            -0.015561397976987216,
+            0.004668780190328835,
+            -0.007479912658259757,
+            0.012657201279639194,
+            -0.01774730033362932,
+            0.004314594593150765,
+            0.00958571617187387,
+            -0.01892876084898881,
+            0.009901041800918215,
+            -0.028675879564512306,
+            -0.004631920242208613,
+            -0.012323455524366223,
+            0.0053804608827285865,
+            -0.000660864620708182,
+            -0.02389088252276507,
+            -0.028944280660230762,
+            -0.0002099667627277945,
+            0.00458533068247518,
+            0.0018878351997997833,
+            0.011621407953503755,
+            0.005970807015151215,
+            0.008542958629199003,
+            0.01023896054756221,
+            -0.00205207247588841,
+            0.009402752124876478,
+            0.04169706217414899,
+            -0.022899912321317197,
+            -0.011574970753549647,
+            0.005321872977369877,
+            0.011499271505460519,
+            -0.02647653483798284,
+            -0.011039791758347888,
+            0.0033424345263339144,
+            -0.005219877979550397,
+            0.03609469691834737,
+            -0.02448746405881972,
+            0.019020618652020673,
+            -0.011392297116547224,
+            0.007625475937968714,
+            -0.021465240076078376,
+            -0.003921088264697311,
+            -0.014265745991522477,
+            -0.00011221615798127438,
+            -0.0031478236827989703,
+            -0.0076053733863387755,
+            0.007485039021296698,
+            -0.001907678661303243,
+            0.0020348246022161925,
+            0.006291436821831794,
+            -0.02173305455485444,
+            -0.0019125499969493854,
+            -0.016432460924913166,
+            0.030367263349759286,
+            -0.016242095671439496,
+            0.005072805486919414,
+            0.0053258139728101506,
+            -0.002810639419737258,
+            0.012368771814660942,
+            0.004165967191536186,
+            0.00513754440091394,
+            -0.004199767241827414,
+            -0.005142642360546972,
+            -0.0053886448467479215,
+            0.017868874549760376,
+            -0.013280048598849825,
+            0.035024701466710065,
+            -0.0100959186045929,
+            -0.007094001877261568,
+            0.0019733042325232155,
+            0.010380430710282744,
+            4.2084202709296443e-05,
+            0.01053565670076933,
+            0.0034270489770133685,
+            -0.0006311078385359693,
+            0.0019774779328212756,
+            -0.017162448473652843,
+            0.003089758682621882,
+            0.01682656561693936,
+            0.005463677441664502,
+            0.004469261499451007,
+            -0.013332130891003062,
+            -0.012050233723282738,
+            0.004538268704442613,
+            -0.007533940408255447,
+            0.0003675737140813486,
+            -0.018816953644773076,
+            0.013972696748575243,
+            -0.0011096683686631858,
+            0.0002715683052236706,
+            0.00524678369365015,
+            0.023056514997908218,
+            0.01813167860994747,
+            -0.002553352602551003,
+            0.03155164978556312,
+            0.0174738879054593,
+            0.0029268039949260673,
+            0.010637295711777793,
+            -0.033222781195984666,
+            -0.013301873512972214,
+            -0.0013721703891642354,
+            0.02238421933149517,
+            -0.01530189703335162,
+            -0.014059029663525449,
+            -0.0037393534403749947,
+            0.0010394601933309767,
+            -0.005177359933828505,
+            0.0042489306011644495,
+            0.011039889511148341,
+            -0.014697595240068863,
+            0.007597149702441915,
+            0.0026780496822907634,
+            0.02092734730183164,
+            0.0004688144432243786,
+            0.017181410417231698,
+            -0.0005083970646221274,
+            0.00749639055342427,
+            0.007319420826337891,
+            0.017541463125355265,
+            -0.01002811389296805,
+            0.023582113845081534,
+            -0.008823933397729605,
+            0.0019889258902730535,
+            -0.01257106158422629,
+            -0.028159594067544737,
+            -0.012624750220149823,
+            0.015247564545279736,
+            0.005765911899842925,
+            0.006478299226263732,
+            -0.016537090297144265,
+            0.006276997440048981,
+            0.015647122171767974,
+            -0.002251683151059454,
+            0.010295358869874306,
+            0.009730378299068289,
+            0.00012816405890877098,
+            0.010181120688239766,
+            -0.00801256937783534,
+            -0.00734257767723293,
+            -0.004497520124843868,
+            0.023982754195729983,
+            -0.005443550092842393,
+            0.022434177122972028,
+            0.012647958478747348,
+            0.00616791857573897,
+            -0.010853355357954727,
+            -0.003340111254357573,
+            0.023389634440057473,
+            0.011863246790550376,
+            0.00029642699874398923,
+            -0.009846034060495678,
+            0.0019458876579524617,
+            0.0054010373765486005,
+            0.012983913038716568,
+            -0.02049427270240938,
+            -0.004344571352918878,
+            0.008461947538684664,
+            -0.008472919299762712,
+            0.004318309005722874,
+            -0.030156444522730416,
+            -0.005812303237222271,
+            0.021354352236942434,
+            0.008392281015122372,
+            -0.010232039722494646,
+            -0.004395682831498155,
+            -0.0006527587802367882,
+            0.0040601850269714945,
+            -0.0024323167053212176,
+            -0.0023729138545912176,
+            -0.008541233856441836,
+            -0.006470933419694742,
+            0.014035696617797647,
+            -0.0052691760528871924,
+            0.008597459268477037,
+            0.00012084859681905111,
+            -0.0017255958688825231,
+            -0.007323529232751411,
+            0.03382792341957058,
+            0.008087732655845718,
+            0.005980666974986827,
+            0.0013613625176327891,
+            0.0042244330945601215,
+            0.000616791811672998,
+            -0.0028374856646453707,
+            0.018075360591135542,
+            0.0156738939725117,
+            0.005054696911814988,
+            0.009038525980341447,
+            0.014672270369917403,
+            -0.019700405219230957,
+            0.012625337070556061,
+            0.012184185186281482,
+            0.019382321278525114,
+            0.0007219498737851579,
+            0.002564970959687783,
+            -0.010601873823737926,
+            -0.0029963537053764736,
+            -0.002070567089566093,
+            0.029563028293936092,
+            0.0005903976560069006,
+            -0.010509758568838935,
+            0.010992421945232951,
+            0.020603868018231348,
+            -0.02101157589451638,
+            -0.001971974816402967,
+            0.005963911670920596,
+            -0.010704207628592966,
+            -0.014466113540150018,
+            -0.021269176952661614,
+            -0.013331247081394942,
+            0.0071597345125287174,
+            -0.008162102923896894,
+            0.02367549407145414,
+            -0.014256771777502395,
+            -0.03960796112042958,
+            -0.014803626428839152,
+            -0.00864320347698291,
+            -0.027531337207277483,
+            -0.007305441146142331,
+            0.00965056093889974,
+            0.016698589600806273,
+            -0.003330867073501157,
+            -0.025198703196464594,
+            0.030483790406960656,
+            0.008872821868559751,
+            0.023708422767352917,
+            -0.021464374508407488,
+            -0.0018617153194555344,
+            -0.00886376784514331,
+            0.014505918224188001,
+            -0.017588761818663975,
+            -0.011756112651591795,
+            0.020014632992762026,
+            0.014356419813769714,
+            0.007246557500044434,
+            -0.004355132555323843,
+            0.0067299398462013936,
+            0.007649499235136303,
+            -0.012515181210363267,
+            0.012239792251154642,
+            -0.013499277313916029,
+            -0.024858295126228433,
+            0.014430561986516884,
+            0.002229494935179828,
+            0.009563004428064055,
+            -0.0015984171085659323,
+            -0.016197659608768088,
+            0.010078159490026602,
+            0.00808729175515505,
+            -0.017508840440512225,
+            -0.004793934415922065,
+            0.007133527615305415,
+            -0.020818869260580315,
+            0.031234686259160864,
+            0.011872765152279094,
+            -0.010692949770503467,
+            -0.001989039629661815,
+            0.004431079028880923,
+            0.016511544608975148,
+            0.007893700385963683,
+            -0.01997314388229276,
+            -0.003708006454566117,
+            0.01530128306203401,
+            0.00822588305477249,
+            0.004861703074728471,
+            -0.010836328783594584,
+            0.03434874763691911,
+            0.006082873924472328,
+            4.192481378506604e-05,
+            -0.0052997095110967115,
+            -0.0076108924981838355,
+            0.004196112457289409,
+            0.018905391806961686,
+            0.013330078288599763,
+            -0.004200590783114919,
+            -0.006630107527854746,
+            0.007778448128358027,
+            -0.01706991160178345,
+            0.007329056462125937,
+            0.009390327499494826,
+            -0.012870674959022891,
+            -0.0017571189976160295,
+            0.016462608870101713,
+            0.00209846661139665,
+            0.004942714850185376,
+            -0.011305050739130339,
+            0.004252776146827917,
+            0.015841821944109317,
+            0.018727567181082646,
+            -0.008399642221255733,
+            0.021448055799259867,
+            -0.020972459569228434,
+            0.027243639465535856,
+            0.028499400850980086,
+            0.016573144403275928,
+            -0.01847096443459798,
+            -0.0022409373140116146,
+            -0.0041737366139895414,
+            -0.014761716560267328,
+            0.0034546020161337894,
+            0.005125870074458691,
+            -0.01174186608185822,
+            -0.009577921232380895,
+            -0.024331993794416026,
+            0.011075959943215117,
+            0.007918147902741738,
+            -0.0015045185909419564,
+            -0.011791155919533844,
+            0.016019077577552835,
+            0.010141868950981925,
+            -0.010938402325619172,
+            -0.005635351352447256,
+            -0.019967361901734836,
+            0.009750427567651833,
+            0.009128294017175602,
+            -0.00300556760891084,
+            -0.011496103417643132,
+            0.012977890524465716,
+            0.0014080059843484115,
+            -0.01292380658817521,
+            0.010841954636959441,
+            -0.03142011651894961,
+            -0.005510537354374977,
+            -0.011027026734432255,
+            -0.003620482919145167,
+            -0.014415606143566519,
+            0.003608572766363589,
+            0.0005034611321129689,
+            -0.011470488351158329,
+            -0.0035005774696278,
+            -0.016207453351921627,
+            0.004676327282262429,
+            0.015021513761697039,
+            0.001679614645967629,
+            -0.006917375094634678,
+            0.021994274901326064,
+            0.018518524483306403,
+            -0.021903538417919984,
+            -0.020116013379669034,
+            0.004708084983787958,
+            -0.03261690161800251,
+            0.015019348187107588,
+            0.00966124471577622,
+            0.011676937880751252,
+            -0.028768229226408037,
+            0.007114534979415504,
+            0.00900420165933673,
+            0.016398927481511824,
+            0.0050879301160439245,
+            0.009453255486450082,
+            0.01222758288108498,
+            -0.0104657614414275,
+            -0.003164502778525433,
+            -5.410382753651127e-06,
+            -0.005618225986727351,
+            0.015607620507415484,
+            0.0008527697935056332,
+            0.03603666488542239,
+            -0.005277207843139309,
+            -0.0009799879252482288,
+            -0.011735731235560851,
+            0.00812046801749312,
+            -0.011713349784345638,
+            0.006334172347602515,
+            -0.0024643499727789167,
+            0.0001240009282616789,
+            0.002965067818077541,
+            0.009442572075213767,
+            -0.01086095783353607,
+            -0.007595379345224076,
+            0.007444897256325813,
+            -0.01482814340478338,
+            -0.00019984717819268013,
+            -0.009339343730980451,
+            0.0019680804106275156,
+            -0.02241031895876346,
+            0.006629838013824249,
+            0.040935644507527846,
+            0.02140216465404127,
+            -0.005481040712269146,
+            -0.012431659209559418,
+            -0.004680230194268222,
+            -0.007229300034633229,
+            -0.006482401275991039,
+            -0.004794698160419071,
+            -0.022762405726589375,
+            -0.006928553505112732,
+            0.004320321244630314,
+            0.013300138724212552,
+            0.009741018117644266,
+            -0.008842135469729511,
+            0.004531760574809748,
+            -0.02662650543210291,
+            -0.016578081411548973,
+            -0.0060021425889846495,
+            -0.015840130031005147,
+            0.006125301421471048,
+            0.027521841635711787,
+            -0.0063432924325983525,
+            0.01789356321527744,
+            0.005142029893259461,
+            0.013034833065163448,
+            -0.005338869351330258,
+            0.00035250243782976347,
+            -0.006010938630154189,
+            0.0032958656042048292,
+            0.003074861351445954,
+            -0.0015947217852680714,
+            0.004151590621941346,
+            -0.007139285822575166,
+            -0.0026091937868967284,
+            0.0011629380311555433,
+            0.0117841884590861,
+            0.006877419488576261,
+            -0.00016252653621991807,
+            -0.01761285866765538,
+            0.022221104524808997,
+            -0.018933508559881868,
+            -0.008840161102111973,
+            0.011092684029552064,
+            -0.009660487856185668,
+            0.00702597023411946,
+            0.020387386977378873,
+            -0.006854213562767843,
+            -0.009583981919303215,
+            0.013028181087649146,
+            -0.0036507672754997286,
+            0.004464368543493108,
+            0.009867428978560914,
+            0.006036303973414791,
+            0.002482944923100156,
+            0.013244650094892797,
+            0.004530294330688855,
+            -0.0032625482538717137,
+            -0.020238533752799943,
+            -0.005186498720507101,
+            0.019015752262933585,
+            -0.02252458872825473,
+            -0.0027641215988942563,
+            0.0044506144202026456,
+            -0.0030317840657635457,
+            -0.01733962766050954,
+            0.004205861489436576,
+            -0.018770275720867473,
+            -0.0033883519277721433,
+            -0.008244668812709337,
+            0.009948090119054079,
+            -0.023700732627507225,
+            0.009109315916173436,
+            -0.011739192106106307,
+            0.02356509175177699,
+            -0.003925851077921686,
+            0.024734433836185024,
+            0.0015142645677087268,
+            -0.013482933030628261,
+            0.0055323884713736554,
+            -0.0008930661843169134,
+            -0.004477731568313757,
+            0.029338490404300612,
+            -0.005418258701133154,
+            0.002249996261871466,
+            0.013491124734151694,
+            -0.021332541202417246,
+            0.021773687217263173,
+            0.0046662696076723836,
+            0.01628697326539432,
+            -0.00986238951830334,
+            -0.013799209313714441,
+            0.025178874553620743,
+            0.03324555035648896,
+            0.008398796308070151,
+            0.011261509865513953,
+            -0.01226890254177634,
+            0.018966175575083295,
+            -0.0038328997984309597,
+            0.004344837325292706,
+            0.00878668880458693,
+            -0.007200842513742962,
+            -0.011500562922303248,
+            -0.004533014670236755,
+            0.027639894892383943,
+            -0.021838848944317424,
+            -0.008920629621611768,
+            -0.005739181836252042,
+            -0.01593272996094732,
+            -0.038167534267554606,
+            -0.004371701247332816,
+            0.019604405563120094,
+            -0.005974358141737469,
+            -0.0018341151594086288,
+            0.014334266690716505,
+            0.00402620072221496,
+            -0.002852569009185304,
+            -0.008195218707838913,
+            -0.0027944153624309856,
+            -0.0059390492900516225,
+            0.016651206007022145,
+            -0.004961864528562493,
+            -0.007929830585906699,
+            0.01145431100211762,
+            -0.005731957927601062,
+            0.012130694936714224,
+            0.0004848418056781251,
+            8.392048424584131e-05,
+            -0.0015174120840537315,
+            0.00822932723013239,
+            0.018620869354789082,
+            -0.0055803454666417465,
+            0.025682778991091174,
+            0.006956936033581638,
+            -0.003495709501259274,
+            -0.005489431577052735,
+            -0.0044154293119170445,
+            -0.024561016942073285,
+            -0.0020864636079347357,
+            0.006593060072668424,
+            0.024333813203596122,
+            -0.02515989187381735,
+            -0.004953266000463339,
+            0.020957806164031402,
+            -0.0012192724005613,
+            -0.011978722479596745,
+            0.014690559986694138,
+            0.0041346025614047875,
+            0.009881584250499358,
+            0.012176393203318002,
+            0.015764530703333238,
+            -0.004923548157246672,
+            -0.0015192775000207833,
+            0.012766482928380375,
+            0.0005745973040874939,
+            0.00997809510696241,
+            0.016495444272181515,
+            0.010467514633877887,
+            -0.021108281220544853,
+            0.0015078843789125156,
+            0.00522593579521476,
+            -0.017481430635971396,
+            0.0034362821987740755,
+            0.005397819166037461,
+            0.011626527721895694,
+            0.004048632382026344,
+            0.013207335578216437,
+            -0.015768003434275563,
+            -0.005916168437120099,
+            0.008153798523776654,
+            0.004319451296835019,
+            -0.02599474886669325,
+            -0.01485009085106089,
+            -0.0006781964116146299,
+            -0.004444144921332929,
+            0.013556265879295017,
+            0.01285768162554067,
+            0.02033626001285354,
+            0.024637349990579514,
+            -0.013833060730799217,
+            -0.00842819657792762,
+            0.009436072455719896,
+            0.015560357465711765,
+            0.005798181411384387,
+            0.007874051721372273,
+            0.009297440071118126,
+            0.0077320823375042,
+            -0.004073209576661068,
+            -0.002973928843567532,
+            0.020205003097225747,
+            -0.019725382393637726,
+            0.0006746659838642147,
+            -0.018417443693472958,
+            0.007530796896254817,
+            0.01140696723506382,
+            0.01129515042745398,
+            0.002816883643872417,
+            -0.005274757308627299,
+            0.005716917047812153,
+            -0.06254973656832136,
+            -0.009191453047502128,
+            0.003384751551032943,
+            0.027256466708746898,
+            -0.010885336140129142,
+            -0.003415707910562931,
+            0.007309577090125959,
+            0.005511970708769456,
+            0.0005572239548625122,
+            0.003186427278034852,
+            0.019698716902282903,
+            -0.02052581021967601,
+            -0.010793968505805883,
+            0.004219269922138548,
+            0.0033111837923746746,
+            -0.007269521534951044,
+            0.004805737092302134,
+            0.014143978208501908,
+            0.01986656063456352,
+            -0.0024048854610842748,
+            0.01629335842915328,
+            -0.0015498603704589015,
+            -0.014496994444493684,
+            -0.002671064325909516,
+            0.0037875833047555925,
+            -0.022466433279066646,
+            -0.005980897597195858,
+            0.0043262697442044435,
+            -0.011226121648007667,
+            0.006978495397818936,
+            0.008452177557316863,
+            -0.003018545465557218,
+            -0.0008303307644324117,
+            0.017796789945838845,
+            0.006591364879905198,
+            -0.013464486184025026,
+            0.0028668426952326957,
+            -0.023377605222213174,
+            0.004325185794796361,
+            0.001416245546006914,
+            0.007809449561523174,
+            -0.008431335139468173,
+            -0.014774499666285603,
+            -0.005933810712528226,
+            0.017746847020610673,
+            -0.01965188730040385,
+            -0.002406875192894708,
+            0.01683009662984443,
+            -0.0002518220110313279,
+            0.005456812494894182,
+            -0.02011241622618092,
+            -0.0009782974741561381,
+            -0.02933296756078644,
+            -0.01171823509606186,
+            0.001162345317872531,
+            0.00892221099991475,
+            -0.006551308022163148,
+            -0.03934356275100932,
+            0.00427623835776202,
+            -0.013111664387776261,
+            -0.008752857644809231,
+            -0.010214546496134218,
+            -0.00518087722852466,
+            0.015240297918667867,
+            -0.012498100033235827,
+            0.01086490299272505,
+            -0.00917777212602538,
+            0.005895055740265625,
+            0.0090554651297836,
+            0.014212841798768113,
+            -0.005544047915369086,
+            -0.0033589999440643385,
+            -0.007924830667307956,
+            0.0022291797354234005,
+            -0.0032976226056924502,
+            0.009615674261248298,
+            -0.0051932155872883,
+            -0.0065636389655770455,
+            0.02426106498596339,
+            0.011445591067574932,
+            0.013312596099532533,
+            0.0055118624272110965,
+            -0.007220620072349783,
+            -0.01916551187613823,
+            -0.003977616240384473,
+            -0.013204506708246246,
+            -0.006315966975649096,
+            0.01434099854765068,
+            0.012588249778961223,
+            0.006357972880218866,
+            0.0007747002406996095,
+            -0.004794439624135275,
+            0.006975593520375694,
+            0.0262335496080837,
+            -0.016481909971520166,
+            -0.0013084662210086592,
+            -0.01369313900978409,
+            0.01921748474212319,
+            0.03333761672336437,
+            -0.0008918419419301244,
+            -0.007699525186523558,
+            0.013757216065348164,
+            0.00819045303047094,
+            -0.007411380297164911,
+            0.02453611080891859,
+            -0.025045462880463647,
+            -0.015989319900023054,
+            -0.011650886878708265,
+            -0.011407253985176972,
+            -0.008386367959227559,
+            0.016331961716602287,
+            -0.00141225438918211,
+            -0.0018505754298268654,
+            0.009557291617446645,
+            0.03357924917013024,
+            0.006916860901290756,
+            0.007030308953738567,
+            -0.002739345982346728,
+            -0.009376522438071562,
+            0.011482637968411055,
+            -0.007259054391879614,
+            -0.04075661435928125,
+            0.004250195164583401,
+            -0.005160505473755424,
+            0.02405623169060588,
+            -0.02197586561375289,
+            -0.00619175761282745,
+            -0.006736129049831973,
+            -0.028200015908139763,
+            -0.005867722104196303,
+            0.02805071895018145,
+            -0.015402198567775478,
+            0.014773294766214671,
+            -0.009502185346558072,
+            -0.0013419858580438616,
+            0.004443749010136947,
+            0.013008783570355656,
+            -0.02289662952366337,
+            0.0013834888740540108,
+            -0.00048549197205067054,
+            0.027969842651517314,
+            0.010437729090975158,
+            -0.0047705656249623744,
+            0.014202218020412936,
+            0.008240861154198411,
+            -0.018637315941614406,
+            0.007314019317343154,
+            -0.005747966765101792,
+            0.014833896739762147,
+            -0.01314847940891154,
+            0.007762100494499689,
+            0.00705026393499352,
+            -0.01278023669088385,
+            0.03016342404000438,
+            -0.015204129985039077,
+            0.007776946006562982,
+            0.009278778571285689,
+            0.009656723862179986,
+            0.01257686993982689,
+            0.004362439504889069,
+            -0.02898418193036562,
+            0.0002144431781454467,
+            -0.03828100004144633,
+            -0.00833748467467171,
+            -0.002611937107022567,
+            -0.002145329666219461,
+            0.007050630951707937,
+            -0.009383645162261629,
+            -0.033976442057557676,
+            0.0006429978367640709,
+            0.01603027598131823,
+            0.017501067413108286,
+            0.0002196997337451293,
+            0.013075379412332973,
+            -0.006197537539554383,
+            -0.021148120787551232,
+            -0.016986448660412497,
+            -0.010207705210051413,
+            -0.02122194744719993,
+            -0.020550470201191853,
+            -0.00908564461857315,
+            0.012090079127547193,
+            -0.009927390005879807,
+            0.013888458967497627,
+            0.029513250508042935,
+            -0.024529175134769883,
+            0.0011144447737101862,
+            0.021977677894839922,
+            -0.00198308986575084,
+            0.014108189311291326,
+            0.001696775006906642,
+            0.014122064251288995,
+            -0.0004166140484018266,
+            -0.010556016685519523,
+            -0.009729362492661684,
+            0.017487929296614076,
+            -0.0026043257634797124,
+            0.016715798948897194,
+            0.012183487338360088,
+            0.003771721868999568,
+            -0.0030887189945795796,
+            0.010667776697461199,
+            -0.026471157050402344,
+            0.014249996582064706,
+            -0.006690961709923224,
+            -0.0181382691440242,
+            -0.008729104191096461,
+            0.003882731567105777,
+            -0.005609384377642956,
+            -0.007794812949125115,
+            0.012967089730551055,
+            -0.00944374111323783,
+            0.02088525981049842,
+            0.0023400121055514958,
+            -0.012196925240774426,
+            0.003648688547842322,
+            -0.03322041209304968,
+            0.0015191765655963997,
+            -0.018402382776013956,
+            -0.024659631472836927,
+            0.03840173429666435,
+            0.011991452465253198,
+            0.006639702644606365,
+            -0.01675838115087676,
+            0.007953102302963526,
+            0.0004257479565099406,
+            0.017484338819046517,
+            0.011068864013925452,
+            0.006466915839538296,
+            0.008364454017976605,
+            -0.023316137291380285,
+            -0.010898421318668592,
+            0.016204569093730564,
+            0.012099473359791148,
+            -0.01017229369980751,
+            0.019431669546223253,
+            0.017955119269367787,
+            0.007300372071724306,
+            -0.006898619924677706,
+            0.01647927363640854,
+            0.021012103522831126,
+            0.010291016618557653,
+            -0.00047955552103474145,
+            0.0008684297441778696,
+            -0.010534247928997246,
+            0.005037930235889528,
+            0.035417626740307205,
+            0.014712595692093886,
+            0.010650306363963653,
+            -0.03626584475133921,
+            0.021764456453493054,
+            -0.011961521673661574,
+            0.009092981192874177,
+            0.006089104887183293,
+            0.017982801101846038,
+            0.006916969397519662,
+            0.014196254958707752,
+            0.011745069030657836,
+            -0.006689305450954901,
+            0.025205280281804708,
+            -0.001601414806100461,
+            -0.011992525142592405,
+            0.003386930324498371,
+            -0.01859193251438437,
+            0.005303261319773131,
+            -0.0041710536965100615,
+            -0.004459393120257307,
+            0.0032461828144552565,
+            -0.0018650956576455287,
+            0.015678225251799887,
+            0.003894142628084471,
+            -0.02355212701026675,
+            -0.0014873755115206368,
+            -0.024284295126348117,
+            0.0027260592623277375,
+            -0.022185571998195344,
+            0.00022883605560765203,
+            -0.005557568353382136,
+            -0.009749382813201403,
+            -0.025385857431333838,
+            -0.009853073898348951,
+            -0.0035843255959309307,
+            0.007661137888845632,
+            -0.010521037433807653,
+            -0.004246698084422667,
+            -0.0019920499321964815,
+            -0.019465113363548715,
+            0.009014255846983305,
+            0.027122533756959888,
+            0.016848221859975445,
+            0.027170883580103264,
+            -0.00019554112591107156,
+            0.013614675849485926,
+            -0.002066822813103284,
+            -0.01603063005944364,
+            0.008824663429777616,
+            0.0075653522381218495,
+            -0.00531780740880296,
+            0.014643348459785112,
+            0.0072634484937344704,
+            -0.012800149015319017,
+            0.009859503660384916,
+            -0.007161712176469609,
+            -0.006437857349742842,
+            -0.009764856421740622,
+            -0.02058868349990357,
+            -0.014866137971603676,
+            0.0012450207572602557,
+            -0.002123265406747006,
+            0.006000473862248991,
+            -0.004054016790850226,
+            0.009584879012937315,
+            -0.00608816807208866,
+            -0.00925801057439376,
+            -0.013962683072792106,
+            -0.008924356118158748,
+            -0.004210064145949253,
+            -0.004381709838058668,
+            -0.011732699976353612,
+            -0.005479523097469261,
+            0.0010680258077037758,
+            -0.0019401640391035755,
+            -0.01397652201595301,
+            0.011650839243603377,
+            0.0058731838291845645,
+            -0.002462743134993977,
+            0.02219008688925881,
+            0.01097316136828334,
+            0.0018083001685932406,
+            0.01366095798812196,
+            0.0003942601242216818,
+            -0.0064063532958756175,
+            -0.03343312148366622,
+            -0.0005083355836126291,
+            -0.02815586367527057,
+            0.025770078162165123,
+            -0.0069114949102666145,
+            -0.010322087105114614,
+            -0.002519047562374146,
+            -0.006063669644021698,
+            0.005873609611264377,
+            -0.002722940652283699,
+            0.0021229525800901114,
+            -0.023673778230768945,
+            0.0028629715379660113,
+            -0.003953653437464966,
+            -0.0036366957271968823,
+            0.007469664864329332,
+            0.0346520690389124,
+            -0.004054282838167587,
+            -0.0034765253405897676,
+            0.005159466707114532,
+            0.011642913651743502,
+            0.027845571220425726,
+            -0.004445196354520585,
+            -0.0032038508043339413,
+            -0.02233838584152591,
+            0.0054112576531374175,
+            -0.00019544838850471866,
+            -0.004733192299951088,
+            0.016664217348411003,
+            -0.016719175418727594,
+            -0.01631658832047864,
+            -0.00545969677717507,
+            -0.014570848053102338,
+            -0.00022620258237138677,
+            0.014454101124666536,
+            0.011056312437637014,
+            -0.004035526363810078,
+            -0.012459378363360557,
+            -0.0022595493558020706,
+            -0.030370952502579162,
+            0.012766010284884204,
+            0.1141689619126009,
+            -0.00766197047210299,
+            -0.00896655658602649,
+            -0.009481112393590025,
+            0.001435269335646347,
+            0.001338755472877713,
+            -0.011739143028767118,
+            0.07166096084053437,
+            -0.0077855767070652395,
+            0.0002139580430342728,
+            -0.011312198831181402,
+            0.001844005266872301,
+            -0.001825412693163496,
+            0.01371932421024605,
+            -0.005025814503687022,
+            -0.018213360991353643,
+            -0.0033612481616769416,
+            -0.02941403830641238,
+            -0.0041137865872933515,
+            -0.005683630256743215,
+            -0.004288198570156578,
+            -0.014982999363445122,
+            0.0067149665791261335,
+            -0.019661973497372313,
+            -0.012688556129040908,
+            0.001124391650175233,
+            -0.0033623453063430534,
+            0.00996944383866679,
+            -0.020496336814603385,
+            -0.0044354321069609536,
+            0.0019875840782527733,
+            0.009706675240563883,
+            0.0011600364428219214,
+            -0.012755178230821602,
+            -0.0026514442584739412,
+            0.004056350166031111,
+            0.01909951327913368,
+            0.014360281326789544,
+            0.003732344102045803,
+            0.008623457695718206,
+            0.014461868085333023,
+            -0.001692149528228451,
+            0.01948035376572433,
+            0.01172550239477637,
+            0.0031966531144076173,
+            0.0004779413449255626,
+            0.0030912562273611517,
+            -0.004597633235071031,
+            0.01582547400214653,
+            -0.014811177637844808,
+            -0.02411202772171917,
+            -0.0071272238837422185,
+            0.018015541137524113,
+            -0.013081287852028074,
+            1.8837689271256874e-05,
+            0.004062803284646432,
+            0.027569726868200254,
+            0.0010564850664255057,
+            -0.01179755720247972,
+            0.0028876134800690334,
+            -0.01638215925131053,
+            0.022944183106912663,
+            0.0026043713064243836,
+            0.03279173899194346,
+            -0.005103903273968361,
+            0.00578213669463383,
+            -0.020744194169889544,
+            -0.007588793365216579,
+            -0.00819862611489168,
+            0.0017401442433921013,
+            0.010993500174794084,
+            0.005411238350013134,
+            0.01620925462604661,
+            -0.00641615069699833,
+            -0.020751730783437995,
+            0.014776878596707538,
+            -0.009552193304486048,
+            -0.03393319554458277,
+            0.008881987295131347,
+            -0.029102297123590998,
+            -0.0045267103602711,
+            0.00368971742585232,
+            -0.014269341759285646,
+            -0.016264809810970175,
+            0.012837302095591106,
+            -0.0011890334180728564,
+            -0.008937992768323412,
+            0.0014444613725248846,
+            -0.017110187133760626,
+            -0.02385668037818897,
+            0.006017405364919834,
+            0.00676670672597817,
+            -0.0077994500447595,
+            0.018312452858615755,
+            -0.027015660037024585,
+            0.006123912473734307,
+            -0.012973676426738582,
+            0.009041974762537394,
+            -0.01580183887345168,
+            0.01163516332546347,
+            0.010887662470499361,
+            -0.001503467664709027,
+            -0.0076234124970223326,
+            0.01231202541338883,
+            0.004852720134894811,
+            0.033760365210093034,
+            0.028658326381800395,
+            0.000995999783627898,
+            -0.0025605367058226737,
+            -0.0020356794893366384,
+            0.011667525096213252,
+            -0.014049700287539817,
+            0.011218246246280464,
+            0.02658113044439717,
+            0.017568631325838108,
+            -0.019133062654008885,
+            0.018050678772708717,
+            -0.010356118256625172,
+            0.012872052420999813,
+            -0.025620154415048252,
+            -0.005135768485044442,
+            0.004936490920616527,
+            0.005163008729256001,
+            0.007425278892536464,
+            0.001410212306439329,
+            0.021891387289347516,
+            0.014999616600766989,
+            0.00027613811156650413,
+            0.00558422822386443,
+            0.01477009526981092,
+            0.010497782599293994,
+            -0.0005567894749233204,
+            -0.00467776506604011,
+            -0.016998035358377343,
+            0.00957271797745408,
+            -0.006065116857898469,
+            0.005138070401846389,
+            0.01698914855850691,
+            -0.01230257809343985,
+            0.011109844802663414,
+            0.007518553660463051,
+            -0.015504773927595823,
+            -0.017169933945839305,
+            0.005552829003043926,
+            -0.02252633790984703,
+            -0.01306722099657118,
+            0.02715689072730398,
+            -0.010513941399990838,
+            -0.02790582732974509,
+            -0.0009847768069205477,
+            0.009797865241108237,
+            -0.002772952073769185,
+            -0.0027968741526000576,
+            0.0037541733651479613,
+            0.010801137722284016,
+            -0.014658156970099418,
+            0.002075415233345957,
+            0.004735264309137368,
+            0.01930804315461151,
+            -0.0014151522239496115,
+            -0.015046118414100617,
+            -0.002288495894227231,
+            0.021592829677937263,
+            0.017174609713578173,
+            0.00265560378681856,
+            -0.016499825201963983,
+            -3.965296928410772e-05,
+            -0.0074142747014567635,
+            0.00021360696629503726,
+            0.00027431174904249525,
+            -0.0056391089282546054,
+            -0.018931681700770225,
+            -0.003888248121485032,
+            0.011181675419027736,
+            -0.00247252920175839,
+            0.01977860312223846,
+            -0.0011527575128223392,
+            -0.015763587433200915,
+            0.001960734212846797,
+            -0.005280860341480037,
+            -0.010372829570478649,
+            0.01351900743232294,
+            -0.009359942374603997,
+            0.006788746813046239,
+            -0.003212015225284119,
+            0.004611143085183367,
+            -0.0003308099698088422,
+            -0.017017265064185255,
+            0.02924142709831441,
+            -0.0018238549856433466,
+            -0.004769274041546295,
+            -0.0014058363976586562,
+            -0.033393331166741945,
+            0.015017165991303838,
+            -0.014574579250749552,
+            0.006745699409061803,
+            0.002705853802154516,
+            0.01272741296341735,
+            0.003253095246601883,
+            -0.008898616461779179,
+            0.0027485606928249326,
+            0.017040360927723885,
+            0.013756402798365262,
+            -0.007292864293090161,
+            0.008694855886257706,
+            -0.0028934389922887676,
+            -0.0037483208383542015,
+            0.01363963339139356,
+            0.004170611039133969,
+            -0.006282657671540625,
+            -0.0009784372731140945
+          ],
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "Mixture-of-Experts (MoE)",
+            "fine-grained expert segmentation",
+            "shared expert isolation"
+          ],
+          "title": "Deepseekmoe: Towards Ultimate Expert Specialization In Mixture-Of-Experts Language Models"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "396d9412-17a9-4b7e-831f-84f4e5006484",
+        "properties": {
+          "page_content": "9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\n",
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "MoE language models",
+            "supervised fine-tuning",
+            "large-scale language models"
+          ],
+          "embedding": [
+            -0.003971853759139776,
+            -0.012639197520911694,
+            -0.032538022845983505,
+            0.01764647848904133,
+            0.028833750635385513,
+            0.017460335046052933,
+            0.0033017343375831842,
+            0.0448235459625721,
+            -0.002440816955640912,
+            0.05025895684957504,
+            -0.011708475649356842,
+            -0.010033177211880684,
+            0.026748934760689735,
+            -0.023044662550091743,
+            -0.0071293264627456665,
+            0.011140735819935799,
+            0.007008332759141922,
+            0.022765446454286575,
+            0.005472642369568348,
+            -0.03823403641581535,
+            0.014482025988399982,
+            -0.014649556018412113,
+            -0.05766750127077103,
+            -0.008972154930233955,
+            0.008334610611200333,
+            0.017088046297430992,
+            0.018679579719901085,
+            0.0209412332624197,
+            -0.06805435568094254,
+            0.020010510459542274,
+            0.045344747602939606,
+            0.012024921365082264,
+            -0.0010371727403253317,
+            -0.006017114035785198,
+            -0.006594161503016949,
+            -0.03143977001309395,
+            0.04270150139927864,
+            0.015459283255040646,
+            -0.024254601448774338,
+            0.02741905488073826,
+            0.02278406172990799,
+            -0.006980410777032375,
+            -0.02874067798256874,
+            -0.009809804148972034,
+            0.00014906085561960936,
+            0.007208437658846378,
+            -0.006026421207934618,
+            0.005379570182412863,
+            -0.0071293264627456665,
+            -0.005030549596995115,
+            -0.012173837050795555,
+            -0.014119044877588749,
+            0.02192779816687107,
+            -0.01348615437746048,
+            -0.025278395041823387,
+            0.010154170915484428,
+            0.014649556018412113,
+            0.0407283715903759,
+            0.04463740065693855,
+            0.026693090796470642,
+            0.021778881549835205,
+            -0.030192604288458824,
+            0.003183067310601473,
+            -0.000935375050175935,
+            0.02533423714339733,
+            0.013030100613832474,
+            0.004416273441165686,
+            0.03518127277493477,
+            0.023658938705921173,
+            -0.02598574385046959,
+            -0.049328237771987915,
+            0.02230008691549301,
+            -0.007487654220312834,
+            -0.012797419913113117,
+            0.004241763148456812,
+            -0.00017014751210808754,
+            0.0030667271930724382,
+            0.0407283715903759,
+            -0.023510023951530457,
+            0.003876454895362258,
+            0.010749832727015018,
+            0.013662991113960743,
+            0.010498537681996822,
+            -0.0014414548641070724,
+            0.04325993359088898,
+            -0.014845007099211216,
+            -0.010051791556179523,
+            -0.09188082069158554,
+            0.0304159764200449,
+            -0.010126249864697456,
+            -0.020475871860980988,
+            0.006919913925230503,
+            0.013616454787552357,
+            -0.011280343867838383,
+            -0.009781882166862488,
+            0.027605198323726654,
+            0.01356991846114397,
+            0.006398709956556559,
+            -0.0263208020478487,
+            0.01287187822163105,
+            -0.013030100613832474,
+            0.0022663066629320383,
+            0.012918413616716862,
+            0.003997448831796646,
+            -0.00012244803656358272,
+            -0.008799971081316471,
+            -0.029466642066836357,
+            0.030267061665654182,
+            0.014156273566186428,
+            0.10290056467056274,
+            0.00873482134193182,
+            -0.0014821739168837667,
+            -0.035088200122117996,
+            0.015133530832827091,
+            -0.028722064569592476,
+            -0.01276949793100357,
+            -0.0439300537109375,
+            -0.03203543275594711,
+            0.005868198815733194,
+            0.0019475346198305488,
+            0.03531157225370407,
+            0.01991743966937065,
+            0.02561345510184765,
+            -0.0014717032900080085,
+            -0.06310291588306427,
+            0.046982817351818085,
+            -0.015608198940753937,
+            -0.02019665576517582,
+            0.010358929634094238,
+            -0.004600090906023979,
+            -0.016101481392979622,
+            -0.018028074875473976,
+            -0.033115070313215256,
+            -0.04102620109915733,
+            0.0028037982992827892,
+            -0.010991820134222507,
+            0.014584405347704887,
+            0.06392195075750351,
+            -0.040951743721961975,
+            -0.01026585791260004,
+            0.015989795327186584,
+            0.010935977101325989,
+            0.009302561171352863,
+            -0.020866775885224342,
+            -0.029299112036824226,
+            -0.0041649783961474895,
+            0.0076691447757184505,
+            0.08391384780406952,
+            -0.007845981977880001,
+            0.02118322066962719,
+            -0.024831648916006088,
+            0.011205886490643024,
+            0.011969078332185745,
+            0.004658260848373175,
+            0.0013623435515910387,
+            -0.013523383066058159,
+            -0.030490435659885406,
+            0.005826316308230162,
+            -0.021034304052591324,
+            0.02980170026421547,
+            0.0025920593179762363,
+            -0.025632068514823914,
+            -0.0009185057133436203,
+            0.01166194025427103,
+            -0.0058030481450259686,
+            0.017702322453260422,
+            0.021127376705408096,
+            0.0077249882742762566,
+            -0.060087379068136215,
+            0.053758472204208374,
+            0.01866096444427967,
+            0.0218347255140543,
+            -0.024329058825969696,
+            0.004721084609627724,
+            0.00043831163202412426,
+            0.017972230911254883,
+            0.0070595224387943745,
+            0.011410645209252834,
+            -0.005281844176352024,
+            0.033561814576387405,
+            -0.015245217829942703,
+            0.023863697424530983,
+            0.021648580208420753,
+            -0.0014717032900080085,
+            0.030453206971287727,
+            0.013495461083948612,
+            0.018474821001291275,
+            0.02894543670117855,
+            -0.0181862972676754,
+            0.005430759862065315,
+            0.014044586569070816,
+            0.00012295703345444053,
+            0.029448026791214943,
+            -0.006738423369824886,
+            -0.007389928214251995,
+            -0.037340544164180756,
+            -0.018484128639101982,
+            0.02455243282020092,
+            -0.0004868836549576372,
+            0.009372364729642868,
+            -0.00881858542561531,
+            -0.03901584446430206,
+            -0.004251070320606232,
+            -0.012946335598826408,
+            0.028870979323983192,
+            0.005100353620946407,
+            -0.011150043457746506,
+            0.005807701963931322,
+            0.022058097645640373,
+            -0.013672297820448875,
+            -0.012434438802301884,
+            -0.007520229555666447,
+            0.002373339608311653,
+            -0.02220701426267624,
+            -0.0041021546348929405,
+            0.0037182322703301907,
+            -0.01921009086072445,
+            0.0027642427012324333,
+            0.008585905656218529,
+            -0.00825549941509962,
+            -0.01627831906080246,
+            0.015040459111332893,
+            0.03441808000206947,
+            -0.016203859820961952,
+            -0.03585138916969299,
+            -0.024310443550348282,
+            -0.022560687735676765,
+            0.018996024504303932,
+            0.028498690575361252,
+            -0.009511973708868027,
+            0.03091856651008129,
+            0.008367185480892658,
+            0.014295881614089012,
+            0.004711777437478304,
+            0.021127376705408096,
+            -0.008074008859694004,
+            -0.011354802176356316,
+            -0.018791265785694122,
+            0.014212116599082947,
+            0.026488332077860832,
+            0.00021682900842279196,
+            0.018260754644870758,
+            -0.04556812345981598,
+            -0.011540945619344711,
+            0.0020394434686750174,
+            -0.0006038055289536715,
+            -0.012406516820192337,
+            -0.016864672303199768,
+            0.01268573384732008,
+            0.029652785509824753,
+            0.008381146937608719,
+            -0.016557535156607628,
+            -0.002878256142139435,
+            0.00434181559830904,
+            -0.04084005579352379,
+            0.052455462515354156,
+            0.03990933671593666,
+            -0.012648504227399826,
+            -0.024366287514567375,
+            -0.033282600343227386,
+            0.05710906907916069,
+            -0.02380785532295704,
+            -0.025222551077604294,
+            0.06961796432733536,
+            -0.013849135488271713,
+            0.019768523052334785,
+            0.05707183852791786,
+            0.0026944386772811413,
+            0.004486077465116978,
+            -0.01794430986046791,
+            0.026227731257677078,
+            -0.021369364112615585,
+            -0.01190392766147852,
+            0.006394056603312492,
+            0.03030429035425186,
+            -0.012332059442996979,
+            -0.01941484957933426,
+            -0.002408241853117943,
+            0.0011081402190029621,
+            0.015059073455631733,
+            0.002245365409180522,
+            0.009539894759654999,
+            0.013514075428247452,
+            0.012536818161606789,
+            0.00968881044536829,
+            -0.0047303917817771435,
+            -0.01110350713133812,
+            -0.04325993359088898,
+            -0.03162591531872749,
+            -0.01740449108183384,
+            -0.035572174936532974,
+            0.0013146440032869577,
+            -0.021909182891249657,
+            0.011205886490643024,
+            0.03786174952983856,
+            -0.009027997963130474,
+            0.003348270431160927,
+            0.018884338438510895,
+            -0.0023140062112361193,
+            -0.0005747204995714128,
+            -0.031718987971544266,
+            -0.018577199429273605,
+            -0.00893958006054163,
+            0.00128555903211236,
+            -0.05126413702964783,
+            -0.01171778328716755,
+            0.008418375626206398,
+            0.03103025257587433,
+            -0.02250484563410282,
+            -0.023621710017323494,
+            0.015878107398748398,
+            -0.019805751740932465,
+            -0.028647607192397118,
+            -0.02931772544980049,
+            -0.019191475585103035,
+            0.05465196445584297,
+            0.016706449910998344,
+            0.025967128574848175,
+            -0.03147700056433678,
+            -0.03423193469643593,
+            0.05025895684957504,
+            0.008474218659102917,
+            -0.07550012320280075,
+            0.036595966666936874,
+            0.0014507620362564921,
+            0.0010063425870612264,
+            -0.005495910067111254,
+            0.05837484821677208,
+            -0.0032575251534581184,
+            -0.0010057608596980572,
+            0.012536818161606789,
+            -0.05748135596513748,
+            -0.00575651228427887,
+            -0.006412670947611332,
+            0.013253473676741123,
+            -0.04642438516020775,
+            -0.024943334981799126,
+            0.004027697257697582,
+            -0.012974257580935955,
+            -0.005700668785721064,
+            -0.042180296033620834,
+            -0.03145838528871536,
+            -0.0028526613023132086,
+            -0.0022756140679121017,
+            -0.007943707518279552,
+            -0.03078826516866684,
+            -0.025929899886250496,
+            0.021164605394005775,
+            0.045344747602939606,
+            0.007641223259270191,
+            0.0017695341957733035,
+            0.023621710017323494,
+            0.0197871383279562,
+            0.005598289426416159,
+            0.020345570519566536,
+            -0.01645515486598015,
+            -0.0375266894698143,
+            -0.005495910067111254,
+            -0.00010630584438331425,
+            0.02680477872490883,
+            -0.00926067866384983,
+            -0.049700524657964706,
+            -0.033599045127630234,
+            0.0193403922021389,
+            -0.03687518462538719,
+            -0.015487205237150192,
+            -0.004313894081860781,
+            -0.033636271953582764,
+            0.013141786679625511,
+            0.00029579491820186377,
+            -0.01730211265385151,
+            0.049328237771987915,
+            -0.023454179987311363,
+            -0.0019975609611719847,
+            -0.023752011358737946,
+            0.003443669294938445,
+            0.05312558263540268,
+            -0.018502742052078247,
+            -0.018028074875473976,
+            0.022672373801469803,
+            -0.01730211265385151,
+            0.01849343627691269,
+            0.04314824566245079,
+            0.009837726131081581,
+            0.004304586909711361,
+            0.014724013395607471,
+            0.006226526573300362,
+            -0.008483526296913624,
+            -0.0386807844042778,
+            0.021871954202651978,
+            0.012220372445881367,
+            -0.045307520776987076,
+            -0.022597916424274445,
+            -0.02479442022740841,
+            0.0007329431246034801,
+            0.011801548302173615,
+            0.016557535156607628,
+            -0.00968881044536829,
+            -0.028387004509568214,
+            0.006421978119760752,
+            -0.016017716377973557,
+            0.010163478553295135,
+            0.00786459632217884,
+            0.014845007099211216,
+            -0.005519178230315447,
+            0.0067756520584225655,
+            0.004006756003946066,
+            0.012788113206624985,
+            0.0362422950565815,
+            -0.02721429616212845,
+            -0.013216244988143444,
+            -0.020271113142371178,
+            -0.05506148189306259,
+            0.0014589058700948954,
+            0.015059073455631733,
+            0.004544247407466173,
+            -0.02455243282020092,
+            0.005709975957870483,
+            0.012546124868094921,
+            -0.04180800914764404,
+            -0.003641447750851512,
+            0.012099378742277622,
+            0.034678682684898376,
+            0.028238089755177498,
+            -0.011596789583563805,
+            0.003436689032241702,
+            0.0011959770927205682,
+            -0.0032435643952339888,
+            -0.0007631916087120771,
+            -0.023100506514310837,
+            0.027493512257933617,
+            0.014649556018412113,
+            0.038941387087106705,
+            -0.023472795262932777,
+            -0.005998499691486359,
+            -0.048211369663476944,
+            -0.003466937458142638,
+            0.030248448252677917,
+            -0.020680630579590797,
+            -0.011736397631466389,
+            -0.0005956617533229291,
+            0.005044510122388601,
+            -0.014733321033418179,
+            -0.006338213104754686,
+            -0.03194236010313034,
+            -0.026674477383494377,
+            -0.04143571853637695,
+            -0.01702289469540119,
+            -0.00934909749776125,
+            -0.01416558027267456,
+            -0.05748135596513748,
+            -0.011364108882844448,
+            -0.09463575482368469,
+            0.009218796156346798,
+            0.04117511585354805,
+            -0.0257437564432621,
+            0.028238089755177498,
+            -0.028796521946787834,
+            -0.0035576827358454466,
+            -0.03501374274492264,
+            0.012648504227399826,
+            0.040616683661937714,
+            0.033115070313215256,
+            -0.015291753225028515,
+            -0.006966450251638889,
+            0.017516177147626877,
+            -0.030732423067092896,
+            -0.03802927955985069,
+            -0.019228706136345863,
+            -0.00846025813370943,
+            0.00572859076783061,
+            0.0018474820535629988,
+            0.03789897635579109,
+            0.04504691809415817,
+            0.04631270095705986,
+            0.03592584654688835,
+            0.00934909749776125,
+            -0.035367414355278015,
+            -0.010740526020526886,
+            0.005216693971306086,
+            0.03071380779147148,
+            -0.03305922448635101,
+            0.01166194025427103,
+            -0.0311233252286911,
+            0.01853066496551037,
+            0.002405914943665266,
+            -0.023454179987311363,
+            0.021853338927030563,
+            0.005947310011833906,
+            0.0013507094699889421,
+            -0.005919388495385647,
+            0.027325982227921486,
+            -0.01828867755830288,
+            0.035125426948070526,
+            -0.028442848473787308,
+            -0.024105684831738472,
+            -0.028684835880994797,
+            -0.011122121475636959,
+            0.031104711815714836,
+            0.019154246896505356,
+            -0.0005947891622781754,
+            -0.002433836692944169,
+            -0.015273138880729675,
+            0.009172259829938412,
+            0.004853712394833565,
+            -0.02032695710659027,
+            -0.016334161162376404,
+            0.026879236102104187,
+            -0.06943181902170181,
+            -0.009763267822563648,
+            0.008306688629090786,
+            0.0005555243697017431,
+            0.00929790735244751,
+            -0.0386807844042778,
+            0.010377543978393078,
+            0.004590783733874559,
+            -0.03990933671593666,
+            0.022877132520079613,
+            0.00926067866384983,
+            0.01971268095076084,
+            -0.004362756852060556,
+            -0.008837200701236725,
+            0.007022293284535408,
+            0.06451760977506638,
+            -0.010563688352704048,
+            0.0022791041992604733,
+            0.007455078884959221,
+            -0.00812519807368517,
+            0.03845741227269173,
+            -0.025222551077604294,
+            -0.028033331036567688,
+            0.02935495413839817,
+            -0.008455604314804077,
+            -0.007650530431419611,
+            -0.015310367569327354,
+            -0.006696540862321854,
+            0.007827367633581161,
+            0.02870344929397106,
+            -0.02373339794576168,
+            -0.005212040152400732,
+            -0.02676755003631115,
+            0.006524357479065657,
+            0.0022500192280858755,
+            0.002908504568040371,
+            0.0061334543861448765,
+            -0.0009958719601854682,
+            0.039685964584350586,
+            -0.0225420743227005,
+            0.03452976793050766,
+            0.02959694154560566,
+            -0.004090521018952131,
+            -0.0045582083985209465,
+            -0.0005939166294410825,
+            0.013179016299545765,
+            -0.01289049256592989,
+            -0.01808391883969307,
+            0.01736726239323616,
+            -0.018679579719901085,
+            -0.013244166038930416,
+            -0.054949793964624405,
+            -0.002410568529739976,
+            0.002610673662275076,
+            -0.010051791556179523,
+            0.006784959230571985,
+            -0.001237859483808279,
+            0.002903850981965661,
+            0.005393530707806349,
+            0.03949981927871704,
+            -0.03827126696705818,
+            0.023714782670140266,
+            0.00015429616905748844,
+            -0.0026525561697781086,
+            0.008716206066310406,
+            0.0035018394701182842,
+            0.007106058299541473,
+            0.05554545670747757,
+            0.012899799272418022,
+            -0.007994897663593292,
+            -0.005063124932348728,
+            0.005002627614885569,
+            0.016296932473778725,
+            -0.016027024015784264,
+            0.0052073863334953785,
+            0.03668903931975365,
+            0.025297008454799652,
+            0.000632890616543591,
+            -0.0017951290355995297,
+            -0.010182092897593975,
+            0.02751212567090988,
+            -0.021034304052591324,
+            0.029820315539836884,
+            0.04307378828525543,
+            0.009470091201364994,
+            -0.0439300537109375,
+            0.0069571430794894695,
+            -0.015431362204253674,
+            -0.009251371026039124,
+            0.0031528188847005367,
+            0.019600993022322655,
+            -0.04381836578249931,
+            0.003245891071856022,
+            0.01145718153566122,
+            0.008320650085806847,
+            -0.021052919328212738,
+            -0.04675944522023201,
+            -0.008674323558807373,
+            -0.0035274343099445105,
+            0.012974257580935955,
+            -0.016846058890223503,
+            -0.006510396488010883,
+            0.005742551293224096,
+            0.016259703785181046,
+            -0.004988667089492083,
+            0.012034228071570396,
+            0.033394284546375275,
+            -0.016408618539571762,
+            0.013783984817564487,
+            -0.007054868619889021,
+            -0.02189056947827339,
+            0.0110476640984416,
+            0.03095579519867897,
+            -0.01307663694024086,
+            0.03546048700809479,
+            -0.0015345270512625575,
+            0.012415824458003044,
+            0.012964949943125248,
+            0.003639120841398835,
+            -0.014817086048424244,
+            0.002092959824949503,
+            0.018335212022066116,
+            -0.021443821489810944,
+            0.013020792976021767,
+            -0.013541997410356998,
+            -0.037992049008607864,
+            0.010051791556179523,
+            -0.0038950692396610975,
+            0.0016752986703068018,
+            0.009041959419846535,
+            0.0062591019086539745,
+            -0.027195680886507034,
+            -0.01611078903079033,
+            0.023342493921518326,
+            0.024105684831738472,
+            0.06098087131977081,
+            -0.014081815257668495,
+            -0.000535164843313396,
+            -0.007617955096065998,
+            0.009493358433246613,
+            0.04460017383098602,
+            0.024459360167384148,
+            0.0038694743998348713,
+            0.004290625918656588,
+            0.027102608233690262,
+            0.046945590525865555,
+            -0.005230654496699572,
+            -0.010377543978393078,
+            0.006394056603312492,
+            0.003197028301656246,
+            0.023789240047335625,
+            0.021574122831225395,
+            -0.0005409818259067833,
+            0.014072508551180363,
+            0.014444797299802303,
+            -0.016538919880986214,
+            0.012937027961015701,
+            -0.04556812345981598,
+            -0.040169939398765564,
+            0.007115365471690893,
+            0.00435112277045846,
+            0.019359005615115166,
+            0.002450124127790332,
+            -0.005039856769144535,
+            -0.0027851839549839497,
+            0.01750687137246132,
+            -0.010535767301917076,
+            -0.0122948307543993,
+            0.02475719153881073,
+            -0.00873482134193182,
+            0.006747730541974306,
+            0.00030801063985563815,
+            -0.009716732427477837,
+            0.0035181271377950907,
+            -0.0014972981298342347,
+            -0.005077085457742214,
+            0.0023989344481378794,
+            -0.008381146937608719,
+            -0.017516177147626877,
+            0.012229680083692074,
+            -0.04396728053689003,
+            0.013886364176869392,
+            0.001635742955841124,
+            -0.008911658078432083,
+            0.002878256142139435,
+            0.0013146440032869577,
+            0.0114944102242589,
+            -0.018381748348474503,
+            -0.005309765692800283,
+            0.012453053146600723,
+            0.026413874700665474,
+            -0.007403889205306768,
+            0.0038904156535863876,
+            0.015142838470637798,
+            -0.007534190081059933,
+            -0.00987495481967926,
+            0.028070559725165367,
+            -0.007822713814675808,
+            0.006859417073428631,
+            -0.008181042037904263,
+            -0.02540869638323784,
+            0.0053888773545622826,
+            0.03793620690703392,
+            0.04865811765193939,
+            0.007538843899965286,
+            0.01648307777941227,
+            0.03123501129448414,
+            0.014835700392723083,
+            0.00822292361408472,
+            0.015422054566442966,
+            -0.04437679797410965,
+            0.01541274692863226,
+            -0.018037382513284683,
+            0.040542226284742355,
+            0.030844109132885933,
+            -0.05044510215520859,
+            -0.02812640182673931,
+            0.019284548237919807,
+            0.023100506514310837,
+            0.017562713474035263,
+            0.013681605458259583,
+            -0.005314419511705637,
+            0.02053171582520008,
+            -0.023249421268701553,
+            -0.01235998049378395,
+            0.01316040102392435,
+            -0.0032435643952339888,
+            -0.004816483706235886,
+            0.005067778285592794,
+            0.03912752866744995,
+            -0.00436741067096591,
+            -0.016911208629608154,
+            0.02131352201104164,
+            -0.005984539166092873,
+            0.005928695667535067,
+            -0.013123172335326672,
+            0.0009958719601854682,
+            0.03916475921869278,
+            0.005393530707806349,
+            0.03761976212263107,
+            -0.021499665454030037,
+            0.012145915068686008,
+            -0.028182245790958405,
+            -0.014733321033418179,
+            0.029243268072605133,
+            -0.013216244988143444,
+            -0.021201834082603455,
+            0.016976360231637955,
+            0.020513100549578667,
+            0.0032156426459550858,
+            0.011168657802045345,
+            -0.013681605458259583,
+            -0.010600917041301727,
+            0.038159579038619995,
+            -0.0029527137521654367,
+            -0.004144037142395973,
+            0.01712527498602867,
+            -0.01812114752829075,
+            -0.011922542005777359,
+            0.016883287578821182,
+            -0.0019580053631216288,
+            -0.032742779701948166,
+            0.02241177298128605,
+            -0.0004022461944259703,
+            0.03620506450533867,
+            -0.010237935930490494,
+            -0.0056634400971233845,
+            0.010535767301917076,
+            -0.006654658354818821,
+            -0.029987845569849014,
+            -0.034176092594861984,
+            0.007455078884959221,
+            -0.027437668293714523,
+            0.010963899083435535,
+            -0.006650005001574755,
+            0.03255663812160492,
+            -0.030248448252677917,
+            0.008669670671224594,
+            0.03030429035425186,
+            -0.00825549941509962,
+            0.022690989077091217,
+            0.032947540283203125,
+            -0.019079789519309998,
+            -0.03525573015213013,
+            -0.003245891071856022,
+            -0.0003592003195080906,
+            0.029578328132629395,
+            0.002880582818761468,
+            0.0026246344204992056,
+            0.004960745573043823,
+            -0.016222475096583366,
+            -0.03510681167244911,
+            0.022430386394262314,
+            -0.04474908858537674,
+            0.007068829610943794,
+            0.026618633419275284,
+            -0.0430365614593029,
+            -0.036800727248191833,
+            -0.018446899950504303,
+            -0.0263208020478487,
+            -0.02159273810684681,
+            0.001237859483808279,
+            0.02639526128768921,
+            -0.017581328749656677,
+            -0.030602121725678444,
+            0.028089173138141632,
+            0.0034948589745908976,
+            -0.012695040553808212,
+            -0.011056970804929733,
+            -0.010126249864697456,
+            -0.027083994820713997,
+            -0.022262856364250183,
+            -0.014035279862582684,
+            0.0009807477472350001,
+            0.004881633911281824,
+            -0.010368237271904945,
+            -0.0225420743227005,
+            -0.013616454787552357,
+            -0.009032651782035828,
+            -0.004825790878385305,
+            0.050147272646427155,
+            0.021090148016810417,
+            0.023863697424530983,
+            0.0009185057133436203,
+            -0.015719885006546974,
+            0.04601486772298813,
+            -0.0007253810181282461,
+            0.02591128461062908,
+            0.005612250417470932,
+            -0.022058097645640373,
+            -0.005844930652529001,
+            0.03715439885854721,
+            -0.018279369920492172,
+            -0.004395332187414169,
+            -0.014724013395607471,
+            -0.00374150020070374,
+            -0.008609173819422722,
+            0.026693090796470642,
+            0.012024921365082264,
+            -0.004769947379827499,
+            0.013458232395350933,
+            -0.01270434819161892,
+            0.02200225554406643,
+            0.005821662489324808,
+            -0.003676349762827158,
+            -0.0010331007651984692,
+            -0.028684835880994797,
+            -0.02114599198102951,
+            0.001068002893589437,
+            0.022765446454286575,
+            -0.016334161162376404,
+            -0.013197630643844604,
+            -0.015291753225028515,
+            -0.021574122831225395,
+            -0.005198079161345959,
+            0.0004831026017200202,
+            0.005407491698861122,
+            0.015850186347961426,
+            -0.003311041509732604,
+            -0.0430365614593029,
+            -0.005272537004202604,
+            -0.03763837739825249,
+            0.00048455686192028224,
+            0.01961960829794407,
+            -0.011243115179240704,
+            0.01128965150564909,
+            0.001805599662475288,
+            -0.0038648208137601614,
+            -0.01832590624690056,
+            0.007636569440364838,
+            0.01982436701655388,
+            -0.006789613049477339,
+            -0.0022197705693542957,
+            0.018986716866493225,
+            0.0021569470409303904,
+            -0.007645876612514257,
+            0.004849058575928211,
+            -0.001855625887401402,
+            -0.006845456548035145,
+            -0.023379722610116005,
+            -0.029280496761202812,
+            -0.013607148081064224,
+            0.0018090897938236594,
+            -0.001733468729071319,
+            -0.024906106293201447,
+            0.018875030800700188,
+            -0.006082264706492424,
+            0.02336110919713974,
+            -0.011326880194246769,
+            0.01316040102392435,
+            -0.0022697970271110535,
+            -0.0090931486338377,
+            0.018214218318462372,
+            -0.010554381646215916,
+            -0.03264970704913139,
+            -0.01316040102392435,
+            0.028517305850982666,
+            0.00034872969263233244,
+            -0.008097277022898197,
+            0.010042484849691391,
+            0.00922810286283493,
+            -0.027437668293714523,
+            -0.0077017201110720634,
+            0.010619531385600567,
+            0.006626736838370562,
+            -0.032333262264728546,
+            0.026506947353482246,
+            0.03266832232475281,
+            0.01235998049378395,
+            -0.03222157806158066,
+            0.048620887100696564,
+            0.008651056326925755,
+            -0.02475719153881073,
+            0.005677401088178158,
+            0.020624786615371704,
+            0.0363725945353508,
+            -0.0299692302942276,
+            -0.007413196377456188,
+            -0.007422503549605608,
+            0.009009383618831635,
+            -0.03473452478647232,
+            0.015235910192131996,
+            -0.02537146769464016,
+            0.01961960829794407,
+            0.032333262264728546,
+            0.0032831199932843447,
+            0.040579456835985184,
+            -0.009511973708868027,
+            0.017693014815449715,
+            0.0060310750268399715,
+            -0.0076691447757184505,
+            0.006459206808358431,
+            -0.02162996679544449,
+            0.01624108850955963,
+            0.019768523052334785,
+            0.02162996679544449,
+            0.009902875870466232,
+            -0.024682732298970222,
+            0.01941484957933426,
+            0.015133530832827091,
+            0.023826468735933304,
+            0.027605198323726654,
+            -0.014658862724900246,
+            0.026842007413506508,
+            0.005812355317175388,
+            -0.013635069131851196,
+            0.004430233966559172,
+            0.009386326186358929,
+            -0.040616683661937714,
+            -0.0012948662042617798,
+            -0.022523459047079086,
+            0.01459371205419302,
+            -0.0005066614830866456,
+            -0.0028224128764122725,
+            -0.02377062663435936,
+            -0.013653683476150036,
+            -7.831148832337931e-05,
+            0.005146889481693506,
+            -0.004358103033155203,
+            -0.04087728634476662,
+            -0.045754265040159225,
+            0.0234355665743351,
+            -0.009800496511161327,
+            -0.040542226284742355,
+            -0.012648504227399826,
+            -0.009311867877840996,
+            -0.0015752461040392518,
+            -0.014109737239778042,
+            -0.01289049256592989,
+            -0.04385559633374214,
+            -0.014435489661991596,
+            0.01958237960934639,
+            0.030043689534068108,
+            -0.02203948423266411,
+            0.030248448252677917,
+            0.006873378064483404,
+            0.015198681503534317,
+            -0.014630941674113274,
+            0.013560611754655838,
+            0.00614741537719965,
+            -0.04638715833425522,
+            0.02894543670117855,
+            -0.0028898902237415314,
+            0.021276293322443962,
+            0.0384201817214489,
+            -0.005272537004202604,
+            -0.010023870505392551,
+            0.017246268689632416,
+            -0.02213255688548088,
+            -0.013551304116845131,
+            -0.008092623203992844,
+            -0.01586880162358284,
+            -0.037005484104156494,
+            -0.04884426295757294,
+            0.016129402443766594,
+            0.04080282896757126,
+            0.00911176297813654,
+            0.014426182955503464,
+            0.018679579719901085,
+            0.004311567172408104,
+            -0.033282600343227386,
+            0.0225420743227005,
+            0.0047932155430316925,
+            -0.012387902475893497,
+            -0.023826468735933304,
+            0.013653683476150036,
+            0.012778805568814278,
+            -0.006198605056852102,
+            -0.006384748965501785,
+            -0.026655862107872963,
+            -0.024477973580360413,
+            0.0021476398687809706,
+            -0.013886364176869392,
+            0.0033250025007873774,
+            -0.022765446454286575,
+            -0.02857314795255661,
+            0.032128505408763885,
+            0.03546048700809479,
+            -0.01235998049378395,
+            -0.00393229816108942,
+            -0.013579226098954678,
+            -0.008404415100812912,
+            0.008920964784920216,
+            0.005230654496699572,
+            0.017953617498278618,
+            -0.019526535645127296,
+            0.01128965150564909,
+            -0.010442694649100304,
+            0.012099378742277622,
+            -0.0762074738740921,
+            -0.01679021492600441,
+            0.025874055922031403,
+            0.022672373801469803,
+            0.0009662052034400403,
+            0.011438566260039806,
+            0.008893043734133244,
+            0.0006619756459258497,
+            -0.035088200122117996,
+            0.01921009086072445,
+            0.013793292455375195,
+            0.016222475096583366,
+            -0.01151302456855774,
+            0.005235308315604925,
+            0.023845084011554718,
+            -0.0006683743558824062,
+            -0.006319598760455847,
+            -0.01760924980044365,
+            0.011540945619344711,
+            0.020103583112359047,
+            0.0026851315051317215,
+            0.013877056539058685,
+            0.010572995990514755,
+            -0.01961960829794407,
+            -0.013011486269533634,
+            -0.0005930440966039896,
+            -0.005668093450367451,
+            -0.010963899083435535,
+            0.008706899359822273,
+            0.0007399235619232059,
+            0.03763837739825249,
+            0.024887491017580032,
+            0.02172303944826126,
+            0.015980487689375877,
+            0.0069571430794894695,
+            0.007469039876013994,
+            -0.0020917963702231646,
+            0.0018323578406125307,
+            0.01746964268386364,
+            0.03272416815161705,
+            -0.022932976484298706,
+            -0.005156196653842926,
+            0.0042533972300589085,
+            0.024831648916006088,
+            -0.008827893063426018,
+            0.025892671197652817,
+            0.030062302947044373,
+            -0.007650530431419611,
+            -0.029503870755434036,
+            0.0032924271654337645,
+            0.018502742052078247,
+            0.009302561171352863,
+            -0.01634346880018711,
+            -0.0027921644505113363,
+            0.013728141784667969,
+            0.0075434972532093525,
+            0.020922617986798286,
+            -0.004241763148456812,
+            -0.014053894206881523,
+            0.01880057342350483,
+            0.09076395630836487,
+            -0.005691361613571644,
+            -0.0012017941335216165,
+            -0.017106659710407257,
+            -0.01921009086072445,
+            0.0014670497039332986,
+            0.0098935691639781,
+            0.016613377258181572,
+            0.017209040001034737,
+            0.00374615378677845,
+            0.004586129914969206,
+            0.01455648336559534,
+            -0.011531638912856579,
+            0.009479397907853127,
+            -0.02421737276017666,
+            0.01804668828845024,
+            -0.014249345287680626,
+            -0.0067756520584225655,
+            -0.011233807541429996,
+            0.023621710017323494,
+            -0.0019568419083952904,
+            0.007720334455370903,
+            0.019154246896505356,
+            0.003871801309287548,
+            -0.0247199609875679,
+            0.006412670947611332,
+            -0.006454553455114365,
+            0.0007160738459788263,
+            -0.038866929709911346,
+            -0.018363134935498238,
+            0.01127103716135025,
+            -0.0014123697765171528,
+            -0.0011232644319534302,
+            -0.004672221839427948,
+            -0.007804099470376968,
+            -0.00025580296642147005,
+            -0.02455243282020092,
+            0.016836751252412796,
+            0.008855815045535564,
+            0.015403440222144127,
+            0.022393157705664635,
+            -0.0010522969532757998,
+            -0.0009138521272689104,
+            0.025315623730421066,
+            -0.0071293264627456665,
+            -0.004043984692543745,
+            0.027232909575104713,
+            -0.023621710017323494,
+            -0.005188771989196539,
+            0.005300458520650864,
+            -0.01692982390522957,
+            -0.002913158154115081,
+            -0.00414636405184865,
+            -0.021034304052591324,
+            -0.013206937350332737,
+            0.0014914810890331864,
+            0.020717859268188477,
+            -0.015468590892851353,
+            -0.0029178117401897907,
+            0.013048714958131313,
+            -0.005286497995257378,
+            0.007585379760712385,
+            0.0032901002559810877,
+            -0.005067778285592794,
+            -0.02114599198102951,
+            -0.026860620826482773,
+            -0.007999550551176071,
+            0.0011936502996832132,
+            2.35407078434946e-05,
+            -0.0037321930285543203,
+            -0.009372364729642868,
+            -0.004830444231629372,
+            -0.01089874841272831,
+            0.007855288684368134,
+            -0.007413196377456188,
+            -0.014928772114217281,
+            0.013988743536174297,
+            0.005226001143455505,
+            0.008436989970505238,
+            -0.031086096540093422,
+            0.013830521143972874,
+            0.020587557926774025,
+            -0.006961796432733536,
+            0.009237410500645638,
+            -0.015198681503534317,
+            -0.00025216734502464533,
+            -0.01685536466538906,
+            0.004043984692543745,
+            0.004632666241377592,
+            -0.01169916894286871,
+            0.005379570182412863,
+            0.007403889205306768,
+            -0.01913563348352909,
+            -0.006929221097379923,
+            -0.02060617320239544,
+            0.00492817023769021,
+            0.011578175239264965,
+            0.009000076912343502,
+            -0.01559889130294323,
+            -0.004886287730187178,
+            -0.03449253737926483,
+            0.016101481392979622,
+            0.017078738659620285,
+            -0.029001280665397644,
+            -0.014584405347704887,
+            0.009046612307429314,
+            0.013588533736765385,
+            -0.01771163009107113,
+            -0.012834648601710796,
+            -0.0009981987532228231,
+            0.005938002839684486,
+            -0.018744729459285736,
+            0.00533768767490983,
+            -0.0037647683639079332,
+            0.01832590624690056,
+            0.0014740300830453634,
+            0.024850262328982353,
+            -0.026581404730677605,
+            0.010107634589076042,
+            0.017004281282424927,
+            -0.027474896982312202,
+            0.0005968251498416066,
+            0.014658862724900246,
+            0.00042202402255497873,
+            0.00020286819199100137,
+            0.0004857202584389597,
+            -0.007594686932861805,
+            -0.008483526296913624,
+            0.004513998981565237,
+            -0.0014658862492069602,
+            -0.026432489976286888,
+            0.01950792223215103,
+            0.01410043053328991,
+            0.013104557991027832,
+            0.008799971081316471,
+            -0.011652632616460323,
+            0.0006020604632794857,
+            0.01573850028216839,
+            0.032742779701948166,
+            -0.0014891542959958315,
+            -0.015654735267162323,
+            0.010014562867581844,
+            0.026637248694896698,
+            0.0003402950242161751,
+            0.0008190348744392395,
+            -0.01603632979094982,
+            -0.026469718664884567,
+            0.038978613913059235,
+            0.018270062282681465,
+            0.01958237960934639,
+            -0.0018253774615004659,
+            -0.01781400851905346,
+            0.01455648336559534,
+            -0.004437214694917202,
+            0.0027595891151577234,
+            -0.015301060862839222,
+            -0.009418901056051254,
+            0.0026572097558528185,
+            0.026655862107872963,
+            -0.005374916363507509,
+            0.03382241725921631,
+            0.012052842415869236,
+            0.009423554874956608,
+            -0.035609401762485504,
+            0.0031528188847005367,
+            -0.02462689019739628,
+            0.011559560894966125,
+            -0.0094142472371459,
+            -0.0026944386772811413,
+            0.008590559475123882,
+            -0.02935495413839817,
+            -0.009032651782035828,
+            0.01011694222688675,
+            0.023063277825713158,
+            0.007366660516709089,
+            -0.023007433861494064,
+            0.020178040489554405,
+            0.0013134806649759412,
+            0.02151828072965145,
+            -0.012052842415869236,
+            0.012266908772289753,
+            -0.008744128048419952,
+            -0.017804700881242752,
+            -0.005170157644897699,
+            0.03374795988202095,
+            -0.008530061691999435,
+            -0.0036577354185283184,
+            0.006254448089748621,
+            -0.0279216431081295,
+            0.032612480223178864,
+            -0.0037182322703301907,
+            -0.0029154850635677576,
+            0.0041975537315011024,
+            -0.012006307020783424,
+            0.00991218350827694,
+            0.0094142472371459,
+            0.008115891367197037,
+            -0.018707500770688057,
+            -0.023714782670140266,
+            -0.04184523597359657,
+            -0.024533817544579506,
+            0.014733321033418179,
+            0.011503716930747032,
+            -0.015114916488528252,
+            0.03387825936079025,
+            -0.02192779816687107,
+            -0.016306240111589432,
+            -0.011447873897850513,
+            0.028684835880994797,
+            -0.03722885996103287,
+            0.00875343568623066,
+            -0.01350476872175932,
+            -0.020587557926774025,
+            -0.00148566416464746,
+            0.004900248255580664,
+            -0.002478045877069235,
+            -0.009176913648843765,
+            -0.02939218282699585,
+            0.0060636503621935844,
+            0.04999835789203644,
+            -0.0055936360731720924,
+            -0.0016415599966421723,
+            0.00418126629665494,
+            0.003774075536057353,
+            -0.014342417940497398,
+            -0.016501691192388535,
+            0.03486482426524162,
+            -0.03806650638580322,
+            0.0017520831897854805,
+            0.01845620572566986,
+            -0.003583277575671673,
+            0.00554244639351964,
+            0.004523306153714657,
+            0.003578623989596963,
+            0.012629889883100986,
+            -0.00727824168279767,
+            0.013579226098954678,
+            0.011056970804929733,
+            0.011624710634350777,
+            0.005291151348501444,
+            0.014910157769918442,
+            0.025185322389006615,
+            -0.01706012338399887,
+            0.024868877604603767,
+            0.025017792358994484,
+            0.00014222587924450636,
+            0.03581416234374046,
+            -0.019433464854955673,
+            -0.026004357263445854,
+            0.005626211408525705,
+            -0.011745704337954521,
+            -0.004711777437478304,
+            0.001653193961828947,
+            -0.0220953281968832,
+            -0.0031993549782782793,
+            -0.011401337571442127,
+            0.0011895783245563507,
+            0.014053894206881523,
+            -0.014724013395607471,
+            -0.0204944871366024,
+            0.015422054566442966,
+            0.013430310413241386,
+            0.02861037850379944,
+            0.01518937386572361,
+            -0.0034971858840435743,
+            0.02881513722240925,
+            -0.011996999382972717,
+            0.014789164066314697,
+            0.008190348744392395,
+            0.011913234367966652,
+            -0.0069571430794894695,
+            0.024329058825969696,
+            0.0019963975064456463,
+            0.009456129744648933,
+            0.001500788377597928,
+            -0.01692051626741886,
+            -0.0034297085367143154,
+            -0.034213319420814514,
+            -0.018344519659876823,
+            0.003041132353246212,
+            -0.00020417702035047114,
+            0.08748781681060791,
+            0.006980410777032375,
+            -0.02090400457382202,
+            0.003069054102525115,
+            0.0011866698041558266,
+            0.005212040152400732,
+            -0.018279369920492172,
+            0.015180067159235477,
+            -0.027456283569335938,
+            -0.020140811800956726,
+            -0.017832623794674873,
+            -0.024980563670396805,
+            0.0030969756189733744,
+            0.004339488688856363,
+            0.0102472435683012,
+            -0.027474896982312202,
+            0.0019824367482215166,
+            -0.006701194681227207,
+            -0.020717859268188477,
+            0.009027997963130474,
+            -0.01924731954932213,
+            0.004816483706235886,
+            0.011419951915740967,
+            -0.00728754885494709,
+            -0.01791638880968094,
+            0.007603994105011225,
+            0.005314419511705637,
+            0.0033878260292112827,
+            -0.01682744361460209,
+            -5.409818550106138e-05,
+            -0.00768775912001729,
+            0.004092847462743521,
+            0.009595737792551517,
+            -0.013821213506162167,
+            0.015505819581449032,
+            0.006366134621202946,
+            -0.008790664374828339,
+            0.024515202268958092,
+            0.00873482134193182,
+            -0.02241177298128605,
+            -0.0018463187152519822,
+            -0.011689861305058002,
+            0.005658786278218031,
+            -0.0016799522563815117,
+            0.004206860903650522,
+            -0.011354802176356316,
+            -0.016287624835968018,
+            -0.02693507820367813,
+            -0.0008167080814018846,
+            -0.006645351182669401,
+            -0.005374916363507509,
+            0.0036554085090756416,
+            -0.006380095612257719,
+            0.011764319613575935,
+            0.00553313922137022,
+            -0.01396082155406475,
+            0.01045200228691101,
+            -0.0005985702155157924,
+            0.013169708661735058,
+            -0.026879236102104187,
+            0.005314419511705637,
+            -0.026655862107872963,
+            0.009642274118959904,
+            0.01330001000314951,
+            -0.038866929709911346,
+            0.015533741563558578,
+            -0.007101404946297407,
+            0.015570970252156258,
+            -0.0014088796451687813,
+            -0.01828867755830288,
+            0.020550329238176346,
+            0.007887864485383034,
+            -0.008841853588819504,
+            0.012332059442996979,
+            0.025073636323213577,
+            0.00967019610106945,
+            0.03830849379301071,
+            -0.016259703785181046,
+            -0.0003152818826492876,
+            -0.0064312852919101715,
+            0.007036254275590181,
+            -0.005449374206364155,
+            -0.0021301887463778257,
+            -0.020475871860980988,
+            0.029243268072605133,
+            -0.008450950495898724,
+            0.007762216962873936,
+            0.02282129041850567,
+            -0.0051050069741904736,
+            -0.009418901056051254,
+            0.004507018718868494,
+            0.012099378742277622,
+            0.018167681992053986,
+            -0.0032342569902539253,
+            0.015096302144229412,
+            -0.015487205237150192,
+            0.012211065739393234,
+            0.008250845596194267,
+            0.0015240564243867993,
+            -0.027903029695153236,
+            -0.022970205172896385,
+            -0.0017858217470347881,
+            0.0012750884052366018,
+            0.002101103775203228,
+            0.012434438802301884,
+            0.00748300040140748,
+            -0.006822188384830952,
+            0.029652785509824753,
+            -0.005798394791781902,
+            0.012099378742277622,
+            -0.0002384973777225241,
+            0.021481052041053772,
+            0.01862373575568199,
+            0.025017792358994484,
+            -0.0010325191542506218,
+            -0.01009832788258791,
+            0.005226001143455505,
+            -0.006859417073428631,
+            0.01991743966937065,
+            0.010759140364825726,
+            -0.01328139565885067,
+            -0.023752011358737946,
+            0.015049765817821026,
+            0.00011735816224245355,
+            -0.0018963449401780963,
+            0.012695040553808212,
+            -0.02336110919713974,
+            0.0009510809904895723,
+            -0.035404644906520844,
+            0.015077687799930573,
+            -0.04240366816520691,
+            -0.01672506518661976,
+            0.005221347324550152,
+            -0.014258652925491333,
+            -0.01026585791260004,
+            0.017795395106077194,
+            0.017748858779668808,
+            -0.008213616907596588,
+            0.014975308440625668,
+            -0.008697591722011566,
+            -0.009753961116075516,
+            0.013811906799674034,
+            0.012387902475893497,
+            -0.02315635047852993,
+            0.015301060862839222,
+            0.005212040152400732,
+            0.0058914669789373875,
+            -0.021871954202651978,
+            -0.02012219838798046,
+            0.006840802729129791,
+            0.01706012338399887,
+            -0.0003562917991075665,
+            0.0016194552881643176,
+            0.006631390191614628,
+            0.003539068391546607,
+            0.00614741537719965,
+            0.003271485911682248,
+            0.0003123733913525939,
+            -0.000290705036604777,
+            0.02274683304131031,
+            0.011140735819935799,
+            -0.015524433925747871,
+            0.0071293264627456665,
+            -0.009335136041045189,
+            0.0005508707836270332,
+            0.009377018548548222,
+            -1.0870535334106535e-05,
+            0.014649556018412113,
+            -0.008176388218998909,
+            0.013653683476150036,
+            0.02043864317238331,
+            -0.007445771712809801,
+            -0.003076034365221858,
+            -0.008948886767029762,
+            0.006891992408782244,
+            0.016082866117358208,
+            0.025148093700408936,
+            -0.017748858779668808,
+            -0.0002715089067351073,
+            -0.009032651782035828,
+            -0.009437515400350094,
+            -0.026842007413506508,
+            0.017050817608833313,
+            0.020289728417992592,
+            0.0073573533445596695,
+            0.017795395106077194,
+            -0.00849748682230711,
+            0.011326880194246769,
+            -0.010191399604082108,
+            -0.0023244768381118774,
+            -0.00374150020070374,
+            -0.012648504227399826,
+            0.002643248997628689,
+            0.023826468735933304,
+            -0.013914286158978939,
+            0.00790182501077652,
+            -0.0013646703446283937,
+            0.03681934252381325,
+            -0.011233807541429996,
+            0.005612250417470932,
+            0.02019665576517582,
+            -0.004979359917342663,
+            -0.011773626320064068,
+            0.02295159175992012,
+            0.008841853588819504,
+            0.0035530291497707367,
+            0.005635518580675125,
+            0.024031227454543114,
+            0.011224500834941864,
+            0.0016043310752138495,
+            0.000546508003026247,
+            -0.04716896265745163,
+            0.00888838991522789,
+            -0.001733468729071319,
+            -0.010638146661221981,
+            0.017525484785437584,
+            0.010619531385600567,
+            0.009395632892847061,
+            -0.003269159235060215,
+            0.019191475585103035,
+            -0.012266908772289753,
+            0.012508896179497242,
+            0.033636271953582764,
+            0.0034157477784901857,
+            -0.019014639779925346,
+            0.0021848685573786497,
+            0.018009459599852562,
+            -0.0024222026113420725,
+            0.00217090779915452,
+            0.006705848034471273,
+            -0.017190424725413322,
+            0.0004281318688299507,
+            0.0036903105210512877,
+            0.029224654659628868,
+            -0.015180067159235477,
+            -0.027828572317957878,
+            0.0004447103419806808,
+            -0.005872852168977261,
+            -0.003073707688599825,
+            0.008841853588819504,
+            0.0032924271654337645,
+            -0.0016217821976169944,
+            -0.012722962535917759,
+            0.0017323052743449807,
+            0.007743602618575096,
+            0.010787061415612698,
+            0.003583277575671673,
+            -0.005965924356132746,
+            -0.008916311897337437,
+            0.021462436765432358,
+            -0.006045036017894745,
+            -0.0038834353908896446,
+            -0.0035925847478210926,
+            -0.006980410777032375,
+            -0.010172785259783268,
+            -0.012778805568814278,
+            0.011122121475636959,
+            -0.001270434819161892,
+            -0.01437033899128437,
+            -0.017004281282424927,
+            0.010461308993399143,
+            -0.0067756520584225655,
+            0.025725141167640686,
+            0.0001714563404675573,
+            -0.008032126352190971,
+            0.003369211684912443,
+            -0.015319675207138062,
+            -0.010535767301917076,
+            0.007752909790724516,
+            -0.002316333120688796,
+            -0.00495609175413847,
+            0.01091736275702715,
+            -0.0032575251534581184,
+            0.018958795815706253,
+            -0.0012913760729134083,
+            -0.008385800756514072,
+            0.002873602556064725,
+            0.0025571573060005903,
+            -0.012806727550923824,
+            -0.0006579037290066481,
+            -0.02131352201104164,
+            -0.013476846739649773,
+            -0.0014519254909828305,
+            0.014091122895479202,
+            -0.010777754709124565,
+            0.005868198815733194,
+            0.0010674211662262678,
+            0.006896646227687597,
+            -0.016371389850974083,
+            -0.010237935930490494,
+            -0.0012890491634607315,
+            0.024198757484555244,
+            0.011001127772033215,
+            -0.02319357916712761,
+            -0.010824290104210377,
+            -0.01007971353828907,
+            0.018055995926260948,
+            -0.013635069131851196,
+            0.004490730818361044,
+            0.013234859332442284,
+            -0.016194554045796394,
+            -0.0009208325645886362,
+            -0.012601968832314014,
+            -0.02220701426267624,
+            0.0003839225973933935,
+            0.032705552875995636,
+            0.012015613727271557,
+            0.017953617498278618,
+            -0.008078662678599358,
+            -0.00011510407057357952,
+            0.03168175742030144,
+            0.0032505446579307318,
+            0.0018253774615004659,
+            -0.01437033899128437,
+            0.009381672367453575,
+            -0.009493358433246613,
+            0.018875030800700188,
+            0.006612775847315788,
+            -0.00827411375939846,
+            0.003231930313631892,
+            -0.02233731560409069,
+            -0.003841552883386612,
+            -0.005244615487754345,
+            -0.009698117151856422,
+            0.012825341895222664,
+            -0.007040907628834248,
+            -0.015543048270046711,
+            0.0005316746537573636,
+            -0.007124672643840313,
+            -0.015980487689375877,
+            -0.008758088573813438,
+            -0.011373416520655155,
+            -0.030062302947044373,
+            -0.027567969635128975,
+            0.0010226302547380328,
+            -0.005868198815733194,
+            -0.028461461886763573,
+            -0.019359005615115166,
+            -0.022393157705664635,
+            0.009097802452743053,
+            -0.012369288131594658,
+            -0.02101569063961506,
+            0.007157247979193926,
+            0.005095699802041054,
+            -0.01842828467488289,
+            -0.026469718664884567,
+            -0.014435489661991596,
+            -0.016334161162376404,
+            0.025278395041823387,
+            0.012713654898107052,
+            -0.0008283421047963202,
+            -0.005174811463803053,
+            0.012220372445881367,
+            0.0032784664072096348,
+            0.01127103716135025,
+            -0.03268693760037422,
+            -0.03129085525870323,
+            -0.011187272146344185,
+            -0.00430691335350275,
+            0.0007451588753610849,
+            -0.03300338238477707,
+            0.002910831244662404,
+            0.003069054102525115,
+            -0.03827126696705818,
+            -0.008706899359822273,
+            0.015040459111332893,
+            -0.0009749307064339519,
+            0.011261729523539543,
+            0.00635217409580946,
+            0.026618633419275284,
+            0.02332388050854206,
+            0.0014437816571444273,
+            4.0937200537882745e-05,
+            0.012211065739393234,
+            -0.014705399051308632,
+            0.0014961347915232182,
+            0.005775126628577709,
+            0.008585905656218529,
+            -0.0004086449043825269,
+            0.015031151473522186,
+            -0.001642723334953189,
+            -0.015403440222144127,
+            0.017311418429017067,
+            0.01541274692863226,
+            0.00934909749776125,
+            0.002934099407866597,
+            0.0009121070033870637,
+            0.0038019970525056124,
+            0.0037880362942814827,
+            0.006770998705178499,
+            -0.01227621641010046,
+            -0.006417324300855398,
+            0.016901900991797447,
+            -0.011829469352960587,
+            -0.013942207209765911,
+            0.019805751740932465,
+            -0.00024489607312716544,
+            -0.023826468735933304,
+            0.02758658491075039,
+            0.006328905932605267,
+            -0.004479097202420235,
+            -0.02537146769464016,
+            -0.0018323578406125307,
+            -0.024664118885993958,
+            0.015543048270046711,
+            0.005407491698861122,
+            0.0015449975617229939,
+            0.022597916424274445,
+            0.01108489278703928,
+            0.0038206116296350956,
+            -0.013206937350332737,
+            0.006231179926544428,
+            0.01106627844274044,
+            0.015896722674369812,
+            -0.03704271465539932,
+            -0.02455243282020092,
+            0.012285523116588593,
+            -0.01841897703707218,
+            -0.005244615487754345,
+            0.012788113206624985,
+            0.0009766757721081376,
+            -0.02019665576517582,
+            0.0031667796429246664,
+            0.003608872415497899,
+            0.004758313298225403,
+            0.012192451395094395,
+            0.016631992533802986,
+            0.0122948307543993,
+            -0.0018521357560530305,
+            -0.022970205172896385,
+            0.006007806863635778,
+            -0.008241538889706135,
+            0.0007137469947338104,
+            -0.023342493921518326,
+            -0.0018102532485499978,
+            0.010610224679112434,
+            0.00767845194786787,
+            0.025874055922031403,
+            0.030676579102873802,
+            0.027903029695153236,
+            0.013244166038930416,
+            -0.004942130763083696,
+            -0.014416875317692757,
+            -0.020959846675395966,
+            0.0019917439203709364,
+            0.002347744768485427,
+            -0.011568867601454258,
+            -0.001557794981636107,
+            0.017488256096839905,
+            0.004011409357190132,
+            0.021816110238432884,
+            -0.005793740972876549,
+            0.017460335046052933,
+            -3.364776057424024e-05,
+            -0.005212040152400732,
+            -0.0008463748381473124,
+            -0.003173760138452053,
+            0.013244166038930416,
+            0.006650005001574755,
+            0.026004357263445854,
+            0.00022031922708265483,
+            0.003708925098180771,
+            0.005481949541717768,
+            -0.0007905315142124891,
+            -0.0008289237739518285,
+            -0.007380621042102575,
+            -0.003539068391546607,
+            -0.009847032837569714,
+            0.034678682684898376,
+            0.025278395041823387,
+            -0.02250484563410282,
+            -0.015124224126338959,
+            -0.012713654898107052,
+            0.02639526128768921,
+            -0.010833597742021084,
+            0.006524357479065657,
+            0.010796369053423405,
+            0.008660363033413887,
+            0.01065676100552082,
+            0.000908035144675523,
+            0.005719283130019903,
+            0.0028898902237415314,
+            0.022672373801469803,
+            -0.012611275538802147,
+            0.008669670671224594,
+            0.003920664079487324,
+            0.008357878774404526,
+            -0.015077687799930573,
+            -0.001618291949853301,
+            0.005872852168977261,
+            -0.014919465407729149,
+            0.006170683074742556,
+            -0.004118442535400391,
+            0.006207912229001522,
+            0.011559560894966125,
+            0.0015775728970766068,
+            -0.01607356034219265,
+            -0.017935002222657204,
+            0.012927721254527569,
+            -0.009781882166862488,
+            0.00916295312345028,
+            -0.018028074875473976,
+            -0.011606096290051937,
+            0.017897773534059525,
+            -0.006119493395090103,
+            -0.011931848712265491,
+            0.0026292880065739155,
+            0.00411146180704236,
+            -5.733389480155893e-05,
+            0.004009082447737455,
+            0.007752909790724516,
+            0.0020592212677001953,
+            0.00535630201920867,
+            0.005947310011833906,
+            -0.002478045877069235,
+            0.002440816955640912,
+            -0.0019789463840425014,
+            0.035739704966545105,
+            -0.009847032837569714,
+            0.006617429666221142,
+            -0.0022372216917574406,
+            -0.0012099378509446979,
+            -0.04154740646481514,
+            0.027772728353738785,
+            -0.006263755261898041,
+            0.015319675207138062,
+            0.00801816489547491,
+            -0.003401787020266056,
+            -0.010768447071313858,
+            0.008353224955499172,
+            0.01050784531980753,
+            -0.02135075069963932,
+            0.015142838470637798,
+            -0.024198757484555244,
+            -0.004218494985252619,
+            -0.002433836692944169,
+            -0.01108489278703928,
+            0.015496511943638325,
+            -0.01459371205419302,
+            0.008092623203992844,
+            -0.01091736275702715,
+            -0.0073713138699531555,
+            0.010144864208996296,
+            -0.0072549739852547646,
+            0.011010434478521347,
+            -0.042924873530864716,
+            0.0067756520584225655,
+            -0.004821137059479952,
+            -0.0010517152259126306,
+            -0.0006689560250379145,
+            -0.02270960435271263,
+            -0.031067483127117157,
+            -0.000392648100387305,
+            0.004853712394833565,
+            -0.005384223535656929,
+            0.00748300040140748,
+            0.005407491698861122,
+            0.004355776589363813,
+            0.0032621787395328283,
+            -0.01954515092074871,
+            0.025390081107616425,
+            0.03665181249380112,
+            -0.011829469352960587,
+            0.013383775018155575,
+            0.007161901798099279,
+            0.008148466236889362,
+            -0.03622367978096008,
+            -0.02352863922715187,
+            -0.008450950495898724,
+            0.009181567467749119,
+            0.02591128461062908,
+            -0.004679202102124691,
+            0.01559889130294323,
+            0.009437515400350094,
+            0.0050910464487969875,
+            -0.02870344929397106,
+            -0.015403440222144127,
+            -0.008046086877584457,
+            -0.015654735267162323,
+            -0.0022442021872848272,
+            -0.0033017343375831842,
+            0.014696091413497925,
+            -0.00473271869122982,
+            -0.018260754644870758,
+            -0.011941156350076199,
+            -0.006864070892333984,
+            -0.0037368466146290302,
+            -0.010154170915484428,
+            0.023547252640128136,
+            -0.010833597742021084,
+            -0.015589584596455097,
+            0.02512947842478752,
+            -0.001938227447681129,
+            0.005793740972876549,
+            0.005235308315604925,
+            0.00750161474570632,
+            -0.003304061247035861,
+            -0.004590783733874559,
+            -0.007673798594623804,
+            0.017627865076065063,
+            0.004202207550406456,
+            0.02874067798256874,
+            -0.004532613325864077,
+            0.00770637346431613,
+            0.0016764620086178184,
+            0.004355776589363813,
+            -0.002505967626348138,
+            0.008599866181612015,
+            0.006003153510391712,
+            -0.003648428013548255,
+            0.0071991304866969585,
+            -0.01350476872175932,
+            0.002410568529739976,
+            0.024943334981799126,
+            0.005984539166092873,
+            0.010414772666990757,
+            -0.012629889883100986,
+            -0.012667118571698666,
+            -0.003418074455112219,
+            -0.016203859820961952,
+            0.015924643725156784,
+            -0.020662017166614532,
+            0.018670272082090378,
+            -0.017460335046052933,
+            -0.001818397082388401,
+            0.0043930052779614925,
+            0.015198681503534317,
+            0.021090148016810417,
+            -0.004346469417214394,
+            0.026469718664884567,
+            0.013728141784667969,
+            0.0035227807238698006,
+            0.004970052745193243,
+            -0.022653760388493538,
+            -0.005514524411410093,
+            -0.011094199493527412,
+            0.003913683816790581,
+            -0.008581251837313175,
+            -0.016734370961785316,
+            -0.0010197217343375087,
+            0.007315470837056637,
+            -0.005398184526711702,
+            0.003701944602653384,
+            0.012657811865210533,
+            0.0015484878094866872,
+            0.0028293931391090155,
+            0.014230730943381786,
+            0.01651099883019924,
+            0.007510922383517027,
+            -0.007622608914971352,
+            0.0040323506109416485,
+            0.006622083019465208,
+            -0.0046280124224722385,
+            0.0009504992631264031,
+            0.017395183444023132,
+            0.01438895333558321,
+            -0.004060272127389908,
+            0.0053656091913580894,
+            -0.0027805303689092398,
+            -0.02676755003631115,
+            -0.014267959631979465,
+            -0.0003990468103438616,
+            -0.0005968251498416066,
+            -0.003506493056192994,
+            -0.015161452814936638,
+            0.004483750555664301,
+            0.012155221775174141,
+            -0.0122948307543993,
+            0.0035181271377950907,
+            0.008399761281907558,
+            -0.009949412196874619,
+            0.0021976660937070847,
+            -0.0034785715397447348,
+            -0.014249345287680626,
+            -0.009791189804673195,
+            0.027623813599348068,
+            -0.01011694222688675,
+            0.013746756128966808,
+            0.013635069131851196,
+            0.00039875597576610744,
+            -0.006440592464059591,
+            -0.0013041734928265214,
+            0.019061176106333733,
+            0.01746964268386364,
+            -0.001688096090219915,
+            -0.00036327220732346177,
+            -0.007552804425358772,
+            -0.006161375902593136,
+            0.01350476872175932,
+            -0.01634346880018711,
+            -0.004311567172408104,
+            0.009577123448252678,
+            -0.00846025813370943,
+            -0.004593110177665949,
+            -0.024533817544579506,
+            0.0021522934548556805,
+            0.009135031141340733,
+            0.0035227807238698006,
+            -0.0068314955569803715,
+            -0.001905652228742838,
+            -0.002934099407866597,
+            0.0058030481450259686,
+            -0.00118550646584481,
+            -0.00687803141772747,
+            -0.016836751252412796,
+            -0.0024175490252673626,
+            0.0070920973084867,
+            -0.006184644065797329,
+            -0.001833521295338869,
+            0.0036251600831747055,
+            -0.006082264706492424,
+            -0.015673348680138588,
+            0.03244495019316673,
+            9.452639642404392e-05,
+            0.021332135424017906,
+            0.004132403060793877,
+            0.0061334543861448765,
+            0.0027246871031820774,
+            0.007222398649901152,
+            0.02561345510184765,
+            0.009614353068172932,
+            0.006747730541974306,
+            0.01621316745877266,
+            0.014975308440625668,
+            -0.014240038581192493,
+            0.0072549739852547646,
+            0.012853262946009636,
+            -0.010163478553295135,
+            0.00417195912450552,
+            0.009251371026039124,
+            -0.0007649366743862629,
+            -0.003988141659647226,
+            -0.0037228858564049006,
+            0.016799522563815117,
+            0.003071380779147148,
+            -0.0008132178918458521,
+            0.012183143757283688,
+            0.01166194025427103,
+            -0.018754037097096443,
+            -0.00033796823117882013,
+            -0.0015135857975110412,
+            0.002343091182410717,
+            -0.005882159806787968,
+            -0.026581404730677605,
+            -0.008320650085806847,
+            0.0033854993525892496,
+            -0.0030434592626988888,
+            0.03091856651008129,
+            -0.00950266607105732,
+            -0.03354320302605629,
+            -0.01563611999154091,
+            -0.001225062063895166,
+            -0.01692982390522957,
+            -0.007966975681483746,
+            0.01276949793100357,
+            0.0295224841684103,
+            0.012415824458003044,
+            -0.013802599161863327,
+            0.024664118885993958,
+            -0.0028131057042628527,
+            0.01750687137246132,
+            -0.00870224554091692,
+            -0.009083840996026993,
+            -0.004509345628321171,
+            0.022113941609859467,
+            -0.008744128048419952,
+            -0.009605045430362225,
+            0.013579226098954678,
+            0.016985666006803513,
+            0.00842768233269453,
+            0.02323080785572529,
+            0.013644376769661903,
+            -0.007022293284535408,
+            -0.014891543425619602,
+            0.003808977548032999,
+            -0.008595212362706661,
+            -0.020252499729394913,
+            0.007324778009206057,
+            -0.014900850132107735,
+            0.005416798871010542,
+            -0.0021348423324525356,
+            -0.023696167394518852,
+            0.005537792574614286,
+            0.0053656091913580894,
+            -0.012043535709381104,
+            -0.012546124868094921,
+            0.000147243044921197,
+            -0.009507319889962673,
+            0.014463411644101143,
+            0.0068314955569803715,
+            -0.00023253493418451399,
+            -0.009251371026039124,
+            0.017832623794674873,
+            0.011001127772033215,
+            0.009968026541173458,
+            -0.00792043935507536,
+            0.0033924796152859926,
+            0.008934926241636276,
+            0.007590033579617739,
+            -0.0014495986979454756,
+            -0.0056634400971233845,
+            0.037880364805459976,
+            0.0030620736069977283,
+            0.012527510523796082,
+            -0.0017451026942580938,
+            0.009865647181868553,
+            0.01086151972413063,
+            0.015291753225028515,
+            0.010256550274789333,
+            -0.003646101336926222,
+            -0.0038275918923318386,
+            0.013458232395350933,
+            -0.012024921365082264,
+            0.0055052172392606735,
+            0.010275164619088173,
+            -0.0046675680205225945,
+            -0.005384223535656929,
+            0.010647453367710114,
+            0.00014309842663351446,
+            0.01812114752829075,
+            -0.005900774151086807,
+            0.009288599714636803,
+            0.018335212022066116,
+            0.009577123448252678,
+            -0.019098404794931412,
+            0.013728141784667969,
+            -0.01556166261434555,
+            0.029950616881251335,
+            0.01917286217212677,
+            -0.01812114752829075,
+            -0.0018975083949044347,
+            -0.011019742116332054,
+            0.009763267822563648,
+            -0.04221752658486366,
+            -0.01190392766147852,
+            0.0023919541854411364,
+            -0.004090521018952131,
+            -0.018884338438510895,
+            -0.027028150856494904,
+            0.0044558290392160416,
+            -0.004430233966559172,
+            -0.009353750385344028,
+            -0.008278767578303814,
+            0.0006509232916869223,
+            0.015924643725156784,
+            -0.005221347324550152,
+            -0.0077715241350233555,
+            -0.024310443550348282,
+            0.014993922784924507,
+            0.00886977557092905,
+            -0.004176612477749586,
+            -0.024403516203165054,
+            0.01455648336559534,
+            0.01603632979094982,
+            -0.00687803141772747,
+            0.01190392766147852,
+            -0.024049842730164528,
+            0.00197545625269413,
+            -0.005700668785721064,
+            -0.01654822751879692,
+            -0.015263832174241543,
+            0.0010999963851645589,
+            -0.012164529412984848,
+            -0.003313368419185281,
+            -0.014119044877588749,
+            -0.009083840996026993,
+            -0.0028945438098162413,
+            0.020475871860980988,
+            0.0025641375686973333,
+            -0.01541274692863226,
+            0.016390005126595497,
+            0.007869250141084194,
+            -0.0229888204485178,
+            -0.01631554774940014,
+            0.0046605877578258514,
+            -0.029876159504055977,
+            0.010805675759911537,
+            0.007869250141084194,
+            0.006826841738075018,
+            -0.02189056947827339,
+            0.007683105766773224,
+            0.02241177298128605,
+            0.007994897663593292,
+            0.0017660439480096102,
+            0.01913563348352909,
+            0.008185694925487041,
+            0.012667118571698666,
+            0.0019696392118930817,
+            -3.335691189931822e-06,
+            -0.005849584471434355,
+            -0.004148690961301327,
+            -0.004714104346930981,
+            0.03802927955985069,
+            -0.01617593877017498,
+            0.009498012252151966,
+            -0.022560687735676765,
+            0.0002395153569523245,
+            -0.01921009086072445,
+            -0.011298958212137222,
+            -0.015980487689375877,
+            0.0008010021410882473,
+            0.02151828072965145,
+            -0.007892518304288387,
+            -0.0002912867348641157,
+            -0.01583157107234001,
+            -0.0058356234803795815,
+            -0.006542971823364496,
+            0.02036418579518795,
+            -0.012741576880216599,
+            0.0018626062665134668,
+            -0.017311418429017067,
+            0.0010249570477753878,
+            0.026693090796470642,
+            0.005556406918913126,
+            -0.013225551694631577,
+            -0.026618633419275284,
+            -0.016194554045796394,
+            -0.009158299304544926,
+            -0.01316040102392435,
+            -0.005239961668848991,
+            -0.019228706136345863,
+            -0.011364108882844448,
+            -0.006473167799413204,
+            0.00654762564226985,
+            0.004039331339299679,
+            -0.015012537129223347,
+            -0.004997974261641502,
+            -0.024664118885993958,
+            -0.0202338844537735,
+            -0.010014562867581844,
+            -0.009470091201364994,
+            0.0013844481436535716,
+            0.02853591926395893,
+            0.0028549879789352417,
+            0.030397363007068634,
+            -0.00415567122399807,
+            0.008790664374828339,
+            0.009828418493270874,
+            0.007203784305602312,
+            -0.01520798821002245,
+            0.007487654220312834,
+            0.014584405347704887,
+            -0.008530061691999435,
+            0.0057890876196324825,
+            -0.0122948307543993,
+            -0.0007463222718797624,
+            0.005249268840998411,
+            0.004970052745193243,
+            0.011261729523539543,
+            -0.007548151072114706,
+            -0.01580365002155304,
+            0.013495461083948612,
+            -0.02040141448378563,
+            0.0011971404310315847,
+            0.00031295508961193264,
+            -0.003236583899706602,
+            0.003632140578702092,
+            0.018102532252669334,
+            -0.01336515974253416,
+            0.001435637823306024,
+            0.01999189704656601,
+            -0.012536818161606789,
+            0.01316040102392435,
+            0.003013210603967309,
+            0.0010214668000116944,
+            0.009511973708868027,
+            0.0011215193662792444,
+            0.005156196653842926,
+            0.0005630864761769772,
+            -0.019228706136345863,
+            -0.006966450251638889,
+            0.00831134244799614,
+            -0.024608274921774864,
+            -0.01128965150564909,
+            0.011177964508533478,
+            0.006254448089748621,
+            -0.017832623794674873,
+            -0.012788113206624985,
+            -0.023863697424530983,
+            0.01166194025427103,
+            -0.005467988550662994,
+            0.023993998765945435,
+            -0.015105608850717545,
+            0.009204835630953312,
+            -0.007738948799669743,
+            0.021164605394005775,
+            -0.0122948307543993,
+            0.005212040152400732,
+            0.0013576899655163288,
+            -0.01665060594677925,
+            0.009567816741764545,
+            -0.012285523116588593,
+            -0.0017567367758601904,
+            0.042775958776474,
+            0.006952489260584116,
+            0.0074271573685109615,
+            0.015068380162119865,
+            -0.022020868957042694,
+            0.01852135732769966,
+            0.002450124127790332,
+            0.018502742052078247,
+            0.010572995990514755,
+            -0.019265934824943542,
+            0.01695774495601654,
+            0.026823392137885094,
+            0.01982436701655388,
+            0.006221872754395008,
+            -0.01147579587996006,
+            0.027903029695153236,
+            0.012862570583820343,
+            0.0006660475628450513,
+            0.0022569994907826185,
+            0.0016846058424562216,
+            -0.011038356460630894,
+            -0.010842905379831791,
+            0.009018691256642342,
+            -0.017730243504047394,
+            -0.013272088021039963,
+            -0.0030923220328986645,
+            0.003080687951296568,
+            -0.010219321586191654,
+            -0.0036996176932007074,
+            0.006719809025526047,
+            0.007399235386401415,
+            0.011419951915740967,
+            0.0016020042821764946,
+            0.01611078903079033,
+            -0.004774601198732853,
+            -0.01893087476491928,
+            0.0002504222502466291,
+            -0.0032598518300801516,
+            0.014947386458516121,
+            -0.029299112036824226,
+            -0.007911132648587227,
+            -0.0004516907501965761,
+            0.005267883650958538,
+            0.006035728845745325,
+            0.007832021452486515,
+            -0.015096302144229412,
+            -0.005682054441422224,
+            -0.003113263286650181,
+            0.016120094805955887,
+            0.0016811155946925282,
+            0.023212192580103874,
+            0.015608198940753937,
+            -0.009968026541173458,
+            -0.006328905932605267,
+            0.0032668323256075382,
+            -0.010982513427734375,
+            0.00017320144979748875,
+            0.0025943859945982695,
+            0.011122121475636959,
+            -0.018847109749913216,
+            -0.008208963088691235,
+            0.025594839826226234,
+            0.02015942707657814,
+            0.00357629731297493,
+            0.026748934760689735,
+            -0.014174887910485268,
+            0.008227577432990074,
+            0.019842982292175293,
+            0.014407568611204624,
+            -0.010572995990514755,
+            -0.010731218382716179,
+            -0.0046675680205225945,
+            0.001287885825149715,
+            -0.0027549355290830135,
+            -0.005179464817047119,
+            0.006161375902593136,
+            -0.022467615082859993,
+            -0.016259703785181046,
+            0.0012157548917457461,
+            -0.013793292455375195,
+            0.0032342569902539253,
+            0.009358404204249382,
+            0.018502742052078247,
+            -0.011736397631466389,
+            0.01787916012108326,
+            -0.026879236102104187,
+            -0.003480898216366768,
+            0.010256550274789333,
+            0.0019545149989426136,
+            -0.016808830201625824,
+            -0.003911356907337904,
+            -0.002536216052249074,
+            -0.009218796156346798,
+            -0.006989717949181795,
+            0.01671575754880905,
+            0.012397210113704205,
+            0.015059073455631733,
+            -0.008195002563297749,
+            0.004423253703862429,
+            0.0023547252640128136,
+            0.009395632892847061,
+            -0.0018847109749913216,
+            -0.009442169219255447,
+            0.006696540862321854,
+            0.0017509197350591421,
+            -0.005863544996827841,
+            -0.005281844176352024,
+            0.023063277825713158,
+            -0.026134658604860306,
+            0.003641447750851512,
+            -0.016501691192388535,
+            0.011410645209252834,
+            0.012257601134479046,
+            0.002092959824949503,
+            -0.005598289426416159,
+            -0.006994371768087149,
+            0.013458232395350933,
+            -0.06496435403823853,
+            -0.008432336151599884,
+            -0.0015938604483380914,
+            0.027623813599348068,
+            -0.01794430986046791,
+            0.003169106552377343,
+            0.008404415100812912,
+            0.015422054566442966,
+            -0.007189823314547539,
+            0.007101404946297407,
+            0.016324853524565697,
+            -0.02261653169989586,
+            -0.011326880194246769,
+            0.00187075010035187,
+            -0.004974706098437309,
+            -0.011838776990771294,
+            0.0010470616398379207,
+            0.004500038456171751,
+            0.018940182402729988,
+            0.0015496512642130256,
+            0.011401337571442127,
+            -0.023510023951530457,
+            -0.013393081724643707,
+            -0.00436042994260788,
+            0.009060573764145374,
+            -0.005910081323236227,
+            0.0040323506109416485,
+            0.00931652169674635,
+            -0.03136531263589859,
+            0.011224500834941864,
+            0.008450950495898724,
+            -0.007241012994199991,
+            0.009605045430362225,
+            0.011326880194246769,
+            0.008474218659102917,
+            0.005267883650958538,
+            0.0011447874130681157,
+            -0.03210989013314247,
+            0.0043162209913134575,
+            0.013337238691747189,
+            -0.009200181812047958,
+            -0.013206937350332737,
+            -0.009977334178984165,
+            -0.0012366961454972625,
+            0.019973281770944595,
+            -0.004530286882072687,
+            -0.0018742403481155634,
+            0.014640248380601406,
+            0.006021767854690552,
+            0.01455648336559534,
+            -0.013337238691747189,
+            0.004942130763083696,
+            -0.010666067712008953,
+            -0.00854867696762085,
+            -0.0014100429834797978,
+            -0.0001836720621213317,
+            -0.0028968704864382744,
+            -0.038159579038619995,
+            0.00965158175677061,
+            -0.0036693692672997713,
+            -0.0197871383279562,
+            -0.01248097512871027,
+            -0.013411696068942547,
+            -0.004532613325864077,
+            -0.010870826430618763,
+            0.006854763720184565,
+            0.0031877208966761827,
+            0.009135031141340733,
+            -0.01336515974253416,
+            0.02496195025742054,
+            -0.014044586569070816,
+            -0.009977334178984165,
+            -0.009107109159231186,
+            -0.0005622139433398843,
+            0.002806125208735466,
+            0.0149846151471138,
+            -0.0038624941371381283,
+            -0.003204008564352989,
+            0.021853338927030563,
+            0.008013512007892132,
+            0.006114840041846037,
+            -0.01188531331717968,
+            -0.018875030800700188,
+            -0.023137735202908516,
+            -0.0029015240725129843,
+            -0.021946411579847336,
+            0.0007562111713923514,
+            0.004853712394833565,
+            0.02377062663435936,
+            0.007720334455370903,
+            -9.13997573661618e-05,
+            0.00436042994260788,
+            0.0038206116296350956,
+            0.019452078267931938,
+            -0.008371839299798012,
+            -0.004046311601996422,
+            -0.0071432869881391525,
+            -0.002633941825479269,
+            0.03482759743928909,
+            -0.008423029445111752,
+            0.00037955984589643776,
+            0.0013239512918516994,
+            -0.008879082277417183,
+            -0.007590033579617739,
+            0.010703296400606632,
+            -0.02881513722240925,
+            -0.014919465407729149,
+            -0.00036181797622703016,
+            0.0024873530492186546,
+            -0.010749832727015018,
+            0.02073647454380989,
+            -0.008502140641212463,
+            0.0023023721296340227,
+            0.009256024844944477,
+            0.024161528795957565,
+            0.022430386394262314,
+            0.023137735202908516,
+            0.007315470837056637,
+            -0.003671696176752448,
+            -0.002410568529739976,
+            -0.0067616915330290794,
+            -0.02496195025742054,
+            0.0161387100815773,
+            -0.0040951743721961975,
+            0.03404578939080238,
+            -0.03428777679800987,
+            0.0012413497315719724,
+            -0.007380621042102575,
+            -0.023472795262932777,
+            -0.009986640885472298,
+            0.023174963891506195,
+            -0.01045200228691101,
+            0.023900926113128662,
+            -0.0003123733913525939,
+            0.01556166261434555,
+            0.007906478829681873,
+            0.008441643789410591,
+            -0.01930316351354122,
+            0.011810855008661747,
+            -0.0023372743744403124,
+            0.01822352595627308,
+            0.009307214990258217,
+            -0.004607071168720722,
+            0.008832546882331371,
+            0.011689861305058002,
+            -0.01518937386572361,
+            -0.011196578852832317,
+            -0.0029015240725129843,
+            0.0018474820535629988,
+            -0.009847032837569714,
+            0.008553329855203629,
+            -0.002062711399048567,
+            -0.01600840874016285,
+            0.02537146769464016,
+            -0.01593395136296749,
+            0.010368237271904945,
+            0.004904902074486017,
+            0.00647782115265727,
+            0.01736726239323616,
+            -0.001187833258882165,
+            -0.027400439605116844,
+            0.0002552212681621313,
+            -0.03222157806158066,
+            -0.012341366149485111,
+            0.001407716190442443,
+            0.006566239986568689,
+            -0.0030434592626988888,
+            -0.010666067712008953,
+            -0.01883780211210251,
+            -0.0060310750268399715,
+            0.024571046233177185,
+            0.03439946472644806,
+            0.007743602618575096,
+            0.011745704337954521,
+            -0.011122121475636959,
+            -0.00967019610106945,
+            -0.006207912229001522,
+            -0.01917286217212677,
+            -0.009004729799926281,
+            -0.005863544996827841,
+            -0.00813915953040123,
+            0.011531638912856579,
+            -0.011568867601454258,
+            0.016538919880986214,
+            0.030118146911263466,
+            -0.031830672174692154,
+            -0.008530061691999435,
+            0.01570127159357071,
+            0.008651056326925755,
+            0.013253473676741123,
+            0.008804624900221825,
+            0.007631916087120771,
+            -0.011233807541429996,
+            -0.005817009136080742,
+            0.000602642132434994,
+            0.0062032584100961685,
+            -0.013802599161863327,
+            0.02162996679544449,
+            0.0010011072736233473,
+            0.002340764505788684,
+            0.004704797174781561,
+            0.01128965150564909,
+            -0.012760191224515438,
+            0.00357629731297493,
+            -0.007613301742821932,
+            -0.02537146769464016,
+            -0.026674477383494377,
+            -0.008134505711495876,
+            -0.006724462378770113,
+            -0.020680630579590797,
+            0.002073182025924325,
+            0.006333559285849333,
+            0.01950792223215103,
+            0.014724013395607471,
+            -0.009246718138456345,
+            -0.007250320166349411,
+            -0.024868877604603767,
+            0.010777754709124565,
+            -0.026376646012067795,
+            -0.02295159175992012,
+            0.027735499665141106,
+            0.013541997410356998,
+            0.014333110302686691,
+            -0.0016043310752138495,
+            0.0024315097834914923,
+            -0.010526459664106369,
+            0.013430310413241386,
+            0.02213255688548088,
+            0.004597763996571302,
+            0.007403889205306768,
+            -0.015105608850717545,
+            -0.002538542728871107,
+            0.020289728417992592,
+            -0.00022119177447166294,
+            -0.005002627614885569,
+            0.022486230358481407,
+            0.015422054566442966,
+            0.01886572316288948,
+            -0.004904902074486017,
+            0.015943259000778198,
+            0.01648307777941227,
+            0.008920964784920216,
+            -0.004106808453798294,
+            0.00020301362383179367,
+            -0.020997075363993645,
+            -0.004835098050534725,
+            0.03337567299604416,
+            0.012341366149485111,
+            0.025762369856238365,
+            -0.0263208020478487,
+            0.004825790878385305,
+            -0.004141710698604584,
+            0.013020792976021767,
+            0.01047061663120985,
+            0.019638223573565483,
+            0.028442848473787308,
+            0.009395632892847061,
+            0.009902875870466232,
+            -0.0012238987255841494,
+            0.029224654659628868,
+            -0.012611275538802147,
+            -0.009902875870466232,
+            -0.0006788449827581644,
+            -0.006622083019465208,
+            -0.004211514722555876,
+            0.00010339733853470534,
+            0.01476124208420515,
+            0.002815432380884886,
+            -0.019061176106333733,
+            0.014146965928375721,
+            0.007054868619889021,
+            -0.018819188699126244,
+            0.012127300724387169,
+            -0.02151828072965145,
+            0.011373416520655155,
+            -0.02738182619214058,
+            -0.0053656091913580894,
+            -0.01377467717975378,
+            -0.009358404204249382,
+            -0.023174963891506195,
+            -0.006933874916285276,
+            0.004034677520394325,
+            0.0082927281036973,
+            -0.0029317724984139204,
+            0.0043301815167069435,
+            -0.0030550931114703417,
+            -0.01600840874016285,
+            0.007929746992886066,
+            0.030937181785702705,
+            0.008716206066310406,
+            0.007450425066053867,
+            -0.0035530291497707367,
+            0.011224500834941864,
+            -0.008972154930233955,
+            -0.02213255688548088,
+            0.0002355888718739152,
+            -0.013467539101839066,
+            -0.008562637493014336,
+            0.013532689772546291,
+            0.006515050306916237,
+            -0.014202809892594814,
+            0.00905591994524002,
+            -0.024440744891762733,
+            -0.0002466411970090121,
+            -0.017786087468266487,
+            -0.00875343568623066,
+            -0.0053656091913580894,
+            -0.015021844767034054,
+            0.0013483826769515872,
+            0.007617955096065998,
+            -0.00890700425952673,
+            -0.006277716252952814,
+            0.00968881044536829,
+            -0.0037926898803561926,
+            -0.014333110302686691,
+            -0.0038880889769643545,
+            -0.008916311897337437,
+            -0.002316333120688796,
+            0.0018905280157923698,
+            -0.00985634047538042,
+            -0.005095699802041054,
+            -0.005407491698861122,
+            -0.026506947353482246,
+            0.006812881212681532,
+            0.019600993022322655,
+            0.004602417815476656,
+            0.020829545333981514,
+            0.0029573673382401466,
+            0.010526459664106369,
+            0.02931772544980049,
+            0.0055238320492208,
+            -0.008748781867325306,
+            -0.024887491017580032,
+            0.02159273810684681,
+            -0.010312393307685852,
+            0.024887491017580032,
+            -0.011447873897850513,
+            -0.014472718350589275,
+            0.0026548828464001417,
+            -0.00554244639351964,
+            -0.005267883650958538,
+            0.00035076565109193325,
+            0.006026421207934618,
+            -0.017497563734650612,
+            0.018009459599852562,
+            -0.011857391335070133,
+            0.008897697553038597,
+            0.010600917041301727,
+            0.041733551770448685,
+            -0.0013134806649759412,
+            -0.007375967688858509,
+            0.004462809301912785,
+            -0.001483337371610105,
+            0.01958237960934639,
+            -0.004497711546719074,
+            0.007282895501703024,
+            -0.009716732427477837,
+            -0.0007445771479979157,
+            -0.009265332482755184,
+            0.0062730624340474606,
+            0.015133530832827091,
+            -0.016771601513028145,
+            -0.01328139565885067,
+            0.0055052172392606735,
+            -0.026506947353482246,
+            0.0086138267070055,
+            0.030248448252677917,
+            0.017562713474035263,
+            -0.007324778009206057,
+            -0.014379646629095078,
+            0.007422503549605608,
+            -0.030080918222665787,
+            -0.0028480077162384987,
+            0.09649720042943954,
+            -0.009339789859950542,
+            -0.005128275137394667,
+            -0.02336110919713974,
+            -0.006524357479065657,
+            0.019228706136345863,
+            -0.014342417940497398,
+            0.07330362498760223,
+            -0.008748781867325306,
+            0.00046128881513141096,
+            -0.0073015098460018635,
+            0.001322787837125361,
+            -0.02053171582520008,
+            0.003978834021836519,
+            -0.02196502685546875,
+            -0.024868877604603767,
+            0.0021266986150294542,
+            -0.030118146911263466,
+            -0.007366660516709089,
+            -0.0034971858840435743,
+            0.008413721807301044,
+            -0.0049188630655407906,
+            0.011857391335070133,
+            -0.009865647181868553,
+            -0.012462359853088856,
+            -0.0011639834847301245,
+            0.005677401088178158,
+            0.015654735267162323,
+            -0.026264959946274757,
+            -0.01991743966937065,
+            0.010545074008405209,
+            0.01900533214211464,
+            0.004469790030270815,
+            -0.008609173819422722,
+            0.001268108026124537,
+            -0.00514223612844944,
+            0.008590559475123882,
+            0.029708629474043846,
+            0.01089874841272831,
+            0.01108489278703928,
+            0.0037531342823058367,
+            -0.011075585149228573,
+            0.015328982844948769,
+            0.007785485126078129,
+            0.013662991113960743,
+            -0.006021767854690552,
+            -0.004816483706235886,
+            -0.0021022669970989227,
+            0.009558509103953838,
+            -0.02220701426267624,
+            -0.014119044877588749,
+            -0.012238986790180206,
+            0.016967052593827248,
+            -0.006938528269529343,
+            -0.0005441812099888921,
+            -0.009791189804673195,
+            0.013579226098954678,
+            -0.008874429389834404,
+            -0.0051236217841506,
+            0.005184118635952473,
+            -0.021276293322443962,
+            0.022467615082859993,
+            0.006864070892333984,
+            0.021983640268445015,
+            -0.002318659797310829,
+            0.003573970403522253,
+            -0.014714706689119339,
+            -0.006580200511962175,
+            0.0106939896941185,
+            -0.004718757700175047,
+            0.00822292361408472,
+            0.002288411371409893,
+            0.02894543670117855,
+            0.0005770472926087677,
+            -0.013523383066058159,
+            0.008250845596194267,
+            -0.011587481945753098,
+            -0.01791638880968094,
+            0.010182092897593975,
+            -0.020382799208164215,
+            0.009060573764145374,
+            -6.11876675975509e-05,
+            -0.025445925071835518,
+            -0.010005255229771137,
+            0.004932823590934277,
+            -0.006924567744135857,
+            -0.013141786679625511,
+            0.013718834146857262,
+            -0.01853066496551037,
+            -0.01071260403841734,
+            -0.007194476667791605,
+            0.02101569063961506,
+            -0.016846058890223503,
+            0.023379722610116005,
+            -0.020680630579590797,
+            0.00634286692366004,
+            -0.00875343568623066,
+            0.02015942707657814,
+            -0.004453502129763365,
+            0.018195604905486107,
+            0.0122948307543993,
+            0.00674307718873024,
+            0.002838700544089079,
+            0.007469039876013994,
+            0.009698117151856422,
+            0.02581821382045746,
+            0.01982436701655388,
+            -0.003245891071856022,
+            -0.008534715510904789,
+            -0.011615403927862644,
+            0.013839827850461006,
+            -0.005793740972876549,
+            -0.00922810286283493,
+            0.011782933957874775,
+            0.016027024015784264,
+            -0.014947386458516121,
+            0.014854314737021923,
+            -0.0028666220605373383,
+            0.01086151972413063,
+            -0.013904978521168232,
+            0.004013736266642809,
+            0.0009057082934305072,
+            0.012369288131594658,
+            0.0075993407517671585,
+            -0.00016491221322212368,
+            0.01541274692863226,
+            0.02040141448378563,
+            -0.0004836842999793589,
+            0.01651099883019924,
+            0.0046210321597754955,
+            0.010377543978393078,
+            0.0021104109473526478,
+            -0.0037926898803561926,
+            -0.00514223612844944,
+            0.0021871954668313265,
+            -0.007892518304288387,
+            0.008786010555922985,
+            0.011056970804929733,
+            -0.010768447071313858,
+            0.002801471622660756,
+            0.012611275538802147,
+            -0.011913234367966652,
+            -0.008511447347700596,
+            0.0038275918923318386,
+            -0.026879236102104187,
+            -0.006850109901279211,
+            0.028089173138141632,
+            -0.018214218318462372,
+            -0.007273588329553604,
+            0.010033177211880684,
+            0.012425131164491177,
+            -0.0010011072736233473,
+            -0.014472718350589275,
+            0.009079188108444214,
+            0.005216693971306086,
+            -0.02557622641324997,
+            0.0024128954391926527,
+            0.011606096290051937,
+            0.018577199429273605,
+            -0.0004990993766114116,
+            -0.010768447071313858,
+            0.0013367487117648125,
+            0.025352852419018745,
+            0.016492383554577827,
+            0.013067329302430153,
+            -0.0050910464487969875,
+            -0.003990468103438616,
+            0.0051375823095440865,
+            0.0012064477195963264,
+            0.002601366490125656,
+            0.00668257987126708,
+            -0.0012983564520254731,
+            -0.0059054275043308735,
+            0.01348615437746048,
+            -0.0010458981851115823,
+            0.012322751805186272,
+            -0.013700219802558422,
+            -0.0186981949955225,
+            -0.0013134806649759412,
+            -0.003874127985909581,
+            -0.012387902475893497,
+            0.019470693543553352,
+            -0.003285446669906378,
+            0.03709855675697327,
+            0.013458232395350933,
+            0.005086392629891634,
+            -0.002601366490125656,
+            -0.008404415100812912,
+            0.016631992533802986,
+            0.0053656091913580894,
+            -0.01520798821002245,
+            0.003927644342184067,
+            -0.029168810695409775,
+            0.021071534603834152,
+            -0.009865647181868553,
+            0.010554381646215916,
+            -0.002633941825479269,
+            0.007906478829681873,
+            0.004876980558037758,
+            -0.011466488242149353,
+            -0.003373865270987153,
+            0.009567816741764545,
+            0.0022220974788069725,
+            -0.012127300724387169,
+            -0.0015612852293998003,
+            0.008678977377712727,
+            -0.006049689371138811,
+            0.012211065739393234,
+            0.005272537004202604,
+            -0.008520754985511303,
+            0.00436741067096591
+          ],
+          "title": "Conclusion"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": true,
+      "properties": {
+        "cosine_similarity": 0.8331248455462625
+      }
+    },
+    {
+      "id": "58f22dbd-acc0-401b-9f33-287c34178d41",
+      "type": "cosine_similarity",
+      "source": {
+        "id": "1e687808-8db8-4320-805d-0282a8d679bc",
+        "properties": {
+          "page_content": "Deepseekmoe: Towards Ultimate Expert Specialization In\n\nMixture-Of-Experts Language Models\n\nDamai Dai∗1,2, Chengqi Deng1, Chenggang Zhao∗1,3, R.X. Xu1, Huazuo Gao1, Deli Chen1, Jiashi Li1, Wangding Zeng1, Xingkai Yu∗1,4, Y. Wu1, Zhenda Xie1, Y.K. Li1, Panpan Huang1, Fuli Luo1, Chong Ruan1, Zhifang Sui2, Wenfeng Liang1 1DeepSeek-AI 2National Key Laboratory for Multimedia Information Processing, Peking University 3Institute for Interdisciplinary Information Sciences, Tsinghua University 4National Key Laboratory for Novel Software Technology, Nanjing University {daidamai, szf}@pku.edu.cn, {wenfeng.liang}@deepseek.com https://github.com/deepseek-ai/DeepSeek-MoE\n\nAbstract\n\nIn the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top- out of experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into ones and activating from them, allowing for a more flexible combination of activated experts; (2) isolating experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5× expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations.\n\nFurther, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\n1. Introduction\n\nRecent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023a). It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017) has emerged as a popular solution. It can\n\nenable parameter scaling, while concurrently keeping computational costs at a modest level.\n\nRecent applications of MoE architectures in Transformers (Vaswani et al., 2017) have yielded successful attempts at scaling language models to a substantial size (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022), accompanied with remarkable performance. These achievements underscore the considerable potential and promise of MoE language models.\n\nDespite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts.\n\nThis architecture manifests two potential issues: (1) Knowledge Hybridity: existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. (2) Knowledge Redundancy: tokens assigned to different experts may require common knowledge. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters.\n\nThese issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models.\n\nIn response to the aforementioned issues, we introduce DeepSeekMoE, an innovative MoE architecture specifically designed towards ultimate expert specialization. Our architecture involves two principal strategies: (1) Fine-Grained Expert Segmentation: while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. (2) Shared Expert Isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.\n\nThrough compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized.\n\nStarting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B (Lepikhin et al., 2021) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5× expert parameters and computation. Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE.\n\nThese studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization.\n\nLeveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B (DeepSeek-AI, 2024), a dense model trained on the same 2T corpus. We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B (Touvron et al., 2023b), which has approximately 2.5 times the activated parameters. Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard1. Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. The experimental results still validate its substantial advantages over the GShard architecture consistently. In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nOur contributions are summarized as follows: - Architectural Innovation. We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation.\n\nEmpirical Validation. We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. Experimental results validate the high 1https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models\n\nScalability. We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B.\n\nAlignment for MoE. We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B.\n\nPublic Release. In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization.\n\n2. Preliminaries: Mixture-Of-Experts For Transformers\n\nWe first introduce a generic MoE architecture commonly used in Transformer language models. A standard Transformer language model is constructed by stacking layers of standard Transformer blocks, where each block can be represented as follows:\n\n$$\\mathbf{u}{1:T}^{l}=\\text{Self-Att}\\left(\\mathbf{h}{1:T}^{l-1}\\right)+\\mathbf{h}{1:T}^{l-1},\\tag{1}$$ $$\\mathbf{h}{t}^{l}=\\text{FFN}\\left(\\mathbf{u}{t}^{l}\\right)+\\mathbf{u}{t}^{l},\\tag{2}$$\n\nwhere denotes the sequence length, Self-Att(·) denotes the self-attention module, FFN(·) denotes the Feed-Forward Network (FFN), u 1: ∈ R× are the hidden states of all tokens after the -th attention module, and h\n\n∈ Ris the output hidden state of the -th token after the -th Transformer block. For brevity, we omit the layer normalization in the above formulations.\n\nA typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022). An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. Then, each token will be assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts. If the -th FFN is substituted with an MoE layer, the computation for its output hidden state h\n\nis expressed as: where denotes the total number of experts, FFN(·) is the -th expert FFN, , denotes the gate value for the -th expert, , denotes the token-to-expert affinity, Topk(·, ) denotes the set comprising highest affinity scores among those calculated for the -th token and all experts, and e\n\nis the centroid of the -th expert in the -th layer. Note that ,is sparse, indicating that only out of gate values are nonzero. This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only experts. Also, in the above formulations, we omit the layer normalization operation for brevity.\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{N}\\left(g_{i,t}\\operatorname{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\operatorname{Topk}({s_{j,t}|1\\leqslant j\\leqslant N},K),\\ 0,&\\text{otherwise},\\end{cases}$$ $$s_{i,t}=\\operatorname{Softmax}{i}\\left(\\mathbf{u}{t}^{l}\\,\\mathbf{e}_{i}^{l}\\right),$$\n\n$$(3)$$\n\n(4) $\\binom{4}{5}$ .\n\n3. Deepseekmoe Architecture\n\nOn top of the generic MoE architecture outlined in Section 2, we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. As illustrated in Figure 2, our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. Both of these strategies are designed to elevate the level of expert specialization.\n\n3.1. Fine-Grained Expert Segmentation\n\nIn scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts.\n\nIn pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. The finer expert segmentation enables a more flexible and adaptable combination of activated experts. To be specific, on top of a typical MoE architecture shown in Figure 2(a), we segment each expert FFN into smaller experts by reducing the FFN intermediate hidden dimension to 1 times its original size. Since each expert becomes smaller, in response, we also increase the number of activated experts to times to keep the same computation cost, as illustrated in Figure 2(b). With the fine-grained expert segmentation, the output of an MoE layer can be expressed as:\n\n$$\\mathbf{h}{t}^{i}=\\sum{i=1}^{\\text{mN}}\\left(g_{i,t}\\text{FFN}{i}\\left(\\mathbf{u}{t}^{i}\\right)\\right)+\\mathbf{u}{t}^{i},\\tag{6}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\text{Tokr}(s_{i,t}|1\\leqslant j\\leqslant m\\text{N}),m\\text{K}),\\ 0,&\\text{otherwise},\\end{cases}$$ (7) $$s_{i,t}=\\text{Softmax}{i}\\left(\\mathbf{u}{t}^{T}\\mathbf{e}_{t}^{i}\\right),\\tag{8}$$ where the total number of expert parameters is equal to $N$ times the number of parameters in a\n\n$$(6)$$\n\n(7) (8) $\\frac{1}{2}$\n\nstandard FFN, and denotes the total number of fine-grained experts. With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to .\n\nFrom a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. As an illustrative example, we consider the case where = 16. A typical top-2 routing strategy can yield 16 2 = 120 possible combinations. By contrast, if each expert is split into 4 smaller experts, the fine-grained routing strategy can yield 64 8 = 4, 426, 165, 368 potential combinations. The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition.\n\n3.2. Shared Expert Isolation\n\nWith a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. This alleviation of redundancy will contribute to a more parameterefficient model with more specialized experts.\n\nTowards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate experts to serve as shared experts. Regardless of the router module, each token will be deterministically assigned to these shared experts. In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by , as depicted in Figure 2(c). With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows:\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{K_{s}}\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)+\\sum_{i=K_{s}+1}^{mN}\\left(g_{i,t}\\;\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},\\tag{9}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\mathrm{Topk}{s_{j,t}|K_{s}+1\\leqslant j\\leqslant mN},mK-K_{s}},\\ 0,&\\text{otherwise},\\end{cases}$$ (10) $$s_{i,t}=\\mathrm{Softmax}{i}\\left(\\mathbf{u}{t}^{l,T}e_{i}^{l}\\right).\\tag{11}$$ $\\mathbf{h}\\mathbf{M}\\mathbf{E}$ the number of closed-compact is $K_{s}$ the total number of routed currents.\n\nFinally, in DeepSeekMoE, the number of shared expert is , the total number of routed experts is − , and the number of nonzero gates is − .\n\nIt is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022). The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint.\n\n3.3. Load Balance Consideration\n\nAutomatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. Firstly, there is a risk of routing collapse (Shazeer et al., 2017), i.e., the model always selects only a few experts, preventing other experts from sufficient training.\n\nSecondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.\n\nExpert-Level Balance Loss. In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. The computation of the balance loss is as follows:\n\n$$\\begin{array}{l}{{{\\mathcal L}{\\mathrm{ExpBal}}=\\alpha{1}\\sum_{i=1}^{N^{\\prime}}f_{i}P_{i},}}\\ {{f_{i}=\\frac{N^{\\prime}}{K^{T}}\\sum_{t=1}^{T}\\mathds{1}\\,(\\mathrm{Token}\\;t\\;\\mathrm{s}\\mathrm{lects}\\;\\mathrm{Expert}\\;i),}}\\ {{P_{i}=\\frac{1}{T}\\sum_{t=1}^{T}s_{i,t},}}\\end{array}$$\n\n(12) $$\\begin{array}{l}\\small\\text{(13)}\\end{array}$$ = (14) $$\\begin{array}{l}\\small\\text{(14)}\\end{array}$$ .\n\n7 where 1 is a hyper-parameter called expert-level balance factor, ′is equal to ( − ) and ′ is equal to ( − ) for brevity. 1(·) denotes the indicator function.\n\nDevice-Level Balance Loss. In addition to the expert-level balance loss, we introduce a devicelevel balance loss. When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. Instead, our primary objective is to ensure balanced computation across the devices. If we partition all routed experts into groups {E1, E2, ..., E}, and deploy each group on a single device, the device-level balance loss is computed as follows:\n\n$$\\mathcal{L}{\\text{DevBal}}=\\alpha{2}\\sum_{i=1}^{D}f_{i}^{\\prime}P_{i}^{\\prime},$$ $$f_{i}^{\\prime}=\\frac{1}{|\\mathcal{E}{i}|}\\sum{j\\in\\mathcal{E}{i}}f{j},$$ $$P_{i}^{\\prime}=\\sum_{j\\in\\mathcal{E}{i}}P{j},$$\n\n(15) $$\\begin{array}{l}\\small\\text{(16)}\\end{array}$$ = (17) .\n\nwhere 2 is a hyper-parameter called device-level balance factor. In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices.\n\n4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n6. Alignment For Deepseekmoe 16B\n\nPrevious research indicates that MoE models typically do not emerge significant gains from fine-tuning (Artetxe et al., 2022; Fedus et al., 2021). However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B.\n\n6.1. Experimental Setup\n\nTraining Data. For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios.\n\nHyper-Parameters. During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).\n\nWe employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10−5 without incorporating any learning rate scheduling strategy.\n\nEvaluation Benchmarks. For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3, with the following adjustments: (1) We exclude Pile (Gao et al., 2020) since chat models are seldom employed for pure language modeling. (2) We exclude\n\nMetric # Shot LLaMA2 DeepSeek DeepSeekMoE SFT 7B Chat 7B Chat 16B # Total Params N/A 6.7B 6.9B 16.4B # Activated Params N/A 6.7B 6.9B 2.8B FLOPs per 4K Tokens N/A 187.9T 183.5T 74.4T HellaSwag (Acc.) 0-shot 67.9 71.0 72.2 PIQA (Acc.) 0-shot 76.9 78.4 79.7 ARC-easy (Acc.) 0-shot 69.7 70.2 69.9 ARC-challenge (Acc.) 0-shot 50.8 50.2 50.0 BBH (EM) 3-shot 39.3 43.1 42.2 RACE-middle (Acc.) 5-shot 63.9 66.1 64.8 RACE-high (Acc.) 5-shot 49.6 50.8 50.6 DROP (EM) 1-shot 40.0 41.7 33.8 GSM8K (EM) 0-shot 63.4 62.6 62.2 MATH (EM) 4-shot 13.5 14.7 15.2 HumanEval (Pass@1) 0-shot 35.4 45.1 45.7 MBPP (Pass@1) 3-shot 27.8 39.0 46.2 TriviaQA (EM) 5-shot 60.1 59.5 63.3 NaturalQuestions (EM) 0-shot 35.2 32.7 35.1 MMLU (Acc.) 0-shot 50.0 49.7 47.2 WinoGrande (Acc.) 0-shot 65.1 68.4 69.0 CLUEWSC (EM) 5-shot 48.4 66.2 68.2 CEval (Acc.) 0-shot 35.1 44.7 40.0 CMMLU (Acc.) 0-shot 36.9 51.2 49.3\n\nCHID (Zheng et al., 2019) due to the observed instability of results, hindering the derivation of solid conclusions. (3) We additionally include BBH (Suzgun et al., 2022) to provide a more comprehensive assessment of the reasoning ability of the chat models.\n\nTable 5 | Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations.\n\n6.2. Evaluations\n\nBaselines. In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. Correspondingly, we construct three chat models, including LLaMA2 SFT 7B3, DeepSeek Chat 7B, and DeepSeekMoE Chat 16B.\n\nSubsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks.\n\nResults. The evaluation results are presented in Table 5. Our key observations include: (1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intensive tasks (TriviaQA, NaturalQuestions). (2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. In addition, it also surpasses DeepSeek Chat 7B. (3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1). However, it is worth noting that, after supervised finetuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. (4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations.\n\n7. Deepseekmoe 145B Ongoing\n\nEncouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available.\n\n7.1. Experimental Setup\n\nTraining Data and Tokenization. For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study.\n\nModel Settings. For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps.\n\nSubsequently, the learning rate keeps constant during the remaining training process. The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. Also, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse.\n\nEvaluation Benchmarks. We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3).\n\n7.2. Evaluations\n\nBaselines. Apart from DeepSeekMoE 145B, we consider three additional models for comparison. DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters.\n\nResults. From the evaluation results presented in Table 6, we have the following observations: (1) Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. (2) On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. (3) At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5.\n\n8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\nAppendices A. Overview Of Hyper-Parameters\n\nWe present the overview of hyper-parameters for DeepSeekMoE across various sizes in Table 7.\n\n# Params # Layers Hidden # Attn # Shared # Routed Relative Sequence Batch Size Learning Size Heads Experts Experts Expert Size Length (Sequence) Rate 2.0B 9 1280 10 1 63 (7 activated) 0.25 2048 2048 1.08e-3 16.4B 28 2048 16 2 64 (6 activated) 0.25 4096 4608 4.2e-4 144.6B 62 4096 32 4 128 (12 activated) 0.125 4096 4608 3.0e-4\n\nTable 7 | Overview of hyper-parameters for DeepSeekMoE across various sizes. The relative expert size is in comparison to a standard FFN.\n\nB. Comparing Deepseekmoe With Larger Models\n\nComparisons among DeepSeekMoE, GShard×1.2, and GShard×1.5 are shown in Table 8. Comparisons among DeepSeekMoE, Dense×4, and Dense×16 are shown in Table 9.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 2.3B 2.8B 1.9B # Activated Expert Params N/A 0.28B 0.35B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.824 1.808 1.808 HellaSwag (Acc.) 0-shot 53.7 54.4 54.8 PIQA (Acc.) 0-shot 71.8 71.1 72.3 ARC-easy (Acc.) 0-shot 46.8 47.3 49.4 ARC-challenge (Acc.) 0-shot 31.7 34.1 34.3 RACE-middle (Acc.) 5-shot 43.7 46.4 44.0 RACE-high (Acc.) 5-shot 31.9 32.4 31.7 HumanEval (Pass@1) 0-shot 3.7 3.0 4.9 MBPP (Pass@1) 3-shot 2.4 2.6 2.2 TriviaQA (EM) 5-shot 15.2 15.7 16.6 NaturalQuestions (EM) 5-shot 4.5 4.7 5.7\n\nTable 8 | Comparison between DeepSeekMoE and larger GShard models.\n\nAt a larger scale of 13B total parameters, we also compare DeepSeekMoE with GShard×1.2 and GShard×1.5, and show results in Table 10. At a larger scale, DeepSeekMoE even outperforms GShard×1.5 distinctly.\n\nMetric # Shot Dense×4 Dense×16 DeepSeekMoE Relative Expert Size N/A 1 1 0.25 # Experts N/A 4 + 0 16 + 0 1 + 63 # Activated Experts N/A 4 + 0 16 + 0 1 + 7 # Total Expert Params N/A 0.47B 1.89B 1.89B # Activated Expert Params N/A 0.47B 1.89B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.908 1.806 1.808 HellaSwag (Acc.) 0-shot 47.6 55.1 54.8 PIQA (Acc.) 0-shot 70.0 71.9 72.3 ARC-easy (Acc.) 0-shot 43.9 51.9 49.4 ARC-challenge (Acc.) 0-shot 30.5 33.8 34.3 RACE-middle (Acc.) 5-shot 42.4 46.3 44.0 RACE-high (Acc.) 5-shot 30.7 33.0 31.7 HumanEval (Pass@1) 0-shot 1.8 4.3 4.9 MBPP (Pass@1) 3-shot 0.2 2.2 2.2 TriviaQA (EM) 5-shot 9.9 16.5 16.6 NaturalQuestions (EM) 5-shot 3.0 6.3 5.7\n\nTable 9 | Comparison between DeepSeekMoE and larger dense baselines.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 15.9B 19.8B 13.3B # Activated Expert Params N/A 2.37B 2.82B 2.05B # Training Tokens N/A 100B 100B 100B HellaSwag (Acc.) 0-shot 66.6 67.7 69.1 PIQA (Acc.) 0-shot 75.6 76.0 75.7 ARC-easy (Acc.) 0-shot 56.8 56.8 58.8 ARC-challenge (Acc.) 0-shot 39.9 37.6 38.5 RACE-middle (Acc.) 5-shot 51.6 50.6 52.4 RACE-high (Acc.) 5-shot 37.4 36.3 38.5 HumanEval (Pass@1) 0-shot 6.1 6.1 9.8 MBPP (Pass@1) 3-shot 7.0 11.6 10.6 TriviaQA (EM) 5-shot 36.5 36.7 38.2 NaturalQuestions (EM) 5-shot 12.6 12.1 13.7\n\nTable 10 | Comparison between DeepSeekMoE and larger GShard models at a larger scale.\n\nC. Training Benchmark Curves Of Deepseekmoe 16B\n\nWe present the benchmark curves during training of DeepSeekMoE 16B and DeepSeek 7B (Dense) in Figure 7 for reference.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/DeepSeekMoE_2.md"
+          },
+          "summary": "DeepSeekMoE is an innovative Mixture-of-Experts (MoE) architecture designed to enhance expert specialization in language models. It employs fine-grained expert segmentation and shared expert isolation to improve performance and reduce computational costs. Starting with a model of 2B parameters, DeepSeekMoE demonstrates comparable performance to larger models like GShard 2.9B and LLaMA2 7B, using significantly fewer computations. The architecture scales up to 16B and 145B parameters, maintaining advantages over traditional MoE architectures. DeepSeekMoE 16B achieves similar results to dense models with only 40% of the computations, and its chat model version shows adaptability in bilingual scenarios. The architecture's public release aims to contribute to advancements in large-scale language models.",
+          "headlines": {
+            "1. Introduction": [],
+            "2. Preliminaries: Mixture-Of-Experts For Transformers": [],
+            "3. Deepseekmoe Architecture": [
+              "3.1. Fine-Grained Expert Segmentation",
+              "3.2. Shared Expert Isolation",
+              "3.3. Load Balance Consideration"
+            ],
+            "4. Validation Experiments": [
+              "4.1. Experimental Setup",
+              "4.1.1. Training Data And Tokenization",
+              "4.1.2. Infrastructures",
+              "4.1.3. Hyper-Parameters",
+              "4.1.4. Evaluation Benchmarks",
+              "4.2. Evaluations",
+              "4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models",
+              "4.4. Ablation Studies",
+              "4.5. Analysis On Expert Specialization"
+            ],
+            "5. Scaling Up To Deepseekmoe 16B": [
+              "5.1. Experimental Setup",
+              "5.1.1. Training Data And Tokenization",
+              "5.1.2. Hyper-Parameters",
+              "5.1.3. Evaluation Benchmarks",
+              "5.2. Evaluations",
+              "5.2.1. Internal Comparison With Deepseek 7B",
+              "5.2.2. Comparison With Open Source Models"
+            ],
+            "6. Alignment For Deepseekmoe 16B": [
+              "6.1. Experimental Setup",
+              "6.2. Evaluations"
+            ],
+            "7. Deepseekmoe 145B Ongoing": [
+              "7.1. Experimental Setup",
+              "7.2. Evaluations"
+            ],
+            "8. Related Work": [],
+            "9. Conclusion": [],
+            "Appendices": [
+              "A. Overview Of Hyper-Parameters",
+              "B. Comparing Deepseekmoe With Larger Models",
+              "C. Training Benchmark Curves Of Deepseekmoe 16B"
+            ]
+          },
+          "summary_embedding": [
+            -0.012465757317841053,
+            -0.022808987647294998,
+            -0.018792515620589256,
+            -0.02308654971420765,
+            0.01797616109251976,
+            0.007200260180979967,
+            0.006490030791610479,
+            0.042613785713911057,
+            -0.014049487188458443,
+            0.03187054023146629,
+            -0.021486490964889526,
+            -0.023592690005898476,
+            0.009045224636793137,
+            -0.014441338367760181,
+            0.005865518003702164,
+            -0.007363531738519669,
+            -0.008743173442780972,
+            -0.005163452122360468,
+            0.018841497600078583,
+            -0.03100520372390747,
+            0.011110605672001839,
+            -0.046075135469436646,
+            -0.020017050206661224,
+            -0.021845687180757523,
+            0.016425084322690964,
+            0.037389107048511505,
+            0.022547753527760506,
+            -0.0028062239289283752,
+            -0.06844329088926315,
+            0.034678805619478226,
+            0.035168617963790894,
+            0.02052319049835205,
+            0.0005438972148112953,
+            -0.00031302153365686536,
+            -0.024833550676703453,
+            -0.028556134551763535,
+            0.03347059711813927,
+            -0.004379750229418278,
+            -0.01453930139541626,
+            0.015976088121533394,
+            0.0414055772125721,
+            0.012106560170650482,
+            -0.02754385396838188,
+            0.0014245413476601243,
+            0.015151568688452244,
+            0.0055961208418011665,
+            -0.01678428053855896,
+            -0.0022960014175623655,
+            -0.002424577483907342,
+            -0.021225256845355034,
+            -0.01400050614029169,
+            0.018825169652700424,
+            0.001296985661610961,
+            -0.008865626528859138,
+            -0.0542386956512928,
+            0.007330877240747213,
+            0.03840138763189316,
+            0.05623060464859009,
+            0.044148534536361694,
+            -0.020180322229862213,
+            0.03369917720556259,
+            -0.030025575309991837,
+            -0.02501314878463745,
+            0.011028970591723919,
+            0.003983817528933287,
+            -0.018531281501054764,
+            0.01784554310142994,
+            -0.005636938381940126,
+            -0.0006556359585374594,
+            0.01751900091767311,
+            -0.03941367194056511,
+            0.020751770585775375,
+            -0.01887415163218975,
+            0.031266435980796814,
+            0.02173139713704586,
+            0.0073798587545752525,
+            0.01872720755636692,
+            0.01678428053855896,
+            0.0036858476232737303,
+            -0.04107903689146042,
+            0.009118696674704552,
+            0.016523046419024467,
+            0.014196431264281273,
+            0.016269976273179054,
+            0.06553706526756287,
+            -0.05391215533018112,
+            -0.004485876299440861,
+            -0.05169166624546051,
+            0.0012561678886413574,
+            -0.004971608519554138,
+            -0.03670336678624153,
+            0.01048201136291027,
+            0.028425518423318863,
+            0.021372200921177864,
+            -0.01063711941242218,
+            0.024392718449234962,
+            -0.022743679583072662,
+            0.0032001156359910965,
+            -0.04062187671661377,
+            0.0019653772469609976,
+            0.004518530797213316,
+            0.016531210392713547,
+            0.0070778070949018,
+            0.004273623693734407,
+            -0.016702644526958466,
+            0.006045116577297449,
+            -0.01905374974012375,
+            0.03291547670960426,
+            -0.008465612307190895,
+            0.08653374016284943,
+            0.008751336485147476,
+            0.004738946910947561,
+            0.004281787667423487,
+            0.02398454025387764,
+            0.012131051160395145,
+            -0.00810233410447836,
+            -0.04718538001179695,
+            -0.015690363943576813,
+            0.011061624623835087,
+            -0.0383034273982048,
+            0.018335357308387756,
+            -0.008759500458836555,
+            0.01282495353370905,
+            -0.04107903689146042,
+            -0.06344719231128693,
+            0.01498013362288475,
+            0.001784758409485221,
+            -0.004065453074872494,
+            -0.0002903166168835014,
+            -0.018923133611679077,
+            0.003930754493921995,
+            -0.004200151655822992,
+            -0.010645282454788685,
+            -0.021502818912267685,
+            0.03159298002719879,
+            0.03657275065779686,
+            0.06527583301067352,
+            0.036801330745220184,
+            -0.0030164355412125587,
+            0.005775718949735165,
+            0.019690508022904396,
+            -0.020784424617886543,
+            0.003481758525595069,
+            -0.01214737817645073,
+            -0.021992631256580353,
+            -0.03820546343922615,
+            0.010971825569868088,
+            0.07379858940839767,
+            -0.029029620811343193,
+            0.013437220826745033,
+            -0.017616963014006615,
+            0.04205866530537605,
+            -0.004567512311041355,
+            0.026727497577667236,
+            -0.016669990494847298,
+            0.004059330560266972,
+            -0.025388672947883606,
+            0.01620466820895672,
+            -0.024964168667793274,
+            0.012571883387863636,
+            -0.000885746325366199,
+            -0.04222193360328674,
+            0.00011116473615402356,
+            0.026890767738223076,
+            -0.010971825569868088,
+            0.02261306345462799,
+            0.0004949158756062388,
+            -0.006302268709987402,
+            -0.05097327008843422,
+            0.038727931678295135,
+            -0.003912386484444141,
+            0.018645571544766426,
+            -0.0018959869630634785,
+            0.0163107942789793,
+            -0.0016408757073804736,
+            0.013216804713010788,
+            0.009543201886117458,
+            0.007114543113857508,
+            -0.033503253012895584,
+            0.02697240374982357,
+            -0.02504580281674862,
+            0.028049994260072708,
+            0.03435226157307625,
+            -0.0030184765346348286,
+            0.03255628049373627,
+            0.019249675795435905,
+            0.03840138763189316,
+            0.0327032245695591,
+            0.030335791409015656,
+            -0.00824927818030119,
+            0.013878053054213524,
+            0.008200296200811863,
+            0.03608293831348419,
+            0.013429056853055954,
+            0.019723162055015564,
+            -0.009730963967740536,
+            -0.014041324146091938,
+            0.048785436898469925,
+            0.007318632211536169,
+            -0.0179598331451416,
+            -0.005441013257950544,
+            -0.03510330989956856,
+            -0.002832755446434021,
+            -0.014425011351704597,
+            0.0323440283536911,
+            0.02158445306122303,
+            -0.01647406443953514,
+            0.035135965794324875,
+            0.024980494752526283,
+            -0.03751972317695618,
+            -0.005232842173427343,
+            -0.02026195637881756,
+            -0.023298801854252815,
+            -0.04215662553906441,
+            0.011624909937381744,
+            0.012833117507398129,
+            -0.013976015150547028,
+            -0.007469657808542252,
+            0.020457882434129715,
+            -0.023625344038009644,
+            -0.0018265966791659594,
+            -0.01356783788651228,
+            0.006547175347805023,
+            -0.020898714661598206,
+            -0.03667071461677551,
+            -0.01048201136291027,
+            0.025862159207463264,
+            0.012637191452085972,
+            0.04362606629729271,
+            -0.013518855907022953,
+            -0.005367541220039129,
+            0.033193036913871765,
+            0.012759645469486713,
+            0.010767736472189426,
+            0.03699725493788719,
+            -0.03027048148214817,
+            0.024621298536658287,
+            -0.020294612273573875,
+            -0.0032878739293664694,
+            0.01707816869020462,
+            -0.0030062312725931406,
+            0.01962519995868206,
+            -0.04535674303770065,
+            -0.024196794256567955,
+            0.04395261034369469,
+            -0.0007107399869710207,
+            0.016302630305290222,
+            -0.005959399044513702,
+            -0.012106560170650482,
+            0.0035899258218705654,
+            0.0027715289033949375,
+            -0.01438419334590435,
+            -0.008620720356702805,
+            0.007473739795386791,
+            -0.01962519995868206,
+            0.03660540655255318,
+            0.03794422745704651,
+            -0.004132802598178387,
+            -0.014131123200058937,
+            -0.03324201703071594,
+            0.010016689077019691,
+            -0.017878197133541107,
+            -0.023951886221766472,
+            0.04078514873981476,
+            -0.032115448266267776,
+            -0.00029312283731997013,
+            0.039478980004787445,
+            0.018808843567967415,
+            -0.014898497611284256,
+            0.00868602842092514,
+            -0.01709449477493763,
+            -0.026302991434931755,
+            -0.017159804701805115,
+            0.03356856107711792,
+            0.024800896644592285,
+            -0.010743245482444763,
+            0.010245268233120441,
+            0.005330805201083422,
+            0.02893165871500969,
+            0.024376392364501953,
+            0.00648594880476594,
+            -0.013812744058668613,
+            0.015029114671051502,
+            0.027641816064715385,
+            -0.0028021421749144793,
+            -7.66609373386018e-05,
+            -0.004010349046438932,
+            -0.007796200457960367,
+            -0.006453294772654772,
+            -0.013069860637187958,
+            -0.005089980084449053,
+            0.0009184005321003497,
+            -0.032295044511556625,
+            0.015967924147844315,
+            -0.005890008993446827,
+            0.010359558276832104,
+            -0.011535110883414745,
+            0.01288209855556488,
+            0.01722511276602745,
+            -0.013494365848600864,
+            -0.0716434046626091,
+            0.014612773433327675,
+            0.02458864450454712,
+            -0.006706364918500185,
+            -0.017943505197763443,
+            0.008481939323246479,
+            0.00016148542636074126,
+            0.038597315549850464,
+            -0.014180104248225689,
+            -0.031103165820240974,
+            0.02680913358926773,
+            -0.017437364906072617,
+            -0.015241367742419243,
+            -0.010661610402166843,
+            -0.00580837344750762,
+            0.04649963974952698,
+            0.01889047957956791,
+            0.011420820839703083,
+            -0.04042595252394676,
+            -0.007449248805642128,
+            0.015078096650540829,
+            0.014506647363305092,
+            -0.04705476388335228,
+            0.0052083516493439674,
+            0.015486273914575577,
+            0.019723162055015564,
+            0.024800896644592285,
+            0.04274440184235573,
+            -0.0021531390957534313,
+            0.01661284640431404,
+            -0.015053605660796165,
+            -0.05136512219905853,
+            -0.03340528905391693,
+            0.029943939298391342,
+            -0.017731253057718277,
+            -0.02906227484345436,
+            -0.01693122461438179,
+            -0.006151242647320032,
+            0.0017939424142241478,
+            0.007126788143068552,
+            -0.01916803978383541,
+            -0.03464614972472191,
+            -0.03970756009221077,
+            0.02277633361518383,
+            0.007983962073922157,
+            0.004269542172551155,
+            -0.042777057737112045,
+            0.0013459670590236783,
+            0.009184005670249462,
+            0.012073906138539314,
+            0.0009954441338777542,
+            -0.019984396174550056,
+            0.015494437888264656,
+            0.0032001156359910965,
+            -0.031070511788129807,
+            -0.035593122243881226,
+            -0.03595232218503952,
+            0.0023919232189655304,
+            0.003781769424676895,
+            0.06279410421848297,
+            0.012988224625587463,
+            -0.03840138763189316,
+            -0.04522612690925598,
+            0.029176564887166023,
+            -0.02052319049835205,
+            -0.016678154468536377,
+            0.0059430720284581184,
+            0.012898425571620464,
+            -0.009877908043563366,
+            0.013192313723266125,
+            0.023560035973787308,
+            0.041601505130529404,
+            -0.005987971555441618,
+            0.012914752587676048,
+            0.023037567734718323,
+            0.012955570593476295,
+            0.0362788625061512,
+            -0.00936360377818346,
+            0.0035409443080425262,
+            0.039936136454343796,
+            -0.012269831262528896,
+            0.03175625205039978,
+            -0.02189466916024685,
+            0.0020398697815835476,
+            0.02249877341091633,
+            0.027184655889868736,
+            -0.0006479826406575739,
+            0.02594379521906376,
+            -0.031543996185064316,
+            0.018368011340498924,
+            -0.014498483389616013,
+            -0.04888340085744858,
+            -0.01557607389986515,
+            0.013102514669299126,
+            0.030384771525859833,
+            0.02756018005311489,
+            -0.0041348435916006565,
+            -0.019723162055015564,
+            -0.01155143789947033,
+            -0.010596301406621933,
+            0.0018419033149257302,
+            0.03285016864538193,
+            0.010506502352654934,
+            0.010890189558267593,
+            -0.011935125105082989,
+            -0.004104230087250471,
+            0.03025415539741516,
+            -0.021192602813243866,
+            0.01650671847164631,
+            0.0041511706076562405,
+            -0.002029665280133486,
+            -0.04320156201720238,
+            -0.035266581922769547,
+            0.007845181971788406,
+            0.011747363954782486,
+            0.014874007552862167,
+            0.0368339866399765,
+            0.025094784796237946,
+            -0.01662100851535797,
+            0.030629679560661316,
+            -0.01905374974012375,
+            0.027984686195850372,
+            0.039054471999406815,
+            0.011624909937381744,
+            -0.018351683393120766,
+            0.012237177230417728,
+            0.002424577483907342,
+            -0.025127438828349113,
+            0.01680060662329197,
+            -0.032735876739025116,
+            0.01738838478922844,
+            -0.02354370802640915,
+            0.025143766775727272,
+            -0.02579685114324093,
+            -0.009290131740272045,
+            -0.004428731743246317,
+            -0.006130834110081196,
+            0.04888340085744858,
+            -0.026727497577667236,
+            -0.0142454132437706,
+            -0.0024694770108908415,
+            -0.0024531499948352575,
+            -0.008718682453036308,
+            0.010000362060964108,
+            -0.045781247317790985,
+            -0.01844964735209942,
+            -0.04016471654176712,
+            0.007355368230491877,
+            0.005412440747022629,
+            -0.030907239764928818,
+            -0.05694899708032608,
+            0.025421326979994774,
+            -0.06331657618284225,
+            -0.019886434078216553,
+            0.051626358181238174,
+            -0.018400665372610092,
+            -0.008212542161345482,
+            0.009641164913773537,
+            0.006375740747898817,
+            -0.029290854930877686,
+            0.0041348435916006565,
+            0.02504580281674862,
+            0.04418119043111801,
+            -0.0045797573402523994,
+            -0.02865409664809704,
+            0.0028674507047981024,
+            -0.027788760140538216,
+            -0.013461710885167122,
+            0.005261414684355259,
+            0.016449574381113052,
+            0.024539662525057793,
+            0.007628847379237413,
+            0.03585435822606087,
+            0.007294141221791506,
+            0.006318595726042986,
+            0.044279150664806366,
+            -0.025666235014796257,
+            -0.027380581945180893,
+            0.02519274689257145,
+            0.04045860469341278,
+            0.023151857778429985,
+            -0.018253721296787262,
+            -0.0142454132437706,
+            0.0053063142113387585,
+            0.0009296254720538855,
+            -0.022139575332403183,
+            0.002808264922350645,
+            -0.013902543112635612,
+            0.005143043119460344,
+            0.03892385587096214,
+            -0.001205145614221692,
+            0.014196431264281273,
+            0.007302304729819298,
+            0.006845145486295223,
+            0.004334850702434778,
+            -0.011731036007404327,
+            -0.023592690005898476,
+            0.00032220553839579225,
+            0.034972693771123886,
+            0.015404638834297657,
+            -0.030515389516949654,
+            -0.00995954405516386,
+            0.034711457788944244,
+            -0.00795130804181099,
+            -0.009118696674704552,
+            -0.009771781973540783,
+            0.003385836724191904,
+            0.02638462744653225,
+            -0.01606588624417782,
+            0.006649219896644354,
+            0.03716052696108818,
+            -0.019870106130838394,
+            0.005543057806789875,
+            -0.025405000895261765,
+            0.0018061877926811576,
+            0.02442537248134613,
+            -0.006216551177203655,
+            0.028098974376916885,
+            5.6315813708351925e-05,
+            0.03025415539741516,
+            -0.004677720367908478,
+            0.008432958275079727,
+            -0.020327266305685043,
+            0.03987082839012146,
+            -0.017633290961384773,
+            0.03402572125196457,
+            0.02067013457417488,
+            -0.01258004643023014,
+            0.04783846437931061,
+            -0.036801330745220184,
+            -0.020784424617886543,
+            0.00584919098764658,
+            0.012743317522108555,
+            0.0028919412288814783,
+            -0.0191517136991024,
+            -0.018400665372610092,
+            -0.010996315628290176,
+            0.02380494214594364,
+            -0.022547753527760506,
+            -0.016865916550159454,
+            -0.007441085297614336,
+            -0.01890680566430092,
+            -0.009673818945884705,
+            -0.007988044060766697,
+            -0.01709449477493763,
+            -0.016547536477446556,
+            0.044573038816452026,
+            -0.06334923207759857,
+            0.016996532678604126,
+            0.017159804701805115,
+            0.011477965861558914,
+            -0.034678805619478226,
+            -0.019510909914970398,
+            -0.017633290961384773,
+            -0.002347023691982031,
+            -0.030743969604372978,
+            0.001702102366834879,
+            0.01725776679813862,
+            0.01138000376522541,
+            -0.05022222548723221,
+            0.012335140258073807,
+            0.02772345207631588,
+            -0.012065742164850235,
+            0.0014949520118534565,
+            -0.023135529831051826,
+            -0.017176130786538124,
+            0.0031776658724993467,
+            0.022123249247670174,
+            0.005975726060569286,
+            -0.008188051171600819,
+            -0.016865916550159454,
+            -0.005241005681455135,
+            -0.006045116577297449,
+            -0.002359268954023719,
+            0.005094061605632305,
+            0.011780017986893654,
+            0.019690508022904396,
+            -0.0024633544962853193,
+            -0.004436895251274109,
+            -0.006396149750798941,
+            0.027364253997802734,
+            -0.001450052484869957,
+            -0.008416631259024143,
+            0.050907962024211884,
+            0.03556047007441521,
+            0.006094098091125488,
+            -0.01108611561357975,
+            -0.03990348428487778,
+            0.010661610402166843,
+            -0.022711025550961494,
+            0.0401320643723011,
+            0.01905374974012375,
+            -0.008298259228467941,
+            -0.028866350650787354,
+            0.010294250212609768,
+            -0.02290695160627365,
+            -0.009975871071219444,
+            0.009469729848206043,
+            0.00726556871086359,
+            -0.03263791650533676,
+            -0.013543346896767616,
+            0.03510330989956856,
+            0.011437147855758667,
+            -0.006841063499450684,
+            -0.036050282418727875,
+            0.008588065393269062,
+            0.01646590232849121,
+            0.011869817040860653,
+            -0.019233347848057747,
+            0.004449140280485153,
+            -0.012041252106428146,
+            0.017470018938183784,
+            0.008849299512803555,
+            -0.017796561121940613,
+            0.018237393349409103,
+            0.004710374400019646,
+            -0.008490103296935558,
+            -0.003912386484444141,
+            -0.0163107942789793,
+            -0.02006603218615055,
+            0.03840138763189316,
+            0.024490682408213615,
+            0.026335645467042923,
+            0.005457340274006128,
+            0.02008235827088356,
+            0.03227871656417847,
+            0.011282040737569332,
+            -0.03134807199239731,
+            0.027935704216361046,
+            0.016996532678604126,
+            -0.04604247957468033,
+            0.021551799029111862,
+            -0.02218855731189251,
+            -0.028670424595475197,
+            0.009649328887462616,
+            -0.0017367975087836385,
+            0.011192241683602333,
+            0.0034837995190173388,
+            -0.003642988856881857,
+            -0.03307874873280525,
+            -0.006236960180103779,
+            0.015771998092532158,
+            0.015502601861953735,
+            -0.0048695639707148075,
+            -0.003296037670224905,
+            -0.011886144056916237,
+            0.00487364549189806,
+            0.009796272963285446,
+            0.039054471999406815,
+            0.001381682581268251,
+            -0.004751192405819893,
+            -0.0018765984568744898,
+            0.031854212284088135,
+            0.0271193478256464,
+            -0.017274094745516777,
+            -0.026286665350198746,
+            0.00580837344750762,
+            -0.014253576286137104,
+            0.026156047359108925,
+            -0.00487364549189806,
+            0.011877980083227158,
+            0.008718682453036308,
+            -0.0007908449042588472,
+            -0.001373519073240459,
+            0.025584599003195763,
+            -0.02710302174091339,
+            -0.021764053031802177,
+            0.031233781948685646,
+            -0.025731543079018593,
+            0.028735732659697533,
+            0.013061696663498878,
+            -0.01557607389986515,
+            -0.015225040726363659,
+            -0.010302413254976273,
+            0.009943217039108276,
+            -0.012277995236217976,
+            0.023853924125432968,
+            0.003608293831348419,
+            0.006392067763954401,
+            0.0003584313380997628,
+            0.02770712412893772,
+            0.024833550676703453,
+            0.007820690982043743,
+            -0.018074123188853264,
+            -0.0016816934803500772,
+            -0.02320083975791931,
+            -0.01680060662329197,
+            0.023772288113832474,
+            -0.018090449273586273,
+            -0.03268689662218094,
+            -0.028556134551763535,
+            0.012114724144339561,
+            0.007392104249447584,
+            0.010816717520356178,
+            0.03461349755525589,
+            -0.030384771525859833,
+            -0.0344502255320549,
+            0.01766594499349594,
+            0.026270337402820587,
+            -0.005551221314817667,
+            -0.021535472944378853,
+            0.03085825778543949,
+            -0.005828781984746456,
+            0.017600636929273605,
+            0.0404912605881691,
+            -0.0016225076979026198,
+            0.010228941217064857,
+            -0.042026009410619736,
+            -0.014057651162147522,
+            -0.0066614653915166855,
+            0.006261450704187155,
+            0.034417569637298584,
+            0.01048201136291027,
+            0.01535565685480833,
+            0.03448288142681122,
+            -0.021192602813243866,
+            -0.0002903166168835014,
+            -0.0022449791431427,
+            -0.02818061038851738,
+            0.03347059711813927,
+            0.025715215131640434,
+            0.011918798089027405,
+            0.019706835970282555,
+            -0.03794422745704651,
+            -0.013037205673754215,
+            0.027070365846157074,
+            0.015053605660796165,
+            0.01273515447974205,
+            0.016694480553269386,
+            0.011249386705458164,
+            0.00868602842092514,
+            -0.021633435040712357,
+            -0.05169166624546051,
+            0.01214737817645073,
+            0.01993541419506073,
+            -0.016449574381113052,
+            0.01057181041687727,
+            0.024980494752526283,
+            0.007620683871209621,
+            -0.007045152597129345,
+            0.012808626517653465,
+            -0.02682545967400074,
+            0.002259265398606658,
+            0.0019010890973731875,
+            -0.014482156373560429,
+            0.038270771503448486,
+            -0.016367938369512558,
+            0.011363675817847252,
+            -0.018204739317297935,
+            0.023870250210165977,
+            0.00011531029304023832,
+            0.0007964573451317847,
+            0.014963806606829166,
+            -0.011347348801791668,
+            -0.04235255345702171,
+            0.0021613026037812233,
+            0.03342161700129509,
+            -0.015200549736618996,
+            0.026449935510754585,
+            -0.0020674217958003283,
+            0.032458316534757614,
+            0.042777057737112045,
+            0.017731253057718277,
+            0.0003497575526125729,
+            0.025682561099529266,
+            0.0036144163459539413,
+            0.0073880222626030445,
+            0.012400448322296143,
+            0.009322785772383213,
+            0.007245160173624754,
+            0.030809277668595314,
+            -0.010261595249176025,
+            0.02173139713704586,
+            -0.04760988429188728,
+            -0.014629100449383259,
+            0.012024925090372562,
+            -0.0013316808035597205,
+            -0.012449429370462894,
+            -0.033029764890670776,
+            0.012563719414174557,
+            -0.03503800183534622,
+            0.022074267268180847,
+            -0.0004959363141097128,
+            0.04333217814564705,
+            -0.02187834307551384,
+            -0.005518566817045212,
+            0.016694480553269386,
+            0.0016510800924152136,
+            0.023282473906874657,
+            -0.0012153501156717539,
+            -0.016416920349001884,
+            -0.0067798369564116,
+            -0.009257477708160877,
+            0.004138925112783909,
+            0.04375668615102768,
+            -0.003722583642229438,
+            -0.0001682458823779598,
+            0.01409030519425869,
+            -0.011959616094827652,
+            -0.03250729665160179,
+            0.045193471014499664,
+            -0.0030837850645184517,
+            0.02484987862408161,
+            0.04979771748185158,
+            -0.06410028040409088,
+            -0.019543563947081566,
+            0.013061696663498878,
+            -0.010441194288432598,
+            0.011363675817847252,
+            0.005547139327973127,
+            0.016114868223667145,
+            0.017894525080919266,
+            -0.005485912784934044,
+            0.01619650423526764,
+            0.005106307100504637,
+            0.028098974376916885,
+            -0.02336410991847515,
+            -0.03296445682644844,
+            -0.02636830136179924,
+            -0.017747581005096436,
+            0.0014480114914476871,
+            -0.015796490013599396,
+            -3.563585551091819e-06,
+            -0.0022204883862286806,
+            -0.04388730227947235,
+            -0.025274382904171944,
+            0.040687184780836105,
+            0.006706364918500185,
+            0.02458864450454712,
+            0.0077268099412322044,
+            0.03418898954987526,
+            0.005885927006602287,
+            -0.009037061594426632,
+            0.030058229342103004,
+            -0.020049704238772392,
+            0.03833607956767082,
+            0.022106921300292015,
+            -0.002832755446434021,
+            0.005865518003702164,
+            0.031086837872862816,
+            -0.033797141164541245,
+            0.008086007088422775,
+            -0.034384917467832565,
+            0.0046695563942193985,
+            -0.008620720356702805,
+            -0.015094423666596413,
+            -0.009526874870061874,
+            -0.02504580281674862,
+            0.011216731742024422,
+            -0.021535472944378853,
+            0.01590261608362198,
+            0.010416703298687935,
+            -0.002916431985795498,
+            0.03448288142681122,
+            -0.014204595237970352,
+            -0.0027796924114227295,
+            0.0024531499948352575,
+            -0.027070365846157074,
+            -0.006343086715787649,
+            -0.003377673216164112,
+            -0.03689929470419884,
+            -0.004502203781157732,
+            0.010988152585923672,
+            -0.00024184548237826675,
+            0.016898570582270622,
+            0.02787039615213871,
+            0.0005092020728625357,
+            -0.022302847355604172,
+            0.002340900944545865,
+            -0.04199335351586342,
+            0.0011877981014549732,
+            0.01483318954706192,
+            -0.017600636929273605,
+            -0.008612556383013725,
+            0.033046092838048935,
+            0.023298801854252815,
+            -0.009396257810294628,
+            -0.035005345940589905,
+            0.017796561121940613,
+            -0.011853490024805069,
+            0.027511198073625565,
+            -0.0002630197268445045,
+            0.019298657774925232,
+            -0.0076941559091210365,
+            0.04493223875761032,
+            0.001369437319226563,
+            0.008645210415124893,
+            -0.021959977224469185,
+            0.007020662073045969,
+            -0.020278284326195717,
+            -0.002406209474429488,
+            -0.006171651650220156,
+            -0.012751481495797634,
+            -0.010669773444533348,
+            0.0032735876739025116,
+            0.007730891928076744,
+            -0.0176822729408741,
+            0.0055961208418011665,
+            -0.015959760174155235,
+            -0.02741323597729206,
+            0.007543129846453667,
+            -0.05965929850935936,
+            -0.0179598331451416,
+            -0.03373183310031891,
+            -0.009477893821895123,
+            0.02008235827088356,
+            0.03510330989956856,
+            -0.012759645469486713,
+            0.00497977202758193,
+            -0.014898497611284256,
+            0.004477712791413069,
+            0.03490738570690155,
+            0.044442422688007355,
+            -0.014759717509150505,
+            0.003381754970178008,
+            -0.010955498553812504,
+            0.033029764890670776,
+            -0.03347059711813927,
+            0.0030062312725931406,
+            0.031086837872862816,
+            -0.005057325586676598,
+            0.007959471084177494,
+            0.011624909937381744,
+            0.020474210381507874,
+            -0.019396619871258736,
+            0.00877582747489214,
+            0.002100075827911496,
+            0.019233347848057747,
+            0.02171507105231285,
+            0.020343592390418053,
+            -0.03539719805121422,
+            0.028409190475940704,
+            0.018384337425231934,
+            0.02444170042872429,
+            0.044442422688007355,
+            -0.03974021226167679,
+            -0.004147088620811701,
+            -0.0032225653994828463,
+            -0.02997659333050251,
+            0.02008235827088356,
+            -0.007530884351581335,
+            -0.01841699331998825,
+            -0.007069643586874008,
+            0.03327467292547226,
+            0.014571955427527428,
+            -0.05051611363887787,
+            0.01980479806661606,
+            0.01740471087396145,
+            0.02055584453046322,
+            0.024474354460835457,
+            -0.01123305968940258,
+            -0.0019072118448093534,
+            0.008996243588626385,
+            0.006804327480494976,
+            0.019266001880168915,
+            0.015257694758474827,
+            0.002012317767366767,
+            -0.0022082431241869926,
+            -0.011837163008749485,
+            0.005224678665399551,
+            -0.009077879600226879,
+            0.010808554477989674,
+            -0.02458864450454712,
+            -0.009698309935629368,
+            0.021388528868556023,
+            0.0031164393294602633,
+            -0.013665799982845783,
+            -0.03729114308953285,
+            -0.045944519340991974,
+            0.0033837957307696342,
+            0.004543021321296692,
+            -0.05365091934800148,
+            0.005322641227394342,
+            -0.026547899469733238,
+            -0.002108239568769932,
+            -0.005502239800989628,
+            0.0019214981002733111,
+            0.005636938381940126,
+            0.0167679525911808,
+            -0.013494365848600864,
+            0.02248244546353817,
+            -0.015502601861953735,
+            0.03165828809142113,
+            0.0055961208418011665,
+            0.007375776767730713,
+            -0.0027796924114227295,
+            0.05590406060218811,
+            -0.003712379140779376,
+            -0.03699725493788719,
+            0.023282473906874657,
+            -0.01311067771166563,
+            -0.0001747512142173946,
+            0.030090883374214172,
+            0.00884113647043705,
+            -0.015257694758474827,
+            0.013437220826745033,
+            -0.030613351613283157,
+            -0.03249097242951393,
+            0.010996315628290176,
+            -0.013755599036812782,
+            -0.0015776080545037985,
+            -0.04568328335881233,
+            -0.009339112788438797,
+            0.004269542172551155,
+            -0.018335357308387756,
+            0.010775899514555931,
+            0.01631895825266838,
+            0.013992343097925186,
+            -0.024245774373412132,
+            0.009053388610482216,
+            0.006865554489195347,
+            0.013453547842800617,
+            0.0007331897504627705,
+            0.014049487188458443,
+            -0.008677864447236061,
+            -0.011306531727313995,
+            -0.018319029361009598,
+            0.0032450153958052397,
+            -0.026792805641889572,
+            0.005155288614332676,
+            -0.003155216109007597,
+            -0.03771565109491348,
+            -0.06521052122116089,
+            -0.0344502255320549,
+            0.016294466331601143,
+            0.03396041318774223,
+            -0.009045224636793137,
+            0.0035144127905368805,
+            -0.00023648813657928258,
+            -0.01093917153775692,
+            0.024523336440324783,
+            -0.005930826533585787,
+            -0.0004829256213270128,
+            0.0035389035474509,
+            0.02050686441361904,
+            -0.009722800925374031,
+            0.005845109466463327,
+            -0.051952898502349854,
+            -0.01586179807782173,
+            0.01812310516834259,
+            0.006306350696831942,
+            0.014629100449383259,
+            -0.009690145961940289,
+            0.01693122461438179,
+            -0.030074557289481163,
+            -0.028147956356406212,
+            0.04130761697888374,
+            0.008277850225567818,
+            0.011486129835247993,
+            -0.003055212553590536,
+            -0.01680060662329197,
+            0.02504580281674862,
+            -0.0020633398089557886,
+            -0.006461458280682564,
+            -0.00817988719791174,
+            0.015820980072021484,
+            0.044279150664806366,
+            0.003255219664424658,
+            0.0009821783751249313,
+            0.017045514658093452,
+            -0.002663361607119441,
+            -0.011575928889214993,
+            0.007677828427404165,
+            0.013592327944934368,
+            -0.016384266316890717,
+            0.010375885292887688,
+            -0.025862159207463264,
+            0.0332583449780941,
+            0.03131541982293129,
+            0.011575928889214993,
+            -0.015045441687107086,
+            0.006130834110081196,
+            0.005330805201083422,
+            0.008086007088422775,
+            -0.001369437319226563,
+            -0.0017133272485807538,
+            0.020719116553664207,
+            -0.03276853263378143,
+            -0.01693122461438179,
+            -0.005347132217139006,
+            0.003932795487344265,
+            0.005277741700410843,
+            0.03396041318774223,
+            -0.0033266509417444468,
+            0.017927179113030434,
+            -0.011657563969492912,
+            0.010579974390566349,
+            0.021764053031802177,
+            0.037095218896865845,
+            -0.024996822699904442,
+            -0.029862305149435997,
+            0.012914752587676048,
+            -0.001296985661610961,
+            0.014065815135836601,
+            0.0010173837654292583,
+            0.014139287173748016,
+            0.0029409227427095175,
+            0.060475654900074005,
+            -0.017763907089829445,
+            0.004628738854080439,
+            0.0004035860183648765,
+            -0.01858026348054409,
+            -0.0013531101867556572,
+            0.004261378664523363,
+            -0.010979988612234592,
+            0.03608293831348419,
+            0.002344982698559761,
+            0.039185091853141785,
+            0.013918871060013771,
+            0.009290131740272045,
+            0.0015143405180424452,
+            -0.013592327944934368,
+            0.008355404250323772,
+            0.0015592400450259447,
+            -0.001132694073021412,
+            0.009461566805839539,
+            0.019298657774925232,
+            0.009714636951684952,
+            0.025274382904171944,
+            0.009788108989596367,
+            0.012857607565820217,
+            -0.029470453038811684,
+            0.03510330989956856,
+            -0.0006224715034477413,
+            0.012996388599276543,
+            -0.02040890045464039,
+            -0.003136848099529743,
+            -0.0008240094175562263,
+            0.015649545937776566,
+            -0.015616890974342823,
+            -0.001201063860207796,
+            -0.01297189760953188,
+            0.040099408477544785,
+            -0.018988441675901413,
+            0.013976015150547028,
+            0.009673818945884705,
+            -0.02865409664809704,
+            -0.007596192881464958,
+            0.01995174214243889,
+            0.014294394291937351,
+            -0.0053144777193665504,
+            -0.020490536466240883,
+            0.005832863971590996,
+            0.015380147844552994,
+            -0.028572462499141693,
+            1.3584674888988957e-05,
+            0.014751553535461426,
+            -0.0164903923869133,
+            -0.00025409081717953086,
+            -0.004702210891991854,
+            0.003975654020905495,
+            -0.009265640750527382,
+            0.007028825581073761,
+            0.016057724133133888,
+            -0.024507008492946625,
+            -0.020049704238772392,
+            0.005722655914723873,
+            0.006571666337549686,
+            0.01444950234144926,
+            0.007739055436104536,
+            0.004355259705334902,
+            -0.020033378154039383,
+            -0.01799248717725277,
+            0.0032674651592969894,
+            -0.01649855636060238,
+            -0.0048614004626870155,
+            0.0020531355403363705,
+            -0.019363965839147568,
+            0.01875986158847809,
+            -0.010743245482444763,
+            0.00017538899555802345,
+            -0.003226647386327386,
+            -0.020800752565264702,
+            0.006204306147992611,
+            -0.004865481983870268,
+            0.019870106130838394,
+            0.003555230563506484,
+            0.02785406820476055,
+            0.02310287579894066,
+            -0.004428731743246317,
+            0.017600636929273605,
+            -0.016155686229467392,
+            0.007983962073922157,
+            -0.031135819852352142,
+            -0.001782717532478273,
+            -0.010408539324998856,
+            0.0036103345919400454,
+            -0.015943434089422226,
+            -0.009126860648393631,
+            -0.0048777274787425995,
+            -0.0006678812787868083,
+            -0.004820582456886768,
+            0.0004275664687156677,
+            0.0027817331720143557,
+            0.02231917530298233,
+            -0.013208640739321709,
+            -0.02803366631269455,
+            -0.017437364906072617,
+            0.006045116577297449,
+            0.019396619871258736,
+            -0.026025431230664253,
+            -0.009184005670249462,
+            -0.005012426059693098,
+            -0.004326687194406986,
+            -0.021404854953289032,
+            -0.015167895704507828,
+            -0.005632856860756874,
+            -0.008045189082622528,
+            -0.010522829368710518,
+            0.0046532293781638145,
+            -0.0035715578123927116,
+            -0.011739199981093407,
+            0.021666089072823524,
+            0.018057795241475105,
+            -0.019837452098727226,
+            -0.019837452098727226,
+            0.013641309924423695,
+            -0.006159406621009111,
+            0.0018878233386203647,
+            0.02008235827088356,
+            0.00029286774224601686,
+            0.005824700463563204,
+            0.020882388576865196,
+            -0.0035225762985646725,
+            -0.01797616109251976,
+            0.003577680326998234,
+            -0.0018888438353314996,
+            -0.012122887186706066,
+            0.003212361130863428,
+            0.02773977816104889,
+            0.0008076822850853205,
+            0.023429417982697487,
+            -0.03056436963379383,
+            -0.011412657797336578,
+            0.006061443593353033,
+            0.02369065210223198,
+            0.00211844383738935,
+            -0.0021694661118090153,
+            0.020131340250372887,
+            0.03941367194056511,
+            0.005775718949735165,
+            0.0015755671774968505,
+            -0.04003410041332245,
+            0.01027792226523161,
+            0.009208496659994125,
+            0.024621298536658287,
+            0.008339077234268188,
+            0.0014653591206297278,
+            -0.012906589545309544,
+            -0.009755454957485199,
+            -0.0017939424142241478,
+            0.014702572487294674,
+            -0.023870250210165977,
+            0.0024755997583270073,
+            -0.03157665207982063,
+            0.005094061605632305,
+            0.0021959978621453047,
+            -0.009559528902173042,
+            0.0335359051823616,
+            0.00980443600565195,
+            -0.02065380848944187,
+            0.015151568688452244,
+            0.005767555441707373,
+            0.007330877240747213,
+            0.022449791431427002,
+            -0.008228869177401066,
+            0.030450081452727318,
+            -0.005049162078648806,
+            0.01213921420276165,
+            0.02742956392467022,
+            0.032588932663202286,
+            -0.013102514669299126,
+            -0.003959327004849911,
+            0.0006097159348428249,
+            -0.0002561317232903093,
+            0.0007995187188498676,
+            -0.008204378187656403,
+            0.0037389106582850218,
+            -0.014800535514950752,
+            0.012710663489997387,
+            -0.019543563947081566,
+            -0.0034286954905837774,
+            0.0038532004691660404,
+            0.01991908811032772,
+            0.0016265894519165158,
+            -0.033633869141340256,
+            0.02055584453046322,
+            -0.015420965850353241,
+            0.0135596739128232,
+            0.001290863030590117,
+            -0.011110605672001839,
+            -0.00029210239881649613,
+            0.020588500425219536,
+            0.010947334580123425,
+            -0.007845181971788406,
+            -0.0135596739128232,
+            -0.028376536443829536,
+            -0.021535472944378853,
+            0.00823703221976757,
+            0.009445239789783955,
+            -0.0028837777208536863,
+            0.0021306893322616816,
+            -0.011869817040860653,
+            0.0006520643946714699,
+            -0.0008122742874547839,
+            0.01740471087396145,
+            -0.0335359051823616,
+            0.021943651139736176,
+            -0.019527235999703407,
+            -0.019543563947081566,
+            0.017045514658093452,
+            0.008049271069467068,
+            0.009445239789783955,
+            0.0033552234526723623,
+            -0.01995174214243889,
+            -0.012326976284384727,
+            0.0338951013982296,
+            -0.01138000376522541,
+            0.006298186723142862,
+            0.004408322740346193,
+            0.012914752587676048,
+            -0.008914608508348465,
+            -0.011926962062716484,
+            0.01784554310142994,
+            -0.009861581027507782,
+            0.013951525092124939,
+            0.0010010566329583526,
+            -0.0254703089594841,
+            0.02739690989255905,
+            -0.019233347848057747,
+            0.03487472981214523,
+            -0.021617108955979347,
+            0.00711862463504076,
+            0.0066614653915166855,
+            0.008792154490947723,
+            -0.006690037902444601,
+            -0.013224967755377293,
+            0.0012786176521331072,
+            0.03807484731078148,
+            -0.02623768337070942,
+            0.024180466309189796,
+            0.022400809451937675,
+            -0.00040766780148260295,
+            0.02065380848944187,
+            -0.008367649279534817,
+            -0.026711169630289078,
+            0.0179598331451416,
+            -0.012596373446285725,
+            -0.026123393326997757,
+            0.013257622718811035,
+            -0.022678371518850327,
+            -0.020278284326195717,
+            -0.026302991434931755,
+            0.004947117529809475,
+            -0.0025694805663079023,
+            -0.005151206627488136,
+            0.009004407562315464,
+            0.02561725303530693,
+            0.020915042608976364,
+            0.019576217979192734,
+            0.02111096680164337,
+            0.01161674689501524,
+            0.022449791431427002,
+            -0.007559456862509251,
+            0.04401791840791702,
+            -0.004947117529809475,
+            0.004632820375263691,
+            -0.0015980169409886003,
+            -0.015657709911465645,
+            0.009771781973540783,
+            0.0119106350466609,
+            0.0029572497587651014,
+            -0.029176564887166023,
+            -0.009061552584171295,
+            -0.03660540655255318,
+            0.005053244065493345,
+            0.0020664012990891933,
+            0.0176822729408741,
+            0.032115448266267776,
+            -0.005865518003702164,
+            0.004592002835124731,
+            -0.005102225113660097,
+            -0.013927034102380276,
+            -0.009420748800039291,
+            -0.02739690989255905,
+            -0.025437654927372932,
+            -0.007653337903320789,
+            -0.013739272020757198,
+            -0.030368445441126823,
+            -0.011722872965037823,
+            0.01551892887800932,
+            -0.007645174395292997,
+            0.011918798089027405,
+            -0.016702644526958466,
+            0.01385356206446886,
+            -0.004110352601855993,
+            -0.004763437435030937,
+            -0.004530776292085648,
+            -0.017600636929273605,
+            -0.006420640274882317,
+            0.014367866329848766,
+            -0.009477893821895123,
+            -0.01237595733255148,
+            -0.02215590327978134,
+            -0.0015459742862731218,
+            -0.01574750803411007,
+            0.003489922033622861,
+            -0.014596446417272091,
+            -0.006771673448383808,
+            -0.003908304497599602,
+            -0.018514955416321754,
+            0.01766594499349594,
+            0.014816862531006336,
+            -0.019380291923880577,
+            -0.0090697156265378,
+            0.02638462744653225,
+            0.042777057737112045,
+            -0.03218075633049011,
+            -0.008200296200811863,
+            0.0020378287881612778,
+            0.018090449273586273,
+            0.010171796195209026,
+            0.016702644526958466,
+            -0.024653952568769455,
+            0.0022286521270871162,
+            -0.021649762988090515,
+            -0.001220452249981463,
+            -0.023739634081721306,
+            0.010775899514555931,
+            0.025715215131640434,
+            -0.010971825569868088,
+            0.025976449251174927,
+            0.01634344831109047,
+            -0.01812310516834259,
+            0.012016761116683483,
+            0.007183933164924383,
+            0.014645427465438843,
+            -0.019755816087126732,
+            0.0167679525911808,
+            -0.015796490013599396,
+            0.031233781948685646,
+            0.00892277155071497,
+            -0.04829562455415726,
+            -0.03503800183534622,
+            -0.013388238847255707,
+            0.0013388239312916994,
+            -0.010702427476644516,
+            0.015037278644740582,
+            0.015967924147844315,
+            0.013959688134491444,
+            -0.019380291923880577,
+            -0.015820980072021484,
+            0.011984107084572315,
+            0.010065670125186443,
+            0.027462217956781387,
+            -0.006649219896644354,
+            -0.004212397150695324,
+            -0.00960034690797329,
+            -0.02249877341091633,
+            -0.016098542138934135,
+            0.0209803506731987,
+            -0.018041469156742096,
+            0.015935270115733147,
+            -0.026139721274375916,
+            -0.012449429370462894,
+            0.017453692853450775,
+            -0.05211617052555084,
+            -0.01453113742172718,
+            0.0007612520130351186,
+            0.03012353740632534,
+            0.004751192405819893,
+            0.0035184945445507765,
+            0.01841699331998825,
+            -0.01993541419506073,
+            -0.0039776950143277645,
+            -0.02099667675793171,
+            0.006832899991422892,
+            -0.011853490024805069,
+            -0.013984179124236107,
+            0.014482156373560429,
+            -0.013845399022102356,
+            -0.01601690612733364,
+            0.0034368589986115694,
+            0.007183933164924383,
+            -0.021094640716910362,
+            0.011951452121138573,
+            0.003945040516555309,
+            0.012171869166195393,
+            0.012171869166195393,
+            0.01199227012693882,
+            0.021551799029111862,
+            0.02295593172311783,
+            -0.007334959227591753,
+            -0.004767519421875477,
+            -0.01004117913544178,
+            -0.0015398516552522779,
+            0.008898280560970306,
+            0.0033797139767557383,
+            -0.002989904023706913,
+            0.007837017998099327,
+            0.010033016093075275,
+            -0.01498013362288475,
+            0.033193036913871765,
+            -0.0035878848284482956,
+            -0.04134026914834976,
+            -0.003238892648369074,
+            -0.03800953924655914,
+            0.020033378154039383,
+            -0.023298801854252815,
+            -0.016996532678604126,
+            0.0032776694279164076,
+            -0.00629410520195961,
+            -0.019902760162949562,
+            0.013608654960989952,
+            0.011404493823647499,
+            0.00258580781519413,
+            0.02021297626197338,
+            0.012269831262528896,
+            -0.005134879611432552,
+            0.026172375306487083,
+            -0.014718899503350258,
+            -0.008751336485147476,
+            0.015567909926176071,
+            0.006853308994323015,
+            0.0022347746416926384,
+            0.0011031010653823614,
+            -0.018825169652700424,
+            0.005098143592476845,
+            0.0164903923869133,
+            0.007216587662696838,
+            -0.000566347036510706,
+            0.014049487188458443,
+            0.012514738366007805,
+            -0.015135240741074085,
+            -0.0002213090192526579,
+            -0.026564225554466248,
+            0.0032817514147609472,
+            0.004567512311041355,
+            0.007506393827497959,
+            -0.006771673448383808,
+            0.006281859707087278,
+            0.00658799335360527,
+            0.014759717509150505,
+            -0.010155469179153442,
+            -0.016980206593871117,
+            -0.002510294783860445,
+            0.004285869188606739,
+            0.016694480553269386,
+            0.03157665207982063,
+            -0.010310577228665352,
+            -0.005734901409596205,
+            -0.0386626236140728,
+            0.02070278860628605,
+            0.03332365304231644,
+            0.011175914667546749,
+            -0.028213264420628548,
+            0.012302485294640064,
+            0.0006515541463159025,
+            0.0019276207312941551,
+            -0.026302991434931755,
+            0.02787039615213871,
+            0.00974729098379612,
+            0.0066614653915166855,
+            0.017861871048808098,
+            0.0022306928876787424,
+            0.0029266364872455597,
+            -0.016302630305290222,
+            0.01018812321126461,
+            0.009086042642593384,
+            -0.017306748777627945,
+            0.0009745250572450459,
+            0.02324981987476349,
+            -0.01423724927008152,
+            -0.003208279376849532,
+            0.0018133309204131365,
+            0.02651524543762207,
+            -0.013820908032357693,
+            -0.014294394291937351,
+            0.0005714492290280759,
+            0.01751900091767311,
+            -0.0007979880319908261,
+            0.012098397128283978,
+            0.04898136481642723,
+            -0.012424939312040806,
+            0.019657853990793228,
+            0.005461421795189381,
+            -0.015290348790585995,
+            -0.032605260610580444,
+            -0.004922627005726099,
+            -0.01830270327627659,
+            0.011200404725968838,
+            0.014343376271426678,
+            -0.015763835981488228,
+            0.0007525782566517591,
+            -0.03144603595137596,
+            0.007743136957287788,
+            0.014735226519405842,
+            0.018776189535856247,
+            -0.002440904499962926,
+            0.006134915631264448,
+            0.027674470096826553,
+            0.02458864450454712,
+            -0.007392104249447584,
+            0.013804581016302109,
+            0.02320083975791931,
+            -0.025062130764126778,
+            -0.003069498809054494,
+            0.006820654962211847,
+            0.004104230087250471,
+            0.0053063142113387585,
+            -0.016963878646492958,
+            0.03915243595838547,
+            -0.005922663025557995,
+            -0.01506176870316267,
+            0.0030266400426626205,
+            -0.009028897620737553,
+            0.0063390047289431095,
+            -0.012563719414174557,
+            0.012637191452085972,
+            -0.008114579133689404,
+            0.00022883480414748192,
+            -0.0059430720284581184,
+            -0.011110605672001839,
+            -0.012653518468141556,
+            -0.0007949267164804041,
+            -0.016531210392713547,
+            0.007432921789586544,
+            0.00981259997934103,
+            0.005045080557465553,
+            0.01025343220680952,
+            -0.0036797248758375645,
+            -0.003136848099529743,
+            -0.016751626506447792,
+            -0.009984034113585949,
+            0.0073716952465474606,
+            0.01872720755636692,
+            -0.028588788583874702,
+            -0.018547609448432922,
+            0.009290131740272045,
+            0.0022898786701261997,
+            0.018776189535856247,
+            0.0158944521099329,
+            -0.0002595757250674069,
+            0.012318813242018223,
+            -0.01964152604341507,
+            -0.009184005670249462,
+            0.009690145961940289,
+            -0.003569516818970442,
+            0.01048201136291027,
+            -0.014759717509150505,
+            0.010033016093075275,
+            -0.011159587651491165,
+            -0.011053460650146008,
+            0.01385356206446886,
+            0.01738838478922844,
+            0.008873790502548218,
+            -0.001294944784604013,
+            3.383413059054874e-05,
+            -0.010506502352654934,
+            -0.024033522233366966,
+            0.004771600943058729,
+            0.027021385729312897,
+            -0.013273949734866619,
+            0.003883813973516226,
+            0.008702355436980724,
+            -0.017502672970294952,
+            -0.019821124151349068,
+            -0.012653518468141556,
+            -0.022106921300292015,
+            0.020572172477841377,
+            0.023413091897964478,
+            -0.02026195637881756,
+            0.00877582747489214,
+            -0.004620575346052647,
+            0.014049487188458443,
+            -0.01400050614029169,
+            0.010612628422677517,
+            0.0317889042198658,
+            -0.020000724121928215,
+            -0.0028654097113758326,
+            0.0036633978597819805,
+            -0.015347493812441826,
+            0.018629245460033417,
+            0.03268689662218094,
+            -0.00176945177372545,
+            0.014041324146091938,
+            0.0008138049161061645,
+            -0.009118696674704552,
+            0.01905374974012375,
+            0.011094278655946255,
+            0.005347132217139006,
+            -0.0042491331696510315,
+            -0.012212686240673065,
+            -0.03010721132159233,
+            0.016702644526958466,
+            0.0129637336358428,
+            -0.0037674831692129374,
+            -0.0011551438365131617,
+            -0.006343086715787649,
+            -0.015404638834297657,
+            0.0035123717971146107,
+            -0.013078023679554462,
+            -0.00803294312208891,
+            -0.008759500458836555,
+            0.004314441699534655,
+            -0.0034960447810590267,
+            -0.009053388610482216,
+            0.007236996199935675,
+            0.0009510547970421612,
+            -0.0010602424154058099,
+            -0.01902109570801258,
+            -0.025078458711504936,
+            -0.00995954405516386,
+            -0.016849588602781296,
+            0.0038287099450826645,
+            0.0005234883283264935,
+            -0.027184655889868736,
+            0.0053961132653057575,
+            -0.008139070123434067,
+            -0.008134988136589527,
+            0.007126788143068552,
+            0.006881881505250931,
+            -0.007437003776431084,
+            -0.010816717520356178,
+            -0.00786150898784399,
+            -0.006865554489195347,
+            0.020033378154039383,
+            0.0034348180051892996,
+            0.012082069180905819,
+            -0.010498338378965855,
+            -0.018025141209363937,
+            -0.016963878646492958,
+            0.016718972474336624,
+            -0.008971752598881721,
+            -0.02083340659737587,
+            -0.01444950234144926,
+            -0.008424794301390648,
+            -0.007665583398193121,
+            -0.01527402177453041,
+            -0.021159948781132698,
+            -0.0032103201374411583,
+            -0.011347348801791668,
+            -0.0067431009374558926,
+            -0.0028960229828953743,
+            0.0013571919407695532,
+            0.00951871182769537,
+            0.00546958576887846,
+            0.00438791373744607,
+            0.021666089072823524,
+            0.010122815147042274,
+            -0.005461421795189381,
+            0.016441410407423973,
+            -0.006183897145092487,
+            -0.004440976772457361,
+            -0.01498013362288475,
+            -0.009918726049363613,
+            0.018547609448432922,
+            -0.0055226488038897514,
+            0.008457448333501816,
+            -0.0009536059224046767,
+            -0.00576347392052412,
+            0.015674035996198654,
+            0.0129637336358428,
+            0.016718972474336624,
+            0.004771600943058729,
+            -0.00012168807734269649,
+            0.012416775338351727,
+            -0.0015745466807857156,
+            0.012555556371808052,
+            -0.0038429962005466223,
+            0.014506647363305092,
+            -0.009722800925374031,
+            -0.018988441675901413,
+            0.004157293122261763,
+            0.006265532691031694,
+            -0.012531065382063389,
+            0.009404421783983707,
+            0.007743136957287788,
+            -0.013747435994446278,
+            -0.02261306345462799,
+            -0.000106317616882734,
+            0.00014362763613462448,
+            0.008122743107378483,
+            0.008971752598881721,
+            -0.008073761127889156,
+            0.012890261597931385,
+            0.004616493359208107,
+            -0.007179851643741131,
+            -0.019657853990793228,
+            0.026009103283286095,
+            0.015608727931976318,
+            0.012269831262528896,
+            -0.052508022636175156,
+            -0.0030980713199824095,
+            0.010743245482444763,
+            -0.015951598063111305,
+            -0.026792805641889572,
+            -0.006404313258826733,
+            -0.008759500458836555,
+            -0.04290767386555672,
+            -0.009624837897717953,
+            -0.0005724696675315499,
+            -0.011388166807591915,
+            0.010400376282632351,
+            0.01573934406042099,
+            0.014612773433327675,
+            0.005489994306117296,
+            -0.007232914678752422,
+            -0.017633290961384773,
+            -0.02788672223687172,
+            0.010057506151497364,
+            -0.013380075804889202,
+            -0.006412476766854525,
+            -0.002263347152620554,
+            -0.007690073922276497,
+            0.0016112826997414231,
+            0.01691489666700363,
+            0.042483169585466385,
+            0.023037567734718323,
+            -0.012335140258073807,
+            -0.020131340250372887,
+            -0.012800462543964386,
+            -0.009192168712615967,
+            0.00633492274209857,
+            0.011592255905270576,
+            -0.004738946910947561,
+            0.0096656559035182,
+            -0.004294032696634531,
+            0.012302485294640064,
+            0.012767808511853218,
+            0.01680060662329197,
+            -2.3023791072773747e-05,
+            0.005579793825745583,
+            -0.0024694770108908415,
+            -0.01738838478922844,
+            -0.004379750229418278,
+            0.008261523209512234,
+            -0.0008872769540175796,
+            0.0019317024853080511,
+            0.014204595237970352,
+            0.019102731719613075,
+            -0.009094206616282463,
+            -0.014612773433327675,
+            -0.016098542138934135,
+            -0.005277741700410843,
+            -0.0012622905196622014,
+            0.029927613213658333,
+            0.02037624642252922,
+            -0.019266001880168915,
+            -0.018237393349409103,
+            -0.018351683393120766,
+            0.00922482367604971,
+            -0.003563394071534276,
+            -0.0064369672909379005,
+            0.01905374974012375,
+            0.023021239787340164,
+            0.017910851165652275,
+            -0.003642988856881857,
+            -0.014865843579173088,
+            0.003804219188168645,
+            0.02847449854016304,
+            0.008367649279534817,
+            0.01078406348824501,
+            0.008726846426725388,
+            -0.0011908593587577343,
+            -0.032441988587379456,
+            -0.0113146947696805,
+            0.00951871182769537,
+            -0.008481939323246479,
+            0.026564225554466248,
+            -0.016531210392713547,
+            0.009388094767928123,
+            -0.005077734589576721,
+            0.0019286411115899682,
+            -0.02365799807012081,
+            0.0014929111348465085,
+            0.010122815147042274,
+            -0.024947840720415115,
+            0.00336950970813632,
+            -0.023592690005898476,
+            -0.00013763252354692668,
+            0.008367649279534817,
+            0.0022837561555206776,
+            0.01650671847164631,
+            -0.0113146947696805,
+            0.0051838611252605915,
+            0.02099667675793171,
+            -0.002832755446434021,
+            -0.01512707769870758,
+            -0.0007842120248824358,
+            0.0005092020728625357,
+            -0.001286781276576221,
+            0.02563357912003994,
+            0.015812816098332405,
+            -0.0036756431218236685,
+            0.020882388576865196,
+            0.008677864447236061,
+            0.0019133344758301973,
+            -0.021764053031802177,
+            0.0023796779569238424,
+            -0.032899148762226105,
+            0.017437364906072617,
+            -0.011126932688057423,
+            0.012490247376263142,
+            0.00868602842092514,
+            -0.01341272983700037,
+            -0.0007388022495433688,
+            -0.0107187544927001,
+            -0.0015725058037787676,
+            -0.007934981025755405,
+            0.009779945947229862,
+            -0.004363423213362694,
+            0.008339077234268188,
+            -0.00024579971795901656,
+            -0.01476788055151701,
+            0.018694553524255753,
+            9.678283277025912e-06,
+            -0.00010925139940809458,
+            0.005441013257950544,
+            0.009935053065419197,
+            0.00786559097468853,
+            0.002009256277233362,
+            0.023870250210165977,
+            -0.031119491904973984,
+            -0.016082214191555977,
+            -0.010204450227320194,
+            0.005543057806789875,
+            -0.007922735065221786,
+            -0.018531281501054764,
+            -0.03257260471582413,
+            0.01318415068089962,
+            0.001293924287892878,
+            -0.0009806477464735508,
+            0.01258004643023014,
+            -0.00942891277372837,
+            -0.0013878053287044168,
+            -0.01606588624417782,
+            -0.014808698557317257,
+            0.006416558753699064,
+            0.0033633869607001543,
+            -0.01222901325672865,
+            -0.023739634081721306,
+            0.014759717509150505,
+            0.0007418635650537908,
+            -0.016980206593871117,
+            -0.017502672970294952,
+            0.00018457300029695034,
+            -0.004804255440831184,
+            0.02204161323606968,
+            -0.016678154468536377,
+            0.003373591462150216,
+            0.0007219648687168956,
+            0.009232986718416214,
+            -0.02889900468289852,
+            0.00716352416202426,
+            0.0006107363733462989,
+            0.0035164537839591503,
+            -0.006518602836877108,
+            -0.021013004705309868,
+            0.011371839791536331,
+            -0.0020674217958003283,
+            -0.014751553535461426,
+            -0.020751770585775375,
+            -0.0020796670578420162,
+            -0.007420676294714212,
+            -0.02724996581673622,
+            0.017910851165652275,
+            0.0009985055075958371,
+            -0.01738838478922844,
+            0.008077843114733696,
+            0.006596156861633062,
+            0.009355440735816956,
+            0.02455599047243595,
+            0.010147306136786938,
+            -0.018776189535856247,
+            -0.0008087027235887945,
+            0.006979844067245722,
+            0.009306458756327629,
+            0.0036450298503041267,
+            0.04222193360328674,
+            -0.008130906149744987,
+            -0.006579829845577478,
+            0.0066614653915166855,
+            -0.008449285291135311,
+            -0.013747435994446278,
+            0.026727497577667236,
+            0.005796127952635288,
+            0.015984252095222473,
+            0.016123032197356224,
+            -0.00945340283215046,
+            -0.005008344538509846,
+            -0.0018786393338814378,
+            0.02065380848944187,
+            0.0073798587545752525,
+            -0.005502239800989628,
+            -0.009714636951684952,
+            0.005881845485419035,
+            -0.00898808054625988,
+            0.0021653843577951193,
+            -0.009526874870061874,
+            -0.005824700463563204,
+            -0.008069679141044617,
+            0.016400592401623726,
+            -0.0003523086488712579,
+            0.027315273880958557,
+            0.020751770585775375,
+            -0.008563575334846973,
+            -0.0005683879135176539,
+            0.017208784818649292,
+            -0.005820618476718664,
+            0.015608727931976318,
+            -0.011192241683602333,
+            -0.007706400938332081,
+            0.006408394780009985,
+            0.024017194285988808,
+            0.012237177230417728,
+            -0.0056042843498289585,
+            0.0002885308349505067,
+            -0.0030348035506904125,
+            -0.010612628422677517,
+            0.013388238847255707,
+            0.015323002822697163,
+            -0.01947825588285923,
+            -0.004722619894891977,
+            -0.0016592437168583274,
+            -0.003300119424238801,
+            0.00945340283215046,
+            0.02305389568209648,
+            0.000171817431692034,
+            0.009184005670249462,
+            0.0012765767751261592,
+            0.03862996771931648,
+            -0.0053144777193665504,
+            0.03102152980864048,
+            -0.0010423846542835236,
+            -0.0059430720284581184,
+            -0.009094206616282463,
+            -0.020441556349396706,
+            -0.011486129835247993,
+            -0.00041481092921458185,
+            -0.005040998570621014,
+            -0.0004956811899319291,
+            -0.020327266305685043,
+            0.0026960158720612526,
+            0.014637264423072338,
+            0.007661501411348581,
+            0.017927179113030434,
+            -0.002889900468289852,
+            0.017437364906072617,
+            0.019690508022904396,
+            -0.01400050614029169,
+            -0.029862305149435997,
+            -0.0006515541463159025,
+            0.01498013362288475,
+            -0.01661284640431404,
+            0.019445601850748062,
+            0.004085862077772617,
+            0.0029531680047512054,
+            0.004779764451086521,
+            -0.006049198564141989,
+            0.018482301384210587,
+            0.02039257436990738,
+            0.01453930139541626,
+            -0.013167822733521461,
+            0.014057651162147522,
+            0.009771781973540783,
+            0.006600238848477602,
+            -0.01646590232849121,
+            -0.013535182923078537,
+            0.002555194543674588,
+            0.0009245232213288546,
+            0.024392718449234962,
+            -0.020588500425219536,
+            0.0016602640971541405,
+            0.010588138364255428,
+            0.01769859902560711,
+            -0.004302196204662323,
+            0.008906444534659386,
+            0.0008740111952647567,
+            0.007134951651096344,
+            -0.0035919665824621916,
+            -0.006457376293838024,
+            -0.006673710886389017,
+            -0.010392212308943272,
+            -0.0020959940738976,
+            -0.02070278860628605,
+            0.002198038622736931,
+            -0.002830714685842395,
+            0.018237393349409103,
+            -0.01018812321126461,
+            0.038140155375003815,
+            0.03399306535720825,
+            -0.0007025764207355678,
+            0.011102442629635334,
+            -0.01709449477493763,
+            -0.003945040516555309,
+            -0.021959977224469185,
+            0.01844964735209942,
+            0.016392430290579796,
+            0.012269831262528896,
+            0.005302232690155506,
+            0.017796561121940613,
+            -0.0006301248213276267,
+            0.014490320347249508,
+            0.0029654132667928934,
+            0.007971717044711113,
+            -0.002265388146042824,
+            0.003938918001949787,
+            -0.031103165820240974,
+            -0.017029186710715294,
+            -0.01138000376522541,
+            0.004796091932803392,
+            -0.006375740747898817,
+            -0.0035205355379730463,
+            0.0084737753495574,
+            -0.005902254022657871,
+            0.0004270562494639307,
+            -0.025992777198553085,
+            -0.0157148540019989,
+            -0.010318740271031857,
+            -0.004040962550789118,
+            -0.01169838197529316,
+            -0.0027694879099726677,
+            -0.0022082431241869926,
+            0.0029552089981734753,
+            0.020947696641087532,
+            -0.012808626517653465,
+            -0.016294466331601143,
+            0.004379750229418278,
+            -0.002263347152620554,
+            -0.014016833156347275,
+            -0.016898570582270622,
+            0.006665547378361225,
+            0.012098397128283978,
+            -0.0017470018938183784,
+            -0.013608654960989952,
+            0.011739199981093407,
+            -0.006628811359405518,
+            0.04663025587797165,
+            -0.013584164902567863,
+            -0.0078002819791436195,
+            0.003720542648807168,
+            0.004698128905147314,
+            -0.009396257810294628,
+            0.007330877240747213,
+            0.013020878657698631,
+            0.01830270327627659,
+            0.0095758568495512,
+            -0.008465612307190895,
+            -0.0024388637393712997,
+            0.007796200457960367,
+            -0.016147522255778313,
+            0.006404313258826733,
+            0.0008525818702764809,
+            -0.014400520361959934,
+            0.006110425107181072,
+            0.028980640694499016,
+            0.013151495717465878,
+            0.01288209855556488,
+            -0.022351829335093498,
+            -0.004094025585800409,
+            -0.009355440735816956,
+            -0.004943036008626223,
+            -0.010090161114931107,
+            0.027070365846157074,
+            -0.018155759200453758,
+            0.02651524543762207,
+            0.00898808054625988,
+            0.0017684312770143151,
+            -0.01176369097083807,
+            0.006343086715787649,
+            0.02832755446434021,
+            0.0028592869639396667,
+            -0.020000724121928215,
+            -0.013592327944934368,
+            0.006314514204859734,
+            0.01707816869020462,
+            -0.002647034591063857,
+            -0.005428767763078213,
+            0.019657853990793228,
+            0.01784554310142994,
+            -0.004334850702434778,
+            -0.00936360377818346,
+            0.0008337036124430597,
+            0.00928196869790554,
+            0.016963878646492958,
+            0.015020951628684998,
+            0.009086042642593384,
+            -0.002647034591063857,
+            0.008037025108933449,
+            -0.008800318464636803,
+            -0.016449574381113052,
+            0.007220669183880091,
+            -0.008432958275079727,
+            -0.016963878646492958,
+            0.01872720755636692,
+            0.015257694758474827,
+            0.0181394312530756,
+            0.0024592725094407797,
+            -0.016735298559069633,
+            0.022302847355604172,
+            0.005755309946835041,
+            0.002998067531734705,
+            0.02160078100860119,
+            -0.022221211344003677,
+            0.013012715615332127,
+            0.02037624642252922,
+            0.027609162032604218,
+            -0.01737205684185028,
+            -0.010906516574323177,
+            -0.004857318475842476,
+            -0.006853308994323015,
+            -0.00176945177372545,
+            -0.008253359235823154,
+            -0.012620864436030388,
+            0.0028735732194036245,
+            -0.030466407537460327,
+            0.018106777220964432,
+            -0.005102225113660097,
+            0.010669773444533348,
+            -0.019266001880168915,
+            0.011363675817847252,
+            -0.0035838030744343996,
+            0.011135096661746502,
+            0.0019612954929471016,
+            -0.030041903257369995,
+            -0.008571738377213478,
+            0.015388311818242073,
+            -0.0030368445441126823,
+            -0.013641309924423695,
+            0.02006603218615055,
+            0.00018942011229228228,
+            -0.022270193323493004,
+            -0.004314441699534655,
+            -0.006934944540262222,
+            0.002193956868723035,
+            -0.0029613315127789974,
+            -0.0052981507033109665,
+            -0.030188847333192825,
+            0.020033378154039383,
+            0.009559528902173042,
+            -0.03072764165699482,
+            -0.003255219664424658,
+            -0.014996460638940334,
+            -0.0025184585247188807,
+            0.008881953544914722,
+            -0.004000144544988871,
+            -0.015551582910120487,
+            0.01784554310142994,
+            0.022711025550961494,
+            -0.0023164101876318455,
+            -0.016057724133133888,
+            -0.0090697156265378,
+            -0.027494871988892555,
+            0.030825603753328323,
+            -0.0066696288995444775,
+            0.0022878379095345736,
+            -0.005322641227394342,
+            0.009388094767928123,
+            0.004228724166750908,
+            0.0157148540019989,
+            0.006081852596253157,
+            -0.002647034591063857,
+            0.026596879586577415,
+            -0.01964152604341507,
+            0.009567692875862122,
+            -1.588864870427642e-05,
+            -0.003308282932266593,
+            0.01890680566430092,
+            0.0007939062779769301,
+            0.021649762988090515,
+            -0.008571738377213478,
+            -0.0001475818717153743,
+            -0.0034388997592031956,
+            0.010898353531956673,
+            -0.0028450009413063526,
+            -0.003465431509539485,
+            -0.001531688030809164,
+            0.011118769645690918,
+            0.01013097818940878,
+            0.003804219188168645,
+            -0.01737205684185028,
+            0.003142970846965909,
+            -0.008547247387468815,
+            -0.007028825581073761,
+            0.01302904263138771,
+            -0.011192241683602333,
+            -0.0018990482203662395,
+            -0.03160930797457695,
+            0.016286302357912064,
+            0.0533243790268898,
+            0.008547247387468815,
+            -0.014433175325393677,
+            -0.0018806802108883858,
+            -0.019755816087126732,
+            0.01535565685480833,
+            0.011004479601979256,
+            -0.016816934570670128,
+            -0.011984107084572315,
+            -0.012310649268329144,
+            0.007649256382137537,
+            0.021306892856955528,
+            -0.006522684823721647,
+            -0.008579902350902557,
+            0.025372346863150597,
+            -0.011788181029260159,
+            -0.010882026515901089,
+            -0.01661284640431404,
+            -0.01635977439582348,
+            0.00241641397587955,
+            0.019902760162949562,
+            -0.0169475506991148,
+            0.021176276728510857,
+            0.00936360377818346,
+            0.0010133018950000405,
+            0.003624620847404003,
+            0.008588065393269062,
+            0.0024878450203686953,
+            0.017878197133541107,
+            0.003883813973516226,
+            -0.01439235731959343,
+            0.0021408938337117434,
+            -0.0030286810360848904,
+            0.004624656867235899,
+            3.4320437407586724e-06,
+            -0.002749078907072544,
+            -0.0065635028295218945,
+            0.003710338147357106,
+            -0.021306892856955528,
+            0.021502818912267685,
+            -0.022678371518850327,
+            -0.004245051648467779,
+            0.005555302836000919,
+            -0.005118552595376968,
+            0.013469874858856201,
+            0.018792515620589256,
+            0.01423724927008152,
+            0.008702355436980724,
+            0.01462093647569418,
+            -0.011331021785736084,
+            0.02323349379003048,
+            -0.013282112777233124,
+            0.0005816536722704768,
+            0.012939243577420712,
+            0.01755165494978428,
+            0.018792515620589256,
+            -0.006245123688131571,
+            -0.010384049266576767,
+            -0.0002946535241790116,
+            0.0129637336358428,
+            -0.022286519408226013,
+            -0.012269831262528896,
+            -0.01386172603815794,
+            0.011486129835247993,
+            -0.004743028432130814,
+            0.025453981012105942,
+            -0.00633492274209857,
+            -0.014653591439127922,
+            -0.015918942168354988,
+            0.014196431264281273,
+            -0.048752784729003906,
+            0.0030348035506904125,
+            -0.0024919267743825912,
+            0.03043375350534916,
+            -0.027021385729312897,
+            0.01738838478922844,
+            -0.0028490826953202486,
+            -0.009355440735816956,
+            0.015429128892719746,
+            -0.015992414206266403,
+            -0.0046613928861916065,
+            -0.0016806729836389422,
+            -0.007539048325270414,
+            -0.00112248957157135,
+            0.017616963014006615,
+            -0.015845470130443573,
+            0.01282495353370905,
+            -0.00989423505961895,
+            0.0027021386194974184,
+            -0.004526694305241108,
+            -0.007751300930976868,
+            0.0196088720113039,
+            0.034548189491033554,
+            -0.007053316105157137,
+            0.0014990337658673525,
+            -0.022547753527760506,
+            0.013192313723266125,
+            -0.006730855442583561,
+            0.02351105399429798,
+            -0.005432849284261465,
+            -0.016898570582270622,
+            -0.013976015150547028,
+            -0.009739127941429615,
+            0.00839214026927948,
+            -0.022433463484048843,
+            -0.011110605672001839,
+            -0.00922482367604971,
+            -0.002573562553152442,
+            -0.018694553524255753,
+            0.004865481983870268,
+            -0.002426618244498968,
+            -0.012506574392318726,
+            -0.008816645480692387,
+            0.01048201136291027,
+            -0.009347276762127876,
+            0.003077662317082286,
+            -0.0011929002357646823,
+            -0.011453475803136826,
+            -0.02369065210223198,
+            0.0158944521099329,
+            0.0001974561164388433,
+            -0.010359558276832104,
+            0.008408467285335064,
+            -0.0018276170594617724,
+            0.001211268245242536,
+            0.016963878646492958,
+            -0.0020664012990891933,
+            0.003467472270131111,
+            0.023478399962186813,
+            0.004057289566844702,
+            -0.010971825569868088,
+            -0.00025906547671183944,
+            0.013535182923078537,
+            -0.005118552595376968,
+            -0.010008525103330612,
+            1.6646010408294387e-05,
+            -0.02984597720205784,
+            -0.008351322263479233,
+            0.006094098091125488,
+            0.018629245460033417,
+            -0.010620792396366596,
+            -0.015853634104132652,
+            0.026319319382309914,
+            -0.006359413731843233,
+            0.019070077687501907,
+            0.002889900468289852,
+            0.0025694805663079023,
+            0.01724144071340561,
+            0.013461710885167122,
+            0.00786150898784399,
+            -0.013927034102380276,
+            -0.02429475635290146,
+            0.0016398552106693387,
+            -0.018090449273586273,
+            0.010579974390566349,
+            -0.011028970591723919,
+            0.0032756286673247814,
+            -0.02398454025387764,
+            -0.012237177230417728,
+            0.004934872500598431,
+            -0.020343592390418053,
+            -0.00040588201954960823,
+            -0.0056940834037959576,
+            0.022237539291381836,
+            -0.005310396198183298,
+            0.008669701404869556,
+            0.0018255761824548244,
+            -0.012237177230417728,
+            0.0009877908742055297,
+            0.009469729848206043,
+            -0.029192892834544182,
+            -0.0007127808639779687,
+            -0.003138889092952013,
+            0.0011418780777603388,
+            0.005351213738322258,
+            0.009755454957485199,
+            0.007322713732719421,
+            0.0027062203735113144,
+            0.007253323681652546,
+            0.010449357330799103,
+            -0.00022041612828616053,
+            0.0197884701192379,
+            0.018172085285186768,
+            -0.007351286243647337,
+            0.008710519410669804,
+            0.024817224591970444,
+            -0.013159659691154957,
+            -0.008718682453036308,
+            0.023266147822141647,
+            -0.006461458280682564,
+            -0.0011122850701212883,
+            -0.01483318954706192,
+            0.00815947912633419,
+            0.0017367975087836385,
+            0.028131630271673203,
+            0.019657853990793228,
+            0.005820618476718664,
+            0.000639819074422121,
+            0.015788326039910316,
+            -0.013020878657698631,
+            0.0035286990460008383,
+            0.015461783856153488,
+            -0.003940958995372057,
+            -0.019821124151349068,
+            0.013878053054213524,
+            0.009502384811639786,
+            -0.003228688146919012,
+            0.01551892887800932,
+            0.011412657797336578,
+            -0.030841931700706482,
+            -0.0011898389784619212,
+            0.010498338378965855,
+            -0.0021449755877256393,
+            -0.008751336485147476,
+            -0.00014273475972004235,
+            -0.012384121306240559,
+            0.021388528868556023,
+            -0.004408322740346193,
+            0.012277995236217976,
+            -0.00815947912633419,
+            -0.017127150669693947,
+            -0.011559601873159409,
+            0.007914572022855282,
+            -0.010547320358455181,
+            -0.0006714528426527977,
+            -0.004022594541311264,
+            -0.005779800936579704,
+            0.017747581005096436,
+            0.004706292413175106,
+            -0.0008964610169641674,
+            -0.0018398624379187822,
+            0.007983962073922157,
+            -0.006979844067245722,
+            -0.008726846426725388,
+            0.004816500470042229,
+            -0.012833117507398129,
+            0.003981776535511017,
+            0.005543057806789875,
+            0.01506176870316267,
+            -0.014857680536806583,
+            -0.021159948781132698,
+            0.007020662073045969,
+            0.02695607766509056,
+            -0.02516009286046028,
+            -0.005232842173427343,
+            0.006245123688131571,
+            0.002093953313305974,
+            0.006914535537362099,
+            -0.006934944540262222,
+            -0.007208424154669046,
+            -0.03657275065779686,
+            -0.0020745648071169853,
+            0.0009005427709780633,
+            0.008963589556515217,
+            -0.008049271069467068,
+            -0.0136494729667902,
+            -0.011135096661746502,
+            0.002410291228443384,
+            0.007134951651096344,
+            -0.02204161323606968,
+            0.0049593630246818066,
+            0.009608510881662369,
+            -0.019559891894459724,
+            0.025127438828349113,
+            -0.010890189558267593,
+            -6.964537897147238e-05,
+            -0.00995954405516386,
+            0.021633435040712357,
+            -0.013486201874911785,
+            -0.018368011340498924,
+            -0.0006898208521306515,
+            0.008355404250323772,
+            0.011926962062716484,
+            -0.0017663904000073671,
+            -0.011371839791536331,
+            0.0003752686607185751,
+            0.01646590232849121,
+            0.019135385751724243,
+            0.002744997153058648,
+            -0.0024837632663547993,
+            -0.01019628718495369,
+            -0.009592183865606785,
+            -0.0073961857706308365,
+            -0.0007449248805642128,
+            0.002091912319883704,
+            0.02035992033779621,
+            0.013469874858856201,
+            0.0024837632663547993,
+            -0.015135240741074085,
+            -0.005257333163172007,
+            0.008939098566770554,
+            0.009412584826350212,
+            -0.011690218932926655,
+            -0.00898808054625988,
+            0.002659279853105545,
+            0.02249877341091633,
+            0.017633290961384773,
+            -0.0014000505907461047,
+            0.0032878739293664694,
+            -0.0016592437168583274,
+            0.012506574392318726,
+            -0.007555375341325998,
+            -0.010269759222865105,
+            -0.006943108048290014,
+            -0.014645427465438843,
+            -0.010800390504300594,
+            -0.0023062059190124273,
+            -0.010179960168898106,
+            0.022629389539361,
+            0.007988044060766697,
+            0.009779945947229862,
+            0.008265605196356773,
+            0.010269759222865105,
+            0.002087830565869808,
+            0.025715215131640434,
+            -0.004424649756401777,
+            -0.011984107084572315,
+            0.007236996199935675,
+            -0.022368155419826508,
+            -0.021535472944378853,
+            0.008257441222667694,
+            0.006367577239871025,
+            0.03595232218503952,
+            -0.026890767738223076,
+            0.0052981507033109665,
+            -0.009779945947229862,
+            -0.010400376282632351,
+            0.007996207103133202,
+            -0.0015439334092661738,
+            -0.004245051648467779,
+            0.022368155419826508,
+            0.00726556871086359,
+            0.0035797213204205036,
+            0.0032817514147609472,
+            0.019837452098727226,
+            -0.005685919895768166,
+            0.01573934406042099,
+            0.0259274672716856,
+            0.024621298536658287,
+            0.009388094767928123,
+            -0.009469729848206043,
+            0.02785406820476055,
+            0.0002895512734539807,
+            -0.0034327772445976734,
+            0.007220669183880091,
+            -0.00305113079957664,
+            -0.0009556467994116247,
+            -0.010033016093075275,
+            0.021764053031802177,
+            -0.001455154619179666,
+            0.004359341226518154,
+            0.02081707864999771,
+            -7.774515688652173e-05,
+            0.006220633164048195,
+            0.01057181041687727,
+            0.002181711606681347,
+            0.003396041225641966,
+            0.010775899514555931,
+            -0.039315707981586456,
+            -0.0006571666453965008,
+            -0.025600925087928772,
+            -0.0018286375561729074,
+            0.0007357408758252859,
+            -0.01266984548419714,
+            0.013600491918623447,
+            -0.002015379024669528,
+            -0.010751409456133842,
+            -0.00029337796149775386,
+            0.021649762988090515,
+            0.005416522268205881,
+            0.004071575589478016,
+            -0.004469549283385277,
+            0.018792515620589256,
+            -0.006955353543162346,
+            -0.017861871048808098,
+            -0.012759645469486713,
+            -0.012367794290184975,
+            -0.01918436773121357,
+            -0.01214737817645073,
+            -0.0024551907554268837,
+            -0.002667443361133337,
+            0.004738946910947561,
+            0.002481722505763173,
+            -0.003228688146919012,
+            -0.02070278860628605,
+            0.015037278644740582,
+            -0.01430255826562643,
+            0.012873934581875801,
+            -0.011053460650146008,
+            0.005963481031358242,
+            -0.004122598096728325,
+            0.012726990506052971,
+            -0.021404854953289032,
+            0.02070278860628605,
+            -0.01536382082849741,
+            0.0038532004691660404,
+            0.019592545926570892,
+            0.006551257334649563,
+            -0.008118661120533943,
+            -0.013404566794633865,
+            -0.022074267268180847,
+            0.021633435040712357,
+            -0.011633073911070824,
+            -0.018237393349409103,
+            -0.004608329851180315,
+            0.009649328887462616,
+            -0.017470018938183784,
+            -0.007367613259702921,
+            0.01707816869020462,
+            -0.0018949664663523436,
+            0.012980061583220959,
+            0.023004913702607155,
+            -0.0052981507033109665,
+            0.0036572751123458147,
+            -0.009167678654193878,
+            0.0010286086471751332,
+            -0.02470293454825878,
+            0.013527019880712032,
+            0.008971752598881721,
+            0.006269614677876234,
+            -0.013935198076069355,
+            -0.013306603766977787,
+            0.0036531933583319187,
+            0.004498121794313192,
+            0.011437147855758667,
+            0.009420748800039291,
+            0.004220560658723116,
+            -0.013918871060013771,
+            -0.027788760140538216,
+            -0.027070365846157074,
+            0.03595232218503952,
+            -0.0005002731923013926,
+            -0.020947696641087532,
+            0.03399306535720825,
+            0.016114868223667145,
+            0.0022980421781539917,
+            -0.012449429370462894,
+            -0.004314441699534655,
+            0.024621298536658287,
+            0.016400592401623726,
+            0.010465684346854687,
+            -0.0004742518358398229,
+            0.005571629852056503,
+            0.0016469983384013176,
+            0.03012353740632534,
+            0.015323002822697163,
+            0.0009459525463171303,
+            -0.038597315549850464,
+            0.011943289078772068,
+            -0.007547211833298206,
+            0.0013408648082986474,
+            0.006702283397316933,
+            0.017780235037207603,
+            0.011935125105082989,
+            0.015567909926176071,
+            0.014865843579173088,
+            -0.003065417055040598,
+            0.0029511270113289356,
+            0.0006163488142192364,
+            -0.009624837897717953,
+            -0.001767410896718502,
+            -0.008881953544914722,
+            -0.005714492406696081,
+            -0.002920513739809394,
+            0.005592038854956627,
+            0.007657419890165329,
+            -0.003983817528933287,
+            0.028049994260072708,
+            -0.0006433906382881105,
+            -0.017600636929273605,
+            -0.009061552584171295,
+            -0.03611559048295021,
+            0.00434301421046257,
+            -0.030939893797039986,
+            -0.011257549747824669,
+            0.0033674687147140503,
+            -0.020457882434129715,
+            -0.024768242612481117,
+            0.0010306495241820812,
+            0.014335212297737598,
+            0.007392104249447584,
+            0.0033327736891806126,
+            -0.01004934310913086,
+            0.003816464450210333,
+            -0.001855169073678553,
+            -0.006461458280682564,
+            0.028833696618676186,
+            0.019559891894459724,
+            0.020457882434129715,
+            -0.01190247107297182,
+            0.019298657774925232,
+            0.005845109466463327,
+            0.02035992033779621,
+            0.010465684346854687,
+            -0.0048695639707148075,
+            0.0074859848245978355,
+            0.009567692875862122,
+            -0.006155324634164572,
+            -0.021519144997000694,
+            0.013192313723266125,
+            -0.0010837126756086946,
+            -0.0052083516493439674,
+            -0.016269976273179054,
+            -0.003534821793437004,
+            -0.015812816098332405,
+            0.010547320358455181,
+            -0.017159804701805115,
+            0.012498411349952221,
+            -0.000802069844212383,
+            0.00168475485406816,
+            -0.015469946898519993,
+            -0.013951525092124939,
+            -0.0012439225101843476,
+            -0.01244126632809639,
+            -0.008971752598881721,
+            0.006163488142192364,
+            -0.007539048325270414,
+            -0.008318668231368065,
+            -0.00535937724635005,
+            -0.011437147855758667,
+            -0.010882026515901089,
+            0.004367504734545946,
+            -0.0010653446661308408,
+            0.020931368693709373,
+            0.00011671340325847268,
+            -0.005824700463563204,
+            0.01190247107297182,
+            0.02787039615213871,
+            0.0016704685986042023,
+            0.004583839327096939,
+            -0.014261740259826183,
+            -0.011118769645690918,
+            -0.02279266156256199,
+            0.016572028398513794,
+            0.0049675265327095985,
+            -0.0344502255320549,
+            0.01438419334590435,
+            0.00824927818030119,
+            0.0015102586476132274,
+            -0.006698201410472393,
+            -0.005334886722266674,
+            -0.005608365871012211,
+            0.006306350696831942,
+            -0.0022837561555206776,
+            0.0005107327597215772,
+            0.0035919665824621916,
+            0.034678805619478226,
+            -0.0040674940682947636,
+            -0.007228832691907883,
+            0.00981259997934103,
+            0.0005130287609063089,
+            0.020196648314595222,
+            -0.005518566817045212,
+            0.0037572786677628756,
+            -0.019282329827547073,
+            -0.016563864424824715,
+            0.0018429238116368651,
+            -0.01722511276602745,
+            0.023282473906874657,
+            -0.005392031744122505,
+            -0.01347803883254528,
+            -0.006906372029334307,
+            -0.020115012302994728,
+            0.012637191452085972,
+            0.02310287579894066,
+            0.007016580086201429,
+            0.01033506728708744,
+            0.012131051160395145,
+            -0.02444170042872429,
+            -0.020457882434129715,
+            0.012171869166195393,
+            0.0013571919407695532,
+            -0.008449285291135311,
+            -0.005404277238994837,
+            -0.001783737912774086,
+            0.007783954963088036,
+            -0.0032633834052830935,
+            0.007881917990744114,
+            0.06491663306951523,
+            -0.01093100756406784,
+            -0.002987863030284643,
+            -0.02966637909412384,
+            -0.004396077245473862,
+            -0.006628811359405518,
+            0.0056287748739123344,
+            0.009151351638138294,
+            -0.023135529831051826,
+            0.009232986718416214,
+            -0.030025575309991837,
+            -0.010628955438733101,
+            0.00824111420661211,
+            0.0028490826953202486,
+            -0.01220452319830656,
+            -0.0009592183632776141,
+            -0.013665799982845783,
+            0.0006107363733462989,
+            -0.009412584826350212,
+            -0.00576347392052412,
+            0.012718827463686466,
+            -0.02321716584265232,
+            0.006298186723142862,
+            0.007592111360281706,
+            -0.006514521315693855,
+            -0.013722945004701614,
+            -0.017861871048808098,
+            -0.011967780068516731,
+            -0.013314766809344292,
+            0.002996026771143079,
+            0.01190247107297182,
+            -0.02607441134750843,
+            0.012922916561365128,
+            -0.0037389106582850218,
+            0.00945340283215046,
+            0.004404240753501654,
+            0.013469874858856201,
+            0.018074123188853264,
+            0.009355440735816956,
+            0.0021429345943033695,
+            -0.008024780079722404,
+            0.012702500447630882,
+            -0.01993541419506073,
+            -0.01635161228477955,
+            0.013837235048413277,
+            0.012016761116683483,
+            0.003061335301026702,
+            0.009828926995396614,
+            0.0020959940738976,
+            0.011886144056916237,
+            0.010775899514555931,
+            -0.019739490002393723,
+            -0.0006648199632763863,
+            0.0007326795603148639,
+            0.03280118480324745,
+            -0.0005938990507274866,
+            0.008857463486492634,
+            0.004420568235218525,
+            0.013159659691154957,
+            -0.012784135527908802,
+            -0.01769859902560711,
+            0.0008087027235887945,
+            -0.01057181041687727,
+            0.021257910877466202,
+            -0.008192133158445358,
+            0.006216551177203655,
+            -0.018612917512655258,
+            -0.027086693793535233,
+            0.006392067763954401,
+            -0.016261812299489975,
+            -0.018221067264676094,
+            0.01317598670721054,
+            0.0077268099412322044,
+            0.0017572063952684402,
+            -0.005098143592476845,
+            -0.020457882434129715,
+            -0.009837090037763119,
+            0.028441844508051872,
+            0.011102442629635334,
+            -0.004120557103306055,
+            0.009584019891917706,
+            -0.009094206616282463,
+            -0.012490247376263142,
+            -0.0017541450215503573,
+            0.0036715613678097725,
+            0.008261523209512234,
+            0.005645101889967918,
+            -0.01386172603815794,
+            -0.00029873527819290757,
+            -0.012400448322296143,
+            -0.021225256845355034,
+            -0.008906444534659386,
+            0.016751626506447792,
+            0.007820690982043743,
+            -0.0038960592355579138,
+            0.015812816098332405,
+            -0.004404240753501654,
+            0.002189875114709139,
+            0.014294394291937351,
+            0.005334886722266674,
+            -0.004285869188606739,
+            -0.0095758568495512,
+            -0.004612411838024855,
+            0.004043003544211388,
+            -0.0025960123166441917,
+            0.006457376293838024,
+            0.017192458733916283,
+            0.0072900597006082535,
+            -0.013755599036812782,
+            0.013135168701410294,
+            0.0007791098323650658,
+            0.0049675265327095985,
+            -0.03699725493788719,
+            -0.003247056156396866,
+            0.015796490013599396,
+            -0.013020878657698631,
+            0.02501314878463745,
+            0.005400195252150297,
+            0.022743679583072662,
+            0.015233203768730164,
+            0.008669701404869556,
+            -0.001542912912555039,
+            0.00464098434895277,
+            0.02759283408522606,
+            -0.003151134354993701,
+            -0.008637047372758389,
+            -0.0020602785516530275,
+            0.009241150692105293,
+            0.0002063850115519017,
+            0.003000108525156975,
+            0.008281932212412357,
+            -0.03562577813863754,
+            0.011706545948982239,
+            -0.019576217979192734,
+            0.0024694770108908415,
+            0.003459308762103319,
+            -0.007183933164924383,
+            -0.014865843579173088,
+            -0.005857354495674372,
+            0.022841641679406166,
+            -0.003224606392905116,
+            0.021078312769532204,
+            0.002885818714275956,
+            0.016106704249978065,
+            -0.011575928889214993,
+            -0.009208496659994125,
+            -0.0004130251472815871,
+            -0.005351213738322258,
+            -0.007539048325270414,
+            0.010596301406621933,
+            0.0034409407526254654,
+            0.005534893833100796,
+            0.002265388146042824,
+            0.0036654386203736067,
+            -0.0036878883838653564,
+            0.02773977816104889,
+            0.003563394071534276,
+            0.007996207103133202,
+            -0.011567764915525913,
+            0.014082142151892185,
+            -0.017029186710715294,
+            0.0070778070949018,
+            0.005575711838901043,
+            -0.026156047359108925,
+            0.016849588602781296,
+            0.004775682929903269,
+            0.006628811359405518,
+            0.013812744058668613,
+            0.01618834026157856,
+            -0.010408539324998856,
+            -0.014816862531006336,
+            0.024049850180745125,
+            -0.0032041973900049925,
+            -0.015225040726363659,
+            -0.006775755435228348,
+            4.438935866346583e-05,
+            0.006559420842677355,
+            0.00373074715025723,
+            0.0013020879123359919,
+            -0.0006204306264407933,
+            -0.010743245482444763,
+            0.004645065870136023,
+            -0.01934763789176941,
+            -0.0007163524278439581,
+            -0.016449574381113052,
+            -0.015633217990398407,
+            0.018678225576877594,
+            -0.01830270327627659,
+            0.009633000940084457,
+            0.0012275953777134418,
+            0.028588788583874702,
+            -0.011706545948982239,
+            -0.012155541218817234,
+            0.0015102586476132274,
+            0.010849371552467346,
+            0.0067512644454836845,
+            -0.010033016093075275,
+            -0.007583947852253914,
+            -0.014180104248225689,
+            -0.010163633152842522,
+            0.015502601861953735,
+            -0.014498483389616013,
+            -0.014343376271426678,
+            0.008914608508348465
+          ],
+          "embedding": [
+            -0.006359657085039302,
+            -0.024594786241399234,
+            -0.02553538924848095,
+            0.005360188465088291,
+            0.012801870977589839,
+            0.021680505862506104,
+            -0.016668048966001273,
+            0.06860852474581301,
+            -0.007122272841157724,
+            0.03471775000705488,
+            -0.009004297848861675,
+            0.01059974144678391,
+            0.02565604599426148,
+            -0.022544127646041287,
+            0.010616674976540785,
+            0.013658194823359737,
+            0.016400329743363844,
+            0.02858546964917478,
+            0.0067612676434693686,
+            -0.04174737193606879,
+            0.03184155110488616,
+            -0.02223105127585514,
+            -0.05506239667974977,
+            -0.0015307551410555752,
+            -0.013169788751379287,
+            0.017681749722384007,
+            0.01682706317229353,
+            0.019652295823816928,
+            -0.05623980369271269,
+            0.030698582851533197,
+            0.02286327124296613,
+            0.016019032539689178,
+            0.005508113353872315,
+            8.58985091215149e-05,
+            -0.019206292853495906,
+            -0.018439290483495144,
+            0.03475023176950151,
+            0.01018612942141759,
+            -0.0252551158857426,
+            0.0334562130666827,
+            0.020462153537485785,
+            0.02260508249714211,
+            -0.03323524989392951,
+            -0.002094251009530707,
+            0.015177442347686773,
+            -0.00018698464350038317,
+            -0.0006788648344273403,
+            0.0007777616893581039,
+            0.013352398615059647,
+            -0.030751111528744813,
+            -0.0317733818445192,
+            -0.006729064081710058,
+            0.009413204899986514,
+            -0.005712894407943592,
+            -0.038961826880527066,
+            -0.002827692206530345,
+            0.017863300537973137,
+            0.07918455388563496,
+            0.048967158919816366,
+            0.018002566325350088,
+            0.040264962620950695,
+            -0.019715100359659787,
+            -0.02211350500879408,
+            0.004918148608475574,
+            0.01062585305866606,
+            -0.014478993521696027,
+            0.0034983131226364672,
+            0.000766228629514787,
+            0.009667268771502067,
+            -0.01558439890697323,
+            -0.04984718709870083,
+            0.029613955432821887,
+            -0.009686854114630624,
+            -0.015279319238124568,
+            0.007460629676325322,
+            -0.006235979350809034,
+            0.012514387473717085,
+            0.02800435263741497,
+            -0.008688925946672778,
+            -0.01236301543374469,
+            0.029755705020188972,
+            0.01158037219446542,
+            -0.005242392206024511,
+            0.009982679027728814,
+            0.052414361714483264,
+            -0.018788500381726227,
+            -0.020335831196027204,
+            -0.08583959943356621,
+            0.029309228216812346,
+            -0.0249285282010673,
+            -0.018929396555426307,
+            0.007710663193334574,
+            0.03096798035958051,
+            -0.017089977064591828,
+            -0.028015107605405817,
+            0.06740703271128606,
+            0.009398795412354399,
+            0.01118769022826835,
+            -0.03041417460855722,
+            0.014776002883150835,
+            0.004319194067673161,
+            0.022507636639041203,
+            0.0053562788646786845,
+            0.007548072258943124,
+            0.011123374308563128,
+            -0.006924362115585165,
+            -0.011118507147092024,
+            0.009563232643295304,
+            0.00657178473678069,
+            0.08313027447380966,
+            -0.004239370156003264,
+            -0.00930406163946907,
+            -0.039202672821898946,
+            0.009761029930556298,
+            -0.02827698465424993,
+            -0.037914533246900244,
+            -0.04496050911855815,
+            -0.017711307900236353,
+            0.01934435542287117,
+            0.004335173027873153,
+            0.03800509969714242,
+            0.031198127357716314,
+            0.031311207941680276,
+            -0.012184553894588813,
+            -0.058985863670856876,
+            0.041442017057707635,
+            -0.012456189292721711,
+            -0.0040966351720394135,
+            0.010849179519959421,
+            0.005243123049347636,
+            0.0011495799963862647,
+            0.003534183676934487,
+            -0.03139340512003631,
+            -0.023876261883905306,
+            0.008382957569752911,
+            -0.004778283217011976,
+            0.03703067589888386,
+            0.06888630688839406,
+            -0.03993532296478411,
+            -0.009472167678222372,
+            0.025246872942105057,
+            -0.01524567912700368,
+            0.01595917939878582,
+            -0.003736723164442467,
+            -0.030836797784967753,
+            0.0026246446163368533,
+            0.008894300082148884,
+            0.08342856279154155,
+            -0.0007476580521704578,
+            0.025149116238550588,
+            0.004972542633631468,
+            0.034495632546556576,
+            -0.009173329764686424,
+            0.014009381543772768,
+            0.004427261668151028,
+            -0.013426138607902904,
+            -0.022234568880321808,
+            0.01976282364689966,
+            -0.03592433789490043,
+            -0.010918278220831188,
+            0.013527180289641174,
+            -0.013023113052242677,
+            -0.008304770302495363,
+            0.003677063799167178,
+            0.0013981822663219421,
+            0.047001163242600175,
+            0.01828332714887986,
+            0.002483375662343689,
+            -0.06820498187523893,
+            0.05295312038995555,
+            0.025038753504342162,
+            0.025286870198436735,
+            -0.0029544808216316115,
+            0.029982752196635428,
+            0.0005600481523874767,
+            0.0128290258663539,
+            0.009023805227424919,
+            -0.0019219166367750596,
+            -0.04482843936478531,
+            0.04523479691409549,
+            -0.004678574027241186,
+            0.010486850804218904,
+            0.006736229865018014,
+            0.02134813656476443,
+            0.028083972673690036,
+            0.016857413440125445,
+            0.0189082447820146,
+            0.03843690613820022,
+            -0.011922027011229613,
+            0.016193433524453224,
+            0.03828207911467065,
+            -0.0013055128404331705,
+            0.043198163443700956,
+            0.024025562893260542,
+            -0.001257851946749882,
+            -0.011726713433348844,
+            -0.021907984099443396,
+            0.02752198420597673,
+            0.004431866336999851,
+            0.015798806714413912,
+            -0.012910782191382367,
+            -0.03180549180206364,
+            0.004550247372715741,
+            -0.022778272841118514,
+            0.031242218240017532,
+            -0.006106984949693483,
+            -0.006259993190497421,
+            0.02187449047033339,
+            0.009334146847221406,
+            -0.009381278960016145,
+            -0.020888853823931134,
+            -0.007502374908776156,
+            -0.010390018386728482,
+            -0.04316693435221676,
+            -0.006274513730720055,
+            0.019152443937064258,
+            -0.01668984824845791,
+            0.018573194036505578,
+            0.0035035262183249925,
+            -0.019779911381789077,
+            -0.0027495798165085993,
+            0.02425481236315913,
+            0.006857723545270935,
+            -0.014733535699087731,
+            -0.026326146471941727,
+            -0.019906660231056833,
+            -0.001033294559612807,
+            0.0159154063026895,
+            0.035649534442431415,
+            -0.026210592096240003,
+            0.014397127229517285,
+            0.005114311681443673,
+            -0.0033104236005207077,
+            0.00324988235167305,
+            0.03764004154553922,
+            -0.017822437760737605,
+            -0.008579701063708334,
+            -0.00032410984200965487,
+            0.018450351071676443,
+            0.017283379764846195,
+            0.006384411611332382,
+            -0.005115090309123598,
+            -0.030590047062583125,
+            -0.011776211359193503,
+            0.01183623914060493,
+            0.0016564847796208331,
+            -0.009530766634361575,
+            -0.004810598258154945,
+            0.03876242515526699,
+            0.027076785774401667,
+            0.005324763136165989,
+            -0.03416412480913939,
+            -0.007943016733586697,
+            0.002114307650456928,
+            -0.02961449347300628,
+            0.05142732770518828,
+            0.05566026433842563,
+            -0.0030729309664974704,
+            -0.00938186948393074,
+            -0.04547908981236149,
+            0.0651216359692231,
+            -0.01813722445944094,
+            -0.0482650853560993,
+            0.044045155892545554,
+            -0.014272653003441165,
+            0.013740652306746851,
+            0.04563732439184395,
+            0.02126513340521316,
+            -0.0054845632439384905,
+            -0.011635625101262898,
+            0.030992082069685376,
+            -0.010051935721068779,
+            -0.050967258971854425,
+            -0.009827942095033494,
+            0.013373878556654464,
+            -0.013017627321534534,
+            -0.009361001542607812,
+            -0.01925867665425919,
+            0.020329052915436022,
+            0.023020741670660188,
+            0.019387043947695172,
+            0.005628632318152202,
+            0.018451555161952665,
+            0.016226845400848455,
+            -0.01384033112836216,
+            0.004762589697426251,
+            -0.012718476561725138,
+            -0.024222765127494463,
+            -0.037487144314010595,
+            -0.019971524166659278,
+            -0.015034588610516958,
+            0.009483936496215338,
+            -0.014301924283553591,
+            -0.0024139665139412125,
+            0.013361045353529458,
+            -0.0005870798721107379,
+            -0.011192319340879789,
+            0.013927695434391159,
+            0.021337539669342293,
+            -0.027300952495815437,
+            -0.04765670695199488,
+            -0.00561706224251562,
+            0.0082776639906139,
+            -0.0030984973749633724,
+            -0.026269016057688548,
+            0.007040524859505573,
+            0.0017859008072463442,
+            0.020091542517503808,
+            -0.018716761130867725,
+            0.0035283051960672963,
+            0.008519383548064053,
+            -0.015247417050336866,
+            -0.016917626868166634,
+            -0.018580550690103854,
+            -0.024697180580294884,
+            0.034202028316737125,
+            0.0204395003463227,
+            0.030392895534267717,
+            -0.04527669147753553,
+            -0.018359428092075348,
+            0.05494107499558624,
+            0.015234754907198422,
+            -0.05066008106884831,
+            0.031139824660661085,
+            0.005016615710206006,
+            0.0019000453624929544,
+            -0.006094364589229031,
+            0.0637324238017742,
+            -0.02711381489292883,
+            -0.02187418906973867,
+            0.004601744864532875,
+            -0.04141121277496072,
+            -0.026689585608019588,
+            -0.005568100750837504,
+            0.01291019036002459,
+            -0.031853041426355644,
+            -0.02815800259372657,
+            -0.007964861641790374,
+            -0.010933515363060793,
+            0.013726639077745378,
+            -0.023956800669647334,
+            -0.015672058590455337,
+            -0.023642025374517057,
+            -0.01288435863544158,
+            -0.0042516881552192385,
+            -0.011162214160460695,
+            -0.02412733256710621,
+            0.010088528028211243,
+            0.03310175458095482,
+            0.023064360903748363,
+            0.001996159684818467,
+            0.009478229572933101,
+            0.014478772877286076,
+            0.00371783700962415,
+            0.007864103610320116,
+            -0.048511515124609,
+            -0.031398754206534776,
+            0.020227872561659397,
+            0.012159811587094092,
+            0.0172163448580163,
+            0.009578952796095119,
+            -0.04154545961112031,
+            -0.05095080393682759,
+            0.02968486144855133,
+            -0.03932395023720674,
+            -0.02063219739061335,
+            0.007512423249865194,
+            -0.011911709468952548,
+            -0.00810951734088105,
+            -0.0008079327452253181,
+            -0.01600701450676789,
+            0.023984909975645632,
+            -0.02418072541081482,
+            0.0033984265728439398,
+            -0.0104343782811971,
+            0.01588977364765743,
+            0.05109529246167992,
+            -0.027800269136366997,
+            -0.027889034481533438,
+            0.03706813210665186,
+            -0.02058120652429308,
+            0.030801396468451574,
+            0.03246127847621613,
+            0.008355517405358826,
+            -0.008605467586247932,
+            0.017693344563968573,
+            0.014499487024471321,
+            0.012431277080478217,
+            -0.040103870071440256,
+            0.017328848567062113,
+            0.02723900253727976,
+            -0.054909074068976875,
+            -0.008547596270650966,
+            -0.00777964838164648,
+            0.027134666572242665,
+            0.02765036594970255,
+            -0.002207173084941375,
+            -0.016838740117653725,
+            -0.028289233739722936,
+            -0.019433376533829132,
+            -0.011053380458946764,
+            0.03247641283550457,
+            0.0007514989009289282,
+            0.007919574219000797,
+            0.001316042436488591,
+            -0.0017934342041762042,
+            0.01924145734197619,
+            0.015260199553572459,
+            0.048259602293134723,
+            -0.03383050622747848,
+            0.0021712225589426266,
+            -0.03508567715696004,
+            -0.04578731401250471,
+            -0.008768228935292693,
+            0.010203318819548585,
+            0.024135236371743855,
+            -0.007467140460133838,
+            0.0028907962385387818,
+            -0.007582083854422456,
+            -0.005384477718912037,
+            -0.01867978482547249,
+            0.01641191108381631,
+            0.037168089880543244,
+            0.0031992495663980464,
+            -0.008533890994695162,
+            0.000778843942291118,
+            -0.004620258538699103,
+            0.010572690335163528,
+            0.033233014627100904,
+            -0.030497362482602697,
+            0.02974571152880676,
+            -0.004274442492343003,
+            0.025341347125910928,
+            -0.04307748954707389,
+            -0.018069609916024403,
+            -0.018491122727274337,
+            -0.02069307802429807,
+            0.05372967226264594,
+            -0.030768978301976772,
+            -0.03662045886993282,
+            -0.003205991669059688,
+            0.016815511161004534,
+            -0.0014152644009276566,
+            -0.0016015439520090104,
+            -0.03984716970293867,
+            -0.010828260334550217,
+            -0.05894985435251326,
+            -0.01638035185618134,
+            -0.021819442096404948,
+            -0.005315102328745982,
+            -0.06251579337157112,
+            0.0033713876515840233,
+            -0.07815974854883057,
+            0.001484860071649606,
+            0.04539427129131382,
+            -0.023530943197353476,
+            0.05142250021536681,
+            -0.024746600229063383,
+            -0.0005885740455257141,
+            -0.044698536284421976,
+            0.0026481001040031785,
+            0.02389178603305878,
+            0.051387657872255674,
+            -0.002949198236157656,
+            0.010674377196609455,
+            0.013322075519401089,
+            -0.026245090721950647,
+            -0.01036724558050635,
+            -0.005321884180696542,
+            0.005958798671856245,
+            0.013755178085972732,
+            -0.0074734329537091675,
+            0.038368016046563674,
+            0.010941421833174982,
+            0.03443115781919206,
+            0.034623617136788874,
+            -0.004361926531008214,
+            -0.03348681614127998,
+            -0.00632870804659212,
+            0.024433331840566692,
+            0.0321998807702602,
+            -0.02899983280320611,
+            0.005958471722097962,
+            -0.015602447981516634,
+            0.015633217539747047,
+            0.006321931116948646,
+            -0.011827541320878442,
+            0.02372204564999124,
+            0.015673426811623124,
+            0.006103300725621424,
+            -0.005165006670545588,
+            0.013589526056189892,
+            -0.025411864292353578,
+            0.03041791947375232,
+            -0.004456865298584878,
+            -0.04276058054583262,
+            -0.0024736078733106825,
+            -0.020059309867945227,
+            0.024883721794000956,
+            0.015578237413840115,
+            -0.00536523508503184,
+            -0.005290645798641171,
+            0.008242987948511412,
+            0.020469675635821673,
+            0.006668354647245447,
+            0.00017729517736814364,
+            0.02373604785121342,
+            0.029915687122900485,
+            -0.02432635861789771,
+            -0.029540533943782886,
+            0.008151297444985344,
+            -0.005730733861939129,
+            0.011318816579801282,
+            -0.02644855229221747,
+            0.00033932872948409137,
+            0.0035505684154028996,
+            -0.03570104343808711,
+            0.03408366016010711,
+            -0.009199700348630527,
+            0.005857598530788056,
+            -0.009538177736472163,
+            -0.0017768806148210118,
+            0.0027410153099351694,
+            0.03898108970435115,
+            -0.007367623392751247,
+            0.009827254226889936,
+            0.015465895035342481,
+            0.000981604396724216,
+            0.05143083558797524,
+            -0.027350877485564314,
+            -0.02183036962301852,
+            0.01885468378168636,
+            -0.007285239007489837,
+            -0.016846837665812908,
+            -0.020546383007536074,
+            -0.011863943814684079,
+            -0.016293192750336218,
+            0.01976516340568883,
+            -0.02387328798681798,
+            -0.011892336568452852,
+            0.007170651391639504,
+            0.0013725014050721622,
+            -0.009809669068563384,
+            0.0016617183704652932,
+            -0.006307590349798603,
+            0.02037693835615797,
+            0.031753656615094446,
+            -0.024892675105483313,
+            0.03577919637821846,
+            0.03178303188315581,
+            0.017260898395251155,
+            -0.02102289690701718,
+            -0.009661035772073294,
+            -0.00048542689442823136,
+            -0.014694631967051945,
+            -0.0326379186158631,
+            0.025443206465204822,
+            0.017194626456982255,
+            -0.011381421003142453,
+            -0.03680783761649673,
+            0.0017216115811536721,
+            0.022819859205829785,
+            -0.013600989058799166,
+            0.028141657788496834,
+            -0.01869047298692401,
+            -0.01366833755869951,
+            -0.00782959271979894,
+            0.031227914441058244,
+            -0.05252392878125891,
+            0.016957842026348195,
+            0.00020880696574616416,
+            0.001154370575234767,
+            0.010892067708194324,
+            -0.008368819756628046,
+            0.012224564683325483,
+            0.031700389750709,
+            0.018063405869147564,
+            0.0010794115307202266,
+            -0.002783668216641561,
+            0.01418234057248712,
+            0.017773269840982066,
+            -0.015653443355569053,
+            -0.000497740485657824,
+            0.019890882604100452,
+            0.022672054781919238,
+            -0.00521797370595221,
+            -0.016686942830774276,
+            -0.018443538351123584,
+            0.025270057639201107,
+            -0.010354431070738671,
+            0.03484047753830186,
+            0.04594785214094975,
+            0.0036441959528283004,
+            -0.05695476763071743,
+            -0.009412209619190693,
+            0.0010371110229966086,
+            -0.01099343927053564,
+            0.01606863921935641,
+            0.012470660511773616,
+            -0.03237187923799263,
+            0.006388672213167589,
+            0.012946570716567364,
+            0.022152554488829627,
+            -0.02540317621005867,
+            -0.023183252921885226,
+            -0.0030984019855787655,
+            -0.007890467768631811,
+            0.018648760402624687,
+            -0.008653405980661855,
+            0.0006228315253506687,
+            0.009672526897422602,
+            0.0005436494014569835,
+            0.011632326145022935,
+            0.005655942007108228,
+            0.021736827172951,
+            -0.001256608167230203,
+            0.013124413886287987,
+            -0.01986808920858981,
+            -0.0270450339809516,
+            0.006500423802582728,
+            0.04438242119211055,
+            -0.014813399694814117,
+            0.030621709833336064,
+            0.0092171224268983,
+            0.016433384739642534,
+            0.024875072913970443,
+            -0.00891470467241686,
+            -0.014124009014664287,
+            0.00845777541223868,
+            0.009019007848927453,
+            -0.04147358041382793,
+            0.012513030150663913,
+            -0.01710243399313324,
+            -0.033144939719206276,
+            0.01904674924133127,
+            -0.00534122554342279,
+            0.012367320751926314,
+            0.004270315800911569,
+            -0.002886460711688153,
+            -0.025696286516581048,
+            0.014137901708875033,
+            0.028135087538950946,
+            0.021797869775098042,
+            0.0296173052503884,
+            -0.012757078368361444,
+            0.0005460601780756187,
+            -0.008748930817524899,
+            0.005470975782187157,
+            0.04187083128854477,
+            0.0008687182830616019,
+            -0.0008032802520818105,
+            0.01507726552487313,
+            0.024945744517309208,
+            0.0376130540370386,
+            -0.015415560911023553,
+            -0.03407353556457147,
+            0.01564969972264396,
+            -0.010104443707865262,
+            0.018855792875436674,
+            0.005503259932317811,
+            0.0049634097823351596,
+            0.0038600156268000702,
+            0.019312419743287703,
+            -0.006635193789316492,
+            0.018798045806429395,
+            -0.02500983683871332,
+            -0.028785757874583755,
+            0.01829411751138161,
+            -0.005277972443258226,
+            0.03338623299668019,
+            0.016225885095995672,
+            -0.029522280454919586,
+            -0.01712097344863975,
+            0.01553706187924322,
+            -0.008465766956343768,
+            -0.024842700926275538,
+            0.01895228828512998,
+            0.007076862934523977,
+            0.008956105593078965,
+            0.0024373779389833218,
+            -0.01106503696703573,
+            0.01125466547367417,
+            -0.000818223040067914,
+            -0.015322790791698395,
+            0.014157131555941659,
+            -0.017853386731281715,
+            -0.014994640991761465,
+            0.01775991780643474,
+            -0.037498078532902435,
+            0.009570197712419421,
+            -0.007687297543935123,
+            0.009050494195378254,
+            -0.004222041469820445,
+            0.007460383234429904,
+            -0.0010804379186060264,
+            -0.014987857934522812,
+            -0.014903749199810126,
+            0.03228485430019625,
+            0.022096121825475306,
+            -0.000410816936759558,
+            0.006728733918931589,
+            -3.971636461500232e-05,
+            -0.026719848032785824,
+            -0.011253751790407687,
+            0.02245467425830765,
+            -0.005664635254398481,
+            0.005134496095445943,
+            -0.01109590855846966,
+            -0.0149616794609307,
+            0.0019726017255571673,
+            0.032668637270108894,
+            0.06066635618119076,
+            0.012001732730659747,
+            0.024422093585231484,
+            0.0189499396761026,
+            0.02119276033547528,
+            -0.008489818917377516,
+            0.006413901834269739,
+            -0.03097413287789443,
+            0.03509225573986143,
+            0.005392594635384506,
+            0.03108937318306421,
+            0.026664331248813712,
+            -0.04722525921137681,
+            -0.0058539171830594774,
+            0.017087531454837677,
+            0.022673712717885323,
+            0.0159020681868943,
+            0.0008382498574190522,
+            -0.008938174268517353,
+            0.035650940044285984,
+            -0.03315250299348084,
+            -0.023647111263155,
+            0.005059382024092534,
+            -0.01439537467317451,
+            -0.010488359956731476,
+            -0.006785295580877329,
+            0.02776449681317373,
+            -0.005584052419598926,
+            -0.004582019433008779,
+            0.009697064252638535,
+            -0.014400206558190682,
+            0.008165676294736836,
+            0.0007624778531368436,
+            0.008681263897172326,
+            0.039249866981334126,
+            0.0005918063050375113,
+            0.024275699182767528,
+            -0.004138313072843555,
+            0.011079529739650594,
+            -0.0158972266904151,
+            -0.016985058543605,
+            0.02276982363172991,
+            0.00351101120496393,
+            -0.03244126813432563,
+            -0.00768819222024939,
+            0.00928180979884261,
+            -0.0013793404539641824,
+            0.005266058566444064,
+            -0.00503433670609399,
+            0.003747448261046857,
+            0.03286230666764075,
+            0.0022383592721685153,
+            0.008606190020480513,
+            0.013593947677257625,
+            0.00030951915677929204,
+            -0.01490307680740127,
+            0.013254591138113545,
+            -2.1379308136707722e-05,
+            -0.012900073369514622,
+            0.03357519186606058,
+            0.006705902094864628,
+            0.04243347200485736,
+            -0.03302953674711988,
+            0.009031534935013593,
+            0.01822227379789576,
+            -0.008491770335368761,
+            -0.02725412856390648,
+            -0.03630788717786838,
+            0.008259145747740535,
+            -0.019003260334340084,
+            0.015781275447792516,
+            -0.010039277287105153,
+            0.029665124310477384,
+            -0.03563370794521629,
+            0.004714858110268169,
+            0.004053913355055313,
+            0.0009503673086552651,
+            0.02068360255395961,
+            0.018524093113914366,
+            -0.002785698934204024,
+            -0.031443835341186485,
+            -0.02066365433385339,
+            0.006869321907853112,
+            0.02431260052207595,
+            -0.010367048093621498,
+            -0.005612149485859478,
+            0.0006316058139530722,
+            -0.024369562712568606,
+            -0.016845124984022828,
+            0.030359419843284823,
+            -0.040974427176153246,
+            0.005252911449822132,
+            0.04272936053727608,
+            -0.04599092715016919,
+            -0.03802288716943514,
+            -0.002314988728575789,
+            -0.024124014894379466,
+            -0.006360771882090754,
+            -0.002068030981218464,
+            0.016669991113175926,
+            0.0036232707799735537,
+            -0.028763764013482795,
+            0.00943401978066451,
+            -0.005173988601101547,
+            0.003971420424839974,
+            -0.0031669592494977393,
+            -0.02216336363004636,
+            -0.01657772032717298,
+            -0.02531962163186398,
+            -0.035174102848865116,
+            -0.0015903434876311088,
+            -0.005158703150821607,
+            -0.028703075279214722,
+            -0.03941940849540492,
+            -0.021696000925262537,
+            -0.0160221488026189,
+            -0.00843213135542359,
+            0.040641882367348774,
+            0.021319060666253945,
+            0.019336844555345886,
+            0.0025252167889382917,
+            -0.020346900609125973,
+            0.049440028814233025,
+            -0.007642182460707816,
+            0.035628691886108926,
+            -0.0017882888000877158,
+            -0.013461669667950893,
+            0.00455558389906464,
+            0.03825428026120858,
+            -0.014248434654950971,
+            -0.0009902062564027718,
+            0.00027811655483232545,
+            -0.0011866253757890136,
+            0.0008535155950871286,
+            0.015433650396777696,
+            0.001307651096343185,
+            -0.007112199543314309,
+            0.030380673552683906,
+            -0.012803247194895328,
+            0.01637501752192592,
+            0.018350814869962304,
+            -0.0013359719137099667,
+            0.008064546833360553,
+            -0.021430659268086814,
+            -0.00731572118780415,
+            0.0044578576331236325,
+            0.008422210247967642,
+            -0.015096885054883586,
+            -0.009964588397149773,
+            -0.02359494467299634,
+            -0.01174145797396426,
+            0.0023855749166744366,
+            -0.004559202896782688,
+            0.004902138778592298,
+            0.034595817216889944,
+            -0.016068744500427584,
+            -0.035106033294403614,
+            0.009997207702122002,
+            -0.02663088875167486,
+            0.02222306406130687,
+            3.983384942889326e-05,
+            0.006838221769859366,
+            -0.0023434342406307616,
+            0.001765445386886839,
+            -0.00169484278667461,
+            0.002862331185439823,
+            -0.018217975204395134,
+            0.0031859734246840257,
+            -0.0032119988065673565,
+            0.0009052950782312657,
+            -0.0012459653052875638,
+            0.00687339892509406,
+            0.006332799551164718,
+            -0.0008019538898592668,
+            -0.002968451740267742,
+            -8.541086355097327e-06,
+            -0.020561816931299912,
+            -0.021642512405352372,
+            -0.028259049261410905,
+            -0.000989887278779396,
+            -0.012359733017834338,
+            -0.027804394906433664,
+            0.011443385424045703,
+            -0.006872684999639775,
+            0.008327844771213648,
+            -0.008969527138034904,
+            0.006219994886600349,
+            -0.008415106130309062,
+            -0.0004143996206881767,
+            0.013119286511794273,
+            -0.02925209282917943,
+            -0.02096907906398536,
+            -0.03096513357712367,
+            0.01762223841790055,
+            -0.00925741670612824,
+            -0.002656382260603827,
+            0.0025223527960181835,
+            0.005092659166535706,
+            -0.033606542487570154,
+            -0.006653438998497412,
+            0.007313682256676618,
+            0.009547290751154595,
+            -0.03287788897629887,
+            0.02446728061355753,
+            0.017023222076526946,
+            0.01874233066565012,
+            -0.02417566312664911,
+            0.04049827067897005,
+            0.014426879325647042,
+            -0.021740606525480655,
+            0.016712907564723732,
+            0.027385150577780248,
+            0.03177163221499527,
+            -0.03469098547555545,
+            0.004481376248622418,
+            -0.01518200063132508,
+            0.018918680020516044,
+            -0.025284921244460287,
+            0.020625170445627613,
+            -0.016707629602243063,
+            0.021278719363178687,
+            0.03520334001264928,
+            -0.005524369131116028,
+            0.03608133014180745,
+            -0.019234548780647943,
+            0.0059985801736444555,
+            0.004041778461022664,
+            -0.01645503683802124,
+            0.01024694092614158,
+            -0.0042700351372843386,
+            0.0006059159967286885,
+            0.009406090340935747,
+            0.019133229156115282,
+            0.019007219188041775,
+            -0.04442208547053466,
+            0.028588965123434923,
+            0.00804120101594811,
+            0.021472086070876265,
+            0.014177820346835521,
+            -0.02708807970982664,
+            0.004742981711172693,
+            0.002072673190279628,
+            -0.004659713164250794,
+            0.0061167904730676514,
+            -0.013796604518930328,
+            -0.034540857808309226,
+            0.006338792319814448,
+            -0.018918703778187926,
+            0.015036937364271984,
+            -0.009026670935442384,
+            0.0011667525636770179,
+            -0.01152652811400482,
+            -0.011938679695598347,
+            0.0006170592505446954,
+            -0.005495151258143897,
+            -0.0008075736908949407,
+            -0.03171721592827769,
+            -0.04427658539412448,
+            0.02036132271908089,
+            -0.0021221965958423804,
+            -0.04320396743189005,
+            0.004111179168317842,
+            -0.025352428814134187,
+            -0.01865918186335911,
+            -0.011237699157960728,
+            -0.025047923887906474,
+            -0.04219630696740606,
+            -0.011108496740166623,
+            0.007785006066683012,
+            0.016810642610918525,
+            -0.009081815651299525,
+            0.040965574143584864,
+            0.0073944364393447965,
+            0.015313576815177263,
+            -0.017731360025694955,
+            0.026137777790243826,
+            -0.019314392060123,
+            -0.03108486330987073,
+            0.02676829566273746,
+            -6.259161089416372e-05,
+            0.003053748984540415,
+            0.014358356168574803,
+            2.8186924071913628e-05,
+            -0.006181789640967465,
+            0.01947730374003864,
+            -0.028739300217145797,
+            -0.023460503178221397,
+            0.0012460072421725688,
+            -0.008850395570726385,
+            -0.016271242375881602,
+            -0.035896509136022194,
+            0.005227881053956234,
+            0.03324337105109496,
+            0.004970419987430122,
+            0.008273497228454482,
+            0.01814500594735864,
+            -0.02229593571615308,
+            -0.024230675455380678,
+            0.000639112529814716,
+            -0.010266640429960994,
+            -0.010257247303470066,
+            -0.021302483296731352,
+            0.019150757066356872,
+            0.01651375496287087,
+            -0.00390189952706486,
+            -0.006971692709239087,
+            -0.010621003777733253,
+            -0.01918206725910585,
+            -0.02020345113211036,
+            -0.013827322378214754,
+            -0.017980206764311026,
+            -0.034153741666571154,
+            -0.032195001867229935,
+            0.03225205339201753,
+            0.026354817588919596,
+            0.0014076409241492998,
+            -0.007959081015805435,
+            -0.023070419006792155,
+            -0.004483328183581872,
+            0.02879516595345542,
+            0.002549934782150634,
+            -0.0001541849716285838,
+            -0.00830090759167373,
+            0.015341095409025125,
+            0.00835695019244165,
+            0.005107844984733648,
+            -0.06676974053142512,
+            -0.006027540973808174,
+            0.01655283585235268,
+            0.01351155740684989,
+            -0.002359114145814062,
+            0.02591301169926259,
+            0.01108099264396926,
+            -0.015361038608114896,
+            -0.019765743462262668,
+            0.035195761515422364,
+            0.02408910733949848,
+            0.023468767772473746,
+            -0.0034492999385544727,
+            -0.004472311490618405,
+            0.007030833258457338,
+            -0.010684252427867153,
+            -0.013326964647936429,
+            -0.004558513842504658,
+            0.026925146859337397,
+            0.016099443021514896,
+            -0.0064331198280825575,
+            0.010848931851939124,
+            -0.0008320209125061849,
+            -0.019444491995041537,
+            -0.001841330829156697,
+            0.008730275423516445,
+            0.0012974082030866743,
+            -0.014611902493400719,
+            -0.005911807768916134,
+            -0.019130604101278263,
+            0.024719468317569562,
+            0.0209456329723249,
+            0.03381727436368338,
+            -0.013237902591972769,
+            -0.0003695142331921278,
+            0.005410360755058108,
+            0.0002841632253252816,
+            0.0023352645111704958,
+            0.004801123870325242,
+            0.02623649813035263,
+            -0.029010710171522985,
+            -0.020741774066460415,
+            0.0068426980682519505,
+            0.013445270043414486,
+            0.0022744032984779294,
+            0.033840485942696216,
+            0.035040080489222324,
+            0.004166066423508316,
+            -0.010046666684988869,
+            -0.006148765564658713,
+            0.007071740742464309,
+            0.01865897335451031,
+            -0.023267411753900076,
+            -0.0005936657905496225,
+            0.004833030519903764,
+            -0.004229582648167378,
+            0.03620610659071198,
+            -0.009961238880412731,
+            -0.007197670122096852,
+            0.022212791947347127,
+            0.09819205436926473,
+            0.00047695621444943017,
+            0.0025354521225913395,
+            -0.029358210234742176,
+            -0.023614016211567963,
+            -0.0012564198366457442,
+            0.00242680708457405,
+            0.00830167274308547,
+            0.028076260517106013,
+            0.0004852400049405906,
+            0.011579274037419117,
+            0.02773972145320728,
+            -0.017217305534161797,
+            0.006752070301141707,
+            -0.011376051474975524,
+            0.024577071515205502,
+            -0.012694034026935636,
+            -0.0019809719792138673,
+            -0.016512649783050146,
+            0.015907402258634816,
+            0.004556574687833768,
+            0.004907510649166062,
+            0.011347630984408937,
+            0.009824484201337644,
+            -0.02650731900265866,
+            0.01756773113723491,
+            -0.005671986127099093,
+            0.004449611508541289,
+            -0.027841346572384462,
+            -0.020124233970502267,
+            0.0025986391930124847,
+            -0.0014440758308989809,
+            -0.009579444248630085,
+            0.006005274872617889,
+            -0.0024984235450999127,
+            0.03765664613203687,
+            -0.03765496603611941,
+            0.01409350051861866,
+            0.0066453175608372555,
+            -0.0015390841089599882,
+            0.011806653942640485,
+            0.017306914776093522,
+            -0.008206256939290151,
+            0.02589800448884443,
+            -0.00798213351353373,
+            0.009846274212000523,
+            0.015230070803055296,
+            -0.03449531102975745,
+            0.009174739184806312,
+            0.010145762060174144,
+            -0.024165660737765746,
+            0.012146877603301336,
+            -0.013784371016298891,
+            -0.004818759788054113,
+            0.005930446700613062,
+            0.007349837554764034,
+            0.010733731974181452,
+            -0.02295571464086674,
+            0.016176424826629898,
+            -0.010947841926210105,
+            0.0037766665765835445,
+            0.013559846998199008,
+            0.024892849808979323,
+            0.0012426907454129122,
+            0.0005950515136086193,
+            -0.01379469431504576,
+            0.005534724429523084,
+            -0.015667467221535825,
+            -0.022401586867694584,
+            -0.014192041775234079,
+            -0.006366846766730035,
+            0.01029598687444803,
+            -0.003972386708834858,
+            0.016971476088808737,
+            -0.010309648640697812,
+            -0.010645860303528099,
+            0.0037818310327362103,
+            0.01040806357838676,
+            0.008316605962234849,
+            -0.02855914079113881,
+            -0.005676310556150579,
+            0.004961084138163161,
+            0.00029285971503001265,
+            0.009592497954559435,
+            -0.014556073011576487,
+            0.013689526487160009,
+            -0.013290113354287095,
+            -0.0010726537498214612,
+            0.013201860397060236,
+            -0.012443552043930505,
+            -0.004012305744087929,
+            -0.01592905426965635,
+            -0.028487066798154617,
+            0.008462143660968396,
+            -0.01365614159523485,
+            0.001076882584445294,
+            0.0019503240125505646,
+            0.004730085646042709,
+            -0.016919069570642004,
+            -0.007860046623166481,
+            -0.033515448866521046,
+            0.011065416478687504,
+            0.02314589687882036,
+            -0.006435733825418645,
+            -0.030351727858789472,
+            -0.009546333526852862,
+            0.011327489786114963,
+            -0.0071765122743172715,
+            -0.01676736383051139,
+            -0.0029390038145944096,
+            0.007617910650439855,
+            -0.008174620754097493,
+            -0.00508943241317381,
+            -0.0025173969010414098,
+            0.0217328037301181,
+            0.016094975605203832,
+            0.01769642142712967,
+            -0.021400862603034036,
+            -0.00976992221747187,
+            0.026043879119776864,
+            -0.03275550610951898,
+            0.012441465975914529,
+            0.024587446266013183,
+            0.003446527051428522,
+            0.005492228054838714,
+            0.0030377554109693714,
+            0.005459159163772825,
+            -0.006985179226831048,
+            0.019925583798328282,
+            0.010870421035240845,
+            -0.019709062313471467,
+            0.005385443078244229,
+            0.02086085539375208,
+            0.0037248737640193097,
+            0.012892299156840998,
+            -0.017546870090541213,
+            -0.0020341807891543716,
+            0.015594779287017958,
+            0.04816214672670215,
+            -0.0032851797178091685,
+            -0.010450846098750195,
+            0.029513742590677183,
+            0.023526004023637855,
+            0.005373455940309873,
+            0.0012120777808635214,
+            -0.02525630456877349,
+            -0.013075130701774785,
+            0.01962235086017781,
+            0.016944870494393874,
+            0.013484925533584295,
+            -0.0023255656480479004,
+            -0.019236081983457366,
+            0.021520156296467717,
+            -0.01701733319731161,
+            0.012422659010697477,
+            -0.023764941037007197,
+            -0.001490790833263327,
+            -0.0035419008954878724,
+            0.021998949056452454,
+            0.0021819635638621092,
+            0.008891505304514552,
+            0.024354552127458608,
+            -0.006978548952551799,
+            -0.019526615985982355,
+            -0.0051799020400150305,
+            -0.014578953088901798,
+            0.022506635912517146,
+            -0.024117708614271965,
+            -0.007008641019892289,
+            0.011379795968360019,
+            -0.012362093847309038,
+            -0.0102534051092702,
+            0.001164934138876502,
+            0.017686807394319725,
+            0.0009423993435129515,
+            -0.014784508280754078,
+            0.01944445593807992,
+            -0.0011359271135469456,
+            0.010410915597949927,
+            -0.01921419010977189,
+            0.009390638453648025,
+            -0.016682442516992777,
+            -0.007849184048209236,
+            0.010343008873053859,
+            0.0012604521199837046,
+            0.003948300703645506,
+            0.002592490121754231,
+            0.007773437307969315,
+            -0.03355658426684504,
+            0.022695505487314516,
+            -0.010751606023585792,
+            -0.003538523688786008,
+            -0.0014125330167969218,
+            -0.01678692701672328,
+            -0.0006918162525881059,
+            0.005816502852598842,
+            0.010878358093847386,
+            -0.016045147059003256,
+            -0.00873270513444376,
+            -0.04920244395446931,
+            -0.030314009649638693,
+            0.023156333024481057,
+            0.013733657599720879,
+            -0.010971506342695331,
+            0.018852159415605117,
+            -0.023760853159532587,
+            -0.010587249404421732,
+            0.0011379869499042905,
+            0.023319856329056813,
+            -0.03539290744576506,
+            0.01215369703816749,
+            -0.014493682360886549,
+            -0.02039288669534806,
+            0.009070984596830142,
+            0.011445354735283254,
+            0.0007259852045513916,
+            -0.0011514627555568814,
+            -0.029350945123925453,
+            -0.010246845821727087,
+            0.02391068177425878,
+            -0.00861718589055971,
+            -0.004723044986826333,
+            0.024716520302409263,
+            0.0033721801437248873,
+            -0.012524663490691517,
+            -0.017955758068233754,
+            0.021612671655463236,
+            -0.026077203994996058,
+            0.007645944309444457,
+            0.010032183011253845,
+            -0.0173991263874487,
+            0.021649599230588117,
+            -0.007529701797485131,
+            0.020327763440441377,
+            0.004227390938169872,
+            -0.011249233530251522,
+            0.020352283974471443,
+            0.013644443758970054,
+            0.004216958800094986,
+            -0.004081669404784771,
+            0.006869353098014179,
+            0.02212732880780841,
+            -0.011021739614266063,
+            0.02219580717345023,
+            0.028777213295471067,
+            0.00017328191188631918,
+            0.03260179554743322,
+            -0.021287655520443285,
+            -0.03159728983004563,
+            0.021740163175540124,
+            -0.01035146858119169,
+            -0.021564906188712388,
+            0.028914245422221707,
+            -0.013674731836508768,
+            -0.00786476052825481,
+            -0.0166718429993922,
+            0.00533027311342322,
+            0.016680336311088843,
+            -0.005246190451159961,
+            -0.014360934066628332,
+            0.0029628103834216345,
+            0.019809487418628386,
+            0.014587234544007958,
+            0.029780009648869975,
+            -0.010778199815920887,
+            0.035180242185168895,
+            -0.002320576872571577,
+            0.008990157317473909,
+            0.012415465232073404,
+            0.006489435152972723,
+            -0.011251531176923445,
+            0.013501894212117511,
+            -0.003446759011266526,
+            0.008230548711588447,
+            -0.001449023474280409,
+            -0.035111679577202855,
+            -0.013709382199848678,
+            -0.04709294504969336,
+            -0.0023752976043235697,
+            -0.0025098360125255215,
+            0.0030975417740913154,
+            0.09736815869389318,
+            -0.005723049647039619,
+            -0.011988817571623457,
+            0.001625059666070763,
+            0.009611697638805556,
+            -0.0009896494420056814,
+            -0.007606836952841873,
+            0.015973152482488318,
+            -0.01343292635775299,
+            -0.02140928016038583,
+            -0.010532701190806147,
+            -0.01288594454368977,
+            0.012079198190522715,
+            0.015929716343161652,
+            -0.0005947359502228821,
+            -0.015207331663761012,
+            0.006524010655124294,
+            -0.004566385899752684,
+            0.0064717704997400245,
+            0.008647099975176308,
+            -0.010414668624343924,
+            -0.0071698879102925555,
+            0.030899143952119196,
+            -0.025301269549419674,
+            -0.024361748354052457,
+            -0.0010579931222861095,
+            -0.009055796737486008,
+            -0.010335459737012047,
+            -0.002692653633719985,
+            -0.0017679501322462056,
+            -0.010519994985692081,
+            -0.010019865263564572,
+            0.006537076627706128,
+            -0.015472130358273033,
+            0.004812857776601667,
+            -0.0047457011269495634,
+            -0.013296069496526007,
+            0.017054968086538724,
+            0.022563785357424853,
+            -0.02841491200807702,
+            0.0006755428946168761,
+            -0.017422510783648978,
+            0.012941211381006863,
+            0.008866069010589792,
+            0.006049618782357251,
+            -0.022451528209097505,
+            -0.023622906582403243,
+            -0.015019272754405791,
+            -0.01927535750568903,
+            -0.024973490129133732,
+            -0.009073381957684092,
+            0.0067692803507600945,
+            -0.001267339788153158,
+            0.021918281111033645,
+            0.000816538714279555,
+            -0.01812545613159305,
+            0.0061084913570894905,
+            0.000568371220952269,
+            0.018292497018668842,
+            -0.027468677288882186,
+            0.0033806422195878524,
+            -0.02307582511848465,
+            0.033508234091571124,
+            0.0039195050746594975,
+            -0.026680111514794824,
+            -0.021558668725755275,
+            -0.003311830287529699,
+            0.018994577319066473,
+            -0.0025745912019675805,
+            -0.00724049447539589,
+            0.014775207617625293,
+            0.0155334760323838,
+            -0.004693118176196483,
+            0.002015625406296403,
+            0.027968079002437102,
+            0.01077998866180723,
+            0.0313892866933904,
+            -0.015236169297590675,
+            0.004945828495617267,
+            -0.027215016713379062,
+            0.010214419950212732,
+            -0.01503548137805401,
+            0.008859554812646242,
+            -0.01290890682557532,
+            0.03531384210182567,
+            -0.012005613861257432,
+            -0.004250698950192301,
+            0.03253288796441852,
+            -0.029150986393345252,
+            -0.0035020051536906334,
+            -0.009255437891304041,
+            -0.004142204892160768,
+            0.013240580426083011,
+            -0.001198093874819309,
+            0.008485141356876356,
+            -0.01854370462655459,
+            9.726096864073609e-05,
+            -0.003221852304410318,
+            0.005404595442127598,
+            -0.02005458688172967,
+            -0.003314118809778734,
+            0.0073427568419969814,
+            0.004259518237708634,
+            0.002408630356035373,
+            0.016420646142725346,
+            0.006662121128241274,
+            -0.0029357701493526907,
+            0.03424224243607692,
+            0.008171895850772245,
+            0.0152521424877615,
+            -0.0018638264193202728,
+            0.0030234211261443034,
+            0.019287963516760636,
+            0.019912956661192856,
+            -0.00048443399729935116,
+            -0.01058935153883094,
+            0.00195113175754497,
+            -0.001140732137433033,
+            0.011048741602327437,
+            0.024396046054912645,
+            -0.021158025221644738,
+            -0.02419562092713724,
+            0.016377661005279716,
+            -0.015076145951740961,
+            0.01903712233378994,
+            0.010541463112889985,
+            -0.010099351302095792,
+            -0.00031558616108876247,
+            -0.042316016201379245,
+            0.010964989031040692,
+            -0.02068939391415103,
+            -0.04004120028915868,
+            -0.00244383573783919,
+            0.0009542507389088937,
+            -0.029439456243505052,
+            0.02822814855547978,
+            0.0023755032182188156,
+            -0.017280221740813364,
+            0.018223709137876153,
+            -0.005548654680726505,
+            -0.0024040301239229064,
+            0.005397711787238679,
+            -0.01092026090471121,
+            -0.010648485086769927,
+            0.008235250252660972,
+            0.010957715075632535,
+            0.005425683656535705,
+            -0.005698919020608095,
+            -0.011284911899894062,
+            0.009975882651638084,
+            0.012318901214538077,
+            -0.0019922884373083155,
+            0.010050520101815593,
+            0.0031720752428247446,
+            0.007352859750323051,
+            -0.0073483049386141005,
+            0.0046483587496626965,
+            -0.005093220128411125,
+            -0.026080057085901446,
+            0.012863580653903103,
+            0.0052434400313520774,
+            -0.011863360599637747,
+            0.0103209502958491,
+            -0.01048162756158923,
+            0.005787157766911021,
+            0.001318945460232246,
+            -0.0059333230422061395,
+            0.004821320238893602,
+            -0.0038922239931512784,
+            0.011267634571983924,
+            0.013256611037442042,
+            0.004592529900805719,
+            -0.013076954459595086,
+            -0.02177318635989856,
+            0.009001291224623467,
+            0.01959767163356408,
+            0.024161754192119852,
+            -0.030416109293488064,
+            0.003784634588325551,
+            -0.0014915849060829743,
+            -0.0020550564975080885,
+            -0.015632785947151753,
+            0.01587790981601866,
+            0.01716742056401084,
+            0.012721563202528083,
+            0.01893393006967701,
+            -0.004009617703771589,
+            0.010539100665843704,
+            -0.02080068888791658,
+            0.013642451766699067,
+            0.0008609869904066794,
+            -0.013629067231663643,
+            0.0014620655123815298,
+            0.027046304278766087,
+            -0.011056783825898616,
+            0.006206969699250696,
+            0.008097527175511791,
+            0.022111870736907183,
+            -0.012509479100313492,
+            -0.003735868296238069,
+            0.01516254968167132,
+            0.0014878674796313539,
+            0.0035887079417890352,
+            0.016533149527222273,
+            0.04729052276110659,
+            0.00102751186013391,
+            0.0016713773449738153,
+            0.0192177920520243,
+            -0.005563519063161466,
+            -0.01114821419109643,
+            -0.003549999863325982,
+            -0.021807893906469106,
+            0.010998558936481785,
+            0.00431279785115251,
+            -0.02186324212032519,
+            0.01024093366826206,
+            0.0125495473550584,
+            0.005268320135120931,
+            -0.0031679212410026603,
+            0.02574575247827441,
+            -0.019807879421996258,
+            0.014935295894965128,
+            0.03329450628384787,
+            0.00493356778223407,
+            -0.011423949333937285,
+            0.007263264249871384,
+            0.0027354459678742013,
+            -0.004494406041906684,
+            -0.00606745413660583,
+            0.010497391290734088,
+            -0.008191702034391382,
+            0.009870458191510176,
+            -0.009580130479693,
+            0.039202091799313755,
+            -0.009619018331262844,
+            -0.010782138248435458,
+            -0.004558645539551881,
+            -0.006241773108460662,
+            0.003478414182938609,
+            -0.003395165694642073,
+            0.005812225135633631,
+            -0.0069045407353789945,
+            -0.014607146220271209,
+            0.0008522982757210454,
+            0.00020175973815948303,
+            0.014001922919116725,
+            0.0011804904415274088,
+            -0.004871874532755353,
+            -0.007877217154706205,
+            0.015243277056007988,
+            -0.006047906208304912,
+            0.0006531937033317192,
+            -0.0011653523616313422,
+            -0.0013662556399359335,
+            -0.007401050012179761,
+            -0.015166105795085528,
+            0.00974609244984173,
+            0.016253733908242134,
+            -0.013857834499211056,
+            -0.005142936208363415,
+            0.012274716212832219,
+            0.008540229125193836,
+            0.011742606168187876,
+            0.0007451338650642946,
+            -0.009255813682572937,
+            0.008292531177374192,
+            -0.015020523035432158,
+            -0.008781317965985481,
+            0.0026250469418136603,
+            0.002381147382977664,
+            0.005839723981046825,
+            0.0006240641828668715,
+            0.004502463508605272,
+            0.005798912705218846,
+            -0.00497936644919367,
+            0.01112666823857836,
+            0.008198424171006548,
+            -0.002452143482105151,
+            -0.01017569410837814,
+            -0.001127589984272638,
+            -0.015733298167661346,
+            -0.016758315411028353,
+            -0.0009286266498936771,
+            0.015175988113750882,
+            -0.02352862898718829,
+            0.018996352975391644,
+            -0.0014524363611708201,
+            -0.0033802546171857704,
+            -0.021392428456030312,
+            -0.0025923983911570183,
+            0.0034214188668518796,
+            0.01255768230394476,
+            0.01858853630745928,
+            -0.02729543519465384,
+            0.010575894388563116,
+            -0.007204725245442141,
+            0.030394929424014717,
+            -0.018328782140005044,
+            -0.004022941273610227,
+            0.026915618079621992,
+            -0.02200018306828758,
+            0.0008791185377513978,
+            -0.011454606955158956,
+            -0.008103032693853504,
+            0.019287674837688177,
+            0.023747720480736777,
+            0.02038287242167783,
+            0.01364489294869211,
+            0.0007758044919789085,
+            0.004768030222445397,
+            0.03572705648994403,
+            0.010568241270862248,
+            0.014828087092859878,
+            -0.007542489523110974,
+            9.457808640264811e-05,
+            -0.02554033312739386,
+            0.01635899434369978,
+            2.587383534566272e-05,
+            -0.012370727506722627,
+            0.007162422186621379,
+            -0.013370882403931942,
+            -0.011066483600541058,
+            -0.007151077450730025,
+            -0.01721910018182534,
+            0.02152239849400241,
+            -0.012322440280680164,
+            0.004587729883596212,
+            -0.009079747696669277,
+            -0.013272707286450417,
+            -0.012953682633400515,
+            -0.014202214807187545,
+            -0.01313141988034687,
+            -0.026041403779538307,
+            -0.027109731260915763,
+            -0.006573285295895077,
+            -0.023780366346334524,
+            -0.013799072648505526,
+            0.005588622594136253,
+            -0.022617902666316757,
+            0.012262325666428604,
+            -0.012689827812482627,
+            -0.01226264872059864,
+            0.012805734807835008,
+            0.020274844315616616,
+            -0.011765231803328348,
+            -0.028494489703400308,
+            -0.018583843821043854,
+            -0.019970998470704358,
+            0.012576201137505969,
+            0.002266288538944273,
+            0.013165889075444961,
+            -0.003018437854660793,
+            0.003227412667880239,
+            -0.014895831590298957,
+            0.018570624675927183,
+            -0.03442390703780963,
+            -0.03108003806963536,
+            -0.013611306765348361,
+            0.005182777066476028,
+            -0.0015782033203578553,
+            -0.011429046504256945,
+            -0.0018606648572920801,
+            -0.00433805271182878,
+            -0.025109572024446088,
+            -0.010134856632190067,
+            0.011010662054484161,
+            -0.015205629927469414,
+            0.0072933565255675036,
+            0.0009018832564038863,
+            0.013870902834260887,
+            0.009587432944616442,
+            0.013393301893492882,
+            -0.006095063698384775,
+            0.004356485794113208,
+            -0.0026476612747324386,
+            0.0004894024149759108,
+            0.009336366919474478,
+            0.00839591561027103,
+            0.010551161369079399,
+            0.002260707777260338,
+            0.007830059223001696,
+            -0.0060927446216633265,
+            0.018065002687169148,
+            0.015436944875538481,
+            -0.0002520689442496089,
+            0.013323098485251487,
+            0.0003769862591044255,
+            0.00022293304921127018,
+            -0.0029970303106340466,
+            0.007888437018607987,
+            -0.011785957933642163,
+            -0.001311704406355858,
+            0.015138190893483804,
+            -0.01707247730569978,
+            -0.013045187198628258,
+            0.03107034781283638,
+            0.0018583638488268697,
+            -0.02495564203304796,
+            0.031146554085787762,
+            0.004220047610942943,
+            -0.005835199594223787,
+            -0.013205086624848782,
+            0.0014059047822222418,
+            -0.011570624844100298,
+            0.003584232809362274,
+            0.014295032061446826,
+            -0.02231984223610487,
+            0.010521094801839404,
+            0.008446892441796151,
+            -0.009681535919980559,
+            -0.012337659795586223,
+            0.009847073709209602,
+            0.016809328445200358,
+            0.004472485504034416,
+            -0.03539093810210523,
+            -0.011957694102753411,
+            0.02145470732008716,
+            -0.029237428225009913,
+            -0.00810041956259673,
+            0.007452962400105787,
+            0.0065154832040426065,
+            -0.019353109017609053,
+            -0.008580955020412362,
+            0.005725794605897509,
+            -0.004940390374637084,
+            -0.003892210172017758,
+            0.017541623128534682,
+            0.00485047043626186,
+            0.013698341792101025,
+            -0.01689738578307474,
+            0.009771458057400557,
+            -0.022222817828602557,
+            0.004021882074676753,
+            -0.023373700500461992,
+            -0.009873403167702603,
+            0.002364143609079849,
+            0.012454271200139762,
+            0.014328556467632212,
+            0.027839363143659326,
+            0.030056404238348527,
+            0.013995696971422429,
+            0.007833677585250953,
+            -0.016265199386953717,
+            -0.018134327237616175,
+            -0.006409513840075933,
+            -0.004033173350204784,
+            -0.008714798059218596,
+            0.0066373547199279235,
+            0.01858354212009877,
+            0.004499712802472377,
+            0.02609078206670711,
+            -0.006227189596272876,
+            0.009039744622014585,
+            -3.127526911951914e-05,
+            0.0008833930131827993,
+            -0.004226083827636182,
+            -0.01971685546584527,
+            0.00481003211084742,
+            -0.0013721482162825508,
+            0.025874758880591167,
+            0.00940496860916206,
+            0.0024915013399541847,
+            0.01615978314811047,
+            -0.006378952757397484,
+            -0.01174653607178726,
+            -0.010540107733238588,
+            -0.0061926268398085176,
+            -0.021752582127981104,
+            0.029059100503124387,
+            0.021195595568253612,
+            -0.010800765264123782,
+            -0.013693582166598082,
+            -0.00903369526059487,
+            0.02795767990556847,
+            0.004080965688836953,
+            -0.0013050212744006933,
+            0.016313311850308505,
+            0.012084555958999215,
+            0.012329325712491252,
+            0.009794698207689061,
+            -0.00036157148861933123,
+            0.0015364709122308352,
+            0.03722522141440073,
+            -0.007427028510548186,
+            0.0026463292682538495,
+            -0.005239324411651515,
+            0.0025987932868461313,
+            -0.009705817520422487,
+            0.007242122655628283,
+            -0.004012676385296197,
+            -0.0158760051470016,
+            0.015003444174308607,
+            -0.022443331305169994,
+            -0.0016749787204471238,
+            -0.008150635190952799,
+            0.0016696640640947303,
+            -0.02781023337840043,
+            -0.0025731206607371845,
+            0.006695065701630532,
+            -0.015547340828489069,
+            0.011816010501093885,
+            -0.01740673229051557,
+            -0.011553604323149105,
+            0.007914961618894248,
+            -0.012034157288065447,
+            0.0016684130271324446,
+            -0.0008268957621923278,
+            0.004460864332388434,
+            0.013426055079012884,
+            -0.011008808570221245,
+            0.012166046631826273,
+            -0.005816284310138776,
+            -0.007872091214834688,
+            -0.0016055129487028885,
+            -0.0030049472973942043,
+            -0.0025600746761171543,
+            -0.0009966231170649917,
+            0.034355910234323155,
+            -0.004446570896048543,
+            0.003761242147499512,
+            -0.01362639743248326,
+            0.0019833052855181806,
+            -0.036176376394999044,
+            0.028344464817309477,
+            -0.008642835889481802,
+            0.011013502221008498,
+            -0.0017508365480865596,
+            -0.013243706268954915,
+            -0.01456517309747086,
+            0.015247299501245178,
+            0.007602596572394442,
+            -0.005854531057349665,
+            0.013558248087347848,
+            -0.019724373614448298,
+            0.008530500822910953,
+            0.004463694543554244,
+            -0.015561397976987216,
+            0.004668780190328835,
+            -0.007479912658259757,
+            0.012657201279639194,
+            -0.01774730033362932,
+            0.004314594593150765,
+            0.00958571617187387,
+            -0.01892876084898881,
+            0.009901041800918215,
+            -0.028675879564512306,
+            -0.004631920242208613,
+            -0.012323455524366223,
+            0.0053804608827285865,
+            -0.000660864620708182,
+            -0.02389088252276507,
+            -0.028944280660230762,
+            -0.0002099667627277945,
+            0.00458533068247518,
+            0.0018878351997997833,
+            0.011621407953503755,
+            0.005970807015151215,
+            0.008542958629199003,
+            0.01023896054756221,
+            -0.00205207247588841,
+            0.009402752124876478,
+            0.04169706217414899,
+            -0.022899912321317197,
+            -0.011574970753549647,
+            0.005321872977369877,
+            0.011499271505460519,
+            -0.02647653483798284,
+            -0.011039791758347888,
+            0.0033424345263339144,
+            -0.005219877979550397,
+            0.03609469691834737,
+            -0.02448746405881972,
+            0.019020618652020673,
+            -0.011392297116547224,
+            0.007625475937968714,
+            -0.021465240076078376,
+            -0.003921088264697311,
+            -0.014265745991522477,
+            -0.00011221615798127438,
+            -0.0031478236827989703,
+            -0.0076053733863387755,
+            0.007485039021296698,
+            -0.001907678661303243,
+            0.0020348246022161925,
+            0.006291436821831794,
+            -0.02173305455485444,
+            -0.0019125499969493854,
+            -0.016432460924913166,
+            0.030367263349759286,
+            -0.016242095671439496,
+            0.005072805486919414,
+            0.0053258139728101506,
+            -0.002810639419737258,
+            0.012368771814660942,
+            0.004165967191536186,
+            0.00513754440091394,
+            -0.004199767241827414,
+            -0.005142642360546972,
+            -0.0053886448467479215,
+            0.017868874549760376,
+            -0.013280048598849825,
+            0.035024701466710065,
+            -0.0100959186045929,
+            -0.007094001877261568,
+            0.0019733042325232155,
+            0.010380430710282744,
+            4.2084202709296443e-05,
+            0.01053565670076933,
+            0.0034270489770133685,
+            -0.0006311078385359693,
+            0.0019774779328212756,
+            -0.017162448473652843,
+            0.003089758682621882,
+            0.01682656561693936,
+            0.005463677441664502,
+            0.004469261499451007,
+            -0.013332130891003062,
+            -0.012050233723282738,
+            0.004538268704442613,
+            -0.007533940408255447,
+            0.0003675737140813486,
+            -0.018816953644773076,
+            0.013972696748575243,
+            -0.0011096683686631858,
+            0.0002715683052236706,
+            0.00524678369365015,
+            0.023056514997908218,
+            0.01813167860994747,
+            -0.002553352602551003,
+            0.03155164978556312,
+            0.0174738879054593,
+            0.0029268039949260673,
+            0.010637295711777793,
+            -0.033222781195984666,
+            -0.013301873512972214,
+            -0.0013721703891642354,
+            0.02238421933149517,
+            -0.01530189703335162,
+            -0.014059029663525449,
+            -0.0037393534403749947,
+            0.0010394601933309767,
+            -0.005177359933828505,
+            0.0042489306011644495,
+            0.011039889511148341,
+            -0.014697595240068863,
+            0.007597149702441915,
+            0.0026780496822907634,
+            0.02092734730183164,
+            0.0004688144432243786,
+            0.017181410417231698,
+            -0.0005083970646221274,
+            0.00749639055342427,
+            0.007319420826337891,
+            0.017541463125355265,
+            -0.01002811389296805,
+            0.023582113845081534,
+            -0.008823933397729605,
+            0.0019889258902730535,
+            -0.01257106158422629,
+            -0.028159594067544737,
+            -0.012624750220149823,
+            0.015247564545279736,
+            0.005765911899842925,
+            0.006478299226263732,
+            -0.016537090297144265,
+            0.006276997440048981,
+            0.015647122171767974,
+            -0.002251683151059454,
+            0.010295358869874306,
+            0.009730378299068289,
+            0.00012816405890877098,
+            0.010181120688239766,
+            -0.00801256937783534,
+            -0.00734257767723293,
+            -0.004497520124843868,
+            0.023982754195729983,
+            -0.005443550092842393,
+            0.022434177122972028,
+            0.012647958478747348,
+            0.00616791857573897,
+            -0.010853355357954727,
+            -0.003340111254357573,
+            0.023389634440057473,
+            0.011863246790550376,
+            0.00029642699874398923,
+            -0.009846034060495678,
+            0.0019458876579524617,
+            0.0054010373765486005,
+            0.012983913038716568,
+            -0.02049427270240938,
+            -0.004344571352918878,
+            0.008461947538684664,
+            -0.008472919299762712,
+            0.004318309005722874,
+            -0.030156444522730416,
+            -0.005812303237222271,
+            0.021354352236942434,
+            0.008392281015122372,
+            -0.010232039722494646,
+            -0.004395682831498155,
+            -0.0006527587802367882,
+            0.0040601850269714945,
+            -0.0024323167053212176,
+            -0.0023729138545912176,
+            -0.008541233856441836,
+            -0.006470933419694742,
+            0.014035696617797647,
+            -0.0052691760528871924,
+            0.008597459268477037,
+            0.00012084859681905111,
+            -0.0017255958688825231,
+            -0.007323529232751411,
+            0.03382792341957058,
+            0.008087732655845718,
+            0.005980666974986827,
+            0.0013613625176327891,
+            0.0042244330945601215,
+            0.000616791811672998,
+            -0.0028374856646453707,
+            0.018075360591135542,
+            0.0156738939725117,
+            0.005054696911814988,
+            0.009038525980341447,
+            0.014672270369917403,
+            -0.019700405219230957,
+            0.012625337070556061,
+            0.012184185186281482,
+            0.019382321278525114,
+            0.0007219498737851579,
+            0.002564970959687783,
+            -0.010601873823737926,
+            -0.0029963537053764736,
+            -0.002070567089566093,
+            0.029563028293936092,
+            0.0005903976560069006,
+            -0.010509758568838935,
+            0.010992421945232951,
+            0.020603868018231348,
+            -0.02101157589451638,
+            -0.001971974816402967,
+            0.005963911670920596,
+            -0.010704207628592966,
+            -0.014466113540150018,
+            -0.021269176952661614,
+            -0.013331247081394942,
+            0.0071597345125287174,
+            -0.008162102923896894,
+            0.02367549407145414,
+            -0.014256771777502395,
+            -0.03960796112042958,
+            -0.014803626428839152,
+            -0.00864320347698291,
+            -0.027531337207277483,
+            -0.007305441146142331,
+            0.00965056093889974,
+            0.016698589600806273,
+            -0.003330867073501157,
+            -0.025198703196464594,
+            0.030483790406960656,
+            0.008872821868559751,
+            0.023708422767352917,
+            -0.021464374508407488,
+            -0.0018617153194555344,
+            -0.00886376784514331,
+            0.014505918224188001,
+            -0.017588761818663975,
+            -0.011756112651591795,
+            0.020014632992762026,
+            0.014356419813769714,
+            0.007246557500044434,
+            -0.004355132555323843,
+            0.0067299398462013936,
+            0.007649499235136303,
+            -0.012515181210363267,
+            0.012239792251154642,
+            -0.013499277313916029,
+            -0.024858295126228433,
+            0.014430561986516884,
+            0.002229494935179828,
+            0.009563004428064055,
+            -0.0015984171085659323,
+            -0.016197659608768088,
+            0.010078159490026602,
+            0.00808729175515505,
+            -0.017508840440512225,
+            -0.004793934415922065,
+            0.007133527615305415,
+            -0.020818869260580315,
+            0.031234686259160864,
+            0.011872765152279094,
+            -0.010692949770503467,
+            -0.001989039629661815,
+            0.004431079028880923,
+            0.016511544608975148,
+            0.007893700385963683,
+            -0.01997314388229276,
+            -0.003708006454566117,
+            0.01530128306203401,
+            0.00822588305477249,
+            0.004861703074728471,
+            -0.010836328783594584,
+            0.03434874763691911,
+            0.006082873924472328,
+            4.192481378506604e-05,
+            -0.0052997095110967115,
+            -0.0076108924981838355,
+            0.004196112457289409,
+            0.018905391806961686,
+            0.013330078288599763,
+            -0.004200590783114919,
+            -0.006630107527854746,
+            0.007778448128358027,
+            -0.01706991160178345,
+            0.007329056462125937,
+            0.009390327499494826,
+            -0.012870674959022891,
+            -0.0017571189976160295,
+            0.016462608870101713,
+            0.00209846661139665,
+            0.004942714850185376,
+            -0.011305050739130339,
+            0.004252776146827917,
+            0.015841821944109317,
+            0.018727567181082646,
+            -0.008399642221255733,
+            0.021448055799259867,
+            -0.020972459569228434,
+            0.027243639465535856,
+            0.028499400850980086,
+            0.016573144403275928,
+            -0.01847096443459798,
+            -0.0022409373140116146,
+            -0.0041737366139895414,
+            -0.014761716560267328,
+            0.0034546020161337894,
+            0.005125870074458691,
+            -0.01174186608185822,
+            -0.009577921232380895,
+            -0.024331993794416026,
+            0.011075959943215117,
+            0.007918147902741738,
+            -0.0015045185909419564,
+            -0.011791155919533844,
+            0.016019077577552835,
+            0.010141868950981925,
+            -0.010938402325619172,
+            -0.005635351352447256,
+            -0.019967361901734836,
+            0.009750427567651833,
+            0.009128294017175602,
+            -0.00300556760891084,
+            -0.011496103417643132,
+            0.012977890524465716,
+            0.0014080059843484115,
+            -0.01292380658817521,
+            0.010841954636959441,
+            -0.03142011651894961,
+            -0.005510537354374977,
+            -0.011027026734432255,
+            -0.003620482919145167,
+            -0.014415606143566519,
+            0.003608572766363589,
+            0.0005034611321129689,
+            -0.011470488351158329,
+            -0.0035005774696278,
+            -0.016207453351921627,
+            0.004676327282262429,
+            0.015021513761697039,
+            0.001679614645967629,
+            -0.006917375094634678,
+            0.021994274901326064,
+            0.018518524483306403,
+            -0.021903538417919984,
+            -0.020116013379669034,
+            0.004708084983787958,
+            -0.03261690161800251,
+            0.015019348187107588,
+            0.00966124471577622,
+            0.011676937880751252,
+            -0.028768229226408037,
+            0.007114534979415504,
+            0.00900420165933673,
+            0.016398927481511824,
+            0.0050879301160439245,
+            0.009453255486450082,
+            0.01222758288108498,
+            -0.0104657614414275,
+            -0.003164502778525433,
+            -5.410382753651127e-06,
+            -0.005618225986727351,
+            0.015607620507415484,
+            0.0008527697935056332,
+            0.03603666488542239,
+            -0.005277207843139309,
+            -0.0009799879252482288,
+            -0.011735731235560851,
+            0.00812046801749312,
+            -0.011713349784345638,
+            0.006334172347602515,
+            -0.0024643499727789167,
+            0.0001240009282616789,
+            0.002965067818077541,
+            0.009442572075213767,
+            -0.01086095783353607,
+            -0.007595379345224076,
+            0.007444897256325813,
+            -0.01482814340478338,
+            -0.00019984717819268013,
+            -0.009339343730980451,
+            0.0019680804106275156,
+            -0.02241031895876346,
+            0.006629838013824249,
+            0.040935644507527846,
+            0.02140216465404127,
+            -0.005481040712269146,
+            -0.012431659209559418,
+            -0.004680230194268222,
+            -0.007229300034633229,
+            -0.006482401275991039,
+            -0.004794698160419071,
+            -0.022762405726589375,
+            -0.006928553505112732,
+            0.004320321244630314,
+            0.013300138724212552,
+            0.009741018117644266,
+            -0.008842135469729511,
+            0.004531760574809748,
+            -0.02662650543210291,
+            -0.016578081411548973,
+            -0.0060021425889846495,
+            -0.015840130031005147,
+            0.006125301421471048,
+            0.027521841635711787,
+            -0.0063432924325983525,
+            0.01789356321527744,
+            0.005142029893259461,
+            0.013034833065163448,
+            -0.005338869351330258,
+            0.00035250243782976347,
+            -0.006010938630154189,
+            0.0032958656042048292,
+            0.003074861351445954,
+            -0.0015947217852680714,
+            0.004151590621941346,
+            -0.007139285822575166,
+            -0.0026091937868967284,
+            0.0011629380311555433,
+            0.0117841884590861,
+            0.006877419488576261,
+            -0.00016252653621991807,
+            -0.01761285866765538,
+            0.022221104524808997,
+            -0.018933508559881868,
+            -0.008840161102111973,
+            0.011092684029552064,
+            -0.009660487856185668,
+            0.00702597023411946,
+            0.020387386977378873,
+            -0.006854213562767843,
+            -0.009583981919303215,
+            0.013028181087649146,
+            -0.0036507672754997286,
+            0.004464368543493108,
+            0.009867428978560914,
+            0.006036303973414791,
+            0.002482944923100156,
+            0.013244650094892797,
+            0.004530294330688855,
+            -0.0032625482538717137,
+            -0.020238533752799943,
+            -0.005186498720507101,
+            0.019015752262933585,
+            -0.02252458872825473,
+            -0.0027641215988942563,
+            0.0044506144202026456,
+            -0.0030317840657635457,
+            -0.01733962766050954,
+            0.004205861489436576,
+            -0.018770275720867473,
+            -0.0033883519277721433,
+            -0.008244668812709337,
+            0.009948090119054079,
+            -0.023700732627507225,
+            0.009109315916173436,
+            -0.011739192106106307,
+            0.02356509175177699,
+            -0.003925851077921686,
+            0.024734433836185024,
+            0.0015142645677087268,
+            -0.013482933030628261,
+            0.0055323884713736554,
+            -0.0008930661843169134,
+            -0.004477731568313757,
+            0.029338490404300612,
+            -0.005418258701133154,
+            0.002249996261871466,
+            0.013491124734151694,
+            -0.021332541202417246,
+            0.021773687217263173,
+            0.0046662696076723836,
+            0.01628697326539432,
+            -0.00986238951830334,
+            -0.013799209313714441,
+            0.025178874553620743,
+            0.03324555035648896,
+            0.008398796308070151,
+            0.011261509865513953,
+            -0.01226890254177634,
+            0.018966175575083295,
+            -0.0038328997984309597,
+            0.004344837325292706,
+            0.00878668880458693,
+            -0.007200842513742962,
+            -0.011500562922303248,
+            -0.004533014670236755,
+            0.027639894892383943,
+            -0.021838848944317424,
+            -0.008920629621611768,
+            -0.005739181836252042,
+            -0.01593272996094732,
+            -0.038167534267554606,
+            -0.004371701247332816,
+            0.019604405563120094,
+            -0.005974358141737469,
+            -0.0018341151594086288,
+            0.014334266690716505,
+            0.00402620072221496,
+            -0.002852569009185304,
+            -0.008195218707838913,
+            -0.0027944153624309856,
+            -0.0059390492900516225,
+            0.016651206007022145,
+            -0.004961864528562493,
+            -0.007929830585906699,
+            0.01145431100211762,
+            -0.005731957927601062,
+            0.012130694936714224,
+            0.0004848418056781251,
+            8.392048424584131e-05,
+            -0.0015174120840537315,
+            0.00822932723013239,
+            0.018620869354789082,
+            -0.0055803454666417465,
+            0.025682778991091174,
+            0.006956936033581638,
+            -0.003495709501259274,
+            -0.005489431577052735,
+            -0.0044154293119170445,
+            -0.024561016942073285,
+            -0.0020864636079347357,
+            0.006593060072668424,
+            0.024333813203596122,
+            -0.02515989187381735,
+            -0.004953266000463339,
+            0.020957806164031402,
+            -0.0012192724005613,
+            -0.011978722479596745,
+            0.014690559986694138,
+            0.0041346025614047875,
+            0.009881584250499358,
+            0.012176393203318002,
+            0.015764530703333238,
+            -0.004923548157246672,
+            -0.0015192775000207833,
+            0.012766482928380375,
+            0.0005745973040874939,
+            0.00997809510696241,
+            0.016495444272181515,
+            0.010467514633877887,
+            -0.021108281220544853,
+            0.0015078843789125156,
+            0.00522593579521476,
+            -0.017481430635971396,
+            0.0034362821987740755,
+            0.005397819166037461,
+            0.011626527721895694,
+            0.004048632382026344,
+            0.013207335578216437,
+            -0.015768003434275563,
+            -0.005916168437120099,
+            0.008153798523776654,
+            0.004319451296835019,
+            -0.02599474886669325,
+            -0.01485009085106089,
+            -0.0006781964116146299,
+            -0.004444144921332929,
+            0.013556265879295017,
+            0.01285768162554067,
+            0.02033626001285354,
+            0.024637349990579514,
+            -0.013833060730799217,
+            -0.00842819657792762,
+            0.009436072455719896,
+            0.015560357465711765,
+            0.005798181411384387,
+            0.007874051721372273,
+            0.009297440071118126,
+            0.0077320823375042,
+            -0.004073209576661068,
+            -0.002973928843567532,
+            0.020205003097225747,
+            -0.019725382393637726,
+            0.0006746659838642147,
+            -0.018417443693472958,
+            0.007530796896254817,
+            0.01140696723506382,
+            0.01129515042745398,
+            0.002816883643872417,
+            -0.005274757308627299,
+            0.005716917047812153,
+            -0.06254973656832136,
+            -0.009191453047502128,
+            0.003384751551032943,
+            0.027256466708746898,
+            -0.010885336140129142,
+            -0.003415707910562931,
+            0.007309577090125959,
+            0.005511970708769456,
+            0.0005572239548625122,
+            0.003186427278034852,
+            0.019698716902282903,
+            -0.02052581021967601,
+            -0.010793968505805883,
+            0.004219269922138548,
+            0.0033111837923746746,
+            -0.007269521534951044,
+            0.004805737092302134,
+            0.014143978208501908,
+            0.01986656063456352,
+            -0.0024048854610842748,
+            0.01629335842915328,
+            -0.0015498603704589015,
+            -0.014496994444493684,
+            -0.002671064325909516,
+            0.0037875833047555925,
+            -0.022466433279066646,
+            -0.005980897597195858,
+            0.0043262697442044435,
+            -0.011226121648007667,
+            0.006978495397818936,
+            0.008452177557316863,
+            -0.003018545465557218,
+            -0.0008303307644324117,
+            0.017796789945838845,
+            0.006591364879905198,
+            -0.013464486184025026,
+            0.0028668426952326957,
+            -0.023377605222213174,
+            0.004325185794796361,
+            0.001416245546006914,
+            0.007809449561523174,
+            -0.008431335139468173,
+            -0.014774499666285603,
+            -0.005933810712528226,
+            0.017746847020610673,
+            -0.01965188730040385,
+            -0.002406875192894708,
+            0.01683009662984443,
+            -0.0002518220110313279,
+            0.005456812494894182,
+            -0.02011241622618092,
+            -0.0009782974741561381,
+            -0.02933296756078644,
+            -0.01171823509606186,
+            0.001162345317872531,
+            0.00892221099991475,
+            -0.006551308022163148,
+            -0.03934356275100932,
+            0.00427623835776202,
+            -0.013111664387776261,
+            -0.008752857644809231,
+            -0.010214546496134218,
+            -0.00518087722852466,
+            0.015240297918667867,
+            -0.012498100033235827,
+            0.01086490299272505,
+            -0.00917777212602538,
+            0.005895055740265625,
+            0.0090554651297836,
+            0.014212841798768113,
+            -0.005544047915369086,
+            -0.0033589999440643385,
+            -0.007924830667307956,
+            0.0022291797354234005,
+            -0.0032976226056924502,
+            0.009615674261248298,
+            -0.0051932155872883,
+            -0.0065636389655770455,
+            0.02426106498596339,
+            0.011445591067574932,
+            0.013312596099532533,
+            0.0055118624272110965,
+            -0.007220620072349783,
+            -0.01916551187613823,
+            -0.003977616240384473,
+            -0.013204506708246246,
+            -0.006315966975649096,
+            0.01434099854765068,
+            0.012588249778961223,
+            0.006357972880218866,
+            0.0007747002406996095,
+            -0.004794439624135275,
+            0.006975593520375694,
+            0.0262335496080837,
+            -0.016481909971520166,
+            -0.0013084662210086592,
+            -0.01369313900978409,
+            0.01921748474212319,
+            0.03333761672336437,
+            -0.0008918419419301244,
+            -0.007699525186523558,
+            0.013757216065348164,
+            0.00819045303047094,
+            -0.007411380297164911,
+            0.02453611080891859,
+            -0.025045462880463647,
+            -0.015989319900023054,
+            -0.011650886878708265,
+            -0.011407253985176972,
+            -0.008386367959227559,
+            0.016331961716602287,
+            -0.00141225438918211,
+            -0.0018505754298268654,
+            0.009557291617446645,
+            0.03357924917013024,
+            0.006916860901290756,
+            0.007030308953738567,
+            -0.002739345982346728,
+            -0.009376522438071562,
+            0.011482637968411055,
+            -0.007259054391879614,
+            -0.04075661435928125,
+            0.004250195164583401,
+            -0.005160505473755424,
+            0.02405623169060588,
+            -0.02197586561375289,
+            -0.00619175761282745,
+            -0.006736129049831973,
+            -0.028200015908139763,
+            -0.005867722104196303,
+            0.02805071895018145,
+            -0.015402198567775478,
+            0.014773294766214671,
+            -0.009502185346558072,
+            -0.0013419858580438616,
+            0.004443749010136947,
+            0.013008783570355656,
+            -0.02289662952366337,
+            0.0013834888740540108,
+            -0.00048549197205067054,
+            0.027969842651517314,
+            0.010437729090975158,
+            -0.0047705656249623744,
+            0.014202218020412936,
+            0.008240861154198411,
+            -0.018637315941614406,
+            0.007314019317343154,
+            -0.005747966765101792,
+            0.014833896739762147,
+            -0.01314847940891154,
+            0.007762100494499689,
+            0.00705026393499352,
+            -0.01278023669088385,
+            0.03016342404000438,
+            -0.015204129985039077,
+            0.007776946006562982,
+            0.009278778571285689,
+            0.009656723862179986,
+            0.01257686993982689,
+            0.004362439504889069,
+            -0.02898418193036562,
+            0.0002144431781454467,
+            -0.03828100004144633,
+            -0.00833748467467171,
+            -0.002611937107022567,
+            -0.002145329666219461,
+            0.007050630951707937,
+            -0.009383645162261629,
+            -0.033976442057557676,
+            0.0006429978367640709,
+            0.01603027598131823,
+            0.017501067413108286,
+            0.0002196997337451293,
+            0.013075379412332973,
+            -0.006197537539554383,
+            -0.021148120787551232,
+            -0.016986448660412497,
+            -0.010207705210051413,
+            -0.02122194744719993,
+            -0.020550470201191853,
+            -0.00908564461857315,
+            0.012090079127547193,
+            -0.009927390005879807,
+            0.013888458967497627,
+            0.029513250508042935,
+            -0.024529175134769883,
+            0.0011144447737101862,
+            0.021977677894839922,
+            -0.00198308986575084,
+            0.014108189311291326,
+            0.001696775006906642,
+            0.014122064251288995,
+            -0.0004166140484018266,
+            -0.010556016685519523,
+            -0.009729362492661684,
+            0.017487929296614076,
+            -0.0026043257634797124,
+            0.016715798948897194,
+            0.012183487338360088,
+            0.003771721868999568,
+            -0.0030887189945795796,
+            0.010667776697461199,
+            -0.026471157050402344,
+            0.014249996582064706,
+            -0.006690961709923224,
+            -0.0181382691440242,
+            -0.008729104191096461,
+            0.003882731567105777,
+            -0.005609384377642956,
+            -0.007794812949125115,
+            0.012967089730551055,
+            -0.00944374111323783,
+            0.02088525981049842,
+            0.0023400121055514958,
+            -0.012196925240774426,
+            0.003648688547842322,
+            -0.03322041209304968,
+            0.0015191765655963997,
+            -0.018402382776013956,
+            -0.024659631472836927,
+            0.03840173429666435,
+            0.011991452465253198,
+            0.006639702644606365,
+            -0.01675838115087676,
+            0.007953102302963526,
+            0.0004257479565099406,
+            0.017484338819046517,
+            0.011068864013925452,
+            0.006466915839538296,
+            0.008364454017976605,
+            -0.023316137291380285,
+            -0.010898421318668592,
+            0.016204569093730564,
+            0.012099473359791148,
+            -0.01017229369980751,
+            0.019431669546223253,
+            0.017955119269367787,
+            0.007300372071724306,
+            -0.006898619924677706,
+            0.01647927363640854,
+            0.021012103522831126,
+            0.010291016618557653,
+            -0.00047955552103474145,
+            0.0008684297441778696,
+            -0.010534247928997246,
+            0.005037930235889528,
+            0.035417626740307205,
+            0.014712595692093886,
+            0.010650306363963653,
+            -0.03626584475133921,
+            0.021764456453493054,
+            -0.011961521673661574,
+            0.009092981192874177,
+            0.006089104887183293,
+            0.017982801101846038,
+            0.006916969397519662,
+            0.014196254958707752,
+            0.011745069030657836,
+            -0.006689305450954901,
+            0.025205280281804708,
+            -0.001601414806100461,
+            -0.011992525142592405,
+            0.003386930324498371,
+            -0.01859193251438437,
+            0.005303261319773131,
+            -0.0041710536965100615,
+            -0.004459393120257307,
+            0.0032461828144552565,
+            -0.0018650956576455287,
+            0.015678225251799887,
+            0.003894142628084471,
+            -0.02355212701026675,
+            -0.0014873755115206368,
+            -0.024284295126348117,
+            0.0027260592623277375,
+            -0.022185571998195344,
+            0.00022883605560765203,
+            -0.005557568353382136,
+            -0.009749382813201403,
+            -0.025385857431333838,
+            -0.009853073898348951,
+            -0.0035843255959309307,
+            0.007661137888845632,
+            -0.010521037433807653,
+            -0.004246698084422667,
+            -0.0019920499321964815,
+            -0.019465113363548715,
+            0.009014255846983305,
+            0.027122533756959888,
+            0.016848221859975445,
+            0.027170883580103264,
+            -0.00019554112591107156,
+            0.013614675849485926,
+            -0.002066822813103284,
+            -0.01603063005944364,
+            0.008824663429777616,
+            0.0075653522381218495,
+            -0.00531780740880296,
+            0.014643348459785112,
+            0.0072634484937344704,
+            -0.012800149015319017,
+            0.009859503660384916,
+            -0.007161712176469609,
+            -0.006437857349742842,
+            -0.009764856421740622,
+            -0.02058868349990357,
+            -0.014866137971603676,
+            0.0012450207572602557,
+            -0.002123265406747006,
+            0.006000473862248991,
+            -0.004054016790850226,
+            0.009584879012937315,
+            -0.00608816807208866,
+            -0.00925801057439376,
+            -0.013962683072792106,
+            -0.008924356118158748,
+            -0.004210064145949253,
+            -0.004381709838058668,
+            -0.011732699976353612,
+            -0.005479523097469261,
+            0.0010680258077037758,
+            -0.0019401640391035755,
+            -0.01397652201595301,
+            0.011650839243603377,
+            0.0058731838291845645,
+            -0.002462743134993977,
+            0.02219008688925881,
+            0.01097316136828334,
+            0.0018083001685932406,
+            0.01366095798812196,
+            0.0003942601242216818,
+            -0.0064063532958756175,
+            -0.03343312148366622,
+            -0.0005083355836126291,
+            -0.02815586367527057,
+            0.025770078162165123,
+            -0.0069114949102666145,
+            -0.010322087105114614,
+            -0.002519047562374146,
+            -0.006063669644021698,
+            0.005873609611264377,
+            -0.002722940652283699,
+            0.0021229525800901114,
+            -0.023673778230768945,
+            0.0028629715379660113,
+            -0.003953653437464966,
+            -0.0036366957271968823,
+            0.007469664864329332,
+            0.0346520690389124,
+            -0.004054282838167587,
+            -0.0034765253405897676,
+            0.005159466707114532,
+            0.011642913651743502,
+            0.027845571220425726,
+            -0.004445196354520585,
+            -0.0032038508043339413,
+            -0.02233838584152591,
+            0.0054112576531374175,
+            -0.00019544838850471866,
+            -0.004733192299951088,
+            0.016664217348411003,
+            -0.016719175418727594,
+            -0.01631658832047864,
+            -0.00545969677717507,
+            -0.014570848053102338,
+            -0.00022620258237138677,
+            0.014454101124666536,
+            0.011056312437637014,
+            -0.004035526363810078,
+            -0.012459378363360557,
+            -0.0022595493558020706,
+            -0.030370952502579162,
+            0.012766010284884204,
+            0.1141689619126009,
+            -0.00766197047210299,
+            -0.00896655658602649,
+            -0.009481112393590025,
+            0.001435269335646347,
+            0.001338755472877713,
+            -0.011739143028767118,
+            0.07166096084053437,
+            -0.0077855767070652395,
+            0.0002139580430342728,
+            -0.011312198831181402,
+            0.001844005266872301,
+            -0.001825412693163496,
+            0.01371932421024605,
+            -0.005025814503687022,
+            -0.018213360991353643,
+            -0.0033612481616769416,
+            -0.02941403830641238,
+            -0.0041137865872933515,
+            -0.005683630256743215,
+            -0.004288198570156578,
+            -0.014982999363445122,
+            0.0067149665791261335,
+            -0.019661973497372313,
+            -0.012688556129040908,
+            0.001124391650175233,
+            -0.0033623453063430534,
+            0.00996944383866679,
+            -0.020496336814603385,
+            -0.0044354321069609536,
+            0.0019875840782527733,
+            0.009706675240563883,
+            0.0011600364428219214,
+            -0.012755178230821602,
+            -0.0026514442584739412,
+            0.004056350166031111,
+            0.01909951327913368,
+            0.014360281326789544,
+            0.003732344102045803,
+            0.008623457695718206,
+            0.014461868085333023,
+            -0.001692149528228451,
+            0.01948035376572433,
+            0.01172550239477637,
+            0.0031966531144076173,
+            0.0004779413449255626,
+            0.0030912562273611517,
+            -0.004597633235071031,
+            0.01582547400214653,
+            -0.014811177637844808,
+            -0.02411202772171917,
+            -0.0071272238837422185,
+            0.018015541137524113,
+            -0.013081287852028074,
+            1.8837689271256874e-05,
+            0.004062803284646432,
+            0.027569726868200254,
+            0.0010564850664255057,
+            -0.01179755720247972,
+            0.0028876134800690334,
+            -0.01638215925131053,
+            0.022944183106912663,
+            0.0026043713064243836,
+            0.03279173899194346,
+            -0.005103903273968361,
+            0.00578213669463383,
+            -0.020744194169889544,
+            -0.007588793365216579,
+            -0.00819862611489168,
+            0.0017401442433921013,
+            0.010993500174794084,
+            0.005411238350013134,
+            0.01620925462604661,
+            -0.00641615069699833,
+            -0.020751730783437995,
+            0.014776878596707538,
+            -0.009552193304486048,
+            -0.03393319554458277,
+            0.008881987295131347,
+            -0.029102297123590998,
+            -0.0045267103602711,
+            0.00368971742585232,
+            -0.014269341759285646,
+            -0.016264809810970175,
+            0.012837302095591106,
+            -0.0011890334180728564,
+            -0.008937992768323412,
+            0.0014444613725248846,
+            -0.017110187133760626,
+            -0.02385668037818897,
+            0.006017405364919834,
+            0.00676670672597817,
+            -0.0077994500447595,
+            0.018312452858615755,
+            -0.027015660037024585,
+            0.006123912473734307,
+            -0.012973676426738582,
+            0.009041974762537394,
+            -0.01580183887345168,
+            0.01163516332546347,
+            0.010887662470499361,
+            -0.001503467664709027,
+            -0.0076234124970223326,
+            0.01231202541338883,
+            0.004852720134894811,
+            0.033760365210093034,
+            0.028658326381800395,
+            0.000995999783627898,
+            -0.0025605367058226737,
+            -0.0020356794893366384,
+            0.011667525096213252,
+            -0.014049700287539817,
+            0.011218246246280464,
+            0.02658113044439717,
+            0.017568631325838108,
+            -0.019133062654008885,
+            0.018050678772708717,
+            -0.010356118256625172,
+            0.012872052420999813,
+            -0.025620154415048252,
+            -0.005135768485044442,
+            0.004936490920616527,
+            0.005163008729256001,
+            0.007425278892536464,
+            0.001410212306439329,
+            0.021891387289347516,
+            0.014999616600766989,
+            0.00027613811156650413,
+            0.00558422822386443,
+            0.01477009526981092,
+            0.010497782599293994,
+            -0.0005567894749233204,
+            -0.00467776506604011,
+            -0.016998035358377343,
+            0.00957271797745408,
+            -0.006065116857898469,
+            0.005138070401846389,
+            0.01698914855850691,
+            -0.01230257809343985,
+            0.011109844802663414,
+            0.007518553660463051,
+            -0.015504773927595823,
+            -0.017169933945839305,
+            0.005552829003043926,
+            -0.02252633790984703,
+            -0.01306722099657118,
+            0.02715689072730398,
+            -0.010513941399990838,
+            -0.02790582732974509,
+            -0.0009847768069205477,
+            0.009797865241108237,
+            -0.002772952073769185,
+            -0.0027968741526000576,
+            0.0037541733651479613,
+            0.010801137722284016,
+            -0.014658156970099418,
+            0.002075415233345957,
+            0.004735264309137368,
+            0.01930804315461151,
+            -0.0014151522239496115,
+            -0.015046118414100617,
+            -0.002288495894227231,
+            0.021592829677937263,
+            0.017174609713578173,
+            0.00265560378681856,
+            -0.016499825201963983,
+            -3.965296928410772e-05,
+            -0.0074142747014567635,
+            0.00021360696629503726,
+            0.00027431174904249525,
+            -0.0056391089282546054,
+            -0.018931681700770225,
+            -0.003888248121485032,
+            0.011181675419027736,
+            -0.00247252920175839,
+            0.01977860312223846,
+            -0.0011527575128223392,
+            -0.015763587433200915,
+            0.001960734212846797,
+            -0.005280860341480037,
+            -0.010372829570478649,
+            0.01351900743232294,
+            -0.009359942374603997,
+            0.006788746813046239,
+            -0.003212015225284119,
+            0.004611143085183367,
+            -0.0003308099698088422,
+            -0.017017265064185255,
+            0.02924142709831441,
+            -0.0018238549856433466,
+            -0.004769274041546295,
+            -0.0014058363976586562,
+            -0.033393331166741945,
+            0.015017165991303838,
+            -0.014574579250749552,
+            0.006745699409061803,
+            0.002705853802154516,
+            0.01272741296341735,
+            0.003253095246601883,
+            -0.008898616461779179,
+            0.0027485606928249326,
+            0.017040360927723885,
+            0.013756402798365262,
+            -0.007292864293090161,
+            0.008694855886257706,
+            -0.0028934389922887676,
+            -0.0037483208383542015,
+            0.01363963339139356,
+            0.004170611039133969,
+            -0.006282657671540625,
+            -0.0009784372731140945
+          ],
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "Mixture-of-Experts (MoE)",
+            "fine-grained expert segmentation",
+            "shared expert isolation"
+          ],
+          "title": "Deepseekmoe: Towards Ultimate Expert Specialization In Mixture-Of-Experts Language Models"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "1084fc41-4c97-458c-a6c9-2140fb99d5a9",
+        "properties": {
+          "page_content": "Appendices A. Overview Of Hyper-Parameters\n\nWe present the overview of hyper-parameters for DeepSeekMoE across various sizes in Table 7.\n\n# Params # Layers Hidden # Attn # Shared # Routed Relative Sequence Batch Size Learning Size Heads Experts Experts Expert Size Length (Sequence) Rate 2.0B 9 1280 10 1 63 (7 activated) 0.25 2048 2048 1.08e-3 16.4B 28 2048 16 2 64 (6 activated) 0.25 4096 4608 4.2e-4 144.6B 62 4096 32 4 128 (12 activated) 0.125 4096 4608 3.0e-4\n\nTable 7 | Overview of hyper-parameters for DeepSeekMoE across various sizes. The relative expert size is in comparison to a standard FFN.\n\nB. Comparing Deepseekmoe With Larger Models\n\nComparisons among DeepSeekMoE, GShard×1.2, and GShard×1.5 are shown in Table 8. Comparisons among DeepSeekMoE, Dense×4, and Dense×16 are shown in Table 9.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 2.3B 2.8B 1.9B # Activated Expert Params N/A 0.28B 0.35B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.824 1.808 1.808 HellaSwag (Acc.) 0-shot 53.7 54.4 54.8 PIQA (Acc.) 0-shot 71.8 71.1 72.3 ARC-easy (Acc.) 0-shot 46.8 47.3 49.4 ARC-challenge (Acc.) 0-shot 31.7 34.1 34.3 RACE-middle (Acc.) 5-shot 43.7 46.4 44.0 RACE-high (Acc.) 5-shot 31.9 32.4 31.7 HumanEval (Pass@1) 0-shot 3.7 3.0 4.9 MBPP (Pass@1) 3-shot 2.4 2.6 2.2 TriviaQA (EM) 5-shot 15.2 15.7 16.6 NaturalQuestions (EM) 5-shot 4.5 4.7 5.7\n\nTable 8 | Comparison between DeepSeekMoE and larger GShard models.\n\nAt a larger scale of 13B total parameters, we also compare DeepSeekMoE with GShard×1.2 and GShard×1.5, and show results in Table 10. At a larger scale, DeepSeekMoE even outperforms GShard×1.5 distinctly.\n\nMetric # Shot Dense×4 Dense×16 DeepSeekMoE Relative Expert Size N/A 1 1 0.25 # Experts N/A 4 + 0 16 + 0 1 + 63 # Activated Experts N/A 4 + 0 16 + 0 1 + 7 # Total Expert Params N/A 0.47B 1.89B 1.89B # Activated Expert Params N/A 0.47B 1.89B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.908 1.806 1.808 HellaSwag (Acc.) 0-shot 47.6 55.1 54.8 PIQA (Acc.) 0-shot 70.0 71.9 72.3 ARC-easy (Acc.) 0-shot 43.9 51.9 49.4 ARC-challenge (Acc.) 0-shot 30.5 33.8 34.3 RACE-middle (Acc.) 5-shot 42.4 46.3 44.0 RACE-high (Acc.) 5-shot 30.7 33.0 31.7 HumanEval (Pass@1) 0-shot 1.8 4.3 4.9 MBPP (Pass@1) 3-shot 0.2 2.2 2.2 TriviaQA (EM) 5-shot 9.9 16.5 16.6 NaturalQuestions (EM) 5-shot 3.0 6.3 5.7\n\nTable 9 | Comparison between DeepSeekMoE and larger dense baselines.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 15.9B 19.8B 13.3B # Activated Expert Params N/A 2.37B 2.82B 2.05B # Training Tokens N/A 100B 100B 100B HellaSwag (Acc.) 0-shot 66.6 67.7 69.1 PIQA (Acc.) 0-shot 75.6 76.0 75.7 ARC-easy (Acc.) 0-shot 56.8 56.8 58.8 ARC-challenge (Acc.) 0-shot 39.9 37.6 38.5 RACE-middle (Acc.) 5-shot 51.6 50.6 52.4 RACE-high (Acc.) 5-shot 37.4 36.3 38.5 HumanEval (Pass@1) 0-shot 6.1 6.1 9.8 MBPP (Pass@1) 3-shot 7.0 11.6 10.6 TriviaQA (EM) 5-shot 36.5 36.7 38.2 NaturalQuestions (EM) 5-shot 12.6 12.1 13.7\n\nTable 10 | Comparison between DeepSeekMoE and larger GShard models at a larger scale.\n\nC. Training Benchmark Curves Of Deepseekmoe 16B\n\nWe present the benchmark curves during training of DeepSeekMoE 16B and DeepSeek 7B (Dense) in Figure 7 for reference.",
+          "keyphrases": [
+            "DeepSeekMoE",
+            "hyper-parameters",
+            "GShard models",
+            "Dense baselines",
+            "training benchmark curves"
+          ],
+          "embedding": [
+            0.020979074761271477,
+            -0.0013986049452796578,
+            -0.019327709451317787,
+            0.0138933714479208,
+            0.0006998290773481131,
+            -0.02645554021000862,
+            0.0001438888575648889,
+            0.05624751001596451,
+            -0.006693925708532333,
+            0.024147000163793564,
+            -0.016437821090221405,
+            -0.0021316087804734707,
+            0.04061009734869003,
+            0.005270044784992933,
+            0.00756173487752676,
+            0.03029749169945717,
+            0.026859955862164497,
+            0.02480417490005493,
+            0.0019567832350730896,
+            -0.041183020919561386,
+            0.017339332029223442,
+            -0.04290178790688515,
+            -0.025309694930911064,
+            0.008383204229176044,
+            -0.016185062006115913,
+            0.009588027372956276,
+            0.022377679124474525,
+            0.014769605360925198,
+            -0.037071458995342255,
+            0.022091219201683998,
+            0.01640412025153637,
+            0.023371869698166847,
+            0.0032205828465521336,
+            -0.005885093938559294,
+            -0.02074316516518593,
+            -0.004515977576375008,
+            0.06709934026002884,
+            0.027870995923876762,
+            -0.010388433001935482,
+            0.018485177308321,
+            0.029404405504465103,
+            0.011500577442348003,
+            -0.013918647542595863,
+            -0.00502149760723114,
+            0.005880881100893021,
+            -0.009082507342100143,
+            -0.018165014684200287,
+            0.014019750989973545,
+            0.011744911782443523,
+            -0.05675303190946579,
+            -0.0348808728158474,
+            -0.017726896330714226,
+            -0.008109381422400475,
+            -0.013952348381280899,
+            -0.0411493182182312,
+            -0.03166239336133003,
+            0.028376515954732895,
+            0.061841931194067,
+            0.04182334616780281,
+            0.008939276449382305,
+            0.02399534359574318,
+            -0.006129428278654814,
+            -0.03993606939911842,
+            0.0007677582907490432,
+            0.008855023421347141,
+            -0.015452057123184204,
+            0.022579887881875038,
+            -0.005105750635266304,
+            -0.006647586356848478,
+            -0.020608359947800636,
+            -0.037745483219623566,
+            0.022377679124474525,
+            -0.003243752522394061,
+            0.006015686318278313,
+            0.02106332778930664,
+            -0.01328674703836441,
+            0.027803592383861542,
+            0.02632073499262333,
+            0.00384195102378726,
+            -0.013918647542595863,
+            0.017726896330714226,
+            0.0043074507266283035,
+            -0.01196397002786398,
+            0.03747587278485298,
+            0.04842880368232727,
+            -0.026287034153938293,
+            -0.040374185889959335,
+            -0.06318998336791992,
+            0.010321030393242836,
+            -0.008244186639785767,
+            -0.018384072929620743,
+            -0.008678090758621693,
+            0.03508307784795761,
+            -0.02180475741624832,
+            -0.03354967013001442,
+            0.06864959746599197,
+            -0.012494766153395176,
+            0.007115192245692015,
+            -0.03442590311169624,
+            -0.0069087715819478035,
+            0.004235835280269384,
+            0.00045523117296397686,
+            0.008661240339279175,
+            0.0043685343116521835,
+            0.005177366081625223,
+            -0.00362710515037179,
+            -0.01878848858177662,
+            0.0002004965499509126,
+            -0.004111561458557844,
+            0.0718512237071991,
+            0.012410513125360012,
+            -0.01227570790797472,
+            -0.048327699303627014,
+            -0.013927072286605835,
+            -0.02032189816236496,
+            -0.038655418902635574,
+            -0.029775120317935944,
+            -0.021568847820162773,
+            0.024281805381178856,
+            -0.002023132750764489,
+            0.033263206481933594,
+            0.020642060786485672,
+            0.030044730752706528,
+            -0.01792910508811474,
+            -0.043070293962955475,
+            0.04421613737940788,
+            0.0006208415725268424,
+            -0.0033933022059500217,
+            -0.0019631022587418556,
+            -0.029842523857951164,
+            0.02118128351867199,
+            -0.0036713380832225084,
+            -0.03408889099955559,
+            -0.030954666435718536,
+            0.00232539139688015,
+            -0.008694942109286785,
+            0.05213594809174538,
+            0.03949795290827751,
+            -0.01417140755802393,
+            -0.0003188461414538324,
+            0.0034438541624695063,
+            -0.04034048691391945,
+            0.029337003827095032,
+            -0.017246652394533157,
+            -0.03181404992938042,
+            0.022327126935124397,
+            0.017895404249429703,
+            0.06558278203010559,
+            -0.0028098479378968477,
+            0.041351526975631714,
+            0.0015386760933324695,
+            0.04003717377781868,
+            0.02350667491555214,
+            0.015763795003294945,
+            -0.009074081666767597,
+            -0.004326407797634602,
+            -0.03693665191531181,
+            0.00613785395398736,
+            -0.03360022231936455,
+            -0.015098193660378456,
+            -0.001018938492052257,
+            -0.004907755646854639,
+            -0.017288779839873314,
+            -0.0007556469063274562,
+            -0.002022079424932599,
+            0.047788478434085846,
+            0.023355018347501755,
+            -0.013244620524346828,
+            -0.06804297864437103,
+            0.03263973444700241,
+            0.04620451480150223,
+            0.007477481383830309,
+            -0.006196831353008747,
+            0.03788029029965401,
+            0.003601829055696726,
+            0.02229342609643936,
+            0.020709464326500893,
+            0.01904124952852726,
+            -0.06531316787004471,
+            0.04522717744112015,
+            -0.015300401486456394,
+            0.003485980909317732,
+            0.01658105105161667,
+            0.011264667846262455,
+            0.0310389194637537,
+            -0.0018630513222888112,
+            0.029033690690994263,
+            0.026859955862164497,
+            -0.012604295276105404,
+            0.02136664092540741,
+            0.031932003796100616,
+            0.008248399011790752,
+            0.04553049057722092,
+            -0.001733511919155717,
+            0.0005845073610544205,
+            -0.018687384203076363,
+            0.008838172070682049,
+            0.03353281691670418,
+            -0.017423585057258606,
+            0.018215566873550415,
+            0.0006161023629829288,
+            -0.027685638517141342,
+            -0.0010357891442254186,
+            -0.013337299227714539,
+            0.027651937678456306,
+            0.0262701828032732,
+            -0.008509584702551365,
+            0.0066855004988610744,
+            0.008404267951846123,
+            -0.016067106276750565,
+            -0.031190576031804085,
+            -0.013935497961938381,
+            -0.017069721594452858,
+            -0.0532480925321579,
+            -0.0025634069461375475,
+            0.03521788492798805,
+            -0.007785005960613489,
+            0.022512484341859818,
+            0.010860252194106579,
+            -0.0323195718228817,
+            0.003799824509769678,
+            0.009933465160429478,
+            0.014179832302033901,
+            0.006390613969415426,
+            -0.017229802906513214,
+            -0.02350667491555214,
+            -0.0032016257755458355,
+            0.008753919042646885,
+            0.032841939479112625,
+            -0.010919229127466679,
+            0.001601866097189486,
+            -0.001936773071065545,
+            -0.01959731988608837,
+            0.0043348330073058605,
+            0.03257232904434204,
+            -0.01013567391782999,
+            -0.00016798004799056798,
+            0.007005663122981787,
+            0.02662404626607895,
+            0.018485177308321,
+            0.01279807835817337,
+            -0.015266700647771358,
+            -0.04192444682121277,
+            -0.016564201563596725,
+            0.01824926771223545,
+            -0.00505098607391119,
+            -0.026742001995444298,
+            -0.01732248067855835,
+            0.057629264891147614,
+            0.024197550490498543,
+            0.023304466158151627,
+            -0.06281927227973938,
+            -0.02283264696598053,
+            0.005535442847758532,
+            -0.01727192848920822,
+            0.05668563023209572,
+            0.059415437281131744,
+            0.005628121085464954,
+            -0.00452861562371254,
+            -0.04121671989560127,
+            0.04023938253521919,
+            -0.01184601616114378,
+            -0.049979064613580704,
+            0.018704235553741455,
+            -0.01627773977816105,
+            0.01627773977816105,
+            0.033633921295404434,
+            0.027685638517141342,
+            0.0044443621300160885,
+            0.008231548592448235,
+            0.04266587644815445,
+            -0.023152809590101242,
+            -0.0458001010119915,
+            -0.02374258264899254,
+            0.021349789574742317,
+            -0.018721086904406548,
+            -0.007696540094912052,
+            -0.02852817252278328,
+            0.02633758634328842,
+            0.05055198818445206,
+            0.02662404626607895,
+            0.018350372090935707,
+            0.016142934560775757,
+            0.025056935846805573,
+            -0.024096447974443436,
+            0.0011490045581012964,
+            -0.023675180971622467,
+            -0.043306201696395874,
+            -0.017710046842694283,
+            -0.015199298039078712,
+            0.0068076676689088345,
+            0.006015686318278313,
+            -0.022849498316645622,
+            0.005493315868079662,
+            0.02606797404587269,
+            0.014407316222786903,
+            -0.03140963613986969,
+            0.011702785268425941,
+            0.05119231343269348,
+            -0.035116780549287796,
+            -0.03194885700941086,
+            0.0317297987639904,
+            0.017844852060079575,
+            -0.0067992424592375755,
+            -0.01885589212179184,
+            0.01466007623821497,
+            -9.174066690320615e-06,
+            0.01113828830420971,
+            -0.009284715168178082,
+            0.0007667051395401359,
+            -0.011938693933188915,
+            -0.0016303015872836113,
+            0.007801856845617294,
+            -0.03386983275413513,
+            -0.008905574679374695,
+            0.019176054745912552,
+            0.022124920040369034,
+            0.034510158002376556,
+            -0.06474024802446365,
+            -0.01166065875440836,
+            0.06969434022903442,
+            0.01836722157895565,
+            -0.04610341414809227,
+            0.025478200986981392,
+            0.019799528643488884,
+            -0.007351101376116276,
+            0.004520190414041281,
+            0.04559789225459099,
+            -0.04684484004974365,
+            -0.014002900570631027,
+            0.029050542041659355,
+            -0.04040788859128952,
+            -0.03139278292655945,
+            0.01892329379916191,
+            0.025023233145475388,
+            -0.06376291066408157,
+            -0.03761067986488342,
+            0.005404850002378225,
+            0.012258856557309628,
+            -0.007397440727800131,
+            -0.021956413984298706,
+            0.0029320153407752514,
+            -0.036094117909669876,
+            -0.01426408626139164,
+            0.0003749272436834872,
+            -0.005135239567607641,
+            -0.028545022010803223,
+            0.002346454653888941,
+            0.020793717354536057,
+            0.01627773977816105,
+            0.00159554707352072,
+            0.012191453948616982,
+            0.03424054756760597,
+            -0.006175767630338669,
+            0.0027550833765417337,
+            -0.028443917632102966,
+            -0.02645554021000862,
+            0.01579749584197998,
+            0.0032079447992146015,
+            0.00201260088942945,
+            -0.017162399366497993,
+            -0.04377802088856697,
+            -0.03366762399673462,
+            0.028545022010803223,
+            -0.020557807758450508,
+            -0.013623761013150215,
+            0.021821608766913414,
+            -0.004482276272028685,
+            -0.011862866580486298,
+            0.012292558327317238,
+            0.002847762079909444,
+            0.030819861218333244,
+            -0.03619522228837013,
+            0.006053600460290909,
+            0.017844852060079575,
+            -0.001491283648647368,
+            0.035116780549287796,
+            -0.010422134771943092,
+            -0.02392794005572796,
+            0.021838458254933357,
+            -0.049069128930568695,
+            0.024871578440070152,
+            0.03848691284656525,
+            0.012831779196858406,
+            0.03066820651292801,
+            0.031055770814418793,
+            -0.009326841682195663,
+            0.011610106565058231,
+            -0.012385237030684948,
+            -0.0024496649857610464,
+            0.033768728375434875,
+            -0.059482838958501816,
+            -0.023034855723381042,
+            0.010801274329423904,
+            0.018872741609811783,
+            0.03943055123090744,
+            0.017221376299858093,
+            -0.010834976099431515,
+            -0.022546187043190002,
+            -0.008138869889080524,
+            0.013640611432492733,
+            0.04927133768796921,
+            -0.0011468982556834817,
+            -0.003104734467342496,
+            -0.005493315868079662,
+            0.026371287181973457,
+            0.006803454831242561,
+            0.012233581393957138,
+            0.023708881810307503,
+            0.0004207399906590581,
+            0.022074367851018906,
+            -0.0037366345059126616,
+            -0.0342910997569561,
+            -0.008581199683248997,
+            0.007616499438881874,
+            0.03515048325061798,
+            -0.0021610974799841642,
+            0.021097028627991676,
+            -0.02473677322268486,
+            0.01978267729282379,
+            -0.039767563343048096,
+            0.020490406081080437,
+            0.031679246574640274,
+            -0.0028561872895807028,
+            -0.0045328279957175255,
+            0.01227570790797472,
+            -0.015157170593738556,
+            0.0019136033952236176,
+            0.029185347259044647,
+            -0.03471236303448677,
+            0.014778031036257744,
+            0.012115626595914364,
+            0.02945495769381523,
+            -0.04320509731769562,
+            -0.022984303534030914,
+            -0.013775416649878025,
+            -0.011685934849083424,
+            0.04320509731769562,
+            -0.04023938253521919,
+            -0.02259673923254013,
+            -0.0018862209981307387,
+            0.0372062623500824,
+            0.019850080832839012,
+            -0.013177217915654182,
+            -0.04327250272035599,
+            -0.033515967428684235,
+            -0.05008016899228096,
+            -0.012284132651984692,
+            -0.03922834247350693,
+            0.002550769131630659,
+            -0.07731083780527115,
+            0.01941196247935295,
+            -0.050113871693611145,
+            0.020642060786485672,
+            0.046811141073703766,
+            -0.008572774939239025,
+            0.030836712568998337,
+            -0.0196478720754385,
+            0.014205108396708965,
+            -0.06251595914363861,
+            -0.0068076676689088345,
+            0.025747811421751976,
+            0.02945495769381523,
+            -0.003494406118988991,
+            -0.003734528087079525,
+            0.019445665180683136,
+            -0.01279807835817337,
+            -0.0065675457008183,
+            -0.008930850774049759,
+            0.02608482539653778,
+            0.008496946655213833,
+            -0.010674894787371159,
+            0.024601968005299568,
+            0.008265250362455845,
+            0.015502609312534332,
+            0.03761067986488342,
+            -0.011365772224962711,
+            -0.04023938253521919,
+            -0.0034417477436363697,
+            0.029117945581674576,
+            0.029707718640565872,
+            -0.031072622165083885,
+            0.014339913614094257,
+            -0.003869333304464817,
+            0.005906157195568085,
+            0.006512781139463186,
+            0.004033627454191446,
+            0.029758270829916,
+            -0.006394826341420412,
+            0.026708299294114113,
+            -0.014811732806265354,
+            0.019850080832839012,
+            -0.026421839371323586,
+            0.03147703781723976,
+            0.012132477015256882,
+            -0.030162686482071877,
+            0.005910370033234358,
+            -0.01628616452217102,
+            0.016918065026402473,
+            0.019361412152647972,
+            -0.013143517076969147,
+            -0.027500281110405922,
+            0.013446828350424767,
+            0.013480530120432377,
+            0.012225155718624592,
+            0.004494914319366217,
+            0.027399176731705666,
+            0.03006158210337162,
+            -0.02227657660841942,
+            -0.03545379266142845,
+            0.0174404364079237,
+            -0.021872159093618393,
+            0.015039216727018356,
+            -0.03427424654364586,
+            0.01426408626139164,
+            0.00647486699745059,
+            -0.03595931455492973,
+            0.04010457545518875,
+            0.020642060786485672,
+            0.0019451983971521258,
+            0.004128412343561649,
+            -0.005990410689264536,
+            -0.009326841682195663,
+            0.05129341781139374,
+            -0.015957577154040337,
+            0.019816379994153976,
+            0.015637414529919624,
+            0.012882331386208534,
+            0.03683554753661156,
+            -0.03322950750589371,
+            -0.014575823210179806,
+            0.01347210444509983,
+            0.014499994926154613,
+            -0.03445960581302643,
+            -0.01365746185183525,
+            -0.02148459479212761,
+            -0.011450025252997875,
+            0.040374185889959335,
+            -0.014525271020829678,
+            -0.02131608873605728,
+            0.007658625952899456,
+            0.0023359230253845453,
+            -0.016934916377067566,
+            0.008092531003057957,
+            -0.015047641471028328,
+            -0.005194216500967741,
+            0.04064379632472992,
+            -0.023978492245078087,
+            0.03622892498970032,
+            0.031561292707920074,
+            0.016859088093042374,
+            -0.042699579149484634,
+            0.0006656011682935059,
+            0.006112577859312296,
+            -0.014188257977366447,
+            -0.01597442850470543,
+            0.034442752599716187,
+            0.014887560158967972,
+            0.01450842060148716,
+            -0.023152809590101242,
+            -0.006205256562680006,
+            0.02362462878227234,
+            -0.023304466158151627,
+            0.02578151412308216,
+            -0.00396411819383502,
+            -0.02332131750881672,
+            -0.03213421255350113,
+            0.007650200743228197,
+            -0.022680992260575294,
+            0.012755951844155788,
+            -0.003985181916505098,
+            0.012292558327317238,
+            0.01236838661134243,
+            -0.025259142741560936,
+            0.0036587000358849764,
+            0.019209755584597588,
+            0.000130460990476422,
+            -0.0003262182872276753,
+            -0.009048805572092533,
+            0.004604443442076445,
+            0.018198715522885323,
+            -0.001285916194319725,
+            0.005215280223637819,
+            0.022006966173648834,
+            0.031005218625068665,
+            0.004381172358989716,
+            -0.003250071546062827,
+            -0.020793717354536057,
+            0.02797210030257702,
+            -0.02357407659292221,
+            0.02773619070649147,
+            0.028174307197332382,
+            0.014921261928975582,
+            -0.03855431452393532,
+            -0.0021800545509904623,
+            -0.016496798023581505,
+            -0.008353715762495995,
+            0.01776059903204441,
+            0.022883199155330658,
+            -0.024281805381178856,
+            -0.0024075384717434645,
+            -0.004147369414567947,
+            0.0215014461427927,
+            0.003218476427718997,
+            -0.030769309028983116,
+            -0.010843401774764061,
+            0.009866062551736832,
+            0.00896455254405737,
+            -0.01456739753484726,
+            0.0019831124227494,
+            0.011888142675161362,
+            0.006150492001324892,
+            -0.0016060788184404373,
+            -0.00011077993985963985,
+            0.01897384598851204,
+            -0.002618171740323305,
+            0.01567954197525978,
+            -0.013750140555202961,
+            -0.013446828350424767,
+            -0.011087736114859581,
+            0.05247296392917633,
+            -0.0032205828465521336,
+            0.03899243474006653,
+            0.014188257977366447,
+            0.009722832590341568,
+            0.04431724175810814,
+            -0.013253046199679375,
+            -0.009309991262853146,
+            0.01721295155584812,
+            -0.01750783808529377,
+            -0.03568970412015915,
+            0.004103136248886585,
+            -0.005025709979236126,
+            -0.028983140364289284,
+            0.01071702130138874,
+            0.005038348026573658,
+            -0.004067328758537769,
+            0.0009341586264781654,
+            0.008846597746014595,
+            -0.026843104511499405,
+            0.03247122839093208,
+            0.005944071337580681,
+            0.011879716999828815,
+            0.034746065735816956,
+            -0.016201911494135857,
+            -0.007974576205015182,
+            0.013792267069220543,
+            -0.01518244668841362,
+            0.028545022010803223,
+            -0.01347210444509983,
+            0.01047268696129322,
+            0.01707814633846283,
+            0.030381744727492332,
+            0.03622892498970032,
+            -0.02492213062942028,
+            -0.06261706352233887,
+            0.011711210012435913,
+            -0.004638144746422768,
+            0.011559554375708103,
+            0.016260888427495956,
+            0.01233468484133482,
+            -0.005640759132802486,
+            0.01811446249485016,
+            -0.007540671620517969,
+            0.028999989852309227,
+            -0.006121003068983555,
+            -0.0326734334230423,
+            0.020962223410606384,
+            0.006007261108607054,
+            0.023034855723381042,
+            0.014339913614094257,
+            -0.03895873203873634,
+            -0.009596452116966248,
+            -0.004596018232405186,
+            0.010270479135215282,
+            -0.03383613005280495,
+            0.00483614020049572,
+            0.010759147815406322,
+            0.017912253737449646,
+            0.002041036495938897,
+            -0.007595436181873083,
+            0.010556939989328384,
+            -0.005227918270975351,
+            -0.016303015872836113,
+            0.0131350914016366,
+            -0.0022664139978587627,
+            -0.013059263117611408,
+            0.026303883641958237,
+            -0.03545379266142845,
+            -0.016993893310427666,
+            -4.235703818267211e-05,
+            0.0013027668464928865,
+            0.013244620524346828,
+            0.003953586798161268,
+            0.0015492078382521868,
+            -0.02896628901362419,
+            -0.03848691284656525,
+            0.04475535824894905,
+            0.022967452183365822,
+            -0.0033153677359223366,
+            -0.0003277980431448668,
+            -0.0004360108869150281,
+            -0.005792415235191584,
+            0.023894239217042923,
+            0.03164554387331009,
+            -0.020658912137150764,
+            0.005004646722227335,
+            -0.027466580271720886,
+            -0.019698424264788628,
+            0.015283551067113876,
+            0.009031955152750015,
+            0.033701326698064804,
+            0.006217894610017538,
+            0.006912984419614077,
+            0.018384072929620743,
+            0.008610688149929047,
+            -0.004507552366703749,
+            -0.003808249719440937,
+            -0.022697841748595238,
+            0.03230271860957146,
+            0.0390598364174366,
+            0.0149381123483181,
+            0.014070303179323673,
+            -0.034746065735816956,
+            -0.003814568743109703,
+            0.010885528288781643,
+            0.02492213062942028,
+            0.02424810267984867,
+            0.006748690269887447,
+            0.001917816000059247,
+            0.015157170593738556,
+            -0.03703775629401207,
+            -0.013573208823800087,
+            0.009798659943044186,
+            -0.014129280112683773,
+            -0.027988949790596962,
+            -0.006087301764637232,
+            0.022916901856660843,
+            -0.005897731985896826,
+            0.0021526720374822617,
+            -0.004659208003431559,
+            -0.011542703956365585,
+            -0.015527885407209396,
+            -0.004541253671050072,
+            0.006693925708532333,
+            0.02380998618900776,
+            0.0044317240826785564,
+            0.024770474061369896,
+            -0.009343692101538181,
+            0.011087736114859581,
+            0.006546482443809509,
+            -0.014162981882691383,
+            0.016488373279571533,
+            -0.013059263117611408,
+            -0.026708299294114113,
+            -0.0092425886541605,
+            0.01781114935874939,
+            -0.00644537853077054,
+            -0.0016945448005571961,
+            -0.01647152192890644,
+            0.005004646722227335,
+            0.036397431045770645,
+            0.008265250362455845,
+            0.017558390274643898,
+            0.023523524403572083,
+            -0.002397006843239069,
+            -0.009318416006863117,
+            0.0131350914016366,
+            0.0055902074091136456,
+            0.0055859945714473724,
+            0.021585699170827866,
+            -0.001512346905656159,
+            0.04387912526726723,
+            -0.05058568716049194,
+            0.008383204229176044,
+            0.027803592383861542,
+            -0.0057081617414951324,
+            -0.005939858499914408,
+            -0.032841939479112625,
+            0.01653892546892166,
+            -0.01678325980901718,
+            0.009798659943044186,
+            -0.011155138723552227,
+            0.028427068144083023,
+            -0.03723996505141258,
+            -0.008378991857171059,
+            -3.804563675657846e-05,
+            0.00811359379440546,
+            0.016083957627415657,
+            0.018653683364391327,
+            -0.006580183748155832,
+            -0.02497268095612526,
+            -0.003673444502055645,
+            0.012882331386208534,
+            0.04290178790688515,
+            -0.003953586798161268,
+            -0.014685352332890034,
+            0.0215014461427927,
+            -0.011171989142894745,
+            -0.020119691267609596,
+            0.0027635085862129927,
+            -0.024821026250720024,
+            0.029471809044480324,
+            0.03612782061100006,
+            -0.041958149522542953,
+            -0.024450311437249184,
+            0.002029451774433255,
+            -0.010936079546809196,
+            0.002504429779946804,
+            0.021012775599956512,
+            0.02743287943303585,
+            -0.004170538857579231,
+            -0.007629137486219406,
+            0.0206926129758358,
+            -0.005316384136676788,
+            0.015949152410030365,
+            0.016665305942296982,
+            -0.011348920874297619,
+            -0.0069551109336316586,
+            -0.017844852060079575,
+            -0.03270713612437248,
+            -0.00037361079012043774,
+            -0.0186031311750412,
+            -0.03481346741318703,
+            -0.0346449613571167,
+            -0.01398605015128851,
+            -0.010278903879225254,
+            -0.003949373960494995,
+            0.0364985354244709,
+            0.02394479140639305,
+            0.0251074880361557,
+            0.015746943652629852,
+            -0.013278322294354439,
+            0.05405692383646965,
+            -0.006816092878580093,
+            0.0460360087454319,
+            -0.020187092944979668,
+            0.0008951914496719837,
+            -0.019816379994153976,
+            0.03226901963353157,
+            -0.014390465803444386,
+            0.011525853537023067,
+            -0.009899764321744442,
+            0.006533844396471977,
+            0.0009504826739430428,
+            0.007574372924864292,
+            -0.0019451983971521258,
+            -0.009419520385563374,
+            0.027079014107584953,
+            -0.0023274978157132864,
+            0.029471809044480324,
+            0.020406153053045273,
+            0.004600231070071459,
+            0.005076262168586254,
+            -0.029707718640565872,
+            -0.00035149429459124804,
+            0.005067836958914995,
+            -0.0066855004988610744,
+            -0.018148163333535194,
+            0.0024201765190809965,
+            -0.007001450285315514,
+            0.007999852299690247,
+            -0.0039051410276442766,
+            -0.009225737303495407,
+            0.014104004949331284,
+            0.025629857555031776,
+            0.0069845994003117085,
+            -0.04391282796859741,
+            0.0035702341701835394,
+            -0.03936314582824707,
+            0.002685574349015951,
+            0.010818125680088997,
+            0.010093546472489834,
+            -0.002889888593927026,
+            0.0057123745791614056,
+            0.009613302536308765,
+            -0.004202133975923061,
+            -0.020962223410606384,
+            0.002352773677557707,
+            -0.015047641471028328,
+            -0.006146279163658619,
+            -0.011281518265604973,
+            3.797981480602175e-05,
+            -0.003614467103034258,
+            -0.0019262413261458278,
+            -0.005986197851598263,
+            -0.0067318398505449295,
+            -0.027146417647600174,
+            -0.02822485938668251,
+            -0.006108365021646023,
+            0.0002985726750921458,
+            -0.00706885289400816,
+            -0.03024693951010704,
+            0.03134223073720932,
+            -0.008539073169231415,
+            0.007136255502700806,
+            -0.0093015655875206,
+            0.009882913902401924,
+            -0.002344348467886448,
+            -0.005960921756923199,
+            -0.004553891718387604,
+            -0.026421839371323586,
+            -0.013994474895298481,
+            -0.03895873203873634,
+            0.016741132363677025,
+            -0.008471670560538769,
+            0.007106767036020756,
+            -0.010506387799978256,
+            0.001101612113416195,
+            -0.018148163333535194,
+            -0.0009278396028093994,
+            -0.002559194341301918,
+            0.027651937678456306,
+            -0.024163849651813507,
+            0.009074081666767597,
+            0.01616821065545082,
+            0.015485758893191814,
+            -0.0346449613571167,
+            0.02008599042892456,
+            0.012191453948616982,
+            -0.019243456423282623,
+            -0.0025423436891287565,
+            0.018350372090935707,
+            0.02576466277241707,
+            -0.013901796191930771,
+            -0.014162981882691383,
+            -0.004326407797634602,
+            0.009630153886973858,
+            -0.025006383657455444,
+            0.00954589992761612,
+            -0.010203076526522636,
+            0.017356181517243385,
+            0.025174889713525772,
+            -0.011256242170929909,
+            0.04023938253521919,
+            -0.03501567617058754,
+            -0.012831779196858406,
+            0.013497380539774895,
+            -0.023675180971622467,
+            0.013177217915654182,
+            -0.0003612360742408782,
+            -0.016690582036972046,
+            0.021754205226898193,
+            -0.006428527645766735,
+            0.01841777376830578,
+            -0.03693665191531181,
+            0.030078433454036713,
+            0.03012898378074169,
+            0.006656011566519737,
+            -0.011079310439527035,
+            -0.019513066858053207,
+            -0.01056536566466093,
+            0.0057418630458414555,
+            -0.0015397293027490377,
+            0.025730961933732033,
+            -0.014331488870084286,
+            -0.02466936968266964,
+            0.004633932374417782,
+            -0.0030478634871542454,
+            0.004692909307777882,
+            -0.02643868885934353,
+            0.004105242434889078,
+            -0.009832361713051796,
+            -0.0017924892017617822,
+            0.0006161023629829288,
+            -0.005990410689264536,
+            -0.009200462140142918,
+            -0.028696678578853607,
+            -0.056416019797325134,
+            0.0014891773462295532,
+            -0.0037913990672677755,
+            -0.023725733160972595,
+            -0.017743747681379318,
+            -0.012402087450027466,
+            -0.014449442736804485,
+            0.018518878147006035,
+            -0.0017208738718181849,
+            -0.023793134838342667,
+            -0.011862866580486298,
+            0.002586576621979475,
+            0.0017630006186664104,
+            0.0024412397760897875,
+            0.02362462878227234,
+            0.011163563467562199,
+            0.031493887305259705,
+            -0.018097611144185066,
+            0.02694420889019966,
+            -0.017238227650523186,
+            -0.03831840679049492,
+            0.03980126604437828,
+            -0.0009425839525647461,
+            -0.022697841748595238,
+            0.010801274329423904,
+            0.0009594346047379076,
+            -0.006483292207121849,
+            0.044115033000707626,
+            -0.0205746591091156,
+            -0.013429977931082249,
+            0.010177800431847572,
+            -0.006441165693104267,
+            -0.016631603240966797,
+            -0.03299359604716301,
+            -0.015393080189824104,
+            0.03240382298827171,
+            -0.006373763084411621,
+            0.013783842325210571,
+            0.035487495362758636,
+            -0.023776285350322723,
+            -0.029353853315114975,
+            0.012073500081896782,
+            -0.020675763487815857,
+            -0.01615135930478573,
+            0.0027782530523836613,
+            0.018636832013726234,
+            0.012098775245249271,
+            -0.0012248324928805232,
+            0.0006171555141918361,
+            -0.004945669323205948,
+            -0.012250431813299656,
+            0.003391195787116885,
+            -0.023776285350322723,
+            -0.035656001418828964,
+            -0.024012194946408272,
+            -0.01162695698440075,
+            0.03280824050307274,
+            0.03457755967974663,
+            -0.00030146888457238674,
+            -0.022495634853839874,
+            -0.02204066701233387,
+            -0.0029256963171064854,
+            0.039329446852207184,
+            0.013758566230535507,
+            0.012410513125360012,
+            -0.005333234556019306,
+            0.010396858677268028,
+            0.0005781883373856544,
+            0.006281084381043911,
+            -0.039093535393476486,
+            -0.008197847753763199,
+            0.019058099016547203,
+            0.001322777010500431,
+            -0.01629459112882614,
+            0.018771637231111526,
+            0.00517315324395895,
+            -0.013497380539774895,
+            -0.022259725257754326,
+            0.018518878147006035,
+            0.013674312271177769,
+            0.008004064671695232,
+            0.0044570001773536205,
+            -0.01873793639242649,
+            0.0005866136634722352,
+            0.009192036464810371,
+            -0.01285705529153347,
+            -0.0011142500443384051,
+            0.021400341764092445,
+            0.012739100493490696,
+            -0.008383204229176044,
+            0.012452639639377594,
+            0.013969198800623417,
+            -0.007115192245692015,
+            -0.008172571659088135,
+            8.287103264592588e-05,
+            0.014778031036257744,
+            -0.007148893550038338,
+            -2.983422928082291e-05,
+            -0.011525853537023067,
+            0.010700170882046223,
+            0.004709760192781687,
+            0.011778613552451134,
+            -0.02443346008658409,
+            -0.006592821795493364,
+            0.0055902074091136456,
+            0.02160254865884781,
+            -0.0079324496909976,
+            -0.013708014041185379,
+            0.009908189997076988,
+            -0.024450311437249184,
+            -0.02185530960559845,
+            0.00659703416749835,
+            0.0009615409071557224,
+            -0.009402669966220856,
+            0.033751875162124634,
+            0.0323195718228817,
+            -0.0010084068635478616,
+            -0.0011511108605191112,
+            0.00995031651109457,
+            0.008180996403098106,
+            0.017608942463994026,
+            -0.022883199155330658,
+            -0.006003048736602068,
+            -0.0013817542931064963,
+            0.016067106276750565,
+            0.04856361076235771,
+            -0.005826116539537907,
+            -0.0022011178079992533,
+            0.015898600220680237,
+            0.0920720174908638,
+            -0.00039783361717127264,
+            0.0067866044119000435,
+            -0.02394479140639305,
+            -0.030769309028983116,
+            0.005017284769564867,
+            0.001989431446418166,
+            0.019058099016547203,
+            0.05786517634987831,
+            0.003043650882318616,
+            0.0014407316921278834,
+            -0.0022179684601724148,
+            0.003943054936826229,
+            -0.0085432855412364,
+            -0.022512484341859818,
+            0.015839623287320137,
+            -0.0067866044119000435,
+            0.010683320462703705,
+            -6.480264710262418e-05,
+            0.01659790240228176,
+            0.004101030062884092,
+            0.008193634450435638,
+            0.006146279163658619,
+            0.017406733706593513,
+            -0.021535146981477737,
+            0.026421839371323586,
+            -0.010219926945865154,
+            0.011525853537023067,
+            -0.014702202752232552,
+            -0.027517132461071014,
+            0.019799528643488884,
+            -4.670134876505472e-05,
+            -0.007490119431167841,
+            -0.004002032335847616,
+            -0.011348920874297619,
+            0.02606797404587269,
+            -0.021400341764092445,
+            0.022849498316645622,
+            0.01261272095143795,
+            -0.020119691267609596,
+            0.010497963055968285,
+            0.008585412986576557,
+            0.0010257840622216463,
+            0.015763795003294945,
+            -0.008821321651339531,
+            0.0161008071154356,
+            0.01242736354470253,
+            -0.027112716808915138,
+            0.009958741255104542,
+            0.014550547115504742,
+            -0.009461646899580956,
+            0.01546048279851675,
+            0.008753919042646885,
+            0.001712448545731604,
+            -0.0010089334100484848,
+            -0.01075072307139635,
+            -0.0043769595213234425,
+            -0.009748108685016632,
+            0.007641775533556938,
+            -0.005552293267101049,
+            0.00820627249777317,
+            0.016134509816765785,
+            0.04782218113541603,
+            0.008320014923810959,
+            0.003953586798161268,
+            3.060764720430598e-05,
+            0.00025091689894907176,
+            -0.015966001898050308,
+            -0.022192321717739105,
+            -0.013649036176502705,
+            -0.009832361713051796,
+            0.024821026250720024,
+            -0.005792415235191584,
+            0.006390613969415426,
+            -0.017179250717163086,
+            0.005463827401399612,
+            0.0017177143599838018,
+            0.010843401774764061,
+            0.0033722389489412308,
+            -0.042396266013383865,
+            -0.0045791673474013805,
+            0.014213534072041512,
+            -0.011104586534202099,
+            -0.001104771508835256,
+            -0.031611841171979904,
+            0.003911459818482399,
+            -0.0158648993819952,
+            -0.0044907014816999435,
+            0.016749558970332146,
+            -0.020153392106294632,
+            0.0006840315763838589,
+            0.0032816664315760136,
+            -0.015940727666020393,
+            0.005122601520270109,
+            -0.008737068623304367,
+            0.0016840130556374788,
+            0.018215566873550415,
+            0.002660298254340887,
+            -0.01298343576490879,
+            -0.002645554021000862,
+            -0.024568265303969383,
+            0.01750783808529377,
+            0.029758270829916,
+            -0.008008277043700218,
+            -0.01941196247935295,
+            -0.005177366081625223,
+            0.014247234910726547,
+            -0.004145262762904167,
+            -0.00814308226108551,
+            0.0031405421905219555,
+            0.003479661885648966,
+            -0.020052287727594376,
+            0.00356602156534791,
+            0.003753485158085823,
+            0.01585647277534008,
+            0.010489537380635738,
+            0.014769605360925198,
+            -0.029286451637744904,
+            -0.023472972214221954,
+            0.007102554198354483,
+            -0.019378261640667915,
+            0.005463827401399612,
+            0.03986866772174835,
+            -0.005404850002378225,
+            0.014348339289426804,
+            0.011011907830834389,
+            0.022006966173648834,
+            -0.023658329620957375,
+            0.017524689435958862,
+            0.0020642061717808247,
+            -0.017710046842694283,
+            -0.010321030393242836,
+            0.016673730686306953,
+            0.00954589992761612,
+            0.02766878716647625,
+            -0.019260307773947716,
+            -0.004570742137730122,
+            0.007696540094912052,
+            0.03398778662085533,
+            0.003955692984163761,
+            -0.014617949724197388,
+            0.027517132461071014,
+            0.047114454209804535,
+            0.008897149935364723,
+            0.01652207411825657,
+            -0.01678325980901718,
+            -0.005101538263261318,
+            0.007785005960613489,
+            0.009849212132394314,
+            0.013337299227714539,
+            -0.0019746869802474976,
+            -0.020962223410606384,
+            0.033027298748493195,
+            -0.009899764321744442,
+            0.002022079424932599,
+            -0.027904696762561798,
+            0.018569430336356163,
+            0.009689130820333958,
+            0.005093112587928772,
+            0.004452787339687347,
+            0.001162695698440075,
+            0.04340730607509613,
+            0.004701334983110428,
+            -0.029303302988409996,
+            0.012065074406564236,
+            0.002390687819570303,
+            0.020153392106294632,
+            -0.013615335337817669,
+            -0.013059263117611408,
+            0.004739248659461737,
+            -0.0018978058360517025,
+            -0.010877102613449097,
+            0.008998253382742405,
+            -0.0005845073610544205,
+            0.019243456423282623,
+            -0.009933465160429478,
+            -0.001282756682485342,
+            -0.0138933714479208,
+            0.0014723266940563917,
+            -0.016067106276750565,
+            0.025360247120261192,
+            -0.014407316222786903,
+            0.004398022778332233,
+            0.009630153886973858,
+            -0.00914990995079279,
+            -0.0034712364431470633,
+            0.007953512482345104,
+            0.009933465160429478,
+            -0.02369203232228756,
+            0.018822189420461655,
+            0.0015323570696637034,
+            0.016977041959762573,
+            0.003606041893362999,
+            -0.01224200613796711,
+            0.007047789636999369,
+            0.005362723488360643,
+            0.017153974622488022,
+            -0.00942794606089592,
+            0.005817691329866648,
+            -0.030904114246368408,
+            -0.02374258264899254,
+            0.038284704089164734,
+            0.015350953675806522,
+            -0.022630440071225166,
+            0.015839623287320137,
+            -0.02148459479212761,
+            -0.006656011566519737,
+            0.01285705529153347,
+            0.01365746185183525,
+            -0.03110632300376892,
+            0.00914990995079279,
+            -0.009040380828082561,
+            0.009166760370135307,
+            0.02241138182580471,
+            0.023119108751416206,
+            0.0038103561382740736,
+            -0.00668971287086606,
+            -0.021400341764092445,
+            -0.006091514602303505,
+            0.007907173596322536,
+            0.006314785685390234,
+            -0.007203658111393452,
+            0.02106332778930664,
+            0.01426408626139164,
+            -0.0055017415434122086,
+            -0.005396424792706966,
+            0.02413014881312847,
+            -0.03043229691684246,
+            0.019816379994153976,
+            0.0010763360187411308,
+            -0.03301044926047325,
+            0.01959731988608837,
+            -0.007325825747102499,
+            0.030196387320756912,
+            -0.0035428518895059824,
+            -0.008762344717979431,
+            0.0012722250539809465,
+            0.01162695698440075,
+            0.0028161669615656137,
+            -0.00014138758706394583,
+            0.007843983359634876,
+            0.020962223410606384,
+            -0.022512484341859818,
+            0.023220213130116463,
+            0.011921843513846397,
+            3.429373100516386e-05,
+            0.014676926657557487,
+            -0.023186512291431427,
+            -0.03011213429272175,
+            0.02743287943303585,
+            -0.011610106565058231,
+            -0.02123183384537697,
+            0.015359378419816494,
+            -0.013615335337817669,
+            -0.012814928777515888,
+            -0.02278209663927555,
+            -0.0092425886541605,
+            0.004014670383185148,
+            -0.006554907653480768,
+            -0.016252463683485985,
+            -0.0033427502494305372,
+            0.021518295630812645,
+            0.01806391030550003,
+            0.022310277447104454,
+            -0.00462129432708025,
+            0.012873905710875988,
+            0.001359637826681137,
+            0.019563619047403336,
+            0.019883781671524048,
+            -0.00939424429088831,
+            -0.0029467595741152763,
+            0.013488955795764923,
+            -0.007363739423453808,
+            0.022192321717739105,
+            0.0008404267719015479,
+            -0.02950550988316536,
+            -0.010969781316816807,
+            -0.03690294921398163,
+            0.007911385968327522,
+            -0.00792823638767004,
+            -0.006487505044788122,
+            0.05395582318305969,
+            0.006175767630338669,
+            -0.01248634047806263,
+            0.007704965304583311,
+            0.0054554021917283535,
+            -0.005341660231351852,
+            -0.00786925945430994,
+            0.017094997689127922,
+            -0.01380911748856306,
+            -0.015266700647771358,
+            -0.013118240982294083,
+            -0.010843401774764061,
+            0.009714406915009022,
+            0.020642060786485672,
+            -0.0033617070876061916,
+            -0.026169078424572945,
+            -0.013210919685661793,
+            -0.020305048674345016,
+            -0.013151941820979118,
+            0.007886109873652458,
+            -0.0042716427706182,
+            -0.0007840823964215815,
+            0.037071458995342255,
+            -0.019496217370033264,
+            -0.013404701836407185,
+            -0.010683320462703705,
+            -0.002651873044669628,
+            -0.01658947765827179,
+            -0.005939858499914408,
+            0.016201911494135857,
+            -0.016867512837052345,
+            -0.0034480667673051357,
+            -0.0056533971801400185,
+            -0.010245203040540218,
+            0.020187092944979668,
+            0.008522222749888897,
+            -0.00668971287086606,
+            0.01156798005104065,
+            0.04886692017316818,
+            -0.03798139467835426,
+            -0.01603340543806553,
+            -0.02515803836286068,
+            0.026489241048693657,
+            0.009259439073503017,
+            0.01670743152499199,
+            -0.02092852257192135,
+            -0.028494469821453094,
+            -0.023725733160972595,
+            -0.012292558327317238,
+            -0.019732125103473663,
+            -0.003188987961038947,
+            0.023102257400751114,
+            0.014803307130932808,
+            0.04283438250422478,
+            0.00532902218401432,
+            0.0035428518895059824,
+            -0.0028772505465894938,
+            0.0022474571596831083,
+            0.011197265237569809,
+            -0.02178790606558323,
+            0.0026834679301828146,
+            0.00024749411386437714,
+            0.026371287181973457,
+            0.01105403434485197,
+            -0.006024111993610859,
+            -0.028865184634923935,
+            0.0021610974799841642,
+            0.012418937869369984,
+            -0.007549096830189228,
+            -0.017069721594452858,
+            0.027837295085191727,
+            0.004549678880721331,
+            -0.0034017274156212807,
+            -0.011281518265604973,
+            0.028511321172118187,
+            0.0027550833765417337,
+            0.028123755007982254,
+            -0.020406153053045273,
+            -0.009040380828082561,
+            -0.022798946127295494,
+            0.009436370804905891,
+            -0.009107783436775208,
+            0.002430708147585392,
+            -0.01025362778455019,
+            0.015696391463279724,
+            -0.01508976798504591,
+            -0.010245203040540218,
+            0.030465997755527496,
+            -0.021198133006691933,
+            -0.0025739388074725866,
+            -0.015578437596559525,
+            -0.028022652491927147,
+            0.00967228040099144,
+            -0.0011005589039996266,
+            0.007852409034967422,
+            -0.006656011566519737,
+            0.003707145806401968,
+            -0.023034855723381042,
+            0.009166760370135307,
+            -0.00963857863098383,
+            -0.011222541332244873,
+            -0.001816712087020278,
+            0.013649036176502705,
+            0.0008383204694837332,
+            0.010337881743907928,
+            0.005034135654568672,
+            -0.03892502933740616,
+            0.01836722157895565,
+            0.0036776571068912745,
+            0.010278903879225254,
+            0.003707145806401968,
+            0.008155720308423042,
+            0.007924024015665054,
+            0.029337003827095032,
+            0.00982393603771925,
+            -0.009874488227069378,
+            -0.011854440905153751,
+            -0.0011216222774237394,
+            0.011163563467562199,
+            0.018653683364391327,
+            -0.004899329971522093,
+            -0.01616821065545082,
+            0.006399039179086685,
+            3.333929998916574e-05,
+            0.03491457179188728,
+            0.008509584702551365,
+            -0.020372450351715088,
+            -0.005337447393685579,
+            -0.04917023330926895,
+            0.020490406081080437,
+            -0.015376229770481586,
+            -0.0339372344315052,
+            0.01075072307139635,
+            -0.0043074507266283035,
+            -0.022428231313824654,
+            0.029353853315114975,
+            -0.004945669323205948,
+            -0.013219344429671764,
+            0.007001450285315514,
+            -0.009343692101538181,
+            -0.006879283115267754,
+            0.0014428379945456982,
+            -0.016623178496956825,
+            6.6851384872279596e-06,
+            0.00356602156534791,
+            0.019176054745912552,
+            -0.0034775554668158293,
+            -0.01549418456852436,
+            0.001500762184150517,
+            0.00623053265735507,
+            0.017861701548099518,
+            0.0021221302449703217,
+            0.003989394288510084,
+            0.0006424314924515784,
+            0.014828583225607872,
+            -0.02664089761674404,
+            0.005366935860365629,
+            -0.0012501084711402655,
+            -0.026792554184794426,
+            0.004553891718387604,
+            -0.006024111993610859,
+            -0.01983322948217392,
+            0.012494766153395176,
+            -0.0009241535444743931,
+            -0.0016408332157880068,
+            -0.00585981784388423,
+            -0.009478497318923473,
+            -0.009697556495666504,
+            -0.008972978219389915,
+            0.008956126868724823,
+            0.016016554087400436,
+            0.007304762024432421,
+            -0.010118822567164898,
+            -0.02431550621986389,
+            0.001289075706154108,
+            0.02529284358024597,
+            0.02357407659292221,
+            -0.035285286605358124,
+            0.014735904522240162,
+            0.005935645662248135,
+            -0.00851379707455635,
+            -0.027163267135620117,
+            0.0310389194637537,
+            0.02197326347231865,
+            0.0038061433006078005,
+            0.007397440727800131,
+            0.01162695698440075,
+            0.007359527051448822,
+            -0.03949795290827751,
+            0.011079310439527035,
+            -0.011020333506166935,
+            -0.009849212132394314,
+            -3.0377268558368087e-05,
+            0.009031955152750015,
+            -0.009882913902401924,
+            -0.011803888715803623,
+            0.007481694221496582,
+            0.02204066701233387,
+            -0.009368968196213245,
+            -0.003660806454718113,
+            0.01678325980901718,
+            0.002243244554847479,
+            0.001279597170650959,
+            0.016808535903692245,
+            0.05621381103992462,
+            0.003601829055696726,
+            -0.0019662617705762386,
+            0.02192271128296852,
+            -0.0018651577411219478,
+            -0.022428231313824654,
+            -0.006196831353008747,
+            -0.01792910508811474,
+            0.008105169050395489,
+            0.010304179973900318,
+            -0.01456739753484726,
+            -0.005088900215923786,
+            -0.0035786593798547983,
+            0.009048805572092533,
+            -0.0046507827937603,
+            0.017828000709414482,
+            -0.00638218829408288,
+            0.01909179985523224,
+            0.03152759000658989,
+            0.01407872885465622,
+            -0.013345724903047085,
+            0.014162981882691383,
+            -0.007473269011825323,
+            -0.0033954083919525146,
+            -0.010666469112038612,
+            0.001936773071065545,
+            0.0015576331643387675,
+            0.005000433884561062,
+            0.0009467966156080365,
+            0.025680409744381905,
+            -0.0012048223288729787,
+            -0.01978267729282379,
+            -0.005805053282529116,
+            -0.0057334378361701965,
+            -0.01224200613796711,
+            0.007755517493933439,
+            0.0014059771783649921,
+            0.0034480667673051357,
+            -0.02081056870520115,
+            0.0031005218625068665,
+            0.005261619575321674,
+            0.005927220452576876,
+            0.006112577859312296,
+            0.0030268002301454544,
+            -0.0038166751619428396,
+            0.02013654261827469,
+            -0.00623053265735507,
+            0.001733511919155717,
+            0.013151941820979118,
+            -0.006415889598429203,
+            -0.010877102613449097,
+            -0.013143517076969147,
+            -0.0007919811177998781,
+            0.019698424264788628,
+            -0.014719054102897644,
+            -0.00436432147398591,
+            0.013556358404457569,
+            0.00863596424460411,
+            0.006710776127874851,
+            -0.0019072843715548515,
+            -0.00604517525061965,
+            0.0026876807678490877,
+            -0.02343927137553692,
+            -0.020557807758450508,
+            -0.008905574679374695,
+            -0.008496946655213833,
+            0.024264954030513763,
+            -0.006681287661194801,
+            0.005956709384918213,
+            -0.00756173487752676,
+            0.006466441787779331,
+            0.01824926771223545,
+            0.005413275212049484,
+            -0.003639743197709322,
+            0.006757115479558706,
+            -0.00021734721667598933,
+            0.002392794005572796,
+            -0.009318416006863117,
+            0.0031658182851970196,
+            0.015291975811123848,
+            -0.010995057411491871,
+            0.0022263936698436737,
+            -0.007738666608929634,
+            -0.0036629128735512495,
+            -0.022866349667310715,
+            0.010430560447275639,
+            -0.001800914527848363,
+            0.009613302536308765,
+            0.007873471826314926,
+            -0.008058829233050346,
+            0.012789652682840824,
+            -0.008694942109286785,
+            0.015713242813944817,
+            -0.012250431813299656,
+            -0.0004007298266515136,
+            0.03017953597009182,
+            -0.011155138723552227,
+            -0.019007546827197075,
+            -0.0034733428619802,
+            -0.016176635399460793,
+            0.02308540791273117,
+            0.027230670675635338,
+            0.015435206703841686,
+            0.016075532883405685,
+            -0.007456418126821518,
+            0.0066517991945147514,
+            0.025680409744381905,
+            0.003513363189995289,
+            0.020962223410606384,
+            -0.01162695698440075,
+            0.011803888715803623,
+            -0.03017953597009182,
+            0.0003238486824557185,
+            -0.0029994179494678974,
+            -0.004992008674889803,
+            0.007843983359634876,
+            -0.02131608873605728,
+            -0.008711792528629303,
+            -0.004992008674889803,
+            -0.00719102006405592,
+            0.0041389442048966885,
+            -0.019732125103473663,
+            0.00952062476426363,
+            -0.018451474606990814,
+            -0.01658947765827179,
+            -0.012006097473204136,
+            -0.01806391030550003,
+            -0.013480530120432377,
+            -0.01721295155584812,
+            -0.030819861218333244,
+            -0.00356602156534791,
+            -0.02062521129846573,
+            -0.025073785334825516,
+            0.018721086904406548,
+            -0.023860538378357887,
+            0.01199767179787159,
+            -0.0008514850633218884,
+            -0.006458016578108072,
+            0.025680409744381905,
+            0.029471809044480324,
+            -0.009866062551736832,
+            -0.023708881810307503,
+            -0.009655429981648922,
+            -0.029117945581674576,
+            0.009714406915009022,
+            0.011028758250176907,
+            0.0092425886541605,
+            -0.01090237870812416,
+            0.0015397293027490377,
+            -0.014516845345497131,
+            0.01732248067855835,
+            -0.032117363065481186,
+            -0.029522361233830452,
+            -0.008029340766370296,
+            -0.006879283115267754,
+            -0.001448103808797896,
+            -0.015350953675806522,
+            0.005969346966594458,
+            -0.018080761656165123,
+            -0.02748342975974083,
+            -0.0030331192538142204,
+            -0.006163129583001137,
+            -0.01304241269826889,
+            0.005219492595642805,
+            -0.00557335652410984,
+            0.022394530475139618,
+            0.010666469112038612,
+            0.015780646353960037,
+            -0.004848778247833252,
+            -0.009192036464810371,
+            -0.014306212775409222,
+            -0.01573851890861988,
+            0.004777162801474333,
+            0.009748108685016632,
+            0.012621146626770496,
+            0.0004154741473030299,
+            0.010076696053147316,
+            0.000991029548458755,
+            0.013210919685661793,
+            0.010270479135215282,
+            0.010910804383456707,
+            0.021164432168006897,
+            -0.0023801561910659075,
+            0.014188257977366447,
+            0.00402730843052268,
+            0.005459614563733339,
+            -0.0029383343644440174,
+            -0.008357929065823555,
+            0.022360829636454582,
+            -0.007658625952899456,
+            -0.013960774056613445,
+            0.029337003827095032,
+            0.004697122145444155,
+            -0.02320336177945137,
+            0.012629571370780468,
+            0.0003151600540149957,
+            -0.018721086904406548,
+            -0.008652815595269203,
+            -0.0009815511293709278,
+            -0.004473851062357426,
+            -0.00765441358089447,
+            0.0014112429926171899,
+            -0.008686516433954239,
+            0.01621876284480095,
+            0.016875937581062317,
+            -0.008930850774049759,
+            -0.008429544046521187,
+            0.017844852060079575,
+            0.016614753752946854,
+            -0.001709289033897221,
+            -0.022209173068404198,
+            -0.008694942109286785,
+            0.02504008449614048,
+            -0.03629632666707039,
+            -0.026893656700849533,
+            0.007700752932578325,
+            0.006420102436095476,
+            -0.009756533429026604,
+            6.574062354047783e-06,
+            0.003983075264841318,
+            -0.007578585296869278,
+            0.0032479651272296906,
+            0.0016208230517804623,
+            0.012680123560130596,
+            0.01484543364495039,
+            -0.011096160858869553,
+            0.010556939989328384,
+            -0.01573851890861988,
+            0.0018672640435397625,
+            -0.020035438239574432,
+            -0.013126665726304054,
+            -0.01380069274455309,
+            0.010371582582592964,
+            0.020591510459780693,
+            0.011829164810478687,
+            0.03889133036136627,
+            0.01466007623821497,
+            0.004014670383185148,
+            -0.006917196791619062,
+            -0.017288779839873314,
+            0.004802438896149397,
+            -0.002948865992948413,
+            -0.0037766548339277506,
+            -0.0021158112213015556,
+            0.014912836253643036,
+            -0.0021779481321573257,
+            0.028123755007982254,
+            0.006158917210996151,
+            0.01236838661134243,
+            -2.6213969249511138e-05,
+            -0.007022513542324305,
+            -0.003983075264841318,
+            0.013101390562951565,
+            5.410642552305944e-05,
+            -0.0023675181437283754,
+            0.0067192018032073975,
+            -0.0018483070889487863,
+            -0.005055198911577463,
+            0.0027656150050461292,
+            -0.011365772224962711,
+            -0.0055101667530834675,
+            -0.017356181517243385,
+            -0.008787620812654495,
+            -0.009175186045467854,
+            0.02485472708940506,
+            0.028865184634923935,
+            -0.015013940632343292,
+            -0.010430560447275639,
+            -0.013455254025757313,
+            0.03589191287755966,
+            -0.004840353038161993,
+            0.0034291099291294813,
+            0.014811732806265354,
+            -0.008753919042646885,
+            0.0024222827050834894,
+            0.012418937869369984,
+            0.0042210910469293594,
+            0.003555489704012871,
+            0.029926776885986328,
+            -0.0031721373088657856,
+            -0.01013567391782999,
+            -0.0035028315614908934,
+            0.003907247446477413,
+            -0.026101676747202873,
+            0.0003712411562446505,
+            0.004840353038161993,
+            -0.010514813475310802,
+            0.013092964887619019,
+            -0.019513066858053207,
+            0.006795029621571302,
+            -0.010632768273353577,
+            -0.0008367407135665417,
+            -0.01976582780480385,
+            -0.0007946140831336379,
+            0.007633350323885679,
+            -0.012587444856762886,
+            -0.003197413170710206,
+            -0.017356181517243385,
+            -0.0020062820985913277,
+            0.014643225818872452,
+            0.007001450285315514,
+            0.006040962412953377,
+            -0.0007472215802408755,
+            0.0032016257755458355,
+            0.022478783503174782,
+            0.0043474710546433926,
+            0.004718185402452946,
+            -0.004541253671050072,
+            0.0042526861652731895,
+            -0.024264954030513763,
+            0.006453803740441799,
+            -0.005762926768511534,
+            -0.0026497666258364916,
+            0.024955831468105316,
+            -0.005842967424541712,
+            0.0009678599308244884,
+            -0.006238957867026329,
+            0.005059411283582449,
+            -0.024568265303969383,
+            0.01508976798504591,
+            -0.01762579381465912,
+            0.010700170882046223,
+            -0.008564349263906479,
+            -0.002706637606024742,
+            -0.00251074880361557,
+            0.009444796480238438,
+            0.012292558327317238,
+            -0.013084539212286472,
+            0.007397440727800131,
+            -0.015822771936655045,
+            0.021686803549528122,
+            -0.003867227118462324,
+            -0.01878848858177662,
+            0.0018314564367756248,
+            -0.008046191185712814,
+            0.008762344717979431,
+            -0.014702202752232552,
+            0.0010515867033973336,
+            0.010396858677268028,
+            -0.007216296158730984,
+            0.005809266120195389,
+            -0.023119108751416206,
+            0.006828730925917625,
+            -0.017490988597273827,
+            0.008796045556664467,
+            -0.012966584414243698,
+            -0.012966584414243698,
+            -0.02118128351867199,
+            -0.012006097473204136,
+            -0.00024551941896788776,
+            0.0051436647772789,
+            0.015713242813944817,
+            -0.006934047676622868,
+            0.005838754586875439,
+            0.021872159093618393,
+            -0.010026143863797188,
+            0.01719610020518303,
+            0.053113289177417755,
+            -0.025444500148296356,
+            -0.016934916377067566,
+            0.00136279733851552,
+            0.010784423910081387,
+            -0.03434165194630623,
+            -0.008581199683248997,
+            0.019395112991333008,
+            -0.016361992806196213,
+            0.029606614261865616,
+            -0.025663558393716812,
+            0.01156798005104065,
+            -0.010708595626056194,
+            0.00796193815767765,
+            -0.017052870243787766,
+            -0.01929400861263275,
+            -0.00991661474108696,
+            -0.003568127751350403,
+            -0.0018462006701156497,
+            -0.02394479140639305,
+            0.009495348669588566,
+            -0.014390465803444386,
+            -0.011045609600841999,
+            0.006150492001324892,
+            -0.0057882023975253105,
+            0.007460630964487791,
+            0.0016608433797955513,
+            0.013758566230535507,
+            -0.020776866003870964,
+            0.0008767610415816307,
+            -0.011045609600841999,
+            0.004747674334794283,
+            -0.0067739663645625114,
+            0.011298369616270065,
+            0.01337942574173212,
+            -0.013092964887619019,
+            -0.002439133357256651,
+            -0.002023132750764489,
+            0.015511034987866879,
+            -0.006504355929791927,
+            0.03683554753661156,
+            -0.008075679652392864,
+            -0.004339045379310846,
+            -0.008871873840689659,
+            -0.0056870984844863415,
+            0.011829164810478687,
+            0.01328674703836441,
+            0.008568561635911465,
+            0.005834541749209166,
+            0.00914990995079279,
+            -0.014112429693341255,
+            0.01536780409514904,
+            0.007949300110340118,
+            0.017659494653344154,
+            0.005307958461344242,
+            -0.013851244933903217,
+            -0.024837875738739967,
+            -0.0066855004988610744,
+            -0.007262635510414839,
+            -0.0029446533881127834,
+            -0.01665687933564186,
+            0.0035512770991772413,
+            0.0067781792022287846,
+            0.007827132940292358,
+            -0.003900928422808647,
+            0.03257232904434204,
+            0.005729225464165211,
+            0.0012753845658153296,
+            0.027635086327791214,
+            0.007793431635946035,
+            0.0015112938126549125,
+            0.017524689435958862,
+            -0.031072622165083885,
+            -0.01404502708464861,
+            -0.021889010444283485,
+            0.011761762201786041,
+            0.0004615501675289124,
+            -0.0318477526307106,
+            -0.0024833662901073694,
+            0.0011121437419205904,
+            -0.01025362778455019,
+            0.012629571370780468,
+            0.0206926129758358,
+            -0.016378844156861305,
+            -0.011947119608521461,
+            0.008037766441702843,
+            0.011070885695517063,
+            -0.005190004128962755,
+            0.027635086327791214,
+            -0.002001016167923808,
+            0.009630153886973858,
+            0.007405866403132677,
+            0.019428813830018044,
+            0.0006540163303725421,
+            0.023523524403572083,
+            -0.006643373519182205,
+            -0.00483614020049572,
+            -0.020490406081080437,
+            -0.008412693627178669,
+            -0.01175333745777607,
+            0.011610106565058231,
+            -0.005206854548305273,
+            0.003789292648434639,
+            -0.013952348381280899,
+            0.007915598340332508,
+            0.014963388442993164,
+            0.011028758250176907,
+            0.0044443621300160885,
+            0.012124051339924335,
+            0.010455836541950703,
+            0.013590059243142605,
+            0.0012353642378002405,
+            -0.020338749513030052,
+            0.008914000354707241,
+            0.017608942463994026,
+            -0.006934047676622868,
+            0.02141719125211239,
+            0.023540375754237175,
+            0.001537623000331223,
+            -0.003188987961038947,
+            0.008518009446561337,
+            0.01652207411825657,
+            0.01976582780480385,
+            -0.01298343576490879,
+            -0.0032332208938896656,
+            0.0036629128735512495,
+            -0.0028983140364289284,
+            -0.0016408332157880068,
+            -0.019327709451317787,
+            -0.000656649237498641,
+            0.008614901453256607,
+            0.0010921335779130459,
+            0.030230088159441948,
+            -0.01823241636157036,
+            -0.01428936142474413,
+            0.01915920339524746,
+            0.006622310262173414,
+            -0.01880533993244171,
+            0.012503191828727722,
+            0.00015968635852914304,
+            0.013960774056613445,
+            -0.011104586534202099,
+            0.005215280223637819,
+            -0.01004299521446228,
+            -0.004503339529037476,
+            0.017524689435958862,
+            -0.009267864748835564,
+            0.011778613552451134,
+            -0.008383204229176044,
+            -0.001989431446418166,
+            -0.02074316516518593,
+            0.02032189816236496,
+            0.02613537758588791,
+            0.01086867693811655,
+            0.008412693627178669,
+            0.004751886706799269,
+            0.007271060720086098,
+            0.00436432147398591,
+            0.01927715726196766,
+            0.020658912137150764,
+            -0.011424749158322811,
+            0.007494332268834114,
+            0.018148163333535194,
+            -0.007249997463077307,
+            0.012840204872190952,
+            0.004827714990824461,
+            0.004970945417881012,
+            -0.007094128988683224,
+            0.006015686318278313,
+            -0.017162399366497993,
+            -0.013033987022936344,
+            0.0012627465184777975,
+            0.032606031745672226,
+            0.008454820141196251,
+            -0.023304466158151627,
+            0.001874636160209775,
+            0.024349207058548927,
+            -0.020894821733236313,
+            -0.0034754490479826927,
+            0.008315801620483398,
+            -0.0019157096976414323,
+            0.012899181805551052,
+            -0.02461881749331951,
+            -0.00786925945430994,
+            0.004810864105820656,
+            -0.006348486989736557,
+            0.016193486750125885,
+            -0.012098775245249271,
+            -0.0390598364174366,
+            -0.01199767179787159,
+            -0.020153392106294632,
+            -0.017861701548099518,
+            -0.019664723426103592,
+            0.0008930850890465081,
+            0.01585647277534008,
+            -0.019024398177862167,
+            -0.018333520740270615,
+            0.016741132363677025,
+            -0.001975740073248744,
+            0.03764437884092331,
+            -0.017373032867908478,
+            0.007536458782851696,
+            -0.01727192848920822,
+            0.01646309718489647,
+            -0.016800111159682274,
+            -0.018502026796340942,
+            0.01475275494158268,
+            -0.0031679244711995125,
+            -0.003721890039741993,
+            0.0023422420490533113,
+            0.00235698651522398,
+            -0.009714406915009022,
+            -0.003024693811312318,
+            0.011652233079075813,
+            0.0006982493214309216,
+            -0.0161008071154356,
+            -0.002645554021000862,
+            0.003528107423335314,
+            0.012882331386208534,
+            0.01578907109797001,
+            -0.0069635361433029175,
+            -0.003808249719440937,
+            0.010211501270532608,
+            -0.01762579381465912,
+            -0.010363157838582993,
+            0.015300401486456394,
+            -0.01732248067855835,
+            0.03191515430808067,
+            0.006592821795493364,
+            -0.01776059903204441,
+            -0.00716153159737587,
+            0.003629211336374283,
+            0.023287614807486534,
+            0.01332044880837202,
+            -0.020726315677165985,
+            -0.008004064671695232,
+            0.007810282055288553,
+            0.012739100493490696,
+            0.005438551306724548,
+            -0.007186807692050934,
+            0.027820443734526634,
+            -0.0007530139992013574,
+            0.00015547369548585266,
+            0.001172174233943224,
+            -0.01056536566466093,
+            -0.0021737355273216963,
+            0.008804471231997013,
+            0.015839623287320137,
+            0.004292706493288279,
+            0.00985763780772686,
+            0.005333234556019306,
+            -0.01635356806218624,
+            0.015713242813944817,
+            0.0068076676689088345,
+            -0.0035449580755084753,
+            -0.016370419412851334,
+            0.011273093521595001,
+            0.007119405083358288,
+            0.016513649374246597,
+            -0.012772802263498306,
+            -0.004882479552179575,
+            -1.0696221579564735e-05,
+            0.00737637747079134,
+            0.0022221810650080442,
+            0.008067254908382893,
+            0.0010705436579883099,
+            0.02938755601644516,
+            0.03282508999109268,
+            0.03312840312719345,
+            -0.025326546281576157,
+            -0.010321030393242836,
+            0.008231548592448235,
+            -0.003157392842695117,
+            -0.004654995631426573,
+            -0.0003354334912728518,
+            -0.01720452681183815,
+            0.006971961818635464,
+            -0.02620278112590313,
+            0.01199767179787159,
+            -0.001448103808797896,
+            0.005147877614945173,
+            -0.010860252194106579,
+            0.0066517991945147514,
+            -0.0013680631527677178,
+            0.0001929927384480834,
+            -0.011424749158322811,
+            -0.019209755584597588,
+            -0.0014586354373022914,
+            0.014002900570631027,
+            -0.0012374705402180552,
+            -0.008122019469738007,
+            0.014061877503991127,
+            -0.006057813297957182,
+            -0.023237064480781555,
+            0.006087301764637232,
+            -0.028882035985589027,
+            0.0021010669879615307,
+            -0.013092964887619019,
+            0.003909353632479906,
+            -0.022680992260575294,
+            0.013421552255749702,
+            -0.0013554252218455076,
+            -0.01267169788479805,
+            0.004941456951200962,
+            -0.017288779839873314,
+            0.0025676197838038206,
+            -0.003197413170710206,
+            0.006272659171372652,
+            -0.010708595626056194,
+            0.028376515954732895,
+            0.014575823210179806,
+            -0.015140320174396038,
+            -0.017052870243787766,
+            0.008446394465863705,
+            -0.02809005416929722,
+            0.021703653037548065,
+            0.010885528288781643,
+            -0.0034733428619802,
+            -0.02406274527311325,
+            0.009748108685016632,
+            0.0036966141778975725,
+            0.02050725556910038,
+            0.010860252194106579,
+            0.009293140843510628,
+            0.018822189420461655,
+            -0.015527885407209396,
+            0.0021800545509904623,
+            -1.19962232929538e-05,
+            -0.00251074880361557,
+            0.014921261928975582,
+            -0.008071467280387878,
+            0.014584247954189777,
+            0.007359527051448822,
+            -0.0044232988730072975,
+            -0.006268446333706379,
+            -0.0030520763248205185,
+            -0.015165596269071102,
+            0.0004289020143914968,
+            -0.0020241858437657356,
+            -0.0008130444912239909,
+            0.006647586356848478,
+            0.010110397823154926,
+            -0.017305631190538406,
+            0.00914990995079279,
+            -0.0012248324928805232,
+            -0.025848915800452232,
+            0.0014038708759471774,
+            -0.016741132363677025,
+            0.002437027171254158,
+            -0.021383490413427353,
+            0.01252004224807024,
+            0.025208590552210808,
+            0.011037183925509453,
+            -0.007073065731674433,
+            -0.006234745029360056,
+            -0.011879716999828815,
+            -0.007485906593501568,
+            -0.010514813475310802,
+            -0.006125215906649828,
+            -0.019209755584597588,
+            -0.003728209063410759,
+            0.011121436953544617,
+            0.011728061363101006,
+            0.00034227909054607153,
+            -0.018350372090935707,
+            -0.005135239567607641,
+            -0.013109815306961536,
+            -0.005400637164711952,
+            -0.008547498844563961,
+            -0.013084539212286472,
+            0.0011974502122029662,
+            0.012545318342745304,
+            -0.011669083498418331,
+            0.029421256855130196,
+            0.005779777187854052,
+            0.006243170239031315,
+            -0.0022158620413392782,
+            -0.007881897501647472,
+            -0.004162113647907972,
+            -0.012216730043292046,
+            -0.0022074368316680193,
+            -0.005341660231351852,
+            -0.0020757908932864666,
+            -0.00905723124742508,
+            0.005813478492200375,
+            -0.0015913344686850905,
+            0.01555316150188446,
+            0.006062025669962168,
+            -0.008126231841742992,
+            -0.002955185016617179,
+            0.031679246574640274,
+            -0.007140468340367079,
+            -0.016614753752946854,
+            -0.00037861333112232387,
+            -0.00638218829408288,
+            0.0037303154822438955,
+            0.007414291612803936,
+            -0.01769319549202919,
+            0.0043895975686609745,
+            0.010851826518774033,
+            0.004794013686478138,
+            -0.0033827705774456263,
+            0.02081056870520115,
+            0.008779195137321949,
+            -0.002984673483297229,
+            0.010329456068575382,
+            0.0032269018702208996,
+            -0.010068271309137344,
+            -0.0044232988730072975,
+            -0.0026476604398339987,
+            0.028814632445573807,
+            -0.03484717011451721,
+            -0.0020547276362776756,
+            0.019732125103473663,
+            0.0044780634343624115,
+            -0.021147580817341805,
+            -0.002312753349542618,
+            -0.026977911591529846,
+            -0.0020673656836152077,
+            -0.009891338646411896,
+            0.013404701836407185,
+            -0.036026716232299805,
+            -0.002512854989618063,
+            -0.00905723124742508,
+            0.0044991266913712025,
+            -0.006373763084411621,
+            0.026893656700849533,
+            -0.0008936116937547922,
+            -0.005505953915417194,
+            0.011011907830834389,
+            0.009689130820333958,
+            0.011947119608521461,
+            0.026590345427393913,
+            -0.006934047676622868,
+            -0.0067529031075537205,
+            0.003083671210333705,
+            -0.028073204681277275,
+            0.02276524528861046,
+            0.00362710515037179,
+            0.0032669221982359886,
+            -0.0004115247866138816,
+            -0.006740265060216188,
+            0.02625333145260811,
+            0.02866297774016857,
+            -0.005223705433309078,
+            -0.005282682832330465,
+            -0.016387268900871277,
+            0.029101094231009483,
+            -0.011441599577665329,
+            0.0011205690680071712,
+            -0.002015760401263833,
+            -0.0121493274345994,
+            -0.01732248067855835,
+            -0.013446828350424767,
+            0.019984886050224304,
+            -0.02008599042892456,
+            -0.0017724790377542377,
+            -0.012882331386208534,
+            -0.001390179619193077,
+            -0.02497268095612526,
+            -0.008467458188533783,
+            0.008572774939239025,
+            -0.014070303179323673,
+            -0.0032416461035609245,
+            0.0003148967516608536,
+            -0.006436952855437994,
+            0.0011311006965115666,
+            -0.0067866044119000435,
+            -0.00031437017605639994,
+            -0.003799824509769678,
+            0.011129862628877163,
+            0.0002694789436645806,
+            -0.008855023421347141,
+            0.019496217370033264,
+            -0.010919229127466679,
+            0.006795029621571302,
+            0.0043221949599683285,
+            -0.004010457545518875,
+            -0.0003080511814914644,
+            0.009697556495666504,
+            0.014188257977366447,
+            -0.013708014041185379,
+            0.021939562633633614,
+            0.009377393871545792,
+            -0.01484543364495039,
+            0.010405284352600574,
+            -0.005354297813028097,
+            -0.019866930320858955,
+            -0.0026687236968427896,
+            0.015022365376353264,
+            0.019850080832839012,
+            -0.022445082664489746,
+            0.0011195158585906029,
+            0.01873793639242649,
+            -0.00021194973669480532,
+            0.0021379278041422367,
+            0.007338463328778744,
+            0.006142066325992346,
+            0.007485906593501568,
+            0.010026143863797188,
+            0.009739683009684086,
+            -0.0037576977629214525,
+            -0.0206926129758358,
+            0.015199298039078712,
+            -0.012140902690589428,
+            0.01561213843524456,
+            0.023776285350322723,
+            -0.010607492178678513,
+            -0.021467743441462517,
+            0.015106619335711002,
+            -0.007186807692050934,
+            -0.02827541157603264,
+            -0.0015039215795695782,
+            -0.0017292991979047656,
+            0.01978267729282379,
+            0.0030815647915005684,
+            -2.3564598450320773e-05,
+            -0.018282968550920486,
+            -1.969750337593723e-05,
+            0.01080970000475645,
+            -0.0017524688737466931,
+            -0.0411493182182312,
+            -0.006217894610017538,
+            0.004263217560946941,
+            -0.00486562866717577,
+            0.014255660586059093,
+            0.0017651069210842252,
+            0.005527017172425985,
+            0.026000572368502617,
+            -0.019698424264788628,
+            -0.010118822567164898,
+            0.006521206349134445,
+            0.016370419412851334,
+            0.010152524337172508,
+            0.011104586534202099,
+            0.006508568301796913,
+            0.015224573202431202,
+            -0.012039798311889172,
+            -0.007102554198354483,
+            0.018266119062900543,
+            -0.013387851417064667,
+            0.014811732806265354,
+            -0.01811446249485016,
+            0.004461213015019894,
+            -0.006310573313385248,
+            0.004739248659461737,
+            0.010969781316816807,
+            -0.00768390204757452,
+            -0.0010357891442254186,
+            -0.040744900703430176,
+            0.004570742137730122,
+            -0.0027213820721954107,
+            0.014710628427565098,
+            -0.01105403434485197,
+            0.004608656279742718,
+            -0.0001983902184292674,
+            0.005834541749209166,
+            0.009689130820333958,
+            -0.0008893990307115018,
+            0.01615135930478573,
+            -0.02601742371916771,
+            -0.025141188874840736,
+            7.082543743308634e-05,
+            -0.006917196791619062,
+            0.008762344717979431,
+            0.020423002541065216,
+            0.012444213964045048,
+            0.023489823564887047,
+            -0.01205664873123169,
+            0.008509584702551365,
+            0.011188839562237263,
+            -0.016875937581062317,
+            -0.003881971351802349,
+            0.0011953439097851515,
+            -0.008400055579841137,
+            -0.02332131750881672,
+            -0.0018177651800215244,
+            -0.004671846050769091,
+            0.008623326197266579,
+            0.0035639151465147734,
+            0.006377975922077894,
+            -0.025427650660276413,
+            0.004160007461905479,
+            0.004265323746949434,
+            -0.0298762246966362,
+            0.00018074968829751015,
+            -0.017423585057258606,
+            0.008627539500594139,
+            -0.0016650561010465026,
+            0.021282386034727097,
+            -0.013126665726304054,
+            -0.022849498316645622,
+            -0.005716587416827679,
+            0.005425913259387016,
+            -0.007389015518128872,
+            -0.01025362778455019,
+            0.015814347192645073,
+            -0.017069721594452858,
+            -0.008008277043700218,
+            -0.021585699170827866,
+            -0.006601247005164623,
+            -0.036026716232299805,
+            -0.006344274617731571,
+            0.01056536566466093,
+            0.005548080429434776,
+            -0.010068271309137344,
+            -0.029657166451215744,
+            0.005889306776225567,
+            -0.0067781792022287846,
+            -0.004549678880721331,
+            -0.01096135564148426,
+            0.0014923367416486144,
+            0.022006966173648834,
+            -0.006411677226424217,
+            0.02148459479212761,
+            -0.0174404364079237,
+            -0.004705547355115414,
+            -0.017061294987797737,
+            0.0044570001773536205,
+            -0.0022179684601724148,
+            -0.004583380185067654,
+            -0.00666022440418601,
+            -0.011736486107110977,
+            0.0024875791277736425,
+            0.002984673483297229,
+            -0.007435354869812727,
+            -0.007532245945185423,
+            0.012267282232642174,
+            0.009141484275460243,
+            0.007831345312297344,
+            0.007843983359634876,
+            0.004840353038161993,
+            -0.012621146626770496,
+            -0.00032700816518627107,
+            -0.006024111993610859,
+            0.005194216500967741,
+            0.009933465160429478,
+            0.0057207997888326645,
+            0.005004646722227335,
+            -0.008897149935364723,
+            0.012081924825906754,
+            0.01899069733917713,
+            0.033263206481933594,
+            -0.01304241269826889,
+            -0.005168940871953964,
+            -0.014011326245963573,
+            0.02166995219886303,
+            0.019024398177862167,
+            0.003317474154755473,
+            0.005088900215923786,
+            0.006925622466951609,
+            -0.0019315071403980255,
+            0.0013238302199169993,
+            0.012098775245249271,
+            -0.00872021820396185,
+            -0.009293140843510628,
+            -0.011323644779622555,
+            0.0013733289670199156,
+            -0.01781114935874939,
+            0.008058829233050346,
+            0.006441165693104267,
+            0.002065259264782071,
+            0.01025362778455019,
+            0.03936314582824707,
+            0.0026139591354876757,
+            -0.0016471522394567728,
+            -0.00238647498190403,
+            -0.01915920339524746,
+            0.02124868519604206,
+            -0.007915598340332508,
+            -0.039295744150877,
+            -0.0024960043374449015,
+            0.002679255325347185,
+            0.01347210444509983,
+            -0.0079324496909976,
+            -0.000739322800654918,
+            -0.011787038296461105,
+            -0.02074316516518593,
+            0.0014459975063800812,
+            0.015039216727018356,
+            -0.0005660769529640675,
+            0.011955545283854008,
+            0.003568127751350403,
+            0.0026350223924964666,
+            0.0034164718817919493,
+            0.009537475183606148,
+            -0.02148459479212761,
+            0.002752976957708597,
+            0.007671264000236988,
+            0.028747230768203735,
+            0.0007682848954573274,
+            -0.007233147043734789,
+            0.017828000709414482,
+            -0.0034164718817919493,
+            -0.007157318759709597,
+            0.008041978813707829,
+            -0.011256242170929909,
+            0.010893953032791615,
+            -0.013143517076969147,
+            0.011795463971793652,
+            0.02290005050599575,
+            -0.0206926129758358,
+            0.005383786745369434,
+            -0.007325825747102499,
+            -0.003957799170166254,
+            0.015696391463279724,
+            0.020102839916944504,
+            0.006255808286368847,
+            0.0010484271915629506,
+            -0.02106332778930664,
+            0.017490988597273827,
+            -0.02081056870520115,
+            -0.0102873295545578,
+            0.005581781733781099,
+            -0.014794881455600262,
+            0.010978206992149353,
+            -0.0038988220039755106,
+            -0.026539793238043785,
+            0.003875652328133583,
+            0.009360543452203274,
+            0.015115044079720974,
+            0.006340061780065298,
+            0.005560718476772308,
+            0.0012048223288729787,
+            -0.012022947892546654,
+            -0.023708881810307503,
+            -0.0058219037018716335,
+            -0.019007546827197075,
+            -0.021889010444283485,
+            -0.006424315273761749,
+            0.007110979408025742,
+            -0.008058829233050346,
+            0.0018704235553741455,
+            0.023001154884696007,
+            -0.018535729497671127,
+            -0.0065675457008183,
+            0.022748393937945366,
+            0.003507044166326523,
+            0.013017136603593826,
+            -0.007460630964487791,
+            0.017423585057258606,
+            0.00954589992761612,
+            -0.012494766153395176,
+            -0.01959731988608837,
+            0.027045313268899918,
+            -0.00495830737054348,
+            0.007321612909436226,
+            0.008846597746014595,
+            0.0012195666786283255,
+            -0.004726610612124205,
+            0.00814308226108551,
+            -0.030465997755527496,
+            0.013151941820979118,
+            -0.00734688900411129,
+            -0.013244620524346828,
+            -0.012208305299282074,
+            -0.00628950959071517,
+            0.001043687923811376,
+            -0.0015302507672458887,
+            0.01603340543806553,
+            -0.006116790696978569,
+            0.01841777376830578,
+            0.003210051218047738,
+            -0.0013785947812721133,
+            0.004608656279742718,
+            -0.0437106192111969,
+            -0.002169522689655423,
+            -0.010851826518774033,
+            -0.022798946127295494,
+            0.023422420024871826,
+            0.010952930897474289,
+            0.00039098801789805293,
+            -0.023557227104902267,
+            0.003938842564821243,
+            -0.004195814952254295,
+            -0.00025855234707705677,
+            0.020406153053045273,
+            0.0026961059775203466,
+            0.0010089334100484848,
+            -0.022731544449925423,
+            -0.0029783546924591064,
+            0.010068271309137344,
+            -0.0028182733803987503,
+            0.0010178852826356888,
+            0.01567954197525978,
+            0.010632768273353577,
+            -0.012393662706017494,
+            -0.018704235553741455,
+            0.005691311322152615,
+            0.016825387254357338,
+            0.008821321651339531,
+            0.0032416461035609245,
+            -0.008071467280387878,
+            0.0044443621300160885,
+            -0.003835632000118494,
+            0.038419511169195175,
+            0.020894821733236313,
+            0.006272659171372652,
+            -0.02504008449614048,
+            0.008475882932543755,
+            -0.019176054745912552,
+            0.004574954975396395,
+            0.010556939989328384,
+            0.00872021820396185,
+            -0.0032269018702208996,
+            -0.007587010972201824,
+            0.017052870243787766,
+            -0.005291108042001724,
+            0.018451474606990814,
+            -0.005286895204335451,
+            -0.00538799911737442,
+            0.018839040771126747,
+            -0.023540375754237175,
+            0.007650200743228197,
+            -0.018586281687021255,
+            -0.011787038296461105,
+            -0.013185643590986729,
+            -0.004248473327606916,
+            0.014087153598666191,
+            0.0013543720124289393,
+            -0.0326734334230423,
+            0.007321612909436226,
+            -0.026775702834129333,
+            0.00860647577792406,
+            -0.011323644779622555,
+            -4.169880776316859e-05,
+            -0.002578151412308216,
+            -0.008496946655213833,
+            -0.02431550621986389,
+            -0.021754205226898193,
+            0.005404850002378225,
+            -0.004570742137730122,
+            -0.0030520763248205185,
+            0.004633932374417782,
+            -0.0018082867609336972,
+            -0.01579749584197998,
+            0.008063041605055332,
+            0.028814632445573807,
+            0.02111387997865677,
+            0.008539073169231415,
+            0.0056196958757936954,
+            0.014727478846907616,
+            0.0011542703723534942,
+            -0.037745483219623566,
+            0.008404267951846123,
+            0.004684484098106623,
+            0.006529631558805704,
+            0.006175767630338669,
+            -0.005539655219763517,
+            0.0017608941998332739,
+            -0.010203076526522636,
+            -0.007553309667855501,
+            0.0029130582697689533,
+            -0.010607492178678513,
+            -0.007169956807047129,
+            -0.031241128221154213,
+            0.003083671210333705,
+            -0.010826550424098969,
+            0.007325825747102499,
+            -0.006971961818635464,
+            0.01641254499554634,
+            0.0016945448005571961,
+            0.00020286618382669985,
+            -0.013817543163895607,
+            -0.013497380539774895,
+            -0.0029320153407752514,
+            5.486338704940863e-05,
+            -0.01885589212179184,
+            -0.008416905999183655,
+            0.005105750635266304,
+            -0.002458090428262949,
+            -0.026657748967409134,
+            0.023489823564887047,
+            -0.000226299132918939,
+            0.011289943940937519,
+            0.023355018347501755,
+            0.020726315677165985,
+            4.5582357415696606e-05,
+            0.004815076943486929,
+            0.0025971082504838705,
+            -0.008404267951846123,
+            -0.01861998252570629,
+            0.007587010972201824,
+            -0.029859373345971107,
+            0.02136664092540741,
+            -0.006146279163658619,
+            -0.018636832013726234,
+            -0.006436952855437994,
+            -0.009082507342100143,
+            0.011011907830834389,
+            0.00015270912263076752,
+            0.01664845459163189,
+            -0.011348920874297619,
+            -0.003734528087079525,
+            -0.005607057828456163,
+            0.0018830614862963557,
+            0.008404267951846123,
+            0.029589762911200523,
+            -0.0032079447992146015,
+            -0.010413709096610546,
+            0.0131940683349967,
+            0.007679689209908247,
+            0.02896628901362419,
+            0.0013954454334452748,
+            -0.0008451660396531224,
+            -0.013868095353245735,
+            -0.0010378954466432333,
+            -0.01075072307139635,
+            -0.007220508996397257,
+            0.02136664092540741,
+            -0.010481111705303192,
+            -0.01995118521153927,
+            -0.00817678403109312,
+            -0.02050725556910038,
+            -0.007717603351920843,
+            0.013025562278926373,
+            -0.004954094998538494,
+            -0.004326407797634602,
+            -0.007182594854384661,
+            -0.01508976798504591,
+            -0.017962805926799774,
+            0.02606797404587269,
+            0.07434512674808502,
+            -0.021265536546707153,
+            -0.009596452116966248,
+            -0.011281518265604973,
+            0.0033954083919525146,
+            -0.004903542809188366,
+            -0.006748690269887447,
+            0.05661822482943535,
+            -0.019984886050224304,
+            0.005868243053555489,
+            -0.020406153053045273,
+            -0.006108365021646023,
+            -8.392419840674847e-05,
+            0.0068877083249390125,
+            0.002439133357256651,
+            -0.008105169050395489,
+            0.009899764321744442,
+            -0.018350372090935707,
+            -0.0057334378361701965,
+            -0.016201911494135857,
+            -0.013977624475955963,
+            -0.0169686172157526,
+            0.011643807403743267,
+            -0.019917482510209084,
+            -0.008956126868724823,
+            -0.0014091366901993752,
+            -0.009588027372956276,
+            0.015317251905798912,
+            -0.009267864748835564,
+            0.009958741255104542,
+            0.0009815511293709278,
+            0.003102628281340003,
+            -0.00939424429088831,
+            -0.010026143863797188,
+            -0.007144680712372065,
+            -0.0023822623770684004,
+            0.005703949369490147,
+            0.0007082544034346938,
+            0.0014491570182144642,
+            0.007292123977094889,
+            -0.0067739663645625114,
+            0.0054764654487371445,
+            0.020355600863695145,
+            0.0009315256611444056,
+            0.004899329971522093,
+            -0.017255079001188278,
+            0.019799528643488884,
+            -0.00517315324395895,
+            0.0289494376629591,
+            0.0057123745791614056,
+            -0.009689130820333958,
+            -0.016640029847621918,
+            0.014247234910726547,
+            -0.013573208823800087,
+            -0.0034227909054607153,
+            0.01983322948217392,
+            0.029168495908379555,
+            0.00789874792098999,
+            -0.012536892667412758,
+            0.0010415816213935614,
+            -4.456210444914177e-05,
+            0.01959731988608837,
+            -0.011348920874297619,
+            0.03230271860957146,
+            -0.011896567419171333,
+            0.004448574967682362,
+            -0.026118526235222816,
+            -0.009402669966220856,
+            -0.006323210895061493,
+            -0.003669231664389372,
+            0.022546187043190002,
+            -0.0044907014816999435,
+            0.007999852299690247,
+            -0.01267169788479805,
+            -0.021046478301286697,
+            0.004642357584089041,
+            -0.0043685343116521835,
+            -0.006723414175212383,
+            0.01579749584197998,
+            -0.029488658532500267,
+            0.004250579513609409,
+            0.009023529477417469,
+            -0.013101390562951565,
+            -0.017356181517243385,
+            0.012814928777515888,
+            0.008202060125768185,
+            -0.009461646899580956,
+            -0.008821321651339531,
+            -0.019395112991333008,
+            -0.02760138548910618,
+            0.010144098661839962,
+            0.0018925400217995048,
+            0.006192618515342474,
+            0.0022221810650080442,
+            -0.011728061363101006,
+            0.020709464326500893,
+            -0.005055198911577463,
+            0.009048805572092533,
+            -0.017726896330714226,
+            0.006281084381043911,
+            0.011155138723552227,
+            0.003974650055170059,
+            -0.0004581273824442178,
+            0.01404502708464861,
+            0.007338463328778744,
+            0.033027298748493195,
+            0.01270539965480566,
+            0.008189422078430653,
+            -0.009992443025112152,
+            -0.009882913902401924,
+            0.012140902690589428,
+            -0.021080179139971733,
+            -0.0007645988371223211,
+            0.016008129343390465,
+            0.010422134771943092,
+            -0.016260888427495956,
+            0.012680123560130596,
+            -0.01720452681183815,
+            0.010818125680088997,
+            -0.03378557786345482,
+            -0.012494766153395176,
+            0.006592821795493364,
+            0.014011326245963573,
+            0.008871873840689659,
+            0.0002376206684857607,
+            0.017592091113328934,
+            0.015376229770481586,
+            0.0031342231668531895,
+            0.006845581810921431,
+            0.022815797477960587,
+            0.004697122145444155,
+            -0.004671846050769091,
+            0.0035091505851596594,
+            -0.005244768690317869,
+            0.0016776940319687128,
+            -0.010919229127466679,
+            0.011483726091682911,
+            0.008897149935364723,
+            -0.006575970910489559,
+            0.01365746185183525,
+            -0.0009689130820333958,
+            -0.012250431813299656,
+            -0.007325825747102499,
+            -0.004587593022733927,
+            -0.008981402963399887,
+            -0.005649184808135033,
+            0.019513066858053207,
+            -0.015696391463279724,
+            -0.008939276449382305,
+            0.0024496649857610464,
+            0.014685352332890034,
+            -0.0025634069461375475,
+            -0.006571758538484573,
+            -0.008678090758621693,
+            -0.00019167628488503397,
+            -0.01205664873123169,
+            0.01450842060148716,
+            0.026742001995444298,
+            0.023894239217042923,
+            -0.004225303418934345,
+            -0.0032858792692422867,
+            -0.0013448934769257903,
+            0.013632185757160187,
+            0.0010505334939807653,
+            0.00133225554600358,
+            -0.007709178142249584,
+            -0.005725012626498938,
+            -0.018653683364391327,
+            -0.0014038708759471774,
+            0.0017956487135961652,
+            0.00014099264808464795,
+            -0.005973559804260731,
+            -0.013421552255749702,
+            0.011003483086824417,
+            0.008846597746014595,
+            0.0174404364079237,
+            -0.014668501913547516,
+            -0.02980882115662098,
+            0.0195299182087183,
+            -0.003911459818482399,
+            -0.004372747149318457,
+            0.003587084822356701,
+            -0.004574954975396395,
+            0.04333990439772606,
+            0.0051436647772789,
+            0.005939858499914408,
+            -0.011441599577665329,
+            -0.008787620812654495,
+            0.015637414529919624,
+            -0.005219492595642805,
+            -0.002031557960435748,
+            -0.019732125103473663,
+            -0.02466936968266964,
+            0.005131026729941368,
+            -0.024332357570528984,
+            -0.001645045937038958,
+            0.002439133357256651,
+            -0.0013027668464928865,
+            -0.013303598389029503,
+            -0.015949152410030365,
+            -0.003936735913157463,
+            0.011045609600841999,
+            0.01719610020518303,
+            -7.589380402350798e-05,
+            0.006116790696978569,
+            -0.00746905617415905,
+            0.003593403846025467,
+            0.01233468484133482,
+            -0.003881971351802349,
+            -0.01647152192890644,
+            -0.007511182688176632
+          ],
+          "title": "Appendices A. Overview Of Hyper-Parameters"
+        },
+        "type": "chunk"
+      },
+      "bidirectional": true,
+      "properties": {
+        "cosine_similarity": 0.8355387392560063
+      }
+    },
+    {
+      "id": "4db7909e-70bc-44fc-9c9d-ec80a4260843",
+      "type": "cosine_similarity",
+      "source": {
+        "id": "65742957-6bda-446d-a61a-0fe20313c760",
+        "properties": {
+          "page_content": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity\n\nWilliam Fedus∗ liamfedus@google.com Barret Zoph∗ barretzoph@google.com Noam Shazeer noam@google.com Google, Mountain View, CA 94043, USA Editor: Alexander Clark\n\nAbstract\n\nIn deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) models defy this and instead select different parameters for each incoming example. The result is a sparsely-activated model—with an outrageous number of parameters—but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs, and training instability. We address these with the introduction of the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques mitigate the instabilities, and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large (Raffel et al., 2019) to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the \"Colossal Clean Crawled Corpus\", and achieve a 4x speedup over the T5-XXL model.12 Keywords: mixture-of-experts, natural language processing, sparsity, large-scale machine learning, distributed computing arXiv:2101.03961v3 [cs.LG] 16 Jun 2022 Contents\n\n1 Introduction 3 2 Switch Transformer 4 2.1 Simplifying Sparse Routing 5 2.2 Efficient Sparse Routing 6 2.3 Putting It All Together: The Switch Transformer 8 2.4 Improved Training and Fine-Tuning Techniques 8 3 Scaling Properties 11 3.1 Scaling Results on a Step-Basis 12 3.2 Scaling Results on a Time-Basis 13 3.3 Scaling Versus a Larger Dense Model 13 4 Downstream Results 14 4.1 Fine-Tuning 14 4.2 Distillation 16 4.3 Multilingual Learning 17 5 Designing Models with Data, Model, and Expert-Parallelism 18 5.1 Data Parallelism 20 5.2 Model Parallelism 20 5.3 Model and Data Parallelism 21 5.4 Expert and Data Parallelism 22 5.5 Expert, Model and Data Parallelism 22 5.6 Towards Trillion Parameter Models 22 6 Related Work 24 7 Discussion 25 8 Future Work 26 9 Conclusion 27 A Switch for Attention 27 B Preventing Token Dropping with No-Token-Left-Behind 29\n\nC Encouraging Exploration Across Experts 29 D Switch Transformers in Lower Compute Regimes 29 E Relation of Upstream to Downstream Model Performance 32 F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/switch_transformers.md"
+          },
+          "summary": "The Switch Transformer is a sparsely-activated model that simplifies the Mixture of Experts (MoE) approach, allowing for efficient scaling to trillion-parameter models with reduced communication and computational costs. It achieves significant speedups in pre-training and fine-tuning by activating only a subset of neural network weights for each input, maintaining a constant computational cost. The model demonstrates superior scaling properties and improved performance across various natural language processing tasks, including multilingual settings. It also allows for effective distillation into smaller dense models, preserving a portion of the quality gains. The architecture is designed to maximize parameter count while keeping floating point operations per example constant, making it suitable for both large-scale and smaller-scale applications.",
+          "headlines": {
+            "1 Introduction": [],
+            "2 Switch Transformer": [
+              "2.1 Simplifying Sparse Routing",
+              "2.2 Efficient Sparse Routing",
+              "2.3 Putting It All Together: The Switch Transformer",
+              "2.4 Improved Training and Fine-Tuning Techniques"
+            ],
+            "3 Scaling Properties": [
+              "3.1 Scaling Results on a Step-Basis",
+              "3.2 Scaling Results on a Time-Basis",
+              "3.3 Scaling Versus a Larger Dense Model"
+            ],
+            "4 Downstream Results": [
+              "4.1 Fine-Tuning",
+              "4.2 Distillation",
+              "4.3 Multilingual Learning"
+            ],
+            "5 Designing Models with Data, Model, and Expert-Parallelism": [
+              "5.1 Data Parallelism",
+              "5.2 Model Parallelism",
+              "5.3 Model and Data Parallelism",
+              "5.4 Expert and Data Parallelism",
+              "5.5 Expert, Model and Data Parallelism",
+              "5.6 Towards Trillion Parameter Models"
+            ],
+            "6 Related Work": [],
+            "7 Discussion": [],
+            "8 Future Work": [],
+            "9 Conclusion": [],
+            "A Switch for Attention": [],
+            "B Preventing Token Dropping with No-Token-Left-Behind": [],
+            "C Encouraging Exploration Across Experts": [],
+            "D Switch Transformers in Lower Compute Regimes": [],
+            "E Relation of Upstream to Downstream Model Performance": [],
+            "F Pseudo Code for Switch Transformers": []
+          },
+          "summary_embedding": [
+            -0.0060447100549936295,
+            -0.00390331051312387,
+            -0.01784370467066765,
+            -0.0021820589900016785,
+            0.013227885589003563,
+            0.011338187381625175,
+            0.0165580902248621,
+            0.024240797385573387,
+            -0.005986624862998724,
+            0.03596621751785278,
+            -0.004852031823247671,
+            -0.0068811336532235146,
+            -0.003574162255972624,
+            0.014242436736822128,
+            0.011578272096812725,
+            -0.033302053809165955,
+            -0.024519605562090874,
+            0.01050176378339529,
+            0.016093410551548004,
+            -0.006795942317694426,
+            0.007473600562661886,
+            -0.030513975769281387,
+            -0.06124480068683624,
+            -0.006838538218289614,
+            -0.0026099516544491053,
+            0.01029265858232975,
+            0.0010329445358365774,
+            -0.010873507708311081,
+            -0.053469158709049225,
+            0.023311438038945198,
+            0.025294071063399315,
+            0.005092116538435221,
+            -0.029584616422653198,
+            0.0009361362317577004,
+            -0.05161044001579285,
+            -0.04448534920811653,
+            0.0379488542675972,
+            0.012561844661831856,
+            -0.04364892467856407,
+            0.039249956607818604,
+            -0.004875265993177891,
+            -0.0022440163884311914,
+            -0.021499186754226685,
+            -0.0034909071400761604,
+            -0.007039899472147226,
+            -0.009851212613284588,
+            -0.02874819189310074,
+            0.0014037203509360552,
+            0.021886419504880905,
+            -0.0056962003000080585,
+            0.023264970630407333,
+            -0.024271776899695396,
+            0.01936165988445282,
+            0.04240977764129639,
+            -0.06957805901765823,
+            0.024659009650349617,
+            0.017549408599734306,
+            0.05120771750807762,
+            0.06511713564395905,
+            0.018339363858103752,
+            0.021669568493962288,
+            -0.014141756109893322,
+            -0.017766257748007774,
+            -0.011872569099068642,
+            0.004673904739320278,
+            -0.014970434829592705,
+            -0.01891246810555458,
+            0.008449427783489227,
+            -0.020461400970816612,
+            0.008209343999624252,
+            -0.05576157942414284,
+            -0.014730350114405155,
+            -0.034200433641672134,
+            -0.01140014547854662,
+            0.017642343416810036,
+            -0.018215449526906013,
+            0.030126741155982018,
+            0.02997184917330742,
+            0.002638994250446558,
+            0.0007822110201232135,
+            0.021049996837973595,
+            0.006509389728307724,
+            -0.0053438181057572365,
+            0.012321759946644306,
+            0.028159597888588905,
+            -0.01450575515627861,
+            -0.010106786154210567,
+            -0.03454120084643364,
+            0.01861817203462124,
+            0.01589204929769039,
+            -0.04194509983062744,
+            -0.012569589540362358,
+            -0.013824224472045898,
+            0.032341714948415756,
+            -0.009549170732498169,
+            0.04098476096987724,
+            -0.010083552449941635,
+            0.001383390510454774,
+            -0.031149037182331085,
+            -0.009463978931307793,
+            0.017069239169359207,
+            0.006172496825456619,
+            -0.016589069738984108,
+            -0.0012846461031585932,
+            0.016170857474207878,
+            0.006617815233767033,
+            -0.030142230913043022,
+            0.04488807171583176,
+            0.023497309535741806,
+            0.055296897888183594,
+            0.05579255893826485,
+            -0.037391237914562225,
+            -0.002445377642288804,
+            0.0344172865152359,
+            -0.0228312686085701,
+            -0.015040136873722076,
+            -0.034014564007520676,
+            -0.01768881268799305,
+            0.035935238003730774,
+            -0.0318460576236248,
+            0.01556677371263504,
+            0.028066661208868027,
+            -0.025712283328175545,
+            0.003488970920443535,
+            -0.03943582624197006,
+            0.003622566582635045,
+            0.01220559049397707,
+            0.006095050368458033,
+            -0.0026293133851140738,
+            -0.013615119270980358,
+            0.008906363509595394,
+            0.003055269829928875,
+            0.00800798274576664,
+            -0.04615819454193115,
+            0.018478767946362495,
+            -0.00454224506393075,
+            0.04033420979976654,
+            0.042502716183662415,
+            -0.02730768360197544,
+            -0.016743963584303856,
+            0.008557853288948536,
+            0.012104909867048264,
+            0.04188314080238342,
+            -0.007287728600203991,
+            -0.00121784838847816,
+            -0.0007836631848476827,
+            0.007322579622268677,
+            0.047552235424518585,
+            0.004646798130124807,
+            0.0020697612781077623,
+            -0.027230238541960716,
+            0.014978179708123207,
+            -0.019733402878046036,
+            0.027648448944091797,
+            0.023373395204544067,
+            0.02942972257733345,
+            -0.04048910364508629,
+            0.017518429085612297,
+            -0.011818356812000275,
+            -0.005568413063883781,
+            -0.002726121572777629,
+            -0.02856232039630413,
+            -0.0074426219798624516,
+            0.033983584493398666,
+            0.015365412458777428,
+            0.012956822291016579,
+            0.03271345794200897,
+            0.0012052633101120591,
+            -0.042502716183662415,
+            0.03804178908467293,
+            0.024550583213567734,
+            0.04640602692961693,
+            0.023466331884264946,
+            0.010796061716973782,
+            0.006571347359567881,
+            0.013375034555792809,
+            -0.019377149641513824,
+            0.015907539054751396,
+            0.02560385875403881,
+            0.009750531986355782,
+            -0.03167567402124405,
+            -0.0009501734166406095,
+            0.01178737822920084,
+            -0.004546117503196001,
+            0.03701949119567871,
+            0.04104671627283096,
+            0.020569827407598495,
+            0.05030933395028114,
+            0.0020155487582087517,
+            0.017936641350388527,
+            0.005936284549534321,
+            0.02820606529712677,
+            0.030467506498098373,
+            -0.005285732913762331,
+            0.005076627247035503,
+            -0.01136916596442461,
+            -0.03385967016220093,
+            0.014110777527093887,
+            0.01561324205249548,
+            -0.008077683858573437,
+            -0.01697630248963833,
+            0.006033093202859163,
+            0.009169681929051876,
+            -0.01795213110744953,
+            -0.017456471920013428,
+            0.005971135571599007,
+            0.018866000697016716,
+            -0.013576395809650421,
+            0.024689987301826477,
+            -0.04442339017987251,
+            0.029119934886693954,
+            -0.033209118992090225,
+            -0.003384418087080121,
+            -0.010641167871654034,
+            -0.0030843124259263277,
+            0.0041666291654109955,
+            0.011314953677356243,
+            0.013406013138592243,
+            0.002042655134573579,
+            -0.013909416273236275,
+            0.024519605562090874,
+            0.018447788432240486,
+            0.012073930352926254,
+            -0.03148980438709259,
+            -0.014397329650819302,
+            -0.0015673263696953654,
+            -0.008743725717067719,
+            -0.034138478338718414,
+            0.026966920122504234,
+            0.014048820361495018,
+            -0.03844451159238815,
+            0.017797237262129784,
+            0.029538147151470184,
+            0.014420564286410809,
+            -0.004790074657648802,
+            -0.0463130883872509,
+            0.04045812413096428,
+            -0.030436528846621513,
+            -0.033333033323287964,
+            -0.010478530079126358,
+            -0.0003146269591525197,
+            0.02894955314695835,
+            -0.0064358157105743885,
+            -0.050835974514484406,
+            0.01184933539479971,
+            0.020631784573197365,
+            0.00010763872705865651,
+            0.005731051322072744,
+            -0.019593998789787292,
+            0.00028631053282879293,
+            -0.03395260497927666,
+            0.0090844901278615,
+            0.026750069111585617,
+            0.008867640048265457,
+            -0.016418687999248505,
+            0.03351890295743942,
+            -0.0004121613164898008,
+            -0.018695618957281113,
+            -0.003889757441356778,
+            0.006885006092488766,
+            0.034912943840026855,
+            0.004836542531847954,
+            -0.02946070022881031,
+            0.05665995925664902,
+            -0.02100352756679058,
+            0.021344292908906937,
+            0.05631919577717781,
+            -0.019439106807112694,
+            -0.008472662419080734,
+            -0.002065889071673155,
+            0.0026196325197815895,
+            -0.0260530486702919,
+            -0.008937342092394829,
+            0.032341714948415756,
+            0.041728246957063675,
+            -0.018742086365818977,
+            -0.001398879918269813,
+            0.006780453026294708,
+            0.009425255469977856,
+            0.004948840010911226,
+            0.02692045085132122,
+            -0.012128143571317196,
+            -0.014559967443346977,
+            0.05198218300938606,
+            0.028314489871263504,
+            -0.012081675231456757,
+            -0.0164031982421875,
+            0.019888296723365784,
+            -0.024844881147146225,
+            -0.020244551822543144,
+            -0.02431824430823326,
+            -0.020414933562278748,
+            -0.0042518205009400845,
+            0.009525936096906662,
+            0.037484172731637955,
+            -0.03968365862965584,
+            0.012747716158628464,
+            0.021700548008084297,
+            0.02981695532798767,
+            -0.019532041624188423,
+            -0.05910727381706238,
+            -0.0016128262504935265,
+            0.03296129032969475,
+            -0.015458349138498306,
+            -0.049379974603652954,
+            0.015164051204919815,
+            0.0260530486702919,
+            0.05957195535302162,
+            -0.001461805310100317,
+            -0.014954946003854275,
+            0.061926331371068954,
+            0.03026614524424076,
+            -0.004383479710668325,
+            -0.009494957514107227,
+            -0.0127864396199584,
+            0.003396035172045231,
+            0.00465454300865531,
+            -0.03364282101392746,
+            -0.041170634329319,
+            -0.011841590516269207,
+            -0.010284913703799248,
+            0.008108663372695446,
+            -0.040922801941633224,
+            0.03531566634774208,
+            -0.021917399019002914,
+            0.010950954630970955,
+            0.015473837964236736,
+            0.037391237914562225,
+            -0.021344292908906937,
+            -0.01284839678555727,
+            -0.01572941243648529,
+            -0.024968795478343964,
+            -0.014010096900165081,
+            0.021917399019002914,
+            0.007682706229388714,
+            3.4699722164077684e-05,
+            0.010075807571411133,
+            -0.024659009650349617,
+            -0.051052823662757874,
+            0.0291044469922781,
+            0.0015605497173964977,
+            -0.015938518568873405,
+            -0.004623564425855875,
+            -0.01977987214922905,
+            -0.01122201792895794,
+            -0.012608313001692295,
+            -0.039249956607818604,
+            0.021111954003572464,
+            -0.01135367713868618,
+            0.015876561403274536,
+            0.0009250032599084079,
+            0.003659353591501713,
+            0.028624277561903,
+            0.01116006076335907,
+            0.0051811798475682735,
+            -0.03255856782197952,
+            -0.018277406692504883,
+            0.010122275911271572,
+            -0.00561100896447897,
+            -0.018339363858103752,
+            -0.013351799920201302,
+            -0.05037129297852516,
+            -0.03305422514677048,
+            -0.008054450154304504,
+            -0.05415068939328194,
+            0.0020697612781077623,
+            -0.014296649023890495,
+            -0.02232012152671814,
+            -0.005990497302263975,
+            0.03107159025967121,
+            -0.003235333366319537,
+            0.025294071063399315,
+            0.0076594725251197815,
+            0.037360258400440216,
+            0.040117356926202774,
+            0.008426194079220295,
+            0.03065337985754013,
+            -0.0029991210903972387,
+            -0.016867877915501595,
+            0.05486319959163666,
+            -0.01881953328847885,
+            0.013529927469789982,
+            -0.008604321628808975,
+            0.008604321628808975,
+            0.03299226611852646,
+            -0.003794885240495205,
+            0.02007416822016239,
+            -0.0311955064535141,
+            -0.07998688519001007,
+            0.01637221872806549,
+            0.028159597888588905,
+            -0.06418777257204056,
+            0.014691627584397793,
+            -0.020275529474020004,
+            0.02052335813641548,
+            0.02411688305437565,
+            -0.0009114501299336553,
+            -0.014064309187233448,
+            0.005595519673079252,
+            0.0020833145827054977,
+            -0.01456771232187748,
+            -0.002691270550712943,
+            -0.009417511522769928,
+            0.008774704299867153,
+            -0.017007282003760338,
+            -0.013406013138592243,
+            -0.034231413155794144,
+            0.010873507708311081,
+            0.01233724970370531,
+            -0.003969140350818634,
+            0.0011500825639814138,
+            -0.03277541697025299,
+            -0.007876322604715824,
+            -0.02145271934568882,
+            0.03382869064807892,
+            0.012058441527187824,
+            -0.030699847266077995,
+            -0.007814365439116955,
+            0.015264731831848621,
+            -0.007876322604715824,
+            0.0008906363509595394,
+            0.024132372811436653,
+            0.02103450708091259,
+            0.0028364830650389194,
+            -0.0009700191440060735,
+            0.02438020147383213,
+            0.007427132688462734,
+            -0.01720864325761795,
+            0.028794659301638603,
+            0.014288905076682568,
+            0.060160547494888306,
+            -0.03271345794200897,
+            0.001297231181524694,
+            -0.04135650396347046,
+            -0.031118059530854225,
+            0.006408709101378918,
+            0.014350862242281437,
+            0.007597515359520912,
+            -0.028252532705664635,
+            0.005978880450129509,
+            0.02004319056868553,
+            0.01469937153160572,
+            0.009704063646495342,
+            0.0037580979987978935,
+            -0.036740683019161224,
+            -0.000691695255227387,
+            -0.011214273050427437,
+            -0.0038781403563916683,
+            0.035656433552503586,
+            -0.008186109364032745,
+            -0.05151750147342682,
+            -0.0006418389966711402,
+            -0.03255856782197952,
+            -0.0353776253759861,
+            0.021313315257430077,
+            -0.00839521549642086,
+            -0.0039207362569868565,
+            -0.013986863195896149,
+            -0.000537770101800561,
+            -0.010524998418986797,
+            0.027509044855833054,
+            0.028345469385385513,
+            0.022490503266453743,
+            -0.05384090170264244,
+            0.009479468688368797,
+            0.00023766436788719147,
+            -0.05817791447043419,
+            0.0009709871956147254,
+            -0.007117345929145813,
+            0.011191039346158504,
+            -0.020213572308421135,
+            -0.02414786070585251,
+            0.011500826105475426,
+            0.03392162546515465,
+            0.014304393902420998,
+            0.0321558453142643,
+            -0.013367289677262306,
+            -0.09250226616859436,
+            0.018292896449565887,
+            -0.008689512498676777,
+            0.04126356914639473,
+            0.00865078903734684,
+            -0.004278926644474268,
+            -0.015845581889152527,
+            -0.00865078903734684,
+            -0.006943091284483671,
+            0.01283290795981884,
+            0.004530628211796284,
+            -0.005626498255878687,
+            0.0016631665639579296,
+            0.013708055019378662,
+            -0.005967263597995043,
+            0.007481344975531101,
+            0.003651608945801854,
+            -0.013189162127673626,
+            -0.06059424951672554,
+            -0.04055105894804001,
+            -0.018540725111961365,
+            0.050835974514484406,
+            -0.0069159846752882,
+            -0.046684831380844116,
+            -0.010850274004042149,
+            0.05058814212679863,
+            0.012259802781045437,
+            -0.0334569476544857,
+            -0.0007841471815481782,
+            0.0050882440991699696,
+            0.015458349138498306,
+            0.004546117503196001,
+            -0.027292195707559586,
+            0.02820606529712677,
+            0.0008412640891037881,
+            0.02300165221095085,
+            -0.018339363858103752,
+            -0.005312839522957802,
+            0.035780347883701324,
+            -0.024844881147146225,
+            0.023791607469320297,
+            0.0055219451896846294,
+            0.031629208475351334,
+            0.002952652983367443,
+            0.010060317814350128,
+            0.004704883322119713,
+            0.0186491496860981,
+            0.004275054205209017,
+            0.0276639387011528,
+            0.011462102644145489,
+            0.006428070832043886,
+            0.021623101085424423,
+            -0.009378788061439991,
+            -0.014513500034809113,
+            0.01885051093995571,
+            -0.0014695499558001757,
+            0.025232113897800446,
+            -0.020538847893476486,
+            0.002726121572777629,
+            -0.0016883367206901312,
+            -0.017967619001865387,
+            -0.040922801941633224,
+            -0.0007405834621749818,
+            0.021747015416622162,
+            -0.011671207845211029,
+            -0.0228312686085701,
+            -0.013715799897909164,
+            -0.01585332676768303,
+            0.0038490979932248592,
+            0.034169457852840424,
+            -0.0398695282638073,
+            0.00963436160236597,
+            -0.000830615172162652,
+            -0.03125746175646782,
+            -0.0239619892090559,
+            -0.01093546487390995,
+            0.014614180661737919,
+            -0.01652711257338524,
+            -0.021173911169171333,
+            -0.028469383716583252,
+            -0.008410705253481865,
+            0.006048582494258881,
+            -0.05065010115504265,
+            -0.004054331686347723,
+            0.019965743646025658,
+            0.0276639387011528,
+            -0.003099801717326045,
+            -0.02177799493074417,
+            0.0012468908680602908,
+            0.022351099178195,
+            0.02936776541173458,
+            0.014226946979761124,
+            0.027586491778492928,
+            -0.002178186783567071,
+            -0.0011675080750137568,
+            -0.0073148347437381744,
+            0.01307299267500639,
+            0.0011287847300991416,
+            0.036616768687963486,
+            0.0122210793197155,
+            -0.020306508988142014,
+            0.0013959755888208747,
+            -0.006431943271309137,
+            0.029181893914937973,
+            -0.02270735427737236,
+            -9.832092473516241e-05,
+            0.03531566634774208,
+            0.027741385623812675,
+            0.004933350719511509,
+            -0.0143121387809515,
+            -0.05142456665635109,
+            0.021592123433947563,
+            -0.01765783317387104,
+            0.031025122851133347,
+            0.016666516661643982,
+            0.003397971158847213,
+            -0.0034212053287774324,
+            0.012368228286504745,
+            -0.022087780758738518,
+            -0.004406713880598545,
+            -0.021142931655049324,
+            0.04389675334095955,
+            -0.043184246867895126,
+            0.008472662419080734,
+            0.029445212334394455,
+            0.011926782317459583,
+            -0.019082851707935333,
+            -0.005425137002021074,
+            0.029212871566414833,
+            -0.00162637943867594,
+            0.01308848150074482,
+            0.0058665829710662365,
+            -0.026935940608382225,
+            0.01202746294438839,
+            -0.031660184264183044,
+            0.02177799493074417,
+            -0.013436991721391678,
+            0.018447788432240486,
+            0.03342596814036369,
+            -0.0030397805385291576,
+            -0.0038239278364926577,
+            -0.016263794153928757,
+            -0.00969631876796484,
+            0.0662323608994484,
+            0.01054823212325573,
+            0.03726732358336449,
+            0.0018974426202476025,
+            -0.009773765690624714,
+            0.007969259284436703,
+            0.0063041565008461475,
+            -0.012329504825174809,
+            0.010494019836187363,
+            0.024178840219974518,
+            0.0012730291346088052,
+            -0.002412462839856744,
+            -0.019408127292990685,
+            -0.03438630700111389,
+            0.003727119415998459,
+            -0.030932186171412468,
+            0.022691864520311356,
+            0.009208405390381813,
+            0.017456471920013428,
+            0.008240322582423687,
+            -0.015907539054751396,
+            0.03200095146894455,
+            0.05461536720395088,
+            0.0019768255297094584,
+            0.013375034555792809,
+            0.007938280701637268,
+            0.007036027032881975,
+            -0.011338187381625175,
+            0.04702559858560562,
+            0.000662652775645256,
+            0.005169562995433807,
+            0.022536970674991608,
+            0.02145271934568882,
+            0.022939693182706833,
+            0.002304037567228079,
+            -0.011694442480802536,
+            0.010478530079126358,
+            -0.0006302220281213522,
+            0.004445436876267195,
+            -0.010695381090044975,
+            -0.027865299955010414,
+            0.007492962293326855,
+            0.01302652433514595,
+            0.022598929703235626,
+            0.021592123433947563,
+            -0.03178409859538078,
+            -0.01855621486902237,
+            0.03364282101392746,
+            -0.02997184917330742,
+            0.03612111136317253,
+            -0.0014424435794353485,
+            -0.023807097226381302,
+            -0.0058356039226055145,
+            0.016929835081100464,
+            0.02656419761478901,
+            -0.029181893914937973,
+            0.01804506592452526,
+            0.029693040996789932,
+            -0.0074426219798624516,
+            0.0005939188995398581,
+            0.012112653814256191,
+            0.02447313815355301,
+            -0.009293596260249615,
+            -0.0024027819745242596,
+            -0.0013320822035893798,
+            -0.011926782317459583,
+            -0.018370341509580612,
+            -0.0044725434854626656,
+            -0.022629907354712486,
+            0.005564541090279818,
+            -0.007365175057202578,
+            -0.003481226274743676,
+            -0.008581086993217468,
+            0.01154729351401329,
+            0.001979729626327753,
+            -0.019857317209243774,
+            -0.022335609421133995,
+            0.01652711257338524,
+            0.03887821361422539,
+            -0.018215449526906013,
+            -0.02106548473238945,
+            0.024442158639431,
+            -0.01759587600827217,
+            -0.01601596549153328,
+            0.036678727716207504,
+            0.013739033602178097,
+            0.007736918982118368,
+            -0.03358086198568344,
+            -0.030049296095967293,
+            -0.03274443745613098,
+            0.006296411622315645,
+            0.03785591572523117,
+            -0.0028752065263688564,
+            0.02569679357111454,
+            0.03701949119567871,
+            0.0057891360484063625,
+            -0.028934063389897346,
+            -0.014823286794126034,
+            -0.045104920864105225,
+            0.060160547494888306,
+            0.0011481463443487883,
+            0.05161044001579285,
+            0.016294771805405617,
+            -0.01456771232187748,
+            0.00419760774821043,
+            -0.013251119293272495,
+            0.026192452758550644,
+            0.02280029095709324,
+            0.025929134339094162,
+            0.006582964211702347,
+            0.022505993023514748,
+            -0.0037522895727306604,
+            -0.012197845615446568,
+            0.007307090330868959,
+            0.014536733739078045,
+            0.0002497654058970511,
+            -0.011392400600016117,
+            0.015783624723553658,
+            -0.01307299267500639,
+            -0.011361422017216682,
+            0.023295948281884193,
+            0.01368482131510973,
+            -0.012344993650913239,
+            0.014064309187233448,
+            -0.008565598167479038,
+            0.01239920686930418,
+            -0.023915521800518036,
+            0.022691864520311356,
+            -0.03262052312493324,
+            0.009587894193828106,
+            0.0015053689712658525,
+            -0.0073148347437381744,
+            0.024550583213567734,
+            -0.0027454833034425974,
+            -0.05071205645799637,
+            0.03550153970718384,
+            -0.008015726692974567,
+            0.0037213109899312258,
+            0.006381602957844734,
+            0.011392400600016117,
+            -0.0012623801594600081,
+            0.03807276859879494,
+            0.01239920686930418,
+            0.015969496220350266,
+            0.02199484594166279,
+            0.011911292560398579,
+            -0.008433938957750797,
+            -0.004096927121281624,
+            -0.0016137943603098392,
+            -0.01784370467066765,
+            0.025944624096155167,
+            0.0018674320308491588,
+            0.03962169960141182,
+            -0.018168980255723,
+            -0.02300165221095085,
+            0.028066661208868027,
+            0.007597515359520912,
+            -0.02351279929280281,
+            -0.0017377089243382215,
+            0.01202746294438839,
+            -0.009185170754790306,
+            0.027385130524635315,
+            0.01196550577878952,
+            0.019113829359412193,
+            -0.018060555681586266,
+            -0.004340883810073137,
+            -0.005072754807770252,
+            -0.015938518568873405,
+            0.02238207869231701,
+            0.020771188661456108,
+            0.011152315884828568,
+            -0.011880313977599144,
+            -0.01307299267500639,
+            0.020600805059075356,
+            0.035036858171224594,
+            -0.0002744515077210963,
+            0.005672966130077839,
+            0.017487451434135437,
+            0.002373739378526807,
+            -0.04470219835639,
+            0.016232814639806747,
+            -0.014025585725903511,
+            0.011214273050427437,
+            0.019593998789787292,
+            -0.021917399019002914,
+            -0.03196997195482254,
+            -0.011539548635482788,
+            -0.0414804182946682,
+            0.00493722315877676,
+            -0.0069159846752882,
+            0.013576395809650421,
+            -0.010176488198339939,
+            -0.0014831031439825892,
+            0.016666516661643982,
+            0.0038103745318949223,
+            0.02177799493074417,
+            -0.011678952723741531,
+            -0.012073930352926254,
+            -0.020213572308421135,
+            -0.020786678418517113,
+            0.004278926644474268,
+            -0.030405549332499504,
+            -0.004488032776862383,
+            0.021375272423028946,
+            -0.04783104360103607,
+            -0.03689557686448097,
+            0.02898053266108036,
+            -0.008503641001880169,
+            0.020755698904395103,
+            0.022955182939767838,
+            -0.015094350092113018,
+            -0.0008717587334103882,
+            -0.018695618957281113,
+            0.02258343994617462,
+            -0.007837600074708462,
+            0.026300877332687378,
+            -0.0027551641687750816,
+            0.0006292539183050394,
+            -0.02659517526626587,
+            0.020306508988142014,
+            0.005227648187428713,
+            0.012275291606783867,
+            -0.02390003204345703,
+            0.006815304048359394,
+            -0.03133491054177284,
+            -0.01784370467066765,
+            0.0180140882730484,
+            -0.011152315884828568,
+            0.027989214286208153,
+            -0.006060199346393347,
+            0.028438406065106392,
+            0.022722844034433365,
+            0.024751944467425346,
+            0.02676555886864662,
+            -0.02363671362400055,
+            0.008495896123349667,
+            0.03726732358336449,
+            0.00865078903734684,
+            -0.015876561403274536,
+            0.008170620538294315,
+            -0.04863648861646652,
+            -0.008062195032835007,
+            0.05030933395028114,
+            0.002127846470102668,
+            0.023141054436564445,
+            0.016294771805405617,
+            -0.002873270306736231,
+            -0.005688455421477556,
+            0.010269423946738243,
+            -0.04296739399433136,
+            -0.023172033950686455,
+            0.022629907354712486,
+            -0.021561143919825554,
+            6.54212328754511e-07,
+            0.03122648410499096,
+            0.012120398692786694,
+            -0.0009109660750254989,
+            -0.05108380317687988,
+            0.02100352756679058,
+            0.01178737822920084,
+            0.014443797990679741,
+            -0.010300402529537678,
+            0.026068538427352905,
+            0.01733255758881569,
+            0.038413532078266144,
+            0.012755461037158966,
+            -0.002679653698578477,
+            -0.03782493621110916,
+            0.009177426807582378,
+            -0.021685058251023293,
+            0.0011471783509477973,
+            0.0074503663927316666,
+            -0.02273833192884922,
+            -0.024782923981547356,
+            0.0043912241235375404,
+            0.004588713403791189,
+            0.011880313977599144,
+            0.022087780758738518,
+            0.009417511522769928,
+            -0.021204888820648193,
+            0.015404135920107365,
+            -0.05446047708392143,
+            -0.015450604259967804,
+            -0.023884542286396027,
+            0.02566581591963768,
+            0.020012211054563522,
+            0.010184233076870441,
+            0.005545179359614849,
+            0.010904486291110516,
+            -0.020988039672374725,
+            -0.009239383973181248,
+            0.020631784573197365,
+            0.011074868962168694,
+            -0.012840652838349342,
+            0.04113965481519699,
+            -0.00011126903700642288,
+            0.031210994347929955,
+            -0.02280029095709324,
+            0.002941036131232977,
+            -0.001241082325577736,
+            -0.03649285435676575,
+            0.009130958467721939,
+            0.01981084980070591,
+            0.02151467651128769,
+            -0.01861817203462124,
+            0.0024143988266587257,
+            0.01048627495765686,
+            0.004766840487718582,
+            0.038351576775312424,
+            0.027121812105178833,
+            -0.044206541031599045,
+            0.011663463898003101,
+            0.012995545752346516,
+            -0.009363298304378986,
+            0.048419639468193054,
+            -0.017518429085612297,
+            0.017673322930932045,
+            0.008240322582423687,
+            -0.01798310875892639,
+            0.02318752370774746,
+            -0.01469937153160572,
+            -0.013282098807394505,
+            0.00650551775470376,
+            0.009192915633320808,
+            -0.0001770381786627695,
+            -0.037577107548713684,
+            -0.017859194427728653,
+            -0.005719434004276991,
+            0.0038297362625598907,
+            0.01562873087823391,
+            0.013498948886990547,
+            0.009967382065951824,
+            0.005161818582564592,
+            -0.00945623405277729,
+            -0.0016089539276435971,
+            0.008124152198433876,
+            -0.0005532593932002783,
+            0.0019216446671634912,
+            -0.01926872320473194,
+            0.02698240801692009,
+            0.02572777308523655,
+            -0.0018354853382334113,
+            -0.04652994126081467,
+            -0.0036709706764668226,
+            0.0037077576853334904,
+            -0.017735280096530914,
+            -0.0035315665882080793,
+            -0.04519785940647125,
+            -0.01810702309012413,
+            0.0143121387809515,
+            -0.014614180661737919,
+            -0.0460033044219017,
+            0.009076746180653572,
+            -0.013886182568967342,
+            0.0063970922492444515,
+            -0.00932457484304905,
+            0.005762029904872179,
+            0.003651608945801854,
+            0.005680711008608341,
+            -0.005909178406000137,
+            0.05108380317687988,
+            -0.03466511517763138,
+            0.028051171451807022,
+            0.008565598167479038,
+            -0.012437930330634117,
+            -0.02261441759765148,
+            0.00887538492679596,
+            -0.044981006532907486,
+            -0.033209118992090225,
+            0.016743963584303856,
+            -0.01781272701919079,
+            0.011074868962168694,
+            0.06161654368042946,
+            0.007330324035137892,
+            -0.023342415690422058,
+            -0.01233724970370531,
+            -0.021049996837973595,
+            -0.04191412031650543,
+            0.003442503046244383,
+            0.005463860463351011,
+            -0.004948840010911226,
+            -0.03807276859879494,
+            -0.021716037765145302,
+            0.04615819454193115,
+            -0.012004229240119457,
+            0.012422440573573112,
+            0.01080380566418171,
+            -0.0019410063978284597,
+            -0.04860550910234451,
+            -0.021019017323851585,
+            0.0395597442984581,
+            0.021886419504880905,
+            -0.008426194079220295,
+            0.0028364830650389194,
+            -0.018277406692504883,
+            -0.003289545886218548,
+            0.006296411622315645,
+            -0.009107724763453007,
+            -0.03562545403838158,
+            0.02067825198173523,
+            0.0054832217283546925,
+            -0.023373395204544067,
+            -0.025805220007896423,
+            -0.03296129032969475,
+            0.009944148361682892,
+            0.02453509531915188,
+            -0.01968693546950817,
+            0.005823987070471048,
+            0.00027203132049180567,
+            -0.000881923595443368,
+            0.00042740863864310086,
+            -0.009959637187421322,
+            -0.019764382392168045,
+            -0.020151615142822266,
+            0.03268248215317726,
+            -0.0060059865936636925,
+            0.0074387495405972,
+            -0.03630698472261429,
+            -0.010974188335239887,
+            0.011322698555886745,
+            -0.002395037328824401,
+            -0.013452480547130108,
+            -0.009092235006392002,
+            -0.01739451475441456,
+            -0.009882191196084023,
+            -0.007876322604715824,
+            0.022660886868834496,
+            0.006168624851852655,
+            0.005409647710621357,
+            0.0023795480374246836,
+            -0.008581086993217468,
+            0.02898053266108036,
+            0.017874684184789658,
+            -0.014188224449753761,
+            -0.00920066051185131,
+            0.03451022133231163,
+            0.02665713243186474,
+            -0.009115469641983509,
+            0.01798310875892639,
+            0.023264970630407333,
+            -0.0010484338272362947,
+            -0.01196550577878952,
+            0.001922612776979804,
+            -0.0009550138493068516,
+            -0.0032972905319184065,
+            0.00845717266201973,
+            -0.015365412458777428,
+            0.0315207801759243,
+            0.04287445917725563,
+            -0.002145271748304367,
+            0.009587894193828106,
+            -0.003727119415998459,
+            -0.008472662419080734,
+            0.017069239169359207,
+            0.009316830895841122,
+            0.04625113308429718,
+            0.01098193321377039,
+            0.005680711008608341,
+            -0.033983584493398666,
+            -0.001229465357027948,
+            -0.01184933539479971,
+            0.03308520466089249,
+            0.02061629481613636,
+            0.00889087375253439,
+            -0.008379725739359856,
+            0.01344473659992218,
+            0.008341003209352493,
+            0.00661007035523653,
+            0.020569827407598495,
+            -0.016542602330446243,
+            -0.017286090180277824,
+            0.030467506498098373,
+            -0.00672236829996109,
+            0.01289486512541771,
+            0.02029101923108101,
+            -0.002089123008772731,
+            0.010842529125511646,
+            0.07100307941436768,
+            -0.006184114143252373,
+            0.015713922679424286,
+            -0.019082851707935333,
+            0.0015789433382451534,
+            -0.007640110794454813,
+            -0.03370477631688118,
+            0.018246427178382874,
+            0.0039962464943528175,
+            0.00865078903734684,
+            0.01198099460452795,
+            0.009053511545062065,
+            -0.007946024648845196,
+            -0.00963436160236597,
+            0.015195030719041824,
+            0.0019080914789810777,
+            0.00033568276558071375,
+            0.001336922636255622,
+            -0.0065210070461034775,
+            -0.004011735785752535,
+            -0.021948376670479774,
+            0.02763296104967594,
+            0.007140580099076033,
+            0.030606910586357117,
+            -0.055296897888183594,
+            0.030095763504505157,
+            -0.00789568480104208,
+            0.02554190158843994,
+            -0.016201836988329887,
+            -0.011090358719229698,
+            0.001609921921044588,
+            0.018153492361307144,
+            0.004491904750466347,
+            0.006462921854108572,
+            0.011888058856129646,
+            0.023233991116285324,
+            -0.03444826602935791,
+            0.017053749412298203,
+            0.023110076785087585,
+            0.0048675211146473885,
+            0.0003535923024173826,
+            0.005165690556168556,
+            0.015458349138498306,
+            0.007198664825409651,
+            -0.023048119619488716,
+            -0.0025692922063171864,
+            0.017611365765333176,
+            -0.02306360937654972,
+            0.0036980770528316498,
+            -0.006923729553818703,
+            0.004623564425855875,
+            -0.0014705180656164885,
+            -0.03509881719946861,
+            -0.00100099784322083,
+            -0.025743262842297554,
+            0.01923774555325508,
+            0.009556914679706097,
+            -0.025867177173495293,
+            0.003932353109121323,
+            0.025650326162576675,
+            -0.0003695656778290868,
+            0.00821708794683218,
+            0.0004683101433329284,
+            0.00869725737720728,
+            -0.00974278710782528,
+            -0.01493945624679327,
+            0.004801691509783268,
+            0.01531120017170906,
+            -0.01493945624679327,
+            -0.007431004662066698,
+            -0.009510447271168232,
+            0.030560443177819252,
+            0.0006674932083114982,
+            0.004189862869679928,
+            -0.005045648664236069,
+            -0.01858719252049923,
+            -0.0032237162813544273,
+            -0.021142931655049324,
+            0.004367990419268608,
+            -0.0009046735358424485,
+            0.04197607934474945,
+            0.0038161829579621553,
+            -0.02029101923108101,
+            0.0032972905319184065,
+            -0.03136589005589485,
+            0.0005000148667022586,
+            -0.007276111748069525,
+            -0.02004319056868553,
+            0.0015005286550149322,
+            0.0016525175888091326,
+            0.012631546705961227,
+            -0.03255856782197952,
+            -0.009301341138780117,
+            0.018416810780763626,
+            -0.00020680672605521977,
+            -0.010687636211514473,
+            -0.017828216776251793,
+            0.00035189816844649613,
+            -0.011338187381625175,
+            -0.027973726391792297,
+            -0.02850036323070526,
+            -0.0026719090528786182,
+            0.013491204008460045,
+            -0.018277406692504883,
+            -0.014993669465184212,
+            0.0031249718740582466,
+            0.021592123433947563,
+            -0.009781510569155216,
+            0.008302279748022556,
+            0.02241305634379387,
+            -0.008255811408162117,
+            -0.005514200776815414,
+            -0.000941460661124438,
+            -0.009130958467721939,
+            0.004267309792339802,
+            0.006300284061580896,
+            0.043339136987924576,
+            -0.032403673976659775,
+            -0.0018238683696836233,
+            0.014815541915595531,
+            0.0034153966698795557,
+            0.0025905899237841368,
+            0.002503462601453066,
+            -0.0007841471815481782,
+            0.01344473659992218,
+            0.01178737822920084,
+            0.015466093085706234,
+            0.00100099784322083,
+            -0.0033011629711836576,
+            0.006408709101378918,
+            -0.021173911169171333,
+            0.015721667557954788,
+            0.027958236634731293,
+            -0.0033999073784798384,
+            0.005525817628949881,
+            -0.013731288723647594,
+            -0.008759214542806149,
+            0.013134949840605259,
+            0.0026641644071787596,
+            -0.005068882368505001,
+            -0.009394276887178421,
+            0.017967619001865387,
+            0.019082851707935333,
+            0.006668155547231436,
+            -0.037453193217515945,
+            -0.02898053266108036,
+            -0.009959637187421322,
+            0.020569827407598495,
+            0.014916222542524338,
+            0.0010339126456528902,
+            -0.010540487244725227,
+            -0.009595638141036034,
+            -0.011903548613190651,
+            0.010989678092300892,
+            0.02721474878489971,
+            -0.032372694462537766,
+            0.0008214184199459851,
+            -0.029383255168795586,
+            0.03068435750901699,
+            0.0020000594668090343,
+            0.011446612887084484,
+            -0.0008185141487047076,
+            -0.015140817500650883,
+            -0.029181893914937973,
+            0.013824224472045898,
+            -0.004395096562802792,
+            0.001024231780320406,
+            -0.0031753121875226498,
+            0.0201051477342844,
+            0.012887120246887207,
+            -0.0014472840121015906,
+            0.01968693546950817,
+            0.008178364485502243,
+            0.016480645164847374,
+            -0.010153254494071007,
+            -0.04135650396347046,
+            0.019965743646025658,
+            0.007562664337456226,
+            0.014219203032553196,
+            0.004739734344184399,
+            0.004588713403791189,
+            -0.020771188661456108,
+            0.002724185585975647,
+            0.004333139397203922,
+            0.014405074529349804,
+            0.011028401553630829,
+            0.0025615475606173277,
+            0.011678952723741531,
+            -0.02228914201259613,
+            0.01929970271885395,
+            -0.01631026156246662,
+            0.01006806269288063,
+            0.004445436876267195,
+            -0.026873983442783356,
+            0.007547175046056509,
+            0.007822110317647457,
+            -0.0019187404541298747,
+            -0.017549408599734306,
+            0.02160761132836342,
+            -0.0034831624943763018,
+            -0.03509881719946861,
+            0.0360281765460968,
+            -0.0043447562493383884,
+            0.019346170127391815,
+            0.03209388628602028,
+            -0.01771979033946991,
+            -0.01993476413190365,
+            0.00906900130212307,
+            0.010408828034996986,
+            -0.002873270306736231,
+            0.017859194427728653,
+            -0.002919738180935383,
+            -0.02232012152671814,
+            0.017766257748007774,
+            -0.028918573632836342,
+            -0.014234691858291626,
+            0.020337486639618874,
+            -0.03624502569437027,
+            -0.0021026763133704662,
+            0.03175312280654907,
+            -0.024070415645837784,
+            0.005734923295676708,
+            0.005196669604629278,
+            -0.0019351978553459048,
+            0.0012236569309607148,
+            -0.006451305001974106,
+            0.03801080957055092,
+            -0.010865762829780579,
+            0.0201051477342844,
+            0.01929970271885395,
+            -0.023388884961605072,
+            0.01977987214922905,
+            -0.001109423115849495,
+            0.01675945147871971,
+            -0.003940097521990538,
+            0.03466511517763138,
+            -0.006017603911459446,
+            -0.011330443434417248,
+            0.006431943271309137,
+            0.008488151244819164,
+            -0.009959637187421322,
+            0.008681767620146275,
+            -0.018122512847185135,
+            0.027152791619300842,
+            0.02091059274971485,
+            -0.0002691270783543587,
+            0.013800990767776966,
+            0.0004421719058882445,
+            -0.020833145827054977,
+            0.00789568480104208,
+            -0.027121812105178833,
+            -0.009967382065951824,
+            0.01646515540778637,
+            -0.005762029904872179,
+            0.007380664348602295,
+            -0.02064727433025837,
+            0.028004704043269157,
+            0.0077524082735180855,
+            -0.013591884635388851,
+            -0.0008824076503515244,
+            0.023760627955198288,
+            -0.005432881414890289,
+            0.009355553425848484,
+            0.02070923149585724,
+            0.015713922679424286,
+            0.013615119270980358,
+            0.016263794153928757,
+            0.03351890295743942,
+            0.005328328814357519,
+            0.005568413063883781,
+            0.003519949736073613,
+            0.016991792246699333,
+            -0.008557853288948536,
+            0.018029578030109406,
+            0.0008093173382803798,
+            -0.013576395809650421,
+            0.021499186754226685,
+            -0.0018684001406654716,
+            -0.013622863218188286,
+            -0.0040891822427511215,
+            -0.004337011836469173,
+            0.03271345794200897,
+            0.022041313350200653,
+            -0.004468671046197414,
+            -0.024442158639431,
+            -0.00394784240052104,
+            0.016682006418704987,
+            -0.003167567541822791,
+            0.011136826127767563,
+            0.0007952801533974707,
+            -0.01586107164621353,
+            -0.009595638141036034,
+            0.0063583687879145145,
+            0.010586955584585667,
+            -0.008921852335333824,
+            0.010346870869398117,
+            -0.0228312686085701,
+            -0.004011735785752535,
+            0.00046613195445388556,
+            -0.004844287410378456,
+            0.0053438181057572365,
+            -0.011314953677356243,
+            0.004948840010911226,
+            0.016062432900071144,
+            0.01026167906820774,
+            0.004832670092582703,
+            0.006989559158682823,
+            0.012066186405718327,
+            0.009161937050521374,
+            -0.01627928391098976,
+            0.00428667152300477,
+            -0.005072754807770252,
+            -0.02345084212720394,
+            0.005831731949001551,
+            0.00902253296226263,
+            -0.01122201792895794,
+            0.009386532939970493,
+            -0.010958699509501457,
+            0.03296129032969475,
+            0.01030814740806818,
+            -0.037391237914562225,
+            -0.0033379499800503254,
+            -0.0016244432190433145,
+            0.014885243959724903,
+            0.012298526242375374,
+            0.008550108410418034,
+            -0.017518429085612297,
+            -0.0047823297791182995,
+            -0.009835722856223583,
+            0.0012827098835259676,
+            -0.012306271120905876,
+            0.017456471920013428,
+            0.022567950189113617,
+            -0.036337960511446,
+            0.01968693546950817,
+            -0.010432061739265919,
+            -0.023652203381061554,
+            0.020476890727877617,
+            0.011500826105475426,
+            0.01942361705005169,
+            -0.009402021765708923,
+            0.018896980211138725,
+            -0.0001331114035565406,
+            0.025510922074317932,
+            -0.004228586331009865,
+            -0.022087780758738518,
+            -0.012499887496232986,
+            -0.03209388628602028,
+            -0.0051811798475682735,
+            0.006819176487624645,
+            -0.00045524101005867124,
+            0.013111716136336327,
+            0.014226946979761124,
+            -0.025712283328175545,
+            0.007295473478734493,
+            0.032589543610811234,
+            0.014637414366006851,
+            0.023311438038945198,
+            -0.011276230216026306,
+            -0.0010794125264510512,
+            -0.0026060794480144978,
+            -0.013770012184977531,
+            -0.03212486580014229,
+            -0.0017493258928880095,
+            -0.036585792899131775,
+            0.005680711008608341,
+            -0.016387708485126495,
+            -0.013096226379275322,
+            0.023264970630407333,
+            -0.034138478338718414,
+            0.0014608372002840042,
+            0.013708055019378662,
+            0.010540487244725227,
+            0.012391461990773678,
+            -0.00162637943867594,
+            0.015032391995191574,
+            -0.029243851080536842,
+            -0.010284913703799248,
+            -0.004491904750466347,
+            -0.01981084980070591,
+            -0.007338068913668394,
+            -0.018060555681586266,
+            -0.013127204962074757,
+            -0.003566417610272765,
+            -0.010060317814350128,
+            0.002247888594865799,
+            0.0032062907703220844,
+            -0.014714861288666725,
+            0.03206290677189827,
+            0.003157886676490307,
+            0.018958937376737595,
+            0.0033418224193155766,
+            0.004449309315532446,
+            0.018788553774356842,
+            -0.0026331855915486813,
+            -0.0058278595097362995,
+            0.030963165685534477,
+            -0.00032672801171429455,
+            0.008759214542806149,
+            0.019082851707935333,
+            0.004724245052784681,
+            0.0038529701996594667,
+            -0.0260530486702919,
+            0.012887120246887207,
+            -0.027447087690234184,
+            0.005080499220639467,
+            -0.0077175572514534,
+            -0.024302754551172256,
+            -0.028128618374466896,
+            -0.006571347359567881,
+            0.012058441527187824,
+            -0.02898053266108036,
+            -0.00613377382978797,
+            0.002315654419362545,
+            -0.00400399137288332,
+            -0.002673845272511244,
+            0.007241260726004839,
+            0.029150914400815964,
+            -0.033209118992090225,
+            0.008751469664275646,
+            0.005595519673079252,
+            -0.0005053392960689962,
+            0.013297587633132935,
+            0.008038961328566074,
+            0.003343758638948202,
+            0.020848635584115982,
+            0.006637176964432001,
+            0.002458930714055896,
+            -0.0025576751213520765,
+            -0.014242436736822128,
+            0.004000118933618069,
+            0.02154565416276455,
+            0.007562664337456226,
+            0.018385831266641617,
+            -0.008914108388125896,
+            0.002621568739414215,
+            -0.01455222349613905,
+            0.005552923772484064,
+            -0.002867461647838354,
+            0.004309905227273703,
+            -0.012236569076776505,
+            0.008093173615634441,
+            0.0028771425131708384,
+            -0.0043447562493383884,
+            0.018060555681586266,
+            -0.002319526858627796,
+            0.016480645164847374,
+            0.004015608225017786,
+            0.015620986931025982,
+            -0.0035412474535405636,
+            0.01178737822920084,
+            0.0005731051205657423,
+            -0.017905661836266518,
+            -0.017022771760821342,
+            -0.039249956607818604,
+            0.007702067960053682,
+            0.03531566634774208,
+            -0.0009564659558236599,
+            -0.03444826602935791,
+            0.010145509615540504,
+            -0.011942271143198013,
+            -0.009975126944482327,
+            -0.022862248122692108,
+            0.013862947933375835,
+            0.012430185452103615,
+            0.007415515370666981,
+            0.012252057902514935,
+            -0.006997303571552038,
+            0.01026167906820774,
+            -0.005216030869632959,
+            0.008356492035090923,
+            -0.0010823167394846678,
+            -0.037422213703393936,
+            0.012344993650913239,
+            0.02216522768139839,
+            -0.002635121811181307,
+            0.0016089539276435971,
+            -0.006145390681922436,
+            0.022103270515799522,
+            -0.013971373438835144,
+            -0.00650551775470376,
+            -0.00046443779137916863,
+            0.00797700323164463,
+            -0.01795213110744953,
+            0.004778457339853048,
+            0.0755879208445549,
+            -0.007775642443448305,
+            0.006122156511992216,
+            0.009270362555980682,
+            -0.015148562379181385,
+            -0.01637221872806549,
+            -0.004561606794595718,
+            -0.020260039716959,
+            0.008844406343996525,
+            -0.0018819533288478851,
+            -0.0110981035977602,
+            0.012468908913433552,
+            0.0053050946444272995,
+            0.002031038049608469,
+            -0.008379725739359856,
+            0.010811550542712212,
+            0.009525936096906662,
+            0.010432061739265919,
+            0.02360573597252369,
+            0.011477591469883919,
+            -0.00858883187174797,
+            -0.011795123107731342,
+            0.01129171997308731,
+            0.0028577810153365135,
+            0.016697494313120842,
+            0.007907302118837833,
+            0.0010648912284523249,
+            0.019501063972711563,
+            -0.002309845993295312,
+            0.02180897258222103,
+            -2.2689444449497387e-05,
+            -0.009301341138780117,
+            -0.004236331209540367,
+            -0.015349923633038998,
+            0.024271776899695396,
+            -0.0020155487582087517,
+            0.0015450604259967804,
+            -0.020213572308421135,
+            0.0013979118084535003,
+            0.013243375346064568,
+            0.011957760900259018,
+            -0.0020407189149409533,
+            -0.0051347119733691216,
+            0.0007154133054427803,
+            -0.016356728971004486,
+            0.032806396484375,
+            0.007578153628855944,
+            0.0007681738352403045,
+            0.0008838597568683326,
+            0.0009288755827583373,
+            -0.0032972905319184065,
+            0.0024608669336885214,
+            -0.004766840487718582,
+            0.009549170732498169,
+            -0.010943209752440453,
+            -0.010796061716973782,
+            0.010075807571411133,
+            -0.02402394637465477,
+            0.003566417610272765,
+            -0.00558777479454875,
+            -0.008062195032835007,
+            0.0076207490637898445,
+            0.00015537731815129519,
+            -0.007392281666398048,
+            -0.0030939930584281683,
+            0.0023408245760947466,
+            0.013429246842861176,
+            0.004406713880598545,
+            0.010850274004042149,
+            0.012956822291016579,
+            0.005359307397156954,
+            -0.006629432085901499,
+            0.015512561425566673,
+            -0.006946963258087635,
+            -0.004228586331009865,
+            -0.0012468908680602908,
+            -0.019996721297502518,
+            -0.0009419447160325944,
+            -0.004708755761384964,
+            0.0015092412941157818,
+            0.0001705036120256409,
+            -8.749050175538287e-05,
+            -0.0018490385264158249,
+            -0.011988739483058453,
+            -0.001831613015383482,
+            -0.011663463898003101,
+            -0.014498010277748108,
+            0.010253935120999813,
+            -0.0041201612912118435,
+            -0.022939693182706833,
+            -0.015349923633038998,
+            -0.020662762224674225,
+            0.02145271934568882,
+            -0.004387352149933577,
+            -0.005800753366202116,
+            0.006733985152095556,
+            -0.010215211659669876,
+            -0.015706177800893784,
+            -0.008069939911365509,
+            -0.0205853171646595,
+            0.010703125037252903,
+            0.003214035416021943,
+            0.0006791101768612862,
+            0.00882891658693552,
+            0.014397329650819302,
+            -0.01589204929769039,
+            0.0111755495890975,
+            0.012182355858385563,
+            0.011361422017216682,
+            -0.006854027509689331,
+            0.0008911203476600349,
+            -0.014103032648563385,
+            0.02596011385321617,
+            0.021623101085424423,
+            0.005169562995433807,
+            -0.04132552444934845,
+            -0.005862710531800985,
+            -0.013344055972993374,
+            -0.010524998418986797,
+            -0.012120398692786694,
+            0.011771888472139835,
+            -0.0038820127956569195,
+            0.0013117524795234203,
+            0.006288666743785143,
+            0.0015063370810821652,
+            0.004367990419268608,
+            -0.0022382077295333147,
+            -0.007709812838584185,
+            -0.03212486580014229,
+            -0.02817508764564991,
+            -0.00863530021160841,
+            0.020461400970816612,
+            0.002838419284671545,
+            -0.0009453330421820283,
+            -0.0036380558740347624,
+            0.0027048238553106785,
+            -0.005882072262465954,
+            -0.005018542055040598,
+            -0.014753584749996662,
+            -0.012739972211420536,
+            0.0101300198584795,
+            0.00802347157150507,
+            0.014195968396961689,
+            -0.011950016021728516,
+            0.0028190575540065765,
+            0.018788553774356842,
+            0.007690451107919216,
+            -0.023357905447483063,
+            0.01135367713868618,
+            -0.010912231169641018,
+            -0.009270362555980682,
+            -0.009433000348508358,
+            -0.009471723809838295,
+            0.0016980175860226154,
+            -0.008317768573760986,
+            0.003945906180888414,
+            -0.03615209087729454,
+            -0.011330443434417248,
+            -0.0013901671627536416,
+            -0.010346870869398117,
+            0.0009864765452221036,
+            0.02524760365486145,
+            0.006815304048359394,
+            0.01432762760668993,
+            -0.0006733016925863922,
+            0.017502939328551292,
+            -0.0018984107300639153,
+            0.018726596608757973,
+            0.012763205915689468,
+            -0.00797700323164463,
+            0.011717676185071468,
+            0.014738094992935658,
+            -0.0033863543067127466,
+            0.015620986931025982,
+            -0.010168743319809437,
+            0.005285732913762331,
+            -0.009556914679706097,
+            -0.003214035416021943,
+            0.002495717955753207,
+            0.030901208519935608,
+            -0.003618694143369794,
+            0.00858883187174797,
+            0.01098193321377039,
+            -0.015241498127579689,
+            0.0026951429899781942,
+            0.014962689951062202,
+            0.01141563430428505,
+            0.007760153152048588,
+            0.0024434414226561785,
+            -0.024225307628512383,
+            0.004193735308945179,
+            0.005037903785705566,
+            0.0031520782504230738,
+            -0.014366351068019867,
+            0.0021859314292669296,
+            0.015675200149416924,
+            0.008550108410418034,
+            -0.025325050577521324,
+            0.016418687999248505,
+            0.0009138703462667763,
+            -0.003909118939191103,
+            0.029119934886693954,
+            -0.011268485337495804,
+            -0.005723306443542242,
+            0.009463978931307793,
+            0.0009980935137718916,
+            0.0030029932968318462,
+            0.008937342092394829,
+            0.0228312686085701,
+            -0.003988501615822315,
+            -0.032341714948415756,
+            -0.022459525614976883,
+            0.015706177800893784,
+            -0.0047823297791182995,
+            -0.022939693182706833,
+            -0.026703599840402603,
+            -0.003113354789093137,
+            -0.02228914201259613,
+            -0.0022323993034660816,
+            -0.020058678463101387,
+            -0.014056564308702946,
+            0.001268188701942563,
+            0.029987338930368423,
+            0.016201836988329887,
+            0.008224832825362682,
+            -0.011051635257899761,
+            -0.029956359416246414,
+            -0.024271776899695396,
+            0.01810702309012413,
+            -0.007055388763546944,
+            -0.005541306920349598,
+            0.00591692328453064,
+            -0.009951893240213394,
+            -0.008480406366288662,
+            0.011802867986261845,
+            0.027137301862239838,
+            -0.002747419523075223,
+            -0.029987338930368423,
+            -0.02688947319984436,
+            -0.014335372485220432,
+            -0.01840132102370262,
+            -0.017704300582408905,
+            -0.008627555333077908,
+            0.0045035220682621,
+            0.00030300996149890125,
+            0.016186347231268883,
+            0.014962689951062202,
+            -0.000480653194244951,
+            0.0028597170021384954,
+            -2.7711375878425315e-05,
+            -0.001763847190886736,
+            0.006888878531754017,
+            -0.004933350719511509,
+            0.026997897773981094,
+            0.014614180661737919,
+            -0.006594581063836813,
+            0.018711106851696968,
+            0.028314489871263504,
+            0.020430423319339752,
+            0.003157886676490307,
+            -0.010695381090044975,
+            -0.014606435783207417,
+            0.009727297350764275,
+            -0.013421501964330673,
+            0.023264970630407333,
+            0.022552460432052612,
+            -0.01480779703706503,
+            -0.014885243959724903,
+            -0.017905661836266518,
+            0.015210519544780254,
+            -0.007562664337456226,
+            -0.008224832825362682,
+            0.012368228286504745,
+            0.022335609421133995,
+            -0.007024410180747509,
+            0.002371803391724825,
+            -0.0010232636705040932,
+            -0.021731525659561157,
+            -0.006122156511992216,
+            -0.004596457816660404,
+            -0.008666278794407845,
+            0.008681767620146275,
+            0.008131897076964378,
+            -0.001928421319462359,
+            -0.006319645792245865,
+            0.005471604876220226,
+            -0.013367289677262306,
+            0.01301103550940752,
+            -0.00789568480104208,
+            0.03379771113395691,
+            -0.009138703346252441,
+            -0.0006733016925863922,
+            -0.022598929703235626,
+            -0.011446612887084484,
+            0.00845717266201973,
+            0.0017967619933187962,
+            0.01682141050696373,
+            -0.0011626676423475146,
+            -0.005688455421477556,
+            0.009913169778883457,
+            -0.0010687635513022542,
+            -0.001450188341550529,
+            -0.006892750971019268,
+            -0.010401083156466484,
+            0.0002884887217078358,
+            0.013258864171802998,
+            -0.006025348324328661,
+            0.0018054747488349676,
+            0.006095050368458033,
+            -0.0175339188426733,
+            0.029027000069618225,
+            0.023884542286396027,
+            0.017255110666155815,
+            0.014660648070275784,
+            0.0006394187803380191,
+            0.011647974140942097,
+            -0.003572226269170642,
+            0.0038820127956569195,
+            -0.020337486639618874,
+            0.021080974489450455,
+            -0.010811550542712212,
+            0.01846327818930149,
+            0.011655719019472599,
+            -0.015233753249049187,
+            -0.005022414494305849,
+            0.021406250074505806,
+            0.009967382065951824,
+            0.00889087375253439,
+            0.0017541663255542517,
+            -0.02470547705888748,
+            0.017766257748007774,
+            0.010099041275680065,
+            -0.020399443805217743,
+            0.019547531381249428,
+            -0.009471723809838295,
+            0.01233724970370531,
+            -0.0005300253978930414,
+            0.004418330732733011,
+            0.0058743273839354515,
+            0.021499186754226685,
+            0.011570527218282223,
+            -0.03884723410010338,
+            -0.013615119270980358,
+            -0.0017357728211209178,
+            0.006857899948954582,
+            -0.0036303112283349037,
+            -0.015233753249049187,
+            -0.041635312139987946,
+            0.01604694314301014,
+            0.009998360648751259,
+            -0.01643417589366436,
+            0.004987563472241163,
+            -0.002427952131256461,
+            -0.0042053526267409325,
+            -0.008170620538294315,
+            0.003186929039657116,
+            -0.004170501604676247,
+            0.02324948087334633,
+            0.0018984107300639153,
+            -0.006946963258087635,
+            0.013258864171802998,
+            0.01344473659992218,
+            -0.021948376670479774,
+            0.0009874446550384164,
+            0.0021375271026045084,
+            0.000636998622212559,
+            0.023683181032538414,
+            -0.014707116410136223,
+            -0.0017657832941040397,
+            -0.01685238815844059,
+            -0.0013069120468571782,
+            -0.021592123433947563,
+            -0.001188805908896029,
+            -0.015187285840511322,
+            -0.008511385880410671,
+            0.025030752643942833,
+            -0.011005166918039322,
+            0.010997422970831394,
+            -0.006687517277896404,
+            0.006943091284483671,
+            -0.006904367823153734,
+            -0.01936165988445282,
+            -0.0008267428493127227,
+            0.009975126944482327,
+            -0.000512115890160203,
+            0.002941036131232977,
+            8.482827252009884e-05,
+            0.002464739140123129,
+            0.008867640048265457,
+            0.006234454456716776,
+            -0.0075278133153915405,
+            0.02489134855568409,
+            -0.012073930352926254,
+            0.0016399325104430318,
+            0.011911292560398579,
+            0.0004743606550619006,
+            -0.010424317792057991,
+            0.025588368996977806,
+            -0.02139076218008995,
+            -0.00800798274576664,
+            -0.00014799568452872336,
+            -0.0016544538084417582,
+            -0.006513262167572975,
+            0.006718495860695839,
+            0.02946070022881031,
+            0.0062654330395162106,
+            0.010447551496326923,
+            -0.0281441081315279,
+            -0.011493081226944923,
+            0.012538610957562923,
+            -0.022273652255535126,
+            -0.0007739823195151985,
+            -0.0004322490422055125,
+            -0.00619960343465209,
+            0.026378324255347252,
+            -0.02022906206548214,
+            -0.01093546487390995,
+            -0.018664639443159103,
+            0.005405775271356106,
+            -0.011121337302029133,
+            0.008294534869492054,
+            -0.0009777637897059321,
+            0.006540368776768446,
+            0.016805920749902725,
+            -0.019563021138310432,
+            0.016031453385949135,
+            0.005119222681969404,
+            -0.016217326745390892,
+            0.010222956538200378,
+            -0.017580386251211166,
+            -0.0008199662552215159,
+            0.020275529474020004,
+            -0.00400399137288332,
+            0.008751469664275646,
+            -0.003363120136782527,
+            0.005208286456763744,
+            -0.006513262167572975,
+            0.0001492057926952839,
+            -0.007249005138874054,
+            0.014428308233618736,
+            -0.016805920749902725,
+            -0.017828216776251793,
+            -0.008821171708405018,
+            0.0016709112096577883,
+            0.021297825500369072,
+            0.011136826127767563,
+            -0.010044828988611698,
+            -0.011059380136430264,
+            -0.011694442480802536,
+            0.041604332625865936,
+            0.005762029904872179,
+            0.027230238541960716,
+            -0.02261441759765148,
+            0.00545998802408576,
+            0.025170156732201576,
+            -0.014242436736822128,
+            0.0012788375606760383,
+            -0.0022788674104958773,
+            0.0053050946444272995,
+            0.011562783271074295,
+            -0.01819995976984501,
+            0.0049139889888465405,
+            -0.00019216447253711522,
+            -0.00782985519617796,
+            -0.000656844291370362,
+            0.003442503046244383,
+            0.00020196630794089288,
+            0.016325751319527626,
+            -0.004027225077152252,
+            -0.016511622816324234,
+            -0.004429947584867477,
+            -0.0003216455806978047,
+            -0.027896279469132423,
+            0.013514438644051552,
+            0.016589069738984108,
+            0.01566745527088642,
+            0.027989214286208153,
+            -0.014513500034809113,
+            0.016031453385949135,
+            0.009533680975437164,
+            0.013901671394705772,
+            -0.010610189288854599,
+            0.008952830918133259,
+            -0.01595400646328926,
+            0.007446494419127703,
+            -0.03454120084643364,
+            -0.007887939922511578,
+            -0.00802347157150507,
+            0.01984182931482792,
+            0.028763681650161743,
+            0.002923610620200634,
+            0.006493900436908007,
+            0.000196883876924403,
+            0.0024202074855566025,
+            -0.01714668609201908,
+            -0.020182594656944275,
+            -0.0003337466041557491,
+            0.0029332914855331182,
+            0.0006950835813768208,
+            -0.004495777189731598,
+            0.00472037261351943,
+            -0.012391461990773678,
+            0.0026835259050130844,
+            -0.013134949840605259,
+            0.005971135571599007,
+            -0.0059982421807944775,
+            0.009603383019566536,
+            0.0019429425010457635,
+            0.020864123478531837,
+            0.02300165221095085,
+            0.004100799560546875,
+            0.0046971384435892105,
+            -0.024333734065294266,
+            0.00920066051185131,
+            -0.013847459107637405,
+            0.012329504825174809,
+            0.019408127292990685,
+            0.008248066529631615,
+            0.006571347359567881,
+            0.01233724970370531,
+            0.012918098829686642,
+            -0.0031772484071552753,
+            0.024287264794111252,
+            0.005014669615775347,
+            -0.0009070937521755695,
+            0.002052335999906063,
+            -0.014954946003854275,
+            -0.0070979841984808445,
+            -0.0015286030247807503,
+            0.0018848575418815017,
+            -0.004731989465653896,
+            0.01906736195087433,
+            0.015256987884640694,
+            -0.01050176378339529,
+            0.017874684184789658,
+            -0.03937387093901634,
+            0.017379024997353554,
+            -0.0027377386577427387,
+            -0.0060911779291927814,
+            -0.00545998802408576,
+            0.00665653869509697,
+            0.004515138920396566,
+            0.003669034456834197,
+            0.03773200139403343,
+            -0.010408828034996986,
+            0.003963331691920757,
+            -0.005847221240401268,
+            0.007121218368411064,
+            -0.020864123478531837,
+            0.008302279748022556,
+            0.005336073227226734,
+            0.02160761132836342,
+            -0.0056497324258089066,
+            -0.0014482521219179034,
+            0.02328045852482319,
+            0.008859895169734955,
+            0.03469609469175339,
+            -0.01566745527088642,
+            -0.004418330732733011,
+            0.01048627495765686,
+            0.0207402091473341,
+            -0.00580462533980608,
+            -0.008976065553724766,
+            0.007903429679572582,
+            0.0035586729645729065,
+            0.021824462339282036,
+            -0.00515407370403409,
+            -0.0006650729919783771,
+            0.0054367538541555405,
+            -0.0060911779291927814,
+            0.009735042229294777,
+            0.011252996511757374,
+            -0.009076746180653572,
+            0.007539430167526007,
+            0.021530164405703545,
+            -0.009975126944482327,
+            -0.008658533915877342,
+            -0.009649851359426975,
+            0.0010658593382686377,
+            -0.03068435750901699,
+            0.001649613375775516,
+            0.005359307397156954,
+            0.015187285840511322,
+            -0.0004603234410751611,
+            -0.009014789015054703,
+            0.021019017323851585,
+            0.017255110666155815,
+            -0.005936284549534321,
+            -0.0028132491279393435,
+            0.0023795480374246836,
+            -0.0017154429806396365,
+            -0.006160879973322153,
+            -0.0006248975405469537,
+            -0.002656419761478901,
+            0.014730350114405155,
+            -0.0015469965292140841,
+            -0.006617815233767033,
+            -0.00756653631106019,
+            0.0033302053343504667,
+            0.010834784246981144,
+            -0.005661349277943373,
+            0.013057502917945385,
+            0.016805920749902725,
+            -0.00031704717548564076,
+            0.019671445712447166,
+            0.01037010457366705,
+            -0.010447551496326923,
+            0.015938518568873405,
+            -0.007984748110175133,
+            0.006563602481037378,
+            0.0164031982421875,
+            -0.01277095079421997,
+            -0.01246116403490305,
+            0.007210282143205404,
+            0.00665653869509697,
+            0.028918573632836342,
+            0.014792308211326599,
+            -0.02109646424651146,
+            0.014908477663993835,
+            0.015768134966492653,
+            -0.013018779456615448,
+            0.003537375247105956,
+            -0.01784370467066765,
+            0.020260039716959,
+            0.019051872193813324,
+            -0.015535795129835606,
+            -0.017224133014678955,
+            -0.014947201125323772,
+            -0.005092116538435221,
+            0.006331262644380331,
+            0.0033263331279158592,
+            -0.00617636926472187,
+            -0.027989214286208153,
+            -0.0049139889888465405,
+            -0.018633661791682243,
+            0.013870692811906338,
+            -0.008674023672938347,
+            0.0007715621031820774,
+            -0.004511266481131315,
+            0.007984748110175133,
+            -0.0033882902935147285,
+            0.032465629279613495,
+            -0.0051811798475682735,
+            -0.02832997962832451,
+            0.004398969002068043,
+            0.016356728971004486,
+            0.012623801827430725,
+            -0.029243851080536842,
+            0.010463040322065353,
+            0.004805563949048519,
+            -0.012530866079032421,
+            0.005990497302263975,
+            -0.02258343994617462,
+            -0.007632365915924311,
+            -0.0009395244996994734,
+            -0.016743963584303856,
+            -0.027121812105178833,
+            -0.008433938957750797,
+            -0.006993431597948074,
+            -0.005401902832090855,
+            -0.009525936096906662,
+            -0.008720491081476212,
+            -0.011477591469883919,
+            0.02303262986242771,
+            -0.007899557240307331,
+            -0.001866463921032846,
+            -0.0014637414133176208,
+            0.00930908601731062,
+            -0.023776117712259293,
+            -0.015605497173964977,
+            0.006389347370713949,
+            -0.002594462363049388,
+            0.022815778851509094,
+            -0.012747716158628464,
+            0.005940156988799572,
+            -0.009316830895841122,
+            0.009851212613284588,
+            0.015241498127579689,
+            0.004956584889441729,
+            0.017440982162952423,
+            -0.009889936074614525,
+            0.0021026763133704662,
+            -0.008627555333077908,
+            -0.00018986526993103325,
+            -1.027075541060185e-05,
+            0.014428308233618736,
+            0.007624621503055096,
+            0.014141756109893322,
+            0.017487451434135437,
+            -0.013553161174058914,
+            0.005003052763640881,
+            -0.0003274540649726987,
+            -0.017255110666155815,
+            0.0027725896798074245,
+            -0.007984748110175133,
+            -0.0008896682411432266,
+            0.001758038648404181,
+            6.395397940650582e-05,
+            -0.02112744189798832,
+            -0.0009124181815423071,
+            0.0018809852190315723,
+            0.0010222955606877804,
+            -0.025201136246323586,
+            0.00889087375253439,
+            -0.00889087375253439,
+            -0.0014918158994987607,
+            -0.019113829359412193,
+            0.019795360043644905,
+            0.026300877332687378,
+            -0.004619691986590624,
+            0.00036206303047947586,
+            -0.013537672348320484,
+            -0.02049238048493862,
+            0.0001489637652412057,
+            -0.011314953677356243,
+            -0.011345932260155678,
+            -0.0022169100120663643,
+            -0.003214035416021943,
+            0.008805682882666588,
+            0.0004189379105810076,
+            -0.009433000348508358,
+            -0.004956584889441729,
+            -0.011764144524931908,
+            -0.017549408599734306,
+            -0.010401083156466484,
+            0.01933068037033081,
+            -0.0018016024259850383,
+            0.0020794421434402466,
+            -0.0032508226577192545,
+            0.0005271211848594248,
+            0.011748654767870903,
+            0.013475715182721615,
+            0.013344055972993374,
+            -0.023915521800518036,
+            -0.017890173941850662,
+            -0.013320821337401867,
+            0.0029487807769328356,
+            -0.007187047973275185,
+            0.004844287410378456,
+            0.026362836360931396,
+            -0.0009695350890979171,
+            -0.01091997604817152,
+            0.002327271504327655,
+            0.019051872193813324,
+            -0.031272951513528824,
+            -0.013390523381531239,
+            -0.025077221915125847,
+            0.02225816436111927,
+            -0.013351799920201302,
+            0.011516314931213856,
+            0.010191977955400944,
+            -0.025944624096155167,
+            0.006768836174160242,
+            -0.007686578668653965,
+            0.026037558913230896,
+            0.01634124107658863,
+            0.0005934348446317017,
+            -0.00944074522703886,
+            0.013622863218188286,
+            -0.0008722427301108837,
+            -0.013576395809650421,
+            0.002201420720666647,
+            0.014645159244537354,
+            -0.014691627584397793,
+            -0.009510447271168232,
+            -0.004507394507527351,
+            0.004313777666538954,
+            0.012763205915689468,
+            -0.0030339721124619246,
+            -0.004031097516417503,
+            -0.011624740436673164,
+            0.013344055972993374,
+            -0.003930416889488697,
+            0.012383717112243176,
+            0.007380664348602295,
+            -0.014056564308702946,
+            -0.019532041624188423,
+            0.033333033323287964,
+            0.00400399137288332,
+            0.00338054564781487,
+            -0.010997422970831394,
+            0.02740062028169632,
+            -0.009897680021822453,
+            0.010571465827524662,
+            -0.006408709101378918,
+            0.0031985461246222258,
+            0.012321759946644306,
+            -0.012104909867048264,
+            -0.01981084980070591,
+            0.021468207240104675,
+            -0.00364192808046937,
+            -0.0010193913476541638,
+            0.008310024626553059,
+            -0.020182594656944275,
+            -0.00663330452516675,
+            0.002180122770369053,
+            0.009541425853967667,
+            -0.011818356812000275,
+            0.006369986105710268,
+            0.017905661836266518,
+            0.003912991378456354,
+            -0.0007672057254239917,
+            0.019996721297502518,
+            -0.01672847382724285,
+            0.011717676185071468,
+            -0.010850274004042149,
+            0.0016825281782075763,
+            0.0006065039779059589,
+            -0.0048597767017781734,
+            0.0058743273839354515,
+            -0.014536733739078045,
+            0.011988739483058453,
+            -6.244135147426277e-05,
+            -0.013979118317365646,
+            0.007601387333124876,
+            -0.013359544798731804,
+            -0.016836898401379585,
+            0.041542377322912216,
+            -0.005301222205162048,
+            -0.0038665232714265585,
+            -0.0056962003000080585,
+            0.01350669376552105,
+            0.02823704481124878,
+            -0.005336073227226734,
+            -0.01153180468827486,
+            -0.010780571959912777,
+            -0.016573579981923103,
+            0.015907539054751396,
+            -0.01810702309012413,
+            0.002495717955753207,
+            -0.010532742366194725,
+            -0.007903429679572582,
+            0.004302160814404488,
+            -0.003516077296808362,
+            0.014722606167197227,
+            0.00950270239263773,
+            0.023357905447483063,
+            0.00797700323164463,
+            -0.00027348342700861394,
+            0.010393339209258556,
+            0.011663463898003101,
+            0.003243078012019396,
+            -0.010509508661925793,
+            -0.00011949774489039555,
+            -0.02894955314695835,
+            -0.008449427783489227,
+            -0.004863648675382137,
+            -0.001485039247199893,
+            -0.00428667152300477,
+            -0.01239920686930418,
+            0.006617815233767033,
+            -0.0022052929271012545,
+            0.013909416273236275,
+            -0.009254872798919678,
+            -0.017766257748007774,
+            0.03389064967632294,
+            0.004495777189731598,
+            0.006466794293373823,
+            -0.008286789990961552,
+            -0.03705047070980072,
+            -0.0032566310837864876,
+            -0.008116407319903374,
+            -0.013529927469789982,
+            0.004538373090326786,
+            0.01283290795981884,
+            -0.014831030741333961,
+            -0.028097640722990036,
+            0.0019390701781958342,
+            -0.006683644838631153,
+            0.02978597767651081,
+            -0.007798876613378525,
+            0.00419760774821043,
+            0.0037251831963658333,
+            0.0011558911064639688,
+            0.01239920686930418,
+            -0.00845717266201973,
+            -0.0022536972537636757,
+            0.016387708485126495,
+            -0.010858018882572651,
+            0.011051635257899761,
+            -0.010966443456709385,
+            -0.02554190158843994,
+            -0.01456771232187748,
+            0.021344292908906937,
+            -0.0004823473282158375,
+            -0.00024903935263864696,
+            0.014823286794126034,
+            0.02354377694427967,
+            0.011260741390287876,
+            0.00969631876796484,
+            0.006408709101378918,
+            -0.001035848748870194,
+            0.005975008010864258,
+            0.006095050368458033,
+            -0.0024163350462913513,
+            -0.013398268260061741,
+            0.017564896494150162,
+            0.009773765690624714,
+            -0.01810702309012413,
+            0.0008514289511367679,
+            0.0036612898111343384,
+            0.0168833676725626,
+            0.006935346405953169,
+            0.009316830895841122,
+            -0.0143121387809515,
+            -0.023760627955198288,
+            -0.00014932679187040776,
+            0.002873270306736231,
+            0.015721667557954788,
+            0.010579210706055164,
+            -0.014319883659482002,
+            0.0005963391158729792,
+            0.008712747134268284,
+            0.020213572308421135,
+            -0.024209819734096527,
+            0.019702425226569176,
+            -0.004673904739320278,
+            -0.0021084847394376993,
+            0.007946024648845196,
+            0.011632485315203667,
+            -0.011872569099068642,
+            -0.00650551775470376,
+            0.006249943748116493,
+            0.003157886676490307,
+            0.03172214329242706,
+            -0.0017260919557884336,
+            0.009665340185165405,
+            -0.010997422970831394,
+            -0.0002405686245765537,
+            0.0006728176376782358,
+            -0.0006820144481025636,
+            -0.002884887158870697,
+            -0.011020656675100327,
+            0.0018974426202476025,
+            -0.020306508988142014,
+            0.0024473138619214296,
+            0.00463130883872509,
+            -0.005684582982212305,
+            -0.0032972905319184065,
+            -0.019222255796194077,
+            -0.0022672503255307674,
+            0.003293418325483799,
+            0.014335372485220432,
+            -0.031691163778305054,
+            0.004972074180841446,
+            0.010672146454453468,
+            -0.002309845993295312,
+            -0.010416572913527489,
+            0.011423379182815552,
+            -0.0026060794480144978,
+            0.011988739483058453,
+            -0.0038142469711601734,
+            -0.0004905760288238525,
+            0.025681305676698685,
+            -0.005308967083692551,
+            0.014707116410136223,
+            -0.01026167906820774,
+            -0.002993312431499362,
+            -0.028252532705664635,
+            0.005235392600297928,
+            -0.007403898518532515,
+            -0.0012081675231456757,
+            -0.004402841441333294,
+            -0.01130720879882574,
+            -0.0026990154292434454,
+            0.002122037811204791,
+            0.008000237867236137,
+            0.012910354882478714,
+            -0.0007725302129983902,
+            -0.010075807571411133,
+            0.004832670092582703,
+            0.0075355577282607555,
+            -0.015520306304097176,
+            -0.0070437719114124775,
+            -0.020306508988142014,
+            0.029662061482667923,
+            -0.013731288723647594,
+            -0.0011123273288831115,
+            -0.020864123478531837,
+            0.010160998441278934,
+            -0.0063119009137153625,
+            -0.002782270545139909,
+            0.004708755761384964,
+            -0.015109838917851448,
+            0.01344473659992218,
+            0.008751469664275646,
+            0.01450575515627861,
+            -0.02151467651128769,
+            -0.01762685552239418,
+            -0.0044725434854626656,
+            -0.002735802438110113,
+            -0.010648912750184536,
+            -0.0005334137240424752,
+            0.002013612538576126,
+            0.021251358091831207,
+            -0.00032503384863957763,
+            0.0042053526267409325,
+            -0.003214035416021943,
+            0.017224133014678955,
+            -0.010137764737010002,
+            -0.013057502917945385,
+            0.0024260159116238356,
+            0.005994369741529226,
+            0.016186347231268883,
+            0.03158273920416832,
+            -0.020693741738796234,
+            -0.00272999401204288,
+            0.0032740565948188305,
+            0.010656657628715038,
+            -0.004085310269147158,
+            6.304640555754304e-05,
+            -0.027803342789411545,
+            -0.04464024305343628,
+            -0.002422143705189228,
+            0.013057502917945385,
+            -0.015636475756764412,
+            0.012670270167291164,
+            0.008743725717067719,
+            0.010021595284342766,
+            -0.005196669604629278,
+            0.02620794251561165,
+            -0.004793947096914053,
+            0.014304393902420998,
+            0.011733165942132473,
+            0.011911292560398579,
+            0.0009874446550384164,
+            0.0007459079497493804,
+            -0.018354853615164757,
+            -0.008255811408162117,
+            -0.0015954007394611835,
+            0.01765783317387104,
+            -0.027478067204356194,
+            0.008526874706149101,
+            -0.011516314931213856,
+            0.013839714229106903,
+            0.0003666614356916398,
+            0.005661349277943373,
+            -0.01771979033946991,
+            0.012445674277842045,
+            -0.0009337160154245794,
+            -0.025325050577521324,
+            0.004805563949048519,
+            0.011276230216026306,
+            -0.010362359695136547,
+            0.006877261213958263,
+            0.019950253888964653,
+            0.0023079097736626863,
+            0.008495896123349667,
+            -0.014126266352832317,
+            0.010703125037252903,
+            0.009649851359426975,
+            -0.021080974489450455,
+            -0.01233724970370531,
+            -0.007059261202812195,
+            -0.006335135083645582,
+            -0.0035470561124384403,
+            -0.00013105422840453684,
+            0.010176488198339939,
+            -0.016000475734472275,
+            0.010099041275680065,
+            -0.006559730041772127,
+            0.006400964688509703,
+            0.00944074522703886,
+            0.001115231541916728,
+            -0.013576395809650421,
+            0.012716737575829029,
+            -0.0218709297478199,
+            0.008426194079220295,
+            -0.04287445917725563,
+            -0.005847221240401268,
+            -0.015078860335052013,
+            0.005204414017498493,
+            -0.003798757679760456,
+            -0.004453181754797697,
+            -0.006203475873917341,
+            -0.005061137955635786,
+            0.00920066051185131,
+            0.021747015416622162,
+            0.012135888449847698,
+            -0.016620049253106117,
+            -0.007551047019660473,
+            0.005425137002021074,
+            0.0004854935978073627,
+            -0.013336311094462872,
+            0.009316830895841122,
+            -0.01576039008796215,
+            0.0009259713697247207,
+            0.006339007057249546,
+            -0.01072635967284441,
+            0.013452480547130108,
+            0.004731989465653896,
+            -0.003401843598112464,
+            0.013390523381531239,
+            -0.0050417762249708176,
+            -0.0057891360484063625,
+            -0.007272239308804274,
+            -0.013739033602178097,
+            -0.003818119177594781,
+            -0.003072695340961218,
+            0.012747716158628464,
+            0.001529571134597063,
+            -0.0003243077953811735,
+            -0.00782985519617796,
+            0.009154192171990871,
+            0.002220782218500972,
+            -0.0013146566925570369,
+            0.005773646757006645,
+            -0.02300165221095085,
+            0.0001287550403503701,
+            0.004755223635584116,
+            -0.0009487213101238012,
+            -0.006246071308851242,
+            -0.012151377275586128,
+            -0.011903548613190651,
+            -0.038165703415870667,
+            0.009363298304378986,
+            0.0019197085639461875,
+            -0.0076207490637898445,
+            0.0064822835847735405,
+            0.017255110666155815,
+            0.01456771232187748,
+            0.016743963584303856,
+            -0.020693741738796234,
+            -0.02303262986242771,
+            0.005556796211749315,
+            -0.009402021765708923,
+            0.02261441759765148,
+            0.004046586807817221,
+            -0.005847221240401268,
+            0.0013243375578895211,
+            -0.013258864171802998,
+            0.008929597213864326,
+            0.005723306443542242,
+            -0.001592496526427567,
+            0.005425137002021074,
+            -0.013692565262317657,
+            -0.010904486291110516,
+            0.01048627495765686,
+            0.032496608793735504,
+            -0.0004261985304765403,
+            0.0024105266202241182,
+            0.023295948281884193,
+            0.015466093085706234,
+            0.016356728971004486,
+            -0.021111954003572464,
+            0.011167804710566998,
+            0.010060317814350128,
+            -0.0018645278178155422,
+            -0.014892988838255405,
+            0.00513083953410387,
+            -0.012716737575829029,
+            -0.010664401575922966,
+            0.014993669465184212,
+            0.007853088900446892,
+            0.017735280096530914,
+            -0.02369867078959942,
+            0.00789568480104208,
+            0.00042765066609717906,
+            0.00013710475468542427,
+            0.011524059809744358,
+            0.012956822291016579,
+            0.03370477631688118,
+            0.027152791619300842,
+            0.020662762224674225,
+            -0.009285852313041687,
+            0.014366351068019867,
+            -0.009192915633320808,
+            -0.011090358719229698,
+            -0.019470084458589554,
+            -0.015249243006110191,
+            -0.011136826127767563,
+            0.017998598515987396,
+            0.008790193125605583,
+            -0.009355553425848484,
+            -0.005947901867330074,
+            0.01586107164621353,
+            -0.003909118939191103,
+            -0.004635181277990341,
+            0.01646515540778637,
+            -0.026269899681210518,
+            0.016836898401379585,
+            -0.000745423894841224,
+            -0.014668392948806286,
+            -0.014691627584397793,
+            -0.004182118456810713,
+            -0.012670270167291164,
+            0.0003964299685321748,
+            -0.020817656069993973,
+            0.013181418180465698,
+            0.024597052484750748,
+            -0.0030920570716261864,
+            -0.004100799560546875,
+            -0.011299464851617813,
+            -0.0005111477803438902,
+            0.018385831266641617,
+            0.0034773540683090687,
+            0.009192915633320808,
+            -0.0023872926831245422,
+            0.013924905098974705,
+            0.00524313747882843,
+            0.0027958236169070005,
+            -0.0007565568666905165,
+            -0.016263794153928757,
+            0.012197845615446568,
+            0.0026835259050130844,
+            0.012135888449847698,
+            -0.022242674604058266,
+            0.007938280701637268,
+            -0.011764144524931908,
+            -0.0023698671720921993,
+            -0.02228914201259613,
+            0.0189744271337986,
+            -0.01610890030860901,
+            0.001046497724018991,
+            0.003841353114694357,
+            0.0006224773242138326,
+            0.009580149315297604,
+            0.01610890030860901,
+            -0.023388884961605072,
+            -0.0003460896550677717,
+            -0.014908477663993835,
+            -0.021917399019002914,
+            -0.01607792265713215,
+            0.004728117026388645,
+            -0.019501063972711563,
+            0.005444498732686043,
+            -0.0025557391345500946,
+            -0.015125328674912453,
+            0.018478767946362495,
+            0.014691627584397793,
+            -0.004379607271403074,
+            0.002412462839856744,
+            0.005425137002021074,
+            -0.0014405074762180448,
+            0.01971791312098503,
+            0.00013274837692733854,
+            -0.0010048701660707593,
+            -0.009425255469977856,
+            -0.0168833676725626,
+            -0.0065752193331718445,
+            -0.035005878657102585,
+            0.0260530486702919,
+            -0.004670032300055027,
+            -0.016511622816324234,
+            0.014319883659482002,
+            0.009115469641983509,
+            -0.014629669487476349,
+            -0.007059261202812195,
+            0.016294771805405617,
+            -0.024287264794111252,
+            0.02244403585791588,
+            -0.009611127898097038,
+            -0.008774704299867153,
+            0.00784921646118164,
+            0.05154848098754883,
+            0.009146448224782944,
+            0.003492843359708786,
+            0.011036145500838757,
+            0.005839476361870766,
+            -0.0013165927957743406,
+            -0.023683181032538414,
+            0.005568413063883781,
+            -0.007190920412540436,
+            -0.003601268632337451,
+            0.002536377403885126,
+            -0.004801691509783268,
+            0.031149037182331085,
+            -0.004379607271403074,
+            -0.005289605353027582,
+            0.004484160337597132,
+            -0.02196386642754078,
+            -0.0001332324172835797,
+            0.023203013464808464,
+            -2.3128106477088295e-05,
+            -0.023590246215462685,
+            -0.006571347359567881,
+            -0.015071115456521511,
+            -0.013770012184977531,
+            -0.018989915028214455,
+            0.005231520161032677,
+            -0.012755461037158966,
+            -0.0033940989524126053,
+            0.0016660707769915462,
+            -0.005940156988799572,
+            -0.002582845278084278,
+            0.009603383019566536,
+            0.05362405255436897,
+            0.014103032648563385,
+            -0.006335135083645582,
+            -0.005854965653270483,
+            0.009192915633320808,
+            -0.010524998418986797,
+            -0.019795360043644905,
+            -0.009820234030485153,
+            -0.029507169499993324,
+            -0.012747716158628464,
+            -0.0143121387809515,
+            -0.010191977955400944,
+            -0.010277168825268745,
+            0.00963436160236597,
+            -0.018122512847185135,
+            -0.007628493942320347,
+            -0.020337486639618874,
+            0.002941036131232977,
+            -0.011268485337495804,
+            -0.014714861288666725,
+            -0.008054450154304504,
+            -0.04807887226343155,
+            0.005796880926936865,
+            0.019593998789787292,
+            0.01263929158449173,
+            0.002515079453587532,
+            0.004383479710668325,
+            -0.01993476413190365,
+            -0.030792783945798874,
+            0.013839714229106903,
+            0.011283975094556808,
+            0.011733165942132473,
+            0.03283737599849701,
+            -0.011709931306540966,
+            -0.00813964195549488,
+            -0.004693266469985247,
+            -0.0017251238459721208,
+            0.012066186405718327,
+            0.00969631876796484,
+            -0.004917861428111792,
+            -0.012375972233712673,
+            0.016217326745390892,
+            -0.01981084980070591,
+            -0.023683181032538414,
+            -0.01672847382724285,
+            0.006157007534056902,
+            0.00813964195549488,
+            -0.008054450154304504,
+            0.002973950933665037,
+            0.0056962003000080585,
+            -0.007159941829741001,
+            -0.00022423222253564745,
+            0.004890755284577608,
+            -0.023048119619488716,
+            0.0008490087930113077,
+            -0.002257569460198283,
+            0.0061376458033919334,
+            0.021266845986247063,
+            -0.010168743319809437,
+            -0.0014075926737859845,
+            -0.0053902859799563885,
+            -0.0028984404634684324,
+            -0.006764963734894991,
+            0.02620794251561165,
+            0.009154192171990871,
+            -0.009882191196084023,
+            -0.01011453103274107,
+            -0.01129171997308731,
+            -0.0011278166202828288,
+            -0.017255110666155815,
+            -0.016186347231268883,
+            0.0015566773945465684,
+            -0.016712984070181847,
+            0.013971373438835144,
+            -0.00876695942133665,
+            0.00400399137288332,
+            -0.014110777527093887,
+            0.02019808255136013,
+            0.01929970271885395,
+            -0.009711808525025845,
+            0.023776117712259293,
+            -0.007709812838584185,
+            -0.038382552564144135,
+            -0.00800798274576664,
+            0.01843230053782463,
+            -0.006118284538388252,
+            0.010037084110081196,
+            -0.011314953677356243,
+            0.008209343999624252,
+            0.00015319912927225232,
+            0.0018122512847185135,
+            -0.0003112386621069163,
+            0.027555514127016068,
+            0.015535795129835606,
+            -0.005560668651014566,
+            -0.005494839046150446,
+            0.004255692940205336,
+            0.009541425853967667,
+            0.017580386251211166,
+            0.012151377275586128,
+            -0.021019017323851585,
+            -0.014257925562560558,
+            -0.016263794153928757,
+            0.011012911796569824,
+            0.015125328674912453,
+            -0.0040891822427511215,
+            0.01691434532403946,
+            0.014126266352832317,
+            -0.019098341464996338,
+            0.0019884423818439245,
+            -0.008193854242563248,
+            -0.00158184755127877,
+            -0.03029712475836277,
+            0.011043890379369259,
+            0.0074387495405972,
+            -0.0072218989953398705,
+            0.036616768687963486,
+            0.009858956560492516,
+            0.037577107548713684,
+            0.018680129200220108,
+            0.0008974128868430853,
+            0.009045767597854137,
+            -0.018370341509580612,
+            0.013065247796475887,
+            0.01500141341239214,
+            -0.012724482454359531,
+            -0.005734923295676708,
+            0.0013446672819554806,
+            0.0007551047019660473,
+            0.012631546705961227,
+            0.02004319056868553,
+            -0.01795213110744953,
+            0.009773765690624714,
+            -0.015032391995191574,
+            -0.005250881891697645,
+            0.006555858068168163,
+            -0.013932649977505207,
+            -0.008495896123349667,
+            -0.00963436160236597,
+            0.019408127292990685,
+            -0.012600568123161793,
+            0.016573579981923103,
+            -0.009595638141036034,
+            0.015520306304097176,
+            -0.029724018648266792,
+            -0.003669034456834197,
+            -0.006679772399365902,
+            -0.019593998789787292,
+            -0.012887120246887207,
+            0.009820234030485153,
+            0.010633422993123531,
+            -0.009301341138780117,
+            -0.011562783271074295,
+            0.020368466153740883,
+            -0.004185990896075964,
+            0.013529927469789982,
+            0.021189400926232338,
+            0.00737291993573308,
+            0.024132372811436653,
+            0.0005605200421996415,
+            0.014164989814162254,
+            0.005463860463351011,
+            0.010362359695136547,
+            -0.004677776712924242,
+            0.005161818582564592,
+            0.00932457484304905,
+            0.014203713275492191,
+            0.017766257748007774,
+            0.013196907006204128,
+            -0.00598275288939476,
+            0.007419387809932232,
+            0.02016710489988327,
+            0.0016254113288596272,
+            -0.030312614515423775,
+            -0.009719553403556347,
+            0.0001736498816171661,
+            0.00924712885171175,
+            0.009185170754790306,
+            3.817877222900279e-05,
+            -0.01093546487390995,
+            0.011144571006298065,
+            0.00969631876796484,
+            0.002286612056195736,
+            0.006280922330915928,
+            -0.01456771232187748,
+            -0.020957060158252716,
+            0.006106667220592499,
+            0.01798310875892639,
+            0.013917161151766777,
+            -0.014722606167197227,
+            0.022784801200032234,
+            0.006145390681922436,
+            -0.021406250074505806,
+            -0.01840132102370262,
+            -0.0009559819591231644,
+            0.0004542729293461889,
+            -0.002265314105898142,
+            0.03720536455512047,
+            -0.0011539548868313432,
+            -0.005955646280199289,
+            -0.0054754773154854774,
+            0.02775687538087368,
+            0.02049238048493862,
+            0.009177426807582378
+          ],
+          "embedding": [
+            0.0008215984936325879,
+            -0.019349418284688353,
+            -0.02322660633029795,
+            0.009747820845194852,
+            0.007353771220903641,
+            0.015252908783199839,
+            -0.012176175829980202,
+            0.051898051094966605,
+            -0.0068130353886146165,
+            0.008189212816890595,
+            0.008102287831931905,
+            0.01682801136011614,
+            0.02005557996049711,
+            -0.020779879580497183,
+            0.002332165268329781,
+            0.02337485065616377,
+            0.006074992895540845,
+            0.02576575146535689,
+            0.0007517286429767504,
+            -0.03763909942314134,
+            0.0011204729720314833,
+            0.00041212613589632806,
+            -0.06089176800314795,
+            0.012852035873395388,
+            -0.008901127092058543,
+            0.01702347055484216,
+            0.005803446395466281,
+            0.017889570275475296,
+            -0.05375047382413552,
+            0.021623778774252724,
+            0.014784317465883424,
+            -0.0018881880857885708,
+            0.0004735328554290247,
+            0.012393802850407656,
+            -0.019234442968055952,
+            -0.019566283611638122,
+            0.04407789373296535,
+            -0.0053565873209102995,
+            -0.026240866282480922,
+            0.048107902270426675,
+            -0.0054681314939387235,
+            -0.0009441397948041965,
+            -0.015205079626883548,
+            0.006885981841893229,
+            0.01403724256866746,
+            -0.026086548246792754,
+            0.0034778946051893067,
+            0.00044587533948775473,
+            0.032851328435891634,
+            -0.03882097333255334,
+            -0.011126344604254894,
+            -0.01648943336384904,
+            0.035810456555802166,
+            -0.019384589526165754,
+            -0.019430702768170285,
+            0.014297440272239852,
+            -0.007883453204663151,
+            0.050996900793086844,
+            0.06007149434153715,
+            0.03406832053483379,
+            0.025193439934671852,
+            0.00027785129663604897,
+            -0.008014391056168083,
+            -0.018806730908490114,
+            0.027193788903098558,
+            -0.006195006670328595,
+            -0.020466000344352826,
+            0.0068939884219589185,
+            0.017252276167086602,
+            -0.045073444648704136,
+            -0.049242269917908754,
+            0.023021140866008313,
+            -0.015676872472071106,
+            -0.024497517908898602,
+            0.005873633718973869,
+            -0.022828585938203683,
+            0.011748341004940436,
+            0.04624205727927451,
+            -0.015442165499674641,
+            0.018565551071491767,
+            0.015246505395507096,
+            0.004119269796224144,
+            0.004862735250819252,
+            -0.000145323321960046,
+            0.05155838952887077,
+            0.0027052423345323936,
+            -0.016218063670586664,
+            -0.07462470920944148,
+            0.043695974746295135,
+            -0.027608273720377708,
+            -0.03860912774695494,
+            0.002823110565150435,
+            0.026471469849508766,
+            -0.0026063577189074103,
+            -0.029230652014151735,
+            0.05684164470477843,
+            0.00022182868300572677,
+            0.015590845552956364,
+            -0.024939617549529242,
+            0.01035803286203917,
+            -0.010225298012384327,
+            0.019571847648759157,
+            0.01841613041441054,
+            0.0156981470538354,
+            0.021764297694793343,
+            -0.018831230007033416,
+            -0.014419529638789993,
+            -0.0029965475899652273,
+            0.028881185012095884,
+            0.09163526460163937,
+            0.022644302207811818,
+            -0.017933909936330133,
+            -0.04663928709016869,
+            0.004962951444105128,
+            -0.041333041174526026,
+            -0.027545628007617527,
+            -0.037142394579894954,
+            -0.013358143162788402,
+            0.02719686432396677,
+            -0.0034971800937504205,
+            0.04413635817371732,
+            0.04903857110328917,
+            0.03186067155630788,
+            0.007831051050269414,
+            -0.025257538623313574,
+            0.023517513773109414,
+            -0.0016527403209505964,
+            -0.0016101860707429347,
+            0.014448515185903624,
+            0.009905018312477684,
+            0.0024114582235163955,
+            0.00172043851384822,
+            -0.03818316221102173,
+            -0.02304537008110578,
+            0.01024969233570089,
+            -0.01901116356327339,
+            0.019791121460848986,
+            0.07985067340821365,
+            -0.051903191272236665,
+            -0.025924646976556942,
+            0.014766511532849252,
+            -0.002073086076894999,
+            0.030959269926800852,
+            0.006394417986617089,
+            -0.028177586693713442,
+            0.010725285336674526,
+            0.0009614052276924192,
+            0.07500253788978564,
+            -0.00036794182149960157,
+            0.004058458332826519,
+            -0.013218269166951234,
+            0.027613539268683245,
+            -0.012995170442480582,
+            0.023388707018488407,
+            0.026389423313105165,
+            -0.003068639956061266,
+            -0.02661967649632026,
+            0.026145274570635913,
+            -0.03300591322174331,
+            -0.004556072066733024,
+            0.020045722084112896,
+            0.0033437343914784865,
+            -0.010224220905483085,
+            -0.0025025498684079703,
+            0.03460103245395426,
+            0.04514362357278504,
+            0.03797031988058962,
+            -0.004586662475576956,
+            -0.06351723722596923,
+            0.06037983484369874,
+            0.019415753124363902,
+            0.028850618647994476,
+            -0.0034810237045849006,
+            0.02957938154808314,
+            0.02642652523231673,
+            0.013187653724133275,
+            -0.007856320804109569,
+            -0.007065882659482447,
+            -0.016672784625197085,
+            0.029020161154209707,
+            -0.013243058921970134,
+            -0.014900932834241732,
+            0.006001769901896033,
+            0.008648456186944271,
+            0.03484763413464497,
+            0.012279368984306564,
+            0.02379643508687133,
+            0.04933044959399941,
+            -0.021047723935647383,
+            -0.007264823701706426,
+            0.0007058774795716792,
+            -0.010512071832195085,
+            0.027489671668749157,
+            0.009731157570361245,
+            -0.03195919640758579,
+            -0.0033626031972691915,
+            -0.04282419484651207,
+            0.01873977910485268,
+            -0.014398285316695653,
+            0.012854631447494207,
+            -0.0017372959978092432,
+            -0.037331867825735776,
+            0.001088209292238142,
+            -0.012345684323522151,
+            0.037096063326414734,
+            0.005050219690024652,
+            0.0008906051515353004,
+            -0.003697709449630797,
+            0.006548479142737107,
+            -0.010727663712299397,
+            -0.007486226350610755,
+            -0.02704618481069485,
+            -0.019752552016312926,
+            -0.014632581296303486,
+            -0.0037501138119724717,
+            0.014588411795703245,
+            -0.007119137071719376,
+            0.016440258608998535,
+            -0.03686645352111013,
+            -0.023546402448713415,
+            0.005628703432152666,
+            0.024988145751166143,
+            0.011971199595808573,
+            -0.010868778329190568,
+            -0.03270046433624425,
+            -0.021950559285357267,
+            -0.011889914022657738,
+            -0.005123971523701009,
+            0.02619713302280345,
+            -0.021990810305904974,
+            0.01165257474485436,
+            -0.009117901804315863,
+            -0.007492082755249399,
+            0.010725902070291232,
+            0.01862811996181951,
+            -0.026694281868322685,
+            -0.01623776415010199,
+            -0.009403997538684658,
+            -0.02207329027097839,
+            0.025939540497196416,
+            0.014178803271471762,
+            -0.001141821318145744,
+            -0.006281187017652108,
+            -0.035993365198554964,
+            0.0026192776111449633,
+            -0.015254976771874396,
+            -0.0116872453009606,
+            -0.015780016491993135,
+            0.021421328283124968,
+            0.02889432250124136,
+            -0.01871009416296692,
+            -0.026540472640514536,
+            0.015703370682110186,
+            -0.0002849235245164438,
+            -0.02594930889995505,
+            0.04491828744809679,
+            0.044063522197410734,
+            -0.037441864491547405,
+            -0.007005339187980476,
+            -0.030381794393931814,
+            0.06901083845539985,
+            -0.017604082400603037,
+            -0.04060591381445593,
+            0.06044850935418443,
+            -0.01756490171206357,
+            0.025539717592705295,
+            0.03452018789549939,
+            0.0016217577522523927,
+            0.00018825054264059277,
+            -0.002563606092068712,
+            0.04770277039047993,
+            -0.0032010143255574255,
+            -0.03425328601475849,
+            -0.02105909767072799,
+            -0.00840881022563103,
+            -0.028699447677549002,
+            -0.015925160861589006,
+            -0.0055750742281166,
+            0.017513987082479483,
+            0.010399718005952185,
+            0.02267201252273823,
+            0.01258511878601025,
+            0.0015081978484714903,
+            0.01874981572091931,
+            0.011158889421057756,
+            -0.002607810686437211,
+            -0.003732023411827874,
+            -0.007787189972165309,
+            -0.05524012485122728,
+            -0.025110689689279583,
+            -0.03946383236703947,
+            0.004443572529325475,
+            -0.018745077598477586,
+            0.005926160888274727,
+            0.039430331582190524,
+            -0.0049500324516682,
+            0.004352795787899792,
+            0.015566139991780173,
+            0.013148156356408315,
+            -0.015279560337042242,
+            -0.03316846614871165,
+            0.0001592209771257332,
+            0.018362411161743347,
+            -0.012706582418607377,
+            -0.03424477956681259,
+            -0.001991022841226589,
+            0.0033467286047492175,
+            0.03357019284647071,
+            -0.024359110796680374,
+            0.00359117029951049,
+            0.005061818038334802,
+            0.011359582931760315,
+            -0.026793992746244076,
+            -0.018170136079817746,
+            -0.017421603453826397,
+            0.03996608926471779,
+            0.02587013058343799,
+            0.015763410145648683,
+            -0.04512536449983358,
+            -0.02293473890636517,
+            0.048575083783480195,
+            -0.009589860756342604,
+            -0.06486262416944208,
+            0.03447666009581266,
+            0.016105654870253654,
+            0.003526582643722421,
+            -0.01354754623151814,
+            0.05401575248587799,
+            -0.03563190295559527,
+            -0.017365166993914202,
+            -0.0046288709698227735,
+            -0.033657212157212645,
+            -0.007686252059880836,
+            -0.001385273642139782,
+            0.048229062789530236,
+            -0.029083855744728694,
+            -0.030465559177859865,
+            0.0024368661249980416,
+            -0.024144031154493057,
+            0.021693278579520123,
+            -0.016405088055634124,
+            -0.018439935250344958,
+            0.006948603359995389,
+            -0.033943037395199695,
+            -0.003007796423804453,
+            -0.016783195426536633,
+            -0.02270589491206155,
+            0.020874438753828814,
+            0.016724666903417947,
+            0.027033184180344852,
+            0.0019847647852695785,
+            0.017749524526371133,
+            0.0204661893602285,
+            -0.005604929968008821,
+            0.028462234633072944,
+            -0.020790459496727066,
+            -0.04183816355941877,
+            0.03946487892663695,
+            0.01192861758684541,
+            -0.0018024841275380242,
+            0.01077082591611702,
+            -0.01475239020295694,
+            -0.03952356245715099,
+            0.03683950018557526,
+            -0.056767672022551764,
+            -0.007507600090017693,
+            0.009530544352246083,
+            -0.00788856051420074,
+            -0.011204694027562521,
+            -0.010587701466318598,
+            -0.028604940377315504,
+            0.023749537635431462,
+            -0.013672419168235691,
+            0.01953449229707597,
+            -0.011449939491554354,
+            -0.012238180529977605,
+            0.05795536247491174,
+            -0.021518581774137322,
+            -0.040672955795925836,
+            0.026066659238628543,
+            -0.014061106927896723,
+            0.03328357230532424,
+            0.03957107406164427,
+            0.030670947051333917,
+            -0.004278577327978564,
+            0.025885946540364826,
+            0.0319106101668716,
+            -0.003462160036198642,
+            -0.04686868330851868,
+            0.00555303933159587,
+            0.02156143971867743,
+            -0.03739529489075782,
+            -0.01025626126223168,
+            -0.02141970191890338,
+            0.022134910980548407,
+            0.017807484142010496,
+            -0.007232643256393455,
+            0.011318041192005274,
+            -0.010489012862455913,
+            -0.013003217163997554,
+            -0.00847279880392288,
+            0.02357274819910594,
+            -0.0016485066139744577,
+            -0.005873113634992993,
+            0.006424249429163587,
+            0.004008703814301658,
+            0.0054639858216457724,
+            0.010198079500408441,
+            0.017694720520798957,
+            -0.017087021465208255,
+            -0.0025558869389290554,
+            -0.016503579212015177,
+            -0.0420082404922411,
+            -0.003563173424359332,
+            0.022924161700566468,
+            0.012345933179990547,
+            -0.00729908668365632,
+            -0.004234683369642856,
+            -0.0022962545569187934,
+            -0.020666205489143392,
+            -0.014796149043465308,
+            0.024562653223873056,
+            0.030217494971086438,
+            0.007810961433007407,
+            0.007781263640449456,
+            -0.0030348559388780655,
+            0.007310876971216526,
+            0.016099591640649783,
+            0.02311929597780174,
+            -0.04160162038353416,
+            0.027152100845897007,
+            -0.00025396315502635146,
+            0.023717123370023598,
+            -0.04186918759097667,
+            -0.017988531857836703,
+            -0.03106663560912813,
+            -0.023595079699181436,
+            0.02437134274097885,
+            -0.02317559303912261,
+            -0.012327263775257306,
+            0.000989566600374836,
+            0.00954921216572729,
+            -0.005155079395685017,
+            -0.017508404061379164,
+            -0.04582321461886958,
+            -0.0014367264231457263,
+            -0.030417229142190933,
+            -0.017829710557332023,
+            -0.001056472801554354,
+            0.005016912084754242,
+            -0.06276644565618253,
+            0.0109965681725148,
+            -0.07270669271977365,
+            -0.004542796125637244,
+            0.040139998232966566,
+            -0.017890441783630643,
+            0.055909102081208445,
+            -0.05029104308417623,
+            0.01673962668706085,
+            -0.04507293803055318,
+            0.015385506084493484,
+            0.025677372277539627,
+            0.030235171586316813,
+            -0.022189978375852563,
+            0.008461188938299745,
+            0.008033707483259569,
+            -0.045089505755429415,
+            -0.04144698743458133,
+            -0.013640256640978024,
+            0.01036782197518379,
+            0.0012696958402094232,
+            -0.016090728932978384,
+            0.023838437877040163,
+            0.011946482002767135,
+            0.03342604921859589,
+            0.04211801335157251,
+            0.002708015395637621,
+            -0.047156646610976634,
+            -0.013104464812562618,
+            -0.003933116891120681,
+            0.03155321708116329,
+            -0.018866109579446184,
+            0.015376812010630058,
+            -0.01955557376085726,
+            0.016255217383093355,
+            0.017868332102274922,
+            -0.020325179506828335,
+            0.013434364256590097,
+            0.004176904701252156,
+            -0.03877060944639854,
+            0.0006475542749369515,
+            0.00042468300294954125,
+            -0.02116992978038693,
+            0.0032654387271108293,
+            -0.00575003786159731,
+            -0.05149954071159492,
+            -0.031382926943128234,
+            -0.015730563640253065,
+            0.0382312502769352,
+            0.004297939848910064,
+            0.019724983545557365,
+            -0.017907869879607064,
+            -0.005165628493894463,
+            0.03698976707579373,
+            0.008501001373266799,
+            -0.01725527821298066,
+            0.006879620077191753,
+            0.02455792200929138,
+            -0.03224193972301503,
+            -0.039181328884413266,
+            -0.0008206986106407542,
+            -0.023306758893533952,
+            0.01672946960876128,
+            -0.02658587698869307,
+            0.005873620623736886,
+            0.0025331170979801232,
+            -0.03490809296591325,
+            0.02456764314036355,
+            0.003291113462353163,
+            -0.004303423672882112,
+            -0.020359544091059775,
+            -0.016892415165120203,
+            0.017174797487220006,
+            0.04391489511478065,
+            0.007327321684948843,
+            -0.003943713832544148,
+            0.00862684231313155,
+            -0.006361172785753631,
+            0.05826688726534012,
+            -0.02284956969636637,
+            -0.026680023999604643,
+            0.027527436347931256,
+            -0.022597669279069723,
+            -0.006277863878949772,
+            -0.017343446085471857,
+            -0.012065192152203348,
+            -0.0004183712490442143,
+            0.019339311388446943,
+            -0.019423562920897294,
+            -0.010212527447243049,
+            0.004783006556748757,
+            0.012824133870627983,
+            -0.004855945108275706,
+            0.006239949458562236,
+            0.006847581690471813,
+            0.03094005096378827,
+            0.026993131409761086,
+            -0.011331881651539968,
+            0.03297014291876675,
+            0.010265600643742144,
+            4.666113318251278e-05,
+            0.001970879281172325,
+            0.002523664335535017,
+            0.0034107811520368574,
+            -0.03143890864030425,
+            -0.01624584278940617,
+            0.013468729048885906,
+            0.005233309338223403,
+            -0.004550544967856448,
+            -0.023783653541634683,
+            -0.009402494239255264,
+            0.03334923712760375,
+            -0.003260424721959948,
+            0.02748395114281811,
+            -0.0058374897745283065,
+            -0.01985840930273332,
+            0.017595418271610163,
+            0.02881743246631328,
+            -0.05928552440605384,
+            0.017062540894682354,
+            -0.0011508547522925793,
+            0.008953761538519997,
+            0.01610561914042553,
+            -0.008894160513648953,
+            0.013301991014533684,
+            0.04080863883897929,
+            0.010738051773755976,
+            -0.023345002072886145,
+            0.004215726310838674,
+            0.01541749448570608,
+            0.01291813299324752,
+            -0.022747942296528025,
+            0.0008640791861357522,
+            0.011008522066277018,
+            0.005893793007012561,
+            0.0026106450193140156,
+            -0.006278394080583205,
+            -0.025659188287146268,
+            0.024180550333596625,
+            -0.0044491292740074925,
+            0.026967973346698617,
+            0.04390909273429254,
+            0.009596855239692042,
+            -0.03686020608700941,
+            -0.013109951664921729,
+            0.0002799680359670127,
+            -0.0038537577602593366,
+            -0.0037552094562213916,
+            0.011105295862811114,
+            -0.04185615640149602,
+            0.019950890919369135,
+            0.006453241355231516,
+            0.015314645427201533,
+            -0.020031193483734364,
+            -0.024530173488182337,
+            0.0031949828520724337,
+            -0.01492645564652531,
+            0.012875827757040941,
+            0.005057170413681063,
+            -0.004319834710426384,
+            0.005386862713494232,
+            0.00030848054013413157,
+            0.008384941028326983,
+            -0.010811526993301901,
+            0.03175068297908541,
+            0.004680987487229685,
+            0.009264547472281849,
+            -0.02856314241700808,
+            -0.021083721348429348,
+            0.020113396771629096,
+            0.03572366771051171,
+            -0.01708529918264109,
+            0.03346921724975398,
+            -0.009042046660366718,
+            0.0033219155387035227,
+            0.0151720168318272,
+            -0.015805318210244414,
+            -0.008057236707694303,
+            -0.0005483009439015931,
+            -0.002917527004423299,
+            -0.014538511124345527,
+            0.007302979537905532,
+            -0.0208529958115422,
+            -0.02965227242147972,
+            0.020064178921373875,
+            -0.00011892622932349096,
+            0.016455188521316146,
+            0.010335044657747514,
+            -0.007816155992852236,
+            -0.0052262330850905165,
+            0.006959344748883874,
+            0.04432147882994085,
+            0.026734484553994495,
+            0.05547661378166294,
+            -0.005787997952888261,
+            -0.004559444854138438,
+            -0.0082295581123363,
+            -0.006588172755707144,
+            0.04524009609473585,
+            0.015915003261638767,
+            -0.004910285095602864,
+            0.017495607570191165,
+            0.01557886571266685,
+            0.03858259260136037,
+            0.0027648525902558316,
+            -0.016912796336835408,
+            0.04014755669075221,
+            -0.004077405782937414,
+            0.008616827127381418,
+            0.011102692199778469,
+            0.0024185742218798562,
+            0.00807141794538637,
+            0.02996274720815659,
+            -0.012429186019907761,
+            0.01680942269506771,
+            -0.04571916592313758,
+            -0.02775530246832996,
+            0.014370280587419414,
+            -0.004931905540824582,
+            0.03419959665162407,
+            0.01668521021731861,
+            -0.023105305446763272,
+            -0.024761385385514877,
+            0.021529428247380984,
+            -0.002603042335561961,
+            -0.024135326707477325,
+            -0.015906833215844473,
+            0.003286605488908195,
+            -0.0023437997044735617,
+            0.0060251848371355166,
+            -0.013481652076194643,
+            0.0002908426939460218,
+            0.01953362396253743,
+            0.0038544889405965057,
+            0.020760460182873287,
+            -0.016224396121239458,
+            -0.00259233446682179,
+            -0.009282861237653619,
+            -0.02952130765234087,
+            0.017988370145301916,
+            0.006138236653154911,
+            -0.007607182955958238,
+            -0.009516767050028553,
+            0.0048347778733230365,
+            -0.005819209642901871,
+            -0.0025851701689263706,
+            -0.01073014664868326,
+            0.012433465684360061,
+            0.025784014942833097,
+            -0.016875624785552303,
+            0.007944349714123104,
+            8.651589872318479e-05,
+            -0.022306044741692753,
+            -0.022127016745066307,
+            0.020143472307319017,
+            0.002792646249762062,
+            0.008768813346304779,
+            -0.004074214276300631,
+            -0.00015024341785520542,
+            0.002990550758627699,
+            0.01951746553838439,
+            0.03896451381932467,
+            0.004232673242619471,
+            0.027876586543280867,
+            0.0145860303663199,
+            0.03994234200854897,
+            0.009657017763624678,
+            0.00696607158576712,
+            -0.026055387694656488,
+            0.02819890357011778,
+            0.00018297502510753014,
+            0.043190622821775236,
+            0.022488402221902314,
+            -0.0407730101161195,
+            -0.0021946518523759066,
+            0.01669520746276111,
+            0.013212623257794862,
+            0.017009844136059122,
+            -0.004537512994041156,
+            -0.014041213424740049,
+            0.028135404528975894,
+            -0.022859586430516003,
+            0.0008637006862481223,
+            -0.004486332934188017,
+            -0.025031795510001887,
+            0.007351370254175368,
+            -0.004567816916440959,
+            0.030025413528233364,
+            0.0014976740344892735,
+            -0.022285096307825804,
+            0.022837170656498543,
+            -0.009551269792616045,
+            0.011045530488684207,
+            0.008809047185288755,
+            0.020065001354321994,
+            0.031993362733068575,
+            0.009143210434415064,
+            0.02133640814803879,
+            0.012241777474889248,
+            0.014771538127546474,
+            -0.008804192497858033,
+            -0.02341427260922065,
+            0.023167424359504862,
+            -0.0011598649565273595,
+            -0.029627604902846688,
+            0.0021991474725787303,
+            0.003261878079344105,
+            -0.0005935713331682358,
+            -0.009144689470851402,
+            -0.005580557862988033,
+            -0.007240836451875213,
+            0.019631377011298905,
+            -0.007085213886352692,
+            0.007017573363709754,
+            -0.004074171791476129,
+            -0.00822871835665378,
+            -0.0421101573155962,
+            0.004210914075286222,
+            0.003408427700622687,
+            -0.01752179330665445,
+            0.006384440404174512,
+            0.011121481025573962,
+            0.043020575455262985,
+            -0.022809979913698254,
+            -0.0005737138409049003,
+            0.019206623212867476,
+            0.0009562551889820241,
+            -0.023427467810802533,
+            -0.038931688376706186,
+            0.007172345672967499,
+            -0.03467050750542982,
+            0.023185301620612612,
+            -0.0007075596339842658,
+            0.02387977204565927,
+            -0.03043468414449851,
+            0.013909304572370437,
+            0.004784668158421625,
+            0.012524879174804915,
+            0.021654512058639427,
+            0.03142146922870715,
+            -0.02053473475540882,
+            -0.050409862638621725,
+            -0.005406667316025549,
+            0.00035138762584085976,
+            0.022018487016014856,
+            -0.026014580970974616,
+            0.00025633004699302783,
+            -0.003349094027291921,
+            -0.02401366769212778,
+            -0.01735996745614562,
+            0.01962340791446025,
+            -0.050011480783403936,
+            0.015042913576492952,
+            0.0262737424555697,
+            -0.02507473901367252,
+            -0.05408889321034101,
+            -0.018988587082325706,
+            -0.028217893770659686,
+            -0.009285770094319318,
+            -0.00661159230987788,
+            0.028597405154740078,
+            0.003525576256054959,
+            -0.011462219358952466,
+            0.025401668390029348,
+            0.006155837251830549,
+            0.0026050693705924593,
+            0.015985722392086222,
+            0.007289910674212967,
+            -0.011452511453623235,
+            -0.014599577191293152,
+            -0.04558584556100621,
+            0.010197536827229795,
+            -0.0045431962637690155,
+            -0.019372598050508005,
+            -0.030147301333718016,
+            -0.0137828704002156,
+            -0.027060515748528666,
+            -0.0017291966950107505,
+            0.053783684195381654,
+            0.03292494290301388,
+            0.005560172296221419,
+            -0.0026085670565688026,
+            -0.010466890489356092,
+            0.036247483457995895,
+            -0.006789746401727701,
+            0.03662690472838588,
+            0.001784522073210422,
+            -0.0242673253380092,
+            0.01169382884548301,
+            0.02892627181542449,
+            0.009858451662166387,
+            -0.0005592283645355213,
+            -0.006160132931009508,
+            -0.008782830470079547,
+            0.0031207630972208394,
+            0.025613233596604418,
+            0.003268170392795548,
+            -0.010062991425127038,
+            0.018106967407336815,
+            0.004480291699976396,
+            0.02010374270557836,
+            0.03555357587371324,
+            0.023489837757983054,
+            -0.021782173652026497,
+            -0.026581762796336082,
+            -0.0013561486781077014,
+            0.0072266106510934505,
+            0.03551967295942857,
+            -0.01651315514034218,
+            0.0001274396319698452,
+            -0.008272735507120127,
+            -0.00894931703949637,
+            0.011149952898504355,
+            -0.0029671593121986116,
+            -0.0030752010285951576,
+            0.01662711184497578,
+            -0.0011672263615879709,
+            -0.033448222499147764,
+            0.0078288560210653,
+            -0.023496331346917084,
+            0.0051040606404415545,
+            -0.004113104164416575,
+            -0.007076525381595441,
+            -0.002157548492138289,
+            0.006302962169276006,
+            -0.008065841090139872,
+            -0.00035242603988955366,
+            -0.01586406637763306,
+            0.008271894439780969,
+            0.007438312999409211,
+            -0.0069617481460406095,
+            0.011402932583388205,
+            0.002666895909532698,
+            0.0005219881149353855,
+            -0.012159425670629751,
+            -0.012042145306711322,
+            -0.00957276588761916,
+            -0.03401863879157551,
+            -0.02604353712472079,
+            -0.024949533021906844,
+            -0.0014209149651479556,
+            -0.006566685925666972,
+            -0.03187796739029736,
+            -0.01209152137926458,
+            -0.011110763124407752,
+            0.007756961097537267,
+            -0.013260548545917543,
+            0.005433285079114047,
+            -6.399098520041351e-05,
+            -0.0044960488826674555,
+            0.016615636438440516,
+            -0.003643291599919672,
+            -0.019488010211707343,
+            -0.03927818878149417,
+            0.008209336177358258,
+            -0.02805593125673831,
+            -0.0038816771550190788,
+            0.007607119219721711,
+            -0.00528681677614909,
+            -0.029502897838743516,
+            -0.009149794737430706,
+            0.008767001199311985,
+            0.003507217008280066,
+            -0.01773349756494436,
+            0.03579670773605949,
+            0.026971617790626075,
+            0.011875179879209696,
+            -0.019891342676443626,
+            0.0508929813275565,
+            -0.0047010735337056275,
+            -0.012544935684845649,
+            0.018297503869683027,
+            0.02355022268131519,
+            0.03273167043015095,
+            -0.024830243320788693,
+            0.0009370814985214882,
+            -0.021937793653233454,
+            0.00535662615596573,
+            -0.035009915581788176,
+            0.0028595659314725012,
+            -0.014395027018699446,
+            0.015596459769560532,
+            0.03397372881171701,
+            -0.013254226072981457,
+            0.011195747181764677,
+            -0.016299384375589934,
+            0.00730244731485402,
+            0.00569944090461748,
+            -0.012916284280345044,
+            -0.005812311882399539,
+            -0.022902815152116493,
+            0.020140276669979015,
+            0.005809799950176385,
+            -0.0008130655714656685,
+            0.0104085826652653,
+            -0.044723432238900435,
+            0.03286429597337671,
+            0.008730225944501239,
+            0.017599189944921396,
+            0.0029762446259779998,
+            -0.0003068671951232415,
+            0.011611551606224214,
+            0.011228401175698422,
+            -0.016029727902740706,
+            0.004345913009102136,
+            -0.004613061137378433,
+            -0.03850533909474302,
+            0.009363351681220008,
+            -0.03631184057978666,
+            0.014427784677340159,
+            0.016026872002233575,
+            -0.004170582842643461,
+            -0.025166189571409366,
+            -0.005428471012333673,
+            0.010827910811085384,
+            -0.009783069664327864,
+            -0.009987805234276443,
+            -0.04163358183060102,
+            -0.025236440364963034,
+            0.034097508113604576,
+            -0.002119474613596975,
+            -0.025826403511707147,
+            -0.010830738112107807,
+            -0.011351132636491575,
+            0.020207687645133,
+            -0.006542026283946858,
+            -0.0006263975804936843,
+            -0.04150011545311013,
+            -0.0073396962917864515,
+            0.011192225907726928,
+            0.027746213794734322,
+            -0.035970017302005566,
+            0.02708126943501373,
+            0.002105633542220187,
+            0.014262674845755191,
+            -0.01626664983003942,
+            -0.005631805361903626,
+            -0.013990539604372344,
+            -0.035877093361375685,
+            0.016487723013906077,
+            -0.008607758248055054,
+            0.004714436224135091,
+            0.04458009880881818,
+            -0.0036872068506712727,
+            -0.009466125348499464,
+            0.007600705817195143,
+            -0.0034590883553048063,
+            -0.010351576085287273,
+            -0.0028564505793999295,
+            0.001959661298507,
+            -0.007080697863126256,
+            -0.04183476693971616,
+            -0.00499888976530217,
+            0.04684391782633148,
+            0.006361744381879857,
+            0.013276793765092465,
+            0.013516220521120684,
+            -0.018335746557932003,
+            -0.045577810515431215,
+            -0.0071616111758423025,
+            -0.0032763880176386808,
+            -0.022708028392758567,
+            -0.0240284945459861,
+            -0.008735040856657758,
+            0.005844145402902923,
+            0.006359368165686724,
+            -0.01120763599008222,
+            -0.022676131361171057,
+            -0.027085968700452136,
+            -0.010526103176253857,
+            -0.02123944716136792,
+            -0.010792145796733375,
+            -0.027142270190771887,
+            -0.028742234538796804,
+            0.02880426953386151,
+            0.007575182232859564,
+            -0.004084088301176117,
+            -0.010005765476121966,
+            -0.01255640546813973,
+            -0.0004774761617900028,
+            -0.0025307994917461232,
+            -0.005822065709999144,
+            0.0038614286313282164,
+            -0.011390482620811868,
+            0.023152086963976554,
+            0.0021220701260546895,
+            0.018329162172128138,
+            -0.05772324571231113,
+            -0.002709083851090071,
+            0.021497896395795905,
+            0.0033025410488410913,
+            -0.005412445141910314,
+            0.029352650459792132,
+            0.011223947499453582,
+            -0.006539460282439711,
+            -0.031355428613718234,
+            0.020942976176820262,
+            0.024122993515785947,
+            0.00817615634503138,
+            -0.012994160510030828,
+            0.01221095914384204,
+            0.01629778384790159,
+            -0.004494985018246821,
+            -0.011161098655072933,
+            0.005586857967039181,
+            0.020941039642372112,
+            0.013580622629911925,
+            0.0013398071867077221,
+            0.030905022925366163,
+            0.0006862084391508121,
+            -0.03521231612862609,
+            -0.01908191995763771,
+            -0.01275995164030872,
+            -0.003584331305986364,
+            -0.015264425022763126,
+            0.005294329654809353,
+            -0.002355465010876059,
+            0.006335052149475905,
+            0.022655379849785256,
+            0.03593231939948758,
+            0.0146937743203089,
+            0.00937330328448886,
+            0.003167462724350819,
+            -0.0013235176101076284,
+            0.012102872918129403,
+            0.008638011773075119,
+            0.0350701240268404,
+            -0.014653221697693864,
+            -0.011003994230902835,
+            -0.0037767288519369537,
+            0.010100131061186566,
+            -0.004113527518662366,
+            0.020414393256136372,
+            0.03274709615649469,
+            -0.0037222949335916876,
+            -0.005604552180246868,
+            -0.006160118231975786,
+            0.0041467454414452565,
+            0.01807879127455274,
+            -0.01252134965321004,
+            0.004364858987501461,
+            0.005549300665159658,
+            -0.0071897162833861,
+            0.018567059255813558,
+            0.0002622074720860951,
+            -0.012983550838417323,
+            0.03266376564402191,
+            0.11449509387240629,
+            0.0017251916835047628,
+            0.007125215166916139,
+            -0.040229997815548584,
+            -0.020537987527148888,
+            -0.00245728735683473,
+            0.005779972828836593,
+            0.007004732428236457,
+            0.02225914982152608,
+            0.006505827715591914,
+            0.0006708045351142873,
+            0.031708522931495536,
+            -0.017334083511709092,
+            -0.002522861743921599,
+            -0.01745656559688129,
+            0.013890451996979048,
+            -0.014312605700272691,
+            -0.013784963572164409,
+            -0.01572417853955075,
+            0.004507425871748977,
+            -0.000601158156253668,
+            0.0016158265715634729,
+            0.01791303234734911,
+            0.0011926917387973374,
+            -0.026393351252156246,
+            0.0006694008039871878,
+            0.001026031515214626,
+            -0.004514419527575117,
+            -0.011320929719992686,
+            -0.02223001723942815,
+            0.016235841442653918,
+            0.014188311619681008,
+            -0.004895218832118094,
+            0.009244697376301004,
+            0.010558796153959677,
+            0.02707485418319703,
+            -0.028380342401606758,
+            0.0040204430418629405,
+            0.007023041225985294,
+            0.01606556581447933,
+            0.017617862031190843,
+            0.0034870847111552862,
+            -0.011253315705053224,
+            0.03238289399766116,
+            -0.010563282687352038,
+            0.0065093536270233385,
+            0.014966802208644462,
+            -0.03541611540945151,
+            -5.821682910348165e-05,
+            0.0061985444938402745,
+            -0.011716201661363997,
+            0.009580035961315467,
+            -0.03120692534097433,
+            -0.0039291243987978845,
+            0.009317468331974374,
+            0.017611312558635673,
+            0.008804996982574765,
+            -0.0034129169537151925,
+            0.010822226698822738,
+            0.012510661458054741,
+            -0.0027147975086409966,
+            -0.008855213644027594,
+            0.01238502686635956,
+            -0.0007687053446954701,
+            0.00450108863711973,
+            -0.00799170686059026,
+            0.0029755595812552293,
+            0.002290650917244691,
+            -0.01554856300008412,
+            -0.011779430686394945,
+            0.00438393126679289,
+            0.0036931891075561957,
+            0.001535662168147833,
+            0.012014408636737167,
+            -0.006087255045485002,
+            -0.0077816220325470226,
+            0.0040573724156794284,
+            0.003136943055301706,
+            -5.505014316363865e-05,
+            -0.030625245024561508,
+            0.01585923084309149,
+            0.0067595435845396405,
+            0.010013308598626031,
+            -0.005315626260228517,
+            -0.00903142414091241,
+            -0.007240920015018893,
+            -0.01228521172101564,
+            -0.010657894223173453,
+            0.007658605592936815,
+            -0.00724924993432376,
+            0.011124174211132038,
+            -0.03128027787357322,
+            -0.013988715435985316,
+            0.01231792993245938,
+            -0.01528161242159377,
+            0.0007675821323979293,
+            -0.008389914312201402,
+            0.0009417242487395995,
+            -0.005119304379761705,
+            0.007887436211143328,
+            -0.03731503169547854,
+            0.007878001990271541,
+            0.02456170736765724,
+            -0.00330924625934981,
+            -0.020259771366167557,
+            -0.009613930457931786,
+            0.0241314385136084,
+            -0.008267831641886344,
+            -0.01645888287387047,
+            0.008421304862524506,
+            0.010140813149093993,
+            -0.0006309118552875393,
+            0.0008237784055549769,
+            -0.003726984191590917,
+            0.01669125388798323,
+            0.02054777616420997,
+            0.041930770964107515,
+            -0.014789610432004559,
+            0.01092349703703152,
+            0.02866938974792608,
+            -0.03378576980722847,
+            0.014728113846952417,
+            0.02518409645155824,
+            0.014590531327650454,
+            -0.0004446956315190408,
+            0.001609777186372437,
+            0.0068939321280085475,
+            -0.014273492115311677,
+            0.009247676157551494,
+            0.013380104440744367,
+            -0.018407746260686045,
+            0.020444430509441124,
+            0.03140071554008003,
+            0.0025839998101985736,
+            0.01355205268903058,
+            -0.006268939830271642,
+            -0.013860692186849113,
+            0.014717364068116991,
+            0.031429975488060896,
+            0.0006919565473572909,
+            -0.018561500446336806,
+            0.019128244642842268,
+            0.006988095806264863,
+            -0.003678529269242034,
+            -0.0021993078588142664,
+            -0.021334919445200388,
+            -0.021003586791369228,
+            0.017304350704792804,
+            0.013508779408333964,
+            0.027794485858413365,
+            -0.010594702782952029,
+            -0.007818089330662161,
+            0.014293688127901867,
+            -0.022368568887004094,
+            0.015426482139881204,
+            -0.02160820059598751,
+            -0.00571499813703566,
+            -0.0077402210272634235,
+            0.03274360648629609,
+            0.001441304806200403,
+            0.02734331677589297,
+            0.016457027764064002,
+            -0.033609420134045206,
+            -0.029790535553657092,
+            -0.00712106584238484,
+            -0.02307933297713104,
+            0.010356561171963577,
+            -0.03901183838032092,
+            -0.00071735150189382,
+            0.008251646656432826,
+            -0.008079917288471352,
+            -0.0007087526632966007,
+            -0.006419191399519291,
+            0.013989344185652506,
+            0.013661633160336152,
+            -0.0228103143826663,
+            0.023644639659577426,
+            0.008343481940133684,
+            0.025998483906975782,
+            -0.02076055376006461,
+            0.019148568238437927,
+            -0.017705363646416224,
+            -0.02777313933338929,
+            0.014358130220942483,
+            0.023585731219788467,
+            0.01635364038245971,
+            -0.007122853965024528,
+            0.007110003583832299,
+            -0.03556761320536796,
+            0.027585457849964405,
+            -0.013064062623866305,
+            -0.012504161289952458,
+            -0.0032379276562094555,
+            -0.011931827532360686,
+            -0.002574161005800279,
+            0.0008533907465694845,
+            0.0034345232246102806,
+            -0.03131462673429805,
+            -0.0002450959360520896,
+            -0.03727416416766574,
+            -0.028944814604815072,
+            0.01360126708805471,
+            -0.00036361284498363273,
+            -0.0049097813612814616,
+            0.02933242705521636,
+            -0.026339136251046315,
+            -0.02693295650849956,
+            -0.009590247544093352,
+            0.02853096456887074,
+            -0.0220673798697511,
+            -0.004519050393279848,
+            0.0015034014871316069,
+            -0.01828226786700525,
+            0.004436383319946364,
+            -0.0012748481042235446,
+            -0.010416065158876917,
+            -0.01105146139270597,
+            -0.03229479099927737,
+            -0.005822133182096844,
+            0.02106215163620738,
+            -0.006957125617688484,
+            0.002746241228040847,
+            0.013312248331456934,
+            -0.010968629118224582,
+            -0.0029946900704643812,
+            0.0017875284441794235,
+            0.023304175718480834,
+            -0.021033941064173063,
+            0.007714334541806199,
+            0.005955442031111047,
+            0.0016405586860131207,
+            0.018406562522853148,
+            -0.0054922197063461825,
+            0.0037690571959124617,
+            0.0162126534419016,
+            -0.009969304596173302,
+            0.017959307638726878,
+            0.007691549802475525,
+            0.006681837561745192,
+            0.002775700618222622,
+            0.007712770283579111,
+            0.009752652973700739,
+            -0.013506020126078112,
+            0.016911141425375933,
+            0.022468715726349706,
+            0.00023233555209372766,
+            0.021066580799163234,
+            -0.02302516292580905,
+            -0.028618544628453015,
+            0.018297878810036087,
+            -0.0225078941299981,
+            -0.012735208680311652,
+            0.016459925496937065,
+            -0.0043032724578332855,
+            -0.00797609922711709,
+            -0.014986057073351055,
+            0.02260618561608301,
+            0.02132930921464508,
+            -0.020249788476562527,
+            -0.012385936327068884,
+            0.0072304431443959505,
+            -0.004373475447051833,
+            0.007086656081865733,
+            0.03195810137269375,
+            -0.015982210897392764,
+            0.006818890093901413,
+            -0.0038306138017521626,
+            0.0030753642449792215,
+            0.017830546304667783,
+            0.009470018737705222,
+            -0.0019815241970236146,
+            0.030100238416495066,
+            -0.001330614013444443,
+            0.016392226282459177,
+            0.0004888028477871524,
+            -0.010948833692356176,
+            0.006343890495626752,
+            -0.026652806306173643,
+            -0.008627570055007363,
+            -0.001203530223345183,
+            -0.009581494451831351,
+            0.07598616930784373,
+            0.003930153552069478,
+            -0.01295957554290434,
+            -0.001673007115291578,
+            -0.001368287102181597,
+            -0.006897179139643672,
+            -0.00960679451603541,
+            0.0287166325268306,
+            -0.016613613331623346,
+            -0.01894527592131826,
+            0.0008370014132002703,
+            -0.006785618888421608,
+            0.00919924251249592,
+            0.01816363936999796,
+            -0.010225961928154075,
+            -0.01604759761423727,
+            0.001577138242530474,
+            -0.003520564936698765,
+            -0.004395434431595494,
+            0.0052399043934902014,
+            -0.010127122101423486,
+            0.006456662777583096,
+            0.026560491731327886,
+            -0.013684536319659296,
+            -0.01425196172663986,
+            0.004084982328707292,
+            -0.0010682084059972982,
+            0.009758641913073978,
+            -0.00864879062601467,
+            -0.00481222050200678,
+            -0.014375330954464725,
+            -0.011378890549403524,
+            0.02505641780042516,
+            -0.02861164504722546,
+            0.0034060847953066354,
+            0.014937227531980433,
+            -0.00016904224334342915,
+            0.01541381851648916,
+            0.0033028855218525644,
+            -0.03615935765718852,
+            -0.013677774380513626,
+            -0.02073839691250734,
+            0.019821074228011413,
+            0.01688422012576042,
+            -0.0033731840788147442,
+            -0.013840993586441808,
+            -0.026149880178874887,
+            -0.007430563876740068,
+            -0.02466234818567362,
+            -0.020708282040996014,
+            -0.019276991448222817,
+            0.006137002799644838,
+            0.005162892426268572,
+            0.022881759407670583,
+            0.0025854880108157857,
+            -0.015058889448404231,
+            0.0029948608544278386,
+            -0.006722836436935071,
+            0.011845298212977357,
+            -0.02747944486818378,
+            0.015311272959213262,
+            -0.01610493464983465,
+            0.025239768880017402,
+            0.007425208902253236,
+            0.0010569008485112994,
+            -0.008822681490988575,
+            0.00010704167529046774,
+            0.014883787023445807,
+            -0.004186968199973372,
+            -0.0007781259002896265,
+            0.02787555889813697,
+            0.02087373813833492,
+            -0.0017783425135865238,
+            0.008743718448136432,
+            0.028026471206552342,
+            0.019081083283793984,
+            0.0325261942213039,
+            -0.013815526479760766,
+            0.005113676604826083,
+            -0.028873600114081546,
+            0.010510475843707614,
+            -0.010871039729515013,
+            -0.010201824683340651,
+            -0.0067946599549165735,
+            0.027756887504389347,
+            -0.007962601205446472,
+            -0.0006687976124826922,
+            0.03469894503563926,
+            -0.014185294847328063,
+            -0.00010543064621374546,
+            0.0007386418834840544,
+            -0.009759481309152999,
+            0.02742722730386853,
+            -0.0073590367291510046,
+            0.018456032721630843,
+            -0.019560882612141943,
+            -0.013705917411254969,
+            -0.011894560208199602,
+            -0.001142768182039817,
+            -0.02576754877141608,
+            -0.008755240066826174,
+            -0.02249287216087102,
+            0.00859172262856532,
+            -0.0046012388553094375,
+            0.018845920253446127,
+            0.0011617844393195566,
+            -0.005275799428018491,
+            0.029069367036565145,
+            0.0088080893110627,
+            0.018100411389202663,
+            -0.0004971948011851863,
+            0.01216566778850985,
+            0.02369448391321832,
+            0.002637001729688794,
+            0.005012948614448268,
+            -0.024213704079796478,
+            0.003420184574622472,
+            -0.0004359900462023086,
+            8.265488264269535e-05,
+            0.016267948861008616,
+            -0.019572032453100303,
+            -0.02216264534756073,
+            0.021871880626992544,
+            -0.012742926070754964,
+            0.011676769226089007,
+            0.0035362870996336194,
+            -0.010732229460038356,
+            0.0006838777488770268,
+            -0.03083438636386314,
+            0.008820746379709355,
+            -0.03216685858285627,
+            -0.032230808828686675,
+            -0.013547681710236494,
+            -0.009994106571421729,
+            0.0008758088338659712,
+            0.028447187234357855,
+            -0.00795317029726568,
+            -0.015797892954136448,
+            0.005363565557248085,
+            -0.011199252697624111,
+            -0.00817162221789202,
+            -0.0007867892785855576,
+            0.005856939999700809,
+            0.009461959355310273,
+            0.005997804620690314,
+            0.011648335019189696,
+            0.007491946658046298,
+            -0.008028800679169293,
+            -0.008837725220736191,
+            0.015889923814605387,
+            -0.0014409716087425222,
+            -0.01176424391171685,
+            0.030430130082762363,
+            -0.013300620185884817,
+            -0.0005592926632214984,
+            -0.0021975669766032472,
+            -0.0015687208316663597,
+            -0.0021652076181411244,
+            -0.014225289143358926,
+            0.01870568404189425,
+            8.334501002782455e-05,
+            -0.014288806218067678,
+            0.0032841743905975634,
+            0.003841780428153241,
+            -0.0036498339483411074,
+            0.008388922960269383,
+            -0.008147275785908849,
+            0.017348147498892,
+            -0.015113395971380741,
+            0.008740452410210708,
+            0.0038335161236289234,
+            0.0043482078469644385,
+            -0.0065755348640655865,
+            -0.01592809184384243,
+            0.009642593996701513,
+            0.012051872087663658,
+            0.02173264012652684,
+            -0.02525660485859878,
+            0.00876161653901566,
+            -0.005374533920781607,
+            -0.014534179652013885,
+            -0.016127866637369738,
+            0.00982303248417706,
+            0.022322910435388785,
+            0.0063086346898091296,
+            0.012139263756190026,
+            0.00013383660542132047,
+            0.005848259030010402,
+            -0.014767198851314148,
+            0.010570158368602513,
+            -0.011355774405066537,
+            -0.018439261453084426,
+            -0.005243587266408389,
+            0.028894120979627323,
+            -0.005492024421080143,
+            0.017938747701972086,
+            0.0021350711250093858,
+            0.029007356490801176,
+            -0.008686781541806358,
+            0.0014883465513564527,
+            0.0039866022469891155,
+            -0.01295271525889559,
+            -0.004272008629260297,
+            0.01976273843646234,
+            0.04882868638601926,
+            0.010204475902027572,
+            -0.0009155109545661691,
+            0.010069071220988797,
+            0.005626418787776309,
+            0.002148101019776153,
+            -0.00805781810841286,
+            -0.030431071067211577,
+            0.008883707378543252,
+            0.005628218215212441,
+            -0.01997825125961566,
+            0.019838234890736068,
+            0.026483365072033395,
+            0.023040623182975964,
+            -0.011585643598171132,
+            0.016539930468533123,
+            -0.010092529089936188,
+            0.0177008735686489,
+            0.028665424114962996,
+            0.013810061902648515,
+            -0.010487150104822674,
+            -0.0020847870330548534,
+            0.006329342599564054,
+            0.004186981876004016,
+            0.00026239408223817384,
+            0.015190614519766204,
+            -0.019155180136775043,
+            0.01005728016778505,
+            0.014725826697964637,
+            0.03322436221319799,
+            8.049930133627564e-05,
+            -0.01645191835557271,
+            0.0016249281996128196,
+            -0.009976155662939693,
+            0.00856399492362911,
+            0.000632427127231252,
+            0.0032242260692672124,
+            -0.009688119824276678,
+            -0.012682779503915694,
+            -0.002851918183031152,
+            0.009403205508474241,
+            0.0021188716361101698,
+            0.002759004011231831,
+            0.007641925150177999,
+            0.0022830062031146204,
+            0.01713024474384216,
+            -0.0011181980841546123,
+            -0.00029141889484139217,
+            0.004601090256674874,
+            0.010509199590505558,
+            -0.01649852835819883,
+            -0.017548369964263514,
+            0.004337237475053179,
+            0.012217494210546133,
+            -0.013159179948224596,
+            -0.013241981632059755,
+            0.015704543608959117,
+            -0.01913575130892795,
+            0.024934102868065366,
+            -0.00942381892138331,
+            -0.004543611336526587,
+            0.000343200128934031,
+            -0.00727538880611703,
+            -0.0038751118497522673,
+            -0.0030064279504724797,
+            0.003984715814505561,
+            0.0019261818110367147,
+            0.007472350699291607,
+            0.008472369788801792,
+            0.014568547873264247,
+            0.006043895352236181,
+            0.0032580830408778395,
+            0.009109104498825864,
+            -0.014523846294121576,
+            -0.014800649887278973,
+            -0.002704791598011488,
+            -0.008895394038700809,
+            0.00902016867266151,
+            -0.006607182154874112,
+            0.00845408382182461,
+            -0.015501867835442603,
+            0.015573642225745032,
+            -0.012200552219016517,
+            -0.0008900215712478553,
+            -0.009666233545144905,
+            -2.7470277092642562e-05,
+            0.008191697904514857,
+            0.014605287653054787,
+            0.0070552942843382215,
+            -0.031921405225964206,
+            0.0017403427534633655,
+            -0.008846568352320488,
+            0.03446319360336241,
+            0.002612606662691094,
+            -0.01060863123202008,
+            0.020352181356568008,
+            -0.01592124132155158,
+            -0.00027704668715660693,
+            -0.01602075523652062,
+            -0.004797557413047212,
+            0.00707939446796679,
+            0.016399017978990946,
+            0.02759049512664933,
+            -0.005697020356865688,
+            -0.007995305872076221,
+            0.006384047872862793,
+            0.029590696670334232,
+            0.012246955508890425,
+            0.024748351124415785,
+            -0.007633712004672087,
+            0.014348228716100414,
+            0.007773260552660029,
+            0.0076045488653348975,
+            0.004321689250440021,
+            -0.002591801797145718,
+            -0.011698574606499644,
+            -0.014974971814360001,
+            -0.0037563183330623384,
+            -0.016987915346053498,
+            0.00012247833530362867,
+            0.021344400276501117,
+            -0.00475351423959752,
+            -0.0033847161875730026,
+            -0.01874186437694691,
+            0.0033461670218611346,
+            -0.010029445302549983,
+            -0.00367421246775698,
+            -0.013133469163793961,
+            -0.0254122035745965,
+            -0.018556369080030412,
+            -0.002155244024999317,
+            -0.00929309186646207,
+            -0.015194269781263309,
+            0.0070354690219415535,
+            -0.016305117309271634,
+            0.0080837166473439,
+            -0.00448473417024141,
+            -0.016508775226236883,
+            -0.00958925666995396,
+            0.01100335153658195,
+            -0.0041819993634961685,
+            -0.01929493648733792,
+            -0.01282300765645703,
+            -0.011300260564838231,
+            0.0051733704121938884,
+            0.005996780699975213,
+            -0.0038111648304178544,
+            -0.0028279639348479987,
+            0.016640121845366295,
+            -0.010197365811240557,
+            0.009878730312873492,
+            -0.03335474019383245,
+            -0.03551826040406959,
+            0.004497847402931096,
+            0.01507007013703286,
+            0.0013085881889541252,
+            -0.02078680101065954,
+            0.010806315988966003,
+            -0.016499186313125556,
+            -0.00858013792516617,
+            -0.008522534127498645,
+            0.009649521088411411,
+            -0.007283003380127847,
+            0.01299542385123007,
+            -5.3226845309105124e-05,
+            0.030477683597373687,
+            0.0074153455615142755,
+            0.004161881195438522,
+            -0.00679081048132963,
+            0.008354912600159931,
+            -0.001621776416108105,
+            0.004080191054849395,
+            0.02120524874226696,
+            0.012440848502230418,
+            -0.00218041171518331,
+            0.006458415098595872,
+            0.005678055538564355,
+            -0.017186537377121235,
+            0.02507960402044456,
+            0.024678178106806115,
+            -0.008205288425409454,
+            0.004108402466376321,
+            0.005783335302169668,
+            -0.008846663924040241,
+            -0.006897707231869481,
+            0.006527795262026734,
+            -0.018474181903007085,
+            -0.0008721381678008483,
+            0.010598084358464908,
+            -0.02255441909004843,
+            -0.004840511019512733,
+            0.021745651362109403,
+            0.004631567631335321,
+            -0.018558972627958834,
+            0.02764788406651861,
+            -0.004942361238163609,
+            -0.0027290999934532494,
+            -0.010876135202650048,
+            0.01671497143541799,
+            -0.012173527207375091,
+            0.008443317346655462,
+            0.011021541861818043,
+            -0.006295639322287843,
+            0.003083685676540999,
+            0.008277206401503182,
+            -0.004504302913715969,
+            -0.0068075424927125676,
+            -0.0010508416962215848,
+            0.010167160518175114,
+            0.0036980815941666336,
+            -0.022704963865525812,
+            -0.006020073261002279,
+            0.02934227992246887,
+            -0.015082922782242507,
+            -0.0050916398557337705,
+            0.005376294716056329,
+            0.007709342013963906,
+            -0.003476236561444464,
+            -0.0023399991233097337,
+            -0.00787486645740744,
+            0.010356414979428607,
+            -0.015950083845400274,
+            0.02076360898078935,
+            0.025608263494550438,
+            0.024504236990840842,
+            -0.014814235990238182,
+            0.010828584133720797,
+            -0.011703992394890898,
+            -0.011047470905668532,
+            -0.015542835222173864,
+            -0.001658296303013357,
+            0.001970208852236451,
+            0.013107754130616248,
+            0.009666887546609378,
+            0.0235853683657408,
+            0.006473544089109194,
+            0.008973300312205858,
+            0.014118601565410305,
+            -0.003902202854013708,
+            -0.018303438244719996,
+            -0.014584226585460755,
+            -0.011977613285028946,
+            -0.01756345491020528,
+            0.0004563602312336091,
+            0.020461659996271687,
+            0.012417744207683839,
+            0.024136626592375227,
+            0.002119213701333967,
+            0.013759902247467922,
+            -2.9525392347047554e-05,
+            -0.000892319792935443,
+            0.00045783825646242434,
+            -0.005238483120312843,
+            0.019255756091120846,
+            -0.004599752813817219,
+            0.01656589522110913,
+            0.0006443773300684925,
+            0.007133940816771615,
+            0.013554761340256566,
+            -0.01704388669741371,
+            -0.003443100892108368,
+            -0.010029620309829285,
+            0.000941391966877521,
+            -0.019633111750884034,
+            0.01934571074127927,
+            0.010273330977233852,
+            0.0017121345993533963,
+            -0.018327189396014198,
+            -0.0029706385099949693,
+            0.03448635131998772,
+            0.019458999453011265,
+            0.0050734354363614525,
+            0.02632443373898792,
+            0.013018762352749236,
+            0.014443611411502168,
+            0.010153370950346075,
+            0.02356766253713842,
+            -0.023479811300626968,
+            0.02095255383200973,
+            -0.011590347850340786,
+            -0.007414136601134069,
+            -0.0035774888741646705,
+            0.0025141508183687115,
+            -0.013588691733097813,
+            0.00982877622088977,
+            -0.0017916539399655826,
+            -0.025386211797192578,
+            0.009164316443206016,
+            -0.008811052979979271,
+            0.010464442616928296,
+            -0.007774569252594923,
+            -0.0002806136360286729,
+            -0.0127084026190163,
+            -0.015184317235990752,
+            0.001259201880504347,
+            -0.009158429132281081,
+            0.023908150915412617,
+            -0.021210848953875233,
+            -0.01321384014436847,
+            0.013997502713857512,
+            -0.006869443694669701,
+            -0.011364239609894453,
+            -0.00374449980963666,
+            0.023673050288879224,
+            -0.009243341446512421,
+            -0.01359367519009039,
+            0.014464166953530989,
+            0.003834735113400469,
+            0.0059713317014342395,
+            -0.007508862694272452,
+            -0.008428705474135382,
+            -0.003280788036976103,
+            0.005998840043285335,
+            0.026615077617284676,
+            -0.014059930972065399,
+            -0.0032833264512892022,
+            -0.009189999799498163,
+            0.004118204537632102,
+            -0.02359889635281631,
+            0.02771075351384059,
+            -0.008871395303200375,
+            0.021705177758560575,
+            -0.0015924449308442096,
+            -0.006239188158271628,
+            -0.024231036074349822,
+            0.027522945599928003,
+            0.003683870966132199,
+            -0.010221180899173243,
+            0.0012768725460662862,
+            -0.052005292736885915,
+            0.016636109796947025,
+            0.008429895913163985,
+            -0.019507994429834353,
+            0.003697519981610208,
+            -0.004773584856005853,
+            0.018983847580558853,
+            -0.022966993228850697,
+            -0.014078986007109447,
+            0.008283766047449966,
+            -0.01153285124362746,
+            0.0031439180465649996,
+            -0.02527749501821793,
+            -0.005754537200570427,
+            0.0011760345167229269,
+            -0.005276936074280004,
+            -0.001705933750221904,
+            -0.020022761608132548,
+            -0.025972227850795115,
+            0.017666017379157135,
+            0.015820875579600418,
+            -0.002278262448951789,
+            0.01320818011561616,
+            0.008709899932124187,
+            0.00836573261091119,
+            0.016034263394940008,
+            0.007124509234103267,
+            0.0081473711608784,
+            0.045498533239850446,
+            -0.019846323044145485,
+            0.007371629414298193,
+            -0.0014854149174929534,
+            0.013951741898629716,
+            -0.030467924817555014,
+            -0.0055144480998344705,
+            0.0062802744565489945,
+            -0.0049373689340857915,
+            0.03510454786147091,
+            -0.017326094216003073,
+            0.01694160526735264,
+            -0.0026075809870445238,
+            0.0007420394265014909,
+            -0.02042003877666893,
+            -0.026156937726134134,
+            -0.014294938857971834,
+            0.0011857406889443987,
+            -0.0010047941261608975,
+            -0.011946583993069242,
+            0.006797001710574735,
+            0.004917161312925866,
+            -0.0020713379451245515,
+            0.015652523103715316,
+            -0.02952061566240878,
+            0.007356800041369564,
+            -0.010733578620243498,
+            0.020008504830359524,
+            -0.00990331048284697,
+            -0.0013319400098308093,
+            0.007236960885181047,
+            -0.004122868383988748,
+            0.013216043472071927,
+            -0.0038337251451538677,
+            0.004227115653354611,
+            -0.005221321656766545,
+            -0.01794343569722877,
+            -0.0016435261495179767,
+            0.021429008432348098,
+            -0.008349684096540855,
+            0.031864497493526205,
+            -0.01264743878037828,
+            0.009174982784686633,
+            0.010692990337462762,
+            0.0006628831036334152,
+            -0.005286285851702483,
+            0.009052971814211658,
+            0.0021431970274545893,
+            -0.008690520448278274,
+            0.006934310986093342,
+            -0.018349449937310143,
+            0.008494451785507706,
+            0.03394632739506833,
+            0.0018511328780472266,
+            -0.0007519661670070526,
+            -0.007266663179613392,
+            -0.01335844201823745,
+            0.018540713029383066,
+            0.005488273906733518,
+            -0.002546531152114063,
+            -0.027536660869590543,
+            0.02007746621157705,
+            0.0012265798861291223,
+            -0.004755325069578531,
+            0.012466557901411086,
+            0.0020495867632654885,
+            0.019989531151046736,
+            -0.010171678639113316,
+            0.030757957899447096,
+            0.010050036220124393,
+            0.002959833051227505,
+            0.0038556928367700113,
+            -0.036655749074851536,
+            -0.01652152778069314,
+            -0.00019357502472740266,
+            0.014271581054865031,
+            -0.012430647252826173,
+            -0.022432580814844155,
+            0.0010346169306977334,
+            0.01148004051449691,
+            -0.0033898642762881046,
+            -0.009650665305271706,
+            0.01560757541276278,
+            -0.020342213902670575,
+            0.006701302394408393,
+            0.007693672045981921,
+            0.0287100214318104,
+            -0.0020735384554022153,
+            -0.0012324838849990056,
+            0.000626416803904792,
+            -0.0055669860801192935,
+            -0.0008037366126870824,
+            0.017251369734470945,
+            -0.013661316947002466,
+            0.0175926211217534,
+            -0.012735241490550407,
+            0.0064713656749838585,
+            -0.0021092378432037563,
+            -0.010629579860050812,
+            -0.020114807971202787,
+            0.0029609495764307357,
+            0.009378003163375899,
+            0.006920144698428806,
+            -0.010438984881449953,
+            0.010715409812532968,
+            0.012197673689706352,
+            -0.00753962861484439,
+            0.014519905132928041,
+            0.022173989895384192,
+            -0.007687503607783572,
+            0.013633321414009471,
+            -0.0041446334728745095,
+            -0.014056264816568229,
+            -0.009985463824440833,
+            0.023929750617160402,
+            -0.0005563806942667597,
+            0.008756997373618809,
+            0.00388125051641897,
+            0.0007856447663040696,
+            -0.011226530517316183,
+            -0.001653447552601681,
+            0.019294448360333633,
+            0.011831934675073843,
+            -0.0013888534822008192,
+            -0.0010525764356699885,
+            -0.0014566936525653407,
+            -0.009427557739483988,
+            0.02049188741459728,
+            -0.022030783927622997,
+            -0.0019604940727076573,
+            0.00461018148439991,
+            -0.022589816803875618,
+            0.0001863117949313752,
+            -0.024570217962989905,
+            -0.01569952241187498,
+            0.011763033781122343,
+            -0.0022530631823737554,
+            -0.016028810867854907,
+            -0.0003242701040942633,
+            7.424127301806556e-05,
+            -0.007815789940508527,
+            -0.004135500768181258,
+            0.006115909992703774,
+            -0.00034193217510010117,
+            -0.0032262189374874842,
+            0.013430154310506884,
+            -0.009996622196870008,
+            0.015431528911575157,
+            0.0067029980830614145,
+            -0.015702167721800483,
+            0.0021406610839557003,
+            0.018500219544264753,
+            -0.0058910087864916085,
+            0.00787021425357441,
+            -0.014681907249972176,
+            -0.0037495551894426303,
+            -0.005787661351944086,
+            0.006051351977821761,
+            0.024849195780885833,
+            0.012969434450643372,
+            0.01139340445077682,
+            0.017405361843075933,
+            0.004325000715054734,
+            -0.01572297405297993,
+            0.0007416798945675923,
+            0.01682727189828363,
+            0.010464036837245312,
+            0.009804326468113703,
+            0.0008975556667684365,
+            0.0043704860361487635,
+            -0.005142176468800175,
+            0.009317064072855551,
+            0.02915482449687158,
+            0.0013434689612997423,
+            0.008696141010760192,
+            0.017276730345147134,
+            0.027038741396131265,
+            -0.0035088997194497725,
+            -0.007469614319788499,
+            0.008739942310300076,
+            -0.009657991604173173,
+            -0.001081629174976152,
+            -0.017000351183542026,
+            -0.012310275736750868,
+            0.016510505259358398,
+            -0.006084627062230492,
+            0.02848913486719956,
+            -0.0010655806791907225,
+            -0.02026498848361482,
+            -0.01565505608194376,
+            0.001404549479440637,
+            -0.01362132895080652,
+            0.005287566256051299,
+            0.014187265685427415,
+            0.01917046513020831,
+            0.004687191682875024,
+            -0.022258337721085656,
+            0.037364962197399806,
+            0.01757108963349427,
+            0.019377870795760123,
+            -0.019422840506328382,
+            -0.006159311237412833,
+            -0.009077095160676276,
+            0.02116697012944498,
+            -0.00880692095488465,
+            -0.009875674856043954,
+            0.024014332004451508,
+            0.017812606845656484,
+            0.005679806286464216,
+            0.006736609904079586,
+            0.0020048663439319585,
+            0.0025414630253661444,
+            -0.015831593851012047,
+            0.017957902498542212,
+            -0.005858125698347523,
+            -0.0058068554681761945,
+            0.017529514573571298,
+            -0.005989581792993925,
+            -0.0030306021203732347,
+            0.00015839997806822045,
+            -0.019697052145531482,
+            0.011618112886207201,
+            0.008256007601576028,
+            -0.017229774429089857,
+            -0.0014757560402567304,
+            -0.0002404480326945295,
+            -0.014118694740134205,
+            0.02004885146995733,
+            0.01694893356262835,
+            -0.0013248884139405398,
+            -0.0051752815372013315,
+            0.008223868147393114,
+            0.007000679132597393,
+            0.005256194890343006,
+            -0.019496098876977206,
+            0.005230597149342019,
+            0.005103909294824105,
+            -0.0014716649591412774,
+            0.010274346323051634,
+            -0.009030594825985526,
+            0.033740380586893826,
+            -0.006505788197604522,
+            0.013065658941136721,
+            -0.0043634731684193745,
+            -0.0055529072007985125,
+            0.001816020676232658,
+            0.013054468478598695,
+            0.009753501030700457,
+            -0.00970446155762939,
+            -0.004172354962129865,
+            0.018697982092775737,
+            -0.011983156146516074,
+            0.0013300791598046338,
+            0.01564072865804035,
+            -0.004858814041438713,
+            0.004708235083775013,
+            0.008635204790345423,
+            -0.004489505237783172,
+            0.01801512858970922,
+            -0.016482572999490366,
+            0.0023953436660054026,
+            0.010694869684962088,
+            0.02390864159830096,
+            -0.019152187046410703,
+            0.021786596486708207,
+            -0.017951050280323393,
+            0.026538582634436314,
+            0.014484894489694262,
+            -0.0035222259457311444,
+            -0.01946200885147006,
+            -0.00850690913394181,
+            0.011654565129063417,
+            -0.021423803876241532,
+            -0.0007985948078290641,
+            -0.0007290222944686444,
+            -0.021367628543092133,
+            -0.011189364017231132,
+            -0.009961779129778331,
+            0.010701637745810138,
+            0.001918633898887013,
+            0.010380983868185359,
+            0.008002182827196827,
+            0.010784760574518087,
+            0.014405640442760822,
+            0.0008958107769384412,
+            -0.010743001445440941,
+            -0.023795589003655273,
+            0.016987708698679597,
+            0.00438463274382093,
+            0.0007862212695139549,
+            -0.01627468994515667,
+            0.011936651269737248,
+            0.013250643815768723,
+            -0.009385605548926286,
+            0.009121882469425302,
+            -0.03716901103204463,
+            -0.004552808410398574,
+            -0.0035704811925430366,
+            -0.0020309197365173475,
+            -0.00908678158053104,
+            0.01559445415882485,
+            -0.013682507836005851,
+            -0.0009590922904844632,
+            0.00011696985057536548,
+            -0.0074756526903548915,
+            -1.5467890432254378e-05,
+            0.020649928837031005,
+            0.008438785939908428,
+            -0.0037382105257999757,
+            0.010376213224654074,
+            0.008826031303751312,
+            -0.013900359449287676,
+            -0.01527474712786084,
+            0.019035366035884273,
+            -0.018776918681908793,
+            0.0063098418544296165,
+            0.01677916431403883,
+            0.020778813096283988,
+            -0.02891862501830075,
+            -0.0007231469961304435,
+            0.022093939332021185,
+            -0.0012807635799950273,
+            0.006370841109790851,
+            0.008781853138377092,
+            0.006754082980602544,
+            0.0029102159062355374,
+            -0.003711173531879644,
+            -4.843770941622283e-06,
+            -0.003940474588075707,
+            0.007155539022862352,
+            0.004366433868349658,
+            0.036453728071874844,
+            -0.013661129348165794,
+            0.002541308805735001,
+            -0.0058838191638835155,
+            -0.00347401304572066,
+            -0.00593469688967835,
+            0.004407665856950772,
+            -0.010055727994798024,
+            0.004330827357536078,
+            -0.001570861535288093,
+            -0.0003731129422346693,
+            0.004883969598300302,
+            -0.001701673379646408,
+            0.007036892564844683,
+            -0.011863650867472625,
+            -0.0003979727750011258,
+            -0.020174851412609312,
+            0.021117284557875267,
+            0.00011666551282792189,
+            0.003662462447326642,
+            0.03723741246290771,
+            0.00476064510662738,
+            -0.00238355131307445,
+            -0.012880057949435798,
+            -0.013551338766321525,
+            -0.012986444917747435,
+            -0.020703821629987552,
+            -0.0036665673548525388,
+            -0.021006311026247164,
+            -0.0052061406234528915,
+            0.010362862594785442,
+            -0.011540184420787572,
+            -0.004023659825622843,
+            -0.015881589631889473,
+            -0.006531981517815174,
+            -0.03114211633687585,
+            -0.01124607488650473,
+            0.007493228366452009,
+            -0.019149285949049886,
+            -0.0039932870926865805,
+            0.026850121590606017,
+            0.001095360635760988,
+            0.01387696424607248,
+            0.0038058248676879625,
+            0.008117686785954085,
+            -0.0045937472017383955,
+            -0.0005669025036141417,
+            -0.01928244877716143,
+            0.003391980330529026,
+            0.008525354854576925,
+            -0.00361507909530282,
+            0.0070008404454106905,
+            -0.008835727374865623,
+            -0.016950017602980382,
+            0.0021654956715219517,
+            0.009334379529385142,
+            -0.006639140003365139,
+            -0.017587472740268683,
+            -0.011049929670847685,
+            0.010401189523318744,
+            -0.02283429470064187,
+            0.006257145380266984,
+            0.016439581169238607,
+            -0.009363030375624585,
+            0.004651575425738153,
+            0.019809183749755425,
+            -0.011810674576781702,
+            -0.005811328501528397,
+            0.008471992070998505,
+            0.013891259999721588,
+            0.0095606134736748,
+            0.02954498207274139,
+            0.0026596809804008996,
+            0.000561870602466584,
+            0.004889263753490895,
+            -0.004277061142504552,
+            -0.017423200644235665,
+            -0.006243787897080996,
+            0.0038507494787521856,
+            0.012873996671952184,
+            -0.009879417960879032,
+            -0.00048575277472139055,
+            0.01682604801673206,
+            0.004530939990187663,
+            -0.0033901248079550284,
+            -0.011868399508738274,
+            -0.020080331108085103,
+            0.0015580259874809898,
+            -0.01362200460317948,
+            0.015354347040143093,
+            0.003390601069331352,
+            0.0015284610287850575,
+            -0.004239880336806429,
+            0.01828645373058079,
+            -0.005013418527935815,
+            0.012749833447770729,
+            -0.006148890994661865,
+            -0.004587831231448011,
+            0.007986533783001116,
+            0.009166578151006778,
+            0.0004939857995863664,
+            0.05421019830078428,
+            0.002838056604877719,
+            0.01333205648597435,
+            0.013419127695838135,
+            -0.014523733369484934,
+            0.014086525147734823,
+            0.012688569853552613,
+            0.018497344547821693,
+            0.013037819487160016,
+            -0.012353776077273462,
+            0.017354801119939964,
+            0.0024084723134891487,
+            0.003909681065813095,
+            0.0022202438064449653,
+            -0.004528576079628071,
+            0.019594683950225115,
+            0.004070222328547378,
+            -0.0074362443729372105,
+            0.004453618452623641,
+            0.005007543168224267,
+            -0.013105524222311019,
+            -0.0112958915514782,
+            0.027105208848863218,
+            -0.01219417557128091,
+            -0.018749971743148304,
+            0.006706594269933445,
+            -0.02008458683853808,
+            -0.03763686535076864,
+            -0.0021332074289244973,
+            0.009992361098878196,
+            0.002795183524495193,
+            -0.00253695604043486,
+            0.01614298064601032,
+            0.016730443047719683,
+            -0.016985296937992726,
+            -0.018245902728910347,
+            0.0025257780765481825,
+            0.0016333325293570182,
+            0.01861807236979066,
+            -0.023168663546030196,
+            -0.006378864850313035,
+            -1.3795465857380962e-05,
+            -0.006829008132963306,
+            0.004850306884033312,
+            0.004703280112191218,
+            -0.0071714759384722965,
+            -0.0027773519096287217,
+            0.002256846645341137,
+            0.01975383140755998,
+            0.013198754584066554,
+            0.01572971337890945,
+            0.002600741959577911,
+            -0.005033508062449482,
+            -0.016817359058724685,
+            -0.005450011991586895,
+            -0.013478330855299777,
+            -0.009091854345715177,
+            -0.003042052092498012,
+            0.008217376329142566,
+            -0.025053469059115735,
+            -0.011965083373220982,
+            0.0119294363235832,
+            0.00391377331459168,
+            0.0003099203772539202,
+            0.007607672728136279,
+            -0.003830132495364688,
+            0.010331053246451333,
+            0.013823890473448383,
+            0.0076112887952959766,
+            -0.004723377809559611,
+            0.009600411526173948,
+            0.011460781436250457,
+            0.006015477068316837,
+            -0.005072711638895421,
+            0.023859899048048694,
+            0.01821920335673539,
+            -0.01680365156728093,
+            -0.002149107383573353,
+            -0.0027829676356423997,
+            -0.01884308738652242,
+            0.009665830801645276,
+            0.00880322398336952,
+            0.008308268605449346,
+            0.003906369351868086,
+            0.00974531700889854,
+            -0.013875904819275925,
+            -0.0050570811397453235,
+            0.002870120865445989,
+            -0.005729556771404781,
+            -0.01901337075686543,
+            0.000297242372701423,
+            -0.011987192261104132,
+            -0.011327157550660895,
+            -0.01409710922165158,
+            -0.003221240464411861,
+            0.005283472511900512,
+            0.021472187640986972,
+            -0.004410881709548286,
+            -0.001855852749039012,
+            0.01351697599517034,
+            0.010060125186506398,
+            0.0021658738823899905,
+            0.0002206474541239167,
+            0.005418828324605768,
+            -0.0027919130944783397,
+            0.0008094042430809068,
+            -0.0015826797905407809,
+            0.021089276765434876,
+            -0.02016611086095107,
+            -0.009413977569213423,
+            -0.009288078098692112,
+            0.005842197094339927,
+            0.028793032462603636,
+            -0.007165634502373454,
+            -0.010104002083017078,
+            -0.01994315293556678,
+            0.0003786527922888834,
+            -0.07810609175343784,
+            -0.005210636756258548,
+            0.001870513618910238,
+            0.03126380113105983,
+            -0.010817888476436834,
+            0.0033673768058629823,
+            -0.002204487854177595,
+            0.014542299655712928,
+            -0.001760107965206874,
+            -0.0013283746982194259,
+            0.0071954806737858056,
+            -0.005640400818010697,
+            -0.006210989097047217,
+            0.005904163661619133,
+            -9.945941028848348e-05,
+            -0.010266938705916303,
+            0.00057263080955709,
+            0.02209498395848068,
+            0.0159398346608872,
+            -0.006575439759804175,
+            0.01816748552021211,
+            -0.00719294553484007,
+            -0.009838738592324234,
+            0.005639288780502127,
+            -0.0019007683459126327,
+            -0.015369619176147573,
+            -0.011475977270803784,
+            0.009862808084641291,
+            -0.030069890644150284,
+            -0.00010725753918890802,
+            0.00943698424646706,
+            -0.013540245857270492,
+            -0.008293042205008774,
+            0.007212974960781723,
+            0.006708343474532583,
+            -0.0012017968151025369,
+            -0.0078026000239507145,
+            -0.02527526731750743,
+            0.0027578400032925625,
+            -0.002446811971857853,
+            -0.004899702349784173,
+            0.0019960465173086183,
+            0.001479503476855694,
+            -0.012645905858466798,
+            0.01216748847452763,
+            -0.013378877368040743,
+            -0.012053251571418766,
+            0.012508842412556306,
+            -0.00027167241745418584,
+            0.01682680930386839,
+            -0.020607299598265426,
+            0.0010702151063620052,
+            -0.028075439824932304,
+            -0.007671848153932823,
+            -0.015486630894165054,
+            -0.0009776136330459065,
+            -0.0012939921569540803,
+            -0.03481880531471923,
+            0.012889447665794033,
+            -0.005200149829198305,
+            -0.006094702500154477,
+            -0.00459778303079732,
+            -0.024583504817899775,
+            0.006072281238752663,
+            0.00607350167867964,
+            -0.006506508970816635,
+            -0.016816439635445445,
+            0.008428048142588516,
+            -0.0011040593013238872,
+            0.004873670535200424,
+            -0.007475800310154908,
+            -0.0005638920132076605,
+            -0.013047135093134946,
+            -0.005463454269764494,
+            -0.0008522562324122834,
+            0.01827123304871433,
+            0.0044055851550317835,
+            -0.014526381222031981,
+            0.016862445236481526,
+            0.013250692156820124,
+            0.01602714293656341,
+            -0.0071089293427753115,
+            -0.0004265739315250755,
+            -0.02750630108787212,
+            -0.00327225092954903,
+            -0.023687644154808972,
+            0.00041213204365507744,
+            0.009447660230507965,
+            0.0203905264889617,
+            0.012694421477479161,
+            0.004620617778996077,
+            -0.001557511764988156,
+            0.0066751090724502,
+            0.02696959560423252,
+            -0.01099183126742224,
+            -0.006969111598423376,
+            -0.01884504192789845,
+            0.01113023717919988,
+            0.025135748541199255,
+            0.003381303139273845,
+            -0.011317957771935042,
+            0.008112598754655146,
+            -0.0009247845526290428,
+            -0.010302980598863016,
+            0.02889587382004902,
+            -0.0391134481322538,
+            -0.015599591108049343,
+            -0.004605824192962023,
+            0.007406520608110251,
+            -0.015596917115435033,
+            0.00770468362805995,
+            -0.012614339656105264,
+            0.001699049329817941,
+            -0.005851477164035241,
+            0.03535087340659861,
+            -0.0031950960727572925,
+            0.0006909856233673865,
+            -0.003395233050301427,
+            0.003531135044131077,
+            0.012355948349937309,
+            -0.004444221317023453,
+            -0.030726300006909638,
+            -0.0022983766869702346,
+            -0.011198252640110788,
+            0.009274666307515955,
+            -0.016207714291658314,
+            -0.017342435976846468,
+            -0.009424737744066259,
+            -0.021827243671837953,
+            -0.012403084982789827,
+            0.029747453318486338,
+            -0.005824001042844527,
+            0.01747569398142269,
+            -0.005219060548639168,
+            -0.008365003057800934,
+            0.00021272411948388247,
+            0.01344335658390732,
+            -0.028498240040284714,
+            0.00041963499785238845,
+            -0.002322931702641349,
+            0.0126393550661767,
+            -0.001595134597082556,
+            -0.00983073906977773,
+            0.006903313245694627,
+            0.0010522208685011098,
+            -0.015648940549713764,
+            -0.0007573245570947632,
+            -0.008533504878052644,
+            0.020475329445149498,
+            0.00027791387213409814,
+            0.001074213629646005,
+            0.013864331671732542,
+            -0.02102352993295736,
+            0.01576938071930521,
+            -0.021722775924283996,
+            0.01893267442161547,
+            -0.006577103101038035,
+            0.005772016664149475,
+            0.008283223343661555,
+            0.014068789332250016,
+            -0.014030042849491258,
+            -0.00663752723080222,
+            -0.034436220972634726,
+            -0.0122621879114314,
+            -0.001946072976826902,
+            -0.005308941654794638,
+            -0.014773123927909993,
+            -0.0011054822266082056,
+            -0.026205461509573584,
+            -0.006460373065662098,
+            0.012822188046808264,
+            0.019900312098359022,
+            -0.008409070094265829,
+            0.014163132434889265,
+            -0.006743711872055091,
+            -0.0028230771810882492,
+            -0.012652021749331811,
+            -0.018061180380009395,
+            -0.01083672324723357,
+            -0.01890966636827258,
+            -0.0018635408070944168,
+            0.004709222372611061,
+            -0.015760618199235518,
+            0.011435131737642627,
+            0.03198874888333405,
+            -0.01567271472650171,
+            0.008243406188694909,
+            0.016411496328333205,
+            0.0034236971031973503,
+            0.011728079139322837,
+            0.0038311415233052795,
+            0.004621789116267879,
+            -0.011109450058055078,
+            -0.011473158133621661,
+            0.006524490579223986,
+            0.00954511918454185,
+            -0.004913142601134428,
+            0.024347072255007807,
+            0.0008745419206472714,
+            0.005045794664611216,
+            0.005304506902446091,
+            0.011488187043525824,
+            -0.013135271908307747,
+            0.019746484188715552,
+            -0.002055210086078267,
+            -0.015571665869889405,
+            -0.01535499169234956,
+            -0.01105419264826655,
+            -0.018626399757822328,
+            0.0017108083455869296,
+            0.0181439079855921,
+            -0.0012493542718277685,
+            0.012137753522284387,
+            -0.00924460959150499,
+            -0.005019437549649743,
+            -0.0027156377206998474,
+            -0.038211954658821295,
+            -0.0021878970332906457,
+            -0.018809487178512628,
+            -0.02498945889484935,
+            0.03505216568400238,
+            -4.26477081213836e-05,
+            0.00997643255528116,
+            -0.010310197159232386,
+            -0.0002454902098986129,
+            -0.011065403358802515,
+            0.014954998647970473,
+            0.004624123403100598,
+            0.008872923678983478,
+            0.008560280683876702,
+            -0.016419500515801093,
+            -0.002997557798375716,
+            0.007472390947008284,
+            -0.00374822278027057,
+            0.012680407710694952,
+            0.015353684671120731,
+            0.020886474047003733,
+            0.011161262397502876,
+            0.002045862158732102,
+            0.00887271411363814,
+            0.009563361150297699,
+            -0.0003513237086717377,
+            -0.010459621030424415,
+            0.015419039131459582,
+            -0.02204848739950703,
+            0.012172257374410352,
+            0.04346634265794738,
+            0.019197433209794286,
+            0.029488039318675732,
+            -0.02272810439246353,
+            0.01448092574459596,
+            -0.003801043409441206,
+            0.009476229966122152,
+            0.0049309968883087685,
+            0.02346191840355209,
+            0.003973840862678076,
+            0.0073546443568171405,
+            0.010647581332452525,
+            -0.00972051412996416,
+            0.025644022498404723,
+            -0.00846669916801108,
+            5.1518540791212095e-05,
+            -0.004407265804145216,
+            -0.026400029105519137,
+            0.01615193008014211,
+            0.0081722536401398,
+            0.0010702314763197463,
+            0.0010263623479474385,
+            -0.0019566604229607643,
+            0.007982949510899019,
+            0.007079322685771456,
+            -0.015274148711633536,
+            -0.003886053737590683,
+            -0.018527455865296014,
+            0.0012864569064749438,
+            -0.024337133883771334,
+            0.010466651250597847,
+            -0.017568706814132484,
+            -0.014822354159709854,
+            -0.021666681078831483,
+            -0.013232016867264097,
+            -0.02011653224168991,
+            0.0035858355517491515,
+            -0.008632755352062642,
+            0.013025728891203198,
+            -0.016962939287139683,
+            -0.021494971929779273,
+            0.008430887589398292,
+            0.00870802359711456,
+            0.007481282677955534,
+            0.009728368603458082,
+            0.014996157358749115,
+            0.01736292711410622,
+            -0.022712386143092265,
+            -0.023163001611599246,
+            -0.006682209717364464,
+            -0.003531495447167166,
+            0.006762379833125297,
+            0.00933681196321128,
+            0.012884185500343249,
+            -0.007077895849501418,
+            -0.007554762054641417,
+            -0.013283067570021503,
+            0.006197966297581871,
+            -0.003993067576677973,
+            -0.009464789349849612,
+            -0.0015662739825834614,
+            -0.0061389795593180686,
+            0.002753664732282926,
+            -0.0001791088888074743,
+            0.0007595261237141144,
+            0.017122834676182343,
+            -0.008176319370751018,
+            -0.009787747261447871,
+            -0.02027228453943018,
+            -0.010683273300243025,
+            -0.015604745521156111,
+            -0.0016279406232527138,
+            -0.006506713191861036,
+            -0.014420323628589382,
+            0.010888206569219648,
+            -0.011930708571093279,
+            -0.00887210117616784,
+            0.010807147705360856,
+            0.0073900910661282935,
+            -0.003431679006289002,
+            0.037804020313161775,
+            0.0011945597874820288,
+            0.00778778394048267,
+            -0.003366543306746973,
+            0.005296727102519836,
+            -0.017761251541598876,
+            -0.026382400396890007,
+            0.0010357716464853195,
+            -0.01775777668381489,
+            0.010807713750049491,
+            -0.008515368407459745,
+            0.0016051389544388124,
+            -0.00034958600424912995,
+            -0.009585854918521076,
+            0.0036549720706351473,
+            -0.008110927907950148,
+            0.004212219431447508,
+            -0.03507890639477636,
+            0.011409377585617926,
+            -0.01819120811524444,
+            -0.008962242411035718,
+            0.01710177594046791,
+            0.037560936862121835,
+            0.0028513388998398104,
+            0.00030058353811717895,
+            0.0014792184333546447,
+            0.004427921639326168,
+            0.021438127785303025,
+            0.006339483221420334,
+            -0.001187747087595264,
+            -0.02237982599829809,
+            0.0062435650294483695,
+            0.002200578793697352,
+            0.010092828474934744,
+            0.023105023225115266,
+            -0.016797326021070008,
+            -0.015378520287395565,
+            -0.012732527281000097,
+            -0.010797463778182385,
+            0.009881499418260473,
+            0.02801767062785128,
+            0.01811139137849365,
+            -0.017358098300028903,
+            -0.020207173948076473,
+            0.0016242901745373126,
+            -0.026331540602868762,
+            0.007269751661479859,
+            0.11940631114242901,
+            -0.0045382135734049305,
+            -0.005994407567723939,
+            -0.008679712231646732,
+            -0.004482553208064535,
+            0.016709571294385293,
+            -0.013387043582971358,
+            0.069179848974036,
+            -0.007221545718220735,
+            0.0037392229364642915,
+            0.0028916224964057385,
+            0.00462729158956839,
+            -0.009727339449635306,
+            0.011570145487622916,
+            -0.011287365791681587,
+            -0.012991994493845187,
+            -0.013076194601468383,
+            -0.0180830261130804,
+            -0.007272768410218907,
+            -0.004155564814758654,
+            0.004204993170487392,
+            -0.005525575983182816,
+            0.008051108079520902,
+            -0.017551645696802648,
+            -0.010800070994485792,
+            0.013007176804761548,
+            -0.0023627485852619817,
+            0.005390986239687555,
+            -0.029668390278675554,
+            -0.0007650391259819867,
+            -0.0002446394046947245,
+            0.021280292985495608,
+            0.003137771886236958,
+            -0.0019077298815862946,
+            -0.010810159675564049,
+            0.007040107942856957,
+            0.023835993694726953,
+            0.021875366566457236,
+            0.0180511566130312,
+            0.012135351541044354,
+            0.020483975362807574,
+            -0.008667079096167644,
+            0.026324312888512866,
+            -0.001552207146726937,
+            -0.00397810465151493,
+            0.0077577136352257004,
+            -0.009849600842081875,
+            -0.00816392268697345,
+            0.00677353736462267,
+            -0.017288900282504148,
+            -0.020418071729964713,
+            -0.020383762873109595,
+            0.025741408097358233,
+            -0.011755094311186617,
+            0.002737129201159793,
+            -0.008266210469668866,
+            0.027287436144516414,
+            -0.006283892307996516,
+            -0.0029440642954701632,
+            0.009428876742171362,
+            -0.03788574758692254,
+            0.008834605607057885,
+            0.006463342392514337,
+            0.026711196829464153,
+            -3.9827332346130654e-05,
+            0.005565561841469247,
+            -0.0006738085601962865,
+            -0.006104320995853506,
+            -0.010792830169647851,
+            2.4565453208679464e-05,
+            0.017983908748172493,
+            0.0035181716362611077,
+            0.016970206528324653,
+            -0.005449808667944701,
+            -0.023171388778833197,
+            0.007092509502738752,
+            -0.01555320330471131,
+            -0.023306285898839978,
+            -0.002028783456704567,
+            -0.03196436161255752,
+            0.0033412618908711163,
+            0.007527633574188688,
+            -0.01890570959143866,
+            -0.017859735533232564,
+            0.010329340599904344,
+            0.004382396539605985,
+            -0.0016746397858953155,
+            0.006643785064338525,
+            -0.03307812909454471,
+            -0.01927475592253288,
+            0.004686970010943116,
+            0.012733525168279355,
+            -0.012993441126970916,
+            0.015291832952199082,
+            -0.020491167895268936,
+            0.016451976170508816,
+            -0.012259387189535237,
+            0.018393274368854132,
+            -0.006997668664226754,
+            0.011843450716200625,
+            0.011457415642860215,
+            0.009560877634897626,
+            -0.01790001115302499,
+            0.0214879415944361,
+            0.0066829070506954115,
+            0.03900547021123392,
+            0.02512484499576633,
+            -0.012651294056575184,
+            -0.0018425680226742782,
+            -0.00011361458320146007,
+            0.014982647266052349,
+            0.0061874424115559,
+            -0.003689464024837091,
+            0.02705902205734107,
+            0.018239469373522537,
+            -0.021070704644859155,
+            0.020037508334843183,
+            -0.00973804771680803,
+            0.018876283979591423,
+            -0.013303080480396818,
+            0.010592424494628194,
+            0.004717870260467274,
+            0.01070131238869639,
+            0.009087656015197565,
+            0.006708673669681232,
+            0.025928065153829043,
+            0.03663367449576964,
+            -0.00511073108390551,
+            0.002161381031392467,
+            0.004864092591944181,
+            0.012814518637357098,
+            0.001556833001747549,
+            0.007915766323215207,
+            -0.007012785514562176,
+            0.005834786334750152,
+            -0.002311533032619821,
+            -0.00122162928193159,
+            0.019525999280692766,
+            0.000824163362963678,
+            0.012875575199417254,
+            0.013259376059185821,
+            -0.002390133687709065,
+            -0.005682644446302278,
+            0.0019262218417782657,
+            -0.018193478045923098,
+            -0.015131696847666434,
+            0.029463723011600933,
+            -0.011447039651934148,
+            -0.008083114389964213,
+            0.0020642857121204753,
+            0.012160202510990566,
+            -0.00866944494225225,
+            -0.005378300635997713,
+            0.0038280364584418335,
+            0.014427318067525318,
+            -0.01799358399064332,
+            0.003381165206832161,
+            0.005252552821854107,
+            0.019433554824232185,
+            -0.00028791985674147885,
+            -0.013487678738391956,
+            0.0005317922815671783,
+            0.014982152801567637,
+            0.011902348165369078,
+            0.0016056390299304837,
+            -0.006935777197137747,
+            -0.00446584737370504,
+            0.004586814866182051,
+            0.006269466956110759,
+            0.0027516492952993445,
+            0.008157662045793032,
+            -0.0025001885993292637,
+            -0.016648747436203294,
+            0.017735757469839637,
+            -0.0030669873770852725,
+            0.018058248627079793,
+            -0.0038238312452176745,
+            -0.0147172470474118,
+            0.0043328417868095355,
+            -0.001888507803054499,
+            -0.018556943283494755,
+            0.009407930200720534,
+            0.002560949424273885,
+            0.010411159836451958,
+            0.0052571230238294825,
+            -0.007499470181994448,
+            -6.145889479758644e-05,
+            -0.015163819825329975,
+            0.022836682800304944,
+            0.016390944816242627,
+            0.001713572907204073,
+            -0.0035687505973373878,
+            -0.031210299824387475,
+            0.0029076511444359886,
+            -0.007822231980296341,
+            0.017821806420465356,
+            0.008250605126658402,
+            0.007908773724009905,
+            0.008351355849444162,
+            -0.00743970642051451,
+            -0.002411981861500092,
+            0.012543476434788293,
+            0.0202407777715137,
+            -0.0024635818448627447,
+            0.02535771413579989,
+            -0.012533449594824386,
+            -0.0022489292647974992,
+            0.0012362917358624392,
+            0.01578939075186126,
+            -0.0008001783766371555,
+            0.006311837812042848
+          ],
+          "keyphrases": [
+            "Switch Transformers",
+            "Mixture of Experts",
+            "sparsely-activated model",
+            "training instability",
+            "multilingual settings"
+          ],
+          "title": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity"
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "addd7c61-bdd3-470e-8c4f-439af824f89e",
+        "properties": {
+          "page_content": "F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+          "embedding": [
+            0.0002445570513886185,
+            -0.025035678422212698,
+            -0.026056660824720954,
+            0.0021234386221176942,
+            -0.007526403638487191,
+            0.018963965001103752,
+            -0.007728070932730228,
+            0.062213823247851435,
+            -0.00501600735659763,
+            0.031461940272958254,
+            0.014585783857839997,
+            0.010118033107401152,
+            0.009983318452435566,
+            -0.03282616973330728,
+            0.015192449242879527,
+            0.01480596701302375,
+            0.019405922018728892,
+            0.022192910412596127,
+            0.008048967251594976,
+            -0.0367043822293015,
+            0.017737679976556845,
+            -0.022757028540067756,
+            -0.06290354368907101,
+            0.016842087557557506,
+            -0.0050319451760917995,
+            0.019311667659179976,
+            0.008548884527537198,
+            0.02344214718766695,
+            -0.042417710048062586,
+            0.01950972309054391,
+            0.021637767293446648,
+            0.010581067128107188,
+            -0.014214247204390872,
+            0.01016365063627519,
+            -0.025782209417014867,
+            -0.02611397356963557,
+            0.034251067799009986,
+            0.0071402676991737055,
+            -0.020147891665364803,
+            0.056472902950962155,
+            -0.006027534100459245,
+            0.005473493744726429,
+            -0.03139313966858068,
+            0.00043634965151753214,
+            0.013171990459657583,
+            -0.016474119463665594,
+            0.0075792906007691105,
+            -0.0032442837467945356,
+            0.02492845005795393,
+            -0.04366675539501486,
+            -0.006033838756045725,
+            -0.018660330752604155,
+            0.02088611082654532,
+            -0.019387265112916365,
+            -0.01222746328548615,
+            0.005977009213829137,
+            0.0012690574665308455,
+            0.05430319716489232,
+            0.058075720286678886,
+            0.02554562736016709,
+            0.02997758641056458,
+            0.00014178556329408164,
+            0.004139467932725944,
+            -0.004454862143349584,
+            0.031094774362479788,
+            0.0017837745263941452,
+            -0.013542834385584962,
+            0.01983846902748998,
+            0.010136662804503968,
+            -0.0387179389182337,
+            -0.06156798393873734,
+            0.015844092498186847,
+            -0.020820695961906926,
+            -0.02678200120240037,
+            -4.570339833017677e-05,
+            -0.015544616932001905,
+            0.005434882475939711,
+            0.050835936350804696,
+            -0.004985250023269435,
+            0.0016509410141492726,
+            0.022801366331672433,
+            -0.004080282248725238,
+            0.0078367492683837,
+            -0.00034876063727630653,
+            0.039772801289415724,
+            0.0288510926430687,
+            -0.022609202992981695,
+            -0.07091181813750308,
+            0.045194222785472735,
+            -0.03225547853169315,
+            -0.032585463285183205,
+            0.01670145480899693,
+            0.03286138231600811,
+            -0.009733112712376757,
+            -0.030197857075781466,
+            0.0731500616714208,
+            -0.005545147701735842,
+            0.017072915518860705,
+            -0.021002529816498382,
+            0.010525361666209598,
+            -0.012745643075817463,
+            0.025352505498532282,
+            -0.003102906957075941,
+            0.016391641977220053,
+            0.02534354270624064,
+            -0.008108690979345334,
+            -0.006451703723989414,
+            0.0065097623653779324,
+            0.04453924610212246,
+            0.0889545538715994,
+            0.019774398083206815,
+            -0.003981259900744729,
+            -0.046615274629371053,
+            0.00732791197790897,
+            -0.02916202133853672,
+            -0.04303144138453202,
+            -0.04367640508681853,
+            -0.010910501188273782,
+            0.023755744360759886,
+            -0.003399898430475389,
+            0.03910227884474275,
+            0.04137187598162877,
+            0.03363334616934253,
+            0.00916121492790106,
+            -0.031686589382074935,
+            0.026907320660055866,
+            -0.01693827759642556,
+            0.002542422168436765,
+            0.00734783450287368,
+            0.008189080458799796,
+            -0.006352648760821389,
+            -0.002333605867785783,
+            -0.03831808898525347,
+            -0.036762698962852246,
+            0.012773386000855496,
+            -0.008349610778914014,
+            0.027936454763857345,
+            0.07015560257394027,
+            -0.03713244149832974,
+            -0.0066245405611366675,
+            0.02802618921440782,
+            0.0017169880395367697,
+            0.023346396749133396,
+            0.007605522879978981,
+            -0.023484167590494896,
+            0.008594274917753499,
+            -0.0071765874950466295,
+            0.0677537727828362,
+            -0.008126000179550069,
+            0.011046754551416857,
+            -0.019516727128214784,
+            0.031654663874166704,
+            0.0007057013357055938,
+            0.021535694192642306,
+            0.03313372731162823,
+            -0.005794425715079591,
+            -0.03865395674172617,
+            0.032271538528276474,
+            -0.029790294001121286,
+            0.0013647166052139265,
+            0.014194415049797637,
+            -0.004899479294176156,
+            -0.015920769574443834,
+            0.0013708196727143433,
+            0.029757298661028133,
+            0.045717552753231847,
+            0.042911032378569906,
+            0.0016019404554248433,
+            -0.06844613496660937,
+            0.06365820914938357,
+            0.012895947255192629,
+            0.03208498337942363,
+            0.009724409704032419,
+            0.042361427690196844,
+            0.008120380964731038,
+            -0.0004384537671641681,
+            -0.005540416382489673,
+            -0.005125920468958917,
+            -0.021186731704155592,
+            0.03258917022091572,
+            -0.01443327736487225,
+            -0.007107908179884635,
+            -0.00206346150761894,
+            0.014042865018508809,
+            0.028517557440008168,
+            0.017283745982594707,
+            0.0029564774390786045,
+            0.04378212980825259,
+            -0.019454732028746872,
+            -0.009437148551161395,
+            0.018820938021162286,
+            0.0008903198503990673,
+            0.025297040557527045,
+            -0.0011429799732923295,
+            -0.01670583390791835,
+            -0.009421860087826363,
+            -0.0328764813310764,
+            0.009563088287577815,
+            -0.0015347427243049344,
+            0.019228666906882602,
+            -0.007174098839673089,
+            -0.04068735475327182,
+            0.008903031576514094,
+            -0.011472954071988202,
+            0.015735403376626513,
+            -0.007084760348074423,
+            0.008729329609340102,
+            -0.00638454270428521,
+            0.016210007209816298,
+            -0.02460475066044075,
+            -0.009047036293783832,
+            -0.022598561172568143,
+            -0.020144302113939427,
+            -0.010886272544512816,
+            -0.010886770387412563,
+            0.008591903125597681,
+            -0.012216526799178626,
+            0.023956058440750026,
+            -0.017402060516065108,
+            -0.018328740623422317,
+            -0.00046940812181371226,
+            0.022921229116282273,
+            0.009026508358954174,
+            -0.00446208425714566,
+            -0.04421346673289319,
+            -0.01214251475197077,
+            -0.016860336137191526,
+            0.005985059770480569,
+            0.02804423359971541,
+            -0.03510963462663318,
+            0.012588036039507566,
+            -0.0018909139908325232,
+            0.0019847602212350957,
+            0.011374423039138836,
+            0.015486762163219968,
+            -0.027040748338117617,
+            -0.016329845569406535,
+            -0.012600824227928017,
+            -0.01765236203915568,
+            0.01673827406092187,
+            0.0016641690585652725,
+            0.003997011410859195,
+            -0.00499922800986537,
+            -0.02782499674212141,
+            -0.008774389172636544,
+            -0.011506812701874534,
+            -0.009783555199790996,
+            -0.015740884114025612,
+            0.013493654174963239,
+            0.014254624157312108,
+            -0.033270488877124596,
+            -0.018446763439664365,
+            0.02247061755496804,
+            -0.0035396859846893343,
+            -0.03759408156279411,
+            0.04544560120093284,
+            0.052893430652542234,
+            -0.02220682841881151,
+            0.002828355769475809,
+            -0.03469141962159207,
+            0.05950248906327849,
+            -0.03444371252359212,
+            -0.03378776392860449,
+            0.06521072243320349,
+            -0.009713026070923203,
+            0.015249223153809483,
+            0.03131206469454657,
+            0.0034865285730464384,
+            -0.0025997752599464445,
+            0.00912836988165194,
+            0.033031075123542926,
+            -0.014878258361522118,
+            -0.013101723481793925,
+            -0.01797681408884685,
+            -0.002397237505725697,
+            -0.03438738791694146,
+            -0.021713511748922477,
+            0.00836495266415799,
+            0.013064910474172053,
+            0.017110113503472666,
+            0.017514586931181946,
+            0.016195079102773242,
+            -0.0024109742901844617,
+            0.01545592172528996,
+            0.001046760697009891,
+            -8.863738247355372e-05,
+            -0.010979717992873124,
+            -0.017762086340346584,
+            -0.052667758929128256,
+            -0.018996338856872796,
+            -0.04421628171000429,
+            -0.005982494173649518,
+            0.0012420734363470585,
+            0.009886466834005627,
+            0.04949340056515024,
+            -0.01295176306623405,
+            0.003508142432105508,
+            0.010930632194984236,
+            0.01019261004578912,
+            -0.006556730312565122,
+            -0.032765993356945154,
+            -0.006485728779853869,
+            0.025709969056450885,
+            -0.022932446214570953,
+            -0.021792413124305354,
+            -0.007047777916806282,
+            0.0038659532636633486,
+            0.02321123092409853,
+            -0.026003302747277822,
+            0.017643847938685852,
+            0.010129447679208158,
+            0.009551458127273026,
+            -0.017747795509094742,
+            -0.02891499441428287,
+            -0.030321565228144824,
+            0.039350178167852384,
+            0.02899873253152933,
+            0.011338359735211436,
+            -0.033634061116901,
+            -0.013629902520109783,
+            0.0376352218933715,
+            -0.0019251015746424632,
+            -0.07267257827375968,
+            0.036685408572352814,
+            0.0050526642351482985,
+            -0.00019289056191322955,
+            -0.013581714265389013,
+            0.05300479823846402,
+            -0.04139717386779843,
+            -0.03335524761191713,
+            -0.00920987783097114,
+            -0.022616612319255443,
+            -0.005274093640024995,
+            -0.0012667939280239298,
+            0.032502321377997974,
+            -0.020089629343721754,
+            -0.024354491736044894,
+            -0.010206744478678104,
+            -0.02465036256948816,
+            0.017966455981144307,
+            -0.012995995315432267,
+            -0.02135780149071486,
+            0.010967719068923269,
+            -0.03402154331207373,
+            -2.0626689329697705e-05,
+            -0.01092309016851862,
+            -0.018683564965672157,
+            0.01571041043301007,
+            0.016259448102797114,
+            0.015403085986247105,
+            0.0020047641027155683,
+            0.015172380396260792,
+            0.008099191720720747,
+            0.004074377809743208,
+            0.01893667379047588,
+            -0.03384452762579117,
+            -0.038822883991989264,
+            0.050315952008617104,
+            0.0008458969404873289,
+            0.0012049589464131246,
+            0.006406225290437734,
+            -0.0042817559898658255,
+            -0.0411169553122232,
+            0.012933323148113837,
+            -0.05784927886369837,
+            0.006262727192870561,
+            0.012928199517490538,
+            -0.01786898028943474,
+            -0.02291183998443847,
+            0.00046953379111417065,
+            -0.024864892714864046,
+            0.02151263275001755,
+            -0.017247234755598105,
+            0.020405349906524557,
+            -0.023068655052252782,
+            -0.006872857342813033,
+            0.05842934123173711,
+            -0.02308642746390288,
+            -0.04925938088613823,
+            0.032829595473385986,
+            -0.007184802367095134,
+            0.03878513331161335,
+            0.055971017386540985,
+            0.016337316575743225,
+            0.006439501224944926,
+            0.026084366083114555,
+            0.013204436213767278,
+            -0.012028383848550473,
+            -0.0383295105300241,
+            0.015805232298071025,
+            0.030751727443425712,
+            -0.04685643586863969,
+            -0.010258856888449854,
+            -0.02541648783174773,
+            0.01599307149086869,
+            0.020046184897703115,
+            -0.0044598427609889935,
+            -0.0012270315730225135,
+            -0.009664326045898607,
+            -0.01520444928791987,
+            -0.02631191309701645,
+            0.02193635473056903,
+            -0.013740214173212537,
+            0.0007504924364311022,
+            -0.001111971939163715,
+            0.0012797636039507028,
+            0.008478066024469051,
+            0.01501219347281274,
+            0.027440349509054086,
+            -0.022728634568050663,
+            0.006916913421268421,
+            -0.008499438828073956,
+            -0.03788213665656746,
+            -0.006718262893819488,
+            0.029198375723488434,
+            0.006169843558767785,
+            -0.016427547826608675,
+            -0.0049009468913267205,
+            0.0022509753971959087,
+            -0.016307639896946134,
+            0.00372456949880952,
+            0.010347360137213142,
+            0.035374109073612686,
+            0.01856811791368892,
+            0.019247903053478954,
+            0.0035591273657627456,
+            0.010292179556783772,
+            0.004040044337368795,
+            0.025869815038962875,
+            -0.04526297785801751,
+            0.030173313035569235,
+            0.01521363974209499,
+            0.009059160353158267,
+            -0.03914219183565228,
+            -0.02367109515190293,
+            -0.025737979702884085,
+            -0.021654077593318197,
+            0.02346511753268162,
+            -0.02730833269330862,
+            -0.006289840626554075,
+            0.006531934540756005,
+            -0.0006804032161296996,
+            0.003928945341546481,
+            -0.0033962179480786624,
+            -0.037975452991752895,
+            -0.009840510213526213,
+            -0.04023861493405766,
+            -0.0241245373830213,
+            0.003822063151525241,
+            0.003249755356115233,
+            -0.06594891648887229,
+            -0.001768385222533634,
+            -0.06222620899240395,
+            -0.013594921028316139,
+            0.03441942883093182,
+            -0.024438842229617237,
+            0.04820486752403123,
+            -0.05486229161412509,
+            0.018219866569279347,
+            -0.046562594596398506,
+            0.011354841468658838,
+            0.021738373879790916,
+            0.04347115691988908,
+            -0.007572536447300832,
+            0.014626550834376365,
+            0.009906079088662038,
+            -0.04530647508884267,
+            -0.04520033458696902,
+            -0.011400230794947595,
+            0.011239630851440557,
+            0.003287355301854498,
+            -0.012304101523783504,
+            0.02623659451177029,
+            0.015903981896266153,
+            0.04300392046214319,
+            0.03414364576071245,
+            -0.0005691335103219632,
+            -0.049506948352089984,
+            -0.027336474868358992,
+            -0.014910043127825028,
+            0.02552137904202476,
+            -0.03055258097953423,
+            0.003401887357920115,
+            -0.03228200232009397,
+            0.013338682663233296,
+            0.018207379090305786,
+            -0.02224210534533673,
+            0.032569850137027664,
+            0.014446434357859929,
+            -0.040301668913492676,
+            0.0012399866793165108,
+            0.0002450513991178417,
+            -0.038300625347349834,
+            0.008119751344998158,
+            -0.013712452991626932,
+            -0.05381257202269606,
+            -0.022199278450525977,
+            -0.014717114803771204,
+            0.02206967996899982,
+            -0.0023470596291977116,
+            0.017222681546831476,
+            -0.029965631581610256,
+            0.0053103260868707786,
+            0.039279027237044066,
+            -0.001286692479487674,
+            -0.01148541970963179,
+            -0.0008706049086180766,
+            0.022795196067681996,
+            -0.01968201411368477,
+            -0.045534614872904305,
+            0.009877856016973188,
+            -0.020566606587457816,
+            0.016002665913994348,
+            -0.015445999456355206,
+            -0.0052940252527317235,
+            0.010360748741468617,
+            -0.03170100753615497,
+            0.02004511444257144,
+            0.009722541379665509,
+            0.006411072442536516,
+            -0.017460683536795665,
+            -0.0033423349847155888,
+            0.015743901543629382,
+            0.047005401368949463,
+            0.0007329338628268773,
+            0.006153613674047577,
+            0.02501606022399451,
+            -0.013054974002495664,
+            0.05036387123988784,
+            -0.02807526639433245,
+            -0.03099731580589117,
+            0.04344605263104773,
+            -0.019489740746149623,
+            -0.017636713524640496,
+            -0.017960525325193457,
+            -0.009792405274000215,
+            -0.0062801692104839674,
+            0.02801421653619339,
+            -0.00712206060099242,
+            -0.005749470909187103,
+            0.020504942834135256,
+            0.013170883165927672,
+            -0.011218763497707175,
+            0.008523945766715354,
+            0.01511514654097483,
+            0.027931571407773977,
+            0.03214582418497515,
+            -0.02345593941074465,
+            0.0333746849861944,
+            0.009324947617983663,
+            -0.004234113006464689,
+            0.005131158074842622,
+            0.0020437263732432953,
+            0.00176803337042029,
+            -0.029372739581870692,
+            -0.01539406767336474,
+            0.015958225909710817,
+            0.007396588215584237,
+            -0.003291618431912445,
+            -0.0316834931316845,
+            -0.0028368107695447525,
+            0.02614797456533501,
+            0.003832488818435695,
+            0.026426519154286857,
+            -0.015605455255333021,
+            -0.02018445776455915,
+            0.008876565653043164,
+            0.023669947107607083,
+            -0.055401884612295824,
+            0.01882053182190406,
+            0.011046506385076857,
+            -2.6003594081234644e-05,
+            0.01591257026228522,
+            -0.0046432737555961385,
+            0.005094314300671912,
+            0.0482159962414822,
+            0.015126643838581288,
+            -0.013318945197936635,
+            0.0020418779929576307,
+            0.015136735547940301,
+            0.001727083579122356,
+            -0.020116332736417462,
+            0.002896210161586164,
+            0.012858057228188658,
+            0.0092589465483404,
+            -0.012984552574495295,
+            -0.005651788746934606,
+            -0.016856929103895944,
+            0.03249171052801259,
+            -0.005843029913406538,
+            0.02336833175997056,
+            0.04800276014595125,
+            0.02261889070736121,
+            -0.03578655846417886,
+            -0.011451513080556048,
+            0.0077334037670644135,
+            0.009415939321950421,
+            -0.012399032514032437,
+            0.006084821107593102,
+            -0.034655634244304866,
+            0.013104268092544735,
+            -0.0016781301605750378,
+            0.012249966178417542,
+            -0.025567673456449492,
+            -0.023488408901029257,
+            0.007732021927617865,
+            -0.006986715221001125,
+            0.01461433573263053,
+            0.00011845838735106802,
+            -0.005506714145045683,
+            -0.006827219009979501,
+            0.0005242424465709722,
+            0.002885486381705293,
+            -0.0042669493792680474,
+            0.021618593454103052,
+            0.00010034640948597114,
+            0.011602625527297699,
+            -0.023477647311478233,
+            -0.0162254255990209,
+            0.031352657169434826,
+            0.03529846802317018,
+            -0.0007059281329032357,
+            0.02568807107901636,
+            -0.00277252926066966,
+            0.010828396988798257,
+            0.00810821945842862,
+            -0.010307381937016405,
+            0.0003407672690264756,
+            -0.003768030765932326,
+            -0.0027367413382233317,
+            -0.012887542072043335,
+            0.0031930180018598015,
+            -0.014086667286395425,
+            -0.01504545079945656,
+            0.02010182276528891,
+            -0.002519095978589373,
+            0.011477055159845283,
+            0.006165264376334778,
+            -0.01139265727997407,
+            -0.016539552549751473,
+            0.01574258576985793,
+            0.05233886430069725,
+            0.022428487725773775,
+            0.044334901638531425,
+            -0.00706941862193002,
+            0.011585028247041065,
+            -0.016760013805097753,
+            0.001211362507398395,
+            0.048283299647824234,
+            0.015403290676022106,
+            -0.013446978976128664,
+            0.024015090204073844,
+            0.013753743307879434,
+            0.0398341544586196,
+            0.0034302789804899806,
+            -0.020152697770903245,
+            0.041614050886963766,
+            0.00392650113407049,
+            0.013465009394885544,
+            0.020487015678247938,
+            0.002430045518615804,
+            0.0035516244267459565,
+            0.02938659684630997,
+            -0.010142701862398358,
+            0.010335745659719125,
+            -0.033187389861872627,
+            -0.021792906328508686,
+            0.02532137764121119,
+            0.0005114146307176893,
+            0.033795722059780535,
+            0.007069260093915293,
+            -0.017628598898476375,
+            -0.017395163521872672,
+            0.021985468266342154,
+            -0.007987351575783343,
+            -0.025037973555114328,
+            0.002627649768366858,
+            0.008700784263932844,
+            -0.004480915144455101,
+            0.006798551384053197,
+            -0.014580806303786968,
+            -0.009741248878656092,
+            0.025066050175629597,
+            0.007330922027951371,
+            0.010472464570407973,
+            -0.024286928122882124,
+            -0.011159235410800293,
+            0.006107141408585794,
+            -0.04181412282877586,
+            0.008647396691893692,
+            0.00528510107024571,
+            -0.006647986735406744,
+            -0.002548771993161688,
+            -0.0011322362644692566,
+            0.00513401828632921,
+            -0.0035963203271502263,
+            -0.014021183815351008,
+            0.022993860073007393,
+            0.02875036422479212,
+            -0.012996574913859303,
+            0.014373647963439562,
+            -0.0093541571449835,
+            -0.019474535598737786,
+            -0.0233122546137944,
+            0.014269723193443946,
+            0.008784376430480408,
+            0.004317715767550436,
+            -0.008273153355003107,
+            0.0010203136295909875,
+            0.001909823045834306,
+            0.02406426043795662,
+            0.049377235716785324,
+            0.005265955549193235,
+            0.029166436381966566,
+            0.017136545810044906,
+            0.03882940870384413,
+            0.0007533463090350569,
+            0.016233986042907378,
+            -0.03311355052072762,
+            0.03082356484543373,
+            -0.0009431717656252606,
+            0.04077734731808,
+            0.0369941736537998,
+            -0.05477277258937169,
+            -0.014658721668336077,
+            0.028327060073149418,
+            0.010896122046930724,
+            0.01436699901738847,
+            -0.003307011398652852,
+            -0.01163199508062231,
+            0.029659278016495517,
+            -0.023936970669211955,
+            -0.010748756889752312,
+            -0.00767811402724157,
+            -0.010785978668072403,
+            0.009408061520238972,
+            -0.0038320809092061126,
+            0.038383104318077226,
+            0.0070413219383093,
+            -0.009684310528333753,
+            0.019154521368080882,
+            -0.006292471963504016,
+            0.01596901434992771,
+            0.0037167338451907566,
+            0.0019879266696342887,
+            0.03104390318244226,
+            -0.003561483519923973,
+            0.028949607591091022,
+            0.015156617705082505,
+            0.017473301120296506,
+            -0.0029321911929339165,
+            -0.017573294769065502,
+            0.027284173131869906,
+            0.009113069584708108,
+            -0.01971507323375489,
+            0.003667023499937692,
+            -0.0025339283534597176,
+            -0.0077697455666997945,
+            -0.010491575421861018,
+            -9.407001004405342e-05,
+            -0.00211520268689493,
+            0.02087980536494742,
+            -0.005582301812064227,
+            -0.0016376727058019977,
+            -0.0064225805186745605,
+            0.0052835419390265485,
+            -0.019662445004865352,
+            0.015117937189193789,
+            0.007575238218238001,
+            -0.018537762525347407,
+            0.014673223139419801,
+            0.019165766004277593,
+            0.0343151018851732,
+            -0.023949650834192267,
+            -0.0009279394158895472,
+            0.01730410035639405,
+            0.005928527001253011,
+            -0.018975612465080345,
+            -0.033873819454780706,
+            0.00825263613547798,
+            -0.01745801647450136,
+            0.027401724356781083,
+            0.008830481774058282,
+            0.0347823637819511,
+            -0.02309575756624861,
+            0.020559439488958053,
+            -1.2525886083543457e-05,
+            0.01593239963830551,
+            0.01455178268663934,
+            0.027993448238508672,
+            -0.017573784980205873,
+            -0.0379374065982424,
+            -0.0030450459335320053,
+            -0.0029968228865394214,
+            0.016851016821808874,
+            -0.013002468483615099,
+            -0.003337198735398279,
+            -0.0019044899095842511,
+            -0.029686441224288696,
+            -0.021487593939979652,
+            0.027796807698461906,
+            -0.04637191752173605,
+            0.0030176823995715597,
+            0.03281049040516897,
+            -0.030047693785808505,
+            -0.04246000386774685,
+            -0.0014603539776430917,
+            -0.03560639883119047,
+            0.0010602583020630973,
+            -0.01222013383414755,
+            0.029128085483749913,
+            0.007388529661657186,
+            -0.020518571527741426,
+            0.018184513437823,
+            0.0076594003031722,
+            -0.004161472492195519,
+            0.006100954284405263,
+            -0.005239699097592758,
+            -0.028136388241670696,
+            -0.015254134018585082,
+            -0.03037392857155718,
+            -1.6317930920712043e-06,
+            -0.002478230556191563,
+            -0.012845180214134591,
+            -0.024383682688014996,
+            -0.000484419407302015,
+            -0.014810859291916117,
+            3.958274758340876e-06,
+            0.05136363130801806,
+            0.02120810926469503,
+            0.005060247818396135,
+            0.012869784532979883,
+            -0.019665446839172217,
+            0.04743686110578455,
+            0.003514534022065111,
+            0.03583012040901262,
+            0.002062991366066978,
+            -0.02173965243546218,
+            -0.0005002960506349255,
+            0.0319890423992129,
+            -0.0016894669649440992,
+            -0.017249094049398075,
+            0.006136324758346181,
+            -0.009220681391996718,
+            0.00837047468537671,
+            0.012871598576797918,
+            0.01579543926540845,
+            -0.0061999233512262265,
+            0.014725777187858035,
+            -0.003927118678273384,
+            0.009336164668870259,
+            0.03122006739418818,
+            0.019771411252821138,
+            -0.024716950288097537,
+            -0.037736399088695305,
+            -0.012523788001500705,
+            0.001234987976751515,
+            0.034477269953960235,
+            -0.013836458205716058,
+            0.003837278407556882,
+            -0.011027628278972294,
+            -0.0054689562005572675,
+            0.005739209890610896,
+            -0.006985434270987544,
+            0.0015806421600885957,
+            0.012196603223058041,
+            -0.006859901421089723,
+            -0.025670287745499374,
+            -0.004011122451643543,
+            -0.023258896941014143,
+            0.012997299613183496,
+            -0.010144866692641839,
+            -0.006816231069049567,
+            -0.005213359047878286,
+            -0.0014167278420750282,
+            -0.0062535968890290545,
+            0.0006662897611595949,
+            -0.014236807237438571,
+            0.01012846684096945,
+            -0.00899852930003816,
+            0.0016620397335915663,
+            0.007838787681725669,
+            0.007725632848979133,
+            0.0033110678307931205,
+            -0.008206916937522372,
+            -0.008777095103563134,
+            -0.01631820698659747,
+            -0.032412484976539334,
+            -0.03000207979123948,
+            -0.02276516767125653,
+            -0.003870969029785218,
+            0.0034978865240734053,
+            -0.032594545526009384,
+            -0.020635972136797354,
+            -0.002367523739258502,
+            0.0030103817989158062,
+            -0.014587341442812854,
+            0.0014779462725289245,
+            0.007969995140846214,
+            -0.0029319878689507335,
+            0.019355924479361485,
+            -0.005448177691636249,
+            -0.021203447333791942,
+            -0.028956667864566195,
+            0.017851274292380093,
+            -0.015891501231440428,
+            -0.010070179095955473,
+            0.0064388768290910715,
+            -0.005633920123057214,
+            -0.03957042012546069,
+            -0.008945181709739484,
+            0.010992989857329521,
+            0.0035378401849368002,
+            -0.025936727040303652,
+            0.031903561853958325,
+            0.030132441728296204,
+            0.01845519141895959,
+            -0.017902241158092712,
+            0.04710441323879975,
+            -0.006739916600200837,
+            -0.012103058303430145,
+            0.016276540022040643,
+            0.023963592487909288,
+            0.028774476787161263,
+            -0.02643829153200483,
+            -0.005349178006882811,
+            -0.010545703050010008,
+            0.005354643441490569,
+            -0.02271588009302058,
+            0.008468528451368553,
+            -0.013601178278798316,
+            0.011576067943282815,
+            0.019227032134009135,
+            -0.013197207810056759,
+            0.02629307658052319,
+            0.0045263515787155985,
+            -6.067227645639678e-05,
+            -0.0037770406201685248,
+            -0.016287414859552752,
+            -0.012319957965328086,
+            -0.034674880061192896,
+            0.020297690806102293,
+            0.01889209267823917,
+            0.003953684662429981,
+            0.021845046751532583,
+            -0.049434011185276405,
+            0.02806744160678342,
+            0.0017849117483741536,
+            0.021763434864703356,
+            0.013889518910180847,
+            -0.001843068102127134,
+            0.013886691089114936,
+            -0.0032117229736863266,
+            -0.019971320584891997,
+            0.006079730961157539,
+            -0.006352535944493006,
+            -0.03316544192496317,
+            0.014736816574462923,
+            -0.03268321622529288,
+            0.013639134163846678,
+            0.015454170145821088,
+            -0.010170396823729307,
+            -0.021708948733883182,
+            -0.008303375382721087,
+            0.0010570623547459114,
+            -0.0014149688171089135,
+            -0.006849845444850982,
+            -0.04346483266888068,
+            -0.03020319096411844,
+            0.03124837430439466,
+            -0.0016720020681567616,
+            -0.03404132565569305,
+            -0.007209824644083462,
+            -0.02711263625315296,
+            0.004780610743996515,
+            -0.004212647822173262,
+            0.0025676935176963222,
+            -0.048750272936498695,
+            -0.01018434342955251,
+            0.011179560412168381,
+            0.031648611061279724,
+            -0.029298960703078986,
+            0.021370266671953315,
+            3.770954396468699e-05,
+            0.006629905234129844,
+            -0.00943632381701963,
+            -0.00484754585233744,
+            -0.012115337995959813,
+            -0.02607251387584821,
+            0.021868809148577698,
+            -0.0020587964818437345,
+            -0.0013502026980372474,
+            0.0341522429571705,
+            -0.00669289309893371,
+            -0.010113192232031375,
+            0.0005934017402521538,
+            -0.011327797122243258,
+            -0.017269539623150844,
+            -0.010467173785198161,
+            0.0005779464354975677,
+            0.0030406502310922306,
+            -0.024755136158113783,
+            0.004943446717510779,
+            0.04566273647510376,
+            0.013607936503018211,
+            0.02534200532143803,
+            0.017104750405207106,
+            -0.03792550084708064,
+            -0.04394052120951292,
+            0.0012600504352102988,
+            -0.005807815271334456,
+            -0.01944685298932453,
+            -0.01732108197375621,
+            0.0011677414026425442,
+            0.00790730598557849,
+            0.0037240231183698074,
+            -0.011427532982681041,
+            -0.02086964367350831,
+            -0.032539146090370744,
+            -0.0136797717535217,
+            -0.009464326388205681,
+            -0.007302204556505735,
+            -0.032703238152349344,
+            -0.015929357859286307,
+            0.026848214129378756,
+            0.01012098852614113,
+            -0.005646819862636667,
+            -0.0107564788588665,
+            -0.024118155140225247,
+            0.007122339033521904,
+            -0.0027183981945039814,
+            -0.009219030314883272,
+            -0.004672057262936543,
+            -0.016450690196817726,
+            0.02348093022809639,
+            0.009139210149358477,
+            0.024997129282118148,
+            -0.058533911093319214,
+            -0.014411932474734742,
+            0.014256038602455438,
+            0.0020188651815054508,
+            0.001417136751974749,
+            0.029836680491427268,
+            0.0028786535004858044,
+            -0.006526872714072965,
+            -0.02967220833325222,
+            0.028817295504169067,
+            0.027449694157944637,
+            0.01800200378294509,
+            -0.010560094796033294,
+            0.0033996190760546383,
+            0.006004691351249025,
+            -0.005679195734863024,
+            -0.011927786207717698,
+            0.0021260244673648285,
+            0.018998574211603905,
+            0.017174632765532803,
+            0.014659925822127794,
+            0.02455156987384756,
+            -0.005882590537857664,
+            -0.03911277764764406,
+            -0.009781110378157883,
+            -0.0119630929326396,
+            -0.004815125964940769,
+            -0.0026631770014339242,
+            0.008903827530107938,
+            -0.011824767329332145,
+            0.013088406716464123,
+            0.030264448923574067,
+            0.042592268021664714,
+            0.021292401951863188,
+            0.006150997237661769,
+            -0.005463801502425317,
+            0.0007399518998122848,
+            0.0038649546282459707,
+            0.011355675550395135,
+            0.035762872771750435,
+            -0.018147218572765546,
+            -0.019704502923991365,
+            0.0015558311499702044,
+            0.023543487977413267,
+            -0.007150197116420024,
+            0.02608209797565159,
+            0.023985156585794257,
+            -0.0003434024317322991,
+            -0.01271338268931938,
+            0.000985282363050309,
+            0.006710364567868783,
+            0.014540943712513688,
+            -0.02154205198718209,
+            0.004039619025717627,
+            0.001865104196024055,
+            -0.009756600776161738,
+            0.029738871754072076,
+            0.00039154552800346406,
+            -0.012981811693538832,
+            0.03825344802886,
+            0.11025963334575106,
+            0.001476339401950677,
+            0.0031187927813794386,
+            -0.031522639386920355,
+            -0.015113380587160163,
+            0.0021786482865071505,
+            -0.005948756393737655,
+            0.020082591515778573,
+            0.012751062800862873,
+            0.0055847042526916265,
+            0.010100961720464159,
+            0.038084609873482715,
+            -0.009203643420546616,
+            0.009804311800033327,
+            -0.01878272400403416,
+            0.017185477997947226,
+            -0.00972716681599207,
+            -0.00857276597428136,
+            -0.007641511960016827,
+            0.004167416313632853,
+            0.00984005584819001,
+            -0.004596999098606572,
+            0.018810908413859465,
+            0.002496310438422421,
+            -0.01613133688770545,
+            0.011505030137914962,
+            0.007437924384075958,
+            -0.005386885126779478,
+            -0.031755982499042136,
+            -0.013811586180238287,
+            0.01190667400285204,
+            -0.0013786018401773159,
+            0.0007177632889203917,
+            0.012342640512536284,
+            0.01205074763420124,
+            0.031217897495485192,
+            -0.0244324065837541,
+            0.0025071378510626696,
+            0.0070486374045588465,
+            0.0159322020159265,
+            0.01098799814774876,
+            0.00029844505852147526,
+            -0.015456787478044924,
+            0.027139764983358832,
+            -0.011978684728746428,
+            0.010378658252224036,
+            0.01146607671795495,
+            -0.04131658683485702,
+            -0.0009967932006190922,
+            -0.005916133731248287,
+            -0.015698754902727707,
+            0.013905118384587169,
+            -0.040163447416240027,
+            -0.005682084326086952,
+            0.004297114532998809,
+            0.010724201884417054,
+            0.0058749102353125955,
+            -0.017715270733111323,
+            0.009962853888582856,
+            0.00955216765250444,
+            0.0025392532243413593,
+            -0.006356691980285544,
+            -0.006218169970979339,
+            -0.008606942887730427,
+            -0.004280171880086814,
+            -0.015669721147884084,
+            0.0057481312044666884,
+            0.0077227094511494055,
+            -0.018751123856436715,
+            -0.01585864749751703,
+            0.009183099781375234,
+            0.006837438250956817,
+            -0.001445622524955126,
+            0.004828557942941401,
+            -0.010034012200008746,
+            -0.004543834537450213,
+            0.01097540631691429,
+            -0.009334158551770033,
+            -0.004804430123454511,
+            -0.02609985520241612,
+            0.013342299390862539,
+            0.011103136832190223,
+            -0.008261332966534102,
+            -0.006944131083745896,
+            -0.016931502665633764,
+            -0.0031318730206304824,
+            -0.0026508016484082988,
+            0.0016110884325639078,
+            0.017071266355203386,
+            -0.009906320666640836,
+            0.010877416311741906,
+            -0.03496707406301918,
+            -0.015357787514985604,
+            0.01622830705930841,
+            -0.01290003882907745,
+            -0.005625420341281716,
+            -0.009838333350984609,
+            0.0006911383780218238,
+            -0.007545081680638308,
+            0.006074752194220675,
+            -0.041320607445282646,
+            0.0035102345636816475,
+            0.018825308640738208,
+            -0.009723969339534365,
+            -0.020829421583878845,
+            -0.009055117268112308,
+            0.010487235049013567,
+            -0.015831829161845813,
+            -0.01780922120240258,
+            0.0016219998217122911,
+            0.010559783293374071,
+            -0.0055135690971065965,
+            -0.005160232484458576,
+            0.006163424486940105,
+            0.020247627989655837,
+            0.015220964429515309,
+            0.04337432385613251,
+            -0.011915355138717245,
+            0.009617347907919507,
+            0.027812934121040484,
+            -0.030652481946156104,
+            0.01037470561447702,
+            0.02599512377987521,
+            0.0037340561802882417,
+            -0.0013375196064018082,
+            -0.010197691376747194,
+            0.004161692215514214,
+            -0.012254402844820636,
+            0.008715552098450461,
+            0.004697361473725416,
+            -0.027978648968732336,
+            0.017196771853396124,
+            0.019448218659009996,
+            0.0038720194629308794,
+            0.0006995062900842749,
+            -0.01527901684789497,
+            -0.011825577013200781,
+            0.019194236643713274,
+            0.027690065919038755,
+            0.005460028121722145,
+            -0.015512497062157167,
+            0.02370643287056251,
+            0.012916257803027072,
+            -0.004973882005506947,
+            0.0022332716531575122,
+            -0.016984605526823043,
+            -0.01443316577897661,
+            0.030942241604983316,
+            0.011563751997976407,
+            0.01908030186367601,
+            -0.021614212853976106,
+            -0.003982310717524619,
+            0.019931579069642925,
+            -0.021029366993395197,
+            0.013006879711144009,
+            -0.019497570222553013,
+            -0.014067925104548129,
+            -0.010687576618198498,
+            0.02884494104893037,
+            0.009229770955288097,
+            0.026701705044769222,
+            0.012715371448003833,
+            -0.021444875656527786,
+            -0.021050591467198086,
+            -0.0027703267406452336,
+            -0.02721606030674425,
+            0.01688256721881862,
+            -0.055990128879151335,
+            0.0016563858333845547,
+            0.010911810464298247,
+            -0.008604898990983188,
+            0.004039521277648332,
+            -0.007858378479197747,
+            0.010021929060776957,
+            0.020926274809879138,
+            -0.02956229623532218,
+            0.01256060127924585,
+            0.01156331569664224,
+            0.0218688589850221,
+            -0.018861065515394883,
+            0.015339087155577661,
+            -0.00841001512858443,
+            -0.027396412391138852,
+            0.008323898095545889,
+            0.01763697170382491,
+            0.014160184414353919,
+            -0.011992345081843988,
+            0.004485633614765869,
+            -0.04000932898876895,
+            0.027999519977069676,
+            -0.016823500188831896,
+            0.002010511519234648,
+            -0.004537991252901591,
+            -0.0067219583064939085,
+            -0.008523354091034399,
+            2.682477256914193e-05,
+            -0.0005371719103087346,
+            -0.030920653652724776,
+            0.0012939076000836422,
+            -0.04144943703666512,
+            -0.02260198516292792,
+            0.013327519374598822,
+            -0.0012897465780583367,
+            -0.005059170593504863,
+            0.02999548027478466,
+            -0.02863650640718448,
+            -0.027272295036911975,
+            -0.014376307736824656,
+            0.03534844385341626,
+            -0.015358480569131909,
+            -0.000540068394391221,
+            -0.009287193719601908,
+            -0.02236594406404711,
+            0.01091078016405303,
+            -0.006469935602381157,
+            -0.0019272207412132534,
+            -0.01387843655423569,
+            -0.03972775413987396,
+            -0.005948332199741248,
+            0.019500395452013777,
+            0.0006122786247500248,
+            0.003656719755271774,
+            0.014473976840529128,
+            -0.0072278373928449675,
+            -0.0036623750417377524,
+            -0.0038809301345520044,
+            0.026929584793529297,
+            -0.022225170775945557,
+            0.009744440210592443,
+            0.004704883568603746,
+            -0.0005991305039095727,
+            0.017317686588588826,
+            0.000351879037187266,
+            0.0024030356296394478,
+            0.0029060795825935717,
+            -0.0035301907818594274,
+            0.02922907976800865,
+            0.0064070048516525,
+            0.0016291092834589405,
+            0.0001988403049139903,
+            0.004746570638478093,
+            0.01395197992587789,
+            -0.01069245020264236,
+            0.0157940052093005,
+            0.02865126209030693,
+            0.00021287181382353683,
+            0.018345429962063935,
+            -0.015921361819192573,
+            -0.036059435316760584,
+            0.019807131219096722,
+            -0.029642240399677812,
+            -0.003504127677138975,
+            0.022473005924273325,
+            -0.012903852741467797,
+            -0.0031750649075827082,
+            -0.01107957248024579,
+            0.027243132530764567,
+            0.023512079029549708,
+            -0.014856101416774535,
+            -0.015142539116300868,
+            0.004337433267586975,
+            -0.010210607143984678,
+            0.010939153927790107,
+            0.029382995339774957,
+            -0.011230337698862454,
+            0.011717621080350227,
+            0.001161341258722717,
+            -0.003778338045364591,
+            0.019300005069271462,
+            0.0027681684372537606,
+            -0.007401677666787755,
+            0.025628069381915978,
+            -0.008115857591022265,
+            0.013043669211084909,
+            -0.004672358463640539,
+            -0.012902494254220482,
+            -0.003106794914564588,
+            -0.017558145644151636,
+            -0.01225024815282763,
+            0.0011522376202485873,
+            -0.005528597019035285,
+            0.06711658458916447,
+            0.010930891527455397,
+            -0.017888423424597618,
+            -0.0059190199704595775,
+            0.00955567741445378,
+            -0.000966061435587513,
+            -0.009466328431452424,
+            0.027063767833815658,
+            -0.016888493246695876,
+            -0.009681104485277698,
+            -3.814655691584766e-05,
+            -0.0233149747136092,
+            0.0030596902889784496,
+            0.010827987186229144,
+            0.0012052973350181351,
+            -0.013994504803884577,
+            0.005318095523120499,
+            -0.004843763992616558,
+            0.0034188401938723923,
+            0.003678626789714963,
+            -0.0025749484272345573,
+            -0.005333294277998365,
+            0.02307826358450567,
+            -0.012791914224263883,
+            -0.008519982879775168,
+            0.01173424692936401,
+            0.001777123104204978,
+            0.003219553788359312,
+            -0.004564745566711077,
+            -0.002015107048152335,
+            -0.017123545909339455,
+            -0.011278616226036531,
+            0.023713473402287875,
+            -0.02733523566514418,
+            -0.00150943777243135,
+            0.017328382962199716,
+            -0.005705544203971564,
+            0.016650187392502975,
+            9.750950361601131e-05,
+            -0.02100066506382883,
+            0.00010090567939930427,
+            -0.024778741347883073,
+            0.010484659495766551,
+            0.013963183459158656,
+            -0.009531952698151101,
+            -0.020878167882780043,
+            -0.027881362421494915,
+            -0.010190031350698889,
+            -0.027246821338460653,
+            -0.0280692647056191,
+            -0.02020582179742232,
+            0.009778340348478517,
+            0.005715364307462615,
+            0.021929057840860695,
+            -0.006380523858228711,
+            -0.027552858236741976,
+            0.015095163660340326,
+            -0.013269344817111162,
+            0.008282918921343354,
+            -0.019991891695114384,
+            0.01754833974582726,
+            -0.014720830704852575,
+            0.02112161088830394,
+            -0.00041409266970033855,
+            -0.004490472334793758,
+            -0.012076561942010787,
+            0.003976913901514849,
+            0.008101357042612863,
+            0.00176590819486174,
+            -0.008977295353115363,
+            0.026242190932756135,
+            0.01707634775369835,
+            -0.007385093173634561,
+            0.01604386075569055,
+            0.031855080733360075,
+            0.014762206220119051,
+            0.038860660002328094,
+            -0.007238177370595944,
+            -0.009237098188374397,
+            -0.02764495845127417,
+            -0.004407870047633689,
+            -0.006820430763682645,
+            -0.004516837004332235,
+            -0.015251921783911087,
+            0.03164122113689215,
+            -0.010371400055822811,
+            -0.01686972159902524,
+            0.03979035122593764,
+            -0.021846728842184655,
+            -0.007084905708666271,
+            0.008642771511425465,
+            -0.013996636040226265,
+            0.03176661229383477,
+            0.007275612529363987,
+            0.018452818352301472,
+            -0.015039890816023917,
+            -0.015988701055097997,
+            -0.0019911190487748244,
+            -0.002806138139429132,
+            -0.020828646560807865,
+            -0.0063794261154720665,
+            -0.01614570050728432,
+            -0.0026319562705484634,
+            -0.00570753504957681,
+            0.015917008930969468,
+            0.0013916917451795444,
+            -0.015420032290910069,
+            0.04488171544231248,
+            0.005033499525006087,
+            0.022768662319611596,
+            0.010777933460735837,
+            0.012372342667053587,
+            0.015819823672091563,
+            0.006759824420801962,
+            -0.0001668726099910387,
+            -0.02211654164646909,
+            0.004261169256891085,
+            -0.005641717035723983,
+            0.0041005490092982915,
+            0.02012094437568505,
+            -0.024436624754739972,
+            -0.029708548399946657,
+            0.024493078535528394,
+            -0.01833101179919427,
+            0.0030908990956393764,
+            -0.0031025537219982234,
+            -0.010059653447038065,
+            -0.011612233298907742,
+            -0.03293899346571458,
+            -0.0005929867206193162,
+            -0.027438115397399914,
+            -0.03635734152790657,
+            -0.006287687204976032,
+            -0.007850474352061472,
+            -0.003574235211373211,
+            0.02449669755605501,
+            0.004269460596739739,
+            -0.029959856511873897,
+            0.01782698044631415,
+            -0.013252733005984911,
+            -0.013342147906120882,
+            0.0029581361690999564,
+            0.004786570391076097,
+            -0.0027618259744494854,
+            0.010720003765859173,
+            0.007898965393358715,
+            0.007358694001293221,
+            -0.010710502803266644,
+            -0.004049671482838962,
+            0.0013263710826446039,
+            0.0030943253037901135,
+            -0.013325799375903166,
+            0.030553973934607993,
+            -0.018812559793353154,
+            -0.004819166852190956,
+            -0.0006499911473215254,
+            0.0034556893023524674,
+            -0.0110929170801761,
+            -0.014862886414702483,
+            0.021004665462647988,
+            0.0031088550726355926,
+            -0.012570481398485388,
+            0.0096952264731202,
+            -0.0018868240406799254,
+            -0.0024767709620225923,
+            0.01176108203579327,
+            -0.002080730326597438,
+            0.014924914157541781,
+            -0.011040818699356567,
+            0.01490073610670544,
+            0.0034649445207609085,
+            0.008645740731109337,
+            0.003603641937848282,
+            -0.01905231764119889,
+            0.009780124784430865,
+            0.016356631955067272,
+            0.013415470760506568,
+            -0.029496530287207723,
+            0.008881639987471632,
+            -0.005975399013716009,
+            -0.009870580036511721,
+            -0.024372445588075428,
+            0.002590423942920311,
+            0.028960961301291905,
+            0.012044179251559706,
+            0.007934516374398097,
+            -0.0077949340827692054,
+            -0.005480592942515702,
+            -0.019923500717513334,
+            0.009041031662948522,
+            -0.01095651774810384,
+            -0.025157908389819465,
+            -0.007578712371715301,
+            0.03430545904801765,
+            -0.010740874257824011,
+            0.01343875386664532,
+            -0.005526757815283282,
+            0.02436552949831661,
+            -0.0027538610388853284,
+            0.0013690344627799137,
+            0.008416180197734094,
+            -0.006091433734127276,
+            -0.001708607133233843,
+            0.013349212286052506,
+            0.041066250931277824,
+            0.0038375614886832564,
+            -0.0050950709988837216,
+            0.01149207527692021,
+            -0.00326389523728132,
+            -0.005712157453687712,
+            -0.00309579814793649,
+            -0.025491974096530076,
+            0.015133356727314608,
+            0.0010745378801251735,
+            -0.017162228172591873,
+            0.021785292616181687,
+            0.027891598422935242,
+            0.013262622112280296,
+            -0.0178471114368855,
+            0.014456217536546996,
+            -0.009278392107874806,
+            0.01629461595238019,
+            0.02423194076020779,
+            0.010067296472207578,
+            -0.011865648674106681,
+            -0.0070073298858753805,
+            0.008985354045706703,
+            0.008096016282907443,
+            0.0004236756235618935,
+            0.020370823466310733,
+            -0.028334250141960886,
+            0.01483270667238479,
+            0.007137078270349882,
+            0.03314918453195744,
+            -0.0006231629401653488,
+            -0.014397909717816884,
+            -0.008918359605259833,
+            -0.002915473990900902,
+            0.011759189078587877,
+            0.003523992368256188,
+            -0.0006853609664806012,
+            -0.004749148984168426,
+            -0.014181883848149463,
+            -0.006778789302277201,
+            0.012376777107762334,
+            -0.0031646329407784245,
+            0.0030514749941358095,
+            -0.0007326603315688041,
+            -0.007811941797535734,
+            0.015661953163847418,
+            0.001669500869963878,
+            -0.0007143628544145883,
+            0.0016729300148826723,
+            6.604560293656475e-06,
+            -0.00467413979302959,
+            -0.019114577369302277,
+            0.008922356418978664,
+            0.00867683974577237,
+            -0.006578536023513047,
+            -0.006467824108991175,
+            0.009324373505311057,
+            -0.01432602608910974,
+            0.01852478469282692,
+            -0.0061254590933158826,
+            -0.006603842196861297,
+            0.00014277641689784565,
+            -0.012328738339684425,
+            -0.003642202698645936,
+            0.0034895681343535914,
+            0.0051453312116433265,
+            -0.000908593523739809,
+            0.0036062764245630334,
+            0.003940522150829955,
+            0.01083964694261132,
+            0.002824079422783228,
+            0.0015266576108775834,
+            0.007280263953807289,
+            -0.01039293741283943,
+            -0.007403606843286745,
+            -0.004993622657038095,
+            -0.006341662331986892,
+            0.007288958567606623,
+            -0.004364975055335228,
+            0.010457360052582658,
+            -0.0062292104145735515,
+            0.020471836369529147,
+            -0.006133009423930811,
+            -0.0024055488697334888,
+            -0.017695183455939345,
+            -0.007234478249839231,
+            0.007156119488831356,
+            0.007878044165936736,
+            0.009359297231370758,
+            -0.04487830669749157,
+            0.005038301058510894,
+            -0.0054192943462008595,
+            0.02941616633783806,
+            -0.009828941844770915,
+            -0.004335130343692854,
+            0.016067824555917796,
+            -0.019100842323053376,
+            0.0037066021889614513,
+            -0.013965982913833292,
+            0.003695036285467973,
+            0.0034558765116311525,
+            0.014205693020177169,
+            0.031307058208919394,
+            -0.007482180869231769,
+            -0.009383819319265733,
+            -0.006384083847148052,
+            0.026967380724567013,
+            0.018643952476557566,
+            0.023548246269020395,
+            -0.011885146764471029,
+            0.00960518654541979,
+            0.001357650512536872,
+            0.018271473493526338,
+            -0.009663920325675169,
+            -0.011163336807225407,
+            -0.000516828412255004,
+            -0.010288703356943627,
+            -0.0032111641654394615,
+            -0.021092460896394025,
+            -0.0033456309643941965,
+            0.025876107752976232,
+            0.0038584058642743987,
+            -0.0036689325941524813,
+            -0.015624044740149702,
+            2.2644755460272697e-05,
+            -0.011931276845804481,
+            -0.0055870010742218364,
+            -0.015697330226950804,
+            -0.028927689083316704,
+            -0.016192017705678192,
+            -0.002914436546908901,
+            -0.017224654082473787,
+            -0.009460990414684924,
+            0.002071634636546831,
+            -0.015533259213802986,
+            0.004918377471991415,
+            -0.015669642231243106,
+            -0.010915412245035698,
+            -0.00046472075835247,
+            0.004672390433470177,
+            -0.004682173537415046,
+            -0.024109027987987815,
+            -0.013628794248618615,
+            -0.0022054473551170926,
+            0.006500170428351522,
+            0.010016298501187265,
+            0.008324628078950177,
+            -0.0069374343736907625,
+            0.011201167788973098,
+            -0.009746811993090597,
+            0.010155637867574787,
+            -0.02773759960959338,
+            -0.04317847702979315,
+            -0.000791583637601122,
+            0.009241551203900427,
+            0.001979688190287368,
+            -0.01884488538245213,
+            0.0038489955050695704,
+            -0.011280060242688173,
+            -0.00847814532468811,
+            -0.007522173019881079,
+            0.01573091287228822,
+            0.004138348969142638,
+            0.013463997539337106,
+            0.0042065270170965705,
+            0.0285467221116185,
+            0.012635383326761538,
+            0.01030078781909351,
+            -0.003092067773698843,
+            0.004763648809248413,
+            -0.0022835042548997662,
+            0.009077632103272405,
+            0.02331475506951857,
+            0.0076225652102231605,
+            -0.002326207128504976,
+            0.00409858249509217,
+            -0.0005698076375460636,
+            -0.005442088641415082,
+            0.018565262151290427,
+            0.02915691852605287,
+            -0.0069811288298562155,
+            0.008120076286611811,
+            -0.0009110258445641983,
+            -0.009900329477406562,
+            0.0038288023984988134,
+            0.005765264445222657,
+            -0.014230725839572923,
+            -0.0024175374911696653,
+            0.009367987848760041,
+            -0.024664375361024386,
+            -0.004113757123212052,
+            0.020371264792326226,
+            0.0024813232005061487,
+            -0.016599537494340785,
+            0.03488073638324099,
+            -0.003709931703179228,
+            0.003010315958426392,
+            -0.00944047118505417,
+            0.016362178907316286,
+            -0.017989464717541103,
+            0.0005390907615384354,
+            0.013131172782591922,
+            -0.005251614069791733,
+            0.005783823857695169,
+            0.003744510952053123,
+            -0.010417674399178015,
+            -0.010824994224858631,
+            -0.0033951133363820767,
+            0.01085129781083312,
+            0.002642227227069058,
+            -0.024446190958178605,
+            -0.017731767444262785,
+            0.019381370848622046,
+            -0.01001206050737818,
+            -0.008290790009659911,
+            0.0009331678418736739,
+            0.0029270629970877303,
+            -0.004637651319894003,
+            -0.004290917518172777,
+            -0.003574121539057569,
+            0.009121344354139133,
+            -0.009485343005168732,
+            0.01637203851901565,
+            0.004043708359084711,
+            0.01843096334879628,
+            -0.011588935298420727,
+            0.005808551475129831,
+            -0.001368952044937682,
+            -0.007242291047654279,
+            -0.019004856431577073,
+            0.0059537808323522675,
+            0.0073777483129045685,
+            0.016051495408653346,
+            0.011060627996179347,
+            0.018569088420830965,
+            0.011345955812555888,
+            0.0039209921807600205,
+            0.001180658155391396,
+            -0.011006781869601802,
+            -0.016371246428912994,
+            -0.014232536816028922,
+            -0.007416908142074357,
+            -0.019561375671062513,
+            -0.0013253236527051844,
+            0.017798265625542643,
+            0.014528038148955117,
+            0.017704367192118375,
+            -0.0048389172759564884,
+            0.016018290057587425,
+            -2.7789810195444745e-05,
+            0.0009215076153434622,
+            -0.005897137279173464,
+            0.001794778099552753,
+            0.016314258230521846,
+            -0.003313666704140227,
+            0.0063152813149371136,
+            0.0002935873451081568,
+            0.01645203621687452,
+            0.022650990579075773,
+            -0.018899997585540014,
+            -0.0014442758993479705,
+            -0.006467599303099793,
+            -0.0021225224281092863,
+            -0.019894752041664215,
+            0.020266472457121338,
+            0.014019662562577204,
+            -0.005002061958298289,
+            -0.00943116174529331,
+            -0.004354703314807638,
+            0.02678118874395904,
+            0.020206530983694557,
+            0.006044260997975214,
+            0.026222728100119382,
+            0.01976601108150248,
+            0.01984865298844998,
+            0.011918890024498122,
+            0.009052182076567042,
+            -0.013092243192366595,
+            0.018358346459726883,
+            -0.01681770553983246,
+            0.0008504895530001561,
+            -0.003223133403363917,
+            0.002743153478851782,
+            -0.007937179998086906,
+            0.013021430914653432,
+            -0.00030955362267617273,
+            -0.025935204965869492,
+            0.007680669831594167,
+            -0.013183173230049736,
+            0.001375652741144253,
+            6.07690288088255e-06,
+            -0.001857629339163609,
+            -0.017652365850892467,
+            -0.0082709435899296,
+            0.0008448530375262078,
+            -0.01435595518048752,
+            0.023866770609514233,
+            -0.026826645910954506,
+            -0.02140002089501948,
+            0.00988141127158139,
+            -0.004436844338114915,
+            -0.012632978351515892,
+            -0.006972217855086974,
+            0.021864041851582767,
+            -0.005139891275508354,
+            -0.016596866242234234,
+            0.014168148679282804,
+            0.0005869000756971533,
+            0.00412958763467087,
+            0.0012364712526909765,
+            -0.008001798306987228,
+            -0.0030416500521329983,
+            0.006860667487819997,
+            0.028693213258586023,
+            -0.011895509688811421,
+            0.0016632663352826491,
+            -0.013799461077520309,
+            -0.0037922015693749754,
+            -0.02974150200357857,
+            0.027752493864472517,
+            -0.011830981692069491,
+            0.01989293240244598,
+            -0.004540523332199857,
+            -0.007667738626842118,
+            -0.021173122202316107,
+            0.035703796333655495,
+            0.004594443089494466,
+            -0.0006664409717898689,
+            0.007256660335524136,
+            -0.047588498838074554,
+            0.01105469709139337,
+            0.006498789757276137,
+            -0.023261890747840144,
+            -0.000964111432535798,
+            -0.003995007813335293,
+            0.02143067593410374,
+            -0.019461700447829744,
+            -0.012777737727876326,
+            0.0021024453304811634,
+            -0.021925886491102078,
+            -0.007056889535073896,
+            -0.0340604475320119,
+            -0.006137802768771592,
+            0.006503135515471076,
+            0.0038385769709340417,
+            0.0020799029890387023,
+            -0.021308499975514292,
+            -0.0309544649831362,
+            0.013308972238482328,
+            0.01651668347077017,
+            0.010007831710070633,
+            0.01200701454477496,
+            0.0067050757442456005,
+            0.011425518484475204,
+            0.00865023114755134,
+            0.00974582806570851,
+            0.008884712478547614,
+            0.039109252942848706,
+            -0.016122498485808056,
+            0.008671479031657285,
+            0.0005699279397862279,
+            0.02156543966774511,
+            -0.02391743870774471,
+            -0.009877938786506814,
+            0.015475531666380014,
+            0.003045729682805746,
+            0.030465388674890067,
+            -0.01511217342930631,
+            0.005735960079359902,
+            0.0030955813611916066,
+            0.003343849304714193,
+            -0.017685941897160877,
+            -0.019795694199965495,
+            -0.015889472669342385,
+            0.0005064000414834533,
+            -0.0007719666713392906,
+            -0.012817517834744753,
+            0.01370266779873307,
+            0.0029457043321269045,
+            0.004380574233134056,
+            0.003997906783171028,
+            -0.02610528890228495,
+            0.0033844277218836307,
+            -0.008397899012048187,
+            0.013703609594527308,
+            -0.0012702392316682096,
+            2.036849934035612e-05,
+            0.007261982817787643,
+            -0.00406175072535136,
+            0.018353316401139956,
+            -0.007206833180919801,
+            -0.004174154024217347,
+            -0.004097578846173991,
+            -0.00573702992914836,
+            -0.0017184308802689716,
+            0.022375713882789285,
+            -0.008746034802967245,
+            0.032729322137817145,
+            -0.014835515993879445,
+            0.01049714579786462,
+            0.013344412726432523,
+            0.006468296817982878,
+            -0.0036641997047404426,
+            0.0003384623491490165,
+            -0.0035345323089769998,
+            -0.006579417470328358,
+            0.015682816949756224,
+            -0.018934655843627496,
+            0.006860259740057359,
+            0.03727162736659208,
+            -0.002536806673033066,
+            -0.00615871353663963,
+            -0.008853057427430264,
+            -0.012968501800661,
+            0.028268388127570685,
+            0.005935705363468247,
+            -0.0013092005585187737,
+            -0.03024974000885413,
+            0.015180604705329429,
+            -0.0037501687534873948,
+            0.0016387074669205704,
+            0.00860407823411825,
+            0.00044285387135976006,
+            0.020127437993717013,
+            -0.0011272951769363546,
+            0.021290480413673304,
+            0.010032359217391118,
+            -0.0010666180914844765,
+            0.011529468490024128,
+            -0.036558434912810016,
+            -0.008579049583792913,
+            0.0038048395503202733,
+            0.019444233855445248,
+            -0.0006623807457136332,
+            -0.015853103419331053,
+            -0.004242407392496502,
+            0.009029483146616182,
+            -0.01059695082306233,
+            -0.013492380798775181,
+            0.016259633969459952,
+            -0.019096544374819852,
+            0.011560512412214506,
+            0.008617334373339341,
+            0.02512750386847096,
+            0.0041202753709624245,
+            0.002613588241299414,
+            0.0035069606381226207,
+            0.006602772841623022,
+            -0.002625433099855357,
+            0.011789793979724633,
+            -0.011592113209370399,
+            0.02102479518464762,
+            -0.008294039267346376,
+            0.007068661518691129,
+            0.0032541671568032306,
+            -0.022166914870027622,
+            -0.018234159989683987,
+            -0.0023642465680105544,
+            0.021312775708470108,
+            0.006016177710817625,
+            -0.012618763344460982,
+            0.0055411096806088885,
+            0.004690341026678943,
+            -0.00547301082703577,
+            0.009118596747082881,
+            0.024398283179066196,
+            -0.0028098889343668772,
+            0.01068095200128235,
+            -0.007137857371825465,
+            -0.027832295014375272,
+            -0.007421428223393234,
+            0.021707007996690945,
+            0.00020946296332069377,
+            0.016793613517175475,
+            0.014396690762872469,
+            0.012889706504719996,
+            -0.00488698200749025,
+            -0.005354238088985329,
+            0.018317372626432005,
+            0.014340608961907827,
+            -0.0048105230366691145,
+            0.001495716612807014,
+            -0.0018529641162211795,
+            -0.017240719704808234,
+            0.026933447120216628,
+            -0.024232238634425658,
+            -0.003438477892192365,
+            0.00892792234782449,
+            -0.012586276728773793,
+            -0.004923824123059311,
+            -0.023468301644204742,
+            -0.02117100775850015,
+            0.014296432515879623,
+            -0.003314359969711106,
+            -0.01750062790930347,
+            -0.006856892040084997,
+            6.621834660421394e-05,
+            -0.003983545180909168,
+            -0.010826144842427663,
+            -0.0028995230412062357,
+            0.0018029406642546938,
+            -0.01286603500476852,
+            0.00937188453692113,
+            -0.00635685147895941,
+            0.010242542894487791,
+            0.0062482230941642404,
+            -0.022974002980633818,
+            0.005235079251883088,
+            0.020160136045340955,
+            -0.008953506137454955,
+            0.0007027144749750591,
+            -0.015318432934167311,
+            -0.006155069535076527,
+            -8.304773776972749e-05,
+            0.010665400866776814,
+            0.013561082526851403,
+            0.019453190615040503,
+            0.008724224096424808,
+            0.017716608184445207,
+            0.0035170519664773133,
+            -0.013169473712817174,
+            0.009862672596286218,
+            0.011645070824075182,
+            0.011745045118639314,
+            0.011761235635339118,
+            -0.0051506969158396755,
+            0.00628898514628358,
+            -0.0049857719357619505,
+            0.008525048536699957,
+            0.027507594807546505,
+            0.0013209667681023782,
+            0.017047984110417937,
+            0.016368796085341072,
+            0.020105811276523177,
+            -0.003759327296302425,
+            3.4626946806238876e-05,
+            0.00035688794161846735,
+            -0.001063984476108484,
+            0.003291497239446139,
+            -0.018279629509070614,
+            -0.01135428882896519,
+            0.017033095140582537,
+            -5.492833742539044e-05,
+            0.024685530931127835,
+            -0.0010090516352729089,
+            -0.0184210790691242,
+            -0.006072072044463014,
+            0.0019766255786695347,
+            -0.0143465314042477,
+            -0.00209268265697904,
+            0.008867797876246725,
+            0.02090777280180668,
+            0.007785784409274212,
+            -0.014694559759226521,
+            0.03266241399157856,
+            0.017596572028431588,
+            0.02301544904486291,
+            -0.02019106568899837,
+            -0.008116927224141509,
+            -0.007678137378694171,
+            0.020318874678520133,
+            -0.0068566512882342135,
+            -0.002510061555870801,
+            0.021473513752723834,
+            0.006645783892214728,
+            0.011211922100954015,
+            0.012286872605920514,
+            0.0003084092164612384,
+            0.0028492628853667647,
+            -0.015161921733689642,
+            0.011855756525919763,
+            -0.005289433905822296,
+            -0.006904878573181685,
+            0.021517401941638887,
+            -0.010444313612915583,
+            -0.009104739247715266,
+            0.005434542886627656,
+            -0.011460046711120444,
+            0.013324525803587403,
+            -0.0016717915349416276,
+            -0.013225789210352699,
+            -0.014868291792978007,
+            -0.004059908595020105,
+            -0.017086039690741954,
+            0.018935544362274584,
+            0.006810606440007176,
+            -0.0016747502037076753,
+            -0.005949836930379681,
+            0.004577809518378731,
+            -0.002387527459083855,
+            -0.0006415824062693734,
+            -0.015923707971787795,
+            0.0035047065077239213,
+            0.009700583796235219,
+            -0.0031324165789907116,
+            0.010643108490671052,
+            -0.013353561795075988,
+            0.033779753889958034,
+            -0.003236875179603599,
+            0.0033068852921624326,
+            0.005106828066670305,
+            -0.003034255184307467,
+            0.001481657575335246,
+            0.013519816427440722,
+            0.009066434673988428,
+            -0.008339461460990311,
+            -0.0009211583209818409,
+            0.013497039042975171,
+            -0.018668722901146233,
+            0.0013796912320793106,
+            0.013796986913155082,
+            -0.015849997808276967,
+            0.0049427770053857125,
+            0.011400118227502903,
+            -0.00639345486098034,
+            0.015340608222895147,
+            -0.01371015934577743,
+            0.007008681477695071,
+            0.009775206231977338,
+            0.028490233702896408,
+            -0.01966685875736091,
+            0.01928758485650401,
+            -0.025635008258084145,
+            0.031283028009357144,
+            0.012122242939206308,
+            -0.005796910058436286,
+            -0.025521126197222552,
+            -0.011593941553769327,
+            0.01283377684428954,
+            -0.021686338476615465,
+            0.0014883363235708173,
+            0.004930128822274494,
+            -0.013589466747020829,
+            -0.014766714233107183,
+            -0.01202283835955618,
+            0.011738578705966321,
+            0.0006399892787922201,
+            0.01589633486439762,
+            -0.001565909739326631,
+            0.006195453613357957,
+            0.00799709051068248,
+            -0.007262779704915372,
+            -0.006717513726969218,
+            -0.01982600498555257,
+            0.01452043414139846,
+            0.004256754663990435,
+            -0.005262312998149605,
+            -0.0224549938750886,
+            0.009397743096285409,
+            0.017707219262338467,
+            -0.005753255460862127,
+            0.013047488144162397,
+            -0.03383526087792666,
+            0.0009452367103428395,
+            -0.005822570832057675,
+            -0.0027621026437538738,
+            -0.011741384193082396,
+            0.005215022845011562,
+            -0.015301048137828595,
+            0.002594097201808009,
+            -0.006446736651739076,
+            -0.016963075831388622,
+            0.002297125878220545,
+            0.02099207084363147,
+            -0.0032941933998622765,
+            -0.0028818889706579017,
+            0.01784874229400347,
+            0.008499720521713671,
+            -0.01436426626087195,
+            -0.004000179664908013,
+            0.006397941984552902,
+            -0.018873859156915206,
+            0.0011599963694687036,
+            0.02198209878751397,
+            0.012652203334020664,
+            -0.026378312815268385,
+            -0.0016397712646032201,
+            0.013833745513458322,
+            0.008230921549345205,
+            0.008667892596847433,
+            0.0040942632709300665,
+            0.014593014365768302,
+            -0.006350891941908016,
+            -0.001374093276890415,
+            -6.3005496707736715e-06,
+            -0.006943874751919358,
+            0.011912879044321251,
+            0.003688355066059739,
+            0.040741622477833674,
+            -0.010639328654064204,
+            0.011006744363538067,
+            -0.01080331270092793,
+            -0.004477468739500037,
+            -0.004503151926350927,
+            0.0027311865257027917,
+            -0.013139063275489816,
+            -0.0001322676781706513,
+            0.0021051078165230533,
+            -0.0019788024041678294,
+            0.00649268452600941,
+            0.003305903531478554,
+            0.013176851471483939,
+            -0.017229299828998786,
+            -0.0024884474570636707,
+            -0.009910839696360717,
+            0.019347666346683084,
+            -0.010724923526200773,
+            0.01051871403061451,
+            0.03256893707453101,
+            0.013242686936629418,
+            0.00022445440868234016,
+            -0.014947304198470308,
+            -0.008836459838385262,
+            -0.008992977994567497,
+            -0.01739161854615299,
+            -0.012908166671408042,
+            -0.022654409684101404,
+            -0.00294050967213434,
+            0.006390611639719366,
+            -0.0010555036392130776,
+            -0.002785796833161689,
+            -0.027340941664129582,
+            -0.00792516100590229,
+            -0.03210910565574123,
+            -0.013679170743667392,
+            0.0056845041078574104,
+            -0.021558178294402857,
+            0.005195998493021035,
+            0.029445968279023734,
+            -0.008256723194649932,
+            0.018403154314213917,
+            0.006923789005793937,
+            0.011433558571124386,
+            -0.0022324062216246424,
+            -0.011870704203994723,
+            -0.011035736680889242,
+            0.00032924465177187114,
+            0.003455564411617082,
+            0.004938023824535148,
+            0.00916834019939669,
+            -0.01732811783799426,
+            -0.010720420451770168,
+            0.00025951769018448803,
+            0.01732173296763328,
+            -0.0036441011455210317,
+            -0.014654182169779525,
+            -0.009890543675136605,
+            0.00934755871928037,
+            -0.017560953324850772,
+            0.0028803276180087,
+            0.008144645056336303,
+            -0.006844751836577803,
+            0.009686357271083834,
+            0.00910269542748879,
+            -0.003634975911996508,
+            -0.01125116892179341,
+            0.015123292946443665,
+            0.011417498955214564,
+            -0.0005683452620824292,
+            0.009547451569266946,
+            0.0031907584166949686,
+            -0.004914343407341038,
+            0.0004946517266559977,
+            -0.013981094295278457,
+            -0.008438177812073363,
+            -0.009698303451884585,
+            -0.0004873949117926506,
+            0.008310275049300703,
+            -0.011244097579407173,
+            0.002589333020166005,
+            0.021677389796728952,
+            0.0016179416889548232,
+            -0.011231602157838225,
+            -0.0045296258349024725,
+            -0.03198651985714177,
+            -0.005752566699226788,
+            -0.017524199510403467,
+            0.014980719037744585,
+            -0.002529640898744345,
+            0.01028755110586937,
+            -0.0021848874276835985,
+            0.015817733088597956,
+            -0.005450667684071798,
+            0.012212046004369696,
+            -0.0008714643856749736,
+            -0.004560548606687623,
+            0.006746171236132563,
+            0.0034423786329218276,
+            -0.0009995673054631,
+            0.05643134636651925,
+            0.010296478797092773,
+            0.014377556149215558,
+            0.013825497434998718,
+            -0.017485595187900386,
+            0.013330532940316836,
+            0.013333976202885402,
+            0.012417102160127097,
+            0.004536479576179834,
+            -0.017256228281240734,
+            0.024337236913894427,
+            0.012755805964855683,
+            0.009622694631579495,
+            -0.0018516129725815202,
+            -0.0015877067248695194,
+            0.019827793457522933,
+            0.018266833138588977,
+            -0.009808363592736615,
+            0.011083663940891296,
+            0.004458465015627857,
+            -0.01068421391840819,
+            -0.0050807436529387556,
+            0.024563333063926475,
+            -0.01325209293057542,
+            -0.016539133573613984,
+            0.0059168302415277515,
+            -0.02488439856049784,
+            -0.029755917745609557,
+            0.001920092403971772,
+            0.006364691645397147,
+            0.002128416061225683,
+            -0.0021827593501562285,
+            0.00636889353893684,
+            0.01505350329221805,
+            -0.02223481012881443,
+            -0.01913380824474348,
+            0.004876437698740333,
+            -0.0031323018872010133,
+            0.015711985535422395,
+            -0.02142499388661488,
+            -0.011646214273628826,
+            0.003244037052013272,
+            -0.006889665577990765,
+            0.004421548112756594,
+            0.004704830748153097,
+            -0.004955793799612646,
+            -0.005239972361031197,
+            0.0007828188358756139,
+            0.019667585310240637,
+            0.017115475760556154,
+            0.02008255444898991,
+            0.012206187649127842,
+            -0.004915394518125827,
+            -0.00997481157928988,
+            0.002664275191354035,
+            -0.017453428141032505,
+            -0.005114569347643741,
+            0.006659824028894164,
+            0.013233844584497872,
+            -0.023794827807567007,
+            -0.007354308951883939,
+            0.012842257789962674,
+            0.011294252132532885,
+            -0.00824836528452984,
+            0.004256336058369505,
+            -0.011193345143979632,
+            0.009248664100690892,
+            0.010363170772580968,
+            0.010981447978991254,
+            -0.01250390246078834,
+            0.008163162620777558,
+            0.002528845973804195,
+            0.007003579605495149,
+            -0.005812703866552746,
+            0.008764871082692903,
+            0.013124123077340675,
+            -0.012212447674686537,
+            -0.0013940507123620205,
+            -0.001544873017903471,
+            -0.020493969663982807,
+            0.005782189204516662,
+            0.007033733269347455,
+            0.012878926833341835,
+            -0.008338295354334415,
+            0.015844568159504493,
+            -0.013423240193733334,
+            -0.00468920516522183,
+            0.0063319500880227525,
+            -0.004924778574104498,
+            -0.012053197667708071,
+            0.0017742020845283848,
+            -0.0035640186446405536,
+            -0.008666749197879684,
+            -0.0013894177719374007,
+            0.0016400693271668141,
+            0.004501038858023909,
+            0.02043221461176279,
+            0.001901506572332553,
+            -0.0056076433740056926,
+            0.019660750419391273,
+            0.003538773720733224,
+            -0.0018659146421562808,
+            0.005354994676719758,
+            0.0006789623595561136,
+            -0.0021516503031452773,
+            -0.0027566542393029298,
+            -0.010552067426241374,
+            0.01699044655141112,
+            -0.01883386136434379,
+            -0.006428123254736374,
+            -0.012514829515253224,
+            0.006923832110523631,
+            0.023065674483579466,
+            -0.0031719793455512423,
+            -0.006131653355136594,
+            -0.015640226612880566,
+            0.003589774262421098,
+            -0.07234225978543331,
+            -0.010636515019878534,
+            0.005436135654579842,
+            0.030362090167699224,
+            -0.008735845879763853,
+            0.0012226978854869956,
+            -0.00802187367908004,
+            0.010460901462092434,
+            -0.0024797678489701584,
+            0.005049311815982583,
+            0.0037643360764659196,
+            -0.003379308997651945,
+            -0.0029803517660911536,
+            0.0047612984744651015,
+            -0.010726918354479238,
+            -0.018870259480662628,
+            0.0028549213021684043,
+            0.025034678854420084,
+            0.009004274035878184,
+            0.004138885468102832,
+            0.009765532829693857,
+            -0.008537379113922283,
+            -0.0014902115840393893,
+            0.008462743183234337,
+            0.007328272099011593,
+            -0.009076904245691212,
+            -0.0008002047467975742,
+            0.008347212876126893,
+            -0.021799374855548816,
+            0.0020253943807330738,
+            0.012246389250647798,
+            -0.008912465890948629,
+            -0.006244789880293639,
+            0.002413038795382646,
+            0.004202974630810528,
+            -0.004339960146144246,
+            -0.009919301413657872,
+            -0.023862183602380987,
+            0.006400371798756263,
+            -0.007410957922309058,
+            -0.0001606112434129549,
+            0.0031662502228666104,
+            0.00012871401736865864,
+            -0.01687187870925491,
+            0.01374600947628664,
+            -0.008834671549515985,
+            -0.0033344289547729495,
+            0.012937173759658114,
+            -0.003267867578821276,
+            0.01864135643890992,
+            -0.014050228437162755,
+            0.007864998326137568,
+            -0.018988002335416778,
+            -0.003944450943320823,
+            -0.0032084001089808037,
+            -0.0023149926514803745,
+            -0.007750311931438483,
+            -0.045019596490520515,
+            0.013573650825957857,
+            -0.004374086208559833,
+            -0.005869005771503507,
+            -0.01250037134701792,
+            -0.016625210662158708,
+            0.011639132814162461,
+            0.004045997475412364,
+            -0.0014885281342871186,
+            -0.020321856013396592,
+            0.006222318025479057,
+            0.011121009394355636,
+            0.00904225037228108,
+            -0.011585148988670874,
+            -0.0018474735072673144,
+            -0.011325387916132704,
+            -0.0025890917877355725,
+            -0.009148717095934154,
+            0.018495348432506,
+            0.002404017871470538,
+            -0.010232069372010613,
+            0.022941087706292104,
+            -0.0024570420644715457,
+            0.020988341201251853,
+            -0.00442363926883869,
+            -0.004485898239018673,
+            -0.020447941631096866,
+            -0.004158168168969816,
+            -0.019358617104349624,
+            0.00036464612100595666,
+            0.0017749921060331022,
+            0.02078748568527368,
+            0.007256675717496499,
+            0.012612722550842472,
+            0.006032799184116686,
+            0.00631983142330159,
+            0.030151438614535002,
+            -0.010852408231267204,
+            0.0022451749744724233,
+            -0.017863828302314567,
+            0.010653806941450981,
+            0.028289026813870025,
+            0.009882042879445822,
+            -0.006920783006708277,
+            0.014053717954960316,
+            0.004932716665921658,
+            -0.00616100713292227,
+            0.025975267677767474,
+            -0.038836632659249616,
+            -0.021059874446788137,
+            -0.0061122622002863615,
+            0.001439949038430725,
+            -3.496872289060673e-05,
+            0.005030571386336924,
+            -0.011007922189367617,
+            0.009222155733548683,
+            0.0026655561130744807,
+            0.03101907459085608,
+            -0.002317540524509921,
+            -0.007348120500263025,
+            0.003987101796623003,
+            0.001511347369284844,
+            0.021229069065914295,
+            -0.01197457000398312,
+            -0.038429200853184124,
+            -0.0031917847963042626,
+            -0.0039488592790253305,
+            0.008371872140716302,
+            -0.012355524487988753,
+            -0.010839396359961,
+            -0.004574476927098761,
+            -0.01882180934875166,
+            -0.010292867074240718,
+            0.025595330740853118,
+            -0.007162452868646569,
+            0.01788200594893745,
+            -0.0076384034977797525,
+            -0.008112046063674713,
+            0.00683079906571602,
+            0.011572398977364099,
+            -0.01932794114157753,
+            0.0034055065356524052,
+            -0.0027824032743918262,
+            0.01627342501880651,
+            0.011718826886349412,
+            -0.013342112644751174,
+            0.006234666245077415,
+            0.01365078200475753,
+            -0.015476695744659024,
+            -0.0015378873781631448,
+            -0.0069121070205125585,
+            0.014866073481143516,
+            -0.003093423905641203,
+            -0.0033762696975023986,
+            0.011958295888528918,
+            -0.021982979614167224,
+            0.01697002117600754,
+            -0.02183323542526832,
+            0.022295922951583082,
+            -0.0006195453455557405,
+            -0.0006320640234089136,
+            0.008423668060539851,
+            0.008314680408345411,
+            -0.013378756052181496,
+            0.0016194432891316378,
+            -0.03416272433582183,
+            -0.009962172336151867,
+            -0.015051382650973002,
+            0.0038009542554177348,
+            -0.010106196779819915,
+            -0.01203081704994182,
+            -0.021869748479536547,
+            -0.013231917312580397,
+            0.009488867268928662,
+            0.02421155136770049,
+            -0.004349181861466944,
+            0.015665735941744613,
+            -0.008077879618601087,
+            -0.00346140349979159,
+            -0.019022501109520248,
+            -0.020158795455478176,
+            -0.011854460388465065,
+            -0.01570428154296937,
+            -0.007073290993912972,
+            0.004794071576762192,
+            -0.014492538032767992,
+            0.013391092874532436,
+            0.03615243857749714,
+            -0.017029836110560772,
+            0.008609740828901341,
+            0.019179133878629843,
+            0.0009923228421593012,
+            0.015393048602064972,
+            0.003790868078763006,
+            0.011710499244312703,
+            -0.014885526287290077,
+            -0.011829126494406653,
+            -0.0005627435441977781,
+            0.0020582596099707785,
+            0.003728052108395647,
+            0.027914733124690644,
+            0.0038604865752177283,
+            0.005302574635039923,
+            0.012293011832771443,
+            0.013320598642264739,
+            -0.010706185765391238,
+            0.015853648683536427,
+            0.004090821159159474,
+            -0.016980287717038522,
+            -0.0031197192883179663,
+            -0.006894138060845641,
+            -0.012739354473335233,
+            -0.002311031608856898,
+            0.0159372988694303,
+            -0.008091532959711314,
+            0.015169701976466646,
+            -0.004316674756425857,
+            -0.015090694840235596,
+            0.0007287957651103694,
+            -0.03810971752015664,
+            -0.0018729979445690102,
+            -0.01301867399306695,
+            -0.023136174288569147,
+            0.04241471786668668,
+            0.00415464531754789,
+            0.020836251111427635,
+            -0.006703690210920993,
+            0.006147438331654669,
+            -0.020673361951422737,
+            0.011783676068789637,
+            -0.0037516561524537765,
+            0.009702463136945385,
+            0.009366943837185086,
+            -0.018879598518030567,
+            -0.01132474738893142,
+            0.018484010977000243,
+            0.0008964474295952695,
+            0.001899853004269779,
+            0.014452586238316783,
+            0.019746940887134506,
+            0.01136527620507647,
+            -0.0020960934961233306,
+            0.02093114800466563,
+            0.018023169361765227,
+            -0.0002475129435970239,
+            -0.007118840223520255,
+            0.013527604253725194,
+            -0.02531530780276405,
+            0.014601163833111726,
+            0.03268673940622878,
+            0.013556815596615061,
+            0.020396267313433348,
+            -0.016161264086739215,
+            0.016259608464481445,
+            -0.006718110409512467,
+            0.017378727943854342,
+            0.001211924791251576,
+            0.01911265175070918,
+            0.014022657097776944,
+            0.01308266544853342,
+            0.006784756253744209,
+            -0.013367188766230505,
+            0.024435211011756285,
+            -0.008342072323415817,
+            -0.0004209982405190692,
+            -0.002823543536521797,
+            -0.023279946325307846,
+            0.01506543259962226,
+            0.0017095020186682082,
+            0.003411744637061027,
+            -0.0035317155649076414,
+            -0.001957702240702538,
+            0.008037874596341196,
+            0.008176809711854463,
+            -0.015780325305129808,
+            0.0031090160698441006,
+            -0.01924979849975192,
+            -0.008566500568783712,
+            -0.024111269290772643,
+            0.003704817310374187,
+            -0.015182531891766549,
+            -0.009971369164255839,
+            -0.019503345629597463,
+            -0.011275291123690537,
+            -0.010129027262987365,
+            0.00026549411615768147,
+            -0.0077553590287942975,
+            0.014297350661562049,
+            -0.017119422159171904,
+            -0.021341131823214652,
+            0.007945720599865192,
+            0.01439604029848557,
+            0.0037256533134793816,
+            0.005472351446140726,
+            0.008325407758477418,
+            0.009082034215555286,
+            -0.010830069811539129,
+            -0.010352768708989164,
+            0.001902868536460922,
+            0.0010307757364948558,
+            0.011572880679973516,
+            0.01480960939553474,
+            0.011328449881374384,
+            -0.013429607187664061,
+            0.0019326261581148884,
+            -0.014862631307250641,
+            0.004686987038977201,
+            -0.0011432874921856958,
+            -0.007026108142521568,
+            -0.0023122147402109927,
+            -0.01067305755615663,
+            0.008329587642130216,
+            0.0051774766682198365,
+            0.0014058609424645312,
+            0.0175095191841222,
+            -0.0016739751490535205,
+            -0.01295313808457633,
+            -0.012767766732175707,
+            -0.011653259281510757,
+            -0.015653450921388538,
+            0.006739771869002279,
+            -0.006908534840842188,
+            -0.006309735406143942,
+            0.00969748644524724,
+            -0.004300978285892996,
+            -0.0071082865141610775,
+            0.009870599453242536,
+            0.0003007194077865931,
+            -0.0011846789159672513,
+            0.03344372215649544,
+            -0.004482311849310301,
+            0.004153681423879411,
+            0.00022516282287549753,
+            -0.00096173441700326,
+            -0.01608715959652527,
+            -0.026963713856310283,
+            -0.0014323662880336336,
+            -0.02273696636049197,
+            0.017938234567205,
+            -0.0058065575851078514,
+            -0.001006759040376689,
+            0.002493842516193206,
+            -0.0042740143630253385,
+            -6.58295494211919e-05,
+            -0.011768915967276026,
+            0.0022431022211178383,
+            -0.029090994997079067,
+            0.009898341521414653,
+            -0.009544941198003441,
+            -0.02154260971685405,
+            0.01133866603878075,
+            0.040150135462722274,
+            -0.0034082812055093122,
+            0.0006477463440760081,
+            0.0010662318198694314,
+            0.010346430535548179,
+            0.014953186668393513,
+            -0.0019239187336010772,
+            -0.01603518108536549,
+            -0.010550241019652155,
+            0.007810013487665037,
+            0.0003366114893320438,
+            0.0030211673366186986,
+            0.025036691996012227,
+            -0.019097101590128545,
+            -0.016027314610013832,
+            -0.011102216330632247,
+            -0.011450582072890358,
+            0.009254748792585386,
+            0.019696746434919966,
+            0.013185751991632784,
+            -0.020709092762886617,
+            -0.017120964154502993,
+            0.00025737692261907426,
+            -0.02662183259737709,
+            -0.0015175233130328546,
+            0.12569047385271798,
+            -0.004250752280902458,
+            -0.009469510190778525,
+            -0.016304049953896647,
+            -0.00863767582150345,
+            0.02183050591695681,
+            -0.008697910752438377,
+            0.06936580824413162,
+            0.0006861269173917291,
+            -0.0029257096931296273,
+            -0.0020896063525880225,
+            -0.002993306834763656,
+            -0.013871927123029641,
+            0.001864949832879178,
+            -0.016847962023455045,
+            -0.01637518023758693,
+            -0.006332388047537449,
+            -0.013325712872379694,
+            -0.0017155737214029734,
+            -0.0061798136617018905,
+            0.004759205228314821,
+            -0.007830001529147273,
+            0.011495001353183024,
+            -0.026107152853762815,
+            -0.010130127689151806,
+            0.005962957535201191,
+            -0.011637257947784772,
+            0.0011307756612654312,
+            -0.030816554116329904,
+            -0.00441741097001083,
+            7.409036337997905e-05,
+            0.016501246370776126,
+            0.0015748302561177006,
+            -0.00940138978060357,
+            -0.011216292373500447,
+            0.006508185064247167,
+            0.021984022174755303,
+            0.02029850289834145,
+            0.020629050747183544,
+            0.009741513502990842,
+            0.017418812192334685,
+            -0.007031919478404256,
+            0.02625468226435525,
+            0.006196721544177288,
+            0.002206946290342702,
+            0.005596780938975919,
+            -0.01211153541496144,
+            -0.008534755574472074,
+            0.0074101096913398124,
+            -0.0219026467376419,
+            -0.0238583746438936,
+            -0.012260621283748488,
+            0.025739429552984337,
+            -0.009054977254128021,
+            -0.0019759724570709175,
+            -0.0030737356218219617,
+            0.024727831360321823,
+            -0.005149317224364814,
+            0.002332136352952536,
+            0.015587381251136284,
+            -0.028035348333895063,
+            0.008643405636450286,
+            0.007622365990511162,
+            0.02564089666732,
+            -0.0032690983288409517,
+            0.0030897564314706603,
+            -0.009711753474608342,
+            -0.009285462125807194,
+            -0.009613024258506932,
+            0.004240295765592783,
+            0.018961596475867332,
+            0.005797211858953558,
+            0.009922313409983964,
+            -0.006335322307636977,
+            -0.015248479856831972,
+            0.018517674622994848,
+            -0.018526204881631703,
+            -0.017805480495530986,
+            -0.0031014632675223346,
+            -0.023499333544416315,
+            0.007508363381003107,
+            0.006594465447638183,
+            -0.013304818586245387,
+            -0.0232989529893049,
+            0.020407788403667446,
+            0.003426820233878876,
+            -0.006520960224510217,
+            0.0028020004149410366,
+            -0.03282622678411901,
+            -0.018099884128291804,
+            0.004341732255376245,
+            0.012442902960644058,
+            -0.006914419410521592,
+            0.01684289728480369,
+            -0.0213314411022478,
+            0.021170682961259797,
+            -0.018444729425471068,
+            0.015400297173773751,
+            -0.005856807329212788,
+            0.01661347113847185,
+            0.011889835104373263,
+            0.0013585650928227913,
+            -0.019926344687386053,
+            0.02233564528528766,
+            0.0006518460244140545,
+            0.029977976063273147,
+            0.026669882018303386,
+            -0.005623183408671759,
+            -0.004497323576380724,
+            -0.017964594171161242,
+            0.011149712487916564,
+            0.0034380718823694635,
+            0.00295265329839474,
+            0.023835186931253683,
+            0.02803963164846445,
+            -0.028097235508129332,
+            0.017703040589712025,
+            -0.012893023698870347,
+            0.0089348477172039,
+            -0.011630331709306501,
+            0.0096768368423985,
+            0.00669273666692988,
+            0.003825890215203911,
+            0.008289398654583944,
+            0.011807275797649763,
+            0.024893949220958504,
+            0.02893733381281628,
+            0.004069052493622539,
+            0.012465630768347759,
+            -0.003600044227856378,
+            0.004954707230784128,
+            0.005985890100113585,
+            0.0014569711209781388,
+            -0.00648453455551435,
+            0.0031945554565833674,
+            -0.006090168119075504,
+            -0.003744222112779555,
+            0.015266602618825367,
+            -0.00028966361595811053,
+            0.015634624351841753,
+            0.012395295870300452,
+            -0.006577117978669073,
+            -0.003069372773552921,
+            -0.0029369830414480856,
+            -0.007598245840968154,
+            -0.017753359726409172,
+            0.025057809370078198,
+            -0.01606095813358513,
+            -0.019631204097884002,
+            0.0008782653868466668,
+            0.0031106969086903075,
+            -0.013675045629189149,
+            -0.008463196400259825,
+            0.008777824363912681,
+            0.014471929439779247,
+            -0.018478175346087953,
+            0.0017862044141924377,
+            -0.0004910793014860654,
+            0.013862986726603731,
+            -0.003641460118879092,
+            -0.00902817956483427,
+            -0.002076665354355939,
+            0.010393957588184604,
+            0.016413601414994997,
+            0.0038017609859922327,
+            -0.008644753777502864,
+            -0.0005484639867925445,
+            0.006452371249117759,
+            0.007391332924183688,
+            0.0036307379469783137,
+            -4.895894824382588e-05,
+            -0.008129811261078888,
+            -0.013655410823646597,
+            0.018829640339982075,
+            -0.007579460691893014,
+            0.016191139169828096,
+            -0.004984562798832502,
+            -0.011846989328030879,
+            0.004110855640416496,
+            0.0004775146965349116,
+            -0.014308885325290177,
+            0.002778951512640393,
+            -0.003583348080466357,
+            0.0017351729844533707,
+            0.006444402813802743,
+            -0.004945263366121632,
+            0.003803172980003619,
+            -0.008178062077842394,
+            0.02700102856098108,
+            0.015477045085093356,
+            0.005818362056942523,
+            -0.00876269298951217,
+            -0.024697543974427114,
+            0.008183329610246195,
+            0.00518473714350693,
+            0.018661120751406378,
+            0.014464207183253898,
+            0.0019868192486975638,
+            0.009827229842256728,
+            -0.013395371817208287,
+            -0.0022376892192351944,
+            0.009711801873474911,
+            0.017963581533215926,
+            -0.0059700736266576305,
+            0.019169874693002874,
+            -0.010316234316657931,
+            -0.0008125759237295765,
+            0.004003428870809722,
+            0.017109043212624506,
+            -0.004590883969173384,
+            0.013958072401225709
+          ],
+          "title": "F Pseudo Code for Switch Transformers 33",
+          "keyphrases": [
+            "Switch Transformers",
+            "Mixture-of-Experts",
+            "sparsely-activated expert model",
+            "natural language tasks",
+            "pre-training and fine-tuning"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": true,
+      "properties": {
+        "cosine_similarity": 0.9351401844718813
+      }
+    },
+    {
+      "id": "d89024b8-3a99-4c2a-a265-614d11f687ea",
+      "type": "cosine_similarity",
+      "source": {
+        "id": "8bd19de5-c2d4-458d-833b-5962c425f915",
+        "properties": {
+          "page_content": "1 Introduction\n\nScaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27]. The final model quality has been found to have a power-law relationship with the amount of data, model size, and compute time [16, 20]. However, training efficiency, which is defined as the total amount of computation used to achieve superior model quality than the state of the art system [21], should receive greater attention as we increase our efforts towards green AI [29].\n\nSparsely gated mixture-of-experts [31] (MoE) provides an effective way to scale model capacity given a fixed computational cost, and has recently played an important role in increasing the training efficiency of large-scale language models [10, 21]. MoE operate by adopting a number of experts, each as a sub-network, and by activating only one or a few experts for each input token. A gating network must be chosen and optimized in order to route each token to the most suited expert(s). For example, recent work has implemented sparse routing via k-means clustering [12], linear assignment to maximize token-expert affinities [22], or hashing [8, 28]. Many of the prior work use a routing strategy concerning the token choice, where each token selects the best one or two experts.\n\nWe argue that the independent token choice of prior work often leads to an imbalanced load of experts, which causes training inefficiency and sub-optimal training of the model. In order to mitigate this\n\nissue, previous sparsely gated networks introduce additional auxiliary losses as regularization to prevent too many tokens being routed to a single expert, but the effectiveness is still limited. Recent approaches [8, 22, 28] explore alternative strategies for routing, but they focus on pre-training only and do not demonstrate performance gain on downstream tasks. Moreover, none of the previous methods consider allocating a variable number of experts to each token based on importance, which can be beneficial.\n\nWe propose a very simple yet effective routing method we are calling expert choice. Unlike conventional MoE where tokens select one or two top-scoring experts, our method lets each expert pick the top-k tokens. Our method guarantees perfect load balancing, allows a variable number of experts for each token, and achieves substantial gains in training efficiency and downstream performance as demonstrated in our experiments. Our major contributions include:\n\nWe identify common pitfalls in conventional MoE such as load imbalance as described in Section 3.1. We then propose a heterogeneous, expert choice method to provide a fluid allocation of model parameters based on a learnt token-to-expert importance. This method intrinsically guarantees load balance without imposing an auxiliary loss.\n\nWe show our method provides over 2× faster training convergence in a 8B/64E (8 billion activated parameters, 64 experts) model, compared to the top-1 and top-2 gating counterparts in Switch Transformer [10] and GShard [21].\n\nWe show our method demonstrates strong scaling when increasing the number of experts from 16 to 128, evaluated in training perplexity.\n\nWe show our method demonstrates strong performance on downstream tasks selected from GLUE and SuperGLUE at all the evaluated scales. More specifically, our 8B/64E model outperforms a T5 11B dense model in 7 out of 11 tasks evaluated.\n\n",
+          "title": "Introduction",
+          "embedding": [
+            0.0024422307033091784,
+            -0.014919951558113098,
+            -0.01768755167722702,
+            -0.02871091105043888,
+            0.020494353026151657,
+            0.03838575258851051,
+            -0.01718577742576599,
+            0.0458810068666935,
+            -0.00809895433485508,
+            0.02212512120604515,
+            -0.01569613441824913,
+            -0.011674097739160061,
+            0.024147899821400642,
+            -0.016072465106844902,
+            -0.003586903912946582,
+            0.029040200635790825,
+            0.00016084715025499463,
+            0.0012985375942662358,
+            0.021921275183558464,
+            -0.021889913827180862,
+            -0.0052411919459700584,
+            -0.014426017180085182,
+            -0.01895766891539097,
+            -0.002695078030228615,
+            0.025621861219406128,
+            0.010239336639642715,
+            0.029526295140385628,
+            -0.00020776601741090417,
+            -0.05397212132811546,
+            0.04054965451359749,
+            0.015868619084358215,
+            0.025010323151946068,
+            -0.020776601508259773,
+            -0.014253532513976097,
+            -0.047417692840099335,
+            -0.029228366911411285,
+            0.05165141448378563,
+            0.018314769491553307,
+            -0.027127185836434364,
+            0.030765051022171974,
+            0.0260295532643795,
+            0.018973348662257195,
+            -0.037350840866565704,
+            -0.009784603491425514,
+            0.0013112779706716537,
+            0.00785982795059681,
+            -0.029761500656604767,
+            0.00613497756421566,
+            0.022031037136912346,
+            -0.01818932592868805,
+            0.02342659793794155,
+            -0.01548444852232933,
+            0.015492288395762444,
+            0.006064415443688631,
+            -0.06635968387126923,
+            0.04058101400732994,
+            0.0031008098740130663,
+            0.008036232553422451,
+            0.05513247475028038,
+            0.019961217418313026,
+            0.018205007538199425,
+            -0.0099727688357234,
+            -0.04525379091501236,
+            0.008130315691232681,
+            0.006036974955350161,
+            -0.019506484270095825,
+            -0.008647770620882511,
+            0.015061075799167156,
+            0.023410918191075325,
+            0.01010605227202177,
+            -0.055038392543792725,
+            0.029275408014655113,
+            -0.019929856061935425,
+            0.005562641192227602,
+            0.006781796459108591,
+            -0.01776595413684845,
+            0.01925559714436531,
+            0.01078031212091446,
+            0.005041265860199928,
+            0.0031556913163512945,
+            -0.01123504526913166,
+            0.038072142750024796,
+            0.004821739625185728,
+            0.014802348800003529,
+            0.04224314168095589,
+            -0.03951474279165268,
+            0.0034300992265343666,
+            -0.06153010204434395,
+            0.01611950621008873,
+            -0.012019068002700806,
+            -0.030733689665794373,
+            -0.003298775525763631,
+            -0.01593918167054653,
+            0.017828676849603653,
+            -0.003900512820109725,
+            0.041521843522787094,
+            0.011423210613429546,
+            -0.006993482355028391,
+            -0.02982422336935997,
+            -0.0009119163732975721,
+            -0.005151029210537672,
+            0.017985479906201363,
+            0.0025931550189852715,
+            -0.023175710812211037,
+            0.019396722316741943,
+            -0.03333664685487747,
+            -0.033023037016391754,
+            0.035281021147966385,
+            0.017907077446579933,
+            0.05739046260714531,
+            0.024100856855511665,
+            -0.011509452946484089,
+            0.011870103888213634,
+            0.025621861219406128,
+            -0.02924404665827751,
+            -0.017279859632253647,
+            -0.03810350224375725,
+            -0.06316086649894714,
+            0.019020389765501022,
+            -0.013657675124704838,
+            0.01680944673717022,
+            0.006762195844203234,
+            -0.012716848403215408,
+            -0.008624250069260597,
+            -0.06369400024414062,
+            0.045347873121500015,
+            -0.014026165939867496,
+            -0.011352648958563805,
+            -0.0018924347823485732,
+            -0.02450854890048504,
+            -0.003341896692290902,
+            0.028397301211953163,
+            -0.03418339043855667,
+            -0.05814312398433685,
+            0.040330126881599426,
+            0.013148060999810696,
+            0.03534374386072159,
+            0.04365438222885132,
+            -0.015782376751303673,
+            -0.022234883159399033,
+            0.026217719539999962,
+            -0.035124216228723526,
+            0.007946070283651352,
+            -0.0005517559475265443,
+            -0.02668813243508339,
+            0.01322646252810955,
+            0.010011970065534115,
+            0.06447802484035492,
+            -0.02052571438252926,
+            0.021889913827180862,
+            -0.02918132394552231,
+            0.057108212262392044,
+            -0.016260631382465363,
+            0.02201535739004612,
+            -0.023018905892968178,
+            -0.0031008098740130663,
+            -0.031219784170389175,
+            0.0018375532235950232,
+            -0.019584886729717255,
+            -0.011485932394862175,
+            -0.02443014644086361,
+            -0.01301477663218975,
+            -0.007569739129394293,
+            0.01749938726425171,
+            0.010686228983104229,
+            0.024931922554969788,
+            0.030827771872282028,
+            0.002036498859524727,
+            -0.07018571346998215,
+            0.026468606665730476,
+            0.018205007538199425,
+            0.01676240563392639,
+            0.01731122098863125,
+            0.019004710018634796,
+            0.006217299960553646,
+            0.026656771078705788,
+            0.011148802936077118,
+            0.011195844039320946,
+            -0.008059753105044365,
+            0.03985971212387085,
+            -0.03716267645359039,
+            -0.007585419807583094,
+            0.01726417988538742,
+            0.0046766954474151134,
+            0.026249079033732414,
+            0.018079563975334167,
+            0.01726417988538742,
+            0.012748208828270435,
+            0.007448215503245592,
+            0.01121936459094286,
+            0.00966699980199337,
+            0.03672362491488457,
+            0.01779731549322605,
+            -0.01638607494533062,
+            0.016433116048574448,
+            -0.02817777544260025,
+            -0.01050590444356203,
+            -0.004990304354578257,
+            0.02722126804292202,
+            0.00673867529258132,
+            -0.02339523658156395,
+            -0.0031027698423713446,
+            0.009196585975587368,
+            -0.022532811388373375,
+            0.008702651597559452,
+            0.006174178794026375,
+            -6.774446228519082e-05,
+            0.009525875560939312,
+            0.007271810434758663,
+            -0.030043749138712883,
+            0.023630443960428238,
+            -0.01404968649148941,
+            -0.008726172149181366,
+            -0.01792275905609131,
+            -0.007032683584839106,
+            -0.000910936389118433,
+            0.004002436064183712,
+            0.006323142908513546,
+            0.006664192769676447,
+            -0.022501451894640923,
+            -0.004837419837713242,
+            0.011454571969807148,
+            0.02488487958908081,
+            0.002355988137423992,
+            -0.0009266168344765902,
+            -0.020353229716420174,
+            -0.006664192769676447,
+            -0.01840885356068611,
+            0.04258811101317406,
+            -0.005386236123740673,
+            -0.03223901242017746,
+            0.009902206249535084,
+            0.014849389903247356,
+            0.010200135409832,
+            0.010286377742886543,
+            -0.042556751519441605,
+            0.00047947888378985226,
+            0.0019267357420176268,
+            0.005268632434308529,
+            -0.01114096213132143,
+            -0.019114473834633827,
+            0.019882814958691597,
+            -0.051431890577077866,
+            -0.014347615651786327,
+            0.029306767508387566,
+            0.02737807296216488,
+            -0.020070981234312057,
+            0.01941240206360817,
+            -0.007851987145841122,
+            -0.004453248810023069,
+            -0.0246967151761055,
+            -0.024336064234375954,
+            0.01234835758805275,
+            0.014378976076841354,
+            -0.0033595373388379812,
+            0.03434019535779953,
+            0.016213588416576385,
+            -0.034089308232069016,
+            -0.017091695219278336,
+            -0.0149669935926795,
+            0.050052009522914886,
+            -0.013006936758756638,
+            -0.015194360166788101,
+            0.06372536718845367,
+            -0.0255748201161623,
+            0.012622765265405178,
+            0.03515557944774628,
+            0.014684745110571384,
+            -0.002234464744105935,
+            -0.024947602301836014,
+            -0.0025970751885324717,
+            -0.05425437167286873,
+            -0.0051588695496320724,
+            -0.013720396906137466,
+            0.04224314168095589,
+            -0.04142775759100914,
+            -0.004076918121427298,
+            -0.005880170036107302,
+            0.01577453687787056,
+            0.03024759516119957,
+            0.006785716395825148,
+            0.00034153988235630095,
+            0.01049022376537323,
+            0.02308162860572338,
+            -0.0034457796718925238,
+            -0.014465218409895897,
+            -0.005151029210537672,
+            -0.018471574410796165,
+            -0.04838988184928894,
+            -0.011046879924833775,
+            -0.01742098480463028,
+            -0.004813899286091328,
+            -0.031454991549253464,
+            -0.00391815323382616,
+            0.03531238064169884,
+            -0.010647027753293514,
+            -0.015421726740896702,
+            0.008522327058017254,
+            0.00809895433485508,
+            -0.022877782583236694,
+            -0.04719816520810127,
+            0.004974624142050743,
+            0.00666027283295989,
+            -0.011321287602186203,
+            -0.0572650171816349,
+            0.006750435568392277,
+            -0.02461831271648407,
+            0.01837749220430851,
+            -0.021999677643179893,
+            -0.0099727688357234,
+            0.025872748345136642,
+            0.0016327272169291973,
+            0.0045120506547391415,
+            -0.02573162503540516,
+            0.0011064520804211497,
+            0.022595534101128578,
+            0.006699474062770605,
+            0.021105891093611717,
+            -0.027958249673247337,
+            -0.03418339043855667,
+            0.020839322358369827,
+            -0.0055077592842280865,
+            -0.06350583583116531,
+            -0.0026676373090595007,
+            -0.02251713164150715,
+            0.0027009581681340933,
+            0.019161514937877655,
+            0.04694727808237076,
+            -0.030184874311089516,
+            0.008663450367748737,
+            -0.023646123707294464,
+            -0.04650822654366493,
+            -0.030467122793197632,
+            -0.0027303590904921293,
+            -0.007957830093801022,
+            -0.026547009125351906,
+            -0.012991256080567837,
+            0.008467445150017738,
+            -0.03901296854019165,
+            0.009274988435208797,
+            -0.014245692640542984,
+            -0.026233399286866188,
+            -0.028648190200328827,
+            0.010427501983940601,
+            0.00468061538413167,
+            -0.002107060980051756,
+            -0.03603368252515793,
+            0.0060722557827830315,
+            0.004939342848956585,
+            -0.005162789486348629,
+            0.0010476503521203995,
+            0.002673517446964979,
+            0.004296444356441498,
+            0.005880170036107302,
+            -0.0027538796421140432,
+            -0.04183544963598251,
+            -0.026750855147838593,
+            0.016072465106844902,
+            0.02090204507112503,
+            -0.001751310657709837,
+            0.029197005555033684,
+            -0.050898753106594086,
+            -0.03230173513293266,
+            0.025951150804758072,
+            -0.06002477928996086,
+            0.006272181402891874,
+            -0.011995547451078892,
+            -0.022376008331775665,
+            -0.00047531374730169773,
+            -0.014073207043111324,
+            -0.02063547819852829,
+            0.026249079033732414,
+            0.02626476064324379,
+            0.017624830827116966,
+            0.017938438802957535,
+            0.013336226344108582,
+            0.02342659793794155,
+            -0.02220352366566658,
+            -0.06391353160142899,
+            0.035625990480184555,
+            -0.01566477306187153,
+            0.007781425025314093,
+            0.017860036343336105,
+            -0.012536522932350636,
+            -0.004010275937616825,
+            0.0303573589771986,
+            0.011713298968970776,
+            0.004022036213427782,
+            -0.018346130847930908,
+            0.05221591144800186,
+            -0.005986013449728489,
+            -0.04886029288172722,
+            -0.0021325417328625917,
+            -0.012120991013944149,
+            0.023489320650696754,
+            0.023348195478320122,
+            -0.00301064713858068,
+            -0.028742272406816483,
+            -0.022830741479992867,
+            -0.007197328377515078,
+            -0.015194360166788101,
+            0.02798961102962494,
+            -0.00789902824908495,
+            0.01466122455894947,
+            -0.0194908045232296,
+            -0.001328918500803411,
+            -0.004104358609765768,
+            0.01139185018837452,
+            0.04287036135792732,
+            -0.040643736720085144,
+            -0.009235787205398083,
+            -0.03361889347434044,
+            -0.03261534497141838,
+            0.00725613022223115,
+            0.017562108114361763,
+            0.03844847157597542,
+            -0.02182719111442566,
+            0.01078031212091446,
+            0.01853429712355137,
+            -0.010349099524319172,
+            -0.012418919242918491,
+            0.03393250331282616,
+            0.04873485118150711,
+            0.03452835977077484,
+            0.011956346221268177,
+            -0.00046306339208967984,
+            0.008992739953100681,
+            0.011415370739996433,
+            0.01152513362467289,
+            0.012967735528945923,
+            0.029338128864765167,
+            -0.008067593909800053,
+            0.006969961803406477,
+            -0.018126605078577995,
+            -0.020118022337555885,
+            -0.01611950621008873,
+            -0.011713298968970776,
+            0.030529843643307686,
+            -0.011768180876970291,
+            -0.005782167427241802,
+            0.0294792540371418,
+            -0.008498805575072765,
+            0.00907114241272211,
+            -0.009392592124640942,
+            -0.023755887523293495,
+            -0.03085913322865963,
+            -0.04202361777424812,
+            -0.005292152985930443,
+            -0.007753984536975622,
+            -0.035092856734991074,
+            -0.0529058501124382,
+            0.010380460880696774,
+            -0.07514073699712753,
+            -0.002481431933119893,
+            0.029965346679091454,
+            0.0010692109353840351,
+            0.0008305740193463862,
+            -0.0010888115502893925,
+            0.008208717219531536,
+            -0.03763309121131897,
+            0.0031478512100875378,
+            0.02243872918188572,
+            0.03089049458503723,
+            -0.0294792540371418,
+            -0.010991998016834259,
+            0.007103245705366135,
+            -0.03879344463348389,
+            -0.0038828724063932896,
+            0.0039887153543531895,
+            0.005190230440348387,
+            0.013312705792486668,
+            -0.03227037563920021,
+            -0.0069033196195960045,
+            0.004982464015483856,
+            0.01630767248570919,
+            0.03835438936948776,
+            -0.0023677486460655928,
+            -0.04512834548950195,
+            0.019004710018634796,
+            -0.0007732423837296665,
+            0.01531196292489767,
+            -0.00893785897642374,
+            -0.020839322358369827,
+            -0.0015778456581756473,
+            -0.00539799639955163,
+            0.0006272181635722518,
+            0.023332515731453896,
+            -0.008530166931450367,
+            0.01787571795284748,
+            -0.01707601360976696,
+            -0.015210039913654327,
+            -0.002385389059782028,
+            -0.006679873447865248,
+            0.018863586708903313,
+            0.0031125701498240232,
+            -0.021121570840477943,
+            -0.026092275977134705,
+            -0.017217138782143593,
+            0.027503516525030136,
+            0.0035927840508520603,
+            -0.040298767387866974,
+            -0.012795249931514263,
+            0.024289023131132126,
+            0.019726011902093887,
+            0.006613231264054775,
+            -0.019616248086094856,
+            0.008208717219531536,
+            0.03024759516119957,
+            -0.044657934457063675,
+            -0.02522985078394413,
+            0.0026931180618703365,
+            0.0033066156320273876,
+            -0.004888381343334913,
+            -0.011148802936077118,
+            0.0006757295341230929,
+            0.04365438222885132,
+            -0.010576466098427773,
+            0.021717429161071777,
+            0.019631927832961082,
+            0.01684080809354782,
+            -0.01898903027176857,
+            -0.013649835251271725,
+            -0.011454571969807148,
+            0.025010323151946068,
+            -0.014669064432382584,
+            0.027205588296055794,
+            0.022219203412532806,
+            0.002944005187600851,
+            0.06447802484035492,
+            -0.026672452688217163,
+            -0.025559140369296074,
+            0.011705459095537663,
+            -0.013579272665083408,
+            0.0009697380592115223,
+            -0.019475122913718224,
+            -0.007812785916030407,
+            -0.004092598333954811,
+            0.022109439596533775,
+            -0.028914757072925568,
+            -0.028507065027952194,
+            0.0017552308272570372,
+            0.010129572823643684,
+            -0.009925727732479572,
+            0.01257572416216135,
+            0.0027950408402830362,
+            0.004539491608738899,
+            0.0212470144033432,
+            -0.04149048030376434,
+            0.011681938543915749,
+            0.035469185560941696,
+            -0.02477511763572693,
+            -0.034402914345264435,
+            -0.031219784170389175,
+            0.011862263083457947,
+            0.0029890865553170443,
+            -0.04434432461857796,
+            -0.015037555247545242,
+            -0.029996708035469055,
+            -0.0015660853823646903,
+            -0.03572007268667221,
+            -0.00958859734237194,
+            0.010082531720399857,
+            -0.0037143074441701174,
+            0.01183090265840292,
+            -0.022987544536590576,
+            -0.005445037502795458,
+            0.028852034360170364,
+            0.04478337615728378,
+            0.012168032117187977,
+            0.020337548106908798,
+            -0.001616066787391901,
+            -0.024978963658213615,
+            -0.015492288395762444,
+            0.012599244713783264,
+            -0.0007776524871587753,
+            0.0031968525145202875,
+            0.0294478926807642,
+            -0.004907981958240271,
+            -0.03763309121131897,
+            0.00632706331089139,
+            0.04095734655857086,
+            -0.020572755485773087,
+            0.007146366871893406,
+            0.04192953556776047,
+            0.025935471057891846,
+            -0.019239917397499084,
+            -0.020008258521556854,
+            -0.031298186630010605,
+            0.004951103124767542,
+            -0.02817777544260025,
+            -0.005201990716159344,
+            0.03688042610883713,
+            -0.006185939069837332,
+            -0.026390204206109047,
+            0.012262115254998207,
+            -0.02297186478972435,
+            -0.009337710216641426,
+            0.01278741005808115,
+            0.01100767869502306,
+            -0.03625321015715599,
+            0.013328385539352894,
+            0.019537845626473427,
+            0.023332515731453896,
+            -0.008882977068424225,
+            -0.018973348662257195,
+            0.021074529737234116,
+            0.006166338454931974,
+            0.0032301736064255238,
+            -0.004202361684292555,
+            0.0075501385144889355,
+            0.019773053005337715,
+            -0.0036045443266630173,
+            0.011305606923997402,
+            -0.0034300992265343666,
+            0.03317984193563461,
+            -0.014473059214651585,
+            -0.002887163544073701,
+            0.01188578363507986,
+            -0.014622023329138756,
+            -0.009314189665019512,
+            0.059523001313209534,
+            0.007213008590042591,
+            0.040486931800842285,
+            -0.003974995110183954,
+            0.018847905099391937,
+            0.024633992463350296,
+            -0.01627631112933159,
+            -0.027205588296055794,
+            0.0006580890621989965,
+            -0.013869361020624638,
+            -0.03085913322865963,
+            -0.01699761115014553,
+            -0.03534374386072159,
+            -0.018471574410796165,
+            -0.00014565671153832227,
+            -0.01404968649148941,
+            0.0007394314161501825,
+            0.028428662568330765,
+            0.010239336639642715,
+            -0.012168032117187977,
+            -0.033901140093803406,
+            0.03885616362094879,
+            0.030874812975525856,
+            0.03261534497141838,
+            -0.007236529607325792,
+            0.019271278753876686,
+            -0.010074691846966743,
+            -0.0058135283179581165,
+            0.020807962864637375,
+            0.015437406487762928,
+            -0.00238342909142375,
+            0.014825869351625443,
+            0.01577453687787056,
+            0.0459437295794487,
+            -0.025621861219406128,
+            -0.03024759516119957,
+            -0.010176614858210087,
+            -0.0260295532643795,
+            0.027817124500870705,
+            0.012983416207134724,
+            -0.015445247292518616,
+            0.015727495774626732,
+            0.017044654116034508,
+            -0.029620377346873283,
+            0.02010234259068966,
+            -0.01374391745775938,
+            -0.03195676580071449,
+            0.030325997620821,
+            -0.016825126484036446,
+            0.013242143206298351,
+            -0.007013082969933748,
+            -0.004033796954900026,
+            -0.020086660981178284,
+            0.009298508986830711,
+            0.01131344772875309,
+            -0.028554106131196022,
+            0.038072142750024796,
+            0.04368574544787407,
+            0.003083169227465987,
+            0.009604278020560741,
+            0.014574982225894928,
+            -0.0007943129749037325,
+            -0.020353229716420174,
+            -0.012724688276648521,
+            0.006679873447865248,
+            -0.001151533331722021,
+            -0.028836354613304138,
+            0.0307807307690382,
+            -0.016103826463222504,
+            0.007495257072150707,
+            -0.01154081430286169,
+            -0.006087936460971832,
+            0.0063192229717969894,
+            -0.0007805926143191755,
+            0.006558349821716547,
+            0.0007997031207196414,
+            -0.03170587867498398,
+            0.014888591133058071,
+            0.031110020354390144,
+            -0.005758646875619888,
+            -0.01079599279910326,
+            0.01737394370138645,
+            -0.02090204507112503,
+            0.009180906228721142,
+            0.028005290776491165,
+            0.015186519362032413,
+            0.004206281621009111,
+            -0.0051980703137815,
+            -0.030310317873954773,
+            -0.020243465900421143,
+            -0.001436721533536911,
+            0.024304702877998352,
+            0.014841550029814243,
+            0.026437245309352875,
+            0.026735173538327217,
+            -0.00780886597931385,
+            0.0013436188455671072,
+            -0.0022677856031805277,
+            -0.03396386280655861,
+            0.04440704733133316,
+            -1.7808952179620974e-05,
+            0.01779731549322605,
+            0.021152932196855545,
+            -0.027095824480056763,
+            -0.027268309146165848,
+            -0.001135852886363864,
+            0.032333094626665115,
+            0.03484196960926056,
+            0.01699761115014553,
+            0.004590453114360571,
+            0.04202361777424812,
+            -0.02464967407286167,
+            -0.03741356357932091,
+            0.017891397699713707,
+            0.030043749138712883,
+            -0.009188746102154255,
+            0.0016738884150981903,
+            0.027017422020435333,
+            -0.011838742531836033,
+            -0.005574401468038559,
+            0.024367425590753555,
+            0.019083112478256226,
+            0.008914338424801826,
+            0.004104358609765768,
+            -0.009659159928560257,
+            0.02369316667318344,
+            0.002107060980051756,
+            0.024336064234375954,
+            -0.01960056647658348,
+            0.007318852003663778,
+            0.00945531390607357,
+            -0.026891978457570076,
+            0.033023037016391754,
+            0.009941407479345798,
+            -0.0502401739358902,
+            0.025668902322649956,
+            0.0032517341896891594,
+            -0.004304284695535898,
+            0.003394818166270852,
+            0.0059076109901070595,
+            0.007789265364408493,
+            0.023363877087831497,
+            0.00446108914911747,
+            -0.004096518736332655,
+            0.030106471851468086,
+            0.023912692442536354,
+            -0.0003197342448402196,
+            0.020666837692260742,
+            -0.0019110552966594696,
+            -0.02074524015188217,
+            -0.002781320596113801,
+            -0.0063780248165130615,
+            0.04105142876505852,
+            -0.0390443317592144,
+            0.014825869351625443,
+            0.02374020777642727,
+            0.003420298919081688,
+            -0.002857762621715665,
+            -0.02320707216858864,
+            -0.008443924598395824,
+            -0.011180163361132145,
+            0.024257661774754524,
+            -0.010145253501832485,
+            0.023301154375076294,
+            -0.02833458036184311,
+            -0.0038926727138459682,
+            0.01353223156183958,
+            -0.030498482286930084,
+            0.011031199246644974,
+            0.02722126804292202,
+            0.0010897915344685316,
+            -0.02010234259068966,
+            -0.005852729547768831,
+            0.036849066615104675,
+            0.055226556956768036,
+            0.0054959990084171295,
+            0.01630767248570919,
+            0.02867954969406128,
+            -0.02140381932258606,
+            -0.0106235072016716,
+            0.019537845626473427,
+            -0.018126605078577995,
+            0.014582822099328041,
+            0.04302716627717018,
+            -0.036347292363643646,
+            -0.03782125562429428,
+            -0.008130315691232681,
+            -0.032333094626665115,
+            0.008898657746613026,
+            -0.012003387324512005,
+            0.012889333069324493,
+            -0.009204426780343056,
+            -0.02841298282146454,
+            0.02798961102962494,
+            0.015170839615166187,
+            0.009384751319885254,
+            -0.014622023329138756,
+            -0.02182719111442566,
+            -0.02411653846502304,
+            -0.031078658998012543,
+            -0.008898657746613026,
+            -0.01765619032084942,
+            0.01975737139582634,
+            -0.0006556389853358269,
+            -0.04224314168095589,
+            -0.007848067209124565,
+            0.0194908045232296,
+            -0.012120991013944149,
+            0.040675096213817596,
+            0.02416357956826687,
+            0.016558559611439705,
+            -0.009267148561775684,
+            -0.03760172799229622,
+            0.05425437167286873,
+            -2.794612191792112e-05,
+            0.03027895651757717,
+            -0.013179421424865723,
+            -0.007655981462448835,
+            0.0026107956655323505,
+            0.027958249673247337,
+            0.005201990716159344,
+            0.009470994584262371,
+            -0.031909722834825516,
+            0.011078240349888802,
+            -0.024304702877998352,
+            -0.011932825669646263,
+            -0.006899399682879448,
+            -0.0034673402551561594,
+            0.02450854890048504,
+            0.00416316045448184,
+            0.006217299960553646,
+            0.02875795215368271,
+            0.012599244713783264,
+            -0.0017483705887570977,
+            -0.023834289982914925,
+            -0.013493030332028866,
+            -0.0015964661724865437,
+            0.003849551547318697,
+            -0.01676240563392639,
+            -0.013053977862000465,
+            -0.03835438936948776,
+            -0.010748950764536858,
+            0.0034810607321560383,
+            0.021152932196855545,
+            0.02151358313858509,
+            0.005688084755092859,
+            -0.010819513350725174,
+            -0.03085913322865963,
+            -0.007405094336718321,
+            -0.04735497012734413,
+            -0.016001902520656586,
+            0.018628379330039024,
+            -0.021043168380856514,
+            0.010356939397752285,
+            0.00260295532643795,
+            0.01590782031416893,
+            -0.0012289555743336678,
+            -0.01925559714436531,
+            0.003136090701445937,
+            -0.009086823090910912,
+            0.011548654176294804,
+            0.0010986118577420712,
+            0.03932657837867737,
+            -0.004084758460521698,
+            0.027660321444272995,
+            -0.029604697600007057,
+            0.008585048839449883,
+            -0.044124796986579895,
+            -0.0097610829398036,
+            -0.03866799920797348,
+            0.009855165146291256,
+            -0.008161676116287708,
+            -0.0338384211063385,
+            0.004523810930550098,
+            0.008357682265341282,
+            -0.002804841147735715,
+            -0.015892140567302704,
+            -0.006468187086284161,
+            0.02294050343334675,
+            -0.0047668581828475,
+            0.008953538723289967,
+            -0.03938930109143257,
+            -0.016166547313332558,
+            -0.007510937284678221,
+            0.03675498440861702,
+            -0.0036241449415683746,
+            0.017436664551496506,
+            -0.015727495774626732,
+            0.016213588416576385,
+            -0.04443840682506561,
+            -0.025825707241892815,
+            0.034057945013046265,
+            0.017389623448252678,
+            -0.021874234080314636,
+            0.026625411584973335,
+            0.019710330292582512,
+            0.04042420908808708,
+            -0.0580490417778492,
+            0.023802928626537323,
+            -0.0018473535310477018,
+            -0.009008420631289482,
+            0.019098792225122452,
+            0.010756791569292545,
+            0.03289759159088135,
+            -0.028961798176169395,
+            -0.014441697858273983,
+            0.006017374340444803,
+            0.020666837692260742,
+            8.042357512749732e-05,
+            0.006691633723676205,
+            -0.02358340285718441,
+            0.014496579766273499,
+            0.017562108114361763,
+            0.005190230440348387,
+            0.02703310176730156,
+            -0.026390204206109047,
+            0.028428662568330765,
+            0.014324095100164413,
+            -0.027770083397626877,
+            0.02132541686296463,
+            -0.027127185836434364,
+            -0.01390072237700224,
+            -0.001531784306280315,
+            0.02780144475400448,
+            0.011360488831996918,
+            -0.060244303196668625,
+            0.02063547819852829,
+            0.03173723816871643,
+            -0.006189859006553888,
+            0.03134522587060928,
+            -0.024053815752267838,
+            0.024665353819727898,
+            0.021372457966208458,
+            0.022799380123615265,
+            0.018440213054418564,
+            0.046037811785936356,
+            -0.00785982795059681,
+            0.01184658333659172,
+            -0.020118022337555885,
+            0.022579854354262352,
+            0.011595695279538631,
+            -0.009541556239128113,
+            -0.037570368498563766,
+            -0.021152932196855545,
+            0.007648141589015722,
+            0.013469509780406952,
+            -0.00831064023077488,
+            -0.03317984193563461,
+            -0.0458810068666935,
+            0.010090372525155544,
+            0.00569592509418726,
+            -0.057484544813632965,
+            -0.013555752113461494,
+            -0.009110343642532825,
+            0.005229431204497814,
+            -0.008459605276584625,
+            0.004061237443238497,
+            -0.028397301211953163,
+            0.015335483476519585,
+            -0.014457378536462784,
+            0.03716267645359039,
+            -0.011485932394862175,
+            0.022140800952911377,
+            -0.015837257727980614,
+            -0.004719816613942385,
+            -0.025292571634054184,
+            0.02488487958908081,
+            -0.021466542035341263,
+            -0.04199225455522537,
+            0.010835194028913975,
+            0.004480689764022827,
+            0.01622927002608776,
+            0.0537525974214077,
+            -0.005884090438485146,
+            -0.010317739099264145,
+            0.020321868360042572,
+            -0.019506484270095825,
+            -0.014002645388245583,
+            0.007679502479732037,
+            -0.005844889208674431,
+            -0.022031037136912346,
+            -0.01665264181792736,
+            -0.012262115254998207,
+            0.0247280765324831,
+            -0.00041602205601520836,
+            0.022611213847994804,
+            0.026390204206109047,
+            0.0195221658796072,
+            -0.02814641408622265,
+            0.009855165146291256,
+            -0.005127508658915758,
+            -0.011031199246644974,
+            0.004245482850819826,
+            0.021356778219342232,
+            -0.004755097441375256,
+            0.009000580757856369,
+            -0.01646447740495205,
+            -0.021121570840477943,
+            -0.03731948137283325,
+            -0.01249732170253992,
+            -0.003145891008898616,
+            0.0031478512100875378,
+            -0.035625990480184555,
+            -0.018126605078577995,
+            0.026060914620757103,
+            0.03070232830941677,
+            -0.028303219005465508,
+            0.0019051751587539911,
+            -0.002246225019916892,
+            -0.01132912840694189,
+            -0.003904432989656925,
+            -0.011791701428592205,
+            0.007040523923933506,
+            -0.015225720591843128,
+            8.691626862855628e-05,
+            -0.002042379230260849,
+            0.0023736287839710712,
+            -0.05792359635233879,
+            -0.017781633883714676,
+            0.02913428284227848,
+            0.002930284710600972,
+            0.00550383934751153,
+            -0.008279279805719852,
+            -0.01322646252810955,
+            -0.007009163033217192,
+            -0.02679789625108242,
+            0.024367425590753555,
+            0.015437406487762928,
+            0.019647609442472458,
+            -0.003559462958946824,
+            0.0017885518027469516,
+            0.018659740686416626,
+            0.0014171210350468755,
+            0.003351696999743581,
+            -0.007279650773853064,
+            0.006507388316094875,
+            0.02182719111442566,
+            -0.012795249931514263,
+            0.0158294178545475,
+            0.005778247490525246,
+            0.002685277722775936,
+            -0.02679789625108242,
+            -0.003682946553453803,
+            0.016793766990303993,
+            0.002657837001606822,
+            0.005296073388308287,
+            -0.0255434587597847,
+            0.038072142750024796,
+            0.04039284959435463,
+            0.025872748345136642,
+            0.020337548106908798,
+            -0.03220765292644501,
+            -0.004206281621009111,
+            -0.00758149940520525,
+            0.01121936459094286,
+            0.021811511367559433,
+            0.020055299624800682,
+            -0.010427501983940601,
+            -0.015743175521492958,
+            0.011062560603022575,
+            -0.02013370208442211,
+            -0.0019473163411021233,
+            0.018894946202635765,
+            0.011266405694186687,
+            -0.006201619748026133,
+            0.009988449513912201,
+            0.0004351325915195048,
+            0.005268632434308529,
+            0.02140381932258606,
+            -0.030529843643307686,
+            -0.025167128071188927,
+            0.0212470144033432,
+            -0.017123056575655937,
+            0.01154081430286169,
+            0.004876621067523956,
+            0.0032517341896891594,
+            0.020776601508259773,
+            0.07445079833269119,
+            -0.020070981234312057,
+            0.0043591661378741264,
+            -0.022768018767237663,
+            -0.02576298639178276,
+            -0.012614925391972065,
+            -0.022109439596533775,
+            0.005378395784646273,
+            0.013571432791650295,
+            -0.00024022944853641093,
+            0.020651157945394516,
+            0.01056862622499466,
+            -0.00644074659794569,
+            0.007216928992420435,
+            -0.01173681952059269,
+            -0.0007139506633393466,
+            -0.006585790775716305,
+            -0.0027087985072284937,
+            0.002058059675619006,
+            0.0015229640994220972,
+            0.002697037998586893,
+            0.02836594171822071,
+            0.012371878139674664,
+            0.013116699643433094,
+            -0.03327392414212227,
+            0.009815963916480541,
+            0.02151358313858509,
+            0.018095243722200394,
+            -0.03317984193563461,
+            -0.021388139575719833,
+            0.004543411545455456,
+            0.005017745308578014,
+            0.012928534299135208,
+            0.009196585975587368,
+            -0.006977802142500877,
+            0.02504168450832367,
+            -0.0355319082736969,
+            0.01814228482544422,
+            0.016527198255062103,
+            -0.011705459095537663,
+            -0.004931502975523472,
+            0.01986713521182537,
+            -0.0020090581383556128,
+            0.006860198453068733,
+            -0.005946812219917774,
+            0.0025794347748160362,
+            0.00653874920681119,
+            -0.021356778219342232,
+            -0.003559462958946824,
+            -0.012983416207134724,
+            -0.001520024030469358,
+            0.021419500932097435,
+            -0.017170097678899765,
+            0.008726172149181366,
+            -0.008655610494315624,
+            0.028836354613304138,
+            0.013414627872407436,
+            -0.029933987185359,
+            0.0007183607667684555,
+            0.023065946996212006,
+            0.01173681952059269,
+            -0.002052179304882884,
+            0.00922794733196497,
+            0.008851616643369198,
+            -0.016825126484036446,
+            6.958201265661046e-05,
+            0.012520842254161835,
+            0.018079563975334167,
+            0.007957830093801022,
+            0.006879799067974091,
+            0.006417225580662489,
+            0.019616248086094856,
+            -0.018785184249281883,
+            -0.02010234259068966,
+            0.00456693209707737,
+            0.010521584190428257,
+            -0.0065426696091890335,
+            -0.020996127277612686,
+            0.007060124538838863,
+            -0.016401754692196846,
+            0.037350840866565704,
+            0.009690520353615284,
+            -0.010803832672536373,
+            -0.0035790635738521814,
+            -0.031141381710767746,
+            -0.0027479995042085648,
+            -0.021983996033668518,
+            -0.010153094306588173,
+            0.019004710018634796,
+            -0.003353656968101859,
+            -0.00965131912380457,
+            -0.01906743273139,
+            -0.011015518568456173,
+            -0.008734012953937054,
+            0.0017326901433989406,
+            -0.008984900079667568,
+            0.01944376341998577,
+            0.01144673116505146,
+            -0.02844434417784214,
+            -0.031799960881471634,
+            -0.03361889347434044,
+            0.011054719798266888,
+            0.0025559139903634787,
+            -0.02913428284227848,
+            -0.011721138842403889,
+            -0.004555171821266413,
+            -0.004080838058143854,
+            -0.008000951260328293,
+            0.008145995438098907,
+            -0.003831910900771618,
+            0.006017374340444803,
+            -0.013830160722136497,
+            -0.005554800853133202,
+            -0.008130315691232681,
+            0.00788726843893528,
+            0.0149513129144907,
+            -0.005488158669322729,
+            -0.03093753568828106,
+            -0.017781633883714676,
+            0.00965131912380457,
+            -0.012826611287891865,
+            0.008145995438098907,
+            0.01684080809354782,
+            -0.004923662636429071,
+            0.00594289181753993,
+            0.01848725602030754,
+            0.018095243722200394,
+            -0.0030753291212022305,
+            0.014363295398652554,
+            -0.0011162522714585066,
+            -0.006385864689946175,
+            0.012607084587216377,
+            0.008953538723289967,
+            0.01487291045486927,
+            0.02507304586470127,
+            -0.010294217616319656,
+            -0.006087936460971832,
+            -0.0016954491147771478,
+            0.03274078667163849,
+            -0.023599082604050636,
+            -0.019804412499070168,
+            0.015688294544816017,
+            0.05425437167286873,
+            -0.0010662708664312959,
+            -0.0020913805346935987,
+            -0.022344646975398064,
+            -0.01102335937321186,
+            0.023489320650696754,
+            0.016793766990303993,
+            0.008036232553422451,
+            -0.010843033902347088,
+            -0.014386816881597042,
+            0.010545105673372746,
+            0.014418177306652069,
+            0.01922423578798771,
+            -0.024053815752267838,
+            0.021278375759720802,
+            -0.026750855147838593,
+            0.036127764731645584,
+            0.0075501385144889355,
+            -0.005974252708256245,
+            0.015539329499006271,
+            0.0036378654185682535,
+            -0.006526988931000233,
+            0.014324095100164413,
+            -0.01070190966129303,
+            0.011893624439835548,
+            0.006868038792163134,
+            -0.0015004234155640006,
+            0.021043168380856514,
+            -0.01108608115464449,
+            -0.0010045290691778064,
+            -0.0027048783376812935,
+            0.02090204507112503,
+            0.0031184502877295017,
+            -0.0038671919610351324,
+            0.01464554388076067,
+            0.0028773632366210222,
+            -0.004351325798779726,
+            -0.010074691846966743,
+            0.007350212894380093,
+            -0.03173723816871643,
+            -0.008138155564665794,
+            0.013924242928624153,
+            -0.005735125858336687,
+            0.002120781457051635,
+            -0.011015518568456173,
+            -0.002642156556248665,
+            -0.03609640523791313,
+            0.01787571795284748,
+            -0.000642898608930409,
+            0.017091695219278336,
+            0.0028656029608100653,
+            -0.00936123076826334,
+            -0.004809979349374771,
+            0.02297186478972435,
+            -0.009565076790750027,
+            -0.006930760573595762,
+            -0.018769502639770508,
+            -0.021497901529073715,
+            -0.025135766714811325,
+            0.021121570840477943,
+            -0.009580757468938828,
+            0.0018189327092841268,
+            -0.007706942968070507,
+            -0.0029126442968845367,
+            -0.00791470892727375,
+            0.012222914025187492,
+            0.018706781789660454,
+            -0.030576884746551514,
+            0.006617151666432619,
+            -0.006170258391648531,
+            -0.022109439596533775,
+            0.006456426810473204,
+            -0.009518035687506199,
+            0.01684080809354782,
+            -0.0009344570571556687,
+            -0.04123959317803383,
+            0.006303542293608189,
+            0.026484286412596703,
+            -0.017969800159335136,
+            -0.0015141437761485577,
+            0.010043330490589142,
+            0.008718332275748253,
+            -0.0026499966625124216,
+            -0.018894946202635765,
+            0.01811092346906662,
+            -0.017436664551496506,
+            0.018675420433282852,
+            0.026844937354326248,
+            -0.008757533505558968,
+            0.028993159532546997,
+            0.0001089056531782262,
+            0.01864405907690525,
+            0.0029832064174115658,
+            0.005641043186187744,
+            0.010082531720399857,
+            0.0038985528517514467,
+            0.020070981234312057,
+            0.00295772566460073,
+            0.007949990220367908,
+            0.038072142750024796,
+            -0.010788151994347572,
+            0.02182719111442566,
+            0.011305606923997402,
+            -0.00019184054690413177,
+            0.01615086756646633,
+            -0.007628540974110365,
+            -0.032489899545907974,
+            0.01566477306187153,
+            -0.027770083397626877,
+            -0.004668855108320713,
+            0.00922794733196497,
+            -0.0147396270185709,
+            -0.00934555009007454,
+            -0.014418177306652069,
+            0.009235787205398083,
+            0.012779570184648037,
+            -0.013908562250435352,
+            -0.0005326454411260784,
+            0.018973348662257195,
+            0.019381040707230568,
+            0.014527940191328526,
+            0.005982093047350645,
+            -0.005182390101253986,
+            0.013830160722136497,
+            0.0030204474460333586,
+            0.023646123707294464,
+            0.0016895688604563475,
+            0.005633202847093344,
+            -0.010592146776616573,
+            0.0145122604444623,
+            0.01508459635078907,
+            -0.0017875717021524906,
+            0.01092927623540163,
+            -0.004845260176807642,
+            -0.005817448254674673,
+            -0.01960056647658348,
+            -0.007636380847543478,
+            -0.009110343642532825,
+            0.006170258391648531,
+            0.04719816520810127,
+            0.0004782538453582674,
+            -0.002846002345904708,
+            -0.01765619032084942,
+            0.004559091757982969,
+            0.012685487046837807,
+            -0.0097454022616148,
+            -0.0034987013787031174,
+            -0.019365360960364342,
+            -0.022720977663993835,
+            0.0034144187811762094,
+            -0.0050843870267271996,
+            0.027001742273569107,
+            -0.003822110593318939,
+            0.02767600119113922,
+            -0.029102923348546028,
+            0.01956920698285103,
+            -0.016135187819600105,
+            -0.0097454022616148,
+            -0.0022560253273695707,
+            -0.012160192243754864,
+            -0.014331934973597527,
+            0.02300322614610195,
+            0.00936123076826334,
+            -0.032678067684173584,
+            -0.0016190068563446403,
+            0.01010605227202177,
+            -0.009306349791586399,
+            0.009008420631289482,
+            0.006860198453068733,
+            0.010803832672536373,
+            -0.0015798057429492474,
+            0.01190930511802435,
+            0.0013592992909252644,
+            -0.001451421994715929,
+            -9.536901052342728e-05,
+            -0.02294050343334675,
+            0.018346130847930908,
+            0.030482802540063858,
+            -0.05265496298670769,
+            -0.01914583332836628,
+            0.020118022337555885,
+            0.00569592509418726,
+            0.0005865469574928284,
+            0.013061817735433578,
+            -0.024367425590753555,
+            0.01027853786945343,
+            -0.018675420433282852,
+            -0.014316254295408726,
+            -0.022140800952911377,
+            -0.005460718180984259,
+            0.0151786794885993,
+            -0.027581918984651566,
+            0.025794345885515213,
+            0.017170097678899765,
+            -0.0015023835003376007,
+            0.006452506873756647,
+            0.0007472716388292611,
+            0.008851616643369198,
+            -0.03255262225866318,
+            -0.006617151666432619,
+            -0.0059664128348231316,
+            0.006009534001350403,
+            0.003655505832284689,
+            -0.02897747792303562,
+            0.014527940191328526,
+            -0.02074524015188217,
+            0.008263599127531052,
+            -0.029949666932225227,
+            0.011321287602186203,
+            0.03622184693813324,
+            -2.7471420253277756e-05,
+            0.009157384745776653,
+            -0.0031282505951821804,
+            0.04218042269349098,
+            0.026625411584973335,
+            0.0015709854196757078,
+            -0.008024472743272781,
+            -0.019694650545716286,
+            -0.002522593131288886,
+            0.00673867529258132,
+            -0.02918132394552231,
+            0.0012113150442019105,
+            -0.035092856734991074,
+            0.02209375984966755,
+            -0.006099696736782789,
+            -0.025512099266052246,
+            0.033556170761585236,
+            -0.020384589210152626,
+            0.002808761317282915,
+            -0.005347034893929958,
+            0.0027068383060395718,
+            0.030561204999685287,
+            0.0024990723468363285,
+            0.03005943074822426,
+            -0.03826030716300011,
+            -0.0013504790840670466,
+            0.0028244417626410723,
+            0.0038162304554134607,
+            -0.010435341857373714,
+            -0.015711814165115356,
+            0.016903528943657875,
+            -0.010200135409832,
+            0.004292524419724941,
+            0.02522985078394413,
+            -0.003655505832284689,
+            -0.013132380321621895,
+            0.021278375759720802,
+            -0.002808761317282915,
+            0.005417597014456987,
+            -0.005460718180984259,
+            0.000567926443181932,
+            0.023222751915454865,
+            0.02568458393216133,
+            -0.01136832870543003,
+            -0.011336968280375004,
+            -0.002781320596113801,
+            -0.00031654915073886514,
+            0.014104568399488926,
+            0.015264921821653843,
+            -0.008200877346098423,
+            -0.020541394129395485,
+            0.006817077286541462,
+            0.009808124043047428,
+            0.018549976870417595,
+            0.0012946174247190356,
+            -0.0255748201161623,
+            -0.008459605276584625,
+            -0.03027895651757717,
+            0.010153094306588173,
+            -0.016558559611439705,
+            -0.005601841956377029,
+            0.0013112779706716537,
+            -0.001341658877208829,
+            -0.009549396112561226,
+            0.014394656755030155,
+            0.015194360166788101,
+            -0.03901296854019165,
+            -0.015813738107681274,
+            -0.0007222809363156557,
+            -0.018502935767173767,
+            0.0024539909791201353,
+            -0.0011662337929010391,
+            0.001961036818102002,
+            0.0021658625919371843,
+            -0.003165491623803973,
+            0.004739417228847742,
+            -0.020070981234312057,
+            -0.0024324303958564997,
+            0.005629282910376787,
+            0.02113725244998932,
+            0.004535571206361055,
+            -0.0015014035161584616,
+            0.0022442650515586138,
+            0.0038907125126570463,
+            0.0017464105039834976,
+            0.0063897850923240185,
+            -0.00673867529258132,
+            0.015108116902410984,
+            -0.00020262086763978004,
+            0.015327643603086472,
+            -0.01906743273139,
+            0.007557978853583336,
+            -0.002369708614423871,
+            -0.005123588256537914,
+            0.015782376751303673,
+            -0.00037020572926849127,
+            -0.0020482593681663275,
+            -0.010294217616319656,
+            0.007605019956827164,
+            0.015586371533572674,
+            -0.022501451894640923,
+            -0.009063302539288998,
+            -0.014614183455705643,
+            0.02711150422692299,
+            0.0153746847063303,
+            0.008122474886476994,
+            -0.03223901242017746,
+            -0.0056488835252821445,
+            -0.012120991013944149,
+            -0.0059193712659180164,
+            -0.015296283178031445,
+            0.018863586708903313,
+            0.019349679350852966,
+            -0.0051784696988761425,
+            0.01619790866971016,
+            -0.0015592251438647509,
+            0.020855003967881203,
+            -0.00924362801015377,
+            0.006476027425378561,
+            -0.0007590319728478789,
+            -0.006315303035080433,
+            0.010168774053454399,
+            0.03521829843521118,
+            -0.017389623448252678,
+            0.008240078575909138,
+            -0.000719830859452486,
+            -0.002530433237552643,
+            -0.00306356861256063,
+            -0.004022036213427782,
+            0.006358424201607704,
+            0.014316254295408726,
+            0.009369071573019028,
+            0.0029342048801481724,
+            0.07056204229593277,
+            -0.003506541484966874,
+            0.020243465900421143,
+            0.006323142908513546,
+            -0.014574982225894928,
+            -0.005891930777579546,
+            0.0022109439596533775,
+            -0.028397301211953163,
+            0.031266823410987854,
+            0.020086660981178284,
+            0.0048413402400910854,
+            0.003782909596338868,
+            -0.014692584984004498,
+            0.023034587502479553,
+            0.004453248810023069,
+            0.0030616086442023516,
+            -0.0049119023606181145,
+            0.005864489823579788,
+            0.047762662172317505,
+            0.012403239496052265,
+            -0.007420775014907122,
+            -0.0022658256348222494,
+            0.014292733743786812,
+            0.0054254368878901005,
+            -0.011493772268295288,
+            -0.0003150791162624955,
+            -0.000643878651317209,
+            0.015037555247545242,
+            -0.001587645965628326,
+            0.021952634677290916,
+            -0.01114096213132143,
+            -0.01826772838830948,
+            0.0018375532235950232,
+            0.004394447430968285,
+            0.01571965590119362,
+            0.009486674331128597,
+            0.008318481035530567,
+            -0.005884090438485146,
+            0.004700215999037027,
+            0.006397624965757132,
+            0.005664564203470945,
+            0.00790294911712408,
+            -0.005852729547768831,
+            -0.008655610494315624,
+            -0.01826772838830948,
+            0.02637452259659767,
+            -0.002616675803437829,
+            0.003259574295952916,
+            -0.016856487840414047,
+            0.002438310533761978,
+            -0.0034712604247033596,
+            -0.007769664749503136,
+            0.010937117040157318,
+            0.017750274389982224,
+            -0.009392592124640942,
+            -0.008185196667909622,
+            0.0147239463403821,
+            -0.017028972506523132,
+            0.004457169212400913,
+            0.012622765265405178,
+            -0.004778618458658457,
+            0.006311382632702589,
+            0.003381097922101617,
+            -0.008898657746613026,
+            0.0012073949910700321,
+            0.0013671396300196648,
+            0.015539329499006271,
+            0.015249241143465042,
+            0.007075804751366377,
+            0.022031037136912346,
+            0.0016395874554291368,
+            0.0009217166807502508,
+            0.01654287800192833,
+            0.015609892085194588,
+            0.0064721074886620045,
+            0.022078078240156174,
+            -0.015445247292518616,
+            -0.016668323427438736,
+            0.003367377445101738,
+            0.006895479746162891,
+            0.0025441537145525217,
+            0.0030243676155805588,
+            0.0036751064471900463,
+            0.010545105673372746,
+            -0.014394656755030155,
+            -0.006946441251784563,
+            -0.009322029538452625,
+            0.018973348662257195,
+            0.022689616307616234,
+            -0.020588435232639313,
+            0.0003922563628293574,
+            0.004958943463861942,
+            0.017091695219278336,
+            -0.001889494713395834,
+            0.006558349821716547,
+            0.01760914921760559,
+            -0.006574030499905348,
+            -0.008389042690396309,
+            -0.02316003106534481,
+            -0.012426760047674179,
+            -0.0006556389853358269,
+            0.019898496568202972,
+            -0.011407529935240746,
+            0.01286581251770258,
+            -0.015633411705493927,
+            -0.0042768437415361404,
+            0.029918305575847626,
+            0.011877943761646748,
+            0.005739046260714531,
+            -0.01330486498773098,
+            -0.011674097739160061,
+            -0.013673355802893639,
+            0.014324095100164413,
+            0.010317739099264145,
+            0.003122370457276702,
+            -0.010121732950210571,
+            -0.0017013292526826262,
+            -0.005515599623322487,
+            -0.008961379528045654,
+            -0.00685235857963562,
+            0.011462411843240261,
+            -0.0027617199812084436,
+            0.003949514124542475,
+            -0.011987706646323204,
+            0.0009903187165036798,
+            -0.010748950764536858,
+            -0.006495628040283918,
+            -0.0015415846137329936,
+            -0.03481060639023781,
+            -0.0299026258289814,
+            -0.001531784306280315,
+            -0.005107908044010401,
+            0.005652803461998701,
+            0.004123959224671125,
+            0.011611375957727432,
+            0.0029185244347900152,
+            0.00937691144645214,
+            0.009251467883586884,
+            0.011062560603022575,
+            3.1728417525300756e-05,
+            -0.004202361684292555,
+            0.011995547451078892,
+            -0.013602794148027897,
+            -0.016244949772953987,
+            0.024916240945458412,
+            0.004751177504658699,
+            0.013461669906973839,
+            -0.028397301211953163,
+            0.011783861555159092,
+            -0.002575514605268836,
+            0.010654868558049202,
+            -0.031141381710767746,
+            -0.0039769550785422325,
+            0.0020443391986191273,
+            -0.0033242562785744667,
+            -0.0101844547316432,
+            -0.028052331879734993,
+            -0.0075383782386779785,
+            -0.017812995240092278,
+            -0.01630767248570919,
+            -0.024759436026215553,
+            0.0050138249062001705,
+            -0.012034748680889606,
+            0.021341098472476006,
+            -0.010286377742886543,
+            0.03725675866007805,
+            0.006040894892066717,
+            -0.0015719655202701688,
+            0.01361063402146101,
+            0.0030478881672024727,
+            -0.0020090581383556128,
+            0.016636962071061134,
+            -0.006942520849406719,
+            -0.0013749798526987433,
+            0.017969800159335136,
+            0.01102335937321186,
+            0.003243893850594759,
+            -0.003602584358304739,
+            0.0029949666932225227,
+            0.026562688872218132,
+            0.01814228482544422,
+            0.00426900340244174,
+            0.005840969271957874,
+            0.002998886862769723,
+            0.016417434439063072,
+            0.009808124043047428,
+            -0.014778828248381615,
+            -0.008145995438098907,
+            0.011611375957727432,
+            -0.01173681952059269,
+            -0.006832757964730263,
+            0.004845260176807642,
+            -0.0097454022616148,
+            -0.023097308352589607,
+            0.012748208828270435,
+            0.006256501190364361,
+            -0.0005919371615163982,
+            -0.019239917397499084,
+            -0.012920694425702095,
+            0.0017620910657569766,
+            0.01225427445024252,
+            0.016323352232575417,
+            -0.01914583332836628,
+            0.002738199196755886,
+            0.008153836242854595,
+            -0.006417225580662489,
+            -0.011580015532672405,
+            0.016950570046901703,
+            0.022313285619020462,
+            0.014888591133058071,
+            -0.04788810759782791,
+            -0.008945698849856853,
+            0.017546428367495537,
+            -0.012183712795376778,
+            -0.043967992067337036,
+            -0.0015455047832801938,
+            -0.0018395131919533014,
+            -0.025982512161135674,
+            0.0052490318194031715,
+            0.006182019133120775,
+            -0.014700425788760185,
+            0.016527198255062103,
+            0.03200380504131317,
+            0.00019159555085934699,
+            0.009102503769099712,
+            -0.011109601706266403,
+            -0.014426017180085182,
+            -0.009047621861100197,
+            0.008726172149181366,
+            -0.01175250019878149,
+            -0.009847325272858143,
+            0.00248927203938365,
+            -0.015555010177195072,
+            0.01848725602030754,
+            0.007569739129394293,
+            0.03150203078985214,
+            0.01132912840694189,
+            -0.00248927203938365,
+            -0.01760914921760559,
+            -0.020964767783880234,
+            -0.009533716365695,
+            0.003741748398169875,
+            0.007997031323611736,
+            0.007005242630839348,
+            0.010160934180021286,
+            0.013328385539352894,
+            0.022673936560750008,
+            -0.006252580787986517,
+            0.014825869351625443,
+            -3.0059307391638868e-05,
+            -0.0014680825406685472,
+            0.0025676742661744356,
+            -0.020149383693933487,
+            0.005143188871443272,
+            0.0018228527624160051,
+            -0.004904062021523714,
+            -0.007013082969933748,
+            -0.003083169227465987,
+            0.032066527754068375,
+            -0.016480157151818275,
+            0.015970543026924133,
+            -0.031078658998012543,
+            -0.009337710216641426,
+            0.001246596104465425,
+            0.02814641408622265,
+            0.012701167725026608,
+            -0.02841298282146454,
+            -0.011297767050564289,
+            -0.026515647768974304,
+            0.02278370037674904,
+            -0.014300573617219925,
+            0.004723736550658941,
+            0.015845099464058876,
+            0.01444953866302967,
+            0.01194850541651249,
+            -0.01165841706097126,
+            -0.00878889486193657,
+            0.011776020750403404,
+            0.021999677643179893,
+            0.007679502479732037,
+            0.0022266244050115347,
+            -0.0009261267841793597,
+            -0.001644487609155476,
+            -0.004304284695535898,
+            -0.017985479906201363,
+            0.006711234338581562,
+            -0.00819303747266531,
+            0.027127185836434364,
+            -0.007597180083394051,
+            0.016746724024415016,
+            -0.0062957024201750755,
+            -0.00582920853048563,
+            -0.02932244911789894,
+            -0.0203689094632864,
+            0.007605019956827164,
+            -0.006703393999487162,
+            0.012959894724190235,
+            -0.019365360960364342,
+            -0.0164958368986845,
+            0.010733271017670631,
+            -0.02010234259068966,
+            -0.002083540428429842,
+            -0.004719816613942385,
+            0.0022442650515586138,
+            0.00676611578091979,
+            0.00736197317019105,
+            -0.012826611287891865,
+            -0.006883719470351934,
+            0.016072465106844902,
+            -0.007663821801543236,
+            0.007640301249921322,
+            0.029871264472603798,
+            0.01768755167722702,
+            0.026844937354326248,
+            0.007048364263027906,
+            0.013751758262515068,
+            -0.010364780202507973,
+            -0.017546428367495537,
+            -0.023301154375076294,
+            0.01278741005808115,
+            -0.0025519938208162785,
+            0.03368161618709564,
+            0.00478645833209157,
+            -0.007820626720786095,
+            -0.0031027698423713446,
+            0.005417597014456987,
+            0.0008408643188886344,
+            -0.01414376962929964,
+            0.0029518455266952515,
+            -0.00551167968660593,
+            0.012826611287891865,
+            0.016033263877034187,
+            -0.01883222535252571,
+            0.02522985078394413,
+            -0.016856487840414047,
+            0.012779570184648037,
+            -0.003602584358304739,
+            0.01630767248570919,
+            0.009753242135047913,
+            -0.0003626105026341975,
+            -0.0015415846137329936,
+            -0.05394076183438301,
+            -0.010458862408995628,
+            0.0038456313777714968,
+            -0.004970703739672899,
+            0.000352075177943334,
+            -0.012638445943593979,
+            -0.038573917001485825,
+            0.0034810607321560383,
+            0.013391107320785522,
+            -0.011556494981050491,
+            0.013422468677163124,
+            -0.002016898477450013,
+            0.006617151666432619,
+            -0.007922549732029438,
+            -0.006260421127080917,
+            0.016480157151818275,
+            0.02573162503540516,
+            -0.021121570840477943,
+            -0.022344646975398064,
+            0.005778247490525246,
+            -0.005543040577322245,
+            -0.014684745110571384,
+            -0.005021665245294571,
+            -0.01598622277379036,
+            0.0025069124531000853,
+            0.023567723110318184,
+            -0.01426137238740921,
+            -0.01776595413684845,
+            0.010145253501832485,
+            0.007757904473692179,
+            -0.024853520095348358,
+            -0.002355988137423992,
+            -0.009815963916480541,
+            -0.010803832672536373,
+            0.018393171951174736,
+            -0.01362631469964981,
+            0.008906497620046139,
+            -0.013124539516866207,
+            0.010396140627563,
+            -0.02825617790222168,
+            -0.01226995512843132,
+            -0.0019502564100548625,
+            -0.008882977068424225,
+            0.028083693236112595,
+            -0.010772472247481346,
+            -0.010200135409832,
+            0.020070981234312057,
+            -0.008663450367748737,
+            0.012567883357405663,
+            0.007855907082557678,
+            0.028648190200328827,
+            -0.0025853149127215147,
+            0.009141704998910427,
+            0.0028930436819791794,
+            0.01390072237700224,
+            0.0011936745140701532,
+            0.035437826067209244,
+            -0.00957291666418314,
+            -0.00030307378619909286,
+            -0.008835935965180397,
+            -0.0017140696290880442,
+            0.0010427501983940601,
+            0.010991998016834259,
+            0.014151609502732754,
+            0.012693327851593494,
+            -0.006244740914553404,
+            -0.0077853454276919365,
+            -0.015500128269195557,
+            -0.004527730867266655,
+            -0.004888381343334913,
+            0.007428614888340235,
+            0.012442439794540405,
+            -0.020039619877934456,
+            0.005358795169740915,
+            -0.012144511565566063,
+            0.0011946546146646142,
+            -0.01587645895779133,
+            -0.0022560253273695707,
+            -0.013634154573082924,
+            -0.004080838058143854,
+            -0.004331725183874369,
+            0.003790749702602625,
+            0.008749693632125854,
+            -0.016401754692196846,
+            0.0199455376714468,
+            0.008153836242854595,
+            -0.0004074467869941145,
+            -0.0028107212856411934,
+            -0.024931922554969788,
+            -0.014010485261678696,
+            0.00014859679504297674,
+            0.016668323427438736,
+            0.0050138249062001705,
+            -0.0051784696988761425,
+            0.002026698552072048,
+            -0.012826611287891865,
+            0.005280392710119486,
+            -0.0009040761506184936,
+            0.0022285846062004566,
+            -0.0030067269690334797,
+            0.0057312059216201305,
+            0.006762195844203234,
+            0.003700587200000882,
+            0.012959894724190235,
+            0.01487291045486927,
+            -0.011760340072214603,
+            -0.00966699980199337,
+            0.012285635806620121,
+            0.015217880718410015,
+            -0.008914338424801826,
+            0.01941240206360817,
+            -0.02878931351006031,
+            0.002820521593093872,
+            0.005233351606875658,
+            -0.0042533231899142265,
+            -0.0021854632068425417,
+            7.748349162284285e-05,
+            0.028475703671574593,
+            -0.0010221695993095636,
+            -0.010388300754129887,
+            -0.0009854185627773404,
+            0.01707601360976696,
+            -0.0006600490887649357,
+            0.006123217288404703,
+            0.023724526166915894,
+            0.00023790188424754888,
+            0.009768922813236713,
+            0.009776762686669827,
+            -0.019506484270095825,
+            -0.003145891008898616,
+            0.004633574280887842,
+            -0.012442439794540405,
+            0.018393171951174736,
+            0.014841550029814243,
+            0.008389042690396309,
+            0.0038005500100553036,
+            0.007510937284678221,
+            0.020086660981178284,
+            0.006374104414135218,
+            -0.002336387522518635,
+            -0.0002511322672944516,
+            0.0066445921547710896,
+            -0.011721138842403889,
+            0.017405303195118904,
+            -0.004363086074590683,
+            -0.015437406487762928,
+            0.006679873447865248,
+            0.011674097739160061,
+            -0.006773956120014191,
+            -0.004033796954900026,
+            0.0022540653590112925,
+            0.005711605306714773,
+            0.005464638117700815,
+            -0.003888752544298768,
+            -0.006574030499905348,
+            0.0012505161575973034,
+            0.017812995240092278,
+            0.010639187879860401,
+            0.016433116048574448,
+            -0.014990514144301414,
+            -0.011697618290781975,
+            0.0023050266318023205,
+            -0.018016841262578964,
+            0.002996926661580801,
+            -0.011517293751239777,
+            0.006299622356891632,
+            -0.007420775014907122,
+            0.02568458393216133,
+            0.019616248086094856,
+            0.007573659066110849,
+            0.011932825669646263,
+            -0.015539329499006271,
+            0.016715364530682564,
+            -0.00913386419415474,
+            0.012120991013944149,
+            -0.018847905099391937,
+            -0.0017346502281725407,
+            0.005296073388308287,
+            0.027628960087895393,
+            0.010302058421075344,
+            0.006064415443688631,
+            -0.001778751495294273,
+            -0.005182390101253986,
+            -0.008318481035530567,
+            -0.009619958698749542,
+            -0.014755306765437126,
+            -0.026233399286866188,
+            -0.00022810160589870065,
+            0.016339033842086792,
+            -0.00195907661691308,
+            0.0022521051578223705,
+            0.014590661972761154,
+            0.013187261298298836,
+            -0.0024128297809511423,
+            -0.013908562250435352,
+            -0.007252209819853306,
+            0.010835194028913975,
+            0.005119668319821358,
+            -0.008357682265341282,
+            -0.009141704998910427,
+            0.008138155564665794,
+            -0.013273504562675953,
+            0.016433116048574448,
+            -0.016182228922843933,
+            -0.009949248284101486,
+            -0.013649835251271725,
+            -0.0005929171456955373,
+            -0.008263599127531052,
+            -0.031454991549253464,
+            0.012152351438999176,
+            0.005707685369998217,
+            -0.0006365284207277,
+            -0.025715945288538933,
+            0.02369316667318344,
+            -0.008569368161261082,
+            0.03656681999564171,
+            -0.01593918167054653,
+            0.015633411705493927,
+            0.0010623506968840957,
+            0.011493772268295288,
+            -0.003392858197912574,
+            0.0029479253571480513,
+            0.007953910157084465,
+            0.002550033852458,
+            0.003055728506296873,
+            -0.00539799639955163,
+            -0.0033066156320273876,
+            0.008898657746613026,
+            -0.006448586471378803,
+            0.008490965701639652,
+            -0.014849389903247356,
+            -0.02320707216858864,
+            -0.01144673116505146,
+            0.009502355009317398,
+            0.014104568399488926,
+            0.00861640926450491,
+            -0.014473059214651585,
+            0.00882809516042471,
+            -0.02435174584388733,
+            -0.0023461878299713135,
+            -0.0031282505951821804,
+            0.004782538395375013,
+            -0.017201457172632217,
+            0.016605600714683533,
+            0.02193695493042469,
+            -3.518914309097454e-05,
+            -0.0007879427867010236,
+            0.01382231991738081,
+            0.010160934180021286,
+            0.009423952549695969,
+            -0.010011970065534115,
+            0.0017826715484261513,
+            -8.403743413509801e-05,
+            0.02044731192290783,
+            -0.002397149335592985,
+            -0.009831644594669342,
+            -0.0035379023756831884,
+            -0.01399480551481247,
+            0.009698361158370972,
+            -0.029275408014655113,
+            -0.0052490318194031715,
+            -0.006523068994283676,
+            0.017170097678899765,
+            0.00684451824054122,
+            0.021639026701450348,
+            0.0065622697584331036,
+            0.005523439962416887,
+            -8.844755939207971e-05,
+            -0.008240078575909138,
+            0.012097470462322235,
+            -0.00809895433485508,
+            -0.007201248314231634,
+            0.006013453938066959,
+            0.004492450039833784,
+            0.016856487840414047,
+            0.008067593909800053,
+            -0.009180906228721142,
+            0.030106471851468086,
+            0.014904271811246872,
+            -0.013939923606812954,
+            0.01814228482544422,
+            -0.01826772838830948,
+            0.020776601508259773,
+            0.01853429712355137,
+            0.010239336639642715,
+            -0.015304123051464558,
+            -0.015680454671382904,
+            0.01784435659646988,
+            -0.010466703213751316,
+            -0.004657094832509756,
+            0.0018052122322842479,
+            -0.017938438802957535,
+            8.092890311672818e-06,
+            -0.011972026899456978,
+            0.015429566614329815,
+            -0.0021991836838424206,
+            -0.00012042098387610167,
+            -0.020776601508259773,
+            -0.00043831768562085927,
+            0.016903528943657875,
+            0.01615086756646633,
+            -0.0033340565860271454,
+            -0.04478337615728378,
+            -0.006178098730742931,
+            0.009855165146291256,
+            -0.00706404447555542,
+            -0.025982512161135674,
+            0.01803252287209034,
+            0.0034104986116290092,
+            -0.019428081810474396,
+            -0.01184658333659172,
+            -0.018502935767173767,
+            -0.019396722316741943,
+            -0.008820255286991596,
+            -0.004563012160360813,
+            -0.015272761695086956,
+            -0.003884832374751568,
+            0.008224397897720337,
+            -0.007213008590042591,
+            -0.006064415443688631,
+            0.004061237443238497,
+            -0.007354132831096649,
+            -0.0003518301818985492,
+            -0.010866554453969002,
+            -0.005629282910376787,
+            0.011133122257888317,
+            0.002452031010761857,
+            -0.01508459635078907,
+            -0.015366844832897186,
+            -0.011564334854483604,
+            -0.010537264868617058,
+            0.004139639902859926,
+            -0.009212266653776169,
+            -0.005080467090010643,
+            -0.020149383693933487,
+            0.0006615191232413054,
+            0.013469509780406952,
+            0.0104118213057518,
+            0.0006154578295536339,
+            0.004931502975523472,
+            0.022313285619020462,
+            -0.006189859006553888,
+            -0.0006242780946195126,
+            -9.363864592160098e-06,
+            -0.002955765463411808,
+            0.02687629871070385,
+            0.004312125034630299,
+            0.01286581251770258,
+            -0.005562641192227602,
+            0.009494515135884285,
+            -0.01975737139582634,
+            -0.0023755887523293495,
+            -0.010850873775780201,
+            -0.019773053005337715,
+            -0.011979866772890091,
+            -5.2125258662272245e-05,
+            0.004743337165564299,
+            -0.013069658540189266,
+            0.0021541023161262274,
+            -0.00467277504503727,
+            0.006734754890203476,
+            -0.01508459635078907,
+            -0.005782167427241802,
+            -0.007134606596082449,
+            -0.010662708431482315,
+            -0.02416357956826687,
+            0.018283409997820854,
+            0.02889907732605934,
+            0.001601366326212883,
+            -0.007554058451205492,
+            -0.01391640305519104,
+            -0.008177356794476509,
+            -0.004041636828333139,
+            -0.018048202618956566,
+            -0.008702651597559452,
+            -0.027926888316869736,
+            -0.013641994446516037,
+            -0.0003746158326976001,
+            0.010772472247481346,
+            0.0036065042950212955,
+            -0.018126605078577995,
+            0.005107908044010401,
+            -0.025527779012918472,
+            -0.025668902322649956,
+            -0.019428081810474396,
+            -0.017295541241765022,
+            0.0025676742661744356,
+            0.003888752544298768,
+            0.0021149013191461563,
+            0.017358262091875076,
+            0.020227786153554916,
+            0.004813899286091328,
+            -0.002424590289592743,
+            0.008592888712882996,
+            -0.014974833466112614,
+            0.0046766954474151134,
+            0.0023305073846131563,
+            -0.013422468677163124,
+            0.02358340285718441,
+            -0.011917144991457462,
+            -0.00353594240732491,
+            0.0008570348145440221,
+            0.0034575401805341244,
+            -0.028193457052111626,
+            -0.002152142347767949,
+            -0.0164958368986845,
+            0.02270529791712761,
+            -0.029651738703250885,
+            -0.0021443020086735487,
+            -0.00416316045448184,
+            -0.007895108312368393,
+            0.026578368619084358,
+            0.021356778219342232,
+            -0.0004645334556698799,
+            0.008984900079667568,
+            0.010709750466048717,
+            -0.0013063778169453144,
+            0.01869110018014908,
+            -0.017483705654740334,
+            -0.014151609502732754,
+            0.022234883159399033,
+            0.008757533505558968,
+            0.0036163046024739742,
+            -0.011242885142564774,
+            0.017562108114361763,
+            0.017468025907874107,
+            0.0059193712659180164,
+            -0.011485932394862175,
+            -0.016244949772953987,
+            0.0007933329907245934,
+            0.019773053005337715,
+            -0.012403239496052265,
+            0.0027440793346613646,
+            -0.010631348006427288,
+            0.008091114461421967,
+            -0.014708265662193298,
+            0.017750274389982224,
+            -0.027503516525030136,
+            -0.005060866475105286,
+            -0.0016092065488919616,
+            0.01795412041246891,
+            -0.008012712001800537,
+            0.015711814165115356,
+            -0.00696604186668992,
+            -0.009470994584262371,
+            0.014888591133058071,
+            -0.0005556760588660836,
+            -0.004739417228847742,
+            0.022909143939614296,
+            0.00653874920681119,
+            -0.0009614078444428742,
+            0.004884461406618357,
+            -0.018424533307552338,
+            0.025010323151946068,
+            -0.010074691846966743,
+            0.003000846831128001,
+            0.0012524762423709035,
+            0.0013710596831515431,
+            0.026844937354326248,
+            0.02634316310286522,
+            0.0057429661974310875,
+            0.008585048839449883,
+            -0.01351655088365078,
+            0.02519848942756653,
+            0.00467277504503727,
+            0.0012720768572762609,
+            -0.0009893386159092188,
+            -0.0024167499504983425,
+            -0.0017454305198043585,
+            -0.009180906228721142,
+            0.015743175521492958,
+            -0.01213667169213295,
+            -0.025935471057891846,
+            -0.009322029538452625,
+            -0.004492450039833784,
+            -0.017107374966144562,
+            0.020055299624800682,
+            0.0004699236014857888,
+            -0.003992635756731033,
+            0.011681938543915749,
+            0.007428614888340235,
+            0.0032360537443310022,
+            -0.008655610494315624,
+            0.0038671919610351324,
+            -0.007130686193704605,
+            -0.013046137988567352,
+            0.012450280599296093,
+            -0.019208556041121483,
+            -0.008984900079667568,
+            -0.008553687483072281,
+            0.009063302539288998,
+            0.010443182662129402,
+            0.00928282830864191,
+            0.015656933188438416,
+            0.004441488534212112,
+            0.00819303747266531,
+            0.012489481829106808,
+            -0.024931922554969788,
+            0.01707601360976696,
+            0.021764470264315605,
+            -0.0005096147651784122,
+            0.001671928446739912,
+            0.00045350813888944685,
+            -0.03446563705801964,
+            0.00027293790481053293,
+            0.0014034006744623184,
+            0.0077853454276919365,
+            -0.013406787998974323,
+            0.0022677856031805277,
+            0.022673936560750008,
+            -0.011109601706266403,
+            0.020996127277612686,
+            0.005970332771539688,
+            0.002479471731930971,
+            0.03343072906136513,
+            0.012230753898620605,
+            0.016339033842086792,
+            -0.0068915593437850475,
+            -0.026625411584973335,
+            0.0054371971637010574,
+            -0.018518615514039993,
+            0.009714040905237198,
+            0.0014386816183105111,
+            0.013148060999810696,
+            -0.01792275905609131,
+            -0.006025214213877916,
+            0.0027715202886611223,
+            -0.011039039120078087,
+            0.01654287800192833,
+            0.0014043806586414576,
+            0.019631927832961082,
+            0.001000609016045928,
+            -0.0028264017309993505,
+            -0.012967735528945923,
+            -0.01404968649148941,
+            0.0029773262795060873,
+            -0.009659159928560257,
+            -0.01699761115014553,
+            0.004966783802956343,
+            -0.008342001587152481,
+            0.007091485429555178,
+            -0.01587645895779133,
+            0.02044731192290783,
+            0.0005806668195873499,
+            0.00853800680488348,
+            0.015523649752140045,
+            0.0010319699067622423,
+            0.017750274389982224,
+            0.010341259650886059,
+            -0.0075501385144889355,
+            0.0062957024201750755,
+            0.0002594625111669302,
+            0.0029636058025062084,
+            0.0017895317869260907,
+            -0.01219155266880989,
+            0.0058135283179581165,
+            -0.004159240517765284,
+            0.00015165939112193882,
+            -0.006479947362095118,
+            -0.004092598333954811,
+            0.0034046184737235308,
+            -0.013406787998974323,
+            -0.00539015606045723,
+            0.002242304850369692,
+            0.001080971327610314,
+            0.02148222178220749,
+            -0.0017905118875205517,
+            0.00878889486193657,
+            0.02228192426264286,
+            -0.015570690855383873,
+            0.006703393999487162,
+            0.010059011168777943,
+            -0.004892301745712757,
+            -0.011156642809510231,
+            0.004661014769226313,
+            0.016166547313332558,
+            -0.01587645895779133,
+            -0.01131344772875309,
+            0.011501613073050976,
+            -0.021921275183558464,
+            0.004390527028590441,
+            -0.003963234834372997,
+            -0.007338452618569136,
+            0.02143518067896366,
+            -0.0012436560355126858,
+            0.032333094626665115,
+            -0.004770778119564056,
+            -0.015460927970707417,
+            -0.01930263824760914,
+            0.0012554163113236427,
+            -0.025512099266052246,
+            -0.005637123249471188,
+            -0.009188746102154255,
+            -0.006848438177257776,
+            0.00958859734237194,
+            0.0006865098839625716,
+            0.004919742234051228,
+            0.005950732156634331,
+            -0.004390527028590441,
+            0.004012236371636391,
+            -0.0031400108709931374,
+            0.007883348502218723,
+            -0.02143518067896366,
+            0.018628379330039024,
+            0.014120249077677727,
+            0.0005228451336733997,
+            -0.019741691648960114,
+            -0.008945698849856853,
+            0.020086660981178284,
+            0.04537923261523247,
+            -0.023222751915454865,
+            -0.0033085758332163095,
+            0.018330451101064682,
+            0.0038691519293934107,
+            0.0014475019415840507,
+            -0.00924362801015377,
+            -0.005570481065660715,
+            -0.04202361777424812,
+            0.0016415475402027369,
+            0.001603326410986483,
+            -0.009306349791586399,
+            0.007048364263027906,
+            -0.017812995240092278,
+            -0.007275730837136507,
+            -0.012638445943593979,
+            -0.020274827256798744,
+            -0.004610053263604641,
+            -0.010341259650886059,
+            -0.006503468379378319,
+            -0.011415370739996433,
+            0.013046137988567352,
+            -0.005774327088147402,
+            0.010082531720399857,
+            -0.004104358609765768,
+            0.01768755167722702,
+            -0.0013240183470770717,
+            0.012544362805783749,
+            -0.0005821368540637195,
+            0.009768922813236713,
+            0.00665243249386549,
+            -0.006789636332541704,
+            -0.008255759254097939,
+            -0.007667741738259792,
+            0.018659740686416626,
+            0.009768922813236713,
+            0.010498063638806343,
+            0.0014337814645841718,
+            -0.012552203610539436,
+            -0.011674097739160061,
+            5.987360782455653e-05,
+            -0.010333418846130371,
+            0.0006835698150098324,
+            0.006742595229297876,
+            0.026970380917191505,
+            -0.0033340565860271454,
+            -0.025527779012918472,
+            0.004649254493415356,
+            0.009847325272858143,
+            0.020917724817991257,
+            -0.013148060999810696,
+            0.007354132831096649,
+            -0.010498063638806343,
+            0.013108859769999981,
+            0.026311801746487617,
+            -0.009212266653776169,
+            0.018283409997820854,
+            -0.006205539684742689,
+            0.013947763480246067,
+            0.006836677901446819,
+            -0.00416316045448184,
+            -0.02963605709373951,
+            -0.008302800357341766,
+            0.0016131267184391618,
+            0.0036319850478321314,
+            -0.009369071573019028,
+            0.030874812975525856,
+            0.010482383891940117,
+            0.016088144853711128,
+            -0.009580757468938828,
+            0.010403981432318687,
+            0.00539015606045723,
+            0.019694650545716286,
+            0.012238594703376293,
+            0.00021413620561361313,
+            0.0019806374330073595,
+            -0.008498805575072765,
+            -0.045504678040742874,
+            0.00957291666418314,
+            0.006668113172054291,
+            0.02538665570318699,
+            -0.04196089506149292,
+            0.01569613441824913,
+            -0.010200135409832,
+            -0.0059899333864450455,
+            -0.003700587200000882,
+            0.007667741738259792,
+            -0.003696667030453682,
+            0.0290245208889246,
+            -0.005790007766336203,
+            0.0019355560652911663,
+            0.016684003174304962,
+            0.014073207043111324,
+            -0.007683422416448593,
+            0.01567261293530464,
+            0.023018905892968178,
+            0.01898903027176857,
+            0.013979124836623669,
+            -0.00748349679633975,
+            0.025621861219406128,
+            0.0006811197381466627,
+            -0.003218413097783923,
+            -0.008240078575909138,
+            -0.016166547313332558,
+            0.0024128297809511423,
+            -0.013116699643433094,
+            0.003659426001831889,
+            -0.00561752263456583,
+            -0.004633574280887842,
+            0.010905755683779716,
+            -0.010991998016834259,
+            0.0017454305198043585,
+            0.0018081523012369871,
+            0.007040523923933506,
+            0.0003800060076173395,
+            0.02703310176730156,
+            -0.032521262764930725,
+            0.014081047847867012,
+            -0.03258398175239563,
+            -0.01657423935830593,
+            0.006942520849406719,
+            -0.0044179679825901985,
+            0.00687195872887969,
+            -0.0195221658796072,
+            -0.017358262091875076,
+            0.00591545132920146,
+            0.01829908974468708,
+            0.01502971537411213,
+            0.00716988742351532,
+            0.006091856397688389,
+            0.004884461406618357,
+            -0.010059011168777943,
+            -0.006299622356891632,
+            -0.009557236917316914,
+            -0.00884377583861351,
+            -0.005049106199294329,
+            -0.02010234259068966,
+            0.015453087165951729,
+            -0.0070562041364610195,
+            -0.007922549732029438,
+            0.016746724024415016,
+            -0.01026285719126463,
+            -0.018424533307552338,
+            0.005539120174944401,
+            -0.0038926727138459682,
+            -0.005025585647672415,
+            -0.016213588416576385,
+            0.00922794733196497,
+            -0.0032752547413110733,
+            0.0039240336045622826,
+            -0.01309317909181118,
+            0.026672452688217163,
+            -0.006785716395825148,
+            0.001506303553469479,
+            0.011924984864890575,
+            -0.009408272802829742,
+            -0.024853520095348358,
+            -0.0021991836838424206,
+            -0.014896431006491184,
+            0.0027323190588504076,
+            -0.0028773632366210222,
+            -0.01673104427754879,
+            -0.008443924598395824,
+            0.0035967042203992605,
+            -0.02698606066405773,
+            -0.011384009383618832,
+            0.0099570881575346,
+            0.011783861555159092,
+            0.009431793354451656,
+            0.011932825669646263,
+            -0.0021011808421462774,
+            0.013148060999810696,
+            -0.029008839279413223,
+            -0.014496579766273499,
+            -0.01362631469964981,
+            -0.009549396112561226,
+            0.01412808895111084,
+            0.0017954119248315692,
+            0.007416854612529278,
+            -0.004480689764022827,
+            -0.005252952221781015,
+            -0.0011201724410057068,
+            0.011776020750403404,
+            0.028804993256926537,
+            0.0003814760420937091,
+            -0.003884832374751568,
+            -0.001314218039624393,
+            -0.011423210613429546,
+            0.028381621465086937,
+            -0.0041121989488601685,
+            -0.0072090886533260345,
+            0.03449700027704239,
+            0.01933399960398674,
+            0.027362391352653503,
+            -0.0001465142413508147,
+            -0.006025214213877916,
+            -0.0005091247148811817,
+            0.004931502975523472,
+            0.004974624142050743,
+            -0.01226995512843132,
+            -0.00842040404677391,
+            -0.004751177504658699,
+            0.03465380147099495,
+            0.0022952265571802855,
+            0.011133122257888317,
+            -0.0307493694126606,
+            0.015225720591843128,
+            -0.003577103605493903,
+            -0.007546218577772379,
+            0.003269374603405595,
+            0.036692261695861816,
+            0.02576298639178276,
+            0.0069738817401230335,
+            0.009447473101317883,
+            8.630374941276386e-05,
+            0.0398910753428936,
+            -0.02438310533761978,
+            -0.020196424797177315,
+            -0.02273665741086006,
+            -0.021717429161071777,
+            0.0016601680545136333,
+            0.007271810434758663,
+            0.003628065111115575,
+            -0.0009770882315933704,
+            0.0015259041683748364,
+            0.028585467487573624,
+            0.0015788257587701082,
+            -0.010341259650886059,
+            -0.0014308413956314325,
+            -0.034089308232069016,
+            0.013618473894894123,
+            -0.017389623448252678,
+            -0.010200135409832,
+            -0.016511518508195877,
+            -0.013006936758756638,
+            -0.025982512161135674,
+            -0.010607827454805374,
+            -0.0009599378099665046,
+            0.015970543026924133,
+            0.010004129260778427,
+            -0.0019120353972539306,
+            0.005613602697849274,
+            -0.023834289982914925,
+            -0.006703393999487162,
+            0.020917724817991257,
+            0.01567261293530464,
+            0.015790216624736786,
+            0.0046531748957931995,
+            0.01622927002608776,
+            -0.0003020937438122928,
+            -0.015923500061035156,
+            -0.0020913805346935987,
+            0.002767600119113922,
+            0.006413305643945932,
+            0.017091695219278336,
+            0.012222914025187492,
+            -0.018769502639770508,
+            0.0071267662569880486,
+            -0.012795249931514263,
+            0.03139226883649826,
+            -0.006197699345648289,
+            0.01205826923251152,
+            0.009682680480182171,
+            0.0032830950804054737,
+            -0.012489481829106808,
+            -0.006158498115837574,
+            -0.00479429867118597,
+            0.01569613441824913,
+            -0.00831064023077488,
+            -0.011062560603022575,
+            -0.005676324479281902,
+            0.009729721583425999,
+            -0.005535200238227844,
+            -0.0010349099757149816,
+            -0.015609892085194588,
+            -0.014347615651786327,
+            0.0030753291212022305,
+            -0.012826611287891865,
+            -0.0010711710201576352,
+            0.019036071375012398,
+            -0.005033425521105528,
+            -0.0004390527028590441,
+            0.014684745110571384,
+            0.016589920967817307,
+            0.004649254493415356,
+            0.018926307559013367,
+            -0.009549396112561226,
+            0.0022932663559913635,
+            -0.021623345091938972,
+            -0.0024716316256672144,
+            -0.032364457845687866,
+            0.019647609442472458,
+            0.0017836516490206122,
+            -0.020070981234312057,
+            0.010944956913590431,
+            -0.0016239070100709796,
+            -0.009165225550532341,
+            -0.0099727688357234,
+            0.002452031010761857,
+            -0.024179259315133095,
+            0.024147899821400642,
+            -0.016048945486545563,
+            0.008906497620046139,
+            0.01161921676248312,
+            0.041333675384521484,
+            0.0038573916535824537,
+            -0.00747565645724535,
+            0.021152932196855545,
+            -0.0002717128663789481,
+            0.02982422336935997,
+            -0.001121152425184846,
+            -0.012081789784133434,
+            -0.022689616307616234,
+            -0.005237271543592215,
+            -0.015202200040221214,
+            -0.014112408272922039,
+            0.0147239463403821,
+            -0.0195221658796072,
+            -0.015217880718410015,
+            -0.010004129260778427,
+            -0.019741691648960114,
+            -0.011509452946484089,
+            0.025982512161135674,
+            0.015758855268359184,
+            -0.011705459095537663,
+            -0.006425065919756889,
+            -0.0055077592842280865,
+            -0.02711150422692299,
+            0.021952634677290916,
+            0.008012712001800537,
+            -0.01983577385544777,
+            -0.0032772149424999952,
+            -0.009768922813236713,
+            0.004378766752779484,
+            0.01588430069386959,
+            -0.0001214010117109865,
+            0.05789223685860634,
+            -0.008804574608802795,
+            -0.0015945062041282654,
+            -0.020008258521556854,
+            0.009298508986830711,
+            -0.01575101539492607,
+            -0.006374104414135218,
+            -0.026672452688217163,
+            -0.008255759254097939,
+            -0.005119668319821358,
+            -0.021105891093611717,
+            -0.0022481849882751703,
+            -0.008287119679152966,
+            0.0057547264732420444,
+            -0.01280309073626995,
+            0.0008050933247432113,
+            -0.01665264181792736,
+            -0.00624866085126996,
+            -0.012089629657566547,
+            -0.012371878139674664,
+            0.01925559714436531,
+            -0.028303219005465508,
+            -0.0033869780600070953,
+            0.008153836242854595,
+            0.01353223156183958,
+            0.006268261466175318,
+            -0.021043168380856514,
+            -0.012614925391972065,
+            -0.011917144991457462,
+            -0.018894946202635765,
+            0.010733271017670631,
+            -0.010741110891103745,
+            0.013140220195055008,
+            -0.03186268359422684,
+            -0.012607084587216377,
+            -0.0019522164948284626,
+            0.019976899027824402,
+            0.01753074675798416,
+            -0.0004733536916319281,
+            -0.0059193712659180164,
+            -0.014002645388245583,
+            0.02278370037674904,
+            -0.013877201825380325,
+            -0.032678067684173584,
+            -0.0013494990998879075,
+            0.002218784298747778,
+            -0.00104079011362046,
+            -0.007816705852746964,
+            0.0027538796421140432,
+            0.004214121960103512,
+            0.004092598333954811,
+            -0.007546218577772379,
+            -0.0016464476939290762,
+            -0.01508459635078907,
+            0.0047668581828475,
+            -3.659793583210558e-05,
+            0.029730141162872314,
+            -0.0035339822061359882,
+            0.008992739953100681,
+            -0.02182719111442566,
+            0.0007546218694187701,
+            -0.01317158155143261,
+            -0.005782167427241802,
+            0.000176037588971667,
+            0.003330136416479945,
+            0.01956920698285103,
+            -0.025794345885515213,
+            -0.013038297183811665,
+            0.01205826923251152,
+            -0.019929856061935425,
+            -0.03572007268667221,
+            0.002736239228397608,
+            -0.029463572427630424,
+            -0.0013053978327661753,
+            -0.003777029225602746,
+            -0.015703974291682243,
+            -0.026217719539999962,
+            0.0060722557827830315,
+            -0.005366635508835316,
+            -0.005786087363958359,
+            0.003516341792419553,
+            -0.015233561396598816,
+            -0.015531489625573158,
+            0.005151029210537672,
+            0.007346292492002249,
+            -0.006503468379378319,
+            0.008561528287827969,
+            -0.0238186102360487,
+            0.011274246498942375,
+            -0.008145995438098907,
+            0.0037358682602643967,
+            0.005135348532348871,
+            0.026123635470867157,
+            0.008561528287827969,
+            -0.004000475630164146,
+            0.009306349791586399,
+            0.005472478456795216,
+            -0.010333418846130371,
+            0.011070400476455688,
+            0.02400677464902401,
+            0.0005184350302442908,
+            0.0032340935431420803,
+            -0.002973406109958887,
+            0.005746886134147644,
+            -0.001685648807324469,
+            -0.007612860295921564,
+            0.020933406427502632,
+            0.01593918167054653,
+            -0.019239917397499084,
+            0.0062369005754590034,
+            -0.020039619877934456,
+            0.010976317338645458,
+            -0.028742272406816483,
+            0.0005703765200451016,
+            0.005840969271957874,
+            -0.0050020646303892136,
+            0.011979866772890091,
+            0.006860198453068733,
+            0.03151771053671837,
+            0.026280440390110016,
+            -0.0018875346286222339,
+            0.0048178196884691715,
+            -0.02002394013106823,
+            0.01280309073626995,
+            0.009118183515965939,
+            0.010066851042211056,
+            -0.010694069787859917,
+            0.00791470892727375,
+            -0.00878889486193657,
+            0.014606342650949955,
+            -0.00260295532643795,
+            -0.014214331284165382,
+            0.006350583862513304,
+            -0.006287862081080675,
+            0.014441697858273983,
+            0.014081047847867012,
+            -0.013148060999810696,
+            -0.026923339813947678,
+            -0.006162418518215418,
+            0.018095243722200394,
+            -0.023912692442536354,
+            0.02443014644086361,
+            0.006464267149567604,
+            0.02265825681388378,
+            -0.009988449513912201,
+            -0.0030655288137495518,
+            -0.001984557369723916,
+            0.002657837001606822,
+            -0.00685235857963562,
+            0.00014345165982376784,
+            0.02243872918188572,
+            -0.009510194882750511,
+            0.005601841956377029,
+            0.012175872921943665,
+            -0.0016052864957600832,
+            0.03622184693813324,
+            0.018769502639770508,
+            0.024978963658213615,
+            -0.0017366103129461408,
+            0.010584305971860886,
+            0.0008266539080068469,
+            0.010537264868617058,
+            0.013845840469002724,
+            -0.010984158143401146,
+            0.0008913357742130756,
+            -0.014347615651786327,
+            0.017028972506523132,
+            0.024665353819727898,
+            0.009463153779506683,
+            0.0002709778491407633,
+            0.019506484270095825,
+            0.023191390559077263,
+            0.0031419708393514156,
+            -0.00707188481464982,
+            -0.007753984536975622,
+            0.017091695219278336,
+            0.01420649141073227,
+            0.007240449544042349,
+            0.0054254368878901005,
+            -0.007189488038420677,
+            -0.006036974955350161,
+            0.019192876294255257,
+            -0.019804412499070168,
+            -0.0018150125397369266,
+            0.0015288442373275757,
+            -0.04042420908808708,
+            0.010286377742886543,
+            -0.00031115900492295623,
+            -0.014669064432382584,
+            0.011180163361132145,
+            0.004072997719049454,
+            -0.010976317338645458,
+            -0.010176614858210087,
+            -0.016448795795440674,
+            -0.006879799067974091,
+            0.007593259681016207,
+            0.0024324303958564997,
+            0.008506646379828453,
+            -0.01050590444356203,
+            -0.005884090438485146,
+            0.019961217418313026,
+            0.003396778367459774,
+            0.0062369005754590034,
+            -0.0001754250843077898
+          ],
+          "keyphrases": [
+            "scaling up model capacity",
+            "training efficiency",
+            "sparsely gated mixture-of-experts",
+            "expert choice method",
+            "downstream performance"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "86d0cb2e-1f6d-4e38-a077-7b7de2fe9e4a",
+        "properties": {
+          "page_content": "3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n",
+          "embedding": [
+            0.010104401037096977,
+            -0.020574966445565224,
+            -0.02167346142232418,
+            -0.031542472541332245,
+            0.03588413819670677,
+            0.03464615345001221,
+            -0.009755672886967659,
+            0.046869076788425446,
+            0.004803731571882963,
+            0.0364421010017395,
+            0.016547156497836113,
+            -0.007223033811897039,
+            0.009546436369419098,
+            -0.0017371026333421469,
+            0.005762734450399876,
+            0.04048734903335571,
+            0.011603932827711105,
+            0.01784616895020008,
+            0.011150586418807507,
+            -0.02915368229150772,
+            0.005383492447435856,
+            -0.01708768494427204,
+            -0.04317255690693855,
+            -0.006194285582751036,
+            0.009546436369419098,
+            0.005871712230145931,
+            0.027340294793248177,
+            0.0036812627222388983,
+            -0.055901139974594116,
+            0.023800702765583992,
+            0.021708333864808083,
+            0.010984940454363823,
+            -0.01007824670523405,
+            -0.010601338930428028,
+            -0.026573093608021736,
+            -0.021969880908727646,
+            0.04634598642587662,
+            0.007685098797082901,
+            -0.006399163510650396,
+            0.042475100606679916,
+            0.03703493997454643,
+            0.012728581205010414,
+            -0.012772172689437866,
+            -0.014175803400576115,
+            -0.003430614247918129,
+            -0.0012979229213669896,
+            -0.004712190479040146,
+            0.0065735275857150555,
+            0.009311044588685036,
+            -0.026172054931521416,
+            -0.001842810888774693,
+            -0.021900134161114693,
+            0.02798544242978096,
+            0.020609840750694275,
+            -0.049379922449588776,
+            0.028072625398635864,
+            -0.004890913609415293,
+            0.06461934745311737,
+            0.047461915761232376,
+            0.014602995477616787,
+            0.03794163465499878,
+            -0.034035876393318176,
+            -0.029275737702846527,
+            -0.003177786245942116,
+            0.03234454616904259,
+            0.007654585409909487,
+            -0.01033979281783104,
+            0.0010679803090170026,
+            0.011717269197106361,
+            -0.005706066265702248,
+            -0.044532597064971924,
+            0.010496720671653748,
+            -0.014925569295883179,
+            0.002881367225199938,
+            0.004034350160509348,
+            -0.020017001777887344,
+            0.03030448593199253,
+            0.0268520750105381,
+            0.014899414964020252,
+            0.010740830563008785,
+            0.010653648525476456,
+            0.03204812481999397,
+            -0.0026961052790284157,
+            0.009555154480040073,
+            0.0364421010017395,
+            -0.029415227472782135,
+            -0.01360040158033371,
+            -0.062178246676921844,
+            0.010540311224758625,
+            0.0028072623535990715,
+            -0.02815980650484562,
+            0.0018340926617383957,
+            -0.006569168530404568,
+            0.0036986989434808493,
+            -0.033094312995672226,
+            0.0727447122335434,
+            0.010653648525476456,
+            0.00023280335881281644,
+            -0.024044813588261604,
+            0.006394804455339909,
+            -0.015762517228722572,
+            0.021969880908727646,
+            0.0006560450419783592,
+            0.004008195362985134,
+            0.0229811929166317,
+            -0.02859571762382984,
+            -0.004690395202487707,
+            0.03815086930990219,
+            0.02148166112601757,
+            0.08676359057426453,
+            0.01007824670523405,
+            -0.017593340948224068,
+            -0.011211613193154335,
+            0.018482597544789314,
+            -0.04278895631432533,
+            -0.017689241096377373,
+            -0.03965040296316147,
+            -0.03215274587273598,
+            0.0114818774163723,
+            -0.025038689374923706,
+            0.01996469311416149,
+            0.02566639892756939,
+            -0.0011344566009938717,
+            -0.025422289967536926,
+            -0.07155904173851013,
+            0.04292844980955124,
+            -0.023696085438132286,
+            0.01780257746577263,
+            0.001439593848772347,
+            -0.0022144243121147156,
+            -0.00422179140150547,
+            0.0037183149252086878,
+            -0.020261110737919807,
+            -0.041882261633872986,
+            0.01856978051364422,
+            0.010479284450411797,
+            0.046834204345941544,
+            0.0633290484547615,
+            -0.021132932975888252,
+            -0.0072448295541107655,
+            0.017375385388731956,
+            -0.022894009947776794,
+            0.003243172774091363,
+            -0.0038142153061926365,
+            -0.029450101777911186,
+            0.013975284993648529,
+            0.028508534654974937,
+            0.08578714728355408,
+            -0.01956365443766117,
+            0.012022406794130802,
+            -0.033286113291978836,
+            0.037662651389837265,
+            -0.009799264371395111,
+            0.023417102172970772,
+            -0.025404853746294975,
+            -0.021708333864808083,
+            -0.025352545082569122,
+            0.010705957189202309,
+            -0.014995315112173557,
+            0.0059632533229887486,
+            -0.005889148451387882,
+            -0.0062945447862148285,
+            -0.018691834062337875,
+            0.010470565408468246,
+            0.007685098797082901,
+            0.047985006123781204,
+            0.029816266149282455,
+            0.005858634598553181,
+            -0.07769665867090225,
+            0.038325235247612,
+            0.017514877021312714,
+            0.036546722054481506,
+            0.01543994341045618,
+            0.033443041145801544,
+            -0.026538221165537834,
+            0.017680522054433823,
+            0.0032475318294018507,
+            0.01094134896993637,
+            -0.03197837993502617,
+            0.044916197657585144,
+            -0.00843050517141819,
+            0.002720080316066742,
+            0.01972058229148388,
+            0.013312701135873795,
+            0.04062684252858162,
+            0.027915697544813156,
+            0.034210242331027985,
+            0.03815086930990219,
+            -0.01332141924649477,
+            0.010453129187226295,
+            0.029380355030298233,
+            0.03515180945396423,
+            0.02796800620853901,
+            0.0027636713348329067,
+            0.0041890982538461685,
+            -0.026590529829263687,
+            -0.006721737328916788,
+            0.009964910335838795,
+            0.007039951626211405,
+            0.016390228644013405,
+            -0.012780890800058842,
+            -0.028124934062361717,
+            0.0220221895724535,
+            -0.028648026287555695,
+            0.030740395188331604,
+            0.008626664988696575,
+            -0.01083673071116209,
+            0.013208082877099514,
+            0.009895164519548416,
+            -0.029833702370524406,
+            0.004243587143719196,
+            -0.006250953767448664,
+            -0.004468080587685108,
+            -0.025021253153681755,
+            -0.01933698169887066,
+            0.00797280017286539,
+            0.0029532923363149166,
+            0.005021686665713787,
+            0.009380790404975414,
+            -0.022545281797647476,
+            0.004284998401999474,
+            0.02127242274582386,
+            0.022876573726534843,
+            -0.018325669690966606,
+            -0.018901070579886436,
+            -0.011438286863267422,
+            -0.01914518140256405,
+            -0.011865478940308094,
+            0.044393107295036316,
+            -0.012711144983768463,
+            -0.009694645181298256,
+            0.020574966445565224,
+            0.006774046458303928,
+            -0.005043482407927513,
+            0.017872322350740433,
+            -0.029484974220395088,
+            -0.010932630859315395,
+            -0.0035592077765613794,
+            0.011072122491896152,
+            0.010243892669677734,
+            -0.00997362844645977,
+            0.01550968922674656,
+            -0.050879452377557755,
+            -0.010932630859315395,
+            0.007881258614361286,
+            0.015082497149705887,
+            -0.0110285310074687,
+            0.02339966595172882,
+            0.012161897495388985,
+            0.016224581748247147,
+            -0.003081885864958167,
+            -0.014829669147729874,
+            0.008221268653869629,
+            0.017384104430675507,
+            0.007881258614361286,
+            0.05712168663740158,
+            0.02338222973048687,
+            -0.03947603702545166,
+            0.006538654677569866,
+            -0.029973194003105164,
+            0.05579652264714241,
+            0.007994594983756542,
+            -0.029833702370524406,
+            0.053215932101011276,
+            -0.023434538394212723,
+            -0.0024193022400140762,
+            0.060853078961372375,
+            -0.0007007258245721459,
+            -0.013548092916607857,
+            0.0013011923292651772,
+            0.02435866929590702,
+            -0.056075502187013626,
+            -0.04899631813168526,
+            -0.02148166112601757,
+            0.02857828140258789,
+            -0.02453303337097168,
+            0.0077156126499176025,
+            -0.015405070967972279,
+            0.04735729843378067,
+            0.036721084266901016,
+            0.018116433173418045,
+            -0.008731283247470856,
+            0.02008674666285515,
+            0.04749678820371628,
+            -0.007977158762514591,
+            -0.004742704331874847,
+            -0.02833417057991028,
+            0.005152460187673569,
+            -0.033216364681720734,
+            -0.009485408663749695,
+            -0.027462350204586983,
+            -0.011133149266242981,
+            -0.02125498652458191,
+            -0.0030317562632262707,
+            0.022336045280098915,
+            0.0041389684192836285,
+            5.1253518904559314e-05,
+            -0.0020389705896377563,
+            0.040522221475839615,
+            -0.024951506406068802,
+            -0.04470696300268173,
+            -0.00825178250670433,
+            0.01704409345984459,
+            0.0038556267973035574,
+            -0.05475033447146416,
+            -0.006250953767448664,
+            -0.0029707287903875113,
+            0.01304243691265583,
+            -0.023539157584309578,
+            -0.00863538309931755,
+            0.016573309898376465,
+            0.002870469354093075,
+            0.009807982482016087,
+            -0.02800287865102291,
+            -0.014629150740802288,
+            0.007406116463243961,
+            0.01841285265982151,
+            0.02817724272608757,
+            -0.0555872842669487,
+            -0.03225736320018768,
+            0.02067958563566208,
+            -0.009040780365467072,
+            -0.06409624963998795,
+            0.007833308540284634,
+            -0.001987751107662916,
+            0.01799437776207924,
+            0.016948193311691284,
+            0.04160328209400177,
+            -0.019581090658903122,
+            -0.022283734753727913,
+            -0.016599465161561966,
+            -0.0364072285592556,
+            -0.028700334951281548,
+            -0.009712082333862782,
+            -0.008299732580780983,
+            -0.028857262805104256,
+            -0.0017839629435911775,
+            0.01342603750526905,
+            -0.03759290650486946,
+            0.006176849361509085,
+            -0.021603714674711227,
+            0.0028486738447099924,
+            -0.03703493997454643,
+            0.007842026650905609,
+            -0.0030339357908815145,
+            -0.01613740064203739,
+            -0.02125498652458191,
+            -0.01561430748552084,
+            0.014725050888955593,
+            -0.0002036518562817946,
+            0.0017142173601314425,
+            0.0229637548327446,
+            0.007484580390155315,
+            0.009328480809926987,
+            -0.008535124361515045,
+            -0.029798829928040504,
+            -0.013382446952164173,
+            0.006586604751646519,
+            0.018534906208515167,
+            0.019999565556645393,
+            0.005501188337802887,
+            -0.04958915710449219,
+            -0.03584926575422287,
+            0.021621152758598328,
+            -0.05890020355582237,
+            0.006394804455339909,
+            -0.0017425514524802566,
+            -0.008622306399047375,
+            0.0033150978852063417,
+            -0.005832480266690254,
+            -0.026294110342860222,
+            0.015570716932415962,
+            0.0086135882884264,
+            0.012275234796106815,
+            0.007488939445465803,
+            0.015980472788214684,
+            0.02852597087621689,
+            -0.02451559714972973,
+            -0.05788889154791832,
+            0.03427998721599579,
+            -0.019075434654951096,
+            0.014210676774382591,
+            0.0440792515873909,
+            0.005475033540278673,
+            -0.006551732309162617,
+            0.025003815069794655,
+            0.013399883173406124,
+            -0.010034655220806599,
+            -0.03407074883580208,
+            0.04376539587974548,
+            -0.003430614247918129,
+            -0.07121030986309052,
+            0.005479392595589161,
+            0.0014810053398832679,
+            0.013103464618325233,
+            0.015283015556633472,
+            -0.003430614247918129,
+            -0.023643774911761284,
+            -0.01956365443766117,
+            -0.014088621363043785,
+            -0.018168741837143898,
+            0.029484974220395088,
+            0.003539591794833541,
+            0.005004250444471836,
+            -0.0006091846735216677,
+            -0.0014341449132189155,
+            0.004677317570894957,
+            0.014027593657374382,
+            0.04080120474100113,
+            -0.0364769771695137,
+            0.0025958458427339792,
+            -0.04275408387184143,
+            -0.03874370828270912,
+            -0.0046380856074392796,
+            0.018360542133450508,
+            0.026485910639166832,
+            -0.025910509750247,
+            0.02320786565542221,
+            0.0100608104839921,
+            -0.015675334259867668,
+            -0.005008609499782324,
+            0.028090061619877815,
+            0.025369981303811073,
+            0.01285063661634922,
+            -0.004533467348664999,
+            0.014690177515149117,
+            -0.012623962946236134,
+            0.0026045641861855984,
+            0.01574508100748062,
+            -0.00010305192699888721,
+            0.03490769863128662,
+            0.001937621389515698,
+            0.018918508663773537,
+            -0.04693882167339325,
+            -0.02699156664311886,
+            -0.027689022943377495,
+            -0.0172620490193367,
+            0.03874370828270912,
+            -0.006503781769424677,
+            -0.012728581205010414,
+            0.020261110737919807,
+            0.004282819107174873,
+            0.0017044093692675233,
+            -0.04066171497106552,
+            -0.028630590066313744,
+            -0.022283734753727913,
+            -0.044916197657585144,
+            -0.005618884228169918,
+            -0.02183038927614689,
+            -0.019057998433709145,
+            -0.05342517048120499,
+            0.0010527233825996518,
+            -0.07811512798070908,
+            -0.005370415281504393,
+            0.04048734903335571,
+            0.004686036147177219,
+            0.009424380958080292,
+            -0.028682898730039597,
+            0.01025261078029871,
+            -0.036337483674287796,
+            -0.02568383514881134,
+            0.017785141244530678,
+            0.04387001320719719,
+            -0.011813169345259666,
+            0.0019997386261820793,
+            0.019057998433709145,
+            -0.05056559666991234,
+            -0.005370415281504393,
+            -0.0031799657735973597,
+            0.007641507778316736,
+            0.012920381501317024,
+            -0.016974348574876785,
+            0.05457597225904465,
+            0.011054685339331627,
+            0.02223142609000206,
+            0.02606743760406971,
+            -0.00020024630066473037,
+            -0.038046251982450485,
+            0.017584621906280518,
+            0.008535124361515045,
+            0.02796800620853901,
+            0.0015681873774155974,
+            0.0073014977388083935,
+            -0.019807765260338783,
+            0.008979752659797668,
+            -0.009485408663749695,
+            0.01646869257092476,
+            0.018866198137402534,
+            0.02645103819668293,
+            -0.008644101209938526,
+            -0.002175192581489682,
+            0.0033434322103857994,
+            -0.0034785643219947815,
+            0.028700334951281548,
+            0.0076371487230062485,
+            -0.038848329335451126,
+            0.006813278421759605,
+            -0.002189359627664089,
+            0.023120682686567307,
+            0.030374230816960335,
+            -0.04108018800616264,
+            -0.02338222973048687,
+            -0.002796364715322852,
+            0.016974348574876785,
+            0.011019812896847725,
+            -0.020662149414420128,
+            -0.0031668883748352528,
+            0.044602345675230026,
+            -0.010479284450411797,
+            -0.032484035938978195,
+            0.01293781865388155,
+            -0.0018504393519833684,
+            0.0059806895442306995,
+            0.0024476363323628902,
+            0.012772172689437866,
+            0.013469628989696503,
+            -0.019598528742790222,
+            0.04087095335125923,
+            -0.004965018481016159,
+            0.013208082877099514,
+            -0.012031124904751778,
+            -0.010052092373371124,
+            -0.015195833519101143,
+            0.03926680237054825,
+            -0.0047906544059515,
+            0.02013905718922615,
+            0.024864325299859047,
+            0.006338136270642281,
+            0.05216974765062332,
+            -0.04125455394387245,
+            -0.03724417835474014,
+            0.005971971433609724,
+            0.0016215863870456815,
+            0.005871712230145931,
+            -0.011272640898823738,
+            -0.030932195484638214,
+            -0.012074715457856655,
+            0.004768858663737774,
+            -0.015535843558609486,
+            -0.015684053301811218,
+            0.00019043832435272634,
+            0.022144244983792305,
+            0.0011257383739575744,
+            0.005475033540278673,
+            0.00287700816988945,
+            0.014585559256374836,
+            0.03581439331173897,
+            -0.02589307352900505,
+            0.010758266784250736,
+            0.03147272393107414,
+            -0.009136680513620377,
+            -0.026764893904328346,
+            -0.027305422350764275,
+            0.015073779039084911,
+            -0.018918508663773537,
+            -0.028874699026346207,
+            3.807404209510423e-05,
+            -0.0034872826654464006,
+            0.008391273207962513,
+            -0.03584926575422287,
+            -0.0076763806864619255,
+            0.0021283321548253298,
+            -0.00196377607062459,
+            0.0007078094058670104,
+            -0.022318609058856964,
+            -0.013591683469712734,
+            0.01741897687315941,
+            0.024219177663326263,
+            0.005727861542254686,
+            0.03028704971075058,
+            -0.010243892669677734,
+            -0.00016768925706855953,
+            0.006525577511638403,
+            0.01589328981935978,
+            0.023050937801599503,
+            0.023312482982873917,
+            0.026573093608021736,
+            0.0009344827267341316,
+            -0.012327543459832668,
+            0.004252305254340172,
+            0.032484035938978195,
+            -0.016058936715126038,
+            -0.0030993223190307617,
+            0.014812232926487923,
+            0.03274558484554291,
+            0.010627493262290955,
+            -0.012188052758574486,
+            -0.02528279833495617,
+            0.015431225299835205,
+            -0.022283734753727913,
+            0.012510626576840878,
+            0.04777577146887779,
+            0.004393976181745529,
+            -0.029240863397717476,
+            0.02430635876953602,
+            -0.009546436369419098,
+            -0.024044813588261604,
+            0.027165930718183517,
+            0.003757546888664365,
+            -0.03271070867776871,
+            -6.98818766977638e-05,
+            -0.00018321855168323964,
+            0.014794795773923397,
+            -0.026015127077698708,
+            -0.015579435043036938,
+            0.02322530187666416,
+            0.009293608367443085,
+            0.0032627887558192015,
+            -0.021603714674711227,
+            0.001829733606427908,
+            0.015344043262302876,
+            0.0018177460879087448,
+            0.009424380958080292,
+            0.009145398624241352,
+            0.01505634281784296,
+            0.006939692422747612,
+            -0.013190646655857563,
+            -0.0032344546634703875,
+            -0.02720080316066742,
+            -0.006098385434597731,
+            0.0652819275856018,
+            -0.007292779628187418,
+            0.03686057776212692,
+            0.0015213270671665668,
+            0.012868072837591171,
+            0.025788454338908195,
+            -0.025160742923617363,
+            -0.02437610551714897,
+            -0.005047841463238001,
+            0.006961487699300051,
+            -0.04292844980955124,
+            -0.013269110582768917,
+            -0.023818138986825943,
+            -0.030217302963137627,
+            0.01933698169887066,
+            0.0005187332862988114,
+            0.0019267236348241568,
+            0.024829450994729996,
+            0.0051742554642260075,
+            -0.015038905665278435,
+            -0.017009221017360687,
+            0.0383601076900959,
+            0.016625620424747467,
+            0.028229551389813423,
+            -0.015989189967513084,
+            0.01625945419073105,
+            -0.008099214173853397,
+            0.003171247662976384,
+            0.03236198052763939,
+            -0.0026982848066836596,
+            -0.0030186788644641638,
+            0.009197707287967205,
+            0.01881388947367668,
+            0.040731459856033325,
+            -0.01745384931564331,
+            -0.030827578157186508,
+            -0.008914366364479065,
+            -0.014236831106245518,
+            0.021969880908727646,
+            -0.0009491947130300105,
+            -0.00041084547410719097,
+            -0.003934090491384268,
+            0.04352128505706787,
+            -0.029572155326604843,
+            0.014428631402552128,
+            -0.024445850402116776,
+            -0.039022691547870636,
+            0.017706677317619324,
+            -0.006102744489908218,
+            0.032693274319171906,
+            0.006883023772388697,
+            -0.007833308540284634,
+            -0.032867636531591415,
+            0.0016749853966757655,
+            0.0086179468780756,
+            -0.03194350749254227,
+            0.020418038591742516,
+            0.02472483366727829,
+            -0.00651250034570694,
+            0.0033063797745853662,
+            -0.016782546415925026,
+            0.008142804726958275,
+            -0.013844511471688747,
+            -0.022946318611502647,
+            0.022737082093954086,
+            -0.004298075567930937,
+            -0.028055187314748764,
+            0.006686864420771599,
+            -0.022458098828792572,
+            0.004707831423729658,
+            -0.0016771649243310094,
+            -0.00017014124023262411,
+            -0.00797280017286539,
+            0.010461847297847271,
+            -0.007571762427687645,
+            -0.01073211245238781,
+            -0.022353481501340866,
+            0.009991064667701721,
+            0.025631526485085487,
+            0.0017588981427252293,
+            -0.004672958515584469,
+            0.006848151329904795,
+            -0.03658159449696541,
+            0.00574093870818615,
+            0.02413199469447136,
+            -0.012205488979816437,
+            -0.008713847026228905,
+            -0.018255924805998802,
+            -0.029380355030298233,
+            -0.002366993110626936,
+            0.01130751334130764,
+            0.03584926575422287,
+            0.025788454338908195,
+            0.030409103259444237,
+            0.03391382470726967,
+            0.010810576379299164,
+            -0.00398204056546092,
+            0.009127962402999401,
+            -0.029031626880168915,
+            0.04006887599825859,
+            0.0040823002345860004,
+            0.02608487382531166,
+            0.016172273084521294,
+            -0.0459623858332634,
+            -0.00968592707067728,
+            0.014681459404528141,
+            0.027898261323571205,
+            0.044148996472358704,
+            0.025038689374923706,
+            -0.0010232995264232159,
+            0.027654150500893593,
+            -0.022196553647518158,
+            -0.02878751792013645,
+            0.01956365443766117,
+            -0.005235282704234123,
+            -0.00918898917734623,
+            -0.0013131798477843404,
+            0.021533969789743423,
+            -0.011385977268218994,
+            0.016058936715126038,
+            0.03415793180465698,
+            -0.008461019024252892,
+            0.012135743163526058,
+            0.0005590550135821104,
+            -0.008286654949188232,
+            0.026764893904328346,
+            -0.007214315701276064,
+            0.03485538810491562,
+            -0.01650356501340866,
+            -0.0018656961619853973,
+            0.000780279457103461,
+            -0.010766984894871712,
+            0.028020314872264862,
+            0.007270984351634979,
+            -0.06172490119934082,
+            0.00873564276844263,
+            0.011063403449952602,
+            0.013835793361067772,
+            -0.001278307056054473,
+            -0.0033957413397729397,
+            -0.025788454338908195,
+            0.019040562212467194,
+            0.008129727095365524,
+            0.02472483366727829,
+            0.010679802857339382,
+            0.019493909552693367,
+            -0.016643056645989418,
+            0.016599465161561966,
+            -0.0026590528432279825,
+            -0.02854340709745884,
+            0.007266624830663204,
+            0.013181927613914013,
+            0.0402083694934845,
+            -0.026538221165537834,
+            -0.002582768676802516,
+            0.010889039374887943,
+            -0.006970205809921026,
+            -0.011194176971912384,
+            -0.030443977564573288,
+            -0.0010756086558103561,
+            -0.028421353548765182,
+            0.02416686899960041,
+            -0.005139382556080818,
+            0.019406726583838463,
+            -0.027357731014490128,
+            0.0063337767496705055,
+            0.00043264098349027336,
+            -0.027654150500893593,
+            0.006390445400029421,
+            0.01782001368701458,
+            0.0031189383007586002,
+            -0.031176306307315826,
+            -0.006006844341754913,
+            0.023242738097906113,
+            0.03619799390435219,
+            0.006713018752634525,
+            -0.006848151329904795,
+            0.021533969789743423,
+            -0.015579435043036938,
+            -0.006150694563984871,
+            0.036546722054481506,
+            -0.025910509750247,
+            0.012231643311679363,
+            0.03759290650486946,
+            -0.03543078899383545,
+            -0.03196094557642937,
+            -0.0004021272761747241,
+            -0.04142891615629196,
+            0.001680434332229197,
+            -0.002975087845697999,
+            0.00835204217582941,
+            -0.0009715351043269038,
+            -0.027113622054457664,
+            0.024812014773488045,
+            -0.0003577189054340124,
+            0.013373728841543198,
+            -0.006848151329904795,
+            0.0021065366454422474,
+            -0.025143306702375412,
+            -0.01862208917737007,
+            -0.010793139226734638,
+            0.00015651904686819762,
+            0.0023059656377881765,
+            -0.008112290874123573,
+            -0.03944116458296776,
+            -0.020976005122065544,
+            0.007257906720042229,
+            0.00045770584256388247,
+            0.03584926575422287,
+            0.018482597544789314,
+            0.012188052758574486,
+            -0.02165602520108223,
+            -0.01898825354874134,
+            0.04756653308868408,
+            -0.026416165754199028,
+            0.0402432419359684,
+            -0.012135743163526058,
+            -0.017488721758127213,
+            0.002946753753349185,
+            0.02625923790037632,
+            -0.020923694595694542,
+            0.016817420721054077,
+            -0.01561430748552084,
+            0.005719143431633711,
+            -0.01409733947366476,
+            -0.004755781497806311,
+            0.009642336517572403,
+            -0.0017796038882806897,
+            0.018255924805998802,
+            0.0009072383400052786,
+            -0.0031123997177928686,
+            0.03183889016509056,
+            0.013652711175382137,
+            0.007763562723994255,
+            -0.016773829236626625,
+            -0.012222925201058388,
+            -0.010766984894871712,
+            0.007087901700288057,
+            -0.015073779039084911,
+            -0.007066106423735619,
+            -0.029659338295459747,
+            0.001694601378403604,
+            0.0007154378108680248,
+            -0.015256861224770546,
+            0.01235369872301817,
+            0.02706131339073181,
+            -0.013295264914631844,
+            -0.03157734498381615,
+            -0.006229158490896225,
+            -0.03944116458296776,
+            -0.021045750007033348,
+            0.019511345773935318,
+            -0.0005190056981518865,
+            0.009127962402999401,
+            0.004250125493854284,
+            0.00043509298120625317,
+            -0.022946318611502647,
+            -0.017976941540837288,
+            0.00714892940595746,
+            -0.009572590701282024,
+            0.00107724335975945,
+            0.00131862866692245,
+            0.01822105050086975,
+            -0.010156710632145405,
+            0.02800287865102291,
+            -0.017497440800070763,
+            0.015265579335391521,
+            -0.029851138591766357,
+            -0.02760184183716774,
+            -0.03199581801891327,
+            0.01304243691265583,
+            -0.014611713588237762,
+            -0.008195114322006702,
+            -0.010113119147717953,
+            0.0004999346565455198,
+            0.007567403372377157,
+            -0.009441817179322243,
+            0.00842614658176899,
+            0.011603932827711105,
+            0.013303983025252819,
+            0.007406116463243961,
+            -0.03244916349649429,
+            -0.02029598504304886,
+            -0.02397506684064865,
+            0.043974634259939194,
+            -0.030513722449541092,
+            0.01703537441790104,
+            -0.017392821609973907,
+            0.005823761690407991,
+            -0.0363723561167717,
+            -0.019075434654951096,
+            0.008953598327934742,
+            0.020749330520629883,
+            -0.025910509750247,
+            0.02741004154086113,
+            0.02934548258781433,
+            0.03013012185692787,
+            -0.036546722054481506,
+            0.025352545082569122,
+            -0.010505438782274723,
+            -0.02512587048113346,
+            0.027357731014490128,
+            0.03731392323970795,
+            0.03679082915186882,
+            -0.013460910879075527,
+            -0.012589089572429657,
+            -0.014690177515149117,
+            0.012318825349211693,
+            -0.0021555765997618437,
+            0.005801966413855553,
+            -0.037069812417030334,
+            0.01862208917737007,
+            0.009895164519548416,
+            0.005121946334838867,
+            0.03773239627480507,
+            -0.01898825354874134,
+            0.02054009400308132,
+            0.007471502758562565,
+            -0.01839541457593441,
+            0.006974565330892801,
+            0.000942111189942807,
+            -0.0004040343628730625,
+            0.003602798795327544,
+            0.017584621906280518,
+            0.005553497467190027,
+            -0.054541099816560745,
+            0.006582245696336031,
+            0.027898261323571205,
+            0.020418038591742516,
+            0.005235282704234123,
+            -0.026398729532957077,
+            0.014629150740802288,
+            0.015823544934391975,
+            0.002135960618034005,
+            0.007035592570900917,
+            0.016538437455892563,
+            -0.011080840602517128,
+            0.00042283302173018456,
+            0.0030339357908815145,
+            0.03720930591225624,
+            0.01475120522081852,
+            -0.013190646655857563,
+            -0.025056125596165657,
+            -0.016364073380827904,
+            0.0066519915126264095,
+            -0.005915303248912096,
+            -0.0004182014672551304,
+            -0.037488289177417755,
+            -0.03598875552415848,
+            0.013521937653422356,
+            -0.01818617805838585,
+            -0.03815086930990219,
+            -0.0042741005308926105,
+            -0.03222249075770378,
+            0.005200410261750221,
+            -0.014890696853399277,
+            -0.019319545477628708,
+            -0.03926680237054825,
+            -0.0038970382884144783,
+            0.00012893410166725516,
+            0.04045247659087181,
+            0.00037842465098947287,
+            0.03476820886135101,
+            -0.007253547664731741,
+            -0.0013676686212420464,
+            -0.004143327474594116,
+            0.022876573726534843,
+            -0.03147272393107414,
+            -0.04529980197548866,
+            0.026573093608021736,
+            0.005832480266690254,
+            0.016024062409996986,
+            0.035744644701480865,
+            -0.013870666734874249,
+            -0.010200301185250282,
+            0.024864325299859047,
+            -0.016747673973441124,
+            -0.010984940454363823,
+            -0.0022231426555663347,
+            0.0025980256032198668,
+            -0.012231643311679363,
+            -0.013521937653422356,
+            -0.011656241491436958,
+            0.05157690867781639,
+            0.005575293209403753,
+            0.027845950797200203,
+            0.04038273170590401,
+            0.0010271136416122317,
+            -0.04059197008609772,
+            0.0032889433205127716,
+            -0.0091018071398139,
+            -0.009642336517572403,
+            -0.004625008441507816,
+            0.033024564385414124,
+            0.001587803359143436,
+            0.017872322350740433,
+            -0.0042174323461949825,
+            -0.016416382044553757,
+            -0.020418038591742516,
+            -0.012318825349211693,
+            -0.00761535344645381,
+            -0.004145507235080004,
+            -0.03373945876955986,
+            -0.01256293524056673,
+            0.031716834753751755,
+            0.025335106998682022,
+            -0.022388353943824768,
+            -0.004925786517560482,
+            -0.01782001368701458,
+            -0.007672021631151438,
+            0.010479284450411797,
+            -0.002959831152111292,
+            -0.00416948227211833,
+            0.009250016883015633,
+            0.018133869394659996,
+            0.01007824670523405,
+            0.00766766257584095,
+            -0.05715656280517578,
+            -0.0019398009171709418,
+            0.011351104825735092,
+            0.010400820523500443,
+            -0.003968963399529457,
+            0.014541968703269958,
+            -0.012580371461808681,
+            -0.01265011727809906,
+            -0.01975545659661293,
+            0.0326235294342041,
+            0.006355572491884232,
+            0.030426539480686188,
+            -0.0005541510181501508,
+            0.009075652807950974,
+            0.02149909734725952,
+            0.004101916216313839,
+            0.012807045131921768,
+            0.00199102028273046,
+            0.019807765260338783,
+            0.00986029114574194,
+            -0.03142041712999344,
+            0.012414725497364998,
+            0.0010429153917357326,
+            -0.0024519956205040216,
+            -0.017872322350740433,
+            0.006922255735844374,
+            0.0023757112212479115,
+            -0.013949130661785603,
+            0.0041215321980416775,
+            -0.022370917722582817,
+            0.005954534746706486,
+            0.031089123338460922,
+            0.033233802765607834,
+            0.009206426329910755,
+            -0.015169679187238216,
+            -0.010810576379299164,
+            0.002057496691122651,
+            0.02008674666285515,
+            0.025404853746294975,
+            0.025003815069794655,
+            -0.0268172025680542,
+            -0.014393758960068226,
+            -0.003729212796315551,
+            -0.001253242138773203,
+            -0.010932630859315395,
+            0.03046141378581524,
+            0.02833417057991028,
+            -0.0008914365898817778,
+            0.0028835467528551817,
+            -0.0051916916854679585,
+            0.01523070689290762,
+            0.01898825354874134,
+            -0.017340512946248055,
+            0.0017665266059339046,
+            0.022283734753727913,
+            -0.020191365852952003,
+            0.022946318611502647,
+            -0.014672741293907166,
+            -0.011429568752646446,
+            0.02013905718922615,
+            0.09366840869188309,
+            -0.005248360335826874,
+            0.01120289508253336,
+            -0.028700334951281548,
+            -0.019214926287531853,
+            -0.008474096655845642,
+            -0.0035330532118678093,
+            0.016765110194683075,
+            0.026503346860408783,
+            0.009511562995612621,
+            0.021359605714678764,
+            0.015466097742319107,
+            -0.01666049286723137,
+            0.014411195181310177,
+            -0.009694645181298256,
+            0.016198428347706795,
+            -0.01362655684351921,
+            -0.004267562180757523,
+            0.010069528594613075,
+            -0.001332795829512179,
+            0.009816700592637062,
+            0.018674397841095924,
+            0.016799984499812126,
+            0.013190646655857563,
+            -0.024864325299859047,
+            0.00949412677437067,
+            0.002284170128405094,
+            -0.007763562723994255,
+            -0.02013905718922615,
+            -0.02322530187666416,
+            0.004999891389161348,
+            0.004428848624229431,
+            0.005296310409903526,
+            0.010331074707210064,
+            -0.0032889433205127716,
+            0.03811599686741829,
+            -0.03319893032312393,
+            0.01496044173836708,
+            0.006917896680533886,
+            -0.013103464618325233,
+            0.011935224756598473,
+            0.016198428347706795,
+            0.004049607086926699,
+            0.012249079532921314,
+            -0.011569059453904629,
+            0.014245549216866493,
+            0.02052265778183937,
+            -0.029781393706798553,
+            -0.00266123260371387,
+            -0.016372792422771454,
+            -0.008474096655845642,
+            0.0026677711866796017,
+            -0.016172273084521294,
+            -0.006006844341754913,
+            -0.00546195637434721,
+            0.01169983297586441,
+            0.0024825092405080795,
+            -0.0325886569917202,
+            0.005060918629169464,
+            0.00261110276915133,
+            0.005928380414843559,
+            0.018168741837143898,
+            0.01632048189640045,
+            0.01765436865389347,
+            -0.011525468900799751,
+            -0.01975545659661293,
+            0.006285826675593853,
+            0.0045029534958302975,
+            -0.01073211245238781,
+            0.0023473771288990974,
+            0.0024912275839596987,
+            0.01588457264006138,
+            -0.011185458861291409,
+            -0.005823761690407991,
+            -0.004141148179769516,
+            0.011664960533380508,
+            0.0031516316812485456,
+            -0.014541968703269958,
+            0.003940629307180643,
+            -0.021150369197130203,
+            0.020644713193178177,
+            0.011577778495848179,
+            -0.0019932000432163477,
+            -0.02088882215321064,
+            -0.02873520739376545,
+            0.01332141924649477,
+            -0.020836513489484787,
+            0.0031952227000147104,
+            0.015771236270666122,
+            -0.011673678644001484,
+            0.013260391540825367,
+            -0.018255924805998802,
+            -0.018866198137402534,
+            -0.0114818774163723,
+            -0.00473834527656436,
+            -0.005226564593613148,
+            -0.007576121482998133,
+            0.007502016611397266,
+            -0.01823848858475685,
+            -0.0306706503033638,
+            -0.033634841442108154,
+            0.019023125991225243,
+            0.01670408435165882,
+            -0.009467972442507744,
+            -0.015823544934391975,
+            -0.004642445128411055,
+            0.02319042943418026,
+            -0.0034415118861943483,
+            0.0007110786973498762,
+            -0.0026830281130969524,
+            0.013391165062785149,
+            -0.007610994391143322,
+            0.00016441992192994803,
+            -0.02284170128405094,
+            0.00044217653339728713,
+            0.0011725988006219268,
+            0.013391165062785149,
+            -0.027357731014490128,
+            -0.003380484413355589,
+            0.0003002332232426852,
+            -0.018098996952176094,
+            -0.004476799163967371,
+            0.04254484549164772,
+            -0.013417319394648075,
+            0.008979752659797668,
+            0.021551406010985374,
+            0.010104401037096977,
+            -0.00977310910820961,
+            0.010095682926476002,
+            0.007850744761526585,
+            -0.0028857262805104256,
+            0.026538221165537834,
+            0.022911446169018745,
+            0.01914518140256405,
+            0.01670408435165882,
+            -0.022161681205034256,
+            0.012528062798082829,
+            0.021969880908727646,
+            0.03752316161990166,
+            -0.017331793904304504,
+            -0.021952442824840546,
+            0.019860073924064636,
+            0.04676445946097374,
+            0.0021032672375440598,
+            -0.0006200824282132089,
+            -0.028648026287555695,
+            -0.017201021313667297,
+            0.024689961224794388,
+            0.020557530224323273,
+            0.016224581748247147,
+            -0.010121837258338928,
+            -0.031507596373558044,
+            0.014280421659350395,
+            -0.00309278373606503,
+            0.006865587551146746,
+            -0.01996469311416149,
+            0.01688716560602188,
+            -0.015004033222794533,
+            0.04257972165942192,
+            -0.012658835388720036,
+            0.004829886369407177,
+            0.017148712649941444,
+            0.011150586418807507,
+            -0.022318609058856964,
+            0.0009263094398193061,
+            -0.01666049286723137,
+            0.027933133766055107,
+            -0.018883634358644485,
+            0.003618055721744895,
+            0.01914518140256405,
+            -0.016608182340860367,
+            -0.014533249661326408,
+            0.004191277548670769,
+            0.019459037110209465,
+            0.006107103545218706,
+            -0.025247925892472267,
+            0.018098996952176094,
+            0.0019190951716154814,
+            0.009171552956104279,
+            -0.009930036962032318,
+            0.023870449513196945,
+            -0.027636714279651642,
+            -0.015387633815407753,
+            0.025003815069794655,
+            0.0041607641614973545,
+            0.013565529137849808,
+            -0.026311546564102173,
+            0.009380790404975414,
+            -0.029049063101410866,
+            0.02336479350924492,
+            0.0032584297005087137,
+            0.014280421659350395,
+            0.0015147883677855134,
+            0.002866110298782587,
+            -0.0015115190763026476,
+            0.014446067623794079,
+            -0.000550881668459624,
+            -0.022527845576405525,
+            -0.01569277234375477,
+            -0.03696519508957863,
+            -0.022737082093954086,
+            0.023818138986825943,
+            -0.002715721260756254,
+            0.003528693923726678,
+            0.02430635876953602,
+            -0.027462350204586983,
+            -0.01435888558626175,
+            0.009572590701282024,
+            0.02549203485250473,
+            -0.043625906109809875,
+            4.093470488442108e-05,
+            0.0013829254312440753,
+            -0.019232362508773804,
+            0.010627493262290955,
+            -0.005440160632133484,
+            0.015500971116125584,
+            0.0018776836805045605,
+            -0.037662651389837265,
+            0.011586496606469154,
+            0.03762777894735336,
+            -0.017924632877111435,
+            0.0028225192800164223,
+            0.017706677317619324,
+            0.012493189424276352,
+            -0.0012390750925987959,
+            -0.0017033196054399014,
+            0.01877901703119278,
+            -0.025945382192730904,
+            0.026102310046553612,
+            0.017026657238602638,
+            -0.014881978742778301,
+            0.007842026650905609,
+            -0.008753078989684582,
+            0.020731894299387932,
+            -0.0013208083109930158,
+            -0.013155773282051086,
+            0.017375385388731956,
+            0.002195898210629821,
+            0.01349578332155943,
+            -0.001912556472234428,
+            0.004304614383727312,
+            0.03107168711721897,
+            -0.01609380915760994,
+            0.02280682697892189,
+            0.01994725689291954,
+            3.6405323044164106e-05,
+            0.015405070967972279,
+            -0.019877510145306587,
+            -0.03312918543815613,
+            0.02221398986876011,
+            -0.013591683469712734,
+            -0.007310215849429369,
+            0.023347357288002968,
+            -0.010453129187226295,
+            -0.006250953767448664,
+            -0.004282819107174873,
+            0.004077941179275513,
+            0.012406007386744022,
+            -0.0230160653591156,
+            -0.015631744638085365,
+            0.022318609058856964,
+            0.017131276428699493,
+            0.022179117426276207,
+            0.023800702765583992,
+            -0.015378915704786777,
+            0.03182145208120346,
+            0.009223862551152706,
+            0.0065735275857150555,
+            0.0076371487230062485,
+            0.008395632728934288,
+            -0.005871712230145931,
+            0.01933698169887066,
+            0.014114775694906712,
+            0.007336370646953583,
+            -0.003565746359527111,
+            -0.02029598504304886,
+            -0.011447004973888397,
+            -0.027532095089554787,
+            -0.00045825072447769344,
+            -0.013914257287979126,
+            0.00704866973683238,
+            0.07797563821077347,
+            -0.006682505365461111,
+            0.0018144766800105572,
+            -0.007868180982768536,
+            0.000739412906114012,
+            0.0016771649243310094,
+            -0.005209128372371197,
+            0.016538437455892563,
+            -0.020662149414420128,
+            -0.020749330520629883,
+            -0.006163771729916334,
+            -0.010967503301799297,
+            0.013460910879075527,
+            0.01741897687315941,
+            0.0030034221708774567,
+            -0.02277195453643799,
+            0.011141867376863956,
+            -0.016355354338884354,
+            -0.011106994934380054,
+            0.010348510928452015,
+            -0.007231752388179302,
+            -0.011656241491436958,
+            0.03295481950044632,
+            -0.009206426329910755,
+            -0.03147272393107414,
+            0.015640461817383766,
+            -0.0024541751481592655,
+            -0.005976330488920212,
+            0.00470347236841917,
+            0.00853076484054327,
+            -0.004986814223229885,
+            0.0005544234300032258,
+            0.013504501432180405,
+            -0.002068394562229514,
+            0.005414006300270557,
+            0.005318105686455965,
+            -0.013565529137849808,
+            0.017296921461820602,
+            0.022527845576405525,
+            -0.033477913588285446,
+            -0.004755781497806311,
+            -0.013940411619842052,
+            0.010479284450411797,
+            0.018709270283579826,
+            0.015535843558609486,
+            -0.022580154240131378,
+            0.0025871277321130037,
+            -0.009154116734862328,
+            -0.012702426873147488,
+            -0.02741004154086113,
+            -0.018255924805998802,
+            0.00723611144348979,
+            -0.00851768720895052,
+            0.018656961619853973,
+            0.007907412946224213,
+            -0.005745298229157925,
+            0.005012968555092812,
+            -0.005178614519536495,
+            0.015518407337367535,
+            -0.0326235294342041,
+            0.010461847297847271,
+            -0.009337198920547962,
+            0.009581308811903,
+            0.01055774837732315,
+            -0.022388353943824768,
+            -0.014759923331439495,
+            -0.023417102172970772,
+            0.016555873677134514,
+            -0.010086964815855026,
+            -0.0020934592466801405,
+            0.02008674666285515,
+            0.0009088730439543724,
+            0.015413789078593254,
+            -0.001491903094574809,
+            0.023242738097906113,
+            0.014140930958092213,
+            0.016590746119618416,
+            -0.01215317938476801,
+            -0.001640112604945898,
+            -0.011124431155622005,
+            0.012719863094389439,
+            -0.02303350158035755,
+            -0.006813278421759605,
+            -0.026015127077698708,
+            0.03598875552415848,
+            -0.007602275814861059,
+            -0.01588457264006138,
+            0.02857828140258789,
+            -0.012414725497364998,
+            -0.0037052377592772245,
+            0.0058847893960773945,
+            -0.015431225299835205,
+            0.023451974615454674,
+            0.0004955755430273712,
+            0.011499314568936825,
+            -0.02549203485250473,
+            0.00863102450966835,
+            -0.007087901700288057,
+            0.006076589692384005,
+            -0.02128986082971096,
+            -0.002423661295324564,
+            0.012135743163526058,
+            -0.010426974855363369,
+            0.00411717314273119,
+            0.021185241639614105,
+            0.005331183318048716,
+            -0.01744513027369976,
+            0.03951091319322586,
+            -0.004025631584227085,
+            0.010653648525476456,
+            0.005475033540278673,
+            -0.0009028792264871299,
+            0.016799984499812126,
+            0.01742769405245781,
+            -0.004925786517560482,
+            -0.016547156497836113,
+            0.0032235567923635244,
+            -0.015021469444036484,
+            0.01561430748552084,
+            0.016233300790190697,
+            -0.02125498652458191,
+            -0.032658401876688004,
+            0.01779385842382908,
+            0.002061855746433139,
+            0.028072625398635864,
+            0.006983283441513777,
+            -0.025997690856456757,
+            -0.005296310409903526,
+            -0.03815086930990219,
+            0.012885509058833122,
+            -0.01801181398332119,
+            -0.02437610551714897,
+            0.002785466844215989,
+            -0.00986029114574194,
+            -0.01856978051364422,
+            0.02512587048113346,
+            0.013652711175382137,
+            -0.02552690915763378,
+            0.010889039374887943,
+            0.0013011923292651772,
+            -0.017549749463796616,
+            0.010052092373371124,
+            0.004350385162979364,
+            -0.012493189424276352,
+            0.007410475518554449,
+            0.005130664445459843,
+            -0.006978924386203289,
+            -0.02890957146883011,
+            -0.01670408435165882,
+            0.01972058229148388,
+            0.010993658564984798,
+            -0.0058455574326217175,
+            0.011290077120065689,
+            -0.00997362844645977,
+            0.0033456117380410433,
+            0.0035221553407609463,
+            0.00901462510228157,
+            -0.013051155023276806,
+            -0.007371243555098772,
+            -0.007162006571888924,
+            0.01973801851272583,
+            -0.016024062409996986,
+            0.007990236394107342,
+            -0.013443474657833576,
+            -0.007981518283486366,
+            0.011429568752646446,
+            -0.0013774766121059656,
+            0.014986597001552582,
+            -0.018744144588708878,
+            0.0073407297022640705,
+            0.01189163327217102,
+            -0.008883852511644363,
+            -0.010671084746718407,
+            -0.02437610551714897,
+            0.019110308960080147,
+            0.01438504084944725,
+            0.004694754257798195,
+            -0.03679082915186882,
+            0.003951527178287506,
+            -0.010453129187226295,
+            -0.0009775288635864854,
+            -0.008796670474112034,
+            0.019249800592660904,
+            0.018831325694918633,
+            0.007985876873135567,
+            0.013277828693389893,
+            -0.0023081451654434204,
+            0.013548092916607857,
+            -0.012414725497364998,
+            0.017593340948224068,
+            -0.004036529455333948,
+            -0.017575904726982117,
+            -0.005714784376323223,
+            0.019406726583838463,
+            -0.001491903094574809,
+            0.007903054356575012,
+            0.004372180439531803,
+            0.010226456448435783,
+            -0.012275234796106815,
+            -0.007493298500776291,
+            0.014045030809938908,
+            0.008378196507692337,
+            -0.0004288267809897661,
+            0.009250016883015633,
+            0.06758353859186172,
+            -0.00870512891560793,
+            0.023853013291954994,
+            0.006311981473118067,
+            -0.009345917031168938,
+            -0.008112290874123573,
+            0.0019932000432163477,
+            -0.02951984666287899,
+            0.023329921066761017,
+            0.011970097199082375,
+            -0.009180271066725254,
+            0.00036126066697761416,
+            -0.006726096384227276,
+            0.018325669690966606,
+            -0.009694645181298256,
+            0.016948193311691284,
+            -0.010052092373371124,
+            0.02435866929590702,
+            0.019162617623806,
+            0.005640679504722357,
+            0.005536061245948076,
+            0.0028203397523611784,
+            0.019459037110209465,
+            0.012868072837591171,
+            -0.002165384590625763,
+            -0.003123297356069088,
+            -0.009877728298306465,
+            0.019389290362596512,
+            -0.001910376944579184,
+            0.030792705714702606,
+            -0.0022144243121147156,
+            -0.005182973574846983,
+            -0.009842854924499989,
+            0.0041607641614973545,
+            0.011569059453904629,
+            0.009520281106233597,
+            0.012876790948212147,
+            -0.0013567708665505052,
+            -0.010679802857339382,
+            0.011909069493412971,
+            0.00863102450966835,
+            0.005326824262738228,
+            -0.0032518908847123384,
+            0.0045247492380440235,
+            -0.008404350839555264,
+            0.014001439325511456,
+            -0.0024781501851975918,
+            0.0015354941133409739,
+            -0.008774874731898308,
+            0.004426669329404831,
+            -0.01312961895018816,
+            0.0005348074482753873,
+            0.009607463143765926,
+            0.012868072837591171,
+            0.0015213270671665668,
+            -0.002273272257298231,
+            0.015405070967972279,
+            0.0025980256032198668,
+            0.0073014977388083935,
+            0.0009486498311161995,
+            -0.009537717327475548,
+            0.019057998433709145,
+            -0.01033979281783104,
+            -0.0019212746992707253,
+            0.010705957189202309,
+            0.006067871581763029,
+            0.008260500617325306,
+            0.01496916078031063,
+            0.009354635141789913,
+            0.003968963399529457,
+            0.0013545913388952613,
+            -0.00565375667065382,
+            -0.003535232739523053,
+            0.0036267738323658705,
+            0.0016771649243310094,
+            0.01646869257092476,
+            -0.019459037110209465,
+            -0.007039951626211405,
+            0.010679802857339382,
+            0.011438286863267422,
+            0.0017839629435911775,
+            0.015684053301811218,
+            0.014428631402552128,
+            0.01994725689291954,
+            -0.022562718018889427,
+            0.003831651760265231,
+            -0.002920599188655615,
+            0.019057998433709145,
+            0.008770515210926533,
+            -0.024114558473229408,
+            0.0003645300166681409,
+            -0.01063621137291193,
+            0.012676272541284561,
+            -0.0067086596973240376,
+            0.0023473771288990974,
+            0.01665177382528782,
+            -0.010121837258338928,
+            -0.009930036962032318,
+            -0.019424164667725563,
+            -0.016983065754175186,
+            0.010976221412420273,
+            0.014437349513173103,
+            0.0014188881032168865,
+            0.019232362508773804,
+            -0.0012412546202540398,
+            -0.008474096655845642,
+            0.030827578157186508,
+            0.016765110194683075,
+            -0.0004157494695391506,
+            -0.008962316438555717,
+            -0.0006544103380292654,
+            -0.015082497149705887,
+            0.01074954867362976,
+            0.004426669329404831,
+            0.005108869168907404,
+            0.003387023229151964,
+            9.174550359603018e-05,
+            -0.016712801530957222,
+            -0.014620431698858738,
+            -0.004668599460273981,
+            0.011045967228710651,
+            -0.003751008305698633,
+            0.009197707287967205,
+            -0.01609380915760994,
+            0.003121117828413844,
+            -0.008173318579792976,
+            -0.010714675299823284,
+            -0.008356400765478611,
+            -0.039406292140483856,
+            -0.02067958563566208,
+            0.0003173972072545439,
+            -0.00473834527656436,
+            -0.002358274767175317,
+            0.009223862551152706,
+            -0.003672544378787279,
+            0.013670147396624088,
+            -0.0015551100950688124,
+            0.0047906544059515,
+            0.008657178841531277,
+            0.01561430748552084,
+            -0.013504501432180405,
+            -0.0006631285650655627,
+            -0.014071185141801834,
+            -0.026782330125570297,
+            0.01707896590232849,
+            0.003199581755325198,
+            0.014446067623794079,
+            -0.015187115408480167,
+            0.009528999216854572,
+            -0.012589089572429657,
+            0.014463504776358604,
+            -0.0421612448990345,
+            -0.009520281106233597,
+            0.007031233515590429,
+            -0.0075979167595505714,
+            -0.005496829282492399,
+            -0.025369981303811073,
+            -0.0012205488746985793,
+            -0.011054685339331627,
+            -0.01666049286723137,
+            -0.019459037110209465,
+            0.00574093870818615,
+            -0.009311044588685036,
+            0.008591792546212673,
+            -0.006120180711150169,
+            0.02761927805840969,
+            0.00532246520742774,
+            -0.001346962875686586,
+            -0.006128899287432432,
+            0.009206426329910755,
+            -0.0007503106608055532,
+            0.012885509058833122,
+            0.008648460730910301,
+            0.005823761690407991,
+            0.023434538394212723,
+            0.0009339378448203206,
+            0.004690395202487707,
+            -0.006141976453363895,
+            0.014541968703269958,
+            0.024689961224794388,
+            0.011342386715114117,
+            0.011455723084509373,
+            0.005967612378299236,
+            0.0005950175691395998,
+            0.011429568752646446,
+            0.017392821609973907,
+            -0.003990759141743183,
+            -0.009171552956104279,
+            0.004869118332862854,
+            -0.01761949434876442,
+            -0.0033347138669341803,
+            0.012885509058833122,
+            -0.006128899287432432,
+            -0.02706131339073181,
+            0.01285063661634922,
+            0.004317691549658775,
+            -0.006673786789178848,
+            -0.016948193311691284,
+            -0.00661711860448122,
+            0.004646804183721542,
+            0.004729627165943384,
+            0.024236613884568214,
+            -0.022144244983792305,
+            -0.008644101209938526,
+            0.011917788535356522,
+            -0.014446067623794079,
+            -0.01063621137291193,
+            0.009790545329451561,
+            0.029484974220395088,
+            0.003496000776067376,
+            -0.05307644233107567,
+            -0.012467035092413425,
+            0.014236831106245518,
+            -0.01503018755465746,
+            -0.025980254635214806,
+            -0.0014428631402552128,
+            0.0038578063249588013,
+            -0.014010157436132431,
+            0.006625836715102196,
+            0.0011606112821027637,
+            -0.011708551086485386,
+            0.017139993607997894,
+            0.028874699026346207,
+            0.01312961895018816,
+            0.0114818774163723,
+            -0.025213053449988365,
+            -0.0008952508214861155,
+            -0.01914518140256405,
+            0.011752142570912838,
+            -0.030566031113266945,
+            -0.013984003104269505,
+            -0.0023342997301369905,
+            0.004141148179769516,
+            0.027444913983345032,
+            0.014376322738826275,
+            0.025962818413972855,
+            0.006778405513614416,
+            0.0037989583797752857,
+            -0.02125498652458191,
+            -0.018447725102305412,
+            0.0033456117380410433,
+            -0.0008304091752506793,
+            -0.002329940674826503,
+            -0.008216909132897854,
+            0.012275234796106815,
+            0.012039843015372753,
+            0.026172054931521416,
+            0.005087073426693678,
+            0.005553497467190027,
+            -2.538836997700855e-05,
+            -0.00494322320446372,
+            0.007593557704240084,
+            -0.019232362508773804,
+            -0.0029925242997705936,
+            -0.004559622146189213,
+            0.015405070967972279,
+            0.0045029534958302975,
+            0.0022275017108768225,
+            0.017357949167490005,
+            -0.0030099607538431883,
+            0.003729212796315551,
+            -0.019982129335403442,
+            -0.014672741293907166,
+            -0.0032518908847123384,
+            0.03525642678141594,
+            0.003971143160015345,
+            -0.02761927805840969,
+            -0.016224581748247147,
+            -0.0153353251516819,
+            0.030792705714702606,
+            -0.009581308811903,
+            0.004315512254834175,
+            0.02683463878929615,
+            0.01593688130378723,
+            0.012388571165502071,
+            0.006399163510650396,
+            -0.01822105050086975,
+            0.005021686665713787,
+            0.02931061014533043,
+            -0.0030317562632262707,
+            -0.010086964815855026,
+            -0.0066912234760820866,
+            0.007144570350646973,
+            -0.007798435632139444,
+            0.007632789667695761,
+            -0.0035068984143435955,
+            -0.013417319394648075,
+            0.029554719105362892,
+            -0.00968592707067728,
+            0.022370917722582817,
+            -0.010130555368959904,
+            0.00268738716840744,
+            -0.038848329335451126,
+            -0.002004097681492567,
+            0.0011453543556854129,
+            0.0019692247733473778,
+            0.004555262625217438,
+            -0.01881388947367668,
+            -0.008753078989684582,
+            0.011839324608445168,
+            -0.01276345457881689,
+            -0.007885618135333061,
+            -0.006137617398053408,
+            0.002983806189149618,
+            0.01977289281785488,
+            -0.00947669055312872,
+            0.0009857022669166327,
+            -0.0210980586707592,
+            0.01256293524056673,
+            -0.00282687833532691,
+            0.0017567186150699854,
+            0.010453129187226295,
+            0.007009437773376703,
+            0.03502975404262543,
+            0.01094134896993637,
+            0.005531701724976301,
+            -0.0032780456822365522,
+            -0.014053748920559883,
+            -0.03218761831521988,
+            0.013295264914631844,
+            -0.010453129187226295,
+            0.023800702765583992,
+            0.0036703648511320353,
+            -0.008103572763502598,
+            -0.0051960512064397335,
+            0.0037139558698982,
+            0.009590026922523975,
+            -0.0041106343269348145,
+            0.013478347100317478,
+            -0.010243892669677734,
+            0.021167805418372154,
+            0.0031298361718654633,
+            -0.005121946334838867,
+            0.012571653351187706,
+            -0.017689241096377373,
+            0.01055774837732315,
+            -0.01744513027369976,
+            0.0028225192800164223,
+            0.01360040158033371,
+            -0.010296201333403587,
+            0.0056276023387908936,
+            -0.052588220685720444,
+            -0.001197663601487875,
+            -0.016521001234650612,
+            -0.00675660977140069,
+            -0.008456660434603691,
+            -0.021394478157162666,
+            -0.03319893032312393,
+            -0.0036660057958215475,
+            0.015370197594165802,
+            0.006551732309162617,
+            0.023260174319148064,
+            0.0009895163821056485,
+            0.012257798574864864,
+            0.012807045131921768,
+            -0.0047906544059515,
+            0.012519344687461853,
+            0.033861514180898666,
+            -0.027741333469748497,
+            -0.019685709848999977,
+            0.005257078446447849,
+            -0.00028552126605063677,
+            -0.015788672491908073,
+            -0.013103464618325233,
+            -0.0017665266059339046,
+            0.0024955866392701864,
+            0.030443977564573288,
+            -0.020191365852952003,
+            -0.005579652264714241,
+            -0.007053028792142868,
+            0.004433208145201206,
+            -0.01918005384504795,
+            -0.0031450928654521704,
+            -0.011708551086485386,
+            -0.012780890800058842,
+            0.016041500493884087,
+            -0.011089558713138103,
+            0.011159304529428482,
+            -0.019511345773935318,
+            0.008626664988696575,
+            -0.015378915704786777,
+            -0.010627493262290955,
+            -0.0031690681353211403,
+            -0.010461847297847271,
+            0.030164994299411774,
+            -0.0041890982538461685,
+            0.0048386044800281525,
+            0.01670408435165882,
+            -0.00853076484054327,
+            0.011603932827711105,
+            -0.0006124540232121944,
+            0.007898694835603237,
+            -0.002704823622480035,
+            -0.0086135882884264,
+            0.0015278656501322985,
+            0.009537717327475548,
+            -0.010950067080557346,
+            0.03846472501754761,
+            -0.013731175102293491,
+            -0.008879492990672588,
+            -0.003859985852614045,
+            -0.004511671606451273,
+            0.002425840822979808,
+            0.0021762822289019823,
+            0.010984940454363823,
+            -0.0012870251666754484,
+            0.006835073698312044,
+            -0.024219177663326263,
+            -0.011717269197106361,
+            0.0013164491392672062,
+            0.006468909326940775,
+            0.003321636701002717,
+            0.008016390725970268,
+            -0.025579217821359634,
+            -0.0020782025530934334,
+            -0.012859354726970196,
+            0.005475033540278673,
+            -0.011499314568936825,
+            0.007332011591643095,
+            -0.01215317938476801,
+            -0.0013295264216139913,
+            0.009337198920547962,
+            0.012946536764502525,
+            0.013861947692930698,
+            -0.006608400493860245,
+            0.026573093608021736,
+            0.014899414964020252,
+            0.008709488436579704,
+            0.005405288189649582,
+            -0.033216364681720734,
+            -0.007802794687449932,
+            0.010365947149693966,
+            0.007519452832639217,
+            0.009232580661773682,
+            -0.017689241096377373,
+            -0.004176021087914705,
+            0.0029402151703834534,
+            -0.007039951626211405,
+            4.651980270864442e-05,
+            0.011080840602517128,
+            -0.010627493262290955,
+            0.005505547393113375,
+            -0.00031521765049546957,
+            0.0027549532242119312,
+            -0.006460190750658512,
+            0.012188052758574486,
+            -0.013286546804010868,
+            -0.008168959058821201,
+            0.0211154967546463,
+            0.02760184183716774,
+            -0.005531701724976301,
+            0.023696085438132286,
+            -0.021917570382356644,
+            0.008487174287438393,
+            -0.005043482407927513,
+            -0.014149649068713188,
+            -0.015300451777875423,
+            0.006883023772388697,
+            0.014777359552681446,
+            0.004760140553116798,
+            -0.013155773282051086,
+            -0.003027397207915783,
+            0.016983065754175186,
+            -0.0010952246375381947,
+            0.006216081324964762,
+            0.01593688130378723,
+            -0.00026971951592713594,
+            0.00017667989595793188,
+            -0.006002485286444426,
+            -0.01876158080995083,
+            -0.006290185730904341,
+            0.017924632877111435,
+            -0.0073799616657197475,
+            0.014262985438108444,
+            0.02320786565542221,
+            0.0005429807933978736,
+            0.011351104825735092,
+            -0.0038469084538519382,
+            0.01351321954280138,
+            0.01505634281784296,
+            -0.005235282704234123,
+            -0.0030600903555750847,
+            -0.0036965194158256054,
+            0.00011858122888952494,
+            0.012022406794130802,
+            -0.01741897687315941,
+            -0.011333668604493141,
+            0.0114818774163723,
+            -0.002121793571859598,
+            -0.008404350839555264,
+            -0.023486848920583725,
+            0.0016313943779096007,
+            0.0077156126499176025,
+            0.011447004973888397,
+            -0.016425101086497307,
+            -0.0005239096935838461,
+            0.000875089957844466,
+            0.010801857337355614,
+            -0.007497657556086779,
+            0.01159521471709013,
+            -0.007907412946224213,
+            -0.00863102450966835,
+            0.006381727289408445,
+            0.0015507509233430028,
+            0.007414834573864937,
+            -0.009999782778322697,
+            0.0015987011138349771,
+            -0.008025108836591244,
+            0.026590529829263687,
+            0.008317168802022934,
+            -0.0013011923292651772,
+            -0.005640679504722357,
+            0.004995532333850861,
+            0.008421787060797215,
+            -0.0100608104839921,
+            0.012815763242542744,
+            0.006107103545218706,
+            -0.002933676354587078,
+            0.010592620819807053,
+            0.015283015556633472,
+            0.0025631526950746775,
+            0.009328480809926987,
+            0.003443691413849592,
+            0.0059022256173193455,
+            -0.007022515404969454,
+            0.0017861425876617432,
+            -0.01550968922674656,
+            -0.020801641047000885,
+            -0.0035831828135997057,
+            0.016991784796118736,
+            -0.010583902709186077,
+            -0.001182406791485846,
+            0.017924632877111435,
+            0.01371373888105154,
+            -0.005034764297306538,
+            0.00551426550373435,
+            0.002217693720012903,
+            -0.0015529305674135685,
+            0.003323816228657961,
+            -0.014280421659350395,
+            -0.012493189424276352,
+            0.008726924657821655,
+            -0.02087138593196869,
+            0.01952878199517727,
+            -0.02165602520108223,
+            -0.035361044108867645,
+            -0.008831542916595936,
+            -0.009197707287967205,
+            -0.01707896590232849,
+            -0.013574247248470783,
+            0.023661212995648384,
+            0.016207145527005196,
+            -0.009354635141789913,
+            -0.034384604543447495,
+            0.03155990689992905,
+            0.00918898917734623,
+            0.027933133766055107,
+            -0.015544561669230461,
+            0.004343846347182989,
+            -0.0015213270671665668,
+            0.014707613736391068,
+            -0.006163771729916334,
+            -0.006686864420771599,
+            0.0091018071398139,
+            0.014620431698858738,
+            0.010897758416831493,
+            0.003245352301746607,
+            -0.003160349791869521,
+            0.007279702462255955,
+            0.002530459314584732,
+            0.0022285915911197662,
+            -0.008583074435591698,
+            -0.02282426506280899,
+            -0.009781827218830585,
+            0.017942069098353386,
+            0.022266298532485962,
+            0.012746017426252365,
+            -0.017131276428699493,
+            0.0020520477555692196,
+            -0.00484296353533864,
+            -0.014053748920559883,
+            -0.009991064667701721,
+            0.009642336517572403,
+            -0.033617403358221054,
+            0.02317299321293831,
+            0.004760140553116798,
+            -0.00627274950966239,
+            -0.0015006213216111064,
+            0.003836010815575719,
+            0.005679911468178034,
+            -0.004313332494348288,
+            0.0002961465797852725,
+            -0.002532638842239976,
+            0.010897758416831493,
+            0.016599465161561966,
+            0.005100150592625141,
+            -0.010400820523500443,
+            0.012388571165502071,
+            -0.0031407338101416826,
+            -0.002761491807177663,
+            -0.013914257287979126,
+            -0.006011203397065401,
+            -0.009502844884991646,
+            0.010261328890919685,
+            0.01483838725835085,
+            0.0054924702271819115,
+            0.006791482679545879,
+            0.004335128236562014,
+            -0.00013084120291750878,
+            -0.0002569146454334259,
+            0.008469737134873867,
+            -0.0005070181796327233,
+            -0.0045639812014997005,
+            0.0010821473551914096,
+            -0.0014679280575364828,
+            0.02320786565542221,
+            -0.00532246520742774,
+            -0.010950067080557346,
+            0.007196879480034113,
+            0.02033085748553276,
+            -0.022370917722582817,
+            0.022091934457421303,
+            -0.022179117426276207,
+            0.023853013291954994,
+            0.03482051566243172,
+            0.013975284993648529,
+            -0.011717269197106361,
+            -0.007096619810909033,
+            0.012815763242542744,
+            -0.012510626576840878,
+            -0.0009600924677215517,
+            0.0018079380970448256,
+            -0.022580154240131378,
+            0.015178397297859192,
+            -0.023347357288002968,
+            0.014280421659350395,
+            0.005121946334838867,
+            -0.00852640625089407,
+            -0.0073232934810221195,
+            -0.0009764390997588634,
+            0.018133869394659996,
+            -0.003990759141743183,
+            -0.0006255313055589795,
+            -0.03028704971075058,
+            -0.006930973846465349,
+            0.02163858897984028,
+            -0.0045247492380440235,
+            -0.013949130661785603,
+            0.007253547664731741,
+            -8.973622243502177e-06,
+            -0.016154836863279343,
+            -0.005876071285456419,
+            -0.03166452422738075,
+            -0.008142804726958275,
+            -0.02264990098774433,
+            -0.0075979167595505714,
+            -0.011577778495848179,
+            -0.012292671017348766,
+            -0.0038512677419930696,
+            -0.016276890411973,
+            0.001949608908034861,
+            -0.001063076313585043,
+            0.0033020207192748785,
+            0.0027091826777905226,
+            -0.009563872590661049,
+            -0.003968963399529457,
+            0.021324733272194862,
+            0.010705957189202309,
+            -0.016041500493884087,
+            -0.026206929236650467,
+            0.011638805270195007,
+            -0.012205488979816437,
+            -0.007593557704240084,
+            0.001547481631860137,
+            -0.005915303248912096,
+            -0.019267236813902855,
+            -0.01860465295612812,
+            0.005967612378299236,
+            0.0003718859807122499,
+            0.0051175872795283794,
+            0.013582965359091759,
+            0.014341449365019798,
+            -0.0019180054077878594,
+            -0.006734814494848251,
+            -1.330718441749923e-05,
+            -0.002195898210629821,
+            0.014027593657374382,
+            0.004348205402493477,
+            0.024445850402116776,
+            -0.008744360879063606,
+            0.0028944446239620447,
+            -0.017924632877111435,
+            0.00449859444051981,
+            -0.007393038831651211,
+            -0.005679911468178034,
+            -0.004064863547682762,
+            -0.011089558713138103,
+            0.005993766710162163,
+            -0.0011126610916107893,
+            -0.00247161160223186,
+            -0.007911772467195988,
+            -0.0020912797190248966,
+            -0.01707024872303009,
+            -0.0029794471338391304,
+            -0.020557530224323273,
+            -0.0013905538944527507,
+            -0.014916851185262203,
+            0.017000501975417137,
+            0.026172054931521416,
+            -0.00513502350077033,
+            -0.0006091846735216677,
+            -0.011229049414396286,
+            -4.08665910072159e-05,
+            -0.011909069493412971,
+            -0.00637736776843667,
+            -0.016416382044553757,
+            -0.0325189083814621,
+            -0.016904601827263832,
+            0.009049498476088047,
+            0.024288922548294067,
+            0.0019398009171709418,
+            -0.024986378848552704,
+            0.0029511128086596727,
+            -0.026607966050505638,
+            -0.014228112995624542,
+            -0.01496916078031063,
+            -0.006529936566948891,
+            -0.009136680513620377,
+            0.017392821609973907,
+            -0.0026350778061896563,
+            0.01571020856499672,
+            0.01747128553688526,
+            0.010906476527452469,
+            0.007497657556086779,
+            0.010880321264266968,
+            -0.009528999216854572,
+            -0.00016932391736190766,
+            0.009869009256362915,
+            -0.006311981473118067,
+            0.011839324608445168,
+            -0.00815152283757925,
+            -0.0011933045461773872,
+            0.01879645325243473,
+            0.008016390725970268,
+            -0.006255313288420439,
+            -0.0026699507143348455,
+            -0.02742747776210308,
+            0.017506157979369164,
+            -0.018552342429757118,
+            -0.00258494820445776,
+            -0.002944574225693941,
+            -0.016738956794142723,
+            0.02319042943418026,
+            0.017323076725006104,
+            -0.005021686665713787,
+            -0.0027549532242119312,
+            0.017889760434627533,
+            -0.006451472640037537,
+            0.00398204056546092,
+            0.0025762300938367844,
+            -0.0033281752839684486,
+            0.009790545329451561,
+            0.0027636713348329067,
+            -0.004577058367431164,
+            -0.0025587936397641897,
+            0.012196770869195461,
+            0.003901397343724966,
+            0.013530656695365906,
+            -0.027131058275699615,
+            -0.010566466487944126,
+            -0.0010303830495104194,
+            0.01935441792011261,
+            -0.005405288189649582,
+            -0.0007906323298811913,
+            -0.013312701135873795,
+            -0.0008854428306221962,
+            -0.005719143431633711,
+            0.023818138986825943,
+            -0.01992982067167759,
+            0.006120180711150169,
+            -0.008395632728934288,
+            0.013861947692930698,
+            -0.0032191977370530367,
+            0.010801857337355614,
+            -0.0007955363253131509,
+            -1.3375295566220302e-05,
+            0.009964910335838795,
+            0.012981409206986427,
+            -0.0022024367935955524,
+            0.024027377367019653,
+            -0.0004737800336442888,
+            -0.006403522565960884,
+            0.0027745692059397697,
+            -0.01895338110625744,
+            0.0229637548327446,
+            0.003402280155569315,
+            0.004315512254834175,
+            -0.0058629936538636684,
+            -0.008199472911655903,
+            0.039789892733097076,
+            0.021516533568501472,
+            0.010392102412879467,
+            -0.000592293159570545,
+            -0.01494300551712513,
+            0.028229551389813423,
+            0.007519452832639217,
+            0.00930232647806406,
+            -0.004012554418295622,
+            0.0027113622054457664,
+            -0.015178397297859192,
+            -0.010976221412420273,
+            0.019807765260338783,
+            -0.019999565556645393,
+            -0.008731283247470856,
+            -0.0026328982785344124,
+            -0.019703146070241928,
+            -0.029293173924088478,
+            0.015474816784262657,
+            0.009764390997588634,
+            0.0028443147893995047,
+            0.008495892398059368,
+            0.0037335718516260386,
+            -0.0015028008492663503,
+            -0.011647523380815983,
+            -0.003986400086432695,
+            -0.008574356324970722,
+            -0.01044441107660532,
+            0.006848151329904795,
+            -0.012004970572888851,
+            0.006717377807945013,
+            -0.008256141096353531,
+            -0.00024533577379770577,
+            0.01285063661634922,
+            0.005091432482004166,
+            0.0062771085649728775,
+            0.003443691413849592,
+            0.005078355316072702,
+            0.01591944508254528,
+            -0.013757329434156418,
+            0.016346637159585953,
+            0.017113838344812393,
+            0.0062945447862148285,
+            -0.0034131777938455343,
+            0.0023103246930986643,
+            -0.029188554733991623,
+            -0.00022381270537152886,
+            -0.0016869729151949286,
+            0.012632681056857109,
+            -0.02163858897984028,
+            0.004733986221253872,
+            0.021568842232227325,
+            -0.0016172273317351937,
+            0.009816700592637062,
+            0.014864541590213776,
+            -0.0001531135058030486,
+            0.028037751093506813,
+            0.011342386715114117,
+            0.02416686899960041,
+            -0.009459254331886768,
+            -0.02852597087621689,
+            0.006368649657815695,
+            -0.0038904997054487467,
+            0.014376322738826275,
+            0.0091018071398139,
+            0.0201216209679842,
+            -0.01369630265980959,
+            -0.006695582531392574,
+            0.007789717521518469,
+            0.0018918508430942893,
+            0.014306576922535896,
+            0.003217018209397793,
+            0.019302109256386757,
+            0.009119244292378426,
+            -0.0008625575574114919,
+            -0.007737408392131329,
+            8.588795753894374e-05,
+            0.006250953767448664,
+            -0.003500359831377864,
+            -0.03047885000705719,
+            0.0016390228411182761,
+            -0.004834245424717665,
+            0.004978095646947622,
+            -0.014245549216866493,
+            0.010409538634121418,
+            -0.0010015039006248116,
+            0.022196553647518158,
+            -0.0041498662903904915,
+            0.004064863547682762,
+            0.01708768494427204,
+            0.00938950851559639,
+            0.004991173278540373,
+            0.015300451777875423,
+            0.005658115725964308,
+            0.004511671606451273,
+            -0.008038186468183994,
+            -0.019493909552693367,
+            0.019162617623806,
+            -0.027462350204586983,
+            0.010653648525476456,
+            -0.00873564276844263,
+            0.004317691549658775,
+            0.003925372380763292,
+            -0.006887382827699184,
+            -0.002567511750385165,
+            0.003607157850638032,
+            0.001491903094574809,
+            -0.059144310653209686,
+            0.0037706242874264717,
+            0.0007388679659925401,
+            0.020452912896871567,
+            -0.00881846621632576,
+            0.0007590288296341896,
+            0.0026394370943307877,
+            0.00390357687138021,
+            -0.013347573578357697,
+            -0.0023604542948305607,
+            0.01571892574429512,
+            -0.01858721673488617,
+            -0.0059632533229887486,
+            0.004038709215819836,
+            -0.005557856522500515,
+            0.0035940806847065687,
+            -0.008779234252870083,
+            -0.002622000640258193,
+            0.02838647924363613,
+            -0.001655369414947927,
+            0.03140297904610634,
+            0.002506484277546406,
+            -0.0004756871494464576,
+            -0.01666049286723137,
+            0.007410475518554449,
+            -0.016738956794142723,
+            -0.00425884360447526,
+            0.008927443064749241,
+            -0.018726706504821777,
+            -0.00123580580111593,
+            0.001693511614575982,
+            -0.0013382446486502886,
+            -0.0077722808346152306,
+            -0.001763257198035717,
+            0.0010429153917357326,
+            -0.02048778533935547,
+            0.015997909009456635,
+            -0.023853013291954994,
+            0.01625945419073105,
+            -0.0034371528308838606,
+            -0.002946753753349185,
+            -0.0013033718569204211,
+            -0.0038098562508821487,
+            0.005666834302246571,
+            0.02531767077744007,
+            -0.015448661521077156,
+            -0.00522220553830266,
+            0.008365118876099586,
+            -0.003567925887182355,
+            0.002730978187173605,
+            -0.008761797100305557,
+            0.006656350567936897,
+            -0.0325886569917202,
+            -0.008112290874123573,
+            0.004413592163473368,
+            0.0031908636447042227,
+            -0.001236895564943552,
+            -0.02758440561592579,
+            0.003960245288908482,
+            -0.00854384247213602,
+            -0.024201741442084312,
+            -0.00027380615938454866,
+            -0.020592402666807175,
+            0.006717377807945013,
+            -0.00881846621632576,
+            0.01761949434876442,
+            -0.014568123035132885,
+            0.011098276823759079,
+            -0.014367603696882725,
+            0.016006626188755035,
+            -0.007406116463243961,
+            -0.009398226626217365,
+            -0.0069266147911548615,
+            -0.006338136270642281,
+            0.01044441107660532,
+            -0.014916851185262203,
+            -0.009511562995612621,
+            -0.007641507778316736,
+            0.005161178298294544,
+            0.011045967228710651,
+            0.013574247248470783,
+            0.0011333668371662498,
+            -0.00880538858473301,
+            -0.016547156497836113,
+            -0.005392210558056831,
+            -0.014881978742778301,
+            -0.0009301236714236438,
+            0.0042174323461949825,
+            0.020853949710726738,
+            0.008160240948200226,
+            -0.003753187833353877,
+            0.011586496606469154,
+            0.020801641047000885,
+            0.014132212847471237,
+            -0.014777359552681446,
+            0.0021697436459362507,
+            -0.010095682926476002,
+            0.013417319394648075,
+            0.022144244983792305,
+            0.002098908182233572,
+            0.002796364715322852,
+            0.006085308268666267,
+            0.007951004430651665,
+            -0.004396155476570129,
+            0.008291014470160007,
+            -0.025457162410020828,
+            -0.012667553499341011,
+            -0.006569168530404568,
+            0.005387851502746344,
+            -0.015378915704786777,
+            0.029973194003105164,
+            -0.003609337378293276,
+            0.004472440108656883,
+            -0.002182820811867714,
+            0.0326409637928009,
+            0.007196879480034113,
+            0.00473834527656436,
+            0.002874828642234206,
+            9.467427298659459e-05,
+            0.009136680513620377,
+            -0.01761077716946602,
+            -0.03888320177793503,
+            0.003254070645198226,
+            -0.000592293159570545,
+            0.01766308583319187,
+            -0.01514352485537529,
+            0.006878664717078209,
+            -0.0036899808328598738,
+            -0.01282448135316372,
+            -0.009633618406951427,
+            0.03138554468750954,
+            -0.005466315429657698,
+            0.03387894853949547,
+            -0.007558684796094894,
+            -0.010104401037096977,
+            0.021725770086050034,
+            0.021900134161114693,
+            -0.014210676774382591,
+            0.005531701724976301,
+            0.015762517228722572,
+            0.020278548821806908,
+            0.005209128372371197,
+            -0.00806434080004692,
+            0.02627667412161827,
+            0.012946536764502525,
+            -0.012807045131921768,
+            -0.0026350778061896563,
+            -0.009572590701282024,
+            0.006726096384227276,
+            -0.012301389127969742,
+            0.005387851502746344,
+            0.006434036418795586,
+            0.004986814223229885,
+            0.012963972985744476,
+            -0.01879645325243473,
+            -0.003831651760265231,
+            0.0073014977388083935,
+            0.013966566883027554,
+            0.00651250034570694,
+            0.017558468505740166,
+            -0.023643774911761284,
+            0.0028966241516172886,
+            -0.02493407018482685,
+            -0.008103572763502598,
+            0.0029968833550810814,
+            0.001748000388033688,
+            0.008875134401023388,
+            -0.01592816226184368,
+            -0.029484974220395088,
+            0.011516750790178776,
+            0.01708768494427204,
+            0.02568383514881134,
+            0.0011126610916107893,
+            0.008914366364479065,
+            0.00560144754126668,
+            -0.013739893212914467,
+            -0.00805998221039772,
+            -0.00997362844645977,
+            -0.01073211245238781,
+            -0.011159304529428482,
+            -0.002761491807177663,
+            0.009119244292378426,
+            -0.010592620819807053,
+            0.0006821996648795903,
+            0.016346637159585953,
+            -0.011717269197106361,
+            -0.0051960512064397335,
+            0.015535843558609486,
+            -0.005008609499782324,
+            0.0007977159111760557,
+            0.0016891524428501725,
+            0.013086027465760708,
+            0.0034480507019907236,
+            -0.0063730087131261826,
+            -0.01591944508254528,
+            0.01215317938476801,
+            -0.00417166156694293,
+            0.0016074193408712745,
+            0.00708354264497757,
+            -0.007933568209409714,
+            -0.0047819362953305244,
+            0.013373728841543198,
+            -0.00854384247213602,
+            0.0016259454423561692,
+            -0.007606634870171547,
+            -0.017549749463796616,
+            -0.0076938169077038765,
+            0.006394804455339909,
+            -0.026730021461844444,
+            -0.013478347100317478,
+            0.016233300790190697,
+            -0.003570105414837599,
+            0.001546391868032515,
+            -0.0005138292908668518,
+            -0.011124431155622005,
+            0.003160349791869521,
+            -0.019598528742790222,
+            -0.01064493041485548,
+            -0.004864759277552366,
+            -0.02662540227174759,
+            0.017549749463796616,
+            0.015091215260326862,
+            0.0033172776456922293,
+            -0.00699636060744524,
+            -0.005592729430645704,
+            0.002567511750385165,
+            0.010348510928452015,
+            0.01763693243265152,
+            0.009311044588685036,
+            -0.005592729430645704,
+            -0.019685709848999977,
+            -0.003443691413849592,
+            0.021324733272194862,
+            -6.375188240781426e-05,
+            -0.016590746119618416,
+            0.020609840750694275,
+            0.007109697442501783,
+            0.014306576922535896,
+            0.0055883703753352165,
+            0.0023451976012438536,
+            0.005383492447435856,
+            0.014114775694906712,
+            -0.0001486181718064472,
+            -0.01783744990825653,
+            -0.013984003104269505,
+            0.0004890369018539786,
+            0.03140297904610634,
+            0.00708354264497757,
+            0.017139993607997894,
+            -0.03623286634683609,
+            0.0006642183288931847,
+            -0.005335542373359203,
+            -0.0038490882143378258,
+            -0.0045988536439836025,
+            0.027671586722135544,
+            0.010766984894871712,
+            -0.0001184450084110722,
+            0.015605589374899864,
+            -0.010043373331427574,
+            0.0422309935092926,
+            -0.015378915704786777,
+            -0.00737560261040926,
+            -0.010052092373371124,
+            -0.01685229316353798,
+            -0.003940629307180643,
+            -0.0011769578559324145,
+            -0.01689588464796543,
+            -0.006647632457315922,
+            -0.006320699583739042,
+            0.021411914378404617,
+            0.0046380856074392796,
+            -0.01665177382528782,
+            0.0017065888969227672,
+            -0.023835577070713043,
+            -0.0010946798138320446,
+            -0.022475536912679672,
+            -0.006403522565960884,
+            -0.012327543459832668,
+            -0.01349578332155943,
+            -0.020993441343307495,
+            -0.02031342126429081,
+            -0.004908350296318531,
+            0.016547156497836113,
+            0.00532246520742774,
+            -0.0069091785699129105,
+            -0.0021043571177870035,
+            -0.014402477070689201,
+            -0.010278765112161636,
+            0.02144678682088852,
+            0.015605589374899864,
+            0.012432162649929523,
+            0.01332141924649477,
+            0.01685229316353798,
+            0.0037597264163196087,
+            -0.034593842923641205,
+            0.006128899287432432,
+            0.0041607641614973545,
+            -0.005435801576822996,
+            0.01782873272895813,
+            0.000511104823090136,
+            -0.010793139226734638,
+            0.013399883173406124,
+            -0.007593557704240084,
+            0.018098996952176094,
+            -0.003212659154087305,
+            -0.004184739198535681,
+            -0.005536061245948076,
+            -0.008740002289414406,
+            -0.007371243555098772,
+            0.00455090356990695,
+            0.0021043571177870035,
+            0.013643993064761162,
+            0.0005775811732746661,
+            -0.004625008441507816,
+            -0.011333668604493141,
+            0.0014602995943278074,
+            -0.0032257363200187683,
+            -0.001628125086426735,
+            -0.023835577070713043,
+            -0.012170616537332535,
+            0.0017120378324761987,
+            -0.0086179468780756,
+            -0.01235369872301817,
+            0.02856084331870079,
+            -0.0017011400777846575,
+            -0.0008009852026589215,
+            0.010688520967960358,
+            0.015675334259867668,
+            0.001266319421119988,
+            0.01837797835469246,
+            -0.012170616537332535,
+            -0.011185458861291409,
+            -0.01649484597146511,
+            -0.01282448135316372,
+            -0.01122033130377531,
+            0.02090625837445259,
+            0.0033979208674281836,
+            -0.00708354264497757,
+            0.010418256744742393,
+            -0.011106994934380054,
+            -0.005056559573858976,
+            -0.0091018071398139,
+            0.011708551086485386,
+            -0.02969421073794365,
+            0.02069702185690403,
+            -0.012711144983768463,
+            0.006024280562996864,
+            0.018639525398612022,
+            0.039231929928064346,
+            -0.006608400493860245,
+            -0.006869946606457233,
+            0.010034655220806599,
+            -0.003068808699026704,
+            0.029205990955233574,
+            -0.0059414575807750225,
+            0.004237048327922821,
+            -0.021551406010985374,
+            0.0018449904164299369,
+            -0.01074954867362976,
+            0.0012815763475373387,
+            0.01991238258779049,
+            -0.0191974900662899,
+            -0.013722456991672516,
+            -0.00675660977140069,
+            -0.016058936715126038,
+            -0.00823870487511158,
+            0.022283734753727913,
+            0.011377259157598019,
+            -0.006111462600529194,
+            -0.008273578248918056,
+            -0.005736579652875662,
+            -0.024201741442084312,
+            0.014716332778334618,
+            0.08174190670251846,
+            -0.009982346557080746,
+            -0.0001858067698776722,
+            0.0003367407189216465,
+            0.0017806936521083117,
+            0.007907412946224213,
+            -0.00031821453012526035,
+            0.05956278741359711,
+            -0.003509078174829483,
+            0.003899217816069722,
+            -0.012998845428228378,
+            0.01083673071116209,
+            -0.01438504084944725,
+            -0.005684270523488522,
+            -0.009712082333862782,
+            -0.01197881530970335,
+            -0.0008167869527824223,
+            -0.02203962579369545,
+            -0.001063076313585043,
+            -0.011996251530945301,
+            -0.005021686665713787,
+            -0.021324733272194862,
+            0.017514877021312714,
+            -0.02087138593196869,
+            -0.01648612879216671,
+            -0.004594494588673115,
+            -0.010705957189202309,
+            0.012118306942284107,
+            -0.016555873677134514,
+            0.002380070276558399,
+            0.007270984351634979,
+            0.014228112995624542,
+            0.007475861813873053,
+            -0.008199472911655903,
+            -0.008408710360527039,
+            -0.0076197125017642975,
+            0.004986814223229885,
+            0.0069876424968242645,
+            -0.0011088468600064516,
+            0.014637868851423264,
+            -0.02088882215321064,
+            -0.016058936715126038,
+            0.007589198648929596,
+            0.0017817834159359336,
+            0.009502844884991646,
+            -0.007820230908691883,
+            0.007449707482010126,
+            0.00020882829267065972,
+            0.022754518315196037,
+            -0.010156710632145405,
+            -0.0268520750105381,
+            -0.007933568209409714,
+            0.008644101209938526,
+            -0.013138337060809135,
+            -0.006102744489908218,
+            0.005331183318048716,
+            0.009807982482016087,
+            0.008081777021288872,
+            -0.009877728298306465,
+            0.0007110786973498762,
+            -0.018918508663773537,
+            0.013373728841543198,
+            0.003238813718780875,
+            0.022196553647518158,
+            -0.012667553499341011,
+            0.010880321264266968,
+            -0.02683463878929615,
+            -0.002574050333350897,
+            -0.014419913291931152,
+            0.0008592882077209651,
+            0.008434864692389965,
+            0.004742704331874847,
+            0.01684357412159443,
+            -0.017192302271723747,
+            -0.017096402123570442,
+            0.009311044588685036,
+            -0.01782873272895813,
+            -0.02376583032310009,
+            -0.0018471699440851808,
+            -0.02624180167913437,
+            -0.0038163948338478804,
+            0.007532530464231968,
+            -0.019389290362596512,
+            -0.01369630265980959,
+            0.0065953233279287815,
+            -0.0033412526827305555,
+            -0.014611713588237762,
+            0.017183585092425346,
+            -0.011970097199082375,
+            -0.03544822707772255,
+            0.003727033268660307,
+            0.008382555097341537,
+            -0.004729627165943384,
+            0.006747891660779715,
+            -0.024410977959632874,
+            0.016015345230698586,
+            -0.002397506730630994,
+            0.004293716512620449,
+            0.008029468357563019,
+            0.02606743760406971,
+            0.0010309278732165694,
+            0.0034545892849564552,
+            -0.005544779356569052,
+            0.014411195181310177,
+            0.005801966413855553,
+            0.023504285141825676,
+            0.026416165754199028,
+            0.0015529305674135685,
+            -0.010958785191178322,
+            -0.01081929448992014,
+            0.01592816226184368,
+            -0.011586496606469154,
+            -0.0048386044800281525,
+            0.022737082093954086,
+            0.013827075250446796,
+            -0.02203962579369545,
+            0.00017259323794860393,
+            -0.02149909734725952,
+            0.0069876424968242645,
+            -0.017148712649941444,
+            0.009467972442507744,
+            0.008731283247470856,
+            -0.005073996260762215,
+            0.02069702185690403,
+            0.010095682926476002,
+            0.030984506011009216,
+            0.01991238258779049,
+            0.002920599188655615,
+            0.011412132531404495,
+            -0.006451472640037537,
+            0.010470565408468246,
+            0.004537826403975487,
+            0.0022275017108768225,
+            -0.006608400493860245,
+            0.006930973846465349,
+            0.0023451976012438536,
+            -0.001170419156551361,
+            0.008583074435591698,
+            -0.007471502758562565,
+            0.009921318851411343,
+            0.0005939278053119779,
+            0.0055273426696658134,
+            -0.004287178162485361,
+            -0.010575184598565102,
+            -0.030339358374476433,
+            -0.003132015699520707,
+            0.020958568900823593,
+            -0.023539157584309578,
+            -0.008003313094377518,
+            0.005932739470154047,
+            0.016983065754175186,
+            -0.008282296359539032,
+            -0.012798327021300793,
+            -0.01708768494427204,
+            0.003105861134827137,
+            -0.0004893093137070537,
+            -0.010313638485968113,
+            0.01747128553688526,
+            0.0009546435903757811,
+            -0.004951941315084696,
+            0.008953598327934742,
+            -0.00680456031113863,
+            0.028212115168571472,
+            0.01505634281784296,
+            0.011150586418807507,
+            -0.012100870721042156,
+            0.014201958663761616,
+            -0.012711144983768463,
+            0.007680739741772413,
+            0.009799264371395111,
+            -0.006416599731892347,
+            -0.021220114082098007,
+            -0.017558468505740166,
+            0.023451974615454674,
+            0.006848151329904795,
+            0.012222925201058388,
+            -0.011987533420324326,
+            -0.0012815763475373387,
+            0.007475861813873053,
+            0.004834245424717665,
+            -0.01611996442079544,
+            -0.00048631246318109334,
+            0.0061986446380615234,
+            0.038604218512773514,
+            0.005087073426693678,
+            0.01072339341044426,
+            -0.0035984397400170565,
+            -0.0002066487359115854,
+            0.019162617623806,
+            -0.01952878199517727,
+            -0.006503781769424677,
+            -0.005832480266690254,
+            -0.02739260531961918,
+            0.016573309898376465,
+            -0.01685229316353798,
+            -0.016451256349682808,
+            0.010976221412420273,
+            -0.00041356991278007627,
+            -0.004422310274094343,
+            -0.01284191757440567,
+            -0.008766156621277332,
+            0.00794228632003069,
+            0.010810576379299164,
+            -0.009136680513620377,
+            0.00532246520742774,
+            0.00357446470297873,
+            -0.0035984397400170565,
+            0.02127242274582386,
+            0.0006467819330282509,
+            -0.002837776206433773,
+            0.007358166389167309
+          ],
+          "title": "Method",
+          "keyphrases": [
+            "mixture-of-experts (MoE) models",
+            "token-choice routing",
+            "load imbalance",
+            "expert choice method",
+            "Transformer architecture"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": true,
+      "properties": {
+        "cosine_similarity": 0.8246007145725218
+      }
+    },
+    {
+      "id": "b76957d8-0f49-4534-8c79-fce9c719eb68",
+      "type": "cosine_similarity",
+      "source": {
+        "id": "8bd19de5-c2d4-458d-833b-5962c425f915",
+        "properties": {
+          "page_content": "1 Introduction\n\nScaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27]. The final model quality has been found to have a power-law relationship with the amount of data, model size, and compute time [16, 20]. However, training efficiency, which is defined as the total amount of computation used to achieve superior model quality than the state of the art system [21], should receive greater attention as we increase our efforts towards green AI [29].\n\nSparsely gated mixture-of-experts [31] (MoE) provides an effective way to scale model capacity given a fixed computational cost, and has recently played an important role in increasing the training efficiency of large-scale language models [10, 21]. MoE operate by adopting a number of experts, each as a sub-network, and by activating only one or a few experts for each input token. A gating network must be chosen and optimized in order to route each token to the most suited expert(s). For example, recent work has implemented sparse routing via k-means clustering [12], linear assignment to maximize token-expert affinities [22], or hashing [8, 28]. Many of the prior work use a routing strategy concerning the token choice, where each token selects the best one or two experts.\n\nWe argue that the independent token choice of prior work often leads to an imbalanced load of experts, which causes training inefficiency and sub-optimal training of the model. In order to mitigate this\n\nissue, previous sparsely gated networks introduce additional auxiliary losses as regularization to prevent too many tokens being routed to a single expert, but the effectiveness is still limited. Recent approaches [8, 22, 28] explore alternative strategies for routing, but they focus on pre-training only and do not demonstrate performance gain on downstream tasks. Moreover, none of the previous methods consider allocating a variable number of experts to each token based on importance, which can be beneficial.\n\nWe propose a very simple yet effective routing method we are calling expert choice. Unlike conventional MoE where tokens select one or two top-scoring experts, our method lets each expert pick the top-k tokens. Our method guarantees perfect load balancing, allows a variable number of experts for each token, and achieves substantial gains in training efficiency and downstream performance as demonstrated in our experiments. Our major contributions include:\n\nWe identify common pitfalls in conventional MoE such as load imbalance as described in Section 3.1. We then propose a heterogeneous, expert choice method to provide a fluid allocation of model parameters based on a learnt token-to-expert importance. This method intrinsically guarantees load balance without imposing an auxiliary loss.\n\nWe show our method provides over 2× faster training convergence in a 8B/64E (8 billion activated parameters, 64 experts) model, compared to the top-1 and top-2 gating counterparts in Switch Transformer [10] and GShard [21].\n\nWe show our method demonstrates strong scaling when increasing the number of experts from 16 to 128, evaluated in training perplexity.\n\nWe show our method demonstrates strong performance on downstream tasks selected from GLUE and SuperGLUE at all the evaluated scales. More specifically, our 8B/64E model outperforms a T5 11B dense model in 7 out of 11 tasks evaluated.\n\n",
+          "title": "Introduction",
+          "embedding": [
+            0.0024422307033091784,
+            -0.014919951558113098,
+            -0.01768755167722702,
+            -0.02871091105043888,
+            0.020494353026151657,
+            0.03838575258851051,
+            -0.01718577742576599,
+            0.0458810068666935,
+            -0.00809895433485508,
+            0.02212512120604515,
+            -0.01569613441824913,
+            -0.011674097739160061,
+            0.024147899821400642,
+            -0.016072465106844902,
+            -0.003586903912946582,
+            0.029040200635790825,
+            0.00016084715025499463,
+            0.0012985375942662358,
+            0.021921275183558464,
+            -0.021889913827180862,
+            -0.0052411919459700584,
+            -0.014426017180085182,
+            -0.01895766891539097,
+            -0.002695078030228615,
+            0.025621861219406128,
+            0.010239336639642715,
+            0.029526295140385628,
+            -0.00020776601741090417,
+            -0.05397212132811546,
+            0.04054965451359749,
+            0.015868619084358215,
+            0.025010323151946068,
+            -0.020776601508259773,
+            -0.014253532513976097,
+            -0.047417692840099335,
+            -0.029228366911411285,
+            0.05165141448378563,
+            0.018314769491553307,
+            -0.027127185836434364,
+            0.030765051022171974,
+            0.0260295532643795,
+            0.018973348662257195,
+            -0.037350840866565704,
+            -0.009784603491425514,
+            0.0013112779706716537,
+            0.00785982795059681,
+            -0.029761500656604767,
+            0.00613497756421566,
+            0.022031037136912346,
+            -0.01818932592868805,
+            0.02342659793794155,
+            -0.01548444852232933,
+            0.015492288395762444,
+            0.006064415443688631,
+            -0.06635968387126923,
+            0.04058101400732994,
+            0.0031008098740130663,
+            0.008036232553422451,
+            0.05513247475028038,
+            0.019961217418313026,
+            0.018205007538199425,
+            -0.0099727688357234,
+            -0.04525379091501236,
+            0.008130315691232681,
+            0.006036974955350161,
+            -0.019506484270095825,
+            -0.008647770620882511,
+            0.015061075799167156,
+            0.023410918191075325,
+            0.01010605227202177,
+            -0.055038392543792725,
+            0.029275408014655113,
+            -0.019929856061935425,
+            0.005562641192227602,
+            0.006781796459108591,
+            -0.01776595413684845,
+            0.01925559714436531,
+            0.01078031212091446,
+            0.005041265860199928,
+            0.0031556913163512945,
+            -0.01123504526913166,
+            0.038072142750024796,
+            0.004821739625185728,
+            0.014802348800003529,
+            0.04224314168095589,
+            -0.03951474279165268,
+            0.0034300992265343666,
+            -0.06153010204434395,
+            0.01611950621008873,
+            -0.012019068002700806,
+            -0.030733689665794373,
+            -0.003298775525763631,
+            -0.01593918167054653,
+            0.017828676849603653,
+            -0.003900512820109725,
+            0.041521843522787094,
+            0.011423210613429546,
+            -0.006993482355028391,
+            -0.02982422336935997,
+            -0.0009119163732975721,
+            -0.005151029210537672,
+            0.017985479906201363,
+            0.0025931550189852715,
+            -0.023175710812211037,
+            0.019396722316741943,
+            -0.03333664685487747,
+            -0.033023037016391754,
+            0.035281021147966385,
+            0.017907077446579933,
+            0.05739046260714531,
+            0.024100856855511665,
+            -0.011509452946484089,
+            0.011870103888213634,
+            0.025621861219406128,
+            -0.02924404665827751,
+            -0.017279859632253647,
+            -0.03810350224375725,
+            -0.06316086649894714,
+            0.019020389765501022,
+            -0.013657675124704838,
+            0.01680944673717022,
+            0.006762195844203234,
+            -0.012716848403215408,
+            -0.008624250069260597,
+            -0.06369400024414062,
+            0.045347873121500015,
+            -0.014026165939867496,
+            -0.011352648958563805,
+            -0.0018924347823485732,
+            -0.02450854890048504,
+            -0.003341896692290902,
+            0.028397301211953163,
+            -0.03418339043855667,
+            -0.05814312398433685,
+            0.040330126881599426,
+            0.013148060999810696,
+            0.03534374386072159,
+            0.04365438222885132,
+            -0.015782376751303673,
+            -0.022234883159399033,
+            0.026217719539999962,
+            -0.035124216228723526,
+            0.007946070283651352,
+            -0.0005517559475265443,
+            -0.02668813243508339,
+            0.01322646252810955,
+            0.010011970065534115,
+            0.06447802484035492,
+            -0.02052571438252926,
+            0.021889913827180862,
+            -0.02918132394552231,
+            0.057108212262392044,
+            -0.016260631382465363,
+            0.02201535739004612,
+            -0.023018905892968178,
+            -0.0031008098740130663,
+            -0.031219784170389175,
+            0.0018375532235950232,
+            -0.019584886729717255,
+            -0.011485932394862175,
+            -0.02443014644086361,
+            -0.01301477663218975,
+            -0.007569739129394293,
+            0.01749938726425171,
+            0.010686228983104229,
+            0.024931922554969788,
+            0.030827771872282028,
+            0.002036498859524727,
+            -0.07018571346998215,
+            0.026468606665730476,
+            0.018205007538199425,
+            0.01676240563392639,
+            0.01731122098863125,
+            0.019004710018634796,
+            0.006217299960553646,
+            0.026656771078705788,
+            0.011148802936077118,
+            0.011195844039320946,
+            -0.008059753105044365,
+            0.03985971212387085,
+            -0.03716267645359039,
+            -0.007585419807583094,
+            0.01726417988538742,
+            0.0046766954474151134,
+            0.026249079033732414,
+            0.018079563975334167,
+            0.01726417988538742,
+            0.012748208828270435,
+            0.007448215503245592,
+            0.01121936459094286,
+            0.00966699980199337,
+            0.03672362491488457,
+            0.01779731549322605,
+            -0.01638607494533062,
+            0.016433116048574448,
+            -0.02817777544260025,
+            -0.01050590444356203,
+            -0.004990304354578257,
+            0.02722126804292202,
+            0.00673867529258132,
+            -0.02339523658156395,
+            -0.0031027698423713446,
+            0.009196585975587368,
+            -0.022532811388373375,
+            0.008702651597559452,
+            0.006174178794026375,
+            -6.774446228519082e-05,
+            0.009525875560939312,
+            0.007271810434758663,
+            -0.030043749138712883,
+            0.023630443960428238,
+            -0.01404968649148941,
+            -0.008726172149181366,
+            -0.01792275905609131,
+            -0.007032683584839106,
+            -0.000910936389118433,
+            0.004002436064183712,
+            0.006323142908513546,
+            0.006664192769676447,
+            -0.022501451894640923,
+            -0.004837419837713242,
+            0.011454571969807148,
+            0.02488487958908081,
+            0.002355988137423992,
+            -0.0009266168344765902,
+            -0.020353229716420174,
+            -0.006664192769676447,
+            -0.01840885356068611,
+            0.04258811101317406,
+            -0.005386236123740673,
+            -0.03223901242017746,
+            0.009902206249535084,
+            0.014849389903247356,
+            0.010200135409832,
+            0.010286377742886543,
+            -0.042556751519441605,
+            0.00047947888378985226,
+            0.0019267357420176268,
+            0.005268632434308529,
+            -0.01114096213132143,
+            -0.019114473834633827,
+            0.019882814958691597,
+            -0.051431890577077866,
+            -0.014347615651786327,
+            0.029306767508387566,
+            0.02737807296216488,
+            -0.020070981234312057,
+            0.01941240206360817,
+            -0.007851987145841122,
+            -0.004453248810023069,
+            -0.0246967151761055,
+            -0.024336064234375954,
+            0.01234835758805275,
+            0.014378976076841354,
+            -0.0033595373388379812,
+            0.03434019535779953,
+            0.016213588416576385,
+            -0.034089308232069016,
+            -0.017091695219278336,
+            -0.0149669935926795,
+            0.050052009522914886,
+            -0.013006936758756638,
+            -0.015194360166788101,
+            0.06372536718845367,
+            -0.0255748201161623,
+            0.012622765265405178,
+            0.03515557944774628,
+            0.014684745110571384,
+            -0.002234464744105935,
+            -0.024947602301836014,
+            -0.0025970751885324717,
+            -0.05425437167286873,
+            -0.0051588695496320724,
+            -0.013720396906137466,
+            0.04224314168095589,
+            -0.04142775759100914,
+            -0.004076918121427298,
+            -0.005880170036107302,
+            0.01577453687787056,
+            0.03024759516119957,
+            0.006785716395825148,
+            0.00034153988235630095,
+            0.01049022376537323,
+            0.02308162860572338,
+            -0.0034457796718925238,
+            -0.014465218409895897,
+            -0.005151029210537672,
+            -0.018471574410796165,
+            -0.04838988184928894,
+            -0.011046879924833775,
+            -0.01742098480463028,
+            -0.004813899286091328,
+            -0.031454991549253464,
+            -0.00391815323382616,
+            0.03531238064169884,
+            -0.010647027753293514,
+            -0.015421726740896702,
+            0.008522327058017254,
+            0.00809895433485508,
+            -0.022877782583236694,
+            -0.04719816520810127,
+            0.004974624142050743,
+            0.00666027283295989,
+            -0.011321287602186203,
+            -0.0572650171816349,
+            0.006750435568392277,
+            -0.02461831271648407,
+            0.01837749220430851,
+            -0.021999677643179893,
+            -0.0099727688357234,
+            0.025872748345136642,
+            0.0016327272169291973,
+            0.0045120506547391415,
+            -0.02573162503540516,
+            0.0011064520804211497,
+            0.022595534101128578,
+            0.006699474062770605,
+            0.021105891093611717,
+            -0.027958249673247337,
+            -0.03418339043855667,
+            0.020839322358369827,
+            -0.0055077592842280865,
+            -0.06350583583116531,
+            -0.0026676373090595007,
+            -0.02251713164150715,
+            0.0027009581681340933,
+            0.019161514937877655,
+            0.04694727808237076,
+            -0.030184874311089516,
+            0.008663450367748737,
+            -0.023646123707294464,
+            -0.04650822654366493,
+            -0.030467122793197632,
+            -0.0027303590904921293,
+            -0.007957830093801022,
+            -0.026547009125351906,
+            -0.012991256080567837,
+            0.008467445150017738,
+            -0.03901296854019165,
+            0.009274988435208797,
+            -0.014245692640542984,
+            -0.026233399286866188,
+            -0.028648190200328827,
+            0.010427501983940601,
+            0.00468061538413167,
+            -0.002107060980051756,
+            -0.03603368252515793,
+            0.0060722557827830315,
+            0.004939342848956585,
+            -0.005162789486348629,
+            0.0010476503521203995,
+            0.002673517446964979,
+            0.004296444356441498,
+            0.005880170036107302,
+            -0.0027538796421140432,
+            -0.04183544963598251,
+            -0.026750855147838593,
+            0.016072465106844902,
+            0.02090204507112503,
+            -0.001751310657709837,
+            0.029197005555033684,
+            -0.050898753106594086,
+            -0.03230173513293266,
+            0.025951150804758072,
+            -0.06002477928996086,
+            0.006272181402891874,
+            -0.011995547451078892,
+            -0.022376008331775665,
+            -0.00047531374730169773,
+            -0.014073207043111324,
+            -0.02063547819852829,
+            0.026249079033732414,
+            0.02626476064324379,
+            0.017624830827116966,
+            0.017938438802957535,
+            0.013336226344108582,
+            0.02342659793794155,
+            -0.02220352366566658,
+            -0.06391353160142899,
+            0.035625990480184555,
+            -0.01566477306187153,
+            0.007781425025314093,
+            0.017860036343336105,
+            -0.012536522932350636,
+            -0.004010275937616825,
+            0.0303573589771986,
+            0.011713298968970776,
+            0.004022036213427782,
+            -0.018346130847930908,
+            0.05221591144800186,
+            -0.005986013449728489,
+            -0.04886029288172722,
+            -0.0021325417328625917,
+            -0.012120991013944149,
+            0.023489320650696754,
+            0.023348195478320122,
+            -0.00301064713858068,
+            -0.028742272406816483,
+            -0.022830741479992867,
+            -0.007197328377515078,
+            -0.015194360166788101,
+            0.02798961102962494,
+            -0.00789902824908495,
+            0.01466122455894947,
+            -0.0194908045232296,
+            -0.001328918500803411,
+            -0.004104358609765768,
+            0.01139185018837452,
+            0.04287036135792732,
+            -0.040643736720085144,
+            -0.009235787205398083,
+            -0.03361889347434044,
+            -0.03261534497141838,
+            0.00725613022223115,
+            0.017562108114361763,
+            0.03844847157597542,
+            -0.02182719111442566,
+            0.01078031212091446,
+            0.01853429712355137,
+            -0.010349099524319172,
+            -0.012418919242918491,
+            0.03393250331282616,
+            0.04873485118150711,
+            0.03452835977077484,
+            0.011956346221268177,
+            -0.00046306339208967984,
+            0.008992739953100681,
+            0.011415370739996433,
+            0.01152513362467289,
+            0.012967735528945923,
+            0.029338128864765167,
+            -0.008067593909800053,
+            0.006969961803406477,
+            -0.018126605078577995,
+            -0.020118022337555885,
+            -0.01611950621008873,
+            -0.011713298968970776,
+            0.030529843643307686,
+            -0.011768180876970291,
+            -0.005782167427241802,
+            0.0294792540371418,
+            -0.008498805575072765,
+            0.00907114241272211,
+            -0.009392592124640942,
+            -0.023755887523293495,
+            -0.03085913322865963,
+            -0.04202361777424812,
+            -0.005292152985930443,
+            -0.007753984536975622,
+            -0.035092856734991074,
+            -0.0529058501124382,
+            0.010380460880696774,
+            -0.07514073699712753,
+            -0.002481431933119893,
+            0.029965346679091454,
+            0.0010692109353840351,
+            0.0008305740193463862,
+            -0.0010888115502893925,
+            0.008208717219531536,
+            -0.03763309121131897,
+            0.0031478512100875378,
+            0.02243872918188572,
+            0.03089049458503723,
+            -0.0294792540371418,
+            -0.010991998016834259,
+            0.007103245705366135,
+            -0.03879344463348389,
+            -0.0038828724063932896,
+            0.0039887153543531895,
+            0.005190230440348387,
+            0.013312705792486668,
+            -0.03227037563920021,
+            -0.0069033196195960045,
+            0.004982464015483856,
+            0.01630767248570919,
+            0.03835438936948776,
+            -0.0023677486460655928,
+            -0.04512834548950195,
+            0.019004710018634796,
+            -0.0007732423837296665,
+            0.01531196292489767,
+            -0.00893785897642374,
+            -0.020839322358369827,
+            -0.0015778456581756473,
+            -0.00539799639955163,
+            0.0006272181635722518,
+            0.023332515731453896,
+            -0.008530166931450367,
+            0.01787571795284748,
+            -0.01707601360976696,
+            -0.015210039913654327,
+            -0.002385389059782028,
+            -0.006679873447865248,
+            0.018863586708903313,
+            0.0031125701498240232,
+            -0.021121570840477943,
+            -0.026092275977134705,
+            -0.017217138782143593,
+            0.027503516525030136,
+            0.0035927840508520603,
+            -0.040298767387866974,
+            -0.012795249931514263,
+            0.024289023131132126,
+            0.019726011902093887,
+            0.006613231264054775,
+            -0.019616248086094856,
+            0.008208717219531536,
+            0.03024759516119957,
+            -0.044657934457063675,
+            -0.02522985078394413,
+            0.0026931180618703365,
+            0.0033066156320273876,
+            -0.004888381343334913,
+            -0.011148802936077118,
+            0.0006757295341230929,
+            0.04365438222885132,
+            -0.010576466098427773,
+            0.021717429161071777,
+            0.019631927832961082,
+            0.01684080809354782,
+            -0.01898903027176857,
+            -0.013649835251271725,
+            -0.011454571969807148,
+            0.025010323151946068,
+            -0.014669064432382584,
+            0.027205588296055794,
+            0.022219203412532806,
+            0.002944005187600851,
+            0.06447802484035492,
+            -0.026672452688217163,
+            -0.025559140369296074,
+            0.011705459095537663,
+            -0.013579272665083408,
+            0.0009697380592115223,
+            -0.019475122913718224,
+            -0.007812785916030407,
+            -0.004092598333954811,
+            0.022109439596533775,
+            -0.028914757072925568,
+            -0.028507065027952194,
+            0.0017552308272570372,
+            0.010129572823643684,
+            -0.009925727732479572,
+            0.01257572416216135,
+            0.0027950408402830362,
+            0.004539491608738899,
+            0.0212470144033432,
+            -0.04149048030376434,
+            0.011681938543915749,
+            0.035469185560941696,
+            -0.02477511763572693,
+            -0.034402914345264435,
+            -0.031219784170389175,
+            0.011862263083457947,
+            0.0029890865553170443,
+            -0.04434432461857796,
+            -0.015037555247545242,
+            -0.029996708035469055,
+            -0.0015660853823646903,
+            -0.03572007268667221,
+            -0.00958859734237194,
+            0.010082531720399857,
+            -0.0037143074441701174,
+            0.01183090265840292,
+            -0.022987544536590576,
+            -0.005445037502795458,
+            0.028852034360170364,
+            0.04478337615728378,
+            0.012168032117187977,
+            0.020337548106908798,
+            -0.001616066787391901,
+            -0.024978963658213615,
+            -0.015492288395762444,
+            0.012599244713783264,
+            -0.0007776524871587753,
+            0.0031968525145202875,
+            0.0294478926807642,
+            -0.004907981958240271,
+            -0.03763309121131897,
+            0.00632706331089139,
+            0.04095734655857086,
+            -0.020572755485773087,
+            0.007146366871893406,
+            0.04192953556776047,
+            0.025935471057891846,
+            -0.019239917397499084,
+            -0.020008258521556854,
+            -0.031298186630010605,
+            0.004951103124767542,
+            -0.02817777544260025,
+            -0.005201990716159344,
+            0.03688042610883713,
+            -0.006185939069837332,
+            -0.026390204206109047,
+            0.012262115254998207,
+            -0.02297186478972435,
+            -0.009337710216641426,
+            0.01278741005808115,
+            0.01100767869502306,
+            -0.03625321015715599,
+            0.013328385539352894,
+            0.019537845626473427,
+            0.023332515731453896,
+            -0.008882977068424225,
+            -0.018973348662257195,
+            0.021074529737234116,
+            0.006166338454931974,
+            0.0032301736064255238,
+            -0.004202361684292555,
+            0.0075501385144889355,
+            0.019773053005337715,
+            -0.0036045443266630173,
+            0.011305606923997402,
+            -0.0034300992265343666,
+            0.03317984193563461,
+            -0.014473059214651585,
+            -0.002887163544073701,
+            0.01188578363507986,
+            -0.014622023329138756,
+            -0.009314189665019512,
+            0.059523001313209534,
+            0.007213008590042591,
+            0.040486931800842285,
+            -0.003974995110183954,
+            0.018847905099391937,
+            0.024633992463350296,
+            -0.01627631112933159,
+            -0.027205588296055794,
+            0.0006580890621989965,
+            -0.013869361020624638,
+            -0.03085913322865963,
+            -0.01699761115014553,
+            -0.03534374386072159,
+            -0.018471574410796165,
+            -0.00014565671153832227,
+            -0.01404968649148941,
+            0.0007394314161501825,
+            0.028428662568330765,
+            0.010239336639642715,
+            -0.012168032117187977,
+            -0.033901140093803406,
+            0.03885616362094879,
+            0.030874812975525856,
+            0.03261534497141838,
+            -0.007236529607325792,
+            0.019271278753876686,
+            -0.010074691846966743,
+            -0.0058135283179581165,
+            0.020807962864637375,
+            0.015437406487762928,
+            -0.00238342909142375,
+            0.014825869351625443,
+            0.01577453687787056,
+            0.0459437295794487,
+            -0.025621861219406128,
+            -0.03024759516119957,
+            -0.010176614858210087,
+            -0.0260295532643795,
+            0.027817124500870705,
+            0.012983416207134724,
+            -0.015445247292518616,
+            0.015727495774626732,
+            0.017044654116034508,
+            -0.029620377346873283,
+            0.02010234259068966,
+            -0.01374391745775938,
+            -0.03195676580071449,
+            0.030325997620821,
+            -0.016825126484036446,
+            0.013242143206298351,
+            -0.007013082969933748,
+            -0.004033796954900026,
+            -0.020086660981178284,
+            0.009298508986830711,
+            0.01131344772875309,
+            -0.028554106131196022,
+            0.038072142750024796,
+            0.04368574544787407,
+            0.003083169227465987,
+            0.009604278020560741,
+            0.014574982225894928,
+            -0.0007943129749037325,
+            -0.020353229716420174,
+            -0.012724688276648521,
+            0.006679873447865248,
+            -0.001151533331722021,
+            -0.028836354613304138,
+            0.0307807307690382,
+            -0.016103826463222504,
+            0.007495257072150707,
+            -0.01154081430286169,
+            -0.006087936460971832,
+            0.0063192229717969894,
+            -0.0007805926143191755,
+            0.006558349821716547,
+            0.0007997031207196414,
+            -0.03170587867498398,
+            0.014888591133058071,
+            0.031110020354390144,
+            -0.005758646875619888,
+            -0.01079599279910326,
+            0.01737394370138645,
+            -0.02090204507112503,
+            0.009180906228721142,
+            0.028005290776491165,
+            0.015186519362032413,
+            0.004206281621009111,
+            -0.0051980703137815,
+            -0.030310317873954773,
+            -0.020243465900421143,
+            -0.001436721533536911,
+            0.024304702877998352,
+            0.014841550029814243,
+            0.026437245309352875,
+            0.026735173538327217,
+            -0.00780886597931385,
+            0.0013436188455671072,
+            -0.0022677856031805277,
+            -0.03396386280655861,
+            0.04440704733133316,
+            -1.7808952179620974e-05,
+            0.01779731549322605,
+            0.021152932196855545,
+            -0.027095824480056763,
+            -0.027268309146165848,
+            -0.001135852886363864,
+            0.032333094626665115,
+            0.03484196960926056,
+            0.01699761115014553,
+            0.004590453114360571,
+            0.04202361777424812,
+            -0.02464967407286167,
+            -0.03741356357932091,
+            0.017891397699713707,
+            0.030043749138712883,
+            -0.009188746102154255,
+            0.0016738884150981903,
+            0.027017422020435333,
+            -0.011838742531836033,
+            -0.005574401468038559,
+            0.024367425590753555,
+            0.019083112478256226,
+            0.008914338424801826,
+            0.004104358609765768,
+            -0.009659159928560257,
+            0.02369316667318344,
+            0.002107060980051756,
+            0.024336064234375954,
+            -0.01960056647658348,
+            0.007318852003663778,
+            0.00945531390607357,
+            -0.026891978457570076,
+            0.033023037016391754,
+            0.009941407479345798,
+            -0.0502401739358902,
+            0.025668902322649956,
+            0.0032517341896891594,
+            -0.004304284695535898,
+            0.003394818166270852,
+            0.0059076109901070595,
+            0.007789265364408493,
+            0.023363877087831497,
+            0.00446108914911747,
+            -0.004096518736332655,
+            0.030106471851468086,
+            0.023912692442536354,
+            -0.0003197342448402196,
+            0.020666837692260742,
+            -0.0019110552966594696,
+            -0.02074524015188217,
+            -0.002781320596113801,
+            -0.0063780248165130615,
+            0.04105142876505852,
+            -0.0390443317592144,
+            0.014825869351625443,
+            0.02374020777642727,
+            0.003420298919081688,
+            -0.002857762621715665,
+            -0.02320707216858864,
+            -0.008443924598395824,
+            -0.011180163361132145,
+            0.024257661774754524,
+            -0.010145253501832485,
+            0.023301154375076294,
+            -0.02833458036184311,
+            -0.0038926727138459682,
+            0.01353223156183958,
+            -0.030498482286930084,
+            0.011031199246644974,
+            0.02722126804292202,
+            0.0010897915344685316,
+            -0.02010234259068966,
+            -0.005852729547768831,
+            0.036849066615104675,
+            0.055226556956768036,
+            0.0054959990084171295,
+            0.01630767248570919,
+            0.02867954969406128,
+            -0.02140381932258606,
+            -0.0106235072016716,
+            0.019537845626473427,
+            -0.018126605078577995,
+            0.014582822099328041,
+            0.04302716627717018,
+            -0.036347292363643646,
+            -0.03782125562429428,
+            -0.008130315691232681,
+            -0.032333094626665115,
+            0.008898657746613026,
+            -0.012003387324512005,
+            0.012889333069324493,
+            -0.009204426780343056,
+            -0.02841298282146454,
+            0.02798961102962494,
+            0.015170839615166187,
+            0.009384751319885254,
+            -0.014622023329138756,
+            -0.02182719111442566,
+            -0.02411653846502304,
+            -0.031078658998012543,
+            -0.008898657746613026,
+            -0.01765619032084942,
+            0.01975737139582634,
+            -0.0006556389853358269,
+            -0.04224314168095589,
+            -0.007848067209124565,
+            0.0194908045232296,
+            -0.012120991013944149,
+            0.040675096213817596,
+            0.02416357956826687,
+            0.016558559611439705,
+            -0.009267148561775684,
+            -0.03760172799229622,
+            0.05425437167286873,
+            -2.794612191792112e-05,
+            0.03027895651757717,
+            -0.013179421424865723,
+            -0.007655981462448835,
+            0.0026107956655323505,
+            0.027958249673247337,
+            0.005201990716159344,
+            0.009470994584262371,
+            -0.031909722834825516,
+            0.011078240349888802,
+            -0.024304702877998352,
+            -0.011932825669646263,
+            -0.006899399682879448,
+            -0.0034673402551561594,
+            0.02450854890048504,
+            0.00416316045448184,
+            0.006217299960553646,
+            0.02875795215368271,
+            0.012599244713783264,
+            -0.0017483705887570977,
+            -0.023834289982914925,
+            -0.013493030332028866,
+            -0.0015964661724865437,
+            0.003849551547318697,
+            -0.01676240563392639,
+            -0.013053977862000465,
+            -0.03835438936948776,
+            -0.010748950764536858,
+            0.0034810607321560383,
+            0.021152932196855545,
+            0.02151358313858509,
+            0.005688084755092859,
+            -0.010819513350725174,
+            -0.03085913322865963,
+            -0.007405094336718321,
+            -0.04735497012734413,
+            -0.016001902520656586,
+            0.018628379330039024,
+            -0.021043168380856514,
+            0.010356939397752285,
+            0.00260295532643795,
+            0.01590782031416893,
+            -0.0012289555743336678,
+            -0.01925559714436531,
+            0.003136090701445937,
+            -0.009086823090910912,
+            0.011548654176294804,
+            0.0010986118577420712,
+            0.03932657837867737,
+            -0.004084758460521698,
+            0.027660321444272995,
+            -0.029604697600007057,
+            0.008585048839449883,
+            -0.044124796986579895,
+            -0.0097610829398036,
+            -0.03866799920797348,
+            0.009855165146291256,
+            -0.008161676116287708,
+            -0.0338384211063385,
+            0.004523810930550098,
+            0.008357682265341282,
+            -0.002804841147735715,
+            -0.015892140567302704,
+            -0.006468187086284161,
+            0.02294050343334675,
+            -0.0047668581828475,
+            0.008953538723289967,
+            -0.03938930109143257,
+            -0.016166547313332558,
+            -0.007510937284678221,
+            0.03675498440861702,
+            -0.0036241449415683746,
+            0.017436664551496506,
+            -0.015727495774626732,
+            0.016213588416576385,
+            -0.04443840682506561,
+            -0.025825707241892815,
+            0.034057945013046265,
+            0.017389623448252678,
+            -0.021874234080314636,
+            0.026625411584973335,
+            0.019710330292582512,
+            0.04042420908808708,
+            -0.0580490417778492,
+            0.023802928626537323,
+            -0.0018473535310477018,
+            -0.009008420631289482,
+            0.019098792225122452,
+            0.010756791569292545,
+            0.03289759159088135,
+            -0.028961798176169395,
+            -0.014441697858273983,
+            0.006017374340444803,
+            0.020666837692260742,
+            8.042357512749732e-05,
+            0.006691633723676205,
+            -0.02358340285718441,
+            0.014496579766273499,
+            0.017562108114361763,
+            0.005190230440348387,
+            0.02703310176730156,
+            -0.026390204206109047,
+            0.028428662568330765,
+            0.014324095100164413,
+            -0.027770083397626877,
+            0.02132541686296463,
+            -0.027127185836434364,
+            -0.01390072237700224,
+            -0.001531784306280315,
+            0.02780144475400448,
+            0.011360488831996918,
+            -0.060244303196668625,
+            0.02063547819852829,
+            0.03173723816871643,
+            -0.006189859006553888,
+            0.03134522587060928,
+            -0.024053815752267838,
+            0.024665353819727898,
+            0.021372457966208458,
+            0.022799380123615265,
+            0.018440213054418564,
+            0.046037811785936356,
+            -0.00785982795059681,
+            0.01184658333659172,
+            -0.020118022337555885,
+            0.022579854354262352,
+            0.011595695279538631,
+            -0.009541556239128113,
+            -0.037570368498563766,
+            -0.021152932196855545,
+            0.007648141589015722,
+            0.013469509780406952,
+            -0.00831064023077488,
+            -0.03317984193563461,
+            -0.0458810068666935,
+            0.010090372525155544,
+            0.00569592509418726,
+            -0.057484544813632965,
+            -0.013555752113461494,
+            -0.009110343642532825,
+            0.005229431204497814,
+            -0.008459605276584625,
+            0.004061237443238497,
+            -0.028397301211953163,
+            0.015335483476519585,
+            -0.014457378536462784,
+            0.03716267645359039,
+            -0.011485932394862175,
+            0.022140800952911377,
+            -0.015837257727980614,
+            -0.004719816613942385,
+            -0.025292571634054184,
+            0.02488487958908081,
+            -0.021466542035341263,
+            -0.04199225455522537,
+            0.010835194028913975,
+            0.004480689764022827,
+            0.01622927002608776,
+            0.0537525974214077,
+            -0.005884090438485146,
+            -0.010317739099264145,
+            0.020321868360042572,
+            -0.019506484270095825,
+            -0.014002645388245583,
+            0.007679502479732037,
+            -0.005844889208674431,
+            -0.022031037136912346,
+            -0.01665264181792736,
+            -0.012262115254998207,
+            0.0247280765324831,
+            -0.00041602205601520836,
+            0.022611213847994804,
+            0.026390204206109047,
+            0.0195221658796072,
+            -0.02814641408622265,
+            0.009855165146291256,
+            -0.005127508658915758,
+            -0.011031199246644974,
+            0.004245482850819826,
+            0.021356778219342232,
+            -0.004755097441375256,
+            0.009000580757856369,
+            -0.01646447740495205,
+            -0.021121570840477943,
+            -0.03731948137283325,
+            -0.01249732170253992,
+            -0.003145891008898616,
+            0.0031478512100875378,
+            -0.035625990480184555,
+            -0.018126605078577995,
+            0.026060914620757103,
+            0.03070232830941677,
+            -0.028303219005465508,
+            0.0019051751587539911,
+            -0.002246225019916892,
+            -0.01132912840694189,
+            -0.003904432989656925,
+            -0.011791701428592205,
+            0.007040523923933506,
+            -0.015225720591843128,
+            8.691626862855628e-05,
+            -0.002042379230260849,
+            0.0023736287839710712,
+            -0.05792359635233879,
+            -0.017781633883714676,
+            0.02913428284227848,
+            0.002930284710600972,
+            0.00550383934751153,
+            -0.008279279805719852,
+            -0.01322646252810955,
+            -0.007009163033217192,
+            -0.02679789625108242,
+            0.024367425590753555,
+            0.015437406487762928,
+            0.019647609442472458,
+            -0.003559462958946824,
+            0.0017885518027469516,
+            0.018659740686416626,
+            0.0014171210350468755,
+            0.003351696999743581,
+            -0.007279650773853064,
+            0.006507388316094875,
+            0.02182719111442566,
+            -0.012795249931514263,
+            0.0158294178545475,
+            0.005778247490525246,
+            0.002685277722775936,
+            -0.02679789625108242,
+            -0.003682946553453803,
+            0.016793766990303993,
+            0.002657837001606822,
+            0.005296073388308287,
+            -0.0255434587597847,
+            0.038072142750024796,
+            0.04039284959435463,
+            0.025872748345136642,
+            0.020337548106908798,
+            -0.03220765292644501,
+            -0.004206281621009111,
+            -0.00758149940520525,
+            0.01121936459094286,
+            0.021811511367559433,
+            0.020055299624800682,
+            -0.010427501983940601,
+            -0.015743175521492958,
+            0.011062560603022575,
+            -0.02013370208442211,
+            -0.0019473163411021233,
+            0.018894946202635765,
+            0.011266405694186687,
+            -0.006201619748026133,
+            0.009988449513912201,
+            0.0004351325915195048,
+            0.005268632434308529,
+            0.02140381932258606,
+            -0.030529843643307686,
+            -0.025167128071188927,
+            0.0212470144033432,
+            -0.017123056575655937,
+            0.01154081430286169,
+            0.004876621067523956,
+            0.0032517341896891594,
+            0.020776601508259773,
+            0.07445079833269119,
+            -0.020070981234312057,
+            0.0043591661378741264,
+            -0.022768018767237663,
+            -0.02576298639178276,
+            -0.012614925391972065,
+            -0.022109439596533775,
+            0.005378395784646273,
+            0.013571432791650295,
+            -0.00024022944853641093,
+            0.020651157945394516,
+            0.01056862622499466,
+            -0.00644074659794569,
+            0.007216928992420435,
+            -0.01173681952059269,
+            -0.0007139506633393466,
+            -0.006585790775716305,
+            -0.0027087985072284937,
+            0.002058059675619006,
+            0.0015229640994220972,
+            0.002697037998586893,
+            0.02836594171822071,
+            0.012371878139674664,
+            0.013116699643433094,
+            -0.03327392414212227,
+            0.009815963916480541,
+            0.02151358313858509,
+            0.018095243722200394,
+            -0.03317984193563461,
+            -0.021388139575719833,
+            0.004543411545455456,
+            0.005017745308578014,
+            0.012928534299135208,
+            0.009196585975587368,
+            -0.006977802142500877,
+            0.02504168450832367,
+            -0.0355319082736969,
+            0.01814228482544422,
+            0.016527198255062103,
+            -0.011705459095537663,
+            -0.004931502975523472,
+            0.01986713521182537,
+            -0.0020090581383556128,
+            0.006860198453068733,
+            -0.005946812219917774,
+            0.0025794347748160362,
+            0.00653874920681119,
+            -0.021356778219342232,
+            -0.003559462958946824,
+            -0.012983416207134724,
+            -0.001520024030469358,
+            0.021419500932097435,
+            -0.017170097678899765,
+            0.008726172149181366,
+            -0.008655610494315624,
+            0.028836354613304138,
+            0.013414627872407436,
+            -0.029933987185359,
+            0.0007183607667684555,
+            0.023065946996212006,
+            0.01173681952059269,
+            -0.002052179304882884,
+            0.00922794733196497,
+            0.008851616643369198,
+            -0.016825126484036446,
+            6.958201265661046e-05,
+            0.012520842254161835,
+            0.018079563975334167,
+            0.007957830093801022,
+            0.006879799067974091,
+            0.006417225580662489,
+            0.019616248086094856,
+            -0.018785184249281883,
+            -0.02010234259068966,
+            0.00456693209707737,
+            0.010521584190428257,
+            -0.0065426696091890335,
+            -0.020996127277612686,
+            0.007060124538838863,
+            -0.016401754692196846,
+            0.037350840866565704,
+            0.009690520353615284,
+            -0.010803832672536373,
+            -0.0035790635738521814,
+            -0.031141381710767746,
+            -0.0027479995042085648,
+            -0.021983996033668518,
+            -0.010153094306588173,
+            0.019004710018634796,
+            -0.003353656968101859,
+            -0.00965131912380457,
+            -0.01906743273139,
+            -0.011015518568456173,
+            -0.008734012953937054,
+            0.0017326901433989406,
+            -0.008984900079667568,
+            0.01944376341998577,
+            0.01144673116505146,
+            -0.02844434417784214,
+            -0.031799960881471634,
+            -0.03361889347434044,
+            0.011054719798266888,
+            0.0025559139903634787,
+            -0.02913428284227848,
+            -0.011721138842403889,
+            -0.004555171821266413,
+            -0.004080838058143854,
+            -0.008000951260328293,
+            0.008145995438098907,
+            -0.003831910900771618,
+            0.006017374340444803,
+            -0.013830160722136497,
+            -0.005554800853133202,
+            -0.008130315691232681,
+            0.00788726843893528,
+            0.0149513129144907,
+            -0.005488158669322729,
+            -0.03093753568828106,
+            -0.017781633883714676,
+            0.00965131912380457,
+            -0.012826611287891865,
+            0.008145995438098907,
+            0.01684080809354782,
+            -0.004923662636429071,
+            0.00594289181753993,
+            0.01848725602030754,
+            0.018095243722200394,
+            -0.0030753291212022305,
+            0.014363295398652554,
+            -0.0011162522714585066,
+            -0.006385864689946175,
+            0.012607084587216377,
+            0.008953538723289967,
+            0.01487291045486927,
+            0.02507304586470127,
+            -0.010294217616319656,
+            -0.006087936460971832,
+            -0.0016954491147771478,
+            0.03274078667163849,
+            -0.023599082604050636,
+            -0.019804412499070168,
+            0.015688294544816017,
+            0.05425437167286873,
+            -0.0010662708664312959,
+            -0.0020913805346935987,
+            -0.022344646975398064,
+            -0.01102335937321186,
+            0.023489320650696754,
+            0.016793766990303993,
+            0.008036232553422451,
+            -0.010843033902347088,
+            -0.014386816881597042,
+            0.010545105673372746,
+            0.014418177306652069,
+            0.01922423578798771,
+            -0.024053815752267838,
+            0.021278375759720802,
+            -0.026750855147838593,
+            0.036127764731645584,
+            0.0075501385144889355,
+            -0.005974252708256245,
+            0.015539329499006271,
+            0.0036378654185682535,
+            -0.006526988931000233,
+            0.014324095100164413,
+            -0.01070190966129303,
+            0.011893624439835548,
+            0.006868038792163134,
+            -0.0015004234155640006,
+            0.021043168380856514,
+            -0.01108608115464449,
+            -0.0010045290691778064,
+            -0.0027048783376812935,
+            0.02090204507112503,
+            0.0031184502877295017,
+            -0.0038671919610351324,
+            0.01464554388076067,
+            0.0028773632366210222,
+            -0.004351325798779726,
+            -0.010074691846966743,
+            0.007350212894380093,
+            -0.03173723816871643,
+            -0.008138155564665794,
+            0.013924242928624153,
+            -0.005735125858336687,
+            0.002120781457051635,
+            -0.011015518568456173,
+            -0.002642156556248665,
+            -0.03609640523791313,
+            0.01787571795284748,
+            -0.000642898608930409,
+            0.017091695219278336,
+            0.0028656029608100653,
+            -0.00936123076826334,
+            -0.004809979349374771,
+            0.02297186478972435,
+            -0.009565076790750027,
+            -0.006930760573595762,
+            -0.018769502639770508,
+            -0.021497901529073715,
+            -0.025135766714811325,
+            0.021121570840477943,
+            -0.009580757468938828,
+            0.0018189327092841268,
+            -0.007706942968070507,
+            -0.0029126442968845367,
+            -0.00791470892727375,
+            0.012222914025187492,
+            0.018706781789660454,
+            -0.030576884746551514,
+            0.006617151666432619,
+            -0.006170258391648531,
+            -0.022109439596533775,
+            0.006456426810473204,
+            -0.009518035687506199,
+            0.01684080809354782,
+            -0.0009344570571556687,
+            -0.04123959317803383,
+            0.006303542293608189,
+            0.026484286412596703,
+            -0.017969800159335136,
+            -0.0015141437761485577,
+            0.010043330490589142,
+            0.008718332275748253,
+            -0.0026499966625124216,
+            -0.018894946202635765,
+            0.01811092346906662,
+            -0.017436664551496506,
+            0.018675420433282852,
+            0.026844937354326248,
+            -0.008757533505558968,
+            0.028993159532546997,
+            0.0001089056531782262,
+            0.01864405907690525,
+            0.0029832064174115658,
+            0.005641043186187744,
+            0.010082531720399857,
+            0.0038985528517514467,
+            0.020070981234312057,
+            0.00295772566460073,
+            0.007949990220367908,
+            0.038072142750024796,
+            -0.010788151994347572,
+            0.02182719111442566,
+            0.011305606923997402,
+            -0.00019184054690413177,
+            0.01615086756646633,
+            -0.007628540974110365,
+            -0.032489899545907974,
+            0.01566477306187153,
+            -0.027770083397626877,
+            -0.004668855108320713,
+            0.00922794733196497,
+            -0.0147396270185709,
+            -0.00934555009007454,
+            -0.014418177306652069,
+            0.009235787205398083,
+            0.012779570184648037,
+            -0.013908562250435352,
+            -0.0005326454411260784,
+            0.018973348662257195,
+            0.019381040707230568,
+            0.014527940191328526,
+            0.005982093047350645,
+            -0.005182390101253986,
+            0.013830160722136497,
+            0.0030204474460333586,
+            0.023646123707294464,
+            0.0016895688604563475,
+            0.005633202847093344,
+            -0.010592146776616573,
+            0.0145122604444623,
+            0.01508459635078907,
+            -0.0017875717021524906,
+            0.01092927623540163,
+            -0.004845260176807642,
+            -0.005817448254674673,
+            -0.01960056647658348,
+            -0.007636380847543478,
+            -0.009110343642532825,
+            0.006170258391648531,
+            0.04719816520810127,
+            0.0004782538453582674,
+            -0.002846002345904708,
+            -0.01765619032084942,
+            0.004559091757982969,
+            0.012685487046837807,
+            -0.0097454022616148,
+            -0.0034987013787031174,
+            -0.019365360960364342,
+            -0.022720977663993835,
+            0.0034144187811762094,
+            -0.0050843870267271996,
+            0.027001742273569107,
+            -0.003822110593318939,
+            0.02767600119113922,
+            -0.029102923348546028,
+            0.01956920698285103,
+            -0.016135187819600105,
+            -0.0097454022616148,
+            -0.0022560253273695707,
+            -0.012160192243754864,
+            -0.014331934973597527,
+            0.02300322614610195,
+            0.00936123076826334,
+            -0.032678067684173584,
+            -0.0016190068563446403,
+            0.01010605227202177,
+            -0.009306349791586399,
+            0.009008420631289482,
+            0.006860198453068733,
+            0.010803832672536373,
+            -0.0015798057429492474,
+            0.01190930511802435,
+            0.0013592992909252644,
+            -0.001451421994715929,
+            -9.536901052342728e-05,
+            -0.02294050343334675,
+            0.018346130847930908,
+            0.030482802540063858,
+            -0.05265496298670769,
+            -0.01914583332836628,
+            0.020118022337555885,
+            0.00569592509418726,
+            0.0005865469574928284,
+            0.013061817735433578,
+            -0.024367425590753555,
+            0.01027853786945343,
+            -0.018675420433282852,
+            -0.014316254295408726,
+            -0.022140800952911377,
+            -0.005460718180984259,
+            0.0151786794885993,
+            -0.027581918984651566,
+            0.025794345885515213,
+            0.017170097678899765,
+            -0.0015023835003376007,
+            0.006452506873756647,
+            0.0007472716388292611,
+            0.008851616643369198,
+            -0.03255262225866318,
+            -0.006617151666432619,
+            -0.0059664128348231316,
+            0.006009534001350403,
+            0.003655505832284689,
+            -0.02897747792303562,
+            0.014527940191328526,
+            -0.02074524015188217,
+            0.008263599127531052,
+            -0.029949666932225227,
+            0.011321287602186203,
+            0.03622184693813324,
+            -2.7471420253277756e-05,
+            0.009157384745776653,
+            -0.0031282505951821804,
+            0.04218042269349098,
+            0.026625411584973335,
+            0.0015709854196757078,
+            -0.008024472743272781,
+            -0.019694650545716286,
+            -0.002522593131288886,
+            0.00673867529258132,
+            -0.02918132394552231,
+            0.0012113150442019105,
+            -0.035092856734991074,
+            0.02209375984966755,
+            -0.006099696736782789,
+            -0.025512099266052246,
+            0.033556170761585236,
+            -0.020384589210152626,
+            0.002808761317282915,
+            -0.005347034893929958,
+            0.0027068383060395718,
+            0.030561204999685287,
+            0.0024990723468363285,
+            0.03005943074822426,
+            -0.03826030716300011,
+            -0.0013504790840670466,
+            0.0028244417626410723,
+            0.0038162304554134607,
+            -0.010435341857373714,
+            -0.015711814165115356,
+            0.016903528943657875,
+            -0.010200135409832,
+            0.004292524419724941,
+            0.02522985078394413,
+            -0.003655505832284689,
+            -0.013132380321621895,
+            0.021278375759720802,
+            -0.002808761317282915,
+            0.005417597014456987,
+            -0.005460718180984259,
+            0.000567926443181932,
+            0.023222751915454865,
+            0.02568458393216133,
+            -0.01136832870543003,
+            -0.011336968280375004,
+            -0.002781320596113801,
+            -0.00031654915073886514,
+            0.014104568399488926,
+            0.015264921821653843,
+            -0.008200877346098423,
+            -0.020541394129395485,
+            0.006817077286541462,
+            0.009808124043047428,
+            0.018549976870417595,
+            0.0012946174247190356,
+            -0.0255748201161623,
+            -0.008459605276584625,
+            -0.03027895651757717,
+            0.010153094306588173,
+            -0.016558559611439705,
+            -0.005601841956377029,
+            0.0013112779706716537,
+            -0.001341658877208829,
+            -0.009549396112561226,
+            0.014394656755030155,
+            0.015194360166788101,
+            -0.03901296854019165,
+            -0.015813738107681274,
+            -0.0007222809363156557,
+            -0.018502935767173767,
+            0.0024539909791201353,
+            -0.0011662337929010391,
+            0.001961036818102002,
+            0.0021658625919371843,
+            -0.003165491623803973,
+            0.004739417228847742,
+            -0.020070981234312057,
+            -0.0024324303958564997,
+            0.005629282910376787,
+            0.02113725244998932,
+            0.004535571206361055,
+            -0.0015014035161584616,
+            0.0022442650515586138,
+            0.0038907125126570463,
+            0.0017464105039834976,
+            0.0063897850923240185,
+            -0.00673867529258132,
+            0.015108116902410984,
+            -0.00020262086763978004,
+            0.015327643603086472,
+            -0.01906743273139,
+            0.007557978853583336,
+            -0.002369708614423871,
+            -0.005123588256537914,
+            0.015782376751303673,
+            -0.00037020572926849127,
+            -0.0020482593681663275,
+            -0.010294217616319656,
+            0.007605019956827164,
+            0.015586371533572674,
+            -0.022501451894640923,
+            -0.009063302539288998,
+            -0.014614183455705643,
+            0.02711150422692299,
+            0.0153746847063303,
+            0.008122474886476994,
+            -0.03223901242017746,
+            -0.0056488835252821445,
+            -0.012120991013944149,
+            -0.0059193712659180164,
+            -0.015296283178031445,
+            0.018863586708903313,
+            0.019349679350852966,
+            -0.0051784696988761425,
+            0.01619790866971016,
+            -0.0015592251438647509,
+            0.020855003967881203,
+            -0.00924362801015377,
+            0.006476027425378561,
+            -0.0007590319728478789,
+            -0.006315303035080433,
+            0.010168774053454399,
+            0.03521829843521118,
+            -0.017389623448252678,
+            0.008240078575909138,
+            -0.000719830859452486,
+            -0.002530433237552643,
+            -0.00306356861256063,
+            -0.004022036213427782,
+            0.006358424201607704,
+            0.014316254295408726,
+            0.009369071573019028,
+            0.0029342048801481724,
+            0.07056204229593277,
+            -0.003506541484966874,
+            0.020243465900421143,
+            0.006323142908513546,
+            -0.014574982225894928,
+            -0.005891930777579546,
+            0.0022109439596533775,
+            -0.028397301211953163,
+            0.031266823410987854,
+            0.020086660981178284,
+            0.0048413402400910854,
+            0.003782909596338868,
+            -0.014692584984004498,
+            0.023034587502479553,
+            0.004453248810023069,
+            0.0030616086442023516,
+            -0.0049119023606181145,
+            0.005864489823579788,
+            0.047762662172317505,
+            0.012403239496052265,
+            -0.007420775014907122,
+            -0.0022658256348222494,
+            0.014292733743786812,
+            0.0054254368878901005,
+            -0.011493772268295288,
+            -0.0003150791162624955,
+            -0.000643878651317209,
+            0.015037555247545242,
+            -0.001587645965628326,
+            0.021952634677290916,
+            -0.01114096213132143,
+            -0.01826772838830948,
+            0.0018375532235950232,
+            0.004394447430968285,
+            0.01571965590119362,
+            0.009486674331128597,
+            0.008318481035530567,
+            -0.005884090438485146,
+            0.004700215999037027,
+            0.006397624965757132,
+            0.005664564203470945,
+            0.00790294911712408,
+            -0.005852729547768831,
+            -0.008655610494315624,
+            -0.01826772838830948,
+            0.02637452259659767,
+            -0.002616675803437829,
+            0.003259574295952916,
+            -0.016856487840414047,
+            0.002438310533761978,
+            -0.0034712604247033596,
+            -0.007769664749503136,
+            0.010937117040157318,
+            0.017750274389982224,
+            -0.009392592124640942,
+            -0.008185196667909622,
+            0.0147239463403821,
+            -0.017028972506523132,
+            0.004457169212400913,
+            0.012622765265405178,
+            -0.004778618458658457,
+            0.006311382632702589,
+            0.003381097922101617,
+            -0.008898657746613026,
+            0.0012073949910700321,
+            0.0013671396300196648,
+            0.015539329499006271,
+            0.015249241143465042,
+            0.007075804751366377,
+            0.022031037136912346,
+            0.0016395874554291368,
+            0.0009217166807502508,
+            0.01654287800192833,
+            0.015609892085194588,
+            0.0064721074886620045,
+            0.022078078240156174,
+            -0.015445247292518616,
+            -0.016668323427438736,
+            0.003367377445101738,
+            0.006895479746162891,
+            0.0025441537145525217,
+            0.0030243676155805588,
+            0.0036751064471900463,
+            0.010545105673372746,
+            -0.014394656755030155,
+            -0.006946441251784563,
+            -0.009322029538452625,
+            0.018973348662257195,
+            0.022689616307616234,
+            -0.020588435232639313,
+            0.0003922563628293574,
+            0.004958943463861942,
+            0.017091695219278336,
+            -0.001889494713395834,
+            0.006558349821716547,
+            0.01760914921760559,
+            -0.006574030499905348,
+            -0.008389042690396309,
+            -0.02316003106534481,
+            -0.012426760047674179,
+            -0.0006556389853358269,
+            0.019898496568202972,
+            -0.011407529935240746,
+            0.01286581251770258,
+            -0.015633411705493927,
+            -0.0042768437415361404,
+            0.029918305575847626,
+            0.011877943761646748,
+            0.005739046260714531,
+            -0.01330486498773098,
+            -0.011674097739160061,
+            -0.013673355802893639,
+            0.014324095100164413,
+            0.010317739099264145,
+            0.003122370457276702,
+            -0.010121732950210571,
+            -0.0017013292526826262,
+            -0.005515599623322487,
+            -0.008961379528045654,
+            -0.00685235857963562,
+            0.011462411843240261,
+            -0.0027617199812084436,
+            0.003949514124542475,
+            -0.011987706646323204,
+            0.0009903187165036798,
+            -0.010748950764536858,
+            -0.006495628040283918,
+            -0.0015415846137329936,
+            -0.03481060639023781,
+            -0.0299026258289814,
+            -0.001531784306280315,
+            -0.005107908044010401,
+            0.005652803461998701,
+            0.004123959224671125,
+            0.011611375957727432,
+            0.0029185244347900152,
+            0.00937691144645214,
+            0.009251467883586884,
+            0.011062560603022575,
+            3.1728417525300756e-05,
+            -0.004202361684292555,
+            0.011995547451078892,
+            -0.013602794148027897,
+            -0.016244949772953987,
+            0.024916240945458412,
+            0.004751177504658699,
+            0.013461669906973839,
+            -0.028397301211953163,
+            0.011783861555159092,
+            -0.002575514605268836,
+            0.010654868558049202,
+            -0.031141381710767746,
+            -0.0039769550785422325,
+            0.0020443391986191273,
+            -0.0033242562785744667,
+            -0.0101844547316432,
+            -0.028052331879734993,
+            -0.0075383782386779785,
+            -0.017812995240092278,
+            -0.01630767248570919,
+            -0.024759436026215553,
+            0.0050138249062001705,
+            -0.012034748680889606,
+            0.021341098472476006,
+            -0.010286377742886543,
+            0.03725675866007805,
+            0.006040894892066717,
+            -0.0015719655202701688,
+            0.01361063402146101,
+            0.0030478881672024727,
+            -0.0020090581383556128,
+            0.016636962071061134,
+            -0.006942520849406719,
+            -0.0013749798526987433,
+            0.017969800159335136,
+            0.01102335937321186,
+            0.003243893850594759,
+            -0.003602584358304739,
+            0.0029949666932225227,
+            0.026562688872218132,
+            0.01814228482544422,
+            0.00426900340244174,
+            0.005840969271957874,
+            0.002998886862769723,
+            0.016417434439063072,
+            0.009808124043047428,
+            -0.014778828248381615,
+            -0.008145995438098907,
+            0.011611375957727432,
+            -0.01173681952059269,
+            -0.006832757964730263,
+            0.004845260176807642,
+            -0.0097454022616148,
+            -0.023097308352589607,
+            0.012748208828270435,
+            0.006256501190364361,
+            -0.0005919371615163982,
+            -0.019239917397499084,
+            -0.012920694425702095,
+            0.0017620910657569766,
+            0.01225427445024252,
+            0.016323352232575417,
+            -0.01914583332836628,
+            0.002738199196755886,
+            0.008153836242854595,
+            -0.006417225580662489,
+            -0.011580015532672405,
+            0.016950570046901703,
+            0.022313285619020462,
+            0.014888591133058071,
+            -0.04788810759782791,
+            -0.008945698849856853,
+            0.017546428367495537,
+            -0.012183712795376778,
+            -0.043967992067337036,
+            -0.0015455047832801938,
+            -0.0018395131919533014,
+            -0.025982512161135674,
+            0.0052490318194031715,
+            0.006182019133120775,
+            -0.014700425788760185,
+            0.016527198255062103,
+            0.03200380504131317,
+            0.00019159555085934699,
+            0.009102503769099712,
+            -0.011109601706266403,
+            -0.014426017180085182,
+            -0.009047621861100197,
+            0.008726172149181366,
+            -0.01175250019878149,
+            -0.009847325272858143,
+            0.00248927203938365,
+            -0.015555010177195072,
+            0.01848725602030754,
+            0.007569739129394293,
+            0.03150203078985214,
+            0.01132912840694189,
+            -0.00248927203938365,
+            -0.01760914921760559,
+            -0.020964767783880234,
+            -0.009533716365695,
+            0.003741748398169875,
+            0.007997031323611736,
+            0.007005242630839348,
+            0.010160934180021286,
+            0.013328385539352894,
+            0.022673936560750008,
+            -0.006252580787986517,
+            0.014825869351625443,
+            -3.0059307391638868e-05,
+            -0.0014680825406685472,
+            0.0025676742661744356,
+            -0.020149383693933487,
+            0.005143188871443272,
+            0.0018228527624160051,
+            -0.004904062021523714,
+            -0.007013082969933748,
+            -0.003083169227465987,
+            0.032066527754068375,
+            -0.016480157151818275,
+            0.015970543026924133,
+            -0.031078658998012543,
+            -0.009337710216641426,
+            0.001246596104465425,
+            0.02814641408622265,
+            0.012701167725026608,
+            -0.02841298282146454,
+            -0.011297767050564289,
+            -0.026515647768974304,
+            0.02278370037674904,
+            -0.014300573617219925,
+            0.004723736550658941,
+            0.015845099464058876,
+            0.01444953866302967,
+            0.01194850541651249,
+            -0.01165841706097126,
+            -0.00878889486193657,
+            0.011776020750403404,
+            0.021999677643179893,
+            0.007679502479732037,
+            0.0022266244050115347,
+            -0.0009261267841793597,
+            -0.001644487609155476,
+            -0.004304284695535898,
+            -0.017985479906201363,
+            0.006711234338581562,
+            -0.00819303747266531,
+            0.027127185836434364,
+            -0.007597180083394051,
+            0.016746724024415016,
+            -0.0062957024201750755,
+            -0.00582920853048563,
+            -0.02932244911789894,
+            -0.0203689094632864,
+            0.007605019956827164,
+            -0.006703393999487162,
+            0.012959894724190235,
+            -0.019365360960364342,
+            -0.0164958368986845,
+            0.010733271017670631,
+            -0.02010234259068966,
+            -0.002083540428429842,
+            -0.004719816613942385,
+            0.0022442650515586138,
+            0.00676611578091979,
+            0.00736197317019105,
+            -0.012826611287891865,
+            -0.006883719470351934,
+            0.016072465106844902,
+            -0.007663821801543236,
+            0.007640301249921322,
+            0.029871264472603798,
+            0.01768755167722702,
+            0.026844937354326248,
+            0.007048364263027906,
+            0.013751758262515068,
+            -0.010364780202507973,
+            -0.017546428367495537,
+            -0.023301154375076294,
+            0.01278741005808115,
+            -0.0025519938208162785,
+            0.03368161618709564,
+            0.00478645833209157,
+            -0.007820626720786095,
+            -0.0031027698423713446,
+            0.005417597014456987,
+            0.0008408643188886344,
+            -0.01414376962929964,
+            0.0029518455266952515,
+            -0.00551167968660593,
+            0.012826611287891865,
+            0.016033263877034187,
+            -0.01883222535252571,
+            0.02522985078394413,
+            -0.016856487840414047,
+            0.012779570184648037,
+            -0.003602584358304739,
+            0.01630767248570919,
+            0.009753242135047913,
+            -0.0003626105026341975,
+            -0.0015415846137329936,
+            -0.05394076183438301,
+            -0.010458862408995628,
+            0.0038456313777714968,
+            -0.004970703739672899,
+            0.000352075177943334,
+            -0.012638445943593979,
+            -0.038573917001485825,
+            0.0034810607321560383,
+            0.013391107320785522,
+            -0.011556494981050491,
+            0.013422468677163124,
+            -0.002016898477450013,
+            0.006617151666432619,
+            -0.007922549732029438,
+            -0.006260421127080917,
+            0.016480157151818275,
+            0.02573162503540516,
+            -0.021121570840477943,
+            -0.022344646975398064,
+            0.005778247490525246,
+            -0.005543040577322245,
+            -0.014684745110571384,
+            -0.005021665245294571,
+            -0.01598622277379036,
+            0.0025069124531000853,
+            0.023567723110318184,
+            -0.01426137238740921,
+            -0.01776595413684845,
+            0.010145253501832485,
+            0.007757904473692179,
+            -0.024853520095348358,
+            -0.002355988137423992,
+            -0.009815963916480541,
+            -0.010803832672536373,
+            0.018393171951174736,
+            -0.01362631469964981,
+            0.008906497620046139,
+            -0.013124539516866207,
+            0.010396140627563,
+            -0.02825617790222168,
+            -0.01226995512843132,
+            -0.0019502564100548625,
+            -0.008882977068424225,
+            0.028083693236112595,
+            -0.010772472247481346,
+            -0.010200135409832,
+            0.020070981234312057,
+            -0.008663450367748737,
+            0.012567883357405663,
+            0.007855907082557678,
+            0.028648190200328827,
+            -0.0025853149127215147,
+            0.009141704998910427,
+            0.0028930436819791794,
+            0.01390072237700224,
+            0.0011936745140701532,
+            0.035437826067209244,
+            -0.00957291666418314,
+            -0.00030307378619909286,
+            -0.008835935965180397,
+            -0.0017140696290880442,
+            0.0010427501983940601,
+            0.010991998016834259,
+            0.014151609502732754,
+            0.012693327851593494,
+            -0.006244740914553404,
+            -0.0077853454276919365,
+            -0.015500128269195557,
+            -0.004527730867266655,
+            -0.004888381343334913,
+            0.007428614888340235,
+            0.012442439794540405,
+            -0.020039619877934456,
+            0.005358795169740915,
+            -0.012144511565566063,
+            0.0011946546146646142,
+            -0.01587645895779133,
+            -0.0022560253273695707,
+            -0.013634154573082924,
+            -0.004080838058143854,
+            -0.004331725183874369,
+            0.003790749702602625,
+            0.008749693632125854,
+            -0.016401754692196846,
+            0.0199455376714468,
+            0.008153836242854595,
+            -0.0004074467869941145,
+            -0.0028107212856411934,
+            -0.024931922554969788,
+            -0.014010485261678696,
+            0.00014859679504297674,
+            0.016668323427438736,
+            0.0050138249062001705,
+            -0.0051784696988761425,
+            0.002026698552072048,
+            -0.012826611287891865,
+            0.005280392710119486,
+            -0.0009040761506184936,
+            0.0022285846062004566,
+            -0.0030067269690334797,
+            0.0057312059216201305,
+            0.006762195844203234,
+            0.003700587200000882,
+            0.012959894724190235,
+            0.01487291045486927,
+            -0.011760340072214603,
+            -0.00966699980199337,
+            0.012285635806620121,
+            0.015217880718410015,
+            -0.008914338424801826,
+            0.01941240206360817,
+            -0.02878931351006031,
+            0.002820521593093872,
+            0.005233351606875658,
+            -0.0042533231899142265,
+            -0.0021854632068425417,
+            7.748349162284285e-05,
+            0.028475703671574593,
+            -0.0010221695993095636,
+            -0.010388300754129887,
+            -0.0009854185627773404,
+            0.01707601360976696,
+            -0.0006600490887649357,
+            0.006123217288404703,
+            0.023724526166915894,
+            0.00023790188424754888,
+            0.009768922813236713,
+            0.009776762686669827,
+            -0.019506484270095825,
+            -0.003145891008898616,
+            0.004633574280887842,
+            -0.012442439794540405,
+            0.018393171951174736,
+            0.014841550029814243,
+            0.008389042690396309,
+            0.0038005500100553036,
+            0.007510937284678221,
+            0.020086660981178284,
+            0.006374104414135218,
+            -0.002336387522518635,
+            -0.0002511322672944516,
+            0.0066445921547710896,
+            -0.011721138842403889,
+            0.017405303195118904,
+            -0.004363086074590683,
+            -0.015437406487762928,
+            0.006679873447865248,
+            0.011674097739160061,
+            -0.006773956120014191,
+            -0.004033796954900026,
+            0.0022540653590112925,
+            0.005711605306714773,
+            0.005464638117700815,
+            -0.003888752544298768,
+            -0.006574030499905348,
+            0.0012505161575973034,
+            0.017812995240092278,
+            0.010639187879860401,
+            0.016433116048574448,
+            -0.014990514144301414,
+            -0.011697618290781975,
+            0.0023050266318023205,
+            -0.018016841262578964,
+            0.002996926661580801,
+            -0.011517293751239777,
+            0.006299622356891632,
+            -0.007420775014907122,
+            0.02568458393216133,
+            0.019616248086094856,
+            0.007573659066110849,
+            0.011932825669646263,
+            -0.015539329499006271,
+            0.016715364530682564,
+            -0.00913386419415474,
+            0.012120991013944149,
+            -0.018847905099391937,
+            -0.0017346502281725407,
+            0.005296073388308287,
+            0.027628960087895393,
+            0.010302058421075344,
+            0.006064415443688631,
+            -0.001778751495294273,
+            -0.005182390101253986,
+            -0.008318481035530567,
+            -0.009619958698749542,
+            -0.014755306765437126,
+            -0.026233399286866188,
+            -0.00022810160589870065,
+            0.016339033842086792,
+            -0.00195907661691308,
+            0.0022521051578223705,
+            0.014590661972761154,
+            0.013187261298298836,
+            -0.0024128297809511423,
+            -0.013908562250435352,
+            -0.007252209819853306,
+            0.010835194028913975,
+            0.005119668319821358,
+            -0.008357682265341282,
+            -0.009141704998910427,
+            0.008138155564665794,
+            -0.013273504562675953,
+            0.016433116048574448,
+            -0.016182228922843933,
+            -0.009949248284101486,
+            -0.013649835251271725,
+            -0.0005929171456955373,
+            -0.008263599127531052,
+            -0.031454991549253464,
+            0.012152351438999176,
+            0.005707685369998217,
+            -0.0006365284207277,
+            -0.025715945288538933,
+            0.02369316667318344,
+            -0.008569368161261082,
+            0.03656681999564171,
+            -0.01593918167054653,
+            0.015633411705493927,
+            0.0010623506968840957,
+            0.011493772268295288,
+            -0.003392858197912574,
+            0.0029479253571480513,
+            0.007953910157084465,
+            0.002550033852458,
+            0.003055728506296873,
+            -0.00539799639955163,
+            -0.0033066156320273876,
+            0.008898657746613026,
+            -0.006448586471378803,
+            0.008490965701639652,
+            -0.014849389903247356,
+            -0.02320707216858864,
+            -0.01144673116505146,
+            0.009502355009317398,
+            0.014104568399488926,
+            0.00861640926450491,
+            -0.014473059214651585,
+            0.00882809516042471,
+            -0.02435174584388733,
+            -0.0023461878299713135,
+            -0.0031282505951821804,
+            0.004782538395375013,
+            -0.017201457172632217,
+            0.016605600714683533,
+            0.02193695493042469,
+            -3.518914309097454e-05,
+            -0.0007879427867010236,
+            0.01382231991738081,
+            0.010160934180021286,
+            0.009423952549695969,
+            -0.010011970065534115,
+            0.0017826715484261513,
+            -8.403743413509801e-05,
+            0.02044731192290783,
+            -0.002397149335592985,
+            -0.009831644594669342,
+            -0.0035379023756831884,
+            -0.01399480551481247,
+            0.009698361158370972,
+            -0.029275408014655113,
+            -0.0052490318194031715,
+            -0.006523068994283676,
+            0.017170097678899765,
+            0.00684451824054122,
+            0.021639026701450348,
+            0.0065622697584331036,
+            0.005523439962416887,
+            -8.844755939207971e-05,
+            -0.008240078575909138,
+            0.012097470462322235,
+            -0.00809895433485508,
+            -0.007201248314231634,
+            0.006013453938066959,
+            0.004492450039833784,
+            0.016856487840414047,
+            0.008067593909800053,
+            -0.009180906228721142,
+            0.030106471851468086,
+            0.014904271811246872,
+            -0.013939923606812954,
+            0.01814228482544422,
+            -0.01826772838830948,
+            0.020776601508259773,
+            0.01853429712355137,
+            0.010239336639642715,
+            -0.015304123051464558,
+            -0.015680454671382904,
+            0.01784435659646988,
+            -0.010466703213751316,
+            -0.004657094832509756,
+            0.0018052122322842479,
+            -0.017938438802957535,
+            8.092890311672818e-06,
+            -0.011972026899456978,
+            0.015429566614329815,
+            -0.0021991836838424206,
+            -0.00012042098387610167,
+            -0.020776601508259773,
+            -0.00043831768562085927,
+            0.016903528943657875,
+            0.01615086756646633,
+            -0.0033340565860271454,
+            -0.04478337615728378,
+            -0.006178098730742931,
+            0.009855165146291256,
+            -0.00706404447555542,
+            -0.025982512161135674,
+            0.01803252287209034,
+            0.0034104986116290092,
+            -0.019428081810474396,
+            -0.01184658333659172,
+            -0.018502935767173767,
+            -0.019396722316741943,
+            -0.008820255286991596,
+            -0.004563012160360813,
+            -0.015272761695086956,
+            -0.003884832374751568,
+            0.008224397897720337,
+            -0.007213008590042591,
+            -0.006064415443688631,
+            0.004061237443238497,
+            -0.007354132831096649,
+            -0.0003518301818985492,
+            -0.010866554453969002,
+            -0.005629282910376787,
+            0.011133122257888317,
+            0.002452031010761857,
+            -0.01508459635078907,
+            -0.015366844832897186,
+            -0.011564334854483604,
+            -0.010537264868617058,
+            0.004139639902859926,
+            -0.009212266653776169,
+            -0.005080467090010643,
+            -0.020149383693933487,
+            0.0006615191232413054,
+            0.013469509780406952,
+            0.0104118213057518,
+            0.0006154578295536339,
+            0.004931502975523472,
+            0.022313285619020462,
+            -0.006189859006553888,
+            -0.0006242780946195126,
+            -9.363864592160098e-06,
+            -0.002955765463411808,
+            0.02687629871070385,
+            0.004312125034630299,
+            0.01286581251770258,
+            -0.005562641192227602,
+            0.009494515135884285,
+            -0.01975737139582634,
+            -0.0023755887523293495,
+            -0.010850873775780201,
+            -0.019773053005337715,
+            -0.011979866772890091,
+            -5.2125258662272245e-05,
+            0.004743337165564299,
+            -0.013069658540189266,
+            0.0021541023161262274,
+            -0.00467277504503727,
+            0.006734754890203476,
+            -0.01508459635078907,
+            -0.005782167427241802,
+            -0.007134606596082449,
+            -0.010662708431482315,
+            -0.02416357956826687,
+            0.018283409997820854,
+            0.02889907732605934,
+            0.001601366326212883,
+            -0.007554058451205492,
+            -0.01391640305519104,
+            -0.008177356794476509,
+            -0.004041636828333139,
+            -0.018048202618956566,
+            -0.008702651597559452,
+            -0.027926888316869736,
+            -0.013641994446516037,
+            -0.0003746158326976001,
+            0.010772472247481346,
+            0.0036065042950212955,
+            -0.018126605078577995,
+            0.005107908044010401,
+            -0.025527779012918472,
+            -0.025668902322649956,
+            -0.019428081810474396,
+            -0.017295541241765022,
+            0.0025676742661744356,
+            0.003888752544298768,
+            0.0021149013191461563,
+            0.017358262091875076,
+            0.020227786153554916,
+            0.004813899286091328,
+            -0.002424590289592743,
+            0.008592888712882996,
+            -0.014974833466112614,
+            0.0046766954474151134,
+            0.0023305073846131563,
+            -0.013422468677163124,
+            0.02358340285718441,
+            -0.011917144991457462,
+            -0.00353594240732491,
+            0.0008570348145440221,
+            0.0034575401805341244,
+            -0.028193457052111626,
+            -0.002152142347767949,
+            -0.0164958368986845,
+            0.02270529791712761,
+            -0.029651738703250885,
+            -0.0021443020086735487,
+            -0.00416316045448184,
+            -0.007895108312368393,
+            0.026578368619084358,
+            0.021356778219342232,
+            -0.0004645334556698799,
+            0.008984900079667568,
+            0.010709750466048717,
+            -0.0013063778169453144,
+            0.01869110018014908,
+            -0.017483705654740334,
+            -0.014151609502732754,
+            0.022234883159399033,
+            0.008757533505558968,
+            0.0036163046024739742,
+            -0.011242885142564774,
+            0.017562108114361763,
+            0.017468025907874107,
+            0.0059193712659180164,
+            -0.011485932394862175,
+            -0.016244949772953987,
+            0.0007933329907245934,
+            0.019773053005337715,
+            -0.012403239496052265,
+            0.0027440793346613646,
+            -0.010631348006427288,
+            0.008091114461421967,
+            -0.014708265662193298,
+            0.017750274389982224,
+            -0.027503516525030136,
+            -0.005060866475105286,
+            -0.0016092065488919616,
+            0.01795412041246891,
+            -0.008012712001800537,
+            0.015711814165115356,
+            -0.00696604186668992,
+            -0.009470994584262371,
+            0.014888591133058071,
+            -0.0005556760588660836,
+            -0.004739417228847742,
+            0.022909143939614296,
+            0.00653874920681119,
+            -0.0009614078444428742,
+            0.004884461406618357,
+            -0.018424533307552338,
+            0.025010323151946068,
+            -0.010074691846966743,
+            0.003000846831128001,
+            0.0012524762423709035,
+            0.0013710596831515431,
+            0.026844937354326248,
+            0.02634316310286522,
+            0.0057429661974310875,
+            0.008585048839449883,
+            -0.01351655088365078,
+            0.02519848942756653,
+            0.00467277504503727,
+            0.0012720768572762609,
+            -0.0009893386159092188,
+            -0.0024167499504983425,
+            -0.0017454305198043585,
+            -0.009180906228721142,
+            0.015743175521492958,
+            -0.01213667169213295,
+            -0.025935471057891846,
+            -0.009322029538452625,
+            -0.004492450039833784,
+            -0.017107374966144562,
+            0.020055299624800682,
+            0.0004699236014857888,
+            -0.003992635756731033,
+            0.011681938543915749,
+            0.007428614888340235,
+            0.0032360537443310022,
+            -0.008655610494315624,
+            0.0038671919610351324,
+            -0.007130686193704605,
+            -0.013046137988567352,
+            0.012450280599296093,
+            -0.019208556041121483,
+            -0.008984900079667568,
+            -0.008553687483072281,
+            0.009063302539288998,
+            0.010443182662129402,
+            0.00928282830864191,
+            0.015656933188438416,
+            0.004441488534212112,
+            0.00819303747266531,
+            0.012489481829106808,
+            -0.024931922554969788,
+            0.01707601360976696,
+            0.021764470264315605,
+            -0.0005096147651784122,
+            0.001671928446739912,
+            0.00045350813888944685,
+            -0.03446563705801964,
+            0.00027293790481053293,
+            0.0014034006744623184,
+            0.0077853454276919365,
+            -0.013406787998974323,
+            0.0022677856031805277,
+            0.022673936560750008,
+            -0.011109601706266403,
+            0.020996127277612686,
+            0.005970332771539688,
+            0.002479471731930971,
+            0.03343072906136513,
+            0.012230753898620605,
+            0.016339033842086792,
+            -0.0068915593437850475,
+            -0.026625411584973335,
+            0.0054371971637010574,
+            -0.018518615514039993,
+            0.009714040905237198,
+            0.0014386816183105111,
+            0.013148060999810696,
+            -0.01792275905609131,
+            -0.006025214213877916,
+            0.0027715202886611223,
+            -0.011039039120078087,
+            0.01654287800192833,
+            0.0014043806586414576,
+            0.019631927832961082,
+            0.001000609016045928,
+            -0.0028264017309993505,
+            -0.012967735528945923,
+            -0.01404968649148941,
+            0.0029773262795060873,
+            -0.009659159928560257,
+            -0.01699761115014553,
+            0.004966783802956343,
+            -0.008342001587152481,
+            0.007091485429555178,
+            -0.01587645895779133,
+            0.02044731192290783,
+            0.0005806668195873499,
+            0.00853800680488348,
+            0.015523649752140045,
+            0.0010319699067622423,
+            0.017750274389982224,
+            0.010341259650886059,
+            -0.0075501385144889355,
+            0.0062957024201750755,
+            0.0002594625111669302,
+            0.0029636058025062084,
+            0.0017895317869260907,
+            -0.01219155266880989,
+            0.0058135283179581165,
+            -0.004159240517765284,
+            0.00015165939112193882,
+            -0.006479947362095118,
+            -0.004092598333954811,
+            0.0034046184737235308,
+            -0.013406787998974323,
+            -0.00539015606045723,
+            0.002242304850369692,
+            0.001080971327610314,
+            0.02148222178220749,
+            -0.0017905118875205517,
+            0.00878889486193657,
+            0.02228192426264286,
+            -0.015570690855383873,
+            0.006703393999487162,
+            0.010059011168777943,
+            -0.004892301745712757,
+            -0.011156642809510231,
+            0.004661014769226313,
+            0.016166547313332558,
+            -0.01587645895779133,
+            -0.01131344772875309,
+            0.011501613073050976,
+            -0.021921275183558464,
+            0.004390527028590441,
+            -0.003963234834372997,
+            -0.007338452618569136,
+            0.02143518067896366,
+            -0.0012436560355126858,
+            0.032333094626665115,
+            -0.004770778119564056,
+            -0.015460927970707417,
+            -0.01930263824760914,
+            0.0012554163113236427,
+            -0.025512099266052246,
+            -0.005637123249471188,
+            -0.009188746102154255,
+            -0.006848438177257776,
+            0.00958859734237194,
+            0.0006865098839625716,
+            0.004919742234051228,
+            0.005950732156634331,
+            -0.004390527028590441,
+            0.004012236371636391,
+            -0.0031400108709931374,
+            0.007883348502218723,
+            -0.02143518067896366,
+            0.018628379330039024,
+            0.014120249077677727,
+            0.0005228451336733997,
+            -0.019741691648960114,
+            -0.008945698849856853,
+            0.020086660981178284,
+            0.04537923261523247,
+            -0.023222751915454865,
+            -0.0033085758332163095,
+            0.018330451101064682,
+            0.0038691519293934107,
+            0.0014475019415840507,
+            -0.00924362801015377,
+            -0.005570481065660715,
+            -0.04202361777424812,
+            0.0016415475402027369,
+            0.001603326410986483,
+            -0.009306349791586399,
+            0.007048364263027906,
+            -0.017812995240092278,
+            -0.007275730837136507,
+            -0.012638445943593979,
+            -0.020274827256798744,
+            -0.004610053263604641,
+            -0.010341259650886059,
+            -0.006503468379378319,
+            -0.011415370739996433,
+            0.013046137988567352,
+            -0.005774327088147402,
+            0.010082531720399857,
+            -0.004104358609765768,
+            0.01768755167722702,
+            -0.0013240183470770717,
+            0.012544362805783749,
+            -0.0005821368540637195,
+            0.009768922813236713,
+            0.00665243249386549,
+            -0.006789636332541704,
+            -0.008255759254097939,
+            -0.007667741738259792,
+            0.018659740686416626,
+            0.009768922813236713,
+            0.010498063638806343,
+            0.0014337814645841718,
+            -0.012552203610539436,
+            -0.011674097739160061,
+            5.987360782455653e-05,
+            -0.010333418846130371,
+            0.0006835698150098324,
+            0.006742595229297876,
+            0.026970380917191505,
+            -0.0033340565860271454,
+            -0.025527779012918472,
+            0.004649254493415356,
+            0.009847325272858143,
+            0.020917724817991257,
+            -0.013148060999810696,
+            0.007354132831096649,
+            -0.010498063638806343,
+            0.013108859769999981,
+            0.026311801746487617,
+            -0.009212266653776169,
+            0.018283409997820854,
+            -0.006205539684742689,
+            0.013947763480246067,
+            0.006836677901446819,
+            -0.00416316045448184,
+            -0.02963605709373951,
+            -0.008302800357341766,
+            0.0016131267184391618,
+            0.0036319850478321314,
+            -0.009369071573019028,
+            0.030874812975525856,
+            0.010482383891940117,
+            0.016088144853711128,
+            -0.009580757468938828,
+            0.010403981432318687,
+            0.00539015606045723,
+            0.019694650545716286,
+            0.012238594703376293,
+            0.00021413620561361313,
+            0.0019806374330073595,
+            -0.008498805575072765,
+            -0.045504678040742874,
+            0.00957291666418314,
+            0.006668113172054291,
+            0.02538665570318699,
+            -0.04196089506149292,
+            0.01569613441824913,
+            -0.010200135409832,
+            -0.0059899333864450455,
+            -0.003700587200000882,
+            0.007667741738259792,
+            -0.003696667030453682,
+            0.0290245208889246,
+            -0.005790007766336203,
+            0.0019355560652911663,
+            0.016684003174304962,
+            0.014073207043111324,
+            -0.007683422416448593,
+            0.01567261293530464,
+            0.023018905892968178,
+            0.01898903027176857,
+            0.013979124836623669,
+            -0.00748349679633975,
+            0.025621861219406128,
+            0.0006811197381466627,
+            -0.003218413097783923,
+            -0.008240078575909138,
+            -0.016166547313332558,
+            0.0024128297809511423,
+            -0.013116699643433094,
+            0.003659426001831889,
+            -0.00561752263456583,
+            -0.004633574280887842,
+            0.010905755683779716,
+            -0.010991998016834259,
+            0.0017454305198043585,
+            0.0018081523012369871,
+            0.007040523923933506,
+            0.0003800060076173395,
+            0.02703310176730156,
+            -0.032521262764930725,
+            0.014081047847867012,
+            -0.03258398175239563,
+            -0.01657423935830593,
+            0.006942520849406719,
+            -0.0044179679825901985,
+            0.00687195872887969,
+            -0.0195221658796072,
+            -0.017358262091875076,
+            0.00591545132920146,
+            0.01829908974468708,
+            0.01502971537411213,
+            0.00716988742351532,
+            0.006091856397688389,
+            0.004884461406618357,
+            -0.010059011168777943,
+            -0.006299622356891632,
+            -0.009557236917316914,
+            -0.00884377583861351,
+            -0.005049106199294329,
+            -0.02010234259068966,
+            0.015453087165951729,
+            -0.0070562041364610195,
+            -0.007922549732029438,
+            0.016746724024415016,
+            -0.01026285719126463,
+            -0.018424533307552338,
+            0.005539120174944401,
+            -0.0038926727138459682,
+            -0.005025585647672415,
+            -0.016213588416576385,
+            0.00922794733196497,
+            -0.0032752547413110733,
+            0.0039240336045622826,
+            -0.01309317909181118,
+            0.026672452688217163,
+            -0.006785716395825148,
+            0.001506303553469479,
+            0.011924984864890575,
+            -0.009408272802829742,
+            -0.024853520095348358,
+            -0.0021991836838424206,
+            -0.014896431006491184,
+            0.0027323190588504076,
+            -0.0028773632366210222,
+            -0.01673104427754879,
+            -0.008443924598395824,
+            0.0035967042203992605,
+            -0.02698606066405773,
+            -0.011384009383618832,
+            0.0099570881575346,
+            0.011783861555159092,
+            0.009431793354451656,
+            0.011932825669646263,
+            -0.0021011808421462774,
+            0.013148060999810696,
+            -0.029008839279413223,
+            -0.014496579766273499,
+            -0.01362631469964981,
+            -0.009549396112561226,
+            0.01412808895111084,
+            0.0017954119248315692,
+            0.007416854612529278,
+            -0.004480689764022827,
+            -0.005252952221781015,
+            -0.0011201724410057068,
+            0.011776020750403404,
+            0.028804993256926537,
+            0.0003814760420937091,
+            -0.003884832374751568,
+            -0.001314218039624393,
+            -0.011423210613429546,
+            0.028381621465086937,
+            -0.0041121989488601685,
+            -0.0072090886533260345,
+            0.03449700027704239,
+            0.01933399960398674,
+            0.027362391352653503,
+            -0.0001465142413508147,
+            -0.006025214213877916,
+            -0.0005091247148811817,
+            0.004931502975523472,
+            0.004974624142050743,
+            -0.01226995512843132,
+            -0.00842040404677391,
+            -0.004751177504658699,
+            0.03465380147099495,
+            0.0022952265571802855,
+            0.011133122257888317,
+            -0.0307493694126606,
+            0.015225720591843128,
+            -0.003577103605493903,
+            -0.007546218577772379,
+            0.003269374603405595,
+            0.036692261695861816,
+            0.02576298639178276,
+            0.0069738817401230335,
+            0.009447473101317883,
+            8.630374941276386e-05,
+            0.0398910753428936,
+            -0.02438310533761978,
+            -0.020196424797177315,
+            -0.02273665741086006,
+            -0.021717429161071777,
+            0.0016601680545136333,
+            0.007271810434758663,
+            0.003628065111115575,
+            -0.0009770882315933704,
+            0.0015259041683748364,
+            0.028585467487573624,
+            0.0015788257587701082,
+            -0.010341259650886059,
+            -0.0014308413956314325,
+            -0.034089308232069016,
+            0.013618473894894123,
+            -0.017389623448252678,
+            -0.010200135409832,
+            -0.016511518508195877,
+            -0.013006936758756638,
+            -0.025982512161135674,
+            -0.010607827454805374,
+            -0.0009599378099665046,
+            0.015970543026924133,
+            0.010004129260778427,
+            -0.0019120353972539306,
+            0.005613602697849274,
+            -0.023834289982914925,
+            -0.006703393999487162,
+            0.020917724817991257,
+            0.01567261293530464,
+            0.015790216624736786,
+            0.0046531748957931995,
+            0.01622927002608776,
+            -0.0003020937438122928,
+            -0.015923500061035156,
+            -0.0020913805346935987,
+            0.002767600119113922,
+            0.006413305643945932,
+            0.017091695219278336,
+            0.012222914025187492,
+            -0.018769502639770508,
+            0.0071267662569880486,
+            -0.012795249931514263,
+            0.03139226883649826,
+            -0.006197699345648289,
+            0.01205826923251152,
+            0.009682680480182171,
+            0.0032830950804054737,
+            -0.012489481829106808,
+            -0.006158498115837574,
+            -0.00479429867118597,
+            0.01569613441824913,
+            -0.00831064023077488,
+            -0.011062560603022575,
+            -0.005676324479281902,
+            0.009729721583425999,
+            -0.005535200238227844,
+            -0.0010349099757149816,
+            -0.015609892085194588,
+            -0.014347615651786327,
+            0.0030753291212022305,
+            -0.012826611287891865,
+            -0.0010711710201576352,
+            0.019036071375012398,
+            -0.005033425521105528,
+            -0.0004390527028590441,
+            0.014684745110571384,
+            0.016589920967817307,
+            0.004649254493415356,
+            0.018926307559013367,
+            -0.009549396112561226,
+            0.0022932663559913635,
+            -0.021623345091938972,
+            -0.0024716316256672144,
+            -0.032364457845687866,
+            0.019647609442472458,
+            0.0017836516490206122,
+            -0.020070981234312057,
+            0.010944956913590431,
+            -0.0016239070100709796,
+            -0.009165225550532341,
+            -0.0099727688357234,
+            0.002452031010761857,
+            -0.024179259315133095,
+            0.024147899821400642,
+            -0.016048945486545563,
+            0.008906497620046139,
+            0.01161921676248312,
+            0.041333675384521484,
+            0.0038573916535824537,
+            -0.00747565645724535,
+            0.021152932196855545,
+            -0.0002717128663789481,
+            0.02982422336935997,
+            -0.001121152425184846,
+            -0.012081789784133434,
+            -0.022689616307616234,
+            -0.005237271543592215,
+            -0.015202200040221214,
+            -0.014112408272922039,
+            0.0147239463403821,
+            -0.0195221658796072,
+            -0.015217880718410015,
+            -0.010004129260778427,
+            -0.019741691648960114,
+            -0.011509452946484089,
+            0.025982512161135674,
+            0.015758855268359184,
+            -0.011705459095537663,
+            -0.006425065919756889,
+            -0.0055077592842280865,
+            -0.02711150422692299,
+            0.021952634677290916,
+            0.008012712001800537,
+            -0.01983577385544777,
+            -0.0032772149424999952,
+            -0.009768922813236713,
+            0.004378766752779484,
+            0.01588430069386959,
+            -0.0001214010117109865,
+            0.05789223685860634,
+            -0.008804574608802795,
+            -0.0015945062041282654,
+            -0.020008258521556854,
+            0.009298508986830711,
+            -0.01575101539492607,
+            -0.006374104414135218,
+            -0.026672452688217163,
+            -0.008255759254097939,
+            -0.005119668319821358,
+            -0.021105891093611717,
+            -0.0022481849882751703,
+            -0.008287119679152966,
+            0.0057547264732420444,
+            -0.01280309073626995,
+            0.0008050933247432113,
+            -0.01665264181792736,
+            -0.00624866085126996,
+            -0.012089629657566547,
+            -0.012371878139674664,
+            0.01925559714436531,
+            -0.028303219005465508,
+            -0.0033869780600070953,
+            0.008153836242854595,
+            0.01353223156183958,
+            0.006268261466175318,
+            -0.021043168380856514,
+            -0.012614925391972065,
+            -0.011917144991457462,
+            -0.018894946202635765,
+            0.010733271017670631,
+            -0.010741110891103745,
+            0.013140220195055008,
+            -0.03186268359422684,
+            -0.012607084587216377,
+            -0.0019522164948284626,
+            0.019976899027824402,
+            0.01753074675798416,
+            -0.0004733536916319281,
+            -0.0059193712659180164,
+            -0.014002645388245583,
+            0.02278370037674904,
+            -0.013877201825380325,
+            -0.032678067684173584,
+            -0.0013494990998879075,
+            0.002218784298747778,
+            -0.00104079011362046,
+            -0.007816705852746964,
+            0.0027538796421140432,
+            0.004214121960103512,
+            0.004092598333954811,
+            -0.007546218577772379,
+            -0.0016464476939290762,
+            -0.01508459635078907,
+            0.0047668581828475,
+            -3.659793583210558e-05,
+            0.029730141162872314,
+            -0.0035339822061359882,
+            0.008992739953100681,
+            -0.02182719111442566,
+            0.0007546218694187701,
+            -0.01317158155143261,
+            -0.005782167427241802,
+            0.000176037588971667,
+            0.003330136416479945,
+            0.01956920698285103,
+            -0.025794345885515213,
+            -0.013038297183811665,
+            0.01205826923251152,
+            -0.019929856061935425,
+            -0.03572007268667221,
+            0.002736239228397608,
+            -0.029463572427630424,
+            -0.0013053978327661753,
+            -0.003777029225602746,
+            -0.015703974291682243,
+            -0.026217719539999962,
+            0.0060722557827830315,
+            -0.005366635508835316,
+            -0.005786087363958359,
+            0.003516341792419553,
+            -0.015233561396598816,
+            -0.015531489625573158,
+            0.005151029210537672,
+            0.007346292492002249,
+            -0.006503468379378319,
+            0.008561528287827969,
+            -0.0238186102360487,
+            0.011274246498942375,
+            -0.008145995438098907,
+            0.0037358682602643967,
+            0.005135348532348871,
+            0.026123635470867157,
+            0.008561528287827969,
+            -0.004000475630164146,
+            0.009306349791586399,
+            0.005472478456795216,
+            -0.010333418846130371,
+            0.011070400476455688,
+            0.02400677464902401,
+            0.0005184350302442908,
+            0.0032340935431420803,
+            -0.002973406109958887,
+            0.005746886134147644,
+            -0.001685648807324469,
+            -0.007612860295921564,
+            0.020933406427502632,
+            0.01593918167054653,
+            -0.019239917397499084,
+            0.0062369005754590034,
+            -0.020039619877934456,
+            0.010976317338645458,
+            -0.028742272406816483,
+            0.0005703765200451016,
+            0.005840969271957874,
+            -0.0050020646303892136,
+            0.011979866772890091,
+            0.006860198453068733,
+            0.03151771053671837,
+            0.026280440390110016,
+            -0.0018875346286222339,
+            0.0048178196884691715,
+            -0.02002394013106823,
+            0.01280309073626995,
+            0.009118183515965939,
+            0.010066851042211056,
+            -0.010694069787859917,
+            0.00791470892727375,
+            -0.00878889486193657,
+            0.014606342650949955,
+            -0.00260295532643795,
+            -0.014214331284165382,
+            0.006350583862513304,
+            -0.006287862081080675,
+            0.014441697858273983,
+            0.014081047847867012,
+            -0.013148060999810696,
+            -0.026923339813947678,
+            -0.006162418518215418,
+            0.018095243722200394,
+            -0.023912692442536354,
+            0.02443014644086361,
+            0.006464267149567604,
+            0.02265825681388378,
+            -0.009988449513912201,
+            -0.0030655288137495518,
+            -0.001984557369723916,
+            0.002657837001606822,
+            -0.00685235857963562,
+            0.00014345165982376784,
+            0.02243872918188572,
+            -0.009510194882750511,
+            0.005601841956377029,
+            0.012175872921943665,
+            -0.0016052864957600832,
+            0.03622184693813324,
+            0.018769502639770508,
+            0.024978963658213615,
+            -0.0017366103129461408,
+            0.010584305971860886,
+            0.0008266539080068469,
+            0.010537264868617058,
+            0.013845840469002724,
+            -0.010984158143401146,
+            0.0008913357742130756,
+            -0.014347615651786327,
+            0.017028972506523132,
+            0.024665353819727898,
+            0.009463153779506683,
+            0.0002709778491407633,
+            0.019506484270095825,
+            0.023191390559077263,
+            0.0031419708393514156,
+            -0.00707188481464982,
+            -0.007753984536975622,
+            0.017091695219278336,
+            0.01420649141073227,
+            0.007240449544042349,
+            0.0054254368878901005,
+            -0.007189488038420677,
+            -0.006036974955350161,
+            0.019192876294255257,
+            -0.019804412499070168,
+            -0.0018150125397369266,
+            0.0015288442373275757,
+            -0.04042420908808708,
+            0.010286377742886543,
+            -0.00031115900492295623,
+            -0.014669064432382584,
+            0.011180163361132145,
+            0.004072997719049454,
+            -0.010976317338645458,
+            -0.010176614858210087,
+            -0.016448795795440674,
+            -0.006879799067974091,
+            0.007593259681016207,
+            0.0024324303958564997,
+            0.008506646379828453,
+            -0.01050590444356203,
+            -0.005884090438485146,
+            0.019961217418313026,
+            0.003396778367459774,
+            0.0062369005754590034,
+            -0.0001754250843077898
+          ],
+          "keyphrases": [
+            "scaling up model capacity",
+            "training efficiency",
+            "sparsely gated mixture-of-experts",
+            "expert choice method",
+            "downstream performance"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "d08eab82-ce8c-4126-8aee-d1681de9b6e5",
+        "properties": {
+          "page_content": "5 Conclusion\n\nWe propose a new routing method for sparsely activated mixture-of-experts (MoE) models. This method addresses load imbalance and under-utilization of experts in conventional MoE methods, and enables selecting different numbers of experts for each token. Our model demonstrates more than 2x training efficiency improvements when compared to the state-of-the-art GShard and Switch Transformer models, and also achieves strong gains when finetuning on 11 datasets in the GLUE and SuperGLUE benchmark.\n\n",
+          "title": "5 Conclusion",
+          "embedding": [
+            -0.014961956068873405,
+            -0.008189860731363297,
+            -0.023845616728067398,
+            0.002967504784464836,
+            -0.00690029701218009,
+            0.024916483089327812,
+            -0.0020832864101976156,
+            0.07137094438076019,
+            -0.014486853033304214,
+            0.04606231302022934,
+            -0.016666291281580925,
+            0.003244647989049554,
+            0.0073112985119223595,
+            -0.015520012937486172,
+            -0.04539867490530014,
+            0.030662959441542625,
+            0.012692022137343884,
+            0.0035896627232432365,
+            0.013717640191316605,
+            -0.03864166513085365,
+            -0.022352438420057297,
+            -0.008853496052324772,
+            -0.018159471452236176,
+            0.0027733161114156246,
+            0.019019179046154022,
+            -0.0032974369823932648,
+            0.000584922730922699,
+            0.008634798228740692,
+            -0.03864166513085365,
+            0.05327180400490761,
+            0.011213925667107105,
+            0.009728288277983665,
+            -0.0011019736994057894,
+            -0.001177386729978025,
+            -0.03257845342159271,
+            -0.007563932333141565,
+            0.07293953746557236,
+            0.025715861469507217,
+            -0.05375444516539574,
+            0.029727837070822716,
+            0.03912430629134178,
+            0.042442481964826584,
+            -0.03233712911605835,
+            -0.008084282279014587,
+            -0.014434064738452435,
+            -0.0042646098881959915,
+            -0.0013913713628426194,
+            -0.005045135505497456,
+            0.02479582279920578,
+            -0.01170411054044962,
+            0.0019154923502355814,
+            -0.010957521386444569,
+            0.014728175476193428,
+            0.0002771430881693959,
+            -0.04500652849674225,
+            0.03791769966483116,
+            0.018596867099404335,
+            0.017103686928749084,
+            0.03366440162062645,
+            0.007273592054843903,
+            -0.00380270485766232,
+            -0.01829521358013153,
+            -0.018657196313142776,
+            0.0008644224726594985,
+            0.003012752626091242,
+            0.00490185059607029,
+            -0.0039478749968111515,
+            -0.011915267445147038,
+            0.037223897874355316,
+            -0.010565373115241528,
+            -0.06045112758874893,
+            0.03665075823664665,
+            -0.04114537686109543,
+            0.0005467448499985039,
+            -0.0135818962007761,
+            -0.0067796362563967705,
+            0.0271637924015522,
+            0.014562266878783703,
+            -0.00047156744403764606,
+            0.001911721657961607,
+            0.012073635123670101,
+            0.00981124211102724,
+            0.0072698211297392845,
+            -0.007443271577358246,
+            0.05547386407852173,
+            -0.044704876840114594,
+            0.006198955699801445,
+            -0.05541353300213814,
+            0.03659042716026306,
+            -0.0024546959903091192,
+            -0.03007473796606064,
+            -0.021115664392709732,
+            -0.021779298782348633,
+            0.008619715459644794,
+            -0.0393354631960392,
+            0.033362749963998795,
+            0.006523231975734234,
+            -0.013197289779782295,
+            -0.008989240042865276,
+            0.021145828068256378,
+            -0.004758565686643124,
+            0.015218360349535942,
+            -0.015444600023329258,
+            0.008808248676359653,
+            0.05411642789840698,
+            -0.033151593059301376,
+            -0.019366079941391945,
+            0.044644545763731,
+            0.01785781793296337,
+            0.05526270717382431,
+            0.03119085170328617,
+            0.0022322272416204214,
+            0.01181722991168499,
+            0.01852145418524742,
+            -0.003199400147423148,
+            -0.028581559658050537,
+            -0.04585115611553192,
+            -0.05535320192575455,
+            0.023498717695474625,
+            -0.028551394119858742,
+            0.022548511624336243,
+            0.020633019506931305,
+            -0.009117442183196545,
+            -0.023845616728067398,
+            -0.03906397894024849,
+            0.01708860509097576,
+            0.0047095469199121,
+            -0.005278916098177433,
+            -0.012586443684995174,
+            -0.016877448186278343,
+            0.00349162588827312,
+            -0.01137983426451683,
+            -0.02776709757745266,
+            -0.03945612534880638,
+            0.016651209443807602,
+            -0.006040588021278381,
+            0.022186528891324997,
+            0.027917923405766487,
+            -0.020135292783379555,
+            -0.019350998103618622,
+            0.013732722960412502,
+            -0.04171851649880409,
+            0.030285894870758057,
+            0.01812930591404438,
+            -0.005037594120949507,
+            0.010459794662892818,
+            0.0033125197514891624,
+            0.053965602070093155,
+            -0.026213588193058968,
+            0.020844176411628723,
+            -0.04105488210916519,
+            0.04775156453251839,
+            -0.005836972501128912,
+            0.03984827175736427,
+            -0.024117104709148407,
+            -0.013838300481438637,
+            -0.01322745531797409,
+            -0.017556166276335716,
+            -0.006719305645674467,
+            -0.022020621225237846,
+            -0.0005962346913293004,
+            -0.0063196164555847645,
+            -0.0033030931372195482,
+            0.03646976873278618,
+            0.014072081074118614,
+            0.021628472954034805,
+            0.041507359594106674,
+            0.0015572801930829883,
+            -0.07173292338848114,
+            0.030828868970274925,
+            0.007684593088924885,
+            0.030919363722205162,
+            0.006839966867119074,
+            0.015897078439593315,
+            0.006066982634365559,
+            0.021387150511145592,
+            0.002356658922508359,
+            0.004947098437696695,
+            0.005754018202424049,
+            0.04135653376579285,
+            -0.02336297370493412,
+            0.001117998966947198,
+            0.015218360349535942,
+            0.012488407082855701,
+            0.03372473269701004,
+            0.023061320185661316,
+            0.016832200810313225,
+            0.015957407653331757,
+            0.004430518485605717,
+            0.0057163117453455925,
+            0.027118545025587082,
+            0.014901624992489815,
+            -0.0032898958306759596,
+            -0.020753679797053337,
+            0.0024188747629523277,
+            -0.024599747732281685,
+            -0.002049350645393133,
+            -0.01785781793296337,
+            0.034961506724357605,
+            0.0012782517587766051,
+            -0.026183422654867172,
+            -0.014984579756855965,
+            0.005836972501128912,
+            -0.03613794967532158,
+            -0.0010991457384079695,
+            0.0015563375782221556,
+            0.016952861100435257,
+            0.005870908498764038,
+            0.013981585390865803,
+            -0.03405654802918434,
+            0.012141506187617779,
+            -0.023679709061980247,
+            -0.006142395548522472,
+            0.004988575354218483,
+            -0.0007941940566524863,
+            0.013423528522253036,
+            -0.01253365445882082,
+            0.02022578939795494,
+            -0.005361870396882296,
+            -0.028837963938713074,
+            -0.007752465084195137,
+            0.02462991327047348,
+            0.004011976066976786,
+            -0.006097147706896067,
+            -0.010210931301116943,
+            -0.01613840088248253,
+            -0.030723290517926216,
+            -0.017948314547538757,
+            0.03155283257365227,
+            0.014449146576225758,
+            -0.015489847399294376,
+            0.020527441054582596,
+            0.007394252810627222,
+            0.0028939771000295877,
+            -0.014924249611794949,
+            -0.05559452623128891,
+            -0.009969609789550304,
+            -0.012797600589692593,
+            -0.000906370987650007,
+            0.0007253796211443841,
+            -0.04506685957312584,
+            0.02573094516992569,
+            -0.03674125298857689,
+            -0.0009040143340826035,
+            0.015791499987244606,
+            0.006938003934919834,
+            -0.006127313245087862,
+            0.011666404083371162,
+            -0.026590652763843536,
+            0.01303138118237257,
+            -0.030059656128287315,
+            -0.003970498684793711,
+            6.586861854884773e-05,
+            0.015173112042248249,
+            -0.023227229714393616,
+            0.034629687666893005,
+            0.021839629858732224,
+            -0.06274368613958359,
+            -0.008318062871694565,
+            -0.01835554465651512,
+            0.05456890910863876,
+            -0.01449439488351345,
+            -0.008144613355398178,
+            0.052306514233350754,
+            -0.009441718459129333,
+            0.002309525618329644,
+            0.031371843069791794,
+            0.027118545025587082,
+            0.01377042941749096,
+            -0.02897370606660843,
+            0.002942995633929968,
+            -0.07517176121473312,
+            -0.012488407082855701,
+            0.013415987603366375,
+            0.034086715430021286,
+            -0.03063279390335083,
+            0.0013838300947099924,
+            0.015474764630198479,
+            0.01895884983241558,
+            0.03170366212725639,
+            0.002475434448570013,
+            0.02963734231889248,
+            0.010452253744006157,
+            0.01011289469897747,
+            0.004140178207308054,
+            -0.004935786128044128,
+            -0.028340237215161324,
+            -0.0015544522320851684,
+            -0.03167349472641945,
+            0.0005425028502941132,
+            -0.008099365048110485,
+            -0.014094705693423748,
+            0.004355105571448803,
+            0.005403347313404083,
+            0.04651479050517082,
+            -0.006949315778911114,
+            -0.012563819997012615,
+            -0.01306154578924179,
+            -0.0002839773951563984,
+            -0.04144703224301338,
+            -0.0399991013109684,
+            0.014946873299777508,
+            0.005105465650558472,
+            -0.023709872737526894,
+            -0.0516730435192585,
+            -0.012880554422736168,
+            -0.022895412519574165,
+            0.035564810037612915,
+            -0.022729502990841866,
+            -0.012805141508579254,
+            0.0416581891477108,
+            -0.003783851396292448,
+            0.03731439262628555,
+            -0.057494934648275375,
+            -0.008604632690548897,
+            0.006545855663716793,
+            -0.005260062403976917,
+            0.014841294847428799,
+            -0.026424745097756386,
+            -0.024056773632764816,
+            -0.010361758060753345,
+            -0.015180653892457485,
+            -0.04760073870420456,
+            0.0017759781330823898,
+            -0.02028612047433853,
+            0.017118770629167557,
+            0.046152807772159576,
+            0.03052721731364727,
+            -0.02469024434685707,
+            0.014072081074118614,
+            -0.008385934866964817,
+            -0.054689567536115646,
+            -0.025595201179385185,
+            0.010097811929881573,
+            -0.020738597959280014,
+            -0.041235875338315964,
+            -0.02043694630265236,
+            0.0005128089687786996,
+            -0.022669173777103424,
+            0.016168564558029175,
+            -0.024539416655898094,
+            -0.03360407054424286,
+            0.000932294235099107,
+            0.0206631850451231,
+            -0.0028920918703079224,
+            -0.017224349081516266,
+            -0.02153797633945942,
+            -0.02562536671757698,
+            0.013793053105473518,
+            -0.004573803395032883,
+            0.0009091990068554878,
+            0.019592318683862686,
+            0.03254828602075577,
+            0.025534870103001595,
+            0.0035406441893428564,
+            -0.03333258256316185,
+            -0.021191077306866646,
+            0.026032596826553345,
+            0.0018824990838766098,
+            -0.0013084170641377568,
+            0.03245779126882553,
+            -0.03653009608387947,
+            -0.04325694590806961,
+            0.005554173607379198,
+            -0.04910900071263313,
+            0.0020418092608451843,
+            -0.021462563425302505,
+            -0.022503264248371124,
+            -0.013898631557822227,
+            0.002277475083246827,
+            0.003847952699288726,
+            0.02909436821937561,
+            0.011658863164484501,
+            -0.010482418350875378,
+            -0.023438386619091034,
+            0.008521678857505322,
+            0.0483548678457737,
+            -0.010060105472803116,
+            -0.02710346318781376,
+            0.03278961032629013,
+            -0.016455134376883507,
+            0.01168148685246706,
+            -0.011425082571804523,
+            -0.022895412519574165,
+            -0.01592724397778511,
+            0.017827652394771576,
+            0.01895884983241558,
+            0.006172561086714268,
+            -0.041235875338315964,
+            0.025474539026618004,
+            -0.0063497815281152725,
+            -0.03840034082531929,
+            0.014056999236345291,
+            0.004128866363316774,
+            0.01239791139960289,
+            0.0005486301961354911,
+            0.013174666091799736,
+            -0.054297421127557755,
+            -0.005844513885676861,
+            0.006036817561835051,
+            -0.03191481530666351,
+            -0.0058595966547727585,
+            -0.008378393948078156,
+            0.02280491590499878,
+            -0.014946873299777508,
+            0.015391810797154903,
+            -0.012405452318489552,
+            0.008144613355398178,
+            0.031975146383047104,
+            -0.03441853076219559,
+            -0.015067534521222115,
+            -0.036228444427251816,
+            -0.0041929674334824085,
+            -0.022458016872406006,
+            0.020361533388495445,
+            0.04334744065999985,
+            -0.017661744728684425,
+            0.00917777232825756,
+            0.0028185639530420303,
+            -0.004241986200213432,
+            -0.022005537524819374,
+            0.016817117109894753,
+            0.0569821260869503,
+            0.031100355088710785,
+            -0.008039034903049469,
+            -0.0006443105521611869,
+            0.02197537198662758,
+            -0.013121876865625381,
+            -0.008340687490999699,
+            0.006051899865269661,
+            0.017752239480614662,
+            0.010701116174459457,
+            0.02286524698138237,
+            -0.040089596062898636,
+            -0.010150601156055927,
+            0.009109900332987309,
+            -0.02469024434685707,
+            0.027812344953417778,
+            -0.03073837235569954,
+            0.018973931670188904,
+            0.017812570556998253,
+            -0.01956215314567089,
+            0.013627144508063793,
+            -0.010022399015724659,
+            -0.023830534890294075,
+            -0.02512763999402523,
+            -0.02496173046529293,
+            -0.008634798228740692,
+            0.017978480085730553,
+            -0.0139966681599617,
+            -0.05287965387105942,
+            0.04301562160253525,
+            -0.06298501044511795,
+            -0.029773086309432983,
+            0.03191481530666351,
+            -0.02027103677392006,
+            -0.014313403517007828,
+            0.008672504685819149,
+            0.01570100337266922,
+            -0.033362749963998795,
+            0.020452028140425682,
+            0.011643780395388603,
+            0.033151593059301376,
+            -0.028023501858115196,
+            -0.024931564927101135,
+            0.0034765431191772223,
+            -0.02739003300666809,
+            -0.006010422948747873,
+            -0.0002648884546943009,
+            0.014667844399809837,
+            0.00637617614120245,
+            -0.033754896372556686,
+            -0.009743370115756989,
+            0.003800819395110011,
+            0.0027506921906024218,
+            0.03348340839147568,
+            -0.004132636822760105,
+            -0.04205033555626869,
+            0.014170118607580662,
+            0.004321169573813677,
+            -0.03284993767738342,
+            0.006406341679394245,
+            0.003346455516293645,
+            -0.009185313247144222,
+            -0.016334474086761475,
+            0.005478760693222284,
+            0.03221647068858147,
+            0.030210481956601143,
+            0.017812570556998253,
+            0.007118995301425457,
+            0.009871572256088257,
+            -0.01338582206517458,
+            -0.01503736898303032,
+            0.010655868798494339,
+            -0.007782630156725645,
+            -0.03191481530666351,
+            -0.013204830698668957,
+            -0.014780964702367783,
+            0.025866687297821045,
+            -0.012563819997012615,
+            -0.01253365445882082,
+            0.005980257410556078,
+            0.045157354325056076,
+            0.014615056104958057,
+            -0.005267603788524866,
+            -0.01336319837719202,
+            0.007601638790220022,
+            0.012707104906439781,
+            -0.034358199685811996,
+            -0.022277025505900383,
+            0.022065868601202965,
+            0.0023283790796995163,
+            0.017375174909830093,
+            -0.024599747732281685,
+            -0.022216694429516792,
+            0.03239746019244194,
+            -0.03248795494437218,
+            0.030919363722205162,
+            0.01580658182501793,
+            0.006202726159244776,
+            -0.006444048136472702,
+            0.004739712458103895,
+            -0.014569807797670364,
+            0.028400568291544914,
+            -0.029350772500038147,
+            0.027193957939743996,
+            0.020421862602233887,
+            -0.006285680457949638,
+            0.060300301760435104,
+            -0.029682589694857597,
+            -0.016334474086761475,
+            0.009607627056539059,
+            -0.0043777297250926495,
+            0.015391810797154903,
+            -0.019094591960310936,
+            -0.007149160373955965,
+            -0.0060745240189135075,
+            0.005018740892410278,
+            -0.029667507857084274,
+            -0.015897078439593315,
+            0.02164355479180813,
+            -0.005987798795104027,
+            -0.0013272702926769853,
+            0.009660416282713413,
+            -0.02286524698138237,
+            0.017495835199952126,
+            -0.006542085204273462,
+            -0.04533834755420685,
+            -0.00045813448377884924,
+            0.02153797633945942,
+            -0.024388590827584267,
+            -0.015912160277366638,
+            -0.008582009002566338,
+            0.0038837739266455173,
+            -0.00595009233802557,
+            -0.02280491590499878,
+            -0.019245419651269913,
+            -0.005655981134623289,
+            -0.008385934866964817,
+            -0.027480527758598328,
+            -0.011779523454606533,
+            0.034086715430021286,
+            0.01294842641800642,
+            -0.004875455982983112,
+            -0.010301426984369755,
+            -0.029667507857084274,
+            0.016002656891942024,
+            0.04373958706855774,
+            0.0034218686632812023,
+            0.006941774394363165,
+            -0.0065948739647865295,
+            -0.0026451139710843563,
+            -0.014841294847428799,
+            0.009864031337201595,
+            0.0017646661726757884,
+            -0.0028223346453160048,
+            0.01790306530892849,
+            -0.011342127807438374,
+            -0.03381522744894028,
+            -0.026967719197273254,
+            0.04304578900337219,
+            -0.023785287514328957,
+            0.004483307711780071,
+            0.04651479050517082,
+            0.036620594561100006,
+            -0.005343017168343067,
+            -0.014305861666798592,
+            -0.03107019141316414,
+            0.0020663184113800526,
+            -0.02010512910783291,
+            0.009509589523077011,
+            0.03294043615460396,
+            -0.00039686134550720453,
+            -0.034297872334718704,
+            -0.013529106974601746,
+            -0.02099500223994255,
+            -0.010346675291657448,
+            0.015716087073087692,
+            0.017013192176818848,
+            -0.034358199685811996,
+            0.005795495584607124,
+            0.018310297280550003,
+            0.028943542391061783,
+            -0.011349668726325035,
+            -0.016983026638627052,
+            0.024765657261013985,
+            0.017224349081516266,
+            0.0009294662741012871,
+            -0.0029222569428384304,
+            -0.00034312953357584774,
+            0.0068098013289272785,
+            -0.004068535752594471,
+            0.033030930906534195,
+            0.0035274471156299114,
+            0.011206384748220444,
+            -0.02116091176867485,
+            0.00479250168427825,
+            0.020738597959280014,
+            -0.006870131939649582,
+            -0.001355550135485828,
+            0.07384449243545532,
+            0.00214173155836761,
+            0.01355173159390688,
+            0.017209265381097794,
+            0.03444869816303253,
+            0.01773715764284134,
+            -0.017993561923503876,
+            -0.03170366212725639,
+            -0.014313403517007828,
+            -0.007718529086560011,
+            -0.020527441054582596,
+            -0.0049697221256792545,
+            -0.012910719960927963,
+            -0.0433172769844532,
+            -0.0223222728818655,
+            -0.0033502262085676193,
+            -0.002466007834300399,
+            0.025263383984565735,
+            -0.0026319166645407677,
+            0.0009822554420679808,
+            -0.027948088943958282,
+            0.03565530478954315,
+            0.02622867003083229,
+            0.023996442556381226,
+            -0.007880667224526405,
+            0.011100806295871735,
+            -0.0043362523429095745,
+            -0.008491513319313526,
+            0.02754085883498192,
+            0.0074885194189846516,
+            -0.008137071505188942,
+            0.03478051349520683,
+            0.012111341580748558,
+            0.05272882804274559,
+            -0.013114335015416145,
+            -0.024343343451619148,
+            -0.017179099842905998,
+            -0.04018009081482887,
+            0.011372293345630169,
+            -0.007710987702012062,
+            0.0001368040539091453,
+            0.014139953069388866,
+            0.017390256747603416,
+            -0.008167237043380737,
+            0.0271637924015522,
+            0.0038498379290103912,
+            -0.015504930168390274,
+            0.025187969207763672,
+            -0.02892845869064331,
+            0.014253072440624237,
+            -0.0035576121881604195,
+            0.011839854530990124,
+            -0.012903179042041302,
+            0.011538201943039894,
+            0.02358921244740486,
+            -0.025761108845472336,
+            0.028551394119858742,
+            0.017254512757062912,
+            -0.0027412655763328075,
+            0.00480758398771286,
+            0.0026168341282755136,
+            -0.0012169786496087909,
+            -0.022080950438976288,
+            -0.034086715430021286,
+            0.02533879689872265,
+            -0.007473436649888754,
+            -0.023332808166742325,
+            0.02425284869968891,
+            -0.017118770629167557,
+            -0.008928908966481686,
+            -0.0017175329849123955,
+            -0.009358763694763184,
+            -0.007579015102237463,
+            0.04292512685060501,
+            -0.008664963766932487,
+            -0.004291004501283169,
+            -0.01640988700091839,
+            0.02689230628311634,
+            0.04226149246096611,
+            -0.008385934866964817,
+            -0.009072193875908852,
+            0.021899959072470665,
+            -0.012450700625777245,
+            0.0015129749663174152,
+            0.041628021746873856,
+            0.006153707858175039,
+            0.0015365416184067726,
+            0.005712541285902262,
+            -0.03393588960170746,
+            -0.029999325051903725,
+            0.005806807428598404,
+            0.016455134376883507,
+            0.016047904267907143,
+            0.04126603901386261,
+            0.03646976873278618,
+            -0.014969496987760067,
+            0.0025037142913788557,
+            0.0007546021952293813,
+            -0.035232994705438614,
+            0.028747467324137688,
+            -0.005486302077770233,
+            0.021447481587529182,
+            0.02485615201294422,
+            -0.026183422654867172,
+            -0.036620594561100006,
+            -0.00034996384056285024,
+            0.030210481956601143,
+            0.0190040972083807,
+            0.014056999236345291,
+            0.014072081074118614,
+            0.023227229714393616,
+            -0.020316284149885178,
+            -0.024946648627519608,
+            0.02012021094560623,
+            0.02506730891764164,
+            -0.009456800296902657,
+            0.01074636448174715,
+            0.017933230847120285,
+            0.009818783961236477,
+            -0.0012282906100153923,
+            0.03134167939424515,
+            0.009019404649734497,
+            0.016289226710796356,
+            0.008287898264825344,
+            -0.011515578255057335,
+            0.02500697784125805,
+            0.010271262377500534,
+            0.03393588960170746,
+            -0.012412993237376213,
+            0.003929021768271923,
+            0.003689585020765662,
+            -0.004087388981133699,
+            0.037163566797971725,
+            0.005946321878582239,
+            -0.04307595267891884,
+            0.0199995506554842,
+            0.011960514821112156,
+            -0.0001776920835254714,
+            -0.006489295978099108,
+            -0.0026469992008060217,
+            0.018446041271090508,
+            0.02043694630265236,
+            0.018038809299468994,
+            0.009532214142382145,
+            0.005659752059727907,
+            0.005618274677544832,
+            0.006138625089079142,
+            0.03167349472641945,
+            0.0020455799531191587,
+            -0.01978839375078678,
+            -0.00230009900406003,
+            0.005505155306309462,
+            0.021115664392709732,
+            -0.019637567922472954,
+            0.01653054729104042,
+            0.019215254113078117,
+            -0.0011934119975194335,
+            -0.01746566966176033,
+            -0.03671108931303024,
+            -0.007303757127374411,
+            -0.013853383250534534,
+            0.013898631557822227,
+            0.008951532654464245,
+            0.018687361851334572,
+            -0.0221563633531332,
+            0.016078069806098938,
+            0.019592318683862686,
+            -0.01950182393193245,
+            0.006496837362647057,
+            0.008061658591032028,
+            -0.014524560421705246,
+            -0.005580568220466375,
+            -0.016108235344290733,
+            0.002053121104836464,
+            0.06382963061332703,
+            -0.014464229345321655,
+            0.009954527020454407,
+            0.024931564927101135,
+            -0.009879114106297493,
+            -0.030119985342025757,
+            0.0009671727893874049,
+            -0.010429629124701023,
+            -0.002450925298035145,
+            0.024373508989810944,
+            -0.03577596694231033,
+            -0.04319661483168602,
+            -0.005290227942168713,
+            -0.03631894290447235,
+            -0.006560938432812691,
+            0.019034262746572495,
+            0.027948088943958282,
+            -0.014162576757371426,
+            -0.02760118804872036,
+            0.02292557805776596,
+            -0.0020418092608451843,
+            0.009765994735062122,
+            -0.019471658393740654,
+            -0.026545405387878418,
+            -0.03030097670853138,
+            -0.03212597221136093,
+            -0.00961516797542572,
+            -0.025097474455833435,
+            0.008408558554947376,
+            0.0012725958367809653,
+            -0.05061726272106171,
+            -0.030602630227804184,
+            0.010670951567590237,
+            -0.002435842528939247,
+            0.04005942866206169,
+            0.039757777005434036,
+            0.003970498684793711,
+            0.028008420020341873,
+            -0.03613794967532158,
+            0.050828419625759125,
+            -0.006530772894620895,
+            0.03592679277062416,
+            -0.007579015102237463,
+            0.0019890200346708298,
+            -0.014358650892972946,
+            0.04379991814494133,
+            0.000892702373676002,
+            0.002547076903283596,
+            -0.038762323558330536,
+            0.0021455022506415844,
+            -0.023996442556381226,
+            -0.00025451916735619307,
+            -0.0026752790436148643,
+            0.008996780961751938,
+            -0.0009907394414767623,
+            -0.014856377616524696,
+            0.002102139638736844,
+            0.03345324471592903,
+            0.02616834081709385,
+            -0.021779298782348633,
+            -0.027088379487395287,
+            -0.030617712065577507,
+            0.0004138292861171067,
+            -0.008725293911993504,
+            -0.002880779793485999,
+            -0.004860373213887215,
+            -0.024313177913427353,
+            0.014396357350051403,
+            -0.016771869733929634,
+            -0.013415987603366375,
+            -0.002205832628533244,
+            0.02545945718884468,
+            -0.01746566966176033,
+            -0.02259376086294651,
+            -0.005814348813146353,
+            -0.029335688799619675,
+            -0.01675678789615631,
+            0.00022930291015654802,
+            -0.02485615201294422,
+            0.017510918900370598,
+            0.01609315164387226,
+            -0.012103799730539322,
+            0.0002860983950085938,
+            0.0022001767065376043,
+            0.016123317182064056,
+            -0.021794380620121956,
+            -0.004166572820395231,
+            0.0025357650592923164,
+            -0.00017439275688957423,
+            -0.023151816800236702,
+            0.021387150511145592,
+            -0.019411327317357063,
+            0.01631939224898815,
+            -0.04844536632299423,
+            -0.011213925667107105,
+            -0.025037143379449844,
+            0.010587996803224087,
+            0.001273538451641798,
+            -0.02990882843732834,
+            -0.014072081074118614,
+            0.00785050168633461,
+            -0.0031447256915271282,
+            -0.02182454615831375,
+            0.008506596088409424,
+            0.03541398420929909,
+            -0.0014385045506060123,
+            0.009683039970695972,
+            -0.029773086309432983,
+            -0.011590991169214249,
+            -0.010158142074942589,
+            0.03079870343208313,
+            0.0037574570160359144,
+            -0.005131860263645649,
+            -0.00574270635843277,
+            0.0005769100971519947,
+            -0.03472018241882324,
+            -0.021899959072470665,
+            -0.003259730525314808,
+            -0.002062547951936722,
+            -0.005392035469412804,
+            0.034509025514125824,
+            0.03405654802918434,
+            0.03686191514134407,
+            -0.04379991814494133,
+            0.019215254113078117,
+            -0.008325604721903801,
+            -0.011108347214758396,
+            0.01325007900595665,
+            0.018777858465909958,
+            0.01840079203248024,
+            -0.016485299915075302,
+            0.001463956548832357,
+            0.003412442049011588,
+            0.013536648824810982,
+            -0.005603192374110222,
+            0.022985907271504402,
+            -0.00719063775613904,
+            0.013408446684479713,
+            0.014260614290833473,
+            -8.476607035845518e-06,
+            0.022955743595957756,
+            -0.006843737326562405,
+            0.009901737794280052,
+            -0.007043581921607256,
+            -0.014094705693423748,
+            0.0017872900934889913,
+            -0.03882265463471413,
+            -0.013016298413276672,
+            0.015384268946945667,
+            0.018777858465909958,
+            -0.012495948001742363,
+            -0.04874701797962189,
+            0.022020621225237846,
+            0.03671108931303024,
+            -0.019275585189461708,
+            0.0273297019302845,
+            -0.0279933363199234,
+            0.016334474086761475,
+            0.011613614857196808,
+            0.013272702693939209,
+            0.004773647990077734,
+            0.020738597959280014,
+            -0.02512763999402523,
+            0.033966053277254105,
+            -0.0019550842698663473,
+            0.025761108845472336,
+            0.016847282648086548,
+            -0.015391810797154903,
+            -0.04573049396276474,
+            -0.016108235344290733,
+            0.006753241643309593,
+            -0.012028386816382408,
+            -0.002897747792303562,
+            -0.043498266488313675,
+            -0.03420737385749817,
+            0.030210481956601143,
+            0.01418520137667656,
+            -0.0493503212928772,
+            0.00014446319255512208,
+            -0.014034374617040157,
+            -0.0030730832368135452,
+            -0.0052261268720030785,
+            -0.028943542391061783,
+            -0.02336297370493412,
+            0.00801641121506691,
+            0.00752245495095849,
+            0.03981810808181763,
+            -0.023046238347887993,
+            0.026741480454802513,
+            -0.01159853208810091,
+            -0.01719418354332447,
+            -0.02005987986922264,
+            -0.00045224284986034036,
+            -0.021326819434762,
+            -0.01481867115944624,
+            0.005094153806567192,
+            0.020738597959280014,
+            0.009660416282713413,
+            0.029396019876003265,
+            -0.012722187675535679,
+            -0.014856377616524696,
+            -0.0005095096421428025,
+            -0.03465985506772995,
+            -0.028506144881248474,
+            0.007032270077615976,
+            0.0006749471067450941,
+            -0.017269596457481384,
+            -0.02070843242108822,
+            -0.03704290837049484,
+            0.016213813796639442,
+            0.011990680359303951,
+            0.01692269556224346,
+            0.03245779126882553,
+            0.0198336411267519,
+            -0.01386092510074377,
+            0.01669645681977272,
+            -0.007888208143413067,
+            0.0009813127107918262,
+            0.01013551838696003,
+            0.018581783398985863,
+            -0.013114335015416145,
+            -0.0027733161114156246,
+            -0.014765881933271885,
+            -0.029003871604800224,
+            -0.01708860509097576,
+            -0.01806897483766079,
+            -0.0008795050671324134,
+            -0.006149936933070421,
+            -0.03891315311193466,
+            -0.013415987603366375,
+            0.024991896003484726,
+            0.02396627888083458,
+            -0.01534656248986721,
+            -0.007511143106967211,
+            -0.00023295573191717267,
+            -0.0208290945738554,
+            -0.004894309211522341,
+            -0.02342330478131771,
+            -0.003672617254778743,
+            -0.024147270247340202,
+            0.013234996236860752,
+            -0.0045813447795808315,
+            0.010052564553916454,
+            -0.053694114089012146,
+            -0.013174666091799736,
+            0.007326381281018257,
+            -0.010105352848768234,
+            -0.0016996223712339997,
+            0.01746566966176033,
+            -0.003291781060397625,
+            -0.009162689559161663,
+            -0.023845616728067398,
+            0.014155035838484764,
+            0.005772871430963278,
+            0.021025167778134346,
+            -0.002831761259585619,
+            -0.00406476529315114,
+            0.03348340839147568,
+            0.025384044274687767,
+            -5.541094287764281e-05,
+            -0.012269708327949047,
+            0.015897078439593315,
+            0.014788505621254444,
+            0.0014507592422887683,
+            0.0044493721798062325,
+            0.009102359414100647,
+            0.0048264372162520885,
+            -0.010889649391174316,
+            -0.00480758398771286,
+            0.024871235713362694,
+            0.006527002435177565,
+            0.005935009568929672,
+            -0.01463013794273138,
+            0.02897370606660843,
+            0.03752554953098297,
+            0.0196677315980196,
+            0.028068749234080315,
+            -0.015685921534895897,
+            -0.011032934300601482,
+            -0.007760006468743086,
+            -0.013762887567281723,
+            0.031432174146175385,
+            0.008861036971211433,
+            -0.009871572256088257,
+            -0.0104145472869277,
+            0.010037481784820557,
+            -0.009939444251358509,
+            0.002166240941733122,
+            0.02269933745265007,
+            0.009592544287443161,
+            0.005580568220466375,
+            -0.00028892638511024415,
+            0.007548849564045668,
+            0.015912160277366638,
+            0.023770203813910484,
+            -0.02975800260901451,
+            -0.027420196682214737,
+            -0.0009120269678533077,
+            0.0003007096820510924,
+            0.012993674725294113,
+            0.01767682656645775,
+            -0.0011594761162996292,
+            0.006658975500613451,
+            0.04334744065999985,
+            -0.015746252611279488,
+            -0.0034369511995464563,
+            -0.014698009938001633,
+            -0.013272702693939209,
+            -0.0022491952404379845,
+            -0.021945208311080933,
+            0.008687587454915047,
+            0.012382828630506992,
+            0.017118770629167557,
+            0.014245531521737576,
+            -0.008023952133953571,
+            0.012322497554123402,
+            0.018446041271090508,
+            0.011515578255057335,
+            -0.004219362046569586,
+            -0.014780964702367783,
+            -0.002564044902101159,
+            0.015248525887727737,
+            0.009034487418830395,
+            0.022835081443190575,
+            0.021326819434762,
+            0.006949315778911114,
+            0.01322745531797409,
+            -0.025052227079868317,
+            0.025746027007699013,
+            0.0144114401191473,
+            0.02419251762330532,
+            -0.03770654276013374,
+            -0.021145828068256378,
+            0.014501935802400112,
+            -0.007141618989408016,
+            -0.0004892423748970032,
+            0.015384268946945667,
+            -0.008091824129223824,
+            0.05125073343515396,
+            -0.0353536531329155,
+            0.00845380686223507,
+            0.02358921244740486,
+            -0.015881994739174843,
+            0.010271262377500534,
+            0.01818963512778282,
+            0.010595538653433323,
+            -0.010678492486476898,
+            -0.02093467302620411,
+            0.00037730109761469066,
+            0.02551978826522827,
+            -0.03818918392062187,
+            -0.018611948937177658,
+            -0.0065345438197255135,
+            -0.0004892423748970032,
+            0.022729502990841866,
+            -0.007552620489150286,
+            0.014645220711827278,
+            -0.008928908966481686,
+            0.011432623490691185,
+            0.001585560035891831,
+            -0.03134167939424515,
+            -0.007537537720054388,
+            0.014637679792940617,
+            0.01231495663523674,
+            -0.009667957201600075,
+            -0.006674057804048061,
+            -0.0010331592056900263,
+            -0.002075745025649667,
+            -0.014268155209720135,
+            0.02722412347793579,
+            0.0018617605092003942,
+            0.01592724397778511,
+            -0.00208705710247159,
+            -0.014207825064659119,
+            0.018551617860794067,
+            -0.01962248422205448,
+            -0.015256066806614399,
+            -0.004521014168858528,
+            0.001125540235079825,
+            -0.0011905840365216136,
+            -0.031100355088710785,
+            0.015361645258963108,
+            -0.013257619924843311,
+            0.034629687666893005,
+            0.006180102005600929,
+            -0.0012207493418827653,
+            -0.002481090370565653,
+            -0.011606073938310146,
+            -0.0056145042181015015,
+            -0.02821957692503929,
+            -0.017872901633381844,
+            0.01686236634850502,
+            -0.013536648824810982,
+            -0.009909279644489288,
+            -0.026515239849686623,
+            -0.021734051406383514,
+            -0.011040475219488144,
+            0.008793165907263756,
+            0.0014347339747473598,
+            0.022850165143609047,
+            0.00864988099783659,
+            -0.017375174909830093,
+            -0.0033671942073851824,
+            -0.041959840804338455,
+            -0.002809137338772416,
+            0.01033159252256155,
+            -0.017541082575917244,
+            -0.022246859967708588,
+            0.013204830698668957,
+            0.00048052275087684393,
+            -0.018596867099404335,
+            0.019411327317357063,
+            0.0019183203112334013,
+            0.004434289410710335,
+            0.0017147050239145756,
+            -0.0023151817731559277,
+            -0.01653054729104042,
+            0.00573139451444149,
+            0.009554837830364704,
+            0.01536918617784977,
+            -0.047238755971193314,
+            -0.007499831262975931,
+            0.003244647989049554,
+            0.006372405681759119,
+            0.006010422948747873,
+            0.01118376012891531,
+            0.004777418915182352,
+            -0.017767323181033134,
+            0.004471995867788792,
+            0.01906442828476429,
+            -0.022563595324754715,
+            0.01977331005036831,
+            -0.0019051230046898127,
+            0.0027883988805115223,
+            0.002799710724502802,
+            -0.007514914032071829,
+            -0.005490072537213564,
+            0.02583652175962925,
+            -0.010452253744006157,
+            0.009622709825634956,
+            0.021507810801267624,
+            0.02309148572385311,
+            -0.011470329947769642,
+            -0.01862703263759613,
+            0.013710098341107368,
+            0.0316433310508728,
+            0.0006923863547854125,
+            -0.014992120675742626,
+            -0.054086264222860336,
+            -0.011206384748220444,
+            0.03218630328774452,
+            0.015897078439593315,
+            0.017390256747603416,
+            -0.006941774394363165,
+            -0.012518571689724922,
+            0.019139841198921204,
+            0.023559046909213066,
+            0.017043357715010643,
+            -0.026545405387878418,
+            0.013928796164691448,
+            -0.020844176411628723,
+            0.03432803601026535,
+            0.0030429179314523935,
+            -0.008008869364857674,
+            0.004143949132412672,
+            -0.013431070372462273,
+            -0.02313673496246338,
+            0.014200284145772457,
+            -0.003846067236736417,
+            0.0028826650232076645,
+            0.006545855663716793,
+            -0.004864144138991833,
+            0.01939624547958374,
+            -0.0008766771061345935,
+            -0.005897303111851215,
+            0.004747253842651844,
+            0.02787267602980137,
+            0.0013659194810315967,
+            0.003993122838437557,
+            0.0011085723526775837,
+            0.010090271010994911,
+            -0.0019946759566664696,
+            -0.012586443684995174,
+            0.022563595324754715,
+            -0.012865472584962845,
+            -0.0056145042181015015,
+            0.01923033595085144,
+            -0.007903290912508965,
+            -0.00023342706845141947,
+            -0.010452253744006157,
+            -0.008838413283228874,
+            -0.03152266889810562,
+            0.026183422654867172,
+            -0.0036066307220607996,
+            0.014396357350051403,
+            0.021010085940361023,
+            -0.0012094373814761639,
+            -0.005410888697952032,
+            0.022337354719638824,
+            0.0002851557219401002,
+            -0.007458353880792856,
+            0.01386092510074377,
+            -0.02562536671757698,
+            -0.02612309157848358,
+            0.03858133405447006,
+            -0.012269708327949047,
+            0.0005542861763387918,
+            -0.0022491952404379845,
+            -0.00906465295702219,
+            -0.007511143106967211,
+            0.020542524755001068,
+            0.01708860509097576,
+            -0.02793300710618496,
+            -0.010595538653433323,
+            -0.010127977468073368,
+            -0.02909436821937561,
+            0.011590991169214249,
+            -0.0018532765097916126,
+            0.02153797633945942,
+            -0.005799266044050455,
+            -0.03155283257365227,
+            -0.004762336146086454,
+            0.03592679277062416,
+            -0.003981810994446278,
+            0.0010048793628811836,
+            0.015323938801884651,
+            0.0028675824869424105,
+            0.015610508620738983,
+            -0.029456350952386856,
+            0.025474539026618004,
+            -0.01640988700091839,
+            0.020255954936146736,
+            0.013483859598636627,
+            -0.014464229345321655,
+            0.027736932039260864,
+            0.013234996236860752,
+            0.005493842996656895,
+            0.006413882598280907,
+            0.0030052114743739367,
+            -0.010859483852982521,
+            -0.007077517919242382,
+            0.008340687490999699,
+            -0.002134190173819661,
+            0.0003591548011172563,
+            0.02594210021197796,
+            0.0003921480383723974,
+            0.013762887567281723,
+            0.001332926214672625,
+            -0.0001895932073239237,
+            0.0017618382116779685,
+            -0.002952422248199582,
+            -0.01944149285554886,
+            0.011304421350359917,
+            -0.015912160277366638,
+            -0.020964836701750755,
+            0.012443158775568008,
+            -0.017269596457481384,
+            -0.01785781793296337,
+            -0.028113998472690582,
+            0.01283530704677105,
+            -0.008861036971211433,
+            -0.007337693125009537,
+            0.009667957201600075,
+            0.011764440685510635,
+            0.015595425851643085,
+            0.02660573646426201,
+            0.019863806664943695,
+            -0.012744811363518238,
+            0.013762887567281723,
+            0.010489960201084614,
+            0.008830872364342213,
+            -0.00939647015184164,
+            0.0018146273214370012,
+            -0.010308968834578991,
+            0.020240871235728264,
+            0.010912273079156876,
+            0.004038370680063963,
+            0.019200170412659645,
+            -0.007669510785490274,
+            -0.01264677383005619,
+            -0.03505200147628784,
+            -0.004882997367531061,
+            -0.001891925698146224,
+            0.007262280210852623,
+            0.04998379200696945,
+            -0.014245531521737576,
+            -0.03300076723098755,
+            -0.016847282648086548,
+            -0.011802148073911667,
+            0.009645333513617516,
+            0.0012443158775568008,
+            0.010723740793764591,
+            -0.013491400517523289,
+            -0.035564810037612915,
+            0.0030825098510831594,
+            0.015776416286826134,
+            0.003531217575073242,
+            -0.0020041028037667274,
+            0.022337354719638824,
+            -0.028113998472690582,
+            0.0180086437612772,
+            -0.007514914032071829,
+            -0.011470329947769642,
+            -0.02088942378759384,
+            -0.0033671942073851824,
+            -0.0076581984758377075,
+            0.023845616728067398,
+            -0.005497613921761513,
+            -0.01978839375078678,
+            -0.003457689890637994,
+            0.011357210576534271,
+            -0.0009516188292764127,
+            -0.005659752059727907,
+            0.004928245209157467,
+            0.006406341679394245,
+            -0.0029769313987344503,
+            0.033905722200870514,
+            -0.0017128196777775884,
+            -0.0024792051408439875,
+            -0.0063196164555847645,
+            -0.0027733161114156246,
+            0.00947942491620779,
+            0.015022286213934422,
+            -0.047087930142879486,
+            -0.019878888502717018,
+            -0.007254738826304674,
+            -0.004079848062247038,
+            0.0051544844172894955,
+            0.012722187675535679,
+            -0.02104024961590767,
+            -0.00375934224575758,
+            -0.011191301979124546,
+            -0.008536760695278645,
+            -0.029516680166125298,
+            -0.010889649391174316,
+            0.03478051349520683,
+            -0.009426635690033436,
+            0.039094142615795135,
+            0.021130746230483055,
+            -0.0020436944905668497,
+            0.015052451752126217,
+            0.00261117797344923,
+            0.002690361812710762,
+            -0.00906465295702219,
+            -0.013936338014900684,
+            -0.0019626254215836525,
+            -0.005486302077770233,
+            0.0025489621330052614,
+            -0.03960695117712021,
+            0.0004977263743057847,
+            -0.0033276022877544165,
+            0.016998108476400375,
+            -0.035685472190380096,
+            -0.008891202509403229,
+            0.026982801035046577,
+            -0.012035928666591644,
+            0.005026282276958227,
+            -0.003996893297880888,
+            0.01653054729104042,
+            0.042231328785419464,
+            0.011455247178673744,
+            -0.003372850129380822,
+            -0.017782405018806458,
+            -0.0002785570686683059,
+            0.004988575354218483,
+            -0.03846067190170288,
+            0.004732171073555946,
+            -0.039697445929050446,
+            0.030180316418409348,
+            -0.00679094810038805,
+            -0.008333145640790462,
+            0.03541398420929909,
+            -0.018038809299468994,
+            0.01473571639508009,
+            0.009879114106297493,
+            -0.022186528891324997,
+            0.02693755365908146,
+            0.002552732825279236,
+            0.018551617860794067,
+            -0.04147719591856003,
+            0.00793345645070076,
+            -0.008295439183712006,
+            0.00950204860419035,
+            -0.01642496883869171,
+            -0.0017646661726757884,
+            0.016575796529650688,
+            -0.010127977468073368,
+            0.014449146576225758,
+            0.020135292783379555,
+            -0.001117998966947198,
+            0.0006494951667264104,
+            0.021899959072470665,
+            -0.006527002435177565,
+            0.0060330466367304325,
+            0.015489847399294376,
+            0.013868466019630432,
+            0.01346877682954073,
+            0.037827201187610626,
+            -0.002782742725685239,
+            0.0012320613022893667,
+            0.003993122838437557,
+            0.001916434965096414,
+            0.010241096839308739,
+            -0.0067796362563967705,
+            -0.001194354728795588,
+            -0.016605960205197334,
+            0.00782033707946539,
+            0.017013192176818848,
+            -0.006379947066307068,
+            -0.001372518134303391,
+            -0.030376389622688293,
+            -0.010037481784820557,
+            -0.04871685057878494,
+            0.006112230475991964,
+            -0.030919363722205162,
+            -0.010150601156055927,
+            -0.008416100405156612,
+            0.018385710194706917,
+            -0.003538758959621191,
+            0.012412993237376213,
+            0.001619496033526957,
+            -0.019984466955065727,
+            -0.005135631188750267,
+            0.008310521952807903,
+            -0.015059992671012878,
+            0.02325739525258541,
+            0.0071680136024951935,
+            -0.006440277211368084,
+            0.00266585242934525,
+            -0.013348115608096123,
+            -0.006142395548522472,
+            -0.015090158209204674,
+            -0.007017187308520079,
+            0.002581012900918722,
+            0.022654090076684952,
+            0.004143949132412672,
+            -0.0013828874798491597,
+            0.00010999705409631133,
+            -0.007752465084195137,
+            0.015791499987244606,
+            0.0027092150412499905,
+            -0.0003994536818936467,
+            0.004984804894775152,
+            0.015135405585169792,
+            0.008355769328773022,
+            -0.0059312391094863415,
+            0.008446265012025833,
+            0.006447818595916033,
+            -0.009592544287443161,
+            0.025821439921855927,
+            -0.008159696124494076,
+            0.009042028337717056,
+            -0.019712980836629868,
+            0.007639345247298479,
+            0.011093264445662498,
+            -0.014456688426434994,
+            -0.008023952133953571,
+            -0.02155306003987789,
+            0.02076876349747181,
+            0.007473436649888754,
+            0.0074885194189846516,
+            -0.024720408022403717,
+            0.014139953069388866,
+            -0.0055730268359184265,
+            -0.0013706327881664038,
+            -0.024780739098787308,
+            0.016213813796639442,
+            0.024886317551136017,
+            0.00906465295702219,
+            0.02027103677392006,
+            -0.003993122838437557,
+            0.008717752061784267,
+            -0.016017738729715347,
+            0.002801596187055111,
+            -0.012330039404332638,
+            -0.020783845335245132,
+            0.0033992447424679995,
+            0.033151593059301376,
+            -0.0016279799165204167,
+            -0.005870908498764038,
+            0.001966396113857627,
+            0.006131083704531193,
+            -0.004841519985347986,
+            -0.007707217242568731,
+            0.012247084639966488,
+            0.01344615314155817,
+            0.004649216774851084,
+            0.006006652023643255,
+            0.06690648943185806,
+            -0.004015746526420116,
+            0.004879226442426443,
+            -0.0011707880767062306,
+            -0.0031767762266099453,
+            -0.0034011302050203085,
+            -0.001456415164284408,
+            -0.02469024434685707,
+            0.019984466955065727,
+            0.0036556492559611797,
+            -0.009057111106812954,
+            0.009675499051809311,
+            -0.021085498854517937,
+            0.0006980423349887133,
+            0.007051123306155205,
+            0.011176219210028648,
+            0.001268825144506991,
+            0.016078069806098938,
+            0.03161316365003586,
+            0.015376728028059006,
+            -0.016666291281580925,
+            0.005180879030376673,
+            0.01116113644093275,
+            0.006406341679394245,
+            -0.00740933557972312,
+            0.02408693917095661,
+            0.0013781740562990308,
+            0.023815451189875603,
+            0.0019852493423968554,
+            0.020859258249402046,
+            0.010995227843523026,
+            -0.005554173607379198,
+            0.0007465895032510161,
+            -0.008582009002566338,
+            0.0064629013650119305,
+            0.012631691992282867,
+            0.0006551511469297111,
+            0.00011624219769146293,
+            0.0008257732843048871,
+            0.011372293345630169,
+            0.00801641121506691,
+            0.0018089713994413614,
+            -0.005953862797468901,
+            -0.007213261444121599,
+            -0.00876300036907196,
+            0.010957521386444569,
+            0.0025565035175532103,
+            -0.0005443881964311004,
+            -0.01063324511051178,
+            -0.002959963632747531,
+            -0.00010993813339155167,
+            0.013913714326918125,
+            0.016605960205197334,
+            0.009358763694763184,
+            -0.012654315680265427,
+            -0.004166572820395231,
+            0.0208290945738554,
+            -0.004852831829339266,
+            0.024916483089327812,
+            0.010670951567590237,
+            0.004422977566719055,
+            0.014418981969356537,
+            -0.016651209443807602,
+            -0.016349555924534798,
+            0.0030768539290875196,
+            -0.0021323049440979958,
+            0.011508036404848099,
+            0.010640786029398441,
+            -0.002104025101289153,
+            0.020406780764460564,
+            -0.007360316812992096,
+            -0.0025828981306403875,
+            0.015731168910861015,
+            0.016832200810313225,
+            -0.004215591587126255,
+            0.005275145173072815,
+            -0.015881994739174843,
+            -0.016334474086761475,
+            -0.005655981134623289,
+            0.00012619908375199884,
+            0.005810578353703022,
+            0.012873013503849506,
+            -5.217053694650531e-05,
+            0.014607514254748821,
+            -0.005870908498764038,
+            -0.025384044274687767,
+            -0.0007234942750073969,
+            0.003993122838437557,
+            0.01603282243013382,
+            -0.007088829763233662,
+            0.002594209974631667,
+            0.0016977370250970125,
+            0.021749133244156837,
+            -0.02099500223994255,
+            0.0038762325420975685,
+            0.0003674031177069992,
+            -0.01325007900595665,
+            0.0005245922366157174,
+            -0.007978703826665878,
+            -0.012458241544663906,
+            0.015331479720771313,
+            0.005561714991927147,
+            -0.013521566055715084,
+            0.018310297280550003,
+            -0.01052766665816307,
+            -0.0012150933034718037,
+            0.02199045568704605,
+            0.010648327879607677,
+            0.004362646955996752,
+            -0.015278690494596958,
+            -0.0056748343631625175,
+            -0.014871460385620594,
+            0.022382603958249092,
+            0.01708860509097576,
+            -0.004260839428752661,
+            0.003921480383723974,
+            0.013679933734238148,
+            0.0046944646164774895,
+            -0.018883436918258667,
+            -0.010648327879607677,
+            -0.0004682681174017489,
+            -0.010210931301116943,
+            0.004728400148451328,
+            0.0072396560572087765,
+            0.02010512910783291,
+            -0.016500383615493774,
+            -0.005995340179651976,
+            -0.006119771860539913,
+            -0.02930552512407303,
+            -0.029275359585881233,
+            -0.004411665257066488,
+            0.0024867465253919363,
+            -0.0024792051408439875,
+            -0.0026545405853539705,
+            -0.0019550842698663473,
+            0.002839302644133568,
+            0.0005000830278731883,
+            -0.008137071505188942,
+            0.013740263879299164,
+            -0.000598591344896704,
+            0.005554173607379198,
+            0.022126199677586555,
+            -0.002910945098847151,
+            -0.013544189743697643,
+            0.03194498270750046,
+            0.011455247178673744,
+            -0.009743370115756989,
+            -0.027344783768057823,
+            0.01211888249963522,
+            -0.011063099838793278,
+            0.017510918900370598,
+            -0.043558597564697266,
+            -0.004962180741131306,
+            -0.0038592645432800055,
+            -0.007356546353548765,
+            -0.011606073938310146,
+            -0.011002768762409687,
+            -0.016349555924534798,
+            -0.006862590555101633,
+            -0.010949979536235332,
+            -0.022231776267290115,
+            0.007963621988892555,
+            -0.005312851630151272,
+            0.02656048908829689,
+            -0.022216694429516792,
+            0.024448921903967857,
+            0.010158142074942589,
+            -0.003913938999176025,
+            0.002268048468977213,
+            -0.008589549921452999,
+            0.00490562105551362,
+            0.020074963569641113,
+            0.010014857165515423,
+            0.00011942369019379839,
+            0.021010085940361023,
+            0.008883661590516567,
+            -0.0038290992379188538,
+            -0.0038988564629107714,
+            -0.005433512851595879,
+            0.0246751606464386,
+            0.004422977566719055,
+            0.004660528618842363,
+            -0.008076741360127926,
+            0.011055557988584042,
+            0.027043132111430168,
+            0.007111453916877508,
+            -0.009336140006780624,
+            -0.005999110639095306,
+            0.0027959400322288275,
+            -0.011658863164484501,
+            -0.008416100405156612,
+            0.011093264445662498,
+            -0.01098768599331379,
+            -0.023543965071439743,
+            0.01002993993461132,
+            0.02291049435734749,
+            0.002745036268606782,
+            -0.015881994739174843,
+            -0.005640898831188679,
+            -0.0024716637562960386,
+            0.006685370113700628,
+            0.017827652394771576,
+            -0.010980145074427128,
+            -0.0144114401191473,
+            0.010542749427258968,
+            0.004057223908603191,
+            -0.00531662255525589,
+            0.02155306003987789,
+            0.030119985342025757,
+            0.011651321314275265,
+            -0.04367925971746445,
+            -0.005663522519171238,
+            0.005867138039320707,
+            -0.013823218643665314,
+            -0.047962721437215805,
+            0.004784960299730301,
+            -0.008416100405156612,
+            -0.013649768196046352,
+            0.014456688426434994,
+            -0.009562378749251366,
+            -0.014999662525951862,
+            0.02104024961590767,
+            0.026213588193058968,
+            0.0052977693267166615,
+            0.012910719960927963,
+            -0.01418520137667656,
+            -0.019909054040908813,
+            -0.020044798031449318,
+            0.00815215427428484,
+            -0.017706992104649544,
+            -0.009607627056539059,
+            0.0023151817731559277,
+            -0.013333032839000225,
+            0.02297082543373108,
+            0.012412993237376213,
+            0.03152266889810562,
+            0.009683039970695972,
+            -0.004702006001025438,
+            -0.02529354766011238,
+            -0.01719418354332447,
+            -0.002509370446205139,
+            -0.0010963176609948277,
+            0.0049697221256792545,
+            0.005456136539578438,
+            0.017510918900370598,
+            0.012963509187102318,
+            0.009517131373286247,
+            -0.013008756563067436,
+            0.015301315113902092,
+            -2.8662863769568503e-05,
+            -0.003806475317105651,
+            -0.014290779829025269,
+            -0.029561929404735565,
+            -0.0033615382853895426,
+            0.006383717525750399,
+            -0.0040911599062383175,
+            -0.004630363546311855,
+            0.0063196164555847645,
+            0.030014406889677048,
+            -0.009192855097353458,
+            0.002863811794668436,
+            -0.037435054779052734,
+            -0.003587777493521571,
+            -0.0034539191983640194,
+            0.026847057044506073,
+            0.01156836748123169,
+            -0.008755459450185299,
+            -0.006323386915028095,
+            -0.01344615314155817,
+            0.012888096272945404,
+            0.01366485096514225,
+            -0.009683039970695972,
+            0.03312142565846443,
+            0.0020116439554840326,
+            0.01396650355309248,
+            -0.0012084946502000093,
+            -0.01002993993461132,
+            0.016078069806098938,
+            0.028883211314678192,
+            -0.011281797662377357,
+            0.012767435051500797,
+            0.01190018467605114,
+            0.005512696225196123,
+            0.0017071636393666267,
+            -0.027254289016127586,
+            0.0022341127041727304,
+            -0.01021847315132618,
+            0.0214927289634943,
+            0.000284920068224892,
+            0.0015572801930829883,
+            -0.02093467302620411,
+            0.018114222213625908,
+            -0.02506730891764164,
+            -0.005060217808932066,
+            0.012443158775568008,
+            -0.0038102460093796253,
+            0.004717088304460049,
+            -0.02186979353427887,
+            -0.010535207577049732,
+            0.015007203444838524,
+            -0.018808024004101753,
+            -0.004396582953631878,
+            -0.012707104906439781,
+            -0.01553509570658207,
+            0.021356984972953796,
+            0.009750911965966225,
+            -0.018415875732898712,
+            -0.0005943493451923132,
+            0.029396019876003265,
+            -0.027239205315709114,
+            0.010753905400633812,
+            0.0180086437612772,
+            0.016711538657546043,
+            0.01344615314155817,
+            0.001777863479219377,
+            0.00884595513343811,
+            -0.0014799818163737655,
+            -0.012473324313759804,
+            -0.020014632493257523,
+            0.0196677315980196,
+            -0.01473571639508009,
+            0.019697897136211395,
+            0.010097811929881573,
+            -0.00854430254548788,
+            -0.005407118238508701,
+            -0.004815125372260809,
+            -0.003229565219953656,
+            -0.010120435617864132,
+            -0.003642451949417591,
+            0.015776416286826134,
+            0.022955743595957756,
+            -0.004068535752594471,
+            -0.018159471452236176,
+            0.017510918900370598,
+            -0.0015959293814375997,
+            0.0015139176975935698,
+            -0.0011679601157084107,
+            -0.006195184774696827,
+            0.015059992671012878,
+            0.010120435617864132,
+            -0.0002375512121943757,
+            -0.06654450297355652,
+            -0.009539755061268806,
+            -0.004287234041839838,
+            -0.004257068503648043,
+            0.002848729258403182,
+            -0.017812570556998253,
+            -0.03951645642518997,
+            0.011191301979124546,
+            0.016892530024051666,
+            -0.01471309270709753,
+            0.01592724397778511,
+            0.005678605288267136,
+            0.0017250742530450225,
+            -0.01740534044802189,
+            -0.002826105337589979,
+            0.021507810801267624,
+            0.03251812234520912,
+            -0.025052227079868317,
+            -0.01211888249963522,
+            0.012247084639966488,
+            -0.004155260976403952,
+            -0.007775088772177696,
+            -0.021326819434762,
+            -0.00029057604842819273,
+            0.01812930591404438,
+            0.02556503564119339,
+            -0.020844176411628723,
+            -0.024011526256799698,
+            4.7928548156050965e-05,
+            0.011085723526775837,
+            -0.025761108845472336,
+            -0.009924361482262611,
+            -0.015625590458512306,
+            -0.0034086713567376137,
+            0.011206384748220444,
+            -0.011870019137859344,
+            0.007624262943863869,
+            -0.012164130806922913,
+            0.005437283311039209,
+            -0.008121988736093044,
+            -0.02485615201294422,
+            0.011470329947769642,
+            -0.006055670790374279,
+            0.035444147884845734,
+            -0.006662745960056782,
+            -0.0030900510028004646,
+            0.02726937085390091,
+            -0.0008281299378722906,
+            0.005916156340390444,
+            0.014049457386136055,
+            0.03002949059009552,
+            -0.014871460385620594,
+            0.011176219210028648,
+            -0.000854524492751807,
+            -0.004882997367531061,
+            0.0005217642756178975,
+            0.03812885656952858,
+            0.004751024302095175,
+            0.0003579764743335545,
+            -0.0009945100173354149,
+            0.0015940440353006124,
+            -0.0011538201943039894,
+            0.015188194811344147,
+            0.025323713198304176,
+            -0.0012480865698307753,
+            -0.011093264445662498,
+            -0.01576133444905281,
+            -0.013868466019630432,
+            -8.383813838008791e-05,
+            0.0030881657730787992,
+            0.011085723526775837,
+            0.014056999236345291,
+            -0.006527002435177565,
+            0.01675678789615631,
+            -0.01146278902888298,
+            0.00815215427428484,
+            -0.024222683161497116,
+            0.013189747929573059,
+            -0.011666404083371162,
+            0.0006047186907380819,
+            0.0031447256915271282,
+            -0.0015261722728610039,
+            0.01325007900595665,
+            -0.00677586579695344,
+            0.004860373213887215,
+            0.012254626490175724,
+            -0.0007579015218652785,
+            0.0024414986837655306,
+            -0.021115664392709732,
+            -0.013906172476708889,
+            -0.003118331078439951,
+            0.021945208311080933,
+            -0.0030730832368135452,
+            -0.014637679792940617,
+            -4.306793925934471e-05,
+            0.004230673890560865,
+            -0.005192190874367952,
+            0.012171671725809574,
+            0.0008700784528627992,
+            -0.00031437829602509737,
+            0.020859258249402046,
+            0.0039063976146280766,
+            -0.010346675291657448,
+            0.01556526031345129,
+            -0.003041032701730728,
+            -0.0069530862383544445,
+            -0.010120435617864132,
+            0.006508149206638336,
+            0.006229120772331953,
+            -0.005177108105272055,
+            0.026364414021372795,
+            -0.03288010507822037,
+            0.0008507538586854935,
+            0.009705663658678532,
+            -0.0014102247077971697,
+            -0.0029486515559256077,
+            0.004102471750229597,
+            0.04105488210916519,
+            -0.009637792594730854,
+            -0.023830534890294075,
+            0.0002169304498238489,
+            0.01769191026687622,
+            -0.011492953635752201,
+            0.002967504784464836,
+            0.01823488436639309,
+            0.004890538286417723,
+            0.0034859697334468365,
+            0.010007316246628761,
+            -0.03704290837049484,
+            -0.004471995867788792,
+            -0.007940997369587421,
+            -0.011206384748220444,
+            0.007186866831034422,
+            0.02088942378759384,
+            0.01690761372447014,
+            0.014026833698153496,
+            0.005784183740615845,
+            0.030949529260396957,
+            0.0037970487028360367,
+            -0.003135299077257514,
+            0.0016317506087943912,
+            0.00720194960013032,
+            -0.013008756563067436,
+            0.018099140375852585,
+            -0.008137071505188942,
+            -0.015520012937486172,
+            0.004260839428752661,
+            -0.009569920599460602,
+            0.004815125372260809,
+            -0.011032934300601482,
+            0.0051243188790977,
+            -0.006904067937284708,
+            0.0005476875230669975,
+            -0.007439500652253628,
+            -0.00021657695469912142,
+            0.013076628558337688,
+            -3.543825732776895e-05,
+            0.011221466585993767,
+            0.012239543721079826,
+            -0.009057111106812954,
+            -0.024644995108246803,
+            0.012443158775568008,
+            -0.008031493052840233,
+            0.005772871430963278,
+            -0.010927355848252773,
+            0.0009073136607185006,
+            -0.0010576684726402164,
+            0.03791769966483116,
+            0.025414209812879562,
+            -0.0003365308803040534,
+            0.00593878049403429,
+            -0.028551394119858742,
+            0.007360316812992096,
+            -0.024991896003484726,
+            0.009826324880123138,
+            -0.013958961702883244,
+            -0.008287898264825344,
+            0.012171671725809574,
+            0.028460897505283356,
+            -0.0046529872342944145,
+            0.004226903431117535,
+            -0.007266050670295954,
+            -0.020150376483798027,
+            -0.007895749993622303,
+            -0.006530772894620895,
+            -0.011500495485961437,
+            -0.02055760659277439,
+            -0.010286344215273857,
+            0.01785781793296337,
+            -0.009026946499943733,
+            -0.0013678048271685839,
+            0.015286232344806194,
+            -0.0017995446687564254,
+            0.005490072537213564,
+            -0.017058439552783966,
+            0.0038837739266455173,
+            0.0062178089283406734,
+            0.01690761372447014,
+            -0.01553509570658207,
+            0.012209378182888031,
+            0.0033332582097500563,
+            -0.013091711327433586,
+            0.008106906898319721,
+            -0.021070415154099464,
+            -0.015082616358995438,
+            -0.0003834283852484077,
+            0.007695905398577452,
+            -0.010535207577049732,
+            -0.030919363722205162,
+            0.009554837830364704,
+            -0.0043777297250926495,
+            -0.007511143106967211,
+            -0.01613840088248253,
+            0.009901737794280052,
+            0.0022322272416204214,
+            0.03387555852532387,
+            -0.017284678295254707,
+            -0.006741929799318314,
+            0.019712980836629868,
+            0.009109900332987309,
+            -0.01642496883869171,
+            -0.009366304613649845,
+            0.004249527119100094,
+            -0.0012886211043223739,
+            0.006821113638579845,
+            -0.01463013794273138,
+            -0.009939444251358509,
+            0.0031145603861659765,
+            0.005637127906084061,
+            0.019471658393740654,
+            0.005324163939803839,
+            -0.0135818962007761,
+            -0.0032653864473104477,
+            0.009351222775876522,
+            0.006911609321832657,
+            0.009388929232954979,
+            -0.016349555924534798,
+            0.002688476350158453,
+            -0.012254626490175724,
+            0.002351003000512719,
+            0.011628697626292706,
+            -0.006361093837767839,
+            -0.006342240143567324,
+            0.022895412519574165,
+            0.017933230847120285,
+            -0.0018061433220282197,
+            -0.013234996236860752,
+            0.0004694464441854507,
+            -0.0002575828111730516,
+            0.01281268335878849,
+            -0.006557167507708073,
+            -0.005595650989562273,
+            0.0013838300947099924,
+            0.01576133444905281,
+            0.003944104071706533,
+            -0.005090383347123861,
+            -0.013943878933787346,
+            -0.015384268946945667,
+            0.0007640288094989955,
+            -0.027480527758598328,
+            0.0033502262085676193,
+            0.0014385045506060123,
+            0.020452028140425682,
+            0.017812570556998253,
+            0.019019179046154022,
+            -0.014358650892972946,
+            0.001570477499626577,
+            0.005007428582757711,
+            -0.00812953058630228,
+            0.002094598487019539,
+            -0.009879114106297493,
+            -0.0190040972083807,
+            -0.006504378281533718,
+            0.0058595966547727585,
+            0.018370626494288445,
+            0.004656758159399033,
+            -0.004309857729822397,
+            0.029411103576421738,
+            0.019366079941391945,
+            -0.02309148572385311,
+            0.011749358847737312,
+            -0.024554500356316566,
+            0.010301426984369755,
+            0.01962248422205448,
+            0.004592656623572111,
+            -0.016078069806098938,
+            -0.024554500356316566,
+            0.0050111995078623295,
+            -0.010301426984369755,
+            -0.008506596088409424,
+            0.008302980102598667,
+            -0.023785287514328957,
+            0.018084056675434113,
+            -0.011975597590208054,
+            0.0018268820131197572,
+            -0.010844401083886623,
+            -0.004241986200213432,
+            -0.010542749427258968,
+            -0.0031748907640576363,
+            0.007115224376320839,
+            0.01690761372447014,
+            0.005388265009969473,
+            -0.03577596694231033,
+            -0.011010310612618923,
+            0.015580343082547188,
+            -0.00520727364346385,
+            -0.0094190938398242,
+            -0.0020436944905668497,
+            0.019652649760246277,
+            -0.02164355479180813,
+            -0.006764553487300873,
+            -0.0376763753592968,
+            -0.02022578939795494,
+            -0.02755594067275524,
+            -0.028340237215161324,
+            -0.01314450055360794,
+            -0.004615280777215958,
+            0.015550177544355392,
+            0.0017986020538955927,
+            -0.008574468083679676,
+            -0.0008361425716429949,
+            -0.0108293192461133,
+            -0.0003174419398419559,
+            -0.010263720527291298,
+            -0.006417653523385525,
+            0.0033596528228372335,
+            0.007375399582087994,
+            0.002143617020919919,
+            -0.012262167409062386,
+            -0.008039034903049469,
+            -0.011809688992798328,
+            -0.010595538653433323,
+            0.00854430254548788,
+            -0.003346455516293645,
+            -0.019682815298438072,
+            0.016108235344290733,
+            0.008197402581572533,
+            0.012005763128399849,
+            0.013189747929573059,
+            0.0009275809279642999,
+            0.032095808535814285,
+            6.698803190374747e-05,
+            -0.0016854824498295784,
+            -6.112583832873497e-06,
+            -0.009026946499943733,
+            0.015376728028059006,
+            0.0077298409305512905,
+            0.008325604721903801,
+            -0.004079848062247038,
+            0.006293221842497587,
+            -0.007443271577358246,
+            0.0058181192725896835,
+            0.00018040223221760243,
+            -0.007971162907779217,
+            0.0003205056127626449,
+            0.007695905398577452,
+            0.002927913097664714,
+            -0.01418520137667656,
+            0.006817342713475227,
+            0.012699563056230545,
+            0.006934233009815216,
+            -0.01576133444905281,
+            -0.01380059402436018,
+            -0.012156588956713676,
+            -0.008499054238200188,
+            -0.028551394119858742,
+            0.022955743595957756,
+            0.043830085545778275,
+            0.000945962849073112,
+            -0.0190040972083807,
+            -0.0044380598701536655,
+            -0.015444600023329258,
+            -0.015007203444838524,
+            -0.0056748343631625175,
+            -0.005912385880947113,
+            -0.02375512197613716,
+            -0.008114447817206383,
+            0.015278690494596958,
+            0.014577348716557026,
+            0.000577381404582411,
+            -0.01570100337266922,
+            -0.005953862797468901,
+            -0.015987573191523552,
+            -0.013302868232131004,
+            -0.00908727664500475,
+            -0.021960290148854256,
+            -0.012518571689724922,
+            0.012043469585478306,
+            -0.01613840088248253,
+            0.015836747363209724,
+            0.020421862602233887,
+            0.00380836077965796,
+            -0.014109788462519646,
+            -0.015437058173120022,
+            -0.011289338581264019,
+            0.01198313944041729,
+            0.010912273079156876,
+            -0.010723740793764591,
+            0.03107019141316414,
+            -0.016440052539110184,
+            -0.009524672292172909,
+            -0.0033596528228372335,
+            0.007217032369226217,
+            -0.014260614290833473,
+            -0.004147719591856003,
+            -0.016621043905615807,
+            0.01220183726400137,
+            -0.014275697059929371,
+            0.006459130439907312,
+            0.007250967901200056,
+            -0.016334474086761475,
+            0.021326819434762,
+            0.025263383984565735,
+            0.0018579898169264197,
+            0.007635574787855148,
+            0.009962067939341068,
+            -0.013928796164691448,
+            0.010784070938825607,
+            -0.004984804894775152,
+            -0.0012575131841003895,
+            0.028129080310463905,
+            0.008536760695278645,
+            -0.0027506921906024218,
+            -0.015188194811344147,
+            0.013627144508063793,
+            0.020798929035663605,
+            -0.0023604296147823334,
+            -0.006870131939649582,
+            -0.013657309114933014,
+            0.013695016503334045,
+            0.0066363513469696045,
+            -0.00605944124981761,
+            -0.0075978683307766914,
+            -0.008989240042865276,
+            -0.0019230336183682084,
+            -0.021779298782348633,
+            0.012149048037827015,
+            -0.023830534890294075,
+            0.00014540585107170045,
+            -0.009494507685303688,
+            0.009667957201600075,
+            -0.0060028815641999245,
+            0.009939444251358509,
+            -0.010806694626808167,
+            0.013619602657854557,
+            0.007367858197540045,
+            0.0004548351571429521,
+            -0.011274255812168121,
+            0.017767323181033134,
+            0.004260839428752661,
+            0.00856692623347044,
+            0.00245281052775681,
+            -0.019170006737113,
+            0.030557380989193916,
+            -0.006625039502978325,
+            0.009434176608920097,
+            -0.010799153707921505,
+            0.012639232911169529,
+            0.021417316049337387,
+            0.028732385486364365,
+            0.011741816997528076,
+            0.0034765431191772223,
+            -0.015225901268422604,
+            0.024147270247340202,
+            0.0015846174210309982,
+            -0.010075188241899014,
+            0.02016545832157135,
+            0.004159031435847282,
+            0.005195961333811283,
+            -0.007032270077615976,
+            0.0090043218806386,
+            -0.02423776499927044,
+            -0.02551978826522827,
+            -0.007511143106967211,
+            -0.007582785561680794,
+            -0.009328598156571388,
+            0.01495441421866417,
+            -0.0017297875601798296,
+            -0.0019946759566664696,
+            0.0037178650964051485,
+            0.0032031708396971226,
+            0.007496060337871313,
+            0.002641343278810382,
+            -0.013574355281889439,
+            -0.012164130806922913,
+            -0.016711538657546043,
+            0.019215254113078117,
+            -0.008106906898319721,
+            0.001139680272899568,
+            -0.0070247286930680275,
+            0.021417316049337387,
+            0.010188307613134384,
+            0.007160472217947245,
+            0.012707104906439781,
+            -0.006877673324197531,
+            0.01729976199567318,
+            0.004649216774851084,
+            -0.012435617856681347,
+            0.015987573191523552,
+            0.026258835569024086,
+            -0.0012443158775568008,
+            -0.00364999333396554,
+            0.00190418038982898,
+            -0.04192967340350151,
+            0.00040935163269750774,
+            -0.014041916467249393,
+            0.018702445551753044,
+            -0.007888208143413067,
+            0.004988575354218483,
+            0.01459997333586216,
+            -0.005350558087229729,
+            -0.0010887763928622007,
+            0.014011750929057598,
+            0.007326381281018257,
+            0.015670839697122574,
+            0.00720194960013032,
+            0.015550177544355392,
+            -0.013657309114933014,
+            -0.03069312497973442,
+            -0.008174777962267399,
+            -0.003989351913332939,
+            0.015437058173120022,
+            0.0037103237118571997,
+            0.016228895634412766,
+            -0.030949529260396957,
+            -0.005116777960211039,
+            -0.006296992301940918,
+            0.002047465182840824,
+            0.014705551788210869,
+            -0.00354630034416914,
+            0.018038809299468994,
+            0.015489847399294376,
+            -0.008755459450185299,
+            -0.02159830741584301,
+            -0.020844176411628723,
+            0.013936338014900684,
+            0.010060105472803116,
+            -0.020361533388495445,
+            0.0020512358751147985,
+            -0.008634798228740692,
+            -0.006244203541427851,
+            -0.02963734231889248,
+            0.024283012375235558,
+            -0.003429410047829151,
+            0.01305400487035513,
+            0.011636238545179367,
+            0.010482418350875378,
+            0.013521566055715084,
+            0.02942618541419506,
+            -0.004690693691372871,
+            0.009683039970695972,
+            0.005984028335660696,
+            -0.009275808930397034,
+            -0.007164243143051863,
+            -0.009864031337201595,
+            0.015881994739174843,
+            -0.0019701668061316013,
+            0.0027412655763328075,
+            -0.003182432148605585,
+            0.017013192176818848,
+            -0.004626592621207237,
+            -0.024056773632764816,
+            -0.004204279277473688,
+            -0.001689253025688231,
+            -0.010429629124701023,
+            0.011802148073911667,
+            -0.008506596088409424,
+            -0.000715952948667109,
+            0.009735829196870327,
+            -0.004607739392668009,
+            -0.008989240042865276,
+            0.006892756093293428,
+            0.0002959963458124548,
+            -0.04117554426193237,
+            0.0005467448499985039,
+            0.004027058836072683,
+            -0.017752239480614662,
+            0.004668070003390312,
+            -0.0076280334033071995,
+            -0.010346675291657448,
+            0.0005962346913293004,
+            0.005927468184381723,
+            0.003774424782022834,
+            0.0221563633531332,
+            -0.0018711871234700084,
+            0.01438127551227808,
+            -0.006862590555101633,
+            -0.0024942876771092415,
+            -0.009750911965966225,
+            0.0135818962007761,
+            -0.020949754863977432,
+            -0.014720633625984192,
+            -0.01580658182501793,
+            -0.005007428582757711,
+            0.004577574320137501,
+            0.00458511570468545,
+            0.009539755061268806,
+            0.0013272702926769853,
+            -0.006444048136472702,
+            0.010701116174459457,
+            -0.006413882598280907,
+            -0.00032121260301209986,
+            -0.013016298413276672,
+            0.008446265012025833,
+            0.02357413060963154,
+            0.005075300578027964,
+            -0.0064930664375424385,
+            -0.013016298413276672,
+            0.01586691290140152,
+            0.03441853076219559,
+            -0.023016072809696198,
+            0.008664963766932487,
+            0.019712980836629868,
+            -0.0031371843069791794,
+            -0.002475434448570013,
+            -0.011583449319005013,
+            0.003039147239178419,
+            -0.019471658393740654,
+            0.009796159341931343,
+            0.006873902399092913,
+            -0.008227567188441753,
+            0.004882997367531061,
+            -0.015776416286826134,
+            0.0017165903700515628,
+            -0.002958078170195222,
+            -0.008710211142897606,
+            -0.005557944066822529,
+            -0.012692022137343884,
+            -0.006078294478356838,
+            0.0035180202685296535,
+            0.008393475785851479,
+            -0.004498390480875969,
+            0.01856670156121254,
+            -0.004038370680063963,
+            0.02253342978656292,
+            0.0022416538558900356,
+            -0.001372518134303391,
+            -0.009124983102083206,
+            0.0070247286930680275,
+            0.00026041080127470195,
+            -0.004769877530634403,
+            0.0017948313616216183,
+            -0.009072193875908852,
+            0.011334586888551712,
+            -0.0034312952775508165,
+            0.01752600073814392,
+            0.007009646389633417,
+            -0.008717752061784267,
+            -0.006308304611593485,
+            0.0027714308816939592,
+            -0.005712541285902262,
+            0.012322497554123402,
+            -0.004592656623572111,
+            0.01894376613199711,
+            -0.013423528522253036,
+            -0.018702445551753044,
+            0.004717088304460049,
+            0.008476430550217628,
+            0.013287785463035107,
+            -0.009532214142382145,
+            0.01002993993461132,
+            -0.0037819661665707827,
+            0.01823488436639309,
+            0.01806897483766079,
+            -0.007243426516652107,
+            0.015007203444838524,
+            -0.01962248422205448,
+            -0.0013781740562990308,
+            0.012737269513309002,
+            -0.002754462882876396,
+            -0.020361533388495445,
+            -0.011651321314275265,
+            -0.018928684294223785,
+            0.010489960201084614,
+            -0.015489847399294376,
+            0.023709872737526894,
+            0.011274255812168121,
+            0.010572914034128189,
+            -0.001968281576409936,
+            0.006198955699801445,
+            -0.0019202056573703885,
+            0.014539642259478569,
+            -0.0016835971036925912,
+            -0.004864144138991833,
+            -0.010595538653433323,
+            -0.002262392546981573,
+            -0.04057224094867706,
+            8.725529187358916e-05,
+            0.014698009938001633,
+            0.03354373946785927,
+            -0.02170388586819172,
+            0.014336027204990387,
+            -0.011651321314275265,
+            -0.0190040972083807,
+            -0.012043469585478306,
+            0.015158030204474926,
+            -0.015263608656823635,
+            0.019170006737113,
+            -0.009743370115756989,
+            -0.0047661070711910725,
+            0.001808028668165207,
+            0.01201330404728651,
+            0.003014638088643551,
+            0.0039478749968111515,
+            0.02402660809457302,
+            0.015203277580440044,
+            0.00648552505299449,
+            -0.003024064702913165,
+            0.011515578255057335,
+            0.00458511570468545,
+            -0.014863918535411358,
+            -0.005463677924126387,
+            -0.0011594761162996292,
+            -0.007639345247298479,
+            -0.010618162341415882,
+            -0.008529219776391983,
+            -0.0037800809368491173,
+            0.0008403845713473856,
+            0.018747692927718163,
+            -0.006010422948747873,
+            -0.0012942770263180137,
+            -0.012299873866140842,
+            0.012164130806922913,
+            -0.0011886986903846264,
+            0.018928684294223785,
+            -0.027359867468476295,
+            0.026741480454802513,
+            -0.03444869816303253,
+            -0.01388354878872633,
+            -0.006632580887526274,
+            -0.00015966364298947155,
+            0.011809688992798328,
+            0.0028845504857599735,
+            -0.015836747363209724,
+            -0.0008521678391844034,
+            -0.006930462550371885,
+            0.002935454249382019,
+            -0.003197514684870839,
+            0.009524672292172909,
+            0.005855826195329428,
+            -0.005659752059727907,
+            -0.021945208311080933,
+            -0.008891202509403229,
+            -0.012563819997012615,
+            -0.008355769328773022,
+            -0.013438611291348934,
+            0.013627144508063793,
+            -0.003331372980028391,
+            0.0006655204924754798,
+            0.023000990971922874,
+            -0.014283237978816032,
+            -0.01613840088248253,
+            0.015082616358995438,
+            -0.003555726958438754,
+            -0.0056446692906320095,
+            -0.01305400487035513,
+            0.007775088772177696,
+            0.0010850057005882263,
+            0.018249966204166412,
+            -0.008876119740307331,
+            0.022337354719638824,
+            -0.0045097023248672485,
+            0.012322497554123402,
+            0.018099140375852585,
+            -0.023453468456864357,
+            -0.030180316418409348,
+            0.004219362046569586,
+            -0.02793300710618496,
+            -0.0010472992435097694,
+            0.0019739374984055758,
+            -0.024946648627519608,
+            -0.014946873299777508,
+            -0.008853496052324772,
+            -0.0255047045648098,
+            -0.01630430854856968,
+            0.012918260879814625,
+            -0.0006914437399245799,
+            0.014253072440624237,
+            0.011621156707406044,
+            -0.016213813796639442,
+            0.01553509570658207,
+            -0.014418981969356537,
+            -0.02490140125155449,
+            0.0008055060170590878,
+            -0.008506596088409424,
+            0.012910719960927963,
+            0.00406476529315114,
+            0.001550681539811194,
+            0.009622709825634956,
+            -0.005320393014699221,
+            -0.0018759004306048155,
+            0.019049344584345818,
+            0.02186979353427887,
+            0.004275921732187271,
+            -0.00418542604893446,
+            0.00616879016160965,
+            -0.016651209443807602,
+            0.029275359585881233,
+            -0.01220183726400137,
+            -0.015097699128091335,
+            0.025097474455833435,
+            0.006285680457949638,
+            0.019215254113078117,
+            0.004453142639249563,
+            0.005128089804202318,
+            0.001582732074894011,
+            0.011357210576534271,
+            -0.0015261722728610039,
+            -0.01104801706969738,
+            -0.014388816431164742,
+            -0.018325379118323326,
+            0.030889200046658516,
+            0.008174777962267399,
+            0.017616497352719307,
+            -0.028672054409980774,
+            0.010799153707921505,
+            -0.008506596088409424,
+            -0.012888096272945404,
+            0.009818783961236477,
+            0.028445815667510033,
+            0.05140155926346779,
+            0.022578677162528038,
+            0.005735164973884821,
+            -0.000906370987650007,
+            0.034297872334718704,
+            -0.01303138118237257,
+            -0.009622709825634956,
+            -0.006455359980463982,
+            -0.01686236634850502,
+            0.004011976066976786,
+            0.012986132875084877,
+            -0.003627369413152337,
+            -0.008393475785851479,
+            -0.010867025703191757,
+            0.03541398420929909,
+            0.005780412815511227,
+            -0.010143060237169266,
+            0.013868466019630432,
+            -0.01956215314567089,
+            0.012216920033097267,
+            -0.017510918900370598,
+            -0.019049344584345818,
+            -0.013415987603366375,
+            0.0009813127107918262,
+            -0.02313673496246338,
+            0.006757012102752924,
+            -0.00854430254548788,
+            0.018913600593805313,
+            -0.010060105472803116,
+            -0.006017964333295822,
+            0.0034331807401031256,
+            -0.005946321878582239,
+            -0.014373733662068844,
+            0.00480758398771286,
+            -0.008023952133953571,
+            0.020783845335245132,
+            0.0005665408098138869,
+            0.008800706826150417,
+            0.0061348541639745235,
+            0.006425194907933474,
+            -0.0028298760298639536,
+            0.0012377172242850065,
+            0.012186754494905472,
+            0.015912160277366638,
+            0.009404012002050877,
+            -0.030346224084496498,
+            0.004253298044204712,
+            -0.013717640191316605,
+            0.014977038837969303,
+            0.00812953058630228,
+            0.01962248422205448,
+            0.016183648258447647,
+            0.013355657458305359,
+            -9.674084867583588e-05,
+            0.004547408781945705,
+            -0.0012961623724550009,
+            -0.0017043356783688068,
+            0.008499054238200188,
+            -0.00024226453388109803,
+            -0.0069530862383544445,
+            -0.0012047240743413568,
+            0.008031493052840233,
+            -0.008921368047595024,
+            -0.019094591960310936,
+            -0.00184950593393296,
+            -0.002364200074225664,
+            -0.01873260922729969,
+            0.0033671942073851824,
+            0.020180542021989822,
+            -0.015987573191523552,
+            -0.0016562598757445812,
+            0.01846112310886383,
+            0.02153797633945942,
+            -0.008204943500459194,
+            0.009230561554431915,
+            -0.004988575354218483,
+            -0.010693575255572796,
+            -0.02766151912510395,
+            -0.000485943048261106,
+            -0.02814416214823723,
+            0.035564810037612915,
+            -0.00886857882142067,
+            -0.018687361851334572,
+            0.012488407082855701,
+            -0.006560938432812691,
+            -0.01713385246694088,
+            -0.0068512787111103535,
+            0.005075300578027964,
+            -0.030165232717990875,
+            0.033966053277254105,
+            -0.016666291281580925,
+            -0.0009907394414767623,
+            0.020482193678617477,
+            0.05173337459564209,
+            -0.0031692348420619965,
+            -0.01746566966176033,
+            0.017254512757062912,
+            0.014426522888243198,
+            0.03188465163111687,
+            -0.01576133444905281,
+            -0.017390256747603416,
+            -0.010814236477017403,
+            -0.012873013503849506,
+            -0.011990680359303951,
+            -0.020814010873436928,
+            0.03604745492339134,
+            -0.012503488920629025,
+            -0.016726622357964516,
+            -0.0040496825240552425,
+            -0.008167237043380737,
+            -0.011289338581264019,
+            0.022654090076684952,
+            0.027450362220406532,
+            -0.0108293192461133,
+            -0.012963509187102318,
+            -0.0012235773028805852,
+            -0.023227229714393616,
+            0.02088942378759384,
+            -0.0005038536619395018,
+            -0.014698009938001633,
+            0.010867025703191757,
+            -0.0036914704833179712,
+            -0.0016949090640991926,
+            0.002579127438366413,
+            0.004751024302095175,
+            0.0569821260869503,
+            -0.015075075440108776,
+            0.002515026368200779,
+            -0.026515239849686623,
+            0.016213813796639442,
+            -0.014863918535411358,
+            -0.012149048037827015,
+            -0.013302868232131004,
+            -0.0031296429224312305,
+            -0.0158216655254364,
+            -0.02737494930624962,
+            -0.001026560552418232,
+            -0.017616497352719307,
+            0.005162025801837444,
+            -0.014464229345321655,
+            0.008069200441241264,
+            -0.024825986474752426,
+            -0.013506483286619186,
+            -0.011063099838793278,
+            -0.024373508989810944,
+            0.015082616358995438,
+            -0.030617712065577507,
+            -0.007790171541273594,
+            0.014449146576225758,
+            0.015218360349535942,
+            -0.007058664690703154,
+            -0.012179212644696236,
+            -0.0075563909485936165,
+            -0.00038366406806744635,
+            -0.05520237609744072,
+            0.002515026368200779,
+            -0.012699563056230545,
+            0.006836195942014456,
+            -0.028310071676969528,
+            0.005146943032741547,
+            0.014351109974086285,
+            0.023000990971922874,
+            0.024267930537462234,
+            0.008385934866964817,
+            0.0024490398354828358,
+            -0.01989397220313549,
+            0.02689230628311634,
+            0.005252521485090256,
+            -0.021839629858732224,
+            0.01313695963472128,
+            -0.003996893297880888,
+            -0.00015777831140439957,
+            -0.01303138118237257,
+            -0.005248750559985638,
+            0.012284791097044945,
+            0.007390482351183891,
+            -0.005791724659502506,
+            -0.007971162907779217,
+            -0.013159583322703838,
+            -0.002450925298035145,
+            0.00605944124981761,
+            0.010467336513102055,
+            0.005803036969155073,
+            0.01044471189379692,
+            -0.016847282648086548,
+            0.0029297983273863792,
+            0.005263833329081535,
+            -0.00219075009226799,
+            0.003046688623726368,
+            -0.007744923699647188,
+            0.01923033595085144,
+            -0.013461235910654068,
+            -0.00782033707946539,
+            0.00209271302446723,
+            -0.02259376086294651,
+            -0.016832200810313225,
+            0.009313516318798065,
+            -0.027902841567993164,
+            0.016500383615493774,
+            -0.002175667556002736,
+            -0.002022956032305956,
+            -0.007903290912508965,
+            0.012465782463550568,
+            0.0031635789200663567,
+            -0.01495441421866417,
+            -0.0007597868097946048,
+            0.0009223962551914155,
+            -0.028883211314678192,
+            0.011621156707406044,
+            0.015459681861102581,
+            -0.006029276177287102,
+            -0.00030047399923205376,
+            -0.029018955305218697,
+            0.009494507685303688,
+            0.0001621381234144792,
+            -0.001130253542214632,
+            0.008793165907263756,
+            0.020255954936146736,
+            0.008227567188441753,
+            -0.012759894132614136,
+            0.012827765196561813,
+            0.013393363915383816,
+            0.005806807428598404,
+            0.00752245495095849,
+            0.024644995108246803,
+            -0.0047962721437215805,
+            0.011621156707406044,
+            -0.011651321314275265,
+            0.00961516797542572,
+            -0.0104145472869277,
+            0.0004543638206087053,
+            0.016605960205197334,
+            0.012307415716350079,
+            -0.005459907464683056,
+            0.005033823195844889,
+            -0.015474764630198479,
+            0.014909166842699051,
+            -0.020633019506931305,
+            0.0008964730077423155,
+            0.012737269513309002,
+            -0.0036066307220607996,
+            0.005599421449005604,
+            0.0135818962007761,
+            0.03830984607338905,
+            0.015851831063628197,
+            0.012480865232646465,
+            0.006093377247452736,
+            -0.02616834081709385,
+            0.003171120071783662,
+            0.0023114110808819532,
+            0.006119771860539913,
+            -0.0024471546057611704,
+            0.004539867863059044,
+            -0.010459794662892818,
+            0.00020526499429252,
+            0.010210931301116943,
+            -0.017118770629167557,
+            0.012865472584962845,
+            -0.010919814929366112,
+            0.007345234509557486,
+            -0.004445601254701614,
+            -0.010919814929366112,
+            -0.021221240982413292,
+            -0.005999110639095306,
+            0.018265048041939735,
+            -0.031824320554733276,
+            0.010716198943555355,
+            0.007039811462163925,
+            0.0288228802382946,
+            -0.02892845869064331,
+            -0.011824771761894226,
+            0.0033841622062027454,
+            -0.006108460016548634,
+            -0.009215478785336018,
+            0.00837085209786892,
+            0.01767682656645775,
+            -0.009652874432504177,
+            0.01534656248986721,
+            0.010821777395904064,
+            -0.006236662156879902,
+            0.03924496844410896,
+            0.007250967901200056,
+            0.016108235344290733,
+            -0.007895749993622303,
+            0.005546632222831249,
+            0.005490072537213564,
+            0.01576133444905281,
+            0.008061658591032028,
+            -0.02021070569753647,
+            0.0027016736567020416,
+            -0.0011566481553018093,
+            0.024539416655898094,
+            0.0052562919445335865,
+            0.005595650989562273,
+            0.0020663184113800526,
+            0.00731506897136569,
+            0.015987573191523552,
+            0.015271149575710297,
+            -0.007213261444121599,
+            -0.007130307145416737,
+            0.007624262943863869,
+            0.003165464149788022,
+            -0.003484084503725171,
+            0.001214150688610971,
+            -0.012518571689724922,
+            -0.004053453449159861,
+            0.017013192176818848,
+            -0.0067381588742136955,
+            -0.001416823361068964,
+            -0.012737269513309002,
+            -0.02452433481812477,
+            0.013453694060444832,
+            0.0018674165476113558,
+            -0.011651321314275265,
+            0.025549951940774918,
+            -0.007695905398577452,
+            -0.011123429983854294,
+            -0.01625906117260456,
+            -0.016334474086761475,
+            0.0139966681599617,
+            0.018280131742358208,
+            0.0010058219777420163,
+            0.0046831523068249226,
+            -0.003197514684870839,
+            -0.0006815457600168884,
+            0.022895412519574165,
+            -0.010776530019938946,
+            0.009728288277983665,
+            -0.002498058369383216
+          ],
+          "keyphrases": [
+            "routing method",
+            "sparsely activated mixture-of-experts",
+            "load imbalance",
+            "training efficiency improvements",
+            "GLUE and SuperGLUE benchmark"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": true,
+      "properties": {
+        "cosine_similarity": 0.8159407751118963
+      }
+    },
+    {
+      "id": "400a0d01-3f33-48b1-a174-2de9e76581e6",
+      "type": "cosine_similarity",
+      "source": {
+        "id": "3639a96c-e9a4-45ed-a3ac-1e18b2cfe8d4",
+        "properties": {
+          "page_content": "2 Related Work\n\nScaling: Various approaches have been proposed to scale up neural network capacity to improve performance. Recent works have successfully scaled models to billions of parameters via various forms of model parallelism [2, 21, 26, 27, 33]. Model parallelism [30] splits weights and tensors across multiple cores while pipeline parallelism [18, 24] splits different layers across devices with micro-batches pipelined to the different layers. To enable continued scaling of neural networks, improving model training and serving efficiency has become a critical research area.\n\nConditional Computation: Computation decisions can be made dynamically based on the input [23, 25]. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation, by activating certain parameters and computation on demand, on a per-example or per-token basis [3]. Conditional convolution layers [1] with task-specific gating has been used to combat catastrophic forgetting when a sequence of learning problems are optimized. The gating decisions may be binary or sparse and continuous, stochastic or deterministic.\n\nMixture of Experts: Sparsely-gated MoE [31] is the first model to demonstrate massive improvements in model capacity, training time, or model quality with gating. Switch Transformer [10] simplifies the gating by selecting only the top expert per token using a softmax over the hidden state and demonstrates better scaling than previous work. All the prior work requires an auxiliary loss to explicitly encourage balancing. This loss term has to be carefully weighted to not overwhelm the primary loss. However, auxiliary loss does not guarantee balancing and a hard capacity factor has to be imposed. As a result, many tokens can still be unprocessed by the MoE layer. Hard MoE [12] with a single decoding layer can be efficiently trained to good effect on large scale hashtag prediction tasks.\n\nBase Layers [22] formulate a linear assignment that maximizes token-expert affinities while ensuring each expert receives an equal number of tokens. Hash layers [8, 28] devise hashing techniques on input tokens. However, the evaluations are limited to pre-training perplexity. THOR [? ] randomly activates experts during training and inference and is trained with a consistency regularization loss.\n\nTHOR has demonstrated strong performance on translation tasks. Different from these prior works, our method is a learnt method that enables heterogeneous MoE and effectively improves downstream fine-tuning performance.\n\n",
+          "embedding": [
+            -0.013259558007121086,
+            -0.007197607308626175,
+            -0.015822459012269974,
+            0.013052377849817276,
+            0.029680538922548294,
+            0.02572108618915081,
+            -0.01866159960627556,
+            0.040392544120550156,
+            -0.039103418588638306,
+            0.017602676525712013,
+            0.020380431786179543,
+            -0.012024147436022758,
+            0.0011442891554906964,
+            0.006088807247579098,
+            0.009645408019423485,
+            -0.014533335343003273,
+            -0.004703766666352749,
+            0.0049493140541017056,
+            0.02260570414364338,
+            -0.03012559376657009,
+            -0.008847379125654697,
+            -0.007220627274364233,
+            -0.03612615540623665,
+            -0.01026694942265749,
+            0.03443801775574684,
+            0.0267186239361763,
+            0.03146075829863548,
+            -0.011226118542253971,
+            -0.04496586322784424,
+            0.025629006326198578,
+            0.015055122785270214,
+            0.009300106205046177,
+            -0.016912074759602547,
+            -0.01592988520860672,
+            -0.0355122871696949,
+            -0.04048462212085724,
+            0.05073622614145279,
+            0.013781346380710602,
+            -0.01992003060877323,
+            0.02774685248732567,
+            0.021807676181197166,
+            0.0150397764518857,
+            -0.03502119332551956,
+            -0.02219134382903576,
+            0.00023379754566121846,
+            0.0007141016540117562,
+            -0.032136011868715286,
+            -0.009069905616343021,
+            -0.008317917585372925,
+            -0.00694822333753109,
+            0.028268640860915184,
+            -0.04450546205043793,
+            0.01573037914931774,
+            0.012584302574396133,
+            -0.038274697959423065,
+            0.039932142943143845,
+            0.013228864409029484,
+            0.005678283050656319,
+            0.053621407598257065,
+            0.03726181387901306,
+            0.011333545669913292,
+            0.006449454929679632,
+            -0.04613221064209938,
+            0.002123601036146283,
+            0.010443436913192272,
+            -0.008847379125654697,
+            -0.00621158117428422,
+            0.0015471403021365404,
+            0.0041819787584245205,
+            0.010289969854056835,
+            -0.0367400236427784,
+            -0.0014166932087391615,
+            -0.011893700808286667,
+            -0.003069342114031315,
+            0.010834777727723122,
+            0.001008087070658803,
+            0.019367549568414688,
+            0.039011336863040924,
+            0.03511327505111694,
+            -0.008678564801812172,
+            -0.01622147299349308,
+            0.03317958861589432,
+            -0.002503432100638747,
+            0.016574447974562645,
+            0.02119380794465542,
+            -0.03713903948664665,
+            -0.0017399332718923688,
+            -0.06629779189825058,
+            -0.018983881920576096,
+            -0.01323653757572174,
+            -0.02991073951125145,
+            0.0056015495210886,
+            -0.01734178327023983,
+            -0.0033398279920220375,
+            -0.020273005589842796,
+            0.04646984115242958,
+            -0.018446745350956917,
+            -0.013482085429131985,
+            -0.032872654497623444,
+            0.004730623681098223,
+            -0.01812446489930153,
+            0.03631031885743141,
+            -0.0027451429050415754,
+            -0.0076541719026863575,
+            0.01571503095328808,
+            0.009092926047742367,
+            -0.05006096884608269,
+            0.034622177481651306,
+            -0.00013188579760026187,
+            0.074830561876297,
+            0.012898909859359264,
+            -0.033210281282663345,
+            0.00029134770738892257,
+            0.04054601117968559,
+            -0.03152214363217354,
+            -0.02452404424548149,
+            -0.04149750620126724,
+            -0.03597268834710121,
+            0.001932726358063519,
+            -0.028406761586666107,
+            0.024416616186499596,
+            0.00759278517216444,
+            -0.0010752289090305567,
+            -0.012430835515260696,
+            -0.05288476496934891,
+            0.050889693200588226,
+            -0.013812039978802204,
+            -0.01692742109298706,
+            -0.005755016580224037,
+            -0.02243689075112343,
+            -0.011356566101312637,
+            0.023541854694485664,
+            -0.032872654497623444,
+            -0.05638381466269493,
+            0.01705019548535347,
+            0.016160085797309875,
+            0.051380787044763565,
+            0.02021161839365959,
+            -0.018738333135843277,
+            -0.010044422000646591,
+            0.03278057277202606,
+            -0.005781873129308224,
+            0.02360324189066887,
+            -0.022022530436515808,
+            -0.028821121901273727,
+            0.013942486606538296,
+            -0.033455830067396164,
+            0.06227695196866989,
+            -0.01651306077837944,
+            0.023787401616573334,
+            -0.013474412262439728,
+            0.04542626440525055,
+            -0.03324097394943237,
+            0.03235086426138878,
+            -0.029388951137661934,
+            0.021500742062926292,
+            0.005755016580224037,
+            -0.00864787120372057,
+            0.0028065296355634928,
+            0.00858648493885994,
+            -0.008279550820589066,
+            -0.020902220159769058,
+            0.005098944529891014,
+            0.030800849199295044,
+            0.0026933476328849792,
+            0.024278495460748672,
+            0.04907878115773201,
+            0.02077944576740265,
+            -0.052915457636117935,
+            0.029726577922701836,
+            0.029695885255932808,
+            0.020426472648978233,
+            0.0444747656583786,
+            0.007554418407380581,
+            0.010604577139019966,
+            0.006587575655430555,
+            0.014993736520409584,
+            0.0070403036661446095,
+            -0.01655910164117813,
+            0.03879648447036743,
+            -0.0241403765976429,
+            -0.0031268922612071037,
+            0.024784937500953674,
+            0.0015365893486887217,
+            0.036617252975702286,
+            0.005482612177729607,
+            0.013512779027223587,
+            0.02136262133717537,
+            0.00933079980313778,
+            0.027808239683508873,
+            0.02021161839365959,
+            0.030171632766723633,
+            0.020150231197476387,
+            -0.024401269853115082,
+            0.03198254480957985,
+            -0.056813523173332214,
+            -0.015945231541991234,
+            -0.009323126636445522,
+            0.028283987194299698,
+            -0.006169377826154232,
+            -0.01812446489930153,
+            -0.017894264310598373,
+            0.008049349300563335,
+            0.002048785798251629,
+            -0.009676100686192513,
+            0.008931785821914673,
+            0.033916231244802475,
+            0.026288915425539017,
+            0.024002255871891975,
+            -0.037937067449092865,
+            -0.0052025350742042065,
+            -0.010903838090598583,
+            -0.013374658301472664,
+            -0.034990500658750534,
+            -0.00218306970782578,
+            0.009461247362196445,
+            0.0012622670037671924,
+            0.009630060754716396,
+            -0.002929303329437971,
+            0.0027355512138456106,
+            0.03015628643333912,
+            -0.0046423799358308315,
+            0.045610424131155014,
+            -0.014425908215343952,
+            -0.0034817850682884455,
+            -0.020503206178545952,
+            -0.006576065439730883,
+            0.003627578727900982,
+            0.01622147299349308,
+            0.011824640445411205,
+            0.0031134639866650105,
+            0.012661036103963852,
+            0.004577156621962786,
+            -0.004465892910957336,
+            0.04435199499130249,
+            -0.016697220504283905,
+            0.01414966769516468,
+            -0.011064978316426277,
+            0.008195143193006516,
+            0.003395459847524762,
+            -0.005912320222705603,
+            0.041466813534498215,
+            -0.047851044684648514,
+            -0.0132749043405056,
+            0.020718058571219444,
+            0.0033820313401520252,
+            -0.03176769241690636,
+            0.012001127935945988,
+            -0.0009505368652753532,
+            -0.01609869860112667,
+            -0.01435684785246849,
+            -0.02364928089082241,
+            -0.005543999373912811,
+            0.00993699487298727,
+            0.00022012938279658556,
+            0.05666005611419678,
+            0.018078425899147987,
+            -0.01528532337397337,
+            -0.01978190988302231,
+            -0.02140866219997406,
+            0.03962520509958267,
+            -0.004730623681098223,
+            -0.035819221287965775,
+            0.013934813439846039,
+            -0.012039494700729847,
+            -0.01883041486144066,
+            0.030938968062400818,
+            0.01470982190221548,
+            -0.003180605825036764,
+            -0.019029920920729637,
+            0.008862725459039211,
+            -0.05260852351784706,
+            -0.005716649815440178,
+            0.0007275299867615104,
+            0.06727997958660126,
+            -0.024984445422887802,
+            -0.010934531688690186,
+            0.0050490680150687695,
+            0.0011596358381211758,
+            0.02364928089082241,
+            -0.00858648493885994,
+            -0.003474111668765545,
+            -0.021961143240332603,
+            0.03003351204097271,
+            -0.008010982535779476,
+            0.005578529555350542,
+            -0.026979517191648483,
+            -0.017955651506781578,
+            -0.04849560558795929,
+            -0.02978796511888504,
+            -0.014510314911603928,
+            0.012607323005795479,
+            -0.05273129791021347,
+            0.017648717388510704,
+            0.026856742799282074,
+            -0.021086379885673523,
+            0.009668427519500256,
+            0.012208308093249798,
+            0.027286451309919357,
+            -0.034407325088977814,
+            -0.05260852351784706,
+            -0.02941964380443096,
+            0.014157340861856937,
+            -0.016942769289016724,
+            -0.05512538552284241,
+            0.0036122319288551807,
+            0.012868217192590237,
+            0.015546217560768127,
+            -0.004849560558795929,
+            -0.0013322862796485424,
+            0.02133192867040634,
+            -0.006986590102314949,
+            0.0062768044881522655,
+            -0.01721900887787342,
+            -0.00873227883130312,
+            0.011793946847319603,
+            0.010282295756042004,
+            -0.000877640035469085,
+            -0.03738458827137947,
+            -0.013466738164424896,
+            0.0236953217536211,
+            0.018600212410092354,
+            -0.07317311316728592,
+            0.011487012729048729,
+            -0.027501305565238,
+            -0.007170750759541988,
+            -0.0042126718908548355,
+            0.03302612155675888,
+            -0.019060615450143814,
+            0.016083352267742157,
+            -0.0020583774894475937,
+            -0.02808448113501072,
+            -0.027639426290988922,
+            -0.006537698674947023,
+            -0.03410039097070694,
+            -0.03207462653517723,
+            -0.028468148782849312,
+            -0.004178142175078392,
+            -0.03143006190657616,
+            -0.029220137745141983,
+            -0.024508696049451828,
+            0.0023173533845692873,
+            -0.00552865257486701,
+            0.017357129603624344,
+            0.01899922825396061,
+            -0.03373207151889801,
+            -0.045364875346422195,
+            0.013190497644245625,
+            -0.006587575655430555,
+            0.01259197574108839,
+            0.0009654040331952274,
+            -0.012062514200806618,
+            0.016282860189676285,
+            -0.003094280604273081,
+            -0.03015628643333912,
+            -0.046193599700927734,
+            -0.013297924771904945,
+            0.014610068872570992,
+            0.008095390163362026,
+            -0.0002774397435132414,
+            0.010006055235862732,
+            -0.04686885327100754,
+            -0.024171069264411926,
+            0.01066596433520317,
+            -0.04008561000227928,
+            -0.024002255871891975,
+            -0.0045694829896092415,
+            -0.025444846600294113,
+            -0.006391904782503843,
+            -0.006844632793217897,
+            -0.0222373828291893,
+            0.03729250654578209,
+            0.018262585625052452,
+            -0.00933079980313778,
+            0.02941964380443096,
+            0.012162268161773682,
+            0.020672019571065903,
+            -0.01617543213069439,
+            -0.04640845209360123,
+            0.030048860237002373,
+            -0.019444283097982407,
+            -0.03098500892519951,
+            8.272836566902697e-05,
+            -0.022252731025218964,
+            0.007792292162775993,
+            0.00012229410640429705,
+            0.015945231541991234,
+            -0.006422598380595446,
+            -0.04573319852352142,
+            0.02932756394147873,
+            0.004872580524533987,
+            -0.04956987500190735,
+            -0.0022137630730867386,
+            -0.04791243001818657,
+            -0.0065607186406850815,
+            0.030340446159243584,
+            -0.004914783872663975,
+            -0.027977053076028824,
+            -0.03459148481488228,
+            -0.0015605686930939555,
+            -0.011433299630880356,
+            0.014625415205955505,
+            -0.013643225654959679,
+            0.013727632351219654,
+            -0.008041676133871078,
+            -0.0188764538615942,
+            -0.010719677433371544,
+            0.015776418149471283,
+            0.03759944066405296,
+            -0.024155722931027412,
+            -0.022682437673211098,
+            -0.03572714328765869,
+            -0.03953312709927559,
+            0.013221191242337227,
+            0.024201761931180954,
+            0.01867694780230522,
+            -0.010972898453474045,
+            0.018078425899147987,
+            0.0068369596265256405,
+            -0.014878636226058006,
+            -0.025153258815407753,
+            0.0029810986015945673,
+            0.05441943556070328,
+            0.00807236973196268,
+            -0.00013368423969950527,
+            0.002990690292790532,
+            0.009798875078558922,
+            0.017188316211104393,
+            0.005555509123951197,
+            -0.014280114322900772,
+            0.04373812675476074,
+            -0.014433581382036209,
+            0.039686594158411026,
+            -0.01065829023718834,
+            -0.02070271223783493,
+            -0.03634101152420044,
+            0.023679975420236588,
+            0.005313798785209656,
+            -0.03443801775574684,
+            -0.012361775152385235,
+            0.029358256608247757,
+            0.005720486398786306,
+            -0.010773390531539917,
+            0.012599649839103222,
+            -0.035358820110559464,
+            -0.029557764530181885,
+            -0.022713132202625275,
+            -0.037108346819877625,
+            -0.00947659369558096,
+            -0.03453009948134422,
+            -0.06365815550088882,
+            -0.004247202072292566,
+            -0.061294760555028915,
+            -0.014878636226058006,
+            0.02820725366473198,
+            -0.0006987549131736159,
+            0.0162982065230608,
+            -0.00065223517594859,
+            -0.007700212299823761,
+            -0.03833608329296112,
+            0.005206371657550335,
+            0.04174305126070976,
+            0.01746455766260624,
+            -0.0380905345082283,
+            -0.017280396074056625,
+            -0.009031538851559162,
+            -0.04263316094875336,
+            -0.029342910274863243,
+            0.009031538851559162,
+            -0.013643225654959679,
+            6.396460958058015e-05,
+            -0.03941035270690918,
+            0.003128810552880168,
+            0.006840796209871769,
+            0.018109118565917015,
+            0.04444407299160957,
+            0.022129956632852554,
+            -0.06040465459227562,
+            0.007987963035702705,
+            -0.003629497019574046,
+            0.027240412309765816,
+            -0.014072933234274387,
+            0.018017038702964783,
+            -0.0006474393303506076,
+            -0.007009610068053007,
+            -0.02662654221057892,
+            0.009054559282958508,
+            -0.0012507569044828415,
+            -0.011878354474902153,
+            -0.0038596978411078453,
+            0.004316262435168028,
+            0.00914663914591074,
+            0.03182907775044441,
+            0.010282295756042004,
+            0.009537980891764164,
+            -0.04082224890589714,
+            -0.012860543094575405,
+            -0.01130285207182169,
+            0.04042323678731918,
+            0.009246393106877804,
+            -0.034990500658750534,
+            0.007316544186323881,
+            0.03253502771258354,
+            0.028406761586666107,
+            0.028115173801779747,
+            -0.011072651483118534,
+            0.012730096466839314,
+            0.020349739119410515,
+            -0.07004238665103912,
+            -0.005758853163570166,
+            -0.0069904266856610775,
+            -0.0037503524217754602,
+            0.0037407607305794954,
+            -0.02289729192852974,
+            -0.00019531087309587747,
+            0.042541082948446274,
+            -0.016896728426218033,
+            -0.00151644682046026,
+            -0.011003591120243073,
+            0.029235484078526497,
+            0.002560982247814536,
+            0.004086061846464872,
+            -0.012607323005795479,
+            0.058470968157052994,
+            -0.017863571643829346,
+            0.013343964703381062,
+            0.006077297497540712,
+            -0.01306772418320179,
+            0.024861671030521393,
+            -0.021715596318244934,
+            -0.007051813416182995,
+            0.005056741181761026,
+            0.0094382269307971,
+            -0.017157621681690216,
+            -0.003107708878815174,
+            -0.0026991027407348156,
+            -0.01741851679980755,
+            0.010811757296323776,
+            -0.0489560067653656,
+            -0.03471425920724869,
+            -0.0022463747300207615,
+            0.01005976926535368,
+            -0.020027456805109978,
+            0.010305316187441349,
+            0.014326154254376888,
+            0.007934249006211758,
+            0.055739253759384155,
+            -0.03143006190657616,
+            -0.0016229146858677268,
+            -0.003309134393930435,
+            -0.023925522342324257,
+            -0.028636962175369263,
+            -0.023004719987511635,
+            -0.0056130592711269855,
+            -0.015822459012269974,
+            -0.04386089742183685,
+            -0.008693911135196686,
+            -0.021439354866743088,
+            0.021270541474223137,
+            -0.04094502329826355,
+            0.016912074759602547,
+            0.008724604733288288,
+            0.0006124296924099326,
+            -0.009008519351482391,
+            -0.01995072327554226,
+            -0.0042625488713383675,
+            0.029757272452116013,
+            0.032166704535484314,
+            0.018861107528209686,
+            0.034867726266384125,
+            -0.005636079702526331,
+            -0.0231888797134161,
+            -0.013581839390099049,
+            0.03572714328765869,
+            -0.00854811817407608,
+            0.020058151334524155,
+            0.004980007652193308,
+            -0.0006061950698494911,
+            -0.0224061980843544,
+            -0.0077270688489079475,
+            0.03646378591656685,
+            -0.02555227279663086,
+            0.014425908215343952,
+            0.05122731998562813,
+            0.032749880105257034,
+            0.0006071542156860232,
+            -0.008640198037028313,
+            -0.01472516916692257,
+            0.005919993855059147,
+            -0.021792329847812653,
+            0.03198254480957985,
+            0.01987398974597454,
+            -0.001803238526917994,
+            -0.010535516776144505,
+            0.01325188484042883,
+            -0.030217673629522324,
+            0.012469202280044556,
+            0.006668145768344402,
+            0.003606477053835988,
+            -0.034867726266384125,
+            -0.02136262133717537,
+            -0.002419025171548128,
+            -0.0007759680738672614,
+            -0.01929081603884697,
+            -0.015553891658782959,
+            -0.02286659926176071,
+            0.002154294401407242,
+            -0.006821612827479839,
+            0.004281732253730297,
+            0.016697220504283905,
+            0.02837606891989708,
+            -0.0006138684111647308,
+            0.03701626509428024,
+            0.00303289364092052,
+            0.023879481479525566,
+            0.004945477470755577,
+            -0.003512478433549404,
+            -0.0070403036661446095,
+            -0.011755580082535744,
+            -0.014487294480204582,
+            0.057273924350738525,
+            0.017019502818584442,
+            0.05957593023777008,
+            -0.011064978316426277,
+            -0.010819431394338608,
+            0.030938968062400818,
+            0.004316262435168028,
+            -0.020733406767249107,
+            0.008670891635119915,
+            -0.016160085797309875,
+            -0.02871369570493698,
+            -0.0027470611967146397,
+            -0.03799845650792122,
+            -0.0188764538615942,
+            -0.022621052339673042,
+            -0.018308626487851143,
+            0.004684583283960819,
+            0.0196591354906559,
+            0.016436327248811722,
+            -0.00016737506666686386,
+            -0.011763254180550575,
+            0.031583528965711594,
+            0.019306162372231483,
+            0.028928549960255623,
+            -0.011809294112026691,
+            0.007331890985369682,
+            -0.01576107181608677,
+            0.0052831051871180534,
+            0.019137348979711533,
+            -0.01253058947622776,
+            0.009453574195504189,
+            -0.0008905887953005731,
+            0.011180078610777855,
+            0.03821330890059471,
+            -0.010036748833954334,
+            -0.03953312709927559,
+            -0.006280641537159681,
+            -0.014778882265090942,
+            0.007278177421540022,
+            0.0067065125331282616,
+            -0.020932912826538086,
+            0.012254348024725914,
+            0.007730905432254076,
+            -0.01479422952979803,
+            0.02128588780760765,
+            -0.007205280475318432,
+            -0.027624079957604408,
+            0.03468356654047966,
+            -0.014134320430457592,
+            0.01787891797721386,
+            -0.017986344173550606,
+            -0.0028755899984389544,
+            -0.025828514248132706,
+            0.022559665143489838,
+            0.01751059666275978,
+            -0.027317145839333534,
+            0.022989371791481972,
+            0.04564111679792404,
+            0.020150231197476387,
+            0.0007615805370733142,
+            0.012415489181876183,
+            0.0057665263302624226,
+            -0.036494478583335876,
+            -0.02057993970811367,
+            -0.006944386754184961,
+            -0.0027950196526944637,
+            -0.016574447974562645,
+            0.030217673629522324,
+            -0.03824400156736374,
+            -0.0031518307514488697,
+            -0.0052025350742042065,
+            -0.024078989401459694,
+            0.0055823661386966705,
+            0.009069905616343021,
+            0.003951778169721365,
+            0.017142275348305702,
+            -0.014172687195241451,
+            0.03020232729613781,
+            0.03566575422883034,
+            0.006322844885289669,
+            -0.017848225310444832,
+            0.011256812140345573,
+            -0.024508696049451828,
+            0.0048457239754498005,
+            0.043001484125852585,
+            0.014072933234274387,
+            0.008494404144585133,
+            -0.020426472648978233,
+            -0.02584386058151722,
+            0.007235974073410034,
+            0.008709258399903774,
+            0.04198860004544258,
+            0.02630426175892353,
+            0.03606477007269859,
+            0.009369166567921638,
+            -0.01199345476925373,
+            -0.013819713145494461,
+            -0.0049493140541017056,
+            -0.023265613242983818,
+            0.021761635318398476,
+            -0.02228342369198799,
+            0.01717296987771988,
+            0.012039494700729847,
+            -0.028759736567735672,
+            0.00017528820899315178,
+            0.01576107181608677,
+            0.03575783595442772,
+            0.04361535236239433,
+            0.023925522342324257,
+            0.0032765227369964123,
+            0.039686594158411026,
+            -0.04493517056107521,
+            -0.03716973215341568,
+            0.00753907160833478,
+            0.017571983858942986,
+            0.009154313243925571,
+            0.0058164033107459545,
+            0.03290334716439247,
+            -0.00947659369558096,
+            -0.003376276232302189,
+            0.012024147436022758,
+            0.02074875310063362,
+            0.002031520940363407,
+            -0.00429324246942997,
+            -0.008095390163362026,
+            0.02289729192852974,
+            0.021086379885673523,
+            0.03937966004014015,
+            -0.020058151334524155,
+            0.01780218444764614,
+            0.0009936995338648558,
+            -0.02552158012986183,
+            0.02559831365942955,
+            0.00021701208606828004,
+            -0.04530349001288414,
+            0.03339444100856781,
+            0.017249703407287598,
+            0.015108836814761162,
+            0.011786273680627346,
+            -0.006131011061370373,
+            0.012553608976304531,
+            0.030524607747793198,
+            -0.0031019540037959814,
+            0.0011423707474023104,
+            0.03115382231771946,
+            -0.003311052918434143,
+            -0.004807357210665941,
+            0.020886873826384544,
+            0.004887927323579788,
+            -0.013167478144168854,
+            0.01867694780230522,
+            -0.012169941328465939,
+            0.044075753539800644,
+            -0.024692857638001442,
+            0.006760226096957922,
+            0.010205562226474285,
+            -0.01883041486144066,
+            -0.007293524220585823,
+            -0.0013188580051064491,
+            -0.022268077358603477,
+            0.004500423092395067,
+            0.018155159428715706,
+            -0.012331082485616207,
+            0.0295731108635664,
+            -0.028606269508600235,
+            0.0012430836213752627,
+            0.015945231541991234,
+            -0.026933476328849792,
+            0.02767011895775795,
+            0.013006336987018585,
+            0.02713298425078392,
+            -0.009576347656548023,
+            -0.014456601813435555,
+            0.04797381907701492,
+            0.0563531219959259,
+            0.0060504404827952385,
+            -0.008578810840845108,
+            0.013742979615926743,
+            0.0003174850717186928,
+            -0.02360324189066887,
+            0.024048294872045517,
+            -0.03572714328765869,
+            0.0162982065230608,
+            0.03107708878815174,
+            -0.014617742039263248,
+            -0.014095953665673733,
+            -0.034622177481651306,
+            -0.0052831051871180534,
+            0.00429324246942997,
+            0.004381485749036074,
+            0.0036966388579458,
+            -0.025659700855612755,
+            -0.015599931590259075,
+            0.022252731025218964,
+            0.005379022099077702,
+            0.0010157604701817036,
+            -0.025076525285840034,
+            0.015530871227383614,
+            -0.01617543213069439,
+            -0.04162028059363365,
+            -0.006330518051981926,
+            -0.012492222711443901,
+            0.025214646011590958,
+            -6.204627425177023e-05,
+            -0.03477564454078674,
+            -0.006718022748827934,
+            -0.005191024858504534,
+            -0.009054559282958508,
+            0.044536154717206955,
+            0.023741360753774643,
+            0.03812123090028763,
+            -0.02170024812221527,
+            -0.03141471743583679,
+            0.02460077777504921,
+            -0.03489841893315315,
+            0.030770154669880867,
+            -0.0032439110800623894,
+            0.007186097092926502,
+            -0.0052025350742042065,
+            0.039348967373371124,
+            -0.012146921828389168,
+            0.008571137674152851,
+            -0.03588061034679413,
+            -0.004980007652193308,
+            -0.01945962943136692,
+            -0.008517424575984478,
+            0.0031153822783380747,
+            0.011870680376887321,
+            -0.009568674489855766,
+            0.005659099668264389,
+            0.04720648378133774,
+            0.031583528965711594,
+            0.0018675028113648295,
+            0.026565156877040863,
+            -0.02862161584198475,
+            -0.017986344173550606,
+            0.01464076153934002,
+            -0.0064072515815496445,
+            -0.00785751547664404,
+            -0.013973180204629898,
+            -0.031798385083675385,
+            -0.012799156829714775,
+            0.011187751777470112,
+            0.013581839390099049,
+            -0.011602113023400307,
+            0.003735005622729659,
+            0.012906583957374096,
+            -0.014172687195241451,
+            -0.008057023398578167,
+            -0.0393182709813118,
+            0.0011222282191738486,
+            0.024370577186346054,
+            -0.027977053076028824,
+            0.013121437281370163,
+            0.002286659786477685,
+            0.007389441132545471,
+            -0.023618588224053383,
+            -0.02087152563035488,
+            0.008218163624405861,
+            -0.012154594995081425,
+            0.005248575005680323,
+            -0.002712531015276909,
+            0.034744951874017715,
+            -0.00800330936908722,
+            0.03351721540093422,
+            -0.012975644320249557,
+            0.01953636296093464,
+            -0.025506233796477318,
+            -0.02389482781291008,
+            -0.04272524267435074,
+            -0.0012977562146261334,
+            0.0029580784030258656,
+            -0.02820725366473198,
+            -0.006495495326817036,
+            -0.001958623994141817,
+            0.017249703407287598,
+            -0.012246674858033657,
+            -0.003274604445323348,
+            0.006622105836868286,
+            -0.01354347262531519,
+            -0.0020104190334677696,
+            -0.03987075388431549,
+            -0.02825329452753067,
+            -0.03406969830393791,
+            0.016835341230034828,
+            -0.022667091339826584,
+            0.017234355211257935,
+            -0.01995072327554226,
+            0.0005678283050656319,
+            -0.047359950840473175,
+            -0.01573037914931774,
+            0.02991073951125145,
+            0.008141430094838142,
+            -0.04204998537898064,
+            0.021470047533512115,
+            0.01755663752555847,
+            0.017479903995990753,
+            -0.02311214618384838,
+            0.018492786213755608,
+            0.002338455058634281,
+            0.004933967255055904,
+            0.024416616186499596,
+            -0.02535276673734188,
+            0.02177698165178299,
+            -0.024907711893320084,
+            -0.021838368847966194,
+            -0.004020838066935539,
+            0.013827386312186718,
+            0.01726504974067211,
+            0.0020104190334677696,
+            -0.01709623634815216,
+            0.016743261367082596,
+            0.018323972821235657,
+            0.003915329463779926,
+            0.022636398673057556,
+            -0.02664189040660858,
+            0.016620486974716187,
+            0.011809294112026691,
+            -0.02323492057621479,
+            0.0036256604362279177,
+            -0.01483259629458189,
+            -0.007174587342888117,
+            0.017525942996144295,
+            0.03146075829863548,
+            0.014617742039263248,
+            -0.03683210536837578,
+            0.012860543094575405,
+            0.011747906915843487,
+            -0.0014147749170660973,
+            0.038274697959423065,
+            0.0027240412309765816,
+            0.0239715613424778,
+            0.03397761657834053,
+            0.0066105956211686134,
+            0.017664063721895218,
+            0.03548159450292587,
+            -0.022912638261914253,
+            -0.009277086704969406,
+            -0.023802747949957848,
+            0.014111299999058247,
+            0.014310807920992374,
+            -0.012753116898238659,
+            -0.012001127935945988,
+            -0.015239283442497253,
+            -0.014118974097073078,
+            0.01879972033202648,
+            -0.0008503036806359887,
+            -0.015745725482702255,
+            -0.029711231589317322,
+            0.024692857638001442,
+            0.0006781327538192272,
+            -0.044658929109573364,
+            0.012323408387601376,
+            -0.014218727126717567,
+            -0.0037868008948862553,
+            -0.0027835096698254347,
+            0.007769272197037935,
+            -0.005605386104434729,
+            0.0033359911758452654,
+            0.004266385454684496,
+            0.05135009437799454,
+            -0.0014166932087391615,
+            0.017311088740825653,
+            -0.008839705027639866,
+            0.01970517635345459,
+            -0.010988244786858559,
+            0.022912638261914253,
+            -0.014556354843080044,
+            -0.051503561437129974,
+            0.02618148922920227,
+            -0.011041957885026932,
+            0.040883637964725494,
+            0.05098177120089531,
+            -0.01841605268418789,
+            -0.010750371031463146,
+            0.03575783595442772,
+            0.004680746700614691,
+            -0.025337418541312218,
+            0.007888209074735641,
+            -0.0012296552304178476,
+            -0.007504541426897049,
+            -0.019613096490502357,
+            -0.012077861465513706,
+            0.019720522686839104,
+            -0.0038712078239768744,
+            0.017372475937008858,
+            0.040638089179992676,
+            0.03161422535777092,
+            -0.021301234140992165,
+            0.009522633627057076,
+            0.01331327110528946,
+            -0.009215699508786201,
+            -0.007796129211783409,
+            0.0254295002669096,
+            -0.017034849151968956,
+            -0.0008320794440805912,
+            -0.012177614495158195,
+            -0.00400165468454361,
+            -0.035941995680332184,
+            0.001303511206060648,
+            -0.016605140641331673,
+            -0.0018761353567242622,
+            -0.007692538667470217,
+            -0.030923621729016304,
+            0.03228947892785072,
+            0.028698349371552467,
+            -0.0070633236318826675,
+            -0.009077579714357853,
+            -0.009468920528888702,
+            0.005037557799369097,
+            0.0036333338357508183,
+            -0.0019969907589256763,
+            0.0034357449039816856,
+            -0.01792495884001255,
+            0.007443154696375132,
+            -0.006775572896003723,
+            -0.01325188484042883,
+            -0.051135238260030746,
+            -0.01867694780230522,
+            0.025045832619071007,
+            0.004830377176403999,
+            -0.01692742109298706,
+            -0.003261175937950611,
+            0.008103063330054283,
+            -0.005862443707883358,
+            -0.02496909908950329,
+            0.03135332837700844,
+            0.01668187417089939,
+            0.01875368133187294,
+            -0.0017456883797422051,
+            -0.0022578847128897905,
+            0.01746455766260624,
+            0.006084970664232969,
+            -0.01547715812921524,
+            -0.009499614126980305,
+            0.021761635318398476,
+            0.004120592027902603,
+            0.020932912826538086,
+            -0.010735023766756058,
+            0.021086379885673523,
+            0.007957269437611103,
+            -0.012123901396989822,
+            -0.0027202044147998095,
+            -0.0012718586949631572,
+            0.004105245228856802,
+            0.016912074759602547,
+            -0.013980853371322155,
+            0.024953750893473625,
+            0.020088844001293182,
+            0.007109363563358784,
+            0.0044889128766953945,
+            -0.01668187417089939,
+            -0.02452404424548149,
+            0.016574447974562645,
+            0.002522615483030677,
+            0.027977053076028824,
+            0.012139247730374336,
+            -0.004013164900243282,
+            -0.02759338542819023,
+            -0.013052377849817276,
+            -0.024217110127210617,
+            0.01005976926535368,
+            0.023664627224206924,
+            0.030632033944129944,
+            -0.015461810864508152,
+            -0.013489758595824242,
+            -0.025276033207774162,
+            0.00469993008300662,
+            0.008908765390515327,
+            -0.027347838506102562,
+            -0.01992003060877323,
+            0.018032385036349297,
+            -0.025291379541158676,
+            0.026488423347473145,
+            -0.0019269713666290045,
+            -0.012852869927883148,
+            0.006675818935036659,
+            0.07071764022111893,
+            -0.013090744614601135,
+            -0.0012200635392218828,
+            -0.028652308508753777,
+            -0.017618024721741676,
+            -0.005931503605097532,
+            -0.008617177605628967,
+            0.0004412179405335337,
+            0.018492786213755608,
+            -0.008624851703643799,
+            0.020902220159769058,
+            0.01668187417089939,
+            0.0013006337685510516,
+            0.0032074626069515944,
+            -0.03526674211025238,
+            0.03032509982585907,
+            -0.004224182106554508,
+            0.0014838350471109152,
+            0.00244204537011683,
+            0.006307498086243868,
+            0.018400706350803375,
+            0.016896728426218033,
+            0.005931503605097532,
+            0.009200353175401688,
+            -0.005912320222705603,
+            -0.003738842438906431,
+            0.016114046797156334,
+            0.019858643412590027,
+            -0.01111869141459465,
+            -0.012944950722157955,
+            0.005728159565478563,
+            0.014901655726134777,
+            -0.004853397142142057,
+            -0.00414744857698679,
+            0.0075736017897725105,
+            0.020472511649131775,
+            -0.01617543213069439,
+            -0.002286659786477685,
+            0.004742133431136608,
+            -0.017372475937008858,
+            0.0007548663415946066,
+            0.01875368133187294,
+            0.007565928157418966,
+            0.015853151679039,
+            -0.015484831295907497,
+            -0.009161986410617828,
+            0.0021255193278193474,
+            -0.022421544417738914,
+            0.003121137386187911,
+            -0.012192961759865284,
+            -0.01058155670762062,
+            -0.00239600520581007,
+            -0.007416298147290945,
+            -0.006487822160124779,
+            -0.006568392273038626,
+            0.004630869720131159,
+            0.009430553764104843,
+            -0.004151285160332918,
+            0.01738782227039337,
+            0.005252411589026451,
+            0.013988526538014412,
+            -0.0053560021333396435,
+            0.01489398255944252,
+            0.02136262133717537,
+            -0.010213236324489117,
+            0.00075822341023013,
+            0.01775614358484745,
+            -0.0006344905705191195,
+            0.006741042714565992,
+            0.004013164900243282,
+            1.707021874608472e-05,
+            0.010842450894415379,
+            -0.004343118984252214,
+            -0.01992003060877323,
+            -0.0055094691924750805,
+            -0.011440972797572613,
+            0.002133192727342248,
+            0.002252129837870598,
+            0.03716973215341568,
+            -0.02513791248202324,
+            0.03235086426138878,
+            0.012914257124066353,
+            -0.014011546969413757,
+            -0.00569362984970212,
+            -0.047482721507549286,
+            0.01338233146816492,
+            -0.014694475568830967,
+            3.977555752499029e-05,
+            0.017970997840166092,
+            -0.00914663914591074,
+            -0.0014799984637647867,
+            0.00502988463267684,
+            -0.005482612177729607,
+            -0.0009457410196773708,
+            -0.01924477517604828,
+            -0.0067563895136117935,
+            0.010382049717009068,
+            0.013842732645571232,
+            -0.03471425920724869,
+            -0.02243689075112343,
+            -0.030263712629675865,
+            0.00392875773832202,
+            0.01883041486144066,
+            -0.010942204855382442,
+            -0.025828514248132706,
+            -0.009537980891764164,
+            0.013405351899564266,
+            -0.01697346195578575,
+            0.006388068199157715,
+            -0.003997818101197481,
+            0.014326154254376888,
+            -0.008832031860947609,
+            0.001874217065051198,
+            -0.01899922825396061,
+            0.01487096305936575,
+            0.0028314681258052588,
+            -0.00420116214081645,
+            -0.030340446159243584,
+            0.007120873779058456,
+            -0.00020574184600263834,
+            -0.01450264174491167,
+            -0.012269695289433002,
+            0.017280396074056625,
+            -0.011134038679301739,
+            -0.004745970480144024,
+            0.016743261367082596,
+            0.006695002317428589,
+            0.0006618269253522158,
+            0.018277931958436966,
+            0.005352165549993515,
+            -0.0005002068355679512,
+            0.005336818750947714,
+            0.013106090947985649,
+            0.0012517160503193736,
+            0.024048294872045517,
+            -0.01617543213069439,
+            0.00997536163777113,
+            -0.014333827421069145,
+            0.033701375126838684,
+            -0.01597592607140541,
+            -0.0016027720412239432,
+            0.012177614495158195,
+            0.040883637964725494,
+            -0.013497431762516499,
+            0.01712692901492119,
+            -0.0291280560195446,
+            -0.015354383736848831,
+            0.03465287387371063,
+            0.01688138209283352,
+            0.010988244786858559,
+            -0.010359029285609722,
+            -0.01332094520330429,
+            0.008801338262856007,
+            0.01470982190221548,
+            0.008824358694255352,
+            -0.0241403765976429,
+            0.02908201701939106,
+            -0.014571702107787132,
+            0.04162028059363365,
+            -0.0001700127759249881,
+            -0.003117300570011139,
+            0.023204226046800613,
+            -0.0009577306336723268,
+            -0.02265174500644207,
+            0.014947696588933468,
+            0.00268183765001595,
+            0.01203182153403759,
+            -0.01066596433520317,
+            0.010527843609452248,
+            0.023787401616573334,
+            0.006537698674947023,
+            -0.005018374416977167,
+            -7.163796544773504e-05,
+            0.016912074759602547,
+            0.0005409715231508017,
+            -0.005033721216022968,
+            0.027731506153941154,
+            -0.007481521461158991,
+            -0.010842450894415379,
+            -0.01875368133187294,
+            -0.0012171859852969646,
+            -0.018247239291667938,
+            -0.00043474353151395917,
+            -0.007700212299823761,
+            0.0008320794440805912,
+            -0.010589230805635452,
+            -0.015154876746237278,
+            -0.003944104537367821,
+            -0.025076525285840034,
+            0.03480634093284607,
+            0.009676100686192513,
+            0.013597185723483562,
+            0.009200353175401688,
+            0.00365059869363904,
+            -0.000640245561953634,
+            0.0214547012001276,
+            -0.0030175470747053623,
+            0.0005649508093483746,
+            -0.005685956217348576,
+            -0.0119857806712389,
+            -0.0122390016913414,
+            0.013980853371322155,
+            -0.008317917585372925,
+            -0.008640198037028313,
+            -0.00844069104641676,
+            -0.01821654476225376,
+            -0.03477564454078674,
+            -0.0013610613532364368,
+            0.008893419057130814,
+            -0.028360720723867416,
+            0.014909329824149609,
+            -0.006441781762987375,
+            -0.01600661873817444,
+            0.0008009064476937056,
+            0.006161704193800688,
+            0.018554173409938812,
+            -0.007972615770995617,
+            -0.03425385802984238,
+            0.031384024769067764,
+            0.01646701991558075,
+            0.003899982897564769,
+            0.0033820313401520252,
+            0.008010982535779476,
+            0.008064696565270424,
+            -0.01395783293992281,
+            -0.0099139753729105,
+            0.028115173801779747,
+            -0.026212181895971298,
+            0.0239715613424778,
+            0.012561283074319363,
+            0.0003963767667300999,
+            0.027240412309765816,
+            -0.019474975764751434,
+            0.02630426175892353,
+            -0.0076042949222028255,
+            -5.1765175157925114e-05,
+            0.015200916677713394,
+            0.02099430002272129,
+            0.020319044589996338,
+            0.02272847853600979,
+            -0.02265174500644207,
+            0.038643017411231995,
+            -0.0023787401150912046,
+            0.03201323747634888,
+            0.006307498086243868,
+            -0.00015094928676262498,
+            0.008486730977892876,
+            0.008172123692929745,
+            -0.017679410055279732,
+            -0.0010176787618547678,
+            -0.03069342114031315,
+            0.010604577139019966,
+            -0.005229391623288393,
+            -0.01867694780230522,
+            -0.020794792100787163,
+            -0.033916231244802475,
+            0.010888490825891495,
+            0.001221981830894947,
+            -0.013037030585110188,
+            -0.004742133431136608,
+            0.0078076389618217945,
+            0.01306772418320179,
+            0.022713132202625275,
+            -0.0037100673653185368,
+            0.0032688493374735117,
+            0.019643789157271385,
+            -0.005298451986163855,
+            0.004331609234213829,
+            -0.01617543213069439,
+            0.02211461029946804,
+            0.005555509123951197,
+            0.030754808336496353,
+            0.02476959116756916,
+            0.008540444076061249,
+            0.004611686337739229,
+            -0.004204998724162579,
+            0.017817530781030655,
+            -0.01600661873817444,
+            -0.010481803677976131,
+            -0.001481916755437851,
+            -0.0038366776425391436,
+            0.06037395820021629,
+            -0.024692857638001442,
+            -0.02506117895245552,
+            -0.005559345707297325,
+            -0.011487012729048729,
+            0.006675818935036659,
+            -0.03247363865375519,
+            -0.028928549960255623,
+            -0.023511160165071487,
+            -0.0175873301923275,
+            -0.0019624605774879456,
+            -0.009998382069170475,
+            0.014049913734197617,
+            -0.00569362984970212,
+            0.011778600513935089,
+            -0.0239715613424778,
+            0.000355372263584286,
+            -0.016651181504130363,
+            -0.005685956217348576,
+            -0.014026893302798271,
+            -0.0070403036661446095,
+            -0.000628735579084605,
+            0.02949637733399868,
+            0.011824640445411205,
+            -0.019229428842663765,
+            -0.0032918695360422134,
+            0.012745442800223827,
+            -0.008279550820589066,
+            -0.007362584583461285,
+            0.012246674858033657,
+            0.01916804164648056,
+            -0.012208308093249798,
+            0.037937067449092865,
+            -0.0037983108777552843,
+            -0.00038582590059377253,
+            -0.005904647056013346,
+            -0.012584302574396133,
+            0.024877017363905907,
+            0.028314681723713875,
+            -0.03984006121754646,
+            -0.0013303679879754782,
+            0.0018694212194532156,
+            0.007600458338856697,
+            -0.005321471951901913,
+            0.0030866072047501802,
+            -0.027117637917399406,
+            -0.0012776136863976717,
+            -0.026872090995311737,
+            -0.01573037914931774,
+            -0.021470047533512115,
+            -0.005923830438405275,
+            0.0032458293717354536,
+            -0.023050758987665176,
+            0.030094899237155914,
+            0.02077944576740265,
+            -0.014487294480204582,
+            -0.0038596978411078453,
+            -0.003500968450680375,
+            0.008126083761453629,
+            -0.03189046308398247,
+            -0.004362302366644144,
+            -0.009745161049067974,
+            -9.597688767826185e-05,
+            -0.00962238758802414,
+            -0.04198860004544258,
+            0.0012239001225680113,
+            -0.003167177317664027,
+            -0.0017955651273950934,
+            -0.008494404144585133,
+            -0.016727915033698082,
+            0.02177698165178299,
+            0.020303698256611824,
+            -0.010880817659199238,
+            -0.008532770909368992,
+            0.045610424131155014,
+            0.008425343781709671,
+            0.0016411389224231243,
+            -0.010888490825891495,
+            0.002762407995760441,
+            0.0028314681258052588,
+            0.008486730977892876,
+            -0.013528125360608101,
+            0.011463993228971958,
+            -0.005298451986163855,
+            0.014118974097073078,
+            -0.0067065125331282616,
+            -0.016114046797156334,
+            0.023204226046800613,
+            -0.02829933539032936,
+            0.000396856339648366,
+            -0.010174869559705257,
+            -0.001934644766151905,
+            0.01395783293992281,
+            0.010259276255965233,
+            0.01726504974067211,
+            -0.03953312709927559,
+            -0.004815030377358198,
+            -0.008386977016925812,
+            -0.0004716715484391898,
+            -0.0059468504041433334,
+            -0.026994863525032997,
+            0.013443718664348125,
+            0.0023557201493531466,
+            -0.010880817659199238,
+            0.018323972821235657,
+            -0.004515769425779581,
+            -0.009960015304386616,
+            0.022375503554940224,
+            -0.0019020329928025603,
+            0.015362057834863663,
+            0.029235484078526497,
+            -0.016789302229881287,
+            0.0183700118213892,
+            0.032504335045814514,
+            -0.009369166567921638,
+            -0.0037139039486646652,
+            -0.0028142030350863934,
+            0.0009936995338648558,
+            0.016328901052474976,
+            0.017525942996144295,
+            0.0007447950774803758,
+            -0.03228947892785072,
+            0.00829489715397358,
+            -0.00218306970782578,
+            0.007918902672827244,
+            0.009514960460364819,
+            -0.014318481087684631,
+            -0.000949098146520555,
+            -0.025997327640652657,
+            0.006952059920877218,
+            -0.0209482591599226,
+            -0.009422880597412586,
+            0.024171069264411926,
+            -0.017786838114261627,
+            -0.019152695313096046,
+            0.003688965691253543,
+            0.02837606891989708,
+            -0.029680538922548294,
+            0.01130285207182169,
+            0.004139775410294533,
+            -0.005812566727399826,
+            0.006361211650073528,
+            0.000759182614274323,
+            -0.01895318739116192,
+            0.004500423092395067,
+            -0.009507287293672562,
+            -0.012169941328465939,
+            -0.020257657393813133,
+            -0.013835059478878975,
+            -0.0006699798395857215,
+            0.014617742039263248,
+            0.015822459012269974,
+            0.028851816430687904,
+            0.0016516897594556212,
+            -0.0069405497051775455,
+            -0.014656108804047108,
+            0.006879162974655628,
+            -0.0064801485277712345,
+            0.012990990653634071,
+            -0.008709258399903774,
+            0.02535276673734188,
+            -0.00811073649674654,
+            -0.0006037971470504999,
+            -0.010497150011360645,
+            -0.013198170810937881,
+            0.008762971498072147,
+            -0.003247747663408518,
+            0.0017994018271565437,
+            -0.011732560582458973,
+            0.004354629199951887,
+            0.01692742109298706,
+            -0.01863090693950653,
+            -0.02177698165178299,
+            -0.023449774831533432,
+            0.021976489573717117,
+            0.03766082972288132,
+            0.0009236801415681839,
+            -0.050214435905218124,
+            -0.0021063359454274178,
+            -0.007677191868424416,
+            -0.012077861465513706,
+            -0.02265174500644207,
+            0.011487012729048729,
+            0.018139811232686043,
+            -0.0008004268747754395,
+            -0.0019893173594027758,
+            0.0031978709157556295,
+            0.015108836814761162,
+            -0.0013207762967795134,
+            0.0053866952657699585,
+            0.00251494231633842,
+            -0.0075007048435509205,
+            0.004155121743679047,
+            0.029941432178020477,
+            -0.027731506153941154,
+            0.00941520743072033,
+            0.002618532394990325,
+            0.011755580082535744,
+            0.004669236950576305,
+            0.009177332744002342,
+            0.01470982190221548,
+            0.0032400742638856173,
+            -0.00709401722997427,
+            0.0196591354906559,
+            0.07206815481185913,
+            -0.0028314681258052588,
+            0.015385077334940434,
+            0.00972981471568346,
+            -0.012822176329791546,
+            -0.004040021449327469,
+            -0.010312989354133606,
+            -0.03848955035209656,
+            0.025398805737495422,
+            0.008034002967178822,
+            -0.012077861465513706,
+            0.006483985111117363,
+            -0.014126647263765335,
+            0.008878071792423725,
+            0.029020629823207855,
+            0.010366703383624554,
+            -0.020794792100787163,
+            0.004281732253730297,
+            0.03453009948134422,
+            0.0214547012001276,
+            0.004542626440525055,
+            -0.01780218444764614,
+            0.006219254340976477,
+            -0.0014195707626640797,
+            -0.0038712078239768744,
+            0.009614714421331882,
+            -0.004465892910957336,
+            0.022375503554940224,
+            -0.004078388214111328,
+            0.014111299999058247,
+            -0.013535798527300358,
+            -0.013911793008446693,
+            0.006115664262324572,
+            -0.001624832977540791,
+            0.0014790392015129328,
+            0.020150231197476387,
+            0.0028314681258052588,
+            -0.006760226096957922,
+            0.0024861672427505255,
+            -0.0005850933375768363,
+            0.013681592419743538,
+            0.004818866960704327,
+            0.007423971313983202,
+            0.006614432204514742,
+            -0.023050758987665176,
+            0.01879972033202648,
+            0.006434108596295118,
+            0.00976818148046732,
+            -0.013037030585110188,
+            -0.005682119634002447,
+            -0.005720486398786306,
+            -0.004757480230182409,
+            0.0018454419914633036,
+            -0.00196437886916101,
+            -0.01573037914931774,
+            -0.004435199312865734,
+            0.009960015304386616,
+            -0.024662164971232414,
+            0.0048764171078801155,
+            0.010842450894415379,
+            0.004991517867892981,
+            0.0009601286146789789,
+            -0.01124146580696106,
+            -0.03256572037935257,
+            5.1435461500659585e-05,
+            -0.005647589452564716,
+            -0.0021255193278193474,
+            0.004853397142142057,
+            0.010919184423983097,
+            0.0222373828291893,
+            0.004396832548081875,
+            -0.0013831222895532846,
+            0.021715596318244934,
+            0.0007064282544888556,
+            -0.0033014612272381783,
+            0.016866035759449005,
+            -0.00735491095110774,
+            -0.013129111379384995,
+            0.00012517160212155432,
+            0.003560436889529228,
+            0.014755862765014172,
+            0.0057358331978321075,
+            0.01617543213069439,
+            0.010351356118917465,
+            -0.02555227279663086,
+            -0.020840832963585854,
+            9.196036990033463e-05,
+            0.026319608092308044,
+            -0.012653362937271595,
+            -0.01912200078368187,
+            -0.00412826519459486,
+            -0.005187188275158405,
+            0.006925203371793032,
+            -0.012231328524649143,
+            0.008977825753390789,
+            0.016451673582196236,
+            -0.011364239268004894,
+            -0.026273569092154503,
+            -0.020472511649131775,
+            -0.005847096908837557,
+            0.009775854647159576,
+            0.009814221411943436,
+            -0.006741042714565992,
+            0.012983317486941814,
+            0.002706776140257716,
+            -0.021761635318398476,
+            0.022835904732346535,
+            0.013489758595824242,
+            0.005689792800694704,
+            -0.005229391623288393,
+            -0.004523443058133125,
+            -0.011479339562356472,
+            0.0068753263913095,
+            0.004070715047419071,
+            -0.007293524220585823,
+            -0.030923621729016304,
+            -0.0024880855344235897,
+            -0.006537698674947023,
+            -0.00125747115816921,
+            -0.009514960460364819,
+            0.00602742051705718,
+            -0.029972126707434654,
+            0.004020838066935539,
+            -0.0060926442965865135,
+            -0.0041819787584245205,
+            -0.006449454929679632,
+            -0.0009423839510418475,
+            -0.009238719940185547,
+            -0.028237948194146156,
+            -0.027271104976534843,
+            0.0011807375121861696,
+            0.01199345476925373,
+            -0.0016392205143347383,
+            -0.002380658406764269,
+            -0.024247802793979645,
+            -0.0004181978583801538,
+            0.008640198037028313,
+            -0.00744699127972126,
+            0.01846209354698658,
+            -0.003890391206368804,
+            -0.006952059920877218,
+            -0.0004949314170517027,
+            -0.0012392469216138124,
+            -0.026749316602945328,
+            0.017080888152122498,
+            0.0031000354792922735,
+            0.014326154254376888,
+            -0.02435522899031639,
+            0.011279832571744919,
+            -0.013366985134780407,
+            0.007780782412737608,
+            -0.01841605268418789,
+            -0.008762971498072147,
+            -0.00786518957465887,
+            -0.012392468750476837,
+            -0.007826822809875011,
+            -0.029143404215574265,
+            0.00844069104641676,
+            -0.008801338262856007,
+            -0.025813167914748192,
+            -0.02908201701939106,
+            0.009353820234537125,
+            -0.015776418149471283,
+            0.015561564825475216,
+            -0.009215699508786201,
+            0.04527279734611511,
+            0.020027456805109978,
+            0.015354383736848831,
+            0.008824358694255352,
+            -0.017863571643829346,
+            -0.0032937878277152777,
+            0.012653362937271595,
+            -0.026273569092154503,
+            0.0003601681091822684,
+            0.012553608976304531,
+            0.01487096305936575,
+            -0.011318199336528778,
+            0.001353388070128858,
+            0.01651306077837944,
+            0.015177897177636623,
+            0.01622147299349308,
+            0.0024036786053329706,
+            -0.009131292812526226,
+            0.010305316187441349,
+            0.023633934557437897,
+            0.02567504718899727,
+            -0.004803520627319813,
+            0.0019662973936647177,
+            0.0011625132756307721,
+            -0.018477439880371094,
+            -0.009077579714357853,
+            -0.013205844908952713,
+            -0.009783527813851833,
+            -0.02659584954380989,
+            0.011525379493832588,
+            -0.0038002291694283485,
+            -0.010013728402554989,
+            -0.01580711267888546,
+            -0.014978389255702496,
+            -0.005133474711328745,
+            0.006599085405468941,
+            0.021546781063079834,
+            -0.010712004266679287,
+            0.010919184423983097,
+            -0.010052095167338848,
+            0.0020545409061014652,
+            -0.00196437886916101,
+            0.0081491032615304,
+            0.027225064113736153,
+            0.010274622589349747,
+            -0.055524397641420364,
+            -0.006499331910163164,
+            0.002762407995760441,
+            -0.010650617070496082,
+            -0.051503561437129974,
+            0.002656899392604828,
+            0.007324217818677425,
+            -0.03185977041721344,
+            0.008655544370412827,
+            -0.0023154348600655794,
+            -0.0079265758395195,
+            0.006595248822122812,
+            0.019935376942157745,
+            -0.016206126660108566,
+            0.013259558007121086,
+            -6.013542588334531e-06,
+            0.004320099018514156,
+            -0.014924676157534122,
+            0.0062230913899838924,
+            -0.015147203579545021,
+            -0.014471948146820068,
+            0.01734178327023983,
+            -0.001127024064771831,
+            -0.002089071087539196,
+            0.012008801102638245,
+            0.02506117895245552,
+            0.02579781971871853,
+            -0.012162268161773682,
+            -0.01592988520860672,
+            -0.01879972033202648,
+            0.0007073874585330486,
+            -0.0055401623249053955,
+            -0.010712004266679287,
+            -0.0019624605774879456,
+            0.008417670615017414,
+            0.01209320779889822,
+            0.01987398974597454,
+            0.008417670615017414,
+            0.006315171252936125,
+            -3.35109798470512e-05,
+            0.009898628108203411,
+            0.0023422916419804096,
+            -0.020303698256611824,
+            -0.0065530454739928246,
+            0.012983317486941814,
+            0.019797256216406822,
+            0.0017936468357220292,
+            0.004297079052776098,
+            0.018185852095484734,
+            -0.00773857906460762,
+            -0.009208026342093945,
+            -0.0336092971265316,
+            -0.0006901223678141832,
+            -0.017203662544488907,
+            0.016236819326877594,
+            0.01945962943136692,
+            -0.02136262133717537,
+            -0.01022858265787363,
+            -0.01866159960627556,
+            0.02394086867570877,
+            -0.010735023766756058,
+            0.005912320222705603,
+            0.0022252730559557676,
+            0.008739951997995377,
+            0.0019068288384005427,
+            -0.008517424575984478,
+            -0.014272441156208515,
+            0.009277086704969406,
+            0.00048318158951587975,
+            0.007661845069378614,
+            -0.0061195008456707,
+            -0.012039494700729847,
+            0.001056045526638627,
+            -0.009860261343419552,
+            -0.028283987194299698,
+            0.011579093523323536,
+            0.0008670891402289271,
+            0.01821654476225376,
+            -0.019597750157117844,
+            0.01675860770046711,
+            0.0007380808820016682,
+            -0.013520452193915844,
+            -0.02788497321307659,
+            -0.015277650207281113,
+            0.003919166047126055,
+            0.003792555769905448,
+            0.012661036103963852,
+            0.0034817850682884455,
+            -0.01658979430794716,
+            0.008739951997995377,
+            -0.005647589452564716,
+            -0.005751179996877909,
+            -0.001697729923762381,
+            -0.01170186698436737,
+            -0.001447386690415442,
+            0.011072651483118534,
+            -0.01037437655031681,
+            -0.005574692506343126,
+            0.01468680240213871,
+            0.006411088164895773,
+            -0.002988772001117468,
+            0.02742457203567028,
+            0.01821654476225376,
+            0.02153143472969532,
+            0.0006488781073130667,
+            0.01066596433520317,
+            -0.0018473602831363678,
+            0.003558518597856164,
+            -0.03505188599228859,
+            0.025122566148638725,
+            -0.01576107181608677,
+            0.023373039439320564,
+            0.008532770909368992,
+            -0.00694822333753109,
+            -0.005881627090275288,
+            0.01331327110528946,
+            0.006088807247579098,
+            -0.004803520627319813,
+            0.00602742051705718,
+            -0.003128810552880168,
+            0.009844915010035038,
+            -0.0013102254597470164,
+            -0.02493840456008911,
+            0.020012110471725464,
+            -0.024416616186499596,
+            0.009990708902478218,
+            -0.002990690292790532,
+            0.009123619645833969,
+            -0.006894509773701429,
+            -5.931863415753469e-05,
+            -0.00457331957295537,
+            -0.040638089179992676,
+            -0.001970133977010846,
+            -0.010773390531539917,
+            0.004450546111911535,
+            0.016819994896650314,
+            -0.00021497384295798838,
+            -0.04220345616340637,
+            -0.005916156806051731,
+            0.01958240196108818,
+            -0.00021737176575697958,
+            0.009883281774818897,
+            -0.008271876722574234,
+            -0.004086061846464872,
+            -0.004320099018514156,
+            -0.007792292162775993,
+            0.016543753445148468,
+            0.013136784546077251,
+            -0.01563829742372036,
+            -0.023618588224053383,
+            0.012507569044828415,
+            -0.01879972033202648,
+            -0.019567055627703667,
+            0.00530996173620224,
+            -0.01267638336867094,
+            0.001553854439407587,
+            0.022268077358603477,
+            -0.004715276882052422,
+            0.011164732277393341,
+            -0.0056130592711269855,
+            0.013282578438520432,
+            -0.01543111726641655,
+            -0.0019471138948574662,
+            -0.0047651538625359535,
+            -0.02348046749830246,
+            0.025567619130015373,
+            -0.018937841057777405,
+            0.0019931539427489042,
+            -0.016528407111763954,
+            0.02228342369198799,
+            -0.020088844001293182,
+            -0.002275149803608656,
+            0.008010982535779476,
+            0.010174869559705257,
+            0.029803311452269554,
+            0.01345906499773264,
+            -0.007408624514937401,
+            0.009169659577310085,
+            0.008824358694255352,
+            0.019827950745821,
+            0.00156632368452847,
+            0.029803311452269554,
+            -0.005271594971418381,
+            0.01041274331510067,
+            -0.001517405966296792,
+            0.018968533724546432,
+            0.0067065125331282616,
+            0.029926085844635963,
+            0.0003174850717186928,
+            0.00015430638450197875,
+            -0.0001380004978273064,
+            0.018600212410092354,
+            -0.004369975998997688,
+            -0.005313798785209656,
+            0.007765435613691807,
+            -0.0059698703698813915,
+            -0.0015797520754858851,
+            -0.020441818982362747,
+            -0.005877790041267872,
+            0.001410938217304647,
+            0.013789019547402859,
+            -0.0005241860635578632,
+            0.005390532314777374,
+            0.0008200898300856352,
+            0.005586202722042799,
+            -0.014617742039263248,
+            0.011402606032788754,
+            -0.015668991953134537,
+            0.01883041486144066,
+            -0.01005976926535368,
+            0.01008278876543045,
+            -0.0009692406747490168,
+            0.00811073649674654,
+            0.011072651483118534,
+            -0.013574165292084217,
+            0.01614473946392536,
+            0.012139247730374336,
+            -0.010589230805635452,
+            0.016313552856445312,
+            -0.007228300906717777,
+            0.005083597730845213,
+            0.0038616161327809095,
+            0.0005870116874575615,
+            0.003153749043121934,
+            -0.018109118565917015,
+            -0.009461247362196445,
+            -0.0001405183138558641,
+            -0.012576629407703876,
+            -0.0030501587316393852,
+            0.012331082485616207,
+            -0.005505632609128952,
+            0.015507850795984268,
+            0.010197889059782028,
+            0.012285041622817516,
+            0.004933967255055904,
+            0.011172405444085598,
+            -0.0014186116168275476,
+            -0.0034299897961318493,
+            0.004684583283960819,
+            0.032964736223220825,
+            0.0008824358810670674,
+            0.025905247777700424,
+            -0.014978389255702496,
+            0.006084970664232969,
+            0.001282409531995654,
+            -0.0016181188402697444,
+            -0.008701585233211517,
+            0.016236819326877594,
+            0.020088844001293182,
+            0.0018934004474431276,
+            -0.011893700808286667,
+            -0.003499049926176667,
+            0.029404297471046448,
+            -0.005501795560121536,
+            0.007454664912074804,
+            0.010696657001972198,
+            -0.006196234375238419,
+            0.016605140641331673,
+            0.004381485749036074,
+            -0.024692857638001442,
+            0.0013457146706059575,
+            0.004047695081681013,
+            -0.019152695313096046,
+            0.020917566493153572,
+            0.02104034088551998,
+            0.0076042949222028255,
+            0.002739387797191739,
+            0.01345906499773264,
+            0.010497150011360645,
+            0.021623514592647552,
+            0.0032400742638856173,
+            0.004580993205308914,
+            0.002643470885232091,
+            -0.012952623888850212,
+            -0.00123445107601583,
+            -0.020963607355952263,
+            -0.011533052660524845,
+            -0.0020583774894475937,
+            0.018983881920576096,
+            0.0026530625764280558,
+            -0.013290251605212688,
+            -0.003959451336413622,
+            0.008855052292346954,
+            0.004381485749036074,
+            -0.006299824919551611,
+            -0.022053223103284836,
+            -0.008202817291021347,
+            0.024094335734844208,
+            0.008095390163362026,
+            0.010550864040851593,
+            -0.0031729324255138636,
+            -0.002160049509257078,
+            -0.0006109909154474735,
+            -0.006614432204514742,
+            -0.0006450414075516164,
+            -0.01597592607140541,
+            -0.0021063359454274178,
+            -0.016666527837514877,
+            0.015346710570156574,
+            0.015791764482855797,
+            0.01763337105512619,
+            0.018139811232686043,
+            -0.02102499268949032,
+            0.017602676525712013,
+            -0.0071438937447965145,
+            0.001838727737776935,
+            -0.0034414997790008783,
+            0.00010826624929904938,
+            0.019275467842817307,
+            0.029388951137661934,
+            0.0056015495210886,
+            0.011149385012686253,
+            -0.0013831222895532846,
+            0.005927667021751404,
+            -0.006457128562033176,
+            -0.013980853371322155,
+            -0.0033935413230210543,
+            -0.01892249472439289,
+            0.004907110705971718,
+            0.017848225310444832,
+            -0.015139530412852764,
+            -0.004922457505017519,
+            0.014349174685776234,
+            0.009867934510111809,
+            0.008340937085449696,
+            -0.008931785821914673,
+            -0.004112918395549059,
+            0.0008714054129086435,
+            0.01557691115885973,
+            -0.007585111539810896,
+            -0.016605140641331673,
+            0.008394651114940643,
+            -0.006146357394754887,
+            0.02123984694480896,
+            -0.004542626440525055,
+            -0.01605265960097313,
+            -0.010282295756042004,
+            0.003186360700055957,
+            -0.021439354866743088,
+            -0.014272441156208515,
+            0.01002140250056982,
+            0.0021312744356691837,
+            0.006936713121831417,
+            -0.017326436936855316,
+            0.013060051016509533,
+            0.004834213759750128,
+            0.03351721540093422,
+            -0.008133756928145885,
+            -0.0005424103001132607,
+            0.0003779127437155694,
+            0.005916156806051731,
+            0.0002481850970070809,
+            -0.01755663752555847,
+            0.026933476328849792,
+            0.006587575655430555,
+            0.013781346380710602,
+            -0.0003863054735120386,
+            -0.012645689770579338,
+            0.006994263269007206,
+            -0.01564597152173519,
+            0.001863666228018701,
+            -0.010205562226474285,
+            -0.020088844001293182,
+            -0.01058155670762062,
+            0.02165420912206173,
+            0.0070403036661446095,
+            -0.00732038076967001,
+            -0.0001907548139570281,
+            -0.0077769458293914795,
+            -0.016359593719244003,
+            -0.009852588176727295,
+            -0.015492504462599754,
+            0.008386977016925812,
+            -0.01663583517074585,
+            0.0029005282558500767,
+            0.008126083761453629,
+            0.009806548245251179,
+            0.005586202722042799,
+            0.021623514592647552,
+            0.002338455058634281,
+            0.016405634582042694,
+            0.0001647373428568244,
+            -0.0019049104303121567,
+            0.005651426035910845,
+            0.014771209098398685,
+            0.0072973608039319515,
+            -0.008555791340768337,
+            -0.0046423799358308315,
+            -0.012691729702055454,
+            0.0005198698490858078,
+            -0.009676100686192513,
+            0.01066596433520317,
+            0.0064302715472877026,
+            -0.0009404656011611223,
+            -0.00678324606269598,
+            0.021715596318244934,
+            -0.0038980646058917046,
+            0.007711722049862146,
+            -0.007957269437611103,
+            -0.004300915636122227,
+            0.01065829023718834,
+            0.0010848206002265215,
+            -0.013765999116003513,
+            0.008210490457713604,
+            0.00919268000870943,
+            -0.002263639820739627,
+            -0.003238155972212553,
+            -0.020672019571065903,
+            0.02874438837170601,
+            0.008885745890438557,
+            -0.010182542726397514,
+            0.014425908215343952,
+            -0.0052831051871180534,
+            0.019306162372231483,
+            0.016282860189676285,
+            0.0185388270765543,
+            -0.018323972821235657,
+            -0.019643789157271385,
+            0.007251320872455835,
+            -0.015561564825475216,
+            -0.0027835096698254347,
+            0.0029715069103986025,
+            -0.0022809049114584923,
+            -0.0002340373321203515,
+            -0.016068005934357643,
+            0.0009150475962087512,
+            -0.0046500531025230885,
+            0.0024938404094427824,
+            0.000545287795830518,
+            0.0077347420156002045,
+            0.0009395064553245902,
+            0.012760790064930916,
+            -0.006196234375238419,
+            -0.01783287711441517,
+            -0.006606759037822485,
+            0.013512779027223587,
+            0.006906019989401102,
+            -0.03600338473916054,
+            0.004980007652193308,
+            -0.012499895878136158,
+            -0.01751059666275978,
+            -0.002232946455478668,
+            -0.02157747559249401,
+            -0.02116311341524124,
+            -0.001316939597018063,
+            0.0009831486968323588,
+            -0.011510033160448074,
+            0.0021869062911719084,
+            0.006648962385952473,
+            -0.009300106205046177,
+            -0.00919268000870943,
+            0.0054595922119915485,
+            -0.0018751762108877301,
+            0.012645689770579338,
+            -0.012392468750476837,
+            -0.010443436913192272,
+            0.035174660384655,
+            0.013635552488267422,
+            -0.00766568211838603,
+            -0.013812039978802204,
+            -0.010873144492506981,
+            -0.014280114322900772,
+            0.010980571620166302,
+            -0.0030827706214040518,
+            -0.00466539990156889,
+            -0.037814296782016754,
+            0.0013543472159653902,
+            0.0011884109117090702,
+            0.01524695660918951,
+            -0.008862725459039211,
+            0.014533335343003273,
+            0.005793383345007896,
+            -0.00058077706489712,
+            -0.012354101985692978,
+            -6.549328645633068e-06,
+            0.01080408412963152,
+            0.010136502794921398,
+            -0.007197607308626175,
+            0.012100880965590477,
+            -0.013006336987018585,
+            0.007527561392635107,
+            -0.014633088372647762,
+            -0.0077769458293914795,
+            -0.01933685503900051,
+            -0.01787891797721386,
+            -0.007516051642596722,
+            0.01634424738585949,
+            0.009890954941511154,
+            -0.01787891797721386,
+            -0.004815030377358198,
+            -0.014525661244988441,
+            0.013451391831040382,
+            -0.011533052660524845,
+            -0.006015910767018795,
+            -0.01306772418320179,
+            0.001044535543769598,
+            -0.018784373998641968,
+            0.01385807991027832,
+            0.01775614358484745,
+            0.01639028638601303,
+            0.0004093255556654185,
+            -0.0053866952657699585,
+            -0.02248293161392212,
+            -0.01870764046907425,
+            -0.004462056327611208,
+            0.002572492463514209,
+            -0.013781346380710602,
+            -0.003744597313925624,
+            0.011732560582458973,
+            0.021915102377533913,
+            -0.0017984426813200116,
+            -0.013129111379384995,
+            0.00854811817407608,
+            -0.016958115622401237,
+            -0.017311088740825653,
+            -0.018277931958436966,
+            0.007358747534453869,
+            0.008172123692929745,
+            0.01605265960097313,
+            0.0024746570270508528,
+            0.006058114115148783,
+            -0.004623196553438902,
+            -0.007730905432254076,
+            -0.024278495460748672,
+            0.005110454745590687,
+            -0.0112030990421772,
+            0.010075115598738194,
+            -0.0021408661268651485,
+            -0.004070715047419071,
+            0.023465121164917946,
+            0.009760508313775063,
+            -0.004995354451239109,
+            -0.00018631866259966046,
+            0.020840832963585854,
+            -0.008686237968504429,
+            -0.00602742051705718,
+            -0.008057023398578167,
+            0.017771491780877113,
+            -0.021961143240332603,
+            0.00016090067219920456,
+            -0.011187751777470112,
+            -0.0222373828291893,
+            0.003272686153650284,
+            0.012200634926557541,
+            0.00606578728184104,
+            -0.003355174558237195,
+            0.003453009994700551,
+            -0.008433017879724503,
+            0.024094335734844208,
+            -0.01435684785246849,
+            0.002348046749830246,
+            0.013620206154882908,
+            -0.0021638860926032066,
+            -0.002465065335854888,
+            0.013727632351219654,
+            -0.005417388863861561,
+            0.009530307725071907,
+            0.0035738651640713215,
+            0.00011857732170028612,
+            -0.009660754352807999,
+            0.016988808289170265,
+            0.02535276673734188,
+            -0.00860950443893671,
+            0.0008152939844876528,
+            -0.003782964078709483,
+            0.00602742051705718,
+            -0.0070403036661446095,
+            0.03201323747634888,
+            -0.008287223987281322,
+            0.0005500836414285004,
+            0.0010100053623318672,
+            0.03161422535777092,
+            -0.0014771209098398685,
+            0.010044422000646591,
+            0.0014541008276864886,
+            -0.0034184798132628202,
+            0.022927986457943916,
+            0.0025667373556643724,
+            0.004531116224825382,
+            0.027102291584014893,
+            -0.013574165292084217,
+            -0.00493780430406332,
+            0.005543999373912811,
+            -0.015200916677713394,
+            0.012392468750476837,
+            -0.02048785798251629,
+            0.0009188842959702015,
+            -0.01522393710911274,
+            0.014280114322900772,
+            0.019981417804956436,
+            0.030386487022042274,
+            -0.014418235048651695,
+            -0.00619239779189229,
+            -0.00728201400488615,
+            0.03143006190657616,
+            0.00044217711547389627,
+            0.006288314703851938,
+            -0.00030045982566662133,
+            0.011471666395664215,
+            -0.015177897177636623,
+            -0.01962844282388687,
+            0.012499895878136158,
+            -0.019551709294319153,
+            -0.03413108363747597,
+            -0.004634706769138575,
+            0.005670609418302774,
+            -0.0030175470747053623,
+            0.0037100673653185368,
+            0.012146921828389168,
+            0.004358465783298016,
+            0.022145302966237068,
+            0.007243647240102291,
+            0.009752834215760231,
+            -0.016482368111610413,
+            3.229703224860714e-06,
+            -0.006848469842225313,
+            -0.025398805737495422,
+            0.016037313267588615,
+            -0.021516088396310806,
+            0.0014138157712295651,
+            -0.008433017879724503,
+            -0.002647307701408863,
+            0.009967688471078873,
+            0.004780500195920467,
+            0.014778882265090942,
+            0.005789546761661768,
+            0.016282860189676285,
+            0.019551709294319153,
+            -0.01573037914931774,
+            0.011586766690015793,
+            0.027025558054447174,
+            0.00113853404764086,
+            -0.017188316211104393,
+            -0.007784618996083736,
+            -0.01975121721625328,
+            -0.01945962943136692,
+            0.011256812140345573,
+            0.02041112445294857,
+            -0.00846371054649353,
+            -0.001863666228018701,
+            0.03904203325510025,
+            -0.013113764114677906,
+            0.023956215009093285,
+            -0.00775776244699955,
+            0.005240901838988066,
+            0.03468356654047966,
+            0.010827104561030865,
+            0.018692294135689735,
+            -0.00613868422806263,
+            -0.034407325088977814,
+            -0.002121682744473219,
+            -0.00680242944508791,
+            0.0065607186406850815,
+            -0.011049631983041763,
+            0.018032385036349297,
+            -0.013673919253051281,
+            -0.019812602549791336,
+            0.02036508545279503,
+            -0.02243689075112343,
+            0.021377967670559883,
+            0.007562091574072838,
+            0.008578810840845108,
+            -0.006269131321460009,
+            -0.011479339562356472,
+            -3.974558421759866e-05,
+            -0.0077270688489079475,
+            0.011318199336528778,
+            0.008034002967178822,
+            -0.011832313612103462,
+            -0.006806266028434038,
+            0.0031729324255138636,
+            -0.004634706769138575,
+            -0.02311214618384838,
+            0.012860543094575405,
+            0.008640198037028313,
+            0.013428371399641037,
+            0.007381767965853214,
+            0.005075924564152956,
+            0.010635270737111568,
+            0.0034817850682884455,
+            -0.013113764114677906,
+            -0.01203182153403759,
+            0.00914663914591074,
+            -0.006127174012362957,
+            -0.0007332850364036858,
+            -0.026687929406762123,
+            0.014702148735523224,
+            -0.022743824869394302,
+            -0.0032573393546044827,
+            -0.0018195443553850055,
+            0.0039786347188055515,
+            0.005486449226737022,
+            -0.01866159960627556,
+            -0.005636079702526331,
+            -0.00028990895953029394,
+            -0.009392186999320984,
+            0.01188602764159441,
+            -0.013620206154882908,
+            0.01646701991558075,
+            0.027056250721216202,
+            -0.01487096305936575,
+            -0.0036199053283780813,
+            0.00554783595725894,
+            0.0077040488831698895,
+            -0.012522916309535503,
+            0.016988808289170265,
+            0.015108836814761162,
+            -0.01205484103411436,
+            0.008847379125654697,
+            0.03083154186606407,
+            -0.006579902023077011,
+            0.0005088393809273839,
+            -0.00012493181566242129,
+            0.0024995955172926188,
+            0.029726577922701836,
+            0.0016066087409853935,
+            0.024293843656778336,
+            0.005873953457921743,
+            -0.0008862725226208568,
+            -0.014610068872570992,
+            -0.014364521019160748,
+            -0.009522633627057076,
+            -0.009990708902478218,
+            -0.005520978942513466,
+            -0.010497150011360645,
+            0.009706794284284115,
+            0.012699402868747711,
+            -0.004676910117268562,
+            0.0038884729146957397,
+            -0.0019164205295965075,
+            0.02131658047437668,
+            -0.005052904598414898,
+            0.013113764114677906,
+            -0.022206690162420273,
+            0.01962844282388687,
+            0.020595286041498184,
+            0.011839987710118294,
+            -0.016313552856445312,
+            -0.005643752869218588,
+            0.016819994896650314,
+            0.025214646011590958,
+            -0.00538285868242383,
+            0.001968215685337782,
+            0.019444283097982407,
+            0.0031384022440761328,
+            0.005252411589026451,
+            -0.0026626542676240206,
+            -0.01987398974597454,
+            -0.027194371446967125,
+            -0.003568110289052129,
+            -0.006737206131219864,
+            0.002689511049538851,
+            0.004097571596503258,
+            -0.011839987710118294,
+            -0.011609786190092564,
+            0.0046500531025230885,
+            -0.005026047583669424,
+            0.0011020856909453869,
+            -0.0099139753729105,
+            -0.007849842309951782,
+            -0.038059841841459274,
+            0.020150231197476387,
+            0.0004304272879380733,
+            0.004396832548081875,
+            0.004239528905600309,
+            0.015108836814761162,
+            -0.007723232265561819,
+            -0.0016229146858677268,
+            0.015085816383361816,
+            0.006069623865187168,
+            -0.0018233810551464558,
+            -0.0003510560200084001,
+            -0.0035719468723982573,
+            -0.007197607308626175,
+            0.0104741295799613,
+            -0.004980007652193308,
+            -0.010274622589349747,
+            0.0084023242816329,
+            -0.012852869927883148,
+            -0.015062796883285046,
+            0.008225836791098118,
+            -0.025153258815407753,
+            0.00501070125028491,
+            -0.000793712679296732,
+            0.010159522294998169,
+            -0.004661563318222761,
+            -0.008809012360870838,
+            0.0010032912250608206,
+            0.013873426243662834,
+            0.007032630033791065,
+            -0.004726787097752094,
+            0.016083352267742157,
+            -0.007400951348245144,
+            0.007124710362404585,
+            0.029220137745141983,
+            -0.007949596270918846,
+            0.008601831272244453,
+            0.018477439880371094,
+            0.0013687347527593374,
+            0.008471384644508362,
+            0.01130285207182169,
+            -0.03643308952450752,
+            -0.016743261367082596,
+            -0.0008809971041046083,
+            0.0051603312604129314,
+            -0.0011174323735758662,
+            0.0188764538615942,
+            0.00919268000870943,
+            0.02725575864315033,
+            -0.006177050992846489,
+            0.012684056535363197,
+            0.0019509505946189165,
+            0.023572547361254692,
+            0.012108555063605309,
+            0.01199345476925373,
+            -0.005747342947870493,
+            -0.0021312744356691837,
+            -0.03588061034679413,
+            0.0035661919973790646,
+            0.022175997495651245,
+            0.019490322098135948,
+            -0.02579781971871853,
+            0.017203662544488907,
+            -0.0057435063645243645,
+            0.0033129712101072073,
+            -0.013428371399641037,
+            0.017694758251309395,
+            0.004392995964735746,
+            0.017571983858942986,
+            -0.008156776428222656,
+            -0.002417106879875064,
+            0.016866035759449005,
+            0.010466456413269043,
+            -0.012914257124066353,
+            0.021470047533512115,
+            0.012806829996407032,
+            0.02119380794465542,
+            0.019198734313249588,
+            -0.018308626487851143,
+            0.00908525288105011,
+            0.00875529833137989,
+            -0.012200634926557541,
+            -0.0081491032615304,
+            0.00457331957295537,
+            0.007232137490063906,
+            -0.016605140641331673,
+            -0.005018374416977167,
+            -0.0005678283050656319,
+            0.00023475672060158104,
+            0.012768463231623173,
+            -0.012668709270656109,
+            0.000462079857243225,
+            0.00412826519459486,
+            0.011080324649810791,
+            -0.0065607186406850815,
+            0.01734178327023983,
+            -0.028729042038321495,
+            0.012223655357956886,
+            -0.022022530436515808,
+            -0.0239715613424778,
+            0.0014195707626640797,
+            -0.004481239710003138,
+            -0.0036333338357508183,
+            -0.0051027811132371426,
+            0.0015154876746237278,
+            0.005647589452564716,
+            0.009983035735785961,
+            0.028391415253281593,
+            0.007715558633208275,
+            0.0013601022073999047,
+            -0.0004165193240623921,
+            -0.02302006632089615,
+            -0.010827104561030865,
+            -0.006054277531802654,
+            -0.0021389478351920843,
+            -0.006445618346333504,
+            -0.0336092971265316,
+            -0.00022852211259305477,
+            -0.0057358331978321075,
+            0.0029216299299150705,
+            0.0008435894851572812,
+            -0.02007349766790867,
+            -0.021500742062926292,
+            0.00926941353827715,
+            -0.008870398625731468,
+            -0.014026893302798271,
+            -0.0168046485632658,
+            0.006472475361078978,
+            0.00037431585951708257,
+            -0.0036563538014888763,
+            0.010151849128305912,
+            0.01846209354698658,
+            -0.004676910117268562,
+            -0.0006843673763796687,
+            0.010090461932122707,
+            0.0026396343018859625,
+            -0.02165420912206173,
+            0.008617177605628967,
+            -0.014111299999058247,
+            0.003951778169721365,
+            -0.01130285207182169,
+            -0.01709623634815216,
+            -0.004964660853147507,
+            -0.003967124503105879,
+            -0.028391415253281593,
+            0.0011673092376440763,
+            0.011832313612103462,
+            -0.007067160215228796,
+            0.013635552488267422,
+            0.020441818982362747,
+            0.010405070148408413,
+            0.00084119156235829,
+            -0.014863288961350918,
+            0.011425626464188099,
+            -0.01457937527447939,
+            -0.012998663820326328,
+            0.01970517635345459,
+            0.012929603457450867,
+            0.0112030990421772,
+            -0.0008503036806359887,
+            -0.014019220136106014,
+            0.011072651483118534,
+            0.015745725482702255,
+            0.025245338678359985,
+            -0.006272967904806137,
+            -0.0007697334513068199,
+            -0.0162982065230608,
+            -0.018446745350956917,
+            0.034161776304244995,
+            -0.011932067573070526,
+            -0.013681592419743538,
+            0.01655910164117813,
+            0.019613096490502357,
+            0.012668709270656109,
+            -0.005616896320134401,
+            -0.004684583283960819,
+            -0.0014684883644804358,
+            0.012346428819000721,
+            0.005820239894092083,
+            -0.025199299678206444,
+            -0.010681310668587685,
+            0.004400669131428003,
+            0.0325964130461216,
+            0.011747906915843487,
+            0.006069623865187168,
+            -0.043001484125852585,
+            0.014172687195241451,
+            0.0008853133767843246,
+            0.0009505368652753532,
+            0.013980853371322155,
+            0.029481031000614166,
+            0.020257657393813133,
+            0.0009706794517114758,
+            0.01962844282388687,
+            -0.015914538875222206,
+            0.02269778586924076,
+            -0.020978953689336777,
+            -0.010919184423983097,
+            -0.0241403765976429,
+            -0.005006864201277494,
+            -0.008479057811200619,
+            0.0028257130179554224,
+            0.009031538851559162,
+            0.009637733921408653,
+            -0.012262022122740746,
+            0.02021161839365959,
+            0.013221191242337227,
+            -0.010044422000646591,
+            -0.001552895293571055,
+            -0.02070271223783493,
+            0.02547553926706314,
+            0.003771454095840454,
+            -0.0008680483442731202,
+            -0.010612250305712223,
+            -0.005018374416977167,
+            -0.021132420748472214,
+            0.0018368094461038709,
+            -0.0028506515081971884,
+            0.012906583957374096,
+            -0.0016085271490737796,
+            0.008348610252141953,
+            0.016942769289016724,
+            -0.015208589844405651,
+            -0.003878881223499775,
+            0.007846006192266941,
+            0.02104034088551998,
+            0.006437945179641247,
+            0.01190904714167118,
+            0.02854488231241703,
+            -0.0013936731265857816,
+            -0.027485959231853485,
+            0.012008801102638245,
+            -0.006403414998203516,
+            -0.003366684541106224,
+            0.01111869141459465,
+            0.02131658047437668,
+            -0.021746288985013962,
+            0.0011557991383597255,
+            -0.007335727568715811,
+            0.020150231197476387,
+            3.1173007300822064e-05,
+            0.013198170810937881,
+            0.009138965979218483,
+            0.003936431370675564,
+            -0.01111869141459465,
+            -0.01421105396002531,
+            -0.008026329800486565,
+            0.01170186698436737,
+            -0.026979517191648483,
+            -0.0024036786053329706,
+            0.001851196982897818,
+            -0.003021383658051491,
+            0.006311334669589996,
+            0.006230764556676149,
+            -0.022099263966083527,
+            -0.010635270737111568,
+            -0.0028564066160470247,
+            -0.010182542726397514,
+            -0.005206371657550335,
+            0.012476875446736813,
+            -0.005409715697169304,
+            0.0005798179190605879,
+            0.00875529833137989,
+            0.009630060754716396,
+            0.004300915636122227,
+            0.025506233796477318,
+            -0.012584302574396133,
+            0.01924477517604828,
+            -0.030647380277514458,
+            0.001827217754907906,
+            -0.008578810840845108,
+            0.024171069264411926,
+            -0.0024497187696397305,
+            -0.019075961783528328,
+            0.012047167867422104,
+            -0.008195143193006516,
+            0.001614282140508294,
+            -0.002810366451740265,
+            -0.008433017879724503,
+            -0.0065300255082547665,
+            0.012545935809612274,
+            -0.018155159428715706,
+            0.004308588802814484,
+            0.025444846600294113,
+            0.0429094024002552,
+            -0.013643225654959679,
+            -0.013121437281370163,
+            0.0052255550399422646,
+            0.009983035735785961,
+            0.024094335734844208,
+            -0.01144864596426487,
+            -0.009307780303061008,
+            -0.01159443985670805,
+            0.0075812749564647675,
+            -0.016896728426218033,
+            -0.023833442479372025,
+            0.011563746258616447,
+            -0.015960579738020897,
+            -0.016850687563419342,
+            -0.018155159428715706,
+            -0.017955651506781578,
+            -0.0027662445791065693,
+            0.013773673214018345,
+            0.005904647056013346,
+            -0.0028046113438904285,
+            -0.0015615278389304876,
+            -0.006111827678978443,
+            -0.021270541474223137,
+            0.016896728426218033,
+            0.005133474711328745,
+            -0.01614473946392536,
+            -0.004373812582343817,
+            -0.012039494700729847,
+            0.002562900772318244,
+            0.0037522707134485245,
+            -0.003075097221881151,
+            0.05506399646401405,
+            0.003518233308568597,
+            0.009507287293672562,
+            -0.02107103355228901,
+            0.004266385454684496,
+            -0.010796410962939262,
+            -0.01074269786477089,
+            -0.007707885466516018,
+            -0.012116228230297565,
+            -0.003617987036705017,
+            -0.011479339562356472,
+            0.0019969907589256763,
+            -0.01543111726641655,
+            0.008133756928145885,
+            -0.013673919253051281,
+            -0.013474412262439728,
+            -0.017571983858942986,
+            -0.0015864662127569318,
+            -0.003451091470196843,
+            0.016114046797156334,
+            0.01325188484042883,
+            -0.016374940052628517,
+            -0.0030904437880963087,
+            0.022268077358603477,
+            0.007895882241427898,
+            0.018968533724546432,
+            -0.0035048050340265036,
+            -0.013129111379384995,
+            -0.020012110471725464,
+            0.0006565514486283064,
+            0.013075397349894047,
+            -0.0075812749564647675,
+            0.002915875054895878,
+            -0.029097363352775574,
+            -0.006748715881258249,
+            0.0014090199256315827,
+            0.01600661873817444,
+            0.01668187417089939,
+            0.008862725459039211,
+            -0.0018732579192146659,
+            -0.020856179296970367,
+            0.0332716703414917,
+            -0.007753925397992134,
+            -0.022743824869394302,
+            0.0019758890848606825,
+            0.007328054402023554,
+            -0.005144984927028418,
+            0.0024036786053329706,
+            0.013351637870073318,
+            0.004903274122625589,
+            -0.008340937085449696,
+            -0.004887927323579788,
+            0.0011836150661110878,
+            -0.021715596318244934,
+            0.01838536001741886,
+            -0.0033263994846493006,
+            0.00854811817407608,
+            -0.004838050343096256,
+            0.004696093499660492,
+            -0.023158187046647072,
+            0.014126647263765335,
+            -0.008164450526237488,
+            -0.0069098565727472305,
+            0.008878071792423725,
+            -0.0023653118405491114,
+            0.02231411822140217,
+            -0.020273005589842796,
+            -0.008908765390515327,
+            0.008855052292346954,
+            -0.022129956632852554,
+            -0.02206856943666935,
+            -0.0004939722712151706,
+            -0.019275467842817307,
+            -0.0014780800556764007,
+            -0.00021233613369986415,
+            -0.032964736223220825,
+            -0.026856742799282074,
+            0.003044403623789549,
+            -0.0073472377844154835,
+            -0.008287223987281322,
+            0.012753116898238659,
+            -7.697334513068199e-05,
+            -0.013850406743586063,
+            -0.003665945492684841,
+            0.01041274331510067,
+            0.00023139962286222726,
+            0.01800169236958027,
+            -0.021930448710918427,
+            0.0020372758153826,
+            0.004431362729519606,
+            -0.004327772185206413,
+            0.006725695915520191,
+            0.01712692901492119,
+            0.0005990013014525175,
+            0.0003234798787161708,
+            0.006084970664232969,
+            0.0034932950511574745,
+            0.010850124061107635,
+            0.015745725482702255,
+            0.015032103285193443,
+            -0.004174305126070976,
+            0.016068005934357643,
+            -0.001623873831704259,
+            2.7096535632153973e-05,
+            -0.02460077777504921,
+            2.6736846848507412e-05,
+            0.012415489181876183,
+            0.004450546111911535,
+            -0.02410968206822872,
+            -0.0031997892074286938,
+            -0.03278057277202606,
+            0.005133474711328745,
+            -0.019382895901799202,
+            0.0013802447356283665,
+            0.017571983858942986,
+            0.00728201400488615,
+            -0.004446709528565407,
+            0.014402887783944607,
+            0.022789865732192993,
+            -0.0010502905352041125,
+            0.0006623064982704818,
+            0.004515769425779581,
+            -0.003224727464839816,
+            0.002537962282076478,
+            0.0022080079652369022,
+            -0.01597592607140541,
+            -0.01414966769516468,
+            0.008156776428222656,
+            0.004611686337739229,
+            0.011310526169836521,
+            0.007385604549199343,
+            -0.015446464531123638,
+            0.012223655357956886,
+            -0.013934813439846039,
+            0.008164450526237488,
+            -0.00473829684779048,
+            -0.013988526538014412,
+            -0.013466738164424896,
+            0.005359838716685772,
+            0.017618024721741676,
+            -0.006012073718011379,
+            0.033425137400627136,
+            -0.00925406627357006,
+            0.01533136423677206,
+            -0.019306162372231483,
+            -0.009491940960288048,
+            -0.010282295756042004,
+            0.01184766087681055,
+            -0.01567666418850422,
+            -0.0009020988363772631,
+            0.0035508451983332634,
+            -0.010573883540928364,
+            -0.010351356118917465,
+            0.005294615402817726,
+            -0.0030309753492474556,
+            0.03956381976604462,
+            -0.0024631470441818237,
+            0.025705739855766296,
+            0.006031257100403309,
+            -0.011517706327140331,
+            -0.010075115598738194,
+            0.002643470885232091,
+            0.00800330936908722,
+            0.0180937722325325,
+            -0.006568392273038626,
+            -0.004354629199951887,
+            0.011041957885026932,
+            0.017955651506781578,
+            0.007251320872455835,
+            0.011563746258616447,
+            -0.004093735013157129,
+            0.0037311690393835306,
+            0.002374903531745076,
+            -0.013666246086359024,
+            -0.0031096271704882383,
+            0.011916721239686012,
+            0.017019502818584442,
+            0.00786518957465887,
+            0.012852869927883148,
+            -0.0003946982033085078,
+            0.004611686337739229,
+            0.013996199704706669,
+            -0.013121437281370163,
+            -0.005240901838988066,
+            0.008241184055805206,
+            -0.01721900887787342,
+            0.009223372675478458,
+            0.0044083427637815475,
+            -0.004638543352484703,
+            0.00893945898860693,
+            -0.005348328500986099,
+            -0.0018483194289729,
+            -0.02099430002272129,
+            -0.014134320430457592,
+            -0.0033494196832180023,
+            0.020932912826538086,
+            -0.006299824919551611,
+            0.0015452218940481544,
+            -0.006790919695049524,
+            -0.02567504718899727,
+            0.01705019548535347,
+            0.004550299607217312,
+            0.0065530454739928246,
+            0.005966033786535263
+          ],
+          "title": "2 Related Work",
+          "keyphrases": [
+            "neural network capacity",
+            "model parallelism",
+            "conditional computation",
+            "mixture of experts",
+            "Switch Transformer"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "294d1382-75f0-4a45-bc91-9727e57552f3",
+        "properties": {
+          "page_content": "1 Introduction And Related Work 1.1 Conditional Computation\n\nExploiting scale in both training data and model size has been central to the success of deep learning. When datasets are sufficiently large, increasing the capacity (number of parameters) of neural networks can give much better prediction accuracy. This has been shown in domains such as text (Sutskever et al., 2014; Bahdanau et al., 2014; Jozefowicz et al., 2016; Wu et al., 2016), images (Krizhevsky et al., 2012; Le et al., 2012), and audio (Hinton et al., 2012; Amodei et al., 2015). For typical deep learning models, where the entire model is activated for every example, this leads to a roughly quadratic blow-up in training costs, as both the model size and the number of training examples increase. Unfortunately, the advances in computing power and distributed computation fall short of meeting such demand.\n\nVarious forms of conditional computation have been proposed as a way to increase model capacity without a proportional increase in computational costs (Davis & Arel, 2013; Bengio et al., 2013; Eigen et al., 2013; Ludovic Denoyer, 2014; Cho & Bengio, 2014; Bengio et al., 2015; Almahairi et al., 2015). In these schemes, large parts of a network are active or inactive on a per-example basis. The gating decisions may be binary or sparse and continuous, stochastic or deterministic. Various forms of reinforcement learning and back-propagation are proposed for trarining the gating decisions.\n\nWhile these ideas are promising in theory, no work to date has yet demonstrated massive improvements in model capacity, training time, or model quality. We blame this on a combination of the following challenges:\n\nModern computing devices, especially GPUs, are much faster at arithmetic than at branching. Most of the works above recognize this and propose turning on/off large chunks of the network with each gating decision.\n\nLarge batch sizes are critical for performance, as they amortize the costs of parameter transfers and updates. Conditional computation reduces the batch sizes for the conditionally active chunks of the network.\n\nNetwork bandwidth can be a bottleneck. A cluster of GPUs may have computational power thousands of times greater than the aggregate inter-device network bandwidth. To be computationally efficient, the relative computational versus network demands of an algorithm must exceed this ratio. Embedding layers, which can be seen as a form of conditional computation, are handicapped by this very problem. Since the embeddings generally need to be sent across the network, the number of (example, parameter) interactions is limited by network bandwidth instead of computational capacity.\n\nDepending on the scheme, loss terms may be necessary to achieve the desired level of sparsity per-chunk and/or per example. Bengio et al. (2015) use three such terms. These issues can affect both model quality and load-balancing.\n\nModel capacity is most critical for very large data sets. The existing literature on conditional computation deals with relatively small image recognition data sets consisting of up to 600,000 images. It is hard to imagine that the labels of these images provide a sufficient signal to adequately train a model with millions, let alone billions of parameters. In this work, we for the first time address all of the above challenges and finally realize the promise of conditional computation. We obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets.\n\n1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1). All parts of the network are trained jointly by back-propagation.\n\nWhile the introduced technique is generic, in this paper we focus on language modeling and machine translation tasks, which are known to benefit from very large models. In particular, we apply a MoE convolutionally between stacked LSTM layers (Hochreiter & Schmidhuber, 1997), as in Figure 1.\n\nThe MoE is called once for each position in the text, selecting a potentially different combination of experts at each position. The different experts tend to become highly specialized based on syntax and semantics (see Appendix E Table 9). On both language modeling and machine translation benchmarks, we improve on best published results at a fraction of the computational cost.\n\n1.3 Related Work On Mixtures Of Experts\n\nSince its introduction more than two decades ago (Jacobs et al., 1991; Jordan & Jacobs, 1994), the mixture-of-experts approach has been the subject of much research. Different types of expert architectures hae been proposed such as SVMs (Collobert et al., 2002), Gaussian Processes (Tresp, 2001; Theis & Bethge, 2015; Deisenroth & Ng, 2015), Dirichlet Processes (Shahbaba & Neal, 2009), and deep networks. Other work has focused on different expert configurations such as a hierarchical structure (Yao et al., 2009), infinite numbers of experts (Rasmussen & Ghahramani, 2002), and adding experts sequentially (Aljundi et al., 2016). Garmash & Monz (2016) suggest an ensemble model in the format of mixture of experts for machine translation. The gating network is trained on a pre-trained ensemble NMT model. The works above concern top-level mixtures of experts. The mixture of experts is the whole model. Eigen et al. (2013) introduce the idea of using multiple MoEs with their own gating networks as parts of a deep model. It is intuitive that the latter approach is more powerful, since complex problems may contain many sub-problems each requiring different experts. They also allude in their conclusion to the potential to introduce sparsity, turning MoEs into a vehicle for computational computation.\n\nOur work builds on this use of MoEs as a general purpose neural network component. While Eigen et al. (2013) uses two stacked MoEs allowing for two sets of gating decisions, our convolutional application of the MoE allows for different gating decisions at each position in the text. We also realize sparse gating and demonstrate its use as a practical way to massively increase model capacity.\n\n",
+          "embedding": [
+            -0.011026207357645035,
+            -0.0030965988989919424,
+            -0.017856089398264885,
+            0.0010101930238306522,
+            0.02138679102063179,
+            0.0306765865534544,
+            -0.014028750360012054,
+            0.025452885776758194,
+            -0.030126724392175674,
+            0.021791953593492508,
+            0.010809156112372875,
+            -0.003031483618542552,
+            0.0071120485663414,
+            -0.010223117657005787,
+            0.014021515846252441,
+            0.00018878943228628486,
+            0.01843489333987236,
+            0.003255769843235612,
+            0.03244193643331528,
+            -0.030329305678606033,
+            0.006352368742227554,
+            -0.010158002376556396,
+            -0.03186313435435295,
+            -0.031226450577378273,
+            0.018391482532024384,
+            0.023152140900492668,
+            0.040400486439466476,
+            -0.0036120957229286432,
+            -0.027333997189998627,
+            0.01920180767774582,
+            0.016163088381290436,
+            0.000524540722835809,
+            -0.03342590481042862,
+            -0.03348378464579582,
+            -0.058574918657541275,
+            -0.044220589101314545,
+            0.04589911922812462,
+            0.018217841163277626,
+            -0.003402279457077384,
+            0.018926875665783882,
+            0.037419646978378296,
+            -0.001556438859552145,
+            -0.03244193643331528,
+            -0.025858048349618912,
+            0.0014379650820046663,
+            0.016626130789518356,
+            -0.016466960310935974,
+            -0.03426516801118851,
+            -0.015902627259492874,
+            -0.01490419078618288,
+            0.02338366210460663,
+            -0.025742286816239357,
+            0.01846383325755596,
+            -0.000727121951058507,
+            -0.040603067725896835,
+            0.033049680292606354,
+            0.026639433577656746,
+            -0.004460404627025127,
+            0.04691202566027641,
+            0.04639110341668129,
+            0.0038635137025266886,
+            0.006019556894898415,
+            -0.04427846893668175,
+            -0.0017617333214730024,
+            0.02446891926229,
+            -0.015251473523676395,
+            -0.024526799097657204,
+            0.021010568365454674,
+            0.018594063818454742,
+            -0.00322140334174037,
+            -0.037680111825466156,
+            0.00494153518229723,
+            -0.03119751065969467,
+            0.01049804873764515,
+            0.005907413549721241,
+            0.01499101147055626,
+            0.02413610741496086,
+            0.04008214548230171,
+            0.021242089569568634,
+            0.011192613281309605,
+            -0.003957569133490324,
+            0.022862739861011505,
+            -0.006706885993480682,
+            0.010975562036037445,
+            0.027218235656619072,
+            -0.02911381796002388,
+            0.005502250976860523,
+            -0.06227926164865494,
+            0.004485727287828922,
+            -0.013276305980980396,
+            -0.0250043123960495,
+            0.013565707951784134,
+            -0.0021397643722593784,
+            0.0019570793956518173,
+            0.010577634908258915,
+            0.03658038377761841,
+            -0.0005281582125462592,
+            -0.013869579881429672,
+            0.005491398740559816,
+            -0.0063632214441895485,
+            -0.01565663516521454,
+            0.01856512390077114,
+            0.007361657451838255,
+            -0.0035723031032830477,
+            0.014448382891714573,
+            -0.0009423645096831024,
+            -0.023730944842100143,
+            0.026205329224467278,
+            0.008710993453860283,
+            0.08670476824045181,
+            -0.009463437832891941,
+            -0.022775918245315552,
+            -0.001728271134197712,
+            0.03600158169865608,
+            -0.0008876494830474257,
+            -0.011489249765872955,
+            -0.04349708557128906,
+            -0.04118186980485916,
+            0.007216956466436386,
+            -0.022110294550657272,
+            0.03328120335936546,
+            0.03843255341053009,
+            -0.015164652839303017,
+            -0.010158002376556396,
+            -0.05591242015361786,
+            0.07883304357528687,
+            -0.013695938512682915,
+            -0.035162314772605896,
+            -0.008703758008778095,
+            -0.027044596150517464,
+            -0.002848798641934991,
+            0.00745209539309144,
+            -0.01866641454398632,
+            -0.02755104750394821,
+            0.04659368470311165,
+            0.012415335513651371,
+            0.03773799166083336,
+            0.022139234468340874,
+            -0.01235022023320198,
+            -0.013630823232233524,
+            0.03244193643331528,
+            0.0003031935775652528,
+            0.004037154838442802,
+            -0.017783738672733307,
+            -0.007980253547430038,
+            0.005802505649626255,
+            -0.022732509300112724,
+            0.057648833841085434,
+            -0.03794057294726372,
+            0.02171960286796093,
+            -0.030300365760922432,
+            0.031023869290947914,
+            -0.016163088381290436,
+            0.03941652178764343,
+            -0.01902816630899906,
+            0.01389128528535366,
+            -0.0250043123960495,
+            -0.017957380041480064,
+            0.006417484022676945,
+            -0.011178143322467804,
+            -0.016987884417176247,
+            -0.019910842180252075,
+            -0.015613225288689137,
+            0.02323896251618862,
+            0.0025485444348305464,
+            0.03073446825146675,
+            0.0352201946079731,
+            0.019476739689707756,
+            -0.0614689365029335,
+            0.019447797909379005,
+            0.004789599217474461,
+            0.021632781252264977,
+            0.019968722015619278,
+            -0.0019860195461660624,
+            0.03134221211075783,
+            0.009297031909227371,
+            0.02205241471529007,
+            0.03252875804901123,
+            -0.005419048015028238,
+            0.0330207422375679,
+            -0.03001096285879612,
+            -0.01251662615686655,
+            0.02878100611269474,
+            0.008117719553411007,
+            0.025091134011745453,
+            0.008877399377524853,
+            0.015801336616277695,
+            -0.007813847623765469,
+            0.0341494083404541,
+            0.0012118698796257377,
+            0.01969379000365734,
+            0.023962466046214104,
+            0.036725085228681564,
+            -0.030589766800403595,
+            0.0233113132417202,
+            -0.04074776917695999,
+            -0.019375449046492577,
+            0.024092696607112885,
+            0.02338366210460663,
+            0.017002353444695473,
+            -0.03368636593222618,
+            -0.03212359547615051,
+            0.02480173110961914,
+            0.01720493473112583,
+            0.013695938512682915,
+            0.009246386587619781,
+            0.013703173957765102,
+            0.0147884301841259,
+            0.017595628276467323,
+            -0.037882693111896515,
+            0.009969891048967838,
+            0.00405524205416441,
+            -0.009940951131284237,
+            -0.04303404316306114,
+            0.006829881574958563,
+            -0.0030152045655995607,
+            -0.022240525111556053,
+            -0.0033299291972070932,
+            -0.0182033721357584,
+            -0.003758605569601059,
+            -0.002535883104428649,
+            0.01681424304842949,
+            0.021459141746163368,
+            -0.021632781252264977,
+            -0.0020529436878859997,
+            -0.017291756346821785,
+            -0.021835362538695335,
+            -0.03235511854290962,
+            0.003633800894021988,
+            -0.005400960333645344,
+            0.02464256063103676,
+            0.03140009194612503,
+            0.017885029315948486,
+            -0.004778746515512466,
+            0.03938758000731468,
+            -0.03368636593222618,
+            0.016626130789518356,
+            -0.0012697502970695496,
+            0.007488270755857229,
+            0.008862929418683052,
+            -0.024353159591555595,
+            0.04178961366415024,
+            -0.0409214086830616,
+            -0.005690362304449081,
+            0.011330079287290573,
+            0.010035006329417229,
+            -0.010794686153531075,
+            0.0044133770279586315,
+            -0.023354722186923027,
+            -0.02649473212659359,
+            0.0013222043635323644,
+            -0.012762618251144886,
+            -0.016553781926631927,
+            0.02205241471529007,
+            -0.02539500594139099,
+            0.03976380452513695,
+            0.030618706718087196,
+            -0.02490302175283432,
+            -0.01976614072918892,
+            -0.006916702259331942,
+            0.04123975336551666,
+            -0.009108920581638813,
+            -0.01268303208053112,
+            0.02569887787103653,
+            -0.025612056255340576,
+            0.0019950633868575096,
+            0.036956604570150375,
+            0.022587807849049568,
+            -0.004930682480335236,
+            -0.01800079084932804,
+            0.0038237208500504494,
+            -0.05845915898680687,
+            0.0035361277405172586,
+            -0.041471272706985474,
+            0.05614394322037697,
+            -0.03736176714301109,
+            0.0041999430395662785,
+            0.0013303437735885382,
+            -0.012111463584005833,
+            0.04028472676873207,
+            0.022935090586543083,
+            -0.009405557066202164,
+            -0.022804860025644302,
+            0.02480173110961914,
+            0.015468524768948555,
+            -0.004123975057154894,
+            -0.015208062715828419,
+            -0.0024273572489619255,
+            -0.05116623267531395,
+            0.002250098856166005,
+            -0.005104323849081993,
+            -0.002651643706485629,
+            -0.048359036445617676,
+            0.015671106055378914,
+            0.028607364743947983,
+            -0.001820518053136766,
+            -0.011402430012822151,
+            -0.01437603309750557,
+            0.02708800509572029,
+            -0.017566686496138573,
+            -0.042397357523441315,
+            -0.006793706677854061,
+            0.013970870524644852,
+            0.008776108734309673,
+            -0.03209465742111206,
+            0.006746678613126278,
+            -0.003434837330132723,
+            0.02543841488659382,
+            -0.0076691466383636,
+            0.004728101193904877,
+            0.018188901245594025,
+            0.0015220724744722247,
+            0.01614861935377121,
+            -0.01747986674308777,
+            -0.006120847538113594,
+            0.02141573093831539,
+            0.021676192060112953,
+            0.0006932076648809016,
+            -0.016872122883796692,
+            -0.03316544368863106,
+            0.032181475311517715,
+            0.002027621027082205,
+            -0.0637841522693634,
+            -0.01426027249544859,
+            -0.03542277589440346,
+            0.01033164281398058,
+            0.004677455872297287,
+            0.03247087821364403,
+            -0.02228393591940403,
+            0.018246781080961227,
+            -0.011713536456227303,
+            -0.03542277589440346,
+            -0.024512330070137978,
+            0.013666998594999313,
+            -0.030879167839884758,
+            -0.03987956419587135,
+            -0.017118114978075027,
+            0.023991405963897705,
+            -0.013117135502398014,
+            -0.029026998206973076,
+            -0.01969379000365734,
+            -0.031255390495061874,
+            0.0030748937278985977,
+            0.027739159762859344,
+            0.01774032786488533,
+            -0.024946432560682297,
+            -0.034033648669719696,
+            0.0057265376672148705,
+            -0.01178588718175888,
+            -0.0018594063585624099,
+            0.0010210456093773246,
+            0.018261251971125603,
+            0.006489834748208523,
+            -0.011257728561758995,
+            -0.017552217468619347,
+            -0.04413377121090889,
+            -0.008407121524214745,
+            0.0005385585827752948,
+            -0.009904775768518448,
+            0.034699272364377975,
+            0.03180525451898575,
+            -0.012125934474170208,
+            -0.011612245813012123,
+            0.014745020307600498,
+            -0.047201428562402725,
+            -0.015367234125733376,
+            -0.025640996173024178,
+            -0.005306905135512352,
+            0.01684318296611309,
+            0.008928044699132442,
+            -0.003950334154069424,
+            0.042889341711997986,
+            0.002532265381887555,
+            -0.013507827185094357,
+            0.007922373712062836,
+            0.006956494878977537,
+            0.028144322335720062,
+            0.0057265376672148705,
+            -0.03637780249118805,
+            0.04381542652845383,
+            -0.02464256063103676,
+            0.010027770884335041,
+            0.013348656706511974,
+            -0.011728006415069103,
+            -0.013761053793132305,
+            0.018550653010606766,
+            -0.005636099260300398,
+            0.025858048349618912,
+            -0.020779047161340714,
+            0.061526816338300705,
+            -0.006761149037629366,
+            -0.05186079815030098,
+            -0.0147884301841259,
+            -0.04089247062802315,
+            0.021271029487252235,
+            0.013377596624195576,
+            -0.002273612655699253,
+            -0.020344944670796394,
+            -0.013030314818024635,
+            0.019288627430796623,
+            0.004000979475677013,
+            -0.009629843756556511,
+            -0.01235022023320198,
+            0.006366838701069355,
+            -0.01578686572611332,
+            -0.005990616511553526,
+            0.007376127410680056,
+            0.007625736761838198,
+            0.04187643527984619,
+            -0.03142903372645378,
+            -0.0031201126985251904,
+            -0.01594603806734085,
+            -0.048127513378858566,
+            0.0049560051411390305,
+            0.03154479339718819,
+            0.0364067442715168,
+            -0.023947997018694878,
+            0.0003253509057685733,
+            0.013175015337765217,
+            0.0018973903497681022,
+            -0.017653508111834526,
+            0.011113028042018414,
+            0.03336802497506142,
+            0.02457020990550518,
+            -0.0187966451048851,
+            -0.005245407111942768,
+            0.0062438431195914745,
+            -0.0020746488589793444,
+            0.00621852045878768,
+            -0.04031366482377052,
+            0.03637780249118805,
+            -0.01235022023320198,
+            0.02998202294111252,
+            -0.028708655387163162,
+            -0.013102664612233639,
+            -0.03481503203511238,
+            -0.010223117657005787,
+            0.03472821041941643,
+            -0.032876040786504745,
+            -0.007188016548752785,
+            0.017262814566493034,
+            -0.002754742978140712,
+            -0.006724973674863577,
+            0.018550653010606766,
+            -0.04636216163635254,
+            -0.019520148634910583,
+            -0.027536578476428986,
+            0.0008266037912108004,
+            -0.0018811115296557546,
+            -0.05871962010860443,
+            -0.061295293271541595,
+            -0.005350315012037754,
+            -0.054233890026807785,
+            -0.019491208717226982,
+            0.03935864195227623,
+            -0.009477907791733742,
+            -0.002881356282159686,
+            -0.019838491454720497,
+            0.009543023072183132,
+            -0.028694184496998787,
+            -0.0034185582771897316,
+            0.026205329224467278,
+            0.017190465703606606,
+            -0.018319131806492805,
+            -0.006236608140170574,
+            0.001457861391827464,
+            -0.062221381813287735,
+            -0.027507638558745384,
+            0.0053466977551579475,
+            -0.02477279119193554,
+            -0.008609702810645103,
+            -0.019780611619353294,
+            0.0038526610005646944,
+            0.01460755430161953,
+            0.00634151604026556,
+            0.037622228264808655,
+            0.0022898914758116007,
+            -0.06980370730161667,
+            0.013182250782847404,
+            0.024454450234770775,
+            0.023137671872973442,
+            -0.023600714281201363,
+            -0.00623299041762948,
+            -0.02228393591940403,
+            -0.022529928013682365,
+            -0.02636450156569481,
+            0.0061859628185629845,
+            0.004109505098313093,
+            0.005440753418952227,
+            -0.013884049840271473,
+            0.009311501868069172,
+            0.0421079583466053,
+            0.005227319430559874,
+            0.0153382932767272,
+            0.0037296651862561703,
+            -0.011366254650056362,
+            -0.020402824506163597,
+            -0.003644653595983982,
+            0.02924404852092266,
+            0.011460309848189354,
+            -0.03681190311908722,
+            -0.00158537901006639,
+            0.04427846893668175,
+            0.03530701622366905,
+            0.02198006398975849,
+            -0.011764181777834892,
+            0.009043805301189423,
+            0.020706696435809135,
+            -0.06013768911361694,
+            -0.00040742341661825776,
+            0.010042241774499416,
+            0.0057844179682433605,
+            -0.02148808166384697,
+            -0.020373884588479996,
+            -0.008754403330385685,
+            0.01953461952507496,
+            -0.013789994642138481,
+            0.008276890963315964,
+            6.890248914714903e-05,
+            0.033049680292606354,
+            0.011236024089157581,
+            -0.012531096115708351,
+            -0.017103644087910652,
+            0.05825657770037651,
+            -0.02539500594139099,
+            0.02878100611269474,
+            0.026842014864087105,
+            -0.023658594116568565,
+            0.04734613001346588,
+            -0.005386490374803543,
+            -0.04057412967085838,
+            0.015801336616277695,
+            0.009022099897265434,
+            -0.02171960286796093,
+            -0.025293715298175812,
+            0.0006579368491657078,
+            -0.007430390454828739,
+            0.002532265381887555,
+            -0.04642004519701004,
+            -0.010266527533531189,
+            -0.0025811020750552416,
+            0.017624568194150925,
+            -0.015642166137695312,
+            0.006019556894898415,
+            0.02600274793803692,
+            -0.00706140324473381,
+            0.041037172079086304,
+            -0.041934315115213394,
+            -0.008045368827879429,
+            0.00811048410832882,
+            -0.03750646859407425,
+            -0.03180525451898575,
+            -0.0013927459949627519,
+            -0.0012588977115228772,
+            -0.018058670684695244,
+            -0.018709823489189148,
+            0.00033439468825235963,
+            -0.008305830880999565,
+            -0.006869674660265446,
+            -0.04254205897450447,
+            0.011662891134619713,
+            -0.013522298075258732,
+            -0.006417484022676945,
+            0.003910541534423828,
+            -0.010592104867100716,
+            -0.019114986062049866,
+            0.024266337975859642,
+            0.03484397381544113,
+            0.024266337975859642,
+            0.023861175402998924,
+            -0.0021126328501850367,
+            -0.021271029487252235,
+            -0.01455690898001194,
+            0.02685648389160633,
+            0.001517550554126501,
+            0.007206104230135679,
+            0.024845141917467117,
+            -0.004927065223455429,
+            0.0015410644700750709,
+            -0.011293903924524784,
+            0.0363488607108593,
+            -0.008703758008778095,
+            0.00639216136187315,
+            0.04485727474093437,
+            0.02795621007680893,
+            0.003259387332946062,
+            -0.006649005692452192,
+            -0.0039539518766105175,
+            0.005277964752167463,
+            -0.03394682705402374,
+            0.01840595155954361,
+            0.019389918074011803,
+            0.0038237208500504494,
+            -0.0056071593426167965,
+            0.006431954447180033,
+            -0.024179518222808838,
+            0.011496485210955143,
+            0.01183653250336647,
+            0.00784278754144907,
+            -0.03660932183265686,
+            0.012545567005872726,
+            0.019259687513113022,
+            0.0014054073253646493,
+            -0.015974977985024452,
+            -0.0116701265797019,
+            -0.014238567091524601,
+            0.024222927168011665,
+            0.001188355963677168,
+            -0.003324502846226096,
+            0.0009622608777135611,
+            0.030994929373264313,
+            0.012777088209986687,
+            0.016727421432733536,
+            -0.022732509300112724,
+            0.02675519324839115,
+            -0.0074159204959869385,
+            0.0068009416572749615,
+            -0.0010110974544659257,
+            -0.014462853781878948,
+            -0.013363126665353775,
+            0.03565429896116257,
+            -0.0019968722481280565,
+            0.04138445109128952,
+            -0.005755477584898472,
+            0.013384832069277763,
+            0.03770904988050461,
+            -0.0027728306595236063,
+            -0.02600274793803692,
+            -0.007712556980550289,
+            0.0007709843921475112,
+            -0.015309353359043598,
+            -0.016293318942189217,
+            -0.027507638558745384,
+            -0.013269071467220783,
+            -0.009427262470126152,
+            -0.017885029315948486,
+            -0.012342985719442368,
+            0.025062192231416702,
+            -0.0037658405490219593,
+            -0.0182033721357584,
+            -0.03727494925260544,
+            0.028940176591277122,
+            0.0004478944465517998,
+            0.03692766651511192,
+            -0.008240715600550175,
+            0.012994139455258846,
+            -0.008616937324404716,
+            0.013920225203037262,
+            0.016307789832353592,
+            0.013023079372942448,
+            0.022370757535099983,
+            0.0005584549508057535,
+            0.00988307036459446,
+            0.03565429896116257,
+            -0.031486913561820984,
+            -0.03348378464579582,
+            0.014976541511714458,
+            -0.01522253267467022,
+            0.022095825523138046,
+            0.0029211491346359253,
+            -0.0050283558666706085,
+            0.021401260048151016,
+            0.006457277107983828,
+            -0.034207288175821304,
+            0.01966485008597374,
+            -0.012444276362657547,
+            -0.03319438174366951,
+            0.03435198962688446,
+            -0.014802900142967701,
+            0.004131210036575794,
+            -0.012480450794100761,
+            -0.0025485444348305464,
+            -0.024353159591555595,
+            0.026711782440543175,
+            0.019809551537036896,
+            -0.025915928184986115,
+            0.02523583360016346,
+            0.030358245596289635,
+            0.013348656706511974,
+            -0.004142062738537788,
+            0.008450531400740147,
+            -0.0048510972410440445,
+            -0.022443106397986412,
+            -0.02549629658460617,
+            0.002917531644925475,
+            0.011641185730695724,
+            -0.005719302222132683,
+            0.021459141746163368,
+            -0.049661342054605484,
+            -0.007481035776436329,
+            0.0005797079065814614,
+            -0.016438020393252373,
+            -0.004764276556670666,
+            0.00960090383887291,
+            0.013761053793132305,
+            0.0030622323974967003,
+            -0.03154479339718819,
+            0.0318920761346817,
+            0.029258519411087036,
+            0.020070012658834457,
+            -0.008732697926461697,
+            0.017566686496138573,
+            -0.006374074146151543,
+            0.012096993625164032,
+            0.031949955970048904,
+            0.01827572099864483,
+            0.008038134314119816,
+            0.0005860385717824101,
+            -0.027898330241441727,
+            0.02015683241188526,
+            0.011330079287290573,
+            0.035827938467264175,
+            0.02128550037741661,
+            0.021864304319024086,
+            0.02225499600172043,
+            -0.022023474797606468,
+            0.006493452005088329,
+            -0.006558567751199007,
+            -0.016235439106822014,
+            0.01658272184431553,
+            -0.010628280229866505,
+            0.03504655510187149,
+            0.012147638946771622,
+            -0.05186079815030098,
+            -0.012314044870436192,
+            0.019809551537036896,
+            0.022356286644935608,
+            0.04372860863804817,
+            0.017335165292024612,
+            0.0025666318833827972,
+            0.04838797450065613,
+            -0.02642238140106201,
+            -0.0306765865534544,
+            0.005871238186955452,
+            0.02025812305510044,
+            -0.024859612807631493,
+            0.009760074317455292,
+            0.02464256063103676,
+            -0.018217841163277626,
+            -0.016365669667720795,
+            0.015613225288689137,
+            0.0020999715197831392,
+            0.0030459535773843527,
+            -0.014137276448309422,
+            -0.008096014149487019,
+            0.02487408183515072,
+            0.006301723420619965,
+            0.036985546350479126,
+            -0.01771138794720173,
+            0.019143927842378616,
+            0.00183589244261384,
+            -0.01473778486251831,
+            0.0284481942653656,
+            0.0011639377335086465,
+            -0.03362848609685898,
+            0.025742286816239357,
+            0.012965199537575245,
+            0.010512519627809525,
+            0.009912010282278061,
+            -0.0009658784256316721,
+            0.021907713264226913,
+            0.04233947768807411,
+            0.010592104867100716,
+            0.0004639472172129899,
+            0.03319438174366951,
+            0.024946432560682297,
+            0.003083937568590045,
+            0.01415898185223341,
+            -0.006649005692452192,
+            -0.013052019290626049,
+            0.011532660573720932,
+            -0.02061987668275833,
+            0.0278838612139225,
+            -0.023846706375479698,
+            -0.012393631041049957,
+            -0.004572547972202301,
+            -0.0021144417114555836,
+            -0.0011386150727048516,
+            -0.012096993625164032,
+            -0.008074309676885605,
+            -0.0187677051872015,
+            0.020402824506163597,
+            -0.007965783588588238,
+            0.037419646978378296,
+            -0.007188016548752785,
+            -0.0012118698796257377,
+            0.015048892237246037,
+            -0.026610491797327995,
+            0.002241055015474558,
+            -0.0010174281196668744,
+            0.021734071895480156,
+            -0.01571451686322689,
+            -0.005599924363195896,
+            0.012929024174809456,
+            0.03912711888551712,
+            -0.009260856546461582,
+            0.0003251247981097549,
+            0.029504509642720222,
+            0.0017798208864405751,
+            -0.03817209228873253,
+            0.028838885948061943,
+            -0.046506863087415695,
+            0.008667582646012306,
+            0.009116156026721,
+            -0.022168176248669624,
+            -0.019737200811505318,
+            -0.025640996173024178,
+            -0.004406142048537731,
+            0.0033009888138622046,
+            -0.01203911378979683,
+            0.020807987079024315,
+            -0.00955025851726532,
+            -0.025423945859074593,
+            0.02998202294111252,
+            0.014665434136986732,
+            0.009470673277974129,
+            -0.02675519324839115,
+            0.014267507009208202,
+            -0.036754023283720016,
+            -0.024193987250328064,
+            -0.0038273383397608995,
+            -0.012531096115708351,
+            0.0076980870217084885,
+            -0.006909467279911041,
+            -0.027666809037327766,
+            -0.014021515846252441,
+            0.00466660363599658,
+            -0.01584474742412567,
+            0.039705920964479446,
+            0.035625357180833817,
+            0.04349708557128906,
+            0.014035985805094242,
+            -0.035856880247592926,
+            0.02626321092247963,
+            -0.011170907877385616,
+            0.02059093490242958,
+            -0.009159565903246403,
+            0.0009902967140078545,
+            0.000590108276810497,
+            0.03637780249118805,
+            -0.02536606416106224,
+            0.010360583662986755,
+            -0.04254205897450447,
+            -0.008023664355278015,
+            -0.027377407997846603,
+            0.0032756661530584097,
+            -0.006352368742227554,
+            -0.001016523689031601,
+            0.009991596452891827,
+            -0.004362731706351042,
+            0.021502550691366196,
+            0.04236841946840286,
+            0.0016324068419635296,
+            0.02977944165468216,
+            -0.03883771598339081,
+            -0.020214714109897614,
+            0.013312481343746185,
+            -0.008595232851803303,
+            -0.013146075420081615,
+            -0.019114986062049866,
+            -0.012538331560790539,
+            -0.021213149651885033,
+            0.00700352294370532,
+            0.013710408471524715,
+            -0.01200293842703104,
+            0.001774394535459578,
+            0.016698481515049934,
+            -0.017291756346821785,
+            -0.0018847290193662047,
+            -0.03371530771255493,
+            -0.011633951216936111,
+            0.01899922639131546,
+            -0.026350030675530434,
+            0.029504509642720222,
+            -0.0045580780133605,
+            0.027970680966973305,
+            -0.02420845814049244,
+            -0.010751275345683098,
+            -0.0021415730006992817,
+            -0.02805750072002411,
+            -0.0020221949089318514,
+            0.016003917902708054,
+            0.03889559581875801,
+            -0.01761009730398655,
+            0.04714354872703552,
+            0.0003796137170866132,
+            0.013565707951784134,
+            -0.024555740877985954,
+            0.005249024368822575,
+            -0.052439600229263306,
+            0.008942514657974243,
+            -0.008660348132252693,
+            -0.033570606261491776,
+            -0.01581580750644207,
+            -0.004829391837120056,
+            0.013189485296607018,
+            -0.022370757535099983,
+            -0.00500665046274662,
+            0.01044740341603756,
+            -0.012560036964714527,
+            0.00022609512961935252,
+            -0.023021910339593887,
+            -0.020185774192214012,
+            -0.007065020501613617,
+            0.012639622204005718,
+            -0.03440986946225166,
+            0.016669541597366333,
+            -0.0352780744433403,
+            -0.0049704755656421185,
+            -0.027464227750897408,
+            -0.021271029487252235,
+            0.02911381796002388,
+            0.027232706546783447,
+            -0.04508879408240318,
+            0.029895203188061714,
+            0.018926875665783882,
+            0.011576070450246334,
+            -0.03976380452513695,
+            0.03840361535549164,
+            0.00019421572505962104,
+            0.010273762978613377,
+            0.003168949391692877,
+            -0.008645878173410892,
+            0.02228393591940403,
+            -0.03047400526702404,
+            -0.013052019290626049,
+            0.00472448393702507,
+            -0.006377691403031349,
+            0.012654092162847519,
+            -0.0010617427760735154,
+            -0.023586243391036987,
+            0.016307789832353592,
+            0.011482015252113342,
+            0.012871143408119678,
+            0.03339696303009987,
+            -0.025583116337656975,
+            0.0352780744433403,
+            0.0045182849280536175,
+            -0.018550653010606766,
+            0.006793706677854061,
+            -0.013507827185094357,
+            0.0034764388110488653,
+            0.029157228767871857,
+            0.03930075839161873,
+            0.008587997406721115,
+            -0.036956604570150375,
+            0.02397693693637848,
+            0.022226056084036827,
+            -0.0035867730621248484,
+            0.02867971546947956,
+            0.011474779807031155,
+            0.024613620713353157,
+            0.03545171767473221,
+            -0.0023441542871296406,
+            0.003197889542207122,
+            0.026017218828201294,
+            -0.023094261065125465,
+            0.007596796378493309,
+            -0.01246598083525896,
+            0.03186313435435295,
+            0.011149203404784203,
+            -0.021401260048151016,
+            -0.004189090803265572,
+            -0.010577634908258915,
+            0.009304266422986984,
+            0.007122900802642107,
+            -0.014101101085543633,
+            -0.02857842482626438,
+            -0.014933131635189056,
+            -0.004286763723939657,
+            0.010360583662986755,
+            -0.06262654066085815,
+            -0.01440497301518917,
+            -0.018883464857935905,
+            -0.010816390626132488,
+            0.0036555060651153326,
+            0.01889793574810028,
+            -0.009123390540480614,
+            -0.0029772205743938684,
+            0.0153382932767272,
+            0.05191867798566818,
+            -0.008566292002797127,
+            0.00704693328589201,
+            0.000250739511102438,
+            0.01034611277282238,
+            -0.01640908047556877,
+            0.03507549315690994,
+            -0.013544002547860146,
+            -0.048561617732048035,
+            0.008768873289227486,
+            -0.025293715298175812,
+            0.01863747462630272,
+            0.047230370342731476,
+            -0.023296842351555824,
+            -0.03733282908797264,
+            0.020171303302049637,
+            0.009297031909227371,
+            -0.02934533916413784,
+            -0.00019738104310818017,
+            -0.008226245641708374,
+            -0.008132189512252808,
+            -0.021068448200821877,
+            -0.004785981960594654,
+            0.04636216163635254,
+            -0.008537352085113525,
+            0.025742286816239357,
+            0.03394682705402374,
+            0.017016824334859848,
+            -0.03134221211075783,
+            0.02562652714550495,
+            0.005017503164708614,
+            -0.00505729578435421,
+            0.002235628664493561,
+            0.04031366482377052,
+            -0.02941768988966942,
+            -0.012328515760600567,
+            -0.030358245596289635,
+            -0.016032857820391655,
+            -0.03883771598339081,
+            -0.0005697597516700625,
+            0.0024888552725315094,
+            0.008725463412702084,
+            -0.03160267323255539,
+            -0.032904982566833496,
+            0.031081749126315117,
+            0.03704342618584633,
+            -0.01427474245429039,
+            -0.005390108097344637,
+            -0.010953856632113457,
+            -0.005194761790335178,
+            0.006739443633705378,
+            -0.013175015337765217,
+            -0.001295977272093296,
+            -0.00739421509206295,
+            0.0142240971326828,
+            0.0035343191120773554,
+            0.0071120485663414,
+            -0.05504421517252922,
+            -0.018116550520062447,
+            0.018623003736138344,
+            0.0025792932137846947,
+            0.00119197356980294,
+            -0.004453169647604227,
+            -0.0051332637667655945,
+            0.00782831758260727,
+            -0.026104038581252098,
+            0.02318108081817627,
+            0.022211585193872452,
+            0.005509485956281424,
+            -0.002067413879558444,
+            0.007075873203575611,
+            0.03273133933544159,
+            0.030184604227542877,
+            -0.014079395681619644,
+            -0.015468524768948555,
+            0.03154479339718819,
+            -0.0019100516801699996,
+            0.014614789746701717,
+            -0.0023495806381106377,
+            0.01568557508289814,
+            -1.6335372492903844e-05,
+            -0.01853618398308754,
+            -0.008204540237784386,
+            -0.01622096821665764,
+            0.005824210587888956,
+            0.03131327033042908,
+            -0.00011587375774979591,
+            0.01953461952507496,
+            0.015092302113771439,
+            0.017089175060391426,
+            0.013847874477505684,
+            -0.009528553113341331,
+            -0.018926875665783882,
+            0.01794290915131569,
+            0.0029537067748606205,
+            0.026104038581252098,
+            0.007093960884958506,
+            -0.0039394814521074295,
+            -0.008906339295208454,
+            -0.011409664526581764,
+            -0.005035590846091509,
+            0.009246386587619781,
+            0.02125656045973301,
+            0.03380212560296059,
+            -0.008117719553411007,
+            -0.02716035582125187,
+            -0.01501995138823986,
+            0.021430201828479767,
+            0.02138679102063179,
+            -0.01907157711684704,
+            -0.021459141746163368,
+            0.01740751601755619,
+            -0.014130041003227234,
+            0.015931567177176476,
+            0.0019932547584176064,
+            0.003203315893188119,
+            -0.010903211310505867,
+            0.07107707113027573,
+            -0.02759445831179619,
+            -0.004398907069116831,
+            -0.02798515185713768,
+            -0.003205124521628022,
+            0.006652622949331999,
+            -0.014781195670366287,
+            -0.0013737539993599057,
+            0.018623003736138344,
+            -0.003024248406291008,
+            0.022399697452783585,
+            0.01601838879287243,
+            -0.012017408385872841,
+            -0.00755338603630662,
+            -0.028071971610188484,
+            0.027174826711416245,
+            -0.008660348132252693,
+            0.0037405178882181644,
+            -0.0038309558294713497,
+            0.011359019204974174,
+            0.03713024780154228,
+            0.024989843368530273,
+            0.013363126665353775,
+            0.01740751601755619,
+            -0.010671690106391907,
+            0.013442711904644966,
+            0.012856673449277878,
+            0.014665434136986732,
+            -0.026017218828201294,
+            -0.015092302113771439,
+            0.001866641454398632,
+            0.006428336724638939,
+            0.008146659471094608,
+            0.005661421921104193,
+            -0.001539255608804524,
+            0.012762618251144886,
+            -0.006446424406021833,
+            0.00756062101572752,
+            -0.0009215637692250311,
+            -0.0204896442592144,
+            -0.009854130446910858,
+            0.025091134011745453,
+            0.0003529344976413995,
+            0.024179518222808838,
+            0.0011567026376724243,
+            -0.005542044062167406,
+            0.002711332868784666,
+            -0.02061987668275833,
+            0.00019195476488675922,
+            -0.01714705489575863,
+            -0.0063849263824522495,
+            0.0036790198646485806,
+            -0.01648143120110035,
+            -0.01195229310542345,
+            -0.01170630194246769,
+            0.000591917079873383,
+            0.0003979274188168347,
+            0.0056288642808794975,
+            0.00756062101572752,
+            -0.0036844462156295776,
+            0.009058275260031223,
+            -0.01016523689031601,
+            0.01011459156870842,
+            -0.007850022986531258,
+            -0.012871143408119678,
+            -0.007705322001129389,
+            0.011402430012822151,
+            0.009427262470126152,
+            -0.004492962267249823,
+            0.01160501129925251,
+            -0.016365669667720795,
+            0.003754987847059965,
+            -0.010924916714429855,
+            -0.025915928184986115,
+            -0.006124464794993401,
+            -0.02739187702536583,
+            0.004521902650594711,
+            -0.013131605461239815,
+            0.022269466891884804,
+            -0.024266337975859642,
+            0.027232706546783447,
+            0.007770437281578779,
+            -0.009297031909227371,
+            -0.00021286855917423964,
+            -0.03620416298508644,
+            0.011395194567739964,
+            -0.01846383325755596,
+            0.0028940176125615835,
+            0.012299574911594391,
+            -0.003438454819843173,
+            0.0056071593426167965,
+            -0.008645878173410892,
+            -0.010121827013790607,
+            -0.003353442996740341,
+            -0.021531492471694946,
+            0.004098652396351099,
+            0.006768384017050266,
+            -0.0031906545627862215,
+            -0.01499101147055626,
+            -0.01727728545665741,
+            -0.023528363555669785,
+            0.017812678590416908,
+            0.022891679778695107,
+            -0.021010568365454674,
+            -0.020518586039543152,
+            -0.005314140114933252,
+            0.0030712762381881475,
+            -0.02708800509572029,
+            0.0033064151648432016,
+            -0.008710993453860283,
+            0.01016523689031601,
+            -0.009065510705113411,
+            0.010606574825942516,
+            -0.003870748681947589,
+            0.020070012658834457,
+            0.018377011641860008,
+            0.02351389452815056,
+            -0.03620416298508644,
+            0.0017409324645996094,
+            0.010599339380860329,
+            -0.009246386587619781,
+            -0.014014280401170254,
+            0.021242089569568634,
+            -0.010403993539512157,
+            -0.0020999715197831392,
+            0.03090810962021351,
+            0.000624926935415715,
+            -0.008964220061898232,
+            0.008472236804664135,
+            -0.0017662551254034042,
+            -0.021531492471694946,
+            0.026436852291226387,
+            0.010830861516296864,
+            0.00905104074627161,
+            0.022573338821530342,
+            -0.003820103360339999,
+            0.005397343076765537,
+            -0.017494337633252144,
+            0.020981628447771072,
+            -0.0036772112362086773,
+            -0.009072745218873024,
+            0.019679320976138115,
+            0.026089569553732872,
+            -0.008233480155467987,
+            0.011510955169796944,
+            -0.03249981999397278,
+            0.0012028261553496122,
+            0.03892453759908676,
+            0.01255280151963234,
+            0.0009378425893373787,
+            -0.014281976968050003,
+            -0.012075289152562618,
+            0.013449947349727154,
+            0.021198678761720657,
+            -0.004442316945642233,
+            -0.02875206619501114,
+            0.008009194396436214,
+            -0.011257728561758995,
+            0.03322332352399826,
+            0.018594063818454742,
+            -0.00976730976253748,
+            0.027464227750897408,
+            -0.004619575571268797,
+            -0.01830466091632843,
+            0.01681424304842949,
+            -0.005379255395382643,
+            -0.004243353381752968,
+            0.011843767017126083,
+            0.0036482710856944323,
+            0.03044506534934044,
+            0.006837116554379463,
+            -0.0055709839798510075,
+            -0.0014605745673179626,
+            0.01661166176199913,
+            -0.002420122269541025,
+            -0.024165047332644463,
+            0.0048510972410440445,
+            0.010288232937455177,
+            -0.0007777672726660967,
+            -0.012588976882398129,
+            0.011966763064265251,
+            -0.014694374985992908,
+            -0.0030423360876739025,
+            0.0008921714033931494,
+            0.006099142134189606,
+            -0.00047615633229725063,
+            -0.0233113132417202,
+            -0.006576654966920614,
+            -0.03449669107794762,
+            0.028708655387163162,
+            0.009441732428967953,
+            0.007538916077464819,
+            0.013218426145613194,
+            0.005419048015028238,
+            0.008624172769486904,
+            -0.0034185582771897316,
+            -0.007850022986531258,
+            0.004485727287828922,
+            -0.009919245727360249,
+            -0.02831796370446682,
+            -0.03163161501288414,
+            0.016394609585404396,
+            -0.021299969404935837,
+            -0.0066236830316483974,
+            0.0028596511110663414,
+            -0.02433868870139122,
+            -0.02205241471529007,
+            0.003123730421066284,
+            0.01661166176199913,
+            -0.022139234468340874,
+            0.01664060167968273,
+            -0.001658633933402598,
+            -0.007350804749876261,
+            0.023730944842100143,
+            -0.008023664355278015,
+            -0.001483184052631259,
+            -0.0116701265797019,
+            -0.02669731341302395,
+            0.013666998594999313,
+            0.01283496804535389,
+            -0.0002509655896574259,
+            0.009608138352632523,
+            0.0012118698796257377,
+            0.010707865469157696,
+            0.009116156026721,
+            -0.01830466091632843,
+            0.01273367740213871,
+            -0.02716035582125187,
+            0.03345484286546707,
+            0.01910051703453064,
+            -0.028129851445555687,
+            0.0352780744433403,
+            -0.008710993453860283,
+            0.015859216451644897,
+            -0.0035288927610963583,
+            0.01400704588741064,
+            0.010845331475138664,
+            0.017393046990036964,
+            0.01601838879287243,
+            0.01399257592856884,
+            0.0041529154404997826,
+            0.01920180767774582,
+            -0.006088289897888899,
+            0.023065321147441864,
+            0.013949165120720863,
+            -3.0748939025215805e-05,
+            0.013580177910625935,
+            0.003338972805067897,
+            -0.018449362367391586,
+            -0.0015220724744722247,
+            -0.021068448200821877,
+            -0.007654676679521799,
+            0.0030622323974967003,
+            -0.0204317644238472,
+            -0.004561695270240307,
+            -0.030821288004517555,
+            0.006598360370844603,
+            0.0016595382476225495,
+            -0.012458746321499348,
+            -0.009362147189676762,
+            0.010765746235847473,
+            0.002649834845215082,
+            0.022775918245315552,
+            -0.011908882297575474,
+            0.02241416648030281,
+            0.008935279212892056,
+            -0.002969985594972968,
+            0.01049804873764515,
+            -0.02457020990550518,
+            0.014499028213322163,
+            0.0017535937950015068,
+            0.022472048178315163,
+            0.021632781252264977,
+            0.0005114271771162748,
+            0.010150766931474209,
+            0.0016197455115616322,
+            0.015352764166891575,
+            -0.015164652839303017,
+            -0.012429806403815746,
+            -0.00489088986068964,
+            -0.015584285371005535,
+            0.05814081430435181,
+            -0.02171960286796093,
+            -0.012314044870436192,
+            -0.017682448029518127,
+            -0.002385755768045783,
+            0.009311501868069172,
+            -0.030126724392175674,
+            -0.018651943653821945,
+            -0.022168176248669624,
+            -0.006797323934733868,
+            -0.02038835547864437,
+            -0.004579782951623201,
+            0.021603841334581375,
+            -0.0011901648249477148,
+            0.007850022986531258,
+            -0.01953461952507496,
+            -0.0068009416572749615,
+            -0.01409386657178402,
+            -0.003257578704506159,
+            -0.014535203576087952,
+            -0.012024643830955029,
+            -0.0042144134640693665,
+            0.021545961499214172,
+            0.01426027249544859,
+            -0.010859801433980465,
+            0.006486217025667429,
+            0.013580177910625935,
+            -0.012010172940790653,
+            0.003698916407302022,
+            0.01714705489575863,
+            0.011691831052303314,
+            -0.00944896787405014,
+            0.01007841620594263,
+            -0.0026842013467103243,
+            -0.005419048015028238,
+            0.004963240120559931,
+            0.0023749032989144325,
+            0.03244193643331528,
+            0.010512519627809525,
+            -0.04520455747842789,
+            -0.00811048410832882,
+            0.010968326590955257,
+            0.006844351999461651,
+            0.0019697407260537148,
+            0.005400960333645344,
+            -0.015309353359043598,
+            0.004250588361173868,
+            -0.03093704953789711,
+            -0.0001339613663731143,
+            -0.0026263210456818342,
+            -0.008645878173410892,
+            0.006634535733610392,
+            -0.006591125391423702,
+            0.01177141722291708,
+            0.01923074759542942,
+            -0.003143626730889082,
+            0.01457137893885374,
+            0.0030712762381881475,
+            0.013218426145613194,
+            -0.03281816095113754,
+            0.008739933371543884,
+            -0.011380724608898163,
+            0.009029335342347622,
+            -0.012328515760600567,
+            -0.035856880247592926,
+            -0.010867035947740078,
+            0.015728985890746117,
+            -0.002018577419221401,
+            -0.005976146552711725,
+            -0.0021922183223068714,
+            0.02626321092247963,
+            0.019780611619353294,
+            -0.0014280168106779456,
+            -0.0065621850080788136,
+            0.030589766800403595,
+            -0.0006945642526261508,
+            0.0002321997017133981,
+            0.0003368817560840398,
+            0.004934300202876329,
+            -0.011829297058284283,
+            0.005697597283869982,
+            -0.01740751601755619,
+            0.0067177386954426765,
+            -0.008052604272961617,
+            0.026899894699454308,
+            -0.0019064341904595494,
+            -0.014614789746701717,
+            0.0193609781563282,
+            -0.032876040786504745,
+            -0.004398907069116831,
+            -0.006478982046246529,
+            -0.00466660363599658,
+            0.021806422621011734,
+            -0.0019480356713756919,
+            0.025076663121581078,
+            -0.05145563557744026,
+            -0.0014379650820046663,
+            -0.004051624797284603,
+            -0.016365669667720795,
+            0.004283146001398563,
+            -0.02135785110294819,
+            0.014130041003227234,
+            -0.0030459535773843527,
+            -0.015757925808429718,
+            0.002360433107241988,
+            -0.008291360922157764,
+            -0.008595232851803303,
+            0.013175015337765217,
+            0.01578686572611332,
+            -0.003349825507029891,
+            0.014253037050366402,
+            0.0039539518766105175,
+            0.015265943482518196,
+            0.017494337633252144,
+            -0.013066490180790424,
+            -0.013558472506701946,
+            0.01178588718175888,
+            0.0006185962702147663,
+            0.01697341352701187,
+            0.008320300839841366,
+            -0.009991596452891827,
+            -0.02749316766858101,
+            0.005447988398373127,
+            -0.006714120972901583,
+            -0.0036301834043115377,
+            -0.018492773175239563,
+            -0.016626130789518356,
+            -0.001278794021345675,
+            -0.01668401248753071,
+            0.00461595831438899,
+            -0.023036381229758263,
+            -0.0010617427760735154,
+            0.008812284097075462,
+            -0.02338366210460663,
+            -0.004847479518502951,
+            0.013580177910625935,
+            0.016235439106822014,
+            -0.03247087821364403,
+            0.003870748681947589,
+            -0.0037369003985077143,
+            -0.028129851445555687,
+            -0.0035126139409840107,
+            0.0025720582343637943,
+            -0.013732113875448704,
+            0.01500548142939806,
+            -0.011344549246132374,
+            -0.017103644087910652,
+            -0.03397576883435249,
+            -0.022298406809568405,
+            -0.007010757923126221,
+            0.019158396869897842,
+            0.005462458357214928,
+            0.004178238101303577,
+            0.0027185678482055664,
+            0.0003034196561202407,
+            0.0076691466383636,
+            0.0284481942653656,
+            -0.013724878430366516,
+            -6.6804045673052315e-06,
+            -0.001633311272598803,
+            0.02061987668275833,
+            -0.004373584408313036,
+            0.004113122820854187,
+            0.004605105612426996,
+            0.000946886430028826,
+            0.0033425905276089907,
+            -0.005317757371813059,
+            -0.006938407197594643,
+            -0.010874271392822266,
+            0.010483578778803349,
+            0.024888552725315094,
+            -0.01910051703453064,
+            -0.0012679415522143245,
+            -0.02028706483542919,
+            0.029461100697517395,
+            0.030126724392175674,
+            0.00784278754144907,
+            -0.0409214086830616,
+            -0.0032340646721422672,
+            -0.0030170134268701077,
+            -0.0014325387310236692,
+            -0.02420845814049244,
+            0.01203911378979683,
+            0.006196815520524979,
+            0.009702194482088089,
+            0.018753234297037125,
+            -0.006583890412002802,
+            0.017002353444695473,
+            -0.008573527447879314,
+            -0.005968911573290825,
+            0.001627884921617806,
+            -0.007878962904214859,
+            0.015077832154929638,
+            0.04115293174982071,
+            -0.028838885948061943,
+            0.00650068698450923,
+            -0.007857258431613445,
+            0.022674627602100372,
+            -0.009543023072183132,
+            0.009195741266012192,
+            0.02061987668275833,
+            0.017161523923277855,
+            -0.0051151760853827,
+            0.016958944499492645,
+            0.07582326233386993,
+            -0.003993744496256113,
+            0.022009003907442093,
+            0.008768873289227486,
+            -0.000894884520675987,
+            0.0041167400777339935,
+            -0.019187336787581444,
+            -0.02305085025727749,
+            0.017233874648809433,
+            0.0039756568148732185,
+            -0.004492962267249823,
+            0.011496485210955143,
+            -0.015164652839303017,
+            0.006319811102002859,
+            0.017827149480581284,
+            0.005386490374803543,
+            -0.012639622204005718,
+            -0.006927554961293936,
+            0.022906150668859482,
+            0.019158396869897842,
+            0.0056288642808794975,
+            -0.010823626071214676,
+            0.01454243902117014,
+            -0.005646951962262392,
+            -0.01661166176199913,
+            0.016192028298974037,
+            -0.01691553369164467,
+            0.018782174214720726,
+            -0.0024978991132229567,
+            0.024787262082099915,
+            -0.007640206720679998,
+            -0.010172472335398197,
+            -0.0016088929260149598,
+            0.00032648135675117373,
+            0.01684318296611309,
+            0.019635910168290138,
+            0.0036573149263858795,
+            6.426754407584667e-05,
+            0.002027621027082205,
+            0.012364690192043781,
+            0.0006294488557614386,
+            0.0032431085128337145,
+            0.00500665046274662,
+            0.003143626730889082,
+            -0.012965199537575245,
+            0.023962466046214104,
+            0.006048496812582016,
+            0.008132189512252808,
+            -0.00616064015775919,
+            -0.005618012044578791,
+            0.004894507583230734,
+            -0.009904775768518448,
+            0.00650068698450923,
+            0.001803334802389145,
+            -0.003291945206001401,
+            -0.01261791680008173,
+            0.029721561819314957,
+            -0.021198678761720657,
+            0.014281976968050003,
+            0.01923074759542942,
+            0.008219010196626186,
+            0.00978177972137928,
+            -0.013189485296607018,
+            -0.018362542614340782,
+            0.00761850131675601,
+            -0.00027221854543313384,
+            0.005864003207534552,
+            0.0051115588285028934,
+            0.001834083697758615,
+            0.016597190871834755,
+            -0.00244182744063437,
+            0.005495015997439623,
+            0.013225660659372807,
+            0.007864492945373058,
+            0.005976146552711725,
+            0.016134148463606834,
+            -0.030329305678606033,
+            -0.018753234297037125,
+            0.004315703641623259,
+            0.00516582140699029,
+            -0.0005634290864691138,
+            0.0043554967269301414,
+            0.01006394624710083,
+            0.011279433965682983,
+            -0.008399886079132557,
+            -0.015497464686632156,
+            -0.014817370101809502,
+            0.04184749722480774,
+            0.014347092248499393,
+            -0.020750107243657112,
+            -0.011959527619183064,
+            -0.0021650870330631733,
+            0.003226829692721367,
+            -0.007198868785053492,
+            0.014933131635189056,
+            0.014962071552872658,
+            -0.018955815583467484,
+            -0.02590145915746689,
+            -0.027102475985884666,
+            -0.012704737484455109,
+            0.014310917817056179,
+            0.013688703998923302,
+            0.010288232937455177,
+            0.005809740629047155,
+            0.02112632989883423,
+            -0.014846310950815678,
+            0.008428826928138733,
+            0.012089759111404419,
+            0.0034782474394887686,
+            -0.009427262470126152,
+            -0.00600146921351552,
+            -0.014477323740720749,
+            0.015873687341809273,
+            0.008276890963315964,
+            -0.005802505649626255,
+            -0.01611967757344246,
+            0.0034438809379935265,
+            -0.014918660745024681,
+            -0.010411228984594345,
+            -0.006714120972901583,
+            -0.0021994535345584154,
+            -0.012082523666322231,
+            0.004927065223455429,
+            -0.009745604358613491,
+            -0.007090343162417412,
+            0.0033896181266754866,
+            -0.004171003121882677,
+            -0.005140498746186495,
+            -0.02685648389160633,
+            -0.026914363726973534,
+            -0.003196080680936575,
+            -0.0024038434494286776,
+            -0.00600146921351552,
+            -0.003590390784665942,
+            -0.029359810054302216,
+            -0.0065404800698161125,
+            0.009022099897265434,
+            -0.011851002462208271,
+            0.021068448200821877,
+            0.0029573242645710707,
+            -0.008906339295208454,
+            0.0033896181266754866,
+            -0.003604860743507743,
+            -0.029866263270378113,
+            0.009977125562727451,
+            0.005639716982841492,
+            0.02208135463297367,
+            -0.0182033721357584,
+            0.0015962315956130624,
+            0.0061497874557971954,
+            0.023759884759783745,
+            -0.031718432903289795,
+            -0.015902627259492874,
+            -0.007654676679521799,
+            -0.010975562036037445,
+            -0.006674328353255987,
+            -0.014354327693581581,
+            0.005060913506895304,
+            -0.008117719553411007,
+            -0.025423945859074593,
+            -0.02281932905316353,
+            -0.00650068698450923,
+            -0.008038134314119816,
+            0.022124765440821648,
+            1.9359395082574338e-05,
+            0.03684084489941597,
+            0.008660348132252693,
+            0.003986509516835213,
+            0.010852565988898277,
+            -0.0030133959371596575,
+            -0.007546151056885719,
+            0.011851002462208271,
+            -0.017581157386302948,
+            -0.006873291917145252,
+            0.003925011493265629,
+            0.01943332888185978,
+            -0.007654676679521799,
+            -0.002360433107241988,
+            0.011040677316486835,
+            0.016857653856277466,
+            0.008776108734309673,
+            0.0027438905090093613,
+            -0.007289307191967964,
+            -0.005610776599496603,
+            0.02215370535850525,
+            0.027073536068201065,
+            0.004728101193904877,
+            0.003436645958572626,
+            0.007806612644344568,
+            -0.002843372290953994,
+            -0.006674328353255987,
+            0.001284220372326672,
+            -0.006337898783385754,
+            -0.02085139788687229,
+            -0.0038635137025266886,
+            -0.004225265700370073,
+            -0.0025720582343637943,
+            -0.02205241471529007,
+            -0.01661166176199913,
+            -0.014875250868499279,
+            0.008956984616816044,
+            0.014470088295638561,
+            -0.019578030332922935,
+            0.01211869902908802,
+            0.006062967237085104,
+            0.006569419987499714,
+            -0.006435571704059839,
+            0.01753774657845497,
+            0.042223718017339706,
+            0.00023073007469065487,
+            -0.03906923905014992,
+            -0.01707470417022705,
+            0.0009111633989959955,
+            -0.012494921684265137,
+            -0.022935090586543083,
+            -0.00756062101572752,
+            0.0031617144122719765,
+            -0.024411039426922798,
+            0.023398132994771004,
+            -0.001627884921617806,
+            -0.0009658784256316721,
+            0.004330174066126347,
+            0.017060235142707825,
+            -0.002376711927354336,
+            0.005332227796316147,
+            -0.014535203576087952,
+            0.003442072309553623,
+            -0.01417345181107521,
+            0.010548694059252739,
+            -0.016857653856277466,
+            -0.018984755501151085,
+            0.006804558914154768,
+            -0.002042091218754649,
+            0.008920809254050255,
+            0.022341815754771233,
+            0.02652367204427719,
+            0.01028099749237299,
+            -0.027869390323758125,
+            -0.01523700263351202,
+            -0.011807592585682869,
+            0.0026950540486723185,
+            -0.014021515846252441,
+            -0.007321864832192659,
+            -0.00761850131675601,
+            0.003545171581208706,
+            0.015323823317885399,
+            0.015251473523676395,
+            0.005151351448148489,
+            0.030647646635770798,
+            -3.1540272175334394e-05,
+            0.006804558914154768,
+            -0.003615713445469737,
+            -0.02801409177482128,
+            -0.005939971189945936,
+            0.005708449985831976,
+            0.009991596452891827,
+            -0.013254600577056408,
+            0.007936843670904636,
+            0.014209627173841,
+            -0.016872122883796692,
+            -0.011279433965682983,
+            -0.02716035582125187,
+            0.010440168902277946,
+            -0.002913914155215025,
+            0.03093704953789711,
+            0.020070012658834457,
+            -0.02138679102063179,
+            -0.015439583919942379,
+            -0.02112632989883423,
+            0.018984755501151085,
+            -0.010953856632113457,
+            -0.0019154779147356749,
+            0.007973019033670425,
+            0.019983192905783653,
+            -0.007712556980550289,
+            -0.009094450622797012,
+            -0.0077921426855027676,
+            0.013876814395189285,
+            0.018753234297037125,
+            7.57983943913132e-05,
+            0.0010300894500687718,
+            -0.0096515491604805,
+            0.010780216194689274,
+            -0.009977125562727451,
+            -0.010592104867100716,
+            0.0051115588285028934,
+            -0.004529137630015612,
+            0.019968722015619278,
+            -0.01638014055788517,
+            0.022529928013682365,
+            0.010360583662986755,
+            -0.009202975779771805,
+            -0.0398506224155426,
+            -0.006522392388433218,
+            -0.006005086470395327,
+            -0.001984210917726159,
+            0.011995702981948853,
+            -0.0027131414972245693,
+            -0.01771138794720173,
+            0.008971454575657845,
+            0.000733000400941819,
+            -0.003031483618542552,
+            0.007604031357914209,
+            0.008096014149487019,
+            0.0007470183190889657,
+            0.011684596538543701,
+            -0.005046443548053503,
+            -0.009702194482088089,
+            0.02195112407207489,
+            0.0014090248150750995,
+            -7.969853322720155e-05,
+            0.02002660185098648,
+            0.02251545712351799,
+            0.010982797481119633,
+            -0.00988307036459446,
+            0.0034547336399555206,
+            -0.009825189597904682,
+            -0.007538916077464819,
+            -0.030850227922201157,
+            0.027579989284276962,
+            -0.007929608225822449,
+            0.03258663788437843,
+            0.013232896104454994,
+            -0.004037154838442802,
+            -0.00075018365168944,
+            0.0074593303725123405,
+            -0.005458840634673834,
+            -0.0006244747783057392,
+            0.017450926825404167,
+            -0.0009233725140802562,
+            0.006261930800974369,
+            -0.0032340646721422672,
+            -0.02028706483542919,
+            0.025915928184986115,
+            -0.0022754215169698,
+            0.005220084451138973,
+            0.011966763064265251,
+            0.009087215177714825,
+            0.007057785522192717,
+            -0.01607626862823963,
+            -0.0028071971610188484,
+            -0.03608839958906174,
+            -0.008016428910195827,
+            -0.0021108242217451334,
+            -0.005697597283869982,
+            -0.003261196194216609,
+            -0.0070831081829965115,
+            -0.04691202566027641,
+            -0.004189090803265572,
+            0.008544587530195713,
+            -0.009868600405752659,
+            0.01853618398308754,
+            -0.016828712075948715,
+            -0.017957380041480064,
+            -0.02115526981651783,
+            0.007195251528173685,
+            0.02002660185098648,
+            0.013587413355708122,
+            -0.003053188556805253,
+            -0.017986319959163666,
+            0.018550653010606766,
+            -0.011800357140600681,
+            -0.008211774751543999,
+            -0.01635120064020157,
+            -0.020055541768670082,
+            0.014549673534929752,
+            0.016105208545923233,
+            -0.011908882297575474,
+            -0.0032431085128337145,
+            -0.007604031357914209,
+            0.013797229155898094,
+            -0.01720493473112583,
+            -0.008327536284923553,
+            0.008096014149487019,
+            0.0018738764338195324,
+            0.012342985719442368,
+            -0.019172867760062218,
+            0.007108430843800306,
+            -0.015801336616277695,
+            0.009043805301189423,
+            -0.01789950020611286,
+            -0.013218426145613194,
+            0.002179556991904974,
+            0.0010237587848678231,
+            0.039705920964479446,
+            0.022920619696378708,
+            -0.027406347915530205,
+            0.01846383325755596,
+            0.004572547972202301,
+            0.016336729750037193,
+            -0.0031038338784128428,
+            0.01467266958206892,
+            -0.01720493473112583,
+            0.016655072569847107,
+            -0.007216956466436386,
+            0.01761009730398655,
+            0.003404088318347931,
+            0.03293392062187195,
+            -0.017566686496138573,
+            0.010432933457195759,
+            0.0015681958757340908,
+            -0.004634045995771885,
+            -0.00705416826531291,
+            0.016785303130745888,
+            0.010411228984594345,
+            0.014520733617246151,
+            0.00616787513718009,
+            -0.023600714281201363,
+            -0.009608138352632523,
+            0.004091417416930199,
+            0.020475175231695175,
+            0.014781195670366287,
+            -0.00349814398214221,
+            -0.0036573149263858795,
+            0.002850607503205538,
+            -0.01028099749237299,
+            0.0096660191193223,
+            -0.013334186747670174,
+            0.007596796378493309,
+            -0.008493942208588123,
+            0.0077921426855027676,
+            -0.008407121524214745,
+            0.0007913329754956067,
+            0.021473610773682594,
+            -0.03510443493723869,
+            0.01510677207261324,
+            -0.003747752867639065,
+            -0.014231331646442413,
+            0.027044596150517464,
+            -0.017393046990036964,
+            -0.005845915526151657,
+            -0.008964220061898232,
+            -0.0008143946761265397,
+            0.014216861687600613,
+            -0.000501479022204876,
+            0.0035704942420125008,
+            -0.006768384017050266,
+            -0.006949259899556637,
+            0.001438869396224618,
+            0.0091016860678792,
+            -0.011851002462208271,
+            0.006677945610135794,
+            -0.004026302136480808,
+            0.021444670855998993,
+            0.008732697926461697,
+            0.006938407197594643,
+            0.00020653790852520615,
+            -0.002971794456243515,
+            0.006062967237085104,
+            0.02182089351117611,
+            0.0048149218782782555,
+            0.023166611790657043,
+            -0.023658594116568565,
+            0.013486122712492943,
+            -0.00016154497279785573,
+            -0.008096014149487019,
+            -0.01694447360932827,
+            -0.0017870558658614755,
+            0.02543841488659382,
+            0.0020475175697356462,
+            -0.02400587685406208,
+            0.00033281202195212245,
+            0.010143532417714596,
+            -0.0014524351572617888,
+            0.01500548142939806,
+            0.01784161850810051,
+            -0.0014343474758788943,
+            0.007282072212547064,
+            -0.002738464158028364,
+            -0.024092696607112885,
+            -0.01807313971221447,
+            -0.003783928230404854,
+            -0.016785303130745888,
+            0.02271803840994835,
+            0.01691553369164467,
+            0.0051694391295313835,
+            -0.0038526610005646944,
+            0.0017517850501462817,
+            0.01522253267467022,
+            0.020041072741150856,
+            0.0056288642808794975,
+            0.00939832255244255,
+            0.0011594158131629229,
+            -0.010309938341379166,
+            0.0016504944069311023,
+            -0.008052604272961617,
+            -0.008797814138233662,
+            -0.012357455678284168,
+            0.009275326505303383,
+            -0.006240225397050381,
+            -0.013449947349727154,
+            -0.01437603309750557,
+            -0.004619575571268797,
+            0.016090737655758858,
+            -0.007546151056885719,
+            -0.026870954781770706,
+            0.0036265659146010876,
+            0.018377011641860008,
+            0.011612245813012123,
+            0.015642166137695312,
+            -0.009087215177714825,
+            -0.011185378767549992,
+            0.007929608225822449,
+            -0.019476739689707756,
+            -0.0029193402733653784,
+            -0.010729570873081684,
+            0.003085746429860592,
+            -0.021545961499214172,
+            0.02937427908182144,
+            0.008472236804664135,
+            0.027739159762859344,
+            0.024729380384087563,
+            -0.01150372065603733,
+            0.004478492308408022,
+            -0.003693490056321025,
+            -0.0002373998868279159,
+            -0.008956984616816044,
+            3.804050720646046e-05,
+            0.0006158831529319286,
+            0.021545961499214172,
+            0.013175015337765217,
+            0.01245151087641716,
+            0.001984210917726159,
+            -0.0009134243591688573,
+            0.002418313641101122,
+            -7.246348832268268e-05,
+            -0.010042241774499416,
+            -0.017856089398264885,
+            0.005277964752167463,
+            0.017002353444695473,
+            -0.004178238101303577,
+            -0.005639716982841492,
+            0.009969891048967838,
+            0.0003594912705011666,
+            -0.0029555156361311674,
+            -0.011612245813012123,
+            -0.011858237907290459,
+            0.018159961327910423,
+            0.026509201154112816,
+            -0.008609702810645103,
+            -0.006547715049237013,
+            0.02115526981651783,
+            -0.001689382828772068,
+            0.03215253725647926,
+            -0.011098558083176613,
+            -0.010621044784784317,
+            -0.0014596701366826892,
+            -0.005212849471718073,
+            -0.01730622537434101,
+            -0.0238756462931633,
+            0.01255280151963234,
+            0.0035397454630583525,
+            -0.0035361277405172586,
+            -0.03047400526702404,
+            0.015034422278404236,
+            0.006135317496955395,
+            0.032876040786504745,
+            -0.001438869396224618,
+            -0.006460894364863634,
+            -0.000922015926335007,
+            -0.0010653602657839656,
+            -0.0013864153297618032,
+            -0.013811699114739895,
+            0.00905104074627161,
+            -0.0032684311736375093,
+            0.0034746299497783184,
+            -0.0020312387496232986,
+            -0.025554176419973373,
+            -0.005693979561328888,
+            0.002465341240167618,
+            -0.00727121951058507,
+            -0.00756062101572752,
+            -0.013131605461239815,
+            -0.0012145830551162362,
+            0.017335165292024612,
+            0.004416994284838438,
+            -0.0010997267672792077,
+            0.008674818091094494,
+            -0.0077559673227369785,
+            -0.008493942208588123,
+            -0.0060665844939649105,
+            -0.01720493473112583,
+            0.01830466091632843,
+            -0.02105397917330265,
+            0.0017563069704920053,
+            0.026350030675530434,
+            -0.004218030720949173,
+            -0.003295562695711851,
+            0.018695354461669922,
+            0.0091016860678792,
+            0.008587997406721115,
+            -0.0006027696072123945,
+            -0.002096354030072689,
+            0.00361028709448874,
+            0.01177141722291708,
+            0.011590540409088135,
+            -0.002790918340906501,
+            -0.00232968432828784,
+            -0.02089480683207512,
+            0.008602467365562916,
+            -0.01889793574810028,
+            0.024657031521201134,
+            0.02061987668275833,
+            -0.0015374468639492989,
+            0.01187270786613226,
+            0.02646579220890999,
+            0.008349240757524967,
+            0.01488972082734108,
+            0.000975826580543071,
+            0.005737389903515577,
+            -0.0025431180838495493,
+            -0.000960452132858336,
+            -0.023253431543707848,
+            0.006135317496955395,
+            0.005831445567309856,
+            0.004308468662202358,
+            -0.0035940082743763924,
+            -0.01976614072918892,
+            0.017421986907720566,
+            0.014151746407151222,
+            -0.018217841163277626,
+            0.021936653181910515,
+            -0.005737389903515577,
+            0.019838491454720497,
+            0.017103644087910652,
+            0.011959527619183064,
+            -0.007311012130230665,
+            -0.014166216365993023,
+            0.013558472506701946,
+            -0.02085139788687229,
+            -0.0024092698004096746,
+            0.005632482003420591,
+            -0.01727728545665741,
+            0.002420122269541025,
+            -0.02025812305510044,
+            -0.004076947458088398,
+            -0.008334770798683167,
+            0.006956494878977537,
+            -0.008327536284923553,
+            0.00628363573923707,
+            0.00583506328985095,
+            -0.0022844651248306036,
+            -0.001457861391827464,
+            -0.03637780249118805,
+            -0.005845915526151657,
+            0.021777482703328133,
+            0.0026028072461485863,
+            -0.04057412967085838,
+            0.021097388118505478,
+            0.004811304621398449,
+            -0.01684318296611309,
+            -0.015280413441359997,
+            -0.021936653181910515,
+            -0.014064925722777843,
+            -0.007625736761838198,
+            0.0037405178882181644,
+            -0.02102503925561905,
+            -0.0029048703145235777,
+            0.011807592585682869,
+            -0.01902816630899906,
+            -0.0102158822119236,
+            -0.008124954998493195,
+            -0.0031056427396833897,
+            0.019086046144366264,
+            -0.011923353187739849,
+            -0.00951408315449953,
+            0.025814637541770935,
+            0.006916702259331942,
+            -0.0058061229065060616,
+            -0.015323823317885399,
+            -0.016134148463606834,
+            -0.0071409884840250015,
+            0.013804464600980282,
+            -0.0025467355735599995,
+            -0.003921393770724535,
+            -0.030155664309859276,
+            -0.0028705038130283356,
+            0.006949259899556637,
+            0.019476739689707756,
+            -0.00043726799776777625,
+            0.002132529392838478,
+            0.009072745218873024,
+            0.00794407818466425,
+            -0.014535203576087952,
+            -7.447008556482615e-06,
+            0.009072745218873024,
+            0.012494921684265137,
+            -0.00015498821449000388,
+            0.019722729921340942,
+            -0.01833360269665718,
+            0.004937917459756136,
+            -0.008718227967619896,
+            0.005220084451138973,
+            -0.01860853284597397,
+            -0.006439189426600933,
+            -0.01627884991466999,
+            0.008052604272961617,
+            0.009275326505303383,
+            -0.018319131806492805,
+            -0.0019172866595909,
+            -0.006866056937724352,
+            0.017856089398264885,
+            -0.027174826711416245,
+            0.001350240083411336,
+            -0.017494337633252144,
+            -0.0015817615203559399,
+            -0.026350030675530434,
+            0.00311287771910429,
+            0.006471747066825628,
+            0.004561695270240307,
+            0.0003723786794580519,
+            -0.007850022986531258,
+            -0.02457020990550518,
+            0.01198846846818924,
+            -0.002181365853175521,
+            -0.001573622110299766,
+            -0.010526989586651325,
+            -0.0017327930545434356,
+            0.011966763064265251,
+            0.016163088381290436,
+            -0.0042361184023320675,
+            -0.014665434136986732,
+            0.01843489333987236,
+            -0.016105208545923233,
+            -0.019549088552594185,
+            -0.015179122798144817,
+            -0.011185378767549992,
+            -0.004644898232072592,
+            0.012639622204005718,
+            0.013804464600980282,
+            0.020779047161340714,
+            0.00961537379771471,
+            0.010765746235847473,
+            -0.007358039729297161,
+            0.00016572773165535182,
+            -0.006638152990490198,
+            -0.0015085067134350538,
+            0.012574506923556328,
+            -0.0054299007169902325,
+            0.02613298036158085,
+            -0.015410644002258778,
+            0.0015274987090379,
+            -0.00025797454873099923,
+            0.0069637298583984375,
+            -0.007354422472417355,
+            0.00015397078823298216,
+            -0.010411228984594345,
+            0.008392651565372944,
+            -0.03455457091331482,
+            -0.006258313078433275,
+            -0.0033263114746659994,
+            -0.011185378767549992,
+            0.03157373145222664,
+            0.0003567781241144985,
+            0.00756062101572752,
+            0.0048149218782782555,
+            0.011395194567739964,
+            -0.008298595435917377,
+            0.016061797738075256,
+            -0.017783738672733307,
+            0.001122336252592504,
+            0.015526404604315758,
+            0.007770437281578779,
+            -0.0012932640966027975,
+            0.006775618996471167,
+            -0.006041261833161116,
+            0.009123390540480614,
+            -0.0014108336763456464,
+            -0.014361563138663769,
+            -0.01450626365840435,
+            0.00017024963744916022,
+            0.013146075420081615,
+            -0.01727728545665741,
+            -0.006670710630714893,
+            -0.006511539686471224,
+            0.01394193060696125,
+            -0.01409386657178402,
+            0.026914363726973534,
+            -0.001209156820550561,
+            -0.017465395852923393,
+            0.01238639559596777,
+            0.030589766800403595,
+            -0.014383267611265182,
+            0.02059093490242958,
+            -0.007546151056885719,
+            -0.003906923811882734,
+            0.004988562781363726,
+            -0.0015482994494959712,
+            -0.0004268675984349102,
+            0.018217841163277626,
+            -0.008204540237784386,
+            -0.007777672726660967,
+            0.011207083240151405,
+            -0.02218264527618885,
+            0.016090737655758858,
+            -0.01584474742412567,
+            0.0010300894500687718,
+            0.004858332220464945,
+            0.0031110690906643867,
+            0.02028706483542919,
+            0.037622228264808655,
+            -0.00014944888243917376,
+            -0.007878962904214859,
+            -0.012234459631145,
+            0.02089480683207512,
+            0.005871238186955452,
+            0.005853150971233845,
+            -0.004388054367154837,
+            0.002447253791615367,
+            -0.01740751601755619,
+            -0.013681468553841114,
+            0.0076691466383636,
+            -0.010606574825942516,
+            -0.03313650190830231,
+            -0.015077832154929638,
+            0.0051115588285028934,
+            0.0063993968069553375,
+            0.018579592928290367,
+            0.006204050499945879,
+            -0.003986509516835213,
+            0.020967157557606697,
+            -0.003581346943974495,
+            0.011496485210955143,
+            -0.008414356037974358,
+            -0.002881356282159686,
+            -0.004876419901847839,
+            -0.023354722186923027,
+            0.024729380384087563,
+            -0.027637869119644165,
+            -0.003433028468862176,
+            -0.011337313801050186,
+            -0.000320150691550225,
+            0.005914648529142141,
+            -0.0031291565392166376,
+            0.01470884494483471,
+            0.007466565817594528,
+            0.016958944499492645,
+            0.01737857609987259,
+            -0.03166055306792259,
+            0.01707470417022705,
+            0.019303098320961,
+            -0.0020149596966803074,
+            -0.007538916077464819,
+            -0.0032449173741042614,
+            -0.012017408385872841,
+            -0.012762618251144886,
+            0.014296446926891804,
+            -0.0004528685531113297,
+            -0.006073819473385811,
+            -0.005654186941683292,
+            0.02287720888853073,
+            -0.007784907706081867,
+            0.02351389452815056,
+            -0.0046955435536801815,
+            0.0027800656389445066,
+            0.02652367204427719,
+            0.007325482089072466,
+            0.020417295396327972,
+            -0.016452491283416748,
+            -0.02271803840994835,
+            -0.007159076165407896,
+            -0.005249024368822575,
+            0.006916702259331942,
+            -0.005715684965252876,
+            0.00944896787405014,
+            -0.021647252142429352,
+            -0.020402824506163597,
+            0.01850724220275879,
+            -0.0199253112077713,
+            0.010736805386841297,
+            0.0061027598567306995,
+            0.01899922639131546,
+            -0.004000979475677013,
+            0.012531096115708351,
+            -0.010382288135588169,
+            -0.01604732871055603,
+            0.011424134485423565,
+            0.0062221381813287735,
+            -0.006652622949331999,
+            0.003928629215806723,
+            -0.011192613281309605,
+            -0.008790578693151474,
+            -0.01638014055788517,
+            -0.0010120017686858773,
+            -0.010961092077195644,
+            0.008161129429936409,
+            0.01787055842578411,
+            0.004959622863680124,
+            0.014745020307600498,
+            0.0032412998843938112,
+            -0.01250215619802475,
+            -0.010273762978613377,
+            0.0035940082743763924,
+            -0.010932152159512043,
+            -0.003747752867639065,
+            -0.022920619696378708,
+            0.0007212434429675341,
+            -0.014361563138663769,
+            -0.01175694726407528,
+            -0.009289796464145184,
+            0.0048510972410440445,
+            0.015497464686632156,
+            -0.018825585022568703,
+            -0.008457766845822334,
+            -0.008703758008778095,
+            0.0023170229978859425,
+            0.007057785522192717,
+            -0.016322258859872818,
+            0.008682052604854107,
+            0.02931639924645424,
+            -0.0034402634482830763,
+            -0.014795665629208088,
+            0.004442316945642233,
+            0.005614394322037697,
+            -0.008378181606531143,
+            0.018058670684695244,
+            0.011395194567739964,
+            -0.013826169073581696,
+            -0.004275911021977663,
+            0.013399302028119564,
+            -0.00019625057757366449,
+            -0.0005494111683219671,
+            0.0009396513924002647,
+            -0.014412208460271358,
+            0.03157373145222664,
+            0.0022808476351201534,
+            0.021082919090986252,
+            -8.054639329202473e-05,
+            -0.006601977627724409,
+            -0.006920319981873035,
+            -0.002239246154204011,
+            -0.006894997321069241,
+            -0.0034493072889745235,
+            -0.0011214318219572306,
+            -0.005668656900525093,
+            0.016307789832353592,
+            0.00704693328589201,
+            -0.002043900080025196,
+            0.0006629109266214073,
+            -0.016655072569847107,
+            0.00517305638641119,
+            -0.005249024368822575,
+            -0.005151351448148489,
+            -0.01804419979453087,
+            0.031284332275390625,
+            0.01261791680008173,
+            0.006728590931743383,
+            -0.006370456423610449,
+            -0.020301533862948418,
+            -0.00013983984536025673,
+            0.030358245596289635,
+            -0.011236024089157581,
+            0.004091417416930199,
+            0.01488972082734108,
+            -0.005162204150110483,
+            -0.002150616841390729,
+            -0.014281976968050003,
+            -0.0028053882997483015,
+            -0.027768099680542946,
+            0.0023061702959239483,
+            -0.006775618996471167,
+            0.006323428824543953,
+            -0.002731229178607464,
+            -0.013630823232233524,
+            -0.015902627259492874,
+            -0.0204607043415308,
+            -0.0062438431195914745,
+            -0.009275326505303383,
+            -0.009702194482088089,
+            -0.010411228984594345,
+            -0.025062192231416702,
+            0.021314440295100212,
+            -0.011843767017126083,
+            -0.0004286763723939657,
+            -0.024599149823188782,
+            0.0182033721357584,
+            -0.01034611277282238,
+            -0.015642166137695312,
+            0.0066453879699110985,
+            0.009789015166461468,
+            0.016192028298974037,
+            0.006124464794993401,
+            -0.004977710545063019,
+            -0.004181855358183384,
+            0.014064925722777843,
+            -0.0066236830316483974,
+            -0.005328610073775053,
+            -0.002212114864960313,
+            -0.021893244236707687,
+            -0.007202486507594585,
+            -0.010367818176746368,
+            -0.01177141722291708,
+            -0.005853150971233845,
+            0.010801920667290688,
+            0.015150182880461216,
+            0.0013104473473504186,
+            -0.01761009730398655,
+            -0.0054877810180187225,
+            0.0034113232977688313,
+            0.010085651651024818,
+            -0.023195551708340645,
+            0.008378181606531143,
+            -0.008356476202607155,
+            0.013732113875448704,
+            0.030792348086833954,
+            0.00577718298882246,
+            -0.004720866214483976,
+            0.0008682053303346038,
+            0.02102503925561905,
+            0.0030423360876739025,
+            0.00938385259360075,
+            -0.030358245596289635,
+            -0.022906150668859482,
+            -0.003950334154069424,
+            0.004923447500914335,
+            -0.005932736210525036,
+            0.030358245596289635,
+            0.008573527447879314,
+            0.01840595155954361,
+            0.006612830329686403,
+            0.023484952747821808,
+            0.006620065309107304,
+            0.022558867931365967,
+            0.011568835936486721,
+            0.003633800894021988,
+            -0.007517211139202118,
+            -0.024251868948340416,
+            -0.02944662980735302,
+            0.007690852042287588,
+            0.016568250954151154,
+            0.02115526981651783,
+            -0.03553853556513786,
+            0.023600714281201363,
+            -0.0015284031396731734,
+            0.0003922750474885106,
+            -0.014679905027151108,
+            0.007929608225822449,
+            0.005599924363195896,
+            0.022139234468340874,
+            0.0003649175341706723,
+            0.006775618996471167,
+            0.01039675809442997,
+            0.024512330070137978,
+            -0.017798209562897682,
+            0.01902816630899906,
+            0.017335165292024612,
+            0.0329628624022007,
+            0.017725858837366104,
+            -0.010461874306201935,
+            0.02095268853008747,
+            0.014202391728758812,
+            -0.004351879004389048,
+            -0.011807592585682869,
+            0.0037658405490219593,
+            -0.007799377664923668,
+            0.006319811102002859,
+            0.006493452005088329,
+            0.004464022349566221,
+            -0.010375053621828556,
+            0.007368892431259155,
+            -0.00728568946942687,
+            0.005216466728597879,
+            0.003527084132656455,
+            0.012096993625164032,
+            0.0013782759197056293,
+            0.02785491943359375,
+            -0.04306298494338989,
+            0.02161831222474575,
+            -0.02652367204427719,
+            -0.013529532589018345,
+            -0.005382873117923737,
+            0.005158586427569389,
+            0.0030748937278985977,
+            -0.009904775768518448,
+            -0.011532660573720932,
+            -0.01472331490367651,
+            0.014846310950815678,
+            0.0198674313724041,
+            0.0018286574631929398,
+            -0.0071120485663414,
+            0.006938407197594643,
+            -0.00183227495290339,
+            -0.01747986674308777,
+            0.0019010078394785523,
+            -0.008124954998493195,
+            -0.011236024089157581,
+            -0.02208135463297367,
+            0.013059254735708237,
+            0.0027420816477388144,
+            0.006710503716021776,
+            0.00017431934247724712,
+            -0.025858048349618912,
+            -0.01399257592856884,
+            0.009875834919512272,
+            0.0010328025091439486,
+            -0.013102664612233639,
+            -0.013276305980980396,
+            0.006750296335667372,
+            -0.0027583607006818056,
+            0.003986509516835213,
+            0.011199848726391792,
+            0.0102158822119236,
+            -0.006641770713031292,
+            0.0036247570533305407,
+            0.01910051703453064,
+            0.005267112050205469,
+            -0.016452491283416748,
+            0.01648143120110035,
+            -0.016061797738075256,
+            0.0020312387496232986,
+            0.00035768249654211104,
+            -0.02028706483542919,
+            -0.011973998509347439,
+            0.008182834833860397,
+            -0.009079980663955212,
+            -0.0016794346738606691,
+            0.016061797738075256,
+            0.011735241860151291,
+            0.019375449046492577,
+            0.022037943825125694,
+            0.004084182437509298,
+            0.016452491283416748,
+            -0.02310873195528984,
+            0.014130041003227234,
+            -0.021878773346543312,
+            -0.010888741351664066,
+            0.01753774657845497,
+            0.007032462861388922,
+            0.006573037710040808,
+            -0.0012263399548828602,
+            -0.004275911021977663,
+            -0.0060448795557022095,
+            0.014745020307600498,
+            0.0329628624022007,
+            -0.0022247761953622103,
+            -0.01235022023320198,
+            -0.017364105209708214,
+            -0.00755338603630662,
+            0.03666720539331436,
+            -0.01522253267467022,
+            -0.014578614383935928,
+            0.021213149651885033,
+            0.02742081694304943,
+            0.00489812484011054,
+            -0.009593668393790722,
+            -0.0063993968069553375,
+            0.0032539612147957087,
+            0.015265943482518196,
+            0.013464417308568954,
+            -0.010353348217904568,
+            -0.003545171581208706,
+            0.006891379598528147,
+            0.02600274793803692,
+            0.00716992886736989,
+            -0.013695938512682915,
+            -0.025351595133543015,
+            0.0063849263824522495,
+            -0.0023369193077087402,
+            0.012661327607929707,
+            0.0018241355428472161,
+            0.028592893853783607,
+            0.02516348287463188,
+            0.0012191049754619598,
+            0.018854524940252304,
+            -0.014238567091524601,
+            0.04338132590055466,
+            -0.019303098320961,
+            -0.00472448393702507,
+            -0.02801409177482128,
+            0.0010409420356154442,
+            -0.008291360922157764,
+            0.007980253547430038,
+            -0.006544097326695919,
+            0.002765595680102706,
+            -0.020750107243657112,
+            0.01979508064687252,
+            0.02079351618885994,
+            -0.023730944842100143,
+            0.006739443633705378,
+            -0.024859612807631493,
+            0.01417345181107521,
+            -0.029258519411087036,
+            -0.00961537379771471,
+            -0.017190465703606606,
+            -0.006269165780395269,
+            -0.025886988267302513,
+            0.0021433818619698286,
+            0.0009595477604307234,
+            0.02238522656261921,
+            -0.0051332637667655945,
+            0.003635609755292535,
+            0.01187270786613226,
+            -0.01437603309750557,
+            -0.011981233023107052,
+            0.019245218485593796,
+            0.02559758722782135,
+            0.007929608225822449,
+            0.018319131806492805,
+            0.02089480683207512,
+            -0.01440497301518917,
+            -0.04427846893668175,
+            0.0009749222081154585,
+            -0.003903306322172284,
+            -0.001428921241313219,
+            0.004565312992781401,
+            0.012972434051334858,
+            -0.02593039907515049,
+            -0.001539255608804524,
+            -0.004160150419920683,
+            0.0033823831472545862,
+            -0.004536372609436512,
+            0.015902627259492874,
+            0.025481825694441795,
+            -0.013218426145613194,
+            -0.013804464600980282,
+            -0.011366254650056362,
+            -0.005361167713999748,
+            0.008436061441898346,
+            -0.015396174043416977,
+            0.0008849363657645881,
+            -0.0034891001414507627,
+            -0.015511934645473957,
+            -0.00976730976253748,
+            0.002360433107241988,
+            -0.012943494133651257,
+            -0.008566292002797127,
+            0.006844351999461651,
+            -0.027637869119644165,
+            -0.01640908047556877,
+            0.019158396869897842,
+            -0.015164652839303017,
+            0.005118793807923794,
+            0.013746583834290504,
+            0.008812284097075462,
+            0.011149203404784203,
+            0.030097784474492073,
+            -0.00984689500182867,
+            0.0038562784902751446,
+            -0.021763013675808907,
+            -0.016770832240581512,
+            -0.0096804890781641,
+            0.01797184906899929,
+            0.003168949391692877,
+            -0.011829297058284283,
+            0.007647441700100899,
+            -0.012379161082208157,
+            -0.013232896104454994,
+            -0.006949259899556637,
+            0.001656825072132051,
+            -0.008645878173410892,
+            0.015265943482518196,
+            -0.01039675809442997,
+            -0.004131210036575794,
+            0.012075289152562618,
+            0.036696143448352814,
+            -0.0035415540914982557,
+            -0.00988307036459446,
+            -0.0014334431616589427,
+            0.01501995138823986,
+            0.008602467365562916,
+            -0.004351879004389048,
+            -0.020634345710277557,
+            -0.015700045973062515,
+            -0.0022302023135125637,
+            -0.010780216194689274,
+            -0.015627695247530937,
+            0.02271803840994835,
+            -0.021632781252264977,
+            -0.018854524940252304,
+            -0.018782174214720726,
+            -0.0158881563693285,
+            0.00048113043885678053,
+            0.03270240128040314,
+            0.0147739602252841,
+            0.0024780025705695152,
+            0.0038020156789571047,
+            -0.010056711733341217,
+            -0.027478698641061783,
+            0.024584680795669556,
+            0.011380724608898163,
+            -0.030097784474492073,
+            -0.009275326505303383,
+            -0.0187966451048851,
+            -0.004066094756126404,
+            0.007821083068847656,
+            -0.0022591424640268087,
+            0.05417601019144058,
+            -0.0007131040329113603,
+            0.006478982046246529,
+            -0.026682842522859573,
+            0.008298595435917377,
+            -0.015555345453321934,
+            -0.02192218415439129,
+            -0.010990031994879246,
+            -0.012299574911594391,
+            -0.009470673277974129,
+            -0.016336729750037193,
+            -0.002657070057466626,
+            -0.011438604444265366,
+            0.006471747066825628,
+            -0.015092302113771439,
+            -0.0034456897992640734,
+            -0.02407822757959366,
+            -0.004507432691752911,
+            0.005683127325028181,
+            -0.0007497314363718033,
+            0.008964220061898232,
+            -0.01863747462630272,
+            -0.006486217025667429,
+            0.0014054073253646493,
+            0.010794686153531075,
+            0.007886198349297047,
+            -0.0038490435108542442,
+            -0.014115571044385433,
+            -0.013167780824005604,
+            -0.008616937324404716,
+            0.01001330092549324,
+            -0.00699628796428442,
+            0.008096014149487019,
+            -0.01664060167968273,
+            -0.014643729664385319,
+            0.004062477499246597,
+            0.03328120335936546,
+            0.01490419078618288,
+            -0.009065510705113411,
+            -0.00961537379771471,
+            -0.014296446926891804,
+            0.034004706889390945,
+            -0.012777088209986687,
+            -0.014585848897695541,
+            0.012625152245163918,
+            0.022443106397986412,
+            -0.0007632971392013133,
+            0.0045508430339396,
+            0.010440168902277946,
+            -0.00010264718730468303,
+            -0.0007316438714042306,
+            0.0005679509486071765,
+            0.005328610073775053,
+            0.00020608570775948465,
+            0.028144322335720062,
+            -0.009065510705113411,
+            0.014419442974030972,
+            -0.004594252910465002,
+            0.006887761875987053,
+            -0.028100911527872086,
+            -0.0024906639009714127,
+            -0.006522392388433218,
+            -0.005965293850749731,
+            0.004771511536091566,
+            -0.0015076022827997804,
+            0.01774032786488533,
+            -0.03044506534934044,
+            -0.027305057272315025,
+            0.016669541597366333,
+            -0.02112632989883423,
+            -0.007958549074828625,
+            0.005433517973870039,
+            -0.014072161167860031,
+            0.008710993453860283,
+            0.006507922429591417,
+            -0.022327346727252007,
+            -0.01607626862823963,
+            0.0019588882569223642,
+            -0.0023206404875963926,
+            0.0027221853379160166,
+            0.01661166176199913,
+            -0.010548694059252739,
+            -0.012545567005872726,
+            -0.00761850131675601,
+            0.012914554215967655,
+            -0.005212849471718073,
+            0.024165047332644463,
+            -0.013261836022138596,
+            -0.0005720206536352634,
+            0.0009541214676573873,
+            -0.005976146552711725,
+            0.015613225288689137,
+            0.015468524768948555,
+            -0.0035071875900030136,
+            -0.0010002448689192533,
+            0.0003527084190864116,
+            -0.006847969256341457,
+            0.011033442802727222,
+            0.010635514743626118,
+            0.01460755430161953,
+            -0.004102270118892193,
+            0.012531096115708351,
+            -0.004297616425901651,
+            0.0027203767094761133,
+            -0.02079351618885994,
+            0.0040588597767055035,
+            0.002065605018287897,
+            0.018232312053442,
+            -0.020214714109897614,
+            -0.011800357140600681,
+            -0.031110690906643867,
+            -0.004529137630015612,
+            -0.01691553369164467,
+            0.0022012621629983187,
+            0.030155664309859276,
+            0.0015817615203559399,
+            -0.010541459545493126,
+            0.014629259705543518,
+            0.031226450577378273,
+            0.009506847709417343,
+            0.0004411115951370448,
+            0.010830861516296864,
+            -0.011532660573720932,
+            0.010592104867100716,
+            0.012480450794100761,
+            -0.01846383325755596,
+            -0.0015636739553883672,
+            -0.0007429486140608788,
+            -0.003494526259601116,
+            0.009825189597904682,
+            0.007517211139202118,
+            -0.004322939086705446,
+            0.017682448029518127,
+            -0.027015654370188713,
+            0.0036591235548257828,
+            -0.0030115870758891106,
+            -0.004681073594838381,
+            -0.027305057272315025,
+            -0.006352368742227554,
+            0.01747986674308777,
+            -0.010519754141569138,
+            0.058574918657541275,
+            0.002271803794428706,
+            0.007922373712062836,
+            -0.027536578476428986,
+            -0.005230937153100967,
+            -0.008906339295208454,
+            0.0034493072889745235,
+            -0.007184398826211691,
+            0.00761126633733511,
+            0.004008214455097914,
+            0.00022835608979221433,
+            -0.0035885819233953953,
+            0.0020583700388669968,
+            0.0022917003370821476,
+            0.016872122883796692,
+            0.00506453076377511,
+            0.03524913638830185,
+            -0.007727027405053377,
+            -0.005151351448148489,
+            -0.011062382720410824,
+            0.005932736210525036,
+            0.019042637199163437,
+            0.0002742533979471773,
+            0.006956494878977537,
+            0.011156437918543816,
+            0.011395194567739964,
+            0.013985340483486652,
+            0.018261251971125603,
+            -0.0033950444776564837,
+            0.00500665046274662,
+            0.01622096821665764,
+            0.00494153518229723,
+            -0.01410833653062582,
+            -0.011127498000860214,
+            -0.004297616425901651,
+            0.018290191888809204,
+            -0.007466565817594528,
+            0.011207083240151405,
+            0.006761149037629366,
+            0.00582782831043005,
+            0.01016523689031601,
+            -0.01178588718175888,
+            -0.00023604331363458186,
+            0.012777088209986687,
+            -0.030097784474492073,
+            0.012574506923556328,
+            0.0029301929753273726,
+            -0.00789343286305666,
+            0.00244906242005527,
+            -0.0076980870217084885,
+            -0.009065510705113411,
+            -0.0020836926996707916,
+            -0.009145095944404602,
+            0.0005069052567705512,
+            0.015251473523676395,
+            -0.012133168987929821,
+            0.0023658594582229853,
+            -0.003762223059311509,
+            -0.017103644087910652,
+            0.02595933899283409,
+            -0.018290191888809204,
+            0.005183909088373184,
+            0.014368797652423382
+          ],
+          "title": "Introduction And Related Work",
+          "keyphrases": [
+            "conditional computation",
+            "model capacity",
+            "Sparsely-Gated Mixture-of-Experts Layer",
+            "language modeling",
+            "machine translation"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": true,
+      "properties": {
+        "cosine_similarity": 0.8290300345650549
+      }
+    },
+    {
+      "id": "8cb929cd-446d-4596-a826-84b8240164e3",
+      "type": "cosine_similarity",
+      "source": {
+        "id": "86d0cb2e-1f6d-4e38-a077-7b7de2fe9e4a",
+        "properties": {
+          "page_content": "3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n",
+          "embedding": [
+            0.010104401037096977,
+            -0.020574966445565224,
+            -0.02167346142232418,
+            -0.031542472541332245,
+            0.03588413819670677,
+            0.03464615345001221,
+            -0.009755672886967659,
+            0.046869076788425446,
+            0.004803731571882963,
+            0.0364421010017395,
+            0.016547156497836113,
+            -0.007223033811897039,
+            0.009546436369419098,
+            -0.0017371026333421469,
+            0.005762734450399876,
+            0.04048734903335571,
+            0.011603932827711105,
+            0.01784616895020008,
+            0.011150586418807507,
+            -0.02915368229150772,
+            0.005383492447435856,
+            -0.01708768494427204,
+            -0.04317255690693855,
+            -0.006194285582751036,
+            0.009546436369419098,
+            0.005871712230145931,
+            0.027340294793248177,
+            0.0036812627222388983,
+            -0.055901139974594116,
+            0.023800702765583992,
+            0.021708333864808083,
+            0.010984940454363823,
+            -0.01007824670523405,
+            -0.010601338930428028,
+            -0.026573093608021736,
+            -0.021969880908727646,
+            0.04634598642587662,
+            0.007685098797082901,
+            -0.006399163510650396,
+            0.042475100606679916,
+            0.03703493997454643,
+            0.012728581205010414,
+            -0.012772172689437866,
+            -0.014175803400576115,
+            -0.003430614247918129,
+            -0.0012979229213669896,
+            -0.004712190479040146,
+            0.0065735275857150555,
+            0.009311044588685036,
+            -0.026172054931521416,
+            -0.001842810888774693,
+            -0.021900134161114693,
+            0.02798544242978096,
+            0.020609840750694275,
+            -0.049379922449588776,
+            0.028072625398635864,
+            -0.004890913609415293,
+            0.06461934745311737,
+            0.047461915761232376,
+            0.014602995477616787,
+            0.03794163465499878,
+            -0.034035876393318176,
+            -0.029275737702846527,
+            -0.003177786245942116,
+            0.03234454616904259,
+            0.007654585409909487,
+            -0.01033979281783104,
+            0.0010679803090170026,
+            0.011717269197106361,
+            -0.005706066265702248,
+            -0.044532597064971924,
+            0.010496720671653748,
+            -0.014925569295883179,
+            0.002881367225199938,
+            0.004034350160509348,
+            -0.020017001777887344,
+            0.03030448593199253,
+            0.0268520750105381,
+            0.014899414964020252,
+            0.010740830563008785,
+            0.010653648525476456,
+            0.03204812481999397,
+            -0.0026961052790284157,
+            0.009555154480040073,
+            0.0364421010017395,
+            -0.029415227472782135,
+            -0.01360040158033371,
+            -0.062178246676921844,
+            0.010540311224758625,
+            0.0028072623535990715,
+            -0.02815980650484562,
+            0.0018340926617383957,
+            -0.006569168530404568,
+            0.0036986989434808493,
+            -0.033094312995672226,
+            0.0727447122335434,
+            0.010653648525476456,
+            0.00023280335881281644,
+            -0.024044813588261604,
+            0.006394804455339909,
+            -0.015762517228722572,
+            0.021969880908727646,
+            0.0006560450419783592,
+            0.004008195362985134,
+            0.0229811929166317,
+            -0.02859571762382984,
+            -0.004690395202487707,
+            0.03815086930990219,
+            0.02148166112601757,
+            0.08676359057426453,
+            0.01007824670523405,
+            -0.017593340948224068,
+            -0.011211613193154335,
+            0.018482597544789314,
+            -0.04278895631432533,
+            -0.017689241096377373,
+            -0.03965040296316147,
+            -0.03215274587273598,
+            0.0114818774163723,
+            -0.025038689374923706,
+            0.01996469311416149,
+            0.02566639892756939,
+            -0.0011344566009938717,
+            -0.025422289967536926,
+            -0.07155904173851013,
+            0.04292844980955124,
+            -0.023696085438132286,
+            0.01780257746577263,
+            0.001439593848772347,
+            -0.0022144243121147156,
+            -0.00422179140150547,
+            0.0037183149252086878,
+            -0.020261110737919807,
+            -0.041882261633872986,
+            0.01856978051364422,
+            0.010479284450411797,
+            0.046834204345941544,
+            0.0633290484547615,
+            -0.021132932975888252,
+            -0.0072448295541107655,
+            0.017375385388731956,
+            -0.022894009947776794,
+            0.003243172774091363,
+            -0.0038142153061926365,
+            -0.029450101777911186,
+            0.013975284993648529,
+            0.028508534654974937,
+            0.08578714728355408,
+            -0.01956365443766117,
+            0.012022406794130802,
+            -0.033286113291978836,
+            0.037662651389837265,
+            -0.009799264371395111,
+            0.023417102172970772,
+            -0.025404853746294975,
+            -0.021708333864808083,
+            -0.025352545082569122,
+            0.010705957189202309,
+            -0.014995315112173557,
+            0.0059632533229887486,
+            -0.005889148451387882,
+            -0.0062945447862148285,
+            -0.018691834062337875,
+            0.010470565408468246,
+            0.007685098797082901,
+            0.047985006123781204,
+            0.029816266149282455,
+            0.005858634598553181,
+            -0.07769665867090225,
+            0.038325235247612,
+            0.017514877021312714,
+            0.036546722054481506,
+            0.01543994341045618,
+            0.033443041145801544,
+            -0.026538221165537834,
+            0.017680522054433823,
+            0.0032475318294018507,
+            0.01094134896993637,
+            -0.03197837993502617,
+            0.044916197657585144,
+            -0.00843050517141819,
+            0.002720080316066742,
+            0.01972058229148388,
+            0.013312701135873795,
+            0.04062684252858162,
+            0.027915697544813156,
+            0.034210242331027985,
+            0.03815086930990219,
+            -0.01332141924649477,
+            0.010453129187226295,
+            0.029380355030298233,
+            0.03515180945396423,
+            0.02796800620853901,
+            0.0027636713348329067,
+            0.0041890982538461685,
+            -0.026590529829263687,
+            -0.006721737328916788,
+            0.009964910335838795,
+            0.007039951626211405,
+            0.016390228644013405,
+            -0.012780890800058842,
+            -0.028124934062361717,
+            0.0220221895724535,
+            -0.028648026287555695,
+            0.030740395188331604,
+            0.008626664988696575,
+            -0.01083673071116209,
+            0.013208082877099514,
+            0.009895164519548416,
+            -0.029833702370524406,
+            0.004243587143719196,
+            -0.006250953767448664,
+            -0.004468080587685108,
+            -0.025021253153681755,
+            -0.01933698169887066,
+            0.00797280017286539,
+            0.0029532923363149166,
+            0.005021686665713787,
+            0.009380790404975414,
+            -0.022545281797647476,
+            0.004284998401999474,
+            0.02127242274582386,
+            0.022876573726534843,
+            -0.018325669690966606,
+            -0.018901070579886436,
+            -0.011438286863267422,
+            -0.01914518140256405,
+            -0.011865478940308094,
+            0.044393107295036316,
+            -0.012711144983768463,
+            -0.009694645181298256,
+            0.020574966445565224,
+            0.006774046458303928,
+            -0.005043482407927513,
+            0.017872322350740433,
+            -0.029484974220395088,
+            -0.010932630859315395,
+            -0.0035592077765613794,
+            0.011072122491896152,
+            0.010243892669677734,
+            -0.00997362844645977,
+            0.01550968922674656,
+            -0.050879452377557755,
+            -0.010932630859315395,
+            0.007881258614361286,
+            0.015082497149705887,
+            -0.0110285310074687,
+            0.02339966595172882,
+            0.012161897495388985,
+            0.016224581748247147,
+            -0.003081885864958167,
+            -0.014829669147729874,
+            0.008221268653869629,
+            0.017384104430675507,
+            0.007881258614361286,
+            0.05712168663740158,
+            0.02338222973048687,
+            -0.03947603702545166,
+            0.006538654677569866,
+            -0.029973194003105164,
+            0.05579652264714241,
+            0.007994594983756542,
+            -0.029833702370524406,
+            0.053215932101011276,
+            -0.023434538394212723,
+            -0.0024193022400140762,
+            0.060853078961372375,
+            -0.0007007258245721459,
+            -0.013548092916607857,
+            0.0013011923292651772,
+            0.02435866929590702,
+            -0.056075502187013626,
+            -0.04899631813168526,
+            -0.02148166112601757,
+            0.02857828140258789,
+            -0.02453303337097168,
+            0.0077156126499176025,
+            -0.015405070967972279,
+            0.04735729843378067,
+            0.036721084266901016,
+            0.018116433173418045,
+            -0.008731283247470856,
+            0.02008674666285515,
+            0.04749678820371628,
+            -0.007977158762514591,
+            -0.004742704331874847,
+            -0.02833417057991028,
+            0.005152460187673569,
+            -0.033216364681720734,
+            -0.009485408663749695,
+            -0.027462350204586983,
+            -0.011133149266242981,
+            -0.02125498652458191,
+            -0.0030317562632262707,
+            0.022336045280098915,
+            0.0041389684192836285,
+            5.1253518904559314e-05,
+            -0.0020389705896377563,
+            0.040522221475839615,
+            -0.024951506406068802,
+            -0.04470696300268173,
+            -0.00825178250670433,
+            0.01704409345984459,
+            0.0038556267973035574,
+            -0.05475033447146416,
+            -0.006250953767448664,
+            -0.0029707287903875113,
+            0.01304243691265583,
+            -0.023539157584309578,
+            -0.00863538309931755,
+            0.016573309898376465,
+            0.002870469354093075,
+            0.009807982482016087,
+            -0.02800287865102291,
+            -0.014629150740802288,
+            0.007406116463243961,
+            0.01841285265982151,
+            0.02817724272608757,
+            -0.0555872842669487,
+            -0.03225736320018768,
+            0.02067958563566208,
+            -0.009040780365467072,
+            -0.06409624963998795,
+            0.007833308540284634,
+            -0.001987751107662916,
+            0.01799437776207924,
+            0.016948193311691284,
+            0.04160328209400177,
+            -0.019581090658903122,
+            -0.022283734753727913,
+            -0.016599465161561966,
+            -0.0364072285592556,
+            -0.028700334951281548,
+            -0.009712082333862782,
+            -0.008299732580780983,
+            -0.028857262805104256,
+            -0.0017839629435911775,
+            0.01342603750526905,
+            -0.03759290650486946,
+            0.006176849361509085,
+            -0.021603714674711227,
+            0.0028486738447099924,
+            -0.03703493997454643,
+            0.007842026650905609,
+            -0.0030339357908815145,
+            -0.01613740064203739,
+            -0.02125498652458191,
+            -0.01561430748552084,
+            0.014725050888955593,
+            -0.0002036518562817946,
+            0.0017142173601314425,
+            0.0229637548327446,
+            0.007484580390155315,
+            0.009328480809926987,
+            -0.008535124361515045,
+            -0.029798829928040504,
+            -0.013382446952164173,
+            0.006586604751646519,
+            0.018534906208515167,
+            0.019999565556645393,
+            0.005501188337802887,
+            -0.04958915710449219,
+            -0.03584926575422287,
+            0.021621152758598328,
+            -0.05890020355582237,
+            0.006394804455339909,
+            -0.0017425514524802566,
+            -0.008622306399047375,
+            0.0033150978852063417,
+            -0.005832480266690254,
+            -0.026294110342860222,
+            0.015570716932415962,
+            0.0086135882884264,
+            0.012275234796106815,
+            0.007488939445465803,
+            0.015980472788214684,
+            0.02852597087621689,
+            -0.02451559714972973,
+            -0.05788889154791832,
+            0.03427998721599579,
+            -0.019075434654951096,
+            0.014210676774382591,
+            0.0440792515873909,
+            0.005475033540278673,
+            -0.006551732309162617,
+            0.025003815069794655,
+            0.013399883173406124,
+            -0.010034655220806599,
+            -0.03407074883580208,
+            0.04376539587974548,
+            -0.003430614247918129,
+            -0.07121030986309052,
+            0.005479392595589161,
+            0.0014810053398832679,
+            0.013103464618325233,
+            0.015283015556633472,
+            -0.003430614247918129,
+            -0.023643774911761284,
+            -0.01956365443766117,
+            -0.014088621363043785,
+            -0.018168741837143898,
+            0.029484974220395088,
+            0.003539591794833541,
+            0.005004250444471836,
+            -0.0006091846735216677,
+            -0.0014341449132189155,
+            0.004677317570894957,
+            0.014027593657374382,
+            0.04080120474100113,
+            -0.0364769771695137,
+            0.0025958458427339792,
+            -0.04275408387184143,
+            -0.03874370828270912,
+            -0.0046380856074392796,
+            0.018360542133450508,
+            0.026485910639166832,
+            -0.025910509750247,
+            0.02320786565542221,
+            0.0100608104839921,
+            -0.015675334259867668,
+            -0.005008609499782324,
+            0.028090061619877815,
+            0.025369981303811073,
+            0.01285063661634922,
+            -0.004533467348664999,
+            0.014690177515149117,
+            -0.012623962946236134,
+            0.0026045641861855984,
+            0.01574508100748062,
+            -0.00010305192699888721,
+            0.03490769863128662,
+            0.001937621389515698,
+            0.018918508663773537,
+            -0.04693882167339325,
+            -0.02699156664311886,
+            -0.027689022943377495,
+            -0.0172620490193367,
+            0.03874370828270912,
+            -0.006503781769424677,
+            -0.012728581205010414,
+            0.020261110737919807,
+            0.004282819107174873,
+            0.0017044093692675233,
+            -0.04066171497106552,
+            -0.028630590066313744,
+            -0.022283734753727913,
+            -0.044916197657585144,
+            -0.005618884228169918,
+            -0.02183038927614689,
+            -0.019057998433709145,
+            -0.05342517048120499,
+            0.0010527233825996518,
+            -0.07811512798070908,
+            -0.005370415281504393,
+            0.04048734903335571,
+            0.004686036147177219,
+            0.009424380958080292,
+            -0.028682898730039597,
+            0.01025261078029871,
+            -0.036337483674287796,
+            -0.02568383514881134,
+            0.017785141244530678,
+            0.04387001320719719,
+            -0.011813169345259666,
+            0.0019997386261820793,
+            0.019057998433709145,
+            -0.05056559666991234,
+            -0.005370415281504393,
+            -0.0031799657735973597,
+            0.007641507778316736,
+            0.012920381501317024,
+            -0.016974348574876785,
+            0.05457597225904465,
+            0.011054685339331627,
+            0.02223142609000206,
+            0.02606743760406971,
+            -0.00020024630066473037,
+            -0.038046251982450485,
+            0.017584621906280518,
+            0.008535124361515045,
+            0.02796800620853901,
+            0.0015681873774155974,
+            0.0073014977388083935,
+            -0.019807765260338783,
+            0.008979752659797668,
+            -0.009485408663749695,
+            0.01646869257092476,
+            0.018866198137402534,
+            0.02645103819668293,
+            -0.008644101209938526,
+            -0.002175192581489682,
+            0.0033434322103857994,
+            -0.0034785643219947815,
+            0.028700334951281548,
+            0.0076371487230062485,
+            -0.038848329335451126,
+            0.006813278421759605,
+            -0.002189359627664089,
+            0.023120682686567307,
+            0.030374230816960335,
+            -0.04108018800616264,
+            -0.02338222973048687,
+            -0.002796364715322852,
+            0.016974348574876785,
+            0.011019812896847725,
+            -0.020662149414420128,
+            -0.0031668883748352528,
+            0.044602345675230026,
+            -0.010479284450411797,
+            -0.032484035938978195,
+            0.01293781865388155,
+            -0.0018504393519833684,
+            0.0059806895442306995,
+            0.0024476363323628902,
+            0.012772172689437866,
+            0.013469628989696503,
+            -0.019598528742790222,
+            0.04087095335125923,
+            -0.004965018481016159,
+            0.013208082877099514,
+            -0.012031124904751778,
+            -0.010052092373371124,
+            -0.015195833519101143,
+            0.03926680237054825,
+            -0.0047906544059515,
+            0.02013905718922615,
+            0.024864325299859047,
+            0.006338136270642281,
+            0.05216974765062332,
+            -0.04125455394387245,
+            -0.03724417835474014,
+            0.005971971433609724,
+            0.0016215863870456815,
+            0.005871712230145931,
+            -0.011272640898823738,
+            -0.030932195484638214,
+            -0.012074715457856655,
+            0.004768858663737774,
+            -0.015535843558609486,
+            -0.015684053301811218,
+            0.00019043832435272634,
+            0.022144244983792305,
+            0.0011257383739575744,
+            0.005475033540278673,
+            0.00287700816988945,
+            0.014585559256374836,
+            0.03581439331173897,
+            -0.02589307352900505,
+            0.010758266784250736,
+            0.03147272393107414,
+            -0.009136680513620377,
+            -0.026764893904328346,
+            -0.027305422350764275,
+            0.015073779039084911,
+            -0.018918508663773537,
+            -0.028874699026346207,
+            3.807404209510423e-05,
+            -0.0034872826654464006,
+            0.008391273207962513,
+            -0.03584926575422287,
+            -0.0076763806864619255,
+            0.0021283321548253298,
+            -0.00196377607062459,
+            0.0007078094058670104,
+            -0.022318609058856964,
+            -0.013591683469712734,
+            0.01741897687315941,
+            0.024219177663326263,
+            0.005727861542254686,
+            0.03028704971075058,
+            -0.010243892669677734,
+            -0.00016768925706855953,
+            0.006525577511638403,
+            0.01589328981935978,
+            0.023050937801599503,
+            0.023312482982873917,
+            0.026573093608021736,
+            0.0009344827267341316,
+            -0.012327543459832668,
+            0.004252305254340172,
+            0.032484035938978195,
+            -0.016058936715126038,
+            -0.0030993223190307617,
+            0.014812232926487923,
+            0.03274558484554291,
+            0.010627493262290955,
+            -0.012188052758574486,
+            -0.02528279833495617,
+            0.015431225299835205,
+            -0.022283734753727913,
+            0.012510626576840878,
+            0.04777577146887779,
+            0.004393976181745529,
+            -0.029240863397717476,
+            0.02430635876953602,
+            -0.009546436369419098,
+            -0.024044813588261604,
+            0.027165930718183517,
+            0.003757546888664365,
+            -0.03271070867776871,
+            -6.98818766977638e-05,
+            -0.00018321855168323964,
+            0.014794795773923397,
+            -0.026015127077698708,
+            -0.015579435043036938,
+            0.02322530187666416,
+            0.009293608367443085,
+            0.0032627887558192015,
+            -0.021603714674711227,
+            0.001829733606427908,
+            0.015344043262302876,
+            0.0018177460879087448,
+            0.009424380958080292,
+            0.009145398624241352,
+            0.01505634281784296,
+            0.006939692422747612,
+            -0.013190646655857563,
+            -0.0032344546634703875,
+            -0.02720080316066742,
+            -0.006098385434597731,
+            0.0652819275856018,
+            -0.007292779628187418,
+            0.03686057776212692,
+            0.0015213270671665668,
+            0.012868072837591171,
+            0.025788454338908195,
+            -0.025160742923617363,
+            -0.02437610551714897,
+            -0.005047841463238001,
+            0.006961487699300051,
+            -0.04292844980955124,
+            -0.013269110582768917,
+            -0.023818138986825943,
+            -0.030217302963137627,
+            0.01933698169887066,
+            0.0005187332862988114,
+            0.0019267236348241568,
+            0.024829450994729996,
+            0.0051742554642260075,
+            -0.015038905665278435,
+            -0.017009221017360687,
+            0.0383601076900959,
+            0.016625620424747467,
+            0.028229551389813423,
+            -0.015989189967513084,
+            0.01625945419073105,
+            -0.008099214173853397,
+            0.003171247662976384,
+            0.03236198052763939,
+            -0.0026982848066836596,
+            -0.0030186788644641638,
+            0.009197707287967205,
+            0.01881388947367668,
+            0.040731459856033325,
+            -0.01745384931564331,
+            -0.030827578157186508,
+            -0.008914366364479065,
+            -0.014236831106245518,
+            0.021969880908727646,
+            -0.0009491947130300105,
+            -0.00041084547410719097,
+            -0.003934090491384268,
+            0.04352128505706787,
+            -0.029572155326604843,
+            0.014428631402552128,
+            -0.024445850402116776,
+            -0.039022691547870636,
+            0.017706677317619324,
+            -0.006102744489908218,
+            0.032693274319171906,
+            0.006883023772388697,
+            -0.007833308540284634,
+            -0.032867636531591415,
+            0.0016749853966757655,
+            0.0086179468780756,
+            -0.03194350749254227,
+            0.020418038591742516,
+            0.02472483366727829,
+            -0.00651250034570694,
+            0.0033063797745853662,
+            -0.016782546415925026,
+            0.008142804726958275,
+            -0.013844511471688747,
+            -0.022946318611502647,
+            0.022737082093954086,
+            -0.004298075567930937,
+            -0.028055187314748764,
+            0.006686864420771599,
+            -0.022458098828792572,
+            0.004707831423729658,
+            -0.0016771649243310094,
+            -0.00017014124023262411,
+            -0.00797280017286539,
+            0.010461847297847271,
+            -0.007571762427687645,
+            -0.01073211245238781,
+            -0.022353481501340866,
+            0.009991064667701721,
+            0.025631526485085487,
+            0.0017588981427252293,
+            -0.004672958515584469,
+            0.006848151329904795,
+            -0.03658159449696541,
+            0.00574093870818615,
+            0.02413199469447136,
+            -0.012205488979816437,
+            -0.008713847026228905,
+            -0.018255924805998802,
+            -0.029380355030298233,
+            -0.002366993110626936,
+            0.01130751334130764,
+            0.03584926575422287,
+            0.025788454338908195,
+            0.030409103259444237,
+            0.03391382470726967,
+            0.010810576379299164,
+            -0.00398204056546092,
+            0.009127962402999401,
+            -0.029031626880168915,
+            0.04006887599825859,
+            0.0040823002345860004,
+            0.02608487382531166,
+            0.016172273084521294,
+            -0.0459623858332634,
+            -0.00968592707067728,
+            0.014681459404528141,
+            0.027898261323571205,
+            0.044148996472358704,
+            0.025038689374923706,
+            -0.0010232995264232159,
+            0.027654150500893593,
+            -0.022196553647518158,
+            -0.02878751792013645,
+            0.01956365443766117,
+            -0.005235282704234123,
+            -0.00918898917734623,
+            -0.0013131798477843404,
+            0.021533969789743423,
+            -0.011385977268218994,
+            0.016058936715126038,
+            0.03415793180465698,
+            -0.008461019024252892,
+            0.012135743163526058,
+            0.0005590550135821104,
+            -0.008286654949188232,
+            0.026764893904328346,
+            -0.007214315701276064,
+            0.03485538810491562,
+            -0.01650356501340866,
+            -0.0018656961619853973,
+            0.000780279457103461,
+            -0.010766984894871712,
+            0.028020314872264862,
+            0.007270984351634979,
+            -0.06172490119934082,
+            0.00873564276844263,
+            0.011063403449952602,
+            0.013835793361067772,
+            -0.001278307056054473,
+            -0.0033957413397729397,
+            -0.025788454338908195,
+            0.019040562212467194,
+            0.008129727095365524,
+            0.02472483366727829,
+            0.010679802857339382,
+            0.019493909552693367,
+            -0.016643056645989418,
+            0.016599465161561966,
+            -0.0026590528432279825,
+            -0.02854340709745884,
+            0.007266624830663204,
+            0.013181927613914013,
+            0.0402083694934845,
+            -0.026538221165537834,
+            -0.002582768676802516,
+            0.010889039374887943,
+            -0.006970205809921026,
+            -0.011194176971912384,
+            -0.030443977564573288,
+            -0.0010756086558103561,
+            -0.028421353548765182,
+            0.02416686899960041,
+            -0.005139382556080818,
+            0.019406726583838463,
+            -0.027357731014490128,
+            0.0063337767496705055,
+            0.00043264098349027336,
+            -0.027654150500893593,
+            0.006390445400029421,
+            0.01782001368701458,
+            0.0031189383007586002,
+            -0.031176306307315826,
+            -0.006006844341754913,
+            0.023242738097906113,
+            0.03619799390435219,
+            0.006713018752634525,
+            -0.006848151329904795,
+            0.021533969789743423,
+            -0.015579435043036938,
+            -0.006150694563984871,
+            0.036546722054481506,
+            -0.025910509750247,
+            0.012231643311679363,
+            0.03759290650486946,
+            -0.03543078899383545,
+            -0.03196094557642937,
+            -0.0004021272761747241,
+            -0.04142891615629196,
+            0.001680434332229197,
+            -0.002975087845697999,
+            0.00835204217582941,
+            -0.0009715351043269038,
+            -0.027113622054457664,
+            0.024812014773488045,
+            -0.0003577189054340124,
+            0.013373728841543198,
+            -0.006848151329904795,
+            0.0021065366454422474,
+            -0.025143306702375412,
+            -0.01862208917737007,
+            -0.010793139226734638,
+            0.00015651904686819762,
+            0.0023059656377881765,
+            -0.008112290874123573,
+            -0.03944116458296776,
+            -0.020976005122065544,
+            0.007257906720042229,
+            0.00045770584256388247,
+            0.03584926575422287,
+            0.018482597544789314,
+            0.012188052758574486,
+            -0.02165602520108223,
+            -0.01898825354874134,
+            0.04756653308868408,
+            -0.026416165754199028,
+            0.0402432419359684,
+            -0.012135743163526058,
+            -0.017488721758127213,
+            0.002946753753349185,
+            0.02625923790037632,
+            -0.020923694595694542,
+            0.016817420721054077,
+            -0.01561430748552084,
+            0.005719143431633711,
+            -0.01409733947366476,
+            -0.004755781497806311,
+            0.009642336517572403,
+            -0.0017796038882806897,
+            0.018255924805998802,
+            0.0009072383400052786,
+            -0.0031123997177928686,
+            0.03183889016509056,
+            0.013652711175382137,
+            0.007763562723994255,
+            -0.016773829236626625,
+            -0.012222925201058388,
+            -0.010766984894871712,
+            0.007087901700288057,
+            -0.015073779039084911,
+            -0.007066106423735619,
+            -0.029659338295459747,
+            0.001694601378403604,
+            0.0007154378108680248,
+            -0.015256861224770546,
+            0.01235369872301817,
+            0.02706131339073181,
+            -0.013295264914631844,
+            -0.03157734498381615,
+            -0.006229158490896225,
+            -0.03944116458296776,
+            -0.021045750007033348,
+            0.019511345773935318,
+            -0.0005190056981518865,
+            0.009127962402999401,
+            0.004250125493854284,
+            0.00043509298120625317,
+            -0.022946318611502647,
+            -0.017976941540837288,
+            0.00714892940595746,
+            -0.009572590701282024,
+            0.00107724335975945,
+            0.00131862866692245,
+            0.01822105050086975,
+            -0.010156710632145405,
+            0.02800287865102291,
+            -0.017497440800070763,
+            0.015265579335391521,
+            -0.029851138591766357,
+            -0.02760184183716774,
+            -0.03199581801891327,
+            0.01304243691265583,
+            -0.014611713588237762,
+            -0.008195114322006702,
+            -0.010113119147717953,
+            0.0004999346565455198,
+            0.007567403372377157,
+            -0.009441817179322243,
+            0.00842614658176899,
+            0.011603932827711105,
+            0.013303983025252819,
+            0.007406116463243961,
+            -0.03244916349649429,
+            -0.02029598504304886,
+            -0.02397506684064865,
+            0.043974634259939194,
+            -0.030513722449541092,
+            0.01703537441790104,
+            -0.017392821609973907,
+            0.005823761690407991,
+            -0.0363723561167717,
+            -0.019075434654951096,
+            0.008953598327934742,
+            0.020749330520629883,
+            -0.025910509750247,
+            0.02741004154086113,
+            0.02934548258781433,
+            0.03013012185692787,
+            -0.036546722054481506,
+            0.025352545082569122,
+            -0.010505438782274723,
+            -0.02512587048113346,
+            0.027357731014490128,
+            0.03731392323970795,
+            0.03679082915186882,
+            -0.013460910879075527,
+            -0.012589089572429657,
+            -0.014690177515149117,
+            0.012318825349211693,
+            -0.0021555765997618437,
+            0.005801966413855553,
+            -0.037069812417030334,
+            0.01862208917737007,
+            0.009895164519548416,
+            0.005121946334838867,
+            0.03773239627480507,
+            -0.01898825354874134,
+            0.02054009400308132,
+            0.007471502758562565,
+            -0.01839541457593441,
+            0.006974565330892801,
+            0.000942111189942807,
+            -0.0004040343628730625,
+            0.003602798795327544,
+            0.017584621906280518,
+            0.005553497467190027,
+            -0.054541099816560745,
+            0.006582245696336031,
+            0.027898261323571205,
+            0.020418038591742516,
+            0.005235282704234123,
+            -0.026398729532957077,
+            0.014629150740802288,
+            0.015823544934391975,
+            0.002135960618034005,
+            0.007035592570900917,
+            0.016538437455892563,
+            -0.011080840602517128,
+            0.00042283302173018456,
+            0.0030339357908815145,
+            0.03720930591225624,
+            0.01475120522081852,
+            -0.013190646655857563,
+            -0.025056125596165657,
+            -0.016364073380827904,
+            0.0066519915126264095,
+            -0.005915303248912096,
+            -0.0004182014672551304,
+            -0.037488289177417755,
+            -0.03598875552415848,
+            0.013521937653422356,
+            -0.01818617805838585,
+            -0.03815086930990219,
+            -0.0042741005308926105,
+            -0.03222249075770378,
+            0.005200410261750221,
+            -0.014890696853399277,
+            -0.019319545477628708,
+            -0.03926680237054825,
+            -0.0038970382884144783,
+            0.00012893410166725516,
+            0.04045247659087181,
+            0.00037842465098947287,
+            0.03476820886135101,
+            -0.007253547664731741,
+            -0.0013676686212420464,
+            -0.004143327474594116,
+            0.022876573726534843,
+            -0.03147272393107414,
+            -0.04529980197548866,
+            0.026573093608021736,
+            0.005832480266690254,
+            0.016024062409996986,
+            0.035744644701480865,
+            -0.013870666734874249,
+            -0.010200301185250282,
+            0.024864325299859047,
+            -0.016747673973441124,
+            -0.010984940454363823,
+            -0.0022231426555663347,
+            0.0025980256032198668,
+            -0.012231643311679363,
+            -0.013521937653422356,
+            -0.011656241491436958,
+            0.05157690867781639,
+            0.005575293209403753,
+            0.027845950797200203,
+            0.04038273170590401,
+            0.0010271136416122317,
+            -0.04059197008609772,
+            0.0032889433205127716,
+            -0.0091018071398139,
+            -0.009642336517572403,
+            -0.004625008441507816,
+            0.033024564385414124,
+            0.001587803359143436,
+            0.017872322350740433,
+            -0.0042174323461949825,
+            -0.016416382044553757,
+            -0.020418038591742516,
+            -0.012318825349211693,
+            -0.00761535344645381,
+            -0.004145507235080004,
+            -0.03373945876955986,
+            -0.01256293524056673,
+            0.031716834753751755,
+            0.025335106998682022,
+            -0.022388353943824768,
+            -0.004925786517560482,
+            -0.01782001368701458,
+            -0.007672021631151438,
+            0.010479284450411797,
+            -0.002959831152111292,
+            -0.00416948227211833,
+            0.009250016883015633,
+            0.018133869394659996,
+            0.01007824670523405,
+            0.00766766257584095,
+            -0.05715656280517578,
+            -0.0019398009171709418,
+            0.011351104825735092,
+            0.010400820523500443,
+            -0.003968963399529457,
+            0.014541968703269958,
+            -0.012580371461808681,
+            -0.01265011727809906,
+            -0.01975545659661293,
+            0.0326235294342041,
+            0.006355572491884232,
+            0.030426539480686188,
+            -0.0005541510181501508,
+            0.009075652807950974,
+            0.02149909734725952,
+            0.004101916216313839,
+            0.012807045131921768,
+            0.00199102028273046,
+            0.019807765260338783,
+            0.00986029114574194,
+            -0.03142041712999344,
+            0.012414725497364998,
+            0.0010429153917357326,
+            -0.0024519956205040216,
+            -0.017872322350740433,
+            0.006922255735844374,
+            0.0023757112212479115,
+            -0.013949130661785603,
+            0.0041215321980416775,
+            -0.022370917722582817,
+            0.005954534746706486,
+            0.031089123338460922,
+            0.033233802765607834,
+            0.009206426329910755,
+            -0.015169679187238216,
+            -0.010810576379299164,
+            0.002057496691122651,
+            0.02008674666285515,
+            0.025404853746294975,
+            0.025003815069794655,
+            -0.0268172025680542,
+            -0.014393758960068226,
+            -0.003729212796315551,
+            -0.001253242138773203,
+            -0.010932630859315395,
+            0.03046141378581524,
+            0.02833417057991028,
+            -0.0008914365898817778,
+            0.0028835467528551817,
+            -0.0051916916854679585,
+            0.01523070689290762,
+            0.01898825354874134,
+            -0.017340512946248055,
+            0.0017665266059339046,
+            0.022283734753727913,
+            -0.020191365852952003,
+            0.022946318611502647,
+            -0.014672741293907166,
+            -0.011429568752646446,
+            0.02013905718922615,
+            0.09366840869188309,
+            -0.005248360335826874,
+            0.01120289508253336,
+            -0.028700334951281548,
+            -0.019214926287531853,
+            -0.008474096655845642,
+            -0.0035330532118678093,
+            0.016765110194683075,
+            0.026503346860408783,
+            0.009511562995612621,
+            0.021359605714678764,
+            0.015466097742319107,
+            -0.01666049286723137,
+            0.014411195181310177,
+            -0.009694645181298256,
+            0.016198428347706795,
+            -0.01362655684351921,
+            -0.004267562180757523,
+            0.010069528594613075,
+            -0.001332795829512179,
+            0.009816700592637062,
+            0.018674397841095924,
+            0.016799984499812126,
+            0.013190646655857563,
+            -0.024864325299859047,
+            0.00949412677437067,
+            0.002284170128405094,
+            -0.007763562723994255,
+            -0.02013905718922615,
+            -0.02322530187666416,
+            0.004999891389161348,
+            0.004428848624229431,
+            0.005296310409903526,
+            0.010331074707210064,
+            -0.0032889433205127716,
+            0.03811599686741829,
+            -0.03319893032312393,
+            0.01496044173836708,
+            0.006917896680533886,
+            -0.013103464618325233,
+            0.011935224756598473,
+            0.016198428347706795,
+            0.004049607086926699,
+            0.012249079532921314,
+            -0.011569059453904629,
+            0.014245549216866493,
+            0.02052265778183937,
+            -0.029781393706798553,
+            -0.00266123260371387,
+            -0.016372792422771454,
+            -0.008474096655845642,
+            0.0026677711866796017,
+            -0.016172273084521294,
+            -0.006006844341754913,
+            -0.00546195637434721,
+            0.01169983297586441,
+            0.0024825092405080795,
+            -0.0325886569917202,
+            0.005060918629169464,
+            0.00261110276915133,
+            0.005928380414843559,
+            0.018168741837143898,
+            0.01632048189640045,
+            0.01765436865389347,
+            -0.011525468900799751,
+            -0.01975545659661293,
+            0.006285826675593853,
+            0.0045029534958302975,
+            -0.01073211245238781,
+            0.0023473771288990974,
+            0.0024912275839596987,
+            0.01588457264006138,
+            -0.011185458861291409,
+            -0.005823761690407991,
+            -0.004141148179769516,
+            0.011664960533380508,
+            0.0031516316812485456,
+            -0.014541968703269958,
+            0.003940629307180643,
+            -0.021150369197130203,
+            0.020644713193178177,
+            0.011577778495848179,
+            -0.0019932000432163477,
+            -0.02088882215321064,
+            -0.02873520739376545,
+            0.01332141924649477,
+            -0.020836513489484787,
+            0.0031952227000147104,
+            0.015771236270666122,
+            -0.011673678644001484,
+            0.013260391540825367,
+            -0.018255924805998802,
+            -0.018866198137402534,
+            -0.0114818774163723,
+            -0.00473834527656436,
+            -0.005226564593613148,
+            -0.007576121482998133,
+            0.007502016611397266,
+            -0.01823848858475685,
+            -0.0306706503033638,
+            -0.033634841442108154,
+            0.019023125991225243,
+            0.01670408435165882,
+            -0.009467972442507744,
+            -0.015823544934391975,
+            -0.004642445128411055,
+            0.02319042943418026,
+            -0.0034415118861943483,
+            0.0007110786973498762,
+            -0.0026830281130969524,
+            0.013391165062785149,
+            -0.007610994391143322,
+            0.00016441992192994803,
+            -0.02284170128405094,
+            0.00044217653339728713,
+            0.0011725988006219268,
+            0.013391165062785149,
+            -0.027357731014490128,
+            -0.003380484413355589,
+            0.0003002332232426852,
+            -0.018098996952176094,
+            -0.004476799163967371,
+            0.04254484549164772,
+            -0.013417319394648075,
+            0.008979752659797668,
+            0.021551406010985374,
+            0.010104401037096977,
+            -0.00977310910820961,
+            0.010095682926476002,
+            0.007850744761526585,
+            -0.0028857262805104256,
+            0.026538221165537834,
+            0.022911446169018745,
+            0.01914518140256405,
+            0.01670408435165882,
+            -0.022161681205034256,
+            0.012528062798082829,
+            0.021969880908727646,
+            0.03752316161990166,
+            -0.017331793904304504,
+            -0.021952442824840546,
+            0.019860073924064636,
+            0.04676445946097374,
+            0.0021032672375440598,
+            -0.0006200824282132089,
+            -0.028648026287555695,
+            -0.017201021313667297,
+            0.024689961224794388,
+            0.020557530224323273,
+            0.016224581748247147,
+            -0.010121837258338928,
+            -0.031507596373558044,
+            0.014280421659350395,
+            -0.00309278373606503,
+            0.006865587551146746,
+            -0.01996469311416149,
+            0.01688716560602188,
+            -0.015004033222794533,
+            0.04257972165942192,
+            -0.012658835388720036,
+            0.004829886369407177,
+            0.017148712649941444,
+            0.011150586418807507,
+            -0.022318609058856964,
+            0.0009263094398193061,
+            -0.01666049286723137,
+            0.027933133766055107,
+            -0.018883634358644485,
+            0.003618055721744895,
+            0.01914518140256405,
+            -0.016608182340860367,
+            -0.014533249661326408,
+            0.004191277548670769,
+            0.019459037110209465,
+            0.006107103545218706,
+            -0.025247925892472267,
+            0.018098996952176094,
+            0.0019190951716154814,
+            0.009171552956104279,
+            -0.009930036962032318,
+            0.023870449513196945,
+            -0.027636714279651642,
+            -0.015387633815407753,
+            0.025003815069794655,
+            0.0041607641614973545,
+            0.013565529137849808,
+            -0.026311546564102173,
+            0.009380790404975414,
+            -0.029049063101410866,
+            0.02336479350924492,
+            0.0032584297005087137,
+            0.014280421659350395,
+            0.0015147883677855134,
+            0.002866110298782587,
+            -0.0015115190763026476,
+            0.014446067623794079,
+            -0.000550881668459624,
+            -0.022527845576405525,
+            -0.01569277234375477,
+            -0.03696519508957863,
+            -0.022737082093954086,
+            0.023818138986825943,
+            -0.002715721260756254,
+            0.003528693923726678,
+            0.02430635876953602,
+            -0.027462350204586983,
+            -0.01435888558626175,
+            0.009572590701282024,
+            0.02549203485250473,
+            -0.043625906109809875,
+            4.093470488442108e-05,
+            0.0013829254312440753,
+            -0.019232362508773804,
+            0.010627493262290955,
+            -0.005440160632133484,
+            0.015500971116125584,
+            0.0018776836805045605,
+            -0.037662651389837265,
+            0.011586496606469154,
+            0.03762777894735336,
+            -0.017924632877111435,
+            0.0028225192800164223,
+            0.017706677317619324,
+            0.012493189424276352,
+            -0.0012390750925987959,
+            -0.0017033196054399014,
+            0.01877901703119278,
+            -0.025945382192730904,
+            0.026102310046553612,
+            0.017026657238602638,
+            -0.014881978742778301,
+            0.007842026650905609,
+            -0.008753078989684582,
+            0.020731894299387932,
+            -0.0013208083109930158,
+            -0.013155773282051086,
+            0.017375385388731956,
+            0.002195898210629821,
+            0.01349578332155943,
+            -0.001912556472234428,
+            0.004304614383727312,
+            0.03107168711721897,
+            -0.01609380915760994,
+            0.02280682697892189,
+            0.01994725689291954,
+            3.6405323044164106e-05,
+            0.015405070967972279,
+            -0.019877510145306587,
+            -0.03312918543815613,
+            0.02221398986876011,
+            -0.013591683469712734,
+            -0.007310215849429369,
+            0.023347357288002968,
+            -0.010453129187226295,
+            -0.006250953767448664,
+            -0.004282819107174873,
+            0.004077941179275513,
+            0.012406007386744022,
+            -0.0230160653591156,
+            -0.015631744638085365,
+            0.022318609058856964,
+            0.017131276428699493,
+            0.022179117426276207,
+            0.023800702765583992,
+            -0.015378915704786777,
+            0.03182145208120346,
+            0.009223862551152706,
+            0.0065735275857150555,
+            0.0076371487230062485,
+            0.008395632728934288,
+            -0.005871712230145931,
+            0.01933698169887066,
+            0.014114775694906712,
+            0.007336370646953583,
+            -0.003565746359527111,
+            -0.02029598504304886,
+            -0.011447004973888397,
+            -0.027532095089554787,
+            -0.00045825072447769344,
+            -0.013914257287979126,
+            0.00704866973683238,
+            0.07797563821077347,
+            -0.006682505365461111,
+            0.0018144766800105572,
+            -0.007868180982768536,
+            0.000739412906114012,
+            0.0016771649243310094,
+            -0.005209128372371197,
+            0.016538437455892563,
+            -0.020662149414420128,
+            -0.020749330520629883,
+            -0.006163771729916334,
+            -0.010967503301799297,
+            0.013460910879075527,
+            0.01741897687315941,
+            0.0030034221708774567,
+            -0.02277195453643799,
+            0.011141867376863956,
+            -0.016355354338884354,
+            -0.011106994934380054,
+            0.010348510928452015,
+            -0.007231752388179302,
+            -0.011656241491436958,
+            0.03295481950044632,
+            -0.009206426329910755,
+            -0.03147272393107414,
+            0.015640461817383766,
+            -0.0024541751481592655,
+            -0.005976330488920212,
+            0.00470347236841917,
+            0.00853076484054327,
+            -0.004986814223229885,
+            0.0005544234300032258,
+            0.013504501432180405,
+            -0.002068394562229514,
+            0.005414006300270557,
+            0.005318105686455965,
+            -0.013565529137849808,
+            0.017296921461820602,
+            0.022527845576405525,
+            -0.033477913588285446,
+            -0.004755781497806311,
+            -0.013940411619842052,
+            0.010479284450411797,
+            0.018709270283579826,
+            0.015535843558609486,
+            -0.022580154240131378,
+            0.0025871277321130037,
+            -0.009154116734862328,
+            -0.012702426873147488,
+            -0.02741004154086113,
+            -0.018255924805998802,
+            0.00723611144348979,
+            -0.00851768720895052,
+            0.018656961619853973,
+            0.007907412946224213,
+            -0.005745298229157925,
+            0.005012968555092812,
+            -0.005178614519536495,
+            0.015518407337367535,
+            -0.0326235294342041,
+            0.010461847297847271,
+            -0.009337198920547962,
+            0.009581308811903,
+            0.01055774837732315,
+            -0.022388353943824768,
+            -0.014759923331439495,
+            -0.023417102172970772,
+            0.016555873677134514,
+            -0.010086964815855026,
+            -0.0020934592466801405,
+            0.02008674666285515,
+            0.0009088730439543724,
+            0.015413789078593254,
+            -0.001491903094574809,
+            0.023242738097906113,
+            0.014140930958092213,
+            0.016590746119618416,
+            -0.01215317938476801,
+            -0.001640112604945898,
+            -0.011124431155622005,
+            0.012719863094389439,
+            -0.02303350158035755,
+            -0.006813278421759605,
+            -0.026015127077698708,
+            0.03598875552415848,
+            -0.007602275814861059,
+            -0.01588457264006138,
+            0.02857828140258789,
+            -0.012414725497364998,
+            -0.0037052377592772245,
+            0.0058847893960773945,
+            -0.015431225299835205,
+            0.023451974615454674,
+            0.0004955755430273712,
+            0.011499314568936825,
+            -0.02549203485250473,
+            0.00863102450966835,
+            -0.007087901700288057,
+            0.006076589692384005,
+            -0.02128986082971096,
+            -0.002423661295324564,
+            0.012135743163526058,
+            -0.010426974855363369,
+            0.00411717314273119,
+            0.021185241639614105,
+            0.005331183318048716,
+            -0.01744513027369976,
+            0.03951091319322586,
+            -0.004025631584227085,
+            0.010653648525476456,
+            0.005475033540278673,
+            -0.0009028792264871299,
+            0.016799984499812126,
+            0.01742769405245781,
+            -0.004925786517560482,
+            -0.016547156497836113,
+            0.0032235567923635244,
+            -0.015021469444036484,
+            0.01561430748552084,
+            0.016233300790190697,
+            -0.02125498652458191,
+            -0.032658401876688004,
+            0.01779385842382908,
+            0.002061855746433139,
+            0.028072625398635864,
+            0.006983283441513777,
+            -0.025997690856456757,
+            -0.005296310409903526,
+            -0.03815086930990219,
+            0.012885509058833122,
+            -0.01801181398332119,
+            -0.02437610551714897,
+            0.002785466844215989,
+            -0.00986029114574194,
+            -0.01856978051364422,
+            0.02512587048113346,
+            0.013652711175382137,
+            -0.02552690915763378,
+            0.010889039374887943,
+            0.0013011923292651772,
+            -0.017549749463796616,
+            0.010052092373371124,
+            0.004350385162979364,
+            -0.012493189424276352,
+            0.007410475518554449,
+            0.005130664445459843,
+            -0.006978924386203289,
+            -0.02890957146883011,
+            -0.01670408435165882,
+            0.01972058229148388,
+            0.010993658564984798,
+            -0.0058455574326217175,
+            0.011290077120065689,
+            -0.00997362844645977,
+            0.0033456117380410433,
+            0.0035221553407609463,
+            0.00901462510228157,
+            -0.013051155023276806,
+            -0.007371243555098772,
+            -0.007162006571888924,
+            0.01973801851272583,
+            -0.016024062409996986,
+            0.007990236394107342,
+            -0.013443474657833576,
+            -0.007981518283486366,
+            0.011429568752646446,
+            -0.0013774766121059656,
+            0.014986597001552582,
+            -0.018744144588708878,
+            0.0073407297022640705,
+            0.01189163327217102,
+            -0.008883852511644363,
+            -0.010671084746718407,
+            -0.02437610551714897,
+            0.019110308960080147,
+            0.01438504084944725,
+            0.004694754257798195,
+            -0.03679082915186882,
+            0.003951527178287506,
+            -0.010453129187226295,
+            -0.0009775288635864854,
+            -0.008796670474112034,
+            0.019249800592660904,
+            0.018831325694918633,
+            0.007985876873135567,
+            0.013277828693389893,
+            -0.0023081451654434204,
+            0.013548092916607857,
+            -0.012414725497364998,
+            0.017593340948224068,
+            -0.004036529455333948,
+            -0.017575904726982117,
+            -0.005714784376323223,
+            0.019406726583838463,
+            -0.001491903094574809,
+            0.007903054356575012,
+            0.004372180439531803,
+            0.010226456448435783,
+            -0.012275234796106815,
+            -0.007493298500776291,
+            0.014045030809938908,
+            0.008378196507692337,
+            -0.0004288267809897661,
+            0.009250016883015633,
+            0.06758353859186172,
+            -0.00870512891560793,
+            0.023853013291954994,
+            0.006311981473118067,
+            -0.009345917031168938,
+            -0.008112290874123573,
+            0.0019932000432163477,
+            -0.02951984666287899,
+            0.023329921066761017,
+            0.011970097199082375,
+            -0.009180271066725254,
+            0.00036126066697761416,
+            -0.006726096384227276,
+            0.018325669690966606,
+            -0.009694645181298256,
+            0.016948193311691284,
+            -0.010052092373371124,
+            0.02435866929590702,
+            0.019162617623806,
+            0.005640679504722357,
+            0.005536061245948076,
+            0.0028203397523611784,
+            0.019459037110209465,
+            0.012868072837591171,
+            -0.002165384590625763,
+            -0.003123297356069088,
+            -0.009877728298306465,
+            0.019389290362596512,
+            -0.001910376944579184,
+            0.030792705714702606,
+            -0.0022144243121147156,
+            -0.005182973574846983,
+            -0.009842854924499989,
+            0.0041607641614973545,
+            0.011569059453904629,
+            0.009520281106233597,
+            0.012876790948212147,
+            -0.0013567708665505052,
+            -0.010679802857339382,
+            0.011909069493412971,
+            0.00863102450966835,
+            0.005326824262738228,
+            -0.0032518908847123384,
+            0.0045247492380440235,
+            -0.008404350839555264,
+            0.014001439325511456,
+            -0.0024781501851975918,
+            0.0015354941133409739,
+            -0.008774874731898308,
+            0.004426669329404831,
+            -0.01312961895018816,
+            0.0005348074482753873,
+            0.009607463143765926,
+            0.012868072837591171,
+            0.0015213270671665668,
+            -0.002273272257298231,
+            0.015405070967972279,
+            0.0025980256032198668,
+            0.0073014977388083935,
+            0.0009486498311161995,
+            -0.009537717327475548,
+            0.019057998433709145,
+            -0.01033979281783104,
+            -0.0019212746992707253,
+            0.010705957189202309,
+            0.006067871581763029,
+            0.008260500617325306,
+            0.01496916078031063,
+            0.009354635141789913,
+            0.003968963399529457,
+            0.0013545913388952613,
+            -0.00565375667065382,
+            -0.003535232739523053,
+            0.0036267738323658705,
+            0.0016771649243310094,
+            0.01646869257092476,
+            -0.019459037110209465,
+            -0.007039951626211405,
+            0.010679802857339382,
+            0.011438286863267422,
+            0.0017839629435911775,
+            0.015684053301811218,
+            0.014428631402552128,
+            0.01994725689291954,
+            -0.022562718018889427,
+            0.003831651760265231,
+            -0.002920599188655615,
+            0.019057998433709145,
+            0.008770515210926533,
+            -0.024114558473229408,
+            0.0003645300166681409,
+            -0.01063621137291193,
+            0.012676272541284561,
+            -0.0067086596973240376,
+            0.0023473771288990974,
+            0.01665177382528782,
+            -0.010121837258338928,
+            -0.009930036962032318,
+            -0.019424164667725563,
+            -0.016983065754175186,
+            0.010976221412420273,
+            0.014437349513173103,
+            0.0014188881032168865,
+            0.019232362508773804,
+            -0.0012412546202540398,
+            -0.008474096655845642,
+            0.030827578157186508,
+            0.016765110194683075,
+            -0.0004157494695391506,
+            -0.008962316438555717,
+            -0.0006544103380292654,
+            -0.015082497149705887,
+            0.01074954867362976,
+            0.004426669329404831,
+            0.005108869168907404,
+            0.003387023229151964,
+            9.174550359603018e-05,
+            -0.016712801530957222,
+            -0.014620431698858738,
+            -0.004668599460273981,
+            0.011045967228710651,
+            -0.003751008305698633,
+            0.009197707287967205,
+            -0.01609380915760994,
+            0.003121117828413844,
+            -0.008173318579792976,
+            -0.010714675299823284,
+            -0.008356400765478611,
+            -0.039406292140483856,
+            -0.02067958563566208,
+            0.0003173972072545439,
+            -0.00473834527656436,
+            -0.002358274767175317,
+            0.009223862551152706,
+            -0.003672544378787279,
+            0.013670147396624088,
+            -0.0015551100950688124,
+            0.0047906544059515,
+            0.008657178841531277,
+            0.01561430748552084,
+            -0.013504501432180405,
+            -0.0006631285650655627,
+            -0.014071185141801834,
+            -0.026782330125570297,
+            0.01707896590232849,
+            0.003199581755325198,
+            0.014446067623794079,
+            -0.015187115408480167,
+            0.009528999216854572,
+            -0.012589089572429657,
+            0.014463504776358604,
+            -0.0421612448990345,
+            -0.009520281106233597,
+            0.007031233515590429,
+            -0.0075979167595505714,
+            -0.005496829282492399,
+            -0.025369981303811073,
+            -0.0012205488746985793,
+            -0.011054685339331627,
+            -0.01666049286723137,
+            -0.019459037110209465,
+            0.00574093870818615,
+            -0.009311044588685036,
+            0.008591792546212673,
+            -0.006120180711150169,
+            0.02761927805840969,
+            0.00532246520742774,
+            -0.001346962875686586,
+            -0.006128899287432432,
+            0.009206426329910755,
+            -0.0007503106608055532,
+            0.012885509058833122,
+            0.008648460730910301,
+            0.005823761690407991,
+            0.023434538394212723,
+            0.0009339378448203206,
+            0.004690395202487707,
+            -0.006141976453363895,
+            0.014541968703269958,
+            0.024689961224794388,
+            0.011342386715114117,
+            0.011455723084509373,
+            0.005967612378299236,
+            0.0005950175691395998,
+            0.011429568752646446,
+            0.017392821609973907,
+            -0.003990759141743183,
+            -0.009171552956104279,
+            0.004869118332862854,
+            -0.01761949434876442,
+            -0.0033347138669341803,
+            0.012885509058833122,
+            -0.006128899287432432,
+            -0.02706131339073181,
+            0.01285063661634922,
+            0.004317691549658775,
+            -0.006673786789178848,
+            -0.016948193311691284,
+            -0.00661711860448122,
+            0.004646804183721542,
+            0.004729627165943384,
+            0.024236613884568214,
+            -0.022144244983792305,
+            -0.008644101209938526,
+            0.011917788535356522,
+            -0.014446067623794079,
+            -0.01063621137291193,
+            0.009790545329451561,
+            0.029484974220395088,
+            0.003496000776067376,
+            -0.05307644233107567,
+            -0.012467035092413425,
+            0.014236831106245518,
+            -0.01503018755465746,
+            -0.025980254635214806,
+            -0.0014428631402552128,
+            0.0038578063249588013,
+            -0.014010157436132431,
+            0.006625836715102196,
+            0.0011606112821027637,
+            -0.011708551086485386,
+            0.017139993607997894,
+            0.028874699026346207,
+            0.01312961895018816,
+            0.0114818774163723,
+            -0.025213053449988365,
+            -0.0008952508214861155,
+            -0.01914518140256405,
+            0.011752142570912838,
+            -0.030566031113266945,
+            -0.013984003104269505,
+            -0.0023342997301369905,
+            0.004141148179769516,
+            0.027444913983345032,
+            0.014376322738826275,
+            0.025962818413972855,
+            0.006778405513614416,
+            0.0037989583797752857,
+            -0.02125498652458191,
+            -0.018447725102305412,
+            0.0033456117380410433,
+            -0.0008304091752506793,
+            -0.002329940674826503,
+            -0.008216909132897854,
+            0.012275234796106815,
+            0.012039843015372753,
+            0.026172054931521416,
+            0.005087073426693678,
+            0.005553497467190027,
+            -2.538836997700855e-05,
+            -0.00494322320446372,
+            0.007593557704240084,
+            -0.019232362508773804,
+            -0.0029925242997705936,
+            -0.004559622146189213,
+            0.015405070967972279,
+            0.0045029534958302975,
+            0.0022275017108768225,
+            0.017357949167490005,
+            -0.0030099607538431883,
+            0.003729212796315551,
+            -0.019982129335403442,
+            -0.014672741293907166,
+            -0.0032518908847123384,
+            0.03525642678141594,
+            0.003971143160015345,
+            -0.02761927805840969,
+            -0.016224581748247147,
+            -0.0153353251516819,
+            0.030792705714702606,
+            -0.009581308811903,
+            0.004315512254834175,
+            0.02683463878929615,
+            0.01593688130378723,
+            0.012388571165502071,
+            0.006399163510650396,
+            -0.01822105050086975,
+            0.005021686665713787,
+            0.02931061014533043,
+            -0.0030317562632262707,
+            -0.010086964815855026,
+            -0.0066912234760820866,
+            0.007144570350646973,
+            -0.007798435632139444,
+            0.007632789667695761,
+            -0.0035068984143435955,
+            -0.013417319394648075,
+            0.029554719105362892,
+            -0.00968592707067728,
+            0.022370917722582817,
+            -0.010130555368959904,
+            0.00268738716840744,
+            -0.038848329335451126,
+            -0.002004097681492567,
+            0.0011453543556854129,
+            0.0019692247733473778,
+            0.004555262625217438,
+            -0.01881388947367668,
+            -0.008753078989684582,
+            0.011839324608445168,
+            -0.01276345457881689,
+            -0.007885618135333061,
+            -0.006137617398053408,
+            0.002983806189149618,
+            0.01977289281785488,
+            -0.00947669055312872,
+            0.0009857022669166327,
+            -0.0210980586707592,
+            0.01256293524056673,
+            -0.00282687833532691,
+            0.0017567186150699854,
+            0.010453129187226295,
+            0.007009437773376703,
+            0.03502975404262543,
+            0.01094134896993637,
+            0.005531701724976301,
+            -0.0032780456822365522,
+            -0.014053748920559883,
+            -0.03218761831521988,
+            0.013295264914631844,
+            -0.010453129187226295,
+            0.023800702765583992,
+            0.0036703648511320353,
+            -0.008103572763502598,
+            -0.0051960512064397335,
+            0.0037139558698982,
+            0.009590026922523975,
+            -0.0041106343269348145,
+            0.013478347100317478,
+            -0.010243892669677734,
+            0.021167805418372154,
+            0.0031298361718654633,
+            -0.005121946334838867,
+            0.012571653351187706,
+            -0.017689241096377373,
+            0.01055774837732315,
+            -0.01744513027369976,
+            0.0028225192800164223,
+            0.01360040158033371,
+            -0.010296201333403587,
+            0.0056276023387908936,
+            -0.052588220685720444,
+            -0.001197663601487875,
+            -0.016521001234650612,
+            -0.00675660977140069,
+            -0.008456660434603691,
+            -0.021394478157162666,
+            -0.03319893032312393,
+            -0.0036660057958215475,
+            0.015370197594165802,
+            0.006551732309162617,
+            0.023260174319148064,
+            0.0009895163821056485,
+            0.012257798574864864,
+            0.012807045131921768,
+            -0.0047906544059515,
+            0.012519344687461853,
+            0.033861514180898666,
+            -0.027741333469748497,
+            -0.019685709848999977,
+            0.005257078446447849,
+            -0.00028552126605063677,
+            -0.015788672491908073,
+            -0.013103464618325233,
+            -0.0017665266059339046,
+            0.0024955866392701864,
+            0.030443977564573288,
+            -0.020191365852952003,
+            -0.005579652264714241,
+            -0.007053028792142868,
+            0.004433208145201206,
+            -0.01918005384504795,
+            -0.0031450928654521704,
+            -0.011708551086485386,
+            -0.012780890800058842,
+            0.016041500493884087,
+            -0.011089558713138103,
+            0.011159304529428482,
+            -0.019511345773935318,
+            0.008626664988696575,
+            -0.015378915704786777,
+            -0.010627493262290955,
+            -0.0031690681353211403,
+            -0.010461847297847271,
+            0.030164994299411774,
+            -0.0041890982538461685,
+            0.0048386044800281525,
+            0.01670408435165882,
+            -0.00853076484054327,
+            0.011603932827711105,
+            -0.0006124540232121944,
+            0.007898694835603237,
+            -0.002704823622480035,
+            -0.0086135882884264,
+            0.0015278656501322985,
+            0.009537717327475548,
+            -0.010950067080557346,
+            0.03846472501754761,
+            -0.013731175102293491,
+            -0.008879492990672588,
+            -0.003859985852614045,
+            -0.004511671606451273,
+            0.002425840822979808,
+            0.0021762822289019823,
+            0.010984940454363823,
+            -0.0012870251666754484,
+            0.006835073698312044,
+            -0.024219177663326263,
+            -0.011717269197106361,
+            0.0013164491392672062,
+            0.006468909326940775,
+            0.003321636701002717,
+            0.008016390725970268,
+            -0.025579217821359634,
+            -0.0020782025530934334,
+            -0.012859354726970196,
+            0.005475033540278673,
+            -0.011499314568936825,
+            0.007332011591643095,
+            -0.01215317938476801,
+            -0.0013295264216139913,
+            0.009337198920547962,
+            0.012946536764502525,
+            0.013861947692930698,
+            -0.006608400493860245,
+            0.026573093608021736,
+            0.014899414964020252,
+            0.008709488436579704,
+            0.005405288189649582,
+            -0.033216364681720734,
+            -0.007802794687449932,
+            0.010365947149693966,
+            0.007519452832639217,
+            0.009232580661773682,
+            -0.017689241096377373,
+            -0.004176021087914705,
+            0.0029402151703834534,
+            -0.007039951626211405,
+            4.651980270864442e-05,
+            0.011080840602517128,
+            -0.010627493262290955,
+            0.005505547393113375,
+            -0.00031521765049546957,
+            0.0027549532242119312,
+            -0.006460190750658512,
+            0.012188052758574486,
+            -0.013286546804010868,
+            -0.008168959058821201,
+            0.0211154967546463,
+            0.02760184183716774,
+            -0.005531701724976301,
+            0.023696085438132286,
+            -0.021917570382356644,
+            0.008487174287438393,
+            -0.005043482407927513,
+            -0.014149649068713188,
+            -0.015300451777875423,
+            0.006883023772388697,
+            0.014777359552681446,
+            0.004760140553116798,
+            -0.013155773282051086,
+            -0.003027397207915783,
+            0.016983065754175186,
+            -0.0010952246375381947,
+            0.006216081324964762,
+            0.01593688130378723,
+            -0.00026971951592713594,
+            0.00017667989595793188,
+            -0.006002485286444426,
+            -0.01876158080995083,
+            -0.006290185730904341,
+            0.017924632877111435,
+            -0.0073799616657197475,
+            0.014262985438108444,
+            0.02320786565542221,
+            0.0005429807933978736,
+            0.011351104825735092,
+            -0.0038469084538519382,
+            0.01351321954280138,
+            0.01505634281784296,
+            -0.005235282704234123,
+            -0.0030600903555750847,
+            -0.0036965194158256054,
+            0.00011858122888952494,
+            0.012022406794130802,
+            -0.01741897687315941,
+            -0.011333668604493141,
+            0.0114818774163723,
+            -0.002121793571859598,
+            -0.008404350839555264,
+            -0.023486848920583725,
+            0.0016313943779096007,
+            0.0077156126499176025,
+            0.011447004973888397,
+            -0.016425101086497307,
+            -0.0005239096935838461,
+            0.000875089957844466,
+            0.010801857337355614,
+            -0.007497657556086779,
+            0.01159521471709013,
+            -0.007907412946224213,
+            -0.00863102450966835,
+            0.006381727289408445,
+            0.0015507509233430028,
+            0.007414834573864937,
+            -0.009999782778322697,
+            0.0015987011138349771,
+            -0.008025108836591244,
+            0.026590529829263687,
+            0.008317168802022934,
+            -0.0013011923292651772,
+            -0.005640679504722357,
+            0.004995532333850861,
+            0.008421787060797215,
+            -0.0100608104839921,
+            0.012815763242542744,
+            0.006107103545218706,
+            -0.002933676354587078,
+            0.010592620819807053,
+            0.015283015556633472,
+            0.0025631526950746775,
+            0.009328480809926987,
+            0.003443691413849592,
+            0.0059022256173193455,
+            -0.007022515404969454,
+            0.0017861425876617432,
+            -0.01550968922674656,
+            -0.020801641047000885,
+            -0.0035831828135997057,
+            0.016991784796118736,
+            -0.010583902709186077,
+            -0.001182406791485846,
+            0.017924632877111435,
+            0.01371373888105154,
+            -0.005034764297306538,
+            0.00551426550373435,
+            0.002217693720012903,
+            -0.0015529305674135685,
+            0.003323816228657961,
+            -0.014280421659350395,
+            -0.012493189424276352,
+            0.008726924657821655,
+            -0.02087138593196869,
+            0.01952878199517727,
+            -0.02165602520108223,
+            -0.035361044108867645,
+            -0.008831542916595936,
+            -0.009197707287967205,
+            -0.01707896590232849,
+            -0.013574247248470783,
+            0.023661212995648384,
+            0.016207145527005196,
+            -0.009354635141789913,
+            -0.034384604543447495,
+            0.03155990689992905,
+            0.00918898917734623,
+            0.027933133766055107,
+            -0.015544561669230461,
+            0.004343846347182989,
+            -0.0015213270671665668,
+            0.014707613736391068,
+            -0.006163771729916334,
+            -0.006686864420771599,
+            0.0091018071398139,
+            0.014620431698858738,
+            0.010897758416831493,
+            0.003245352301746607,
+            -0.003160349791869521,
+            0.007279702462255955,
+            0.002530459314584732,
+            0.0022285915911197662,
+            -0.008583074435591698,
+            -0.02282426506280899,
+            -0.009781827218830585,
+            0.017942069098353386,
+            0.022266298532485962,
+            0.012746017426252365,
+            -0.017131276428699493,
+            0.0020520477555692196,
+            -0.00484296353533864,
+            -0.014053748920559883,
+            -0.009991064667701721,
+            0.009642336517572403,
+            -0.033617403358221054,
+            0.02317299321293831,
+            0.004760140553116798,
+            -0.00627274950966239,
+            -0.0015006213216111064,
+            0.003836010815575719,
+            0.005679911468178034,
+            -0.004313332494348288,
+            0.0002961465797852725,
+            -0.002532638842239976,
+            0.010897758416831493,
+            0.016599465161561966,
+            0.005100150592625141,
+            -0.010400820523500443,
+            0.012388571165502071,
+            -0.0031407338101416826,
+            -0.002761491807177663,
+            -0.013914257287979126,
+            -0.006011203397065401,
+            -0.009502844884991646,
+            0.010261328890919685,
+            0.01483838725835085,
+            0.0054924702271819115,
+            0.006791482679545879,
+            0.004335128236562014,
+            -0.00013084120291750878,
+            -0.0002569146454334259,
+            0.008469737134873867,
+            -0.0005070181796327233,
+            -0.0045639812014997005,
+            0.0010821473551914096,
+            -0.0014679280575364828,
+            0.02320786565542221,
+            -0.00532246520742774,
+            -0.010950067080557346,
+            0.007196879480034113,
+            0.02033085748553276,
+            -0.022370917722582817,
+            0.022091934457421303,
+            -0.022179117426276207,
+            0.023853013291954994,
+            0.03482051566243172,
+            0.013975284993648529,
+            -0.011717269197106361,
+            -0.007096619810909033,
+            0.012815763242542744,
+            -0.012510626576840878,
+            -0.0009600924677215517,
+            0.0018079380970448256,
+            -0.022580154240131378,
+            0.015178397297859192,
+            -0.023347357288002968,
+            0.014280421659350395,
+            0.005121946334838867,
+            -0.00852640625089407,
+            -0.0073232934810221195,
+            -0.0009764390997588634,
+            0.018133869394659996,
+            -0.003990759141743183,
+            -0.0006255313055589795,
+            -0.03028704971075058,
+            -0.006930973846465349,
+            0.02163858897984028,
+            -0.0045247492380440235,
+            -0.013949130661785603,
+            0.007253547664731741,
+            -8.973622243502177e-06,
+            -0.016154836863279343,
+            -0.005876071285456419,
+            -0.03166452422738075,
+            -0.008142804726958275,
+            -0.02264990098774433,
+            -0.0075979167595505714,
+            -0.011577778495848179,
+            -0.012292671017348766,
+            -0.0038512677419930696,
+            -0.016276890411973,
+            0.001949608908034861,
+            -0.001063076313585043,
+            0.0033020207192748785,
+            0.0027091826777905226,
+            -0.009563872590661049,
+            -0.003968963399529457,
+            0.021324733272194862,
+            0.010705957189202309,
+            -0.016041500493884087,
+            -0.026206929236650467,
+            0.011638805270195007,
+            -0.012205488979816437,
+            -0.007593557704240084,
+            0.001547481631860137,
+            -0.005915303248912096,
+            -0.019267236813902855,
+            -0.01860465295612812,
+            0.005967612378299236,
+            0.0003718859807122499,
+            0.0051175872795283794,
+            0.013582965359091759,
+            0.014341449365019798,
+            -0.0019180054077878594,
+            -0.006734814494848251,
+            -1.330718441749923e-05,
+            -0.002195898210629821,
+            0.014027593657374382,
+            0.004348205402493477,
+            0.024445850402116776,
+            -0.008744360879063606,
+            0.0028944446239620447,
+            -0.017924632877111435,
+            0.00449859444051981,
+            -0.007393038831651211,
+            -0.005679911468178034,
+            -0.004064863547682762,
+            -0.011089558713138103,
+            0.005993766710162163,
+            -0.0011126610916107893,
+            -0.00247161160223186,
+            -0.007911772467195988,
+            -0.0020912797190248966,
+            -0.01707024872303009,
+            -0.0029794471338391304,
+            -0.020557530224323273,
+            -0.0013905538944527507,
+            -0.014916851185262203,
+            0.017000501975417137,
+            0.026172054931521416,
+            -0.00513502350077033,
+            -0.0006091846735216677,
+            -0.011229049414396286,
+            -4.08665910072159e-05,
+            -0.011909069493412971,
+            -0.00637736776843667,
+            -0.016416382044553757,
+            -0.0325189083814621,
+            -0.016904601827263832,
+            0.009049498476088047,
+            0.024288922548294067,
+            0.0019398009171709418,
+            -0.024986378848552704,
+            0.0029511128086596727,
+            -0.026607966050505638,
+            -0.014228112995624542,
+            -0.01496916078031063,
+            -0.006529936566948891,
+            -0.009136680513620377,
+            0.017392821609973907,
+            -0.0026350778061896563,
+            0.01571020856499672,
+            0.01747128553688526,
+            0.010906476527452469,
+            0.007497657556086779,
+            0.010880321264266968,
+            -0.009528999216854572,
+            -0.00016932391736190766,
+            0.009869009256362915,
+            -0.006311981473118067,
+            0.011839324608445168,
+            -0.00815152283757925,
+            -0.0011933045461773872,
+            0.01879645325243473,
+            0.008016390725970268,
+            -0.006255313288420439,
+            -0.0026699507143348455,
+            -0.02742747776210308,
+            0.017506157979369164,
+            -0.018552342429757118,
+            -0.00258494820445776,
+            -0.002944574225693941,
+            -0.016738956794142723,
+            0.02319042943418026,
+            0.017323076725006104,
+            -0.005021686665713787,
+            -0.0027549532242119312,
+            0.017889760434627533,
+            -0.006451472640037537,
+            0.00398204056546092,
+            0.0025762300938367844,
+            -0.0033281752839684486,
+            0.009790545329451561,
+            0.0027636713348329067,
+            -0.004577058367431164,
+            -0.0025587936397641897,
+            0.012196770869195461,
+            0.003901397343724966,
+            0.013530656695365906,
+            -0.027131058275699615,
+            -0.010566466487944126,
+            -0.0010303830495104194,
+            0.01935441792011261,
+            -0.005405288189649582,
+            -0.0007906323298811913,
+            -0.013312701135873795,
+            -0.0008854428306221962,
+            -0.005719143431633711,
+            0.023818138986825943,
+            -0.01992982067167759,
+            0.006120180711150169,
+            -0.008395632728934288,
+            0.013861947692930698,
+            -0.0032191977370530367,
+            0.010801857337355614,
+            -0.0007955363253131509,
+            -1.3375295566220302e-05,
+            0.009964910335838795,
+            0.012981409206986427,
+            -0.0022024367935955524,
+            0.024027377367019653,
+            -0.0004737800336442888,
+            -0.006403522565960884,
+            0.0027745692059397697,
+            -0.01895338110625744,
+            0.0229637548327446,
+            0.003402280155569315,
+            0.004315512254834175,
+            -0.0058629936538636684,
+            -0.008199472911655903,
+            0.039789892733097076,
+            0.021516533568501472,
+            0.010392102412879467,
+            -0.000592293159570545,
+            -0.01494300551712513,
+            0.028229551389813423,
+            0.007519452832639217,
+            0.00930232647806406,
+            -0.004012554418295622,
+            0.0027113622054457664,
+            -0.015178397297859192,
+            -0.010976221412420273,
+            0.019807765260338783,
+            -0.019999565556645393,
+            -0.008731283247470856,
+            -0.0026328982785344124,
+            -0.019703146070241928,
+            -0.029293173924088478,
+            0.015474816784262657,
+            0.009764390997588634,
+            0.0028443147893995047,
+            0.008495892398059368,
+            0.0037335718516260386,
+            -0.0015028008492663503,
+            -0.011647523380815983,
+            -0.003986400086432695,
+            -0.008574356324970722,
+            -0.01044441107660532,
+            0.006848151329904795,
+            -0.012004970572888851,
+            0.006717377807945013,
+            -0.008256141096353531,
+            -0.00024533577379770577,
+            0.01285063661634922,
+            0.005091432482004166,
+            0.0062771085649728775,
+            0.003443691413849592,
+            0.005078355316072702,
+            0.01591944508254528,
+            -0.013757329434156418,
+            0.016346637159585953,
+            0.017113838344812393,
+            0.0062945447862148285,
+            -0.0034131777938455343,
+            0.0023103246930986643,
+            -0.029188554733991623,
+            -0.00022381270537152886,
+            -0.0016869729151949286,
+            0.012632681056857109,
+            -0.02163858897984028,
+            0.004733986221253872,
+            0.021568842232227325,
+            -0.0016172273317351937,
+            0.009816700592637062,
+            0.014864541590213776,
+            -0.0001531135058030486,
+            0.028037751093506813,
+            0.011342386715114117,
+            0.02416686899960041,
+            -0.009459254331886768,
+            -0.02852597087621689,
+            0.006368649657815695,
+            -0.0038904997054487467,
+            0.014376322738826275,
+            0.0091018071398139,
+            0.0201216209679842,
+            -0.01369630265980959,
+            -0.006695582531392574,
+            0.007789717521518469,
+            0.0018918508430942893,
+            0.014306576922535896,
+            0.003217018209397793,
+            0.019302109256386757,
+            0.009119244292378426,
+            -0.0008625575574114919,
+            -0.007737408392131329,
+            8.588795753894374e-05,
+            0.006250953767448664,
+            -0.003500359831377864,
+            -0.03047885000705719,
+            0.0016390228411182761,
+            -0.004834245424717665,
+            0.004978095646947622,
+            -0.014245549216866493,
+            0.010409538634121418,
+            -0.0010015039006248116,
+            0.022196553647518158,
+            -0.0041498662903904915,
+            0.004064863547682762,
+            0.01708768494427204,
+            0.00938950851559639,
+            0.004991173278540373,
+            0.015300451777875423,
+            0.005658115725964308,
+            0.004511671606451273,
+            -0.008038186468183994,
+            -0.019493909552693367,
+            0.019162617623806,
+            -0.027462350204586983,
+            0.010653648525476456,
+            -0.00873564276844263,
+            0.004317691549658775,
+            0.003925372380763292,
+            -0.006887382827699184,
+            -0.002567511750385165,
+            0.003607157850638032,
+            0.001491903094574809,
+            -0.059144310653209686,
+            0.0037706242874264717,
+            0.0007388679659925401,
+            0.020452912896871567,
+            -0.00881846621632576,
+            0.0007590288296341896,
+            0.0026394370943307877,
+            0.00390357687138021,
+            -0.013347573578357697,
+            -0.0023604542948305607,
+            0.01571892574429512,
+            -0.01858721673488617,
+            -0.0059632533229887486,
+            0.004038709215819836,
+            -0.005557856522500515,
+            0.0035940806847065687,
+            -0.008779234252870083,
+            -0.002622000640258193,
+            0.02838647924363613,
+            -0.001655369414947927,
+            0.03140297904610634,
+            0.002506484277546406,
+            -0.0004756871494464576,
+            -0.01666049286723137,
+            0.007410475518554449,
+            -0.016738956794142723,
+            -0.00425884360447526,
+            0.008927443064749241,
+            -0.018726706504821777,
+            -0.00123580580111593,
+            0.001693511614575982,
+            -0.0013382446486502886,
+            -0.0077722808346152306,
+            -0.001763257198035717,
+            0.0010429153917357326,
+            -0.02048778533935547,
+            0.015997909009456635,
+            -0.023853013291954994,
+            0.01625945419073105,
+            -0.0034371528308838606,
+            -0.002946753753349185,
+            -0.0013033718569204211,
+            -0.0038098562508821487,
+            0.005666834302246571,
+            0.02531767077744007,
+            -0.015448661521077156,
+            -0.00522220553830266,
+            0.008365118876099586,
+            -0.003567925887182355,
+            0.002730978187173605,
+            -0.008761797100305557,
+            0.006656350567936897,
+            -0.0325886569917202,
+            -0.008112290874123573,
+            0.004413592163473368,
+            0.0031908636447042227,
+            -0.001236895564943552,
+            -0.02758440561592579,
+            0.003960245288908482,
+            -0.00854384247213602,
+            -0.024201741442084312,
+            -0.00027380615938454866,
+            -0.020592402666807175,
+            0.006717377807945013,
+            -0.00881846621632576,
+            0.01761949434876442,
+            -0.014568123035132885,
+            0.011098276823759079,
+            -0.014367603696882725,
+            0.016006626188755035,
+            -0.007406116463243961,
+            -0.009398226626217365,
+            -0.0069266147911548615,
+            -0.006338136270642281,
+            0.01044441107660532,
+            -0.014916851185262203,
+            -0.009511562995612621,
+            -0.007641507778316736,
+            0.005161178298294544,
+            0.011045967228710651,
+            0.013574247248470783,
+            0.0011333668371662498,
+            -0.00880538858473301,
+            -0.016547156497836113,
+            -0.005392210558056831,
+            -0.014881978742778301,
+            -0.0009301236714236438,
+            0.0042174323461949825,
+            0.020853949710726738,
+            0.008160240948200226,
+            -0.003753187833353877,
+            0.011586496606469154,
+            0.020801641047000885,
+            0.014132212847471237,
+            -0.014777359552681446,
+            0.0021697436459362507,
+            -0.010095682926476002,
+            0.013417319394648075,
+            0.022144244983792305,
+            0.002098908182233572,
+            0.002796364715322852,
+            0.006085308268666267,
+            0.007951004430651665,
+            -0.004396155476570129,
+            0.008291014470160007,
+            -0.025457162410020828,
+            -0.012667553499341011,
+            -0.006569168530404568,
+            0.005387851502746344,
+            -0.015378915704786777,
+            0.029973194003105164,
+            -0.003609337378293276,
+            0.004472440108656883,
+            -0.002182820811867714,
+            0.0326409637928009,
+            0.007196879480034113,
+            0.00473834527656436,
+            0.002874828642234206,
+            9.467427298659459e-05,
+            0.009136680513620377,
+            -0.01761077716946602,
+            -0.03888320177793503,
+            0.003254070645198226,
+            -0.000592293159570545,
+            0.01766308583319187,
+            -0.01514352485537529,
+            0.006878664717078209,
+            -0.0036899808328598738,
+            -0.01282448135316372,
+            -0.009633618406951427,
+            0.03138554468750954,
+            -0.005466315429657698,
+            0.03387894853949547,
+            -0.007558684796094894,
+            -0.010104401037096977,
+            0.021725770086050034,
+            0.021900134161114693,
+            -0.014210676774382591,
+            0.005531701724976301,
+            0.015762517228722572,
+            0.020278548821806908,
+            0.005209128372371197,
+            -0.00806434080004692,
+            0.02627667412161827,
+            0.012946536764502525,
+            -0.012807045131921768,
+            -0.0026350778061896563,
+            -0.009572590701282024,
+            0.006726096384227276,
+            -0.012301389127969742,
+            0.005387851502746344,
+            0.006434036418795586,
+            0.004986814223229885,
+            0.012963972985744476,
+            -0.01879645325243473,
+            -0.003831651760265231,
+            0.0073014977388083935,
+            0.013966566883027554,
+            0.00651250034570694,
+            0.017558468505740166,
+            -0.023643774911761284,
+            0.0028966241516172886,
+            -0.02493407018482685,
+            -0.008103572763502598,
+            0.0029968833550810814,
+            0.001748000388033688,
+            0.008875134401023388,
+            -0.01592816226184368,
+            -0.029484974220395088,
+            0.011516750790178776,
+            0.01708768494427204,
+            0.02568383514881134,
+            0.0011126610916107893,
+            0.008914366364479065,
+            0.00560144754126668,
+            -0.013739893212914467,
+            -0.00805998221039772,
+            -0.00997362844645977,
+            -0.01073211245238781,
+            -0.011159304529428482,
+            -0.002761491807177663,
+            0.009119244292378426,
+            -0.010592620819807053,
+            0.0006821996648795903,
+            0.016346637159585953,
+            -0.011717269197106361,
+            -0.0051960512064397335,
+            0.015535843558609486,
+            -0.005008609499782324,
+            0.0007977159111760557,
+            0.0016891524428501725,
+            0.013086027465760708,
+            0.0034480507019907236,
+            -0.0063730087131261826,
+            -0.01591944508254528,
+            0.01215317938476801,
+            -0.00417166156694293,
+            0.0016074193408712745,
+            0.00708354264497757,
+            -0.007933568209409714,
+            -0.0047819362953305244,
+            0.013373728841543198,
+            -0.00854384247213602,
+            0.0016259454423561692,
+            -0.007606634870171547,
+            -0.017549749463796616,
+            -0.0076938169077038765,
+            0.006394804455339909,
+            -0.026730021461844444,
+            -0.013478347100317478,
+            0.016233300790190697,
+            -0.003570105414837599,
+            0.001546391868032515,
+            -0.0005138292908668518,
+            -0.011124431155622005,
+            0.003160349791869521,
+            -0.019598528742790222,
+            -0.01064493041485548,
+            -0.004864759277552366,
+            -0.02662540227174759,
+            0.017549749463796616,
+            0.015091215260326862,
+            0.0033172776456922293,
+            -0.00699636060744524,
+            -0.005592729430645704,
+            0.002567511750385165,
+            0.010348510928452015,
+            0.01763693243265152,
+            0.009311044588685036,
+            -0.005592729430645704,
+            -0.019685709848999977,
+            -0.003443691413849592,
+            0.021324733272194862,
+            -6.375188240781426e-05,
+            -0.016590746119618416,
+            0.020609840750694275,
+            0.007109697442501783,
+            0.014306576922535896,
+            0.0055883703753352165,
+            0.0023451976012438536,
+            0.005383492447435856,
+            0.014114775694906712,
+            -0.0001486181718064472,
+            -0.01783744990825653,
+            -0.013984003104269505,
+            0.0004890369018539786,
+            0.03140297904610634,
+            0.00708354264497757,
+            0.017139993607997894,
+            -0.03623286634683609,
+            0.0006642183288931847,
+            -0.005335542373359203,
+            -0.0038490882143378258,
+            -0.0045988536439836025,
+            0.027671586722135544,
+            0.010766984894871712,
+            -0.0001184450084110722,
+            0.015605589374899864,
+            -0.010043373331427574,
+            0.0422309935092926,
+            -0.015378915704786777,
+            -0.00737560261040926,
+            -0.010052092373371124,
+            -0.01685229316353798,
+            -0.003940629307180643,
+            -0.0011769578559324145,
+            -0.01689588464796543,
+            -0.006647632457315922,
+            -0.006320699583739042,
+            0.021411914378404617,
+            0.0046380856074392796,
+            -0.01665177382528782,
+            0.0017065888969227672,
+            -0.023835577070713043,
+            -0.0010946798138320446,
+            -0.022475536912679672,
+            -0.006403522565960884,
+            -0.012327543459832668,
+            -0.01349578332155943,
+            -0.020993441343307495,
+            -0.02031342126429081,
+            -0.004908350296318531,
+            0.016547156497836113,
+            0.00532246520742774,
+            -0.0069091785699129105,
+            -0.0021043571177870035,
+            -0.014402477070689201,
+            -0.010278765112161636,
+            0.02144678682088852,
+            0.015605589374899864,
+            0.012432162649929523,
+            0.01332141924649477,
+            0.01685229316353798,
+            0.0037597264163196087,
+            -0.034593842923641205,
+            0.006128899287432432,
+            0.0041607641614973545,
+            -0.005435801576822996,
+            0.01782873272895813,
+            0.000511104823090136,
+            -0.010793139226734638,
+            0.013399883173406124,
+            -0.007593557704240084,
+            0.018098996952176094,
+            -0.003212659154087305,
+            -0.004184739198535681,
+            -0.005536061245948076,
+            -0.008740002289414406,
+            -0.007371243555098772,
+            0.00455090356990695,
+            0.0021043571177870035,
+            0.013643993064761162,
+            0.0005775811732746661,
+            -0.004625008441507816,
+            -0.011333668604493141,
+            0.0014602995943278074,
+            -0.0032257363200187683,
+            -0.001628125086426735,
+            -0.023835577070713043,
+            -0.012170616537332535,
+            0.0017120378324761987,
+            -0.0086179468780756,
+            -0.01235369872301817,
+            0.02856084331870079,
+            -0.0017011400777846575,
+            -0.0008009852026589215,
+            0.010688520967960358,
+            0.015675334259867668,
+            0.001266319421119988,
+            0.01837797835469246,
+            -0.012170616537332535,
+            -0.011185458861291409,
+            -0.01649484597146511,
+            -0.01282448135316372,
+            -0.01122033130377531,
+            0.02090625837445259,
+            0.0033979208674281836,
+            -0.00708354264497757,
+            0.010418256744742393,
+            -0.011106994934380054,
+            -0.005056559573858976,
+            -0.0091018071398139,
+            0.011708551086485386,
+            -0.02969421073794365,
+            0.02069702185690403,
+            -0.012711144983768463,
+            0.006024280562996864,
+            0.018639525398612022,
+            0.039231929928064346,
+            -0.006608400493860245,
+            -0.006869946606457233,
+            0.010034655220806599,
+            -0.003068808699026704,
+            0.029205990955233574,
+            -0.0059414575807750225,
+            0.004237048327922821,
+            -0.021551406010985374,
+            0.0018449904164299369,
+            -0.01074954867362976,
+            0.0012815763475373387,
+            0.01991238258779049,
+            -0.0191974900662899,
+            -0.013722456991672516,
+            -0.00675660977140069,
+            -0.016058936715126038,
+            -0.00823870487511158,
+            0.022283734753727913,
+            0.011377259157598019,
+            -0.006111462600529194,
+            -0.008273578248918056,
+            -0.005736579652875662,
+            -0.024201741442084312,
+            0.014716332778334618,
+            0.08174190670251846,
+            -0.009982346557080746,
+            -0.0001858067698776722,
+            0.0003367407189216465,
+            0.0017806936521083117,
+            0.007907412946224213,
+            -0.00031821453012526035,
+            0.05956278741359711,
+            -0.003509078174829483,
+            0.003899217816069722,
+            -0.012998845428228378,
+            0.01083673071116209,
+            -0.01438504084944725,
+            -0.005684270523488522,
+            -0.009712082333862782,
+            -0.01197881530970335,
+            -0.0008167869527824223,
+            -0.02203962579369545,
+            -0.001063076313585043,
+            -0.011996251530945301,
+            -0.005021686665713787,
+            -0.021324733272194862,
+            0.017514877021312714,
+            -0.02087138593196869,
+            -0.01648612879216671,
+            -0.004594494588673115,
+            -0.010705957189202309,
+            0.012118306942284107,
+            -0.016555873677134514,
+            0.002380070276558399,
+            0.007270984351634979,
+            0.014228112995624542,
+            0.007475861813873053,
+            -0.008199472911655903,
+            -0.008408710360527039,
+            -0.0076197125017642975,
+            0.004986814223229885,
+            0.0069876424968242645,
+            -0.0011088468600064516,
+            0.014637868851423264,
+            -0.02088882215321064,
+            -0.016058936715126038,
+            0.007589198648929596,
+            0.0017817834159359336,
+            0.009502844884991646,
+            -0.007820230908691883,
+            0.007449707482010126,
+            0.00020882829267065972,
+            0.022754518315196037,
+            -0.010156710632145405,
+            -0.0268520750105381,
+            -0.007933568209409714,
+            0.008644101209938526,
+            -0.013138337060809135,
+            -0.006102744489908218,
+            0.005331183318048716,
+            0.009807982482016087,
+            0.008081777021288872,
+            -0.009877728298306465,
+            0.0007110786973498762,
+            -0.018918508663773537,
+            0.013373728841543198,
+            0.003238813718780875,
+            0.022196553647518158,
+            -0.012667553499341011,
+            0.010880321264266968,
+            -0.02683463878929615,
+            -0.002574050333350897,
+            -0.014419913291931152,
+            0.0008592882077209651,
+            0.008434864692389965,
+            0.004742704331874847,
+            0.01684357412159443,
+            -0.017192302271723747,
+            -0.017096402123570442,
+            0.009311044588685036,
+            -0.01782873272895813,
+            -0.02376583032310009,
+            -0.0018471699440851808,
+            -0.02624180167913437,
+            -0.0038163948338478804,
+            0.007532530464231968,
+            -0.019389290362596512,
+            -0.01369630265980959,
+            0.0065953233279287815,
+            -0.0033412526827305555,
+            -0.014611713588237762,
+            0.017183585092425346,
+            -0.011970097199082375,
+            -0.03544822707772255,
+            0.003727033268660307,
+            0.008382555097341537,
+            -0.004729627165943384,
+            0.006747891660779715,
+            -0.024410977959632874,
+            0.016015345230698586,
+            -0.002397506730630994,
+            0.004293716512620449,
+            0.008029468357563019,
+            0.02606743760406971,
+            0.0010309278732165694,
+            0.0034545892849564552,
+            -0.005544779356569052,
+            0.014411195181310177,
+            0.005801966413855553,
+            0.023504285141825676,
+            0.026416165754199028,
+            0.0015529305674135685,
+            -0.010958785191178322,
+            -0.01081929448992014,
+            0.01592816226184368,
+            -0.011586496606469154,
+            -0.0048386044800281525,
+            0.022737082093954086,
+            0.013827075250446796,
+            -0.02203962579369545,
+            0.00017259323794860393,
+            -0.02149909734725952,
+            0.0069876424968242645,
+            -0.017148712649941444,
+            0.009467972442507744,
+            0.008731283247470856,
+            -0.005073996260762215,
+            0.02069702185690403,
+            0.010095682926476002,
+            0.030984506011009216,
+            0.01991238258779049,
+            0.002920599188655615,
+            0.011412132531404495,
+            -0.006451472640037537,
+            0.010470565408468246,
+            0.004537826403975487,
+            0.0022275017108768225,
+            -0.006608400493860245,
+            0.006930973846465349,
+            0.0023451976012438536,
+            -0.001170419156551361,
+            0.008583074435591698,
+            -0.007471502758562565,
+            0.009921318851411343,
+            0.0005939278053119779,
+            0.0055273426696658134,
+            -0.004287178162485361,
+            -0.010575184598565102,
+            -0.030339358374476433,
+            -0.003132015699520707,
+            0.020958568900823593,
+            -0.023539157584309578,
+            -0.008003313094377518,
+            0.005932739470154047,
+            0.016983065754175186,
+            -0.008282296359539032,
+            -0.012798327021300793,
+            -0.01708768494427204,
+            0.003105861134827137,
+            -0.0004893093137070537,
+            -0.010313638485968113,
+            0.01747128553688526,
+            0.0009546435903757811,
+            -0.004951941315084696,
+            0.008953598327934742,
+            -0.00680456031113863,
+            0.028212115168571472,
+            0.01505634281784296,
+            0.011150586418807507,
+            -0.012100870721042156,
+            0.014201958663761616,
+            -0.012711144983768463,
+            0.007680739741772413,
+            0.009799264371395111,
+            -0.006416599731892347,
+            -0.021220114082098007,
+            -0.017558468505740166,
+            0.023451974615454674,
+            0.006848151329904795,
+            0.012222925201058388,
+            -0.011987533420324326,
+            -0.0012815763475373387,
+            0.007475861813873053,
+            0.004834245424717665,
+            -0.01611996442079544,
+            -0.00048631246318109334,
+            0.0061986446380615234,
+            0.038604218512773514,
+            0.005087073426693678,
+            0.01072339341044426,
+            -0.0035984397400170565,
+            -0.0002066487359115854,
+            0.019162617623806,
+            -0.01952878199517727,
+            -0.006503781769424677,
+            -0.005832480266690254,
+            -0.02739260531961918,
+            0.016573309898376465,
+            -0.01685229316353798,
+            -0.016451256349682808,
+            0.010976221412420273,
+            -0.00041356991278007627,
+            -0.004422310274094343,
+            -0.01284191757440567,
+            -0.008766156621277332,
+            0.00794228632003069,
+            0.010810576379299164,
+            -0.009136680513620377,
+            0.00532246520742774,
+            0.00357446470297873,
+            -0.0035984397400170565,
+            0.02127242274582386,
+            0.0006467819330282509,
+            -0.002837776206433773,
+            0.007358166389167309
+          ],
+          "title": "Method",
+          "keyphrases": [
+            "mixture-of-experts (MoE) models",
+            "token-choice routing",
+            "load imbalance",
+            "expert choice method",
+            "Transformer architecture"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "33412f87-7418-4ce2-9116-377a27fd2732",
+        "properties": {
+          "page_content": "4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n",
+          "title": "4 Experiments",
+          "embedding": [
+            0.009938324801623821,
+            -0.009144282899796963,
+            -0.026211904361844063,
+            -0.022352691739797592,
+            0.016043050214648247,
+            0.016247965395450592,
+            -0.02783413790166378,
+            0.0537557490170002,
+            -0.02827811799943447,
+            -0.01755429059267044,
+            0.010237157344818115,
+            0.001297788112424314,
+            0.014762339182198048,
+            -0.01001516729593277,
+            0.007487895432859659,
+            0.015283161774277687,
+            0.019347287714481354,
+            0.019910801202058792,
+            0.02535809576511383,
+            -0.04251963645219803,
+            0.0020213902462273836,
+            -0.032034873962402344,
+            -0.058332156389951706,
+            0.0034600566141307354,
+            0.001006959704682231,
+            0.01755429059267044,
+            0.011645940132439137,
+            -0.0015795447397977114,
+            -0.053584985435009,
+            0.04794985428452492,
+            0.016555335372686386,
+            0.0044910297729074955,
+            -0.007833687588572502,
+            -0.004798400681465864,
+            -0.04204150289297104,
+            -0.028961164876818657,
+            0.05720513314008713,
+            0.005165537819266319,
+            -0.005519868340343237,
+            0.05481446906924248,
+            0.020252324640750885,
+            0.010245694778859615,
+            -0.03961668908596039,
+            -0.009904172271490097,
+            0.010134699754416943,
+            0.001490962109528482,
+            -0.0010427129454910755,
+            -0.007479357533156872,
+            0.02295035682618618,
+            -0.04146091267466545,
+            -0.005549751687794924,
+            -0.03811398521065712,
+            0.01863008923828602,
+            0.013780459761619568,
+            -0.04426140338182449,
+            0.023223575204610825,
+            -0.0052978782914578915,
+            0.03527934476733208,
+            0.057854026556015015,
+            0.02783413790166378,
+            0.024162763729691505,
+            -0.02590453252196312,
+            -0.04340759664773941,
+            -0.0036180110182613134,
+            0.010143238119781017,
+            0.008119713515043259,
+            -0.01344747468829155,
+            -0.010527451522648335,
+            0.046173933893442154,
+            -0.023701708763837814,
+            -0.05327761545777321,
+            0.027680452913045883,
+            -0.013584083877503872,
+            -0.008081291802227497,
+            0.0036372218746691942,
+            -0.03517688810825348,
+            0.012807118706405163,
+            0.03548425808548927,
+            0.01004078146070242,
+            0.015018481761217117,
+            0.001224147155880928,
+            0.008418546058237553,
+            -0.0014920294051989913,
+            0.017912890762090683,
+            0.028517184779047966,
+            -0.01417321152985096,
+            -0.006309640593826771,
+            -0.0686461552977562,
+            0.025767924264073372,
+            -0.007368362508714199,
+            -0.020115714520215988,
+            0.00026281276950612664,
+            -0.007099412847310305,
+            0.0033597343135625124,
+            -0.020781684666872025,
+            0.07472527027130127,
+            -0.013105952180922031,
+            0.014386664144694805,
+            -0.02096952311694622,
+            0.007300057914108038,
+            -0.031932417303323746,
+            0.02236976847052574,
+            0.01146664097905159,
+            0.005916888825595379,
+            0.038318898528814316,
+            -0.02834642305970192,
+            -0.005869929678738117,
+            0.0122009152546525,
+            0.0292856115847826,
+            0.08387809246778488,
+            0.01419028826057911,
+            -0.014352511614561081,
+            -0.025563009083271027,
+            -0.016538260504603386,
+            -0.046105626970529556,
+            -0.0292856115847826,
+            -0.025170257315039635,
+            -0.027099864557385445,
+            0.036338064819574356,
+            -0.0010613900376483798,
+            0.012388752773404121,
+            0.05338007211685181,
+            0.015863751992583275,
+            -0.00874299369752407,
+            -0.039241012185811996,
+            0.0438857264816761,
+            -0.006198645569384098,
+            0.010809208266437054,
+            0.002330895746126771,
+            -0.0146428057923913,
+            -0.006185838486999273,
+            0.011833777651190758,
+            -0.013763383962213993,
+            -0.029456373304128647,
+            0.031949494034051895,
+            -0.01727253384888172,
+            0.03944592550396919,
+            0.06239628419280052,
+            -0.03562086820602417,
+            -0.02001325786113739,
+            0.03189826384186745,
+            -0.035825781524181366,
+            0.018288565799593925,
+            -0.0005250918911769986,
+            -0.030993228778243065,
+            0.026246055960655212,
+            0.01562468521296978,
+            0.06092773377895355,
+            -0.024196917191147804,
+            0.028705023229122162,
+            -0.019176525995135307,
+            0.01628211699426174,
+            -0.020627999678254128,
+            0.03596239164471626,
+            0.0023351646959781647,
+            -0.024265222251415253,
+            -0.015752756968140602,
+            0.014694035053253174,
+            -0.014275669120252132,
+            -0.022830823436379433,
+            0.019740039482712746,
+            0.02100367471575737,
+            -0.010450609028339386,
+            -0.0049051265232264996,
+            0.012013077735900879,
+            0.04989653453230858,
+            0.016205275431275368,
+            -0.014130521565675735,
+            -0.07479357719421387,
+            0.04801815748214722,
+            0.021703798323869705,
+            0.01958635449409485,
+            0.023104043677449226,
+            0.035825781524181366,
+            -0.001056053675711155,
+            0.030566323548555374,
+            0.0019007899099960923,
+            -0.013601160608232021,
+            -0.02044016122817993,
+            0.043646663427352905,
+            -0.0039019023533910513,
+            -0.0019274713704362512,
+            0.0076373121701180935,
+            0.018288565799593925,
+            0.03123229369521141,
+            0.005430218297988176,
+            0.010399380698800087,
+            0.039719145745038986,
+            -0.011586173437535763,
+            0.01414759736508131,
+            0.009460192173719406,
+            0.02091829478740692,
+            0.04299776628613472,
+            0.02245514839887619,
+            -0.008670419454574585,
+            -0.02194286324083805,
+            0.00032551426556892693,
+            0.0219599399715662,
+            0.01811780408024788,
+            0.02199409157037735,
+            -0.012013077735900879,
+            -0.0220111683011055,
+            0.009383349679410458,
+            -0.02199409157037735,
+            0.01682855375111103,
+            -0.00167666538618505,
+            0.01859593763947487,
+            0.013865840621292591,
+            -0.006258412264287472,
+            -0.01677732542157173,
+            0.0002502724528312683,
+            -0.018186109140515327,
+            -0.008380125276744366,
+            -0.010945817455649376,
+            -0.016495568677783012,
+            0.03364003449678421,
+            -0.020747533068060875,
+            0.006599935237318277,
+            -0.01810072734951973,
+            -0.011885006912052631,
+            -0.005835777148604393,
+            0.03596239164471626,
+            0.012209453620016575,
+            -0.01851055584847927,
+            -0.026058217510581017,
+            -0.011108040809631348,
+            -0.018817927688360214,
+            -0.006463326048105955,
+            0.0342206209897995,
+            -0.01750306226313114,
+            0.006659701932221651,
+            -0.004211407620459795,
+            -0.002409872831776738,
+            0.004258367232978344,
+            0.033435117453336716,
+            -0.03082246705889702,
+            -0.01676025055348873,
+            0.023582175374031067,
+            0.008059946820139885,
+            0.029046546667814255,
+            -0.016888320446014404,
+            -0.006275488529354334,
+            -0.024231068789958954,
+            -0.016973702237010002,
+            -0.01643580198287964,
+            -0.005118578672409058,
+            -0.02199409157037735,
+            -0.016025975346565247,
+            0.02790244296193123,
+            0.013686541467905045,
+            -0.00790199264883995,
+            -0.019313136115670204,
+            -0.002333030104637146,
+            0.02542640082538128,
+            -0.013592622242867947,
+            0.04064125940203667,
+            0.050545431673526764,
+            -0.05713682621717453,
+            -0.0122521435841918,
+            -0.03661128506064415,
+            0.07684271782636642,
+            -0.015377080999314785,
+            -0.03032725863158703,
+            0.033964481204748154,
+            -0.029388070106506348,
+            -0.008990597911179066,
+            0.026997406035661697,
+            0.025272715836763382,
+            -0.014719649218022823,
+            0.01123611256480217,
+            0.03756754845380783,
+            -0.030139420181512833,
+            -0.0439881831407547,
+            -0.03777246177196503,
+            0.019859572872519493,
+            -0.025289790704846382,
+            -0.01419028826057911,
+            -0.0110311983153224,
+            0.023530947044491768,
+            0.01801534742116928,
+            0.010664060711860657,
+            0.007547662127763033,
+            0.006843270733952522,
+            0.021157361567020416,
+            0.02144765481352806,
+            0.007073798682540655,
+            -0.03237639740109444,
+            0.0003433908859733492,
+            -0.03944592550396919,
+            -0.009093054570257664,
+            -0.025289790704846382,
+            -0.02095244638621807,
+            -0.023821242153644562,
+            0.00949434470385313,
+            0.03275207430124283,
+            -0.0008143192972056568,
+            0.0067365444265306,
+            0.015044095925986767,
+            0.015488076023757458,
+            -0.03807983547449112,
+            -0.03555256128311157,
+            0.006348061840981245,
+            0.01682855375111103,
+            -0.0055625587701797485,
+            -0.059220120310783386,
+            -0.00042770442087203264,
+            -0.003165493020787835,
+            -0.005016121547669172,
+            -0.028909936547279358,
+            0.010476223193109035,
+            0.013345018029212952,
+            0.010237157344818115,
+            0.0110311983153224,
+            -0.027526767924427986,
+            -0.014651344157755375,
+            -0.0036073385272175074,
+            0.006019345950335264,
+            0.02100367471575737,
+            -0.040709562599658966,
+            -0.022830823436379433,
+            0.03818229213356972,
+            0.0003810117777902633,
+            -0.06598227471113205,
+            -0.005387527868151665,
+            -0.010757979936897755,
+            0.015436847694218159,
+            -0.0005656477296724916,
+            0.034476764500141144,
+            -0.03121521882712841,
+            -0.019296059384942055,
+            -0.020815838128328323,
+            -0.051979828625917435,
+            -0.03531349450349808,
+            -0.015436847694218159,
+            0.007389707490801811,
+            -0.038353051990270615,
+            -0.024265222251415253,
+            0.004111085087060928,
+            -0.0268949493765831,
+            0.007398245856165886,
+            -0.013567008078098297,
+            0.015633223578333855,
+            -0.03852381557226181,
+            -0.010092009790241718,
+            0.004262635949999094,
+            -0.01861301250755787,
+            -0.005507061257958412,
+            0.014369587413966656,
+            0.013925607316195965,
+            0.014719649218022823,
+            0.0017268265364691615,
+            0.015266085974872112,
+            0.0062071834690868855,
+            -0.00769280968233943,
+            0.007833687588572502,
+            -0.04091447591781616,
+            0.0014461372047662735,
+            0.018852079287171364,
+            0.011432488448917866,
+            -0.011082426644861698,
+            0.004151640925556421,
+            -0.010373766534030437,
+            -0.03034433349967003,
+            0.029917430132627487,
+            -0.05419972911477089,
+            0.0024141420144587755,
+            0.005554020404815674,
+            -0.016410188749432564,
+            -0.008384393528103828,
+            -0.009690719656646252,
+            -0.03708941861987114,
+            -0.01248267199844122,
+            0.0039019023533910513,
+            0.00019050588889513165,
+            0.009349197149276733,
+            0.002870929194614291,
+            0.05471201241016388,
+            -0.003165493020787835,
+            -0.04484199360013008,
+            0.023308956995606422,
+            -0.03671374171972275,
+            0.018749622628092766,
+            0.04245132952928543,
+            0.020713381469249725,
+            -0.017212767153978348,
+            0.04791570082306862,
+            0.0195692777633667,
+            0.000224524803343229,
+            -0.016999315470457077,
+            0.025153182446956635,
+            -0.007099412847310305,
+            -0.061303410679101944,
+            -0.004448339343070984,
+            -0.018749622628092766,
+            0.018664240837097168,
+            0.016674868762493134,
+            -0.002407738473266363,
+            -0.010032243095338345,
+            0.009426039643585682,
+            -0.01510386262089014,
+            -0.017443295568227768,
+            0.01705908216536045,
+            -0.011458102613687515,
+            -0.004525181837379932,
+            0.007526317145675421,
+            0.013148642145097256,
+            0.01271320041269064,
+            0.008853988721966743,
+            0.023992003872990608,
+            -0.045627497136592865,
+            0.009127207100391388,
+            -0.01997910626232624,
+            -0.036918655037879944,
+            -0.002990462351590395,
+            0.01318279467523098,
+            0.04183658957481384,
+            -0.02779998630285263,
+            0.0019253368955105543,
+            0.005007583647966385,
+            -0.013515779748558998,
+            -0.021106131374835968,
+            0.00341309723444283,
+            0.04600317031145096,
+            -0.0015763428527861834,
+            0.021618416532874107,
+            -0.011304416693747044,
+            -0.03562086820602417,
+            0.008811297826468945,
+            0.04142676293849945,
+            -0.005028928630053997,
+            0.01663217879831791,
+            0.009434578008949757,
+            0.03548425808548927,
+            -0.04590071365237236,
+            -0.007833687588572502,
+            -0.02344556525349617,
+            -0.01813488081097603,
+            0.04105108603835106,
+            -0.03075416199862957,
+            -0.01365238893777132,
+            0.03524519130587578,
+            -0.005306416191160679,
+            0.02441890724003315,
+            -0.018271489068865776,
+            -0.02588745579123497,
+            0.0014600115828216076,
+            -0.03712356835603714,
+            -0.00708660576492548,
+            -0.01806657575070858,
+            -0.006160224322229624,
+            -0.06557244807481766,
+            0.00010459146869834512,
+            -0.07431544363498688,
+            -0.002902946900576353,
+            0.025136105716228485,
+            -0.006552976090461016,
+            0.04429555684328079,
+            -0.026228979229927063,
+            0.02098659984767437,
+            -0.05553166940808296,
+            -0.01367800310254097,
+            0.00978463888168335,
+            0.03375956416130066,
+            0.007611697539687157,
+            -0.001995776081457734,
+            0.0015507286880165339,
+            -0.026997406035661697,
+            -0.0018303507240489125,
+            -0.007282981649041176,
+            -0.001387437921948731,
+            4.89605481561739e-05,
+            -0.018852079287171364,
+            0.00950288213789463,
+            0.000838332693092525,
+            0.01610281690955162,
+            0.031454283744096756,
+            0.022711291909217834,
+            -0.03070293366909027,
+            -0.008862526156008244,
+            0.005195421166718006,
+            0.02587038092315197,
+            -0.02978082001209259,
+            0.003897633170709014,
+            -0.03464752435684204,
+            0.014864795841276646,
+            5.636466448777355e-05,
+            0.016657792031764984,
+            0.008008718490600586,
+            0.017016392201185226,
+            -0.040197279304265976,
+            0.0030160765163600445,
+            -0.0016254368238151073,
+            -0.027202321216464043,
+            0.020867066457867622,
+            0.002305281348526478,
+            -0.022147778421640396,
+            0.0054472945630550385,
+            -0.004207138437777758,
+            0.024213993921875954,
+            0.04473953694105148,
+            -0.0010592554463073611,
+            -0.02930268831551075,
+            -0.012704662047326565,
+            0.042212262749671936,
+            0.02633143588900566,
+            -0.028483033180236816,
+            0.017118848860263824,
+            0.02103782817721367,
+            -0.02341141365468502,
+            -0.021089056506752968,
+            0.013925607316195965,
+            -0.016179660335183144,
+            -0.0007694943924434483,
+            -0.005468640010803938,
+            0.010399380698800087,
+            0.01240582950413227,
+            -0.029883278533816338,
+            0.05676115304231644,
+            -0.016820017248392105,
+            -0.006834732368588448,
+            -0.03034433349967003,
+            -0.006168762221932411,
+            -0.014531810767948627,
+            0.05553166940808296,
+            0.0028175662737339735,
+            0.02728770114481449,
+            0.01658095046877861,
+            0.002264725510030985,
+            0.05628301948308945,
+            -0.03678204491734505,
+            -0.03451091796159744,
+            0.018681317567825317,
+            -0.0055625587701797485,
+            -0.00671093026176095,
+            -0.022728366777300835,
+            -0.014207364059984684,
+            -0.013353556394577026,
+            0.016965163871645927,
+            -0.0268778745085001,
+            -0.012260681949555874,
+            0.019740039482712746,
+            0.017690900713205338,
+            -2.0469710761972237e-06,
+            0.02293328195810318,
+            0.012644895352423191,
+            0.014284206554293633,
+            0.017878737300634384,
+            -0.020867066457867622,
+            0.019142374396324158,
+            0.024111535400152206,
+            0.009946862235665321,
+            -0.01763967238366604,
+            -0.017878737300634384,
+            0.007347017526626587,
+            -0.0038613462820649147,
+            -0.007125027012079954,
+            0.01437812577933073,
+            0.007043915335088968,
+            -0.004047049675136805,
+            -0.027151092886924744,
+            -0.018356870859861374,
+            0.02093537151813507,
+            -0.0023757205344736576,
+            0.01539415679872036,
+            -0.00928943045437336,
+            -0.010612832382321358,
+            0.03763585537672043,
+            0.01861301250755787,
+            -0.046139780431985855,
+            0.026280207559466362,
+            0.01051037572324276,
+            -0.006228528916835785,
+            -0.013729231432080269,
+            0.005626594182103872,
+            0.009101592935621738,
+            0.010220080614089966,
+            0.016572412103414536,
+            -0.005097233224660158,
+            -0.03328143432736397,
+            0.01564176194369793,
+            0.026211904361844063,
+            -0.020798761397600174,
+            0.012021616101264954,
+            0.00017663151083979756,
+            0.022762520238757133,
+            -0.005028928630053997,
+            -0.012311910279095173,
+            -0.01562468521296978,
+            0.009741948917508125,
+            -0.012235067784786224,
+            0.01123611256480217,
+            0.037396788597106934,
+            0.015914980322122574,
+            -0.03565501794219017,
+            0.015735680237412453,
+            -0.01686270721256733,
+            -0.015684451907873154,
+            0.03261546418070793,
+            0.008264861069619656,
+            -0.02636558935046196,
+            -0.007142103277146816,
+            -0.006527361460030079,
+            0.02001325786113739,
+            -0.02102075144648552,
+            -0.008508196100592613,
+            0.004418455995619297,
+            0.0035646480973809958,
+            0.009938324801623821,
+            0.006745082791894674,
+            0.005016121547669172,
+            0.019159449264407158,
+            0.013806073926389217,
+            -0.0005186883499845862,
+            0.012730276212096214,
+            0.011577636003494263,
+            0.002902946900576353,
+            0.0075391242280602455,
+            -0.010997045785188675,
+            -0.02295035682618618,
+            0.004734364803880453,
+            0.05945918336510658,
+            -0.005852853413671255,
+            0.03876287862658501,
+            0.010407919064164162,
+            0.028961164876818657,
+            0.021686721593141556,
+            -0.024299373850226402,
+            -0.01681147888302803,
+            -0.018151957541704178,
+            -0.021737949922680855,
+            -0.04453462362289429,
+            -0.01510386262089014,
+            -0.01629919372498989,
+            -0.013404784724116325,
+            0.005694898776710033,
+            -0.00013474156730808318,
+            -0.0013116624904796481,
+            0.011355645954608917,
+            -0.002003246918320656,
+            -0.012491210363805294,
+            -0.003355465130880475,
+            0.05324346199631691,
+            -0.0011056812945753336,
+            0.017912890762090683,
+            -0.004576410632580519,
+            0.015539304353296757,
+            -0.007300057914108038,
+            -0.016111355274915695,
+            0.029131926596164703,
+            0.0035561099648475647,
+            -0.0007540191290900111,
+            0.012106996960937977,
+            0.004469684325158596,
+            0.04149506613612175,
+            -0.005887005943804979,
+            -0.026058217510581017,
+            0.011910621076822281,
+            -0.017187153920531273,
+            0.020371858030557632,
+            0.00769707839936018,
+            -0.007607428822666407,
+            0.001932807732373476,
+            0.012909576296806335,
+            -0.027116939425468445,
+            0.019705887883901596,
+            -0.020645076408982277,
+            -0.045149363577365875,
+            0.009280892089009285,
+            -0.01175693515688181,
+            0.023035738617181778,
+            0.001117421081289649,
+            -0.010800670832395554,
+            -0.03507443144917488,
+            0.0021387888118624687,
+            0.0039040367119014263,
+            -0.03372541442513466,
+            0.006548706907778978,
+            0.01863008923828602,
+            0.015539304353296757,
+            0.009485806338489056,
+            -0.012465596199035645,
+            0.003758889390155673,
+            0.0031847036443650723,
+            -0.012371676973998547,
+            0.02776583470404148,
+            -0.004742903169244528,
+            -0.031983643770217896,
+            0.035859934985637665,
+            -0.02733892947435379,
+            0.009468729607760906,
+            -0.020730456337332726,
+            -0.012346062809228897,
+            -0.01608574204146862,
+            0.00441418681293726,
+            -0.02148180827498436,
+            -0.002561423694714904,
+            -0.02983205020427704,
+            0.027577996253967285,
+            0.009383349679410458,
+            -0.015249010175466537,
+            -0.004862436093389988,
+            0.009101592935621738,
+            -0.025255639106035233,
+            0.007846495136618614,
+            0.012585128657519817,
+            -0.010646984912455082,
+            -0.01367800310254097,
+            -0.00706099160015583,
+            -0.0014685495989397168,
+            0.005972386337816715,
+            0.0158210601657629,
+            0.04672037065029144,
+            0.004431263078004122,
+            0.03712356835603714,
+            0.02337726205587387,
+            0.013532855547964573,
+            0.0037012575194239616,
+            0.0047898623161017895,
+            -0.0067322757095098495,
+            0.04012897238135338,
+            0.006249873898923397,
+            0.0219599399715662,
+            0.008247784338891506,
+            -0.044671230018138885,
+            0.008503926917910576,
+            0.03264961391687393,
+            0.026621730998158455,
+            0.018715469166636467,
+            0.009588262997567654,
+            -0.0024013349320739508,
+            0.021191513165831566,
+            -0.020252324640750885,
+            -0.01705908216536045,
+            0.0035902622621506453,
+            -0.007530585862696171,
+            -0.0056650154292583466,
+            -0.009716334752738476,
+            0.03387909755110741,
+            -0.003229528432711959,
+            -0.008862526156008244,
+            0.018186109140515327,
+            -0.007432397920638323,
+            0.00475997943431139,
+            0.006245605181902647,
+            0.0011163539020344615,
+            0.02739015780389309,
+            0.008683227002620697,
+            0.027680452913045883,
+            0.005391797050833702,
+            0.007573276292532682,
+            0.0058571225963532925,
+            -0.009451653808355331,
+            0.018920384347438812,
+            -0.0097590247169137,
+            -0.033366814255714417,
+            -0.009443115442991257,
+            0.0016393112018704414,
+            0.014070754870772362,
+            0.0004023569927085191,
+            -0.014062216505408287,
+            -0.011893544346094131,
+            0.007859301753342152,
+            0.012533900327980518,
+            0.02107197977602482,
+            0.018169032409787178,
+            -0.005016121547669172,
+            -0.024726277217268944,
+            0.012593667022883892,
+            0.011056812480092049,
+            -0.01861301250755787,
+            0.008465506136417389,
+            0.012542438693344593,
+            0.057410046458244324,
+            -0.044705383479595184,
+            0.016538260504603386,
+            0.02151595987379551,
+            -0.004896588623523712,
+            -0.015249010175466537,
+            -0.010851899161934853,
+            0.0031633584294468164,
+            -0.017477449029684067,
+            0.014975790865719318,
+            -0.01798119582235813,
+            0.011193421669304371,
+            -0.014745263382792473,
+            0.00743666710332036,
+            0.012866885401308537,
+            -0.009682182222604752,
+            0.003942457959055901,
+            0.020371858030557632,
+            0.0028068935498595238,
+            -0.0077867284417152405,
+            0.005387527868151665,
+            0.01390853151679039,
+            0.020354781299829483,
+            0.006066305562853813,
+            -0.0005933965439908206,
+            0.019876649603247643,
+            -0.028175661340355873,
+            -0.025170257315039635,
+            0.01909114606678486,
+            -0.03551841154694557,
+            0.005472908727824688,
+            0.02790244296193123,
+            -0.04296361654996872,
+            -0.05020390823483467,
+            0.0027407235465943813,
+            -0.039719145745038986,
+            0.0004191663465462625,
+            -0.0023287611547857523,
+            0.01536854263395071,
+            -0.004057722166180611,
+            -0.017212767153978348,
+            0.0146428057923913,
+            -0.012397291138768196,
+            -0.0067322757095098495,
+            0.009835867211222649,
+            0.009016212075948715,
+            -0.024572592228651047,
+            -0.0016222350532189012,
+            -0.030890770256519318,
+            0.009169897064566612,
+            -0.017844585701823235,
+            -0.019142374396324158,
+            -0.025528857484459877,
+            -0.0005656477296724916,
+            -0.010638446547091007,
+            -0.001638244022615254,
+            0.051501695066690445,
+            0.021652569994330406,
+            0.020166942849755287,
+            -0.007726961746811867,
+            -0.023650480434298515,
+            0.05457540228962898,
+            0.0018132745753973722,
+            0.03814813867211342,
+            -0.012380215339362621,
+            -0.0002006448630709201,
+            -0.0008852921309880912,
+            0.022113624960184097,
+            -0.017101772129535675,
+            -0.0011547752656042576,
+            -0.011022660881280899,
+            -0.00586566049605608,
+            -0.0070780678652226925,
+            0.014045140706002712,
+            0.015974747017025948,
+            -0.002610517665743828,
+            0.014156135730445385,
+            0.01221799198538065,
+            0.022694215178489685,
+            0.03130060061812401,
+            0.016230888664722443,
+            -0.0034664601553231478,
+            -0.022199006751179695,
+            -0.012918113730847836,
+            -0.008200825192034245,
+            0.008999135345220566,
+            -0.008525271899998188,
+            -0.0031847036443650723,
+            -0.02631436102092266,
+            -0.020730456337332726,
+            -0.011312955059111118,
+            -0.01292665209621191,
+            -0.0031462821643799543,
+            0.02641681768000126,
+            -0.002469639526680112,
+            -0.03763585537672043,
+            -0.01511240005493164,
+            -0.03524519130587578,
+            -0.015607609413564205,
+            -0.0017876604106277227,
+            -0.01748598739504814,
+            0.016538260504603386,
+            -0.003780234605073929,
+            -0.007833687588572502,
+            -0.0008964983280748129,
+            -0.018169032409787178,
+            0.009690719656646252,
+            -0.011048275046050549,
+            -0.000543768925126642,
+            -0.0017503062263131142,
+            0.013345018029212952,
+            -0.008205094374716282,
+            0.008973521180450916,
+            -0.01412198320031166,
+            0.006868884898722172,
+            -0.048120614141225815,
+            -0.027731681242585182,
+            -0.027099864557385445,
+            0.022557605057954788,
+            -0.015214857645332813,
+            -0.023189423605799675,
+            0.008734455332159996,
+            -0.013686541467905045,
+            0.008939368650317192,
+            -0.012021616101264954,
+            0.008388662710785866,
+            0.019210679456591606,
+            0.009024749509990215,
+            -0.002614786848425865,
+            -0.013669464737176895,
+            -0.02390662208199501,
+            -0.02585330419242382,
+            0.023650480434298515,
+            -0.026485122740268707,
+            0.01071528997272253,
+            -0.012021616101264954,
+            0.008585038594901562,
+            -0.029917430132627487,
+            -0.020730456337332726,
+            0.0044910297729074955,
+            0.011347107589244843,
+            -0.021140284836292267,
+            0.044671230018138885,
+            0.04043634235858917,
+            0.029985735192894936,
+            -0.020252324640750885,
+            0.05139923840761185,
+            -0.022796671837568283,
+            0.007312864996492863,
+            0.036440521478652954,
+            0.030446790158748627,
+            0.026092370972037315,
+            -0.015581995248794556,
+            -0.012294834479689598,
+            -0.015496614389121532,
+            0.013404784724116325,
+            -0.024333525449037552,
+            0.002821835223585367,
+            -0.02042308636009693,
+            0.013643850572407246,
+            0.03451091796159744,
+            -0.004917933605611324,
+            0.029934506863355637,
+            -0.013720693998038769,
+            0.030361410230398178,
+            0.0017652479000389576,
+            -0.026587579399347305,
+            0.015667375177145004,
+            -0.014232978224754333,
+            0.004418455995619297,
+            0.017964119091629982,
+            -0.006087650544941425,
+            0.013575546443462372,
+            -0.061337560415267944,
+            0.028141509741544724,
+            0.01758844405412674,
+            0.01123611256480217,
+            -0.002180411946028471,
+            -0.024726277217268944,
+            -0.0021142419427633286,
+            0.009340658783912659,
+            -0.0016745307948440313,
+            0.011859391815960407,
+            0.01193623524159193,
+            -0.033418044447898865,
+            0.010578680783510208,
+            -0.013635313138365746,
+            0.024299373850226402,
+            0.006258412264287472,
+            0.0007476155878975987,
+            -0.03961668908596039,
+            -0.00560524920001626,
+            0.014830644242465496,
+            0.01095435582101345,
+            -0.005989462602883577,
+            -0.04624223709106445,
+            -0.030105268582701683,
+            0.025563009083271027,
+            0.005110040307044983,
+            -0.05167245492339134,
+            0.014557424932718277,
+            -0.015334390103816986,
+            0.018766699358820915,
+            -0.008708841167390347,
+            -0.025972837582230568,
+            -0.029985735192894936,
+            -0.017187153920531273,
+            0.016726097092032433,
+            0.048564594238996506,
+            0.004207138437777758,
+            0.013345018029212952,
+            -0.0048538981936872005,
+            0.006651164032518864,
+            -0.024760430678725243,
+            0.012030153535306454,
+            -0.027151092886924744,
+            -0.02971251681447029,
+            0.029012393206357956,
+            0.009981014765799046,
+            -0.011330030858516693,
+            0.019330210983753204,
+            -0.017084697261452675,
+            -0.0014141193823888898,
+            0.02240392006933689,
+            -0.0006163425859995186,
+            -0.007295788731426001,
+            -0.010382304899394512,
+            0.013618236407637596,
+            -0.0014354645973071456,
+            -0.01762259565293789,
+            -0.009571187198162079,
+            0.04241717979311943,
+            0.010092009790241718,
+            0.018835002556443214,
+            0.03261546418070793,
+            -0.0018933190731331706,
+            -0.02400907874107361,
+            -0.0027215126901865005,
+            -0.026638807728886604,
+            -0.017033468931913376,
+            -0.014548887498676777,
+            0.016674868762493134,
+            0.023530947044491768,
+            0.015249010175466537,
+            -0.027526767924427986,
+            -0.030019886791706085,
+            -0.01950097270309925,
+            -0.022199006751179695,
+            -0.0034493841230869293,
+            -0.0004213008505757898,
+            -0.026502197608351707,
+            0.007005494087934494,
+            0.0341181643307209,
+            0.03357172757387161,
+            -0.0010870042024180293,
+            -0.009016212075948715,
+            -0.010638446547091007,
+            0.0007721625734120607,
+            0.015231933444738388,
+            -0.015018481761217117,
+            0.0012326851719990373,
+            0.005028928630053997,
+            0.005383259151130915,
+            0.0029285612981766462,
+            -0.007095144130289555,
+            -0.05747834965586662,
+            0.003961668815463781,
+            0.02544347569346428,
+            0.005084426142275333,
+            -0.0008377990452572703,
+            0.02629728429019451,
+            0.0008276600856333971,
+            0.008123982697725296,
+            -0.03142013028264046,
+            0.011108040809631348,
+            0.01803242415189743,
+            0.015163629315793514,
+            -0.018151957541704178,
+            0.0122521435841918,
+            0.0064206356182694435,
+            0.004811207763850689,
+            -0.00030950538348406553,
+            0.010143238119781017,
+            0.028636718168854713,
+            0.019620506092905998,
+            -0.014736725017428398,
+            0.023821242153644562,
+            0.0006435577524825931,
+            -0.012072844430804253,
+            -0.024333525449037552,
+            -0.001040044822730124,
+            0.0029883277602493763,
+            -0.011176345869898796,
+            0.014702572487294674,
+            -0.0023949311580508947,
+            0.0007956422632560134,
+            0.04139260947704315,
+            0.0437491200864315,
+            0.013481627218425274,
+            -0.011295879259705544,
+            0.003304236801341176,
+            0.026502197608351707,
+            0.020132791250944138,
+            0.004862436093389988,
+            0.011458102613687515,
+            -0.03331558406352997,
+            -0.019620506092905998,
+            -0.002981924219056964,
+            0.0014290610561147332,
+            -0.0026788224931806326,
+            0.03876287862658501,
+            0.029524678364396095,
+            0.009682182222604752,
+            -0.005741858389228582,
+            -0.011364183388650417,
+            0.013643850572407246,
+            0.020201096311211586,
+            -0.023599252104759216,
+            -0.0038015798199921846,
+            0.0016745307948440313,
+            -0.017170077189803123,
+            0.02288205362856388,
+            -0.0070311082527041435,
+            -0.00928943045437336,
+            0.02645096927881241,
+            0.09186973422765732,
+            -0.00464471522718668,
+            0.012141148559749126,
+            -0.03948007896542549,
+            -0.021823331713676453,
+            -0.006975610740482807,
+            -0.0036671049892902374,
+            0.015402695164084435,
+            0.02739015780389309,
+            0.009135744534432888,
+            0.007133565377444029,
+            0.017878737300634384,
+            -0.004606293980032206,
+            0.012508286163210869,
+            -0.02826104313135147,
+            0.020166942849755287,
+            -0.005007583647966385,
+            -0.0038335975259542465,
+            -0.011389797553420067,
+            -0.005989462602883577,
+            0.011765473522245884,
+            0.005468640010803938,
+            0.017016392201185226,
+            0.0019893725402653217,
+            -0.0024760430678725243,
+            0.0014333301223814487,
+            0.003031018190085888,
+            -0.010877513326704502,
+            -0.009383349679410458,
+            -0.021396426483988762,
+            0.003827193984761834,
+            0.0009914844995364547,
+            -0.007884915918111801,
+            0.012755890376865864,
+            0.005502792075276375,
+            0.024043232202529907,
+            -0.028175661340355873,
+            -0.0030609015375375748,
+            0.00633525475859642,
+            -0.008956445381045341,
+            0.010390842333436012,
+            0.012380215339362621,
+            -0.008640536107122898,
+            0.022830823436379433,
+            -0.01466841995716095,
+            0.014762339182198048,
+            0.01170570682734251,
+            -0.021601341664791107,
+            -0.004155910108238459,
+            -0.009938324801623821,
+            -0.02192578837275505,
+            0.020252324640750885,
+            -0.018715469166636467,
+            0.005033197812736034,
+            0.00073320756200701,
+            0.006774966139346361,
+            -0.0010496501345187426,
+            -0.01740914396941662,
+            0.010348152369260788,
+            -0.008785683661699295,
+            0.006839001551270485,
+            -0.005058811977505684,
+            0.018254414200782776,
+            0.012166762724518776,
+            0.0028090281412005424,
+            -0.0011622461024671793,
+            0.014600115828216076,
+            -0.00372900627553463,
+            -0.018886230885982513,
+            -0.0036713741719722748,
+            0.01633334532380104,
+            0.01051037572324276,
+            0.0002606782363727689,
+            -0.010305461473762989,
+            -0.0146342683583498,
+            0.014139058999717236,
+            0.000977076473645866,
+            -0.0018122073961421847,
+            -0.0005285604856908321,
+            -0.03760170191526413,
+            0.002606248715892434,
+            0.013532855547964573,
+            0.012704662047326565,
+            -0.005818700883537531,
+            -0.017477449029684067,
+            0.007983104325830936,
+            -0.004190062638372183,
+            -0.0003204447857569903,
+            0.01661510206758976,
+            -0.019774191081523895,
+            0.01002370472997427,
+            -0.014779414981603622,
+            -0.03278622403740883,
+            -0.0029776552692055702,
+            -0.008380125276744366,
+            0.005822970066219568,
+            -0.005592442117631435,
+            -0.0071805245243012905,
+            -0.01414759736508131,
+            -0.005741858389228582,
+            -0.02979789674282074,
+            0.008947907015681267,
+            0.0317445807158947,
+            0.004473953507840633,
+            -0.028978241607546806,
+            -0.012158225290477276,
+            0.025580085813999176,
+            -0.0024931191001087427,
+            -0.006915844511240721,
+            0.003009672975167632,
+            0.0012572321575134993,
+            -0.005784548819065094,
+            -0.009656568057835102,
+            -0.012935190461575985,
+            0.013063261285424232,
+            0.01320840884000063,
+            0.011244650930166245,
+            -0.03357172757387161,
+            0.010664060711860657,
+            0.01813488081097603,
+            -0.02042308636009693,
+            0.011748396791517735,
+            0.028585489839315414,
+            0.0029435029719024897,
+            0.010399380698800087,
+            0.007334209978580475,
+            0.008286206051707268,
+            -0.017153002321720123,
+            0.016683407127857208,
+            0.009186973795294762,
+            -0.012559514492750168,
+            0.004563603550195694,
+            0.007381169591099024,
+            0.008542348630726337,
+            0.01960342936217785,
+            -0.011603250168263912,
+            0.013609698973596096,
+            -0.0068816919811069965,
+            0.03321312740445137,
+            -0.011919158510863781,
+            -0.014881872572004795,
+            0.03229101747274399,
+            0.027560919523239136,
+            -0.004563603550195694,
+            0.017246920615434647,
+            -0.01943266950547695,
+            -0.01267050951719284,
+            0.019296059384942055,
+            0.0005907283630222082,
+            0.019313136115670204,
+            -0.011304416693747044,
+            -0.02595576085150242,
+            0.029012393206357956,
+            -0.008106905966997147,
+            -0.001259366748854518,
+            -0.015829598531126976,
+            0.0043458822183310986,
+            -0.0005789885181002319,
+            0.037328481674194336,
+            0.012653433717787266,
+            0.0011985328746959567,
+            0.026570502668619156,
+            -0.00769280968233943,
+            -0.025067800655961037,
+            0.001990439835935831,
+            -0.015214857645332813,
+            0.013336479663848877,
+            -0.027970748022198677,
+            -0.00560524920001626,
+            0.014506196603178978,
+            0.003026749240234494,
+            -0.002644670195877552,
+            0.01666633039712906,
+            -0.007513510063290596,
+            0.011791087687015533,
+            -0.019244831055402756,
+            0.016717558726668358,
+            -0.011517869308590889,
+            0.004473953507840633,
+            -0.011910621076822281,
+            0.026092370972037315,
+            -0.025563009083271027,
+            -0.00999809056520462,
+            0.013660927303135395,
+            -0.008354511111974716,
+            0.0006376878009177744,
+            -0.02252345345914364,
+            -0.008089830167591572,
+            -0.020371858030557632,
+            0.01679440215229988,
+            -0.005682091694325209,
+            0.011902082711458206,
+            -0.00306303589604795,
+            0.0025528857950121164,
+            -0.008585038594901562,
+            0.010322538204491138,
+            -0.0009199780761264265,
+            -0.017417682334780693,
+            -0.005968117620795965,
+            -0.03025895357131958,
+            -0.030361410230398178,
+            0.03072001039981842,
+            -0.00880275946110487,
+            0.003923247568309307,
+            0.02728770114481449,
+            -0.02540932409465313,
+            -0.005485715810209513,
+            0.001918933354318142,
+            0.022728366777300835,
+            -0.030651705339550972,
+            0.007914799265563488,
+            -0.009391887113451958,
+            -0.016452878713607788,
+            0.007184793706983328,
+            -0.0008874266059137881,
+            0.015880826860666275,
+            -0.0025635582860559225,
+            -0.04924764111638069,
+            -0.0007081269286572933,
+            0.017135925590991974,
+            -0.013806073926389217,
+            0.008081291802227497,
+            0.02151595987379551,
+            -0.009630953893065453,
+            -0.008439891040325165,
+            -0.0011334300506860018,
+            0.025989914312958717,
+            -0.022625910118222237,
+            0.030019886791706085,
+            0.005767472553998232,
+            0.004713019821792841,
+            0.005340568721294403,
+            0.002962713595479727,
+            0.01730668731033802,
+            0.002080089645460248,
+            0.004258367232978344,
+            0.025238562375307083,
+            0.013148642145097256,
+            0.0037418133579194546,
+            -0.0032978332601487637,
+            -0.0030352871399372816,
+            0.016538260504603386,
+            -0.018442250788211823,
+            0.02100367471575737,
+            -0.0004714620590675622,
+            0.00013560871593654156,
+            0.016256503760814667,
+            -0.01484772004187107,
+            -0.027219397947192192,
+            0.03075416199862957,
+            -0.022335615009069443,
+            0.01757136732339859,
+            0.023770013824105263,
+            -0.01339624635875225,
+            -0.004922202788293362,
+            -0.007073798682540655,
+            0.001534719718620181,
+            0.018339794129133224,
+            -0.0134218605235219,
+            -0.023308956995606422,
+            -0.00030790449818596244,
+            0.0220111683011055,
+            0.02293328195810318,
+            0.028175661340355873,
+            -0.021140284836292267,
+            0.01412198320031166,
+            0.007466550450772047,
+            -0.008311820216476917,
+            0.016469955444335938,
+            0.008452698588371277,
+            -0.0025272713974118233,
+            0.021293969824910164,
+            0.0075903525575995445,
+            0.004576410632580519,
+            -0.008896678686141968,
+            -0.013242561370134354,
+            -0.01267050951719284,
+            -0.006749351508915424,
+            -0.006164493504911661,
+            -0.008264861069619656,
+            -0.01709323562681675,
+            0.09548988193273544,
+            -0.011867930181324482,
+            -0.0028410458471626043,
+            -0.009195511229336262,
+            0.0026041141245514154,
+            -0.006949996575713158,
+            0.0013479492627084255,
+            0.023086966946721077,
+            -0.022728366777300835,
+            -0.03671374171972275,
+            0.011167807504534721,
+            -0.004070529248565435,
+            0.006288295611739159,
+            0.03712356835603714,
+            0.00038314631092362106,
+            -0.016589488834142685,
+            -6.773765198886395e-05,
+            -0.003372541395947337,
+            0.01295226626098156,
+            0.0028197006322443485,
+            -0.0036158766597509384,
+            0.0109799699857831,
+            0.029200231656432152,
+            -0.017810434103012085,
+            -0.02144765481352806,
+            -0.012525362893939018,
+            0.0032615463715046644,
+            -0.002578499959781766,
+            0.003707661060616374,
+            0.0051996903494000435,
+            -0.014053679071366787,
+            -0.0134218605235219,
+            0.018817927688360214,
+            -0.016094280406832695,
+            0.0008025794522836804,
+            -0.012627819553017616,
+            -0.008222170174121857,
+            0.01753721572458744,
+            0.01757136732339859,
+            -0.03406693786382675,
+            -0.009921248070895672,
+            -0.03855796530842781,
+            0.018749622628092766,
+            0.01643580198287964,
+            0.014540349133312702,
+            -0.020884141325950623,
+            -0.012644895352423191,
+            -0.004520913120359182,
+            -0.02981497347354889,
+            -0.02243807166814804,
+            -0.026536351069808006,
+            0.009605339728295803,
+            0.00365856708958745,
+            0.027595072984695435,
+            0.005259457044303417,
+            -0.006262681446969509,
+            0.00801725685596466,
+            -0.01025423314422369,
+            0.00755620002746582,
+            -0.02146473154425621,
+            0.007107951212674379,
+            -0.0011409008875489235,
+            0.0012529630912467837,
+            0.005592442117631435,
+            -0.008853988721966743,
+            -0.01735791563987732,
+            -0.0079276068136096,
+            0.01466841995716095,
+            -0.015752756968140602,
+            -0.00821790099143982,
+            0.020798761397600174,
+            0.009383349679410458,
+            0.00931504461914301,
+            0.015616146847605705,
+            0.045661646872758865,
+            0.008273398503661156,
+            0.00783795677125454,
+            -0.011056812480092049,
+            0.009648029692471027,
+            -0.01813488081097603,
+            0.00745801255106926,
+            -0.019330210983753204,
+            -0.006292564328759909,
+            -0.019722962751984596,
+            0.04012897238135338,
+            -0.009366272948682308,
+            -0.01635042205452919,
+            0.033435117453336716,
+            -0.011808163486421108,
+            0.01910822093486786,
+            0.0035475720651447773,
+            -0.022830823436379433,
+            0.03352050110697746,
+            0.003876287955790758,
+            0.012918113730847836,
+            -0.025682542473077774,
+            0.0035859933122992516,
+            -0.00877714529633522,
+            0.006032153032720089,
+            -0.011586173437535763,
+            0.003494208911433816,
+            0.0023885276168584824,
+            -0.004841091111302376,
+            -0.008760069496929646,
+            0.018698394298553467,
+            0.006595666520297527,
+            0.0002183880569646135,
+            0.028056127950549126,
+            -0.000954130373429507,
+            0.018220260739326477,
+            6.110063259257004e-05,
+            0.006010808050632477,
+            0.027680452913045883,
+            0.002821835223585367,
+            0.006809118203818798,
+            -0.01714446395635605,
+            -0.0009199780761264265,
+            -0.009485806338489056,
+            0.009596801362931728,
+            0.02639974094927311,
+            -0.010612832382321358,
+            -0.031078608706593513,
+            0.022574681788682938,
+            -0.007048184517771006,
+            0.015966208651661873,
+            0.002999000484123826,
+            -0.014062216505408287,
+            -0.0023437028285115957,
+            -0.027680452913045883,
+            0.014454968273639679,
+            -0.017349377274513245,
+            -0.024265222251415253,
+            -0.007231753319501877,
+            -0.01045914739370346,
+            -0.004845359828323126,
+            0.038353051990270615,
+            0.012474133633077145,
+            -0.024572592228651047,
+            0.006817656569182873,
+            -0.003974475897848606,
+            -0.005989462602883577,
+            0.0057119750417768955,
+            0.005976655520498753,
+            0.004098278004676104,
+            0.010945817455649376,
+            -0.0065700518898665905,
+            -0.01320840884000063,
+            -0.03324728086590767,
+            -0.02240392006933689,
+            0.014941639266908169,
+            0.00028068936080671847,
+            -0.01315718051046133,
+            0.028483033180236816,
+            -0.01002370472997427,
+            -0.005575365852564573,
+            0.0071292961947619915,
+            0.00013887719251215458,
+            -0.0022796671837568283,
+            -0.017844585701823235,
+            0.002273263642564416,
+            0.006245605181902647,
+            -0.014950176700949669,
+            0.009844405576586723,
+            0.0032124524004757404,
+            -0.014540349133312702,
+            0.005635132547467947,
+            -0.0015027020126581192,
+            -0.003562513506039977,
+            -0.02481165900826454,
+            0.0059595792554318905,
+            0.00708233704790473,
+            -0.0018036692636087537,
+            0.002390662208199501,
+            -0.018169032409787178,
+            0.01801534742116928,
+            0.001955220242962241,
+            0.0071762558072805405,
+            -0.029166078194975853,
+            0.007282981649041176,
+            -0.006740813609212637,
+            0.0015080382581800222,
+            -0.0195692777633667,
+            0.009161358699202538,
+            0.020798761397600174,
+            0.012491210363805294,
+            -0.0005250918911769986,
+            0.010442070662975311,
+            0.008175211027264595,
+            -0.012704662047326565,
+            0.003950996324419975,
+            -0.02050846628844738,
+            -0.007308595813810825,
+            -0.010057857260107994,
+            0.011372721754014492,
+            -0.004219945520162582,
+            0.015983285382390022,
+            0.009511420503258705,
+            0.005600980017334223,
+            -0.009699258022010326,
+            0.00743666710332036,
+            0.01292665209621191,
+            0.005874198395758867,
+            -0.0011099503608420491,
+            0.017827508971095085,
+            0.07363239675760269,
+            0.005336299538612366,
+            0.013618236407637596,
+            6.870485958643258e-05,
+            -0.013071799650788307,
+            0.0015624684747308493,
+            -0.004939279053360224,
+            -0.01748598739504814,
+            0.026126522570848465,
+            0.026587579399347305,
+            -0.006382214371114969,
+            0.0004917400074191391,
+            0.009016212075948715,
+            0.005558289587497711,
+            0.005485715810209513,
+            0.022216081619262695,
+            -0.0077867284417152405,
+            0.01753721572458744,
+            0.024179840460419655,
+            0.005541213322430849,
+            -0.0019093280425295234,
+            -0.0022305732127279043,
+            0.009152821265161037,
+            0.004209273029118776,
+            -0.009050363674759865,
+            0.01150079257786274,
+            -0.01437812577933073,
+            0.01909114606678486,
+            0.011389797553420067,
+            0.02146473154425621,
+            0.007073798682540655,
+            -0.01704200729727745,
+            -0.018356870859861374,
+            0.0041281613521277905,
+            -0.0013479492627084255,
+            0.0059083509258925915,
+            0.0063224476762115955,
+            0.005477177910506725,
+            -0.011611787602305412,
+            -0.0041793896816670895,
+            0.010356690734624863,
+            -0.002471773885190487,
+            -0.0008500725380145013,
+            0.01757136732339859,
+            -0.01268758624792099,
+            0.01168009266257286,
+            -0.001839956152252853,
+            -0.0009984216885641217,
+            0.009682182222604752,
+            0.0014354645973071456,
+            -0.011825240217149258,
+            -0.004166582599282265,
+            0.0021003675647079945,
+            0.01217530108988285,
+            -0.013259637169539928,
+            -0.01048476155847311,
+            0.011825240217149258,
+            -0.006168762221932411,
+            0.009169897064566612,
+            -0.009579724632203579,
+            -0.0025315405800938606,
+            -0.00039648707024753094,
+            -0.008085560984909534,
+            -0.0014578771078959107,
+            0.007820880971848965,
+            0.0008324627415277064,
+            0.010518914088606834,
+            0.011560559272766113,
+            0.01150079257786274,
+            0.003560379147529602,
+            0.004002224653959274,
+            -0.002964847953990102,
+            -0.0064206356182694435,
+            -0.011432488448917866,
+            8.15119783510454e-05,
+            0.005776010919362307,
+            -0.02242099680006504,
+            -0.00024800453684292734,
+            0.008465506136417389,
+            0.005046004895120859,
+            -0.0007710952777415514,
+            0.020849989727139473,
+            0.001264702994376421,
+            0.007820880971848965,
+            -0.025221485644578934,
+            0.0011238247388973832,
+            0.006890229880809784,
+            0.007718423847109079,
+            0.012798581272363663,
+            -0.028636718168854713,
+            -0.0038528083823621273,
+            -0.0038634808734059334,
+            0.0171273872256279,
+            -0.00011606451880652457,
+            -0.0033106403425335884,
+            0.021686721593141556,
+            -0.0007833687704987824,
+            -0.003430173499509692,
+            -0.00755620002746582,
+            -0.005579635035246611,
+            0.008004449307918549,
+            0.012994957156479359,
+            0.024623820558190346,
+            -0.0019968433771282434,
+            -0.0055967108346521854,
+            -0.00809409935027361,
+            0.0244530588388443,
+            0.015249010175466537,
+            0.01863008923828602,
+            -0.0219087116420269,
+            0.010638446547091007,
+            -0.013865840621292591,
+            0.009067440405488014,
+            0.011714245192706585,
+            -0.010911665856838226,
+            0.0004290384822525084,
+            -0.010757979936897755,
+            -0.017135925590991974,
+            -0.022711291909217834,
+            0.00502039073035121,
+            0.02839765138924122,
+            -0.0002676154254004359,
+            0.007193332072347403,
+            -0.016358960419893265,
+            -0.0018442251021042466,
+            -0.011312955059111118,
+            -0.025750847533345222,
+            -0.016222350299358368,
+            -0.021772101521492004,
+            -0.02141350321471691,
+            -0.0038592119235545397,
+            -0.002131317974999547,
+            -0.01907406933605671,
+            0.00013153978215996176,
+            -0.0012817791430279613,
+            0.01902284100651741,
+            -0.0014888276346027851,
+            0.0015624684747308493,
+            0.012337524443864822,
+            0.012943727895617485,
+            -0.012824195437133312,
+            -0.004228483885526657,
+            -0.019261907786130905,
+            -0.024179840460419655,
+            0.0032487392891198397,
+            -0.007726961746811867,
+            0.013874378986656666,
+            -0.008234977722167969,
+            0.004260501824319363,
+            -0.017187153920531273,
+            0.014702572487294674,
+            -0.03579162806272507,
+            -0.024128612130880356,
+            0.0006531630642712116,
+            -0.0040299734100699425,
+            -0.009605339728295803,
+            -0.02585330419242382,
+            0.01198746357113123,
+            -0.019620506092905998,
+            0.002104636514559388,
+            -0.018442250788211823,
+            0.011398335918784142,
+            -0.015283161774277687,
+            0.018681317567825317,
+            -0.00025520852068439126,
+            0.03179580718278885,
+            0.004405648913234472,
+            0.007978835143148899,
+            -0.004525181837379932,
+            0.003327716374769807,
+            -0.005468640010803938,
+            0.006365138106048107,
+            0.006988417822867632,
+            0.008055677637457848,
+            0.007825150154531002,
+            -0.004768517334014177,
+            0.004061991348862648,
+            0.005110040307044983,
+            0.030515095219016075,
+            0.028021976351737976,
+            0.0027706066612154245,
+            0.0014674824196845293,
+            0.0034685947466641665,
+            0.004576410632580519,
+            -0.00038021133514121175,
+            0.008862526156008244,
+            -0.011287340894341469,
+            -0.005391797050833702,
+            0.01417321152985096,
+            -0.012448519468307495,
+            -0.0037695621140301228,
+            0.015377080999314785,
+            -0.015086785890161991,
+            -0.026092370972037315,
+            0.021823331713676453,
+            -0.006552976090461016,
+            -0.002450428670272231,
+            -0.006689585279673338,
+            0.008115444332361221,
+            -0.005946772173047066,
+            0.004194331355392933,
+            0.028107358142733574,
+            -0.005737589206546545,
+            -0.010945817455649376,
+            0.012636357918381691,
+            -0.002330895746126771,
+            -0.007948951795697212,
+            0.02291620522737503,
+            0.035894084721803665,
+            0.00999809056520462,
+            -0.026041142642498016,
+            -0.007381169591099024,
+            0.02144765481352806,
+            -0.00973341055214405,
+            -0.021755026653409004,
+            0.008273398503661156,
+            0.007522047962993383,
+            -0.005421680398285389,
+            0.018937459215521812,
+            0.004239156376570463,
+            -0.010792132467031479,
+            0.009041826240718365,
+            0.022711291909217834,
+            0.021686721593141556,
+            0.008947907015681267,
+            -0.023548023775219917,
+            0.01025423314422369,
+            -0.026228979229927063,
+            -0.013831689022481441,
+            -0.023821242153644562,
+            0.003581724362447858,
+            0.003857077332213521,
+            -0.003257277188822627,
+            0.026536351069808006,
+            0.011876468546688557,
+            0.011645940132439137,
+            0.0030950538348406553,
+            0.023052813485264778,
+            -0.015223395079374313,
+            -0.028107358142733574,
+            -0.0009733410552144051,
+            -0.015479537658393383,
+            -0.0018282162491232157,
+            0.002166537567973137,
+            0.007150641642510891,
+            0.005852853413671255,
+            0.01953512616455555,
+            0.000376209121895954,
+            0.0019584218971431255,
+            -2.4263486920972355e-05,
+            0.010100548155605793,
+            -0.015727141872048378,
+            -0.019705887883901596,
+            -0.008922292850911617,
+            6.470263178925961e-05,
+            0.011637401767075062,
+            -0.005208228249102831,
+            -3.448583447607234e-05,
+            0.007846495136618614,
+            -0.015897903591394424,
+            -0.0097590247169137,
+            -0.014412278309464455,
+            -0.0029306956566870213,
+            -0.005071619059890509,
+            0.03136890381574631,
+            0.014437892474234104,
+            -0.004747172351926565,
+            -0.011543483473360538,
+            -0.005741858389228582,
+            0.03370833769440651,
+            0.0017204229952767491,
+            0.012422905303537846,
+            0.0146342683583498,
+            0.015684451907873154,
+            0.027014482766389847,
+            0.012807118706405163,
+            0.003018211107701063,
+            0.002516598906368017,
+            0.02346264198422432,
+            -0.013951221480965614,
+            0.008730186149477959,
+            -0.011645940132439137,
+            -0.003521957667544484,
+            -0.010937280021607876,
+            0.015078248456120491,
+            -0.013003494590520859,
+            -0.011278802528977394,
+            0.023069890215992928,
+            -0.0004381102044135332,
+            0.013276712968945503,
+            -0.0004242358263581991,
+            -0.003953130915760994,
+            -0.02534101903438568,
+            0.0015464596217498183,
+            0.0006531630642712116,
+            -0.004401379730552435,
+            0.010578680783510208,
+            -0.027475539594888687,
+            0.0013660927070304751,
+            0.015240471810102463,
+            -0.007116489112377167,
+            0.00038981667603366077,
+            -0.010826284997165203,
+            0.005904081743210554,
+            0.00011606451880652457,
+            -0.02385539375245571,
+            0.015488076023757458,
+            -0.006698123179376125,
+            0.01243144366890192,
+            -0.0018548977095633745,
+            -0.013788998126983643,
+            0.008687496185302734,
+            0.019296059384942055,
+            0.027560919523239136,
+            -0.007825150154531002,
+            0.0005528406472876668,
+            -0.0001855698210420087,
+            -0.006027883850038052,
+            -0.02737308293581009,
+            0.02054261974990368,
+            -0.007863570936024189,
+            0.01953512616455555,
+            -0.009682182222604752,
+            -0.018186109140515327,
+            -0.014856258407235146,
+            0.011927696876227856,
+            0.007603159639984369,
+            0.0010304395109415054,
+            0.01489894837141037,
+            -0.024077383801341057,
+            0.020867066457867622,
+            0.0033383890986442566,
+            -0.0064718639478087425,
+            0.008303281851112843,
+            0.004230618476867676,
+            0.004695943556725979,
+            -0.008384393528103828,
+            0.007658657152205706,
+            0.007569007575511932,
+            -0.007253098301589489,
+            0.004326671827584505,
+            -0.04057295247912407,
+            0.010732365772128105,
+            -0.011090965010225773,
+            0.005878467578440905,
+            -0.004742903169244528,
+            -0.022745443508028984,
+            -0.035825781524181366,
+            0.004567872267216444,
+            0.019313136115670204,
+            -0.020235247910022736,
+            0.018408099189400673,
+            0.012124072760343552,
+            0.014574501663446426,
+            0.01859593763947487,
+            0.008853988721966743,
+            0.01461719162762165,
+            0.0438857264816761,
+            -0.024231068789958954,
+            6.483604374807328e-05,
+            0.008734455332159996,
+            0.011381260119378567,
+            -0.013404784724116325,
+            -0.010126162320375443,
+            0.005784548819065094,
+            -0.0015197781613096595,
+            0.04067540913820267,
+            -0.019210679456591606,
+            0.010886050760746002,
+            -0.010962894186377525,
+            0.008371586911380291,
+            -0.009562648832798004,
+            -0.0036158766597509384,
+            -0.012559514492750168,
+            -0.011893544346094131,
+            0.014745263382792473,
+            -0.019671734422445297,
+            -0.005460101645439863,
+            -0.007282981649041176,
+            0.01048476155847311,
+            -0.006096188444644213,
+            -0.020730456337332726,
+            0.018647165969014168,
+            -0.0018079383298754692,
+            0.029968658462166786,
+            -0.012755890376865864,
+            0.00574612757191062,
+            0.022591758519411087,
+            -0.010604294948279858,
+            0.018732545897364616,
+            -0.004362958483397961,
+            0.008853988721966743,
+            0.0055625587701797485,
+            -0.0065188235603272915,
+            -0.0014685495989397168,
+            0.01389145478606224,
+            -0.0027385889552533627,
+            0.029934506863355637,
+            -0.017400605604052544,
+            -0.009818791411817074,
+            0.009707796387374401,
+            0.0073769004084169865,
+            0.0042370217852294445,
+            0.003878422547131777,
+            0.009579724632203579,
+            -0.01071528997272253,
+            0.0009728074073791504,
+            -0.021686721593141556,
+            -0.015453923493623734,
+            -0.0011483717244118452,
+            -0.004683136474341154,
+            -0.0018730411538854241,
+            -0.002467504935339093,
+            -0.02044016122817993,
+            0.004627638962119818,
+            -0.0021579996682703495,
+            -8.451364556094632e-05,
+            -0.024487212300300598,
+            -0.00769280968233943,
+            -0.010057857260107994,
+            -0.005161269102245569,
+            -0.003139878623187542,
+            0.018271489068865776,
+            0.0034856710117310286,
+            -0.004862436093389988,
+            0.0293197650462389,
+            0.005404604133218527,
+            0.01536854263395071,
+            0.00280262460000813,
+            -0.05334591865539551,
+            -0.011014122515916824,
+            -0.004563603550195694,
+            -0.009912709705531597,
+            -0.007261636666953564,
+            -0.008456967771053314,
+            -0.010681137442588806,
+            -0.0019061262719333172,
+            0.0032914294861257076,
+            -0.004170851781964302,
+            0.0122521435841918,
+            -0.01700785383582115,
+            0.025494705885648727,
+            -0.004017166327685118,
+            0.015744218602776527,
+            -0.011722782626748085,
+            0.011654478497803211,
+            -0.006194376386702061,
+            -0.010851899161934853,
+            0.010544528253376484,
+            0.0146342683583498,
+            -0.0030544979963451624,
+            0.020371858030557632,
+            -0.014736725017428398,
+            0.003197510726749897,
+            -0.008303281851112843,
+            -0.02143057994544506,
+            -0.015991821885108948,
+            0.011295879259705544,
+            0.03130060061812401,
+            0.010066395625472069,
+            -0.0037524858489632607,
+            0.004072663839906454,
+            0.021669644862413406,
+            -0.004352285992354155,
+            0.0063181789591908455,
+            0.03025895357131958,
+            -0.0010982104577124119,
+            0.017742129042744637,
+            0.0001935742620844394,
+            -0.005028928630053997,
+            -0.002362913452088833,
+            0.01808365248143673,
+            -0.002766337711364031,
+            0.0057119750417768955,
+            0.02441890724003315,
+            0.010997045785188675,
+            -0.0064718639478087425,
+            0.008230708539485931,
+            0.014309820719063282,
+            0.013874378986656666,
+            -0.015351466834545135,
+            -0.0122009152546525,
+            -0.011534945107996464,
+            -0.015778370201587677,
+            0.007291519548743963,
+            -0.019261907786130905,
+            -0.004049184266477823,
+            0.007829418405890465,
+            -0.015138015151023865,
+            -0.012807118706405163,
+            -0.023548023775219917,
+            -0.015914980322122574,
+            0.01863008923828602,
+            0.005357644986361265,
+            -0.01465988252311945,
+            -0.004369362257421017,
+            0.0013714289525523782,
+            0.006181569304317236,
+            -0.0054985228925943375,
+            0.021840406581759453,
+            -0.004098278004676104,
+            -0.011125117540359497,
+            0.019330210983753204,
+            0.0017897948855534196,
+            0.005097233224660158,
+            -0.0064206356182694435,
+            0.004550796467810869,
+            -0.011364183388650417,
+            0.022096550092101097,
+            -0.003470729338005185,
+            0.012636357918381691,
+            -0.0037033918779343367,
+            0.004631908144801855,
+            -0.001978699816390872,
+            0.00821790099143982,
+            0.004017166327685118,
+            0.010629909113049507,
+            -0.005656477529555559,
+            0.004525181837379932,
+            0.012491210363805294,
+            -0.01641872711479664,
+            0.007265905383974314,
+            0.004559334367513657,
+            -4.365760105429217e-05,
+            0.005780279636383057,
+            -0.007645850069820881,
+            -0.005417411215603352,
+            -0.015488076023757458,
+            0.010100548155605793,
+            0.030873695388436317,
+            -0.0016649254830554128,
+            -0.007240291219204664,
+            0.01704200729727745,
+            0.015735680237412453,
+            -0.003816521493718028,
+            0.0061218030750751495,
+            0.009280892089009285,
+            -0.003148416755720973,
+            0.01370361726731062,
+            -0.024487212300300598,
+            -0.029524678364396095,
+            0.01763967238366604,
+            -0.020867066457867622,
+            0.022779595106840134,
+            -0.019210679456591606,
+            -0.027731681242585182,
+            -0.01847640424966812,
+            -0.0032530082389712334,
+            -0.019842496141791344,
+            -0.01611989364027977,
+            0.02532394416630268,
+            0.00685180863365531,
+            -0.006949996575713158,
+            -0.031112760305404663,
+            0.04050464928150177,
+            0.014489120803773403,
+            0.038318898528814316,
+            -0.011603250168263912,
+            -2.554753518779762e-05,
+            -0.007504971697926521,
+            0.012064306065440178,
+            -0.01146664097905159,
+            -0.014258592389523983,
+            0.007509240880608559,
+            0.028175661340355873,
+            0.015231933444738388,
+            -0.015223395079374313,
+            -0.007944682613015175,
+            0.015991821885108948,
+            -0.007043915335088968,
+            -0.0005480379913933575,
+            -0.008836911991238594,
+            -0.014830644242465496,
+            -0.010066395625472069,
+            -0.0005312286084517837,
+            0.012243606150150299,
+            0.016179660335183144,
+            -0.0007257367251440883,
+            0.01390853151679039,
+            -0.009955400601029396,
+            -0.014420815743505955,
+            -0.004602024797350168,
+            0.005327761638909578,
+            -0.02252345345914364,
+            0.02105490304529667,
+            0.002292474266141653,
+            -0.009537034668028355,
+            0.0030865157023072243,
+            -0.004572141449898481,
+            0.012602205388247967,
+            0.009391887113451958,
+            -0.0038421356584876776,
+            -0.0036137420684099197,
+            0.00572051340714097,
+            -0.001715086749754846,
+            0.015001405030488968,
+            -0.011714245192706585,
+            0.011842316016554832,
+            -0.007432397920638323,
+            -0.00019931078713852912,
+            -0.009648029692471027,
+            -0.005238111596554518,
+            -0.010493299923837185,
+            0.0028794673271477222,
+            0.012943727895617485,
+            -0.014548887498676777,
+            0.012807118706405163,
+            -0.0008132520597428083,
+            -0.00949434470385313,
+            -0.015138015151023865,
+            -0.0055155991576612,
+            -0.011364183388650417,
+            -0.016683407127857208,
+            0.007069529499858618,
+            0.0007855033036321402,
+            0.019347287714481354,
+            0.0030054040253162384,
+            -0.004572141449898481,
+            -0.00017262929759453982,
+            0.018903307616710663,
+            -0.02580207586288452,
+            0.03403278440237045,
+            -0.00010045584349427372,
+            0.022711291909217834,
+            0.04251963645219803,
+            0.00954557303339243,
+            -0.006663971114903688,
+            -0.013754845596849918,
+            0.005801625084131956,
+            -0.013609698973596096,
+            0.009981014765799046,
+            0.014335434883832932,
+            -0.03235932067036629,
+            -0.003944592550396919,
+            -0.013976835645735264,
+            0.011586173437535763,
+            -0.007957490161061287,
+            0.0032850259449332952,
+            -0.010587218217551708,
+            0.011765473522245884,
+            0.01679440215229988,
+            -0.009075978770852089,
+            -0.004311730153858662,
+            -0.027492616325616837,
+            0.00032524747075513005,
+            0.016674868762493134,
+            -0.010911665856838226,
+            -0.01684563048183918,
+            0.010945817455649376,
+            -0.007218946237117052,
+            -0.015453923493623734,
+            -0.006898768246173859,
+            -0.03278622403740883,
+            -0.00020504731219261885,
+            -0.01076651830226183,
+            -0.003927516285330057,
+            -0.0028794673271477222,
+            0.0011024795239791274,
+            -0.0033917520195245743,
+            -0.0067066615447402,
+            -0.0025891726836562157,
+            0.0067664277739822865,
+            0.005011852364987135,
+            0.002080089645460248,
+            0.0005987327895127237,
+            -0.00023506399884354323,
+            0.009255277924239635,
+            0.010168852284550667,
+            -0.007419590838253498,
+            -0.017212767153978348,
+            0.010834822431206703,
+            -0.020730456337332726,
+            0.005938234273344278,
+            0.00783795677125454,
+            0.011039736680686474,
+            -0.0220111683011055,
+            -0.010868974961340427,
+            0.009818791411817074,
+            0.003835732117295265,
+            0.01902284100651741,
+            0.010886050760746002,
+            0.0022455148864537477,
+            -0.0035646480973809958,
+            0.0028495839796960354,
+            -1.0947752343781758e-05,
+            -0.0012508286163210869,
+            0.015308775939047337,
+            0.0006504949415102601,
+            0.02974666841328144,
+            -0.012055767700076103,
+            0.011201960034668446,
+            -0.009340658783912659,
+            0.0013842361513525248,
+            -0.009741948917508125,
+            -0.001973363570868969,
+            -0.004194331355392933,
+            -0.006484671495854855,
+            0.013148642145097256,
+            0.007714154664427042,
+            -0.004239156376570463,
+            -0.01416467409580946,
+            0.01194477267563343,
+            -0.023035738617181778,
+            -0.012269220314919949,
+            -0.00881983619183302,
+            0.012935190461575985,
+            -0.0053960662335157394,
+            0.01170570682734251,
+            0.03457922115921974,
+            -0.0018559650052338839,
+            0.0007860369514673948,
+            -0.013746308162808418,
+            -0.008311820216476917,
+            -0.02728770114481449,
+            -0.01510386262089014,
+            -0.008922292850911617,
+            -0.03454506769776344,
+            -0.011919158510863781,
+            0.008239246904850006,
+            0.008111175149679184,
+            0.0014237246941775084,
+            -0.019176525995135307,
+            -0.008888140320777893,
+            -0.01492456253618002,
+            -0.017460372298955917,
+            -0.00924673955887556,
+            -0.01175693515688181,
+            -0.0037119300104677677,
+            0.020884141325950623,
+            -0.004162313882261515,
+            0.014267130754888058,
+            0.016939548775553703,
+            1.3132299500284716e-05,
+            0.015334390103816986,
+            0.009665105491876602,
+            -0.009127207100391388,
+            -0.004725826904177666,
+            0.006885961163789034,
+            0.0027706066612154245,
+            0.018749622628092766,
+            -0.006702392362058163,
+            -0.014796491712331772,
+            0.004298923071473837,
+            0.005571096669882536,
+            -0.012346062809228897,
+            0.010655523277819157,
+            -0.00476424815133214,
+            0.02054261974990368,
+            -0.018254414200782776,
+            -0.011885006912052631,
+            0.014472044073045254,
+            -0.008862526156008244,
+            0.02394077368080616,
+            0.013951221480965614,
+            -0.015291700139641762,
+            0.007167717441916466,
+            0.013029108755290508,
+            0.007615966722369194,
+            0.0018495614640414715,
+            0.022147778421640396,
+            -0.0011846584966406226,
+            0.0031590894795954227,
+            0.010237157344818115,
+            -0.018835002556443214,
+            -0.02042308636009693,
+            -0.003118533408269286,
+            0.02247222512960434,
+            0.007961759343743324,
+            -0.016222350299358368,
+            0.0037930416874587536,
+            0.012849809601902962,
+            0.012072844430804253,
+            0.0039787450805306435,
+            -0.001204936415888369,
+            -0.02248930186033249,
+            0.0035155541263520718,
+            0.0001910395221784711,
+            0.018664240837097168,
+            -0.011116579174995422,
+            0.001461078878492117,
+            -0.011389797553420067,
+            0.012747352942824364,
+            -0.005221035331487656,
+            0.01193623524159193,
+            0.0011761204805225134,
+            -0.005353375803679228,
+            0.0001780989987310022,
+            0.014941639266908169,
+            0.014070754870772362,
+            0.037738312035799026,
+            -0.005916888825595379,
+            -0.006480402313172817,
+            0.017170077189803123,
+            -0.012388752773404121,
+            0.018715469166636467,
+            0.006497478578239679,
+            0.0170249305665493,
+            -0.011090965010225773,
+            0.0011344973463565111,
+            0.04115354269742966,
+            0.015949131920933723,
+            -6.583659705938771e-05,
+            -0.008760069496929646,
+            -0.00805140845477581,
+            0.0244530588388443,
+            0.009067440405488014,
+            -2.7882166250492446e-05,
+            0.015744218602776527,
+            0.006096188444644213,
+            -0.016324806958436966,
+            -0.012918113730847836,
+            0.03555256128311157,
+            -0.007718423847109079,
+            -0.022045321762561798,
+            -0.0009311842732131481,
+            -0.009383349679410458,
+            -0.02884163148701191,
+            0.013959759846329689,
+            0.013788998126983643,
+            -0.008162403479218483,
+            0.006992687005549669,
+            0.008439891040325165,
+            -0.004117488861083984,
+            -0.004435532260686159,
+            0.0005346972029656172,
+            -0.0007049251580610871,
+            0.006177300587296486,
+            0.005092964507639408,
+            -0.014463506639003754,
+            -0.006787773221731186,
+            -0.005669284611940384,
+            -0.004064125940203667,
+            0.009016212075948715,
+            0.005058811977505684,
+            0.004870974458754063,
+            -0.008482581935822964,
+            0.018339794129133224,
+            0.02298451028764248,
+            -0.0122009152546525,
+            0.012115534394979477,
+            0.005233842879533768,
+            -0.0146428057923913,
+            -0.004614831879734993,
+            -0.01121903583407402,
+            -0.026570502668619156,
+            -0.006113264709711075,
+            0.0033490615896880627,
+            0.015428309328854084,
+            -0.022796671837568283,
+            0.0008500725380145013,
+            0.007790997624397278,
+            2.7365211280994117e-05,
+            -0.008879602886736393,
+            0.007573276292532682,
+            -0.00044157879892736673,
+            0.022301463410258293,
+            0.019808344542980194,
+            0.025631314143538475,
+            -0.008414276875555515,
+            -0.00499904528260231,
+            0.014301283285021782,
+            0.00433947890996933,
+            0.009374811314046383,
+            0.027543844655156136,
+            0.022164853289723396,
+            -0.013857303187251091,
+            0.006429173983633518,
+            0.000640355923678726,
+            -0.014250054024159908,
+            0.0066682398319244385,
+            0.013054722920060158,
+            0.012414366938173771,
+            0.0028410458471626043,
+            -0.0013415457215160131,
+            -0.015957670286297798,
+            0.004824014846235514,
+            0.018408099189400673,
+            -0.017118848860263824,
+            -0.03471583127975464,
+            -0.011842316016554832,
+            -0.012158225290477276,
+            -0.00624133599922061,
+            -0.012875423766672611,
+            0.01440373994410038,
+            0.004418455995619297,
+            0.019705887883901596,
+            0.0002309283590875566,
+            -0.01022861897945404,
+            0.014250054024159908,
+            0.005110040307044983,
+            -0.0016638581873849034,
+            0.0005352308508008718,
+            0.007231753319501877,
+            -0.0030331527814269066,
+            -0.0022476494777947664,
+            -0.00027055040118284523,
+            0.016239427030086517,
+            -0.02532394416630268,
+            0.008947907015681267,
+            -0.017067620530724525,
+            0.0017951311310753226,
+            0.015249010175466537,
+            -0.003775965655222535,
+            -0.016025975346565247,
+            0.0011216901475563645,
+            -0.008619191125035286,
+            -0.07288104295730591,
+            -0.005882736761122942,
+            0.004512374754995108,
+            0.023992003872990608,
+            -0.013336479663848877,
+            -0.0024824466090649366,
+            0.0018474268727004528,
+            0.008657612837851048,
+            -0.010134699754416943,
+            0.003995821345597506,
+            0.010390842333436012,
+            -0.003319178242236376,
+            -0.009221125394105911,
+            0.002508060773834586,
+            -0.008106905966997147,
+            -0.0038805571384727955,
+            -0.00783795677125454,
+            0.003402424743399024,
+            0.025221485644578934,
+            -0.010868974961340427,
+            0.024657974019646645,
+            0.009699258022010326,
+            -0.0075433929450809956,
+            -0.0027428579051047564,
+            -0.007761114276945591,
+            -0.027543844655156136,
+            -0.012397291138768196,
+            -0.003396020969375968,
+            -0.01292665209621191,
+            0.0029264267068356276,
+            0.001692674239166081,
+            -0.007167717441916466,
+            -0.004348016809672117,
+            0.0021547977812588215,
+            0.00030336863710545003,
+            -0.022147778421640396,
+            -0.0009407896432094276,
+            -0.014267130754888058,
+            0.02105490304529667,
+            -0.012371676973998547,
+            -0.004461146425455809,
+            0.0030544979963451624,
+            -0.010109085589647293,
+            0.004580679349601269,
+            0.019313136115670204,
+            -0.007308595813810825,
+            -0.009682182222604752,
+            0.014045140706002712,
+            -0.003690584795549512,
+            0.00185916677583009,
+            -0.01798119582235813,
+            0.010032243095338345,
+            -0.03232516720890999,
+            -0.003583858720958233,
+            -0.009981014765799046,
+            -0.0031847036443650723,
+            -0.0067151994444429874,
+            -0.03162504732608795,
+            0.015138015151023865,
+            -0.007039646618068218,
+            -0.02346264198422432,
+            -0.0011793222511187196,
+            -0.01859593763947487,
+            0.01419028826057911,
+            -0.01487333420664072,
+            0.01393414568156004,
+            -0.020645076408982277,
+            0.012533900327980518,
+            0.012798581272363663,
+            0.021123208105564117,
+            0.006493209395557642,
+            -0.005302147474139929,
+            0.002676687901839614,
+            0.0018730411538854241,
+            0.014352511614561081,
+            -0.008905217051506042,
+            0.00462336977943778,
+            -0.002390662208199501,
+            0.005737589206546545,
+            0.009426039643585682,
+            0.02107197977602482,
+            -0.0005896611255593598,
+            -0.0037823691964149475,
+            -0.017468910664319992,
+            0.00757754547521472,
+            -0.01559907104820013,
+            0.005485715810209513,
+            0.0025251370389014482,
+            0.023787088692188263,
+            0.010886050760746002,
+            0.0035859933122992516,
+            0.004900857340544462,
+            0.004520913120359182,
+            0.01861301250755787,
+            -0.006890229880809784,
+            -0.004747172351926565,
+            -0.018186109140515327,
+            0.019688811153173447,
+            0.007705616764724255,
+            -0.005306416191160679,
+            -0.011902082711458206,
+            0.006010808050632477,
+            0.002134519862011075,
+            0.0017577770631760359,
+            0.02288205362856388,
+            -0.02974666841328144,
+            -0.008964983746409416,
+            -0.007808073423802853,
+            -0.009682182222604752,
+            -0.017844585701823235,
+            0.017084697261452675,
+            -0.008012987673282623,
+            0.00461910106241703,
+            -0.012790042906999588,
+            0.033418044447898865,
+            0.002969117136672139,
+            0.00513992365449667,
+            0.013729231432080269,
+            -0.00024373548512812704,
+            0.017844585701823235,
+            -0.008367317728698254,
+            -0.04036803916096687,
+            -0.0031911071855574846,
+            0.0034515184815973043,
+            0.007645850069820881,
+            -0.010903127491474152,
+            0.004862436093389988,
+            -0.006339523941278458,
+            -0.01854470930993557,
+            -0.014232978224754333,
+            0.037840768694877625,
+            -0.005225304514169693,
+            0.017947042360901833,
+            -0.016145508736371994,
+            -0.0033682722132653,
+            0.007107951212674379,
+            0.022762520238757133,
+            -0.012602205388247967,
+            0.00791053008288145,
+            0.009571187198162079,
+            0.02199409157037735,
+            0.00978463888168335,
+            -0.01565883681178093,
+            0.023735860362648964,
+            0.005741858389228582,
+            -0.0023565099108964205,
+            0.005276532843708992,
+            -0.0030352871399372816,
+            0.016000360250473022,
+            -0.014813567511737347,
+            -0.011876468546688557,
+            0.002303146990016103,
+            -0.010168852284550667,
+            0.0037055264692753553,
+            -0.024589668959379196,
+            0.00037540867924690247,
+            -0.009844405576586723,
+            0.021089056506752968,
+            0.01027130987495184,
+            0.00877714529633522,
+            -0.019637582823634148,
+            -0.0048026698641479015,
+            -0.02645096927881241,
+            -0.014574501663446426,
+            -0.01022861897945404,
+            -0.0011451698374003172,
+            -0.004431263078004122,
+            -0.007765382993966341,
+            -0.03133475035429001,
+            -0.004202869720757008,
+            0.013541393913328648,
+            0.014975790865719318,
+            -0.0013714289525523782,
+            0.012089920230209827,
+            -0.00019544197130016983,
+            -0.008106905966997147,
+            -0.02236976847052574,
+            -0.010407919064164162,
+            -0.02091829478740692,
+            -0.03123229369521141,
+            -0.022199006751179695,
+            0.008670419454574585,
+            -0.012021616101264954,
+            -0.002941368380561471,
+            0.028961164876818657,
+            -0.022130701690912247,
+            0.007513510063290596,
+            0.005400335416197777,
+            0.0028666602447628975,
+            0.014856258407235146,
+            0.0054472945630550385,
+            0.0003634020104072988,
+            0.0037652929313480854,
+            -0.008341703563928604,
+            -0.008081291802227497,
+            0.01799827069044113,
+            -0.003331985557451844,
+            0.005097233224660158,
+            0.010809208266437054,
+            -0.006749351508915424,
+            -0.005438756663352251,
+            0.02392369881272316,
+            -0.015718603506684303,
+            0.002044870052486658,
+            -0.008610652759671211,
+            -0.014079293236136436,
+            -0.00949434470385313,
+            -0.005793086718767881,
+            -0.01813488081097603,
+            -0.015240471810102463,
+            0.01689685881137848,
+            -0.0004028906114399433,
+            0.016025975346565247,
+            -0.005289340391755104,
+            -0.005007583647966385,
+            0.014437892474234104,
+            -0.03558671474456787,
+            -0.003402424743399024,
+            0.006377945188432932,
+            -0.026211904361844063,
+            0.02484581060707569,
+            0.005276532843708992,
+            0.01605158858001232,
+            -0.013780459761619568,
+            0.0026318628806620836,
+            -0.006365138106048107,
+            0.004358689300715923,
+            0.018903307616710663,
+            0.007308595813810825,
+            0.004858167376369238,
+            -0.006689585279673338,
+            -0.008047140203416348,
+            0.016077203676104546,
+            -0.007415322121232748,
+            0.0018719738582149148,
+            0.016785863786935806,
+            0.018425175920128822,
+            0.012004539370536804,
+            0.0059553105384111404,
+            0.013071799650788307,
+            0.01750306226313114,
+            -0.007043915335088968,
+            -0.005835777148604393,
+            -0.013481627218425274,
+            -0.00802579429000616,
+            -0.0013105951948091388,
+            0.03859211876988411,
+            0.018920384347438812,
+            0.007385438773781061,
+            -0.02054261974990368,
+            0.013456013053655624,
+            0.0043928418308496475,
+            6.841302820248529e-06,
+            -0.0027385889552533627,
+            0.02293328195810318,
+            0.0024952536914497614,
+            0.007125027012079954,
+            0.006258412264287472,
+            -0.017212767153978348,
+            0.03309359401464462,
+            -0.020713381469249725,
+            0.0008532743086107075,
+            -0.013319403864443302,
+            -0.012166762724518776,
+            0.015530765987932682,
+            0.002104636514559388,
+            -0.004164448007941246,
+            -0.006296833511441946,
+            -0.0033575997222214937,
+            0.013515779748558998,
+            0.008499657735228539,
+            -0.012875423766672611,
+            0.004516643937677145,
+            -0.02542640082538128,
+            0.004315999336540699,
+            -0.015334390103816986,
+            0.021755026653409004,
+            -0.004559334367513657,
+            -0.004674598574638367,
+            -0.019261907786130905,
+            -0.017067620530724525,
+            -0.00999809056520462,
+            0.002403469290584326,
+            -0.004755710251629353,
+            0.004422725178301334,
+            -0.009468729607760906,
+            -0.008499657735228539,
+            0.0027599341701716185,
+            0.0036628360394388437,
+            0.012465596199035645,
+            0.013669464737176895,
+            0.019893724471330643,
+            0.017759205773472786,
+            -0.01536000519990921,
+            -0.0341864712536335,
+            0.0006056699785403907,
+            0.005084426142275333,
+            -0.0036436254158616066,
+            0.018664240837097168,
+            -0.001938143977895379,
+            -0.0009968207450583577,
+            -0.0007150641758926213,
+            0.0005419012159109116,
+            0.02004740945994854,
+            -0.011014122515916824,
+            -0.016657792031764984,
+            0.00021545309573411942,
+            0.008853988721966743,
+            0.00502039073035121,
+            -0.0025934416335076094,
+            -0.010296924039721489,
+            -0.0009455922991037369,
+            0.0020341973286122084,
+            0.006032153032720089,
+            -0.009101592935621738,
+            -0.0010496501345187426,
+            -0.0013671600027009845,
+            -0.0030544979963451624,
+            -0.006254143081605434,
+            -0.009844405576586723,
+            0.008222170174121857,
+            -0.012602205388247967,
+            -0.01123611256480217,
+            0.016222350299358368,
+            -0.015086785890161991,
+            0.001136631821282208,
+            0.023821242153644562,
+            0.022557605057954788,
+            -0.0008842248353175819,
+            0.006774966139346361,
+            -0.007936145178973675,
+            -0.011517869308590889,
+            -0.02785121463239193,
+            -0.019637582823634148,
+            -0.00806848518550396,
+            0.017947042360901833,
+            -0.002053408185020089,
+            -0.004049184266477823,
+            -0.0018869155319407582,
+            -0.008580769412219524,
+            0.008444160223007202,
+            -0.013814612291753292,
+            0.011620325967669487,
+            -0.034425534307956696,
+            0.013088875450193882,
+            -0.006702392362058163,
+            0.0010485828388482332,
+            0.015009943395853043,
+            0.04351005330681801,
+            -0.007351286243647337,
+            0.0029541754629462957,
+            0.001540056080557406,
+            0.01389145478606224,
+            0.028209814801812172,
+            0.004369362257421017,
+            0.0006136744632385671,
+            -0.01765674725174904,
+            0.004091874696314335,
+            0.002264725510030985,
+            -0.013456013053655624,
+            0.009579724632203579,
+            -0.023035738617181778,
+            -0.023155272006988525,
+            -0.009067440405488014,
+            -0.021123208105564117,
+            -0.002608383307233453,
+            0.002011785050854087,
+            0.012883962132036686,
+            -0.009878558106720448,
+            -0.012098458595573902,
+            -0.0005656477296724916,
+            -0.032991137355566025,
+            0.016709022223949432,
+            0.09665105491876602,
+            0.00875153113156557,
+            -0.003908305894583464,
+            -0.0021515958942472935,
+            0.002499522641301155,
+            0.009562648832798004,
+            -0.004768517334014177,
+            0.055360905826091766,
+            -0.018852079287171364,
+            0.008311820216476917,
+            -0.0008105839369818568,
+            0.008064216002821922,
+            -0.007223214954137802,
+            0.015530765987932682,
+            -0.012209453620016575,
+            -0.005951041355729103,
+            -0.009460192173719406,
+            -0.01123611256480217,
+            -0.0017246920615434647,
+            -0.016640717163681984,
+            -0.004986238200217485,
+            -0.020645076408982277,
+            0.013336479663848877,
+            -0.03234224393963814,
+            -0.011509330943226814,
+            -0.012474133633077145,
+            -0.0063224476762115955,
+            0.006066305562853813,
+            -0.015684451907873154,
+            -0.002595576224848628,
+            -0.002099300269037485,
+            0.015402695164084435,
+            0.0013863706262782216,
+            -0.00232235761359334,
+            -0.008328896015882492,
+            0.0005405671545304358,
+            0.005024659913033247,
+            0.003940323833376169,
+            0.006557244807481766,
+            0.006792041938751936,
+            -0.005848584230989218,
+            -0.008811297826468945,
+            0.010655523277819157,
+            -0.002510195365175605,
+            -0.0027919518761336803,
+            -0.008871064521372318,
+            0.010382304899394512,
+            -0.013481627218425274,
+            0.018356870859861374,
+            -0.013840226456522942,
+            -0.029507601633667946,
+            -0.015547842718660831,
+            0.02100367471575737,
+            -0.01996202953159809,
+            -0.0039381892420351505,
+            0.010553065687417984,
+            0.02829519473016262,
+            0.004113219678401947,
+            -0.001529383473098278,
+            -0.004909395705908537,
+            -0.017204230651259422,
+            0.009511420503258705,
+            0.008196556009352207,
+            0.018442250788211823,
+            -0.012371676973998547,
+            0.016487032175064087,
+            -0.018288565799593925,
+            0.006177300587296486,
+            -0.01663217879831791,
+            0.0006083381595090032,
+            0.00011866596469189972,
+            -0.0041793896816670895,
+            0.01608574204146862,
+            -0.01534292846918106,
+            -0.026263132691383362,
+            0.01900576427578926,
+            -0.013635313138365746,
+            -0.03307652100920677,
+            0.003374675754457712,
+            -0.034937821328639984,
+            -0.0025827689096331596,
+            0.009716334752738476,
+            -0.02149888314306736,
+            -0.018305642530322075,
+            0.007487895432859659,
+            -0.010066395625472069,
+            -0.008871064521372318,
+            0.00502039073035121,
+            -0.01689685881137848,
+            -0.012149686925113201,
+            -0.001152640674263239,
+            0.006753620691597462,
+            -0.00499477656558156,
+            0.0057632033713161945,
+            -0.026075294241309166,
+            0.014010988175868988,
+            -0.0170249305665493,
+            0.01144102681428194,
+            -0.010587218217551708,
+            0.022762520238757133,
+            0.011748396791517735,
+            -0.010356690734624863,
+            -0.0008303282083943486,
+            0.006036422215402126,
+            0.0005264259525574744,
+            0.026126522570848465,
+            0.03237639740109444,
+            -0.0011889275629073381,
+            0.008469774387776852,
+            0.007586083374917507,
+            0.003921112976968288,
+            -0.011014122515916824,
+            0.0015923518221825361,
+            0.014830644242465496,
+            0.01992787793278694,
+            -0.013729231432080269,
+            0.008879602886736393,
+            -0.026672959327697754,
+            0.02103782817721367,
+            -0.022096550092101097,
+            -0.000982412719167769,
+            0.010066395625472069,
+            -0.004777055233716965,
+            0.007090874947607517,
+            0.0066682398319244385,
+            0.021362274885177612,
+            0.03375956416130066,
+            0.00807702261954546,
+            0.008879602886736393,
+            0.007257367484271526,
+            -0.004798400681465864,
+            0.002044870052486658,
+            0.009135744534432888,
+            -0.005869929678738117,
+            -0.0040940092876553535,
+            -0.010083471424877644,
+            -0.006685316096991301,
+            0.016205275431275368,
+            0.0008079157560132444,
+            0.012602205388247967,
+            0.007018301170319319,
+            -0.007428129203617573,
+            -0.005669284611940384,
+            0.007014031987637281,
+            -0.023787088692188263,
+            -0.00950288213789463,
+            0.025221485644578934,
+            -0.013814612291753292,
+            -0.043202683329582214,
+            -0.0024333526380360126,
+            0.008853988721966743,
+            -0.017964119091629982,
+            -0.01245705783367157,
+            -0.004772786516696215,
+            0.010117623955011368,
+            0.0007241358398459852,
+            -0.007569007575511932,
+            0.028175661340355873,
+            0.00502039073035121,
+            -0.005169807001948357,
+            0.003571051638573408,
+            -2.8232359909452498e-05,
+            0.019244831055402756,
+            0.01536000519990921,
+            0.008294744417071342,
+            -0.03266669064760208,
+            -0.007304327096790075,
+            -0.005780279636383057,
+            0.013114489614963531,
+            0.002813297091051936,
+            0.003423769725486636,
+            -0.019740039482712746,
+            -0.020798761397600174,
+            0.019210679456591606,
+            0.004935009870678186,
+            0.019825419411063194,
+            -0.00867895781993866,
+            -0.006548706907778978,
+            -0.004841091111302376,
+            -0.004742903169244528,
+            0.00499477656558156,
+            0.0076415808871388435,
+            0.0038933642208576202,
+            0.03377664089202881,
+            -0.001280711847357452,
+            0.004042780492454767,
+            -0.0036350872833281755,
+            -0.0002441357064526528,
+            0.014787953346967697,
+            -0.00790199264883995,
+            0.008905217051506042,
+            0.018800850957632065,
+            -0.02887578494846821,
+            0.011398335918784142,
+            -0.01679440215229988,
+            -0.00828193686902523,
+            0.016000360250473022,
+            -0.0005320290802046657,
+            -0.004922202788293362,
+            -0.014583039097487926,
+            0.0008127184119075537,
+            0.007752575911581516,
+            0.01996202953159809,
+            -0.004091874696314335,
+            0.011287340894341469,
+            -0.009648029692471027,
+            -0.004305326379835606,
+            0.010083471424877644,
+            0.006087650544941425,
+            -0.0033213128335773945,
+            0.006232798099517822
+          ],
+          "keyphrases": [
+            "MoE models",
+            "expert choice",
+            "training efficiency",
+            "fine-tuning on GLUE and SuperGLUE",
+            "capacity factor"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": true,
+      "properties": {
+        "cosine_similarity": 0.8285278694280345
+      }
+    },
+    {
+      "id": "f7a0c54d-3589-41e9-a6cd-57a7760303fb",
+      "type": "cosine_similarity",
+      "source": {
+        "id": "33412f87-7418-4ce2-9116-377a27fd2732",
+        "properties": {
+          "page_content": "4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n",
+          "title": "4 Experiments",
+          "embedding": [
+            0.009938324801623821,
+            -0.009144282899796963,
+            -0.026211904361844063,
+            -0.022352691739797592,
+            0.016043050214648247,
+            0.016247965395450592,
+            -0.02783413790166378,
+            0.0537557490170002,
+            -0.02827811799943447,
+            -0.01755429059267044,
+            0.010237157344818115,
+            0.001297788112424314,
+            0.014762339182198048,
+            -0.01001516729593277,
+            0.007487895432859659,
+            0.015283161774277687,
+            0.019347287714481354,
+            0.019910801202058792,
+            0.02535809576511383,
+            -0.04251963645219803,
+            0.0020213902462273836,
+            -0.032034873962402344,
+            -0.058332156389951706,
+            0.0034600566141307354,
+            0.001006959704682231,
+            0.01755429059267044,
+            0.011645940132439137,
+            -0.0015795447397977114,
+            -0.053584985435009,
+            0.04794985428452492,
+            0.016555335372686386,
+            0.0044910297729074955,
+            -0.007833687588572502,
+            -0.004798400681465864,
+            -0.04204150289297104,
+            -0.028961164876818657,
+            0.05720513314008713,
+            0.005165537819266319,
+            -0.005519868340343237,
+            0.05481446906924248,
+            0.020252324640750885,
+            0.010245694778859615,
+            -0.03961668908596039,
+            -0.009904172271490097,
+            0.010134699754416943,
+            0.001490962109528482,
+            -0.0010427129454910755,
+            -0.007479357533156872,
+            0.02295035682618618,
+            -0.04146091267466545,
+            -0.005549751687794924,
+            -0.03811398521065712,
+            0.01863008923828602,
+            0.013780459761619568,
+            -0.04426140338182449,
+            0.023223575204610825,
+            -0.0052978782914578915,
+            0.03527934476733208,
+            0.057854026556015015,
+            0.02783413790166378,
+            0.024162763729691505,
+            -0.02590453252196312,
+            -0.04340759664773941,
+            -0.0036180110182613134,
+            0.010143238119781017,
+            0.008119713515043259,
+            -0.01344747468829155,
+            -0.010527451522648335,
+            0.046173933893442154,
+            -0.023701708763837814,
+            -0.05327761545777321,
+            0.027680452913045883,
+            -0.013584083877503872,
+            -0.008081291802227497,
+            0.0036372218746691942,
+            -0.03517688810825348,
+            0.012807118706405163,
+            0.03548425808548927,
+            0.01004078146070242,
+            0.015018481761217117,
+            0.001224147155880928,
+            0.008418546058237553,
+            -0.0014920294051989913,
+            0.017912890762090683,
+            0.028517184779047966,
+            -0.01417321152985096,
+            -0.006309640593826771,
+            -0.0686461552977562,
+            0.025767924264073372,
+            -0.007368362508714199,
+            -0.020115714520215988,
+            0.00026281276950612664,
+            -0.007099412847310305,
+            0.0033597343135625124,
+            -0.020781684666872025,
+            0.07472527027130127,
+            -0.013105952180922031,
+            0.014386664144694805,
+            -0.02096952311694622,
+            0.007300057914108038,
+            -0.031932417303323746,
+            0.02236976847052574,
+            0.01146664097905159,
+            0.005916888825595379,
+            0.038318898528814316,
+            -0.02834642305970192,
+            -0.005869929678738117,
+            0.0122009152546525,
+            0.0292856115847826,
+            0.08387809246778488,
+            0.01419028826057911,
+            -0.014352511614561081,
+            -0.025563009083271027,
+            -0.016538260504603386,
+            -0.046105626970529556,
+            -0.0292856115847826,
+            -0.025170257315039635,
+            -0.027099864557385445,
+            0.036338064819574356,
+            -0.0010613900376483798,
+            0.012388752773404121,
+            0.05338007211685181,
+            0.015863751992583275,
+            -0.00874299369752407,
+            -0.039241012185811996,
+            0.0438857264816761,
+            -0.006198645569384098,
+            0.010809208266437054,
+            0.002330895746126771,
+            -0.0146428057923913,
+            -0.006185838486999273,
+            0.011833777651190758,
+            -0.013763383962213993,
+            -0.029456373304128647,
+            0.031949494034051895,
+            -0.01727253384888172,
+            0.03944592550396919,
+            0.06239628419280052,
+            -0.03562086820602417,
+            -0.02001325786113739,
+            0.03189826384186745,
+            -0.035825781524181366,
+            0.018288565799593925,
+            -0.0005250918911769986,
+            -0.030993228778243065,
+            0.026246055960655212,
+            0.01562468521296978,
+            0.06092773377895355,
+            -0.024196917191147804,
+            0.028705023229122162,
+            -0.019176525995135307,
+            0.01628211699426174,
+            -0.020627999678254128,
+            0.03596239164471626,
+            0.0023351646959781647,
+            -0.024265222251415253,
+            -0.015752756968140602,
+            0.014694035053253174,
+            -0.014275669120252132,
+            -0.022830823436379433,
+            0.019740039482712746,
+            0.02100367471575737,
+            -0.010450609028339386,
+            -0.0049051265232264996,
+            0.012013077735900879,
+            0.04989653453230858,
+            0.016205275431275368,
+            -0.014130521565675735,
+            -0.07479357719421387,
+            0.04801815748214722,
+            0.021703798323869705,
+            0.01958635449409485,
+            0.023104043677449226,
+            0.035825781524181366,
+            -0.001056053675711155,
+            0.030566323548555374,
+            0.0019007899099960923,
+            -0.013601160608232021,
+            -0.02044016122817993,
+            0.043646663427352905,
+            -0.0039019023533910513,
+            -0.0019274713704362512,
+            0.0076373121701180935,
+            0.018288565799593925,
+            0.03123229369521141,
+            0.005430218297988176,
+            0.010399380698800087,
+            0.039719145745038986,
+            -0.011586173437535763,
+            0.01414759736508131,
+            0.009460192173719406,
+            0.02091829478740692,
+            0.04299776628613472,
+            0.02245514839887619,
+            -0.008670419454574585,
+            -0.02194286324083805,
+            0.00032551426556892693,
+            0.0219599399715662,
+            0.01811780408024788,
+            0.02199409157037735,
+            -0.012013077735900879,
+            -0.0220111683011055,
+            0.009383349679410458,
+            -0.02199409157037735,
+            0.01682855375111103,
+            -0.00167666538618505,
+            0.01859593763947487,
+            0.013865840621292591,
+            -0.006258412264287472,
+            -0.01677732542157173,
+            0.0002502724528312683,
+            -0.018186109140515327,
+            -0.008380125276744366,
+            -0.010945817455649376,
+            -0.016495568677783012,
+            0.03364003449678421,
+            -0.020747533068060875,
+            0.006599935237318277,
+            -0.01810072734951973,
+            -0.011885006912052631,
+            -0.005835777148604393,
+            0.03596239164471626,
+            0.012209453620016575,
+            -0.01851055584847927,
+            -0.026058217510581017,
+            -0.011108040809631348,
+            -0.018817927688360214,
+            -0.006463326048105955,
+            0.0342206209897995,
+            -0.01750306226313114,
+            0.006659701932221651,
+            -0.004211407620459795,
+            -0.002409872831776738,
+            0.004258367232978344,
+            0.033435117453336716,
+            -0.03082246705889702,
+            -0.01676025055348873,
+            0.023582175374031067,
+            0.008059946820139885,
+            0.029046546667814255,
+            -0.016888320446014404,
+            -0.006275488529354334,
+            -0.024231068789958954,
+            -0.016973702237010002,
+            -0.01643580198287964,
+            -0.005118578672409058,
+            -0.02199409157037735,
+            -0.016025975346565247,
+            0.02790244296193123,
+            0.013686541467905045,
+            -0.00790199264883995,
+            -0.019313136115670204,
+            -0.002333030104637146,
+            0.02542640082538128,
+            -0.013592622242867947,
+            0.04064125940203667,
+            0.050545431673526764,
+            -0.05713682621717453,
+            -0.0122521435841918,
+            -0.03661128506064415,
+            0.07684271782636642,
+            -0.015377080999314785,
+            -0.03032725863158703,
+            0.033964481204748154,
+            -0.029388070106506348,
+            -0.008990597911179066,
+            0.026997406035661697,
+            0.025272715836763382,
+            -0.014719649218022823,
+            0.01123611256480217,
+            0.03756754845380783,
+            -0.030139420181512833,
+            -0.0439881831407547,
+            -0.03777246177196503,
+            0.019859572872519493,
+            -0.025289790704846382,
+            -0.01419028826057911,
+            -0.0110311983153224,
+            0.023530947044491768,
+            0.01801534742116928,
+            0.010664060711860657,
+            0.007547662127763033,
+            0.006843270733952522,
+            0.021157361567020416,
+            0.02144765481352806,
+            0.007073798682540655,
+            -0.03237639740109444,
+            0.0003433908859733492,
+            -0.03944592550396919,
+            -0.009093054570257664,
+            -0.025289790704846382,
+            -0.02095244638621807,
+            -0.023821242153644562,
+            0.00949434470385313,
+            0.03275207430124283,
+            -0.0008143192972056568,
+            0.0067365444265306,
+            0.015044095925986767,
+            0.015488076023757458,
+            -0.03807983547449112,
+            -0.03555256128311157,
+            0.006348061840981245,
+            0.01682855375111103,
+            -0.0055625587701797485,
+            -0.059220120310783386,
+            -0.00042770442087203264,
+            -0.003165493020787835,
+            -0.005016121547669172,
+            -0.028909936547279358,
+            0.010476223193109035,
+            0.013345018029212952,
+            0.010237157344818115,
+            0.0110311983153224,
+            -0.027526767924427986,
+            -0.014651344157755375,
+            -0.0036073385272175074,
+            0.006019345950335264,
+            0.02100367471575737,
+            -0.040709562599658966,
+            -0.022830823436379433,
+            0.03818229213356972,
+            0.0003810117777902633,
+            -0.06598227471113205,
+            -0.005387527868151665,
+            -0.010757979936897755,
+            0.015436847694218159,
+            -0.0005656477296724916,
+            0.034476764500141144,
+            -0.03121521882712841,
+            -0.019296059384942055,
+            -0.020815838128328323,
+            -0.051979828625917435,
+            -0.03531349450349808,
+            -0.015436847694218159,
+            0.007389707490801811,
+            -0.038353051990270615,
+            -0.024265222251415253,
+            0.004111085087060928,
+            -0.0268949493765831,
+            0.007398245856165886,
+            -0.013567008078098297,
+            0.015633223578333855,
+            -0.03852381557226181,
+            -0.010092009790241718,
+            0.004262635949999094,
+            -0.01861301250755787,
+            -0.005507061257958412,
+            0.014369587413966656,
+            0.013925607316195965,
+            0.014719649218022823,
+            0.0017268265364691615,
+            0.015266085974872112,
+            0.0062071834690868855,
+            -0.00769280968233943,
+            0.007833687588572502,
+            -0.04091447591781616,
+            0.0014461372047662735,
+            0.018852079287171364,
+            0.011432488448917866,
+            -0.011082426644861698,
+            0.004151640925556421,
+            -0.010373766534030437,
+            -0.03034433349967003,
+            0.029917430132627487,
+            -0.05419972911477089,
+            0.0024141420144587755,
+            0.005554020404815674,
+            -0.016410188749432564,
+            -0.008384393528103828,
+            -0.009690719656646252,
+            -0.03708941861987114,
+            -0.01248267199844122,
+            0.0039019023533910513,
+            0.00019050588889513165,
+            0.009349197149276733,
+            0.002870929194614291,
+            0.05471201241016388,
+            -0.003165493020787835,
+            -0.04484199360013008,
+            0.023308956995606422,
+            -0.03671374171972275,
+            0.018749622628092766,
+            0.04245132952928543,
+            0.020713381469249725,
+            -0.017212767153978348,
+            0.04791570082306862,
+            0.0195692777633667,
+            0.000224524803343229,
+            -0.016999315470457077,
+            0.025153182446956635,
+            -0.007099412847310305,
+            -0.061303410679101944,
+            -0.004448339343070984,
+            -0.018749622628092766,
+            0.018664240837097168,
+            0.016674868762493134,
+            -0.002407738473266363,
+            -0.010032243095338345,
+            0.009426039643585682,
+            -0.01510386262089014,
+            -0.017443295568227768,
+            0.01705908216536045,
+            -0.011458102613687515,
+            -0.004525181837379932,
+            0.007526317145675421,
+            0.013148642145097256,
+            0.01271320041269064,
+            0.008853988721966743,
+            0.023992003872990608,
+            -0.045627497136592865,
+            0.009127207100391388,
+            -0.01997910626232624,
+            -0.036918655037879944,
+            -0.002990462351590395,
+            0.01318279467523098,
+            0.04183658957481384,
+            -0.02779998630285263,
+            0.0019253368955105543,
+            0.005007583647966385,
+            -0.013515779748558998,
+            -0.021106131374835968,
+            0.00341309723444283,
+            0.04600317031145096,
+            -0.0015763428527861834,
+            0.021618416532874107,
+            -0.011304416693747044,
+            -0.03562086820602417,
+            0.008811297826468945,
+            0.04142676293849945,
+            -0.005028928630053997,
+            0.01663217879831791,
+            0.009434578008949757,
+            0.03548425808548927,
+            -0.04590071365237236,
+            -0.007833687588572502,
+            -0.02344556525349617,
+            -0.01813488081097603,
+            0.04105108603835106,
+            -0.03075416199862957,
+            -0.01365238893777132,
+            0.03524519130587578,
+            -0.005306416191160679,
+            0.02441890724003315,
+            -0.018271489068865776,
+            -0.02588745579123497,
+            0.0014600115828216076,
+            -0.03712356835603714,
+            -0.00708660576492548,
+            -0.01806657575070858,
+            -0.006160224322229624,
+            -0.06557244807481766,
+            0.00010459146869834512,
+            -0.07431544363498688,
+            -0.002902946900576353,
+            0.025136105716228485,
+            -0.006552976090461016,
+            0.04429555684328079,
+            -0.026228979229927063,
+            0.02098659984767437,
+            -0.05553166940808296,
+            -0.01367800310254097,
+            0.00978463888168335,
+            0.03375956416130066,
+            0.007611697539687157,
+            -0.001995776081457734,
+            0.0015507286880165339,
+            -0.026997406035661697,
+            -0.0018303507240489125,
+            -0.007282981649041176,
+            -0.001387437921948731,
+            4.89605481561739e-05,
+            -0.018852079287171364,
+            0.00950288213789463,
+            0.000838332693092525,
+            0.01610281690955162,
+            0.031454283744096756,
+            0.022711291909217834,
+            -0.03070293366909027,
+            -0.008862526156008244,
+            0.005195421166718006,
+            0.02587038092315197,
+            -0.02978082001209259,
+            0.003897633170709014,
+            -0.03464752435684204,
+            0.014864795841276646,
+            5.636466448777355e-05,
+            0.016657792031764984,
+            0.008008718490600586,
+            0.017016392201185226,
+            -0.040197279304265976,
+            0.0030160765163600445,
+            -0.0016254368238151073,
+            -0.027202321216464043,
+            0.020867066457867622,
+            0.002305281348526478,
+            -0.022147778421640396,
+            0.0054472945630550385,
+            -0.004207138437777758,
+            0.024213993921875954,
+            0.04473953694105148,
+            -0.0010592554463073611,
+            -0.02930268831551075,
+            -0.012704662047326565,
+            0.042212262749671936,
+            0.02633143588900566,
+            -0.028483033180236816,
+            0.017118848860263824,
+            0.02103782817721367,
+            -0.02341141365468502,
+            -0.021089056506752968,
+            0.013925607316195965,
+            -0.016179660335183144,
+            -0.0007694943924434483,
+            -0.005468640010803938,
+            0.010399380698800087,
+            0.01240582950413227,
+            -0.029883278533816338,
+            0.05676115304231644,
+            -0.016820017248392105,
+            -0.006834732368588448,
+            -0.03034433349967003,
+            -0.006168762221932411,
+            -0.014531810767948627,
+            0.05553166940808296,
+            0.0028175662737339735,
+            0.02728770114481449,
+            0.01658095046877861,
+            0.002264725510030985,
+            0.05628301948308945,
+            -0.03678204491734505,
+            -0.03451091796159744,
+            0.018681317567825317,
+            -0.0055625587701797485,
+            -0.00671093026176095,
+            -0.022728366777300835,
+            -0.014207364059984684,
+            -0.013353556394577026,
+            0.016965163871645927,
+            -0.0268778745085001,
+            -0.012260681949555874,
+            0.019740039482712746,
+            0.017690900713205338,
+            -2.0469710761972237e-06,
+            0.02293328195810318,
+            0.012644895352423191,
+            0.014284206554293633,
+            0.017878737300634384,
+            -0.020867066457867622,
+            0.019142374396324158,
+            0.024111535400152206,
+            0.009946862235665321,
+            -0.01763967238366604,
+            -0.017878737300634384,
+            0.007347017526626587,
+            -0.0038613462820649147,
+            -0.007125027012079954,
+            0.01437812577933073,
+            0.007043915335088968,
+            -0.004047049675136805,
+            -0.027151092886924744,
+            -0.018356870859861374,
+            0.02093537151813507,
+            -0.0023757205344736576,
+            0.01539415679872036,
+            -0.00928943045437336,
+            -0.010612832382321358,
+            0.03763585537672043,
+            0.01861301250755787,
+            -0.046139780431985855,
+            0.026280207559466362,
+            0.01051037572324276,
+            -0.006228528916835785,
+            -0.013729231432080269,
+            0.005626594182103872,
+            0.009101592935621738,
+            0.010220080614089966,
+            0.016572412103414536,
+            -0.005097233224660158,
+            -0.03328143432736397,
+            0.01564176194369793,
+            0.026211904361844063,
+            -0.020798761397600174,
+            0.012021616101264954,
+            0.00017663151083979756,
+            0.022762520238757133,
+            -0.005028928630053997,
+            -0.012311910279095173,
+            -0.01562468521296978,
+            0.009741948917508125,
+            -0.012235067784786224,
+            0.01123611256480217,
+            0.037396788597106934,
+            0.015914980322122574,
+            -0.03565501794219017,
+            0.015735680237412453,
+            -0.01686270721256733,
+            -0.015684451907873154,
+            0.03261546418070793,
+            0.008264861069619656,
+            -0.02636558935046196,
+            -0.007142103277146816,
+            -0.006527361460030079,
+            0.02001325786113739,
+            -0.02102075144648552,
+            -0.008508196100592613,
+            0.004418455995619297,
+            0.0035646480973809958,
+            0.009938324801623821,
+            0.006745082791894674,
+            0.005016121547669172,
+            0.019159449264407158,
+            0.013806073926389217,
+            -0.0005186883499845862,
+            0.012730276212096214,
+            0.011577636003494263,
+            0.002902946900576353,
+            0.0075391242280602455,
+            -0.010997045785188675,
+            -0.02295035682618618,
+            0.004734364803880453,
+            0.05945918336510658,
+            -0.005852853413671255,
+            0.03876287862658501,
+            0.010407919064164162,
+            0.028961164876818657,
+            0.021686721593141556,
+            -0.024299373850226402,
+            -0.01681147888302803,
+            -0.018151957541704178,
+            -0.021737949922680855,
+            -0.04453462362289429,
+            -0.01510386262089014,
+            -0.01629919372498989,
+            -0.013404784724116325,
+            0.005694898776710033,
+            -0.00013474156730808318,
+            -0.0013116624904796481,
+            0.011355645954608917,
+            -0.002003246918320656,
+            -0.012491210363805294,
+            -0.003355465130880475,
+            0.05324346199631691,
+            -0.0011056812945753336,
+            0.017912890762090683,
+            -0.004576410632580519,
+            0.015539304353296757,
+            -0.007300057914108038,
+            -0.016111355274915695,
+            0.029131926596164703,
+            0.0035561099648475647,
+            -0.0007540191290900111,
+            0.012106996960937977,
+            0.004469684325158596,
+            0.04149506613612175,
+            -0.005887005943804979,
+            -0.026058217510581017,
+            0.011910621076822281,
+            -0.017187153920531273,
+            0.020371858030557632,
+            0.00769707839936018,
+            -0.007607428822666407,
+            0.001932807732373476,
+            0.012909576296806335,
+            -0.027116939425468445,
+            0.019705887883901596,
+            -0.020645076408982277,
+            -0.045149363577365875,
+            0.009280892089009285,
+            -0.01175693515688181,
+            0.023035738617181778,
+            0.001117421081289649,
+            -0.010800670832395554,
+            -0.03507443144917488,
+            0.0021387888118624687,
+            0.0039040367119014263,
+            -0.03372541442513466,
+            0.006548706907778978,
+            0.01863008923828602,
+            0.015539304353296757,
+            0.009485806338489056,
+            -0.012465596199035645,
+            0.003758889390155673,
+            0.0031847036443650723,
+            -0.012371676973998547,
+            0.02776583470404148,
+            -0.004742903169244528,
+            -0.031983643770217896,
+            0.035859934985637665,
+            -0.02733892947435379,
+            0.009468729607760906,
+            -0.020730456337332726,
+            -0.012346062809228897,
+            -0.01608574204146862,
+            0.00441418681293726,
+            -0.02148180827498436,
+            -0.002561423694714904,
+            -0.02983205020427704,
+            0.027577996253967285,
+            0.009383349679410458,
+            -0.015249010175466537,
+            -0.004862436093389988,
+            0.009101592935621738,
+            -0.025255639106035233,
+            0.007846495136618614,
+            0.012585128657519817,
+            -0.010646984912455082,
+            -0.01367800310254097,
+            -0.00706099160015583,
+            -0.0014685495989397168,
+            0.005972386337816715,
+            0.0158210601657629,
+            0.04672037065029144,
+            0.004431263078004122,
+            0.03712356835603714,
+            0.02337726205587387,
+            0.013532855547964573,
+            0.0037012575194239616,
+            0.0047898623161017895,
+            -0.0067322757095098495,
+            0.04012897238135338,
+            0.006249873898923397,
+            0.0219599399715662,
+            0.008247784338891506,
+            -0.044671230018138885,
+            0.008503926917910576,
+            0.03264961391687393,
+            0.026621730998158455,
+            0.018715469166636467,
+            0.009588262997567654,
+            -0.0024013349320739508,
+            0.021191513165831566,
+            -0.020252324640750885,
+            -0.01705908216536045,
+            0.0035902622621506453,
+            -0.007530585862696171,
+            -0.0056650154292583466,
+            -0.009716334752738476,
+            0.03387909755110741,
+            -0.003229528432711959,
+            -0.008862526156008244,
+            0.018186109140515327,
+            -0.007432397920638323,
+            0.00475997943431139,
+            0.006245605181902647,
+            0.0011163539020344615,
+            0.02739015780389309,
+            0.008683227002620697,
+            0.027680452913045883,
+            0.005391797050833702,
+            0.007573276292532682,
+            0.0058571225963532925,
+            -0.009451653808355331,
+            0.018920384347438812,
+            -0.0097590247169137,
+            -0.033366814255714417,
+            -0.009443115442991257,
+            0.0016393112018704414,
+            0.014070754870772362,
+            0.0004023569927085191,
+            -0.014062216505408287,
+            -0.011893544346094131,
+            0.007859301753342152,
+            0.012533900327980518,
+            0.02107197977602482,
+            0.018169032409787178,
+            -0.005016121547669172,
+            -0.024726277217268944,
+            0.012593667022883892,
+            0.011056812480092049,
+            -0.01861301250755787,
+            0.008465506136417389,
+            0.012542438693344593,
+            0.057410046458244324,
+            -0.044705383479595184,
+            0.016538260504603386,
+            0.02151595987379551,
+            -0.004896588623523712,
+            -0.015249010175466537,
+            -0.010851899161934853,
+            0.0031633584294468164,
+            -0.017477449029684067,
+            0.014975790865719318,
+            -0.01798119582235813,
+            0.011193421669304371,
+            -0.014745263382792473,
+            0.00743666710332036,
+            0.012866885401308537,
+            -0.009682182222604752,
+            0.003942457959055901,
+            0.020371858030557632,
+            0.0028068935498595238,
+            -0.0077867284417152405,
+            0.005387527868151665,
+            0.01390853151679039,
+            0.020354781299829483,
+            0.006066305562853813,
+            -0.0005933965439908206,
+            0.019876649603247643,
+            -0.028175661340355873,
+            -0.025170257315039635,
+            0.01909114606678486,
+            -0.03551841154694557,
+            0.005472908727824688,
+            0.02790244296193123,
+            -0.04296361654996872,
+            -0.05020390823483467,
+            0.0027407235465943813,
+            -0.039719145745038986,
+            0.0004191663465462625,
+            -0.0023287611547857523,
+            0.01536854263395071,
+            -0.004057722166180611,
+            -0.017212767153978348,
+            0.0146428057923913,
+            -0.012397291138768196,
+            -0.0067322757095098495,
+            0.009835867211222649,
+            0.009016212075948715,
+            -0.024572592228651047,
+            -0.0016222350532189012,
+            -0.030890770256519318,
+            0.009169897064566612,
+            -0.017844585701823235,
+            -0.019142374396324158,
+            -0.025528857484459877,
+            -0.0005656477296724916,
+            -0.010638446547091007,
+            -0.001638244022615254,
+            0.051501695066690445,
+            0.021652569994330406,
+            0.020166942849755287,
+            -0.007726961746811867,
+            -0.023650480434298515,
+            0.05457540228962898,
+            0.0018132745753973722,
+            0.03814813867211342,
+            -0.012380215339362621,
+            -0.0002006448630709201,
+            -0.0008852921309880912,
+            0.022113624960184097,
+            -0.017101772129535675,
+            -0.0011547752656042576,
+            -0.011022660881280899,
+            -0.00586566049605608,
+            -0.0070780678652226925,
+            0.014045140706002712,
+            0.015974747017025948,
+            -0.002610517665743828,
+            0.014156135730445385,
+            0.01221799198538065,
+            0.022694215178489685,
+            0.03130060061812401,
+            0.016230888664722443,
+            -0.0034664601553231478,
+            -0.022199006751179695,
+            -0.012918113730847836,
+            -0.008200825192034245,
+            0.008999135345220566,
+            -0.008525271899998188,
+            -0.0031847036443650723,
+            -0.02631436102092266,
+            -0.020730456337332726,
+            -0.011312955059111118,
+            -0.01292665209621191,
+            -0.0031462821643799543,
+            0.02641681768000126,
+            -0.002469639526680112,
+            -0.03763585537672043,
+            -0.01511240005493164,
+            -0.03524519130587578,
+            -0.015607609413564205,
+            -0.0017876604106277227,
+            -0.01748598739504814,
+            0.016538260504603386,
+            -0.003780234605073929,
+            -0.007833687588572502,
+            -0.0008964983280748129,
+            -0.018169032409787178,
+            0.009690719656646252,
+            -0.011048275046050549,
+            -0.000543768925126642,
+            -0.0017503062263131142,
+            0.013345018029212952,
+            -0.008205094374716282,
+            0.008973521180450916,
+            -0.01412198320031166,
+            0.006868884898722172,
+            -0.048120614141225815,
+            -0.027731681242585182,
+            -0.027099864557385445,
+            0.022557605057954788,
+            -0.015214857645332813,
+            -0.023189423605799675,
+            0.008734455332159996,
+            -0.013686541467905045,
+            0.008939368650317192,
+            -0.012021616101264954,
+            0.008388662710785866,
+            0.019210679456591606,
+            0.009024749509990215,
+            -0.002614786848425865,
+            -0.013669464737176895,
+            -0.02390662208199501,
+            -0.02585330419242382,
+            0.023650480434298515,
+            -0.026485122740268707,
+            0.01071528997272253,
+            -0.012021616101264954,
+            0.008585038594901562,
+            -0.029917430132627487,
+            -0.020730456337332726,
+            0.0044910297729074955,
+            0.011347107589244843,
+            -0.021140284836292267,
+            0.044671230018138885,
+            0.04043634235858917,
+            0.029985735192894936,
+            -0.020252324640750885,
+            0.05139923840761185,
+            -0.022796671837568283,
+            0.007312864996492863,
+            0.036440521478652954,
+            0.030446790158748627,
+            0.026092370972037315,
+            -0.015581995248794556,
+            -0.012294834479689598,
+            -0.015496614389121532,
+            0.013404784724116325,
+            -0.024333525449037552,
+            0.002821835223585367,
+            -0.02042308636009693,
+            0.013643850572407246,
+            0.03451091796159744,
+            -0.004917933605611324,
+            0.029934506863355637,
+            -0.013720693998038769,
+            0.030361410230398178,
+            0.0017652479000389576,
+            -0.026587579399347305,
+            0.015667375177145004,
+            -0.014232978224754333,
+            0.004418455995619297,
+            0.017964119091629982,
+            -0.006087650544941425,
+            0.013575546443462372,
+            -0.061337560415267944,
+            0.028141509741544724,
+            0.01758844405412674,
+            0.01123611256480217,
+            -0.002180411946028471,
+            -0.024726277217268944,
+            -0.0021142419427633286,
+            0.009340658783912659,
+            -0.0016745307948440313,
+            0.011859391815960407,
+            0.01193623524159193,
+            -0.033418044447898865,
+            0.010578680783510208,
+            -0.013635313138365746,
+            0.024299373850226402,
+            0.006258412264287472,
+            0.0007476155878975987,
+            -0.03961668908596039,
+            -0.00560524920001626,
+            0.014830644242465496,
+            0.01095435582101345,
+            -0.005989462602883577,
+            -0.04624223709106445,
+            -0.030105268582701683,
+            0.025563009083271027,
+            0.005110040307044983,
+            -0.05167245492339134,
+            0.014557424932718277,
+            -0.015334390103816986,
+            0.018766699358820915,
+            -0.008708841167390347,
+            -0.025972837582230568,
+            -0.029985735192894936,
+            -0.017187153920531273,
+            0.016726097092032433,
+            0.048564594238996506,
+            0.004207138437777758,
+            0.013345018029212952,
+            -0.0048538981936872005,
+            0.006651164032518864,
+            -0.024760430678725243,
+            0.012030153535306454,
+            -0.027151092886924744,
+            -0.02971251681447029,
+            0.029012393206357956,
+            0.009981014765799046,
+            -0.011330030858516693,
+            0.019330210983753204,
+            -0.017084697261452675,
+            -0.0014141193823888898,
+            0.02240392006933689,
+            -0.0006163425859995186,
+            -0.007295788731426001,
+            -0.010382304899394512,
+            0.013618236407637596,
+            -0.0014354645973071456,
+            -0.01762259565293789,
+            -0.009571187198162079,
+            0.04241717979311943,
+            0.010092009790241718,
+            0.018835002556443214,
+            0.03261546418070793,
+            -0.0018933190731331706,
+            -0.02400907874107361,
+            -0.0027215126901865005,
+            -0.026638807728886604,
+            -0.017033468931913376,
+            -0.014548887498676777,
+            0.016674868762493134,
+            0.023530947044491768,
+            0.015249010175466537,
+            -0.027526767924427986,
+            -0.030019886791706085,
+            -0.01950097270309925,
+            -0.022199006751179695,
+            -0.0034493841230869293,
+            -0.0004213008505757898,
+            -0.026502197608351707,
+            0.007005494087934494,
+            0.0341181643307209,
+            0.03357172757387161,
+            -0.0010870042024180293,
+            -0.009016212075948715,
+            -0.010638446547091007,
+            0.0007721625734120607,
+            0.015231933444738388,
+            -0.015018481761217117,
+            0.0012326851719990373,
+            0.005028928630053997,
+            0.005383259151130915,
+            0.0029285612981766462,
+            -0.007095144130289555,
+            -0.05747834965586662,
+            0.003961668815463781,
+            0.02544347569346428,
+            0.005084426142275333,
+            -0.0008377990452572703,
+            0.02629728429019451,
+            0.0008276600856333971,
+            0.008123982697725296,
+            -0.03142013028264046,
+            0.011108040809631348,
+            0.01803242415189743,
+            0.015163629315793514,
+            -0.018151957541704178,
+            0.0122521435841918,
+            0.0064206356182694435,
+            0.004811207763850689,
+            -0.00030950538348406553,
+            0.010143238119781017,
+            0.028636718168854713,
+            0.019620506092905998,
+            -0.014736725017428398,
+            0.023821242153644562,
+            0.0006435577524825931,
+            -0.012072844430804253,
+            -0.024333525449037552,
+            -0.001040044822730124,
+            0.0029883277602493763,
+            -0.011176345869898796,
+            0.014702572487294674,
+            -0.0023949311580508947,
+            0.0007956422632560134,
+            0.04139260947704315,
+            0.0437491200864315,
+            0.013481627218425274,
+            -0.011295879259705544,
+            0.003304236801341176,
+            0.026502197608351707,
+            0.020132791250944138,
+            0.004862436093389988,
+            0.011458102613687515,
+            -0.03331558406352997,
+            -0.019620506092905998,
+            -0.002981924219056964,
+            0.0014290610561147332,
+            -0.0026788224931806326,
+            0.03876287862658501,
+            0.029524678364396095,
+            0.009682182222604752,
+            -0.005741858389228582,
+            -0.011364183388650417,
+            0.013643850572407246,
+            0.020201096311211586,
+            -0.023599252104759216,
+            -0.0038015798199921846,
+            0.0016745307948440313,
+            -0.017170077189803123,
+            0.02288205362856388,
+            -0.0070311082527041435,
+            -0.00928943045437336,
+            0.02645096927881241,
+            0.09186973422765732,
+            -0.00464471522718668,
+            0.012141148559749126,
+            -0.03948007896542549,
+            -0.021823331713676453,
+            -0.006975610740482807,
+            -0.0036671049892902374,
+            0.015402695164084435,
+            0.02739015780389309,
+            0.009135744534432888,
+            0.007133565377444029,
+            0.017878737300634384,
+            -0.004606293980032206,
+            0.012508286163210869,
+            -0.02826104313135147,
+            0.020166942849755287,
+            -0.005007583647966385,
+            -0.0038335975259542465,
+            -0.011389797553420067,
+            -0.005989462602883577,
+            0.011765473522245884,
+            0.005468640010803938,
+            0.017016392201185226,
+            0.0019893725402653217,
+            -0.0024760430678725243,
+            0.0014333301223814487,
+            0.003031018190085888,
+            -0.010877513326704502,
+            -0.009383349679410458,
+            -0.021396426483988762,
+            0.003827193984761834,
+            0.0009914844995364547,
+            -0.007884915918111801,
+            0.012755890376865864,
+            0.005502792075276375,
+            0.024043232202529907,
+            -0.028175661340355873,
+            -0.0030609015375375748,
+            0.00633525475859642,
+            -0.008956445381045341,
+            0.010390842333436012,
+            0.012380215339362621,
+            -0.008640536107122898,
+            0.022830823436379433,
+            -0.01466841995716095,
+            0.014762339182198048,
+            0.01170570682734251,
+            -0.021601341664791107,
+            -0.004155910108238459,
+            -0.009938324801623821,
+            -0.02192578837275505,
+            0.020252324640750885,
+            -0.018715469166636467,
+            0.005033197812736034,
+            0.00073320756200701,
+            0.006774966139346361,
+            -0.0010496501345187426,
+            -0.01740914396941662,
+            0.010348152369260788,
+            -0.008785683661699295,
+            0.006839001551270485,
+            -0.005058811977505684,
+            0.018254414200782776,
+            0.012166762724518776,
+            0.0028090281412005424,
+            -0.0011622461024671793,
+            0.014600115828216076,
+            -0.00372900627553463,
+            -0.018886230885982513,
+            -0.0036713741719722748,
+            0.01633334532380104,
+            0.01051037572324276,
+            0.0002606782363727689,
+            -0.010305461473762989,
+            -0.0146342683583498,
+            0.014139058999717236,
+            0.000977076473645866,
+            -0.0018122073961421847,
+            -0.0005285604856908321,
+            -0.03760170191526413,
+            0.002606248715892434,
+            0.013532855547964573,
+            0.012704662047326565,
+            -0.005818700883537531,
+            -0.017477449029684067,
+            0.007983104325830936,
+            -0.004190062638372183,
+            -0.0003204447857569903,
+            0.01661510206758976,
+            -0.019774191081523895,
+            0.01002370472997427,
+            -0.014779414981603622,
+            -0.03278622403740883,
+            -0.0029776552692055702,
+            -0.008380125276744366,
+            0.005822970066219568,
+            -0.005592442117631435,
+            -0.0071805245243012905,
+            -0.01414759736508131,
+            -0.005741858389228582,
+            -0.02979789674282074,
+            0.008947907015681267,
+            0.0317445807158947,
+            0.004473953507840633,
+            -0.028978241607546806,
+            -0.012158225290477276,
+            0.025580085813999176,
+            -0.0024931191001087427,
+            -0.006915844511240721,
+            0.003009672975167632,
+            0.0012572321575134993,
+            -0.005784548819065094,
+            -0.009656568057835102,
+            -0.012935190461575985,
+            0.013063261285424232,
+            0.01320840884000063,
+            0.011244650930166245,
+            -0.03357172757387161,
+            0.010664060711860657,
+            0.01813488081097603,
+            -0.02042308636009693,
+            0.011748396791517735,
+            0.028585489839315414,
+            0.0029435029719024897,
+            0.010399380698800087,
+            0.007334209978580475,
+            0.008286206051707268,
+            -0.017153002321720123,
+            0.016683407127857208,
+            0.009186973795294762,
+            -0.012559514492750168,
+            0.004563603550195694,
+            0.007381169591099024,
+            0.008542348630726337,
+            0.01960342936217785,
+            -0.011603250168263912,
+            0.013609698973596096,
+            -0.0068816919811069965,
+            0.03321312740445137,
+            -0.011919158510863781,
+            -0.014881872572004795,
+            0.03229101747274399,
+            0.027560919523239136,
+            -0.004563603550195694,
+            0.017246920615434647,
+            -0.01943266950547695,
+            -0.01267050951719284,
+            0.019296059384942055,
+            0.0005907283630222082,
+            0.019313136115670204,
+            -0.011304416693747044,
+            -0.02595576085150242,
+            0.029012393206357956,
+            -0.008106905966997147,
+            -0.001259366748854518,
+            -0.015829598531126976,
+            0.0043458822183310986,
+            -0.0005789885181002319,
+            0.037328481674194336,
+            0.012653433717787266,
+            0.0011985328746959567,
+            0.026570502668619156,
+            -0.00769280968233943,
+            -0.025067800655961037,
+            0.001990439835935831,
+            -0.015214857645332813,
+            0.013336479663848877,
+            -0.027970748022198677,
+            -0.00560524920001626,
+            0.014506196603178978,
+            0.003026749240234494,
+            -0.002644670195877552,
+            0.01666633039712906,
+            -0.007513510063290596,
+            0.011791087687015533,
+            -0.019244831055402756,
+            0.016717558726668358,
+            -0.011517869308590889,
+            0.004473953507840633,
+            -0.011910621076822281,
+            0.026092370972037315,
+            -0.025563009083271027,
+            -0.00999809056520462,
+            0.013660927303135395,
+            -0.008354511111974716,
+            0.0006376878009177744,
+            -0.02252345345914364,
+            -0.008089830167591572,
+            -0.020371858030557632,
+            0.01679440215229988,
+            -0.005682091694325209,
+            0.011902082711458206,
+            -0.00306303589604795,
+            0.0025528857950121164,
+            -0.008585038594901562,
+            0.010322538204491138,
+            -0.0009199780761264265,
+            -0.017417682334780693,
+            -0.005968117620795965,
+            -0.03025895357131958,
+            -0.030361410230398178,
+            0.03072001039981842,
+            -0.00880275946110487,
+            0.003923247568309307,
+            0.02728770114481449,
+            -0.02540932409465313,
+            -0.005485715810209513,
+            0.001918933354318142,
+            0.022728366777300835,
+            -0.030651705339550972,
+            0.007914799265563488,
+            -0.009391887113451958,
+            -0.016452878713607788,
+            0.007184793706983328,
+            -0.0008874266059137881,
+            0.015880826860666275,
+            -0.0025635582860559225,
+            -0.04924764111638069,
+            -0.0007081269286572933,
+            0.017135925590991974,
+            -0.013806073926389217,
+            0.008081291802227497,
+            0.02151595987379551,
+            -0.009630953893065453,
+            -0.008439891040325165,
+            -0.0011334300506860018,
+            0.025989914312958717,
+            -0.022625910118222237,
+            0.030019886791706085,
+            0.005767472553998232,
+            0.004713019821792841,
+            0.005340568721294403,
+            0.002962713595479727,
+            0.01730668731033802,
+            0.002080089645460248,
+            0.004258367232978344,
+            0.025238562375307083,
+            0.013148642145097256,
+            0.0037418133579194546,
+            -0.0032978332601487637,
+            -0.0030352871399372816,
+            0.016538260504603386,
+            -0.018442250788211823,
+            0.02100367471575737,
+            -0.0004714620590675622,
+            0.00013560871593654156,
+            0.016256503760814667,
+            -0.01484772004187107,
+            -0.027219397947192192,
+            0.03075416199862957,
+            -0.022335615009069443,
+            0.01757136732339859,
+            0.023770013824105263,
+            -0.01339624635875225,
+            -0.004922202788293362,
+            -0.007073798682540655,
+            0.001534719718620181,
+            0.018339794129133224,
+            -0.0134218605235219,
+            -0.023308956995606422,
+            -0.00030790449818596244,
+            0.0220111683011055,
+            0.02293328195810318,
+            0.028175661340355873,
+            -0.021140284836292267,
+            0.01412198320031166,
+            0.007466550450772047,
+            -0.008311820216476917,
+            0.016469955444335938,
+            0.008452698588371277,
+            -0.0025272713974118233,
+            0.021293969824910164,
+            0.0075903525575995445,
+            0.004576410632580519,
+            -0.008896678686141968,
+            -0.013242561370134354,
+            -0.01267050951719284,
+            -0.006749351508915424,
+            -0.006164493504911661,
+            -0.008264861069619656,
+            -0.01709323562681675,
+            0.09548988193273544,
+            -0.011867930181324482,
+            -0.0028410458471626043,
+            -0.009195511229336262,
+            0.0026041141245514154,
+            -0.006949996575713158,
+            0.0013479492627084255,
+            0.023086966946721077,
+            -0.022728366777300835,
+            -0.03671374171972275,
+            0.011167807504534721,
+            -0.004070529248565435,
+            0.006288295611739159,
+            0.03712356835603714,
+            0.00038314631092362106,
+            -0.016589488834142685,
+            -6.773765198886395e-05,
+            -0.003372541395947337,
+            0.01295226626098156,
+            0.0028197006322443485,
+            -0.0036158766597509384,
+            0.0109799699857831,
+            0.029200231656432152,
+            -0.017810434103012085,
+            -0.02144765481352806,
+            -0.012525362893939018,
+            0.0032615463715046644,
+            -0.002578499959781766,
+            0.003707661060616374,
+            0.0051996903494000435,
+            -0.014053679071366787,
+            -0.0134218605235219,
+            0.018817927688360214,
+            -0.016094280406832695,
+            0.0008025794522836804,
+            -0.012627819553017616,
+            -0.008222170174121857,
+            0.01753721572458744,
+            0.01757136732339859,
+            -0.03406693786382675,
+            -0.009921248070895672,
+            -0.03855796530842781,
+            0.018749622628092766,
+            0.01643580198287964,
+            0.014540349133312702,
+            -0.020884141325950623,
+            -0.012644895352423191,
+            -0.004520913120359182,
+            -0.02981497347354889,
+            -0.02243807166814804,
+            -0.026536351069808006,
+            0.009605339728295803,
+            0.00365856708958745,
+            0.027595072984695435,
+            0.005259457044303417,
+            -0.006262681446969509,
+            0.00801725685596466,
+            -0.01025423314422369,
+            0.00755620002746582,
+            -0.02146473154425621,
+            0.007107951212674379,
+            -0.0011409008875489235,
+            0.0012529630912467837,
+            0.005592442117631435,
+            -0.008853988721966743,
+            -0.01735791563987732,
+            -0.0079276068136096,
+            0.01466841995716095,
+            -0.015752756968140602,
+            -0.00821790099143982,
+            0.020798761397600174,
+            0.009383349679410458,
+            0.00931504461914301,
+            0.015616146847605705,
+            0.045661646872758865,
+            0.008273398503661156,
+            0.00783795677125454,
+            -0.011056812480092049,
+            0.009648029692471027,
+            -0.01813488081097603,
+            0.00745801255106926,
+            -0.019330210983753204,
+            -0.006292564328759909,
+            -0.019722962751984596,
+            0.04012897238135338,
+            -0.009366272948682308,
+            -0.01635042205452919,
+            0.033435117453336716,
+            -0.011808163486421108,
+            0.01910822093486786,
+            0.0035475720651447773,
+            -0.022830823436379433,
+            0.03352050110697746,
+            0.003876287955790758,
+            0.012918113730847836,
+            -0.025682542473077774,
+            0.0035859933122992516,
+            -0.00877714529633522,
+            0.006032153032720089,
+            -0.011586173437535763,
+            0.003494208911433816,
+            0.0023885276168584824,
+            -0.004841091111302376,
+            -0.008760069496929646,
+            0.018698394298553467,
+            0.006595666520297527,
+            0.0002183880569646135,
+            0.028056127950549126,
+            -0.000954130373429507,
+            0.018220260739326477,
+            6.110063259257004e-05,
+            0.006010808050632477,
+            0.027680452913045883,
+            0.002821835223585367,
+            0.006809118203818798,
+            -0.01714446395635605,
+            -0.0009199780761264265,
+            -0.009485806338489056,
+            0.009596801362931728,
+            0.02639974094927311,
+            -0.010612832382321358,
+            -0.031078608706593513,
+            0.022574681788682938,
+            -0.007048184517771006,
+            0.015966208651661873,
+            0.002999000484123826,
+            -0.014062216505408287,
+            -0.0023437028285115957,
+            -0.027680452913045883,
+            0.014454968273639679,
+            -0.017349377274513245,
+            -0.024265222251415253,
+            -0.007231753319501877,
+            -0.01045914739370346,
+            -0.004845359828323126,
+            0.038353051990270615,
+            0.012474133633077145,
+            -0.024572592228651047,
+            0.006817656569182873,
+            -0.003974475897848606,
+            -0.005989462602883577,
+            0.0057119750417768955,
+            0.005976655520498753,
+            0.004098278004676104,
+            0.010945817455649376,
+            -0.0065700518898665905,
+            -0.01320840884000063,
+            -0.03324728086590767,
+            -0.02240392006933689,
+            0.014941639266908169,
+            0.00028068936080671847,
+            -0.01315718051046133,
+            0.028483033180236816,
+            -0.01002370472997427,
+            -0.005575365852564573,
+            0.0071292961947619915,
+            0.00013887719251215458,
+            -0.0022796671837568283,
+            -0.017844585701823235,
+            0.002273263642564416,
+            0.006245605181902647,
+            -0.014950176700949669,
+            0.009844405576586723,
+            0.0032124524004757404,
+            -0.014540349133312702,
+            0.005635132547467947,
+            -0.0015027020126581192,
+            -0.003562513506039977,
+            -0.02481165900826454,
+            0.0059595792554318905,
+            0.00708233704790473,
+            -0.0018036692636087537,
+            0.002390662208199501,
+            -0.018169032409787178,
+            0.01801534742116928,
+            0.001955220242962241,
+            0.0071762558072805405,
+            -0.029166078194975853,
+            0.007282981649041176,
+            -0.006740813609212637,
+            0.0015080382581800222,
+            -0.0195692777633667,
+            0.009161358699202538,
+            0.020798761397600174,
+            0.012491210363805294,
+            -0.0005250918911769986,
+            0.010442070662975311,
+            0.008175211027264595,
+            -0.012704662047326565,
+            0.003950996324419975,
+            -0.02050846628844738,
+            -0.007308595813810825,
+            -0.010057857260107994,
+            0.011372721754014492,
+            -0.004219945520162582,
+            0.015983285382390022,
+            0.009511420503258705,
+            0.005600980017334223,
+            -0.009699258022010326,
+            0.00743666710332036,
+            0.01292665209621191,
+            0.005874198395758867,
+            -0.0011099503608420491,
+            0.017827508971095085,
+            0.07363239675760269,
+            0.005336299538612366,
+            0.013618236407637596,
+            6.870485958643258e-05,
+            -0.013071799650788307,
+            0.0015624684747308493,
+            -0.004939279053360224,
+            -0.01748598739504814,
+            0.026126522570848465,
+            0.026587579399347305,
+            -0.006382214371114969,
+            0.0004917400074191391,
+            0.009016212075948715,
+            0.005558289587497711,
+            0.005485715810209513,
+            0.022216081619262695,
+            -0.0077867284417152405,
+            0.01753721572458744,
+            0.024179840460419655,
+            0.005541213322430849,
+            -0.0019093280425295234,
+            -0.0022305732127279043,
+            0.009152821265161037,
+            0.004209273029118776,
+            -0.009050363674759865,
+            0.01150079257786274,
+            -0.01437812577933073,
+            0.01909114606678486,
+            0.011389797553420067,
+            0.02146473154425621,
+            0.007073798682540655,
+            -0.01704200729727745,
+            -0.018356870859861374,
+            0.0041281613521277905,
+            -0.0013479492627084255,
+            0.0059083509258925915,
+            0.0063224476762115955,
+            0.005477177910506725,
+            -0.011611787602305412,
+            -0.0041793896816670895,
+            0.010356690734624863,
+            -0.002471773885190487,
+            -0.0008500725380145013,
+            0.01757136732339859,
+            -0.01268758624792099,
+            0.01168009266257286,
+            -0.001839956152252853,
+            -0.0009984216885641217,
+            0.009682182222604752,
+            0.0014354645973071456,
+            -0.011825240217149258,
+            -0.004166582599282265,
+            0.0021003675647079945,
+            0.01217530108988285,
+            -0.013259637169539928,
+            -0.01048476155847311,
+            0.011825240217149258,
+            -0.006168762221932411,
+            0.009169897064566612,
+            -0.009579724632203579,
+            -0.0025315405800938606,
+            -0.00039648707024753094,
+            -0.008085560984909534,
+            -0.0014578771078959107,
+            0.007820880971848965,
+            0.0008324627415277064,
+            0.010518914088606834,
+            0.011560559272766113,
+            0.01150079257786274,
+            0.003560379147529602,
+            0.004002224653959274,
+            -0.002964847953990102,
+            -0.0064206356182694435,
+            -0.011432488448917866,
+            8.15119783510454e-05,
+            0.005776010919362307,
+            -0.02242099680006504,
+            -0.00024800453684292734,
+            0.008465506136417389,
+            0.005046004895120859,
+            -0.0007710952777415514,
+            0.020849989727139473,
+            0.001264702994376421,
+            0.007820880971848965,
+            -0.025221485644578934,
+            0.0011238247388973832,
+            0.006890229880809784,
+            0.007718423847109079,
+            0.012798581272363663,
+            -0.028636718168854713,
+            -0.0038528083823621273,
+            -0.0038634808734059334,
+            0.0171273872256279,
+            -0.00011606451880652457,
+            -0.0033106403425335884,
+            0.021686721593141556,
+            -0.0007833687704987824,
+            -0.003430173499509692,
+            -0.00755620002746582,
+            -0.005579635035246611,
+            0.008004449307918549,
+            0.012994957156479359,
+            0.024623820558190346,
+            -0.0019968433771282434,
+            -0.0055967108346521854,
+            -0.00809409935027361,
+            0.0244530588388443,
+            0.015249010175466537,
+            0.01863008923828602,
+            -0.0219087116420269,
+            0.010638446547091007,
+            -0.013865840621292591,
+            0.009067440405488014,
+            0.011714245192706585,
+            -0.010911665856838226,
+            0.0004290384822525084,
+            -0.010757979936897755,
+            -0.017135925590991974,
+            -0.022711291909217834,
+            0.00502039073035121,
+            0.02839765138924122,
+            -0.0002676154254004359,
+            0.007193332072347403,
+            -0.016358960419893265,
+            -0.0018442251021042466,
+            -0.011312955059111118,
+            -0.025750847533345222,
+            -0.016222350299358368,
+            -0.021772101521492004,
+            -0.02141350321471691,
+            -0.0038592119235545397,
+            -0.002131317974999547,
+            -0.01907406933605671,
+            0.00013153978215996176,
+            -0.0012817791430279613,
+            0.01902284100651741,
+            -0.0014888276346027851,
+            0.0015624684747308493,
+            0.012337524443864822,
+            0.012943727895617485,
+            -0.012824195437133312,
+            -0.004228483885526657,
+            -0.019261907786130905,
+            -0.024179840460419655,
+            0.0032487392891198397,
+            -0.007726961746811867,
+            0.013874378986656666,
+            -0.008234977722167969,
+            0.004260501824319363,
+            -0.017187153920531273,
+            0.014702572487294674,
+            -0.03579162806272507,
+            -0.024128612130880356,
+            0.0006531630642712116,
+            -0.0040299734100699425,
+            -0.009605339728295803,
+            -0.02585330419242382,
+            0.01198746357113123,
+            -0.019620506092905998,
+            0.002104636514559388,
+            -0.018442250788211823,
+            0.011398335918784142,
+            -0.015283161774277687,
+            0.018681317567825317,
+            -0.00025520852068439126,
+            0.03179580718278885,
+            0.004405648913234472,
+            0.007978835143148899,
+            -0.004525181837379932,
+            0.003327716374769807,
+            -0.005468640010803938,
+            0.006365138106048107,
+            0.006988417822867632,
+            0.008055677637457848,
+            0.007825150154531002,
+            -0.004768517334014177,
+            0.004061991348862648,
+            0.005110040307044983,
+            0.030515095219016075,
+            0.028021976351737976,
+            0.0027706066612154245,
+            0.0014674824196845293,
+            0.0034685947466641665,
+            0.004576410632580519,
+            -0.00038021133514121175,
+            0.008862526156008244,
+            -0.011287340894341469,
+            -0.005391797050833702,
+            0.01417321152985096,
+            -0.012448519468307495,
+            -0.0037695621140301228,
+            0.015377080999314785,
+            -0.015086785890161991,
+            -0.026092370972037315,
+            0.021823331713676453,
+            -0.006552976090461016,
+            -0.002450428670272231,
+            -0.006689585279673338,
+            0.008115444332361221,
+            -0.005946772173047066,
+            0.004194331355392933,
+            0.028107358142733574,
+            -0.005737589206546545,
+            -0.010945817455649376,
+            0.012636357918381691,
+            -0.002330895746126771,
+            -0.007948951795697212,
+            0.02291620522737503,
+            0.035894084721803665,
+            0.00999809056520462,
+            -0.026041142642498016,
+            -0.007381169591099024,
+            0.02144765481352806,
+            -0.00973341055214405,
+            -0.021755026653409004,
+            0.008273398503661156,
+            0.007522047962993383,
+            -0.005421680398285389,
+            0.018937459215521812,
+            0.004239156376570463,
+            -0.010792132467031479,
+            0.009041826240718365,
+            0.022711291909217834,
+            0.021686721593141556,
+            0.008947907015681267,
+            -0.023548023775219917,
+            0.01025423314422369,
+            -0.026228979229927063,
+            -0.013831689022481441,
+            -0.023821242153644562,
+            0.003581724362447858,
+            0.003857077332213521,
+            -0.003257277188822627,
+            0.026536351069808006,
+            0.011876468546688557,
+            0.011645940132439137,
+            0.0030950538348406553,
+            0.023052813485264778,
+            -0.015223395079374313,
+            -0.028107358142733574,
+            -0.0009733410552144051,
+            -0.015479537658393383,
+            -0.0018282162491232157,
+            0.002166537567973137,
+            0.007150641642510891,
+            0.005852853413671255,
+            0.01953512616455555,
+            0.000376209121895954,
+            0.0019584218971431255,
+            -2.4263486920972355e-05,
+            0.010100548155605793,
+            -0.015727141872048378,
+            -0.019705887883901596,
+            -0.008922292850911617,
+            6.470263178925961e-05,
+            0.011637401767075062,
+            -0.005208228249102831,
+            -3.448583447607234e-05,
+            0.007846495136618614,
+            -0.015897903591394424,
+            -0.0097590247169137,
+            -0.014412278309464455,
+            -0.0029306956566870213,
+            -0.005071619059890509,
+            0.03136890381574631,
+            0.014437892474234104,
+            -0.004747172351926565,
+            -0.011543483473360538,
+            -0.005741858389228582,
+            0.03370833769440651,
+            0.0017204229952767491,
+            0.012422905303537846,
+            0.0146342683583498,
+            0.015684451907873154,
+            0.027014482766389847,
+            0.012807118706405163,
+            0.003018211107701063,
+            0.002516598906368017,
+            0.02346264198422432,
+            -0.013951221480965614,
+            0.008730186149477959,
+            -0.011645940132439137,
+            -0.003521957667544484,
+            -0.010937280021607876,
+            0.015078248456120491,
+            -0.013003494590520859,
+            -0.011278802528977394,
+            0.023069890215992928,
+            -0.0004381102044135332,
+            0.013276712968945503,
+            -0.0004242358263581991,
+            -0.003953130915760994,
+            -0.02534101903438568,
+            0.0015464596217498183,
+            0.0006531630642712116,
+            -0.004401379730552435,
+            0.010578680783510208,
+            -0.027475539594888687,
+            0.0013660927070304751,
+            0.015240471810102463,
+            -0.007116489112377167,
+            0.00038981667603366077,
+            -0.010826284997165203,
+            0.005904081743210554,
+            0.00011606451880652457,
+            -0.02385539375245571,
+            0.015488076023757458,
+            -0.006698123179376125,
+            0.01243144366890192,
+            -0.0018548977095633745,
+            -0.013788998126983643,
+            0.008687496185302734,
+            0.019296059384942055,
+            0.027560919523239136,
+            -0.007825150154531002,
+            0.0005528406472876668,
+            -0.0001855698210420087,
+            -0.006027883850038052,
+            -0.02737308293581009,
+            0.02054261974990368,
+            -0.007863570936024189,
+            0.01953512616455555,
+            -0.009682182222604752,
+            -0.018186109140515327,
+            -0.014856258407235146,
+            0.011927696876227856,
+            0.007603159639984369,
+            0.0010304395109415054,
+            0.01489894837141037,
+            -0.024077383801341057,
+            0.020867066457867622,
+            0.0033383890986442566,
+            -0.0064718639478087425,
+            0.008303281851112843,
+            0.004230618476867676,
+            0.004695943556725979,
+            -0.008384393528103828,
+            0.007658657152205706,
+            0.007569007575511932,
+            -0.007253098301589489,
+            0.004326671827584505,
+            -0.04057295247912407,
+            0.010732365772128105,
+            -0.011090965010225773,
+            0.005878467578440905,
+            -0.004742903169244528,
+            -0.022745443508028984,
+            -0.035825781524181366,
+            0.004567872267216444,
+            0.019313136115670204,
+            -0.020235247910022736,
+            0.018408099189400673,
+            0.012124072760343552,
+            0.014574501663446426,
+            0.01859593763947487,
+            0.008853988721966743,
+            0.01461719162762165,
+            0.0438857264816761,
+            -0.024231068789958954,
+            6.483604374807328e-05,
+            0.008734455332159996,
+            0.011381260119378567,
+            -0.013404784724116325,
+            -0.010126162320375443,
+            0.005784548819065094,
+            -0.0015197781613096595,
+            0.04067540913820267,
+            -0.019210679456591606,
+            0.010886050760746002,
+            -0.010962894186377525,
+            0.008371586911380291,
+            -0.009562648832798004,
+            -0.0036158766597509384,
+            -0.012559514492750168,
+            -0.011893544346094131,
+            0.014745263382792473,
+            -0.019671734422445297,
+            -0.005460101645439863,
+            -0.007282981649041176,
+            0.01048476155847311,
+            -0.006096188444644213,
+            -0.020730456337332726,
+            0.018647165969014168,
+            -0.0018079383298754692,
+            0.029968658462166786,
+            -0.012755890376865864,
+            0.00574612757191062,
+            0.022591758519411087,
+            -0.010604294948279858,
+            0.018732545897364616,
+            -0.004362958483397961,
+            0.008853988721966743,
+            0.0055625587701797485,
+            -0.0065188235603272915,
+            -0.0014685495989397168,
+            0.01389145478606224,
+            -0.0027385889552533627,
+            0.029934506863355637,
+            -0.017400605604052544,
+            -0.009818791411817074,
+            0.009707796387374401,
+            0.0073769004084169865,
+            0.0042370217852294445,
+            0.003878422547131777,
+            0.009579724632203579,
+            -0.01071528997272253,
+            0.0009728074073791504,
+            -0.021686721593141556,
+            -0.015453923493623734,
+            -0.0011483717244118452,
+            -0.004683136474341154,
+            -0.0018730411538854241,
+            -0.002467504935339093,
+            -0.02044016122817993,
+            0.004627638962119818,
+            -0.0021579996682703495,
+            -8.451364556094632e-05,
+            -0.024487212300300598,
+            -0.00769280968233943,
+            -0.010057857260107994,
+            -0.005161269102245569,
+            -0.003139878623187542,
+            0.018271489068865776,
+            0.0034856710117310286,
+            -0.004862436093389988,
+            0.0293197650462389,
+            0.005404604133218527,
+            0.01536854263395071,
+            0.00280262460000813,
+            -0.05334591865539551,
+            -0.011014122515916824,
+            -0.004563603550195694,
+            -0.009912709705531597,
+            -0.007261636666953564,
+            -0.008456967771053314,
+            -0.010681137442588806,
+            -0.0019061262719333172,
+            0.0032914294861257076,
+            -0.004170851781964302,
+            0.0122521435841918,
+            -0.01700785383582115,
+            0.025494705885648727,
+            -0.004017166327685118,
+            0.015744218602776527,
+            -0.011722782626748085,
+            0.011654478497803211,
+            -0.006194376386702061,
+            -0.010851899161934853,
+            0.010544528253376484,
+            0.0146342683583498,
+            -0.0030544979963451624,
+            0.020371858030557632,
+            -0.014736725017428398,
+            0.003197510726749897,
+            -0.008303281851112843,
+            -0.02143057994544506,
+            -0.015991821885108948,
+            0.011295879259705544,
+            0.03130060061812401,
+            0.010066395625472069,
+            -0.0037524858489632607,
+            0.004072663839906454,
+            0.021669644862413406,
+            -0.004352285992354155,
+            0.0063181789591908455,
+            0.03025895357131958,
+            -0.0010982104577124119,
+            0.017742129042744637,
+            0.0001935742620844394,
+            -0.005028928630053997,
+            -0.002362913452088833,
+            0.01808365248143673,
+            -0.002766337711364031,
+            0.0057119750417768955,
+            0.02441890724003315,
+            0.010997045785188675,
+            -0.0064718639478087425,
+            0.008230708539485931,
+            0.014309820719063282,
+            0.013874378986656666,
+            -0.015351466834545135,
+            -0.0122009152546525,
+            -0.011534945107996464,
+            -0.015778370201587677,
+            0.007291519548743963,
+            -0.019261907786130905,
+            -0.004049184266477823,
+            0.007829418405890465,
+            -0.015138015151023865,
+            -0.012807118706405163,
+            -0.023548023775219917,
+            -0.015914980322122574,
+            0.01863008923828602,
+            0.005357644986361265,
+            -0.01465988252311945,
+            -0.004369362257421017,
+            0.0013714289525523782,
+            0.006181569304317236,
+            -0.0054985228925943375,
+            0.021840406581759453,
+            -0.004098278004676104,
+            -0.011125117540359497,
+            0.019330210983753204,
+            0.0017897948855534196,
+            0.005097233224660158,
+            -0.0064206356182694435,
+            0.004550796467810869,
+            -0.011364183388650417,
+            0.022096550092101097,
+            -0.003470729338005185,
+            0.012636357918381691,
+            -0.0037033918779343367,
+            0.004631908144801855,
+            -0.001978699816390872,
+            0.00821790099143982,
+            0.004017166327685118,
+            0.010629909113049507,
+            -0.005656477529555559,
+            0.004525181837379932,
+            0.012491210363805294,
+            -0.01641872711479664,
+            0.007265905383974314,
+            0.004559334367513657,
+            -4.365760105429217e-05,
+            0.005780279636383057,
+            -0.007645850069820881,
+            -0.005417411215603352,
+            -0.015488076023757458,
+            0.010100548155605793,
+            0.030873695388436317,
+            -0.0016649254830554128,
+            -0.007240291219204664,
+            0.01704200729727745,
+            0.015735680237412453,
+            -0.003816521493718028,
+            0.0061218030750751495,
+            0.009280892089009285,
+            -0.003148416755720973,
+            0.01370361726731062,
+            -0.024487212300300598,
+            -0.029524678364396095,
+            0.01763967238366604,
+            -0.020867066457867622,
+            0.022779595106840134,
+            -0.019210679456591606,
+            -0.027731681242585182,
+            -0.01847640424966812,
+            -0.0032530082389712334,
+            -0.019842496141791344,
+            -0.01611989364027977,
+            0.02532394416630268,
+            0.00685180863365531,
+            -0.006949996575713158,
+            -0.031112760305404663,
+            0.04050464928150177,
+            0.014489120803773403,
+            0.038318898528814316,
+            -0.011603250168263912,
+            -2.554753518779762e-05,
+            -0.007504971697926521,
+            0.012064306065440178,
+            -0.01146664097905159,
+            -0.014258592389523983,
+            0.007509240880608559,
+            0.028175661340355873,
+            0.015231933444738388,
+            -0.015223395079374313,
+            -0.007944682613015175,
+            0.015991821885108948,
+            -0.007043915335088968,
+            -0.0005480379913933575,
+            -0.008836911991238594,
+            -0.014830644242465496,
+            -0.010066395625472069,
+            -0.0005312286084517837,
+            0.012243606150150299,
+            0.016179660335183144,
+            -0.0007257367251440883,
+            0.01390853151679039,
+            -0.009955400601029396,
+            -0.014420815743505955,
+            -0.004602024797350168,
+            0.005327761638909578,
+            -0.02252345345914364,
+            0.02105490304529667,
+            0.002292474266141653,
+            -0.009537034668028355,
+            0.0030865157023072243,
+            -0.004572141449898481,
+            0.012602205388247967,
+            0.009391887113451958,
+            -0.0038421356584876776,
+            -0.0036137420684099197,
+            0.00572051340714097,
+            -0.001715086749754846,
+            0.015001405030488968,
+            -0.011714245192706585,
+            0.011842316016554832,
+            -0.007432397920638323,
+            -0.00019931078713852912,
+            -0.009648029692471027,
+            -0.005238111596554518,
+            -0.010493299923837185,
+            0.0028794673271477222,
+            0.012943727895617485,
+            -0.014548887498676777,
+            0.012807118706405163,
+            -0.0008132520597428083,
+            -0.00949434470385313,
+            -0.015138015151023865,
+            -0.0055155991576612,
+            -0.011364183388650417,
+            -0.016683407127857208,
+            0.007069529499858618,
+            0.0007855033036321402,
+            0.019347287714481354,
+            0.0030054040253162384,
+            -0.004572141449898481,
+            -0.00017262929759453982,
+            0.018903307616710663,
+            -0.02580207586288452,
+            0.03403278440237045,
+            -0.00010045584349427372,
+            0.022711291909217834,
+            0.04251963645219803,
+            0.00954557303339243,
+            -0.006663971114903688,
+            -0.013754845596849918,
+            0.005801625084131956,
+            -0.013609698973596096,
+            0.009981014765799046,
+            0.014335434883832932,
+            -0.03235932067036629,
+            -0.003944592550396919,
+            -0.013976835645735264,
+            0.011586173437535763,
+            -0.007957490161061287,
+            0.0032850259449332952,
+            -0.010587218217551708,
+            0.011765473522245884,
+            0.01679440215229988,
+            -0.009075978770852089,
+            -0.004311730153858662,
+            -0.027492616325616837,
+            0.00032524747075513005,
+            0.016674868762493134,
+            -0.010911665856838226,
+            -0.01684563048183918,
+            0.010945817455649376,
+            -0.007218946237117052,
+            -0.015453923493623734,
+            -0.006898768246173859,
+            -0.03278622403740883,
+            -0.00020504731219261885,
+            -0.01076651830226183,
+            -0.003927516285330057,
+            -0.0028794673271477222,
+            0.0011024795239791274,
+            -0.0033917520195245743,
+            -0.0067066615447402,
+            -0.0025891726836562157,
+            0.0067664277739822865,
+            0.005011852364987135,
+            0.002080089645460248,
+            0.0005987327895127237,
+            -0.00023506399884354323,
+            0.009255277924239635,
+            0.010168852284550667,
+            -0.007419590838253498,
+            -0.017212767153978348,
+            0.010834822431206703,
+            -0.020730456337332726,
+            0.005938234273344278,
+            0.00783795677125454,
+            0.011039736680686474,
+            -0.0220111683011055,
+            -0.010868974961340427,
+            0.009818791411817074,
+            0.003835732117295265,
+            0.01902284100651741,
+            0.010886050760746002,
+            0.0022455148864537477,
+            -0.0035646480973809958,
+            0.0028495839796960354,
+            -1.0947752343781758e-05,
+            -0.0012508286163210869,
+            0.015308775939047337,
+            0.0006504949415102601,
+            0.02974666841328144,
+            -0.012055767700076103,
+            0.011201960034668446,
+            -0.009340658783912659,
+            0.0013842361513525248,
+            -0.009741948917508125,
+            -0.001973363570868969,
+            -0.004194331355392933,
+            -0.006484671495854855,
+            0.013148642145097256,
+            0.007714154664427042,
+            -0.004239156376570463,
+            -0.01416467409580946,
+            0.01194477267563343,
+            -0.023035738617181778,
+            -0.012269220314919949,
+            -0.00881983619183302,
+            0.012935190461575985,
+            -0.0053960662335157394,
+            0.01170570682734251,
+            0.03457922115921974,
+            -0.0018559650052338839,
+            0.0007860369514673948,
+            -0.013746308162808418,
+            -0.008311820216476917,
+            -0.02728770114481449,
+            -0.01510386262089014,
+            -0.008922292850911617,
+            -0.03454506769776344,
+            -0.011919158510863781,
+            0.008239246904850006,
+            0.008111175149679184,
+            0.0014237246941775084,
+            -0.019176525995135307,
+            -0.008888140320777893,
+            -0.01492456253618002,
+            -0.017460372298955917,
+            -0.00924673955887556,
+            -0.01175693515688181,
+            -0.0037119300104677677,
+            0.020884141325950623,
+            -0.004162313882261515,
+            0.014267130754888058,
+            0.016939548775553703,
+            1.3132299500284716e-05,
+            0.015334390103816986,
+            0.009665105491876602,
+            -0.009127207100391388,
+            -0.004725826904177666,
+            0.006885961163789034,
+            0.0027706066612154245,
+            0.018749622628092766,
+            -0.006702392362058163,
+            -0.014796491712331772,
+            0.004298923071473837,
+            0.005571096669882536,
+            -0.012346062809228897,
+            0.010655523277819157,
+            -0.00476424815133214,
+            0.02054261974990368,
+            -0.018254414200782776,
+            -0.011885006912052631,
+            0.014472044073045254,
+            -0.008862526156008244,
+            0.02394077368080616,
+            0.013951221480965614,
+            -0.015291700139641762,
+            0.007167717441916466,
+            0.013029108755290508,
+            0.007615966722369194,
+            0.0018495614640414715,
+            0.022147778421640396,
+            -0.0011846584966406226,
+            0.0031590894795954227,
+            0.010237157344818115,
+            -0.018835002556443214,
+            -0.02042308636009693,
+            -0.003118533408269286,
+            0.02247222512960434,
+            0.007961759343743324,
+            -0.016222350299358368,
+            0.0037930416874587536,
+            0.012849809601902962,
+            0.012072844430804253,
+            0.0039787450805306435,
+            -0.001204936415888369,
+            -0.02248930186033249,
+            0.0035155541263520718,
+            0.0001910395221784711,
+            0.018664240837097168,
+            -0.011116579174995422,
+            0.001461078878492117,
+            -0.011389797553420067,
+            0.012747352942824364,
+            -0.005221035331487656,
+            0.01193623524159193,
+            0.0011761204805225134,
+            -0.005353375803679228,
+            0.0001780989987310022,
+            0.014941639266908169,
+            0.014070754870772362,
+            0.037738312035799026,
+            -0.005916888825595379,
+            -0.006480402313172817,
+            0.017170077189803123,
+            -0.012388752773404121,
+            0.018715469166636467,
+            0.006497478578239679,
+            0.0170249305665493,
+            -0.011090965010225773,
+            0.0011344973463565111,
+            0.04115354269742966,
+            0.015949131920933723,
+            -6.583659705938771e-05,
+            -0.008760069496929646,
+            -0.00805140845477581,
+            0.0244530588388443,
+            0.009067440405488014,
+            -2.7882166250492446e-05,
+            0.015744218602776527,
+            0.006096188444644213,
+            -0.016324806958436966,
+            -0.012918113730847836,
+            0.03555256128311157,
+            -0.007718423847109079,
+            -0.022045321762561798,
+            -0.0009311842732131481,
+            -0.009383349679410458,
+            -0.02884163148701191,
+            0.013959759846329689,
+            0.013788998126983643,
+            -0.008162403479218483,
+            0.006992687005549669,
+            0.008439891040325165,
+            -0.004117488861083984,
+            -0.004435532260686159,
+            0.0005346972029656172,
+            -0.0007049251580610871,
+            0.006177300587296486,
+            0.005092964507639408,
+            -0.014463506639003754,
+            -0.006787773221731186,
+            -0.005669284611940384,
+            -0.004064125940203667,
+            0.009016212075948715,
+            0.005058811977505684,
+            0.004870974458754063,
+            -0.008482581935822964,
+            0.018339794129133224,
+            0.02298451028764248,
+            -0.0122009152546525,
+            0.012115534394979477,
+            0.005233842879533768,
+            -0.0146428057923913,
+            -0.004614831879734993,
+            -0.01121903583407402,
+            -0.026570502668619156,
+            -0.006113264709711075,
+            0.0033490615896880627,
+            0.015428309328854084,
+            -0.022796671837568283,
+            0.0008500725380145013,
+            0.007790997624397278,
+            2.7365211280994117e-05,
+            -0.008879602886736393,
+            0.007573276292532682,
+            -0.00044157879892736673,
+            0.022301463410258293,
+            0.019808344542980194,
+            0.025631314143538475,
+            -0.008414276875555515,
+            -0.00499904528260231,
+            0.014301283285021782,
+            0.00433947890996933,
+            0.009374811314046383,
+            0.027543844655156136,
+            0.022164853289723396,
+            -0.013857303187251091,
+            0.006429173983633518,
+            0.000640355923678726,
+            -0.014250054024159908,
+            0.0066682398319244385,
+            0.013054722920060158,
+            0.012414366938173771,
+            0.0028410458471626043,
+            -0.0013415457215160131,
+            -0.015957670286297798,
+            0.004824014846235514,
+            0.018408099189400673,
+            -0.017118848860263824,
+            -0.03471583127975464,
+            -0.011842316016554832,
+            -0.012158225290477276,
+            -0.00624133599922061,
+            -0.012875423766672611,
+            0.01440373994410038,
+            0.004418455995619297,
+            0.019705887883901596,
+            0.0002309283590875566,
+            -0.01022861897945404,
+            0.014250054024159908,
+            0.005110040307044983,
+            -0.0016638581873849034,
+            0.0005352308508008718,
+            0.007231753319501877,
+            -0.0030331527814269066,
+            -0.0022476494777947664,
+            -0.00027055040118284523,
+            0.016239427030086517,
+            -0.02532394416630268,
+            0.008947907015681267,
+            -0.017067620530724525,
+            0.0017951311310753226,
+            0.015249010175466537,
+            -0.003775965655222535,
+            -0.016025975346565247,
+            0.0011216901475563645,
+            -0.008619191125035286,
+            -0.07288104295730591,
+            -0.005882736761122942,
+            0.004512374754995108,
+            0.023992003872990608,
+            -0.013336479663848877,
+            -0.0024824466090649366,
+            0.0018474268727004528,
+            0.008657612837851048,
+            -0.010134699754416943,
+            0.003995821345597506,
+            0.010390842333436012,
+            -0.003319178242236376,
+            -0.009221125394105911,
+            0.002508060773834586,
+            -0.008106905966997147,
+            -0.0038805571384727955,
+            -0.00783795677125454,
+            0.003402424743399024,
+            0.025221485644578934,
+            -0.010868974961340427,
+            0.024657974019646645,
+            0.009699258022010326,
+            -0.0075433929450809956,
+            -0.0027428579051047564,
+            -0.007761114276945591,
+            -0.027543844655156136,
+            -0.012397291138768196,
+            -0.003396020969375968,
+            -0.01292665209621191,
+            0.0029264267068356276,
+            0.001692674239166081,
+            -0.007167717441916466,
+            -0.004348016809672117,
+            0.0021547977812588215,
+            0.00030336863710545003,
+            -0.022147778421640396,
+            -0.0009407896432094276,
+            -0.014267130754888058,
+            0.02105490304529667,
+            -0.012371676973998547,
+            -0.004461146425455809,
+            0.0030544979963451624,
+            -0.010109085589647293,
+            0.004580679349601269,
+            0.019313136115670204,
+            -0.007308595813810825,
+            -0.009682182222604752,
+            0.014045140706002712,
+            -0.003690584795549512,
+            0.00185916677583009,
+            -0.01798119582235813,
+            0.010032243095338345,
+            -0.03232516720890999,
+            -0.003583858720958233,
+            -0.009981014765799046,
+            -0.0031847036443650723,
+            -0.0067151994444429874,
+            -0.03162504732608795,
+            0.015138015151023865,
+            -0.007039646618068218,
+            -0.02346264198422432,
+            -0.0011793222511187196,
+            -0.01859593763947487,
+            0.01419028826057911,
+            -0.01487333420664072,
+            0.01393414568156004,
+            -0.020645076408982277,
+            0.012533900327980518,
+            0.012798581272363663,
+            0.021123208105564117,
+            0.006493209395557642,
+            -0.005302147474139929,
+            0.002676687901839614,
+            0.0018730411538854241,
+            0.014352511614561081,
+            -0.008905217051506042,
+            0.00462336977943778,
+            -0.002390662208199501,
+            0.005737589206546545,
+            0.009426039643585682,
+            0.02107197977602482,
+            -0.0005896611255593598,
+            -0.0037823691964149475,
+            -0.017468910664319992,
+            0.00757754547521472,
+            -0.01559907104820013,
+            0.005485715810209513,
+            0.0025251370389014482,
+            0.023787088692188263,
+            0.010886050760746002,
+            0.0035859933122992516,
+            0.004900857340544462,
+            0.004520913120359182,
+            0.01861301250755787,
+            -0.006890229880809784,
+            -0.004747172351926565,
+            -0.018186109140515327,
+            0.019688811153173447,
+            0.007705616764724255,
+            -0.005306416191160679,
+            -0.011902082711458206,
+            0.006010808050632477,
+            0.002134519862011075,
+            0.0017577770631760359,
+            0.02288205362856388,
+            -0.02974666841328144,
+            -0.008964983746409416,
+            -0.007808073423802853,
+            -0.009682182222604752,
+            -0.017844585701823235,
+            0.017084697261452675,
+            -0.008012987673282623,
+            0.00461910106241703,
+            -0.012790042906999588,
+            0.033418044447898865,
+            0.002969117136672139,
+            0.00513992365449667,
+            0.013729231432080269,
+            -0.00024373548512812704,
+            0.017844585701823235,
+            -0.008367317728698254,
+            -0.04036803916096687,
+            -0.0031911071855574846,
+            0.0034515184815973043,
+            0.007645850069820881,
+            -0.010903127491474152,
+            0.004862436093389988,
+            -0.006339523941278458,
+            -0.01854470930993557,
+            -0.014232978224754333,
+            0.037840768694877625,
+            -0.005225304514169693,
+            0.017947042360901833,
+            -0.016145508736371994,
+            -0.0033682722132653,
+            0.007107951212674379,
+            0.022762520238757133,
+            -0.012602205388247967,
+            0.00791053008288145,
+            0.009571187198162079,
+            0.02199409157037735,
+            0.00978463888168335,
+            -0.01565883681178093,
+            0.023735860362648964,
+            0.005741858389228582,
+            -0.0023565099108964205,
+            0.005276532843708992,
+            -0.0030352871399372816,
+            0.016000360250473022,
+            -0.014813567511737347,
+            -0.011876468546688557,
+            0.002303146990016103,
+            -0.010168852284550667,
+            0.0037055264692753553,
+            -0.024589668959379196,
+            0.00037540867924690247,
+            -0.009844405576586723,
+            0.021089056506752968,
+            0.01027130987495184,
+            0.00877714529633522,
+            -0.019637582823634148,
+            -0.0048026698641479015,
+            -0.02645096927881241,
+            -0.014574501663446426,
+            -0.01022861897945404,
+            -0.0011451698374003172,
+            -0.004431263078004122,
+            -0.007765382993966341,
+            -0.03133475035429001,
+            -0.004202869720757008,
+            0.013541393913328648,
+            0.014975790865719318,
+            -0.0013714289525523782,
+            0.012089920230209827,
+            -0.00019544197130016983,
+            -0.008106905966997147,
+            -0.02236976847052574,
+            -0.010407919064164162,
+            -0.02091829478740692,
+            -0.03123229369521141,
+            -0.022199006751179695,
+            0.008670419454574585,
+            -0.012021616101264954,
+            -0.002941368380561471,
+            0.028961164876818657,
+            -0.022130701690912247,
+            0.007513510063290596,
+            0.005400335416197777,
+            0.0028666602447628975,
+            0.014856258407235146,
+            0.0054472945630550385,
+            0.0003634020104072988,
+            0.0037652929313480854,
+            -0.008341703563928604,
+            -0.008081291802227497,
+            0.01799827069044113,
+            -0.003331985557451844,
+            0.005097233224660158,
+            0.010809208266437054,
+            -0.006749351508915424,
+            -0.005438756663352251,
+            0.02392369881272316,
+            -0.015718603506684303,
+            0.002044870052486658,
+            -0.008610652759671211,
+            -0.014079293236136436,
+            -0.00949434470385313,
+            -0.005793086718767881,
+            -0.01813488081097603,
+            -0.015240471810102463,
+            0.01689685881137848,
+            -0.0004028906114399433,
+            0.016025975346565247,
+            -0.005289340391755104,
+            -0.005007583647966385,
+            0.014437892474234104,
+            -0.03558671474456787,
+            -0.003402424743399024,
+            0.006377945188432932,
+            -0.026211904361844063,
+            0.02484581060707569,
+            0.005276532843708992,
+            0.01605158858001232,
+            -0.013780459761619568,
+            0.0026318628806620836,
+            -0.006365138106048107,
+            0.004358689300715923,
+            0.018903307616710663,
+            0.007308595813810825,
+            0.004858167376369238,
+            -0.006689585279673338,
+            -0.008047140203416348,
+            0.016077203676104546,
+            -0.007415322121232748,
+            0.0018719738582149148,
+            0.016785863786935806,
+            0.018425175920128822,
+            0.012004539370536804,
+            0.0059553105384111404,
+            0.013071799650788307,
+            0.01750306226313114,
+            -0.007043915335088968,
+            -0.005835777148604393,
+            -0.013481627218425274,
+            -0.00802579429000616,
+            -0.0013105951948091388,
+            0.03859211876988411,
+            0.018920384347438812,
+            0.007385438773781061,
+            -0.02054261974990368,
+            0.013456013053655624,
+            0.0043928418308496475,
+            6.841302820248529e-06,
+            -0.0027385889552533627,
+            0.02293328195810318,
+            0.0024952536914497614,
+            0.007125027012079954,
+            0.006258412264287472,
+            -0.017212767153978348,
+            0.03309359401464462,
+            -0.020713381469249725,
+            0.0008532743086107075,
+            -0.013319403864443302,
+            -0.012166762724518776,
+            0.015530765987932682,
+            0.002104636514559388,
+            -0.004164448007941246,
+            -0.006296833511441946,
+            -0.0033575997222214937,
+            0.013515779748558998,
+            0.008499657735228539,
+            -0.012875423766672611,
+            0.004516643937677145,
+            -0.02542640082538128,
+            0.004315999336540699,
+            -0.015334390103816986,
+            0.021755026653409004,
+            -0.004559334367513657,
+            -0.004674598574638367,
+            -0.019261907786130905,
+            -0.017067620530724525,
+            -0.00999809056520462,
+            0.002403469290584326,
+            -0.004755710251629353,
+            0.004422725178301334,
+            -0.009468729607760906,
+            -0.008499657735228539,
+            0.0027599341701716185,
+            0.0036628360394388437,
+            0.012465596199035645,
+            0.013669464737176895,
+            0.019893724471330643,
+            0.017759205773472786,
+            -0.01536000519990921,
+            -0.0341864712536335,
+            0.0006056699785403907,
+            0.005084426142275333,
+            -0.0036436254158616066,
+            0.018664240837097168,
+            -0.001938143977895379,
+            -0.0009968207450583577,
+            -0.0007150641758926213,
+            0.0005419012159109116,
+            0.02004740945994854,
+            -0.011014122515916824,
+            -0.016657792031764984,
+            0.00021545309573411942,
+            0.008853988721966743,
+            0.00502039073035121,
+            -0.0025934416335076094,
+            -0.010296924039721489,
+            -0.0009455922991037369,
+            0.0020341973286122084,
+            0.006032153032720089,
+            -0.009101592935621738,
+            -0.0010496501345187426,
+            -0.0013671600027009845,
+            -0.0030544979963451624,
+            -0.006254143081605434,
+            -0.009844405576586723,
+            0.008222170174121857,
+            -0.012602205388247967,
+            -0.01123611256480217,
+            0.016222350299358368,
+            -0.015086785890161991,
+            0.001136631821282208,
+            0.023821242153644562,
+            0.022557605057954788,
+            -0.0008842248353175819,
+            0.006774966139346361,
+            -0.007936145178973675,
+            -0.011517869308590889,
+            -0.02785121463239193,
+            -0.019637582823634148,
+            -0.00806848518550396,
+            0.017947042360901833,
+            -0.002053408185020089,
+            -0.004049184266477823,
+            -0.0018869155319407582,
+            -0.008580769412219524,
+            0.008444160223007202,
+            -0.013814612291753292,
+            0.011620325967669487,
+            -0.034425534307956696,
+            0.013088875450193882,
+            -0.006702392362058163,
+            0.0010485828388482332,
+            0.015009943395853043,
+            0.04351005330681801,
+            -0.007351286243647337,
+            0.0029541754629462957,
+            0.001540056080557406,
+            0.01389145478606224,
+            0.028209814801812172,
+            0.004369362257421017,
+            0.0006136744632385671,
+            -0.01765674725174904,
+            0.004091874696314335,
+            0.002264725510030985,
+            -0.013456013053655624,
+            0.009579724632203579,
+            -0.023035738617181778,
+            -0.023155272006988525,
+            -0.009067440405488014,
+            -0.021123208105564117,
+            -0.002608383307233453,
+            0.002011785050854087,
+            0.012883962132036686,
+            -0.009878558106720448,
+            -0.012098458595573902,
+            -0.0005656477296724916,
+            -0.032991137355566025,
+            0.016709022223949432,
+            0.09665105491876602,
+            0.00875153113156557,
+            -0.003908305894583464,
+            -0.0021515958942472935,
+            0.002499522641301155,
+            0.009562648832798004,
+            -0.004768517334014177,
+            0.055360905826091766,
+            -0.018852079287171364,
+            0.008311820216476917,
+            -0.0008105839369818568,
+            0.008064216002821922,
+            -0.007223214954137802,
+            0.015530765987932682,
+            -0.012209453620016575,
+            -0.005951041355729103,
+            -0.009460192173719406,
+            -0.01123611256480217,
+            -0.0017246920615434647,
+            -0.016640717163681984,
+            -0.004986238200217485,
+            -0.020645076408982277,
+            0.013336479663848877,
+            -0.03234224393963814,
+            -0.011509330943226814,
+            -0.012474133633077145,
+            -0.0063224476762115955,
+            0.006066305562853813,
+            -0.015684451907873154,
+            -0.002595576224848628,
+            -0.002099300269037485,
+            0.015402695164084435,
+            0.0013863706262782216,
+            -0.00232235761359334,
+            -0.008328896015882492,
+            0.0005405671545304358,
+            0.005024659913033247,
+            0.003940323833376169,
+            0.006557244807481766,
+            0.006792041938751936,
+            -0.005848584230989218,
+            -0.008811297826468945,
+            0.010655523277819157,
+            -0.002510195365175605,
+            -0.0027919518761336803,
+            -0.008871064521372318,
+            0.010382304899394512,
+            -0.013481627218425274,
+            0.018356870859861374,
+            -0.013840226456522942,
+            -0.029507601633667946,
+            -0.015547842718660831,
+            0.02100367471575737,
+            -0.01996202953159809,
+            -0.0039381892420351505,
+            0.010553065687417984,
+            0.02829519473016262,
+            0.004113219678401947,
+            -0.001529383473098278,
+            -0.004909395705908537,
+            -0.017204230651259422,
+            0.009511420503258705,
+            0.008196556009352207,
+            0.018442250788211823,
+            -0.012371676973998547,
+            0.016487032175064087,
+            -0.018288565799593925,
+            0.006177300587296486,
+            -0.01663217879831791,
+            0.0006083381595090032,
+            0.00011866596469189972,
+            -0.0041793896816670895,
+            0.01608574204146862,
+            -0.01534292846918106,
+            -0.026263132691383362,
+            0.01900576427578926,
+            -0.013635313138365746,
+            -0.03307652100920677,
+            0.003374675754457712,
+            -0.034937821328639984,
+            -0.0025827689096331596,
+            0.009716334752738476,
+            -0.02149888314306736,
+            -0.018305642530322075,
+            0.007487895432859659,
+            -0.010066395625472069,
+            -0.008871064521372318,
+            0.00502039073035121,
+            -0.01689685881137848,
+            -0.012149686925113201,
+            -0.001152640674263239,
+            0.006753620691597462,
+            -0.00499477656558156,
+            0.0057632033713161945,
+            -0.026075294241309166,
+            0.014010988175868988,
+            -0.0170249305665493,
+            0.01144102681428194,
+            -0.010587218217551708,
+            0.022762520238757133,
+            0.011748396791517735,
+            -0.010356690734624863,
+            -0.0008303282083943486,
+            0.006036422215402126,
+            0.0005264259525574744,
+            0.026126522570848465,
+            0.03237639740109444,
+            -0.0011889275629073381,
+            0.008469774387776852,
+            0.007586083374917507,
+            0.003921112976968288,
+            -0.011014122515916824,
+            0.0015923518221825361,
+            0.014830644242465496,
+            0.01992787793278694,
+            -0.013729231432080269,
+            0.008879602886736393,
+            -0.026672959327697754,
+            0.02103782817721367,
+            -0.022096550092101097,
+            -0.000982412719167769,
+            0.010066395625472069,
+            -0.004777055233716965,
+            0.007090874947607517,
+            0.0066682398319244385,
+            0.021362274885177612,
+            0.03375956416130066,
+            0.00807702261954546,
+            0.008879602886736393,
+            0.007257367484271526,
+            -0.004798400681465864,
+            0.002044870052486658,
+            0.009135744534432888,
+            -0.005869929678738117,
+            -0.0040940092876553535,
+            -0.010083471424877644,
+            -0.006685316096991301,
+            0.016205275431275368,
+            0.0008079157560132444,
+            0.012602205388247967,
+            0.007018301170319319,
+            -0.007428129203617573,
+            -0.005669284611940384,
+            0.007014031987637281,
+            -0.023787088692188263,
+            -0.00950288213789463,
+            0.025221485644578934,
+            -0.013814612291753292,
+            -0.043202683329582214,
+            -0.0024333526380360126,
+            0.008853988721966743,
+            -0.017964119091629982,
+            -0.01245705783367157,
+            -0.004772786516696215,
+            0.010117623955011368,
+            0.0007241358398459852,
+            -0.007569007575511932,
+            0.028175661340355873,
+            0.00502039073035121,
+            -0.005169807001948357,
+            0.003571051638573408,
+            -2.8232359909452498e-05,
+            0.019244831055402756,
+            0.01536000519990921,
+            0.008294744417071342,
+            -0.03266669064760208,
+            -0.007304327096790075,
+            -0.005780279636383057,
+            0.013114489614963531,
+            0.002813297091051936,
+            0.003423769725486636,
+            -0.019740039482712746,
+            -0.020798761397600174,
+            0.019210679456591606,
+            0.004935009870678186,
+            0.019825419411063194,
+            -0.00867895781993866,
+            -0.006548706907778978,
+            -0.004841091111302376,
+            -0.004742903169244528,
+            0.00499477656558156,
+            0.0076415808871388435,
+            0.0038933642208576202,
+            0.03377664089202881,
+            -0.001280711847357452,
+            0.004042780492454767,
+            -0.0036350872833281755,
+            -0.0002441357064526528,
+            0.014787953346967697,
+            -0.00790199264883995,
+            0.008905217051506042,
+            0.018800850957632065,
+            -0.02887578494846821,
+            0.011398335918784142,
+            -0.01679440215229988,
+            -0.00828193686902523,
+            0.016000360250473022,
+            -0.0005320290802046657,
+            -0.004922202788293362,
+            -0.014583039097487926,
+            0.0008127184119075537,
+            0.007752575911581516,
+            0.01996202953159809,
+            -0.004091874696314335,
+            0.011287340894341469,
+            -0.009648029692471027,
+            -0.004305326379835606,
+            0.010083471424877644,
+            0.006087650544941425,
+            -0.0033213128335773945,
+            0.006232798099517822
+          ],
+          "keyphrases": [
+            "MoE models",
+            "expert choice",
+            "training efficiency",
+            "fine-tuning on GLUE and SuperGLUE",
+            "capacity factor"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "1ea6fc21-365a-41fe-9b07-1e5411ca0bc9",
+        "properties": {
+          "page_content": "4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n",
+          "title": "Validation Experiments",
+          "embedding": [
+            -0.010754532180726528,
+            -0.023697586730122566,
+            -0.02424471639096737,
+            -0.004748920910060406,
+            0.0029044931288808584,
+            0.014695581048727036,
+            -0.013233717530965805,
+            0.03884626179933548,
+            -0.0167387705296278,
+            0.032195206731557846,
+            -0.028296900913119316,
+            0.003022040706127882,
+            0.033374954015016556,
+            -0.032622650265693665,
+            -0.007941939868032932,
+            0.013165325857698917,
+            0.035289913415908813,
+            0.03330656513571739,
+            0.011327309533953667,
+            -0.037751998752355576,
+            0.04247099906206131,
+            -0.036520954221487045,
+            -0.060047563165426254,
+            -0.011079391464591026,
+            -0.019577009603381157,
+            0.02089354395866394,
+            0.00764272827655077,
+            0.016584891825914383,
+            -0.0636381059885025,
+            0.03330656513571739,
+            0.00491989916190505,
+            0.023150455206632614,
+            0.0002941360289696604,
+            0.008754086680710316,
+            -0.011498288251459599,
+            -0.03205842152237892,
+            0.022381054237484932,
+            -0.0009312971960753202,
+            -0.015362396836280823,
+            0.042607780545949936,
+            0.03381950035691261,
+            0.01822628267109394,
+            -0.023372728377580643,
+            -0.004340710584074259,
+            0.016832809895277023,
+            0.00703148078173399,
+            -0.013173874467611313,
+            -0.010062070563435555,
+            0.015704352408647537,
+            -0.04240260645747185,
+            -0.03546088933944702,
+            -0.011600874364376068,
+            0.008172760717570782,
+            -0.01200267393141985,
+            -0.03761521726846695,
+            0.007856450974941254,
+            0.016140347346663475,
+            0.026159673929214478,
+            0.04093219339847565,
+            0.015439336188137531,
+            0.027305228635668755,
+            -0.018602434545755386,
+            -0.019765086472034454,
+            0.010164657607674599,
+            0.0010231980122625828,
+            -0.03443502262234688,
+            0.0016926847165450454,
+            -0.011720559559762478,
+            0.017046531662344933,
+            -0.034383729100227356,
+            -0.04520665109157562,
+            0.02380017377436161,
+            -0.009292668662965298,
+            -0.010993901640176773,
+            -0.0009270227164961398,
+            -0.010198852978646755,
+            0.031203532591462135,
+            0.027544597163796425,
+            -0.0026907704304903746,
+            0.000909390626475215,
+            0.014806716702878475,
+            0.02405664138495922,
+            -0.023714683949947357,
+            0.0028040434699505568,
+            0.04373623803257942,
+            -0.0030434129294008017,
+            -0.02814302034676075,
+            -0.06527949869632721,
+            0.06415104120969772,
+            -0.0167815163731575,
+            -0.013644064776599407,
+            0.0002978761913254857,
+            0.023646293208003044,
+            -0.019577009603381157,
+            -0.029237281531095505,
+            0.05844036862254143,
+            -0.007394809741526842,
+            0.0007074225577525795,
+            -0.018619531765580177,
+            0.014601543545722961,
+            -0.009771407581865788,
+            0.010865667834877968,
+            -0.008617304265499115,
+            0.014422016218304634,
+            0.032622650265693665,
+            -0.024877335876226425,
+            -0.014447662979364395,
+            0.012831917963922024,
+            0.013712456449866295,
+            0.10628008842468262,
+            -0.016217287629842758,
+            -0.005920122377574444,
+            -0.03257135674357414,
+            0.0011006725253537297,
+            -0.009745760820806026,
+            -0.05006243288516998,
+            -0.026826487854123116,
+            -0.025373173877596855,
+            0.031237727031111717,
+            -0.008035978302359581,
+            0.033836595714092255,
+            0.032434575259685516,
+            0.045035671442747116,
+            -0.0132422661408782,
+            -0.04804489016532898,
+            0.046950630843639374,
+            -0.0051549943163990974,
+            0.000823367154225707,
+            -0.0025347527116537094,
+            -0.003041275776922703,
+            -0.0017194000538438559,
+            -0.007373437285423279,
+            -0.01822628267109394,
+            -0.01443056482821703,
+            0.011831695213913918,
+            0.0043449848890304565,
+            0.034298237413167953,
+            0.053003259003162384,
+            -0.04373623803257942,
+            -0.009685918688774109,
+            0.026792293414473534,
+            -0.029938293620944023,
+            0.010985353030264378,
+            0.006091100629419088,
+            -0.02403954230248928,
+            0.015011890791356564,
+            0.0010638053063303232,
+            0.055465348064899445,
+            0.0003606572572607547,
+            0.02374888025224209,
+            0.0011113586369901896,
+            0.04274456575512886,
+            -0.008360836654901505,
+            0.04363365098834038,
+            0.010592103004455566,
+            0.0010258695110678673,
+            -0.020927738398313522,
+            0.018602434545755386,
+            -0.039427585899829865,
+            -0.000900841667316854,
+            0.03171646595001221,
+            -0.0211500097066164,
+            -0.009899641387164593,
+            0.012660940177738667,
+            0.004398415796458721,
+            0.06869906187057495,
+            -0.01395182590931654,
+            -0.0074888477101922035,
+            -0.0781712606549263,
+            0.04982306435704231,
+            0.023971151560544968,
+            0.00814711395651102,
+            0.005907298997044563,
+            0.04182128235697746,
+            0.003832050133496523,
+            0.01309693418443203,
+            0.012438667938113213,
+            -0.0098397983238101,
+            -0.05026760697364807,
+            0.03207552060484886,
+            -0.0026587119791656733,
+            0.012908858247101307,
+            -0.009130238555371761,
+            0.026860684156417847,
+            0.05505499988794327,
+            0.022021999582648277,
+            0.01165216788649559,
+            0.027476206421852112,
+            -0.020671270787715912,
+            0.01749962382018566,
+            0.037033889442682266,
+            -0.0011412798194214702,
+            0.03744423761963844,
+            0.026980368420481682,
+            -0.005979964509606361,
+            -0.013105483725667,
+            -0.028348194435238838,
+            0.013797945342957973,
+            0.006086825858801603,
+            0.03405886888504028,
+            -0.02034641243517399,
+            -0.03525571525096893,
+            0.007664100266993046,
+            -0.0034281141124665737,
+            0.04079541191458702,
+            -0.00934396218508482,
+            0.007822255603969097,
+            0.023099161684513092,
+            0.010942608118057251,
+            -0.004847233649343252,
+            -0.024210521951317787,
+            -0.008540363982319832,
+            -0.01188298873603344,
+            -0.0402824766933918,
+            -0.011891537345945835,
+            0.03157968446612358,
+            -0.015353847295045853,
+            0.01938893459737301,
+            -0.005915847606956959,
+            -0.007800883147865534,
+            -0.023543706163764,
+            0.018072402104735374,
+            0.0032464496325701475,
+            -0.008027428761124611,
+            -0.0069075217470526695,
+            0.003357585519552231,
+            -0.0019352601375430822,
+            0.003267821855843067,
+            0.048181671649217606,
+            -0.02402244508266449,
+            0.012113809585571289,
+            0.008527540601789951,
+            0.0011316622840240598,
+            0.02380017377436161,
+            0.03346044570207596,
+            -0.035563476383686066,
+            -0.012934505008161068,
+            -0.0014618640998378396,
+            0.009514939971268177,
+            0.020876444876194,
+            -0.009873994626104832,
+            -0.0097030159085989,
+            -0.03378530219197273,
+            -0.0005765173118561506,
+            0.00013043770741205662,
+            -0.015712901949882507,
+            -0.008467698469758034,
+            0.017166217789053917,
+            0.042641978710889816,
+            0.032434575259685516,
+            0.02034641243517399,
+            -0.035289913415908813,
+            -0.00907894503325224,
+            0.027082955464720726,
+            -0.014610092155635357,
+            0.034315336495637894,
+            0.0504043884575367,
+            -0.029459552839398384,
+            -0.018773412331938744,
+            -0.03980373963713646,
+            0.07112695276737213,
+            -0.023731783032417297,
+            -0.02964762970805168,
+            0.042368412017822266,
+            -0.01408860832452774,
+            0.0021778354421257973,
+            0.03187034651637077,
+            0.035324107855558395,
+            -0.0035307009238749743,
+            0.012053967453539371,
+            0.03499924764037132,
+            -0.0194060318171978,
+            -0.04691643267869949,
+            -0.02523639053106308,
+            -0.025116706266999245,
+            -0.016619086265563965,
+            -0.010275793261826038,
+            -0.017311548814177513,
+            0.010087717324495316,
+            0.020278021693229675,
+            0.013635516166687012,
+            -0.010882765986025333,
+            0.007022931706160307,
+            0.009985130280256271,
+            -0.009976581670343876,
+            -0.000917939527425915,
+            -0.027561694383621216,
+            -0.0071939099580049515,
+            -0.028467880561947823,
+            -0.012053967453539371,
+            -0.014977695420384407,
+            -0.006685249973088503,
+            -0.025783522054553032,
+            0.01889309659600258,
+            0.023167554289102554,
+            -0.002190658822655678,
+            -0.008459148928523064,
+            0.01238737441599369,
+            0.03472568467259407,
+            -0.028108825907111168,
+            -0.03812815248966217,
+            0.008916515856981277,
+            -0.0034558980260044336,
+            0.009762858040630817,
+            -0.015037537552416325,
+            0.010677591897547245,
+            -0.011395701207220554,
+            0.010540809482336044,
+            -0.010703238658607006,
+            -0.010002228431403637,
+            -0.0006646779947914183,
+            0.021953608840703964,
+            0.00801460538059473,
+            -0.014592994004487991,
+            -0.017593663185834885,
+            0.027715574949979782,
+            0.014293782413005829,
+            0.0334775410592556,
+            -0.029545042663812637,
+            -0.023099161684513092,
+            0.042573586106300354,
+            0.004015851765871048,
+            -0.043086521327495575,
+            0.016148896887898445,
+            0.01671312376856804,
+            0.011797499842941761,
+            0.01942313089966774,
+            0.048181671649217606,
+            -0.033870793879032135,
+            -0.038196541368961334,
+            0.005420010536909103,
+            -0.031938739120960236,
+            -0.030929965898394585,
+            0.014883656986057758,
+            -0.006227883044630289,
+            -0.05125928297638893,
+            -0.031203532591462135,
+            -0.014020216651260853,
+            -0.019337641075253487,
+            0.0022098938934504986,
+            -0.015046087093651295,
+            0.004941271618008614,
+            -0.03741004317998886,
+            0.0025689483154565096,
+            0.0004648471367545426,
+            -0.018328869715332985,
+            -0.028724348172545433,
+            0.0027548871003091335,
+            0.004488179460167885,
+            0.03542669489979744,
+            0.0017097825184464455,
+            0.01128456462174654,
+            -0.012951603159308434,
+            -0.000579188868869096,
+            0.003056236309930682,
+            -0.05016501992940903,
+            -0.019474424421787262,
+            0.038983043283224106,
+            0.009027652442455292,
+            0.006048355717211962,
+            0.011874440126121044,
+            -0.028211412951350212,
+            -0.0592268668115139,
+            0.029716020449995995,
+            -0.04643769562244415,
+            0.015268358401954174,
+            0.004415513481944799,
+            -0.0010344184702262282,
+            -0.001160514890216291,
+            -0.0007758138235658407,
+            -0.0067707388661801815,
+            0.023167554289102554,
+            -0.007407633122056723,
+            0.012498511001467705,
+            -0.0028788463678210974,
+            0.03669193387031555,
+            0.0358712375164032,
+            -0.00673226872459054,
+            -0.038948848843574524,
+            0.02956213988363743,
+            -0.008668597787618637,
+            0.025612542405724525,
+            0.01889309659600258,
+            0.02961343340575695,
+            -0.017918521538376808,
+            0.032417476177215576,
+            0.002891669748350978,
+            0.024125032126903534,
+            -0.03877786919474602,
+            0.023167554289102554,
+            0.0052191112190485,
+            -0.054268497973680496,
+            -0.00860020611435175,
+            -0.0073777115903794765,
+            0.022244270890951157,
+            0.008822478353977203,
+            -0.02477474883198738,
+            -0.021970706060528755,
+            0.0009409147314727306,
+            -0.02817721664905548,
+            -0.007574336603283882,
+            0.04568539187312126,
+            -0.012558353133499622,
+            0.0014875108608976007,
+            -0.017414135858416557,
+            0.014174097217619419,
+            0.02886112965643406,
+            0.027510400861501694,
+            0.03354593366384506,
+            -0.013986021280288696,
+            0.020021554082632065,
+            -0.022278467193245888,
+            -0.037033889442682266,
+            -0.02595449984073639,
+            0.0073050460778176785,
+            0.03939339146018028,
+            0.009728662669658661,
+            0.006039807107299566,
+            -0.0013913356233388186,
+            0.00790346972644329,
+            -0.015370945446193218,
+            0.027989141643047333,
+            0.044796302914619446,
+            -0.002494145417585969,
+            -0.011464091949164867,
+            0.007505945395678282,
+            0.015046087093651295,
+            -0.018619531765580177,
+            0.027373619377613068,
+            -0.025424467399716377,
+            0.02255203202366829,
+            0.004744646605104208,
+            0.02065417356789112,
+            -0.047702934592962265,
+            -0.004556570667773485,
+            -0.0023082063999027014,
+            -0.03445211797952652,
+            0.05946623906493187,
+            -0.028690151870250702,
+            -0.013524380512535572,
+            0.0029386887326836586,
+            -0.0011316622840240598,
+            -0.01313113048672676,
+            -0.026980368420481682,
+            -0.014849461615085602,
+            -0.0061894129030406475,
+            -0.04554860666394234,
+            -0.029801510274410248,
+            -0.014815266244113445,
+            -0.01213945634663105,
+            -0.06240706518292427,
+            0.004058596212416887,
+            -0.0741361752152443,
+            0.007206733338534832,
+            0.027715574949979782,
+            -0.016661832109093666,
+            0.03963275998830795,
+            -0.022637521848082542,
+            0.02374888025224209,
+            -0.04076121747493744,
+            -0.0157299991697073,
+            0.03258845582604408,
+            0.047463566064834595,
+            0.010848570615053177,
+            0.011498288251459599,
+            0.021047422662377357,
+            -0.03199003264307976,
+            -0.0026758096646517515,
+            -0.00860875565558672,
+            -0.0018219870980829,
+            0.03332366421818733,
+            -0.008553187362849712,
+            0.021765531972050667,
+            0.0051549943163990974,
+            0.024142129346728325,
+            0.031442902982234955,
+            -0.002782671246677637,
+            -0.023868564516305923,
+            0.012857564724981785,
+            0.025544151663780212,
+            0.013054190203547478,
+            -0.014251037500798702,
+            0.012686586938798428,
+            -0.01579839177429676,
+            0.007719668094068766,
+            0.008484795689582825,
+            0.009908189997076988,
+            0.034315336495637894,
+            0.024142129346728325,
+            -0.013105483725667,
+            0.0029814334120601416,
+            0.009865445084869862,
+            -0.03645256534218788,
+            0.042641978710889816,
+            0.0051849158480763435,
+            -0.03953017294406891,
+            -0.0005487333401106298,
+            -0.007617081515491009,
+            0.020551586523652077,
+            0.02156035788357258,
+            0.00936105940490961,
+            -0.022671716287732124,
+            -0.00020944836433045566,
+            0.016841357573866844,
+            0.008660049177706242,
+            -0.012182201258838177,
+            0.02450118400156498,
+            0.03833332657814026,
+            -0.020260922610759735,
+            -0.030639303848147392,
+            0.00922427698969841,
+            -0.010489515960216522,
+            0.0007736766128800809,
+            -0.02715134806931019,
+            -0.004394141025841236,
+            0.018072402104735374,
+            -0.044762108474969864,
+            0.05177221819758415,
+            -0.013575674034655094,
+            0.008685695938766003,
+            -0.012943054549396038,
+            -0.002703593811020255,
+            -0.02138938009738922,
+            0.034588903188705444,
+            0.014131353236734867,
+            0.02867305465042591,
+            0.016106151044368744,
+            0.010318538174033165,
+            0.028040433302521706,
+            -0.020278021693229675,
+            -0.023099161684513092,
+            0.014422016218304634,
+            -0.007980410009622574,
+            -0.00467198109254241,
+            -0.02959633618593216,
+            -0.024107934907078743,
+            -0.01892729289829731,
+            0.02527058683335781,
+            -0.01530255377292633,
+            -0.008566010743379593,
+            0.010951157659292221,
+            0.0014981969725340605,
+            -0.002131885150447488,
+            0.013943277299404144,
+            -0.007638453505933285,
+            0.008707067929208279,
+            0.04192386940121651,
+            -0.013934727758169174,
+            0.02159455418586731,
+            0.03611060976982117,
+            0.011472641490399837,
+            -0.018516944721341133,
+            -0.0064800758846104145,
+            -0.0007293291273526847,
+            -0.01066904328763485,
+            -0.03570026159286499,
+            0.035289913415908813,
+            0.020996129140257835,
+            -0.0021842471323907375,
+            -0.037239063531160355,
+            -0.012583999894559383,
+            0.014635738916695118,
+            -0.014037314802408218,
+            0.027732674032449722,
+            -0.015943722799420357,
+            -0.015653058886528015,
+            0.00474037230014801,
+            0.021970706060528755,
+            -0.049857258796691895,
+            -0.012977249920368195,
+            0.010899864137172699,
+            0.002707868115976453,
+            0.008057350292801857,
+            -0.006779287941753864,
+            0.013455988839268684,
+            0.020568683743476868,
+            0.004945545922964811,
+            0.0036439741961658,
+            -0.0038085407577455044,
+            0.01647375524044037,
+            0.013455988839268684,
+            -0.017593663185834885,
+            -0.009241375140845776,
+            0.02210748940706253,
+            0.0230820644646883,
+            -0.030211858451366425,
+            -0.023919858038425446,
+            -0.00583035871386528,
+            0.021902315318584442,
+            -0.004753195680677891,
+            0.03689710795879364,
+            0.0388120636343956,
+            0.011438445188105106,
+            -0.04318910837173462,
+            -0.014516053721308708,
+            0.0017290175892412663,
+            -0.010660494677722454,
+            0.016431011259555817,
+            -0.0012438668636605144,
+            -0.042812954634428024,
+            0.002143639838322997,
+            -0.0036055040545761585,
+            0.028450781479477882,
+            -0.040214087814092636,
+            -0.011959929019212723,
+            0.009617527015507221,
+            -0.012951603159308434,
+            0.003302017692476511,
+            0.012182201258838177,
+            0.008484795689582825,
+            0.021269695833325386,
+            0.012703684158623219,
+            -0.005526872351765633,
+            -0.0014607955235987902,
+            0.016550695523619652,
+            0.004535198211669922,
+            0.026604216545820236,
+            -0.018328869715332985,
+            -0.017345745116472244,
+            0.01844855397939682,
+            0.03860688954591751,
+            -0.012327532283961773,
+            0.04076121747493744,
+            0.007514494471251965,
+            0.012062516063451767,
+            0.012088162824511528,
+            -0.02453538030385971,
+            -0.01598646678030491,
+            -0.011951380409300327,
+            0.006257804110646248,
+            -0.037512630224227905,
+            0.014858010224997997,
+            -0.02380017377436161,
+            -0.027134248986840248,
+            0.02619386836886406,
+            -0.006014160346239805,
+            0.010686141438782215,
+            0.022244270890951157,
+            -0.012891761027276516,
+            -0.027271032333374023,
+            0.03369981423020363,
+            0.034349530935287476,
+            0.008433502167463303,
+            0.04677965119481087,
+            -0.010395478457212448,
+            0.006779287941753864,
+            0.005702124908566475,
+            0.0024920080322772264,
+            0.044762108474969864,
+            -0.008236877620220184,
+            -0.01200267393141985,
+            0.007535866927355528,
+            0.009779956191778183,
+            0.04206065088510513,
+            -0.015413690358400345,
+            -0.049412716180086136,
+            0.019508618861436844,
+            -0.014507505111396313,
+            0.009856896474957466,
+            -0.006655328441411257,
+            0.008198407478630543,
+            -0.005890200845897198,
+            0.010053521953523159,
+            -0.004522374831140041,
+            0.010113364085555077,
+            -0.005843182094395161,
+            -0.014327977783977985,
+            0.015174319967627525,
+            -0.01018175482749939,
+            0.030280249193310738,
+            0.009557684883475304,
+            -0.02503121644258499,
+            -0.026501629501581192,
+            0.01746542938053608,
+            0.0019277798710390925,
+            -0.028724348172545433,
+            0.009882543236017227,
+            0.0023872838355600834,
+            0.010104815475642681,
+            0.004381317645311356,
+            -0.012242043390870094,
+            0.015926625579595566,
+            0.01676441729068756,
+            -0.022210076451301575,
+            0.03911982476711273,
+            -0.014285233803093433,
+            -0.014644287526607513,
+            0.019235054031014442,
+            -0.03730745613574982,
+            0.008424953557550907,
+            -0.012524157762527466,
+            0.010831472463905811,
+            0.00491989916190505,
+            0.004667706321924925,
+            -0.011352956295013428,
+            -0.0026715353596955538,
+            -0.02352660894393921,
+            0.03281072899699211,
+            0.0272197388112545,
+            -0.0012834055814892054,
+            0.005958592053502798,
+            -0.007535866927355528,
+            -0.012900309637188911,
+            0.0024984197225421667,
+            0.026142574846744537,
+            -0.024381499737501144,
+            -0.0017899286467581987,
+            -0.0006310166209004819,
+            0.007694021798670292,
+            0.00551832327619195,
+            0.022432347759604454,
+            0.04900236800312996,
+            -0.0038192267529666424,
+            0.028980813920497894,
+            0.035050541162490845,
+            0.024142129346728325,
+            -0.003259273013100028,
+            0.004761744290590286,
+            -0.025150902569293976,
+            0.03645256534218788,
+            0.008950712159276009,
+            0.016670379787683487,
+            0.017798837274312973,
+            -0.04312071576714516,
+            -0.009634625166654587,
+            0.019525716081261635,
+            0.02205619588494301,
+            0.020089944824576378,
+            -0.003881206503137946,
+            -0.0035884061362594366,
+            0.03298170492053032,
+            -0.025920303538441658,
+            -0.019354738295078278,
+            0.004154771566390991,
+            -0.0053174239583313465,
+            -0.005732045974582434,
+            -0.0069032469764351845,
+            0.01865372806787491,
+            -0.014584445394575596,
+            -0.004458257928490639,
+            0.016867004334926605,
+            0.001722605898976326,
+            0.012113809585571289,
+            0.016165994107723236,
+            -0.00024137321452144533,
+            0.022415248677134514,
+            -0.004932722542434931,
+            0.021509064361453056,
+            0.005330247338861227,
+            -0.003663209266960621,
+            -0.006911796052008867,
+            -0.010275793261826038,
+            0.029015010222792625,
+            -0.006980187259614468,
+            -0.034640196710824966,
+            -0.01360986940562725,
+            -0.009284119121730328,
+            0.004156908951699734,
+            0.0011391426669433713,
+            0.0050524077378213406,
+            0.01744833029806614,
+            0.026501629501581192,
+            -0.00046751866466365755,
+            0.004280867986381054,
+            0.014336527325212955,
+            -0.0024193422868847847,
+            -0.010506614111363888,
+            0.019559912383556366,
+            0.008779733441770077,
+            -0.01624293439090252,
+            0.018756315112113953,
+            0.0030861576087772846,
+            0.04688223823904991,
+            -0.044522739946842194,
+            0.018209183588624,
+            0.007805157452821732,
+            0.0006956678116694093,
+            -0.014960597269237041,
+            -0.03327237069606781,
+            0.0036097783595323563,
+            -0.014122803695499897,
+            -0.0033597226720303297,
+            -0.024689260870218277,
+            0.00786499958485365,
+            -0.018089499324560165,
+            0.014045863412320614,
+            0.011600874364376068,
+            -0.0016349796205759048,
+            -0.002028229646384716,
+            0.017542369663715363,
+            -0.001940603251568973,
+            0.00787782296538353,
+            -0.017362842336297035,
+            0.010942608118057251,
+            0.022381054237484932,
+            -0.010677591897547245,
+            0.008638676255941391,
+            0.0073306928388774395,
+            -0.02793784812092781,
+            -0.028536271303892136,
+            0.013336303643882275,
+            -0.03546088933944702,
+            0.007681198418140411,
+            0.031203532591462135,
+            -0.032434575259685516,
+            -0.028519174084067345,
+            0.0019085448002442718,
+            -0.021406477317214012,
+            -0.017867228016257286,
+            0.011446994729340076,
+            0.018021108582615852,
+            -0.004216751083731651,
+            -0.030451226979494095,
+            0.013173874467611313,
+            -0.007809431757777929,
+            0.00698873633518815,
+            0.00015388043539132923,
+            -0.02569803223013878,
+            -0.02569803223013878,
+            0.004984016064554453,
+            -0.028211412951350212,
+            -0.007138342130929232,
+            -0.0005297654424794018,
+            -0.024466989561915398,
+            -0.043804630637168884,
+            -0.015242711640894413,
+            -0.013507282361388206,
+            -0.010156108066439629,
+            0.0480106957256794,
+            0.0196795966476202,
+            0.026894880458712578,
+            -0.0003211185394320637,
+            -0.018106596544384956,
+            0.04773712903261185,
+            0.0009954140987247229,
+            0.041752889752388,
+            0.013387597166001797,
+            -0.008335189893841743,
+            -0.006693798583000898,
+            0.03392208740115166,
+            -0.015584668144583702,
+            -0.007775236386805773,
+            -0.010651945136487484,
+            -0.0015836860984563828,
+            0.0025091059505939484,
+            0.022449444979429245,
+            0.0019085448002442718,
+            0.004037224221974611,
+            0.027100054547190666,
+            -0.023201748728752136,
+            0.002739926567301154,
+            0.01238737441599369,
+            -0.006937442813068628,
+            0.008215505629777908,
+            -0.02838239073753357,
+            -0.008300994522869587,
+            0.019081173464655876,
+            0.006044081412255764,
+            -0.007437554188072681,
+            -0.0022440897300839424,
+            -0.027134248986840248,
+            -0.0014052275801077485,
+            0.0038106779102236032,
+            -0.005826083943247795,
+            0.005744869355112314,
+            0.04459112882614136,
+            -0.004817312583327293,
+            -0.039940521121025085,
+            -0.00029493749025277793,
+            -0.03634997829794884,
+            0.0001610935723874718,
+            0.00010799681331263855,
+            -0.0033191153779625893,
+            0.005488402210175991,
+            0.0018422907451167703,
+            -0.012472864240407944,
+            0.012130907736718655,
+            -0.028741445392370224,
+            0.019782183691859245,
+            0.004817312583327293,
+            -0.007574336603283882,
+            -0.005056682042777538,
+            0.01988477073609829,
+            0.008151388727128506,
+            -0.013148227706551552,
+            -0.0014960597036406398,
+            0.0028852580580860376,
+            -0.026125477626919746,
+            -0.020773857831954956,
+            -0.03369981423020363,
+            -0.0022505014203488827,
+            -0.009583331644535065,
+            -0.03010927140712738,
+            -0.006937442813068628,
+            -0.0003617258626036346,
+            0.007775236386805773,
+            -0.020295118913054466,
+            0.0006956678116694093,
+            0.005552519112825394,
+            0.012199298478662968,
+            0.00624070642516017,
+            -0.007339241914451122,
+            -0.004263770300894976,
+            -0.03033154271543026,
+            0.003584131831303239,
+            -0.0015345298452302814,
+            0.0038619714323431253,
+            -0.016063407063484192,
+            0.006351842079311609,
+            -0.016183091327548027,
+            -0.0035264266189187765,
+            0.011335858143866062,
+            0.01800401136279106,
+            -0.028741445392370224,
+            0.036726128309965134,
+            0.02889532595872879,
+            0.017320098355412483,
+            -0.038914650678634644,
+            0.028587564826011658,
+            -0.0012331806356087327,
+            0.001522775157354772,
+            0.0030177661683410406,
+            0.021731335669755936,
+            0.026039987802505493,
+            -0.016431011259555817,
+            -0.006531369406729937,
+            0.010224499739706516,
+            0.028194313868880272,
+            -0.031545490026474,
+            -0.007266575936228037,
+            1.9886239897459745e-05,
+            0.02378307469189167,
+            0.027783967554569244,
+            0.023116260766983032,
+            0.04271036759018898,
+            -0.0053131491877138615,
+            0.002891669748350978,
+            -0.005082328803837299,
+            -0.013926179148256779,
+            0.0024727729614824057,
+            -0.0028767092153429985,
+            -0.0017952717607840896,
+            0.011489738710224628,
+            0.012695135548710823,
+            0.017593663185834885,
+            -0.07126373797655106,
+            0.038743674755096436,
+            0.026570022106170654,
+            0.012105260975658894,
+            0.008399306796491146,
+            -0.024706358090043068,
+            -0.0017909972229972482,
+            -0.0022205801215022802,
+            0.0018604571232572198,
+            0.013438890688121319,
+            -0.010874217376112938,
+            -0.02475765161216259,
+            0.016567792743444443,
+            -0.019457325339317322,
+            0.017149118706583977,
+            0.0050267609767615795,
+            0.00503958435729146,
+            -0.024603771045804024,
+            -0.014721227809786797,
+            0.006830581463873386,
+            0.008065898902714252,
+            0.008100095205008984,
+            -0.03857269510626793,
+            -0.044727910310029984,
+            0.01750817336142063,
+            0.008035978302359581,
+            -0.04862621799111366,
+            -0.006586937233805656,
+            -0.017388489097356796,
+            -0.004210339393466711,
+            -0.006326195318251848,
+            -0.01767915114760399,
+            -0.040897998958826065,
+            0.002064562402665615,
+            0.017302999272942543,
+            0.018534043803811073,
+            5.349749335437082e-05,
+            0.028826933354139328,
+            0.0017183314776048064,
+            -0.0022355406545102596,
+            -0.030434129759669304,
+            0.026860684156417847,
+            -0.007326418533921242,
+            -0.022278467193245888,
+            0.024142129346728325,
+            0.01628567837178707,
+            -0.019970260560512543,
+            -0.0038769319653511047,
+            0.0014928538585081697,
+            -0.008446325547993183,
+            0.034554705023765564,
+            -0.0017129883635789156,
+            -0.008933614008128643,
+            0.0026907704304903746,
+            -0.0020998266991227865,
+            -0.00959188025444746,
+            -0.02479184791445732,
+            0.00145438383333385,
+            0.025869010016322136,
+            -0.007749589625746012,
+            0.006437331438064575,
+            0.04202645644545555,
+            -0.012353179045021534,
+            -0.019850576296448708,
+            0.00043145293602719903,
+            -0.02839948795735836,
+            0.010318538174033165,
+            -0.006685249973088503,
+            0.030280249193310738,
+            0.004206065088510513,
+            0.013002896681427956,
+            -0.016644733026623726,
+            -0.01066904328763485,
+            -0.021970706060528755,
+            -0.0641852393746376,
+            -0.012122358195483685,
+            -0.005561067722737789,
+            -0.024894434958696365,
+            -0.02398824878036976,
+            0.03234908729791641,
+            0.022398151457309723,
+            0.010609201155602932,
+            -0.002107307082042098,
+            -0.01891019567847252,
+            -0.004458257928490639,
+            0.03009217418730259,
+            -0.005458480678498745,
+            0.017371391877532005,
+            -0.011617972515523434,
+            0.01627713069319725,
+            -0.006578388623893261,
+            0.0051336223259568214,
+            -0.053960736840963364,
+            -0.005167817696928978,
+            0.02983570657670498,
+            0.01993606425821781,
+            0.007240929175168276,
+            0.010019325651228428,
+            0.013293559662997723,
+            -0.007501671090722084,
+            -0.02764718420803547,
+            0.02525348775088787,
+            0.017132021486759186,
+            0.01991896703839302,
+            -0.015935173258185387,
+            0.007587159983813763,
+            0.011788951233029366,
+            0.00288098375312984,
+            0.000775279535446316,
+            0.0008281759219244123,
+            0.025133803486824036,
+            0.003549935994669795,
+            -0.009275570511817932,
+            0.015721451491117477,
+            -0.007163988891988993,
+            -0.021269695833325386,
+            -0.004233849234879017,
+            0.008959260769188404,
+            0.004035086836665869,
+            0.001731154858134687,
+            -0.0015922350576147437,
+            -0.008984907530248165,
+            0.02886112965643406,
+            0.03539250046014786,
+            0.03183615207672119,
+            -0.0057876138016581535,
+            -0.005137896630913019,
+            0.0016456657322123647,
+            0.003513603238388896,
+            -0.007659825962036848,
+            -0.0018294673645868897,
+            0.019799282774329185,
+            -0.030006684362888336,
+            -0.011276016011834145,
+            -0.009318315424025059,
+            0.016362618654966354,
+            0.003522152081131935,
+            0.04411239176988602,
+            0.02111581526696682,
+            0.006522820331156254,
+            -0.00557389110326767,
+            0.0037209142465144396,
+            0.004389866720885038,
+            0.029237281531095505,
+            -0.02282559685409069,
+            9.023110033012927e-05,
+            -0.0034217024222016335,
+            -0.017901424318552017,
+            0.019046977162361145,
+            -0.007300771772861481,
+            0.009540586732327938,
+            0.02205619588494301,
+            0.09212308377027512,
+            -0.0065527418628335,
+            0.022449444979429245,
+            -0.027852358296513557,
+            -0.024090835824608803,
+            0.00291517935693264,
+            0.006313371937721968,
+            0.002265461953356862,
+            0.022039096802473068,
+            0.0054499320685863495,
+            0.02525348775088787,
+            0.0272197388112545,
+            0.0009067190694622695,
+            0.006698073353618383,
+            -0.010292891412973404,
+            0.004118438810110092,
+            -0.013182424008846283,
+            -0.009549135342240334,
+            -0.009437999688088894,
+            0.008779733441770077,
+            0.019012782722711563,
+            0.00911314133554697,
+            0.007531592156738043,
+            -0.004084242973476648,
+            -0.015482081100344658,
+            0.026518728584051132,
+            0.010292891412973404,
+            -0.004475356079638004,
+            -0.015601766295731068,
+            -0.015114477835595608,
+            0.012404472567141056,
+            -0.006886149290949106,
+            -0.005894475616514683,
+            0.01721751131117344,
+            0.005069505423307419,
+            0.02865595556795597,
+            -0.03369981423020363,
+            0.008890869095921516,
+            0.014259587042033672,
+            -0.0010857119923457503,
+            0.007245203480124474,
+            0.021423576399683952,
+            -0.0022633245680481195,
+            0.0242789126932621,
+            -0.01468703243881464,
+            0.00874553807079792,
+            0.010959706269204617,
+            -0.021201303228735924,
+            0.0036055040545761585,
+            0.01990186981856823,
+            -0.003381095128133893,
+            0.015413690358400345,
+            -0.0052191112190485,
+            -0.001976936124265194,
+            0.0025903205387294292,
+            0.006582662928849459,
+            0.00363756250590086,
+            -0.01988477073609829,
+            0.01188298873603344,
+            -0.01872211880981922,
+            0.009172983467578888,
+            -0.004565119277685881,
+            0.012190749868750572,
+            -0.008424953557550907,
+            0.011370054446160793,
+            -0.006121021695435047,
+            0.006847679149359465,
+            -0.0105750048533082,
+            -0.016559245064854622,
+            -0.012515608221292496,
+            0.012960151769220829,
+            0.007249478250741959,
+            -0.013558575883507729,
+            0.008309543132781982,
+            -0.008228329010307789,
+            -0.002067768247798085,
+            0.014960597269237041,
+            -0.011746206320822239,
+            0.00407355697825551,
+            -0.039461780339479446,
+            -0.004535198211669922,
+            0.018089499324560165,
+            -0.001959838205948472,
+            0.0027719850186258554,
+            -0.007352065294981003,
+            0.011199075728654861,
+            -0.008288171142339706,
+            -0.0024086562916636467,
+            0.0254757609218359,
+            -0.019782183691859245,
+            -0.017328646034002304,
+            -0.032913316041231155,
+            -0.05009663105010986,
+            -0.002131885150447488,
+            -0.009326864033937454,
+            0.005864554084837437,
+            -0.00790346972644329,
+            0.017815934494137764,
+            -0.0040179891511797905,
+            -0.007095597684383392,
+            -0.024125032126903534,
+            0.010395478457212448,
+            0.028211412951350212,
+            0.0008682489860802889,
+            -0.03079318441450596,
+            -0.012344630435109138,
+            0.010463869199156761,
+            0.0006646779947914183,
+            -0.029049206525087357,
+            0.0083693852648139,
+            0.0006181932403706014,
+            -0.014473309740424156,
+            -0.008882320486009121,
+            -0.005971415434032679,
+            0.021440673619508743,
+            0.024193422868847847,
+            0.005753418430685997,
+            -0.019782183691859245,
+            -0.004225300159305334,
+            0.03258845582604408,
+            -0.018294673413038254,
+            0.003994479309767485,
+            0.023868564516305923,
+            -0.00264375121332705,
+            0.00982270110398531,
+            0.01054935809224844,
+            0.018072402104735374,
+            -0.008352288044989109,
+            0.012660940177738667,
+            0.007561513222754002,
+            -0.015328200533986092,
+            0.008548912592232227,
+            0.006668151821941137,
+            0.016405364498496056,
+            0.006890423595905304,
+            -0.022244270890951157,
+            -0.004411239176988602,
+            0.015832586213946342,
+            0.053276825696229935,
+            -0.006706621963530779,
+            -0.008506168611347675,
+            0.03617899864912033,
+            0.024415696039795876,
+            0.007416181731969118,
+            0.005937220063060522,
+            -0.03231488913297653,
+            -0.004646334331482649,
+            0.0008709204848855734,
+            0.005236208904534578,
+            0.0132422661408782,
+            0.0035029170103371143,
+            -0.007770961616188288,
+            0.018978586420416832,
+            -0.001043501659296453,
+            0.014293782413005829,
+            -0.020107043907046318,
+            0.009514939971268177,
+            -0.00618513859808445,
+            0.03161387890577316,
+            0.01042112521827221,
+            -0.0010632710764184594,
+            0.03318687900900841,
+            0.006719445344060659,
+            -0.03344334661960602,
+            0.006125296000391245,
+            -0.007638453505933285,
+            0.011352956295013428,
+            -0.02260332554578781,
+            -0.007894921116530895,
+            0.009899641387164593,
+            0.0012289062142372131,
+            -0.018123695626854897,
+            0.011900086887180805,
+            0.007142616901546717,
+            0.009378157556056976,
+            -0.0005217508296482265,
+            0.006356116849929094,
+            0.008553187362849712,
+            -0.004522374831140041,
+            -0.008168485946953297,
+            0.005599537864327431,
+            -0.016336971893906593,
+            -0.0040179891511797905,
+            0.029425358399748802,
+            -0.0006726926076225936,
+            0.010309988632798195,
+            0.0004135536728426814,
+            -0.0004116835771128535,
+            -0.018978586420416832,
+            0.009207178838551044,
+            -0.024569574743509293,
+            -0.006415958981961012,
+            0.005595263559371233,
+            -0.01225059200078249,
+            -0.006142393685877323,
+            0.014413466677069664,
+            0.0022633245680481195,
+            -0.021047422662377357,
+            -0.008061625063419342,
+            -0.030485423281788826,
+            -0.021218402311205864,
+            0.031904544681310654,
+            0.006518546026200056,
+            -0.005646557081490755,
+            0.015379494056105614,
+            -0.022876890376210213,
+            -0.0054969508200883865,
+            -0.005035309586673975,
+            0.016636185348033905,
+            -0.020021554082632065,
+            0.004398415796458721,
+            -0.010634847916662693,
+            -0.016431011259555817,
+            0.023099161684513092,
+            0.008326641283929348,
+            0.0011220448650419712,
+            0.006133845075964928,
+            -0.02497992292046547,
+            -0.014285233803093433,
+            0.01798691228032112,
+            0.005702124908566475,
+            -0.008228329010307789,
+            0.01795271784067154,
+            -0.0012609646655619144,
+            -0.007215282414108515,
+            -0.013780847191810608,
+            0.01841435767710209,
+            -0.02183392271399498,
+            0.026535825803875923,
+            0.012156554497778416,
+            -0.010472417809069157,
+            0.008822478353977203,
+            0.004992565140128136,
+            0.01287466287612915,
+            -0.0003889755462296307,
+            -0.015370945446193218,
+            0.013421793468296528,
+            0.013712456449866295,
+            0.002194933360442519,
+            -0.0021778354421257973,
+            0.0037572472356259823,
+            0.005826083943247795,
+            -0.0054456572979688644,
+            0.02333853207528591,
+            0.023594999685883522,
+            0.0002186651690863073,
+            0.015080282464623451,
+            -0.02084225043654442,
+            -0.019611205905675888,
+            0.02378307469189167,
+            -0.004595040809363127,
+            -0.0314599983394146,
+            0.038709476590156555,
+            -0.010292891412973404,
+            -0.007236654870212078,
+            -0.01993606425821781,
+            0.00771111948415637,
+            0.020448999479413033,
+            -0.021885216236114502,
+            -0.010002228431403637,
+            -0.00110708421561867,
+            0.008848125115036964,
+            0.027852358296513557,
+            0.020568683743476868,
+            -0.015507727861404419,
+            0.02788655459880829,
+            0.009668820537626743,
+            0.008044526912271976,
+            0.011874440126121044,
+            0.005561067722737789,
+            -0.010840021073818207,
+            0.010044972412288189,
+            -0.0005241552134975791,
+            0.005296051502227783,
+            -0.00516354339197278,
+            -0.02231266349554062,
+            -0.015644511207938194,
+            -0.04178708791732788,
+            -0.0036589347291737795,
+            -0.008604480884969234,
+            0.01553337462246418,
+            0.08624143153429031,
+            -0.014037314802408218,
+            -0.009309765882790089,
+            -0.0020271609537303448,
+            0.009976581670343876,
+            -0.01720896176993847,
+            0.012566901743412018,
+            0.024466989561915398,
+            -0.013626967556774616,
+            -0.0329304113984108,
+            0.0025518503971397877,
+            -0.012994347140192986,
+            0.007805157452821732,
+            0.017542369663715363,
+            0.0019245740259066224,
+            -0.023680489510297775,
+            -0.0018433593213558197,
+            -0.020004456862807274,
+            0.001239592325873673,
+            0.010369831696152687,
+            -0.016131797805428505,
+            -0.010771630331873894,
+            0.023902760818600655,
+            -0.02017543464899063,
+            -0.025167999789118767,
+            -0.012925956398248672,
+            -0.007745314855128527,
+            -0.011891537345945835,
+            0.017815934494137764,
+            0.008647225797176361,
+            -0.0186708252876997,
+            -0.008595932275056839,
+            0.005915847606956959,
+            -0.011378603056073189,
+            8.963001164374873e-05,
+            0.011925733648240566,
+            -0.0033169782254844904,
+            0.008788282051682472,
+            0.01482381485402584,
+            -0.03400757536292076,
+            -0.003128902055323124,
+            -0.014704130589962006,
+            0.01360986940562725,
+            0.006557016167789698,
+            0.028057532384991646,
+            -0.028279803693294525,
+            -0.021765531972050667,
+            -0.00981415156275034,
+            -0.01892729289829731,
+            -0.03082738071680069,
+            -0.02692907489836216,
+            0.0058175353333354,
+            0.00636466545984149,
+            0.021526163443922997,
+            0.014729776419699192,
+            -0.014396369457244873,
+            0.010096265934407711,
+            -0.0033981928136199713,
+            0.008984907530248165,
+            -0.014404918067157269,
+            -0.0018454965902492404,
+            -0.015550472773611546,
+            0.015131575986742973,
+            0.017662053927779198,
+            -0.002583908848464489,
+            -0.018380163237452507,
+            -0.019730890169739723,
+            0.010429673828184605,
+            -0.007604258134961128,
+            -0.004637785255908966,
+            0.02668970637023449,
+            0.003182332729920745,
+            0.0019096133764833212,
+            0.007527317851781845,
+            0.023406922817230225,
+            0.008664323017001152,
+            0.03546088933944702,
+            -0.023406922817230225,
+            0.001999377040192485,
+            -0.0189443901181221,
+            0.0031866072677075863,
+            -0.023218847811222076,
+            0.005112249869853258,
+            -0.01942313089966774,
+            0.03417855501174927,
+            -0.006133845075964928,
+            -0.007852176204323769,
+            0.030468326061964035,
+            -0.02624516189098358,
+            0.002979296026751399,
+            -0.009754309430718422,
+            0.004650608636438847,
+            0.022141683846712112,
+            -0.0038256384432315826,
+            -0.017285902053117752,
+            -0.03925660625100136,
+            0.003522152081131935,
+            -0.009899641387164593,
+            -0.006091100629419088,
+            -0.011318760924041271,
+            0.01773044466972351,
+            0.005065230652689934,
+            0.005569616798311472,
+            -0.0015633824514225125,
+            0.016892651095986366,
+            0.0013913356233388186,
+            0.0035456616897135973,
+            0.029955390840768814,
+            -0.001200053608044982,
+            0.012558353133499622,
+            0.0033533109817653894,
+            0.005689301528036594,
+            0.023492412641644478,
+            -0.007745314855128527,
+            -0.0015206378884613514,
+            -0.017362842336297035,
+            0.0067237201146781445,
+            -0.0007555101765319705,
+            0.02520219422876835,
+            0.01539659220725298,
+            -0.01771334744989872,
+            -0.040658630430698395,
+            0.02159455418586731,
+            -0.011352956295013428,
+            0.027373619377613068,
+            -0.009497841820120811,
+            -0.006569839548319578,
+            -0.0003144397051073611,
+            -0.03754682466387749,
+            0.003182332729920745,
+            -0.009720114059746265,
+            -0.052798084914684296,
+            -0.018089499324560165,
+            -0.007548689842224121,
+            -0.026296455413103104,
+            0.03342624753713608,
+            -0.0014650699449703097,
+            -0.026775194332003593,
+            0.024706358090043068,
+            -0.0034516234882175922,
+            -0.00698873633518815,
+            0.012233494780957699,
+            -0.016704576089978218,
+            -0.009378157556056976,
+            -0.0032443124800920486,
+            -0.005116524174809456,
+            0.0006069728406146169,
+            -0.012421570718288422,
+            -0.019235054031014442,
+            0.015866782516241074,
+            -0.006980187259614468,
+            0.0008538226829841733,
+            0.015866782516241074,
+            -0.0020496018696576357,
+            0.025065412744879723,
+            -0.014122803695499897,
+            0.005385815165936947,
+            0.007612806744873524,
+            -0.013883434236049652,
+            0.012301885522902012,
+            0.003847010899335146,
+            -0.015430787578225136,
+            0.01580693945288658,
+            0.009215728379786015,
+            0.00848907046020031,
+            0.0041611832566559315,
+            -0.014533151872456074,
+            0.0022889713291078806,
+            -0.011421347968280315,
+            0.006646779831498861,
+            0.013909080997109413,
+            0.005304600577801466,
+            -0.011276016011834145,
+            -0.016653282567858696,
+            0.013857787474989891,
+            0.007501671090722084,
+            0.020004456862807274,
+            -0.03180195763707161,
+            0.007386260665953159,
+            -0.004676255397498608,
+            0.006941717118024826,
+            -0.01227623876184225,
+            0.025287684053182602,
+            0.022620422765612602,
+            0.02111581526696682,
+            0.006356116849929094,
+            0.0028767092153429985,
+            0.010515162721276283,
+            -0.01966249942779541,
+            -0.0033618600573390722,
+            -0.0204148031771183,
+            -0.0006417027907446027,
+            -0.0008324504015035927,
+            0.01743123307824135,
+            -0.012318983674049377,
+            0.013105483725667,
+            0.006522820331156254,
+            0.015447885729372501,
+            -0.02887822687625885,
+            0.0039047158788889647,
+            0.0070400298573076725,
+            0.0013421793701127172,
+            0.0066211330704391,
+            0.0031224903650581837,
+            0.08371095359325409,
+            0.010472417809069157,
+            -0.0070870486088097095,
+            0.003968833014369011,
+            -0.009437999688088894,
+            -0.0077538639307022095,
+            0.015251260250806808,
+            -0.01081437524408102,
+            0.012233494780957699,
+            0.007347790524363518,
+            -0.013148227706551552,
+            0.004582217428833246,
+            0.0073563395999372005,
+            0.008108643814921379,
+            -0.01623438484966755,
+            0.011190527118742466,
+            -0.005253307055681944,
+            0.0038961670361459255,
+            0.025133803486824036,
+            0.01652504876255989,
+            -0.005723497364670038,
+            0.00971156544983387,
+            0.006467252504080534,
+            0.0009863307932391763,
+            -0.012404472567141056,
+            0.006390312220901251,
+            -0.004152634646743536,
+            0.023013673722743988,
+            -0.016208738088607788,
+            0.0344863161444664,
+            -0.0005706399097107351,
+            -0.003855559742078185,
+            -0.010882765986025333,
+            -0.003383232280611992,
+            0.007501671090722084,
+            -0.008322366513311863,
+            0.01114778220653534,
+            -0.013712456449866295,
+            0.0005973553052172065,
+            -0.013045640662312508,
+            -0.008189858868718147,
+            0.004058596212416887,
+            -0.007540141232311726,
+            -0.0073777115903794765,
+            -0.0009211453725583851,
+            0.001756801619194448,
+            -4.190970867057331e-05,
+            -0.0012331806356087327,
+            -0.0012919544242322445,
+            -0.009437999688088894,
+            -0.004513826221227646,
+            -0.013601320795714855,
+            -1.5010688912298065e-05,
+            0.02137228287756443,
+            -0.003682444104924798,
+            -0.006809209007769823,
+            0.01044677197933197,
+            0.003128902055323124,
+            0.02429600991308689,
+            -0.006095374934375286,
+            -0.014558798633515835,
+            -0.0021564632188528776,
+            -0.007005834020674229,
+            -0.007390534970909357,
+            0.00033928497578017414,
+            0.0038790691178292036,
+            0.006180863827466965,
+            0.003287056926637888,
+            0.012438667938113213,
+            0.006018434651196003,
+            -0.008831026963889599,
+            0.011763304471969604,
+            0.003120353212580085,
+            -0.0009190081618726254,
+            -0.01891019567847252,
+            -0.003242175094783306,
+            -0.013550027273595333,
+            -0.01199412439018488,
+            -0.011455543339252472,
+            0.017345745116472244,
+            -0.01723460853099823,
+            0.019765086472034454,
+            -0.006339018698781729,
+            0.00765555165708065,
+            -0.02549285814166069,
+            0.002449263585731387,
+            0.008899418637156487,
+            0.011250369250774384,
+            0.006702347658574581,
+            -0.012857564724981785,
+            -0.002013268880546093,
+            -0.0010862462222576141,
+            0.02963053248822689,
+            -0.008702793158590794,
+            -0.010010777041316032,
+            0.02816011942923069,
+            -0.014413466677069664,
+            -0.01019030436873436,
+            0.0014362173387780786,
+            -0.0013090522261336446,
+            0.02552705444395542,
+            0.011455543339252472,
+            0.017328646034002304,
+            0.0054456572979688644,
+            0.002840376226231456,
+            0.0013379048323258758,
+            0.023936955258250237,
+            0.016071956604719162,
+            0.010908412747085094,
+            -0.01418264675885439,
+            -0.008446325547993183,
+            -0.01942313089966774,
+            0.014216842129826546,
+            0.0033746834378689528,
+            -0.01530255377292633,
+            0.011224722489714622,
+            -0.021953608840703964,
+            -0.019064076244831085,
+            -0.023697586730122566,
+            -0.010386928915977478,
+            0.013438890688121319,
+            -0.011686364188790321,
+            -0.005505499895662069,
+            -0.017089277505874634,
+            -0.005454206373542547,
+            -0.010455320589244366,
+            -0.015148673206567764,
+            -0.01673022285103798,
+            -0.02598869428038597,
+            -0.022688815370202065,
+            -0.0098397983238101,
+            -0.017294451594352722,
+            -0.008300994522869587,
+            0.012797722592949867,
+            -0.020448999479413033,
+            0.017662053927779198,
+            -0.005065230652689934,
+            0.0037893056869506836,
+            0.01188298873603344,
+            0.0216116514056921,
+            -0.01748252660036087,
+            -0.031904544681310654,
+            -0.02402244508266449,
+            -0.02135518379509449,
+            0.022705912590026855,
+            0.0021927962079644203,
+            0.024142129346728325,
+            -0.007125518750399351,
+            0.016157444566488266,
+            -0.018363064154982567,
+            0.019303444772958755,
+            -0.04202645644545555,
+            -0.03222940117120743,
+            -0.007168263662606478,
+            0.00034836819395422935,
+            -0.004362082574516535,
+            -0.0010199921671301126,
+            -0.015661608427762985,
+            -0.010993901640176773,
+            -0.007168263662606478,
+            -0.006210785359144211,
+            -0.009685918688774109,
+            -0.01516577135771513,
+            0.01007916871458292,
+            0.004370631650090218,
+            0.016165994107723236,
+            0.001985484967008233,
+            0.015559021383523941,
+            -0.01516577135771513,
+            0.006945991888642311,
+            -0.0022269918117672205,
+            -0.005753418430685997,
+            0.004646334331482649,
+            0.003804266219958663,
+            0.0044924537651240826,
+            0.0002427089784760028,
+            0.0011017411015927792,
+            -0.00029386888490989804,
+            0.017362842336297035,
+            0.013361950404942036,
+            -0.007146891206502914,
+            0.01773044466972351,
+            6.485151970991865e-05,
+            0.0016787927597761154,
+            -0.010429673828184605,
+            -0.003214391181245446,
+            -0.00911314133554697,
+            -0.008882320486009121,
+            0.015362396836280823,
+            -0.019730890169739723,
+            -0.010626298375427723,
+            0.026843586936593056,
+            -0.0018348104786127806,
+            -0.03443502262234688,
+            0.02742491289973259,
+            7.86015789344674e-06,
+            -0.0024364402052015066,
+            -0.01369535829871893,
+            -0.007078499998897314,
+            -0.0006363596767187119,
+            -0.0005978895933367312,
+            0.017337195575237274,
+            -0.018038205802440643,
+            -0.005249032285064459,
+            0.00630909763276577,
+            -0.0040436359122395515,
+            -0.0021254734601825476,
+            0.006497173570096493,
+            0.014028766192495823,
+            -0.00047820480540394783,
+            -0.03501634672284126,
+            -0.012190749868750572,
+            0.020568683743476868,
+            -0.017106374725699425,
+            -0.014994793571531773,
+            0.011233271099627018,
+            -0.001976936124265194,
+            -0.012601097114384174,
+            -0.003911127801984549,
+            -0.005223385989665985,
+            -0.010993901640176773,
+            0.008442051708698273,
+            0.010224499739706516,
+            0.008339464664459229,
+            0.02210748940706253,
+            -0.014447662979364395,
+            -0.005757692735642195,
+            -0.015251260250806808,
+            0.0031075298320502043,
+            -0.022962380200624466,
+            -0.00015214394079521298,
+            -0.009745760820806026,
+            0.0016766554908826947,
+            0.012575451284646988,
+            0.01749962382018566,
+            0.028724348172545433,
+            0.0089678093791008,
+            0.0072879483923316,
+            -0.010848570615053177,
+            -0.01456734724342823,
+            0.004680529702454805,
+            0.0031417254358530045,
+            -0.00558671448379755,
+            0.006655328441411257,
+            0.019286347553133965,
+            0.0013261501444503665,
+            0.019303444772958755,
+            0.0025133804883807898,
+            0.0037978545296937227,
+            -2.8652215405600145e-05,
+            -0.010455320589244366,
+            -0.006706621963530779,
+            -0.02374888025224209,
+            0.0007453583530150354,
+            0.005779065191745758,
+            0.01889309659600258,
+            0.0005540763959288597,
+            0.0030370012391358614,
+            0.026416141539812088,
+            -0.0029771588742733,
+            -0.014345075935125351,
+            -0.012592548504471779,
+            -0.011942830868065357,
+            -0.00874553807079792,
+            0.034332435578107834,
+            0.01454170048236847,
+            -0.01454170048236847,
+            -0.015234163030982018,
+            -0.011566678993403912,
+            0.03033154271543026,
+            0.00442833686247468,
+            -0.0004928982816636562,
+            0.021902315318584442,
+            0.018294673413038254,
+            0.017345745116472244,
+            0.016362618654966354,
+            0.009420901536941528,
+            0.0027912200894206762,
+            0.020466096699237823,
+            0.0019480835180729628,
+            0.015088831074535847,
+            -0.008172760717570782,
+            0.008642951026558876,
+            -0.005864554084837437,
+            0.012712233699858189,
+            -0.010677591897547245,
+            -0.011797499842941761,
+            0.008433502167463303,
+            -0.014003119431436062,
+            0.002117993077263236,
+            -0.007373437285423279,
+            0.00948074460029602,
+            -0.02017543464899063,
+            -0.007599983364343643,
+            0.012190749868750572,
+            -0.014533151872456074,
+            0.0024043817538768053,
+            -0.022893989458680153,
+            -0.0009761789697222412,
+            0.014695581048727036,
+            -0.0025368898641318083,
+            0.0066125839948654175,
+            0.005321698263287544,
+            0.00012549536768347025,
+            0.010104815475642681,
+            -0.014789619483053684,
+            0.004030812531709671,
+            -0.005377266090363264,
+            -0.004834410268813372,
+            -0.00637748884037137,
+            -0.0011712011182680726,
+            0.0009542724001221359,
+            -0.005556793417781591,
+            0.01991896703839302,
+            0.008091545663774014,
+            0.00435994565486908,
+            -0.005270404741168022,
+            -0.012831917963922024,
+            -0.026159673929214478,
+            0.0145758967846632,
+            -0.015447885729372501,
+            0.01528545655310154,
+            0.0013550026342272758,
+            -0.003092569299042225,
+            -0.014370722696185112,
+            0.010643396526575089,
+            0.0056337337009608746,
+            0.00474037230014801,
+            0.015114477835595608,
+            -0.011481190100312233,
+            0.009583331644535065,
+            0.009147336706519127,
+            -0.009267021901905537,
+            -0.0067493668757379055,
+            0.0005936151137575507,
+            0.006279176566749811,
+            -0.0232701413333416,
+            -0.0016862730262801051,
+            0.008292445912957191,
+            -0.01962830312550068,
+            0.008762635290622711,
+            -0.03730745613574982,
+            0.0020068571902811527,
+            -0.012088162824511528,
+            0.011258917860686779,
+            -0.002776259556412697,
+            -0.033374954015016556,
+            -0.026518728584051132,
+            0.0016745183384045959,
+            0.0040179891511797905,
+            -0.009993678890168667,
+            0.009010554291307926,
+            0.009275570511817932,
+            0.011532483622431755,
+            0.01443056482821703,
+            0.0098397983238101,
+            0.010113364085555077,
+            0.04223163053393364,
+            -0.04151352122426033,
+            -0.007146891206502914,
+            0.0070656766183674335,
+            0.02209039032459259,
+            -0.010267244651913643,
+            -0.011361504904925823,
+            0.008685695938766003,
+            -0.003859834047034383,
+            0.034349530935287476,
+            -0.024877335876226425,
+            0.014994793571531773,
+            -0.0040543219074606895,
+            -0.00010605995339574292,
+            -0.024159228429198265,
+            -0.0020325039513409138,
+            -0.01969669573009014,
+            0.0032357636373490095,
+            -0.002079523168504238,
+            -0.017191864550113678,
+            0.018021108582615852,
+            -0.004595040809363127,
+            0.012789173983037472,
+            0.007813706062734127,
+            -0.01625148393213749,
+            0.004526649136096239,
+            -0.015593216754496098,
+            0.032896216958761215,
+            -0.012105260975658894,
+            0.005911573302000761,
+            0.00873698852956295,
+            -0.006779287941753864,
+            0.016439558938145638,
+            0.009352510794997215,
+            -0.006044081412255764,
+            -0.0051592690870165825,
+            -0.002759161638095975,
+            -0.0023680487647652626,
+            0.008493345230817795,
+            -0.021474869921803474,
+            0.027493303641676903,
+            -0.006894698366522789,
+            -0.014037314802408218,
+            0.006266353186219931,
+            -0.001722605898976326,
+            0.002626653527840972,
+            0.004406964406371117,
+            -0.0018059578724205494,
+            -0.0007939802599139512,
+            -0.004958369303494692,
+            -0.017798837274312973,
+            -0.010130461305379868,
+            0.0010467075044289231,
+            -0.0016114701284095645,
+            0.013105483725667,
+            -0.010540809482336044,
+            -0.016601989045739174,
+            0.005112249869853258,
+            0.00026929075829684734,
+            -0.003975244704633951,
+            -0.016807163134217262,
+            0.007215282414108515,
+            -0.007544415537267923,
+            0.005372991785407066,
+            -0.008168485946953297,
+            0.016448108479380608,
+            0.008313817903399467,
+            -0.0003630616411101073,
+            0.01767915114760399,
+            0.024706358090043068,
+            0.02061997726559639,
+            0.01152393501251936,
+            -0.040214087814092636,
+            -0.0189443901181221,
+            -0.0025091059505939484,
+            0.018807608634233475,
+            -0.0008981701685115695,
+            -0.006420233752578497,
+            -0.0017632133094593883,
+            0.0029835705645382404,
+            -0.003084020223468542,
+            -0.0007474955637007952,
+            0.016918297857046127,
+            -0.020192531868815422,
+            0.0032742335461080074,
+            -0.012720782309770584,
+            0.014404918067157269,
+            0.0006860502762719989,
+            -0.0006737611838616431,
+            5.767176844528876e-05,
+            0.00945509783923626,
+            0.014900755137205124,
+            0.014661385677754879,
+            0.0014159136917442083,
+            0.019064076244831085,
+            -0.003663209266960621,
+            -0.0016510088462382555,
+            -0.018636630848050117,
+            -0.01696104370057583,
+            -0.0034494863357394934,
+            0.01298579853028059,
+            -0.0015601766062900424,
+            0.011575227603316307,
+            -0.014191195368766785,
+            0.01649085246026516,
+            0.025680935010313988,
+            -0.012840467505156994,
+            0.005069505423307419,
+            0.002564673777669668,
+            -0.001211808412335813,
+            0.0064800758846104145,
+            -0.007381986360996962,
+            -0.004030812531709671,
+            -0.012438667938113213,
+            0.01868792437016964,
+            -0.0032208028715103865,
+            0.018277576193213463,
+            0.013635516166687012,
+            0.011968477629125118,
+            -0.00032993461354635656,
+            0.008305269293487072,
+            0.018636630848050117,
+            0.014413466677069664,
+            -0.0005158734857104719,
+            0.0011615835828706622,
+            -0.01284901611506939,
+            0.0023594999220222235,
+            0.016311325132846832,
+            -0.02715134806931019,
+            -0.007685472723096609,
+            0.011840244755148888,
+            -0.016567792743444443,
+            0.0025603994727134705,
+            -0.02597159706056118,
+            -0.011472641490399837,
+            0.012053967453539371,
+            0.010233048349618912,
+            -0.011053744703531265,
+            0.001883966731838882,
+            0.001705508097074926,
+            -0.0008377934573218226,
+            -0.00936105940490961,
+            0.0167815163731575,
+            -0.005868828855454922,
+            -0.011258917860686779,
+            0.011686364188790321,
+            -0.00981415156275034,
+            0.012695135548710823,
+            0.004603589419275522,
+            0.011746206320822239,
+            -0.0024578124284744263,
+            0.026877781376242638,
+            0.01287466287612915,
+            0.007236654870212078,
+            0.0004386660875752568,
+            0.005937220063060522,
+            -0.004847233649343252,
+            0.007407633122056723,
+            0.026570022106170654,
+            -0.00048728802357800305,
+            -0.0116350706666708,
+            -0.0013197384541854262,
+            0.010951157659292221,
+            -0.01988477073609829,
+            0.0023124809376895428,
+            0.005676478147506714,
+            0.028211412951350212,
+            0.014327977783977985,
+            0.0025796345435082912,
+            -0.0204148031771183,
+            -0.00023709874949418008,
+            -0.00041595802758820355,
+            0.04106897860765457,
+            0.0026116929948329926,
+            -0.012532706372439861,
+            0.0021489830687642097,
+            0.013849238865077496,
+            -0.005590988788753748,
+            -0.004834410268813372,
+            -0.014310880564153194,
+            -0.008027428761124611,
+            -0.006018434651196003,
+            -0.0198676735162735,
+            0.001279131043702364,
+            0.005800437182188034,
+            -0.013319206424057484,
+            0.007471750024706125,
+            -0.006403135601431131,
+            -0.03031444549560547,
+            -0.005932945758104324,
+            -0.0034131535794585943,
+            -0.023731783032417297,
+            -0.007125518750399351,
+            0.0089678093791008,
+            0.005915847606956959,
+            -0.014122803695499897,
+            -0.025338977575302124,
+            0.020996129140257835,
+            0.007668375037610531,
+            0.03392208740115166,
+            -0.012669488787651062,
+            -0.007185361348092556,
+            -0.00948074460029602,
+            0.008557462133467197,
+            -0.023150455206632614,
+            -0.008715616539120674,
+            0.009061847813427448,
+            0.01767915114760399,
+            0.012404472567141056,
+            -0.011370054446160793,
+            0.0015163634670898318,
+            0.012122358195483685,
+            -0.00010085045505547896,
+            0.02764718420803547,
+            -0.009104591794312,
+            -0.014994793571531773,
+            0.010754532180726528,
+            -0.003259273013100028,
+            0.01916666328907013,
+            -0.0006203305092640221,
+            -0.0025924579240381718,
+            0.012917407788336277,
+            0.0038341875188052654,
+            -0.012934505008161068,
+            -0.005240483675152063,
+            0.008647225797176361,
+            -0.028553368523716927,
+            0.02792074903845787,
+            0.015909526497125626,
+            -0.02816011942923069,
+            -0.008305269293487072,
+            -0.0025091059505939484,
+            0.017140571027994156,
+            0.020089944824576378,
+            -0.011446994729340076,
+            -0.009728662669658661,
+            0.014960597269237041,
+            0.0034217024222016335,
+            0.0077538639307022095,
+            -0.0026565745938569307,
+            0.01896148920059204,
+            -0.004370631650090218,
+            -0.007800883147865534,
+            -0.003408879041671753,
+            -0.012789173983037472,
+            -0.0003505054337438196,
+            0.015618863515555859,
+            0.012977249920368195,
+            -0.005984238814562559,
+            -0.008771184831857681,
+            0.015208516269922256,
+            -0.022432347759604454,
+            0.0010841090697795153,
+            0.010908412747085094,
+            -0.01745687983930111,
+            -0.007561513222754002,
+            0.01767915114760399,
+            -0.0031395882833749056,
+            0.011258917860686779,
+            -0.007381986360996962,
+            0.007856450974941254,
+            0.0067237201146781445,
+            0.031135139986872673,
+            -0.007476024329662323,
+            0.03525571525096893,
+            -0.009309765882790089,
+            0.01991896703839302,
+            0.037991367280483246,
+            0.01455025002360344,
+            -0.0031118043698370457,
+            -0.008335189893841743,
+            0.009198630228638649,
+            -0.010096265934407711,
+            0.002415067981928587,
+            0.013524380512535572,
+            -0.026809390634298325,
+            -0.01418264675885439,
+            -0.011378603056073189,
+            0.013233717530965805,
+            0.0034580351784825325,
+            -0.013661162927746773,
+            -0.011575227603316307,
+            0.019508618861436844,
+            0.010352733545005322,
+            -0.0038705202750861645,
+            -0.007869274355471134,
+            -0.02157745696604252,
+            0.0020688369404524565,
+            0.012088162824511528,
+            -0.009164434857666492,
+            -0.011532483622431755,
+            0.016670379787683487,
+            -0.01261819526553154,
+            0.0027228286489844322,
+            -0.008300994522869587,
+            -0.027989141643047333,
+            -0.005800437182188034,
+            -0.00491989916190505,
+            -0.0019844165071845055,
+            -0.005565342493355274,
+            0.006471527274698019,
+            0.005227660294622183,
+            -0.01824337989091873,
+            0.0024278913624584675,
+            -0.008788282051682472,
+            0.00861302949488163,
+            0.00436635734513402,
+            0.006249255500733852,
+            -0.0019384659826755524,
+            0.014225390739738941,
+            0.013661162927746773,
+            -0.005377266090363264,
+            -0.020773857831954956,
+            0.012772075831890106,
+            -0.03634997829794884,
+            0.020722564309835434,
+            0.008694244548678398,
+            0.0025411644019186497,
+            -0.03086157515645027,
+            0.01018175482749939,
+            0.01115633174777031,
+            0.0160035640001297,
+            0.011370054446160793,
+            0.0016766554908826947,
+            0.013190972618758678,
+            -0.01114778220653534,
+            0.006766464561223984,
+            -7.192273642431246e-06,
+            -0.01482381485402584,
+            0.020688369870185852,
+            0.011814597994089127,
+            0.02913469448685646,
+            0.002951512113213539,
+            -0.0011209761723876,
+            0.001963044051080942,
+            0.00899345614016056,
+            -0.007685472723096609,
+            0.0030904319137334824,
+            0.004954094998538494,
+            0.0058175353333354,
+            0.013190972618758678,
+            0.007745314855128527,
+            -0.00860875565558672,
+            0.007839352823793888,
+            0.0026736725121736526,
+            -0.014020216651260853,
+            -0.000775279535446316,
+            -0.006997285410761833,
+            0.007552964612841606,
+            -0.01505463570356369,
+            0.004560844972729683,
+            0.04342847689986229,
+            0.012413021177053452,
+            -0.0034323884174227715,
+            -0.017354292795062065,
+            -0.011079391464591026,
+            -0.01273788046091795,
+            -0.01272933091968298,
+            -0.0053644427098333836,
+            -0.02303077094256878,
+            -0.003434525802731514,
+            0.0065912120044231415,
+            0.02453538030385971,
+            0.01092551089823246,
+            -0.017191864550113678,
+            -0.005813260562717915,
+            -0.02380017377436161,
+            -0.010566456243395805,
+            -0.016191640868782997,
+            -0.016319874674081802,
+            -0.008061625063419342,
+            0.018978586420416832,
+            -0.008839575573801994,
+            0.01408860832452774,
+            0.002968610031530261,
+            -0.0002393695613136515,
+            -0.0020431901793926954,
+            -0.000852754048537463,
+            -0.0051549943163990974,
+            0.0015911663649603724,
+            0.003152411663904786,
+            -0.0005375128821469843,
+            0.020722564309835434,
+            -0.007800883147865534,
+            0.003866245737299323,
+            -0.0026929075829684734,
+            0.01542223896831274,
+            0.007108421064913273,
+            0.0028681603725999594,
+            -0.0238172709941864,
+            0.02304787002503872,
+            -0.024381499737501144,
+            -0.014156999997794628,
+            0.02284269593656063,
+            -0.008108643814921379,
+            0.004804489202797413,
+            0.024107934907078743,
+            0.00029493749025277793,
+            -0.0009697672794573009,
+            0.015123026445508003,
+            0.004045772831887007,
+            0.011258917860686779,
+            0.015242711640894413,
+            0.007082774303853512,
+            -0.0052875024266541,
+            0.019098270684480667,
+            0.0073306928388774395,
+            -0.006403135601431131,
+            -0.01626003161072731,
+            0.0014244626509025693,
+            0.004238123539835215,
+            -0.010044972412288189,
+            -0.004137673880904913,
+            0.008630127646028996,
+            0.004453983623534441,
+            -0.007843627594411373,
+            -0.0012972975382581353,
+            -0.024706358090043068,
+            -0.005821809638291597,
+            -0.01530255377292633,
+            -0.006039807107299566,
+            -0.023406922817230225,
+            0.005676478147506714,
+            -0.009053298272192478,
+            0.01672167330980301,
+            0.00016403225890826434,
+            0.024347303435206413,
+            0.0020154062658548355,
+            -0.0007266575703397393,
+            -0.0051592690870165825,
+            0.009318315424025059,
+            -0.0007266575703397393,
+            0.027236836031079292,
+            -0.00011026761785615236,
+            -0.005911573302000761,
+            0.01032708678394556,
+            -0.01769625023007393,
+            0.020876444876194,
+            0.006356116849929094,
+            0.01454170048236847,
+            -0.015610314905643463,
+            -0.0073563395999372005,
+            0.035289913415908813,
+            0.024843141436576843,
+            0.0035029170103371143,
+            0.019816379994153976,
+            -0.010455320589244366,
+            0.016311325132846832,
+            0.003013491863384843,
+            -0.002389421220868826,
+            0.011010999791324139,
+            -0.005420010536909103,
+            -0.014627190306782722,
+            -0.006997285410761833,
+            0.03911982476711273,
+            -0.014832363463938236,
+            -6.972707342356443e-05,
+            -0.005060956347733736,
+            -0.010104815475642681,
+            -0.030929965898394585,
+            0.011609423905611038,
+            0.011951380409300327,
+            -0.024706358090043068,
+            -0.011566678993403912,
+            0.0032699592411518097,
+            -0.00026448199059814215,
+            -0.0030348640866577625,
+            -0.005706399213522673,
+            0.006509996950626373,
+            -0.0017375665483996272,
+            0.009275570511817932,
+            0.005569616798311472,
+            -0.0009617527248337865,
+            0.00013798479631077498,
+            0.005539695732295513,
+            0.011780401691794395,
+            0.006206510588526726,
+            0.0005498019745573401,
+            -0.007381986360996962,
+            0.00558671448379755,
+            0.010745983570814133,
+            -0.015088831074535847,
+            0.020739663392305374,
+            0.013823592104017735,
+            -0.016576342284679413,
+            0.0038149524480104446,
+            -0.019132466986775398,
+            -0.0329304113984108,
+            0.004787391051650047,
+            -0.0009029789362102747,
+            0.021748434752225876,
+            -0.018311770632863045,
+            -0.00995948351919651,
+            0.00442833686247468,
+            0.003908990416675806,
+            -0.011421347968280315,
+            0.019371837377548218,
+            0.007215282414108515,
+            -0.0017664191545918584,
+            0.01745687983930111,
+            0.016593439504504204,
+            -0.006403135601431131,
+            -0.00907894503325224,
+            0.018824705854058266,
+            -0.00437276903539896,
+            0.02744201011955738,
+            0.026621313765645027,
+            0.009207178838551044,
+            -0.018602434545755386,
+            0.0018262615194544196,
+            0.0080231549218297,
+            -0.02036350965499878,
+            -0.005590988788753748,
+            0.015670157968997955,
+            0.005402912851423025,
+            -0.0003841667785309255,
+            -0.0050737797282636166,
+            -0.022175880149006844,
+            0.007407633122056723,
+            0.011840244755148888,
+            -0.005189190153032541,
+            -0.028809836134314537,
+            -0.0011199075961485505,
+            -0.01431942917406559,
+            -0.0007694021333009005,
+            0.0010648739989846945,
+            0.01236172765493393,
+            0.009036201052367687,
+            0.028074629604816437,
+            -0.009865445084869862,
+            -0.00885667372494936,
+            0.010173206217586994,
+            0.009797054342925549,
+            0.012583999894559383,
+            0.012327532283961773,
+            0.013986021280288696,
+            0.007292222697287798,
+            -0.015960820019245148,
+            -0.004966918379068375,
+            0.007527317851781845,
+            -0.00899345614016056,
+            0.003985930699855089,
+            -0.01358422264456749,
+            0.020278021693229675,
+            0.015259809792041779,
+            0.0008260387112386525,
+            0.004804489202797413,
+            -0.004753195680677891,
+            -0.0012684449320659041,
+            -0.06483495235443115,
+            -0.013532929122447968,
+            0.00861302949488163,
+            0.021252596750855446,
+            -0.016396814957261086,
+            0.00046244275290519,
+            0.0014287370722740889,
+            0.008788282051682472,
+            -0.011481190100312233,
+            0.01020740158855915,
+            0.014524603262543678,
+            -0.005223385989665985,
+            -0.0035349754616618156,
+            0.005099426489323378,
+            -0.0016403226181864738,
+            -0.00013371034583542496,
+            0.004958369303494692,
+            0.014900755137205124,
+            0.021406477317214012,
+            -0.004385592415928841,
+            0.018585337325930595,
+            0.004112027119845152,
+            -0.015080282464623451,
+            -0.0027014564257115126,
+            0.013575674034655094,
+            -0.02328723855316639,
+            -0.004607864189893007,
+            0.007480298634618521,
+            -0.006980187259614468,
+            0.010908412747085094,
+            -0.004069282673299313,
+            -0.004406964406371117,
+            0.008677146397531033,
+            0.01420829351991415,
+            0.002276147948578,
+            -0.026775194332003593,
+            0.0034665842540562153,
+            -0.011233271099627018,
+            0.0035093287006020546,
+            0.01020740158855915,
+            -0.008219779469072819,
+            1.5344630810432136e-05,
+            -0.015328200533986092,
+            -0.002739926567301154,
+            0.024860238656401634,
+            -0.004289417061954737,
+            0.0067707388661801815,
+            0.017302999272942543,
+            -0.007018657401204109,
+            0.002389421220868826,
+            -0.02862176112830639,
+            0.007796608377248049,
+            -0.0369655005633831,
+            -0.0004579011583700776,
+            0.004984016064554453,
+            0.004097066354006529,
+            -0.010104815475642681,
+            -0.030211858451366425,
+            0.008412130177021027,
+            -0.00474037230014801,
+            -0.006753641180694103,
+            -0.022723009809851646,
+            -0.007937666028738022,
+            0.008429228328168392,
+            -0.006941717118024826,
+            0.010147559456527233,
+            -0.017662053927779198,
+            0.017815934494137764,
+            0.015439336188137531,
+            0.013302108272910118,
+            0.003968833014369011,
+            0.0032379007898271084,
+            0.005129347555339336,
+            0.007946214638650417,
+            0.004701902158558369,
+            0.004697627853602171,
+            -0.01019030436873436,
+            -0.011464091949164867,
+            0.01575564593076706,
+            0.004255221225321293,
+            0.00860875565558672,
+            -0.0018604571232572198,
+            0.002479184651747346,
+            -0.0073050460778176785,
+            -0.00045629823580384254,
+            -0.014396369457244873,
+            0.0021596690639853477,
+            0.01199412439018488,
+            0.015499179251492023,
+            0.0014650699449703097,
+            -0.00625352980569005,
+            -0.007634179200977087,
+            -0.00667670089751482,
+            0.020209630951285362,
+            -0.017149118706583977,
+            -0.006698073353618383,
+            -0.012635293416678905,
+            0.021526163443922997,
+            0.01599501632153988,
+            0.0035563476849347353,
+            -0.006826307158917189,
+            -0.004092792049050331,
+            0.0097030159085989,
+            0.008570285513997078,
+            0.006766464561223984,
+            -0.023663390427827835,
+            -0.005377266090363264,
+            -0.019081173464655876,
+            -0.01539659220725298,
+            -0.01406296156346798,
+            0.021218402311205864,
+            -0.004924173932522535,
+            -0.007471750024706125,
+            0.008454875089228153,
+            0.024655064567923546,
+            0.007916293106973171,
+            0.007715393789112568,
+            0.00503958435729146,
+            -0.01726025529205799,
+            0.012071064673364162,
+            -0.010139010846614838,
+            -0.03665773943066597,
+            0.0065912120044231415,
+            -0.00971156544983387,
+            0.02598869428038597,
+            -0.017183315008878708,
+            -0.0011167017510160804,
+            -0.002635202370584011,
+            -0.030878672376275063,
+            -0.005638008005917072,
+            0.02961343340575695,
+            -0.010737434960901737,
+            0.011669266037642956,
+            -0.010062070563435555,
+            -0.009950934909284115,
+            0.01019030436873436,
+            0.013062738813459873,
+            -0.011720559559762478,
+            -0.002940826117992401,
+            0.014721227809786797,
+            0.02453538030385971,
+            0.0035093287006020546,
+            -0.004548021592199802,
+            0.009429451078176498,
+            0.0123702771961689,
+            -0.006804934702813625,
+            0.019559912383556366,
+            -0.007561513222754002,
+            -0.0015078145079314709,
+            -0.00656556524336338,
+            0.0012235631002113223,
+            0.006193687207996845,
+            -0.019577009603381157,
+            0.017320098355412483,
+            -0.009916738606989384,
+            0.008873771876096725,
+            0.0028702975250780582,
+            0.01726025529205799,
+            0.011267467401921749,
+            0.01896148920059204,
+            -0.02891242317855358,
+            -0.001095329411327839,
+            -0.0276813805103302,
+            -0.015772745013237,
+            -0.010703238658607006,
+            -0.00873698852956295,
+            -0.002485596342012286,
+            -0.007796608377248049,
+            -0.023868564516305923,
+            -0.005625184625387192,
+            0.0007939802599139512,
+            0.0036781697999686003,
+            -0.017362842336297035,
+            0.01007061917334795,
+            1.5286190318875015e-05,
+            -0.016593439504504204,
+            -0.020825151354074478,
+            -0.002087003318592906,
+            -0.022021999582648277,
+            -0.012464314699172974,
+            -0.005005388520658016,
+            -0.0022248546592891216,
+            -0.0012139456812292337,
+            -0.0005172092351131141,
+            0.027544597163796425,
+            -0.019457325339317322,
+            0.004381317645311356,
+            0.02744201011955738,
+            -0.0019352601375430822,
+            0.005035309586673975,
+            -0.006668151821941137,
+            0.012532706372439861,
+            0.0032913314644247293,
+            -0.003973107319325209,
+            -0.022688815370202065,
+            0.02521929331123829,
+            0.0005311011918820441,
+            0.013669711537659168,
+            0.010139010846614838,
+            -0.008065898902714252,
+            -0.005783339496701956,
+            0.00485150795429945,
+            -0.03253716230392456,
+            0.0028531996067613363,
+            -0.013011445291340351,
+            -0.01865372806787491,
+            -0.005539695732295513,
+            -0.004586491733789444,
+            -0.006561290472745895,
+            -0.0071939099580049515,
+            0.017285902053117752,
+            -0.004377043340355158,
+            0.02376597747206688,
+            -0.007146891206502914,
+            -0.00453947251662612,
+            0.010908412747085094,
+            -0.04301812872290611,
+            -0.007240929175168276,
+            -0.006129570305347443,
+            -0.01843145675957203,
+            0.019799282774329185,
+            0.012173651717603207,
+            -0.001380649395287037,
+            -0.020243825390934944,
+            0.019816379994153976,
+            -0.0030028056353330612,
+            0.02552705444395542,
+            0.010634847916662693,
+            0.010173206217586994,
+            0.004364219959825277,
+            -0.010840021073818207,
+            -0.010113364085555077,
+            0.018311770632863045,
+            0.006304823327809572,
+            -0.011609423905611038,
+            0.01873921789228916,
+            0.004445434547960758,
+            -0.00186686881352216,
+            -0.003041275776922703,
+            0.018756315112113953,
+            0.024603771045804024,
+            0.013002896681427956,
+            0.002274010796099901,
+            -0.016174543648958206,
+            -0.006475801579654217,
+            0.0037038165610283613,
+            0.036418367177248,
+            0.012036869302392006,
+            0.012994347140192986,
+            -0.008202682249248028,
+            0.027373619377613068,
+            -0.009181532077491283,
+            0.009044749662280083,
+            0.00863440241664648,
+            0.017174765467643738,
+            0.0035328383091837168,
+            0.020329315215349197,
+            0.010857119224965572,
+            -0.00764272827655077,
+            0.02405664138495922,
+            0.007625630125403404,
+            0.003767933463677764,
+            0.006625407375395298,
+            -0.01844855397939682,
+            0.012626743875443935,
+            -0.011609423905611038,
+            -0.005950043443590403,
+            0.004505277145653963,
+            0.002058150712400675,
+            0.02374888025224209,
+            0.008027428761124611,
+            -0.03135741129517555,
+            -0.0017867228016257286,
+            -0.023612096905708313,
+            0.004340710584074259,
+            -0.0167387705296278,
+            0.012447217479348183,
+            0.004092792049050331,
+            -0.01774754375219345,
+            -0.018790511414408684,
+            -0.010728885419666767,
+            -0.01942313089966774,
+            -0.004924173932522535,
+            -0.012173651717603207,
+            -0.010840021073818207,
+            0.006954540498554707,
+            -0.0025069687981158495,
+            0.0024813220370560884,
+            0.006963089574128389,
+            0.019046977162361145,
+            0.020260922610759735,
+            0.0035819944459944963,
+            0.018824705854058266,
+            -0.0009019103017635643,
+            -0.055431149899959564,
+            0.0036589347291737795,
+            0.009309765882790089,
+            -0.0002863885893020779,
+            0.026108380407094955,
+            -0.004041498526930809,
+            0.00042557556298561394,
+            0.007223831489682198,
+            0.006937442813068628,
+            0.013276461511850357,
+            -0.004210339393466711,
+            -0.025886109098792076,
+            -0.019337641075253487,
+            0.0040393611416220665,
+            -0.006155217066407204,
+            0.008100095205008984,
+            -0.006685249973088503,
+            0.004761744290590286,
+            0.002353088231757283,
+            -0.009745760820806026,
+            -0.006595486309379339,
+            0.009617527015507221,
+            0.0014469034504145384,
+            -0.010386928915977478,
+            -0.013173874467611313,
+            -0.0018690060824155807,
+            0.01824337989091873,
+            -0.009019102901220322,
+            -0.024364402517676353,
+            0.009660271927714348,
+            -0.010010777041316032,
+            -0.0014821677468717098,
+            0.025869010016322136,
+            0.02429600991308689,
+            -0.003970969934016466,
+            0.019371837377548218,
+            -0.0021190617699176073,
+            -0.016405364498496056,
+            -0.03029734641313553,
+            -0.006873325910419226,
+            -0.028997913002967834,
+            0.027253935113549232,
+            -0.006424508057534695,
+            -0.0013176011852920055,
+            0.0035285637713968754,
+            -0.00413553649559617,
+            0.003434525802731514,
+            -0.009788504801690578,
+            -0.007672649342566729,
+            -0.02716844528913498,
+            0.007527317851781845,
+            -0.0069075217470526695,
+            -0.008172760717570782,
+            0.00031550831045024097,
+            0.039701152592897415,
+            -0.004556570667773485,
+            0.00850189384073019,
+            0.006232157349586487,
+            0.011865891516208649,
+            0.024193422868847847,
+            -0.008463423699140549,
+            -0.012096711434423923,
+            -0.015046087093651295,
+            -0.0020154062658548355,
+            0.00014052275219000876,
+            -0.003975244704633951,
+            0.022876890376210213,
+            -0.013019993901252747,
+            -0.015191418118774891,
+            -0.00034489520476199687,
+            -0.023389825597405434,
+            -0.011737657710909843,
+            0.009968032129108906,
+            0.012908858247101307,
+            0.007535866927355528,
+            -0.014003119431436062,
+            -0.014268135651946068,
+            -0.022945282980799675,
+            0.01800401136279106,
+            0.0976627841591835,
+            0.0012545529752969742,
+            -0.009070396423339844,
+            0.0026758096646517515,
+            0.010771630331873894,
+            -0.0008372591692022979,
+            -0.017559466883540154,
+            0.05919267237186432,
+            -0.013567124493420124,
+            0.005856005474925041,
+            -0.009010554291307926,
+            0.01007916871458292,
+            0.005031035281717777,
+            -0.00013257493264973164,
+            0.005928670987486839,
+            -0.008796831592917442,
+            0.006129570305347443,
+            -0.03645256534218788,
+            -0.00186686881352216,
+            -0.011344407685101032,
+            -0.006381763610988855,
+            -0.016824260354042053,
+            0.009275570511817932,
+            -0.02911759726703167,
+            -0.00922427698969841,
+            0.0020880720112472773,
+            -0.0059543177485466,
+            0.012336080893874168,
+            -0.01628567837178707,
+            -0.0017995461821556091,
+            0.005368717480450869,
+            0.005120798945426941,
+            -0.0036867186427116394,
+            -0.0074632009491324425,
+            -0.013310656882822514,
+            0.005065230652689934,
+            0.015576119534671307,
+            0.005548244342207909,
+            -0.008228329010307789,
+            0.00959188025444746,
+            0.016388265416026115,
+            0.0009510665549896657,
+            0.013661162927746773,
+            0.009412352927029133,
+            0.0007950488943606615,
+            0.0007875685696490109,
+            0.019611205905675888,
+            -0.016345521435141563,
+            0.02137228287756443,
+            -0.015883879736065865,
+            -0.02843368425965309,
+            -0.006236432120203972,
+            0.006411684677004814,
+            -0.014413466677069664,
+            0.0010910549899563193,
+            0.006582662928849459,
+            0.04100058600306511,
+            0.00863440241664648,
+            -0.0034281141124665737,
+            -0.0009259541402570903,
+            -0.013669711537659168,
+            0.023954054340720177,
+            0.01091696135699749,
+            0.030502520501613617,
+            0.007604258134961128,
+            0.00373373762704432,
+            -0.007001559715718031,
+            -0.009557684883475304,
+            -0.003487956477329135,
+            0.00474037230014801,
+            0.0025069687981158495,
+            0.004415513481944799,
+            0.008523265831172466,
+            -0.014498956501483917,
+            -0.02036350965499878,
+            0.017610760405659676,
+            -0.00474037230014801,
+            -0.023663390427827835,
+            0.012609646655619144,
+            -0.0316309779882431,
+            0.0047831167466938496,
+            0.0010344184702262282,
+            -0.005877377465367317,
+            -0.013960374519228935,
+            0.006300548557192087,
+            0.01151538547128439,
+            -0.009378157556056976,
+            -0.015482081100344658,
+            -0.008454875089228153,
+            -0.010592103004455566,
+            0.0001227570464834571,
+            0.0037657960783690214,
+            -0.011583777144551277,
+            -0.00394318625330925,
+            -0.01796981506049633,
+            0.007142616901546717,
+            -0.003013491863384843,
+            0.004654882941395044,
+            -0.011122135445475578,
+            0.011558130383491516,
+            0.020226728171110153,
+            -0.009788504801690578,
+            -0.007069950923323631,
+            0.013105483725667,
+            0.003400330198928714,
+            0.016089053824543953,
+            0.03368271514773369,
+            -0.0033704089000821114,
+            0.0033511738292872906,
+            0.003849148051813245,
+            0.0058731031604111195,
+            0.0008874839986674488,
+            0.015909526497125626,
+            0.03621319308876991,
+            0.01942313089966774,
+            -0.0250141192227602,
+            0.02186811901628971,
+            -0.019713792949914932,
+            0.021936509758234024,
+            -0.02231266349554062,
+            -0.0030797459185123444,
+            0.007005834020674229,
+            -0.007394809741526842,
+            0.004146222956478596,
+            0.00624070642516017,
+            0.017627857625484467,
+            0.022363955155014992,
+            0.000617124664131552,
+            -0.0003163097717333585,
+            0.004860057029873133,
+            0.021953608840703964,
+            -0.0025411644019186497,
+            0.007125518750399351,
+            -0.013421793468296528,
+            0.004411239176988602,
+            -0.007202459033578634,
+            0.0021767669823020697,
+            0.008395032025873661,
+            -0.014952048659324646,
+            0.013344853185117245,
+            0.012891761027276516,
+            -0.017302999272942543,
+            0.0043342988938093185,
+            0.0025945950765162706,
+            -0.018568238243460655,
+            -0.018978586420416832,
+            0.023372728377580643,
+            -0.013635516166687012,
+            -0.012660940177738667,
+            -0.009891091845929623,
+            -0.01152393501251936,
+            -0.0201412383466959,
+            -0.005056682042777538,
+            -0.0017471840837970376,
+            0.005753418430685997,
+            -0.0024449890479445457,
+            0.006467252504080534,
+            0.01573854871094227,
+            0.005612361244857311,
+            -0.006437331438064575,
+            -0.005561067722737789,
+            -0.0015163634670898318,
+            0.020107043907046318,
+            0.005013937596231699,
+            0.0026458885986357927,
+            -0.04031667485833168,
+            0.0013143953401595354,
+            -0.010540809482336044,
+            0.006428782362490892,
+            0.0031417254358530045,
+            -0.004971192684024572,
+            -0.01942313089966774,
+            -0.014755423180758953,
+            0.01872211880981922,
+            -0.0103014400228858,
+            0.024637967348098755,
+            4.564986011246219e-05,
+            -0.012053967453539371,
+            -0.0018999958410859108,
+            0.003960283938795328,
+            0.0016029211692512035,
+            0.004415513481944799,
+            -0.00800178200006485,
+            0.016174543648958206,
+            -0.011694912798702717,
+            0.010908412747085094,
+            -0.00401157746091485,
+            -0.024706358090043068,
+            0.010891314595937729,
+            -0.01744833029806614,
+            0.0004752661334350705,
+            -0.00666387751698494,
+            -0.030040880665183067,
+            0.018551141023635864,
+            -0.023731783032417297,
+            -0.0033233899157494307,
+            0.009643173776566982,
+            0.018841803073883057,
+            0.0017461155075579882,
+            -0.010976804420351982,
+            0.00630909763276577,
+            0.017362842336297035,
+            0.013481635600328445,
+            -0.004062870983034372,
+            0.012438667938113213,
+            -0.004971192684024572,
+            0.0025603994727134705,
+            0.005172092467546463,
+            0.008399306796491146,
+            -0.006804934702813625,
+            0.007805157452821732
+          ],
+          "keyphrases": [
+            "DeepSeekMoE",
+            "multilingual corpus",
+            "HuggingFace Tokenizer",
+            "MoE architectures",
+            "evaluation benchmarks"
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": true,
+      "properties": {
+        "cosine_similarity": 0.8001567173517833
+      }
+    },
+    {
+      "id": "92df043f-4148-431b-a766-65e92886cc01",
+      "type": "cosine_similarity",
+      "source": {
+        "id": "4ef9096a-b51d-4ed6-b15e-5f4caff268e6",
+        "properties": {
+          "page_content": "1. Introduction\n\nRecent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023a). It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017) has emerged as a popular solution. It can\n\nenable parameter scaling, while concurrently keeping computational costs at a modest level.\n\nRecent applications of MoE architectures in Transformers (Vaswani et al., 2017) have yielded successful attempts at scaling language models to a substantial size (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022), accompanied with remarkable performance. These achievements underscore the considerable potential and promise of MoE language models.\n\nDespite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts.\n\nThis architecture manifests two potential issues: (1) Knowledge Hybridity: existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. (2) Knowledge Redundancy: tokens assigned to different experts may require common knowledge. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters.\n\nThese issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models.\n\nIn response to the aforementioned issues, we introduce DeepSeekMoE, an innovative MoE architecture specifically designed towards ultimate expert specialization. Our architecture involves two principal strategies: (1) Fine-Grained Expert Segmentation: while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. (2) Shared Expert Isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.\n\nThrough compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized.\n\nStarting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B (Lepikhin et al., 2021) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5× expert parameters and computation. Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE.\n\nThese studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization.\n\nLeveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B (DeepSeek-AI, 2024), a dense model trained on the same 2T corpus. We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B (Touvron et al., 2023b), which has approximately 2.5 times the activated parameters. Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard1. Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. The experimental results still validate its substantial advantages over the GShard architecture consistently. In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nOur contributions are summarized as follows: - Architectural Innovation. We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation.\n\nEmpirical Validation. We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. Experimental results validate the high 1https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models\n\nScalability. We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B.\n\nAlignment for MoE. We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B.\n\nPublic Release. In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization.\n\n",
+          "embedding": [
+            -0.007097613997757435,
+            -0.024122508242726326,
+            -0.01228793989866972,
+            -0.023231398314237595,
+            0.03164222836494446,
+            -0.012272306717932224,
+            -0.00579612422734499,
+            0.04649406298995018,
+            -0.012303573079407215,
+            0.0454309843480587,
+            -0.02185564860701561,
+            -0.010310300625860691,
+            0.03736409172415733,
+            -0.01593836396932602,
+            0.0027690858114510775,
+            0.013272850774228573,
+            0.008512446656823158,
+            0.021105241030454636,
+            0.017853470519185066,
+            -0.029391001909971237,
+            0.004928464535623789,
+            -0.04117866978049278,
+            -0.02241845615208149,
+            -0.022606058046221733,
+            0.011209228076040745,
+            0.016493353992700577,
+            0.025873461738228798,
+            0.010279033333063126,
+            -0.04374256357550621,
+            0.036988887935876846,
+            0.026092329993844032,
+            0.026905274018645287,
+            0.012608426623046398,
+            0.006147878244519234,
+            -0.01596963219344616,
+            -0.022027617320418358,
+            0.03842717036604881,
+            0.020776936784386635,
+            -0.013640238903462887,
+            0.023888004943728447,
+            0.03273657336831093,
+            0.01783783733844757,
+            -0.028609326109290123,
+            -0.008363928645849228,
+            0.020730035379529,
+            0.000511997495777905,
+            -0.039646584540605545,
+            0.003970911726355553,
+            0.006726318504661322,
+            -0.012373924255371094,
+            -0.017962904646992683,
+            0.0026518343947827816,
+            -0.0023176681715995073,
+            0.005772673990577459,
+            -0.04880782216787338,
+            0.008121608756482601,
+            0.03786436468362808,
+            0.007570527959614992,
+            0.06381599605083466,
+            -0.00819195993244648,
+            0.02632683329284191,
+            -0.028968896716833115,
+            -0.011349929496645927,
+            0.005518629681318998,
+            0.012342656962573528,
+            -0.03111068718135357,
+            -0.009755311533808708,
+            0.004252315033227205,
+            -0.001601457828655839,
+            0.008856384083628654,
+            -0.03999052196741104,
+            0.02268422581255436,
+            -0.021433543413877487,
+            0.01341355312615633,
+            0.022637324407696724,
+            0.0019063112558797002,
+            0.03542553633451462,
+            0.012115971185266972,
+            0.00941919069737196,
+            -0.028968896716833115,
+            0.010451002046465874,
+            0.03836463764309883,
+            0.0045532602816820145,
+            0.023387733846902847,
+            0.046181391924619675,
+            -0.033017974346876144,
+            -0.01883838139474392,
+            -0.07066347450017929,
+            0.019416820257902145,
+            -0.0036641042679548264,
+            -0.036301013082265854,
+            -0.0024915907997637987,
+            0.014390647411346436,
+            -0.017790935933589935,
+            -0.015133239328861237,
+            0.030250845476984978,
+            -0.009114337153732777,
+            -0.009606792591512203,
+            -0.04149134084582329,
+            0.0034120138734579086,
+            0.0041936892084777355,
+            0.012100338004529476,
+            0.0006917828577570617,
+            -0.01863514631986618,
+            -0.017353197559714317,
+            0.007664328906685114,
+            -0.01122486125677824,
+            0.027843283489346504,
+            -0.009442640468478203,
+            0.07210175693035126,
+            0.011662599630653858,
+            -0.013014898635447025,
+            -0.00774249667301774,
+            0.01697799377143383,
+            -0.01190491858869791,
+            -0.024028707295656204,
+            -0.042460616677999496,
+            -0.03342444822192192,
+            0.007097613997757435,
+            -0.03351824730634689,
+            0.0031052061822265387,
+            -0.007050713524222374,
+            0.022011984139680862,
+            -0.02262169122695923,
+            -0.07103867828845978,
+            0.041272468864917755,
+            -0.008317028172314167,
+            -0.01979202590882778,
+            -0.002986000617966056,
+            -0.007121064234524965,
+            0.0018261895747855306,
+            0.004967548418790102,
+            -0.03423738852143288,
+            -0.027171043679118156,
+            0.033737119287252426,
+            0.028562424704432487,
+            0.04386763274669647,
+            0.06197123974561691,
+            -0.014328112825751305,
+            0.018463177606463432,
+            0.016243217512965202,
+            -0.01963569037616253,
+            0.007547077722847462,
+            -0.013358835130929947,
+            -0.029656771570444107,
+            -0.01833810843527317,
+            0.01499253697693348,
+            0.06312812119722366,
+            -0.00027554063126444817,
+            0.005131700076162815,
+            -0.03043844737112522,
+            0.05531136319041252,
+            -0.02162114717066288,
+            0.02626429870724678,
+            -0.001939532463438809,
+            0.002087073866277933,
+            -0.018291208893060684,
+            0.014077977277338505,
+            -0.019745124503970146,
+            0.003943553194403648,
+            0.01255370955914259,
+            -0.0338309183716774,
+            0.00569450668990612,
+            0.01824430748820305,
+            0.004725228995084763,
+            0.03333064541220665,
+            0.006179145537316799,
+            0.0007611566106788814,
+            -0.06187743693590164,
+            0.02789018489420414,
+            0.007937915623188019,
+            0.02185564860701561,
+            -0.011428097262978554,
+            0.008035624399781227,
+            0.0017568158218637109,
+            0.0012887875782325864,
+            0.02079256996512413,
+            0.0060306270606815815,
+            -0.042304281145334244,
+            0.016868559643626213,
+            -0.028703127056360245,
+            0.024951083585619926,
+            0.027937084436416626,
+            0.01647772081196308,
+            0.029172131791710854,
+            0.01813487336039543,
+            0.04702560231089592,
+            0.028796928003430367,
+            0.017259396612644196,
+            0.01966695673763752,
+            0.01883838139474392,
+            0.007789397146552801,
+            0.03207996487617493,
+            0.011084159836173058,
+            0.010638604871928692,
+            -0.026076696813106537,
+            -0.0023919271770864725,
+            0.025123052299022675,
+            0.022371554747223854,
+            0.00022302180877886713,
+            -0.0031208398286253214,
+            -0.027405545115470886,
+            0.0038243476301431656,
+            -0.006437098141759634,
+            0.011725133284926414,
+            -0.0005163944442756474,
+            -0.003394426079466939,
+            0.02990690805017948,
+            0.02507615275681019,
+            -0.02055806666612625,
+            -0.013718406669795513,
+            -0.013202500529587269,
+            -0.03495652973651886,
+            -0.05943860858678818,
+            0.00731257488951087,
+            0.005628064274787903,
+            -0.002409514971077442,
+            0.012209772132337093,
+            0.024497712031006813,
+            -0.03864603862166405,
+            -0.006269038189202547,
+            -0.005651514511555433,
+            0.015563160181045532,
+            -0.02762441523373127,
+            -0.0013708635233342648,
+            -0.0062494962476193905,
+            0.012584976851940155,
+            0.0018232582369819283,
+            0.0441177673637867,
+            -0.0175720676779747,
+            -0.0032107324805110693,
+            0.018682045862078667,
+            0.00900490302592516,
+            0.02573275938630104,
+            0.03989672288298607,
+            -0.02431011013686657,
+            0.012209772132337093,
+            -0.0007171873585321009,
+            0.011154510080814362,
+            0.017368830740451813,
+            0.00593682611361146,
+            -0.002163287252187729,
+            -0.030500980094075203,
+            -0.018181772902607918,
+            0.035269200801849365,
+            0.02026103064417839,
+            0.00014326647215057164,
+            0.04396143555641174,
+            0.0065035405568778515,
+            0.021496077999472618,
+            0.013390102423727512,
+            -0.016165051609277725,
+            -0.008066891692578793,
+            0.006788852158933878,
+            -0.02623303234577179,
+            0.04271075502038002,
+            0.034831464290618896,
+            -0.0005466843722388148,
+            -0.010341567918658257,
+            -0.03958405181765556,
+            0.019682591781020164,
+            -0.028265388682484627,
+            -0.016071248799562454,
+            0.04943316429853439,
+            -0.01714996248483658,
+            0.02401307411491871,
+            0.04552478343248367,
+            0.01860387809574604,
+            -0.009575525298714638,
+            -0.0058977422304451466,
+            0.0014216724084690213,
+            -0.011396829970180988,
+            -0.02925029955804348,
+            0.013546437956392765,
+            0.003519494319334626,
+            -0.03176729381084442,
+            0.024169407784938812,
+            -0.0020089060999453068,
+            0.01830684207379818,
+            0.01532084122300148,
+            0.02158987894654274,
+            -0.009817845188081264,
+            0.02543572336435318,
+            0.010239950381219387,
+            -0.019651323556900024,
+            -0.0199014600366354,
+            -0.008270127698779106,
+            -0.04264821857213974,
+            -0.01573512889444828,
+            -0.013640238903462887,
+            -0.012413008138537407,
+            0.012037803418934345,
+            -0.05046497657895088,
+            0.004490726161748171,
+            0.00925503857433796,
+            -0.003300625132396817,
+            -0.017165595665574074,
+            0.020089061930775642,
+            0.004693961702287197,
+            -0.0008246677462011576,
+            -0.06691142916679382,
+            0.008473362773656845,
+            0.011529714800417423,
+            0.001870158826932311,
+            -0.02576402761042118,
+            0.023622235283255577,
+            -0.012209772132337093,
+            0.047901079058647156,
+            -0.01425776258111,
+            -0.03664495050907135,
+            0.0108574740588665,
+            -0.014296846464276314,
+            -0.0075079938396811485,
+            -0.007574436254799366,
+            0.0036054786760360003,
+            0.058719467371702194,
+            0.04042826220393181,
+            0.013007081113755703,
+            -0.03176729381084442,
+            -0.020683135837316513,
+            0.022699858993291855,
+            0.0027397728990763426,
+            -0.04705686867237091,
+            0.019057249650359154,
+            0.015461542643606663,
+            -0.010826206766068935,
+            0.024450812488794327,
+            0.04302342236042023,
+            -0.023794203996658325,
+            0.017024893313646317,
+            -0.011193593963980675,
+            -0.032580237835645676,
+            -0.031251389533281326,
+            0.012358291074633598,
+            -0.005334935616701841,
+            -0.04430537298321724,
+            -0.0290157962590456,
+            -0.0036191579420119524,
+            -0.007433734834194183,
+            0.009817845188081264,
+            -0.015180139802396297,
+            -0.039052512496709824,
+            -0.036394815891981125,
+            0.031517159193754196,
+            0.0017431365558877587,
+            0.0038946985732764006,
+            -0.03601961210370064,
+            0.0033651133999228477,
+            0.005702323280274868,
+            0.00542873702943325,
+            0.001134406658820808,
+            0.009090886451303959,
+            0.009294122457504272,
+            0.0013835657155141234,
+            -0.01187365222722292,
+            -0.038614772260189056,
+            -0.029203400015830994,
+            -0.0011451547034084797,
+            0.011209228076040745,
+            0.04055332764983177,
+            0.03014140948653221,
+            -0.05312267318367958,
+            -0.04149134084582329,
+            0.02957860380411148,
+            -0.03705142065882683,
+            -0.017228130251169205,
+            0.0033592507243156433,
+            0.00022241112310439348,
+            0.002487682504579425,
+            0.010724589228630066,
+            0.00532321073114872,
+            0.04427410289645195,
+            -0.0007357521099038422,
+            0.012710045091807842,
+            0.01567259430885315,
+            0.013460453599691391,
+            0.02424757555127144,
+            -0.015680411830544472,
+            -0.020354831591248512,
+            0.04755714163184166,
+            -0.023669136688113213,
+            0.01641518622636795,
+            0.011365562677383423,
+            0.011037259362637997,
+            0.013906008563935757,
+            0.017290662974119186,
+            0.001894586137495935,
+            0.011865834705531597,
+            -0.058844536542892456,
+            0.014375013299286366,
+            0.004736953880637884,
+            -0.031954895704984665,
+            -0.024482078850269318,
+            -0.014695500954985619,
+            0.02729611098766327,
+            0.03207996487617493,
+            0.00952080823481083,
+            -0.024466445669531822,
+            -0.022668590769171715,
+            -0.00555380480363965,
+            0.007304758299142122,
+            0.04596252366900444,
+            -0.0008617973071523011,
+            0.02049553394317627,
+            -0.015492809936404228,
+            0.010279033333063126,
+            0.01527394074946642,
+            0.01324940100312233,
+            0.026655137538909912,
+            -0.00941919069737196,
+            -3.502273102640174e-05,
+            -0.043930165469646454,
+            -0.027139775454998016,
+            -0.013937275856733322,
+            0.006112703122198582,
+            0.02712414227426052,
+            0.026389367878437042,
+            0.006374564487487078,
+            -0.003955278545618057,
+            0.022011984139680862,
+            -0.030500980094075203,
+            0.016165051609277725,
+            0.03608214482665062,
+            0.026670770719647408,
+            -0.016665322706103325,
+            0.0023587059695273638,
+            0.01933865435421467,
+            0.0032713122200220823,
+            0.01016178261488676,
+            -0.0252012200653553,
+            0.028859462589025497,
+            -0.022996895015239716,
+            0.029422268271446228,
+            -0.0237160362303257,
+            -0.027217943221330643,
+            -0.030954353511333466,
+            -0.003388563636690378,
+            0.03458132594823837,
+            -0.013046164996922016,
+            -0.01112324371933937,
+            -0.008449913002550602,
+            0.02380983717739582,
+            -0.014961270615458488,
+            -0.0068513862788677216,
+            -0.03702015429735184,
+            -0.017900370061397552,
+            -0.05071511119604111,
+            0.0032380910124629736,
+            0.003027038648724556,
+            -0.022199586033821106,
+            -0.044680576771497726,
+            0.014445364475250244,
+            -0.09092450141906738,
+            -0.004307032562792301,
+            0.032580237835645676,
+            -0.020683135837316513,
+            0.006538716144859791,
+            0.0038204393349587917,
+            0.011740767396986485,
+            -0.03877110779285431,
+            0.0010366971837356687,
+            0.03226756677031517,
+            0.0514029860496521,
+            -0.008637514896690845,
+            -0.01996399462223053,
+            0.00903616938740015,
+            -0.0172437634319067,
+            -0.01904161646962166,
+            0.008434279821813107,
+            0.0228874608874321,
+            0.020135963335633278,
+            0.0036289289128035307,
+            0.026358099654316902,
+            0.01718122884631157,
+            0.030860550701618195,
+            0.03336191177368164,
+            -0.01271786168217659,
+            -0.04918302595615387,
+            0.02768694795668125,
+            0.039083778858184814,
+            0.009739677421748638,
+            -0.015281757339835167,
+            -0.01543809287250042,
+            -0.0034159221686422825,
+            -0.0024661864154040813,
+            0.0010112927993759513,
+            0.0008114769589155912,
+            -0.00029141842969693244,
+            0.01521922368556261,
+            0.028109053149819374,
+            -0.009481724351644516,
+            0.026780204847455025,
+            0.003990453667938709,
+            0.02194944955408573,
+            0.004506359808146954,
+            -0.02513868547976017,
+            -0.010255583561956882,
+            -0.0037227298598736525,
+            0.02481038309633732,
+            0.005276310257613659,
+            -0.03505033254623413,
+            0.006155695300549269,
+            0.029406635090708733,
+            -0.01286637969315052,
+            0.008465546183288097,
+            0.013452637009322643,
+            -0.00037764699663966894,
+            0.016337020322680473,
+            -0.01910415105521679,
+            -0.0001468084374209866,
+            0.02330956608057022,
+            -0.01797853782773018,
+            0.008160692639648914,
+            -0.028593692928552628,
+            -0.012061254121363163,
+            0.007492360193282366,
+            -0.011396829970180988,
+            0.017556432634592056,
+            -0.001251657959073782,
+            0.015680411830544472,
+            -0.005045715719461441,
+            0.0027983984909951687,
+            -0.010654238052666187,
+            0.03501906618475914,
+            -0.00401390390470624,
+            0.03150152415037155,
+            0.021214675158262253,
+            -0.015250490047037601,
+            0.044524241238832474,
+            -0.03295544162392616,
+            -0.025873461738228798,
+            0.01159224845468998,
+            0.006386289373040199,
+            -0.011537531390786171,
+            -0.015336474403738976,
+            -0.013616788201034069,
+            -0.00037911266554147005,
+            0.01718122884631157,
+            -0.02209015190601349,
+            -0.024153774604201317,
+            -0.020010894164443016,
+            -0.010318117216229439,
+            -0.0031638317741453648,
+            0.000499783840496093,
+            -0.0059446427039802074,
+            -0.014593882486224174,
+            0.04880782216787338,
+            -0.05543643236160278,
+            0.01943245530128479,
+            0.047838546335697174,
+            0.013171233236789703,
+            -0.04252314940094948,
+            -0.023997440934181213,
+            -0.014539165422320366,
+            -0.0075509860180318356,
+            -0.041835278272628784,
+            0.014109243638813496,
+            0.002925420878455043,
+            0.005772673990577459,
+            -0.048995424062013626,
+            -0.0009712319006212056,
+            0.005784399341791868,
+            -0.022996895015239716,
+            0.014015442691743374,
+            -0.022934360429644585,
+            -0.022606058046221733,
+            0.01424212846904993,
+            0.019416820257902145,
+            -0.006104886066168547,
+            0.013741856440901756,
+            -0.0035761657636612654,
+            -0.007980907335877419,
+            -0.0021456994581967592,
+            -0.0009365450241602957,
+            0.0011207273928448558,
+            0.019229218363761902,
+            0.004365657921880484,
+            -0.00630812207236886,
+            -0.0013132150052115321,
+            -0.005002723541110754,
+            0.030688581988215446,
+            -0.008864200673997402,
+            -0.009661509655416012,
+            0.03958405181765556,
+            0.035800740122795105,
+            -0.010724589228630066,
+            -0.02218395285308361,
+            -0.023825472220778465,
+            0.014797118492424488,
+            -0.02948480285704136,
+            0.028265388682484627,
+            0.036426082253456116,
+            -0.011670416221022606,
+            -0.03162659332156181,
+            0.005577255040407181,
+            -0.022105785086750984,
+            -0.003996316343545914,
+            0.01704052835702896,
+            0.01804107241332531,
+            -0.04418030381202698,
+            -0.006003268528729677,
+            0.01688419282436371,
+            0.01238955743610859,
+            -0.009137787856161594,
+            -0.03064168244600296,
+            0.008653149008750916,
+            0.01904161646962166,
+            0.013335385359823704,
+            -0.0027808109298348427,
+            0.00702335499227047,
+            -0.0165871549397707,
+            -0.0011353837326169014,
+            -0.000332700670696795,
+            -0.013429186306893826,
+            0.03573820739984512,
+            0.0010484223021194339,
+            0.010294667445123196,
+            -0.00013398408191278577,
+            -0.02005779556930065,
+            -0.012944547459483147,
+            0.03551933914422989,
+            0.01399199292063713,
+            0.038614772260189056,
+            0.0019502805080264807,
+            0.015578794293105602,
+            0.03817703574895859,
+            -0.002221912844106555,
+            -0.0273586455732584,
+            0.021668046712875366,
+            0.0010064073139801621,
+            -0.0441177673637867,
+            0.00657389173284173,
+            -0.03128265589475632,
+            -0.04593125730752945,
+            0.023653503507375717,
+            -0.013624604791402817,
+            0.022793659940361977,
+            -0.005432645324617624,
+            -0.004381291568279266,
+            -0.0318298302590847,
+            0.0035038606729358435,
+            0.02188691683113575,
+            0.015664778649806976,
+            0.04424283653497696,
+            -0.0051629673689603806,
+            -0.005643697455525398,
+            0.011217044666409492,
+            -0.001060147536918521,
+            0.037989433854818344,
+            0.01185801811516285,
+            -0.014281212352216244,
+            0.013859108090400696,
+            0.030266478657722473,
+            0.04761967435479164,
+            -0.02928156778216362,
+            -0.03539426997303963,
+            0.008676598779857159,
+            0.0044633676297962666,
+            0.041303738951683044,
+            0.006831844337284565,
+            0.006155695300549269,
+            0.016790391877293587,
+            0.014695500954985619,
+            -0.008567164652049541,
+            0.024028707295656204,
+            -0.03014140948653221,
+            -0.034768927842378616,
+            0.03670748323202133,
+            -0.008145059458911419,
+            0.012858563102781773,
+            0.01800980605185032,
+            -0.01425776258111,
+            -0.016821658238768578,
+            0.005639789160341024,
+            0.01091219112277031,
+            -0.015344290994107723,
+            0.030798017978668213,
+            0.03389345109462738,
+            0.02398180589079857,
+            -0.0048581138253211975,
+            0.012694410979747772,
+            0.0009091863757930696,
+            0.002092936309054494,
+            -0.01744699850678444,
+            0.007003813050687313,
+            -0.0008837819332256913,
+            -0.01618068479001522,
+            0.043992701917886734,
+            -0.0290157962590456,
+            -0.012928914278745651,
+            -0.0022844469640403986,
+            -0.002614704892039299,
+            0.005502996034920216,
+            0.003972866106778383,
+            0.018619511276483536,
+            -0.012788211926817894,
+            -0.018916549161076546,
+            0.017368830740451813,
+            0.024747848510742188,
+            0.002688963897526264,
+            -0.0072461324743926525,
+            0.023966172710061073,
+            -0.007937915623188019,
+            0.009278489276766777,
+            0.02304379642009735,
+            0.004572802223265171,
+            0.012366107665002346,
+            -0.021542979404330254,
+            -0.006890470162034035,
+            -0.0059876348823308945,
+            0.0075079938396811485,
+            0.044680576771497726,
+            0.0025365371257066727,
+            0.027186676859855652,
+            0.028296655043959618,
+            -0.004799488000571728,
+            -0.009028352797031403,
+            0.0014529394684359431,
+            -0.013859108090400696,
+            0.046806734055280685,
+            0.0006092183757573366,
+            0.007265674415975809,
+            0.013468270190060139,
+            -0.05134045332670212,
+            -0.010779306292533875,
+            0.016024349257349968,
+            0.0012506808852776885,
+            0.03339318186044693,
+            -0.003781355684623122,
+            -0.0033846551086753607,
+            0.014804935082793236,
+            -0.035863276571035385,
+            -0.039677850902080536,
+            0.012366107665002346,
+            0.006018901709467173,
+            -0.011959636583924294,
+            -0.004299215506762266,
+            0.013749673031270504,
+            -0.005788307636976242,
+            -0.004346115980297327,
+            0.009184688329696655,
+            -0.006624700501561165,
+            0.008965819142758846,
+            0.009215954691171646,
+            -0.008739132434129715,
+            0.03986545279622078,
+            -0.009606792591512203,
+            0.03727029263973236,
+            0.0007645764271728694,
+            0.0010112927993759513,
+            0.006867019925266504,
+            -0.01979202590882778,
+            0.03345571458339691,
+            -0.0003559066681191325,
+            -0.04533718153834343,
+            0.0027827650774270296,
+            0.02082383632659912,
+            -0.008231043815612793,
+            0.009825661778450012,
+            -0.006233862601220608,
+            0.015711678192019463,
+            0.03952151536941528,
+            0.0055928886868059635,
+            0.01132647879421711,
+            0.017259396612644196,
+            0.0048581138253211975,
+            0.0018779755337163806,
+            0.031188854947686195,
+            0.0011090021580457687,
+            -0.009231588803231716,
+            0.029672404751181602,
+            -0.018713314086198807,
+            0.037457894533872604,
+            -0.047807276248931885,
+            -0.007414192892611027,
+            0.03561313822865486,
+            0.0005046692676842213,
+            -0.015383374877274036,
+            -0.04196034371852875,
+            0.025029251351952553,
+            -0.03229883313179016,
+            0.021183408796787262,
+            -0.003157969331368804,
+            0.033612050116062164,
+            -0.023825472220778465,
+            0.002618613187223673,
+            0.01883838139474392,
+            -0.013077432289719582,
+            0.02477911487221718,
+            0.014875286258757114,
+            -0.01064642146229744,
+            0.0021417911630123854,
+            -0.013225950300693512,
+            0.006738043390214443,
+            0.04646279662847519,
+            -0.009888196364045143,
+            0.004029537551105022,
+            0.007511902134865522,
+            -0.02224648743867874,
+            -0.018353741616010666,
+            0.04987090080976486,
+            0.0013239630497992039,
+            0.023293932899832726,
+            0.03661368414759636,
+            -0.041897810995578766,
+            -0.02168367989361286,
+            0.02135537751019001,
+            0.004912830889225006,
+            0.0037325008306652308,
+            0.014031076803803444,
+            0.016258852556347847,
+            0.014867469668388367,
+            -0.020386097952723503,
+            0.03411232307553291,
+            0.011131060309708118,
+            0.00669895950704813,
+            -0.019651323556900024,
+            -0.024169407784938812,
+            -0.020776936784386635,
+            -0.03398725390434265,
+            -0.02102707326412201,
+            -0.00927067268639803,
+            0.010200866498053074,
+            -0.008473362773656845,
+            -0.04883908852934837,
+            -0.017790935933589935,
+            0.026358099654316902,
+            -0.0069412789307534695,
+            0.04583745449781418,
+            0.0015369695611298084,
+            0.036957621574401855,
+            0.0012604518560692668,
+            -0.016993626952171326,
+            0.050590042024850845,
+            -0.027937084436416626,
+            0.04158513993024826,
+            0.022840559482574463,
+            -0.004025629255920649,
+            -0.008410829119384289,
+            0.04752587527036667,
+            -0.024122508242726326,
+            0.0032537246588617563,
+            -0.029688037931919098,
+            -0.003392471931874752,
+            -0.024951083585619926,
+            -0.0011197502026334405,
+            -0.0028746118769049644,
+            -0.008684415370225906,
+            0.024357011541724205,
+            -0.015242673456668854,
+            0.020401732996106148,
+            0.019385553896427155,
+            -0.0021437453106045723,
+            0.008739132434129715,
+            -0.020089061930775642,
+            0.012561526149511337,
+            -0.016399553045630455,
+            0.004568893928080797,
+            -0.009919462725520134,
+            0.001229184796102345,
+            -0.03279910609126091,
+            -0.00606189388781786,
+            -0.0067224097438156605,
+            -0.011193593963980675,
+            0.015367741696536541,
+            0.038552239537239075,
+            -0.0045884354040026665,
+            -0.035206668078899384,
+            0.007781580090522766,
+            -0.04008432477712631,
+            0.010169599205255508,
+            0.007144514936953783,
+            0.002470094710588455,
+            -0.009161237627267838,
+            0.01877584680914879,
+            0.022011984139680862,
+            -0.012772578746080399,
+            -0.033737119287252426,
+            0.009724044241011143,
+            -0.006038443651050329,
+            0.016524622216820717,
+            0.0029469169676303864,
+            0.0334869809448719,
+            -0.004009995609521866,
+            0.01880711503326893,
+            -0.012358291074633598,
+            -0.004693961702287197,
+            -0.017556432634592056,
+            0.0023293932899832726,
+            -0.02380983717739582,
+            -0.00526458490639925,
+            -0.00030778476502746344,
+            -0.013358835130929947,
+            -0.00802780780941248,
+            0.0027280477806925774,
+            0.011146693490445614,
+            -0.014156145043671131,
+            0.010732405818998814,
+            -0.019057249650359154,
+            -0.010044530965387821,
+            0.0072109573520720005,
+            -0.03226756677031517,
+            -0.014070160686969757,
+            -0.038552239537239075,
+            -0.003859523218125105,
+            0.013499537482857704,
+            0.03358078375458717,
+            -0.007562711369246244,
+            0.010724589228630066,
+            -0.030250845476984978,
+            -0.0007328208303079009,
+            0.023590968921780586,
+            0.023856738582253456,
+            -0.03545680269598961,
+            0.003523402614519,
+            0.025123052299022675,
+            0.0293441005051136,
+            -0.035206668078899384,
+            0.015625694766640663,
+            0.0351441353559494,
+            0.007093705702573061,
+            0.021371010690927505,
+            -0.0105291698127985,
+            0.042992155998945236,
+            -0.03786436468362808,
+            0.01160788256675005,
+            -0.004826846532523632,
+            0.024294476956129074,
+            0.007840205915272236,
+            0.024591512978076935,
+            -0.02049553394317627,
+            0.0328303724527359,
+            0.033612050116062164,
+            0.02032356522977352,
+            0.04418030381202698,
+            -0.03658241778612137,
+            -0.017415732145309448,
+            0.012538076378405094,
+            -0.01966695673763752,
+            0.01245209202170372,
+            -0.0023606601171195507,
+            -0.02481038309633732,
+            0.009755311533808708,
+            0.024325743317604065,
+            0.008637514896690845,
+            -0.059376075863838196,
+            0.023528434336185455,
+            0.02165241353213787,
+            0.0131008829921484,
+            0.02374730445444584,
+            -0.0371452234685421,
+            0.0004274788370821625,
+            0.015930548310279846,
+            0.015242673456668854,
+            0.020933272317051888,
+            0.017337564378976822,
+            -0.010349384509027004,
+            -0.0008661942556500435,
+            -0.023137597367167473,
+            0.006769310217350721,
+            -0.0035722574684768915,
+            0.010052347555756569,
+            -0.02646753564476967,
+            -0.028187220916152,
+            0.00872349925339222,
+            0.004994906950742006,
+            -0.0021535162813961506,
+            -0.03723902255296707,
+            -0.03933391347527504,
+            0.013749673031270504,
+            0.00819195993244648,
+            -0.044618040323257446,
+            -0.012921096757054329,
+            -0.03823956847190857,
+            -0.01447663176804781,
+            -0.021636780351400375,
+            0.0018261895747855306,
+            -0.010185232385993004,
+            0.0026928724255412817,
+            -0.005202050786465406,
+            0.025513891130685806,
+            -0.010451002046465874,
+            0.04552478343248367,
+            0.006566074676811695,
+            -0.003556623822078109,
+            -0.030125776305794716,
+            0.04802614822983742,
+            -0.007433734834194183,
+            -0.03933391347527504,
+            0.02454461343586445,
+            -0.017790935933589935,
+            0.011662599630653858,
+            0.03187672793865204,
+            -0.007949640974402428,
+            -0.014883102849125862,
+            0.013851291500031948,
+            -0.023262664675712585,
+            -0.042398083955049515,
+            0.007074163760989904,
+            -0.016337020322680473,
+            -0.0293441005051136,
+            -0.041460070759058,
+            -0.004834663588553667,
+            0.018713314086198807,
+            -0.025779660791158676,
+            0.004701778758317232,
+            0.01824430748820305,
+            0.019354287534952164,
+            -0.038520973175764084,
+            0.008551530539989471,
+            -0.00254044565372169,
+            -0.0017480220412835479,
+            -0.010787122882902622,
+            0.009591159410774708,
+            -0.0032380910124629736,
+            -0.004393016919493675,
+            -0.021449178457260132,
+            -0.005784399341791868,
+            -0.029203400015830994,
+            -0.010279033333063126,
+            -0.012006537057459354,
+            -0.027905818074941635,
+            -0.04589999094605446,
+            -0.04483691230416298,
+            0.025482622906565666,
+            0.03126702457666397,
+            -0.003568348940461874,
+            0.0028277114033699036,
+            -0.008379561826586723,
+            -0.012647510506212711,
+            0.019526256248354912,
+            0.00017880828818306327,
+            0.010771489702165127,
+            -0.0133510185405612,
+            0.016634056344628334,
+            -0.010279033333063126,
+            0.00540528679266572,
+            -0.04336735978722572,
+            -0.020370464771986008,
+            0.032986707985401154,
+            0.007062438875436783,
+            -0.0007914465386420488,
+            -0.004721320234239101,
+            0.013468270190060139,
+            -0.02434137649834156,
+            -0.01286637969315052,
+            0.03392472118139267,
+            0.01721249520778656,
+            0.01452353224158287,
+            -0.015000354498624802,
+            -0.008817300200462341,
+            0.003703187918290496,
+            -0.011412463150918484,
+            0.0049362811259925365,
+            -0.004615794401615858,
+            0.009075253270566463,
+            0.013804391026496887,
+            0.013569887727499008,
+            0.013898191973567009,
+            0.02235592156648636,
+            -0.009043985977768898,
+            -0.00649181567132473,
+            -0.003738363506272435,
+            0.008465546183288097,
+            -0.006272946484386921,
+            -0.006894378457218409,
+            -0.025873461738228798,
+            0.033674582839012146,
+            0.01890091598033905,
+            0.010998175479471684,
+            -0.015227040275931358,
+            -0.003757905215024948,
+            0.009356657043099403,
+            -0.009958546608686447,
+            -0.005823483224958181,
+            0.0037930808030068874,
+            0.025029251351952553,
+            -0.02194944955408573,
+            -0.013061799108982086,
+            -0.010865290649235249,
+            -0.0026479260995984077,
+            0.006870928220450878,
+            0.03773929551243782,
+            0.018431909382343292,
+            -0.008528080768883228,
+            -0.008488996885716915,
+            0.002384110586717725,
+            0.017337564378976822,
+            0.023919273167848587,
+            -0.03539426997303963,
+            -0.030751116573810577,
+            0.008105975575745106,
+            -0.0038829734548926353,
+            0.021136507391929626,
+            0.00532321073114872,
+            0.003744225949048996,
+            0.003537081880494952,
+            0.0722893550992012,
+            -0.0008554462110623717,
+            0.010067981667816639,
+            -0.017900370061397552,
+            -0.01520358957350254,
+            -0.0013493674341589212,
+            -0.005565530154854059,
+            -0.01308524888008833,
+            0.036957621574401855,
+            0.001335688168182969,
+            0.044618040323257446,
+            0.011865834705531597,
+            -0.006296396721154451,
+            -0.01190491858869791,
+            -0.0006805462762713432,
+            0.012084703892469406,
+            -0.0005608522333204746,
+            -0.0018662504153326154,
+            -0.0025658500380814075,
+            0.008973635733127594,
+            0.006499632261693478,
+            0.02570149302482605,
+            0.0005916306981816888,
+            0.017384463921189308,
+            -0.02424757555127144,
+            0.03601961210370064,
+            0.01711869426071644,
+            0.012842929922044277,
+            -0.019416820257902145,
+            -0.008207593113183975,
+            0.0028003526385873556,
+            0.0044633676297962666,
+            -0.007707321085035801,
+            -0.0006507448852062225,
+            -0.008660965599119663,
+            0.030454080551862717,
+            -0.03007887676358223,
+            0.01668095588684082,
+            0.011725133284926414,
+            -0.018197407945990562,
+            -0.005940734408795834,
+            0.015352108515799046,
+            -0.002077302895486355,
+            0.004459459334611893,
+            -0.006112703122198582,
+            0.0052450429648160934,
+            0.024357011541724205,
+            -0.028187220916152,
+            -0.006128336302936077,
+            0.010458819568157196,
+            -0.01493781991302967,
+            -0.0036445623263716698,
+            0.004221048206090927,
+            0.00956770870834589,
+            -0.014875286258757114,
+            0.011428097262978554,
+            0.02434137649834156,
+            -0.027937084436416626,
+            -0.009450457990169525,
+            0.011310845613479614,
+            -0.00010235064837615937,
+            0.01866641268134117,
+            0.023403367027640343,
+            0.008739132434129715,
+            -0.006448823492974043,
+            -0.0031423356849700212,
+            0.010341567918658257,
+            -0.009677143767476082,
+            -0.007468909956514835,
+            -0.0069842711091041565,
+            -0.010716771706938744,
+            0.01940118707716465,
+            -0.01591491512954235,
+            -0.0026928724255412817,
+            -0.003937690518796444,
+            -0.0106933219358325,
+            0.001270222826860845,
+            -0.013069615699350834,
+            0.017071794718503952,
+            0.0013249401235952973,
+            0.017931638285517693,
+            0.010951275005936623,
+            0.004846388474106789,
+            0.023590968921780586,
+            -0.012655327096581459,
+            0.012248856015503407,
+            -0.032017432153224945,
+            -0.0018027393380180001,
+            0.009145604446530342,
+            0.006022810470312834,
+            -0.01635265350341797,
+            -0.019948361441493034,
+            -0.013194683939218521,
+            0.0032244117464870214,
+            -0.009372290223836899,
+            0.0065465327352285385,
+            0.014296846464276314,
+            0.037926897406578064,
+            -0.016133783385157585,
+            -0.030469713732600212,
+            -0.015094155445694923,
+            0.012905463576316833,
+            0.016243217512965202,
+            -0.023888004943728447,
+            -0.009669327177107334,
+            -0.007269582711160183,
+            0.0034432809334248304,
+            -0.013171233236789703,
+            -0.01833810843527317,
+            -0.01668095588684082,
+            -0.003011405235156417,
+            -0.025889094918966293,
+            0.0013493674341589212,
+            -0.0043773832730948925,
+            0.0067224097438156605,
+            0.019479354843497276,
+            -0.0034725936129689217,
+            -0.02179311402142048,
+            -0.02490418404340744,
+            0.0175720676779747,
+            -0.02434137649834156,
+            0.008231043815612793,
+            0.025623325258493423,
+            -0.017853470519185066,
+            0.004928464535623789,
+            0.026670770719647408,
+            0.004948006477206945,
+            -0.009333206340670586,
+            0.012670961208641529,
+            0.009778761304914951,
+            0.006362839136272669,
+            0.008700049482285976,
+            0.0190728846937418,
+            0.012061254121363163,
+            0.020933272317051888,
+            -0.021605512127280235,
+            -0.013593338429927826,
+            -0.004025629255920649,
+            0.03626974672079086,
+            -0.0010327888885512948,
+            -0.00878603383898735,
+            0.014617333188652992,
+            0.0361446775496006,
+            0.005663239397108555,
+            -0.005585072096437216,
+            -0.01786910369992256,
+            0.0029586420860141516,
+            0.014601700007915497,
+            0.005291943438351154,
+            0.002679192926734686,
+            -0.00567105645313859,
+            -0.01048226933926344,
+            -0.00022790727962274104,
+            -0.0009438732522539794,
+            0.004697869997471571,
+            -0.026389367878437042,
+            0.018697679042816162,
+            -0.023419000208377838,
+            0.014171778224408627,
+            -0.0013063752558082342,
+            0.0007059507770463824,
+            0.03093871846795082,
+            -0.0022023709025233984,
+            -0.022340288385748863,
+            0.002501361770555377,
+            0.0019502805080264807,
+            0.013601155020296574,
+            0.022230852395296097,
+            -0.010943458415567875,
+            0.01261624414473772,
+            -0.003331892192363739,
+            0.0014382830122485757,
+            0.007461093366146088,
+            0.030469713732600212,
+            0.002923466730862856,
+            -0.011045075953006744,
+            0.020354831591248512,
+            0.009638059884309769,
+            0.0007425918010994792,
+            -0.007703412789851427,
+            0.020886370912194252,
+            -0.015602244064211845,
+            0.012491175904870033,
+            -0.012663144618272781,
+            -0.006323755253106356,
+            3.625936528806051e-07,
+            0.0010777352144941688,
+            -0.0023626144975423813,
+            -0.030688581988215446,
+            0.022965628653764725,
+            0.0005041807307861745,
+            0.0177440345287323,
+            -0.009505175054073334,
+            -0.019307386130094528,
+            0.002489636652171612,
+            0.020933272317051888,
+            -0.006855294574052095,
+            -0.01261624414473772,
+            -0.014304663054645061,
+            -0.014335930347442627,
+            -0.020073428750038147,
+            0.006695051211863756,
+            0.004060804378241301,
+            -0.015039438381791115,
+            0.0053740194998681545,
+            -0.003746180096641183,
+            -0.019119784235954285,
+            0.005280218552798033,
+            0.02410687506198883,
+            -0.027765115723013878,
+            0.03162659332156181,
+            -0.015860198065638542,
+            -0.015297390520572662,
+            0.009075253270566463,
+            0.0035175401717424393,
+            0.0005569438217207789,
+            0.005502996034920216,
+            -0.02135537751019001,
+            0.0020694860722869635,
+            0.020417366176843643,
+            -0.01704052835702896,
+            -0.0019932726863771677,
+            0.010075798258185387,
+            0.017493899911642075,
+            0.0007289124769158661,
+            -0.018025439232587814,
+            0.02490418404340744,
+            -0.016227584332227707,
+            0.02454461343586445,
+            0.02126157470047474,
+            -0.041804008185863495,
+            0.042116679251194,
+            -0.024654047563672066,
+            0.015828929841518402,
+            0.002237546257674694,
+            -0.006437098141759634,
+            0.013069615699350834,
+            0.018885280936956406,
+            -0.005256768316030502,
+            -0.0012780395336449146,
+            0.002804261166602373,
+            0.03167349472641945,
+            -0.023403367027640343,
+            0.024528980255126953,
+            0.02132410928606987,
+            -0.00010387735528638586,
+            0.02481038309633732,
+            -0.021230308338999748,
+            -0.029766205698251724,
+            0.019620057195425034,
+            -0.023465899750590324,
+            -0.034174855798482895,
+            0.020042162388563156,
+            -0.008418645709753036,
+            -0.008184143342077732,
+            -0.020339198410511017,
+            0.006386289373040199,
+            0.0054639121517539024,
+            -0.013007081113755703,
+            0.002849207492545247,
+            0.0027397728990763426,
+            0.02606106363236904,
+            0.022574789822101593,
+            0.01520358957350254,
+            -0.003320167073979974,
+            0.018166139721870422,
+            -0.0015731221064925194,
+            0.032580237835645676,
+            -0.0020304021891206503,
+            0.0004528832796495408,
+            -0.011052892543375492,
+            -0.0021456994581967592,
+            0.0034198306966573,
+            0.0039220573380589485,
+            0.017384463921189308,
+            -0.02069876901805401,
+            -0.015868013724684715,
+            -0.0351441353559494,
+            -0.0025971170980483294,
+            -0.003539036260917783,
+            0.010708955116569996,
+            0.03508159890770912,
+            -0.00903616938740015,
+            -0.013999809511005878,
+            0.009192504920065403,
+            -0.0068083941005170345,
+            -0.0002442736295051873,
+            -0.02330956608057022,
+            -0.020354831591248512,
+            -0.01493781991302967,
+            -0.013272850774228573,
+            -0.008856384083628654,
+            -0.020042162388563156,
+            0.012960180640220642,
+            -0.010599520988762379,
+            0.01048226933926344,
+            -0.02795271761715412,
+            0.014406280592083931,
+            -0.012280123308300972,
+            0.009059620089828968,
+            -0.005456095561385155,
+            -0.024200676009058952,
+            -0.004357841331511736,
+            0.02653006836771965,
+            -0.0034745479933917522,
+            -0.021449178457260132,
+            -0.01786910369992256,
+            0.005202050786465406,
+            -0.014844018965959549,
+            -0.0016180684324353933,
+            -0.015101972036063671,
+            -0.009614609181880951,
+            0.0008300417684949934,
+            0.0009849112248048186,
+            0.005475637502968311,
+            0.010443185456097126,
+            -0.0029332374688237906,
+            -0.018150506541132927,
+            0.014015442691743374,
+            0.04139753803610802,
+            -0.04386763274669647,
+            -0.017134329304099083,
+            0.012076887302100658,
+            0.010404101572930813,
+            -0.0036308830603957176,
+            0.010474452748894691,
+            -0.013554254546761513,
+            -0.003349479753524065,
+            -0.024388277903199196,
+            -0.01308524888008833,
+            -0.02513868547976017,
+            0.00749626848846674,
+            0.017931638285517693,
+            -0.008137242868542671,
+            0.03273657336831093,
+            0.030391545966267586,
+            -0.01877584680914879,
+            -0.0013884512009099126,
+            0.010583886876702309,
+            0.015281757339835167,
+            -0.029031431302428246,
+            0.016368286684155464,
+            -0.009911646135151386,
+            0.021167773753404617,
+            0.00968496035784483,
+            -0.021152140572667122,
+            -0.0146720502525568,
+            -0.016524622216820717,
+            0.007011629641056061,
+            -0.013874741271138191,
+            -3.935858694603667e-05,
+            0.027264844626188278,
+            0.015500626526772976,
+            -0.007484543602913618,
+            -0.014109243638813496,
+            0.018650779500603676,
+            0.013335385359823704,
+            0.019573155790567398,
+            -0.00540528679266572,
+            -0.0038399812765419483,
+            -0.007801122032105923,
+            -0.00946609117090702,
+            -0.018947815522551537,
+            0.00927067268639803,
+            -0.02984437346458435,
+            0.019057249650359154,
+            -0.01854134537279606,
+            -0.007437643129378557,
+            0.03273657336831093,
+            -0.04652532935142517,
+            -0.012913280166685581,
+            -0.013280668295919895,
+            0.02895326353609562,
+            0.015977447852492332,
+            0.004623610991984606,
+            0.009919462725520134,
+            -0.021449178457260132,
+            0.0033729299902915955,
+            -0.00321659492328763,
+            0.003099343739449978,
+            -0.016837291419506073,
+            -0.016399553045630455,
+            0.012413008138537407,
+            0.002137882634997368,
+            -0.004228864796459675,
+            0.0069061038084328175,
+            -0.005420919973403215,
+            -0.013538620434701443,
+            0.002679192926734686,
+            -0.0007513856398873031,
+            0.0015975494170561433,
+            0.008817300200462341,
+            0.002515041036531329,
+            0.01495345402508974,
+            0.01112324371933937,
+            -0.008660965599119663,
+            -0.005647606216371059,
+            -0.005647606216371059,
+            0.00088085065362975,
+            0.010153966024518013,
+            0.005284126847982407,
+            -0.006398014724254608,
+            -0.011631332337856293,
+            0.003376838518306613,
+            -0.007996541447937489,
+            0.026186130940914154,
+            -0.008418645709753036,
+            -0.042335547506809235,
+            0.002751498017460108,
+            -0.042366817593574524,
+            0.02354406751692295,
+            -0.02573275938630104,
+            -0.020651869475841522,
+            -0.009974180720746517,
+            -0.003695371327921748,
+            -0.019979627802968025,
+            0.009380106814205647,
+            0.007121064234524965,
+            -0.007941823452711105,
+            0.00792619027197361,
+            0.001134406658820808,
+            -0.015023804269731045,
+            0.0105291698127985,
+            -0.0101148821413517,
+            -0.010552620515227318,
+            0.005710139870643616,
+            0.0073868343606591225,
+            0.008848567493259907,
+            -0.006272946484386921,
+            -0.007472818251699209,
+            0.009505175054073334,
+            0.014734584838151932,
+            0.005229409784078598,
+            0.0053740194998681545,
+            0.020667502656579018,
+            0.013319752179086208,
+            -0.022496622055768967,
+            0.0023606601171195507,
+            -0.008019991219043732,
+            0.0038380271289497614,
+            0.003476502140983939,
+            -0.005932917352765799,
+            -0.009192504920065403,
+            0.003963095135986805,
+            -0.0003766699228435755,
+            4.7236411774065346e-05,
+            -0.0018340062815696,
+            -0.01614941656589508,
+            -0.005491270683705807,
+            -0.012006537057459354,
+            0.004221048206090927,
+            0.0300163421779871,
+            -0.006233862601220608,
+            -0.006398014724254608,
+            -0.019682591781020164,
+            0.018494443967938423,
+            0.03404978662729263,
+            0.017556432634592056,
+            -0.028671860694885254,
+            0.0005080891423858702,
+            -0.006417556665837765,
+            0.003300625132396817,
+            -0.025513891130685806,
+            0.022856194525957108,
+            0.018744580447673798,
+            0.009458274580538273,
+            0.011834568344056606,
+            -0.0025502166245132685,
+            0.005721865221858025,
+            -0.01548499334603548,
+            0.012858563102781773,
+            -0.004557168576866388,
+            -0.012014353647828102,
+            0.005006631836295128,
+            0.03942771628499031,
+            -0.015993082895874977,
+            -0.009677143767476082,
+            0.0030192218255251646,
+            0.02424757555127144,
+            -0.011889285407960415,
+            -0.007238315884023905,
+            0.011717316694557667,
+            0.009700593538582325,
+            0.004580618813633919,
+            0.00878603383898735,
+            0.06472273916006088,
+            0.0021867374889552593,
+            0.01638391986489296,
+            0.01283511333167553,
+            -0.021136507391929626,
+            -0.01919795200228691,
+            -0.005780491046607494,
+            -0.020354831591248512,
+            0.009012719616293907,
+            0.00900490302592516,
+            -0.011576615273952484,
+            -0.004068621434271336,
+            -0.020214131101965904,
+            0.0060736192390322685,
+            0.00731257488951087,
+            0.011936185881495476,
+            -0.0024192859418690205,
+            0.006870928220450878,
+            0.04274202138185501,
+            0.019448088482022285,
+            -0.0017196862027049065,
+            0.008105975575745106,
+            0.01419522799551487,
+            -0.009489541873335838,
+            -0.012420824728906155,
+            -0.002659651217982173,
+            0.0026107963640242815,
+            0.00877039972692728,
+            -0.010927824303507805,
+            0.033205579966306686,
+            -0.010419735684990883,
+            -0.015774212777614594,
+            0.011005992069840431,
+            -0.008066891692578793,
+            0.011013808660209179,
+            -0.012311390601098537,
+            0.017916003242135048,
+            -0.016665322706103325,
+            -0.0033221212215721607,
+            -0.0002916626981459558,
+            0.00019004486966878176,
+            0.017603334039449692,
+            0.0018095789710059762,
+            -0.010497902520000935,
+            0.0016815796261653304,
+            0.013851291500031948,
+            -0.008590614423155785,
+            0.003435464110225439,
+            -0.006261221133172512,
+            0.003959186840802431,
+            -0.011420279741287231,
+            -0.015555343590676785,
+            0.007191415410488844,
+            0.018822748214006424,
+            -0.026154864579439163,
+            -0.013210317119956017,
+            0.026576969772577286,
+            0.0071601481176912785,
+            0.002671376336365938,
+            0.013593338429927826,
+            -0.0058117578737437725,
+            0.0030954352114349604,
+            -0.016243217512965202,
+            -0.01777530275285244,
+            0.0061635118909180164,
+            -0.0048932889476418495,
+            0.019557522609829903,
+            -0.0024056066758930683,
+            0.004514176398515701,
+            -0.002837482374161482,
+            -0.008535897359251976,
+            0.019510623067617416,
+            0.015430275350809097,
+            -0.005170783959329128,
+            -0.006269038189202547,
+            0.011459363624453545,
+            -0.01827557571232319,
+            -0.020948905497789383,
+            0.003243953688070178,
+            0.020401732996106148,
+            -0.0065113576129078865,
+            0.004994906950742006,
+            -0.00030558629077859223,
+            -0.0012457953998818994,
+            -0.014664233662188053,
+            -0.009974180720746517,
+            -0.020902004092931747,
+            0.01919795200228691,
+            0.01336665265262127,
+            -0.02251225709915161,
+            0.0030192218255251646,
+            -0.005932917352765799,
+            0.02052680030465126,
+            -0.017587700858712196,
+            -0.003423738991841674,
+            0.033080510795116425,
+            -0.017759669572114944,
+            -0.004557168576866388,
+            -0.008231043815612793,
+            -0.014930003322660923,
+            0.016790391877293587,
+            0.03867730870842934,
+            0.0012037804117426276,
+            0.01014614850282669,
+            0.0029156499076634645,
+            -0.00452590174973011,
+            0.03558187186717987,
+            0.012647510506212711,
+            0.0004885472590103745,
+            -0.0032771748956292868,
+            -0.013835657387971878,
+            -0.02626429870724678,
+            0.0049440981820225716,
+            0.0026518343947827816,
+            -0.002016722923144698,
+            -5.20302819495555e-05,
+            -0.006014993414282799,
+            -0.007906648330390453,
+            0.006659876089543104,
+            -0.021668046712875366,
+            0.007797213736921549,
+            -0.016931092366576195,
+            -0.00276517728343606,
+            -0.00231375964358449,
+            -0.012796029448509216,
+            -0.005303668789565563,
+            0.010802756063640118,
+            -0.0016786482883617282,
+            -0.026123598217964172,
+            -0.03404978662729263,
+            -0.011670416221022606,
+            -0.012311390601098537,
+            -0.0007938892231322825,
+            0.00760179478675127,
+            -0.0267020370811224,
+            0.010224316269159317,
+            -0.0095364423468709,
+            -0.0014138557016849518,
+            0.0013679323019459844,
+            0.011983086355030537,
+            -0.006968637928366661,
+            0.0020362648647278547,
+            -0.02925029955804348,
+            -0.023434633389115334,
+            0.02135537751019001,
+            -0.003925965633243322,
+            0.012303573079407215,
+            -0.012538076378405094,
+            -0.004846388474106789,
+            -0.012592793442308903,
+            0.0177440345287323,
+            -0.017603334039449692,
+            -0.014007626101374626,
+            -0.014617333188652992,
+            -0.0005115089588798583,
+            -0.005366202909499407,
+            -0.009231588803231716,
+            -0.010083614848554134,
+            -0.010638604871928692,
+            -0.021152140572667122,
+            -0.014812751673161983,
+            -0.008442096412181854,
+            0.001930738682858646,
+            0.020151596516370773,
+            -0.006237770896404982,
+            0.026311200112104416,
+            0.02126157470047474,
+            0.00823886040598154,
+            -0.00520595908164978,
+            0.01425776258111,
+            -0.006937370635569096,
+            0.007679962553083897,
+            -0.00952080823481083,
+            0.0003239068028051406,
+            0.01671222411096096,
+            0.012850746512413025,
+            0.008410829119384289,
+            -0.004197597969323397,
+            0.002706551691517234,
+            0.01916668564081192,
+            0.015078521333634853,
+            0.01664968952536583,
+            0.005291943438351154,
+            0.008731315843760967,
+            0.010615154169499874,
+            0.0015399008989334106,
+            -0.005377927795052528,
+            0.0008955070516094565,
+            0.011701683513820171,
+            -0.011568798683583736,
+            -0.02005779556930065,
+            0.01329630147665739,
+            0.00930193904787302,
+            -0.018916549161076546,
+            0.009215954691171646,
+            0.0006575845764018595,
+            -0.009645876474678516,
+            -0.01836937665939331,
+            -0.014586065895855427,
+            -0.007805030792951584,
+            0.015430275350809097,
+            0.007918373681604862,
+            -0.022637324407696724,
+            0.01567259430885315,
+            0.010153966024518013,
+            -0.0015311070019379258,
+            -0.02712414227426052,
+            0.018619511276483536,
+            0.014054526574909687,
+            0.010638604871928692,
+            -0.04821375012397766,
+            -0.006069710943847895,
+            0.020730035379529,
+            -0.01329630147665739,
+            -0.026076696813106537,
+            -0.007625245023518801,
+            -0.008879834786057472,
+            -0.03902124613523483,
+            -0.0030758932698518038,
+            0.0029586420860141516,
+            -0.014844018965959549,
+            -0.003974820487201214,
+            0.015539710409939289,
+            0.00391228636726737,
+            0.018979083746671677,
+            -0.007781580090522766,
+            -0.01336665265262127,
+            -0.019213585183024406,
+            0.005202050786465406,
+            -0.010787122882902622,
+            -0.024263210594654083,
+            0.0027554063126444817,
+            -0.008465546183288097,
+            0.0022180043160915375,
+            0.01700926013290882,
+            0.042835820466279984,
+            0.01786910369992256,
+            -0.019917093217372894,
+            -0.025357555598020554,
+            -0.019307386130094528,
+            -0.007953548803925514,
+            0.009137787856161594,
+            0.007179690059274435,
+            -0.0021456994581967592,
+            0.003595707705244422,
+            0.006593433208763599,
+            0.031923629343509674,
+            0.0009536442230455577,
+            0.020886370912194252,
+            -3.169450064888224e-05,
+            -0.003814576892182231,
+            -0.002132020192220807,
+            -0.018979083746671677,
+            -0.0016053662402555346,
+            0.0029664586763828993,
+            0.013139965943992138,
+            0.004826846532523632,
+            0.005385744851082563,
+            0.016665322706103325,
+            -0.005628064274787903,
+            -0.0036035242956131697,
+            -0.017056161537766457,
+            -0.004428192041814327,
+            -0.004330482799559832,
+            0.03717648983001709,
+            0.020636234432458878,
+            -0.0199014600366354,
+            -0.0165871549397707,
+            -0.017431365326046944,
+            0.02194944955408573,
+            -0.00830921158194542,
+            -0.008997085504233837,
+            0.02099580504000187,
+            0.010239950381219387,
+            0.0015311070019379258,
+            0.0027280477806925774,
+            -0.006612975150346756,
+            0.011357746087014675,
+            0.02570149302482605,
+            0.012561526149511337,
+            -0.001964936964213848,
+            -0.004002179019153118,
+            -0.00016915948071982712,
+            -0.019323019310832024,
+            -0.01641518622636795,
+            0.00010302239388693124,
+            -0.012928914278745651,
+            0.025982895866036415,
+            -0.023106329143047333,
+            0.004314849153161049,
+            -0.01424212846904993,
+            -0.0024075608234852552,
+            -0.021996350958943367,
+            -0.011514080688357353,
+            0.014758034609258175,
+            -0.0237160362303257,
+            0.0054170116782188416,
+            -0.016759123653173447,
+            0.0010660100961104035,
+            0.011209228076040745,
+            -0.005159058608114719,
+            0.012561526149511337,
+            -0.00763697037473321,
+            0.004084255080670118,
+            0.004287490621209145,
+            0.00872349925339222,
+            -0.009145604446530342,
+            0.0015164506621658802,
+            0.005538171622902155,
+            -0.014578249305486679,
+            0.009239405393600464,
+            0.00925503857433796,
+            -0.004748679231852293,
+            0.03917757794260979,
+            0.015281757339835167,
+            0.00014949544856790453,
+            -0.009395739994943142,
+            -0.0038673398084938526,
+            -0.03140772506594658,
+            0.027014708146452904,
+            -0.012663144618272781,
+            0.021386643871665,
+            0.005983726587146521,
+            -0.01490655355155468,
+            -0.0016161142848432064,
+            -0.002688963897526264,
+            0.013194683939218521,
+            -0.009935096837580204,
+            0.0008969727205112576,
+            -0.013624604791402817,
+            0.011990902945399284,
+            0.00476822117343545,
+            -0.01016178261488676,
+            0.012295756489038467,
+            -0.015305208042263985,
+            0.0016571521991863847,
+            -0.007183598354458809,
+            0.0022844469640403986,
+            0.01588364690542221,
+            -0.0033416631631553173,
+            0.0028589784633368254,
+            -0.034675128757953644,
+            -0.015570977702736855,
+            -0.011529714800417423,
+            -8.244234777521342e-05,
+            0.007976999506354332,
+            -0.01754079945385456,
+            -0.028077786788344383,
+            -0.015977447852492332,
+            0.0099507300183177,
+            -0.0005926077719777822,
+            0.00028848715010099113,
+            -0.004658786579966545,
+            0.005225501023232937,
+            0.0009604838560335338,
+            -0.007468909956514835,
+            0.017525166273117065,
+            0.014875286258757114,
+            -0.026358099654316902,
+            -0.03081365115940571,
+            0.004564985167235136,
+            -0.0003722730034496635,
+            -0.027608782052993774,
+            -0.014077977277338505,
+            0.001066987169906497,
+            -0.0006072642281651497,
+            0.021199041977524757,
+            -0.020776936784386635,
+            0.0014588020276278257,
+            -0.00043260856182314456,
+            0.0030055425595492125,
+            -0.036394815891981125,
+            0.0030954352114349604,
+            -0.003144290065392852,
+            -0.0029273750260472298,
+            0.005565530154854059,
+            -0.01542245876044035,
+            0.0013278713449835777,
+            -0.008864200673997402,
+            -0.011428097262978554,
+            -0.025748392567038536,
+            -0.004592344164848328,
+            0.001534038339741528,
+            -8.647285721963271e-05,
+            0.023121964186429977,
+            -0.00622604601085186,
+            -0.019776392728090286,
+            0.002987954765558243,
+            -0.0019424638012424111,
+            -0.0030758932698518038,
+            0.018447542563080788,
+            0.020010894164443016,
+            -0.00651526590809226,
+            -0.0015105879865586758,
+            -0.0016258851392194629,
+            -0.001716754981316626,
+            -0.0004516619083005935,
+            0.039146311581134796,
+            -0.008551530539989471,
+            -0.006882653571665287,
+            0.0050261737778782845,
+            0.0016591063467785716,
+            -0.005444370210170746,
+            0.012100338004529476,
+            0.0015594427241012454,
+            0.007918373681604862,
+            0.0018388917669653893,
+            -0.010130515322089195,
+            -0.01824430748820305,
+            0.002528720535337925,
+            0.013851291500031948,
+            0.00581957446411252,
+            0.004197597969323397,
+            -0.014562616124749184,
+            0.008082525804638863,
+            -0.0075978864915668964,
+            -0.008778216317296028,
+            -0.010771489702165127,
+            0.007816756144165993,
+            -0.00444382568821311,
+            0.019526256248354912,
+            0.0003461357264313847,
+            0.021480444818735123,
+            0.022340288385748863,
+            -0.008442096412181854,
+            0.017024893313646317,
+            0.031485892832279205,
+            8.806063851807266e-05,
+            0.004744770936667919,
+            -0.011365562677383423,
+            -0.014679866842925549,
+            -0.013233767822384834,
+            0.03389345109462738,
+            0.004541534930467606,
+            -0.004267948679625988,
+            0.0018193499417975545,
+            -0.007316483184695244,
+            -0.01283511333167553,
+            0.012413008138537407,
+            0.003392471931874752,
+            -0.018556978553533554,
+            -0.004740862175822258,
+            0.009239405393600464,
+            0.006878744810819626,
+            0.012928914278745651,
+            0.02158987894654274,
+            -0.003869294188916683,
+            0.0028316196985542774,
+            0.004959731362760067,
+            0.029953807592391968,
+            -0.006374564487487078,
+            0.028593692928552628,
+            -0.0062494962476193905,
+            -0.0034823645837605,
+            -0.01107634324580431,
+            -0.015750762075185776,
+            -0.008426462300121784,
+            0.0029312833212316036,
+            0.0030602598562836647,
+            0.006186962127685547,
+            -0.01833810843527317,
+            0.006257312837988138,
+            0.02809341996908188,
+            0.0030055425595492125,
+            0.014734584838151932,
+            0.009278489276766777,
+            0.012741311453282833,
+            0.017321931198239326,
+            -0.00479167141020298,
+            -0.014851835556328297,
+            -0.003451097756624222,
+            0.020808203145861626,
+            -0.006683326326310635,
+            0.024654047563672066,
+            -0.0034745479933917522,
+            0.004350024741142988,
+            -0.00021972411195747554,
+            -0.0038810193073004484,
+            0.020479900762438774,
+            0.01500817108899355,
+            0.01664968952536583,
+            -0.001603411976248026,
+            0.00850463006645441,
+            0.0018349833553656936,
+            0.0017480220412835479,
+            -0.012913280166685581,
+            -0.016508987173438072,
+            0.01786910369992256,
+            0.0038477980997413397,
+            0.023168863728642464,
+            -0.022199586033821106,
+            -0.0007113247411325574,
+            0.006018901709467173,
+            0.016430821269750595,
+            0.004131155554205179,
+            0.010552620515227318,
+            -0.005444370210170746,
+            0.009544258937239647,
+            -0.005905558820813894,
+            0.0016913504805415869,
+            -0.01824430748820305,
+            -0.011600065045058727,
+            0.011631332337856293,
+            -0.00819195993244648,
+            0.010685505345463753,
+            0.001274131122045219,
+            0.009622426703572273,
+            -0.013843473978340626,
+            0.029672404751181602,
+            0.021402277052402496,
+            0.015234856866300106,
+            0.01499253697693348,
+            -0.0024779115337878466,
+            -0.007910557091236115,
+            -0.012420824728906155,
+            0.024560246616601944,
+            0.006800577510148287,
+            0.0029156499076634645,
+            -0.004475092515349388,
+            0.024153774604201317,
+            -0.0004384711501188576,
+            0.005151242017745972,
+            -0.0005886994185857475,
+            0.012967998161911964,
+            -0.01638391986489296,
+            0.008825116790831089,
+            -0.023559702560305595,
+            -0.013210317119956017,
+            -0.01913541741669178,
+            0.019588788971304893,
+            0.012741311453282833,
+            -0.0047721294686198235,
+            0.01543809287250042,
+            0.017603334039449692,
+            -0.0027319560758769512,
+            -0.027280477806925774,
+            -0.009786577895283699,
+            1.4351075151353143e-05,
+            -0.0032380910124629736,
+            -0.015727313235402107,
+            -0.0099507300183177,
+            0.002714368514716625,
+            -0.004885472357273102,
+            0.015547527000308037,
+            -0.0037637678906321526,
+            -0.024591512978076935,
+            -0.005686689633876085,
+            -0.007175781764090061,
+            -0.016555888578295708,
+            -0.013577704317867756,
+            0.008184143342077732,
+            0.005749223753809929,
+            -0.011506264097988605,
+            -0.018916549161076546,
+            0.018494443967938423,
+            -0.004693961702287197,
+            0.03214249759912491,
+            -0.015664778649806976,
+            -0.002808169461786747,
+            0.0053740194998681545,
+            0.01940118707716465,
+            0.0002865329443011433,
+            -0.0021183406934142113,
+            0.009638059884309769,
+            0.010615154169499874,
+            0.003232228569686413,
+            -0.007734679616987705,
+            0.0012917189160361886,
+            -0.0002904413267970085,
+            -0.019385553896427155,
+            0.005893833935260773,
+            -0.0005007609142921865,
+            -0.014070160686969757,
+            0.003351433901116252,
+            0.014335930347442627,
+            0.009622426703572273,
+            0.005452187266200781,
+            -0.02789018489420414,
+            0.012897646985948086,
+            -0.008582797832787037,
+            -0.011256128549575806,
+            -0.0016356561100110412,
+            0.01851007714867592,
+            -0.021965082734823227,
+            0.027937084436416626,
+            0.01419522799551487,
+            -0.00941919069737196,
+            0.005166875664144754,
+            0.019948361441493034,
+            0.017024893313646317,
+            0.011474997736513615,
+            -0.023794203996658325,
+            -0.004221048206090927,
+            0.01425776258111,
+            0.006304213311523199,
+            -0.0006717524374835193,
+            -0.0020206314511597157,
+            0.02463841438293457,
+            0.0034139680210500956,
+            -0.0018310750601813197,
+            -0.0070272632874548435,
+            -0.0018193499417975545,
+            0.009309755638241768,
+            0.021183408796787262,
+            0.009927279315888882,
+            0.018947815522551537,
+            -0.00131028366740793,
+            0.01578984595835209,
+            -0.004295307211577892,
+            -0.009669327177107334,
+            0.014844018965959549,
+            -0.0005882108816877007,
+            -0.009841295890510082,
+            0.01668095588684082,
+            0.008418645709753036,
+            0.012373924255371094,
+            -0.0008901330293156207,
+            -0.009872562251985073,
+            0.02871876023709774,
+            0.015109788626432419,
+            -0.0008701026090420783,
+            0.02046426571905613,
+            -0.02685837261378765,
+            0.024951083585619926,
+            0.012772578746080399,
+            0.0160868838429451,
+            -0.022340288385748863,
+            -0.004475092515349388,
+            0.00048488311585970223,
+            -0.018259940668940544,
+            -0.008700049482285976,
+            0.0037950349505990744,
+            -0.009989813901484013,
+            -0.0020264938939362764,
+            -0.037395358085632324,
+            0.01303053181618452,
+            -0.009427007287740707,
+            0.003249816130846739,
+            -0.015227040275931358,
+            -0.001274131122045219,
+            0.00532321073114872,
+            0.008543713949620724,
+            -0.007746404968202114,
+            -0.030469713732600212,
+            -0.004744770936667919,
+            0.016759123653173447,
+            -0.00835611205548048,
+            -0.019354287534952164,
+            0.024716582149267197,
+            -0.0004824404022656381,
+            -0.021699313074350357,
+            0.0041155219078063965,
+            -0.010544802993535995,
+            -0.007363383658230305,
+            -0.005518629681318998,
+            -0.008105975575745106,
+            -0.02424757555127144,
+            0.013843473978340626,
+            0.0019102196674793959,
+            -0.02046426571905613,
+            0.010232132859528065,
+            -0.01378094032406807,
+            0.004111613612622023,
+            0.008551530539989471,
+            -0.008895467966794968,
+            -0.011748583987355232,
+            0.027749482542276382,
+            0.022324655205011368,
+            -0.017618967220187187,
+            -0.015742946416139603,
+            9.386213787365705e-05,
+            -0.025060519576072693,
+            0.023700403049588203,
+            -0.00470959534868598,
+            0.0019825247582048178,
+            -0.011131060309708118,
+            0.011600065045058727,
+            0.010826206766068935,
+            0.012624060735106468,
+            -0.003554669674485922,
+            -0.0005579209537245333,
+            0.026514435186982155,
+            -0.009090886451303959,
+            0.003509723348543048,
+            -8.564843483327422e-06,
+            -0.012413008138537407,
+            0.02384110540151596,
+            -0.004486817866563797,
+            0.023950539529323578,
+            -0.006237770896404982,
+            -0.0006483022007159889,
+            -0.014961270615458488,
+            0.0026440175715833902,
+            -0.0031462442129850388,
+            -0.0069842711091041565,
+            -0.010286850854754448,
+            0.01283511333167553,
+            0.01213942188769579,
+            0.006835752632468939,
+            -0.010255583561956882,
+            0.0012184367515146732,
+            0.00038790650432929397,
+            -0.016696590930223465,
+            0.0023489349987357855,
+            -0.019713858142495155,
+            0.0009829570772126317,
+            -0.03329937905073166,
+            0.009708410128951073,
+            0.04430537298321724,
+            0.016634056344628334,
+            -0.008129426278173923,
+            -0.003537081880494952,
+            -0.01611815020442009,
+            0.005225501023232937,
+            -0.005706231575459242,
+            -0.011725133284926414,
+            -0.0136793227866292,
+            -0.01605561561882496,
+            0.0035312194377183914,
+            0.017916003242135048,
+            -0.0026518343947827816,
+            -0.020745670422911644,
+            0.02298126183450222,
+            -0.014500081539154053,
+            -0.026655137538909912,
+            -0.01754079945385456,
+            -0.02096453867852688,
+            0.007324300240725279,
+            0.020745670422911644,
+            -0.00579612422734499,
+            0.026576969772577286,
+            0.012428641319274902,
+            0.0028140321373939514,
+            -0.0010904374066740274,
+            0.01813487336039543,
+            -0.011349929496645927,
+            0.00452590174973011,
+            -0.0013816115679219365,
+            -0.020354831591248512,
+            0.0014783439692109823,
+            -0.008715682663023472,
+            -0.005600705277174711,
+            0.001242864178493619,
+            -0.005624155513942242,
+            -0.00260297954082489,
+            -0.0075978864915668964,
+            -0.017728401347994804,
+            0.02327829785645008,
+            -0.02348153479397297,
+            -0.002851161640137434,
+            -0.0005007609142921865,
+            -0.009591159410774708,
+            0.0111076096072793,
+            0.023950539529323578,
+            0.01261624414473772,
+            -0.015430275350809097,
+            0.01833810843527317,
+            -0.00037154016899876297,
+            0.026545701548457146,
+            -0.004662694875150919,
+            0.005585072096437216,
+            0.012694410979747772,
+            0.0008129425696097314,
+            0.01685292460024357,
+            -0.00710543105378747,
+            -0.011029441840946674,
+            0.0042913989163935184,
+            -0.00018686932162381709,
+            -0.021402277052402496,
+            -0.009755311533808708,
+            -0.0023782479111105204,
+            0.012217588722705841,
+            -0.017165595665574074,
+            0.015727313235402107,
+            -0.012741311453282833,
+            -0.001535992487333715,
+            -0.00387124833650887,
+            0.01397635880857706,
+            -0.04855768755078316,
+            -0.004963639657944441,
+            -0.008199776522815228,
+            0.03489399701356888,
+            -0.012444275431334972,
+            0.022434089332818985,
+            0.0040002246387302876,
+            -0.007621336728334427,
+            0.011834568344056606,
+            -0.010130515322089195,
+            -0.0004521504743024707,
+            0.015070704743266106,
+            -0.007347750477492809,
+            0.0017431365558877587,
+            0.016993626952171326,
+            -0.01836937665939331,
+            0.01674349047243595,
+            -0.019541889429092407,
+            0.0031071603298187256,
+            -0.012061254121363163,
+            -0.00640973960980773,
+            0.021386643871665,
+            0.0267020370811224,
+            -0.005815666168928146,
+            0.007398559246212244,
+            -0.009184688329696655,
+            0.007691687438637018,
+            -0.004252315033227205,
+            0.011545347981154919,
+            0.0010748038766905665,
+            -0.010232132859528065,
+            -0.0019317157566547394,
+            -0.009661509655416012,
+            0.020651869475841522,
+            -0.013382285833358765,
+            -0.012491175904870033,
+            -0.009559892117977142,
+            -0.004803396295756102,
+            -0.03351824730634689,
+            -0.005983726587146521,
+            -0.008137242868542671,
+            -0.012483359314501286,
+            -0.008426462300121784,
+            0.012733494862914085,
+            0.0010884832590818405,
+            0.0024290569126605988,
+            -0.009544258937239647,
+            -0.01144373044371605,
+            -0.013866924680769444,
+            0.015977447852492332,
+            0.004052987787872553,
+            -0.004111613612622023,
+            0.0038380271289497614,
+            0.00037056306609883904,
+            0.004795579705387354,
+            -0.00015438091941177845,
+            -0.00807470828294754,
+            -0.0032830373384058475,
+            0.012381740845739841,
+            0.006398014724254608,
+            -0.021011440083384514,
+            0.010388468392193317,
+            0.01893218234181404,
+            -0.0033572965767234564,
+            -0.006057985592633486,
+            -0.0064605483785271645,
+            -0.013382285833358765,
+            -0.004557168576866388,
+            0.0017665867926552892,
+            0.020917639136314392,
+            -0.01661842316389084,
+            -0.0013845429057255387,
+            0.019541889429092407,
+            -0.0027749482542276382,
+            0.023028161376714706,
+            0.0025384915061295033,
+            0.009935096837580204,
+            0.00988037884235382,
+            0.011740767396986485,
+            0.0062924884259700775,
+            -0.010951275005936623,
+            -0.02162114717066288,
+            0.001587778446264565,
+            -0.022434089332818985,
+            0.01588364690542221,
+            0.0023333015851676464,
+            -0.005100433249026537,
+            -0.021840015426278114,
+            -0.012506809085607529,
+            0.009512991644442081,
+            -0.02971930429339409,
+            -0.0007259811973199248,
+            -0.00737120071426034,
+            0.014265579171478748,
+            -0.007879289798438549,
+            0.0033866094890981913,
+            0.006042351946234703,
+            -0.007566619664430618,
+            0.00028799858409911394,
+            -0.0024505530018359423,
+            -0.01800980605185032,
+            -0.0045962524600327015,
+            0.00026381551288068295,
+            0.010372835211455822,
+            0.0038575688377022743,
+            0.015539710409939289,
+            0.023231398314237595,
+            0.031689126044511795,
+            -0.009653693065047264,
+            0.0018340062815696,
+            0.0009956592693924904,
+            0.016806025058031082,
+            0.005616338923573494,
+            0.0006727295112796128,
+            0.001513519324362278,
+            0.02510741911828518,
+            -0.011670416221022606,
+            -0.015899281948804855,
+            0.01596963219344616,
+            -0.020182862877845764,
+            -0.00010515979374758899,
+            -0.02029229700565338,
+            0.012084703892469406,
+            0.0011334295850247145,
+            0.013335385359823704,
+            0.013796573504805565,
+            -0.001073826802894473,
+            0.00819195993244648,
+            0.024794749915599823,
+            -0.005995451472699642,
+            0.0027358646038919687,
+            0.027233576402068138,
+            -0.00872349925339222,
+            -0.004779946058988571,
+            -0.000544730166438967,
+            0.002515041036531329,
+            0.00041257814154960215,
+            0.010599520988762379,
+            0.017822202295064926,
+            -0.027608782052993774,
+            -0.006034535355865955,
+            0.01240519154816866,
+            0.001964936964213848,
+            -0.006558258086442947,
+            0.004705687053501606,
+            0.0033553424291312695,
+            0.014851835556328297,
+            0.0003659218782559037,
+            0.01688419282436371,
+            -0.004819029942154884,
+            -0.02543572336435318,
+            -0.00930193904787302,
+            0.005835208110511303,
+            -0.012506809085607529,
+            -0.0032263658940792084,
+            -0.0034999523777514696,
+            -0.009512991644442081,
+            0.014265579171478748,
+            -0.008285760879516602,
+            0.0012340702814981341,
+            0.009395739994943142,
+            0.003802851540967822,
+            -0.007418101187795401,
+            -0.00215156190097332,
+            0.011881468817591667,
+            -0.012678777799010277,
+            0.004752587527036667,
+            0.022309020161628723,
+            0.003931828308850527,
+            -0.016821658238768578,
+            -0.018025439232587814,
+            0.006323755253106356,
+            0.03017267771065235,
+            -0.012194138951599598,
+            -0.0028765660244971514,
+            0.018916549161076546,
+            -0.0006644242093898356,
+            0.008215410634875298,
+            -0.002986000617966056,
+            -0.015023804269731045,
+            -0.03773929551243782,
+            -0.0019072884460911155,
+            -0.0018975174752995372,
+            0.005268493201583624,
+            -0.0003336777735967189,
+            -0.02374730445444584,
+            -0.007199232000857592,
+            -0.0103650176897645,
+            -0.002759314840659499,
+            -0.011021625250577927,
+            0.002632292453199625,
+            0.0008007289143279195,
+            -0.023684769868850708,
+            0.014031076803803444,
+            -0.012319207191467285,
+            0.010888740420341492,
+            -0.006100977770984173,
+            0.0190728846937418,
+            -0.0025169954169541597,
+            -0.0015457634581252933,
+            -0.008950185030698776,
+            0.006175236776471138,
+            0.013788756914436817,
+            0.0068513862788677216,
+            -0.011998720467090607,
+            0.0012467725900933146,
+            0.022856194525957108,
+            0.011818934231996536,
+            0.006179145537316799,
+            0.006683326326310635,
+            -0.011310845613479614,
+            -0.017884736880660057,
+            0.0020108604803681374,
+            -0.010451002046465874,
+            0.0033260295167565346,
+            0.007781580090522766,
+            0.014851835556328297,
+            0.005663239397108555,
+            -0.013694955967366695,
+            0.004760404117405415,
+            0.0103650176897645,
+            0.01694672554731369,
+            -0.014593882486224174,
+            -0.0034725936129689217,
+            -0.00925503857433796,
+            0.012483359314501286,
+            0.01916668564081192,
+            0.0042366813868284225,
+            0.006147878244519234,
+            0.004092071671038866,
+            0.015774212777614594,
+            -0.004228864796459675,
+            0.008332661353051662,
+            -0.013741856440901756,
+            -0.016493353992700577,
+            -0.019182318821549416,
+            0.004932372830808163,
+            0.00021899129205849022,
+            0.02745244652032852,
+            0.004557168576866388,
+            0.01976075768470764,
+            0.007457185070961714,
+            0.012749128974974155,
+            0.0020812111906707287,
+            0.022434089332818985,
+            -0.0009277511853724718,
+            -0.024153774604201317,
+            0.01206907071173191,
+            -0.021871281787753105,
+            -0.03070421703159809,
+            0.006843569688498974,
+            0.006612975150346756,
+            0.0381457693874836,
+            -0.022590424865484238,
+            0.011834568344056606,
+            -0.003156015183776617,
+            -0.023387733846902847,
+            -0.0022844469640403986,
+            -0.001389428274706006,
+            -0.0038380271289497614,
+            0.026358099654316902,
+            -0.0005198142607696354,
+            0.006315938662737608,
+            0.003521448466926813,
+            0.009810028597712517,
+            -0.00983347836881876,
+            0.021761847659945488,
+            0.01632138527929783,
+            0.018150506541132927,
+            0.005788307636976242,
+            -0.01112324371933937,
+            0.016993626952171326,
+            0.0010904374066740274,
+            -0.004955823067575693,
+            0.004568893928080797,
+            -0.005022265482693911,
+            0.0028257572557777166,
+            -0.0005564552848227322,
+            0.01474240142852068,
+            0.008707866072654724,
+            -0.015133239328861237,
+            0.010036714375019073,
+            -0.004568893928080797,
+            0.007898831740021706,
+            0.01949498802423477,
+            0.011975269764661789,
+            -0.002401698147878051,
+            0.009614609181880951,
+            -0.04430537298321724,
+            0.006140061654150486,
+            -0.03339318186044693,
+            -0.01233484037220478,
+            0.00270264339633286,
+            -0.017400098964571953,
+            0.0021105241030454636,
+            -0.01042755227535963,
+            -0.0175720676779747,
+            -0.006859203334897757,
+            0.019307386130094528,
+            0.01877584680914879,
+            0.004092071671038866,
+            0.00378917227499187,
+            0.010443185456097126,
+            -0.010873107239603996,
+            -0.013812207616865635,
+            -0.01394509244710207,
+            -0.012944547459483147,
+            -0.02049553394317627,
+            -0.0032556788064539433,
+            0.0009736746433191001,
+            -0.006441006902605295,
+            -0.002018677070736885,
+            0.009129970334470272,
+            -0.022043250501155853,
+            -0.010974724777042866,
+            0.014109243638813496,
+            -0.003392471931874752,
+            0.0006732181063853204,
+            -0.019119784235954285,
+            0.004256223328411579,
+            -0.003243953688070178,
+            0.00798481609672308,
+            -0.00581957446411252,
+            0.02431011013686657,
+            -0.01382784079760313,
+            -0.005053532775491476,
+            0.016821658238768578,
+            0.005491270683705807,
+            -0.007097613997757435,
+            -0.013546437956392765,
+            -0.03439372405409813,
+            0.025826560333371162,
+            -0.009770944714546204,
+            -0.01727502979338169,
+            -0.013288484886288643,
+            0.014984720386564732,
+            -0.009333206340670586,
+            -0.0018574565183371305,
+            0.011005992069840431,
+            0.0014988628681749105,
+            0.019713858142495155,
+            0.024826016277074814,
+            -0.010349384509027004,
+            0.0014206953346729279,
+            -0.01542245876044035,
+            0.012569342739880085,
+            -0.024700947105884552,
+            -0.009966363199055195,
+            0.013874741271138191,
+            0.012029986828565598,
+            0.002149607753381133,
+            -0.01661842316389084,
+            0.009661509655416012,
+            0.008363928645849228,
+            0.013906008563935757,
+            0.016993626952171326,
+            0.005467820446938276,
+            -0.010873107239603996,
+            -0.007914464920759201,
+            -0.024951083585619926,
+            0.0240443404763937,
+            0.005909467115998268,
+            -0.010974724777042866,
+            0.04368003085255623,
+            0.01993272639811039,
+            0.011998720467090607,
+            -0.01783783733844757,
+            -0.010552620515227318,
+            0.01664968952536583,
+            0.01218632236123085,
+            0.010802756063640118,
+            -0.00024036523245740682,
+            0.004932372830808163,
+            -0.0024603239726275206,
+            0.022168319672346115,
+            0.018213041126728058,
+            0.010185232385993004,
+            -0.05137171968817711,
+            0.014976903796195984,
+            0.002907833084464073,
+            -0.00520595908164978,
+            0.01685292460024357,
+            0.019948361441493034,
+            0.01217068824917078,
+            0.009606792591512203,
+            0.019854558631777763,
+            -0.00018735785852186382,
+            0.02194944955408573,
+            -0.009083069860935211,
+            -0.015758579596877098,
+            -0.00031950988341122866,
+            -0.013804391026496887,
+            -0.002776902401819825,
+            -0.019291752949357033,
+            0.009903829544782639,
+            0.00450245151296258,
+            -0.005237226374447346,
+            0.0240443404763937,
+            0.004209322854876518,
+            -0.029078330844640732,
+            -0.007656512316316366,
+            -0.029031431302428246,
+            0.0013796574203297496,
+            -0.019854558631777763,
+            -0.003818485187366605,
+            -0.014859652146697044,
+            -0.014421914704144001,
+            -0.026842739433050156,
+            -0.012241039425134659,
+            -0.00030363211408257484,
+            0.015023804269731045,
+            -0.001865273341536522,
+            -0.015633512288331985,
+            0.004529810044914484,
+            -0.03839590400457382,
+            -0.008582797832787037,
+            0.03645734861493111,
+            0.00819195993244648,
+            0.031954895704984665,
+            -0.008614065125584602,
+            0.01446099765598774,
+            -0.0010142240207642317,
+            -0.04261695221066475,
+            -0.0031657861545681953,
+            -0.003800897393375635,
+            0.00893455184996128,
+            0.007488451898097992,
+            0.0048932889476418495,
+            -0.012147238478064537,
+            0.016368286684155464,
+            -0.0014314433792605996,
+            0.00898926891386509,
+            0.0030524430330842733,
+            0.005151242017745972,
+            -0.010966908186674118,
+            0.010521353222429752,
+            -0.016368286684155464,
+            0.004248406738042831,
+            0.0019346470944583416,
+            0.004174147732555866,
+            -0.016868559643626213,
+            -0.016430821269750595,
+            -0.0037051422987133265,
+            -0.004576710518449545,
+            -0.012889830395579338,
+            -0.002970367204397917,
+            -0.011670416221022606,
+            -0.006655967328697443,
+            -0.008856384083628654,
+            -0.01431247964501381,
+            -0.011146693490445614,
+            0.0049362811259925365,
+            0.0027573604602366686,
+            0.008840750902891159,
+            0.014234311878681183,
+            0.003171648597344756,
+            0.007961365394294262,
+            0.02573275938630104,
+            0.001182284322567284,
+            -0.001272176974453032,
+            -0.01671222411096096,
+            0.001556511502712965,
+            -0.030157042667269707,
+            0.014453181065618992,
+            0.011748583987355232,
+            -0.025748392567038536,
+            0.006093161180615425,
+            -1.3885428415960632e-05,
+            0.0028570243157446384,
+            0.002175012370571494,
+            -0.0007220727857202291,
+            -0.014062343165278435,
+            0.003333846339955926,
+            -0.0180723387748003,
+            -0.00032488390570506454,
+            0.012467725202441216,
+            0.0401155911386013,
+            3.362579036547686e-06,
+            -0.0008657057187519968,
+            0.014429731294512749,
+            0.007707321085035801,
+            0.022074518725275993,
+            -0.01596963219344616,
+            -0.007015538401901722,
+            -0.03129829093813896,
+            -0.005342752672731876,
+            -0.005925100762397051,
+            -0.02198071777820587,
+            0.012241039425134659,
+            -0.0012545892968773842,
+            -0.010341567918658257,
+            -0.01366368867456913,
+            -0.02238718792796135,
+            0.012217588722705841,
+            0.02241845615208149,
+            0.008825116790831089,
+            0.007183598354458809,
+            0.008707866072654724,
+            -0.024450812488794327,
+            -0.020808203145861626,
+            0.029562970623373985,
+            0.013616788201034069,
+            -0.009231588803231716,
+            -0.01664968952536583,
+            -0.013585521839559078,
+            0.0031247481238096952,
+            -0.0006238748319447041,
+            0.0004951425944454968,
+            0.06219010800123215,
+            -0.018760213628411293,
+            0.0021847831085324287,
+            -0.034768927842378616,
+            -0.000998102012090385,
+            -0.009208138100802898,
+            0.013179049827158451,
+            0.005135608371347189,
+            -0.030188310891389847,
+            0.01618068479001522,
+            -0.030626049265265465,
+            -0.007257857825607061,
+            0.0015750762540847063,
+            -0.008137242868542671,
+            -0.010615154169499874,
+            -0.0023919271770864725,
+            -0.01827557571232319,
+            0.00013215202488936484,
+            -0.013327568769454956,
+            0.0021144323982298374,
+            0.002075348747894168,
+            -0.024700947105884552,
+            0.0015144963981583714,
+            0.006120519712567329,
+            -0.0046314275823533535,
+            -0.004350024741142988,
+            -0.019510623067617416,
+            -0.02019849605858326,
+            -0.014156145043671131,
+            -0.002032356569543481,
+            0.018025439232587814,
+            -0.02424757555127144,
+            0.01697799377143383,
+            -0.0032087783329188824,
+            0.004721320234239101,
+            0.003218549070879817,
+            0.008598431013524532,
+            0.018119240179657936,
+            0.01336665265262127,
+            0.0007044851081445813,
+            -0.010466636158525944,
+            0.01797853782773018,
+            -0.017947271466255188,
+            -0.02188691683113575,
+            0.001725548878312111,
+            0.0034081055782735348,
+            4.7083740355446935e-05,
+            0.0025932088028639555,
+            0.0045962524600327015,
+            0.013569887727499008,
+            0.007500177249312401,
+            -0.016196317970752716,
+            -0.002585391979664564,
+            -0.0027827650774270296,
+            0.029828740283846855,
+            0.0024603239726275206,
+            0.03276783972978592,
+            0.0012545892968773842,
+            0.015782030299305916,
+            -0.0014167869230732322,
+            -0.014570432715117931,
+            -0.017431365326046944,
+            -0.01314778346568346,
+            0.018728947266936302,
+            0.0016972130397334695,
+            0.007629153784364462,
+            -0.023700403049588203,
+            -0.01596963219344616,
+            0.003118885448202491,
+            -0.004049079492688179,
+            -0.01704052835702896,
+            0.0180723387748003,
+            -0.0177440345287323,
+            0.011521898210048676,
+            0.00032415110035799444,
+            -0.015023804269731045,
+            -0.017525166273117065,
+            0.014367196708917618,
+            0.0016923275543376803,
+            -0.000899904000107199,
+            0.012076887302100658,
+            -0.013257217593491077,
+            -0.011826751753687859,
+            0.002767131431028247,
+            0.00219259993173182,
+            -0.006824027746915817,
+            0.013616788201034069,
+            -0.016430821269750595,
+            0.005604614038020372,
+            -0.006824027746915817,
+            -0.0017802660586312413,
+            -0.010849656537175179,
+            0.013921641744673252,
+            0.012921096757054329,
+            -0.0015057026175782084,
+            0.012108154594898224,
+            0.012053437530994415,
+            0.003245907835662365,
+            0.014703317545354366,
+            0.005835208110511303,
+            -0.006378472782671452,
+            -0.010028897784650326,
+            0.0021124782506376505,
+            0.008676598779857159,
+            -0.004088163375854492,
+            -0.00383021030575037,
+            0.033080510795116425,
+            0.01668095588684082,
+            -0.017759669572114944,
+            0.021496077999472618,
+            -0.0038946985732764006,
+            0.007179690059274435,
+            -0.03186109662055969,
+            0.0004763335455209017,
+            0.013655872084200382,
+            -0.010873107239603996,
+            0.008223227225244045,
+            0.007816756144165993,
+            0.028171587735414505,
+            0.012194138951599598,
+            0.00819195993244648,
+            -0.0050769830122590065,
+            0.0021105241030454636,
+            0.013186866417527199,
+            0.0027964443434029818,
+            -0.002470094710588455,
+            -0.0026967807207256556,
+            0.01996399462223053,
+            -0.016337020322680473,
+            0.020370464771986008,
+            0.013335385359823704,
+            -0.029375368729233742,
+            0.005217684432864189,
+            -0.019010350108146667,
+            0.003961140755563974,
+            -0.006319846957921982,
+            -0.0037168674170970917,
+            -0.021386643871665,
+            0.007339933421462774,
+            0.02945353463292122,
+            -0.011772033758461475,
+            0.029734939336776733,
+            0.004326574504375458,
+            0.02016722969710827,
+            -0.009450457990169525,
+            -0.006198687478899956,
+            -0.009208138100802898,
+            0.0010054302401840687,
+            -0.008371745236217976,
+            0.011154510080814362,
+            0.0002304721565451473,
+            0.005846933461725712,
+            0.01424212846904993,
+            0.004060804378241301,
+            -0.003523402614519,
+            0.028828194364905357,
+            0.005925100762397051,
+            0.008231043815612793,
+            -0.013007081113755703,
+            0.005159058608114719,
+            -0.007863656617701054,
+            0.0038243476301431656,
+            0.0019854558631777763,
+            -0.008637514896690845,
+            9.306825086241588e-05,
+            -0.0021730579901486635,
+            0.013585521839559078,
+            0.009614609181880951,
+            0.013960725627839565,
+            -0.004678328521549702,
+            -0.008121608756482601,
+            0.025685859844088554,
+            0.001968845259398222,
+            -0.004310940857976675,
+            0.004729137290269136,
+            0.004881564062088728,
+            0.016931092366576195,
+            0.012960180640220642,
+            0.0006043329485692084,
+            -0.0021574245765805244,
+            -0.025185586884617805,
+            0.01212378777563572,
+            -0.018353741616010666,
+            -0.00039841028046794236,
+            -0.005698414985090494,
+            -0.022731125354766846,
+            0.013053981587290764,
+            -0.005248951260000467,
+            -0.003607432823628187,
+            -0.005542079918086529,
+            0.022965628653764725,
+            0.004064713139086962,
+            -0.014343746937811375,
+            -0.00399240804836154,
+            0.007468909956514835,
+            0.02765568159520626,
+            -0.0022121418733149767,
+            -0.00042381472303532064,
+            -0.010622970759868622,
+            -0.015289573930203915,
+            0.012749128974974155,
+            -0.015742946416139603,
+            -0.013624604791402817,
+            0.003101297887042165
+          ],
+          "title": "DeepSeekMoE: Achieving Ultimate Expert Specialization in Mixture-of-Experts Architectures",
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "Mixture-of-Experts (MoE)",
+            "expert specialization",
+            "fine-grained expert segmentation",
+            "shared expert isolation"
+          ]
+        },
+        "type": "chunk"
+      },
+      "target": {
+        "id": "c6bf0f92-84c8-44b9-8b28-7fcab6058203",
+        "properties": {
+          "page_content": "8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n",
+          "keyphrases": [
+            "Mixture of Experts (MoE)",
+            "language model training",
+            "DeepSeekMoE architecture",
+            "expert-choice routing strategy",
+            "training instability"
+          ],
+          "title": "Related Work",
+          "embedding": [
+            -0.0004454151203390211,
+            -0.01349730882793665,
+            -0.018033159896731377,
+            0.005232766270637512,
+            0.01874188706278801,
+            0.010268663987517357,
+            -0.011331754736602306,
+            0.03266443312168121,
+            -0.018993878737092018,
+            0.018789134919643402,
+            -0.005878495052456856,
+            -0.009221323765814304,
+            0.007355009205639362,
+            -0.008567719720304012,
+            0.0011782583314925432,
+            0.02784508839249611,
+            0.018474144861102104,
+            0.015410871244966984,
+            0.002397859003394842,
+            -0.03512135148048401,
+            0.019734105095267296,
+            -0.02189178392291069,
+            -0.044067058712244034,
+            -0.013772924430668354,
+            0.014253283850848675,
+            0.01693069562315941,
+            0.03858623653650284,
+            -0.006386416032910347,
+            -0.05959605053067207,
+            0.03386139124631882,
+            0.020521579310297966,
+            0.02291550114750862,
+            -0.01582823134958744,
+            -0.014930511824786663,
+            -0.02899480052292347,
+            -0.013481559231877327,
+            0.05077634006738663,
+            0.021734289824962616,
+            -0.016410963609814644,
+            0.027687594294548035,
+            0.0467129722237587,
+            0.013268941082060337,
+            -0.02280525304377079,
+            -0.0008283244678750634,
+            0.005512319505214691,
+            0.0011831800220534205,
+            -0.019009627401828766,
+            -0.016332216560840607,
+            0.010497031733393669,
+            -0.028175828978419304,
+            -0.018883632495999336,
+            -0.027278108522295952,
+            -0.006665969267487526,
+            0.0006486818892881274,
+            -0.0642578974366188,
+            0.007378633599728346,
+            0.01752917654812336,
+            0.03389289230108261,
+            0.05058734491467476,
+            0.0015080131124705076,
+            0.012308222241699696,
+            -0.019419115036725998,
+            -0.01686769910156727,
+            -0.008260604925453663,
+            -0.00022885970247443765,
+            -0.01097739115357399,
+            0.012016857042908669,
+            -0.013324064202606678,
+            0.01982860080897808,
+            0.016221968457102776,
+            -0.040854163467884064,
+            0.027026114985346794,
+            -0.023876218125224113,
+            0.009426066651940346,
+            -0.006969146896153688,
+            0.007095142733305693,
+            0.019135624170303345,
+            0.026600880548357964,
+            -0.0019509674748405814,
+            -0.007252637762576342,
+            0.0025514166336506605,
+            0.017324432730674744,
+            -0.007122704293578863,
+            0.0022108338307589293,
+            0.05014635995030403,
+            -0.04687046632170677,
+            -0.02584490366280079,
+            -0.07049469649791718,
+            0.00882758665829897,
+            0.00037503460771404207,
+            -0.024789689108729362,
+            -0.003289673710241914,
+            0.028490817174315453,
+            0.008835460990667343,
+            -0.040602169930934906,
+            0.06312393397092819,
+            -0.014056415297091007,
+            -0.015662861987948418,
+            -0.03512135148048401,
+            -0.008355101570487022,
+            0.0005320372874848545,
+            -0.0049768369644880295,
+            0.011575871147215366,
+            0.010709649883210659,
+            0.012197976000607014,
+            -0.012812205590307713,
+            -0.033199913799762726,
+            0.04687046632170677,
+            -0.005713125690817833,
+            0.08523621410131454,
+            0.0021379925310611725,
+            -0.009945799596607685,
+            -0.003252268536016345,
+            0.03537334129214287,
+            -0.021435048431158066,
+            -0.008386600762605667,
+            -0.028349071741104126,
+            -0.04935888573527336,
+            0.024852687492966652,
+            -0.02348248101770878,
+            0.018631640821695328,
+            -0.0007800916791893542,
+            0.016395213082432747,
+            -0.019214371219277382,
+            -0.0721326395869255,
+            0.05739112198352814,
+            -0.0026774124708026648,
+            -0.015111630782485008,
+            -0.003021932439878583,
+            0.007201451808214188,
+            -0.0008194653782993555,
+            0.0037345965392887592,
+            -0.01735593192279339,
+            -0.019088376313447952,
+            0.029136545956134796,
+            0.023513980209827423,
+            0.03861773759126663,
+            0.046240486204624176,
+            -0.020301084965467453,
+            -0.01452889945358038,
+            0.023734472692012787,
+            -0.029514534398913383,
+            0.006630532909184694,
+            0.011567996814846992,
+            -0.01746617816388607,
+            0.011599495075643063,
+            0.011662493459880352,
+            0.06967572122812271,
+            -0.019860100001096725,
+            0.017009442672133446,
+            0.0022895813453942537,
+            0.0643208920955658,
+            0.0025435418356209993,
+            0.030475253239274025,
+            -0.014765141531825066,
+            0.02178153768181801,
+            -0.028160078451037407,
+            0.0025061366613954306,
+            -0.020222337916493416,
+            -0.003464886685833335,
+            -0.002510074060410261,
+            -0.04094865918159485,
+            -0.002821126487106085,
+            0.028853056952357292,
+            0.0015119505114853382,
+            0.01776541955769062,
+            0.033199913799762726,
+            0.003850748995319009,
+            -0.05962754786014557,
+            0.023198990151286125,
+            0.03452287241816521,
+            0.010985265485942364,
+            -0.009103202261030674,
+            0.021624041721224785,
+            0.011056138202548027,
+            0.010575778782367706,
+            0.02067907340824604,
+            0.008189732208848,
+            -0.022710757330060005,
+            0.03972020000219345,
+            -0.02886880561709404,
+            0.0018830477492883801,
+            0.03228644281625748,
+            0.0016743671149015427,
+            0.044791534543037415,
+            0.012906703166663647,
+            0.026852872222661972,
+            0.01693069562315941,
+            0.012316097505390644,
+            0.029939770698547363,
+            0.007166015449911356,
+            0.010536405257880688,
+            0.03540484234690666,
+            -0.015095881186425686,
+            0.008158233016729355,
+            -0.02255326136946678,
+            -0.012867328710854053,
+            0.006488787941634655,
+            0.00988280214369297,
+            -0.005815497133880854,
+            -0.0021753974724560976,
+            -0.04201962426304817,
+            0.02212802693247795,
+            -0.0055831922218203545,
+            0.021986281499266624,
+            0.0026144145522266626,
+            0.012615337036550045,
+            0.007630625274032354,
+            0.03559383377432823,
+            -0.03852323815226555,
+            -0.011788489297032356,
+            -0.007878679782152176,
+            0.005535943899303675,
+            -0.044791534543037415,
+            -0.020033344626426697,
+            0.016725953668355942,
+            -0.004776031244546175,
+            0.021135808899998665,
+            0.009276446886360645,
+            -0.02159254439175129,
+            -0.006626595743000507,
+            -0.006445476785302162,
+            0.018363898620009422,
+            -0.01533999852836132,
+            -0.030491001904010773,
+            -0.0351843498647213,
+            0.01300907414406538,
+            0.011536497622728348,
+            0.03590882569551468,
+            -0.0009178996551781893,
+            -0.020600326359272003,
+            0.014788766391575336,
+            0.001092620543204248,
+            0.0039216214790940285,
+            0.02274225652217865,
+            -0.04646097868680954,
+            0.00845747347921133,
+            0.0026537880767136812,
+            0.011520748026669025,
+            0.02085231803357601,
+            0.007394383195787668,
+            0.017182687297463417,
+            -0.035751331597566605,
+            0.004086991306394339,
+            0.03688529506325722,
+            0.011638869531452656,
+            -0.016103848814964294,
+            0.028065582737326622,
+            4.574118065647781e-05,
+            -0.012883078306913376,
+            -0.0033290472347289324,
+            -0.03222344443202019,
+            -0.012623212300240993,
+            0.0008617921266704798,
+            -0.016757450997829437,
+            0.05269777774810791,
+            0.043090589344501495,
+            -0.019324617460370064,
+            -0.014631271362304688,
+            -0.05918656289577484,
+            0.039688702672719955,
+            -0.02408096194267273,
+            -0.025262173265218735,
+            0.04416155442595482,
+            -0.016725953668355942,
+            0.020159341394901276,
+            0.032475437968969345,
+            0.012820080853998661,
+            -0.00011873635230585933,
+            -0.03527884557843208,
+            -0.004051554948091507,
+            -0.04523251950740814,
+            -0.03801925480365753,
+            0.005126457195729017,
+            0.03170371428132057,
+            -0.03559383377432823,
+            -0.0055792550556361675,
+            -0.009481189772486687,
+            0.030191762372851372,
+            0.028837306424975395,
+            0.02049008011817932,
+            -0.0016379464650526643,
+            0.007969239726662636,
+            0.016379464417696,
+            -0.03208170086145401,
+            0.013394936919212341,
+            -0.0046460977755486965,
+            -0.026522131636738777,
+            -0.010308037512004375,
+            -0.015749484300613403,
+            -0.006760466378182173,
+            -4.174228524789214e-05,
+            -0.04693346470594406,
+            0.00440985569730401,
+            0.03540484234690666,
+            -0.0037050663959234953,
+            -0.01795441284775734,
+            0.03323141112923622,
+            0.033451907336711884,
+            -0.027860838919878006,
+            -0.06167498230934143,
+            -0.001828908920288086,
+            0.021450798958539963,
+            -0.0024529823567718267,
+            -0.05616266280412674,
+            -0.003445199690759182,
+            -0.008276354521512985,
+            0.030554000288248062,
+            -0.01710394024848938,
+            -0.04201962426304817,
+            0.0032680181320756674,
+            -0.0005128426128067076,
+            -0.013481559231877327,
+            -0.03433387726545334,
+            -0.00764243770390749,
+            0.034365374594926834,
+            0.01437140442430973,
+            0.025813406333327293,
+            -0.04384656623005867,
+            -0.02929404191672802,
+            0.01270983461290598,
+            0.0394052118062973,
+            -0.06284044682979584,
+            0.010308037512004375,
+            -0.014292657375335693,
+            -0.001888953847810626,
+            0.004728782922029495,
+            0.043878063559532166,
+            -0.03291642293334007,
+            0.009016579948365688,
+            -0.00609505083411932,
+            -0.04167313501238823,
+            -0.034963857382535934,
+            -0.010827770456671715,
+            -0.00495715020224452,
+            -0.016332216560840607,
+            -0.02297849766910076,
+            0.002760097151622176,
+            -0.014269033446907997,
+            0.003340859431773424,
+            -0.02165554091334343,
+            0.004846903961151838,
+            -0.017859915271401405,
+            0.021734289824962616,
+            0.010056045837700367,
+            -0.013938293792307377,
+            -0.025703158229589462,
+            -0.004941401071846485,
+            0.02097831293940544,
+            0.012670460157096386,
+            0.00115561846178025,
+            -0.0034688240848481655,
+            0.002754191169515252,
+            -0.003376295790076256,
+            -0.009693807922303677,
+            -0.06545485556125641,
+            -0.017261436209082603,
+            -0.017607923597097397,
+            0.015552615746855736,
+            0.016056599095463753,
+            0.010725399479269981,
+            -0.046901967376470566,
+            -0.05118582397699356,
+            0.028538066893815994,
+            -0.03549933806061745,
+            -0.001561167649924755,
+            -0.0025927589740604162,
+            -0.0015926666092127562,
+            -0.004386231303215027,
+            0.016568457707762718,
+            -0.005606816615909338,
+            0.03360940143465996,
+            -0.012016857042908669,
+            0.007650312036275864,
+            0.004480728413909674,
+            0.020269587635993958,
+            0.0333574078977108,
+            -0.011103386990725994,
+            -0.011048263870179653,
+            0.049106892198324203,
+            -0.013631178997457027,
+            0.025088928639888763,
+            0.005638315342366695,
+            0.001570026739500463,
+            0.0033861391711980104,
+            0.02382897026836872,
+            0.014198160730302334,
+            0.005350887309759855,
+            -0.03713728487491608,
+            0.027089113369584084,
+            0.010363160632550716,
+            -0.060919005423784256,
+            -0.015088006854057312,
+            0.006595096550881863,
+            0.017418930307030678,
+            0.016536958515644073,
+            -0.003559383563697338,
+            -0.014662770554423332,
+            -0.013457934372127056,
+            -0.008481097407639027,
+            -0.018521394580602646,
+            0.036412809044122696,
+            -0.0033054230734705925,
+            0.010725399479269981,
+            -0.027215110138058662,
+            0.0045752255246043205,
+            0.01328469067811966,
+            0.016206219792366028,
+            0.02942003682255745,
+            -0.015922728925943375,
+            -0.015017134137451649,
+            -0.04718545451760292,
+            -0.03288492560386658,
+            -0.00928432121872902,
+            0.013245316222310066,
+            0.030128763988614082,
+            0.01600935123860836,
+            0.046429481357336044,
+            -0.012757082469761372,
+            0.00840235035866499,
+            -0.027624595910310745,
+            0.03795625641942024,
+            0.049043893814086914,
+            0.009985173121094704,
+            -0.020631825551390648,
+            0.02171853929758072,
+            0.009985173121094704,
+            -0.01995459757745266,
+            0.012308222241699696,
+            -0.020631825551390648,
+            0.040413178503513336,
+            -0.022458765655755997,
+            0.03874373435974121,
+            -0.04337408021092415,
+            -0.010709649883210659,
+            -0.03414488211274147,
+            0.003462917869910598,
+            0.02856956608593464,
+            -0.022395767271518707,
+            -0.021608293056488037,
+            -0.0017294903518632054,
+            0.0272938571870327,
+            0.012410594150424004,
+            -0.0009070718660950661,
+            -0.02293124981224537,
+            -0.021198807284235954,
+            -0.03534184396266937,
+            0.006890399381518364,
+            -0.014465902000665665,
+            -0.017371682450175285,
+            -0.0533907525241375,
+            0.014245408587157726,
+            -0.07143966108560562,
+            0.010993140749633312,
+            0.0485084131360054,
+            -0.012276723049581051,
+            0.016662955284118652,
+            0.0007077424670569599,
+            -0.004256298299878836,
+            -0.02935704030096531,
+            0.00967018399387598,
+            0.049831368029117584,
+            0.04028718173503876,
+            -0.021482296288013458,
+            -0.017670921981334686,
+            0.009150451049208641,
+            -0.028601063415408134,
+            -0.01589122973382473,
+            -0.005209141876548529,
+            0.01704094186425209,
+            0.014481651596724987,
+            -0.02589215338230133,
+            0.02638038620352745,
+            0.008465347811579704,
+            0.004953213036060333,
+            0.028396321460604668,
+            0.01331618893891573,
+            -0.0460829921066761,
+            0.014347780495882034,
+            0.029372788965702057,
+            0.001872219960205257,
+            -0.010378910228610039,
+            -0.004106678068637848,
+            0.00592968100681901,
+            0.01819065399467945,
+            -0.017576424404978752,
+            0.007114829961210489,
+            0.005591067019850016,
+            -0.01583610661327839,
+            0.020694822072982788,
+            0.011930234730243683,
+            0.01128450594842434,
+            -0.01067027635872364,
+            0.0218287855386734,
+            -0.00031375925755128264,
+            -0.019907347857952118,
+            -0.009426066651940346,
+            -0.00981980375945568,
+            0.02061607502400875,
+            0.00038364759529940784,
+            -0.03445987403392792,
+            0.011111261323094368,
+            0.04372056946158409,
+            -0.001538527780212462,
+            0.01003242190927267,
+            0.010812020860612392,
+            0.030223261564970016,
+            0.028538066893815994,
+            -0.022710757330060005,
+            -0.02825457602739334,
+            0.01691494695842266,
+            -0.009394567459821701,
+            0.004547663498669863,
+            -0.022632010281085968,
+            -0.014237534254789352,
+            0.03408188372850418,
+            -0.016725953668355942,
+            0.033136915415525436,
+            -0.002256113803014159,
+            0.015969976782798767,
+            -0.004846903961151838,
+            0.0010483250953257084,
+            -0.006433664355427027,
+            0.035215847194194794,
+            -0.026396136730909348,
+            0.014930511824786663,
+            0.007110892329365015,
+            0.002866406226530671,
+            0.05723362788558006,
+            -0.03814525157213211,
+            -0.03625531494617462,
+            0.00262228911742568,
+            0.030853239819407463,
+            -0.017796916887164116,
+            -0.005913931410759687,
+            -0.009599311277270317,
+            0.0031813958194106817,
+            0.018757635727524757,
+            -0.0351843498647213,
+            -0.01855289377272129,
+            0.01331618893891573,
+            -0.01801741123199463,
+            0.003470792667940259,
+            -0.0006501584430225194,
+            0.0018436741083860397,
+            0.00524457823485136,
+            0.0618639774620533,
+            -0.05175280570983887,
+            0.016946446150541306,
+            0.03603482246398926,
+            -0.0007520379149354994,
+            -0.03282192721962929,
+            -0.02625439129769802,
+            -0.005736749619245529,
+            -0.003931465093046427,
+            -0.03067999705672264,
+            -1.9440769392531365e-05,
+            -0.004043680150061846,
+            0.007024270016700029,
+            -0.03669629991054535,
+            0.014623397029936314,
+            0.011355378665030003,
+            -0.002449044957756996,
+            0.003744439920410514,
+            -0.01391466986387968,
+            -0.002553385216742754,
+            -0.003896028734743595,
+            0.03367239981889725,
+            -0.011064013466238976,
+            -0.002878218423575163,
+            -0.00967018399387598,
+            -0.003775938879698515,
+            -0.007197514642030001,
+            -0.0027738779317587614,
+            -0.00845747347921133,
+            0.0031873020343482494,
+            0.012237349525094032,
+            0.001938170986250043,
+            -0.012426343746483326,
+            -0.0028171890880912542,
+            0.03657030314207077,
+            -0.02510467916727066,
+            -0.021183056756854057,
+            0.051280323415994644,
+            0.040476176887750626,
+            0.004374419339001179,
+            -0.021907532587647438,
+            -0.010260789655148983,
+            0.021498046815395355,
+            -0.019749853760004044,
+            0.013670552521944046,
+            0.032176196575164795,
+            -0.007547940593212843,
+            -0.028364822268486023,
+            -0.00964655913412571,
+            -0.014410778880119324,
+            0.0019509674748405814,
+            -0.00562256621196866,
+            0.024065213277935982,
+            -0.044539544731378555,
+            0.008488972671329975,
+            0.019938847050070763,
+            0.002590790158137679,
+            -0.012394844554364681,
+            -0.011040388606488705,
+            -0.003295579692348838,
+            0.02178153768181801,
+            0.0009813897777348757,
+            -0.01893088035285473,
+            0.00861496850848198,
+            0.013418560847640038,
+            0.006158048752695322,
+            0.01893088035285473,
+            -0.005732812453061342,
+            0.02352973073720932,
+            0.009733181446790695,
+            -0.0005241625476628542,
+            0.014056415297091007,
+            -0.019970346242189407,
+            -0.00488234031945467,
+            0.061517488211393356,
+            0.011623119935393333,
+            0.025734657421708107,
+            -0.01431628130376339,
+            0.020631825551390648,
+            0.03814525157213211,
+            -0.0016999599756672978,
+            -0.020820818841457367,
+            0.008362976834177971,
+            0.00283490726724267,
+            -0.044665537774562836,
+            -0.0134894335642457,
+            -0.023923467844724655,
+            -0.037326280027627945,
+            0.0035357594024389982,
+            -0.021214555948972702,
+            0.01801741123199463,
+            0.025199174880981445,
+            0.0065832845866680145,
+            -0.030884739011526108,
+            -0.01197748351842165,
+            0.022710757330060005,
+            0.02741985209286213,
+            0.03449137136340141,
+            -0.01233972143381834,
+            0.010103294625878334,
+            -0.003295579692348838,
+            0.013694177381694317,
+            0.037987757474184036,
+            0.006925835739821196,
+            -0.0031006797216832638,
+            0.008059798739850521,
+            0.029703527688980103,
+            0.03675929829478264,
+            -0.024584945291280746,
+            -0.052130796015262604,
+            -0.0028703436255455017,
+            -0.01686769910156727,
+            0.018600141629576683,
+            -0.001023716526106,
+            0.003746408736333251,
+            0.0011428219731897116,
+            0.006654157303273678,
+            -0.008740964345633984,
+            0.02893180400133133,
+            0.008028299547731876,
+            -0.021797286346554756,
+            0.05159531161189079,
+            -0.015481743961572647,
+            0.024112461134791374,
+            -0.001423359732143581,
+            -0.02590790204703808,
+            -0.028711311519145966,
+            0.013111446052789688,
+            0.007390445563942194,
+            -0.03546784073114395,
+            0.03742077574133873,
+            0.033451907336711884,
+            0.0030731181614100933,
+            0.0012127102818340063,
+            0.0166787039488554,
+            0.016962194815278053,
+            -0.014780891127884388,
+            -0.023876218125224113,
+            0.02286825142800808,
+            -0.008677965961396694,
+            -0.021939031779766083,
+            0.01982860080897808,
+            -0.018521394580602646,
+            -0.008268479257822037,
+            0.0042484235018491745,
+            -0.001532621681690216,
+            -0.010182041674852371,
+            0.014395029284060001,
+            0.016521209850907326,
+            -0.015332123264670372,
+            -0.02527792379260063,
+            0.031215479597449303,
+            0.021513795480132103,
+            0.0038231874350458384,
+            -0.02165554091334343,
+            0.013473683968186378,
+            -0.022537512704730034,
+            -0.0010286382166668773,
+            0.043752070516347885,
+            -0.002433295361697674,
+            0.012946076691150665,
+            -0.010237164795398712,
+            -0.029719278216362,
+            0.002671506255865097,
+            0.011213633231818676,
+            0.03789326176047325,
+            0.01885213330388069,
+            0.01795441284775734,
+            0.019450614228844643,
+            -0.01631646603345871,
+            0.009528438560664654,
+            -0.014560398645699024,
+            -0.03294792026281357,
+            0.03179820999503136,
+            -0.006024178117513657,
+            0.025577163323760033,
+            0.008465347811579704,
+            -0.0321604460477829,
+            -0.004728782922029495,
+            0.011040388606488705,
+            0.013001199811697006,
+            0.023120243102312088,
+            0.013867421075701714,
+            0.003108554519712925,
+            0.031388722360134125,
+            -0.014513149857521057,
+            -0.051091328263282776,
+            0.02710486389696598,
+            0.0063076685182750225,
+            -0.006701405625790358,
+            -0.0016143221873790026,
+            0.020364083349704742,
+            -0.003027838421985507,
+            0.00017139868577942252,
+            0.01270195934921503,
+            -0.010961641557514668,
+            0.00882758665829897,
+            -0.006992771290242672,
+            -0.005469008814543486,
+            0.02414396032691002,
+            0.011827862821519375,
+            0.024065213277935982,
+            -0.019497862085700035,
+            0.006347042508423328,
+            0.017324432730674744,
+            -0.009748931042850018,
+            0.014568272978067398,
+            0.0034314189106225967,
+            -0.05011485889554024,
+            0.01607234962284565,
+            0.017576424404978752,
+            -0.005173705518245697,
+            0.007103017531335354,
+            -0.013402811251580715,
+            0.013804423622786999,
+            0.04627198725938797,
+            0.0016330246580764651,
+            0.0066856564953923225,
+            0.013205942697823048,
+            -0.005012273322790861,
+            -0.009804054163396358,
+            0.03112098202109337,
+            0.00845747347921133,
+            -0.008796087466180325,
+            0.009079578332602978,
+            0.001405641552992165,
+            0.03499535471200943,
+            -0.04828792065382004,
+            0.010717524215579033,
+            0.012520840391516685,
+            -0.018584391102194786,
+            -0.012812205590307713,
+            -0.02886880561709404,
+            0.009520563296973705,
+            -0.03044375404715538,
+            0.01814340613782406,
+            -0.022789504379034042,
+            0.021608293056488037,
+            -0.035688333213329315,
+            -0.0018732043681666255,
+            0.012623212300240993,
+            -0.0058076223358511925,
+            0.003575132926926017,
+            0.012473591603338718,
+            -0.007575502153486013,
+            -0.009323694743216038,
+            -0.018096158280968666,
+            0.023671476170420647,
+            0.05575317516922951,
+            -0.0003388600016478449,
+            0.00024116398708429188,
+            0.015930604189634323,
+            -0.01443440280854702,
+            -0.02620714344084263,
+            0.02953028306365013,
+            -0.016773201525211334,
+            0.015623488463461399,
+            0.03723178058862686,
+            -0.03468036651611328,
+            -0.027262357994914055,
+            0.01543449517339468,
+            -0.013725675642490387,
+            0.008347227238118649,
+            0.0020651512313634157,
+            0.02159254439175129,
+            -0.00283490726724267,
+            -0.026096895337104797,
+            0.028538066893815994,
+            0.006197422277182341,
+            0.009418192319571972,
+            -0.021671291440725327,
+            -0.011095511727035046,
+            -0.027325356379151344,
+            -0.023797471076250076,
+            -0.011867236346006393,
+            -0.010174167342483997,
+            0.006508474703878164,
+            -0.016820449382066727,
+            -0.057832106947898865,
+            -0.015229751355946064,
+            0.023907717317342758,
+            -0.01412728801369667,
+            0.05263477936387062,
+            0.01844264753162861,
+            0.016269218176603317,
+            0.0048035928048193455,
+            -0.04022418335080147,
+            0.030176013708114624,
+            -0.017560675740242004,
+            0.03143597021698952,
+            0.016088098287582397,
+            0.006839213892817497,
+            -0.0059611801989376545,
+            0.05058734491467476,
+            -0.02389196865260601,
+            0.007126641925424337,
+            -0.019686855375766754,
+            0.006473038345575333,
+            -0.02499443292617798,
+            -0.004508289974182844,
+            -0.00521701667457819,
+            -0.011465624906122684,
+            0.010063921101391315,
+            -0.011426251381635666,
+            0.024049462750554085,
+            0.02541966922581196,
+            0.0009026423213072121,
+            0.020301084965467453,
+            -0.030900489538908005,
+            -0.022206773981451988,
+            -0.004740594886243343,
+            0.0018899381393566728,
+            -0.02080507017672062,
+            -0.016332216560840607,
+            -0.040066689252853394,
+            -0.012788581661880016,
+            -0.0002187701902585104,
+            0.01212710328400135,
+            0.0040909284725785255,
+            0.02838057093322277,
+            -0.0017038973746821284,
+            -0.03209745138883591,
+            0.0071896398440003395,
+            -0.029514534398913383,
+            -0.0005906056612730026,
+            0.01391466986387968,
+            -0.017859915271401405,
+            -0.005945430602878332,
+            0.030648497864603996,
+            0.0016812575049698353,
+            -0.0150092588737607,
+            -0.016032975167036057,
+            0.005567443091422319,
+            -0.016237718984484673,
+            -0.0035554461646825075,
+            -0.01028441358357668,
+            0.03364089876413345,
+            -0.015332123264670372,
+            0.007780245505273342,
+            0.0005694422870874405,
+            4.2634346755221486e-05,
+            -0.03439687564969063,
+            -0.012890953570604324,
+            -0.019686855375766754,
+            0.013371312990784645,
+            0.0017777230823412538,
+            -0.027073364704847336,
+            0.007902303710579872,
+            0.0009838506812229753,
+            -0.004531914368271828,
+            -0.021324802190065384,
+            0.0032621119171380997,
+            0.005339075345546007,
+            -0.01771816983819008,
+            0.001696022693067789,
+            -0.033388908952474594,
+            -0.02831757441163063,
+            -0.029719278216362,
+            0.019419115036725998,
+            -0.0030435880180448294,
+            0.021183056756854057,
+            -0.013205942697823048,
+            0.012969700619578362,
+            -0.017261436209082603,
+            0.004590974655002356,
+            0.01570223644375801,
+            0.024663692340254784,
+            -0.03160921484231949,
+            0.013898920267820358,
+            0.01582035794854164,
+            0.021561045199632645,
+            -0.04523251950740814,
+            0.010709649883210659,
+            0.013631178997457027,
+            -0.0015227783005684614,
+            0.016332216560840607,
+            -0.010835645720362663,
+            0.03354640305042267,
+            -0.02093106508255005,
+            -0.0010749023640528321,
+            -0.005587129853665829,
+            0.013520932756364346,
+            0.01861589029431343,
+            0.012883078306913376,
+            -0.030412254855036736,
+            0.03110523149371147,
+            0.02352973073720932,
+            -0.012662585824728012,
+            0.04614599049091339,
+            -0.02656938135623932,
+            0.016458211466670036,
+            0.018600141629576683,
+            -0.02827032469213009,
+            0.0015660893404856324,
+            0.0045791626907885075,
+            -0.015631362795829773,
+            -0.014206035062670708,
+            0.025340920314192772,
+            0.00750462943688035,
+            -0.04346857964992523,
+            0.015001384541392326,
+            0.018237903714179993,
+            0.01119000930339098,
+            0.00433504581451416,
+            -0.023939216509461403,
+            0.009174074977636337,
+            0.028128579258918762,
+            0.0006875634426251054,
+            0.030475253239274025,
+            0.015064381994307041,
+            -0.02323048934340477,
+            0.011930234730243683,
+            -0.005795810371637344,
+            0.007406195160001516,
+            -0.01831665076315403,
+            0.00392752792686224,
+            -0.00747706787660718,
+            -0.014654895290732384,
+            -0.007232951000332832,
+            0.029640531167387962,
+            -0.003964932635426521,
+            -0.018883632495999336,
+            -0.03833424672484398,
+            0.015804607421159744,
+            0.009252822026610374,
+            -0.06709280610084534,
+            -0.00876458827406168,
+            -0.03386139124631882,
+            -0.02140354923903942,
+            -0.008906333707273006,
+            0.012749208137392998,
+            -0.015851857140660286,
+            -0.002389984205365181,
+            0.006154111120849848,
+            0.015599864535033703,
+            -0.01564711332321167,
+            0.02085231803357601,
+            0.006024178117513657,
+            0.005291827023029327,
+            -0.013993416912853718,
+            0.042051125317811966,
+            -0.007563690189272165,
+            -0.03814525157213211,
+            0.032176196575164795,
+            -0.012820080853998661,
+            0.003575132926926017,
+            0.024065213277935982,
+            -0.0019421083852648735,
+            -0.022002030164003372,
+            0.028963303193449974,
+            -0.026285890489816666,
+            -0.03512135148048401,
+            0.01522187702357769,
+            -0.006547848228365183,
+            -0.020285336300730705,
+            -0.02056882716715336,
+            -0.004681534133851528,
+            0.012733458541333675,
+            -0.02050582878291607,
+            0.018489895388484,
+            0.03266443312168121,
+            0.0052524530328810215,
+            -0.016206219792366028,
+            0.012457842007279396,
+            -0.0032247069757431746,
+            -0.0015030914219096303,
+            -0.005173705518245697,
+            0.024364452809095383,
+            -0.004055492114275694,
+            0.0033625150099396706,
+            -0.019119873642921448,
+            0.0030239010229706764,
+            -0.016088098287582397,
+            -0.012670460157096386,
+            -0.013457934372127056,
+            -0.025214925408363342,
+            -0.02493143454194069,
+            -0.04088566079735756,
+            0.03531034663319588,
+            0.024458950385451317,
+            -0.017324432730674744,
+            -0.010599403642117977,
+            -0.00861496850848198,
+            0.0013957980554550886,
+            0.028396321460604668,
+            -0.002578978193923831,
+            -0.0036597864236682653,
+            -0.02231702022254467,
+            0.011686117388308048,
+            -0.010583654046058655,
+            0.0014322188217192888,
+            -0.06296644359827042,
+            -0.01806465908885002,
+            0.034428372979164124,
+            0.007181765045970678,
+            -0.007996801286935806,
+            0.005917869042605162,
+            0.02376597188413143,
+            -0.020222337916493416,
+            -0.012481466867029667,
+            0.04390956461429596,
+            0.0027305670082569122,
+            0.0021458673290908337,
+            0.0070203328505158424,
+            -0.008347227238118649,
+            0.010961641557514668,
+            -0.004445292055606842,
+            -0.007729059550911188,
+            0.0005728874821215868,
+            0.010969515889883041,
+            0.009654434397816658,
+            0.00638247886672616,
+            -0.00442954245954752,
+            0.020710572600364685,
+            -0.011402626521885395,
+            -0.0043665445409715176,
+            0.006543911062180996,
+            0.019749853760004044,
+            -0.004783906042575836,
+            0.00025014611310325563,
+            -0.020301084965467453,
+            0.023986464366316795,
+            0.020773570984601974,
+            0.016158971935510635,
+            -0.0004537820350378752,
+            -0.008733089081943035,
+            -0.008059798739850521,
+            0.004401980899274349,
+            0.0030317758210003376,
+            0.017497677356004715,
+            0.028112830594182014,
+            -0.019482113420963287,
+            -0.016536958515644073,
+            -0.0012373188510537148,
+            0.0025336984544992447,
+            -0.004968962632119656,
+            0.03990919515490532,
+            0.02431720495223999,
+            0.005598941817879677,
+            -0.013969792984426022,
+            -0.0007269371417351067,
+            0.00891420803964138,
+            0.03505835309624672,
+            -0.026112645864486694,
+            -0.01571011170744896,
+            0.019907347857952118,
+            -0.0016330246580764651,
+            0.03433387726545334,
+            -0.021387800574302673,
+            0.014261158183217049,
+            -0.0026006337720900774,
+            0.08655916899442673,
+            -0.023372234776616096,
+            -0.005098895635455847,
+            -0.016174720600247383,
+            -0.014450152404606342,
+            0.004220861941576004,
+            -0.00459491228684783,
+            -0.003289673710241914,
+            0.031624965369701385,
+            -0.002029714873060584,
+            0.017182687297463417,
+            -0.0004545202828012407,
+            -0.008898458443582058,
+            0.008020425215363503,
+            -0.0035180412232875824,
+            0.018206404522061348,
+            -0.006681718863546848,
+            0.008969331160187721,
+            -0.003004214260727167,
+            0.0008957519312389195,
+            0.01322169229388237,
+            0.03291642293334007,
+            0.003939339891076088,
+            0.022537512704730034,
+            -0.02419120818376541,
+            0.018631640821695328,
+            0.007677873596549034,
+            0.016647204756736755,
+            -0.019986096769571304,
+            -0.017418930307030678,
+            -0.0007298901909962296,
+            0.013709926046431065,
+            -0.008575594052672386,
+            0.010906518436968327,
+            -0.003833030816167593,
+            0.039216216653585434,
+            -0.026396136730909348,
+            0.007047894410789013,
+            0.010835645720362663,
+            -0.004051554948091507,
+            0.009118951857089996,
+            0.02208077721297741,
+            0.017985912039875984,
+            0.005118582397699356,
+            -0.02297849766910076,
+            0.014135162346065044,
+            0.02293124981224537,
+            -0.01734018325805664,
+            0.0019470300758257508,
+            0.006539973430335522,
+            -0.0015119505114853382,
+            0.0004030883719678968,
+            0.005835183896124363,
+            -0.00024300962104462087,
+            -0.008331477642059326,
+            0.004512227606028318,
+            -0.007961364462971687,
+            -0.035562336444854736,
+            -0.0027384415734559298,
+            0.012851579114794731,
+            -0.00626435736194253,
+            0.031451720744371414,
+            0.029735026881098747,
+            0.019671106711030006,
+            -0.009811929427087307,
+            -0.006874650251120329,
+            0.004177550785243511,
+            -0.0040751793421804905,
+            -0.0011890861205756664,
+            -0.013607555069029331,
+            -0.014308406971395016,
+            0.014111538417637348,
+            -0.022474514320492744,
+            -0.00888270977884531,
+            0.0018338306108489633,
+            -0.005532006733119488,
+            -0.00459491228684783,
+            -0.0025002306792885065,
+            0.017481928691267967,
+            -0.005724937655031681,
+            0.02037983387708664,
+            0.01048915646970272,
+            -0.011528623290359974,
+            0.011308129876852036,
+            -0.026900120079517365,
+            0.009567812085151672,
+            -0.026963118463754654,
+            -0.010756897740066051,
+            0.009213448502123356,
+            -0.003992494195699692,
+            -0.017985912039875984,
+            -0.010473407804965973,
+            -0.016095973551273346,
+            -0.005220954306423664,
+            -0.00745738111436367,
+            -0.0006659079226665199,
+            0.013607555069029331,
+            0.008150358684360981,
+            -0.01601722650229931,
+            -0.016599956899881363,
+            -0.03694828972220421,
+            0.007843242958188057,
+            0.012993324548006058,
+            -0.025246424600481987,
+            -0.014119412750005722,
+            -0.007567627355456352,
+            0.008433849550783634,
+            -0.01233972143381834,
+            -0.0014351718127727509,
+            -0.0033802331890910864,
+            0.014883263036608696,
+            -0.02493143454194069,
+            0.006827401462942362,
+            -0.01929311826825142,
+            0.01558411493897438,
+            0.009583561681210995,
+            0.009260697290301323,
+            -0.03612931817770004,
+            -0.01885213330388069,
+            0.023561229929327965,
+            -0.008205481804907322,
+            -0.009504813700914383,
+            0.024049462750554085,
+            -0.012969700619578362,
+            0.011134886182844639,
+            0.019875850528478622,
+            0.01145775057375431,
+            -0.0026537880767136812,
+            0.0218287855386734,
+            -0.006213171873241663,
+            7.259527774294838e-05,
+            -0.004512227606028318,
+            0.024332953616976738,
+            0.0063588544726371765,
+            0.032790426164865494,
+            -0.015300624072551727,
+            -0.018521394580602646,
+            -0.005760374013334513,
+            0.0442875511944294,
+            0.007559752557426691,
+            -0.02723085880279541,
+            0.018174905329942703,
+            0.029136545956134796,
+            -0.0032916422933340073,
+            0.005445384420454502,
+            -0.038239747285842896,
+            0.0004163770063314587,
+            0.015505367890000343,
+            0.021923283115029335,
+            0.011292380280792713,
+            -0.009922175668179989,
+            -0.018363898620009422,
+            0.007181765045970678,
+            0.002352579263970256,
+            0.006039927247911692,
+            -0.03990919515490532,
+            0.011449875310063362,
+            -0.020946815609931946,
+            0.01970260590314865,
+            0.00861496850848198,
+            -0.009378817863762379,
+            0.022458765655755997,
+            0.0028152205049991608,
+            -0.017088191583752632,
+            0.014710018411278725,
+            -0.005980866961181164,
+            0.0006673844181932509,
+            0.03329440951347351,
+            -0.0036184440832585096,
+            0.012717708945274353,
+            -0.007815681397914886,
+            0.0061383615247905254,
+            0.0031046171206980944,
+            0.03505835309624672,
+            -0.0008091297931969166,
+            -0.014458026736974716,
+            0.0068628378212451935,
+            0.005417822860181332,
+            -0.0005571380024775863,
+            -0.005059522110968828,
+            0.0077448091469705105,
+            -0.012670460157096386,
+            0.0014400935033336282,
+            -0.003053431399166584,
+            -0.006315543316304684,
+            -0.0030396506190299988,
+            0.002159648109227419,
+            0.00047543758410029113,
+            -0.033136915415525436,
+            0.02178153768181801,
+            -0.0042484235018491745,
+            0.013843797147274017,
+            -0.005913931410759687,
+            -0.01531637366861105,
+            0.0023643914610147476,
+            0.02074207179248333,
+            -0.005492632742971182,
+            -0.011898735538125038,
+            -0.01595422811806202,
+            -0.02238001860678196,
+            -0.026301639154553413,
+            0.028522316366434097,
+            0.0024254205636680126,
+            -0.01607234962284565,
+            0.008481097407639027,
+            -0.009591436013579369,
+            -0.025545664131641388,
+            0.006520286668092012,
+            0.018395397812128067,
+            -0.041641637682914734,
+            0.016048725694417953,
+            -0.006953397300094366,
+            -0.0025573226157575846,
+            0.028648313134908676,
+            0.006732904817909002,
+            -0.003504260443150997,
+            0.014450152404606342,
+            -0.0027384415734559298,
+            0.011945984326303005,
+            0.02165554091334343,
+            0.0023072995245456696,
+            0.0066384077072143555,
+            0.014198160730302334,
+            0.015521117486059666,
+            -0.002368328860029578,
+            -0.01837964914739132,
+            0.027624595910310745,
+            -0.021545294672250748,
+            0.02025383710861206,
+            0.021261803805828094,
+            -0.026427635923027992,
+            0.029388537630438805,
+            -0.00997729878872633,
+            0.019182872027158737,
+            0.0034491370897740126,
+            -0.00592968100681901,
+            -0.01233972143381834,
+            0.004949275404214859,
+            0.007484942674636841,
+            0.009300070814788342,
+            0.010331662371754646,
+            0.044004060328006744,
+            -0.01364692859351635,
+            0.028601063415408134,
+            0.03401888534426689,
+            -0.00010723184095695615,
+            0.02255326136946678,
+            -0.018584391102194786,
+            -0.02123030461370945,
+            0.007488879840821028,
+            -0.015780983492732048,
+            -0.03025476075708866,
+            0.0032483311370015144,
+            -0.01452889945358038,
+            -0.008284228853881359,
+            -0.016269218176603317,
+            -0.0019539203494787216,
+            -0.0022324894089251757,
+            -0.0009164231596514583,
+            0.004220861941576004,
+            0.020348334684967995,
+            0.026049647480249405,
+            0.02376597188413143,
+            0.01024504005908966,
+            0.00747706787660718,
+            0.01148924883455038,
+            -0.009930050000548363,
+            0.026175644248723984,
+            -0.0011713679414242506,
+            -0.0028624688275158405,
+            -0.0172929335385561,
+            0.007055769208818674,
+            -0.00029013503808528185,
+            -0.005110707599669695,
+            0.007425881922245026,
+            -0.022175274789333344,
+            -0.0015906979097053409,
+            -0.04687046632170677,
+            -0.002504168078303337,
+            -0.0007677873945795,
+            0.010394659824669361,
+            0.03965720161795616,
+            -0.007449506316334009,
+            -0.017922913655638695,
+            0.008284228853881359,
+            -0.01685194857418537,
+            0.004098803270608187,
+            -0.01814340613782406,
+            -0.015591990202665329,
+            -0.010756897740066051,
+            -0.02414396032691002,
+            -0.009315820410847664,
+            -0.013812297955155373,
+            0.015143129974603653,
+            -0.00876458827406168,
+            0.011229382827877998,
+            -0.018710387870669365,
+            0.00041736135608516634,
+            -0.006221046671271324,
+            -0.008433849550783634,
+            -0.002559291198849678,
+            -0.020348334684967995,
+            -0.012379094958305359,
+            0.022710757330060005,
+            -0.008780337870121002,
+            -0.029609031975269318,
+            -0.006311606150120497,
+            0.004232673905789852,
+            -0.014859639108181,
+            0.0037365653552114964,
+            0.00592968100681901,
+            -0.005917869042605162,
+            -0.008985080756247044,
+            0.00961506087332964,
+            0.018726136535406113,
+            0.012457842007279396,
+            -0.0028644376434385777,
+            0.00395705783739686,
+            0.02291550114750862,
+            0.02348248101770878,
+            -0.0454845130443573,
+            -0.005102833267301321,
+            0.017686670646071434,
+            0.016347965225577354,
+            -0.0070321448147296906,
+            0.018411148339509964,
+            -0.01507225725799799,
+            0.011827862821519375,
+            -0.02250601351261139,
+            -0.00612261239439249,
+            -0.026915868744254112,
+            0.00845747347921133,
+            0.01734018325805664,
+            -0.006768341176211834,
+            0.035215847194194794,
+            0.020600326359272003,
+            -0.016788950189948082,
+            0.020994063466787338,
+            -0.0013150819577276707,
+            0.011174259707331657,
+            -0.027089113369584084,
+            0.01364692859351635,
+            -0.009229198098182678,
+            0.008236980065703392,
+            0.014206035062670708,
+            -0.020049093291163445,
+            -0.025340920314192772,
+            -0.01222159992903471,
+            0.012174352072179317,
+            0.003065243363380432,
+            -0.0027896275278180838,
+            0.022222522646188736,
+            0.015143129974603653,
+            -0.01197748351842165,
+            -0.01740317977964878,
+            0.023561229929327965,
+            0.01276495773345232,
+            0.015418745577335358,
+            -0.0032581747509539127,
+            -0.0033290472347289324,
+            -0.017009442672133446,
+            0.0005093974177725613,
+            -0.02450619824230671,
+            -0.001176289631985128,
+            -0.002449044957756996,
+            0.021529546007514,
+            -0.007681811228394508,
+            -0.006224983837455511,
+            -0.007177827879786491,
+            -0.02808133140206337,
+            -0.000330000912072137,
+            0.00023365837114397436,
+            0.01252871472388506,
+            0.007658186834305525,
+            0.009473315440118313,
+            0.017481928691267967,
+            -0.026364637538790703,
+            0.002179334871470928,
+            -0.019482113420963287,
+            0.004126364830881357,
+            -0.00928432121872902,
+            -0.014292657375335693,
+            0.01033953670412302,
+            0.003470792667940259,
+            -0.00480753043666482,
+            0.01012691855430603,
+            0.003260143334046006,
+            -0.017387431114912033,
+            0.029939770698547363,
+            -0.004006274975836277,
+            0.002279737964272499,
+            0.003663723822683096,
+            0.005736749619245529,
+            0.0019736073445528746,
+            0.02827032469213009,
+            -0.025388170033693314,
+            -0.016773201525211334,
+            -0.018600141629576683,
+            0.0024372327607125044,
+            0.004988649394363165,
+            0.005386323668062687,
+            -0.0010079670464619994,
+            -0.00013190193567425013,
+            0.002657725475728512,
+            -0.0070321448147296906,
+            0.027876587584614754,
+            0.0040909284725785255,
+            -0.03382989391684532,
+            -0.024395952001214027,
+            -0.04038167744874954,
+            0.008307852782309055,
+            -0.01982860080897808,
+            -0.029766526073217392,
+            -0.010008797980844975,
+            -0.02274225652217865,
+            -0.035940323024988174,
+            0.018474144861102104,
+            0.020757820457220078,
+            -0.02025383710861206,
+            -0.008205481804907322,
+            -0.0028447506483644247,
+            -0.013142945244908333,
+            0.010048171505331993,
+            -0.0029805900994688272,
+            -0.016269218176603317,
+            0.010355286300182343,
+            0.014765141531825066,
+            0.0068628378212451935,
+            -0.00867009162902832,
+            -0.016347965225577354,
+            0.00028521331842057407,
+            0.023813221603631973,
+            0.0064651635475456715,
+            -0.01107188779860735,
+            0.007925928570330143,
+            0.006398227997124195,
+            -0.015111630782485008,
+            0.007437694352120161,
+            -0.023088743910193443,
+            -0.000766310840845108,
+            0.0025021994952112436,
+            0.014056415297091007,
+            -0.01644246280193329,
+            0.003551508765667677,
+            0.005618628580123186,
+            -0.0036873482167720795,
+            0.003545602783560753,
+            -0.018757635727524757,
+            0.00671715522184968,
+            -0.009748931042850018,
+            0.0019913255237042904,
+            0.028128579258918762,
+            -0.007232951000332832,
+            -0.005161893554031849,
+            -0.032412439584732056,
+            0.01642671227455139,
+            0.03360940143465996,
+            0.013678427785634995,
+            -0.021513795480132103,
+            -0.015985727310180664,
+            0.004283859860152006,
+            -0.0016468055546283722,
+            -0.019198622554540634,
+            0.015473868697881699,
+            0.019450614228844643,
+            0.012197976000607014,
+            0.015505367890000343,
+            0.013166569173336029,
+            0.014379279688000679,
+            -0.025451166555285454,
+            0.001903718919493258,
+            -0.0020454642362892628,
+            -0.02170279063284397,
+            0.012024731375277042,
+            0.02061607502400875,
+            -0.02378172241151333,
+            -0.003651911858469248,
+            0.0016153064789250493,
+            0.02165554091334343,
+            -0.003827124834060669,
+            -0.002248239004984498,
+            0.01808040775358677,
+            0.010252914391458035,
+            0.015119505114853382,
+            0.004795718006789684,
+            0.061045002192258835,
+            -0.003384170588105917,
+            0.007122704293578863,
+            0.008575594052672386,
+            -0.005106770433485508,
+            -0.01965535618364811,
+            -0.0010197791270911694,
+            -0.026537882164120674,
+            0.006654157303273678,
+            0.007752683945000172,
+            -0.02778209187090397,
+            -0.005902119446545839,
+            -0.009008705615997314,
+            -0.006595096550881863,
+            0.018048910424113274,
+            0.0047484696842730045,
+            -0.0020179026760160923,
+            0.006260420195758343,
+            0.037987757474184036,
+            0.014686394482851028,
+            -0.009504813700914383,
+            0.0023072995245456696,
+            0.010150542482733727,
+            0.007760558743029833,
+            -0.008268479257822037,
+            0.009040203876793385,
+            0.0004434464208316058,
+            0.004461041651666164,
+            -0.00800467561930418,
+            0.039625704288482666,
+            -0.009040203876793385,
+            0.0009188840049318969,
+            0.0023801408242434263,
+            -0.014426528476178646,
+            0.010938017629086971,
+            0.002990433480590582,
+            0.00729201128706336,
+            -0.007095142733305693,
+            0.0013308314373716712,
+            0.002222646027803421,
+            -0.0020001844968646765,
+            -0.0012343658600002527,
+            -0.009559936821460724,
+            -0.01655270904302597,
+            -0.005587129853665829,
+            0.013016949407756329,
+            0.0005856839707121253,
+            0.0052367039024829865,
+            -0.01812765747308731,
+            -0.0072447629645466805,
+            -0.0007126641576178372,
+            -0.011134886182844639,
+            0.007221138570457697,
+            0.024899935349822044,
+            -0.027498601004481316,
+            -0.004283859860152006,
+            0.014166661538183689,
+            -0.00015171183622442186,
+            0.007327447645366192,
+            0.014607647433876991,
+            -0.013119320385158062,
+            0.008000737987458706,
+            -0.016040850430727005,
+            -0.014347780495882034,
+            0.0007456396706402302,
+            -0.006492725107818842,
+            0.0023506106808781624,
+            -0.01307207252830267,
+            -0.0012058198917657137,
+            0.01279645599424839,
+            0.0006412993534468114,
+            0.012079854495823383,
+            0.014544649049639702,
+            -0.00679196510463953,
+            -0.00038364759529940784,
+            0.012694085016846657,
+            -0.012780707329511642,
+            -0.024128209799528122,
+            0.0007146328571252525,
+            0.023876218125224113,
+            -0.00492958864197135,
+            0.00383499963209033,
+            -0.0028506566304713488,
+            0.0024825125001370907,
+            -0.012662585824728012,
+            -0.013001199811697006,
+            -0.012213725596666336,
+            0.0209153164178133,
+            0.004571287892758846,
+            -0.022758005186915398,
+            0.0008258636225946248,
+            0.0055831922218203545,
+            0.009103202261030674,
+            -0.017497677356004715,
+            0.004000368993729353,
+            0.02995551936328411,
+            -0.01479664072394371,
+            -0.0041814884170889854,
+            -0.0008224184275604784,
+            -0.013812297955155373,
+            0.008044049143791199,
+            0.017088191583752632,
+            0.005484757944941521,
+            0.016363713890314102,
+            -0.003575132926926017,
+            -0.009441816247999668,
+            0.03038075566291809,
+            0.015308499336242676,
+            0.004185425583273172,
+            -0.006536036264151335,
+            -0.012788581661880016,
+            -0.028333323076367378,
+            0.004374419339001179,
+            0.010623027570545673,
+            -0.014292657375335693,
+            -0.0005876526702195406,
+            -0.012276723049581051,
+            -0.018001660704612732,
+            -0.004488603211939335,
+            -0.006524223834276199,
+            0.012300347909331322,
+            -0.00976468063890934,
+            0.010804146528244019,
+            -0.0003120366600342095,
+            -0.0035377279855310917,
+            -0.002352579263970256,
+            0.0002889046154450625,
+            -0.009410317055881023,
+            -0.03291642293334007,
+            -0.028412070125341415,
+            -0.008898458443582058,
+            -0.008142483420670033,
+            0.006528161466121674,
+            0.0009907410712912679,
+            -0.030664246529340744,
+            0.010300163179636002,
+            -0.00012433479423634708,
+            -0.014843889512121677,
+            0.015190377831459045,
+            0.009898550808429718,
+            -0.002703005215153098,
+            -0.007528253830969334,
+            -0.007807807065546513,
+            -0.02874280884861946,
+            0.01589910499751568,
+            0.0021872096695005894,
+            0.009260697290301323,
+            -0.014237534254789352,
+            0.0010424189968034625,
+            -0.021009812131524086,
+            0.01727718487381935,
+            -0.026474883779883385,
+            -0.015095881186425686,
+            -0.01522187702357769,
+            -0.0022128026466816664,
+            -0.0060320524498820305,
+            -0.017781168222427368,
+            -0.021907532587647438,
+            -0.006161985918879509,
+            -0.025640161707997322,
+            -0.022899750620126724,
+            -0.0017609893111512065,
+            -0.0034353563096374273,
+            0.010583654046058655,
+            -0.007040019612759352,
+            0.027986833825707436,
+            0.01619047112762928,
+            0.006579347420483828,
+            -0.011560121551156044,
+            -0.013442184776067734,
+            -0.0020198714919388294,
+            0.0025179488584399223,
+            -0.009158325381577015,
+            0.002384078223258257,
+            0.019797101616859436,
+            -0.0036184440832585096,
+            0.0045240395702421665,
+            0.004106678068637848,
+            -0.004035805352032185,
+            0.016836199909448624,
+            0.007988926023244858,
+            0.011788489297032356,
+            0.006315543316304684,
+            0.0016094004968181252,
+            0.03379839286208153,
+            0.016473960131406784,
+            -0.0035869451239705086,
+            0.0022324894089251757,
+            0.020962564274668694,
+            -0.02170279063284397,
+            -0.01571011170744896,
+            0.015450244769454002,
+            0.001383001683279872,
+            -0.03546784073114395,
+            0.02067907340824604,
+            0.01661570556461811,
+            -0.009796179831027985,
+            -0.010764773003757,
+            -0.007961364462971687,
+            -0.00021323325927369297,
+            -0.005181580316275358,
+            0.008544095791876316,
+            -0.011560121551156044,
+            0.013426436111330986,
+            0.006496662274003029,
+            -0.0064651635475456715,
+            -0.014269033446907997,
+            0.01422178465873003,
+            0.027971085160970688,
+            0.0035790703259408474,
+            -0.04819342494010925,
+            -0.012276723049581051,
+            0.01704094186425209,
+            -0.02140354923903942,
+            -0.025797655805945396,
+            0.00011172290396643803,
+            -0.0075715649873018265,
+            -0.023608477786183357,
+            -0.007063644006848335,
+            -0.00392752792686224,
+            -0.016836199909448624,
+            0.012883078306913376,
+            0.01064665149897337,
+            0.012426343746483326,
+            0.008205481804907322,
+            -0.015867605805397034,
+            -0.012064104899764061,
+            -0.042366113513708115,
+            0.00788655411452055,
+            -0.024836936965584755,
+            -0.021939031779766083,
+            0.004902027081698179,
+            -0.006827401462942362,
+            0.008740964345633984,
+            0.01452889945358038,
+            0.039153218269348145,
+            0.01255233958363533,
+            -0.015080131590366364,
+            -0.025687409564852715,
+            -0.017907164990901947,
+            -0.007961364462971687,
+            0.0022758005652576685,
+            -0.0074180071242153645,
+            -0.0035003230441361666,
+            0.0033192038536071777,
+            0.002246270189061761,
+            0.029309790581464767,
+            0.001104432623833418,
+            0.018962379544973373,
+            -3.389830453670584e-05,
+            0.00928432121872902,
+            -0.0038920913357287645,
+            -0.025025932118296623,
+            0.006213171873241663,
+            0.004106678068637848,
+            0.00794167723506689,
+            0.0068628378212451935,
+            0.005106770433485508,
+            0.016584208235144615,
+            -0.015867605805397034,
+            0.005394198466092348,
+            -0.026222892105579376,
+            -0.006469100713729858,
+            -0.014418653212487698,
+            0.0294042881578207,
+            0.014733643271028996,
+            -0.02327773906290531,
+            -0.013174444437026978,
+            -0.018993878737092018,
+            0.017970161512494087,
+            -0.008685841225087643,
+            0.009756805375218391,
+            0.01355243194848299,
+            0.01040253508836031,
+            0.006205297075212002,
+            0.0053548249416053295,
+            -0.005012273322790861,
+            0.007736934348940849,
+            0.02310449443757534,
+            -0.007106955163180828,
+            0.008000737987458706,
+            -0.0013849702663719654,
+            -0.0041814884170889854,
+            -0.014812390320003033,
+            -0.01722993701696396,
+            -0.0012570057297125459,
+            -0.00373853393830359,
+            0.01935611665248871,
+            -0.017623674124479294,
+            0.020001845434308052,
+            -0.011906610801815987,
+            0.005181580316275358,
+            -0.026490632444620132,
+            -0.008481097407639027,
+            0.01644246280193329,
+            -0.015505367890000343,
+            0.0024195145815610886,
+            -0.015458119101822376,
+            -0.003754283534362912,
+            0.006665969267487526,
+            0.001975575927644968,
+            0.001650742837227881,
+            0.005575317423790693,
+            0.005201267544180155,
+            0.016662955284118652,
+            0.008110984228551388,
+            -0.017450429499149323,
+            -0.0032562059350311756,
+            0.008370851166546345,
+            -0.015135254710912704,
+            0.006961272098124027,
+            0.005862745922058821,
+            0.0009361099801026285,
+            0.04082266613841057,
+            0.0066384077072143555,
+            0.002084837993606925,
+            -0.009898550808429718,
+            0.0037798762787133455,
+            -0.036727797240018845,
+            0.02904205024242401,
+            -0.0016576332272961736,
+            0.016221968457102776,
+            0.001489310641773045,
+            -0.014465902000665665,
+            0.0045791626907885075,
+            -0.005154018756002188,
+            0.014828139916062355,
+            -0.006287981756031513,
+            0.006969146896153688,
+            -0.00131114455871284,
+            0.017387431114912033,
+            -0.007339260075241327,
+            -0.018899381160736084,
+            0.009142575785517693,
+            -0.011316005140542984,
+            0.0007323510362766683,
+            -0.009693807922303677,
+            0.0055280691012740135,
+            0.01352880708873272,
+            0.0070203328505158424,
+            0.005480820778757334,
+            -0.03808225318789482,
+            -0.008622842840850353,
+            -0.003271955531090498,
+            0.002777815330773592,
+            -0.0024687317200005054,
+            -0.023970715701580048,
+            -0.024584945291280746,
+            -0.0002655264688655734,
+            0.02170279063284397,
+            -0.006587222218513489,
+            -0.0011831800220534205,
+            0.009118951857089996,
+            0.0015503398608416319,
+            -0.011749115772545338,
+            -0.006980959326028824,
+            0.007421944756060839,
+            0.021246055141091347,
+            -0.025514164939522743,
+            -0.020883817225694656,
+            0.012048356235027313,
+            0.0029274355620145798,
+            -0.019875850528478622,
+            -0.01033953670412302,
+            -0.0020651512313634157,
+            0.008512596599757671,
+            0.03439687564969063,
+            -0.017686670646071434,
+            0.001204835600219667,
+            0.0008947675814852118,
+            0.014080039225518703,
+            -0.026301639154553413,
+            -0.005043772514909506,
+            -0.01112701091915369,
+            -0.010418283753097057,
+            0.01148924883455038,
+            -0.01631646603345871,
+            0.0070518315769732,
+            -0.002840813249349594,
+            -0.002578978193923831,
+            -0.014064289629459381,
+            0.0009021501755341887,
+            0.0013810329837724566,
+            0.0016064473893493414,
+            0.03316841647028923,
+            -0.009567812085151672,
+            -0.013505183160305023,
+            0.0018968285294249654,
+            0.00041391613194718957,
+            0.014922636561095715,
+            0.010465532541275024,
+            0.01976560428738594,
+            -0.008331477642059326,
+            0.001558214658871293,
+            0.00961506087332964,
+            0.014623397029936314,
+            0.0005468024173751473,
+            0.042240116745233536,
+            -0.005795810371637344,
+            -0.006725030019879341,
+            -0.011788489297032356,
+            -0.005390261299908161,
+            -0.01233972143381834,
+            0.01491476222872734,
+            0.00269119325093925,
+            0.010371035896241665,
+            -0.0020828694105148315,
+            -0.008945707231760025,
+            0.012111353687942028,
+            0.007445569150149822,
+            0.010560029186308384,
+            0.013205942697823048,
+            -0.004014149773865938,
+            -0.02001759596168995,
+            0.010063921101391315,
+            -0.010048171505331993,
+            0.006059614475816488,
+            -0.001052262494340539,
+            0.008866960182785988,
+            -0.007996801286935806,
+            0.018631640821695328,
+            -0.0028703436255455017,
+            0.017607923597097397,
+            0.015560491010546684,
+            -0.019545109942555428,
+            0.017072441056370735,
+            0.01782841607928276,
+            -0.010788396932184696,
+            0.012780707329511642,
+            -0.01648971065878868,
+            -0.010237164795398712,
+            -0.0052524530328810215,
+            0.02176578715443611,
+            -0.013142945244908333,
+            -0.013324064202606678,
+            -0.010922268033027649,
+            0.00395508948713541,
+            -0.007988926023244858,
+            0.017324432730674744,
+            0.00588243268430233,
+            -0.009890676476061344,
+            -0.019686855375766754,
+            0.015505367890000343,
+            0.004433480091392994,
+            0.005213079508394003,
+            0.022474514320492744,
+            -0.007898367010056973,
+            0.005878495052456856,
+            0.014717893674969673,
+            0.030349256470799446,
+            -0.008323602378368378,
+            0.02831757441163063,
+            -0.011111261323094368,
+            0.00266756908968091,
+            -0.0037601895164698362,
+            -0.016158971935510635,
+            -0.012402718886733055,
+            -8.889845776138827e-05,
+            0.0050280229188501835,
+            0.012930327095091343,
+            -0.028601063415408134,
+            0.00861496850848198,
+            0.026222892105579376,
+            0.0027502537705004215,
+            0.008937832899391651,
+            0.0026537880767136812,
+            0.006890399381518364,
+            0.005386323668062687,
+            0.0029943708796054125,
+            -0.022364268079400063,
+            -0.007126641925424337,
+            0.00323848775587976,
+            -0.010851395316421986,
+            0.02104131132364273,
+            0.014355655759572983,
+            0.01067027635872364,
+            -0.002246270189061761,
+            -0.0012619274202734232,
+            0.025151927024126053,
+            0.01959235966205597,
+            0.004051554948091507,
+            -0.001164477551355958,
+            0.006854963023215532,
+            0.0023191117215901613,
+            0.011875111609697342,
+            -0.014190285466611385,
+            -0.014166661538183689,
+            0.007331385277211666,
+            0.00262228911742568,
+            0.010890768840909004,
+            -0.01680470071732998,
+            -0.006449413951486349,
+            -0.003303454490378499,
+            0.01746617816388607,
+            -0.009993048384785652,
+            -0.0018879694398492575,
+            0.0012599588371813297,
+            0.015253376215696335,
+            0.00039250668487511575,
+            -0.0032680181320756674,
+            -0.004724845290184021,
+            -0.014119412750005722,
+            0.0047996556386351585,
+            -0.009725307114422321,
+            0.004894152283668518,
+            -0.005661939736455679,
+            0.00881971139460802,
+            -0.007311698514968157,
+            0.04378356784582138,
+            0.027876587584614754,
+            0.008512596599757671,
+            0.017371682450175285,
+            -0.003986588213592768,
+            0.0017127564642578363,
+            -0.016119597479701042,
+            0.020222337916493416,
+            0.005598941817879677,
+            4.361868923297152e-05,
+            0.003966901451349258,
+            0.024915684014558792,
+            0.004728782922029495,
+            0.009733181446790695,
+            0.0006260420195758343,
+            0.006394290830940008,
+            0.0026439446955919266,
+            0.009410317055881023,
+            -0.018474144861102104,
+            -0.01507225725799799,
+            -0.014497400261461735,
+            0.02316749282181263,
+            -0.005512319505214691,
+            -0.008567719720304012,
+            0.013733550906181335,
+            0.011111261323094368,
+            -0.002498262096196413,
+            -0.014670644886791706,
+            -0.005693438928574324,
+            -0.0008736042072996497,
+            -0.005161893554031849,
+            -0.016269218176603317,
+            0.00297862128354609,
+            0.0020316834561526775,
+            -0.012182226404547691,
+            0.021372050046920776,
+            -0.012166476808488369,
+            -0.020395582541823387,
+            -0.003490479663014412,
+            -0.006843151059001684,
+            -0.04022418335080147,
+            -0.019056877121329308,
+            0.011906610801815987,
+            0.010788396932184696,
+            -0.008591343648731709,
+            -0.03480635955929756,
+            0.021261803805828094,
+            0.0060832384042441845,
+            0.03808225318789482,
+            -0.026065398007631302,
+            0.0023762034252285957,
+            0.0006777200032956898,
+            0.00397477624937892,
+            -0.011268756352365017,
+            -4.0081209590425715e-05,
+            0.019419115036725998,
+            -0.0007800916791893542,
+            0.00812673382461071,
+            -0.00542963482439518,
+            -0.010922268033027649,
+            0.014686394482851028,
+            -0.007209326606243849,
+            0.012906703166663647,
+            -0.006161985918879509,
+            -0.014442277140915394,
+            0.0032424251548945904,
+            0.016631456092000008,
+            0.012607462704181671,
+            -0.0017993786605075002,
+            -0.014670644886791706,
+            -0.009567812085151672,
+            0.00433504581451416,
+            -0.013331938534975052,
+            -0.012890953570604324,
+            0.02378172241151333,
+            -0.0166787039488554,
+            0.018111906945705414,
+            0.013205942697823048,
+            -0.017182687297463417,
+            0.004839029163122177,
+            0.006035990081727505,
+            0.006008428521454334,
+            0.007567627355456352,
+            -0.015064381994307041,
+            -0.019434863701462746,
+            0.004476791247725487,
+            0.026128394529223442,
+            -0.00240376521833241,
+            -0.006020240485668182,
+            0.012544464319944382,
+            -0.0026951306499540806,
+            0.008512596599757671,
+            -0.023387985303997993,
+            -0.0023821096401661634,
+            0.010993140749633312,
+            0.015568365342915058,
+            0.015686485916376114,
+            0.02335648611187935,
+            0.009181949310004711,
+            0.012402718886733055,
+            -0.012323971837759018,
+            0.005177643150091171,
+            0.022726505994796753,
+            -0.009630809538066387,
+            -0.018883632495999336,
+            0.011379002593457699,
+            0.01763942278921604,
+            0.023687224835157394,
+            0.0018456426914781332,
+            -0.010268663987517357,
+            0.030916238203644753,
+            0.012284598313272,
+            -0.009063828736543655,
+            0.014135162346065044,
+            -0.010906518436968327,
+            0.024962933734059334,
+            0.010237164795398712,
+            0.016174720600247383,
+            -0.01219010166823864,
+            -0.010835645720362663,
+            0.006646282505244017,
+            -0.011567996814846992,
+            -0.007410132791846991,
+            0.01249721646308899,
+            -0.020175090059638023,
+            -0.003384170588105917,
+            -0.031325723975896835,
+            0.00638247886672616,
+            0.003254237351939082,
+            0.0035790703259408474,
+            -0.02461644448339939,
+            0.005658002570271492,
+            -0.006146236322820187,
+            0.008079485967755318,
+            -0.011048263870179653,
+            -0.031262725591659546,
+            0.0039058721158653498,
+            -0.006496662274003029,
+            0.002059245016425848,
+            -0.01433990616351366,
+            0.01589122973382473,
+            0.007114829961210489,
+            -0.021088559180498123,
+            0.0010315912077203393,
+            -0.01904112659394741,
+            -0.0072565749287605286,
+            0.0003491956158541143,
+            -0.0023880156222730875,
+            -0.016347965225577354,
+            0.011363252997398376,
+            0.005886369850486517,
+            -0.02527792379260063,
+            -0.006937648169696331,
+            -0.0024687317200005054,
+            0.0007170937024056911,
+            0.016048725694417953,
+            -0.006055676843971014,
+            0.0022324894089251757,
+            0.033388908952474594,
+            0.015867605805397034,
+            -0.008488972671329975,
+            -0.025624411180615425,
+            -0.0028644376434385777,
+            -0.02359272725880146,
+            0.015631362795829773,
+            -0.0007953489548526704,
+            0.005650127772241831,
+            -0.029971269890666008,
+            0.005150081589818001,
+            0.006709280423820019,
+            0.01746617816388607,
+            0.006488787941634655,
+            0.01033953670412302,
+            0.026837121695280075,
+            -0.005106770433485508,
+            -0.001383001683279872,
+            -1.0535543879086617e-05,
+            0.00679196510463953,
+            0.018411148339509964,
+            -0.0036184440832585096,
+            0.01812765747308731,
+            -0.009607185609638691,
+            0.0029136547818779945,
+            -0.012654710561037064,
+            0.0035554461646825075,
+            -0.003462917869910598,
+            -0.013095696456730366,
+            -9.954166307579726e-05,
+            0.012686209753155708,
+            0.003395982552319765,
+            -0.0012727552093565464,
+            -0.018836384639143944,
+            -0.003833030816167593,
+            -0.0012530683306977153,
+            -0.021261803805828094,
+            0.0029628719203174114,
+            -0.01970260590314865,
+            -0.01124513242393732,
+            -0.02778209187090397,
+            -0.0025435418356209993,
+            0.04438204690814018,
+            0.00955206248909235,
+            -0.012331846170127392,
+            0.002647882094606757,
+            -0.02067907340824604,
+            0.0009956627618521452,
+            -0.012213725596666336,
+            -0.0036479744594544172,
+            -0.006886462215334177,
+            -0.013820173218846321,
+            0.014631271362304688,
+            0.01735593192279339,
+            -0.00404761778190732,
+            -0.006701405625790358,
+            0.012481466867029667,
+            -0.01070177461951971,
+            -0.009504813700914383,
+            -0.017576424404978752,
+            -0.017922913655638695,
+            -0.00785505585372448,
+            0.018883632495999336,
+            -0.005795810371637344,
+            0.02127755433320999,
+            0.008544095791876316,
+            0.0034353563096374273,
+            -0.00692977337166667,
+            0.0015168722020462155,
+            -0.021498046815395355,
+            -0.0006855947431176901,
+            0.006287981756031513,
+            -0.01433990616351366,
+            0.016599956899881363,
+            -0.006815589498728514,
+            -0.0031361160799860954,
+            0.008725214749574661,
+            0.004413793329149485,
+            -0.008725214749574661,
+            -0.005516257137060165,
+            -0.017135439440608025,
+            0.018820634111762047,
+            -0.018332399427890778,
+            -0.0027502537705004215,
+            0.00022246147273108363,
+            -0.009969423525035381,
+            0.012158602476119995,
+            0.015591990202665329,
+            0.007481005042791367,
+            -0.010378910228610039,
+            0.006610846146941185,
+            -0.010292287915945053,
+            0.0221437755972147,
+            -0.001872219960205257,
+            0.01612747274339199,
+            0.013024823740124702,
+            0.005433572456240654,
+            0.006284044589847326,
+            0.001713740755803883,
+            -0.012544464319944382,
+            -0.006949460133910179,
+            0.011064013466238976,
+            -0.02571890875697136,
+            -0.01349730882793665,
+            0.005917869042605162,
+            0.002801439492031932,
+            -0.01148924883455038,
+            -0.0007451475248672068,
+            -0.009205574169754982,
+            0.00037798762787133455,
+            -0.008481097407639027,
+            0.016710203140974045,
+            -0.03433387726545334,
+            -0.00861496850848198,
+            0.0017580362036824226,
+            0.031136730685830116,
+            -0.008449598215520382,
+            0.01619047112762928,
+            0.007532190997153521,
+            0.006488787941634655,
+            0.023687224835157394,
+            0.001509981811977923,
+            0.007662124466150999,
+            0.02116730809211731,
+            -0.011095511727035046,
+            0.00785505585372448,
+            0.0058036851696670055,
+            -0.01995459757745266,
+            0.025120427832007408,
+            -0.0061856103129684925,
+            0.005906056612730026,
+            0.002665600273758173,
+            0.001731458934955299,
+            0.02571890875697136,
+            0.027624595910310745,
+            -0.00870158988982439,
+            -0.005665877368301153,
+            -0.013379187323153019,
+            0.027262357994914055,
+            -0.004945338238030672,
+            0.007610938511788845,
+            -0.010197791270911694,
+            0.002689224435016513,
+            0.0007668030448257923,
+            -0.01024504005908966,
+            0.021482296288013458,
+            -0.016032975167036057,
+            -0.011875111609697342,
+            -0.017560675740242004,
+            -0.002472669119015336,
+            -0.024222707375884056,
+            0.013536682352423668,
+            0.0025829155929386616,
+            -0.003352671628817916,
+            0.0023821096401661634,
+            0.006484850309789181,
+            -0.003496385645121336,
+            -0.00940244272351265,
+            -0.0020887753926217556,
+            -0.01170186698436737,
+            -0.020348334684967995,
+            0.0166787039488554,
+            -0.007248700130730867,
+            -0.002116336952894926,
+            -0.0012127102818340063,
+            0.009811929427087307,
+            0.014985634945333004,
+            0.02129330299794674,
+            0.002915623364970088,
+            0.0066344705410301685,
+            0.007311698514968157,
+            -0.0019332492956891656,
+            -0.012032606638967991,
+            0.020427081733942032,
+            0.0197183545678854,
+            0.0003445199690759182,
+            -0.0031695838551968336,
+            -0.0109143927693367,
+            -0.035562336444854736,
+            -0.009733181446790695,
+            0.0008125749882310629,
+            0.011938109062612057,
+            -0.012442093342542648,
+            -0.007103017531335354,
+            0.039027221500873566,
+            -0.003220769576728344,
+            0.02110430970788002,
+            -0.004189362749457359,
+            3.842013029498048e-05,
+            0.01674170233309269,
+            0.004606724251061678,
+            0.012142852880060673,
+            -0.01264683622866869,
+            -0.02327773906290531,
+            0.002559291198849678,
+            -0.012961826287209988,
+            0.002697099233046174,
+            0.0004523055104073137,
+            -0.0002670029934961349,
+            -0.019497862085700035,
+            -0.008504721336066723,
+            0.005189455114305019,
+            -0.01128450594842434,
+            0.010040296241641045,
+            -0.004157864023000002,
+            0.017387431114912033,
+            -0.0030179950408637524,
+            0.0037897196598351,
+            -0.008583469316363335,
+            -0.006811652332544327,
+            0.0011575871612876654,
+            -0.0017304746434092522,
+            -0.017198437824845314,
+            -8.342305227415636e-05,
+            -0.0020887753926217556,
+            -0.0003410747740417719,
+            0.008410224691033363,
+            0.008378725498914719,
+            0.01328469067811966,
+            0.01385954674333334,
+            -0.0009119936148636043,
+            0.006996708456426859,
+            0.009008705615997314,
+            0.02735685557126999,
+            -0.007185702212154865,
+            0.0068077147006988525,
+            0.002291549928486347,
+            0.008012549951672554,
+            -0.012946076691150665,
+            -0.03285342454910278,
+            0.015426619909703732,
+            -0.015780983492732048,
+            -0.0048035928048193455,
+            -0.014717893674969673,
+            0.01040253508836031,
+            -0.004445292055606842,
+            0.0010876987362280488,
+            0.006098988000303507,
+            0.00845747347921133,
+            0.004937463440001011,
+            0.019308868795633316,
+            -0.007213264238089323,
+            0.0071935770101845264,
+            0.021309053525328636,
+            -0.003295579692348838,
+            -0.002515980275347829,
+            0.006563597824424505,
+            0.009323694743216038,
+            -0.01644246280193329,
+            0.01103251427412033,
+            0.024647943675518036,
+            -0.023198990151286125,
+            0.004441354889422655,
+            0.002891999203711748,
+            -0.01070177461951971,
+            0.004126364830881357,
+            0.0065320986323058605,
+            -0.00033024701406247914,
+            0.023009996861219406,
+            0.004587037488818169,
+            0.019576609134674072,
+            0.004022024571895599,
+            -0.014670644886791706,
+            -0.015599864535033703,
+            0.006646282505244017,
+            -0.010890768840909004,
+            -0.011016764678061008,
+            -0.008063736371695995,
+            -0.010693900287151337,
+            0.0054611340165138245,
+            -0.004787843208760023,
+            -0.004022024571895599,
+            -0.0037739702966064215,
+            -0.011253006756305695,
+            0.006827401462942362,
+            -0.01148924883455038,
+            0.001115260412916541,
+            -0.00638247886672616,
+            -0.005280014593154192,
+            0.00945756584405899,
+            0.011851487681269646,
+            -0.010008797980844975,
+            -0.02674262598156929,
+            0.005555630661547184,
+            0.031199729070067406,
+            -0.006728967186063528,
+            0.007929865270853043,
+            0.012355471029877663,
+            -0.00745738111436367,
+            0.000588144815992564,
+            -0.006732904817909002,
+            -0.004650035407394171,
+            -0.03105798363685608,
+            -0.0010227321181446314,
+            -0.01067027635872364,
+            0.0003351687337271869,
+            -0.014245408587157726,
+            -0.015017134137451649,
+            -0.013166569173336029,
+            0.004968962632119656,
+            -0.013142945244908333,
+            -0.005043772514909506,
+            0.004149989224970341,
+            0.006988833658397198,
+            -0.016379464417696,
+            0.017576424404978752,
+            -0.00943394098430872,
+            -0.003933433908969164,
+            -0.00592968100681901,
+            0.033451907336711884,
+            -0.005406010430306196,
+            -0.003504260443150997,
+            -0.00492958864197135,
+            0.008874834515154362,
+            0.006776215974241495,
+            0.002053339034318924,
+            -0.0029038111679255962,
+            0.0021379925310611725,
+            0.02323048934340477,
+            0.006232858635485172,
+            0.003397951368242502,
+            0.015536867082118988,
+            -0.013473683968186378,
+            -0.008044049143791199,
+            -0.012654710561037064,
+            -0.01067027635872364,
+            0.0020454642362892628,
+            0.010985265485942364,
+            0.02365572564303875,
+            -0.005083146039396524,
+            -0.014985634945333004,
+            0.011473499238491058,
+            0.006772278342396021,
+            0.02140354923903942,
+            -0.011843612417578697,
+            5.6138298532459885e-05,
+            0.00812673382461071,
+            0.02376597188413143,
+            0.031813960522413254,
+            -0.000510381767526269,
+            7.437940075760707e-05,
+            0.0083393519744277,
+            0.010615152306854725,
+            -0.004980774596333504,
+            0.00664234533905983,
+            -0.01704094186425209,
+            -0.02795533463358879,
+            -0.008898458443582058,
+            0.006646282505244017,
+            -0.001707834773696959,
+            0.018647389486432076,
+            0.01007179543375969,
+            0.016757450997829437,
+            0.02382897026836872,
+            0.023025747388601303,
+            0.00039545970503240824,
+            0.02037983387708664,
+            -0.004425605293363333,
+            -0.004417730495333672,
+            0.0025691348128020763,
+            -0.015048632398247719,
+            -0.03471186384558678,
+            0.006118674762547016,
+            0.00442954245954752,
+            0.024773940443992615,
+            -0.015757359564304352,
+            0.023561229929327965,
+            -0.004146052058786154,
+            -0.016158971935510635,
+            -0.005898181814700365,
+            0.020789319649338722,
+            -0.008772462606430054,
+            0.020789319649338722,
+            0.0013239410473033786,
+            -0.0010945892427116632,
+            0.00988280214369297,
+            -0.0011201821034774184,
+            -0.01379654835909605,
+            0.011253006756305695,
+            0.01763942278921604,
+            0.03770426660776138,
+            0.009630809538066387,
+            -0.008118859492242336,
+            0.01752917654812336,
+            0.001776738790795207,
+            -0.008394475094974041,
+            -0.004433480091392994,
+            -0.013820173218846321,
+            0.0076936231926083565,
+            -0.0074180071242153645,
+            0.01697794534265995,
+            0.002640007296577096,
+            -0.005260327830910683,
+            0.014725768007338047,
+            -0.012481466867029667,
+            0.01219010166823864,
+            0.021246055141091347,
+            0.0011949921026825905,
+            0.005886369850486517,
+            0.020001845434308052,
+            -0.033136915415525436,
+            0.011528623290359974,
+            -0.034837860614061356,
+            -0.005138269625604153,
+            0.012064104899764061,
+            -0.01861589029431343,
+            0.007732997182756662,
+            -0.018584391102194786,
+            -0.013253191486001015,
+            -0.002553385216742754,
+            0.014599772170186043,
+            0.02497868239879608,
+            0.006555723026394844,
+            -0.017670921981334686,
+            0.01179636362940073,
+            -0.00876458827406168,
+            -0.01825365237891674,
+            -0.008874834515154362,
+            -0.019419115036725998,
+            -0.011693992651998997,
+            -0.003850748995319009,
+            0.011938109062612057,
+            -0.008811837062239647,
+            0.0037011289969086647,
+            0.002248239004984498,
+            -0.0221437755972147,
+            -0.013883170671761036,
+            0.015914853662252426,
+            0.003413700731471181,
+            -0.00664234533905983,
+            -0.007343197241425514,
+            0.013016949407756329,
+            -0.010756897740066051,
+            0.007323510479182005,
+            -0.008181856945157051,
+            0.014253283850848675,
+            -0.012142852880060673,
+            -0.00319123943336308,
+            0.007079393602907658,
+            0.0035869451239705086,
+            -0.018033159896731377,
+            -0.004732720088213682,
+            -0.02493143454194069,
+            0.015355747193098068,
+            0.0029766527004539967,
+            -0.017481928691267967,
+            -0.00692977337166667,
+            0.007732997182756662,
+            -0.017922913655638695,
+            -0.004894152283668518,
+            0.008717339485883713,
+            -0.003990525845438242,
+            0.025340920314192772,
+            0.022301269695162773,
+            -0.01248934119939804,
+            0.00016143222455866635,
+            -0.009213448502123356,
+            0.009985173121094704,
+            -0.01856864243745804,
+            -0.004484665580093861,
+            0.02650638297200203,
+            0.007315635681152344,
+            -0.00550444470718503,
+            -0.009937925264239311,
+            0.001173336640931666,
+            0.002771909348666668,
+            0.017607923597097397,
+            0.02493143454194069,
+            0.00621710903942585,
+            -0.010985265485942364,
+            -0.022458765655755997,
+            -0.027388354763388634,
+            0.027372604236006737,
+            0.0007485927199013531,
+            -0.004118490032851696,
+            0.013276815414428711,
+            0.02529367245733738,
+            0.023939216509461403,
+            -0.0062013594433665276,
+            -0.00483509199693799,
+            0.018300902098417282,
+            0.00997729878872633,
+            -0.005402073264122009,
+            -0.010166292078793049,
+            0.003392045386135578,
+            -0.0071345167234539986,
+            0.03414488211274147,
+            0.010111168958246708,
+            0.004417730495333672,
+            -0.046366482973098755,
+            0.009993048384785652,
+            -0.01619047112762928,
+            -0.003758220700547099,
+            0.008236980065703392,
+            0.021734289824962616,
+            0.010095419362187386,
+            0.013032699003815651,
+            0.007984988391399384,
+            -0.007236888166517019,
+            0.02861681394279003,
+            -0.0028703436255455017,
+            -0.010623027570545673,
+            -0.007272324524819851,
+            -0.025687409564852715,
+            0.004342920612543821,
+            -0.016410963609814644,
+            0.0021517733111977577,
+            0.005677689332515001,
+            -0.00918982457369566,
+            0.02590790204703808,
+            0.00794167723506689,
+            -0.02382897026836872,
+            0.00588243268430233,
+            -0.03682229667901993,
+            0.019482113420963287,
+            -0.02619139291346073,
+            -0.009323694743216038,
+            -0.004862653557211161,
+            -0.003758220700547099,
+            -0.025923652574419975,
+            -0.0037365653552114964,
+            -0.004783906042575836,
+            0.016883447766304016,
+            0.0024017964024096727,
+            -0.00013805407797917724,
+            -0.00013399365707300603,
+            -0.007103017531335354,
+            0.002041527070105076,
+            0.012410594150424004,
+            0.022521764039993286,
+            0.01571798510849476,
+            -0.0037897196598351,
+            0.018521394580602646,
+            -0.0005768248811364174,
+            -0.03704278916120529,
+            0.0037168783601373434,
+            -0.011662493459880352,
+            0.008103109896183014,
+            0.015332123264670372,
+            0.009591436013579369,
+            -0.01370205171406269,
+            0.013190193101763725,
+            -0.012150727212429047,
+            0.015395121648907661,
+            0.01619047112762928,
+            0.004894152283668518,
+            -0.0009976314613595605,
+            0.0157652348279953,
+            -0.02389196865260601,
+            0.00020941892580594867,
+            -0.00626435736194253,
+            0.006713218055665493,
+            -0.009930050000548363,
+            -0.0033251100685447454,
+            0.002041527070105076,
+            0.0006855947431176901,
+            -0.0034038573503494263,
+            -0.012064104899764061,
+            -0.017387431114912033,
+            -0.021498046815395355,
+            -0.01428478304296732,
+            -0.020820818841457367,
+            -0.011001015082001686,
+            0.030711494386196136,
+            -0.008622842840850353,
+            0.004559475928544998,
+            0.015025008469820023,
+            0.018710387870669365,
+            0.002596696373075247,
+            0.01856864243745804,
+            0.008260604925453663,
+            -0.007299886085093021,
+            -0.01727718487381935,
+            0.005543818697333336,
+            -0.025435417890548706,
+            0.03310541808605194,
+            0.00933944433927536,
+            -0.014032791368663311,
+            0.012969700619578362,
+            -0.01070177461951971,
+            -0.004346857778728008,
+            -0.007784182671457529,
+            0.0060163033194839954,
+            -0.025498416274785995,
+            0.01455252431333065,
+            -0.011749115772545338,
+            0.006961272098124027,
+            0.002305330941453576,
+            0.03505835309624672,
+            -0.014489525929093361,
+            -0.008929957635700703,
+            0.012032606638967991,
+            0.019119873642921448,
+            0.026553630828857422,
+            -0.009441816247999668,
+            -0.004937463440001011,
+            -0.01814340613782406,
+            -0.00357119576074183,
+            -0.005906056612730026,
+            -0.018631640821695328,
+            0.03590882569551468,
+            -0.007847180590033531,
+            -0.021387800574302673,
+            -0.00824485532939434,
+            -0.02006484381854534,
+            0.0031262726988643408,
+            0.0315462164580822,
+            0.01040253508836031,
+            0.006890399381518364,
+            -8.92060634214431e-05,
+            -0.020285336300730705,
+            -0.021324802190065384,
+            0.019797101616859436,
+            0.012166476808488369,
+            -0.019844351336359978,
+            -0.0009695776388980448,
+            -0.009394567459821701,
+            -0.00216752290725708,
+            0.0006609861738979816,
+            0.0021773662883788347,
+            0.05780060961842537,
+            -0.008087360300123692,
+            0.0013229567557573318,
+            -0.024065213277935982,
+            -0.001320003648288548,
+            -0.003332984633743763,
+            -0.0035318220034241676,
+            0.005646190140396357,
+            -0.00626435736194253,
+            0.01128450594842434,
+            -0.03480635955929756,
+            0.006784090772271156,
+            -0.010607277974486351,
+            -0.009654434397816658,
+            -0.016111722216010094,
+            0.00166747672483325,
+            -0.018285151571035385,
+            -4.6664001274621114e-05,
+            0.00021680150530301034,
+            -0.012134977616369724,
+            0.004083054140210152,
+            -0.013883170671761036,
+            -0.0003961979818996042,
+            0.024553446099162102,
+            0.011686117388308048,
+            -0.0013268940383568406,
+            -0.022222522646188736,
+            -0.022395767271518707,
+            0.0015562459593638778,
+            -0.010394659824669361,
+            0.01801741123199463,
+            -0.013229567557573318,
+            0.017182687297463417,
+            -0.02165554091334343,
+            -0.00046780891716480255,
+            0.0019765603356063366,
+            0.006232858635485172,
+            0.01861589029431343,
+            -0.00010969270078931004,
+            0.009693807922303677,
+            -0.014993509277701378,
+            0.02535667084157467,
+            -0.006949460133910179,
+            -0.02019083872437477,
+            0.0007328432402573526,
+            0.005181580316275358,
+            -0.0044216676615178585,
+            -0.0003430434735491872,
+            0.011056138202548027,
+            0.019419115036725998,
+            -0.0024766065180301666,
+            -0.007839306257665157,
+            -0.014064289629459381,
+            -0.009930050000548363,
+            0.0194033645093441,
+            0.0009297117940150201,
+            0.02055307850241661,
+            -0.011536497622728348,
+            0.00918982457369566,
+            -0.017623674124479294,
+            -0.007662124466150999,
+            -0.00907170306891203,
+            -0.004022024571895599,
+            0.016725953668355942,
+            -0.0007599126547574997,
+            0.01512738037854433,
+            -0.018001660704612732,
+            -0.021907532587647438,
+            0.0006098003359511495,
+            -0.02348248101770878,
+            -0.018411148339509964,
+            0.009205574169754982,
+            -0.01855289377272129,
+            -0.0025986649561673403,
+            0.001046356395818293,
+            -0.020883817225694656,
+            -0.013473683968186378,
+            0.018694639205932617,
+            -0.001501122722402215,
+            -0.0029628719203174114,
+            0.008496847003698349,
+            -0.006343104876577854,
+            -0.021813036873936653,
+            -0.0039688702672719955,
+            0.009512688964605331,
+            0.009748931042850018,
+            0.012993324548006058,
+            -0.022222522646188736,
+            0.004724845290184021,
+            0.0015719954390078783,
+            -0.005350887309759855,
+            -0.007095142733305693,
+            0.018174905329942703,
+            0.008047986775636673,
+            0.004531914368271828,
+            0.007768433541059494,
+            0.010008797980844975,
+            -0.0009479221189394593,
+            0.008370851166546345,
+            0.02146654762327671,
+            0.00108376145362854,
+            -0.002827032469213009,
+            -0.016347965225577354,
+            0.00612261239439249,
+            -0.012292472645640373,
+            0.015473868697881699,
+            0.02589215338230133,
+            0.008355101570487022,
+            -0.016332216560840607,
+            0.020883817225694656,
+            -0.014481651596724987,
+            0.015151004306972027,
+            -0.0282073263078928,
+            -0.00988280214369297,
+            0.018159156665205956,
+            -0.0011546340538188815,
+            0.01842689700424671,
+            -0.003108554519712925,
+            0.024065213277935982,
+            0.012481466867029667,
+            0.006417915225028992,
+            1.7518224922241643e-05,
+            -0.0007170937024056911,
+            0.010063921101391315,
+            -0.005189455114305019,
+            0.00014851271407678723,
+            -0.005740687251091003,
+            0.009134701453149319,
+            -0.006551785860210657,
+            0.00027832292835228145,
+            0.010725399479269981,
+            -0.028065582737326622,
+            0.014332030899822712,
+            -0.02274225652217865,
+            -0.0036735672038048506,
+            -0.0024431387428194284,
+            -0.007146328687667847,
+            -0.02031683549284935,
+            -0.003079024376347661,
+            0.019088376313447952,
+            -0.018489895388484,
+            0.04819342494010925,
+            0.0012028669007122517,
+            0.018300902098417282,
+            -0.007453443482518196,
+            0.002006090711802244,
+            -0.015143129974603653,
+            -0.00035928512807004154,
+            -0.02674262598156929,
+            0.00807161070406437,
+            0.003821218851953745,
+            0.0024254205636680126,
+            0.0025238548405468464,
+            0.006902211811393499,
+            -0.002173428889364004,
+            0.026648128405213356,
+            0.0002606047492008656,
+            0.01127663068473339,
+            -0.004295671824365854,
+            0.010717524215579033,
+            -0.019261619076132774,
+            0.004327171016484499,
+            0.0034491370897740126,
+            -0.00797711405903101,
+            -0.003933433908969164,
+            -0.007091205567121506,
+            0.0221437755972147,
+            0.016631456092000008,
+            0.009662308730185032,
+            -0.008020425215363503,
+            -0.0013042541686445475,
+            0.006803777534514666,
+            -0.0030396506190299988,
+            -0.011756990104913712,
+            -0.0038724045734852552,
+            0.005969054531306028,
+            0.012512966059148312,
+            -0.0009011658257804811,
+            0.011961733922362328,
+            0.005362699739634991,
+            -0.008008613251149654,
+            0.01297757588326931,
+            -0.020096343010663986,
+            -0.006390353664755821,
+            -0.009685933589935303,
+            -0.033987388014793396,
+            0.009292195551097393,
+            -0.0017304746434092522,
+            0.003090836340561509,
+            -0.007229013368487358,
+            0.010260789655148983,
+            -0.009520563296973705,
+            -0.01322169229388237,
+            -0.006512411870062351,
+            0.002208865247666836,
+            0.023639976978302002,
+            -0.006193485110998154,
+            0.00240376521833241,
+            -0.013355563394725323,
+            -0.013804423622786999,
+            0.016899196431040764,
+            -0.01170186698436737,
+            -0.007343197241425514,
+            0.009827678091824055
+          ]
+        },
+        "type": "chunk"
+      },
+      "bidirectional": true,
+      "properties": {
+        "cosine_similarity": 0.838717725661055
+      }
+    },
+    {
+      "id": "100e0e79-35e9-44ec-b9ba-89143b9f1b54",
+      "type": "summary_cosine_similarity",
+      "source": {
+        "id": "b570d71e-4f83-486e-b70f-b0c6217db455",
+        "properties": {
+          "page_content": "Mixture-Of-Experts With Expert Choice Routing\n\nYanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon Google, Mountain View, CA, USA {yanqiz, taole, hanxiaol, dunan, huangyp, vzhao, adai, zhifengc, qvl, jlaudon}@google.com\n\nAbstract\n\nSparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choice method. Instead of letting tokens select the top-k experts, we have experts selecting the top-k tokens. As a result, each token can be routed to a variable number of experts and each expert can have a fixed bucket size. We systematically study pre-training speedups using the same computational resources of the Switch Transformer top-1 and GShard top-2 gating of prior work and find that our method improves training convergence time by more than 2×. For the same computational cost, our method demonstrates higher performance in fine-tuning 11 selected tasks in the GLUE and SuperGLUE benchmarks. For a smaller activation cost, our method outperforms the T5 dense model in 7 out of the 11 tasks.\n\n1 Introduction\n\nScaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing the performance of computer vision architectures [4, 11, 13, 14] as well as neural language models [2, 20, 26, 27]. The final model quality has been found to have a power-law relationship with the amount of data, model size, and compute time [16, 20]. However, training efficiency, which is defined as the total amount of computation used to achieve superior model quality than the state of the art system [21], should receive greater attention as we increase our efforts towards green AI [29].\n\nSparsely gated mixture-of-experts [31] (MoE) provides an effective way to scale model capacity given a fixed computational cost, and has recently played an important role in increasing the training efficiency of large-scale language models [10, 21]. MoE operate by adopting a number of experts, each as a sub-network, and by activating only one or a few experts for each input token. A gating network must be chosen and optimized in order to route each token to the most suited expert(s). For example, recent work has implemented sparse routing via k-means clustering [12], linear assignment to maximize token-expert affinities [22], or hashing [8, 28]. Many of the prior work use a routing strategy concerning the token choice, where each token selects the best one or two experts.\n\nWe argue that the independent token choice of prior work often leads to an imbalanced load of experts, which causes training inefficiency and sub-optimal training of the model. In order to mitigate this\n\nissue, previous sparsely gated networks introduce additional auxiliary losses as regularization to prevent too many tokens being routed to a single expert, but the effectiveness is still limited. Recent approaches [8, 22, 28] explore alternative strategies for routing, but they focus on pre-training only and do not demonstrate performance gain on downstream tasks. Moreover, none of the previous methods consider allocating a variable number of experts to each token based on importance, which can be beneficial.\n\nWe propose a very simple yet effective routing method we are calling expert choice. Unlike conventional MoE where tokens select one or two top-scoring experts, our method lets each expert pick the top-k tokens. Our method guarantees perfect load balancing, allows a variable number of experts for each token, and achieves substantial gains in training efficiency and downstream performance as demonstrated in our experiments. Our major contributions include:\n\nWe identify common pitfalls in conventional MoE such as load imbalance as described in Section 3.1. We then propose a heterogeneous, expert choice method to provide a fluid allocation of model parameters based on a learnt token-to-expert importance. This method intrinsically guarantees load balance without imposing an auxiliary loss.\n\nWe show our method provides over 2× faster training convergence in a 8B/64E (8 billion activated parameters, 64 experts) model, compared to the top-1 and top-2 gating counterparts in Switch Transformer [10] and GShard [21].\n\nWe show our method demonstrates strong scaling when increasing the number of experts from 16 to 128, evaluated in training perplexity.\n\nWe show our method demonstrates strong performance on downstream tasks selected from GLUE and SuperGLUE at all the evaluated scales. More specifically, our 8B/64E model outperforms a T5 11B dense model in 7 out of 11 tasks evaluated.\n\n2 Related Work\n\nScaling: Various approaches have been proposed to scale up neural network capacity to improve performance. Recent works have successfully scaled models to billions of parameters via various forms of model parallelism [2, 21, 26, 27, 33]. Model parallelism [30] splits weights and tensors across multiple cores while pipeline parallelism [18, 24] splits different layers across devices with micro-batches pipelined to the different layers. To enable continued scaling of neural networks, improving model training and serving efficiency has become a critical research area.\n\nConditional Computation: Computation decisions can be made dynamically based on the input [23, 25]. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation, by activating certain parameters and computation on demand, on a per-example or per-token basis [3]. Conditional convolution layers [1] with task-specific gating has been used to combat catastrophic forgetting when a sequence of learning problems are optimized. The gating decisions may be binary or sparse and continuous, stochastic or deterministic.\n\nMixture of Experts: Sparsely-gated MoE [31] is the first model to demonstrate massive improvements in model capacity, training time, or model quality with gating. Switch Transformer [10] simplifies the gating by selecting only the top expert per token using a softmax over the hidden state and demonstrates better scaling than previous work. All the prior work requires an auxiliary loss to explicitly encourage balancing. This loss term has to be carefully weighted to not overwhelm the primary loss. However, auxiliary loss does not guarantee balancing and a hard capacity factor has to be imposed. As a result, many tokens can still be unprocessed by the MoE layer. Hard MoE [12] with a single decoding layer can be efficiently trained to good effect on large scale hashtag prediction tasks.\n\nBase Layers [22] formulate a linear assignment that maximizes token-expert affinities while ensuring each expert receives an equal number of tokens. Hash layers [8, 28] devise hashing techniques on input tokens. However, the evaluations are limited to pre-training perplexity. THOR [? ] randomly activates experts during training and inference and is trained with a consistency regularization loss.\n\nTHOR has demonstrated strong performance on translation tasks. Different from these prior works, our method is a learnt method that enables heterogeneous MoE and effectively improves downstream fine-tuning performance.\n\n3 Method\n\nWe first identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models and then present our method using expert choice to tackle these problems.\n\n3.1 Pitfalls Of Token-Choice Routing\n\nMoE can be computationally advantageous compared to a dense model, a routing strategy must be used to assign each token to the most-suited experts. Conventional MoE models employ token-choice routing which independently selects the top-k experts for each token [10, 21, 31]. We argue that this strategy has a few pitfalls that lead to sub-optimal training.\n\nLoad Imbalance: Token-choice routing often lead to poor load balancing across experts. That is, some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts can be under specialized because a lot of model capacity in the under-utilized experts are wasted. On the other side, some tokens will not be processed, since over-utilized experts can only take a maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can also hurt step latency, thus inference time, as the step latency can be determined by the most loaded expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this auxiliary loss does not guarantee a balanced load, especially during the important early stages of training. Indeed, we empirically observe that the over-capacity ratio can reach 20%–40% for some experts in token choice routing, indicating that a significant portion of the tokens routed to these experts will be dropped.\n\nUnder Specialization: Each MoE layer uses a gating network to learn token-to-expert affinity.\n\nIdeally, the learnt gating network should produce the affinity such that similar or relevant tokens are routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that are not sufficiently specialized. Under specialization may result by imposing an large auxiliary loss which favors more load balanced but less effective routing. Finding the right balance on the auxiliary loss to promote both load balancing and specialization is challenging for token-choice routing.\n\nSame Compute for Every Token: Finally, in a token-choice strategy each token receives exactly k experts and therefore occupies the same amount of compute. We hypothesize that this is not necessary nor desired. Instead, a MoE model should flexibly allocate its compute resource based on the complexity of the input. Motivated by the aforementioned observations, we next describe a simple yet effective method which produces load balanced assignments based on expert choice.\n\n3.2 Heterogeneous Moe Via Expert Choice\n\nDifferent from conventional routing, an expert choice method independently selects top-k tokens for each expert, where k is a fixed expert capacity (i.e. the number of tokens each expert can take).\n\nDespite its simplicity, expert choice achieves perfect load balancing by design. It also enables more flexible allocation of model compute since tokens can be received by a variable number of experts.\n\n$$k={\\frac{n\\times c}{c}}$$\n\ne(1) where n is the total number of tokens in the input batch (such as batch size × sequence length), c is the capacity factor, and e is the number of experts. The capacity factor c denotes on average how many experts are utilized by a token. Given input token representations X ∈ R n×d where d is the model hidden dimension, our method produces a token-to-expert assignment denoted by three output matrices I, G and P. The matrix I is an index matrix where I[i, j] specifies j-th selected token of the i-th expert. The gating matrix G ∈ R e×k denotes the weight of expert for the selected token, and P ∈ R e×k×n refers to an one-hot version of I that will be used to gather tokens for each expert.\n\nThese matrices are computed using a gating function,\n\n$$\\begin{array}{l l}{{S=\\mathrm{Softmax}(X\\cdot W_{g}),}}&{{S\\in\\mathbb{R}^{n\\times e}}}\\ {{G,I=\\mathrm{TopK}(S^{\\top},k),P=\\mathrm{Onehot}(I)}}\\end{array}$$\n\n$$(2)$$\n\nwhere S denotes the token-to-expert affinity scores, Wg ∈ R d×e denotes the expert embeddings, and T opK() selects the k largest entries for each row of S>.\n\nSimilar to Switch Transformer [10] and GShard [21], we apply mixture of experts and the gating function in the dense feed-forward (FFN) layer, as it is the most computationally expensive part in a Transformer-based network. The input to the gated FFN, denoted by Xin ∈ R e×k×d, is produced using the permutation matrix P. Here Xin[i] ∈ R k×d denotes the input of the i-th expert. Similarly, let W1 and W2 denote the parameters of gated FFN in which W1[i] and W2[i] ∈ R d×d 0denote the parameter matrices of the i-th expert. We compute the output of each expert Xe[i] as follows, $$X_{i n}=P\\cdot X$$ $$\\forall i:\\ \\ X_{e}[i]=\\mathrm{{GeLU}}(X_{i n}[i]\\cdot W_{1}[i])\\cdot W_{2}[i]^{\\top}$$\n\n(3) We omit the bias terms here for brevity. The finally output of the gated FFN layer Xout ∈ R n×dcan be obtained given Xe, the permutation and gating matrices P and G,\n\n$$({\\mathfrak{I}})$$ $$X_{\\mathrm{out}}[l,d]=\\sum_{i,j}P[i,j,l]\\;G[i,j]\\;X_{e}[i,j,d]$$ $$(4)$$\n\nP[i, j, l] G[i, j] Xei, j, d Both Xe and Xout can be efficiently computed using Einstein summation (einsum) operations.\n\n3.3 Expert Choice With Additional Constraint\n\nWe also consider regularizing our expert choice routing by limiting the maximum number of experts for each token. We are interested in whether adding this constraint improves pre-training and finetuning results. More importantly, it helps analyzing to what degree using a variable number of experts per token affects the model performance.\n\nLet A ∈ R e×n be a positive matrix where A[i, j] represents whether the i-th expert selects j-th token.\n\nWe solve the following entropy-regularized linear programming problem\n\n$$\\begin{array}{l}{{\\operatorname{max}{A}\\;\\left\\langle S^{\\top},A\\right\\rangle+\\lambda H(A)}}\\ {{\\forall i:\\;\\sum{j^{\\prime}}A[i,j^{\\prime}]=k;\\;\\;\\forall j:\\;\\sum_{i^{\\prime}}A[i^{\\prime},j]\\leq b;\\;\\;\\forall i,j:\\;0\\leq A[i,j]\\leq1}}\\end{array}$$ $$\\mathbf{s.t.}$$ s.t. ∀i : where < S>, A >* denotes the inner product, H(A) is the sum of element-wise entropy1, and b > 0 is an integer that upper bounds the selection for each token. Adding a small entropy term gives a near-integer solution while enabling a fast iterative solver we can run on TPUs. Specifically, the solution space is the intersection of three convex sets each satisfying one of the linear constraints.\n\nWe use Dykstra's algorithm [9] that alternatively projects the intermediate solution onto one of the convex sets.2 After A is computed, the routing indices I is selected using T opK(A, k) instead.\n\n1H(A) = Pij −A[i, j] log A[i, j] 2We use λ = 0.001 and a maximum of 100 iterations.\n\nModel Type nparams nact-params L M H nheads dhead E 0.1B Dense 130M 130M - 0.1B/16E MoE 548M 145M 16 0.1B/32E MoE 1.0B 145M 12 768 3,072 12 64 32 0.1B/64E MoE 1.9B 145M 64 0.1B/128E MoE 3.7B 145M 128 8B Dense 8.7B 8.7B 32 4,096 16,384 32 128 - 8B/64E MoE 143B 9.8B 64\n\n3.4 Model Architecture\n\nAt the high level, we adopt the idea of sparsely activated Mixture-of-Experts (MoE) [31]. We use a Transformer architecture and replace the feed-forward component of every other Transformer layer with a MoE layer, following recent practice [10, 21]. Interleaving regular Transformer layers and MoE layers empirically improves model performance and training efficiency, probably because forcing some shared components in between MoE layers can mitigate the negative effects of skipping tokens. Several additional modifications adopted in recent work have been applied in our experiments. For example, we replace the standard positional embedding with per-layer relative positional bias [5].\n\nIn the non-MoE feed-forward sub-layers (only every other layers are MoE layers), we replace the first linear projection and the activation function with the Gated Linear Unit [6], which computes the component-wise product of two linear transformation of the input, followed by a Gaussian Error Linear Unit [15] activation function.\n\nAs described earlier, each MoE layer consists of a group of independent feed-forward networks as denoted as \"experts\". The gating function in Eq. (2) uses a softmax activation function to model a probability distribution over these experts. This distribution denotes the preference over experts of each incoming token, which is computed similarly in a conventional gating network [10, 21, 31]. During training, each MoE layer's learnable gating network described in Eq. (2) is trained to use the input to activate the best subset of experts using a top-k function along the token dimension. An \"shuffle\" stage and an \"unshuffle\" stage are inserted to the MoE layer, where the first stage gathers the tokens to their designated experts while the second stage permutes the tokens back to their original order in the input batch. This step is formulated in Eq. (3) and Eq. (4).\n\nSimilar to conventional MoE method, there are more parameters in the MoE layer. However, the activated model size per token can be comparable to a dense layer because during training or inference, only a limited subset of experts is activated for any given token. For instance, Switch Transformer [10] has only one activated expert while GShard [21] uses two experts per token. In our method, the number of activated experts can vary for each token but the overall computation is kept the same as the baseline architectures by fixing the capacity factor c in Eq. (1). Unless otherwise specified, we set c = 2 such that our method can be directly compared to the top-2 token-choice gating in GShard.\n\nWe train several variants of our architecture at the 100M scale (i.e. 100M expert size) by increasing the number of experts to understand the scaling effects of our method. We also train a 8B scale MoE model. The large MoE model is partitioned with a 2D sharding algorithm as presented in GSPMD [36], which fully exploits the 2D topology of the TPU cluster [19]. Across different scales and setups, our method outperforms related work and demonstrates strong downstream task performance on selected tasks in GLUE and SuperGLUE.\n\n4 Experiments 4.1 Setup\n\nTable 1 summarizes the hyperparameter settings of different MoE models. As a reference point, we also include the respective dense model configurations with comparable numbers of activated parameters per-token during inference. To study of the effect of scaling the number of experts, we\n\nstudied varying the number of experts but fixing the per expert size to 100M parameters. For example, 0.1B/64E represents the architecture of an approximately 100M parameter dense model with every other layer replaced by a 64-expert MoE layer. The MoE model degenerates into a dense transformer architecture when each MoE layer only has one expert. While nparams is the total number of trainable parameters, nact−params represents the number of activated parameters per token. L is the total number of Transformer layers, M is the model dimension, H is the hidden dimension after the projection in each transformer layer, nheads is the number of attention heads, and dhead is the hidden dimension of each attention head.\n\nDataset: We use the high-quality dataset from GLaM [? ] of 1.6 trillion tokens that are representative of a wide range of natural language use cases. An in-house classifier is trained to classify between a collection of curated text and other webpages and estimate the content quality of a webpage. A high-quality filtered subset of webpages are combined with books, Wikipedia pages, conversations, forums, and news to create the final dataset. The data and mixture weights can be found in Table 3 in the GLaM paper.\n\nModel Training: Our model training follows the setups of GLaM [? ] where a maximum sequence length of 1024 tokens is adopted. We use an Adafactor optimizer [32] with first-moment decay β1 = 0 and second-moment decay β2 = 0.99. We keep the learning rate constant for the first 10K training steps, and then decay it with an inverse square root schedule. Unlike most related works, we do not impose any auxiliary loss for load balance, such as described in Switch Transformer [10] and GShard [21]. We use the SentencePiece subword tokenizer with a vocabulary of size of 256K. The largest model (8B/64E) is trained on 512 TPU V4 chips. We use a dropout rate of 0 during training as the number of tokens in the training data corpus is much greater than the total number of tokens during training.\n\nModel Evaluation: We mainly focus on evaluating the finetuning performance on the 11 selected tasks from GLUE and SuperGLUE benchmarks [34, 35].\n\n4.2 Training Efficiency\n\nWe first study training efficiency and convergence. We use expert choice with a capacity factor of 2 (EC-CF2) to match the activated model size and computational cost on a per token basis in GShard top-2 gating and run both for a fixed number of steps. The results are shown in Fig. 2 (a). Comparing to GShard top-2 gating, which showed stronger performance in both perplexity in the evaluation dataset and fine-tuning on downstream tasks compared to Switch Transformer top-1 gating, EC-CF2 converges more than 2x faster during training. More specifically, EC-CF2 reaches the same perplexity as GShard top-2 in less than half the steps, and with each GShard top-2 step being 20% slower than our method. As explained in Section 3.1, the slower step time in top-2 gating is due to load imbalance\n\n100M/128E 100M/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 77.4 76.5 76.9 73.2 77.5 79.7 CB acc dev 87.5 80.9 89.1 85.9 84.4 89.1 CoLA acc dev 78.9 84.0 86.7 64.1 85.2 88.3 MNLI acc dev 82.3 83.6 84.9 80.8 85.2 86.7 MRPC acc dev 82.6 81.0 83.1 81.3 81.3 84.4 QNLI acc dev 89.5 88.6 89.0 89.4 89.7 91.3 QQP acc dev 90.6 90.3 90.4 88.9 90.5 91.0 RTE acc dev 77.0 78.9 78.5 74.1 79.3 81.6 SST2 acc dev 92.0 94.5 94.6 91.8 95.1 95.1 WiC acc dev 67.8 65.5 68.1 64.4 67.8 65.6 WNLI acc dev 65.6 70.3 67.2 68.8 68.8 71.7 Avg - - 81.0 81.3 82.6 78.4 82.2 84.0 100M/32E 8B/64E Name Metric Split ST Top-1 GS Top-2 EC-CF2 ST Top-1 GS Top-2 EC-CF2 BoolQ acc dev 74.5 79.0 79.3 89.1 89.5 89.2 CB acc dev 80.6 81.3 92.2 93.8 96.7 100 CoLA acc dev 87.5 92.2 93.8 88.3 87.5 89.1 MNLI acc dev 83.1 87.8 88.0 90.7 91.4 91.1 MRPC acc dev 82.3 85.2 84.4 89.3 91.7 90.6 QNLI acc dev 91.6 91.9 92.5 94.5 94.9 95.0 QQP acc dev 90.1 91.5 92.0 92.1 92.5 93.8 RTE acc dev 75.0 79.1 78.1 91.0 92.2 95.2 SST2 acc dev 93.3 94.4 95.4 97.1 98.0 97.7 WiC acc dev 62.5 65.9 69.8 74.5 76.4 83.8 WNLI acc dev 65.6 64.1 68.8 78.1 82.8 92.8 Avg - - 80.6 83.5 85.0 88.9 90.3 92.6\n\nTable 2: Expert choice with capacity factor of 2 (EC-CF2) outperforms Top-1 gating in Switch Transformer (ST) and top-2 gating in GShard (GS) on GLUE and SuperGLUE tasks. Note that with an expert size of 100M parameters, 100M/32E works best for our method and Ghard Top-2 while 100M/128E works better for Switch Transformer Top-1. Our method consistently outperforms the others across all the scales.\n\nwhere some experts can receive a lot more tokens than the desired capacity. As a result, the step latency will be bottlenecked by the most loaded expert.\n\n4.3 Scaling The Number Of Experts 7\n\nAs presented in Table 1, increasing the number of experts effectively increases model capacity without increasing activated model size. We scale the number of experts while fixing the expert size to 100M parameters for both expert choice (EC) and GShard (Top-2) methods and find both methods work well in terms of perplexity on the evaluation dataset during pre-training. As demonstrated in Fig. 2 (b), having more experts consistently improves training perplexity.\n\n4.4 Fine-Tuning On Glue And Superglue\n\nTo validate whether improved perplexity directly translates to better performance in downstream tasks, we perform fine-tuning on 11 selected tasks from GLUE and SuperGLUE. We compare three MoE methods including Switch Transformer top-1 gating (ST Top-1), GShard top-2 gating (GS Top-2) and our method (EC-CF2) that matches the activation memory size and computational cost of GS Top-2. Indicated by the results in Table 2, our EC-CF2 method consistently outperforms the related methods and yields more than 2% average accuracy increase in a large 8B/64E setting. Table 3 further compares our 8B/64E model against its dense counterpart. Again, our method achieves stronger fine-tuning results, increasing the average score by 3.4 point.\n\nInterestingly, we observe the 100M/32E model setting works the best for both GS Top-2 and EC-CF2, even though the effective model capacity is smaller than that of 100M/64E and 100M/128E. This result indicates that a good training perplexity does not always translate to better performance of downstream tasks.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 3: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nFigure 3: Distribution of the number of experts routed to per token in a 100M/64E model.\n\nLayer. Method Max # of Experts Avg acc. EC-CAP2 2 83.2 ± 0.4 EC-CAP3 3 84.0 ± 0.4 EC-CF2 - 84.0 ± 0.2 Hash Layer - 81.3 ± 0.1\n\n4.5 Heterogeneity Matters\n\nCapped Expert Choice: We regularized expert choice by limiting the maximum number of experts for each token, using the method described in Section 3.3. Table 4 reports the average accuracy on the 11 selected datasets. EC-CAP2 is the variant of our expert choice method by limiting the number of experts of each token to 2. This decreases the fine-tuning accuracy by 0.8 points on average. In addition, EC-CAP3 allows a maximum of 3 experts per token and achieves on par results compared to the vanilla expert choice method. This ablation study confirms that allowing variable number of experts per token is indeed helpful.\n\nVariable Experts per Token: We compute statistics on token-to-expert routing, particularly on the ratio of tokens that have been routed to a certain number of experts. According to Fig. 3, a majority of tokens have been routed to one or two experts while 23% have been routed to three or four experts and only about 3% tokens have been routed to more than 4 experts. This plot verifies our hypothesis that our method learns to allocate a variable number experts to tokens, which can be beneficial for important tokens.\n\n4.6 Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [28]. We use mod x to map a token ID to an expert ID. This ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 4. Hashing based routing performs worse than expert choice in terms of average scores and variance. This indicates that load balancing alone does not generate all the benefits.\n\n4.7 Ablation\n\nCapacity Factor: We study the capacity factor in our expert choice method and compare the training perplexity with the baseline top-1 gating method used in Switch Transformer. As described in Eq. (1), the capacity factor determines how many experts on average each token can be routed to, thus the bucket size k of each expert. In all our previous experiments, we use a capacity factor of 2, which matches the computational footprint of the top-2 gating used in GShard method. To match the computation cost on a per-token basis fairly with top-1 gating used in Switch Transformer, we reduce the capacity factor to 1 and plot the training perplexity in Fig. 4 (a). Not surprisingly, using a smaller capacity factor yields higher perplexity, but our method still significantly outperforms top-1 gating.\n\nWe further push the capacity factor down to 0.5, and observe that it still outperforms the top-1 gating.\n\nComparison with Dense Models on Pre-training: We compare our method with dense models on pre-training. As shown in Fig. 4 (b), our method consistently outperforms the dense method in\n\nperplexity and convergence time. For a small expert size of 100M parameters, the benefit of sparse gating is even more significant. Orthogonal to results presented in Fig. 2 (b), where scaling the number of experts improves model performance, Fig. 4 (b) shows that increasing expert capacity also significantly increases model performance.\n\n5 Conclusion\n\nWe propose a new routing method for sparsely activated mixture-of-experts (MoE) models. This method addresses load imbalance and under-utilization of experts in conventional MoE methods, and enables selecting different numbers of experts for each token. Our model demonstrates more than 2x training efficiency improvements when compared to the state-of-the-art GShard and Switch Transformer models, and also achieves strong gains when finetuning on 11 datasets in the GLUE and SuperGLUE benchmark.\n\n6 Limitations\n\nThe expert choice method might not immediately apply to auto-regressive text generation as our current implementation takes in the past and future tokens to perform the top-k selection. One possible solution is to collect a large batch of input sequences, dispatch tokens of the same sequence into separate groups, and perform expert choice routing for each group. Another scenario where the expert choice method does not immediately apply is when the batch size becomes very small during serving or inference. A global top-k can be selected instead and we can cap the number of times each expert or token gets selected. We leave these possible improvements for future work.\n\nAnother long-standing issue with MoE has been the large memory footprint. Even though computational cost can be reduced using sparsely gated networks, the total number of parameters increases linearly or sub-linearly with the number of experts. Increasing the number of experts requires reservation of a large number of hardware devices. Therefore, dynamic (used) power is saved while static (reserved) power is not. Power saving techniques such as the ability to put hardware devices into low power states while not in use [17] can help with reducing the reserved power requirements.\n\nReferences\n\n[1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In CVPR, pages 3930–3939. Computer Vision Foundation / IEEE, 2020.\n\n[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. In Advances in Neural Information Processing Systems.\n\n[3] Kyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning, 2014.\n\n[4] Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. CoAtNet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems, 2021.\n\n[5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.\n\nTransformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.\n\n[6] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 933–941. JMLR.org, 2017.\n\n[7] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixtureof-experts, 2021.\n\n[8] Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan.\n\nTricks for training sparse translation models, 2021.\n\n[9] Richard L Dykstra. An iterative procedure for obtaining i-projections onto the intersection of convex sets. The annals of Probability, pages 975–984, 1985.\n\n[10] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.\n\n[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. NAS-FPN: learning scalable feature pyramid architecture for object detection. In CVPR, pages 7036–7045. Computer Vision Foundation / IEEE, 2019.\n\n[12] Sam Gross, Marc'Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision, 2017.\n\n[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.\n\n[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 630–645, Cham, 2016. Springer International Publishing.\n\n[15] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2016.\n\n[16] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.\n\n[17] Ping Huang, Zuocheng Xing, Tianran Wang, Qiang Wei, Hongyan Wang, and Guitao Fu. A brief survey on power gating design. In 2010 10th IEEE International Conference on Solid-State and Integrated Circuit Technology, pages 788–790, 2010.\n\n[18] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 103–112, 2019.\n\n[19] Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David A. Patterson. A domain-specific supercomputer for training deep neural networks. Commun. ACM, 63(7):67–78, 2020.\n\n[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.\n\n[21] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.\n\n[22] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6265–6274. PMLR, 18–24 Jul 2021.\n\n[24] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. New York, NY, USA, 2019. Association for Computing Machinery.\n\n[25] Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, Cédric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. In ICLR. OpenReview.net, 2021.\n\n[26] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.\n\n[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.\n\n[28] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models, 2021.\n\n[30] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 10435–10444, Red Hook, NY, USA, 2018. Curran Associates Inc.\n\n[31] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E.\n\nHinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. In ICLR (Poster). OpenReview.net, 2017.\n\n[32] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR, 10–15 Jul 2018. [23] Min Lin, Jie Fu, and Yoshua Bengio. Conditional computation for continual learning, 2019.\n\n[29] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai, 2019.\n\n[33] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.\n\n[34] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc.\n\n[35] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.\n\nGLUE: A multi-task benchmark and analysis platform for natural language understanding.\n\nIn Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, November 2018. Association for Computational Linguistics.\n\n[36] Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake A. Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen. GSPMD: general and scalable parallelization for ML computation graphs. CoRR, abs/2105.04663, 2021.\n\n7 Checklist\n\n(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Yes (b) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes (c) Did you discuss any potential negative societal impacts of your work? N/A. Not any. (d) Did you describe the limitations of your work? Yes (a) Did you include the code, data, and instructions needed to reproduce the main experimental results? Yes. We include details in the experiment setup to help reproduce the main results. (b) Did you specify all the training details? Yes (c) Did you report error bars? Yes (d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? Yes (a) If your work uses existing assets, did you cite the creators? Yes (b) Did you mention the license of the assets? No. The used dataset is not released yet. (c) Did you include any new assets either in the supplemental material or as a URL? No. The dataset is not released yet.\n\n(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? No. Not using persons' data. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes. The dataset does not contain any personally identifiable information or offensive content.\n\nA Comparison On Fine-Tuning With A Dense Model\n\nOur 8B MoE model achieves stronger pre-training perplexity than its dense counterpart. However, a better perplexity does not always directly translate to downstream performance as demonstrated in Section 4.4. To this end, we compare fine-tuning performance of the 8B dense model and MoE model in Table 1. As shown in the table, our MoE model using expert choice routing consistently outperforms the dense model across the 11 tasks in GLUE and SuperGLUE.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Dense 8B 88.2 100 86.4 91.3 86.7 94.7 91.2 92.2 97.2 75.6 78.1 89.2 EC-CF2 8B/64E 89.2 100 89.1 91.1 90.6 95.0 93.8 95.2 97.7 83.8 92.8 92.6\n\nTable 1: Comparison between Dense 8B and Expert Choice (EC-CF2) 8B/64E models: Our method significantly outperforms the dense model in downstream tasks.\n\nB Capacity Factor\n\nWe evaluate the downstream task fine-tuning performance by varying the capacity factors. Note that a capacity factor of n indicates on average how many experts each token can be received. EC-CF2 is our baseline expert choice, which matches GShard top-2 gating computational footprint. EC-CF1, however, matches Switch Transformer top-1 gating computational footprint. EC-CF0.5 further verifies that an aggressively lowered capacity factor can provide strong enough performance, that almost matches the top-2 gating baseline.\n\nModel BoolQ CB CoLA MNLI MRPC QNLI QQP RTE SST2 WiC WNLI Avg Top-2 78.1 87.0 88.3 85.0 82.6 90.1 90.7 81.6 94.7 68.2 67.2 83.0±0.3 EC-CAP2 78.2 88.0 88.5 85.7 83.0 90.8 91.1 80.0 95.4 70.4 64.1 83.2±0.4 EC-CAP3 78.5 91.7 89.3 86.3 83.5 90.9 91.1 81.8 94.9 70.0 65.6 84.0±0.4 EC-CF2 79.1 89.6 89.3 86.8 84.3 91.3 91.2 81.1 95.2 68.1 68.0 84.0±0.2 EC-CF1 77.4 90.6 88.0 85.5 83.6 90.3 91.2 79.8 95.3 66.5 64.9 83.0±0.2 EC-CF0.5 77.4 89.6 86.3 85.2 82.7 91.7 91.0 79.6 94.9 67.3 63.5 83.0 ±0.05 Hash Layers 76.1 85.2 86.7 83.4 82.5 90.0 90.3 75.7 94.0 67.4 63.3 81.3±1.0\n\nTable 2: Comparison between different routing methods in fine-tuning of 100M/64E models. We perform 3 independent fine-tuning runs for each method and report the average results. This gives more accurate difference between the variants of expert choice method, since they achieve close fine-tuning results. We do not report averaged results in other experiments.\n\nC Capped Expert Choice\n\nAs described in Section 4.5, the maximum number of experts each token is assigned can be capped by an entropy-regularized linear programming. Figure 1 compares the validation perplexity when training the 100M/64E models using the base expert choice method (EC-BASE), expert choice capped by two experts per token (EC-CAP2), expert choice capped by three experts per token (EC-CAP3), and GShard top-2 gating.\n\nAs shown in the figure, restricting the number of experts to 2 degrades the perplexity compared to the base expert choice method. This suggests that a more flexible allocation of experts (e.g. more than 2 experts for a token) can enhance model expressiveness. On the other hand, our EC-CAP2 and EC-CAP3 methods still outperform the top-2 gating method by a clear margin. We believe this confirms the effectiveness of a load balanced training, provided by our method. Finally, EC-CAP3 obtains comparable perplexity to EC-BASE. As indicated by Figure 3, only a little fraction of tokens use more than 3 experts therefore we see little or no difference between EC-BASE and EC-CAP3 variants. We present the fine-tuning results of these methods in Table 2.\n\nD Comparison With Hash Layer\n\nIn this section, we compare our method with Hash Layers [? ]. We use mod x to map a token ID to an expert ID. This in some way ensures load balance and generates specialized experts. The fine-tuning results are presented in the last row in Table 2. Hashing based routing performs much worse than expert choice in terms of average scores and variance.\n\nE Fine-Tuning Details\n\nWe did a hyperparameter search for both baseline models and expert choice method. For fine-tuning of the 8B dense model, we use a constant learning rate of 0.0001 and a dropout rate of 0.1. We freeze the attention layer and feed-forward layer while leaving the embedding and layer normalization trainable. This setting has been found optimal for the 8B dense model. For MoE 8B/64E models including GShard top-2 gating and expert choice, we found continuing the learning rate from the pre-trained model while using a square root learning rate decay works better. In addition, we do not apply parameter freezing for fine-tuning MoE models. For models with 100M expert size, we use a constant learning rate of 0.0001 and no dropout is used.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/2202.09368v2.md"
+          },
+          "headlines": {
+            "1 Introduction": [],
+            "2 Related Work": [],
+            "3 Method": [
+              "3.1 Pitfalls Of Token-Choice Routing",
+              "3.2 Heterogeneous Moe Via Expert Choice",
+              "3.3 Expert Choice With Additional Constraint",
+              "3.4 Model Architecture"
+            ],
+            "4 Experiments": [
+              "4.1 Setup",
+              "4.2 Training Efficiency",
+              "4.3 Scaling The Number Of Experts",
+              "4.4 Fine-Tuning On Glue And Superglue",
+              "4.5 Heterogeneity Matters",
+              "4.6 Comparison With Hash Layer",
+              "4.7 Ablation"
+            ],
+            "5 Conclusion": [],
+            "6 Limitations": [],
+            "A Comparison On Fine-Tuning With A Dense Model": [],
+            "B Capacity Factor": [],
+            "C Capped Expert Choice": [],
+            "D Comparison With Hash Layer": [],
+            "E Fine-Tuning Details": []
+          },
+          "summary": "The paper introduces a new routing method for sparsely-activated Mixture-of-Experts (MoE) models called 'expert choice,' which addresses load imbalance and under-utilization issues in conventional MoE methods. Unlike traditional token-choice routing, expert choice allows experts to select the top-k tokens, ensuring perfect load balancing and enabling variable expert allocation per token. This method improves training efficiency, achieving over 2x faster convergence compared to existing models like Switch Transformer and GShard. It also demonstrates superior performance on downstream tasks in the GLUE and SuperGLUE benchmarks. The study highlights the benefits of flexible expert allocation and shows that expert choice outperforms hashing-based routing methods. The paper also discusses potential limitations and future improvements for the expert choice method.",
+          "summary_embedding": [
+            -0.009726986289024353,
+            -0.0337262786924839,
+            -0.023302342742681503,
+            -0.019878201186656952,
+            -0.025953780859708786,
+            0.04560471698641777,
+            -0.006874797400087118,
+            0.039665497839450836,
+            -0.006973279174417257,
+            0.034544438123703,
+            -0.025787118822336197,
+            -0.00363057479262352,
+            -0.01052999310195446,
+            0.0054657477885484695,
+            -0.024499278515577316,
+            0.036786794662475586,
+            0.010007280856370926,
+            0.003272630739957094,
+            0.006102093029767275,
+            -0.04048365354537964,
+            -0.018666116520762444,
+            -0.013522328808903694,
+            -0.03230207785964012,
+            0.008969432674348354,
+            0.02484775334596634,
+            0.0016713523073121905,
+            0.0026230288203805685,
+            0.003624893259257078,
+            -0.05311964824795723,
+            0.04339266195893288,
+            0.018681267276406288,
+            0.023711422458291054,
+            -0.016332851722836494,
+            -0.012832955457270145,
+            -0.039120059460401535,
+            -0.0197418425232172,
+            0.030786970630288124,
+            0.039968520402908325,
+            -0.019287310540676117,
+            0.03357476741075516,
+            0.018060073256492615,
+            0.02454473078250885,
+            -0.021272100508213043,
+            -0.029817303642630577,
+            -0.0061892117373645306,
+            -0.007840678095817566,
+            -0.008355814032256603,
+            0.01242387667298317,
+            0.016984347254037857,
+            -0.013007191941142082,
+            0.0212569497525692,
+            -0.01371171697974205,
+            0.03384748846292496,
+            0.008916404098272324,
+            -0.042332086712121964,
+            0.03984731063246727,
+            -0.008007340133190155,
+            0.012105703353881836,
+            0.03281721472740173,
+            0.010333028621971607,
+            -0.005287723150104284,
+            -0.021544819697737694,
+            -0.03378688171505928,
+            0.0010529992869123816,
+            0.00014014738553669304,
+            0.005950582213699818,
+            -0.003069985192269087,
+            -0.009742137044668198,
+            0.03227177634835243,
+            -0.003939277958124876,
+            -0.0740584209561348,
+            0.050089433789253235,
+            -0.026590125635266304,
+            -0.007628562860190868,
+            -0.028938541188836098,
+            -0.01736312545835972,
+            0.037362534552812576,
+            0.005662711802870035,
+            0.0098709212616086,
+            0.0106284748762846,
+            -0.0007892759749665856,
+            0.028923390433192253,
+            0.00910579226911068,
+            -0.024923507124185562,
+            0.04478655755519867,
+            -0.04790768027305603,
+            0.01801462098956108,
+            -0.04336235672235489,
+            0.022756904363632202,
+            1.6393929399782792e-05,
+            -0.020726662129163742,
+            -0.0013304532039910555,
+            -0.016666175797581673,
+            0.0218023881316185,
+            -0.01730252057313919,
+            0.026802239939570427,
+            0.013825350441038609,
+            -0.015802564099431038,
+            -0.029817303642630577,
+            0.025378039106726646,
+            0.003367325058206916,
+            0.010499690659344196,
+            -0.02075696364045143,
+            0.014332910999655724,
+            0.04393810033798218,
+            -0.02593863010406494,
+            -0.02271145209670067,
+            0.027681002393364906,
+            0.014408666640520096,
+            0.0633314698934555,
+            0.036544378846883774,
+            0.006624804809689522,
+            0.022362977266311646,
+            0.020575150847434998,
+            -0.028968842700123787,
+            -0.01963578537106514,
+            -0.046362269669771194,
+            -0.04757435619831085,
+            0.019878201186656952,
+            -0.04566532000899315,
+            0.0050869714468717575,
+            0.038544319570064545,
+            -0.013704141601920128,
+            -0.014446543529629707,
+            -0.04021093621850014,
+            0.017847958952188492,
+            0.0003908502112608403,
+            0.011976920068264008,
+            -0.0022082682698965073,
+            0.00033285003155469894,
+            -0.011825408786535263,
+            0.008878526277840137,
+            -0.03778676688671112,
+            -0.025196226313710213,
+            0.025181075558066368,
+            0.022862963378429413,
+            0.024120500311255455,
+            0.028802180662751198,
+            -0.03077181987464428,
+            -0.009984553791582584,
+            0.021378157660365105,
+            -0.04242299124598503,
+            -0.006556625012308359,
+            0.011332998983561993,
+            0.0026703758630901575,
+            0.003872992005199194,
+            0.032786913216114044,
+            0.052998438477516174,
+            -0.03172633796930313,
+            0.02065090648829937,
+            -0.010431510396301746,
+            0.04957429692149162,
+            0.00024691506405360997,
+            0.028968842700123787,
+            -0.019514575600624084,
+            -0.0134389977902174,
+            -0.029529431834816933,
+            -0.009795165620744228,
+            -0.01356020662933588,
+            -0.008484598249197006,
+            -0.002998017705976963,
+            -0.009575475007295609,
+            0.0040453351102769375,
+            0.0331808403134346,
+            0.00273855566047132,
+            0.030680913478136063,
+            0.014325335621833801,
+            0.0051475754007697105,
+            -0.08745196461677551,
+            0.026059838011860847,
+            -0.0033616432920098305,
+            0.005859675817191601,
+            0.0062460280023515224,
+            0.03013547509908676,
+            -0.0312718041241169,
+            0.020938776433467865,
+            0.026620427146553993,
+            0.0011145505122840405,
+            0.016969196498394012,
+            0.036847397685050964,
+            -0.04181694984436035,
+            0.009643655270338058,
+            0.012590537779033184,
+            -0.0033881578128784895,
+            0.04584713280200958,
+            0.011628445237874985,
+            0.019514575600624084,
+            0.018529755994677544,
+            0.003569970605894923,
+            0.00297907879576087,
+            0.006533898413181305,
+            0.03724132478237152,
+            0.003462019143626094,
+            -0.009833043441176414,
+            0.014976831153035164,
+            -0.010757259093225002,
+            -0.011219366453588009,
+            -0.037998881191015244,
+            0.014454119838774204,
+            0.008742166683077812,
+            -0.017454031854867935,
+            -0.02959003672003746,
+            0.0026949963066726923,
+            -0.031908150762319565,
+            0.026559824123978615,
+            -0.002445003716275096,
+            0.005738467443734407,
+            -0.010454237461090088,
+            -0.006757376715540886,
+            -0.035423196852207184,
+            0.028453707695007324,
+            -0.02095392718911171,
+            0.0053142374381423,
+            0.0007258309051394463,
+            -0.029438525438308716,
+            0.0060073984786868095,
+            0.017863109707832336,
+            -0.0025965142995119095,
+            0.0025302283465862274,
+            -0.02349930815398693,
+            0.024468975141644478,
+            0.025635607540607452,
+            0.004518806003034115,
+            -0.014037464745342731,
+            -0.013113250024616718,
+            -0.003605954349040985,
+            -0.009742137044668198,
+            -0.016196493059396744,
+            0.024468975141644478,
+            -0.014810169115662575,
+            0.00308892410248518,
+            0.021847840398550034,
+            0.014264730736613274,
+            -0.007045246660709381,
+            -0.004007457755506039,
+            -0.04066546633839607,
+            -0.01218903437256813,
+            -0.04760465770959854,
+            -0.011628445237874985,
+            -0.0034601252991706133,
+            -0.028696123510599136,
+            0.0061513339169323444,
+            -0.04481685906648636,
+            -0.01235569640994072,
+            0.02309022843837738,
+            0.018135828897356987,
+            0.0019488062243908644,
+            -0.009173971600830555,
+            -0.022605394944548607,
+            8.486965816700831e-05,
+            -0.01763584464788437,
+            -0.017044952139258385,
+            0.015393485315144062,
+            0.031059689819812775,
+            -0.03101423755288124,
+            0.04024123772978783,
+            0.003893824527040124,
+            -0.08066429197788239,
+            0.010393633507192135,
+            -0.026362858712673187,
+            0.07011914253234863,
+            -0.0025453795678913593,
+            0.014075342565774918,
+            0.06781618297100067,
+            -0.012431452050805092,
+            -0.02007516659796238,
+            0.03142331540584564,
+            -0.008249756880104542,
+            -1.1311515663692262e-05,
+            -0.02884763479232788,
+            -0.02739313244819641,
+            -0.053271155804395676,
+            -0.009704259224236012,
+            -0.010787560604512691,
+            0.03360506892204285,
+            -0.024317465722560883,
+            -0.0007656024536117911,
+            0.012620840221643448,
+            0.02983245439827442,
+            0.016635872423648834,
+            -0.0015681355725973845,
+            0.008272483013570309,
+            0.02881733328104019,
+            0.0274234339594841,
+            -0.0036154238041490316,
+            0.00814369972795248,
+            -0.034211114048957825,
+            -0.0068066176027059555,
+            -0.0493621826171875,
+            0.011302697472274303,
+            -0.02596893161535263,
+            -0.02596893161535263,
+            0.007643714081496,
+            -0.011476934887468815,
+            0.03748374432325363,
+            -0.024696242064237595,
+            -0.007503566797822714,
+            0.005037730094045401,
+            0.012855681590735912,
+            -0.017757052555680275,
+            -0.03612014651298523,
+            -0.0028502948116511106,
+            0.0026381798088550568,
+            0.007560383062809706,
+            -0.053240854293107986,
+            0.004499867558479309,
+            -0.018469152972102165,
+            0.0279688723385334,
+            -0.02139330841600895,
+            -0.013688990846276283,
+            0.042029064148664474,
+            0.011363301426172256,
+            0.027499189600348473,
+            -0.026468917727470398,
+            0.0017849853029474616,
+            0.012825379148125648,
+            -0.0020056227222085,
+            0.032514192163944244,
+            -0.030574856325984,
+            -0.03257479891180992,
+            0.001548249856568873,
+            -0.007840678095817566,
+            -0.0625436082482338,
+            -0.011386028490960598,
+            -0.00981789268553257,
+            0.017317671328783035,
+            0.0446956530213356,
+            0.04557441547513008,
+            -0.01068150345236063,
+            0.002174178371205926,
+            -0.02213571034371853,
+            -0.04542290419340134,
+            -0.02643861435353756,
+            0.0016486257081851363,
+            -0.019999410957098007,
+            -0.02789311669766903,
+            -0.0018313854234293103,
+            0.011211791075766087,
+            -0.013651113025844097,
+            0.01134057529270649,
+            -0.018560059368610382,
+            -0.01862066239118576,
+            -0.012332970276474953,
+            0.018575210124254227,
+            0.009901223704218864,
+            -0.0018796793883666396,
+            -0.015037435106933117,
+            -0.00980274099856615,
+            0.018969137221574783,
+            -0.011279970407485962,
+            0.0009388927719555795,
+            0.01188601367175579,
+            0.024347767233848572,
+            -0.007083124481141567,
+            0.010961798019707203,
+            -0.034756552428007126,
+            -0.01966608688235283,
+            0.007984613068401814,
+            0.002719616750255227,
+            0.020711511373519897,
+            0.008742166683077812,
+            -0.04275631532073021,
+            -0.044301725924015045,
+            0.016044981777668,
+            -0.0562104657292366,
+            0.028120383620262146,
+            -0.03327174484729767,
+            -0.012514783069491386,
+            -0.02671133354306221,
+            0.0013418165035545826,
+            -0.01828734017908573,
+            0.022438732907176018,
+            0.03778676688671112,
+            -0.0011486404109746218,
+            0.005503625608980656,
+            0.014658658765256405,
+            0.04384719207882881,
+            -0.05021063983440399,
+            -0.041180603206157684,
+            0.03754434734582901,
+            -0.01601468026638031,
+            0.0185146052390337,
+            0.00021744151308666915,
+            0.010348179377615452,
+            -0.014431392773985863,
+            0.022772056981921196,
+            0.013757170177996159,
+            -0.008764892816543579,
+            -0.030574856325984,
+            0.04945308715105057,
+            -0.023650817573070526,
+            -0.049513693898916245,
+            0.0018323323456570506,
+            0.018878230825066566,
+            0.018635815009474754,
+            -0.014870774000883102,
+            -0.02907489985227585,
+            -0.024317465722560883,
+            -0.006060427520424128,
+            -0.00947699323296547,
+            -0.03402930125594139,
+            0.011363301426172256,
+            0.0048445542342960835,
+            0.016666175797581673,
+            -0.00645056739449501,
+            -0.0073141781613230705,
+            -0.009370936080813408,
+            0.022559940814971924,
+            0.012635990977287292,
+            -0.03175663948059082,
+            -0.017484333366155624,
+            -0.02528713271021843,
+            0.004359719809144735,
+            0.010136065073311329,
+            0.022166013717651367,
+            0.024287162348628044,
+            -0.0323626808822155,
+            0.04218057543039322,
+            0.01550711877644062,
+            -0.008727015927433968,
+            -0.0019317612750455737,
+            0.02478714846074581,
+            0.04324115067720413,
+            0.0218023881316185,
+            0.026514369994401932,
+            0.002107892418280244,
+            8.705945947440341e-05,
+            -0.015620751306414604,
+            -0.0011391709558665752,
+            0.01862066239118576,
+            0.04911976307630539,
+            -0.012151156552135944,
+            0.01716616004705429,
+            -0.03578682243824005,
+            0.00396200455725193,
+            -0.02580226957798004,
+            -0.030786970630288124,
+            0.03745344281196594,
+            -0.015060162171721458,
+            0.0191964041441679,
+            0.025044716894626617,
+            -0.022469034418463707,
+            -0.011075431481003761,
+            -0.019423669204115868,
+            -0.026181045919656754,
+            -0.01702980138361454,
+            -0.03281721472740173,
+            -0.018135828897356987,
+            -0.002653330797329545,
+            -0.035635314881801605,
+            -0.04987731948494911,
+            -0.004329417832195759,
+            -0.046392571181058884,
+            -0.010507266037166119,
+            0.017378276214003563,
+            -0.011166337877511978,
+            -0.025711363181471825,
+            -0.005336964037269354,
+            0.003973367623984814,
+            -0.022196315228939056,
+            -0.0066588944755494595,
+            0.008439145050942898,
+            0.035938333719968796,
+            -0.02298417128622532,
+            -0.013060221448540688,
+            -0.0024317465722560883,
+            -0.02887793630361557,
+            -0.007878555916249752,
+            0.008999735116958618,
+            0.007723257411271334,
+            -0.0012357590021565557,
+            -0.05196816474199295,
+            -0.006916462909430265,
+            -0.012992041185498238,
+            0.029196109622716904,
+            0.026468917727470398,
+            0.0012168202083557844,
+            -0.02783251367509365,
+            0.008749742060899734,
+            -0.0072535742074251175,
+            0.02339324913918972,
+            -0.007992188446223736,
+            -0.022908415645360947,
+            -0.02034788578748703,
+            0.030786970630288124,
+            -0.0159086212515831,
+            0.009113367646932602,
+            0.009976978413760662,
+            0.0121360057964921,
+            -0.03899884968996048,
+            -0.001445980160497129,
+            0.0005648507503792644,
+            -0.010128489695489407,
+            0.00423093605786562,
+            -0.006738437805324793,
+            -0.007079336792230606,
+            -0.022090258076786995,
+            -0.0218023881316185,
+            0.03818069398403168,
+            -0.009113367646932602,
+            -0.0035661826841533184,
+            -0.01088604237884283,
+            0.03436262533068657,
+            -0.00595815759152174,
+            -0.01993880607187748,
+            -0.019347915425896645,
+            0.014431392773985863,
+            0.05105910077691078,
+            -0.028529461473226547,
+            -0.03984731063246727,
+            0.0011136035900563002,
+            0.030150625854730606,
+            0.004140029661357403,
+            -0.016908591613173485,
+            0.0035851215943694115,
+            0.056392278522253036,
+            -0.011742077767848969,
+            0.028362801298499107,
+            0.027771908789873123,
+            0.02295386977493763,
+            -0.02021152526140213,
+            -0.01678738370537758,
+            -0.020999381318688393,
+            0.011098157614469528,
+            -0.03027183562517166,
+            0.027059808373451233,
+            0.018969137221574783,
+            0.01538590993732214,
+            0.05054396390914917,
+            -0.03178694099187851,
+            -0.02515077404677868,
+            0.008499749936163425,
+            0.0021722845267504454,
+            0.011007251217961311,
+            -0.007727045100182295,
+            -0.01801462098956108,
+            -0.001950700068846345,
+            -0.014022313989698887,
+            -0.019817598164081573,
+            -0.020938776433467865,
+            0.00954517349600792,
+            0.015696506947278976,
+            0.002649543108418584,
+            0.010863316245377064,
+            0.0016590419691056013,
+            0.01686313934624195,
+            0.01321930717676878,
+            -0.0444229319691658,
+            -7.936555630294606e-05,
+            0.03378688171505928,
+            -0.0293476190418005,
+            -0.012264790013432503,
+            -0.016060132533311844,
+            -0.004439263138920069,
+            -0.01706010289490223,
+            -0.02762039750814438,
+            -0.006874797400087118,
+            -0.01889338344335556,
+            -0.00814369972795248,
+            -0.028196139261126518,
+            -0.009992130100727081,
+            0.01949942484498024,
+            0.022332675755023956,
+            0.001196934375911951,
+            -0.006783891003578901,
+            -0.005901341326534748,
+            0.02092362567782402,
+            0.032514192163944244,
+            0.01952972821891308,
+            0.0058331615291535854,
+            -0.015158643946051598,
+            -0.004295328166335821,
+            -0.012370847165584564,
+            0.016060132533311844,
+            0.012537509202957153,
+            6.344509893096983e-05,
+            0.03493836522102356,
+            0.004170331638306379,
+            -0.022726602852344513,
+            -0.001230077352374792,
+            0.024923507124185562,
+            -0.021590271964669228,
+            -0.0108330138027668,
+            0.040029123425483704,
+            0.0085679292678833,
+            -0.0024506852496415377,
+            -0.019287310540676117,
+            -0.019075196236371994,
+            0.02630225569009781,
+            -0.02133270539343357,
+            0.002083271974697709,
+            0.0482713058590889,
+            0.00035344602656550705,
+            -0.029802151024341583,
+            0.017954016104340553,
+            -0.013242034241557121,
+            -0.018560059368610382,
+            0.01848430372774601,
+            0.0012859469279646873,
+            -0.039968520402908325,
+            0.019014591351151466,
+            0.018408548086881638,
+            0.022635696455836296,
+            -0.02918095886707306,
+            0.0002127068000845611,
+            0.03233237937092781,
+            0.004037759732455015,
+            0.001595596899278462,
+            -0.008530051447451115,
+            -0.0006065162015147507,
+            0.010431510396301746,
+            0.001746160676702857,
+            0.02854461409151554,
+            -0.0030434709042310715,
+            0.027771908789873123,
+            -0.006727074272930622,
+            -0.008863375522196293,
+            0.019181253388524055,
+            -0.02109028771519661,
+            0.013166278600692749,
+            0.0685434341430664,
+            0.0021950111258774996,
+            0.025847723707556725,
+            0.007181606721132994,
+            0.024347767233848572,
+            0.02018122375011444,
+            -0.027271922677755356,
+            -0.02021152526140213,
+            -0.005397567991167307,
+            -0.0014033677289262414,
+            -0.03787767142057419,
+            0.011128460057079792,
+            -0.021302402019500732,
+            -0.013249609619379044,
+            -0.0022347825579345226,
+            -0.002316219499334693,
+            -0.006405114196240902,
+            0.034211114048957825,
+            -0.0014630251098424196,
+            -0.018560059368610382,
+            -0.035423196852207184,
+            0.037726160138845444,
+            0.04048365354537964,
+            0.0389382466673851,
+            -0.005984671879559755,
+            0.03548380360007286,
+            0.0030756669584661722,
+            0.0025302283465862274,
+            0.041453324258327484,
+            0.004791525658220053,
+            -0.011567841283977032,
+            0.02295386977493763,
+            0.012416300363838673,
+            0.03351416438817978,
+            -0.02045394293963909,
+            -0.02630225569009781,
+            -0.003299145260825753,
+            -0.026423463597893715,
+            0.012492056004703045,
+            -0.019726691767573357,
+            0.0018768386216834188,
+            0.007117214612662792,
+            0.015196521766483784,
+            -0.016044981777668,
+            0.014719262719154358,
+            -0.006102093029767275,
+            -0.021014532074332237,
+            0.011719351634383202,
+            -0.007780073676258326,
+            0.011961768381297588,
+            -0.02262054570019245,
+            0.001138224033638835,
+            -0.04063516482710838,
+            0.008280059322714806,
+            0.014022313989698887,
+            -0.01598437689244747,
+            0.03205966204404831,
+            0.020969079807400703,
+            -0.01801462098956108,
+            -0.0009682480012997985,
+            0.0034714885987341404,
+            -0.010393633507192135,
+            -0.011067855171859264,
+            -0.019923655316233635,
+            0.02786281518638134,
+            -0.015802564099431038,
+            -0.025908326730132103,
+            0.02454473078250885,
+            -0.01743887923657894,
+            0.015151068568229675,
+            -0.004299115855246782,
+            0.00888610165566206,
+            -0.019120648503303528,
+            0.02284781076014042,
+            -0.023741723969578743,
+            -0.0009128518868237734,
+            -0.020287280902266502,
+            0.0028389315120875835,
+            0.014135947450995445,
+            -0.018272189423441887,
+            -0.02139330841600895,
+            0.019150950014591217,
+            -0.005420294590294361,
+            -0.003931702114641666,
+            0.010014856234192848,
+            0.0030983935575932264,
+            -0.0029715034179389477,
+            -0.004549108445644379,
+            -0.03281721472740173,
+            -0.02915065549314022,
+            0.0033086147159337997,
+            0.011386028490960598,
+            0.015257125720381737,
+            0.027574945241212845,
+            0.03715042024850845,
+            0.006052851676940918,
+            0.0036949669010937214,
+            0.0009327376610599458,
+            -0.020726662129163742,
+            0.042635105550289154,
+            -0.020469093695282936,
+            0.040847279131412506,
+            0.032544493675231934,
+            -0.0197418425232172,
+            -0.023908386006951332,
+            0.0029052174650132656,
+            0.03115059621632099,
+            0.03151422366499901,
+            0.025211378931999207,
+            0.012310243211686611,
+            0.03690800443291664,
+            -0.004359719809144735,
+            -0.032211173325777054,
+            0.015802564099431038,
+            0.022256920114159584,
+            -0.008105821907520294,
+            0.009673957712948322,
+            0.027438584715127945,
+            0.013279912061989307,
+            -0.007757347077131271,
+            0.030150625854730606,
+            0.0012234487803652883,
+            0.02396899089217186,
+            0.02498411200940609,
+            -0.02190844528377056,
+            0.01743887923657894,
+            -0.0001154677156591788,
+            0.006727074272930622,
+            -0.009840618818998337,
+            0.004348356742411852,
+            0.011810258030891418,
+            0.018272189423441887,
+            0.018908534198999405,
+            0.003246116451919079,
+            -0.07060398161411285,
+            0.019726691767573357,
+            0.004507442936301231,
+            -0.02160542458295822,
+            0.00390140013769269,
+            -0.0024866692256182432,
+            0.012704171240329742,
+            0.02105998620390892,
+            0.012461753562092781,
+            -0.01790856383740902,
+            0.0013626492582261562,
+            0.011961768381297588,
+            -0.00595815759152174,
+            0.04730163514614105,
+            0.0019791084341704845,
+            -0.01437078882008791,
+            -0.016105586662888527,
+            0.013726867735385895,
+            0.02525683119893074,
+            -0.02355991117656231,
+            0.006325571332126856,
+            0.005621046293526888,
+            -0.004147605039179325,
+            -0.010742107406258583,
+            -0.030377892777323723,
+            -0.005280147306621075,
+            -0.03436262533068657,
+            0.023150833323597908,
+            0.015491968020796776,
+            0.014764715917408466,
+            -0.025044716894626617,
+            -0.004280176945030689,
+            0.0046400148421525955,
+            -0.021287251263856888,
+            0.003261267440393567,
+            0.01161329448223114,
+            0.0005104016163386405,
+            -0.020984230563044548,
+            0.00959820207208395,
+            0.013469300232827663,
+            0.06848283112049103,
+            0.0007442962378263474,
+            0.029771849513053894,
+            0.0422411784529686,
+            -0.0012243957025930285,
+            -0.006371024530380964,
+            0.0036286809481680393,
+            -0.019953956827521324,
+            0.003473382443189621,
+            0.025878025218844414,
+            -0.03224147483706474,
+            -0.055058982223272324,
+            -0.007545232307165861,
+            -0.04624105989933014,
+            0.001489539397880435,
+            0.014711687341332436,
+            0.023044776171445847,
+            -0.0037782976869493723,
+            -0.034817155450582504,
+            0.016908591613173485,
+            -0.003071879269555211,
+            0.007014944683760405,
+            -0.02593863010406494,
+            -0.0294839795678854,
+            -0.026014383882284164,
+            -0.02339324913918972,
+            -0.012908710166811943,
+            -0.022726602852344513,
+            0.012590537779033184,
+            -0.0015103721525520086,
+            -0.06587684899568558,
+            -0.009780014865100384,
+            0.016166189685463905,
+            0.001200722181238234,
+            0.011939042247831821,
+            0.011083006858825684,
+            -0.007083124481141567,
+            -0.0018275976181030273,
+            -0.022559940814971924,
+            0.03124150261282921,
+            0.005177877843379974,
+            0.03584742918610573,
+            -0.006503595970571041,
+            -0.009128518402576447,
+            0.01073453202843666,
+            0.04527139291167259,
+            0.0017916138749569654,
+            0.010226971469819546,
+            -0.020560000091791153,
+            0.0015065843472257257,
+            -0.02284781076014042,
+            -0.0031135445460677147,
+            -0.008863375522196293,
+            -0.0021211495622992516,
+            0.015893470495939255,
+            0.0019753205124288797,
+            -0.010204244405031204,
+            0.025438643991947174,
+            0.02251448854804039,
+            -0.009605777449905872,
+            -0.015969226136803627,
+            -0.02437806874513626,
+            -0.014545026235282421,
+            -0.0002627763315103948,
+            -0.010007280856370926,
+            -0.022453883662819862,
+            -0.020878173410892487,
+            -0.002566212322562933,
+            -0.02559015527367592,
+            0.022529639303684235,
+            0.01220418605953455,
+            -4.882668508798815e-05,
+            -0.006723286584019661,
+            -0.038544319570064545,
+            -0.003727162955328822,
+            -0.03118089959025383,
+            -0.005374841392040253,
+            0.03287781774997711,
+            -0.03745344281196594,
+            -0.0002231231628684327,
+            0.014491996727883816,
+            -0.01993880607187748,
+            0.0028938541654497385,
+            0.008961857296526432,
+            0.02769615314900875,
+            -0.027408283203840256,
+            0.0001795638381736353,
+            0.006954340264201164,
+            0.016908591613173485,
+            -0.009878496639430523,
+            0.027196168899536133,
+            -0.04275631532073021,
+            0.011037553660571575,
+            -0.037120118737220764,
+            -0.020423639565706253,
+            -0.01523439958691597,
+            0.008363389410078526,
+            -0.005018791649490595,
+            -0.02637801133096218,
+            -0.010454237461090088,
+            0.006026337388902903,
+            -0.00693918950855732,
+            -0.01902974210679531,
+            -0.010620899498462677,
+            0.027438584715127945,
+            0.00931033119559288,
+            0.007977037690579891,
+            -0.02955973520874977,
+            -0.01631770096719265,
+            0.006280118133872747,
+            0.052089374512434006,
+            -0.01584801822900772,
+            0.014938953332602978,
+            -0.006639956030994654,
+            0.02342355251312256,
+            -0.028484009206295013,
+            -0.01872672140598297,
+            0.0315445251762867,
+            0.013946558348834515,
+            0.007064185570925474,
+            0.028953691944479942,
+            0.027196168899536133,
+            0.027105262503027916,
+            -0.04654408246278763,
+            0.009886072017252445,
+            -0.002613559365272522,
+            -0.008757317438721657,
+            0.03402930125594139,
+            0.01916610263288021,
+            0.022150862962007523,
+            0.0009744031121954322,
+            -0.01730252057313919,
+            0.0139844361692667,
+            0.014885924756526947,
+            0.007681591901928186,
+            -0.014242004603147507,
+            -0.02660527639091015,
+            0.003367325058206916,
+            -0.007992188446223736,
+            0.00047276070108637214,
+            0.020726662129163742,
+            -0.009052763693034649,
+            0.030999086797237396,
+            -0.014605630189180374,
+            -0.02593863010406494,
+            0.008916404098272324,
+            -0.02268115058541298,
+            -0.0041741193272173405,
+            0.0004476667381823063,
+            0.010083036497235298,
+            -0.023847781121730804,
+            -0.04669559374451637,
+            0.019408518448472023,
+            0.028393102809786797,
+            0.005579380784183741,
+            0.01328748743981123,
+            -0.035332292318344116,
+            0.008992158807814121,
+            0.012310243211686611,
+            0.011348150670528412,
+            0.0019847899675369263,
+            0.035362593829631805,
+            -0.00028834378463216126,
+            0.012885984033346176,
+            -0.008620957843959332,
+            0.028362801298499107,
+            0.00667025800794363,
+            -0.0323626808822155,
+            -0.0378473699092865,
+            -0.01513591781258583,
+            0.015257125720381737,
+            0.006692984607070684,
+            -0.011840560473501682,
+            -0.02278720773756504,
+            -0.038817036896944046,
+            0.015787413343787193,
+            -0.012916285544633865,
+            -0.038271598517894745,
+            -0.010158791206777096,
+            -0.009742137044668198,
+            0.016635872423648834,
+            -0.006916462909430265,
+            -0.016802534461021423,
+            -0.016090434044599533,
+            0.005821797996759415,
+            7.161247049225494e-05,
+            0.0244083721190691,
+            -0.001251857029274106,
+            -0.006855858489871025,
+            -0.016393456608057022,
+            -0.01188601367175579,
+            -0.0035074723418802023,
+            0.03602924197912216,
+            -0.014219277538359165,
+            -0.02383263036608696,
+            0.014984406530857086,
+            0.02657497487962246,
+            0.011636020615696907,
+            0.03227177634835243,
+            0.007181606721132994,
+            -0.014022313989698887,
+            0.003016956616193056,
+            -0.033029329031705856,
+            -0.0046400148421525955,
+            0.005196816753596067,
+            -0.006427840795367956,
+            -0.015651054680347443,
+            -0.01815097965300083,
+            -0.015832867473363876,
+            0.030756669119000435,
+            -0.02487805485725403,
+            0.0218023881316185,
+            0.030347591266036034,
+            0.003174148965626955,
+            -0.02915065549314022,
+            0.013598084449768066,
+            -0.0072005451656877995,
+            0.01862066239118576,
+            0.0019014590652659535,
+            0.028014326468110085,
+            -0.007461901288479567,
+            -0.004818039946258068,
+            -0.02501441352069378,
+            -0.03696860745549202,
+            -0.023484155535697937,
+            -0.008817922323942184,
+            0.0027537066489458084,
+            0.020984230563044548,
+            -0.0375746488571167,
+            -0.017757052555680275,
+            0.009855770505964756,
+            0.015757111832499504,
+            -0.03027183562517166,
+            -0.0025908327661454678,
+            -0.010158791206777096,
+            -0.0004055278259329498,
+            0.005140000022947788,
+            -0.02227207086980343,
+            -0.008977008052170277,
+            -0.015726808458566666,
+            -0.0010435298318043351,
+            0.006166485138237476,
+            -0.0007409819518215954,
+            -0.05805889517068863,
+            -0.013969285413622856,
+            0.018635815009474754,
+            0.0026400736533105373,
+            0.008908828720450401,
+            0.015938924625515938,
+            -0.0111739132553339,
+            0.0031533162109553814,
+            -0.026756787672638893,
+            0.023893235251307487,
+            -0.009492144919931889,
+            0.009355784393846989,
+            -0.013742019422352314,
+            0.012514783069491386,
+            0.014014738611876965,
+            0.004795313347131014,
+            0.0010681503918021917,
+            -0.0054051438346505165,
+            0.01601468026638031,
+            0.028377952054142952,
+            -0.023756874725222588,
+            0.011454207822680473,
+            -0.0002514130319468677,
+            -0.0032158142421394587,
+            -0.029196109622716904,
+            0.00509454682469368,
+            0.003931702114641666,
+            -0.005109698045998812,
+            0.016651025041937828,
+            -0.036544378846883774,
+            0.01662072166800499,
+            0.02177208475768566,
+            0.032544493675231934,
+            0.014832896180450916,
+            -0.017287369817495346,
+            -0.011658747680485249,
+            -0.012385998852550983,
+            -0.002624922664836049,
+            0.022181164473295212,
+            0.012120855040848255,
+            -0.01815097965300083,
+            -0.024287162348628044,
+            0.011249667964875698,
+            0.0005283934879116714,
+            -0.01203752402216196,
+            0.010795135982334614,
+            -0.013166278600692749,
+            0.01349202636629343,
+            0.012711746618151665,
+            0.00866641104221344,
+            0.00888610165566206,
+            0.022862963378429413,
+            -0.011878437362611294,
+            -0.017711598426103592,
+            0.00656041270121932,
+            -0.008916404098272324,
+            -0.0034089903347194195,
+            -0.012598113156855106,
+            0.010795135982334614,
+            0.014878349378705025,
+            0.06490717828273773,
+            -0.013143551535904408,
+            0.0009526234352961183,
+            -0.020438792183995247,
+            -0.031938452273607254,
+            -0.0017944546416401863,
+            -0.020908474922180176,
+            0.0111739132553339,
+            0.008370965719223022,
+            0.01598437689244747,
+            0.012022373266518116,
+            -0.0021912232041358948,
+            0.00795431062579155,
+            0.019317612051963806,
+            -0.010423935018479824,
+            7.255941454786807e-05,
+            -0.006503595970571041,
+            0.0013683309080079198,
+            0.0035851215943694115,
+            -0.011719351634383202,
+            -0.0026949963066726923,
+            0.035695917904376984,
+            0.01068150345236063,
+            0.011651171371340752,
+            -0.03445352986454964,
+            0.0020435003098100424,
+            0.020150920376181602,
+            0.007602048572152853,
+            -0.008302785456180573,
+            -0.01563590206205845,
+            0.01706010289490223,
+            -0.015317730605602264,
+            -0.0022499337792396545,
+            0.002888172399252653,
+            -0.010151215828955173,
+            0.04581683129072189,
+            -0.019787294790148735,
+            -0.0011742077767848969,
+            0.031847547739744186,
+            0.0030207443051040173,
+            -0.0040983641520142555,
+            0.005408931523561478,
+            -0.003473382443189621,
+            -0.0069013116881251335,
+            -0.02139330841600895,
+            0.0031381649896502495,
+            0.00667025800794363,
+            -0.019287310540676117,
+            -0.0072005451656877995,
+            -0.007719469256699085,
+            0.006897523999214172,
+            0.023105379194021225,
+            -0.009492144919931889,
+            -0.00016145357221830636,
+            -0.008575504645705223,
+            0.009848194196820259,
+            -0.0011912527261301875,
+            -0.044301725924015045,
+            -0.006147546228021383,
+            0.021181194111704826,
+            0.000973456131760031,
+            0.0005757405888289213,
+            -0.007450537756085396,
+            0.005007428117096424,
+            -0.017423728480935097,
+            -0.008454295806586742,
+            0.02369627170264721,
+            0.01736312545835972,
+            -0.009113367646932602,
+            0.010863316245377064,
+            -0.0035680767614394426,
+            0.00433320552110672,
+            -0.020014561712741852,
+            -0.01095422264188528,
+            0.003462019143626094,
+            0.015272277407348156,
+            -0.0009819786064326763,
+            -0.029029447585344315,
+            -0.0012272365856915712,
+            -0.00024194362049456686,
+            0.02918095886707306,
+            0.01486319862306118,
+            -0.008961857296526432,
+            -0.021211495622992516,
+            -0.013242034241557121,
+            -0.001237652963027358,
+            -0.015726808458566666,
+            -0.009620928205549717,
+            0.011855711229145527,
+            0.0009137988090515137,
+            0.008992158807814121,
+            -0.018135828897356987,
+            -0.027347678318619728,
+            -0.016166189685463905,
+            0.0018465365283191204,
+            -0.012870832346379757,
+            0.003340810537338257,
+            0.015514694154262543,
+            -0.03430201858282089,
+            -0.015060162171721458,
+            -0.03805948421359062,
+            0.008280059322714806,
+            -0.0029658216517418623,
+            -0.02024182677268982,
+            -0.023605365306138992,
+            0.009484568610787392,
+            0.019181253388524055,
+            -0.0015548784285783768,
+            0.019847899675369263,
+            -0.000889651826582849,
+            0.020060013979673386,
+            -0.0073141781613230705,
+            0.0031703610438853502,
+            -0.013007191941142082,
+            0.0030245319940149784,
+            0.008333087898790836,
+            0.02890823967754841,
+            -0.023075077682733536,
+            -0.006503595970571041,
+            -0.004685468040406704,
+            0.004746072459965944,
+            0.012658718042075634,
+            0.014613205567002296,
+            0.020741812884807587,
+            -0.008272483013570309,
+            -0.0027612822595983744,
+            0.012484480626881123,
+            -0.003799130441620946,
+            0.007295239716768265,
+            0.002390081062912941,
+            0.0019677451346069574,
+            0.004162756260484457,
+            0.008477022871375084,
+            0.0053710537031292915,
+            0.027650700882077217,
+            -0.0028275682125240564,
+            0.0031703610438853502,
+            0.024620486423373222,
+            0.02322658896446228,
+            -0.018878230825066566,
+            -0.028074929490685463,
+            0.002024561632424593,
+            0.050695475190877914,
+            0.007245998363941908,
+            -0.006810405291616917,
+            -0.03927157074213028,
+            -0.0031760428100824356,
+            0.028635520488023758,
+            0.023650817573070526,
+            0.01584801822900772,
+            -0.010567869991064072,
+            -0.012098127976059914,
+            0.02772645466029644,
+            0.013295062817633152,
+            0.02616589516401291,
+            -0.025302285328507423,
+            0.010704229585826397,
+            -0.03869583085179329,
+            0.0449683703482151,
+            0.004439263138920069,
+            -0.014476845972239971,
+            0.0011884119594469666,
+            0.010386057198047638,
+            -0.004537744913250208,
+            0.006431628484278917,
+            -5.8059369621332735e-05,
+            -0.0008086882880888879,
+            0.011923890560865402,
+            0.012007221579551697,
+            0.004265025723725557,
+            -0.02399929240345955,
+            0.007359631359577179,
+            0.004272601567208767,
+            0.03005971945822239,
+            0.005431658122688532,
+            -0.0013863227795809507,
+            0.012385998852550983,
+            -0.005810434930026531,
+            0.008855799213051796,
+            -0.010719381272792816,
+            0.020469093695282936,
+            -0.02918095886707306,
+            1.0986003871948924e-05,
+            0.02051454596221447,
+            -0.0015785519499331713,
+            0.0038199631962925196,
+            -0.023741723969578743,
+            0.01186328660696745,
+            -0.052968136966228485,
+            0.021166043356060982,
+            -0.018741872161626816,
+            0.021741783246397972,
+            0.019590331241488457,
+            -0.009734561666846275,
+            -0.012052674777805805,
+            0.03545350208878517,
+            0.0026760573964565992,
+            -0.014264730736613274,
+            -0.010643625631928444,
+            -0.017999470233917236,
+            -0.02830219641327858,
+            0.041968461126089096,
+            -0.0006117243901826441,
+            0.0028275682125240564,
+            0.0024222771171480417,
+            -0.010666352696716785,
+            -0.008923979476094246,
+            0.001954487757757306,
+            0.020499395206570625,
+            -0.03369597718119621,
+            -0.011128460057079792,
+            -0.008689138107001781,
+            -0.02566591091454029,
+            -0.00020300064352340996,
+            -0.011961768381297588,
+            0.026211349293589592,
+            0.01240872498601675,
+            -0.03696860745549202,
+            0.003780191531404853,
+            0.023650817573070526,
+            0.001603172393515706,
+            0.018120678141713142,
+            0.002130619017407298,
+            0.0010795136913657188,
+            0.009742137044668198,
+            -0.010476963594555855,
+            0.026559824123978615,
+            -0.009204274043440819,
+            0.008658835664391518,
+            -0.001950700068846345,
+            -0.017287369817495346,
+            0.017969166859984398,
+            0.004083212930709124,
+            0.024120500311255455,
+            0.011954193003475666,
+            0.011605718173086643,
+            0.01112088467925787,
+            -0.019878201186656952,
+            0.020090317353606224,
+            -0.0018891488434746861,
+            0.009075489826500416,
+            0.024726543575525284,
+            -0.005863463506102562,
+            0.020832719281315804,
+            0.015257125720381737,
+            -0.00020229043730068952,
+            0.011295121163129807,
+            -0.020120618864893913,
+            -0.011779955588281155,
+            0.0085679292678833,
+            -0.009605777449905872,
+            -0.010325453244149685,
+            0.009901223704218864,
+            -0.010105762630701065,
+            -0.015052586793899536,
+            -0.0071247899904847145,
+            0.016635872423648834,
+            0.0028976418543606997,
+            -0.02610529027879238,
+            -1.7326075976598077e-05,
+            0.008901252411305904,
+            0.006166485138237476,
+            0.018741872161626816,
+            0.011317848227918148,
+            -0.006692984607070684,
+            0.010363331064581871,
+            0.011476934887468815,
+            0.0053104497492313385,
+            -0.005079395603388548,
+            -0.013431422412395477,
+            -0.0024241709616035223,
+            0.018090376630425453,
+            0.009810317307710648,
+            -0.01922670565545559,
+            0.004564259201288223,
+            0.0008877579239197075,
+            -0.01420412678271532,
+            -0.019347915425896645,
+            -0.0038483713287860155,
+            -0.004170331638306379,
+            0.0027915844693779945,
+            0.0509984977543354,
+            -0.0006216673064045608,
+            0.007893706671893597,
+            -0.025741664692759514,
+            -0.006299056578427553,
+            -0.00027556007262319326,
+            -0.0036097420379519463,
+            0.01272689737379551,
+            -0.01824188604950905,
+            -0.03724132478237152,
+            0.012711746618151665,
+            -0.0012584856012836099,
+            0.014923802576959133,
+            -0.00297907879576087,
+            0.014226853847503662,
+            -0.017347972840070724,
+            0.01916610263288021,
+            -0.0011117096291854978,
+            -0.016029831022024155,
+            0.0029374135192483664,
+            -0.011143610812723637,
+            -0.021317552775144577,
+            0.005916492082178593,
+            -0.006848283112049103,
+            -0.03896854817867279,
+            0.0029885482508689165,
+            -0.010423935018479824,
+            0.0011798894265666604,
+            -0.003912763670086861,
+            0.020166072994470596,
+            -0.0061513339169323444,
+            0.0008697660523466766,
+            0.02177208475768566,
+            0.0017026013229042292,
+            0.01601468026638031,
+            -0.014499573037028313,
+            0.010999675840139389,
+            0.012598113156855106,
+            0.02210540883243084,
+            -0.032180868089199066,
+            -0.01393140759319067,
+            0.014211702160537243,
+            0.0026116655208170414,
+            -0.005382417235523462,
+            0.020635755732655525,
+            -0.01294658798724413,
+            0.019650936126708984,
+            -0.0005577487172558904,
+            -0.011704200878739357,
+            -0.03524138405919075,
+            -0.016529815271496773,
+            -3.343888238305226e-05,
+            -0.008439145050942898,
+            0.025317436084151268,
+            0.01760554127395153,
+            -0.0108330138027668,
+            0.026468917727470398,
+            0.0037574649322777987,
+            0.014598054811358452,
+            -0.014257155358791351,
+            -0.005999823100864887,
+            -0.0003773563075810671,
+            -0.0119314668700099,
+            0.015264702029526234,
+            -0.03512017801403999,
+            0.0008342557121068239,
+            -0.00839369185268879,
+            0.005242269951850176,
+            -0.0254992488771677,
+            0.018257036805152893,
+            0.025817420333623886,
+            -0.0031419529113918543,
+            -0.005037730094045401,
+            -0.022802358493208885,
+            0.028968842700123787,
+            0.03049910068511963,
+            0.0024828813038766384,
+            -0.0013825349742546678,
+            -0.00352072948589921,
+            0.0033218718599528074,
+            0.0211205892264843,
+            -0.037998881191015244,
+            -0.009098216891288757,
+            -0.028696123510599136,
+            0.042332086712121964,
+            -0.008098246529698372,
+            -0.013431422412395477,
+            0.01862066239118576,
+            -0.0024715180043131113,
+            0.02355991117656231,
+            -0.004689255729317665,
+            -0.004855917301028967,
+            0.03148392215371132,
+            0.011514811776578426,
+            0.02772645466029644,
+            -0.04015033319592476,
+            0.014704111963510513,
+            -0.006867222022265196,
+            0.004867280833423138,
+            -0.013484450988471508,
+            0.005302873905748129,
+            0.02580226957798004,
+            -0.018060073256492615,
+            0.0068066176027059555,
+            0.030665762722492218,
+            0.00748084019869566,
+            0.002448791405186057,
+            0.0165904201567173,
+            0.003890036838129163,
+            0.010855740867555141,
+            0.000841831264551729,
+            0.0052574207074940205,
+            0.022423582151532173,
+            0.014454119838774204,
+            -0.016029831022024155,
+            -0.015120766125619411,
+            -0.00044790346873924136,
+            -0.004450626205652952,
+            0.005261208396404982,
+            0.013242034241557121,
+            0.0061172437854111195,
+            -0.015681356191635132,
+            0.01052999310195446,
+            0.014158673584461212,
+            0.013128400780260563,
+            -0.0039847311563789845,
+            -0.03412020578980446,
+            -0.02443867363035679,
+            -0.021438762545585632,
+            0.003104075090959668,
+            -0.0008281006012111902,
+            -0.0021211495622992516,
+            0.0017954016802832484,
+            0.017347972840070724,
+            -0.0008352026343345642,
+            0.00379155483096838,
+            0.01686313934624195,
+            -0.01415109820663929,
+            -0.022090258076786995,
+            -0.0035964848939329386,
+            -0.027529491111636162,
+            0.0053710537031292915,
+            0.001844642567448318,
+            -0.00965123064815998,
+            0.008530051447451115,
+            -0.0046059247106313705,
+            -0.007223271764814854,
+            -0.012196609750390053,
+            -0.02007516659796238,
+            0.005780132487416267,
+            0.016878290101885796,
+            0.005730891600251198,
+            -0.009272454306483269,
+            -0.003939277958124876,
+            0.011575416661798954,
+            0.017484333366155624,
+            0.017317671328783035,
+            -0.009295180439949036,
+            0.009628504514694214,
+            0.0004022135108243674,
+            0.004670316819101572,
+            -0.01834794320166111,
+            0.0045869858004152775,
+            -0.0027953721582889557,
+            -0.008658835664391518,
+            0.01105270441621542,
+            -0.00031201730598695576,
+            0.01628739945590496,
+            -0.013090522959828377,
+            0.005624833982437849,
+            0.010333028621971607,
+            -0.020590301603078842,
+            -0.0067308624275028706,
+            -0.01692374423146248,
+            0.026484068483114243,
+            -0.00504530593752861,
+            0.00045405857963487506,
+            -0.02978700026869774,
+            0.0033616432920098305,
+            0.000848933297675103,
+            -0.000845619011670351,
+            -0.012764775194227695,
+            0.010749682784080505,
+            0.03242328763008118,
+            0.0005179771687835455,
+            0.0216660276055336,
+            -0.0036154238041490316,
+            0.0007480839849449694,
+            -0.00785582885146141,
+            0.004977126140147448,
+            -0.004916521720588207,
+            -0.009901223704218864,
+            -0.010863316245377064,
+            0.021726632490754128,
+            -0.0033029329497367144,
+            0.0003356908564455807,
+            0.011332998983561993,
+            -0.012067826464772224,
+            0.0014620780711993575,
+            -0.0047687990590929985,
+            0.00971941091120243,
+            0.0061702728271484375,
+            0.009113367646932602,
+            -0.01176480483263731,
+            0.05451354384422302,
+            -0.0106284748762846,
+            0.005537715740501881,
+            -0.00439380994066596,
+            -0.022226616740226746,
+            -0.017317671328783035,
+            0.012711746618151665,
+            -0.017544938251376152,
+            0.03375658020377159,
+            0.024150803685188293,
+            0.0006467612111009657,
+            0.010393633507192135,
+            -0.006958128418773413,
+            0.023923536762595177,
+            -0.002602196065708995,
+            0.0010795136913657188,
+            -0.004855917301028967,
+            0.015802564099431038,
+            0.04511988162994385,
+            0.017393426969647408,
+            -0.001152428099885583,
+            0.00829521007835865,
+            0.020575150847434998,
+            0.00814369972795248,
+            -0.0020075165666639805,
+            0.014219277538359165,
+            -0.01024212222546339,
+            0.00900731049478054,
+            -0.004458202049136162,
+            0.027438584715127945,
+            0.00814369972795248,
+            -0.006310420110821724,
+            -0.0036836036015301943,
+            -0.0008209985098801553,
+            0.014173824340105057,
+            0.006950552575290203,
+            0.01963578537106514,
+            -0.00461350055411458,
+            -0.0012575386790558696,
+            0.013870803639292717,
+            0.014491996727883816,
+            -0.00552635220810771,
+            0.001724380999803543,
+            -0.01435563713312149,
+            -0.010204244405031204,
+            0.013575357384979725,
+            0.006086941808462143,
+            0.006310420110821724,
+            -0.014514723792672157,
+            -0.00980274099856615,
+            -0.004670316819101572,
+            0.0015435151290148497,
+            0.015052586793899536,
+            0.011060279794037342,
+            -0.004814252257347107,
+            0.006052851676940918,
+            0.013825350441038609,
+            -0.014014738611876965,
+            0.006147546228021383,
+            0.007208121009171009,
+            0.008855799213051796,
+            0.004787737503647804,
+            -0.010817863047122955,
+            -0.00428396463394165,
+            -0.0063937511295080185,
+            0.0012670081341639161,
+            0.016832837834954262,
+            0.013181429356336594,
+            0.0049581872299313545,
+            0.027529491111636162,
+            -0.00839369185268879,
+            -0.012302667833864689,
+            0.02339324913918972,
+            0.011257244274020195,
+            0.0031703610438853502,
+            0.014651083387434483,
+            -0.012635990977287292,
+            -0.012446602806448936,
+            0.006655106786638498,
+            0.004848341923207045,
+            0.006049063988029957,
+            0.009151245467364788,
+            0.00450365524739027,
+            0.007333117071539164,
+            -0.007098275702446699,
+            -0.007427811156958342,
+            0.0014810169814154506,
+            0.01642375811934471,
+            0.020393338054418564,
+            -0.024968961253762245,
+            -0.0018806264270097017,
+            0.0009052763343788683,
+            0.010878467001020908,
+            -0.003316190093755722,
+            0.012310243211686611,
+            5.65205882594455e-05,
+            -0.0074240234680473804,
+            0.010158791206777096,
+            -0.02119634486734867,
+            -0.012681444175541401,
+            0.0012717428617179394,
+            0.013037494383752346,
+            -0.019590331241488457,
+            0.018226735293865204,
+            -0.008893677033483982,
+            -0.013348091393709183,
+            0.013893529772758484,
+            0.020014561712741852,
+            0.004090788308531046,
+            -0.007738408166915178,
+            -0.021862991154193878,
+            -0.01127239502966404,
+            0.016060132533311844,
+            0.020272130146622658,
+            0.000947888707742095,
+            -0.0006472347304224968,
+            0.010173942893743515,
+            -0.013113250024616718,
+            -0.023605365306138992,
+            -0.005230906419456005,
+            0.010386057198047638,
+            0.01746918261051178,
+            0.018832778558135033,
+            -0.006685409229248762,
+            0.02443867363035679,
+            -0.0059127043932676315,
+            -0.015499543398618698,
+            2.86597842205083e-05,
+            -0.02802947722375393,
+            -0.017423728480935097,
+            0.00013778003631159663,
+            0.00246773031540215,
+            0.008598231710493565,
+            0.0008555619278922677,
+            0.01943882182240486,
+            0.000904329412151128,
+            0.017590390518307686,
+            -0.000659071491099894,
+            -0.002009410411119461,
+            -0.0013995799235999584,
+            -0.0015340456739068031,
+            0.02254479005932808,
+            -0.012166308239102364,
+            -0.01176480483263731,
+            0.022302372381091118,
+            0.006852070800960064,
+            0.014719262719154358,
+            -0.027635548263788223,
+            0.013757170177996159,
+            -0.015499543398618698,
+            0.017423728480935097,
+            -0.03408990427851677,
+            0.006420265417546034,
+            0.01489350013434887,
+            -0.010348179377615452,
+            -0.020499395206570625,
+            -0.029650641605257988,
+            -0.024150803685188293,
+            -0.0038710979279130697,
+            -0.010855740867555141,
+            -0.016847988590598106,
+            0.006121031939983368,
+            -0.005802859086543322,
+            0.018499454483389854,
+            -0.007079336792230606,
+            0.033877789974212646,
+            0.006969491485506296,
+            -0.004056698642671108,
+            0.003835114184767008,
+            0.022469034418463707,
+            -0.006102093029767275,
+            0.02109028771519661,
+            0.004408961161971092,
+            -0.001467759720981121,
+            0.013681414537131786,
+            -0.006552837323397398,
+            -0.002361672930419445,
+            0.005821797996759415,
+            -0.010484539903700352,
+            0.022014502435922623,
+            0.005924067925661802,
+            -0.007708106189966202,
+            -0.0006775368237867951,
+            -0.011454207822680473,
+            0.014726839028298855,
+            0.006530110724270344,
+            0.0022480397019535303,
+            -0.011295121163129807,
+            0.011249667964875698,
+            -0.004859705455601215,
+            0.0040642740204930305,
+            0.009484568610787392,
+            -0.01774190180003643,
+            -0.028862785547971725,
+            0.011416330002248287,
+            0.019120648503303528,
+            0.006382387597113848,
+            -0.0055755930952727795,
+            -0.0025681061670184135,
+            0.00839369185268879,
+            0.010651201009750366,
+            0.022150862962007523,
+            -0.012984465807676315,
+            -8.889415767043829e-05,
+            0.013204156421124935,
+            -0.007170243188738823,
+            -0.01056029461324215,
+            0.00487485621124506,
+            0.023438703268766403,
+            0.0068823727779090405,
+            -0.04327145218849182,
+            -0.018469152972102165,
+            0.01909034699201584,
+            0.007136153522878885,
+            -0.04154422879219055,
+            0.0026381798088550568,
+            -0.003689285134896636,
+            -0.023787178099155426,
+            0.016105586662888527,
+            -0.0011817833874374628,
+            -0.01828734017908573,
+            0.028756728395819664,
+            0.02830219641327858,
+            0.01834794320166111,
+            2.6987840101355687e-05,
+            -0.013332940638065338,
+            -0.0318172425031662,
+            -0.023590214550495148,
+            0.008280059322714806,
+            -0.02660527639091015,
+            -0.0072725131176412106,
+            0.011242092587053776,
+            -0.017454031854867935,
+            0.02972639724612236,
+            -0.008507325313985348,
+            0.026938600465655327,
+            0.011446632444858551,
+            0.001691238023340702,
+            -0.013446573168039322,
+            -0.018802475184202194,
+            -0.01350717805325985,
+            -0.008196728304028511,
+            0.014090493321418762,
+            0.017923714593052864,
+            0.011007251217961311,
+            0.011332998983561993,
+            0.02251448854804039,
+            -0.012158732861280441,
+            0.007969462312757969,
+            -2.888172457460314e-05,
+            -0.008961857296526432,
+            0.0026343921199440956,
+            -0.018393397331237793,
+            0.00868156272917986,
+            0.0062308767810463905,
+            0.005458172410726547,
+            -0.007208121009171009,
+            0.010742107406258583,
+            0.02712041325867176,
+            -0.018847929313778877,
+            0.011401179246604443,
+            -0.031362712383270264,
+            -0.014787442982196808,
+            -0.01056029461324215,
+            0.026862844824790955,
+            0.007757347077131271,
+            -0.015726808458566666,
+            -0.007514929864555597,
+            -0.01545409020036459,
+            0.012885984033346176,
+            0.0032442226074635983,
+            0.005764981731772423,
+            0.015817714855074883,
+            0.021893294528126717,
+            0.01746918261051178,
+            -0.00795431062579155,
+            0.006852070800960064,
+            0.004674104508012533,
+            0.028090080246329308,
+            -0.01777220331132412,
+            0.020802417770028114,
+            0.009954252280294895,
+            -0.003935490269213915,
+            0.005685438401997089,
+            -0.014257155358791351,
+            0.008530051447451115,
+            -0.007325541693717241,
+            0.030514251440763474,
+            0.0033067206386476755,
+            0.014075342565774918,
+            -0.02062060497701168,
+            0.006575563922524452,
+            -0.03172633796930313,
+            0.009022461250424385,
+            0.013704141601920128,
+            0.012976890429854393,
+            -0.007677804213017225,
+            -0.024181105196475983,
+            -0.0119314668700099,
+            0.0057990713976323605,
+            -0.01279507763683796,
+            -0.01046181283891201,
+            -0.0034828518982976675,
+            0.012446602806448936,
+            0.018847929313778877,
+            -0.01848430372774601,
+            -0.019923655316233635,
+            -0.009583051316440105,
+            0.02898399345576763,
+            -0.015787413343787193,
+            -0.001471547526307404,
+            0.02413565292954445,
+            0.021241798996925354,
+            0.015022284351289272,
+            0.002155239460989833,
+            0.020090317353606224,
+            -0.0007111532613635063,
+            -0.0237265732139349,
+            -0.02813553437590599,
+            0.007302815094590187,
+            -0.006912675220519304,
+            0.01943882182240486,
+            -0.0035548193845897913,
+            -0.005658924113959074,
+            -0.0034033088013529778,
+            -0.007503566797822714,
+            -0.0165904201567173,
+            -0.01787826046347618,
+            0.0021647089160978794,
+            -0.008454295806586742,
+            0.005863463506102562,
+            -0.0002795845502987504,
+            -0.013105674646794796,
+            0.013401119969785213,
+            -0.014075342565774918,
+            0.005772557109594345,
+            -0.002653330797329545,
+            0.001962063368409872,
+            0.0035926972050219774,
+            0.009113367646932602,
+            -0.007336904760450125,
+            -0.06381630152463913,
+            -0.013105674646794796,
+            -0.003704436356201768,
+            0.006071790587157011,
+            -0.00019578021601773798,
+            -0.016635872423648834,
+            -0.013196581043303013,
+            0.01706010289490223,
+            0.008863375522196293,
+            -0.013870803639292717,
+            0.016954045742750168,
+            0.009961827658116817,
+            0.003622999181970954,
+            -0.024044746533036232,
+            -0.0008148433989845216,
+            0.008757317438721657,
+            0.032756611704826355,
+            -0.008643684908747673,
+            -0.0003839849086944014,
+            0.0036684523802250624,
+            -0.0033332351595163345,
+            -0.010052734054625034,
+            -0.014582903124392033,
+            -0.011848135851323605,
+            0.011908739805221558,
+            0.01790856383740902,
+            -0.014264730736613274,
+            -0.02007516659796238,
+            -0.0007106798002496362,
+            0.012931437231600285,
+            -0.026590125635266304,
+            0.0007244104635901749,
+            -0.007787649054080248,
+            -0.01052999310195446,
+            0.018302490934729576,
+            -0.019014591351151466,
+            0.005249845329672098,
+            0.00048294031876139343,
+            0.008507325313985348,
+            -0.023544760420918465,
+            -0.024893205612897873,
+            0.0035131541080772877,
+            -0.023847781121730804,
+            0.03612014651298523,
+            -0.021590271964669228,
+            -0.012082977220416069,
+            0.034483831375837326,
+            -0.0037423139438033104,
+            0.01702980138361454,
+            0.0023995505180209875,
+            0.02739313244819641,
+            -0.011439057067036629,
+            -0.00617784820497036,
+            0.012499631382524967,
+            0.009575475007295609,
+            0.0032593735959380865,
+            0.03775646165013313,
+            -0.01257538702338934,
+            -0.013901105150580406,
+            -0.005556654185056686,
+            -0.012954163365066051,
+            0.0006775368237867951,
+            0.014431392773985863,
+            0.028393102809786797,
+            0.004257450345903635,
+            -0.00628769351169467,
+            -0.006143758539110422,
+            -0.0014649189542979002,
+            -0.004242299124598503,
+            -0.00947699323296547,
+            -0.00715130427852273,
+            0.017287369817495346,
+            -0.025241680443286896,
+            0.016711628064513206,
+            -0.006420265417546034,
+            0.021135739982128143,
+            -0.011757229454815388,
+            0.007007369305938482,
+            -0.008961857296526432,
+            0.003492321353405714,
+            0.0023635667748749256,
+            -0.005109698045998812,
+            0.009287605062127113,
+            -0.014772292226552963,
+            0.008454295806586742,
+            0.013742019422352314,
+            0.0011183382011950016,
+            -0.010590597055852413,
+            -0.019272159785032272,
+            0.0022972808219492435,
+            -0.000710206339135766,
+            0.01181783340871334,
+            -0.0047498601488769054,
+            -0.000981031684204936,
+            0.0002921315317507833,
+            -0.014476845972239971,
+            -0.009575475007295609,
+            0.0006050958181731403,
+            0.005302873905748129,
+            -0.008961857296526432,
+            0.010378481820225716,
+            -0.005693014245480299,
+            -0.00034160923678427935,
+            0.02627195231616497,
+            0.006526322569698095,
+            -0.007632351014763117,
+            -0.014098069630563259,
+            0.01563590206205845,
+            0.015370759181678295,
+            -0.007454325910657644,
+            0.01933276280760765,
+            -0.016196493059396744,
+            -0.0008366230758838356,
+            0.013916256837546825,
+            -0.009219424799084663,
+            0.0017698341980576515,
+            -0.008196728304028511,
+            0.03554440662264824,
+            -0.0008787619881331921,
+            -0.0035415622405707836,
+            -0.0006401326390914619,
+            0.012976890429854393,
+            -0.0026779514737427235,
+            0.001182730309665203,
+            0.01132542360574007,
+            0.005321812815964222,
+            0.0185146052390337,
+            0.00328020635060966,
+            -0.03378688171505928,
+            -0.007780073676258326,
+            -0.0065490491688251495,
+            -0.008090670220553875,
+            0.009022461250424385,
+            0.025196226313710213,
+            0.01990850456058979,
+            0.006408901885151863,
+            0.001881573349237442,
+            0.018423698842525482,
+            0.015757111832499504,
+            -0.01489350013434887,
+            0.016363153234124184,
+            0.02802947722375393,
+            -0.010030006989836693,
+            0.022741753607988358,
+            -0.005253633018583059,
+            -0.008893677033483982,
+            0.008946705609560013,
+            -0.0015378334792330861,
+            -0.017393426969647408,
+            -0.015681356191635132,
+            0.005022579338401556,
+            -0.008999735116958618,
+            -0.0022897052112966776,
+            -0.018605511635541916,
+            0.004670316819101572,
+            0.017105557024478912,
+            -0.004965762607753277,
+            -3.5243989259470254e-05,
+            0.019484274089336395,
+            -0.006965703796595335,
+            -0.016908591613173485,
+            0.006325571332126856,
+            -0.015007133595645428,
+            0.007946735247969627,
+            -0.013666263781487942,
+            0.010492115281522274,
+            -0.006098304875195026,
+            0.018817627802491188,
+            0.005446809343993664,
+            -0.008090670220553875,
+            -0.0008958069374784827,
+            -0.018984289839863777,
+            0.02065090648829937,
+            -0.033089932054281235,
+            0.00839369185268879,
+            -0.026226500049233437,
+            -0.0009781909175217152,
+            0.0008849170990288258,
+            0.014605630189180374,
+            -0.001218714052811265,
+            0.019696388393640518,
+            -0.002751812804490328,
+            -0.020090317353606224,
+            -0.007086912170052528,
+            -0.0010634156642481685,
+            -0.01618134044110775,
+            -0.030923331156373024,
+            -0.004272601567208767,
+            0.0036267871037125587,
+            -0.004711982328444719,
+            0.0021135739516466856,
+            0.017984317615628242,
+            -0.005613470915704966,
+            0.009173971600830555,
+            -0.02690829709172249,
+            -0.007416448090225458,
+            -0.00355103169567883,
+            0.0010776197304949164,
+            -0.011242092587053776,
+            0.0010596278589218855,
+            0.011226941831409931,
+            -0.019044892862439156,
+            0.013461724855005741,
+            -0.021378157660365105,
+            -0.004371083341538906,
+            -0.008507325313985348,
+            -0.00135412672534585,
+            -0.013446573168039322,
+            -0.028120383620262146,
+            0.025029566138982773,
+            -0.00021495578403118998,
+            -0.003528305096551776,
+            -0.011999646201729774,
+            0.023181134834885597,
+            0.002914686920121312,
+            0.027817361056804657,
+            -0.03157482668757439,
+            0.0013427634257823229,
+            0.011552689597010612,
+            0.0025984083767980337,
+            -0.017287369817495346,
+            0.004746072459965944,
+            -0.005234694108366966,
+            0.02210540883243084,
+            -0.0011694730492308736,
+            -0.0005340751959010959,
+            -0.008234606124460697,
+            0.030999086797237396,
+            0.009136094711720943,
+            0.0012944694608449936,
+            -0.013226882554590702,
+            -0.008060368709266186,
+            -0.002759388415142894,
+            -0.0011069749016314745,
+            0.012855681590735912,
+            0.01290113478899002,
+            -0.012105703353881836,
+            0.0013607552973553538,
+            -0.006878585088998079,
+            0.0011694730492308736,
+            -0.002935519441962242,
+            -0.0004966709529981017,
+            -0.01149208564311266,
+            0.003257479751482606,
+            0.016272246837615967,
+            -0.00780280027538538,
+            -0.006480869371443987,
+            0.0007750718505121768,
+            0.009181547909975052,
+            0.012461753562092781,
+            -0.012742049060761929,
+            -0.018226735293865204,
+            -0.011908739805221558,
+            0.016029831022024155,
+            0.010522416792809963,
+            -0.008204303681850433,
+            -0.013234458863735199,
+            -0.02583257108926773,
+            0.01169662456959486,
+            -0.03293842449784279,
+            -0.012242063879966736,
+            0.005821797996759415,
+            0.028559764847159386,
+            0.019999410957098007,
+            0.012226912193000317,
+            0.001332347048446536,
+            0.002950670663267374,
+            0.00023342114582192153,
+            -0.01916610263288021,
+            0.014408666640520096,
+            -0.011143610812723637,
+            -0.018166130408644676,
+            0.0075906855054199696,
+            0.011939042247831821,
+            0.03148392215371132,
+            0.006942977197468281,
+            -0.0010160685051232576,
+            0.01686313934624195,
+            -0.001218714052811265,
+            -0.013582932762801647,
+            0.01420412678271532,
+            -0.024256860837340355,
+            0.016438908874988556,
+            0.02860521711409092,
+            -0.004780162125825882,
+            0.0008531945641152561,
+            -0.020938776433467865,
+            0.00482561532407999,
+            -0.011529963463544846,
+            -0.002177966060116887,
+            -0.003287781961262226,
+            -0.03460504114627838,
+            0.007946735247969627,
+            -0.006825556512922049,
+            0.01046181283891201,
+            -0.00814369972795248,
+            0.006655106786638498,
+            -0.012431452050805092,
+            0.004253662656992674,
+            0.01513591781258583,
+            0.016105586662888527,
+            0.007060397882014513,
+            -0.025211378931999207,
+            -0.009196698665618896,
+            0.008105821907520294,
+            -0.01349202636629343,
+            -0.02295386977493763,
+            -0.006852070800960064,
+            0.009840618818998337,
+            -0.01061332318931818,
+            -0.00488243205472827,
+            -0.028180986642837524,
+            -0.013681414537131786,
+            -0.0014327229000627995,
+            -0.017817657440900803,
+            -0.005556654185056686,
+            -0.003100287402048707,
+            0.01571165770292282,
+            -0.0049733384512364864,
+            -0.006185423582792282,
+            0.004742284305393696,
+            -0.023878084495663643,
+            0.0037423139438033104,
+            -0.008280059322714806,
+            0.0008266801596619189,
+            -0.0032517979852855206,
+            -0.0014762822538614273,
+            -0.007969462312757969,
+            -0.014332910999655724,
+            0.006893736310303211,
+            -0.016469212248921394,
+            0.002954458352178335,
+            -0.015173795633018017,
+            -0.016499513760209084,
+            -0.00682934420183301,
+            0.00775355938822031,
+            0.021575121209025383,
+            0.006541473791003227,
+            0.021484214812517166,
+            0.005469535943120718,
+            0.028680972754955292,
+            0.0003953482082579285,
+            0.003041577059775591,
+            -8.958956641436089e-06,
+            0.003232859307900071,
+            0.019347915425896645,
+            0.006571775767952204,
+            0.018984289839863777,
+            -0.008742166683077812,
+            0.010514841414988041,
+            -0.010355755686759949,
+            0.010014856234192848,
+            0.006333146709948778,
+            -0.006473293993622065,
+            -0.003799130441620946,
+            -0.010348179377615452,
+            -0.0031419529113918543,
+            -0.016544966027140617,
+            0.0027366618160158396,
+            0.00693161366507411,
+            0.0071588801220059395,
+            -0.011582992039620876,
+            -0.005230906419456005,
+            -0.014704111963510513,
+            -0.0064354161731898785,
+            -0.01831764169037342,
+            0.014385939575731754,
+            0.029226411134004593,
+            -0.016332851722836494,
+            -0.01977214403450489,
+            -0.007772498298436403,
+            0.0007296186522580683,
+            -0.006636167876422405,
+            -0.01240872498601675,
+            -0.009151245467364788,
+            -0.035059571266174316,
+            -0.017666146159172058,
+            0.007586897816509008,
+            0.02484775334596634,
+            0.01387837901711464,
+            -0.013628385961055756,
+            -0.0006983695784583688,
+            -0.016984347254037857,
+            -0.010787560604512691,
+            -0.01535560842603445,
+            -0.02054484933614731,
+            -0.02498411200940609,
+            0.0077952248975634575,
+            -0.004033972043544054,
+            0.014931377954781055,
+            0.008113397285342216,
+            0.0032631612848490477,
+            -0.004371083341538906,
+            -0.0024109138175845146,
+            -0.013196581043303013,
+            0.0012925754999741912,
+            -0.004787737503647804,
+            -0.0017291157273575664,
+            0.03554440662264824,
+            -0.008098246529698372,
+            -0.005871038883924484,
+            0.011060279794037342,
+            0.011128460057079792,
+            -0.013764745555818081,
+            0.0006642796797677875,
+            -0.011249667964875698,
+            0.021847840398550034,
+            -0.021620575338602066,
+            0.019257009029388428,
+            0.0014507147716358304,
+            -0.010840589180588722,
+            0.028999146074056625,
+            0.01892368495464325,
+            0.00743159931153059,
+            0.006579351611435413,
+            0.01672677882015705,
+            0.0044316877610981464,
+            0.016196493059396744,
+            -0.0026552248746156693,
+            -0.004632438998669386,
+            0.009295180439949036,
+            0.015681356191635132,
+            -0.000702630786690861,
+            -0.006942977197468281,
+            0.016438908874988556,
+            0.004056698642671108,
+            0.0075414446182549,
+            -0.01356020662933588,
+            -0.008961857296526432,
+            0.001507531269453466,
+            0.006257391534745693,
+            -0.005939218681305647,
+            -0.0026003022212535143,
+            0.005530139897018671,
+            0.003943065647035837,
+            -0.013302638195455074,
+            0.01689344085752964,
+            -0.016166189685463905,
+            0.008742166683077812,
+            -0.0037972365971654654,
+            0.0121360057964921,
+            -0.006799042224884033,
+            0.0013153020991012454,
+            -0.01818128302693367,
+            -0.003869204083457589,
+            0.0038597346283495426,
+            0.012173883616924286,
+            -0.01262841559946537,
+            0.025484098121523857,
+            0.008166425861418247,
+            0.006602078210562468,
+            0.0059467945247888565,
+            -0.012855681590735912,
+            0.027211319655179977,
+            0.005969521123915911,
+            0.009886072017252445,
+            -0.008893677033483982,
+            0.011136035434901714,
+            0.035362593829631805,
+            0.03060515783727169,
+            0.014135947450995445,
+            0.004469565115869045,
+            -0.018999440595507622,
+            0.015120766125619411,
+            0.0018427487229928374,
+            0.0023806116078048944,
+            0.00863610953092575,
+            -0.011348150670528412,
+            0.01305264513939619,
+            -0.009484568610787392,
+            0.015241974964737892,
+            -0.021272100508213043,
+            -0.011749654076993465,
+            -0.014408666640520096,
+            -0.021075136959552765,
+            -0.01430260855704546,
+            0.034180812537670135,
+            -0.002346521709114313,
+            -0.010393633507192135,
+            0.0045112306252121925,
+            0.008477022871375084,
+            -0.006079366430640221,
+            0.0053142374381423,
+            -0.008204303681850433,
+            -0.013749594800174236,
+            -0.01889338344335556,
+            0.006628592498600483,
+            -0.01550711877644062,
+            0.007651289459317923,
+            -0.003304826794192195,
+            0.0037688282318413258,
+            0.007014944683760405,
+            0.018105527386069298,
+            0.00530666159465909,
+            -0.00135412672534585,
+            0.0014033677289262414,
+            0.0016865032957866788,
+            -0.00931033119559288,
+            0.0108330138027668,
+            0.015408637002110481,
+            -0.0011921997647732496,
+            -0.011279970407485962,
+            -0.009136094711720943,
+            -0.042877525091171265,
+            0.0006264019757509232,
+            -0.013128400780260563,
+            0.0064354161731898785,
+            -0.011196639388799667,
+            0.0022234192583709955,
+            0.015863168984651566,
+            -0.0054051438346505165,
+            -0.0010918239131569862,
+            0.02193874679505825,
+            0.013014768250286579,
+            0.021075136959552765,
+            0.02065090648829937,
+            0.014188976027071476,
+            -0.015302578918635845,
+            -0.03469594568014145,
+            0.002221525413915515,
+            -0.010030006989836693,
+            0.013772320933640003,
+            0.004238511435687542,
+            0.016060132533311844,
+            -0.023893235251307487,
+            -0.011878437362611294,
+            -0.006662682630121708,
+            0.014431392773985863,
+            0.01491622719913721,
+            -0.0032271775417029858,
+            0.02690829709172249,
+            0.009280029684305191,
+            0.0020510759204626083,
+            -0.02281750924885273,
+            0.003312402404844761,
+            0.005321812815964222,
+            -0.008370965719223022,
+            -0.016681326553225517,
+            -0.0004552422615233809,
+            -0.0012168202083557844,
+            -0.0007817004225216806,
+            -0.021650876849889755,
+            0.018832778558135033,
+            -0.004018820822238922,
+            0.0037574649322777987,
+            0.013840501196682453,
+            0.01516621932387352,
+            0.017984317615628242,
+            0.008370965719223022,
+            0.001497115008533001,
+            -0.006605865899473429,
+            -0.0007973249885253608,
+            -0.0005828426801599562,
+            -0.007389933802187443,
+            -0.005598319694399834,
+            0.006382387597113848,
+            -0.01134057529270649,
+            0.008810346014797688,
+            0.000896753859706223,
+            0.006113456096500158,
+            0.010605747811496258,
+            -0.004836978856474161,
+            -0.011628445237874985,
+            0.014628356322646141,
+            -0.0010548931313678622,
+            -0.0055755930952727795,
+            -0.011060279794037342,
+            0.003624893259257078,
+            0.013575357384979725,
+            -0.003096499713137746,
+            -0.013499601744115353,
+            0.00910579226911068,
+            0.002009410411119461,
+            -0.027438584715127945,
+            0.015499543398618698,
+            0.010295150801539421,
+            -0.013370818458497524,
+            0.004980913829058409,
+            -0.003371112747117877,
+            -0.03027183562517166,
+            0.004791525658220053,
+            0.002660906407982111,
+            -0.010643625631928444,
+            0.028938541188836098,
+            -0.003821857040748,
+            0.029256712645292282,
+            -0.004980913829058409,
+            -0.006905099377036095,
+            -0.011355726048350334,
+            0.009416389279067516,
+            -0.018681267276406288,
+            -0.0037953427527099848,
+            -0.008727015927433968,
+            -0.00915882084518671,
+            -0.0009838725673034787,
+            -0.005185453221201897,
+            0.006984642706811428,
+            0.005121061112731695,
+            -0.012704171240329742,
+            0.006757376715540886,
+            -0.010257272981107235,
+            -0.00020856392802670598,
+            0.0006718551740050316,
+            -0.008408842608332634,
+            0.013885954394936562,
+            0.005996035411953926,
+            -0.008340663276612759,
+            -0.0005032995832152665,
+            0.027817361056804657,
+            0.05051366239786148,
+            -0.018863080069422722,
+            0.0009270560112781823,
+            0.01452229917049408,
+            -0.0028824908658862114,
+            -0.0021590273827314377,
+            -0.011976920068264008,
+            0.005821797996759415,
+            -0.019423669204115868,
+            0.013317788951098919,
+            0.006473293993622065,
+            -0.008833073079586029,
+            0.003202557098120451,
+            -0.016711628064513206,
+            0.002621134975925088,
+            -0.0018058179412037134,
+            -0.018681267276406288,
+            -0.015529844909906387,
+            -0.012060250155627728,
+            -0.024181105196475983,
+            0.0004235197266098112,
+            -0.0008546149474568665,
+            -0.013870803639292717,
+            0.006579351611435413,
+            0.008431569673120975,
+            0.032483890652656555,
+            0.0011912527261301875,
+            0.010128489695489407,
+            -2.1513333194889128e-05,
+            -0.0010416359873488545,
+            -0.005988460034132004,
+            -0.01523439958691597,
+            0.00015387803432531655,
+            -0.002342734020203352,
+            0.022590242326259613,
+            0.005549078807234764,
+            0.017560089007019997,
+            0.0017887729918584228,
+            -0.016514664515852928,
+            -0.00596573343500495,
+            -0.007408872712403536,
+            -0.018060073256492615,
+            0.0062649669125676155,
+            -0.005473323632031679,
+            0.009605777449905872,
+            -0.015863168984651566,
+            -0.02731737680733204,
+            0.00792400911450386,
+            0.012234487570822239,
+            0.028514310717582703,
+            -0.0035623949952423573,
+            0.001716805505566299,
+            -0.005533927585929632,
+            0.016408607363700867,
+            0.032180868089199066,
+            -0.0008550884085707366,
+            0.006950552575290203,
+            -0.019999410957098007,
+            0.0017404790269210935,
+            0.013704141601920128,
+            -0.008401267230510712,
+            -0.01821158453822136,
+            0.003939277958124876,
+            -0.020635755732655525,
+            0.004583198111504316,
+            -0.020393338054418564,
+            0.030408194288611412,
+            -0.0023522034753113985,
+            0.008105821907520294,
+            0.004310478921979666,
+            0.01760554127395153,
+            -0.0031646795105189085,
+            0.010545143857598305,
+            0.013916256837546825,
+            0.014908651821315289,
+            0.0009980766335502267,
+            -0.011189064010977745,
+            -0.03790797293186188,
+            0.001859793672338128,
+            0.00817400123924017,
+            0.018635815009474754,
+            -0.01980244740843773,
+            0.011681473813951015,
+            -0.01296931505203247,
+            -0.020363036543130875,
+            -0.01003758329898119,
+            0.011575416661798954,
+            -0.006139970384538174,
+            0.03378688171505928,
+            -0.005874827038496733,
+            -0.008204303681850433,
+            0.005136212334036827,
+            0.010454237461090088,
+            0.0066778333857655525,
+            0.014704111963510513,
+            0.0326354019343853,
+            0.010196669027209282,
+            0.014173824340105057,
+            -0.004859705455601215,
+            0.02399929240345955,
+            -0.003931702114641666,
+            0.003954428713768721,
+            0.00509454682469368,
+            -0.022832660004496574,
+            -0.0040983641520142555,
+            -0.003931702114641666,
+            -0.016711628064513206,
+            -0.007458113599568605,
+            0.0032366469968110323,
+            0.008802770636975765,
+            -0.014045040123164654,
+            -0.00649980828166008,
+            -0.01350717805325985,
+            0.01218903437256813,
+            0.002730980049818754,
+            0.018302490934729576,
+            -0.029802151024341583,
+            0.009825468063354492,
+            -0.017484333366155624,
+            -0.017044952139258385,
+            0.006511171814054251,
+            -0.0038332203403115273,
+            0.006855858489871025,
+            -0.018272189423441887,
+            -0.01318900566548109,
+            0.012817803770303726,
+            0.0007897494360804558,
+            0.013060221448540688,
+            -0.009825468063354492,
+            0.0009706153068691492,
+            0.006064215209335089,
+            -0.009833043441176414,
+            -0.010961798019707203,
+            -0.007977037690579891,
+            -0.009681533090770245,
+            -0.008848223835229874,
+            0.005621046293526888,
+            0.01656011864542961,
+            -0.013742019422352314,
+            -0.003685497445985675,
+            0.016135888174176216,
+            0.006416477728635073,
+            -0.008166425861418247,
+            0.004075637552887201,
+            -0.003793448908254504,
+            -9.948806837201118e-05,
+            0.0013740125577896833,
+            0.012439027428627014,
+            -0.0017632056260481477,
+            0.024665940552949905,
+            -0.02654467150568962,
+            0.007598260883241892,
+            -0.0054051438346505165,
+            0.005446809343993664,
+            0.018226735293865204,
+            -0.02366596832871437,
+            -0.013082947582006454,
+            0.006276329979300499,
+            -0.01966608688235283,
+            -0.008060368709266186,
+            0.0054051438346505165,
+            -0.00016405766655225307,
+            -0.019923655316233635,
+            -0.012923861853778362,
+            -0.034786853939294815,
+            -0.02309022843837738,
+            0.024665940552949905,
+            0.008401267230510712,
+            -0.0073672072030603886,
+            0.018060073256492615,
+            -0.018438849598169327,
+            0.0172722190618515,
+            -0.002225313102826476,
+            -0.006613441277295351,
+            0.003935490269213915,
+            0.007674016058444977,
+            0.025362888351082802,
+            0.002653330797329545,
+            0.008264907635748386,
+            -0.0042082094587385654,
+            -0.014931377954781055,
+            -0.006321783177554607,
+            0.01980244740843773,
+            0.02471139281988144,
+            0.018605511635541916,
+            -0.018575210124254227,
+            0.010742107406258583,
+            -0.011749654076993465,
+            0.01818128302693367,
+            -0.006537686102092266,
+            0.00039037675014697015,
+            0.01859036087989807,
+            0.007969462312757969,
+            0.025241680443286896,
+            0.019923655316233635,
+            0.006852070800960064,
+            0.014128371141850948,
+            0.011908739805221558,
+            0.001204509986564517,
+            -0.012832955457270145,
+            -0.01154511421918869,
+            -0.02840825356543064,
+            0.042847223579883575,
+            0.0025434857234358788,
+            0.008317936211824417,
+            -0.027635548263788223,
+            -0.00021389048197306693,
+            -0.0034942151978611946,
+            -0.018832778558135033,
+            0.005939218681305647,
+            0.03260510042309761,
+            0.03284751623868942,
+            0.021438762545585632,
+            -2.888172457460314e-05,
+            0.0012234487803652883,
+            0.0329081192612648,
+            -0.013673839159309864,
+            -0.00937851145863533,
+            -0.01034060399979353,
+            -0.018393397331237793,
+            -0.006852070800960064,
+            -0.0032423287630081177,
+            -0.01031030248850584,
+            -0.0061172437854111195,
+            -0.009272454306483269,
+            0.0238629337400198,
+            0.0001374249259242788,
+            -0.0022594030015170574,
+            0.006605865899473429,
+            -0.020878173410892487,
+            0.011537538841366768,
+            -0.02813553437590599,
+            -0.013431422412395477,
+            -0.008651260286569595,
+            -0.006950552575290203,
+            -0.018802475184202194,
+            0.008643684908747673,
+            -0.006942977197468281,
+            0.021438762545585632,
+            0.002914686920121312,
+            -0.023590214550495148,
+            0.007530081085860729,
+            0.015317730605602264,
+            -0.0003825644962489605,
+            0.014037464745342731,
+            -0.003890036838129163,
+            0.009060339070856571,
+            0.0045112306252121925,
+            0.011681473813951015,
+            0.016954045742750168,
+            0.010651201009750366,
+            0.009030036628246307,
+            0.003437398700043559,
+            -0.001167579204775393,
+            0.01878732442855835,
+            -0.0007163614500313997,
+            -0.03405960276722908,
+            2.9236827685963362e-05,
+            -0.014628356322646141,
+            0.029499130323529243,
+            -0.009795165620744228,
+            0.009514871053397655,
+            0.008105821907520294,
+            0.007098275702446699,
+            -0.0067119235172867775,
+            0.007916433736681938,
+            -0.008742166683077812,
+            0.0052763596177101135,
+            0.015832867473363876,
+            -0.0042877523228526115,
+            -0.0014980619307607412,
+            0.007840678095817566,
+            0.00407184986397624,
+            -0.005454384721815586,
+            -0.019711541011929512,
+            -0.012287517078220844,
+            0.00726114958524704,
+            -0.02065090648829937,
+            -0.004367295652627945,
+            0.01161329448223114,
+            -0.005454384721815586,
+            -0.009620928205549717,
+            0.01393140759319067,
+            0.02136300690472126,
+            -0.002382505452260375,
+            0.022893264889717102,
+            -0.01349202636629343,
+            -0.009833043441176414,
+            -0.007984613068401814,
+            -0.004374871030449867,
+            -0.041120000183582306,
+            0.025984082370996475,
+            0.01068150345236063,
+            -0.006871009711176157,
+            0.01895398646593094,
+            0.0022499337792396545,
+            -0.002954458352178335,
+            -0.0011041341349482536,
+            0.00010499217023607343,
+            -0.03281721472740173,
+            0.03660498186945915,
+            -0.007007369305938482,
+            0.0005449649761430919,
+            0.010272424668073654,
+            0.042029064148664474,
+            -0.004590773954987526,
+            -0.005859675817191601,
+            0.01225721463561058,
+            0.005336964037269354,
+            0.031938452273607254,
+            -0.010416359640657902,
+            -0.014340486377477646,
+            -0.01815097965300083,
+            -0.004833190701901913,
+            -0.01939336769282818,
+            -0.0038313264958560467,
+            0.014037464745342731,
+            -0.019650936126708984,
+            -0.011242092587053776,
+            -0.001287840772420168,
+            -0.012537509202957153,
+            -0.0010747789638116956,
+            0.008598231710493565,
+            0.017832808196544647,
+            -0.011348150670528412,
+            -0.004450626205652952,
+            -2.2904154320713133e-05,
+            -0.005003640428185463,
+            0.0185146052390337,
+            0.0020056227222085,
+            -0.01702980138361454,
+            0.01321930717676878,
+            -0.00526499655097723,
+            0.006749800872057676,
+            0.0074429623782634735,
+            0.007939159870147705,
+            0.05145302787423134,
+            0.010317877866327763,
+            -0.001339922659099102,
+            -0.01607528328895569,
+            0.023196285590529442,
+            -0.01749948412179947,
+            -0.014181400649249554,
+            -0.02119634486734867,
+            -0.0029923359397798777,
+            -0.007060397882014513,
+            -0.028014326468110085,
+            -9.137987944995984e-05,
+            0.0016221113037317991,
+            -0.0024980325251817703,
+            -0.010045158676803112,
+            0.0005208179936744273,
+            -0.011529963463544846,
+            -0.006954340264201164,
+            -0.008545203134417534,
+            -0.016044981777668,
+            0.02240842953324318,
+            -0.021817538887262344,
+            -0.008855799213051796,
+            0.011514811776578426,
+            0.03118089959025383,
+            -0.005011215806007385,
+            -0.014173824340105057,
+            -0.019817598164081573,
+            0.0025643182452768087,
+            -0.01763584464788437,
+            0.012181458994746208,
+            -0.01225721463561058,
+            0.015045011416077614,
+            -0.03142331540584564,
+            -0.006079366430640221,
+            -0.008992158807814121,
+            0.012840530835092068,
+            0.019817598164081573,
+            -0.0025093958247452974,
+            0.00606800289824605,
+            -0.010295150801539421,
+            0.030211230739951134,
+            -0.00012286569108255208,
+            -0.018605511635541916,
+            0.00873459130525589,
+            0.007977037690579891,
+            0.012537509202957153,
+            -0.004435475450009108,
+            0.00257757562212646,
+            0.011567841283977032,
+            0.012742049060761929,
+            -0.01634800247848034,
+            -0.003174148965626955,
+            -0.017969166859984398,
+            0.005996035411953926,
+            0.006999793462455273,
+            0.021711481735110283,
+            -0.0013550736475735903,
+            0.018272189423441887,
+            -0.01563590206205845,
+            0.00019412307301536202,
+            -0.011749654076993465,
+            -0.0082270298153162,
+            -0.0028237802907824516,
+            -0.003958216868340969,
+            0.006685409229248762,
+            -0.028211290016770363,
+            -0.01274962443858385,
+            0.005287723150104284,
+            -0.004992276895791292,
+            -0.03405960276722908,
+            0.0053710537031292915,
+            -0.02725677192211151,
+            0.013128400780260563,
+            -0.002397656673565507,
+            -0.017847958952188492,
+            -0.00030846628942526877,
+            0.026332557201385498,
+            -0.003677921835333109,
+            -0.012537509202957153,
+            0.015544996596872807,
+            -0.006757376715540886,
+            -0.02119634486734867,
+            -0.005420294590294361,
+            0.014658658765256405,
+            -0.0010719380807131529,
+            -0.012393574230372906,
+            -0.029635490849614143,
+            0.012992041185498238,
+            0.0026874206960201263,
+            -0.007386146113276482,
+            0.010893618687987328,
+            0.025953780859708786,
+            0.008151275105774403,
+            -0.005924067925661802,
+            0.0015757111832499504,
+            0.0049581872299313545,
+            0.00300938100554049,
+            0.020287280902266502,
+            0.02728707529604435,
+            -0.008014915511012077,
+            -0.0011145505122840405,
+            -0.014954105019569397,
+            0.015802564099431038,
+            0.0013020449550822377,
+            0.010507266037166119,
+            0.021893294528126717,
+            0.007045246660709381,
+            -0.005738467443734407,
+            0.003905187826603651,
+            -0.023135682567954063,
+            0.01420412678271532,
+            -0.012272365391254425,
+            0.000622140767518431,
+            0.008749742060899734,
+            -0.007283876184374094,
+            0.01437078882008791,
+            -0.0015245762187987566,
+            0.0339989997446537,
+            0.025211378931999207,
+            0.0043218424543738365,
+            0.0007362472242675722,
+            -0.028559764847159386,
+            0.003547244006767869,
+            0.008651260286569595,
+            0.014499573037028313,
+            -0.0076664406806230545,
+            0.006261179223656654,
+            -0.004355932120233774,
+            0.00959820207208395,
+            -0.0065490491688251495,
+            -0.015893470495939255,
+            0.006113456096500158,
+            0.0012689019786193967,
+            0.006795254070311785,
+            0.0023408401757478714,
+            -0.008863375522196293,
+            -0.024362917989492416,
+            -0.025984082370996475,
+            0.021984200924634933,
+            -0.03918066248297691,
+            0.0073293293826282024,
+            0.011234517209231853,
+            0.03324144333600998,
+            -0.008158850483596325,
+            -0.004980913829058409,
+            0.006276329979300499,
+            -0.016711628064513206,
+            -0.004147605039179325,
+            0.008204303681850433,
+            0.012385998852550983,
+            -0.002420383272692561,
+            0.00974971242249012,
+            0.01318900566548109,
+            -0.012378423474729061,
+            0.03660498186945915,
+            0.016514664515852928,
+            0.01467380952090025,
+            -0.004007457755506039,
+            0.01312082540243864,
+            0.0066172294318675995,
+            0.019044892862439156,
+            0.025165924802422523,
+            -0.017454031854867935,
+            -0.002365460619330406,
+            -0.02292356640100479,
+            0.024165954440832138,
+            0.00726114958524704,
+            0.004556683823466301,
+            -0.008310360834002495,
+            0.014870774000883102,
+            0.0017954016802832484,
+            0.0028199926018714905,
+            -0.006696772295981646,
+            -0.010173942893743515,
+            0.019241856411099434,
+            0.0034317169338464737,
+            0.014870774000883102,
+            4.8501173296244815e-05,
+            -0.005503625608980656,
+            0.006052851676940918,
+            0.014832896180450916,
+            -0.023590214550495148,
+            -0.00203592493198812,
+            -0.019484274089336395,
+            -0.03202936053276062,
+            0.009022461250424385,
+            0.0016135887708514929,
+            -0.007052822504192591,
+            0.03230207785964012,
+            -0.010083036497235298,
+            -0.019787294790148735,
+            -0.008067944087088108,
+            -0.013113250024616718,
+            -0.010469388216733932,
+            0.0008271536789834499,
+            0.003908975515514612,
+            -0.00012428610352799296,
+            -0.009317907504737377,
+            -0.005749830510467291,
+            0.01990850456058979,
+            -0.0030358952935785055,
+            -0.00753386877477169,
+            0.00868156272917986
+          ],
+          "embedding": [
+            0.0040707582550429606,
+            -0.010230625151989194,
+            -0.021520242101903186,
+            -0.010115756270640508,
+            0.026738734586159035,
+            0.01796370233223267,
+            -0.012245199029994153,
+            0.06506924494147726,
+            -0.0014087587222033442,
+            0.009300068719271429,
+            0.003787396973047584,
+            0.002818772639754174,
+            0.022129077062176306,
+            -0.021526709808445683,
+            -0.0034388135367225824,
+            0.02763761747879405,
+            0.013210013805395276,
+            0.025255872672296602,
+            0.00029011480909986703,
+            -0.05438974762290585,
+            0.016671640187490865,
+            -0.016426252259893328,
+            -0.056591180651012554,
+            0.009335648425523075,
+            0.006141624459418827,
+            0.02568943956694641,
+            0.019377132054042116,
+            0.017946464337078746,
+            -0.057184776985927686,
+            0.03208527139096687,
+            0.030132392063755547,
+            -0.003846032371791839,
+            -0.008376524427531415,
+            -0.0010513963058713435,
+            -0.02734868388724371,
+            -0.014879400370545479,
+            0.04383920959748825,
+            0.02563613659516289,
+            -0.020117467074438997,
+            0.0398016699155716,
+            0.029896497884197775,
+            -0.0004195844530067313,
+            -0.023302720788253625,
+            -0.001168337916577517,
+            0.009149021369661628,
+            -0.02508145721900992,
+            -0.004307594138480351,
+            0.00550940415564303,
+            0.014016998918718476,
+            -0.028607379616884317,
+            -0.0018317656587526954,
+            -0.01973464940415756,
+            0.03762738392056255,
+            0.015358106058583114,
+            -0.03931907682104936,
+            0.011519637298832444,
+            0.004482080558226775,
+            0.0747867063383242,
+            0.05550679018263077,
+            0.023243129601931176,
+            0.02671631686665238,
+            -0.009686146566573536,
+            -0.04088582189725495,
+            -0.0065344623594924085,
+            0.019617295999325712,
+            -0.004722087402575777,
+            -0.002871844114437583,
+            0.003935087265300519,
+            0.03229890413393201,
+            -0.02813577047904649,
+            -0.05124172132597234,
+            0.03325235407841968,
+            -0.008687655557217192,
+            -0.011027174838246948,
+            -0.003653553063547375,
+            -0.02350737945182988,
+            0.016223087939183468,
+            0.03789252356462714,
+            0.0034350623378603083,
+            0.00195158657888739,
+            0.011789712862232108,
+            -0.0004435439660769551,
+            -0.0037330456918125894,
+            0.0031552713243866858,
+            0.04465432992298031,
+            -0.0032465421571915712,
+            -0.0075366743861922745,
+            -0.059958752731410256,
+            0.016784616861008125,
+            -0.028734506065772967,
+            -0.03035575736273943,
+            0.02680390892611741,
+            0.02338368719038086,
+            -0.0030658375437784277,
+            -0.03754896050794851,
+            0.07005659523248504,
+            0.0020711575553570695,
+            0.013034425083076923,
+            -0.02316832262132332,
+            0.021332142030732894,
+            -0.0318756233020585,
+            0.02270782509507388,
+            0.01570810309454529,
+            0.004788124787612736,
+            0.026727250200142456,
+            -0.022380142778058276,
+            -0.008593910379436904,
+            0.014439754382387034,
+            0.015007697617383174,
+            0.07662531963612977,
+            0.011195998137723795,
+            -0.010809283538470098,
+            -0.042227822120219305,
+            0.003824320933275175,
+            -0.03797629427607094,
+            -0.02347390684453348,
+            -0.0526876440280276,
+            -0.017799861680287913,
+            0.01701482796266917,
+            0.002730395144917155,
+            0.03789451759899825,
+            0.038338545350313946,
+            0.028463789278406586,
+            0.015494583705011921,
+            -0.05700237266350479,
+            0.047895170359075166,
+            -0.0064406894534929406,
+            -0.0007954721295065988,
+            0.012543029839729358,
+            -0.004703334868741216,
+            -0.0060569679284068,
+            0.012324901254711192,
+            -0.019883935809784068,
+            -0.04253565942154215,
+            0.02164409276211115,
+            -0.0075367555415808635,
+            0.03617159147722157,
+            0.07567745988686005,
+            -0.04225977020036783,
+            -0.022548837589622003,
+            0.029811602145850144,
+            -0.02175792196819533,
+            0.02069861458115363,
+            -0.00941331829664397,
+            -0.035479639885943264,
+            0.020352552580119105,
+            0.003678588588550461,
+            0.06776105274336729,
+            -0.02346047721661687,
+            0.02721075076349443,
+            -0.029386781707644253,
+            0.023530831921117993,
+            -0.02077660533100372,
+            0.020022475323372906,
+            0.024195118676513715,
+            -0.028378518720494666,
+            -0.012900969023913307,
+            0.019264089970719624,
+            -0.0374983624629265,
+            -0.019726171557384355,
+            0.00979636085642416,
+            -0.014479528435832063,
+            -0.014533384583777662,
+            0.005904623758767224,
+            0.010344475500263744,
+            0.05659646358026115,
+            0.04163866113482093,
+            0.007152494819169405,
+            -0.07584500130974763,
+            0.05471858597878099,
+            0.03580211013427615,
+            0.044161429444475306,
+            0.02258180566046248,
+            0.00980779221334715,
+            0.0011193505603397395,
+            0.018086566768892507,
+            0.015605066702191871,
+            -0.00954727399396313,
+            -0.027950079146919375,
+            0.03028514025485407,
+            -0.006351669761257431,
+            0.005095804801686436,
+            0.009620040085743162,
+            0.019748001227987134,
+            0.033709535059510916,
+            0.008203559956940196,
+            0.007389752602218563,
+            0.04197771878351742,
+            -0.02067363758471765,
+            0.007543739188471913,
+            0.011207941261799052,
+            0.013964827251172813,
+            0.03839562075248879,
+            0.0009775244543292637,
+            0.006469507611492216,
+            -0.022808235395746654,
+            -0.024795004398633292,
+            0.006249024864814684,
+            0.011292877109066925,
+            0.01596839619826819,
+            -0.0014105396434174883,
+            -0.04152075438277094,
+            0.008267788411079337,
+            -0.028834191212891387,
+            0.030940526405027906,
+            -0.0018271056434014425,
+            0.006185475047820603,
+            -0.007920574674727123,
+            0.012907560753043616,
+            -0.015165704767187252,
+            -0.021962244496450727,
+            0.0037071053072833147,
+            -0.013508376064336042,
+            -0.015178823540513864,
+            -0.02518557053061311,
+            0.011693256916760944,
+            -0.01716015674592331,
+            0.02048897513447166,
+            -0.003412078966361745,
+            -0.001807298511378529,
+            -0.0012575655363779904,
+            0.03116103961636643,
+            0.011668533361641842,
+            -0.011637010548030312,
+            -0.040260195687925236,
+            -0.011129376691522074,
+            -0.0034493180842871908,
+            -0.02098244789711689,
+            0.026658743116016938,
+            -0.01923736325953796,
+            -0.009723937181300192,
+            -0.013134679989565051,
+            -0.011974548914818263,
+            -0.004028712301052159,
+            0.019628329175867303,
+            -0.028192034966932684,
+            -0.012638520793012082,
+            -0.011568652932013111,
+            -0.0025919910519445737,
+            0.012497908463405111,
+            0.015113186712897145,
+            0.014880607940212641,
+            -0.03059227859452334,
+            -0.021899716217105084,
+            -0.0008513506358489523,
+            0.011985931118464679,
+            -0.01321584595585435,
+            -0.011735288366213981,
+            0.010710683541435907,
+            0.01961722172525993,
+            -0.023571946726192585,
+            -0.03364281880125354,
+            0.0014460076743756465,
+            0.016563167828467187,
+            -0.02891589059260944,
+            0.04595166331401113,
+            0.034271553243501465,
+            -0.027557971577291153,
+            -0.007138821520412525,
+            -0.039508407807651844,
+            0.06814329636830853,
+            0.00022136123564675344,
+            -0.039718016705753735,
+            0.0503457594963753,
+            -0.003580581891576583,
+            0.013820796398521455,
+            0.02054050419088757,
+            0.035682661890720495,
+            -0.0013220739026852943,
+            -0.0052759394697328675,
+            0.04733018719162478,
+            -0.0187845855131196,
+            -0.03982699468608443,
+            -0.023905460465070062,
+            0.002631233133039351,
+            -0.03090995554989513,
+            -0.012617612051488022,
+            -0.013605787969473513,
+            0.039672783793764745,
+            0.008669582242097126,
+            0.009974277556736362,
+            0.005535636914843583,
+            0.014180668493093584,
+            0.012042659947850441,
+            0.011870115226306776,
+            0.00451259454348581,
+            -0.014023048592193627,
+            -0.007769033996392623,
+            -0.05715987527668358,
+            0.003374564518384728,
+            -0.028801158523506985,
+            -0.00756331369170698,
+            -0.02163054894055572,
+            0.005367181573233089,
+            0.02308705505078254,
+            -0.005438909398653553,
+            -0.00873620909426313,
+            -0.0036530652237719077,
+            0.013920407174296543,
+            -0.01539613379172909,
+            -0.03908211213269472,
+            -0.016810726548844992,
+            0.007460173854386443,
+            -0.007069189426742819,
+            -0.041197222498539215,
+            -0.003843233633335066,
+            0.004373367480223065,
+            0.016194121660459908,
+            -0.029165150066068787,
+            -0.00735139702658342,
+            0.017257895082337448,
+            0.014153313280536095,
+            -0.002376178042741926,
+            -0.023223700836427403,
+            -0.0319877537205969,
+            0.02088060774052704,
+            0.0012829290529515294,
+            0.03251550040316258,
+            -0.04128453069131847,
+            -0.023376114412063578,
+            0.04083022735076889,
+            0.005690491480259003,
+            -0.05810403800689653,
+            0.016582434816685452,
+            0.010402038964417569,
+            0.016795644054313566,
+            -0.00908590043098552,
+            0.05003656634510042,
+            -0.037216197954986735,
+            -0.013027960261454162,
+            -0.007090943677204207,
+            -0.05175371787508879,
+            -0.0020065660448063322,
+            -0.014908489552053589,
+            0.010627472758306387,
+            -0.03275473161158777,
+            -0.017334608434133866,
+            0.025321047439048318,
+            -0.0033515195870701753,
+            0.011526068468647207,
+            -0.026783879816504975,
+            -0.0012830935419413407,
+            -0.017506908439998053,
+            -0.02114853058440477,
+            0.004878884893669383,
+            -0.020936900837115968,
+            -0.0025587027781733642,
+            0.01557850246191957,
+            0.023652335794299834,
+            0.011489428313303988,
+            0.0019089137782552768,
+            0.02054907941022243,
+            0.014361232983780595,
+            -0.0056413109308506085,
+            -0.0010300050480036684,
+            -0.03413262505887213,
+            -0.018821160485010483,
+            0.03361989907065425,
+            0.013125889701529858,
+            -0.003742167496379789,
+            0.008813100942466033,
+            -0.0383265293638935,
+            -0.035316957263713485,
+            0.03020499323039515,
+            -0.05001623971243151,
+            -0.00855479667673597,
+            -0.005939674940109814,
+            -0.026451831943555135,
+            -0.008032873049527808,
+            0.0009431571685654594,
+            -0.014387605128858954,
+            0.016382559541755014,
+            -0.024909510232038502,
+            0.003715736505444385,
+            -0.006016712902409688,
+            -0.009110197692342108,
+            0.05507263160354303,
+            -0.022103031032598353,
+            -0.0475415595313825,
+            0.021958434235402655,
+            -0.016670809659931243,
+            0.0323519363801999,
+            0.03867494951405658,
+            0.011124508964363016,
+            -0.008002445714431074,
+            0.034041838454818577,
+            0.018400123235267234,
+            -0.01161754789471199,
+            -0.025901844576206774,
+            0.021800298132870245,
+            0.00943488425697727,
+            -0.05906691123008531,
+            -0.00782549274387971,
+            -0.019357446956797447,
+            0.008682376344965976,
+            0.02863077694455766,
+            0.0024925977373894447,
+            -0.002682170968419412,
+            -0.013590235283325244,
+            -0.021511149507762284,
+            -0.03255453916518719,
+            0.009039307015853905,
+            0.008305286476500875,
+            0.009312625027668458,
+            -0.0014144165766983945,
+            -0.007853869334376857,
+            0.009067065760942163,
+            0.022485201611182105,
+            0.03966189080765722,
+            -0.041627289519629344,
+            0.002707086456239785,
+            -0.022678178050044222,
+            -0.03653753142622053,
+            0.0036574646778835588,
+            0.029996160786576367,
+            0.02901163765559891,
+            -0.01971777193183787,
+            -0.0036589934841109884,
+            0.009181234938554578,
+            -0.025979483002227535,
+            -0.00773645103995767,
+            0.004280770859737866,
+            0.04116668301017644,
+            0.006201085381597264,
+            0.01402728258112097,
+            0.001060407317110527,
+            -0.00920801784000087,
+            0.02109880889040528,
+            0.025211022083982727,
+            -0.029866678264942617,
+            0.014439702646820226,
+            0.009379206296908577,
+            0.04156435812161122,
+            -0.037309655469525846,
+            -0.011819731348299495,
+            -0.02766498020409985,
+            -0.010454143230794476,
+            0.027911532296566614,
+            -0.04127890742215716,
+            -0.006948536183086887,
+            0.010207784113909938,
+            -0.0016584043356136647,
+            0.016437495428919636,
+            -0.014607861750686952,
+            -0.03222311919805017,
+            -0.000610723203118213,
+            -0.02867445198803213,
+            -0.024772816792038133,
+            -0.016510309753766612,
+            -0.018581201775667577,
+            -0.07450654873867177,
+            0.00026832248829154255,
+            -0.06724696990816072,
+            -0.02447002541264633,
+            0.04885569249258513,
+            -0.010195604207134053,
+            0.03332905935704749,
+            -0.02815272112271314,
+            -0.005296340304911926,
+            -0.05908835042526019,
+            0.0038476172058970774,
+            0.025753576863447866,
+            0.02386003857017594,
+            0.005206447371391696,
+            0.024799057871126032,
+            0.012060738268265606,
+            -0.04146503002980051,
+            -0.024949907556862643,
+            -0.014503475223770958,
+            -0.0016340466065744286,
+            -0.0034009167013795906,
+            -0.031298432995411005,
+            0.036241272149880994,
+            0.008902644391348814,
+            0.03550223922667804,
+            0.04463301409716232,
+            -0.010385984413759852,
+            -0.03305004479734889,
+            -0.006952739338820577,
+            -0.0175981588092108,
+            0.042668533202572856,
+            -0.017112184620086308,
+            0.005149173014526686,
+            -0.03219589107997635,
+            0.024181808150698054,
+            0.0014339790422945406,
+            0.01254721190611376,
+            0.011099458649404119,
+            0.0027626836767364253,
+            -0.026883640446090158,
+            0.0024160770766463607,
+            0.023874514847956222,
+            -0.022558433617669373,
+            0.00917667715473093,
+            -0.006754091166465801,
+            -0.033686935718776144,
+            -0.01078389638118685,
+            -0.013595327486167114,
+            0.024712781935748277,
+            0.013168464807451798,
+            0.005875864604862434,
+            -0.02417814420558463,
+            -0.0023978800459714144,
+            0.028650463366555304,
+            0.020176980768301288,
+            -0.02118184201861616,
+            0.019082340279671355,
+            0.019734925740100277,
+            -0.04209645071102001,
+            -0.04217310001316092,
+            0.006553913719098956,
+            -0.003735719227468833,
+            0.0061793909220459,
+            -0.028285889794908054,
+            0.008181788076484304,
+            0.004432488437404309,
+            -0.01743905248466261,
+            0.041221269621866216,
+            -0.014892354419172107,
+            0.005087102208335201,
+            -0.022198383716186407,
+            -0.011936388003605221,
+            -0.00027487042024290816,
+            0.044599215801158845,
+            0.008068975171827777,
+            0.003496950921841073,
+            0.02238779276447619,
+            0.0054839808262270315,
+            0.04816723616939378,
+            -0.02685221260466647,
+            -0.0190356946337995,
+            0.016464136459560113,
+            -0.0176092780782677,
+            -0.012513274095432353,
+            -0.010695421445334002,
+            -0.004784348908457072,
+            -0.022203041334932165,
+            0.010788155380415583,
+            -0.018353897272137543,
+            -0.02277431189907027,
+            -0.003945465850796918,
+            0.016024639792926797,
+            -0.006248368166794934,
+            -0.0038769803892794875,
+            0.0016292556540434277,
+            0.02412864830338344,
+            0.02892955453819876,
+            -0.02408022470555968,
+            0.019190282372105023,
+            0.018941584034503536,
+            -0.006709516380602965,
+            -0.020956377933747532,
+            -0.01964885206488053,
+            0.01434623424964373,
+            -0.01352968586212152,
+            -0.013872556892707697,
+            0.004052798542613835,
+            0.009201147374299997,
+            -0.004086067985859617,
+            -0.04185829685685201,
+            -0.014668698123513235,
+            0.003179403563664022,
+            -0.0006415014901450319,
+            0.019360184344240947,
+            -0.0150817538567364,
+            -0.029771999278536083,
+            0.014221776097046751,
+            0.022944884879589678,
+            -0.04493425220030206,
+            0.02117577719533236,
+            -0.0030648751497261983,
+            -0.00016251533643103604,
+            -3.817296018601414e-05,
+            0.008727767382077138,
+            0.026914779443266063,
+            0.03456565094655899,
+            0.01974806772696219,
+            0.00450800007152441,
+            -0.008952599700759673,
+            0.025184190461155245,
+            0.009846044083064704,
+            -0.025168735183134738,
+            0.010983353565382029,
+            0.016740721743326522,
+            0.0070838930012654296,
+            0.009173879407277282,
+            -0.006078288608766152,
+            -0.018269347893097643,
+            0.013181494188066667,
+            -0.01435693747022231,
+            0.01678907949532146,
+            0.05154463552784974,
+            -0.00039632596424461964,
+            -0.02461439550772063,
+            0.008418147037164186,
+            -0.006544319842335593,
+            -0.010065129286193278,
+            0.02214628550460128,
+            0.0029515574751370417,
+            -0.030862152033847717,
+            0.007509556025223583,
+            0.00675485304374519,
+            0.01894936231559203,
+            -0.02355525295639599,
+            -0.01387885858544803,
+            -0.00431398020087544,
+            0.0021208056468763523,
+            0.01580766504149989,
+            -0.004739477121073512,
+            -0.013204823447960706,
+            0.02994098229517376,
+            0.025496728234662373,
+            -0.00026124946738213036,
+            0.015060064980768175,
+            0.022262345898533903,
+            0.009968408664790368,
+            0.007588431141088797,
+            -0.0083063413609779,
+            -0.031336272171937095,
+            0.0018957858383561667,
+            0.05800811417728381,
+            -0.014937085296865802,
+            0.04313875432235027,
+            0.02018476425923236,
+            0.015657644644987107,
+            0.017880263053253607,
+            -0.01662268082870562,
+            -0.008740053682875323,
+            -0.0017630394204179638,
+            -0.008294417990024647,
+            -0.027613464270562948,
+            -0.005953209556351086,
+            -0.02379961210881074,
+            -0.013878676554701543,
+            0.015418315245979704,
+            -0.014764728065382074,
+            0.0027420640870858457,
+            0.010395538988699663,
+            -0.006590633833835581,
+            -0.00841537209927409,
+            -0.003943772700797381,
+            0.03376338017637296,
+            0.011980354721458463,
+            0.027761929822019422,
+            -0.00011128539412582973,
+            0.009342430952107655,
+            -0.018419526564396063,
+            -0.006839752805318664,
+            0.03271190887864607,
+            0.006011520427944665,
+            0.004652835471939905,
+            0.01485920920301398,
+            0.014305626381398871,
+            0.038015677219487023,
+            -0.005102005401565164,
+            -0.018204460895989052,
+            0.02065495133597456,
+            -0.0049699013356374775,
+            0.019149036693562712,
+            0.008825981577388056,
+            -0.002660055848523568,
+            -0.004292379561953203,
+            0.03090270710860331,
+            -0.020243440437217503,
+            0.018735727616321387,
+            -0.02238236089688497,
+            -0.0276584413929402,
+            0.023028408906176946,
+            0.008320436733620496,
+            0.04047641118026783,
+            0.011463851066078041,
+            -0.02247143742166039,
+            -0.034588725413285185,
+            0.009827713930789184,
+            0.00337071450636392,
+            -0.024446280199879062,
+            0.017542777160283324,
+            0.00780836483490212,
+            0.008453886667283516,
+            0.016624116261219637,
+            -0.004175682824655864,
+            -0.0042784362933758835,
+            -0.001546656717896957,
+            -0.008819739495525477,
+            0.01406790007187259,
+            -0.015487658289541583,
+            -0.021457182907140035,
+            0.024687169583245905,
+            -0.03915911962859764,
+            0.01771290139419629,
+            -0.006510046907173026,
+            -0.00997450880725535,
+            -0.02059560682587115,
+            1.5939836949198004e-05,
+            -0.007159493752862143,
+            -0.0004782363814453761,
+            -0.017503181957042913,
+            0.016643971600145645,
+            0.024056442566869802,
+            -0.013209211443909658,
+            -0.00445521990927108,
+            0.0020313626821877526,
+            -0.029235692455820056,
+            0.006662209097255912,
+            0.016993027053110635,
+            0.00529940612845993,
+            0.0026104743851393716,
+            0.0001707934714983891,
+            -0.031289446611163674,
+            -0.005782208620284939,
+            0.03053516889069683,
+            0.05590704747438521,
+            0.021628254147289437,
+            0.02403314981042378,
+            0.02908606663381421,
+            0.02717206823892633,
+            -0.003198844932040613,
+            0.0016217508103442348,
+            -0.01620660606431983,
+            0.027098323143010743,
+            -0.011664168290676943,
+            0.04322175021589083,
+            0.008652323184522541,
+            -0.049493020736445056,
+            0.004537958157696794,
+            0.00943239454147692,
+            0.034918603888916544,
+            0.016678716235690318,
+            0.0005887414054803805,
+            -0.005894338664998101,
+            0.03279165663336323,
+            -0.02735726709032382,
+            -0.011962070182991011,
+            -0.00036014407560325284,
+            -0.019079003730607717,
+            -0.015938295169860884,
+            0.010357611943935459,
+            0.04278924678845588,
+            0.0009957483598272451,
+            -0.0071422790468850215,
+            0.01571201177743999,
+            -0.009445307674383,
+            0.02286338492075306,
+            0.007336440384050498,
+            0.0042809497342459904,
+            0.025581467690095977,
+            -0.0025007871391868105,
+            0.02300539823997411,
+            -0.000628898771317254,
+            0.004582863909134464,
+            -0.015960578501883316,
+            -0.00930649396114688,
+            0.027838455754803444,
+            -0.011112268986721,
+            -0.029253746504396393,
+            -0.008734785561831333,
+            0.00472579547270089,
+            -0.0003805825712986138,
+            0.0021854358149324227,
+            -0.01183885533661493,
+            -0.010840951201591629,
+            0.019965691788904057,
+            -0.004922943301424112,
+            0.003800048754664761,
+            0.012664049429916252,
+            -0.0030124219674414897,
+            -0.01910844996750822,
+            0.01889983984736845,
+            -0.003858283778519141,
+            -0.024168821338992813,
+            0.020083955711766945,
+            0.01850497889545421,
+            0.045156567155119454,
+            -0.01586824055785609,
+            0.004612757419198795,
+            0.017497774574602505,
+            -0.002172088240279357,
+            -0.019617327081566827,
+            -0.014944795772093997,
+            -0.018212462639041492,
+            -0.015975283672064702,
+            0.012837956594691596,
+            -0.01243789877725244,
+            0.019500571353639314,
+            -0.0380735226388175,
+            0.012590900534826689,
+            0.004832168601136412,
+            0.007505385730121269,
+            0.016011983211212992,
+            0.025994560026860358,
+            0.002157436199771672,
+            -0.033465476168065784,
+            -0.01014599891204166,
+            0.007845373376916211,
+            0.010093432683262838,
+            -0.002849185178237036,
+            -0.018147004183434554,
+            0.003633790611809425,
+            -0.0266147420637404,
+            -0.018644422659309035,
+            0.022926737299761104,
+            -0.04798772015599113,
+            0.017908415301694906,
+            0.024859125158586466,
+            -0.04343699525707025,
+            -0.04886699607556352,
+            -0.011459188432595229,
+            -0.036774381316312464,
+            0.0014128005215813578,
+            -0.0005256067584181517,
+            0.011194992852264102,
+            -0.009342140079829378,
+            -0.018598671981710183,
+            0.01584789414301952,
+            -0.0036412641242200418,
+            -0.004225840239935549,
+            -0.00616696736018045,
+            -0.0025598631463178384,
+            -0.010188803157718543,
+            -0.03244694662411977,
+            -0.03290891136780909,
+            -0.00033742479296272127,
+            -0.009743927051414455,
+            -0.02334288359945272,
+            -0.03447062782872289,
+            -0.008728968107967028,
+            -0.037753659749578956,
+            -0.00515561961442376,
+            0.04070890150714597,
+            0.027517147608335957,
+            0.009449414978133334,
+            0.008086900089929912,
+            -0.013989034604490105,
+            0.05691047488419298,
+            -0.020425883381389086,
+            0.039466620642512155,
+            0.005301493569940605,
+            -0.0037222353899918703,
+            0.013075425214878277,
+            0.027710826422351564,
+            -0.006253665106995653,
+            -0.0017412928073238833,
+            -0.01928096630475438,
+            -0.012920476859996416,
+            0.009444408011467955,
+            0.015799884767184885,
+            -0.0025083061619197524,
+            0.001274457251739847,
+            0.00930533161700852,
+            -0.0006652478540264343,
+            0.017258038491335247,
+            0.03996268817486439,
+            0.003833844612473671,
+            -0.00826404141706295,
+            -0.018822002570963614,
+            -0.003188844900317347,
+            -0.011807006674248829,
+            0.010482058734260632,
+            -0.015410635475145028,
+            -0.011540803927522013,
+            -0.01249428777269578,
+            -0.017081856786754605,
+            -0.002183034976477535,
+            0.003114834030705039,
+            0.0027475411722881747,
+            0.02261627635192778,
+            -0.015872027027080824,
+            -0.022487581027490516,
+            -0.0035823117885461506,
+            -0.02398778449664045,
+            8.13716819067491e-05,
+            -0.01050060782407259,
+            -0.015616181283381218,
+            0.013676480044963827,
+            -0.021513173575512936,
+            -0.01091399089076612,
+            -0.008699176177861256,
+            -0.0006986953850759474,
+            0.005853723710671249,
+            -0.00993355748621621,
+            -0.00010241843582159751,
+            0.01150872694850241,
+            0.005559552266827916,
+            -0.012336449703762907,
+            0.0039431718621504,
+            -0.012740550272925562,
+            -0.004432059174951158,
+            -0.03592309122711106,
+            -0.019665686718735042,
+            -0.01846795637714379,
+            0.005642094448288269,
+            -0.005108964028543505,
+            -0.020586678054151854,
+            0.00668364061048941,
+            0.0034775127050453774,
+            0.004713169526661792,
+            -0.010723667875209928,
+            0.013944735241748791,
+            0.010882420168586074,
+            -0.003518245916227596,
+            0.013827334494230662,
+            -0.001330399936559009,
+            -0.020286697504699764,
+            -0.011139922402071095,
+            0.03538258948179278,
+            -0.024961826464903997,
+            -0.0018019328351763667,
+            -0.0026423695690168066,
+            0.00597003516870726,
+            -0.03071835946501977,
+            -0.013634212232744325,
+            0.02164400349828422,
+            -0.005846552894831441,
+            -0.026367014426434226,
+            0.03399116480234033,
+            0.03473258231629753,
+            0.010041605592929843,
+            -0.013423384340589794,
+            0.04730054254696893,
+            -0.012968978954938308,
+            -0.018274800841400665,
+            0.026826644936918647,
+            0.0278389611094493,
+            0.02525164095926168,
+            -0.029782550447890466,
+            -0.003010713086487275,
+            -0.02393629084357885,
+            0.017993366287762866,
+            -0.0263476663180971,
+            0.01864011499179424,
+            -0.02332263152537849,
+            0.016014908881756034,
+            0.01868464825773604,
+            -0.003104041044182249,
+            0.026313491086684425,
+            -0.0052945248919367815,
+            0.015253731232325074,
+            0.0012371308759955573,
+            -0.02119765517013431,
+            0.007795466385020642,
+            -0.006469500272298062,
+            0.025762169721880456,
+            0.00833992768202466,
+            0.01801901639978976,
+            0.022019940825351116,
+            -0.0409522298213741,
+            0.02003698451081499,
+            -0.001367916864679543,
+            0.01782984895971748,
+            0.0038516134262842373,
+            -0.007885432294782304,
+            0.014221728513400065,
+            0.010991352188016265,
+            -0.012851099261924757,
+            0.003840539873231741,
+            0.0008923482291826014,
+            -0.028928104542716937,
+            0.007066004768771198,
+            -0.026914574065060926,
+            0.025866907400807087,
+            -0.005526774784152086,
+            -0.00998122210956923,
+            -0.020744532673097317,
+            -0.0071425207262770865,
+            0.003754468896273156,
+            -0.012747874997429746,
+            -0.011385805601999295,
+            -0.024700457669046855,
+            -0.02401291382882952,
+            0.029591582802488962,
+            -0.014054332096344003,
+            -0.04015978298687865,
+            0.001234440863385445,
+            -0.023301255402129767,
+            0.01193125007773239,
+            -0.005793124129267758,
+            -0.024440100649464773,
+            -0.042563399082257154,
+            -0.017548888269389457,
+            -0.0009394887961485922,
+            0.022739763644266633,
+            -0.00114156949394707,
+            0.041791395064755515,
+            0.0008816696205437911,
+            0.022207132857533028,
+            -0.015547580543606755,
+            0.005319731264840819,
+            -0.02071229859314312,
+            -0.041495579742912285,
+            0.03660432378985971,
+            -0.014637031447181937,
+            0.025854844497435178,
+            0.03324325296704186,
+            -0.013009347371118843,
+            0.001354504885302625,
+            0.02652570709474851,
+            -0.014618100923440427,
+            -0.010782232747863238,
+            -0.0034936699515378212,
+            0.003461670978569965,
+            -0.006468116285021298,
+            -0.027793221545159466,
+            -0.007752458229193391,
+            0.04193706715422074,
+            0.017894069367616637,
+            0.021928994684423043,
+            0.022606401934291136,
+            -0.017972404868085358,
+            -0.026402048654300563,
+            0.003656646576244765,
+            -0.013129722990153284,
+            -0.018090360146368677,
+            -0.01624598856366761,
+            0.012397150789705746,
+            0.011741002980073786,
+            -0.004052422390675261,
+            -0.01836750423615859,
+            -0.020929335029248472,
+            -0.016327412417122416,
+            -0.0042615345226698015,
+            -0.008458117095535,
+            -0.005002582649405199,
+            -0.012158173174049797,
+            -0.020409616620022143,
+            0.03706368540614453,
+            0.0285565642978937,
+            -0.0041732933884846566,
+            -0.00607154864704767,
+            -0.011142150176334957,
+            0.008834431806990914,
+            0.006021232046087652,
+            -0.005028870471297621,
+            -2.3396043292139426e-05,
+            0.005112033372713154,
+            0.022822312435625314,
+            0.007271235299348001,
+            0.0035791789459533856,
+            -0.06340305811365254,
+            -0.005392000517722054,
+            0.022560201096078862,
+            0.012581401789917604,
+            -0.0032248219865223496,
+            0.025180718735801837,
+            0.0006451724964970927,
+            -0.009714514747463593,
+            -0.018389422219956937,
+            0.024728862150150004,
+            0.008760329201960964,
+            0.020988700992999768,
+            -0.014974763313686516,
+            0.003845297486923296,
+            0.0006445983847138896,
+            0.00750394530065325,
+            -0.0019792992847026793,
+            -0.008578254004439607,
+            0.025776677078006516,
+            0.016593438298233645,
+            -0.005695578040595484,
+            0.01604018433477423,
+            -0.0017497260488899657,
+            -0.021460543443428924,
+            -0.010265213095609554,
+            0.00042017209251926397,
+            -0.010570864288876387,
+            -0.018015790888295944,
+            0.017569414447505878,
+            -0.002601036282535863,
+            0.0072399377790164435,
+            0.030281707151249355,
+            0.0423183067569767,
+            0.009566493398534713,
+            -0.005928603293907002,
+            -0.0037936911251953654,
+            0.015477873304924939,
+            0.01513127995244932,
+            0.010741256034652754,
+            0.03015858543221727,
+            -0.029247492875021843,
+            -0.006690306960497428,
+            0.0046653727410352576,
+            0.004610405633429778,
+            -0.011692179099382171,
+            0.030055897497741527,
+            0.04569730592994177,
+            0.006736813244286893,
+            -0.004799978513117371,
+            0.002121067028052434,
+            0.011299889107780231,
+            0.007768137218140627,
+            -0.018927080714961783,
+            -0.0012222962555201746,
+            -0.006009911026251445,
+            -0.014001766533604134,
+            0.026941585318261085,
+            -0.003960837039351648,
+            -0.011848232223811548,
+            0.02903357274500332,
+            0.1105267435019781,
+            -0.006011029338781496,
+            -0.0053646998281195206,
+            -0.03242321704621517,
+            -0.015194102811332289,
+            -0.008860590919147765,
+            0.0056988205877963076,
+            0.01635957585050017,
+            0.020327661913559888,
+            -0.0008943408274661007,
+            0.016820860249955754,
+            0.02822175614376368,
+            -0.013854827839129885,
+            0.007906244895327178,
+            -0.030891381690403466,
+            0.015702117924362928,
+            -0.005383974926093977,
+            -0.012032955400313954,
+            -0.0026169511828034054,
+            -0.008618307035330928,
+            -0.0037356680273096105,
+            0.0009666430093455632,
+            0.011296307527684261,
+            0.004427516814276546,
+            -0.014484198313369681,
+            0.002829529156309784,
+            -0.008595300985863424,
+            -0.005122996529848575,
+            -0.02393571073523279,
+            -0.024119245543788293,
+            0.005223147047205755,
+            0.0019389305548629142,
+            6.344308292376579e-05,
+            0.006848965054539454,
+            -0.003810712095343454,
+            0.030149081317333418,
+            -0.01700322008272963,
+            0.0012288912810468446,
+            0.01193226302726573,
+            0.012332749933413344,
+            0.01688602002694633,
+            0.0028600984823761366,
+            -1.9882692552530084e-06,
+            0.021172977047420227,
+            -0.010514036509402561,
+            0.01460334896316005,
+            0.013091417303145243,
+            -0.04103572381458182,
+            0.0089746401463863,
+            -0.0027841289484449873,
+            -0.024252447218794344,
+            0.004446472209246425,
+            -0.018987867343500102,
+            -0.00881974688349601,
+            0.0005365625696166774,
+            0.012054719659248512,
+            0.01960300421041832,
+            -0.008553943699846696,
+            0.02111201386480852,
+            -0.0003501657740808554,
+            0.004177088753111459,
+            -0.012879944670340026,
+            0.0065222126975051685,
+            -0.007563354540402343,
+            0.0008168128607271199,
+            -0.018213241937597355,
+            0.012526561612175514,
+            -0.0008285725179894544,
+            -0.015737003159436623,
+            -0.011337688275309921,
+            0.004737346137976519,
+            -0.0027993545140114705,
+            0.0030843163114700313,
+            0.010153740340626154,
+            -0.008815302484351509,
+            0.004919275910399053,
+            -0.0017541042538165297,
+            0.002562816687714991,
+            0.014285249186421434,
+            -0.029816964593980453,
+            0.005357227738099325,
+            0.009154842495811638,
+            -0.00048782303639333943,
+            -0.013074976152495576,
+            -0.004054505559072866,
+            0.0028322102449485015,
+            -0.016877328079971877,
+            -0.012128199639779175,
+            0.023421330164419888,
+            -0.012565145848820333,
+            0.008609070188894315,
+            -0.010936598722317274,
+            -0.018401801734378692,
+            -0.01066119413133089,
+            -0.005254229780560808,
+            0.01234969508270852,
+            -0.0038427767204741877,
+            -0.006947594635952202,
+            -0.01756894543880097,
+            -0.007890897373987378,
+            -0.047246698568222466,
+            0.009617876521097624,
+            0.027157555263862723,
+            -0.011355758309667893,
+            -0.04050311381480088,
+            -0.01017774338301804,
+            0.02437257208967103,
+            -0.004056329295095018,
+            -0.008345450168215274,
+            -0.001599171409135998,
+            0.018358386463243014,
+            0.0027207445181251406,
+            -0.007292967749349572,
+            -0.011267861428599221,
+            0.01657507035043879,
+            0.015105061280816214,
+            0.021419854579631276,
+            -0.02088952706825817,
+            0.006295984081511286,
+            0.020973020776936996,
+            -0.03073246543555915,
+            0.0023720435876790876,
+            0.03363966216709148,
+            0.010412588613998134,
+            0.013023311559810034,
+            -0.0010217662933572685,
+            -0.0075567381078977215,
+            -0.008248818794295818,
+            0.011848200323549865,
+            0.005066463524567732,
+            -0.023174489297821385,
+            0.021823825355858556,
+            0.020368868756448753,
+            0.0026042856858509524,
+            0.003457067317891926,
+            0.0005539745115410357,
+            0.01062670844108589,
+            0.009948889803890108,
+            0.05100741556350918,
+            -0.006493714124814164,
+            -0.014021751055930108,
+            0.015063988102064609,
+            0.02488059237697814,
+            -0.0023499941119596228,
+            0.01087282522970853,
+            -0.02612218935931224,
+            -0.021423880914461062,
+            0.023389294741183817,
+            0.008967763132667892,
+            0.009637806238448432,
+            -0.010815706635624337,
+            -0.011390999462441815,
+            0.031750675158698435,
+            -0.01307829462211914,
+            0.00759749911213318,
+            -0.02169895957209634,
+            0.01387879409130081,
+            -0.008606362268863441,
+            0.032229658009209815,
+            -0.0009066257436946125,
+            0.011062436424260079,
+            0.028010486866468023,
+            -0.007835805879241447,
+            -0.019657486763625547,
+            0.003967480939987674,
+            -0.015577952617624695,
+            0.02188429826317828,
+            -0.020502758155233298,
+            -0.0002362500702658591,
+            0.0040935406015014265,
+            -0.010056462958039452,
+            -0.0057038083589231615,
+            0.011475170331848666,
+            0.0015345076181482267,
+            0.006359262977587827,
+            -0.020293014401106,
+            0.01369706440475074,
+            -0.003989482215141903,
+            0.007887824197823247,
+            -0.015925992045349828,
+            0.014370223626942766,
+            -0.02309139937510304,
+            -0.021808736033141787,
+            0.008142621179624918,
+            0.0108996933426044,
+            -0.007589050511458233,
+            -0.025971539510030145,
+            0.02250992080022228,
+            -0.024980833805349545,
+            0.024979008145354243,
+            -0.006622339940663538,
+            0.002566378488088469,
+            0.004913874740363343,
+            -0.008102103508732738,
+            -0.0028098905427027783,
+            0.0057865587191235565,
+            -0.0073509382400023995,
+            -0.014123756810537343,
+            -0.019453380788786282,
+            -0.04083629135586275,
+            -0.03177927047231389,
+            0.022927581786088044,
+            0.011557232599801421,
+            0.0017073832658476193,
+            0.02339552660545296,
+            -0.02247471449131386,
+            -0.017890866287500053,
+            -0.009775068417716798,
+            0.025184899522229628,
+            -0.04408541669037361,
+            0.012104255392615549,
+            -0.014669458760022245,
+            -0.0245258242481677,
+            0.006274252098230249,
+            -0.004161960147481513,
+            0.009299171724026455,
+            -0.010779421730862708,
+            -0.04713627671791054,
+            -0.005192684784377356,
+            0.02330975631183273,
+            -0.0036802876364168935,
+            -0.006197548344287773,
+            0.03048284650641829,
+            -0.0019503797069087845,
+            -0.005038254331399784,
+            -0.005524875459245342,
+            0.027783596017224226,
+            -0.024950083604372984,
+            0.010988398531346805,
+            0.008304937912856357,
+            -0.008485065895193593,
+            0.006572665697403556,
+            0.0011518846966102158,
+            0.011927348472282199,
+            0.015632588644277183,
+            -0.015559413716826264,
+            0.02262990861928957,
+            0.011446850286896198,
+            0.006506075777765303,
+            0.008015225053785824,
+            0.00010054430757843335,
+            0.0022975742216194894,
+            -0.024948925449668525,
+            0.026047924967851216,
+            0.01888211557921096,
+            0.0001632181017876392,
+            0.012553094476616919,
+            -0.015042621225242777,
+            -0.02782009986290425,
+            0.02606646297992026,
+            -0.016847653820092335,
+            -0.00045272379465939274,
+            0.020053091486344633,
+            -0.016210036358455773,
+            -0.007930550661303182,
+            -0.01123514849773852,
+            -0.0162675150307978,
+            0.01803076559921804,
+            -0.012160217543572069,
+            -0.02216594536057653,
+            0.002594675636753233,
+            0.016443123828746582,
+            0.02134480656850119,
+            0.026694575953313482,
+            -0.011470930697351533,
+            0.017768927048191052,
+            0.0008653408391607816,
+            0.001865765599795092,
+            0.025745954171649543,
+            0.01702677011968492,
+            -0.008474889109168202,
+            0.025347176686722313,
+            -0.0016686223698600755,
+            -0.002421921060670558,
+            -0.012265504084322075,
+            -0.0165214436458303,
+            -0.007545471138114208,
+            -0.015333235556146196,
+            -0.0051830106590765685,
+            0.000762746782664525,
+            -0.0018118022696726346,
+            0.10876576838038368,
+            0.0034948044540582626,
+            -0.011608460689239234,
+            -0.005870263500417206,
+            0.007510774308250375,
+            0.0023525201199670765,
+            -0.006846154764389575,
+            0.016082721103476375,
+            -0.018616626204086575,
+            -0.023593744079394,
+            0.004743481974389998,
+            -0.0009262286319469741,
+            0.00870122878944883,
+            0.02029523251993269,
+            -0.008608837818169835,
+            -0.0200567711498164,
+            0.007089477722191928,
+            0.005535149644041221,
+            0.0023640746516071774,
+            0.005256431824732927,
+            -0.0006118671321824933,
+            0.008209920240361308,
+            0.02448422122961942,
+            -0.015622892878860465,
+            -0.029770836607612903,
+            0.010535371601298532,
+            -0.00015994156470228762,
+            -0.007193425789067776,
+            -0.011046316463797128,
+            0.014728841492009627,
+            -0.0011488451990980268,
+            -0.02751404133172324,
+            0.017663132712536528,
+            -0.019055335178900415,
+            0.014643246200933506,
+            0.0025348345163347464,
+            -0.00016387173410118788,
+            0.022565953069005714,
+            0.013101158251834834,
+            -0.04272085076882881,
+            -0.0034442020309914823,
+            0.01041205537746844,
+            0.011059422927946552,
+            0.0116446016862942,
+            0.0012992535381387887,
+            -0.018670801241345514,
+            -0.011788984490283379,
+            -0.009109849103208542,
+            -0.021268488414296117,
+            -0.012523033785938503,
+            -0.011393853408628214,
+            0.010743646386439633,
+            0.0012964030814325153,
+            0.023809566099779113,
+            -0.003030535542277875,
+            -0.006526351867141901,
+            -0.000430699447948508,
+            0.00880561288237257,
+            0.009694646009259077,
+            -0.02079011138439714,
+            0.008294338719512953,
+            -0.01823922789220214,
+            0.011819769685519291,
+            0.006416044253693268,
+            -0.022799568832306067,
+            -0.009911619276012527,
+            0.007152786695893951,
+            0.008393677195358034,
+            -0.013569953300860946,
+            -0.006503988731872938,
+            0.02299498754558268,
+            0.018870619152710703,
+            0.012233964644738905,
+            0.024994616203196145,
+            0.03838969722235787,
+            0.005744255912479976,
+            0.027682123067526515,
+            -0.016791471092091604,
+            0.008459745383680297,
+            -0.022493868174622695,
+            0.012902262638928338,
+            -0.00460399519266689,
+            -0.0050431343566427886,
+            -0.018188192498411616,
+            0.033765105245412236,
+            -0.009159751849801773,
+            -0.0027150644587327763,
+            0.034778224828120714,
+            -0.0066502021793471075,
+            0.004525230911930458,
+            0.005096748531775903,
+            -0.016067049494073893,
+            0.033274672789154185,
+            0.01686831040872425,
+            0.016353966592947262,
+            -0.021631941883929817,
+            0.011748317746828848,
+            -0.005675701689448899,
+            0.0001273442591116645,
+            -0.020738819037390263,
+            -0.0037528323797385733,
+            0.0027725541343763104,
+            0.0059860509282538546,
+            -0.016556134518370437,
+            0.02033623476757896,
+            0.00019647023567641008,
+            0.0052895744489259875,
+            0.03253462247073863,
+            0.01043311189804924,
+            0.019182938878855655,
+            0.010051051068017807,
+            0.019523739618278902,
+            0.0281939391326836,
+            0.025685787482623124,
+            0.0029932688462254147,
+            -0.018668945726139946,
+            -0.0009326901784812825,
+            -0.007880352863625262,
+            0.010492063055849727,
+            0.022849902214818293,
+            -0.022619228461640748,
+            -0.02772644874135741,
+            0.018566981184526675,
+            -0.0022608185345048414,
+            0.020031103834122595,
+            0.007544464625538984,
+            -0.020935181884939796,
+            0.003407355515103286,
+            -0.023178230166337178,
+            -0.0005418155871620593,
+            -0.031039187720144967,
+            -0.028591174077243604,
+            -0.0007659241904525392,
+            0.011945344534674635,
+            -0.01688153017778311,
+            0.03357788163306365,
+            0.017667778671101954,
+            -0.014289109343082812,
+            0.01604684528817524,
+            -0.011054587325539086,
+            -0.0036959091138078683,
+            -0.005819853870004682,
+            -0.00203755010992417,
+            -0.02157944474979592,
+            0.005138614763345542,
+            -0.0033509138150859593,
+            -0.0024871060745482883,
+            -0.016936575573598154,
+            -0.021901830366872595,
+            0.016375173843894617,
+            0.016742770709641697,
+            -0.006606252542838758,
+            0.039771905404651496,
+            -0.004238071340064403,
+            0.00018364315349545657,
+            0.011690521707403059,
+            0.0010018392078563795,
+            -0.013651921068215563,
+            -0.010829994512513501,
+            0.012362647379788833,
+            0.011321403447882543,
+            -0.018942043512252663,
+            0.005763773608775982,
+            -0.009131565038297889,
+            0.002089216671850269,
+            0.005438887787329238,
+            -0.008183835275629478,
+            0.006690280329947951,
+            -0.01673229763262922,
+            0.015833397876364723,
+            0.00295474085300386,
+            0.003186554859191664,
+            -0.011797859525926772,
+            -0.020991927615680524,
+            0.01576413532776261,
+            0.0032403514518527287,
+            0.018819824905201635,
+            -0.026755218746060662,
+            -0.00012946467986739473,
+            -0.009552681500884924,
+            -0.0028941357661144826,
+            -0.012533800273548772,
+            0.013975640628890588,
+            0.016864739319953755,
+            0.0063941926578529,
+            0.01781016436903586,
+            -0.0026268224264609724,
+            0.015368284773773678,
+            -0.018122416427266218,
+            0.008686009180697955,
+            -0.006801074032721299,
+            -0.018537521789181048,
+            0.0013237338264105415,
+            0.027863741654435434,
+            -0.004670012662973029,
+            0.011066853115455966,
+            0.011704884649534624,
+            0.018954056680143452,
+            -0.0012626215734593707,
+            -0.0030850559460101866,
+            0.015610404665171367,
+            -0.005838226405658572,
+            -0.006861883833171456,
+            0.020872966233916948,
+            0.032951653763679145,
+            -0.006716215648910268,
+            -0.0021292599951407863,
+            0.010886596327481677,
+            -0.004901684487088035,
+            -0.008644584974417106,
+            -0.009587037829603014,
+            -0.03158137900717491,
+            0.024369089446706894,
+            0.012092412439477474,
+            -0.009601042951586109,
+            0.022690862146548786,
+            0.020752792552278052,
+            0.011412743968296708,
+            -0.010471860708195726,
+            0.02291536038589634,
+            -0.01268035142112569,
+            0.024168869865226136,
+            0.032482831775271157,
+            0.004400973556358811,
+            0.0022312906286559167,
+            0.0021175489661843985,
+            0.01691348460546639,
+            0.006050357346231155,
+            -0.007068133214675996,
+            0.011656003476409986,
+            -0.018487923180009552,
+            0.01598530891381001,
+            -0.004432366596808033,
+            0.02808371007118948,
+            -0.01337145216499331,
+            -0.010473700033368562,
+            -0.00537325376270318,
+            0.00010726431372328729,
+            0.006445701087575359,
+            -0.003324002570256215,
+            0.005255443653813165,
+            0.0007072577303252132,
+            -0.017825649146944237,
+            0.003050128130393781,
+            0.004825967170196209,
+            0.009018799597337101,
+            0.004103653497834134,
+            0.007249957401824204,
+            -0.012360355929483,
+            0.01441885664789707,
+            0.004401820363035808,
+            -0.0026190938717927086,
+            -0.0007163225760398784,
+            0.011124538295057815,
+            -0.013232822471919533,
+            -0.004895624967029031,
+            0.005281228035321008,
+            0.0059275753353475414,
+            -0.007507924998741734,
+            -0.004484140989405051,
+            0.011908076549392969,
+            -0.0037917291321648483,
+            0.006351543504726163,
+            -0.004409464890606014,
+            -0.0037760571637986827,
+            0.005303661313434895,
+            -0.00974511544836468,
+            -0.004204167898342997,
+            -0.0007093125445559502,
+            -0.006869250680233214,
+            0.00786202375969793,
+            0.005971033227117194,
+            0.010606085272785863,
+            0.006969437776542303,
+            0.004293129058563288,
+            0.004114452739123398,
+            0.004071017374828283,
+            -0.002445681623750424,
+            -0.00592480769363354,
+            -0.004617501328052899,
+            -0.012067458701076778,
+            -0.008335239153829813,
+            0.009025028998949398,
+            0.008653779421856954,
+            -0.019363018792788574,
+            0.02048011578221327,
+            -0.001188320960771659,
+            0.00772165044232104,
+            -0.01602086976290928,
+            -0.000907553574811279,
+            0.005464880629734941,
+            0.008267702703609528,
+            0.016136762227127184,
+            -0.03654209526492064,
+            0.005179791231187316,
+            -0.0037321883493524766,
+            0.016026013709955444,
+            -0.008215703411373182,
+            -0.006979816286354862,
+            0.02564060526656963,
+            -0.01450763388985563,
+            -0.0009847758228920171,
+            -0.016142304341196283,
+            -0.01407778890808498,
+            0.010815725252959506,
+            0.02050940280233426,
+            0.020684808835848383,
+            -0.0004965802897137407,
+            -0.0005781445948759235,
+            0.00036179841583004193,
+            0.022775804864342608,
+            0.015545681427566235,
+            0.018697154644607316,
+            -0.004543500782944068,
+            0.007028436664826706,
+            -0.023057215501463973,
+            0.009912581556496658,
+            0.0011795417129232596,
+            -0.008215027850908477,
+            -0.008554137076267468,
+            -0.008703378564520125,
+            -0.010448538943182258,
+            -0.011773811327190638,
+            0.004106616839960258,
+            0.013895013678116032,
+            0.004190549543250594,
+            0.0031571190600981074,
+            -0.0225385788570314,
+            -0.006243193914542516,
+            -0.011229641377928767,
+            -0.01674214043453308,
+            -0.007367389908398082,
+            -0.02386334969150212,
+            -0.028021339684039734,
+            0.0002794207370355175,
+            -0.02411861258809571,
+            -0.015958336406924612,
+            0.005268815864773021,
+            -0.01546610225139129,
+            0.00898600137695074,
+            -0.008396481690803911,
+            -0.011873486990732417,
+            0.0074743033679134105,
+            0.015336444788692,
+            -0.017676977079590463,
+            -0.010165317702640217,
+            -0.01690524990506033,
+            -0.013460451917691695,
+            0.010918550283871637,
+            0.00595960977280955,
+            0.009250157926438213,
+            -0.012447979337850305,
+            0.0062932435120863365,
+            -0.012673905639312696,
+            0.015073013552721701,
+            -0.03769166067890086,
+            -0.029992249463405965,
+            -0.00014990754540999537,
+            0.007288036630677436,
+            -0.0020735742042505426,
+            -0.018695120921334806,
+            0.0003860116505754868,
+            -0.003608995261360232,
+            -0.014900191335144973,
+            -0.013866216738266176,
+            0.025251475305524053,
+            -0.006788483107616133,
+            0.01223227424986397,
+            0.0034937277040015626,
+            0.03438333560343413,
+            0.008803852645562129,
+            0.014666206816302367,
+            -0.005709421955160623,
+            0.003742433643393657,
+            -0.0033706258426092744,
+            0.006906464031352315,
+            0.0076578888497945185,
+            0.009276592191710524,
+            0.008025321004305927,
+            0.004576470294889512,
+            0.002887346543823031,
+            -0.001925468214331082,
+            0.02125241545487307,
+            0.025671028960818562,
+            -0.00037467906648160044,
+            0.008981063192139038,
+            0.0009387900084090577,
+            0.0002735789978680719,
+            -0.008374290340790827,
+            0.006547013284767913,
+            -0.008274637814401893,
+            0.004041066530254948,
+            0.012897549078838394,
+            -0.009150863940242148,
+            -0.010211568689323908,
+            0.016575088843292566,
+            -0.012380099999574783,
+            -0.03420455578618604,
+            0.026550654680719987,
+            0.005324512466403649,
+            -0.0035670106106754245,
+            -0.010657928468872608,
+            0.0037176027853158167,
+            -0.01103891334887315,
+            0.008096915671982816,
+            0.022587659771132487,
+            -0.0013527917944322358,
+            -0.0018085171207746743,
+            0.0038503794877980586,
+            -0.011870484733648887,
+            -0.007881489342351292,
+            0.006462772034381968,
+            0.0315696514747912,
+            0.01326417653779577,
+            -0.020400369539140532,
+            -0.015223808971473513,
+            0.020406862349006738,
+            -0.023142176712111575,
+            -0.006254959806032159,
+            0.009039506093607052,
+            0.0009087520492922078,
+            -0.022766046838415165,
+            0.0009662037093312435,
+            0.004701966191595815,
+            0.005946458676746661,
+            -0.006358986495019584,
+            0.03237532341275415,
+            0.001075992257265679,
+            0.015892020271402903,
+            -0.017189453434945788,
+            0.0017514632185789299,
+            -0.014189278671121677,
+            -0.014265447388465805,
+            -0.029809460132405256,
+            0.0005557491671071693,
+            -0.0030324198417091704,
+            0.011038984948790744,
+            0.024808067315720335,
+            0.01145103533194245,
+            0.014432246996007246,
+            0.010552852947578788,
+            0.01513689173494211,
+            -0.013653722180565311,
+            -0.029009757850432732,
+            -0.005171522291984226,
+            -0.018001048743500243,
+            -0.02297482628212251,
+            0.003962613844063587,
+            0.013645125285548414,
+            0.006278662690346082,
+            0.023786381386575035,
+            0.0021738516691483163,
+            0.0009580591750940011,
+            -2.690805459518147e-05,
+            0.008894091070322921,
+            -0.004952203441302471,
+            -0.006553986197076102,
+            0.003570189040872711,
+            -0.000652714137223501,
+            0.021228360881461853,
+            -0.005758946428212041,
+            7.897153237591187e-05,
+            0.014242175492137011,
+            -0.005622256348934008,
+            -0.0035009192529791146,
+            -0.017339926143652377,
+            0.006248842389703505,
+            -0.01622011586976866,
+            0.025783086900871728,
+            0.018450783573486916,
+            -0.009109295293546267,
+            -0.013469521573468915,
+            -0.011259555201313116,
+            0.02773097452736661,
+            0.004631282554715908,
+            0.010787055220617197,
+            0.018228322680582464,
+            0.01461254259249465,
+            0.01784830193964943,
+            0.004664725683522323,
+            -0.0034339781146649384,
+            -0.0007692550982545399,
+            0.021176646860433974,
+            -0.014700178590249292,
+            -0.002701208084390198,
+            -0.008842788249773623,
+            0.0021799005046816444,
+            -0.008431299000066363,
+            0.006086447848362669,
+            -0.010793016137040172,
+            -0.015251381840165954,
+            0.0189503457652375,
+            -0.02660767353735015,
+            0.016423753600066336,
+            -0.00593016170680532,
+            -0.00482891500342298,
+            -0.027459214027897378,
+            0.010728039528784636,
+            0.0027866316191608297,
+            0.0022519974195000687,
+            0.010770472550461752,
+            -0.021427634123401408,
+            -0.01867717541189363,
+            0.008532444554010673,
+            -0.011457117350255678,
+            -0.007152301670244725,
+            -0.00348619742388509,
+            0.0154715607296319,
+            -0.00017898955187849443,
+            -0.009925365923862246,
+            0.02085093731060963,
+            -0.007175309371250024,
+            0.01223979370120031,
+            0.005256127464980655,
+            -0.008756077893734508,
+            -0.0028438320920811362,
+            0.021022708898898203,
+            0.027557382862284808,
+            -0.008342206702778956,
+            0.014526066580760486,
+            -0.011883797998224506,
+            -0.01088335352319397,
+            -0.03479444009382274,
+            0.030104838505937447,
+            -0.013305732257226255,
+            0.026773747582411633,
+            -0.0006968265979630097,
+            -0.017037729111584094,
+            -0.016206207453228583,
+            0.01625225876515654,
+            -0.0024290584023212254,
+            -0.009366287391027166,
+            0.01465885301870365,
+            -0.02428628495473643,
+            0.0058336571592899735,
+            -0.0006627935593670669,
+            -0.011531762838191602,
+            0.013275508677482622,
+            0.0019223252930710324,
+            0.010738622785461513,
+            -0.02030405805119001,
+            0.0009661595529028194,
+            0.009263716373456188,
+            -0.016791855691574455,
+            0.006057830565306032,
+            -0.03730952786847912,
+            0.0029341721588188943,
+            -0.014833895684348656,
+            0.005725615593914644,
+            0.001498755253963699,
+            -0.01820708352937343,
+            -0.03652692232407375,
+            0.00043665701245599497,
+            0.01913883585802903,
+            -0.007975988799790905,
+            0.020538595326175857,
+            0.008243569241768276,
+            0.008421404577931517,
+            0.002314738067443065,
+            0.0016702403734608475,
+            0.008279538988821465,
+            0.03679456861213399,
+            -0.01775927634148114,
+            -0.004836606118958108,
+            -0.0012151329859941935,
+            0.010644195708319258,
+            -0.02028470571116305,
+            -0.009287515726906219,
+            -0.0036506912311261536,
+            -0.013134467570201307,
+            0.030615862647585222,
+            -0.020265555441523482,
+            0.025634449009756734,
+            -0.009300770899023007,
+            0.009274072905190077,
+            -0.02097685009125915,
+            -0.009797008690754956,
+            -0.010661540100131183,
+            0.0019532291898233895,
+            0.013465005282849381,
+            -0.009151430285942593,
+            0.0071467934925125145,
+            -0.006655608635025676,
+            0.01020327561163374,
+            -0.001472597695707201,
+            -0.02443508800166349,
+            -0.0024998811551575563,
+            -0.01107321258077183,
+            0.02456164421154801,
+            -0.0019619696941303388,
+            0.0005043179699158694,
+            0.010951580163871948,
+            -0.00853593956432799,
+            0.023927336608555987,
+            -0.008855702280766362,
+            0.0010093679746665581,
+            -0.004863269000020324,
+            -0.004005691415117686,
+            0.001698212044518236,
+            0.01822024022650613,
+            -0.005707404715849356,
+            0.03722587302228139,
+            -0.01708925706528118,
+            -0.0065500772743641925,
+            -0.00015538686795361824,
+            0.010852925330001283,
+            0.010035359503147549,
+            0.008754719437760327,
+            0.004067820167375348,
+            -0.014861417626433965,
+            0.009346669946626484,
+            -0.020136029447262147,
+            0.007501669351583119,
+            0.023178490169347605,
+            -0.006840359541885465,
+            0.0006676450965357345,
+            -0.01283320739092204,
+            -0.015610538827742276,
+            0.006308543164022682,
+            0.002329420005586303,
+            -0.002295739637504597,
+            -0.02181189272979859,
+            0.006298804075737111,
+            -0.014398470126723037,
+            0.004007419942728446,
+            -0.0007497656993578918,
+            0.013863867106289847,
+            0.016451166273826513,
+            -0.006740209707090658,
+            0.039292260347522444,
+            0.01871723939316731,
+            0.009109005310505462,
+            0.005674782595856756,
+            -0.040588444759736254,
+            -0.008993850998154852,
+            -0.0010197185016393919,
+            0.004866150984083949,
+            -0.01141354216871663,
+            -0.018944881712855702,
+            -0.002646110151132781,
+            0.000308585036997592,
+            -0.010681954707313127,
+            -0.009091585604191968,
+            0.01751599002710486,
+            -0.006034424764302979,
+            0.00872057856926618,
+            0.004146961781109887,
+            0.023758065635807138,
+            -0.009727802672285544,
+            0.01176257119851645,
+            -0.0071822083765114165,
+            -0.005456197751568526,
+            0.0006726765703478756,
+            0.011011628907288813,
+            -0.008031792262886802,
+            0.009763340867550022,
+            -0.004279891138185881,
+            0.008355720233269893,
+            -0.008552596384067825,
+            -0.023952129458400563,
+            -0.014972827087541732,
+            0.014347535404251018,
+            0.011651260496709961,
+            0.008137738462704715,
+            -0.013624435569539285,
+            0.0031728721674035352,
+            0.00675830277744976,
+            -0.009184784787581942,
+            0.009161771311910335,
+            0.017733082848622708,
+            -0.001411542568472906,
+            -0.0017830167499636113,
+            -0.0016130003349297027,
+            -0.012173528776899308,
+            0.0011196437753239162,
+            0.02284887358189995,
+            -0.003054479643937998,
+            0.014614283913976716,
+            0.010689486748036806,
+            0.009643292262102645,
+            -0.00810930837603355,
+            0.001915955511174678,
+            0.02341863048264366,
+            0.015022291783105098,
+            -0.004625715884342598,
+            -0.011422973894623295,
+            0.004259757717597968,
+            -0.010332744704689279,
+            0.007373328473086509,
+            -0.02117940113144938,
+            -0.004672698999328004,
+            0.008896311380763048,
+            -0.007028862917603513,
+            -0.01732432285099198,
+            -0.028026173555055017,
+            -0.016765805892242715,
+            0.019819329380181,
+            0.0027213760678977965,
+            -0.013291251876048186,
+            -0.0032919084721662363,
+            0.0031214883337300926,
+            0.002553005168048115,
+            -0.0071104806975682885,
+            0.005310583052470149,
+            0.007359183453835734,
+            -0.008674133980486269,
+            0.017665958608281126,
+            -0.000521269431423141,
+            -0.0008020251454616109,
+            -0.0005442374261144659,
+            0.0006127687598711624,
+            -0.016387752806126442,
+            0.01666161912459431,
+            -0.010949228376419892,
+            0.015198110119816624,
+            -0.008966726991404666,
+            0.008600499511690635,
+            -0.007617043351740469,
+            0.009458641139070436,
+            0.011597957400445055,
+            0.012755037944648426,
+            -0.004403680820063604,
+            0.008519241473418196,
+            0.004807816191718704,
+            -0.008147125240985166,
+            0.007017601015398136,
+            0.01296082929962557,
+            0.00582353932160707,
+            0.0013634237970372698,
+            -0.0008246894322793264,
+            0.0022035670958705576,
+            -0.002480976219285683,
+            0.0056293125704658135,
+            0.012714850523393719,
+            -0.002497639449055006,
+            -0.007135683201188999,
+            0.014988249771676699,
+            0.014955895630183352,
+            -0.022994948923833993,
+            0.007121634434082677,
+            0.005835971825639792,
+            -0.0030507120821557008,
+            0.002145266953387315,
+            -0.027334525318320587,
+            -0.020359474070022213,
+            0.01901995970350808,
+            -0.015260454474605018,
+            0.02850460517419317,
+            -0.011624138792780823,
+            -0.02284242513728605,
+            -0.01277489645734181,
+            -0.004209529633534193,
+            -0.022684500849916897,
+            -0.004894414098943742,
+            0.01919165657366256,
+            0.013401560972810082,
+            -0.004313549020265677,
+            -0.019089369171102138,
+            0.03863248408555474,
+            0.010497951840980463,
+            0.03508157919687289,
+            -0.008597970944278045,
+            0.0023354831464448313,
+            -0.01563401808449416,
+            0.015551176593436512,
+            0.001075882802133782,
+            -0.011575692834796734,
+            0.01505716108583101,
+            0.031836226915668404,
+            0.0034740889237207494,
+            -0.00617206855371193,
+            0.0018420207148235309,
+            0.00418433325800744,
+            -0.0008167766650900381,
+            0.011160989178055499,
+            -0.018497472318034093,
+            -0.011951262348399922,
+            0.0013093376768780122,
+            0.00216347903176653,
+            0.0059233460419514045,
+            -0.001778091863744448,
+            -0.013317639704511549,
+            0.011433308892643204,
+            0.011596810919469926,
+            -0.01518143105525228,
+            -0.007497373860338335,
+            0.0050957665991362405,
+            -0.018974405317819656,
+            0.01872472902494418,
+            0.011859522398789263,
+            0.000792413325332956,
+            -0.0006503947129906383,
+            0.00808635755624928,
+            0.003749813548995223,
+            0.0033657799335872368,
+            -0.007862215303617144,
+            -0.0028014669075116876,
+            0.0009892310856576799,
+            3.834379772674841e-06,
+            0.015402211364447726,
+            -0.015264893397161102,
+            0.02488855366696587,
+            -0.0025643639525691083,
+            0.00953993868921819,
+            -0.017460590918773188,
+            -0.0017911712931080155,
+            -0.00402374787529868,
+            0.01838051670627292,
+            0.006753072393532114,
+            -0.010130676216186492,
+            0.004921510943896829,
+            0.007975201081296845,
+            -0.012589786869771252,
+            0.00018319054705558588,
+            -0.0022606180909706688,
+            -0.014791528072476771,
+            -0.0031171076370692706,
+            0.017873825561816544,
+            0.007593688317551089,
+            0.0037375070591255375,
+            -0.015863143964928172,
+            0.007192324815779268,
+            0.007522990667369163,
+            0.024504875417877527,
+            -0.026309816223960274,
+            0.024779042752840836,
+            -0.022104087334088543,
+            0.02165781034613893,
+            0.0319051741666687,
+            0.007538146369110045,
+            -0.01690181186708103,
+            -0.006595947249169521,
+            0.006373991305923002,
+            -0.023410814842488123,
+            -0.005494017994983188,
+            0.016903736810723716,
+            -0.01792400152372258,
+            0.000330865061689199,
+            -0.014660413003063089,
+            0.0040825880154611895,
+            0.008248037865838212,
+            0.011540333451799686,
+            -0.0033578830043738085,
+            0.018327525495676525,
+            0.020171745352159914,
+            -0.01138759854371235,
+            -0.0004221149053107702,
+            -0.022321409200941777,
+            0.005046865873981027,
+            0.022509643273449936,
+            0.001111272240506322,
+            -0.009540467231190513,
+            0.018376261423745193,
+            0.008480532791922674,
+            -0.020801891594934313,
+            0.008436374227753589,
+            -0.036175982655998785,
+            -0.010454118523313122,
+            -0.009328507118928412,
+            -0.001218633166683384,
+            -0.004142773602181115,
+            -0.0002494371580772723,
+            -0.011365548842513265,
+            -7.699548642315061e-05,
+            -0.006889339420877534,
+            -0.0037173070609413474,
+            0.014953978004522174,
+            0.011678957032329096,
+            -0.003501047405385249,
+            -0.0073984932581048915,
+            0.008493128567102267,
+            0.006163937693564917,
+            -0.017034696006318337,
+            -0.013268418502042576,
+            0.0037122128689059085,
+            -0.020067656401141467,
+            0.005148329575092478,
+            0.00459729178556505,
+            0.014031669598917171,
+            -0.01849235806598133,
+            -0.009298127665365717,
+            0.019716512208448042,
+            0.0094466405158553,
+            -0.0008988916054855498,
+            0.004264260983102956,
+            -0.00032312548009733906,
+            0.002279987803645912,
+            -0.0008902345374346296,
+            -6.145528806973222e-06,
+            0.003602229465148649,
+            0.012503366799381277,
+            0.007759393855482671,
+            0.03694402922447051,
+            -0.016692148708641812,
+            0.005873765863350649,
+            -0.014187609702966002,
+            -0.0037592517335595654,
+            -0.00749939010396784,
+            -0.0004308157309481927,
+            -0.0022282234382577843,
+            0.0021999996319642854,
+            0.0052176773733353925,
+            0.009353171029508977,
+            0.002155405440765206,
+            -0.00823565584692709,
+            0.009691447250255694,
+            -0.015228173095533029,
+            0.003853858061067531,
+            -0.008207910291425473,
+            0.01017972550354623,
+            -0.009695280827515048,
+            0.0028705587799331246,
+            0.03849358164875786,
+            0.01130021300937487,
+            -0.004934175853966098,
+            -0.014300147534361459,
+            -0.0007796051618993811,
+            -0.01737877501558196,
+            -0.008579119859817895,
+            -0.005434792488602325,
+            -0.02617338653902575,
+            -0.00838916510137031,
+            0.015344260464413648,
+            0.011202567142310492,
+            0.014124853863170132,
+            -0.023894182185855824,
+            0.008897943489322673,
+            -0.03001670823127986,
+            -0.011951507895140876,
+            -0.001530967140156163,
+            -0.015467812535428609,
+            0.0010110999991167703,
+            0.023623196504289838,
+            -0.007762735863648054,
+            0.012654652822750325,
+            0.012867958170258448,
+            0.018777522196571934,
+            0.006390986057157016,
+            0.00038045736974183937,
+            -0.009444911783760569,
+            0.009534310165417939,
+            0.008806031651319704,
+            0.0080954543713255,
+            0.013161833638467553,
+            -0.0022840529693446553,
+            -0.018060162439023572,
+            0.005469288590000856,
+            0.014354475778235131,
+            -0.0033504276914600585,
+            -0.001615191819057537,
+            -0.011088493180661976,
+            0.016039040565630432,
+            -0.023559046333872156,
+            -0.009482398794557168,
+            0.009918576960319548,
+            -0.0027308309155029842,
+            0.009212063547988036,
+            0.0160181200081126,
+            -0.01881373001297218,
+            -0.0024168618343039703,
+            0.016244889977796682,
+            0.0024104546308670025,
+            -0.00022606425578303413,
+            0.016792244238218245,
+            -0.0007616857360728623,
+            -0.004279120886828465,
+            0.00997764266310123,
+            -0.0075631917138318655,
+            7.3878137053128405e-06,
+            -0.002416364915008295,
+            -0.0028253498168087964,
+            0.02054128727706496,
+            -0.021281801678871736,
+            -0.005379259148621041,
+            0.02257306103533458,
+            0.015687105622685975,
+            -0.007843745111084413,
+            6.258498042855072e-05,
+            -0.0219506172079994,
+            0.0019925886885244454,
+            -0.007768133999661569,
+            0.016524850592507592,
+            -0.007957949830060354,
+            0.0030085044673901665,
+            -0.01361024620290394,
+            0.011563781350927133,
+            -0.01277407169098418,
+            0.00873716215529134,
+            -0.002986943948333218,
+            -0.017567646684567492,
+            -0.0053109390854215125,
+            0.004561236136619266,
+            0.0014045175130801308,
+            0.044958041347040185,
+            -0.005539136232703934,
+            0.003729517522976541,
+            0.018086335927383804,
+            -0.014199453349727933,
+            0.02079719360289355,
+            0.016663293302592568,
+            0.00991445855219406,
+            -0.00033967370741635794,
+            -0.0018726748849858836,
+            0.023373354806639515,
+            0.013258226441642464,
+            0.00861986998150065,
+            -0.0014340218811281975,
+            -0.008516541227326329,
+            0.018239230461345617,
+            0.005458629943667832,
+            -0.012421896028975013,
+            0.012732721349758832,
+            0.013559877132508506,
+            -0.013720939472355347,
+            -0.011545210834395917,
+            0.01616358223872008,
+            -0.019904594319155373,
+            -0.018260016571763644,
+            -0.0036124537591368864,
+            -0.01614510591132948,
+            -0.03864699384205461,
+            0.01619401261888991,
+            0.018351607973242735,
+            0.006666313000123845,
+            0.013647410298896132,
+            0.002597680061427196,
+            0.004545530267244157,
+            -0.018531082430861854,
+            -0.0077618625909995275,
+            -0.008085144524135461,
+            0.003744735923004248,
+            0.011059457364156674,
+            -0.011335432904114519,
+            -0.009514379703581616,
+            -0.00034500702695418285,
+            -0.007433132060989644,
+            0.01971420270994951,
+            0.0020753610290475642,
+            0.0034506706065161024,
+            -0.00628437298309269,
+            0.017966485854173167,
+            0.026789678964152908,
+            -0.007532446825077476,
+            0.021886182869092073,
+            -0.002864230253056434,
+            -0.0015870360120997492,
+            -0.014393519359298496,
+            -0.0014439412842964876,
+            -0.011527182213472351,
+            0.0005453493437517964,
+            -0.0006791481533636673,
+            0.014093767997869645,
+            -0.03288824642147967,
+            -0.003957301296796813,
+            0.014444592778851293,
+            0.007538052312876599,
+            -0.005057755472988024,
+            0.011357919882949903,
+            -0.002964451028327082,
+            0.016977317604413924,
+            0.0075201055190542146,
+            0.023391646079994784,
+            -0.016588823248397513,
+            -0.004504858318770565,
+            0.006660877064954993,
+            0.0034631255931793638,
+            0.0014484203909961972,
+            0.01912983041187014,
+            0.023439941258581887,
+            -0.014715886385460541,
+            -0.0013596123366001458,
+            -0.0029020372473879614,
+            -0.021414221343490594,
+            0.014976806123253727,
+            0.0074729481304182865,
+            0.010967351226114886,
+            0.009806663105020302,
+            0.015465076881663691,
+            -0.02109587527354609,
+            0.0011671121739822202,
+            0.020567458365051382,
+            -0.008618333549105003,
+            -0.018815706906171112,
+            -0.009804829158883123,
+            -0.00966828714719191,
+            -0.006078893678150302,
+            -0.00803783313451146,
+            0.005523023167028105,
+            0.0022647506733877573,
+            0.01652997604669998,
+            -0.008204200434441975,
+            -0.008404609147866811,
+            0.008489247125986078,
+            0.007546785673285454,
+            -0.0033093451283570316,
+            0.010614050963656496,
+            0.0030633179859393587,
+            -0.006940540318011839,
+            0.00894884343338777,
+            -0.002787816848430143,
+            0.014347598920245487,
+            -0.02441769157218451,
+            0.004717742147185283,
+            -0.012096876825337124,
+            0.006405151160675699,
+            0.0041568790527989905,
+            0.0016702066682613428,
+            -0.0041159211459004,
+            0.0035217362856754353,
+            0.0008893479210743541,
+            -0.07433312151926533,
+            -0.012629681414004524,
+            -0.0035797012285864486,
+            0.033657838713146884,
+            -0.019695806809053914,
+            0.004477927056336216,
+            0.011966065644382396,
+            0.0096717896523898,
+            0.003070713232538593,
+            0.006136726101157844,
+            0.006112385569942848,
+            -0.0245258242481677,
+            -0.00015284418977092615,
+            0.004821040694679944,
+            0.0020347004045943454,
+            -0.007532408630122127,
+            0.00750565079228405,
+            0.015327710245879296,
+            0.020989926895388523,
+            -0.010970305087289488,
+            0.017227663358885748,
+            -0.003352718496981928,
+            -0.006068892126250978,
+            0.0014285750322333686,
+            -0.010153799722483554,
+            -0.017307415843726394,
+            -0.006495008086641125,
+            0.0068983699213297075,
+            -0.001982568363667046,
+            0.0044032687939897855,
+            0.003333596495330213,
+            -0.0032171586993948696,
+            0.0072365127663633925,
+            0.013920326463272646,
+            0.006415916101287133,
+            -0.013675562796300214,
+            0.003003782758807665,
+            -0.02500375035312058,
+            0.011945988879567646,
+            -0.012708975181025296,
+            -0.0009488161508767279,
+            -0.0047093877027523014,
+            -0.0035750724825159503,
+            -0.002213435677949806,
+            0.02043567538125188,
+            -0.014304009033474186,
+            -0.01802899775650995,
+            0.008187227768196673,
+            -0.011217013195839762,
+            0.008300740149622635,
+            -0.03382831122769974,
+            0.012386335491333264,
+            -0.0276029934382802,
+            -0.0002243752790856292,
+            -0.004521387639042107,
+            -0.0010631721066129095,
+            -0.009491184157278669,
+            -0.03274559774619223,
+            0.015845705808609598,
+            -0.007204751554079551,
+            -0.020338337341085228,
+            0.0007143424225187802,
+            -0.021267094083674304,
+            0.014150458196361147,
+            -0.01851165553224022,
+            0.0033450353123933058,
+            -0.018775439467431685,
+            0.002782289821140401,
+            -0.0018711350216089556,
+            0.010502195116579441,
+            -0.00557814442809019,
+            -0.0007130962242755068,
+            -0.013072988887382269,
+            -0.002121841212381393,
+            -0.006653444103331213,
+            0.009394907690516121,
+            0.0028678074167728367,
+            -0.009601577734902932,
+            0.011153571864462314,
+            0.013046116892519414,
+            0.019812860521973938,
+            -0.00539047787915588,
+            -0.00555139958411929,
+            -0.011553689090179406,
+            0.009852063270798534,
+            -0.016326212706634428,
+            -0.00827402913960903,
+            0.008982303148984887,
+            0.015216947130173639,
+            0.01026889105233137,
+            0.003352028462252994,
+            0.003150801750574548,
+            0.002531620731961765,
+            0.026744691368233237,
+            -0.0015120532859309638,
+            0.0008324701960729416,
+            -0.014508988496009931,
+            0.022127525293249926,
+            0.02772430884375744,
+            -0.0012206102581726523,
+            -0.00018859639594030037,
+            0.017437360169132967,
+            -0.004697052522250805,
+            -0.0038111184371524645,
+            0.024015246959315593,
+            -0.038396108183338655,
+            -0.008416825232533771,
+            -0.006788816882617399,
+            -0.006043428198944214,
+            -0.009587366247924467,
+            0.007294190273465812,
+            0.00041966080769004275,
+            0.0024709692753547823,
+            -0.0005250904505954556,
+            0.03452858797518386,
+            0.003403515654095309,
+            -0.0003392060101496309,
+            -0.0032980864446307116,
+            0.0019125121221204029,
+            0.0032987517853214346,
+            -0.009116761755779728,
+            -0.038030085122772624,
+            0.006006334000396066,
+            0.007932785488877407,
+            0.00975009749364395,
+            -0.008120121513849434,
+            -0.008864461019882246,
+            0.004909348738798099,
+            -0.02273129283865203,
+            -0.01372636307167243,
+            0.02625229144362647,
+            -0.008523028986127237,
+            0.029005877324974557,
+            -0.022435059061182174,
+            -0.013135836801818153,
+            -0.004126433322528265,
+            0.010283975013795537,
+            -0.019732856462444506,
+            0.004343860338939202,
+            -0.004022387065239562,
+            0.023733721026005954,
+            0.0077578144626961245,
+            -0.008371631446146718,
+            0.012386735614153616,
+            0.0064789380436815714,
+            -0.0026638642827013837,
+            0.00136333052963848,
+            -0.007209517067748103,
+            0.016916505654836485,
+            -0.011819784026452944,
+            -0.005833909403097991,
+            0.008977527508585844,
+            -0.0024556434204852946,
+            0.02509293948918158,
+            -0.0237762245071987,
+            0.005288171058938098,
+            -0.00019047961064753207,
+            0.01822542471604037,
+            0.008127347078975925,
+            0.008361588667386882,
+            -0.011835407590060994,
+            0.010004482616697591,
+            -0.03378107067882622,
+            -0.015433906677849963,
+            -0.007702910250795191,
+            -0.0013877702264578865,
+            -0.002059152251387601,
+            -0.015745218548968338,
+            -0.026393221735795527,
+            0.006294583740809379,
+            0.016267525575472445,
+            0.023291528020980265,
+            -0.004683197930892445,
+            0.012912830736650997,
+            -0.002831583991769044,
+            -0.008904271394710672,
+            -0.011441882849296418,
+            -0.012202629605208118,
+            -0.012470773094383122,
+            -0.014560694782340029,
+            -0.017045701603711442,
+            0.012139665577738337,
+            -0.020079061996873537,
+            0.013911518328924777,
+            0.0347210775522539,
+            -0.017176114991595492,
+            -0.0018941772168839354,
+            0.019702722555622765,
+            -0.004221469612316137,
+            0.01212088353057276,
+            0.0063707489618353725,
+            0.015298014532351866,
+            0.0038643174573634725,
+            -0.012181479717826009,
+            -0.007374005740809251,
+            0.019411457664902376,
+            -0.004902591844710158,
+            0.014591446725717338,
+            0.015069599884557399,
+            -0.008675672164385891,
+            0.002544198867387345,
+            0.01951865904496276,
+            -0.01762101777798757,
+            0.01665654707715257,
+            -0.00458402229480694,
+            -0.009936206685538294,
+            -0.013345740581684638,
+            -0.005336416724426228,
+            -0.019469457091169776,
+            0.0061608454109892425,
+            0.018931676738452464,
+            -0.00533013594306834,
+            0.009310871143727683,
+            -0.0037670793158045,
+            -0.005045151384034014,
+            0.0002041909886640482,
+            -0.04152751596227068,
+            -0.008340911336217609,
+            -0.0006651584370005978,
+            -0.02505748815737022,
+            0.025954102004571416,
+            0.007205602377144382,
+            0.022602344303400163,
+            -0.017168610245491406,
+            0.004459735690564842,
+            -0.005882024209102714,
+            0.013217517324383834,
+            0.02255689669623969,
+            0.0017984057469785134,
+            0.005699128497215882,
+            -0.02353849721739205,
+            -0.0034711576718386185,
+            0.015924913374553738,
+            -0.003604478141616307,
+            -0.0034993903155063817,
+            0.012592308877056896,
+            0.012864938348003884,
+            0.01567418642605425,
+            0.008801753550506258,
+            0.015606443428689292,
+            0.010589933994371628,
+            0.008928721556432368,
+            -0.005805129999077126,
+            -0.001025103064751441,
+            -0.015907755521365925,
+            0.0045874330999415985,
+            0.031701049450012876,
+            0.012594483199773746,
+            0.016783858873835704,
+            -0.01778536268185936,
+            0.00789523134933153,
+            -0.0012706411671644011,
+            0.008325108613530198,
+            -0.001677173669891956,
+            0.029540136288412976,
+            0.011739999605699826,
+            0.015670966746894652,
+            0.008732334090105053,
+            -0.016237731209426752,
+            0.045334690225148,
+            -0.016911928942220152,
+            -0.0021139311364009113,
+            -0.01058970056572469,
+            -0.021705338531028618,
+            0.007324792557651635,
+            -0.006556582104686972,
+            0.0018243188317438144,
+            0.010780292738415671,
+            -0.006471850255151019,
+            0.0006997395548596146,
+            0.003907433192252573,
+            -0.023129313269763224,
+            -0.002550875642832235,
+            -0.02237917712764273,
+            -0.003107775639788617,
+            -0.02621608419628401,
+            -0.00012482032027919036,
+            -0.016207377503696517,
+            -0.0038712739995960753,
+            -0.023315508569288476,
+            -0.014963211090562272,
+            -0.01633110043829169,
+            -0.004241081909202262,
+            -0.008295402942465353,
+            0.004976396647108597,
+            -0.007402107755970892,
+            -0.020321297548927468,
+            0.0020986169430507757,
+            0.016803273183070004,
+            0.01434729320047051,
+            0.018612970083178984,
+            0.00603868809307069,
+            0.014278001198851581,
+            -0.0026350854954985117,
+            0.017171016209107656,
+            -0.006968968261909459,
+            0.006881847166565678,
+            0.011454001418121107,
+            0.02140935294963631,
+            0.011121833270235448,
+            -0.012801756637470525,
+            0.0010870041670931706,
+            -0.009655388034828094,
+            0.002094601818569396,
+            -0.00828486034340067,
+            -0.009190975158350585,
+            0.0017762581946855365,
+            -0.007261996868906865,
+            0.004674405202069334,
+            0.0030526253833799426,
+            -0.0014082232541965316,
+            0.010419302200840906,
+            -0.0024200053377751805,
+            -0.012208685875688886,
+            -0.0033037330470612695,
+            -0.017975940046938498,
+            -0.006958042903103732,
+            -0.0014496522238488239,
+            -0.011711502944602638,
+            -0.01016130418405875,
+            0.003321436189059608,
+            -0.00538282506552848,
+            -0.009353581612407435,
+            0.013884147453136417,
+            0.0001309738696853529,
+            0.0005878330075918333,
+            0.023495280602974504,
+            0.006035028203642483,
+            0.008087594627921345,
+            0.013144288945385904,
+            -0.01023995000994454,
+            -0.022574958230635715,
+            -0.0243435727055662,
+            0.007867022215096418,
+            -0.01573478573423379,
+            0.02324699908478915,
+            -0.007929809922043611,
+            -0.00561630876431978,
+            0.0020982723335057625,
+            -0.013699317926847062,
+            0.008578431496569186,
+            -0.011271381061801356,
+            0.0014450645788554738,
+            -0.019581451799757368,
+            0.018059311800267413,
+            -0.010917911869373693,
+            -0.004302687883018328,
+            0.009466103226841532,
+            0.034049086327052606,
+            -0.0005689605174080737,
+            -0.007410970644432269,
+            -0.006594150877740108,
+            -0.00022810271313786974,
+            0.018766765773409123,
+            -0.0029041815756733333,
+            0.005758781908059541,
+            -0.01563519107805783,
+            0.006571628287171105,
+            -0.006132372252825422,
+            0.0012423261344297225,
+            0.01513074831647933,
+            -0.026306320707079913,
+            -0.015491710256868008,
+            -0.007857352164317122,
+            -0.024893430640551818,
+            0.002253574511878983,
+            0.017579477884206534,
+            0.020505606775521504,
+            -0.020324505029926956,
+            -0.02513332780748072,
+            0.005150966733930881,
+            -0.026580994694344065,
+            0.0037390577082486594,
+            0.1241031878517593,
+            -0.002138404898309707,
+            0.002720723037908521,
+            -0.011425036886011179,
+            0.0022399461009316216,
+            0.005389407121650262,
+            -0.019552543955456567,
+            0.05950746473186985,
+            -0.00638023587296763,
+            -0.004023798021748334,
+            0.002170545188776296,
+            0.001785855390676113,
+            -0.005831534307818096,
+            0.008643501093778434,
+            -0.017602787281866816,
+            -0.02184792441891879,
+            -0.0052215705748838905,
+            -0.021252169339074325,
+            -0.008509621422071373,
+            -0.013000137524907565,
+            0.002893880203681282,
+            -0.015821609145923173,
+            0.010397692426614709,
+            -0.01638916243796692,
+            -0.010283706937701179,
+            -0.01103621454761223,
+            -0.00373552043529251,
+            0.01563553822905027,
+            -0.03188533919653949,
+            -0.004873550301425776,
+            -0.005183590054152726,
+            0.018218971950313273,
+            0.006004328765748325,
+            0.004357734370247252,
+            -0.014230318971458851,
+            0.0064792070801456235,
+            0.0215589173714926,
+            0.0055272493764401305,
+            0.014643335833970546,
+            0.010618070641357003,
+            -0.0022177303813575846,
+            -0.0024402761885758007,
+            0.017541799738501007,
+            -0.005445774414228888,
+            -0.00023502936119626386,
+            -0.00537569046685066,
+            -0.005974703247446444,
+            -0.010712244203425036,
+            0.01843613166635215,
+            -0.009938564643456672,
+            -0.017388592763277214,
+            -0.0015797412875366156,
+            0.02028434083164951,
+            -0.012761534382003973,
+            0.0031906812114950982,
+            0.000788436210275669,
+            0.020589196433948156,
+            0.0007726855314648301,
+            -0.005481260996289263,
+            0.007165014826294827,
+            -0.02956334397512344,
+            0.004095715232997753,
+            0.0003568512047626187,
+            0.027290768799658912,
+            -0.0017631129386615335,
+            0.01552542586826711,
+            -0.020871286597113327,
+            0.004388125824324094,
+            -0.016926008631435403,
+            0.005099899758322398,
+            0.012560923968841436,
+            0.002781049890810361,
+            0.010889021753593997,
+            -0.008500358897706648,
+            -0.021353460542554444,
+            0.012663934009161902,
+            -0.018657094130860155,
+            -0.029156017587920958,
+            0.0018889388761887053,
+            -0.028158076315864772,
+            -0.003048854609750186,
+            0.006812072971079798,
+            -0.035217412591558014,
+            -0.02251675180883659,
+            0.007396524512996277,
+            -0.00995571600201846,
+            -0.011912753666073882,
+            0.010675240017836214,
+            -0.02645155743041798,
+            -0.018137648527851668,
+            0.0010666944556449141,
+            0.012588800077165337,
+            -0.004561825939541096,
+            0.007555558090958502,
+            -0.028558603070876605,
+            0.014816816621445352,
+            -0.01313764261744744,
+            0.016036065063831813,
+            -0.0037797584923144134,
+            0.022217516257304868,
+            0.011131597210722573,
+            0.008795376259031849,
+            -0.004950080558710245,
+            0.021409590863550987,
+            0.005574351961953293,
+            0.051358937509801736,
+            0.031972009383746405,
+            0.0019821003238211276,
+            -0.003251257665378234,
+            0.003313045083311649,
+            0.008294776423730887,
+            -0.00940948257229233,
+            0.005114631360856872,
+            0.028053227377164806,
+            0.016738013668710083,
+            -0.018755924318109216,
+            0.006606711923035118,
+            -0.02702692774986498,
+            0.017950884766781056,
+            -0.022054653845163,
+            0.0003755944703013671,
+            0.007766253634286795,
+            0.0016206993755471613,
+            0.010682347575070117,
+            0.0013842930079185415,
+            0.01705234255261425,
+            0.023975191051210525,
+            -0.002976402980885077,
+            0.017466027107477605,
+            0.012734909275073793,
+            0.018143161773162015,
+            -0.0038399373197265827,
+            0.006929440921265761,
+            -0.012003095293274786,
+            -0.0008883470597690258,
+            0.0020490527756307267,
+            -0.004231091383919565,
+            0.012128848448049833,
+            0.008340302519160299,
+            0.012989849936937843,
+            0.002974488532907779,
+            0.003198854381461613,
+            -0.005309753760578859,
+            -0.0017024982049936384,
+            -0.01763395035509857,
+            -0.01819680226734474,
+            0.028417724274372468,
+            -0.020217879775485576,
+            -0.050118184195774004,
+            0.0025509548991664425,
+            0.014185761006969567,
+            -0.0012021669304388173,
+            -0.0051260576846881515,
+            -0.0005774778872927327,
+            0.005718864494530703,
+            -0.016918749210038116,
+            -0.0015472638835434902,
+            0.008125845907627804,
+            0.020347547062899797,
+            -0.002036218493206914,
+            -0.01079739518866248,
+            0.0038112958713600906,
+            0.02169339303105242,
+            0.01667594322285835,
+            0.012338193092567552,
+            -0.009798409233733167,
+            0.0003198215558702285,
+            -1.6308740825163337e-05,
+            0.0076870827874737325,
+            0.004356644356656038,
+            0.003500023005601613,
+            -0.019968549869439228,
+            -0.008634740687755112,
+            0.012781565963545847,
+            -0.0025637758121512927,
+            0.010243567793632335,
+            -0.00042296044205132426,
+            -0.003069251553118065,
+            -0.009344199449486244,
+            -0.004694953215957636,
+            -0.009583993744231791,
+            0.013783878036521561,
+            -0.007705982042252044,
+            -0.01549567538665377,
+            0.00020666298495823114,
+            0.0005515775451348472,
+            0.001667367278489879,
+            -0.004518431526236256,
+            0.03302324454129141,
+            0.009531990824506103,
+            -0.0004937348108340977,
+            -0.004047613286672229,
+            -0.0327383112522095,
+            0.012456150771224677,
+            -0.009677533214778067,
+            0.0006455811519981003,
+            0.020012976827426605,
+            -0.00020277620481825933,
+            0.002168432791818704,
+            -0.015657551629104767,
+            0.0032924357559238502,
+            0.006161295735841883,
+            0.013814165416788555,
+            0.0005164426482972658,
+            0.01076512534105537,
+            -0.006498977875968572,
+            -0.010325144017898257,
+            0.004804158067334203,
+            0.0104064855510473,
+            -0.0012188100892986933,
+            0.005171197790296134
+          ],
+          "title": "Mixture-Of-Experts With Expert Choice Routing",
+          "keyphrases": [
+            "Mixture-Of-Experts",
+            "Expert Choice Routing",
+            "sparsely-activated models",
+            "training efficiency",
+            "downstream performance"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "14339b1f-53f8-479b-9da7-9430b174b9ee",
+        "properties": {
+          "page_content": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer\n\nNoam Shazeer1, Azalia Mirhoseini∗†1, Krzysztof Maziarz∗2, Andy Davis1, Quoc Le1, Geoffrey Hinton1and Jeff Dean1 1Google Brain, {noam,azalia,andydavis,qvl,geoffhinton,jeff}@google.com 2Jagiellonian University, Cracow, krzysztof.maziarz@student.uj.edu.pl\n\nAbstract\n\nThe capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost.\n\n1 Introduction And Related Work 1.1 Conditional Computation\n\nExploiting scale in both training data and model size has been central to the success of deep learning. When datasets are sufficiently large, increasing the capacity (number of parameters) of neural networks can give much better prediction accuracy. This has been shown in domains such as text (Sutskever et al., 2014; Bahdanau et al., 2014; Jozefowicz et al., 2016; Wu et al., 2016), images (Krizhevsky et al., 2012; Le et al., 2012), and audio (Hinton et al., 2012; Amodei et al., 2015). For typical deep learning models, where the entire model is activated for every example, this leads to a roughly quadratic blow-up in training costs, as both the model size and the number of training examples increase. Unfortunately, the advances in computing power and distributed computation fall short of meeting such demand.\n\nVarious forms of conditional computation have been proposed as a way to increase model capacity without a proportional increase in computational costs (Davis & Arel, 2013; Bengio et al., 2013; Eigen et al., 2013; Ludovic Denoyer, 2014; Cho & Bengio, 2014; Bengio et al., 2015; Almahairi et al., 2015). In these schemes, large parts of a network are active or inactive on a per-example basis. The gating decisions may be binary or sparse and continuous, stochastic or deterministic. Various forms of reinforcement learning and back-propagation are proposed for trarining the gating decisions.\n\nWhile these ideas are promising in theory, no work to date has yet demonstrated massive improvements in model capacity, training time, or model quality. We blame this on a combination of the following challenges:\n\nModern computing devices, especially GPUs, are much faster at arithmetic than at branching. Most of the works above recognize this and propose turning on/off large chunks of the network with each gating decision.\n\nLarge batch sizes are critical for performance, as they amortize the costs of parameter transfers and updates. Conditional computation reduces the batch sizes for the conditionally active chunks of the network.\n\nNetwork bandwidth can be a bottleneck. A cluster of GPUs may have computational power thousands of times greater than the aggregate inter-device network bandwidth. To be computationally efficient, the relative computational versus network demands of an algorithm must exceed this ratio. Embedding layers, which can be seen as a form of conditional computation, are handicapped by this very problem. Since the embeddings generally need to be sent across the network, the number of (example, parameter) interactions is limited by network bandwidth instead of computational capacity.\n\nDepending on the scheme, loss terms may be necessary to achieve the desired level of sparsity per-chunk and/or per example. Bengio et al. (2015) use three such terms. These issues can affect both model quality and load-balancing.\n\nModel capacity is most critical for very large data sets. The existing literature on conditional computation deals with relatively small image recognition data sets consisting of up to 600,000 images. It is hard to imagine that the labels of these images provide a sufficient signal to adequately train a model with millions, let alone billions of parameters. In this work, we for the first time address all of the above challenges and finally realize the promise of conditional computation. We obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets.\n\n1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1). All parts of the network are trained jointly by back-propagation.\n\nWhile the introduced technique is generic, in this paper we focus on language modeling and machine translation tasks, which are known to benefit from very large models. In particular, we apply a MoE convolutionally between stacked LSTM layers (Hochreiter & Schmidhuber, 1997), as in Figure 1.\n\nThe MoE is called once for each position in the text, selecting a potentially different combination of experts at each position. The different experts tend to become highly specialized based on syntax and semantics (see Appendix E Table 9). On both language modeling and machine translation benchmarks, we improve on best published results at a fraction of the computational cost.\n\n1.3 Related Work On Mixtures Of Experts\n\nSince its introduction more than two decades ago (Jacobs et al., 1991; Jordan & Jacobs, 1994), the mixture-of-experts approach has been the subject of much research. Different types of expert architectures hae been proposed such as SVMs (Collobert et al., 2002), Gaussian Processes (Tresp, 2001; Theis & Bethge, 2015; Deisenroth & Ng, 2015), Dirichlet Processes (Shahbaba & Neal, 2009), and deep networks. Other work has focused on different expert configurations such as a hierarchical structure (Yao et al., 2009), infinite numbers of experts (Rasmussen & Ghahramani, 2002), and adding experts sequentially (Aljundi et al., 2016). Garmash & Monz (2016) suggest an ensemble model in the format of mixture of experts for machine translation. The gating network is trained on a pre-trained ensemble NMT model. The works above concern top-level mixtures of experts. The mixture of experts is the whole model. Eigen et al. (2013) introduce the idea of using multiple MoEs with their own gating networks as parts of a deep model. It is intuitive that the latter approach is more powerful, since complex problems may contain many sub-problems each requiring different experts. They also allude in their conclusion to the potential to introduce sparsity, turning MoEs into a vehicle for computational computation.\n\nOur work builds on this use of MoEs as a general purpose neural network component. While Eigen et al. (2013) uses two stacked MoEs allowing for two sets of gating decisions, our convolutional application of the MoE allows for different gating decisions at each position in the text. We also realize sparse gating and demonstrate its use as a practical way to massively increase model capacity.\n\n2 The Structure Of The Mixture-Of-Experts Layer\n\nThe Mixture-of-Experts (MoE) layer consists of a set of n \"expert networks\" E1, · · · , En, and a \"gating network\" G whose output is a sparse n-dimensional vector. Figure 1 shows an overview of the MoE module. The experts are themselves neural networks, each with their own parameters.\n\nAlthough in principle we only require that the experts accept the same sized inputs and produce the same-sized outputs, in our initial investigations in this paper, we restrict ourselves to the case where the models are feed-forward networks with identical architectures, but with separate parameters.\n\nLet us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows:\n\n$$y=\\sum_{i=1}^{n}G(x){i}E{i}(x)$$ $$(\\mathbf{l})$$ G(x)iEi(x) (1) We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example. If the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network. In the following we focus on ordinary MoEs. We provide more details on hierarchical MoEs in Appendix B. Our implementation is related to other models of conditional computation. A MoE whose experts are simple weight matrices is similar to the parameterized weight matrix proposed in (Cho & Bengio, 2014). A MoE whose experts have one hidden layer is similar to the block-wise dropout described in (Bengio et al., 2015), where the dropped-out layer is sandwiched between fully-activated layers.\n\n2.1 Gating Network\n\nSoftmax Gating: A simple choice of non-sparse gating function (Jordan & Jacobs, 1994) is to multiply the input by a trainable weight matrix Wg and then apply the Sof tmax function.\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ $\\eqref{eq:walpha}$. Gσ(x) = Sof tmax(x · Wg) (2) Noisy Top-K Gating: We add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to −∞ (which causes the corresponding gate values to equal 0). The sparsity serves to save computation, as described above. While this form of sparsity creates some theoretically scary discontinuities in the output of gating function, we have not yet observed this to be a problem in practice. The noise term helps with load balancing, as will be discussed in Appendix A. The amount of noise per component is controlled by a second trainable weight matrix Wnoise.\n\n$$G(x)=S o f t m a x(K e e p T o p K(H(x),k))$$ $$(4)$$ $$(S)$$ G(x) = Sof tmax(KeepT opK(H(x), k)) (3) $$H(x){i}=(x\\cdot W{g}){i}+S t a n d a r d N o r m a l()\\cdot S o f t p l u s((x\\cdot W{n o i s e}){i})$$ $KeepTopK(v,k){i}=\\begin{cases}v_{i}&\\text{if}v_{i}\\text{is in the top}k\\text{elements of}v.\\ -\\infty&\\text{otherwise.}\\end{cases}$ Training the Gating Network We train the gating network by simple back-propagation, along with the rest of the model. If we choose k > 1, the gate values for the top k experts have nonzero derivatives with respect to the weights of the gating network. This type of occasionally-sensitive behavior is described in (Bengio et al., 2013) with respect to noisy rectifiers. Gradients also backpropagate through the gating network to its inputs. Our method differs here from (Bengio et al., 2015) who use boolean gates and a REINFORCE-style approach to train the gating network.\n\n3 Addressing Performance Challenges 3.1 The Shrinking Batch Problem\n\nOn modern CPUs and GPUs, large batch sizes are necessary for computational efficiency, so as to amortize the overhead of parameter loads and updates. If the gating network chooses k out of n experts for each example, then for a batch of b examples, each expert receives a much smaller batch of approximately kb n b examples. This causes a naive MoE implementation to become very inefficient as the number of experts increases. The solution to this shrinking batch problem is to make the original batch size as large as possible. However, batch size tends to be limited by the memory necessary to store activations between the forwards and backwards passes. We propose the following techniques for increasing the batch size: Mixing Data Parallelism and Model Parallelism: In a conventional distributed training setting, multiple copies of the model on different devices asynchronously process distinct batches of data, and parameters are synchronized through a set of parameter servers. In our technique, these different batches run synchronously so that they can be combined for the MoE layer. We distribute the standard layers of the model and the gating network according to conventional data-parallel schemes, but keep only one shared copy of each expert. Each expert in the MoE layer receives a combined batch consisting of the relevant examples from all of the data-parallel input batches. The same set of devices function as data-parallel replicas (for the standard layers and the gating networks) and as model-parallel shards (each hosting a subset of the experts). If the model is distributed over d devices, and each device processes a batch of size b, each expert receives a batch of approximately kbd nexamples. Thus, we achieve a factor of d improvement in expert batch size.\n\nIn the case of a hierarchical MoE (Section B), the primary gating network employs data parallelism, and the secondary MoEs employ model parallelism. Each secondary MoE resides on one device.\n\nThis technique allows us to increase the number of experts (and hence the number of parameters) by proportionally increasing the number of devices in the training cluster. The total batch size increases, keeping the batch size per expert constant. The memory and bandwidth requirements per device also remain constant, as do the step times, as does the amount of time necessary to process a number of training examples equal to the number of parameters in the model. It is our goal to train a trillionparameter model on a trillion-word corpus. We have not scaled our systems this far as of the writing of this paper, but it should be possible by adding more hardware.\n\nTaking Advantage of Convolutionality: In our language models, we apply the same MoE to each time step of the previous layer. If we wait for the previous layer to finish, we can apply the MoE to all the time steps together as one big batch. Doing so increases the size of the input batch to the MoE layer by a factor of the number of unrolled time steps.\n\nIncreasing Batch Size for a Recurrent MoE: We suspect that even more powerful models may involve applying a MoE recurrently. For example, the weight matrices of a LSTM or other RNN could be replaced by a MoE. Sadly, such models break the convolutional trick from the last paragraph, since the input to the MoE at one timestep depends on the output of the MoE at the previous timestep. Gruslys et al. (2016) describe a technique for drastically reducing the number of stored activations in an unrolled RNN, at the cost of recomputing forward activations. This would allow for a large increase in batch size.\n\n3.2 Network Bandwidth\n\nAnother major performance concern in distributed computing is network bandwidth. Since the experts are stationary (see above) and the number of gating parameters is small, most of the communication involves sending the inputs and outputs of the experts across the network. To maintain computational efficiency, the ratio of an expert's computation to the size of its input and output must exceed the ratio of computational to network capacity of the computing device. For GPUs, this may be thousands to one. In our experiments, we use experts with one hidden layer containing thousands of RELU-activated units. Since the weight matrices in the expert have sizes input_size×hidden_size and hidden_size × output_size, the ratio of computation to input and output is equal to the size of the hidden layer. Conveniently, we can increase computational efficiency simply by using a larger hidden layer, or more hidden layers.\n\n4 Balancing Expert Utilization\n\nWe have observed that the gating network tends to converge to a state where it always produces large weights for the same few experts. This imbalance is self-reinforcing, as the favored experts are trained more rapidly and thus are selected even more by the gating network. Eigen et al. (2013) describe the same phenomenon, and use a hard constraint at the beginning of training to avoid this local minimum. Bengio et al. (2015) include a soft constraint on the batch-wise average of each gate.1 We take a soft constraint approach. We define the importance of an expert relative to a batch of training examples to be the batchwise sum of the gate values for that expert. We define an additional loss Limportance, which is added to the overall loss function for the model. This loss is equal to the square of the coefficient of variation of the set of importance values, multiplied by a hand-tuned scaling factor wimportance. This additional loss encourages all experts to have equal importance.\n\n$$I m p o r t a n c e(X)=\\sum_{x\\in X}G(x)$$ $$L_{importance}(X)=w_{importance}\\cdot CV(Importance(X))^{2}\\tag{7}$$ $$(6)$$\n\nWhile this loss function can ensure equal importance, experts may still receive very different numbers of examples. For example, one expert may receive a few examples with large weights, and another may receive many examples with small weights. This can cause memory and performance problems on distributed hardware. To solve this problem, we introduce a second loss function, Lload , which ensures balanced loads. Appendix A contains the definition of this function, along with experimental results.\n\n5 Experiments 5.1 1 Billion Word Language Modeling Benchmark\n\nDataset: This dataset, introduced by (Chelba et al., 2013) consists of shuffled unique sentences from news articles, totaling approximately 829 million words, with a vocabulary of 793,471 words.\n\nPrevious State-of-the-Art: The best previously published results (Jozefowicz et al., 2016) use models consisting of one or more stacked Long Short-Term Memory (LSTM) layers (Hochreiter & Schmidhuber, 1997; Gers et al., 2000). The number of parameters in the LSTM layers of these models vary from 2 million to 151 million. Quality increases greatly with parameter count, as do computational costs. Results for these models form the top line of Figure 2-right. MoE Models: Our models consist of two stacked LSTM layers with a MoE layer between them (see Figure 1). We vary the sizes of the layers and the number of experts. For full details on model architecture, training regimen, additional baselines and results, see Appendix C.\n\nLow Computation, Varied Capacity: To investigate the effects of adding capacity, we trained\n\na series of MoE models all with roughly equal computational costs: about 8 million multiply-andadds per training example per timestep in the forwards pass, excluding the softmax layer. We call this metric (ops/timestep). We trained models with flat MoEs containing 4, 32, and 256 experts, and models with hierarchical MoEs containing 256, 1024, and 4096 experts. Each expert had about 1 million parameters. For all the MoE layers, 4 experts were active per input.\n\nThe results of these models are shown in Figure 2-left. The model with 4 always-active experts performed (unsurprisingly) similarly to the computationally-matched baseline models, while the largest of the models (4096 experts) achieved an impressive 24% lower perplexity on the test set. Figure 2: Model comparison on 1-Billion-Word Language-Modeling Benchmark. On the left, we plot test perplexity as a function of model capacity for models with similar computational budgets of approximately 8-million-ops-per-timestep. On the right, we plot test perplexity as a function of computational budget. The top line represents the LSTM models from (Jozefowicz et al., 2016).\n\nThe bottom line represents 4-billion parameter MoE models with different computational budgets.\n\nVaried Computation, High Capacity: In addition to the largest model from the previous section, we trained two more MoE models with similarly high capacity (4 billion parameters), but higher computation budgets. These models had larger LSTMs, and fewer but larger and experts. Details\n\nTest Test #Parameters ops/timestep Training TFLOPS Perplexity Perplexity excluding embedding Time /GPU 10 epochs 100 epochs and softmax layers 10 epochs Best Published Results 34.7 30.6 151 million 151 million 59 hours, 32 k40s 1.09 Low-Budget MoE Model 34.1 4303 million 8.9 million 15 hours, 16 k40s 0.74 Medium-Budget MoE Model 31.3 4313 million 33.8 million 17 hours, 32 k40s 1.22 High-Budget MoE Model 28.0 4371 million 142.7 million 47 hours, 32 k40s 1.56\n\nTable 1: Summary of high-capacity MoE-augmented models with varying computational budgets, vs. best previously published results (Jozefowicz et al., 2016). Details in Appendix C. can be found in Appendix C.2. Results of these three models form the bottom line of Figure 2-right.\n\nTable 1 compares the results of these models to the best previously-published result on this dataset .\n\nEven the fastest of these models beats the best published result (when controlling for the number of training epochs), despite requiring only 6% of the computation.\n\nComputational Efficiency: We trained our models using TensorFlow (Abadi et al., 2016) on clusters containing 16-32 Tesla K40 GPUs. For each of our models, we determine computational efficiency in TFLOPS/GPU by dividing the number of floating point operations required to process one training batch by the observed step time and the number of GPUs in the cluster. The operation counts used here are higher than the ones we report in our ops/timestep numbers in that we include the backwards pass, we include the importance-sampling-based training of the softmax layer, and we count a multiply-and-add as two separate operations. For all of our MoE models, the floating point operations involved in the experts represent between 37% and 46% of the total.\n\nFor our baseline models wtih no MoE, observed computational efficiency ranged from 1.07-1.29 TFLOPS/GPU. For our low-computation MoE models, computation efficiency ranged from 0.740.90 TFLOPS/GPU, except for the 4-expert model which did not make full use of the available parallelism. Our highest-computation MoE model was more efficient at 1.56 TFLOPS/GPU, likely due to the larger matrices. These numbers represent a significant fraction of the theoretical maximum of 4.29 TFLOPS/GPU claimed by NVIDIA. Detailed results are in Appendix C, Table 7.\n\n5.2 100 BILLION WORD GOOGLE NEWS CORPUS\n\nOn the 1-billion-word corpus, adding additional capacity seems to produce diminishing returns as the number of parameters in the MoE layer exceeds 1 billion, as can be seen in Figure 2-left. We hypothesized that for a larger training set, even higher capacities would produce significant quality improvements. We constructed a similar training set consisting of shuffled unique sentences from Google's internal news corpus, totalling roughly 100 billion words. Similarly to the previous section, we tested a series of models with similar computational costs of about 8 million ops/timestep. In addition to a baseline LSTM model, we trained models augmented with MoE layers containing 32, 256, 1024, 4096, 16384, 65536, and 131072 experts. This corresponds to up to 137 billion parameters in the MoE layer. Details on architecture, training, and results are given in Appendix D.\n\nResults: Figure 3 shows test perplexity as a function of capacity after training on 10 billion words (top line) and 100 billion words (bottom line). When training over the full 100 billion words, test perplexity improves significantly up to 65536 experts (68 billion parameters), dropping 39% lower than the computationally matched baseline, but degrades at 131072 experts, possibly a result of too much sparsity. The widening gap between the two lines demonstrates (unsurprisingly) that increased model capacity helps more on larger training sets. Even at 65536 experts (99.994% layer sparsity), computational efficiency for the model stays at a respectable 0.72 TFLOPS/GPU.\n\n5.3 Machine Translation (Single Language Pair)\n\nModel Architecture: Our model was a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decreased the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We inserted MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). Each MoE layer contained up to 2048 experts each with about two million parameters, adding a total of about 8 billion parameters to the models. Further details on model architecture, testing procedure and results can be found in Appendix E.\n\nDatasets: We benchmarked our method on the WMT'14 En→Fr and En→De corpora, whose training sets have 36M sentence pairs and 5M sentence pairs, respectively. The experimental protocols were also similar to those in (Wu et al., 2016): newstest2014 was used as the test set to compare against previous work (Luong et al., 2015a; Zhou et al., 2016; Wu et al., 2016), while the combination of newstest2012 and newstest2013 was used as the development set. We also tested the same model on a Google's Production English to French data.\n\nTable 2: Results on WMT'14 En→ Fr newstest2014 (bold values represent best results). Model Test Test ops/timenstep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.69 40.35 85M 8.7B 3 days/64 k40s MoE with 2048 Experts (longer training) 2.63 40.56 85M 8.7B 6 days/64 k40s GNMT (Wu et al., 2016) 2.79 39.22 214M 278M 6 days/96 k80s GNMT+RL (Wu et al., 2016) 2.96 39.92 214M 278M 6 days/96 k80s PBMT (Durrani et al., 2014) 37.0 LSTM (6-layer) (Luong et al., 2015b) 31.5 LSTM (6-layer+PosUnk) (Luong et al., 2015b) 33.1 DeepAtt (Zhou et al., 2016) 37.7 DeepAtt+PosUnk (Zhou et al., 2016) 39.2\n\nTable 3: Results on WMT'14 En → De newstest2014 (bold values represent best results). Model Test Test ops/timestep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 4.64 26.03 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 5.25 24.91 214M 278M 1 day/96 k80s GNMT +RL (Wu et al., 2016) 8.08 24.66 214M 278M 1 day/96 k80s PBMT (Durrani et al., 2014) 20.7 DeepAtt (Zhou et al., 2016) 20.6\n\nTable 4: Results on the Google Production En→ Fr dataset (bold values represent best results). Model Eval Eval Test Test ops/timestep Total Training Perplexity BLEU Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.60 37.27 2.69 36.57 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 2.78 35.80 2.87 35.56 214M 278M 6 days/96 k80s\n\nResults: Tables 2, 3, and 4 show the results of our largest models, compared with published results. Our approach achieved BLEU scores of 40.56 and 26.03 on the WMT'14 En→Fr and En→De benchmarks. As our models did not use RL refinement, these results constitute significant gains of 1.34 and 1.12 BLEU score on top of the strong baselines in (Wu et al., 2016). The perplexity scores are also better.2 On the Google Production dataset, our model achieved 1.01 higher test BLEU score even after training for only one sixth of the time.\n\n5.4 Multilingual Machine Translation\n\nDataset: (Johnson et al., 2016) train a single GNMT (Wu et al., 2016) model on a very large combined dataset of twelve language pairs. Results are somewhat worse than those for 12 separately trained single-pair GNMT models. This is not surprising, given that the twelve models have 12 times the capacity and twelve times the aggregate training of the one model. We repeat this experiment with a single MoE-augmented model. See Appendix E for details on model architecture.\n\nWe train our model on the same dataset as (Johnson et al., 2016) and process the same number of training examples (about 3 billion sentence pairs). Our training time was shorter due to the lower computational budget of our model.\n\nResults: Results for the single-pair GNMT models, the multilingual GNMT model and the multilingual MoE model are given in Table 5. The MoE model achieves 19% lower perplexity on the dev set than the multilingual GNMT model. On BLEU score, the MoE model significantly beats the multilingual GNMT model on 11 of the 12 language pairs (by as much as 5.84 points), and even beats the monolingual GNMT models on 8 of 12 language pairs. The poor performance on English → Korean seems to be a result of severe overtraining, as for the rarer language pairs a small number of real examples were highly oversampled in the training corpus.\n\nTable 5: Multilingual Machine Translation (bold values represent best results). GNMT-Mono GNMT-Multi MoE-Multi MoE-Multi vs. GNMT-Multi Parameters 278M / model 278M 8.7B ops/timestep 212M 212M 102M training time, hardware various 21 days, 96 k20s 12 days, 64 k40s Perplexity (dev) 4.14 3.35 -19% French → English Test BLEU 36.47 34.40 37.46 +3.06 German → English Test BLEU 31.77 31.17 34.80 +3.63 Japanese → English Test BLEU 23.41 21.62 25.91 +4.29 Korean → English Test BLEU 25.42 22.87 28.71 +5.84 Portuguese → English Test BLEU 44.40 42.53 46.13 +3.60 Spanish → English Test BLEU 38.00 36.04 39.39 +3.35 English → French Test BLEU 35.37 34.00 36.59 +2.59 English → German Test BLEU 26.43 23.15 24.53 +1.38 English → Japanese Test BLEU 23.66 21.10 22.78 +1.68 English → Korean Test BLEU 19.75 18.41 16.62 -1.79 English → Portuguese Test BLEU 38.40 37.35 37.90 +0.55 English → Spanish Test BLEU 34.50 34.25 36.21 +1.96\n\n6 Conclusion\n\nThis work is the first to demonstrate major wins from conditional computation in deep networks.\n\nWe carefully identified the design considerations and challenges of conditional computing and addressed them with a combination of algorithmic and engineering solutions. While we focused on text, conditional computation may help in other domains as well, provided sufficiently large training sets. We look forward to seeing many novel implementations and applications of conditional computation in the years to come.\n\nAcknowledgments\n\nWe would like to thank all of the members of the Google Brain and Google Translate teams who helped us with this project, in particular Zhifeng Chen, Yonghui Wu, and Melvin Johnson. Thanks also to our anonymous ICLR reviewers for the helpful suggestions on making this paper better.\n\n2Reported perplexities relative to the tokenization used by both our models and GNMT.\n\nReferences\n\nMartín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Gordon Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda B. Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. CoRR, abs/1603.04467, 2016. URL http://arxiv.org/abs/1603.04467.\n\nRahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. CoRR, abs/1611.06194, 2016. URL http://arxiv.org/abs/1611.\n\n06194.\n\nA. Almahairi, N. Ballas, T. Cooijmans, Y. Zheng, H. Larochelle, and A. Courville. Dynamic Capacity Networks. ArXiv e-prints, November 2015.\n\nDario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan, Christopher Fougner, Tony Han, Awni Y. Hannun, Billy Jun, Patrick LeGresley, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595, 2015. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.\n\nEmmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297, 2015.\n\nYoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.\n\nCiprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling.\n\narXiv preprint arXiv:1312.3005, 2013.\n\nK. Cho and Y. Bengio. Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning. ArXiv e-prints, June 2014. Ronan Collobert, Samy Bengio, and Yoshua Bengio. A parallel mixture of SVMs for very large scale problems. Neural Computing, 2002.\n\nAndrew Davis and Itamar Arel. Low-rank approximations for conditional feedforward computation in deep neural networks. arXiv preprint arXiv:1312.4461, 2013. Marc Peter Deisenroth and Jun Wei Ng. Distributed Gaussian processes. In ICML, 2015.\n\nJohn Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization, 2010.\n\nNadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. Edinburgh's phrase-based machine translation systems for wmt-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation, 2014.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013. Ekaterina Garmash and Christof Monz. Ensemble learning for multi-source neural machine translation. In staff.science.uva.nl/c.monz, 2016.\n\nFelix A. Gers, Jürgen A. Schmidhuber, and Fred A. Cummins. Learning to forget: Continual prediction with lstm. Neural Computation, 2000.\n\nAudrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. Memory-efficient backpropagation through time. CoRR, abs/1606.03401, 2016. URL http://arxiv.org/ abs/1606.03401. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2015.\n\nGeoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 2012. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.\n\nSergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.\n\nRobert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computing, 1991.\n\nMelvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation.\n\nCoRR, abs/1611.04558, 2016. URL http://arxiv.org/abs/1611.04558.\n\nMichael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm.\n\nNeural Computing, 1994. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.\n\nDiederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.\n\nReinhard Kneser and Hermann. Ney. Improved backingoff for m-gram language modeling., 1995.\n\nAlex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.\n\nQuoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeffrey Dean, and Andrew Y. Ng. Building high-level features using large scale unsupervised learning. In ICML, 2012. Patrick Gallinari Ludovic Denoyer. Deep sequential neural network. arXiv preprint arXiv:1410.0510, 2014.\n\nMinh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attentionbased neural machine translation. EMNLP, 2015a.\n\nMinh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the rare word problem in neural machine translation. ACL, 2015b.\n\nCarl Edward Rasmussen and Zoubin Ghahramani. Infinite mixtures of Gaussian process experts.\n\nNIPS, 2002.\n\nHasim Sak, Andrew W Senior, and Françoise Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH, pp. 338–342, 2014.\n\nMike Schuster and Kaisuke Nakajima. Japanese and Korean voice search. ICASSP, 2012.\n\nBabak Shahbaba and Radford Neal. Nonlinear models using dirichlet process mixtures. JMLR, 2009.\n\nIlya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks.\n\nIn NIPS, 2014.\n\nLucas Theis and Matthias Bethge. Generative image modeling using spatial LSTMs. In NIPS, 2015. Volker Tresp. Mixtures of Gaussian Processes. In NIPS, 2001.\n\nYonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.\n\nBangpeng Yao, Dirk Walther, Diane Beck, and Li Fei-fei. Hierarchical mixture of classification experts uncovers interactions between brain regions. In NIPS. 2009.\n\nWojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization.\n\narXiv preprint arXiv:1409.2329, 2014.\n\nJie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with fast-forward connections for neural machine translation. arXiv preprint arXiv:1606.04199, 2016.\n\nAppendices A Load-Balancing Loss\n\nAs discussed in section 4, for load-balancing purposes, we want to define an additional loss function to encourage experts to receive roughly equal numbers of training examples. Unfortunately, the number of examples received by an expert is a discrete quantity, so it can not be used in backpropagation. Instead, we define a smooth estimator Load(X) of the number of examples assigned to each expert for a batch X of inputs. The smoothness allows us to back-propagate gradients through the estimator. This is the purpose of the noise term in the gating function. We define P(x, i) as the probability that G(x)iis nonzero, given a new random choice of noise on element i, but keeping the already-sampled choices of noise on the other elements. To compute P(x, i), we note that the G(x)iis nonzero if and only if H(x)iis greater than the k th-greatest element of H(x) excluding itself. The probability works out to be:\n\n$$P(x,i)=P r\\Big((x\\cdot W_{g}){i}+S t a n d x N o r a l()\\cdot S o f t p u s((x\\cdot W{n o i s e})_{i})$$ $$>k t h_e x c l u d i n(H(x),k,i)\\Big)$$ $$({\\boldsymbol{8}})$$\n\n$$(9)$$ $$(10)$$ (8) Where kth_excluding(v, k, i) means the kth highest component of v, excluding component i. Simplifying, we get:\n\n$$P(x,i)=\\Phi{\\Big(}{\\frac{(x\\cdot W_{g}){i}-k t h_c x c l u d i n g(H(x),k,i)}{S o f t p l u s((x\\cdot W{n o i s e})_{i})}}{\\Big)}$$\n\nWhere Φ is the CDF of the standard normal distribution.\n\n$$L o a d(X){i}=\\sum{x\\in X}P(x,i)$$\n\n$$(11)$$ P(x, i) (10) We can now define the load loss to be the square of the coefficient of variation of the load vector, multiplied by a hand-tuned scaling factor wload.\n\n$$L_{l o a d}(X)=w_{l o a d}\\cdot C V(L o a d(X))^{2}$$ Lload(X) = wload · CV (Load(X))2(11) Initial Load Imbalance: To avoid out-of-memory errors, we need to initialize the network in a state of approximately equal expert load (since the soft constraints need some time to work). To accomplish this, we initialize the matrices Wg and Wnoise to all zeros, which yields no signal and some noise.\n\nExperiments: We trained a set of models with identical architecture (the MoE-256 model described in Appendix C), using different values of wimportance and wload. We trained each model for 10 epochs, then measured perplexity on the test set. We also measured the coefficients of variation in Importance and Load, as well as ratio of the load on the most overloaded expert to the average load. This last value is significant for load balancing purposes on distributed hardware. All of these metrics were averaged over several training batches.\n\nwimportance wload Test Perplexity CV (Importance(X)) CV (Load(X)) max(Load(X)) mean(Load(X)) 0.0 0.0 39.8 3.04 3.01 17.80 0.2 0.0 35.6 0.06 0.17 1.47 0.0 0.2 35.7 0.22 0.04 1.15 0.1 0.1 35.6 0.06 0.05 1.14 0.01 0.01 35.7 0.48 0.11 1.37 1.0 1.0 35.7 0.03 0.02 1.07\n\nTable 6: Experiments with different combinations of losses.\n\n13 Results: Results are reported in Table 6. All the combinations containing at least one the two losses led to very similar model quality, where having no loss was much worse. Models with higher values of wload had lower loads on the most overloaded expert.\n\nB Hierachical Mixture Of Experts\n\nIf the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network.3If the hierarchical MoE consists of a groups of b experts each, we denote the primary gating network by Gprimary, the secondary gating networks by (G1, G2..Ga), and the expert networks by (E0,0, E0,1..Ea,b). The output of the MoE is given by:\n\n$$y_{H}=\\sum_{i=1}^{a}\\sum_{j=1}^{b}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}\\cdot E{i,j}(x)$$ $$(12)$$ $$(13)$$ $$(14)$$\n\nOur metrics of expert utilization change to the following:\n\n$$I m p o r t a n c e_{H}(X){i,j}=\\sum{x\\in X}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}$$ $$L o a d{H}(X){i,j}={\\frac{L o a d{p r i m a r y}(X){i}\\cdot L o a d{i}(X^{(i)})_{j}}{|X^{(i)}|}}$$\n\nLoadprimary and Loadi deonte the Load functions for the primary gating network and i th secondary gating network respectively. X(i) denotes the subset of X for which Gprimary(x)i > 0.\n\nIt would seem simpler to let LoadH(X)i,j = Loadi(Xi)j , but this would not have a gradient with respect to the primary gating network, so we use the formulation above.\n\nC 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS C.1 8-MILLION-OPERATIONS-PER-TIMESTEP MODELS Model Architecture: Our model consists of five layers: a word embedding layer, a recurrent Long Short-Term Memory (LSTM) layer (Hochreiter & Schmidhuber, 1997; Gers et al., 2000), a MoE layer, a second LSTM layer, and a softmax layer. The dimensionality of the embedding layer, the number of units in each LSTM layer, and the input and output dimensionality of the MoE layer are all equal to 512. For every layer other than the softmax, we apply drouput (Zaremba et al., 2014) to the layer output, dropping each activation with probability DropP rob, otherwise dividing by (1 − DropP rob). After dropout, the output of the previous layer is added to the layer output.\n\nThis residual connection encourages gradient flow (He et al., 2015).\n\nMoE Layer Architecture: Each expert in the MoE layer is a feed forward network with one ReLU-activated hidden layer of size 1024 and an output layer of size 512. Thus, each expert contains [512 ∗ 1024] + [1024 ∗ 512] = 1M parameters. The output of the MoE layer is passed through a sigmoid function before dropout. We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts.\n\nWe call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers. Thus, each example is processed by exactly 4 experts for a total of 4M ops/timestep. The two LSTM layers contribute 2M ops/timestep each for the desired total of 8M.\n\n3 We have not found the need for deeper hierarchies.\n\nComputationally-Matched Baselines: The MoE-4 model does not employ sparsity, since all 4 experts are always used. In addition, we trained four more computationally-matched baseline models with no sparsity:\n\nMoE-1-Wide: The MoE layer consists of a single \"expert\" containing one ReLU-activated hidden layer of size 4096.\n\nMoE-1-Deep: The MoE layer consists of a single \"expert\" containing four ReLU-activated hidden layers, each with size 1024.\n\n4xLSTM-512: We replace the MoE layer with two additional 512-unit LSTM layers.\n\nLSTM-2048-512: The model contains one 2048-unit LSTM layer (and no MoE). The output of the LSTM is projected down to 512 dimensions (Sak et al., 2014). The next timestep of the LSTM receives the projected output. This is identical to one of the models published in (Jozefowicz et al., 2016). We re-ran it to account for differences in training regimen, and obtained results very similar to the published ones. Training: The models were trained on a cluster of 16 K40 GPUs using the synchronous method described in Section 3. Each batch consisted of a set of sentences totaling roughly 300,000 words. In the interest of time, we limited training to 10 epochs, (27,000 steps). Training took 12-16 hours for all models, except for MoE-4, which took 18 hours (since all the expert computation was performed on only 4 of 16 GPUs). We used the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 1000 training steps, and decreased after that so as to be proportional to the inverse square root of the step number. The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016). For each model, we performed a hyper-parmeter search to find the best dropout probability, in increments of 0.1.\n\nTo ensure balanced expert utilization we set wimportance = 0.1 and wload = 0.1, as described in Section 4 and Appendix A.\n\nResults: We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol. Results are reported in Table 7. For each model, we report the test perplexity, the computational budget, the parameter counts, the value of DropP rob, and the computational efficiency.\n\nModel Test Test ops/timestep #Params excluding Total Drop- TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params P rob per GPU 10 epochs (final) (millions) (billions) (observed) Kneser-Ney 5-gram* 67.6 0.00001 1.8 LSTM-512-512* 54.1 2.4 2.4 0.8 0.1 LSTM-1024-512* 48.2 4.7 4.7 0.8 0.1 LSTM-2048-512* 45.0 43.7 9.4 9.4 0.8 0.1 0.61 LSTM-2048-512 44.7 9.4 9.4 0.8 0.1 1.21 4xLSTM-512 46.0 8.4 8.4 0.8 0.1 1.07 MoE-1-Wide 46.1 8.4 8.4 0.8 0.1 1.29 MoE-1-Deep 45.7 8.4 8.4 0.8 0.1 1.29 MoE-4 45.0 8.4 8.4 0.8 0.1 0.52 MoE-32 39.7 8.4 37.8 0.9 0.1 0.87 MoE-256 35.7 8.6 272.9 1.1 0.1 0.81 MoE-256-h 36.0 8.4 272.9 1.1 0.1 0.89 MoE-1024-h 34.6 8.5 1079.0 1.9 0.2 0.90 MoE-4096-h 34.1 8.9 4303.4 5.1 0.2 0.74 2xLSTM-8192-1024* 34.7 30.6 151.0 151.0 1.8 0.25 1.09 MoE-34M 31.3 33.8 4313.9 6.0 0.3 1.22 MoE-143M 28.0 142.7 4371.1 6.0 0.4 1.56\n\nC.2 More Expensive Models\n\nWe ran two additional models (MoE-34M and MoE-143M) to investigate the effects of adding more computation in the presence of a large MoE layer. These models have computation budgets of 34M and 143M ops/timestep. Similar to the models above, these models use a MoE layer between two LSTM layers. The dimensionality of the embedding layer, and the input and output dimensionality of the MoE layer are set to 1024 instead of 512. For MoE-34M, the LSTM layers have 1024 units.\n\nFor MoE-143M, the LSTM layers have 4096 units and an output projection of size 1024 (Sak et al., 2014). MoE-34M uses a hierarchical MoE layer with 1024 experts, each with a hidden layer of size 2048. MoE-143M uses a hierarchical MoE layer with 256 experts, each with a hidden layer of size 8192. Both models have 4B parameters in the MoE layers. We searched for the best DropP rob for each model, and trained each model for 10 epochs.\n\nThe two models achieved test perplexity of 31.3 and 28.0 respectively, showing that even in the presence of a large MoE, more computation is still useful. Results are reported at the bottom of Table 7. The larger of the two models has a similar computational budget to the best published model from the literature, and training times are similar. Comparing after 10 epochs, our model has a lower test perplexity by 18%.\n\nD 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS Model Architecture: The models are similar in structure to the 8-million-operations-per-timestep models described in the previous section. We vary the number of experts between models, using an ordinary MoE layer with 32 experts and hierarchical MoE layers with 256, 1024, 4096, 16384, 65536 and 131072 experts. For the hierarchical MoE layers, the first level branching factors are 32, 32, 64, 128, 256 and 256, respectively.\n\nTraining: Models are trained on a cluster of 32 Tesla K40 GPUs, except for the last two models, which are trained on clusters of 64 and 128 GPUs so as to have enough memory for all the parameters. For all models, training batch sizes are approximately 2.5 million words. Models are trained once-through over about 100 billion words. We implement several memory optimizations in order to fit up to 1 billion parameters per GPU. First, we do not store the activations of the hidden layers of the experts, but instead recompute them on the backwards pass. Secondly, we modify the optimizer on the expert parameters to require less auxiliary storage: The Adam optimizer (Kingma & Ba, 2015) keeps first and second moment estimates of the perparameter gradients. This triples the required memory. To avoid keeping a first-moment estimator, we set β1 = 0. To reduce the size of the second moment estimator, we replace it with a factored approximation. For a matrix of parameters, instead of maintaining a full matrix of second-moment estimators, we maintain vectors of row-wise and column-wise averages of that matrix. At each step, the matrix of estimators is taken to be the outer product of those two vectors divided by the mean of either one. This technique could similarly be applied to Adagrad (Duchi et al., 2010).\n\nModel Test Test ops/timestep #Params excluding Total TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params per GPU .1 epochs 1 epoch (millions) (billions) (observed) Kneser-Ney 5-gram 67.1 45.3 0.00001 76.0 4xLSTM-512 54.5 47.0 8.4 8.4 0.1 1.23 MoE-32 48.5 40.4 8.4 37.8 0.1 0.83 MoE-256-h 42.8 35.3 8.4 272.9 0.4 1.11 MoE-1024-h 40.3 32.7 8.5 1079.0 1.2 1.14 MoE-4096-h 38.9 30.9 8.6 4303.4 4.4 1.07 MoE-16384-h 38.2 29.7 8.8 17201.0 17.3 0.96 MoE-65536-h 38.2 28.9 9.2 68791.0 68.9 0.72 MoE-131072-h 39.8 29.2 9.7 137577.6 137.7 0.30\n\nResults: We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8. Perplexity after 100 billion training words is 39% lower for the 68-billion-parameter MoE model than for the baseline model. It is notable that the measured computational efficiency of the largest model (0.30 TFLOPS/GPU) is very low compared to the other models. This is likely a result of the fact that, for purposes of comparison to the other models, we did not increase the training batch size proportionally to the number of GPUs. For comparison, we include results for a computationally matched baseline model consisting of 4 LSTMs, and for an unpruned 5-gram model with Kneser-Ney smoothing (Kneser & Ney, 1995).4\n\nE Machine Translation - Experimental Details\n\nModel Architecture for Single Language Pair MoE Models: Our model is a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decrease the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We insert MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). We use an attention mechanism between the encoder and decoder, with the first decoder LSTM receiving output from and providing input for the attention 5. All of the layers in our model have input and output dimensionality of 512. Our LSTM layers have 2048 hidden units, with a 512-dimensional output projection. We add residual connections around all LSTM and MoE layers to encourage gradient flow (He et al., 2015). Similar to GNMT, to effectively deal with rare words, we used subword units (also known as \"wordpieces\") (Schuster & Nakajima, 2012) for inputs and outputs in our system.\n\nWe use a shared source and target vocabulary of 32K wordpieces. We also used the same beam search technique as proposed in (Wu et al., 2016).\n\nWe train models with different numbers of experts in the MoE layers. In addition to a baseline model with no MoE layers, we train models with flat MoE layers containing 32 experts, and models with hierarchical MoE layers containing 512 and 2048 experts. The flat MoE layers use k = 4 and the hierarchical MoE models use k = 2 at each level of the gating network. Thus, each input is processed by exactly 4 experts in each MoE layer. Each expert in the MoE layer is a feed forward network with one hidden layer of size 2048 and ReLU activation. Thus, each expert contains [512 ∗ 2048] + [2048 ∗ 512] = 2M parameters. The output of the MoE layer is passed through a sigmoid function. We use the strictly-balanced gating function described in Appendix F.\n\nModel Architecture for Multilingual MoE Model: We used the same model architecture as for the single-language-pair models, with the following exceptions: We used noisy-top-k gating as described in Section 2.1, not the scheme from Appendix F. The MoE layers in the encoder and decoder are non-hierarchical MoEs with n = 512 experts, and k = 2. Each expert has a larger hidden layer of size 8192. This doubles the amount of computation in the MoE layers, raising the computational budget of the entire model from 85M to 102M ops/timestep. Training: We trained our networks using the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 2000 training steps, held constant for an additional 8000 steps, and decreased after that so as to be proportional to the inverse square root of the step number. For the single-language-pair models, similarly to (Wu et al., 2016), we applied dropout (Zaremba et al., 2014) to the output of all embedding, LSTM and MoE layers, using DropP rob = 0.4. Training was done synchronously on a cluster of up to 64 GPUs as described in section 3. Each training batch consisted of a set of sentence pairs containing roughly 16000 words per GPU.\n\nTo ensure balanced expert utilization we set wimportance = 0.01 and wload = 0.01, as described in Section 4 and Appendix A. Metrics: We evaluated our models using the perplexity and the standard BLEU score metric. We reported tokenized BLEU score as computed by the multi-bleu.pl script, downloaded from the public implementation of Moses (on Github), which was also used in (Luong et al., 2015a). Results: Tables 2, 3 and 4 in Section 5.3 show comparisons of our results to other published methods. Figure 4 shows test perplexity as a function of number of words in the (training data's) source sentences processed for models with different numbers of experts. As can be seen from the Figure, as we increased the number of experts to approach 2048, the test perplexity of our model continued to improve.\n\nWe found that the experts indeed become highly specialized by syntax and/or semantics, as can be seen in Table 9. For example, one expert is used when the indefinite article \"a\" introduces the direct object in a verb phrase indicating importance or leadership.\n\nExpert 381 Expert 752 Expert 2004 ... with researchers , ... ... plays a core ... ... with rapidly growing ... ... to innovation . ... plays a critical ... ... under static conditions ... ... tics researchers . ... provides a legislative ... ... to swift ly ... ... the generation of ... ... play a leading ... ... to dras tically ... ... technology innovations is ... ... assume a leadership ... ... the rapid and ... ... technological innovations , ... ... plays a central ... ... the fast est ... ... support innovation throughout ... ... taken a leading ... ... the Quick Method ... ... role innovation will ... ... established a reconciliation ... ... rec urrent ) ... ... research scienti st ... ... played a vital ... ... provides quick access ... ... promoting innovation where ... ... have a central ... ... of volatile organic ... ... ... ...\n\nDue to some peculiarities in our infrastructure which have since been fixed, at the time we ran some of the machine translation experiments, our models ran faster if every expert received exactly the same batch size. To accommodate this, we used a different gating function which we describe below. Recall that we define the softmax gating function to be:\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ Gσ(x) = Sof tmax(x · Wg) (15) Sparse Gating (alternate formulation): To obtain a sparse gating vector, we multiply Gσ(x) component-wise with a sparse mask M(Gσ(x)) and normalize the output. The mask itself is a function of Gσ(x) and specifies which experts are assigned to each input example:\n\n$$(15)$$ $$G(x){i}=\\frac{G{\\sigma}(x){i}M(G{\\sigma}(x)){i}}{\\sum{j=1}^{n}G_{\\sigma}(x){j}M(G{\\sigma}(x))_{j}}\\tag{1}$$ $$(17)$$\n\n$$(16)$$\n\nTop-K Mask: To implement top-k gating in this formulation, we would let M(v) = T opK(v, k), where:\n\n$$T o p K(v,k){i}=\\begin{cases}1&{\\mathrm{if~}}v{i}{\\mathrm{~is~in~the~top~}}k{\\mathrm{~elements~of~}}v.\\ 0&{\\mathrm{otherwise.}}\\end{cases}$$ 0 otherwise. (17) Batchwise Mask: To force each expert to receive the exact same number of examples, we introduce an alternative mask function, Mbatchwise(X, m), which operates over batches of input vectors.\n\nInstead of keeping the top k values per example, we keep the top m values per expert across the training batch, where m = k|X| n, so that each example is sent to an average of k experts.\n\n$$M_{b a t c h w i s e}(X,m){j,i}=\\begin{cases}1&\\text{if}X{j,i}\\text{is in the top}m\\text{values for to expert}i\\ 0&\\text{otherwise}\\end{cases}$$\n\nAs our experiments suggest and also observed in (Ioffe & Szegedy, 2015), using a batchwise function during training (such as Mbatchwise) requires modifications to the inference when we may not have a large batch of examples. Our solution to this is to train a vector T of per-expert threshold values to approximate the effects of the batchwise mask. We use the following mask at inference time:\n\n$$(18)$$ $$M_{threshold}(x,T){i}=\\begin{cases}1&\\text{if}x{i}>T_{i}\\ 0&\\text{otherwise}\\end{cases}\\tag{1}$$ $$(19)$$\n\nTo learn the threshold values, we apply an additional loss at training time which is minimized when the batchwise mask and the threshold mask are identical.\n\n$$L_{batchwise}(X,T,m)=\\sum_{j=1}^{|X|}\\sum_{i=1}^{n}(M_{threshold}(x,T){i}-M{batchwise}(X,m){j,i})(X{j,i}-T_{i})\\tag{20}$$\n\nG ATTENTION FUNCTION The attention mechanism described in GNMT (Wu et al., 2016) involves a learned \"Attention Function\" A(xi, yj ) which takes a \"source vector\" xi and a \"target vector\" yj , and must be computed for every source time step i and target time step j. In GNMT, the attention function is implemented as a feed forward neural network with a hidden layer of size n. It can be expressed as:\n\n$$A_{G N M T}(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}t a n h((x_{i}U){d}+(y{j}W)_{d})$$\n\nWhere U and W are trainable weight matrices and V is a trainable weight vector.\n\nFor performance reasons, in our models, we used a slightly different attention function:\n\n$$A(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}tanh((x_{i}U){d})tanh((y{j}W)_{d})\\tag{22}$$\n\nWith our attention function, we can simultaneously compute the attention function on multiple source time steps and multiple target time steps using optimized matrix multiplications. We found little difference in quality between the two functions.\n\n$$(21)$$",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/1701.06538v1.md"
+          },
+          "headlines": {
+            "1 Introduction And Related Work": [
+              "1.1 Conditional Computation",
+              "1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER",
+              "1.3 Related Work On Mixtures Of Experts"
+            ],
+            "2 The Structure Of The Mixture-Of-Experts Layer": [
+              "2.1 Gating Network"
+            ],
+            "3 Addressing Performance Challenges": [
+              "3.1 The Shrinking Batch Problem",
+              "3.2 Network Bandwidth"
+            ],
+            "4 Balancing Expert Utilization": [],
+            "5 Experiments": [
+              "5.1 1 Billion Word Language Modeling Benchmark",
+              "5.2 100 BILLION WORD GOOGLE NEWS CORPUS",
+              "5.3 Machine Translation (Single Language Pair)",
+              "5.4 Multilingual Machine Translation"
+            ],
+            "6 Conclusion": [],
+            "Appendices": [
+              "A Load-Balancing Loss",
+              "B Hierachical Mixture Of Experts",
+              "C 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS",
+              "D 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS",
+              "E Machine Translation - Experimental Details"
+            ]
+          },
+          "summary": "The paper introduces a Sparsely-Gated Mixture-of-Experts (MoE) layer to enhance neural network capacity without a proportional increase in computation. This approach uses a trainable gating network to select a sparse combination of experts for each input, significantly improving model capacity and performance in language modeling and machine translation tasks. The MoE layer, with up to 137 billion parameters, achieves better results than state-of-the-art models at lower computational costs. The paper addresses challenges like batch size and network bandwidth, proposing solutions like mixing data and model parallelism. Experiments show that MoE models outperform traditional models in various benchmarks, demonstrating the potential of conditional computation in deep learning.",
+          "summary_embedding": [
+            -0.01644345559179783,
+            -0.016995344310998917,
+            -0.016372699290513992,
+            -0.014349107630550861,
+            0.02282555401325226,
+            0.045934125781059265,
+            -0.014646278694272041,
+            0.03563220426440239,
+            -0.004829026758670807,
+            0.005617944523692131,
+            -0.00932550523430109,
+            -0.008143897168338299,
+            0.005575491581112146,
+            0.01538213063031435,
+            -0.005653322208672762,
+            -0.020306676626205444,
+            -0.013832597061991692,
+            -0.023235931992530823,
+            0.028344441205263138,
+            -0.020080260932445526,
+            0.011823155917227268,
+            -0.020943470299243927,
+            -0.025868017226457596,
+            -0.016500059515237808,
+            0.025061409920454025,
+            0.024806693196296692,
+            0.02583971433341503,
+            0.003827843815088272,
+            -0.03492465242743492,
+            0.033651065081357956,
+            0.014674580655992031,
+            0.012488252483308315,
+            -0.031500112265348434,
+            -0.009162768721580505,
+            -0.05351905897259712,
+            -0.05227377265691757,
+            0.04143410921096802,
+            0.022245362401008606,
+            -0.016231190413236618,
+            0.023717066273093224,
+            0.017264213413000107,
+            0.005816058721393347,
+            -0.020632149651646614,
+            -0.020844414830207825,
+            0.003700484987348318,
+            -0.013103820383548737,
+            -0.01983969286084175,
+            -0.013528349809348583,
+            -0.01828308403491974,
+            -0.01577835902571678,
+            0.009551920928061008,
+            0.0002856731880456209,
+            0.0029186424799263477,
+            -0.003424540627747774,
+            -0.047830358147621155,
+            0.0248208437114954,
+            0.019853845238685608,
+            0.03492465242743492,
+            0.04633035510778427,
+            0.013011839240789413,
+            0.01516986545175314,
+            -0.014271277002990246,
+            -0.04021712392568588,
+            0.005497660953551531,
+            -0.00041656987741589546,
+            -0.005380915477871895,
+            -0.005819596350193024,
+            0.008575502783060074,
+            0.008724087849259377,
+            -0.01143400277942419,
+            -0.04202845320105553,
+            0.013952880166471004,
+            -0.020561393350362778,
+            0.00010143388499272987,
+            0.01354957651346922,
+            0.008292482234537601,
+            0.02677368000149727,
+            0.03195294365286827,
+            0.023971782997250557,
+            -0.008724087849259377,
+            -0.01340806670486927,
+            0.019924599677324295,
+            -0.0022676968947052956,
+            0.007018893025815487,
+            0.04570771008729935,
+            -0.04033033177256584,
+            0.02163686975836754,
+            -0.025726506486535072,
+            -0.015339677222073078,
+            -0.005349075887352228,
+            -0.022471778094768524,
+            0.02806142158806324,
+            -0.007344366051256657,
+            0.017801949754357338,
+            0.017094401642680168,
+            0.01594817079603672,
+            0.013330236077308655,
+            -0.001894464367069304,
+            -0.022938761860132217,
+            0.0018360915128141642,
+            -0.01541043259203434,
+            0.02114158496260643,
+            0.0015230007702484727,
+            0.005469358991831541,
+            0.027594437822699547,
+            0.031160488724708557,
+            -0.023490650579333305,
+            0.009502392262220383,
+            0.02422650158405304,
+            0.06967949122190475,
+            0.015424583107233047,
+            -0.006435164716094732,
+            -0.0024640418123453856,
+            0.04143410921096802,
+            0.0015513027319684625,
+            0.0038349193055182695,
+            -0.06826439499855042,
+            -0.03022652305662632,
+            -0.0011365016689524055,
+            -0.02254253253340721,
+            0.02227366529405117,
+            0.01002597901970148,
+            -0.009750034660100937,
+            -0.022457627579569817,
+            -0.04516997188329697,
+            0.05190584436058998,
+            -0.00545874610543251,
+            -0.016259491443634033,
+            -0.005554265342652798,
+            -0.05340585112571716,
+            0.00039136342820711434,
+            0.015042506158351898,
+            -0.018481198698282242,
+            -0.03670767694711685,
+            0.036141637712717056,
+            0.0007398316520266235,
+            0.03741522878408432,
+            0.012459950521588326,
+            -0.01708024926483631,
+            -0.020094411447644234,
+            0.022259512916207314,
+            -0.011639192700386047,
+            -0.0038313816767185926,
+            -0.027339721098542213,
+            0.0065236082300543785,
+            -0.028103873133659363,
+            -0.0014566679019480944,
+            0.06730213016271591,
+            -0.04816998168826103,
+            0.02946236915886402,
+            -0.029660483822226524,
+            0.023660462349653244,
+            -0.008575502783060074,
+            0.0278916098177433,
+            -0.010238244198262691,
+            0.005840822588652372,
+            -0.013457595370709896,
+            -0.0010586712742224336,
+            -0.020745357498526573,
+            0.0034210027661174536,
+            -0.016259491443634033,
+            -0.027750099077820778,
+            -0.007054270710796118,
+            0.031273696571588516,
+            0.01698119193315506,
+            0.025655752047896385,
+            0.025429336354136467,
+            0.01246702577918768,
+            -0.04935866594314575,
+            0.02595292218029499,
+            0.0018926954362541437,
+            0.013514199294149876,
+            0.013181651011109352,
+            -0.0010418669553473592,
+            0.025599148124456406,
+            0.012049571610987186,
+            0.025061409920454025,
+            0.037443529814481735,
+            0.0027682881336659193,
+            0.04737752676010132,
+            -0.015693452209234238,
+            -0.005292471963912249,
+            0.028528403490781784,
+            0.000505455827806145,
+            0.02465103194117546,
+            -0.001686621573753655,
+            0.013330236077308655,
+            0.000776978034991771,
+            0.03834919258952141,
+            -0.006339645478874445,
+            0.029518973082304,
+            0.021608568727970123,
+            0.028316138312220573,
+            -0.014759486541152,
+            0.023122724145650864,
+            -0.019754787907004356,
+            -0.02292460948228836,
+            0.035547297447919846,
+            0.004082561936229467,
+            0.01886327564716339,
+            -0.028457649052143097,
+            -0.030679354444146156,
+            0.015452885068953037,
+            -0.01855195313692093,
+            0.016556663438677788,
+            0.03639635816216469,
+            0.0008273909334093332,
+            -0.0025719432160258293,
+            0.026292547583580017,
+            -0.04245298355817795,
+            0.012856177985668182,
+            -0.01532552670687437,
+            -0.02855670638382435,
+            -0.019528372213244438,
+            -0.00875946506857872,
+            -0.0018095583654940128,
+            -0.013691086322069168,
+            -0.0037570889107882977,
+            -0.007563706487417221,
+            -0.02459442801773548,
+            0.00935380719602108,
+            0.013627407141029835,
+            0.009849091991782188,
+            -0.018665160983800888,
+            0.0019952901639044285,
+            -0.008257105015218258,
+            -0.0160047747194767,
+            -0.03752843663096428,
+            0.015268922783434391,
+            -0.0024888061452656984,
+            0.009191070683300495,
+            0.04078316316008568,
+            0.028485950082540512,
+            -0.005055442452430725,
+            0.0329718142747879,
+            -0.033707667142152786,
+            0.017915157601237297,
+            -0.008320784196257591,
+            0.009191070683300495,
+            0.002989397384226322,
+            -0.025542544201016426,
+            0.027424626052379608,
+            -0.03792466223239899,
+            -0.038151077926158905,
+            0.022995365783572197,
+            0.011858533136546612,
+            0.0017096169758588076,
+            -0.019386861473321915,
+            -0.017717044800519943,
+            0.0008499440737068653,
+            0.002752368338406086,
+            -0.018834972754120827,
+            -0.01019579079002142,
+            0.02227366529405117,
+            -0.025500090792775154,
+            0.036085035651922226,
+            0.023391593247652054,
+            -0.03520767390727997,
+            -0.00608492735773325,
+            -0.01903308741748333,
+            0.059037946164608,
+            -0.0011506526498124003,
+            -0.010301923379302025,
+            0.022938761860132217,
+            -0.027268964797258377,
+            -0.004524780437350273,
+            0.022301966324448586,
+            0.01828308403491974,
+            -0.008278331719338894,
+            0.003983504604548216,
+            -0.022287815809249878,
+            -0.05281151086091995,
+            -0.010577867738902569,
+            -0.0003305583668407053,
+            0.054368119686841965,
+            -0.012367968447506428,
+            0.0020695829298347235,
+            -0.006413938011974096,
+            0.004100250545889139,
+            0.03704730048775673,
+            0.002140337834134698,
+            0.0025082635693252087,
+            -0.013698162510991096,
+            0.0371888130903244,
+            0.022316116839647293,
+            0.005883275531232357,
+            -0.012162779457867146,
+            0.005377377849072218,
+            -0.04899074137210846,
+            -0.0009118546731770039,
+            0.005741765722632408,
+            0.007386818993836641,
+            -0.025910470634698868,
+            0.017717044800519943,
+            0.009205221198499203,
+            -0.0024693484883755445,
+            -0.009792488068342209,
+            0.01173117384314537,
+            0.025429336354136467,
+            -0.019938750192523003,
+            -0.05700020492076874,
+            0.0034705311991274357,
+            0.012014193460345268,
+            -0.004054259974509478,
+            -0.019401011988520622,
+            0.006792477332055569,
+            0.0021934041287750006,
+            0.022698193788528442,
+            -0.011271266266703606,
+            -0.0029221803415566683,
+            0.040585052222013474,
+            0.009516543708741665,
+            -0.0016813150141388178,
+            -0.017717044800519943,
+            -0.011794853024184704,
+            0.018070818856358528,
+            0.004740582779049873,
+            0.0035289041697978973,
+            -0.01751893013715744,
+            -0.029264256358146667,
+            0.02800481766462326,
+            0.019967053085565567,
+            -0.048905834555625916,
+            -0.008200501091778278,
+            -0.019273653626441956,
+            0.0019599127117544413,
+            0.008695785887539387,
+            0.04613224044442177,
+            -0.00590803986415267,
+            0.008313708938658237,
+            -0.020179318264126778,
+            -0.05003791302442551,
+            -0.02708500251173973,
+            0.03707560524344444,
+            -0.028684064745903015,
+            -0.04276430234313011,
+            -0.010252394713461399,
+            0.016740625724196434,
+            -0.009403335861861706,
+            -0.018679311498999596,
+            -0.011738249100744724,
+            -0.04969828948378563,
+            -0.003497064346447587,
+            0.0337359681725502,
+            0.017264213413000107,
+            -0.008441068232059479,
+            -0.033877480775117874,
+            0.019641580060124397,
+            -0.007160402834415436,
+            -0.006474079564213753,
+            0.0007862646016292274,
+            0.008794843219220638,
+            0.02400008589029312,
+            0.011879759840667248,
+            -0.02763689123094082,
+            -0.03427370637655258,
+            -0.0024888061452656984,
+            0.014929298311471939,
+            -0.00878776703029871,
+            0.0245378240942955,
+            0.024679332971572876,
+            -0.022995365783572197,
+            -0.01654251106083393,
+            0.015750056132674217,
+            -0.05779265984892845,
+            -0.010287772864103317,
+            -0.029235953465104103,
+            -0.006183984223753214,
+            0.0009613831643946469,
+            0.0061096916906535625,
+            0.00012304731353651732,
+            0.03846240043640137,
+            0.008801918476819992,
+            0.006014172453433275,
+            0.009191070683300495,
+            0.01913214474916458,
+            0.028471799567341805,
+            0.009155693463981152,
+            -0.03517937287688255,
+            0.02850010246038437,
+            -0.017051948234438896,
+            0.007811348885297775,
+            -0.025754809379577637,
+            -0.013033065013587475,
+            0.00443633645772934,
+            0.02725481428205967,
+            -0.000503244751598686,
+            0.018849123269319534,
+            -0.021919889375567436,
+            0.037500135600566864,
+            -0.029915200546383858,
+            -0.05278320610523224,
+            -0.02005195803940296,
+            -0.008285406976938248,
+            0.015764206647872925,
+            0.017278363928198814,
+            -0.019924599677324295,
+            -0.0286557637155056,
+            -0.021750077605247498,
+            0.006470541935414076,
+            0.010542490519583225,
+            -0.021099131554365158,
+            -0.010627396404743195,
+            0.01502835564315319,
+            -0.00404364662244916,
+            -0.01886327564716339,
+            0.013684011064469814,
+            0.015693452209234238,
+            0.03169822692871094,
+            -0.03308502212166786,
+            0.004287751391530037,
+            -0.025967074558138847,
+            -0.04327373951673508,
+            0.004096712917089462,
+            0.03166992589831352,
+            0.027509532868862152,
+            -0.018636859953403473,
+            0.010754755698144436,
+            0.003990580327808857,
+            0.004234685096889734,
+            -0.0033803186379373074,
+            0.023603858426213264,
+            0.03291521221399307,
+            0.02763689123094082,
+            -0.011653343215584755,
+            0.004921008367091417,
+            -0.01559439580887556,
+            -0.0011108530452474952,
+            0.01865101046860218,
+            -0.017816102132201195,
+            0.033877480775117874,
+            -0.029405765235424042,
+            0.02104252763092518,
+            -0.035122767090797424,
+            0.006141531281173229,
+            -0.012099100276827812,
+            -0.00404364662244916,
+            0.04833979532122612,
+            -0.017646290361881256,
+            -0.0019546060357242823,
+            -0.008101443760097027,
+            -0.014575524255633354,
+            -0.02158026583492756,
+            0.008766540326178074,
+            -0.04033033177256584,
+            -0.012367968447506428,
+            -0.028429346159100533,
+            0.003424540627747774,
+            0.019910449162125587,
+            -0.06277380883693695,
+            -0.06237758323550224,
+            0.012275987304747105,
+            -0.04918885603547096,
+            -0.026419905945658684,
+            0.02963218092918396,
+            -0.002984090941026807,
+            0.0012780115939676762,
+            -0.0024003623984754086,
+            0.0068101659417152405,
+            -0.029207652434706688,
+            0.00425944896414876,
+            -0.000887974863871932,
+            0.03271709755063057,
+            -0.021056678146123886,
+            -0.008122670464217663,
+            0.004878554958850145,
+            -0.06860402226448059,
+            -0.028344441205263138,
+            0.011271266266703606,
+            -0.009459939785301685,
+            -0.0066615804098546505,
+            -0.017915157601237297,
+            -0.004383270163089037,
+            0.0008950503543019295,
+            0.0032830305863171816,
+            0.05801907554268837,
+            -0.01983969286084175,
+            -0.053773775696754456,
+            0.03633975237607956,
+            0.00887267291545868,
+            0.029603879898786545,
+            -0.025556694716215134,
+            -0.017533080652356148,
+            -0.01855195313692093,
+            -0.020504789426922798,
+            -0.04123599827289581,
+            0.00404364662244916,
+            -0.005918653216212988,
+            0.009212297387421131,
+            -0.0009693430620245636,
+            0.02763689123094082,
+            0.029235953465104103,
+            0.023108573630452156,
+            0.012750045396387577,
+            0.01079013291746378,
+            -0.022372720763087273,
+            -0.021594416350126266,
+            0.0012222920777276158,
+            0.04683978855609894,
+            0.024636881425976753,
+            -0.02675952948629856,
+            -0.011504758149385452,
+            0.03953787684440613,
+            0.0419718474149704,
+            -0.01974063739180565,
+            -0.02276895008981228,
+            0.00764861237257719,
+            0.03266049548983574,
+            -0.04477374628186226,
+            0.005932804197072983,
+            0.03410389646887779,
+            0.014405711553990841,
+            -0.005356151144951582,
+            -0.010294848121702671,
+            0.022570835426449776,
+            0.03152841329574585,
+            -0.008724087849259377,
+            0.01876421831548214,
+            0.016910437494516373,
+            0.029349161311984062,
+            -0.0067889392375946045,
+            0.0005766529939137399,
+            -0.024735936895012856,
+            0.055188875645399094,
+            -0.03059444949030876,
+            0.02909444272518158,
+            0.010316074825823307,
+            -0.00593634182587266,
+            0.039056744426488876,
+            -0.016641568392515182,
+            -0.0431322306394577,
+            0.01913214474916458,
+            -0.006979977712035179,
+            -0.0055613406002521515,
+            -0.03444352000951767,
+            -0.007592008449137211,
+            -0.007507102098315954,
+            -0.006714646704494953,
+            -0.022556684911251068,
+            -0.02606613002717495,
+            0.004415110219269991,
+            0.00973588414490223,
+            -0.006484692916274071,
+            0.00698705343529582,
+            0.02092931978404522,
+            -0.013507124036550522,
+            0.02525952458381653,
+            -0.046047333627939224,
+            -0.0008366775000467896,
+            -0.0038596836384385824,
+            -0.025712355971336365,
+            -0.0345001220703125,
+            -0.0005770951975136995,
+            -0.0024251264985650778,
+            -0.024622729048132896,
+            -0.00914154201745987,
+            -0.008752389810979366,
+            -0.0023119186516851187,
+            -0.004938696976751089,
+            -0.055188875645399094,
+            0.008695785887539387,
+            0.010846736840903759,
+            -0.002713453024625778,
+            0.0055507272481918335,
+            -0.010613245889544487,
+            -0.03039633482694626,
+            0.03039633482694626,
+            0.04129260033369064,
+            0.033707667142152786,
+            0.014334957115352154,
+            -0.0074646491557359695,
+            -0.017165156081318855,
+            0.006060163024812937,
+            0.021721776574850082,
+            0.00902833417057991,
+            0.007846726104617119,
+            0.03877372294664383,
+            0.00011331850691931322,
+            -0.008349086157977581,
+            -0.009764186106622219,
+            0.028740668669342995,
+            -0.001085204305127263,
+            0.023603858426213264,
+            0.050802066922187805,
+            0.030424637719988823,
+            0.020589696243405342,
+            -0.01681138016283512,
+            -0.02010856196284294,
+            -0.00851889792829752,
+            -0.02855670638382435,
+            0.0364246591925621,
+            0.012127402238547802,
+            -0.005232329946011305,
+            -0.011540135368704796,
+            0.019061388447880745,
+            -0.033339742571115494,
+            -0.0037358624394983053,
+            0.00048422932741232216,
+            0.013924578204751015,
+            -0.03379257395863533,
+            0.0002480846014805138,
+            0.031613320112228394,
+            0.014957600273191929,
+            -0.016995344310998917,
+            -0.0188774261623621,
+            0.010549565777182579,
+            0.020462337881326675,
+            0.018948180601000786,
+            0.00037743354914709926,
+            0.00897173024713993,
+            -0.00022121983056422323,
+            -0.0005319889169186354,
+            0.01590571738779545,
+            -0.007952858693897724,
+            0.036254845559597015,
+            -0.003525366308167577,
+            -0.01325240544974804,
+            -0.0037500134203583,
+            -0.021438755095005035,
+            -0.024467067793011665,
+            0.043556760996580124,
+            0.023971782997250557,
+            0.04661337286233902,
+            -0.005660397466272116,
+            -0.004245297983288765,
+            0.025825563818216324,
+            0.008221727795898914,
+            -0.03288691118359566,
+            0.000265110022155568,
+            0.005281858611851931,
+            -0.02892463095486164,
+            0.016641568392515182,
+            -0.02476423978805542,
+            -0.016896286979317665,
+            -0.006771250627934933,
+            -0.0015557248843833804,
+            -0.028188779950141907,
+            0.01703779585659504,
+            0.0031309074256569147,
+            -0.0027099153958261013,
+            -0.039764292538166046,
+            0.006880920846015215,
+            0.020419884473085403,
+            0.015311375260353088,
+            0.0037358624394983053,
+            0.011525984853506088,
+            -0.016613267362117767,
+            0.002126186853274703,
+            0.03379257395863533,
+            0.004878554958850145,
+            0.02005195803940296,
+            0.0027647505048662424,
+            0.023773670196533203,
+            0.03633975237607956,
+            -0.01681138016283512,
+            -0.02189158834517002,
+            0.01519816741347313,
+            -0.02233026921749115,
+            0.029943503439426422,
+            0.004889168310910463,
+            -0.012283062562346458,
+            0.022033097222447395,
+            -0.009827865287661552,
+            -0.0027346794959157705,
+            0.01659911684691906,
+            -0.022740647196769714,
+            -0.017717044800519943,
+            0.03183973580598831,
+            -0.039452970027923584,
+            0.012856177985668182,
+            -0.008547200821340084,
+            0.015863263979554176,
+            -0.023066120222210884,
+            0.011172209866344929,
+            0.007100261282175779,
+            -0.01573590561747551,
+            0.010797208175063133,
+            0.009693430736660957,
+            0.00545874610543251,
+            0.0018926954362541437,
+            0.02179253101348877,
+            -0.0006049550138413906,
+            -0.011257115751504898,
+            -0.018155725672841072,
+            -0.004694592207670212,
+            -0.002614396158605814,
+            -0.016556663438677788,
+            0.01865101046860218,
+            -0.041066184639930725,
+            -0.013280707411468029,
+            -0.008879749104380608,
+            -0.013988257385790348,
+            -0.010011828504502773,
+            0.024184048175811768,
+            0.025669902563095093,
+            -0.002361447084695101,
+            -0.031217092648148537,
+            0.01790100708603859,
+            0.024198200553655624,
+            0.003654494183138013,
+            -0.023646311834454536,
+            0.0302831269800663,
+            -0.008936353027820587,
+            0.014971751719713211,
+            0.026136886328458786,
+            0.014094389975070953,
+            0.004595535341650248,
+            -0.010500037111341953,
+            -0.02114158496260643,
+            0.01481609046459198,
+            0.010507113300263882,
+            0.033537857234478,
+            0.014745336025953293,
+            0.02071705460548401,
+            0.01594817079603672,
+            -0.023320838809013367,
+            -0.011214662343263626,
+            -0.003482913365587592,
+            -0.022514231503009796,
+            0.03093407303094864,
+            0.0006129149114713073,
+            0.02114158496260643,
+            0.026618018746376038,
+            -0.03319823369383812,
+            -0.01187268365174532,
+            0.032688796520233154,
+            0.02779255248606205,
+            0.05108508840203285,
+            0.025316128507256508,
+            0.015396281145513058,
+            0.027226513251662254,
+            -0.03585861995816231,
+            -0.036368053406476974,
+            0.001548649393953383,
+            0.04256619140505791,
+            -0.008426916785538197,
+            0.00967928022146225,
+            0.03900013864040375,
+            -0.006781863979995251,
+            -0.01611798256635666,
+            0.019457615911960602,
+            0.0026710000820457935,
+            0.002448122017085552,
+            -0.0033113325480371714,
+            -0.01335146278142929,
+            0.03175482898950577,
+            -0.0051474240608513355,
+            0.013570803217589855,
+            -0.019273653626441956,
+            0.027113303542137146,
+            0.0001799829478841275,
+            -0.003229964291676879,
+            0.028344441205263138,
+            -0.002660386962816119,
+            -0.051990751177072525,
+            0.024212351068854332,
+            0.019811391830444336,
+            -0.011561362072825432,
+            -0.0036332677118480206,
+            -0.003792466362938285,
+            0.04567940905690193,
+            0.034698236733675,
+            0.029009537771344185,
+            -0.0020501252729445696,
+            0.0168679840862751,
+            0.018792519345879555,
+            0.0053667644970119,
+            0.014278353191912174,
+            -0.0017025414854288101,
+            -0.007853801362216473,
+            0.021056678146123886,
+            -0.020787810906767845,
+            0.03226426616311073,
+            -0.016570813953876495,
+            -0.020122714340686798,
+            0.008738238364458084,
+            -0.011172209866344929,
+            -0.0009083169279620051,
+            -0.006311343517154455,
+            -0.0008813415770418942,
+            -0.0029664020985364914,
+            0.016938740387558937,
+            -0.004446949809789658,
+            0.04661337286233902,
+            -0.013931653462350368,
+            -0.008808993734419346,
+            0.006972902454435825,
+            -0.025981225073337555,
+            0.0013346156338229775,
+            0.0028726516757160425,
+            0.027000095695257187,
+            -0.01989629678428173,
+            -0.00970758218318224,
+            0.01606137864291668,
+            0.06039644405245781,
+            -0.008639181964099407,
+            -0.0026267783250659704,
+            0.027226513251662254,
+            -0.002787745790556073,
+            -0.03551899641752243,
+            0.012919857166707516,
+            -0.04494355618953705,
+            0.017589686438441277,
+            0.026349151507019997,
+            -0.04986810311675072,
+            -0.022089701145887375,
+            -0.02347649820148945,
+            -0.02043403498828411,
+            0.023434046655893326,
+            -0.009976450353860855,
+            0.013867974281311035,
+            -0.017801949754357338,
+            -0.009014182724058628,
+            0.020518941804766655,
+            0.009176919236779213,
+            0.03735862299799919,
+            -0.027551984414458275,
+            -0.016783079132437706,
+            -0.027820853516459465,
+            -0.04304732382297516,
+            0.009558996185660362,
+            -0.0278916098177433,
+            0.0027099153958261013,
+            -0.0016291331266984344,
+            -0.040302030742168427,
+            -0.037981268018484116,
+            0.005278320983052254,
+            -0.017985913902521133,
+            0.01516986545175314,
+            0.020915169268846512,
+            0.02731141820549965,
+            -0.0020253609400242567,
+            -0.030198222026228905,
+            0.029235953465104103,
+            -0.007344366051256657,
+            0.03223596513271332,
+            -0.00689507182687521,
+            0.005066055804491043,
+            -0.0070436573587358,
+            0.031160488724708557,
+            -0.023009516298770905,
+            0.018792519345879555,
+            -0.026801982894539833,
+            -0.004086099565029144,
+            -0.018990634009242058,
+            -0.01774534583091736,
+            0.004224071744829416,
+            0.004521242808550596,
+            7.888073741924018e-05,
+            0.0048750173300504684,
+            0.005777143407613039,
+            0.024806693196296692,
+            0.0021421066485345364,
+            0.02589631825685501,
+            -0.027877457439899445,
+            -0.025316128507256508,
+            0.01719345711171627,
+            -0.02077365852892399,
+            0.005678086541593075,
+            -0.006261814851313829,
+            -0.0302831269800663,
+            -0.01202834490686655,
+            0.023391593247652054,
+            0.02162271924316883,
+            -0.005490585695952177,
+            0.002644466934725642,
+            0.02206140011548996,
+            -0.026023678481578827,
+            0.00437973253428936,
+            -0.03781145438551903,
+            -0.018396291881799698,
+            0.029773691669106483,
+            -0.03195294365286827,
+            0.020745357498526573,
+            0.003185742534697056,
+            0.03483974561095238,
+            -0.017292514443397522,
+            -0.01947176828980446,
+            0.01654251106083393,
+            -0.01008258294314146,
+            0.004142703488469124,
+            0.01606137864291668,
+            0.01616043597459793,
+            -0.0015070808585733175,
+            0.05380208045244217,
+            0.0008167776977643371,
+            0.006909222807735205,
+            -0.049075648188591,
+            0.004496478475630283,
+            -0.044858649373054504,
+            0.0002768288250081241,
+            -0.013337312266230583,
+            -0.02514631673693657,
+            0.0014027173165231943,
+            0.001001182827167213,
+            0.001963450340554118,
+            -0.01913214474916458,
+            -0.01989629678428173,
+            0.011596739292144775,
+            -0.015863263979554176,
+            0.004928083624690771,
+            -0.06418890506029129,
+            -0.01855195313692093,
+            -0.004974074196070433,
+            0.0043337419629096985,
+            0.0007699025445617735,
+            0.03342464938759804,
+            -0.01478778850287199,
+            0.010117961093783379,
+            -0.01549533847719431,
+            -0.026363302022218704,
+            0.04010391607880592,
+            0.018750067800283432,
+            -0.023773670196533203,
+            0.024693485349416733,
+            0.012106175534427166,
+            0.03099067695438862,
+            -0.0318114347755909,
+            0.020349130034446716,
+            0.018523650243878365,
+            -0.0029823218937963247,
+            0.006551910191774368,
+            0.008575502783060074,
+            0.01832553744316101,
+            -0.015764206647872925,
+            -0.0003632825391832739,
+            0.011419852264225483,
+            0.010033054277300835,
+            0.031924642622470856,
+            0.01403071079403162,
+            -0.03183973580598831,
+            0.019047237932682037,
+            0.01616043597459793,
+            0.02298121340572834,
+            0.033820874989032745,
+            -0.027481229975819588,
+            0.038575608283281326,
+            -0.010386829264461994,
+            -0.024042539298534393,
+            0.020801961421966553,
+            -0.032688796520233154,
+            -0.006679269019514322,
+            0.030735958367586136,
+            0.04398128762841225,
+            -0.0015566092915832996,
+            -0.044632233679294586,
+            0.01535382866859436,
+            0.01316749956458807,
+            0.0027806703001260757,
+            0.028811423107981682,
+            0.011264191009104252,
+            0.021820832043886185,
+            0.025273675099015236,
+            0.011419852264225483,
+            -0.0037252490874379873,
+            0.024962352588772774,
+            -0.008101443760097027,
+            -0.004446949809789658,
+            -0.0007035697344690561,
+            0.03206615149974823,
+            0.014702882617712021,
+            -0.017603836953639984,
+            -0.036962397396564484,
+            -0.007620310410857201,
+            0.009452863596379757,
+            -0.007903330028057098,
+            -0.027127455919981003,
+            -0.03950957581400871,
+            -0.026787832379341125,
+            -0.005667473189532757,
+            0.01505665760487318,
+            -0.06724552065134048,
+            -0.007620310410857201,
+            -0.012233533896505833,
+            0.017717044800519943,
+            0.0022906921803951263,
+            0.0181132722645998,
+            0.001277127186767757,
+            -0.009403335861861706,
+            0.004209920763969421,
+            0.050802066922187805,
+            -0.03059444949030876,
+            0.014242975041270256,
+            0.011292492970824242,
+            0.0032105068676173687,
+            0.002787745790556073,
+            0.038688816130161285,
+            -0.010988246649503708,
+            -0.030424637719988823,
+            0.011575513519346714,
+            -0.017292514443397522,
+            0.003383856499567628,
+            0.05006621778011322,
+            -0.009481165558099747,
+            -0.027495380491018295,
+            0.020915169268846512,
+            -0.009148617275059223,
+            -0.015523640438914299,
+            -0.0027258351910859346,
+            -0.006035399157553911,
+            0.0006283925613388419,
+            -0.03832089155912399,
+            -0.015750056132674217,
+            0.03212275728583336,
+            0.008617955259978771,
+            0.014603826217353344,
+            0.025344429537653923,
+            0.010209942236542702,
+            -0.030481241643428802,
+            0.027481229975819588,
+            -0.0005678086308762431,
+            0.005221716593950987,
+            0.013323160819709301,
+            0.03424540534615517,
+            -0.014264201745390892,
+            -0.020264223217964172,
+            -0.040245428681373596,
+            -0.013853822834789753,
+            -0.0463869571685791,
+            0.001943992800079286,
+            0.006919836159795523,
+            -0.012919857166707516,
+            -0.04366996884346008,
+            -0.0329718142747879,
+            0.021127434447407722,
+            0.03178313374519348,
+            -0.017023645341396332,
+            0.0025294902734458447,
+            -0.008483520708978176,
+            -0.007882103323936462,
+            -0.0017520699184387922,
+            -0.013217028230428696,
+            -0.006378560326993465,
+            -0.006229975260794163,
+            0.015452885068953037,
+            -0.004652139265090227,
+            -0.0036934094969183207,
+            -0.0498398020863533,
+            -0.018778368830680847,
+            0.004082561936229467,
+            -0.008221727795898914,
+            0.007323139347136021,
+            -0.006955213379114866,
+            -0.002727604005485773,
+            -0.012474101036787033,
+            -0.03829259052872658,
+            0.024184048175811768,
+            0.017278363928198814,
+            0.0035501306410878897,
+            -0.008922201581299305,
+            -0.018367990851402283,
+            0.03294351324439049,
+            0.012523629702627659,
+            -0.017547233030200005,
+            -0.018580254167318344,
+            0.025981225073337555,
+            0.026957644149661064,
+            0.008122670464217663,
+            -0.006137993652373552,
+            0.029886899515986443,
+            0.005695775151252747,
+            -0.022938761860132217,
+            -0.003983504604548216,
+            -0.005055442452430725,
+            -0.0038844477385282516,
+            0.020561393350362778,
+            -0.010556641966104507,
+            0.0300001073628664,
+            0.038717120885849,
+            0.012870329432189465,
+            0.0039198254235088825,
+            -0.0033714743331074715,
+            -0.008922201581299305,
+            0.015976471826434135,
+            0.006481155287474394,
+            0.008525974117219448,
+            0.014759486541152,
+            -0.016358548775315285,
+            -0.020405733957886696,
+            0.006215824279934168,
+            -0.007107336539775133,
+            0.020122714340686798,
+            0.01708024926483631,
+            0.016429303213953972,
+            0.0007610581233166158,
+            -0.028287837281823158,
+            -0.0018024828750640154,
+            0.02411329373717308,
+            0.02010856196284294,
+            -0.004655676893889904,
+            -0.023264234885573387,
+            0.014398636296391487,
+            -0.015806660056114197,
+            -0.009969375096261501,
+            0.0033431723713874817,
+            0.006229975260794163,
+            -0.003083147807046771,
+            0.0571134127676487,
+            -0.020193468779325485,
+            -0.011419852264225483,
+            -0.010287772864103317,
+            -0.010867963545024395,
+            0.0012072566896677017,
+            -0.01828308403491974,
+            0.008964654989540577,
+            0.025613298639655113,
+            -0.009764186106622219,
+            0.03568880632519722,
+            0.0228114016354084,
+            0.006912760436534882,
+            0.005260631907731295,
+            -0.027070851996541023,
+            0.012304289266467094,
+            -0.016047226265072823,
+            0.005660397466272116,
+            0.018467046320438385,
+            0.017320817336440086,
+            0.0076132346875965595,
+            0.026306698098778725,
+            0.019386861473321915,
+            0.016684021800756454,
+            -0.022783100605010986,
+            0.025457637384533882,
+            -0.007832575589418411,
+            0.027877457439899445,
+            -0.01611798256635666,
+            -0.002253545681014657,
+            -0.012990612536668777,
+            0.008681634441018105,
+            0.002416282193735242,
+            -0.0019351483788341284,
+            0.00036129256477579474,
+            0.016783079132437706,
+            -0.014589674770832062,
+            0.01062032114714384,
+            0.013316085562109947,
+            -0.027919910848140717,
+            -0.010372678749263287,
+            0.016344398260116577,
+            -0.00668988237157464,
+            0.006877383217215538,
+            -0.011865608394145966,
+            -0.0017202302115038037,
+            0.011009473353624344,
+            -0.0436982698738575,
+            0.0104929618537426,
+            0.006279503460973501,
+            -0.015155714936554432,
+            0.014801939949393272,
+            -0.009799563325941563,
+            -0.008235878311097622,
+            -0.013592029921710491,
+            0.003792466362938285,
+            -0.00030225637601688504,
+            -0.0018290160223841667,
+            -0.00022995365725364536,
+            0.00389859895221889,
+            0.010047205723822117,
+            -0.0045990729704499245,
+            -0.010450509376823902,
+            -0.0028638073708862066,
+            -0.02265574224293232,
+            -0.0010047205723822117,
+            0.018735915422439575,
+            -0.004591997712850571,
+            0.0005222601466812193,
+            0.0145189194008708,
+            -0.005830209702253342,
+            0.010556641966104507,
+            -0.005221716593950987,
+            -0.008320784196257591,
+            -0.010768906213343143,
+            -0.033764272928237915,
+            0.0026710000820457935,
+            -0.015254771336913109,
+            0.008865597657859325,
+            -0.012544856406748295,
+            0.04027372971177101,
+            0.01022409275174141,
+            -0.00837738811969757,
+            -0.002796590095385909,
+            -0.0228114016354084,
+            -0.004475251771509647,
+            -0.016938740387558937,
+            -0.01184438169002533,
+            0.008342010900378227,
+            0.012629762291908264,
+            0.005614406894892454,
+            -0.018410442396998405,
+            0.003665107535198331,
+            0.0030955299735069275,
+            -0.005575491581112146,
+            -0.008370312862098217,
+            0.010853812098503113,
+            -0.0006248548161238432,
+            -0.01800006441771984,
+            -0.01957082375884056,
+            -0.027424626052379608,
+            0.0150000536814332,
+            0.014589674770832062,
+            -0.01011088490486145,
+            -0.022754797711968422,
+            -0.007471724878996611,
+            0.007107336539775133,
+            -0.02064630016684532,
+            0.0006779210525564849,
+            0.006891534198075533,
+            0.008717012591660023,
+            0.004117939155548811,
+            0.009636826813220978,
+            0.0007699025445617735,
+            0.0121698547154665,
+            0.018566103652119637,
+            0.03271709755063057,
+            -0.0396793857216835,
+            -0.009537769481539726,
+            0.0057028504088521,
+            -0.001465512323193252,
+            -0.013853822834789753,
+            0.009672204032540321,
+            0.0024375086650252342,
+            -0.0026426981203258038,
+            0.024424616247415543,
+            -0.008044839836657047,
+            -0.01505665760487318,
+            0.01184438169002533,
+            0.00887267291545868,
+            -0.019075538963079453,
+            0.02746707946062088,
+            0.006169833242893219,
+            -0.01011088490486145,
+            0.02259913645684719,
+            -0.012360893189907074,
+            0.012658064253628254,
+            -0.005522425286471844,
+            0.023660462349653244,
+            -0.019004784524440765,
+            0.0006306036957539618,
+            0.015665151178836823,
+            0.03883032873272896,
+            -0.00437973253428936,
+            -0.004567233379930258,
+            -0.03421710431575775,
+            0.005115584470331669,
+            0.028853876516222954,
+            0.02666047215461731,
+            -0.0012346742441877723,
+            -0.005596718285232782,
+            -0.0019864458590745926,
+            0.010768906213343143,
+            0.011589664034545422,
+            0.017561383545398712,
+            -0.029547275975346565,
+            0.00022011429246049374,
+            -0.016938740387558937,
+            0.025500090792775154,
+            0.014632128179073334,
+            0.006951675750315189,
+            0.04072656109929085,
+            0.0027789014857262373,
+            0.003841994795948267,
+            0.027452928945422173,
+            -0.00779719790443778,
+            0.003546592779457569,
+            -0.015566092915832996,
+            0.0003175129240844399,
+            0.022853855043649673,
+            0.0016370931407436728,
+            0.0020784272346645594,
+            0.013337312266230583,
+            0.029320860281586647,
+            -0.022471778094768524,
+            -0.023929331451654434,
+            0.003482913365587592,
+            0.007185167167335749,
+            -0.010570792481303215,
+            0.0038738346192985773,
+            0.006275965832173824,
+            -0.006007096730172634,
+            0.008900974877178669,
+            0.006332569755613804,
+            -0.0003166284877806902,
+            0.0039198254235088825,
+            -0.00308845448307693,
+            0.012962310574948788,
+            -0.02702839858829975,
+            0.036849189549684525,
+            0.004439874552190304,
+            0.010075507685542107,
+            0.015268922783434391,
+            0.001627364312298596,
+            0.004917470272630453,
+            0.005869124550372362,
+            0.010188715532422066,
+            0.018636859953403473,
+            -0.00956607237458229,
+            -0.027481229975819588,
+            -0.03166992589831352,
+            0.020363280549645424,
+            -0.01173117384314537,
+            0.0035872769076377153,
+            0.0074646491557359695,
+            -0.01828308403491974,
+            -0.007507102098315954,
+            -0.009976450353860855,
+            0.015764206647872925,
+            -0.03189634159207344,
+            0.012990612536668777,
+            -0.0043231286108493805,
+            -0.028669914230704308,
+            0.00851889792829752,
+            -0.010181640274822712,
+            -0.0045884596183896065,
+            -0.0035094465129077435,
+            -0.029264256358146667,
+            -0.007931631989777088,
+            0.03908504545688629,
+            -0.02141045406460762,
+            0.024184048175811768,
+            0.008257105015218258,
+            -0.00014449491573031992,
+            -0.0040365708991885185,
+            -0.013563727959990501,
+            0.005331386812031269,
+            -0.01778779923915863,
+            0.025457637384533882,
+            0.010294848121702671,
+            -0.013924578204751015,
+            0.0322076641023159,
+            -0.011179285123944283,
+            0.027438776567578316,
+            -0.018580254167318344,
+            0.036198243498802185,
+            0.007422196213155985,
+            0.00674648629501462,
+            0.01974063739180565,
+            -0.004025958012789488,
+            0.008228803053498268,
+            0.04559450224041939,
+            -0.018523650243878365,
+            0.02460857853293419,
+            0.020349130034446716,
+            -0.00022685811563860625,
+            0.017915157601237297,
+            0.0028390432707965374,
+            -0.023165177553892136,
+            0.009495317004621029,
+            -0.01135617308318615,
+            -0.009955224581062794,
+            0.01583496294915676,
+            -0.019316107034683228,
+            -0.018721764907240868,
+            -0.025613298639655113,
+            0.0029416377656161785,
+            -0.003350247861817479,
+            -0.0077618202194571495,
+            -0.0019616815261542797,
+            0.01454722136259079,
+            0.008724087849259377,
+            0.025669902563095093,
+            0.0008326975512318313,
+            0.011971740983426571,
+            0.01511326152831316,
+            -0.00035576484515331686,
+            0.02313687466084957,
+            -0.03229256719350815,
+            0.01246702577918768,
+            -0.004517704714089632,
+            -0.00509789539501071,
+            0.023858575150370598,
+            0.0003747802402358502,
+            -0.0004877670726273209,
+            -0.0024888061452656984,
+            0.0005496777012012899,
+            -0.014589674770832062,
+            -0.0057099261321127415,
+            0.000907432462554425,
+            -0.005175726022571325,
+            0.04299071803689003,
+            -0.00701535539701581,
+            -0.004772422835230827,
+            -0.039169952273368835,
+            -0.01918874867260456,
+            0.00623351288959384,
+            -0.01583496294915676,
+            -0.018778368830680847,
+            -0.006307805422693491,
+            0.005801907740533352,
+            -0.04893413558602333,
+            0.008342010900378227,
+            0.03336804360151291,
+            -0.012219383381307125,
+            0.00259493850171566,
+            -0.010981171391904354,
+            -0.006969364359974861,
+            -0.0029115669894963503,
+            -7.766463386360556e-05,
+            -0.002412744564935565,
+            -0.020830262452363968,
+            -0.005617944523692131,
+            0.003364398842677474,
+            -0.0008795727044343948,
+            -0.025047259405255318,
+            0.0006177793256938457,
+            0.018849123269319534,
+            -0.003979966975748539,
+            -0.0011152751976624131,
+            -0.001825478277169168,
+            0.006399787031114101,
+            -0.01508495956659317,
+            0.008009462617337704,
+            -0.0076344613917171955,
+            0.003841994795948267,
+            -0.0030495391692966223,
+            -0.002214630600064993,
+            0.02844349853694439,
+            0.021594416350126266,
+            -0.04808507859706879,
+            -0.01405901275575161,
+            0.023009516298770905,
+            0.007280686404556036,
+            -0.004740582779049873,
+            0.01178777776658535,
+            -0.031613320112228394,
+            -0.00028943203506059945,
+            -0.026787832379341125,
+            0.02200479619204998,
+            -0.009205221198499203,
+            -0.007386818993836641,
+            0.016967041417956352,
+            -0.026632171124219894,
+            0.016344398260116577,
+            0.00794578343629837,
+            -0.00030181417241692543,
+            -0.000493515923153609,
+            0.0063856360502541065,
+            0.0008972614887170494,
+            -0.01659911684691906,
+            0.006442239973694086,
+            -0.0048502529971301556,
+            0.007959933951497078,
+            -0.005087282508611679,
+            -0.05527378246188164,
+            -0.009933997876942158,
+            -0.005497660953551531,
+            -0.008936353027820587,
+            -0.012820800766348839,
+            0.008136821910738945,
+            0.02827368676662445,
+            0.01606137864291668,
+            -0.009863242506980896,
+            -4.8312376748071983e-05,
+            0.033000119030475616,
+            -0.0008300442132167518,
+            0.009955224581062794,
+            -0.011349096894264221,
+            0.00015709814033471048,
+            -0.0006911876262165606,
+            -0.014660430140793324,
+            -0.019061388447880745,
+            0.024042539298534393,
+            -0.028344441205263138,
+            0.02378782071173191,
+            -0.01843874529004097,
+            -0.0130825936794281,
+            0.01757553406059742,
+            -0.032575588673353195,
+            -0.005108508747071028,
+            0.007450498174875975,
+            0.0090566361322999,
+            0.021608568727970123,
+            0.0032794929575175047,
+            0.020589696243405342,
+            -0.03716050833463669,
+            -0.009134466759860516,
+            -0.010160413570702076,
+            -0.00794578343629837,
+            -0.010231168940663338,
+            -0.011596739292144775,
+            0.021863285452127457,
+            -0.015240620821714401,
+            -0.016952890902757645,
+            0.007054270710796118,
+            0.0030212372075766325,
+            -0.011313719674944878,
+            0.015750056132674217,
+            0.00017810352437663823,
+            0.004712280817329884,
+            0.013011839240789413,
+            -0.0008923970744945109,
+            0.04086806997656822,
+            0.0253019779920578,
+            -0.0030212372075766325,
+            -0.008568426594138145,
+            0.021438755095005035,
+            0.0004285097820684314,
+            0.0150000536814332,
+            0.009375033900141716,
+            0.023264234885573387,
+            -0.02092931978404522,
+            0.009997677057981491,
+            -0.005501199048012495,
+            0.014603826217353344,
+            -0.009495317004621029,
+            -0.015339677222073078,
+            0.00479718716815114,
+            -0.01577835902571678,
+            0.010153338313102722,
+            -0.010860888287425041,
+            0.008221727795898914,
+            0.01913214474916458,
+            0.002467579673975706,
+            -0.00941041111946106,
+            0.01146230474114418,
+            0.02075950801372528,
+            -0.025641601532697678,
+            0.008851447142660618,
+            -0.0031998935155570507,
+            -0.0021916350815445185,
+            0.021127434447407722,
+            0.010301923379302025,
+            -0.01594817079603672,
+            0.01573590561747551,
+            -0.008879749104380608,
+            -0.0015053119277581573,
+            -0.01882082223892212,
+            -0.03291521221399307,
+            -0.0015283073298633099,
+            0.012396270409226418,
+            0.004800724796950817,
+            0.0150000536814332,
+            0.011264191009104252,
+            -0.0008950503543019295,
+            -0.006601438857614994,
+            0.0008433107868768275,
+            -0.013485897332429886,
+            -0.004963461309671402,
+            0.010231168940663338,
+            0.017051948234438896,
+            -0.0037217114586383104,
+            0.015650998800992966,
+            0.0088160689920187,
+            0.006137993652373552,
+            -0.0010480580385774374,
+            0.0036138100549578667,
+            -0.002060738392174244,
+            -0.007783046457916498,
+            0.019938750192523003,
+            0.019075538963079453,
+            -0.018509499728679657,
+            -0.0024870370980352163,
+            -0.02141045406460762,
+            0.025882167741656303,
+            0.022188758477568626,
+            0.011165134608745575,
+            -0.04222656786441803,
+            0.00012050455552525818,
+            0.00752832880243659,
+            -0.0013744152383878827,
+            -0.031103884801268578,
+            0.008172199130058289,
+            0.005221716593950987,
+            0.004330204334110022,
+            0.020844414830207825,
+            -0.018424594774842262,
+            0.010853812098503113,
+            0.0070861103013157845,
+            -0.0005766529939137399,
+            0.01019579079002142,
+            -0.0073372903279960155,
+            0.008688710629940033,
+            0.038717120885849,
+            -0.018410442396998405,
+            0.005303084850311279,
+            0.0062370505183935165,
+            0.023320838809013367,
+            -0.011228813789784908,
+            -0.00031485961517319083,
+            0.00800238735973835,
+            0.021608568727970123,
+            -0.012934008613228798,
+            0.005883275531232357,
+            0.067075714468956,
+            -0.012912781909108162,
+            0.031613320112228394,
+            0.020533092319965363,
+            -0.004970536567270756,
+            -0.0104929618537426,
+            -0.012445799075067043,
+            -0.0181132722645998,
+            0.021283095702528954,
+            0.01175947580486536,
+            0.003419233951717615,
+            0.01941516436636448,
+            -0.01610383205115795,
+            0.006636816076934338,
+            0.017547233030200005,
+            0.012601460330188274,
+            -0.007379743270576,
+            -0.0036173479165881872,
+            0.019004784524440765,
+            0.018509499728679657,
+            -0.0030849166214466095,
+            0.00019424450874794275,
+            0.01508495956659317,
+            -0.022075550630688667,
+            -0.012530704960227013,
+            0.01644345559179783,
+            -0.0037358624394983053,
+            0.017773648723959923,
+            -0.02227366529405117,
+            0.024198200553655624,
+            -0.004531855694949627,
+            -0.017179306596517563,
+            0.004698129836469889,
+            -0.007514177821576595,
+            0.022033097222447395,
+            -0.001578720286488533,
+            0.004857328720390797,
+            0.000821199850179255,
+            -0.0018015984678640962,
+            0.01349297259002924,
+            0.002826661104336381,
+            -0.0005120891146361828,
+            0.010507113300263882,
+            -0.0009702275274321437,
+            -0.0013496510218828917,
+            0.024509521201252937,
+            1.9374700059415773e-05,
+            0.005989408120512962,
+            0.0032936439383774996,
+            -0.009261825121939182,
+            -0.004429261200129986,
+            -0.004443412180989981,
+            0.0011842612875625491,
+            0.003877372248098254,
+            -0.007029506377875805,
+            -0.02384442463517189,
+            0.022471778094768524,
+            -0.008540124632418156,
+            0.01175947580486536,
+            0.011349096894264221,
+            0.003665107535198331,
+            0.006824316922575235,
+            -0.009771261364221573,
+            -0.017320817336440086,
+            0.0006633278680965304,
+            -0.01617458648979664,
+            0.006912760436534882,
+            -0.001656550681218505,
+            0.0020235921256244183,
+            0.009445788338780403,
+            -0.002570174168795347,
+            0.0015123875346034765,
+            0.011306644417345524,
+            0.011497682891786098,
+            0.003254728624597192,
+            0.004577846731990576,
+            -0.018721764907240868,
+            -0.020915169268846512,
+            0.005288933869451284,
+            0.013924578204751015,
+            0.004273600410670042,
+            0.0015309606678783894,
+            0.017603836953639984,
+            -0.006166295614093542,
+            -0.022967062890529633,
+            -0.010117961093783379,
+            -0.01584911346435547,
+            0.039396367967128754,
+            0.01800006441771984,
+            -0.02525952458381653,
+            -0.000530220044311136,
+            -0.009728807955980301,
+            0.010450509376823902,
+            -0.00404364662244916,
+            0.014575524255633354,
+            0.005476434715092182,
+            -0.02426895499229431,
+            -0.016797229647636414,
+            -0.010662773624062538,
+            -0.005257094278931618,
+            0.020080260932445526,
+            0.027014248073101044,
+            -0.004956385586410761,
+            0.0061026159673929214,
+            0.019584976136684418,
+            -0.010938717983663082,
+            0.009849091991782188,
+            0.020957622677087784,
+            0.0031698227394372225,
+            -0.007570781745016575,
+            -0.004553082399070263,
+            -0.016514210030436516,
+            0.024467067793011665,
+            0.014250051230192184,
+            -0.005104971118271351,
+            -0.02465103194117546,
+            0.004110863897949457,
+            -0.02428310550749302,
+            -0.01616043597459793,
+            -0.007301913108676672,
+            -0.014143918640911579,
+            -0.004931621253490448,
+            0.0027594438288360834,
+            -0.004836102016270161,
+            0.011278342455625534,
+            -0.0031592093873769045,
+            -0.004531855694949627,
+            -0.0041851564310491085,
+            -0.025584997609257698,
+            -0.02524537406861782,
+            0.007691065315157175,
+            -0.003005317412316799,
+            0.005596718285232782,
+            0.000650945701636374,
+            -0.013400991447269917,
+            -0.011589664034545422,
+            0.00970758218318224,
+            -0.007988235913217068,
+            0.012813725508749485,
+            0.0031149876303970814,
+            -0.009183995425701141,
+            0.0017193458043038845,
+            -0.0012311364989727736,
+            -0.0059752571396529675,
+            0.019924599677324295,
+            0.00916984397917986,
+            0.011724098585546017,
+            -0.03118879161775112,
+            -0.0021633331198245287,
+            0.008349086157977581,
+            0.01828308403491974,
+            -0.02244347706437111,
+            -0.008313708938658237,
+            0.005179263651371002,
+            -0.011228813789784908,
+            -0.013507124036550522,
+            -0.028952933847904205,
+            -0.0030460015404969454,
+            8.175515540642664e-05,
+            -0.018636859953403473,
+            -0.029886899515986443,
+            0.0009038947173394263,
+            -0.007860877551138401,
+            0.016882136464118958,
+            -0.006643891800194979,
+            0.01790100708603859,
+            0.00870286114513874,
+            0.0036173479165881872,
+            0.011554286815226078,
+            0.0025082635693252087,
+            -0.013266556896269321,
+            0.008426916785538197,
+            -0.02671707607805729,
+            -0.01516986545175314,
+            0.009792488068342209,
+            0.014214673079550266,
+            -0.0026126273442059755,
+            5.074458022136241e-05,
+            0.0058372849598526955,
+            0.024254804477095604,
+            0.013584954664111137,
+            0.012474101036787033,
+            0.003042463678866625,
+            -0.005303084850311279,
+            0.005568416323512793,
+            0.0227123461663723,
+            0.008603804744780064,
+            0.004011806566268206,
+            0.007592008449137211,
+            0.00389859895221889,
+            0.0009932229295372963,
+            -0.005943417549133301,
+            -0.00843399204313755,
+            -0.021537812426686287,
+            -0.008908051066100597,
+            0.019938750192523003,
+            -0.010422207415103912,
+            -0.03005671128630638,
+            0.0006023016758263111,
+            -0.0032777239102870226,
+            0.007988235913217068,
+            0.02595292218029499,
+            -0.014172220602631569,
+            0.004970536567270756,
+            0.005536576267331839,
+            -0.007910405285656452,
+            -0.00614860700443387,
+            0.022967062890529633,
+            0.02639160305261612,
+            -0.0034404604230076075,
+            -0.04800017178058624,
+            -0.01671232469379902,
+            0.004305440001189709,
+            -0.007768895477056503,
+            -0.034047290682792664,
+            -0.015424583107233047,
+            0.002757675014436245,
+            -0.03206615149974823,
+            0.010068432427942753,
+            -0.012948159128427505,
+            -0.01214155275374651,
+            0.020264223217964172,
+            0.03223596513271332,
+            0.005154499784111977,
+            0.00014847487909719348,
+            -0.0214670579880476,
+            -0.0021686397958546877,
+            -0.021382151171565056,
+            0.010174565017223358,
+            -0.008801918476819992,
+            -0.010584943927824497,
+            0.0059611061587929726,
+            -0.012191081419587135,
+            0.013266556896269321,
+            0.010797208175063133,
+            0.024778390303254128,
+            0.01584911346435547,
+            -0.01876421831548214,
+            -0.0024746551644057035,
+            -0.01108730398118496,
+            -0.0016450530383735895,
+            -0.009820790030062199,
+            0.006360871717333794,
+            -0.0031592093873769045,
+            0.011264191009104252,
+            0.015184016898274422,
+            0.004432798828929663,
+            0.00046786724124103785,
+            0.038151077926158905,
+            -2.4487851987942122e-05,
+            -0.0003418349369894713,
+            0.006803090218454599,
+            -0.015750056132674217,
+            0.008837295696139336,
+            0.012191081419587135,
+            0.005051904823631048,
+            -0.007698140572756529,
+            0.006739411037415266,
+            0.03099067695438862,
+            -0.01516986545175314,
+            -0.009014182724058628,
+            -0.023504801094532013,
+            0.007549555506557226,
+            -0.0004656561359297484,
+            0.029971804469823837,
+            0.020533092319965363,
+            -0.027509532868862152,
+            -0.016018925234675407,
+            -0.02179253101348877,
+            0.01267929095774889,
+            -0.011497682891786098,
+            -0.003704022616147995,
+            0.012240610085427761,
+            0.02367461286485195,
+            0.0018838511314243078,
+            -0.014455240219831467,
+            -0.008950503543019295,
+            0.008943428285419941,
+            0.014617976732552052,
+            0.003497064346447587,
+            0.007570781745016575,
+            0.00743634719401598,
+            0.008886824361979961,
+            -0.014801939949393272,
+            -0.012254760600626469,
+            0.00767691433429718,
+            -0.010202866978943348,
+            0.01794346049427986,
+            -0.009403335861861706,
+            0.016768928617239,
+            -0.004645063541829586,
+            -0.011964665725827217,
+            -0.0320095494389534,
+            -0.002292460994794965,
+            0.005037753842771053,
+            0.004627374932169914,
+            0.0034705311991274357,
+            -0.0022217060904949903,
+            -0.013160424306988716,
+            0.004673365969210863,
+            0.005073131527751684,
+            1.875283123808913e-05,
+            -0.008313708938658237,
+            -0.00479718716815114,
+            0.004252373706549406,
+            -0.001031253719702363,
+            -0.0073372903279960155,
+            -0.0059858704917132854,
+            0.018481198698282242,
+            0.0066509670577943325,
+            0.00800238735973835,
+            0.03234917297959328,
+            0.02974538877606392,
+            -0.0016353242099285126,
+            -0.004606148693710566,
+            -0.00023304918431676924,
+            -0.01589156687259674,
+            -0.0057205394841730595,
+            -0.023320838809013367,
+            0.022259512916207314,
+            -0.009665128774940968,
+            0.021820832043886185,
+            0.019457615911960602,
+            0.0039304387755692005,
+            -0.008391539566218853,
+            -0.0011586126638576388,
+            -0.007308988366276026,
+            -0.003560743760317564,
+            0.016641568392515182,
+            0.006742948666214943,
+            0.01226891204714775,
+            0.00030933189555071294,
+            -0.028401045128703117,
+            0.026405755430459976,
+            0.000579748535528779,
+            0.0034015451092272997,
+            0.008908051066100597,
+            0.014129767194390297,
+            0.013209952972829342,
+            -0.01235381793230772,
+            0.01768874190747738,
+            -0.03280200436711311,
+            -0.0033484790474176407,
+            -0.016500059515237808,
+            0.002076658420264721,
+            -0.0018520113080739975,
+            -0.0008145665633492172,
+            -0.05354736000299454,
+            0.0003842879377771169,
+            0.005313698202371597,
+            -0.015339677222073078,
+            0.029349161311984062,
+            -0.020589696243405342,
+            -0.017165156081318855,
+            -0.032858606427907944,
+            -0.010238244198262691,
+            0.018099121749401093,
+            0.005617944523692131,
+            0.00035178486723452806,
+            -0.02660386823117733,
+            0.015750056132674217,
+            -0.0047476585023105145,
+            -0.00743634719401598,
+            -0.029858596622943878,
+            -0.010938717983663082,
+            0.009403335861861706,
+            0.004485865123569965,
+            -0.0130825936794281,
+            0.009898620657622814,
+            -0.002890340518206358,
+            0.010867963545024395,
+            -0.0138467475771904,
+            -0.00420284504070878,
+            0.0010020672343671322,
+            -0.005363226868212223,
+            0.012615610845386982,
+            -0.030537845566868782,
+            0.011391550302505493,
+            -0.01352127455174923,
+            0.0008548084879294038,
+            -0.017858553677797318,
+            -0.027707645669579506,
+            0.0020288988016545773,
+            -0.016132133081555367,
+            0.039990708231925964,
+            0.010188715532422066,
+            -0.01481609046459198,
+            0.016995344310998917,
+            -0.002080196049064398,
+            0.017377421259880066,
+            0.0014955832157284021,
+            0.017023645341396332,
+            -0.011971740983426571,
+            0.021056678146123886,
+            -0.0041745430789887905,
+            0.02156611531972885,
+            0.00600002147257328,
+            0.034047290682792664,
+            -0.02033497765660286,
+            0.0070825726725161076,
+            0.003792466362938285,
+            0.005798369646072388,
+            -0.0026816134341061115,
+            0.021424604579806328,
+            0.017985913902521133,
+            0.019698183983564377,
+            0.0037110981065779924,
+            -0.024084990844130516,
+            -0.006137993652373552,
+            0.004867942072451115,
+            0.023858575150370598,
+            0.011150983162224293,
+            -0.016415152698755264,
+            -0.021056678146123886,
+            0.015226469375193119,
+            -0.007860877551138401,
+            0.006771250627934933,
+            -0.00851889792829752,
+            0.00037544354563578963,
+            -0.013259481638669968,
+            0.012233533896505833,
+            -0.012523629702627659,
+            0.0032335021533071995,
+            0.01043635793030262,
+            -0.027806703001260757,
+            0.01511326152831316,
+            -0.00371817359700799,
+            -0.014313730411231518,
+            0.023320838809013367,
+            -0.015877414494752884,
+            -0.013988257385790348,
+            0.005501199048012495,
+            0.008610880002379417,
+            0.01746232621371746,
+            -0.002761212643235922,
+            0.002669231267645955,
+            -0.020504789426922798,
+            -0.012445799075067043,
+            0.004591997712850571,
+            0.02210385352373123,
+            -0.015820810571312904,
+            0.012431648559868336,
+            -0.018099121749401093,
+            0.01364863384515047,
+            0.016202887520194054,
+            0.007149789482355118,
+            0.00573115237057209,
+            -0.007726442534476519,
+            -0.0011471149045974016,
+            0.032151058316230774,
+            -0.0029876285698264837,
+            0.02704254910349846,
+            -0.014915147796273232,
+            0.009615600109100342,
+            0.00518987700343132,
+            -0.013089669868350029,
+            -0.014589674770832062,
+            0.011108530685305595,
+            0.02428310550749302,
+            -0.009530694223940372,
+            -0.023660462349653244,
+            0.0018484735628589988,
+            0.014264201745390892,
+            0.010337301529943943,
+            0.009403335861861706,
+            -0.001997058978304267,
+            -0.0037358624394983053,
+            0.01484439242631197,
+            -0.0032918748911470175,
+            -0.0286557637155056,
+            -0.01633024774491787,
+            -0.004400959238409996,
+            -0.017490629106760025,
+            0.020745357498526573,
+            0.02367461286485195,
+            0.010507113300263882,
+            0.0037570889107882977,
+            0.005540114361792803,
+            0.018297234550118446,
+            0.01843874529004097,
+            0.013330236077308655,
+            0.009304278530180454,
+            0.012700516730546951,
+            -0.004319590982049704,
+            0.02800481766462326,
+            -0.010860888287425041,
+            -0.01046465989202261,
+            -0.015707602724432945,
+            0.0016636261716485023,
+            -0.005193414632230997,
+            -0.012431648559868336,
+            -0.0069481381215155125,
+            0.004369119182229042,
+            0.013330236077308655,
+            -0.012240610085427761,
+            -0.007149789482355118,
+            0.01016748882830143,
+            0.011971740983426571,
+            0.01403071079403162,
+            0.01719345711171627,
+            -0.01633024774491787,
+            -0.011278342455625534,
+            0.0009834941010922194,
+            -0.02844349853694439,
+            -0.0160047747194767,
+            -0.01767459139227867,
+            0.009580222889780998,
+            -0.014469391666352749,
+            0.03724541515111923,
+            0.028471799567341805,
+            0.0073514413088560104,
+            0.031613320112228394,
+            -0.028401045128703117,
+            0.00970758218318224,
+            -0.014398636296391487,
+            0.007132100872695446,
+            -0.004188694059848785,
+            0.003902136581018567,
+            0.002448122017085552,
+            0.020419884473085403,
+            0.00791748147457838,
+            0.014264201745390892,
+            0.008150972425937653,
+            -0.021934039890766144,
+            -0.00662620272487402,
+            -0.012559006921947002,
+            -0.016401002183556557,
+            -0.02595292218029499,
+            0.013719388283789158,
+            0.005087282508611679,
+            -0.003074303502216935,
+            -0.0002704166399780661,
+            0.006566061172634363,
+            -0.0013195802457630634,
+            0.010443433187901974,
+            -0.027608588337898254,
+            -0.01311089564114809,
+            0.0069269114173948765,
+            0.01449061743915081,
+            -0.02114158496260643,
+            0.0014637433923780918,
+            0.008900974877178669,
+            -0.004602610599249601,
+            0.035660505294799805,
+            -0.014674580655992031,
+            -0.008610880002379417,
+            0.0027806703001260757,
+            0.005062518175691366,
+            -0.017660440877079964,
+            -0.02319347858428955,
+            0.0070648835971951485,
+            0.011575513519346714,
+            0.010867963545024395,
+            -0.028570856899023056,
+            0.02087271586060524,
+            -0.0054339817725121975,
+            0.03832089155912399,
+            0.00401534466072917,
+            -0.013337312266230583,
+            0.003962278366088867,
+            0.0011860302183777094,
+            -0.012360893189907074,
+            0.002060738392174244,
+            -0.000340508297085762,
+            0.003700484987348318,
+            0.011228813789784908,
+            -0.014801939949393272,
+            -0.016967041417956352,
+            0.016910437494516373,
+            -0.011476456187665462,
+            -0.0035395172890275717,
+            -0.00943871308118105,
+            -0.016669871285557747,
+            0.007308988366276026,
+            0.025047259405255318,
+            0.0041745430789887905,
+            0.001242634141817689,
+            -0.011632117442786694,
+            -0.0038596836384385824,
+            -0.008575502783060074,
+            0.0011718792375177145,
+            -0.0207029040902853,
+            0.027396325021982193,
+            -0.01801421493291855,
+            0.002410975517705083,
+            0.021099131554365158,
+            0.008709936402738094,
+            -0.003944589756429195,
+            0.016316095367074013,
+            0.026363302022218704,
+            0.01205664686858654,
+            0.0031539027113467455,
+            -0.00902833417057991,
+            1.648646139074117e-05,
+            0.014957600273191929,
+            0.0015123875346034765,
+            -0.0015663381200283766,
+            0.0032140444964170456,
+            -0.006583749782294035,
+            -0.001859086798503995,
+            -0.017561383545398712,
+            0.005564878229051828,
+            0.02125479280948639,
+            -0.004132090136408806,
+            0.020476488396525383,
+            0.015509488992393017,
+            0.011625041253864765,
+            0.010726453736424446,
+            -0.0012390963966026902,
+            -0.002150951186195016,
+            -0.0021668709814548492,
+            -0.007393894251435995,
+            -0.018297234550118446,
+            0.006470541935414076,
+            0.0034493047278374434,
+            0.01822648011147976,
+            0.020504789426922798,
+            -0.02308027073740959,
+            0.010146263055503368,
+            0.014985902234911919,
+            -0.013011839240789413,
+            0.0235755555331707,
+            -0.02006610855460167,
+            0.0024605041835457087,
+            0.02156611531972885,
+            0.013337312266230583,
+            -0.014271277002990246,
+            -0.009219372645020485,
+            0.004662752617150545,
+            -0.01543873455375433,
+            0.000872939417604357,
+            -0.01478778850287199,
+            -0.010783057659864426,
+            0.014533070847392082,
+            -0.01490099634975195,
+            0.01229721400886774,
+            -0.006969364359974861,
+            0.0070967236533761024,
+            -0.017165156081318855,
+            0.015014204196631908,
+            0.008632106706500053,
+            0.01757553406059742,
+            0.0021279556676745415,
+            -0.02518877014517784,
+            -0.006290116813033819,
+            0.021169887855648994,
+            0.009509467519819736,
+            -0.02497650496661663,
+            0.019613277167081833,
+            0.0020094411447644234,
+            -0.019528372213244438,
+            -0.022401023656129837,
+            -0.012686366215348244,
+            -0.005055442452430725,
+            -0.017264213413000107,
+            -0.0011276573641225696,
+            -0.024792540818452835,
+            0.003972891718149185,
+            0.009106164798140526,
+            -0.011978816241025925,
+            -0.019273653626441956,
+            -0.0002315014135092497,
+            -0.01040805596858263,
+            0.006219361908733845,
+            -0.01132787112146616,
+            -0.019853845238685608,
+            0.0070436573587358,
+            0.012127402238547802,
+            -0.0027647505048662424,
+            -0.005646246485412121,
+            -0.02562745101749897,
+            -0.006919836159795523,
+            0.020179318264126778,
+            -0.003994117956608534,
+            -0.004974074196070433,
+            -0.01703779585659504,
+            0.01108730398118496,
+            0.011165134608745575,
+            0.020901018753647804,
+            0.010931642726063728,
+            0.005674548447132111,
+            0.013634482398629189,
+            -0.0029416377656161785,
+            -0.007156865205615759,
+            -9.645892532716971e-06,
+            0.004284213297069073,
+            0.0069233737885952,
+            0.0016636261716485023,
+            0.020235922187566757,
+            -0.01692458800971508,
+            0.018934030085802078,
+            -0.012290137819945812,
+            0.008794843219220638,
+            -0.01855195313692093,
+            -0.0003155229496769607,
+            -0.01067692507058382,
+            0.00015002264990471303,
+            -0.0038455326575785875,
+            -0.009544845670461655,
+            1.9595809135353193e-05,
+            0.00143986358307302,
+            0.008511822670698166,
+            -0.016783079132437706,
+            0.009615600109100342,
+            -0.010839661583304405,
+            -0.004308977629989386,
+            -0.019542522728443146,
+            0.017448175698518753,
+            0.013344387523829937,
+            0.007308988366276026,
+            -0.008278331719338894,
+            -0.006300730165094137,
+            -0.02282555401325226,
+            0.008582578040659428,
+            -0.0014637433923780918,
+            0.002214630600064993,
+            -0.028301987797021866,
+            -0.004358506295830011,
+            0.0024587351363152266,
+            0.020306676626205444,
+            0.0033661676570773125,
+            -0.010712302289903164,
+            0.008978805504739285,
+            -0.023377442732453346,
+            -0.0030159305315464735,
+            -0.023349139839410782,
+            -0.004075486212968826,
+            -0.005112046375870705,
+            0.010103809647262096,
+            0.008101443760097027,
+            0.01549533847719431,
+            -0.009898620657622814,
+            0.01478778850287199,
+            -0.004793649073690176,
+            0.012049571610987186,
+            -0.0012152165872976184,
+            0.01337976474314928,
+            -0.0003111007681582123,
+            -0.01505665760487318,
+            0.02158026583492756,
+            -0.008738238364458084,
+            0.001741456682793796,
+            -0.0052853962406516075,
+            0.0027346794959157705,
+            -0.017702894285321236,
+            -0.004553082399070263,
+            -0.014455240219831467,
+            0.01957082375884056,
+            -0.022584985941648483,
+            0.004153316840529442,
+            -0.005055442452430725,
+            -0.016075529158115387,
+            0.029434068128466606,
+            0.022896308451890945,
+            -0.0035483618266880512,
+            0.015070809051394463,
+            0.0017989451298490167,
+            0.0015061964513733983,
+            0.016202887520194054,
+            -0.015806660056114197,
+            -0.013025989755988121,
+            0.035066161304712296,
+            0.0176321379840374,
+            0.0004313841927796602,
+            -0.0023260696325451136,
+            -0.0037323245778679848,
+            0.0032193511724472046,
+            0.008023613132536411,
+            -0.010450509376823902,
+            -0.01141277700662613,
+            -0.017051948234438896,
+            0.009997677057981491,
+            -0.006155682262033224,
+            0.004446949809789658,
+            -0.00025891896802932024,
+            -7.617214578203857e-05,
+            -0.01516986545175314,
+            0.022584985941648483,
+            -0.017717044800519943,
+            -0.00602832343429327,
+            0.004687516950070858,
+            0.0207029040902853,
+            -0.02541518583893776,
+            0.01616043597459793,
+            -0.007107336539775133,
+            -0.013429293408989906,
+            0.012573158368468285,
+            -0.015127412974834442,
+            -0.0107335289940238,
+            -0.003175129182636738,
+            -0.008009462617337704,
+            -0.022754797711968422,
+            0.017929309979081154,
+            -0.013627407141029835,
+            0.015650998800992966,
+            -0.00287972716614604,
+            0.000562502013053745,
+            0.002057200763374567,
+            0.0032582662533968687,
+            0.02250008098781109,
+            0.030735958367586136,
+            0.010415131226181984,
+            -0.005465821363031864,
+            -0.021000074222683907,
+            0.012983537279069424,
+            0.0020589695777744055,
+            0.013202877715229988,
+            -0.002253545681014657,
+            -0.002135031158104539,
+            -0.016613267362117767,
+            -0.023122724145650864,
+            -0.0013010069960728288,
+            -0.01761798746883869,
+            -0.02552839368581772,
+            -0.01774534583091736,
+            -0.007047194987535477,
+            0.011349096894264221,
+            0.020405733957886696,
+            0.0009755341452546418,
+            -0.005384453106671572,
+            0.01559439580887556,
+            -0.006941062398254871,
+            0.002080196049064398,
+            -0.006594363134354353,
+            -0.002032436430454254,
+            -0.008200501091778278,
+            -0.021863285452127457,
+            0.029660483822226524,
+            -0.010103809647262096,
+            0.01141277700662613,
+            -0.005472897086292505,
+            -0.00965097825974226,
+            0.005575491581112146,
+            0.00900003220885992,
+            0.024849144741892815,
+            0.006452853325754404,
+            0.03141520544886589,
+            0.020151015371084213,
+            -0.018891576677560806,
+            -0.0006071660900488496,
+            0.022684043273329735,
+            -0.0037570889107882977,
+            -0.004889168310910463,
+            -0.0031786670442670584,
+            -0.017985913902521133,
+            -0.012007118202745914,
+            0.007839650847017765,
+            -0.013471745885908604,
+            -0.006636816076934338,
+            -0.00837738811969757,
+            0.018070818856358528,
+            -0.017773648723959923,
+            0.025698205456137657,
+            -0.006488230545073748,
+            0.005154499784111977,
+            0.031924642622470856,
+            0.014363259077072144,
+            0.019698183983564377,
+            -0.011794853024184704,
+            -0.031075581908226013,
+            -0.00683846790343523,
+            -0.007882103323936462,
+            0.009558996185660362,
+            -0.01062032114714384,
+            0.005204027984291315,
+            -0.02006610855460167,
+            -0.018948180601000786,
+            0.012863253243267536,
+            -0.017476476728916168,
+            0.013924578204751015,
+            -0.009955224581062794,
+            0.027226513251662254,
+            -0.001228483160957694,
+            0.0121698547154665,
+            -0.002614396158605814,
+            -0.017801949754357338,
+            0.01205664686858654,
+            0.004089637193828821,
+            -0.023943481966853142,
+            0.010337301529943943,
+            -0.001864393474534154,
+            -0.014037786051630974,
+            -0.018410442396998405,
+            -0.001228483160957694,
+            -0.005200490355491638,
+            -0.009658053517341614,
+            0.025457637384533882,
+            0.014101465232670307,
+            0.016231190413236618,
+            0.0035359796602278948,
+            0.0016335553955286741,
+            -0.011073152534663677,
+            0.009573147632181644,
+            -0.006803090218454599,
+            0.000612030504271388,
+            -0.0034404604230076075,
+            0.011893910355865955,
+            -0.004252373706549406,
+            -0.01543873455375433,
+            0.002324300818145275,
+            0.008561351336538792,
+            0.0143278818577528,
+            5.947839599684812e-05,
+            0.009587298147380352,
+            0.0009790719486773014,
+            -0.009764186106622219,
+            -0.00046388726332224905,
+            -0.0060884649865329266,
+            0.0069375247694551945,
+            0.02400008589029312,
+            -0.012516554445028305,
+            -0.01681138016283512,
+            0.014504768885672092,
+            0.004033033270388842,
+            -0.017589686438441277,
+            0.018367990851402283,
+            0.004103788174688816,
+            -0.022358570247888565,
+            0.0035412863362580538,
+            0.010662773624062538,
+            -0.009523618966341019,
+            -0.014469391666352749,
+            -0.0010339069413021207,
+            -0.015509488992393017,
+            0.03130199760198593,
+            0.00025073791039176285,
+            0.016910437494516373,
+            -0.01062032114714384,
+            -0.0025967073161154985,
+            -0.013309009373188019,
+            0.013103820383548737,
+            -0.015750056132674217,
+            -0.007273610681295395,
+            0.00035620704875327647,
+            -0.02298121340572834,
+            0.01741987280547619,
+            0.0013425755314528942,
+            0.00794578343629837,
+            0.003436922561377287,
+            -0.010556641966104507,
+            0.009820790030062199,
+            0.0018237093463540077,
+            0.006668656133115292,
+            -0.014915147796273232,
+            0.023618008941411972,
+            0.006413938011974096,
+            0.011235889047384262,
+            -0.01352127455174923,
+            -0.01546703651547432,
+            0.002253545681014657,
+            0.04134920611977577,
+            -0.01306136790663004,
+            0.0002593612007331103,
+            0.01562269777059555,
+            7.081024523358792e-05,
+            0.004156854469329119,
+            -0.0023755980655550957,
+            -0.0034493047278374434,
+            -0.034585028886795044,
+            0.005370302125811577,
+            -0.0006328147719614208,
+            0.008073141798377037,
+            -0.008992956951260567,
+            -0.005430444143712521,
+            -0.0040684109553694725,
+            -0.008625030517578125,
+            -0.00314859626814723,
+            -0.01478778850287199,
+            0.008363237604498863,
+            -0.014533070847392082,
+            -0.028910480439662933,
+            0.03166992589831352,
+            -0.006887996103614569,
+            -0.00533846253529191,
+            -0.021396303549408913,
+            0.007570781745016575,
+            -0.01370523776859045,
+            0.002663924591615796,
+            0.010967019945383072,
+            0.008865597657859325,
+            0.008738238364458084,
+            0.002762981690466404,
+            0.0009516543359495699,
+            -0.004255911335349083,
+            0.010273621417582035,
+            -0.00835616234689951,
+            -0.0005943417199887335,
+            -0.0014186371117830276,
+            -0.02725481428205967,
+            -0.01405901275575161,
+            -0.010592019185423851,
+            -0.007811348885297775,
+            -0.00435143057256937,
+            0.012608535587787628,
+            0.007174553815275431,
+            -0.0017485321732237935,
+            -0.022938761860132217,
+            -0.014745336025953293,
+            -0.0034103896468877792,
+            0.008349086157977581,
+            -0.01838214136660099,
+            0.01064862310886383,
+            0.0016830838285386562,
+            0.009976450353860855,
+            0.014292503707110882,
+            0.00302831269800663,
+            0.0016742395237088203,
+            0.003458149265497923,
+            0.014886845834553242,
+            0.012601460330188274,
+            -0.005801907740533352,
+            -0.021382151171565056,
+            -0.0024605041835457087,
+            -0.007068421691656113,
+            -0.0034687623847275972,
+            -0.012332591228187084,
+            0.03039633482694626,
+            0.0029805530793964863,
+            0.0176321379840374,
+            0.007832575589418411,
+            0.009573147632181644,
+            0.010507113300263882,
+            0.029377464205026627,
+            0.006049550138413906,
+            -0.0029929352458566427,
+            -0.008603804744780064,
+            -0.02141045406460762,
+            -0.013146273791790009,
+            0.009898620657622814,
+            0.013294858857989311,
+            0.03597182780504227,
+            -0.034896351397037506,
+            0.012162779457867146,
+            -0.011101454496383667,
+            -0.008264180272817612,
+            -0.008533049374818802,
+            0.0044717141427099705,
+            0.010068432427942753,
+            0.014370334334671497,
+            -0.0023402206134051085,
+            -0.01175947580486536,
+            0.009070787578821182,
+            0.023547254502773285,
+            -0.012983537279069424,
+            0.03016991913318634,
+            0.027325568720698357,
+            0.023066120222210884,
+            0.01761798746883869,
+            -0.012537780217826366,
+            0.026462359353899956,
+            -6.484030018327758e-05,
+            0.0006261814851313829,
+            -0.018084971234202385,
+            -0.00587973790243268,
+            -0.004294826649129391,
+            -0.004362043924629688,
+            0.013316085562109947,
+            -0.0025471788831055164,
+            -0.0008468485320918262,
+            0.015099111013114452,
+            -0.0037323245778679848,
+            -3.927453508367762e-05,
+            -0.0059752571396529675,
+            0.015127412974834442,
+            0.0002699744363781065,
+            0.019457615911960602,
+            -0.04270770028233528,
+            0.019726485013961792,
+            -0.021396303549408913,
+            -0.003102605463936925,
+            -0.0020890405867248774,
+            0.009523618966341019,
+            0.00764861237257719,
+            -0.007142714224755764,
+            -0.014334957115352154,
+            0.004524780437350273,
+            0.014066088013350964,
+            -0.0006726144347339869,
+            0.005080206785351038,
+            -0.0027541371528059244,
+            0.0017228835495188832,
+            -0.010089659132063389,
+            -0.005862049292773008,
+            5.842812606715597e-05,
+            -0.010570792481303215,
+            -0.008646257221698761,
+            -0.03232087194919586,
+            0.01062032114714384,
+            0.010209942236542702,
+            0.007294837385416031,
+            0.004252373706549406,
+            -0.022018946707248688,
+            -0.012318440712988377,
+            0.012948159128427505,
+            -0.01132787112146616,
+            -0.002570174168795347,
+            -0.0020005968399345875,
+            0.005387990735471249,
+            0.0004705205501522869,
+            0.010160413570702076,
+            -0.003472300246357918,
+            0.010938717983663082,
+            -0.014603826217353344,
+            0.018084971234202385,
+            0.03229256719350815,
+            -0.00533846253529191,
+            -0.019967053085565567,
+            0.000907432462554425,
+            -0.009077862836420536,
+            0.003813692834228277,
+            0.007641536649316549,
+            -0.021962342783808708,
+            -0.02173592709004879,
+            0.005087282508611679,
+            -0.013153349049389362,
+            -0.007783046457916498,
+            0.014830241911113262,
+            0.005221716593950987,
+            0.010839661583304405,
+            0.01751893013715744,
+            0.010889190249145031,
+            0.00840569008141756,
+            -0.026533113792538643,
+            0.0006734988419339061,
+            -0.012686366215348244,
+            -0.00590803986415267,
+            0.002239394700154662,
+            -0.0017591454088687897,
+            0.004591997712850571,
+            -0.00010115749319083989,
+            -0.011200511828064919,
+            -0.0007057808106765151,
+            0.010903340764343739,
+            0.027919910848140717,
+            -0.004146241117268801,
+            -0.0022570835426449776,
+            -0.014257126487791538,
+            0.0010100272484123707,
+            0.036481261253356934,
+            -0.012275987304747105,
+            -0.017434025183320045,
+            0.01757553406059742,
+            0.027452928945422173,
+            0.005954030901193619,
+            -0.0032529598101973534,
+            -0.0027364485431462526,
+            0.00039600671152584255,
+            0.011978816241025925,
+            0.007719367276877165,
+            -0.002692226553335786,
+            -0.002131493529304862,
+            0.004761809483170509,
+            0.035773713141679764,
+            -0.0015203474322333932,
+            -0.010330225341022015,
+            -0.024240652099251747,
+            0.010974096134305,
+            0.00277182599529624,
+            0.004754733759909868,
+            0.002313687466084957,
+            0.023504801094532013,
+            0.019655730575323105,
+            0.014858543872833252,
+            0.00470520555973053,
+            -0.009771261364221573,
+            0.03005671128630638,
+            -0.008172199130058289,
+            -0.004542469047009945,
+            -0.02838689461350441,
+            -0.005087282508611679,
+            -0.015650998800992966,
+            0.008929276838898659,
+            -0.0002221042668679729,
+            -0.005016527138650417,
+            -0.013422218151390553,
+            0.019726485013961792,
+            0.01064862310886383,
+            -0.011073152534663677,
+            0.012608535587787628,
+            -0.027070851996541023,
+            0.02346234768629074,
+            -0.03826428949832916,
+            -0.0071993181481957436,
+            -0.006421013735234737,
+            -0.010514188557863235,
+            -0.02627839520573616,
+            -0.0030353881884366274,
+            0.007372668012976646,
+            0.014250051230192184,
+            0.0022570835426449776,
+            0.0022818478755652905,
+            0.02282555401325226,
+            0.0016415152931585908,
+            -0.01508495956659317,
+            0.017985913902521133,
+            0.03492465242743492,
+            0.012926933355629444,
+            0.00155837822239846,
+            0.015990622341632843,
+            0.0027789014857262373,
+            0.0021633331198245287,
+            0.00788917951285839,
+            -0.0034493047278374434,
+            -0.007938707247376442,
+            0.008582578040659428,
+            0.006990591064095497,
+            -0.02763689123094082,
+            0.003824306186288595,
+            -0.00010408719390397891,
+            -0.009898620657622814,
+            -0.022429324686527252,
+            0.011568437330424786,
+            0.01913214474916458,
+            -0.004224071744829416,
+            -0.006771250627934933,
+            -0.0043337419629096985,
+            0.0006836698739789426,
+            0.006785401608794928,
+            -0.002412744564935565,
+            -0.000515184598043561,
+            -0.0015176940942183137,
+            -0.02368876338005066,
+            -0.010316074825823307,
+            0.013570803217589855,
+            -0.005175726022571325,
+            0.010358527302742004,
+            0.005242943298071623,
+            -0.018792519345879555,
+            -0.016938740387558937,
+            0.006852618884295225,
+            -0.016188737004995346,
+            0.01364863384515047,
+            0.00027948213391937315,
+            -0.0023172253277152777,
+            0.024962352588772774,
+            0.03594352304935455,
+            -0.013188726268708706,
+            0.005119122099131346,
+            -0.019386861473321915,
+            -0.0245378240942955,
+            -0.019004784524440765,
+            0.022613288834691048,
+            0.008306633681058884,
+            -0.018792519345879555,
+            0.012304289266467094,
+            -0.002752368338406086,
+            -0.008865597657859325,
+            -0.0066721937619149685,
+            -0.006845543161034584,
+            -0.008547200821340084,
+            0.020533092319965363,
+            -0.004935159347951412,
+            0.0038950610905885696,
+            0.019047237932682037,
+            0.04075486212968826,
+            -0.008292482234537601,
+            -0.01349297259002924,
+            0.0054339817725121975,
+            0.004878554958850145,
+            0.0018626245437189937,
+            -0.004814875777810812,
+            -0.008016537874937057,
+            -0.0185944065451622,
+            -0.01740572229027748,
+            -0.009615600109100342,
+            -0.0020041344687342644,
+            0.020122714340686798,
+            -0.007768895477056503,
+            -0.01354957651346922,
+            -0.0245378240942955,
+            -0.028188779950141907,
+            -0.006580212153494358,
+            0.02319347858428955,
+            0.017179306596517563,
+            -0.00013200224202591926,
+            -4.3337418901501223e-05,
+            -0.012042495422065258,
+            -0.01968403346836567,
+            0.015509488992393017,
+            0.00037876018905080855,
+            -0.01773119531571865,
+            -0.0065448349341750145,
+            -0.015919867902994156,
+            0.005787756759673357,
+            -0.0051368107087910175,
+            0.004574308637529612,
+            0.05057565122842789,
+            0.0005271245026960969,
+            0.008441068232059479,
+            -0.026575567200779915,
+            0.009106164798140526,
+            -0.01583496294915676,
+            -0.017377421259880066,
+            -0.00012702727690339088,
+            -0.022301966324448586,
+            -0.0030955299735069275,
+            -0.018792519345879555,
+            -0.007386818993836641,
+            0.008568426594138145,
+            0.009799563325941563,
+            -0.013896276243031025,
+            -0.013669860549271107,
+            -0.024693485349416733,
+            -0.0035872769076377153,
+            -0.001539805089123547,
+            0.009311353787779808,
+            0.007825499400496483,
+            -0.03693409264087677,
+            0.002536565763875842,
+            -0.0009507699287496507,
+            0.0005779796629212797,
+            0.01208494883030653,
+            -0.008667483925819397,
+            -0.003063690150156617,
+            -0.0214670579880476,
+            0.006452853325754404,
+            0.007103798910975456,
+            -0.013500047847628593,
+            0.012375044636428356,
+            -0.012863253243267536,
+            -0.007669838611036539,
+            -0.0005426021525636315,
+            0.03379257395863533,
+            0.011794853024184704,
+            -0.0025153392925858498,
+            -0.00654129683971405,
+            -0.011150983162224293,
+            0.02654726430773735,
+            -0.011349096894264221,
+            -0.026405755430459976,
+            0.028910480439662933,
+            0.013245330192148685,
+            0.0005974372616037726,
+            0.020151015371084213,
+            -0.0017936384538188577,
+            -0.01397410687059164,
+            0.009601449593901634,
+            0.004266524687409401,
+            0.012424572370946407,
+            -0.00014460546663030982,
+            0.0214670579880476,
+            0.0018555490532889962,
+            0.0027399861719459295,
+            -0.0036226543597877026,
+            0.004687516950070858,
+            -0.03093407303094864,
+            -0.0006549257086589932,
+            -0.003102605463936925,
+            -0.008398614823818207,
+            0.011207587085664272,
+            -0.002421588869765401,
+            0.006558985915035009,
+            -0.022188758477568626,
+            -0.028485950082540512,
+            0.011342021636664867,
+            -0.016231190413236618,
+            -0.021396303549408913,
+            0.015339677222073078,
+            -0.0007641536649316549,
+            0.00614860700443387,
+            -0.007443422917276621,
+            -0.023278385400772095,
+            -0.019386861473321915,
+            0.02064630016684532,
+            0.012212308123707771,
+            -0.01190098561346531,
+            0.012750045396387577,
+            -0.021113283932209015,
+            -0.012948159128427505,
+            -0.001894464367069304,
+            0.0090566361322999,
+            -0.0019864458590745926,
+            0.01325240544974804,
+            -0.012198156677186489,
+            -0.00334494118578732,
+            0.0031592093873769045,
+            -0.011589664034545422,
+            0.008158047683537006,
+            0.01978308893740177,
+            0.008193425834178925,
+            -0.009339655749499798,
+            0.003422771580517292,
+            -0.011922212317585945,
+            0.013500047847628593,
+            0.015537790954113007,
+            0.018311386927962303,
+            -0.006498843897134066,
+            0.0020412809681147337,
+            -0.01370523776859045,
+            0.0063962494023144245,
+            -0.009827865287661552,
+            -0.004878554958850145,
+            -0.0005333155859261751,
+            0.004921008367091417,
+            -0.011179285123944283,
+            -0.0025772498920559883,
+            -0.026830283924937248,
+            -0.0013823752524331212,
+            -0.020533092319965363,
+            0.005303084850311279,
+            0.023207630962133408,
+            -0.007514177821576595,
+            0.0006045127520337701,
+            -0.0039092120714485645,
+            0.014476466923952103,
+            0.020688753575086594,
+            0.00870286114513874,
+            0.00482548912987113,
+            -0.006081389728933573,
+            0.02179253101348877,
+            0.011603815481066704,
+            -0.017816102132201195,
+            0.00164239970035851,
+            -0.001509734196588397,
+            -0.0014336726162582636,
+            0.011625041253864765,
+            0.004443412180989981,
+            -0.020844414830207825,
+            0.0028602697420865297,
+            -0.010089659132063389,
+            0.007818424142897129,
+            0.014702882617712021,
+            -0.00031906069489195943,
+            -0.02486329711973667,
+            -0.009827865287661552,
+            0.021919889375567436,
+            -0.015070809051394463,
+            -0.0008291598060168326,
+            0.00764861237257719,
+            0.023264234885573387,
+            -0.024792540818452835,
+            -0.006831392180174589,
+            -0.0023455272894352674,
+            -0.003933976404368877,
+            -0.005193414632230997,
+            0.0010489424457773566,
+            0.0214670579880476,
+            0.004415110219269991,
+            -0.004234685096889734,
+            0.005724077112972736,
+            0.006470541935414076,
+            0.02325008250772953,
+            0.0040577976033091545,
+            0.028047269210219383,
+            -0.00943871308118105,
+            -0.0023809047415852547,
+            -0.0019988277927041054,
+            -4.775960405822843e-05,
+            0.0160047747194767,
+            -0.005430444143712521,
+            0.020320827141404152,
+            0.009445788338780403,
+            0.0036934094969183207,
+            0.016514210030436516,
+            0.02319347858428955,
+            -0.006767712999135256,
+            -0.004793649073690176,
+            0.019160445779561996,
+            0.0053667644970119,
+            -0.008858522400259972,
+            -0.02194819226861,
+            0.01141277700662613,
+            0.0014124460285529494,
+            -0.0006956097786314785,
+            0.012106175534427166,
+            0.0024375086650252342,
+            0.006300730165094137,
+            0.008172199130058289,
+            -0.016684021800756454,
+            0.004241760354489088,
+            -0.004698129836469889,
+            -0.01860855706036091,
+            0.010910416021943092,
+            -0.012092024087905884,
+            -5.201595922699198e-05,
+            -0.0010002984199672937,
+            0.014165145345032215,
+            -0.01703779585659504,
+            -0.0019103841623291373,
+            -0.0070719593204557896,
+            -1.3024719009990804e-05,
+            -0.0009162768837995827,
+            0.008320784196257591,
+            -0.006254739128053188,
+            -0.010938717983663082,
+            -0.010938717983663082,
+            0.027509532868862152,
+            -0.004991763271391392,
+            0.003916287794709206,
+            0.019330257549881935
+          ],
+          "embedding": [
+            0.007204997251390758,
+            -0.03173324616100502,
+            -0.020505790884701024,
+            0.0037451608231038273,
+            0.009817085391002229,
+            0.026883177166072393,
+            -0.016477826654817607,
+            0.06949311853773875,
+            -0.01224876667472692,
+            0.031952566873802865,
+            0.013633671951171586,
+            -0.005041437355104989,
+            0.02663392750069342,
+            -0.022099132865267587,
+            0.006665702000169926,
+            0.0030467129715919033,
+            0.007931802508415542,
+            0.009984477355413895,
+            -0.010362836464183985,
+            -0.03180848181251013,
+            -0.013542377333474449,
+            -0.013572345022577053,
+            -0.03979985300694044,
+            -0.007698412130292187,
+            -0.005953452234356532,
+            0.018890228917932694,
+            0.02217446994660065,
+            0.005158357501784182,
+            -0.05468694715835175,
+            0.00789994048345594,
+            0.017130184174786147,
+            -0.0026591475669416114,
+            -0.02081141846859738,
+            -0.008403928318906045,
+            -0.030925346724945978,
+            -0.03855023691059801,
+            0.038853354785097736,
+            0.01612011653434478,
+            -0.035635702395196575,
+            0.026266623757525934,
+            0.043450373922427316,
+            -0.007254334835408145,
+            -0.015376804869420366,
+            -0.0017276202867766907,
+            0.004650414207094434,
+            -0.002365514361535186,
+            0.011906139905776183,
+            -0.0005664194302009289,
+            -0.0191723447249157,
+            -0.025546124518037998,
+            0.01025005257280377,
+            -0.014436532079436437,
+            0.029277529346694118,
+            -0.0045414983847733344,
+            -0.02260315456913697,
+            0.01717918656248626,
+            0.013666132349359846,
+            0.057804090189845364,
+            0.056600576093100735,
+            0.051208022172527734,
+            0.03553760896972431,
+            0.02094599510567085,
+            -0.02438674472832137,
+            -0.0027797634437333432,
+            0.03614248430646102,
+            -0.026729308357113124,
+            -0.002251394534452815,
+            0.018929251842788906,
+            0.00689697483318378,
+            -0.026603606434115974,
+            -0.0541443728344337,
+            0.027164630314707807,
+            -0.004335272389193875,
+            -0.013967321193992958,
+            0.003323121377857836,
+            -0.0245295956412367,
+            0.024489335129134787,
+            0.06212059558640156,
+            0.009214058207162371,
+            -0.002082293435905329,
+            0.013659934201796686,
+            -0.003344037927706573,
+            -0.00689041317919184,
+            0.020235785390418023,
+            0.0212739457082623,
+            0.0016782346585238381,
+            -0.013484221664059555,
+            -0.06465582506604173,
+            0.011882797526607326,
+            -0.032970655964890155,
+            -0.04027700439305653,
+            0.025396847912528105,
+            0.013861157517716707,
+            0.00604596157583048,
+            -0.00744611407420936,
+            0.04038084298991919,
+            0.021507532290328625,
+            0.005983822806623904,
+            -0.012711110369915742,
+            0.02410138142963901,
+            -0.018372791517880082,
+            0.021016674128298246,
+            0.013387285956415756,
+            0.02039383525127615,
+            0.010700292312227496,
+            0.0053934660497349735,
+            -0.014807678014449691,
+            0.009422805142740376,
+            0.0414357625490679,
+            0.12206398267002624,
+            -0.010877509834324155,
+            -0.011079302503786522,
+            -0.04107714853756596,
+            0.006631397986165343,
+            -0.031914267104222384,
+            -0.0179965295457466,
+            -0.057530036803978266,
+            -0.0320488164246567,
+            0.008208202363320509,
+            -0.0020203090915064162,
+            0.03502055780919601,
+            0.04464599164916666,
+            0.019332152079287082,
+            0.017133943087431935,
+            -0.039885565008919065,
+            0.04703059988373572,
+            -0.029370696964304416,
+            0.0008494086835971101,
+            0.0014082189719987074,
+            -0.030973762582071615,
+            -0.006561157852024518,
+            0.011634653995510726,
+            -0.019896826887411713,
+            -0.034275921680252924,
+            0.026274465530039866,
+            -0.007728234125236442,
+            0.04814514040690998,
+            0.06550407296280117,
+            -0.02749051703615343,
+            -0.010041099502187074,
+            0.02019601967763537,
+            -0.01871701432558228,
+            0.008532829759864928,
+            -0.020164138072967613,
+            -0.045196925901010855,
+            -0.004523517988516156,
+            -0.002113469149506399,
+            0.07929650663710812,
+            -0.01111269708042164,
+            0.03358254444720164,
+            -0.019973382759721527,
+            -0.0007145592863325417,
+            -0.0011740862928390678,
+            0.02166182480535788,
+            0.004543228023398081,
+            -0.013377688428761516,
+            -0.01758250073647576,
+            0.009548292014596834,
+            -0.04753617258004735,
+            -0.007125744019792479,
+            -0.009930358894299303,
+            -0.007931672044235996,
+            -0.024561936537429774,
+            0.012979195012881556,
+            0.009209145250061479,
+            0.04586756662401364,
+            0.04311807301838248,
+            -0.0036241105681042675,
+            -0.06579256720896123,
+            0.04967191755416882,
+            0.015475886231466817,
+            0.03157651656768371,
+            0.02515941109300714,
+            0.01773554572598468,
+            0.005824418191900522,
+            0.016317635547562007,
+            0.007650872352957098,
+            0.010015166465492963,
+            -0.033203307142867844,
+            0.040272188258443496,
+            -0.013408423788025047,
+            0.01036004294342104,
+            0.012391622448703885,
+            0.04214857862444935,
+            0.03418032388760535,
+            0.015263018079889338,
+            0.018786704580603317,
+            0.02704685343985817,
+            0.015348111464412869,
+            0.0013031600434754704,
+            0.03241427528588285,
+            0.011689924942195664,
+            0.04544406114742012,
+            0.016186615914968734,
+            0.011624500626103958,
+            -0.008272953078864385,
+            -0.02363080189189375,
+            0.02784298119317616,
+            -0.010585980875564109,
+            0.01455013604305903,
+            0.01749228557615982,
+            -0.054742056843690476,
+            -0.005876213780171779,
+            -0.010556605098730201,
+            0.003967456397644932,
+            0.014663888851767762,
+            0.0015548476559947807,
+            -0.003858367677774835,
+            0.012504286824484335,
+            -0.025806882498628832,
+            -0.014216299453397403,
+            0.0031620924516417956,
+            -0.006448390918604307,
+            -0.02785577502346934,
+            -0.00766402687921159,
+            0.005275000635711884,
+            -0.0090448751942,
+            0.019702662630533,
+            0.007009091837467867,
+            -0.01847307100043082,
+            -0.0006774105206839772,
+            0.032952656101827446,
+            0.015900033044760432,
+            -0.02170121261931376,
+            -0.017964879081469612,
+            -0.005062849745084668,
+            -0.03171847232054206,
+            -0.038177398521606544,
+            0.029426291950046455,
+            -0.025687201683643784,
+            0.030158867757101797,
+            0.015780411479509624,
+            -0.007011484049169482,
+            -0.012114023533118393,
+            0.020661407942701628,
+            -0.025067530380881376,
+            0.009777190971413269,
+            -0.004160771288889298,
+            -0.028825369485727344,
+            0.008307290529469729,
+            -0.014415903546305744,
+            -0.007561235629544945,
+            -0.03201704135271719,
+            -0.015182869388110948,
+            -0.00501561450194676,
+            -0.0003863746169672397,
+            -0.01073403445816663,
+            -0.03321103354401307,
+            -0.002304357889348034,
+            0.017001213960120047,
+            0.00028222146729658097,
+            -0.04002882852650754,
+            0.014509588052058002,
+            -0.004150541038999649,
+            -0.034333837380045466,
+            0.0564137596224324,
+            0.03584898691608747,
+            -0.006830698606679202,
+            0.006121456445207273,
+            -0.04505674213649008,
+            0.04358467937307492,
+            0.010847859073780811,
+            -0.04918983713702805,
+            0.049190442935189385,
+            -0.00528960443684769,
+            0.01419195371129309,
+            0.040316024601793406,
+            0.004600843996974165,
+            -0.006780970176765864,
+            0.005206651850992678,
+            0.03397517915308201,
+            -0.010037223444627569,
+            -0.027377913828275806,
+            -0.03328136706915221,
+            0.02562568440972934,
+            -0.03485770844248482,
+            -0.020069979230687374,
+            -0.0052041408068332524,
+            0.029818953377223505,
+            0.028789460147370508,
+            0.004071274130619537,
+            0.01928375754300958,
+            0.0014970947734202088,
+            0.010502599795027616,
+            0.016914145702515263,
+            0.011942576738188502,
+            -0.014909953528043828,
+            -0.003432489584767786,
+            -0.05733768990013957,
+            0.0030710780388681002,
+            -0.02627280695622545,
+            0.007930906480371135,
+            -0.031227917075485834,
+            0.013132797714872345,
+            0.002514351081823354,
+            -0.007152021475569949,
+            -0.0066245205132327055,
+            -0.005279224195782313,
+            0.028512607222275155,
+            -0.00921960825771865,
+            -0.0359704105589433,
+            -0.021347597388353182,
+            0.011200044645812517,
+            -0.0004171294784927543,
+            -0.016784468873218786,
+            -0.0018746544944553878,
+            0.007749103417240419,
+            0.02390272097796833,
+            -0.005128540108845805,
+            0.006397849195755672,
+            0.01214922062611769,
+            0.01163371266250103,
+            0.012307186722130844,
+            -0.004982589662882946,
+            -0.016343724553037307,
+            0.019806887143115643,
+            0.013139657463054087,
+            0.0205801609265216,
+            -0.03767676007629612,
+            -0.0249879011550972,
+            0.05547392586363707,
+            0.005903313549319051,
+            -0.05568806298774435,
+            0.018827693208840233,
+            -0.007931178941307645,
+            0.008183992600768387,
+            -0.010319022946041053,
+            0.05111352384595183,
+            -0.024897203290807753,
+            -0.02096678205188722,
+            0.006436523335212348,
+            -0.04917084523640266,
+            -0.010459284852442706,
+            -0.011138558192296276,
+            0.014789370043397917,
+            -0.03105725978752438,
+            -0.004934745004818259,
+            0.015595801375817122,
+            -4.687585727052446e-05,
+            0.0028886067775750254,
+            -0.00850352807499054,
+            -0.0034885579171087815,
+            0.016339681487281293,
+            -0.0034050317046826813,
+            -0.008309570448487812,
+            -0.003615165868022203,
+            -0.027638409919916757,
+            0.023352910161586186,
+            0.026364511529991978,
+            0.001249936082895606,
+            0.001987948967948376,
+            0.01756629246122769,
+            0.004656075485833687,
+            -0.006752579977941239,
+            -0.013681608638716132,
+            -0.03405936202918687,
+            -0.0401867651078737,
+            0.022991180183813725,
+            -0.02318908140203705,
+            0.018299512436825923,
+            0.00904081847012441,
+            0.019697091729447287,
+            -0.019236130984371835,
+            0.015159917756323988,
+            -0.0387297415849285,
+            -0.018793498297161176,
+            -0.005256288213225733,
+            -0.010414697711744128,
+            0.0007039894599589104,
+            -0.01739645696111503,
+            -0.013728845328097506,
+            0.022709522955937343,
+            -0.022789243441781803,
+            -0.0036763802589970026,
+            -0.01429671717544519,
+            0.0023674126354206932,
+            0.049448298344058644,
+            -0.009926033487593551,
+            -0.023371285193090733,
+            0.020846792916650956,
+            -0.026996519120163966,
+            0.011246908457705283,
+            0.04091365159640201,
+            0.005469162850197613,
+            0.005640919201612091,
+            0.029575753750229278,
+            -0.0007860452837141593,
+            -0.0052805237594885516,
+            -0.009782745443933828,
+            0.021178503581190618,
+            -0.0009176512672832309,
+            -0.05245650420332918,
+            -0.007698364493410058,
+            -0.039027418406703554,
+            0.005448840777995117,
+            0.017581625031764185,
+            0.025522517097715556,
+            -0.020775492472580457,
+            -0.015906320383873178,
+            -0.000643669839811819,
+            -0.0086896084429604,
+            0.0029399233482444638,
+            0.014381240985647907,
+            0.021086693776922936,
+            0.002436575081260609,
+            -0.004913514868101904,
+            -0.011582101006213432,
+            0.014400297663309155,
+            0.0467181039341492,
+            -0.01722692180137712,
+            0.0011622224560709315,
+            -0.013036540368441797,
+            -0.06208454638917413,
+            -0.019160212331432766,
+            0.0351496273293468,
+            0.02429814055116254,
+            -0.009896662169762772,
+            -0.01647101145477158,
+            -0.002915821099802562,
+            -0.03557863734177486,
+            -0.01786433999615293,
+            -0.010699364747311715,
+            0.02390901423299708,
+            0.007213809581960646,
+            0.0005601364043388203,
+            0.00916614601011734,
+            0.014737491634516238,
+            0.019303599465763203,
+            0.008864535607492504,
+            -0.0575839606069681,
+            0.018051837786406608,
+            0.006315830080225406,
+            0.02525619013082567,
+            -0.015671924543113065,
+            0.003988943108210209,
+            -0.03179878281917948,
+            -0.029440641244429775,
+            0.040586322748158446,
+            -0.03418722657593859,
+            -0.030650542761032924,
+            0.006566745611226466,
+            0.009571089937153852,
+            0.004491158558315032,
+            -0.011352813042040581,
+            -0.02535505987721779,
+            -0.013381948772812133,
+            -0.030918676056735445,
+            -0.02192708443081135,
+            -0.005232667170255381,
+            -0.03342408726436631,
+            -0.07498634115550039,
+            -0.006245638452568239,
+            -0.049946896465756174,
+            -0.03900385398225758,
+            0.04458524125677807,
+            -0.006106489377959481,
+            0.02398410968928338,
+            -0.04615183642824994,
+            -0.01165382599292015,
+            -0.04890013373371541,
+            0.013624324077017377,
+            0.01476398305411351,
+            0.020202554298470666,
+            0.00014553149242532853,
+            0.036224316512987846,
+            0.00021498496267774312,
+            -0.05055208658013767,
+            -0.057482169168904106,
+            -0.003960451575823462,
+            -0.010217078509789989,
+            -0.005593738972150311,
+            -0.034090261748104665,
+            0.05835341009022349,
+            0.0116359661127526,
+            0.04203678184748103,
+            0.03447818335186531,
+            -0.006747564131379555,
+            -0.025756962754841885,
+            -0.028123416424745064,
+            -0.008073625379614702,
+            0.047544241458556825,
+            -0.029605671083240564,
+            0.008171081332993893,
+            -0.026007452205474395,
+            0.0067084589781962805,
+            -0.0070645448617940995,
+            -0.0012298101311792947,
+            0.012372286141738867,
+            0.006377319686763927,
+            -0.015024537877377194,
+            0.007934745664790507,
+            0.026041755081605937,
+            -0.007303731324541215,
+            0.0407000170257006,
+            -0.012990703097937777,
+            -0.033189084013863494,
+            -0.022525679964848666,
+            0.01657717698191603,
+            0.03440284885162039,
+            0.010230633738702899,
+            -0.008707808754455952,
+            -0.0363777020540216,
+            0.011184076269109602,
+            0.031774012431843524,
+            0.01504703222643543,
+            -0.022509833303978505,
+            0.025230974050270107,
+            0.017502662694813216,
+            -0.018700579139537456,
+            -0.03511734747709348,
+            0.015162183119327901,
+            -0.017711638132862063,
+            0.018190335050681287,
+            -0.01590174153215615,
+            0.011103110869700727,
+            0.0025189370696524936,
+            -0.00920305384736575,
+            0.01929652151514404,
+            -0.006196715938358087,
+            0.011426611271994685,
+            0.007058186102245829,
+            -0.018191352599117545,
+            0.0027202185072220937,
+            0.057310566886380025,
+            -0.000531639834386418,
+            0.01494975681909879,
+            0.03845692752400366,
+            -0.001971953300066856,
+            0.045554909803324765,
+            -0.02532332999916258,
+            -0.042616337063845953,
+            0.02633963974172818,
+            -0.02970705577683783,
+            -0.01624154998005974,
+            -0.02620993449678864,
+            -0.021490817420853802,
+            -0.004038424045768319,
+            0.009940887982818171,
+            -0.02692933845395842,
+            -0.023801570856681796,
+            -0.001948460540662095,
+            0.02290911497874141,
+            -0.03300532378029689,
+            0.006525488380791647,
+            0.007039306647666766,
+            -0.003685585408710752,
+            0.03183735852538168,
+            -0.027197064244321705,
+            0.006861928286688311,
+            0.011209869849682412,
+            -0.001094897323854524,
+            -0.005328676992017907,
+            -0.004007474089818478,
+            0.010796946231051235,
+            -0.030543173497906445,
+            -0.012879972141706173,
+            0.00935894527375688,
+            0.0017573071057611829,
+            -0.010050882212831222,
+            -0.03102132152986303,
+            -0.002915141291985554,
+            0.007634137769426993,
+            -0.011094093612593696,
+            0.01199609810193786,
+            -0.010458016758814742,
+            -0.04929347607956551,
+            0.010973132159729983,
+            0.028148117094907874,
+            -0.025504019123994758,
+            0.012470127357818321,
+            0.009182729379865986,
+            -0.016020133756226275,
+            0.016320111225714855,
+            -0.012051271446461976,
+            0.024098195821827578,
+            0.04317620443481784,
+            0.024592886889234396,
+            0.007727787037954994,
+            0.00771746776156926,
+            -0.0003147538723468374,
+            0.022572868970938624,
+            -0.016683340264320543,
+            0.013672085891597506,
+            0.020827021901755415,
+            0.02092309219721837,
+            0.0057130785714458055,
+            -0.003901728981013356,
+            -0.0036968415923910314,
+            0.002227024825579113,
+            -0.02106678364798051,
+            0.014578546755457675,
+            0.03128101864424301,
+            -0.005234627629617124,
+            -0.022394744692442837,
+            0.002430617645746389,
+            -0.015780389432069438,
+            4.7277885648917e-05,
+            0.013357803020920043,
+            0.013777651543690907,
+            -0.027713499926132385,
+            -0.0018999112228756628,
+            0.028364630573666852,
+            0.009146793705085586,
+            -0.009891297866910313,
+            -0.01644269527910728,
+            -0.0003903768506971414,
+            0.006512217314869576,
+            0.02408785204739448,
+            0.0018406499545367782,
+            -0.007360473750031044,
+            0.01484497577088404,
+            0.016723277095327158,
+            0.003629911330658209,
+            -0.0004820567356447713,
+            0.02394406451074397,
+            0.023794019615473026,
+            0.001632266615911801,
+            -0.02064379152040184,
+            -0.02755693521684878,
+            0.027335938400284726,
+            0.02769248863677192,
+            -0.006832353240623308,
+            0.027324973360715134,
+            0.003937207909967385,
+            0.013580154030133622,
+            0.02076818768224654,
+            -0.01784151382870951,
+            -0.017617903734311997,
+            0.0017316476451235418,
+            -0.009605294810027804,
+            -0.009829196696575069,
+            0.004891780008779742,
+            -0.011655402383771472,
+            -0.01124379942906224,
+            0.043205454954628145,
+            0.004713701851783307,
+            0.003560551284131962,
+            0.007894840447501912,
+            -0.018881214526749082,
+            -0.00945538175144614,
+            0.0019040259741796817,
+            0.029505863720301916,
+            -0.001400002868254808,
+            0.03315866433970851,
+            -0.00830542045504805,
+            0.008602408964961391,
+            -0.03202191233287069,
+            -0.006978289755114341,
+            0.0326499698728755,
+            0.015314305362524091,
+            0.015104436300213667,
+            0.0021257042489266787,
+            -0.0010840650756005028,
+            0.02434348355847526,
+            -0.007174433778817674,
+            -0.04839373063710409,
+            0.022696994450018916,
+            -0.004779511451775743,
+            0.023588216956749154,
+            0.0382733980620746,
+            -0.004232286553944853,
+            0.003815507826545929,
+            0.023531223314846965,
+            -0.01887650634213847,
+            0.015055403379782985,
+            -0.01918944132635701,
+            -0.017597900721414636,
+            0.02176630010543207,
+            -0.008896608357798277,
+            0.01732387427588561,
+            0.01764363756727683,
+            -0.012180043444768312,
+            -0.019003360499321047,
+            0.01229139850966224,
+            0.0025710091163655626,
+            -0.017075658234265562,
+            0.01797412486853335,
+            0.018372010308966135,
+            0.006723179570342314,
+            0.015417677746651683,
+            -0.00047634684109332134,
+            0.017608296628666153,
+            -0.001049984114818522,
+            -0.008416990841821452,
+            0.01827827907328822,
+            -0.009568717277044265,
+            -0.01658089563620308,
+            0.009808478422454319,
+            -0.05193977747210182,
+            0.010644335931025957,
+            0.0016781339270411392,
+            -0.01562285100294917,
+            0.00405036330963539,
+            0.007102724536527262,
+            0.005195069570261247,
+            0.0028773630901681767,
+            0.004610474981005765,
+            0.042921486858787584,
+            0.013213672397112759,
+            -0.0064030584930089415,
+            -0.010711457508461704,
+            -0.017480309425763254,
+            -0.027151012983864797,
+            -0.005204425551873271,
+            0.009927884476991038,
+            0.022601230152836282,
+            0.008046297278661014,
+            -0.0031858480011267704,
+            -0.02589628103025927,
+            0.019723430185137954,
+            0.03604835658149313,
+            0.062423099201768716,
+            0.010930268386637612,
+            0.02925564269324381,
+            0.02439517070384333,
+            0.023930432096006114,
+            0.013036828091946605,
+            0.006910311087503408,
+            -0.032995841694293995,
+            0.012813650252889403,
+            0.01333714655191133,
+            0.03678966264802025,
+            0.012571138353467183,
+            -0.05610151838497368,
+            3.0777966587107344e-05,
+            0.009791774904454618,
+            0.041959488802722524,
+            0.01956296954085397,
+            0.000155217569370113,
+            -0.007285602454264675,
+            0.031547870838711856,
+            -0.026518385135879206,
+            -0.012299593280363843,
+            -0.0026551378470968594,
+            -0.006190117573235779,
+            -0.003807907001161504,
+            -0.004991957475076049,
+            0.036983035940964804,
+            -0.015648565314743963,
+            -0.019256463332683493,
+            0.0004300362961407231,
+            -0.0015686065121077472,
+            0.0011311524136286674,
+            0.0032236641835276464,
+            0.013795345278976164,
+            0.015385029265915279,
+            -0.004800448639353721,
+            0.026514926384796363,
+            -0.008699453553651681,
+            0.022186817580012775,
+            -0.013427553742493343,
+            -0.011927259975371873,
+            0.017776228476138103,
+            -0.021243797085501972,
+            -0.01667280589187595,
+            -0.003688045116580565,
+            0.006189970874536939,
+            0.0020911968266404467,
+            0.0006097761986763103,
+            -0.004685468039463817,
+            -0.01011357666327168,
+            0.022631082517271542,
+            0.012398000826637277,
+            -0.006721588925828951,
+            0.018142966317821054,
+            -0.0021330638043572654,
+            -0.011869184647489572,
+            0.025073455250141923,
+            -0.0015392945266926425,
+            -0.0337085890598252,
+            0.01267302113143573,
+            -0.013438601165482187,
+            0.02673290895281225,
+            -0.009586762706902736,
+            -0.011208527168019762,
+            0.019144928964011168,
+            -0.0017591316941248897,
+            -0.005459150067129732,
+            -0.029230079356228025,
+            0.004679497471366242,
+            -0.011723195942933286,
+            0.015899377439148873,
+            0.0022054234777254627,
+            0.029187759252864872,
+            -0.018290694895057157,
+            0.00792533824593637,
+            0.011719198071061566,
+            0.014612375997487872,
+            0.005419943379355563,
+            0.010515265723144554,
+            -0.006893314445458116,
+            -0.04424495426064596,
+            5.8241472764594054e-05,
+            -0.005719143565673274,
+            0.0038220021965914754,
+            0.013106940982933836,
+            -0.01203955335201393,
+            0.011260626096963215,
+            -0.0279896016489392,
+            -0.0295228261396555,
+            0.036474811902902415,
+            -0.04900747926861378,
+            0.019261089742811332,
+            0.00014104646541594166,
+            -0.03057197731717424,
+            -0.04331886560073797,
+            -0.03232957159604667,
+            -0.019401402365464362,
+            0.004102283356333861,
+            0.0002610070704924167,
+            0.0012280372705500212,
+            -0.014025012323929194,
+            -0.014135990809256075,
+            0.013269444858490211,
+            0.00992155580941576,
+            0.011225162609266308,
+            -0.006545044517282357,
+            0.002368862850663672,
+            -0.02063183105457681,
+            -0.026981383275694142,
+            -0.02774194980145225,
+            0.013337303400678109,
+            -0.0025902548982556788,
+            -0.02794339093668326,
+            -0.025606606546251152,
+            -0.0029852498102770587,
+            -0.01906873442473352,
+            0.006508845592362198,
+            0.03268861529625085,
+            0.0355975143313428,
+            0.010066906387952277,
+            0.01690250467284985,
+            -0.017201544758038183,
+            0.048345309114140714,
+            -0.019031171948182872,
+            0.03188329748476941,
+            -0.0005845169448621133,
+            -0.024140202991373587,
+            0.00011597263700677706,
+            0.03203623528903764,
+            -0.019285565857567896,
+            -0.01332755507465857,
+            -0.010914187315481855,
+            -0.003374912271519852,
+            0.02419326154642231,
+            0.02443825785057301,
+            -0.0037923303809811746,
+            0.00022800399604730175,
+            0.015893265651968885,
+            0.003154527829977343,
+            0.007688529734165207,
+            0.030623379985627763,
+            0.014052282574575454,
+            0.01127235345371133,
+            -0.04164707563955819,
+            -0.011811587719246128,
+            -0.003763086694358232,
+            0.005909410979841885,
+            0.002913220159138154,
+            -0.002671613084376487,
+            0.0045765194162907885,
+            -0.007241703279856166,
+            -0.005855949422953439,
+            0.0016973541436777935,
+            -0.001576326424468957,
+            0.008393873711385013,
+            -0.0037217094191171083,
+            -0.03498879678492834,
+            -0.026096985039773694,
+            -0.021280075788049788,
+            0.006437497348154639,
+            0.0070214556591898375,
+            -0.00972721128392027,
+            0.01949553762684561,
+            -0.0029404319702066146,
+            0.000271037702827666,
+            -0.010734867909488861,
+            -0.0029784341515646075,
+            -0.01206464664516104,
+            -0.019544778160755956,
+            0.00035139045159739705,
+            0.013457601448003152,
+            0.00025947376767512006,
+            -0.03165522602526639,
+            0.011979811705146604,
+            0.006642849985850598,
+            0.013181162886675883,
+            -0.024499404540711422,
+            -0.02310714308893576,
+            -0.028028912907092744,
+            0.010570122662701183,
+            -0.0066866321204157725,
+            -0.024617571284817428,
+            -0.004097807033448707,
+            -0.0074576460628928256,
+            -0.0020417256533259887,
+            -0.02470953739335365,
+            -0.011408461668188081,
+            0.023537349682260345,
+            -0.006373652338136973,
+            0.01936338676879376,
+            -0.017636435070085086,
+            -0.017036091876814843,
+            -0.002198685044404603,
+            0.015584758061011048,
+            -0.038943923931820196,
+            -0.002076529978164001,
+            -0.008452513321945455,
+            0.006551055965075869,
+            -0.028840587857625097,
+            -0.007128279504360441,
+            0.0199397499488538,
+            0.012424319366271584,
+            -0.03518875279748495,
+            0.013596027106451115,
+            0.022191403935861113,
+            0.01218507466930854,
+            -0.0038551718784437783,
+            0.03910075115429626,
+            -0.0025215641203030336,
+            -0.005422688956956598,
+            0.016759888275451905,
+            0.04462538468871483,
+            0.023708097134660096,
+            -0.03152743757744915,
+            -0.008668774531128668,
+            -0.015224825785549947,
+            0.004102648962663789,
+            -0.025867795193966987,
+            0.004676262922216584,
+            -0.011739753544166856,
+            0.01645951757031186,
+            0.011253400919230348,
+            0.013958707917471253,
+            0.0376596076542735,
+            -0.005077354977296842,
+            0.02668910096392324,
+            0.0010332284250006117,
+            -0.00856723931523911,
+            0.0014431148305642308,
+            -0.01412579094241589,
+            0.029892393553325205,
+            0.02269599036836208,
+            0.014421409701746036,
+            0.029248195108770955,
+            -0.03588867295960255,
+            0.02458655544632528,
+            -0.0010013539341895672,
+            0.008368648583431747,
+            0.018150734970019018,
+            0.02628022565029391,
+            0.015423867373844644,
+            0.012477080028047823,
+            -0.02513018750675568,
+            -0.022180001014682045,
+            -0.0027550350630288997,
+            -0.031944843677988374,
+            -0.0014668093006914267,
+            -0.009789309083948147,
+            0.026386027599666254,
+            0.0046988411391149,
+            -0.019803160322638583,
+            -0.003931806845108628,
+            0.004482097428470392,
+            -0.00244660100774939,
+            -0.010441732346892935,
+            -0.004649318716095894,
+            -0.01869464992812615,
+            -0.017838643679294407,
+            0.023677878122153204,
+            -0.023958455640817666,
+            -0.025238725541146534,
+            0.014582798102040157,
+            0.0013223590277629197,
+            -0.002295641913710489,
+            0.0008238518461567685,
+            -0.02115533275282518,
+            -0.04214275933987015,
+            -0.027817083427786927,
+            0.0037588803464689412,
+            0.03565056002940463,
+            -0.026788865866928923,
+            0.014162550819567465,
+            -0.011336583458519,
+            0.019175861037658344,
+            -0.005127229083828615,
+            0.015531116410874209,
+            -0.018251290727445846,
+            -0.044630374763679266,
+            0.021833307809708984,
+            -0.012087009688208446,
+            0.02720742725750304,
+            0.021497809277370574,
+            -0.026937653159100156,
+            -0.013572733244188407,
+            0.01358214612023832,
+            0.0020408253706382332,
+            -0.009652962552750464,
+            -0.0063411929391088626,
+            -0.010533309167065193,
+            0.003901271588567942,
+            -0.04496345296706673,
+            -0.008761188928422701,
+            0.06948704056954039,
+            0.005916180375965137,
+            0.03157905519638486,
+            0.014668818318138927,
+            -0.028448496686381258,
+            -0.04008949536631331,
+            -0.00462253589823598,
+            -0.0003701522833020314,
+            -0.00024410762276269235,
+            -0.01850585249309217,
+            0.017699615637510956,
+            -0.002712069912576265,
+            -0.009767338815933488,
+            -0.013119464222297351,
+            -0.013743610418034631,
+            -0.030375365173543157,
+            -0.0030316327877227783,
+            -0.008632689470358149,
+            -0.006565457264917239,
+            -0.0328089275944709,
+            -0.011713996290006943,
+            0.029773910045828737,
+            0.02740100904353622,
+            -0.0061926915497348915,
+            0.00638737826122608,
+            -0.010244154924811975,
+            0.0010396203272677069,
+            -0.001264454445070139,
+            -0.004059969507786729,
+            0.008715730726831364,
+            0.0014767015415289076,
+            0.031074767756160294,
+            0.004761858344125113,
+            0.00987870172044607,
+            -0.059926117793699066,
+            -0.009294021263133636,
+            0.021188432871397475,
+            0.016602826531924377,
+            -0.009125784136292983,
+            0.013946188378030194,
+            -0.0017670794602549428,
+            -0.006837334179368295,
+            -0.028290055353601393,
+            0.024556541126233806,
+            0.04543639597357669,
+            0.0031311343860416473,
+            -0.011199639773247286,
+            0.014965677855595444,
+            0.0018248533506196013,
+            0.015507721118783542,
+            -0.011409402378400194,
+            -0.01338737990904486,
+            0.03063410115541065,
+            0.009305362935335817,
+            0.0025578631174065385,
+            0.012727648343848194,
+            -0.0038680554900795763,
+            -0.01173215289303824,
+            -0.01568862495293563,
+            -0.004101703129244437,
+            -0.03649200408657064,
+            -0.0022107849490954137,
+            0.009659780679763728,
+            0.0034091693454195806,
+            0.008848380026939594,
+            0.01842435611499587,
+            0.03546926214224266,
+            0.013478278853234764,
+            0.005970293629431872,
+            -0.0069197533761813455,
+            0.023196445250565058,
+            0.010403914879205351,
+            0.010017272564548626,
+            0.015076807790462658,
+            -0.029649260409273275,
+            -0.012737152920662164,
+            0.00373276777994811,
+            0.013239909450830258,
+            0.0001682544254742693,
+            0.02030652051051447,
+            0.014143961923579586,
+            0.0010147733405793957,
+            -0.012509446388074923,
+            0.0020437769941487626,
+            0.007405578085797006,
+            0.013458276946120546,
+            0.0006779114540819997,
+            -0.004250745498730309,
+            0.0009149988500358458,
+            -0.008355906072311008,
+            0.040391939952386505,
+            -0.01831218885700067,
+            -0.016793844108223176,
+            0.035064882870542116,
+            0.11416006426189942,
+            0.0010630856112440689,
+            -0.015660014600661857,
+            -0.017142534874931403,
+            -0.0086899013413391,
+            0.016611825572132997,
+            0.01609650977226868,
+            0.009780106186629793,
+            0.009622367309076686,
+            -0.000672926017804395,
+            0.024310949465350565,
+            0.02944698490225121,
+            -0.0037169531700178868,
+            -0.010990838766452194,
+            -0.036819646475041955,
+            0.022923962979125287,
+            -0.004596366586768123,
+            -0.007108041562635549,
+            -0.002604794313552557,
+            0.012641223813290853,
+            0.013668775645501733,
+            0.005125100040230003,
+            0.0077993841846437815,
+            -0.011500851674334447,
+            -0.017334458072700107,
+            0.0056819751688881774,
+            -0.015982234644378358,
+            -0.0076533013358874935,
+            -0.013127172654362694,
+            -0.011837346554842119,
+            -0.006562312362371076,
+            0.0029877420008976425,
+            -0.008572723854050838,
+            0.00854083969164562,
+            -0.010939639783598493,
+            0.016185642759084087,
+            -0.008052282332943615,
+            -0.0011992212936609377,
+            0.0010378646897070758,
+            -0.0025399515501741718,
+            0.010712865639618352,
+            0.01451357949543628,
+            -0.02521401163091652,
+            0.020836357346214075,
+            -0.008252813367155418,
+            0.0006409807249997337,
+            0.015338809580802425,
+            -0.04672526649678653,
+            0.022728876784937357,
+            0.005129225753457052,
+            -0.018372222092794564,
+            0.0066182736895667805,
+            -0.02048375780196863,
+            -0.00896382274306482,
+            0.0021738974344880794,
+            0.004491167150658828,
+            0.0034280799511763956,
+            0.015611133773752265,
+            0.018126362497548308,
+            0.00014383671280220997,
+            0.007927708693242186,
+            -0.006973076292062532,
+            0.008118313055163714,
+            -0.006791836664375275,
+            -0.01164567445668625,
+            -0.010894577039363014,
+            0.00014731542306100506,
+            -0.008865132658571428,
+            -0.011415696755798914,
+            -0.017922318360382588,
+            0.00358538001727405,
+            0.00289327637128695,
+            0.010406054711003452,
+            -0.007757607207626101,
+            -0.010994775479337082,
+            -0.011716096880261664,
+            -0.0013333474316266438,
+            -0.01231144419953554,
+            0.009792603083441917,
+            -0.02884953407490196,
+            0.0029097409399144306,
+            0.025277274938032477,
+            0.009263530315933738,
+            -0.028037993274175277,
+            -0.02532252134134532,
+            -0.0018596440554143638,
+            -0.013847737486669237,
+            0.009910107581119272,
+            0.013952962280235523,
+            -0.011807495289410774,
+            0.003896325037406752,
+            -0.02144921729633576,
+            -0.01628545945817568,
+            -0.0040148671532777005,
+            -0.016032493124255323,
+            0.004580041957001551,
+            -0.011375180364988195,
+            -0.007789996450583554,
+            -0.022041665680945066,
+            -0.005890065942573431,
+            -0.050455058269193866,
+            0.014401045842891128,
+            0.030219548461217365,
+            -0.004580747668649223,
+            -0.028684680969792875,
+            -0.015525870112797619,
+            0.011121401958547706,
+            -0.008879749268609245,
+            -0.019454243014778435,
+            -0.007839434838464604,
+            0.018908534797107714,
+            -0.0019283067432864216,
+            0.0024769961730655835,
+            -0.006732653593696035,
+            0.027604805021030646,
+            0.01366593691107021,
+            0.03740746059041736,
+            -0.014662268336336578,
+            0.02539666271785517,
+            0.02371809783638648,
+            -0.036126848124425154,
+            -0.007627445211958312,
+            0.03657481892016626,
+            0.01092515982671624,
+            0.0011046573579274704,
+            0.012350047525582187,
+            -0.006340802546498992,
+            -0.031330519599469066,
+            -0.0013691360532301728,
+            -0.0014676630169621005,
+            -0.023111265345539388,
+            0.027359543575867225,
+            0.018091458372134273,
+            0.014237156751820934,
+            0.0030993618578507172,
+            -0.0014037082734049385,
+            0.013257269860435359,
+            0.01021315143964527,
+            0.030885417052806473,
+            -0.003225065956381245,
+            -0.012825341172770057,
+            0.013476039084668826,
+            0.01852164843904835,
+            -0.009965495459579544,
+            0.012541079912270015,
+            -0.00824122613985912,
+            -0.018124134280975132,
+            0.03192263766222747,
+            0.0001484308749572035,
+            0.010888733498667343,
+            -0.010649515858317311,
+            -0.013082335214246402,
+            0.032614733822945205,
+            -0.00999075761269065,
+            0.004257643407198955,
+            -0.030723533031261387,
+            -0.003820776505926347,
+            -0.010471792928758869,
+            0.01737608180148319,
+            0.004597677757231315,
+            0.02095148515295301,
+            0.029253627700524396,
+            -0.011604450820008121,
+            -0.01996325509217509,
+            -0.001158626158150122,
+            -0.020583793738536985,
+            0.004168757459099133,
+            -0.03632338333261702,
+            -0.019373660230711246,
+            0.022259216603413172,
+            -0.010364620831688015,
+            -0.004751193403783363,
+            0.00444187509644351,
+            0.0008572202661850934,
+            0.014943779567410408,
+            -0.025408263453313398,
+            0.017755758370543148,
+            0.004627392860417132,
+            0.007402073071525438,
+            -0.024913329140362466,
+            0.013760771254234878,
+            -0.006087346663657887,
+            -0.03780469033982951,
+            0.005509388120163784,
+            0.03756141956381493,
+            0.02381019168406692,
+            -0.024580249929652097,
+            0.01807584361080568,
+            -0.03111039245576191,
+            0.023245542501125073,
+            -0.004868090951368273,
+            5.1544460403943004e-05,
+            0.00640235757127173,
+            -0.004698373089469779,
+            -0.00621383035264185,
+            -0.0025132415458431587,
+            -0.008746634331769936,
+            -0.01611801559493664,
+            -0.01193535623883546,
+            -0.03381789115349148,
+            -0.024119754887378317,
+            0.013881102852385332,
+            0.0019232668966222687,
+            -0.0095263729740736,
+            0.018652966857927805,
+            -0.03775912497511837,
+            -0.012022352025516115,
+            -0.023289377270709736,
+            0.023579672990954228,
+            -0.028605627815260902,
+            0.019235397553197357,
+            -0.025838655372560505,
+            -0.021458526449310157,
+            0.01508157885177254,
+            -0.010064396230734452,
+            -0.0016708624560165368,
+            -0.015042071642751917,
+            -0.03670785416972153,
+            -0.006267614265977843,
+            0.02298763967146357,
+            -0.012506444728541897,
+            0.00029871598584066644,
+            0.02671390618094565,
+            -0.003059865833779805,
+            -0.011103452976820622,
+            -0.0017382684168257506,
+            0.010923055147701782,
+            -0.02938686260440486,
+            0.014300097108092281,
+            0.0038571851420598074,
+            -0.0037614499297117365,
+            0.0010991440540809893,
+            -0.010108413486243776,
+            0.02127952161226619,
+            0.009142545227414532,
+            -0.0030260119510809205,
+            0.040589877142650374,
+            0.008677453889680373,
+            0.0017339508605300814,
+            0.008732950895530873,
+            0.01063352447960556,
+            0.006618635029823979,
+            -0.014986307792626314,
+            0.016905861015681957,
+            0.013101237900847744,
+            0.0002162791516784468,
+            0.019309343995079022,
+            0.0021263376627204313,
+            -0.024516267132902416,
+            0.013339486637573757,
+            -0.02311659751478777,
+            0.0020575982953961246,
+            0.02391695586495566,
+            -0.020440172706421535,
+            -0.001656132876701879,
+            -0.006304133667169861,
+            0.000543457661904495,
+            0.022135707414160034,
+            -0.0230903871986617,
+            -0.023444763940153112,
+            0.009604600033024832,
+            0.01721387169484476,
+            0.01827905276638783,
+            0.02824773106648473,
+            -0.007100625424382998,
+            0.01348510178676837,
+            0.0002604386361122316,
+            -0.011802178189607182,
+            0.007664157334475143,
+            0.0074175377458313594,
+            -0.0044008460682833875,
+            0.025491035291032547,
+            -0.0017015687016376205,
+            -0.006134010425966364,
+            -0.005379657499404915,
+            -0.01584573519366957,
+            0.0034386456145185213,
+            -0.013860742817569676,
+            -0.013355238174583658,
+            0.0040342831827141845,
+            -0.0208160950322342,
+            0.10904592104153767,
+            -0.0037038089210379766,
+            -0.019472012964045247,
+            0.0063554319446707044,
+            0.003716719884684827,
+            -0.003692786470980806,
+            -0.003353843210051379,
+            0.007788605566526459,
+            -0.011254480693190834,
+            -0.0045751956417398095,
+            -0.013976039724381875,
+            -0.001989262807706565,
+            0.010067374586433014,
+            0.009201983771978006,
+            -0.005902980676079493,
+            -0.02029714318363332,
+            0.013031676901335606,
+            -0.016970144750162643,
+            0.008874670945038486,
+            0.0008311255074953139,
+            0.00884167159995975,
+            0.001716603195908536,
+            0.01854951250316076,
+            -0.009310881405486074,
+            -0.01325242738760595,
+            0.029667589863813217,
+            -0.005616511284128333,
+            -0.0038179509972925954,
+            -0.0004690716809300012,
+            -0.005068974756744239,
+            0.0021204231125051274,
+            -0.027958274497127258,
+            0.022127088236172387,
+            -0.017271838492631593,
+            0.010267709087809265,
+            0.01316800560257366,
+            0.005599367128954229,
+            0.01817806970002991,
+            -0.007115594036834107,
+            -0.02627643123342057,
+            0.006542742528003751,
+            -0.003433874053082844,
+            0.006631427284537971,
+            0.02316295746074422,
+            -0.014573351469966297,
+            -0.033520864104177005,
+            -0.013707022143307006,
+            -0.01877574599842729,
+            -0.005992004412448576,
+            -0.0043698881826095435,
+            -0.01481352026460129,
+            0.006424121402963946,
+            0.01382938882387991,
+            0.018136021761515388,
+            0.0012697483790911433,
+            -0.011531136974861559,
+            0.00609187274277241,
+            0.01076133059240578,
+            0.008400887242930113,
+            -0.01533504932731406,
+            0.011439227832642668,
+            -0.028718487721012004,
+            0.01522697877548034,
+            -0.012452665599305595,
+            -0.024938748080573947,
+            -0.0022082450217489366,
+            0.017589399637694264,
+            0.0037824832255756763,
+            -0.004355445812154662,
+            -0.018778668474337727,
+            0.019825087149714933,
+            0.024692878851503508,
+            0.003883497401229232,
+            0.019470881143244197,
+            0.041834635328088364,
+            -0.003544509491805351,
+            0.027139164223766474,
+            -0.02129723143323512,
+            0.01461354860170749,
+            -0.03441729954693155,
+            0.001893713808284104,
+            -0.004222657393633789,
+            0.0013677766871517654,
+            -0.003811928806805752,
+            0.020142937759360904,
+            -0.009158202577174012,
+            0.0030638058587375543,
+            0.024854053046139163,
+            -0.018665378030958756,
+            0.00805903734779081,
+            0.010625429315145887,
+            -0.012856237251902233,
+            0.028535299711292306,
+            -0.006335094045840286,
+            0.00305422126851784,
+            -0.0297883488914759,
+            -0.0029192872993071216,
+            0.0026067044379754047,
+            -0.006097749614529602,
+            -0.00968023644800927,
+            -0.007433916539502931,
+            -0.006202274026852405,
+            0.0038354607448269555,
+            -0.021471361566683354,
+            0.003195646827154078,
+            -0.004390751282171024,
+            -0.003323447423321121,
+            0.02518759723554948,
+            0.025857746688368054,
+            0.016893707217894103,
+            0.012739170140289531,
+            0.008553625305650129,
+            0.019805351588748434,
+            0.01654964927067909,
+            0.011549329129728739,
+            -0.020026013396046632,
+            0.01150644479272541,
+            -0.004159611921476011,
+            -0.0022937297602724912,
+            0.014815466072709131,
+            -0.01572171840394285,
+            -0.02276187015864222,
+            0.024693009795978448,
+            -0.019725315346836404,
+            0.00998768979020729,
+            -1.1702210281215188e-06,
+            -0.011129745730507545,
+            -0.002591293237053776,
+            -0.018034357254990988,
+            0.00012070821244376275,
+            -0.023051045150566527,
+            -0.028465270110388716,
+            -0.00048603008831669926,
+            -0.000574546300981984,
+            -0.012992623897880395,
+            0.02064747165046032,
+            0.014820638761258362,
+            -0.024820174379739057,
+            0.02666518374464728,
+            -0.017430076811109155,
+            -0.01495051200726725,
+            -0.005861181182847845,
+            0.00100881727574559,
+            -0.0055223282819321295,
+            0.0034719859452138017,
+            -0.010180752654175722,
+            -0.009961678082468036,
+            -0.012134423047419183,
+            -0.028073672819317354,
+            0.011574683159318163,
+            0.016120674453124317,
+            -0.015043147648505721,
+            0.031834693135959095,
+            -0.008802805725050891,
+            -0.0018698230152007877,
+            0.016179742266521146,
+            0.005096232332051617,
+            -0.017187077376166815,
+            -0.018999569809271876,
+            0.012251308792871571,
+            0.013279090498325945,
+            -0.009588297039279112,
+            0.006630431200804809,
+            -0.014130176783207386,
+            -0.005092662672039885,
+            0.007020236623084732,
+            -0.005049363863784624,
+            0.0055051712211635915,
+            -0.00995314895304919,
+            0.0052743456988306485,
+            0.005841370337169303,
+            -0.0029434616355351975,
+            0.004677049987338193,
+            -0.018568941264514256,
+            0.020031253265629777,
+            0.0018725390936655962,
+            0.01332532735283085,
+            -0.03518855189467458,
+            -0.0056298723325832025,
+            -0.005369475318146738,
+            -0.004650364775945294,
+            -0.020833510091412353,
+            0.020445280915606132,
+            0.0159991245532085,
+            0.013202409908533162,
+            0.023094507005426774,
+            -0.014905666713082246,
+            0.01572052456975675,
+            -0.020945766915121455,
+            0.013114000985348841,
+            0.003349101658930482,
+            -0.019829605346106578,
+            -0.001423750749031516,
+            0.04557459007627153,
+            -0.005847231973329081,
+            0.011986110554869603,
+            0.0012709842140249691,
+            0.026758922819943853,
+            -0.0001635536762342011,
+            0.0065419519929442995,
+            0.005997692702992206,
+            -0.009935128108338126,
+            -0.0008057798606777003,
+            0.00865712443881613,
+            0.046254320605750854,
+            -0.01181132866321511,
+            0.008170688314239364,
+            0.013099051984057827,
+            0.0034971494510865624,
+            -0.0008386400616584293,
+            -0.015346506644390764,
+            -0.03067678541405639,
+            0.019305962963428017,
+            0.007736916644077731,
+            -0.012756525606221703,
+            0.008734673666457216,
+            0.015137771675476126,
+            0.012891824153867332,
+            0.003895886821685762,
+            0.02155154331885366,
+            -0.012761690789748027,
+            0.018528313792219413,
+            0.024663453412919027,
+            0.009077230086690327,
+            9.184526542175535e-05,
+            -0.000751529849753638,
+            0.006791927267880182,
+            -0.0004433479107716552,
+            -0.009832230053513483,
+            0.015858232096630655,
+            -0.014861341148074698,
+            0.017790924034108806,
+            2.7216806238946785e-05,
+            0.03744841066552559,
+            -0.015216207585985075,
+            -0.010138319613387346,
+            -0.007774325714088762,
+            -0.0016997034352120362,
+            0.003971564314593291,
+            0.0018686537276849653,
+            0.006051669054361943,
+            -0.0008319624938748146,
+            -0.017326561560776794,
+            0.01667737942301944,
+            0.005719808546053257,
+            -0.0007405637872110778,
+            0.013750752065593804,
+            0.0038732323799683453,
+            -0.00367645612814479,
+            0.023580906347714314,
+            0.006238541579764762,
+            -0.005235795826199542,
+            0.0028182323816636243,
+            -0.006443210286549713,
+            -0.016255209299456314,
+            -0.003061441395209219,
+            0.004597970789372988,
+            -0.0016590920570710628,
+            -0.011138261849069419,
+            -0.0019423173609729848,
+            0.020355534503272887,
+            -0.004612410336124759,
+            0.013346750536413418,
+            -0.005400060374036387,
+            0.006252154576727367,
+            0.010601459925435238,
+            -0.012346569079682625,
+            0.0014154620309235822,
+            0.010406076751218612,
+            0.0013983417564564663,
+            -0.007140180075225315,
+            0.0010544410919327675,
+            0.007757599672903709,
+            0.006505350216372801,
+            0.00483063937962671,
+            0.00836630436955631,
+            0.013284451514874498,
+            0.0033749897293287438,
+            -0.006243293993633622,
+            0.0013949831763016042,
+            -0.013269921385800992,
+            -0.006308639321247727,
+            0.0010906169976769569,
+            -0.0010152835709869353,
+            -0.02017142197515583,
+            0.010787630901414118,
+            0.005355784839945777,
+            0.007838485470991034,
+            -0.01706932891696025,
+            0.009576030158827565,
+            -0.0008144311254998525,
+            0.02829832828245172,
+            0.007407953684956539,
+            -0.02769403281764041,
+            0.014266001222243134,
+            -0.010492199740754951,
+            0.022575782293320284,
+            -0.0016349316593207844,
+            -0.005656505606143177,
+            0.008025514398322551,
+            -0.009634281579228592,
+            -0.005991026158213869,
+            -0.022366083313133788,
+            -0.025825209755160946,
+            0.016871087849192924,
+            0.013583173973358155,
+            0.021125252303971605,
+            0.00012555627355712715,
+            0.015347535950809997,
+            0.0020128011217859444,
+            0.015164099887550086,
+            0.0024544903332736167,
+            0.012507136277293888,
+            -0.011798050869794805,
+            0.0104828761281283,
+            -0.0137805761114781,
+            0.014620357150314097,
+            -0.005246639228547017,
+            -0.006986408834884282,
+            -0.008849580196616098,
+            -0.005442165099441538,
+            -0.020891551850452786,
+            -0.009155930286949747,
+            -0.0013590151948670868,
+            0.010665872513236615,
+            -0.0005386423407222223,
+            -0.0027671778465775897,
+            -0.03237311596935737,
+            -0.004875062533405715,
+            -0.006670982068843501,
+            -0.01942244110096111,
+            -0.0045975373348006885,
+            -0.026137783023267008,
+            -0.01657173843643047,
+            -0.0007784046498232684,
+            -0.015037306676062236,
+            -0.006889680260149007,
+            -0.02009705080543122,
+            -0.027192867101205106,
+            3.0161964469866572e-05,
+            -0.009647644858034833,
+            -0.005380165957043753,
+            0.013815092649296373,
+            0.011902366689169847,
+            -0.01572592170758115,
+            -0.022321702398924333,
+            -0.01088937403483412,
+            -0.0165754067023749,
+            0.01646435743868707,
+            0.009914938007210463,
+            0.018834863207293908,
+            -0.003724245912768107,
+            0.005643505694722323,
+            0.0009581443359000331,
+            0.02327704258596552,
+            -0.02532422120223976,
+            -0.028694220894404807,
+            0.0019298778426469338,
+            0.0025970175688368323,
+            0.006086164866310174,
+            -0.009414665394188936,
+            0.001200390869257535,
+            -0.007782899559317819,
+            -0.0228264209954853,
+            -0.00898519580978142,
+            0.008156835270668449,
+            -0.010036630164522355,
+            0.02230664068547531,
+            0.012332230358869292,
+            0.013699197379189934,
+            0.010593587938184292,
+            0.01576174456497367,
+            -0.0001654241831584687,
+            0.014471905257729428,
+            -0.008675915624727887,
+            -0.0031070346099204425,
+            0.0038358404385932777,
+            0.005878417037187274,
+            -0.00625022998235275,
+            0.028077114529659795,
+            -0.0011836453716817525,
+            0.0002962695480296182,
+            0.014405782572641941,
+            0.02697667559474498,
+            -0.004710313427099278,
+            0.0210512043261309,
+            -0.002393165025068831,
+            -0.006135816286212752,
+            -0.008429776018235104,
+            0.008607617299702318,
+            -0.009529291136733401,
+            0.011133497451942684,
+            0.019864404836361926,
+            -0.014041315457579166,
+            -0.007351413712676263,
+            0.02950254974910368,
+            -0.014564388091367897,
+            -0.01825317106331622,
+            0.02325964952838705,
+            -0.007414295676409368,
+            -0.0039042587057605484,
+            0.00377144546378926,
+            0.007169856503233239,
+            -0.004036744966949995,
+            -0.012928341573802416,
+            0.016090527447479415,
+            -0.00011159893712484579,
+            0.014146137810890391,
+            -0.0004655839804281017,
+            -0.006372231942983921,
+            -0.010047382825720416,
+            0.002373224247019062,
+            0.026818813126567614,
+            0.013483627627077692,
+            -0.017895710143841333,
+            -0.013785856764479817,
+            0.010826610341182479,
+            -0.01538893818197323,
+            -0.004425206001945227,
+            0.02155595194812746,
+            0.0028768153668128553,
+            -0.020543027467698483,
+            0.008028975884363618,
+            0.0003835165718138906,
+            0.00037046947047892953,
+            -0.00029363296913641716,
+            0.030135739208689217,
+            0.001343516577762509,
+            0.00018948658255495327,
+            -0.00875053830853609,
+            0.008606573235959595,
+            -0.01280305089741873,
+            0.012786478836532904,
+            -0.02437993139337839,
+            0.007648181775166282,
+            -0.00629541989541625,
+            0.004579605315599212,
+            0.016658386927051592,
+            0.01218098627022323,
+            0.015031618813778413,
+            0.008237200519537417,
+            -0.008942965899756683,
+            0.0009093265012901892,
+            -0.025302175262520477,
+            -0.001333639155622843,
+            -0.017971876771137367,
+            -0.02036433247634014,
+            0.0002463842837918767,
+            0.022144712689630985,
+            0.018559482073124734,
+            0.011579840028464503,
+            -0.008657014440886656,
+            0.0166696911919226,
+            -2.9834620942882283e-05,
+            0.011845827764518911,
+            -0.00037901176785495324,
+            -0.006138374154553732,
+            0.004711058953619512,
+            -0.003622694588720417,
+            0.013542405259906128,
+            -0.014730567462694827,
+            0.012076044672004906,
+            0.01690223095263866,
+            -0.009108632851807137,
+            -0.008946251274440141,
+            -0.013992423874475428,
+            0.00048381889145447367,
+            -0.015909599902931202,
+            0.035489252674454214,
+            0.02252772349379229,
+            -0.01210464970923921,
+            -0.010469120693943236,
+            -0.014164999313775756,
+            0.019496048806752753,
+            0.010427539351244087,
+            0.0028018908451425276,
+            0.017091768406027965,
+            0.021395638416767112,
+            0.008817247593404623,
+            -0.006081063711583319,
+            -0.0021216503501144983,
+            -0.003781810594530787,
+            0.020143098218482827,
+            -0.003078172339607037,
+            -0.011350171884507555,
+            -0.012516379645909518,
+            0.005766102930704529,
+            -0.01626433668691071,
+            0.0019294643928566657,
+            0.008641558521764407,
+            -0.015057688533773374,
+            0.019407986696657772,
+            -0.028126197703647868,
+            0.012364374199657305,
+            -0.0044278434183731225,
+            -0.007251147925369614,
+            -0.03496220945653279,
+            0.006389489026974697,
+            0.008941259259632024,
+            0.003941752277454314,
+            0.023003838655784314,
+            -0.024528514847470928,
+            -0.018361931455105486,
+            0.0037791061321962543,
+            0.00047162060848090104,
+            -0.00711894577052754,
+            0.0010138871688747137,
+            0.004128392769213672,
+            -0.00674546985015164,
+            0.0017307097443508704,
+            0.0033354800298095546,
+            -0.002829898077850263,
+            0.013252947102333348,
+            0.0081724269619999,
+            -0.021449979263734438,
+            0.001368791856860285,
+            0.01590656734732547,
+            0.01452008402429858,
+            -0.02391847163018108,
+            0.006331397860689212,
+            -0.02064812934794864,
+            -0.000631340242718475,
+            -0.031272866000888556,
+            0.016859793097564958,
+            -0.0069008624795683276,
+            0.01899718801899956,
+            -0.0015377436942771224,
+            -0.0065810685260109004,
+            -0.012822620655323715,
+            0.02137604937338179,
+            -0.0019121137078344135,
+            -0.0025598772127157477,
+            0.015678293057066646,
+            -0.025700865215693987,
+            0.017825186616033165,
+            -0.0026230848396158204,
+            -0.017342967698019966,
+            0.017199139867162275,
+            0.004536019178633225,
+            0.007224666811309087,
+            -0.023505589665315567,
+            0.0007208041198255396,
+            0.003900164485347562,
+            -0.017250324025622685,
+            0.004470862192961212,
+            -0.030807400122398566,
+            0.0006579147990327597,
+            -0.013222907940559794,
+            -0.0020514994691204847,
+            -0.0046763235724575445,
+            -0.014050035576183385,
+            -0.039181147615786484,
+            -0.005429639279537522,
+            0.016065742005148733,
+            -0.0009029636981539927,
+            0.01800715198737081,
+            -0.002730931639143803,
+            0.002947680833843522,
+            0.004912692811020822,
+            0.00818378920043427,
+            0.006915895602313347,
+            0.02274116906914758,
+            -0.0057854922886970365,
+            0.00471999900678522,
+            0.007058014277137233,
+            0.014638081356210031,
+            -0.01719565951201012,
+            -0.015170377598372723,
+            -0.016015921063369496,
+            -0.006351213642269936,
+            0.01739665344303718,
+            -0.017488885259028475,
+            0.013303332445465686,
+            -0.00429428898520701,
+            0.014825338166443021,
+            -0.008686427208690176,
+            -0.01575945086724703,
+            0.0028564767084934386,
+            -0.007570417028925067,
+            0.001007670647768344,
+            -0.02323908008484089,
+            0.02019729564706863,
+            -0.00847581038203716,
+            0.010508144473197669,
+            0.01729824052022313,
+            -0.021364529179605813,
+            -0.009742851959432839,
+            -0.014342281441152413,
+            0.029896210049029176,
+            0.013925163184699139,
+            -0.009392252140772913,
+            0.003927230832659267,
+            0.008873561269230905,
+            0.011529536139797032,
+            -0.012209070214071445,
+            -0.0027679974125029362,
+            0.002411914349172495,
+            -0.0062056207925691356,
+            -0.002025687569076897,
+            0.03090134105002816,
+            0.0038285457265343245,
+            0.024877474647546086,
+            -0.013214997316123852,
+            -0.0033763842539861907,
+            0.00023379106197363714,
+            0.005134764372494098,
+            0.01183867975321486,
+            -0.0017966997954747142,
+            -0.008029791503028367,
+            -0.01179500476055557,
+            0.02599911330072515,
+            -0.021852083993691557,
+            0.007024505421035273,
+            0.026784024996764876,
+            0.001390202993990949,
+            0.0029956956045771046,
+            -0.013131691460546865,
+            -0.006298736864509989,
+            0.02523640664202355,
+            0.010030824059858138,
+            0.0041523782603032225,
+            -0.026465819482580182,
+            0.0032128899272377593,
+            -0.0061901409332948115,
+            -0.008681758126710213,
+            0.009866461769956222,
+            -0.0028095454857224164,
+            0.01188012041754054,
+            -0.012191996383095982,
+            0.028459939543805705,
+            0.010711502132045176,
+            0.01047154118931583,
+            0.025888652452698974,
+            -0.02079614305315134,
+            0.0042203106489994615,
+            -0.010829796140534066,
+            0.0006773493086316061,
+            0.005596640511307504,
+            -0.027510562413905437,
+            0.001969898603871003,
+            0.010390866563404346,
+            -0.005642706366801436,
+            -0.01882994179544761,
+            0.01187970346127799,
+            -0.019814385265412387,
+            0.024479579275617887,
+            0.003231341626501863,
+            0.020056914356101588,
+            -0.0053276448188067154,
+            0.009834685132275868,
+            0.0002742942778577165,
+            0.0031361331890882127,
+            -0.0008074598092128465,
+            0.009009012303347016,
+            0.005786777410144894,
+            0.013719989127068968,
+            -0.025528370602188686,
+            0.006821572597883499,
+            -0.0029197127345352626,
+            -0.030539872624857384,
+            -0.013383097680839732,
+            0.010291485622655026,
+            0.012218533188117704,
+            -0.009144425807890594,
+            -0.010378679518611277,
+            0.0030042579827305603,
+            0.005748243116238633,
+            -0.006692403885450407,
+            0.006913672151000981,
+            0.01629516912493545,
+            -0.002533517074771728,
+            -0.0019083178803585312,
+            -0.0035007340321378893,
+            -0.01154609935927537,
+            -0.010016998439072824,
+            0.023193275321990937,
+            -0.00482990342915347,
+            0.017943025772239603,
+            0.025873224580389702,
+            0.0039876567755322,
+            -0.0057577932736143506,
+            -0.005460363755206188,
+            0.01478435999841998,
+            0.014195457012279476,
+            0.004669295894868759,
+            -0.009702220679239768,
+            0.007974305230005956,
+            -0.007760086293456636,
+            0.008375077126777163,
+            -0.020260917110786147,
+            0.002199401871514877,
+            0.0026191373298767766,
+            -0.0022089262577237776,
+            -0.015213807046175918,
+            -0.01956788452420682,
+            -0.021898368819136302,
+            0.012634659463647929,
+            0.00034643102239488474,
+            -0.009241772775582378,
+            -0.013645250997151429,
+            0.0016665106990117016,
+            0.0025630500346914592,
+            0.0002095521560180936,
+            -0.007783022628694597,
+            0.009546147813008391,
+            -0.01265287317178631,
+            0.007445223311736028,
+            -0.004007029181436962,
+            0.0051846833906574515,
+            -0.0016916014415925766,
+            -0.016413938859402438,
+            -0.0210868129634149,
+            0.01920794362224444,
+            -0.012897042227841011,
+            0.02269303737601583,
+            -0.0014935878147854765,
+            3.062680855559304e-05,
+            0.0033132626755401546,
+            0.007274487651476732,
+            0.012110042834251237,
+            0.011426424896797977,
+            0.004307086698938411,
+            0.008206932254416481,
+            0.004673427176334148,
+            -0.013124571552764586,
+            0.004324963759928715,
+            0.0039216033705804084,
+            -0.010449887922072297,
+            0.005298323150690876,
+            -0.0010196552573491,
+            0.006128327172621373,
+            0.012167492341907742,
+            0.01923525204293908,
+            0.01908107550907106,
+            -0.0012031208519043073,
+            -0.0010601912677170903,
+            0.014474374858125607,
+            0.004540954805526089,
+            -0.004883537463424679,
+            -0.002772988629590886,
+            -0.013363713078069854,
+            -0.002572403646338979,
+            0.011541515690646252,
+            -0.02376161747414617,
+            -0.018215231053002656,
+            0.021831219299143226,
+            -0.004991882316041097,
+            0.033796174834149546,
+            -0.013140970283091536,
+            -0.016197177724633614,
+            -0.00464190076219332,
+            0.0026389653040644607,
+            -0.013788049032378498,
+            -0.010300859744205445,
+            0.010545541278382187,
+            0.013848738866656793,
+            0.009903590151120531,
+            -0.0005984085581408575,
+            0.02922816576443645,
+            0.0019900932217491907,
+            0.02245395592422119,
+            0.0018978407151245295,
+            -0.007029294098689854,
+            -0.01440575120349887,
+            0.010840472832356266,
+            0.005529335652955532,
+            0.009050748012161991,
+            0.017791269808484802,
+            0.014502057454389247,
+            0.006265567405788131,
+            0.013928705812292101,
+            0.0012903002372833693,
+            0.0010858377120770652,
+            -0.002355229463060164,
+            -0.01692711767136341,
+            -0.01456786846097011,
+            -0.01627640642218236,
+            0.003873864312554098,
+            0.012620350492241247,
+            0.00525890570023481,
+            0.0007393789113866752,
+            -0.007153210751899716,
+            0.009254566395058412,
+            0.0065087881512835375,
+            -0.012279152412209152,
+            -0.016530068283509593,
+            0.004060671955726967,
+            -0.02210570272789274,
+            0.007849467899009437,
+            0.01554233481902349,
+            0.00219885032769439,
+            -0.002655449524219968,
+            0.0006843524304842362,
+            0.006272944169764281,
+            0.005531927368102191,
+            -0.0036710328198476905,
+            -0.00544113514775027,
+            0.011264661763762068,
+            -0.006379654234440906,
+            0.009537150372938296,
+            -0.00290031548037575,
+            0.03718190949929003,
+            -0.011976080887998194,
+            0.012120408392610773,
+            -0.001751472085435711,
+            0.008481703962951994,
+            0.008067669857949423,
+            0.010772687289903202,
+            -0.003805089040741351,
+            -0.0022164543131735167,
+            -0.008753625032781523,
+            0.006751953442322185,
+            -0.01582177621255226,
+            0.013631301238130705,
+            -0.00030685726238985963,
+            -0.010728958775974582,
+            0.01055472059121978,
+            0.006788316621149835,
+            0.004413382316882087,
+            -0.003262024961860459,
+            -0.01972283203505511,
+            0.014937729522976615,
+            0.0018799533869990667,
+            0.019247008681353865,
+            -0.023276106189004046,
+            0.029043727846427652,
+            -0.02019024213343764,
+            0.023083643395685142,
+            0.01865875504942157,
+            0.001179436167200093,
+            -0.02070329911663634,
+            -0.007618131527214554,
+            0.014871709122115308,
+            -0.02659818889735726,
+            -0.006234719259825804,
+            0.005986829551040766,
+            -0.014639912579831785,
+            0.00037813423546252645,
+            -0.01924149909676064,
+            0.00010832498610532006,
+            0.011013130149723431,
+            -0.011782407678373089,
+            0.00438725832178584,
+            0.01608349750921335,
+            0.019347883652009713,
+            -0.009760017612545174,
+            0.003869246694835168,
+            -0.024240324890794748,
+            0.010958789926998691,
+            0.020511684636649225,
+            0.008883830392748497,
+            -0.01761944522796127,
+            0.021817973960311446,
+            0.010652131228027521,
+            -0.012575679152857703,
+            -0.0012753095453828826,
+            -0.028191274775392804,
+            -0.012421170815054556,
+            -0.02188248722244861,
+            -0.0011670175621882146,
+            -0.014436683528976771,
+            -0.0029783880907228383,
+            0.0009506986717516226,
+            -0.004719165363484719,
+            -0.01179163390107033,
+            -0.019543732196676634,
+            0.016606805875498898,
+            0.0227313413004497,
+            -0.006738735201626204,
+            -0.014547240981002825,
+            0.013866974327185123,
+            0.031694077725890536,
+            -0.016189209297529324,
+            -0.00542110554672736,
+            0.0028061798782555557,
+            -0.012964876141500577,
+            0.018946046750284537,
+            0.0039419760434645495,
+            0.000322115192590907,
+            -0.01991851369210666,
+            0.002489725351392778,
+            0.010835458511052207,
+            0.01684964045708685,
+            0.0005541314524621196,
+            0.010855852688653808,
+            -0.007549882748861744,
+            0.010096800951420057,
+            -0.011832260836468588,
+            -3.6179995603517093e-06,
+            -0.007800366796516945,
+            0.007001049318404058,
+            -0.004424776169203575,
+            0.034356203288485945,
+            -0.02817071276766299,
+            0.004989631403681216,
+            -0.014539644354369607,
+            -0.010888763939716889,
+            -0.0027322086509797843,
+            0.0004865534340981866,
+            -0.010775238981581461,
+            -0.005162220412866082,
+            -0.0011263271110752205,
+            0.00913081242511139,
+            0.005844327320514489,
+            -0.013886529519306775,
+            0.019171827605726503,
+            -0.030984919318656618,
+            0.004469458202753611,
+            -0.008553175891755315,
+            0.010511707988936336,
+            -0.0151185694875566,
+            0.013830711686954931,
+            0.01301959345025159,
+            0.008205984085944315,
+            0.007740367849216328,
+            -0.01532123052721834,
+            -0.01678314222392063,
+            0.001376468747213867,
+            0.0036953018347776244,
+            -0.00931855481896886,
+            -0.033391028922085425,
+            -0.007072913446166969,
+            0.005331350534425606,
+            0.011614453225043,
+            0.008156128603003278,
+            -0.02345206873048703,
+            0.005003870024809975,
+            -0.024178720646704597,
+            -0.012115112684614331,
+            -0.007750557665424973,
+            -0.011580152272614108,
+            -0.006117298954192534,
+            0.026449765548206453,
+            0.007989237291215073,
+            0.016691517078382376,
+            0.004954133565782356,
+            0.02742809127725913,
+            0.011670893621991404,
+            -0.011736840457474895,
+            -0.003349151747880166,
+            0.00035419034843196817,
+            0.005507651806218584,
+            -0.008706590052024251,
+            0.011793241980257321,
+            -0.0013295035978611142,
+            -0.0065061682758755625,
+            -0.004398354347597199,
+            0.009372816666628725,
+            0.014188676762501037,
+            -0.003007980559117429,
+            -0.008724255229333351,
+            0.015749776067989826,
+            -0.030899793617096834,
+            -0.015215033379100091,
+            0.0016152308157299517,
+            -0.007813175908301852,
+            0.017806770680528907,
+            0.01169056345888971,
+            -0.026835412610223673,
+            -0.004538235170332766,
+            0.010532316940606419,
+            0.004755930024691789,
+            0.0095408889025506,
+            0.00954782703036282,
+            0.012079064562163901,
+            -0.007829549014214643,
+            0.01046722658021748,
+            -0.011519369840999348,
+            -0.007092961453184232,
+            -0.007241799314569802,
+            -0.008077934089061052,
+            0.021467547268987214,
+            -0.025816030568238717,
+            -0.006558853543772633,
+            0.01134670818701549,
+            0.013027491377727365,
+            -0.002301991242932548,
+            -0.006299073927206214,
+            -0.017357989759336386,
+            0.008311464809363397,
+            -0.013922568511594478,
+            0.025633907424998367,
+            -0.004854655498817219,
+            -0.005336807899100369,
+            -0.005228056880188717,
+            0.01958682721272445,
+            -0.016774409737540904,
+            0.02072573220949266,
+            -0.014449250679704904,
+            -0.007857351393427215,
+            0.001673929749689194,
+            0.006890008607998747,
+            -0.002948835017234416,
+            0.03734128725793149,
+            -0.00556474133167756,
+            -0.0014544201137417158,
+            0.02209084868209573,
+            -0.01883610080898967,
+            0.017901421953179442,
+            0.01563604483998435,
+            0.012327159100148019,
+            0.0018161034827768648,
+            -0.010253034391420751,
+            0.019979728662282904,
+            0.020823925491027635,
+            0.0139490143219451,
+            0.009577521724445478,
+            -0.013478761809038993,
+            0.014934933784421556,
+            -0.0003725422088504494,
+            -0.0024362115214204105,
+            0.012785534912074068,
+            0.016441944143888537,
+            -0.02164522371903646,
+            -0.013152102018736346,
+            0.0063548333449661805,
+            -0.016547629104776748,
+            -0.024875443975589356,
+            -0.008991095813132973,
+            -0.021885127280977718,
+            -0.023336714382596175,
+            0.00688008599534903,
+            0.016482610587564443,
+            0.0035797836729533367,
+            0.013111012464332918,
+            0.006820471349350837,
+            0.018274678506489214,
+            -0.00980248993077149,
+            -0.01582838461385605,
+            -0.0009454586621769173,
+            -0.0004906493597104733,
+            0.009787688665732045,
+            -0.01835583040075359,
+            -0.013977939077760832,
+            0.004340902272034388,
+            -0.020118484730757877,
+            0.01421736173162507,
+            -0.006428386730058143,
+            0.016756376129345597,
+            -0.0004630315831872392,
+            0.011701948921076007,
+            0.02621773278421472,
+            -0.001391567406453007,
+            0.01262026995055899,
+            0.004696906079976175,
+            -0.0030577364576645635,
+            -0.010624066224426196,
+            0.0009420498021845824,
+            -0.0044079288776429516,
+            -0.0004845606666642234,
+            0.006481113327122954,
+            0.020269778435314,
+            -0.03321232649491074,
+            -0.009425498978427855,
+            0.01246754422779043,
+            -0.001658574527323383,
+            -0.005816985985181387,
+            0.013050062223188965,
+            0.00011195028631856881,
+            0.016573827662919836,
+            -0.0017453203044868104,
+            0.01417371165305189,
+            -0.008165910519539772,
+            -0.0007461950113024975,
+            0.0016492309267666983,
+            0.005076015595439873,
+            0.005829299318055953,
+            0.012146800191035666,
+            0.01148884639506138,
+            -0.021051327863689618,
+            -0.0026357096198729156,
+            0.0036146148489310443,
+            -0.010673282438301611,
+            0.014967444393862347,
+            0.005737933867764028,
+            0.01391411712868596,
+            0.0029556189893790237,
+            0.01167806258173173,
+            -0.024440974650111174,
+            -0.01029944007931845,
+            0.01385509328958837,
+            0.00869671728965227,
+            -0.00998315356917989,
+            -0.016580053751364735,
+            -0.007061005072623325,
+            -0.00477175200214453,
+            -0.0021182980051220428,
+            -0.009871524928312118,
+            -0.0005933941177237234,
+            0.006284920058476413,
+            5.466820734762947e-05,
+            -0.012842295882002122,
+            0.015810530885294202,
+            0.004572385791133295,
+            0.003685966629168175,
+            0.005093714133656017,
+            0.011877541830396259,
+            -0.013130170365831912,
+            0.01258190257102377,
+            0.011433121961378686,
+            0.01142665981860032,
+            -0.013710025169815697,
+            -0.001774030746825879,
+            -0.01109881340948764,
+            0.0065684584345163735,
+            0.009983753894796107,
+            0.005346443477816739,
+            -0.002062387245572686,
+            0.004950044847352647,
+            0.006105199826576752,
+            -0.06391185159544624,
+            -0.02131511418723256,
+            0.0027165737784844413,
+            0.033389688791557785,
+            -0.007167593552251769,
+            -0.0037646833460996797,
+            0.002883510089742145,
+            0.01356887065340832,
+            0.009730171586767032,
+            0.00418566245984083,
+            0.020320934773473808,
+            -0.008814826860651366,
+            -0.003014193747732543,
+            0.001559573791453607,
+            0.003499420226345028,
+            -0.008703837291821111,
+            0.0044844701302276645,
+            0.0053963630238108035,
+            0.027764369548110287,
+            0.001469968540767627,
+            0.015793086297019008,
+            -0.008505290826588508,
+            -0.004075888558568453,
+            -0.009166710762668385,
+            -0.002860047211143503,
+            -0.011127820563172684,
+            0.003937110717404271,
+            0.02054186721680431,
+            -0.006126180963243227,
+            0.009346433819831456,
+            0.01057504086030076,
+            0.0001650851447316893,
+            -0.012033423322524951,
+            0.0049587335923558,
+            -0.0032240390628760535,
+            -0.005568976687946761,
+            -0.0024667040429312737,
+            -0.03474027725033016,
+            0.026855947684394643,
+            -0.004369626674503285,
+            0.009252512430957932,
+            -0.005081834388513357,
+            -0.0037896610731020337,
+            -0.014201609270756739,
+            0.017871088029615106,
+            -0.008249444723433136,
+            -0.020092257756971717,
+            0.009532994544485572,
+            -0.004991989149115383,
+            0.006464417738670472,
+            -0.02643078108339741,
+            0.0029138922065812495,
+            -0.017232410749655366,
+            -0.003999293090834831,
+            -0.006924488971625995,
+            0.01318082244178942,
+            -0.007232695119856389,
+            -0.02660711037908311,
+            0.01979251106803667,
+            -0.019485860712345603,
+            -0.011977904020170568,
+            -0.0177530973032872,
+            -0.014524003566945937,
+            0.004874952064727108,
+            -0.015326044325348929,
+            0.002047771741041311,
+            -0.009697340332342246,
+            -0.003652304998155628,
+            -0.0008878698629674222,
+            0.0017782435347417623,
+            -0.013017776298713044,
+            -0.018272697230002275,
+            -0.01353020425912225,
+            -0.00757239449706798,
+            -0.0008250000619957538,
+            0.01299142281772714,
+            -0.001679606555140765,
+            -0.007329941421303076,
+            0.0044605462460153116,
+            0.010231192277666706,
+            0.0006558644644452857,
+            0.0024231733856020414,
+            -0.010728844680699885,
+            -0.014137434553169443,
+            0.005230053181525096,
+            -0.012086468475542818,
+            -0.009459717916037003,
+            -0.0028185227600210556,
+            0.014984527039312847,
+            0.008189464334213754,
+            -0.009139240774931914,
+            -0.005639157579163322,
+            0.014325949891859377,
+            0.009020303519690971,
+            -0.013310205617445609,
+            0.01567574314514385,
+            -0.025369802629268523,
+            0.021964786021408878,
+            0.030536905527039065,
+            0.004945288795196981,
+            -0.01168303908203322,
+            0.02281128628230144,
+            -0.004072286832145419,
+            0.0028278384348551146,
+            0.01740522162288968,
+            -0.042558465579514926,
+            -0.013146562547084278,
+            0.002903743316339479,
+            -0.015297177099602701,
+            -0.010963312697387696,
+            0.008804938803398296,
+            -0.0026727734429658817,
+            0.005273238046953793,
+            0.004257613402172282,
+            0.03512246556126453,
+            0.002258612643297096,
+            -0.0002885708728599208,
+            -0.0019720190285930487,
+            0.012312613129996533,
+            0.010451003939611476,
+            -0.015556620910136185,
+            -0.02387856865043814,
+            -0.004583090523619507,
+            -0.0037828776396525835,
+            -0.0014588238692793338,
+            -0.004361834167147271,
+            -0.008159679676329573,
+            0.012243319920992457,
+            -0.017279964797769668,
+            -0.013748032758693542,
+            0.018394195628121487,
+            -0.006411113522842359,
+            0.019002611495040334,
+            -0.014970362556522149,
+            -0.0077437581301661946,
+            0.0017311115715410025,
+            0.024439691379769795,
+            -0.025013956798520212,
+            0.007359134924497539,
+            -0.0031406470488477937,
+            0.017574824561755627,
+            0.00419409250132525,
+            -0.0046803792187546656,
+            0.010556972371044426,
+            0.015229789583480183,
+            -0.004608293492221483,
+            -0.014449487985966966,
+            -0.01172850291078532,
+            0.016823024209386785,
+            -0.012304887597699588,
+            0.0011773351576246157,
+            0.002262670226244277,
+            -0.0023810578015456716,
+            0.03477119000959688,
+            -0.03442178291363675,
+            0.004062693040714608,
+            0.0029249583507755944,
+            0.003291349443702718,
+            0.006461525089528308,
+            0.006047777261425826,
+            -0.020642802018978404,
+            0.008916088381876748,
+            -0.03316762661930299,
+            -0.006367254689957291,
+            -0.005404276207844267,
+            0.0009189006423526284,
+            0.009401330264376041,
+            -0.011106443887384533,
+            -0.02259405372588916,
+            -0.006010439054323966,
+            0.0070744551115721095,
+            0.010262607009446879,
+            -0.002856616377476421,
+            0.019719451109535182,
+            0.008426509638139982,
+            0.0015877648339862691,
+            -0.010655792508751512,
+            -0.015527851374834499,
+            -0.015460974859638219,
+            -0.017847462632188386,
+            -0.02236964311096428,
+            0.011841749854626384,
+            -0.00576688153869993,
+            0.009054902130717224,
+            0.014317707346504314,
+            -0.015366657284925976,
+            0.002910211008094564,
+            0.017761685988205086,
+            0.007974806171830457,
+            0.002692237107055498,
+            0.0032405477366987947,
+            0.02685230091638845,
+            -0.0009116884421301454,
+            -0.0032228977180251673,
+            0.003158213042682977,
+            0.015872199488167846,
+            0.0004966905858799325,
+            0.019981401695749582,
+            0.015201538351780685,
+            0.005359075814444414,
+            -0.004668738193285921,
+            0.019853988551405383,
+            -0.009844394286458195,
+            0.013476144827747423,
+            -0.00411789677405284,
+            -0.00830656434595337,
+            -0.008496132520586954,
+            0.006442581751999304,
+            -0.008138786054348571,
+            0.006977337946948795,
+            0.02419128791801387,
+            -0.0005734827808485311,
+            -0.0028842142002271044,
+            0.005503033191779872,
+            -0.01696639779775455,
+            -0.00389835761601045,
+            -0.03366077608692946,
+            -0.009942613075976742,
+            -0.01708630689497001,
+            -0.02910756562213045,
+            0.026536268897199546,
+            0.00219827495662445,
+            0.02586503862171508,
+            -0.01905122645609761,
+            0.008935637454859214,
+            -0.01385806889332683,
+            0.029747093931541325,
+            0.009163959012862627,
+            0.0008611776492791918,
+            0.003598831885031635,
+            -0.02103045281939351,
+            -0.01150941825152158,
+            0.027341774725604333,
+            -0.0006700608710615124,
+            -0.012089044078073347,
+            0.022396400579038037,
+            0.017491565611764764,
+            0.0051760367056280536,
+            -0.00559536785318233,
+            0.01979262595019399,
+            0.002306380546194741,
+            0.011973863166872444,
+            -0.0028136750606972223,
+            0.012911235736067303,
+            -0.01510421118359169,
+            -0.003338602157155174,
+            0.039758808572572274,
+            0.007290642519242007,
+            -0.001802921446521437,
+            -0.013696774990705774,
+            0.003941198181491921,
+            -0.0008308293948698925,
+            0.0200351513103389,
+            -0.00014327587644797734,
+            0.028291423013030337,
+            -0.0001085859276231927,
+            -0.010723064585989723,
+            0.020443848393732245,
+            -0.017860296930396913,
+            0.04540901629084378,
+            -0.018671578423261854,
+            0.000852031295629291,
+            -0.010639853151638835,
+            -0.019017194262730474,
+            0.003861296738239369,
+            0.0010977143012920602,
+            0.0008387482016061222,
+            0.0060155816314618,
+            -0.010780239332444652,
+            0.002152214497259421,
+            0.010556338812873054,
+            -0.027636192450621728,
+            0.0010938992111792307,
+            -0.014385202899929138,
+            -0.01353192238925814,
+            -0.026667778226211785,
+            -0.00017403398236549974,
+            -0.015357674131635848,
+            -0.013956722505869091,
+            -0.01985038526848704,
+            -0.008927903879720431,
+            -0.009580214158077466,
+            -0.0020314441852913386,
+            -0.004755584878514854,
+            -0.00018388548107372635,
+            -0.002918855567842974,
+            -0.0015941852262316128,
+            -0.014230272061190995,
+            0.02133992212037431,
+            0.015283345362445234,
+            0.004642765060980717,
+            0.012829570028541471,
+            0.01708952142208596,
+            -0.004770647543823141,
+            0.006604762788847952,
+            -0.0025369836845081078,
+            0.008368000846861623,
+            0.014785031726639733,
+            0.012074738887486398,
+            0.02124518119032885,
+            -0.015459818988572564,
+            -0.004992865135357168,
+            -0.012722391036055767,
+            -0.012210202603282509,
+            -0.017850065946244383,
+            -0.004300429148015694,
+            -0.0014387298144714717,
+            -0.023128300275416423,
+            0.004468952290723408,
+            -0.0026602931688877443,
+            -0.005768852559887139,
+            0.007815212168719318,
+            0.0005862972228454734,
+            -0.002265719947629544,
+            -0.00019323737674204774,
+            -0.02735820445266249,
+            0.00024009122317215594,
+            -0.00018701993387558173,
+            -0.014663710240203917,
+            -0.00450797561624596,
+            0.010584731989847334,
+            -0.011966527748437568,
+            -0.00602367292291343,
+            0.0111155226518017,
+            0.0015565179432852256,
+            0.006062862961835896,
+            0.016952559697728433,
+            0.0005012157272316869,
+            0.011160668242334889,
+            0.014993782560797772,
+            -0.01591303023837098,
+            -0.015356412848444082,
+            -0.0236025426434855,
+            -0.004514538154596937,
+            -0.009753334017065119,
+            0.007553951535816575,
+            -0.004616595532996391,
+            -0.013023153809236325,
+            0.006040285197946957,
+            -0.0006343451398642784,
+            0.008338564073158257,
+            -0.017608246105292328,
+            -8.346168119689692e-05,
+            -0.020944870308705715,
+            0.009230123321316848,
+            -0.00531237224627929,
+            -0.00445985089300067,
+            0.01410675485522911,
+            0.02895945580213926,
+            -0.007799830351886844,
+            -0.009455128322575888,
+            -0.005663072244883114,
+            0.012817229945992764,
+            0.012474431030896143,
+            -0.004423141749110039,
+            -0.0018294512146253298,
+            -0.012817101093394574,
+            0.0004755435377918658,
+            -0.01129485262081729,
+            0.0062747111125005155,
+            0.017199374798326412,
+            -0.021965117839794063,
+            -0.01296097993683748,
+            -0.016765996091978336,
+            -0.016882855472266523,
+            0.004372970343291638,
+            0.017290565252244258,
+            0.022670788364378462,
+            -0.008456547449078897,
+            -0.019564049850942417,
+            0.006898353991539946,
+            -0.01620125883985687,
+            0.0037108638122592746,
+            0.12887455743463638,
+            -0.0028864852421660366,
+            -0.011487804449328495,
+            -0.022683334123299446,
+            -0.005467192611880829,
+            0.01210557696712196,
+            -0.01555159294496261,
+            0.06709672179195036,
+            -0.015071089552631105,
+            0.004364902817496082,
+            -0.004899872946473657,
+            0.0006310685609937571,
+            -0.017610935588191347,
+            -0.005962991715597585,
+            -0.007768819050529025,
+            -0.02688607975562596,
+            0.000883763319535686,
+            -0.009935313912803587,
+            -0.012883104064962595,
+            0.002126368813790577,
+            -0.0013697265269800346,
+            -0.016131186838419288,
+            0.01575514418037863,
+            -0.020677990768697592,
+            -0.007251917191831732,
+            -0.0029325623780992118,
+            -0.00600710212646933,
+            0.014573093764769918,
+            -0.03253336117394778,
+            0.002973725493330224,
+            -0.013601417220439602,
+            0.011905788192118131,
+            0.001542117126832946,
+            -0.010789486980776022,
+            -0.013862139885514524,
+            0.007086571810314664,
+            0.015658913897173155,
+            0.017039624764452546,
+            0.01901374401756777,
+            0.004797954569087795,
+            0.003928999581093576,
+            -0.008046722893614655,
+            0.021974371139905104,
+            -0.00697335955527197,
+            0.0008975473383295958,
+            -0.00970589295353857,
+            -0.004346869073499034,
+            0.0031279458769925416,
+            0.02303983493541673,
+            -0.012560380384977343,
+            -0.01536378830925542,
+            0.0012035987157572355,
+            0.033313215063595594,
+            -0.010845084209743817,
+            0.013587126023659577,
+            0.0029867620644300073,
+            0.020679673435988446,
+            -0.0006640787253300063,
+            0.0025071780490108306,
+            0.022554193713122565,
+            -0.018082906254599396,
+            0.018230000220500718,
+            -0.004451773302171552,
+            0.01915083025790679,
+            -0.004183521043835777,
+            0.015019320101074747,
+            -0.027014055979204272,
+            -0.005362044589023088,
+            -0.012355559881051253,
+            0.003995309003198048,
+            0.020680455787871383,
+            0.0054391755324872375,
+            0.009865676762244514,
+            -0.01645386866657314,
+            -0.01760657627715638,
+            0.022969828924328126,
+            -0.021566687194482292,
+            -0.015826424849757402,
+            0.00012804774676298237,
+            -0.015021970627042762,
+            0.006000610438452896,
+            0.010126179023323158,
+            -0.02912121211344025,
+            -0.02046150879033127,
+            0.004881850039543051,
+            -0.0023469840669773042,
+            -0.00915786433296249,
+            0.006547377770403066,
+            -0.022329063637464074,
+            -0.015702777677743352,
+            -0.005692904224111613,
+            0.01207999389934878,
+            -0.008429160038192635,
+            0.010075061963131637,
+            -0.020106712664936504,
+            0.0036787045650556907,
+            -0.002764965946892891,
+            0.010307702251974629,
+            0.004697169247387421,
+            0.011192285826515207,
+            0.0030113901179382946,
+            0.0014774681536380047,
+            -0.02778270000997313,
+            0.010043261430740039,
+            -5.178406450222847e-05,
+            0.04216636255860709,
+            0.030977096295679525,
+            0.004479857028809617,
+            -5.798020745008712e-05,
+            -0.009996471945562616,
+            0.012476009017187798,
+            -0.019306907391548302,
+            -0.005092322343599383,
+            0.01856463597985508,
+            0.02195467141594795,
+            -0.02236232264139305,
+            0.0052217593371508145,
+            -0.016766955100332014,
+            0.016133299654723637,
+            -0.011208763137897244,
+            0.006248024409576809,
+            0.011601988603146743,
+            0.0022330878390492527,
+            0.010317781892717903,
+            -0.005895142237225119,
+            0.019670196605183973,
+            0.01186677200787666,
+            -0.002339367025714548,
+            0.02367408853110795,
+            0.006858134385159451,
+            0.004479683673962342,
+            0.004798719889930319,
+            -0.009604173975576537,
+            -0.009045602998213713,
+            0.009230969488015385,
+            0.004083616606774546,
+            -0.007944628589259331,
+            0.011836323287097144,
+            0.015100350195476968,
+            0.01885317701149666,
+            -0.0011707847499298092,
+            -0.00684548433712413,
+            -0.006550059418141426,
+            0.001899758393077906,
+            -0.024577381680573596,
+            -0.02855511854789239,
+            0.021238935710272536,
+            -0.010427135883958709,
+            -0.019187769881105637,
+            -0.0015864640188683347,
+            0.016360037306061392,
+            -0.008853972319021563,
+            -0.01627167079143546,
+            0.001270414822216867,
+            0.02255755055961612,
+            -0.005959411125391464,
+            0.006009654905468478,
+            0.005909233773418581,
+            0.005874361519568704,
+            -0.0027859463941992753,
+            -0.024599789577567636,
+            0.0036612374968223453,
+            0.03221346865585554,
+            0.016461563205582293,
+            0.016139782455492073,
+            -0.007158168059808459,
+            0.001838455290310879,
+            0.007530954381521755,
+            0.005438791523412295,
+            0.015674000912615536,
+            0.008604106101542998,
+            -0.006990462920227135,
+            0.00244736628387359,
+            0.012213048255418866,
+            -0.007329299946651028,
+            0.017424711328323798,
+            0.001973882532418776,
+            -0.0133178791530314,
+            -0.008373452013951822,
+            -0.010462150707776556,
+            -0.014575483571750984,
+            0.0027958704799249897,
+            -0.011401916623878957,
+            0.0008016041012669959,
+            0.003953958717263787,
+            0.010260753041492368,
+            0.013961649189650642,
+            -0.0019275791443041316,
+            0.017019168629298462,
+            0.011534594243860177,
+            -0.0017859096307142257,
+            -0.002970739203219076,
+            -0.03142778833370656,
+            0.010207789755749909,
+            -0.014730553506704018,
+            -0.0037611416875830746,
+            0.01825790347981639,
+            -0.007766890847082858,
+            -0.0032778433418881666,
+            -0.021003405429807113,
+            0.0024512381432602736,
+            0.008768616692363321,
+            0.01558713989712387,
+            -0.012819542714293536,
+            0.009201155584074711,
+            -0.0012254337002053017,
+            -0.0057451574939563045,
+            0.00977251392462179,
+            0.0042356831792011935,
+            -0.0033433823868435786,
+            0.011499764125503874
+          ],
+          "title": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer",
+          "keyphrases": [
+            "Sparsely-Gated Mixture-of-Experts Layer",
+            "conditional computation",
+            "neural network",
+            "language modeling",
+            "machine translation"
+          ]
+        },
+        "type": "document"
+      },
+      "bidirectional": true,
+      "properties": {
+        "summary_cosine_similarity": 0.6562208111520341
+      }
+    },
+    {
+      "id": "2508dca3-1113-483c-b3da-d0dab8609366",
+      "type": "summary_cosine_similarity",
+      "source": {
+        "id": "14339b1f-53f8-479b-9da7-9430b174b9ee",
+        "properties": {
+          "page_content": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer\n\nNoam Shazeer1, Azalia Mirhoseini∗†1, Krzysztof Maziarz∗2, Andy Davis1, Quoc Le1, Geoffrey Hinton1and Jeff Dean1 1Google Brain, {noam,azalia,andydavis,qvl,geoffhinton,jeff}@google.com 2Jagiellonian University, Cracow, krzysztof.maziarz@student.uj.edu.pl\n\nAbstract\n\nThe capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost.\n\n1 Introduction And Related Work 1.1 Conditional Computation\n\nExploiting scale in both training data and model size has been central to the success of deep learning. When datasets are sufficiently large, increasing the capacity (number of parameters) of neural networks can give much better prediction accuracy. This has been shown in domains such as text (Sutskever et al., 2014; Bahdanau et al., 2014; Jozefowicz et al., 2016; Wu et al., 2016), images (Krizhevsky et al., 2012; Le et al., 2012), and audio (Hinton et al., 2012; Amodei et al., 2015). For typical deep learning models, where the entire model is activated for every example, this leads to a roughly quadratic blow-up in training costs, as both the model size and the number of training examples increase. Unfortunately, the advances in computing power and distributed computation fall short of meeting such demand.\n\nVarious forms of conditional computation have been proposed as a way to increase model capacity without a proportional increase in computational costs (Davis & Arel, 2013; Bengio et al., 2013; Eigen et al., 2013; Ludovic Denoyer, 2014; Cho & Bengio, 2014; Bengio et al., 2015; Almahairi et al., 2015). In these schemes, large parts of a network are active or inactive on a per-example basis. The gating decisions may be binary or sparse and continuous, stochastic or deterministic. Various forms of reinforcement learning and back-propagation are proposed for trarining the gating decisions.\n\nWhile these ideas are promising in theory, no work to date has yet demonstrated massive improvements in model capacity, training time, or model quality. We blame this on a combination of the following challenges:\n\nModern computing devices, especially GPUs, are much faster at arithmetic than at branching. Most of the works above recognize this and propose turning on/off large chunks of the network with each gating decision.\n\nLarge batch sizes are critical for performance, as they amortize the costs of parameter transfers and updates. Conditional computation reduces the batch sizes for the conditionally active chunks of the network.\n\nNetwork bandwidth can be a bottleneck. A cluster of GPUs may have computational power thousands of times greater than the aggregate inter-device network bandwidth. To be computationally efficient, the relative computational versus network demands of an algorithm must exceed this ratio. Embedding layers, which can be seen as a form of conditional computation, are handicapped by this very problem. Since the embeddings generally need to be sent across the network, the number of (example, parameter) interactions is limited by network bandwidth instead of computational capacity.\n\nDepending on the scheme, loss terms may be necessary to achieve the desired level of sparsity per-chunk and/or per example. Bengio et al. (2015) use three such terms. These issues can affect both model quality and load-balancing.\n\nModel capacity is most critical for very large data sets. The existing literature on conditional computation deals with relatively small image recognition data sets consisting of up to 600,000 images. It is hard to imagine that the labels of these images provide a sufficient signal to adequately train a model with millions, let alone billions of parameters. In this work, we for the first time address all of the above challenges and finally realize the promise of conditional computation. We obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets.\n\n1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1). All parts of the network are trained jointly by back-propagation.\n\nWhile the introduced technique is generic, in this paper we focus on language modeling and machine translation tasks, which are known to benefit from very large models. In particular, we apply a MoE convolutionally between stacked LSTM layers (Hochreiter & Schmidhuber, 1997), as in Figure 1.\n\nThe MoE is called once for each position in the text, selecting a potentially different combination of experts at each position. The different experts tend to become highly specialized based on syntax and semantics (see Appendix E Table 9). On both language modeling and machine translation benchmarks, we improve on best published results at a fraction of the computational cost.\n\n1.3 Related Work On Mixtures Of Experts\n\nSince its introduction more than two decades ago (Jacobs et al., 1991; Jordan & Jacobs, 1994), the mixture-of-experts approach has been the subject of much research. Different types of expert architectures hae been proposed such as SVMs (Collobert et al., 2002), Gaussian Processes (Tresp, 2001; Theis & Bethge, 2015; Deisenroth & Ng, 2015), Dirichlet Processes (Shahbaba & Neal, 2009), and deep networks. Other work has focused on different expert configurations such as a hierarchical structure (Yao et al., 2009), infinite numbers of experts (Rasmussen & Ghahramani, 2002), and adding experts sequentially (Aljundi et al., 2016). Garmash & Monz (2016) suggest an ensemble model in the format of mixture of experts for machine translation. The gating network is trained on a pre-trained ensemble NMT model. The works above concern top-level mixtures of experts. The mixture of experts is the whole model. Eigen et al. (2013) introduce the idea of using multiple MoEs with their own gating networks as parts of a deep model. It is intuitive that the latter approach is more powerful, since complex problems may contain many sub-problems each requiring different experts. They also allude in their conclusion to the potential to introduce sparsity, turning MoEs into a vehicle for computational computation.\n\nOur work builds on this use of MoEs as a general purpose neural network component. While Eigen et al. (2013) uses two stacked MoEs allowing for two sets of gating decisions, our convolutional application of the MoE allows for different gating decisions at each position in the text. We also realize sparse gating and demonstrate its use as a practical way to massively increase model capacity.\n\n2 The Structure Of The Mixture-Of-Experts Layer\n\nThe Mixture-of-Experts (MoE) layer consists of a set of n \"expert networks\" E1, · · · , En, and a \"gating network\" G whose output is a sparse n-dimensional vector. Figure 1 shows an overview of the MoE module. The experts are themselves neural networks, each with their own parameters.\n\nAlthough in principle we only require that the experts accept the same sized inputs and produce the same-sized outputs, in our initial investigations in this paper, we restrict ourselves to the case where the models are feed-forward networks with identical architectures, but with separate parameters.\n\nLet us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows:\n\n$$y=\\sum_{i=1}^{n}G(x){i}E{i}(x)$$ $$(\\mathbf{l})$$ G(x)iEi(x) (1) We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example. If the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network. In the following we focus on ordinary MoEs. We provide more details on hierarchical MoEs in Appendix B. Our implementation is related to other models of conditional computation. A MoE whose experts are simple weight matrices is similar to the parameterized weight matrix proposed in (Cho & Bengio, 2014). A MoE whose experts have one hidden layer is similar to the block-wise dropout described in (Bengio et al., 2015), where the dropped-out layer is sandwiched between fully-activated layers.\n\n2.1 Gating Network\n\nSoftmax Gating: A simple choice of non-sparse gating function (Jordan & Jacobs, 1994) is to multiply the input by a trainable weight matrix Wg and then apply the Sof tmax function.\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ $\\eqref{eq:walpha}$. Gσ(x) = Sof tmax(x · Wg) (2) Noisy Top-K Gating: We add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to −∞ (which causes the corresponding gate values to equal 0). The sparsity serves to save computation, as described above. While this form of sparsity creates some theoretically scary discontinuities in the output of gating function, we have not yet observed this to be a problem in practice. The noise term helps with load balancing, as will be discussed in Appendix A. The amount of noise per component is controlled by a second trainable weight matrix Wnoise.\n\n$$G(x)=S o f t m a x(K e e p T o p K(H(x),k))$$ $$(4)$$ $$(S)$$ G(x) = Sof tmax(KeepT opK(H(x), k)) (3) $$H(x){i}=(x\\cdot W{g}){i}+S t a n d a r d N o r m a l()\\cdot S o f t p l u s((x\\cdot W{n o i s e}){i})$$ $KeepTopK(v,k){i}=\\begin{cases}v_{i}&\\text{if}v_{i}\\text{is in the top}k\\text{elements of}v.\\ -\\infty&\\text{otherwise.}\\end{cases}$ Training the Gating Network We train the gating network by simple back-propagation, along with the rest of the model. If we choose k > 1, the gate values for the top k experts have nonzero derivatives with respect to the weights of the gating network. This type of occasionally-sensitive behavior is described in (Bengio et al., 2013) with respect to noisy rectifiers. Gradients also backpropagate through the gating network to its inputs. Our method differs here from (Bengio et al., 2015) who use boolean gates and a REINFORCE-style approach to train the gating network.\n\n3 Addressing Performance Challenges 3.1 The Shrinking Batch Problem\n\nOn modern CPUs and GPUs, large batch sizes are necessary for computational efficiency, so as to amortize the overhead of parameter loads and updates. If the gating network chooses k out of n experts for each example, then for a batch of b examples, each expert receives a much smaller batch of approximately kb n b examples. This causes a naive MoE implementation to become very inefficient as the number of experts increases. The solution to this shrinking batch problem is to make the original batch size as large as possible. However, batch size tends to be limited by the memory necessary to store activations between the forwards and backwards passes. We propose the following techniques for increasing the batch size: Mixing Data Parallelism and Model Parallelism: In a conventional distributed training setting, multiple copies of the model on different devices asynchronously process distinct batches of data, and parameters are synchronized through a set of parameter servers. In our technique, these different batches run synchronously so that they can be combined for the MoE layer. We distribute the standard layers of the model and the gating network according to conventional data-parallel schemes, but keep only one shared copy of each expert. Each expert in the MoE layer receives a combined batch consisting of the relevant examples from all of the data-parallel input batches. The same set of devices function as data-parallel replicas (for the standard layers and the gating networks) and as model-parallel shards (each hosting a subset of the experts). If the model is distributed over d devices, and each device processes a batch of size b, each expert receives a batch of approximately kbd nexamples. Thus, we achieve a factor of d improvement in expert batch size.\n\nIn the case of a hierarchical MoE (Section B), the primary gating network employs data parallelism, and the secondary MoEs employ model parallelism. Each secondary MoE resides on one device.\n\nThis technique allows us to increase the number of experts (and hence the number of parameters) by proportionally increasing the number of devices in the training cluster. The total batch size increases, keeping the batch size per expert constant. The memory and bandwidth requirements per device also remain constant, as do the step times, as does the amount of time necessary to process a number of training examples equal to the number of parameters in the model. It is our goal to train a trillionparameter model on a trillion-word corpus. We have not scaled our systems this far as of the writing of this paper, but it should be possible by adding more hardware.\n\nTaking Advantage of Convolutionality: In our language models, we apply the same MoE to each time step of the previous layer. If we wait for the previous layer to finish, we can apply the MoE to all the time steps together as one big batch. Doing so increases the size of the input batch to the MoE layer by a factor of the number of unrolled time steps.\n\nIncreasing Batch Size for a Recurrent MoE: We suspect that even more powerful models may involve applying a MoE recurrently. For example, the weight matrices of a LSTM or other RNN could be replaced by a MoE. Sadly, such models break the convolutional trick from the last paragraph, since the input to the MoE at one timestep depends on the output of the MoE at the previous timestep. Gruslys et al. (2016) describe a technique for drastically reducing the number of stored activations in an unrolled RNN, at the cost of recomputing forward activations. This would allow for a large increase in batch size.\n\n3.2 Network Bandwidth\n\nAnother major performance concern in distributed computing is network bandwidth. Since the experts are stationary (see above) and the number of gating parameters is small, most of the communication involves sending the inputs and outputs of the experts across the network. To maintain computational efficiency, the ratio of an expert's computation to the size of its input and output must exceed the ratio of computational to network capacity of the computing device. For GPUs, this may be thousands to one. In our experiments, we use experts with one hidden layer containing thousands of RELU-activated units. Since the weight matrices in the expert have sizes input_size×hidden_size and hidden_size × output_size, the ratio of computation to input and output is equal to the size of the hidden layer. Conveniently, we can increase computational efficiency simply by using a larger hidden layer, or more hidden layers.\n\n4 Balancing Expert Utilization\n\nWe have observed that the gating network tends to converge to a state where it always produces large weights for the same few experts. This imbalance is self-reinforcing, as the favored experts are trained more rapidly and thus are selected even more by the gating network. Eigen et al. (2013) describe the same phenomenon, and use a hard constraint at the beginning of training to avoid this local minimum. Bengio et al. (2015) include a soft constraint on the batch-wise average of each gate.1 We take a soft constraint approach. We define the importance of an expert relative to a batch of training examples to be the batchwise sum of the gate values for that expert. We define an additional loss Limportance, which is added to the overall loss function for the model. This loss is equal to the square of the coefficient of variation of the set of importance values, multiplied by a hand-tuned scaling factor wimportance. This additional loss encourages all experts to have equal importance.\n\n$$I m p o r t a n c e(X)=\\sum_{x\\in X}G(x)$$ $$L_{importance}(X)=w_{importance}\\cdot CV(Importance(X))^{2}\\tag{7}$$ $$(6)$$\n\nWhile this loss function can ensure equal importance, experts may still receive very different numbers of examples. For example, one expert may receive a few examples with large weights, and another may receive many examples with small weights. This can cause memory and performance problems on distributed hardware. To solve this problem, we introduce a second loss function, Lload , which ensures balanced loads. Appendix A contains the definition of this function, along with experimental results.\n\n5 Experiments 5.1 1 Billion Word Language Modeling Benchmark\n\nDataset: This dataset, introduced by (Chelba et al., 2013) consists of shuffled unique sentences from news articles, totaling approximately 829 million words, with a vocabulary of 793,471 words.\n\nPrevious State-of-the-Art: The best previously published results (Jozefowicz et al., 2016) use models consisting of one or more stacked Long Short-Term Memory (LSTM) layers (Hochreiter & Schmidhuber, 1997; Gers et al., 2000). The number of parameters in the LSTM layers of these models vary from 2 million to 151 million. Quality increases greatly with parameter count, as do computational costs. Results for these models form the top line of Figure 2-right. MoE Models: Our models consist of two stacked LSTM layers with a MoE layer between them (see Figure 1). We vary the sizes of the layers and the number of experts. For full details on model architecture, training regimen, additional baselines and results, see Appendix C.\n\nLow Computation, Varied Capacity: To investigate the effects of adding capacity, we trained\n\na series of MoE models all with roughly equal computational costs: about 8 million multiply-andadds per training example per timestep in the forwards pass, excluding the softmax layer. We call this metric (ops/timestep). We trained models with flat MoEs containing 4, 32, and 256 experts, and models with hierarchical MoEs containing 256, 1024, and 4096 experts. Each expert had about 1 million parameters. For all the MoE layers, 4 experts were active per input.\n\nThe results of these models are shown in Figure 2-left. The model with 4 always-active experts performed (unsurprisingly) similarly to the computationally-matched baseline models, while the largest of the models (4096 experts) achieved an impressive 24% lower perplexity on the test set. Figure 2: Model comparison on 1-Billion-Word Language-Modeling Benchmark. On the left, we plot test perplexity as a function of model capacity for models with similar computational budgets of approximately 8-million-ops-per-timestep. On the right, we plot test perplexity as a function of computational budget. The top line represents the LSTM models from (Jozefowicz et al., 2016).\n\nThe bottom line represents 4-billion parameter MoE models with different computational budgets.\n\nVaried Computation, High Capacity: In addition to the largest model from the previous section, we trained two more MoE models with similarly high capacity (4 billion parameters), but higher computation budgets. These models had larger LSTMs, and fewer but larger and experts. Details\n\nTest Test #Parameters ops/timestep Training TFLOPS Perplexity Perplexity excluding embedding Time /GPU 10 epochs 100 epochs and softmax layers 10 epochs Best Published Results 34.7 30.6 151 million 151 million 59 hours, 32 k40s 1.09 Low-Budget MoE Model 34.1 4303 million 8.9 million 15 hours, 16 k40s 0.74 Medium-Budget MoE Model 31.3 4313 million 33.8 million 17 hours, 32 k40s 1.22 High-Budget MoE Model 28.0 4371 million 142.7 million 47 hours, 32 k40s 1.56\n\nTable 1: Summary of high-capacity MoE-augmented models with varying computational budgets, vs. best previously published results (Jozefowicz et al., 2016). Details in Appendix C. can be found in Appendix C.2. Results of these three models form the bottom line of Figure 2-right.\n\nTable 1 compares the results of these models to the best previously-published result on this dataset .\n\nEven the fastest of these models beats the best published result (when controlling for the number of training epochs), despite requiring only 6% of the computation.\n\nComputational Efficiency: We trained our models using TensorFlow (Abadi et al., 2016) on clusters containing 16-32 Tesla K40 GPUs. For each of our models, we determine computational efficiency in TFLOPS/GPU by dividing the number of floating point operations required to process one training batch by the observed step time and the number of GPUs in the cluster. The operation counts used here are higher than the ones we report in our ops/timestep numbers in that we include the backwards pass, we include the importance-sampling-based training of the softmax layer, and we count a multiply-and-add as two separate operations. For all of our MoE models, the floating point operations involved in the experts represent between 37% and 46% of the total.\n\nFor our baseline models wtih no MoE, observed computational efficiency ranged from 1.07-1.29 TFLOPS/GPU. For our low-computation MoE models, computation efficiency ranged from 0.740.90 TFLOPS/GPU, except for the 4-expert model which did not make full use of the available parallelism. Our highest-computation MoE model was more efficient at 1.56 TFLOPS/GPU, likely due to the larger matrices. These numbers represent a significant fraction of the theoretical maximum of 4.29 TFLOPS/GPU claimed by NVIDIA. Detailed results are in Appendix C, Table 7.\n\n5.2 100 BILLION WORD GOOGLE NEWS CORPUS\n\nOn the 1-billion-word corpus, adding additional capacity seems to produce diminishing returns as the number of parameters in the MoE layer exceeds 1 billion, as can be seen in Figure 2-left. We hypothesized that for a larger training set, even higher capacities would produce significant quality improvements. We constructed a similar training set consisting of shuffled unique sentences from Google's internal news corpus, totalling roughly 100 billion words. Similarly to the previous section, we tested a series of models with similar computational costs of about 8 million ops/timestep. In addition to a baseline LSTM model, we trained models augmented with MoE layers containing 32, 256, 1024, 4096, 16384, 65536, and 131072 experts. This corresponds to up to 137 billion parameters in the MoE layer. Details on architecture, training, and results are given in Appendix D.\n\nResults: Figure 3 shows test perplexity as a function of capacity after training on 10 billion words (top line) and 100 billion words (bottom line). When training over the full 100 billion words, test perplexity improves significantly up to 65536 experts (68 billion parameters), dropping 39% lower than the computationally matched baseline, but degrades at 131072 experts, possibly a result of too much sparsity. The widening gap between the two lines demonstrates (unsurprisingly) that increased model capacity helps more on larger training sets. Even at 65536 experts (99.994% layer sparsity), computational efficiency for the model stays at a respectable 0.72 TFLOPS/GPU.\n\n5.3 Machine Translation (Single Language Pair)\n\nModel Architecture: Our model was a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decreased the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We inserted MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). Each MoE layer contained up to 2048 experts each with about two million parameters, adding a total of about 8 billion parameters to the models. Further details on model architecture, testing procedure and results can be found in Appendix E.\n\nDatasets: We benchmarked our method on the WMT'14 En→Fr and En→De corpora, whose training sets have 36M sentence pairs and 5M sentence pairs, respectively. The experimental protocols were also similar to those in (Wu et al., 2016): newstest2014 was used as the test set to compare against previous work (Luong et al., 2015a; Zhou et al., 2016; Wu et al., 2016), while the combination of newstest2012 and newstest2013 was used as the development set. We also tested the same model on a Google's Production English to French data.\n\nTable 2: Results on WMT'14 En→ Fr newstest2014 (bold values represent best results). Model Test Test ops/timenstep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.69 40.35 85M 8.7B 3 days/64 k40s MoE with 2048 Experts (longer training) 2.63 40.56 85M 8.7B 6 days/64 k40s GNMT (Wu et al., 2016) 2.79 39.22 214M 278M 6 days/96 k80s GNMT+RL (Wu et al., 2016) 2.96 39.92 214M 278M 6 days/96 k80s PBMT (Durrani et al., 2014) 37.0 LSTM (6-layer) (Luong et al., 2015b) 31.5 LSTM (6-layer+PosUnk) (Luong et al., 2015b) 33.1 DeepAtt (Zhou et al., 2016) 37.7 DeepAtt+PosUnk (Zhou et al., 2016) 39.2\n\nTable 3: Results on WMT'14 En → De newstest2014 (bold values represent best results). Model Test Test ops/timestep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 4.64 26.03 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 5.25 24.91 214M 278M 1 day/96 k80s GNMT +RL (Wu et al., 2016) 8.08 24.66 214M 278M 1 day/96 k80s PBMT (Durrani et al., 2014) 20.7 DeepAtt (Zhou et al., 2016) 20.6\n\nTable 4: Results on the Google Production En→ Fr dataset (bold values represent best results). Model Eval Eval Test Test ops/timestep Total Training Perplexity BLEU Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.60 37.27 2.69 36.57 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 2.78 35.80 2.87 35.56 214M 278M 6 days/96 k80s\n\nResults: Tables 2, 3, and 4 show the results of our largest models, compared with published results. Our approach achieved BLEU scores of 40.56 and 26.03 on the WMT'14 En→Fr and En→De benchmarks. As our models did not use RL refinement, these results constitute significant gains of 1.34 and 1.12 BLEU score on top of the strong baselines in (Wu et al., 2016). The perplexity scores are also better.2 On the Google Production dataset, our model achieved 1.01 higher test BLEU score even after training for only one sixth of the time.\n\n5.4 Multilingual Machine Translation\n\nDataset: (Johnson et al., 2016) train a single GNMT (Wu et al., 2016) model on a very large combined dataset of twelve language pairs. Results are somewhat worse than those for 12 separately trained single-pair GNMT models. This is not surprising, given that the twelve models have 12 times the capacity and twelve times the aggregate training of the one model. We repeat this experiment with a single MoE-augmented model. See Appendix E for details on model architecture.\n\nWe train our model on the same dataset as (Johnson et al., 2016) and process the same number of training examples (about 3 billion sentence pairs). Our training time was shorter due to the lower computational budget of our model.\n\nResults: Results for the single-pair GNMT models, the multilingual GNMT model and the multilingual MoE model are given in Table 5. The MoE model achieves 19% lower perplexity on the dev set than the multilingual GNMT model. On BLEU score, the MoE model significantly beats the multilingual GNMT model on 11 of the 12 language pairs (by as much as 5.84 points), and even beats the monolingual GNMT models on 8 of 12 language pairs. The poor performance on English → Korean seems to be a result of severe overtraining, as for the rarer language pairs a small number of real examples were highly oversampled in the training corpus.\n\nTable 5: Multilingual Machine Translation (bold values represent best results). GNMT-Mono GNMT-Multi MoE-Multi MoE-Multi vs. GNMT-Multi Parameters 278M / model 278M 8.7B ops/timestep 212M 212M 102M training time, hardware various 21 days, 96 k20s 12 days, 64 k40s Perplexity (dev) 4.14 3.35 -19% French → English Test BLEU 36.47 34.40 37.46 +3.06 German → English Test BLEU 31.77 31.17 34.80 +3.63 Japanese → English Test BLEU 23.41 21.62 25.91 +4.29 Korean → English Test BLEU 25.42 22.87 28.71 +5.84 Portuguese → English Test BLEU 44.40 42.53 46.13 +3.60 Spanish → English Test BLEU 38.00 36.04 39.39 +3.35 English → French Test BLEU 35.37 34.00 36.59 +2.59 English → German Test BLEU 26.43 23.15 24.53 +1.38 English → Japanese Test BLEU 23.66 21.10 22.78 +1.68 English → Korean Test BLEU 19.75 18.41 16.62 -1.79 English → Portuguese Test BLEU 38.40 37.35 37.90 +0.55 English → Spanish Test BLEU 34.50 34.25 36.21 +1.96\n\n6 Conclusion\n\nThis work is the first to demonstrate major wins from conditional computation in deep networks.\n\nWe carefully identified the design considerations and challenges of conditional computing and addressed them with a combination of algorithmic and engineering solutions. While we focused on text, conditional computation may help in other domains as well, provided sufficiently large training sets. We look forward to seeing many novel implementations and applications of conditional computation in the years to come.\n\nAcknowledgments\n\nWe would like to thank all of the members of the Google Brain and Google Translate teams who helped us with this project, in particular Zhifeng Chen, Yonghui Wu, and Melvin Johnson. Thanks also to our anonymous ICLR reviewers for the helpful suggestions on making this paper better.\n\n2Reported perplexities relative to the tokenization used by both our models and GNMT.\n\nReferences\n\nMartín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Gordon Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda B. Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. CoRR, abs/1603.04467, 2016. URL http://arxiv.org/abs/1603.04467.\n\nRahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. CoRR, abs/1611.06194, 2016. URL http://arxiv.org/abs/1611.\n\n06194.\n\nA. Almahairi, N. Ballas, T. Cooijmans, Y. Zheng, H. Larochelle, and A. Courville. Dynamic Capacity Networks. ArXiv e-prints, November 2015.\n\nDario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan, Christopher Fougner, Tony Han, Awni Y. Hannun, Billy Jun, Patrick LeGresley, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595, 2015. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.\n\nEmmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297, 2015.\n\nYoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.\n\nCiprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling.\n\narXiv preprint arXiv:1312.3005, 2013.\n\nK. Cho and Y. Bengio. Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning. ArXiv e-prints, June 2014. Ronan Collobert, Samy Bengio, and Yoshua Bengio. A parallel mixture of SVMs for very large scale problems. Neural Computing, 2002.\n\nAndrew Davis and Itamar Arel. Low-rank approximations for conditional feedforward computation in deep neural networks. arXiv preprint arXiv:1312.4461, 2013. Marc Peter Deisenroth and Jun Wei Ng. Distributed Gaussian processes. In ICML, 2015.\n\nJohn Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization, 2010.\n\nNadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. Edinburgh's phrase-based machine translation systems for wmt-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation, 2014.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013. Ekaterina Garmash and Christof Monz. Ensemble learning for multi-source neural machine translation. In staff.science.uva.nl/c.monz, 2016.\n\nFelix A. Gers, Jürgen A. Schmidhuber, and Fred A. Cummins. Learning to forget: Continual prediction with lstm. Neural Computation, 2000.\n\nAudrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. Memory-efficient backpropagation through time. CoRR, abs/1606.03401, 2016. URL http://arxiv.org/ abs/1606.03401. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2015.\n\nGeoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 2012. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.\n\nSergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.\n\nRobert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computing, 1991.\n\nMelvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation.\n\nCoRR, abs/1611.04558, 2016. URL http://arxiv.org/abs/1611.04558.\n\nMichael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm.\n\nNeural Computing, 1994. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.\n\nDiederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.\n\nReinhard Kneser and Hermann. Ney. Improved backingoff for m-gram language modeling., 1995.\n\nAlex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.\n\nQuoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeffrey Dean, and Andrew Y. Ng. Building high-level features using large scale unsupervised learning. In ICML, 2012. Patrick Gallinari Ludovic Denoyer. Deep sequential neural network. arXiv preprint arXiv:1410.0510, 2014.\n\nMinh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attentionbased neural machine translation. EMNLP, 2015a.\n\nMinh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the rare word problem in neural machine translation. ACL, 2015b.\n\nCarl Edward Rasmussen and Zoubin Ghahramani. Infinite mixtures of Gaussian process experts.\n\nNIPS, 2002.\n\nHasim Sak, Andrew W Senior, and Françoise Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH, pp. 338–342, 2014.\n\nMike Schuster and Kaisuke Nakajima. Japanese and Korean voice search. ICASSP, 2012.\n\nBabak Shahbaba and Radford Neal. Nonlinear models using dirichlet process mixtures. JMLR, 2009.\n\nIlya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks.\n\nIn NIPS, 2014.\n\nLucas Theis and Matthias Bethge. Generative image modeling using spatial LSTMs. In NIPS, 2015. Volker Tresp. Mixtures of Gaussian Processes. In NIPS, 2001.\n\nYonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.\n\nBangpeng Yao, Dirk Walther, Diane Beck, and Li Fei-fei. Hierarchical mixture of classification experts uncovers interactions between brain regions. In NIPS. 2009.\n\nWojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization.\n\narXiv preprint arXiv:1409.2329, 2014.\n\nJie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with fast-forward connections for neural machine translation. arXiv preprint arXiv:1606.04199, 2016.\n\nAppendices A Load-Balancing Loss\n\nAs discussed in section 4, for load-balancing purposes, we want to define an additional loss function to encourage experts to receive roughly equal numbers of training examples. Unfortunately, the number of examples received by an expert is a discrete quantity, so it can not be used in backpropagation. Instead, we define a smooth estimator Load(X) of the number of examples assigned to each expert for a batch X of inputs. The smoothness allows us to back-propagate gradients through the estimator. This is the purpose of the noise term in the gating function. We define P(x, i) as the probability that G(x)iis nonzero, given a new random choice of noise on element i, but keeping the already-sampled choices of noise on the other elements. To compute P(x, i), we note that the G(x)iis nonzero if and only if H(x)iis greater than the k th-greatest element of H(x) excluding itself. The probability works out to be:\n\n$$P(x,i)=P r\\Big((x\\cdot W_{g}){i}+S t a n d x N o r a l()\\cdot S o f t p u s((x\\cdot W{n o i s e})_{i})$$ $$>k t h_e x c l u d i n(H(x),k,i)\\Big)$$ $$({\\boldsymbol{8}})$$\n\n$$(9)$$ $$(10)$$ (8) Where kth_excluding(v, k, i) means the kth highest component of v, excluding component i. Simplifying, we get:\n\n$$P(x,i)=\\Phi{\\Big(}{\\frac{(x\\cdot W_{g}){i}-k t h_c x c l u d i n g(H(x),k,i)}{S o f t p l u s((x\\cdot W{n o i s e})_{i})}}{\\Big)}$$\n\nWhere Φ is the CDF of the standard normal distribution.\n\n$$L o a d(X){i}=\\sum{x\\in X}P(x,i)$$\n\n$$(11)$$ P(x, i) (10) We can now define the load loss to be the square of the coefficient of variation of the load vector, multiplied by a hand-tuned scaling factor wload.\n\n$$L_{l o a d}(X)=w_{l o a d}\\cdot C V(L o a d(X))^{2}$$ Lload(X) = wload · CV (Load(X))2(11) Initial Load Imbalance: To avoid out-of-memory errors, we need to initialize the network in a state of approximately equal expert load (since the soft constraints need some time to work). To accomplish this, we initialize the matrices Wg and Wnoise to all zeros, which yields no signal and some noise.\n\nExperiments: We trained a set of models with identical architecture (the MoE-256 model described in Appendix C), using different values of wimportance and wload. We trained each model for 10 epochs, then measured perplexity on the test set. We also measured the coefficients of variation in Importance and Load, as well as ratio of the load on the most overloaded expert to the average load. This last value is significant for load balancing purposes on distributed hardware. All of these metrics were averaged over several training batches.\n\nwimportance wload Test Perplexity CV (Importance(X)) CV (Load(X)) max(Load(X)) mean(Load(X)) 0.0 0.0 39.8 3.04 3.01 17.80 0.2 0.0 35.6 0.06 0.17 1.47 0.0 0.2 35.7 0.22 0.04 1.15 0.1 0.1 35.6 0.06 0.05 1.14 0.01 0.01 35.7 0.48 0.11 1.37 1.0 1.0 35.7 0.03 0.02 1.07\n\nTable 6: Experiments with different combinations of losses.\n\n13 Results: Results are reported in Table 6. All the combinations containing at least one the two losses led to very similar model quality, where having no loss was much worse. Models with higher values of wload had lower loads on the most overloaded expert.\n\nB Hierachical Mixture Of Experts\n\nIf the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network.3If the hierarchical MoE consists of a groups of b experts each, we denote the primary gating network by Gprimary, the secondary gating networks by (G1, G2..Ga), and the expert networks by (E0,0, E0,1..Ea,b). The output of the MoE is given by:\n\n$$y_{H}=\\sum_{i=1}^{a}\\sum_{j=1}^{b}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}\\cdot E{i,j}(x)$$ $$(12)$$ $$(13)$$ $$(14)$$\n\nOur metrics of expert utilization change to the following:\n\n$$I m p o r t a n c e_{H}(X){i,j}=\\sum{x\\in X}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}$$ $$L o a d{H}(X){i,j}={\\frac{L o a d{p r i m a r y}(X){i}\\cdot L o a d{i}(X^{(i)})_{j}}{|X^{(i)}|}}$$\n\nLoadprimary and Loadi deonte the Load functions for the primary gating network and i th secondary gating network respectively. X(i) denotes the subset of X for which Gprimary(x)i > 0.\n\nIt would seem simpler to let LoadH(X)i,j = Loadi(Xi)j , but this would not have a gradient with respect to the primary gating network, so we use the formulation above.\n\nC 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS C.1 8-MILLION-OPERATIONS-PER-TIMESTEP MODELS Model Architecture: Our model consists of five layers: a word embedding layer, a recurrent Long Short-Term Memory (LSTM) layer (Hochreiter & Schmidhuber, 1997; Gers et al., 2000), a MoE layer, a second LSTM layer, and a softmax layer. The dimensionality of the embedding layer, the number of units in each LSTM layer, and the input and output dimensionality of the MoE layer are all equal to 512. For every layer other than the softmax, we apply drouput (Zaremba et al., 2014) to the layer output, dropping each activation with probability DropP rob, otherwise dividing by (1 − DropP rob). After dropout, the output of the previous layer is added to the layer output.\n\nThis residual connection encourages gradient flow (He et al., 2015).\n\nMoE Layer Architecture: Each expert in the MoE layer is a feed forward network with one ReLU-activated hidden layer of size 1024 and an output layer of size 512. Thus, each expert contains [512 ∗ 1024] + [1024 ∗ 512] = 1M parameters. The output of the MoE layer is passed through a sigmoid function before dropout. We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts.\n\nWe call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers. Thus, each example is processed by exactly 4 experts for a total of 4M ops/timestep. The two LSTM layers contribute 2M ops/timestep each for the desired total of 8M.\n\n3 We have not found the need for deeper hierarchies.\n\nComputationally-Matched Baselines: The MoE-4 model does not employ sparsity, since all 4 experts are always used. In addition, we trained four more computationally-matched baseline models with no sparsity:\n\nMoE-1-Wide: The MoE layer consists of a single \"expert\" containing one ReLU-activated hidden layer of size 4096.\n\nMoE-1-Deep: The MoE layer consists of a single \"expert\" containing four ReLU-activated hidden layers, each with size 1024.\n\n4xLSTM-512: We replace the MoE layer with two additional 512-unit LSTM layers.\n\nLSTM-2048-512: The model contains one 2048-unit LSTM layer (and no MoE). The output of the LSTM is projected down to 512 dimensions (Sak et al., 2014). The next timestep of the LSTM receives the projected output. This is identical to one of the models published in (Jozefowicz et al., 2016). We re-ran it to account for differences in training regimen, and obtained results very similar to the published ones. Training: The models were trained on a cluster of 16 K40 GPUs using the synchronous method described in Section 3. Each batch consisted of a set of sentences totaling roughly 300,000 words. In the interest of time, we limited training to 10 epochs, (27,000 steps). Training took 12-16 hours for all models, except for MoE-4, which took 18 hours (since all the expert computation was performed on only 4 of 16 GPUs). We used the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 1000 training steps, and decreased after that so as to be proportional to the inverse square root of the step number. The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016). For each model, we performed a hyper-parmeter search to find the best dropout probability, in increments of 0.1.\n\nTo ensure balanced expert utilization we set wimportance = 0.1 and wload = 0.1, as described in Section 4 and Appendix A.\n\nResults: We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol. Results are reported in Table 7. For each model, we report the test perplexity, the computational budget, the parameter counts, the value of DropP rob, and the computational efficiency.\n\nModel Test Test ops/timestep #Params excluding Total Drop- TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params P rob per GPU 10 epochs (final) (millions) (billions) (observed) Kneser-Ney 5-gram* 67.6 0.00001 1.8 LSTM-512-512* 54.1 2.4 2.4 0.8 0.1 LSTM-1024-512* 48.2 4.7 4.7 0.8 0.1 LSTM-2048-512* 45.0 43.7 9.4 9.4 0.8 0.1 0.61 LSTM-2048-512 44.7 9.4 9.4 0.8 0.1 1.21 4xLSTM-512 46.0 8.4 8.4 0.8 0.1 1.07 MoE-1-Wide 46.1 8.4 8.4 0.8 0.1 1.29 MoE-1-Deep 45.7 8.4 8.4 0.8 0.1 1.29 MoE-4 45.0 8.4 8.4 0.8 0.1 0.52 MoE-32 39.7 8.4 37.8 0.9 0.1 0.87 MoE-256 35.7 8.6 272.9 1.1 0.1 0.81 MoE-256-h 36.0 8.4 272.9 1.1 0.1 0.89 MoE-1024-h 34.6 8.5 1079.0 1.9 0.2 0.90 MoE-4096-h 34.1 8.9 4303.4 5.1 0.2 0.74 2xLSTM-8192-1024* 34.7 30.6 151.0 151.0 1.8 0.25 1.09 MoE-34M 31.3 33.8 4313.9 6.0 0.3 1.22 MoE-143M 28.0 142.7 4371.1 6.0 0.4 1.56\n\nC.2 More Expensive Models\n\nWe ran two additional models (MoE-34M and MoE-143M) to investigate the effects of adding more computation in the presence of a large MoE layer. These models have computation budgets of 34M and 143M ops/timestep. Similar to the models above, these models use a MoE layer between two LSTM layers. The dimensionality of the embedding layer, and the input and output dimensionality of the MoE layer are set to 1024 instead of 512. For MoE-34M, the LSTM layers have 1024 units.\n\nFor MoE-143M, the LSTM layers have 4096 units and an output projection of size 1024 (Sak et al., 2014). MoE-34M uses a hierarchical MoE layer with 1024 experts, each with a hidden layer of size 2048. MoE-143M uses a hierarchical MoE layer with 256 experts, each with a hidden layer of size 8192. Both models have 4B parameters in the MoE layers. We searched for the best DropP rob for each model, and trained each model for 10 epochs.\n\nThe two models achieved test perplexity of 31.3 and 28.0 respectively, showing that even in the presence of a large MoE, more computation is still useful. Results are reported at the bottom of Table 7. The larger of the two models has a similar computational budget to the best published model from the literature, and training times are similar. Comparing after 10 epochs, our model has a lower test perplexity by 18%.\n\nD 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS Model Architecture: The models are similar in structure to the 8-million-operations-per-timestep models described in the previous section. We vary the number of experts between models, using an ordinary MoE layer with 32 experts and hierarchical MoE layers with 256, 1024, 4096, 16384, 65536 and 131072 experts. For the hierarchical MoE layers, the first level branching factors are 32, 32, 64, 128, 256 and 256, respectively.\n\nTraining: Models are trained on a cluster of 32 Tesla K40 GPUs, except for the last two models, which are trained on clusters of 64 and 128 GPUs so as to have enough memory for all the parameters. For all models, training batch sizes are approximately 2.5 million words. Models are trained once-through over about 100 billion words. We implement several memory optimizations in order to fit up to 1 billion parameters per GPU. First, we do not store the activations of the hidden layers of the experts, but instead recompute them on the backwards pass. Secondly, we modify the optimizer on the expert parameters to require less auxiliary storage: The Adam optimizer (Kingma & Ba, 2015) keeps first and second moment estimates of the perparameter gradients. This triples the required memory. To avoid keeping a first-moment estimator, we set β1 = 0. To reduce the size of the second moment estimator, we replace it with a factored approximation. For a matrix of parameters, instead of maintaining a full matrix of second-moment estimators, we maintain vectors of row-wise and column-wise averages of that matrix. At each step, the matrix of estimators is taken to be the outer product of those two vectors divided by the mean of either one. This technique could similarly be applied to Adagrad (Duchi et al., 2010).\n\nModel Test Test ops/timestep #Params excluding Total TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params per GPU .1 epochs 1 epoch (millions) (billions) (observed) Kneser-Ney 5-gram 67.1 45.3 0.00001 76.0 4xLSTM-512 54.5 47.0 8.4 8.4 0.1 1.23 MoE-32 48.5 40.4 8.4 37.8 0.1 0.83 MoE-256-h 42.8 35.3 8.4 272.9 0.4 1.11 MoE-1024-h 40.3 32.7 8.5 1079.0 1.2 1.14 MoE-4096-h 38.9 30.9 8.6 4303.4 4.4 1.07 MoE-16384-h 38.2 29.7 8.8 17201.0 17.3 0.96 MoE-65536-h 38.2 28.9 9.2 68791.0 68.9 0.72 MoE-131072-h 39.8 29.2 9.7 137577.6 137.7 0.30\n\nResults: We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8. Perplexity after 100 billion training words is 39% lower for the 68-billion-parameter MoE model than for the baseline model. It is notable that the measured computational efficiency of the largest model (0.30 TFLOPS/GPU) is very low compared to the other models. This is likely a result of the fact that, for purposes of comparison to the other models, we did not increase the training batch size proportionally to the number of GPUs. For comparison, we include results for a computationally matched baseline model consisting of 4 LSTMs, and for an unpruned 5-gram model with Kneser-Ney smoothing (Kneser & Ney, 1995).4\n\nE Machine Translation - Experimental Details\n\nModel Architecture for Single Language Pair MoE Models: Our model is a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decrease the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We insert MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). We use an attention mechanism between the encoder and decoder, with the first decoder LSTM receiving output from and providing input for the attention 5. All of the layers in our model have input and output dimensionality of 512. Our LSTM layers have 2048 hidden units, with a 512-dimensional output projection. We add residual connections around all LSTM and MoE layers to encourage gradient flow (He et al., 2015). Similar to GNMT, to effectively deal with rare words, we used subword units (also known as \"wordpieces\") (Schuster & Nakajima, 2012) for inputs and outputs in our system.\n\nWe use a shared source and target vocabulary of 32K wordpieces. We also used the same beam search technique as proposed in (Wu et al., 2016).\n\nWe train models with different numbers of experts in the MoE layers. In addition to a baseline model with no MoE layers, we train models with flat MoE layers containing 32 experts, and models with hierarchical MoE layers containing 512 and 2048 experts. The flat MoE layers use k = 4 and the hierarchical MoE models use k = 2 at each level of the gating network. Thus, each input is processed by exactly 4 experts in each MoE layer. Each expert in the MoE layer is a feed forward network with one hidden layer of size 2048 and ReLU activation. Thus, each expert contains [512 ∗ 2048] + [2048 ∗ 512] = 2M parameters. The output of the MoE layer is passed through a sigmoid function. We use the strictly-balanced gating function described in Appendix F.\n\nModel Architecture for Multilingual MoE Model: We used the same model architecture as for the single-language-pair models, with the following exceptions: We used noisy-top-k gating as described in Section 2.1, not the scheme from Appendix F. The MoE layers in the encoder and decoder are non-hierarchical MoEs with n = 512 experts, and k = 2. Each expert has a larger hidden layer of size 8192. This doubles the amount of computation in the MoE layers, raising the computational budget of the entire model from 85M to 102M ops/timestep. Training: We trained our networks using the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 2000 training steps, held constant for an additional 8000 steps, and decreased after that so as to be proportional to the inverse square root of the step number. For the single-language-pair models, similarly to (Wu et al., 2016), we applied dropout (Zaremba et al., 2014) to the output of all embedding, LSTM and MoE layers, using DropP rob = 0.4. Training was done synchronously on a cluster of up to 64 GPUs as described in section 3. Each training batch consisted of a set of sentence pairs containing roughly 16000 words per GPU.\n\nTo ensure balanced expert utilization we set wimportance = 0.01 and wload = 0.01, as described in Section 4 and Appendix A. Metrics: We evaluated our models using the perplexity and the standard BLEU score metric. We reported tokenized BLEU score as computed by the multi-bleu.pl script, downloaded from the public implementation of Moses (on Github), which was also used in (Luong et al., 2015a). Results: Tables 2, 3 and 4 in Section 5.3 show comparisons of our results to other published methods. Figure 4 shows test perplexity as a function of number of words in the (training data's) source sentences processed for models with different numbers of experts. As can be seen from the Figure, as we increased the number of experts to approach 2048, the test perplexity of our model continued to improve.\n\nWe found that the experts indeed become highly specialized by syntax and/or semantics, as can be seen in Table 9. For example, one expert is used when the indefinite article \"a\" introduces the direct object in a verb phrase indicating importance or leadership.\n\nExpert 381 Expert 752 Expert 2004 ... with researchers , ... ... plays a core ... ... with rapidly growing ... ... to innovation . ... plays a critical ... ... under static conditions ... ... tics researchers . ... provides a legislative ... ... to swift ly ... ... the generation of ... ... play a leading ... ... to dras tically ... ... technology innovations is ... ... assume a leadership ... ... the rapid and ... ... technological innovations , ... ... plays a central ... ... the fast est ... ... support innovation throughout ... ... taken a leading ... ... the Quick Method ... ... role innovation will ... ... established a reconciliation ... ... rec urrent ) ... ... research scienti st ... ... played a vital ... ... provides quick access ... ... promoting innovation where ... ... have a central ... ... of volatile organic ... ... ... ...\n\nDue to some peculiarities in our infrastructure which have since been fixed, at the time we ran some of the machine translation experiments, our models ran faster if every expert received exactly the same batch size. To accommodate this, we used a different gating function which we describe below. Recall that we define the softmax gating function to be:\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ Gσ(x) = Sof tmax(x · Wg) (15) Sparse Gating (alternate formulation): To obtain a sparse gating vector, we multiply Gσ(x) component-wise with a sparse mask M(Gσ(x)) and normalize the output. The mask itself is a function of Gσ(x) and specifies which experts are assigned to each input example:\n\n$$(15)$$ $$G(x){i}=\\frac{G{\\sigma}(x){i}M(G{\\sigma}(x)){i}}{\\sum{j=1}^{n}G_{\\sigma}(x){j}M(G{\\sigma}(x))_{j}}\\tag{1}$$ $$(17)$$\n\n$$(16)$$\n\nTop-K Mask: To implement top-k gating in this formulation, we would let M(v) = T opK(v, k), where:\n\n$$T o p K(v,k){i}=\\begin{cases}1&{\\mathrm{if~}}v{i}{\\mathrm{~is~in~the~top~}}k{\\mathrm{~elements~of~}}v.\\ 0&{\\mathrm{otherwise.}}\\end{cases}$$ 0 otherwise. (17) Batchwise Mask: To force each expert to receive the exact same number of examples, we introduce an alternative mask function, Mbatchwise(X, m), which operates over batches of input vectors.\n\nInstead of keeping the top k values per example, we keep the top m values per expert across the training batch, where m = k|X| n, so that each example is sent to an average of k experts.\n\n$$M_{b a t c h w i s e}(X,m){j,i}=\\begin{cases}1&\\text{if}X{j,i}\\text{is in the top}m\\text{values for to expert}i\\ 0&\\text{otherwise}\\end{cases}$$\n\nAs our experiments suggest and also observed in (Ioffe & Szegedy, 2015), using a batchwise function during training (such as Mbatchwise) requires modifications to the inference when we may not have a large batch of examples. Our solution to this is to train a vector T of per-expert threshold values to approximate the effects of the batchwise mask. We use the following mask at inference time:\n\n$$(18)$$ $$M_{threshold}(x,T){i}=\\begin{cases}1&\\text{if}x{i}>T_{i}\\ 0&\\text{otherwise}\\end{cases}\\tag{1}$$ $$(19)$$\n\nTo learn the threshold values, we apply an additional loss at training time which is minimized when the batchwise mask and the threshold mask are identical.\n\n$$L_{batchwise}(X,T,m)=\\sum_{j=1}^{|X|}\\sum_{i=1}^{n}(M_{threshold}(x,T){i}-M{batchwise}(X,m){j,i})(X{j,i}-T_{i})\\tag{20}$$\n\nG ATTENTION FUNCTION The attention mechanism described in GNMT (Wu et al., 2016) involves a learned \"Attention Function\" A(xi, yj ) which takes a \"source vector\" xi and a \"target vector\" yj , and must be computed for every source time step i and target time step j. In GNMT, the attention function is implemented as a feed forward neural network with a hidden layer of size n. It can be expressed as:\n\n$$A_{G N M T}(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}t a n h((x_{i}U){d}+(y{j}W)_{d})$$\n\nWhere U and W are trainable weight matrices and V is a trainable weight vector.\n\nFor performance reasons, in our models, we used a slightly different attention function:\n\n$$A(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}tanh((x_{i}U){d})tanh((y{j}W)_{d})\\tag{22}$$\n\nWith our attention function, we can simultaneously compute the attention function on multiple source time steps and multiple target time steps using optimized matrix multiplications. We found little difference in quality between the two functions.\n\n$$(21)$$",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/1701.06538v1.md"
+          },
+          "headlines": {
+            "1 Introduction And Related Work": [
+              "1.1 Conditional Computation",
+              "1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER",
+              "1.3 Related Work On Mixtures Of Experts"
+            ],
+            "2 The Structure Of The Mixture-Of-Experts Layer": [
+              "2.1 Gating Network"
+            ],
+            "3 Addressing Performance Challenges": [
+              "3.1 The Shrinking Batch Problem",
+              "3.2 Network Bandwidth"
+            ],
+            "4 Balancing Expert Utilization": [],
+            "5 Experiments": [
+              "5.1 1 Billion Word Language Modeling Benchmark",
+              "5.2 100 BILLION WORD GOOGLE NEWS CORPUS",
+              "5.3 Machine Translation (Single Language Pair)",
+              "5.4 Multilingual Machine Translation"
+            ],
+            "6 Conclusion": [],
+            "Appendices": [
+              "A Load-Balancing Loss",
+              "B Hierachical Mixture Of Experts",
+              "C 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS",
+              "D 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS",
+              "E Machine Translation - Experimental Details"
+            ]
+          },
+          "summary": "The paper introduces a Sparsely-Gated Mixture-of-Experts (MoE) layer to enhance neural network capacity without a proportional increase in computation. This approach uses a trainable gating network to select a sparse combination of experts for each input, significantly improving model capacity and performance in language modeling and machine translation tasks. The MoE layer, with up to 137 billion parameters, achieves better results than state-of-the-art models at lower computational costs. The paper addresses challenges like batch size and network bandwidth, proposing solutions like mixing data and model parallelism. Experiments show that MoE models outperform traditional models in various benchmarks, demonstrating the potential of conditional computation in deep learning.",
+          "summary_embedding": [
+            -0.01644345559179783,
+            -0.016995344310998917,
+            -0.016372699290513992,
+            -0.014349107630550861,
+            0.02282555401325226,
+            0.045934125781059265,
+            -0.014646278694272041,
+            0.03563220426440239,
+            -0.004829026758670807,
+            0.005617944523692131,
+            -0.00932550523430109,
+            -0.008143897168338299,
+            0.005575491581112146,
+            0.01538213063031435,
+            -0.005653322208672762,
+            -0.020306676626205444,
+            -0.013832597061991692,
+            -0.023235931992530823,
+            0.028344441205263138,
+            -0.020080260932445526,
+            0.011823155917227268,
+            -0.020943470299243927,
+            -0.025868017226457596,
+            -0.016500059515237808,
+            0.025061409920454025,
+            0.024806693196296692,
+            0.02583971433341503,
+            0.003827843815088272,
+            -0.03492465242743492,
+            0.033651065081357956,
+            0.014674580655992031,
+            0.012488252483308315,
+            -0.031500112265348434,
+            -0.009162768721580505,
+            -0.05351905897259712,
+            -0.05227377265691757,
+            0.04143410921096802,
+            0.022245362401008606,
+            -0.016231190413236618,
+            0.023717066273093224,
+            0.017264213413000107,
+            0.005816058721393347,
+            -0.020632149651646614,
+            -0.020844414830207825,
+            0.003700484987348318,
+            -0.013103820383548737,
+            -0.01983969286084175,
+            -0.013528349809348583,
+            -0.01828308403491974,
+            -0.01577835902571678,
+            0.009551920928061008,
+            0.0002856731880456209,
+            0.0029186424799263477,
+            -0.003424540627747774,
+            -0.047830358147621155,
+            0.0248208437114954,
+            0.019853845238685608,
+            0.03492465242743492,
+            0.04633035510778427,
+            0.013011839240789413,
+            0.01516986545175314,
+            -0.014271277002990246,
+            -0.04021712392568588,
+            0.005497660953551531,
+            -0.00041656987741589546,
+            -0.005380915477871895,
+            -0.005819596350193024,
+            0.008575502783060074,
+            0.008724087849259377,
+            -0.01143400277942419,
+            -0.04202845320105553,
+            0.013952880166471004,
+            -0.020561393350362778,
+            0.00010143388499272987,
+            0.01354957651346922,
+            0.008292482234537601,
+            0.02677368000149727,
+            0.03195294365286827,
+            0.023971782997250557,
+            -0.008724087849259377,
+            -0.01340806670486927,
+            0.019924599677324295,
+            -0.0022676968947052956,
+            0.007018893025815487,
+            0.04570771008729935,
+            -0.04033033177256584,
+            0.02163686975836754,
+            -0.025726506486535072,
+            -0.015339677222073078,
+            -0.005349075887352228,
+            -0.022471778094768524,
+            0.02806142158806324,
+            -0.007344366051256657,
+            0.017801949754357338,
+            0.017094401642680168,
+            0.01594817079603672,
+            0.013330236077308655,
+            -0.001894464367069304,
+            -0.022938761860132217,
+            0.0018360915128141642,
+            -0.01541043259203434,
+            0.02114158496260643,
+            0.0015230007702484727,
+            0.005469358991831541,
+            0.027594437822699547,
+            0.031160488724708557,
+            -0.023490650579333305,
+            0.009502392262220383,
+            0.02422650158405304,
+            0.06967949122190475,
+            0.015424583107233047,
+            -0.006435164716094732,
+            -0.0024640418123453856,
+            0.04143410921096802,
+            0.0015513027319684625,
+            0.0038349193055182695,
+            -0.06826439499855042,
+            -0.03022652305662632,
+            -0.0011365016689524055,
+            -0.02254253253340721,
+            0.02227366529405117,
+            0.01002597901970148,
+            -0.009750034660100937,
+            -0.022457627579569817,
+            -0.04516997188329697,
+            0.05190584436058998,
+            -0.00545874610543251,
+            -0.016259491443634033,
+            -0.005554265342652798,
+            -0.05340585112571716,
+            0.00039136342820711434,
+            0.015042506158351898,
+            -0.018481198698282242,
+            -0.03670767694711685,
+            0.036141637712717056,
+            0.0007398316520266235,
+            0.03741522878408432,
+            0.012459950521588326,
+            -0.01708024926483631,
+            -0.020094411447644234,
+            0.022259512916207314,
+            -0.011639192700386047,
+            -0.0038313816767185926,
+            -0.027339721098542213,
+            0.0065236082300543785,
+            -0.028103873133659363,
+            -0.0014566679019480944,
+            0.06730213016271591,
+            -0.04816998168826103,
+            0.02946236915886402,
+            -0.029660483822226524,
+            0.023660462349653244,
+            -0.008575502783060074,
+            0.0278916098177433,
+            -0.010238244198262691,
+            0.005840822588652372,
+            -0.013457595370709896,
+            -0.0010586712742224336,
+            -0.020745357498526573,
+            0.0034210027661174536,
+            -0.016259491443634033,
+            -0.027750099077820778,
+            -0.007054270710796118,
+            0.031273696571588516,
+            0.01698119193315506,
+            0.025655752047896385,
+            0.025429336354136467,
+            0.01246702577918768,
+            -0.04935866594314575,
+            0.02595292218029499,
+            0.0018926954362541437,
+            0.013514199294149876,
+            0.013181651011109352,
+            -0.0010418669553473592,
+            0.025599148124456406,
+            0.012049571610987186,
+            0.025061409920454025,
+            0.037443529814481735,
+            0.0027682881336659193,
+            0.04737752676010132,
+            -0.015693452209234238,
+            -0.005292471963912249,
+            0.028528403490781784,
+            0.000505455827806145,
+            0.02465103194117546,
+            -0.001686621573753655,
+            0.013330236077308655,
+            0.000776978034991771,
+            0.03834919258952141,
+            -0.006339645478874445,
+            0.029518973082304,
+            0.021608568727970123,
+            0.028316138312220573,
+            -0.014759486541152,
+            0.023122724145650864,
+            -0.019754787907004356,
+            -0.02292460948228836,
+            0.035547297447919846,
+            0.004082561936229467,
+            0.01886327564716339,
+            -0.028457649052143097,
+            -0.030679354444146156,
+            0.015452885068953037,
+            -0.01855195313692093,
+            0.016556663438677788,
+            0.03639635816216469,
+            0.0008273909334093332,
+            -0.0025719432160258293,
+            0.026292547583580017,
+            -0.04245298355817795,
+            0.012856177985668182,
+            -0.01532552670687437,
+            -0.02855670638382435,
+            -0.019528372213244438,
+            -0.00875946506857872,
+            -0.0018095583654940128,
+            -0.013691086322069168,
+            -0.0037570889107882977,
+            -0.007563706487417221,
+            -0.02459442801773548,
+            0.00935380719602108,
+            0.013627407141029835,
+            0.009849091991782188,
+            -0.018665160983800888,
+            0.0019952901639044285,
+            -0.008257105015218258,
+            -0.0160047747194767,
+            -0.03752843663096428,
+            0.015268922783434391,
+            -0.0024888061452656984,
+            0.009191070683300495,
+            0.04078316316008568,
+            0.028485950082540512,
+            -0.005055442452430725,
+            0.0329718142747879,
+            -0.033707667142152786,
+            0.017915157601237297,
+            -0.008320784196257591,
+            0.009191070683300495,
+            0.002989397384226322,
+            -0.025542544201016426,
+            0.027424626052379608,
+            -0.03792466223239899,
+            -0.038151077926158905,
+            0.022995365783572197,
+            0.011858533136546612,
+            0.0017096169758588076,
+            -0.019386861473321915,
+            -0.017717044800519943,
+            0.0008499440737068653,
+            0.002752368338406086,
+            -0.018834972754120827,
+            -0.01019579079002142,
+            0.02227366529405117,
+            -0.025500090792775154,
+            0.036085035651922226,
+            0.023391593247652054,
+            -0.03520767390727997,
+            -0.00608492735773325,
+            -0.01903308741748333,
+            0.059037946164608,
+            -0.0011506526498124003,
+            -0.010301923379302025,
+            0.022938761860132217,
+            -0.027268964797258377,
+            -0.004524780437350273,
+            0.022301966324448586,
+            0.01828308403491974,
+            -0.008278331719338894,
+            0.003983504604548216,
+            -0.022287815809249878,
+            -0.05281151086091995,
+            -0.010577867738902569,
+            -0.0003305583668407053,
+            0.054368119686841965,
+            -0.012367968447506428,
+            0.0020695829298347235,
+            -0.006413938011974096,
+            0.004100250545889139,
+            0.03704730048775673,
+            0.002140337834134698,
+            0.0025082635693252087,
+            -0.013698162510991096,
+            0.0371888130903244,
+            0.022316116839647293,
+            0.005883275531232357,
+            -0.012162779457867146,
+            0.005377377849072218,
+            -0.04899074137210846,
+            -0.0009118546731770039,
+            0.005741765722632408,
+            0.007386818993836641,
+            -0.025910470634698868,
+            0.017717044800519943,
+            0.009205221198499203,
+            -0.0024693484883755445,
+            -0.009792488068342209,
+            0.01173117384314537,
+            0.025429336354136467,
+            -0.019938750192523003,
+            -0.05700020492076874,
+            0.0034705311991274357,
+            0.012014193460345268,
+            -0.004054259974509478,
+            -0.019401011988520622,
+            0.006792477332055569,
+            0.0021934041287750006,
+            0.022698193788528442,
+            -0.011271266266703606,
+            -0.0029221803415566683,
+            0.040585052222013474,
+            0.009516543708741665,
+            -0.0016813150141388178,
+            -0.017717044800519943,
+            -0.011794853024184704,
+            0.018070818856358528,
+            0.004740582779049873,
+            0.0035289041697978973,
+            -0.01751893013715744,
+            -0.029264256358146667,
+            0.02800481766462326,
+            0.019967053085565567,
+            -0.048905834555625916,
+            -0.008200501091778278,
+            -0.019273653626441956,
+            0.0019599127117544413,
+            0.008695785887539387,
+            0.04613224044442177,
+            -0.00590803986415267,
+            0.008313708938658237,
+            -0.020179318264126778,
+            -0.05003791302442551,
+            -0.02708500251173973,
+            0.03707560524344444,
+            -0.028684064745903015,
+            -0.04276430234313011,
+            -0.010252394713461399,
+            0.016740625724196434,
+            -0.009403335861861706,
+            -0.018679311498999596,
+            -0.011738249100744724,
+            -0.04969828948378563,
+            -0.003497064346447587,
+            0.0337359681725502,
+            0.017264213413000107,
+            -0.008441068232059479,
+            -0.033877480775117874,
+            0.019641580060124397,
+            -0.007160402834415436,
+            -0.006474079564213753,
+            0.0007862646016292274,
+            0.008794843219220638,
+            0.02400008589029312,
+            0.011879759840667248,
+            -0.02763689123094082,
+            -0.03427370637655258,
+            -0.0024888061452656984,
+            0.014929298311471939,
+            -0.00878776703029871,
+            0.0245378240942955,
+            0.024679332971572876,
+            -0.022995365783572197,
+            -0.01654251106083393,
+            0.015750056132674217,
+            -0.05779265984892845,
+            -0.010287772864103317,
+            -0.029235953465104103,
+            -0.006183984223753214,
+            0.0009613831643946469,
+            0.0061096916906535625,
+            0.00012304731353651732,
+            0.03846240043640137,
+            0.008801918476819992,
+            0.006014172453433275,
+            0.009191070683300495,
+            0.01913214474916458,
+            0.028471799567341805,
+            0.009155693463981152,
+            -0.03517937287688255,
+            0.02850010246038437,
+            -0.017051948234438896,
+            0.007811348885297775,
+            -0.025754809379577637,
+            -0.013033065013587475,
+            0.00443633645772934,
+            0.02725481428205967,
+            -0.000503244751598686,
+            0.018849123269319534,
+            -0.021919889375567436,
+            0.037500135600566864,
+            -0.029915200546383858,
+            -0.05278320610523224,
+            -0.02005195803940296,
+            -0.008285406976938248,
+            0.015764206647872925,
+            0.017278363928198814,
+            -0.019924599677324295,
+            -0.0286557637155056,
+            -0.021750077605247498,
+            0.006470541935414076,
+            0.010542490519583225,
+            -0.021099131554365158,
+            -0.010627396404743195,
+            0.01502835564315319,
+            -0.00404364662244916,
+            -0.01886327564716339,
+            0.013684011064469814,
+            0.015693452209234238,
+            0.03169822692871094,
+            -0.03308502212166786,
+            0.004287751391530037,
+            -0.025967074558138847,
+            -0.04327373951673508,
+            0.004096712917089462,
+            0.03166992589831352,
+            0.027509532868862152,
+            -0.018636859953403473,
+            0.010754755698144436,
+            0.003990580327808857,
+            0.004234685096889734,
+            -0.0033803186379373074,
+            0.023603858426213264,
+            0.03291521221399307,
+            0.02763689123094082,
+            -0.011653343215584755,
+            0.004921008367091417,
+            -0.01559439580887556,
+            -0.0011108530452474952,
+            0.01865101046860218,
+            -0.017816102132201195,
+            0.033877480775117874,
+            -0.029405765235424042,
+            0.02104252763092518,
+            -0.035122767090797424,
+            0.006141531281173229,
+            -0.012099100276827812,
+            -0.00404364662244916,
+            0.04833979532122612,
+            -0.017646290361881256,
+            -0.0019546060357242823,
+            -0.008101443760097027,
+            -0.014575524255633354,
+            -0.02158026583492756,
+            0.008766540326178074,
+            -0.04033033177256584,
+            -0.012367968447506428,
+            -0.028429346159100533,
+            0.003424540627747774,
+            0.019910449162125587,
+            -0.06277380883693695,
+            -0.06237758323550224,
+            0.012275987304747105,
+            -0.04918885603547096,
+            -0.026419905945658684,
+            0.02963218092918396,
+            -0.002984090941026807,
+            0.0012780115939676762,
+            -0.0024003623984754086,
+            0.0068101659417152405,
+            -0.029207652434706688,
+            0.00425944896414876,
+            -0.000887974863871932,
+            0.03271709755063057,
+            -0.021056678146123886,
+            -0.008122670464217663,
+            0.004878554958850145,
+            -0.06860402226448059,
+            -0.028344441205263138,
+            0.011271266266703606,
+            -0.009459939785301685,
+            -0.0066615804098546505,
+            -0.017915157601237297,
+            -0.004383270163089037,
+            0.0008950503543019295,
+            0.0032830305863171816,
+            0.05801907554268837,
+            -0.01983969286084175,
+            -0.053773775696754456,
+            0.03633975237607956,
+            0.00887267291545868,
+            0.029603879898786545,
+            -0.025556694716215134,
+            -0.017533080652356148,
+            -0.01855195313692093,
+            -0.020504789426922798,
+            -0.04123599827289581,
+            0.00404364662244916,
+            -0.005918653216212988,
+            0.009212297387421131,
+            -0.0009693430620245636,
+            0.02763689123094082,
+            0.029235953465104103,
+            0.023108573630452156,
+            0.012750045396387577,
+            0.01079013291746378,
+            -0.022372720763087273,
+            -0.021594416350126266,
+            0.0012222920777276158,
+            0.04683978855609894,
+            0.024636881425976753,
+            -0.02675952948629856,
+            -0.011504758149385452,
+            0.03953787684440613,
+            0.0419718474149704,
+            -0.01974063739180565,
+            -0.02276895008981228,
+            0.00764861237257719,
+            0.03266049548983574,
+            -0.04477374628186226,
+            0.005932804197072983,
+            0.03410389646887779,
+            0.014405711553990841,
+            -0.005356151144951582,
+            -0.010294848121702671,
+            0.022570835426449776,
+            0.03152841329574585,
+            -0.008724087849259377,
+            0.01876421831548214,
+            0.016910437494516373,
+            0.029349161311984062,
+            -0.0067889392375946045,
+            0.0005766529939137399,
+            -0.024735936895012856,
+            0.055188875645399094,
+            -0.03059444949030876,
+            0.02909444272518158,
+            0.010316074825823307,
+            -0.00593634182587266,
+            0.039056744426488876,
+            -0.016641568392515182,
+            -0.0431322306394577,
+            0.01913214474916458,
+            -0.006979977712035179,
+            -0.0055613406002521515,
+            -0.03444352000951767,
+            -0.007592008449137211,
+            -0.007507102098315954,
+            -0.006714646704494953,
+            -0.022556684911251068,
+            -0.02606613002717495,
+            0.004415110219269991,
+            0.00973588414490223,
+            -0.006484692916274071,
+            0.00698705343529582,
+            0.02092931978404522,
+            -0.013507124036550522,
+            0.02525952458381653,
+            -0.046047333627939224,
+            -0.0008366775000467896,
+            -0.0038596836384385824,
+            -0.025712355971336365,
+            -0.0345001220703125,
+            -0.0005770951975136995,
+            -0.0024251264985650778,
+            -0.024622729048132896,
+            -0.00914154201745987,
+            -0.008752389810979366,
+            -0.0023119186516851187,
+            -0.004938696976751089,
+            -0.055188875645399094,
+            0.008695785887539387,
+            0.010846736840903759,
+            -0.002713453024625778,
+            0.0055507272481918335,
+            -0.010613245889544487,
+            -0.03039633482694626,
+            0.03039633482694626,
+            0.04129260033369064,
+            0.033707667142152786,
+            0.014334957115352154,
+            -0.0074646491557359695,
+            -0.017165156081318855,
+            0.006060163024812937,
+            0.021721776574850082,
+            0.00902833417057991,
+            0.007846726104617119,
+            0.03877372294664383,
+            0.00011331850691931322,
+            -0.008349086157977581,
+            -0.009764186106622219,
+            0.028740668669342995,
+            -0.001085204305127263,
+            0.023603858426213264,
+            0.050802066922187805,
+            0.030424637719988823,
+            0.020589696243405342,
+            -0.01681138016283512,
+            -0.02010856196284294,
+            -0.00851889792829752,
+            -0.02855670638382435,
+            0.0364246591925621,
+            0.012127402238547802,
+            -0.005232329946011305,
+            -0.011540135368704796,
+            0.019061388447880745,
+            -0.033339742571115494,
+            -0.0037358624394983053,
+            0.00048422932741232216,
+            0.013924578204751015,
+            -0.03379257395863533,
+            0.0002480846014805138,
+            0.031613320112228394,
+            0.014957600273191929,
+            -0.016995344310998917,
+            -0.0188774261623621,
+            0.010549565777182579,
+            0.020462337881326675,
+            0.018948180601000786,
+            0.00037743354914709926,
+            0.00897173024713993,
+            -0.00022121983056422323,
+            -0.0005319889169186354,
+            0.01590571738779545,
+            -0.007952858693897724,
+            0.036254845559597015,
+            -0.003525366308167577,
+            -0.01325240544974804,
+            -0.0037500134203583,
+            -0.021438755095005035,
+            -0.024467067793011665,
+            0.043556760996580124,
+            0.023971782997250557,
+            0.04661337286233902,
+            -0.005660397466272116,
+            -0.004245297983288765,
+            0.025825563818216324,
+            0.008221727795898914,
+            -0.03288691118359566,
+            0.000265110022155568,
+            0.005281858611851931,
+            -0.02892463095486164,
+            0.016641568392515182,
+            -0.02476423978805542,
+            -0.016896286979317665,
+            -0.006771250627934933,
+            -0.0015557248843833804,
+            -0.028188779950141907,
+            0.01703779585659504,
+            0.0031309074256569147,
+            -0.0027099153958261013,
+            -0.039764292538166046,
+            0.006880920846015215,
+            0.020419884473085403,
+            0.015311375260353088,
+            0.0037358624394983053,
+            0.011525984853506088,
+            -0.016613267362117767,
+            0.002126186853274703,
+            0.03379257395863533,
+            0.004878554958850145,
+            0.02005195803940296,
+            0.0027647505048662424,
+            0.023773670196533203,
+            0.03633975237607956,
+            -0.01681138016283512,
+            -0.02189158834517002,
+            0.01519816741347313,
+            -0.02233026921749115,
+            0.029943503439426422,
+            0.004889168310910463,
+            -0.012283062562346458,
+            0.022033097222447395,
+            -0.009827865287661552,
+            -0.0027346794959157705,
+            0.01659911684691906,
+            -0.022740647196769714,
+            -0.017717044800519943,
+            0.03183973580598831,
+            -0.039452970027923584,
+            0.012856177985668182,
+            -0.008547200821340084,
+            0.015863263979554176,
+            -0.023066120222210884,
+            0.011172209866344929,
+            0.007100261282175779,
+            -0.01573590561747551,
+            0.010797208175063133,
+            0.009693430736660957,
+            0.00545874610543251,
+            0.0018926954362541437,
+            0.02179253101348877,
+            -0.0006049550138413906,
+            -0.011257115751504898,
+            -0.018155725672841072,
+            -0.004694592207670212,
+            -0.002614396158605814,
+            -0.016556663438677788,
+            0.01865101046860218,
+            -0.041066184639930725,
+            -0.013280707411468029,
+            -0.008879749104380608,
+            -0.013988257385790348,
+            -0.010011828504502773,
+            0.024184048175811768,
+            0.025669902563095093,
+            -0.002361447084695101,
+            -0.031217092648148537,
+            0.01790100708603859,
+            0.024198200553655624,
+            0.003654494183138013,
+            -0.023646311834454536,
+            0.0302831269800663,
+            -0.008936353027820587,
+            0.014971751719713211,
+            0.026136886328458786,
+            0.014094389975070953,
+            0.004595535341650248,
+            -0.010500037111341953,
+            -0.02114158496260643,
+            0.01481609046459198,
+            0.010507113300263882,
+            0.033537857234478,
+            0.014745336025953293,
+            0.02071705460548401,
+            0.01594817079603672,
+            -0.023320838809013367,
+            -0.011214662343263626,
+            -0.003482913365587592,
+            -0.022514231503009796,
+            0.03093407303094864,
+            0.0006129149114713073,
+            0.02114158496260643,
+            0.026618018746376038,
+            -0.03319823369383812,
+            -0.01187268365174532,
+            0.032688796520233154,
+            0.02779255248606205,
+            0.05108508840203285,
+            0.025316128507256508,
+            0.015396281145513058,
+            0.027226513251662254,
+            -0.03585861995816231,
+            -0.036368053406476974,
+            0.001548649393953383,
+            0.04256619140505791,
+            -0.008426916785538197,
+            0.00967928022146225,
+            0.03900013864040375,
+            -0.006781863979995251,
+            -0.01611798256635666,
+            0.019457615911960602,
+            0.0026710000820457935,
+            0.002448122017085552,
+            -0.0033113325480371714,
+            -0.01335146278142929,
+            0.03175482898950577,
+            -0.0051474240608513355,
+            0.013570803217589855,
+            -0.019273653626441956,
+            0.027113303542137146,
+            0.0001799829478841275,
+            -0.003229964291676879,
+            0.028344441205263138,
+            -0.002660386962816119,
+            -0.051990751177072525,
+            0.024212351068854332,
+            0.019811391830444336,
+            -0.011561362072825432,
+            -0.0036332677118480206,
+            -0.003792466362938285,
+            0.04567940905690193,
+            0.034698236733675,
+            0.029009537771344185,
+            -0.0020501252729445696,
+            0.0168679840862751,
+            0.018792519345879555,
+            0.0053667644970119,
+            0.014278353191912174,
+            -0.0017025414854288101,
+            -0.007853801362216473,
+            0.021056678146123886,
+            -0.020787810906767845,
+            0.03226426616311073,
+            -0.016570813953876495,
+            -0.020122714340686798,
+            0.008738238364458084,
+            -0.011172209866344929,
+            -0.0009083169279620051,
+            -0.006311343517154455,
+            -0.0008813415770418942,
+            -0.0029664020985364914,
+            0.016938740387558937,
+            -0.004446949809789658,
+            0.04661337286233902,
+            -0.013931653462350368,
+            -0.008808993734419346,
+            0.006972902454435825,
+            -0.025981225073337555,
+            0.0013346156338229775,
+            0.0028726516757160425,
+            0.027000095695257187,
+            -0.01989629678428173,
+            -0.00970758218318224,
+            0.01606137864291668,
+            0.06039644405245781,
+            -0.008639181964099407,
+            -0.0026267783250659704,
+            0.027226513251662254,
+            -0.002787745790556073,
+            -0.03551899641752243,
+            0.012919857166707516,
+            -0.04494355618953705,
+            0.017589686438441277,
+            0.026349151507019997,
+            -0.04986810311675072,
+            -0.022089701145887375,
+            -0.02347649820148945,
+            -0.02043403498828411,
+            0.023434046655893326,
+            -0.009976450353860855,
+            0.013867974281311035,
+            -0.017801949754357338,
+            -0.009014182724058628,
+            0.020518941804766655,
+            0.009176919236779213,
+            0.03735862299799919,
+            -0.027551984414458275,
+            -0.016783079132437706,
+            -0.027820853516459465,
+            -0.04304732382297516,
+            0.009558996185660362,
+            -0.0278916098177433,
+            0.0027099153958261013,
+            -0.0016291331266984344,
+            -0.040302030742168427,
+            -0.037981268018484116,
+            0.005278320983052254,
+            -0.017985913902521133,
+            0.01516986545175314,
+            0.020915169268846512,
+            0.02731141820549965,
+            -0.0020253609400242567,
+            -0.030198222026228905,
+            0.029235953465104103,
+            -0.007344366051256657,
+            0.03223596513271332,
+            -0.00689507182687521,
+            0.005066055804491043,
+            -0.0070436573587358,
+            0.031160488724708557,
+            -0.023009516298770905,
+            0.018792519345879555,
+            -0.026801982894539833,
+            -0.004086099565029144,
+            -0.018990634009242058,
+            -0.01774534583091736,
+            0.004224071744829416,
+            0.004521242808550596,
+            7.888073741924018e-05,
+            0.0048750173300504684,
+            0.005777143407613039,
+            0.024806693196296692,
+            0.0021421066485345364,
+            0.02589631825685501,
+            -0.027877457439899445,
+            -0.025316128507256508,
+            0.01719345711171627,
+            -0.02077365852892399,
+            0.005678086541593075,
+            -0.006261814851313829,
+            -0.0302831269800663,
+            -0.01202834490686655,
+            0.023391593247652054,
+            0.02162271924316883,
+            -0.005490585695952177,
+            0.002644466934725642,
+            0.02206140011548996,
+            -0.026023678481578827,
+            0.00437973253428936,
+            -0.03781145438551903,
+            -0.018396291881799698,
+            0.029773691669106483,
+            -0.03195294365286827,
+            0.020745357498526573,
+            0.003185742534697056,
+            0.03483974561095238,
+            -0.017292514443397522,
+            -0.01947176828980446,
+            0.01654251106083393,
+            -0.01008258294314146,
+            0.004142703488469124,
+            0.01606137864291668,
+            0.01616043597459793,
+            -0.0015070808585733175,
+            0.05380208045244217,
+            0.0008167776977643371,
+            0.006909222807735205,
+            -0.049075648188591,
+            0.004496478475630283,
+            -0.044858649373054504,
+            0.0002768288250081241,
+            -0.013337312266230583,
+            -0.02514631673693657,
+            0.0014027173165231943,
+            0.001001182827167213,
+            0.001963450340554118,
+            -0.01913214474916458,
+            -0.01989629678428173,
+            0.011596739292144775,
+            -0.015863263979554176,
+            0.004928083624690771,
+            -0.06418890506029129,
+            -0.01855195313692093,
+            -0.004974074196070433,
+            0.0043337419629096985,
+            0.0007699025445617735,
+            0.03342464938759804,
+            -0.01478778850287199,
+            0.010117961093783379,
+            -0.01549533847719431,
+            -0.026363302022218704,
+            0.04010391607880592,
+            0.018750067800283432,
+            -0.023773670196533203,
+            0.024693485349416733,
+            0.012106175534427166,
+            0.03099067695438862,
+            -0.0318114347755909,
+            0.020349130034446716,
+            0.018523650243878365,
+            -0.0029823218937963247,
+            0.006551910191774368,
+            0.008575502783060074,
+            0.01832553744316101,
+            -0.015764206647872925,
+            -0.0003632825391832739,
+            0.011419852264225483,
+            0.010033054277300835,
+            0.031924642622470856,
+            0.01403071079403162,
+            -0.03183973580598831,
+            0.019047237932682037,
+            0.01616043597459793,
+            0.02298121340572834,
+            0.033820874989032745,
+            -0.027481229975819588,
+            0.038575608283281326,
+            -0.010386829264461994,
+            -0.024042539298534393,
+            0.020801961421966553,
+            -0.032688796520233154,
+            -0.006679269019514322,
+            0.030735958367586136,
+            0.04398128762841225,
+            -0.0015566092915832996,
+            -0.044632233679294586,
+            0.01535382866859436,
+            0.01316749956458807,
+            0.0027806703001260757,
+            0.028811423107981682,
+            0.011264191009104252,
+            0.021820832043886185,
+            0.025273675099015236,
+            0.011419852264225483,
+            -0.0037252490874379873,
+            0.024962352588772774,
+            -0.008101443760097027,
+            -0.004446949809789658,
+            -0.0007035697344690561,
+            0.03206615149974823,
+            0.014702882617712021,
+            -0.017603836953639984,
+            -0.036962397396564484,
+            -0.007620310410857201,
+            0.009452863596379757,
+            -0.007903330028057098,
+            -0.027127455919981003,
+            -0.03950957581400871,
+            -0.026787832379341125,
+            -0.005667473189532757,
+            0.01505665760487318,
+            -0.06724552065134048,
+            -0.007620310410857201,
+            -0.012233533896505833,
+            0.017717044800519943,
+            0.0022906921803951263,
+            0.0181132722645998,
+            0.001277127186767757,
+            -0.009403335861861706,
+            0.004209920763969421,
+            0.050802066922187805,
+            -0.03059444949030876,
+            0.014242975041270256,
+            0.011292492970824242,
+            0.0032105068676173687,
+            0.002787745790556073,
+            0.038688816130161285,
+            -0.010988246649503708,
+            -0.030424637719988823,
+            0.011575513519346714,
+            -0.017292514443397522,
+            0.003383856499567628,
+            0.05006621778011322,
+            -0.009481165558099747,
+            -0.027495380491018295,
+            0.020915169268846512,
+            -0.009148617275059223,
+            -0.015523640438914299,
+            -0.0027258351910859346,
+            -0.006035399157553911,
+            0.0006283925613388419,
+            -0.03832089155912399,
+            -0.015750056132674217,
+            0.03212275728583336,
+            0.008617955259978771,
+            0.014603826217353344,
+            0.025344429537653923,
+            0.010209942236542702,
+            -0.030481241643428802,
+            0.027481229975819588,
+            -0.0005678086308762431,
+            0.005221716593950987,
+            0.013323160819709301,
+            0.03424540534615517,
+            -0.014264201745390892,
+            -0.020264223217964172,
+            -0.040245428681373596,
+            -0.013853822834789753,
+            -0.0463869571685791,
+            0.001943992800079286,
+            0.006919836159795523,
+            -0.012919857166707516,
+            -0.04366996884346008,
+            -0.0329718142747879,
+            0.021127434447407722,
+            0.03178313374519348,
+            -0.017023645341396332,
+            0.0025294902734458447,
+            -0.008483520708978176,
+            -0.007882103323936462,
+            -0.0017520699184387922,
+            -0.013217028230428696,
+            -0.006378560326993465,
+            -0.006229975260794163,
+            0.015452885068953037,
+            -0.004652139265090227,
+            -0.0036934094969183207,
+            -0.0498398020863533,
+            -0.018778368830680847,
+            0.004082561936229467,
+            -0.008221727795898914,
+            0.007323139347136021,
+            -0.006955213379114866,
+            -0.002727604005485773,
+            -0.012474101036787033,
+            -0.03829259052872658,
+            0.024184048175811768,
+            0.017278363928198814,
+            0.0035501306410878897,
+            -0.008922201581299305,
+            -0.018367990851402283,
+            0.03294351324439049,
+            0.012523629702627659,
+            -0.017547233030200005,
+            -0.018580254167318344,
+            0.025981225073337555,
+            0.026957644149661064,
+            0.008122670464217663,
+            -0.006137993652373552,
+            0.029886899515986443,
+            0.005695775151252747,
+            -0.022938761860132217,
+            -0.003983504604548216,
+            -0.005055442452430725,
+            -0.0038844477385282516,
+            0.020561393350362778,
+            -0.010556641966104507,
+            0.0300001073628664,
+            0.038717120885849,
+            0.012870329432189465,
+            0.0039198254235088825,
+            -0.0033714743331074715,
+            -0.008922201581299305,
+            0.015976471826434135,
+            0.006481155287474394,
+            0.008525974117219448,
+            0.014759486541152,
+            -0.016358548775315285,
+            -0.020405733957886696,
+            0.006215824279934168,
+            -0.007107336539775133,
+            0.020122714340686798,
+            0.01708024926483631,
+            0.016429303213953972,
+            0.0007610581233166158,
+            -0.028287837281823158,
+            -0.0018024828750640154,
+            0.02411329373717308,
+            0.02010856196284294,
+            -0.004655676893889904,
+            -0.023264234885573387,
+            0.014398636296391487,
+            -0.015806660056114197,
+            -0.009969375096261501,
+            0.0033431723713874817,
+            0.006229975260794163,
+            -0.003083147807046771,
+            0.0571134127676487,
+            -0.020193468779325485,
+            -0.011419852264225483,
+            -0.010287772864103317,
+            -0.010867963545024395,
+            0.0012072566896677017,
+            -0.01828308403491974,
+            0.008964654989540577,
+            0.025613298639655113,
+            -0.009764186106622219,
+            0.03568880632519722,
+            0.0228114016354084,
+            0.006912760436534882,
+            0.005260631907731295,
+            -0.027070851996541023,
+            0.012304289266467094,
+            -0.016047226265072823,
+            0.005660397466272116,
+            0.018467046320438385,
+            0.017320817336440086,
+            0.0076132346875965595,
+            0.026306698098778725,
+            0.019386861473321915,
+            0.016684021800756454,
+            -0.022783100605010986,
+            0.025457637384533882,
+            -0.007832575589418411,
+            0.027877457439899445,
+            -0.01611798256635666,
+            -0.002253545681014657,
+            -0.012990612536668777,
+            0.008681634441018105,
+            0.002416282193735242,
+            -0.0019351483788341284,
+            0.00036129256477579474,
+            0.016783079132437706,
+            -0.014589674770832062,
+            0.01062032114714384,
+            0.013316085562109947,
+            -0.027919910848140717,
+            -0.010372678749263287,
+            0.016344398260116577,
+            -0.00668988237157464,
+            0.006877383217215538,
+            -0.011865608394145966,
+            -0.0017202302115038037,
+            0.011009473353624344,
+            -0.0436982698738575,
+            0.0104929618537426,
+            0.006279503460973501,
+            -0.015155714936554432,
+            0.014801939949393272,
+            -0.009799563325941563,
+            -0.008235878311097622,
+            -0.013592029921710491,
+            0.003792466362938285,
+            -0.00030225637601688504,
+            -0.0018290160223841667,
+            -0.00022995365725364536,
+            0.00389859895221889,
+            0.010047205723822117,
+            -0.0045990729704499245,
+            -0.010450509376823902,
+            -0.0028638073708862066,
+            -0.02265574224293232,
+            -0.0010047205723822117,
+            0.018735915422439575,
+            -0.004591997712850571,
+            0.0005222601466812193,
+            0.0145189194008708,
+            -0.005830209702253342,
+            0.010556641966104507,
+            -0.005221716593950987,
+            -0.008320784196257591,
+            -0.010768906213343143,
+            -0.033764272928237915,
+            0.0026710000820457935,
+            -0.015254771336913109,
+            0.008865597657859325,
+            -0.012544856406748295,
+            0.04027372971177101,
+            0.01022409275174141,
+            -0.00837738811969757,
+            -0.002796590095385909,
+            -0.0228114016354084,
+            -0.004475251771509647,
+            -0.016938740387558937,
+            -0.01184438169002533,
+            0.008342010900378227,
+            0.012629762291908264,
+            0.005614406894892454,
+            -0.018410442396998405,
+            0.003665107535198331,
+            0.0030955299735069275,
+            -0.005575491581112146,
+            -0.008370312862098217,
+            0.010853812098503113,
+            -0.0006248548161238432,
+            -0.01800006441771984,
+            -0.01957082375884056,
+            -0.027424626052379608,
+            0.0150000536814332,
+            0.014589674770832062,
+            -0.01011088490486145,
+            -0.022754797711968422,
+            -0.007471724878996611,
+            0.007107336539775133,
+            -0.02064630016684532,
+            0.0006779210525564849,
+            0.006891534198075533,
+            0.008717012591660023,
+            0.004117939155548811,
+            0.009636826813220978,
+            0.0007699025445617735,
+            0.0121698547154665,
+            0.018566103652119637,
+            0.03271709755063057,
+            -0.0396793857216835,
+            -0.009537769481539726,
+            0.0057028504088521,
+            -0.001465512323193252,
+            -0.013853822834789753,
+            0.009672204032540321,
+            0.0024375086650252342,
+            -0.0026426981203258038,
+            0.024424616247415543,
+            -0.008044839836657047,
+            -0.01505665760487318,
+            0.01184438169002533,
+            0.00887267291545868,
+            -0.019075538963079453,
+            0.02746707946062088,
+            0.006169833242893219,
+            -0.01011088490486145,
+            0.02259913645684719,
+            -0.012360893189907074,
+            0.012658064253628254,
+            -0.005522425286471844,
+            0.023660462349653244,
+            -0.019004784524440765,
+            0.0006306036957539618,
+            0.015665151178836823,
+            0.03883032873272896,
+            -0.00437973253428936,
+            -0.004567233379930258,
+            -0.03421710431575775,
+            0.005115584470331669,
+            0.028853876516222954,
+            0.02666047215461731,
+            -0.0012346742441877723,
+            -0.005596718285232782,
+            -0.0019864458590745926,
+            0.010768906213343143,
+            0.011589664034545422,
+            0.017561383545398712,
+            -0.029547275975346565,
+            0.00022011429246049374,
+            -0.016938740387558937,
+            0.025500090792775154,
+            0.014632128179073334,
+            0.006951675750315189,
+            0.04072656109929085,
+            0.0027789014857262373,
+            0.003841994795948267,
+            0.027452928945422173,
+            -0.00779719790443778,
+            0.003546592779457569,
+            -0.015566092915832996,
+            0.0003175129240844399,
+            0.022853855043649673,
+            0.0016370931407436728,
+            0.0020784272346645594,
+            0.013337312266230583,
+            0.029320860281586647,
+            -0.022471778094768524,
+            -0.023929331451654434,
+            0.003482913365587592,
+            0.007185167167335749,
+            -0.010570792481303215,
+            0.0038738346192985773,
+            0.006275965832173824,
+            -0.006007096730172634,
+            0.008900974877178669,
+            0.006332569755613804,
+            -0.0003166284877806902,
+            0.0039198254235088825,
+            -0.00308845448307693,
+            0.012962310574948788,
+            -0.02702839858829975,
+            0.036849189549684525,
+            0.004439874552190304,
+            0.010075507685542107,
+            0.015268922783434391,
+            0.001627364312298596,
+            0.004917470272630453,
+            0.005869124550372362,
+            0.010188715532422066,
+            0.018636859953403473,
+            -0.00956607237458229,
+            -0.027481229975819588,
+            -0.03166992589831352,
+            0.020363280549645424,
+            -0.01173117384314537,
+            0.0035872769076377153,
+            0.0074646491557359695,
+            -0.01828308403491974,
+            -0.007507102098315954,
+            -0.009976450353860855,
+            0.015764206647872925,
+            -0.03189634159207344,
+            0.012990612536668777,
+            -0.0043231286108493805,
+            -0.028669914230704308,
+            0.00851889792829752,
+            -0.010181640274822712,
+            -0.0045884596183896065,
+            -0.0035094465129077435,
+            -0.029264256358146667,
+            -0.007931631989777088,
+            0.03908504545688629,
+            -0.02141045406460762,
+            0.024184048175811768,
+            0.008257105015218258,
+            -0.00014449491573031992,
+            -0.0040365708991885185,
+            -0.013563727959990501,
+            0.005331386812031269,
+            -0.01778779923915863,
+            0.025457637384533882,
+            0.010294848121702671,
+            -0.013924578204751015,
+            0.0322076641023159,
+            -0.011179285123944283,
+            0.027438776567578316,
+            -0.018580254167318344,
+            0.036198243498802185,
+            0.007422196213155985,
+            0.00674648629501462,
+            0.01974063739180565,
+            -0.004025958012789488,
+            0.008228803053498268,
+            0.04559450224041939,
+            -0.018523650243878365,
+            0.02460857853293419,
+            0.020349130034446716,
+            -0.00022685811563860625,
+            0.017915157601237297,
+            0.0028390432707965374,
+            -0.023165177553892136,
+            0.009495317004621029,
+            -0.01135617308318615,
+            -0.009955224581062794,
+            0.01583496294915676,
+            -0.019316107034683228,
+            -0.018721764907240868,
+            -0.025613298639655113,
+            0.0029416377656161785,
+            -0.003350247861817479,
+            -0.0077618202194571495,
+            -0.0019616815261542797,
+            0.01454722136259079,
+            0.008724087849259377,
+            0.025669902563095093,
+            0.0008326975512318313,
+            0.011971740983426571,
+            0.01511326152831316,
+            -0.00035576484515331686,
+            0.02313687466084957,
+            -0.03229256719350815,
+            0.01246702577918768,
+            -0.004517704714089632,
+            -0.00509789539501071,
+            0.023858575150370598,
+            0.0003747802402358502,
+            -0.0004877670726273209,
+            -0.0024888061452656984,
+            0.0005496777012012899,
+            -0.014589674770832062,
+            -0.0057099261321127415,
+            0.000907432462554425,
+            -0.005175726022571325,
+            0.04299071803689003,
+            -0.00701535539701581,
+            -0.004772422835230827,
+            -0.039169952273368835,
+            -0.01918874867260456,
+            0.00623351288959384,
+            -0.01583496294915676,
+            -0.018778368830680847,
+            -0.006307805422693491,
+            0.005801907740533352,
+            -0.04893413558602333,
+            0.008342010900378227,
+            0.03336804360151291,
+            -0.012219383381307125,
+            0.00259493850171566,
+            -0.010981171391904354,
+            -0.006969364359974861,
+            -0.0029115669894963503,
+            -7.766463386360556e-05,
+            -0.002412744564935565,
+            -0.020830262452363968,
+            -0.005617944523692131,
+            0.003364398842677474,
+            -0.0008795727044343948,
+            -0.025047259405255318,
+            0.0006177793256938457,
+            0.018849123269319534,
+            -0.003979966975748539,
+            -0.0011152751976624131,
+            -0.001825478277169168,
+            0.006399787031114101,
+            -0.01508495956659317,
+            0.008009462617337704,
+            -0.0076344613917171955,
+            0.003841994795948267,
+            -0.0030495391692966223,
+            -0.002214630600064993,
+            0.02844349853694439,
+            0.021594416350126266,
+            -0.04808507859706879,
+            -0.01405901275575161,
+            0.023009516298770905,
+            0.007280686404556036,
+            -0.004740582779049873,
+            0.01178777776658535,
+            -0.031613320112228394,
+            -0.00028943203506059945,
+            -0.026787832379341125,
+            0.02200479619204998,
+            -0.009205221198499203,
+            -0.007386818993836641,
+            0.016967041417956352,
+            -0.026632171124219894,
+            0.016344398260116577,
+            0.00794578343629837,
+            -0.00030181417241692543,
+            -0.000493515923153609,
+            0.0063856360502541065,
+            0.0008972614887170494,
+            -0.01659911684691906,
+            0.006442239973694086,
+            -0.0048502529971301556,
+            0.007959933951497078,
+            -0.005087282508611679,
+            -0.05527378246188164,
+            -0.009933997876942158,
+            -0.005497660953551531,
+            -0.008936353027820587,
+            -0.012820800766348839,
+            0.008136821910738945,
+            0.02827368676662445,
+            0.01606137864291668,
+            -0.009863242506980896,
+            -4.8312376748071983e-05,
+            0.033000119030475616,
+            -0.0008300442132167518,
+            0.009955224581062794,
+            -0.011349096894264221,
+            0.00015709814033471048,
+            -0.0006911876262165606,
+            -0.014660430140793324,
+            -0.019061388447880745,
+            0.024042539298534393,
+            -0.028344441205263138,
+            0.02378782071173191,
+            -0.01843874529004097,
+            -0.0130825936794281,
+            0.01757553406059742,
+            -0.032575588673353195,
+            -0.005108508747071028,
+            0.007450498174875975,
+            0.0090566361322999,
+            0.021608568727970123,
+            0.0032794929575175047,
+            0.020589696243405342,
+            -0.03716050833463669,
+            -0.009134466759860516,
+            -0.010160413570702076,
+            -0.00794578343629837,
+            -0.010231168940663338,
+            -0.011596739292144775,
+            0.021863285452127457,
+            -0.015240620821714401,
+            -0.016952890902757645,
+            0.007054270710796118,
+            0.0030212372075766325,
+            -0.011313719674944878,
+            0.015750056132674217,
+            0.00017810352437663823,
+            0.004712280817329884,
+            0.013011839240789413,
+            -0.0008923970744945109,
+            0.04086806997656822,
+            0.0253019779920578,
+            -0.0030212372075766325,
+            -0.008568426594138145,
+            0.021438755095005035,
+            0.0004285097820684314,
+            0.0150000536814332,
+            0.009375033900141716,
+            0.023264234885573387,
+            -0.02092931978404522,
+            0.009997677057981491,
+            -0.005501199048012495,
+            0.014603826217353344,
+            -0.009495317004621029,
+            -0.015339677222073078,
+            0.00479718716815114,
+            -0.01577835902571678,
+            0.010153338313102722,
+            -0.010860888287425041,
+            0.008221727795898914,
+            0.01913214474916458,
+            0.002467579673975706,
+            -0.00941041111946106,
+            0.01146230474114418,
+            0.02075950801372528,
+            -0.025641601532697678,
+            0.008851447142660618,
+            -0.0031998935155570507,
+            -0.0021916350815445185,
+            0.021127434447407722,
+            0.010301923379302025,
+            -0.01594817079603672,
+            0.01573590561747551,
+            -0.008879749104380608,
+            -0.0015053119277581573,
+            -0.01882082223892212,
+            -0.03291521221399307,
+            -0.0015283073298633099,
+            0.012396270409226418,
+            0.004800724796950817,
+            0.0150000536814332,
+            0.011264191009104252,
+            -0.0008950503543019295,
+            -0.006601438857614994,
+            0.0008433107868768275,
+            -0.013485897332429886,
+            -0.004963461309671402,
+            0.010231168940663338,
+            0.017051948234438896,
+            -0.0037217114586383104,
+            0.015650998800992966,
+            0.0088160689920187,
+            0.006137993652373552,
+            -0.0010480580385774374,
+            0.0036138100549578667,
+            -0.002060738392174244,
+            -0.007783046457916498,
+            0.019938750192523003,
+            0.019075538963079453,
+            -0.018509499728679657,
+            -0.0024870370980352163,
+            -0.02141045406460762,
+            0.025882167741656303,
+            0.022188758477568626,
+            0.011165134608745575,
+            -0.04222656786441803,
+            0.00012050455552525818,
+            0.00752832880243659,
+            -0.0013744152383878827,
+            -0.031103884801268578,
+            0.008172199130058289,
+            0.005221716593950987,
+            0.004330204334110022,
+            0.020844414830207825,
+            -0.018424594774842262,
+            0.010853812098503113,
+            0.0070861103013157845,
+            -0.0005766529939137399,
+            0.01019579079002142,
+            -0.0073372903279960155,
+            0.008688710629940033,
+            0.038717120885849,
+            -0.018410442396998405,
+            0.005303084850311279,
+            0.0062370505183935165,
+            0.023320838809013367,
+            -0.011228813789784908,
+            -0.00031485961517319083,
+            0.00800238735973835,
+            0.021608568727970123,
+            -0.012934008613228798,
+            0.005883275531232357,
+            0.067075714468956,
+            -0.012912781909108162,
+            0.031613320112228394,
+            0.020533092319965363,
+            -0.004970536567270756,
+            -0.0104929618537426,
+            -0.012445799075067043,
+            -0.0181132722645998,
+            0.021283095702528954,
+            0.01175947580486536,
+            0.003419233951717615,
+            0.01941516436636448,
+            -0.01610383205115795,
+            0.006636816076934338,
+            0.017547233030200005,
+            0.012601460330188274,
+            -0.007379743270576,
+            -0.0036173479165881872,
+            0.019004784524440765,
+            0.018509499728679657,
+            -0.0030849166214466095,
+            0.00019424450874794275,
+            0.01508495956659317,
+            -0.022075550630688667,
+            -0.012530704960227013,
+            0.01644345559179783,
+            -0.0037358624394983053,
+            0.017773648723959923,
+            -0.02227366529405117,
+            0.024198200553655624,
+            -0.004531855694949627,
+            -0.017179306596517563,
+            0.004698129836469889,
+            -0.007514177821576595,
+            0.022033097222447395,
+            -0.001578720286488533,
+            0.004857328720390797,
+            0.000821199850179255,
+            -0.0018015984678640962,
+            0.01349297259002924,
+            0.002826661104336381,
+            -0.0005120891146361828,
+            0.010507113300263882,
+            -0.0009702275274321437,
+            -0.0013496510218828917,
+            0.024509521201252937,
+            1.9374700059415773e-05,
+            0.005989408120512962,
+            0.0032936439383774996,
+            -0.009261825121939182,
+            -0.004429261200129986,
+            -0.004443412180989981,
+            0.0011842612875625491,
+            0.003877372248098254,
+            -0.007029506377875805,
+            -0.02384442463517189,
+            0.022471778094768524,
+            -0.008540124632418156,
+            0.01175947580486536,
+            0.011349096894264221,
+            0.003665107535198331,
+            0.006824316922575235,
+            -0.009771261364221573,
+            -0.017320817336440086,
+            0.0006633278680965304,
+            -0.01617458648979664,
+            0.006912760436534882,
+            -0.001656550681218505,
+            0.0020235921256244183,
+            0.009445788338780403,
+            -0.002570174168795347,
+            0.0015123875346034765,
+            0.011306644417345524,
+            0.011497682891786098,
+            0.003254728624597192,
+            0.004577846731990576,
+            -0.018721764907240868,
+            -0.020915169268846512,
+            0.005288933869451284,
+            0.013924578204751015,
+            0.004273600410670042,
+            0.0015309606678783894,
+            0.017603836953639984,
+            -0.006166295614093542,
+            -0.022967062890529633,
+            -0.010117961093783379,
+            -0.01584911346435547,
+            0.039396367967128754,
+            0.01800006441771984,
+            -0.02525952458381653,
+            -0.000530220044311136,
+            -0.009728807955980301,
+            0.010450509376823902,
+            -0.00404364662244916,
+            0.014575524255633354,
+            0.005476434715092182,
+            -0.02426895499229431,
+            -0.016797229647636414,
+            -0.010662773624062538,
+            -0.005257094278931618,
+            0.020080260932445526,
+            0.027014248073101044,
+            -0.004956385586410761,
+            0.0061026159673929214,
+            0.019584976136684418,
+            -0.010938717983663082,
+            0.009849091991782188,
+            0.020957622677087784,
+            0.0031698227394372225,
+            -0.007570781745016575,
+            -0.004553082399070263,
+            -0.016514210030436516,
+            0.024467067793011665,
+            0.014250051230192184,
+            -0.005104971118271351,
+            -0.02465103194117546,
+            0.004110863897949457,
+            -0.02428310550749302,
+            -0.01616043597459793,
+            -0.007301913108676672,
+            -0.014143918640911579,
+            -0.004931621253490448,
+            0.0027594438288360834,
+            -0.004836102016270161,
+            0.011278342455625534,
+            -0.0031592093873769045,
+            -0.004531855694949627,
+            -0.0041851564310491085,
+            -0.025584997609257698,
+            -0.02524537406861782,
+            0.007691065315157175,
+            -0.003005317412316799,
+            0.005596718285232782,
+            0.000650945701636374,
+            -0.013400991447269917,
+            -0.011589664034545422,
+            0.00970758218318224,
+            -0.007988235913217068,
+            0.012813725508749485,
+            0.0031149876303970814,
+            -0.009183995425701141,
+            0.0017193458043038845,
+            -0.0012311364989727736,
+            -0.0059752571396529675,
+            0.019924599677324295,
+            0.00916984397917986,
+            0.011724098585546017,
+            -0.03118879161775112,
+            -0.0021633331198245287,
+            0.008349086157977581,
+            0.01828308403491974,
+            -0.02244347706437111,
+            -0.008313708938658237,
+            0.005179263651371002,
+            -0.011228813789784908,
+            -0.013507124036550522,
+            -0.028952933847904205,
+            -0.0030460015404969454,
+            8.175515540642664e-05,
+            -0.018636859953403473,
+            -0.029886899515986443,
+            0.0009038947173394263,
+            -0.007860877551138401,
+            0.016882136464118958,
+            -0.006643891800194979,
+            0.01790100708603859,
+            0.00870286114513874,
+            0.0036173479165881872,
+            0.011554286815226078,
+            0.0025082635693252087,
+            -0.013266556896269321,
+            0.008426916785538197,
+            -0.02671707607805729,
+            -0.01516986545175314,
+            0.009792488068342209,
+            0.014214673079550266,
+            -0.0026126273442059755,
+            5.074458022136241e-05,
+            0.0058372849598526955,
+            0.024254804477095604,
+            0.013584954664111137,
+            0.012474101036787033,
+            0.003042463678866625,
+            -0.005303084850311279,
+            0.005568416323512793,
+            0.0227123461663723,
+            0.008603804744780064,
+            0.004011806566268206,
+            0.007592008449137211,
+            0.00389859895221889,
+            0.0009932229295372963,
+            -0.005943417549133301,
+            -0.00843399204313755,
+            -0.021537812426686287,
+            -0.008908051066100597,
+            0.019938750192523003,
+            -0.010422207415103912,
+            -0.03005671128630638,
+            0.0006023016758263111,
+            -0.0032777239102870226,
+            0.007988235913217068,
+            0.02595292218029499,
+            -0.014172220602631569,
+            0.004970536567270756,
+            0.005536576267331839,
+            -0.007910405285656452,
+            -0.00614860700443387,
+            0.022967062890529633,
+            0.02639160305261612,
+            -0.0034404604230076075,
+            -0.04800017178058624,
+            -0.01671232469379902,
+            0.004305440001189709,
+            -0.007768895477056503,
+            -0.034047290682792664,
+            -0.015424583107233047,
+            0.002757675014436245,
+            -0.03206615149974823,
+            0.010068432427942753,
+            -0.012948159128427505,
+            -0.01214155275374651,
+            0.020264223217964172,
+            0.03223596513271332,
+            0.005154499784111977,
+            0.00014847487909719348,
+            -0.0214670579880476,
+            -0.0021686397958546877,
+            -0.021382151171565056,
+            0.010174565017223358,
+            -0.008801918476819992,
+            -0.010584943927824497,
+            0.0059611061587929726,
+            -0.012191081419587135,
+            0.013266556896269321,
+            0.010797208175063133,
+            0.024778390303254128,
+            0.01584911346435547,
+            -0.01876421831548214,
+            -0.0024746551644057035,
+            -0.01108730398118496,
+            -0.0016450530383735895,
+            -0.009820790030062199,
+            0.006360871717333794,
+            -0.0031592093873769045,
+            0.011264191009104252,
+            0.015184016898274422,
+            0.004432798828929663,
+            0.00046786724124103785,
+            0.038151077926158905,
+            -2.4487851987942122e-05,
+            -0.0003418349369894713,
+            0.006803090218454599,
+            -0.015750056132674217,
+            0.008837295696139336,
+            0.012191081419587135,
+            0.005051904823631048,
+            -0.007698140572756529,
+            0.006739411037415266,
+            0.03099067695438862,
+            -0.01516986545175314,
+            -0.009014182724058628,
+            -0.023504801094532013,
+            0.007549555506557226,
+            -0.0004656561359297484,
+            0.029971804469823837,
+            0.020533092319965363,
+            -0.027509532868862152,
+            -0.016018925234675407,
+            -0.02179253101348877,
+            0.01267929095774889,
+            -0.011497682891786098,
+            -0.003704022616147995,
+            0.012240610085427761,
+            0.02367461286485195,
+            0.0018838511314243078,
+            -0.014455240219831467,
+            -0.008950503543019295,
+            0.008943428285419941,
+            0.014617976732552052,
+            0.003497064346447587,
+            0.007570781745016575,
+            0.00743634719401598,
+            0.008886824361979961,
+            -0.014801939949393272,
+            -0.012254760600626469,
+            0.00767691433429718,
+            -0.010202866978943348,
+            0.01794346049427986,
+            -0.009403335861861706,
+            0.016768928617239,
+            -0.004645063541829586,
+            -0.011964665725827217,
+            -0.0320095494389534,
+            -0.002292460994794965,
+            0.005037753842771053,
+            0.004627374932169914,
+            0.0034705311991274357,
+            -0.0022217060904949903,
+            -0.013160424306988716,
+            0.004673365969210863,
+            0.005073131527751684,
+            1.875283123808913e-05,
+            -0.008313708938658237,
+            -0.00479718716815114,
+            0.004252373706549406,
+            -0.001031253719702363,
+            -0.0073372903279960155,
+            -0.0059858704917132854,
+            0.018481198698282242,
+            0.0066509670577943325,
+            0.00800238735973835,
+            0.03234917297959328,
+            0.02974538877606392,
+            -0.0016353242099285126,
+            -0.004606148693710566,
+            -0.00023304918431676924,
+            -0.01589156687259674,
+            -0.0057205394841730595,
+            -0.023320838809013367,
+            0.022259512916207314,
+            -0.009665128774940968,
+            0.021820832043886185,
+            0.019457615911960602,
+            0.0039304387755692005,
+            -0.008391539566218853,
+            -0.0011586126638576388,
+            -0.007308988366276026,
+            -0.003560743760317564,
+            0.016641568392515182,
+            0.006742948666214943,
+            0.01226891204714775,
+            0.00030933189555071294,
+            -0.028401045128703117,
+            0.026405755430459976,
+            0.000579748535528779,
+            0.0034015451092272997,
+            0.008908051066100597,
+            0.014129767194390297,
+            0.013209952972829342,
+            -0.01235381793230772,
+            0.01768874190747738,
+            -0.03280200436711311,
+            -0.0033484790474176407,
+            -0.016500059515237808,
+            0.002076658420264721,
+            -0.0018520113080739975,
+            -0.0008145665633492172,
+            -0.05354736000299454,
+            0.0003842879377771169,
+            0.005313698202371597,
+            -0.015339677222073078,
+            0.029349161311984062,
+            -0.020589696243405342,
+            -0.017165156081318855,
+            -0.032858606427907944,
+            -0.010238244198262691,
+            0.018099121749401093,
+            0.005617944523692131,
+            0.00035178486723452806,
+            -0.02660386823117733,
+            0.015750056132674217,
+            -0.0047476585023105145,
+            -0.00743634719401598,
+            -0.029858596622943878,
+            -0.010938717983663082,
+            0.009403335861861706,
+            0.004485865123569965,
+            -0.0130825936794281,
+            0.009898620657622814,
+            -0.002890340518206358,
+            0.010867963545024395,
+            -0.0138467475771904,
+            -0.00420284504070878,
+            0.0010020672343671322,
+            -0.005363226868212223,
+            0.012615610845386982,
+            -0.030537845566868782,
+            0.011391550302505493,
+            -0.01352127455174923,
+            0.0008548084879294038,
+            -0.017858553677797318,
+            -0.027707645669579506,
+            0.0020288988016545773,
+            -0.016132133081555367,
+            0.039990708231925964,
+            0.010188715532422066,
+            -0.01481609046459198,
+            0.016995344310998917,
+            -0.002080196049064398,
+            0.017377421259880066,
+            0.0014955832157284021,
+            0.017023645341396332,
+            -0.011971740983426571,
+            0.021056678146123886,
+            -0.0041745430789887905,
+            0.02156611531972885,
+            0.00600002147257328,
+            0.034047290682792664,
+            -0.02033497765660286,
+            0.0070825726725161076,
+            0.003792466362938285,
+            0.005798369646072388,
+            -0.0026816134341061115,
+            0.021424604579806328,
+            0.017985913902521133,
+            0.019698183983564377,
+            0.0037110981065779924,
+            -0.024084990844130516,
+            -0.006137993652373552,
+            0.004867942072451115,
+            0.023858575150370598,
+            0.011150983162224293,
+            -0.016415152698755264,
+            -0.021056678146123886,
+            0.015226469375193119,
+            -0.007860877551138401,
+            0.006771250627934933,
+            -0.00851889792829752,
+            0.00037544354563578963,
+            -0.013259481638669968,
+            0.012233533896505833,
+            -0.012523629702627659,
+            0.0032335021533071995,
+            0.01043635793030262,
+            -0.027806703001260757,
+            0.01511326152831316,
+            -0.00371817359700799,
+            -0.014313730411231518,
+            0.023320838809013367,
+            -0.015877414494752884,
+            -0.013988257385790348,
+            0.005501199048012495,
+            0.008610880002379417,
+            0.01746232621371746,
+            -0.002761212643235922,
+            0.002669231267645955,
+            -0.020504789426922798,
+            -0.012445799075067043,
+            0.004591997712850571,
+            0.02210385352373123,
+            -0.015820810571312904,
+            0.012431648559868336,
+            -0.018099121749401093,
+            0.01364863384515047,
+            0.016202887520194054,
+            0.007149789482355118,
+            0.00573115237057209,
+            -0.007726442534476519,
+            -0.0011471149045974016,
+            0.032151058316230774,
+            -0.0029876285698264837,
+            0.02704254910349846,
+            -0.014915147796273232,
+            0.009615600109100342,
+            0.00518987700343132,
+            -0.013089669868350029,
+            -0.014589674770832062,
+            0.011108530685305595,
+            0.02428310550749302,
+            -0.009530694223940372,
+            -0.023660462349653244,
+            0.0018484735628589988,
+            0.014264201745390892,
+            0.010337301529943943,
+            0.009403335861861706,
+            -0.001997058978304267,
+            -0.0037358624394983053,
+            0.01484439242631197,
+            -0.0032918748911470175,
+            -0.0286557637155056,
+            -0.01633024774491787,
+            -0.004400959238409996,
+            -0.017490629106760025,
+            0.020745357498526573,
+            0.02367461286485195,
+            0.010507113300263882,
+            0.0037570889107882977,
+            0.005540114361792803,
+            0.018297234550118446,
+            0.01843874529004097,
+            0.013330236077308655,
+            0.009304278530180454,
+            0.012700516730546951,
+            -0.004319590982049704,
+            0.02800481766462326,
+            -0.010860888287425041,
+            -0.01046465989202261,
+            -0.015707602724432945,
+            0.0016636261716485023,
+            -0.005193414632230997,
+            -0.012431648559868336,
+            -0.0069481381215155125,
+            0.004369119182229042,
+            0.013330236077308655,
+            -0.012240610085427761,
+            -0.007149789482355118,
+            0.01016748882830143,
+            0.011971740983426571,
+            0.01403071079403162,
+            0.01719345711171627,
+            -0.01633024774491787,
+            -0.011278342455625534,
+            0.0009834941010922194,
+            -0.02844349853694439,
+            -0.0160047747194767,
+            -0.01767459139227867,
+            0.009580222889780998,
+            -0.014469391666352749,
+            0.03724541515111923,
+            0.028471799567341805,
+            0.0073514413088560104,
+            0.031613320112228394,
+            -0.028401045128703117,
+            0.00970758218318224,
+            -0.014398636296391487,
+            0.007132100872695446,
+            -0.004188694059848785,
+            0.003902136581018567,
+            0.002448122017085552,
+            0.020419884473085403,
+            0.00791748147457838,
+            0.014264201745390892,
+            0.008150972425937653,
+            -0.021934039890766144,
+            -0.00662620272487402,
+            -0.012559006921947002,
+            -0.016401002183556557,
+            -0.02595292218029499,
+            0.013719388283789158,
+            0.005087282508611679,
+            -0.003074303502216935,
+            -0.0002704166399780661,
+            0.006566061172634363,
+            -0.0013195802457630634,
+            0.010443433187901974,
+            -0.027608588337898254,
+            -0.01311089564114809,
+            0.0069269114173948765,
+            0.01449061743915081,
+            -0.02114158496260643,
+            0.0014637433923780918,
+            0.008900974877178669,
+            -0.004602610599249601,
+            0.035660505294799805,
+            -0.014674580655992031,
+            -0.008610880002379417,
+            0.0027806703001260757,
+            0.005062518175691366,
+            -0.017660440877079964,
+            -0.02319347858428955,
+            0.0070648835971951485,
+            0.011575513519346714,
+            0.010867963545024395,
+            -0.028570856899023056,
+            0.02087271586060524,
+            -0.0054339817725121975,
+            0.03832089155912399,
+            0.00401534466072917,
+            -0.013337312266230583,
+            0.003962278366088867,
+            0.0011860302183777094,
+            -0.012360893189907074,
+            0.002060738392174244,
+            -0.000340508297085762,
+            0.003700484987348318,
+            0.011228813789784908,
+            -0.014801939949393272,
+            -0.016967041417956352,
+            0.016910437494516373,
+            -0.011476456187665462,
+            -0.0035395172890275717,
+            -0.00943871308118105,
+            -0.016669871285557747,
+            0.007308988366276026,
+            0.025047259405255318,
+            0.0041745430789887905,
+            0.001242634141817689,
+            -0.011632117442786694,
+            -0.0038596836384385824,
+            -0.008575502783060074,
+            0.0011718792375177145,
+            -0.0207029040902853,
+            0.027396325021982193,
+            -0.01801421493291855,
+            0.002410975517705083,
+            0.021099131554365158,
+            0.008709936402738094,
+            -0.003944589756429195,
+            0.016316095367074013,
+            0.026363302022218704,
+            0.01205664686858654,
+            0.0031539027113467455,
+            -0.00902833417057991,
+            1.648646139074117e-05,
+            0.014957600273191929,
+            0.0015123875346034765,
+            -0.0015663381200283766,
+            0.0032140444964170456,
+            -0.006583749782294035,
+            -0.001859086798503995,
+            -0.017561383545398712,
+            0.005564878229051828,
+            0.02125479280948639,
+            -0.004132090136408806,
+            0.020476488396525383,
+            0.015509488992393017,
+            0.011625041253864765,
+            0.010726453736424446,
+            -0.0012390963966026902,
+            -0.002150951186195016,
+            -0.0021668709814548492,
+            -0.007393894251435995,
+            -0.018297234550118446,
+            0.006470541935414076,
+            0.0034493047278374434,
+            0.01822648011147976,
+            0.020504789426922798,
+            -0.02308027073740959,
+            0.010146263055503368,
+            0.014985902234911919,
+            -0.013011839240789413,
+            0.0235755555331707,
+            -0.02006610855460167,
+            0.0024605041835457087,
+            0.02156611531972885,
+            0.013337312266230583,
+            -0.014271277002990246,
+            -0.009219372645020485,
+            0.004662752617150545,
+            -0.01543873455375433,
+            0.000872939417604357,
+            -0.01478778850287199,
+            -0.010783057659864426,
+            0.014533070847392082,
+            -0.01490099634975195,
+            0.01229721400886774,
+            -0.006969364359974861,
+            0.0070967236533761024,
+            -0.017165156081318855,
+            0.015014204196631908,
+            0.008632106706500053,
+            0.01757553406059742,
+            0.0021279556676745415,
+            -0.02518877014517784,
+            -0.006290116813033819,
+            0.021169887855648994,
+            0.009509467519819736,
+            -0.02497650496661663,
+            0.019613277167081833,
+            0.0020094411447644234,
+            -0.019528372213244438,
+            -0.022401023656129837,
+            -0.012686366215348244,
+            -0.005055442452430725,
+            -0.017264213413000107,
+            -0.0011276573641225696,
+            -0.024792540818452835,
+            0.003972891718149185,
+            0.009106164798140526,
+            -0.011978816241025925,
+            -0.019273653626441956,
+            -0.0002315014135092497,
+            -0.01040805596858263,
+            0.006219361908733845,
+            -0.01132787112146616,
+            -0.019853845238685608,
+            0.0070436573587358,
+            0.012127402238547802,
+            -0.0027647505048662424,
+            -0.005646246485412121,
+            -0.02562745101749897,
+            -0.006919836159795523,
+            0.020179318264126778,
+            -0.003994117956608534,
+            -0.004974074196070433,
+            -0.01703779585659504,
+            0.01108730398118496,
+            0.011165134608745575,
+            0.020901018753647804,
+            0.010931642726063728,
+            0.005674548447132111,
+            0.013634482398629189,
+            -0.0029416377656161785,
+            -0.007156865205615759,
+            -9.645892532716971e-06,
+            0.004284213297069073,
+            0.0069233737885952,
+            0.0016636261716485023,
+            0.020235922187566757,
+            -0.01692458800971508,
+            0.018934030085802078,
+            -0.012290137819945812,
+            0.008794843219220638,
+            -0.01855195313692093,
+            -0.0003155229496769607,
+            -0.01067692507058382,
+            0.00015002264990471303,
+            -0.0038455326575785875,
+            -0.009544845670461655,
+            1.9595809135353193e-05,
+            0.00143986358307302,
+            0.008511822670698166,
+            -0.016783079132437706,
+            0.009615600109100342,
+            -0.010839661583304405,
+            -0.004308977629989386,
+            -0.019542522728443146,
+            0.017448175698518753,
+            0.013344387523829937,
+            0.007308988366276026,
+            -0.008278331719338894,
+            -0.006300730165094137,
+            -0.02282555401325226,
+            0.008582578040659428,
+            -0.0014637433923780918,
+            0.002214630600064993,
+            -0.028301987797021866,
+            -0.004358506295830011,
+            0.0024587351363152266,
+            0.020306676626205444,
+            0.0033661676570773125,
+            -0.010712302289903164,
+            0.008978805504739285,
+            -0.023377442732453346,
+            -0.0030159305315464735,
+            -0.023349139839410782,
+            -0.004075486212968826,
+            -0.005112046375870705,
+            0.010103809647262096,
+            0.008101443760097027,
+            0.01549533847719431,
+            -0.009898620657622814,
+            0.01478778850287199,
+            -0.004793649073690176,
+            0.012049571610987186,
+            -0.0012152165872976184,
+            0.01337976474314928,
+            -0.0003111007681582123,
+            -0.01505665760487318,
+            0.02158026583492756,
+            -0.008738238364458084,
+            0.001741456682793796,
+            -0.0052853962406516075,
+            0.0027346794959157705,
+            -0.017702894285321236,
+            -0.004553082399070263,
+            -0.014455240219831467,
+            0.01957082375884056,
+            -0.022584985941648483,
+            0.004153316840529442,
+            -0.005055442452430725,
+            -0.016075529158115387,
+            0.029434068128466606,
+            0.022896308451890945,
+            -0.0035483618266880512,
+            0.015070809051394463,
+            0.0017989451298490167,
+            0.0015061964513733983,
+            0.016202887520194054,
+            -0.015806660056114197,
+            -0.013025989755988121,
+            0.035066161304712296,
+            0.0176321379840374,
+            0.0004313841927796602,
+            -0.0023260696325451136,
+            -0.0037323245778679848,
+            0.0032193511724472046,
+            0.008023613132536411,
+            -0.010450509376823902,
+            -0.01141277700662613,
+            -0.017051948234438896,
+            0.009997677057981491,
+            -0.006155682262033224,
+            0.004446949809789658,
+            -0.00025891896802932024,
+            -7.617214578203857e-05,
+            -0.01516986545175314,
+            0.022584985941648483,
+            -0.017717044800519943,
+            -0.00602832343429327,
+            0.004687516950070858,
+            0.0207029040902853,
+            -0.02541518583893776,
+            0.01616043597459793,
+            -0.007107336539775133,
+            -0.013429293408989906,
+            0.012573158368468285,
+            -0.015127412974834442,
+            -0.0107335289940238,
+            -0.003175129182636738,
+            -0.008009462617337704,
+            -0.022754797711968422,
+            0.017929309979081154,
+            -0.013627407141029835,
+            0.015650998800992966,
+            -0.00287972716614604,
+            0.000562502013053745,
+            0.002057200763374567,
+            0.0032582662533968687,
+            0.02250008098781109,
+            0.030735958367586136,
+            0.010415131226181984,
+            -0.005465821363031864,
+            -0.021000074222683907,
+            0.012983537279069424,
+            0.0020589695777744055,
+            0.013202877715229988,
+            -0.002253545681014657,
+            -0.002135031158104539,
+            -0.016613267362117767,
+            -0.023122724145650864,
+            -0.0013010069960728288,
+            -0.01761798746883869,
+            -0.02552839368581772,
+            -0.01774534583091736,
+            -0.007047194987535477,
+            0.011349096894264221,
+            0.020405733957886696,
+            0.0009755341452546418,
+            -0.005384453106671572,
+            0.01559439580887556,
+            -0.006941062398254871,
+            0.002080196049064398,
+            -0.006594363134354353,
+            -0.002032436430454254,
+            -0.008200501091778278,
+            -0.021863285452127457,
+            0.029660483822226524,
+            -0.010103809647262096,
+            0.01141277700662613,
+            -0.005472897086292505,
+            -0.00965097825974226,
+            0.005575491581112146,
+            0.00900003220885992,
+            0.024849144741892815,
+            0.006452853325754404,
+            0.03141520544886589,
+            0.020151015371084213,
+            -0.018891576677560806,
+            -0.0006071660900488496,
+            0.022684043273329735,
+            -0.0037570889107882977,
+            -0.004889168310910463,
+            -0.0031786670442670584,
+            -0.017985913902521133,
+            -0.012007118202745914,
+            0.007839650847017765,
+            -0.013471745885908604,
+            -0.006636816076934338,
+            -0.00837738811969757,
+            0.018070818856358528,
+            -0.017773648723959923,
+            0.025698205456137657,
+            -0.006488230545073748,
+            0.005154499784111977,
+            0.031924642622470856,
+            0.014363259077072144,
+            0.019698183983564377,
+            -0.011794853024184704,
+            -0.031075581908226013,
+            -0.00683846790343523,
+            -0.007882103323936462,
+            0.009558996185660362,
+            -0.01062032114714384,
+            0.005204027984291315,
+            -0.02006610855460167,
+            -0.018948180601000786,
+            0.012863253243267536,
+            -0.017476476728916168,
+            0.013924578204751015,
+            -0.009955224581062794,
+            0.027226513251662254,
+            -0.001228483160957694,
+            0.0121698547154665,
+            -0.002614396158605814,
+            -0.017801949754357338,
+            0.01205664686858654,
+            0.004089637193828821,
+            -0.023943481966853142,
+            0.010337301529943943,
+            -0.001864393474534154,
+            -0.014037786051630974,
+            -0.018410442396998405,
+            -0.001228483160957694,
+            -0.005200490355491638,
+            -0.009658053517341614,
+            0.025457637384533882,
+            0.014101465232670307,
+            0.016231190413236618,
+            0.0035359796602278948,
+            0.0016335553955286741,
+            -0.011073152534663677,
+            0.009573147632181644,
+            -0.006803090218454599,
+            0.000612030504271388,
+            -0.0034404604230076075,
+            0.011893910355865955,
+            -0.004252373706549406,
+            -0.01543873455375433,
+            0.002324300818145275,
+            0.008561351336538792,
+            0.0143278818577528,
+            5.947839599684812e-05,
+            0.009587298147380352,
+            0.0009790719486773014,
+            -0.009764186106622219,
+            -0.00046388726332224905,
+            -0.0060884649865329266,
+            0.0069375247694551945,
+            0.02400008589029312,
+            -0.012516554445028305,
+            -0.01681138016283512,
+            0.014504768885672092,
+            0.004033033270388842,
+            -0.017589686438441277,
+            0.018367990851402283,
+            0.004103788174688816,
+            -0.022358570247888565,
+            0.0035412863362580538,
+            0.010662773624062538,
+            -0.009523618966341019,
+            -0.014469391666352749,
+            -0.0010339069413021207,
+            -0.015509488992393017,
+            0.03130199760198593,
+            0.00025073791039176285,
+            0.016910437494516373,
+            -0.01062032114714384,
+            -0.0025967073161154985,
+            -0.013309009373188019,
+            0.013103820383548737,
+            -0.015750056132674217,
+            -0.007273610681295395,
+            0.00035620704875327647,
+            -0.02298121340572834,
+            0.01741987280547619,
+            0.0013425755314528942,
+            0.00794578343629837,
+            0.003436922561377287,
+            -0.010556641966104507,
+            0.009820790030062199,
+            0.0018237093463540077,
+            0.006668656133115292,
+            -0.014915147796273232,
+            0.023618008941411972,
+            0.006413938011974096,
+            0.011235889047384262,
+            -0.01352127455174923,
+            -0.01546703651547432,
+            0.002253545681014657,
+            0.04134920611977577,
+            -0.01306136790663004,
+            0.0002593612007331103,
+            0.01562269777059555,
+            7.081024523358792e-05,
+            0.004156854469329119,
+            -0.0023755980655550957,
+            -0.0034493047278374434,
+            -0.034585028886795044,
+            0.005370302125811577,
+            -0.0006328147719614208,
+            0.008073141798377037,
+            -0.008992956951260567,
+            -0.005430444143712521,
+            -0.0040684109553694725,
+            -0.008625030517578125,
+            -0.00314859626814723,
+            -0.01478778850287199,
+            0.008363237604498863,
+            -0.014533070847392082,
+            -0.028910480439662933,
+            0.03166992589831352,
+            -0.006887996103614569,
+            -0.00533846253529191,
+            -0.021396303549408913,
+            0.007570781745016575,
+            -0.01370523776859045,
+            0.002663924591615796,
+            0.010967019945383072,
+            0.008865597657859325,
+            0.008738238364458084,
+            0.002762981690466404,
+            0.0009516543359495699,
+            -0.004255911335349083,
+            0.010273621417582035,
+            -0.00835616234689951,
+            -0.0005943417199887335,
+            -0.0014186371117830276,
+            -0.02725481428205967,
+            -0.01405901275575161,
+            -0.010592019185423851,
+            -0.007811348885297775,
+            -0.00435143057256937,
+            0.012608535587787628,
+            0.007174553815275431,
+            -0.0017485321732237935,
+            -0.022938761860132217,
+            -0.014745336025953293,
+            -0.0034103896468877792,
+            0.008349086157977581,
+            -0.01838214136660099,
+            0.01064862310886383,
+            0.0016830838285386562,
+            0.009976450353860855,
+            0.014292503707110882,
+            0.00302831269800663,
+            0.0016742395237088203,
+            0.003458149265497923,
+            0.014886845834553242,
+            0.012601460330188274,
+            -0.005801907740533352,
+            -0.021382151171565056,
+            -0.0024605041835457087,
+            -0.007068421691656113,
+            -0.0034687623847275972,
+            -0.012332591228187084,
+            0.03039633482694626,
+            0.0029805530793964863,
+            0.0176321379840374,
+            0.007832575589418411,
+            0.009573147632181644,
+            0.010507113300263882,
+            0.029377464205026627,
+            0.006049550138413906,
+            -0.0029929352458566427,
+            -0.008603804744780064,
+            -0.02141045406460762,
+            -0.013146273791790009,
+            0.009898620657622814,
+            0.013294858857989311,
+            0.03597182780504227,
+            -0.034896351397037506,
+            0.012162779457867146,
+            -0.011101454496383667,
+            -0.008264180272817612,
+            -0.008533049374818802,
+            0.0044717141427099705,
+            0.010068432427942753,
+            0.014370334334671497,
+            -0.0023402206134051085,
+            -0.01175947580486536,
+            0.009070787578821182,
+            0.023547254502773285,
+            -0.012983537279069424,
+            0.03016991913318634,
+            0.027325568720698357,
+            0.023066120222210884,
+            0.01761798746883869,
+            -0.012537780217826366,
+            0.026462359353899956,
+            -6.484030018327758e-05,
+            0.0006261814851313829,
+            -0.018084971234202385,
+            -0.00587973790243268,
+            -0.004294826649129391,
+            -0.004362043924629688,
+            0.013316085562109947,
+            -0.0025471788831055164,
+            -0.0008468485320918262,
+            0.015099111013114452,
+            -0.0037323245778679848,
+            -3.927453508367762e-05,
+            -0.0059752571396529675,
+            0.015127412974834442,
+            0.0002699744363781065,
+            0.019457615911960602,
+            -0.04270770028233528,
+            0.019726485013961792,
+            -0.021396303549408913,
+            -0.003102605463936925,
+            -0.0020890405867248774,
+            0.009523618966341019,
+            0.00764861237257719,
+            -0.007142714224755764,
+            -0.014334957115352154,
+            0.004524780437350273,
+            0.014066088013350964,
+            -0.0006726144347339869,
+            0.005080206785351038,
+            -0.0027541371528059244,
+            0.0017228835495188832,
+            -0.010089659132063389,
+            -0.005862049292773008,
+            5.842812606715597e-05,
+            -0.010570792481303215,
+            -0.008646257221698761,
+            -0.03232087194919586,
+            0.01062032114714384,
+            0.010209942236542702,
+            0.007294837385416031,
+            0.004252373706549406,
+            -0.022018946707248688,
+            -0.012318440712988377,
+            0.012948159128427505,
+            -0.01132787112146616,
+            -0.002570174168795347,
+            -0.0020005968399345875,
+            0.005387990735471249,
+            0.0004705205501522869,
+            0.010160413570702076,
+            -0.003472300246357918,
+            0.010938717983663082,
+            -0.014603826217353344,
+            0.018084971234202385,
+            0.03229256719350815,
+            -0.00533846253529191,
+            -0.019967053085565567,
+            0.000907432462554425,
+            -0.009077862836420536,
+            0.003813692834228277,
+            0.007641536649316549,
+            -0.021962342783808708,
+            -0.02173592709004879,
+            0.005087282508611679,
+            -0.013153349049389362,
+            -0.007783046457916498,
+            0.014830241911113262,
+            0.005221716593950987,
+            0.010839661583304405,
+            0.01751893013715744,
+            0.010889190249145031,
+            0.00840569008141756,
+            -0.026533113792538643,
+            0.0006734988419339061,
+            -0.012686366215348244,
+            -0.00590803986415267,
+            0.002239394700154662,
+            -0.0017591454088687897,
+            0.004591997712850571,
+            -0.00010115749319083989,
+            -0.011200511828064919,
+            -0.0007057808106765151,
+            0.010903340764343739,
+            0.027919910848140717,
+            -0.004146241117268801,
+            -0.0022570835426449776,
+            -0.014257126487791538,
+            0.0010100272484123707,
+            0.036481261253356934,
+            -0.012275987304747105,
+            -0.017434025183320045,
+            0.01757553406059742,
+            0.027452928945422173,
+            0.005954030901193619,
+            -0.0032529598101973534,
+            -0.0027364485431462526,
+            0.00039600671152584255,
+            0.011978816241025925,
+            0.007719367276877165,
+            -0.002692226553335786,
+            -0.002131493529304862,
+            0.004761809483170509,
+            0.035773713141679764,
+            -0.0015203474322333932,
+            -0.010330225341022015,
+            -0.024240652099251747,
+            0.010974096134305,
+            0.00277182599529624,
+            0.004754733759909868,
+            0.002313687466084957,
+            0.023504801094532013,
+            0.019655730575323105,
+            0.014858543872833252,
+            0.00470520555973053,
+            -0.009771261364221573,
+            0.03005671128630638,
+            -0.008172199130058289,
+            -0.004542469047009945,
+            -0.02838689461350441,
+            -0.005087282508611679,
+            -0.015650998800992966,
+            0.008929276838898659,
+            -0.0002221042668679729,
+            -0.005016527138650417,
+            -0.013422218151390553,
+            0.019726485013961792,
+            0.01064862310886383,
+            -0.011073152534663677,
+            0.012608535587787628,
+            -0.027070851996541023,
+            0.02346234768629074,
+            -0.03826428949832916,
+            -0.0071993181481957436,
+            -0.006421013735234737,
+            -0.010514188557863235,
+            -0.02627839520573616,
+            -0.0030353881884366274,
+            0.007372668012976646,
+            0.014250051230192184,
+            0.0022570835426449776,
+            0.0022818478755652905,
+            0.02282555401325226,
+            0.0016415152931585908,
+            -0.01508495956659317,
+            0.017985913902521133,
+            0.03492465242743492,
+            0.012926933355629444,
+            0.00155837822239846,
+            0.015990622341632843,
+            0.0027789014857262373,
+            0.0021633331198245287,
+            0.00788917951285839,
+            -0.0034493047278374434,
+            -0.007938707247376442,
+            0.008582578040659428,
+            0.006990591064095497,
+            -0.02763689123094082,
+            0.003824306186288595,
+            -0.00010408719390397891,
+            -0.009898620657622814,
+            -0.022429324686527252,
+            0.011568437330424786,
+            0.01913214474916458,
+            -0.004224071744829416,
+            -0.006771250627934933,
+            -0.0043337419629096985,
+            0.0006836698739789426,
+            0.006785401608794928,
+            -0.002412744564935565,
+            -0.000515184598043561,
+            -0.0015176940942183137,
+            -0.02368876338005066,
+            -0.010316074825823307,
+            0.013570803217589855,
+            -0.005175726022571325,
+            0.010358527302742004,
+            0.005242943298071623,
+            -0.018792519345879555,
+            -0.016938740387558937,
+            0.006852618884295225,
+            -0.016188737004995346,
+            0.01364863384515047,
+            0.00027948213391937315,
+            -0.0023172253277152777,
+            0.024962352588772774,
+            0.03594352304935455,
+            -0.013188726268708706,
+            0.005119122099131346,
+            -0.019386861473321915,
+            -0.0245378240942955,
+            -0.019004784524440765,
+            0.022613288834691048,
+            0.008306633681058884,
+            -0.018792519345879555,
+            0.012304289266467094,
+            -0.002752368338406086,
+            -0.008865597657859325,
+            -0.0066721937619149685,
+            -0.006845543161034584,
+            -0.008547200821340084,
+            0.020533092319965363,
+            -0.004935159347951412,
+            0.0038950610905885696,
+            0.019047237932682037,
+            0.04075486212968826,
+            -0.008292482234537601,
+            -0.01349297259002924,
+            0.0054339817725121975,
+            0.004878554958850145,
+            0.0018626245437189937,
+            -0.004814875777810812,
+            -0.008016537874937057,
+            -0.0185944065451622,
+            -0.01740572229027748,
+            -0.009615600109100342,
+            -0.0020041344687342644,
+            0.020122714340686798,
+            -0.007768895477056503,
+            -0.01354957651346922,
+            -0.0245378240942955,
+            -0.028188779950141907,
+            -0.006580212153494358,
+            0.02319347858428955,
+            0.017179306596517563,
+            -0.00013200224202591926,
+            -4.3337418901501223e-05,
+            -0.012042495422065258,
+            -0.01968403346836567,
+            0.015509488992393017,
+            0.00037876018905080855,
+            -0.01773119531571865,
+            -0.0065448349341750145,
+            -0.015919867902994156,
+            0.005787756759673357,
+            -0.0051368107087910175,
+            0.004574308637529612,
+            0.05057565122842789,
+            0.0005271245026960969,
+            0.008441068232059479,
+            -0.026575567200779915,
+            0.009106164798140526,
+            -0.01583496294915676,
+            -0.017377421259880066,
+            -0.00012702727690339088,
+            -0.022301966324448586,
+            -0.0030955299735069275,
+            -0.018792519345879555,
+            -0.007386818993836641,
+            0.008568426594138145,
+            0.009799563325941563,
+            -0.013896276243031025,
+            -0.013669860549271107,
+            -0.024693485349416733,
+            -0.0035872769076377153,
+            -0.001539805089123547,
+            0.009311353787779808,
+            0.007825499400496483,
+            -0.03693409264087677,
+            0.002536565763875842,
+            -0.0009507699287496507,
+            0.0005779796629212797,
+            0.01208494883030653,
+            -0.008667483925819397,
+            -0.003063690150156617,
+            -0.0214670579880476,
+            0.006452853325754404,
+            0.007103798910975456,
+            -0.013500047847628593,
+            0.012375044636428356,
+            -0.012863253243267536,
+            -0.007669838611036539,
+            -0.0005426021525636315,
+            0.03379257395863533,
+            0.011794853024184704,
+            -0.0025153392925858498,
+            -0.00654129683971405,
+            -0.011150983162224293,
+            0.02654726430773735,
+            -0.011349096894264221,
+            -0.026405755430459976,
+            0.028910480439662933,
+            0.013245330192148685,
+            0.0005974372616037726,
+            0.020151015371084213,
+            -0.0017936384538188577,
+            -0.01397410687059164,
+            0.009601449593901634,
+            0.004266524687409401,
+            0.012424572370946407,
+            -0.00014460546663030982,
+            0.0214670579880476,
+            0.0018555490532889962,
+            0.0027399861719459295,
+            -0.0036226543597877026,
+            0.004687516950070858,
+            -0.03093407303094864,
+            -0.0006549257086589932,
+            -0.003102605463936925,
+            -0.008398614823818207,
+            0.011207587085664272,
+            -0.002421588869765401,
+            0.006558985915035009,
+            -0.022188758477568626,
+            -0.028485950082540512,
+            0.011342021636664867,
+            -0.016231190413236618,
+            -0.021396303549408913,
+            0.015339677222073078,
+            -0.0007641536649316549,
+            0.00614860700443387,
+            -0.007443422917276621,
+            -0.023278385400772095,
+            -0.019386861473321915,
+            0.02064630016684532,
+            0.012212308123707771,
+            -0.01190098561346531,
+            0.012750045396387577,
+            -0.021113283932209015,
+            -0.012948159128427505,
+            -0.001894464367069304,
+            0.0090566361322999,
+            -0.0019864458590745926,
+            0.01325240544974804,
+            -0.012198156677186489,
+            -0.00334494118578732,
+            0.0031592093873769045,
+            -0.011589664034545422,
+            0.008158047683537006,
+            0.01978308893740177,
+            0.008193425834178925,
+            -0.009339655749499798,
+            0.003422771580517292,
+            -0.011922212317585945,
+            0.013500047847628593,
+            0.015537790954113007,
+            0.018311386927962303,
+            -0.006498843897134066,
+            0.0020412809681147337,
+            -0.01370523776859045,
+            0.0063962494023144245,
+            -0.009827865287661552,
+            -0.004878554958850145,
+            -0.0005333155859261751,
+            0.004921008367091417,
+            -0.011179285123944283,
+            -0.0025772498920559883,
+            -0.026830283924937248,
+            -0.0013823752524331212,
+            -0.020533092319965363,
+            0.005303084850311279,
+            0.023207630962133408,
+            -0.007514177821576595,
+            0.0006045127520337701,
+            -0.0039092120714485645,
+            0.014476466923952103,
+            0.020688753575086594,
+            0.00870286114513874,
+            0.00482548912987113,
+            -0.006081389728933573,
+            0.02179253101348877,
+            0.011603815481066704,
+            -0.017816102132201195,
+            0.00164239970035851,
+            -0.001509734196588397,
+            -0.0014336726162582636,
+            0.011625041253864765,
+            0.004443412180989981,
+            -0.020844414830207825,
+            0.0028602697420865297,
+            -0.010089659132063389,
+            0.007818424142897129,
+            0.014702882617712021,
+            -0.00031906069489195943,
+            -0.02486329711973667,
+            -0.009827865287661552,
+            0.021919889375567436,
+            -0.015070809051394463,
+            -0.0008291598060168326,
+            0.00764861237257719,
+            0.023264234885573387,
+            -0.024792540818452835,
+            -0.006831392180174589,
+            -0.0023455272894352674,
+            -0.003933976404368877,
+            -0.005193414632230997,
+            0.0010489424457773566,
+            0.0214670579880476,
+            0.004415110219269991,
+            -0.004234685096889734,
+            0.005724077112972736,
+            0.006470541935414076,
+            0.02325008250772953,
+            0.0040577976033091545,
+            0.028047269210219383,
+            -0.00943871308118105,
+            -0.0023809047415852547,
+            -0.0019988277927041054,
+            -4.775960405822843e-05,
+            0.0160047747194767,
+            -0.005430444143712521,
+            0.020320827141404152,
+            0.009445788338780403,
+            0.0036934094969183207,
+            0.016514210030436516,
+            0.02319347858428955,
+            -0.006767712999135256,
+            -0.004793649073690176,
+            0.019160445779561996,
+            0.0053667644970119,
+            -0.008858522400259972,
+            -0.02194819226861,
+            0.01141277700662613,
+            0.0014124460285529494,
+            -0.0006956097786314785,
+            0.012106175534427166,
+            0.0024375086650252342,
+            0.006300730165094137,
+            0.008172199130058289,
+            -0.016684021800756454,
+            0.004241760354489088,
+            -0.004698129836469889,
+            -0.01860855706036091,
+            0.010910416021943092,
+            -0.012092024087905884,
+            -5.201595922699198e-05,
+            -0.0010002984199672937,
+            0.014165145345032215,
+            -0.01703779585659504,
+            -0.0019103841623291373,
+            -0.0070719593204557896,
+            -1.3024719009990804e-05,
+            -0.0009162768837995827,
+            0.008320784196257591,
+            -0.006254739128053188,
+            -0.010938717983663082,
+            -0.010938717983663082,
+            0.027509532868862152,
+            -0.004991763271391392,
+            0.003916287794709206,
+            0.019330257549881935
+          ],
+          "embedding": [
+            0.007204997251390758,
+            -0.03173324616100502,
+            -0.020505790884701024,
+            0.0037451608231038273,
+            0.009817085391002229,
+            0.026883177166072393,
+            -0.016477826654817607,
+            0.06949311853773875,
+            -0.01224876667472692,
+            0.031952566873802865,
+            0.013633671951171586,
+            -0.005041437355104989,
+            0.02663392750069342,
+            -0.022099132865267587,
+            0.006665702000169926,
+            0.0030467129715919033,
+            0.007931802508415542,
+            0.009984477355413895,
+            -0.010362836464183985,
+            -0.03180848181251013,
+            -0.013542377333474449,
+            -0.013572345022577053,
+            -0.03979985300694044,
+            -0.007698412130292187,
+            -0.005953452234356532,
+            0.018890228917932694,
+            0.02217446994660065,
+            0.005158357501784182,
+            -0.05468694715835175,
+            0.00789994048345594,
+            0.017130184174786147,
+            -0.0026591475669416114,
+            -0.02081141846859738,
+            -0.008403928318906045,
+            -0.030925346724945978,
+            -0.03855023691059801,
+            0.038853354785097736,
+            0.01612011653434478,
+            -0.035635702395196575,
+            0.026266623757525934,
+            0.043450373922427316,
+            -0.007254334835408145,
+            -0.015376804869420366,
+            -0.0017276202867766907,
+            0.004650414207094434,
+            -0.002365514361535186,
+            0.011906139905776183,
+            -0.0005664194302009289,
+            -0.0191723447249157,
+            -0.025546124518037998,
+            0.01025005257280377,
+            -0.014436532079436437,
+            0.029277529346694118,
+            -0.0045414983847733344,
+            -0.02260315456913697,
+            0.01717918656248626,
+            0.013666132349359846,
+            0.057804090189845364,
+            0.056600576093100735,
+            0.051208022172527734,
+            0.03553760896972431,
+            0.02094599510567085,
+            -0.02438674472832137,
+            -0.0027797634437333432,
+            0.03614248430646102,
+            -0.026729308357113124,
+            -0.002251394534452815,
+            0.018929251842788906,
+            0.00689697483318378,
+            -0.026603606434115974,
+            -0.0541443728344337,
+            0.027164630314707807,
+            -0.004335272389193875,
+            -0.013967321193992958,
+            0.003323121377857836,
+            -0.0245295956412367,
+            0.024489335129134787,
+            0.06212059558640156,
+            0.009214058207162371,
+            -0.002082293435905329,
+            0.013659934201796686,
+            -0.003344037927706573,
+            -0.00689041317919184,
+            0.020235785390418023,
+            0.0212739457082623,
+            0.0016782346585238381,
+            -0.013484221664059555,
+            -0.06465582506604173,
+            0.011882797526607326,
+            -0.032970655964890155,
+            -0.04027700439305653,
+            0.025396847912528105,
+            0.013861157517716707,
+            0.00604596157583048,
+            -0.00744611407420936,
+            0.04038084298991919,
+            0.021507532290328625,
+            0.005983822806623904,
+            -0.012711110369915742,
+            0.02410138142963901,
+            -0.018372791517880082,
+            0.021016674128298246,
+            0.013387285956415756,
+            0.02039383525127615,
+            0.010700292312227496,
+            0.0053934660497349735,
+            -0.014807678014449691,
+            0.009422805142740376,
+            0.0414357625490679,
+            0.12206398267002624,
+            -0.010877509834324155,
+            -0.011079302503786522,
+            -0.04107714853756596,
+            0.006631397986165343,
+            -0.031914267104222384,
+            -0.0179965295457466,
+            -0.057530036803978266,
+            -0.0320488164246567,
+            0.008208202363320509,
+            -0.0020203090915064162,
+            0.03502055780919601,
+            0.04464599164916666,
+            0.019332152079287082,
+            0.017133943087431935,
+            -0.039885565008919065,
+            0.04703059988373572,
+            -0.029370696964304416,
+            0.0008494086835971101,
+            0.0014082189719987074,
+            -0.030973762582071615,
+            -0.006561157852024518,
+            0.011634653995510726,
+            -0.019896826887411713,
+            -0.034275921680252924,
+            0.026274465530039866,
+            -0.007728234125236442,
+            0.04814514040690998,
+            0.06550407296280117,
+            -0.02749051703615343,
+            -0.010041099502187074,
+            0.02019601967763537,
+            -0.01871701432558228,
+            0.008532829759864928,
+            -0.020164138072967613,
+            -0.045196925901010855,
+            -0.004523517988516156,
+            -0.002113469149506399,
+            0.07929650663710812,
+            -0.01111269708042164,
+            0.03358254444720164,
+            -0.019973382759721527,
+            -0.0007145592863325417,
+            -0.0011740862928390678,
+            0.02166182480535788,
+            0.004543228023398081,
+            -0.013377688428761516,
+            -0.01758250073647576,
+            0.009548292014596834,
+            -0.04753617258004735,
+            -0.007125744019792479,
+            -0.009930358894299303,
+            -0.007931672044235996,
+            -0.024561936537429774,
+            0.012979195012881556,
+            0.009209145250061479,
+            0.04586756662401364,
+            0.04311807301838248,
+            -0.0036241105681042675,
+            -0.06579256720896123,
+            0.04967191755416882,
+            0.015475886231466817,
+            0.03157651656768371,
+            0.02515941109300714,
+            0.01773554572598468,
+            0.005824418191900522,
+            0.016317635547562007,
+            0.007650872352957098,
+            0.010015166465492963,
+            -0.033203307142867844,
+            0.040272188258443496,
+            -0.013408423788025047,
+            0.01036004294342104,
+            0.012391622448703885,
+            0.04214857862444935,
+            0.03418032388760535,
+            0.015263018079889338,
+            0.018786704580603317,
+            0.02704685343985817,
+            0.015348111464412869,
+            0.0013031600434754704,
+            0.03241427528588285,
+            0.011689924942195664,
+            0.04544406114742012,
+            0.016186615914968734,
+            0.011624500626103958,
+            -0.008272953078864385,
+            -0.02363080189189375,
+            0.02784298119317616,
+            -0.010585980875564109,
+            0.01455013604305903,
+            0.01749228557615982,
+            -0.054742056843690476,
+            -0.005876213780171779,
+            -0.010556605098730201,
+            0.003967456397644932,
+            0.014663888851767762,
+            0.0015548476559947807,
+            -0.003858367677774835,
+            0.012504286824484335,
+            -0.025806882498628832,
+            -0.014216299453397403,
+            0.0031620924516417956,
+            -0.006448390918604307,
+            -0.02785577502346934,
+            -0.00766402687921159,
+            0.005275000635711884,
+            -0.0090448751942,
+            0.019702662630533,
+            0.007009091837467867,
+            -0.01847307100043082,
+            -0.0006774105206839772,
+            0.032952656101827446,
+            0.015900033044760432,
+            -0.02170121261931376,
+            -0.017964879081469612,
+            -0.005062849745084668,
+            -0.03171847232054206,
+            -0.038177398521606544,
+            0.029426291950046455,
+            -0.025687201683643784,
+            0.030158867757101797,
+            0.015780411479509624,
+            -0.007011484049169482,
+            -0.012114023533118393,
+            0.020661407942701628,
+            -0.025067530380881376,
+            0.009777190971413269,
+            -0.004160771288889298,
+            -0.028825369485727344,
+            0.008307290529469729,
+            -0.014415903546305744,
+            -0.007561235629544945,
+            -0.03201704135271719,
+            -0.015182869388110948,
+            -0.00501561450194676,
+            -0.0003863746169672397,
+            -0.01073403445816663,
+            -0.03321103354401307,
+            -0.002304357889348034,
+            0.017001213960120047,
+            0.00028222146729658097,
+            -0.04002882852650754,
+            0.014509588052058002,
+            -0.004150541038999649,
+            -0.034333837380045466,
+            0.0564137596224324,
+            0.03584898691608747,
+            -0.006830698606679202,
+            0.006121456445207273,
+            -0.04505674213649008,
+            0.04358467937307492,
+            0.010847859073780811,
+            -0.04918983713702805,
+            0.049190442935189385,
+            -0.00528960443684769,
+            0.01419195371129309,
+            0.040316024601793406,
+            0.004600843996974165,
+            -0.006780970176765864,
+            0.005206651850992678,
+            0.03397517915308201,
+            -0.010037223444627569,
+            -0.027377913828275806,
+            -0.03328136706915221,
+            0.02562568440972934,
+            -0.03485770844248482,
+            -0.020069979230687374,
+            -0.0052041408068332524,
+            0.029818953377223505,
+            0.028789460147370508,
+            0.004071274130619537,
+            0.01928375754300958,
+            0.0014970947734202088,
+            0.010502599795027616,
+            0.016914145702515263,
+            0.011942576738188502,
+            -0.014909953528043828,
+            -0.003432489584767786,
+            -0.05733768990013957,
+            0.0030710780388681002,
+            -0.02627280695622545,
+            0.007930906480371135,
+            -0.031227917075485834,
+            0.013132797714872345,
+            0.002514351081823354,
+            -0.007152021475569949,
+            -0.0066245205132327055,
+            -0.005279224195782313,
+            0.028512607222275155,
+            -0.00921960825771865,
+            -0.0359704105589433,
+            -0.021347597388353182,
+            0.011200044645812517,
+            -0.0004171294784927543,
+            -0.016784468873218786,
+            -0.0018746544944553878,
+            0.007749103417240419,
+            0.02390272097796833,
+            -0.005128540108845805,
+            0.006397849195755672,
+            0.01214922062611769,
+            0.01163371266250103,
+            0.012307186722130844,
+            -0.004982589662882946,
+            -0.016343724553037307,
+            0.019806887143115643,
+            0.013139657463054087,
+            0.0205801609265216,
+            -0.03767676007629612,
+            -0.0249879011550972,
+            0.05547392586363707,
+            0.005903313549319051,
+            -0.05568806298774435,
+            0.018827693208840233,
+            -0.007931178941307645,
+            0.008183992600768387,
+            -0.010319022946041053,
+            0.05111352384595183,
+            -0.024897203290807753,
+            -0.02096678205188722,
+            0.006436523335212348,
+            -0.04917084523640266,
+            -0.010459284852442706,
+            -0.011138558192296276,
+            0.014789370043397917,
+            -0.03105725978752438,
+            -0.004934745004818259,
+            0.015595801375817122,
+            -4.687585727052446e-05,
+            0.0028886067775750254,
+            -0.00850352807499054,
+            -0.0034885579171087815,
+            0.016339681487281293,
+            -0.0034050317046826813,
+            -0.008309570448487812,
+            -0.003615165868022203,
+            -0.027638409919916757,
+            0.023352910161586186,
+            0.026364511529991978,
+            0.001249936082895606,
+            0.001987948967948376,
+            0.01756629246122769,
+            0.004656075485833687,
+            -0.006752579977941239,
+            -0.013681608638716132,
+            -0.03405936202918687,
+            -0.0401867651078737,
+            0.022991180183813725,
+            -0.02318908140203705,
+            0.018299512436825923,
+            0.00904081847012441,
+            0.019697091729447287,
+            -0.019236130984371835,
+            0.015159917756323988,
+            -0.0387297415849285,
+            -0.018793498297161176,
+            -0.005256288213225733,
+            -0.010414697711744128,
+            0.0007039894599589104,
+            -0.01739645696111503,
+            -0.013728845328097506,
+            0.022709522955937343,
+            -0.022789243441781803,
+            -0.0036763802589970026,
+            -0.01429671717544519,
+            0.0023674126354206932,
+            0.049448298344058644,
+            -0.009926033487593551,
+            -0.023371285193090733,
+            0.020846792916650956,
+            -0.026996519120163966,
+            0.011246908457705283,
+            0.04091365159640201,
+            0.005469162850197613,
+            0.005640919201612091,
+            0.029575753750229278,
+            -0.0007860452837141593,
+            -0.0052805237594885516,
+            -0.009782745443933828,
+            0.021178503581190618,
+            -0.0009176512672832309,
+            -0.05245650420332918,
+            -0.007698364493410058,
+            -0.039027418406703554,
+            0.005448840777995117,
+            0.017581625031764185,
+            0.025522517097715556,
+            -0.020775492472580457,
+            -0.015906320383873178,
+            -0.000643669839811819,
+            -0.0086896084429604,
+            0.0029399233482444638,
+            0.014381240985647907,
+            0.021086693776922936,
+            0.002436575081260609,
+            -0.004913514868101904,
+            -0.011582101006213432,
+            0.014400297663309155,
+            0.0467181039341492,
+            -0.01722692180137712,
+            0.0011622224560709315,
+            -0.013036540368441797,
+            -0.06208454638917413,
+            -0.019160212331432766,
+            0.0351496273293468,
+            0.02429814055116254,
+            -0.009896662169762772,
+            -0.01647101145477158,
+            -0.002915821099802562,
+            -0.03557863734177486,
+            -0.01786433999615293,
+            -0.010699364747311715,
+            0.02390901423299708,
+            0.007213809581960646,
+            0.0005601364043388203,
+            0.00916614601011734,
+            0.014737491634516238,
+            0.019303599465763203,
+            0.008864535607492504,
+            -0.0575839606069681,
+            0.018051837786406608,
+            0.006315830080225406,
+            0.02525619013082567,
+            -0.015671924543113065,
+            0.003988943108210209,
+            -0.03179878281917948,
+            -0.029440641244429775,
+            0.040586322748158446,
+            -0.03418722657593859,
+            -0.030650542761032924,
+            0.006566745611226466,
+            0.009571089937153852,
+            0.004491158558315032,
+            -0.011352813042040581,
+            -0.02535505987721779,
+            -0.013381948772812133,
+            -0.030918676056735445,
+            -0.02192708443081135,
+            -0.005232667170255381,
+            -0.03342408726436631,
+            -0.07498634115550039,
+            -0.006245638452568239,
+            -0.049946896465756174,
+            -0.03900385398225758,
+            0.04458524125677807,
+            -0.006106489377959481,
+            0.02398410968928338,
+            -0.04615183642824994,
+            -0.01165382599292015,
+            -0.04890013373371541,
+            0.013624324077017377,
+            0.01476398305411351,
+            0.020202554298470666,
+            0.00014553149242532853,
+            0.036224316512987846,
+            0.00021498496267774312,
+            -0.05055208658013767,
+            -0.057482169168904106,
+            -0.003960451575823462,
+            -0.010217078509789989,
+            -0.005593738972150311,
+            -0.034090261748104665,
+            0.05835341009022349,
+            0.0116359661127526,
+            0.04203678184748103,
+            0.03447818335186531,
+            -0.006747564131379555,
+            -0.025756962754841885,
+            -0.028123416424745064,
+            -0.008073625379614702,
+            0.047544241458556825,
+            -0.029605671083240564,
+            0.008171081332993893,
+            -0.026007452205474395,
+            0.0067084589781962805,
+            -0.0070645448617940995,
+            -0.0012298101311792947,
+            0.012372286141738867,
+            0.006377319686763927,
+            -0.015024537877377194,
+            0.007934745664790507,
+            0.026041755081605937,
+            -0.007303731324541215,
+            0.0407000170257006,
+            -0.012990703097937777,
+            -0.033189084013863494,
+            -0.022525679964848666,
+            0.01657717698191603,
+            0.03440284885162039,
+            0.010230633738702899,
+            -0.008707808754455952,
+            -0.0363777020540216,
+            0.011184076269109602,
+            0.031774012431843524,
+            0.01504703222643543,
+            -0.022509833303978505,
+            0.025230974050270107,
+            0.017502662694813216,
+            -0.018700579139537456,
+            -0.03511734747709348,
+            0.015162183119327901,
+            -0.017711638132862063,
+            0.018190335050681287,
+            -0.01590174153215615,
+            0.011103110869700727,
+            0.0025189370696524936,
+            -0.00920305384736575,
+            0.01929652151514404,
+            -0.006196715938358087,
+            0.011426611271994685,
+            0.007058186102245829,
+            -0.018191352599117545,
+            0.0027202185072220937,
+            0.057310566886380025,
+            -0.000531639834386418,
+            0.01494975681909879,
+            0.03845692752400366,
+            -0.001971953300066856,
+            0.045554909803324765,
+            -0.02532332999916258,
+            -0.042616337063845953,
+            0.02633963974172818,
+            -0.02970705577683783,
+            -0.01624154998005974,
+            -0.02620993449678864,
+            -0.021490817420853802,
+            -0.004038424045768319,
+            0.009940887982818171,
+            -0.02692933845395842,
+            -0.023801570856681796,
+            -0.001948460540662095,
+            0.02290911497874141,
+            -0.03300532378029689,
+            0.006525488380791647,
+            0.007039306647666766,
+            -0.003685585408710752,
+            0.03183735852538168,
+            -0.027197064244321705,
+            0.006861928286688311,
+            0.011209869849682412,
+            -0.001094897323854524,
+            -0.005328676992017907,
+            -0.004007474089818478,
+            0.010796946231051235,
+            -0.030543173497906445,
+            -0.012879972141706173,
+            0.00935894527375688,
+            0.0017573071057611829,
+            -0.010050882212831222,
+            -0.03102132152986303,
+            -0.002915141291985554,
+            0.007634137769426993,
+            -0.011094093612593696,
+            0.01199609810193786,
+            -0.010458016758814742,
+            -0.04929347607956551,
+            0.010973132159729983,
+            0.028148117094907874,
+            -0.025504019123994758,
+            0.012470127357818321,
+            0.009182729379865986,
+            -0.016020133756226275,
+            0.016320111225714855,
+            -0.012051271446461976,
+            0.024098195821827578,
+            0.04317620443481784,
+            0.024592886889234396,
+            0.007727787037954994,
+            0.00771746776156926,
+            -0.0003147538723468374,
+            0.022572868970938624,
+            -0.016683340264320543,
+            0.013672085891597506,
+            0.020827021901755415,
+            0.02092309219721837,
+            0.0057130785714458055,
+            -0.003901728981013356,
+            -0.0036968415923910314,
+            0.002227024825579113,
+            -0.02106678364798051,
+            0.014578546755457675,
+            0.03128101864424301,
+            -0.005234627629617124,
+            -0.022394744692442837,
+            0.002430617645746389,
+            -0.015780389432069438,
+            4.7277885648917e-05,
+            0.013357803020920043,
+            0.013777651543690907,
+            -0.027713499926132385,
+            -0.0018999112228756628,
+            0.028364630573666852,
+            0.009146793705085586,
+            -0.009891297866910313,
+            -0.01644269527910728,
+            -0.0003903768506971414,
+            0.006512217314869576,
+            0.02408785204739448,
+            0.0018406499545367782,
+            -0.007360473750031044,
+            0.01484497577088404,
+            0.016723277095327158,
+            0.003629911330658209,
+            -0.0004820567356447713,
+            0.02394406451074397,
+            0.023794019615473026,
+            0.001632266615911801,
+            -0.02064379152040184,
+            -0.02755693521684878,
+            0.027335938400284726,
+            0.02769248863677192,
+            -0.006832353240623308,
+            0.027324973360715134,
+            0.003937207909967385,
+            0.013580154030133622,
+            0.02076818768224654,
+            -0.01784151382870951,
+            -0.017617903734311997,
+            0.0017316476451235418,
+            -0.009605294810027804,
+            -0.009829196696575069,
+            0.004891780008779742,
+            -0.011655402383771472,
+            -0.01124379942906224,
+            0.043205454954628145,
+            0.004713701851783307,
+            0.003560551284131962,
+            0.007894840447501912,
+            -0.018881214526749082,
+            -0.00945538175144614,
+            0.0019040259741796817,
+            0.029505863720301916,
+            -0.001400002868254808,
+            0.03315866433970851,
+            -0.00830542045504805,
+            0.008602408964961391,
+            -0.03202191233287069,
+            -0.006978289755114341,
+            0.0326499698728755,
+            0.015314305362524091,
+            0.015104436300213667,
+            0.0021257042489266787,
+            -0.0010840650756005028,
+            0.02434348355847526,
+            -0.007174433778817674,
+            -0.04839373063710409,
+            0.022696994450018916,
+            -0.004779511451775743,
+            0.023588216956749154,
+            0.0382733980620746,
+            -0.004232286553944853,
+            0.003815507826545929,
+            0.023531223314846965,
+            -0.01887650634213847,
+            0.015055403379782985,
+            -0.01918944132635701,
+            -0.017597900721414636,
+            0.02176630010543207,
+            -0.008896608357798277,
+            0.01732387427588561,
+            0.01764363756727683,
+            -0.012180043444768312,
+            -0.019003360499321047,
+            0.01229139850966224,
+            0.0025710091163655626,
+            -0.017075658234265562,
+            0.01797412486853335,
+            0.018372010308966135,
+            0.006723179570342314,
+            0.015417677746651683,
+            -0.00047634684109332134,
+            0.017608296628666153,
+            -0.001049984114818522,
+            -0.008416990841821452,
+            0.01827827907328822,
+            -0.009568717277044265,
+            -0.01658089563620308,
+            0.009808478422454319,
+            -0.05193977747210182,
+            0.010644335931025957,
+            0.0016781339270411392,
+            -0.01562285100294917,
+            0.00405036330963539,
+            0.007102724536527262,
+            0.005195069570261247,
+            0.0028773630901681767,
+            0.004610474981005765,
+            0.042921486858787584,
+            0.013213672397112759,
+            -0.0064030584930089415,
+            -0.010711457508461704,
+            -0.017480309425763254,
+            -0.027151012983864797,
+            -0.005204425551873271,
+            0.009927884476991038,
+            0.022601230152836282,
+            0.008046297278661014,
+            -0.0031858480011267704,
+            -0.02589628103025927,
+            0.019723430185137954,
+            0.03604835658149313,
+            0.062423099201768716,
+            0.010930268386637612,
+            0.02925564269324381,
+            0.02439517070384333,
+            0.023930432096006114,
+            0.013036828091946605,
+            0.006910311087503408,
+            -0.032995841694293995,
+            0.012813650252889403,
+            0.01333714655191133,
+            0.03678966264802025,
+            0.012571138353467183,
+            -0.05610151838497368,
+            3.0777966587107344e-05,
+            0.009791774904454618,
+            0.041959488802722524,
+            0.01956296954085397,
+            0.000155217569370113,
+            -0.007285602454264675,
+            0.031547870838711856,
+            -0.026518385135879206,
+            -0.012299593280363843,
+            -0.0026551378470968594,
+            -0.006190117573235779,
+            -0.003807907001161504,
+            -0.004991957475076049,
+            0.036983035940964804,
+            -0.015648565314743963,
+            -0.019256463332683493,
+            0.0004300362961407231,
+            -0.0015686065121077472,
+            0.0011311524136286674,
+            0.0032236641835276464,
+            0.013795345278976164,
+            0.015385029265915279,
+            -0.004800448639353721,
+            0.026514926384796363,
+            -0.008699453553651681,
+            0.022186817580012775,
+            -0.013427553742493343,
+            -0.011927259975371873,
+            0.017776228476138103,
+            -0.021243797085501972,
+            -0.01667280589187595,
+            -0.003688045116580565,
+            0.006189970874536939,
+            0.0020911968266404467,
+            0.0006097761986763103,
+            -0.004685468039463817,
+            -0.01011357666327168,
+            0.022631082517271542,
+            0.012398000826637277,
+            -0.006721588925828951,
+            0.018142966317821054,
+            -0.0021330638043572654,
+            -0.011869184647489572,
+            0.025073455250141923,
+            -0.0015392945266926425,
+            -0.0337085890598252,
+            0.01267302113143573,
+            -0.013438601165482187,
+            0.02673290895281225,
+            -0.009586762706902736,
+            -0.011208527168019762,
+            0.019144928964011168,
+            -0.0017591316941248897,
+            -0.005459150067129732,
+            -0.029230079356228025,
+            0.004679497471366242,
+            -0.011723195942933286,
+            0.015899377439148873,
+            0.0022054234777254627,
+            0.029187759252864872,
+            -0.018290694895057157,
+            0.00792533824593637,
+            0.011719198071061566,
+            0.014612375997487872,
+            0.005419943379355563,
+            0.010515265723144554,
+            -0.006893314445458116,
+            -0.04424495426064596,
+            5.8241472764594054e-05,
+            -0.005719143565673274,
+            0.0038220021965914754,
+            0.013106940982933836,
+            -0.01203955335201393,
+            0.011260626096963215,
+            -0.0279896016489392,
+            -0.0295228261396555,
+            0.036474811902902415,
+            -0.04900747926861378,
+            0.019261089742811332,
+            0.00014104646541594166,
+            -0.03057197731717424,
+            -0.04331886560073797,
+            -0.03232957159604667,
+            -0.019401402365464362,
+            0.004102283356333861,
+            0.0002610070704924167,
+            0.0012280372705500212,
+            -0.014025012323929194,
+            -0.014135990809256075,
+            0.013269444858490211,
+            0.00992155580941576,
+            0.011225162609266308,
+            -0.006545044517282357,
+            0.002368862850663672,
+            -0.02063183105457681,
+            -0.026981383275694142,
+            -0.02774194980145225,
+            0.013337303400678109,
+            -0.0025902548982556788,
+            -0.02794339093668326,
+            -0.025606606546251152,
+            -0.0029852498102770587,
+            -0.01906873442473352,
+            0.006508845592362198,
+            0.03268861529625085,
+            0.0355975143313428,
+            0.010066906387952277,
+            0.01690250467284985,
+            -0.017201544758038183,
+            0.048345309114140714,
+            -0.019031171948182872,
+            0.03188329748476941,
+            -0.0005845169448621133,
+            -0.024140202991373587,
+            0.00011597263700677706,
+            0.03203623528903764,
+            -0.019285565857567896,
+            -0.01332755507465857,
+            -0.010914187315481855,
+            -0.003374912271519852,
+            0.02419326154642231,
+            0.02443825785057301,
+            -0.0037923303809811746,
+            0.00022800399604730175,
+            0.015893265651968885,
+            0.003154527829977343,
+            0.007688529734165207,
+            0.030623379985627763,
+            0.014052282574575454,
+            0.01127235345371133,
+            -0.04164707563955819,
+            -0.011811587719246128,
+            -0.003763086694358232,
+            0.005909410979841885,
+            0.002913220159138154,
+            -0.002671613084376487,
+            0.0045765194162907885,
+            -0.007241703279856166,
+            -0.005855949422953439,
+            0.0016973541436777935,
+            -0.001576326424468957,
+            0.008393873711385013,
+            -0.0037217094191171083,
+            -0.03498879678492834,
+            -0.026096985039773694,
+            -0.021280075788049788,
+            0.006437497348154639,
+            0.0070214556591898375,
+            -0.00972721128392027,
+            0.01949553762684561,
+            -0.0029404319702066146,
+            0.000271037702827666,
+            -0.010734867909488861,
+            -0.0029784341515646075,
+            -0.01206464664516104,
+            -0.019544778160755956,
+            0.00035139045159739705,
+            0.013457601448003152,
+            0.00025947376767512006,
+            -0.03165522602526639,
+            0.011979811705146604,
+            0.006642849985850598,
+            0.013181162886675883,
+            -0.024499404540711422,
+            -0.02310714308893576,
+            -0.028028912907092744,
+            0.010570122662701183,
+            -0.0066866321204157725,
+            -0.024617571284817428,
+            -0.004097807033448707,
+            -0.0074576460628928256,
+            -0.0020417256533259887,
+            -0.02470953739335365,
+            -0.011408461668188081,
+            0.023537349682260345,
+            -0.006373652338136973,
+            0.01936338676879376,
+            -0.017636435070085086,
+            -0.017036091876814843,
+            -0.002198685044404603,
+            0.015584758061011048,
+            -0.038943923931820196,
+            -0.002076529978164001,
+            -0.008452513321945455,
+            0.006551055965075869,
+            -0.028840587857625097,
+            -0.007128279504360441,
+            0.0199397499488538,
+            0.012424319366271584,
+            -0.03518875279748495,
+            0.013596027106451115,
+            0.022191403935861113,
+            0.01218507466930854,
+            -0.0038551718784437783,
+            0.03910075115429626,
+            -0.0025215641203030336,
+            -0.005422688956956598,
+            0.016759888275451905,
+            0.04462538468871483,
+            0.023708097134660096,
+            -0.03152743757744915,
+            -0.008668774531128668,
+            -0.015224825785549947,
+            0.004102648962663789,
+            -0.025867795193966987,
+            0.004676262922216584,
+            -0.011739753544166856,
+            0.01645951757031186,
+            0.011253400919230348,
+            0.013958707917471253,
+            0.0376596076542735,
+            -0.005077354977296842,
+            0.02668910096392324,
+            0.0010332284250006117,
+            -0.00856723931523911,
+            0.0014431148305642308,
+            -0.01412579094241589,
+            0.029892393553325205,
+            0.02269599036836208,
+            0.014421409701746036,
+            0.029248195108770955,
+            -0.03588867295960255,
+            0.02458655544632528,
+            -0.0010013539341895672,
+            0.008368648583431747,
+            0.018150734970019018,
+            0.02628022565029391,
+            0.015423867373844644,
+            0.012477080028047823,
+            -0.02513018750675568,
+            -0.022180001014682045,
+            -0.0027550350630288997,
+            -0.031944843677988374,
+            -0.0014668093006914267,
+            -0.009789309083948147,
+            0.026386027599666254,
+            0.0046988411391149,
+            -0.019803160322638583,
+            -0.003931806845108628,
+            0.004482097428470392,
+            -0.00244660100774939,
+            -0.010441732346892935,
+            -0.004649318716095894,
+            -0.01869464992812615,
+            -0.017838643679294407,
+            0.023677878122153204,
+            -0.023958455640817666,
+            -0.025238725541146534,
+            0.014582798102040157,
+            0.0013223590277629197,
+            -0.002295641913710489,
+            0.0008238518461567685,
+            -0.02115533275282518,
+            -0.04214275933987015,
+            -0.027817083427786927,
+            0.0037588803464689412,
+            0.03565056002940463,
+            -0.026788865866928923,
+            0.014162550819567465,
+            -0.011336583458519,
+            0.019175861037658344,
+            -0.005127229083828615,
+            0.015531116410874209,
+            -0.018251290727445846,
+            -0.044630374763679266,
+            0.021833307809708984,
+            -0.012087009688208446,
+            0.02720742725750304,
+            0.021497809277370574,
+            -0.026937653159100156,
+            -0.013572733244188407,
+            0.01358214612023832,
+            0.0020408253706382332,
+            -0.009652962552750464,
+            -0.0063411929391088626,
+            -0.010533309167065193,
+            0.003901271588567942,
+            -0.04496345296706673,
+            -0.008761188928422701,
+            0.06948704056954039,
+            0.005916180375965137,
+            0.03157905519638486,
+            0.014668818318138927,
+            -0.028448496686381258,
+            -0.04008949536631331,
+            -0.00462253589823598,
+            -0.0003701522833020314,
+            -0.00024410762276269235,
+            -0.01850585249309217,
+            0.017699615637510956,
+            -0.002712069912576265,
+            -0.009767338815933488,
+            -0.013119464222297351,
+            -0.013743610418034631,
+            -0.030375365173543157,
+            -0.0030316327877227783,
+            -0.008632689470358149,
+            -0.006565457264917239,
+            -0.0328089275944709,
+            -0.011713996290006943,
+            0.029773910045828737,
+            0.02740100904353622,
+            -0.0061926915497348915,
+            0.00638737826122608,
+            -0.010244154924811975,
+            0.0010396203272677069,
+            -0.001264454445070139,
+            -0.004059969507786729,
+            0.008715730726831364,
+            0.0014767015415289076,
+            0.031074767756160294,
+            0.004761858344125113,
+            0.00987870172044607,
+            -0.059926117793699066,
+            -0.009294021263133636,
+            0.021188432871397475,
+            0.016602826531924377,
+            -0.009125784136292983,
+            0.013946188378030194,
+            -0.0017670794602549428,
+            -0.006837334179368295,
+            -0.028290055353601393,
+            0.024556541126233806,
+            0.04543639597357669,
+            0.0031311343860416473,
+            -0.011199639773247286,
+            0.014965677855595444,
+            0.0018248533506196013,
+            0.015507721118783542,
+            -0.011409402378400194,
+            -0.01338737990904486,
+            0.03063410115541065,
+            0.009305362935335817,
+            0.0025578631174065385,
+            0.012727648343848194,
+            -0.0038680554900795763,
+            -0.01173215289303824,
+            -0.01568862495293563,
+            -0.004101703129244437,
+            -0.03649200408657064,
+            -0.0022107849490954137,
+            0.009659780679763728,
+            0.0034091693454195806,
+            0.008848380026939594,
+            0.01842435611499587,
+            0.03546926214224266,
+            0.013478278853234764,
+            0.005970293629431872,
+            -0.0069197533761813455,
+            0.023196445250565058,
+            0.010403914879205351,
+            0.010017272564548626,
+            0.015076807790462658,
+            -0.029649260409273275,
+            -0.012737152920662164,
+            0.00373276777994811,
+            0.013239909450830258,
+            0.0001682544254742693,
+            0.02030652051051447,
+            0.014143961923579586,
+            0.0010147733405793957,
+            -0.012509446388074923,
+            0.0020437769941487626,
+            0.007405578085797006,
+            0.013458276946120546,
+            0.0006779114540819997,
+            -0.004250745498730309,
+            0.0009149988500358458,
+            -0.008355906072311008,
+            0.040391939952386505,
+            -0.01831218885700067,
+            -0.016793844108223176,
+            0.035064882870542116,
+            0.11416006426189942,
+            0.0010630856112440689,
+            -0.015660014600661857,
+            -0.017142534874931403,
+            -0.0086899013413391,
+            0.016611825572132997,
+            0.01609650977226868,
+            0.009780106186629793,
+            0.009622367309076686,
+            -0.000672926017804395,
+            0.024310949465350565,
+            0.02944698490225121,
+            -0.0037169531700178868,
+            -0.010990838766452194,
+            -0.036819646475041955,
+            0.022923962979125287,
+            -0.004596366586768123,
+            -0.007108041562635549,
+            -0.002604794313552557,
+            0.012641223813290853,
+            0.013668775645501733,
+            0.005125100040230003,
+            0.0077993841846437815,
+            -0.011500851674334447,
+            -0.017334458072700107,
+            0.0056819751688881774,
+            -0.015982234644378358,
+            -0.0076533013358874935,
+            -0.013127172654362694,
+            -0.011837346554842119,
+            -0.006562312362371076,
+            0.0029877420008976425,
+            -0.008572723854050838,
+            0.00854083969164562,
+            -0.010939639783598493,
+            0.016185642759084087,
+            -0.008052282332943615,
+            -0.0011992212936609377,
+            0.0010378646897070758,
+            -0.0025399515501741718,
+            0.010712865639618352,
+            0.01451357949543628,
+            -0.02521401163091652,
+            0.020836357346214075,
+            -0.008252813367155418,
+            0.0006409807249997337,
+            0.015338809580802425,
+            -0.04672526649678653,
+            0.022728876784937357,
+            0.005129225753457052,
+            -0.018372222092794564,
+            0.0066182736895667805,
+            -0.02048375780196863,
+            -0.00896382274306482,
+            0.0021738974344880794,
+            0.004491167150658828,
+            0.0034280799511763956,
+            0.015611133773752265,
+            0.018126362497548308,
+            0.00014383671280220997,
+            0.007927708693242186,
+            -0.006973076292062532,
+            0.008118313055163714,
+            -0.006791836664375275,
+            -0.01164567445668625,
+            -0.010894577039363014,
+            0.00014731542306100506,
+            -0.008865132658571428,
+            -0.011415696755798914,
+            -0.017922318360382588,
+            0.00358538001727405,
+            0.00289327637128695,
+            0.010406054711003452,
+            -0.007757607207626101,
+            -0.010994775479337082,
+            -0.011716096880261664,
+            -0.0013333474316266438,
+            -0.01231144419953554,
+            0.009792603083441917,
+            -0.02884953407490196,
+            0.0029097409399144306,
+            0.025277274938032477,
+            0.009263530315933738,
+            -0.028037993274175277,
+            -0.02532252134134532,
+            -0.0018596440554143638,
+            -0.013847737486669237,
+            0.009910107581119272,
+            0.013952962280235523,
+            -0.011807495289410774,
+            0.003896325037406752,
+            -0.02144921729633576,
+            -0.01628545945817568,
+            -0.0040148671532777005,
+            -0.016032493124255323,
+            0.004580041957001551,
+            -0.011375180364988195,
+            -0.007789996450583554,
+            -0.022041665680945066,
+            -0.005890065942573431,
+            -0.050455058269193866,
+            0.014401045842891128,
+            0.030219548461217365,
+            -0.004580747668649223,
+            -0.028684680969792875,
+            -0.015525870112797619,
+            0.011121401958547706,
+            -0.008879749268609245,
+            -0.019454243014778435,
+            -0.007839434838464604,
+            0.018908534797107714,
+            -0.0019283067432864216,
+            0.0024769961730655835,
+            -0.006732653593696035,
+            0.027604805021030646,
+            0.01366593691107021,
+            0.03740746059041736,
+            -0.014662268336336578,
+            0.02539666271785517,
+            0.02371809783638648,
+            -0.036126848124425154,
+            -0.007627445211958312,
+            0.03657481892016626,
+            0.01092515982671624,
+            0.0011046573579274704,
+            0.012350047525582187,
+            -0.006340802546498992,
+            -0.031330519599469066,
+            -0.0013691360532301728,
+            -0.0014676630169621005,
+            -0.023111265345539388,
+            0.027359543575867225,
+            0.018091458372134273,
+            0.014237156751820934,
+            0.0030993618578507172,
+            -0.0014037082734049385,
+            0.013257269860435359,
+            0.01021315143964527,
+            0.030885417052806473,
+            -0.003225065956381245,
+            -0.012825341172770057,
+            0.013476039084668826,
+            0.01852164843904835,
+            -0.009965495459579544,
+            0.012541079912270015,
+            -0.00824122613985912,
+            -0.018124134280975132,
+            0.03192263766222747,
+            0.0001484308749572035,
+            0.010888733498667343,
+            -0.010649515858317311,
+            -0.013082335214246402,
+            0.032614733822945205,
+            -0.00999075761269065,
+            0.004257643407198955,
+            -0.030723533031261387,
+            -0.003820776505926347,
+            -0.010471792928758869,
+            0.01737608180148319,
+            0.004597677757231315,
+            0.02095148515295301,
+            0.029253627700524396,
+            -0.011604450820008121,
+            -0.01996325509217509,
+            -0.001158626158150122,
+            -0.020583793738536985,
+            0.004168757459099133,
+            -0.03632338333261702,
+            -0.019373660230711246,
+            0.022259216603413172,
+            -0.010364620831688015,
+            -0.004751193403783363,
+            0.00444187509644351,
+            0.0008572202661850934,
+            0.014943779567410408,
+            -0.025408263453313398,
+            0.017755758370543148,
+            0.004627392860417132,
+            0.007402073071525438,
+            -0.024913329140362466,
+            0.013760771254234878,
+            -0.006087346663657887,
+            -0.03780469033982951,
+            0.005509388120163784,
+            0.03756141956381493,
+            0.02381019168406692,
+            -0.024580249929652097,
+            0.01807584361080568,
+            -0.03111039245576191,
+            0.023245542501125073,
+            -0.004868090951368273,
+            5.1544460403943004e-05,
+            0.00640235757127173,
+            -0.004698373089469779,
+            -0.00621383035264185,
+            -0.0025132415458431587,
+            -0.008746634331769936,
+            -0.01611801559493664,
+            -0.01193535623883546,
+            -0.03381789115349148,
+            -0.024119754887378317,
+            0.013881102852385332,
+            0.0019232668966222687,
+            -0.0095263729740736,
+            0.018652966857927805,
+            -0.03775912497511837,
+            -0.012022352025516115,
+            -0.023289377270709736,
+            0.023579672990954228,
+            -0.028605627815260902,
+            0.019235397553197357,
+            -0.025838655372560505,
+            -0.021458526449310157,
+            0.01508157885177254,
+            -0.010064396230734452,
+            -0.0016708624560165368,
+            -0.015042071642751917,
+            -0.03670785416972153,
+            -0.006267614265977843,
+            0.02298763967146357,
+            -0.012506444728541897,
+            0.00029871598584066644,
+            0.02671390618094565,
+            -0.003059865833779805,
+            -0.011103452976820622,
+            -0.0017382684168257506,
+            0.010923055147701782,
+            -0.02938686260440486,
+            0.014300097108092281,
+            0.0038571851420598074,
+            -0.0037614499297117365,
+            0.0010991440540809893,
+            -0.010108413486243776,
+            0.02127952161226619,
+            0.009142545227414532,
+            -0.0030260119510809205,
+            0.040589877142650374,
+            0.008677453889680373,
+            0.0017339508605300814,
+            0.008732950895530873,
+            0.01063352447960556,
+            0.006618635029823979,
+            -0.014986307792626314,
+            0.016905861015681957,
+            0.013101237900847744,
+            0.0002162791516784468,
+            0.019309343995079022,
+            0.0021263376627204313,
+            -0.024516267132902416,
+            0.013339486637573757,
+            -0.02311659751478777,
+            0.0020575982953961246,
+            0.02391695586495566,
+            -0.020440172706421535,
+            -0.001656132876701879,
+            -0.006304133667169861,
+            0.000543457661904495,
+            0.022135707414160034,
+            -0.0230903871986617,
+            -0.023444763940153112,
+            0.009604600033024832,
+            0.01721387169484476,
+            0.01827905276638783,
+            0.02824773106648473,
+            -0.007100625424382998,
+            0.01348510178676837,
+            0.0002604386361122316,
+            -0.011802178189607182,
+            0.007664157334475143,
+            0.0074175377458313594,
+            -0.0044008460682833875,
+            0.025491035291032547,
+            -0.0017015687016376205,
+            -0.006134010425966364,
+            -0.005379657499404915,
+            -0.01584573519366957,
+            0.0034386456145185213,
+            -0.013860742817569676,
+            -0.013355238174583658,
+            0.0040342831827141845,
+            -0.0208160950322342,
+            0.10904592104153767,
+            -0.0037038089210379766,
+            -0.019472012964045247,
+            0.0063554319446707044,
+            0.003716719884684827,
+            -0.003692786470980806,
+            -0.003353843210051379,
+            0.007788605566526459,
+            -0.011254480693190834,
+            -0.0045751956417398095,
+            -0.013976039724381875,
+            -0.001989262807706565,
+            0.010067374586433014,
+            0.009201983771978006,
+            -0.005902980676079493,
+            -0.02029714318363332,
+            0.013031676901335606,
+            -0.016970144750162643,
+            0.008874670945038486,
+            0.0008311255074953139,
+            0.00884167159995975,
+            0.001716603195908536,
+            0.01854951250316076,
+            -0.009310881405486074,
+            -0.01325242738760595,
+            0.029667589863813217,
+            -0.005616511284128333,
+            -0.0038179509972925954,
+            -0.0004690716809300012,
+            -0.005068974756744239,
+            0.0021204231125051274,
+            -0.027958274497127258,
+            0.022127088236172387,
+            -0.017271838492631593,
+            0.010267709087809265,
+            0.01316800560257366,
+            0.005599367128954229,
+            0.01817806970002991,
+            -0.007115594036834107,
+            -0.02627643123342057,
+            0.006542742528003751,
+            -0.003433874053082844,
+            0.006631427284537971,
+            0.02316295746074422,
+            -0.014573351469966297,
+            -0.033520864104177005,
+            -0.013707022143307006,
+            -0.01877574599842729,
+            -0.005992004412448576,
+            -0.0043698881826095435,
+            -0.01481352026460129,
+            0.006424121402963946,
+            0.01382938882387991,
+            0.018136021761515388,
+            0.0012697483790911433,
+            -0.011531136974861559,
+            0.00609187274277241,
+            0.01076133059240578,
+            0.008400887242930113,
+            -0.01533504932731406,
+            0.011439227832642668,
+            -0.028718487721012004,
+            0.01522697877548034,
+            -0.012452665599305595,
+            -0.024938748080573947,
+            -0.0022082450217489366,
+            0.017589399637694264,
+            0.0037824832255756763,
+            -0.004355445812154662,
+            -0.018778668474337727,
+            0.019825087149714933,
+            0.024692878851503508,
+            0.003883497401229232,
+            0.019470881143244197,
+            0.041834635328088364,
+            -0.003544509491805351,
+            0.027139164223766474,
+            -0.02129723143323512,
+            0.01461354860170749,
+            -0.03441729954693155,
+            0.001893713808284104,
+            -0.004222657393633789,
+            0.0013677766871517654,
+            -0.003811928806805752,
+            0.020142937759360904,
+            -0.009158202577174012,
+            0.0030638058587375543,
+            0.024854053046139163,
+            -0.018665378030958756,
+            0.00805903734779081,
+            0.010625429315145887,
+            -0.012856237251902233,
+            0.028535299711292306,
+            -0.006335094045840286,
+            0.00305422126851784,
+            -0.0297883488914759,
+            -0.0029192872993071216,
+            0.0026067044379754047,
+            -0.006097749614529602,
+            -0.00968023644800927,
+            -0.007433916539502931,
+            -0.006202274026852405,
+            0.0038354607448269555,
+            -0.021471361566683354,
+            0.003195646827154078,
+            -0.004390751282171024,
+            -0.003323447423321121,
+            0.02518759723554948,
+            0.025857746688368054,
+            0.016893707217894103,
+            0.012739170140289531,
+            0.008553625305650129,
+            0.019805351588748434,
+            0.01654964927067909,
+            0.011549329129728739,
+            -0.020026013396046632,
+            0.01150644479272541,
+            -0.004159611921476011,
+            -0.0022937297602724912,
+            0.014815466072709131,
+            -0.01572171840394285,
+            -0.02276187015864222,
+            0.024693009795978448,
+            -0.019725315346836404,
+            0.00998768979020729,
+            -1.1702210281215188e-06,
+            -0.011129745730507545,
+            -0.002591293237053776,
+            -0.018034357254990988,
+            0.00012070821244376275,
+            -0.023051045150566527,
+            -0.028465270110388716,
+            -0.00048603008831669926,
+            -0.000574546300981984,
+            -0.012992623897880395,
+            0.02064747165046032,
+            0.014820638761258362,
+            -0.024820174379739057,
+            0.02666518374464728,
+            -0.017430076811109155,
+            -0.01495051200726725,
+            -0.005861181182847845,
+            0.00100881727574559,
+            -0.0055223282819321295,
+            0.0034719859452138017,
+            -0.010180752654175722,
+            -0.009961678082468036,
+            -0.012134423047419183,
+            -0.028073672819317354,
+            0.011574683159318163,
+            0.016120674453124317,
+            -0.015043147648505721,
+            0.031834693135959095,
+            -0.008802805725050891,
+            -0.0018698230152007877,
+            0.016179742266521146,
+            0.005096232332051617,
+            -0.017187077376166815,
+            -0.018999569809271876,
+            0.012251308792871571,
+            0.013279090498325945,
+            -0.009588297039279112,
+            0.006630431200804809,
+            -0.014130176783207386,
+            -0.005092662672039885,
+            0.007020236623084732,
+            -0.005049363863784624,
+            0.0055051712211635915,
+            -0.00995314895304919,
+            0.0052743456988306485,
+            0.005841370337169303,
+            -0.0029434616355351975,
+            0.004677049987338193,
+            -0.018568941264514256,
+            0.020031253265629777,
+            0.0018725390936655962,
+            0.01332532735283085,
+            -0.03518855189467458,
+            -0.0056298723325832025,
+            -0.005369475318146738,
+            -0.004650364775945294,
+            -0.020833510091412353,
+            0.020445280915606132,
+            0.0159991245532085,
+            0.013202409908533162,
+            0.023094507005426774,
+            -0.014905666713082246,
+            0.01572052456975675,
+            -0.020945766915121455,
+            0.013114000985348841,
+            0.003349101658930482,
+            -0.019829605346106578,
+            -0.001423750749031516,
+            0.04557459007627153,
+            -0.005847231973329081,
+            0.011986110554869603,
+            0.0012709842140249691,
+            0.026758922819943853,
+            -0.0001635536762342011,
+            0.0065419519929442995,
+            0.005997692702992206,
+            -0.009935128108338126,
+            -0.0008057798606777003,
+            0.00865712443881613,
+            0.046254320605750854,
+            -0.01181132866321511,
+            0.008170688314239364,
+            0.013099051984057827,
+            0.0034971494510865624,
+            -0.0008386400616584293,
+            -0.015346506644390764,
+            -0.03067678541405639,
+            0.019305962963428017,
+            0.007736916644077731,
+            -0.012756525606221703,
+            0.008734673666457216,
+            0.015137771675476126,
+            0.012891824153867332,
+            0.003895886821685762,
+            0.02155154331885366,
+            -0.012761690789748027,
+            0.018528313792219413,
+            0.024663453412919027,
+            0.009077230086690327,
+            9.184526542175535e-05,
+            -0.000751529849753638,
+            0.006791927267880182,
+            -0.0004433479107716552,
+            -0.009832230053513483,
+            0.015858232096630655,
+            -0.014861341148074698,
+            0.017790924034108806,
+            2.7216806238946785e-05,
+            0.03744841066552559,
+            -0.015216207585985075,
+            -0.010138319613387346,
+            -0.007774325714088762,
+            -0.0016997034352120362,
+            0.003971564314593291,
+            0.0018686537276849653,
+            0.006051669054361943,
+            -0.0008319624938748146,
+            -0.017326561560776794,
+            0.01667737942301944,
+            0.005719808546053257,
+            -0.0007405637872110778,
+            0.013750752065593804,
+            0.0038732323799683453,
+            -0.00367645612814479,
+            0.023580906347714314,
+            0.006238541579764762,
+            -0.005235795826199542,
+            0.0028182323816636243,
+            -0.006443210286549713,
+            -0.016255209299456314,
+            -0.003061441395209219,
+            0.004597970789372988,
+            -0.0016590920570710628,
+            -0.011138261849069419,
+            -0.0019423173609729848,
+            0.020355534503272887,
+            -0.004612410336124759,
+            0.013346750536413418,
+            -0.005400060374036387,
+            0.006252154576727367,
+            0.010601459925435238,
+            -0.012346569079682625,
+            0.0014154620309235822,
+            0.010406076751218612,
+            0.0013983417564564663,
+            -0.007140180075225315,
+            0.0010544410919327675,
+            0.007757599672903709,
+            0.006505350216372801,
+            0.00483063937962671,
+            0.00836630436955631,
+            0.013284451514874498,
+            0.0033749897293287438,
+            -0.006243293993633622,
+            0.0013949831763016042,
+            -0.013269921385800992,
+            -0.006308639321247727,
+            0.0010906169976769569,
+            -0.0010152835709869353,
+            -0.02017142197515583,
+            0.010787630901414118,
+            0.005355784839945777,
+            0.007838485470991034,
+            -0.01706932891696025,
+            0.009576030158827565,
+            -0.0008144311254998525,
+            0.02829832828245172,
+            0.007407953684956539,
+            -0.02769403281764041,
+            0.014266001222243134,
+            -0.010492199740754951,
+            0.022575782293320284,
+            -0.0016349316593207844,
+            -0.005656505606143177,
+            0.008025514398322551,
+            -0.009634281579228592,
+            -0.005991026158213869,
+            -0.022366083313133788,
+            -0.025825209755160946,
+            0.016871087849192924,
+            0.013583173973358155,
+            0.021125252303971605,
+            0.00012555627355712715,
+            0.015347535950809997,
+            0.0020128011217859444,
+            0.015164099887550086,
+            0.0024544903332736167,
+            0.012507136277293888,
+            -0.011798050869794805,
+            0.0104828761281283,
+            -0.0137805761114781,
+            0.014620357150314097,
+            -0.005246639228547017,
+            -0.006986408834884282,
+            -0.008849580196616098,
+            -0.005442165099441538,
+            -0.020891551850452786,
+            -0.009155930286949747,
+            -0.0013590151948670868,
+            0.010665872513236615,
+            -0.0005386423407222223,
+            -0.0027671778465775897,
+            -0.03237311596935737,
+            -0.004875062533405715,
+            -0.006670982068843501,
+            -0.01942244110096111,
+            -0.0045975373348006885,
+            -0.026137783023267008,
+            -0.01657173843643047,
+            -0.0007784046498232684,
+            -0.015037306676062236,
+            -0.006889680260149007,
+            -0.02009705080543122,
+            -0.027192867101205106,
+            3.0161964469866572e-05,
+            -0.009647644858034833,
+            -0.005380165957043753,
+            0.013815092649296373,
+            0.011902366689169847,
+            -0.01572592170758115,
+            -0.022321702398924333,
+            -0.01088937403483412,
+            -0.0165754067023749,
+            0.01646435743868707,
+            0.009914938007210463,
+            0.018834863207293908,
+            -0.003724245912768107,
+            0.005643505694722323,
+            0.0009581443359000331,
+            0.02327704258596552,
+            -0.02532422120223976,
+            -0.028694220894404807,
+            0.0019298778426469338,
+            0.0025970175688368323,
+            0.006086164866310174,
+            -0.009414665394188936,
+            0.001200390869257535,
+            -0.007782899559317819,
+            -0.0228264209954853,
+            -0.00898519580978142,
+            0.008156835270668449,
+            -0.010036630164522355,
+            0.02230664068547531,
+            0.012332230358869292,
+            0.013699197379189934,
+            0.010593587938184292,
+            0.01576174456497367,
+            -0.0001654241831584687,
+            0.014471905257729428,
+            -0.008675915624727887,
+            -0.0031070346099204425,
+            0.0038358404385932777,
+            0.005878417037187274,
+            -0.00625022998235275,
+            0.028077114529659795,
+            -0.0011836453716817525,
+            0.0002962695480296182,
+            0.014405782572641941,
+            0.02697667559474498,
+            -0.004710313427099278,
+            0.0210512043261309,
+            -0.002393165025068831,
+            -0.006135816286212752,
+            -0.008429776018235104,
+            0.008607617299702318,
+            -0.009529291136733401,
+            0.011133497451942684,
+            0.019864404836361926,
+            -0.014041315457579166,
+            -0.007351413712676263,
+            0.02950254974910368,
+            -0.014564388091367897,
+            -0.01825317106331622,
+            0.02325964952838705,
+            -0.007414295676409368,
+            -0.0039042587057605484,
+            0.00377144546378926,
+            0.007169856503233239,
+            -0.004036744966949995,
+            -0.012928341573802416,
+            0.016090527447479415,
+            -0.00011159893712484579,
+            0.014146137810890391,
+            -0.0004655839804281017,
+            -0.006372231942983921,
+            -0.010047382825720416,
+            0.002373224247019062,
+            0.026818813126567614,
+            0.013483627627077692,
+            -0.017895710143841333,
+            -0.013785856764479817,
+            0.010826610341182479,
+            -0.01538893818197323,
+            -0.004425206001945227,
+            0.02155595194812746,
+            0.0028768153668128553,
+            -0.020543027467698483,
+            0.008028975884363618,
+            0.0003835165718138906,
+            0.00037046947047892953,
+            -0.00029363296913641716,
+            0.030135739208689217,
+            0.001343516577762509,
+            0.00018948658255495327,
+            -0.00875053830853609,
+            0.008606573235959595,
+            -0.01280305089741873,
+            0.012786478836532904,
+            -0.02437993139337839,
+            0.007648181775166282,
+            -0.00629541989541625,
+            0.004579605315599212,
+            0.016658386927051592,
+            0.01218098627022323,
+            0.015031618813778413,
+            0.008237200519537417,
+            -0.008942965899756683,
+            0.0009093265012901892,
+            -0.025302175262520477,
+            -0.001333639155622843,
+            -0.017971876771137367,
+            -0.02036433247634014,
+            0.0002463842837918767,
+            0.022144712689630985,
+            0.018559482073124734,
+            0.011579840028464503,
+            -0.008657014440886656,
+            0.0166696911919226,
+            -2.9834620942882283e-05,
+            0.011845827764518911,
+            -0.00037901176785495324,
+            -0.006138374154553732,
+            0.004711058953619512,
+            -0.003622694588720417,
+            0.013542405259906128,
+            -0.014730567462694827,
+            0.012076044672004906,
+            0.01690223095263866,
+            -0.009108632851807137,
+            -0.008946251274440141,
+            -0.013992423874475428,
+            0.00048381889145447367,
+            -0.015909599902931202,
+            0.035489252674454214,
+            0.02252772349379229,
+            -0.01210464970923921,
+            -0.010469120693943236,
+            -0.014164999313775756,
+            0.019496048806752753,
+            0.010427539351244087,
+            0.0028018908451425276,
+            0.017091768406027965,
+            0.021395638416767112,
+            0.008817247593404623,
+            -0.006081063711583319,
+            -0.0021216503501144983,
+            -0.003781810594530787,
+            0.020143098218482827,
+            -0.003078172339607037,
+            -0.011350171884507555,
+            -0.012516379645909518,
+            0.005766102930704529,
+            -0.01626433668691071,
+            0.0019294643928566657,
+            0.008641558521764407,
+            -0.015057688533773374,
+            0.019407986696657772,
+            -0.028126197703647868,
+            0.012364374199657305,
+            -0.0044278434183731225,
+            -0.007251147925369614,
+            -0.03496220945653279,
+            0.006389489026974697,
+            0.008941259259632024,
+            0.003941752277454314,
+            0.023003838655784314,
+            -0.024528514847470928,
+            -0.018361931455105486,
+            0.0037791061321962543,
+            0.00047162060848090104,
+            -0.00711894577052754,
+            0.0010138871688747137,
+            0.004128392769213672,
+            -0.00674546985015164,
+            0.0017307097443508704,
+            0.0033354800298095546,
+            -0.002829898077850263,
+            0.013252947102333348,
+            0.0081724269619999,
+            -0.021449979263734438,
+            0.001368791856860285,
+            0.01590656734732547,
+            0.01452008402429858,
+            -0.02391847163018108,
+            0.006331397860689212,
+            -0.02064812934794864,
+            -0.000631340242718475,
+            -0.031272866000888556,
+            0.016859793097564958,
+            -0.0069008624795683276,
+            0.01899718801899956,
+            -0.0015377436942771224,
+            -0.0065810685260109004,
+            -0.012822620655323715,
+            0.02137604937338179,
+            -0.0019121137078344135,
+            -0.0025598772127157477,
+            0.015678293057066646,
+            -0.025700865215693987,
+            0.017825186616033165,
+            -0.0026230848396158204,
+            -0.017342967698019966,
+            0.017199139867162275,
+            0.004536019178633225,
+            0.007224666811309087,
+            -0.023505589665315567,
+            0.0007208041198255396,
+            0.003900164485347562,
+            -0.017250324025622685,
+            0.004470862192961212,
+            -0.030807400122398566,
+            0.0006579147990327597,
+            -0.013222907940559794,
+            -0.0020514994691204847,
+            -0.0046763235724575445,
+            -0.014050035576183385,
+            -0.039181147615786484,
+            -0.005429639279537522,
+            0.016065742005148733,
+            -0.0009029636981539927,
+            0.01800715198737081,
+            -0.002730931639143803,
+            0.002947680833843522,
+            0.004912692811020822,
+            0.00818378920043427,
+            0.006915895602313347,
+            0.02274116906914758,
+            -0.0057854922886970365,
+            0.00471999900678522,
+            0.007058014277137233,
+            0.014638081356210031,
+            -0.01719565951201012,
+            -0.015170377598372723,
+            -0.016015921063369496,
+            -0.006351213642269936,
+            0.01739665344303718,
+            -0.017488885259028475,
+            0.013303332445465686,
+            -0.00429428898520701,
+            0.014825338166443021,
+            -0.008686427208690176,
+            -0.01575945086724703,
+            0.0028564767084934386,
+            -0.007570417028925067,
+            0.001007670647768344,
+            -0.02323908008484089,
+            0.02019729564706863,
+            -0.00847581038203716,
+            0.010508144473197669,
+            0.01729824052022313,
+            -0.021364529179605813,
+            -0.009742851959432839,
+            -0.014342281441152413,
+            0.029896210049029176,
+            0.013925163184699139,
+            -0.009392252140772913,
+            0.003927230832659267,
+            0.008873561269230905,
+            0.011529536139797032,
+            -0.012209070214071445,
+            -0.0027679974125029362,
+            0.002411914349172495,
+            -0.0062056207925691356,
+            -0.002025687569076897,
+            0.03090134105002816,
+            0.0038285457265343245,
+            0.024877474647546086,
+            -0.013214997316123852,
+            -0.0033763842539861907,
+            0.00023379106197363714,
+            0.005134764372494098,
+            0.01183867975321486,
+            -0.0017966997954747142,
+            -0.008029791503028367,
+            -0.01179500476055557,
+            0.02599911330072515,
+            -0.021852083993691557,
+            0.007024505421035273,
+            0.026784024996764876,
+            0.001390202993990949,
+            0.0029956956045771046,
+            -0.013131691460546865,
+            -0.006298736864509989,
+            0.02523640664202355,
+            0.010030824059858138,
+            0.0041523782603032225,
+            -0.026465819482580182,
+            0.0032128899272377593,
+            -0.0061901409332948115,
+            -0.008681758126710213,
+            0.009866461769956222,
+            -0.0028095454857224164,
+            0.01188012041754054,
+            -0.012191996383095982,
+            0.028459939543805705,
+            0.010711502132045176,
+            0.01047154118931583,
+            0.025888652452698974,
+            -0.02079614305315134,
+            0.0042203106489994615,
+            -0.010829796140534066,
+            0.0006773493086316061,
+            0.005596640511307504,
+            -0.027510562413905437,
+            0.001969898603871003,
+            0.010390866563404346,
+            -0.005642706366801436,
+            -0.01882994179544761,
+            0.01187970346127799,
+            -0.019814385265412387,
+            0.024479579275617887,
+            0.003231341626501863,
+            0.020056914356101588,
+            -0.0053276448188067154,
+            0.009834685132275868,
+            0.0002742942778577165,
+            0.0031361331890882127,
+            -0.0008074598092128465,
+            0.009009012303347016,
+            0.005786777410144894,
+            0.013719989127068968,
+            -0.025528370602188686,
+            0.006821572597883499,
+            -0.0029197127345352626,
+            -0.030539872624857384,
+            -0.013383097680839732,
+            0.010291485622655026,
+            0.012218533188117704,
+            -0.009144425807890594,
+            -0.010378679518611277,
+            0.0030042579827305603,
+            0.005748243116238633,
+            -0.006692403885450407,
+            0.006913672151000981,
+            0.01629516912493545,
+            -0.002533517074771728,
+            -0.0019083178803585312,
+            -0.0035007340321378893,
+            -0.01154609935927537,
+            -0.010016998439072824,
+            0.023193275321990937,
+            -0.00482990342915347,
+            0.017943025772239603,
+            0.025873224580389702,
+            0.0039876567755322,
+            -0.0057577932736143506,
+            -0.005460363755206188,
+            0.01478435999841998,
+            0.014195457012279476,
+            0.004669295894868759,
+            -0.009702220679239768,
+            0.007974305230005956,
+            -0.007760086293456636,
+            0.008375077126777163,
+            -0.020260917110786147,
+            0.002199401871514877,
+            0.0026191373298767766,
+            -0.0022089262577237776,
+            -0.015213807046175918,
+            -0.01956788452420682,
+            -0.021898368819136302,
+            0.012634659463647929,
+            0.00034643102239488474,
+            -0.009241772775582378,
+            -0.013645250997151429,
+            0.0016665106990117016,
+            0.0025630500346914592,
+            0.0002095521560180936,
+            -0.007783022628694597,
+            0.009546147813008391,
+            -0.01265287317178631,
+            0.007445223311736028,
+            -0.004007029181436962,
+            0.0051846833906574515,
+            -0.0016916014415925766,
+            -0.016413938859402438,
+            -0.0210868129634149,
+            0.01920794362224444,
+            -0.012897042227841011,
+            0.02269303737601583,
+            -0.0014935878147854765,
+            3.062680855559304e-05,
+            0.0033132626755401546,
+            0.007274487651476732,
+            0.012110042834251237,
+            0.011426424896797977,
+            0.004307086698938411,
+            0.008206932254416481,
+            0.004673427176334148,
+            -0.013124571552764586,
+            0.004324963759928715,
+            0.0039216033705804084,
+            -0.010449887922072297,
+            0.005298323150690876,
+            -0.0010196552573491,
+            0.006128327172621373,
+            0.012167492341907742,
+            0.01923525204293908,
+            0.01908107550907106,
+            -0.0012031208519043073,
+            -0.0010601912677170903,
+            0.014474374858125607,
+            0.004540954805526089,
+            -0.004883537463424679,
+            -0.002772988629590886,
+            -0.013363713078069854,
+            -0.002572403646338979,
+            0.011541515690646252,
+            -0.02376161747414617,
+            -0.018215231053002656,
+            0.021831219299143226,
+            -0.004991882316041097,
+            0.033796174834149546,
+            -0.013140970283091536,
+            -0.016197177724633614,
+            -0.00464190076219332,
+            0.0026389653040644607,
+            -0.013788049032378498,
+            -0.010300859744205445,
+            0.010545541278382187,
+            0.013848738866656793,
+            0.009903590151120531,
+            -0.0005984085581408575,
+            0.02922816576443645,
+            0.0019900932217491907,
+            0.02245395592422119,
+            0.0018978407151245295,
+            -0.007029294098689854,
+            -0.01440575120349887,
+            0.010840472832356266,
+            0.005529335652955532,
+            0.009050748012161991,
+            0.017791269808484802,
+            0.014502057454389247,
+            0.006265567405788131,
+            0.013928705812292101,
+            0.0012903002372833693,
+            0.0010858377120770652,
+            -0.002355229463060164,
+            -0.01692711767136341,
+            -0.01456786846097011,
+            -0.01627640642218236,
+            0.003873864312554098,
+            0.012620350492241247,
+            0.00525890570023481,
+            0.0007393789113866752,
+            -0.007153210751899716,
+            0.009254566395058412,
+            0.0065087881512835375,
+            -0.012279152412209152,
+            -0.016530068283509593,
+            0.004060671955726967,
+            -0.02210570272789274,
+            0.007849467899009437,
+            0.01554233481902349,
+            0.00219885032769439,
+            -0.002655449524219968,
+            0.0006843524304842362,
+            0.006272944169764281,
+            0.005531927368102191,
+            -0.0036710328198476905,
+            -0.00544113514775027,
+            0.011264661763762068,
+            -0.006379654234440906,
+            0.009537150372938296,
+            -0.00290031548037575,
+            0.03718190949929003,
+            -0.011976080887998194,
+            0.012120408392610773,
+            -0.001751472085435711,
+            0.008481703962951994,
+            0.008067669857949423,
+            0.010772687289903202,
+            -0.003805089040741351,
+            -0.0022164543131735167,
+            -0.008753625032781523,
+            0.006751953442322185,
+            -0.01582177621255226,
+            0.013631301238130705,
+            -0.00030685726238985963,
+            -0.010728958775974582,
+            0.01055472059121978,
+            0.006788316621149835,
+            0.004413382316882087,
+            -0.003262024961860459,
+            -0.01972283203505511,
+            0.014937729522976615,
+            0.0018799533869990667,
+            0.019247008681353865,
+            -0.023276106189004046,
+            0.029043727846427652,
+            -0.02019024213343764,
+            0.023083643395685142,
+            0.01865875504942157,
+            0.001179436167200093,
+            -0.02070329911663634,
+            -0.007618131527214554,
+            0.014871709122115308,
+            -0.02659818889735726,
+            -0.006234719259825804,
+            0.005986829551040766,
+            -0.014639912579831785,
+            0.00037813423546252645,
+            -0.01924149909676064,
+            0.00010832498610532006,
+            0.011013130149723431,
+            -0.011782407678373089,
+            0.00438725832178584,
+            0.01608349750921335,
+            0.019347883652009713,
+            -0.009760017612545174,
+            0.003869246694835168,
+            -0.024240324890794748,
+            0.010958789926998691,
+            0.020511684636649225,
+            0.008883830392748497,
+            -0.01761944522796127,
+            0.021817973960311446,
+            0.010652131228027521,
+            -0.012575679152857703,
+            -0.0012753095453828826,
+            -0.028191274775392804,
+            -0.012421170815054556,
+            -0.02188248722244861,
+            -0.0011670175621882146,
+            -0.014436683528976771,
+            -0.0029783880907228383,
+            0.0009506986717516226,
+            -0.004719165363484719,
+            -0.01179163390107033,
+            -0.019543732196676634,
+            0.016606805875498898,
+            0.0227313413004497,
+            -0.006738735201626204,
+            -0.014547240981002825,
+            0.013866974327185123,
+            0.031694077725890536,
+            -0.016189209297529324,
+            -0.00542110554672736,
+            0.0028061798782555557,
+            -0.012964876141500577,
+            0.018946046750284537,
+            0.0039419760434645495,
+            0.000322115192590907,
+            -0.01991851369210666,
+            0.002489725351392778,
+            0.010835458511052207,
+            0.01684964045708685,
+            0.0005541314524621196,
+            0.010855852688653808,
+            -0.007549882748861744,
+            0.010096800951420057,
+            -0.011832260836468588,
+            -3.6179995603517093e-06,
+            -0.007800366796516945,
+            0.007001049318404058,
+            -0.004424776169203575,
+            0.034356203288485945,
+            -0.02817071276766299,
+            0.004989631403681216,
+            -0.014539644354369607,
+            -0.010888763939716889,
+            -0.0027322086509797843,
+            0.0004865534340981866,
+            -0.010775238981581461,
+            -0.005162220412866082,
+            -0.0011263271110752205,
+            0.00913081242511139,
+            0.005844327320514489,
+            -0.013886529519306775,
+            0.019171827605726503,
+            -0.030984919318656618,
+            0.004469458202753611,
+            -0.008553175891755315,
+            0.010511707988936336,
+            -0.0151185694875566,
+            0.013830711686954931,
+            0.01301959345025159,
+            0.008205984085944315,
+            0.007740367849216328,
+            -0.01532123052721834,
+            -0.01678314222392063,
+            0.001376468747213867,
+            0.0036953018347776244,
+            -0.00931855481896886,
+            -0.033391028922085425,
+            -0.007072913446166969,
+            0.005331350534425606,
+            0.011614453225043,
+            0.008156128603003278,
+            -0.02345206873048703,
+            0.005003870024809975,
+            -0.024178720646704597,
+            -0.012115112684614331,
+            -0.007750557665424973,
+            -0.011580152272614108,
+            -0.006117298954192534,
+            0.026449765548206453,
+            0.007989237291215073,
+            0.016691517078382376,
+            0.004954133565782356,
+            0.02742809127725913,
+            0.011670893621991404,
+            -0.011736840457474895,
+            -0.003349151747880166,
+            0.00035419034843196817,
+            0.005507651806218584,
+            -0.008706590052024251,
+            0.011793241980257321,
+            -0.0013295035978611142,
+            -0.0065061682758755625,
+            -0.004398354347597199,
+            0.009372816666628725,
+            0.014188676762501037,
+            -0.003007980559117429,
+            -0.008724255229333351,
+            0.015749776067989826,
+            -0.030899793617096834,
+            -0.015215033379100091,
+            0.0016152308157299517,
+            -0.007813175908301852,
+            0.017806770680528907,
+            0.01169056345888971,
+            -0.026835412610223673,
+            -0.004538235170332766,
+            0.010532316940606419,
+            0.004755930024691789,
+            0.0095408889025506,
+            0.00954782703036282,
+            0.012079064562163901,
+            -0.007829549014214643,
+            0.01046722658021748,
+            -0.011519369840999348,
+            -0.007092961453184232,
+            -0.007241799314569802,
+            -0.008077934089061052,
+            0.021467547268987214,
+            -0.025816030568238717,
+            -0.006558853543772633,
+            0.01134670818701549,
+            0.013027491377727365,
+            -0.002301991242932548,
+            -0.006299073927206214,
+            -0.017357989759336386,
+            0.008311464809363397,
+            -0.013922568511594478,
+            0.025633907424998367,
+            -0.004854655498817219,
+            -0.005336807899100369,
+            -0.005228056880188717,
+            0.01958682721272445,
+            -0.016774409737540904,
+            0.02072573220949266,
+            -0.014449250679704904,
+            -0.007857351393427215,
+            0.001673929749689194,
+            0.006890008607998747,
+            -0.002948835017234416,
+            0.03734128725793149,
+            -0.00556474133167756,
+            -0.0014544201137417158,
+            0.02209084868209573,
+            -0.01883610080898967,
+            0.017901421953179442,
+            0.01563604483998435,
+            0.012327159100148019,
+            0.0018161034827768648,
+            -0.010253034391420751,
+            0.019979728662282904,
+            0.020823925491027635,
+            0.0139490143219451,
+            0.009577521724445478,
+            -0.013478761809038993,
+            0.014934933784421556,
+            -0.0003725422088504494,
+            -0.0024362115214204105,
+            0.012785534912074068,
+            0.016441944143888537,
+            -0.02164522371903646,
+            -0.013152102018736346,
+            0.0063548333449661805,
+            -0.016547629104776748,
+            -0.024875443975589356,
+            -0.008991095813132973,
+            -0.021885127280977718,
+            -0.023336714382596175,
+            0.00688008599534903,
+            0.016482610587564443,
+            0.0035797836729533367,
+            0.013111012464332918,
+            0.006820471349350837,
+            0.018274678506489214,
+            -0.00980248993077149,
+            -0.01582838461385605,
+            -0.0009454586621769173,
+            -0.0004906493597104733,
+            0.009787688665732045,
+            -0.01835583040075359,
+            -0.013977939077760832,
+            0.004340902272034388,
+            -0.020118484730757877,
+            0.01421736173162507,
+            -0.006428386730058143,
+            0.016756376129345597,
+            -0.0004630315831872392,
+            0.011701948921076007,
+            0.02621773278421472,
+            -0.001391567406453007,
+            0.01262026995055899,
+            0.004696906079976175,
+            -0.0030577364576645635,
+            -0.010624066224426196,
+            0.0009420498021845824,
+            -0.0044079288776429516,
+            -0.0004845606666642234,
+            0.006481113327122954,
+            0.020269778435314,
+            -0.03321232649491074,
+            -0.009425498978427855,
+            0.01246754422779043,
+            -0.001658574527323383,
+            -0.005816985985181387,
+            0.013050062223188965,
+            0.00011195028631856881,
+            0.016573827662919836,
+            -0.0017453203044868104,
+            0.01417371165305189,
+            -0.008165910519539772,
+            -0.0007461950113024975,
+            0.0016492309267666983,
+            0.005076015595439873,
+            0.005829299318055953,
+            0.012146800191035666,
+            0.01148884639506138,
+            -0.021051327863689618,
+            -0.0026357096198729156,
+            0.0036146148489310443,
+            -0.010673282438301611,
+            0.014967444393862347,
+            0.005737933867764028,
+            0.01391411712868596,
+            0.0029556189893790237,
+            0.01167806258173173,
+            -0.024440974650111174,
+            -0.01029944007931845,
+            0.01385509328958837,
+            0.00869671728965227,
+            -0.00998315356917989,
+            -0.016580053751364735,
+            -0.007061005072623325,
+            -0.00477175200214453,
+            -0.0021182980051220428,
+            -0.009871524928312118,
+            -0.0005933941177237234,
+            0.006284920058476413,
+            5.466820734762947e-05,
+            -0.012842295882002122,
+            0.015810530885294202,
+            0.004572385791133295,
+            0.003685966629168175,
+            0.005093714133656017,
+            0.011877541830396259,
+            -0.013130170365831912,
+            0.01258190257102377,
+            0.011433121961378686,
+            0.01142665981860032,
+            -0.013710025169815697,
+            -0.001774030746825879,
+            -0.01109881340948764,
+            0.0065684584345163735,
+            0.009983753894796107,
+            0.005346443477816739,
+            -0.002062387245572686,
+            0.004950044847352647,
+            0.006105199826576752,
+            -0.06391185159544624,
+            -0.02131511418723256,
+            0.0027165737784844413,
+            0.033389688791557785,
+            -0.007167593552251769,
+            -0.0037646833460996797,
+            0.002883510089742145,
+            0.01356887065340832,
+            0.009730171586767032,
+            0.00418566245984083,
+            0.020320934773473808,
+            -0.008814826860651366,
+            -0.003014193747732543,
+            0.001559573791453607,
+            0.003499420226345028,
+            -0.008703837291821111,
+            0.0044844701302276645,
+            0.0053963630238108035,
+            0.027764369548110287,
+            0.001469968540767627,
+            0.015793086297019008,
+            -0.008505290826588508,
+            -0.004075888558568453,
+            -0.009166710762668385,
+            -0.002860047211143503,
+            -0.011127820563172684,
+            0.003937110717404271,
+            0.02054186721680431,
+            -0.006126180963243227,
+            0.009346433819831456,
+            0.01057504086030076,
+            0.0001650851447316893,
+            -0.012033423322524951,
+            0.0049587335923558,
+            -0.0032240390628760535,
+            -0.005568976687946761,
+            -0.0024667040429312737,
+            -0.03474027725033016,
+            0.026855947684394643,
+            -0.004369626674503285,
+            0.009252512430957932,
+            -0.005081834388513357,
+            -0.0037896610731020337,
+            -0.014201609270756739,
+            0.017871088029615106,
+            -0.008249444723433136,
+            -0.020092257756971717,
+            0.009532994544485572,
+            -0.004991989149115383,
+            0.006464417738670472,
+            -0.02643078108339741,
+            0.0029138922065812495,
+            -0.017232410749655366,
+            -0.003999293090834831,
+            -0.006924488971625995,
+            0.01318082244178942,
+            -0.007232695119856389,
+            -0.02660711037908311,
+            0.01979251106803667,
+            -0.019485860712345603,
+            -0.011977904020170568,
+            -0.0177530973032872,
+            -0.014524003566945937,
+            0.004874952064727108,
+            -0.015326044325348929,
+            0.002047771741041311,
+            -0.009697340332342246,
+            -0.003652304998155628,
+            -0.0008878698629674222,
+            0.0017782435347417623,
+            -0.013017776298713044,
+            -0.018272697230002275,
+            -0.01353020425912225,
+            -0.00757239449706798,
+            -0.0008250000619957538,
+            0.01299142281772714,
+            -0.001679606555140765,
+            -0.007329941421303076,
+            0.0044605462460153116,
+            0.010231192277666706,
+            0.0006558644644452857,
+            0.0024231733856020414,
+            -0.010728844680699885,
+            -0.014137434553169443,
+            0.005230053181525096,
+            -0.012086468475542818,
+            -0.009459717916037003,
+            -0.0028185227600210556,
+            0.014984527039312847,
+            0.008189464334213754,
+            -0.009139240774931914,
+            -0.005639157579163322,
+            0.014325949891859377,
+            0.009020303519690971,
+            -0.013310205617445609,
+            0.01567574314514385,
+            -0.025369802629268523,
+            0.021964786021408878,
+            0.030536905527039065,
+            0.004945288795196981,
+            -0.01168303908203322,
+            0.02281128628230144,
+            -0.004072286832145419,
+            0.0028278384348551146,
+            0.01740522162288968,
+            -0.042558465579514926,
+            -0.013146562547084278,
+            0.002903743316339479,
+            -0.015297177099602701,
+            -0.010963312697387696,
+            0.008804938803398296,
+            -0.0026727734429658817,
+            0.005273238046953793,
+            0.004257613402172282,
+            0.03512246556126453,
+            0.002258612643297096,
+            -0.0002885708728599208,
+            -0.0019720190285930487,
+            0.012312613129996533,
+            0.010451003939611476,
+            -0.015556620910136185,
+            -0.02387856865043814,
+            -0.004583090523619507,
+            -0.0037828776396525835,
+            -0.0014588238692793338,
+            -0.004361834167147271,
+            -0.008159679676329573,
+            0.012243319920992457,
+            -0.017279964797769668,
+            -0.013748032758693542,
+            0.018394195628121487,
+            -0.006411113522842359,
+            0.019002611495040334,
+            -0.014970362556522149,
+            -0.0077437581301661946,
+            0.0017311115715410025,
+            0.024439691379769795,
+            -0.025013956798520212,
+            0.007359134924497539,
+            -0.0031406470488477937,
+            0.017574824561755627,
+            0.00419409250132525,
+            -0.0046803792187546656,
+            0.010556972371044426,
+            0.015229789583480183,
+            -0.004608293492221483,
+            -0.014449487985966966,
+            -0.01172850291078532,
+            0.016823024209386785,
+            -0.012304887597699588,
+            0.0011773351576246157,
+            0.002262670226244277,
+            -0.0023810578015456716,
+            0.03477119000959688,
+            -0.03442178291363675,
+            0.004062693040714608,
+            0.0029249583507755944,
+            0.003291349443702718,
+            0.006461525089528308,
+            0.006047777261425826,
+            -0.020642802018978404,
+            0.008916088381876748,
+            -0.03316762661930299,
+            -0.006367254689957291,
+            -0.005404276207844267,
+            0.0009189006423526284,
+            0.009401330264376041,
+            -0.011106443887384533,
+            -0.02259405372588916,
+            -0.006010439054323966,
+            0.0070744551115721095,
+            0.010262607009446879,
+            -0.002856616377476421,
+            0.019719451109535182,
+            0.008426509638139982,
+            0.0015877648339862691,
+            -0.010655792508751512,
+            -0.015527851374834499,
+            -0.015460974859638219,
+            -0.017847462632188386,
+            -0.02236964311096428,
+            0.011841749854626384,
+            -0.00576688153869993,
+            0.009054902130717224,
+            0.014317707346504314,
+            -0.015366657284925976,
+            0.002910211008094564,
+            0.017761685988205086,
+            0.007974806171830457,
+            0.002692237107055498,
+            0.0032405477366987947,
+            0.02685230091638845,
+            -0.0009116884421301454,
+            -0.0032228977180251673,
+            0.003158213042682977,
+            0.015872199488167846,
+            0.0004966905858799325,
+            0.019981401695749582,
+            0.015201538351780685,
+            0.005359075814444414,
+            -0.004668738193285921,
+            0.019853988551405383,
+            -0.009844394286458195,
+            0.013476144827747423,
+            -0.00411789677405284,
+            -0.00830656434595337,
+            -0.008496132520586954,
+            0.006442581751999304,
+            -0.008138786054348571,
+            0.006977337946948795,
+            0.02419128791801387,
+            -0.0005734827808485311,
+            -0.0028842142002271044,
+            0.005503033191779872,
+            -0.01696639779775455,
+            -0.00389835761601045,
+            -0.03366077608692946,
+            -0.009942613075976742,
+            -0.01708630689497001,
+            -0.02910756562213045,
+            0.026536268897199546,
+            0.00219827495662445,
+            0.02586503862171508,
+            -0.01905122645609761,
+            0.008935637454859214,
+            -0.01385806889332683,
+            0.029747093931541325,
+            0.009163959012862627,
+            0.0008611776492791918,
+            0.003598831885031635,
+            -0.02103045281939351,
+            -0.01150941825152158,
+            0.027341774725604333,
+            -0.0006700608710615124,
+            -0.012089044078073347,
+            0.022396400579038037,
+            0.017491565611764764,
+            0.0051760367056280536,
+            -0.00559536785318233,
+            0.01979262595019399,
+            0.002306380546194741,
+            0.011973863166872444,
+            -0.0028136750606972223,
+            0.012911235736067303,
+            -0.01510421118359169,
+            -0.003338602157155174,
+            0.039758808572572274,
+            0.007290642519242007,
+            -0.001802921446521437,
+            -0.013696774990705774,
+            0.003941198181491921,
+            -0.0008308293948698925,
+            0.0200351513103389,
+            -0.00014327587644797734,
+            0.028291423013030337,
+            -0.0001085859276231927,
+            -0.010723064585989723,
+            0.020443848393732245,
+            -0.017860296930396913,
+            0.04540901629084378,
+            -0.018671578423261854,
+            0.000852031295629291,
+            -0.010639853151638835,
+            -0.019017194262730474,
+            0.003861296738239369,
+            0.0010977143012920602,
+            0.0008387482016061222,
+            0.0060155816314618,
+            -0.010780239332444652,
+            0.002152214497259421,
+            0.010556338812873054,
+            -0.027636192450621728,
+            0.0010938992111792307,
+            -0.014385202899929138,
+            -0.01353192238925814,
+            -0.026667778226211785,
+            -0.00017403398236549974,
+            -0.015357674131635848,
+            -0.013956722505869091,
+            -0.01985038526848704,
+            -0.008927903879720431,
+            -0.009580214158077466,
+            -0.0020314441852913386,
+            -0.004755584878514854,
+            -0.00018388548107372635,
+            -0.002918855567842974,
+            -0.0015941852262316128,
+            -0.014230272061190995,
+            0.02133992212037431,
+            0.015283345362445234,
+            0.004642765060980717,
+            0.012829570028541471,
+            0.01708952142208596,
+            -0.004770647543823141,
+            0.006604762788847952,
+            -0.0025369836845081078,
+            0.008368000846861623,
+            0.014785031726639733,
+            0.012074738887486398,
+            0.02124518119032885,
+            -0.015459818988572564,
+            -0.004992865135357168,
+            -0.012722391036055767,
+            -0.012210202603282509,
+            -0.017850065946244383,
+            -0.004300429148015694,
+            -0.0014387298144714717,
+            -0.023128300275416423,
+            0.004468952290723408,
+            -0.0026602931688877443,
+            -0.005768852559887139,
+            0.007815212168719318,
+            0.0005862972228454734,
+            -0.002265719947629544,
+            -0.00019323737674204774,
+            -0.02735820445266249,
+            0.00024009122317215594,
+            -0.00018701993387558173,
+            -0.014663710240203917,
+            -0.00450797561624596,
+            0.010584731989847334,
+            -0.011966527748437568,
+            -0.00602367292291343,
+            0.0111155226518017,
+            0.0015565179432852256,
+            0.006062862961835896,
+            0.016952559697728433,
+            0.0005012157272316869,
+            0.011160668242334889,
+            0.014993782560797772,
+            -0.01591303023837098,
+            -0.015356412848444082,
+            -0.0236025426434855,
+            -0.004514538154596937,
+            -0.009753334017065119,
+            0.007553951535816575,
+            -0.004616595532996391,
+            -0.013023153809236325,
+            0.006040285197946957,
+            -0.0006343451398642784,
+            0.008338564073158257,
+            -0.017608246105292328,
+            -8.346168119689692e-05,
+            -0.020944870308705715,
+            0.009230123321316848,
+            -0.00531237224627929,
+            -0.00445985089300067,
+            0.01410675485522911,
+            0.02895945580213926,
+            -0.007799830351886844,
+            -0.009455128322575888,
+            -0.005663072244883114,
+            0.012817229945992764,
+            0.012474431030896143,
+            -0.004423141749110039,
+            -0.0018294512146253298,
+            -0.012817101093394574,
+            0.0004755435377918658,
+            -0.01129485262081729,
+            0.0062747111125005155,
+            0.017199374798326412,
+            -0.021965117839794063,
+            -0.01296097993683748,
+            -0.016765996091978336,
+            -0.016882855472266523,
+            0.004372970343291638,
+            0.017290565252244258,
+            0.022670788364378462,
+            -0.008456547449078897,
+            -0.019564049850942417,
+            0.006898353991539946,
+            -0.01620125883985687,
+            0.0037108638122592746,
+            0.12887455743463638,
+            -0.0028864852421660366,
+            -0.011487804449328495,
+            -0.022683334123299446,
+            -0.005467192611880829,
+            0.01210557696712196,
+            -0.01555159294496261,
+            0.06709672179195036,
+            -0.015071089552631105,
+            0.004364902817496082,
+            -0.004899872946473657,
+            0.0006310685609937571,
+            -0.017610935588191347,
+            -0.005962991715597585,
+            -0.007768819050529025,
+            -0.02688607975562596,
+            0.000883763319535686,
+            -0.009935313912803587,
+            -0.012883104064962595,
+            0.002126368813790577,
+            -0.0013697265269800346,
+            -0.016131186838419288,
+            0.01575514418037863,
+            -0.020677990768697592,
+            -0.007251917191831732,
+            -0.0029325623780992118,
+            -0.00600710212646933,
+            0.014573093764769918,
+            -0.03253336117394778,
+            0.002973725493330224,
+            -0.013601417220439602,
+            0.011905788192118131,
+            0.001542117126832946,
+            -0.010789486980776022,
+            -0.013862139885514524,
+            0.007086571810314664,
+            0.015658913897173155,
+            0.017039624764452546,
+            0.01901374401756777,
+            0.004797954569087795,
+            0.003928999581093576,
+            -0.008046722893614655,
+            0.021974371139905104,
+            -0.00697335955527197,
+            0.0008975473383295958,
+            -0.00970589295353857,
+            -0.004346869073499034,
+            0.0031279458769925416,
+            0.02303983493541673,
+            -0.012560380384977343,
+            -0.01536378830925542,
+            0.0012035987157572355,
+            0.033313215063595594,
+            -0.010845084209743817,
+            0.013587126023659577,
+            0.0029867620644300073,
+            0.020679673435988446,
+            -0.0006640787253300063,
+            0.0025071780490108306,
+            0.022554193713122565,
+            -0.018082906254599396,
+            0.018230000220500718,
+            -0.004451773302171552,
+            0.01915083025790679,
+            -0.004183521043835777,
+            0.015019320101074747,
+            -0.027014055979204272,
+            -0.005362044589023088,
+            -0.012355559881051253,
+            0.003995309003198048,
+            0.020680455787871383,
+            0.0054391755324872375,
+            0.009865676762244514,
+            -0.01645386866657314,
+            -0.01760657627715638,
+            0.022969828924328126,
+            -0.021566687194482292,
+            -0.015826424849757402,
+            0.00012804774676298237,
+            -0.015021970627042762,
+            0.006000610438452896,
+            0.010126179023323158,
+            -0.02912121211344025,
+            -0.02046150879033127,
+            0.004881850039543051,
+            -0.0023469840669773042,
+            -0.00915786433296249,
+            0.006547377770403066,
+            -0.022329063637464074,
+            -0.015702777677743352,
+            -0.005692904224111613,
+            0.01207999389934878,
+            -0.008429160038192635,
+            0.010075061963131637,
+            -0.020106712664936504,
+            0.0036787045650556907,
+            -0.002764965946892891,
+            0.010307702251974629,
+            0.004697169247387421,
+            0.011192285826515207,
+            0.0030113901179382946,
+            0.0014774681536380047,
+            -0.02778270000997313,
+            0.010043261430740039,
+            -5.178406450222847e-05,
+            0.04216636255860709,
+            0.030977096295679525,
+            0.004479857028809617,
+            -5.798020745008712e-05,
+            -0.009996471945562616,
+            0.012476009017187798,
+            -0.019306907391548302,
+            -0.005092322343599383,
+            0.01856463597985508,
+            0.02195467141594795,
+            -0.02236232264139305,
+            0.0052217593371508145,
+            -0.016766955100332014,
+            0.016133299654723637,
+            -0.011208763137897244,
+            0.006248024409576809,
+            0.011601988603146743,
+            0.0022330878390492527,
+            0.010317781892717903,
+            -0.005895142237225119,
+            0.019670196605183973,
+            0.01186677200787666,
+            -0.002339367025714548,
+            0.02367408853110795,
+            0.006858134385159451,
+            0.004479683673962342,
+            0.004798719889930319,
+            -0.009604173975576537,
+            -0.009045602998213713,
+            0.009230969488015385,
+            0.004083616606774546,
+            -0.007944628589259331,
+            0.011836323287097144,
+            0.015100350195476968,
+            0.01885317701149666,
+            -0.0011707847499298092,
+            -0.00684548433712413,
+            -0.006550059418141426,
+            0.001899758393077906,
+            -0.024577381680573596,
+            -0.02855511854789239,
+            0.021238935710272536,
+            -0.010427135883958709,
+            -0.019187769881105637,
+            -0.0015864640188683347,
+            0.016360037306061392,
+            -0.008853972319021563,
+            -0.01627167079143546,
+            0.001270414822216867,
+            0.02255755055961612,
+            -0.005959411125391464,
+            0.006009654905468478,
+            0.005909233773418581,
+            0.005874361519568704,
+            -0.0027859463941992753,
+            -0.024599789577567636,
+            0.0036612374968223453,
+            0.03221346865585554,
+            0.016461563205582293,
+            0.016139782455492073,
+            -0.007158168059808459,
+            0.001838455290310879,
+            0.007530954381521755,
+            0.005438791523412295,
+            0.015674000912615536,
+            0.008604106101542998,
+            -0.006990462920227135,
+            0.00244736628387359,
+            0.012213048255418866,
+            -0.007329299946651028,
+            0.017424711328323798,
+            0.001973882532418776,
+            -0.0133178791530314,
+            -0.008373452013951822,
+            -0.010462150707776556,
+            -0.014575483571750984,
+            0.0027958704799249897,
+            -0.011401916623878957,
+            0.0008016041012669959,
+            0.003953958717263787,
+            0.010260753041492368,
+            0.013961649189650642,
+            -0.0019275791443041316,
+            0.017019168629298462,
+            0.011534594243860177,
+            -0.0017859096307142257,
+            -0.002970739203219076,
+            -0.03142778833370656,
+            0.010207789755749909,
+            -0.014730553506704018,
+            -0.0037611416875830746,
+            0.01825790347981639,
+            -0.007766890847082858,
+            -0.0032778433418881666,
+            -0.021003405429807113,
+            0.0024512381432602736,
+            0.008768616692363321,
+            0.01558713989712387,
+            -0.012819542714293536,
+            0.009201155584074711,
+            -0.0012254337002053017,
+            -0.0057451574939563045,
+            0.00977251392462179,
+            0.0042356831792011935,
+            -0.0033433823868435786,
+            0.011499764125503874
+          ],
+          "title": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer",
+          "keyphrases": [
+            "Sparsely-Gated Mixture-of-Experts Layer",
+            "conditional computation",
+            "neural network",
+            "language modeling",
+            "machine translation"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "1e687808-8db8-4320-805d-0282a8d679bc",
+        "properties": {
+          "page_content": "Deepseekmoe: Towards Ultimate Expert Specialization In\n\nMixture-Of-Experts Language Models\n\nDamai Dai∗1,2, Chengqi Deng1, Chenggang Zhao∗1,3, R.X. Xu1, Huazuo Gao1, Deli Chen1, Jiashi Li1, Wangding Zeng1, Xingkai Yu∗1,4, Y. Wu1, Zhenda Xie1, Y.K. Li1, Panpan Huang1, Fuli Luo1, Chong Ruan1, Zhifang Sui2, Wenfeng Liang1 1DeepSeek-AI 2National Key Laboratory for Multimedia Information Processing, Peking University 3Institute for Interdisciplinary Information Sciences, Tsinghua University 4National Key Laboratory for Novel Software Technology, Nanjing University {daidamai, szf}@pku.edu.cn, {wenfeng.liang}@deepseek.com https://github.com/deepseek-ai/DeepSeek-MoE\n\nAbstract\n\nIn the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top- out of experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into ones and activating from them, allowing for a more flexible combination of activated experts; (2) isolating experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5× expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations.\n\nFurther, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\n1. Introduction\n\nRecent research and practices have empirically demonstrated that, with sufficient training data available, scaling language models with increased parameters and computational budgets can yield remarkably stronger models (Brown et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; Touvron et al., 2023a). It is imperative to acknowledge, however, that the endeavor to scale models to an extremely large scale is also associated with exceedingly high computational costs. Considering the substantial costs, the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017) has emerged as a popular solution. It can\n\nenable parameter scaling, while concurrently keeping computational costs at a modest level.\n\nRecent applications of MoE architectures in Transformers (Vaswani et al., 2017) have yielded successful attempts at scaling language models to a substantial size (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022), accompanied with remarkable performance. These achievements underscore the considerable potential and promise of MoE language models.\n\nDespite the promising potential of MoE architectures, existing MoE architectures potentially suffer from issues of knowledge hybridity and knowledge redundancy, which limit the expert specialization, i.e., each expert acquires non-overlapping and focused knowledge. Conventional MoE architectures substitute the Feed-Forward Networks (FFNs) in a Transformer with MoE layers. Each MoE layer consists of multiple experts, with each structurally identical to a standard FFN, and each token is assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts.\n\nThis architecture manifests two potential issues: (1) Knowledge Hybridity: existing MoE practices often employ a limited number of experts (e.g., 8 or 16), and thus tokens assigned to a specific expert will be likely to cover diverse knowledge. Consequently, the designated expert will intend to assemble vastly different types of knowledge in its parameters, which are hard to utilize simultaneously. (2) Knowledge Redundancy: tokens assigned to different experts may require common knowledge. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby leading to redundancy in expert parameters.\n\nThese issues collectively hinder the expert specialization in existing MoE practices, preventing them from reaching the theoretical upper-bound performance of MoE models.\n\nIn response to the aforementioned issues, we introduce DeepSeekMoE, an innovative MoE architecture specifically designed towards ultimate expert specialization. Our architecture involves two principal strategies: (1) Fine-Grained Expert Segmentation: while maintaining the number of parameters constant, we segment the experts into a finer grain by splitting the FFN intermediate hidden dimension. Correspondingly, keeping a constant computational cost, we also activate more fine-grained experts to enable a more flexible and adaptable combination of activated experts. Fine-grained expert segmentation allows diverse knowledge to be decomposed more finely and be learned more precisely into different experts, where each expert will retain a higher level of specialization. In addition, the increased flexibility in combining activated experts also contributes to a more accurate and targeted knowledge acquisition. (2) Shared Expert Isolation: we isolate certain experts to serve as shared experts that are always activated, aiming at capturing and consolidating common knowledge across varying contexts.\n\nThrough compressing common knowledge into these shared experts, redundancy among other routed experts will be mitigated. This can enhance the parameter efficiency and ensure that each routed expert retains specialized by focusing on distinctive aspects. These architectural innovations in DeepSeekMoE offer opportunities to train a parameter-efficient MoE language model where each expert is highly specialized.\n\nStarting from a modest scale with 2B parameters, we validate the advantages of the DeepSeekMoE architecture. We conduct evaluations on 12 zero-shot or few-shot benchmarks spanning diverse tasks. Empirical results indicate that DeepSeekMoE 2B surpasses GShard 2B (Lepikhin et al., 2021) by a substantial margin, and even matches GShard 2.9B, a larger MoE model with 1.5× expert parameters and computation. Remarkably, we find that DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with an equivalent number of parameters, which sets the strict upper bound of MoE language models. In pursuit of deeper insights, we conduct elaborate ablation studies and analysis on the expert specialization for DeepSeekMoE.\n\nThese studies validate the effectiveness of fine-grained expert segmentation and shared expert isolation, and provide empirical evidence supporting the assertion that DeepSeekMoE can achieve a high level of expert specialization.\n\nLeveraging our architecture, we subsequently scale up the model parameters to 16B and train DeepSeekMoE 16B on a large-scale corpus with 2T tokens. Evaluation results reveal that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B (DeepSeek-AI, 2024), a dense model trained on the same 2T corpus. We also compare DeepSeekMoE with open source models and the evaluations demonstrate that DeepSeekMoE 16B consistently outperforms models with a similar number of activated parameters by a large margin, and achieves comparable performance with LLaMA2 7B (Touvron et al., 2023b), which has approximately 2.5 times the activated parameters. Figure 1 demonstrates the evaluation results on the Open LLM Leaderboard1. Additionally, we conduct supervised fine-tuning (SFT) for alignment, transforming the model into a chat model. Evaluation results show that DeepSeekMoE Chat 16B also achieves comparable performance with DeepSeek Chat 7B and LLaMA2 SFT 7B in the chat setting. Encouraged by these results, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. The experimental results still validate its substantial advantages over the GShard architecture consistently. In addition, it shows performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nOur contributions are summarized as follows: - Architectural Innovation. We introduce DeepSeekMoE, an innovative MoE architecture aiming at achieving ultimate expert specialization, which employs two principal strategies of fine-grained expert segmentation and shared expert isolation.\n\nEmpirical Validation. We conduct extensive experiments to empirically validate the effectiveness of the DeepSeekMoE architecture. Experimental results validate the high 1https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard level of expert specialization in DeepSeekMoE 2B, and indicate that DeepSeekMoE 2B can nearly approach the upper bound performance for MoE models\n\nScalability. We scale up DeepSeekMoE to train a 16B model and show that with only about 40% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B and LLaMA2 7B. We also undertake a preliminary endeavor to scale up DeepSeekMoE to 145B, highlighting its consistent advantages over the GShard architecture and showing a comparable performance with DeepSeek 67B.\n\nAlignment for MoE. We successfully perform supervised fine-tuning on DeepSeekMoE 16B to create an aligned chat model, showcasing the adaptability and versatility of DeepSeekMoE 16B.\n\nPublic Release. In the spirit of open research, we release the model checkpoint of DeepSeekMoE 16B to the public. Notably, this model can be deployed on a single GPU with 40GB of memory without the need for quantization.\n\n2. Preliminaries: Mixture-Of-Experts For Transformers\n\nWe first introduce a generic MoE architecture commonly used in Transformer language models. A standard Transformer language model is constructed by stacking layers of standard Transformer blocks, where each block can be represented as follows:\n\n$$\\mathbf{u}{1:T}^{l}=\\text{Self-Att}\\left(\\mathbf{h}{1:T}^{l-1}\\right)+\\mathbf{h}{1:T}^{l-1},\\tag{1}$$ $$\\mathbf{h}{t}^{l}=\\text{FFN}\\left(\\mathbf{u}{t}^{l}\\right)+\\mathbf{u}{t}^{l},\\tag{2}$$\n\nwhere denotes the sequence length, Self-Att(·) denotes the self-attention module, FFN(·) denotes the Feed-Forward Network (FFN), u 1: ∈ R× are the hidden states of all tokens after the -th attention module, and h\n\n∈ Ris the output hidden state of the -th token after the -th Transformer block. For brevity, we omit the layer normalization in the above formulations.\n\nA typical practice to construct an MoE language model usually substitutes FFNs in a Transformer with MoE layers at specified intervals (Du et al., 2022; Fedus et al., 2021; Lepikhin et al., 2021; Zoph, 2022). An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN. Then, each token will be assigned to one (Fedus et al., 2021) or two (Lepikhin et al., 2021) experts. If the -th FFN is substituted with an MoE layer, the computation for its output hidden state h\n\nis expressed as: where denotes the total number of experts, FFN(·) is the -th expert FFN, , denotes the gate value for the -th expert, , denotes the token-to-expert affinity, Topk(·, ) denotes the set comprising highest affinity scores among those calculated for the -th token and all experts, and e\n\nis the centroid of the -th expert in the -th layer. Note that ,is sparse, indicating that only out of gate values are nonzero. This sparsity property ensures computational efficiency within an MoE layer, i.e., each token will be assigned to and computed in only experts. Also, in the above formulations, we omit the layer normalization operation for brevity.\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{N}\\left(g_{i,t}\\operatorname{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\operatorname{Topk}({s_{j,t}|1\\leqslant j\\leqslant N},K),\\ 0,&\\text{otherwise},\\end{cases}$$ $$s_{i,t}=\\operatorname{Softmax}{i}\\left(\\mathbf{u}{t}^{l}\\,\\mathbf{e}_{i}^{l}\\right),$$\n\n$$(3)$$\n\n(4) $\\binom{4}{5}$ .\n\n3. Deepseekmoe Architecture\n\nOn top of the generic MoE architecture outlined in Section 2, we introduce DeepSeekMoE, which is specifically designed to exploit the potential of expert specialization. As illustrated in Figure 2, our architecture incorporates two principal strategies: fine-grained expert segmentation and shared expert isolation. Both of these strategies are designed to elevate the level of expert specialization.\n\n3.1. Fine-Grained Expert Segmentation\n\nIn scenarios where the number of experts is limited, tokens assigned to a particular expert will be more likely to cover diverse types of knowledge. As a consequence, the designated expert will intend to learn vastly different types of knowledge in its parameters, and they are hard to be simultaneously utilized. However, if each token can be routed to more experts, diverse knowledge will gain the potential to be decomposed and learned in different experts respectively. In this context, each expert can still retain a high level of expert specialization, contributing to a more focused knowledge distribution across experts.\n\nIn pursuit of the goal, while maintaining a consistent number of expert parameters and computational cost, we segment the experts with a finer grain. The finer expert segmentation enables a more flexible and adaptable combination of activated experts. To be specific, on top of a typical MoE architecture shown in Figure 2(a), we segment each expert FFN into smaller experts by reducing the FFN intermediate hidden dimension to 1 times its original size. Since each expert becomes smaller, in response, we also increase the number of activated experts to times to keep the same computation cost, as illustrated in Figure 2(b). With the fine-grained expert segmentation, the output of an MoE layer can be expressed as:\n\n$$\\mathbf{h}{t}^{i}=\\sum{i=1}^{\\text{mN}}\\left(g_{i,t}\\text{FFN}{i}\\left(\\mathbf{u}{t}^{i}\\right)\\right)+\\mathbf{u}{t}^{i},\\tag{6}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\text{Tokr}(s_{i,t}|1\\leqslant j\\leqslant m\\text{N}),m\\text{K}),\\ 0,&\\text{otherwise},\\end{cases}$$ (7) $$s_{i,t}=\\text{Softmax}{i}\\left(\\mathbf{u}{t}^{T}\\mathbf{e}_{t}^{i}\\right),\\tag{8}$$ where the total number of expert parameters is equal to $N$ times the number of parameters in a\n\n$$(6)$$\n\n(7) (8) $\\frac{1}{2}$\n\nstandard FFN, and denotes the total number of fine-grained experts. With the fine-grained expert segmentation strategy, the number of nonzero gates will also increases to .\n\nFrom a combinatorial perspective, the fine-grained expert segmentation strategy substantially enhances the combinatorial flexibility of activated experts. As an illustrative example, we consider the case where = 16. A typical top-2 routing strategy can yield 16 2 = 120 possible combinations. By contrast, if each expert is split into 4 smaller experts, the fine-grained routing strategy can yield 64 8 = 4, 426, 165, 368 potential combinations. The surge in combinatorial flexibility enhances the potential for achieving more accurate and targeted knowledge acquisition.\n\n3.2. Shared Expert Isolation\n\nWith a conventional routing strategy, tokens assigned to different experts may necessitate some common knowledge or information. As a result, multiple experts may converge in acquiring shared knowledge in their respective parameters, thereby resulting in redundancy in expert parameters. However, if there are shared experts dedicated to capturing and consolidating common knowledge across varying contexts, the parameter redundancy among other routed experts will be alleviated. This alleviation of redundancy will contribute to a more parameterefficient model with more specialized experts.\n\nTowards this objective, in addition to the fine-grained expert segmentation strategy, we further isolate experts to serve as shared experts. Regardless of the router module, each token will be deterministically assigned to these shared experts. In order to maintain a constant computational cost, the number of activated experts among the other routed experts will be decreased by , as depicted in Figure 2(c). With the shared expert isolation strategy integrated, an MoE layer in the complete DeepSeekMoE architecture is formulated as follows:\n\n$$\\mathbf{h}{t}^{l}=\\sum{i=1}^{K_{s}}\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)+\\sum_{i=K_{s}+1}^{mN}\\left(g_{i,t}\\;\\mathrm{FFN}{i}\\left(\\mathbf{u}{t}^{l}\\right)\\right)+\\mathbf{u}{t}^{l},\\tag{9}$$ $$g{i,t}=\\begin{cases}s_{i,t},&s_{i,t}\\in\\mathrm{Topk}{s_{j,t}|K_{s}+1\\leqslant j\\leqslant mN},mK-K_{s}},\\ 0,&\\text{otherwise},\\end{cases}$$ (10) $$s_{i,t}=\\mathrm{Softmax}{i}\\left(\\mathbf{u}{t}^{l,T}e_{i}^{l}\\right).\\tag{11}$$ $\\mathbf{h}\\mathbf{M}\\mathbf{E}$ the number of closed-compact is $K_{s}$ the total number of routed currents.\n\nFinally, in DeepSeekMoE, the number of shared expert is , the total number of routed experts is − , and the number of nonzero gates is − .\n\nIt is worth noting that the prototype of shared expert isolation can be credited to Rajbhandari et al. (2022). The key distinction lies in the fact that they derive this strategy from an engineering perspective, while we approach it from an algorithmic standpoint.\n\n3.3. Load Balance Consideration\n\nAutomatically learned routing strategies may encounter the issue of load imbalance, which manifests two notable defects. Firstly, there is a risk of routing collapse (Shazeer et al., 2017), i.e., the model always selects only a few experts, preventing other experts from sufficient training.\n\nSecondly, if experts are distributed across multiple devices, load imbalance can exacerbate computation bottlenecks.\n\nExpert-Level Balance Loss. In order to mitigate the risk of routing collapse, we also employ an expert-level balance loss. The computation of the balance loss is as follows:\n\n$$\\begin{array}{l}{{{\\mathcal L}{\\mathrm{ExpBal}}=\\alpha{1}\\sum_{i=1}^{N^{\\prime}}f_{i}P_{i},}}\\ {{f_{i}=\\frac{N^{\\prime}}{K^{T}}\\sum_{t=1}^{T}\\mathds{1}\\,(\\mathrm{Token}\\;t\\;\\mathrm{s}\\mathrm{lects}\\;\\mathrm{Expert}\\;i),}}\\ {{P_{i}=\\frac{1}{T}\\sum_{t=1}^{T}s_{i,t},}}\\end{array}$$\n\n(12) $$\\begin{array}{l}\\small\\text{(13)}\\end{array}$$ = (14) $$\\begin{array}{l}\\small\\text{(14)}\\end{array}$$ .\n\n7 where 1 is a hyper-parameter called expert-level balance factor, ′is equal to ( − ) and ′ is equal to ( − ) for brevity. 1(·) denotes the indicator function.\n\nDevice-Level Balance Loss. In addition to the expert-level balance loss, we introduce a devicelevel balance loss. When aiming to alleviate computation bottlenecks, it becomes unnecessary to enforce strict balance constraints at the expert level, because excessive constraints on load balance will compromise model performance. Instead, our primary objective is to ensure balanced computation across the devices. If we partition all routed experts into groups {E1, E2, ..., E}, and deploy each group on a single device, the device-level balance loss is computed as follows:\n\n$$\\mathcal{L}{\\text{DevBal}}=\\alpha{2}\\sum_{i=1}^{D}f_{i}^{\\prime}P_{i}^{\\prime},$$ $$f_{i}^{\\prime}=\\frac{1}{|\\mathcal{E}{i}|}\\sum{j\\in\\mathcal{E}{i}}f{j},$$ $$P_{i}^{\\prime}=\\sum_{j\\in\\mathcal{E}{i}}P{j},$$\n\n(15) $$\\begin{array}{l}\\small\\text{(16)}\\end{array}$$ = (17) .\n\nwhere 2 is a hyper-parameter called device-level balance factor. In practice, we set a small expert-level balance factor to mitigate the risk of routing collapse, and meanwhile set a larger device-level balance factor to promote balanced computation across the devices.\n\n4. Validation Experiments 4.1. Experimental Setup 4.1.1. Training Data And Tokenization\n\nOur training data is sampled from a large-scale multilingual corpus created by DeepSeek-AI. The corpus primarily focuses on English and Chinese but also encompasses other languages. It is derived from diverse sources, including web text, mathematical material, coding scripts, published literature, and various other textual materials. For the purpose of validation experiments, we sample a subset containing 100B tokens from the corpus to train our models. For tokenization, we utilize the HuggingFace Tokenizer2tools to train byte pair encoding (BPE) (Sennrich et al., 2016) tokenizers on a smaller subset of the training corpus. In the validation experiments, we prepare a tokenizer with a vocabulary size of 8K, and the vocabulary size will be scaled up when training larger models.\n\n4.1.2. Infrastructures\n\nWe conduct experiments based on HAI-LLM (High-Flyer, 2023), an efficient and light-weight training framework which integrates multiple parallelism strategies, including tensor parallelism (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019), ZeRO data parallelism (Rajbhandari et al., 2020), PipeDream pipeline parallelism (Harlap et al., 2018), and more specifically, expert parallelism (Lepikhin et al., 2021) by combining data and tensor parallelism.\n\nIn order to optimize performance, we develop GPU kernels with CUDA and Triton (Tillet et al., 2019) for gating algorithms and fusing computations across linear layers in different experts.\n\nAll experiments are carried out on clusters equipped with NVIDIA A100 or H800 GPUs.\n\nEach node in the A100 cluster contains 8 GPUs connected pairwise via the NVLink bridge.\n\nThe H800 cluster also features 8 GPUs per node, interconnected using NVLink and NVSwitch within nodes. For both A100 and H800 clusters, InfiniBand interconnects are utilized to facilitate communication across nodes.\n\n4.1.3. Hyper-Parameters\n\nModel Settings. In the validation experiments, we set the number of Transformer layers to 9 and the hidden dimension to 1280. We employ the multi-head attention mechanism with a total of 10 attention heads, where each head has a dimension of 128. For initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs with MoE layers, and ensure that the total number of expert parameters equals 16 times that of a standard FFN. Additionally, we keep the activated expert parameters, including shared expert parameters and activated routed expert parameters, as 2 times that of a standard FFN. Under this configuration, each MoE model has approximately 2B total parameters, with the number of activated parameters around 0.3B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for validation experiments is set to 1.08 × 10−3, and the gradient clipping norm is set to 1.0. The batch size is set to 2K, and with a maximum sequence length of 2K, each training batch contains 4M tokens. Correspondingly, the total number of training steps is set to 25,000 to achieve 100B training tokens. Due to the abundance of training data, we do not use dropout during training. Given the relatively small model size, all parameters, including expert parameters, are deployed on a single GPU device to avoid unbalanced computation.\n\nCorrespondingly, we do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set an expert-level balance factor of 0.01.\n\nFor readability, we also present an overview table of hyper-parameters for DeepSeekMoE across different sizes in Appendix A.\n\n4.1.4. Evaluation Benchmarks\n\nWe conduct evaluations on a wide range of benchmarks covering various types of tasks. We list the benchmarks as follows.\n\nLanguage Modeling. For language modeling, we evaluate the models on the test set of Pile (Gao et al., 2020), and the evaluation metric is the cross-entropy loss.\n\nLanguage Understanding and Reasoning. For language understanding and reasoning, we consider HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge and ARCeasy (Clark et al., 2018). The evaluation metric for these tasks is accuracy.\n\nReading Comprehension. For reading comprehension, we use RACE-high and RACE-middle Lai et al. (2017), and the evaluation metric is accuracy.\n\nCode Generation. For code generation, we evaluate the models on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021). The evaluation metric is Pass@1, which represents the pass rate for only one generation attempt.\n\nClosed-Book Question Answering. For closed-book question answering, we consider TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\n4.2. Evaluations\n\nBaselines. Including DeepSeekMoE, we compare five models for validation experiments.\n\nDense denotes a standard dense Transformer language model with 0.2B total parameters. Hash Layer (Roller et al., 2021) is an MoE architecture based on top-1 hash routing, with 2.0B total parameters and 0.2B activated parameters, aligned with the dense baseline. Switch Transformer (Fedus et al., 2021) is another well-known MoE architecture based on top-1 learnable routing, with total parameters and activated parameters the same as Hash Layer. GShard (Lepikhin et al., 2021) employs a top-2 learnable routing strategy, with 2.0B total parameters and 0.3B activated parameters since one more expert is activated compared to top-1 routing methods. DeepSeekMoE has 1 shared expert and 63 routed experts, where each expert is 0.25 times the size of a standard FFN. Including DeepSeekMoE, all compared models share the same training corpus and training hyper-parameters. All compared MoE models have the same number of total parameters, and GShard has the same number of activated parameters as DeepSeekMoE.\n\nResults. We present the evaluation results in Table 1. For all demonstrated models, we report the final evaluation results after training on 100B tokens. From the table, we make the following observations: (1) With sparse architectures and more total parameters, Hash Layer\n\nMetric # Shot Dense Hash Layer Switch GShard DeepSeekMoE # Total Params N/A 0.2B 2.0B 2.0B 2.0B 2.0B # Activated Params N/A 0.2B 0.2B 0.2B 0.3B 0.3B FLOPs per 2K Tokens N/A 2.9T 2.9T 2.9T 4.3T 4.3T # Training Tokens N/A 100B 100B 100B 100B 100B Pile (Loss) N/A 2.060 1.932 1.881 1.867 1.808 HellaSwag (Acc.) 0-shot 38.8 46.2 49.1 50.5 54.8 PIQA (Acc.) 0-shot 66.8 68.4 70.5 70.6 72.3 ARC-easy (Acc.) 0-shot 41.0 45.3 45.9 43.9 49.4 ARC-challenge (Acc.) 0-shot 26.0 28.2 30.2 31.6 34.3 RACE-middle (Acc.) 5-shot 38.8 38.8 43.6 42.1 44.0 RACE-high (Acc.) 5-shot 29.0 30.0 30.9 30.4 31.7 HumanEval (Pass@1) 0-shot 0.0 1.2 2.4 3.7 4.9 MBPP (Pass@1) 3-shot 0.2 0.6 0.4 0.2 2.2 TriviaQA (EM) 5-shot 4.9 6.5 8.9 10.2 16.6 NaturalQuestions (EM) 5-shot 1.4 1.4 2.5 3.2 5.7\n\nTable 1 | Evaluation results for validation experiments. Bold font indicates the best. Compared with other MoE architectures, DeepSeekMoE exhibits a substantial performance advantage.\n\nand Switch Transformer achieve significantly stronger performance than the dense baseline with the same number of activated parameters. (2) Compared with Hash Layer and Switch Transformer, GShard has more activated parameters and achieves slightly better performance than Switch Transformer. (3) With the same number of total parameters and activated parameters, DeepSeekMoE demonstrates overwhelming advantages over GShard. These results showcase the superiority of our DeepSeekMoE architecture within the existing landscape of MoE architectures.\n\n4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models\n\nWe have demonstrated that DeepSeekMoE outperforms the dense baseline and other MoE architectures. In order to provide a more precise understanding of the performance of DeepSeekMoE, we compare it with larger baselines with more total parameters or activated parameters. The comparisons enable us to estimate the required model size of GShard or dense baselines to achieve equivalent performance to DeepSeekMoE.\n\nComparison with GShard×1.5. Table 2 shows the comparison between DeepSeekMoE and a larger GShard model with 1.5 times the expert size, which results in 1.5 times both expert parameters and expert computation. Overall, we observe that DeepSeekMoE achieves comparable performance with GShard×1.5, underscoring the significant advantage inherent in the DeepSeekMoE architecture. In addition to the comparison with GShard×1.5, we also show the comparison with GShard×1.2 in Appendix B.\n\nFurthermore, we increase the number of total parameters of DeepSeekMoE to 13.3B and compare it with GShard×1.2 and GShard×1.5 with 15.9B and 19.8B total parameters, respectively.\n\nWe find that at a larger scale, DeepSeekMoE can even outperform GShard×1.5 distinctly. These\n\nMetric # Shot GShard×1.5 Dense×16 DeepSeekMoE Relative Expert Size N/A 1.5 1 0.25 # Experts N/A 0 + 16 16 + 0 1 + 63 # Activated Experts N/A 0 + 2 16 + 0 1 + 7 # Total Expert Params N/A 2.83B 1.89B 1.89B # Activated Expert Params N/A 0.35B 1.89B 0.24B FLOPs per 2K Tokens N/A 5.8T 24.6T 4.3T # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.808 1.806 1.808 HellaSwag (Acc.) 0-shot 54.4 55.1 54.8 PIQA (Acc.) 0-shot 71.1 71.9 72.3 ARC-easy (Acc.) 0-shot 47.3 51.9 49.4 ARC-challenge (Acc.) 0-shot 34.1 33.8 34.3 RACE-middle (Acc.) 5-shot 46.4 46.3 44.0 RACE-high (Acc.) 5-shot 32.4 33.0 31.7 HumanEval (Pass@1) 0-shot 3.0 4.3 4.9 MBPP (Pass@1) 3-shot 2.6 2.2 2.2 TriviaQA (EM) 5-shot 15.7 16.5 16.6 NaturalQuestions (EM) 5-shot 4.7 6.3 5.7\n\nTable 2 | Comparisons among DeepSeekMoE, larger GShard models, and larger dense models.\n\nIn the line of \"# Experts\", + denotes shared experts and routed experts. In the line of \"# Activated Experts\", + denotes activated shared experts and activated routed experts. DeepSeekMoE achieves comparable performance with a GShard model containing 1.5 times expert parameters and computation. In addition, DeepSeekMoE nearly approaches the performance of a dense model with 16 times FFN parameters, which sets the upper bound for MoE models in terms of the model capacity.\n\nResults Are Also Provided In Appendix B.\n\nComparison with Dense×16. Table 2 also shows the comparison between DeepSeekMoE and larger dense models. For a fair comparison, we do not use the widely used ratio (1:2) between the attention and FFN parameters. Instead, we configure 16 shared experts where each expert has the same number of parameters as a standard FFN. This architecture mimics a dense model with 16 times standard FFN parameters. From the table, we find that DeepSeekMoE nearly approaches the performance of Dense×16, which sets the strict upper bound of MoE models in terms of the model capacity. These results suggest that, at least at the scale of about 2B parameters and 100B training tokens, the performance of DeepSeekMoE aligns closely with the theoretical upper bound of MoE models. Also, we provide additional comparisons with Dense×4 in Appendix B.\n\n4.4. Ablation Studies\n\nIn order to substantiate the effectiveness of the fine-grained expert segmentation and shared expert isolation strategies, we conduct ablation studies for DeepSeekMoE and present the results in Figure 3. For a fair comparison, we ensure all models included in the comparison have the\n\nsame number of total parameters and activated parameters.\n\nShared Expert Isolation. In order to evaluate the influence of the shared expert isolation strategy, we isolate one expert as the shared one based on GShard. From Figure 3, we observe that compared with GShard, the intentional isolation of a shared expert yields improved performance across a majority of benchmarks. These results support the proposition that the shared expert isolation strategy contributes to a stronger model performance.\n\nFine-Grained Expert Segmentation. In order to assess the effectiveness of the fine-grained expert segmentation strategy, we conduct a more detailed comparison by further segmenting the experts into a finer grain. To be specific, we segment each expert into 2 or 4 smaller experts, resulting in a total of 32 (1 shared + 31 routed) or 64 (1 shared + 63 routed) experts. Figure 3 reveals a consistent trend that the continuous refinement of expert segmentation granularity corresponds to a continuous enhancement in overall model performance. These findings provide empirical substantiation for the effectiveness of the fine-grained expert segmentation strategy.\n\nRatios Between Shared and Routed Experts. In addition, we investigate the best ratio of shared experts and routed experts. Based on the finest granularity with 64 total experts and keeping the number of total experts and activated experts constant, we attempt to isolate 1, 2, and 4 experts as shared ones. We find that different ratios of the shared experts and routed experts do not significantly impact the performance, and 1, 2, and 4 shared experts achieve a Pile loss of 1.808, 1.806, and 1.811, respectively. Considering that the ratio of 1:3 yields a marginally better Pile loss, when scaling up DeepSeekMoE, we keep the ratio between shared experts and activated routed experts as 1:3.\n\n4.5. Analysis On Expert Specialization\n\nIn this section, we conduct an empirical analysis on the expert specialization of DeepSeekMoE 2B. DeepSeekMoE 2B in this section refers to the model reported in Table 1, i.e., comprising 2.0B total parameters, with 1 shared expert and 7 out of 63 routed experts being activated.\n\nDeepSeekMoE Exhibits Lower Redundancy Among Routed Experts. In order to assess the redundancy among routed experts, we disable varying ratios of top routed experts and evaluate the Pile loss. To be specific, for each token, we mask a certain ratio of experts with the highest routing probability, and then select top-K experts from the remaining routed experts. For fairness, we compare DeepSeekMoE with GShard×1.5 since they have the same Pile loss when no experts are disabled. As shown in Figure 4, compared with GShard×1.5, DeepSeekMoE is more sensitive to the disabling of top routed experts. This sensitivity suggests a lower level of parameter redundancy in DeepSeekMoE, since each routed expert is more irreplaceable. In contrast, GShard×1.5 exhibits greater redundancy among its expert parameters, so it can buffer the performance drop when top routed experts are disabled.\n\nShared Experts Are Irreplaceable by Routed Experts. In order to investigate the role of the shared expert in DeepSeekMoE, we disable it and activate one more routed expert. The evaluation on Pile shows a significant increase in the Pile loss, rising from 1.808 to 2.414, even though we maintain the same computational cost. This result highlights the crucial function of the shared expert and indicates that the shared expert captures fundamental and essential knowledge not shared with routed experts, making it irreplaceable by routed ones.\n\nDeepSeekMoE Acquires Knowledge More Accurately. In order to validate our claim that higher flexibility in combining activated experts contributes to a more accurate and targeted knowledge acquisition, we investigate whether DeepSeekMoE can acquire requisite knowledge with fewer activated experts. To be specific, we vary the number of activated routed experts from 3 to 7 and evaluate the resulting Pile loss. As demonstrated in Figure 5, even with only\n\n4 routed experts activated, DeepSeekMoE achieves a Pile loss comparable with GShard. This observation supports the proposition that DeepSeekMoE can acquire requisite knowledge more accurately and efficiently.\n\nEncouraged by these findings, in order to validate the expert specialization and accurate knowledge acquisition of DeepSeekMoE more rigorously, we train a new model from scratch.\n\nThis model comprises 1 shared expert and 63 routed experts, where only 3 routed experts are activated. The evaluation results shown in Figure 6 demonstrate that, even with the same total expert parameters and only half of the activated expert parameters, DeepSeekMoE still outperforms GShard. This highlights the ability of DeepSeekMoE to leverage expert parameters more efficiently, i.e., the proportion of effective parameters in the activated experts is much higher than that of GShard.\n\n5. Scaling Up To Deepseekmoe 16B\n\nWith the DeepSeekMoE architecture, we scale up our MoE model to a larger scale with 16B total parameters and train it on 2T tokens. Our results demonstrate that compared with LLaMA2 7B, DeepSeekMoE 16B achieves superior performance with only about 40% of computations.\n\n5.1. Experimental Setup 5.1.1. Training Data And Tokenization\n\nWe sample the training data from the same corpus as described in Section 4.1.1. Different from the validation experiments, we sample a larger amount of data with 2T tokens, aligning with the number of training tokens of LLaMA2 7B. We also use the HuggingFace Tokenizer tools to train a BPE tokenizer, but the vocabulary size is set to 100K for DeepSeekMoE 16B.\n\n5.1.2. Hyper-Parameters\n\nModel Settings. For DeepSeekMoE 16B, we set the number of Transformer layers to 28 and the hidden dimension to 2048. We employ the multi-head attention mechanism with a total of 16 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. We substitute all FFNs except for the first layer with MoE layers, since we observe that the load balance status converges especially slower for the first layer. Each MoE layer consists of 2 shared experts and 64 routed experts, where each expert is 0.25 times the size of a standard FFN. Each token will be routed to these 2 shared experts and 6 out of 64 routed experts. An even finer expert segmentation granularity is not employed due to the potential reduction in computational efficiency associated with excessively small expert sizes. At a larger scale over 16B, a finer granularity can still be employed. Under our configuration, DeepSeekMoE 16B has approximately 16.4B total parameters, with the number of activated parameters around 2.8B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. The learning rate is also scheduled using a warmup-and-step-decay strategy. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps. Subsequently, the learning rate is multiplied by 0.316 at 80% of the training steps, and again by 0.316 at 90% of the training steps.\n\nThe maximum learning rate for DeepSeekMoE 16B is set to 4.2 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. Correspondingly, the total number of training steps is set to 106,449 to achieve 2T training tokens. Due to the abundance of training data, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the experts will be deployed on the same device.\n\nTherefore, we also do not drop any tokens during training and do not employ the device-level balance loss. In order to prevent routing collapse, we set a quite small expert-level balance factor of 0.001 because we find that under our parallelization strategy, a higher expert-level balance factor cannot increase the computation efficiency, but instead, it will compromise the model performance.\n\n5.1.3. Evaluation Benchmarks\n\nIn addition to the benchmarks used in the validation experiments, we incorporate additional benchmarks for a more comprehensive evaluation. We introduce the distinctions from the benchmarks used in validation experiments as follows.\n\nLanguage Modeling. For language modeling, we also evaluate the models on the test set of Pile (Gao et al., 2020). Since the tokenizer used in DeepSeekMoE 16B is different from that used in LLaMA2 7B. For a fair comparison, we use bits per byte (BPB) as the evaluation metric.\n\nReading Comprehension. For reading comprehension, we additionally consider DROP (Dua et al., 2019). The evaluation metric is the Exactly Matching (EM) rate.\n\nMath Reasoning. For math reasoning, we additionally incorporate GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), using EM as the evaluation metric.\n\nMulti-Subject Multiple-Choice. For multi-subject multiple-choice, we additionally evaluate the models on MMLU (Hendrycks et al., 2020). The evaluation metric is accuracy.\n\nDisambiguation. For disambiguation, we additionally consider WinoGrande (Sakaguchi et al., 2019) and the evaluation metric is accuracy.\n\nChinese Benchmarks. Since DeepSeekMoE 16B is pretrained on a bilingual corpus, we also evaluate it on four Chinese benchmarks. CLUEWSC (Xu et al., 2020) is a Chinese disambiguation benchmark. CEval (Huang et al., 2023) and CMMLU (Li et al., 2023) are two Chinese multisubject multiple-choice benchmarks with a similar form to MMLU. CHID (Zheng et al., 2019) is a Chinese idiom completion benchmark, aiming to evaluate the understanding of Chinese culture. The evaluation metrics for the aforementioned Chinese benchmarks are accuracy or EM.\n\nOpen LLM Leaderboard. We evaluate all of the aforementioned benchmarks based on our internal evaluation framework. In order to compare DeepSeekMoE 16B with open source models fairly and conveniently, we additionally evaluate DeepSeekMoE 16B on the Open LLM Leaderboard. The Open LLM Leaderboard is a public leaderboard supported by HuggingFace, it consists of six tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021).\n\n5.2. Evaluations 5.2.1. Internal Comparison With Deepseek 7B\n\nWe first conduct an internal comparison between DeepSeekMoE 16B and DeepSeek 7B (DeepSeekAI, 2024), a dense language model with 6.9B parameters. Ensuring fairness, both models are trained on the same corpus with 2T tokens. This enables an accurate assessment of the effectiveness of our MoE architecture, independent of the influence of the training data.\n\nMetric # Shot DeepSeek 7B (Dense) DeepSeekMoE 16B # Total Params N/A 6.9B 16.4B # Activated Params N/A 6.9B 2.8B FLOPs per 4K Tokens N/A 183.5T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.75 0.74 HellaSwag (Acc.) 0-shot 75.4 77.1 PIQA (Acc.) 0-shot 79.2 80.2 ARC-easy (Acc.) 0-shot 67.9 68.1 ARC-challenge (Acc.) 0-shot 48.1 49.8 RACE-middle (Acc.) 5-shot 63.2 61.9 RACE-high (Acc.) 5-shot 46.5 46.4 DROP (EM) 1-shot 34.9 32.9 GSM8K (EM) 8-shot 17.4 18.8 MATH (EM) 4-shot 3.3 4.3 HumanEval (Pass@1) 0-shot 26.2 26.8 MBPP (Pass@1) 3-shot 39.0 39.2 TriviaQA (EM) 5-shot 59.7 64.8 NaturalQuestions (EM) 5-shot 22.2 25.5 MMLU (Acc.) 5-shot 48.2 45.0 WinoGrande (Acc.) 0-shot 70.5 70.2 CLUEWSC (EM) 5-shot 73.1 72.1 CEval (Acc.) 5-shot 45.0 40.6 CMMLU (Acc.) 5-shot 47.2 42.5 CHID (Acc.) 0-shot 89.3 89.4\n\nTable 3 | Comparison between DeepSeek 7B and DeepSeekMoE 16B. Bold font indicates the best or near the best. With only 40.5% of computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B.\n\nThe evaluation results are presented in Table 3, yielding the following observations: (1) On the whole, with about only 40% of the computations, DeepSeekMoE 16B achieves comparable performance with DeepSeek 7B. (2) DeepSeekMoE 16B exhibits notable strengths in language modeling and knowledge-intensive tasks such as Pile, HellaSwag, TriviaQA, and NaturalQuestions. Given that in an MoE model, FFN parameters are much heavier than attention parameters, these outcomes align with the proposition that FFNs in Transformers exhibit the capability for knowledge memorization (Dai et al., 2022a). (3) Compared with the excellent performance on other tasks, DeepSeekMoE exhibits limitations in addressing multiple-choice tasks. This inadequacy stems from the limited attention parameters in DeepSeekMoE 16B (DeepSeekMoE 16B has only about 0.5B attention parameters, while DeepSeek 7B has 2.5B attention parameters).\n\nOur earlier investigation on DeepSeek 7B reveals a positive correlation between the attention capacity and performance on multiple-choice tasks. For example, DeepSeek 7B MQA, which is equipped with the multi-query attention mechanism (Shazeer, 2019), also struggled in MMLUlike tasks. In addition, for a more comprehensive understanding of the training process of DeepSeekMoE 16B, we also provide the benchmark curves of DeepSeekMoE 16B and DeepSeek 7B (Dense) during training in Appendix C for reference.\n\nCritically, due to the modest number of parameters in DeepSeekMoE 16B, it enables singledevice deployment on a GPU with 40GB of memory. With appropriate operator optimizations, it can achieve nearly 2.5 times the inference speed of a 7B dense model.\n\nMetric # Shot LLaMA2 7B DeepSeekMoE 16B # Total Params N/A 6.7B 16.4B # Activated Params N/A 6.7B 2.8B FLOPs per 4K Tokens N/A 187.9T 74.4T # Training Tokens N/A 2T 2T Pile (BPB) N/A 0.76 0.74 HellaSwag (Acc.) 0-shot 75.6 77.1 PIQA (Acc.) 0-shot 78.0 80.2 ARC-easy (Acc.) 0-shot 69.1 68.1 ARC-challenge (Acc.) 0-shot 49.0 49.8 RACE-middle (Acc.) 5-shot 60.7 61.9 RACE-high (Acc.) 5-shot 45.8 46.4 DROP (EM) 1-shot 34.0 32.9 GSM8K (EM) 8-shot 15.5 18.8 MATH (EM) 4-shot 2.6 4.3 HumanEval (Pass@1) 0-shot 14.6 26.8 MBPP (Pass@1) 3-shot 21.8 39.2 TriviaQA (EM) 5-shot 63.8 64.8 NaturalQuestions (EM) 5-shot 25.5 25.5 MMLU (Acc.) 5-shot 45.8 45.0 WinoGrande (Acc.) 0-shot 69.6 70.2 CLUEWSC (EM) 5-shot 64.0 72.1 CEval (Acc.) 5-shot 33.9 40.6 CMMLU (Acc.) 5-shot 32.6 42.5 CHID (Acc.) 0-shot 37.9 89.4\n\nTable 4 | Comparison between LLaMA2 7B and DeepSeekMoE 16B. With only 39.6% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks.\n\n5.2.2. Comparison With Open Source Models\n\nInternal Comparison with LLaMA2 7B. In the realm of open source models, we mainly compare DeepSeekMoE 16B with LLaMA2 7B (Touvron et al., 2023b), a well-known and strong open source language model with 6.7B parameters. Both DeepSeekMoE 16B and LLaMA2 7B are pretrained on 2T tokens. Compared with LLaMA2 7B, DeepSeekMoE has 245% of total parameters but only needs 39.6% of computations. The results on our internal benchmarks are presented in Table 4, leading to the following observations. (1) Among the evaluated benchmarks, with only about 40% of computations, DeepSeekMoE 16B outperforms LLaMA2 7B on the majority of benchmarks. (2) The math reasoning and code generation capabilities of DeepSeekMoE 16B are stronger than LLaMA2 7B, attributed to the enriched presence of mathematical and coderelated text in our pretraining corpus. (3) Given the presence of Chinese texts in our pretraining corpus, DeepSeekMoE 16B exhibits a substantial performance advantage over LLaMA2 7B on Chinese benchmarks. (4) Despite being trained on fewer English texts, DeepSeekMoE 16B achieves comparable or better performance compared with LLaMA2 7B on English understanding or knowledge-intensive benchmarks, which demonstrates the exceptional capabilities of DeepSeekMoE 16B.\n\nEvaluation on Open LLM Leaderboard. Beyond our internal evaluations, we also evaluate DeepSeekMoE 16B on the Open LLM Leaderboard and compare it with other open source models. In addition to LLaMA2 7B, we take a broader set of open source models into consideration, including LLaMA 7B (Touvron et al., 2023a), Falcon 7B (Almazrouei et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), RedPajama-INCITE 7B and 3B (Together-AI, 2023), Open LLaMA 7B and 3B (Geng and Liu, 2023), OPT 2.7B (Zhang et al., 2022), Pythia 2.8B (Biderman et al., 2023), GPT-neo 2.7B (Black et al., 2021), and BLOOM 3B (Scao et al., 2022). The evaluation results, as presented in Figure 1, show that DeepSeekMoE 16B consistently outperforms models with similar activated parameters by a large margin. Moreover, it achieves comparable performance with LLaMA2 7B, which has approximately 2.5 times the activated parameters.\n\n6. Alignment For Deepseekmoe 16B\n\nPrevious research indicates that MoE models typically do not emerge significant gains from fine-tuning (Artetxe et al., 2022; Fedus et al., 2021). However, Shen et al. (2023) present findings suggesting that MoE models can indeed benefit from instruction tuning. In order to assess whether DeepSeekMoE 16B can benefit from fine-tuning, we conduct supervised fine-tuning to construct a chat model based on DeepSeekMoE 16B. The experimental results reveal that DeepSeekMoE Chat 16B also achieves comparable performance with LLaMA2 SFT 7B and DeepSeek Chat 7B.\n\n6.1. Experimental Setup\n\nTraining Data. For training the chat model, we conduct supervised fine-tuning (SFT) on our in-house curated data, comprising 1.4M training examples. This dataset spans a broad range of categories including math, code, writing, question answering, reasoning, summarization, and more. The majority of our SFT training data is in English and Chinese, rendering the chat model versatile and applicable in bilingual scenarios.\n\nHyper-Parameters. During supervised fine-tuning, we set the batch size to 1024 examples and conduct training over 8 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019).\n\nWe employ a maximum sequence length of 4K, and pack the training examples as densely as possible until reaching the sequence length limit. We do not use dropout for supervised fine-tuning, and simply set a constant learning rate of 10−5 without incorporating any learning rate scheduling strategy.\n\nEvaluation Benchmarks. For the evaluation of the chat models, we employ benchmarks similar to those used in Section 5.1.3, with the following adjustments: (1) We exclude Pile (Gao et al., 2020) since chat models are seldom employed for pure language modeling. (2) We exclude\n\nMetric # Shot LLaMA2 DeepSeek DeepSeekMoE SFT 7B Chat 7B Chat 16B # Total Params N/A 6.7B 6.9B 16.4B # Activated Params N/A 6.7B 6.9B 2.8B FLOPs per 4K Tokens N/A 187.9T 183.5T 74.4T HellaSwag (Acc.) 0-shot 67.9 71.0 72.2 PIQA (Acc.) 0-shot 76.9 78.4 79.7 ARC-easy (Acc.) 0-shot 69.7 70.2 69.9 ARC-challenge (Acc.) 0-shot 50.8 50.2 50.0 BBH (EM) 3-shot 39.3 43.1 42.2 RACE-middle (Acc.) 5-shot 63.9 66.1 64.8 RACE-high (Acc.) 5-shot 49.6 50.8 50.6 DROP (EM) 1-shot 40.0 41.7 33.8 GSM8K (EM) 0-shot 63.4 62.6 62.2 MATH (EM) 4-shot 13.5 14.7 15.2 HumanEval (Pass@1) 0-shot 35.4 45.1 45.7 MBPP (Pass@1) 3-shot 27.8 39.0 46.2 TriviaQA (EM) 5-shot 60.1 59.5 63.3 NaturalQuestions (EM) 0-shot 35.2 32.7 35.1 MMLU (Acc.) 0-shot 50.0 49.7 47.2 WinoGrande (Acc.) 0-shot 65.1 68.4 69.0 CLUEWSC (EM) 5-shot 48.4 66.2 68.2 CEval (Acc.) 0-shot 35.1 44.7 40.0 CMMLU (Acc.) 0-shot 36.9 51.2 49.3\n\nCHID (Zheng et al., 2019) due to the observed instability of results, hindering the derivation of solid conclusions. (3) We additionally include BBH (Suzgun et al., 2022) to provide a more comprehensive assessment of the reasoning ability of the chat models.\n\nTable 5 | Comparison among LLaMA2 SFT 7B, DeepSeek Chat 7B and DeepSeekMoE Chat 16B, with all of these three models fine-tuned on the same SFT data. Compared with both 7B dense models, DeepSeekMoE Chat 16B still achieves comparable or better performance on the majority of benchmarks with only 40% of computations.\n\n6.2. Evaluations\n\nBaselines. In order to validate the potential of DeepSeekMoE 16B after alignment, we conduct supervised fine-tuning for LLaMA2 7B, DeepSeek 7B, and DeepSeekMoE 16B, where we utilize totally the same fine-tuning data to ensure fairness. Correspondingly, we construct three chat models, including LLaMA2 SFT 7B3, DeepSeek Chat 7B, and DeepSeekMoE Chat 16B.\n\nSubsequently, we compare DeepSeekMoE Chat 16B with the other two dense chat models (with about 2.5 times the FLOPs) across a wide range of downstream tasks.\n\nResults. The evaluation results are presented in Table 5. Our key observations include: (1) DeepSeekMoE Chat 16B, while consuming nearly 40% of computations, achieves comparable performance with 7B dense models across language understanding and reasoning (PIQA, ARC, BBH), machine reading comprehension (RACE), mathematical (GSM8K, MATH), and knowledge-intensive tasks (TriviaQA, NaturalQuestions). (2) On code generation tasks, DeepSeekMoE Chat 16B significantly outperforms LLaMA2 SFT 7B, demonstrating notable improvements on HumanEval and MBPP. In addition, it also surpasses DeepSeek Chat 7B. (3) On multiple-choice question answering benchmarks including MMLU, CEval, and CMMLU, DeepSeekMoE Chat 16B still falls behind DeepSeek Chat 7B, consistent with the observations for the base model (Section 5.2.1). However, it is worth noting that, after supervised finetuning, the performance gap between DeepSeekMoE 16B and DeepSeek 7B is narrowed. (4) Benefiting from the pretraining on a bilingual corpus, DeepSeekMoE Chat 16B notably outperforms LLaMA2 SFT 7B on all Chinese benchmarks. These results demonstrate the balanced capabilities of DeepSeekMoE 16B in both Chinese and English, enhancing its versatility and applicability in diverse scenarios. In conclusion, the evaluation for the chat models highlights the potential of DeepSeekMoE 16B in benefiting from alignment, and validates its consistent advantages in achieving comparable performance with dense models while using only about 40% of computations.\n\n7. Deepseekmoe 145B Ongoing\n\nEncouraged by the outstanding performance of DeepSeekMoE 16B, we further undertake a preliminary endeavor to scale up DeepSeekMoE to 145B. In this initial study, DeepSeekMoE 145B is trained on 245B tokens, but it has demonstrated consistent advantages over the GShard architecture and shown promise to match or exceed the performance of DeepSeek 67B (Dense). Furthermore, upon the completion of the final version and full training of DeepSeekMoE 145B, we also plan to make it publicly available.\n\n7.1. Experimental Setup\n\nTraining Data and Tokenization. For DeepSeekMoE 145B, we employ exactly the same training corpus and tokenizer as DeepSeekMoE 16B, with the only difference being that DeepSeekMoE 145B is trained on 245B tokens for an initial study.\n\nModel Settings. For DeepSeekMoE 145B, we set the number of Transformer layers to 62 and the hidden dimension to 4096. We employ the multi-head attention mechanism with a total of 32 attention heads, where each head has a dimension of 128. As for initialization, all learnable parameters are randomly initialized with a standard deviation of 0.006. As in DeepSeekMoE 16B, we also substitute all FFNs except for the first layer with MoE layers. Each MoE layer consists of 4 shared experts and 128 routed experts, where each expert is 0.125 times the size of a standard FFN. Each token will be routed to these 4 shared experts and 12 out of 128 routed experts. Under this configuration, DeepSeekMoE 145 has approximately 144.6B total parameters, with the number of activated parameters around 22.2B.\n\nTraining Settings. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with hyper-parameters set to 1 = 0.9, 2 = 0.95, and weight_decay = 0.1. For the preliminary study of DeepSeekMoE 145B, we employ a warmup-and-constant learning rate scheduler. Initially, the learning rate linearly increases from 0 to the maximum value during the first 2K steps.\n\nSubsequently, the learning rate keeps constant during the remaining training process. The maximum learning rate for DeepSeekMoE 145B is set to 3.0 × 10−4, and the gradient clipping norm is set to 1.0. The batch size is set to 4.5K, and with a maximum sequence length of 4K, each training batch contains 18M tokens. We train DeepSeekMoE 145B for 13,000 steps, achieving 245B training tokens. Also, we do not use dropout during training. We leverage pipeline parallelism to deploy different layers of a model on different devices, and for each layer, all the routed experts will be uniformly deployed on 4 devices (i.e., expert parallelism combined with data parallelism). Since we employ expert parallelism for DeepSeekMoE 145B, the device-level load balance should be considered to reduce the computational bottleneck. In response, we set the device-level balance factor to 0.05 to encourage balanced computation across devices. Also, we still set a small expert-level balance factor of 0.003 to prevent routing collapse.\n\nEvaluation Benchmarks. We evaluate DeepSeekMoE 145B on exactly the same internal benchmarks as used for DeepSeekMoE 16B (see Section 5.1.3).\n\n7.2. Evaluations\n\nBaselines. Apart from DeepSeekMoE 145B, we consider three additional models for comparison. DeepSeek 67B (Dense) is a dense model with 67.4B total parameters (refer to DeepSeek-AI (2024) for the model and training details). GShard 137B shares the same hidden dimension and number of layers as DeepSeekMoE 145B, but follows the GShard architecture. Note that DeepSeekMoE 145B aligns the intermediate hidden dimension in each expert to a multiple of 64 for computation efficiency, so its model size is 6% larger than GShard 137B. DeepSeekMoE 142B (Half Activated) has a similar architecture to DeepSeekMoE 145B, but it contains only 2 shared experts, and only 6 out of 128 routed experts are activated. It is noteworthy that all compared models, including DeepSeekMoE 145B, share the same training corpus. In addition, all MoE models in the comparison are trained from scratch and share the same training hyper-parameters.\n\nResults. From the evaluation results presented in Table 6, we have the following observations: (1) Despite having comparable total parameters and computations, DeepSeekMoE 145B significantly outperforms GShard 137B, highlighting the advantages of the DeepSeekMoE architecture again. (2) On the whole, with only 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B (Dense). Consistent with the findings from DeepSeekMoE 16B, DeepSeekMoE 145B exhibits remarkable strengths in language modeling and knowledge-intensive tasks, but with limitations in multiple-choice tasks. (3) At a larger scale, the performance of DeepSeekMoE 142B (Half Activated) does not lag behind too much from DeepSeekMoE 145B. In addition, despite having only a half of activated expert parameters, DeepSeekMoE 142B (Half Activated) still match the performance of DeepSeek 67B (Dense), with only 18.2% of computations. It also outperforms GShard 137B, which aligns with the conclusion from Section 4.5.\n\n8. Related Work\n\nThe Mixture of Experts (MoE) technique is first proposed by Jacobs et al. (1991); Jordan and Jacobs (1994) to deal with different samples with independent expert modules. Shazeer et al. (2017) introduce MoE into language model training and build a large-scale LSTM-based (Hochreiter and Schmidhuber, 1997) MoE models. As Transformer become the most popular architecture\n\nMetric # Shot DeepSeek GShard DeepSeekMoE DeepSeekMoE 142B 67B (Dense) 137B 145B (Half Activated) # Total Params N/A 67.4B 136.5B 144.6B 142.3B # Activated Params N/A 67.4B 21.6B 22.2B 12.2B Relative Expert Size N/A N/A 1 0.125 0.125 # Experts N/A N/A 0 + 16 4 + 128 2 + 128 # Activated Experts N/A N/A 0 + 2 4 + 12 2 + 6 FLOPs per 4K Tokens N/A 2057.5T 572.7T 585.6T 374.6T # Training Tokens N/A 245B 245B 245B 245B Pile (Loss.) N/A 1.905 1.961 1.876 1.888 HellaSwag (Acc.) 0-shot 74.8 72.0 75.8 74.9 PIQA (Acc.) 0-shot 79.8 77.6 80.7 80.2 ARC-easy (Acc.) 0-shot 69.0 64.0 69.7 67.9 ARC-challenge (Acc.) 0-shot 50.4 45.8 48.8 49.0 RACE-middle (Acc.) 5-shot 63.2 59.2 62.1 59.5 RACE-high (Acc.) 5-shot 46.9 43.5 45.5 42.6 DROP (EM) 1-shot 27.5 21.6 27.8 28.9 GSM8K (EM) 8-shot 11.8 6.4 12.2 13.8 MATH (EM) 4-shot 2.1 1.6 3.1 2.8 HumanEval (Pass@1) 0-shot 23.8 17.7 19.5 23.2 MBPP (Pass@1) 3-shot 33.6 27.6 33.2 32.0 TriviaQA (EM) 5-shot 57.2 52.5 61.1 59.8 NaturalQuestions (EM) 5-shot 22.6 19.0 25.0 23.5 MMLU (Acc.) 5-shot 45.1 26.3 39.4 37.5 WinoGrande (Acc.) 0-shot 70.7 67.6 71.9 70.8 CLUEWSC (EM) 5-shot 69.1 65.7 71.9 72.6 CEval (Acc.) 5-shot 40.3 26.2 37.1 32.8 CMMLU (Acc.) 5-shot 40.6 25.4 35.9 31.9 CHID (Acc.) 0-shot 88.5 86.9 90.3 88.3\n\nTable 6 | Comparison among DeepSeek 67B (Dense) and MoE models at the scale of about 140B total parameters. In the lines of \"# Experts\" and \"# Activated Experts\", + denotes shared experts and routed experts, respectively. Bold font indicates the best or near the best performance excluding the last column. DeepSeekMoE 145B, and even DeepSeekMoE 142B (Half Activated) that has only a half of activated expert parameters, outperform GShard 137B by a large margin. Moreover, with 28.5% of computations, DeepSeekMoE 145B achieves comparable performance with DeepSeek 67B.\n\nfor NLP, many attempts extend FFNs in a Transformer as MoE layers to build MoE language models. GShard (Lepikhin et al., 2021) and Switch Transformer (Fedus et al., 2021) are pioneers which employ learnable top-2 or top-1 routing strategies to scale the MoE language models to an extremely large scale. Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022b) use fixed routing strategies for more stable routing and training. Zhou et al. (2022) propose an expert-choice routing strategy, where each token can be assigned to different numbers of experts.\n\nZoph (2022) focus on the issues of training instability and fine-tuning difficulty in MoE models, and propose ST-MoE to overcome these challenges. In addition to research on MoE architectures and training strategies, recent years have also witnessed the emergence of numerous large-scale language or multimodal models (Du et al., 2022; Lin et al., 2021; Ren et al., 2023; Xue et al., 2023) based on existing MoE architectures. By and large, most of the previous MoE models are based on conventional top-1 or top-2 routing strategies, leaving large room for improving expert specialization. In response, our DeepSeekMoE architecture aims to improve the expert specialization to the utmost extent.\n\n9. Conclusion\n\nIn this paper, we introduce the DeepSeekMoE architecture for MoE language models, with the objective of achieving ultimate expert specialization. Through fine-grained expert segmentation and shared expert isolation, DeepSeekMoE achieves significantly higher expert specialization and performance compared with prevailing MoE architectures. Starting with a modest scale of 2B parameters, we validate the advantages of DeepSeekMoE, demonstrating its capability to approach the upper bound performance for MoE models. Furthermore, we provide empirical evidence to show that DeepSeekMoE has a higher level of expert specialization than GShard.\n\nScaling up to a larger scale of 16B total parameters, we train DeepSeekMoE 16B on 2T tokens and demonstrate its outstanding performance comparable with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations. Additionally, supervised fine-tuning is conducted for alignment to construct an MoE chat model based on DeepSeekMoE 16B, further showing its adaptability and versatility. Further, we perform a preliminary exploration to scale DeepSeekMoE to 145B parameters. We find that DeepSeekMoE 145B still keeps substantial advantages over the GShard architecture, and demonstrates comparable performance with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.\n\nFor research purposes, we release the model checkpoint of DeepSeekMoE 16B to the public, which can be deployed on a single GPU with 40GB of memory. We aspire for this work to provide valuable insights for both academia and industry, and contribute to the accelerated advancement of large-scale language models.\n\nReferences\n\nE. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance, 2023.\n\nM. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasunuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, P. S. Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. T. Diab, Z. Kozareva, and V. Stoyanov.\n\nEfficient large scale language modeling with mixtures of experts. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11699–11732. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022 .EMNLP-MAIN.804. URL https://doi.org/10.18653/v1/2022.emnlp-main.804.\n\nJ. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR, 2023. URL https: //proceedings.mlr.press/v202/biderman23a.html.\n\nY. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439. AAAI Press, 2020. doi: 10.1609/aaai.v34i05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239.\n\nS. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https://doi.org/10.5281/ zenodo.5297715. If you use this misc, please cite it using these metadata.\n\nT. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8 ac142f64a-Abstract.html.\n\nM. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.\n\nURL https://arxiv.org/abs/2107.03374.\n\nP. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457.\n\nK. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\nD. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502. Association for Computational Linguistics, 2022a. doi: 10.18653/V1/2022.ACL-LONG.581. URL https://doi.org/10.1 8653/v1/2022.acl-long.581.\n\nD. Dai, L. Dong, S. Ma, B. Zheng, Z. Sui, B. Chang, and F. Wei. Stablemoe: Stable routing strategy for mixture of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7085–7095.\n\nAssociation for Computational Linguistics, 2022b. doi: 10.18653/V1/2022.ACL-LONG.489.\n\nURL https://doi.org/10.18653/v1/2022.acl-long.489.\n\nDeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024.\n\nN. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. S. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5547–5569. PMLR, 2022.\n\nURL https://proceedings.mlr.press/v162/du22c.html.\n\nD. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2368– 2378. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1246. URL https://doi.org/10.18653/v1/n19-1246. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961.\n\nL. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.\n\nX. Geng and H. Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github.com/openlm-research/open_llama.\n\nA. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B.\n\nGibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR, abs/1806.03377, 2018. URL http://arxiv.org/abs/1806.03377. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.\n\nD. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.\n\nMeasuring mathematical problem solving with the math dataset, 2021.\n\nHigh-Flyer. Hai-llm: An efficient and lightweight tool for training large models, 2023. URL https://www.high-flyer.cn/en/blog/hai-llm.\n\nS. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computing, 9(8):1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.\n\nJ. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre.\n\nTraining compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550 /arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556.\n\nY. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023.\n\nR. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts.\n\nNeural Computing, 3(1):79–87, 1991. URL https://doi.org/10.1162/neco.1991.3.1.\n\n79.\n\nM. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Computing, 6(2):181–214, 1994. URL https://doi.org/10.1162/neco.1994.6.2.181.\n\nM. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017.\n\nV. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro.\n\nReducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.\n\nT. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research.\n\nTransactions of the Association of Computational Linguistics, 2019.\n\nG. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785–794. Association for Computational Linguistics, 2017. doi: 10.18653/V1/D17-1082. URL https://doi.org/10.18653/v1/d1 7-1082.\n\nD. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.\n\nGshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.\n\nURL https://openreview.net/forum?id=qrwe7XHTmYb.\n\nH. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023.\n\nJ. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, J. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A chinese multimodal pretrainer. CoRR, abs/2103.00823, 2021.\n\nURL https://arxiv.org/abs/2103.00823.\n\nS. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3214–3252. Association for Computational Linguistics, 2022.\n\ndoi: 10.18653/V1/2022.ACL-LONG.229. URL https://doi.org/10.18653/v1/2022.a cl-long.229.\n\nI. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.\n\nOpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.\n\nD. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15, 2021.\n\nOpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774.\n\nURL https://doi.org/10.48550/arXiv.2303.08774.\n\nS. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, page 20.\n\nIEEE/ACM, 2020. doi: 10.1109/SC41405.2020.00024. URL https://doi.org/10.1109/SC 41405.2020.00024.\n\nS. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation AI scale. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022. URL https://proceedings.mlr.press/v162/rajbh andari22a.html.\n\nX. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, A. Bout, I. Piontkovskaya, J. Wei, X. Jiang, T. Su, Q. Liu, and J. Yao. Pangu-Σ: Towards trillion parameter language model with sparse heterogeneous computing. CoRR, abs/2303.10845, 2023. URL https://doi.org/10.48550/arXiv.2303.10845.\n\nS. Roller, S. Sukhbaatar, A. Szlam, and J. Weston. Hash layers for large sparse models. CoRR, abs/2106.04426, 2021. URL https://arxiv.org/abs/2106.04426.\n\nK. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.\n\nT. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/ARXIV.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100.\n\nR. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16-1162. URL https: //doi.org/10.18653/v1/p16-1162.\n\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150.\n\nN. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.\n\nS. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou.\n\nFlan-moe: Scaling instruction-finetuned language models with sparse mixture of experts.\n\nCoRR, abs/2305.14705, 2023. doi: 10.48550/ARXIV.2305.14705. URL https://doi.org/10 .48550/arXiv.2305.14705.\n\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.\n\nM. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.\n\nP. Tillet, H. T. Kung, and D. Cox. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/331550 8.3329973. URL https://doi.org/10.1145/3315508.3329973.\n\nTogether-AI. Redpajama-data: An open source recipe to reproduce llama training dataset, April 2023. URL https://github.com/togethercomputer/RedPajama-Data.\n\nH. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/arXiv.230 2.13971. URL https://doi.org/10.48550/arXiv.2302.13971.\n\nH. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M.\n\nSmith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. doi: 10.48550/arXiv.2307.09288. URL https://doi.org/10.48550/arXiv.2307. 09288.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998–6008, 2017.\n\nURL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd 053c1c4a845aa-Abstract.html. B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.\n\nhttps://github.com/kingoflolz/mesh-transformer-jax, May 2021.\n\nL. Xu, H. Hu, X. Zhang, L. Li, C. Cao, Y. Li, Y. Xu, K. Sun, D. Yu, C. Yu, Y. Tian, Q. Dong, W. Liu, B. Shi, Y. Cui, J. Li, J. Zeng, R. Wang, W. Xie, Y. Li, Y. Patterson, Z. Tian, Y. Zhang, H. Zhou, S. Liu, Z. Zhao, Q. Zhao, C. Yue, X. Zhang, Z. Yang, K. Richardson, and Z. Lan. CLUE: A chinese language understanding evaluation benchmark. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 4762–4772. International Committee on Computational Linguistics, 2020. doi: 10.18653/V1/2020.COLING-MAIN.419. URL https://doi.org/10.18653/v1/2020.coling-main.419.\n\nF. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, and Y. You. Openmoe: Open mixture-of-experts language models. https://github.com/XueFuzhao/OpenMoE, 2023.\n\nR. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1472. URL https://doi.org/10.18653/v1/p1 9-1472.\n\nS. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.\n\nC. Zheng, M. Huang, and A. Sun. Chid: A large-scale chinese idiom dataset for cloze test. In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 778–787. Association for Computational Linguistics, 2019.\n\ndoi: 10.18653/V1/P19-1075. URL https://doi.org/10.18653/v1/p19-1075. Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Z. Chen, Q. V. Le, and J. Laudon.\n\nMixture-of-experts with expert choice routing. In NeurIPS, 2022. URL http://papers.nip s.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abs tract-Conference.html.\n\nB. Zoph. Designing effective sparse expert models. In IEEE International Parallel and Distributed Processing Symposium, IPDPS Workshops 2022, Lyon, France, May 30 - June 3, 2022, page 1044. IEEE, 2022. URL https://doi.org/10.1109/IPDPSW55747.2022.0 0171.\n\nAppendices A. Overview Of Hyper-Parameters\n\nWe present the overview of hyper-parameters for DeepSeekMoE across various sizes in Table 7.\n\n# Params # Layers Hidden # Attn # Shared # Routed Relative Sequence Batch Size Learning Size Heads Experts Experts Expert Size Length (Sequence) Rate 2.0B 9 1280 10 1 63 (7 activated) 0.25 2048 2048 1.08e-3 16.4B 28 2048 16 2 64 (6 activated) 0.25 4096 4608 4.2e-4 144.6B 62 4096 32 4 128 (12 activated) 0.125 4096 4608 3.0e-4\n\nTable 7 | Overview of hyper-parameters for DeepSeekMoE across various sizes. The relative expert size is in comparison to a standard FFN.\n\nB. Comparing Deepseekmoe With Larger Models\n\nComparisons among DeepSeekMoE, GShard×1.2, and GShard×1.5 are shown in Table 8. Comparisons among DeepSeekMoE, Dense×4, and Dense×16 are shown in Table 9.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 2.3B 2.8B 1.9B # Activated Expert Params N/A 0.28B 0.35B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.824 1.808 1.808 HellaSwag (Acc.) 0-shot 53.7 54.4 54.8 PIQA (Acc.) 0-shot 71.8 71.1 72.3 ARC-easy (Acc.) 0-shot 46.8 47.3 49.4 ARC-challenge (Acc.) 0-shot 31.7 34.1 34.3 RACE-middle (Acc.) 5-shot 43.7 46.4 44.0 RACE-high (Acc.) 5-shot 31.9 32.4 31.7 HumanEval (Pass@1) 0-shot 3.7 3.0 4.9 MBPP (Pass@1) 3-shot 2.4 2.6 2.2 TriviaQA (EM) 5-shot 15.2 15.7 16.6 NaturalQuestions (EM) 5-shot 4.5 4.7 5.7\n\nTable 8 | Comparison between DeepSeekMoE and larger GShard models.\n\nAt a larger scale of 13B total parameters, we also compare DeepSeekMoE with GShard×1.2 and GShard×1.5, and show results in Table 10. At a larger scale, DeepSeekMoE even outperforms GShard×1.5 distinctly.\n\nMetric # Shot Dense×4 Dense×16 DeepSeekMoE Relative Expert Size N/A 1 1 0.25 # Experts N/A 4 + 0 16 + 0 1 + 63 # Activated Experts N/A 4 + 0 16 + 0 1 + 7 # Total Expert Params N/A 0.47B 1.89B 1.89B # Activated Expert Params N/A 0.47B 1.89B 0.24B # Training Tokens N/A 100B 100B 100B Pile (Loss) N/A 1.908 1.806 1.808 HellaSwag (Acc.) 0-shot 47.6 55.1 54.8 PIQA (Acc.) 0-shot 70.0 71.9 72.3 ARC-easy (Acc.) 0-shot 43.9 51.9 49.4 ARC-challenge (Acc.) 0-shot 30.5 33.8 34.3 RACE-middle (Acc.) 5-shot 42.4 46.3 44.0 RACE-high (Acc.) 5-shot 30.7 33.0 31.7 HumanEval (Pass@1) 0-shot 1.8 4.3 4.9 MBPP (Pass@1) 3-shot 0.2 2.2 2.2 TriviaQA (EM) 5-shot 9.9 16.5 16.6 NaturalQuestions (EM) 5-shot 3.0 6.3 5.7\n\nTable 9 | Comparison between DeepSeekMoE and larger dense baselines.\n\nMetric # Shot GShard×1.2 GShard×1.5 DeepSeekMoE Relative Expert Size N/A 1.2 1.5 0.25 # Experts N/A 0 + 16 0 + 16 1 + 63 # Activated Experts N/A 0 + 2 0 + 2 1 + 7 # Total Expert Params N/A 15.9B 19.8B 13.3B # Activated Expert Params N/A 2.37B 2.82B 2.05B # Training Tokens N/A 100B 100B 100B HellaSwag (Acc.) 0-shot 66.6 67.7 69.1 PIQA (Acc.) 0-shot 75.6 76.0 75.7 ARC-easy (Acc.) 0-shot 56.8 56.8 58.8 ARC-challenge (Acc.) 0-shot 39.9 37.6 38.5 RACE-middle (Acc.) 5-shot 51.6 50.6 52.4 RACE-high (Acc.) 5-shot 37.4 36.3 38.5 HumanEval (Pass@1) 0-shot 6.1 6.1 9.8 MBPP (Pass@1) 3-shot 7.0 11.6 10.6 TriviaQA (EM) 5-shot 36.5 36.7 38.2 NaturalQuestions (EM) 5-shot 12.6 12.1 13.7\n\nTable 10 | Comparison between DeepSeekMoE and larger GShard models at a larger scale.\n\nC. Training Benchmark Curves Of Deepseekmoe 16B\n\nWe present the benchmark curves during training of DeepSeekMoE 16B and DeepSeek 7B (Dense) in Figure 7 for reference.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/DeepSeekMoE_2.md"
+          },
+          "summary": "DeepSeekMoE is an innovative Mixture-of-Experts (MoE) architecture designed to enhance expert specialization in language models. It employs fine-grained expert segmentation and shared expert isolation to improve performance and reduce computational costs. Starting with a model of 2B parameters, DeepSeekMoE demonstrates comparable performance to larger models like GShard 2.9B and LLaMA2 7B, using significantly fewer computations. The architecture scales up to 16B and 145B parameters, maintaining advantages over traditional MoE architectures. DeepSeekMoE 16B achieves similar results to dense models with only 40% of the computations, and its chat model version shows adaptability in bilingual scenarios. The architecture's public release aims to contribute to advancements in large-scale language models.",
+          "headlines": {
+            "1. Introduction": [],
+            "2. Preliminaries: Mixture-Of-Experts For Transformers": [],
+            "3. Deepseekmoe Architecture": [
+              "3.1. Fine-Grained Expert Segmentation",
+              "3.2. Shared Expert Isolation",
+              "3.3. Load Balance Consideration"
+            ],
+            "4. Validation Experiments": [
+              "4.1. Experimental Setup",
+              "4.1.1. Training Data And Tokenization",
+              "4.1.2. Infrastructures",
+              "4.1.3. Hyper-Parameters",
+              "4.1.4. Evaluation Benchmarks",
+              "4.2. Evaluations",
+              "4.3. Deepseekmoe Aligns Closely With The Upper Bound Of Moe Models",
+              "4.4. Ablation Studies",
+              "4.5. Analysis On Expert Specialization"
+            ],
+            "5. Scaling Up To Deepseekmoe 16B": [
+              "5.1. Experimental Setup",
+              "5.1.1. Training Data And Tokenization",
+              "5.1.2. Hyper-Parameters",
+              "5.1.3. Evaluation Benchmarks",
+              "5.2. Evaluations",
+              "5.2.1. Internal Comparison With Deepseek 7B",
+              "5.2.2. Comparison With Open Source Models"
+            ],
+            "6. Alignment For Deepseekmoe 16B": [
+              "6.1. Experimental Setup",
+              "6.2. Evaluations"
+            ],
+            "7. Deepseekmoe 145B Ongoing": [
+              "7.1. Experimental Setup",
+              "7.2. Evaluations"
+            ],
+            "8. Related Work": [],
+            "9. Conclusion": [],
+            "Appendices": [
+              "A. Overview Of Hyper-Parameters",
+              "B. Comparing Deepseekmoe With Larger Models",
+              "C. Training Benchmark Curves Of Deepseekmoe 16B"
+            ]
+          },
+          "summary_embedding": [
+            -0.012465757317841053,
+            -0.022808987647294998,
+            -0.018792515620589256,
+            -0.02308654971420765,
+            0.01797616109251976,
+            0.007200260180979967,
+            0.006490030791610479,
+            0.042613785713911057,
+            -0.014049487188458443,
+            0.03187054023146629,
+            -0.021486490964889526,
+            -0.023592690005898476,
+            0.009045224636793137,
+            -0.014441338367760181,
+            0.005865518003702164,
+            -0.007363531738519669,
+            -0.008743173442780972,
+            -0.005163452122360468,
+            0.018841497600078583,
+            -0.03100520372390747,
+            0.011110605672001839,
+            -0.046075135469436646,
+            -0.020017050206661224,
+            -0.021845687180757523,
+            0.016425084322690964,
+            0.037389107048511505,
+            0.022547753527760506,
+            -0.0028062239289283752,
+            -0.06844329088926315,
+            0.034678805619478226,
+            0.035168617963790894,
+            0.02052319049835205,
+            0.0005438972148112953,
+            -0.00031302153365686536,
+            -0.024833550676703453,
+            -0.028556134551763535,
+            0.03347059711813927,
+            -0.004379750229418278,
+            -0.01453930139541626,
+            0.015976088121533394,
+            0.0414055772125721,
+            0.012106560170650482,
+            -0.02754385396838188,
+            0.0014245413476601243,
+            0.015151568688452244,
+            0.0055961208418011665,
+            -0.01678428053855896,
+            -0.0022960014175623655,
+            -0.002424577483907342,
+            -0.021225256845355034,
+            -0.01400050614029169,
+            0.018825169652700424,
+            0.001296985661610961,
+            -0.008865626528859138,
+            -0.0542386956512928,
+            0.007330877240747213,
+            0.03840138763189316,
+            0.05623060464859009,
+            0.044148534536361694,
+            -0.020180322229862213,
+            0.03369917720556259,
+            -0.030025575309991837,
+            -0.02501314878463745,
+            0.011028970591723919,
+            0.003983817528933287,
+            -0.018531281501054764,
+            0.01784554310142994,
+            -0.005636938381940126,
+            -0.0006556359585374594,
+            0.01751900091767311,
+            -0.03941367194056511,
+            0.020751770585775375,
+            -0.01887415163218975,
+            0.031266435980796814,
+            0.02173139713704586,
+            0.0073798587545752525,
+            0.01872720755636692,
+            0.01678428053855896,
+            0.0036858476232737303,
+            -0.04107903689146042,
+            0.009118696674704552,
+            0.016523046419024467,
+            0.014196431264281273,
+            0.016269976273179054,
+            0.06553706526756287,
+            -0.05391215533018112,
+            -0.004485876299440861,
+            -0.05169166624546051,
+            0.0012561678886413574,
+            -0.004971608519554138,
+            -0.03670336678624153,
+            0.01048201136291027,
+            0.028425518423318863,
+            0.021372200921177864,
+            -0.01063711941242218,
+            0.024392718449234962,
+            -0.022743679583072662,
+            0.0032001156359910965,
+            -0.04062187671661377,
+            0.0019653772469609976,
+            0.004518530797213316,
+            0.016531210392713547,
+            0.0070778070949018,
+            0.004273623693734407,
+            -0.016702644526958466,
+            0.006045116577297449,
+            -0.01905374974012375,
+            0.03291547670960426,
+            -0.008465612307190895,
+            0.08653374016284943,
+            0.008751336485147476,
+            0.004738946910947561,
+            0.004281787667423487,
+            0.02398454025387764,
+            0.012131051160395145,
+            -0.00810233410447836,
+            -0.04718538001179695,
+            -0.015690363943576813,
+            0.011061624623835087,
+            -0.0383034273982048,
+            0.018335357308387756,
+            -0.008759500458836555,
+            0.01282495353370905,
+            -0.04107903689146042,
+            -0.06344719231128693,
+            0.01498013362288475,
+            0.001784758409485221,
+            -0.004065453074872494,
+            -0.0002903166168835014,
+            -0.018923133611679077,
+            0.003930754493921995,
+            -0.004200151655822992,
+            -0.010645282454788685,
+            -0.021502818912267685,
+            0.03159298002719879,
+            0.03657275065779686,
+            0.06527583301067352,
+            0.036801330745220184,
+            -0.0030164355412125587,
+            0.005775718949735165,
+            0.019690508022904396,
+            -0.020784424617886543,
+            0.003481758525595069,
+            -0.01214737817645073,
+            -0.021992631256580353,
+            -0.03820546343922615,
+            0.010971825569868088,
+            0.07379858940839767,
+            -0.029029620811343193,
+            0.013437220826745033,
+            -0.017616963014006615,
+            0.04205866530537605,
+            -0.004567512311041355,
+            0.026727497577667236,
+            -0.016669990494847298,
+            0.004059330560266972,
+            -0.025388672947883606,
+            0.01620466820895672,
+            -0.024964168667793274,
+            0.012571883387863636,
+            -0.000885746325366199,
+            -0.04222193360328674,
+            0.00011116473615402356,
+            0.026890767738223076,
+            -0.010971825569868088,
+            0.02261306345462799,
+            0.0004949158756062388,
+            -0.006302268709987402,
+            -0.05097327008843422,
+            0.038727931678295135,
+            -0.003912386484444141,
+            0.018645571544766426,
+            -0.0018959869630634785,
+            0.0163107942789793,
+            -0.0016408757073804736,
+            0.013216804713010788,
+            0.009543201886117458,
+            0.007114543113857508,
+            -0.033503253012895584,
+            0.02697240374982357,
+            -0.02504580281674862,
+            0.028049994260072708,
+            0.03435226157307625,
+            -0.0030184765346348286,
+            0.03255628049373627,
+            0.019249675795435905,
+            0.03840138763189316,
+            0.0327032245695591,
+            0.030335791409015656,
+            -0.00824927818030119,
+            0.013878053054213524,
+            0.008200296200811863,
+            0.03608293831348419,
+            0.013429056853055954,
+            0.019723162055015564,
+            -0.009730963967740536,
+            -0.014041324146091938,
+            0.048785436898469925,
+            0.007318632211536169,
+            -0.0179598331451416,
+            -0.005441013257950544,
+            -0.03510330989956856,
+            -0.002832755446434021,
+            -0.014425011351704597,
+            0.0323440283536911,
+            0.02158445306122303,
+            -0.01647406443953514,
+            0.035135965794324875,
+            0.024980494752526283,
+            -0.03751972317695618,
+            -0.005232842173427343,
+            -0.02026195637881756,
+            -0.023298801854252815,
+            -0.04215662553906441,
+            0.011624909937381744,
+            0.012833117507398129,
+            -0.013976015150547028,
+            -0.007469657808542252,
+            0.020457882434129715,
+            -0.023625344038009644,
+            -0.0018265966791659594,
+            -0.01356783788651228,
+            0.006547175347805023,
+            -0.020898714661598206,
+            -0.03667071461677551,
+            -0.01048201136291027,
+            0.025862159207463264,
+            0.012637191452085972,
+            0.04362606629729271,
+            -0.013518855907022953,
+            -0.005367541220039129,
+            0.033193036913871765,
+            0.012759645469486713,
+            0.010767736472189426,
+            0.03699725493788719,
+            -0.03027048148214817,
+            0.024621298536658287,
+            -0.020294612273573875,
+            -0.0032878739293664694,
+            0.01707816869020462,
+            -0.0030062312725931406,
+            0.01962519995868206,
+            -0.04535674303770065,
+            -0.024196794256567955,
+            0.04395261034369469,
+            -0.0007107399869710207,
+            0.016302630305290222,
+            -0.005959399044513702,
+            -0.012106560170650482,
+            0.0035899258218705654,
+            0.0027715289033949375,
+            -0.01438419334590435,
+            -0.008620720356702805,
+            0.007473739795386791,
+            -0.01962519995868206,
+            0.03660540655255318,
+            0.03794422745704651,
+            -0.004132802598178387,
+            -0.014131123200058937,
+            -0.03324201703071594,
+            0.010016689077019691,
+            -0.017878197133541107,
+            -0.023951886221766472,
+            0.04078514873981476,
+            -0.032115448266267776,
+            -0.00029312283731997013,
+            0.039478980004787445,
+            0.018808843567967415,
+            -0.014898497611284256,
+            0.00868602842092514,
+            -0.01709449477493763,
+            -0.026302991434931755,
+            -0.017159804701805115,
+            0.03356856107711792,
+            0.024800896644592285,
+            -0.010743245482444763,
+            0.010245268233120441,
+            0.005330805201083422,
+            0.02893165871500969,
+            0.024376392364501953,
+            0.00648594880476594,
+            -0.013812744058668613,
+            0.015029114671051502,
+            0.027641816064715385,
+            -0.0028021421749144793,
+            -7.66609373386018e-05,
+            -0.004010349046438932,
+            -0.007796200457960367,
+            -0.006453294772654772,
+            -0.013069860637187958,
+            -0.005089980084449053,
+            0.0009184005321003497,
+            -0.032295044511556625,
+            0.015967924147844315,
+            -0.005890008993446827,
+            0.010359558276832104,
+            -0.011535110883414745,
+            0.01288209855556488,
+            0.01722511276602745,
+            -0.013494365848600864,
+            -0.0716434046626091,
+            0.014612773433327675,
+            0.02458864450454712,
+            -0.006706364918500185,
+            -0.017943505197763443,
+            0.008481939323246479,
+            0.00016148542636074126,
+            0.038597315549850464,
+            -0.014180104248225689,
+            -0.031103165820240974,
+            0.02680913358926773,
+            -0.017437364906072617,
+            -0.015241367742419243,
+            -0.010661610402166843,
+            -0.00580837344750762,
+            0.04649963974952698,
+            0.01889047957956791,
+            0.011420820839703083,
+            -0.04042595252394676,
+            -0.007449248805642128,
+            0.015078096650540829,
+            0.014506647363305092,
+            -0.04705476388335228,
+            0.0052083516493439674,
+            0.015486273914575577,
+            0.019723162055015564,
+            0.024800896644592285,
+            0.04274440184235573,
+            -0.0021531390957534313,
+            0.01661284640431404,
+            -0.015053605660796165,
+            -0.05136512219905853,
+            -0.03340528905391693,
+            0.029943939298391342,
+            -0.017731253057718277,
+            -0.02906227484345436,
+            -0.01693122461438179,
+            -0.006151242647320032,
+            0.0017939424142241478,
+            0.007126788143068552,
+            -0.01916803978383541,
+            -0.03464614972472191,
+            -0.03970756009221077,
+            0.02277633361518383,
+            0.007983962073922157,
+            0.004269542172551155,
+            -0.042777057737112045,
+            0.0013459670590236783,
+            0.009184005670249462,
+            0.012073906138539314,
+            0.0009954441338777542,
+            -0.019984396174550056,
+            0.015494437888264656,
+            0.0032001156359910965,
+            -0.031070511788129807,
+            -0.035593122243881226,
+            -0.03595232218503952,
+            0.0023919232189655304,
+            0.003781769424676895,
+            0.06279410421848297,
+            0.012988224625587463,
+            -0.03840138763189316,
+            -0.04522612690925598,
+            0.029176564887166023,
+            -0.02052319049835205,
+            -0.016678154468536377,
+            0.0059430720284581184,
+            0.012898425571620464,
+            -0.009877908043563366,
+            0.013192313723266125,
+            0.023560035973787308,
+            0.041601505130529404,
+            -0.005987971555441618,
+            0.012914752587676048,
+            0.023037567734718323,
+            0.012955570593476295,
+            0.0362788625061512,
+            -0.00936360377818346,
+            0.0035409443080425262,
+            0.039936136454343796,
+            -0.012269831262528896,
+            0.03175625205039978,
+            -0.02189466916024685,
+            0.0020398697815835476,
+            0.02249877341091633,
+            0.027184655889868736,
+            -0.0006479826406575739,
+            0.02594379521906376,
+            -0.031543996185064316,
+            0.018368011340498924,
+            -0.014498483389616013,
+            -0.04888340085744858,
+            -0.01557607389986515,
+            0.013102514669299126,
+            0.030384771525859833,
+            0.02756018005311489,
+            -0.0041348435916006565,
+            -0.019723162055015564,
+            -0.01155143789947033,
+            -0.010596301406621933,
+            0.0018419033149257302,
+            0.03285016864538193,
+            0.010506502352654934,
+            0.010890189558267593,
+            -0.011935125105082989,
+            -0.004104230087250471,
+            0.03025415539741516,
+            -0.021192602813243866,
+            0.01650671847164631,
+            0.0041511706076562405,
+            -0.002029665280133486,
+            -0.04320156201720238,
+            -0.035266581922769547,
+            0.007845181971788406,
+            0.011747363954782486,
+            0.014874007552862167,
+            0.0368339866399765,
+            0.025094784796237946,
+            -0.01662100851535797,
+            0.030629679560661316,
+            -0.01905374974012375,
+            0.027984686195850372,
+            0.039054471999406815,
+            0.011624909937381744,
+            -0.018351683393120766,
+            0.012237177230417728,
+            0.002424577483907342,
+            -0.025127438828349113,
+            0.01680060662329197,
+            -0.032735876739025116,
+            0.01738838478922844,
+            -0.02354370802640915,
+            0.025143766775727272,
+            -0.02579685114324093,
+            -0.009290131740272045,
+            -0.004428731743246317,
+            -0.006130834110081196,
+            0.04888340085744858,
+            -0.026727497577667236,
+            -0.0142454132437706,
+            -0.0024694770108908415,
+            -0.0024531499948352575,
+            -0.008718682453036308,
+            0.010000362060964108,
+            -0.045781247317790985,
+            -0.01844964735209942,
+            -0.04016471654176712,
+            0.007355368230491877,
+            0.005412440747022629,
+            -0.030907239764928818,
+            -0.05694899708032608,
+            0.025421326979994774,
+            -0.06331657618284225,
+            -0.019886434078216553,
+            0.051626358181238174,
+            -0.018400665372610092,
+            -0.008212542161345482,
+            0.009641164913773537,
+            0.006375740747898817,
+            -0.029290854930877686,
+            0.0041348435916006565,
+            0.02504580281674862,
+            0.04418119043111801,
+            -0.0045797573402523994,
+            -0.02865409664809704,
+            0.0028674507047981024,
+            -0.027788760140538216,
+            -0.013461710885167122,
+            0.005261414684355259,
+            0.016449574381113052,
+            0.024539662525057793,
+            0.007628847379237413,
+            0.03585435822606087,
+            0.007294141221791506,
+            0.006318595726042986,
+            0.044279150664806366,
+            -0.025666235014796257,
+            -0.027380581945180893,
+            0.02519274689257145,
+            0.04045860469341278,
+            0.023151857778429985,
+            -0.018253721296787262,
+            -0.0142454132437706,
+            0.0053063142113387585,
+            0.0009296254720538855,
+            -0.022139575332403183,
+            0.002808264922350645,
+            -0.013902543112635612,
+            0.005143043119460344,
+            0.03892385587096214,
+            -0.001205145614221692,
+            0.014196431264281273,
+            0.007302304729819298,
+            0.006845145486295223,
+            0.004334850702434778,
+            -0.011731036007404327,
+            -0.023592690005898476,
+            0.00032220553839579225,
+            0.034972693771123886,
+            0.015404638834297657,
+            -0.030515389516949654,
+            -0.00995954405516386,
+            0.034711457788944244,
+            -0.00795130804181099,
+            -0.009118696674704552,
+            -0.009771781973540783,
+            0.003385836724191904,
+            0.02638462744653225,
+            -0.01606588624417782,
+            0.006649219896644354,
+            0.03716052696108818,
+            -0.019870106130838394,
+            0.005543057806789875,
+            -0.025405000895261765,
+            0.0018061877926811576,
+            0.02442537248134613,
+            -0.006216551177203655,
+            0.028098974376916885,
+            5.6315813708351925e-05,
+            0.03025415539741516,
+            -0.004677720367908478,
+            0.008432958275079727,
+            -0.020327266305685043,
+            0.03987082839012146,
+            -0.017633290961384773,
+            0.03402572125196457,
+            0.02067013457417488,
+            -0.01258004643023014,
+            0.04783846437931061,
+            -0.036801330745220184,
+            -0.020784424617886543,
+            0.00584919098764658,
+            0.012743317522108555,
+            0.0028919412288814783,
+            -0.0191517136991024,
+            -0.018400665372610092,
+            -0.010996315628290176,
+            0.02380494214594364,
+            -0.022547753527760506,
+            -0.016865916550159454,
+            -0.007441085297614336,
+            -0.01890680566430092,
+            -0.009673818945884705,
+            -0.007988044060766697,
+            -0.01709449477493763,
+            -0.016547536477446556,
+            0.044573038816452026,
+            -0.06334923207759857,
+            0.016996532678604126,
+            0.017159804701805115,
+            0.011477965861558914,
+            -0.034678805619478226,
+            -0.019510909914970398,
+            -0.017633290961384773,
+            -0.002347023691982031,
+            -0.030743969604372978,
+            0.001702102366834879,
+            0.01725776679813862,
+            0.01138000376522541,
+            -0.05022222548723221,
+            0.012335140258073807,
+            0.02772345207631588,
+            -0.012065742164850235,
+            0.0014949520118534565,
+            -0.023135529831051826,
+            -0.017176130786538124,
+            0.0031776658724993467,
+            0.022123249247670174,
+            0.005975726060569286,
+            -0.008188051171600819,
+            -0.016865916550159454,
+            -0.005241005681455135,
+            -0.006045116577297449,
+            -0.002359268954023719,
+            0.005094061605632305,
+            0.011780017986893654,
+            0.019690508022904396,
+            -0.0024633544962853193,
+            -0.004436895251274109,
+            -0.006396149750798941,
+            0.027364253997802734,
+            -0.001450052484869957,
+            -0.008416631259024143,
+            0.050907962024211884,
+            0.03556047007441521,
+            0.006094098091125488,
+            -0.01108611561357975,
+            -0.03990348428487778,
+            0.010661610402166843,
+            -0.022711025550961494,
+            0.0401320643723011,
+            0.01905374974012375,
+            -0.008298259228467941,
+            -0.028866350650787354,
+            0.010294250212609768,
+            -0.02290695160627365,
+            -0.009975871071219444,
+            0.009469729848206043,
+            0.00726556871086359,
+            -0.03263791650533676,
+            -0.013543346896767616,
+            0.03510330989956856,
+            0.011437147855758667,
+            -0.006841063499450684,
+            -0.036050282418727875,
+            0.008588065393269062,
+            0.01646590232849121,
+            0.011869817040860653,
+            -0.019233347848057747,
+            0.004449140280485153,
+            -0.012041252106428146,
+            0.017470018938183784,
+            0.008849299512803555,
+            -0.017796561121940613,
+            0.018237393349409103,
+            0.004710374400019646,
+            -0.008490103296935558,
+            -0.003912386484444141,
+            -0.0163107942789793,
+            -0.02006603218615055,
+            0.03840138763189316,
+            0.024490682408213615,
+            0.026335645467042923,
+            0.005457340274006128,
+            0.02008235827088356,
+            0.03227871656417847,
+            0.011282040737569332,
+            -0.03134807199239731,
+            0.027935704216361046,
+            0.016996532678604126,
+            -0.04604247957468033,
+            0.021551799029111862,
+            -0.02218855731189251,
+            -0.028670424595475197,
+            0.009649328887462616,
+            -0.0017367975087836385,
+            0.011192241683602333,
+            0.0034837995190173388,
+            -0.003642988856881857,
+            -0.03307874873280525,
+            -0.006236960180103779,
+            0.015771998092532158,
+            0.015502601861953735,
+            -0.0048695639707148075,
+            -0.003296037670224905,
+            -0.011886144056916237,
+            0.00487364549189806,
+            0.009796272963285446,
+            0.039054471999406815,
+            0.001381682581268251,
+            -0.004751192405819893,
+            -0.0018765984568744898,
+            0.031854212284088135,
+            0.0271193478256464,
+            -0.017274094745516777,
+            -0.026286665350198746,
+            0.00580837344750762,
+            -0.014253576286137104,
+            0.026156047359108925,
+            -0.00487364549189806,
+            0.011877980083227158,
+            0.008718682453036308,
+            -0.0007908449042588472,
+            -0.001373519073240459,
+            0.025584599003195763,
+            -0.02710302174091339,
+            -0.021764053031802177,
+            0.031233781948685646,
+            -0.025731543079018593,
+            0.028735732659697533,
+            0.013061696663498878,
+            -0.01557607389986515,
+            -0.015225040726363659,
+            -0.010302413254976273,
+            0.009943217039108276,
+            -0.012277995236217976,
+            0.023853924125432968,
+            0.003608293831348419,
+            0.006392067763954401,
+            0.0003584313380997628,
+            0.02770712412893772,
+            0.024833550676703453,
+            0.007820690982043743,
+            -0.018074123188853264,
+            -0.0016816934803500772,
+            -0.02320083975791931,
+            -0.01680060662329197,
+            0.023772288113832474,
+            -0.018090449273586273,
+            -0.03268689662218094,
+            -0.028556134551763535,
+            0.012114724144339561,
+            0.007392104249447584,
+            0.010816717520356178,
+            0.03461349755525589,
+            -0.030384771525859833,
+            -0.0344502255320549,
+            0.01766594499349594,
+            0.026270337402820587,
+            -0.005551221314817667,
+            -0.021535472944378853,
+            0.03085825778543949,
+            -0.005828781984746456,
+            0.017600636929273605,
+            0.0404912605881691,
+            -0.0016225076979026198,
+            0.010228941217064857,
+            -0.042026009410619736,
+            -0.014057651162147522,
+            -0.0066614653915166855,
+            0.006261450704187155,
+            0.034417569637298584,
+            0.01048201136291027,
+            0.01535565685480833,
+            0.03448288142681122,
+            -0.021192602813243866,
+            -0.0002903166168835014,
+            -0.0022449791431427,
+            -0.02818061038851738,
+            0.03347059711813927,
+            0.025715215131640434,
+            0.011918798089027405,
+            0.019706835970282555,
+            -0.03794422745704651,
+            -0.013037205673754215,
+            0.027070365846157074,
+            0.015053605660796165,
+            0.01273515447974205,
+            0.016694480553269386,
+            0.011249386705458164,
+            0.00868602842092514,
+            -0.021633435040712357,
+            -0.05169166624546051,
+            0.01214737817645073,
+            0.01993541419506073,
+            -0.016449574381113052,
+            0.01057181041687727,
+            0.024980494752526283,
+            0.007620683871209621,
+            -0.007045152597129345,
+            0.012808626517653465,
+            -0.02682545967400074,
+            0.002259265398606658,
+            0.0019010890973731875,
+            -0.014482156373560429,
+            0.038270771503448486,
+            -0.016367938369512558,
+            0.011363675817847252,
+            -0.018204739317297935,
+            0.023870250210165977,
+            0.00011531029304023832,
+            0.0007964573451317847,
+            0.014963806606829166,
+            -0.011347348801791668,
+            -0.04235255345702171,
+            0.0021613026037812233,
+            0.03342161700129509,
+            -0.015200549736618996,
+            0.026449935510754585,
+            -0.0020674217958003283,
+            0.032458316534757614,
+            0.042777057737112045,
+            0.017731253057718277,
+            0.0003497575526125729,
+            0.025682561099529266,
+            0.0036144163459539413,
+            0.0073880222626030445,
+            0.012400448322296143,
+            0.009322785772383213,
+            0.007245160173624754,
+            0.030809277668595314,
+            -0.010261595249176025,
+            0.02173139713704586,
+            -0.04760988429188728,
+            -0.014629100449383259,
+            0.012024925090372562,
+            -0.0013316808035597205,
+            -0.012449429370462894,
+            -0.033029764890670776,
+            0.012563719414174557,
+            -0.03503800183534622,
+            0.022074267268180847,
+            -0.0004959363141097128,
+            0.04333217814564705,
+            -0.02187834307551384,
+            -0.005518566817045212,
+            0.016694480553269386,
+            0.0016510800924152136,
+            0.023282473906874657,
+            -0.0012153501156717539,
+            -0.016416920349001884,
+            -0.0067798369564116,
+            -0.009257477708160877,
+            0.004138925112783909,
+            0.04375668615102768,
+            -0.003722583642229438,
+            -0.0001682458823779598,
+            0.01409030519425869,
+            -0.011959616094827652,
+            -0.03250729665160179,
+            0.045193471014499664,
+            -0.0030837850645184517,
+            0.02484987862408161,
+            0.04979771748185158,
+            -0.06410028040409088,
+            -0.019543563947081566,
+            0.013061696663498878,
+            -0.010441194288432598,
+            0.011363675817847252,
+            0.005547139327973127,
+            0.016114868223667145,
+            0.017894525080919266,
+            -0.005485912784934044,
+            0.01619650423526764,
+            0.005106307100504637,
+            0.028098974376916885,
+            -0.02336410991847515,
+            -0.03296445682644844,
+            -0.02636830136179924,
+            -0.017747581005096436,
+            0.0014480114914476871,
+            -0.015796490013599396,
+            -3.563585551091819e-06,
+            -0.0022204883862286806,
+            -0.04388730227947235,
+            -0.025274382904171944,
+            0.040687184780836105,
+            0.006706364918500185,
+            0.02458864450454712,
+            0.0077268099412322044,
+            0.03418898954987526,
+            0.005885927006602287,
+            -0.009037061594426632,
+            0.030058229342103004,
+            -0.020049704238772392,
+            0.03833607956767082,
+            0.022106921300292015,
+            -0.002832755446434021,
+            0.005865518003702164,
+            0.031086837872862816,
+            -0.033797141164541245,
+            0.008086007088422775,
+            -0.034384917467832565,
+            0.0046695563942193985,
+            -0.008620720356702805,
+            -0.015094423666596413,
+            -0.009526874870061874,
+            -0.02504580281674862,
+            0.011216731742024422,
+            -0.021535472944378853,
+            0.01590261608362198,
+            0.010416703298687935,
+            -0.002916431985795498,
+            0.03448288142681122,
+            -0.014204595237970352,
+            -0.0027796924114227295,
+            0.0024531499948352575,
+            -0.027070365846157074,
+            -0.006343086715787649,
+            -0.003377673216164112,
+            -0.03689929470419884,
+            -0.004502203781157732,
+            0.010988152585923672,
+            -0.00024184548237826675,
+            0.016898570582270622,
+            0.02787039615213871,
+            0.0005092020728625357,
+            -0.022302847355604172,
+            0.002340900944545865,
+            -0.04199335351586342,
+            0.0011877981014549732,
+            0.01483318954706192,
+            -0.017600636929273605,
+            -0.008612556383013725,
+            0.033046092838048935,
+            0.023298801854252815,
+            -0.009396257810294628,
+            -0.035005345940589905,
+            0.017796561121940613,
+            -0.011853490024805069,
+            0.027511198073625565,
+            -0.0002630197268445045,
+            0.019298657774925232,
+            -0.0076941559091210365,
+            0.04493223875761032,
+            0.001369437319226563,
+            0.008645210415124893,
+            -0.021959977224469185,
+            0.007020662073045969,
+            -0.020278284326195717,
+            -0.002406209474429488,
+            -0.006171651650220156,
+            -0.012751481495797634,
+            -0.010669773444533348,
+            0.0032735876739025116,
+            0.007730891928076744,
+            -0.0176822729408741,
+            0.0055961208418011665,
+            -0.015959760174155235,
+            -0.02741323597729206,
+            0.007543129846453667,
+            -0.05965929850935936,
+            -0.0179598331451416,
+            -0.03373183310031891,
+            -0.009477893821895123,
+            0.02008235827088356,
+            0.03510330989956856,
+            -0.012759645469486713,
+            0.00497977202758193,
+            -0.014898497611284256,
+            0.004477712791413069,
+            0.03490738570690155,
+            0.044442422688007355,
+            -0.014759717509150505,
+            0.003381754970178008,
+            -0.010955498553812504,
+            0.033029764890670776,
+            -0.03347059711813927,
+            0.0030062312725931406,
+            0.031086837872862816,
+            -0.005057325586676598,
+            0.007959471084177494,
+            0.011624909937381744,
+            0.020474210381507874,
+            -0.019396619871258736,
+            0.00877582747489214,
+            0.002100075827911496,
+            0.019233347848057747,
+            0.02171507105231285,
+            0.020343592390418053,
+            -0.03539719805121422,
+            0.028409190475940704,
+            0.018384337425231934,
+            0.02444170042872429,
+            0.044442422688007355,
+            -0.03974021226167679,
+            -0.004147088620811701,
+            -0.0032225653994828463,
+            -0.02997659333050251,
+            0.02008235827088356,
+            -0.007530884351581335,
+            -0.01841699331998825,
+            -0.007069643586874008,
+            0.03327467292547226,
+            0.014571955427527428,
+            -0.05051611363887787,
+            0.01980479806661606,
+            0.01740471087396145,
+            0.02055584453046322,
+            0.024474354460835457,
+            -0.01123305968940258,
+            -0.0019072118448093534,
+            0.008996243588626385,
+            0.006804327480494976,
+            0.019266001880168915,
+            0.015257694758474827,
+            0.002012317767366767,
+            -0.0022082431241869926,
+            -0.011837163008749485,
+            0.005224678665399551,
+            -0.009077879600226879,
+            0.010808554477989674,
+            -0.02458864450454712,
+            -0.009698309935629368,
+            0.021388528868556023,
+            0.0031164393294602633,
+            -0.013665799982845783,
+            -0.03729114308953285,
+            -0.045944519340991974,
+            0.0033837957307696342,
+            0.004543021321296692,
+            -0.05365091934800148,
+            0.005322641227394342,
+            -0.026547899469733238,
+            -0.002108239568769932,
+            -0.005502239800989628,
+            0.0019214981002733111,
+            0.005636938381940126,
+            0.0167679525911808,
+            -0.013494365848600864,
+            0.02248244546353817,
+            -0.015502601861953735,
+            0.03165828809142113,
+            0.0055961208418011665,
+            0.007375776767730713,
+            -0.0027796924114227295,
+            0.05590406060218811,
+            -0.003712379140779376,
+            -0.03699725493788719,
+            0.023282473906874657,
+            -0.01311067771166563,
+            -0.0001747512142173946,
+            0.030090883374214172,
+            0.00884113647043705,
+            -0.015257694758474827,
+            0.013437220826745033,
+            -0.030613351613283157,
+            -0.03249097242951393,
+            0.010996315628290176,
+            -0.013755599036812782,
+            -0.0015776080545037985,
+            -0.04568328335881233,
+            -0.009339112788438797,
+            0.004269542172551155,
+            -0.018335357308387756,
+            0.010775899514555931,
+            0.01631895825266838,
+            0.013992343097925186,
+            -0.024245774373412132,
+            0.009053388610482216,
+            0.006865554489195347,
+            0.013453547842800617,
+            0.0007331897504627705,
+            0.014049487188458443,
+            -0.008677864447236061,
+            -0.011306531727313995,
+            -0.018319029361009598,
+            0.0032450153958052397,
+            -0.026792805641889572,
+            0.005155288614332676,
+            -0.003155216109007597,
+            -0.03771565109491348,
+            -0.06521052122116089,
+            -0.0344502255320549,
+            0.016294466331601143,
+            0.03396041318774223,
+            -0.009045224636793137,
+            0.0035144127905368805,
+            -0.00023648813657928258,
+            -0.01093917153775692,
+            0.024523336440324783,
+            -0.005930826533585787,
+            -0.0004829256213270128,
+            0.0035389035474509,
+            0.02050686441361904,
+            -0.009722800925374031,
+            0.005845109466463327,
+            -0.051952898502349854,
+            -0.01586179807782173,
+            0.01812310516834259,
+            0.006306350696831942,
+            0.014629100449383259,
+            -0.009690145961940289,
+            0.01693122461438179,
+            -0.030074557289481163,
+            -0.028147956356406212,
+            0.04130761697888374,
+            0.008277850225567818,
+            0.011486129835247993,
+            -0.003055212553590536,
+            -0.01680060662329197,
+            0.02504580281674862,
+            -0.0020633398089557886,
+            -0.006461458280682564,
+            -0.00817988719791174,
+            0.015820980072021484,
+            0.044279150664806366,
+            0.003255219664424658,
+            0.0009821783751249313,
+            0.017045514658093452,
+            -0.002663361607119441,
+            -0.011575928889214993,
+            0.007677828427404165,
+            0.013592327944934368,
+            -0.016384266316890717,
+            0.010375885292887688,
+            -0.025862159207463264,
+            0.0332583449780941,
+            0.03131541982293129,
+            0.011575928889214993,
+            -0.015045441687107086,
+            0.006130834110081196,
+            0.005330805201083422,
+            0.008086007088422775,
+            -0.001369437319226563,
+            -0.0017133272485807538,
+            0.020719116553664207,
+            -0.03276853263378143,
+            -0.01693122461438179,
+            -0.005347132217139006,
+            0.003932795487344265,
+            0.005277741700410843,
+            0.03396041318774223,
+            -0.0033266509417444468,
+            0.017927179113030434,
+            -0.011657563969492912,
+            0.010579974390566349,
+            0.021764053031802177,
+            0.037095218896865845,
+            -0.024996822699904442,
+            -0.029862305149435997,
+            0.012914752587676048,
+            -0.001296985661610961,
+            0.014065815135836601,
+            0.0010173837654292583,
+            0.014139287173748016,
+            0.0029409227427095175,
+            0.060475654900074005,
+            -0.017763907089829445,
+            0.004628738854080439,
+            0.0004035860183648765,
+            -0.01858026348054409,
+            -0.0013531101867556572,
+            0.004261378664523363,
+            -0.010979988612234592,
+            0.03608293831348419,
+            0.002344982698559761,
+            0.039185091853141785,
+            0.013918871060013771,
+            0.009290131740272045,
+            0.0015143405180424452,
+            -0.013592327944934368,
+            0.008355404250323772,
+            0.0015592400450259447,
+            -0.001132694073021412,
+            0.009461566805839539,
+            0.019298657774925232,
+            0.009714636951684952,
+            0.025274382904171944,
+            0.009788108989596367,
+            0.012857607565820217,
+            -0.029470453038811684,
+            0.03510330989956856,
+            -0.0006224715034477413,
+            0.012996388599276543,
+            -0.02040890045464039,
+            -0.003136848099529743,
+            -0.0008240094175562263,
+            0.015649545937776566,
+            -0.015616890974342823,
+            -0.001201063860207796,
+            -0.01297189760953188,
+            0.040099408477544785,
+            -0.018988441675901413,
+            0.013976015150547028,
+            0.009673818945884705,
+            -0.02865409664809704,
+            -0.007596192881464958,
+            0.01995174214243889,
+            0.014294394291937351,
+            -0.0053144777193665504,
+            -0.020490536466240883,
+            0.005832863971590996,
+            0.015380147844552994,
+            -0.028572462499141693,
+            1.3584674888988957e-05,
+            0.014751553535461426,
+            -0.0164903923869133,
+            -0.00025409081717953086,
+            -0.004702210891991854,
+            0.003975654020905495,
+            -0.009265640750527382,
+            0.007028825581073761,
+            0.016057724133133888,
+            -0.024507008492946625,
+            -0.020049704238772392,
+            0.005722655914723873,
+            0.006571666337549686,
+            0.01444950234144926,
+            0.007739055436104536,
+            0.004355259705334902,
+            -0.020033378154039383,
+            -0.01799248717725277,
+            0.0032674651592969894,
+            -0.01649855636060238,
+            -0.0048614004626870155,
+            0.0020531355403363705,
+            -0.019363965839147568,
+            0.01875986158847809,
+            -0.010743245482444763,
+            0.00017538899555802345,
+            -0.003226647386327386,
+            -0.020800752565264702,
+            0.006204306147992611,
+            -0.004865481983870268,
+            0.019870106130838394,
+            0.003555230563506484,
+            0.02785406820476055,
+            0.02310287579894066,
+            -0.004428731743246317,
+            0.017600636929273605,
+            -0.016155686229467392,
+            0.007983962073922157,
+            -0.031135819852352142,
+            -0.001782717532478273,
+            -0.010408539324998856,
+            0.0036103345919400454,
+            -0.015943434089422226,
+            -0.009126860648393631,
+            -0.0048777274787425995,
+            -0.0006678812787868083,
+            -0.004820582456886768,
+            0.0004275664687156677,
+            0.0027817331720143557,
+            0.02231917530298233,
+            -0.013208640739321709,
+            -0.02803366631269455,
+            -0.017437364906072617,
+            0.006045116577297449,
+            0.019396619871258736,
+            -0.026025431230664253,
+            -0.009184005670249462,
+            -0.005012426059693098,
+            -0.004326687194406986,
+            -0.021404854953289032,
+            -0.015167895704507828,
+            -0.005632856860756874,
+            -0.008045189082622528,
+            -0.010522829368710518,
+            0.0046532293781638145,
+            -0.0035715578123927116,
+            -0.011739199981093407,
+            0.021666089072823524,
+            0.018057795241475105,
+            -0.019837452098727226,
+            -0.019837452098727226,
+            0.013641309924423695,
+            -0.006159406621009111,
+            0.0018878233386203647,
+            0.02008235827088356,
+            0.00029286774224601686,
+            0.005824700463563204,
+            0.020882388576865196,
+            -0.0035225762985646725,
+            -0.01797616109251976,
+            0.003577680326998234,
+            -0.0018888438353314996,
+            -0.012122887186706066,
+            0.003212361130863428,
+            0.02773977816104889,
+            0.0008076822850853205,
+            0.023429417982697487,
+            -0.03056436963379383,
+            -0.011412657797336578,
+            0.006061443593353033,
+            0.02369065210223198,
+            0.00211844383738935,
+            -0.0021694661118090153,
+            0.020131340250372887,
+            0.03941367194056511,
+            0.005775718949735165,
+            0.0015755671774968505,
+            -0.04003410041332245,
+            0.01027792226523161,
+            0.009208496659994125,
+            0.024621298536658287,
+            0.008339077234268188,
+            0.0014653591206297278,
+            -0.012906589545309544,
+            -0.009755454957485199,
+            -0.0017939424142241478,
+            0.014702572487294674,
+            -0.023870250210165977,
+            0.0024755997583270073,
+            -0.03157665207982063,
+            0.005094061605632305,
+            0.0021959978621453047,
+            -0.009559528902173042,
+            0.0335359051823616,
+            0.00980443600565195,
+            -0.02065380848944187,
+            0.015151568688452244,
+            0.005767555441707373,
+            0.007330877240747213,
+            0.022449791431427002,
+            -0.008228869177401066,
+            0.030450081452727318,
+            -0.005049162078648806,
+            0.01213921420276165,
+            0.02742956392467022,
+            0.032588932663202286,
+            -0.013102514669299126,
+            -0.003959327004849911,
+            0.0006097159348428249,
+            -0.0002561317232903093,
+            0.0007995187188498676,
+            -0.008204378187656403,
+            0.0037389106582850218,
+            -0.014800535514950752,
+            0.012710663489997387,
+            -0.019543563947081566,
+            -0.0034286954905837774,
+            0.0038532004691660404,
+            0.01991908811032772,
+            0.0016265894519165158,
+            -0.033633869141340256,
+            0.02055584453046322,
+            -0.015420965850353241,
+            0.0135596739128232,
+            0.001290863030590117,
+            -0.011110605672001839,
+            -0.00029210239881649613,
+            0.020588500425219536,
+            0.010947334580123425,
+            -0.007845181971788406,
+            -0.0135596739128232,
+            -0.028376536443829536,
+            -0.021535472944378853,
+            0.00823703221976757,
+            0.009445239789783955,
+            -0.0028837777208536863,
+            0.0021306893322616816,
+            -0.011869817040860653,
+            0.0006520643946714699,
+            -0.0008122742874547839,
+            0.01740471087396145,
+            -0.0335359051823616,
+            0.021943651139736176,
+            -0.019527235999703407,
+            -0.019543563947081566,
+            0.017045514658093452,
+            0.008049271069467068,
+            0.009445239789783955,
+            0.0033552234526723623,
+            -0.01995174214243889,
+            -0.012326976284384727,
+            0.0338951013982296,
+            -0.01138000376522541,
+            0.006298186723142862,
+            0.004408322740346193,
+            0.012914752587676048,
+            -0.008914608508348465,
+            -0.011926962062716484,
+            0.01784554310142994,
+            -0.009861581027507782,
+            0.013951525092124939,
+            0.0010010566329583526,
+            -0.0254703089594841,
+            0.02739690989255905,
+            -0.019233347848057747,
+            0.03487472981214523,
+            -0.021617108955979347,
+            0.00711862463504076,
+            0.0066614653915166855,
+            0.008792154490947723,
+            -0.006690037902444601,
+            -0.013224967755377293,
+            0.0012786176521331072,
+            0.03807484731078148,
+            -0.02623768337070942,
+            0.024180466309189796,
+            0.022400809451937675,
+            -0.00040766780148260295,
+            0.02065380848944187,
+            -0.008367649279534817,
+            -0.026711169630289078,
+            0.0179598331451416,
+            -0.012596373446285725,
+            -0.026123393326997757,
+            0.013257622718811035,
+            -0.022678371518850327,
+            -0.020278284326195717,
+            -0.026302991434931755,
+            0.004947117529809475,
+            -0.0025694805663079023,
+            -0.005151206627488136,
+            0.009004407562315464,
+            0.02561725303530693,
+            0.020915042608976364,
+            0.019576217979192734,
+            0.02111096680164337,
+            0.01161674689501524,
+            0.022449791431427002,
+            -0.007559456862509251,
+            0.04401791840791702,
+            -0.004947117529809475,
+            0.004632820375263691,
+            -0.0015980169409886003,
+            -0.015657709911465645,
+            0.009771781973540783,
+            0.0119106350466609,
+            0.0029572497587651014,
+            -0.029176564887166023,
+            -0.009061552584171295,
+            -0.03660540655255318,
+            0.005053244065493345,
+            0.0020664012990891933,
+            0.0176822729408741,
+            0.032115448266267776,
+            -0.005865518003702164,
+            0.004592002835124731,
+            -0.005102225113660097,
+            -0.013927034102380276,
+            -0.009420748800039291,
+            -0.02739690989255905,
+            -0.025437654927372932,
+            -0.007653337903320789,
+            -0.013739272020757198,
+            -0.030368445441126823,
+            -0.011722872965037823,
+            0.01551892887800932,
+            -0.007645174395292997,
+            0.011918798089027405,
+            -0.016702644526958466,
+            0.01385356206446886,
+            -0.004110352601855993,
+            -0.004763437435030937,
+            -0.004530776292085648,
+            -0.017600636929273605,
+            -0.006420640274882317,
+            0.014367866329848766,
+            -0.009477893821895123,
+            -0.01237595733255148,
+            -0.02215590327978134,
+            -0.0015459742862731218,
+            -0.01574750803411007,
+            0.003489922033622861,
+            -0.014596446417272091,
+            -0.006771673448383808,
+            -0.003908304497599602,
+            -0.018514955416321754,
+            0.01766594499349594,
+            0.014816862531006336,
+            -0.019380291923880577,
+            -0.0090697156265378,
+            0.02638462744653225,
+            0.042777057737112045,
+            -0.03218075633049011,
+            -0.008200296200811863,
+            0.0020378287881612778,
+            0.018090449273586273,
+            0.010171796195209026,
+            0.016702644526958466,
+            -0.024653952568769455,
+            0.0022286521270871162,
+            -0.021649762988090515,
+            -0.001220452249981463,
+            -0.023739634081721306,
+            0.010775899514555931,
+            0.025715215131640434,
+            -0.010971825569868088,
+            0.025976449251174927,
+            0.01634344831109047,
+            -0.01812310516834259,
+            0.012016761116683483,
+            0.007183933164924383,
+            0.014645427465438843,
+            -0.019755816087126732,
+            0.0167679525911808,
+            -0.015796490013599396,
+            0.031233781948685646,
+            0.00892277155071497,
+            -0.04829562455415726,
+            -0.03503800183534622,
+            -0.013388238847255707,
+            0.0013388239312916994,
+            -0.010702427476644516,
+            0.015037278644740582,
+            0.015967924147844315,
+            0.013959688134491444,
+            -0.019380291923880577,
+            -0.015820980072021484,
+            0.011984107084572315,
+            0.010065670125186443,
+            0.027462217956781387,
+            -0.006649219896644354,
+            -0.004212397150695324,
+            -0.00960034690797329,
+            -0.02249877341091633,
+            -0.016098542138934135,
+            0.0209803506731987,
+            -0.018041469156742096,
+            0.015935270115733147,
+            -0.026139721274375916,
+            -0.012449429370462894,
+            0.017453692853450775,
+            -0.05211617052555084,
+            -0.01453113742172718,
+            0.0007612520130351186,
+            0.03012353740632534,
+            0.004751192405819893,
+            0.0035184945445507765,
+            0.01841699331998825,
+            -0.01993541419506073,
+            -0.0039776950143277645,
+            -0.02099667675793171,
+            0.006832899991422892,
+            -0.011853490024805069,
+            -0.013984179124236107,
+            0.014482156373560429,
+            -0.013845399022102356,
+            -0.01601690612733364,
+            0.0034368589986115694,
+            0.007183933164924383,
+            -0.021094640716910362,
+            0.011951452121138573,
+            0.003945040516555309,
+            0.012171869166195393,
+            0.012171869166195393,
+            0.01199227012693882,
+            0.021551799029111862,
+            0.02295593172311783,
+            -0.007334959227591753,
+            -0.004767519421875477,
+            -0.01004117913544178,
+            -0.0015398516552522779,
+            0.008898280560970306,
+            0.0033797139767557383,
+            -0.002989904023706913,
+            0.007837017998099327,
+            0.010033016093075275,
+            -0.01498013362288475,
+            0.033193036913871765,
+            -0.0035878848284482956,
+            -0.04134026914834976,
+            -0.003238892648369074,
+            -0.03800953924655914,
+            0.020033378154039383,
+            -0.023298801854252815,
+            -0.016996532678604126,
+            0.0032776694279164076,
+            -0.00629410520195961,
+            -0.019902760162949562,
+            0.013608654960989952,
+            0.011404493823647499,
+            0.00258580781519413,
+            0.02021297626197338,
+            0.012269831262528896,
+            -0.005134879611432552,
+            0.026172375306487083,
+            -0.014718899503350258,
+            -0.008751336485147476,
+            0.015567909926176071,
+            0.006853308994323015,
+            0.0022347746416926384,
+            0.0011031010653823614,
+            -0.018825169652700424,
+            0.005098143592476845,
+            0.0164903923869133,
+            0.007216587662696838,
+            -0.000566347036510706,
+            0.014049487188458443,
+            0.012514738366007805,
+            -0.015135240741074085,
+            -0.0002213090192526579,
+            -0.026564225554466248,
+            0.0032817514147609472,
+            0.004567512311041355,
+            0.007506393827497959,
+            -0.006771673448383808,
+            0.006281859707087278,
+            0.00658799335360527,
+            0.014759717509150505,
+            -0.010155469179153442,
+            -0.016980206593871117,
+            -0.002510294783860445,
+            0.004285869188606739,
+            0.016694480553269386,
+            0.03157665207982063,
+            -0.010310577228665352,
+            -0.005734901409596205,
+            -0.0386626236140728,
+            0.02070278860628605,
+            0.03332365304231644,
+            0.011175914667546749,
+            -0.028213264420628548,
+            0.012302485294640064,
+            0.0006515541463159025,
+            0.0019276207312941551,
+            -0.026302991434931755,
+            0.02787039615213871,
+            0.00974729098379612,
+            0.0066614653915166855,
+            0.017861871048808098,
+            0.0022306928876787424,
+            0.0029266364872455597,
+            -0.016302630305290222,
+            0.01018812321126461,
+            0.009086042642593384,
+            -0.017306748777627945,
+            0.0009745250572450459,
+            0.02324981987476349,
+            -0.01423724927008152,
+            -0.003208279376849532,
+            0.0018133309204131365,
+            0.02651524543762207,
+            -0.013820908032357693,
+            -0.014294394291937351,
+            0.0005714492290280759,
+            0.01751900091767311,
+            -0.0007979880319908261,
+            0.012098397128283978,
+            0.04898136481642723,
+            -0.012424939312040806,
+            0.019657853990793228,
+            0.005461421795189381,
+            -0.015290348790585995,
+            -0.032605260610580444,
+            -0.004922627005726099,
+            -0.01830270327627659,
+            0.011200404725968838,
+            0.014343376271426678,
+            -0.015763835981488228,
+            0.0007525782566517591,
+            -0.03144603595137596,
+            0.007743136957287788,
+            0.014735226519405842,
+            0.018776189535856247,
+            -0.002440904499962926,
+            0.006134915631264448,
+            0.027674470096826553,
+            0.02458864450454712,
+            -0.007392104249447584,
+            0.013804581016302109,
+            0.02320083975791931,
+            -0.025062130764126778,
+            -0.003069498809054494,
+            0.006820654962211847,
+            0.004104230087250471,
+            0.0053063142113387585,
+            -0.016963878646492958,
+            0.03915243595838547,
+            -0.005922663025557995,
+            -0.01506176870316267,
+            0.0030266400426626205,
+            -0.009028897620737553,
+            0.0063390047289431095,
+            -0.012563719414174557,
+            0.012637191452085972,
+            -0.008114579133689404,
+            0.00022883480414748192,
+            -0.0059430720284581184,
+            -0.011110605672001839,
+            -0.012653518468141556,
+            -0.0007949267164804041,
+            -0.016531210392713547,
+            0.007432921789586544,
+            0.00981259997934103,
+            0.005045080557465553,
+            0.01025343220680952,
+            -0.0036797248758375645,
+            -0.003136848099529743,
+            -0.016751626506447792,
+            -0.009984034113585949,
+            0.0073716952465474606,
+            0.01872720755636692,
+            -0.028588788583874702,
+            -0.018547609448432922,
+            0.009290131740272045,
+            0.0022898786701261997,
+            0.018776189535856247,
+            0.0158944521099329,
+            -0.0002595757250674069,
+            0.012318813242018223,
+            -0.01964152604341507,
+            -0.009184005670249462,
+            0.009690145961940289,
+            -0.003569516818970442,
+            0.01048201136291027,
+            -0.014759717509150505,
+            0.010033016093075275,
+            -0.011159587651491165,
+            -0.011053460650146008,
+            0.01385356206446886,
+            0.01738838478922844,
+            0.008873790502548218,
+            -0.001294944784604013,
+            3.383413059054874e-05,
+            -0.010506502352654934,
+            -0.024033522233366966,
+            0.004771600943058729,
+            0.027021385729312897,
+            -0.013273949734866619,
+            0.003883813973516226,
+            0.008702355436980724,
+            -0.017502672970294952,
+            -0.019821124151349068,
+            -0.012653518468141556,
+            -0.022106921300292015,
+            0.020572172477841377,
+            0.023413091897964478,
+            -0.02026195637881756,
+            0.00877582747489214,
+            -0.004620575346052647,
+            0.014049487188458443,
+            -0.01400050614029169,
+            0.010612628422677517,
+            0.0317889042198658,
+            -0.020000724121928215,
+            -0.0028654097113758326,
+            0.0036633978597819805,
+            -0.015347493812441826,
+            0.018629245460033417,
+            0.03268689662218094,
+            -0.00176945177372545,
+            0.014041324146091938,
+            0.0008138049161061645,
+            -0.009118696674704552,
+            0.01905374974012375,
+            0.011094278655946255,
+            0.005347132217139006,
+            -0.0042491331696510315,
+            -0.012212686240673065,
+            -0.03010721132159233,
+            0.016702644526958466,
+            0.0129637336358428,
+            -0.0037674831692129374,
+            -0.0011551438365131617,
+            -0.006343086715787649,
+            -0.015404638834297657,
+            0.0035123717971146107,
+            -0.013078023679554462,
+            -0.00803294312208891,
+            -0.008759500458836555,
+            0.004314441699534655,
+            -0.0034960447810590267,
+            -0.009053388610482216,
+            0.007236996199935675,
+            0.0009510547970421612,
+            -0.0010602424154058099,
+            -0.01902109570801258,
+            -0.025078458711504936,
+            -0.00995954405516386,
+            -0.016849588602781296,
+            0.0038287099450826645,
+            0.0005234883283264935,
+            -0.027184655889868736,
+            0.0053961132653057575,
+            -0.008139070123434067,
+            -0.008134988136589527,
+            0.007126788143068552,
+            0.006881881505250931,
+            -0.007437003776431084,
+            -0.010816717520356178,
+            -0.00786150898784399,
+            -0.006865554489195347,
+            0.020033378154039383,
+            0.0034348180051892996,
+            0.012082069180905819,
+            -0.010498338378965855,
+            -0.018025141209363937,
+            -0.016963878646492958,
+            0.016718972474336624,
+            -0.008971752598881721,
+            -0.02083340659737587,
+            -0.01444950234144926,
+            -0.008424794301390648,
+            -0.007665583398193121,
+            -0.01527402177453041,
+            -0.021159948781132698,
+            -0.0032103201374411583,
+            -0.011347348801791668,
+            -0.0067431009374558926,
+            -0.0028960229828953743,
+            0.0013571919407695532,
+            0.00951871182769537,
+            0.00546958576887846,
+            0.00438791373744607,
+            0.021666089072823524,
+            0.010122815147042274,
+            -0.005461421795189381,
+            0.016441410407423973,
+            -0.006183897145092487,
+            -0.004440976772457361,
+            -0.01498013362288475,
+            -0.009918726049363613,
+            0.018547609448432922,
+            -0.0055226488038897514,
+            0.008457448333501816,
+            -0.0009536059224046767,
+            -0.00576347392052412,
+            0.015674035996198654,
+            0.0129637336358428,
+            0.016718972474336624,
+            0.004771600943058729,
+            -0.00012168807734269649,
+            0.012416775338351727,
+            -0.0015745466807857156,
+            0.012555556371808052,
+            -0.0038429962005466223,
+            0.014506647363305092,
+            -0.009722800925374031,
+            -0.018988441675901413,
+            0.004157293122261763,
+            0.006265532691031694,
+            -0.012531065382063389,
+            0.009404421783983707,
+            0.007743136957287788,
+            -0.013747435994446278,
+            -0.02261306345462799,
+            -0.000106317616882734,
+            0.00014362763613462448,
+            0.008122743107378483,
+            0.008971752598881721,
+            -0.008073761127889156,
+            0.012890261597931385,
+            0.004616493359208107,
+            -0.007179851643741131,
+            -0.019657853990793228,
+            0.026009103283286095,
+            0.015608727931976318,
+            0.012269831262528896,
+            -0.052508022636175156,
+            -0.0030980713199824095,
+            0.010743245482444763,
+            -0.015951598063111305,
+            -0.026792805641889572,
+            -0.006404313258826733,
+            -0.008759500458836555,
+            -0.04290767386555672,
+            -0.009624837897717953,
+            -0.0005724696675315499,
+            -0.011388166807591915,
+            0.010400376282632351,
+            0.01573934406042099,
+            0.014612773433327675,
+            0.005489994306117296,
+            -0.007232914678752422,
+            -0.017633290961384773,
+            -0.02788672223687172,
+            0.010057506151497364,
+            -0.013380075804889202,
+            -0.006412476766854525,
+            -0.002263347152620554,
+            -0.007690073922276497,
+            0.0016112826997414231,
+            0.01691489666700363,
+            0.042483169585466385,
+            0.023037567734718323,
+            -0.012335140258073807,
+            -0.020131340250372887,
+            -0.012800462543964386,
+            -0.009192168712615967,
+            0.00633492274209857,
+            0.011592255905270576,
+            -0.004738946910947561,
+            0.0096656559035182,
+            -0.004294032696634531,
+            0.012302485294640064,
+            0.012767808511853218,
+            0.01680060662329197,
+            -2.3023791072773747e-05,
+            0.005579793825745583,
+            -0.0024694770108908415,
+            -0.01738838478922844,
+            -0.004379750229418278,
+            0.008261523209512234,
+            -0.0008872769540175796,
+            0.0019317024853080511,
+            0.014204595237970352,
+            0.019102731719613075,
+            -0.009094206616282463,
+            -0.014612773433327675,
+            -0.016098542138934135,
+            -0.005277741700410843,
+            -0.0012622905196622014,
+            0.029927613213658333,
+            0.02037624642252922,
+            -0.019266001880168915,
+            -0.018237393349409103,
+            -0.018351683393120766,
+            0.00922482367604971,
+            -0.003563394071534276,
+            -0.0064369672909379005,
+            0.01905374974012375,
+            0.023021239787340164,
+            0.017910851165652275,
+            -0.003642988856881857,
+            -0.014865843579173088,
+            0.003804219188168645,
+            0.02847449854016304,
+            0.008367649279534817,
+            0.01078406348824501,
+            0.008726846426725388,
+            -0.0011908593587577343,
+            -0.032441988587379456,
+            -0.0113146947696805,
+            0.00951871182769537,
+            -0.008481939323246479,
+            0.026564225554466248,
+            -0.016531210392713547,
+            0.009388094767928123,
+            -0.005077734589576721,
+            0.0019286411115899682,
+            -0.02365799807012081,
+            0.0014929111348465085,
+            0.010122815147042274,
+            -0.024947840720415115,
+            0.00336950970813632,
+            -0.023592690005898476,
+            -0.00013763252354692668,
+            0.008367649279534817,
+            0.0022837561555206776,
+            0.01650671847164631,
+            -0.0113146947696805,
+            0.0051838611252605915,
+            0.02099667675793171,
+            -0.002832755446434021,
+            -0.01512707769870758,
+            -0.0007842120248824358,
+            0.0005092020728625357,
+            -0.001286781276576221,
+            0.02563357912003994,
+            0.015812816098332405,
+            -0.0036756431218236685,
+            0.020882388576865196,
+            0.008677864447236061,
+            0.0019133344758301973,
+            -0.021764053031802177,
+            0.0023796779569238424,
+            -0.032899148762226105,
+            0.017437364906072617,
+            -0.011126932688057423,
+            0.012490247376263142,
+            0.00868602842092514,
+            -0.01341272983700037,
+            -0.0007388022495433688,
+            -0.0107187544927001,
+            -0.0015725058037787676,
+            -0.007934981025755405,
+            0.009779945947229862,
+            -0.004363423213362694,
+            0.008339077234268188,
+            -0.00024579971795901656,
+            -0.01476788055151701,
+            0.018694553524255753,
+            9.678283277025912e-06,
+            -0.00010925139940809458,
+            0.005441013257950544,
+            0.009935053065419197,
+            0.00786559097468853,
+            0.002009256277233362,
+            0.023870250210165977,
+            -0.031119491904973984,
+            -0.016082214191555977,
+            -0.010204450227320194,
+            0.005543057806789875,
+            -0.007922735065221786,
+            -0.018531281501054764,
+            -0.03257260471582413,
+            0.01318415068089962,
+            0.001293924287892878,
+            -0.0009806477464735508,
+            0.01258004643023014,
+            -0.00942891277372837,
+            -0.0013878053287044168,
+            -0.01606588624417782,
+            -0.014808698557317257,
+            0.006416558753699064,
+            0.0033633869607001543,
+            -0.01222901325672865,
+            -0.023739634081721306,
+            0.014759717509150505,
+            0.0007418635650537908,
+            -0.016980206593871117,
+            -0.017502672970294952,
+            0.00018457300029695034,
+            -0.004804255440831184,
+            0.02204161323606968,
+            -0.016678154468536377,
+            0.003373591462150216,
+            0.0007219648687168956,
+            0.009232986718416214,
+            -0.02889900468289852,
+            0.00716352416202426,
+            0.0006107363733462989,
+            0.0035164537839591503,
+            -0.006518602836877108,
+            -0.021013004705309868,
+            0.011371839791536331,
+            -0.0020674217958003283,
+            -0.014751553535461426,
+            -0.020751770585775375,
+            -0.0020796670578420162,
+            -0.007420676294714212,
+            -0.02724996581673622,
+            0.017910851165652275,
+            0.0009985055075958371,
+            -0.01738838478922844,
+            0.008077843114733696,
+            0.006596156861633062,
+            0.009355440735816956,
+            0.02455599047243595,
+            0.010147306136786938,
+            -0.018776189535856247,
+            -0.0008087027235887945,
+            0.006979844067245722,
+            0.009306458756327629,
+            0.0036450298503041267,
+            0.04222193360328674,
+            -0.008130906149744987,
+            -0.006579829845577478,
+            0.0066614653915166855,
+            -0.008449285291135311,
+            -0.013747435994446278,
+            0.026727497577667236,
+            0.005796127952635288,
+            0.015984252095222473,
+            0.016123032197356224,
+            -0.00945340283215046,
+            -0.005008344538509846,
+            -0.0018786393338814378,
+            0.02065380848944187,
+            0.0073798587545752525,
+            -0.005502239800989628,
+            -0.009714636951684952,
+            0.005881845485419035,
+            -0.00898808054625988,
+            0.0021653843577951193,
+            -0.009526874870061874,
+            -0.005824700463563204,
+            -0.008069679141044617,
+            0.016400592401623726,
+            -0.0003523086488712579,
+            0.027315273880958557,
+            0.020751770585775375,
+            -0.008563575334846973,
+            -0.0005683879135176539,
+            0.017208784818649292,
+            -0.005820618476718664,
+            0.015608727931976318,
+            -0.011192241683602333,
+            -0.007706400938332081,
+            0.006408394780009985,
+            0.024017194285988808,
+            0.012237177230417728,
+            -0.0056042843498289585,
+            0.0002885308349505067,
+            -0.0030348035506904125,
+            -0.010612628422677517,
+            0.013388238847255707,
+            0.015323002822697163,
+            -0.01947825588285923,
+            -0.004722619894891977,
+            -0.0016592437168583274,
+            -0.003300119424238801,
+            0.00945340283215046,
+            0.02305389568209648,
+            0.000171817431692034,
+            0.009184005670249462,
+            0.0012765767751261592,
+            0.03862996771931648,
+            -0.0053144777193665504,
+            0.03102152980864048,
+            -0.0010423846542835236,
+            -0.0059430720284581184,
+            -0.009094206616282463,
+            -0.020441556349396706,
+            -0.011486129835247993,
+            -0.00041481092921458185,
+            -0.005040998570621014,
+            -0.0004956811899319291,
+            -0.020327266305685043,
+            0.0026960158720612526,
+            0.014637264423072338,
+            0.007661501411348581,
+            0.017927179113030434,
+            -0.002889900468289852,
+            0.017437364906072617,
+            0.019690508022904396,
+            -0.01400050614029169,
+            -0.029862305149435997,
+            -0.0006515541463159025,
+            0.01498013362288475,
+            -0.01661284640431404,
+            0.019445601850748062,
+            0.004085862077772617,
+            0.0029531680047512054,
+            0.004779764451086521,
+            -0.006049198564141989,
+            0.018482301384210587,
+            0.02039257436990738,
+            0.01453930139541626,
+            -0.013167822733521461,
+            0.014057651162147522,
+            0.009771781973540783,
+            0.006600238848477602,
+            -0.01646590232849121,
+            -0.013535182923078537,
+            0.002555194543674588,
+            0.0009245232213288546,
+            0.024392718449234962,
+            -0.020588500425219536,
+            0.0016602640971541405,
+            0.010588138364255428,
+            0.01769859902560711,
+            -0.004302196204662323,
+            0.008906444534659386,
+            0.0008740111952647567,
+            0.007134951651096344,
+            -0.0035919665824621916,
+            -0.006457376293838024,
+            -0.006673710886389017,
+            -0.010392212308943272,
+            -0.0020959940738976,
+            -0.02070278860628605,
+            0.002198038622736931,
+            -0.002830714685842395,
+            0.018237393349409103,
+            -0.01018812321126461,
+            0.038140155375003815,
+            0.03399306535720825,
+            -0.0007025764207355678,
+            0.011102442629635334,
+            -0.01709449477493763,
+            -0.003945040516555309,
+            -0.021959977224469185,
+            0.01844964735209942,
+            0.016392430290579796,
+            0.012269831262528896,
+            0.005302232690155506,
+            0.017796561121940613,
+            -0.0006301248213276267,
+            0.014490320347249508,
+            0.0029654132667928934,
+            0.007971717044711113,
+            -0.002265388146042824,
+            0.003938918001949787,
+            -0.031103165820240974,
+            -0.017029186710715294,
+            -0.01138000376522541,
+            0.004796091932803392,
+            -0.006375740747898817,
+            -0.0035205355379730463,
+            0.0084737753495574,
+            -0.005902254022657871,
+            0.0004270562494639307,
+            -0.025992777198553085,
+            -0.0157148540019989,
+            -0.010318740271031857,
+            -0.004040962550789118,
+            -0.01169838197529316,
+            -0.0027694879099726677,
+            -0.0022082431241869926,
+            0.0029552089981734753,
+            0.020947696641087532,
+            -0.012808626517653465,
+            -0.016294466331601143,
+            0.004379750229418278,
+            -0.002263347152620554,
+            -0.014016833156347275,
+            -0.016898570582270622,
+            0.006665547378361225,
+            0.012098397128283978,
+            -0.0017470018938183784,
+            -0.013608654960989952,
+            0.011739199981093407,
+            -0.006628811359405518,
+            0.04663025587797165,
+            -0.013584164902567863,
+            -0.0078002819791436195,
+            0.003720542648807168,
+            0.004698128905147314,
+            -0.009396257810294628,
+            0.007330877240747213,
+            0.013020878657698631,
+            0.01830270327627659,
+            0.0095758568495512,
+            -0.008465612307190895,
+            -0.0024388637393712997,
+            0.007796200457960367,
+            -0.016147522255778313,
+            0.006404313258826733,
+            0.0008525818702764809,
+            -0.014400520361959934,
+            0.006110425107181072,
+            0.028980640694499016,
+            0.013151495717465878,
+            0.01288209855556488,
+            -0.022351829335093498,
+            -0.004094025585800409,
+            -0.009355440735816956,
+            -0.004943036008626223,
+            -0.010090161114931107,
+            0.027070365846157074,
+            -0.018155759200453758,
+            0.02651524543762207,
+            0.00898808054625988,
+            0.0017684312770143151,
+            -0.01176369097083807,
+            0.006343086715787649,
+            0.02832755446434021,
+            0.0028592869639396667,
+            -0.020000724121928215,
+            -0.013592327944934368,
+            0.006314514204859734,
+            0.01707816869020462,
+            -0.002647034591063857,
+            -0.005428767763078213,
+            0.019657853990793228,
+            0.01784554310142994,
+            -0.004334850702434778,
+            -0.00936360377818346,
+            0.0008337036124430597,
+            0.00928196869790554,
+            0.016963878646492958,
+            0.015020951628684998,
+            0.009086042642593384,
+            -0.002647034591063857,
+            0.008037025108933449,
+            -0.008800318464636803,
+            -0.016449574381113052,
+            0.007220669183880091,
+            -0.008432958275079727,
+            -0.016963878646492958,
+            0.01872720755636692,
+            0.015257694758474827,
+            0.0181394312530756,
+            0.0024592725094407797,
+            -0.016735298559069633,
+            0.022302847355604172,
+            0.005755309946835041,
+            0.002998067531734705,
+            0.02160078100860119,
+            -0.022221211344003677,
+            0.013012715615332127,
+            0.02037624642252922,
+            0.027609162032604218,
+            -0.01737205684185028,
+            -0.010906516574323177,
+            -0.004857318475842476,
+            -0.006853308994323015,
+            -0.00176945177372545,
+            -0.008253359235823154,
+            -0.012620864436030388,
+            0.0028735732194036245,
+            -0.030466407537460327,
+            0.018106777220964432,
+            -0.005102225113660097,
+            0.010669773444533348,
+            -0.019266001880168915,
+            0.011363675817847252,
+            -0.0035838030744343996,
+            0.011135096661746502,
+            0.0019612954929471016,
+            -0.030041903257369995,
+            -0.008571738377213478,
+            0.015388311818242073,
+            -0.0030368445441126823,
+            -0.013641309924423695,
+            0.02006603218615055,
+            0.00018942011229228228,
+            -0.022270193323493004,
+            -0.004314441699534655,
+            -0.006934944540262222,
+            0.002193956868723035,
+            -0.0029613315127789974,
+            -0.0052981507033109665,
+            -0.030188847333192825,
+            0.020033378154039383,
+            0.009559528902173042,
+            -0.03072764165699482,
+            -0.003255219664424658,
+            -0.014996460638940334,
+            -0.0025184585247188807,
+            0.008881953544914722,
+            -0.004000144544988871,
+            -0.015551582910120487,
+            0.01784554310142994,
+            0.022711025550961494,
+            -0.0023164101876318455,
+            -0.016057724133133888,
+            -0.0090697156265378,
+            -0.027494871988892555,
+            0.030825603753328323,
+            -0.0066696288995444775,
+            0.0022878379095345736,
+            -0.005322641227394342,
+            0.009388094767928123,
+            0.004228724166750908,
+            0.0157148540019989,
+            0.006081852596253157,
+            -0.002647034591063857,
+            0.026596879586577415,
+            -0.01964152604341507,
+            0.009567692875862122,
+            -1.588864870427642e-05,
+            -0.003308282932266593,
+            0.01890680566430092,
+            0.0007939062779769301,
+            0.021649762988090515,
+            -0.008571738377213478,
+            -0.0001475818717153743,
+            -0.0034388997592031956,
+            0.010898353531956673,
+            -0.0028450009413063526,
+            -0.003465431509539485,
+            -0.001531688030809164,
+            0.011118769645690918,
+            0.01013097818940878,
+            0.003804219188168645,
+            -0.01737205684185028,
+            0.003142970846965909,
+            -0.008547247387468815,
+            -0.007028825581073761,
+            0.01302904263138771,
+            -0.011192241683602333,
+            -0.0018990482203662395,
+            -0.03160930797457695,
+            0.016286302357912064,
+            0.0533243790268898,
+            0.008547247387468815,
+            -0.014433175325393677,
+            -0.0018806802108883858,
+            -0.019755816087126732,
+            0.01535565685480833,
+            0.011004479601979256,
+            -0.016816934570670128,
+            -0.011984107084572315,
+            -0.012310649268329144,
+            0.007649256382137537,
+            0.021306892856955528,
+            -0.006522684823721647,
+            -0.008579902350902557,
+            0.025372346863150597,
+            -0.011788181029260159,
+            -0.010882026515901089,
+            -0.01661284640431404,
+            -0.01635977439582348,
+            0.00241641397587955,
+            0.019902760162949562,
+            -0.0169475506991148,
+            0.021176276728510857,
+            0.00936360377818346,
+            0.0010133018950000405,
+            0.003624620847404003,
+            0.008588065393269062,
+            0.0024878450203686953,
+            0.017878197133541107,
+            0.003883813973516226,
+            -0.01439235731959343,
+            0.0021408938337117434,
+            -0.0030286810360848904,
+            0.004624656867235899,
+            3.4320437407586724e-06,
+            -0.002749078907072544,
+            -0.0065635028295218945,
+            0.003710338147357106,
+            -0.021306892856955528,
+            0.021502818912267685,
+            -0.022678371518850327,
+            -0.004245051648467779,
+            0.005555302836000919,
+            -0.005118552595376968,
+            0.013469874858856201,
+            0.018792515620589256,
+            0.01423724927008152,
+            0.008702355436980724,
+            0.01462093647569418,
+            -0.011331021785736084,
+            0.02323349379003048,
+            -0.013282112777233124,
+            0.0005816536722704768,
+            0.012939243577420712,
+            0.01755165494978428,
+            0.018792515620589256,
+            -0.006245123688131571,
+            -0.010384049266576767,
+            -0.0002946535241790116,
+            0.0129637336358428,
+            -0.022286519408226013,
+            -0.012269831262528896,
+            -0.01386172603815794,
+            0.011486129835247993,
+            -0.004743028432130814,
+            0.025453981012105942,
+            -0.00633492274209857,
+            -0.014653591439127922,
+            -0.015918942168354988,
+            0.014196431264281273,
+            -0.048752784729003906,
+            0.0030348035506904125,
+            -0.0024919267743825912,
+            0.03043375350534916,
+            -0.027021385729312897,
+            0.01738838478922844,
+            -0.0028490826953202486,
+            -0.009355440735816956,
+            0.015429128892719746,
+            -0.015992414206266403,
+            -0.0046613928861916065,
+            -0.0016806729836389422,
+            -0.007539048325270414,
+            -0.00112248957157135,
+            0.017616963014006615,
+            -0.015845470130443573,
+            0.01282495353370905,
+            -0.00989423505961895,
+            0.0027021386194974184,
+            -0.004526694305241108,
+            -0.007751300930976868,
+            0.0196088720113039,
+            0.034548189491033554,
+            -0.007053316105157137,
+            0.0014990337658673525,
+            -0.022547753527760506,
+            0.013192313723266125,
+            -0.006730855442583561,
+            0.02351105399429798,
+            -0.005432849284261465,
+            -0.016898570582270622,
+            -0.013976015150547028,
+            -0.009739127941429615,
+            0.00839214026927948,
+            -0.022433463484048843,
+            -0.011110605672001839,
+            -0.00922482367604971,
+            -0.002573562553152442,
+            -0.018694553524255753,
+            0.004865481983870268,
+            -0.002426618244498968,
+            -0.012506574392318726,
+            -0.008816645480692387,
+            0.01048201136291027,
+            -0.009347276762127876,
+            0.003077662317082286,
+            -0.0011929002357646823,
+            -0.011453475803136826,
+            -0.02369065210223198,
+            0.0158944521099329,
+            0.0001974561164388433,
+            -0.010359558276832104,
+            0.008408467285335064,
+            -0.0018276170594617724,
+            0.001211268245242536,
+            0.016963878646492958,
+            -0.0020664012990891933,
+            0.003467472270131111,
+            0.023478399962186813,
+            0.004057289566844702,
+            -0.010971825569868088,
+            -0.00025906547671183944,
+            0.013535182923078537,
+            -0.005118552595376968,
+            -0.010008525103330612,
+            1.6646010408294387e-05,
+            -0.02984597720205784,
+            -0.008351322263479233,
+            0.006094098091125488,
+            0.018629245460033417,
+            -0.010620792396366596,
+            -0.015853634104132652,
+            0.026319319382309914,
+            -0.006359413731843233,
+            0.019070077687501907,
+            0.002889900468289852,
+            0.0025694805663079023,
+            0.01724144071340561,
+            0.013461710885167122,
+            0.00786150898784399,
+            -0.013927034102380276,
+            -0.02429475635290146,
+            0.0016398552106693387,
+            -0.018090449273586273,
+            0.010579974390566349,
+            -0.011028970591723919,
+            0.0032756286673247814,
+            -0.02398454025387764,
+            -0.012237177230417728,
+            0.004934872500598431,
+            -0.020343592390418053,
+            -0.00040588201954960823,
+            -0.0056940834037959576,
+            0.022237539291381836,
+            -0.005310396198183298,
+            0.008669701404869556,
+            0.0018255761824548244,
+            -0.012237177230417728,
+            0.0009877908742055297,
+            0.009469729848206043,
+            -0.029192892834544182,
+            -0.0007127808639779687,
+            -0.003138889092952013,
+            0.0011418780777603388,
+            0.005351213738322258,
+            0.009755454957485199,
+            0.007322713732719421,
+            0.0027062203735113144,
+            0.007253323681652546,
+            0.010449357330799103,
+            -0.00022041612828616053,
+            0.0197884701192379,
+            0.018172085285186768,
+            -0.007351286243647337,
+            0.008710519410669804,
+            0.024817224591970444,
+            -0.013159659691154957,
+            -0.008718682453036308,
+            0.023266147822141647,
+            -0.006461458280682564,
+            -0.0011122850701212883,
+            -0.01483318954706192,
+            0.00815947912633419,
+            0.0017367975087836385,
+            0.028131630271673203,
+            0.019657853990793228,
+            0.005820618476718664,
+            0.000639819074422121,
+            0.015788326039910316,
+            -0.013020878657698631,
+            0.0035286990460008383,
+            0.015461783856153488,
+            -0.003940958995372057,
+            -0.019821124151349068,
+            0.013878053054213524,
+            0.009502384811639786,
+            -0.003228688146919012,
+            0.01551892887800932,
+            0.011412657797336578,
+            -0.030841931700706482,
+            -0.0011898389784619212,
+            0.010498338378965855,
+            -0.0021449755877256393,
+            -0.008751336485147476,
+            -0.00014273475972004235,
+            -0.012384121306240559,
+            0.021388528868556023,
+            -0.004408322740346193,
+            0.012277995236217976,
+            -0.00815947912633419,
+            -0.017127150669693947,
+            -0.011559601873159409,
+            0.007914572022855282,
+            -0.010547320358455181,
+            -0.0006714528426527977,
+            -0.004022594541311264,
+            -0.005779800936579704,
+            0.017747581005096436,
+            0.004706292413175106,
+            -0.0008964610169641674,
+            -0.0018398624379187822,
+            0.007983962073922157,
+            -0.006979844067245722,
+            -0.008726846426725388,
+            0.004816500470042229,
+            -0.012833117507398129,
+            0.003981776535511017,
+            0.005543057806789875,
+            0.01506176870316267,
+            -0.014857680536806583,
+            -0.021159948781132698,
+            0.007020662073045969,
+            0.02695607766509056,
+            -0.02516009286046028,
+            -0.005232842173427343,
+            0.006245123688131571,
+            0.002093953313305974,
+            0.006914535537362099,
+            -0.006934944540262222,
+            -0.007208424154669046,
+            -0.03657275065779686,
+            -0.0020745648071169853,
+            0.0009005427709780633,
+            0.008963589556515217,
+            -0.008049271069467068,
+            -0.0136494729667902,
+            -0.011135096661746502,
+            0.002410291228443384,
+            0.007134951651096344,
+            -0.02204161323606968,
+            0.0049593630246818066,
+            0.009608510881662369,
+            -0.019559891894459724,
+            0.025127438828349113,
+            -0.010890189558267593,
+            -6.964537897147238e-05,
+            -0.00995954405516386,
+            0.021633435040712357,
+            -0.013486201874911785,
+            -0.018368011340498924,
+            -0.0006898208521306515,
+            0.008355404250323772,
+            0.011926962062716484,
+            -0.0017663904000073671,
+            -0.011371839791536331,
+            0.0003752686607185751,
+            0.01646590232849121,
+            0.019135385751724243,
+            0.002744997153058648,
+            -0.0024837632663547993,
+            -0.01019628718495369,
+            -0.009592183865606785,
+            -0.0073961857706308365,
+            -0.0007449248805642128,
+            0.002091912319883704,
+            0.02035992033779621,
+            0.013469874858856201,
+            0.0024837632663547993,
+            -0.015135240741074085,
+            -0.005257333163172007,
+            0.008939098566770554,
+            0.009412584826350212,
+            -0.011690218932926655,
+            -0.00898808054625988,
+            0.002659279853105545,
+            0.02249877341091633,
+            0.017633290961384773,
+            -0.0014000505907461047,
+            0.0032878739293664694,
+            -0.0016592437168583274,
+            0.012506574392318726,
+            -0.007555375341325998,
+            -0.010269759222865105,
+            -0.006943108048290014,
+            -0.014645427465438843,
+            -0.010800390504300594,
+            -0.0023062059190124273,
+            -0.010179960168898106,
+            0.022629389539361,
+            0.007988044060766697,
+            0.009779945947229862,
+            0.008265605196356773,
+            0.010269759222865105,
+            0.002087830565869808,
+            0.025715215131640434,
+            -0.004424649756401777,
+            -0.011984107084572315,
+            0.007236996199935675,
+            -0.022368155419826508,
+            -0.021535472944378853,
+            0.008257441222667694,
+            0.006367577239871025,
+            0.03595232218503952,
+            -0.026890767738223076,
+            0.0052981507033109665,
+            -0.009779945947229862,
+            -0.010400376282632351,
+            0.007996207103133202,
+            -0.0015439334092661738,
+            -0.004245051648467779,
+            0.022368155419826508,
+            0.00726556871086359,
+            0.0035797213204205036,
+            0.0032817514147609472,
+            0.019837452098727226,
+            -0.005685919895768166,
+            0.01573934406042099,
+            0.0259274672716856,
+            0.024621298536658287,
+            0.009388094767928123,
+            -0.009469729848206043,
+            0.02785406820476055,
+            0.0002895512734539807,
+            -0.0034327772445976734,
+            0.007220669183880091,
+            -0.00305113079957664,
+            -0.0009556467994116247,
+            -0.010033016093075275,
+            0.021764053031802177,
+            -0.001455154619179666,
+            0.004359341226518154,
+            0.02081707864999771,
+            -7.774515688652173e-05,
+            0.006220633164048195,
+            0.01057181041687727,
+            0.002181711606681347,
+            0.003396041225641966,
+            0.010775899514555931,
+            -0.039315707981586456,
+            -0.0006571666453965008,
+            -0.025600925087928772,
+            -0.0018286375561729074,
+            0.0007357408758252859,
+            -0.01266984548419714,
+            0.013600491918623447,
+            -0.002015379024669528,
+            -0.010751409456133842,
+            -0.00029337796149775386,
+            0.021649762988090515,
+            0.005416522268205881,
+            0.004071575589478016,
+            -0.004469549283385277,
+            0.018792515620589256,
+            -0.006955353543162346,
+            -0.017861871048808098,
+            -0.012759645469486713,
+            -0.012367794290184975,
+            -0.01918436773121357,
+            -0.01214737817645073,
+            -0.0024551907554268837,
+            -0.002667443361133337,
+            0.004738946910947561,
+            0.002481722505763173,
+            -0.003228688146919012,
+            -0.02070278860628605,
+            0.015037278644740582,
+            -0.01430255826562643,
+            0.012873934581875801,
+            -0.011053460650146008,
+            0.005963481031358242,
+            -0.004122598096728325,
+            0.012726990506052971,
+            -0.021404854953289032,
+            0.02070278860628605,
+            -0.01536382082849741,
+            0.0038532004691660404,
+            0.019592545926570892,
+            0.006551257334649563,
+            -0.008118661120533943,
+            -0.013404566794633865,
+            -0.022074267268180847,
+            0.021633435040712357,
+            -0.011633073911070824,
+            -0.018237393349409103,
+            -0.004608329851180315,
+            0.009649328887462616,
+            -0.017470018938183784,
+            -0.007367613259702921,
+            0.01707816869020462,
+            -0.0018949664663523436,
+            0.012980061583220959,
+            0.023004913702607155,
+            -0.0052981507033109665,
+            0.0036572751123458147,
+            -0.009167678654193878,
+            0.0010286086471751332,
+            -0.02470293454825878,
+            0.013527019880712032,
+            0.008971752598881721,
+            0.006269614677876234,
+            -0.013935198076069355,
+            -0.013306603766977787,
+            0.0036531933583319187,
+            0.004498121794313192,
+            0.011437147855758667,
+            0.009420748800039291,
+            0.004220560658723116,
+            -0.013918871060013771,
+            -0.027788760140538216,
+            -0.027070365846157074,
+            0.03595232218503952,
+            -0.0005002731923013926,
+            -0.020947696641087532,
+            0.03399306535720825,
+            0.016114868223667145,
+            0.0022980421781539917,
+            -0.012449429370462894,
+            -0.004314441699534655,
+            0.024621298536658287,
+            0.016400592401623726,
+            0.010465684346854687,
+            -0.0004742518358398229,
+            0.005571629852056503,
+            0.0016469983384013176,
+            0.03012353740632534,
+            0.015323002822697163,
+            0.0009459525463171303,
+            -0.038597315549850464,
+            0.011943289078772068,
+            -0.007547211833298206,
+            0.0013408648082986474,
+            0.006702283397316933,
+            0.017780235037207603,
+            0.011935125105082989,
+            0.015567909926176071,
+            0.014865843579173088,
+            -0.003065417055040598,
+            0.0029511270113289356,
+            0.0006163488142192364,
+            -0.009624837897717953,
+            -0.001767410896718502,
+            -0.008881953544914722,
+            -0.005714492406696081,
+            -0.002920513739809394,
+            0.005592038854956627,
+            0.007657419890165329,
+            -0.003983817528933287,
+            0.028049994260072708,
+            -0.0006433906382881105,
+            -0.017600636929273605,
+            -0.009061552584171295,
+            -0.03611559048295021,
+            0.00434301421046257,
+            -0.030939893797039986,
+            -0.011257549747824669,
+            0.0033674687147140503,
+            -0.020457882434129715,
+            -0.024768242612481117,
+            0.0010306495241820812,
+            0.014335212297737598,
+            0.007392104249447584,
+            0.0033327736891806126,
+            -0.01004934310913086,
+            0.003816464450210333,
+            -0.001855169073678553,
+            -0.006461458280682564,
+            0.028833696618676186,
+            0.019559891894459724,
+            0.020457882434129715,
+            -0.01190247107297182,
+            0.019298657774925232,
+            0.005845109466463327,
+            0.02035992033779621,
+            0.010465684346854687,
+            -0.0048695639707148075,
+            0.0074859848245978355,
+            0.009567692875862122,
+            -0.006155324634164572,
+            -0.021519144997000694,
+            0.013192313723266125,
+            -0.0010837126756086946,
+            -0.0052083516493439674,
+            -0.016269976273179054,
+            -0.003534821793437004,
+            -0.015812816098332405,
+            0.010547320358455181,
+            -0.017159804701805115,
+            0.012498411349952221,
+            -0.000802069844212383,
+            0.00168475485406816,
+            -0.015469946898519993,
+            -0.013951525092124939,
+            -0.0012439225101843476,
+            -0.01244126632809639,
+            -0.008971752598881721,
+            0.006163488142192364,
+            -0.007539048325270414,
+            -0.008318668231368065,
+            -0.00535937724635005,
+            -0.011437147855758667,
+            -0.010882026515901089,
+            0.004367504734545946,
+            -0.0010653446661308408,
+            0.020931368693709373,
+            0.00011671340325847268,
+            -0.005824700463563204,
+            0.01190247107297182,
+            0.02787039615213871,
+            0.0016704685986042023,
+            0.004583839327096939,
+            -0.014261740259826183,
+            -0.011118769645690918,
+            -0.02279266156256199,
+            0.016572028398513794,
+            0.0049675265327095985,
+            -0.0344502255320549,
+            0.01438419334590435,
+            0.00824927818030119,
+            0.0015102586476132274,
+            -0.006698201410472393,
+            -0.005334886722266674,
+            -0.005608365871012211,
+            0.006306350696831942,
+            -0.0022837561555206776,
+            0.0005107327597215772,
+            0.0035919665824621916,
+            0.034678805619478226,
+            -0.0040674940682947636,
+            -0.007228832691907883,
+            0.00981259997934103,
+            0.0005130287609063089,
+            0.020196648314595222,
+            -0.005518566817045212,
+            0.0037572786677628756,
+            -0.019282329827547073,
+            -0.016563864424824715,
+            0.0018429238116368651,
+            -0.01722511276602745,
+            0.023282473906874657,
+            -0.005392031744122505,
+            -0.01347803883254528,
+            -0.006906372029334307,
+            -0.020115012302994728,
+            0.012637191452085972,
+            0.02310287579894066,
+            0.007016580086201429,
+            0.01033506728708744,
+            0.012131051160395145,
+            -0.02444170042872429,
+            -0.020457882434129715,
+            0.012171869166195393,
+            0.0013571919407695532,
+            -0.008449285291135311,
+            -0.005404277238994837,
+            -0.001783737912774086,
+            0.007783954963088036,
+            -0.0032633834052830935,
+            0.007881917990744114,
+            0.06491663306951523,
+            -0.01093100756406784,
+            -0.002987863030284643,
+            -0.02966637909412384,
+            -0.004396077245473862,
+            -0.006628811359405518,
+            0.0056287748739123344,
+            0.009151351638138294,
+            -0.023135529831051826,
+            0.009232986718416214,
+            -0.030025575309991837,
+            -0.010628955438733101,
+            0.00824111420661211,
+            0.0028490826953202486,
+            -0.01220452319830656,
+            -0.0009592183632776141,
+            -0.013665799982845783,
+            0.0006107363733462989,
+            -0.009412584826350212,
+            -0.00576347392052412,
+            0.012718827463686466,
+            -0.02321716584265232,
+            0.006298186723142862,
+            0.007592111360281706,
+            -0.006514521315693855,
+            -0.013722945004701614,
+            -0.017861871048808098,
+            -0.011967780068516731,
+            -0.013314766809344292,
+            0.002996026771143079,
+            0.01190247107297182,
+            -0.02607441134750843,
+            0.012922916561365128,
+            -0.0037389106582850218,
+            0.00945340283215046,
+            0.004404240753501654,
+            0.013469874858856201,
+            0.018074123188853264,
+            0.009355440735816956,
+            0.0021429345943033695,
+            -0.008024780079722404,
+            0.012702500447630882,
+            -0.01993541419506073,
+            -0.01635161228477955,
+            0.013837235048413277,
+            0.012016761116683483,
+            0.003061335301026702,
+            0.009828926995396614,
+            0.0020959940738976,
+            0.011886144056916237,
+            0.010775899514555931,
+            -0.019739490002393723,
+            -0.0006648199632763863,
+            0.0007326795603148639,
+            0.03280118480324745,
+            -0.0005938990507274866,
+            0.008857463486492634,
+            0.004420568235218525,
+            0.013159659691154957,
+            -0.012784135527908802,
+            -0.01769859902560711,
+            0.0008087027235887945,
+            -0.01057181041687727,
+            0.021257910877466202,
+            -0.008192133158445358,
+            0.006216551177203655,
+            -0.018612917512655258,
+            -0.027086693793535233,
+            0.006392067763954401,
+            -0.016261812299489975,
+            -0.018221067264676094,
+            0.01317598670721054,
+            0.0077268099412322044,
+            0.0017572063952684402,
+            -0.005098143592476845,
+            -0.020457882434129715,
+            -0.009837090037763119,
+            0.028441844508051872,
+            0.011102442629635334,
+            -0.004120557103306055,
+            0.009584019891917706,
+            -0.009094206616282463,
+            -0.012490247376263142,
+            -0.0017541450215503573,
+            0.0036715613678097725,
+            0.008261523209512234,
+            0.005645101889967918,
+            -0.01386172603815794,
+            -0.00029873527819290757,
+            -0.012400448322296143,
+            -0.021225256845355034,
+            -0.008906444534659386,
+            0.016751626506447792,
+            0.007820690982043743,
+            -0.0038960592355579138,
+            0.015812816098332405,
+            -0.004404240753501654,
+            0.002189875114709139,
+            0.014294394291937351,
+            0.005334886722266674,
+            -0.004285869188606739,
+            -0.0095758568495512,
+            -0.004612411838024855,
+            0.004043003544211388,
+            -0.0025960123166441917,
+            0.006457376293838024,
+            0.017192458733916283,
+            0.0072900597006082535,
+            -0.013755599036812782,
+            0.013135168701410294,
+            0.0007791098323650658,
+            0.0049675265327095985,
+            -0.03699725493788719,
+            -0.003247056156396866,
+            0.015796490013599396,
+            -0.013020878657698631,
+            0.02501314878463745,
+            0.005400195252150297,
+            0.022743679583072662,
+            0.015233203768730164,
+            0.008669701404869556,
+            -0.001542912912555039,
+            0.00464098434895277,
+            0.02759283408522606,
+            -0.003151134354993701,
+            -0.008637047372758389,
+            -0.0020602785516530275,
+            0.009241150692105293,
+            0.0002063850115519017,
+            0.003000108525156975,
+            0.008281932212412357,
+            -0.03562577813863754,
+            0.011706545948982239,
+            -0.019576217979192734,
+            0.0024694770108908415,
+            0.003459308762103319,
+            -0.007183933164924383,
+            -0.014865843579173088,
+            -0.005857354495674372,
+            0.022841641679406166,
+            -0.003224606392905116,
+            0.021078312769532204,
+            0.002885818714275956,
+            0.016106704249978065,
+            -0.011575928889214993,
+            -0.009208496659994125,
+            -0.0004130251472815871,
+            -0.005351213738322258,
+            -0.007539048325270414,
+            0.010596301406621933,
+            0.0034409407526254654,
+            0.005534893833100796,
+            0.002265388146042824,
+            0.0036654386203736067,
+            -0.0036878883838653564,
+            0.02773977816104889,
+            0.003563394071534276,
+            0.007996207103133202,
+            -0.011567764915525913,
+            0.014082142151892185,
+            -0.017029186710715294,
+            0.0070778070949018,
+            0.005575711838901043,
+            -0.026156047359108925,
+            0.016849588602781296,
+            0.004775682929903269,
+            0.006628811359405518,
+            0.013812744058668613,
+            0.01618834026157856,
+            -0.010408539324998856,
+            -0.014816862531006336,
+            0.024049850180745125,
+            -0.0032041973900049925,
+            -0.015225040726363659,
+            -0.006775755435228348,
+            4.438935866346583e-05,
+            0.006559420842677355,
+            0.00373074715025723,
+            0.0013020879123359919,
+            -0.0006204306264407933,
+            -0.010743245482444763,
+            0.004645065870136023,
+            -0.01934763789176941,
+            -0.0007163524278439581,
+            -0.016449574381113052,
+            -0.015633217990398407,
+            0.018678225576877594,
+            -0.01830270327627659,
+            0.009633000940084457,
+            0.0012275953777134418,
+            0.028588788583874702,
+            -0.011706545948982239,
+            -0.012155541218817234,
+            0.0015102586476132274,
+            0.010849371552467346,
+            0.0067512644454836845,
+            -0.010033016093075275,
+            -0.007583947852253914,
+            -0.014180104248225689,
+            -0.010163633152842522,
+            0.015502601861953735,
+            -0.014498483389616013,
+            -0.014343376271426678,
+            0.008914608508348465
+          ],
+          "embedding": [
+            -0.006359657085039302,
+            -0.024594786241399234,
+            -0.02553538924848095,
+            0.005360188465088291,
+            0.012801870977589839,
+            0.021680505862506104,
+            -0.016668048966001273,
+            0.06860852474581301,
+            -0.007122272841157724,
+            0.03471775000705488,
+            -0.009004297848861675,
+            0.01059974144678391,
+            0.02565604599426148,
+            -0.022544127646041287,
+            0.010616674976540785,
+            0.013658194823359737,
+            0.016400329743363844,
+            0.02858546964917478,
+            0.0067612676434693686,
+            -0.04174737193606879,
+            0.03184155110488616,
+            -0.02223105127585514,
+            -0.05506239667974977,
+            -0.0015307551410555752,
+            -0.013169788751379287,
+            0.017681749722384007,
+            0.01682706317229353,
+            0.019652295823816928,
+            -0.05623980369271269,
+            0.030698582851533197,
+            0.02286327124296613,
+            0.016019032539689178,
+            0.005508113353872315,
+            8.58985091215149e-05,
+            -0.019206292853495906,
+            -0.018439290483495144,
+            0.03475023176950151,
+            0.01018612942141759,
+            -0.0252551158857426,
+            0.0334562130666827,
+            0.020462153537485785,
+            0.02260508249714211,
+            -0.03323524989392951,
+            -0.002094251009530707,
+            0.015177442347686773,
+            -0.00018698464350038317,
+            -0.0006788648344273403,
+            0.0007777616893581039,
+            0.013352398615059647,
+            -0.030751111528744813,
+            -0.0317733818445192,
+            -0.006729064081710058,
+            0.009413204899986514,
+            -0.005712894407943592,
+            -0.038961826880527066,
+            -0.002827692206530345,
+            0.017863300537973137,
+            0.07918455388563496,
+            0.048967158919816366,
+            0.018002566325350088,
+            0.040264962620950695,
+            -0.019715100359659787,
+            -0.02211350500879408,
+            0.004918148608475574,
+            0.01062585305866606,
+            -0.014478993521696027,
+            0.0034983131226364672,
+            0.000766228629514787,
+            0.009667268771502067,
+            -0.01558439890697323,
+            -0.04984718709870083,
+            0.029613955432821887,
+            -0.009686854114630624,
+            -0.015279319238124568,
+            0.007460629676325322,
+            -0.006235979350809034,
+            0.012514387473717085,
+            0.02800435263741497,
+            -0.008688925946672778,
+            -0.01236301543374469,
+            0.029755705020188972,
+            0.01158037219446542,
+            -0.005242392206024511,
+            0.009982679027728814,
+            0.052414361714483264,
+            -0.018788500381726227,
+            -0.020335831196027204,
+            -0.08583959943356621,
+            0.029309228216812346,
+            -0.0249285282010673,
+            -0.018929396555426307,
+            0.007710663193334574,
+            0.03096798035958051,
+            -0.017089977064591828,
+            -0.028015107605405817,
+            0.06740703271128606,
+            0.009398795412354399,
+            0.01118769022826835,
+            -0.03041417460855722,
+            0.014776002883150835,
+            0.004319194067673161,
+            0.022507636639041203,
+            0.0053562788646786845,
+            0.007548072258943124,
+            0.011123374308563128,
+            -0.006924362115585165,
+            -0.011118507147092024,
+            0.009563232643295304,
+            0.00657178473678069,
+            0.08313027447380966,
+            -0.004239370156003264,
+            -0.00930406163946907,
+            -0.039202672821898946,
+            0.009761029930556298,
+            -0.02827698465424993,
+            -0.037914533246900244,
+            -0.04496050911855815,
+            -0.017711307900236353,
+            0.01934435542287117,
+            0.004335173027873153,
+            0.03800509969714242,
+            0.031198127357716314,
+            0.031311207941680276,
+            -0.012184553894588813,
+            -0.058985863670856876,
+            0.041442017057707635,
+            -0.012456189292721711,
+            -0.0040966351720394135,
+            0.010849179519959421,
+            0.005243123049347636,
+            0.0011495799963862647,
+            0.003534183676934487,
+            -0.03139340512003631,
+            -0.023876261883905306,
+            0.008382957569752911,
+            -0.004778283217011976,
+            0.03703067589888386,
+            0.06888630688839406,
+            -0.03993532296478411,
+            -0.009472167678222372,
+            0.025246872942105057,
+            -0.01524567912700368,
+            0.01595917939878582,
+            -0.003736723164442467,
+            -0.030836797784967753,
+            0.0026246446163368533,
+            0.008894300082148884,
+            0.08342856279154155,
+            -0.0007476580521704578,
+            0.025149116238550588,
+            0.004972542633631468,
+            0.034495632546556576,
+            -0.009173329764686424,
+            0.014009381543772768,
+            0.004427261668151028,
+            -0.013426138607902904,
+            -0.022234568880321808,
+            0.01976282364689966,
+            -0.03592433789490043,
+            -0.010918278220831188,
+            0.013527180289641174,
+            -0.013023113052242677,
+            -0.008304770302495363,
+            0.003677063799167178,
+            0.0013981822663219421,
+            0.047001163242600175,
+            0.01828332714887986,
+            0.002483375662343689,
+            -0.06820498187523893,
+            0.05295312038995555,
+            0.025038753504342162,
+            0.025286870198436735,
+            -0.0029544808216316115,
+            0.029982752196635428,
+            0.0005600481523874767,
+            0.0128290258663539,
+            0.009023805227424919,
+            -0.0019219166367750596,
+            -0.04482843936478531,
+            0.04523479691409549,
+            -0.004678574027241186,
+            0.010486850804218904,
+            0.006736229865018014,
+            0.02134813656476443,
+            0.028083972673690036,
+            0.016857413440125445,
+            0.0189082447820146,
+            0.03843690613820022,
+            -0.011922027011229613,
+            0.016193433524453224,
+            0.03828207911467065,
+            -0.0013055128404331705,
+            0.043198163443700956,
+            0.024025562893260542,
+            -0.001257851946749882,
+            -0.011726713433348844,
+            -0.021907984099443396,
+            0.02752198420597673,
+            0.004431866336999851,
+            0.015798806714413912,
+            -0.012910782191382367,
+            -0.03180549180206364,
+            0.004550247372715741,
+            -0.022778272841118514,
+            0.031242218240017532,
+            -0.006106984949693483,
+            -0.006259993190497421,
+            0.02187449047033339,
+            0.009334146847221406,
+            -0.009381278960016145,
+            -0.020888853823931134,
+            -0.007502374908776156,
+            -0.010390018386728482,
+            -0.04316693435221676,
+            -0.006274513730720055,
+            0.019152443937064258,
+            -0.01668984824845791,
+            0.018573194036505578,
+            0.0035035262183249925,
+            -0.019779911381789077,
+            -0.0027495798165085993,
+            0.02425481236315913,
+            0.006857723545270935,
+            -0.014733535699087731,
+            -0.026326146471941727,
+            -0.019906660231056833,
+            -0.001033294559612807,
+            0.0159154063026895,
+            0.035649534442431415,
+            -0.026210592096240003,
+            0.014397127229517285,
+            0.005114311681443673,
+            -0.0033104236005207077,
+            0.00324988235167305,
+            0.03764004154553922,
+            -0.017822437760737605,
+            -0.008579701063708334,
+            -0.00032410984200965487,
+            0.018450351071676443,
+            0.017283379764846195,
+            0.006384411611332382,
+            -0.005115090309123598,
+            -0.030590047062583125,
+            -0.011776211359193503,
+            0.01183623914060493,
+            0.0016564847796208331,
+            -0.009530766634361575,
+            -0.004810598258154945,
+            0.03876242515526699,
+            0.027076785774401667,
+            0.005324763136165989,
+            -0.03416412480913939,
+            -0.007943016733586697,
+            0.002114307650456928,
+            -0.02961449347300628,
+            0.05142732770518828,
+            0.05566026433842563,
+            -0.0030729309664974704,
+            -0.00938186948393074,
+            -0.04547908981236149,
+            0.0651216359692231,
+            -0.01813722445944094,
+            -0.0482650853560993,
+            0.044045155892545554,
+            -0.014272653003441165,
+            0.013740652306746851,
+            0.04563732439184395,
+            0.02126513340521316,
+            -0.0054845632439384905,
+            -0.011635625101262898,
+            0.030992082069685376,
+            -0.010051935721068779,
+            -0.050967258971854425,
+            -0.009827942095033494,
+            0.013373878556654464,
+            -0.013017627321534534,
+            -0.009361001542607812,
+            -0.01925867665425919,
+            0.020329052915436022,
+            0.023020741670660188,
+            0.019387043947695172,
+            0.005628632318152202,
+            0.018451555161952665,
+            0.016226845400848455,
+            -0.01384033112836216,
+            0.004762589697426251,
+            -0.012718476561725138,
+            -0.024222765127494463,
+            -0.037487144314010595,
+            -0.019971524166659278,
+            -0.015034588610516958,
+            0.009483936496215338,
+            -0.014301924283553591,
+            -0.0024139665139412125,
+            0.013361045353529458,
+            -0.0005870798721107379,
+            -0.011192319340879789,
+            0.013927695434391159,
+            0.021337539669342293,
+            -0.027300952495815437,
+            -0.04765670695199488,
+            -0.00561706224251562,
+            0.0082776639906139,
+            -0.0030984973749633724,
+            -0.026269016057688548,
+            0.007040524859505573,
+            0.0017859008072463442,
+            0.020091542517503808,
+            -0.018716761130867725,
+            0.0035283051960672963,
+            0.008519383548064053,
+            -0.015247417050336866,
+            -0.016917626868166634,
+            -0.018580550690103854,
+            -0.024697180580294884,
+            0.034202028316737125,
+            0.0204395003463227,
+            0.030392895534267717,
+            -0.04527669147753553,
+            -0.018359428092075348,
+            0.05494107499558624,
+            0.015234754907198422,
+            -0.05066008106884831,
+            0.031139824660661085,
+            0.005016615710206006,
+            0.0019000453624929544,
+            -0.006094364589229031,
+            0.0637324238017742,
+            -0.02711381489292883,
+            -0.02187418906973867,
+            0.004601744864532875,
+            -0.04141121277496072,
+            -0.026689585608019588,
+            -0.005568100750837504,
+            0.01291019036002459,
+            -0.031853041426355644,
+            -0.02815800259372657,
+            -0.007964861641790374,
+            -0.010933515363060793,
+            0.013726639077745378,
+            -0.023956800669647334,
+            -0.015672058590455337,
+            -0.023642025374517057,
+            -0.01288435863544158,
+            -0.0042516881552192385,
+            -0.011162214160460695,
+            -0.02412733256710621,
+            0.010088528028211243,
+            0.03310175458095482,
+            0.023064360903748363,
+            0.001996159684818467,
+            0.009478229572933101,
+            0.014478772877286076,
+            0.00371783700962415,
+            0.007864103610320116,
+            -0.048511515124609,
+            -0.031398754206534776,
+            0.020227872561659397,
+            0.012159811587094092,
+            0.0172163448580163,
+            0.009578952796095119,
+            -0.04154545961112031,
+            -0.05095080393682759,
+            0.02968486144855133,
+            -0.03932395023720674,
+            -0.02063219739061335,
+            0.007512423249865194,
+            -0.011911709468952548,
+            -0.00810951734088105,
+            -0.0008079327452253181,
+            -0.01600701450676789,
+            0.023984909975645632,
+            -0.02418072541081482,
+            0.0033984265728439398,
+            -0.0104343782811971,
+            0.01588977364765743,
+            0.05109529246167992,
+            -0.027800269136366997,
+            -0.027889034481533438,
+            0.03706813210665186,
+            -0.02058120652429308,
+            0.030801396468451574,
+            0.03246127847621613,
+            0.008355517405358826,
+            -0.008605467586247932,
+            0.017693344563968573,
+            0.014499487024471321,
+            0.012431277080478217,
+            -0.040103870071440256,
+            0.017328848567062113,
+            0.02723900253727976,
+            -0.054909074068976875,
+            -0.008547596270650966,
+            -0.00777964838164648,
+            0.027134666572242665,
+            0.02765036594970255,
+            -0.002207173084941375,
+            -0.016838740117653725,
+            -0.028289233739722936,
+            -0.019433376533829132,
+            -0.011053380458946764,
+            0.03247641283550457,
+            0.0007514989009289282,
+            0.007919574219000797,
+            0.001316042436488591,
+            -0.0017934342041762042,
+            0.01924145734197619,
+            0.015260199553572459,
+            0.048259602293134723,
+            -0.03383050622747848,
+            0.0021712225589426266,
+            -0.03508567715696004,
+            -0.04578731401250471,
+            -0.008768228935292693,
+            0.010203318819548585,
+            0.024135236371743855,
+            -0.007467140460133838,
+            0.0028907962385387818,
+            -0.007582083854422456,
+            -0.005384477718912037,
+            -0.01867978482547249,
+            0.01641191108381631,
+            0.037168089880543244,
+            0.0031992495663980464,
+            -0.008533890994695162,
+            0.000778843942291118,
+            -0.004620258538699103,
+            0.010572690335163528,
+            0.033233014627100904,
+            -0.030497362482602697,
+            0.02974571152880676,
+            -0.004274442492343003,
+            0.025341347125910928,
+            -0.04307748954707389,
+            -0.018069609916024403,
+            -0.018491122727274337,
+            -0.02069307802429807,
+            0.05372967226264594,
+            -0.030768978301976772,
+            -0.03662045886993282,
+            -0.003205991669059688,
+            0.016815511161004534,
+            -0.0014152644009276566,
+            -0.0016015439520090104,
+            -0.03984716970293867,
+            -0.010828260334550217,
+            -0.05894985435251326,
+            -0.01638035185618134,
+            -0.021819442096404948,
+            -0.005315102328745982,
+            -0.06251579337157112,
+            0.0033713876515840233,
+            -0.07815974854883057,
+            0.001484860071649606,
+            0.04539427129131382,
+            -0.023530943197353476,
+            0.05142250021536681,
+            -0.024746600229063383,
+            -0.0005885740455257141,
+            -0.044698536284421976,
+            0.0026481001040031785,
+            0.02389178603305878,
+            0.051387657872255674,
+            -0.002949198236157656,
+            0.010674377196609455,
+            0.013322075519401089,
+            -0.026245090721950647,
+            -0.01036724558050635,
+            -0.005321884180696542,
+            0.005958798671856245,
+            0.013755178085972732,
+            -0.0074734329537091675,
+            0.038368016046563674,
+            0.010941421833174982,
+            0.03443115781919206,
+            0.034623617136788874,
+            -0.004361926531008214,
+            -0.03348681614127998,
+            -0.00632870804659212,
+            0.024433331840566692,
+            0.0321998807702602,
+            -0.02899983280320611,
+            0.005958471722097962,
+            -0.015602447981516634,
+            0.015633217539747047,
+            0.006321931116948646,
+            -0.011827541320878442,
+            0.02372204564999124,
+            0.015673426811623124,
+            0.006103300725621424,
+            -0.005165006670545588,
+            0.013589526056189892,
+            -0.025411864292353578,
+            0.03041791947375232,
+            -0.004456865298584878,
+            -0.04276058054583262,
+            -0.0024736078733106825,
+            -0.020059309867945227,
+            0.024883721794000956,
+            0.015578237413840115,
+            -0.00536523508503184,
+            -0.005290645798641171,
+            0.008242987948511412,
+            0.020469675635821673,
+            0.006668354647245447,
+            0.00017729517736814364,
+            0.02373604785121342,
+            0.029915687122900485,
+            -0.02432635861789771,
+            -0.029540533943782886,
+            0.008151297444985344,
+            -0.005730733861939129,
+            0.011318816579801282,
+            -0.02644855229221747,
+            0.00033932872948409137,
+            0.0035505684154028996,
+            -0.03570104343808711,
+            0.03408366016010711,
+            -0.009199700348630527,
+            0.005857598530788056,
+            -0.009538177736472163,
+            -0.0017768806148210118,
+            0.0027410153099351694,
+            0.03898108970435115,
+            -0.007367623392751247,
+            0.009827254226889936,
+            0.015465895035342481,
+            0.000981604396724216,
+            0.05143083558797524,
+            -0.027350877485564314,
+            -0.02183036962301852,
+            0.01885468378168636,
+            -0.007285239007489837,
+            -0.016846837665812908,
+            -0.020546383007536074,
+            -0.011863943814684079,
+            -0.016293192750336218,
+            0.01976516340568883,
+            -0.02387328798681798,
+            -0.011892336568452852,
+            0.007170651391639504,
+            0.0013725014050721622,
+            -0.009809669068563384,
+            0.0016617183704652932,
+            -0.006307590349798603,
+            0.02037693835615797,
+            0.031753656615094446,
+            -0.024892675105483313,
+            0.03577919637821846,
+            0.03178303188315581,
+            0.017260898395251155,
+            -0.02102289690701718,
+            -0.009661035772073294,
+            -0.00048542689442823136,
+            -0.014694631967051945,
+            -0.0326379186158631,
+            0.025443206465204822,
+            0.017194626456982255,
+            -0.011381421003142453,
+            -0.03680783761649673,
+            0.0017216115811536721,
+            0.022819859205829785,
+            -0.013600989058799166,
+            0.028141657788496834,
+            -0.01869047298692401,
+            -0.01366833755869951,
+            -0.00782959271979894,
+            0.031227914441058244,
+            -0.05252392878125891,
+            0.016957842026348195,
+            0.00020880696574616416,
+            0.001154370575234767,
+            0.010892067708194324,
+            -0.008368819756628046,
+            0.012224564683325483,
+            0.031700389750709,
+            0.018063405869147564,
+            0.0010794115307202266,
+            -0.002783668216641561,
+            0.01418234057248712,
+            0.017773269840982066,
+            -0.015653443355569053,
+            -0.000497740485657824,
+            0.019890882604100452,
+            0.022672054781919238,
+            -0.00521797370595221,
+            -0.016686942830774276,
+            -0.018443538351123584,
+            0.025270057639201107,
+            -0.010354431070738671,
+            0.03484047753830186,
+            0.04594785214094975,
+            0.0036441959528283004,
+            -0.05695476763071743,
+            -0.009412209619190693,
+            0.0010371110229966086,
+            -0.01099343927053564,
+            0.01606863921935641,
+            0.012470660511773616,
+            -0.03237187923799263,
+            0.006388672213167589,
+            0.012946570716567364,
+            0.022152554488829627,
+            -0.02540317621005867,
+            -0.023183252921885226,
+            -0.0030984019855787655,
+            -0.007890467768631811,
+            0.018648760402624687,
+            -0.008653405980661855,
+            0.0006228315253506687,
+            0.009672526897422602,
+            0.0005436494014569835,
+            0.011632326145022935,
+            0.005655942007108228,
+            0.021736827172951,
+            -0.001256608167230203,
+            0.013124413886287987,
+            -0.01986808920858981,
+            -0.0270450339809516,
+            0.006500423802582728,
+            0.04438242119211055,
+            -0.014813399694814117,
+            0.030621709833336064,
+            0.0092171224268983,
+            0.016433384739642534,
+            0.024875072913970443,
+            -0.00891470467241686,
+            -0.014124009014664287,
+            0.00845777541223868,
+            0.009019007848927453,
+            -0.04147358041382793,
+            0.012513030150663913,
+            -0.01710243399313324,
+            -0.033144939719206276,
+            0.01904674924133127,
+            -0.00534122554342279,
+            0.012367320751926314,
+            0.004270315800911569,
+            -0.002886460711688153,
+            -0.025696286516581048,
+            0.014137901708875033,
+            0.028135087538950946,
+            0.021797869775098042,
+            0.0296173052503884,
+            -0.012757078368361444,
+            0.0005460601780756187,
+            -0.008748930817524899,
+            0.005470975782187157,
+            0.04187083128854477,
+            0.0008687182830616019,
+            -0.0008032802520818105,
+            0.01507726552487313,
+            0.024945744517309208,
+            0.0376130540370386,
+            -0.015415560911023553,
+            -0.03407353556457147,
+            0.01564969972264396,
+            -0.010104443707865262,
+            0.018855792875436674,
+            0.005503259932317811,
+            0.0049634097823351596,
+            0.0038600156268000702,
+            0.019312419743287703,
+            -0.006635193789316492,
+            0.018798045806429395,
+            -0.02500983683871332,
+            -0.028785757874583755,
+            0.01829411751138161,
+            -0.005277972443258226,
+            0.03338623299668019,
+            0.016225885095995672,
+            -0.029522280454919586,
+            -0.01712097344863975,
+            0.01553706187924322,
+            -0.008465766956343768,
+            -0.024842700926275538,
+            0.01895228828512998,
+            0.007076862934523977,
+            0.008956105593078965,
+            0.0024373779389833218,
+            -0.01106503696703573,
+            0.01125466547367417,
+            -0.000818223040067914,
+            -0.015322790791698395,
+            0.014157131555941659,
+            -0.017853386731281715,
+            -0.014994640991761465,
+            0.01775991780643474,
+            -0.037498078532902435,
+            0.009570197712419421,
+            -0.007687297543935123,
+            0.009050494195378254,
+            -0.004222041469820445,
+            0.007460383234429904,
+            -0.0010804379186060264,
+            -0.014987857934522812,
+            -0.014903749199810126,
+            0.03228485430019625,
+            0.022096121825475306,
+            -0.000410816936759558,
+            0.006728733918931589,
+            -3.971636461500232e-05,
+            -0.026719848032785824,
+            -0.011253751790407687,
+            0.02245467425830765,
+            -0.005664635254398481,
+            0.005134496095445943,
+            -0.01109590855846966,
+            -0.0149616794609307,
+            0.0019726017255571673,
+            0.032668637270108894,
+            0.06066635618119076,
+            0.012001732730659747,
+            0.024422093585231484,
+            0.0189499396761026,
+            0.02119276033547528,
+            -0.008489818917377516,
+            0.006413901834269739,
+            -0.03097413287789443,
+            0.03509225573986143,
+            0.005392594635384506,
+            0.03108937318306421,
+            0.026664331248813712,
+            -0.04722525921137681,
+            -0.0058539171830594774,
+            0.017087531454837677,
+            0.022673712717885323,
+            0.0159020681868943,
+            0.0008382498574190522,
+            -0.008938174268517353,
+            0.035650940044285984,
+            -0.03315250299348084,
+            -0.023647111263155,
+            0.005059382024092534,
+            -0.01439537467317451,
+            -0.010488359956731476,
+            -0.006785295580877329,
+            0.02776449681317373,
+            -0.005584052419598926,
+            -0.004582019433008779,
+            0.009697064252638535,
+            -0.014400206558190682,
+            0.008165676294736836,
+            0.0007624778531368436,
+            0.008681263897172326,
+            0.039249866981334126,
+            0.0005918063050375113,
+            0.024275699182767528,
+            -0.004138313072843555,
+            0.011079529739650594,
+            -0.0158972266904151,
+            -0.016985058543605,
+            0.02276982363172991,
+            0.00351101120496393,
+            -0.03244126813432563,
+            -0.00768819222024939,
+            0.00928180979884261,
+            -0.0013793404539641824,
+            0.005266058566444064,
+            -0.00503433670609399,
+            0.003747448261046857,
+            0.03286230666764075,
+            0.0022383592721685153,
+            0.008606190020480513,
+            0.013593947677257625,
+            0.00030951915677929204,
+            -0.01490307680740127,
+            0.013254591138113545,
+            -2.1379308136707722e-05,
+            -0.012900073369514622,
+            0.03357519186606058,
+            0.006705902094864628,
+            0.04243347200485736,
+            -0.03302953674711988,
+            0.009031534935013593,
+            0.01822227379789576,
+            -0.008491770335368761,
+            -0.02725412856390648,
+            -0.03630788717786838,
+            0.008259145747740535,
+            -0.019003260334340084,
+            0.015781275447792516,
+            -0.010039277287105153,
+            0.029665124310477384,
+            -0.03563370794521629,
+            0.004714858110268169,
+            0.004053913355055313,
+            0.0009503673086552651,
+            0.02068360255395961,
+            0.018524093113914366,
+            -0.002785698934204024,
+            -0.031443835341186485,
+            -0.02066365433385339,
+            0.006869321907853112,
+            0.02431260052207595,
+            -0.010367048093621498,
+            -0.005612149485859478,
+            0.0006316058139530722,
+            -0.024369562712568606,
+            -0.016845124984022828,
+            0.030359419843284823,
+            -0.040974427176153246,
+            0.005252911449822132,
+            0.04272936053727608,
+            -0.04599092715016919,
+            -0.03802288716943514,
+            -0.002314988728575789,
+            -0.024124014894379466,
+            -0.006360771882090754,
+            -0.002068030981218464,
+            0.016669991113175926,
+            0.0036232707799735537,
+            -0.028763764013482795,
+            0.00943401978066451,
+            -0.005173988601101547,
+            0.003971420424839974,
+            -0.0031669592494977393,
+            -0.02216336363004636,
+            -0.01657772032717298,
+            -0.02531962163186398,
+            -0.035174102848865116,
+            -0.0015903434876311088,
+            -0.005158703150821607,
+            -0.028703075279214722,
+            -0.03941940849540492,
+            -0.021696000925262537,
+            -0.0160221488026189,
+            -0.00843213135542359,
+            0.040641882367348774,
+            0.021319060666253945,
+            0.019336844555345886,
+            0.0025252167889382917,
+            -0.020346900609125973,
+            0.049440028814233025,
+            -0.007642182460707816,
+            0.035628691886108926,
+            -0.0017882888000877158,
+            -0.013461669667950893,
+            0.00455558389906464,
+            0.03825428026120858,
+            -0.014248434654950971,
+            -0.0009902062564027718,
+            0.00027811655483232545,
+            -0.0011866253757890136,
+            0.0008535155950871286,
+            0.015433650396777696,
+            0.001307651096343185,
+            -0.007112199543314309,
+            0.030380673552683906,
+            -0.012803247194895328,
+            0.01637501752192592,
+            0.018350814869962304,
+            -0.0013359719137099667,
+            0.008064546833360553,
+            -0.021430659268086814,
+            -0.00731572118780415,
+            0.0044578576331236325,
+            0.008422210247967642,
+            -0.015096885054883586,
+            -0.009964588397149773,
+            -0.02359494467299634,
+            -0.01174145797396426,
+            0.0023855749166744366,
+            -0.004559202896782688,
+            0.004902138778592298,
+            0.034595817216889944,
+            -0.016068744500427584,
+            -0.035106033294403614,
+            0.009997207702122002,
+            -0.02663088875167486,
+            0.02222306406130687,
+            3.983384942889326e-05,
+            0.006838221769859366,
+            -0.0023434342406307616,
+            0.001765445386886839,
+            -0.00169484278667461,
+            0.002862331185439823,
+            -0.018217975204395134,
+            0.0031859734246840257,
+            -0.0032119988065673565,
+            0.0009052950782312657,
+            -0.0012459653052875638,
+            0.00687339892509406,
+            0.006332799551164718,
+            -0.0008019538898592668,
+            -0.002968451740267742,
+            -8.541086355097327e-06,
+            -0.020561816931299912,
+            -0.021642512405352372,
+            -0.028259049261410905,
+            -0.000989887278779396,
+            -0.012359733017834338,
+            -0.027804394906433664,
+            0.011443385424045703,
+            -0.006872684999639775,
+            0.008327844771213648,
+            -0.008969527138034904,
+            0.006219994886600349,
+            -0.008415106130309062,
+            -0.0004143996206881767,
+            0.013119286511794273,
+            -0.02925209282917943,
+            -0.02096907906398536,
+            -0.03096513357712367,
+            0.01762223841790055,
+            -0.00925741670612824,
+            -0.002656382260603827,
+            0.0025223527960181835,
+            0.005092659166535706,
+            -0.033606542487570154,
+            -0.006653438998497412,
+            0.007313682256676618,
+            0.009547290751154595,
+            -0.03287788897629887,
+            0.02446728061355753,
+            0.017023222076526946,
+            0.01874233066565012,
+            -0.02417566312664911,
+            0.04049827067897005,
+            0.014426879325647042,
+            -0.021740606525480655,
+            0.016712907564723732,
+            0.027385150577780248,
+            0.03177163221499527,
+            -0.03469098547555545,
+            0.004481376248622418,
+            -0.01518200063132508,
+            0.018918680020516044,
+            -0.025284921244460287,
+            0.020625170445627613,
+            -0.016707629602243063,
+            0.021278719363178687,
+            0.03520334001264928,
+            -0.005524369131116028,
+            0.03608133014180745,
+            -0.019234548780647943,
+            0.0059985801736444555,
+            0.004041778461022664,
+            -0.01645503683802124,
+            0.01024694092614158,
+            -0.0042700351372843386,
+            0.0006059159967286885,
+            0.009406090340935747,
+            0.019133229156115282,
+            0.019007219188041775,
+            -0.04442208547053466,
+            0.028588965123434923,
+            0.00804120101594811,
+            0.021472086070876265,
+            0.014177820346835521,
+            -0.02708807970982664,
+            0.004742981711172693,
+            0.002072673190279628,
+            -0.004659713164250794,
+            0.0061167904730676514,
+            -0.013796604518930328,
+            -0.034540857808309226,
+            0.006338792319814448,
+            -0.018918703778187926,
+            0.015036937364271984,
+            -0.009026670935442384,
+            0.0011667525636770179,
+            -0.01152652811400482,
+            -0.011938679695598347,
+            0.0006170592505446954,
+            -0.005495151258143897,
+            -0.0008075736908949407,
+            -0.03171721592827769,
+            -0.04427658539412448,
+            0.02036132271908089,
+            -0.0021221965958423804,
+            -0.04320396743189005,
+            0.004111179168317842,
+            -0.025352428814134187,
+            -0.01865918186335911,
+            -0.011237699157960728,
+            -0.025047923887906474,
+            -0.04219630696740606,
+            -0.011108496740166623,
+            0.007785006066683012,
+            0.016810642610918525,
+            -0.009081815651299525,
+            0.040965574143584864,
+            0.0073944364393447965,
+            0.015313576815177263,
+            -0.017731360025694955,
+            0.026137777790243826,
+            -0.019314392060123,
+            -0.03108486330987073,
+            0.02676829566273746,
+            -6.259161089416372e-05,
+            0.003053748984540415,
+            0.014358356168574803,
+            2.8186924071913628e-05,
+            -0.006181789640967465,
+            0.01947730374003864,
+            -0.028739300217145797,
+            -0.023460503178221397,
+            0.0012460072421725688,
+            -0.008850395570726385,
+            -0.016271242375881602,
+            -0.035896509136022194,
+            0.005227881053956234,
+            0.03324337105109496,
+            0.004970419987430122,
+            0.008273497228454482,
+            0.01814500594735864,
+            -0.02229593571615308,
+            -0.024230675455380678,
+            0.000639112529814716,
+            -0.010266640429960994,
+            -0.010257247303470066,
+            -0.021302483296731352,
+            0.019150757066356872,
+            0.01651375496287087,
+            -0.00390189952706486,
+            -0.006971692709239087,
+            -0.010621003777733253,
+            -0.01918206725910585,
+            -0.02020345113211036,
+            -0.013827322378214754,
+            -0.017980206764311026,
+            -0.034153741666571154,
+            -0.032195001867229935,
+            0.03225205339201753,
+            0.026354817588919596,
+            0.0014076409241492998,
+            -0.007959081015805435,
+            -0.023070419006792155,
+            -0.004483328183581872,
+            0.02879516595345542,
+            0.002549934782150634,
+            -0.0001541849716285838,
+            -0.00830090759167373,
+            0.015341095409025125,
+            0.00835695019244165,
+            0.005107844984733648,
+            -0.06676974053142512,
+            -0.006027540973808174,
+            0.01655283585235268,
+            0.01351155740684989,
+            -0.002359114145814062,
+            0.02591301169926259,
+            0.01108099264396926,
+            -0.015361038608114896,
+            -0.019765743462262668,
+            0.035195761515422364,
+            0.02408910733949848,
+            0.023468767772473746,
+            -0.0034492999385544727,
+            -0.004472311490618405,
+            0.007030833258457338,
+            -0.010684252427867153,
+            -0.013326964647936429,
+            -0.004558513842504658,
+            0.026925146859337397,
+            0.016099443021514896,
+            -0.0064331198280825575,
+            0.010848931851939124,
+            -0.0008320209125061849,
+            -0.019444491995041537,
+            -0.001841330829156697,
+            0.008730275423516445,
+            0.0012974082030866743,
+            -0.014611902493400719,
+            -0.005911807768916134,
+            -0.019130604101278263,
+            0.024719468317569562,
+            0.0209456329723249,
+            0.03381727436368338,
+            -0.013237902591972769,
+            -0.0003695142331921278,
+            0.005410360755058108,
+            0.0002841632253252816,
+            0.0023352645111704958,
+            0.004801123870325242,
+            0.02623649813035263,
+            -0.029010710171522985,
+            -0.020741774066460415,
+            0.0068426980682519505,
+            0.013445270043414486,
+            0.0022744032984779294,
+            0.033840485942696216,
+            0.035040080489222324,
+            0.004166066423508316,
+            -0.010046666684988869,
+            -0.006148765564658713,
+            0.007071740742464309,
+            0.01865897335451031,
+            -0.023267411753900076,
+            -0.0005936657905496225,
+            0.004833030519903764,
+            -0.004229582648167378,
+            0.03620610659071198,
+            -0.009961238880412731,
+            -0.007197670122096852,
+            0.022212791947347127,
+            0.09819205436926473,
+            0.00047695621444943017,
+            0.0025354521225913395,
+            -0.029358210234742176,
+            -0.023614016211567963,
+            -0.0012564198366457442,
+            0.00242680708457405,
+            0.00830167274308547,
+            0.028076260517106013,
+            0.0004852400049405906,
+            0.011579274037419117,
+            0.02773972145320728,
+            -0.017217305534161797,
+            0.006752070301141707,
+            -0.011376051474975524,
+            0.024577071515205502,
+            -0.012694034026935636,
+            -0.0019809719792138673,
+            -0.016512649783050146,
+            0.015907402258634816,
+            0.004556574687833768,
+            0.004907510649166062,
+            0.011347630984408937,
+            0.009824484201337644,
+            -0.02650731900265866,
+            0.01756773113723491,
+            -0.005671986127099093,
+            0.004449611508541289,
+            -0.027841346572384462,
+            -0.020124233970502267,
+            0.0025986391930124847,
+            -0.0014440758308989809,
+            -0.009579444248630085,
+            0.006005274872617889,
+            -0.0024984235450999127,
+            0.03765664613203687,
+            -0.03765496603611941,
+            0.01409350051861866,
+            0.0066453175608372555,
+            -0.0015390841089599882,
+            0.011806653942640485,
+            0.017306914776093522,
+            -0.008206256939290151,
+            0.02589800448884443,
+            -0.00798213351353373,
+            0.009846274212000523,
+            0.015230070803055296,
+            -0.03449531102975745,
+            0.009174739184806312,
+            0.010145762060174144,
+            -0.024165660737765746,
+            0.012146877603301336,
+            -0.013784371016298891,
+            -0.004818759788054113,
+            0.005930446700613062,
+            0.007349837554764034,
+            0.010733731974181452,
+            -0.02295571464086674,
+            0.016176424826629898,
+            -0.010947841926210105,
+            0.0037766665765835445,
+            0.013559846998199008,
+            0.024892849808979323,
+            0.0012426907454129122,
+            0.0005950515136086193,
+            -0.01379469431504576,
+            0.005534724429523084,
+            -0.015667467221535825,
+            -0.022401586867694584,
+            -0.014192041775234079,
+            -0.006366846766730035,
+            0.01029598687444803,
+            -0.003972386708834858,
+            0.016971476088808737,
+            -0.010309648640697812,
+            -0.010645860303528099,
+            0.0037818310327362103,
+            0.01040806357838676,
+            0.008316605962234849,
+            -0.02855914079113881,
+            -0.005676310556150579,
+            0.004961084138163161,
+            0.00029285971503001265,
+            0.009592497954559435,
+            -0.014556073011576487,
+            0.013689526487160009,
+            -0.013290113354287095,
+            -0.0010726537498214612,
+            0.013201860397060236,
+            -0.012443552043930505,
+            -0.004012305744087929,
+            -0.01592905426965635,
+            -0.028487066798154617,
+            0.008462143660968396,
+            -0.01365614159523485,
+            0.001076882584445294,
+            0.0019503240125505646,
+            0.004730085646042709,
+            -0.016919069570642004,
+            -0.007860046623166481,
+            -0.033515448866521046,
+            0.011065416478687504,
+            0.02314589687882036,
+            -0.006435733825418645,
+            -0.030351727858789472,
+            -0.009546333526852862,
+            0.011327489786114963,
+            -0.0071765122743172715,
+            -0.01676736383051139,
+            -0.0029390038145944096,
+            0.007617910650439855,
+            -0.008174620754097493,
+            -0.00508943241317381,
+            -0.0025173969010414098,
+            0.0217328037301181,
+            0.016094975605203832,
+            0.01769642142712967,
+            -0.021400862603034036,
+            -0.00976992221747187,
+            0.026043879119776864,
+            -0.03275550610951898,
+            0.012441465975914529,
+            0.024587446266013183,
+            0.003446527051428522,
+            0.005492228054838714,
+            0.0030377554109693714,
+            0.005459159163772825,
+            -0.006985179226831048,
+            0.019925583798328282,
+            0.010870421035240845,
+            -0.019709062313471467,
+            0.005385443078244229,
+            0.02086085539375208,
+            0.0037248737640193097,
+            0.012892299156840998,
+            -0.017546870090541213,
+            -0.0020341807891543716,
+            0.015594779287017958,
+            0.04816214672670215,
+            -0.0032851797178091685,
+            -0.010450846098750195,
+            0.029513742590677183,
+            0.023526004023637855,
+            0.005373455940309873,
+            0.0012120777808635214,
+            -0.02525630456877349,
+            -0.013075130701774785,
+            0.01962235086017781,
+            0.016944870494393874,
+            0.013484925533584295,
+            -0.0023255656480479004,
+            -0.019236081983457366,
+            0.021520156296467717,
+            -0.01701733319731161,
+            0.012422659010697477,
+            -0.023764941037007197,
+            -0.001490790833263327,
+            -0.0035419008954878724,
+            0.021998949056452454,
+            0.0021819635638621092,
+            0.008891505304514552,
+            0.024354552127458608,
+            -0.006978548952551799,
+            -0.019526615985982355,
+            -0.0051799020400150305,
+            -0.014578953088901798,
+            0.022506635912517146,
+            -0.024117708614271965,
+            -0.007008641019892289,
+            0.011379795968360019,
+            -0.012362093847309038,
+            -0.0102534051092702,
+            0.001164934138876502,
+            0.017686807394319725,
+            0.0009423993435129515,
+            -0.014784508280754078,
+            0.01944445593807992,
+            -0.0011359271135469456,
+            0.010410915597949927,
+            -0.01921419010977189,
+            0.009390638453648025,
+            -0.016682442516992777,
+            -0.007849184048209236,
+            0.010343008873053859,
+            0.0012604521199837046,
+            0.003948300703645506,
+            0.002592490121754231,
+            0.007773437307969315,
+            -0.03355658426684504,
+            0.022695505487314516,
+            -0.010751606023585792,
+            -0.003538523688786008,
+            -0.0014125330167969218,
+            -0.01678692701672328,
+            -0.0006918162525881059,
+            0.005816502852598842,
+            0.010878358093847386,
+            -0.016045147059003256,
+            -0.00873270513444376,
+            -0.04920244395446931,
+            -0.030314009649638693,
+            0.023156333024481057,
+            0.013733657599720879,
+            -0.010971506342695331,
+            0.018852159415605117,
+            -0.023760853159532587,
+            -0.010587249404421732,
+            0.0011379869499042905,
+            0.023319856329056813,
+            -0.03539290744576506,
+            0.01215369703816749,
+            -0.014493682360886549,
+            -0.02039288669534806,
+            0.009070984596830142,
+            0.011445354735283254,
+            0.0007259852045513916,
+            -0.0011514627555568814,
+            -0.029350945123925453,
+            -0.010246845821727087,
+            0.02391068177425878,
+            -0.00861718589055971,
+            -0.004723044986826333,
+            0.024716520302409263,
+            0.0033721801437248873,
+            -0.012524663490691517,
+            -0.017955758068233754,
+            0.021612671655463236,
+            -0.026077203994996058,
+            0.007645944309444457,
+            0.010032183011253845,
+            -0.0173991263874487,
+            0.021649599230588117,
+            -0.007529701797485131,
+            0.020327763440441377,
+            0.004227390938169872,
+            -0.011249233530251522,
+            0.020352283974471443,
+            0.013644443758970054,
+            0.004216958800094986,
+            -0.004081669404784771,
+            0.006869353098014179,
+            0.02212732880780841,
+            -0.011021739614266063,
+            0.02219580717345023,
+            0.028777213295471067,
+            0.00017328191188631918,
+            0.03260179554743322,
+            -0.021287655520443285,
+            -0.03159728983004563,
+            0.021740163175540124,
+            -0.01035146858119169,
+            -0.021564906188712388,
+            0.028914245422221707,
+            -0.013674731836508768,
+            -0.00786476052825481,
+            -0.0166718429993922,
+            0.00533027311342322,
+            0.016680336311088843,
+            -0.005246190451159961,
+            -0.014360934066628332,
+            0.0029628103834216345,
+            0.019809487418628386,
+            0.014587234544007958,
+            0.029780009648869975,
+            -0.010778199815920887,
+            0.035180242185168895,
+            -0.002320576872571577,
+            0.008990157317473909,
+            0.012415465232073404,
+            0.006489435152972723,
+            -0.011251531176923445,
+            0.013501894212117511,
+            -0.003446759011266526,
+            0.008230548711588447,
+            -0.001449023474280409,
+            -0.035111679577202855,
+            -0.013709382199848678,
+            -0.04709294504969336,
+            -0.0023752976043235697,
+            -0.0025098360125255215,
+            0.0030975417740913154,
+            0.09736815869389318,
+            -0.005723049647039619,
+            -0.011988817571623457,
+            0.001625059666070763,
+            0.009611697638805556,
+            -0.0009896494420056814,
+            -0.007606836952841873,
+            0.015973152482488318,
+            -0.01343292635775299,
+            -0.02140928016038583,
+            -0.010532701190806147,
+            -0.01288594454368977,
+            0.012079198190522715,
+            0.015929716343161652,
+            -0.0005947359502228821,
+            -0.015207331663761012,
+            0.006524010655124294,
+            -0.004566385899752684,
+            0.0064717704997400245,
+            0.008647099975176308,
+            -0.010414668624343924,
+            -0.0071698879102925555,
+            0.030899143952119196,
+            -0.025301269549419674,
+            -0.024361748354052457,
+            -0.0010579931222861095,
+            -0.009055796737486008,
+            -0.010335459737012047,
+            -0.002692653633719985,
+            -0.0017679501322462056,
+            -0.010519994985692081,
+            -0.010019865263564572,
+            0.006537076627706128,
+            -0.015472130358273033,
+            0.004812857776601667,
+            -0.0047457011269495634,
+            -0.013296069496526007,
+            0.017054968086538724,
+            0.022563785357424853,
+            -0.02841491200807702,
+            0.0006755428946168761,
+            -0.017422510783648978,
+            0.012941211381006863,
+            0.008866069010589792,
+            0.006049618782357251,
+            -0.022451528209097505,
+            -0.023622906582403243,
+            -0.015019272754405791,
+            -0.01927535750568903,
+            -0.024973490129133732,
+            -0.009073381957684092,
+            0.0067692803507600945,
+            -0.001267339788153158,
+            0.021918281111033645,
+            0.000816538714279555,
+            -0.01812545613159305,
+            0.0061084913570894905,
+            0.000568371220952269,
+            0.018292497018668842,
+            -0.027468677288882186,
+            0.0033806422195878524,
+            -0.02307582511848465,
+            0.033508234091571124,
+            0.0039195050746594975,
+            -0.026680111514794824,
+            -0.021558668725755275,
+            -0.003311830287529699,
+            0.018994577319066473,
+            -0.0025745912019675805,
+            -0.00724049447539589,
+            0.014775207617625293,
+            0.0155334760323838,
+            -0.004693118176196483,
+            0.002015625406296403,
+            0.027968079002437102,
+            0.01077998866180723,
+            0.0313892866933904,
+            -0.015236169297590675,
+            0.004945828495617267,
+            -0.027215016713379062,
+            0.010214419950212732,
+            -0.01503548137805401,
+            0.008859554812646242,
+            -0.01290890682557532,
+            0.03531384210182567,
+            -0.012005613861257432,
+            -0.004250698950192301,
+            0.03253288796441852,
+            -0.029150986393345252,
+            -0.0035020051536906334,
+            -0.009255437891304041,
+            -0.004142204892160768,
+            0.013240580426083011,
+            -0.001198093874819309,
+            0.008485141356876356,
+            -0.01854370462655459,
+            9.726096864073609e-05,
+            -0.003221852304410318,
+            0.005404595442127598,
+            -0.02005458688172967,
+            -0.003314118809778734,
+            0.0073427568419969814,
+            0.004259518237708634,
+            0.002408630356035373,
+            0.016420646142725346,
+            0.006662121128241274,
+            -0.0029357701493526907,
+            0.03424224243607692,
+            0.008171895850772245,
+            0.0152521424877615,
+            -0.0018638264193202728,
+            0.0030234211261443034,
+            0.019287963516760636,
+            0.019912956661192856,
+            -0.00048443399729935116,
+            -0.01058935153883094,
+            0.00195113175754497,
+            -0.001140732137433033,
+            0.011048741602327437,
+            0.024396046054912645,
+            -0.021158025221644738,
+            -0.02419562092713724,
+            0.016377661005279716,
+            -0.015076145951740961,
+            0.01903712233378994,
+            0.010541463112889985,
+            -0.010099351302095792,
+            -0.00031558616108876247,
+            -0.042316016201379245,
+            0.010964989031040692,
+            -0.02068939391415103,
+            -0.04004120028915868,
+            -0.00244383573783919,
+            0.0009542507389088937,
+            -0.029439456243505052,
+            0.02822814855547978,
+            0.0023755032182188156,
+            -0.017280221740813364,
+            0.018223709137876153,
+            -0.005548654680726505,
+            -0.0024040301239229064,
+            0.005397711787238679,
+            -0.01092026090471121,
+            -0.010648485086769927,
+            0.008235250252660972,
+            0.010957715075632535,
+            0.005425683656535705,
+            -0.005698919020608095,
+            -0.011284911899894062,
+            0.009975882651638084,
+            0.012318901214538077,
+            -0.0019922884373083155,
+            0.010050520101815593,
+            0.0031720752428247446,
+            0.007352859750323051,
+            -0.0073483049386141005,
+            0.0046483587496626965,
+            -0.005093220128411125,
+            -0.026080057085901446,
+            0.012863580653903103,
+            0.0052434400313520774,
+            -0.011863360599637747,
+            0.0103209502958491,
+            -0.01048162756158923,
+            0.005787157766911021,
+            0.001318945460232246,
+            -0.0059333230422061395,
+            0.004821320238893602,
+            -0.0038922239931512784,
+            0.011267634571983924,
+            0.013256611037442042,
+            0.004592529900805719,
+            -0.013076954459595086,
+            -0.02177318635989856,
+            0.009001291224623467,
+            0.01959767163356408,
+            0.024161754192119852,
+            -0.030416109293488064,
+            0.003784634588325551,
+            -0.0014915849060829743,
+            -0.0020550564975080885,
+            -0.015632785947151753,
+            0.01587790981601866,
+            0.01716742056401084,
+            0.012721563202528083,
+            0.01893393006967701,
+            -0.004009617703771589,
+            0.010539100665843704,
+            -0.02080068888791658,
+            0.013642451766699067,
+            0.0008609869904066794,
+            -0.013629067231663643,
+            0.0014620655123815298,
+            0.027046304278766087,
+            -0.011056783825898616,
+            0.006206969699250696,
+            0.008097527175511791,
+            0.022111870736907183,
+            -0.012509479100313492,
+            -0.003735868296238069,
+            0.01516254968167132,
+            0.0014878674796313539,
+            0.0035887079417890352,
+            0.016533149527222273,
+            0.04729052276110659,
+            0.00102751186013391,
+            0.0016713773449738153,
+            0.0192177920520243,
+            -0.005563519063161466,
+            -0.01114821419109643,
+            -0.003549999863325982,
+            -0.021807893906469106,
+            0.010998558936481785,
+            0.00431279785115251,
+            -0.02186324212032519,
+            0.01024093366826206,
+            0.0125495473550584,
+            0.005268320135120931,
+            -0.0031679212410026603,
+            0.02574575247827441,
+            -0.019807879421996258,
+            0.014935295894965128,
+            0.03329450628384787,
+            0.00493356778223407,
+            -0.011423949333937285,
+            0.007263264249871384,
+            0.0027354459678742013,
+            -0.004494406041906684,
+            -0.00606745413660583,
+            0.010497391290734088,
+            -0.008191702034391382,
+            0.009870458191510176,
+            -0.009580130479693,
+            0.039202091799313755,
+            -0.009619018331262844,
+            -0.010782138248435458,
+            -0.004558645539551881,
+            -0.006241773108460662,
+            0.003478414182938609,
+            -0.003395165694642073,
+            0.005812225135633631,
+            -0.0069045407353789945,
+            -0.014607146220271209,
+            0.0008522982757210454,
+            0.00020175973815948303,
+            0.014001922919116725,
+            0.0011804904415274088,
+            -0.004871874532755353,
+            -0.007877217154706205,
+            0.015243277056007988,
+            -0.006047906208304912,
+            0.0006531937033317192,
+            -0.0011653523616313422,
+            -0.0013662556399359335,
+            -0.007401050012179761,
+            -0.015166105795085528,
+            0.00974609244984173,
+            0.016253733908242134,
+            -0.013857834499211056,
+            -0.005142936208363415,
+            0.012274716212832219,
+            0.008540229125193836,
+            0.011742606168187876,
+            0.0007451338650642946,
+            -0.009255813682572937,
+            0.008292531177374192,
+            -0.015020523035432158,
+            -0.008781317965985481,
+            0.0026250469418136603,
+            0.002381147382977664,
+            0.005839723981046825,
+            0.0006240641828668715,
+            0.004502463508605272,
+            0.005798912705218846,
+            -0.00497936644919367,
+            0.01112666823857836,
+            0.008198424171006548,
+            -0.002452143482105151,
+            -0.01017569410837814,
+            -0.001127589984272638,
+            -0.015733298167661346,
+            -0.016758315411028353,
+            -0.0009286266498936771,
+            0.015175988113750882,
+            -0.02352862898718829,
+            0.018996352975391644,
+            -0.0014524363611708201,
+            -0.0033802546171857704,
+            -0.021392428456030312,
+            -0.0025923983911570183,
+            0.0034214188668518796,
+            0.01255768230394476,
+            0.01858853630745928,
+            -0.02729543519465384,
+            0.010575894388563116,
+            -0.007204725245442141,
+            0.030394929424014717,
+            -0.018328782140005044,
+            -0.004022941273610227,
+            0.026915618079621992,
+            -0.02200018306828758,
+            0.0008791185377513978,
+            -0.011454606955158956,
+            -0.008103032693853504,
+            0.019287674837688177,
+            0.023747720480736777,
+            0.02038287242167783,
+            0.01364489294869211,
+            0.0007758044919789085,
+            0.004768030222445397,
+            0.03572705648994403,
+            0.010568241270862248,
+            0.014828087092859878,
+            -0.007542489523110974,
+            9.457808640264811e-05,
+            -0.02554033312739386,
+            0.01635899434369978,
+            2.587383534566272e-05,
+            -0.012370727506722627,
+            0.007162422186621379,
+            -0.013370882403931942,
+            -0.011066483600541058,
+            -0.007151077450730025,
+            -0.01721910018182534,
+            0.02152239849400241,
+            -0.012322440280680164,
+            0.004587729883596212,
+            -0.009079747696669277,
+            -0.013272707286450417,
+            -0.012953682633400515,
+            -0.014202214807187545,
+            -0.01313141988034687,
+            -0.026041403779538307,
+            -0.027109731260915763,
+            -0.006573285295895077,
+            -0.023780366346334524,
+            -0.013799072648505526,
+            0.005588622594136253,
+            -0.022617902666316757,
+            0.012262325666428604,
+            -0.012689827812482627,
+            -0.01226264872059864,
+            0.012805734807835008,
+            0.020274844315616616,
+            -0.011765231803328348,
+            -0.028494489703400308,
+            -0.018583843821043854,
+            -0.019970998470704358,
+            0.012576201137505969,
+            0.002266288538944273,
+            0.013165889075444961,
+            -0.003018437854660793,
+            0.003227412667880239,
+            -0.014895831590298957,
+            0.018570624675927183,
+            -0.03442390703780963,
+            -0.03108003806963536,
+            -0.013611306765348361,
+            0.005182777066476028,
+            -0.0015782033203578553,
+            -0.011429046504256945,
+            -0.0018606648572920801,
+            -0.00433805271182878,
+            -0.025109572024446088,
+            -0.010134856632190067,
+            0.011010662054484161,
+            -0.015205629927469414,
+            0.0072933565255675036,
+            0.0009018832564038863,
+            0.013870902834260887,
+            0.009587432944616442,
+            0.013393301893492882,
+            -0.006095063698384775,
+            0.004356485794113208,
+            -0.0026476612747324386,
+            0.0004894024149759108,
+            0.009336366919474478,
+            0.00839591561027103,
+            0.010551161369079399,
+            0.002260707777260338,
+            0.007830059223001696,
+            -0.0060927446216633265,
+            0.018065002687169148,
+            0.015436944875538481,
+            -0.0002520689442496089,
+            0.013323098485251487,
+            0.0003769862591044255,
+            0.00022293304921127018,
+            -0.0029970303106340466,
+            0.007888437018607987,
+            -0.011785957933642163,
+            -0.001311704406355858,
+            0.015138190893483804,
+            -0.01707247730569978,
+            -0.013045187198628258,
+            0.03107034781283638,
+            0.0018583638488268697,
+            -0.02495564203304796,
+            0.031146554085787762,
+            0.004220047610942943,
+            -0.005835199594223787,
+            -0.013205086624848782,
+            0.0014059047822222418,
+            -0.011570624844100298,
+            0.003584232809362274,
+            0.014295032061446826,
+            -0.02231984223610487,
+            0.010521094801839404,
+            0.008446892441796151,
+            -0.009681535919980559,
+            -0.012337659795586223,
+            0.009847073709209602,
+            0.016809328445200358,
+            0.004472485504034416,
+            -0.03539093810210523,
+            -0.011957694102753411,
+            0.02145470732008716,
+            -0.029237428225009913,
+            -0.00810041956259673,
+            0.007452962400105787,
+            0.0065154832040426065,
+            -0.019353109017609053,
+            -0.008580955020412362,
+            0.005725794605897509,
+            -0.004940390374637084,
+            -0.003892210172017758,
+            0.017541623128534682,
+            0.00485047043626186,
+            0.013698341792101025,
+            -0.01689738578307474,
+            0.009771458057400557,
+            -0.022222817828602557,
+            0.004021882074676753,
+            -0.023373700500461992,
+            -0.009873403167702603,
+            0.002364143609079849,
+            0.012454271200139762,
+            0.014328556467632212,
+            0.027839363143659326,
+            0.030056404238348527,
+            0.013995696971422429,
+            0.007833677585250953,
+            -0.016265199386953717,
+            -0.018134327237616175,
+            -0.006409513840075933,
+            -0.004033173350204784,
+            -0.008714798059218596,
+            0.0066373547199279235,
+            0.01858354212009877,
+            0.004499712802472377,
+            0.02609078206670711,
+            -0.006227189596272876,
+            0.009039744622014585,
+            -3.127526911951914e-05,
+            0.0008833930131827993,
+            -0.004226083827636182,
+            -0.01971685546584527,
+            0.00481003211084742,
+            -0.0013721482162825508,
+            0.025874758880591167,
+            0.00940496860916206,
+            0.0024915013399541847,
+            0.01615978314811047,
+            -0.006378952757397484,
+            -0.01174653607178726,
+            -0.010540107733238588,
+            -0.0061926268398085176,
+            -0.021752582127981104,
+            0.029059100503124387,
+            0.021195595568253612,
+            -0.010800765264123782,
+            -0.013693582166598082,
+            -0.00903369526059487,
+            0.02795767990556847,
+            0.004080965688836953,
+            -0.0013050212744006933,
+            0.016313311850308505,
+            0.012084555958999215,
+            0.012329325712491252,
+            0.009794698207689061,
+            -0.00036157148861933123,
+            0.0015364709122308352,
+            0.03722522141440073,
+            -0.007427028510548186,
+            0.0026463292682538495,
+            -0.005239324411651515,
+            0.0025987932868461313,
+            -0.009705817520422487,
+            0.007242122655628283,
+            -0.004012676385296197,
+            -0.0158760051470016,
+            0.015003444174308607,
+            -0.022443331305169994,
+            -0.0016749787204471238,
+            -0.008150635190952799,
+            0.0016696640640947303,
+            -0.02781023337840043,
+            -0.0025731206607371845,
+            0.006695065701630532,
+            -0.015547340828489069,
+            0.011816010501093885,
+            -0.01740673229051557,
+            -0.011553604323149105,
+            0.007914961618894248,
+            -0.012034157288065447,
+            0.0016684130271324446,
+            -0.0008268957621923278,
+            0.004460864332388434,
+            0.013426055079012884,
+            -0.011008808570221245,
+            0.012166046631826273,
+            -0.005816284310138776,
+            -0.007872091214834688,
+            -0.0016055129487028885,
+            -0.0030049472973942043,
+            -0.0025600746761171543,
+            -0.0009966231170649917,
+            0.034355910234323155,
+            -0.004446570896048543,
+            0.003761242147499512,
+            -0.01362639743248326,
+            0.0019833052855181806,
+            -0.036176376394999044,
+            0.028344464817309477,
+            -0.008642835889481802,
+            0.011013502221008498,
+            -0.0017508365480865596,
+            -0.013243706268954915,
+            -0.01456517309747086,
+            0.015247299501245178,
+            0.007602596572394442,
+            -0.005854531057349665,
+            0.013558248087347848,
+            -0.019724373614448298,
+            0.008530500822910953,
+            0.004463694543554244,
+            -0.015561397976987216,
+            0.004668780190328835,
+            -0.007479912658259757,
+            0.012657201279639194,
+            -0.01774730033362932,
+            0.004314594593150765,
+            0.00958571617187387,
+            -0.01892876084898881,
+            0.009901041800918215,
+            -0.028675879564512306,
+            -0.004631920242208613,
+            -0.012323455524366223,
+            0.0053804608827285865,
+            -0.000660864620708182,
+            -0.02389088252276507,
+            -0.028944280660230762,
+            -0.0002099667627277945,
+            0.00458533068247518,
+            0.0018878351997997833,
+            0.011621407953503755,
+            0.005970807015151215,
+            0.008542958629199003,
+            0.01023896054756221,
+            -0.00205207247588841,
+            0.009402752124876478,
+            0.04169706217414899,
+            -0.022899912321317197,
+            -0.011574970753549647,
+            0.005321872977369877,
+            0.011499271505460519,
+            -0.02647653483798284,
+            -0.011039791758347888,
+            0.0033424345263339144,
+            -0.005219877979550397,
+            0.03609469691834737,
+            -0.02448746405881972,
+            0.019020618652020673,
+            -0.011392297116547224,
+            0.007625475937968714,
+            -0.021465240076078376,
+            -0.003921088264697311,
+            -0.014265745991522477,
+            -0.00011221615798127438,
+            -0.0031478236827989703,
+            -0.0076053733863387755,
+            0.007485039021296698,
+            -0.001907678661303243,
+            0.0020348246022161925,
+            0.006291436821831794,
+            -0.02173305455485444,
+            -0.0019125499969493854,
+            -0.016432460924913166,
+            0.030367263349759286,
+            -0.016242095671439496,
+            0.005072805486919414,
+            0.0053258139728101506,
+            -0.002810639419737258,
+            0.012368771814660942,
+            0.004165967191536186,
+            0.00513754440091394,
+            -0.004199767241827414,
+            -0.005142642360546972,
+            -0.0053886448467479215,
+            0.017868874549760376,
+            -0.013280048598849825,
+            0.035024701466710065,
+            -0.0100959186045929,
+            -0.007094001877261568,
+            0.0019733042325232155,
+            0.010380430710282744,
+            4.2084202709296443e-05,
+            0.01053565670076933,
+            0.0034270489770133685,
+            -0.0006311078385359693,
+            0.0019774779328212756,
+            -0.017162448473652843,
+            0.003089758682621882,
+            0.01682656561693936,
+            0.005463677441664502,
+            0.004469261499451007,
+            -0.013332130891003062,
+            -0.012050233723282738,
+            0.004538268704442613,
+            -0.007533940408255447,
+            0.0003675737140813486,
+            -0.018816953644773076,
+            0.013972696748575243,
+            -0.0011096683686631858,
+            0.0002715683052236706,
+            0.00524678369365015,
+            0.023056514997908218,
+            0.01813167860994747,
+            -0.002553352602551003,
+            0.03155164978556312,
+            0.0174738879054593,
+            0.0029268039949260673,
+            0.010637295711777793,
+            -0.033222781195984666,
+            -0.013301873512972214,
+            -0.0013721703891642354,
+            0.02238421933149517,
+            -0.01530189703335162,
+            -0.014059029663525449,
+            -0.0037393534403749947,
+            0.0010394601933309767,
+            -0.005177359933828505,
+            0.0042489306011644495,
+            0.011039889511148341,
+            -0.014697595240068863,
+            0.007597149702441915,
+            0.0026780496822907634,
+            0.02092734730183164,
+            0.0004688144432243786,
+            0.017181410417231698,
+            -0.0005083970646221274,
+            0.00749639055342427,
+            0.007319420826337891,
+            0.017541463125355265,
+            -0.01002811389296805,
+            0.023582113845081534,
+            -0.008823933397729605,
+            0.0019889258902730535,
+            -0.01257106158422629,
+            -0.028159594067544737,
+            -0.012624750220149823,
+            0.015247564545279736,
+            0.005765911899842925,
+            0.006478299226263732,
+            -0.016537090297144265,
+            0.006276997440048981,
+            0.015647122171767974,
+            -0.002251683151059454,
+            0.010295358869874306,
+            0.009730378299068289,
+            0.00012816405890877098,
+            0.010181120688239766,
+            -0.00801256937783534,
+            -0.00734257767723293,
+            -0.004497520124843868,
+            0.023982754195729983,
+            -0.005443550092842393,
+            0.022434177122972028,
+            0.012647958478747348,
+            0.00616791857573897,
+            -0.010853355357954727,
+            -0.003340111254357573,
+            0.023389634440057473,
+            0.011863246790550376,
+            0.00029642699874398923,
+            -0.009846034060495678,
+            0.0019458876579524617,
+            0.0054010373765486005,
+            0.012983913038716568,
+            -0.02049427270240938,
+            -0.004344571352918878,
+            0.008461947538684664,
+            -0.008472919299762712,
+            0.004318309005722874,
+            -0.030156444522730416,
+            -0.005812303237222271,
+            0.021354352236942434,
+            0.008392281015122372,
+            -0.010232039722494646,
+            -0.004395682831498155,
+            -0.0006527587802367882,
+            0.0040601850269714945,
+            -0.0024323167053212176,
+            -0.0023729138545912176,
+            -0.008541233856441836,
+            -0.006470933419694742,
+            0.014035696617797647,
+            -0.0052691760528871924,
+            0.008597459268477037,
+            0.00012084859681905111,
+            -0.0017255958688825231,
+            -0.007323529232751411,
+            0.03382792341957058,
+            0.008087732655845718,
+            0.005980666974986827,
+            0.0013613625176327891,
+            0.0042244330945601215,
+            0.000616791811672998,
+            -0.0028374856646453707,
+            0.018075360591135542,
+            0.0156738939725117,
+            0.005054696911814988,
+            0.009038525980341447,
+            0.014672270369917403,
+            -0.019700405219230957,
+            0.012625337070556061,
+            0.012184185186281482,
+            0.019382321278525114,
+            0.0007219498737851579,
+            0.002564970959687783,
+            -0.010601873823737926,
+            -0.0029963537053764736,
+            -0.002070567089566093,
+            0.029563028293936092,
+            0.0005903976560069006,
+            -0.010509758568838935,
+            0.010992421945232951,
+            0.020603868018231348,
+            -0.02101157589451638,
+            -0.001971974816402967,
+            0.005963911670920596,
+            -0.010704207628592966,
+            -0.014466113540150018,
+            -0.021269176952661614,
+            -0.013331247081394942,
+            0.0071597345125287174,
+            -0.008162102923896894,
+            0.02367549407145414,
+            -0.014256771777502395,
+            -0.03960796112042958,
+            -0.014803626428839152,
+            -0.00864320347698291,
+            -0.027531337207277483,
+            -0.007305441146142331,
+            0.00965056093889974,
+            0.016698589600806273,
+            -0.003330867073501157,
+            -0.025198703196464594,
+            0.030483790406960656,
+            0.008872821868559751,
+            0.023708422767352917,
+            -0.021464374508407488,
+            -0.0018617153194555344,
+            -0.00886376784514331,
+            0.014505918224188001,
+            -0.017588761818663975,
+            -0.011756112651591795,
+            0.020014632992762026,
+            0.014356419813769714,
+            0.007246557500044434,
+            -0.004355132555323843,
+            0.0067299398462013936,
+            0.007649499235136303,
+            -0.012515181210363267,
+            0.012239792251154642,
+            -0.013499277313916029,
+            -0.024858295126228433,
+            0.014430561986516884,
+            0.002229494935179828,
+            0.009563004428064055,
+            -0.0015984171085659323,
+            -0.016197659608768088,
+            0.010078159490026602,
+            0.00808729175515505,
+            -0.017508840440512225,
+            -0.004793934415922065,
+            0.007133527615305415,
+            -0.020818869260580315,
+            0.031234686259160864,
+            0.011872765152279094,
+            -0.010692949770503467,
+            -0.001989039629661815,
+            0.004431079028880923,
+            0.016511544608975148,
+            0.007893700385963683,
+            -0.01997314388229276,
+            -0.003708006454566117,
+            0.01530128306203401,
+            0.00822588305477249,
+            0.004861703074728471,
+            -0.010836328783594584,
+            0.03434874763691911,
+            0.006082873924472328,
+            4.192481378506604e-05,
+            -0.0052997095110967115,
+            -0.0076108924981838355,
+            0.004196112457289409,
+            0.018905391806961686,
+            0.013330078288599763,
+            -0.004200590783114919,
+            -0.006630107527854746,
+            0.007778448128358027,
+            -0.01706991160178345,
+            0.007329056462125937,
+            0.009390327499494826,
+            -0.012870674959022891,
+            -0.0017571189976160295,
+            0.016462608870101713,
+            0.00209846661139665,
+            0.004942714850185376,
+            -0.011305050739130339,
+            0.004252776146827917,
+            0.015841821944109317,
+            0.018727567181082646,
+            -0.008399642221255733,
+            0.021448055799259867,
+            -0.020972459569228434,
+            0.027243639465535856,
+            0.028499400850980086,
+            0.016573144403275928,
+            -0.01847096443459798,
+            -0.0022409373140116146,
+            -0.0041737366139895414,
+            -0.014761716560267328,
+            0.0034546020161337894,
+            0.005125870074458691,
+            -0.01174186608185822,
+            -0.009577921232380895,
+            -0.024331993794416026,
+            0.011075959943215117,
+            0.007918147902741738,
+            -0.0015045185909419564,
+            -0.011791155919533844,
+            0.016019077577552835,
+            0.010141868950981925,
+            -0.010938402325619172,
+            -0.005635351352447256,
+            -0.019967361901734836,
+            0.009750427567651833,
+            0.009128294017175602,
+            -0.00300556760891084,
+            -0.011496103417643132,
+            0.012977890524465716,
+            0.0014080059843484115,
+            -0.01292380658817521,
+            0.010841954636959441,
+            -0.03142011651894961,
+            -0.005510537354374977,
+            -0.011027026734432255,
+            -0.003620482919145167,
+            -0.014415606143566519,
+            0.003608572766363589,
+            0.0005034611321129689,
+            -0.011470488351158329,
+            -0.0035005774696278,
+            -0.016207453351921627,
+            0.004676327282262429,
+            0.015021513761697039,
+            0.001679614645967629,
+            -0.006917375094634678,
+            0.021994274901326064,
+            0.018518524483306403,
+            -0.021903538417919984,
+            -0.020116013379669034,
+            0.004708084983787958,
+            -0.03261690161800251,
+            0.015019348187107588,
+            0.00966124471577622,
+            0.011676937880751252,
+            -0.028768229226408037,
+            0.007114534979415504,
+            0.00900420165933673,
+            0.016398927481511824,
+            0.0050879301160439245,
+            0.009453255486450082,
+            0.01222758288108498,
+            -0.0104657614414275,
+            -0.003164502778525433,
+            -5.410382753651127e-06,
+            -0.005618225986727351,
+            0.015607620507415484,
+            0.0008527697935056332,
+            0.03603666488542239,
+            -0.005277207843139309,
+            -0.0009799879252482288,
+            -0.011735731235560851,
+            0.00812046801749312,
+            -0.011713349784345638,
+            0.006334172347602515,
+            -0.0024643499727789167,
+            0.0001240009282616789,
+            0.002965067818077541,
+            0.009442572075213767,
+            -0.01086095783353607,
+            -0.007595379345224076,
+            0.007444897256325813,
+            -0.01482814340478338,
+            -0.00019984717819268013,
+            -0.009339343730980451,
+            0.0019680804106275156,
+            -0.02241031895876346,
+            0.006629838013824249,
+            0.040935644507527846,
+            0.02140216465404127,
+            -0.005481040712269146,
+            -0.012431659209559418,
+            -0.004680230194268222,
+            -0.007229300034633229,
+            -0.006482401275991039,
+            -0.004794698160419071,
+            -0.022762405726589375,
+            -0.006928553505112732,
+            0.004320321244630314,
+            0.013300138724212552,
+            0.009741018117644266,
+            -0.008842135469729511,
+            0.004531760574809748,
+            -0.02662650543210291,
+            -0.016578081411548973,
+            -0.0060021425889846495,
+            -0.015840130031005147,
+            0.006125301421471048,
+            0.027521841635711787,
+            -0.0063432924325983525,
+            0.01789356321527744,
+            0.005142029893259461,
+            0.013034833065163448,
+            -0.005338869351330258,
+            0.00035250243782976347,
+            -0.006010938630154189,
+            0.0032958656042048292,
+            0.003074861351445954,
+            -0.0015947217852680714,
+            0.004151590621941346,
+            -0.007139285822575166,
+            -0.0026091937868967284,
+            0.0011629380311555433,
+            0.0117841884590861,
+            0.006877419488576261,
+            -0.00016252653621991807,
+            -0.01761285866765538,
+            0.022221104524808997,
+            -0.018933508559881868,
+            -0.008840161102111973,
+            0.011092684029552064,
+            -0.009660487856185668,
+            0.00702597023411946,
+            0.020387386977378873,
+            -0.006854213562767843,
+            -0.009583981919303215,
+            0.013028181087649146,
+            -0.0036507672754997286,
+            0.004464368543493108,
+            0.009867428978560914,
+            0.006036303973414791,
+            0.002482944923100156,
+            0.013244650094892797,
+            0.004530294330688855,
+            -0.0032625482538717137,
+            -0.020238533752799943,
+            -0.005186498720507101,
+            0.019015752262933585,
+            -0.02252458872825473,
+            -0.0027641215988942563,
+            0.0044506144202026456,
+            -0.0030317840657635457,
+            -0.01733962766050954,
+            0.004205861489436576,
+            -0.018770275720867473,
+            -0.0033883519277721433,
+            -0.008244668812709337,
+            0.009948090119054079,
+            -0.023700732627507225,
+            0.009109315916173436,
+            -0.011739192106106307,
+            0.02356509175177699,
+            -0.003925851077921686,
+            0.024734433836185024,
+            0.0015142645677087268,
+            -0.013482933030628261,
+            0.0055323884713736554,
+            -0.0008930661843169134,
+            -0.004477731568313757,
+            0.029338490404300612,
+            -0.005418258701133154,
+            0.002249996261871466,
+            0.013491124734151694,
+            -0.021332541202417246,
+            0.021773687217263173,
+            0.0046662696076723836,
+            0.01628697326539432,
+            -0.00986238951830334,
+            -0.013799209313714441,
+            0.025178874553620743,
+            0.03324555035648896,
+            0.008398796308070151,
+            0.011261509865513953,
+            -0.01226890254177634,
+            0.018966175575083295,
+            -0.0038328997984309597,
+            0.004344837325292706,
+            0.00878668880458693,
+            -0.007200842513742962,
+            -0.011500562922303248,
+            -0.004533014670236755,
+            0.027639894892383943,
+            -0.021838848944317424,
+            -0.008920629621611768,
+            -0.005739181836252042,
+            -0.01593272996094732,
+            -0.038167534267554606,
+            -0.004371701247332816,
+            0.019604405563120094,
+            -0.005974358141737469,
+            -0.0018341151594086288,
+            0.014334266690716505,
+            0.00402620072221496,
+            -0.002852569009185304,
+            -0.008195218707838913,
+            -0.0027944153624309856,
+            -0.0059390492900516225,
+            0.016651206007022145,
+            -0.004961864528562493,
+            -0.007929830585906699,
+            0.01145431100211762,
+            -0.005731957927601062,
+            0.012130694936714224,
+            0.0004848418056781251,
+            8.392048424584131e-05,
+            -0.0015174120840537315,
+            0.00822932723013239,
+            0.018620869354789082,
+            -0.0055803454666417465,
+            0.025682778991091174,
+            0.006956936033581638,
+            -0.003495709501259274,
+            -0.005489431577052735,
+            -0.0044154293119170445,
+            -0.024561016942073285,
+            -0.0020864636079347357,
+            0.006593060072668424,
+            0.024333813203596122,
+            -0.02515989187381735,
+            -0.004953266000463339,
+            0.020957806164031402,
+            -0.0012192724005613,
+            -0.011978722479596745,
+            0.014690559986694138,
+            0.0041346025614047875,
+            0.009881584250499358,
+            0.012176393203318002,
+            0.015764530703333238,
+            -0.004923548157246672,
+            -0.0015192775000207833,
+            0.012766482928380375,
+            0.0005745973040874939,
+            0.00997809510696241,
+            0.016495444272181515,
+            0.010467514633877887,
+            -0.021108281220544853,
+            0.0015078843789125156,
+            0.00522593579521476,
+            -0.017481430635971396,
+            0.0034362821987740755,
+            0.005397819166037461,
+            0.011626527721895694,
+            0.004048632382026344,
+            0.013207335578216437,
+            -0.015768003434275563,
+            -0.005916168437120099,
+            0.008153798523776654,
+            0.004319451296835019,
+            -0.02599474886669325,
+            -0.01485009085106089,
+            -0.0006781964116146299,
+            -0.004444144921332929,
+            0.013556265879295017,
+            0.01285768162554067,
+            0.02033626001285354,
+            0.024637349990579514,
+            -0.013833060730799217,
+            -0.00842819657792762,
+            0.009436072455719896,
+            0.015560357465711765,
+            0.005798181411384387,
+            0.007874051721372273,
+            0.009297440071118126,
+            0.0077320823375042,
+            -0.004073209576661068,
+            -0.002973928843567532,
+            0.020205003097225747,
+            -0.019725382393637726,
+            0.0006746659838642147,
+            -0.018417443693472958,
+            0.007530796896254817,
+            0.01140696723506382,
+            0.01129515042745398,
+            0.002816883643872417,
+            -0.005274757308627299,
+            0.005716917047812153,
+            -0.06254973656832136,
+            -0.009191453047502128,
+            0.003384751551032943,
+            0.027256466708746898,
+            -0.010885336140129142,
+            -0.003415707910562931,
+            0.007309577090125959,
+            0.005511970708769456,
+            0.0005572239548625122,
+            0.003186427278034852,
+            0.019698716902282903,
+            -0.02052581021967601,
+            -0.010793968505805883,
+            0.004219269922138548,
+            0.0033111837923746746,
+            -0.007269521534951044,
+            0.004805737092302134,
+            0.014143978208501908,
+            0.01986656063456352,
+            -0.0024048854610842748,
+            0.01629335842915328,
+            -0.0015498603704589015,
+            -0.014496994444493684,
+            -0.002671064325909516,
+            0.0037875833047555925,
+            -0.022466433279066646,
+            -0.005980897597195858,
+            0.0043262697442044435,
+            -0.011226121648007667,
+            0.006978495397818936,
+            0.008452177557316863,
+            -0.003018545465557218,
+            -0.0008303307644324117,
+            0.017796789945838845,
+            0.006591364879905198,
+            -0.013464486184025026,
+            0.0028668426952326957,
+            -0.023377605222213174,
+            0.004325185794796361,
+            0.001416245546006914,
+            0.007809449561523174,
+            -0.008431335139468173,
+            -0.014774499666285603,
+            -0.005933810712528226,
+            0.017746847020610673,
+            -0.01965188730040385,
+            -0.002406875192894708,
+            0.01683009662984443,
+            -0.0002518220110313279,
+            0.005456812494894182,
+            -0.02011241622618092,
+            -0.0009782974741561381,
+            -0.02933296756078644,
+            -0.01171823509606186,
+            0.001162345317872531,
+            0.00892221099991475,
+            -0.006551308022163148,
+            -0.03934356275100932,
+            0.00427623835776202,
+            -0.013111664387776261,
+            -0.008752857644809231,
+            -0.010214546496134218,
+            -0.00518087722852466,
+            0.015240297918667867,
+            -0.012498100033235827,
+            0.01086490299272505,
+            -0.00917777212602538,
+            0.005895055740265625,
+            0.0090554651297836,
+            0.014212841798768113,
+            -0.005544047915369086,
+            -0.0033589999440643385,
+            -0.007924830667307956,
+            0.0022291797354234005,
+            -0.0032976226056924502,
+            0.009615674261248298,
+            -0.0051932155872883,
+            -0.0065636389655770455,
+            0.02426106498596339,
+            0.011445591067574932,
+            0.013312596099532533,
+            0.0055118624272110965,
+            -0.007220620072349783,
+            -0.01916551187613823,
+            -0.003977616240384473,
+            -0.013204506708246246,
+            -0.006315966975649096,
+            0.01434099854765068,
+            0.012588249778961223,
+            0.006357972880218866,
+            0.0007747002406996095,
+            -0.004794439624135275,
+            0.006975593520375694,
+            0.0262335496080837,
+            -0.016481909971520166,
+            -0.0013084662210086592,
+            -0.01369313900978409,
+            0.01921748474212319,
+            0.03333761672336437,
+            -0.0008918419419301244,
+            -0.007699525186523558,
+            0.013757216065348164,
+            0.00819045303047094,
+            -0.007411380297164911,
+            0.02453611080891859,
+            -0.025045462880463647,
+            -0.015989319900023054,
+            -0.011650886878708265,
+            -0.011407253985176972,
+            -0.008386367959227559,
+            0.016331961716602287,
+            -0.00141225438918211,
+            -0.0018505754298268654,
+            0.009557291617446645,
+            0.03357924917013024,
+            0.006916860901290756,
+            0.007030308953738567,
+            -0.002739345982346728,
+            -0.009376522438071562,
+            0.011482637968411055,
+            -0.007259054391879614,
+            -0.04075661435928125,
+            0.004250195164583401,
+            -0.005160505473755424,
+            0.02405623169060588,
+            -0.02197586561375289,
+            -0.00619175761282745,
+            -0.006736129049831973,
+            -0.028200015908139763,
+            -0.005867722104196303,
+            0.02805071895018145,
+            -0.015402198567775478,
+            0.014773294766214671,
+            -0.009502185346558072,
+            -0.0013419858580438616,
+            0.004443749010136947,
+            0.013008783570355656,
+            -0.02289662952366337,
+            0.0013834888740540108,
+            -0.00048549197205067054,
+            0.027969842651517314,
+            0.010437729090975158,
+            -0.0047705656249623744,
+            0.014202218020412936,
+            0.008240861154198411,
+            -0.018637315941614406,
+            0.007314019317343154,
+            -0.005747966765101792,
+            0.014833896739762147,
+            -0.01314847940891154,
+            0.007762100494499689,
+            0.00705026393499352,
+            -0.01278023669088385,
+            0.03016342404000438,
+            -0.015204129985039077,
+            0.007776946006562982,
+            0.009278778571285689,
+            0.009656723862179986,
+            0.01257686993982689,
+            0.004362439504889069,
+            -0.02898418193036562,
+            0.0002144431781454467,
+            -0.03828100004144633,
+            -0.00833748467467171,
+            -0.002611937107022567,
+            -0.002145329666219461,
+            0.007050630951707937,
+            -0.009383645162261629,
+            -0.033976442057557676,
+            0.0006429978367640709,
+            0.01603027598131823,
+            0.017501067413108286,
+            0.0002196997337451293,
+            0.013075379412332973,
+            -0.006197537539554383,
+            -0.021148120787551232,
+            -0.016986448660412497,
+            -0.010207705210051413,
+            -0.02122194744719993,
+            -0.020550470201191853,
+            -0.00908564461857315,
+            0.012090079127547193,
+            -0.009927390005879807,
+            0.013888458967497627,
+            0.029513250508042935,
+            -0.024529175134769883,
+            0.0011144447737101862,
+            0.021977677894839922,
+            -0.00198308986575084,
+            0.014108189311291326,
+            0.001696775006906642,
+            0.014122064251288995,
+            -0.0004166140484018266,
+            -0.010556016685519523,
+            -0.009729362492661684,
+            0.017487929296614076,
+            -0.0026043257634797124,
+            0.016715798948897194,
+            0.012183487338360088,
+            0.003771721868999568,
+            -0.0030887189945795796,
+            0.010667776697461199,
+            -0.026471157050402344,
+            0.014249996582064706,
+            -0.006690961709923224,
+            -0.0181382691440242,
+            -0.008729104191096461,
+            0.003882731567105777,
+            -0.005609384377642956,
+            -0.007794812949125115,
+            0.012967089730551055,
+            -0.00944374111323783,
+            0.02088525981049842,
+            0.0023400121055514958,
+            -0.012196925240774426,
+            0.003648688547842322,
+            -0.03322041209304968,
+            0.0015191765655963997,
+            -0.018402382776013956,
+            -0.024659631472836927,
+            0.03840173429666435,
+            0.011991452465253198,
+            0.006639702644606365,
+            -0.01675838115087676,
+            0.007953102302963526,
+            0.0004257479565099406,
+            0.017484338819046517,
+            0.011068864013925452,
+            0.006466915839538296,
+            0.008364454017976605,
+            -0.023316137291380285,
+            -0.010898421318668592,
+            0.016204569093730564,
+            0.012099473359791148,
+            -0.01017229369980751,
+            0.019431669546223253,
+            0.017955119269367787,
+            0.007300372071724306,
+            -0.006898619924677706,
+            0.01647927363640854,
+            0.021012103522831126,
+            0.010291016618557653,
+            -0.00047955552103474145,
+            0.0008684297441778696,
+            -0.010534247928997246,
+            0.005037930235889528,
+            0.035417626740307205,
+            0.014712595692093886,
+            0.010650306363963653,
+            -0.03626584475133921,
+            0.021764456453493054,
+            -0.011961521673661574,
+            0.009092981192874177,
+            0.006089104887183293,
+            0.017982801101846038,
+            0.006916969397519662,
+            0.014196254958707752,
+            0.011745069030657836,
+            -0.006689305450954901,
+            0.025205280281804708,
+            -0.001601414806100461,
+            -0.011992525142592405,
+            0.003386930324498371,
+            -0.01859193251438437,
+            0.005303261319773131,
+            -0.0041710536965100615,
+            -0.004459393120257307,
+            0.0032461828144552565,
+            -0.0018650956576455287,
+            0.015678225251799887,
+            0.003894142628084471,
+            -0.02355212701026675,
+            -0.0014873755115206368,
+            -0.024284295126348117,
+            0.0027260592623277375,
+            -0.022185571998195344,
+            0.00022883605560765203,
+            -0.005557568353382136,
+            -0.009749382813201403,
+            -0.025385857431333838,
+            -0.009853073898348951,
+            -0.0035843255959309307,
+            0.007661137888845632,
+            -0.010521037433807653,
+            -0.004246698084422667,
+            -0.0019920499321964815,
+            -0.019465113363548715,
+            0.009014255846983305,
+            0.027122533756959888,
+            0.016848221859975445,
+            0.027170883580103264,
+            -0.00019554112591107156,
+            0.013614675849485926,
+            -0.002066822813103284,
+            -0.01603063005944364,
+            0.008824663429777616,
+            0.0075653522381218495,
+            -0.00531780740880296,
+            0.014643348459785112,
+            0.0072634484937344704,
+            -0.012800149015319017,
+            0.009859503660384916,
+            -0.007161712176469609,
+            -0.006437857349742842,
+            -0.009764856421740622,
+            -0.02058868349990357,
+            -0.014866137971603676,
+            0.0012450207572602557,
+            -0.002123265406747006,
+            0.006000473862248991,
+            -0.004054016790850226,
+            0.009584879012937315,
+            -0.00608816807208866,
+            -0.00925801057439376,
+            -0.013962683072792106,
+            -0.008924356118158748,
+            -0.004210064145949253,
+            -0.004381709838058668,
+            -0.011732699976353612,
+            -0.005479523097469261,
+            0.0010680258077037758,
+            -0.0019401640391035755,
+            -0.01397652201595301,
+            0.011650839243603377,
+            0.0058731838291845645,
+            -0.002462743134993977,
+            0.02219008688925881,
+            0.01097316136828334,
+            0.0018083001685932406,
+            0.01366095798812196,
+            0.0003942601242216818,
+            -0.0064063532958756175,
+            -0.03343312148366622,
+            -0.0005083355836126291,
+            -0.02815586367527057,
+            0.025770078162165123,
+            -0.0069114949102666145,
+            -0.010322087105114614,
+            -0.002519047562374146,
+            -0.006063669644021698,
+            0.005873609611264377,
+            -0.002722940652283699,
+            0.0021229525800901114,
+            -0.023673778230768945,
+            0.0028629715379660113,
+            -0.003953653437464966,
+            -0.0036366957271968823,
+            0.007469664864329332,
+            0.0346520690389124,
+            -0.004054282838167587,
+            -0.0034765253405897676,
+            0.005159466707114532,
+            0.011642913651743502,
+            0.027845571220425726,
+            -0.004445196354520585,
+            -0.0032038508043339413,
+            -0.02233838584152591,
+            0.0054112576531374175,
+            -0.00019544838850471866,
+            -0.004733192299951088,
+            0.016664217348411003,
+            -0.016719175418727594,
+            -0.01631658832047864,
+            -0.00545969677717507,
+            -0.014570848053102338,
+            -0.00022620258237138677,
+            0.014454101124666536,
+            0.011056312437637014,
+            -0.004035526363810078,
+            -0.012459378363360557,
+            -0.0022595493558020706,
+            -0.030370952502579162,
+            0.012766010284884204,
+            0.1141689619126009,
+            -0.00766197047210299,
+            -0.00896655658602649,
+            -0.009481112393590025,
+            0.001435269335646347,
+            0.001338755472877713,
+            -0.011739143028767118,
+            0.07166096084053437,
+            -0.0077855767070652395,
+            0.0002139580430342728,
+            -0.011312198831181402,
+            0.001844005266872301,
+            -0.001825412693163496,
+            0.01371932421024605,
+            -0.005025814503687022,
+            -0.018213360991353643,
+            -0.0033612481616769416,
+            -0.02941403830641238,
+            -0.0041137865872933515,
+            -0.005683630256743215,
+            -0.004288198570156578,
+            -0.014982999363445122,
+            0.0067149665791261335,
+            -0.019661973497372313,
+            -0.012688556129040908,
+            0.001124391650175233,
+            -0.0033623453063430534,
+            0.00996944383866679,
+            -0.020496336814603385,
+            -0.0044354321069609536,
+            0.0019875840782527733,
+            0.009706675240563883,
+            0.0011600364428219214,
+            -0.012755178230821602,
+            -0.0026514442584739412,
+            0.004056350166031111,
+            0.01909951327913368,
+            0.014360281326789544,
+            0.003732344102045803,
+            0.008623457695718206,
+            0.014461868085333023,
+            -0.001692149528228451,
+            0.01948035376572433,
+            0.01172550239477637,
+            0.0031966531144076173,
+            0.0004779413449255626,
+            0.0030912562273611517,
+            -0.004597633235071031,
+            0.01582547400214653,
+            -0.014811177637844808,
+            -0.02411202772171917,
+            -0.0071272238837422185,
+            0.018015541137524113,
+            -0.013081287852028074,
+            1.8837689271256874e-05,
+            0.004062803284646432,
+            0.027569726868200254,
+            0.0010564850664255057,
+            -0.01179755720247972,
+            0.0028876134800690334,
+            -0.01638215925131053,
+            0.022944183106912663,
+            0.0026043713064243836,
+            0.03279173899194346,
+            -0.005103903273968361,
+            0.00578213669463383,
+            -0.020744194169889544,
+            -0.007588793365216579,
+            -0.00819862611489168,
+            0.0017401442433921013,
+            0.010993500174794084,
+            0.005411238350013134,
+            0.01620925462604661,
+            -0.00641615069699833,
+            -0.020751730783437995,
+            0.014776878596707538,
+            -0.009552193304486048,
+            -0.03393319554458277,
+            0.008881987295131347,
+            -0.029102297123590998,
+            -0.0045267103602711,
+            0.00368971742585232,
+            -0.014269341759285646,
+            -0.016264809810970175,
+            0.012837302095591106,
+            -0.0011890334180728564,
+            -0.008937992768323412,
+            0.0014444613725248846,
+            -0.017110187133760626,
+            -0.02385668037818897,
+            0.006017405364919834,
+            0.00676670672597817,
+            -0.0077994500447595,
+            0.018312452858615755,
+            -0.027015660037024585,
+            0.006123912473734307,
+            -0.012973676426738582,
+            0.009041974762537394,
+            -0.01580183887345168,
+            0.01163516332546347,
+            0.010887662470499361,
+            -0.001503467664709027,
+            -0.0076234124970223326,
+            0.01231202541338883,
+            0.004852720134894811,
+            0.033760365210093034,
+            0.028658326381800395,
+            0.000995999783627898,
+            -0.0025605367058226737,
+            -0.0020356794893366384,
+            0.011667525096213252,
+            -0.014049700287539817,
+            0.011218246246280464,
+            0.02658113044439717,
+            0.017568631325838108,
+            -0.019133062654008885,
+            0.018050678772708717,
+            -0.010356118256625172,
+            0.012872052420999813,
+            -0.025620154415048252,
+            -0.005135768485044442,
+            0.004936490920616527,
+            0.005163008729256001,
+            0.007425278892536464,
+            0.001410212306439329,
+            0.021891387289347516,
+            0.014999616600766989,
+            0.00027613811156650413,
+            0.00558422822386443,
+            0.01477009526981092,
+            0.010497782599293994,
+            -0.0005567894749233204,
+            -0.00467776506604011,
+            -0.016998035358377343,
+            0.00957271797745408,
+            -0.006065116857898469,
+            0.005138070401846389,
+            0.01698914855850691,
+            -0.01230257809343985,
+            0.011109844802663414,
+            0.007518553660463051,
+            -0.015504773927595823,
+            -0.017169933945839305,
+            0.005552829003043926,
+            -0.02252633790984703,
+            -0.01306722099657118,
+            0.02715689072730398,
+            -0.010513941399990838,
+            -0.02790582732974509,
+            -0.0009847768069205477,
+            0.009797865241108237,
+            -0.002772952073769185,
+            -0.0027968741526000576,
+            0.0037541733651479613,
+            0.010801137722284016,
+            -0.014658156970099418,
+            0.002075415233345957,
+            0.004735264309137368,
+            0.01930804315461151,
+            -0.0014151522239496115,
+            -0.015046118414100617,
+            -0.002288495894227231,
+            0.021592829677937263,
+            0.017174609713578173,
+            0.00265560378681856,
+            -0.016499825201963983,
+            -3.965296928410772e-05,
+            -0.0074142747014567635,
+            0.00021360696629503726,
+            0.00027431174904249525,
+            -0.0056391089282546054,
+            -0.018931681700770225,
+            -0.003888248121485032,
+            0.011181675419027736,
+            -0.00247252920175839,
+            0.01977860312223846,
+            -0.0011527575128223392,
+            -0.015763587433200915,
+            0.001960734212846797,
+            -0.005280860341480037,
+            -0.010372829570478649,
+            0.01351900743232294,
+            -0.009359942374603997,
+            0.006788746813046239,
+            -0.003212015225284119,
+            0.004611143085183367,
+            -0.0003308099698088422,
+            -0.017017265064185255,
+            0.02924142709831441,
+            -0.0018238549856433466,
+            -0.004769274041546295,
+            -0.0014058363976586562,
+            -0.033393331166741945,
+            0.015017165991303838,
+            -0.014574579250749552,
+            0.006745699409061803,
+            0.002705853802154516,
+            0.01272741296341735,
+            0.003253095246601883,
+            -0.008898616461779179,
+            0.0027485606928249326,
+            0.017040360927723885,
+            0.013756402798365262,
+            -0.007292864293090161,
+            0.008694855886257706,
+            -0.0028934389922887676,
+            -0.0037483208383542015,
+            0.01363963339139356,
+            0.004170611039133969,
+            -0.006282657671540625,
+            -0.0009784372731140945
+          ],
+          "keyphrases": [
+            "DeepSeekMoE architecture",
+            "expert specialization",
+            "Mixture-of-Experts (MoE)",
+            "fine-grained expert segmentation",
+            "shared expert isolation"
+          ],
+          "title": "Deepseekmoe: Towards Ultimate Expert Specialization In Mixture-Of-Experts Language Models"
+        },
+        "type": "document"
+      },
+      "bidirectional": true,
+      "properties": {
+        "summary_cosine_similarity": 0.7431007605177407
+      }
+    },
+    {
+      "id": "9f37ec0a-5922-4868-9fe9-9bbf4ec64cd8",
+      "type": "summary_cosine_similarity",
+      "source": {
+        "id": "14339b1f-53f8-479b-9da7-9430b174b9ee",
+        "properties": {
+          "page_content": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer\n\nNoam Shazeer1, Azalia Mirhoseini∗†1, Krzysztof Maziarz∗2, Andy Davis1, Quoc Le1, Geoffrey Hinton1and Jeff Dean1 1Google Brain, {noam,azalia,andydavis,qvl,geoffhinton,jeff}@google.com 2Jagiellonian University, Cracow, krzysztof.maziarz@student.uj.edu.pl\n\nAbstract\n\nThe capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost.\n\n1 Introduction And Related Work 1.1 Conditional Computation\n\nExploiting scale in both training data and model size has been central to the success of deep learning. When datasets are sufficiently large, increasing the capacity (number of parameters) of neural networks can give much better prediction accuracy. This has been shown in domains such as text (Sutskever et al., 2014; Bahdanau et al., 2014; Jozefowicz et al., 2016; Wu et al., 2016), images (Krizhevsky et al., 2012; Le et al., 2012), and audio (Hinton et al., 2012; Amodei et al., 2015). For typical deep learning models, where the entire model is activated for every example, this leads to a roughly quadratic blow-up in training costs, as both the model size and the number of training examples increase. Unfortunately, the advances in computing power and distributed computation fall short of meeting such demand.\n\nVarious forms of conditional computation have been proposed as a way to increase model capacity without a proportional increase in computational costs (Davis & Arel, 2013; Bengio et al., 2013; Eigen et al., 2013; Ludovic Denoyer, 2014; Cho & Bengio, 2014; Bengio et al., 2015; Almahairi et al., 2015). In these schemes, large parts of a network are active or inactive on a per-example basis. The gating decisions may be binary or sparse and continuous, stochastic or deterministic. Various forms of reinforcement learning and back-propagation are proposed for trarining the gating decisions.\n\nWhile these ideas are promising in theory, no work to date has yet demonstrated massive improvements in model capacity, training time, or model quality. We blame this on a combination of the following challenges:\n\nModern computing devices, especially GPUs, are much faster at arithmetic than at branching. Most of the works above recognize this and propose turning on/off large chunks of the network with each gating decision.\n\nLarge batch sizes are critical for performance, as they amortize the costs of parameter transfers and updates. Conditional computation reduces the batch sizes for the conditionally active chunks of the network.\n\nNetwork bandwidth can be a bottleneck. A cluster of GPUs may have computational power thousands of times greater than the aggregate inter-device network bandwidth. To be computationally efficient, the relative computational versus network demands of an algorithm must exceed this ratio. Embedding layers, which can be seen as a form of conditional computation, are handicapped by this very problem. Since the embeddings generally need to be sent across the network, the number of (example, parameter) interactions is limited by network bandwidth instead of computational capacity.\n\nDepending on the scheme, loss terms may be necessary to achieve the desired level of sparsity per-chunk and/or per example. Bengio et al. (2015) use three such terms. These issues can affect both model quality and load-balancing.\n\nModel capacity is most critical for very large data sets. The existing literature on conditional computation deals with relatively small image recognition data sets consisting of up to 600,000 images. It is hard to imagine that the labels of these images provide a sufficient signal to adequately train a model with millions, let alone billions of parameters. In this work, we for the first time address all of the above challenges and finally realize the promise of conditional computation. We obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets.\n\n1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1). All parts of the network are trained jointly by back-propagation.\n\nWhile the introduced technique is generic, in this paper we focus on language modeling and machine translation tasks, which are known to benefit from very large models. In particular, we apply a MoE convolutionally between stacked LSTM layers (Hochreiter & Schmidhuber, 1997), as in Figure 1.\n\nThe MoE is called once for each position in the text, selecting a potentially different combination of experts at each position. The different experts tend to become highly specialized based on syntax and semantics (see Appendix E Table 9). On both language modeling and machine translation benchmarks, we improve on best published results at a fraction of the computational cost.\n\n1.3 Related Work On Mixtures Of Experts\n\nSince its introduction more than two decades ago (Jacobs et al., 1991; Jordan & Jacobs, 1994), the mixture-of-experts approach has been the subject of much research. Different types of expert architectures hae been proposed such as SVMs (Collobert et al., 2002), Gaussian Processes (Tresp, 2001; Theis & Bethge, 2015; Deisenroth & Ng, 2015), Dirichlet Processes (Shahbaba & Neal, 2009), and deep networks. Other work has focused on different expert configurations such as a hierarchical structure (Yao et al., 2009), infinite numbers of experts (Rasmussen & Ghahramani, 2002), and adding experts sequentially (Aljundi et al., 2016). Garmash & Monz (2016) suggest an ensemble model in the format of mixture of experts for machine translation. The gating network is trained on a pre-trained ensemble NMT model. The works above concern top-level mixtures of experts. The mixture of experts is the whole model. Eigen et al. (2013) introduce the idea of using multiple MoEs with their own gating networks as parts of a deep model. It is intuitive that the latter approach is more powerful, since complex problems may contain many sub-problems each requiring different experts. They also allude in their conclusion to the potential to introduce sparsity, turning MoEs into a vehicle for computational computation.\n\nOur work builds on this use of MoEs as a general purpose neural network component. While Eigen et al. (2013) uses two stacked MoEs allowing for two sets of gating decisions, our convolutional application of the MoE allows for different gating decisions at each position in the text. We also realize sparse gating and demonstrate its use as a practical way to massively increase model capacity.\n\n2 The Structure Of The Mixture-Of-Experts Layer\n\nThe Mixture-of-Experts (MoE) layer consists of a set of n \"expert networks\" E1, · · · , En, and a \"gating network\" G whose output is a sparse n-dimensional vector. Figure 1 shows an overview of the MoE module. The experts are themselves neural networks, each with their own parameters.\n\nAlthough in principle we only require that the experts accept the same sized inputs and produce the same-sized outputs, in our initial investigations in this paper, we restrict ourselves to the case where the models are feed-forward networks with identical architectures, but with separate parameters.\n\nLet us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows:\n\n$$y=\\sum_{i=1}^{n}G(x){i}E{i}(x)$$ $$(\\mathbf{l})$$ G(x)iEi(x) (1) We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example. If the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network. In the following we focus on ordinary MoEs. We provide more details on hierarchical MoEs in Appendix B. Our implementation is related to other models of conditional computation. A MoE whose experts are simple weight matrices is similar to the parameterized weight matrix proposed in (Cho & Bengio, 2014). A MoE whose experts have one hidden layer is similar to the block-wise dropout described in (Bengio et al., 2015), where the dropped-out layer is sandwiched between fully-activated layers.\n\n2.1 Gating Network\n\nSoftmax Gating: A simple choice of non-sparse gating function (Jordan & Jacobs, 1994) is to multiply the input by a trainable weight matrix Wg and then apply the Sof tmax function.\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ $\\eqref{eq:walpha}$. Gσ(x) = Sof tmax(x · Wg) (2) Noisy Top-K Gating: We add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to −∞ (which causes the corresponding gate values to equal 0). The sparsity serves to save computation, as described above. While this form of sparsity creates some theoretically scary discontinuities in the output of gating function, we have not yet observed this to be a problem in practice. The noise term helps with load balancing, as will be discussed in Appendix A. The amount of noise per component is controlled by a second trainable weight matrix Wnoise.\n\n$$G(x)=S o f t m a x(K e e p T o p K(H(x),k))$$ $$(4)$$ $$(S)$$ G(x) = Sof tmax(KeepT opK(H(x), k)) (3) $$H(x){i}=(x\\cdot W{g}){i}+S t a n d a r d N o r m a l()\\cdot S o f t p l u s((x\\cdot W{n o i s e}){i})$$ $KeepTopK(v,k){i}=\\begin{cases}v_{i}&\\text{if}v_{i}\\text{is in the top}k\\text{elements of}v.\\ -\\infty&\\text{otherwise.}\\end{cases}$ Training the Gating Network We train the gating network by simple back-propagation, along with the rest of the model. If we choose k > 1, the gate values for the top k experts have nonzero derivatives with respect to the weights of the gating network. This type of occasionally-sensitive behavior is described in (Bengio et al., 2013) with respect to noisy rectifiers. Gradients also backpropagate through the gating network to its inputs. Our method differs here from (Bengio et al., 2015) who use boolean gates and a REINFORCE-style approach to train the gating network.\n\n3 Addressing Performance Challenges 3.1 The Shrinking Batch Problem\n\nOn modern CPUs and GPUs, large batch sizes are necessary for computational efficiency, so as to amortize the overhead of parameter loads and updates. If the gating network chooses k out of n experts for each example, then for a batch of b examples, each expert receives a much smaller batch of approximately kb n b examples. This causes a naive MoE implementation to become very inefficient as the number of experts increases. The solution to this shrinking batch problem is to make the original batch size as large as possible. However, batch size tends to be limited by the memory necessary to store activations between the forwards and backwards passes. We propose the following techniques for increasing the batch size: Mixing Data Parallelism and Model Parallelism: In a conventional distributed training setting, multiple copies of the model on different devices asynchronously process distinct batches of data, and parameters are synchronized through a set of parameter servers. In our technique, these different batches run synchronously so that they can be combined for the MoE layer. We distribute the standard layers of the model and the gating network according to conventional data-parallel schemes, but keep only one shared copy of each expert. Each expert in the MoE layer receives a combined batch consisting of the relevant examples from all of the data-parallel input batches. The same set of devices function as data-parallel replicas (for the standard layers and the gating networks) and as model-parallel shards (each hosting a subset of the experts). If the model is distributed over d devices, and each device processes a batch of size b, each expert receives a batch of approximately kbd nexamples. Thus, we achieve a factor of d improvement in expert batch size.\n\nIn the case of a hierarchical MoE (Section B), the primary gating network employs data parallelism, and the secondary MoEs employ model parallelism. Each secondary MoE resides on one device.\n\nThis technique allows us to increase the number of experts (and hence the number of parameters) by proportionally increasing the number of devices in the training cluster. The total batch size increases, keeping the batch size per expert constant. The memory and bandwidth requirements per device also remain constant, as do the step times, as does the amount of time necessary to process a number of training examples equal to the number of parameters in the model. It is our goal to train a trillionparameter model on a trillion-word corpus. We have not scaled our systems this far as of the writing of this paper, but it should be possible by adding more hardware.\n\nTaking Advantage of Convolutionality: In our language models, we apply the same MoE to each time step of the previous layer. If we wait for the previous layer to finish, we can apply the MoE to all the time steps together as one big batch. Doing so increases the size of the input batch to the MoE layer by a factor of the number of unrolled time steps.\n\nIncreasing Batch Size for a Recurrent MoE: We suspect that even more powerful models may involve applying a MoE recurrently. For example, the weight matrices of a LSTM or other RNN could be replaced by a MoE. Sadly, such models break the convolutional trick from the last paragraph, since the input to the MoE at one timestep depends on the output of the MoE at the previous timestep. Gruslys et al. (2016) describe a technique for drastically reducing the number of stored activations in an unrolled RNN, at the cost of recomputing forward activations. This would allow for a large increase in batch size.\n\n3.2 Network Bandwidth\n\nAnother major performance concern in distributed computing is network bandwidth. Since the experts are stationary (see above) and the number of gating parameters is small, most of the communication involves sending the inputs and outputs of the experts across the network. To maintain computational efficiency, the ratio of an expert's computation to the size of its input and output must exceed the ratio of computational to network capacity of the computing device. For GPUs, this may be thousands to one. In our experiments, we use experts with one hidden layer containing thousands of RELU-activated units. Since the weight matrices in the expert have sizes input_size×hidden_size and hidden_size × output_size, the ratio of computation to input and output is equal to the size of the hidden layer. Conveniently, we can increase computational efficiency simply by using a larger hidden layer, or more hidden layers.\n\n4 Balancing Expert Utilization\n\nWe have observed that the gating network tends to converge to a state where it always produces large weights for the same few experts. This imbalance is self-reinforcing, as the favored experts are trained more rapidly and thus are selected even more by the gating network. Eigen et al. (2013) describe the same phenomenon, and use a hard constraint at the beginning of training to avoid this local minimum. Bengio et al. (2015) include a soft constraint on the batch-wise average of each gate.1 We take a soft constraint approach. We define the importance of an expert relative to a batch of training examples to be the batchwise sum of the gate values for that expert. We define an additional loss Limportance, which is added to the overall loss function for the model. This loss is equal to the square of the coefficient of variation of the set of importance values, multiplied by a hand-tuned scaling factor wimportance. This additional loss encourages all experts to have equal importance.\n\n$$I m p o r t a n c e(X)=\\sum_{x\\in X}G(x)$$ $$L_{importance}(X)=w_{importance}\\cdot CV(Importance(X))^{2}\\tag{7}$$ $$(6)$$\n\nWhile this loss function can ensure equal importance, experts may still receive very different numbers of examples. For example, one expert may receive a few examples with large weights, and another may receive many examples with small weights. This can cause memory and performance problems on distributed hardware. To solve this problem, we introduce a second loss function, Lload , which ensures balanced loads. Appendix A contains the definition of this function, along with experimental results.\n\n5 Experiments 5.1 1 Billion Word Language Modeling Benchmark\n\nDataset: This dataset, introduced by (Chelba et al., 2013) consists of shuffled unique sentences from news articles, totaling approximately 829 million words, with a vocabulary of 793,471 words.\n\nPrevious State-of-the-Art: The best previously published results (Jozefowicz et al., 2016) use models consisting of one or more stacked Long Short-Term Memory (LSTM) layers (Hochreiter & Schmidhuber, 1997; Gers et al., 2000). The number of parameters in the LSTM layers of these models vary from 2 million to 151 million. Quality increases greatly with parameter count, as do computational costs. Results for these models form the top line of Figure 2-right. MoE Models: Our models consist of two stacked LSTM layers with a MoE layer between them (see Figure 1). We vary the sizes of the layers and the number of experts. For full details on model architecture, training regimen, additional baselines and results, see Appendix C.\n\nLow Computation, Varied Capacity: To investigate the effects of adding capacity, we trained\n\na series of MoE models all with roughly equal computational costs: about 8 million multiply-andadds per training example per timestep in the forwards pass, excluding the softmax layer. We call this metric (ops/timestep). We trained models with flat MoEs containing 4, 32, and 256 experts, and models with hierarchical MoEs containing 256, 1024, and 4096 experts. Each expert had about 1 million parameters. For all the MoE layers, 4 experts were active per input.\n\nThe results of these models are shown in Figure 2-left. The model with 4 always-active experts performed (unsurprisingly) similarly to the computationally-matched baseline models, while the largest of the models (4096 experts) achieved an impressive 24% lower perplexity on the test set. Figure 2: Model comparison on 1-Billion-Word Language-Modeling Benchmark. On the left, we plot test perplexity as a function of model capacity for models with similar computational budgets of approximately 8-million-ops-per-timestep. On the right, we plot test perplexity as a function of computational budget. The top line represents the LSTM models from (Jozefowicz et al., 2016).\n\nThe bottom line represents 4-billion parameter MoE models with different computational budgets.\n\nVaried Computation, High Capacity: In addition to the largest model from the previous section, we trained two more MoE models with similarly high capacity (4 billion parameters), but higher computation budgets. These models had larger LSTMs, and fewer but larger and experts. Details\n\nTest Test #Parameters ops/timestep Training TFLOPS Perplexity Perplexity excluding embedding Time /GPU 10 epochs 100 epochs and softmax layers 10 epochs Best Published Results 34.7 30.6 151 million 151 million 59 hours, 32 k40s 1.09 Low-Budget MoE Model 34.1 4303 million 8.9 million 15 hours, 16 k40s 0.74 Medium-Budget MoE Model 31.3 4313 million 33.8 million 17 hours, 32 k40s 1.22 High-Budget MoE Model 28.0 4371 million 142.7 million 47 hours, 32 k40s 1.56\n\nTable 1: Summary of high-capacity MoE-augmented models with varying computational budgets, vs. best previously published results (Jozefowicz et al., 2016). Details in Appendix C. can be found in Appendix C.2. Results of these three models form the bottom line of Figure 2-right.\n\nTable 1 compares the results of these models to the best previously-published result on this dataset .\n\nEven the fastest of these models beats the best published result (when controlling for the number of training epochs), despite requiring only 6% of the computation.\n\nComputational Efficiency: We trained our models using TensorFlow (Abadi et al., 2016) on clusters containing 16-32 Tesla K40 GPUs. For each of our models, we determine computational efficiency in TFLOPS/GPU by dividing the number of floating point operations required to process one training batch by the observed step time and the number of GPUs in the cluster. The operation counts used here are higher than the ones we report in our ops/timestep numbers in that we include the backwards pass, we include the importance-sampling-based training of the softmax layer, and we count a multiply-and-add as two separate operations. For all of our MoE models, the floating point operations involved in the experts represent between 37% and 46% of the total.\n\nFor our baseline models wtih no MoE, observed computational efficiency ranged from 1.07-1.29 TFLOPS/GPU. For our low-computation MoE models, computation efficiency ranged from 0.740.90 TFLOPS/GPU, except for the 4-expert model which did not make full use of the available parallelism. Our highest-computation MoE model was more efficient at 1.56 TFLOPS/GPU, likely due to the larger matrices. These numbers represent a significant fraction of the theoretical maximum of 4.29 TFLOPS/GPU claimed by NVIDIA. Detailed results are in Appendix C, Table 7.\n\n5.2 100 BILLION WORD GOOGLE NEWS CORPUS\n\nOn the 1-billion-word corpus, adding additional capacity seems to produce diminishing returns as the number of parameters in the MoE layer exceeds 1 billion, as can be seen in Figure 2-left. We hypothesized that for a larger training set, even higher capacities would produce significant quality improvements. We constructed a similar training set consisting of shuffled unique sentences from Google's internal news corpus, totalling roughly 100 billion words. Similarly to the previous section, we tested a series of models with similar computational costs of about 8 million ops/timestep. In addition to a baseline LSTM model, we trained models augmented with MoE layers containing 32, 256, 1024, 4096, 16384, 65536, and 131072 experts. This corresponds to up to 137 billion parameters in the MoE layer. Details on architecture, training, and results are given in Appendix D.\n\nResults: Figure 3 shows test perplexity as a function of capacity after training on 10 billion words (top line) and 100 billion words (bottom line). When training over the full 100 billion words, test perplexity improves significantly up to 65536 experts (68 billion parameters), dropping 39% lower than the computationally matched baseline, but degrades at 131072 experts, possibly a result of too much sparsity. The widening gap between the two lines demonstrates (unsurprisingly) that increased model capacity helps more on larger training sets. Even at 65536 experts (99.994% layer sparsity), computational efficiency for the model stays at a respectable 0.72 TFLOPS/GPU.\n\n5.3 Machine Translation (Single Language Pair)\n\nModel Architecture: Our model was a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decreased the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We inserted MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). Each MoE layer contained up to 2048 experts each with about two million parameters, adding a total of about 8 billion parameters to the models. Further details on model architecture, testing procedure and results can be found in Appendix E.\n\nDatasets: We benchmarked our method on the WMT'14 En→Fr and En→De corpora, whose training sets have 36M sentence pairs and 5M sentence pairs, respectively. The experimental protocols were also similar to those in (Wu et al., 2016): newstest2014 was used as the test set to compare against previous work (Luong et al., 2015a; Zhou et al., 2016; Wu et al., 2016), while the combination of newstest2012 and newstest2013 was used as the development set. We also tested the same model on a Google's Production English to French data.\n\nTable 2: Results on WMT'14 En→ Fr newstest2014 (bold values represent best results). Model Test Test ops/timenstep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.69 40.35 85M 8.7B 3 days/64 k40s MoE with 2048 Experts (longer training) 2.63 40.56 85M 8.7B 6 days/64 k40s GNMT (Wu et al., 2016) 2.79 39.22 214M 278M 6 days/96 k80s GNMT+RL (Wu et al., 2016) 2.96 39.92 214M 278M 6 days/96 k80s PBMT (Durrani et al., 2014) 37.0 LSTM (6-layer) (Luong et al., 2015b) 31.5 LSTM (6-layer+PosUnk) (Luong et al., 2015b) 33.1 DeepAtt (Zhou et al., 2016) 37.7 DeepAtt+PosUnk (Zhou et al., 2016) 39.2\n\nTable 3: Results on WMT'14 En → De newstest2014 (bold values represent best results). Model Test Test ops/timestep Total Training Perplexity BLEU #Parameters Time MoE with 2048 Experts 4.64 26.03 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 5.25 24.91 214M 278M 1 day/96 k80s GNMT +RL (Wu et al., 2016) 8.08 24.66 214M 278M 1 day/96 k80s PBMT (Durrani et al., 2014) 20.7 DeepAtt (Zhou et al., 2016) 20.6\n\nTable 4: Results on the Google Production En→ Fr dataset (bold values represent best results). Model Eval Eval Test Test ops/timestep Total Training Perplexity BLEU Perplexity BLEU #Parameters Time MoE with 2048 Experts 2.60 37.27 2.69 36.57 85M 8.7B 1 day/64 k40s GNMT (Wu et al., 2016) 2.78 35.80 2.87 35.56 214M 278M 6 days/96 k80s\n\nResults: Tables 2, 3, and 4 show the results of our largest models, compared with published results. Our approach achieved BLEU scores of 40.56 and 26.03 on the WMT'14 En→Fr and En→De benchmarks. As our models did not use RL refinement, these results constitute significant gains of 1.34 and 1.12 BLEU score on top of the strong baselines in (Wu et al., 2016). The perplexity scores are also better.2 On the Google Production dataset, our model achieved 1.01 higher test BLEU score even after training for only one sixth of the time.\n\n5.4 Multilingual Machine Translation\n\nDataset: (Johnson et al., 2016) train a single GNMT (Wu et al., 2016) model on a very large combined dataset of twelve language pairs. Results are somewhat worse than those for 12 separately trained single-pair GNMT models. This is not surprising, given that the twelve models have 12 times the capacity and twelve times the aggregate training of the one model. We repeat this experiment with a single MoE-augmented model. See Appendix E for details on model architecture.\n\nWe train our model on the same dataset as (Johnson et al., 2016) and process the same number of training examples (about 3 billion sentence pairs). Our training time was shorter due to the lower computational budget of our model.\n\nResults: Results for the single-pair GNMT models, the multilingual GNMT model and the multilingual MoE model are given in Table 5. The MoE model achieves 19% lower perplexity on the dev set than the multilingual GNMT model. On BLEU score, the MoE model significantly beats the multilingual GNMT model on 11 of the 12 language pairs (by as much as 5.84 points), and even beats the monolingual GNMT models on 8 of 12 language pairs. The poor performance on English → Korean seems to be a result of severe overtraining, as for the rarer language pairs a small number of real examples were highly oversampled in the training corpus.\n\nTable 5: Multilingual Machine Translation (bold values represent best results). GNMT-Mono GNMT-Multi MoE-Multi MoE-Multi vs. GNMT-Multi Parameters 278M / model 278M 8.7B ops/timestep 212M 212M 102M training time, hardware various 21 days, 96 k20s 12 days, 64 k40s Perplexity (dev) 4.14 3.35 -19% French → English Test BLEU 36.47 34.40 37.46 +3.06 German → English Test BLEU 31.77 31.17 34.80 +3.63 Japanese → English Test BLEU 23.41 21.62 25.91 +4.29 Korean → English Test BLEU 25.42 22.87 28.71 +5.84 Portuguese → English Test BLEU 44.40 42.53 46.13 +3.60 Spanish → English Test BLEU 38.00 36.04 39.39 +3.35 English → French Test BLEU 35.37 34.00 36.59 +2.59 English → German Test BLEU 26.43 23.15 24.53 +1.38 English → Japanese Test BLEU 23.66 21.10 22.78 +1.68 English → Korean Test BLEU 19.75 18.41 16.62 -1.79 English → Portuguese Test BLEU 38.40 37.35 37.90 +0.55 English → Spanish Test BLEU 34.50 34.25 36.21 +1.96\n\n6 Conclusion\n\nThis work is the first to demonstrate major wins from conditional computation in deep networks.\n\nWe carefully identified the design considerations and challenges of conditional computing and addressed them with a combination of algorithmic and engineering solutions. While we focused on text, conditional computation may help in other domains as well, provided sufficiently large training sets. We look forward to seeing many novel implementations and applications of conditional computation in the years to come.\n\nAcknowledgments\n\nWe would like to thank all of the members of the Google Brain and Google Translate teams who helped us with this project, in particular Zhifeng Chen, Yonghui Wu, and Melvin Johnson. Thanks also to our anonymous ICLR reviewers for the helpful suggestions on making this paper better.\n\n2Reported perplexities relative to the tokenization used by both our models and GNMT.\n\nReferences\n\nMartín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Gordon Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda B. Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. CoRR, abs/1603.04467, 2016. URL http://arxiv.org/abs/1603.04467.\n\nRahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. CoRR, abs/1611.06194, 2016. URL http://arxiv.org/abs/1611.\n\n06194.\n\nA. Almahairi, N. Ballas, T. Cooijmans, Y. Zheng, H. Larochelle, and A. Courville. Dynamic Capacity Networks. ArXiv e-prints, November 2015.\n\nDario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan, Christopher Fougner, Tony Han, Awni Y. Hannun, Billy Jun, Patrick LeGresley, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595, 2015. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.\n\nEmmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297, 2015.\n\nYoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.\n\nCiprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling.\n\narXiv preprint arXiv:1312.3005, 2013.\n\nK. Cho and Y. Bengio. Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning. ArXiv e-prints, June 2014. Ronan Collobert, Samy Bengio, and Yoshua Bengio. A parallel mixture of SVMs for very large scale problems. Neural Computing, 2002.\n\nAndrew Davis and Itamar Arel. Low-rank approximations for conditional feedforward computation in deep neural networks. arXiv preprint arXiv:1312.4461, 2013. Marc Peter Deisenroth and Jun Wei Ng. Distributed Gaussian processes. In ICML, 2015.\n\nJohn Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization, 2010.\n\nNadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. Edinburgh's phrase-based machine translation systems for wmt-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation, 2014.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013. Ekaterina Garmash and Christof Monz. Ensemble learning for multi-source neural machine translation. In staff.science.uva.nl/c.monz, 2016.\n\nFelix A. Gers, Jürgen A. Schmidhuber, and Fred A. Cummins. Learning to forget: Continual prediction with lstm. Neural Computation, 2000.\n\nAudrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. Memory-efficient backpropagation through time. CoRR, abs/1606.03401, 2016. URL http://arxiv.org/ abs/1606.03401. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2015.\n\nGeoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 2012. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.\n\nSergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.\n\nRobert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computing, 1991.\n\nMelvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot translation.\n\nCoRR, abs/1611.04558, 2016. URL http://arxiv.org/abs/1611.04558.\n\nMichael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm.\n\nNeural Computing, 1994. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.\n\nDiederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.\n\nReinhard Kneser and Hermann. Ney. Improved backingoff for m-gram language modeling., 1995.\n\nAlex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.\n\nQuoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeffrey Dean, and Andrew Y. Ng. Building high-level features using large scale unsupervised learning. In ICML, 2012. Patrick Gallinari Ludovic Denoyer. Deep sequential neural network. arXiv preprint arXiv:1410.0510, 2014.\n\nMinh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attentionbased neural machine translation. EMNLP, 2015a.\n\nMinh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the rare word problem in neural machine translation. ACL, 2015b.\n\nCarl Edward Rasmussen and Zoubin Ghahramani. Infinite mixtures of Gaussian process experts.\n\nNIPS, 2002.\n\nHasim Sak, Andrew W Senior, and Françoise Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH, pp. 338–342, 2014.\n\nMike Schuster and Kaisuke Nakajima. Japanese and Korean voice search. ICASSP, 2012.\n\nBabak Shahbaba and Radford Neal. Nonlinear models using dirichlet process mixtures. JMLR, 2009.\n\nIlya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks.\n\nIn NIPS, 2014.\n\nLucas Theis and Matthias Bethge. Generative image modeling using spatial LSTMs. In NIPS, 2015. Volker Tresp. Mixtures of Gaussian Processes. In NIPS, 2001.\n\nYonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.\n\nBangpeng Yao, Dirk Walther, Diane Beck, and Li Fei-fei. Hierarchical mixture of classification experts uncovers interactions between brain regions. In NIPS. 2009.\n\nWojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization.\n\narXiv preprint arXiv:1409.2329, 2014.\n\nJie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with fast-forward connections for neural machine translation. arXiv preprint arXiv:1606.04199, 2016.\n\nAppendices A Load-Balancing Loss\n\nAs discussed in section 4, for load-balancing purposes, we want to define an additional loss function to encourage experts to receive roughly equal numbers of training examples. Unfortunately, the number of examples received by an expert is a discrete quantity, so it can not be used in backpropagation. Instead, we define a smooth estimator Load(X) of the number of examples assigned to each expert for a batch X of inputs. The smoothness allows us to back-propagate gradients through the estimator. This is the purpose of the noise term in the gating function. We define P(x, i) as the probability that G(x)iis nonzero, given a new random choice of noise on element i, but keeping the already-sampled choices of noise on the other elements. To compute P(x, i), we note that the G(x)iis nonzero if and only if H(x)iis greater than the k th-greatest element of H(x) excluding itself. The probability works out to be:\n\n$$P(x,i)=P r\\Big((x\\cdot W_{g}){i}+S t a n d x N o r a l()\\cdot S o f t p u s((x\\cdot W{n o i s e})_{i})$$ $$>k t h_e x c l u d i n(H(x),k,i)\\Big)$$ $$({\\boldsymbol{8}})$$\n\n$$(9)$$ $$(10)$$ (8) Where kth_excluding(v, k, i) means the kth highest component of v, excluding component i. Simplifying, we get:\n\n$$P(x,i)=\\Phi{\\Big(}{\\frac{(x\\cdot W_{g}){i}-k t h_c x c l u d i n g(H(x),k,i)}{S o f t p l u s((x\\cdot W{n o i s e})_{i})}}{\\Big)}$$\n\nWhere Φ is the CDF of the standard normal distribution.\n\n$$L o a d(X){i}=\\sum{x\\in X}P(x,i)$$\n\n$$(11)$$ P(x, i) (10) We can now define the load loss to be the square of the coefficient of variation of the load vector, multiplied by a hand-tuned scaling factor wload.\n\n$$L_{l o a d}(X)=w_{l o a d}\\cdot C V(L o a d(X))^{2}$$ Lload(X) = wload · CV (Load(X))2(11) Initial Load Imbalance: To avoid out-of-memory errors, we need to initialize the network in a state of approximately equal expert load (since the soft constraints need some time to work). To accomplish this, we initialize the matrices Wg and Wnoise to all zeros, which yields no signal and some noise.\n\nExperiments: We trained a set of models with identical architecture (the MoE-256 model described in Appendix C), using different values of wimportance and wload. We trained each model for 10 epochs, then measured perplexity on the test set. We also measured the coefficients of variation in Importance and Load, as well as ratio of the load on the most overloaded expert to the average load. This last value is significant for load balancing purposes on distributed hardware. All of these metrics were averaged over several training batches.\n\nwimportance wload Test Perplexity CV (Importance(X)) CV (Load(X)) max(Load(X)) mean(Load(X)) 0.0 0.0 39.8 3.04 3.01 17.80 0.2 0.0 35.6 0.06 0.17 1.47 0.0 0.2 35.7 0.22 0.04 1.15 0.1 0.1 35.6 0.06 0.05 1.14 0.01 0.01 35.7 0.48 0.11 1.37 1.0 1.0 35.7 0.03 0.02 1.07\n\nTable 6: Experiments with different combinations of losses.\n\n13 Results: Results are reported in Table 6. All the combinations containing at least one the two losses led to very similar model quality, where having no loss was much worse. Models with higher values of wload had lower loads on the most overloaded expert.\n\nB Hierachical Mixture Of Experts\n\nIf the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of \"experts\", each of which is itself a secondary mixture-of-experts with its own gating network.3If the hierarchical MoE consists of a groups of b experts each, we denote the primary gating network by Gprimary, the secondary gating networks by (G1, G2..Ga), and the expert networks by (E0,0, E0,1..Ea,b). The output of the MoE is given by:\n\n$$y_{H}=\\sum_{i=1}^{a}\\sum_{j=1}^{b}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}\\cdot E{i,j}(x)$$ $$(12)$$ $$(13)$$ $$(14)$$\n\nOur metrics of expert utilization change to the following:\n\n$$I m p o r t a n c e_{H}(X){i,j}=\\sum{x\\in X}G_{p r i m a r y}(x){i}\\cdot G{i}(x){j}$$ $$L o a d{H}(X){i,j}={\\frac{L o a d{p r i m a r y}(X){i}\\cdot L o a d{i}(X^{(i)})_{j}}{|X^{(i)}|}}$$\n\nLoadprimary and Loadi deonte the Load functions for the primary gating network and i th secondary gating network respectively. X(i) denotes the subset of X for which Gprimary(x)i > 0.\n\nIt would seem simpler to let LoadH(X)i,j = Loadi(Xi)j , but this would not have a gradient with respect to the primary gating network, so we use the formulation above.\n\nC 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS C.1 8-MILLION-OPERATIONS-PER-TIMESTEP MODELS Model Architecture: Our model consists of five layers: a word embedding layer, a recurrent Long Short-Term Memory (LSTM) layer (Hochreiter & Schmidhuber, 1997; Gers et al., 2000), a MoE layer, a second LSTM layer, and a softmax layer. The dimensionality of the embedding layer, the number of units in each LSTM layer, and the input and output dimensionality of the MoE layer are all equal to 512. For every layer other than the softmax, we apply drouput (Zaremba et al., 2014) to the layer output, dropping each activation with probability DropP rob, otherwise dividing by (1 − DropP rob). After dropout, the output of the previous layer is added to the layer output.\n\nThis residual connection encourages gradient flow (He et al., 2015).\n\nMoE Layer Architecture: Each expert in the MoE layer is a feed forward network with one ReLU-activated hidden layer of size 1024 and an output layer of size 512. Thus, each expert contains [512 ∗ 1024] + [1024 ∗ 512] = 1M parameters. The output of the MoE layer is passed through a sigmoid function before dropout. We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts.\n\nWe call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers. Thus, each example is processed by exactly 4 experts for a total of 4M ops/timestep. The two LSTM layers contribute 2M ops/timestep each for the desired total of 8M.\n\n3 We have not found the need for deeper hierarchies.\n\nComputationally-Matched Baselines: The MoE-4 model does not employ sparsity, since all 4 experts are always used. In addition, we trained four more computationally-matched baseline models with no sparsity:\n\nMoE-1-Wide: The MoE layer consists of a single \"expert\" containing one ReLU-activated hidden layer of size 4096.\n\nMoE-1-Deep: The MoE layer consists of a single \"expert\" containing four ReLU-activated hidden layers, each with size 1024.\n\n4xLSTM-512: We replace the MoE layer with two additional 512-unit LSTM layers.\n\nLSTM-2048-512: The model contains one 2048-unit LSTM layer (and no MoE). The output of the LSTM is projected down to 512 dimensions (Sak et al., 2014). The next timestep of the LSTM receives the projected output. This is identical to one of the models published in (Jozefowicz et al., 2016). We re-ran it to account for differences in training regimen, and obtained results very similar to the published ones. Training: The models were trained on a cluster of 16 K40 GPUs using the synchronous method described in Section 3. Each batch consisted of a set of sentences totaling roughly 300,000 words. In the interest of time, we limited training to 10 epochs, (27,000 steps). Training took 12-16 hours for all models, except for MoE-4, which took 18 hours (since all the expert computation was performed on only 4 of 16 GPUs). We used the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 1000 training steps, and decreased after that so as to be proportional to the inverse square root of the step number. The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016). For each model, we performed a hyper-parmeter search to find the best dropout probability, in increments of 0.1.\n\nTo ensure balanced expert utilization we set wimportance = 0.1 and wload = 0.1, as described in Section 4 and Appendix A.\n\nResults: We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol. Results are reported in Table 7. For each model, we report the test perplexity, the computational budget, the parameter counts, the value of DropP rob, and the computational efficiency.\n\nModel Test Test ops/timestep #Params excluding Total Drop- TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params P rob per GPU 10 epochs (final) (millions) (billions) (observed) Kneser-Ney 5-gram* 67.6 0.00001 1.8 LSTM-512-512* 54.1 2.4 2.4 0.8 0.1 LSTM-1024-512* 48.2 4.7 4.7 0.8 0.1 LSTM-2048-512* 45.0 43.7 9.4 9.4 0.8 0.1 0.61 LSTM-2048-512 44.7 9.4 9.4 0.8 0.1 1.21 4xLSTM-512 46.0 8.4 8.4 0.8 0.1 1.07 MoE-1-Wide 46.1 8.4 8.4 0.8 0.1 1.29 MoE-1-Deep 45.7 8.4 8.4 0.8 0.1 1.29 MoE-4 45.0 8.4 8.4 0.8 0.1 0.52 MoE-32 39.7 8.4 37.8 0.9 0.1 0.87 MoE-256 35.7 8.6 272.9 1.1 0.1 0.81 MoE-256-h 36.0 8.4 272.9 1.1 0.1 0.89 MoE-1024-h 34.6 8.5 1079.0 1.9 0.2 0.90 MoE-4096-h 34.1 8.9 4303.4 5.1 0.2 0.74 2xLSTM-8192-1024* 34.7 30.6 151.0 151.0 1.8 0.25 1.09 MoE-34M 31.3 33.8 4313.9 6.0 0.3 1.22 MoE-143M 28.0 142.7 4371.1 6.0 0.4 1.56\n\nC.2 More Expensive Models\n\nWe ran two additional models (MoE-34M and MoE-143M) to investigate the effects of adding more computation in the presence of a large MoE layer. These models have computation budgets of 34M and 143M ops/timestep. Similar to the models above, these models use a MoE layer between two LSTM layers. The dimensionality of the embedding layer, and the input and output dimensionality of the MoE layer are set to 1024 instead of 512. For MoE-34M, the LSTM layers have 1024 units.\n\nFor MoE-143M, the LSTM layers have 4096 units and an output projection of size 1024 (Sak et al., 2014). MoE-34M uses a hierarchical MoE layer with 1024 experts, each with a hidden layer of size 2048. MoE-143M uses a hierarchical MoE layer with 256 experts, each with a hidden layer of size 8192. Both models have 4B parameters in the MoE layers. We searched for the best DropP rob for each model, and trained each model for 10 epochs.\n\nThe two models achieved test perplexity of 31.3 and 28.0 respectively, showing that even in the presence of a large MoE, more computation is still useful. Results are reported at the bottom of Table 7. The larger of the two models has a similar computational budget to the best published model from the literature, and training times are similar. Comparing after 10 epochs, our model has a lower test perplexity by 18%.\n\nD 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS Model Architecture: The models are similar in structure to the 8-million-operations-per-timestep models described in the previous section. We vary the number of experts between models, using an ordinary MoE layer with 32 experts and hierarchical MoE layers with 256, 1024, 4096, 16384, 65536 and 131072 experts. For the hierarchical MoE layers, the first level branching factors are 32, 32, 64, 128, 256 and 256, respectively.\n\nTraining: Models are trained on a cluster of 32 Tesla K40 GPUs, except for the last two models, which are trained on clusters of 64 and 128 GPUs so as to have enough memory for all the parameters. For all models, training batch sizes are approximately 2.5 million words. Models are trained once-through over about 100 billion words. We implement several memory optimizations in order to fit up to 1 billion parameters per GPU. First, we do not store the activations of the hidden layers of the experts, but instead recompute them on the backwards pass. Secondly, we modify the optimizer on the expert parameters to require less auxiliary storage: The Adam optimizer (Kingma & Ba, 2015) keeps first and second moment estimates of the perparameter gradients. This triples the required memory. To avoid keeping a first-moment estimator, we set β1 = 0. To reduce the size of the second moment estimator, we replace it with a factored approximation. For a matrix of parameters, instead of maintaining a full matrix of second-moment estimators, we maintain vectors of row-wise and column-wise averages of that matrix. At each step, the matrix of estimators is taken to be the outer product of those two vectors divided by the mean of either one. This technique could similarly be applied to Adagrad (Duchi et al., 2010).\n\nModel Test Test ops/timestep #Params excluding Total TFLOPS Perplexity Perplexity (millions) embed. & softmax #Params per GPU .1 epochs 1 epoch (millions) (billions) (observed) Kneser-Ney 5-gram 67.1 45.3 0.00001 76.0 4xLSTM-512 54.5 47.0 8.4 8.4 0.1 1.23 MoE-32 48.5 40.4 8.4 37.8 0.1 0.83 MoE-256-h 42.8 35.3 8.4 272.9 0.4 1.11 MoE-1024-h 40.3 32.7 8.5 1079.0 1.2 1.14 MoE-4096-h 38.9 30.9 8.6 4303.4 4.4 1.07 MoE-16384-h 38.2 29.7 8.8 17201.0 17.3 0.96 MoE-65536-h 38.2 28.9 9.2 68791.0 68.9 0.72 MoE-131072-h 39.8 29.2 9.7 137577.6 137.7 0.30\n\nResults: We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8. Perplexity after 100 billion training words is 39% lower for the 68-billion-parameter MoE model than for the baseline model. It is notable that the measured computational efficiency of the largest model (0.30 TFLOPS/GPU) is very low compared to the other models. This is likely a result of the fact that, for purposes of comparison to the other models, we did not increase the training batch size proportionally to the number of GPUs. For comparison, we include results for a computationally matched baseline model consisting of 4 LSTMs, and for an unpruned 5-gram model with Kneser-Ney smoothing (Kneser & Ney, 1995).4\n\nE Machine Translation - Experimental Details\n\nModel Architecture for Single Language Pair MoE Models: Our model is a modified version of the GNMT model described in (Wu et al., 2016). To reduce computation, we decrease the number of LSTM layers in the encoder and decoder from 9 and 8 to 3 and 2 respectively. We insert MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). We use an attention mechanism between the encoder and decoder, with the first decoder LSTM receiving output from and providing input for the attention 5. All of the layers in our model have input and output dimensionality of 512. Our LSTM layers have 2048 hidden units, with a 512-dimensional output projection. We add residual connections around all LSTM and MoE layers to encourage gradient flow (He et al., 2015). Similar to GNMT, to effectively deal with rare words, we used subword units (also known as \"wordpieces\") (Schuster & Nakajima, 2012) for inputs and outputs in our system.\n\nWe use a shared source and target vocabulary of 32K wordpieces. We also used the same beam search technique as proposed in (Wu et al., 2016).\n\nWe train models with different numbers of experts in the MoE layers. In addition to a baseline model with no MoE layers, we train models with flat MoE layers containing 32 experts, and models with hierarchical MoE layers containing 512 and 2048 experts. The flat MoE layers use k = 4 and the hierarchical MoE models use k = 2 at each level of the gating network. Thus, each input is processed by exactly 4 experts in each MoE layer. Each expert in the MoE layer is a feed forward network with one hidden layer of size 2048 and ReLU activation. Thus, each expert contains [512 ∗ 2048] + [2048 ∗ 512] = 2M parameters. The output of the MoE layer is passed through a sigmoid function. We use the strictly-balanced gating function described in Appendix F.\n\nModel Architecture for Multilingual MoE Model: We used the same model architecture as for the single-language-pair models, with the following exceptions: We used noisy-top-k gating as described in Section 2.1, not the scheme from Appendix F. The MoE layers in the encoder and decoder are non-hierarchical MoEs with n = 512 experts, and k = 2. Each expert has a larger hidden layer of size 8192. This doubles the amount of computation in the MoE layers, raising the computational budget of the entire model from 85M to 102M ops/timestep. Training: We trained our networks using the Adam optimizer (Kingma & Ba, 2015). The base learning rate was increased linearly for the first 2000 training steps, held constant for an additional 8000 steps, and decreased after that so as to be proportional to the inverse square root of the step number. For the single-language-pair models, similarly to (Wu et al., 2016), we applied dropout (Zaremba et al., 2014) to the output of all embedding, LSTM and MoE layers, using DropP rob = 0.4. Training was done synchronously on a cluster of up to 64 GPUs as described in section 3. Each training batch consisted of a set of sentence pairs containing roughly 16000 words per GPU.\n\nTo ensure balanced expert utilization we set wimportance = 0.01 and wload = 0.01, as described in Section 4 and Appendix A. Metrics: We evaluated our models using the perplexity and the standard BLEU score metric. We reported tokenized BLEU score as computed by the multi-bleu.pl script, downloaded from the public implementation of Moses (on Github), which was also used in (Luong et al., 2015a). Results: Tables 2, 3 and 4 in Section 5.3 show comparisons of our results to other published methods. Figure 4 shows test perplexity as a function of number of words in the (training data's) source sentences processed for models with different numbers of experts. As can be seen from the Figure, as we increased the number of experts to approach 2048, the test perplexity of our model continued to improve.\n\nWe found that the experts indeed become highly specialized by syntax and/or semantics, as can be seen in Table 9. For example, one expert is used when the indefinite article \"a\" introduces the direct object in a verb phrase indicating importance or leadership.\n\nExpert 381 Expert 752 Expert 2004 ... with researchers , ... ... plays a core ... ... with rapidly growing ... ... to innovation . ... plays a critical ... ... under static conditions ... ... tics researchers . ... provides a legislative ... ... to swift ly ... ... the generation of ... ... play a leading ... ... to dras tically ... ... technology innovations is ... ... assume a leadership ... ... the rapid and ... ... technological innovations , ... ... plays a central ... ... the fast est ... ... support innovation throughout ... ... taken a leading ... ... the Quick Method ... ... role innovation will ... ... established a reconciliation ... ... rec urrent ) ... ... research scienti st ... ... played a vital ... ... provides quick access ... ... promoting innovation where ... ... have a central ... ... of volatile organic ... ... ... ...\n\nDue to some peculiarities in our infrastructure which have since been fixed, at the time we ran some of the machine translation experiments, our models ran faster if every expert received exactly the same batch size. To accommodate this, we used a different gating function which we describe below. Recall that we define the softmax gating function to be:\n\n$$G_{\\sigma}(x)=S o f t m a x(x\\cdot W_{g})$$ Gσ(x) = Sof tmax(x · Wg) (15) Sparse Gating (alternate formulation): To obtain a sparse gating vector, we multiply Gσ(x) component-wise with a sparse mask M(Gσ(x)) and normalize the output. The mask itself is a function of Gσ(x) and specifies which experts are assigned to each input example:\n\n$$(15)$$ $$G(x){i}=\\frac{G{\\sigma}(x){i}M(G{\\sigma}(x)){i}}{\\sum{j=1}^{n}G_{\\sigma}(x){j}M(G{\\sigma}(x))_{j}}\\tag{1}$$ $$(17)$$\n\n$$(16)$$\n\nTop-K Mask: To implement top-k gating in this formulation, we would let M(v) = T opK(v, k), where:\n\n$$T o p K(v,k){i}=\\begin{cases}1&{\\mathrm{if~}}v{i}{\\mathrm{~is~in~the~top~}}k{\\mathrm{~elements~of~}}v.\\ 0&{\\mathrm{otherwise.}}\\end{cases}$$ 0 otherwise. (17) Batchwise Mask: To force each expert to receive the exact same number of examples, we introduce an alternative mask function, Mbatchwise(X, m), which operates over batches of input vectors.\n\nInstead of keeping the top k values per example, we keep the top m values per expert across the training batch, where m = k|X| n, so that each example is sent to an average of k experts.\n\n$$M_{b a t c h w i s e}(X,m){j,i}=\\begin{cases}1&\\text{if}X{j,i}\\text{is in the top}m\\text{values for to expert}i\\ 0&\\text{otherwise}\\end{cases}$$\n\nAs our experiments suggest and also observed in (Ioffe & Szegedy, 2015), using a batchwise function during training (such as Mbatchwise) requires modifications to the inference when we may not have a large batch of examples. Our solution to this is to train a vector T of per-expert threshold values to approximate the effects of the batchwise mask. We use the following mask at inference time:\n\n$$(18)$$ $$M_{threshold}(x,T){i}=\\begin{cases}1&\\text{if}x{i}>T_{i}\\ 0&\\text{otherwise}\\end{cases}\\tag{1}$$ $$(19)$$\n\nTo learn the threshold values, we apply an additional loss at training time which is minimized when the batchwise mask and the threshold mask are identical.\n\n$$L_{batchwise}(X,T,m)=\\sum_{j=1}^{|X|}\\sum_{i=1}^{n}(M_{threshold}(x,T){i}-M{batchwise}(X,m){j,i})(X{j,i}-T_{i})\\tag{20}$$\n\nG ATTENTION FUNCTION The attention mechanism described in GNMT (Wu et al., 2016) involves a learned \"Attention Function\" A(xi, yj ) which takes a \"source vector\" xi and a \"target vector\" yj , and must be computed for every source time step i and target time step j. In GNMT, the attention function is implemented as a feed forward neural network with a hidden layer of size n. It can be expressed as:\n\n$$A_{G N M T}(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}t a n h((x_{i}U){d}+(y{j}W)_{d})$$\n\nWhere U and W are trainable weight matrices and V is a trainable weight vector.\n\nFor performance reasons, in our models, we used a slightly different attention function:\n\n$$A(x_{i},y_{j})=\\sum_{d=1}^{n}V_{d}tanh((x_{i}U){d})tanh((y{j}W)_{d})\\tag{22}$$\n\nWith our attention function, we can simultaneously compute the attention function on multiple source time steps and multiple target time steps using optimized matrix multiplications. We found little difference in quality between the two functions.\n\n$$(21)$$",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/1701.06538v1.md"
+          },
+          "headlines": {
+            "1 Introduction And Related Work": [
+              "1.1 Conditional Computation",
+              "1.2 OUR APPROACH: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER",
+              "1.3 Related Work On Mixtures Of Experts"
+            ],
+            "2 The Structure Of The Mixture-Of-Experts Layer": [
+              "2.1 Gating Network"
+            ],
+            "3 Addressing Performance Challenges": [
+              "3.1 The Shrinking Batch Problem",
+              "3.2 Network Bandwidth"
+            ],
+            "4 Balancing Expert Utilization": [],
+            "5 Experiments": [
+              "5.1 1 Billion Word Language Modeling Benchmark",
+              "5.2 100 BILLION WORD GOOGLE NEWS CORPUS",
+              "5.3 Machine Translation (Single Language Pair)",
+              "5.4 Multilingual Machine Translation"
+            ],
+            "6 Conclusion": [],
+            "Appendices": [
+              "A Load-Balancing Loss",
+              "B Hierachical Mixture Of Experts",
+              "C 1 BILLION WORD LANGUAGE MODELING BENCHMARK - EXPERIMENTAL DETAILS",
+              "D 100 BILLION WORD GOOGLE NEWS CORPUS - EXPERIMENTAL DETAILS",
+              "E Machine Translation - Experimental Details"
+            ]
+          },
+          "summary": "The paper introduces a Sparsely-Gated Mixture-of-Experts (MoE) layer to enhance neural network capacity without a proportional increase in computation. This approach uses a trainable gating network to select a sparse combination of experts for each input, significantly improving model capacity and performance in language modeling and machine translation tasks. The MoE layer, with up to 137 billion parameters, achieves better results than state-of-the-art models at lower computational costs. The paper addresses challenges like batch size and network bandwidth, proposing solutions like mixing data and model parallelism. Experiments show that MoE models outperform traditional models in various benchmarks, demonstrating the potential of conditional computation in deep learning.",
+          "summary_embedding": [
+            -0.01644345559179783,
+            -0.016995344310998917,
+            -0.016372699290513992,
+            -0.014349107630550861,
+            0.02282555401325226,
+            0.045934125781059265,
+            -0.014646278694272041,
+            0.03563220426440239,
+            -0.004829026758670807,
+            0.005617944523692131,
+            -0.00932550523430109,
+            -0.008143897168338299,
+            0.005575491581112146,
+            0.01538213063031435,
+            -0.005653322208672762,
+            -0.020306676626205444,
+            -0.013832597061991692,
+            -0.023235931992530823,
+            0.028344441205263138,
+            -0.020080260932445526,
+            0.011823155917227268,
+            -0.020943470299243927,
+            -0.025868017226457596,
+            -0.016500059515237808,
+            0.025061409920454025,
+            0.024806693196296692,
+            0.02583971433341503,
+            0.003827843815088272,
+            -0.03492465242743492,
+            0.033651065081357956,
+            0.014674580655992031,
+            0.012488252483308315,
+            -0.031500112265348434,
+            -0.009162768721580505,
+            -0.05351905897259712,
+            -0.05227377265691757,
+            0.04143410921096802,
+            0.022245362401008606,
+            -0.016231190413236618,
+            0.023717066273093224,
+            0.017264213413000107,
+            0.005816058721393347,
+            -0.020632149651646614,
+            -0.020844414830207825,
+            0.003700484987348318,
+            -0.013103820383548737,
+            -0.01983969286084175,
+            -0.013528349809348583,
+            -0.01828308403491974,
+            -0.01577835902571678,
+            0.009551920928061008,
+            0.0002856731880456209,
+            0.0029186424799263477,
+            -0.003424540627747774,
+            -0.047830358147621155,
+            0.0248208437114954,
+            0.019853845238685608,
+            0.03492465242743492,
+            0.04633035510778427,
+            0.013011839240789413,
+            0.01516986545175314,
+            -0.014271277002990246,
+            -0.04021712392568588,
+            0.005497660953551531,
+            -0.00041656987741589546,
+            -0.005380915477871895,
+            -0.005819596350193024,
+            0.008575502783060074,
+            0.008724087849259377,
+            -0.01143400277942419,
+            -0.04202845320105553,
+            0.013952880166471004,
+            -0.020561393350362778,
+            0.00010143388499272987,
+            0.01354957651346922,
+            0.008292482234537601,
+            0.02677368000149727,
+            0.03195294365286827,
+            0.023971782997250557,
+            -0.008724087849259377,
+            -0.01340806670486927,
+            0.019924599677324295,
+            -0.0022676968947052956,
+            0.007018893025815487,
+            0.04570771008729935,
+            -0.04033033177256584,
+            0.02163686975836754,
+            -0.025726506486535072,
+            -0.015339677222073078,
+            -0.005349075887352228,
+            -0.022471778094768524,
+            0.02806142158806324,
+            -0.007344366051256657,
+            0.017801949754357338,
+            0.017094401642680168,
+            0.01594817079603672,
+            0.013330236077308655,
+            -0.001894464367069304,
+            -0.022938761860132217,
+            0.0018360915128141642,
+            -0.01541043259203434,
+            0.02114158496260643,
+            0.0015230007702484727,
+            0.005469358991831541,
+            0.027594437822699547,
+            0.031160488724708557,
+            -0.023490650579333305,
+            0.009502392262220383,
+            0.02422650158405304,
+            0.06967949122190475,
+            0.015424583107233047,
+            -0.006435164716094732,
+            -0.0024640418123453856,
+            0.04143410921096802,
+            0.0015513027319684625,
+            0.0038349193055182695,
+            -0.06826439499855042,
+            -0.03022652305662632,
+            -0.0011365016689524055,
+            -0.02254253253340721,
+            0.02227366529405117,
+            0.01002597901970148,
+            -0.009750034660100937,
+            -0.022457627579569817,
+            -0.04516997188329697,
+            0.05190584436058998,
+            -0.00545874610543251,
+            -0.016259491443634033,
+            -0.005554265342652798,
+            -0.05340585112571716,
+            0.00039136342820711434,
+            0.015042506158351898,
+            -0.018481198698282242,
+            -0.03670767694711685,
+            0.036141637712717056,
+            0.0007398316520266235,
+            0.03741522878408432,
+            0.012459950521588326,
+            -0.01708024926483631,
+            -0.020094411447644234,
+            0.022259512916207314,
+            -0.011639192700386047,
+            -0.0038313816767185926,
+            -0.027339721098542213,
+            0.0065236082300543785,
+            -0.028103873133659363,
+            -0.0014566679019480944,
+            0.06730213016271591,
+            -0.04816998168826103,
+            0.02946236915886402,
+            -0.029660483822226524,
+            0.023660462349653244,
+            -0.008575502783060074,
+            0.0278916098177433,
+            -0.010238244198262691,
+            0.005840822588652372,
+            -0.013457595370709896,
+            -0.0010586712742224336,
+            -0.020745357498526573,
+            0.0034210027661174536,
+            -0.016259491443634033,
+            -0.027750099077820778,
+            -0.007054270710796118,
+            0.031273696571588516,
+            0.01698119193315506,
+            0.025655752047896385,
+            0.025429336354136467,
+            0.01246702577918768,
+            -0.04935866594314575,
+            0.02595292218029499,
+            0.0018926954362541437,
+            0.013514199294149876,
+            0.013181651011109352,
+            -0.0010418669553473592,
+            0.025599148124456406,
+            0.012049571610987186,
+            0.025061409920454025,
+            0.037443529814481735,
+            0.0027682881336659193,
+            0.04737752676010132,
+            -0.015693452209234238,
+            -0.005292471963912249,
+            0.028528403490781784,
+            0.000505455827806145,
+            0.02465103194117546,
+            -0.001686621573753655,
+            0.013330236077308655,
+            0.000776978034991771,
+            0.03834919258952141,
+            -0.006339645478874445,
+            0.029518973082304,
+            0.021608568727970123,
+            0.028316138312220573,
+            -0.014759486541152,
+            0.023122724145650864,
+            -0.019754787907004356,
+            -0.02292460948228836,
+            0.035547297447919846,
+            0.004082561936229467,
+            0.01886327564716339,
+            -0.028457649052143097,
+            -0.030679354444146156,
+            0.015452885068953037,
+            -0.01855195313692093,
+            0.016556663438677788,
+            0.03639635816216469,
+            0.0008273909334093332,
+            -0.0025719432160258293,
+            0.026292547583580017,
+            -0.04245298355817795,
+            0.012856177985668182,
+            -0.01532552670687437,
+            -0.02855670638382435,
+            -0.019528372213244438,
+            -0.00875946506857872,
+            -0.0018095583654940128,
+            -0.013691086322069168,
+            -0.0037570889107882977,
+            -0.007563706487417221,
+            -0.02459442801773548,
+            0.00935380719602108,
+            0.013627407141029835,
+            0.009849091991782188,
+            -0.018665160983800888,
+            0.0019952901639044285,
+            -0.008257105015218258,
+            -0.0160047747194767,
+            -0.03752843663096428,
+            0.015268922783434391,
+            -0.0024888061452656984,
+            0.009191070683300495,
+            0.04078316316008568,
+            0.028485950082540512,
+            -0.005055442452430725,
+            0.0329718142747879,
+            -0.033707667142152786,
+            0.017915157601237297,
+            -0.008320784196257591,
+            0.009191070683300495,
+            0.002989397384226322,
+            -0.025542544201016426,
+            0.027424626052379608,
+            -0.03792466223239899,
+            -0.038151077926158905,
+            0.022995365783572197,
+            0.011858533136546612,
+            0.0017096169758588076,
+            -0.019386861473321915,
+            -0.017717044800519943,
+            0.0008499440737068653,
+            0.002752368338406086,
+            -0.018834972754120827,
+            -0.01019579079002142,
+            0.02227366529405117,
+            -0.025500090792775154,
+            0.036085035651922226,
+            0.023391593247652054,
+            -0.03520767390727997,
+            -0.00608492735773325,
+            -0.01903308741748333,
+            0.059037946164608,
+            -0.0011506526498124003,
+            -0.010301923379302025,
+            0.022938761860132217,
+            -0.027268964797258377,
+            -0.004524780437350273,
+            0.022301966324448586,
+            0.01828308403491974,
+            -0.008278331719338894,
+            0.003983504604548216,
+            -0.022287815809249878,
+            -0.05281151086091995,
+            -0.010577867738902569,
+            -0.0003305583668407053,
+            0.054368119686841965,
+            -0.012367968447506428,
+            0.0020695829298347235,
+            -0.006413938011974096,
+            0.004100250545889139,
+            0.03704730048775673,
+            0.002140337834134698,
+            0.0025082635693252087,
+            -0.013698162510991096,
+            0.0371888130903244,
+            0.022316116839647293,
+            0.005883275531232357,
+            -0.012162779457867146,
+            0.005377377849072218,
+            -0.04899074137210846,
+            -0.0009118546731770039,
+            0.005741765722632408,
+            0.007386818993836641,
+            -0.025910470634698868,
+            0.017717044800519943,
+            0.009205221198499203,
+            -0.0024693484883755445,
+            -0.009792488068342209,
+            0.01173117384314537,
+            0.025429336354136467,
+            -0.019938750192523003,
+            -0.05700020492076874,
+            0.0034705311991274357,
+            0.012014193460345268,
+            -0.004054259974509478,
+            -0.019401011988520622,
+            0.006792477332055569,
+            0.0021934041287750006,
+            0.022698193788528442,
+            -0.011271266266703606,
+            -0.0029221803415566683,
+            0.040585052222013474,
+            0.009516543708741665,
+            -0.0016813150141388178,
+            -0.017717044800519943,
+            -0.011794853024184704,
+            0.018070818856358528,
+            0.004740582779049873,
+            0.0035289041697978973,
+            -0.01751893013715744,
+            -0.029264256358146667,
+            0.02800481766462326,
+            0.019967053085565567,
+            -0.048905834555625916,
+            -0.008200501091778278,
+            -0.019273653626441956,
+            0.0019599127117544413,
+            0.008695785887539387,
+            0.04613224044442177,
+            -0.00590803986415267,
+            0.008313708938658237,
+            -0.020179318264126778,
+            -0.05003791302442551,
+            -0.02708500251173973,
+            0.03707560524344444,
+            -0.028684064745903015,
+            -0.04276430234313011,
+            -0.010252394713461399,
+            0.016740625724196434,
+            -0.009403335861861706,
+            -0.018679311498999596,
+            -0.011738249100744724,
+            -0.04969828948378563,
+            -0.003497064346447587,
+            0.0337359681725502,
+            0.017264213413000107,
+            -0.008441068232059479,
+            -0.033877480775117874,
+            0.019641580060124397,
+            -0.007160402834415436,
+            -0.006474079564213753,
+            0.0007862646016292274,
+            0.008794843219220638,
+            0.02400008589029312,
+            0.011879759840667248,
+            -0.02763689123094082,
+            -0.03427370637655258,
+            -0.0024888061452656984,
+            0.014929298311471939,
+            -0.00878776703029871,
+            0.0245378240942955,
+            0.024679332971572876,
+            -0.022995365783572197,
+            -0.01654251106083393,
+            0.015750056132674217,
+            -0.05779265984892845,
+            -0.010287772864103317,
+            -0.029235953465104103,
+            -0.006183984223753214,
+            0.0009613831643946469,
+            0.0061096916906535625,
+            0.00012304731353651732,
+            0.03846240043640137,
+            0.008801918476819992,
+            0.006014172453433275,
+            0.009191070683300495,
+            0.01913214474916458,
+            0.028471799567341805,
+            0.009155693463981152,
+            -0.03517937287688255,
+            0.02850010246038437,
+            -0.017051948234438896,
+            0.007811348885297775,
+            -0.025754809379577637,
+            -0.013033065013587475,
+            0.00443633645772934,
+            0.02725481428205967,
+            -0.000503244751598686,
+            0.018849123269319534,
+            -0.021919889375567436,
+            0.037500135600566864,
+            -0.029915200546383858,
+            -0.05278320610523224,
+            -0.02005195803940296,
+            -0.008285406976938248,
+            0.015764206647872925,
+            0.017278363928198814,
+            -0.019924599677324295,
+            -0.0286557637155056,
+            -0.021750077605247498,
+            0.006470541935414076,
+            0.010542490519583225,
+            -0.021099131554365158,
+            -0.010627396404743195,
+            0.01502835564315319,
+            -0.00404364662244916,
+            -0.01886327564716339,
+            0.013684011064469814,
+            0.015693452209234238,
+            0.03169822692871094,
+            -0.03308502212166786,
+            0.004287751391530037,
+            -0.025967074558138847,
+            -0.04327373951673508,
+            0.004096712917089462,
+            0.03166992589831352,
+            0.027509532868862152,
+            -0.018636859953403473,
+            0.010754755698144436,
+            0.003990580327808857,
+            0.004234685096889734,
+            -0.0033803186379373074,
+            0.023603858426213264,
+            0.03291521221399307,
+            0.02763689123094082,
+            -0.011653343215584755,
+            0.004921008367091417,
+            -0.01559439580887556,
+            -0.0011108530452474952,
+            0.01865101046860218,
+            -0.017816102132201195,
+            0.033877480775117874,
+            -0.029405765235424042,
+            0.02104252763092518,
+            -0.035122767090797424,
+            0.006141531281173229,
+            -0.012099100276827812,
+            -0.00404364662244916,
+            0.04833979532122612,
+            -0.017646290361881256,
+            -0.0019546060357242823,
+            -0.008101443760097027,
+            -0.014575524255633354,
+            -0.02158026583492756,
+            0.008766540326178074,
+            -0.04033033177256584,
+            -0.012367968447506428,
+            -0.028429346159100533,
+            0.003424540627747774,
+            0.019910449162125587,
+            -0.06277380883693695,
+            -0.06237758323550224,
+            0.012275987304747105,
+            -0.04918885603547096,
+            -0.026419905945658684,
+            0.02963218092918396,
+            -0.002984090941026807,
+            0.0012780115939676762,
+            -0.0024003623984754086,
+            0.0068101659417152405,
+            -0.029207652434706688,
+            0.00425944896414876,
+            -0.000887974863871932,
+            0.03271709755063057,
+            -0.021056678146123886,
+            -0.008122670464217663,
+            0.004878554958850145,
+            -0.06860402226448059,
+            -0.028344441205263138,
+            0.011271266266703606,
+            -0.009459939785301685,
+            -0.0066615804098546505,
+            -0.017915157601237297,
+            -0.004383270163089037,
+            0.0008950503543019295,
+            0.0032830305863171816,
+            0.05801907554268837,
+            -0.01983969286084175,
+            -0.053773775696754456,
+            0.03633975237607956,
+            0.00887267291545868,
+            0.029603879898786545,
+            -0.025556694716215134,
+            -0.017533080652356148,
+            -0.01855195313692093,
+            -0.020504789426922798,
+            -0.04123599827289581,
+            0.00404364662244916,
+            -0.005918653216212988,
+            0.009212297387421131,
+            -0.0009693430620245636,
+            0.02763689123094082,
+            0.029235953465104103,
+            0.023108573630452156,
+            0.012750045396387577,
+            0.01079013291746378,
+            -0.022372720763087273,
+            -0.021594416350126266,
+            0.0012222920777276158,
+            0.04683978855609894,
+            0.024636881425976753,
+            -0.02675952948629856,
+            -0.011504758149385452,
+            0.03953787684440613,
+            0.0419718474149704,
+            -0.01974063739180565,
+            -0.02276895008981228,
+            0.00764861237257719,
+            0.03266049548983574,
+            -0.04477374628186226,
+            0.005932804197072983,
+            0.03410389646887779,
+            0.014405711553990841,
+            -0.005356151144951582,
+            -0.010294848121702671,
+            0.022570835426449776,
+            0.03152841329574585,
+            -0.008724087849259377,
+            0.01876421831548214,
+            0.016910437494516373,
+            0.029349161311984062,
+            -0.0067889392375946045,
+            0.0005766529939137399,
+            -0.024735936895012856,
+            0.055188875645399094,
+            -0.03059444949030876,
+            0.02909444272518158,
+            0.010316074825823307,
+            -0.00593634182587266,
+            0.039056744426488876,
+            -0.016641568392515182,
+            -0.0431322306394577,
+            0.01913214474916458,
+            -0.006979977712035179,
+            -0.0055613406002521515,
+            -0.03444352000951767,
+            -0.007592008449137211,
+            -0.007507102098315954,
+            -0.006714646704494953,
+            -0.022556684911251068,
+            -0.02606613002717495,
+            0.004415110219269991,
+            0.00973588414490223,
+            -0.006484692916274071,
+            0.00698705343529582,
+            0.02092931978404522,
+            -0.013507124036550522,
+            0.02525952458381653,
+            -0.046047333627939224,
+            -0.0008366775000467896,
+            -0.0038596836384385824,
+            -0.025712355971336365,
+            -0.0345001220703125,
+            -0.0005770951975136995,
+            -0.0024251264985650778,
+            -0.024622729048132896,
+            -0.00914154201745987,
+            -0.008752389810979366,
+            -0.0023119186516851187,
+            -0.004938696976751089,
+            -0.055188875645399094,
+            0.008695785887539387,
+            0.010846736840903759,
+            -0.002713453024625778,
+            0.0055507272481918335,
+            -0.010613245889544487,
+            -0.03039633482694626,
+            0.03039633482694626,
+            0.04129260033369064,
+            0.033707667142152786,
+            0.014334957115352154,
+            -0.0074646491557359695,
+            -0.017165156081318855,
+            0.006060163024812937,
+            0.021721776574850082,
+            0.00902833417057991,
+            0.007846726104617119,
+            0.03877372294664383,
+            0.00011331850691931322,
+            -0.008349086157977581,
+            -0.009764186106622219,
+            0.028740668669342995,
+            -0.001085204305127263,
+            0.023603858426213264,
+            0.050802066922187805,
+            0.030424637719988823,
+            0.020589696243405342,
+            -0.01681138016283512,
+            -0.02010856196284294,
+            -0.00851889792829752,
+            -0.02855670638382435,
+            0.0364246591925621,
+            0.012127402238547802,
+            -0.005232329946011305,
+            -0.011540135368704796,
+            0.019061388447880745,
+            -0.033339742571115494,
+            -0.0037358624394983053,
+            0.00048422932741232216,
+            0.013924578204751015,
+            -0.03379257395863533,
+            0.0002480846014805138,
+            0.031613320112228394,
+            0.014957600273191929,
+            -0.016995344310998917,
+            -0.0188774261623621,
+            0.010549565777182579,
+            0.020462337881326675,
+            0.018948180601000786,
+            0.00037743354914709926,
+            0.00897173024713993,
+            -0.00022121983056422323,
+            -0.0005319889169186354,
+            0.01590571738779545,
+            -0.007952858693897724,
+            0.036254845559597015,
+            -0.003525366308167577,
+            -0.01325240544974804,
+            -0.0037500134203583,
+            -0.021438755095005035,
+            -0.024467067793011665,
+            0.043556760996580124,
+            0.023971782997250557,
+            0.04661337286233902,
+            -0.005660397466272116,
+            -0.004245297983288765,
+            0.025825563818216324,
+            0.008221727795898914,
+            -0.03288691118359566,
+            0.000265110022155568,
+            0.005281858611851931,
+            -0.02892463095486164,
+            0.016641568392515182,
+            -0.02476423978805542,
+            -0.016896286979317665,
+            -0.006771250627934933,
+            -0.0015557248843833804,
+            -0.028188779950141907,
+            0.01703779585659504,
+            0.0031309074256569147,
+            -0.0027099153958261013,
+            -0.039764292538166046,
+            0.006880920846015215,
+            0.020419884473085403,
+            0.015311375260353088,
+            0.0037358624394983053,
+            0.011525984853506088,
+            -0.016613267362117767,
+            0.002126186853274703,
+            0.03379257395863533,
+            0.004878554958850145,
+            0.02005195803940296,
+            0.0027647505048662424,
+            0.023773670196533203,
+            0.03633975237607956,
+            -0.01681138016283512,
+            -0.02189158834517002,
+            0.01519816741347313,
+            -0.02233026921749115,
+            0.029943503439426422,
+            0.004889168310910463,
+            -0.012283062562346458,
+            0.022033097222447395,
+            -0.009827865287661552,
+            -0.0027346794959157705,
+            0.01659911684691906,
+            -0.022740647196769714,
+            -0.017717044800519943,
+            0.03183973580598831,
+            -0.039452970027923584,
+            0.012856177985668182,
+            -0.008547200821340084,
+            0.015863263979554176,
+            -0.023066120222210884,
+            0.011172209866344929,
+            0.007100261282175779,
+            -0.01573590561747551,
+            0.010797208175063133,
+            0.009693430736660957,
+            0.00545874610543251,
+            0.0018926954362541437,
+            0.02179253101348877,
+            -0.0006049550138413906,
+            -0.011257115751504898,
+            -0.018155725672841072,
+            -0.004694592207670212,
+            -0.002614396158605814,
+            -0.016556663438677788,
+            0.01865101046860218,
+            -0.041066184639930725,
+            -0.013280707411468029,
+            -0.008879749104380608,
+            -0.013988257385790348,
+            -0.010011828504502773,
+            0.024184048175811768,
+            0.025669902563095093,
+            -0.002361447084695101,
+            -0.031217092648148537,
+            0.01790100708603859,
+            0.024198200553655624,
+            0.003654494183138013,
+            -0.023646311834454536,
+            0.0302831269800663,
+            -0.008936353027820587,
+            0.014971751719713211,
+            0.026136886328458786,
+            0.014094389975070953,
+            0.004595535341650248,
+            -0.010500037111341953,
+            -0.02114158496260643,
+            0.01481609046459198,
+            0.010507113300263882,
+            0.033537857234478,
+            0.014745336025953293,
+            0.02071705460548401,
+            0.01594817079603672,
+            -0.023320838809013367,
+            -0.011214662343263626,
+            -0.003482913365587592,
+            -0.022514231503009796,
+            0.03093407303094864,
+            0.0006129149114713073,
+            0.02114158496260643,
+            0.026618018746376038,
+            -0.03319823369383812,
+            -0.01187268365174532,
+            0.032688796520233154,
+            0.02779255248606205,
+            0.05108508840203285,
+            0.025316128507256508,
+            0.015396281145513058,
+            0.027226513251662254,
+            -0.03585861995816231,
+            -0.036368053406476974,
+            0.001548649393953383,
+            0.04256619140505791,
+            -0.008426916785538197,
+            0.00967928022146225,
+            0.03900013864040375,
+            -0.006781863979995251,
+            -0.01611798256635666,
+            0.019457615911960602,
+            0.0026710000820457935,
+            0.002448122017085552,
+            -0.0033113325480371714,
+            -0.01335146278142929,
+            0.03175482898950577,
+            -0.0051474240608513355,
+            0.013570803217589855,
+            -0.019273653626441956,
+            0.027113303542137146,
+            0.0001799829478841275,
+            -0.003229964291676879,
+            0.028344441205263138,
+            -0.002660386962816119,
+            -0.051990751177072525,
+            0.024212351068854332,
+            0.019811391830444336,
+            -0.011561362072825432,
+            -0.0036332677118480206,
+            -0.003792466362938285,
+            0.04567940905690193,
+            0.034698236733675,
+            0.029009537771344185,
+            -0.0020501252729445696,
+            0.0168679840862751,
+            0.018792519345879555,
+            0.0053667644970119,
+            0.014278353191912174,
+            -0.0017025414854288101,
+            -0.007853801362216473,
+            0.021056678146123886,
+            -0.020787810906767845,
+            0.03226426616311073,
+            -0.016570813953876495,
+            -0.020122714340686798,
+            0.008738238364458084,
+            -0.011172209866344929,
+            -0.0009083169279620051,
+            -0.006311343517154455,
+            -0.0008813415770418942,
+            -0.0029664020985364914,
+            0.016938740387558937,
+            -0.004446949809789658,
+            0.04661337286233902,
+            -0.013931653462350368,
+            -0.008808993734419346,
+            0.006972902454435825,
+            -0.025981225073337555,
+            0.0013346156338229775,
+            0.0028726516757160425,
+            0.027000095695257187,
+            -0.01989629678428173,
+            -0.00970758218318224,
+            0.01606137864291668,
+            0.06039644405245781,
+            -0.008639181964099407,
+            -0.0026267783250659704,
+            0.027226513251662254,
+            -0.002787745790556073,
+            -0.03551899641752243,
+            0.012919857166707516,
+            -0.04494355618953705,
+            0.017589686438441277,
+            0.026349151507019997,
+            -0.04986810311675072,
+            -0.022089701145887375,
+            -0.02347649820148945,
+            -0.02043403498828411,
+            0.023434046655893326,
+            -0.009976450353860855,
+            0.013867974281311035,
+            -0.017801949754357338,
+            -0.009014182724058628,
+            0.020518941804766655,
+            0.009176919236779213,
+            0.03735862299799919,
+            -0.027551984414458275,
+            -0.016783079132437706,
+            -0.027820853516459465,
+            -0.04304732382297516,
+            0.009558996185660362,
+            -0.0278916098177433,
+            0.0027099153958261013,
+            -0.0016291331266984344,
+            -0.040302030742168427,
+            -0.037981268018484116,
+            0.005278320983052254,
+            -0.017985913902521133,
+            0.01516986545175314,
+            0.020915169268846512,
+            0.02731141820549965,
+            -0.0020253609400242567,
+            -0.030198222026228905,
+            0.029235953465104103,
+            -0.007344366051256657,
+            0.03223596513271332,
+            -0.00689507182687521,
+            0.005066055804491043,
+            -0.0070436573587358,
+            0.031160488724708557,
+            -0.023009516298770905,
+            0.018792519345879555,
+            -0.026801982894539833,
+            -0.004086099565029144,
+            -0.018990634009242058,
+            -0.01774534583091736,
+            0.004224071744829416,
+            0.004521242808550596,
+            7.888073741924018e-05,
+            0.0048750173300504684,
+            0.005777143407613039,
+            0.024806693196296692,
+            0.0021421066485345364,
+            0.02589631825685501,
+            -0.027877457439899445,
+            -0.025316128507256508,
+            0.01719345711171627,
+            -0.02077365852892399,
+            0.005678086541593075,
+            -0.006261814851313829,
+            -0.0302831269800663,
+            -0.01202834490686655,
+            0.023391593247652054,
+            0.02162271924316883,
+            -0.005490585695952177,
+            0.002644466934725642,
+            0.02206140011548996,
+            -0.026023678481578827,
+            0.00437973253428936,
+            -0.03781145438551903,
+            -0.018396291881799698,
+            0.029773691669106483,
+            -0.03195294365286827,
+            0.020745357498526573,
+            0.003185742534697056,
+            0.03483974561095238,
+            -0.017292514443397522,
+            -0.01947176828980446,
+            0.01654251106083393,
+            -0.01008258294314146,
+            0.004142703488469124,
+            0.01606137864291668,
+            0.01616043597459793,
+            -0.0015070808585733175,
+            0.05380208045244217,
+            0.0008167776977643371,
+            0.006909222807735205,
+            -0.049075648188591,
+            0.004496478475630283,
+            -0.044858649373054504,
+            0.0002768288250081241,
+            -0.013337312266230583,
+            -0.02514631673693657,
+            0.0014027173165231943,
+            0.001001182827167213,
+            0.001963450340554118,
+            -0.01913214474916458,
+            -0.01989629678428173,
+            0.011596739292144775,
+            -0.015863263979554176,
+            0.004928083624690771,
+            -0.06418890506029129,
+            -0.01855195313692093,
+            -0.004974074196070433,
+            0.0043337419629096985,
+            0.0007699025445617735,
+            0.03342464938759804,
+            -0.01478778850287199,
+            0.010117961093783379,
+            -0.01549533847719431,
+            -0.026363302022218704,
+            0.04010391607880592,
+            0.018750067800283432,
+            -0.023773670196533203,
+            0.024693485349416733,
+            0.012106175534427166,
+            0.03099067695438862,
+            -0.0318114347755909,
+            0.020349130034446716,
+            0.018523650243878365,
+            -0.0029823218937963247,
+            0.006551910191774368,
+            0.008575502783060074,
+            0.01832553744316101,
+            -0.015764206647872925,
+            -0.0003632825391832739,
+            0.011419852264225483,
+            0.010033054277300835,
+            0.031924642622470856,
+            0.01403071079403162,
+            -0.03183973580598831,
+            0.019047237932682037,
+            0.01616043597459793,
+            0.02298121340572834,
+            0.033820874989032745,
+            -0.027481229975819588,
+            0.038575608283281326,
+            -0.010386829264461994,
+            -0.024042539298534393,
+            0.020801961421966553,
+            -0.032688796520233154,
+            -0.006679269019514322,
+            0.030735958367586136,
+            0.04398128762841225,
+            -0.0015566092915832996,
+            -0.044632233679294586,
+            0.01535382866859436,
+            0.01316749956458807,
+            0.0027806703001260757,
+            0.028811423107981682,
+            0.011264191009104252,
+            0.021820832043886185,
+            0.025273675099015236,
+            0.011419852264225483,
+            -0.0037252490874379873,
+            0.024962352588772774,
+            -0.008101443760097027,
+            -0.004446949809789658,
+            -0.0007035697344690561,
+            0.03206615149974823,
+            0.014702882617712021,
+            -0.017603836953639984,
+            -0.036962397396564484,
+            -0.007620310410857201,
+            0.009452863596379757,
+            -0.007903330028057098,
+            -0.027127455919981003,
+            -0.03950957581400871,
+            -0.026787832379341125,
+            -0.005667473189532757,
+            0.01505665760487318,
+            -0.06724552065134048,
+            -0.007620310410857201,
+            -0.012233533896505833,
+            0.017717044800519943,
+            0.0022906921803951263,
+            0.0181132722645998,
+            0.001277127186767757,
+            -0.009403335861861706,
+            0.004209920763969421,
+            0.050802066922187805,
+            -0.03059444949030876,
+            0.014242975041270256,
+            0.011292492970824242,
+            0.0032105068676173687,
+            0.002787745790556073,
+            0.038688816130161285,
+            -0.010988246649503708,
+            -0.030424637719988823,
+            0.011575513519346714,
+            -0.017292514443397522,
+            0.003383856499567628,
+            0.05006621778011322,
+            -0.009481165558099747,
+            -0.027495380491018295,
+            0.020915169268846512,
+            -0.009148617275059223,
+            -0.015523640438914299,
+            -0.0027258351910859346,
+            -0.006035399157553911,
+            0.0006283925613388419,
+            -0.03832089155912399,
+            -0.015750056132674217,
+            0.03212275728583336,
+            0.008617955259978771,
+            0.014603826217353344,
+            0.025344429537653923,
+            0.010209942236542702,
+            -0.030481241643428802,
+            0.027481229975819588,
+            -0.0005678086308762431,
+            0.005221716593950987,
+            0.013323160819709301,
+            0.03424540534615517,
+            -0.014264201745390892,
+            -0.020264223217964172,
+            -0.040245428681373596,
+            -0.013853822834789753,
+            -0.0463869571685791,
+            0.001943992800079286,
+            0.006919836159795523,
+            -0.012919857166707516,
+            -0.04366996884346008,
+            -0.0329718142747879,
+            0.021127434447407722,
+            0.03178313374519348,
+            -0.017023645341396332,
+            0.0025294902734458447,
+            -0.008483520708978176,
+            -0.007882103323936462,
+            -0.0017520699184387922,
+            -0.013217028230428696,
+            -0.006378560326993465,
+            -0.006229975260794163,
+            0.015452885068953037,
+            -0.004652139265090227,
+            -0.0036934094969183207,
+            -0.0498398020863533,
+            -0.018778368830680847,
+            0.004082561936229467,
+            -0.008221727795898914,
+            0.007323139347136021,
+            -0.006955213379114866,
+            -0.002727604005485773,
+            -0.012474101036787033,
+            -0.03829259052872658,
+            0.024184048175811768,
+            0.017278363928198814,
+            0.0035501306410878897,
+            -0.008922201581299305,
+            -0.018367990851402283,
+            0.03294351324439049,
+            0.012523629702627659,
+            -0.017547233030200005,
+            -0.018580254167318344,
+            0.025981225073337555,
+            0.026957644149661064,
+            0.008122670464217663,
+            -0.006137993652373552,
+            0.029886899515986443,
+            0.005695775151252747,
+            -0.022938761860132217,
+            -0.003983504604548216,
+            -0.005055442452430725,
+            -0.0038844477385282516,
+            0.020561393350362778,
+            -0.010556641966104507,
+            0.0300001073628664,
+            0.038717120885849,
+            0.012870329432189465,
+            0.0039198254235088825,
+            -0.0033714743331074715,
+            -0.008922201581299305,
+            0.015976471826434135,
+            0.006481155287474394,
+            0.008525974117219448,
+            0.014759486541152,
+            -0.016358548775315285,
+            -0.020405733957886696,
+            0.006215824279934168,
+            -0.007107336539775133,
+            0.020122714340686798,
+            0.01708024926483631,
+            0.016429303213953972,
+            0.0007610581233166158,
+            -0.028287837281823158,
+            -0.0018024828750640154,
+            0.02411329373717308,
+            0.02010856196284294,
+            -0.004655676893889904,
+            -0.023264234885573387,
+            0.014398636296391487,
+            -0.015806660056114197,
+            -0.009969375096261501,
+            0.0033431723713874817,
+            0.006229975260794163,
+            -0.003083147807046771,
+            0.0571134127676487,
+            -0.020193468779325485,
+            -0.011419852264225483,
+            -0.010287772864103317,
+            -0.010867963545024395,
+            0.0012072566896677017,
+            -0.01828308403491974,
+            0.008964654989540577,
+            0.025613298639655113,
+            -0.009764186106622219,
+            0.03568880632519722,
+            0.0228114016354084,
+            0.006912760436534882,
+            0.005260631907731295,
+            -0.027070851996541023,
+            0.012304289266467094,
+            -0.016047226265072823,
+            0.005660397466272116,
+            0.018467046320438385,
+            0.017320817336440086,
+            0.0076132346875965595,
+            0.026306698098778725,
+            0.019386861473321915,
+            0.016684021800756454,
+            -0.022783100605010986,
+            0.025457637384533882,
+            -0.007832575589418411,
+            0.027877457439899445,
+            -0.01611798256635666,
+            -0.002253545681014657,
+            -0.012990612536668777,
+            0.008681634441018105,
+            0.002416282193735242,
+            -0.0019351483788341284,
+            0.00036129256477579474,
+            0.016783079132437706,
+            -0.014589674770832062,
+            0.01062032114714384,
+            0.013316085562109947,
+            -0.027919910848140717,
+            -0.010372678749263287,
+            0.016344398260116577,
+            -0.00668988237157464,
+            0.006877383217215538,
+            -0.011865608394145966,
+            -0.0017202302115038037,
+            0.011009473353624344,
+            -0.0436982698738575,
+            0.0104929618537426,
+            0.006279503460973501,
+            -0.015155714936554432,
+            0.014801939949393272,
+            -0.009799563325941563,
+            -0.008235878311097622,
+            -0.013592029921710491,
+            0.003792466362938285,
+            -0.00030225637601688504,
+            -0.0018290160223841667,
+            -0.00022995365725364536,
+            0.00389859895221889,
+            0.010047205723822117,
+            -0.0045990729704499245,
+            -0.010450509376823902,
+            -0.0028638073708862066,
+            -0.02265574224293232,
+            -0.0010047205723822117,
+            0.018735915422439575,
+            -0.004591997712850571,
+            0.0005222601466812193,
+            0.0145189194008708,
+            -0.005830209702253342,
+            0.010556641966104507,
+            -0.005221716593950987,
+            -0.008320784196257591,
+            -0.010768906213343143,
+            -0.033764272928237915,
+            0.0026710000820457935,
+            -0.015254771336913109,
+            0.008865597657859325,
+            -0.012544856406748295,
+            0.04027372971177101,
+            0.01022409275174141,
+            -0.00837738811969757,
+            -0.002796590095385909,
+            -0.0228114016354084,
+            -0.004475251771509647,
+            -0.016938740387558937,
+            -0.01184438169002533,
+            0.008342010900378227,
+            0.012629762291908264,
+            0.005614406894892454,
+            -0.018410442396998405,
+            0.003665107535198331,
+            0.0030955299735069275,
+            -0.005575491581112146,
+            -0.008370312862098217,
+            0.010853812098503113,
+            -0.0006248548161238432,
+            -0.01800006441771984,
+            -0.01957082375884056,
+            -0.027424626052379608,
+            0.0150000536814332,
+            0.014589674770832062,
+            -0.01011088490486145,
+            -0.022754797711968422,
+            -0.007471724878996611,
+            0.007107336539775133,
+            -0.02064630016684532,
+            0.0006779210525564849,
+            0.006891534198075533,
+            0.008717012591660023,
+            0.004117939155548811,
+            0.009636826813220978,
+            0.0007699025445617735,
+            0.0121698547154665,
+            0.018566103652119637,
+            0.03271709755063057,
+            -0.0396793857216835,
+            -0.009537769481539726,
+            0.0057028504088521,
+            -0.001465512323193252,
+            -0.013853822834789753,
+            0.009672204032540321,
+            0.0024375086650252342,
+            -0.0026426981203258038,
+            0.024424616247415543,
+            -0.008044839836657047,
+            -0.01505665760487318,
+            0.01184438169002533,
+            0.00887267291545868,
+            -0.019075538963079453,
+            0.02746707946062088,
+            0.006169833242893219,
+            -0.01011088490486145,
+            0.02259913645684719,
+            -0.012360893189907074,
+            0.012658064253628254,
+            -0.005522425286471844,
+            0.023660462349653244,
+            -0.019004784524440765,
+            0.0006306036957539618,
+            0.015665151178836823,
+            0.03883032873272896,
+            -0.00437973253428936,
+            -0.004567233379930258,
+            -0.03421710431575775,
+            0.005115584470331669,
+            0.028853876516222954,
+            0.02666047215461731,
+            -0.0012346742441877723,
+            -0.005596718285232782,
+            -0.0019864458590745926,
+            0.010768906213343143,
+            0.011589664034545422,
+            0.017561383545398712,
+            -0.029547275975346565,
+            0.00022011429246049374,
+            -0.016938740387558937,
+            0.025500090792775154,
+            0.014632128179073334,
+            0.006951675750315189,
+            0.04072656109929085,
+            0.0027789014857262373,
+            0.003841994795948267,
+            0.027452928945422173,
+            -0.00779719790443778,
+            0.003546592779457569,
+            -0.015566092915832996,
+            0.0003175129240844399,
+            0.022853855043649673,
+            0.0016370931407436728,
+            0.0020784272346645594,
+            0.013337312266230583,
+            0.029320860281586647,
+            -0.022471778094768524,
+            -0.023929331451654434,
+            0.003482913365587592,
+            0.007185167167335749,
+            -0.010570792481303215,
+            0.0038738346192985773,
+            0.006275965832173824,
+            -0.006007096730172634,
+            0.008900974877178669,
+            0.006332569755613804,
+            -0.0003166284877806902,
+            0.0039198254235088825,
+            -0.00308845448307693,
+            0.012962310574948788,
+            -0.02702839858829975,
+            0.036849189549684525,
+            0.004439874552190304,
+            0.010075507685542107,
+            0.015268922783434391,
+            0.001627364312298596,
+            0.004917470272630453,
+            0.005869124550372362,
+            0.010188715532422066,
+            0.018636859953403473,
+            -0.00956607237458229,
+            -0.027481229975819588,
+            -0.03166992589831352,
+            0.020363280549645424,
+            -0.01173117384314537,
+            0.0035872769076377153,
+            0.0074646491557359695,
+            -0.01828308403491974,
+            -0.007507102098315954,
+            -0.009976450353860855,
+            0.015764206647872925,
+            -0.03189634159207344,
+            0.012990612536668777,
+            -0.0043231286108493805,
+            -0.028669914230704308,
+            0.00851889792829752,
+            -0.010181640274822712,
+            -0.0045884596183896065,
+            -0.0035094465129077435,
+            -0.029264256358146667,
+            -0.007931631989777088,
+            0.03908504545688629,
+            -0.02141045406460762,
+            0.024184048175811768,
+            0.008257105015218258,
+            -0.00014449491573031992,
+            -0.0040365708991885185,
+            -0.013563727959990501,
+            0.005331386812031269,
+            -0.01778779923915863,
+            0.025457637384533882,
+            0.010294848121702671,
+            -0.013924578204751015,
+            0.0322076641023159,
+            -0.011179285123944283,
+            0.027438776567578316,
+            -0.018580254167318344,
+            0.036198243498802185,
+            0.007422196213155985,
+            0.00674648629501462,
+            0.01974063739180565,
+            -0.004025958012789488,
+            0.008228803053498268,
+            0.04559450224041939,
+            -0.018523650243878365,
+            0.02460857853293419,
+            0.020349130034446716,
+            -0.00022685811563860625,
+            0.017915157601237297,
+            0.0028390432707965374,
+            -0.023165177553892136,
+            0.009495317004621029,
+            -0.01135617308318615,
+            -0.009955224581062794,
+            0.01583496294915676,
+            -0.019316107034683228,
+            -0.018721764907240868,
+            -0.025613298639655113,
+            0.0029416377656161785,
+            -0.003350247861817479,
+            -0.0077618202194571495,
+            -0.0019616815261542797,
+            0.01454722136259079,
+            0.008724087849259377,
+            0.025669902563095093,
+            0.0008326975512318313,
+            0.011971740983426571,
+            0.01511326152831316,
+            -0.00035576484515331686,
+            0.02313687466084957,
+            -0.03229256719350815,
+            0.01246702577918768,
+            -0.004517704714089632,
+            -0.00509789539501071,
+            0.023858575150370598,
+            0.0003747802402358502,
+            -0.0004877670726273209,
+            -0.0024888061452656984,
+            0.0005496777012012899,
+            -0.014589674770832062,
+            -0.0057099261321127415,
+            0.000907432462554425,
+            -0.005175726022571325,
+            0.04299071803689003,
+            -0.00701535539701581,
+            -0.004772422835230827,
+            -0.039169952273368835,
+            -0.01918874867260456,
+            0.00623351288959384,
+            -0.01583496294915676,
+            -0.018778368830680847,
+            -0.006307805422693491,
+            0.005801907740533352,
+            -0.04893413558602333,
+            0.008342010900378227,
+            0.03336804360151291,
+            -0.012219383381307125,
+            0.00259493850171566,
+            -0.010981171391904354,
+            -0.006969364359974861,
+            -0.0029115669894963503,
+            -7.766463386360556e-05,
+            -0.002412744564935565,
+            -0.020830262452363968,
+            -0.005617944523692131,
+            0.003364398842677474,
+            -0.0008795727044343948,
+            -0.025047259405255318,
+            0.0006177793256938457,
+            0.018849123269319534,
+            -0.003979966975748539,
+            -0.0011152751976624131,
+            -0.001825478277169168,
+            0.006399787031114101,
+            -0.01508495956659317,
+            0.008009462617337704,
+            -0.0076344613917171955,
+            0.003841994795948267,
+            -0.0030495391692966223,
+            -0.002214630600064993,
+            0.02844349853694439,
+            0.021594416350126266,
+            -0.04808507859706879,
+            -0.01405901275575161,
+            0.023009516298770905,
+            0.007280686404556036,
+            -0.004740582779049873,
+            0.01178777776658535,
+            -0.031613320112228394,
+            -0.00028943203506059945,
+            -0.026787832379341125,
+            0.02200479619204998,
+            -0.009205221198499203,
+            -0.007386818993836641,
+            0.016967041417956352,
+            -0.026632171124219894,
+            0.016344398260116577,
+            0.00794578343629837,
+            -0.00030181417241692543,
+            -0.000493515923153609,
+            0.0063856360502541065,
+            0.0008972614887170494,
+            -0.01659911684691906,
+            0.006442239973694086,
+            -0.0048502529971301556,
+            0.007959933951497078,
+            -0.005087282508611679,
+            -0.05527378246188164,
+            -0.009933997876942158,
+            -0.005497660953551531,
+            -0.008936353027820587,
+            -0.012820800766348839,
+            0.008136821910738945,
+            0.02827368676662445,
+            0.01606137864291668,
+            -0.009863242506980896,
+            -4.8312376748071983e-05,
+            0.033000119030475616,
+            -0.0008300442132167518,
+            0.009955224581062794,
+            -0.011349096894264221,
+            0.00015709814033471048,
+            -0.0006911876262165606,
+            -0.014660430140793324,
+            -0.019061388447880745,
+            0.024042539298534393,
+            -0.028344441205263138,
+            0.02378782071173191,
+            -0.01843874529004097,
+            -0.0130825936794281,
+            0.01757553406059742,
+            -0.032575588673353195,
+            -0.005108508747071028,
+            0.007450498174875975,
+            0.0090566361322999,
+            0.021608568727970123,
+            0.0032794929575175047,
+            0.020589696243405342,
+            -0.03716050833463669,
+            -0.009134466759860516,
+            -0.010160413570702076,
+            -0.00794578343629837,
+            -0.010231168940663338,
+            -0.011596739292144775,
+            0.021863285452127457,
+            -0.015240620821714401,
+            -0.016952890902757645,
+            0.007054270710796118,
+            0.0030212372075766325,
+            -0.011313719674944878,
+            0.015750056132674217,
+            0.00017810352437663823,
+            0.004712280817329884,
+            0.013011839240789413,
+            -0.0008923970744945109,
+            0.04086806997656822,
+            0.0253019779920578,
+            -0.0030212372075766325,
+            -0.008568426594138145,
+            0.021438755095005035,
+            0.0004285097820684314,
+            0.0150000536814332,
+            0.009375033900141716,
+            0.023264234885573387,
+            -0.02092931978404522,
+            0.009997677057981491,
+            -0.005501199048012495,
+            0.014603826217353344,
+            -0.009495317004621029,
+            -0.015339677222073078,
+            0.00479718716815114,
+            -0.01577835902571678,
+            0.010153338313102722,
+            -0.010860888287425041,
+            0.008221727795898914,
+            0.01913214474916458,
+            0.002467579673975706,
+            -0.00941041111946106,
+            0.01146230474114418,
+            0.02075950801372528,
+            -0.025641601532697678,
+            0.008851447142660618,
+            -0.0031998935155570507,
+            -0.0021916350815445185,
+            0.021127434447407722,
+            0.010301923379302025,
+            -0.01594817079603672,
+            0.01573590561747551,
+            -0.008879749104380608,
+            -0.0015053119277581573,
+            -0.01882082223892212,
+            -0.03291521221399307,
+            -0.0015283073298633099,
+            0.012396270409226418,
+            0.004800724796950817,
+            0.0150000536814332,
+            0.011264191009104252,
+            -0.0008950503543019295,
+            -0.006601438857614994,
+            0.0008433107868768275,
+            -0.013485897332429886,
+            -0.004963461309671402,
+            0.010231168940663338,
+            0.017051948234438896,
+            -0.0037217114586383104,
+            0.015650998800992966,
+            0.0088160689920187,
+            0.006137993652373552,
+            -0.0010480580385774374,
+            0.0036138100549578667,
+            -0.002060738392174244,
+            -0.007783046457916498,
+            0.019938750192523003,
+            0.019075538963079453,
+            -0.018509499728679657,
+            -0.0024870370980352163,
+            -0.02141045406460762,
+            0.025882167741656303,
+            0.022188758477568626,
+            0.011165134608745575,
+            -0.04222656786441803,
+            0.00012050455552525818,
+            0.00752832880243659,
+            -0.0013744152383878827,
+            -0.031103884801268578,
+            0.008172199130058289,
+            0.005221716593950987,
+            0.004330204334110022,
+            0.020844414830207825,
+            -0.018424594774842262,
+            0.010853812098503113,
+            0.0070861103013157845,
+            -0.0005766529939137399,
+            0.01019579079002142,
+            -0.0073372903279960155,
+            0.008688710629940033,
+            0.038717120885849,
+            -0.018410442396998405,
+            0.005303084850311279,
+            0.0062370505183935165,
+            0.023320838809013367,
+            -0.011228813789784908,
+            -0.00031485961517319083,
+            0.00800238735973835,
+            0.021608568727970123,
+            -0.012934008613228798,
+            0.005883275531232357,
+            0.067075714468956,
+            -0.012912781909108162,
+            0.031613320112228394,
+            0.020533092319965363,
+            -0.004970536567270756,
+            -0.0104929618537426,
+            -0.012445799075067043,
+            -0.0181132722645998,
+            0.021283095702528954,
+            0.01175947580486536,
+            0.003419233951717615,
+            0.01941516436636448,
+            -0.01610383205115795,
+            0.006636816076934338,
+            0.017547233030200005,
+            0.012601460330188274,
+            -0.007379743270576,
+            -0.0036173479165881872,
+            0.019004784524440765,
+            0.018509499728679657,
+            -0.0030849166214466095,
+            0.00019424450874794275,
+            0.01508495956659317,
+            -0.022075550630688667,
+            -0.012530704960227013,
+            0.01644345559179783,
+            -0.0037358624394983053,
+            0.017773648723959923,
+            -0.02227366529405117,
+            0.024198200553655624,
+            -0.004531855694949627,
+            -0.017179306596517563,
+            0.004698129836469889,
+            -0.007514177821576595,
+            0.022033097222447395,
+            -0.001578720286488533,
+            0.004857328720390797,
+            0.000821199850179255,
+            -0.0018015984678640962,
+            0.01349297259002924,
+            0.002826661104336381,
+            -0.0005120891146361828,
+            0.010507113300263882,
+            -0.0009702275274321437,
+            -0.0013496510218828917,
+            0.024509521201252937,
+            1.9374700059415773e-05,
+            0.005989408120512962,
+            0.0032936439383774996,
+            -0.009261825121939182,
+            -0.004429261200129986,
+            -0.004443412180989981,
+            0.0011842612875625491,
+            0.003877372248098254,
+            -0.007029506377875805,
+            -0.02384442463517189,
+            0.022471778094768524,
+            -0.008540124632418156,
+            0.01175947580486536,
+            0.011349096894264221,
+            0.003665107535198331,
+            0.006824316922575235,
+            -0.009771261364221573,
+            -0.017320817336440086,
+            0.0006633278680965304,
+            -0.01617458648979664,
+            0.006912760436534882,
+            -0.001656550681218505,
+            0.0020235921256244183,
+            0.009445788338780403,
+            -0.002570174168795347,
+            0.0015123875346034765,
+            0.011306644417345524,
+            0.011497682891786098,
+            0.003254728624597192,
+            0.004577846731990576,
+            -0.018721764907240868,
+            -0.020915169268846512,
+            0.005288933869451284,
+            0.013924578204751015,
+            0.004273600410670042,
+            0.0015309606678783894,
+            0.017603836953639984,
+            -0.006166295614093542,
+            -0.022967062890529633,
+            -0.010117961093783379,
+            -0.01584911346435547,
+            0.039396367967128754,
+            0.01800006441771984,
+            -0.02525952458381653,
+            -0.000530220044311136,
+            -0.009728807955980301,
+            0.010450509376823902,
+            -0.00404364662244916,
+            0.014575524255633354,
+            0.005476434715092182,
+            -0.02426895499229431,
+            -0.016797229647636414,
+            -0.010662773624062538,
+            -0.005257094278931618,
+            0.020080260932445526,
+            0.027014248073101044,
+            -0.004956385586410761,
+            0.0061026159673929214,
+            0.019584976136684418,
+            -0.010938717983663082,
+            0.009849091991782188,
+            0.020957622677087784,
+            0.0031698227394372225,
+            -0.007570781745016575,
+            -0.004553082399070263,
+            -0.016514210030436516,
+            0.024467067793011665,
+            0.014250051230192184,
+            -0.005104971118271351,
+            -0.02465103194117546,
+            0.004110863897949457,
+            -0.02428310550749302,
+            -0.01616043597459793,
+            -0.007301913108676672,
+            -0.014143918640911579,
+            -0.004931621253490448,
+            0.0027594438288360834,
+            -0.004836102016270161,
+            0.011278342455625534,
+            -0.0031592093873769045,
+            -0.004531855694949627,
+            -0.0041851564310491085,
+            -0.025584997609257698,
+            -0.02524537406861782,
+            0.007691065315157175,
+            -0.003005317412316799,
+            0.005596718285232782,
+            0.000650945701636374,
+            -0.013400991447269917,
+            -0.011589664034545422,
+            0.00970758218318224,
+            -0.007988235913217068,
+            0.012813725508749485,
+            0.0031149876303970814,
+            -0.009183995425701141,
+            0.0017193458043038845,
+            -0.0012311364989727736,
+            -0.0059752571396529675,
+            0.019924599677324295,
+            0.00916984397917986,
+            0.011724098585546017,
+            -0.03118879161775112,
+            -0.0021633331198245287,
+            0.008349086157977581,
+            0.01828308403491974,
+            -0.02244347706437111,
+            -0.008313708938658237,
+            0.005179263651371002,
+            -0.011228813789784908,
+            -0.013507124036550522,
+            -0.028952933847904205,
+            -0.0030460015404969454,
+            8.175515540642664e-05,
+            -0.018636859953403473,
+            -0.029886899515986443,
+            0.0009038947173394263,
+            -0.007860877551138401,
+            0.016882136464118958,
+            -0.006643891800194979,
+            0.01790100708603859,
+            0.00870286114513874,
+            0.0036173479165881872,
+            0.011554286815226078,
+            0.0025082635693252087,
+            -0.013266556896269321,
+            0.008426916785538197,
+            -0.02671707607805729,
+            -0.01516986545175314,
+            0.009792488068342209,
+            0.014214673079550266,
+            -0.0026126273442059755,
+            5.074458022136241e-05,
+            0.0058372849598526955,
+            0.024254804477095604,
+            0.013584954664111137,
+            0.012474101036787033,
+            0.003042463678866625,
+            -0.005303084850311279,
+            0.005568416323512793,
+            0.0227123461663723,
+            0.008603804744780064,
+            0.004011806566268206,
+            0.007592008449137211,
+            0.00389859895221889,
+            0.0009932229295372963,
+            -0.005943417549133301,
+            -0.00843399204313755,
+            -0.021537812426686287,
+            -0.008908051066100597,
+            0.019938750192523003,
+            -0.010422207415103912,
+            -0.03005671128630638,
+            0.0006023016758263111,
+            -0.0032777239102870226,
+            0.007988235913217068,
+            0.02595292218029499,
+            -0.014172220602631569,
+            0.004970536567270756,
+            0.005536576267331839,
+            -0.007910405285656452,
+            -0.00614860700443387,
+            0.022967062890529633,
+            0.02639160305261612,
+            -0.0034404604230076075,
+            -0.04800017178058624,
+            -0.01671232469379902,
+            0.004305440001189709,
+            -0.007768895477056503,
+            -0.034047290682792664,
+            -0.015424583107233047,
+            0.002757675014436245,
+            -0.03206615149974823,
+            0.010068432427942753,
+            -0.012948159128427505,
+            -0.01214155275374651,
+            0.020264223217964172,
+            0.03223596513271332,
+            0.005154499784111977,
+            0.00014847487909719348,
+            -0.0214670579880476,
+            -0.0021686397958546877,
+            -0.021382151171565056,
+            0.010174565017223358,
+            -0.008801918476819992,
+            -0.010584943927824497,
+            0.0059611061587929726,
+            -0.012191081419587135,
+            0.013266556896269321,
+            0.010797208175063133,
+            0.024778390303254128,
+            0.01584911346435547,
+            -0.01876421831548214,
+            -0.0024746551644057035,
+            -0.01108730398118496,
+            -0.0016450530383735895,
+            -0.009820790030062199,
+            0.006360871717333794,
+            -0.0031592093873769045,
+            0.011264191009104252,
+            0.015184016898274422,
+            0.004432798828929663,
+            0.00046786724124103785,
+            0.038151077926158905,
+            -2.4487851987942122e-05,
+            -0.0003418349369894713,
+            0.006803090218454599,
+            -0.015750056132674217,
+            0.008837295696139336,
+            0.012191081419587135,
+            0.005051904823631048,
+            -0.007698140572756529,
+            0.006739411037415266,
+            0.03099067695438862,
+            -0.01516986545175314,
+            -0.009014182724058628,
+            -0.023504801094532013,
+            0.007549555506557226,
+            -0.0004656561359297484,
+            0.029971804469823837,
+            0.020533092319965363,
+            -0.027509532868862152,
+            -0.016018925234675407,
+            -0.02179253101348877,
+            0.01267929095774889,
+            -0.011497682891786098,
+            -0.003704022616147995,
+            0.012240610085427761,
+            0.02367461286485195,
+            0.0018838511314243078,
+            -0.014455240219831467,
+            -0.008950503543019295,
+            0.008943428285419941,
+            0.014617976732552052,
+            0.003497064346447587,
+            0.007570781745016575,
+            0.00743634719401598,
+            0.008886824361979961,
+            -0.014801939949393272,
+            -0.012254760600626469,
+            0.00767691433429718,
+            -0.010202866978943348,
+            0.01794346049427986,
+            -0.009403335861861706,
+            0.016768928617239,
+            -0.004645063541829586,
+            -0.011964665725827217,
+            -0.0320095494389534,
+            -0.002292460994794965,
+            0.005037753842771053,
+            0.004627374932169914,
+            0.0034705311991274357,
+            -0.0022217060904949903,
+            -0.013160424306988716,
+            0.004673365969210863,
+            0.005073131527751684,
+            1.875283123808913e-05,
+            -0.008313708938658237,
+            -0.00479718716815114,
+            0.004252373706549406,
+            -0.001031253719702363,
+            -0.0073372903279960155,
+            -0.0059858704917132854,
+            0.018481198698282242,
+            0.0066509670577943325,
+            0.00800238735973835,
+            0.03234917297959328,
+            0.02974538877606392,
+            -0.0016353242099285126,
+            -0.004606148693710566,
+            -0.00023304918431676924,
+            -0.01589156687259674,
+            -0.0057205394841730595,
+            -0.023320838809013367,
+            0.022259512916207314,
+            -0.009665128774940968,
+            0.021820832043886185,
+            0.019457615911960602,
+            0.0039304387755692005,
+            -0.008391539566218853,
+            -0.0011586126638576388,
+            -0.007308988366276026,
+            -0.003560743760317564,
+            0.016641568392515182,
+            0.006742948666214943,
+            0.01226891204714775,
+            0.00030933189555071294,
+            -0.028401045128703117,
+            0.026405755430459976,
+            0.000579748535528779,
+            0.0034015451092272997,
+            0.008908051066100597,
+            0.014129767194390297,
+            0.013209952972829342,
+            -0.01235381793230772,
+            0.01768874190747738,
+            -0.03280200436711311,
+            -0.0033484790474176407,
+            -0.016500059515237808,
+            0.002076658420264721,
+            -0.0018520113080739975,
+            -0.0008145665633492172,
+            -0.05354736000299454,
+            0.0003842879377771169,
+            0.005313698202371597,
+            -0.015339677222073078,
+            0.029349161311984062,
+            -0.020589696243405342,
+            -0.017165156081318855,
+            -0.032858606427907944,
+            -0.010238244198262691,
+            0.018099121749401093,
+            0.005617944523692131,
+            0.00035178486723452806,
+            -0.02660386823117733,
+            0.015750056132674217,
+            -0.0047476585023105145,
+            -0.00743634719401598,
+            -0.029858596622943878,
+            -0.010938717983663082,
+            0.009403335861861706,
+            0.004485865123569965,
+            -0.0130825936794281,
+            0.009898620657622814,
+            -0.002890340518206358,
+            0.010867963545024395,
+            -0.0138467475771904,
+            -0.00420284504070878,
+            0.0010020672343671322,
+            -0.005363226868212223,
+            0.012615610845386982,
+            -0.030537845566868782,
+            0.011391550302505493,
+            -0.01352127455174923,
+            0.0008548084879294038,
+            -0.017858553677797318,
+            -0.027707645669579506,
+            0.0020288988016545773,
+            -0.016132133081555367,
+            0.039990708231925964,
+            0.010188715532422066,
+            -0.01481609046459198,
+            0.016995344310998917,
+            -0.002080196049064398,
+            0.017377421259880066,
+            0.0014955832157284021,
+            0.017023645341396332,
+            -0.011971740983426571,
+            0.021056678146123886,
+            -0.0041745430789887905,
+            0.02156611531972885,
+            0.00600002147257328,
+            0.034047290682792664,
+            -0.02033497765660286,
+            0.0070825726725161076,
+            0.003792466362938285,
+            0.005798369646072388,
+            -0.0026816134341061115,
+            0.021424604579806328,
+            0.017985913902521133,
+            0.019698183983564377,
+            0.0037110981065779924,
+            -0.024084990844130516,
+            -0.006137993652373552,
+            0.004867942072451115,
+            0.023858575150370598,
+            0.011150983162224293,
+            -0.016415152698755264,
+            -0.021056678146123886,
+            0.015226469375193119,
+            -0.007860877551138401,
+            0.006771250627934933,
+            -0.00851889792829752,
+            0.00037544354563578963,
+            -0.013259481638669968,
+            0.012233533896505833,
+            -0.012523629702627659,
+            0.0032335021533071995,
+            0.01043635793030262,
+            -0.027806703001260757,
+            0.01511326152831316,
+            -0.00371817359700799,
+            -0.014313730411231518,
+            0.023320838809013367,
+            -0.015877414494752884,
+            -0.013988257385790348,
+            0.005501199048012495,
+            0.008610880002379417,
+            0.01746232621371746,
+            -0.002761212643235922,
+            0.002669231267645955,
+            -0.020504789426922798,
+            -0.012445799075067043,
+            0.004591997712850571,
+            0.02210385352373123,
+            -0.015820810571312904,
+            0.012431648559868336,
+            -0.018099121749401093,
+            0.01364863384515047,
+            0.016202887520194054,
+            0.007149789482355118,
+            0.00573115237057209,
+            -0.007726442534476519,
+            -0.0011471149045974016,
+            0.032151058316230774,
+            -0.0029876285698264837,
+            0.02704254910349846,
+            -0.014915147796273232,
+            0.009615600109100342,
+            0.00518987700343132,
+            -0.013089669868350029,
+            -0.014589674770832062,
+            0.011108530685305595,
+            0.02428310550749302,
+            -0.009530694223940372,
+            -0.023660462349653244,
+            0.0018484735628589988,
+            0.014264201745390892,
+            0.010337301529943943,
+            0.009403335861861706,
+            -0.001997058978304267,
+            -0.0037358624394983053,
+            0.01484439242631197,
+            -0.0032918748911470175,
+            -0.0286557637155056,
+            -0.01633024774491787,
+            -0.004400959238409996,
+            -0.017490629106760025,
+            0.020745357498526573,
+            0.02367461286485195,
+            0.010507113300263882,
+            0.0037570889107882977,
+            0.005540114361792803,
+            0.018297234550118446,
+            0.01843874529004097,
+            0.013330236077308655,
+            0.009304278530180454,
+            0.012700516730546951,
+            -0.004319590982049704,
+            0.02800481766462326,
+            -0.010860888287425041,
+            -0.01046465989202261,
+            -0.015707602724432945,
+            0.0016636261716485023,
+            -0.005193414632230997,
+            -0.012431648559868336,
+            -0.0069481381215155125,
+            0.004369119182229042,
+            0.013330236077308655,
+            -0.012240610085427761,
+            -0.007149789482355118,
+            0.01016748882830143,
+            0.011971740983426571,
+            0.01403071079403162,
+            0.01719345711171627,
+            -0.01633024774491787,
+            -0.011278342455625534,
+            0.0009834941010922194,
+            -0.02844349853694439,
+            -0.0160047747194767,
+            -0.01767459139227867,
+            0.009580222889780998,
+            -0.014469391666352749,
+            0.03724541515111923,
+            0.028471799567341805,
+            0.0073514413088560104,
+            0.031613320112228394,
+            -0.028401045128703117,
+            0.00970758218318224,
+            -0.014398636296391487,
+            0.007132100872695446,
+            -0.004188694059848785,
+            0.003902136581018567,
+            0.002448122017085552,
+            0.020419884473085403,
+            0.00791748147457838,
+            0.014264201745390892,
+            0.008150972425937653,
+            -0.021934039890766144,
+            -0.00662620272487402,
+            -0.012559006921947002,
+            -0.016401002183556557,
+            -0.02595292218029499,
+            0.013719388283789158,
+            0.005087282508611679,
+            -0.003074303502216935,
+            -0.0002704166399780661,
+            0.006566061172634363,
+            -0.0013195802457630634,
+            0.010443433187901974,
+            -0.027608588337898254,
+            -0.01311089564114809,
+            0.0069269114173948765,
+            0.01449061743915081,
+            -0.02114158496260643,
+            0.0014637433923780918,
+            0.008900974877178669,
+            -0.004602610599249601,
+            0.035660505294799805,
+            -0.014674580655992031,
+            -0.008610880002379417,
+            0.0027806703001260757,
+            0.005062518175691366,
+            -0.017660440877079964,
+            -0.02319347858428955,
+            0.0070648835971951485,
+            0.011575513519346714,
+            0.010867963545024395,
+            -0.028570856899023056,
+            0.02087271586060524,
+            -0.0054339817725121975,
+            0.03832089155912399,
+            0.00401534466072917,
+            -0.013337312266230583,
+            0.003962278366088867,
+            0.0011860302183777094,
+            -0.012360893189907074,
+            0.002060738392174244,
+            -0.000340508297085762,
+            0.003700484987348318,
+            0.011228813789784908,
+            -0.014801939949393272,
+            -0.016967041417956352,
+            0.016910437494516373,
+            -0.011476456187665462,
+            -0.0035395172890275717,
+            -0.00943871308118105,
+            -0.016669871285557747,
+            0.007308988366276026,
+            0.025047259405255318,
+            0.0041745430789887905,
+            0.001242634141817689,
+            -0.011632117442786694,
+            -0.0038596836384385824,
+            -0.008575502783060074,
+            0.0011718792375177145,
+            -0.0207029040902853,
+            0.027396325021982193,
+            -0.01801421493291855,
+            0.002410975517705083,
+            0.021099131554365158,
+            0.008709936402738094,
+            -0.003944589756429195,
+            0.016316095367074013,
+            0.026363302022218704,
+            0.01205664686858654,
+            0.0031539027113467455,
+            -0.00902833417057991,
+            1.648646139074117e-05,
+            0.014957600273191929,
+            0.0015123875346034765,
+            -0.0015663381200283766,
+            0.0032140444964170456,
+            -0.006583749782294035,
+            -0.001859086798503995,
+            -0.017561383545398712,
+            0.005564878229051828,
+            0.02125479280948639,
+            -0.004132090136408806,
+            0.020476488396525383,
+            0.015509488992393017,
+            0.011625041253864765,
+            0.010726453736424446,
+            -0.0012390963966026902,
+            -0.002150951186195016,
+            -0.0021668709814548492,
+            -0.007393894251435995,
+            -0.018297234550118446,
+            0.006470541935414076,
+            0.0034493047278374434,
+            0.01822648011147976,
+            0.020504789426922798,
+            -0.02308027073740959,
+            0.010146263055503368,
+            0.014985902234911919,
+            -0.013011839240789413,
+            0.0235755555331707,
+            -0.02006610855460167,
+            0.0024605041835457087,
+            0.02156611531972885,
+            0.013337312266230583,
+            -0.014271277002990246,
+            -0.009219372645020485,
+            0.004662752617150545,
+            -0.01543873455375433,
+            0.000872939417604357,
+            -0.01478778850287199,
+            -0.010783057659864426,
+            0.014533070847392082,
+            -0.01490099634975195,
+            0.01229721400886774,
+            -0.006969364359974861,
+            0.0070967236533761024,
+            -0.017165156081318855,
+            0.015014204196631908,
+            0.008632106706500053,
+            0.01757553406059742,
+            0.0021279556676745415,
+            -0.02518877014517784,
+            -0.006290116813033819,
+            0.021169887855648994,
+            0.009509467519819736,
+            -0.02497650496661663,
+            0.019613277167081833,
+            0.0020094411447644234,
+            -0.019528372213244438,
+            -0.022401023656129837,
+            -0.012686366215348244,
+            -0.005055442452430725,
+            -0.017264213413000107,
+            -0.0011276573641225696,
+            -0.024792540818452835,
+            0.003972891718149185,
+            0.009106164798140526,
+            -0.011978816241025925,
+            -0.019273653626441956,
+            -0.0002315014135092497,
+            -0.01040805596858263,
+            0.006219361908733845,
+            -0.01132787112146616,
+            -0.019853845238685608,
+            0.0070436573587358,
+            0.012127402238547802,
+            -0.0027647505048662424,
+            -0.005646246485412121,
+            -0.02562745101749897,
+            -0.006919836159795523,
+            0.020179318264126778,
+            -0.003994117956608534,
+            -0.004974074196070433,
+            -0.01703779585659504,
+            0.01108730398118496,
+            0.011165134608745575,
+            0.020901018753647804,
+            0.010931642726063728,
+            0.005674548447132111,
+            0.013634482398629189,
+            -0.0029416377656161785,
+            -0.007156865205615759,
+            -9.645892532716971e-06,
+            0.004284213297069073,
+            0.0069233737885952,
+            0.0016636261716485023,
+            0.020235922187566757,
+            -0.01692458800971508,
+            0.018934030085802078,
+            -0.012290137819945812,
+            0.008794843219220638,
+            -0.01855195313692093,
+            -0.0003155229496769607,
+            -0.01067692507058382,
+            0.00015002264990471303,
+            -0.0038455326575785875,
+            -0.009544845670461655,
+            1.9595809135353193e-05,
+            0.00143986358307302,
+            0.008511822670698166,
+            -0.016783079132437706,
+            0.009615600109100342,
+            -0.010839661583304405,
+            -0.004308977629989386,
+            -0.019542522728443146,
+            0.017448175698518753,
+            0.013344387523829937,
+            0.007308988366276026,
+            -0.008278331719338894,
+            -0.006300730165094137,
+            -0.02282555401325226,
+            0.008582578040659428,
+            -0.0014637433923780918,
+            0.002214630600064993,
+            -0.028301987797021866,
+            -0.004358506295830011,
+            0.0024587351363152266,
+            0.020306676626205444,
+            0.0033661676570773125,
+            -0.010712302289903164,
+            0.008978805504739285,
+            -0.023377442732453346,
+            -0.0030159305315464735,
+            -0.023349139839410782,
+            -0.004075486212968826,
+            -0.005112046375870705,
+            0.010103809647262096,
+            0.008101443760097027,
+            0.01549533847719431,
+            -0.009898620657622814,
+            0.01478778850287199,
+            -0.004793649073690176,
+            0.012049571610987186,
+            -0.0012152165872976184,
+            0.01337976474314928,
+            -0.0003111007681582123,
+            -0.01505665760487318,
+            0.02158026583492756,
+            -0.008738238364458084,
+            0.001741456682793796,
+            -0.0052853962406516075,
+            0.0027346794959157705,
+            -0.017702894285321236,
+            -0.004553082399070263,
+            -0.014455240219831467,
+            0.01957082375884056,
+            -0.022584985941648483,
+            0.004153316840529442,
+            -0.005055442452430725,
+            -0.016075529158115387,
+            0.029434068128466606,
+            0.022896308451890945,
+            -0.0035483618266880512,
+            0.015070809051394463,
+            0.0017989451298490167,
+            0.0015061964513733983,
+            0.016202887520194054,
+            -0.015806660056114197,
+            -0.013025989755988121,
+            0.035066161304712296,
+            0.0176321379840374,
+            0.0004313841927796602,
+            -0.0023260696325451136,
+            -0.0037323245778679848,
+            0.0032193511724472046,
+            0.008023613132536411,
+            -0.010450509376823902,
+            -0.01141277700662613,
+            -0.017051948234438896,
+            0.009997677057981491,
+            -0.006155682262033224,
+            0.004446949809789658,
+            -0.00025891896802932024,
+            -7.617214578203857e-05,
+            -0.01516986545175314,
+            0.022584985941648483,
+            -0.017717044800519943,
+            -0.00602832343429327,
+            0.004687516950070858,
+            0.0207029040902853,
+            -0.02541518583893776,
+            0.01616043597459793,
+            -0.007107336539775133,
+            -0.013429293408989906,
+            0.012573158368468285,
+            -0.015127412974834442,
+            -0.0107335289940238,
+            -0.003175129182636738,
+            -0.008009462617337704,
+            -0.022754797711968422,
+            0.017929309979081154,
+            -0.013627407141029835,
+            0.015650998800992966,
+            -0.00287972716614604,
+            0.000562502013053745,
+            0.002057200763374567,
+            0.0032582662533968687,
+            0.02250008098781109,
+            0.030735958367586136,
+            0.010415131226181984,
+            -0.005465821363031864,
+            -0.021000074222683907,
+            0.012983537279069424,
+            0.0020589695777744055,
+            0.013202877715229988,
+            -0.002253545681014657,
+            -0.002135031158104539,
+            -0.016613267362117767,
+            -0.023122724145650864,
+            -0.0013010069960728288,
+            -0.01761798746883869,
+            -0.02552839368581772,
+            -0.01774534583091736,
+            -0.007047194987535477,
+            0.011349096894264221,
+            0.020405733957886696,
+            0.0009755341452546418,
+            -0.005384453106671572,
+            0.01559439580887556,
+            -0.006941062398254871,
+            0.002080196049064398,
+            -0.006594363134354353,
+            -0.002032436430454254,
+            -0.008200501091778278,
+            -0.021863285452127457,
+            0.029660483822226524,
+            -0.010103809647262096,
+            0.01141277700662613,
+            -0.005472897086292505,
+            -0.00965097825974226,
+            0.005575491581112146,
+            0.00900003220885992,
+            0.024849144741892815,
+            0.006452853325754404,
+            0.03141520544886589,
+            0.020151015371084213,
+            -0.018891576677560806,
+            -0.0006071660900488496,
+            0.022684043273329735,
+            -0.0037570889107882977,
+            -0.004889168310910463,
+            -0.0031786670442670584,
+            -0.017985913902521133,
+            -0.012007118202745914,
+            0.007839650847017765,
+            -0.013471745885908604,
+            -0.006636816076934338,
+            -0.00837738811969757,
+            0.018070818856358528,
+            -0.017773648723959923,
+            0.025698205456137657,
+            -0.006488230545073748,
+            0.005154499784111977,
+            0.031924642622470856,
+            0.014363259077072144,
+            0.019698183983564377,
+            -0.011794853024184704,
+            -0.031075581908226013,
+            -0.00683846790343523,
+            -0.007882103323936462,
+            0.009558996185660362,
+            -0.01062032114714384,
+            0.005204027984291315,
+            -0.02006610855460167,
+            -0.018948180601000786,
+            0.012863253243267536,
+            -0.017476476728916168,
+            0.013924578204751015,
+            -0.009955224581062794,
+            0.027226513251662254,
+            -0.001228483160957694,
+            0.0121698547154665,
+            -0.002614396158605814,
+            -0.017801949754357338,
+            0.01205664686858654,
+            0.004089637193828821,
+            -0.023943481966853142,
+            0.010337301529943943,
+            -0.001864393474534154,
+            -0.014037786051630974,
+            -0.018410442396998405,
+            -0.001228483160957694,
+            -0.005200490355491638,
+            -0.009658053517341614,
+            0.025457637384533882,
+            0.014101465232670307,
+            0.016231190413236618,
+            0.0035359796602278948,
+            0.0016335553955286741,
+            -0.011073152534663677,
+            0.009573147632181644,
+            -0.006803090218454599,
+            0.000612030504271388,
+            -0.0034404604230076075,
+            0.011893910355865955,
+            -0.004252373706549406,
+            -0.01543873455375433,
+            0.002324300818145275,
+            0.008561351336538792,
+            0.0143278818577528,
+            5.947839599684812e-05,
+            0.009587298147380352,
+            0.0009790719486773014,
+            -0.009764186106622219,
+            -0.00046388726332224905,
+            -0.0060884649865329266,
+            0.0069375247694551945,
+            0.02400008589029312,
+            -0.012516554445028305,
+            -0.01681138016283512,
+            0.014504768885672092,
+            0.004033033270388842,
+            -0.017589686438441277,
+            0.018367990851402283,
+            0.004103788174688816,
+            -0.022358570247888565,
+            0.0035412863362580538,
+            0.010662773624062538,
+            -0.009523618966341019,
+            -0.014469391666352749,
+            -0.0010339069413021207,
+            -0.015509488992393017,
+            0.03130199760198593,
+            0.00025073791039176285,
+            0.016910437494516373,
+            -0.01062032114714384,
+            -0.0025967073161154985,
+            -0.013309009373188019,
+            0.013103820383548737,
+            -0.015750056132674217,
+            -0.007273610681295395,
+            0.00035620704875327647,
+            -0.02298121340572834,
+            0.01741987280547619,
+            0.0013425755314528942,
+            0.00794578343629837,
+            0.003436922561377287,
+            -0.010556641966104507,
+            0.009820790030062199,
+            0.0018237093463540077,
+            0.006668656133115292,
+            -0.014915147796273232,
+            0.023618008941411972,
+            0.006413938011974096,
+            0.011235889047384262,
+            -0.01352127455174923,
+            -0.01546703651547432,
+            0.002253545681014657,
+            0.04134920611977577,
+            -0.01306136790663004,
+            0.0002593612007331103,
+            0.01562269777059555,
+            7.081024523358792e-05,
+            0.004156854469329119,
+            -0.0023755980655550957,
+            -0.0034493047278374434,
+            -0.034585028886795044,
+            0.005370302125811577,
+            -0.0006328147719614208,
+            0.008073141798377037,
+            -0.008992956951260567,
+            -0.005430444143712521,
+            -0.0040684109553694725,
+            -0.008625030517578125,
+            -0.00314859626814723,
+            -0.01478778850287199,
+            0.008363237604498863,
+            -0.014533070847392082,
+            -0.028910480439662933,
+            0.03166992589831352,
+            -0.006887996103614569,
+            -0.00533846253529191,
+            -0.021396303549408913,
+            0.007570781745016575,
+            -0.01370523776859045,
+            0.002663924591615796,
+            0.010967019945383072,
+            0.008865597657859325,
+            0.008738238364458084,
+            0.002762981690466404,
+            0.0009516543359495699,
+            -0.004255911335349083,
+            0.010273621417582035,
+            -0.00835616234689951,
+            -0.0005943417199887335,
+            -0.0014186371117830276,
+            -0.02725481428205967,
+            -0.01405901275575161,
+            -0.010592019185423851,
+            -0.007811348885297775,
+            -0.00435143057256937,
+            0.012608535587787628,
+            0.007174553815275431,
+            -0.0017485321732237935,
+            -0.022938761860132217,
+            -0.014745336025953293,
+            -0.0034103896468877792,
+            0.008349086157977581,
+            -0.01838214136660099,
+            0.01064862310886383,
+            0.0016830838285386562,
+            0.009976450353860855,
+            0.014292503707110882,
+            0.00302831269800663,
+            0.0016742395237088203,
+            0.003458149265497923,
+            0.014886845834553242,
+            0.012601460330188274,
+            -0.005801907740533352,
+            -0.021382151171565056,
+            -0.0024605041835457087,
+            -0.007068421691656113,
+            -0.0034687623847275972,
+            -0.012332591228187084,
+            0.03039633482694626,
+            0.0029805530793964863,
+            0.0176321379840374,
+            0.007832575589418411,
+            0.009573147632181644,
+            0.010507113300263882,
+            0.029377464205026627,
+            0.006049550138413906,
+            -0.0029929352458566427,
+            -0.008603804744780064,
+            -0.02141045406460762,
+            -0.013146273791790009,
+            0.009898620657622814,
+            0.013294858857989311,
+            0.03597182780504227,
+            -0.034896351397037506,
+            0.012162779457867146,
+            -0.011101454496383667,
+            -0.008264180272817612,
+            -0.008533049374818802,
+            0.0044717141427099705,
+            0.010068432427942753,
+            0.014370334334671497,
+            -0.0023402206134051085,
+            -0.01175947580486536,
+            0.009070787578821182,
+            0.023547254502773285,
+            -0.012983537279069424,
+            0.03016991913318634,
+            0.027325568720698357,
+            0.023066120222210884,
+            0.01761798746883869,
+            -0.012537780217826366,
+            0.026462359353899956,
+            -6.484030018327758e-05,
+            0.0006261814851313829,
+            -0.018084971234202385,
+            -0.00587973790243268,
+            -0.004294826649129391,
+            -0.004362043924629688,
+            0.013316085562109947,
+            -0.0025471788831055164,
+            -0.0008468485320918262,
+            0.015099111013114452,
+            -0.0037323245778679848,
+            -3.927453508367762e-05,
+            -0.0059752571396529675,
+            0.015127412974834442,
+            0.0002699744363781065,
+            0.019457615911960602,
+            -0.04270770028233528,
+            0.019726485013961792,
+            -0.021396303549408913,
+            -0.003102605463936925,
+            -0.0020890405867248774,
+            0.009523618966341019,
+            0.00764861237257719,
+            -0.007142714224755764,
+            -0.014334957115352154,
+            0.004524780437350273,
+            0.014066088013350964,
+            -0.0006726144347339869,
+            0.005080206785351038,
+            -0.0027541371528059244,
+            0.0017228835495188832,
+            -0.010089659132063389,
+            -0.005862049292773008,
+            5.842812606715597e-05,
+            -0.010570792481303215,
+            -0.008646257221698761,
+            -0.03232087194919586,
+            0.01062032114714384,
+            0.010209942236542702,
+            0.007294837385416031,
+            0.004252373706549406,
+            -0.022018946707248688,
+            -0.012318440712988377,
+            0.012948159128427505,
+            -0.01132787112146616,
+            -0.002570174168795347,
+            -0.0020005968399345875,
+            0.005387990735471249,
+            0.0004705205501522869,
+            0.010160413570702076,
+            -0.003472300246357918,
+            0.010938717983663082,
+            -0.014603826217353344,
+            0.018084971234202385,
+            0.03229256719350815,
+            -0.00533846253529191,
+            -0.019967053085565567,
+            0.000907432462554425,
+            -0.009077862836420536,
+            0.003813692834228277,
+            0.007641536649316549,
+            -0.021962342783808708,
+            -0.02173592709004879,
+            0.005087282508611679,
+            -0.013153349049389362,
+            -0.007783046457916498,
+            0.014830241911113262,
+            0.005221716593950987,
+            0.010839661583304405,
+            0.01751893013715744,
+            0.010889190249145031,
+            0.00840569008141756,
+            -0.026533113792538643,
+            0.0006734988419339061,
+            -0.012686366215348244,
+            -0.00590803986415267,
+            0.002239394700154662,
+            -0.0017591454088687897,
+            0.004591997712850571,
+            -0.00010115749319083989,
+            -0.011200511828064919,
+            -0.0007057808106765151,
+            0.010903340764343739,
+            0.027919910848140717,
+            -0.004146241117268801,
+            -0.0022570835426449776,
+            -0.014257126487791538,
+            0.0010100272484123707,
+            0.036481261253356934,
+            -0.012275987304747105,
+            -0.017434025183320045,
+            0.01757553406059742,
+            0.027452928945422173,
+            0.005954030901193619,
+            -0.0032529598101973534,
+            -0.0027364485431462526,
+            0.00039600671152584255,
+            0.011978816241025925,
+            0.007719367276877165,
+            -0.002692226553335786,
+            -0.002131493529304862,
+            0.004761809483170509,
+            0.035773713141679764,
+            -0.0015203474322333932,
+            -0.010330225341022015,
+            -0.024240652099251747,
+            0.010974096134305,
+            0.00277182599529624,
+            0.004754733759909868,
+            0.002313687466084957,
+            0.023504801094532013,
+            0.019655730575323105,
+            0.014858543872833252,
+            0.00470520555973053,
+            -0.009771261364221573,
+            0.03005671128630638,
+            -0.008172199130058289,
+            -0.004542469047009945,
+            -0.02838689461350441,
+            -0.005087282508611679,
+            -0.015650998800992966,
+            0.008929276838898659,
+            -0.0002221042668679729,
+            -0.005016527138650417,
+            -0.013422218151390553,
+            0.019726485013961792,
+            0.01064862310886383,
+            -0.011073152534663677,
+            0.012608535587787628,
+            -0.027070851996541023,
+            0.02346234768629074,
+            -0.03826428949832916,
+            -0.0071993181481957436,
+            -0.006421013735234737,
+            -0.010514188557863235,
+            -0.02627839520573616,
+            -0.0030353881884366274,
+            0.007372668012976646,
+            0.014250051230192184,
+            0.0022570835426449776,
+            0.0022818478755652905,
+            0.02282555401325226,
+            0.0016415152931585908,
+            -0.01508495956659317,
+            0.017985913902521133,
+            0.03492465242743492,
+            0.012926933355629444,
+            0.00155837822239846,
+            0.015990622341632843,
+            0.0027789014857262373,
+            0.0021633331198245287,
+            0.00788917951285839,
+            -0.0034493047278374434,
+            -0.007938707247376442,
+            0.008582578040659428,
+            0.006990591064095497,
+            -0.02763689123094082,
+            0.003824306186288595,
+            -0.00010408719390397891,
+            -0.009898620657622814,
+            -0.022429324686527252,
+            0.011568437330424786,
+            0.01913214474916458,
+            -0.004224071744829416,
+            -0.006771250627934933,
+            -0.0043337419629096985,
+            0.0006836698739789426,
+            0.006785401608794928,
+            -0.002412744564935565,
+            -0.000515184598043561,
+            -0.0015176940942183137,
+            -0.02368876338005066,
+            -0.010316074825823307,
+            0.013570803217589855,
+            -0.005175726022571325,
+            0.010358527302742004,
+            0.005242943298071623,
+            -0.018792519345879555,
+            -0.016938740387558937,
+            0.006852618884295225,
+            -0.016188737004995346,
+            0.01364863384515047,
+            0.00027948213391937315,
+            -0.0023172253277152777,
+            0.024962352588772774,
+            0.03594352304935455,
+            -0.013188726268708706,
+            0.005119122099131346,
+            -0.019386861473321915,
+            -0.0245378240942955,
+            -0.019004784524440765,
+            0.022613288834691048,
+            0.008306633681058884,
+            -0.018792519345879555,
+            0.012304289266467094,
+            -0.002752368338406086,
+            -0.008865597657859325,
+            -0.0066721937619149685,
+            -0.006845543161034584,
+            -0.008547200821340084,
+            0.020533092319965363,
+            -0.004935159347951412,
+            0.0038950610905885696,
+            0.019047237932682037,
+            0.04075486212968826,
+            -0.008292482234537601,
+            -0.01349297259002924,
+            0.0054339817725121975,
+            0.004878554958850145,
+            0.0018626245437189937,
+            -0.004814875777810812,
+            -0.008016537874937057,
+            -0.0185944065451622,
+            -0.01740572229027748,
+            -0.009615600109100342,
+            -0.0020041344687342644,
+            0.020122714340686798,
+            -0.007768895477056503,
+            -0.01354957651346922,
+            -0.0245378240942955,
+            -0.028188779950141907,
+            -0.006580212153494358,
+            0.02319347858428955,
+            0.017179306596517563,
+            -0.00013200224202591926,
+            -4.3337418901501223e-05,
+            -0.012042495422065258,
+            -0.01968403346836567,
+            0.015509488992393017,
+            0.00037876018905080855,
+            -0.01773119531571865,
+            -0.0065448349341750145,
+            -0.015919867902994156,
+            0.005787756759673357,
+            -0.0051368107087910175,
+            0.004574308637529612,
+            0.05057565122842789,
+            0.0005271245026960969,
+            0.008441068232059479,
+            -0.026575567200779915,
+            0.009106164798140526,
+            -0.01583496294915676,
+            -0.017377421259880066,
+            -0.00012702727690339088,
+            -0.022301966324448586,
+            -0.0030955299735069275,
+            -0.018792519345879555,
+            -0.007386818993836641,
+            0.008568426594138145,
+            0.009799563325941563,
+            -0.013896276243031025,
+            -0.013669860549271107,
+            -0.024693485349416733,
+            -0.0035872769076377153,
+            -0.001539805089123547,
+            0.009311353787779808,
+            0.007825499400496483,
+            -0.03693409264087677,
+            0.002536565763875842,
+            -0.0009507699287496507,
+            0.0005779796629212797,
+            0.01208494883030653,
+            -0.008667483925819397,
+            -0.003063690150156617,
+            -0.0214670579880476,
+            0.006452853325754404,
+            0.007103798910975456,
+            -0.013500047847628593,
+            0.012375044636428356,
+            -0.012863253243267536,
+            -0.007669838611036539,
+            -0.0005426021525636315,
+            0.03379257395863533,
+            0.011794853024184704,
+            -0.0025153392925858498,
+            -0.00654129683971405,
+            -0.011150983162224293,
+            0.02654726430773735,
+            -0.011349096894264221,
+            -0.026405755430459976,
+            0.028910480439662933,
+            0.013245330192148685,
+            0.0005974372616037726,
+            0.020151015371084213,
+            -0.0017936384538188577,
+            -0.01397410687059164,
+            0.009601449593901634,
+            0.004266524687409401,
+            0.012424572370946407,
+            -0.00014460546663030982,
+            0.0214670579880476,
+            0.0018555490532889962,
+            0.0027399861719459295,
+            -0.0036226543597877026,
+            0.004687516950070858,
+            -0.03093407303094864,
+            -0.0006549257086589932,
+            -0.003102605463936925,
+            -0.008398614823818207,
+            0.011207587085664272,
+            -0.002421588869765401,
+            0.006558985915035009,
+            -0.022188758477568626,
+            -0.028485950082540512,
+            0.011342021636664867,
+            -0.016231190413236618,
+            -0.021396303549408913,
+            0.015339677222073078,
+            -0.0007641536649316549,
+            0.00614860700443387,
+            -0.007443422917276621,
+            -0.023278385400772095,
+            -0.019386861473321915,
+            0.02064630016684532,
+            0.012212308123707771,
+            -0.01190098561346531,
+            0.012750045396387577,
+            -0.021113283932209015,
+            -0.012948159128427505,
+            -0.001894464367069304,
+            0.0090566361322999,
+            -0.0019864458590745926,
+            0.01325240544974804,
+            -0.012198156677186489,
+            -0.00334494118578732,
+            0.0031592093873769045,
+            -0.011589664034545422,
+            0.008158047683537006,
+            0.01978308893740177,
+            0.008193425834178925,
+            -0.009339655749499798,
+            0.003422771580517292,
+            -0.011922212317585945,
+            0.013500047847628593,
+            0.015537790954113007,
+            0.018311386927962303,
+            -0.006498843897134066,
+            0.0020412809681147337,
+            -0.01370523776859045,
+            0.0063962494023144245,
+            -0.009827865287661552,
+            -0.004878554958850145,
+            -0.0005333155859261751,
+            0.004921008367091417,
+            -0.011179285123944283,
+            -0.0025772498920559883,
+            -0.026830283924937248,
+            -0.0013823752524331212,
+            -0.020533092319965363,
+            0.005303084850311279,
+            0.023207630962133408,
+            -0.007514177821576595,
+            0.0006045127520337701,
+            -0.0039092120714485645,
+            0.014476466923952103,
+            0.020688753575086594,
+            0.00870286114513874,
+            0.00482548912987113,
+            -0.006081389728933573,
+            0.02179253101348877,
+            0.011603815481066704,
+            -0.017816102132201195,
+            0.00164239970035851,
+            -0.001509734196588397,
+            -0.0014336726162582636,
+            0.011625041253864765,
+            0.004443412180989981,
+            -0.020844414830207825,
+            0.0028602697420865297,
+            -0.010089659132063389,
+            0.007818424142897129,
+            0.014702882617712021,
+            -0.00031906069489195943,
+            -0.02486329711973667,
+            -0.009827865287661552,
+            0.021919889375567436,
+            -0.015070809051394463,
+            -0.0008291598060168326,
+            0.00764861237257719,
+            0.023264234885573387,
+            -0.024792540818452835,
+            -0.006831392180174589,
+            -0.0023455272894352674,
+            -0.003933976404368877,
+            -0.005193414632230997,
+            0.0010489424457773566,
+            0.0214670579880476,
+            0.004415110219269991,
+            -0.004234685096889734,
+            0.005724077112972736,
+            0.006470541935414076,
+            0.02325008250772953,
+            0.0040577976033091545,
+            0.028047269210219383,
+            -0.00943871308118105,
+            -0.0023809047415852547,
+            -0.0019988277927041054,
+            -4.775960405822843e-05,
+            0.0160047747194767,
+            -0.005430444143712521,
+            0.020320827141404152,
+            0.009445788338780403,
+            0.0036934094969183207,
+            0.016514210030436516,
+            0.02319347858428955,
+            -0.006767712999135256,
+            -0.004793649073690176,
+            0.019160445779561996,
+            0.0053667644970119,
+            -0.008858522400259972,
+            -0.02194819226861,
+            0.01141277700662613,
+            0.0014124460285529494,
+            -0.0006956097786314785,
+            0.012106175534427166,
+            0.0024375086650252342,
+            0.006300730165094137,
+            0.008172199130058289,
+            -0.016684021800756454,
+            0.004241760354489088,
+            -0.004698129836469889,
+            -0.01860855706036091,
+            0.010910416021943092,
+            -0.012092024087905884,
+            -5.201595922699198e-05,
+            -0.0010002984199672937,
+            0.014165145345032215,
+            -0.01703779585659504,
+            -0.0019103841623291373,
+            -0.0070719593204557896,
+            -1.3024719009990804e-05,
+            -0.0009162768837995827,
+            0.008320784196257591,
+            -0.006254739128053188,
+            -0.010938717983663082,
+            -0.010938717983663082,
+            0.027509532868862152,
+            -0.004991763271391392,
+            0.003916287794709206,
+            0.019330257549881935
+          ],
+          "embedding": [
+            0.007204997251390758,
+            -0.03173324616100502,
+            -0.020505790884701024,
+            0.0037451608231038273,
+            0.009817085391002229,
+            0.026883177166072393,
+            -0.016477826654817607,
+            0.06949311853773875,
+            -0.01224876667472692,
+            0.031952566873802865,
+            0.013633671951171586,
+            -0.005041437355104989,
+            0.02663392750069342,
+            -0.022099132865267587,
+            0.006665702000169926,
+            0.0030467129715919033,
+            0.007931802508415542,
+            0.009984477355413895,
+            -0.010362836464183985,
+            -0.03180848181251013,
+            -0.013542377333474449,
+            -0.013572345022577053,
+            -0.03979985300694044,
+            -0.007698412130292187,
+            -0.005953452234356532,
+            0.018890228917932694,
+            0.02217446994660065,
+            0.005158357501784182,
+            -0.05468694715835175,
+            0.00789994048345594,
+            0.017130184174786147,
+            -0.0026591475669416114,
+            -0.02081141846859738,
+            -0.008403928318906045,
+            -0.030925346724945978,
+            -0.03855023691059801,
+            0.038853354785097736,
+            0.01612011653434478,
+            -0.035635702395196575,
+            0.026266623757525934,
+            0.043450373922427316,
+            -0.007254334835408145,
+            -0.015376804869420366,
+            -0.0017276202867766907,
+            0.004650414207094434,
+            -0.002365514361535186,
+            0.011906139905776183,
+            -0.0005664194302009289,
+            -0.0191723447249157,
+            -0.025546124518037998,
+            0.01025005257280377,
+            -0.014436532079436437,
+            0.029277529346694118,
+            -0.0045414983847733344,
+            -0.02260315456913697,
+            0.01717918656248626,
+            0.013666132349359846,
+            0.057804090189845364,
+            0.056600576093100735,
+            0.051208022172527734,
+            0.03553760896972431,
+            0.02094599510567085,
+            -0.02438674472832137,
+            -0.0027797634437333432,
+            0.03614248430646102,
+            -0.026729308357113124,
+            -0.002251394534452815,
+            0.018929251842788906,
+            0.00689697483318378,
+            -0.026603606434115974,
+            -0.0541443728344337,
+            0.027164630314707807,
+            -0.004335272389193875,
+            -0.013967321193992958,
+            0.003323121377857836,
+            -0.0245295956412367,
+            0.024489335129134787,
+            0.06212059558640156,
+            0.009214058207162371,
+            -0.002082293435905329,
+            0.013659934201796686,
+            -0.003344037927706573,
+            -0.00689041317919184,
+            0.020235785390418023,
+            0.0212739457082623,
+            0.0016782346585238381,
+            -0.013484221664059555,
+            -0.06465582506604173,
+            0.011882797526607326,
+            -0.032970655964890155,
+            -0.04027700439305653,
+            0.025396847912528105,
+            0.013861157517716707,
+            0.00604596157583048,
+            -0.00744611407420936,
+            0.04038084298991919,
+            0.021507532290328625,
+            0.005983822806623904,
+            -0.012711110369915742,
+            0.02410138142963901,
+            -0.018372791517880082,
+            0.021016674128298246,
+            0.013387285956415756,
+            0.02039383525127615,
+            0.010700292312227496,
+            0.0053934660497349735,
+            -0.014807678014449691,
+            0.009422805142740376,
+            0.0414357625490679,
+            0.12206398267002624,
+            -0.010877509834324155,
+            -0.011079302503786522,
+            -0.04107714853756596,
+            0.006631397986165343,
+            -0.031914267104222384,
+            -0.0179965295457466,
+            -0.057530036803978266,
+            -0.0320488164246567,
+            0.008208202363320509,
+            -0.0020203090915064162,
+            0.03502055780919601,
+            0.04464599164916666,
+            0.019332152079287082,
+            0.017133943087431935,
+            -0.039885565008919065,
+            0.04703059988373572,
+            -0.029370696964304416,
+            0.0008494086835971101,
+            0.0014082189719987074,
+            -0.030973762582071615,
+            -0.006561157852024518,
+            0.011634653995510726,
+            -0.019896826887411713,
+            -0.034275921680252924,
+            0.026274465530039866,
+            -0.007728234125236442,
+            0.04814514040690998,
+            0.06550407296280117,
+            -0.02749051703615343,
+            -0.010041099502187074,
+            0.02019601967763537,
+            -0.01871701432558228,
+            0.008532829759864928,
+            -0.020164138072967613,
+            -0.045196925901010855,
+            -0.004523517988516156,
+            -0.002113469149506399,
+            0.07929650663710812,
+            -0.01111269708042164,
+            0.03358254444720164,
+            -0.019973382759721527,
+            -0.0007145592863325417,
+            -0.0011740862928390678,
+            0.02166182480535788,
+            0.004543228023398081,
+            -0.013377688428761516,
+            -0.01758250073647576,
+            0.009548292014596834,
+            -0.04753617258004735,
+            -0.007125744019792479,
+            -0.009930358894299303,
+            -0.007931672044235996,
+            -0.024561936537429774,
+            0.012979195012881556,
+            0.009209145250061479,
+            0.04586756662401364,
+            0.04311807301838248,
+            -0.0036241105681042675,
+            -0.06579256720896123,
+            0.04967191755416882,
+            0.015475886231466817,
+            0.03157651656768371,
+            0.02515941109300714,
+            0.01773554572598468,
+            0.005824418191900522,
+            0.016317635547562007,
+            0.007650872352957098,
+            0.010015166465492963,
+            -0.033203307142867844,
+            0.040272188258443496,
+            -0.013408423788025047,
+            0.01036004294342104,
+            0.012391622448703885,
+            0.04214857862444935,
+            0.03418032388760535,
+            0.015263018079889338,
+            0.018786704580603317,
+            0.02704685343985817,
+            0.015348111464412869,
+            0.0013031600434754704,
+            0.03241427528588285,
+            0.011689924942195664,
+            0.04544406114742012,
+            0.016186615914968734,
+            0.011624500626103958,
+            -0.008272953078864385,
+            -0.02363080189189375,
+            0.02784298119317616,
+            -0.010585980875564109,
+            0.01455013604305903,
+            0.01749228557615982,
+            -0.054742056843690476,
+            -0.005876213780171779,
+            -0.010556605098730201,
+            0.003967456397644932,
+            0.014663888851767762,
+            0.0015548476559947807,
+            -0.003858367677774835,
+            0.012504286824484335,
+            -0.025806882498628832,
+            -0.014216299453397403,
+            0.0031620924516417956,
+            -0.006448390918604307,
+            -0.02785577502346934,
+            -0.00766402687921159,
+            0.005275000635711884,
+            -0.0090448751942,
+            0.019702662630533,
+            0.007009091837467867,
+            -0.01847307100043082,
+            -0.0006774105206839772,
+            0.032952656101827446,
+            0.015900033044760432,
+            -0.02170121261931376,
+            -0.017964879081469612,
+            -0.005062849745084668,
+            -0.03171847232054206,
+            -0.038177398521606544,
+            0.029426291950046455,
+            -0.025687201683643784,
+            0.030158867757101797,
+            0.015780411479509624,
+            -0.007011484049169482,
+            -0.012114023533118393,
+            0.020661407942701628,
+            -0.025067530380881376,
+            0.009777190971413269,
+            -0.004160771288889298,
+            -0.028825369485727344,
+            0.008307290529469729,
+            -0.014415903546305744,
+            -0.007561235629544945,
+            -0.03201704135271719,
+            -0.015182869388110948,
+            -0.00501561450194676,
+            -0.0003863746169672397,
+            -0.01073403445816663,
+            -0.03321103354401307,
+            -0.002304357889348034,
+            0.017001213960120047,
+            0.00028222146729658097,
+            -0.04002882852650754,
+            0.014509588052058002,
+            -0.004150541038999649,
+            -0.034333837380045466,
+            0.0564137596224324,
+            0.03584898691608747,
+            -0.006830698606679202,
+            0.006121456445207273,
+            -0.04505674213649008,
+            0.04358467937307492,
+            0.010847859073780811,
+            -0.04918983713702805,
+            0.049190442935189385,
+            -0.00528960443684769,
+            0.01419195371129309,
+            0.040316024601793406,
+            0.004600843996974165,
+            -0.006780970176765864,
+            0.005206651850992678,
+            0.03397517915308201,
+            -0.010037223444627569,
+            -0.027377913828275806,
+            -0.03328136706915221,
+            0.02562568440972934,
+            -0.03485770844248482,
+            -0.020069979230687374,
+            -0.0052041408068332524,
+            0.029818953377223505,
+            0.028789460147370508,
+            0.004071274130619537,
+            0.01928375754300958,
+            0.0014970947734202088,
+            0.010502599795027616,
+            0.016914145702515263,
+            0.011942576738188502,
+            -0.014909953528043828,
+            -0.003432489584767786,
+            -0.05733768990013957,
+            0.0030710780388681002,
+            -0.02627280695622545,
+            0.007930906480371135,
+            -0.031227917075485834,
+            0.013132797714872345,
+            0.002514351081823354,
+            -0.007152021475569949,
+            -0.0066245205132327055,
+            -0.005279224195782313,
+            0.028512607222275155,
+            -0.00921960825771865,
+            -0.0359704105589433,
+            -0.021347597388353182,
+            0.011200044645812517,
+            -0.0004171294784927543,
+            -0.016784468873218786,
+            -0.0018746544944553878,
+            0.007749103417240419,
+            0.02390272097796833,
+            -0.005128540108845805,
+            0.006397849195755672,
+            0.01214922062611769,
+            0.01163371266250103,
+            0.012307186722130844,
+            -0.004982589662882946,
+            -0.016343724553037307,
+            0.019806887143115643,
+            0.013139657463054087,
+            0.0205801609265216,
+            -0.03767676007629612,
+            -0.0249879011550972,
+            0.05547392586363707,
+            0.005903313549319051,
+            -0.05568806298774435,
+            0.018827693208840233,
+            -0.007931178941307645,
+            0.008183992600768387,
+            -0.010319022946041053,
+            0.05111352384595183,
+            -0.024897203290807753,
+            -0.02096678205188722,
+            0.006436523335212348,
+            -0.04917084523640266,
+            -0.010459284852442706,
+            -0.011138558192296276,
+            0.014789370043397917,
+            -0.03105725978752438,
+            -0.004934745004818259,
+            0.015595801375817122,
+            -4.687585727052446e-05,
+            0.0028886067775750254,
+            -0.00850352807499054,
+            -0.0034885579171087815,
+            0.016339681487281293,
+            -0.0034050317046826813,
+            -0.008309570448487812,
+            -0.003615165868022203,
+            -0.027638409919916757,
+            0.023352910161586186,
+            0.026364511529991978,
+            0.001249936082895606,
+            0.001987948967948376,
+            0.01756629246122769,
+            0.004656075485833687,
+            -0.006752579977941239,
+            -0.013681608638716132,
+            -0.03405936202918687,
+            -0.0401867651078737,
+            0.022991180183813725,
+            -0.02318908140203705,
+            0.018299512436825923,
+            0.00904081847012441,
+            0.019697091729447287,
+            -0.019236130984371835,
+            0.015159917756323988,
+            -0.0387297415849285,
+            -0.018793498297161176,
+            -0.005256288213225733,
+            -0.010414697711744128,
+            0.0007039894599589104,
+            -0.01739645696111503,
+            -0.013728845328097506,
+            0.022709522955937343,
+            -0.022789243441781803,
+            -0.0036763802589970026,
+            -0.01429671717544519,
+            0.0023674126354206932,
+            0.049448298344058644,
+            -0.009926033487593551,
+            -0.023371285193090733,
+            0.020846792916650956,
+            -0.026996519120163966,
+            0.011246908457705283,
+            0.04091365159640201,
+            0.005469162850197613,
+            0.005640919201612091,
+            0.029575753750229278,
+            -0.0007860452837141593,
+            -0.0052805237594885516,
+            -0.009782745443933828,
+            0.021178503581190618,
+            -0.0009176512672832309,
+            -0.05245650420332918,
+            -0.007698364493410058,
+            -0.039027418406703554,
+            0.005448840777995117,
+            0.017581625031764185,
+            0.025522517097715556,
+            -0.020775492472580457,
+            -0.015906320383873178,
+            -0.000643669839811819,
+            -0.0086896084429604,
+            0.0029399233482444638,
+            0.014381240985647907,
+            0.021086693776922936,
+            0.002436575081260609,
+            -0.004913514868101904,
+            -0.011582101006213432,
+            0.014400297663309155,
+            0.0467181039341492,
+            -0.01722692180137712,
+            0.0011622224560709315,
+            -0.013036540368441797,
+            -0.06208454638917413,
+            -0.019160212331432766,
+            0.0351496273293468,
+            0.02429814055116254,
+            -0.009896662169762772,
+            -0.01647101145477158,
+            -0.002915821099802562,
+            -0.03557863734177486,
+            -0.01786433999615293,
+            -0.010699364747311715,
+            0.02390901423299708,
+            0.007213809581960646,
+            0.0005601364043388203,
+            0.00916614601011734,
+            0.014737491634516238,
+            0.019303599465763203,
+            0.008864535607492504,
+            -0.0575839606069681,
+            0.018051837786406608,
+            0.006315830080225406,
+            0.02525619013082567,
+            -0.015671924543113065,
+            0.003988943108210209,
+            -0.03179878281917948,
+            -0.029440641244429775,
+            0.040586322748158446,
+            -0.03418722657593859,
+            -0.030650542761032924,
+            0.006566745611226466,
+            0.009571089937153852,
+            0.004491158558315032,
+            -0.011352813042040581,
+            -0.02535505987721779,
+            -0.013381948772812133,
+            -0.030918676056735445,
+            -0.02192708443081135,
+            -0.005232667170255381,
+            -0.03342408726436631,
+            -0.07498634115550039,
+            -0.006245638452568239,
+            -0.049946896465756174,
+            -0.03900385398225758,
+            0.04458524125677807,
+            -0.006106489377959481,
+            0.02398410968928338,
+            -0.04615183642824994,
+            -0.01165382599292015,
+            -0.04890013373371541,
+            0.013624324077017377,
+            0.01476398305411351,
+            0.020202554298470666,
+            0.00014553149242532853,
+            0.036224316512987846,
+            0.00021498496267774312,
+            -0.05055208658013767,
+            -0.057482169168904106,
+            -0.003960451575823462,
+            -0.010217078509789989,
+            -0.005593738972150311,
+            -0.034090261748104665,
+            0.05835341009022349,
+            0.0116359661127526,
+            0.04203678184748103,
+            0.03447818335186531,
+            -0.006747564131379555,
+            -0.025756962754841885,
+            -0.028123416424745064,
+            -0.008073625379614702,
+            0.047544241458556825,
+            -0.029605671083240564,
+            0.008171081332993893,
+            -0.026007452205474395,
+            0.0067084589781962805,
+            -0.0070645448617940995,
+            -0.0012298101311792947,
+            0.012372286141738867,
+            0.006377319686763927,
+            -0.015024537877377194,
+            0.007934745664790507,
+            0.026041755081605937,
+            -0.007303731324541215,
+            0.0407000170257006,
+            -0.012990703097937777,
+            -0.033189084013863494,
+            -0.022525679964848666,
+            0.01657717698191603,
+            0.03440284885162039,
+            0.010230633738702899,
+            -0.008707808754455952,
+            -0.0363777020540216,
+            0.011184076269109602,
+            0.031774012431843524,
+            0.01504703222643543,
+            -0.022509833303978505,
+            0.025230974050270107,
+            0.017502662694813216,
+            -0.018700579139537456,
+            -0.03511734747709348,
+            0.015162183119327901,
+            -0.017711638132862063,
+            0.018190335050681287,
+            -0.01590174153215615,
+            0.011103110869700727,
+            0.0025189370696524936,
+            -0.00920305384736575,
+            0.01929652151514404,
+            -0.006196715938358087,
+            0.011426611271994685,
+            0.007058186102245829,
+            -0.018191352599117545,
+            0.0027202185072220937,
+            0.057310566886380025,
+            -0.000531639834386418,
+            0.01494975681909879,
+            0.03845692752400366,
+            -0.001971953300066856,
+            0.045554909803324765,
+            -0.02532332999916258,
+            -0.042616337063845953,
+            0.02633963974172818,
+            -0.02970705577683783,
+            -0.01624154998005974,
+            -0.02620993449678864,
+            -0.021490817420853802,
+            -0.004038424045768319,
+            0.009940887982818171,
+            -0.02692933845395842,
+            -0.023801570856681796,
+            -0.001948460540662095,
+            0.02290911497874141,
+            -0.03300532378029689,
+            0.006525488380791647,
+            0.007039306647666766,
+            -0.003685585408710752,
+            0.03183735852538168,
+            -0.027197064244321705,
+            0.006861928286688311,
+            0.011209869849682412,
+            -0.001094897323854524,
+            -0.005328676992017907,
+            -0.004007474089818478,
+            0.010796946231051235,
+            -0.030543173497906445,
+            -0.012879972141706173,
+            0.00935894527375688,
+            0.0017573071057611829,
+            -0.010050882212831222,
+            -0.03102132152986303,
+            -0.002915141291985554,
+            0.007634137769426993,
+            -0.011094093612593696,
+            0.01199609810193786,
+            -0.010458016758814742,
+            -0.04929347607956551,
+            0.010973132159729983,
+            0.028148117094907874,
+            -0.025504019123994758,
+            0.012470127357818321,
+            0.009182729379865986,
+            -0.016020133756226275,
+            0.016320111225714855,
+            -0.012051271446461976,
+            0.024098195821827578,
+            0.04317620443481784,
+            0.024592886889234396,
+            0.007727787037954994,
+            0.00771746776156926,
+            -0.0003147538723468374,
+            0.022572868970938624,
+            -0.016683340264320543,
+            0.013672085891597506,
+            0.020827021901755415,
+            0.02092309219721837,
+            0.0057130785714458055,
+            -0.003901728981013356,
+            -0.0036968415923910314,
+            0.002227024825579113,
+            -0.02106678364798051,
+            0.014578546755457675,
+            0.03128101864424301,
+            -0.005234627629617124,
+            -0.022394744692442837,
+            0.002430617645746389,
+            -0.015780389432069438,
+            4.7277885648917e-05,
+            0.013357803020920043,
+            0.013777651543690907,
+            -0.027713499926132385,
+            -0.0018999112228756628,
+            0.028364630573666852,
+            0.009146793705085586,
+            -0.009891297866910313,
+            -0.01644269527910728,
+            -0.0003903768506971414,
+            0.006512217314869576,
+            0.02408785204739448,
+            0.0018406499545367782,
+            -0.007360473750031044,
+            0.01484497577088404,
+            0.016723277095327158,
+            0.003629911330658209,
+            -0.0004820567356447713,
+            0.02394406451074397,
+            0.023794019615473026,
+            0.001632266615911801,
+            -0.02064379152040184,
+            -0.02755693521684878,
+            0.027335938400284726,
+            0.02769248863677192,
+            -0.006832353240623308,
+            0.027324973360715134,
+            0.003937207909967385,
+            0.013580154030133622,
+            0.02076818768224654,
+            -0.01784151382870951,
+            -0.017617903734311997,
+            0.0017316476451235418,
+            -0.009605294810027804,
+            -0.009829196696575069,
+            0.004891780008779742,
+            -0.011655402383771472,
+            -0.01124379942906224,
+            0.043205454954628145,
+            0.004713701851783307,
+            0.003560551284131962,
+            0.007894840447501912,
+            -0.018881214526749082,
+            -0.00945538175144614,
+            0.0019040259741796817,
+            0.029505863720301916,
+            -0.001400002868254808,
+            0.03315866433970851,
+            -0.00830542045504805,
+            0.008602408964961391,
+            -0.03202191233287069,
+            -0.006978289755114341,
+            0.0326499698728755,
+            0.015314305362524091,
+            0.015104436300213667,
+            0.0021257042489266787,
+            -0.0010840650756005028,
+            0.02434348355847526,
+            -0.007174433778817674,
+            -0.04839373063710409,
+            0.022696994450018916,
+            -0.004779511451775743,
+            0.023588216956749154,
+            0.0382733980620746,
+            -0.004232286553944853,
+            0.003815507826545929,
+            0.023531223314846965,
+            -0.01887650634213847,
+            0.015055403379782985,
+            -0.01918944132635701,
+            -0.017597900721414636,
+            0.02176630010543207,
+            -0.008896608357798277,
+            0.01732387427588561,
+            0.01764363756727683,
+            -0.012180043444768312,
+            -0.019003360499321047,
+            0.01229139850966224,
+            0.0025710091163655626,
+            -0.017075658234265562,
+            0.01797412486853335,
+            0.018372010308966135,
+            0.006723179570342314,
+            0.015417677746651683,
+            -0.00047634684109332134,
+            0.017608296628666153,
+            -0.001049984114818522,
+            -0.008416990841821452,
+            0.01827827907328822,
+            -0.009568717277044265,
+            -0.01658089563620308,
+            0.009808478422454319,
+            -0.05193977747210182,
+            0.010644335931025957,
+            0.0016781339270411392,
+            -0.01562285100294917,
+            0.00405036330963539,
+            0.007102724536527262,
+            0.005195069570261247,
+            0.0028773630901681767,
+            0.004610474981005765,
+            0.042921486858787584,
+            0.013213672397112759,
+            -0.0064030584930089415,
+            -0.010711457508461704,
+            -0.017480309425763254,
+            -0.027151012983864797,
+            -0.005204425551873271,
+            0.009927884476991038,
+            0.022601230152836282,
+            0.008046297278661014,
+            -0.0031858480011267704,
+            -0.02589628103025927,
+            0.019723430185137954,
+            0.03604835658149313,
+            0.062423099201768716,
+            0.010930268386637612,
+            0.02925564269324381,
+            0.02439517070384333,
+            0.023930432096006114,
+            0.013036828091946605,
+            0.006910311087503408,
+            -0.032995841694293995,
+            0.012813650252889403,
+            0.01333714655191133,
+            0.03678966264802025,
+            0.012571138353467183,
+            -0.05610151838497368,
+            3.0777966587107344e-05,
+            0.009791774904454618,
+            0.041959488802722524,
+            0.01956296954085397,
+            0.000155217569370113,
+            -0.007285602454264675,
+            0.031547870838711856,
+            -0.026518385135879206,
+            -0.012299593280363843,
+            -0.0026551378470968594,
+            -0.006190117573235779,
+            -0.003807907001161504,
+            -0.004991957475076049,
+            0.036983035940964804,
+            -0.015648565314743963,
+            -0.019256463332683493,
+            0.0004300362961407231,
+            -0.0015686065121077472,
+            0.0011311524136286674,
+            0.0032236641835276464,
+            0.013795345278976164,
+            0.015385029265915279,
+            -0.004800448639353721,
+            0.026514926384796363,
+            -0.008699453553651681,
+            0.022186817580012775,
+            -0.013427553742493343,
+            -0.011927259975371873,
+            0.017776228476138103,
+            -0.021243797085501972,
+            -0.01667280589187595,
+            -0.003688045116580565,
+            0.006189970874536939,
+            0.0020911968266404467,
+            0.0006097761986763103,
+            -0.004685468039463817,
+            -0.01011357666327168,
+            0.022631082517271542,
+            0.012398000826637277,
+            -0.006721588925828951,
+            0.018142966317821054,
+            -0.0021330638043572654,
+            -0.011869184647489572,
+            0.025073455250141923,
+            -0.0015392945266926425,
+            -0.0337085890598252,
+            0.01267302113143573,
+            -0.013438601165482187,
+            0.02673290895281225,
+            -0.009586762706902736,
+            -0.011208527168019762,
+            0.019144928964011168,
+            -0.0017591316941248897,
+            -0.005459150067129732,
+            -0.029230079356228025,
+            0.004679497471366242,
+            -0.011723195942933286,
+            0.015899377439148873,
+            0.0022054234777254627,
+            0.029187759252864872,
+            -0.018290694895057157,
+            0.00792533824593637,
+            0.011719198071061566,
+            0.014612375997487872,
+            0.005419943379355563,
+            0.010515265723144554,
+            -0.006893314445458116,
+            -0.04424495426064596,
+            5.8241472764594054e-05,
+            -0.005719143565673274,
+            0.0038220021965914754,
+            0.013106940982933836,
+            -0.01203955335201393,
+            0.011260626096963215,
+            -0.0279896016489392,
+            -0.0295228261396555,
+            0.036474811902902415,
+            -0.04900747926861378,
+            0.019261089742811332,
+            0.00014104646541594166,
+            -0.03057197731717424,
+            -0.04331886560073797,
+            -0.03232957159604667,
+            -0.019401402365464362,
+            0.004102283356333861,
+            0.0002610070704924167,
+            0.0012280372705500212,
+            -0.014025012323929194,
+            -0.014135990809256075,
+            0.013269444858490211,
+            0.00992155580941576,
+            0.011225162609266308,
+            -0.006545044517282357,
+            0.002368862850663672,
+            -0.02063183105457681,
+            -0.026981383275694142,
+            -0.02774194980145225,
+            0.013337303400678109,
+            -0.0025902548982556788,
+            -0.02794339093668326,
+            -0.025606606546251152,
+            -0.0029852498102770587,
+            -0.01906873442473352,
+            0.006508845592362198,
+            0.03268861529625085,
+            0.0355975143313428,
+            0.010066906387952277,
+            0.01690250467284985,
+            -0.017201544758038183,
+            0.048345309114140714,
+            -0.019031171948182872,
+            0.03188329748476941,
+            -0.0005845169448621133,
+            -0.024140202991373587,
+            0.00011597263700677706,
+            0.03203623528903764,
+            -0.019285565857567896,
+            -0.01332755507465857,
+            -0.010914187315481855,
+            -0.003374912271519852,
+            0.02419326154642231,
+            0.02443825785057301,
+            -0.0037923303809811746,
+            0.00022800399604730175,
+            0.015893265651968885,
+            0.003154527829977343,
+            0.007688529734165207,
+            0.030623379985627763,
+            0.014052282574575454,
+            0.01127235345371133,
+            -0.04164707563955819,
+            -0.011811587719246128,
+            -0.003763086694358232,
+            0.005909410979841885,
+            0.002913220159138154,
+            -0.002671613084376487,
+            0.0045765194162907885,
+            -0.007241703279856166,
+            -0.005855949422953439,
+            0.0016973541436777935,
+            -0.001576326424468957,
+            0.008393873711385013,
+            -0.0037217094191171083,
+            -0.03498879678492834,
+            -0.026096985039773694,
+            -0.021280075788049788,
+            0.006437497348154639,
+            0.0070214556591898375,
+            -0.00972721128392027,
+            0.01949553762684561,
+            -0.0029404319702066146,
+            0.000271037702827666,
+            -0.010734867909488861,
+            -0.0029784341515646075,
+            -0.01206464664516104,
+            -0.019544778160755956,
+            0.00035139045159739705,
+            0.013457601448003152,
+            0.00025947376767512006,
+            -0.03165522602526639,
+            0.011979811705146604,
+            0.006642849985850598,
+            0.013181162886675883,
+            -0.024499404540711422,
+            -0.02310714308893576,
+            -0.028028912907092744,
+            0.010570122662701183,
+            -0.0066866321204157725,
+            -0.024617571284817428,
+            -0.004097807033448707,
+            -0.0074576460628928256,
+            -0.0020417256533259887,
+            -0.02470953739335365,
+            -0.011408461668188081,
+            0.023537349682260345,
+            -0.006373652338136973,
+            0.01936338676879376,
+            -0.017636435070085086,
+            -0.017036091876814843,
+            -0.002198685044404603,
+            0.015584758061011048,
+            -0.038943923931820196,
+            -0.002076529978164001,
+            -0.008452513321945455,
+            0.006551055965075869,
+            -0.028840587857625097,
+            -0.007128279504360441,
+            0.0199397499488538,
+            0.012424319366271584,
+            -0.03518875279748495,
+            0.013596027106451115,
+            0.022191403935861113,
+            0.01218507466930854,
+            -0.0038551718784437783,
+            0.03910075115429626,
+            -0.0025215641203030336,
+            -0.005422688956956598,
+            0.016759888275451905,
+            0.04462538468871483,
+            0.023708097134660096,
+            -0.03152743757744915,
+            -0.008668774531128668,
+            -0.015224825785549947,
+            0.004102648962663789,
+            -0.025867795193966987,
+            0.004676262922216584,
+            -0.011739753544166856,
+            0.01645951757031186,
+            0.011253400919230348,
+            0.013958707917471253,
+            0.0376596076542735,
+            -0.005077354977296842,
+            0.02668910096392324,
+            0.0010332284250006117,
+            -0.00856723931523911,
+            0.0014431148305642308,
+            -0.01412579094241589,
+            0.029892393553325205,
+            0.02269599036836208,
+            0.014421409701746036,
+            0.029248195108770955,
+            -0.03588867295960255,
+            0.02458655544632528,
+            -0.0010013539341895672,
+            0.008368648583431747,
+            0.018150734970019018,
+            0.02628022565029391,
+            0.015423867373844644,
+            0.012477080028047823,
+            -0.02513018750675568,
+            -0.022180001014682045,
+            -0.0027550350630288997,
+            -0.031944843677988374,
+            -0.0014668093006914267,
+            -0.009789309083948147,
+            0.026386027599666254,
+            0.0046988411391149,
+            -0.019803160322638583,
+            -0.003931806845108628,
+            0.004482097428470392,
+            -0.00244660100774939,
+            -0.010441732346892935,
+            -0.004649318716095894,
+            -0.01869464992812615,
+            -0.017838643679294407,
+            0.023677878122153204,
+            -0.023958455640817666,
+            -0.025238725541146534,
+            0.014582798102040157,
+            0.0013223590277629197,
+            -0.002295641913710489,
+            0.0008238518461567685,
+            -0.02115533275282518,
+            -0.04214275933987015,
+            -0.027817083427786927,
+            0.0037588803464689412,
+            0.03565056002940463,
+            -0.026788865866928923,
+            0.014162550819567465,
+            -0.011336583458519,
+            0.019175861037658344,
+            -0.005127229083828615,
+            0.015531116410874209,
+            -0.018251290727445846,
+            -0.044630374763679266,
+            0.021833307809708984,
+            -0.012087009688208446,
+            0.02720742725750304,
+            0.021497809277370574,
+            -0.026937653159100156,
+            -0.013572733244188407,
+            0.01358214612023832,
+            0.0020408253706382332,
+            -0.009652962552750464,
+            -0.0063411929391088626,
+            -0.010533309167065193,
+            0.003901271588567942,
+            -0.04496345296706673,
+            -0.008761188928422701,
+            0.06948704056954039,
+            0.005916180375965137,
+            0.03157905519638486,
+            0.014668818318138927,
+            -0.028448496686381258,
+            -0.04008949536631331,
+            -0.00462253589823598,
+            -0.0003701522833020314,
+            -0.00024410762276269235,
+            -0.01850585249309217,
+            0.017699615637510956,
+            -0.002712069912576265,
+            -0.009767338815933488,
+            -0.013119464222297351,
+            -0.013743610418034631,
+            -0.030375365173543157,
+            -0.0030316327877227783,
+            -0.008632689470358149,
+            -0.006565457264917239,
+            -0.0328089275944709,
+            -0.011713996290006943,
+            0.029773910045828737,
+            0.02740100904353622,
+            -0.0061926915497348915,
+            0.00638737826122608,
+            -0.010244154924811975,
+            0.0010396203272677069,
+            -0.001264454445070139,
+            -0.004059969507786729,
+            0.008715730726831364,
+            0.0014767015415289076,
+            0.031074767756160294,
+            0.004761858344125113,
+            0.00987870172044607,
+            -0.059926117793699066,
+            -0.009294021263133636,
+            0.021188432871397475,
+            0.016602826531924377,
+            -0.009125784136292983,
+            0.013946188378030194,
+            -0.0017670794602549428,
+            -0.006837334179368295,
+            -0.028290055353601393,
+            0.024556541126233806,
+            0.04543639597357669,
+            0.0031311343860416473,
+            -0.011199639773247286,
+            0.014965677855595444,
+            0.0018248533506196013,
+            0.015507721118783542,
+            -0.011409402378400194,
+            -0.01338737990904486,
+            0.03063410115541065,
+            0.009305362935335817,
+            0.0025578631174065385,
+            0.012727648343848194,
+            -0.0038680554900795763,
+            -0.01173215289303824,
+            -0.01568862495293563,
+            -0.004101703129244437,
+            -0.03649200408657064,
+            -0.0022107849490954137,
+            0.009659780679763728,
+            0.0034091693454195806,
+            0.008848380026939594,
+            0.01842435611499587,
+            0.03546926214224266,
+            0.013478278853234764,
+            0.005970293629431872,
+            -0.0069197533761813455,
+            0.023196445250565058,
+            0.010403914879205351,
+            0.010017272564548626,
+            0.015076807790462658,
+            -0.029649260409273275,
+            -0.012737152920662164,
+            0.00373276777994811,
+            0.013239909450830258,
+            0.0001682544254742693,
+            0.02030652051051447,
+            0.014143961923579586,
+            0.0010147733405793957,
+            -0.012509446388074923,
+            0.0020437769941487626,
+            0.007405578085797006,
+            0.013458276946120546,
+            0.0006779114540819997,
+            -0.004250745498730309,
+            0.0009149988500358458,
+            -0.008355906072311008,
+            0.040391939952386505,
+            -0.01831218885700067,
+            -0.016793844108223176,
+            0.035064882870542116,
+            0.11416006426189942,
+            0.0010630856112440689,
+            -0.015660014600661857,
+            -0.017142534874931403,
+            -0.0086899013413391,
+            0.016611825572132997,
+            0.01609650977226868,
+            0.009780106186629793,
+            0.009622367309076686,
+            -0.000672926017804395,
+            0.024310949465350565,
+            0.02944698490225121,
+            -0.0037169531700178868,
+            -0.010990838766452194,
+            -0.036819646475041955,
+            0.022923962979125287,
+            -0.004596366586768123,
+            -0.007108041562635549,
+            -0.002604794313552557,
+            0.012641223813290853,
+            0.013668775645501733,
+            0.005125100040230003,
+            0.0077993841846437815,
+            -0.011500851674334447,
+            -0.017334458072700107,
+            0.0056819751688881774,
+            -0.015982234644378358,
+            -0.0076533013358874935,
+            -0.013127172654362694,
+            -0.011837346554842119,
+            -0.006562312362371076,
+            0.0029877420008976425,
+            -0.008572723854050838,
+            0.00854083969164562,
+            -0.010939639783598493,
+            0.016185642759084087,
+            -0.008052282332943615,
+            -0.0011992212936609377,
+            0.0010378646897070758,
+            -0.0025399515501741718,
+            0.010712865639618352,
+            0.01451357949543628,
+            -0.02521401163091652,
+            0.020836357346214075,
+            -0.008252813367155418,
+            0.0006409807249997337,
+            0.015338809580802425,
+            -0.04672526649678653,
+            0.022728876784937357,
+            0.005129225753457052,
+            -0.018372222092794564,
+            0.0066182736895667805,
+            -0.02048375780196863,
+            -0.00896382274306482,
+            0.0021738974344880794,
+            0.004491167150658828,
+            0.0034280799511763956,
+            0.015611133773752265,
+            0.018126362497548308,
+            0.00014383671280220997,
+            0.007927708693242186,
+            -0.006973076292062532,
+            0.008118313055163714,
+            -0.006791836664375275,
+            -0.01164567445668625,
+            -0.010894577039363014,
+            0.00014731542306100506,
+            -0.008865132658571428,
+            -0.011415696755798914,
+            -0.017922318360382588,
+            0.00358538001727405,
+            0.00289327637128695,
+            0.010406054711003452,
+            -0.007757607207626101,
+            -0.010994775479337082,
+            -0.011716096880261664,
+            -0.0013333474316266438,
+            -0.01231144419953554,
+            0.009792603083441917,
+            -0.02884953407490196,
+            0.0029097409399144306,
+            0.025277274938032477,
+            0.009263530315933738,
+            -0.028037993274175277,
+            -0.02532252134134532,
+            -0.0018596440554143638,
+            -0.013847737486669237,
+            0.009910107581119272,
+            0.013952962280235523,
+            -0.011807495289410774,
+            0.003896325037406752,
+            -0.02144921729633576,
+            -0.01628545945817568,
+            -0.0040148671532777005,
+            -0.016032493124255323,
+            0.004580041957001551,
+            -0.011375180364988195,
+            -0.007789996450583554,
+            -0.022041665680945066,
+            -0.005890065942573431,
+            -0.050455058269193866,
+            0.014401045842891128,
+            0.030219548461217365,
+            -0.004580747668649223,
+            -0.028684680969792875,
+            -0.015525870112797619,
+            0.011121401958547706,
+            -0.008879749268609245,
+            -0.019454243014778435,
+            -0.007839434838464604,
+            0.018908534797107714,
+            -0.0019283067432864216,
+            0.0024769961730655835,
+            -0.006732653593696035,
+            0.027604805021030646,
+            0.01366593691107021,
+            0.03740746059041736,
+            -0.014662268336336578,
+            0.02539666271785517,
+            0.02371809783638648,
+            -0.036126848124425154,
+            -0.007627445211958312,
+            0.03657481892016626,
+            0.01092515982671624,
+            0.0011046573579274704,
+            0.012350047525582187,
+            -0.006340802546498992,
+            -0.031330519599469066,
+            -0.0013691360532301728,
+            -0.0014676630169621005,
+            -0.023111265345539388,
+            0.027359543575867225,
+            0.018091458372134273,
+            0.014237156751820934,
+            0.0030993618578507172,
+            -0.0014037082734049385,
+            0.013257269860435359,
+            0.01021315143964527,
+            0.030885417052806473,
+            -0.003225065956381245,
+            -0.012825341172770057,
+            0.013476039084668826,
+            0.01852164843904835,
+            -0.009965495459579544,
+            0.012541079912270015,
+            -0.00824122613985912,
+            -0.018124134280975132,
+            0.03192263766222747,
+            0.0001484308749572035,
+            0.010888733498667343,
+            -0.010649515858317311,
+            -0.013082335214246402,
+            0.032614733822945205,
+            -0.00999075761269065,
+            0.004257643407198955,
+            -0.030723533031261387,
+            -0.003820776505926347,
+            -0.010471792928758869,
+            0.01737608180148319,
+            0.004597677757231315,
+            0.02095148515295301,
+            0.029253627700524396,
+            -0.011604450820008121,
+            -0.01996325509217509,
+            -0.001158626158150122,
+            -0.020583793738536985,
+            0.004168757459099133,
+            -0.03632338333261702,
+            -0.019373660230711246,
+            0.022259216603413172,
+            -0.010364620831688015,
+            -0.004751193403783363,
+            0.00444187509644351,
+            0.0008572202661850934,
+            0.014943779567410408,
+            -0.025408263453313398,
+            0.017755758370543148,
+            0.004627392860417132,
+            0.007402073071525438,
+            -0.024913329140362466,
+            0.013760771254234878,
+            -0.006087346663657887,
+            -0.03780469033982951,
+            0.005509388120163784,
+            0.03756141956381493,
+            0.02381019168406692,
+            -0.024580249929652097,
+            0.01807584361080568,
+            -0.03111039245576191,
+            0.023245542501125073,
+            -0.004868090951368273,
+            5.1544460403943004e-05,
+            0.00640235757127173,
+            -0.004698373089469779,
+            -0.00621383035264185,
+            -0.0025132415458431587,
+            -0.008746634331769936,
+            -0.01611801559493664,
+            -0.01193535623883546,
+            -0.03381789115349148,
+            -0.024119754887378317,
+            0.013881102852385332,
+            0.0019232668966222687,
+            -0.0095263729740736,
+            0.018652966857927805,
+            -0.03775912497511837,
+            -0.012022352025516115,
+            -0.023289377270709736,
+            0.023579672990954228,
+            -0.028605627815260902,
+            0.019235397553197357,
+            -0.025838655372560505,
+            -0.021458526449310157,
+            0.01508157885177254,
+            -0.010064396230734452,
+            -0.0016708624560165368,
+            -0.015042071642751917,
+            -0.03670785416972153,
+            -0.006267614265977843,
+            0.02298763967146357,
+            -0.012506444728541897,
+            0.00029871598584066644,
+            0.02671390618094565,
+            -0.003059865833779805,
+            -0.011103452976820622,
+            -0.0017382684168257506,
+            0.010923055147701782,
+            -0.02938686260440486,
+            0.014300097108092281,
+            0.0038571851420598074,
+            -0.0037614499297117365,
+            0.0010991440540809893,
+            -0.010108413486243776,
+            0.02127952161226619,
+            0.009142545227414532,
+            -0.0030260119510809205,
+            0.040589877142650374,
+            0.008677453889680373,
+            0.0017339508605300814,
+            0.008732950895530873,
+            0.01063352447960556,
+            0.006618635029823979,
+            -0.014986307792626314,
+            0.016905861015681957,
+            0.013101237900847744,
+            0.0002162791516784468,
+            0.019309343995079022,
+            0.0021263376627204313,
+            -0.024516267132902416,
+            0.013339486637573757,
+            -0.02311659751478777,
+            0.0020575982953961246,
+            0.02391695586495566,
+            -0.020440172706421535,
+            -0.001656132876701879,
+            -0.006304133667169861,
+            0.000543457661904495,
+            0.022135707414160034,
+            -0.0230903871986617,
+            -0.023444763940153112,
+            0.009604600033024832,
+            0.01721387169484476,
+            0.01827905276638783,
+            0.02824773106648473,
+            -0.007100625424382998,
+            0.01348510178676837,
+            0.0002604386361122316,
+            -0.011802178189607182,
+            0.007664157334475143,
+            0.0074175377458313594,
+            -0.0044008460682833875,
+            0.025491035291032547,
+            -0.0017015687016376205,
+            -0.006134010425966364,
+            -0.005379657499404915,
+            -0.01584573519366957,
+            0.0034386456145185213,
+            -0.013860742817569676,
+            -0.013355238174583658,
+            0.0040342831827141845,
+            -0.0208160950322342,
+            0.10904592104153767,
+            -0.0037038089210379766,
+            -0.019472012964045247,
+            0.0063554319446707044,
+            0.003716719884684827,
+            -0.003692786470980806,
+            -0.003353843210051379,
+            0.007788605566526459,
+            -0.011254480693190834,
+            -0.0045751956417398095,
+            -0.013976039724381875,
+            -0.001989262807706565,
+            0.010067374586433014,
+            0.009201983771978006,
+            -0.005902980676079493,
+            -0.02029714318363332,
+            0.013031676901335606,
+            -0.016970144750162643,
+            0.008874670945038486,
+            0.0008311255074953139,
+            0.00884167159995975,
+            0.001716603195908536,
+            0.01854951250316076,
+            -0.009310881405486074,
+            -0.01325242738760595,
+            0.029667589863813217,
+            -0.005616511284128333,
+            -0.0038179509972925954,
+            -0.0004690716809300012,
+            -0.005068974756744239,
+            0.0021204231125051274,
+            -0.027958274497127258,
+            0.022127088236172387,
+            -0.017271838492631593,
+            0.010267709087809265,
+            0.01316800560257366,
+            0.005599367128954229,
+            0.01817806970002991,
+            -0.007115594036834107,
+            -0.02627643123342057,
+            0.006542742528003751,
+            -0.003433874053082844,
+            0.006631427284537971,
+            0.02316295746074422,
+            -0.014573351469966297,
+            -0.033520864104177005,
+            -0.013707022143307006,
+            -0.01877574599842729,
+            -0.005992004412448576,
+            -0.0043698881826095435,
+            -0.01481352026460129,
+            0.006424121402963946,
+            0.01382938882387991,
+            0.018136021761515388,
+            0.0012697483790911433,
+            -0.011531136974861559,
+            0.00609187274277241,
+            0.01076133059240578,
+            0.008400887242930113,
+            -0.01533504932731406,
+            0.011439227832642668,
+            -0.028718487721012004,
+            0.01522697877548034,
+            -0.012452665599305595,
+            -0.024938748080573947,
+            -0.0022082450217489366,
+            0.017589399637694264,
+            0.0037824832255756763,
+            -0.004355445812154662,
+            -0.018778668474337727,
+            0.019825087149714933,
+            0.024692878851503508,
+            0.003883497401229232,
+            0.019470881143244197,
+            0.041834635328088364,
+            -0.003544509491805351,
+            0.027139164223766474,
+            -0.02129723143323512,
+            0.01461354860170749,
+            -0.03441729954693155,
+            0.001893713808284104,
+            -0.004222657393633789,
+            0.0013677766871517654,
+            -0.003811928806805752,
+            0.020142937759360904,
+            -0.009158202577174012,
+            0.0030638058587375543,
+            0.024854053046139163,
+            -0.018665378030958756,
+            0.00805903734779081,
+            0.010625429315145887,
+            -0.012856237251902233,
+            0.028535299711292306,
+            -0.006335094045840286,
+            0.00305422126851784,
+            -0.0297883488914759,
+            -0.0029192872993071216,
+            0.0026067044379754047,
+            -0.006097749614529602,
+            -0.00968023644800927,
+            -0.007433916539502931,
+            -0.006202274026852405,
+            0.0038354607448269555,
+            -0.021471361566683354,
+            0.003195646827154078,
+            -0.004390751282171024,
+            -0.003323447423321121,
+            0.02518759723554948,
+            0.025857746688368054,
+            0.016893707217894103,
+            0.012739170140289531,
+            0.008553625305650129,
+            0.019805351588748434,
+            0.01654964927067909,
+            0.011549329129728739,
+            -0.020026013396046632,
+            0.01150644479272541,
+            -0.004159611921476011,
+            -0.0022937297602724912,
+            0.014815466072709131,
+            -0.01572171840394285,
+            -0.02276187015864222,
+            0.024693009795978448,
+            -0.019725315346836404,
+            0.00998768979020729,
+            -1.1702210281215188e-06,
+            -0.011129745730507545,
+            -0.002591293237053776,
+            -0.018034357254990988,
+            0.00012070821244376275,
+            -0.023051045150566527,
+            -0.028465270110388716,
+            -0.00048603008831669926,
+            -0.000574546300981984,
+            -0.012992623897880395,
+            0.02064747165046032,
+            0.014820638761258362,
+            -0.024820174379739057,
+            0.02666518374464728,
+            -0.017430076811109155,
+            -0.01495051200726725,
+            -0.005861181182847845,
+            0.00100881727574559,
+            -0.0055223282819321295,
+            0.0034719859452138017,
+            -0.010180752654175722,
+            -0.009961678082468036,
+            -0.012134423047419183,
+            -0.028073672819317354,
+            0.011574683159318163,
+            0.016120674453124317,
+            -0.015043147648505721,
+            0.031834693135959095,
+            -0.008802805725050891,
+            -0.0018698230152007877,
+            0.016179742266521146,
+            0.005096232332051617,
+            -0.017187077376166815,
+            -0.018999569809271876,
+            0.012251308792871571,
+            0.013279090498325945,
+            -0.009588297039279112,
+            0.006630431200804809,
+            -0.014130176783207386,
+            -0.005092662672039885,
+            0.007020236623084732,
+            -0.005049363863784624,
+            0.0055051712211635915,
+            -0.00995314895304919,
+            0.0052743456988306485,
+            0.005841370337169303,
+            -0.0029434616355351975,
+            0.004677049987338193,
+            -0.018568941264514256,
+            0.020031253265629777,
+            0.0018725390936655962,
+            0.01332532735283085,
+            -0.03518855189467458,
+            -0.0056298723325832025,
+            -0.005369475318146738,
+            -0.004650364775945294,
+            -0.020833510091412353,
+            0.020445280915606132,
+            0.0159991245532085,
+            0.013202409908533162,
+            0.023094507005426774,
+            -0.014905666713082246,
+            0.01572052456975675,
+            -0.020945766915121455,
+            0.013114000985348841,
+            0.003349101658930482,
+            -0.019829605346106578,
+            -0.001423750749031516,
+            0.04557459007627153,
+            -0.005847231973329081,
+            0.011986110554869603,
+            0.0012709842140249691,
+            0.026758922819943853,
+            -0.0001635536762342011,
+            0.0065419519929442995,
+            0.005997692702992206,
+            -0.009935128108338126,
+            -0.0008057798606777003,
+            0.00865712443881613,
+            0.046254320605750854,
+            -0.01181132866321511,
+            0.008170688314239364,
+            0.013099051984057827,
+            0.0034971494510865624,
+            -0.0008386400616584293,
+            -0.015346506644390764,
+            -0.03067678541405639,
+            0.019305962963428017,
+            0.007736916644077731,
+            -0.012756525606221703,
+            0.008734673666457216,
+            0.015137771675476126,
+            0.012891824153867332,
+            0.003895886821685762,
+            0.02155154331885366,
+            -0.012761690789748027,
+            0.018528313792219413,
+            0.024663453412919027,
+            0.009077230086690327,
+            9.184526542175535e-05,
+            -0.000751529849753638,
+            0.006791927267880182,
+            -0.0004433479107716552,
+            -0.009832230053513483,
+            0.015858232096630655,
+            -0.014861341148074698,
+            0.017790924034108806,
+            2.7216806238946785e-05,
+            0.03744841066552559,
+            -0.015216207585985075,
+            -0.010138319613387346,
+            -0.007774325714088762,
+            -0.0016997034352120362,
+            0.003971564314593291,
+            0.0018686537276849653,
+            0.006051669054361943,
+            -0.0008319624938748146,
+            -0.017326561560776794,
+            0.01667737942301944,
+            0.005719808546053257,
+            -0.0007405637872110778,
+            0.013750752065593804,
+            0.0038732323799683453,
+            -0.00367645612814479,
+            0.023580906347714314,
+            0.006238541579764762,
+            -0.005235795826199542,
+            0.0028182323816636243,
+            -0.006443210286549713,
+            -0.016255209299456314,
+            -0.003061441395209219,
+            0.004597970789372988,
+            -0.0016590920570710628,
+            -0.011138261849069419,
+            -0.0019423173609729848,
+            0.020355534503272887,
+            -0.004612410336124759,
+            0.013346750536413418,
+            -0.005400060374036387,
+            0.006252154576727367,
+            0.010601459925435238,
+            -0.012346569079682625,
+            0.0014154620309235822,
+            0.010406076751218612,
+            0.0013983417564564663,
+            -0.007140180075225315,
+            0.0010544410919327675,
+            0.007757599672903709,
+            0.006505350216372801,
+            0.00483063937962671,
+            0.00836630436955631,
+            0.013284451514874498,
+            0.0033749897293287438,
+            -0.006243293993633622,
+            0.0013949831763016042,
+            -0.013269921385800992,
+            -0.006308639321247727,
+            0.0010906169976769569,
+            -0.0010152835709869353,
+            -0.02017142197515583,
+            0.010787630901414118,
+            0.005355784839945777,
+            0.007838485470991034,
+            -0.01706932891696025,
+            0.009576030158827565,
+            -0.0008144311254998525,
+            0.02829832828245172,
+            0.007407953684956539,
+            -0.02769403281764041,
+            0.014266001222243134,
+            -0.010492199740754951,
+            0.022575782293320284,
+            -0.0016349316593207844,
+            -0.005656505606143177,
+            0.008025514398322551,
+            -0.009634281579228592,
+            -0.005991026158213869,
+            -0.022366083313133788,
+            -0.025825209755160946,
+            0.016871087849192924,
+            0.013583173973358155,
+            0.021125252303971605,
+            0.00012555627355712715,
+            0.015347535950809997,
+            0.0020128011217859444,
+            0.015164099887550086,
+            0.0024544903332736167,
+            0.012507136277293888,
+            -0.011798050869794805,
+            0.0104828761281283,
+            -0.0137805761114781,
+            0.014620357150314097,
+            -0.005246639228547017,
+            -0.006986408834884282,
+            -0.008849580196616098,
+            -0.005442165099441538,
+            -0.020891551850452786,
+            -0.009155930286949747,
+            -0.0013590151948670868,
+            0.010665872513236615,
+            -0.0005386423407222223,
+            -0.0027671778465775897,
+            -0.03237311596935737,
+            -0.004875062533405715,
+            -0.006670982068843501,
+            -0.01942244110096111,
+            -0.0045975373348006885,
+            -0.026137783023267008,
+            -0.01657173843643047,
+            -0.0007784046498232684,
+            -0.015037306676062236,
+            -0.006889680260149007,
+            -0.02009705080543122,
+            -0.027192867101205106,
+            3.0161964469866572e-05,
+            -0.009647644858034833,
+            -0.005380165957043753,
+            0.013815092649296373,
+            0.011902366689169847,
+            -0.01572592170758115,
+            -0.022321702398924333,
+            -0.01088937403483412,
+            -0.0165754067023749,
+            0.01646435743868707,
+            0.009914938007210463,
+            0.018834863207293908,
+            -0.003724245912768107,
+            0.005643505694722323,
+            0.0009581443359000331,
+            0.02327704258596552,
+            -0.02532422120223976,
+            -0.028694220894404807,
+            0.0019298778426469338,
+            0.0025970175688368323,
+            0.006086164866310174,
+            -0.009414665394188936,
+            0.001200390869257535,
+            -0.007782899559317819,
+            -0.0228264209954853,
+            -0.00898519580978142,
+            0.008156835270668449,
+            -0.010036630164522355,
+            0.02230664068547531,
+            0.012332230358869292,
+            0.013699197379189934,
+            0.010593587938184292,
+            0.01576174456497367,
+            -0.0001654241831584687,
+            0.014471905257729428,
+            -0.008675915624727887,
+            -0.0031070346099204425,
+            0.0038358404385932777,
+            0.005878417037187274,
+            -0.00625022998235275,
+            0.028077114529659795,
+            -0.0011836453716817525,
+            0.0002962695480296182,
+            0.014405782572641941,
+            0.02697667559474498,
+            -0.004710313427099278,
+            0.0210512043261309,
+            -0.002393165025068831,
+            -0.006135816286212752,
+            -0.008429776018235104,
+            0.008607617299702318,
+            -0.009529291136733401,
+            0.011133497451942684,
+            0.019864404836361926,
+            -0.014041315457579166,
+            -0.007351413712676263,
+            0.02950254974910368,
+            -0.014564388091367897,
+            -0.01825317106331622,
+            0.02325964952838705,
+            -0.007414295676409368,
+            -0.0039042587057605484,
+            0.00377144546378926,
+            0.007169856503233239,
+            -0.004036744966949995,
+            -0.012928341573802416,
+            0.016090527447479415,
+            -0.00011159893712484579,
+            0.014146137810890391,
+            -0.0004655839804281017,
+            -0.006372231942983921,
+            -0.010047382825720416,
+            0.002373224247019062,
+            0.026818813126567614,
+            0.013483627627077692,
+            -0.017895710143841333,
+            -0.013785856764479817,
+            0.010826610341182479,
+            -0.01538893818197323,
+            -0.004425206001945227,
+            0.02155595194812746,
+            0.0028768153668128553,
+            -0.020543027467698483,
+            0.008028975884363618,
+            0.0003835165718138906,
+            0.00037046947047892953,
+            -0.00029363296913641716,
+            0.030135739208689217,
+            0.001343516577762509,
+            0.00018948658255495327,
+            -0.00875053830853609,
+            0.008606573235959595,
+            -0.01280305089741873,
+            0.012786478836532904,
+            -0.02437993139337839,
+            0.007648181775166282,
+            -0.00629541989541625,
+            0.004579605315599212,
+            0.016658386927051592,
+            0.01218098627022323,
+            0.015031618813778413,
+            0.008237200519537417,
+            -0.008942965899756683,
+            0.0009093265012901892,
+            -0.025302175262520477,
+            -0.001333639155622843,
+            -0.017971876771137367,
+            -0.02036433247634014,
+            0.0002463842837918767,
+            0.022144712689630985,
+            0.018559482073124734,
+            0.011579840028464503,
+            -0.008657014440886656,
+            0.0166696911919226,
+            -2.9834620942882283e-05,
+            0.011845827764518911,
+            -0.00037901176785495324,
+            -0.006138374154553732,
+            0.004711058953619512,
+            -0.003622694588720417,
+            0.013542405259906128,
+            -0.014730567462694827,
+            0.012076044672004906,
+            0.01690223095263866,
+            -0.009108632851807137,
+            -0.008946251274440141,
+            -0.013992423874475428,
+            0.00048381889145447367,
+            -0.015909599902931202,
+            0.035489252674454214,
+            0.02252772349379229,
+            -0.01210464970923921,
+            -0.010469120693943236,
+            -0.014164999313775756,
+            0.019496048806752753,
+            0.010427539351244087,
+            0.0028018908451425276,
+            0.017091768406027965,
+            0.021395638416767112,
+            0.008817247593404623,
+            -0.006081063711583319,
+            -0.0021216503501144983,
+            -0.003781810594530787,
+            0.020143098218482827,
+            -0.003078172339607037,
+            -0.011350171884507555,
+            -0.012516379645909518,
+            0.005766102930704529,
+            -0.01626433668691071,
+            0.0019294643928566657,
+            0.008641558521764407,
+            -0.015057688533773374,
+            0.019407986696657772,
+            -0.028126197703647868,
+            0.012364374199657305,
+            -0.0044278434183731225,
+            -0.007251147925369614,
+            -0.03496220945653279,
+            0.006389489026974697,
+            0.008941259259632024,
+            0.003941752277454314,
+            0.023003838655784314,
+            -0.024528514847470928,
+            -0.018361931455105486,
+            0.0037791061321962543,
+            0.00047162060848090104,
+            -0.00711894577052754,
+            0.0010138871688747137,
+            0.004128392769213672,
+            -0.00674546985015164,
+            0.0017307097443508704,
+            0.0033354800298095546,
+            -0.002829898077850263,
+            0.013252947102333348,
+            0.0081724269619999,
+            -0.021449979263734438,
+            0.001368791856860285,
+            0.01590656734732547,
+            0.01452008402429858,
+            -0.02391847163018108,
+            0.006331397860689212,
+            -0.02064812934794864,
+            -0.000631340242718475,
+            -0.031272866000888556,
+            0.016859793097564958,
+            -0.0069008624795683276,
+            0.01899718801899956,
+            -0.0015377436942771224,
+            -0.0065810685260109004,
+            -0.012822620655323715,
+            0.02137604937338179,
+            -0.0019121137078344135,
+            -0.0025598772127157477,
+            0.015678293057066646,
+            -0.025700865215693987,
+            0.017825186616033165,
+            -0.0026230848396158204,
+            -0.017342967698019966,
+            0.017199139867162275,
+            0.004536019178633225,
+            0.007224666811309087,
+            -0.023505589665315567,
+            0.0007208041198255396,
+            0.003900164485347562,
+            -0.017250324025622685,
+            0.004470862192961212,
+            -0.030807400122398566,
+            0.0006579147990327597,
+            -0.013222907940559794,
+            -0.0020514994691204847,
+            -0.0046763235724575445,
+            -0.014050035576183385,
+            -0.039181147615786484,
+            -0.005429639279537522,
+            0.016065742005148733,
+            -0.0009029636981539927,
+            0.01800715198737081,
+            -0.002730931639143803,
+            0.002947680833843522,
+            0.004912692811020822,
+            0.00818378920043427,
+            0.006915895602313347,
+            0.02274116906914758,
+            -0.0057854922886970365,
+            0.00471999900678522,
+            0.007058014277137233,
+            0.014638081356210031,
+            -0.01719565951201012,
+            -0.015170377598372723,
+            -0.016015921063369496,
+            -0.006351213642269936,
+            0.01739665344303718,
+            -0.017488885259028475,
+            0.013303332445465686,
+            -0.00429428898520701,
+            0.014825338166443021,
+            -0.008686427208690176,
+            -0.01575945086724703,
+            0.0028564767084934386,
+            -0.007570417028925067,
+            0.001007670647768344,
+            -0.02323908008484089,
+            0.02019729564706863,
+            -0.00847581038203716,
+            0.010508144473197669,
+            0.01729824052022313,
+            -0.021364529179605813,
+            -0.009742851959432839,
+            -0.014342281441152413,
+            0.029896210049029176,
+            0.013925163184699139,
+            -0.009392252140772913,
+            0.003927230832659267,
+            0.008873561269230905,
+            0.011529536139797032,
+            -0.012209070214071445,
+            -0.0027679974125029362,
+            0.002411914349172495,
+            -0.0062056207925691356,
+            -0.002025687569076897,
+            0.03090134105002816,
+            0.0038285457265343245,
+            0.024877474647546086,
+            -0.013214997316123852,
+            -0.0033763842539861907,
+            0.00023379106197363714,
+            0.005134764372494098,
+            0.01183867975321486,
+            -0.0017966997954747142,
+            -0.008029791503028367,
+            -0.01179500476055557,
+            0.02599911330072515,
+            -0.021852083993691557,
+            0.007024505421035273,
+            0.026784024996764876,
+            0.001390202993990949,
+            0.0029956956045771046,
+            -0.013131691460546865,
+            -0.006298736864509989,
+            0.02523640664202355,
+            0.010030824059858138,
+            0.0041523782603032225,
+            -0.026465819482580182,
+            0.0032128899272377593,
+            -0.0061901409332948115,
+            -0.008681758126710213,
+            0.009866461769956222,
+            -0.0028095454857224164,
+            0.01188012041754054,
+            -0.012191996383095982,
+            0.028459939543805705,
+            0.010711502132045176,
+            0.01047154118931583,
+            0.025888652452698974,
+            -0.02079614305315134,
+            0.0042203106489994615,
+            -0.010829796140534066,
+            0.0006773493086316061,
+            0.005596640511307504,
+            -0.027510562413905437,
+            0.001969898603871003,
+            0.010390866563404346,
+            -0.005642706366801436,
+            -0.01882994179544761,
+            0.01187970346127799,
+            -0.019814385265412387,
+            0.024479579275617887,
+            0.003231341626501863,
+            0.020056914356101588,
+            -0.0053276448188067154,
+            0.009834685132275868,
+            0.0002742942778577165,
+            0.0031361331890882127,
+            -0.0008074598092128465,
+            0.009009012303347016,
+            0.005786777410144894,
+            0.013719989127068968,
+            -0.025528370602188686,
+            0.006821572597883499,
+            -0.0029197127345352626,
+            -0.030539872624857384,
+            -0.013383097680839732,
+            0.010291485622655026,
+            0.012218533188117704,
+            -0.009144425807890594,
+            -0.010378679518611277,
+            0.0030042579827305603,
+            0.005748243116238633,
+            -0.006692403885450407,
+            0.006913672151000981,
+            0.01629516912493545,
+            -0.002533517074771728,
+            -0.0019083178803585312,
+            -0.0035007340321378893,
+            -0.01154609935927537,
+            -0.010016998439072824,
+            0.023193275321990937,
+            -0.00482990342915347,
+            0.017943025772239603,
+            0.025873224580389702,
+            0.0039876567755322,
+            -0.0057577932736143506,
+            -0.005460363755206188,
+            0.01478435999841998,
+            0.014195457012279476,
+            0.004669295894868759,
+            -0.009702220679239768,
+            0.007974305230005956,
+            -0.007760086293456636,
+            0.008375077126777163,
+            -0.020260917110786147,
+            0.002199401871514877,
+            0.0026191373298767766,
+            -0.0022089262577237776,
+            -0.015213807046175918,
+            -0.01956788452420682,
+            -0.021898368819136302,
+            0.012634659463647929,
+            0.00034643102239488474,
+            -0.009241772775582378,
+            -0.013645250997151429,
+            0.0016665106990117016,
+            0.0025630500346914592,
+            0.0002095521560180936,
+            -0.007783022628694597,
+            0.009546147813008391,
+            -0.01265287317178631,
+            0.007445223311736028,
+            -0.004007029181436962,
+            0.0051846833906574515,
+            -0.0016916014415925766,
+            -0.016413938859402438,
+            -0.0210868129634149,
+            0.01920794362224444,
+            -0.012897042227841011,
+            0.02269303737601583,
+            -0.0014935878147854765,
+            3.062680855559304e-05,
+            0.0033132626755401546,
+            0.007274487651476732,
+            0.012110042834251237,
+            0.011426424896797977,
+            0.004307086698938411,
+            0.008206932254416481,
+            0.004673427176334148,
+            -0.013124571552764586,
+            0.004324963759928715,
+            0.0039216033705804084,
+            -0.010449887922072297,
+            0.005298323150690876,
+            -0.0010196552573491,
+            0.006128327172621373,
+            0.012167492341907742,
+            0.01923525204293908,
+            0.01908107550907106,
+            -0.0012031208519043073,
+            -0.0010601912677170903,
+            0.014474374858125607,
+            0.004540954805526089,
+            -0.004883537463424679,
+            -0.002772988629590886,
+            -0.013363713078069854,
+            -0.002572403646338979,
+            0.011541515690646252,
+            -0.02376161747414617,
+            -0.018215231053002656,
+            0.021831219299143226,
+            -0.004991882316041097,
+            0.033796174834149546,
+            -0.013140970283091536,
+            -0.016197177724633614,
+            -0.00464190076219332,
+            0.0026389653040644607,
+            -0.013788049032378498,
+            -0.010300859744205445,
+            0.010545541278382187,
+            0.013848738866656793,
+            0.009903590151120531,
+            -0.0005984085581408575,
+            0.02922816576443645,
+            0.0019900932217491907,
+            0.02245395592422119,
+            0.0018978407151245295,
+            -0.007029294098689854,
+            -0.01440575120349887,
+            0.010840472832356266,
+            0.005529335652955532,
+            0.009050748012161991,
+            0.017791269808484802,
+            0.014502057454389247,
+            0.006265567405788131,
+            0.013928705812292101,
+            0.0012903002372833693,
+            0.0010858377120770652,
+            -0.002355229463060164,
+            -0.01692711767136341,
+            -0.01456786846097011,
+            -0.01627640642218236,
+            0.003873864312554098,
+            0.012620350492241247,
+            0.00525890570023481,
+            0.0007393789113866752,
+            -0.007153210751899716,
+            0.009254566395058412,
+            0.0065087881512835375,
+            -0.012279152412209152,
+            -0.016530068283509593,
+            0.004060671955726967,
+            -0.02210570272789274,
+            0.007849467899009437,
+            0.01554233481902349,
+            0.00219885032769439,
+            -0.002655449524219968,
+            0.0006843524304842362,
+            0.006272944169764281,
+            0.005531927368102191,
+            -0.0036710328198476905,
+            -0.00544113514775027,
+            0.011264661763762068,
+            -0.006379654234440906,
+            0.009537150372938296,
+            -0.00290031548037575,
+            0.03718190949929003,
+            -0.011976080887998194,
+            0.012120408392610773,
+            -0.001751472085435711,
+            0.008481703962951994,
+            0.008067669857949423,
+            0.010772687289903202,
+            -0.003805089040741351,
+            -0.0022164543131735167,
+            -0.008753625032781523,
+            0.006751953442322185,
+            -0.01582177621255226,
+            0.013631301238130705,
+            -0.00030685726238985963,
+            -0.010728958775974582,
+            0.01055472059121978,
+            0.006788316621149835,
+            0.004413382316882087,
+            -0.003262024961860459,
+            -0.01972283203505511,
+            0.014937729522976615,
+            0.0018799533869990667,
+            0.019247008681353865,
+            -0.023276106189004046,
+            0.029043727846427652,
+            -0.02019024213343764,
+            0.023083643395685142,
+            0.01865875504942157,
+            0.001179436167200093,
+            -0.02070329911663634,
+            -0.007618131527214554,
+            0.014871709122115308,
+            -0.02659818889735726,
+            -0.006234719259825804,
+            0.005986829551040766,
+            -0.014639912579831785,
+            0.00037813423546252645,
+            -0.01924149909676064,
+            0.00010832498610532006,
+            0.011013130149723431,
+            -0.011782407678373089,
+            0.00438725832178584,
+            0.01608349750921335,
+            0.019347883652009713,
+            -0.009760017612545174,
+            0.003869246694835168,
+            -0.024240324890794748,
+            0.010958789926998691,
+            0.020511684636649225,
+            0.008883830392748497,
+            -0.01761944522796127,
+            0.021817973960311446,
+            0.010652131228027521,
+            -0.012575679152857703,
+            -0.0012753095453828826,
+            -0.028191274775392804,
+            -0.012421170815054556,
+            -0.02188248722244861,
+            -0.0011670175621882146,
+            -0.014436683528976771,
+            -0.0029783880907228383,
+            0.0009506986717516226,
+            -0.004719165363484719,
+            -0.01179163390107033,
+            -0.019543732196676634,
+            0.016606805875498898,
+            0.0227313413004497,
+            -0.006738735201626204,
+            -0.014547240981002825,
+            0.013866974327185123,
+            0.031694077725890536,
+            -0.016189209297529324,
+            -0.00542110554672736,
+            0.0028061798782555557,
+            -0.012964876141500577,
+            0.018946046750284537,
+            0.0039419760434645495,
+            0.000322115192590907,
+            -0.01991851369210666,
+            0.002489725351392778,
+            0.010835458511052207,
+            0.01684964045708685,
+            0.0005541314524621196,
+            0.010855852688653808,
+            -0.007549882748861744,
+            0.010096800951420057,
+            -0.011832260836468588,
+            -3.6179995603517093e-06,
+            -0.007800366796516945,
+            0.007001049318404058,
+            -0.004424776169203575,
+            0.034356203288485945,
+            -0.02817071276766299,
+            0.004989631403681216,
+            -0.014539644354369607,
+            -0.010888763939716889,
+            -0.0027322086509797843,
+            0.0004865534340981866,
+            -0.010775238981581461,
+            -0.005162220412866082,
+            -0.0011263271110752205,
+            0.00913081242511139,
+            0.005844327320514489,
+            -0.013886529519306775,
+            0.019171827605726503,
+            -0.030984919318656618,
+            0.004469458202753611,
+            -0.008553175891755315,
+            0.010511707988936336,
+            -0.0151185694875566,
+            0.013830711686954931,
+            0.01301959345025159,
+            0.008205984085944315,
+            0.007740367849216328,
+            -0.01532123052721834,
+            -0.01678314222392063,
+            0.001376468747213867,
+            0.0036953018347776244,
+            -0.00931855481896886,
+            -0.033391028922085425,
+            -0.007072913446166969,
+            0.005331350534425606,
+            0.011614453225043,
+            0.008156128603003278,
+            -0.02345206873048703,
+            0.005003870024809975,
+            -0.024178720646704597,
+            -0.012115112684614331,
+            -0.007750557665424973,
+            -0.011580152272614108,
+            -0.006117298954192534,
+            0.026449765548206453,
+            0.007989237291215073,
+            0.016691517078382376,
+            0.004954133565782356,
+            0.02742809127725913,
+            0.011670893621991404,
+            -0.011736840457474895,
+            -0.003349151747880166,
+            0.00035419034843196817,
+            0.005507651806218584,
+            -0.008706590052024251,
+            0.011793241980257321,
+            -0.0013295035978611142,
+            -0.0065061682758755625,
+            -0.004398354347597199,
+            0.009372816666628725,
+            0.014188676762501037,
+            -0.003007980559117429,
+            -0.008724255229333351,
+            0.015749776067989826,
+            -0.030899793617096834,
+            -0.015215033379100091,
+            0.0016152308157299517,
+            -0.007813175908301852,
+            0.017806770680528907,
+            0.01169056345888971,
+            -0.026835412610223673,
+            -0.004538235170332766,
+            0.010532316940606419,
+            0.004755930024691789,
+            0.0095408889025506,
+            0.00954782703036282,
+            0.012079064562163901,
+            -0.007829549014214643,
+            0.01046722658021748,
+            -0.011519369840999348,
+            -0.007092961453184232,
+            -0.007241799314569802,
+            -0.008077934089061052,
+            0.021467547268987214,
+            -0.025816030568238717,
+            -0.006558853543772633,
+            0.01134670818701549,
+            0.013027491377727365,
+            -0.002301991242932548,
+            -0.006299073927206214,
+            -0.017357989759336386,
+            0.008311464809363397,
+            -0.013922568511594478,
+            0.025633907424998367,
+            -0.004854655498817219,
+            -0.005336807899100369,
+            -0.005228056880188717,
+            0.01958682721272445,
+            -0.016774409737540904,
+            0.02072573220949266,
+            -0.014449250679704904,
+            -0.007857351393427215,
+            0.001673929749689194,
+            0.006890008607998747,
+            -0.002948835017234416,
+            0.03734128725793149,
+            -0.00556474133167756,
+            -0.0014544201137417158,
+            0.02209084868209573,
+            -0.01883610080898967,
+            0.017901421953179442,
+            0.01563604483998435,
+            0.012327159100148019,
+            0.0018161034827768648,
+            -0.010253034391420751,
+            0.019979728662282904,
+            0.020823925491027635,
+            0.0139490143219451,
+            0.009577521724445478,
+            -0.013478761809038993,
+            0.014934933784421556,
+            -0.0003725422088504494,
+            -0.0024362115214204105,
+            0.012785534912074068,
+            0.016441944143888537,
+            -0.02164522371903646,
+            -0.013152102018736346,
+            0.0063548333449661805,
+            -0.016547629104776748,
+            -0.024875443975589356,
+            -0.008991095813132973,
+            -0.021885127280977718,
+            -0.023336714382596175,
+            0.00688008599534903,
+            0.016482610587564443,
+            0.0035797836729533367,
+            0.013111012464332918,
+            0.006820471349350837,
+            0.018274678506489214,
+            -0.00980248993077149,
+            -0.01582838461385605,
+            -0.0009454586621769173,
+            -0.0004906493597104733,
+            0.009787688665732045,
+            -0.01835583040075359,
+            -0.013977939077760832,
+            0.004340902272034388,
+            -0.020118484730757877,
+            0.01421736173162507,
+            -0.006428386730058143,
+            0.016756376129345597,
+            -0.0004630315831872392,
+            0.011701948921076007,
+            0.02621773278421472,
+            -0.001391567406453007,
+            0.01262026995055899,
+            0.004696906079976175,
+            -0.0030577364576645635,
+            -0.010624066224426196,
+            0.0009420498021845824,
+            -0.0044079288776429516,
+            -0.0004845606666642234,
+            0.006481113327122954,
+            0.020269778435314,
+            -0.03321232649491074,
+            -0.009425498978427855,
+            0.01246754422779043,
+            -0.001658574527323383,
+            -0.005816985985181387,
+            0.013050062223188965,
+            0.00011195028631856881,
+            0.016573827662919836,
+            -0.0017453203044868104,
+            0.01417371165305189,
+            -0.008165910519539772,
+            -0.0007461950113024975,
+            0.0016492309267666983,
+            0.005076015595439873,
+            0.005829299318055953,
+            0.012146800191035666,
+            0.01148884639506138,
+            -0.021051327863689618,
+            -0.0026357096198729156,
+            0.0036146148489310443,
+            -0.010673282438301611,
+            0.014967444393862347,
+            0.005737933867764028,
+            0.01391411712868596,
+            0.0029556189893790237,
+            0.01167806258173173,
+            -0.024440974650111174,
+            -0.01029944007931845,
+            0.01385509328958837,
+            0.00869671728965227,
+            -0.00998315356917989,
+            -0.016580053751364735,
+            -0.007061005072623325,
+            -0.00477175200214453,
+            -0.0021182980051220428,
+            -0.009871524928312118,
+            -0.0005933941177237234,
+            0.006284920058476413,
+            5.466820734762947e-05,
+            -0.012842295882002122,
+            0.015810530885294202,
+            0.004572385791133295,
+            0.003685966629168175,
+            0.005093714133656017,
+            0.011877541830396259,
+            -0.013130170365831912,
+            0.01258190257102377,
+            0.011433121961378686,
+            0.01142665981860032,
+            -0.013710025169815697,
+            -0.001774030746825879,
+            -0.01109881340948764,
+            0.0065684584345163735,
+            0.009983753894796107,
+            0.005346443477816739,
+            -0.002062387245572686,
+            0.004950044847352647,
+            0.006105199826576752,
+            -0.06391185159544624,
+            -0.02131511418723256,
+            0.0027165737784844413,
+            0.033389688791557785,
+            -0.007167593552251769,
+            -0.0037646833460996797,
+            0.002883510089742145,
+            0.01356887065340832,
+            0.009730171586767032,
+            0.00418566245984083,
+            0.020320934773473808,
+            -0.008814826860651366,
+            -0.003014193747732543,
+            0.001559573791453607,
+            0.003499420226345028,
+            -0.008703837291821111,
+            0.0044844701302276645,
+            0.0053963630238108035,
+            0.027764369548110287,
+            0.001469968540767627,
+            0.015793086297019008,
+            -0.008505290826588508,
+            -0.004075888558568453,
+            -0.009166710762668385,
+            -0.002860047211143503,
+            -0.011127820563172684,
+            0.003937110717404271,
+            0.02054186721680431,
+            -0.006126180963243227,
+            0.009346433819831456,
+            0.01057504086030076,
+            0.0001650851447316893,
+            -0.012033423322524951,
+            0.0049587335923558,
+            -0.0032240390628760535,
+            -0.005568976687946761,
+            -0.0024667040429312737,
+            -0.03474027725033016,
+            0.026855947684394643,
+            -0.004369626674503285,
+            0.009252512430957932,
+            -0.005081834388513357,
+            -0.0037896610731020337,
+            -0.014201609270756739,
+            0.017871088029615106,
+            -0.008249444723433136,
+            -0.020092257756971717,
+            0.009532994544485572,
+            -0.004991989149115383,
+            0.006464417738670472,
+            -0.02643078108339741,
+            0.0029138922065812495,
+            -0.017232410749655366,
+            -0.003999293090834831,
+            -0.006924488971625995,
+            0.01318082244178942,
+            -0.007232695119856389,
+            -0.02660711037908311,
+            0.01979251106803667,
+            -0.019485860712345603,
+            -0.011977904020170568,
+            -0.0177530973032872,
+            -0.014524003566945937,
+            0.004874952064727108,
+            -0.015326044325348929,
+            0.002047771741041311,
+            -0.009697340332342246,
+            -0.003652304998155628,
+            -0.0008878698629674222,
+            0.0017782435347417623,
+            -0.013017776298713044,
+            -0.018272697230002275,
+            -0.01353020425912225,
+            -0.00757239449706798,
+            -0.0008250000619957538,
+            0.01299142281772714,
+            -0.001679606555140765,
+            -0.007329941421303076,
+            0.0044605462460153116,
+            0.010231192277666706,
+            0.0006558644644452857,
+            0.0024231733856020414,
+            -0.010728844680699885,
+            -0.014137434553169443,
+            0.005230053181525096,
+            -0.012086468475542818,
+            -0.009459717916037003,
+            -0.0028185227600210556,
+            0.014984527039312847,
+            0.008189464334213754,
+            -0.009139240774931914,
+            -0.005639157579163322,
+            0.014325949891859377,
+            0.009020303519690971,
+            -0.013310205617445609,
+            0.01567574314514385,
+            -0.025369802629268523,
+            0.021964786021408878,
+            0.030536905527039065,
+            0.004945288795196981,
+            -0.01168303908203322,
+            0.02281128628230144,
+            -0.004072286832145419,
+            0.0028278384348551146,
+            0.01740522162288968,
+            -0.042558465579514926,
+            -0.013146562547084278,
+            0.002903743316339479,
+            -0.015297177099602701,
+            -0.010963312697387696,
+            0.008804938803398296,
+            -0.0026727734429658817,
+            0.005273238046953793,
+            0.004257613402172282,
+            0.03512246556126453,
+            0.002258612643297096,
+            -0.0002885708728599208,
+            -0.0019720190285930487,
+            0.012312613129996533,
+            0.010451003939611476,
+            -0.015556620910136185,
+            -0.02387856865043814,
+            -0.004583090523619507,
+            -0.0037828776396525835,
+            -0.0014588238692793338,
+            -0.004361834167147271,
+            -0.008159679676329573,
+            0.012243319920992457,
+            -0.017279964797769668,
+            -0.013748032758693542,
+            0.018394195628121487,
+            -0.006411113522842359,
+            0.019002611495040334,
+            -0.014970362556522149,
+            -0.0077437581301661946,
+            0.0017311115715410025,
+            0.024439691379769795,
+            -0.025013956798520212,
+            0.007359134924497539,
+            -0.0031406470488477937,
+            0.017574824561755627,
+            0.00419409250132525,
+            -0.0046803792187546656,
+            0.010556972371044426,
+            0.015229789583480183,
+            -0.004608293492221483,
+            -0.014449487985966966,
+            -0.01172850291078532,
+            0.016823024209386785,
+            -0.012304887597699588,
+            0.0011773351576246157,
+            0.002262670226244277,
+            -0.0023810578015456716,
+            0.03477119000959688,
+            -0.03442178291363675,
+            0.004062693040714608,
+            0.0029249583507755944,
+            0.003291349443702718,
+            0.006461525089528308,
+            0.006047777261425826,
+            -0.020642802018978404,
+            0.008916088381876748,
+            -0.03316762661930299,
+            -0.006367254689957291,
+            -0.005404276207844267,
+            0.0009189006423526284,
+            0.009401330264376041,
+            -0.011106443887384533,
+            -0.02259405372588916,
+            -0.006010439054323966,
+            0.0070744551115721095,
+            0.010262607009446879,
+            -0.002856616377476421,
+            0.019719451109535182,
+            0.008426509638139982,
+            0.0015877648339862691,
+            -0.010655792508751512,
+            -0.015527851374834499,
+            -0.015460974859638219,
+            -0.017847462632188386,
+            -0.02236964311096428,
+            0.011841749854626384,
+            -0.00576688153869993,
+            0.009054902130717224,
+            0.014317707346504314,
+            -0.015366657284925976,
+            0.002910211008094564,
+            0.017761685988205086,
+            0.007974806171830457,
+            0.002692237107055498,
+            0.0032405477366987947,
+            0.02685230091638845,
+            -0.0009116884421301454,
+            -0.0032228977180251673,
+            0.003158213042682977,
+            0.015872199488167846,
+            0.0004966905858799325,
+            0.019981401695749582,
+            0.015201538351780685,
+            0.005359075814444414,
+            -0.004668738193285921,
+            0.019853988551405383,
+            -0.009844394286458195,
+            0.013476144827747423,
+            -0.00411789677405284,
+            -0.00830656434595337,
+            -0.008496132520586954,
+            0.006442581751999304,
+            -0.008138786054348571,
+            0.006977337946948795,
+            0.02419128791801387,
+            -0.0005734827808485311,
+            -0.0028842142002271044,
+            0.005503033191779872,
+            -0.01696639779775455,
+            -0.00389835761601045,
+            -0.03366077608692946,
+            -0.009942613075976742,
+            -0.01708630689497001,
+            -0.02910756562213045,
+            0.026536268897199546,
+            0.00219827495662445,
+            0.02586503862171508,
+            -0.01905122645609761,
+            0.008935637454859214,
+            -0.01385806889332683,
+            0.029747093931541325,
+            0.009163959012862627,
+            0.0008611776492791918,
+            0.003598831885031635,
+            -0.02103045281939351,
+            -0.01150941825152158,
+            0.027341774725604333,
+            -0.0006700608710615124,
+            -0.012089044078073347,
+            0.022396400579038037,
+            0.017491565611764764,
+            0.0051760367056280536,
+            -0.00559536785318233,
+            0.01979262595019399,
+            0.002306380546194741,
+            0.011973863166872444,
+            -0.0028136750606972223,
+            0.012911235736067303,
+            -0.01510421118359169,
+            -0.003338602157155174,
+            0.039758808572572274,
+            0.007290642519242007,
+            -0.001802921446521437,
+            -0.013696774990705774,
+            0.003941198181491921,
+            -0.0008308293948698925,
+            0.0200351513103389,
+            -0.00014327587644797734,
+            0.028291423013030337,
+            -0.0001085859276231927,
+            -0.010723064585989723,
+            0.020443848393732245,
+            -0.017860296930396913,
+            0.04540901629084378,
+            -0.018671578423261854,
+            0.000852031295629291,
+            -0.010639853151638835,
+            -0.019017194262730474,
+            0.003861296738239369,
+            0.0010977143012920602,
+            0.0008387482016061222,
+            0.0060155816314618,
+            -0.010780239332444652,
+            0.002152214497259421,
+            0.010556338812873054,
+            -0.027636192450621728,
+            0.0010938992111792307,
+            -0.014385202899929138,
+            -0.01353192238925814,
+            -0.026667778226211785,
+            -0.00017403398236549974,
+            -0.015357674131635848,
+            -0.013956722505869091,
+            -0.01985038526848704,
+            -0.008927903879720431,
+            -0.009580214158077466,
+            -0.0020314441852913386,
+            -0.004755584878514854,
+            -0.00018388548107372635,
+            -0.002918855567842974,
+            -0.0015941852262316128,
+            -0.014230272061190995,
+            0.02133992212037431,
+            0.015283345362445234,
+            0.004642765060980717,
+            0.012829570028541471,
+            0.01708952142208596,
+            -0.004770647543823141,
+            0.006604762788847952,
+            -0.0025369836845081078,
+            0.008368000846861623,
+            0.014785031726639733,
+            0.012074738887486398,
+            0.02124518119032885,
+            -0.015459818988572564,
+            -0.004992865135357168,
+            -0.012722391036055767,
+            -0.012210202603282509,
+            -0.017850065946244383,
+            -0.004300429148015694,
+            -0.0014387298144714717,
+            -0.023128300275416423,
+            0.004468952290723408,
+            -0.0026602931688877443,
+            -0.005768852559887139,
+            0.007815212168719318,
+            0.0005862972228454734,
+            -0.002265719947629544,
+            -0.00019323737674204774,
+            -0.02735820445266249,
+            0.00024009122317215594,
+            -0.00018701993387558173,
+            -0.014663710240203917,
+            -0.00450797561624596,
+            0.010584731989847334,
+            -0.011966527748437568,
+            -0.00602367292291343,
+            0.0111155226518017,
+            0.0015565179432852256,
+            0.006062862961835896,
+            0.016952559697728433,
+            0.0005012157272316869,
+            0.011160668242334889,
+            0.014993782560797772,
+            -0.01591303023837098,
+            -0.015356412848444082,
+            -0.0236025426434855,
+            -0.004514538154596937,
+            -0.009753334017065119,
+            0.007553951535816575,
+            -0.004616595532996391,
+            -0.013023153809236325,
+            0.006040285197946957,
+            -0.0006343451398642784,
+            0.008338564073158257,
+            -0.017608246105292328,
+            -8.346168119689692e-05,
+            -0.020944870308705715,
+            0.009230123321316848,
+            -0.00531237224627929,
+            -0.00445985089300067,
+            0.01410675485522911,
+            0.02895945580213926,
+            -0.007799830351886844,
+            -0.009455128322575888,
+            -0.005663072244883114,
+            0.012817229945992764,
+            0.012474431030896143,
+            -0.004423141749110039,
+            -0.0018294512146253298,
+            -0.012817101093394574,
+            0.0004755435377918658,
+            -0.01129485262081729,
+            0.0062747111125005155,
+            0.017199374798326412,
+            -0.021965117839794063,
+            -0.01296097993683748,
+            -0.016765996091978336,
+            -0.016882855472266523,
+            0.004372970343291638,
+            0.017290565252244258,
+            0.022670788364378462,
+            -0.008456547449078897,
+            -0.019564049850942417,
+            0.006898353991539946,
+            -0.01620125883985687,
+            0.0037108638122592746,
+            0.12887455743463638,
+            -0.0028864852421660366,
+            -0.011487804449328495,
+            -0.022683334123299446,
+            -0.005467192611880829,
+            0.01210557696712196,
+            -0.01555159294496261,
+            0.06709672179195036,
+            -0.015071089552631105,
+            0.004364902817496082,
+            -0.004899872946473657,
+            0.0006310685609937571,
+            -0.017610935588191347,
+            -0.005962991715597585,
+            -0.007768819050529025,
+            -0.02688607975562596,
+            0.000883763319535686,
+            -0.009935313912803587,
+            -0.012883104064962595,
+            0.002126368813790577,
+            -0.0013697265269800346,
+            -0.016131186838419288,
+            0.01575514418037863,
+            -0.020677990768697592,
+            -0.007251917191831732,
+            -0.0029325623780992118,
+            -0.00600710212646933,
+            0.014573093764769918,
+            -0.03253336117394778,
+            0.002973725493330224,
+            -0.013601417220439602,
+            0.011905788192118131,
+            0.001542117126832946,
+            -0.010789486980776022,
+            -0.013862139885514524,
+            0.007086571810314664,
+            0.015658913897173155,
+            0.017039624764452546,
+            0.01901374401756777,
+            0.004797954569087795,
+            0.003928999581093576,
+            -0.008046722893614655,
+            0.021974371139905104,
+            -0.00697335955527197,
+            0.0008975473383295958,
+            -0.00970589295353857,
+            -0.004346869073499034,
+            0.0031279458769925416,
+            0.02303983493541673,
+            -0.012560380384977343,
+            -0.01536378830925542,
+            0.0012035987157572355,
+            0.033313215063595594,
+            -0.010845084209743817,
+            0.013587126023659577,
+            0.0029867620644300073,
+            0.020679673435988446,
+            -0.0006640787253300063,
+            0.0025071780490108306,
+            0.022554193713122565,
+            -0.018082906254599396,
+            0.018230000220500718,
+            -0.004451773302171552,
+            0.01915083025790679,
+            -0.004183521043835777,
+            0.015019320101074747,
+            -0.027014055979204272,
+            -0.005362044589023088,
+            -0.012355559881051253,
+            0.003995309003198048,
+            0.020680455787871383,
+            0.0054391755324872375,
+            0.009865676762244514,
+            -0.01645386866657314,
+            -0.01760657627715638,
+            0.022969828924328126,
+            -0.021566687194482292,
+            -0.015826424849757402,
+            0.00012804774676298237,
+            -0.015021970627042762,
+            0.006000610438452896,
+            0.010126179023323158,
+            -0.02912121211344025,
+            -0.02046150879033127,
+            0.004881850039543051,
+            -0.0023469840669773042,
+            -0.00915786433296249,
+            0.006547377770403066,
+            -0.022329063637464074,
+            -0.015702777677743352,
+            -0.005692904224111613,
+            0.01207999389934878,
+            -0.008429160038192635,
+            0.010075061963131637,
+            -0.020106712664936504,
+            0.0036787045650556907,
+            -0.002764965946892891,
+            0.010307702251974629,
+            0.004697169247387421,
+            0.011192285826515207,
+            0.0030113901179382946,
+            0.0014774681536380047,
+            -0.02778270000997313,
+            0.010043261430740039,
+            -5.178406450222847e-05,
+            0.04216636255860709,
+            0.030977096295679525,
+            0.004479857028809617,
+            -5.798020745008712e-05,
+            -0.009996471945562616,
+            0.012476009017187798,
+            -0.019306907391548302,
+            -0.005092322343599383,
+            0.01856463597985508,
+            0.02195467141594795,
+            -0.02236232264139305,
+            0.0052217593371508145,
+            -0.016766955100332014,
+            0.016133299654723637,
+            -0.011208763137897244,
+            0.006248024409576809,
+            0.011601988603146743,
+            0.0022330878390492527,
+            0.010317781892717903,
+            -0.005895142237225119,
+            0.019670196605183973,
+            0.01186677200787666,
+            -0.002339367025714548,
+            0.02367408853110795,
+            0.006858134385159451,
+            0.004479683673962342,
+            0.004798719889930319,
+            -0.009604173975576537,
+            -0.009045602998213713,
+            0.009230969488015385,
+            0.004083616606774546,
+            -0.007944628589259331,
+            0.011836323287097144,
+            0.015100350195476968,
+            0.01885317701149666,
+            -0.0011707847499298092,
+            -0.00684548433712413,
+            -0.006550059418141426,
+            0.001899758393077906,
+            -0.024577381680573596,
+            -0.02855511854789239,
+            0.021238935710272536,
+            -0.010427135883958709,
+            -0.019187769881105637,
+            -0.0015864640188683347,
+            0.016360037306061392,
+            -0.008853972319021563,
+            -0.01627167079143546,
+            0.001270414822216867,
+            0.02255755055961612,
+            -0.005959411125391464,
+            0.006009654905468478,
+            0.005909233773418581,
+            0.005874361519568704,
+            -0.0027859463941992753,
+            -0.024599789577567636,
+            0.0036612374968223453,
+            0.03221346865585554,
+            0.016461563205582293,
+            0.016139782455492073,
+            -0.007158168059808459,
+            0.001838455290310879,
+            0.007530954381521755,
+            0.005438791523412295,
+            0.015674000912615536,
+            0.008604106101542998,
+            -0.006990462920227135,
+            0.00244736628387359,
+            0.012213048255418866,
+            -0.007329299946651028,
+            0.017424711328323798,
+            0.001973882532418776,
+            -0.0133178791530314,
+            -0.008373452013951822,
+            -0.010462150707776556,
+            -0.014575483571750984,
+            0.0027958704799249897,
+            -0.011401916623878957,
+            0.0008016041012669959,
+            0.003953958717263787,
+            0.010260753041492368,
+            0.013961649189650642,
+            -0.0019275791443041316,
+            0.017019168629298462,
+            0.011534594243860177,
+            -0.0017859096307142257,
+            -0.002970739203219076,
+            -0.03142778833370656,
+            0.010207789755749909,
+            -0.014730553506704018,
+            -0.0037611416875830746,
+            0.01825790347981639,
+            -0.007766890847082858,
+            -0.0032778433418881666,
+            -0.021003405429807113,
+            0.0024512381432602736,
+            0.008768616692363321,
+            0.01558713989712387,
+            -0.012819542714293536,
+            0.009201155584074711,
+            -0.0012254337002053017,
+            -0.0057451574939563045,
+            0.00977251392462179,
+            0.0042356831792011935,
+            -0.0033433823868435786,
+            0.011499764125503874
+          ],
+          "title": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer",
+          "keyphrases": [
+            "Sparsely-Gated Mixture-of-Experts Layer",
+            "conditional computation",
+            "neural network",
+            "language modeling",
+            "machine translation"
+          ]
+        },
+        "type": "document"
+      },
+      "target": {
+        "id": "65742957-6bda-446d-a61a-0fe20313c760",
+        "properties": {
+          "page_content": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity\n\nWilliam Fedus∗ liamfedus@google.com Barret Zoph∗ barretzoph@google.com Noam Shazeer noam@google.com Google, Mountain View, CA 94043, USA Editor: Alexander Clark\n\nAbstract\n\nIn deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) models defy this and instead select different parameters for each incoming example. The result is a sparsely-activated model—with an outrageous number of parameters—but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs, and training instability. We address these with the introduction of the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques mitigate the instabilities, and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large (Raffel et al., 2019) to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the \"Colossal Clean Crawled Corpus\", and achieve a 4x speedup over the T5-XXL model.12 Keywords: mixture-of-experts, natural language processing, sparsity, large-scale machine learning, distributed computing arXiv:2101.03961v3 [cs.LG] 16 Jun 2022 Contents\n\n1 Introduction 3 2 Switch Transformer 4 2.1 Simplifying Sparse Routing 5 2.2 Efficient Sparse Routing 6 2.3 Putting It All Together: The Switch Transformer 8 2.4 Improved Training and Fine-Tuning Techniques 8 3 Scaling Properties 11 3.1 Scaling Results on a Step-Basis 12 3.2 Scaling Results on a Time-Basis 13 3.3 Scaling Versus a Larger Dense Model 13 4 Downstream Results 14 4.1 Fine-Tuning 14 4.2 Distillation 16 4.3 Multilingual Learning 17 5 Designing Models with Data, Model, and Expert-Parallelism 18 5.1 Data Parallelism 20 5.2 Model Parallelism 20 5.3 Model and Data Parallelism 21 5.4 Expert and Data Parallelism 22 5.5 Expert, Model and Data Parallelism 22 5.6 Towards Trillion Parameter Models 22 6 Related Work 24 7 Discussion 25 8 Future Work 26 9 Conclusion 27 A Switch for Attention 27 B Preventing Token Dropping with No-Token-Left-Behind 29\n\nC Encouraging Exploration Across Experts 29 D Switch Transformers in Lower Compute Regimes 29 E Relation of Upstream to Downstream Model Performance 32 F Pseudo Code for Switch Transformers 33\n\n1. Introduction\n\nLarge scale training has been an effective path towards flexible and powerful neural language models (Radford et al., 2018; Kaplan et al., 2020; Brown et al., 2020). Simple architectures— backed by a generous computational budget, data set size and parameter count—surpass more complicated algorithms (Sutton, 2019). An approach followed in Radford et al. (2018); Raffel et al. (2019); Brown et al. (2020) expands the model size of a densely-activated Transformer (Vaswani et al., 2017). While effective, it is also extremely computationally intensive (Strubell et al., 2019). Inspired by the success of model scale, but seeking greater computational efficiency, we instead propose a sparsely-activated expert model: the Switch Transformer. In our case the sparsity comes from activating a subset of the neural network weights for each incoming example.\n\nFigure 1: Scaling and sample efficiency of Switch Transformers. Left Plot: Scaling properties for increasingly sparse (more experts) Switch Transformers. Right Plot: Negative log perplexity comparing Switch Transformers to T5 (Raffel et al., 2019) models using the same compute budget. Sparse training is an active area of research and engineering (Gray et al., 2017; Gale et al., 2020), but as of today, machine learning libraries and hardware accelerators still cater to dense matrix multiplications. To have an efficient sparse algorithm, we start with the Mixture-of-Expert (MoE) paradigm (Jacobs et al., 1991; Jordan and Jacobs, 1994; Shazeer et al., 2017), and simplify it to yield training stability and computational benefits. MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020), however, widespread adoption is hindered by complexity, communication costs, and training instabilities.\n\nWe address these issues, and then go beyond translation, to find that these class of algorithms are broadly valuable in natural language. We measure superior scaling on a diverse set of natural language tasks and across three regimes in NLP: pre-training, finetuning and multi-task training. While this work focuses on scale, we also show that the Switch Transformer architecture not only excels in the domain of supercomputers, but is beneficial even with only a few computational cores. Further, our large sparse models can be distilled (Hinton et al., 2015) into small dense versions while preserving 30% of the sparse model quality gain. Our contributions are the following: - The Switch Transformer architecture, which simplifies and improves over Mixture of Experts.\n\nScaling properties and a benchmark against the strongly tuned T5 model (Raffel et al., 2019) where we measure 7x+ pre-training speedups while still using the same FLOPS per token. We further show the improvements hold even with limited computational resources, using as few as two experts.\n\nSuccessful distillation of sparse pre-trained and specialized fine-tuned models into small dense models. We reduce the model size by up to 99% while preserving 30% of the quality gains of the large sparse teacher.\n\nImproved pre-training and fine-tuning techniques: (1) selective precision training that enables training with lower bfloat16 precision (2) an initialization scheme that allows for scaling to a larger number of experts and (3) increased expert regularization that improves sparse model fine-tuning and multi-task training.\n\nA measurement of the pre-training benefits on multilingual data where we find a universal improvement across all 101 languages and with 91% of languages benefiting from 4x+ speedups over the mT5 baseline (Xue et al., 2020).\n\nAn increase in the scale of neural language models achieved by efficiently combining data, model, and expert-parallelism to create models with up to a trillion parameters. These models improve the pre-training speed of a strongly tuned T5-XXL baseline by 4x.\n\n2. Switch Transformer\n\nThe guiding design principle for Switch Transformers is to maximize the parameter count of a Transformer model (Vaswani et al., 2017) in a simple and computationally efficient way.\n\nThe benefit of scale was exhaustively studied in Kaplan et al. (2020) which uncovered powerlaw scaling with model size, data set size and computational budget. Importantly, this work advocates training large models on relatively small amounts of data as the computationally optimal approach.\n\nHeeding these results, we investigate a fourth axis: increase the parameter count while keeping the floating point operations (FLOPs) per example constant. Our hypothesis is that the parameter count, independent of total computation performed, is a separately important axis on which to scale. We achieve this by designing a sparsely activated model that efficiently uses hardware designed for dense matrix multiplications such as GPUs and TPUs. Our work here focuses on TPU architectures, but these class of models may be similarly trained on GPU clusters. In our distributed training setup, our sparsely activated layers split unique weights on different devices. Therefore, the weights of the model increase with the number of devices, all while maintaining a manageable memory and computational footprint on each device.\n\nFigure 2: Illustration of a Switch Transformer encoder block. We replace the dense feed forward network (FFN) layer present in the Transformer with a sparse Switch FFN layer (light blue). The layer operates independently on the tokens in the sequence. We diagram two tokens (x1 = \"More\" and x2 = \"Parameters\" below) being routed (solid lines) across four FFN experts, where the router independently routes each token. The switch FFN layer returns the output of the selected FFN multiplied by the router gate value (dotted-line).\n\n2.1 Simplifying Sparse Routing\n\nMixture of Expert Routing. Shazeer et al. (2017) proposed a natural language Mixtureof-Experts (MoE) layer which takes as an input a token representation x and then routes this to the best determined top-k experts, selected from a set {Ei(x)} N i=1 of N experts.\n\nThe router variable Wr produces logits h(x) = Wr · x which are normalized via a softmax distribution over the available N experts at that layer. The gate-value for expert i is given by,\n\n$$p_{i}(x)=\\frac{e^{h(x){i}}}{\\sum{j}^{N}e^{h(x)_{j}}}.\\tag{1}$$ $$(1)$$\n\n$$\\left(2\\right)$$\n\nThe top-k gate values are selected for routing the token x. If T is the set of selected top-k indices then the output computation of the layer is the linearly weighted combination of each expert's computation on the token by the gate value,\n\n$$y=\\sum_{i\\in{\\mathcal{T}}}p_{i}(x)E_{i}(x).$$ pi(x)Ei(x). (2) Switch Routing: Rethinking Mixture-of-Experts. Shazeer et al. (2017) conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions. The authors intuited that learning to route would not work without the ability to compare at least two experts. Ramachandran and Le (2018) went further to study the top-k decision and found that higher k-values in lower layers in the model were important for models with many routing layers. Contrary to these ideas, we instead use a simplified strategy where we route to only a single expert. We show this simplification preserves model quality, reduces routing computation and performs better. This k = 1 routing strategy is later referred to as a Switch layer. Note that for both MoE and Switch Routing, the gate value pi(x) in Equation 2 permits differentiability of the router.\n\nThe benefits for the Switch layer are three-fold: (1) The router computation is reduced as we are only routing a token to a single expert. (2) The batch size (expert capacity) of each expert can be at least halved since each token is only being routed to a single expert.3 (3) The routing implementation is simplified and communication costs are reduced. Figure 3 shows an example of routing with different expert capacity factors.\n\nFigure 3: Illustration of token routing dynamics. Each expert processes a fixed batch-size of tokens modulated by the capacity factor. Each token is routed to the expert with the highest router probability, but each expert has a fixed batch size of (total tokens / num experts) × capacity factor. If the tokens are unevenly dispatched then certain experts will overflow (denoted by dotted red lines), resulting in these tokens not being processed by this layer. A larger capacity factor alleviates this overflow issue, but also increases computation and communication costs (depicted by padded white/empty slots).\n\n2.2 Efficient Sparse Routing\n\nWe use Mesh-Tensorflow (MTF) (Shazeer et al., 2018) which is a library, with similar semantics and API to Tensorflow (Abadi et al., 2016) that facilitates efficient distributed data and model parallel architectures. It does so by abstracting the physical set of cores to a logical mesh of processors. Tensors and computations may then be sharded per named dimensions, facilitating easy partitioning of models across dimensions. We design our model with TPUs in mind, which require statically declared sizes. Below we describe our distributed Switch Transformer implementation.\n\nSee Section 2.2 for a technical description.\n\n$$\\left({\\boldsymbol{3}}\\right)$$\n\nDistributed Switch Implementation. All of our tensor shapes are statically determined at compilation time, but our computation is dynamic due to the routing decisions at training and inference. Because of this, one important technical consideration is how to set the expert capacity. The expert capacity—the number of tokens each expert computes—is set by evenly dividing the number of tokens in the batch across the number of experts, and then further expanding by a capacity factor,\n\n$${\\mathrm{~expert~capacity}}=!!\\left({\\frac{{\\mathrm{tokens~per~batch}}}{{\\mathrm{number~of~experts}}}}\\right)\\times{\\mathrm{capacity~factor}}.$$\n\nA capacity factor greater than 1.0 creates additional buffer to accommodate for when tokens are not perfectly balanced across experts. If too many tokens are routed to an expert (referred to later as dropped tokens), computation is skipped and the token representation is passed directly to the next layer through the residual connection. Increasing the expert capacity is not without drawbacks, however, since high values will result in wasted computation and memory. This trade-off is explained in Figure 3. Empirically we find ensuring lower rates of dropped tokens are important for the scaling of sparse expert-models.\n\nThroughout our experiments we didn't notice any dependency on the number of experts for the number of tokens dropped (typically < 1%). Using the auxiliary load balancing loss (next section) with a high enough coefficient ensured good load balancing. We study the impact that these design decisions have on model quality and speed in Table 1.\n\nA Differentiable Load Balancing Loss. To encourage a balanced load across experts we add an auxiliary loss (Shazeer et al., 2017, 2018; Lepikhin et al., 2020). As in Shazeer et al. (2018); Lepikhin et al. (2020), Switch Transformers simplifies the original design in Shazeer et al. (2017) which had separate load-balancing and importance-weighting losses.\n\nFor each Switch layer, this auxiliary loss is added to the total model loss during training.\n\nGiven N experts indexed by i = 1 to N and a batch B with T tokens, the auxiliary loss is computed as the scaled dot-product between vectors f and P,\n\n$$\\operatorname{loss}=\\alpha\\cdot N\\cdot\\sum_{i=1}^{N}f_{i}\\cdot P_{i}$$ $$\\left(4\\right)$$\n\n$$\\left({\\mathfrak{h}}\\right)$$ $$\\left({\\mathfrak{h}}\\right)$$ fi· Pi (4) where $ f_{i}$ is the frac where fiis the fraction of tokens dispatched to expert i,\n\ndispatched to expert $i$, . $$f_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}\\mathbbm{1}{{\\mathrm{argmax}}\\,p(x)=i}$$ 1{argmax p(x) = i} (5) and Piis the fraction of the router probability allocated for expert i,\n\nrt $\\,i_1$ ? $$P_{i}={\\frac{1}{T}}\\sum_{x\\in{\\mathcal{B}}}p_{i}(x).$$ pi(x). (6) Since we seek uniform routing of the batch of tokens across the N experts, we desire both vectors to have values of 1/N. The auxiliary loss of Equation 4 encourages uniform routing since it is minimized under a uniform distribution. The objective can also be differentiated as\n\nthe P-vector is differentiable, but the f-vector is not. The final loss is multiplied by expert count N to keep the loss constant as the number of experts varies since under uniform routing PN i=1(fi· Pi) = PN i=1( 1 N · 1 N ) = 1N . Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10−2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10−1to 10−5 in powers of 10 and found 10−2 balanced load quickly without interfering with training loss.\n\n2.3 Putting It All Together: The Switch Transformer\n\nOur first test of the Switch Transformer starts with pre-training on the \"Colossal Clean Crawled Corpus\" (C4), introduced in (Raffel et al., 2019). For our pre-training objective, we use a masked language modeling task (Taylor, 1953; Fedus et al., 2018; Devlin et al., 2018) where the model is trained to predict missing tokens. In our pre-training setting, as determined in Raffel et al. (2019) to be optimal, we drop out 15% of tokens and then replace the masked sequence with a single sentinel token. To compare our models, we record the negative log perplexity.4 Throughout all tables in the paper, ↑ indicates that a higher value for that metric is better and vice-versa for ↓. A comparison of all the models studied in this work are in Table 9.\n\nA head-to-head comparison of the Switch Transformer and the MoE Transformer is presented in Table 1. Our Switch Transformer model is FLOP-matched to 'T5-Base' (Raffel et al., 2019) (same amount of computation per token is applied). The MoE Transformer, using top-2 routing, has two experts which each apply a separate FFN to each token and thus its FLOPS are larger. All models were trained for the same number of steps on identical hardware. Note that the MoE model going from capacity factor 2.0 to 1.25 actually slows down (840 to 790) in the above experiment setup, which is unexpected.5 We highlight three key findings from Table 1: (1) Switch Transformers outperform both carefully tuned dense models and MoE Transformers on a speed-quality basis. For a fixed amount of computation and wall-clock time, Switch Transformers achieve the best result. (2) The Switch Transformer has a smaller computational footprint than the MoE counterpart. If we increase its size to match the training speed of the MoE Transformer, we find this outperforms all MoE and Dense models on a per step basis as well. (3) Switch Transformers perform better at lower capacity factors (1.0, 1.25). Smaller expert capacities are indicative of the scenario in the large model regime where model memory is very scarce and the capacity factor will want to be made as small as possible.\n\n2.4 Improved Training And Fine-Tuning Techniques\n\nSparse expert models may introduce training difficulties over a vanilla Transformer. Instability can result because of the hard-switching (routing) decisions at each of these layers.\n\nFurther, low precision formats like bfloat16 (Wang and Kanwar, 2019) can exacerbate issues\n\nModel Capacity Quality after Time to Quality Speed (↑) Factor 100k steps (↑) Threshold (↓) (examples/sec) (Neg. Log Perp.) (hours) T5-Base - -1.731 Not achieved† 1600 T5-Large - -1.550 131.1 470 MoE-Base 2.0 -1.547 68.7 840 Switch-Base 2.0 -1.554 72.8 860 MoE-Base 1.25 -1.559 80.7 790 Switch-Base 1.25 -1.553 65.0 910 MoE-Base 1.0 -1.572 80.1 860 Switch-Base 1.0 -1.561 62.8 1000 Switch-Base+ 1.0 -1.534 67.6 780\n\nTable 1: Benchmarking Switch versus MoE. Head-to-head comparison measuring per step and per time benefits of the Switch Transformer over the MoE Transformer and T5 dense baselines. We measure quality by the negative log perplexity and the time to reach an arbitrary chosen quality threshold of Neg. Log Perp.=-1.50. All MoE and Switch Transformer models use 128 experts, with experts at every other feed-forward layer. For Switch-Base+, we increase the model size until it matches the speed of the MoE model by increasing the model hidden-size from 768 to 896 and the number of heads from 14 to 16. All models are trained with the same amount of computation (32 cores) and on the same hardware (TPUv3). Further note that all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50. † T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained.\n\nin the softmax computation for our router. We describe training difficulties here and the methods we use to overcome them to achieve stable and scalable training.\n\nSelective precision with large sparse models. Model instability hinders the ability to train using efficient bfloat16 precision, and as a result, Lepikhin et al. (2020) trains with float32 precision throughout their MoE Transformer. However, we show that by instead selectively casting to float32 precision within a localized part of the model, stability may be achieved, without incurring expensive communication cost of float32 tensors. This technique is inline with modern mixed precision training strategies where certain parts of the model and gradient updates are done in higher precision Micikevicius et al. (2017). Table 2 shows that our approach permits nearly equal speed to bfloat16 training while conferring the training stability of float32.\n\nTo achieve this, we cast the router input to float32 precision. The router function takes the tokens as input and produces the dispatch and combine tensors used for the selection and recombination of expert computation (refer to Code Block 15 in the Appendix for details). Importantly, the float32 precision is only used within the body of the router function—on computations local to that device. Because the resulting dispatch and combine tensors are recast to bfloat16 precision at the end of the function, no expensive float32 tensors\n\nModel Quality Speed (precision) (Neg. Log Perp.) (↑) (Examples/sec) (↑) Switch-Base (float32) -1.718 1160 Switch-Base (bfloat16) -3.780 [diverged] 1390 Switch-Base (Selective precision) -1.716 1390\n\nTable 2: Selective precision. We cast the local routing operations to float32 while preserving bfloat16 precision elsewhere to stabilize our model while achieving nearly equal speed to (unstable) bfloat16-precision training. We measure the quality of a 32 expert model after a fixed step count early in training its speed performance. For both Switch-Base in float32 and with Selective prevision we notice similar learning dynamics. are broadcast through all-to-all communication operations, but we still benefit from the increased stability of float32.\n\nSmaller parameter initialization for stability. Appropriate initialization is critical to successful training in deep learning and we especially observe this to be true for Switch Transformer. We initialize our weight matrices by drawing elements from a truncated normal distribution with mean µ = 0 and standard deviation σ =ps/n where s is a scale hyper-parameter and n is the number of input units in the weight tensor (e.g. fan-in).6 As an additional remedy to the instability, we recommend reducing the default Transformer initialization scale s = 1.0 by a factor of 10. This both improves quality and reduces the likelihood of destabilized training in our experiments. Table 3 measures the improvement of the model quality and reduction of the variance early in training. We find that\n\nModel (Initialization scale) Average Quality Std. Dev. of Quality (Neg. Log Perp.) (Neg. Log Perp.) Switch-Base (0.1x-init) -2.72 0.01 Switch-Base (1.0x-init) -3.60 0.68\n\nTable 3: Reduced initialization scale improves stability. Reducing the initialization scale results in better model quality and more stable training of Switch Transformer. Here we record the average and standard deviation of model quality, measured by the negative log perplexity, of a 32 expert model after 3.5k steps (3 random seeds each). the average model quality, as measured by the Neg. Log Perp., is dramatically improved and there is a far reduced variance across runs. Further, this same initialization scheme is broadly effective for models spanning several orders of magnitude. We use the same approach to stably train models as small as our 223M parameter baseline to enormous models in excess of one trillion parameters.\n\nRegularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.\n\nModel (dropout) GLUE CNNDM SQuAD SuperGLUE T5-Base (d=0.1) 82.9 19.6 83.5 72.4 Switch-Base (d=0.1) 84.7 19.1 83.7 73.0 Switch-Base (d=0.2) 84.4 19.2 83.9 73.2 Switch-Base (d=0.3) 83.9 19.6 83.4 70.7 Switch-Base (d=0.1, ed=0.4) 85.2 19.6 83.7 73.0\n\nTable 4: Fine-tuning regularization results. A sweep of dropout rates while fine-tuning Switch Transformer models pre-trained on 34B tokens of the C4 data set (higher numbers are better). We observe that using a lower standard dropout rate at all non-expert layer, with a much larger dropout rate on the expert feed-forward layers, to perform the best.\n\nWe thus propose a simple way to alleviate this issue during fine-tuning: increase the dropout inside the experts, which we name as expert dropout. During fine-tuning we simply increase the dropout rate by a significant amount only at the interim feed-forward computation at each expert layer. Table 4 has the results for our expert dropout protocol.\n\nWe observe that simply increasing the dropout across all layers leads to worse performance.\n\nHowever, setting a smaller dropout rate (0.1) at non-expert layers and a much larger dropout rate (0.4) at expert layers leads to performance improvements on four smaller downstream tasks.\n\n3. Scaling Properties\n\nWe present a study of the scaling properties of the Switch Transformer architecture during pre-training. Per Kaplan et al. (2020), we consider a regime where the model is not bottlenecked by either the computational budget or amount of data. To avoid the data bottleneck, we use the large C4 corpus with over 180B target tokens (Raffel et al., 2019) and we train until diminishing returns are observed.\n\nThe number of experts is the most efficient dimension for scaling our model. Increasing the experts keeps the computational cost approximately fixed since the model only selects one expert per token, regardless of the number of experts to choose from. The router must compute a probability distribution over more experts, however, this is a lightweight computation of cost O(dmodel × num experts) where dmodel is the embedding dimension of tokens passed between the layers. In this section, we consider the scaling properties on a step-basis and a time-basis with a fixed computational budget.\n\n3.1 Scaling Results On A Step-Basis\n\nFigure 4 demonstrates consistent scaling benefits with the number of experts when training all models for a fixed number of steps. We observe a clear trend: when keeping the FLOPS per token fixed, having more parameters (experts) speeds up training. The left Figure demonstrates consistent scaling properties (with fixed FLOPS per token) between sparse model parameters and test loss. This reveals the advantage of scaling along this additional axis of sparse model parameters. Our right Figure measures sample efficiency of a dense model variant and four FLOP-matched sparse variants. We find that increasing the number of experts leads to more sample efficient models. Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time. In addition, consistent with the findings of Kaplan et al. (2020), we find that larger models are also more sample efficient—learning more quickly for a fixed number of observed tokens.\n\nFigure 4: Scaling properties of the Switch Transformer. Left Plot: We measure the quality improvement, as measured by perplexity, as the parameters increase by scaling the number of experts. The top-left point corresponds to the T5-Base model with 223M parameters. Moving from top-left to bottom-right, we double the number of experts from 2, 4, 8 and so on until the bottom-right point of a 256 expert model with 14.7B parameters. Despite all models using an equal computational budget, we observe consistent improvements scaling the number of experts. Right Plot: Negative log perplexity per step sweeping over the number of experts. The dense baseline is shown with the purple line and we note improved sample efficiency of our Switch-Base models.\n\n3.2 Scaling Results On A Time-Basis\n\nFigure 4 demonstrates that on a step basis, as we increase the number of experts, the performance consistently improves. While our models have roughly the same amount of FLOPS per token as the baseline, our Switch Transformers incurs additional communication costs across devices as well as the extra computation of the routing mechanism. Therefore, the increased sample efficiency observed on a step-basis doesn't necessarily translate to a better model quality as measured by wall-clock. This raises the question: For a fixed training duration and computational budget, should one train a dense or a sparse model?\n\nFigure 5: Speed advantage of Switch Transformer. All models trained on 32 TPUv3 cores with equal FLOPs per example. For a fixed amount of computation and training time, Switch Transformers significantly outperform the dense Transformer baseline. Our 64 expert Switch-Base model achieves the same quality in one-seventh the time of the T5-Base and continues to improve.\n\nFigures 5 and 6 address this question. Figure 5 measures the pre-training model quality as a function of time. For a fixed training duration and computational budget, Switch Transformers yield a substantial speed-up. In this setting, our Switch-Base 64 expert model trains in one-seventh the time that it would take the T5-Base to get similar perplexity.\n\n3.3 Scaling Versus A Larger Dense Model\n\nThe above analysis shows that a computationally-matched dense model is outpaced by its Switch counterpart. Figure 6 considers a different scenario: what if we instead had allocated our resources to a larger dense model? We do so now, measuring Switch-Base against the next strong baseline, T5-Large. But despite T5-Large applying 3.5x more FLOPs per token, Switch-Base is still more sample efficient and yields a 2.5x speedup. Furthermore, more gains can be had simply by designing a new, larger sparse version, Switch-Large, which is FLOP-matched to T5-Large. We do this and demonstrate superior scaling and fine-tuning in the following section.\n\n4. Downstream Results\n\nSection 3 demonstrated the superior scaling properties while pre-training, but we now validate that these gains translate to improved language learning abilities on downstream tasks. We begin by fine-tuning on a diverse set of NLP tasks. Next we study reducing the memory footprint of our sparse models by over 90% by distilling into small—and easily deployed—dense baselines. Finally, we conclude this section measuring the improvements in a multi-task, multilingual setting, where we show that Switch Transformers are strong multi-task learners, improving over the multilingual T5-base model across all 101 languages.\n\n4.1 Fine-Tuning\n\nBaseline and Switch models used for fine-tuning. Our baselines are the highly-tuned 223M parameter T5-Base model and the 739M parameter T5-Large model (Raffel et al., 2019). For both versions, we design a FLOP-matched Switch Transformer, with many more parameters, which is summarized in Table 9.\n\n7 Our baselines differ slightly from those in Raffel et al. (2019) because we pre-train on an improved C4 corpus which removes intraexample text duplication and thus increases the efficacy as a pre-training task Lee et al.\n\nFLOPS are calculated for the forward pass as done in Kaplan et al. (2020).\n\n(2021). In our protocol we pre-train with 220 (1,048,576) tokens per batch for 550k steps amounting to 576B total tokens. We then fine-tune across a diverse set of tasks using a dropout rate of 0.1 for all layers except the Switch layers, which use a dropout rate of 0.4 (see Table 4). We fine-tune using a batch-size of 1M for 16k steps and for each task, we evaluate model quality every 200-steps and report the peak performance as computed on the validation set.\n\nFine-tuning tasks and data sets. We select tasks probing language capabilities including question answering, summarization and knowledge about the world. The language benchmarks GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) are handled as composite mixtures with all the tasks blended in proportion to the amount of tokens present in each. These benchmarks consist of tasks requiring sentiment analysis (SST2), word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP), natural language inference (MNLI, QNLI, RTE, CB), question answering (MultiRC, RECORD, BoolQ), coreference resolution (WNLI, WSC) and sentence completion (COPA) and sentence acceptability (CoLA). The CNNDM (Hermann et al., 2015) and BBC XSum (Narayan et al., 2018) data sets are used to measure the ability to summarize articles. Question answering is probed with the SQuAD data set (Rajpurkar et al., 2016) and the ARC Reasoning Challenge (Clark et al., 2018). And as in Roberts et al. (2020), we evaluate the knowledge of our models by fine-tuning on three closed-book question answering data sets: Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) and Trivia QA (Joshi et al., 2017). Closed-book refers to questions posed with no supplemental reference or context material. To gauge the model's common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020). And finally, we test our model's natural language inference capabilities on the Adversarial NLI Benchmark (Nie et al., 2019).\n\nFine-tuning metrics. The following evaluation metrics are used throughout the paper: We report the average scores across all subtasks for GLUE and SuperGLUE. The Rouge-2 metric is used both the CNNDM and XSum. In SQuAD and the closed book tasks (Web, Natural, and Trivia Questions) we report the percentage of answers exactly matching the target (refer to Roberts et al. (2020) for further details and deficiency of this measure). Finally, in ARC Easy, ARC Challenge, ANLI, and Winogrande we report the accuracy of the generated responses.\n\nFine-tuning results. We observe significant downstream improvements across many natural language tasks. Notable improvements come from SuperGLUE, where we find FLOP-matched Switch variants improve by 4.4 and 2 percentage points over the T5-Base and T5-Large baselines, respectively as well as large improvements in Winogrande, closed book Trivia QA, and XSum.8In our fine-tuning study, the only tasks where we do not observe gains are on the AI2 Reasoning Challenge (ARC) data sets where the T5-Base outperforms Switch-Base on the challenge data set and T5-Large outperforms Switch-Large on the easy data set. Taken as a whole, we observe significant improvements spanning both reasoning and knowledge-heavy tasks. This validates our architecture, not just as one that pre-trains well, but can translate quality improvements to downstream tasks via fine-tuning.\n\nModel GLUE SQuAD SuperGLUE Winogrande (XL) T5-Base 84.3 85.5 75.1 66.6 Switch-Base 86.7 87.2 79.5 73.3 T5-Large 87.8 88.1 82.7 79.1 Switch-Large 88.5 88.6 84.7 83.0 Model XSum ANLI (R3) ARC Easy ARC Chal. T5-Base 18.7 51.8 56.7 35.5 Switch-Base 20.3 54.0 61.3 32.8 T5-Large 20.9 56.6 68.8 35.5 Switch-Large 22.3 58.6 66.0 35.5 Model CB Web QA CB Natural QA CB Trivia QA T5-Base 26.6 25.8 24.5 Switch-Base 27.4 26.8 30.7 T5-Large 27.7 27.6 29.5 Switch-Large 31.3 29.5 36.9\n\nTable 5: Fine-tuning results. Fine-tuning results of T5 baselines and Switch models across a diverse set of natural language tests (validation sets; higher numbers are better).\n\nWe compare FLOP-matched Switch models to the T5-Base and T5-Large baselines. For most tasks considered, we find significant improvements of the Switchvariants. We observe gains across both model sizes and across both reasoning and knowledge-heavy language tasks.\n\n4.2 Distillation\n\nDeploying massive neural networks with billions, or trillions, of parameters is inconvenient.\n\nTo alleviate this, we study distilling (Hinton et al., 2015) large sparse models into small dense models. Future work could additionally study distilling large models into smaller sparse models.\n\nDistillation techniques. In Table 6 we study a variety of distillation techniques.\n\nThese techniques are built off of Sanh et al. (2019), who study distillation methods for BERT models. We find that initializing the dense model with the non-expert weights yields a modest improvement. This is possible since all models are FLOP matched, so non-expert layers will have the same dimensions. Since expert layers are usually only added at every or every other FFN layer in a Transformer, this allows for many of the weights to be initialized with trained parameters. Furthermore, we observe a distillation improvement using a mixture of 0.25 for the teacher probabilities and 0.75 for the ground truth label. By combining both techniques we preserve ≈ 30% of the quality gains from the larger sparse models with only ≈ 1/20th of the parameters. The quality gain refers to the percent of\n\nTechnique Parameters Quality (↑) T5-Base 223M -1.636 Switch-Base 3,800M -1.444 Distillation 223M (3%) -1.631 + Init. non-expert weights from teacher 223M (20%) -1.598 + 0.75 mix of hard and soft loss 223M (29%) -1.580 Initialization Baseline (no distillation) Init. non-expert weights from teacher 223M -1.639\n\nthe quality difference between Switch-Base (Teacher) and T5-Base (Student). Therefore, a quality gain of 100% implies the Student equals the performance of the Teacher.\n\nTable 6: Distilling Switch Transformers for Language Modeling. Initializing T5-Base with the non-expert weights from Switch-Base and using a loss from a mixture of teacher and ground-truth labels obtains the best performance. We can distill 30% of the performance improvement of a large sparse model with 100x more parameters back into a small dense model. For a final baseline, we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation.\n\nAchievable compression rates. Using our best distillation technique described in Table 6, we distill a wide variety of sparse models into dense models. We distill SwitchBase versions, sweeping over an increasing number of experts, which corresponds to varying between 1.1B to 14.7B parameters. Through distillation, we can preserve 37% of the quality gain of the 1.1B parameter model while compressing 82%. At the extreme, where we compress the model 99%, we are still able to maintain 28% of the teacher's model quality improvement.\n\nDistilling a fine-tuned model. We conclude this with a study of distilling a finetuned sparse model into a dense model. Table 8 shows results of distilling a 7.4B parameter Switch-Base model, fine-tuned on the SuperGLUE task, into the 223M T5-Base. Similar to our pre-training results, we find we are able to preserve 30% of the gains of the sparse model when distilling into a FLOP matched dense variant. One potential future avenue, not considered here, may examine the specific experts being used for fine-tuning tasks and extracting them to achieve better model compression.\n\n4.3 Multilingual Learning\n\nIn our final set of downstream experiments, we measure the model quality and speed tradeoffs while pre-training on a mixture of 101 different languages. We build and benchmark off the recent work of mT5 (Xue et al., 2020), a multilingual extension to T5. We pre-train on the multilingual variant of the Common Crawl data set (mC4) spanning 101 languages introduced in mT5, but due to script variants within certain languages, the mixture contains 107 tasks.\n\nIn Figure 7 we plot the quality improvement in negative log perplexity for all languages of a FLOP-matched Switch model, mSwitch-Base to the T5 base variant, mT5-Base. After\n\nDense Sparse Parameters 223M 1.1B 2.0B 3.8B 7.4B 14.7B Pre-trained Neg. Log Perp. (↑) -1.636 -1.505 -1.474 -1.444 -1.432 -1.427 Distilled Neg. Log Perp. (↑) - -1.587 -1.585 -1.579 -1.582 -1.578 Percent of Teacher Performance - 37% 32% 30 % 27 % 28 % Compression Percent - 82 % 90 % 95 % 97 % 99 %\n\nTable 7: Distillation compression rates. We measure the quality when distilling large sparse models into a dense baseline. Our baseline, T5-Base, has a -1.636 Neg. Log Perp. quality. In the right columns, we then distill increasingly large sparse models into this same architecture. Through a combination of weight-initialization and a mixture of hard and soft losses, we can shrink our sparse teachers by 95%+ while preserving 30% of the quality gain. However, for significantly better and larger pre-trained teachers, we expect larger student models would be necessary to achieve these compression rates.\n\nModel Parameters FLOPS SuperGLUE (↑) T5-Base 223M 124B 74.6 Switch-Base 7410M 124B 81.3 Distilled T5-Base 223M 124B (30%) 76.6\n\nTable 8: Distilling a fine-tuned SuperGLUE model. We distill a Switch-Base model finetuned on the SuperGLUE tasks into a T5-Base model. We observe that on smaller data sets our large sparse model can be an effective teacher for distillation. We find that we again achieve 30% of the teacher's performance on a 97% compressed model. pre-training both versions for 1M steps, we find that on all 101 languages considered, Switch Transformer increases the final negative log perplexity over the baseline. In Figure 8, we present a different view and now histogram the per step speed-up of using Switch Transformer over the mT5-Base.9 We find a mean speed-up over mT5-Base of 5x and that 91% of languages achieve at least a 4x speedup. This presents evidence that Switch Transformers are effective multi-task and multi-lingual learners.\n\n5. Designing Models With Data, Model, And Expert-Parallelism\n\nArbitrarily increasing the number of experts is subject to diminishing returns (Figure 4). Here we describe complementary scaling strategies. The common way to scale a Transformer is to increase dimensions in tandem, like dmodel or df f . This increases both the parameters\n\nFigure 7: Multilingual pre-training on 101 languages. Improvements of Switch T5 Base model over dense baseline when multi-task training on 101 languages. We observe Switch Transformers to do quite well in the multi-task training setup and yield improvements on all 101 languages.\n\nFigure 8: Multilingual pre-training on 101 languages. We histogram for each language, the step speedup of Switch Transformers over the FLOP matched T5 dense baseline to reach the same quality. Over all 101 languages, we achieve a mean step speedup over mT5-Base of 5x and, for 91% of languages, we record a 4x, or greater, speedup to reach the final perplexity of mT5-Base.\n\nand computation performed and is ultimately limited by the memory per accelerator. Once it exceeds the size of the accelerator's memory, single program multiple data (SPMD) modelparallelism can be employed. This section studies the trade-offs of combining data, model, and expert-parallelism.\n\nReviewing the Feed-Forward Network (FFN) Layer. We use the FFN layer as an example of how data, model and expert-parallelism works in Mesh TensorFlow (Shazeer et al., 2018) and review it briefly here. We assume B tokens in the batch, each of dimension dmodel. Both the input (x) and output (y) of the FFN are of size [B, dmodel] and the intermediate (h) is of size [B, df f ] where df f is typically several times larger than dmodel. In the FFN, the intermediate is h = xWin and then the output of the layer is y = ReLU(h)Wout.\n\nThus Win and Wout are applied independently to each token and have sizes [dmodel, df f ] and [df f , dmodel].\n\nWe describe two aspects of partitioning: how the weights and batches of data divide over cores, depicted in Figure 9. We denote all cores available as N which Mesh Tensorflow may then remap into a logical multidimensional mesh of processors. Here we create a two-dimensional logical mesh, with one dimension representing the number of ways for data-parallel sharding (n) and the other, the model-parallel sharding (m). The total cores must equal the ways to shard across both data and model-parallelism, e.g. N = n × m.\n\nTo shard the layer across cores, the tensors containing that batch of B tokens are sharded across n data-parallel cores, so each core contains B/n tokens. Tensors and variables with df f are then sharded across m model-parallel cores. For the variants with experts-layers, we consider E experts, each of which can process up to C tokens.\n\nTerm Description B Number of tokens in the batch. N Number of total cores. n Number of ways for data-parallelism sharding. m Number of ways for model-parallelism sharding. E Number of experts in Switch layers. C Expert capacity, the batch size of each expert.\n\n5.1 Data Parallelism\n\nWhen training data parallel models, which is the standard for distributed training, then all cores are allocated to the data-parallel dimension or n = N, m = 1. This has the advantage that no communication is needed until the entire forward and backward pass is finished and the gradients need to be then aggregated across all cores. This corresponds to the left-most column of Figure 9.\n\n5.2 Model Parallelism\n\nWe now consider a scenario where all cores are allocated exclusively to the model-parallel dimension and so n = 1, m = N. Now all cores must keep the full B tokens and each core will contain a unique slice of the weights. For each forward and backward pass, a communication cost is now incurred. Each core sends a tensor of [B, dmodel] to compute the second matrix multiplication ReLU(h)Wout because the df f dimension is partitioned and must be summed over. As a general rule, whenever a dimension that is partitioned across cores must be summed, then an all-reduce operation is added for both the forward and backward pass. This contrasts with pure data parallelism where an all-reduce only occurs at the end of the entire forward and backward pass.\n\nFigure 9: Data and weight partitioning strategies. Each 4×4 dotted-line grid represents 16 cores and the shaded squares are the data contained on that core (either model weights or batch of tokens). We illustrate both how the model weights and the data tensors are split for each strategy. First Row: illustration of how model weights are split across the cores. Shapes of different sizes in this row represent larger weight matrices in the Feed Forward Network (FFN) layers (e.g larger df f sizes). Each color of the shaded squares identifies a unique weight matrix. The number of parameters per core is fixed, but larger weight matrices will apply more computation to each token. Second Row: illustration of how the data batch is split across cores. Each core holds the same number of tokens which maintains a fixed memory usage across all strategies. The partitioning strategies have different properties of allowing each core to either have the same tokens or different tokens across cores, which is what the different colors symbolize.\n\n5.3 Model And Data Parallelism\n\nIt is common to mix both model and data parallelism for large scale models, which was done in the largest T5 models (Raffel et al., 2019; Xue et al., 2020) and in GPT-3 (Brown et al., 2020). With a total of N = n × m cores, now each core will be responsible for B/n tokens and df f /m of both the weights and intermediate activation. In the forward and backward pass each core communicates a tensor of size [B/n, dmodel] in an all-reduce operation.\n\n5.4 Expert And Data Parallelism\n\nNext we describe the partitioning strategy for expert and data parallelism. Switch Transformers will allocate all of their cores to the data partitioning dimension n, which will also correspond to the number of experts in the model. For each token per core a router locally computes assignments to the experts. The output is a binary matrix of size [n, B/n, E, C] which is partitioned across the first dimension and determines expert assignment. This binary matrix is then used to do a gather via matrix multiplication with the input tensor of [n, B/n, dmodel].\n\neinsum([n, B/n, dmodel], [n, B/n, E, C], dimension = [B/n]) (7) resulting in the final tensor of shape [n, E, C, dmodel], which is sharded across the first dimension. Because each core has its own expert, we do an all-to-all communication of size [E, C, dmodel] to now shard the E dimension instead of the n-dimension. There are additional communication costs of bfloat16 tensors of size E×C ×dmodel in the forward pass to analogusly receive the tokens from each expert located on different cores. See Appendix F for a detailed analysis of the expert partitioning code.\n\n5.5 Expert, Model And Data Parallelism\n\nIn the design of our best model, we seek to balance the FLOPS per token and the parameter count. When we scale the number of experts, we increase the number of parameters, but do not change the FLOPs per token. In order to increase FLOPs, we must also increase the df f dimension (which also increases parameters, but at a slower rate). This presents a trade-off: as we increase df f we will run out of memory per core, which then necessitates increasing m. But since we have a fixed number of cores N, and N = n × m, we must decrease n, which forces use of a smaller batch-size (in order to hold tokens per core constant).\n\nWhen combining both model and expert-parallelism, we will have all-to-all communication costs from routing the tokens to the correct experts along with the internal all-reduce communications from the model parallelism. Balancing the FLOPS, communication costs and memory per core becomes quite complex when combining all three methods where the best mapping is empirically determined. See our further analysis in section 5.6 for how the number of experts effects the downstream performance as well.\n\n5.6 Towards Trillion Parameter Models\n\nCombining expert, model and data parallelism, we design two large Switch Transformer models, one with 395 billion and 1.6 trillion parameters, respectively. We study how these models perform on both up-stream pre-training as language models and their downstream fine-tuning performance. The parameters, FLOPs per sequence and hyper-parameters of the two different models are listed below in Table 9. Standard hyper-parameters of the Transformer, including dmodel, df f , dkv, number of heads and number of layers are described, as well as a less common feature, F F NGEGLU , which refers to a variation of the FFN layer where the expansion matrix is substituted with two sets of weights which are non-linearly combined (Shazeer, 2020).\n\nThe Switch-C model is designed using only expert-parallelism, and no model-parallelism, as described earlier in Section 5.4. As a result, the hyper-parameters controlling the width,\n\nModel Parameters FLOPs/seq dmodel F F NGEGLU df f dkv Num. Heads T5-Base 0.2B 124B 768 X 2048 64 12 T5-Large 0.7B 425B 1024 X 2816 64 16 T5-XXL 11B 6.3T 4096 X 10240 64 64 Switch-Base 7B 124B 768 X 2048 64 12 Switch-Large 26B 425B 1024 X 2816 64 16 Switch-XXL 395B 6.3T 4096 X 10240 64 64 Switch-C 1571B 890B 2080 6144 64 32 Model Expert Freq. Num. Layers Num Experts Neg. Log Perp. @250k Neg. Log Perp. @ 500k T5-Base - 12 - -1.599 -1.556 T5-Large - 24 - -1.402 -1.350 T5-XXL - 24 - -1.147 -1.095 Switch-Base 1/2 12 128 -1.370 -1.306 Switch-Large 1/2 24 128 -1.248 -1.177 Switch-XXL 1/2 24 64 -1.086 -1.008 Switch-C 1 15 2048 -1.096 -1.043\n\nTable 9: Switch model design and pre-training performance. We compare the hyperparameters and pre-training performance of the T5 models to our Switch Transformer variants. The last two columns record the pre-training model quality on the C4 data set after 250k and 500k steps, respectively. We observe that the SwitchC Transformer variant is 4x faster to a fixed perplexity (with the same compute budget) than the T5-XXL model, with the gap increasing as training progresses.\n\ndepth, number of heads, and so on, are all much smaller than the T5-XXL model. In contrast, the Switch-XXL is FLOP-matched to the T5-XXL model, which allows for larger dimensions of the hyper-parameters, but at the expense of additional communication costs induced by model-parallelism (see Section 5.5 for more details).\n\nSample efficiency versus T5-XXL. In the final two columns of Table 9 we record the negative log perplexity on the C4 corpus after 250k and 500k steps, respectively. After 250k steps, we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061.10 To contextualize the significance of a gap of 0.061, we note that the T5-XXL model had to train for an additional 250k steps to increase 0.052. The gap continues to increase with additional training, with the Switch-XXL model out-performing the T5-XXL by 0.087 by 500k steps.\n\nTraining instability. However, as described in the introduction, large sparse models can be unstable, and as we increase the scale, we encounter some sporadic issues. We find that the larger Switch-C model, with 1.6T parameters and 2048 experts, exhibits no training instability at all. Instead, the Switch XXL version, with nearly 10x larger FLOPs per sequence, is sometimes unstable. As a result, though this is our better model on a step-basis, we do not pre-train for a full 1M steps, in-line with the final reported results of T5 (Raffel et al., 2019).\n\nReasoning fine-tuning performance. As a preliminary assessment of the model quality, we use a Switch-XXL model partially pre-trained on 503B tokens, or approximately half the text used by the T5-XXL model. Using this checkpoint, we conduct multi-task training for efficiency, where all tasks are learned jointly, rather than individually fine-tuned.\n\nWe find that SQuAD accuracy on the validation set increases to 89.7 versus state-of-the-art of 91.3. Next, the average SuperGLUE test score is recorded at 87.5 versus the T5 version obtaining a score of 89.3 compared to the state-of-the-art of 90.0 (Wang et al., 2019). On ANLI (Nie et al., 2019), Switch XXL improves over the prior state-of-the-art to get a 65.7 accuracy versus the prior best of 49.4 (Yang et al., 2020). We note that while the SwitchXXL has state-of-the-art Neg. Log Perp. on the upstream pre-training task, its gains have not yet fully translated to SOTA downstream performance. We study this issue more in Appendix E.\n\nKnowledge-based fine-tuning performance. Finally, we also conduct an early examination of the model's knowledge with three closed-book knowledge-based tasks: Natural Questions, WebQuestions and TriviaQA, without additional pre-training using Salient Span Masking (Guu et al., 2020). In all three cases, we observe improvements over the prior stateof-the-art T5-XXL model (without SSM). Natural Questions exact match increases to 34.4 versus the prior best of 32.8, Web Questions increases to 41.0 over 37.2, and TriviaQA increases to 47.5 versus 42.9.\n\nSumming up, despite training on less than half the data of other models, we already find comparable, and sometimes state-of-the-art, model quality. Currently, the Switch Transformer translates substantial upstream gains better to knowledge-based tasks, than reasoning-tasks (see Appendix E). Extracting stronger fine-tuning performance from large expert models is an active research question, and the pre-training perplexity indicates future improvements should be possible.\n\n6. Related Work\n\nThe importance of scale in neural networks is widely recognized and several approaches have been proposed. Recent works have scaled models to billions of parameters through using model parallelism (e.g. splitting weights and tensors across multiple cores) (Shazeer et al., 2018; Rajbhandari et al., 2019; Raffel et al., 2019; Brown et al., 2020; Shoeybi et al., 2019). Alternatively, Harlap et al. (2018); Huang et al. (2019) propose using pipeline based model parallelism, where different layers are split across devices and micro-batches are pipelined to the different layers. Finally, Product Key networks (Lample et al., 2019) were proposed to scale up the capacity of neural networks by doing a lookup for learnable embeddings based on the incoming token representations to a given layer.\n\nOur work studies a specific model in a class of methods that do conditional computation, where computation decisions are made dynamically based on the input. Cho and Bengio (2014) proposed adaptively selecting weights based on certain bit patterns occuring in the model hidden-states. Eigen et al. (2013) built stacked expert layers with dense matrix multiplications and ReLU activations and showed promising results on jittered MNIST and monotone speech. In computer vision Puigcerver et al. (2020) manually route tokens based on semantic classes during upstream pre-training and then select the relevant experts to be used according to the downstream task.\n\nMixture of Experts (MoE), in the context of modern deep learning architectures, was proven effective in Shazeer et al. (2017). That work added an MoE layer which was stacked between LSTM (Hochreiter and Schmidhuber, 1997) layers, and tokens were separately routed to combinations of experts. This resulted in state-of-the-art results in language modeling and machine translation benchmarks. The MoE layer was reintroduced into the Transformer architecture by the Mesh Tensorflow library (Shazeer et al., 2018) where MoE layers were introduced as a substitute of the FFN layers, however, there were no accompanying NLP results. More recently, through advances in machine learning infrastructure, GShard (Lepikhin et al., 2020), which extended the XLA compiler, used the MoE Transformer to dramatically improve machine translation across 100 languages. Finally Fan et al.\n\n(2021) chooses a different deterministic MoE strategy to split the model parameters into non-overlapping groups of languages.\n\nSparsity along the sequence length dimension (L) in the Transformer attention patterns has been a successful technique to reduce the attention complexity from O(L 2) (Child et al., 2019; Correia et al., 2019; Sukhbaatar et al., 2019; Kitaev et al., 2020; Zaheer et al., 2020; Beltagy et al., 2020). This has enabled learning longer sequences than previously possible. This version of the Switch Transformer does not employ attention sparsity, but these techniques are complimentary, and, as future work, these could be combined to potentially improve learning on tasks requiring long contexts.\n\n7. Discussion\n\nWe pose and discuss questions about the Switch Transformer, and sparse expert models generally, where sparsity refers to weights, not on attention patterns.\n\nIsn't Switch Transformer better due to sheer parameter count? Yes, and by design! Parameters, independent of the total FLOPs used, are a useful axis to scale neural language models. Large models have been exhaustively shown to perform better (Kaplan et al., 2020). But in this case, our model is more sample efficient and faster while using the same computational resources.\n\nI don't have access to a supercomputer—is this still useful for me? Though this work has focused on extremely large models, we also find that models with as few as two experts improves performance while easily fitting within memory constraints of commonly available GPUs or TPUs (details in Appendix D). We therefore believe our techniques are useful in small-scale settings.\n\nDo sparse models outperform dense models on the speed-accuracy Pareto curve? Yes. Across a wide variety of different models sizes, sparse models outperform dense models per step and on wall clock time. Our controlled experiments show for a fixed amount of computation and time, sparse models outperform dense models.\n\nI can't deploy a trillion parameter model—can we shrink these models? We cannot fully preserve the model quality, but compression rates of 10 to 100x are achievable by distilling our sparse models into dense models while achieving ≈30% of the quality gain of the expert model.\n\nWhy use Switch Transformer instead of a model-parallel dense model? On a time basis, Switch Transformers can be far more efficient than dense-models with sharded parameters (Figure 6). Also, we point out that this decision is not mutually exclusive—we can, and do, use model-parallelism in Switch Transformers, increasing the FLOPs per token, but incurring the slowdown of conventional model-parallelism.\n\nWhy aren't sparse models widely used already? The motivation to try sparse models has been stymied by the massive success of scaling dense models (the success of which is partially driven by co-adaptation with deep learning hardware as argued in Hooker (2020)). Further, sparse models have been subject to multiple issues including (1) model complexity, (2) training difficulties, and (3) communication costs. Switch Transformer makes strides to alleviate these issues.\n\n8. Future Work\n\nThis paper lays out a simplified architecture, improved training procedures, and a study of how sparse models scale. However, there remain many open future directions which we briefly describe here:\n\nA significant challenge is further improving training stability for the largest models. While our stability techniques were effective for our Switch-Base, Switch-Large and Switch-C models (no observed instability), they were not sufficient for Switch-XXL. We have taken early steps towards stabilizing these models, which we think may be generally useful for large models, including using regularizers for improving stability and adapted forms of gradient clipping, but this remains unsolved.\n\nGenerally we find that improved pre-training quality leads to better downstream results (Appendix E), though we sometimes encounter striking anomalies. For instance, despite similar perplexities modeling the C4 data set, the 1.6T parameter Switch-C achieves only an 87.7 exact match score in SQuAD, which compares unfavorably to 89.6 for the smaller Switch-XXL model. One notable difference is that the SwitchXXL model applies ≈10x the FLOPS per token than the Switch-C model, even though it has ≈4x less unique parameters (395B vs 1.6T). This suggests a poorly understood dependence between fine-tuning quality, FLOPS per token and number of parameters.\n\nPerform a comprehensive study of scaling relationships to guide the design of architectures blending data, model and expert-parallelism. Ideally, given the specs of a hardware configuration (computation, memory, communication) one could more rapidly design an optimal model. And, vice versa, this may also help in the design of future hardware.\n\nOur work falls within the family of adaptive computation algorithms. Our approach always used identical, homogeneous experts, but future designs (facilitated by more flexible infrastructure) could support heterogeneous experts. This would enable more flexible adaptation by routing to larger experts when more computation is desired— perhaps for harder examples.\n\nInvestigating expert layers outside the FFN layer of the Transformer. We find preliminary evidence that this similarly can improve model quality. In Appendix A, we report quality improvement adding these inside Self-Attention layers, where our layer replaces the weight matrices which produce Q, K, V. However, due to training instabilities with the bfloat16 format, we instead leave this as an area for future work.\n\nExamining Switch Transformer in new and across different modalities. We have thus far only considered language, but we believe that model sparsity can similarly provide advantages in new modalities, as well as multi-modal networks. This list could easily be extended, but we hope this gives a flavor for the types of challenges that we are thinking about and what we suspect are promising future directions.\n\n9. Conclusion\n\nSwitch Transformers are scalable and effective natural language learners. We simplify Mixture of Experts to produce an architecture that is easy to understand, stable to train and vastly more sample efficient than equivalently-sized dense models. We find that these models excel across a diverse set of natural language tasks and in different training regimes, including pre-training, fine-tuning and multi-task training. These advances make it possible to train models with hundreds of billion to trillion parameters and which achieve substantial speedups relative to dense T5 baselines. We hope our work motivates sparse models as an effective architecture and that this encourages researchers and practitioners to consider these flexible models in natural language tasks, and beyond.\n\nAcknowledgments\n\nThe authors would like to thank Margaret Li who provided months of key insights into algorithmic improvements and suggestions for empirical studies. Hugo Larochelle for sage advising and clarifying comments on the draft, Irwan Bello for detailed comments and careful revisions, Colin Raffel and Adam Roberts for timely advice on neural language models and the T5 code-base, Yoshua Bengio for advising and encouragement on research in adaptive computation, Jascha Sohl-dickstein for interesting new directions for stabilizing new large scale models and paper revisions, and the Google Brain Team for useful discussions on the paper. Blake Hechtman who provided invaluable help in profiling and improving the training performance of our models.\n\nA. Switch For Attention\n\nShazeer et al. (2018); Lepikhin et al. (2020) designed MoE Transformers (Shazeer et al., 2017) by adding MoE layers into the dense feedfoward network (FFN) computations of the Transformer. Similarly, our work also replaced the FFN layer in the Transformer, but we briefly explore here an alternate design. We add Switch layers into the Transformer Self-Attention layers. To do so, we replace the trainable weight matrices that produce the queries, keys and values with Switch layers as seen in Figure 10.\n\nTable 10 records the quality after a fixed number of steps as well as training time for several variants. Though we find improvements, we also found these layers to be more unstable when using bfloat16 precision and thus we did not include them in the final variant.\n\nModel Precision Quality Quality Speed @100k Steps (↑) @16H (↑) (ex/sec) (↑) Experts FF float32 -1.548 -1.614 1480 Expert Attention float32 -1.524 -1.606 1330 Expert Attention bfloat16 [diverges] [diverges] - Experts FF + Attention float32 -1.513 -1.607 1240 Expert FF + Attention bfloat16 [diverges] [diverges] -\n\nHowever, when these layers do train stably, we believe the preliminary positive results suggests a future promising direction.\n\nTable 10: Switch attention layer results. All models have 32 experts and train with 524k tokens per batch. Experts FF is when experts replace the FFN in the Transformer, which is our standard setup throughout the paper. Experts FF + Attention is when experts are used to replace both the FFN and the Self-Attention layers. When training with bfloat16 precision the models that have experts attention diverge.\n\nB. Preventing Token Dropping With No-Token-Left-Behind\n\nDue to software constraints on TPU accelerators, the shapes of our Tensors must be statically sized. As a result, each expert has a finite and fixed capacity to process token representations. This, however, presents an issue for our model which dynamically routes tokens at run-time that may result in an uneven distribution over experts. If the number of tokens sent to an expert is less than the expert capacity, then the computation may simply be padded - an inefficient use of the hardware, but mathematically correct. However, when the number of tokens sent to an expert is larger than its capacity (expert overflow), a protocol is needed to handle this. Lepikhin et al. (2020) adapts a Mixture-of-Expert model and addresses expert overflow by passing its representation to the next layer without processing through a residual connection which we also follow.\n\nWe suspected that having no computation applied to tokens could be very wasteful, especially since if there is overflow on one expert, that means another expert will have extra capacity. With this intuition we create No-Token-Left-Behind, which iteratively reroutes any tokens that are at first routed to an expert that is overflowing. Figure 11 shows a graphical description of this method, which will allow us to guarantee almost no tokens will be dropped during training and inference. We hypothesised that this could improve performance and further stabilize training, but we found no empirical benefits. We suspect that once the network learns associations between different tokens and experts, if this association is changed (e.g. sending a token to its second highest expert) then performance could be degraded.\n\nC. Encouraging Exploration Across Experts\n\nAt each expert-layer, the router determines to which expert to send the token. This is a discrete decision over the available experts, conditioned on information about the token's representation. Based on the incoming token representation, the router determines the best expert, however, it receives no counterfactual information about how well it would have done selecting an alternate expert. As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018). These issues have been similarly noted and addressed differently by Rosenbaum et al. (2017) which demonstrated success in multi-task learning. This particular setting most closely matches that of a contextual bandit (Robbins, 1952). Deterministically selecting the top expert always amounts to an exploitative strategy - we consider balancing exploration to seek better expert assignment.\n\nTo introduce exploration, we consider several approaches: 1) deterministic or argmax 2) sampling from the softmax distribution 3) input dropout on the incoming representation 4) multiplicative jitter noise on the incoming representation. The resulting impact on model quality is reported in Table 11. Throughout this work, we use input jitter to inject noise as we have found it to empirically perform the best.\n\nD. Switch Transformers In Lower Compute Regimes\n\nSwitch Transformer is also an effective architecture at small scales as well as in regimes with thousands of cores and trillions of parameters. Many of our prior experiments were\n\nFigure 11: Diagram of the No-Token-Left-Behind Routing. Stage 1 is equivalent to Switch routing where tokens are routed to the expert with the highest probability from the router. In Stage 2 we look at all tokens that have overflowed and route them to the expert with which has the second highest probability. Tokens can still be overflowed if their second highest expert has too many tokens, but this allows most of the tokens to be routed. This process can be iterated to guarantee virtually no tokens are dropped at all.\n\nModel Quality (Neg. Log Perp.) (↑) Argmax -1.471 Sample softmax -1.570 Input dropout -1.480 Input jitter -1.468\n\nat the scale of 10B+ parameter models, but we show in Figure 12 as few as 2 experts produce compelling gains over a FLOP-matched counterpart. Even if a super computer is not readily available, training Switch Transformers with 2, 4, or 8 experts (as we typically recommend one expert per core) results in solid improvements over T5 dense baselines.\n\nE. Relation Of Upstream To Downstream Model Performance\n\nThere is no guarantee that a model's quality on a pre-training objective will translate to downstream task results. Figure 13 presents the correlation of the upstream model quality, for both dense and Switch models, on the C4 pre-training task with two downstream task measures: average SuperGLUE performance and TriviaQA score. We choose these two tasks as one probes the model's reasoning and the other factual knowledge.\n\nFigure 13: Upstream pre-trained quality to downstream model quality. We correlate the upstream performance with downstream quality on both SuperGLUE and TriviaQA (SOTA recorded without SSM), reasoning and knowledge-heavy benchmarks, respectively (validation sets). We find that, as with the baseline, the Switch model scales with improvements in the upstream pre-training task. For SuperGLUE, we find a loosely linear relation between negative log perplexity and the average SuperGLUE score. However, the dense model often performs better for a fixed perplexity, particularly in the large-scale regime. Conversely, on the knowledge-heavy task, TriviaQA, we find that the Switch Transformer may follow an improved scaling relationship - for a given upstream perplexity, it does better than a dense counterpart. Further statistics (expensive to collect and left to future work) would be necessary to confirm these observations.\n\nWe find a consistent correlation, indicating that for both baseline and Switch models, improved pre-training leads to better downstream results. Additionally, for a fixed upstream perplexity we find that both Switch and dense models perform similarly in the small to medium model size regime. However, in the largest model regime (T5-11B/T5-XXL) our largest Switch models, as mentioned in Section 5.6, do not always translate their upstream perplexity well to downstream fine-tuning on the SuperGLUE task. This warrants future investigation and study to fully realize the potential of sparse models. Understanding the fine-tuning dynamics with expert-models is very complicated and is dependent on regularization, load-balancing, and fine-tuning hyper-parameters.\n\nF. Pseudo Code For Switch Transformers\n\nPseudocode for Switch Transformers in Mesh Tensorflow (Shazeer et al., 2018). No model parallelism is being used for the below code (see 5.4 for more details).\n\nimport mesh tensorflow as mtf\n\n``` def load balance loss(router probs, expert mask): \"\"\"Calculate load−balancing loss to ensure diverse expert routing.\"\"\" # router probs is the probability assigned for each expert per token. # router probs shape: [num cores, tokens per core, num experts] # expert index contains the expert with the highest router probability in one−hot format. # expert mask shape: [num cores, tokens per core, num experts] # For each core, get the fraction of tokens routed to each expert. # density 1 shape: [num cores, num experts] density 1 = mtf.reduce mean(expert mask, reduced dim=tokens per core) # For each core, get fraction of probability mass assigned to each expert # from the router across all tokens. # density 1 proxy shape: [num cores, num experts] density 1 proxy = mtf.reduce mean(router probs, reduced dim=tokens per core) # density l for a single core: vector of length num experts that sums to 1. # density l proxy for a single core: vector of length num experts that sums to 1. # Want both vectors to have uniform allocation (1/num experts) across all num expert elements. # The two vectors will be pushed towards uniform allocation when the dot product is minimized. loss = mtf.reduce mean(density 1 proxy ∗ density 1) ∗ (num experts ˆ 2) return loss\n\n```\n\nFigure 14: Pseudo code for the load balance loss for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def router(inputs, capacity factor): \"\"\"Produce the combine and dispatch tensors used for sending and receiving tokens from their highest probability expert. \"\"\" # Core layout is split across num cores for all tensors and operations. # inputs shape: [num cores, tokens per core, d model] router weights = mtf.Variable(shape=[d model, num experts]) # router logits shape: [num cores, tokens per core, num experts] router logits = mtf.einsum([inputs, router weights], reduced dim=d model) if is training: # Add noise for exploration across experts. router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps) # Convert input to softmax operation from bfloat16 to float32 for stability. router logits = mtf.to float32(router logits) # Probabilities for each token of what expert it should be sent to. router probs = mtf.softmax(router logits, axis=−1) # Get the top−1 expert for each token. expert gate is the top−1 probability # from the router for each token. expert index is what expert each token # is going to be routed to. # expert gate shape: [num cores, tokens per core] # expert index shape: [num cores, tokens per core] expert gate, expert index = mtf.top 1(router probs, reduced dim=num experts) # expert mask shape: [num cores, tokens per core, num experts] expert mask = mtf.one hot(expert index, dimension=num experts) # Compute load balancing loss. aux loss = load balance loss(router probs, expert mask) # Experts have a fixed capacity, ensure we do not exceed it. Construct # the batch indices, to each expert, with position in expert # make sure that not more that expert capacity examples can be routed to # each expert. position in expert = mtf.cumsum(expert mask, dimension=tokens per core) ∗ expert mask # Keep only tokens that fit within expert capacity. expert mask ∗= mtf.less(position in expert, expert capacity) expert mask flat = mtf.reduce sum(expert mask, reduced dim=experts dim) # Mask out the experts that have overflowed the expert capacity. expert gate ∗= expert mask flat # combine tensor used for combining expert outputs and scaling with router probability. # combine tensor shape: [num cores, tokens per core, num experts, expert capacity] combine tensor = ( expert gate ∗ expert mask flat ∗ mtf.one hot(expert index, dimension=num experts) ∗ mtf.one hot(position in expert, dimension=expert capacity)) # Cast back outputs to bfloat16 for the rest of the layer. combine tensor = mtf.to bfloat16(combine tensor) # Create binary dispatch tensor that is 1 if the token gets routed to the corresponding expert. # dispatch tensor shape: [num cores, tokens per core, num experts, expert capacity] dispatch tensor = mtf.cast(combine tensor, tf.bool) return dispatch tensor, combine tensor, aux loss\n\n```\n\nFigure 15: Pseudo code for the router for Switch Transformers in Mesh Tensorflow.\n\nimport mesh tensorflow as mtf\n\n``` def switch layer(inputs, n, capacity factor, num experts): \"\"\"Distributed switch transformer feed−forward layer.\"\"\" # num cores (n) = total cores for training the model (scalar). # d model = model hidden size (scalar). # num experts = total number of experts. # capacity factor = extra buffer for each expert. # inputs shape: [batch, seq len, d model] batch, seq len, d model = inputs.get shape() # Each core will route tokens per core tokens to the correct experts. tokens per core = batch ∗ seq len / num cores # Each expert will have shape [num cores, expert capacity, d model]. # Each core is responsible for sending expert capacity tokens # to each expert. expert capacity = tokens per core ∗ capacity factor / num experts # Reshape to setup per core expert dispatching. # shape: [batch, seq len, d model] −> [num cores, tokens per core, d model] # Core layout: [n, 1, 1] −> [n, 1, 1] inputs = mtf.reshape(inputs, [num cores, tokens per core, d model]) # Core Layout: [n, 1, 1] −> [n, 1, 1, 1], [n, 1, 1, 1] # dispatch tensor (boolean) shape: [num cores, tokens per core, num experts, expert capacity] # dispatch tensor is used for routing tokens to the correct expert. # combine tensor (float) shape: [num cores, tokens per core, num experts, expert capacity] # combine tensor used for combining expert outputs and scaling with router # probability. dispatch tensor, combine tensor, aux loss = router(inputs, expert capacity) # Matmul with large boolean tensor to assign tokens to the correct expert. # Core Layout: [n, 1, 1], −> [1, n, 1, 1] # expert inputs shape: [num experts, num cores, expert capacity, d model] expert inputs = mtf.einsum([inputs, dispatch tensor], reduce dims=[tokens per core]) # All−to−All communication. Cores split across num cores and now we want to split # across num experts. This sends tokens, routed locally, to the correct expert now # split across different cores. # Core layout: [1, n, 1, 1] −> [n, 1, 1, 1] expert inputs = mtf.reshape(expert inputs, [num experts, num cores, expert capacity, d model]) # Standard feed forward computation, where each expert will have its own # unique set of parameters. # Total unique parameters created: num experts ∗ (d model ∗ d ff ∗ 2). # expert outputs shape: [num experts, num cores, expert capacity, d model] expert outputs = feed forward(expert inputs) # All−to−All communication. Cores are currently split across the experts # dimension, which needs to be switched back to being split across num cores. # Core Layout: [n, 1, 1, 1] −> [1, n, 1, 1] expert outputs = mtf.reshape(expert outputs, [num experts, num cores, expert capacity, d model]) # Convert back to input shape and multiply outputs of experts by the routing probability. # expert outputs shape: [num experts, num cores, tokens per core, d model] # expert outputs combined shape: [num cores, tokens per core, d model] # Core Layout: [1, n, 1, 1] −> [n, 1, 1] expert outputs combined = mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core]) # Remove tokens per core shapes used for local routing dispatching to match input shape. # Core Layout: [n, 1, 1] −> [n, 1, 1] outputs = mtf.reshape(expert outputs combined, [batch, seq len, d model]) return outputs, aux loss\n\n```\n\nFigure 16: Pseudo code of the Switch Transformer layer in Mesh Tensorflow.\n\nReferences\n\nMart´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265–283, 2016.\n\nIz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.\n\nJonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.\n\nTom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.\n\nRewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.\n\nKyunghyun Cho and Yoshua Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014.\n\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Gon¸calo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers.\n\narXiv preprint arXiv:1909.00015, 2019.\n\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.\n\nDavid Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013.\n\nAngela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48, 2021. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736, 2018.\n\nTrevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020.\n\nScott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights.\n\nhttps://openai.com/blog/block-sparse-gpu-kernels/, 2017.\n\nKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.\n\nKarl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\nSepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.\n\nSara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020.\n\nYanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in neural information processing systems, pages 103–112, 2019. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.\n\nMichael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.\n\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.\n\nNikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.\n\narXiv preprint arXiv:2001.04451, 2020.\n\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.\n\nGuillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv´e J´egou. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pages 8548–8559, 2019.\n\nKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.\n\nDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020.\n\nPaulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.\n\nMixed precision training. arXiv preprint arXiv:1710.03740, 2017.\n\nShashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018.\n\nYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.\n\nAdversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019.\n\nJoan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, Andr´e Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.\n\nAlec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.\n\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\n\nSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019.\n\nPranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.\n\nPrajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models.\n\nIn International Conference on Learning Representations, 2018.\n\nHerbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527–535, 1952.\n\nAdam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020.\n\nClemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020.\n\nVictor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.\n\nNoam Shazeer. Glu variants improve transformer, 2020.\n\nNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-ofexperts layer. arXiv preprint arXiv:1701.06538, 2017.\n\nNoam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. URL http://www.cs. toronto.edu/~rsalakhu/papers/srivastava14a.pdf. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.\n\nSainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799, 2019.\n\nRich Sutton. The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019.\n\nRichard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Stanford University, 2018.\n\nWilson L Taylor. \"cloze procedure\": A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953.\n\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.\n\nAlex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for generalpurpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3266–3280, 2019.\n\nShibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus.\n\nGoogle Cloud Blog, 2019.\n\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.\n\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.\n\nManzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020.",
+          "document_metadata": {
+            "source": "../../datasets/Sample_Docs_Markdown/switch_transformers.md"
+          },
+          "summary": "The Switch Transformer is a sparsely-activated model that simplifies the Mixture of Experts (MoE) approach, allowing for efficient scaling to trillion-parameter models with reduced communication and computational costs. It achieves significant speedups in pre-training and fine-tuning by activating only a subset of neural network weights for each input, maintaining a constant computational cost. The model demonstrates superior scaling properties and improved performance across various natural language processing tasks, including multilingual settings. It also allows for effective distillation into smaller dense models, preserving a portion of the quality gains. The architecture is designed to maximize parameter count while keeping floating point operations per example constant, making it suitable for both large-scale and smaller-scale applications.",
+          "headlines": {
+            "1 Introduction": [],
+            "2 Switch Transformer": [
+              "2.1 Simplifying Sparse Routing",
+              "2.2 Efficient Sparse Routing",
+              "2.3 Putting It All Together: The Switch Transformer",
+              "2.4 Improved Training and Fine-Tuning Techniques"
+            ],
+            "3 Scaling Properties": [
+              "3.1 Scaling Results on a Step-Basis",
+              "3.2 Scaling Results on a Time-Basis",
+              "3.3 Scaling Versus a Larger Dense Model"
+            ],
+            "4 Downstream Results": [
+              "4.1 Fine-Tuning",
+              "4.2 Distillation",
+              "4.3 Multilingual Learning"
+            ],
+            "5 Designing Models with Data, Model, and Expert-Parallelism": [
+              "5.1 Data Parallelism",
+              "5.2 Model Parallelism",
+              "5.3 Model and Data Parallelism",
+              "5.4 Expert and Data Parallelism",
+              "5.5 Expert, Model and Data Parallelism",
+              "5.6 Towards Trillion Parameter Models"
+            ],
+            "6 Related Work": [],
+            "7 Discussion": [],
+            "8 Future Work": [],
+            "9 Conclusion": [],
+            "A Switch for Attention": [],
+            "B Preventing Token Dropping with No-Token-Left-Behind": [],
+            "C Encouraging Exploration Across Experts": [],
+            "D Switch Transformers in Lower Compute Regimes": [],
+            "E Relation of Upstream to Downstream Model Performance": [],
+            "F Pseudo Code for Switch Transformers": []
+          },
+          "summary_embedding": [
+            -0.0060447100549936295,
+            -0.00390331051312387,
+            -0.01784370467066765,
+            -0.0021820589900016785,
+            0.013227885589003563,
+            0.011338187381625175,
+            0.0165580902248621,
+            0.024240797385573387,
+            -0.005986624862998724,
+            0.03596621751785278,
+            -0.004852031823247671,
+            -0.0068811336532235146,
+            -0.003574162255972624,
+            0.014242436736822128,
+            0.011578272096812725,
+            -0.033302053809165955,
+            -0.024519605562090874,
+            0.01050176378339529,
+            0.016093410551548004,
+            -0.006795942317694426,
+            0.007473600562661886,
+            -0.030513975769281387,
+            -0.06124480068683624,
+            -0.006838538218289614,
+            -0.0026099516544491053,
+            0.01029265858232975,
+            0.0010329445358365774,
+            -0.010873507708311081,
+            -0.053469158709049225,
+            0.023311438038945198,
+            0.025294071063399315,
+            0.005092116538435221,
+            -0.029584616422653198,
+            0.0009361362317577004,
+            -0.05161044001579285,
+            -0.04448534920811653,
+            0.0379488542675972,
+            0.012561844661831856,
+            -0.04364892467856407,
+            0.039249956607818604,
+            -0.004875265993177891,
+            -0.0022440163884311914,
+            -0.021499186754226685,
+            -0.0034909071400761604,
+            -0.007039899472147226,
+            -0.009851212613284588,
+            -0.02874819189310074,
+            0.0014037203509360552,
+            0.021886419504880905,
+            -0.0056962003000080585,
+            0.023264970630407333,
+            -0.024271776899695396,
+            0.01936165988445282,
+            0.04240977764129639,
+            -0.06957805901765823,
+            0.024659009650349617,
+            0.017549408599734306,
+            0.05120771750807762,
+            0.06511713564395905,
+            0.018339363858103752,
+            0.021669568493962288,
+            -0.014141756109893322,
+            -0.017766257748007774,
+            -0.011872569099068642,
+            0.004673904739320278,
+            -0.014970434829592705,
+            -0.01891246810555458,
+            0.008449427783489227,
+            -0.020461400970816612,
+            0.008209343999624252,
+            -0.05576157942414284,
+            -0.014730350114405155,
+            -0.034200433641672134,
+            -0.01140014547854662,
+            0.017642343416810036,
+            -0.018215449526906013,
+            0.030126741155982018,
+            0.02997184917330742,
+            0.002638994250446558,
+            0.0007822110201232135,
+            0.021049996837973595,
+            0.006509389728307724,
+            -0.0053438181057572365,
+            0.012321759946644306,
+            0.028159597888588905,
+            -0.01450575515627861,
+            -0.010106786154210567,
+            -0.03454120084643364,
+            0.01861817203462124,
+            0.01589204929769039,
+            -0.04194509983062744,
+            -0.012569589540362358,
+            -0.013824224472045898,
+            0.032341714948415756,
+            -0.009549170732498169,
+            0.04098476096987724,
+            -0.010083552449941635,
+            0.001383390510454774,
+            -0.031149037182331085,
+            -0.009463978931307793,
+            0.017069239169359207,
+            0.006172496825456619,
+            -0.016589069738984108,
+            -0.0012846461031585932,
+            0.016170857474207878,
+            0.006617815233767033,
+            -0.030142230913043022,
+            0.04488807171583176,
+            0.023497309535741806,
+            0.055296897888183594,
+            0.05579255893826485,
+            -0.037391237914562225,
+            -0.002445377642288804,
+            0.0344172865152359,
+            -0.0228312686085701,
+            -0.015040136873722076,
+            -0.034014564007520676,
+            -0.01768881268799305,
+            0.035935238003730774,
+            -0.0318460576236248,
+            0.01556677371263504,
+            0.028066661208868027,
+            -0.025712283328175545,
+            0.003488970920443535,
+            -0.03943582624197006,
+            0.003622566582635045,
+            0.01220559049397707,
+            0.006095050368458033,
+            -0.0026293133851140738,
+            -0.013615119270980358,
+            0.008906363509595394,
+            0.003055269829928875,
+            0.00800798274576664,
+            -0.04615819454193115,
+            0.018478767946362495,
+            -0.00454224506393075,
+            0.04033420979976654,
+            0.042502716183662415,
+            -0.02730768360197544,
+            -0.016743963584303856,
+            0.008557853288948536,
+            0.012104909867048264,
+            0.04188314080238342,
+            -0.007287728600203991,
+            -0.00121784838847816,
+            -0.0007836631848476827,
+            0.007322579622268677,
+            0.047552235424518585,
+            0.004646798130124807,
+            0.0020697612781077623,
+            -0.027230238541960716,
+            0.014978179708123207,
+            -0.019733402878046036,
+            0.027648448944091797,
+            0.023373395204544067,
+            0.02942972257733345,
+            -0.04048910364508629,
+            0.017518429085612297,
+            -0.011818356812000275,
+            -0.005568413063883781,
+            -0.002726121572777629,
+            -0.02856232039630413,
+            -0.0074426219798624516,
+            0.033983584493398666,
+            0.015365412458777428,
+            0.012956822291016579,
+            0.03271345794200897,
+            0.0012052633101120591,
+            -0.042502716183662415,
+            0.03804178908467293,
+            0.024550583213567734,
+            0.04640602692961693,
+            0.023466331884264946,
+            0.010796061716973782,
+            0.006571347359567881,
+            0.013375034555792809,
+            -0.019377149641513824,
+            0.015907539054751396,
+            0.02560385875403881,
+            0.009750531986355782,
+            -0.03167567402124405,
+            -0.0009501734166406095,
+            0.01178737822920084,
+            -0.004546117503196001,
+            0.03701949119567871,
+            0.04104671627283096,
+            0.020569827407598495,
+            0.05030933395028114,
+            0.0020155487582087517,
+            0.017936641350388527,
+            0.005936284549534321,
+            0.02820606529712677,
+            0.030467506498098373,
+            -0.005285732913762331,
+            0.005076627247035503,
+            -0.01136916596442461,
+            -0.03385967016220093,
+            0.014110777527093887,
+            0.01561324205249548,
+            -0.008077683858573437,
+            -0.01697630248963833,
+            0.006033093202859163,
+            0.009169681929051876,
+            -0.01795213110744953,
+            -0.017456471920013428,
+            0.005971135571599007,
+            0.018866000697016716,
+            -0.013576395809650421,
+            0.024689987301826477,
+            -0.04442339017987251,
+            0.029119934886693954,
+            -0.033209118992090225,
+            -0.003384418087080121,
+            -0.010641167871654034,
+            -0.0030843124259263277,
+            0.0041666291654109955,
+            0.011314953677356243,
+            0.013406013138592243,
+            0.002042655134573579,
+            -0.013909416273236275,
+            0.024519605562090874,
+            0.018447788432240486,
+            0.012073930352926254,
+            -0.03148980438709259,
+            -0.014397329650819302,
+            -0.0015673263696953654,
+            -0.008743725717067719,
+            -0.034138478338718414,
+            0.026966920122504234,
+            0.014048820361495018,
+            -0.03844451159238815,
+            0.017797237262129784,
+            0.029538147151470184,
+            0.014420564286410809,
+            -0.004790074657648802,
+            -0.0463130883872509,
+            0.04045812413096428,
+            -0.030436528846621513,
+            -0.033333033323287964,
+            -0.010478530079126358,
+            -0.0003146269591525197,
+            0.02894955314695835,
+            -0.0064358157105743885,
+            -0.050835974514484406,
+            0.01184933539479971,
+            0.020631784573197365,
+            0.00010763872705865651,
+            0.005731051322072744,
+            -0.019593998789787292,
+            0.00028631053282879293,
+            -0.03395260497927666,
+            0.0090844901278615,
+            0.026750069111585617,
+            0.008867640048265457,
+            -0.016418687999248505,
+            0.03351890295743942,
+            -0.0004121613164898008,
+            -0.018695618957281113,
+            -0.003889757441356778,
+            0.006885006092488766,
+            0.034912943840026855,
+            0.004836542531847954,
+            -0.02946070022881031,
+            0.05665995925664902,
+            -0.02100352756679058,
+            0.021344292908906937,
+            0.05631919577717781,
+            -0.019439106807112694,
+            -0.008472662419080734,
+            -0.002065889071673155,
+            0.0026196325197815895,
+            -0.0260530486702919,
+            -0.008937342092394829,
+            0.032341714948415756,
+            0.041728246957063675,
+            -0.018742086365818977,
+            -0.001398879918269813,
+            0.006780453026294708,
+            0.009425255469977856,
+            0.004948840010911226,
+            0.02692045085132122,
+            -0.012128143571317196,
+            -0.014559967443346977,
+            0.05198218300938606,
+            0.028314489871263504,
+            -0.012081675231456757,
+            -0.0164031982421875,
+            0.019888296723365784,
+            -0.024844881147146225,
+            -0.020244551822543144,
+            -0.02431824430823326,
+            -0.020414933562278748,
+            -0.0042518205009400845,
+            0.009525936096906662,
+            0.037484172731637955,
+            -0.03968365862965584,
+            0.012747716158628464,
+            0.021700548008084297,
+            0.02981695532798767,
+            -0.019532041624188423,
+            -0.05910727381706238,
+            -0.0016128262504935265,
+            0.03296129032969475,
+            -0.015458349138498306,
+            -0.049379974603652954,
+            0.015164051204919815,
+            0.0260530486702919,
+            0.05957195535302162,
+            -0.001461805310100317,
+            -0.014954946003854275,
+            0.061926331371068954,
+            0.03026614524424076,
+            -0.004383479710668325,
+            -0.009494957514107227,
+            -0.0127864396199584,
+            0.003396035172045231,
+            0.00465454300865531,
+            -0.03364282101392746,
+            -0.041170634329319,
+            -0.011841590516269207,
+            -0.010284913703799248,
+            0.008108663372695446,
+            -0.040922801941633224,
+            0.03531566634774208,
+            -0.021917399019002914,
+            0.010950954630970955,
+            0.015473837964236736,
+            0.037391237914562225,
+            -0.021344292908906937,
+            -0.01284839678555727,
+            -0.01572941243648529,
+            -0.024968795478343964,
+            -0.014010096900165081,
+            0.021917399019002914,
+            0.007682706229388714,
+            3.4699722164077684e-05,
+            0.010075807571411133,
+            -0.024659009650349617,
+            -0.051052823662757874,
+            0.0291044469922781,
+            0.0015605497173964977,
+            -0.015938518568873405,
+            -0.004623564425855875,
+            -0.01977987214922905,
+            -0.01122201792895794,
+            -0.012608313001692295,
+            -0.039249956607818604,
+            0.021111954003572464,
+            -0.01135367713868618,
+            0.015876561403274536,
+            0.0009250032599084079,
+            0.003659353591501713,
+            0.028624277561903,
+            0.01116006076335907,
+            0.0051811798475682735,
+            -0.03255856782197952,
+            -0.018277406692504883,
+            0.010122275911271572,
+            -0.00561100896447897,
+            -0.018339363858103752,
+            -0.013351799920201302,
+            -0.05037129297852516,
+            -0.03305422514677048,
+            -0.008054450154304504,
+            -0.05415068939328194,
+            0.0020697612781077623,
+            -0.014296649023890495,
+            -0.02232012152671814,
+            -0.005990497302263975,
+            0.03107159025967121,
+            -0.003235333366319537,
+            0.025294071063399315,
+            0.0076594725251197815,
+            0.037360258400440216,
+            0.040117356926202774,
+            0.008426194079220295,
+            0.03065337985754013,
+            -0.0029991210903972387,
+            -0.016867877915501595,
+            0.05486319959163666,
+            -0.01881953328847885,
+            0.013529927469789982,
+            -0.008604321628808975,
+            0.008604321628808975,
+            0.03299226611852646,
+            -0.003794885240495205,
+            0.02007416822016239,
+            -0.0311955064535141,
+            -0.07998688519001007,
+            0.01637221872806549,
+            0.028159597888588905,
+            -0.06418777257204056,
+            0.014691627584397793,
+            -0.020275529474020004,
+            0.02052335813641548,
+            0.02411688305437565,
+            -0.0009114501299336553,
+            -0.014064309187233448,
+            0.005595519673079252,
+            0.0020833145827054977,
+            -0.01456771232187748,
+            -0.002691270550712943,
+            -0.009417511522769928,
+            0.008774704299867153,
+            -0.017007282003760338,
+            -0.013406013138592243,
+            -0.034231413155794144,
+            0.010873507708311081,
+            0.01233724970370531,
+            -0.003969140350818634,
+            0.0011500825639814138,
+            -0.03277541697025299,
+            -0.007876322604715824,
+            -0.02145271934568882,
+            0.03382869064807892,
+            0.012058441527187824,
+            -0.030699847266077995,
+            -0.007814365439116955,
+            0.015264731831848621,
+            -0.007876322604715824,
+            0.0008906363509595394,
+            0.024132372811436653,
+            0.02103450708091259,
+            0.0028364830650389194,
+            -0.0009700191440060735,
+            0.02438020147383213,
+            0.007427132688462734,
+            -0.01720864325761795,
+            0.028794659301638603,
+            0.014288905076682568,
+            0.060160547494888306,
+            -0.03271345794200897,
+            0.001297231181524694,
+            -0.04135650396347046,
+            -0.031118059530854225,
+            0.006408709101378918,
+            0.014350862242281437,
+            0.007597515359520912,
+            -0.028252532705664635,
+            0.005978880450129509,
+            0.02004319056868553,
+            0.01469937153160572,
+            0.009704063646495342,
+            0.0037580979987978935,
+            -0.036740683019161224,
+            -0.000691695255227387,
+            -0.011214273050427437,
+            -0.0038781403563916683,
+            0.035656433552503586,
+            -0.008186109364032745,
+            -0.05151750147342682,
+            -0.0006418389966711402,
+            -0.03255856782197952,
+            -0.0353776253759861,
+            0.021313315257430077,
+            -0.00839521549642086,
+            -0.0039207362569868565,
+            -0.013986863195896149,
+            -0.000537770101800561,
+            -0.010524998418986797,
+            0.027509044855833054,
+            0.028345469385385513,
+            0.022490503266453743,
+            -0.05384090170264244,
+            0.009479468688368797,
+            0.00023766436788719147,
+            -0.05817791447043419,
+            0.0009709871956147254,
+            -0.007117345929145813,
+            0.011191039346158504,
+            -0.020213572308421135,
+            -0.02414786070585251,
+            0.011500826105475426,
+            0.03392162546515465,
+            0.014304393902420998,
+            0.0321558453142643,
+            -0.013367289677262306,
+            -0.09250226616859436,
+            0.018292896449565887,
+            -0.008689512498676777,
+            0.04126356914639473,
+            0.00865078903734684,
+            -0.004278926644474268,
+            -0.015845581889152527,
+            -0.00865078903734684,
+            -0.006943091284483671,
+            0.01283290795981884,
+            0.004530628211796284,
+            -0.005626498255878687,
+            0.0016631665639579296,
+            0.013708055019378662,
+            -0.005967263597995043,
+            0.007481344975531101,
+            0.003651608945801854,
+            -0.013189162127673626,
+            -0.06059424951672554,
+            -0.04055105894804001,
+            -0.018540725111961365,
+            0.050835974514484406,
+            -0.0069159846752882,
+            -0.046684831380844116,
+            -0.010850274004042149,
+            0.05058814212679863,
+            0.012259802781045437,
+            -0.0334569476544857,
+            -0.0007841471815481782,
+            0.0050882440991699696,
+            0.015458349138498306,
+            0.004546117503196001,
+            -0.027292195707559586,
+            0.02820606529712677,
+            0.0008412640891037881,
+            0.02300165221095085,
+            -0.018339363858103752,
+            -0.005312839522957802,
+            0.035780347883701324,
+            -0.024844881147146225,
+            0.023791607469320297,
+            0.0055219451896846294,
+            0.031629208475351334,
+            0.002952652983367443,
+            0.010060317814350128,
+            0.004704883322119713,
+            0.0186491496860981,
+            0.004275054205209017,
+            0.0276639387011528,
+            0.011462102644145489,
+            0.006428070832043886,
+            0.021623101085424423,
+            -0.009378788061439991,
+            -0.014513500034809113,
+            0.01885051093995571,
+            -0.0014695499558001757,
+            0.025232113897800446,
+            -0.020538847893476486,
+            0.002726121572777629,
+            -0.0016883367206901312,
+            -0.017967619001865387,
+            -0.040922801941633224,
+            -0.0007405834621749818,
+            0.021747015416622162,
+            -0.011671207845211029,
+            -0.0228312686085701,
+            -0.013715799897909164,
+            -0.01585332676768303,
+            0.0038490979932248592,
+            0.034169457852840424,
+            -0.0398695282638073,
+            0.00963436160236597,
+            -0.000830615172162652,
+            -0.03125746175646782,
+            -0.0239619892090559,
+            -0.01093546487390995,
+            0.014614180661737919,
+            -0.01652711257338524,
+            -0.021173911169171333,
+            -0.028469383716583252,
+            -0.008410705253481865,
+            0.006048582494258881,
+            -0.05065010115504265,
+            -0.004054331686347723,
+            0.019965743646025658,
+            0.0276639387011528,
+            -0.003099801717326045,
+            -0.02177799493074417,
+            0.0012468908680602908,
+            0.022351099178195,
+            0.02936776541173458,
+            0.014226946979761124,
+            0.027586491778492928,
+            -0.002178186783567071,
+            -0.0011675080750137568,
+            -0.0073148347437381744,
+            0.01307299267500639,
+            0.0011287847300991416,
+            0.036616768687963486,
+            0.0122210793197155,
+            -0.020306508988142014,
+            0.0013959755888208747,
+            -0.006431943271309137,
+            0.029181893914937973,
+            -0.02270735427737236,
+            -9.832092473516241e-05,
+            0.03531566634774208,
+            0.027741385623812675,
+            0.004933350719511509,
+            -0.0143121387809515,
+            -0.05142456665635109,
+            0.021592123433947563,
+            -0.01765783317387104,
+            0.031025122851133347,
+            0.016666516661643982,
+            0.003397971158847213,
+            -0.0034212053287774324,
+            0.012368228286504745,
+            -0.022087780758738518,
+            -0.004406713880598545,
+            -0.021142931655049324,
+            0.04389675334095955,
+            -0.043184246867895126,
+            0.008472662419080734,
+            0.029445212334394455,
+            0.011926782317459583,
+            -0.019082851707935333,
+            -0.005425137002021074,
+            0.029212871566414833,
+            -0.00162637943867594,
+            0.01308848150074482,
+            0.0058665829710662365,
+            -0.026935940608382225,
+            0.01202746294438839,
+            -0.031660184264183044,
+            0.02177799493074417,
+            -0.013436991721391678,
+            0.018447788432240486,
+            0.03342596814036369,
+            -0.0030397805385291576,
+            -0.0038239278364926577,
+            -0.016263794153928757,
+            -0.00969631876796484,
+            0.0662323608994484,
+            0.01054823212325573,
+            0.03726732358336449,
+            0.0018974426202476025,
+            -0.009773765690624714,
+            0.007969259284436703,
+            0.0063041565008461475,
+            -0.012329504825174809,
+            0.010494019836187363,
+            0.024178840219974518,
+            0.0012730291346088052,
+            -0.002412462839856744,
+            -0.019408127292990685,
+            -0.03438630700111389,
+            0.003727119415998459,
+            -0.030932186171412468,
+            0.022691864520311356,
+            0.009208405390381813,
+            0.017456471920013428,
+            0.008240322582423687,
+            -0.015907539054751396,
+            0.03200095146894455,
+            0.05461536720395088,
+            0.0019768255297094584,
+            0.013375034555792809,
+            0.007938280701637268,
+            0.007036027032881975,
+            -0.011338187381625175,
+            0.04702559858560562,
+            0.000662652775645256,
+            0.005169562995433807,
+            0.022536970674991608,
+            0.02145271934568882,
+            0.022939693182706833,
+            0.002304037567228079,
+            -0.011694442480802536,
+            0.010478530079126358,
+            -0.0006302220281213522,
+            0.004445436876267195,
+            -0.010695381090044975,
+            -0.027865299955010414,
+            0.007492962293326855,
+            0.01302652433514595,
+            0.022598929703235626,
+            0.021592123433947563,
+            -0.03178409859538078,
+            -0.01855621486902237,
+            0.03364282101392746,
+            -0.02997184917330742,
+            0.03612111136317253,
+            -0.0014424435794353485,
+            -0.023807097226381302,
+            -0.0058356039226055145,
+            0.016929835081100464,
+            0.02656419761478901,
+            -0.029181893914937973,
+            0.01804506592452526,
+            0.029693040996789932,
+            -0.0074426219798624516,
+            0.0005939188995398581,
+            0.012112653814256191,
+            0.02447313815355301,
+            -0.009293596260249615,
+            -0.0024027819745242596,
+            -0.0013320822035893798,
+            -0.011926782317459583,
+            -0.018370341509580612,
+            -0.0044725434854626656,
+            -0.022629907354712486,
+            0.005564541090279818,
+            -0.007365175057202578,
+            -0.003481226274743676,
+            -0.008581086993217468,
+            0.01154729351401329,
+            0.001979729626327753,
+            -0.019857317209243774,
+            -0.022335609421133995,
+            0.01652711257338524,
+            0.03887821361422539,
+            -0.018215449526906013,
+            -0.02106548473238945,
+            0.024442158639431,
+            -0.01759587600827217,
+            -0.01601596549153328,
+            0.036678727716207504,
+            0.013739033602178097,
+            0.007736918982118368,
+            -0.03358086198568344,
+            -0.030049296095967293,
+            -0.03274443745613098,
+            0.006296411622315645,
+            0.03785591572523117,
+            -0.0028752065263688564,
+            0.02569679357111454,
+            0.03701949119567871,
+            0.0057891360484063625,
+            -0.028934063389897346,
+            -0.014823286794126034,
+            -0.045104920864105225,
+            0.060160547494888306,
+            0.0011481463443487883,
+            0.05161044001579285,
+            0.016294771805405617,
+            -0.01456771232187748,
+            0.00419760774821043,
+            -0.013251119293272495,
+            0.026192452758550644,
+            0.02280029095709324,
+            0.025929134339094162,
+            0.006582964211702347,
+            0.022505993023514748,
+            -0.0037522895727306604,
+            -0.012197845615446568,
+            0.007307090330868959,
+            0.014536733739078045,
+            0.0002497654058970511,
+            -0.011392400600016117,
+            0.015783624723553658,
+            -0.01307299267500639,
+            -0.011361422017216682,
+            0.023295948281884193,
+            0.01368482131510973,
+            -0.012344993650913239,
+            0.014064309187233448,
+            -0.008565598167479038,
+            0.01239920686930418,
+            -0.023915521800518036,
+            0.022691864520311356,
+            -0.03262052312493324,
+            0.009587894193828106,
+            0.0015053689712658525,
+            -0.0073148347437381744,
+            0.024550583213567734,
+            -0.0027454833034425974,
+            -0.05071205645799637,
+            0.03550153970718384,
+            -0.008015726692974567,
+            0.0037213109899312258,
+            0.006381602957844734,
+            0.011392400600016117,
+            -0.0012623801594600081,
+            0.03807276859879494,
+            0.01239920686930418,
+            0.015969496220350266,
+            0.02199484594166279,
+            0.011911292560398579,
+            -0.008433938957750797,
+            -0.004096927121281624,
+            -0.0016137943603098392,
+            -0.01784370467066765,
+            0.025944624096155167,
+            0.0018674320308491588,
+            0.03962169960141182,
+            -0.018168980255723,
+            -0.02300165221095085,
+            0.028066661208868027,
+            0.007597515359520912,
+            -0.02351279929280281,
+            -0.0017377089243382215,
+            0.01202746294438839,
+            -0.009185170754790306,
+            0.027385130524635315,
+            0.01196550577878952,
+            0.019113829359412193,
+            -0.018060555681586266,
+            -0.004340883810073137,
+            -0.005072754807770252,
+            -0.015938518568873405,
+            0.02238207869231701,
+            0.020771188661456108,
+            0.011152315884828568,
+            -0.011880313977599144,
+            -0.01307299267500639,
+            0.020600805059075356,
+            0.035036858171224594,
+            -0.0002744515077210963,
+            0.005672966130077839,
+            0.017487451434135437,
+            0.002373739378526807,
+            -0.04470219835639,
+            0.016232814639806747,
+            -0.014025585725903511,
+            0.011214273050427437,
+            0.019593998789787292,
+            -0.021917399019002914,
+            -0.03196997195482254,
+            -0.011539548635482788,
+            -0.0414804182946682,
+            0.00493722315877676,
+            -0.0069159846752882,
+            0.013576395809650421,
+            -0.010176488198339939,
+            -0.0014831031439825892,
+            0.016666516661643982,
+            0.0038103745318949223,
+            0.02177799493074417,
+            -0.011678952723741531,
+            -0.012073930352926254,
+            -0.020213572308421135,
+            -0.020786678418517113,
+            0.004278926644474268,
+            -0.030405549332499504,
+            -0.004488032776862383,
+            0.021375272423028946,
+            -0.04783104360103607,
+            -0.03689557686448097,
+            0.02898053266108036,
+            -0.008503641001880169,
+            0.020755698904395103,
+            0.022955182939767838,
+            -0.015094350092113018,
+            -0.0008717587334103882,
+            -0.018695618957281113,
+            0.02258343994617462,
+            -0.007837600074708462,
+            0.026300877332687378,
+            -0.0027551641687750816,
+            0.0006292539183050394,
+            -0.02659517526626587,
+            0.020306508988142014,
+            0.005227648187428713,
+            0.012275291606783867,
+            -0.02390003204345703,
+            0.006815304048359394,
+            -0.03133491054177284,
+            -0.01784370467066765,
+            0.0180140882730484,
+            -0.011152315884828568,
+            0.027989214286208153,
+            -0.006060199346393347,
+            0.028438406065106392,
+            0.022722844034433365,
+            0.024751944467425346,
+            0.02676555886864662,
+            -0.02363671362400055,
+            0.008495896123349667,
+            0.03726732358336449,
+            0.00865078903734684,
+            -0.015876561403274536,
+            0.008170620538294315,
+            -0.04863648861646652,
+            -0.008062195032835007,
+            0.05030933395028114,
+            0.002127846470102668,
+            0.023141054436564445,
+            0.016294771805405617,
+            -0.002873270306736231,
+            -0.005688455421477556,
+            0.010269423946738243,
+            -0.04296739399433136,
+            -0.023172033950686455,
+            0.022629907354712486,
+            -0.021561143919825554,
+            6.54212328754511e-07,
+            0.03122648410499096,
+            0.012120398692786694,
+            -0.0009109660750254989,
+            -0.05108380317687988,
+            0.02100352756679058,
+            0.01178737822920084,
+            0.014443797990679741,
+            -0.010300402529537678,
+            0.026068538427352905,
+            0.01733255758881569,
+            0.038413532078266144,
+            0.012755461037158966,
+            -0.002679653698578477,
+            -0.03782493621110916,
+            0.009177426807582378,
+            -0.021685058251023293,
+            0.0011471783509477973,
+            0.0074503663927316666,
+            -0.02273833192884922,
+            -0.024782923981547356,
+            0.0043912241235375404,
+            0.004588713403791189,
+            0.011880313977599144,
+            0.022087780758738518,
+            0.009417511522769928,
+            -0.021204888820648193,
+            0.015404135920107365,
+            -0.05446047708392143,
+            -0.015450604259967804,
+            -0.023884542286396027,
+            0.02566581591963768,
+            0.020012211054563522,
+            0.010184233076870441,
+            0.005545179359614849,
+            0.010904486291110516,
+            -0.020988039672374725,
+            -0.009239383973181248,
+            0.020631784573197365,
+            0.011074868962168694,
+            -0.012840652838349342,
+            0.04113965481519699,
+            -0.00011126903700642288,
+            0.031210994347929955,
+            -0.02280029095709324,
+            0.002941036131232977,
+            -0.001241082325577736,
+            -0.03649285435676575,
+            0.009130958467721939,
+            0.01981084980070591,
+            0.02151467651128769,
+            -0.01861817203462124,
+            0.0024143988266587257,
+            0.01048627495765686,
+            0.004766840487718582,
+            0.038351576775312424,
+            0.027121812105178833,
+            -0.044206541031599045,
+            0.011663463898003101,
+            0.012995545752346516,
+            -0.009363298304378986,
+            0.048419639468193054,
+            -0.017518429085612297,
+            0.017673322930932045,
+            0.008240322582423687,
+            -0.01798310875892639,
+            0.02318752370774746,
+            -0.01469937153160572,
+            -0.013282098807394505,
+            0.00650551775470376,
+            0.009192915633320808,
+            -0.0001770381786627695,
+            -0.037577107548713684,
+            -0.017859194427728653,
+            -0.005719434004276991,
+            0.0038297362625598907,
+            0.01562873087823391,
+            0.013498948886990547,
+            0.009967382065951824,
+            0.005161818582564592,
+            -0.00945623405277729,
+            -0.0016089539276435971,
+            0.008124152198433876,
+            -0.0005532593932002783,
+            0.0019216446671634912,
+            -0.01926872320473194,
+            0.02698240801692009,
+            0.02572777308523655,
+            -0.0018354853382334113,
+            -0.04652994126081467,
+            -0.0036709706764668226,
+            0.0037077576853334904,
+            -0.017735280096530914,
+            -0.0035315665882080793,
+            -0.04519785940647125,
+            -0.01810702309012413,
+            0.0143121387809515,
+            -0.014614180661737919,
+            -0.0460033044219017,
+            0.009076746180653572,
+            -0.013886182568967342,
+            0.0063970922492444515,
+            -0.00932457484304905,
+            0.005762029904872179,
+            0.003651608945801854,
+            0.005680711008608341,
+            -0.005909178406000137,
+            0.05108380317687988,
+            -0.03466511517763138,
+            0.028051171451807022,
+            0.008565598167479038,
+            -0.012437930330634117,
+            -0.02261441759765148,
+            0.00887538492679596,
+            -0.044981006532907486,
+            -0.033209118992090225,
+            0.016743963584303856,
+            -0.01781272701919079,
+            0.011074868962168694,
+            0.06161654368042946,
+            0.007330324035137892,
+            -0.023342415690422058,
+            -0.01233724970370531,
+            -0.021049996837973595,
+            -0.04191412031650543,
+            0.003442503046244383,
+            0.005463860463351011,
+            -0.004948840010911226,
+            -0.03807276859879494,
+            -0.021716037765145302,
+            0.04615819454193115,
+            -0.012004229240119457,
+            0.012422440573573112,
+            0.01080380566418171,
+            -0.0019410063978284597,
+            -0.04860550910234451,
+            -0.021019017323851585,
+            0.0395597442984581,
+            0.021886419504880905,
+            -0.008426194079220295,
+            0.0028364830650389194,
+            -0.018277406692504883,
+            -0.003289545886218548,
+            0.006296411622315645,
+            -0.009107724763453007,
+            -0.03562545403838158,
+            0.02067825198173523,
+            0.0054832217283546925,
+            -0.023373395204544067,
+            -0.025805220007896423,
+            -0.03296129032969475,
+            0.009944148361682892,
+            0.02453509531915188,
+            -0.01968693546950817,
+            0.005823987070471048,
+            0.00027203132049180567,
+            -0.000881923595443368,
+            0.00042740863864310086,
+            -0.009959637187421322,
+            -0.019764382392168045,
+            -0.020151615142822266,
+            0.03268248215317726,
+            -0.0060059865936636925,
+            0.0074387495405972,
+            -0.03630698472261429,
+            -0.010974188335239887,
+            0.011322698555886745,
+            -0.002395037328824401,
+            -0.013452480547130108,
+            -0.009092235006392002,
+            -0.01739451475441456,
+            -0.009882191196084023,
+            -0.007876322604715824,
+            0.022660886868834496,
+            0.006168624851852655,
+            0.005409647710621357,
+            0.0023795480374246836,
+            -0.008581086993217468,
+            0.02898053266108036,
+            0.017874684184789658,
+            -0.014188224449753761,
+            -0.00920066051185131,
+            0.03451022133231163,
+            0.02665713243186474,
+            -0.009115469641983509,
+            0.01798310875892639,
+            0.023264970630407333,
+            -0.0010484338272362947,
+            -0.01196550577878952,
+            0.001922612776979804,
+            -0.0009550138493068516,
+            -0.0032972905319184065,
+            0.00845717266201973,
+            -0.015365412458777428,
+            0.0315207801759243,
+            0.04287445917725563,
+            -0.002145271748304367,
+            0.009587894193828106,
+            -0.003727119415998459,
+            -0.008472662419080734,
+            0.017069239169359207,
+            0.009316830895841122,
+            0.04625113308429718,
+            0.01098193321377039,
+            0.005680711008608341,
+            -0.033983584493398666,
+            -0.001229465357027948,
+            -0.01184933539479971,
+            0.03308520466089249,
+            0.02061629481613636,
+            0.00889087375253439,
+            -0.008379725739359856,
+            0.01344473659992218,
+            0.008341003209352493,
+            0.00661007035523653,
+            0.020569827407598495,
+            -0.016542602330446243,
+            -0.017286090180277824,
+            0.030467506498098373,
+            -0.00672236829996109,
+            0.01289486512541771,
+            0.02029101923108101,
+            -0.002089123008772731,
+            0.010842529125511646,
+            0.07100307941436768,
+            -0.006184114143252373,
+            0.015713922679424286,
+            -0.019082851707935333,
+            0.0015789433382451534,
+            -0.007640110794454813,
+            -0.03370477631688118,
+            0.018246427178382874,
+            0.0039962464943528175,
+            0.00865078903734684,
+            0.01198099460452795,
+            0.009053511545062065,
+            -0.007946024648845196,
+            -0.00963436160236597,
+            0.015195030719041824,
+            0.0019080914789810777,
+            0.00033568276558071375,
+            0.001336922636255622,
+            -0.0065210070461034775,
+            -0.004011735785752535,
+            -0.021948376670479774,
+            0.02763296104967594,
+            0.007140580099076033,
+            0.030606910586357117,
+            -0.055296897888183594,
+            0.030095763504505157,
+            -0.00789568480104208,
+            0.02554190158843994,
+            -0.016201836988329887,
+            -0.011090358719229698,
+            0.001609921921044588,
+            0.018153492361307144,
+            0.004491904750466347,
+            0.006462921854108572,
+            0.011888058856129646,
+            0.023233991116285324,
+            -0.03444826602935791,
+            0.017053749412298203,
+            0.023110076785087585,
+            0.0048675211146473885,
+            0.0003535923024173826,
+            0.005165690556168556,
+            0.015458349138498306,
+            0.007198664825409651,
+            -0.023048119619488716,
+            -0.0025692922063171864,
+            0.017611365765333176,
+            -0.02306360937654972,
+            0.0036980770528316498,
+            -0.006923729553818703,
+            0.004623564425855875,
+            -0.0014705180656164885,
+            -0.03509881719946861,
+            -0.00100099784322083,
+            -0.025743262842297554,
+            0.01923774555325508,
+            0.009556914679706097,
+            -0.025867177173495293,
+            0.003932353109121323,
+            0.025650326162576675,
+            -0.0003695656778290868,
+            0.00821708794683218,
+            0.0004683101433329284,
+            0.00869725737720728,
+            -0.00974278710782528,
+            -0.01493945624679327,
+            0.004801691509783268,
+            0.01531120017170906,
+            -0.01493945624679327,
+            -0.007431004662066698,
+            -0.009510447271168232,
+            0.030560443177819252,
+            0.0006674932083114982,
+            0.004189862869679928,
+            -0.005045648664236069,
+            -0.01858719252049923,
+            -0.0032237162813544273,
+            -0.021142931655049324,
+            0.004367990419268608,
+            -0.0009046735358424485,
+            0.04197607934474945,
+            0.0038161829579621553,
+            -0.02029101923108101,
+            0.0032972905319184065,
+            -0.03136589005589485,
+            0.0005000148667022586,
+            -0.007276111748069525,
+            -0.02004319056868553,
+            0.0015005286550149322,
+            0.0016525175888091326,
+            0.012631546705961227,
+            -0.03255856782197952,
+            -0.009301341138780117,
+            0.018416810780763626,
+            -0.00020680672605521977,
+            -0.010687636211514473,
+            -0.017828216776251793,
+            0.00035189816844649613,
+            -0.011338187381625175,
+            -0.027973726391792297,
+            -0.02850036323070526,
+            -0.0026719090528786182,
+            0.013491204008460045,
+            -0.018277406692504883,
+            -0.014993669465184212,
+            0.0031249718740582466,
+            0.021592123433947563,
+            -0.009781510569155216,
+            0.008302279748022556,
+            0.02241305634379387,
+            -0.008255811408162117,
+            -0.005514200776815414,
+            -0.000941460661124438,
+            -0.009130958467721939,
+            0.004267309792339802,
+            0.006300284061580896,
+            0.043339136987924576,
+            -0.032403673976659775,
+            -0.0018238683696836233,
+            0.014815541915595531,
+            0.0034153966698795557,
+            0.0025905899237841368,
+            0.002503462601453066,
+            -0.0007841471815481782,
+            0.01344473659992218,
+            0.01178737822920084,
+            0.015466093085706234,
+            0.00100099784322083,
+            -0.0033011629711836576,
+            0.006408709101378918,
+            -0.021173911169171333,
+            0.015721667557954788,
+            0.027958236634731293,
+            -0.0033999073784798384,
+            0.005525817628949881,
+            -0.013731288723647594,
+            -0.008759214542806149,
+            0.013134949840605259,
+            0.0026641644071787596,
+            -0.005068882368505001,
+            -0.009394276887178421,
+            0.017967619001865387,
+            0.019082851707935333,
+            0.006668155547231436,
+            -0.037453193217515945,
+            -0.02898053266108036,
+            -0.009959637187421322,
+            0.020569827407598495,
+            0.014916222542524338,
+            0.0010339126456528902,
+            -0.010540487244725227,
+            -0.009595638141036034,
+            -0.011903548613190651,
+            0.010989678092300892,
+            0.02721474878489971,
+            -0.032372694462537766,
+            0.0008214184199459851,
+            -0.029383255168795586,
+            0.03068435750901699,
+            0.0020000594668090343,
+            0.011446612887084484,
+            -0.0008185141487047076,
+            -0.015140817500650883,
+            -0.029181893914937973,
+            0.013824224472045898,
+            -0.004395096562802792,
+            0.001024231780320406,
+            -0.0031753121875226498,
+            0.0201051477342844,
+            0.012887120246887207,
+            -0.0014472840121015906,
+            0.01968693546950817,
+            0.008178364485502243,
+            0.016480645164847374,
+            -0.010153254494071007,
+            -0.04135650396347046,
+            0.019965743646025658,
+            0.007562664337456226,
+            0.014219203032553196,
+            0.004739734344184399,
+            0.004588713403791189,
+            -0.020771188661456108,
+            0.002724185585975647,
+            0.004333139397203922,
+            0.014405074529349804,
+            0.011028401553630829,
+            0.0025615475606173277,
+            0.011678952723741531,
+            -0.02228914201259613,
+            0.01929970271885395,
+            -0.01631026156246662,
+            0.01006806269288063,
+            0.004445436876267195,
+            -0.026873983442783356,
+            0.007547175046056509,
+            0.007822110317647457,
+            -0.0019187404541298747,
+            -0.017549408599734306,
+            0.02160761132836342,
+            -0.0034831624943763018,
+            -0.03509881719946861,
+            0.0360281765460968,
+            -0.0043447562493383884,
+            0.019346170127391815,
+            0.03209388628602028,
+            -0.01771979033946991,
+            -0.01993476413190365,
+            0.00906900130212307,
+            0.010408828034996986,
+            -0.002873270306736231,
+            0.017859194427728653,
+            -0.002919738180935383,
+            -0.02232012152671814,
+            0.017766257748007774,
+            -0.028918573632836342,
+            -0.014234691858291626,
+            0.020337486639618874,
+            -0.03624502569437027,
+            -0.0021026763133704662,
+            0.03175312280654907,
+            -0.024070415645837784,
+            0.005734923295676708,
+            0.005196669604629278,
+            -0.0019351978553459048,
+            0.0012236569309607148,
+            -0.006451305001974106,
+            0.03801080957055092,
+            -0.010865762829780579,
+            0.0201051477342844,
+            0.01929970271885395,
+            -0.023388884961605072,
+            0.01977987214922905,
+            -0.001109423115849495,
+            0.01675945147871971,
+            -0.003940097521990538,
+            0.03466511517763138,
+            -0.006017603911459446,
+            -0.011330443434417248,
+            0.006431943271309137,
+            0.008488151244819164,
+            -0.009959637187421322,
+            0.008681767620146275,
+            -0.018122512847185135,
+            0.027152791619300842,
+            0.02091059274971485,
+            -0.0002691270783543587,
+            0.013800990767776966,
+            0.0004421719058882445,
+            -0.020833145827054977,
+            0.00789568480104208,
+            -0.027121812105178833,
+            -0.009967382065951824,
+            0.01646515540778637,
+            -0.005762029904872179,
+            0.007380664348602295,
+            -0.02064727433025837,
+            0.028004704043269157,
+            0.0077524082735180855,
+            -0.013591884635388851,
+            -0.0008824076503515244,
+            0.023760627955198288,
+            -0.005432881414890289,
+            0.009355553425848484,
+            0.02070923149585724,
+            0.015713922679424286,
+            0.013615119270980358,
+            0.016263794153928757,
+            0.03351890295743942,
+            0.005328328814357519,
+            0.005568413063883781,
+            0.003519949736073613,
+            0.016991792246699333,
+            -0.008557853288948536,
+            0.018029578030109406,
+            0.0008093173382803798,
+            -0.013576395809650421,
+            0.021499186754226685,
+            -0.0018684001406654716,
+            -0.013622863218188286,
+            -0.0040891822427511215,
+            -0.004337011836469173,
+            0.03271345794200897,
+            0.022041313350200653,
+            -0.004468671046197414,
+            -0.024442158639431,
+            -0.00394784240052104,
+            0.016682006418704987,
+            -0.003167567541822791,
+            0.011136826127767563,
+            0.0007952801533974707,
+            -0.01586107164621353,
+            -0.009595638141036034,
+            0.0063583687879145145,
+            0.010586955584585667,
+            -0.008921852335333824,
+            0.010346870869398117,
+            -0.0228312686085701,
+            -0.004011735785752535,
+            0.00046613195445388556,
+            -0.004844287410378456,
+            0.0053438181057572365,
+            -0.011314953677356243,
+            0.004948840010911226,
+            0.016062432900071144,
+            0.01026167906820774,
+            0.004832670092582703,
+            0.006989559158682823,
+            0.012066186405718327,
+            0.009161937050521374,
+            -0.01627928391098976,
+            0.00428667152300477,
+            -0.005072754807770252,
+            -0.02345084212720394,
+            0.005831731949001551,
+            0.00902253296226263,
+            -0.01122201792895794,
+            0.009386532939970493,
+            -0.010958699509501457,
+            0.03296129032969475,
+            0.01030814740806818,
+            -0.037391237914562225,
+            -0.0033379499800503254,
+            -0.0016244432190433145,
+            0.014885243959724903,
+            0.012298526242375374,
+            0.008550108410418034,
+            -0.017518429085612297,
+            -0.0047823297791182995,
+            -0.009835722856223583,
+            0.0012827098835259676,
+            -0.012306271120905876,
+            0.017456471920013428,
+            0.022567950189113617,
+            -0.036337960511446,
+            0.01968693546950817,
+            -0.010432061739265919,
+            -0.023652203381061554,
+            0.020476890727877617,
+            0.011500826105475426,
+            0.01942361705005169,
+            -0.009402021765708923,
+            0.018896980211138725,
+            -0.0001331114035565406,
+            0.025510922074317932,
+            -0.004228586331009865,
+            -0.022087780758738518,
+            -0.012499887496232986,
+            -0.03209388628602028,
+            -0.0051811798475682735,
+            0.006819176487624645,
+            -0.00045524101005867124,
+            0.013111716136336327,
+            0.014226946979761124,
+            -0.025712283328175545,
+            0.007295473478734493,
+            0.032589543610811234,
+            0.014637414366006851,
+            0.023311438038945198,
+            -0.011276230216026306,
+            -0.0010794125264510512,
+            -0.0026060794480144978,
+            -0.013770012184977531,
+            -0.03212486580014229,
+            -0.0017493258928880095,
+            -0.036585792899131775,
+            0.005680711008608341,
+            -0.016387708485126495,
+            -0.013096226379275322,
+            0.023264970630407333,
+            -0.034138478338718414,
+            0.0014608372002840042,
+            0.013708055019378662,
+            0.010540487244725227,
+            0.012391461990773678,
+            -0.00162637943867594,
+            0.015032391995191574,
+            -0.029243851080536842,
+            -0.010284913703799248,
+            -0.004491904750466347,
+            -0.01981084980070591,
+            -0.007338068913668394,
+            -0.018060555681586266,
+            -0.013127204962074757,
+            -0.003566417610272765,
+            -0.010060317814350128,
+            0.002247888594865799,
+            0.0032062907703220844,
+            -0.014714861288666725,
+            0.03206290677189827,
+            0.003157886676490307,
+            0.018958937376737595,
+            0.0033418224193155766,
+            0.004449309315532446,
+            0.018788553774356842,
+            -0.0026331855915486813,
+            -0.0058278595097362995,
+            0.030963165685534477,
+            -0.00032672801171429455,
+            0.008759214542806149,
+            0.019082851707935333,
+            0.004724245052784681,
+            0.0038529701996594667,
+            -0.0260530486702919,
+            0.012887120246887207,
+            -0.027447087690234184,
+            0.005080499220639467,
+            -0.0077175572514534,
+            -0.024302754551172256,
+            -0.028128618374466896,
+            -0.006571347359567881,
+            0.012058441527187824,
+            -0.02898053266108036,
+            -0.00613377382978797,
+            0.002315654419362545,
+            -0.00400399137288332,
+            -0.002673845272511244,
+            0.007241260726004839,
+            0.029150914400815964,
+            -0.033209118992090225,
+            0.008751469664275646,
+            0.005595519673079252,
+            -0.0005053392960689962,
+            0.013297587633132935,
+            0.008038961328566074,
+            0.003343758638948202,
+            0.020848635584115982,
+            0.006637176964432001,
+            0.002458930714055896,
+            -0.0025576751213520765,
+            -0.014242436736822128,
+            0.004000118933618069,
+            0.02154565416276455,
+            0.007562664337456226,
+            0.018385831266641617,
+            -0.008914108388125896,
+            0.002621568739414215,
+            -0.01455222349613905,
+            0.005552923772484064,
+            -0.002867461647838354,
+            0.004309905227273703,
+            -0.012236569076776505,
+            0.008093173615634441,
+            0.0028771425131708384,
+            -0.0043447562493383884,
+            0.018060555681586266,
+            -0.002319526858627796,
+            0.016480645164847374,
+            0.004015608225017786,
+            0.015620986931025982,
+            -0.0035412474535405636,
+            0.01178737822920084,
+            0.0005731051205657423,
+            -0.017905661836266518,
+            -0.017022771760821342,
+            -0.039249956607818604,
+            0.007702067960053682,
+            0.03531566634774208,
+            -0.0009564659558236599,
+            -0.03444826602935791,
+            0.010145509615540504,
+            -0.011942271143198013,
+            -0.009975126944482327,
+            -0.022862248122692108,
+            0.013862947933375835,
+            0.012430185452103615,
+            0.007415515370666981,
+            0.012252057902514935,
+            -0.006997303571552038,
+            0.01026167906820774,
+            -0.005216030869632959,
+            0.008356492035090923,
+            -0.0010823167394846678,
+            -0.037422213703393936,
+            0.012344993650913239,
+            0.02216522768139839,
+            -0.002635121811181307,
+            0.0016089539276435971,
+            -0.006145390681922436,
+            0.022103270515799522,
+            -0.013971373438835144,
+            -0.00650551775470376,
+            -0.00046443779137916863,
+            0.00797700323164463,
+            -0.01795213110744953,
+            0.004778457339853048,
+            0.0755879208445549,
+            -0.007775642443448305,
+            0.006122156511992216,
+            0.009270362555980682,
+            -0.015148562379181385,
+            -0.01637221872806549,
+            -0.004561606794595718,
+            -0.020260039716959,
+            0.008844406343996525,
+            -0.0018819533288478851,
+            -0.0110981035977602,
+            0.012468908913433552,
+            0.0053050946444272995,
+            0.002031038049608469,
+            -0.008379725739359856,
+            0.010811550542712212,
+            0.009525936096906662,
+            0.010432061739265919,
+            0.02360573597252369,
+            0.011477591469883919,
+            -0.00858883187174797,
+            -0.011795123107731342,
+            0.01129171997308731,
+            0.0028577810153365135,
+            0.016697494313120842,
+            0.007907302118837833,
+            0.0010648912284523249,
+            0.019501063972711563,
+            -0.002309845993295312,
+            0.02180897258222103,
+            -2.2689444449497387e-05,
+            -0.009301341138780117,
+            -0.004236331209540367,
+            -0.015349923633038998,
+            0.024271776899695396,
+            -0.0020155487582087517,
+            0.0015450604259967804,
+            -0.020213572308421135,
+            0.0013979118084535003,
+            0.013243375346064568,
+            0.011957760900259018,
+            -0.0020407189149409533,
+            -0.0051347119733691216,
+            0.0007154133054427803,
+            -0.016356728971004486,
+            0.032806396484375,
+            0.007578153628855944,
+            0.0007681738352403045,
+            0.0008838597568683326,
+            0.0009288755827583373,
+            -0.0032972905319184065,
+            0.0024608669336885214,
+            -0.004766840487718582,
+            0.009549170732498169,
+            -0.010943209752440453,
+            -0.010796061716973782,
+            0.010075807571411133,
+            -0.02402394637465477,
+            0.003566417610272765,
+            -0.00558777479454875,
+            -0.008062195032835007,
+            0.0076207490637898445,
+            0.00015537731815129519,
+            -0.007392281666398048,
+            -0.0030939930584281683,
+            0.0023408245760947466,
+            0.013429246842861176,
+            0.004406713880598545,
+            0.010850274004042149,
+            0.012956822291016579,
+            0.005359307397156954,
+            -0.006629432085901499,
+            0.015512561425566673,
+            -0.006946963258087635,
+            -0.004228586331009865,
+            -0.0012468908680602908,
+            -0.019996721297502518,
+            -0.0009419447160325944,
+            -0.004708755761384964,
+            0.0015092412941157818,
+            0.0001705036120256409,
+            -8.749050175538287e-05,
+            -0.0018490385264158249,
+            -0.011988739483058453,
+            -0.001831613015383482,
+            -0.011663463898003101,
+            -0.014498010277748108,
+            0.010253935120999813,
+            -0.0041201612912118435,
+            -0.022939693182706833,
+            -0.015349923633038998,
+            -0.020662762224674225,
+            0.02145271934568882,
+            -0.004387352149933577,
+            -0.005800753366202116,
+            0.006733985152095556,
+            -0.010215211659669876,
+            -0.015706177800893784,
+            -0.008069939911365509,
+            -0.0205853171646595,
+            0.010703125037252903,
+            0.003214035416021943,
+            0.0006791101768612862,
+            0.00882891658693552,
+            0.014397329650819302,
+            -0.01589204929769039,
+            0.0111755495890975,
+            0.012182355858385563,
+            0.011361422017216682,
+            -0.006854027509689331,
+            0.0008911203476600349,
+            -0.014103032648563385,
+            0.02596011385321617,
+            0.021623101085424423,
+            0.005169562995433807,
+            -0.04132552444934845,
+            -0.005862710531800985,
+            -0.013344055972993374,
+            -0.010524998418986797,
+            -0.012120398692786694,
+            0.011771888472139835,
+            -0.0038820127956569195,
+            0.0013117524795234203,
+            0.006288666743785143,
+            0.0015063370810821652,
+            0.004367990419268608,
+            -0.0022382077295333147,
+            -0.007709812838584185,
+            -0.03212486580014229,
+            -0.02817508764564991,
+            -0.00863530021160841,
+            0.020461400970816612,
+            0.002838419284671545,
+            -0.0009453330421820283,
+            -0.0036380558740347624,
+            0.0027048238553106785,
+            -0.005882072262465954,
+            -0.005018542055040598,
+            -0.014753584749996662,
+            -0.012739972211420536,
+            0.0101300198584795,
+            0.00802347157150507,
+            0.014195968396961689,
+            -0.011950016021728516,
+            0.0028190575540065765,
+            0.018788553774356842,
+            0.007690451107919216,
+            -0.023357905447483063,
+            0.01135367713868618,
+            -0.010912231169641018,
+            -0.009270362555980682,
+            -0.009433000348508358,
+            -0.009471723809838295,
+            0.0016980175860226154,
+            -0.008317768573760986,
+            0.003945906180888414,
+            -0.03615209087729454,
+            -0.011330443434417248,
+            -0.0013901671627536416,
+            -0.010346870869398117,
+            0.0009864765452221036,
+            0.02524760365486145,
+            0.006815304048359394,
+            0.01432762760668993,
+            -0.0006733016925863922,
+            0.017502939328551292,
+            -0.0018984107300639153,
+            0.018726596608757973,
+            0.012763205915689468,
+            -0.00797700323164463,
+            0.011717676185071468,
+            0.014738094992935658,
+            -0.0033863543067127466,
+            0.015620986931025982,
+            -0.010168743319809437,
+            0.005285732913762331,
+            -0.009556914679706097,
+            -0.003214035416021943,
+            0.002495717955753207,
+            0.030901208519935608,
+            -0.003618694143369794,
+            0.00858883187174797,
+            0.01098193321377039,
+            -0.015241498127579689,
+            0.0026951429899781942,
+            0.014962689951062202,
+            0.01141563430428505,
+            0.007760153152048588,
+            0.0024434414226561785,
+            -0.024225307628512383,
+            0.004193735308945179,
+            0.005037903785705566,
+            0.0031520782504230738,
+            -0.014366351068019867,
+            0.0021859314292669296,
+            0.015675200149416924,
+            0.008550108410418034,
+            -0.025325050577521324,
+            0.016418687999248505,
+            0.0009138703462667763,
+            -0.003909118939191103,
+            0.029119934886693954,
+            -0.011268485337495804,
+            -0.005723306443542242,
+            0.009463978931307793,
+            0.0009980935137718916,
+            0.0030029932968318462,
+            0.008937342092394829,
+            0.0228312686085701,
+            -0.003988501615822315,
+            -0.032341714948415756,
+            -0.022459525614976883,
+            0.015706177800893784,
+            -0.0047823297791182995,
+            -0.022939693182706833,
+            -0.026703599840402603,
+            -0.003113354789093137,
+            -0.02228914201259613,
+            -0.0022323993034660816,
+            -0.020058678463101387,
+            -0.014056564308702946,
+            0.001268188701942563,
+            0.029987338930368423,
+            0.016201836988329887,
+            0.008224832825362682,
+            -0.011051635257899761,
+            -0.029956359416246414,
+            -0.024271776899695396,
+            0.01810702309012413,
+            -0.007055388763546944,
+            -0.005541306920349598,
+            0.00591692328453064,
+            -0.009951893240213394,
+            -0.008480406366288662,
+            0.011802867986261845,
+            0.027137301862239838,
+            -0.002747419523075223,
+            -0.029987338930368423,
+            -0.02688947319984436,
+            -0.014335372485220432,
+            -0.01840132102370262,
+            -0.017704300582408905,
+            -0.008627555333077908,
+            0.0045035220682621,
+            0.00030300996149890125,
+            0.016186347231268883,
+            0.014962689951062202,
+            -0.000480653194244951,
+            0.0028597170021384954,
+            -2.7711375878425315e-05,
+            -0.001763847190886736,
+            0.006888878531754017,
+            -0.004933350719511509,
+            0.026997897773981094,
+            0.014614180661737919,
+            -0.006594581063836813,
+            0.018711106851696968,
+            0.028314489871263504,
+            0.020430423319339752,
+            0.003157886676490307,
+            -0.010695381090044975,
+            -0.014606435783207417,
+            0.009727297350764275,
+            -0.013421501964330673,
+            0.023264970630407333,
+            0.022552460432052612,
+            -0.01480779703706503,
+            -0.014885243959724903,
+            -0.017905661836266518,
+            0.015210519544780254,
+            -0.007562664337456226,
+            -0.008224832825362682,
+            0.012368228286504745,
+            0.022335609421133995,
+            -0.007024410180747509,
+            0.002371803391724825,
+            -0.0010232636705040932,
+            -0.021731525659561157,
+            -0.006122156511992216,
+            -0.004596457816660404,
+            -0.008666278794407845,
+            0.008681767620146275,
+            0.008131897076964378,
+            -0.001928421319462359,
+            -0.006319645792245865,
+            0.005471604876220226,
+            -0.013367289677262306,
+            0.01301103550940752,
+            -0.00789568480104208,
+            0.03379771113395691,
+            -0.009138703346252441,
+            -0.0006733016925863922,
+            -0.022598929703235626,
+            -0.011446612887084484,
+            0.00845717266201973,
+            0.0017967619933187962,
+            0.01682141050696373,
+            -0.0011626676423475146,
+            -0.005688455421477556,
+            0.009913169778883457,
+            -0.0010687635513022542,
+            -0.001450188341550529,
+            -0.006892750971019268,
+            -0.010401083156466484,
+            0.0002884887217078358,
+            0.013258864171802998,
+            -0.006025348324328661,
+            0.0018054747488349676,
+            0.006095050368458033,
+            -0.0175339188426733,
+            0.029027000069618225,
+            0.023884542286396027,
+            0.017255110666155815,
+            0.014660648070275784,
+            0.0006394187803380191,
+            0.011647974140942097,
+            -0.003572226269170642,
+            0.0038820127956569195,
+            -0.020337486639618874,
+            0.021080974489450455,
+            -0.010811550542712212,
+            0.01846327818930149,
+            0.011655719019472599,
+            -0.015233753249049187,
+            -0.005022414494305849,
+            0.021406250074505806,
+            0.009967382065951824,
+            0.00889087375253439,
+            0.0017541663255542517,
+            -0.02470547705888748,
+            0.017766257748007774,
+            0.010099041275680065,
+            -0.020399443805217743,
+            0.019547531381249428,
+            -0.009471723809838295,
+            0.01233724970370531,
+            -0.0005300253978930414,
+            0.004418330732733011,
+            0.0058743273839354515,
+            0.021499186754226685,
+            0.011570527218282223,
+            -0.03884723410010338,
+            -0.013615119270980358,
+            -0.0017357728211209178,
+            0.006857899948954582,
+            -0.0036303112283349037,
+            -0.015233753249049187,
+            -0.041635312139987946,
+            0.01604694314301014,
+            0.009998360648751259,
+            -0.01643417589366436,
+            0.004987563472241163,
+            -0.002427952131256461,
+            -0.0042053526267409325,
+            -0.008170620538294315,
+            0.003186929039657116,
+            -0.004170501604676247,
+            0.02324948087334633,
+            0.0018984107300639153,
+            -0.006946963258087635,
+            0.013258864171802998,
+            0.01344473659992218,
+            -0.021948376670479774,
+            0.0009874446550384164,
+            0.0021375271026045084,
+            0.000636998622212559,
+            0.023683181032538414,
+            -0.014707116410136223,
+            -0.0017657832941040397,
+            -0.01685238815844059,
+            -0.0013069120468571782,
+            -0.021592123433947563,
+            -0.001188805908896029,
+            -0.015187285840511322,
+            -0.008511385880410671,
+            0.025030752643942833,
+            -0.011005166918039322,
+            0.010997422970831394,
+            -0.006687517277896404,
+            0.006943091284483671,
+            -0.006904367823153734,
+            -0.01936165988445282,
+            -0.0008267428493127227,
+            0.009975126944482327,
+            -0.000512115890160203,
+            0.002941036131232977,
+            8.482827252009884e-05,
+            0.002464739140123129,
+            0.008867640048265457,
+            0.006234454456716776,
+            -0.0075278133153915405,
+            0.02489134855568409,
+            -0.012073930352926254,
+            0.0016399325104430318,
+            0.011911292560398579,
+            0.0004743606550619006,
+            -0.010424317792057991,
+            0.025588368996977806,
+            -0.02139076218008995,
+            -0.00800798274576664,
+            -0.00014799568452872336,
+            -0.0016544538084417582,
+            -0.006513262167572975,
+            0.006718495860695839,
+            0.02946070022881031,
+            0.0062654330395162106,
+            0.010447551496326923,
+            -0.0281441081315279,
+            -0.011493081226944923,
+            0.012538610957562923,
+            -0.022273652255535126,
+            -0.0007739823195151985,
+            -0.0004322490422055125,
+            -0.00619960343465209,
+            0.026378324255347252,
+            -0.02022906206548214,
+            -0.01093546487390995,
+            -0.018664639443159103,
+            0.005405775271356106,
+            -0.011121337302029133,
+            0.008294534869492054,
+            -0.0009777637897059321,
+            0.006540368776768446,
+            0.016805920749902725,
+            -0.019563021138310432,
+            0.016031453385949135,
+            0.005119222681969404,
+            -0.016217326745390892,
+            0.010222956538200378,
+            -0.017580386251211166,
+            -0.0008199662552215159,
+            0.020275529474020004,
+            -0.00400399137288332,
+            0.008751469664275646,
+            -0.003363120136782527,
+            0.005208286456763744,
+            -0.006513262167572975,
+            0.0001492057926952839,
+            -0.007249005138874054,
+            0.014428308233618736,
+            -0.016805920749902725,
+            -0.017828216776251793,
+            -0.008821171708405018,
+            0.0016709112096577883,
+            0.021297825500369072,
+            0.011136826127767563,
+            -0.010044828988611698,
+            -0.011059380136430264,
+            -0.011694442480802536,
+            0.041604332625865936,
+            0.005762029904872179,
+            0.027230238541960716,
+            -0.02261441759765148,
+            0.00545998802408576,
+            0.025170156732201576,
+            -0.014242436736822128,
+            0.0012788375606760383,
+            -0.0022788674104958773,
+            0.0053050946444272995,
+            0.011562783271074295,
+            -0.01819995976984501,
+            0.0049139889888465405,
+            -0.00019216447253711522,
+            -0.00782985519617796,
+            -0.000656844291370362,
+            0.003442503046244383,
+            0.00020196630794089288,
+            0.016325751319527626,
+            -0.004027225077152252,
+            -0.016511622816324234,
+            -0.004429947584867477,
+            -0.0003216455806978047,
+            -0.027896279469132423,
+            0.013514438644051552,
+            0.016589069738984108,
+            0.01566745527088642,
+            0.027989214286208153,
+            -0.014513500034809113,
+            0.016031453385949135,
+            0.009533680975437164,
+            0.013901671394705772,
+            -0.010610189288854599,
+            0.008952830918133259,
+            -0.01595400646328926,
+            0.007446494419127703,
+            -0.03454120084643364,
+            -0.007887939922511578,
+            -0.00802347157150507,
+            0.01984182931482792,
+            0.028763681650161743,
+            0.002923610620200634,
+            0.006493900436908007,
+            0.000196883876924403,
+            0.0024202074855566025,
+            -0.01714668609201908,
+            -0.020182594656944275,
+            -0.0003337466041557491,
+            0.0029332914855331182,
+            0.0006950835813768208,
+            -0.004495777189731598,
+            0.00472037261351943,
+            -0.012391461990773678,
+            0.0026835259050130844,
+            -0.013134949840605259,
+            0.005971135571599007,
+            -0.0059982421807944775,
+            0.009603383019566536,
+            0.0019429425010457635,
+            0.020864123478531837,
+            0.02300165221095085,
+            0.004100799560546875,
+            0.0046971384435892105,
+            -0.024333734065294266,
+            0.00920066051185131,
+            -0.013847459107637405,
+            0.012329504825174809,
+            0.019408127292990685,
+            0.008248066529631615,
+            0.006571347359567881,
+            0.01233724970370531,
+            0.012918098829686642,
+            -0.0031772484071552753,
+            0.024287264794111252,
+            0.005014669615775347,
+            -0.0009070937521755695,
+            0.002052335999906063,
+            -0.014954946003854275,
+            -0.0070979841984808445,
+            -0.0015286030247807503,
+            0.0018848575418815017,
+            -0.004731989465653896,
+            0.01906736195087433,
+            0.015256987884640694,
+            -0.01050176378339529,
+            0.017874684184789658,
+            -0.03937387093901634,
+            0.017379024997353554,
+            -0.0027377386577427387,
+            -0.0060911779291927814,
+            -0.00545998802408576,
+            0.00665653869509697,
+            0.004515138920396566,
+            0.003669034456834197,
+            0.03773200139403343,
+            -0.010408828034996986,
+            0.003963331691920757,
+            -0.005847221240401268,
+            0.007121218368411064,
+            -0.020864123478531837,
+            0.008302279748022556,
+            0.005336073227226734,
+            0.02160761132836342,
+            -0.0056497324258089066,
+            -0.0014482521219179034,
+            0.02328045852482319,
+            0.008859895169734955,
+            0.03469609469175339,
+            -0.01566745527088642,
+            -0.004418330732733011,
+            0.01048627495765686,
+            0.0207402091473341,
+            -0.00580462533980608,
+            -0.008976065553724766,
+            0.007903429679572582,
+            0.0035586729645729065,
+            0.021824462339282036,
+            -0.00515407370403409,
+            -0.0006650729919783771,
+            0.0054367538541555405,
+            -0.0060911779291927814,
+            0.009735042229294777,
+            0.011252996511757374,
+            -0.009076746180653572,
+            0.007539430167526007,
+            0.021530164405703545,
+            -0.009975126944482327,
+            -0.008658533915877342,
+            -0.009649851359426975,
+            0.0010658593382686377,
+            -0.03068435750901699,
+            0.001649613375775516,
+            0.005359307397156954,
+            0.015187285840511322,
+            -0.0004603234410751611,
+            -0.009014789015054703,
+            0.021019017323851585,
+            0.017255110666155815,
+            -0.005936284549534321,
+            -0.0028132491279393435,
+            0.0023795480374246836,
+            -0.0017154429806396365,
+            -0.006160879973322153,
+            -0.0006248975405469537,
+            -0.002656419761478901,
+            0.014730350114405155,
+            -0.0015469965292140841,
+            -0.006617815233767033,
+            -0.00756653631106019,
+            0.0033302053343504667,
+            0.010834784246981144,
+            -0.005661349277943373,
+            0.013057502917945385,
+            0.016805920749902725,
+            -0.00031704717548564076,
+            0.019671445712447166,
+            0.01037010457366705,
+            -0.010447551496326923,
+            0.015938518568873405,
+            -0.007984748110175133,
+            0.006563602481037378,
+            0.0164031982421875,
+            -0.01277095079421997,
+            -0.01246116403490305,
+            0.007210282143205404,
+            0.00665653869509697,
+            0.028918573632836342,
+            0.014792308211326599,
+            -0.02109646424651146,
+            0.014908477663993835,
+            0.015768134966492653,
+            -0.013018779456615448,
+            0.003537375247105956,
+            -0.01784370467066765,
+            0.020260039716959,
+            0.019051872193813324,
+            -0.015535795129835606,
+            -0.017224133014678955,
+            -0.014947201125323772,
+            -0.005092116538435221,
+            0.006331262644380331,
+            0.0033263331279158592,
+            -0.00617636926472187,
+            -0.027989214286208153,
+            -0.0049139889888465405,
+            -0.018633661791682243,
+            0.013870692811906338,
+            -0.008674023672938347,
+            0.0007715621031820774,
+            -0.004511266481131315,
+            0.007984748110175133,
+            -0.0033882902935147285,
+            0.032465629279613495,
+            -0.0051811798475682735,
+            -0.02832997962832451,
+            0.004398969002068043,
+            0.016356728971004486,
+            0.012623801827430725,
+            -0.029243851080536842,
+            0.010463040322065353,
+            0.004805563949048519,
+            -0.012530866079032421,
+            0.005990497302263975,
+            -0.02258343994617462,
+            -0.007632365915924311,
+            -0.0009395244996994734,
+            -0.016743963584303856,
+            -0.027121812105178833,
+            -0.008433938957750797,
+            -0.006993431597948074,
+            -0.005401902832090855,
+            -0.009525936096906662,
+            -0.008720491081476212,
+            -0.011477591469883919,
+            0.02303262986242771,
+            -0.007899557240307331,
+            -0.001866463921032846,
+            -0.0014637414133176208,
+            0.00930908601731062,
+            -0.023776117712259293,
+            -0.015605497173964977,
+            0.006389347370713949,
+            -0.002594462363049388,
+            0.022815778851509094,
+            -0.012747716158628464,
+            0.005940156988799572,
+            -0.009316830895841122,
+            0.009851212613284588,
+            0.015241498127579689,
+            0.004956584889441729,
+            0.017440982162952423,
+            -0.009889936074614525,
+            0.0021026763133704662,
+            -0.008627555333077908,
+            -0.00018986526993103325,
+            -1.027075541060185e-05,
+            0.014428308233618736,
+            0.007624621503055096,
+            0.014141756109893322,
+            0.017487451434135437,
+            -0.013553161174058914,
+            0.005003052763640881,
+            -0.0003274540649726987,
+            -0.017255110666155815,
+            0.0027725896798074245,
+            -0.007984748110175133,
+            -0.0008896682411432266,
+            0.001758038648404181,
+            6.395397940650582e-05,
+            -0.02112744189798832,
+            -0.0009124181815423071,
+            0.0018809852190315723,
+            0.0010222955606877804,
+            -0.025201136246323586,
+            0.00889087375253439,
+            -0.00889087375253439,
+            -0.0014918158994987607,
+            -0.019113829359412193,
+            0.019795360043644905,
+            0.026300877332687378,
+            -0.004619691986590624,
+            0.00036206303047947586,
+            -0.013537672348320484,
+            -0.02049238048493862,
+            0.0001489637652412057,
+            -0.011314953677356243,
+            -0.011345932260155678,
+            -0.0022169100120663643,
+            -0.003214035416021943,
+            0.008805682882666588,
+            0.0004189379105810076,
+            -0.009433000348508358,
+            -0.004956584889441729,
+            -0.011764144524931908,
+            -0.017549408599734306,
+            -0.010401083156466484,
+            0.01933068037033081,
+            -0.0018016024259850383,
+            0.0020794421434402466,
+            -0.0032508226577192545,
+            0.0005271211848594248,
+            0.011748654767870903,
+            0.013475715182721615,
+            0.013344055972993374,
+            -0.023915521800518036,
+            -0.017890173941850662,
+            -0.013320821337401867,
+            0.0029487807769328356,
+            -0.007187047973275185,
+            0.004844287410378456,
+            0.026362836360931396,
+            -0.0009695350890979171,
+            -0.01091997604817152,
+            0.002327271504327655,
+            0.019051872193813324,
+            -0.031272951513528824,
+            -0.013390523381531239,
+            -0.025077221915125847,
+            0.02225816436111927,
+            -0.013351799920201302,
+            0.011516314931213856,
+            0.010191977955400944,
+            -0.025944624096155167,
+            0.006768836174160242,
+            -0.007686578668653965,
+            0.026037558913230896,
+            0.01634124107658863,
+            0.0005934348446317017,
+            -0.00944074522703886,
+            0.013622863218188286,
+            -0.0008722427301108837,
+            -0.013576395809650421,
+            0.002201420720666647,
+            0.014645159244537354,
+            -0.014691627584397793,
+            -0.009510447271168232,
+            -0.004507394507527351,
+            0.004313777666538954,
+            0.012763205915689468,
+            -0.0030339721124619246,
+            -0.004031097516417503,
+            -0.011624740436673164,
+            0.013344055972993374,
+            -0.003930416889488697,
+            0.012383717112243176,
+            0.007380664348602295,
+            -0.014056564308702946,
+            -0.019532041624188423,
+            0.033333033323287964,
+            0.00400399137288332,
+            0.00338054564781487,
+            -0.010997422970831394,
+            0.02740062028169632,
+            -0.009897680021822453,
+            0.010571465827524662,
+            -0.006408709101378918,
+            0.0031985461246222258,
+            0.012321759946644306,
+            -0.012104909867048264,
+            -0.01981084980070591,
+            0.021468207240104675,
+            -0.00364192808046937,
+            -0.0010193913476541638,
+            0.008310024626553059,
+            -0.020182594656944275,
+            -0.00663330452516675,
+            0.002180122770369053,
+            0.009541425853967667,
+            -0.011818356812000275,
+            0.006369986105710268,
+            0.017905661836266518,
+            0.003912991378456354,
+            -0.0007672057254239917,
+            0.019996721297502518,
+            -0.01672847382724285,
+            0.011717676185071468,
+            -0.010850274004042149,
+            0.0016825281782075763,
+            0.0006065039779059589,
+            -0.0048597767017781734,
+            0.0058743273839354515,
+            -0.014536733739078045,
+            0.011988739483058453,
+            -6.244135147426277e-05,
+            -0.013979118317365646,
+            0.007601387333124876,
+            -0.013359544798731804,
+            -0.016836898401379585,
+            0.041542377322912216,
+            -0.005301222205162048,
+            -0.0038665232714265585,
+            -0.0056962003000080585,
+            0.01350669376552105,
+            0.02823704481124878,
+            -0.005336073227226734,
+            -0.01153180468827486,
+            -0.010780571959912777,
+            -0.016573579981923103,
+            0.015907539054751396,
+            -0.01810702309012413,
+            0.002495717955753207,
+            -0.010532742366194725,
+            -0.007903429679572582,
+            0.004302160814404488,
+            -0.003516077296808362,
+            0.014722606167197227,
+            0.00950270239263773,
+            0.023357905447483063,
+            0.00797700323164463,
+            -0.00027348342700861394,
+            0.010393339209258556,
+            0.011663463898003101,
+            0.003243078012019396,
+            -0.010509508661925793,
+            -0.00011949774489039555,
+            -0.02894955314695835,
+            -0.008449427783489227,
+            -0.004863648675382137,
+            -0.001485039247199893,
+            -0.00428667152300477,
+            -0.01239920686930418,
+            0.006617815233767033,
+            -0.0022052929271012545,
+            0.013909416273236275,
+            -0.009254872798919678,
+            -0.017766257748007774,
+            0.03389064967632294,
+            0.004495777189731598,
+            0.006466794293373823,
+            -0.008286789990961552,
+            -0.03705047070980072,
+            -0.0032566310837864876,
+            -0.008116407319903374,
+            -0.013529927469789982,
+            0.004538373090326786,
+            0.01283290795981884,
+            -0.014831030741333961,
+            -0.028097640722990036,
+            0.0019390701781958342,
+            -0.006683644838631153,
+            0.02978597767651081,
+            -0.007798876613378525,
+            0.00419760774821043,
+            0.0037251831963658333,
+            0.0011558911064639688,
+            0.01239920686930418,
+            -0.00845717266201973,
+            -0.0022536972537636757,
+            0.016387708485126495,
+            -0.010858018882572651,
+            0.011051635257899761,
+            -0.010966443456709385,
+            -0.02554190158843994,
+            -0.01456771232187748,
+            0.021344292908906937,
+            -0.0004823473282158375,
+            -0.00024903935263864696,
+            0.014823286794126034,
+            0.02354377694427967,
+            0.011260741390287876,
+            0.00969631876796484,
+            0.006408709101378918,
+            -0.001035848748870194,
+            0.005975008010864258,
+            0.006095050368458033,
+            -0.0024163350462913513,
+            -0.013398268260061741,
+            0.017564896494150162,
+            0.009773765690624714,
+            -0.01810702309012413,
+            0.0008514289511367679,
+            0.0036612898111343384,
+            0.0168833676725626,
+            0.006935346405953169,
+            0.009316830895841122,
+            -0.0143121387809515,
+            -0.023760627955198288,
+            -0.00014932679187040776,
+            0.002873270306736231,
+            0.015721667557954788,
+            0.010579210706055164,
+            -0.014319883659482002,
+            0.0005963391158729792,
+            0.008712747134268284,
+            0.020213572308421135,
+            -0.024209819734096527,
+            0.019702425226569176,
+            -0.004673904739320278,
+            -0.0021084847394376993,
+            0.007946024648845196,
+            0.011632485315203667,
+            -0.011872569099068642,
+            -0.00650551775470376,
+            0.006249943748116493,
+            0.003157886676490307,
+            0.03172214329242706,
+            -0.0017260919557884336,
+            0.009665340185165405,
+            -0.010997422970831394,
+            -0.0002405686245765537,
+            0.0006728176376782358,
+            -0.0006820144481025636,
+            -0.002884887158870697,
+            -0.011020656675100327,
+            0.0018974426202476025,
+            -0.020306508988142014,
+            0.0024473138619214296,
+            0.00463130883872509,
+            -0.005684582982212305,
+            -0.0032972905319184065,
+            -0.019222255796194077,
+            -0.0022672503255307674,
+            0.003293418325483799,
+            0.014335372485220432,
+            -0.031691163778305054,
+            0.004972074180841446,
+            0.010672146454453468,
+            -0.002309845993295312,
+            -0.010416572913527489,
+            0.011423379182815552,
+            -0.0026060794480144978,
+            0.011988739483058453,
+            -0.0038142469711601734,
+            -0.0004905760288238525,
+            0.025681305676698685,
+            -0.005308967083692551,
+            0.014707116410136223,
+            -0.01026167906820774,
+            -0.002993312431499362,
+            -0.028252532705664635,
+            0.005235392600297928,
+            -0.007403898518532515,
+            -0.0012081675231456757,
+            -0.004402841441333294,
+            -0.01130720879882574,
+            -0.0026990154292434454,
+            0.002122037811204791,
+            0.008000237867236137,
+            0.012910354882478714,
+            -0.0007725302129983902,
+            -0.010075807571411133,
+            0.004832670092582703,
+            0.0075355577282607555,
+            -0.015520306304097176,
+            -0.0070437719114124775,
+            -0.020306508988142014,
+            0.029662061482667923,
+            -0.013731288723647594,
+            -0.0011123273288831115,
+            -0.020864123478531837,
+            0.010160998441278934,
+            -0.0063119009137153625,
+            -0.002782270545139909,
+            0.004708755761384964,
+            -0.015109838917851448,
+            0.01344473659992218,
+            0.008751469664275646,
+            0.01450575515627861,
+            -0.02151467651128769,
+            -0.01762685552239418,
+            -0.0044725434854626656,
+            -0.002735802438110113,
+            -0.010648912750184536,
+            -0.0005334137240424752,
+            0.002013612538576126,
+            0.021251358091831207,
+            -0.00032503384863957763,
+            0.0042053526267409325,
+            -0.003214035416021943,
+            0.017224133014678955,
+            -0.010137764737010002,
+            -0.013057502917945385,
+            0.0024260159116238356,
+            0.005994369741529226,
+            0.016186347231268883,
+            0.03158273920416832,
+            -0.020693741738796234,
+            -0.00272999401204288,
+            0.0032740565948188305,
+            0.010656657628715038,
+            -0.004085310269147158,
+            6.304640555754304e-05,
+            -0.027803342789411545,
+            -0.04464024305343628,
+            -0.002422143705189228,
+            0.013057502917945385,
+            -0.015636475756764412,
+            0.012670270167291164,
+            0.008743725717067719,
+            0.010021595284342766,
+            -0.005196669604629278,
+            0.02620794251561165,
+            -0.004793947096914053,
+            0.014304393902420998,
+            0.011733165942132473,
+            0.011911292560398579,
+            0.0009874446550384164,
+            0.0007459079497493804,
+            -0.018354853615164757,
+            -0.008255811408162117,
+            -0.0015954007394611835,
+            0.01765783317387104,
+            -0.027478067204356194,
+            0.008526874706149101,
+            -0.011516314931213856,
+            0.013839714229106903,
+            0.0003666614356916398,
+            0.005661349277943373,
+            -0.01771979033946991,
+            0.012445674277842045,
+            -0.0009337160154245794,
+            -0.025325050577521324,
+            0.004805563949048519,
+            0.011276230216026306,
+            -0.010362359695136547,
+            0.006877261213958263,
+            0.019950253888964653,
+            0.0023079097736626863,
+            0.008495896123349667,
+            -0.014126266352832317,
+            0.010703125037252903,
+            0.009649851359426975,
+            -0.021080974489450455,
+            -0.01233724970370531,
+            -0.007059261202812195,
+            -0.006335135083645582,
+            -0.0035470561124384403,
+            -0.00013105422840453684,
+            0.010176488198339939,
+            -0.016000475734472275,
+            0.010099041275680065,
+            -0.006559730041772127,
+            0.006400964688509703,
+            0.00944074522703886,
+            0.001115231541916728,
+            -0.013576395809650421,
+            0.012716737575829029,
+            -0.0218709297478199,
+            0.008426194079220295,
+            -0.04287445917725563,
+            -0.005847221240401268,
+            -0.015078860335052013,
+            0.005204414017498493,
+            -0.003798757679760456,
+            -0.004453181754797697,
+            -0.006203475873917341,
+            -0.005061137955635786,
+            0.00920066051185131,
+            0.021747015416622162,
+            0.012135888449847698,
+            -0.016620049253106117,
+            -0.007551047019660473,
+            0.005425137002021074,
+            0.0004854935978073627,
+            -0.013336311094462872,
+            0.009316830895841122,
+            -0.01576039008796215,
+            0.0009259713697247207,
+            0.006339007057249546,
+            -0.01072635967284441,
+            0.013452480547130108,
+            0.004731989465653896,
+            -0.003401843598112464,
+            0.013390523381531239,
+            -0.0050417762249708176,
+            -0.0057891360484063625,
+            -0.007272239308804274,
+            -0.013739033602178097,
+            -0.003818119177594781,
+            -0.003072695340961218,
+            0.012747716158628464,
+            0.001529571134597063,
+            -0.0003243077953811735,
+            -0.00782985519617796,
+            0.009154192171990871,
+            0.002220782218500972,
+            -0.0013146566925570369,
+            0.005773646757006645,
+            -0.02300165221095085,
+            0.0001287550403503701,
+            0.004755223635584116,
+            -0.0009487213101238012,
+            -0.006246071308851242,
+            -0.012151377275586128,
+            -0.011903548613190651,
+            -0.038165703415870667,
+            0.009363298304378986,
+            0.0019197085639461875,
+            -0.0076207490637898445,
+            0.0064822835847735405,
+            0.017255110666155815,
+            0.01456771232187748,
+            0.016743963584303856,
+            -0.020693741738796234,
+            -0.02303262986242771,
+            0.005556796211749315,
+            -0.009402021765708923,
+            0.02261441759765148,
+            0.004046586807817221,
+            -0.005847221240401268,
+            0.0013243375578895211,
+            -0.013258864171802998,
+            0.008929597213864326,
+            0.005723306443542242,
+            -0.001592496526427567,
+            0.005425137002021074,
+            -0.013692565262317657,
+            -0.010904486291110516,
+            0.01048627495765686,
+            0.032496608793735504,
+            -0.0004261985304765403,
+            0.0024105266202241182,
+            0.023295948281884193,
+            0.015466093085706234,
+            0.016356728971004486,
+            -0.021111954003572464,
+            0.011167804710566998,
+            0.010060317814350128,
+            -0.0018645278178155422,
+            -0.014892988838255405,
+            0.00513083953410387,
+            -0.012716737575829029,
+            -0.010664401575922966,
+            0.014993669465184212,
+            0.007853088900446892,
+            0.017735280096530914,
+            -0.02369867078959942,
+            0.00789568480104208,
+            0.00042765066609717906,
+            0.00013710475468542427,
+            0.011524059809744358,
+            0.012956822291016579,
+            0.03370477631688118,
+            0.027152791619300842,
+            0.020662762224674225,
+            -0.009285852313041687,
+            0.014366351068019867,
+            -0.009192915633320808,
+            -0.011090358719229698,
+            -0.019470084458589554,
+            -0.015249243006110191,
+            -0.011136826127767563,
+            0.017998598515987396,
+            0.008790193125605583,
+            -0.009355553425848484,
+            -0.005947901867330074,
+            0.01586107164621353,
+            -0.003909118939191103,
+            -0.004635181277990341,
+            0.01646515540778637,
+            -0.026269899681210518,
+            0.016836898401379585,
+            -0.000745423894841224,
+            -0.014668392948806286,
+            -0.014691627584397793,
+            -0.004182118456810713,
+            -0.012670270167291164,
+            0.0003964299685321748,
+            -0.020817656069993973,
+            0.013181418180465698,
+            0.024597052484750748,
+            -0.0030920570716261864,
+            -0.004100799560546875,
+            -0.011299464851617813,
+            -0.0005111477803438902,
+            0.018385831266641617,
+            0.0034773540683090687,
+            0.009192915633320808,
+            -0.0023872926831245422,
+            0.013924905098974705,
+            0.00524313747882843,
+            0.0027958236169070005,
+            -0.0007565568666905165,
+            -0.016263794153928757,
+            0.012197845615446568,
+            0.0026835259050130844,
+            0.012135888449847698,
+            -0.022242674604058266,
+            0.007938280701637268,
+            -0.011764144524931908,
+            -0.0023698671720921993,
+            -0.02228914201259613,
+            0.0189744271337986,
+            -0.01610890030860901,
+            0.001046497724018991,
+            0.003841353114694357,
+            0.0006224773242138326,
+            0.009580149315297604,
+            0.01610890030860901,
+            -0.023388884961605072,
+            -0.0003460896550677717,
+            -0.014908477663993835,
+            -0.021917399019002914,
+            -0.01607792265713215,
+            0.004728117026388645,
+            -0.019501063972711563,
+            0.005444498732686043,
+            -0.0025557391345500946,
+            -0.015125328674912453,
+            0.018478767946362495,
+            0.014691627584397793,
+            -0.004379607271403074,
+            0.002412462839856744,
+            0.005425137002021074,
+            -0.0014405074762180448,
+            0.01971791312098503,
+            0.00013274837692733854,
+            -0.0010048701660707593,
+            -0.009425255469977856,
+            -0.0168833676725626,
+            -0.0065752193331718445,
+            -0.035005878657102585,
+            0.0260530486702919,
+            -0.004670032300055027,
+            -0.016511622816324234,
+            0.014319883659482002,
+            0.009115469641983509,
+            -0.014629669487476349,
+            -0.007059261202812195,
+            0.016294771805405617,
+            -0.024287264794111252,
+            0.02244403585791588,
+            -0.009611127898097038,
+            -0.008774704299867153,
+            0.00784921646118164,
+            0.05154848098754883,
+            0.009146448224782944,
+            0.003492843359708786,
+            0.011036145500838757,
+            0.005839476361870766,
+            -0.0013165927957743406,
+            -0.023683181032538414,
+            0.005568413063883781,
+            -0.007190920412540436,
+            -0.003601268632337451,
+            0.002536377403885126,
+            -0.004801691509783268,
+            0.031149037182331085,
+            -0.004379607271403074,
+            -0.005289605353027582,
+            0.004484160337597132,
+            -0.02196386642754078,
+            -0.0001332324172835797,
+            0.023203013464808464,
+            -2.3128106477088295e-05,
+            -0.023590246215462685,
+            -0.006571347359567881,
+            -0.015071115456521511,
+            -0.013770012184977531,
+            -0.018989915028214455,
+            0.005231520161032677,
+            -0.012755461037158966,
+            -0.0033940989524126053,
+            0.0016660707769915462,
+            -0.005940156988799572,
+            -0.002582845278084278,
+            0.009603383019566536,
+            0.05362405255436897,
+            0.014103032648563385,
+            -0.006335135083645582,
+            -0.005854965653270483,
+            0.009192915633320808,
+            -0.010524998418986797,
+            -0.019795360043644905,
+            -0.009820234030485153,
+            -0.029507169499993324,
+            -0.012747716158628464,
+            -0.0143121387809515,
+            -0.010191977955400944,
+            -0.010277168825268745,
+            0.00963436160236597,
+            -0.018122512847185135,
+            -0.007628493942320347,
+            -0.020337486639618874,
+            0.002941036131232977,
+            -0.011268485337495804,
+            -0.014714861288666725,
+            -0.008054450154304504,
+            -0.04807887226343155,
+            0.005796880926936865,
+            0.019593998789787292,
+            0.01263929158449173,
+            0.002515079453587532,
+            0.004383479710668325,
+            -0.01993476413190365,
+            -0.030792783945798874,
+            0.013839714229106903,
+            0.011283975094556808,
+            0.011733165942132473,
+            0.03283737599849701,
+            -0.011709931306540966,
+            -0.00813964195549488,
+            -0.004693266469985247,
+            -0.0017251238459721208,
+            0.012066186405718327,
+            0.00969631876796484,
+            -0.004917861428111792,
+            -0.012375972233712673,
+            0.016217326745390892,
+            -0.01981084980070591,
+            -0.023683181032538414,
+            -0.01672847382724285,
+            0.006157007534056902,
+            0.00813964195549488,
+            -0.008054450154304504,
+            0.002973950933665037,
+            0.0056962003000080585,
+            -0.007159941829741001,
+            -0.00022423222253564745,
+            0.004890755284577608,
+            -0.023048119619488716,
+            0.0008490087930113077,
+            -0.002257569460198283,
+            0.0061376458033919334,
+            0.021266845986247063,
+            -0.010168743319809437,
+            -0.0014075926737859845,
+            -0.0053902859799563885,
+            -0.0028984404634684324,
+            -0.006764963734894991,
+            0.02620794251561165,
+            0.009154192171990871,
+            -0.009882191196084023,
+            -0.01011453103274107,
+            -0.01129171997308731,
+            -0.0011278166202828288,
+            -0.017255110666155815,
+            -0.016186347231268883,
+            0.0015566773945465684,
+            -0.016712984070181847,
+            0.013971373438835144,
+            -0.00876695942133665,
+            0.00400399137288332,
+            -0.014110777527093887,
+            0.02019808255136013,
+            0.01929970271885395,
+            -0.009711808525025845,
+            0.023776117712259293,
+            -0.007709812838584185,
+            -0.038382552564144135,
+            -0.00800798274576664,
+            0.01843230053782463,
+            -0.006118284538388252,
+            0.010037084110081196,
+            -0.011314953677356243,
+            0.008209343999624252,
+            0.00015319912927225232,
+            0.0018122512847185135,
+            -0.0003112386621069163,
+            0.027555514127016068,
+            0.015535795129835606,
+            -0.005560668651014566,
+            -0.005494839046150446,
+            0.004255692940205336,
+            0.009541425853967667,
+            0.017580386251211166,
+            0.012151377275586128,
+            -0.021019017323851585,
+            -0.014257925562560558,
+            -0.016263794153928757,
+            0.011012911796569824,
+            0.015125328674912453,
+            -0.0040891822427511215,
+            0.01691434532403946,
+            0.014126266352832317,
+            -0.019098341464996338,
+            0.0019884423818439245,
+            -0.008193854242563248,
+            -0.00158184755127877,
+            -0.03029712475836277,
+            0.011043890379369259,
+            0.0074387495405972,
+            -0.0072218989953398705,
+            0.036616768687963486,
+            0.009858956560492516,
+            0.037577107548713684,
+            0.018680129200220108,
+            0.0008974128868430853,
+            0.009045767597854137,
+            -0.018370341509580612,
+            0.013065247796475887,
+            0.01500141341239214,
+            -0.012724482454359531,
+            -0.005734923295676708,
+            0.0013446672819554806,
+            0.0007551047019660473,
+            0.012631546705961227,
+            0.02004319056868553,
+            -0.01795213110744953,
+            0.009773765690624714,
+            -0.015032391995191574,
+            -0.005250881891697645,
+            0.006555858068168163,
+            -0.013932649977505207,
+            -0.008495896123349667,
+            -0.00963436160236597,
+            0.019408127292990685,
+            -0.012600568123161793,
+            0.016573579981923103,
+            -0.009595638141036034,
+            0.015520306304097176,
+            -0.029724018648266792,
+            -0.003669034456834197,
+            -0.006679772399365902,
+            -0.019593998789787292,
+            -0.012887120246887207,
+            0.009820234030485153,
+            0.010633422993123531,
+            -0.009301341138780117,
+            -0.011562783271074295,
+            0.020368466153740883,
+            -0.004185990896075964,
+            0.013529927469789982,
+            0.021189400926232338,
+            0.00737291993573308,
+            0.024132372811436653,
+            0.0005605200421996415,
+            0.014164989814162254,
+            0.005463860463351011,
+            0.010362359695136547,
+            -0.004677776712924242,
+            0.005161818582564592,
+            0.00932457484304905,
+            0.014203713275492191,
+            0.017766257748007774,
+            0.013196907006204128,
+            -0.00598275288939476,
+            0.007419387809932232,
+            0.02016710489988327,
+            0.0016254113288596272,
+            -0.030312614515423775,
+            -0.009719553403556347,
+            0.0001736498816171661,
+            0.00924712885171175,
+            0.009185170754790306,
+            3.817877222900279e-05,
+            -0.01093546487390995,
+            0.011144571006298065,
+            0.00969631876796484,
+            0.002286612056195736,
+            0.006280922330915928,
+            -0.01456771232187748,
+            -0.020957060158252716,
+            0.006106667220592499,
+            0.01798310875892639,
+            0.013917161151766777,
+            -0.014722606167197227,
+            0.022784801200032234,
+            0.006145390681922436,
+            -0.021406250074505806,
+            -0.01840132102370262,
+            -0.0009559819591231644,
+            0.0004542729293461889,
+            -0.002265314105898142,
+            0.03720536455512047,
+            -0.0011539548868313432,
+            -0.005955646280199289,
+            -0.0054754773154854774,
+            0.02775687538087368,
+            0.02049238048493862,
+            0.009177426807582378
+          ],
+          "embedding": [
+            0.0008215984936325879,
+            -0.019349418284688353,
+            -0.02322660633029795,
+            0.009747820845194852,
+            0.007353771220903641,
+            0.015252908783199839,
+            -0.012176175829980202,
+            0.051898051094966605,
+            -0.0068130353886146165,
+            0.008189212816890595,
+            0.008102287831931905,
+            0.01682801136011614,
+            0.02005557996049711,
+            -0.020779879580497183,
+            0.002332165268329781,
+            0.02337485065616377,
+            0.006074992895540845,
+            0.02576575146535689,
+            0.0007517286429767504,
+            -0.03763909942314134,
+            0.0011204729720314833,
+            0.00041212613589632806,
+            -0.06089176800314795,
+            0.012852035873395388,
+            -0.008901127092058543,
+            0.01702347055484216,
+            0.005803446395466281,
+            0.017889570275475296,
+            -0.05375047382413552,
+            0.021623778774252724,
+            0.014784317465883424,
+            -0.0018881880857885708,
+            0.0004735328554290247,
+            0.012393802850407656,
+            -0.019234442968055952,
+            -0.019566283611638122,
+            0.04407789373296535,
+            -0.0053565873209102995,
+            -0.026240866282480922,
+            0.048107902270426675,
+            -0.0054681314939387235,
+            -0.0009441397948041965,
+            -0.015205079626883548,
+            0.006885981841893229,
+            0.01403724256866746,
+            -0.026086548246792754,
+            0.0034778946051893067,
+            0.00044587533948775473,
+            0.032851328435891634,
+            -0.03882097333255334,
+            -0.011126344604254894,
+            -0.01648943336384904,
+            0.035810456555802166,
+            -0.019384589526165754,
+            -0.019430702768170285,
+            0.014297440272239852,
+            -0.007883453204663151,
+            0.050996900793086844,
+            0.06007149434153715,
+            0.03406832053483379,
+            0.025193439934671852,
+            0.00027785129663604897,
+            -0.008014391056168083,
+            -0.018806730908490114,
+            0.027193788903098558,
+            -0.006195006670328595,
+            -0.020466000344352826,
+            0.0068939884219589185,
+            0.017252276167086602,
+            -0.045073444648704136,
+            -0.049242269917908754,
+            0.023021140866008313,
+            -0.015676872472071106,
+            -0.024497517908898602,
+            0.005873633718973869,
+            -0.022828585938203683,
+            0.011748341004940436,
+            0.04624205727927451,
+            -0.015442165499674641,
+            0.018565551071491767,
+            0.015246505395507096,
+            0.004119269796224144,
+            0.004862735250819252,
+            -0.000145323321960046,
+            0.05155838952887077,
+            0.0027052423345323936,
+            -0.016218063670586664,
+            -0.07462470920944148,
+            0.043695974746295135,
+            -0.027608273720377708,
+            -0.03860912774695494,
+            0.002823110565150435,
+            0.026471469849508766,
+            -0.0026063577189074103,
+            -0.029230652014151735,
+            0.05684164470477843,
+            0.00022182868300572677,
+            0.015590845552956364,
+            -0.024939617549529242,
+            0.01035803286203917,
+            -0.010225298012384327,
+            0.019571847648759157,
+            0.01841613041441054,
+            0.0156981470538354,
+            0.021764297694793343,
+            -0.018831230007033416,
+            -0.014419529638789993,
+            -0.0029965475899652273,
+            0.028881185012095884,
+            0.09163526460163937,
+            0.022644302207811818,
+            -0.017933909936330133,
+            -0.04663928709016869,
+            0.004962951444105128,
+            -0.041333041174526026,
+            -0.027545628007617527,
+            -0.037142394579894954,
+            -0.013358143162788402,
+            0.02719686432396677,
+            -0.0034971800937504205,
+            0.04413635817371732,
+            0.04903857110328917,
+            0.03186067155630788,
+            0.007831051050269414,
+            -0.025257538623313574,
+            0.023517513773109414,
+            -0.0016527403209505964,
+            -0.0016101860707429347,
+            0.014448515185903624,
+            0.009905018312477684,
+            0.0024114582235163955,
+            0.00172043851384822,
+            -0.03818316221102173,
+            -0.02304537008110578,
+            0.01024969233570089,
+            -0.01901116356327339,
+            0.019791121460848986,
+            0.07985067340821365,
+            -0.051903191272236665,
+            -0.025924646976556942,
+            0.014766511532849252,
+            -0.002073086076894999,
+            0.030959269926800852,
+            0.006394417986617089,
+            -0.028177586693713442,
+            0.010725285336674526,
+            0.0009614052276924192,
+            0.07500253788978564,
+            -0.00036794182149960157,
+            0.004058458332826519,
+            -0.013218269166951234,
+            0.027613539268683245,
+            -0.012995170442480582,
+            0.023388707018488407,
+            0.026389423313105165,
+            -0.003068639956061266,
+            -0.02661967649632026,
+            0.026145274570635913,
+            -0.03300591322174331,
+            -0.004556072066733024,
+            0.020045722084112896,
+            0.0033437343914784865,
+            -0.010224220905483085,
+            -0.0025025498684079703,
+            0.03460103245395426,
+            0.04514362357278504,
+            0.03797031988058962,
+            -0.004586662475576956,
+            -0.06351723722596923,
+            0.06037983484369874,
+            0.019415753124363902,
+            0.028850618647994476,
+            -0.0034810237045849006,
+            0.02957938154808314,
+            0.02642652523231673,
+            0.013187653724133275,
+            -0.007856320804109569,
+            -0.007065882659482447,
+            -0.016672784625197085,
+            0.029020161154209707,
+            -0.013243058921970134,
+            -0.014900932834241732,
+            0.006001769901896033,
+            0.008648456186944271,
+            0.03484763413464497,
+            0.012279368984306564,
+            0.02379643508687133,
+            0.04933044959399941,
+            -0.021047723935647383,
+            -0.007264823701706426,
+            0.0007058774795716792,
+            -0.010512071832195085,
+            0.027489671668749157,
+            0.009731157570361245,
+            -0.03195919640758579,
+            -0.0033626031972691915,
+            -0.04282419484651207,
+            0.01873977910485268,
+            -0.014398285316695653,
+            0.012854631447494207,
+            -0.0017372959978092432,
+            -0.037331867825735776,
+            0.001088209292238142,
+            -0.012345684323522151,
+            0.037096063326414734,
+            0.005050219690024652,
+            0.0008906051515353004,
+            -0.003697709449630797,
+            0.006548479142737107,
+            -0.010727663712299397,
+            -0.007486226350610755,
+            -0.02704618481069485,
+            -0.019752552016312926,
+            -0.014632581296303486,
+            -0.0037501138119724717,
+            0.014588411795703245,
+            -0.007119137071719376,
+            0.016440258608998535,
+            -0.03686645352111013,
+            -0.023546402448713415,
+            0.005628703432152666,
+            0.024988145751166143,
+            0.011971199595808573,
+            -0.010868778329190568,
+            -0.03270046433624425,
+            -0.021950559285357267,
+            -0.011889914022657738,
+            -0.005123971523701009,
+            0.02619713302280345,
+            -0.021990810305904974,
+            0.01165257474485436,
+            -0.009117901804315863,
+            -0.007492082755249399,
+            0.010725902070291232,
+            0.01862811996181951,
+            -0.026694281868322685,
+            -0.01623776415010199,
+            -0.009403997538684658,
+            -0.02207329027097839,
+            0.025939540497196416,
+            0.014178803271471762,
+            -0.001141821318145744,
+            -0.006281187017652108,
+            -0.035993365198554964,
+            0.0026192776111449633,
+            -0.015254976771874396,
+            -0.0116872453009606,
+            -0.015780016491993135,
+            0.021421328283124968,
+            0.02889432250124136,
+            -0.01871009416296692,
+            -0.026540472640514536,
+            0.015703370682110186,
+            -0.0002849235245164438,
+            -0.02594930889995505,
+            0.04491828744809679,
+            0.044063522197410734,
+            -0.037441864491547405,
+            -0.007005339187980476,
+            -0.030381794393931814,
+            0.06901083845539985,
+            -0.017604082400603037,
+            -0.04060591381445593,
+            0.06044850935418443,
+            -0.01756490171206357,
+            0.025539717592705295,
+            0.03452018789549939,
+            0.0016217577522523927,
+            0.00018825054264059277,
+            -0.002563606092068712,
+            0.04770277039047993,
+            -0.0032010143255574255,
+            -0.03425328601475849,
+            -0.02105909767072799,
+            -0.00840881022563103,
+            -0.028699447677549002,
+            -0.015925160861589006,
+            -0.0055750742281166,
+            0.017513987082479483,
+            0.010399718005952185,
+            0.02267201252273823,
+            0.01258511878601025,
+            0.0015081978484714903,
+            0.01874981572091931,
+            0.011158889421057756,
+            -0.002607810686437211,
+            -0.003732023411827874,
+            -0.007787189972165309,
+            -0.05524012485122728,
+            -0.025110689689279583,
+            -0.03946383236703947,
+            0.004443572529325475,
+            -0.018745077598477586,
+            0.005926160888274727,
+            0.039430331582190524,
+            -0.0049500324516682,
+            0.004352795787899792,
+            0.015566139991780173,
+            0.013148156356408315,
+            -0.015279560337042242,
+            -0.03316846614871165,
+            0.0001592209771257332,
+            0.018362411161743347,
+            -0.012706582418607377,
+            -0.03424477956681259,
+            -0.001991022841226589,
+            0.0033467286047492175,
+            0.03357019284647071,
+            -0.024359110796680374,
+            0.00359117029951049,
+            0.005061818038334802,
+            0.011359582931760315,
+            -0.026793992746244076,
+            -0.018170136079817746,
+            -0.017421603453826397,
+            0.03996608926471779,
+            0.02587013058343799,
+            0.015763410145648683,
+            -0.04512536449983358,
+            -0.02293473890636517,
+            0.048575083783480195,
+            -0.009589860756342604,
+            -0.06486262416944208,
+            0.03447666009581266,
+            0.016105654870253654,
+            0.003526582643722421,
+            -0.01354754623151814,
+            0.05401575248587799,
+            -0.03563190295559527,
+            -0.017365166993914202,
+            -0.0046288709698227735,
+            -0.033657212157212645,
+            -0.007686252059880836,
+            -0.001385273642139782,
+            0.048229062789530236,
+            -0.029083855744728694,
+            -0.030465559177859865,
+            0.0024368661249980416,
+            -0.024144031154493057,
+            0.021693278579520123,
+            -0.016405088055634124,
+            -0.018439935250344958,
+            0.006948603359995389,
+            -0.033943037395199695,
+            -0.003007796423804453,
+            -0.016783195426536633,
+            -0.02270589491206155,
+            0.020874438753828814,
+            0.016724666903417947,
+            0.027033184180344852,
+            0.0019847647852695785,
+            0.017749524526371133,
+            0.0204661893602285,
+            -0.005604929968008821,
+            0.028462234633072944,
+            -0.020790459496727066,
+            -0.04183816355941877,
+            0.03946487892663695,
+            0.01192861758684541,
+            -0.0018024841275380242,
+            0.01077082591611702,
+            -0.01475239020295694,
+            -0.03952356245715099,
+            0.03683950018557526,
+            -0.056767672022551764,
+            -0.007507600090017693,
+            0.009530544352246083,
+            -0.00788856051420074,
+            -0.011204694027562521,
+            -0.010587701466318598,
+            -0.028604940377315504,
+            0.023749537635431462,
+            -0.013672419168235691,
+            0.01953449229707597,
+            -0.011449939491554354,
+            -0.012238180529977605,
+            0.05795536247491174,
+            -0.021518581774137322,
+            -0.040672955795925836,
+            0.026066659238628543,
+            -0.014061106927896723,
+            0.03328357230532424,
+            0.03957107406164427,
+            0.030670947051333917,
+            -0.004278577327978564,
+            0.025885946540364826,
+            0.0319106101668716,
+            -0.003462160036198642,
+            -0.04686868330851868,
+            0.00555303933159587,
+            0.02156143971867743,
+            -0.03739529489075782,
+            -0.01025626126223168,
+            -0.02141970191890338,
+            0.022134910980548407,
+            0.017807484142010496,
+            -0.007232643256393455,
+            0.011318041192005274,
+            -0.010489012862455913,
+            -0.013003217163997554,
+            -0.00847279880392288,
+            0.02357274819910594,
+            -0.0016485066139744577,
+            -0.005873113634992993,
+            0.006424249429163587,
+            0.004008703814301658,
+            0.0054639858216457724,
+            0.010198079500408441,
+            0.017694720520798957,
+            -0.017087021465208255,
+            -0.0025558869389290554,
+            -0.016503579212015177,
+            -0.0420082404922411,
+            -0.003563173424359332,
+            0.022924161700566468,
+            0.012345933179990547,
+            -0.00729908668365632,
+            -0.004234683369642856,
+            -0.0022962545569187934,
+            -0.020666205489143392,
+            -0.014796149043465308,
+            0.024562653223873056,
+            0.030217494971086438,
+            0.007810961433007407,
+            0.007781263640449456,
+            -0.0030348559388780655,
+            0.007310876971216526,
+            0.016099591640649783,
+            0.02311929597780174,
+            -0.04160162038353416,
+            0.027152100845897007,
+            -0.00025396315502635146,
+            0.023717123370023598,
+            -0.04186918759097667,
+            -0.017988531857836703,
+            -0.03106663560912813,
+            -0.023595079699181436,
+            0.02437134274097885,
+            -0.02317559303912261,
+            -0.012327263775257306,
+            0.000989566600374836,
+            0.00954921216572729,
+            -0.005155079395685017,
+            -0.017508404061379164,
+            -0.04582321461886958,
+            -0.0014367264231457263,
+            -0.030417229142190933,
+            -0.017829710557332023,
+            -0.001056472801554354,
+            0.005016912084754242,
+            -0.06276644565618253,
+            0.0109965681725148,
+            -0.07270669271977365,
+            -0.004542796125637244,
+            0.040139998232966566,
+            -0.017890441783630643,
+            0.055909102081208445,
+            -0.05029104308417623,
+            0.01673962668706085,
+            -0.04507293803055318,
+            0.015385506084493484,
+            0.025677372277539627,
+            0.030235171586316813,
+            -0.022189978375852563,
+            0.008461188938299745,
+            0.008033707483259569,
+            -0.045089505755429415,
+            -0.04144698743458133,
+            -0.013640256640978024,
+            0.01036782197518379,
+            0.0012696958402094232,
+            -0.016090728932978384,
+            0.023838437877040163,
+            0.011946482002767135,
+            0.03342604921859589,
+            0.04211801335157251,
+            0.002708015395637621,
+            -0.047156646610976634,
+            -0.013104464812562618,
+            -0.003933116891120681,
+            0.03155321708116329,
+            -0.018866109579446184,
+            0.015376812010630058,
+            -0.01955557376085726,
+            0.016255217383093355,
+            0.017868332102274922,
+            -0.020325179506828335,
+            0.013434364256590097,
+            0.004176904701252156,
+            -0.03877060944639854,
+            0.0006475542749369515,
+            0.00042468300294954125,
+            -0.02116992978038693,
+            0.0032654387271108293,
+            -0.00575003786159731,
+            -0.05149954071159492,
+            -0.031382926943128234,
+            -0.015730563640253065,
+            0.0382312502769352,
+            0.004297939848910064,
+            0.019724983545557365,
+            -0.017907869879607064,
+            -0.005165628493894463,
+            0.03698976707579373,
+            0.008501001373266799,
+            -0.01725527821298066,
+            0.006879620077191753,
+            0.02455792200929138,
+            -0.03224193972301503,
+            -0.039181328884413266,
+            -0.0008206986106407542,
+            -0.023306758893533952,
+            0.01672946960876128,
+            -0.02658587698869307,
+            0.005873620623736886,
+            0.0025331170979801232,
+            -0.03490809296591325,
+            0.02456764314036355,
+            0.003291113462353163,
+            -0.004303423672882112,
+            -0.020359544091059775,
+            -0.016892415165120203,
+            0.017174797487220006,
+            0.04391489511478065,
+            0.007327321684948843,
+            -0.003943713832544148,
+            0.00862684231313155,
+            -0.006361172785753631,
+            0.05826688726534012,
+            -0.02284956969636637,
+            -0.026680023999604643,
+            0.027527436347931256,
+            -0.022597669279069723,
+            -0.006277863878949772,
+            -0.017343446085471857,
+            -0.012065192152203348,
+            -0.0004183712490442143,
+            0.019339311388446943,
+            -0.019423562920897294,
+            -0.010212527447243049,
+            0.004783006556748757,
+            0.012824133870627983,
+            -0.004855945108275706,
+            0.006239949458562236,
+            0.006847581690471813,
+            0.03094005096378827,
+            0.026993131409761086,
+            -0.011331881651539968,
+            0.03297014291876675,
+            0.010265600643742144,
+            4.666113318251278e-05,
+            0.001970879281172325,
+            0.002523664335535017,
+            0.0034107811520368574,
+            -0.03143890864030425,
+            -0.01624584278940617,
+            0.013468729048885906,
+            0.005233309338223403,
+            -0.004550544967856448,
+            -0.023783653541634683,
+            -0.009402494239255264,
+            0.03334923712760375,
+            -0.003260424721959948,
+            0.02748395114281811,
+            -0.0058374897745283065,
+            -0.01985840930273332,
+            0.017595418271610163,
+            0.02881743246631328,
+            -0.05928552440605384,
+            0.017062540894682354,
+            -0.0011508547522925793,
+            0.008953761538519997,
+            0.01610561914042553,
+            -0.008894160513648953,
+            0.013301991014533684,
+            0.04080863883897929,
+            0.010738051773755976,
+            -0.023345002072886145,
+            0.004215726310838674,
+            0.01541749448570608,
+            0.01291813299324752,
+            -0.022747942296528025,
+            0.0008640791861357522,
+            0.011008522066277018,
+            0.005893793007012561,
+            0.0026106450193140156,
+            -0.006278394080583205,
+            -0.025659188287146268,
+            0.024180550333596625,
+            -0.0044491292740074925,
+            0.026967973346698617,
+            0.04390909273429254,
+            0.009596855239692042,
+            -0.03686020608700941,
+            -0.013109951664921729,
+            0.0002799680359670127,
+            -0.0038537577602593366,
+            -0.0037552094562213916,
+            0.011105295862811114,
+            -0.04185615640149602,
+            0.019950890919369135,
+            0.006453241355231516,
+            0.015314645427201533,
+            -0.020031193483734364,
+            -0.024530173488182337,
+            0.0031949828520724337,
+            -0.01492645564652531,
+            0.012875827757040941,
+            0.005057170413681063,
+            -0.004319834710426384,
+            0.005386862713494232,
+            0.00030848054013413157,
+            0.008384941028326983,
+            -0.010811526993301901,
+            0.03175068297908541,
+            0.004680987487229685,
+            0.009264547472281849,
+            -0.02856314241700808,
+            -0.021083721348429348,
+            0.020113396771629096,
+            0.03572366771051171,
+            -0.01708529918264109,
+            0.03346921724975398,
+            -0.009042046660366718,
+            0.0033219155387035227,
+            0.0151720168318272,
+            -0.015805318210244414,
+            -0.008057236707694303,
+            -0.0005483009439015931,
+            -0.002917527004423299,
+            -0.014538511124345527,
+            0.007302979537905532,
+            -0.0208529958115422,
+            -0.02965227242147972,
+            0.020064178921373875,
+            -0.00011892622932349096,
+            0.016455188521316146,
+            0.010335044657747514,
+            -0.007816155992852236,
+            -0.0052262330850905165,
+            0.006959344748883874,
+            0.04432147882994085,
+            0.026734484553994495,
+            0.05547661378166294,
+            -0.005787997952888261,
+            -0.004559444854138438,
+            -0.0082295581123363,
+            -0.006588172755707144,
+            0.04524009609473585,
+            0.015915003261638767,
+            -0.004910285095602864,
+            0.017495607570191165,
+            0.01557886571266685,
+            0.03858259260136037,
+            0.0027648525902558316,
+            -0.016912796336835408,
+            0.04014755669075221,
+            -0.004077405782937414,
+            0.008616827127381418,
+            0.011102692199778469,
+            0.0024185742218798562,
+            0.00807141794538637,
+            0.02996274720815659,
+            -0.012429186019907761,
+            0.01680942269506771,
+            -0.04571916592313758,
+            -0.02775530246832996,
+            0.014370280587419414,
+            -0.004931905540824582,
+            0.03419959665162407,
+            0.01668521021731861,
+            -0.023105305446763272,
+            -0.024761385385514877,
+            0.021529428247380984,
+            -0.002603042335561961,
+            -0.024135326707477325,
+            -0.015906833215844473,
+            0.003286605488908195,
+            -0.0023437997044735617,
+            0.0060251848371355166,
+            -0.013481652076194643,
+            0.0002908426939460218,
+            0.01953362396253743,
+            0.0038544889405965057,
+            0.020760460182873287,
+            -0.016224396121239458,
+            -0.00259233446682179,
+            -0.009282861237653619,
+            -0.02952130765234087,
+            0.017988370145301916,
+            0.006138236653154911,
+            -0.007607182955958238,
+            -0.009516767050028553,
+            0.0048347778733230365,
+            -0.005819209642901871,
+            -0.0025851701689263706,
+            -0.01073014664868326,
+            0.012433465684360061,
+            0.025784014942833097,
+            -0.016875624785552303,
+            0.007944349714123104,
+            8.651589872318479e-05,
+            -0.022306044741692753,
+            -0.022127016745066307,
+            0.020143472307319017,
+            0.002792646249762062,
+            0.008768813346304779,
+            -0.004074214276300631,
+            -0.00015024341785520542,
+            0.002990550758627699,
+            0.01951746553838439,
+            0.03896451381932467,
+            0.004232673242619471,
+            0.027876586543280867,
+            0.0145860303663199,
+            0.03994234200854897,
+            0.009657017763624678,
+            0.00696607158576712,
+            -0.026055387694656488,
+            0.02819890357011778,
+            0.00018297502510753014,
+            0.043190622821775236,
+            0.022488402221902314,
+            -0.0407730101161195,
+            -0.0021946518523759066,
+            0.01669520746276111,
+            0.013212623257794862,
+            0.017009844136059122,
+            -0.004537512994041156,
+            -0.014041213424740049,
+            0.028135404528975894,
+            -0.022859586430516003,
+            0.0008637006862481223,
+            -0.004486332934188017,
+            -0.025031795510001887,
+            0.007351370254175368,
+            -0.004567816916440959,
+            0.030025413528233364,
+            0.0014976740344892735,
+            -0.022285096307825804,
+            0.022837170656498543,
+            -0.009551269792616045,
+            0.011045530488684207,
+            0.008809047185288755,
+            0.020065001354321994,
+            0.031993362733068575,
+            0.009143210434415064,
+            0.02133640814803879,
+            0.012241777474889248,
+            0.014771538127546474,
+            -0.008804192497858033,
+            -0.02341427260922065,
+            0.023167424359504862,
+            -0.0011598649565273595,
+            -0.029627604902846688,
+            0.0021991474725787303,
+            0.003261878079344105,
+            -0.0005935713331682358,
+            -0.009144689470851402,
+            -0.005580557862988033,
+            -0.007240836451875213,
+            0.019631377011298905,
+            -0.007085213886352692,
+            0.007017573363709754,
+            -0.004074171791476129,
+            -0.00822871835665378,
+            -0.0421101573155962,
+            0.004210914075286222,
+            0.003408427700622687,
+            -0.01752179330665445,
+            0.006384440404174512,
+            0.011121481025573962,
+            0.043020575455262985,
+            -0.022809979913698254,
+            -0.0005737138409049003,
+            0.019206623212867476,
+            0.0009562551889820241,
+            -0.023427467810802533,
+            -0.038931688376706186,
+            0.007172345672967499,
+            -0.03467050750542982,
+            0.023185301620612612,
+            -0.0007075596339842658,
+            0.02387977204565927,
+            -0.03043468414449851,
+            0.013909304572370437,
+            0.004784668158421625,
+            0.012524879174804915,
+            0.021654512058639427,
+            0.03142146922870715,
+            -0.02053473475540882,
+            -0.050409862638621725,
+            -0.005406667316025549,
+            0.00035138762584085976,
+            0.022018487016014856,
+            -0.026014580970974616,
+            0.00025633004699302783,
+            -0.003349094027291921,
+            -0.02401366769212778,
+            -0.01735996745614562,
+            0.01962340791446025,
+            -0.050011480783403936,
+            0.015042913576492952,
+            0.0262737424555697,
+            -0.02507473901367252,
+            -0.05408889321034101,
+            -0.018988587082325706,
+            -0.028217893770659686,
+            -0.009285770094319318,
+            -0.00661159230987788,
+            0.028597405154740078,
+            0.003525576256054959,
+            -0.011462219358952466,
+            0.025401668390029348,
+            0.006155837251830549,
+            0.0026050693705924593,
+            0.015985722392086222,
+            0.007289910674212967,
+            -0.011452511453623235,
+            -0.014599577191293152,
+            -0.04558584556100621,
+            0.010197536827229795,
+            -0.0045431962637690155,
+            -0.019372598050508005,
+            -0.030147301333718016,
+            -0.0137828704002156,
+            -0.027060515748528666,
+            -0.0017291966950107505,
+            0.053783684195381654,
+            0.03292494290301388,
+            0.005560172296221419,
+            -0.0026085670565688026,
+            -0.010466890489356092,
+            0.036247483457995895,
+            -0.006789746401727701,
+            0.03662690472838588,
+            0.001784522073210422,
+            -0.0242673253380092,
+            0.01169382884548301,
+            0.02892627181542449,
+            0.009858451662166387,
+            -0.0005592283645355213,
+            -0.006160132931009508,
+            -0.008782830470079547,
+            0.0031207630972208394,
+            0.025613233596604418,
+            0.003268170392795548,
+            -0.010062991425127038,
+            0.018106967407336815,
+            0.004480291699976396,
+            0.02010374270557836,
+            0.03555357587371324,
+            0.023489837757983054,
+            -0.021782173652026497,
+            -0.026581762796336082,
+            -0.0013561486781077014,
+            0.0072266106510934505,
+            0.03551967295942857,
+            -0.01651315514034218,
+            0.0001274396319698452,
+            -0.008272735507120127,
+            -0.00894931703949637,
+            0.011149952898504355,
+            -0.0029671593121986116,
+            -0.0030752010285951576,
+            0.01662711184497578,
+            -0.0011672263615879709,
+            -0.033448222499147764,
+            0.0078288560210653,
+            -0.023496331346917084,
+            0.0051040606404415545,
+            -0.004113104164416575,
+            -0.007076525381595441,
+            -0.002157548492138289,
+            0.006302962169276006,
+            -0.008065841090139872,
+            -0.00035242603988955366,
+            -0.01586406637763306,
+            0.008271894439780969,
+            0.007438312999409211,
+            -0.0069617481460406095,
+            0.011402932583388205,
+            0.002666895909532698,
+            0.0005219881149353855,
+            -0.012159425670629751,
+            -0.012042145306711322,
+            -0.00957276588761916,
+            -0.03401863879157551,
+            -0.02604353712472079,
+            -0.024949533021906844,
+            -0.0014209149651479556,
+            -0.006566685925666972,
+            -0.03187796739029736,
+            -0.01209152137926458,
+            -0.011110763124407752,
+            0.007756961097537267,
+            -0.013260548545917543,
+            0.005433285079114047,
+            -6.399098520041351e-05,
+            -0.0044960488826674555,
+            0.016615636438440516,
+            -0.003643291599919672,
+            -0.019488010211707343,
+            -0.03927818878149417,
+            0.008209336177358258,
+            -0.02805593125673831,
+            -0.0038816771550190788,
+            0.007607119219721711,
+            -0.00528681677614909,
+            -0.029502897838743516,
+            -0.009149794737430706,
+            0.008767001199311985,
+            0.003507217008280066,
+            -0.01773349756494436,
+            0.03579670773605949,
+            0.026971617790626075,
+            0.011875179879209696,
+            -0.019891342676443626,
+            0.0508929813275565,
+            -0.0047010735337056275,
+            -0.012544935684845649,
+            0.018297503869683027,
+            0.02355022268131519,
+            0.03273167043015095,
+            -0.024830243320788693,
+            0.0009370814985214882,
+            -0.021937793653233454,
+            0.00535662615596573,
+            -0.035009915581788176,
+            0.0028595659314725012,
+            -0.014395027018699446,
+            0.015596459769560532,
+            0.03397372881171701,
+            -0.013254226072981457,
+            0.011195747181764677,
+            -0.016299384375589934,
+            0.00730244731485402,
+            0.00569944090461748,
+            -0.012916284280345044,
+            -0.005812311882399539,
+            -0.022902815152116493,
+            0.020140276669979015,
+            0.005809799950176385,
+            -0.0008130655714656685,
+            0.0104085826652653,
+            -0.044723432238900435,
+            0.03286429597337671,
+            0.008730225944501239,
+            0.017599189944921396,
+            0.0029762446259779998,
+            -0.0003068671951232415,
+            0.011611551606224214,
+            0.011228401175698422,
+            -0.016029727902740706,
+            0.004345913009102136,
+            -0.004613061137378433,
+            -0.03850533909474302,
+            0.009363351681220008,
+            -0.03631184057978666,
+            0.014427784677340159,
+            0.016026872002233575,
+            -0.004170582842643461,
+            -0.025166189571409366,
+            -0.005428471012333673,
+            0.010827910811085384,
+            -0.009783069664327864,
+            -0.009987805234276443,
+            -0.04163358183060102,
+            -0.025236440364963034,
+            0.034097508113604576,
+            -0.002119474613596975,
+            -0.025826403511707147,
+            -0.010830738112107807,
+            -0.011351132636491575,
+            0.020207687645133,
+            -0.006542026283946858,
+            -0.0006263975804936843,
+            -0.04150011545311013,
+            -0.0073396962917864515,
+            0.011192225907726928,
+            0.027746213794734322,
+            -0.035970017302005566,
+            0.02708126943501373,
+            0.002105633542220187,
+            0.014262674845755191,
+            -0.01626664983003942,
+            -0.005631805361903626,
+            -0.013990539604372344,
+            -0.035877093361375685,
+            0.016487723013906077,
+            -0.008607758248055054,
+            0.004714436224135091,
+            0.04458009880881818,
+            -0.0036872068506712727,
+            -0.009466125348499464,
+            0.007600705817195143,
+            -0.0034590883553048063,
+            -0.010351576085287273,
+            -0.0028564505793999295,
+            0.001959661298507,
+            -0.007080697863126256,
+            -0.04183476693971616,
+            -0.00499888976530217,
+            0.04684391782633148,
+            0.006361744381879857,
+            0.013276793765092465,
+            0.013516220521120684,
+            -0.018335746557932003,
+            -0.045577810515431215,
+            -0.0071616111758423025,
+            -0.0032763880176386808,
+            -0.022708028392758567,
+            -0.0240284945459861,
+            -0.008735040856657758,
+            0.005844145402902923,
+            0.006359368165686724,
+            -0.01120763599008222,
+            -0.022676131361171057,
+            -0.027085968700452136,
+            -0.010526103176253857,
+            -0.02123944716136792,
+            -0.010792145796733375,
+            -0.027142270190771887,
+            -0.028742234538796804,
+            0.02880426953386151,
+            0.007575182232859564,
+            -0.004084088301176117,
+            -0.010005765476121966,
+            -0.01255640546813973,
+            -0.0004774761617900028,
+            -0.0025307994917461232,
+            -0.005822065709999144,
+            0.0038614286313282164,
+            -0.011390482620811868,
+            0.023152086963976554,
+            0.0021220701260546895,
+            0.018329162172128138,
+            -0.05772324571231113,
+            -0.002709083851090071,
+            0.021497896395795905,
+            0.0033025410488410913,
+            -0.005412445141910314,
+            0.029352650459792132,
+            0.011223947499453582,
+            -0.006539460282439711,
+            -0.031355428613718234,
+            0.020942976176820262,
+            0.024122993515785947,
+            0.00817615634503138,
+            -0.012994160510030828,
+            0.01221095914384204,
+            0.01629778384790159,
+            -0.004494985018246821,
+            -0.011161098655072933,
+            0.005586857967039181,
+            0.020941039642372112,
+            0.013580622629911925,
+            0.0013398071867077221,
+            0.030905022925366163,
+            0.0006862084391508121,
+            -0.03521231612862609,
+            -0.01908191995763771,
+            -0.01275995164030872,
+            -0.003584331305986364,
+            -0.015264425022763126,
+            0.005294329654809353,
+            -0.002355465010876059,
+            0.006335052149475905,
+            0.022655379849785256,
+            0.03593231939948758,
+            0.0146937743203089,
+            0.00937330328448886,
+            0.003167462724350819,
+            -0.0013235176101076284,
+            0.012102872918129403,
+            0.008638011773075119,
+            0.0350701240268404,
+            -0.014653221697693864,
+            -0.011003994230902835,
+            -0.0037767288519369537,
+            0.010100131061186566,
+            -0.004113527518662366,
+            0.020414393256136372,
+            0.03274709615649469,
+            -0.0037222949335916876,
+            -0.005604552180246868,
+            -0.006160118231975786,
+            0.0041467454414452565,
+            0.01807879127455274,
+            -0.01252134965321004,
+            0.004364858987501461,
+            0.005549300665159658,
+            -0.0071897162833861,
+            0.018567059255813558,
+            0.0002622074720860951,
+            -0.012983550838417323,
+            0.03266376564402191,
+            0.11449509387240629,
+            0.0017251916835047628,
+            0.007125215166916139,
+            -0.040229997815548584,
+            -0.020537987527148888,
+            -0.00245728735683473,
+            0.005779972828836593,
+            0.007004732428236457,
+            0.02225914982152608,
+            0.006505827715591914,
+            0.0006708045351142873,
+            0.031708522931495536,
+            -0.017334083511709092,
+            -0.002522861743921599,
+            -0.01745656559688129,
+            0.013890451996979048,
+            -0.014312605700272691,
+            -0.013784963572164409,
+            -0.01572417853955075,
+            0.004507425871748977,
+            -0.000601158156253668,
+            0.0016158265715634729,
+            0.01791303234734911,
+            0.0011926917387973374,
+            -0.026393351252156246,
+            0.0006694008039871878,
+            0.001026031515214626,
+            -0.004514419527575117,
+            -0.011320929719992686,
+            -0.02223001723942815,
+            0.016235841442653918,
+            0.014188311619681008,
+            -0.004895218832118094,
+            0.009244697376301004,
+            0.010558796153959677,
+            0.02707485418319703,
+            -0.028380342401606758,
+            0.0040204430418629405,
+            0.007023041225985294,
+            0.01606556581447933,
+            0.017617862031190843,
+            0.0034870847111552862,
+            -0.011253315705053224,
+            0.03238289399766116,
+            -0.010563282687352038,
+            0.0065093536270233385,
+            0.014966802208644462,
+            -0.03541611540945151,
+            -5.821682910348165e-05,
+            0.0061985444938402745,
+            -0.011716201661363997,
+            0.009580035961315467,
+            -0.03120692534097433,
+            -0.0039291243987978845,
+            0.009317468331974374,
+            0.017611312558635673,
+            0.008804996982574765,
+            -0.0034129169537151925,
+            0.010822226698822738,
+            0.012510661458054741,
+            -0.0027147975086409966,
+            -0.008855213644027594,
+            0.01238502686635956,
+            -0.0007687053446954701,
+            0.00450108863711973,
+            -0.00799170686059026,
+            0.0029755595812552293,
+            0.002290650917244691,
+            -0.01554856300008412,
+            -0.011779430686394945,
+            0.00438393126679289,
+            0.0036931891075561957,
+            0.001535662168147833,
+            0.012014408636737167,
+            -0.006087255045485002,
+            -0.0077816220325470226,
+            0.0040573724156794284,
+            0.003136943055301706,
+            -5.505014316363865e-05,
+            -0.030625245024561508,
+            0.01585923084309149,
+            0.0067595435845396405,
+            0.010013308598626031,
+            -0.005315626260228517,
+            -0.00903142414091241,
+            -0.007240920015018893,
+            -0.01228521172101564,
+            -0.010657894223173453,
+            0.007658605592936815,
+            -0.00724924993432376,
+            0.011124174211132038,
+            -0.03128027787357322,
+            -0.013988715435985316,
+            0.01231792993245938,
+            -0.01528161242159377,
+            0.0007675821323979293,
+            -0.008389914312201402,
+            0.0009417242487395995,
+            -0.005119304379761705,
+            0.007887436211143328,
+            -0.03731503169547854,
+            0.007878001990271541,
+            0.02456170736765724,
+            -0.00330924625934981,
+            -0.020259771366167557,
+            -0.009613930457931786,
+            0.0241314385136084,
+            -0.008267831641886344,
+            -0.01645888287387047,
+            0.008421304862524506,
+            0.010140813149093993,
+            -0.0006309118552875393,
+            0.0008237784055549769,
+            -0.003726984191590917,
+            0.01669125388798323,
+            0.02054777616420997,
+            0.041930770964107515,
+            -0.014789610432004559,
+            0.01092349703703152,
+            0.02866938974792608,
+            -0.03378576980722847,
+            0.014728113846952417,
+            0.02518409645155824,
+            0.014590531327650454,
+            -0.0004446956315190408,
+            0.001609777186372437,
+            0.0068939321280085475,
+            -0.014273492115311677,
+            0.009247676157551494,
+            0.013380104440744367,
+            -0.018407746260686045,
+            0.020444430509441124,
+            0.03140071554008003,
+            0.0025839998101985736,
+            0.01355205268903058,
+            -0.006268939830271642,
+            -0.013860692186849113,
+            0.014717364068116991,
+            0.031429975488060896,
+            0.0006919565473572909,
+            -0.018561500446336806,
+            0.019128244642842268,
+            0.006988095806264863,
+            -0.003678529269242034,
+            -0.0021993078588142664,
+            -0.021334919445200388,
+            -0.021003586791369228,
+            0.017304350704792804,
+            0.013508779408333964,
+            0.027794485858413365,
+            -0.010594702782952029,
+            -0.007818089330662161,
+            0.014293688127901867,
+            -0.022368568887004094,
+            0.015426482139881204,
+            -0.02160820059598751,
+            -0.00571499813703566,
+            -0.0077402210272634235,
+            0.03274360648629609,
+            0.001441304806200403,
+            0.02734331677589297,
+            0.016457027764064002,
+            -0.033609420134045206,
+            -0.029790535553657092,
+            -0.00712106584238484,
+            -0.02307933297713104,
+            0.010356561171963577,
+            -0.03901183838032092,
+            -0.00071735150189382,
+            0.008251646656432826,
+            -0.008079917288471352,
+            -0.0007087526632966007,
+            -0.006419191399519291,
+            0.013989344185652506,
+            0.013661633160336152,
+            -0.0228103143826663,
+            0.023644639659577426,
+            0.008343481940133684,
+            0.025998483906975782,
+            -0.02076055376006461,
+            0.019148568238437927,
+            -0.017705363646416224,
+            -0.02777313933338929,
+            0.014358130220942483,
+            0.023585731219788467,
+            0.01635364038245971,
+            -0.007122853965024528,
+            0.007110003583832299,
+            -0.03556761320536796,
+            0.027585457849964405,
+            -0.013064062623866305,
+            -0.012504161289952458,
+            -0.0032379276562094555,
+            -0.011931827532360686,
+            -0.002574161005800279,
+            0.0008533907465694845,
+            0.0034345232246102806,
+            -0.03131462673429805,
+            -0.0002450959360520896,
+            -0.03727416416766574,
+            -0.028944814604815072,
+            0.01360126708805471,
+            -0.00036361284498363273,
+            -0.0049097813612814616,
+            0.02933242705521636,
+            -0.026339136251046315,
+            -0.02693295650849956,
+            -0.009590247544093352,
+            0.02853096456887074,
+            -0.0220673798697511,
+            -0.004519050393279848,
+            0.0015034014871316069,
+            -0.01828226786700525,
+            0.004436383319946364,
+            -0.0012748481042235446,
+            -0.010416065158876917,
+            -0.01105146139270597,
+            -0.03229479099927737,
+            -0.005822133182096844,
+            0.02106215163620738,
+            -0.006957125617688484,
+            0.002746241228040847,
+            0.013312248331456934,
+            -0.010968629118224582,
+            -0.0029946900704643812,
+            0.0017875284441794235,
+            0.023304175718480834,
+            -0.021033941064173063,
+            0.007714334541806199,
+            0.005955442031111047,
+            0.0016405586860131207,
+            0.018406562522853148,
+            -0.0054922197063461825,
+            0.0037690571959124617,
+            0.0162126534419016,
+            -0.009969304596173302,
+            0.017959307638726878,
+            0.007691549802475525,
+            0.006681837561745192,
+            0.002775700618222622,
+            0.007712770283579111,
+            0.009752652973700739,
+            -0.013506020126078112,
+            0.016911141425375933,
+            0.022468715726349706,
+            0.00023233555209372766,
+            0.021066580799163234,
+            -0.02302516292580905,
+            -0.028618544628453015,
+            0.018297878810036087,
+            -0.0225078941299981,
+            -0.012735208680311652,
+            0.016459925496937065,
+            -0.0043032724578332855,
+            -0.00797609922711709,
+            -0.014986057073351055,
+            0.02260618561608301,
+            0.02132930921464508,
+            -0.020249788476562527,
+            -0.012385936327068884,
+            0.0072304431443959505,
+            -0.004373475447051833,
+            0.007086656081865733,
+            0.03195810137269375,
+            -0.015982210897392764,
+            0.006818890093901413,
+            -0.0038306138017521626,
+            0.0030753642449792215,
+            0.017830546304667783,
+            0.009470018737705222,
+            -0.0019815241970236146,
+            0.030100238416495066,
+            -0.001330614013444443,
+            0.016392226282459177,
+            0.0004888028477871524,
+            -0.010948833692356176,
+            0.006343890495626752,
+            -0.026652806306173643,
+            -0.008627570055007363,
+            -0.001203530223345183,
+            -0.009581494451831351,
+            0.07598616930784373,
+            0.003930153552069478,
+            -0.01295957554290434,
+            -0.001673007115291578,
+            -0.001368287102181597,
+            -0.006897179139643672,
+            -0.00960679451603541,
+            0.0287166325268306,
+            -0.016613613331623346,
+            -0.01894527592131826,
+            0.0008370014132002703,
+            -0.006785618888421608,
+            0.00919924251249592,
+            0.01816363936999796,
+            -0.010225961928154075,
+            -0.01604759761423727,
+            0.001577138242530474,
+            -0.003520564936698765,
+            -0.004395434431595494,
+            0.0052399043934902014,
+            -0.010127122101423486,
+            0.006456662777583096,
+            0.026560491731327886,
+            -0.013684536319659296,
+            -0.01425196172663986,
+            0.004084982328707292,
+            -0.0010682084059972982,
+            0.009758641913073978,
+            -0.00864879062601467,
+            -0.00481222050200678,
+            -0.014375330954464725,
+            -0.011378890549403524,
+            0.02505641780042516,
+            -0.02861164504722546,
+            0.0034060847953066354,
+            0.014937227531980433,
+            -0.00016904224334342915,
+            0.01541381851648916,
+            0.0033028855218525644,
+            -0.03615935765718852,
+            -0.013677774380513626,
+            -0.02073839691250734,
+            0.019821074228011413,
+            0.01688422012576042,
+            -0.0033731840788147442,
+            -0.013840993586441808,
+            -0.026149880178874887,
+            -0.007430563876740068,
+            -0.02466234818567362,
+            -0.020708282040996014,
+            -0.019276991448222817,
+            0.006137002799644838,
+            0.005162892426268572,
+            0.022881759407670583,
+            0.0025854880108157857,
+            -0.015058889448404231,
+            0.0029948608544278386,
+            -0.006722836436935071,
+            0.011845298212977357,
+            -0.02747944486818378,
+            0.015311272959213262,
+            -0.01610493464983465,
+            0.025239768880017402,
+            0.007425208902253236,
+            0.0010569008485112994,
+            -0.008822681490988575,
+            0.00010704167529046774,
+            0.014883787023445807,
+            -0.004186968199973372,
+            -0.0007781259002896265,
+            0.02787555889813697,
+            0.02087373813833492,
+            -0.0017783425135865238,
+            0.008743718448136432,
+            0.028026471206552342,
+            0.019081083283793984,
+            0.0325261942213039,
+            -0.013815526479760766,
+            0.005113676604826083,
+            -0.028873600114081546,
+            0.010510475843707614,
+            -0.010871039729515013,
+            -0.010201824683340651,
+            -0.0067946599549165735,
+            0.027756887504389347,
+            -0.007962601205446472,
+            -0.0006687976124826922,
+            0.03469894503563926,
+            -0.014185294847328063,
+            -0.00010543064621374546,
+            0.0007386418834840544,
+            -0.009759481309152999,
+            0.02742722730386853,
+            -0.0073590367291510046,
+            0.018456032721630843,
+            -0.019560882612141943,
+            -0.013705917411254969,
+            -0.011894560208199602,
+            -0.001142768182039817,
+            -0.02576754877141608,
+            -0.008755240066826174,
+            -0.02249287216087102,
+            0.00859172262856532,
+            -0.0046012388553094375,
+            0.018845920253446127,
+            0.0011617844393195566,
+            -0.005275799428018491,
+            0.029069367036565145,
+            0.0088080893110627,
+            0.018100411389202663,
+            -0.0004971948011851863,
+            0.01216566778850985,
+            0.02369448391321832,
+            0.002637001729688794,
+            0.005012948614448268,
+            -0.024213704079796478,
+            0.003420184574622472,
+            -0.0004359900462023086,
+            8.265488264269535e-05,
+            0.016267948861008616,
+            -0.019572032453100303,
+            -0.02216264534756073,
+            0.021871880626992544,
+            -0.012742926070754964,
+            0.011676769226089007,
+            0.0035362870996336194,
+            -0.010732229460038356,
+            0.0006838777488770268,
+            -0.03083438636386314,
+            0.008820746379709355,
+            -0.03216685858285627,
+            -0.032230808828686675,
+            -0.013547681710236494,
+            -0.009994106571421729,
+            0.0008758088338659712,
+            0.028447187234357855,
+            -0.00795317029726568,
+            -0.015797892954136448,
+            0.005363565557248085,
+            -0.011199252697624111,
+            -0.00817162221789202,
+            -0.0007867892785855576,
+            0.005856939999700809,
+            0.009461959355310273,
+            0.005997804620690314,
+            0.011648335019189696,
+            0.007491946658046298,
+            -0.008028800679169293,
+            -0.008837725220736191,
+            0.015889923814605387,
+            -0.0014409716087425222,
+            -0.01176424391171685,
+            0.030430130082762363,
+            -0.013300620185884817,
+            -0.0005592926632214984,
+            -0.0021975669766032472,
+            -0.0015687208316663597,
+            -0.0021652076181411244,
+            -0.014225289143358926,
+            0.01870568404189425,
+            8.334501002782455e-05,
+            -0.014288806218067678,
+            0.0032841743905975634,
+            0.003841780428153241,
+            -0.0036498339483411074,
+            0.008388922960269383,
+            -0.008147275785908849,
+            0.017348147498892,
+            -0.015113395971380741,
+            0.008740452410210708,
+            0.0038335161236289234,
+            0.0043482078469644385,
+            -0.0065755348640655865,
+            -0.01592809184384243,
+            0.009642593996701513,
+            0.012051872087663658,
+            0.02173264012652684,
+            -0.02525660485859878,
+            0.00876161653901566,
+            -0.005374533920781607,
+            -0.014534179652013885,
+            -0.016127866637369738,
+            0.00982303248417706,
+            0.022322910435388785,
+            0.0063086346898091296,
+            0.012139263756190026,
+            0.00013383660542132047,
+            0.005848259030010402,
+            -0.014767198851314148,
+            0.010570158368602513,
+            -0.011355774405066537,
+            -0.018439261453084426,
+            -0.005243587266408389,
+            0.028894120979627323,
+            -0.005492024421080143,
+            0.017938747701972086,
+            0.0021350711250093858,
+            0.029007356490801176,
+            -0.008686781541806358,
+            0.0014883465513564527,
+            0.0039866022469891155,
+            -0.01295271525889559,
+            -0.004272008629260297,
+            0.01976273843646234,
+            0.04882868638601926,
+            0.010204475902027572,
+            -0.0009155109545661691,
+            0.010069071220988797,
+            0.005626418787776309,
+            0.002148101019776153,
+            -0.00805781810841286,
+            -0.030431071067211577,
+            0.008883707378543252,
+            0.005628218215212441,
+            -0.01997825125961566,
+            0.019838234890736068,
+            0.026483365072033395,
+            0.023040623182975964,
+            -0.011585643598171132,
+            0.016539930468533123,
+            -0.010092529089936188,
+            0.0177008735686489,
+            0.028665424114962996,
+            0.013810061902648515,
+            -0.010487150104822674,
+            -0.0020847870330548534,
+            0.006329342599564054,
+            0.004186981876004016,
+            0.00026239408223817384,
+            0.015190614519766204,
+            -0.019155180136775043,
+            0.01005728016778505,
+            0.014725826697964637,
+            0.03322436221319799,
+            8.049930133627564e-05,
+            -0.01645191835557271,
+            0.0016249281996128196,
+            -0.009976155662939693,
+            0.00856399492362911,
+            0.000632427127231252,
+            0.0032242260692672124,
+            -0.009688119824276678,
+            -0.012682779503915694,
+            -0.002851918183031152,
+            0.009403205508474241,
+            0.0021188716361101698,
+            0.002759004011231831,
+            0.007641925150177999,
+            0.0022830062031146204,
+            0.01713024474384216,
+            -0.0011181980841546123,
+            -0.00029141889484139217,
+            0.004601090256674874,
+            0.010509199590505558,
+            -0.01649852835819883,
+            -0.017548369964263514,
+            0.004337237475053179,
+            0.012217494210546133,
+            -0.013159179948224596,
+            -0.013241981632059755,
+            0.015704543608959117,
+            -0.01913575130892795,
+            0.024934102868065366,
+            -0.00942381892138331,
+            -0.004543611336526587,
+            0.000343200128934031,
+            -0.00727538880611703,
+            -0.0038751118497522673,
+            -0.0030064279504724797,
+            0.003984715814505561,
+            0.0019261818110367147,
+            0.007472350699291607,
+            0.008472369788801792,
+            0.014568547873264247,
+            0.006043895352236181,
+            0.0032580830408778395,
+            0.009109104498825864,
+            -0.014523846294121576,
+            -0.014800649887278973,
+            -0.002704791598011488,
+            -0.008895394038700809,
+            0.00902016867266151,
+            -0.006607182154874112,
+            0.00845408382182461,
+            -0.015501867835442603,
+            0.015573642225745032,
+            -0.012200552219016517,
+            -0.0008900215712478553,
+            -0.009666233545144905,
+            -2.7470277092642562e-05,
+            0.008191697904514857,
+            0.014605287653054787,
+            0.0070552942843382215,
+            -0.031921405225964206,
+            0.0017403427534633655,
+            -0.008846568352320488,
+            0.03446319360336241,
+            0.002612606662691094,
+            -0.01060863123202008,
+            0.020352181356568008,
+            -0.01592124132155158,
+            -0.00027704668715660693,
+            -0.01602075523652062,
+            -0.004797557413047212,
+            0.00707939446796679,
+            0.016399017978990946,
+            0.02759049512664933,
+            -0.005697020356865688,
+            -0.007995305872076221,
+            0.006384047872862793,
+            0.029590696670334232,
+            0.012246955508890425,
+            0.024748351124415785,
+            -0.007633712004672087,
+            0.014348228716100414,
+            0.007773260552660029,
+            0.0076045488653348975,
+            0.004321689250440021,
+            -0.002591801797145718,
+            -0.011698574606499644,
+            -0.014974971814360001,
+            -0.0037563183330623384,
+            -0.016987915346053498,
+            0.00012247833530362867,
+            0.021344400276501117,
+            -0.00475351423959752,
+            -0.0033847161875730026,
+            -0.01874186437694691,
+            0.0033461670218611346,
+            -0.010029445302549983,
+            -0.00367421246775698,
+            -0.013133469163793961,
+            -0.0254122035745965,
+            -0.018556369080030412,
+            -0.002155244024999317,
+            -0.00929309186646207,
+            -0.015194269781263309,
+            0.0070354690219415535,
+            -0.016305117309271634,
+            0.0080837166473439,
+            -0.00448473417024141,
+            -0.016508775226236883,
+            -0.00958925666995396,
+            0.01100335153658195,
+            -0.0041819993634961685,
+            -0.01929493648733792,
+            -0.01282300765645703,
+            -0.011300260564838231,
+            0.0051733704121938884,
+            0.005996780699975213,
+            -0.0038111648304178544,
+            -0.0028279639348479987,
+            0.016640121845366295,
+            -0.010197365811240557,
+            0.009878730312873492,
+            -0.03335474019383245,
+            -0.03551826040406959,
+            0.004497847402931096,
+            0.01507007013703286,
+            0.0013085881889541252,
+            -0.02078680101065954,
+            0.010806315988966003,
+            -0.016499186313125556,
+            -0.00858013792516617,
+            -0.008522534127498645,
+            0.009649521088411411,
+            -0.007283003380127847,
+            0.01299542385123007,
+            -5.3226845309105124e-05,
+            0.030477683597373687,
+            0.0074153455615142755,
+            0.004161881195438522,
+            -0.00679081048132963,
+            0.008354912600159931,
+            -0.001621776416108105,
+            0.004080191054849395,
+            0.02120524874226696,
+            0.012440848502230418,
+            -0.00218041171518331,
+            0.006458415098595872,
+            0.005678055538564355,
+            -0.017186537377121235,
+            0.02507960402044456,
+            0.024678178106806115,
+            -0.008205288425409454,
+            0.004108402466376321,
+            0.005783335302169668,
+            -0.008846663924040241,
+            -0.006897707231869481,
+            0.006527795262026734,
+            -0.018474181903007085,
+            -0.0008721381678008483,
+            0.010598084358464908,
+            -0.02255441909004843,
+            -0.004840511019512733,
+            0.021745651362109403,
+            0.004631567631335321,
+            -0.018558972627958834,
+            0.02764788406651861,
+            -0.004942361238163609,
+            -0.0027290999934532494,
+            -0.010876135202650048,
+            0.01671497143541799,
+            -0.012173527207375091,
+            0.008443317346655462,
+            0.011021541861818043,
+            -0.006295639322287843,
+            0.003083685676540999,
+            0.008277206401503182,
+            -0.004504302913715969,
+            -0.0068075424927125676,
+            -0.0010508416962215848,
+            0.010167160518175114,
+            0.0036980815941666336,
+            -0.022704963865525812,
+            -0.006020073261002279,
+            0.02934227992246887,
+            -0.015082922782242507,
+            -0.0050916398557337705,
+            0.005376294716056329,
+            0.007709342013963906,
+            -0.003476236561444464,
+            -0.0023399991233097337,
+            -0.00787486645740744,
+            0.010356414979428607,
+            -0.015950083845400274,
+            0.02076360898078935,
+            0.025608263494550438,
+            0.024504236990840842,
+            -0.014814235990238182,
+            0.010828584133720797,
+            -0.011703992394890898,
+            -0.011047470905668532,
+            -0.015542835222173864,
+            -0.001658296303013357,
+            0.001970208852236451,
+            0.013107754130616248,
+            0.009666887546609378,
+            0.0235853683657408,
+            0.006473544089109194,
+            0.008973300312205858,
+            0.014118601565410305,
+            -0.003902202854013708,
+            -0.018303438244719996,
+            -0.014584226585460755,
+            -0.011977613285028946,
+            -0.01756345491020528,
+            0.0004563602312336091,
+            0.020461659996271687,
+            0.012417744207683839,
+            0.024136626592375227,
+            0.002119213701333967,
+            0.013759902247467922,
+            -2.9525392347047554e-05,
+            -0.000892319792935443,
+            0.00045783825646242434,
+            -0.005238483120312843,
+            0.019255756091120846,
+            -0.004599752813817219,
+            0.01656589522110913,
+            0.0006443773300684925,
+            0.007133940816771615,
+            0.013554761340256566,
+            -0.01704388669741371,
+            -0.003443100892108368,
+            -0.010029620309829285,
+            0.000941391966877521,
+            -0.019633111750884034,
+            0.01934571074127927,
+            0.010273330977233852,
+            0.0017121345993533963,
+            -0.018327189396014198,
+            -0.0029706385099949693,
+            0.03448635131998772,
+            0.019458999453011265,
+            0.0050734354363614525,
+            0.02632443373898792,
+            0.013018762352749236,
+            0.014443611411502168,
+            0.010153370950346075,
+            0.02356766253713842,
+            -0.023479811300626968,
+            0.02095255383200973,
+            -0.011590347850340786,
+            -0.007414136601134069,
+            -0.0035774888741646705,
+            0.0025141508183687115,
+            -0.013588691733097813,
+            0.00982877622088977,
+            -0.0017916539399655826,
+            -0.025386211797192578,
+            0.009164316443206016,
+            -0.008811052979979271,
+            0.010464442616928296,
+            -0.007774569252594923,
+            -0.0002806136360286729,
+            -0.0127084026190163,
+            -0.015184317235990752,
+            0.001259201880504347,
+            -0.009158429132281081,
+            0.023908150915412617,
+            -0.021210848953875233,
+            -0.01321384014436847,
+            0.013997502713857512,
+            -0.006869443694669701,
+            -0.011364239609894453,
+            -0.00374449980963666,
+            0.023673050288879224,
+            -0.009243341446512421,
+            -0.01359367519009039,
+            0.014464166953530989,
+            0.003834735113400469,
+            0.0059713317014342395,
+            -0.007508862694272452,
+            -0.008428705474135382,
+            -0.003280788036976103,
+            0.005998840043285335,
+            0.026615077617284676,
+            -0.014059930972065399,
+            -0.0032833264512892022,
+            -0.009189999799498163,
+            0.004118204537632102,
+            -0.02359889635281631,
+            0.02771075351384059,
+            -0.008871395303200375,
+            0.021705177758560575,
+            -0.0015924449308442096,
+            -0.006239188158271628,
+            -0.024231036074349822,
+            0.027522945599928003,
+            0.003683870966132199,
+            -0.010221180899173243,
+            0.0012768725460662862,
+            -0.052005292736885915,
+            0.016636109796947025,
+            0.008429895913163985,
+            -0.019507994429834353,
+            0.003697519981610208,
+            -0.004773584856005853,
+            0.018983847580558853,
+            -0.022966993228850697,
+            -0.014078986007109447,
+            0.008283766047449966,
+            -0.01153285124362746,
+            0.0031439180465649996,
+            -0.02527749501821793,
+            -0.005754537200570427,
+            0.0011760345167229269,
+            -0.005276936074280004,
+            -0.001705933750221904,
+            -0.020022761608132548,
+            -0.025972227850795115,
+            0.017666017379157135,
+            0.015820875579600418,
+            -0.002278262448951789,
+            0.01320818011561616,
+            0.008709899932124187,
+            0.00836573261091119,
+            0.016034263394940008,
+            0.007124509234103267,
+            0.0081473711608784,
+            0.045498533239850446,
+            -0.019846323044145485,
+            0.007371629414298193,
+            -0.0014854149174929534,
+            0.013951741898629716,
+            -0.030467924817555014,
+            -0.0055144480998344705,
+            0.0062802744565489945,
+            -0.0049373689340857915,
+            0.03510454786147091,
+            -0.017326094216003073,
+            0.01694160526735264,
+            -0.0026075809870445238,
+            0.0007420394265014909,
+            -0.02042003877666893,
+            -0.026156937726134134,
+            -0.014294938857971834,
+            0.0011857406889443987,
+            -0.0010047941261608975,
+            -0.011946583993069242,
+            0.006797001710574735,
+            0.004917161312925866,
+            -0.0020713379451245515,
+            0.015652523103715316,
+            -0.02952061566240878,
+            0.007356800041369564,
+            -0.010733578620243498,
+            0.020008504830359524,
+            -0.00990331048284697,
+            -0.0013319400098308093,
+            0.007236960885181047,
+            -0.004122868383988748,
+            0.013216043472071927,
+            -0.0038337251451538677,
+            0.004227115653354611,
+            -0.005221321656766545,
+            -0.01794343569722877,
+            -0.0016435261495179767,
+            0.021429008432348098,
+            -0.008349684096540855,
+            0.031864497493526205,
+            -0.01264743878037828,
+            0.009174982784686633,
+            0.010692990337462762,
+            0.0006628831036334152,
+            -0.005286285851702483,
+            0.009052971814211658,
+            0.0021431970274545893,
+            -0.008690520448278274,
+            0.006934310986093342,
+            -0.018349449937310143,
+            0.008494451785507706,
+            0.03394632739506833,
+            0.0018511328780472266,
+            -0.0007519661670070526,
+            -0.007266663179613392,
+            -0.01335844201823745,
+            0.018540713029383066,
+            0.005488273906733518,
+            -0.002546531152114063,
+            -0.027536660869590543,
+            0.02007746621157705,
+            0.0012265798861291223,
+            -0.004755325069578531,
+            0.012466557901411086,
+            0.0020495867632654885,
+            0.019989531151046736,
+            -0.010171678639113316,
+            0.030757957899447096,
+            0.010050036220124393,
+            0.002959833051227505,
+            0.0038556928367700113,
+            -0.036655749074851536,
+            -0.01652152778069314,
+            -0.00019357502472740266,
+            0.014271581054865031,
+            -0.012430647252826173,
+            -0.022432580814844155,
+            0.0010346169306977334,
+            0.01148004051449691,
+            -0.0033898642762881046,
+            -0.009650665305271706,
+            0.01560757541276278,
+            -0.020342213902670575,
+            0.006701302394408393,
+            0.007693672045981921,
+            0.0287100214318104,
+            -0.0020735384554022153,
+            -0.0012324838849990056,
+            0.000626416803904792,
+            -0.0055669860801192935,
+            -0.0008037366126870824,
+            0.017251369734470945,
+            -0.013661316947002466,
+            0.0175926211217534,
+            -0.012735241490550407,
+            0.0064713656749838585,
+            -0.0021092378432037563,
+            -0.010629579860050812,
+            -0.020114807971202787,
+            0.0029609495764307357,
+            0.009378003163375899,
+            0.006920144698428806,
+            -0.010438984881449953,
+            0.010715409812532968,
+            0.012197673689706352,
+            -0.00753962861484439,
+            0.014519905132928041,
+            0.022173989895384192,
+            -0.007687503607783572,
+            0.013633321414009471,
+            -0.0041446334728745095,
+            -0.014056264816568229,
+            -0.009985463824440833,
+            0.023929750617160402,
+            -0.0005563806942667597,
+            0.008756997373618809,
+            0.00388125051641897,
+            0.0007856447663040696,
+            -0.011226530517316183,
+            -0.001653447552601681,
+            0.019294448360333633,
+            0.011831934675073843,
+            -0.0013888534822008192,
+            -0.0010525764356699885,
+            -0.0014566936525653407,
+            -0.009427557739483988,
+            0.02049188741459728,
+            -0.022030783927622997,
+            -0.0019604940727076573,
+            0.00461018148439991,
+            -0.022589816803875618,
+            0.0001863117949313752,
+            -0.024570217962989905,
+            -0.01569952241187498,
+            0.011763033781122343,
+            -0.0022530631823737554,
+            -0.016028810867854907,
+            -0.0003242701040942633,
+            7.424127301806556e-05,
+            -0.007815789940508527,
+            -0.004135500768181258,
+            0.006115909992703774,
+            -0.00034193217510010117,
+            -0.0032262189374874842,
+            0.013430154310506884,
+            -0.009996622196870008,
+            0.015431528911575157,
+            0.0067029980830614145,
+            -0.015702167721800483,
+            0.0021406610839557003,
+            0.018500219544264753,
+            -0.0058910087864916085,
+            0.00787021425357441,
+            -0.014681907249972176,
+            -0.0037495551894426303,
+            -0.005787661351944086,
+            0.006051351977821761,
+            0.024849195780885833,
+            0.012969434450643372,
+            0.01139340445077682,
+            0.017405361843075933,
+            0.004325000715054734,
+            -0.01572297405297993,
+            0.0007416798945675923,
+            0.01682727189828363,
+            0.010464036837245312,
+            0.009804326468113703,
+            0.0008975556667684365,
+            0.0043704860361487635,
+            -0.005142176468800175,
+            0.009317064072855551,
+            0.02915482449687158,
+            0.0013434689612997423,
+            0.008696141010760192,
+            0.017276730345147134,
+            0.027038741396131265,
+            -0.0035088997194497725,
+            -0.007469614319788499,
+            0.008739942310300076,
+            -0.009657991604173173,
+            -0.001081629174976152,
+            -0.017000351183542026,
+            -0.012310275736750868,
+            0.016510505259358398,
+            -0.006084627062230492,
+            0.02848913486719956,
+            -0.0010655806791907225,
+            -0.02026498848361482,
+            -0.01565505608194376,
+            0.001404549479440637,
+            -0.01362132895080652,
+            0.005287566256051299,
+            0.014187265685427415,
+            0.01917046513020831,
+            0.004687191682875024,
+            -0.022258337721085656,
+            0.037364962197399806,
+            0.01757108963349427,
+            0.019377870795760123,
+            -0.019422840506328382,
+            -0.006159311237412833,
+            -0.009077095160676276,
+            0.02116697012944498,
+            -0.00880692095488465,
+            -0.009875674856043954,
+            0.024014332004451508,
+            0.017812606845656484,
+            0.005679806286464216,
+            0.006736609904079586,
+            0.0020048663439319585,
+            0.0025414630253661444,
+            -0.015831593851012047,
+            0.017957902498542212,
+            -0.005858125698347523,
+            -0.0058068554681761945,
+            0.017529514573571298,
+            -0.005989581792993925,
+            -0.0030306021203732347,
+            0.00015839997806822045,
+            -0.019697052145531482,
+            0.011618112886207201,
+            0.008256007601576028,
+            -0.017229774429089857,
+            -0.0014757560402567304,
+            -0.0002404480326945295,
+            -0.014118694740134205,
+            0.02004885146995733,
+            0.01694893356262835,
+            -0.0013248884139405398,
+            -0.0051752815372013315,
+            0.008223868147393114,
+            0.007000679132597393,
+            0.005256194890343006,
+            -0.019496098876977206,
+            0.005230597149342019,
+            0.005103909294824105,
+            -0.0014716649591412774,
+            0.010274346323051634,
+            -0.009030594825985526,
+            0.033740380586893826,
+            -0.006505788197604522,
+            0.013065658941136721,
+            -0.0043634731684193745,
+            -0.0055529072007985125,
+            0.001816020676232658,
+            0.013054468478598695,
+            0.009753501030700457,
+            -0.00970446155762939,
+            -0.004172354962129865,
+            0.018697982092775737,
+            -0.011983156146516074,
+            0.0013300791598046338,
+            0.01564072865804035,
+            -0.004858814041438713,
+            0.004708235083775013,
+            0.008635204790345423,
+            -0.004489505237783172,
+            0.01801512858970922,
+            -0.016482572999490366,
+            0.0023953436660054026,
+            0.010694869684962088,
+            0.02390864159830096,
+            -0.019152187046410703,
+            0.021786596486708207,
+            -0.017951050280323393,
+            0.026538582634436314,
+            0.014484894489694262,
+            -0.0035222259457311444,
+            -0.01946200885147006,
+            -0.00850690913394181,
+            0.011654565129063417,
+            -0.021423803876241532,
+            -0.0007985948078290641,
+            -0.0007290222944686444,
+            -0.021367628543092133,
+            -0.011189364017231132,
+            -0.009961779129778331,
+            0.010701637745810138,
+            0.001918633898887013,
+            0.010380983868185359,
+            0.008002182827196827,
+            0.010784760574518087,
+            0.014405640442760822,
+            0.0008958107769384412,
+            -0.010743001445440941,
+            -0.023795589003655273,
+            0.016987708698679597,
+            0.00438463274382093,
+            0.0007862212695139549,
+            -0.01627468994515667,
+            0.011936651269737248,
+            0.013250643815768723,
+            -0.009385605548926286,
+            0.009121882469425302,
+            -0.03716901103204463,
+            -0.004552808410398574,
+            -0.0035704811925430366,
+            -0.0020309197365173475,
+            -0.00908678158053104,
+            0.01559445415882485,
+            -0.013682507836005851,
+            -0.0009590922904844632,
+            0.00011696985057536548,
+            -0.0074756526903548915,
+            -1.5467890432254378e-05,
+            0.020649928837031005,
+            0.008438785939908428,
+            -0.0037382105257999757,
+            0.010376213224654074,
+            0.008826031303751312,
+            -0.013900359449287676,
+            -0.01527474712786084,
+            0.019035366035884273,
+            -0.018776918681908793,
+            0.0063098418544296165,
+            0.01677916431403883,
+            0.020778813096283988,
+            -0.02891862501830075,
+            -0.0007231469961304435,
+            0.022093939332021185,
+            -0.0012807635799950273,
+            0.006370841109790851,
+            0.008781853138377092,
+            0.006754082980602544,
+            0.0029102159062355374,
+            -0.003711173531879644,
+            -4.843770941622283e-06,
+            -0.003940474588075707,
+            0.007155539022862352,
+            0.004366433868349658,
+            0.036453728071874844,
+            -0.013661129348165794,
+            0.002541308805735001,
+            -0.0058838191638835155,
+            -0.00347401304572066,
+            -0.00593469688967835,
+            0.004407665856950772,
+            -0.010055727994798024,
+            0.004330827357536078,
+            -0.001570861535288093,
+            -0.0003731129422346693,
+            0.004883969598300302,
+            -0.001701673379646408,
+            0.007036892564844683,
+            -0.011863650867472625,
+            -0.0003979727750011258,
+            -0.020174851412609312,
+            0.021117284557875267,
+            0.00011666551282792189,
+            0.003662462447326642,
+            0.03723741246290771,
+            0.00476064510662738,
+            -0.00238355131307445,
+            -0.012880057949435798,
+            -0.013551338766321525,
+            -0.012986444917747435,
+            -0.020703821629987552,
+            -0.0036665673548525388,
+            -0.021006311026247164,
+            -0.0052061406234528915,
+            0.010362862594785442,
+            -0.011540184420787572,
+            -0.004023659825622843,
+            -0.015881589631889473,
+            -0.006531981517815174,
+            -0.03114211633687585,
+            -0.01124607488650473,
+            0.007493228366452009,
+            -0.019149285949049886,
+            -0.0039932870926865805,
+            0.026850121590606017,
+            0.001095360635760988,
+            0.01387696424607248,
+            0.0038058248676879625,
+            0.008117686785954085,
+            -0.0045937472017383955,
+            -0.0005669025036141417,
+            -0.01928244877716143,
+            0.003391980330529026,
+            0.008525354854576925,
+            -0.00361507909530282,
+            0.0070008404454106905,
+            -0.008835727374865623,
+            -0.016950017602980382,
+            0.0021654956715219517,
+            0.009334379529385142,
+            -0.006639140003365139,
+            -0.017587472740268683,
+            -0.011049929670847685,
+            0.010401189523318744,
+            -0.02283429470064187,
+            0.006257145380266984,
+            0.016439581169238607,
+            -0.009363030375624585,
+            0.004651575425738153,
+            0.019809183749755425,
+            -0.011810674576781702,
+            -0.005811328501528397,
+            0.008471992070998505,
+            0.013891259999721588,
+            0.0095606134736748,
+            0.02954498207274139,
+            0.0026596809804008996,
+            0.000561870602466584,
+            0.004889263753490895,
+            -0.004277061142504552,
+            -0.017423200644235665,
+            -0.006243787897080996,
+            0.0038507494787521856,
+            0.012873996671952184,
+            -0.009879417960879032,
+            -0.00048575277472139055,
+            0.01682604801673206,
+            0.004530939990187663,
+            -0.0033901248079550284,
+            -0.011868399508738274,
+            -0.020080331108085103,
+            0.0015580259874809898,
+            -0.01362200460317948,
+            0.015354347040143093,
+            0.003390601069331352,
+            0.0015284610287850575,
+            -0.004239880336806429,
+            0.01828645373058079,
+            -0.005013418527935815,
+            0.012749833447770729,
+            -0.006148890994661865,
+            -0.004587831231448011,
+            0.007986533783001116,
+            0.009166578151006778,
+            0.0004939857995863664,
+            0.05421019830078428,
+            0.002838056604877719,
+            0.01333205648597435,
+            0.013419127695838135,
+            -0.014523733369484934,
+            0.014086525147734823,
+            0.012688569853552613,
+            0.018497344547821693,
+            0.013037819487160016,
+            -0.012353776077273462,
+            0.017354801119939964,
+            0.0024084723134891487,
+            0.003909681065813095,
+            0.0022202438064449653,
+            -0.004528576079628071,
+            0.019594683950225115,
+            0.004070222328547378,
+            -0.0074362443729372105,
+            0.004453618452623641,
+            0.005007543168224267,
+            -0.013105524222311019,
+            -0.0112958915514782,
+            0.027105208848863218,
+            -0.01219417557128091,
+            -0.018749971743148304,
+            0.006706594269933445,
+            -0.02008458683853808,
+            -0.03763686535076864,
+            -0.0021332074289244973,
+            0.009992361098878196,
+            0.002795183524495193,
+            -0.00253695604043486,
+            0.01614298064601032,
+            0.016730443047719683,
+            -0.016985296937992726,
+            -0.018245902728910347,
+            0.0025257780765481825,
+            0.0016333325293570182,
+            0.01861807236979066,
+            -0.023168663546030196,
+            -0.006378864850313035,
+            -1.3795465857380962e-05,
+            -0.006829008132963306,
+            0.004850306884033312,
+            0.004703280112191218,
+            -0.0071714759384722965,
+            -0.0027773519096287217,
+            0.002256846645341137,
+            0.01975383140755998,
+            0.013198754584066554,
+            0.01572971337890945,
+            0.002600741959577911,
+            -0.005033508062449482,
+            -0.016817359058724685,
+            -0.005450011991586895,
+            -0.013478330855299777,
+            -0.009091854345715177,
+            -0.003042052092498012,
+            0.008217376329142566,
+            -0.025053469059115735,
+            -0.011965083373220982,
+            0.0119294363235832,
+            0.00391377331459168,
+            0.0003099203772539202,
+            0.007607672728136279,
+            -0.003830132495364688,
+            0.010331053246451333,
+            0.013823890473448383,
+            0.0076112887952959766,
+            -0.004723377809559611,
+            0.009600411526173948,
+            0.011460781436250457,
+            0.006015477068316837,
+            -0.005072711638895421,
+            0.023859899048048694,
+            0.01821920335673539,
+            -0.01680365156728093,
+            -0.002149107383573353,
+            -0.0027829676356423997,
+            -0.01884308738652242,
+            0.009665830801645276,
+            0.00880322398336952,
+            0.008308268605449346,
+            0.003906369351868086,
+            0.00974531700889854,
+            -0.013875904819275925,
+            -0.0050570811397453235,
+            0.002870120865445989,
+            -0.005729556771404781,
+            -0.01901337075686543,
+            0.000297242372701423,
+            -0.011987192261104132,
+            -0.011327157550660895,
+            -0.01409710922165158,
+            -0.003221240464411861,
+            0.005283472511900512,
+            0.021472187640986972,
+            -0.004410881709548286,
+            -0.001855852749039012,
+            0.01351697599517034,
+            0.010060125186506398,
+            0.0021658738823899905,
+            0.0002206474541239167,
+            0.005418828324605768,
+            -0.0027919130944783397,
+            0.0008094042430809068,
+            -0.0015826797905407809,
+            0.021089276765434876,
+            -0.02016611086095107,
+            -0.009413977569213423,
+            -0.009288078098692112,
+            0.005842197094339927,
+            0.028793032462603636,
+            -0.007165634502373454,
+            -0.010104002083017078,
+            -0.01994315293556678,
+            0.0003786527922888834,
+            -0.07810609175343784,
+            -0.005210636756258548,
+            0.001870513618910238,
+            0.03126380113105983,
+            -0.010817888476436834,
+            0.0033673768058629823,
+            -0.002204487854177595,
+            0.014542299655712928,
+            -0.001760107965206874,
+            -0.0013283746982194259,
+            0.0071954806737858056,
+            -0.005640400818010697,
+            -0.006210989097047217,
+            0.005904163661619133,
+            -9.945941028848348e-05,
+            -0.010266938705916303,
+            0.00057263080955709,
+            0.02209498395848068,
+            0.0159398346608872,
+            -0.006575439759804175,
+            0.01816748552021211,
+            -0.00719294553484007,
+            -0.009838738592324234,
+            0.005639288780502127,
+            -0.0019007683459126327,
+            -0.015369619176147573,
+            -0.011475977270803784,
+            0.009862808084641291,
+            -0.030069890644150284,
+            -0.00010725753918890802,
+            0.00943698424646706,
+            -0.013540245857270492,
+            -0.008293042205008774,
+            0.007212974960781723,
+            0.006708343474532583,
+            -0.0012017968151025369,
+            -0.0078026000239507145,
+            -0.02527526731750743,
+            0.0027578400032925625,
+            -0.002446811971857853,
+            -0.004899702349784173,
+            0.0019960465173086183,
+            0.001479503476855694,
+            -0.012645905858466798,
+            0.01216748847452763,
+            -0.013378877368040743,
+            -0.012053251571418766,
+            0.012508842412556306,
+            -0.00027167241745418584,
+            0.01682680930386839,
+            -0.020607299598265426,
+            0.0010702151063620052,
+            -0.028075439824932304,
+            -0.007671848153932823,
+            -0.015486630894165054,
+            -0.0009776136330459065,
+            -0.0012939921569540803,
+            -0.03481880531471923,
+            0.012889447665794033,
+            -0.005200149829198305,
+            -0.006094702500154477,
+            -0.00459778303079732,
+            -0.024583504817899775,
+            0.006072281238752663,
+            0.00607350167867964,
+            -0.006506508970816635,
+            -0.016816439635445445,
+            0.008428048142588516,
+            -0.0011040593013238872,
+            0.004873670535200424,
+            -0.007475800310154908,
+            -0.0005638920132076605,
+            -0.013047135093134946,
+            -0.005463454269764494,
+            -0.0008522562324122834,
+            0.01827123304871433,
+            0.0044055851550317835,
+            -0.014526381222031981,
+            0.016862445236481526,
+            0.013250692156820124,
+            0.01602714293656341,
+            -0.0071089293427753115,
+            -0.0004265739315250755,
+            -0.02750630108787212,
+            -0.00327225092954903,
+            -0.023687644154808972,
+            0.00041213204365507744,
+            0.009447660230507965,
+            0.0203905264889617,
+            0.012694421477479161,
+            0.004620617778996077,
+            -0.001557511764988156,
+            0.0066751090724502,
+            0.02696959560423252,
+            -0.01099183126742224,
+            -0.006969111598423376,
+            -0.01884504192789845,
+            0.01113023717919988,
+            0.025135748541199255,
+            0.003381303139273845,
+            -0.011317957771935042,
+            0.008112598754655146,
+            -0.0009247845526290428,
+            -0.010302980598863016,
+            0.02889587382004902,
+            -0.0391134481322538,
+            -0.015599591108049343,
+            -0.004605824192962023,
+            0.007406520608110251,
+            -0.015596917115435033,
+            0.00770468362805995,
+            -0.012614339656105264,
+            0.001699049329817941,
+            -0.005851477164035241,
+            0.03535087340659861,
+            -0.0031950960727572925,
+            0.0006909856233673865,
+            -0.003395233050301427,
+            0.003531135044131077,
+            0.012355948349937309,
+            -0.004444221317023453,
+            -0.030726300006909638,
+            -0.0022983766869702346,
+            -0.011198252640110788,
+            0.009274666307515955,
+            -0.016207714291658314,
+            -0.017342435976846468,
+            -0.009424737744066259,
+            -0.021827243671837953,
+            -0.012403084982789827,
+            0.029747453318486338,
+            -0.005824001042844527,
+            0.01747569398142269,
+            -0.005219060548639168,
+            -0.008365003057800934,
+            0.00021272411948388247,
+            0.01344335658390732,
+            -0.028498240040284714,
+            0.00041963499785238845,
+            -0.002322931702641349,
+            0.0126393550661767,
+            -0.001595134597082556,
+            -0.00983073906977773,
+            0.006903313245694627,
+            0.0010522208685011098,
+            -0.015648940549713764,
+            -0.0007573245570947632,
+            -0.008533504878052644,
+            0.020475329445149498,
+            0.00027791387213409814,
+            0.001074213629646005,
+            0.013864331671732542,
+            -0.02102352993295736,
+            0.01576938071930521,
+            -0.021722775924283996,
+            0.01893267442161547,
+            -0.006577103101038035,
+            0.005772016664149475,
+            0.008283223343661555,
+            0.014068789332250016,
+            -0.014030042849491258,
+            -0.00663752723080222,
+            -0.034436220972634726,
+            -0.0122621879114314,
+            -0.001946072976826902,
+            -0.005308941654794638,
+            -0.014773123927909993,
+            -0.0011054822266082056,
+            -0.026205461509573584,
+            -0.006460373065662098,
+            0.012822188046808264,
+            0.019900312098359022,
+            -0.008409070094265829,
+            0.014163132434889265,
+            -0.006743711872055091,
+            -0.0028230771810882492,
+            -0.012652021749331811,
+            -0.018061180380009395,
+            -0.01083672324723357,
+            -0.01890966636827258,
+            -0.0018635408070944168,
+            0.004709222372611061,
+            -0.015760618199235518,
+            0.011435131737642627,
+            0.03198874888333405,
+            -0.01567271472650171,
+            0.008243406188694909,
+            0.016411496328333205,
+            0.0034236971031973503,
+            0.011728079139322837,
+            0.0038311415233052795,
+            0.004621789116267879,
+            -0.011109450058055078,
+            -0.011473158133621661,
+            0.006524490579223986,
+            0.00954511918454185,
+            -0.004913142601134428,
+            0.024347072255007807,
+            0.0008745419206472714,
+            0.005045794664611216,
+            0.005304506902446091,
+            0.011488187043525824,
+            -0.013135271908307747,
+            0.019746484188715552,
+            -0.002055210086078267,
+            -0.015571665869889405,
+            -0.01535499169234956,
+            -0.01105419264826655,
+            -0.018626399757822328,
+            0.0017108083455869296,
+            0.0181439079855921,
+            -0.0012493542718277685,
+            0.012137753522284387,
+            -0.00924460959150499,
+            -0.005019437549649743,
+            -0.0027156377206998474,
+            -0.038211954658821295,
+            -0.0021878970332906457,
+            -0.018809487178512628,
+            -0.02498945889484935,
+            0.03505216568400238,
+            -4.26477081213836e-05,
+            0.00997643255528116,
+            -0.010310197159232386,
+            -0.0002454902098986129,
+            -0.011065403358802515,
+            0.014954998647970473,
+            0.004624123403100598,
+            0.008872923678983478,
+            0.008560280683876702,
+            -0.016419500515801093,
+            -0.002997557798375716,
+            0.007472390947008284,
+            -0.00374822278027057,
+            0.012680407710694952,
+            0.015353684671120731,
+            0.020886474047003733,
+            0.011161262397502876,
+            0.002045862158732102,
+            0.00887271411363814,
+            0.009563361150297699,
+            -0.0003513237086717377,
+            -0.010459621030424415,
+            0.015419039131459582,
+            -0.02204848739950703,
+            0.012172257374410352,
+            0.04346634265794738,
+            0.019197433209794286,
+            0.029488039318675732,
+            -0.02272810439246353,
+            0.01448092574459596,
+            -0.003801043409441206,
+            0.009476229966122152,
+            0.0049309968883087685,
+            0.02346191840355209,
+            0.003973840862678076,
+            0.0073546443568171405,
+            0.010647581332452525,
+            -0.00972051412996416,
+            0.025644022498404723,
+            -0.00846669916801108,
+            5.1518540791212095e-05,
+            -0.004407265804145216,
+            -0.026400029105519137,
+            0.01615193008014211,
+            0.0081722536401398,
+            0.0010702314763197463,
+            0.0010263623479474385,
+            -0.0019566604229607643,
+            0.007982949510899019,
+            0.007079322685771456,
+            -0.015274148711633536,
+            -0.003886053737590683,
+            -0.018527455865296014,
+            0.0012864569064749438,
+            -0.024337133883771334,
+            0.010466651250597847,
+            -0.017568706814132484,
+            -0.014822354159709854,
+            -0.021666681078831483,
+            -0.013232016867264097,
+            -0.02011653224168991,
+            0.0035858355517491515,
+            -0.008632755352062642,
+            0.013025728891203198,
+            -0.016962939287139683,
+            -0.021494971929779273,
+            0.008430887589398292,
+            0.00870802359711456,
+            0.007481282677955534,
+            0.009728368603458082,
+            0.014996157358749115,
+            0.01736292711410622,
+            -0.022712386143092265,
+            -0.023163001611599246,
+            -0.006682209717364464,
+            -0.003531495447167166,
+            0.006762379833125297,
+            0.00933681196321128,
+            0.012884185500343249,
+            -0.007077895849501418,
+            -0.007554762054641417,
+            -0.013283067570021503,
+            0.006197966297581871,
+            -0.003993067576677973,
+            -0.009464789349849612,
+            -0.0015662739825834614,
+            -0.0061389795593180686,
+            0.002753664732282926,
+            -0.0001791088888074743,
+            0.0007595261237141144,
+            0.017122834676182343,
+            -0.008176319370751018,
+            -0.009787747261447871,
+            -0.02027228453943018,
+            -0.010683273300243025,
+            -0.015604745521156111,
+            -0.0016279406232527138,
+            -0.006506713191861036,
+            -0.014420323628589382,
+            0.010888206569219648,
+            -0.011930708571093279,
+            -0.00887210117616784,
+            0.010807147705360856,
+            0.0073900910661282935,
+            -0.003431679006289002,
+            0.037804020313161775,
+            0.0011945597874820288,
+            0.00778778394048267,
+            -0.003366543306746973,
+            0.005296727102519836,
+            -0.017761251541598876,
+            -0.026382400396890007,
+            0.0010357716464853195,
+            -0.01775777668381489,
+            0.010807713750049491,
+            -0.008515368407459745,
+            0.0016051389544388124,
+            -0.00034958600424912995,
+            -0.009585854918521076,
+            0.0036549720706351473,
+            -0.008110927907950148,
+            0.004212219431447508,
+            -0.03507890639477636,
+            0.011409377585617926,
+            -0.01819120811524444,
+            -0.008962242411035718,
+            0.01710177594046791,
+            0.037560936862121835,
+            0.0028513388998398104,
+            0.00030058353811717895,
+            0.0014792184333546447,
+            0.004427921639326168,
+            0.021438127785303025,
+            0.006339483221420334,
+            -0.001187747087595264,
+            -0.02237982599829809,
+            0.0062435650294483695,
+            0.002200578793697352,
+            0.010092828474934744,
+            0.023105023225115266,
+            -0.016797326021070008,
+            -0.015378520287395565,
+            -0.012732527281000097,
+            -0.010797463778182385,
+            0.009881499418260473,
+            0.02801767062785128,
+            0.01811139137849365,
+            -0.017358098300028903,
+            -0.020207173948076473,
+            0.0016242901745373126,
+            -0.026331540602868762,
+            0.007269751661479859,
+            0.11940631114242901,
+            -0.0045382135734049305,
+            -0.005994407567723939,
+            -0.008679712231646732,
+            -0.004482553208064535,
+            0.016709571294385293,
+            -0.013387043582971358,
+            0.069179848974036,
+            -0.007221545718220735,
+            0.0037392229364642915,
+            0.0028916224964057385,
+            0.00462729158956839,
+            -0.009727339449635306,
+            0.011570145487622916,
+            -0.011287365791681587,
+            -0.012991994493845187,
+            -0.013076194601468383,
+            -0.0180830261130804,
+            -0.007272768410218907,
+            -0.004155564814758654,
+            0.004204993170487392,
+            -0.005525575983182816,
+            0.008051108079520902,
+            -0.017551645696802648,
+            -0.010800070994485792,
+            0.013007176804761548,
+            -0.0023627485852619817,
+            0.005390986239687555,
+            -0.029668390278675554,
+            -0.0007650391259819867,
+            -0.0002446394046947245,
+            0.021280292985495608,
+            0.003137771886236958,
+            -0.0019077298815862946,
+            -0.010810159675564049,
+            0.007040107942856957,
+            0.023835993694726953,
+            0.021875366566457236,
+            0.0180511566130312,
+            0.012135351541044354,
+            0.020483975362807574,
+            -0.008667079096167644,
+            0.026324312888512866,
+            -0.001552207146726937,
+            -0.00397810465151493,
+            0.0077577136352257004,
+            -0.009849600842081875,
+            -0.00816392268697345,
+            0.00677353736462267,
+            -0.017288900282504148,
+            -0.020418071729964713,
+            -0.020383762873109595,
+            0.025741408097358233,
+            -0.011755094311186617,
+            0.002737129201159793,
+            -0.008266210469668866,
+            0.027287436144516414,
+            -0.006283892307996516,
+            -0.0029440642954701632,
+            0.009428876742171362,
+            -0.03788574758692254,
+            0.008834605607057885,
+            0.006463342392514337,
+            0.026711196829464153,
+            -3.9827332346130654e-05,
+            0.005565561841469247,
+            -0.0006738085601962865,
+            -0.006104320995853506,
+            -0.010792830169647851,
+            2.4565453208679464e-05,
+            0.017983908748172493,
+            0.0035181716362611077,
+            0.016970206528324653,
+            -0.005449808667944701,
+            -0.023171388778833197,
+            0.007092509502738752,
+            -0.01555320330471131,
+            -0.023306285898839978,
+            -0.002028783456704567,
+            -0.03196436161255752,
+            0.0033412618908711163,
+            0.007527633574188688,
+            -0.01890570959143866,
+            -0.017859735533232564,
+            0.010329340599904344,
+            0.004382396539605985,
+            -0.0016746397858953155,
+            0.006643785064338525,
+            -0.03307812909454471,
+            -0.01927475592253288,
+            0.004686970010943116,
+            0.012733525168279355,
+            -0.012993441126970916,
+            0.015291832952199082,
+            -0.020491167895268936,
+            0.016451976170508816,
+            -0.012259387189535237,
+            0.018393274368854132,
+            -0.006997668664226754,
+            0.011843450716200625,
+            0.011457415642860215,
+            0.009560877634897626,
+            -0.01790001115302499,
+            0.0214879415944361,
+            0.0066829070506954115,
+            0.03900547021123392,
+            0.02512484499576633,
+            -0.012651294056575184,
+            -0.0018425680226742782,
+            -0.00011361458320146007,
+            0.014982647266052349,
+            0.0061874424115559,
+            -0.003689464024837091,
+            0.02705902205734107,
+            0.018239469373522537,
+            -0.021070704644859155,
+            0.020037508334843183,
+            -0.00973804771680803,
+            0.018876283979591423,
+            -0.013303080480396818,
+            0.010592424494628194,
+            0.004717870260467274,
+            0.01070131238869639,
+            0.009087656015197565,
+            0.006708673669681232,
+            0.025928065153829043,
+            0.03663367449576964,
+            -0.00511073108390551,
+            0.002161381031392467,
+            0.004864092591944181,
+            0.012814518637357098,
+            0.001556833001747549,
+            0.007915766323215207,
+            -0.007012785514562176,
+            0.005834786334750152,
+            -0.002311533032619821,
+            -0.00122162928193159,
+            0.019525999280692766,
+            0.000824163362963678,
+            0.012875575199417254,
+            0.013259376059185821,
+            -0.002390133687709065,
+            -0.005682644446302278,
+            0.0019262218417782657,
+            -0.018193478045923098,
+            -0.015131696847666434,
+            0.029463723011600933,
+            -0.011447039651934148,
+            -0.008083114389964213,
+            0.0020642857121204753,
+            0.012160202510990566,
+            -0.00866944494225225,
+            -0.005378300635997713,
+            0.0038280364584418335,
+            0.014427318067525318,
+            -0.01799358399064332,
+            0.003381165206832161,
+            0.005252552821854107,
+            0.019433554824232185,
+            -0.00028791985674147885,
+            -0.013487678738391956,
+            0.0005317922815671783,
+            0.014982152801567637,
+            0.011902348165369078,
+            0.0016056390299304837,
+            -0.006935777197137747,
+            -0.00446584737370504,
+            0.004586814866182051,
+            0.006269466956110759,
+            0.0027516492952993445,
+            0.008157662045793032,
+            -0.0025001885993292637,
+            -0.016648747436203294,
+            0.017735757469839637,
+            -0.0030669873770852725,
+            0.018058248627079793,
+            -0.0038238312452176745,
+            -0.0147172470474118,
+            0.0043328417868095355,
+            -0.001888507803054499,
+            -0.018556943283494755,
+            0.009407930200720534,
+            0.002560949424273885,
+            0.010411159836451958,
+            0.0052571230238294825,
+            -0.007499470181994448,
+            -6.145889479758644e-05,
+            -0.015163819825329975,
+            0.022836682800304944,
+            0.016390944816242627,
+            0.001713572907204073,
+            -0.0035687505973373878,
+            -0.031210299824387475,
+            0.0029076511444359886,
+            -0.007822231980296341,
+            0.017821806420465356,
+            0.008250605126658402,
+            0.007908773724009905,
+            0.008351355849444162,
+            -0.00743970642051451,
+            -0.002411981861500092,
+            0.012543476434788293,
+            0.0202407777715137,
+            -0.0024635818448627447,
+            0.02535771413579989,
+            -0.012533449594824386,
+            -0.0022489292647974992,
+            0.0012362917358624392,
+            0.01578939075186126,
+            -0.0008001783766371555,
+            0.006311837812042848
+          ],
+          "keyphrases": [
+            "Switch Transformers",
+            "Mixture of Experts",
+            "sparsely-activated model",
+            "training instability",
+            "multilingual settings"
+          ],
+          "title": "Switch Transformers: Scaling To Trillion Parameter Models With Simple And Efficient Sparsity"
+        },
+        "type": "document"
+      },
+      "bidirectional": true,
+      "properties": {
+        "summary_cosine_similarity": 0.6420223818521993
+      }
+    }
+  ]
+}
\ No newline at end of file
diff --git a/tests/e2e/test_fullflow.py b/tests/e2e/test_fullflow.py
index a2bdc3bc9..3db616dd4 100644
--- a/tests/e2e/test_fullflow.py
+++ b/tests/e2e/test_fullflow.py
@@ -1,14 +1,19 @@
 from datasets import load_dataset
 
-from ragas import evaluate
-from ragas.metrics import answer_relevancy, context_precision, faithfulness
-from ragas.metrics.critique import harmfulness
+from ragas import EvaluationDataset, evaluate
+from ragas.llms import llm_factory
+from ragas.metrics import AnswerRelevancy, ContextPrecision, Faithfulness
 
 
 def test_evaluate_e2e():
-    ds = load_dataset("explodinggradients/fiqa", "ragas_eval")["baseline"]
+    dataset = load_dataset("explodinggradients/amnesty_qa", "english_v3")
+    eval_dataset = EvaluationDataset.from_hf_dataset(dataset["eval"])  # type: ignore
     result = evaluate(
-        ds.select(range(3)),
-        metrics=[answer_relevancy, context_precision, faithfulness, harmfulness],
+        eval_dataset,
+        metrics=[
+            AnswerRelevancy(llm=llm_factory()),
+            ContextPrecision(llm=llm_factory()),
+            Faithfulness(llm=llm_factory()),
+        ],
     )
     assert result is not None
diff --git a/tests/e2e/test_testset_generation.py b/tests/e2e/test_testset_generation.py
new file mode 100644
index 000000000..fb682b3f6
--- /dev/null
+++ b/tests/e2e/test_testset_generation.py
@@ -0,0 +1,34 @@
+import os
+import uuid
+
+from ragas.testset import TestsetGenerator
+
+
+def test_testset_generation():
+    from ragas.llms import llm_factory
+    from ragas.testset.graph import KnowledgeGraph
+
+    kg = KnowledgeGraph.load(
+        os.path.join(os.path.dirname(__file__), "scratchpad_kg.json")
+    )
+    tg = TestsetGenerator(llm=llm_factory(), knowledge_graph=kg)
+    testset = tg.generate(testset_size=10)
+    assert testset is not None
+
+
+def test_transforms():
+    from ragas.embeddings import embedding_factory
+    from ragas.llms import llm_factory
+    from ragas.testset.graph import KnowledgeGraph, Node
+    from ragas.testset.transforms import apply_transforms, default_transforms
+
+    transforms = default_transforms(
+        llm=llm_factory(), embedding_model=embedding_factory()
+    )
+
+    kg = KnowledgeGraph()
+    kg.nodes.append(Node(id=uuid.uuid4(), properties={"page_content": "Hello, world!"}))
+    assert len(kg.nodes) == 1
+
+    apply_transforms(kg, transforms)
+    assert len(kg.nodes) == 1